Übersicht Letzte Änderungen Letzte Änderungen in der Umgebung KurzAnleitung Blog RSS

WikiFormatsTokenizer

wikiformat.py:

 """Tokenize WikiFormat web pages.
 
 A WikiFormat is a special kind of data format. The data format ONLY
 relies on (1) bold text, (2) italic text, and (3) hyperlinks. All wiki
 support these three things.
 
 This module tokenizes a web page's bold text, italic text, and
 hyperlinks. That way, it's easier to make new WikiFormats.
 
 WikiFormatParser  -- fill list with tokens parsed from HTML
 tokenize_string  -- return tokens from text
 tokenize_webpage  -- return tokens from URL
 """
 
 import urllib
 
 import HTMLParser
 
 
 BOLD = "B"
 ITALIC = "I"
 A_HREF = "A_HREF"
 
 
 class WikiFormatParser(HTMLParser.HTMLParser):
 
     """Interpret XHTML tag events and record WikiFormat tokens.
 
     A SAX-like handler for XHTML events.
 
     The handler tokenizes bold-tagged text, italic text, and href URL
     targets.
     """
 
     def __init__(self, tokens):
         HTMLParser.HTMLParser.__init__(self)
         self.tokens = tokens
         self.last_text = ""
 
     def handle_starttag(self, tag, attrs):
         self.last_text = ""
         if tag == "a":
             attrs = dict(attrs)
             if attrs.has_key("href"):
                 self.tokens.append((A_HREF, attrs["href"]))
 
     def handle_endtag(self, tag):
         if tag in ["i", "em"]:
             self.tokens.append((ITALIC, self.last_text))
         elif tag in ["b", "strong"]:
             self.tokens.append((BOLD, self.last_text))
         self.last_text = ""
 
     def handle_data(self, data):
         self.last_text = self.last_text + data
 
 
 def tokenize_string(text):
     """Return WikiFormat tokens taken from text."""
     tokens = []
     parser = WikiFormatParser(tokens)
     parser.feed(text)
     parser.close()
     return tokens
 
 
 def tokenize_webpage(url):
     """Return WikiFormat tokens taken from a web page."""
     return tokenize_string(urllib.urlopen(url).read())
 
 
 if __name__ == "__main__":
     print tokenize_webpage("http://communitywiki.org/WikiFormats")

Very simple.

Output is something like:

 [('A_HREF', 'http://www.emacswiki.org/cw/SiteMap'),
  ('A_HREF', 'http://www.emacswiki.org/cw/SiteMap'),
  ('A_HREF', 'http://www.emacswiki.org/cw/RecentChanges'),
  ('A_HREF', 'http://www.emacswiki.org/cw/RecentNearChanges'),
  ('A_HREF', 'http://www.emacswiki.org/cw/HowTo'),
  ('A_HREF', 'http://www.emacswiki.org/cw/Blog'),
  ('A_HREF', 'http://www.emacswiki.org/cw/2005-04-23'),
  ('A_HREF', 'http://www.emacswiki.org/cw/2005-04-22'),
  ('A_HREF', 'http://www.emacswiki.org/cw/atom'),
  ('A_HREF', 'http://www.emacswiki.org/cw/rss'),
  ('A_HREF', 'http://www.emacswiki.org/cw?action=rc;raw=1'),
  ('A_HREF', 'http://www.emacswiki.org/cw?search=WikiFormats'),
  ('A_HREF', 'http://www.emacswiki.org/cw/MicroFormats'),
  ('A_HREF', 'http://www.emacswiki.org/cw/MachineCodeBlocks'),
  ('I', '3 things'),
  ('B', 'bold text'),
  ('I', 'italic text'),
  ('A_HREF', 'http://communitywiki.org/WikiFormats'),
  ('A_HREF', 'http://www.emacswiki.org/cw?action=edit;id=MicroFormat'),
  ('B', 'VCARD'),
  ('B', '(any key)'),
  ('I', '(any value)'),
  ('B', 'VCARD'),
  ('B', '(any key)'),
  ('A_HREF', 'http://communitywiki.org/WikiFormats'),
  ('I', 'is'),
  ('A_HREF',
   'http://www.emacswiki.org/cgi-bin/latin-1.pl?url=http://www.usemod.com/cgi-bin/mb.pl?WikiEngine'),
  ('I', 'text.'),
  ('I', 'the linked URL.'),
  ('B', 'WIKINODE'),
  ('B', 'neighbor'),
  ('A_HREF',
   'http://www.emacswiki.org/cgi-bin/latin-1.pl?url=http://www.usemod.com/cgi-bin/mb.pl?WikiNode'),
  ('B', 'neighbor'),
  ('A_HREF', 'http://onebigsoup.wiki.taoriver.net/moin.cgi/WikiNode'),
  ('B', 'neighbor'),
  ('A_HREF', 'http://is-root.de/wiki/index.php/WikiNode'),
  ('B', 'WIKINODE'),
  ('A_HREF',
   'http://www.emacswiki.org/cgi-bin/latin-1.pl?url=http://www.usemod.com/cgi-bin/mb.pl?WardsWiki'),
  ('A_HREF',
   'http://www.emacswiki.org/cgi-bin/latin-1.pl?url=http://www.usemod.com/cgi-bin/mb.pl?WardsWiki'),
  ('I', 'plain text in series,'),
  ('A_HREF', 'http://www.emacswiki.org/cw/MachineCodeBlocks'),
  ('A_HREF', 'http://www.emacswiki.org/cw/WikiFormats'),
  ('A_HREF', 'http://www.emacswiki.org/cw/LionKimbro'),
  ('A_HREF', 'http://www.emacswiki.org/cw/LionKimbro'),
  ('A_HREF', 'http://www.emacswiki.org/cw?action=edit;id=WikiFormat'),
  ('I', 'and'),
  ('A_HREF', 'http://www.emacswiki.org/cw/LionKimbro'),
  ('A_HREF', 'http://www.emacswiki.org/cw/LionKimbro'),
  ('A_HREF', 'http://www.emacswiki.org/cw?action=edit;id=WikiFormatsTokenizer'),
  ('A_HREF', 'http://www.emacswiki.org/cw/SiteMap'),
  ('A_HREF', 'http://www.emacswiki.org/cw/RecentChanges'),
  ('A_HREF', 'http://www.emacswiki.org/cw/RecentNearChanges'),
  ('A_HREF', 'http://www.emacswiki.org/cw/HowTo'),
  ('A_HREF', 'http://www.emacswiki.org/cw/Blog'),
  ('A_HREF', 'http://www.emacswiki.org/cw/2005-04-23'),
  ('A_HREF', 'http://www.emacswiki.org/cw/2005-04-22'),
  ('A_HREF', 'http://www.emacswiki.org/cw/atom'),
  ('A_HREF', 'http://www.emacswiki.org/cw/rss'),
  ('A_HREF', 'http://www.emacswiki.org/cw?action=rc;raw=1'),
  ('A_HREF', 'http://www.emacswiki.org/cw?action=edit;id=WikiFormats'),
  ('A_HREF', 'http://www.emacswiki.org/cw?action=history;id=WikiFormats'),
  ('A_HREF', 'http://www.emacswiki.org/cw?action=admin;id=WikiFormats'),
  ('A_HREF', 'http://www.emacswiki.org/cw/LionKimbro'),
  ('A_HREF', 'http://www.emacswiki.org/cw?action=browse;diff=1;id=WikiFormats'),
  ('A_HREF', 'http://creativecommons.org/licenses/sa/1.0/'),
  ('A_HREF', 'http://creativecommons.org/licenses/sa/1.0/'),
  ('A_HREF', '/FDL'),
  ('A_HREF', 'http://www.fsf.org/'),
  ('A_HREF', '/cgi-bin/community/CommunityWikiLicense'),
  ('A_HREF', 'http://www.emacswiki.org/cw/EditNearLinks'),
  ('A_HREF', 'http://www.emacswiki.org/cw?action=edit;id=WardsWiki'),
  ('A_HREF', 'http://www.emacswiki.org/cw?action=edit;id=WikiEngine')]

Highlighting the WikiNode block in the above:

  ('B', 'WIKINODE'),
  ('B', 'neighbor'),
  ('A_HREF',
   'http://www.emacswiki.org/cgi-bin/latin-1.pl?url=http://www.usemod.com/cgi-bin/mb.pl?WikiNode'),
  ('B', 'neighbor'),
  ('A_HREF', 'http://onebigsoup.wiki.taoriver.net/moin.cgi/WikiNode'),
  ('B', 'neighbor'),
  ('A_HREF', 'http://is-root.de/wiki/index.php/WikiNode'),
  ('B', 'WIKINODE'),

It should be simple to make little WikiFormats on top of such a tool.


Do you have a good way to avoid the page-wrap from the HTML, so that only page data is considered? Or are you completely relying on the WIKINODE markers? I've been adding semi-structured data to some pages in a wiki I've been working on, using ReStructuredText tables with a .. class: some-interesting-class-name before the table; adding a sample table and the .. class: markup in a template page makes it easy to see how to use a particular structure. But I've not actually needed to extract the data in a script (yet). --FredDrake

Diese Seiten lokal erstellen: ReStructuredText PatrickAnderson

Sprachen: