wikiformat.py:
"""Tokenize WikiFormat web pages.
A WikiFormat is a special kind of data format. The data format ONLY
relies on (1) bold text, (2) italic text, and (3) hyperlinks. All wiki
support these three things.
This module tokenizes a web page's bold text, italic text, and
hyperlinks. That way, it's easier to make new WikiFormats.
WikiFormatParser -- fill list with tokens parsed from HTML
tokenize_string -- return tokens from text
tokenize_webpage -- return tokens from URL
"""
import urllib
import HTMLParser
BOLD = "B"
ITALIC = "I"
A_HREF = "A_HREF"
class WikiFormatParser(HTMLParser.HTMLParser):
"""Interpret XHTML tag events and record WikiFormat tokens.
A SAX-like handler for XHTML events.
The handler tokenizes bold-tagged text, italic text, and href URL
targets.
"""
def __init__(self, tokens):
HTMLParser.HTMLParser.__init__(self)
self.tokens = tokens
self.last_text = ""
def handle_starttag(self, tag, attrs):
self.last_text = ""
if tag == "a":
attrs = dict(attrs)
if attrs.has_key("href"):
self.tokens.append((A_HREF, attrs["href"]))
def handle_endtag(self, tag):
if tag in ["i", "em"]:
self.tokens.append((ITALIC, self.last_text))
elif tag in ["b", "strong"]:
self.tokens.append((BOLD, self.last_text))
self.last_text = ""
def handle_data(self, data):
self.last_text = self.last_text + data
def tokenize_string(text):
"""Return WikiFormat tokens taken from text."""
tokens = []
parser = WikiFormatParser(tokens)
parser.feed(text)
parser.close()
return tokens
def tokenize_webpage(url):
"""Return WikiFormat tokens taken from a web page."""
return tokenize_string(urllib.urlopen(url).read())
if __name__ == "__main__":
print tokenize_webpage("http://communitywiki.org/WikiFormats")Very simple.
Output is something like:
[('A_HREF', 'http://www.emacswiki.org/cw/SiteMap'),
('A_HREF', 'http://www.emacswiki.org/cw/SiteMap'),
('A_HREF', 'http://www.emacswiki.org/cw/RecentChanges'),
('A_HREF', 'http://www.emacswiki.org/cw/RecentNearChanges'),
('A_HREF', 'http://www.emacswiki.org/cw/HowTo'),
('A_HREF', 'http://www.emacswiki.org/cw/Blog'),
('A_HREF', 'http://www.emacswiki.org/cw/2005-04-23'),
('A_HREF', 'http://www.emacswiki.org/cw/2005-04-22'),
('A_HREF', 'http://www.emacswiki.org/cw/atom'),
('A_HREF', 'http://www.emacswiki.org/cw/rss'),
('A_HREF', 'http://www.emacswiki.org/cw?action=rc;raw=1'),
('A_HREF', 'http://www.emacswiki.org/cw?search=WikiFormats'),
('A_HREF', 'http://www.emacswiki.org/cw/MicroFormats'),
('A_HREF', 'http://www.emacswiki.org/cw/MachineCodeBlocks'),
('I', '3 things'),
('B', 'bold text'),
('I', 'italic text'),
('A_HREF', 'http://communitywiki.org/WikiFormats'),
('A_HREF', 'http://www.emacswiki.org/cw?action=edit;id=MicroFormat'),
('B', 'VCARD'),
('B', '(any key)'),
('I', '(any value)'),
('B', 'VCARD'),
('B', '(any key)'),
('A_HREF', 'http://communitywiki.org/WikiFormats'),
('I', 'is'),
('A_HREF',
'http://www.emacswiki.org/cgi-bin/latin-1.pl?url=http://www.usemod.com/cgi-bin/mb.pl?WikiEngine'),
('I', 'text.'),
('I', 'the linked URL.'),
('B', 'WIKINODE'),
('B', 'neighbor'),
('A_HREF',
'http://www.emacswiki.org/cgi-bin/latin-1.pl?url=http://www.usemod.com/cgi-bin/mb.pl?WikiNode'),
('B', 'neighbor'),
('A_HREF', 'http://onebigsoup.wiki.taoriver.net/moin.cgi/WikiNode'),
('B', 'neighbor'),
('A_HREF', 'http://is-root.de/wiki/index.php/WikiNode'),
('B', 'WIKINODE'),
('A_HREF',
'http://www.emacswiki.org/cgi-bin/latin-1.pl?url=http://www.usemod.com/cgi-bin/mb.pl?WardsWiki'),
('A_HREF',
'http://www.emacswiki.org/cgi-bin/latin-1.pl?url=http://www.usemod.com/cgi-bin/mb.pl?WardsWiki'),
('I', 'plain text in series,'),
('A_HREF', 'http://www.emacswiki.org/cw/MachineCodeBlocks'),
('A_HREF', 'http://www.emacswiki.org/cw/WikiFormats'),
('A_HREF', 'http://www.emacswiki.org/cw/LionKimbro'),
('A_HREF', 'http://www.emacswiki.org/cw/LionKimbro'),
('A_HREF', 'http://www.emacswiki.org/cw?action=edit;id=WikiFormat'),
('I', 'and'),
('A_HREF', 'http://www.emacswiki.org/cw/LionKimbro'),
('A_HREF', 'http://www.emacswiki.org/cw/LionKimbro'),
('A_HREF', 'http://www.emacswiki.org/cw?action=edit;id=WikiFormatsTokenizer'),
('A_HREF', 'http://www.emacswiki.org/cw/SiteMap'),
('A_HREF', 'http://www.emacswiki.org/cw/RecentChanges'),
('A_HREF', 'http://www.emacswiki.org/cw/RecentNearChanges'),
('A_HREF', 'http://www.emacswiki.org/cw/HowTo'),
('A_HREF', 'http://www.emacswiki.org/cw/Blog'),
('A_HREF', 'http://www.emacswiki.org/cw/2005-04-23'),
('A_HREF', 'http://www.emacswiki.org/cw/2005-04-22'),
('A_HREF', 'http://www.emacswiki.org/cw/atom'),
('A_HREF', 'http://www.emacswiki.org/cw/rss'),
('A_HREF', 'http://www.emacswiki.org/cw?action=rc;raw=1'),
('A_HREF', 'http://www.emacswiki.org/cw?action=edit;id=WikiFormats'),
('A_HREF', 'http://www.emacswiki.org/cw?action=history;id=WikiFormats'),
('A_HREF', 'http://www.emacswiki.org/cw?action=admin;id=WikiFormats'),
('A_HREF', 'http://www.emacswiki.org/cw/LionKimbro'),
('A_HREF', 'http://www.emacswiki.org/cw?action=browse;diff=1;id=WikiFormats'),
('A_HREF', 'http://creativecommons.org/licenses/sa/1.0/'),
('A_HREF', 'http://creativecommons.org/licenses/sa/1.0/'),
('A_HREF', '/FDL'),
('A_HREF', 'http://www.fsf.org/'),
('A_HREF', '/cgi-bin/community/CommunityWikiLicense'),
('A_HREF', 'http://www.emacswiki.org/cw/EditNearLinks'),
('A_HREF', 'http://www.emacswiki.org/cw?action=edit;id=WardsWiki'),
('A_HREF', 'http://www.emacswiki.org/cw?action=edit;id=WikiEngine')]Highlighting the WikiNode block in the above:
('B', 'WIKINODE'),
('B', 'neighbor'),
('A_HREF',
'http://www.emacswiki.org/cgi-bin/latin-1.pl?url=http://www.usemod.com/cgi-bin/mb.pl?WikiNode'),
('B', 'neighbor'),
('A_HREF', 'http://onebigsoup.wiki.taoriver.net/moin.cgi/WikiNode'),
('B', 'neighbor'),
('A_HREF', 'http://is-root.de/wiki/index.php/WikiNode'),
('B', 'WIKINODE'),It should be simple to make little WikiFormats on top of such a tool.
Do you have a good way to avoid the page-wrap from the HTML, so that only page data is considered? Or are you completely relying on the WIKINODE markers? I've been adding semi-structured data to some pages in a wiki I've been working on, using ReStructuredText tables with a .. class: some-interesting-class-name before the table; adding a sample table and the .. class: markup in a template page makes it easy to see how to use a particular structure. But I've not actually needed to extract the data in a script (yet). --FredDrake