On Wed, May 12, 2004 at 03:57:47PM -0400, rob caSSon wrote:
> my main problem right now is the parsing....ugh, ugh...ugh....
Boy howdy! It's like a rat's nest sometimes.
I haven't done HTML parsing in Perl for about a year now, but with
Python I am having good success with the microdom module from the
Twisted Framework[1]. The parseString() function, which returns a
largely DOM-like object, has an optional argument 'beExtremelyLenient',
which, if true, lets it handle a lot of crummy HTML. Usually I run the
HTML file through tidy first, and in some cases I need to do a little
manual fix-up before that. Something like this:
-------------------------Snippet-------------------------
import urllib
from mx.Tidy import tidy
from twisted.web import microdom
rawdoc = urllib.urlopen('http://foo.com/trash.html').read()
#-- fix up raw HTML here before passing to tidy, if necessary
tidydoc = tidy(rawdoc)[2]
domdoc = microdom.parseString(tidydoc, beExtremelyLenient=True)
#-- do stuff with quasi-DOM object
---------------------------------------------------------
I use Mozilla's DOM inspector as a visual guide to the DOM of
database-generated web pages, to work out the functions to get the
wanted parts of the document.
I'm using these techniques presently to scrape patent records from
the USPTO site.
Chuck
[1] http://www.twistedmatrix.com/
