On Thu, May 13, 2004 at 08:27:01AM -0500, Eric Lease Morgan wrote:
> On May 12, 2004, at 9:44 PM, Chuck Bearden wrote:
>
> >>my main problem right now is the parsing....ugh, ugh...ugh....
> >
> >Boy howdy!  It's like a rat's nest sometimes.
>
> Screen scraping is for the birds!!

Albatross!  Albatross!  Bloody albatross!

Seriously, I do it as little as possible.  It happens to be the best way
forward for the project I'm involved in.  I don't have the luxury of
saying that I don't need data from the USPTO, and that I'll just wait
for a proper OAI server or XML-RPC interface to it.  When I compare it
to being able to harvest valid XML records in batches of 5000 from
PubMed, well then I do feel the albatross around my neck.

That being said, I have found that web pages of records generated from
databases and templates *are* indeed structured data, it's just that
they are (to borrow a PC phrase) "differently structured".  The HTML may
be crappy, but it's crappy in a consistent way, and it can be fixed
and navigated in a consistent way.  Necessity is the mother of invention
and the midwife of screen-scraping.

Chuck