On Thu, May 13, 2004 at 08:27:01AM -0500, Eric Lease Morgan wrote:
> On May 12, 2004, at 9:44 PM, Chuck Bearden wrote:
> >>my main problem right now is the parsing....ugh, ugh...ugh....
> >Boy howdy! It's like a rat's nest sometimes.
> Screen scraping is for the birds!!
Albatross! Albatross! Bloody albatross!
Seriously, I do it as little as possible. It happens to be the best way
forward for the project I'm involved in. I don't have the luxury of
saying that I don't need data from the USPTO, and that I'll just wait
for a proper OAI server or XML-RPC interface to it. When I compare it
to being able to harvest valid XML records in batches of 5000 from
PubMed, well then I do feel the albatross around my neck.
That being said, I have found that web pages of records generated from
databases and templates *are* indeed structured data, it's just that
they are (to borrow a PC phrase) "differently structured". The HTML may
be crappy, but it's crappy in a consistent way, and it can be fixed
and navigated in a consistent way. Necessity is the mother of invention
and the midwife of screen-scraping.