rob caSSon wrote:

>[snip]its very incomplete, but if anyone feels like taking a look, here it is:
>there is a tarball with the cgis, php scripts, and example result lists
>for the dbs
>
>i'm mainly posting this in case someone can figure out how better to
>parse some of the database search result lists (jstor has been
>particularly problematic)....oh, how i long for xml output.....
>
>the current scrapers i've got are ebsco, jstor, dataware (ohio-centric),
>and the III catalog....i'll tackle lexis, and a few others in the near
>future.....
>
>anyway, comments/help appreciated....i'm using perl, lwp,
>html::treebuilder, cgi, and uri, and my perl is rusty at best....
>
I did a couple of proof-of-concept things over the winter using php and
curl, which is available on a number of platforms.  I can't comment on
how well it performs relative to the configuration you're using but
libcurl is a smart,  reasonably well supported toolset.  In my
configuration, I attempted to derive a number of results from the first
screen back using a simple regular expression.  Failing that I simply
put up a "success" flag, largely based on the *absence* of the target's
variation on "no records could be found..." message.

In terms of the navigation metaphor [note to those using the archives:
this URL is subject to change]: try
   http://roy.halinet.on.ca/GreatLakes/Search/search.php
A search like "Chicora" returns a meaningful, but not overwhelming set
of results from a number of targets including BGSU in Ohio.

When I tackled Ebsco, I ran into issues of site authentication via
cookies that were passed to the search gateway but not on to the client
browser. Peter Binkley, at the University of Alberta recommended a proxy
configuration to balance off this issue.  Essentially those connections
would have to continue to operate inside a search gateway proxied session.

I don't know how the perl tools stack up in terms of parallel search
streams.  The php/curl combination is purely serial and the last targets
will time out if there is a tardy responder in the middle of the serial
queue.  Art Rhyno, at the University of Windsor, suggested a parallel
approach might be possible in a Cocoon environment.  This has the
advantage of passing all the inbound HTML pages through JTidy and giving
you the XHTML/XML compliant input stream you wanted (in most cases, even
when the output from the target was some distance from compliance).

Walter Lewis
Halton Hills