I'd suggest contacting the vendor. Whoever handles your the electronic subscriptions will have some relationship with a customer rep. Explain what you're trying to do and you might get the customer rep to advocate for you.
-Tod
On Nov 14, 2013, at 9:30 AM, Eric Lease Morgan <[log in to unmask]> wrote:
> Thank you for the replies, and after a bit of investigation I learned that I don’t need to do authentication because the vendor does IP authentication. Nice! On the other hand, I was still not able to resolve my original problem.
>
> I needed/wanted to download ten’s of thousands, if not hundred’s of thousands of citations for text mining analysis. The Web interface to the database/index limits output to 4,000 items and selecting the set of these items is beyond tedious — it is cruel and unusual punishment. I then got the idea of using EndNote’s z39.50 client, and after a bit of back & forth I got it working, but the downloading process was too slow. I then got the bright idea of writing my own z39.50 client (below). Unfortunately, I learned that the 4,000 record limit is more than that. A person can only download the first 4,000 records in a found set. Requests for record 4001, 4002, etc. fail. This is true in my locally written client as well as in EndNote.
>
> Alas, it looks as if I am unable to download the data I need/require, unless somebody at the vendor give me a data dump. On the other hand, since my locally written client is so short and simple, I think I can create a Web-based interface to query many different z39.50 targets and provide on-the-fly text mining analysis against the results.
>
> In short, I learned a great many things.
>
> —
> Eric Lease Morgan
> University of Notre Dame
>
>
> #!/usr/bin/perl
>
> # nytimes-search.pl - rudimentary z39.50 client to query the NY Times
>
> # Eric Lease Morgan <[log in to unmask]>
> # November 13, 2013 - first cut; "Happy Birthday, Steve!"
>
> # usage: ./nytimes-search.pl > nytimes.marc
>
>
> # configure
> use constant DB => 'hnpnewyorktimes';
> use constant HOST => 'fedsearch.proquest.com';
> use constant PORT => 210;
> use constant QUERY => '@attr 1=1016 "trade or tariff"';
> use constant SYNTAX => 'usmarc';
>
> # require
> use strict;
> use ZOOM;
>
> # do the work
> eval {
>
> # connect; configure; search
> my $conn = new ZOOM::Connection( HOST, PORT, databaseName => DB );
> $conn->option( preferredRecordSyntax => SYNTAX );
> my $rs = $conn->search_pqf( QUERY );
>
> # requests > 4000 return errors
> # print $rs->record( 4001 )->raw;
>
> # retrieve; will break at record 4,000 because of vendor limitations
> for my $i ( 0 .. $rs->size ) {
>
> print STDERR "\tRetrieving record #$i\r";
> print $rs->record( $i )->raw;
>
> }
>
> };
>
> # report errors
> if ( $@ ) { print STDERR "Error ", $@->code, ": ", $@->message, "\n" }
>
> # done
> exit;
