Haven't used the Hathi API before. Is multithreading possible or do
tech/policy constraints make that approach a nonstarter or otherwise not
worth pursuing?

Peeking the documentation, I noticed a htd:numpages element. If that is
usable, it would prevent the need to rely on errors to detect the document
end.

kyle

On Sun, Feb 10, 2019 at 5:51 PM Eric Lease Morgan <[log in to unmask]> wrote:

>
> I've finally figured out how to get raw OCR text out of the HathiTrust
> API, but it is really slow. Any hints out there?
>
> To use the HathiTrust Data API a person needs to first get a couple of
> access tokens. Applications then need to use the tokens to authenticate.
> Once this is done, a simple URL can be sent and cool stuff will be
> returned. For example, the following URL will return the first page of OCR:
>
>   https://babel.hathitrust.org/cgi/htd/volume/pageocr/uva.x000274833/1?v=2
>
> By continually incrementing the URL, other pages can be gotten:
>
>   https://babel.hathitrust.org/cgi/htd/volume/pageocr/uva.x000274833/2?v=2
>   https://babel.hathitrust.org/cgi/htd/volume/pageocr/uva.x000274833/3?v=2
>   https://babel.hathitrust.org/cgi/htd/volume/pageocr/uva.x000274833/4?v=2
>
> By incrementing the URL until an error is returned, one can get the whole
> of the document. I don't think there is a way to get the whole of the
> document in one go.
>
> Similarly, a person can get page images:
>
>
> https://babel.hathitrust.org/cgi/htd/volume/pageimage/uva.x000274833/1?v=2
>
> https://babel.hathitrust.org/cgi/htd/volume/pageimage/uva.x000274833/2?v=2
>
> https://babel.hathitrust.org/cgi/htd/volume/pageimage/uva.x000274833/3?v=2
>
> https://babel.hathitrust.org/cgi/htd/volume/pageimage/uva.x000274833/4?v=2
>
> Again, by incrementing the URL until an error is returned, all the images
> can be downloaded, and a PDF file could be created.
>
> By combining the traditional reading of a book (PDF) with the text mining
> of the OCR, very interesting things can take place. Thorough understanding
> could be obtained.
>
> Unfortunately, continually requesting individual pages seems laborious,
> not to mention,  s l o w .  It takes ten's of minutes to do the good work.
>
> Attached is the code I use to do the work. Can you suggest ways things
> could be sped up? Am I missing something when it comes to the API? Maybe if
> I do the work in a HathiTrust Research Center "capsule" things would be
> faster?
>
> --
> Eric Morgan
>
>
>
>
>