I'm going to use the "FIFIO API" term from now on...
For a while I scraped Ex Libris's status webpage (http://status.exlibrisgroup.com/) to monitor the status of our catalogue/discovery layer instances and, in the event of an issue, automatically put a notice on the discovery layer to tell users what was wrong, that it was being worked on, and what other options they had in the meantime.
Eventually Ex Libris created a RESTful API with the same content so I've gratefully upgraded my code to use that instead.
From: Code for Libraries [mailto:[log in to unmask]] On Behalf Of Jason Bengtson
Sent: Wednesday, 29 November 2017 8:00 a.m.
To: [log in to unmask]
Subject: Re: [CODE4LIB] Anyone web scraping to benefit their library?
I use web scraping sometimes to extract data from systems that lack APIs.
I'm doing this to get current library job openings from our University jobs application, for instance. I use the structure of their website in a way similar to an API query, scrape the results, and extract only what I need.
I jokingly call it a FIFIO API (Fine, I'll Figure It Out). Obviously, such a tool is inherently unstable, and has to be closely managed. When used with things like the jobs application, which maintain a relatively stable uri structure over time, however, it can be a pretty good tool when you have nothing else. I also used screen scraping as part of a tool I built years ago to allow authorized staff to create announcements within a special libguide that they then pushed to the EZ Proxy login page. I wrote
a book chapter on that one: "Leveraging LibGuides as an EZProxy
Notifications Interface." Innovative Libguides Applications: Real World Examples. New York: Rowman & Littlefield, 2016
On Tue, Nov 28, 2017 at 12:24 PM, Brad Coffield <[log in to unmask]
> I think there's likely a lot of possibilities out there and was hoping
> to hear examples of web scraping for libraries. Your example might
> just inspire me or another reader to do something similar. At the very
> least, the ideas will be interesting!
> Brad Coffield, MLIS
> Assistant Information and Web Services Librarian Saint Francis
> [log in to unmask]
P Please consider the environment before you print this email.
"The contents of this e-mail (including any attachments) may be confidential and/or subject to copyright. Any unauthorised use, distribution, or copying of the contents is expressly prohibited. If you have received this e-mail in error, please advise the sender by return e-mail or telephone and then delete this e-mail together with all attachments from your system."