 |
 |
Yes, I did ask, and ask, and ask, and waited for 2 months. There was something political going on internally with that group that was well beyond my pay grade. I did explain the potential problems to my boss and she was providing cover. I did it in batches as Google Sheets limits the amount of ImportXML that you can do in a 24 hour span, so I wasn't hammering anyone's web server into oblivion. It's funny, I actually had to do a fair amount to get the old V1 LibGuides link checker to stop hammering my ILS into going offline back in 2010-2011. On Tue, Nov 28, 2017 at 2:18 PM, Bill Dueber <[log in to unmask]> wrote: > Brett, did you ask the folks at the Large University Library if they could > set something up for you? I don't have a good sense of how other > institutions deal with things like this. > > In any case, I know I'd much rather talk about setting up an API or a > nightly dump or something rather than have my analytics (and bandwidth!) > blown by a screen scraper. I might say "no," but at least it would be an > informed "no" :-) > > On Tue, Nov 28, 2017 at 2:08 PM, Brett <[log in to unmask]> wrote: > > > I leveraged the IMPORTXML() and xpath features in Google Sheets to pull > > information from a large university website to help create a set of > weeding > > lists for a branch campus. They needed extra details about what was in > > off-site storage and what was held at the central campus library. > > > > This was very much like Jason's FIFO API, the central reporting group had > > sent me a spreadsheet with horrible data that I would have had to sort > out > > almost completely manually, but the call numbers were pristine. I used > the > > call numbers as a key to query the catalog with limits for each campus I > > needed to check, and then it dumped all of the necessary content > (holdings, > > dates, etc) into the spreadsheet. > > > > I've also used Feed43 as a way to modify certain RSS feeds and scrape > > websites to only display the content I want. > > > > Brett Williams > > > > > > On Tue, Nov 28, 2017 at 1:24 PM, Brad Coffield < > > [log in to unmask]> > > wrote: > > > > > I think there's likely a lot of possibilities out there and was hoping > to > > > hear examples of web scraping for libraries. Your example might just > > > inspire me or another reader to do something similar. At the very > least, > > > the ideas will be interesting! > > > > > > Brad > > > > > > > > > -- > > > Brad Coffield, MLIS > > > Assistant Information and Web Services Librarian > > > Saint Francis University > > > 814-472-3315 > > > [log in to unmask] > > > > > > > > > -- > Bill Dueber > Library Systems Programmer > University of Michigan Library >