 |
 |
There's a Very Large Catalog that includes the holdings of many libraries. Links from that VLC into our OPAC (aka Walter) assume that our MARCs include the relevant VLC #s (or sometimes an ISBN) which they don't always do (often because many of our VLC numbers appear to be obsolete). So, when Walter can't find anything for a given VLC# he screen scrapes the VLC to find out the title and author (if there is one) so that he can then send you into the OPAC with a keyword search that might work. I've been keeping a log of titles that have required this intervention and am happy to report that it works well. When it doesn't we now have the information we need to follow-up and remove our holdings from the Very Large Catalog. Guy <http://www.drew.edu/?utm_source=FIL_Email_Footer&utm_medium=email&utm_campaign=FIL%2BEmail%2BFooter> *Guy Dobson <http://www.drew.edu/directory/?q=email:gdobson&utm_source=FIL_Email_Footer&utm_medium=email&utm_campaign=FIL%2BEmail%2BFooter>* Systems Librarian | Library <http://www.drew.edu/library?utm_source=FIL_Email_Footer&utm_medium=email&utm_campaign=FIL%2BEmail%2BFooter> Drew University | 36 Madison Ave | Madison, NJ 07940 (973) 408-3207 | drew.edu <http://www.drew.edu/?utm_source=FIL_Email_Footer&utm_medium=email&utm_campaign=FIL%2BEmail%2BFooter> <http://www.drew.edu/undergraduate/?utm_source=FIL_Email_Footer&utm_medium=email&utm_campaign=FIL%2BEmail%2BFooter> On Tue, Nov 28, 2017 at 3:13 PM, Kenny Ketner <[log in to unmask]> wrote: > Brad et al, > > > We use wget scripts to back up our internet archive pages, which, oddly > enough, are the instructions given by internet archive itself. :/ > > > > Kenny Ketner > Information Products Lead > Montana State Library > 406-444-2870 > [log in to unmask] > kennyketner.com > > On Tue, Nov 28, 2017 at 12:31 PM, Brett <[log in to unmask]> > wrote: > > > Yes, I did ask, and ask, and ask, and waited for 2 months. There was > > something political going on internally with that group that was well > > beyond my pay grade. > > > > I did explain the potential problems to my boss and she was providing > > cover. > > > > I did it in batches as Google Sheets limits the amount of ImportXML that > > you can do in a 24 hour span, so I wasn't hammering anyone's web server > > into oblivion. > > > > It's funny, I actually had to do a fair amount to get the old V1 > LibGuides > > link checker to stop hammering my ILS into going offline back in > 2010-2011. > > > > > > > > On Tue, Nov 28, 2017 at 2:18 PM, Bill Dueber <[log in to unmask]> wrote: > > > > > Brett, did you ask the folks at the Large University Library if they > > could > > > set something up for you? I don't have a good sense of how other > > > institutions deal with things like this. > > > > > > In any case, I know I'd much rather talk about setting up an API or a > > > nightly dump or something rather than have my analytics (and > bandwidth!) > > > blown by a screen scraper. I might say "no," but at least it would be > an > > > informed "no" :-) > > > > > > On Tue, Nov 28, 2017 at 2:08 PM, Brett <[log in to unmask]> > > wrote: > > > > > > > I leveraged the IMPORTXML() and xpath features in Google Sheets to > pull > > > > information from a large university website to help create a set of > > > weeding > > > > lists for a branch campus. They needed extra details about what was > in > > > > off-site storage and what was held at the central campus library. > > > > > > > > This was very much like Jason's FIFO API, the central reporting group > > had > > > > sent me a spreadsheet with horrible data that I would have had to > sort > > > out > > > > almost completely manually, but the call numbers were pristine. I > used > > > the > > > > call numbers as a key to query the catalog with limits for each > campus > > I > > > > needed to check, and then it dumped all of the necessary content > > > (holdings, > > > > dates, etc) into the spreadsheet. > > > > > > > > I've also used Feed43 as a way to modify certain RSS feeds and scrape > > > > websites to only display the content I want. > > > > > > > > Brett Williams > > > > > > > > > > > > On Tue, Nov 28, 2017 at 1:24 PM, Brad Coffield < > > > > [log in to unmask]> > > > > wrote: > > > > > > > > > I think there's likely a lot of possibilities out there and was > > hoping > > > to > > > > > hear examples of web scraping for libraries. Your example might > just > > > > > inspire me or another reader to do something similar. At the very > > > least, > > > > > the ideas will be interesting! > > > > > > > > > > Brad > > > > > > > > > > > > > > > -- > > > > > Brad Coffield, MLIS > > > > > Assistant Information and Web Services Librarian > > > > > Saint Francis University > > > > > 814-472-3315 > > > > > [log in to unmask] > > > > > > > > > > > > > > > > > > > > > -- > > > Bill Dueber > > > Library Systems Programmer > > > University of Michigan Library > > > > > >