Brad et al,


We use wget scripts to back up our internet archive pages, which, oddly
enough, are the instructions given by internet archive itself. :/



Kenny Ketner
Information Products Lead
Montana State Library
406-444-2870
[log in to unmask]
kennyketner.com

On Tue, Nov 28, 2017 at 12:31 PM, Brett <[log in to unmask]> wrote:

> Yes, I did ask, and ask, and ask, and waited for 2 months. There was
> something political going on internally with that group that was well
> beyond my pay grade.
>
> I did explain the potential problems to my boss and she was providing
> cover.
>
> I did it in batches as Google Sheets limits the amount of ImportXML that
> you can do in a 24 hour span, so I wasn't hammering anyone's web server
> into oblivion.
>
> It's funny, I actually had to do a fair amount to get the old V1 LibGuides
> link checker to stop hammering my ILS into going offline back in 2010-2011.
>
>
>
> On Tue, Nov 28, 2017 at 2:18 PM, Bill Dueber <[log in to unmask]> wrote:
>
> > Brett, did you ask the folks at the Large University Library if they
> could
> > set something up for you? I don't have a good sense of how other
> > institutions deal with things like this.
> >
> > In any case, I know I'd much rather talk about setting up an API or a
> > nightly dump or something rather than have my analytics (and bandwidth!)
> > blown by a screen scraper. I might say "no," but at least it would be an
> > informed "no" :-)
> >
> > On Tue, Nov 28, 2017 at 2:08 PM, Brett <[log in to unmask]>
> wrote:
> >
> > > I leveraged the IMPORTXML() and xpath features in Google Sheets to pull
> > > information from a large university website to help create a set of
> > weeding
> > > lists for a branch campus. They needed  extra details about what was in
> > > off-site storage and what was held at the central campus library.
> > >
> > > This was very much like Jason's FIFO API, the central reporting group
> had
> > > sent me a spreadsheet with horrible data that I would have had to sort
> > out
> > > almost completely manually, but the call numbers were pristine. I used
> > the
> > > call numbers as a key to query the catalog with limits for each campus
> I
> > > needed to check, and then it dumped all of the necessary content
> > (holdings,
> > > dates, etc) into the spreadsheet.
> > >
> > > I've also used Feed43 as a way to modify certain RSS feeds and scrape
> > > websites  to only display the content I want.
> > >
> > > Brett Williams
> > >
> > >
> > > On Tue, Nov 28, 2017 at 1:24 PM, Brad Coffield <
> > > [log in to unmask]>
> > > wrote:
> > >
> > > > I think there's likely a lot of possibilities out there and was
> hoping
> > to
> > > > hear examples of web scraping for libraries. Your example might just
> > > > inspire me or another reader to do something similar. At the very
> > least,
> > > > the ideas will be interesting!
> > > >
> > > > Brad
> > > >
> > > >
> > > > --
> > > > Brad Coffield, MLIS
> > > > Assistant Information and Web Services Librarian
> > > > Saint Francis University
> > > > 814-472-3315
> > > > [log in to unmask]
> > > >
> > >
> >
> >
> >
> > --
> > Bill Dueber
> > Library Systems Programmer
> > University of Michigan Library
> >
>