 |
 |
Thanks for the advice all. I'm trying httrack now but the other wget options are good to know about, especially Alex's point about saving a WARC file. One clarification: I definitely don't want to deal with the database, nor can I. We don't have admin or server access. Even if we did, I don't think preserving the db would be wise or necessary. Best, Eric On Mon, Oct 6, 2014 at 9:24 AM, Alexander Duryee <[log in to unmask]> wrote: > I was dealing with a lot of sites that would shunt the user around based on > their user agent (e.g. very old sites that had completely different pages > for Netscape and IE), so I needed something neutral that wouldn't get > caught in a browser-specific branch. Suffice to say, nothing ever checks > for Amiga browsers :) > > On Mon, Oct 6, 2014 at 12:08 PM, Little, James Clarence IV < > [log in to unmask]> wrote: > > > I love that user agent. > > > > This the wget command I've used to back up sites that have pretty urls: > > > > wget -v --mirror -p --html-extension -e robots=off --base=./ -k -P ./ > <URL> > > > > > > – Jamie > > ________________________________________ > > From: Code for Libraries <[log in to unmask]> on behalf of > > Alexander Duryee <[log in to unmask]> > > Sent: Monday, October 06, 2014 11:51 AM > > To: [log in to unmask] > > Subject: Re: [CODE4LIB] wget archiving for dummies > > > > I've used wget extensively for web preservation. It's a remarkably > > powerful tool, but there are some notable features/caveats to be aware > of: > > > > 1) You absolutely should use the --warc-file=<NAME> and > > --warc-header=<STRING> options. These will create a WARC file alongside > > the usual wget filedump, which captures essential information (process > > provenance, server request/responses, raw data before wget adjusts it) > for > > preservation. The warc-header option includes user-added metadata, such > as > > the name, purpose, etc. of the capture. It's likely that you won't use > the > > WARC for access, but keeping it as a preservation copy of the site is > > invaluable. > > > > 2) Javascript, AJAX queries, links in rich media, and such are completely > > opaque to wget. As such, you'll need to QC aggressively to ensure that > you > > captured everything you intended to. My method was to run a generic wget > > capture[1], QC it, and manually download missing objects. I'd then pass > > everything back into wget to create a complete WARC file containing the > > full capture. It's janky, but gets the job done. > > > > 3) Do be careful of commenting options, which often turn into spider > > traps. The latest versions of wget have regex support, so you can > > blacklist certain URLs that you know will trap the crawler. > > > > If the site is proving stubborn, I can take a look off-list. > > > > Best of luck, > > Alex > > > > [1] I've used the following successfully: wget > > --user-agent="AmigaVoyager/3.2 > > (AmigaOS/MC680x0)" --warc-file=<FILENAME> --warc-header="<STRING>" > > --page-requisites -e robots=off --random-wait --wait=5 --recursive > > --level=0 > > --no-parent --convert-links <URL> > > >