I was dealing with a lot of sites that would shunt the user around based on
their user agent (e.g. very old sites that had completely different pages
for Netscape and IE), so I needed something neutral that wouldn't get
caught in a browser-specific branch.  Suffice to say, nothing ever checks
for Amiga browsers :)

On Mon, Oct 6, 2014 at 12:08 PM, Little, James Clarence IV <
[log in to unmask]> wrote:

> I love that user agent.
>
> This the wget command I've used to back up sites that have pretty urls:
>
> wget -v --mirror -p --html-extension -e robots=off --base=./ -k -P ./ <URL>
>
>
> – Jamie
> ________________________________________
> From: Code for Libraries <[log in to unmask]> on behalf of
> Alexander Duryee <[log in to unmask]>
> Sent: Monday, October 06, 2014 11:51 AM
> To: [log in to unmask]
> Subject: Re: [CODE4LIB] wget archiving for dummies
>
> I've used wget extensively for web preservation.  It's a remarkably
> powerful tool, but there are some notable features/caveats to be aware of:
>
> 1) You absolutely should use the --warc-file=<NAME> and
> --warc-header=<STRING> options.  These will create a WARC file alongside
> the usual wget filedump, which captures essential information (process
> provenance, server request/responses, raw data before wget adjusts it) for
> preservation.  The warc-header option includes user-added metadata, such as
> the name, purpose, etc. of the capture.  It's likely that you won't use the
> WARC for access, but keeping it as a preservation copy of the site is
> invaluable.
>
> 2) Javascript, AJAX queries, links in rich media, and such are completely
> opaque to wget.  As such, you'll need to QC aggressively to ensure that you
> captured everything you intended to.  My method was to run a generic wget
> capture[1], QC it, and manually download missing objects.  I'd then pass
> everything back into wget to create a complete WARC file containing the
> full capture.  It's janky, but gets the job done.
>
> 3) Do be careful of commenting options, which often turn into spider
> traps.  The latest versions of wget have regex support, so you can
> blacklist certain URLs that you know will trap the crawler.
>
> If the site is proving stubborn, I can take a look off-list.
>
> Best of luck,
> Alex
>
> [1] I've used the following successfully: wget
> --user-agent="AmigaVoyager/3.2
> (AmigaOS/MC680x0)" --warc-file=<FILENAME> --warc-header="<STRING>"
> --page-requisites -e robots=off --random-wait --wait=5 --recursive
> --level=0
> --no-parent --convert-links <URL>
>