I've used wget extensively for web preservation. It's a remarkably
powerful tool, but there are some notable features/caveats to be aware of:
1) You absolutely should use the --warc-file=<NAME> and
--warc-header=<STRING> options. These will create a WARC file alongside
the usual wget filedump, which captures essential information (process
provenance, server request/responses, raw data before wget adjusts it) for
preservation. The warc-header option includes user-added metadata, such as
the name, purpose, etc. of the capture. It's likely that you won't use the
WARC for access, but keeping it as a preservation copy of the site is
invaluable.
2) Javascript, AJAX queries, links in rich media, and such are completely
opaque to wget. As such, you'll need to QC aggressively to ensure that you
captured everything you intended to. My method was to run a generic wget
capture[1], QC it, and manually download missing objects. I'd then pass
everything back into wget to create a complete WARC file containing the
full capture. It's janky, but gets the job done.
3) Do be careful of commenting options, which often turn into spider
traps. The latest versions of wget have regex support, so you can
blacklist certain URLs that you know will trap the crawler.
If the site is proving stubborn, I can take a look off-list.
Best of luck,
Alex
[1] I've used the following successfully: wget --user-agent="AmigaVoyager/3.2
(AmigaOS/MC680x0)" --warc-file=<FILENAME> --warc-header="<STRING>"
--page-requisites -e robots=off --random-wait --wait=5 --recursive --level=0
--no-parent --convert-links <URL>
|