 |
 |
Hi Edward, We're currently using the warc-tools library for WARC creation. It's written in Python, but there are a few pre-built utilities that come with the package that might suit your needs? http://code.hanzoarchives.com/warc-tools -Kurt ________________________________________ From: Code for Libraries [[log in to unmask]] on behalf of Edward M. Corrado [[log in to unmask]] Sent: Wednesday, November 23, 2011 5:30 PM To: [log in to unmask] Subject: [CODE4LIB] Web archiving and WARC Hello All, I need to harvest a few Web sites in order to preserve them. I'd really like to preserve them using the WARC file format [1] since it is a standard for digital preservation. I looked at I looked at Web Curator Tool (WCT) and Heritrix and they seem to be good at what they do but are built to work on a much larger scale then what I'd like to do -- and that comes with a cost of increased complexity. Tools like wget are simple to use and can easily be scripted to accomplish my limited task, except the standard wget and similar tools I am familiar with do not support WARC. Also, I haven't been able to find a tool that can convert zipped files created with wget to WARC. I did find a version of wget with warc support built in [1] from the Archive Team so that may be my solution, but compile software with "dirty" written into the name of the zip file is maybe not the best longterm solution. Does anyone know of any other simples tool to create a WARC file (either from harvesting or converting a wget or similar mirror/archive)? Edward [1] http://archiveteam.org/index.php?title=Wget_with_WARC_output