 |
 |
On Jul 24, 2009, at 2:20 PM, [Chris Stockwell] wrote: > Over the next few years, I am tasked to download 30,000 archival > masters > from Internet Archive into an archive for long-term staff access > that we may > preserve with LOCKSS. These are masters of Montana state > publications. I > have a hierarchy in mind to receive these files. The hierarchy is > state > agency\year\title\pub_date\*.pdf. > > I am intending to download the files in batches of 200 - 500 pdfs, > but am > thinking that if I slot them automatically into the archive > hierarchy, misplaced > or missing files could be very hard to find as the total grows. I > will be logging > the downloads, which should give me some control. Are there other > strategies > for ensuring that I can readily correct download errors? I am > looking for > recommendations for the simplest way to maintain reasonable control > over the > download process. A couple things: If you already have archive.org identifiers picked out, you can use something like this python script to download them all from IA: http://blog.openlibrary.org/2008/11/24/bulk-access-to-ocr-for-1-million-books/ You can use the archive.org advanced search engine to produce xml, json, or csv file with all identifiers for a particular contributing institution: http://www.archive.org/advancedsearch.php eg. all identifier for Montana State Library (http://www.archive.org/details/MontanaStateLibrary ) as an xml file (change rows=10 to rows=10000 to get them all): http://www.archive.org/advancedsearch.php?q=collection%3A%22montanastatelib%22&fl%5B%5D=creator&fl%5B%5D=identifier&fl%5B%5D=title&sort%5B%5D=&sort%5B%5D=&sort%5B%5D=&rows=10&fmt=xml&xmlsearch=Search Also, if you have an archive.org identifier, then you can get the files.xml that contains md5 and sha1 hashes, so you can verify your download: To pull the files.xml, use a /download/id/id_files.xml url. e.g.: http://www.archive.org/download/librariesoffutur00lickuoft/librariesoffutur00lickuoft_files.xml -raj