On Jul 24, 2009, at 9:43 PM, raj kumar wrote:

>> Over the next few years, I am tasked to download 30,000 archival  
>> masters from Internet Archive into an archive for long-term staff  
>> access that we may preserve with LOCKSS. These are masters of  
>> Montana state publications. I have a hierarchy in mind to receive  
>> these files. The hierarchy is state agency\year\title\pub_date\*.pdf.
>>
>> I am intending to download the files in batches of 200 - 500 pdfs,  
>> but am thinking that if I slot them automatically into the archive  
>> hierarchy, misplaced or missing files could be very hard to find as  
>> the total grows. I will be logging the downloads, which should give  
>> me some control. Are there other strategies for ensuring that I can  
>> readily correct download errors? I am looking for recommendations  
>> for the simplest way to maintain reasonable control over the  
>> download process.
>
> A couple things:
>
> If you already have archive.org identifiers picked out, you can use
> something like this python script to download them all from IA:
> http://blog.openlibrary.org/2008/11/24/bulk-access-to-ocr-for-1-million-books/



'Sounds fun, and such a project is something I advocate not only for  
retrospective preservation purposes put for general collection  
building as well, but that is another story.

Without some sort of metadata it will not be possible for you to save  
your files in the hierarchy outlined above. State agency. Year. Title.  
Publication date. One the other hand, if metadata containing these  
values is readily accessible in the downloaded file itself or, as Ed  
mentioned, a part of some sort of manifest (or MARC record), then you  
are golden. I used Raj's script as a model for a similar process [1]:

   * write a cool query against Open Library returning identifiers
   * feed identifiers to mirroring program; I used wget
   * download file as well as metadata
   * parse metadata and process associated file accordingly

If you're really luck, then the "cool query" written against Open  
Library will also return the necessary metadata and you could use that  
as a guide to save your file

Good luck.

[1] similar process - http://infomotions.com/blog/2009/06/interent-archive-content-in-discovery-systems/

-- 
Eric Lease Morgan