 |
 |
Peter, I've seen no official information or documentation from the Internet Archive either. I've actually been quite frustrated by several issues for a while now. For example: If you go to http://www.archive.org/details/nonexistentidentifier you'll get a human-readable web page stating that the item cannot be found. That page, however, is served up with an HTTP status of 200 OK, not 404 NOT FOUND. In addition, I've noticed that when certain requests fail due to system load and other issues, I get back an HTML page saying something like "the system is experiencing slowness," but again with a 200 OK instead of a 503 SERVICE UNAVAILABLE (ideally with a Retry-After header). These things alone make it extremely difficult to automate any large-scale metadata retrieval from the Internet Archive, and that's without any attempt to download content. I'm working on a post documenting some of the techniques and strategies that have worked for us, but it's not quite ready for human consumption yet. Michael -- Michael B. Klein Digital Initiatives Technology Librarian Boston Public Library [log in to unmask] > From: "Binkley, Peter" <[log in to unmask]> > Reply-To: "Code for Libraries <[log in to unmask]>" > <[log in to unmask]> > Date: Thu, 5 Jun 2008 13:08:13 -0600 > To: <[log in to unmask]> > Conversation: [CODE4LIB] Internet Archive collection codes? > Subject: Re: [CODE4LIB] Internet Archive collection codes? > > While we're on the subject, are there any more up-to-date instructions > for harvesting from Internet Archive than these? > http://biodiversitylibrary.blogspot.com/2008/03/harvesting-process-from- > internet_14.html > > And does IA provide guidelines for harvesting (traffic limits etc.)? I > clicked around the site a bit and didn't find them, but could easily > have missed them. > > Peter