Because the IA hasn't devoted resources to documenting this stuff, I
guess.  If they actually want their stuff to be used by folks like us,
then seems to me resources devoted to such would be resources well spent.

Jonathan

K.G. Schneider wrote:
> But why are there hurdles?
>
> Karen G. Schneider
>
> On Wed, 27 Feb 2008 07:29:57 -0600, "Chris Freeland"
> <[log in to unmask]> said:
>
>> Roy, do you have an answer in mind?
>>
>> To me & my project it's the content that is open, which is why it's worth
>> the hurdles.  Once you 'crack the nut' you can grab metadata, scans, and
>> derivatives and ingest, parse, recombine, remix...as we've done for BHL.
>>
>> Access to OCA content may not be standards-based, but it works.
>>
>> Chris
>>
>> -----Original Message-----
>> From: "Roy Tennant" <[log in to unmask]>
>> To: "[log in to unmask]" <[log in to unmask]>
>> Sent: 2/27/2008 5:28 AM
>> Subject: Re: [CODE4LIB] oca api?
>>
>> So what, exactly, is "open" about this? Anyone care to guess?
>> Roy
>>
>>
>> On 2/26/08 10:29 AM, "Chris Freeland" <[log in to unmask]> wrote:
>>
>>
>>> My guess is that, yes, the query interface we've been discussing here
>>> and the 'all sorts of interfaces that none of us knew about' are the
>>> same.  It's not documented that I'm aware of.  We've found out about it
>>> by literally sitting next to IA developers and asking questions.
>>>
>>> Chris
>>> -----Original Message-----
>>> From: Code for Libraries [mailto:[log in to unmask]] On Behalf Of
>>> Jonathan Rochkind
>>> Sent: Tuesday, February 26, 2008 12:18 PM
>>> To: [log in to unmask]
>>> Subject: Re: [CODE4LIB] oca api?
>>>
>>> So in answer to my question here at the Code4Lib conference, after
>>> Brewster's keynote, Brewster suggests there are all sorts of interfaces
>>> that none of us knew about. Or at least I didn't know about, and haven't
>>> been able to figure out in months of trying!  I'm going to try and
>>> corner him and ask for an email of who we should contact.
>>>
>>> Perhaps it's the XML interface that you guys know about already. Is that
>>> documented anywhere? How the heck did you find out about it?
>>>
>>> Jonathan
>>>
>>>
>>>
>>>>>> Steve Toub <[log in to unmask]> 02/25/08 9:41 PM >>>
>>>>>>
>>> I'll add that when IA told me about
>>> http://www.archive.org/services/search.php interface to return
>>> XML, they asked that we not send more than 100 records at time since
>>> doing more would adversely
>>> affect production services. Which made it seem like OAI-PMH was a better
>>> way to go.
>>>
>>> Chris, can you explain a bit more about what this means: "We found their
>>> OAI interface to pull
>>> scanned items inconsistently based on date of scanning...."? I'm having
>>> trouble parsing.
>>>
>>>
>>>    --SET
>>>
>>>
>>>
>>>
>>> --- Chris Freeland <[log in to unmask]> wrote:
>>>
>>>
>>>> Jonathan - No, I don't believe it's documented - at least not anywhere
>>>> publicly.  If any IA/OCA folks are lurking, here's an opportunity to
>>>> make a bunch of techies happy...
>>>>
>>>> Chris
>>>>
>>>> -----Original Message-----
>>>> From: Code for Libraries [mailto:[log in to unmask]] On Behalf
>>>>
>>> Of
>>>
>>>> Jonathan Rochkind
>>>> Sent: Monday, February 25, 2008 2:48 PM
>>>> To: [log in to unmask]
>>>> Subject: Re: [CODE4LIB] oca api?
>>>>
>>>> I hadn't known this "custom query interface" existed! This is welcome
>>>> news. Is this documented anywhere?
>>>>
>>>> Jonathan
>>>>
>>>>
>>>>
>>>>>>> Chris Freeland <[log in to unmask]> 02/25/08 2:51 PM >>>
>>>>>>>
>>>> Steve & Tim,
>>>>
>>>> I'm the tech director for the Biodiversity Heritage Library (BHL),
>>>>
>>> which
>>>
>>>> is a consortium of 10 natural history libraries who have partnered
>>>>
>>> with
>>>
>>>> Internet Archive (IA)/OCA for scanning our collections.  We've just
>>>> launched our revamped portal, complete with more than 7,500 books &
>>>>
>>> 2.8
>>>
>>>> million pages scanned by IA & other digitization partners, at:
>>>> http://www.biodiversitylibrary.org
>>>>
>>>> To build this portal we ingest metadata from IA.  We found their OAI
>>>> interface to pull scanned items inconsistently based on date of
>>>> scanning, so we switched to using their custom query interface.
>>>>
>>> Here's
>>>
>>>> an example of a query we fire off:
>>>>
>>>>
>>>>
>>> http://www.archive.org/services/search.php?query=collection:(biodiversit
>>>
>>> y)+AND+updatedate:%5b2007-10-31+TO+2007-11-30%5d+AND+-contributor:(MBLWH
>>>
>>>> OI%20Library)&limit=10&submit=submit
>>>>
>>>> This is returning scanned items from the "biodiversity" collection,
>>>> updated between 10/31/2007 - 11/30/2007, restricted to one of our
>>>> contributing libraries (MBLWHOI Library), and limited to 10 results.
>>>>
>>>> The results are styled in the browser; view source to see the good
>>>> stuff.  We use this list to grab the identifiers we've yet to ingest.
>>>>
>>>> Some background: When a book is scanned through IA/OCA scanning, they
>>>> create their own unique identifier (like "annalesacademiae21univ") and
>>>> grab a MARC record from the contributing library's catalog.  All of
>>>>
>>> the
>>>
>>>> scanned files, derivatives, and metadata files are stored on IA's
>>>> clusters in a directory named with the identifier.
>>>>
>>>> Steve mentioned using their /details/ directive, then sniffing the
>>>>
>>> page
>>>
>>>> to get the cluster location and the files for downloading.  An easier
>>>> method is to use their /download/ directive, as in:
>>>>
>>>> http://www.archive.org/download/ID$, or in the example above:
>>>> http://www.archive.org/download/annalesacademiae21univ
>>>>
>>>> That automatically does a lookup on the cluster, which means you don't
>>>> have to scrape info off pages.  You can also address any files within
>>>> that directory, as in:
>>>>
>>>>
>>> http://www.archive.org/download/annalesacademiae21univ/annalesacademiae2
>>>
>>>> 1univ_marc.xml
>>>>
>>>> The only way to get standard identifiers (ISBN, ISSN, OCLC, LCCN) for
>>>> these scanned books is to grab them out of the MARC record.  So the
>>>> long-winded answer to your question, Tim, is no, there's no simple way
>>>> to crossref what IA has scanned with your catalog - THAT I KNOW OF.
>>>>
>>> Big
>>>
>>>> caveat on that last part.
>>>>
>>>> Happy to help with any other questions I can,
>>>>
>>>> Chris Freeland
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Code for Libraries [mailto:[log in to unmask]] On Behalf
>>>>
>>> Of
>>>
>>>> Steve Toub
>>>> Sent: Sunday, February 24, 2008 11:20 PM
>>>> To: [log in to unmask]
>>>> Subject: Re: [CODE4LIB] oca api?
>>>>
>>>> --- Tim Shearer <[log in to unmask]> wrote:
>>>>
>>>>
>>>>> Hi Folks,
>>>>>
>>>>> I'm looking into tapping the texts in the Open Content Alliance.
>>>>>
>>>>> A few questions...
>>>>>
>>>>> As near as I can tell, they don't expose (perhaps even store?) any
>>>>>
>>>> common
>>>>
>>>>> unique identifiers (oclc number, issn, isbn, loc number).
>>>>>
>>>> I poked around in this world a few months ago in my previous job at
>>>> California Digital Library,
>>>> also an OCA partner.
>>>>
>>>> The unique key seems to be text string identifier (one that seems to
>>>>
>>> be
>>>
>>>> completely different from
>>>> the text string identifier in Open Library). Apparently there was talk
>>>> at the last partner meeting
>>>> about moving to ISBNs:
>>>>
>>>>
>>> http://dilettantes.code4lib.org/2007/10/22/tales-from-the-open-content-a
>>>
>>>> lliance/
>>>>
>>>> To obtain identifiers in bulk, I think the recommended approach is the
>>>> OAI-PMH interface, which
>>>> seems more reliable in recent months:
>>>>
>>>> http://www.archive.org/services/oai.php?verb=Identify
>>>>
>>>>
>>>>
>>> http://www.archive.org/services/oai.php?verb=ListIdentifiers&metadataPre
>>>
>>>> fix=oai_dc&set=collection:cdl
>>>>
>>>> etc.
>>>>
>>>>
>>>> Additional instructions if you want to grab the content files.
>>>>
>>>> From any book's metadata page (e.g.,
>>>> http://www.archive.org/details/chemicallecturee00newtrich)
>>>> click through on the "Usage Rights: See Terms" link; the rights are on
>>>>
>>> a
>>>
>>>> pane on the left-hand
>>>> side.
>>>>
>>>> Once you know the identifier, you can grab the content files, using
>>>>
>>> this
>>>
>>>> syntax:
>>>>     http://www.archive.org/details/$ID
>>>> Like so:
>>>>     http://www.archive.org/details/chemicallecturee00newtrich
>>>>
>>>> And then sniff the page to find the FTP link:
>>>>     ftp://ia340915.us.archive.org/2/items/chemicallecturee00newtrich
>>>>
>>>> But I think they prefer to use HTTP for these, not the FTP, so switch
>>>> this to:
>>>>     http://ia340915.us.archive.org/2/items/chemicallecturee00newtrich
>>>>
>>>> Hope this helps!
>>>>
>>>>   --SET
>>>>
>>>>
>>>>
>>>>> We're a contributer so I can use curl to grab our records via http
>>>>>
>>>> (and
>>>>
>>>>> regexp my way to our local catalog identifiers, which they do
>>>>> store/expose).
>>>>>
>>>>> I've played a bit with the z39.50 interface at indexdata
>>>>> (http://www.indexdata.dk/opencontent/), but I'm not confident about
>>>>>
>>>> the
>>>>
>>>>> content behind it.  I get very limited results, for instance I can't
>>>>>
>>>> find
>>>>
>>>>> any UNC records and we're fairly new to the game.
>>>>>
>>>>> Again, I'm looking for unique identifiers in what I can get back and
>>>>>
>>>> it's
>>>>
>>>>> slim pickings.
>>>>>
>>>>> Anyone cracked this nut?  Got any life lessons for me?
>>>>>
>>>>> Thanks!
>>>>> Tim
>>>>>
>>>>> +++++++++++++++++++++++++++++++++++++++++++
>>>>> Tim Shearer
>>>>>
>>>>> Web Development Coordinator
>>>>> The University Library
>>>>> University of North Carolina at Chapel Hill
>>>>> [log in to unmask]
>>>>> 919-962-1288
>>>>> +++++++++++++++++++++++++++++++++++++++++++
>>>>>
>>>>>
>> --
>>
>
>

--
Jonathan Rochkind
Digital Services Software Engineer
The Sheridan Libraries
Johns Hopkins University
410.516.8886
rochkind (at) jhu.edu