Great idea, Tim!
The open library tech list that Bess mentions is [log in to unmask],
described at
http://mail.archive.org/cgi-bin/mailman/listinfo/ol-tech
-Jodi
Jodi Schneider
Science Library Specialist
Amherst College
413-542-2076
>-----Original Message-----
>From: Code for Libraries [mailto:[log in to unmask]] On
>Behalf Of Tim Shearer
>Sent: Thursday, March 06, 2008 8:47 AM
>To: [log in to unmask]
>Subject: [CODE4LIB] musing on oca apiRe: [CODE4LIB] oca api?
>
>Howdy folks,
>
>I've been playing and thinking. I'd like to have what amounts
>to a unique
>identifier index to oca digitized texts. I want to be able to
>pull all the
>records that have oclc numbers, issns, isbns, etc. I want it to be
>lightweight, fast, searchable.
>
>Would anyone else want/use such a thing?
>
>I'm thinking about building something like this.
>
>If I do, it would be ideal if wouldn't be a duplication of
>effort, so anyone
>got this in the works? And if it would meet the needs of others.
>
>My basic notion is to crawl the site (starting with
>"americana", the American
>Libraries. Pull the oca unique identifier (e.g.
>northcarolinayea1910rale) and
>associate it with
>
>unique identifiers (oclc numbers, issns, isbns, lc numbers)
>contributing institution's alias and unique catalog identifier
>upload date
>
>That's all I was thinking of. Then there's what you might be
>able to do with
>it:
>
> Give me all the oca unique identifiers that have oclc numbers
> Give me all the oca unique identifiers with isbns that were
> uploaded between x and y date
> Give me the oca unique identifier for this oclc number
>
>Planning to do:
>
> keep crawling it and keep it up to date.
>
>Things I wasn't planning to do:
>
> worry about other unique ids (you'd have to go to xISBN or
> ThingISBN yourself)
> worry about storing anything else from oca.
>
>It would be good for being able to add an 856 to matches in
>your catalog. It
>would not be good for grabbing all marc records for all of oca.
>
>Anyhow, is this duplication of effort? Would you like
>something like this?
>What else would you like it to do (keeping in mind this is an
>unfunded pet
>project)? How would you want to talk to it? I was thinking
>of a web service,
>but hadn't thought too much about how to query it or how I'd
>deliver results.
>
>Of course I'm being an idiot and trying out new tools at the
>same time (python
>to see what the buzz is all about, sqlite just to learn it (it
>may not work
>out)).
>
>Thoughts? Vicious criticism?
>
>-t
>
>
>On Tue, 26 Feb 2008, Chris Freeland wrote:
>
>> My guess is that, yes, the query interface we've been discussing here
>> and the 'all sorts of interfaces that none of us knew about' are the
>> same. It's not documented that I'm aware of. We've found
>out about it
>> by literally sitting next to IA developers and asking questions.
>>
>> Chris
>> -----Original Message-----
>> From: Code for Libraries [mailto:[log in to unmask]]
>On Behalf Of
>> Jonathan Rochkind
>> Sent: Tuesday, February 26, 2008 12:18 PM
>> To: [log in to unmask]
>> Subject: Re: [CODE4LIB] oca api?
>>
>> So in answer to my question here at the Code4Lib conference, after
>> Brewster's keynote, Brewster suggests there are all sorts of
>interfaces
>> that none of us knew about. Or at least I didn't know about,
>and haven't
>> been able to figure out in months of trying! I'm going to try and
>> corner him and ask for an email of who we should contact.
>>
>> Perhaps it's the XML interface that you guys know about
>already. Is that
>> documented anywhere? How the heck did you find out about it?
>>
>> Jonathan
>>
>>
>>>>> Steve Toub <[log in to unmask]> 02/25/08 9:41 PM >>>
>> I'll add that when IA told me about
>> http://www.archive.org/services/search.php interface to return
>> XML, they asked that we not send more than 100 records at time since
>> doing more would adversely
>> affect production services. Which made it seem like OAI-PMH
>was a better
>> way to go.
>>
>> Chris, can you explain a bit more about what this means: "We
>found their
>> OAI interface to pull
>> scanned items inconsistently based on date of scanning...."?
>I'm having
>> trouble parsing.
>>
>>
>> --SET
>>
>>
>>
>>
>> --- Chris Freeland <[log in to unmask]> wrote:
>>
>>> Jonathan - No, I don't believe it's documented - at least
>not anywhere
>>> publicly. If any IA/OCA folks are lurking, here's an opportunity to
>>> make a bunch of techies happy...
>>>
>>> Chris
>>>
>>> -----Original Message-----
>>> From: Code for Libraries [mailto:[log in to unmask]] On Behalf
>> Of
>>> Jonathan Rochkind
>>> Sent: Monday, February 25, 2008 2:48 PM
>>> To: [log in to unmask]
>>> Subject: Re: [CODE4LIB] oca api?
>>>
>>> I hadn't known this "custom query interface" existed! This
>is welcome
>>> news. Is this documented anywhere?
>>>
>>> Jonathan
>>>
>>>
>>>>>> Chris Freeland <[log in to unmask]> 02/25/08 2:51 PM >>>
>>> Steve & Tim,
>>>
>>> I'm the tech director for the Biodiversity Heritage Library (BHL),
>> which
>>> is a consortium of 10 natural history libraries who have partnered
>> with
>>> Internet Archive (IA)/OCA for scanning our collections. We've just
>>> launched our revamped portal, complete with more than 7,500 books &
>> 2.8
>>> million pages scanned by IA & other digitization partners, at:
>>> http://www.biodiversitylibrary.org
>>>
>>> To build this portal we ingest metadata from IA. We found their OAI
>>> interface to pull scanned items inconsistently based on date of
>>> scanning, so we switched to using their custom query interface.
>> Here's
>>> an example of a query we fire off:
>>>
>>>
>>
>http://www.archive.org/services/search.php?query=collection:(bi
odiversit
>>>
>>
>y)+AND+updatedate:%5b2007-10-31+TO+2007-11-30%5d+AND+-contribut
or:(MBLWH
>>> OI%20Library)&limit=10&submit=submit
>>>
>>> This is returning scanned items from the "biodiversity" collection,
>>> updated between 10/31/2007 - 11/30/2007, restricted to one of our
>>> contributing libraries (MBLWHOI Library), and limited to 10 results.
>>>
>>> The results are styled in the browser; view source to see the good
>>> stuff. We use this list to grab the identifiers we've yet
>to ingest.
>>>
>>> Some background: When a book is scanned through IA/OCA
>scanning, they
>>> create their own unique identifier (like
>"annalesacademiae21univ") and
>>> grab a MARC record from the contributing library's catalog. All of
>> the
>>> scanned files, derivatives, and metadata files are stored on IA's
>>> clusters in a directory named with the identifier.
>>>
>>> Steve mentioned using their /details/ directive, then sniffing the
>> page
>>> to get the cluster location and the files for downloading.
>An easier
>>> method is to use their /download/ directive, as in:
>>>
>>> http://www.archive.org/download/ID$, or in the example above:
>>> http://www.archive.org/download/annalesacademiae21univ
>>>
>>> That automatically does a lookup on the cluster, which
>means you don't
>>> have to scrape info off pages. You can also address any
>files within
>>> that directory, as in:
>>>
>>
>http://www.archive.org/download/annalesacademiae21univ/annalesa
>cademiae2
>>> 1univ_marc.xml
>>>
>>> The only way to get standard identifiers (ISBN, ISSN, OCLC,
>LCCN) for
>>> these scanned books is to grab them out of the MARC record. So the
>>> long-winded answer to your question, Tim, is no, there's no
>simple way
>>> to crossref what IA has scanned with your catalog - THAT I KNOW OF.
>> Big
>>> caveat on that last part.
>>>
>>> Happy to help with any other questions I can,
>>>
>>> Chris Freeland
>>>
>>>
>>> -----Original Message-----
>>> From: Code for Libraries [mailto:[log in to unmask]] On Behalf
>> Of
>>> Steve Toub
>>> Sent: Sunday, February 24, 2008 11:20 PM
>>> To: [log in to unmask]
>>> Subject: Re: [CODE4LIB] oca api?
>>>
>>> --- Tim Shearer <[log in to unmask]> wrote:
>>>
>>>> Hi Folks,
>>>>
>>>> I'm looking into tapping the texts in the Open Content Alliance.
>>>>
>>>> A few questions...
>>>>
>>>> As near as I can tell, they don't expose (perhaps even store?) any
>>> common
>>>> unique identifiers (oclc number, issn, isbn, loc number).
>>>
>>> I poked around in this world a few months ago in my previous job at
>>> California Digital Library,
>>> also an OCA partner.
>>>
>>> The unique key seems to be text string identifier (one that seems to
>> be
>>> completely different from
>>> the text string identifier in Open Library). Apparently
>there was talk
>>> at the last partner meeting
>>> about moving to ISBNs:
>>>
>>
>http://dilettantes.code4lib.org/2007/10/22/tales-from-the-open-
>content-a
>>> lliance/
>>>
>>> To obtain identifiers in bulk, I think the recommended
>approach is the
>>> OAI-PMH interface, which
>>> seems more reliable in recent months:
>>>
>>> http://www.archive.org/services/oai.php?verb=Identify
>>>
>>>
>>
>http://www.archive.org/services/oai.php?verb=ListIdentifiers&me
tadataPre
>>> fix=oai_dc&set=collection:cdl
>>>
>>> etc.
>>>
>>>
>>> Additional instructions if you want to grab the content files.
>>>
>>> From any book's metadata page (e.g.,
>>> http://www.archive.org/details/chemicallecturee00newtrich)
>>> click through on the "Usage Rights: See Terms" link; the
>rights are on
>> a
>>> pane on the left-hand
>>> side.
>>>
>>> Once you know the identifier, you can grab the content files, using
>> this
>>> syntax:
>>> http://www.archive.org/details/$ID
>>> Like so:
>>> http://www.archive.org/details/chemicallecturee00newtrich
>>>
>>> And then sniff the page to find the FTP link:
>>> ftp://ia340915.us.archive.org/2/items/chemicallecturee00newtrich
>>>
>>> But I think they prefer to use HTTP for these, not the FTP,
>so switch
>>> this to:
>>>
>http://ia340915.us.archive.org/2/items/chemicallecturee00newtrich
>>>
>>> Hope this helps!
>>>
>>> --SET
>>>
>>>
>>>> We're a contributer so I can use curl to grab our records via http
>>> (and
>>>> regexp my way to our local catalog identifiers, which they do
>>>> store/expose).
>>>>
>>>> I've played a bit with the z39.50 interface at indexdata
>>>> (http://www.indexdata.dk/opencontent/), but I'm not confident about
>>> the
>>>> content behind it. I get very limited results, for
>instance I can't
>>> find
>>>> any UNC records and we're fairly new to the game.
>>>>
>>>> Again, I'm looking for unique identifiers in what I can
>get back and
>>> it's
>>>> slim pickings.
>>>>
>>>> Anyone cracked this nut? Got any life lessons for me?
>>>>
>>>> Thanks!
>>>> Tim
>>>>
>>>> +++++++++++++++++++++++++++++++++++++++++++
>>>> Tim Shearer
>>>>
>>>> Web Development Coordinator
>>>> The University Library
>>>> University of North Carolina at Chapel Hill
>>>> [log in to unmask]
>>>> 919-962-1288
>>>> +++++++++++++++++++++++++++++++++++++++++++
>>>>
>>>
>>
>
|
