This is pretty similar to the project that Art Rhyno and I have been
working on for a couple of months now.  Thankfully, I just got the
go-ahead to make it the top development priority, so hopefully we'll
actually have something to see in the near future.  Like Eric, we don't
have any problem with (and there aren't touching) any of the backend
stuff (cataloging, acq, circ), but have major issues with the public
interface.

Although the way we're extracting records from our catalog is a little
different (and there are reasons for it), the way I would recommend
getting the data out of the opac is not via z39.50, but through
whatever sort of marcdump utility your ILS has.  You can then use
marc4j (or something similar) to transform the marc to xml (we're going
to MODS, for example).  Although we're currently just writing this dump
to a filesystem (broken up by LCC... again, there are reasons that
don't exactly apply to this project), but I anticipate this will
eventually go into a METS record and a Berkeley xmldb for storage.  For
indexing, we're using Lucene (Art is accessing it via Cocoon, I am
through PyLucene) and we're, so far, pretty happy with the results.

If Lucene has issues, we'll look at Zebra (as John mentioned), although
Zebra's indexes are enormous.  The nice thing about Zebra, though, is
that it would forgo the need for the Berkeley DB, since it stores the
XML record.  The built-in Z39.50 server is a nice bonus, as well.
Backups would be XTF (http://www.cdlib.org/inside/projects/xtf/) and
Xapian.  Swish-e isn't really an option since it can't index utf-8.

The idea then is to be able to make stronger relationships between our
site's content... eliminate the silos.  A search that brings back a
couple of items that are in a particular subject guide would get a link
to the subject... or at least links to the other "top" items from that
guide (good tie in with MyLibrary, Eric).  Something that's on reserve
would have links to reserve policies or a course guide for that course
or whatever.

Journals would have links to the databases they are indexed in.

Yes, there's some infrastructure that needs to be worked out... :)

But the goal is to have something to at least see by the end of the
year (calendar, not school).

We'll see :)

-Ross.

On Oct 27, 2005, at 5:58 PM, Eric Lease Morgan wrote:

> On Oct 27, 2005, at 2:06 PM, Andrew Nagy wrote:
>
>>>  http://mylibrary.ockham.org/
>>
>> I have been thinking of ways, similiar to what you have done that you
>> mentioned below with the Ockham project, to allow more modern day
>> access
>> with our library catalog.  I have been beginning to think about
>> devising
>> a way to index/harvest our entire catalog (and allow this indexing
>> process to run every so often) to allow our own custom access methods.
>> We could then generate our own custom RSS feeds of new books, allow
>> more
>> efficient/enticing search interfaces, etc.
>>
>> Do you know of any existing software for indexing or harvesting a
>> catalog into another datastore (SQL Database, XML Database, etc).
>> I am
>> sure I could fetch all of the records somehow through Z39.50 and
>> dump it
>> into a MySQL database, but maybe there is some better method?
>
> I too have thought about harvesting content from my local catalog and
> providing new interfaces to the content, and I might go about this in
> a number of different ways.
>
> 1. I might use OAI to harvest the content, cache is locally, and
> provide services against the cache. This cache might be saved on a
> file system, but more likely into a relational database.
>
> 2. I might simply dump all the MARC records from my catalog,
> transform them into something more readable, say sets of HTML/XML
> records, and provide services against these files.
>
> The weakest link in my chain would be my indexer. Relational
> databases are notoriously ill-equipped to handle free text searching.
> Yes, you can implement it and you can use various database-specific
> features to implement free text searching, but they still won't work
> as well as an indexer. My only experience with indexers lies in
> things like swish-e and Plucene. I sincerely wonder whether or not
> these indexers would be up to the task.
>
> Supposing I could find/use an indexer that was satisfactory, I would
> then provide simple and advanced (SRU/OpenSearch) search features
> against the index of holdings. Search results would then be enhanced
> with the features such as borrow, re-new, review, put on reserve,
> save as citation, email, "get it for me", put on hold, "what's new?",
> view as RSS, etc. These services would require a list of authorized
> users of the system -- a patron database.
>
> In short, since I would have direct access to the data, and since I
> would have direct to the index, I would use my skills to provide
> services them. For the most part, I don't mind back-end,
> administrative, data-entry interfaces to our various systems, but I
> do have problems with the end-user interfaces. Let me use those back-
> ends to create and store my data, then give me unfettered access to
> the data and I will provide my own end-user interfaces. Another
> alternative is to exploit (industry standard) Web Services computing
> techniques against the existing integrated library system. In this
> way you get XML data (information without presentation) back and you
> can begin to do the same things.
>
> --
> Eric Lease Morgan
> University Libraries of Notre Dame
>