Andrew, just as an additional data point, we have millions of records
indexed in our Lucene-based XTF system, and the response isn't too
bad even on a development server.
Roy
On Oct 28, 2005, at 1:31 PM, Andrew Nagy wrote:
> Wow! Thanks for such a detailed reply ... this is awesome.
>
> I am thinking about storing the data from the catalog in an XML
> database
> as well, however since I know very little about these I am greatly
> concerned about the scalability ... can they handle the 800,000+
> records
> we have in our catalog? If I am just using it as a store, and then
> use
> some sort of indexer, this shouldn't be a concern?
>
> Lucene seems enticing over Zebra since it is a z39 interface which
> from
> what I can understand will not let me do fancy searches such as
> what was
> recently cataloged in the past 7 days, etc.
> What about Xapian or XTF, did you test these out at all? I guess
> lucene
> seems like a better product because it is an apache project?
>
> Thanks for all the Info!
>
> Andrew
>
>
> Ross Singer wrote:
>
>
>> This is pretty similar to the project that Art Rhyno and I have been
>> working on for a couple of months now. Thankfully, I just got the
>> go-ahead to make it the top development priority, so hopefully we'll
>> actually have something to see in the near future. Like Eric, we
>> don't
>> have any problem with (and there aren't touching) any of the backend
>> stuff (cataloging, acq, circ), but have major issues with the public
>> interface.
>>
>> Although the way we're extracting records from our catalog is a
>> little
>> different (and there are reasons for it), the way I would recommend
>> getting the data out of the opac is not via z39.50, but through
>> whatever sort of marcdump utility your ILS has. You can then use
>> marc4j (or something similar) to transform the marc to xml (we're
>> going
>> to MODS, for example). Although we're currently just writing this
>> dump
>> to a filesystem (broken up by LCC... again, there are reasons that
>> don't exactly apply to this project), but I anticipate this will
>> eventually go into a METS record and a Berkeley xmldb for
>> storage. For
>> indexing, we're using Lucene (Art is accessing it via Cocoon, I am
>> through PyLucene) and we're, so far, pretty happy with the results.
>>
>> If Lucene has issues, we'll look at Zebra (as John mentioned),
>> although
>> Zebra's indexes are enormous. The nice thing about Zebra, though, is
>> that it would forgo the need for the Berkeley DB, since it stores the
>> XML record. The built-in Z39.50 server is a nice bonus, as well.
>> Backups would be XTF (http://www.cdlib.org/inside/projects/xtf/) and
>> Xapian. Swish-e isn't really an option since it can't index utf-8.
>>
>> The idea then is to be able to make stronger relationships between
>> our
>> site's content... eliminate the silos. A search that brings back a
>> couple of items that are in a particular subject guide would get a
>> link
>> to the subject... or at least links to the other "top" items from
>> that
>> guide (good tie in with MyLibrary, Eric). Something that's on
>> reserve
>> would have links to reserve policies or a course guide for that
>> course
>> or whatever.
>>
>> Journals would have links to the databases they are indexed in.
>>
>> Yes, there's some infrastructure that needs to be worked out... :)
>>
>> But the goal is to have something to at least see by the end of the
>> year (calendar, not school).
>>
>> We'll see :)
>>
>> -Ross.
>>
>> On Oct 27, 2005, at 5:58 PM, Eric Lease Morgan wrote:
>>
>>
>>> On Oct 27, 2005, at 2:06 PM, Andrew Nagy wrote:
>>>
>>>
>>>>> http://mylibrary.ockham.org/
>>>>>
>>>>
>>>>
>>>> I have been thinking of ways, similiar to what you have done
>>>> that you
>>>> mentioned below with the Ockham project, to allow more modern day
>>>> access
>>>> with our library catalog. I have been beginning to think about
>>>> devising
>>>> a way to index/harvest our entire catalog (and allow this indexing
>>>> process to run every so often) to allow our own custom access
>>>> methods.
>>>> We could then generate our own custom RSS feeds of new books, allow
>>>> more
>>>> efficient/enticing search interfaces, etc.
>>>>
>>>> Do you know of any existing software for indexing or harvesting a
>>>> catalog into another datastore (SQL Database, XML Database, etc).
>>>> I am
>>>> sure I could fetch all of the records somehow through Z39.50 and
>>>> dump it
>>>> into a MySQL database, but maybe there is some better method?
>>>>
>>>
>>>
>>> I too have thought about harvesting content from my local catalog
>>> and
>>> providing new interfaces to the content, and I might go about
>>> this in
>>> a number of different ways.
>>>
>>> 1. I might use OAI to harvest the content, cache is locally, and
>>> provide services against the cache. This cache might be saved on a
>>> file system, but more likely into a relational database.
>>>
>>> 2. I might simply dump all the MARC records from my catalog,
>>> transform them into something more readable, say sets of HTML/XML
>>> records, and provide services against these files.
>>>
>>> The weakest link in my chain would be my indexer. Relational
>>> databases are notoriously ill-equipped to handle free text
>>> searching.
>>> Yes, you can implement it and you can use various database-specific
>>> features to implement free text searching, but they still won't work
>>> as well as an indexer. My only experience with indexers lies in
>>> things like swish-e and Plucene. I sincerely wonder whether or not
>>> these indexers would be up to the task.
>>>
>>> Supposing I could find/use an indexer that was satisfactory, I would
>>> then provide simple and advanced (SRU/OpenSearch) search features
>>> against the index of holdings. Search results would then be enhanced
>>> with the features such as borrow, re-new, review, put on reserve,
>>> save as citation, email, "get it for me", put on hold, "what's
>>> new?",
>>> view as RSS, etc. These services would require a list of authorized
>>> users of the system -- a patron database.
>>>
>>> In short, since I would have direct access to the data, and since I
>>> would have direct to the index, I would use my skills to provide
>>> services them. For the most part, I don't mind back-end,
>>> administrative, data-entry interfaces to our various systems, but I
>>> do have problems with the end-user interfaces. Let me use those
>>> back-
>>> ends to create and store my data, then give me unfettered access to
>>> the data and I will provide my own end-user interfaces. Another
>>> alternative is to exploit (industry standard) Web Services computing
>>> techniques against the existing integrated library system. In this
>>> way you get XML data (information without presentation) back and you
>>> can begin to do the same things.
>>>
>>> --
>>> Eric Lease Morgan
>>> University Libraries of Notre Dame
>>>
>>>
>
>
|
