On Feb 23, 2006, at 9:29 PM, Eric Lease Morgan wrote:

> I have mirrored 50 GB of journal data from the open access journal
> literature, and I'm curious to know, what indexer would you use to
> index this data?
>
> --
> Eric Lease Morgan
> University Libraries of Notre Dame


Hi, Eric.

What format is the data in? (e.g. xml? plain text?) Are you willing
to do coding yourself, or do you need an out-of-box solution?

Generally speaking, lucene is a very fast indexing solution, but it
is not something you can just download and use; you have a find an
application that builds on the lucene indexer. If your data is XML
you might want to take a look at the martini project at http://
sourceforge.net/projects/martini. It was built for Olive XML, but it
works with any xml schema. With some tweaking, it will let you
describe your xml documents, put them into a lucene index, and
search / display them via a cocoon application. It's very much in
development, though, so be prepared to shape it to fit your own needs.

I've also been experimenting with Berkeley DB's XML databases, and
those are very fast, too, and would handle the kind of size you're
talking about. One of my co-workers recently figured out how to pull
documents directly from Berkeley DB XML into a cocoon pipeline, so
that's handy, too. If this is of interest, I can send you some code
that you could use as a template.

Both of these assume the documents are in XML, and that you're
willing to do a lot of coding.

Another option might be eXist. It's fairly easy to get up and
running, and it can be more of an out-of-the-box solution (as long as
you're prepared to open a complicated box) to index, search, and
display documents, but I've heard it can be unstable for very large
data sets like the one you're talking about.

I'd love to hear about other people's recent experiences with eXist.
At my library we keep wanting to use it, but having been burned by a
previous release we're feeling a little hesitant.

Cheers,
Bess


Elizabeth (Bess) Sadler
Metadata Specialist for User Projects

Digital Research and Instructional Services (DRIS)
Box 400129
Alderman Library
University of Virginia
Charlottesville, VA 22904

[log in to unmask]
(434) 243-2305