The Blacklight code is not currently using XML or XSLT. It's indexing
binary MARC files. I don't know it's speed, but I hear it's pretty fast.

But for the kind of test set I want, even waiting half an hour is too
long. I want a test set where I can make a change to my indexing
configuration and then see the results in a few minutes, if not seconds.

This has become apparent in my attempts to get the indexer _working_,
where I might not be actually changing the index mapping at all, I'm
just changing the indexer configuration until I know it's working. When
I get to actually messing with indexer mapping to try out new ideas, I
believe it will also be important, however.

Jonathan

Casey Durfee wrote:
> I strongly agree that we need something like this.  The LoC records that
> Casey donated are a great resource but far from ideal from this purpose.
> They're pretty homogeneous.  I do think it needs to be bigger than 10,000
> though.  100,000 would be a better target.  And I would like to see a
> UNIMARC/DANMARC-based one as well as a MARC21 based one (can one's parser
> handle DANMARC's "ø" subfield?).
>
> I don't know about Blacklight or VuFind, but using our MarcThing package +
> Solr we can index up to 1000 records a second.  I know using XSLT severely
> limits how fast you can index (I'll refrain from giving another rant about
> how wrong it is to use XSL to handle MARC -- the Society for Prevention of
> Cruelty to Dead Horses has my number as it is.)  But I'd still expect you
> can do a good 50-100 records a second.  That's only a half hour to an hour
> of work to index 100,000 records.  You could run it over your lunch break.
> Seems reasonable to me.
>
> In addition to a wide variety of languages, encodings, formats and so forth,
> it would definitely need to have records explicitly designed to break
> things.  Blank MARC tags, extraneous subfield markers, non-printing control
> characters, incorrect length fixed fields etc.  The kind of stuff that
> should never happen in theory, but happens frequently in real life (I'm
> looking at you, Horizon's MARC export utility).
>
> The legal aspect of this is the difficult part.  We (LibraryThing) could
> easily grab 200 random records from 500 different Z39.50 sources worldwide.
> Technically, it could be done in a couple of hours.  Legally, I don't think
> it could ever be done, sadly.
>
> --Casey
>
>
> On Fri, May 9, 2008 at 9:33 AM, Bess Sadler <[log in to unmask]> wrote:
>
>
>> Those of us involved in the Blacklight and VuFind projects are
>> spending lots of time recently thinking about marc records indexing.
>> We're about to start running some performance tests, and we want to
>> create unit tests for our marc to solr indexer, and also people
>> wanting to download and play with the software need to have easy
>> access to a small but representative set of marc records that they
>> can play with.
>>
>> According to the combined brainstorming of Jonathan Rochkind and
>> myself, the ideal record set should:
>>
>> 1. contain about 10k records, enough to really see the features, but
>> small enough that you could index it in a few minutes on a typical
>> desktop
>> 2. contain a distribution of kinds of records, e.g., books, CDs,
>> musical scores, DVDs, special collection items, etc.
>> 3. contain a distribution of languages, so we can test unicode handling
>> 4. contain holdings information in addition to bib records
>> 5. contain a distribution of typical errors one might encounter with
>> marc records in the wild
>>
>> It seems to me that the set that Casey donated to Open Library
>> (http://www.archive.org/details/marc_records_scriblio_net) would be a
>> good place from which to draw records, because although IANAL, this
>> seems to sidestep any legal hurdles. I'd also love to see the ability
>> for the community to contribute test cases. Assuming such a set
>> doesn't exist already (see my question below) this seems like the
>> ideal sort of project for code4lib to host, too.
>>
>> Since code4lib is my lazyweb, I'm asking you:
>>
>> 1. Does something like this exist already and I just don't know about
>> it?
>> 2. If not, do you have suggestions on how to go about making such a
>> data set? I have some ideas on how to do it bit by bit, and we have a
>> certain small set of records that we're already using for testing,
>> but maybe there's a better method that I don't know about?
>> 3. Are there features missing from the above list that would make
>> this more useful?
>>
>> Thoughts? Comments?
>>
>> Thanks!
>> Bess
>>
>>
>> Elizabeth (Bess) Sadler
>> Research and Development Librarian
>> Digital Scholarship Services
>> Box 400129
>> Alderman Library
>> University of Virginia
>> Charlottesville, VA 22904
>>
>> [log in to unmask]
>> (434) 243-2305
>>
>>
>
>

--
Jonathan Rochkind
Digital Services Software Engineer
The Sheridan Libraries
Johns Hopkins University
410.516.8886
rochkind (at) jhu.edu