I know there are two parts of this discussion (speed on the one hand,
applicability/features on teh other), but for the former, running a little
benchmark just isn't that hard. Aren't we supposed to, you know, prefer to
make decisions based on data?

Note: I'm only testing deserialization because there's isn't, as of now, a
fast serialization option for ruby-marc. It uses REXML, and it's dog-slow. I
already looked marc-in-json vs marc binary at
http://robotlibrarian.billdueber.com/sizespeed-of-various-marc-serializations-using-ruby-marc/

Benchmark Source: http://gist.github.com/645683

18,883 records as either an XML collection or newline-delimited json.
Open the file, read every record, pull out a title. Repeat 5 times for a
total of 94,415 records (i.e., just under 100K records total).

Under ruby-marc, using the libxml deserializer is the fastest option. If
you're using the REXML parser, well,  god help us all.

ruby 1.8.7 (2010-08-16 patchlevel 302) [i686-darwin9.8.0]. User time
reported in seconds.

  xml w/libxml         227 seconds
  marc-in-json w/yajl  130 seconds


So....quite a bit faster (more than 40%). For a million records (assuming I
can just say 10*these_values) you're talking about a difference of 16
minutes due to just reading speed. Assuming, of course, you're running your
code on my desktop. Today.

For the 8M records I have to deal with, that'd be roughly 8M * ((227-130)
/ 94,415)  = 7806 seconds, or about 130 minutes. Soooo...a lot.

Of course, if you're using a slower XML library or a slower JSON library,
your numbers will vary quite a bit. REXML is unforgivingly slow, and
json/pure (and even 'json') are quite a bit slower than yajl. And don't
forget that you need to serialize these things from your source somehow...

 -Bill-



On Mon, Oct 25, 2010 at 4:23 PM, Stephen Meyer <[log in to unmask]>wrote:

> Kyle Banerjee wrote:
>
>> On Mon, Oct 25, 2010 at 12:38 PM, Tim Spalding <[log in to unmask]>
>> wrote:
>>
>>  Does processing speed of something matter anymore? You'd have to be
>>> doing a LOT of processing to care, wouldn't you?
>>>
>>>
>> Data migrations and data dumps are a common use case. Needing to break or
>> make hundreds of thousands or millions of records is not uncommon.
>>
>> kyle
>>
>
> To make this concrete, we processes the MARC records from 14 separate ILS's
> throughout the University of Wisconsin System. We extract, sort on OCLC
> number, dedup and merge pieces from any campus that has a record for the
> work. The MARC that we then index and display here
>
>  http://forward.library.wisconsin.edu/catalog/ocm37443537?school_code=WU
>
> is not identical to the version of the MARC record from any of the 4
> schools that hold it.
>
> We extract 13 million records and dedup down to 8 million every week. Speed
> is paramount.
>
> -sm
> --
> Stephen Meyer
> Library Application Developer
> UW-Madison Libraries
> 436 Memorial Library
> 728 State St.
> Madison, WI 53706
>
> [log in to unmask]
> 608-265-2844 (ph)
>
>
> "Just don't let the human factor fail to be a factor at all."
> - Andrew Bird, "Tables and Chairs"
>



-- 
Bill Dueber
Library Systems Programmer
University of Michigan Library