Kyle Banerjee wrote:
> On Mon, Oct 25, 2010 at 12:38 PM, Tim Spalding <[log in to unmask]> wrote:
>> Does processing speed of something matter anymore? You'd have to be
>> doing a LOT of processing to care, wouldn't you?
> Data migrations and data dumps are a common use case. Needing to break or
> make hundreds of thousands or millions of records is not uncommon.
To make this concrete, we processes the MARC records from 14 separate
ILS's throughout the University of Wisconsin System. We extract, sort on
OCLC number, dedup and merge pieces from any campus that has a record
for the work. The MARC that we then index and display here
is not identical to the version of the MARC record from any of the 4
schools that hold it.
We extract 13 million records and dedup down to 8 million every week.
Speed is paramount.
Library Application Developer
436 Memorial Library
728 State St.
Madison, WI 53706
[log in to unmask]
"Just don't let the human factor fail to be a factor at all."
- Andrew Bird, "Tables and Chairs"