 |
 |
Kyle Banerjee wrote: > On Mon, Oct 25, 2010 at 12:38 PM, Tim Spalding <[log in to unmask]> wrote: > >> Does processing speed of something matter anymore? You'd have to be >> doing a LOT of processing to care, wouldn't you? >> > > Data migrations and data dumps are a common use case. Needing to break or > make hundreds of thousands or millions of records is not uncommon. > > kyle To make this concrete, we processes the MARC records from 14 separate ILS's throughout the University of Wisconsin System. We extract, sort on OCLC number, dedup and merge pieces from any campus that has a record for the work. The MARC that we then index and display here http://forward.library.wisconsin.edu/catalog/ocm37443537?school_code=WU is not identical to the version of the MARC record from any of the 4 schools that hold it. We extract 13 million records and dedup down to 8 million every week. Speed is paramount. -sm -- Stephen Meyer Library Application Developer UW-Madison Libraries 436 Memorial Library 728 State St. Madison, WI 53706 [log in to unmask] 608-265-2844 (ph) "Just don't let the human factor fail to be a factor at all." - Andrew Bird, "Tables and Chairs"