Hi, Mike.

I don't know of any off-the-shelf software that does de-duplication  
of the kind you're describing, but it would be pretty useful. That  
would be awesome if someone wanted to build something like that into  
marc4j. Has anyone published any good algorithms for de-duping? As I  
understand it, if you have two records that are 100% identical except  
for holdings information, that's pretty easy. It gets harder when one  
record is more complete than the other, and very hard when one record  
has even slightly different information than the other, to tell  
whether they are the same record and decide whose information to  
privilege. Are there any good de-duping guidelines out there? When a  
library contracts out the de-duping of their catalog, what kind of  
specific guidelines are they expected to provide? Anyone know?

I remember the open library folks were very interested in this  
question. Any open library folks on this list? Did that effort to de- 
dupe all those contributed marc records ever go anywhere?

Bess

On Oct 20, 2008, at 1:12 PM, Michael Beccaria wrote:

> Very cool! I noticed that a feature, MarcDirStreamReader, is  
> capable of
> iterating over all marc record files in a given directory. Does anyone
> know of any de-duplicating efforts done with marc4j? For example,
> libraries that have similar holdings would have their records merged
> into one record with a location tag somewhere. I know places do it
> (consortia etc.) but I haven't been able to find a good open program
> that handles stuff like that.
>
> Mike Beccaria
> Systems Librarian
> Head of Digital Initiatives
> Paul Smith's College
> 518.327.6376
> [log in to unmask]
>
> ---
> This message may contain confidential information and is intended only
> for the individual named. If you are not the named addressee you  
> should
> not disseminate, distribute or copy this e-mail. Please notify the
> sender immediately by e-mail if you have received this e-mail by  
> mistake
> and delete this e-mail from your system.
>
> -----Original Message-----
> From: Code for Libraries [mailto:[log in to unmask]] On  
> Behalf Of
> Bess Sadler
> Sent: Monday, October 20, 2008 11:12 AM
> To: [log in to unmask]
> Subject: [CODE4LIB] marc4j 2.4 released
>
> Dear Code4Libbers,
>
> I'm very pleased to announce that for the first time in almost two
> years there has been a new release of marc4j. Release 2.4 is a minor
> release in the sense that it shouldn't break any existing code, but
> it's a major release in the sense that it represents an influx of new
> people into the development of this project, and a significant
> improvement in marc4j's ability to handle malformed or mis-encoded
> marc records.
>
> Release notes are here: http://marc4j.tigris.org/files/documents/
> 220/44060/changes.txt
>
> And the project website, including download links, is here: http://
> marc4j.tigris.org/
>
> We've been using this new marc4j code in solrmarc since solrmarc
> started, so if you're using Blacklight or VuFind, you're probably
> using it already, just in an unreleased form.
>
> Bravo to Bob Haschart, Wayne Graham, and Bas Peters for making these
> improvements to marc4j and getting this release out the door.
>
> Bess
>
> Elizabeth (Bess) Sadler
> Research and Development Librarian
> Digital Scholarship Services
> Box 400129
> Alderman Library
> University of Virginia
> Charlottesville, VA 22904
>
> [log in to unmask]
> (434) 243-2305