To me, "de-duplication" means throwing out some records as duplicates. 
Are we talking about that, or are we talking about what I call "work set 
grouping" and others (erroneously in my opinion) call "FRBRization"?

If the latter, I don't think there is any mature open source software 
that addresses that yet. Or for that matter, any proprietary 
for-purchase software that you could use as a component in your own 
tools. Various proprietary software includes a work set grouping feature 
in it's "black box" (AquaBrowser, Primo, I believe the VTLS ILS).  But I 
don't know of anything available to do it for you in your own tool.

I've been just starting to give some thought to how to accomplish this, 
and it's a bit of a tricky problem on several grounds, including 
computationally (doing it in a way that performs efficiently). One 
choice is whether you group records at the indexing stage, or on-demand 
at the retrieval stage. Both have performance implications--we really 
don't want to slow down retrieval OR indexing.  Usually if you have the 
choice, you put the slow down at indexing since it only happens "once" 
in abstract theory. But in fact, with what we do, when indexing that's 
already been optmized and does not have this feature can take hours or 
even days with some of our corpuses, and when in fact we do re-index 
from time to time (including 'incremental' addition to the index of new 
and changed records)---we really don't want to slow down indexing either.

Jonathan

Bess Sadler wrote:
> Hi, Mike.
>
> I don't know of any off-the-shelf software that does de-duplication of 
> the kind you're describing, but it would be pretty useful. That would 
> be awesome if someone wanted to build something like that into marc4j. 
> Has anyone published any good algorithms for de-duping? As I 
> understand it, if you have two records that are 100% identical except 
> for holdings information, that's pretty easy. It gets harder when one 
> record is more complete than the other, and very hard when one record 
> has even slightly different information than the other, to tell 
> whether they are the same record and decide whose information to 
> privilege. Are there any good de-duping guidelines out there? When a 
> library contracts out the de-duping of their catalog, what kind of 
> specific guidelines are they expected to provide? Anyone know?
>
> I remember the open library folks were very interested in this 
> question. Any open library folks on this list? Did that effort to 
> de-dupe all those contributed marc records ever go anywhere?
>
> Bess
>
> On Oct 20, 2008, at 1:12 PM, Michael Beccaria wrote:
>
>> Very cool! I noticed that a feature, MarcDirStreamReader, is capable of
>> iterating over all marc record files in a given directory. Does anyone
>> know of any de-duplicating efforts done with marc4j? For example,
>> libraries that have similar holdings would have their records merged
>> into one record with a location tag somewhere. I know places do it
>> (consortia etc.) but I haven't been able to find a good open program
>> that handles stuff like that.
>>
>> Mike Beccaria
>> Systems Librarian
>> Head of Digital Initiatives
>> Paul Smith's College
>> 518.327.6376
>> [log in to unmask]
>>
>> ---
>> This message may contain confidential information and is intended only
>> for the individual named. If you are not the named addressee you should
>> not disseminate, distribute or copy this e-mail. Please notify the
>> sender immediately by e-mail if you have received this e-mail by mistake
>> and delete this e-mail from your system.
>>
>> -----Original Message-----
>> From: Code for Libraries [mailto:[log in to unmask]] On Behalf Of
>> Bess Sadler
>> Sent: Monday, October 20, 2008 11:12 AM
>> To: [log in to unmask]
>> Subject: [CODE4LIB] marc4j 2.4 released
>>
>> Dear Code4Libbers,
>>
>> I'm very pleased to announce that for the first time in almost two
>> years there has been a new release of marc4j. Release 2.4 is a minor
>> release in the sense that it shouldn't break any existing code, but
>> it's a major release in the sense that it represents an influx of new
>> people into the development of this project, and a significant
>> improvement in marc4j's ability to handle malformed or mis-encoded
>> marc records.
>>
>> Release notes are here: http://marc4j.tigris.org/files/documents/
>> 220/44060/changes.txt
>>
>> And the project website, including download links, is here: http://
>> marc4j.tigris.org/
>>
>> We've been using this new marc4j code in solrmarc since solrmarc
>> started, so if you're using Blacklight or VuFind, you're probably
>> using it already, just in an unreleased form.
>>
>> Bravo to Bob Haschart, Wayne Graham, and Bas Peters for making these
>> improvements to marc4j and getting this release out the door.
>>
>> Bess
>>
>> Elizabeth (Bess) Sadler
>> Research and Development Librarian
>> Digital Scholarship Services
>> Box 400129
>> Alderman Library
>> University of Virginia
>> Charlottesville, VA 22904
>>
>> [log in to unmask]
>> (434) 243-2305
>

-- 
Jonathan Rochkind
Digital Services Software Engineer
The Sheridan Libraries
Johns Hopkins University
410.516.8886 
rochkind (at) jhu.edu