I've wondered if standard number matching  (ISBN, LCCN, OCLC,  
ISSN ...) would be a big piece.  Isn't there such a service from OCLC,  
and another flavor of something-or-other from LibraryThing?

- Naomi

On Oct 20, 2008, at 12:21 PM, Jonathan Rochkind wrote:

> To me, "de-duplication" means throwing out some records as  
> duplicates. Are we talking about that, or are we talking about what  
> I call "work set grouping" and others (erroneously in my opinion)  
> call "FRBRization"?
>
> If the latter, I don't think there is any mature open source  
> software that addresses that yet. Or for that matter, any  
> proprietary for-purchase software that you could use as a component  
> in your own tools. Various proprietary software includes a work set  
> grouping feature in it's "black box" (AquaBrowser, Primo, I believe  
> the VTLS ILS).  But I don't know of anything available to do it for  
> you in your own tool.
>
> I've been just starting to give some thought to how to accomplish  
> this, and it's a bit of a tricky problem on several grounds,  
> including computationally (doing it in a way that performs  
> efficiently). One choice is whether you group records at the  
> indexing stage, or on-demand at the retrieval stage. Both have  
> performance implications--we really don't want to slow down  
> retrieval OR indexing.  Usually if you have the choice, you put the  
> slow down at indexing since it only happens "once" in abstract  
> theory. But in fact, with what we do, when indexing that's already  
> been optmized and does not have this feature can take hours or even  
> days with some of our corpuses, and when in fact we do re-index from  
> time to time (including 'incremental' addition to the index of new  
> and changed records)---we really don't want to slow down indexing  
> either.
>
> Jonathan
>
> Bess Sadler wrote:
>> Hi, Mike.
>>
>> I don't know of any off-the-shelf software that does de-duplication  
>> of the kind you're describing, but it would be pretty useful. That  
>> would be awesome if someone wanted to build something like that  
>> into marc4j. Has anyone published any good algorithms for de- 
>> duping? As I understand it, if you have two records that are 100%  
>> identical except for holdings information, that's pretty easy. It  
>> gets harder when one record is more complete than the other, and  
>> very hard when one record has even slightly different information  
>> than the other, to tell whether they are the same record and decide  
>> whose information to privilege. Are there any good de-duping  
>> guidelines out there? When a library contracts out the de-duping of  
>> their catalog, what kind of specific guidelines are they expected  
>> to provide? Anyone know?
>>
>> I remember the open library folks were very interested in this  
>> question. Any open library folks on this list? Did that effort to  
>> de-dupe all those contributed marc records ever go anywhere?
>>
>> Bess
>>
>> On Oct 20, 2008, at 1:12 PM, Michael Beccaria wrote:
>>
>>> Very cool! I noticed that a feature, MarcDirStreamReader, is  
>>> capable of
>>> iterating over all marc record files in a given directory. Does  
>>> anyone
>>> know of any de-duplicating efforts done with marc4j? For example,
>>> libraries that have similar holdings would have their records merged
>>> into one record with a location tag somewhere. I know places do it
>>> (consortia etc.) but I haven't been able to find a good open program
>>> that handles stuff like that.
>>>
>>> Mike Beccaria
>>> Systems Librarian
>>> Head of Digital Initiatives
>>> Paul Smith's College
>>> 518.327.6376
>>> [log in to unmask]
>>>
>>>
> -- 
> Jonathan Rochkind
> Digital Services Software Engineer
> The Sheridan Libraries
> Johns Hopkins University
> 410.516.8886 rochkind (at) jhu.edu

Naomi Dushay
[log in to unmask]