Hi all:

My student, Yee Fan Tan, and I published a short technical column on
record linkage tasks (very similar to the de-dup task discussed here)
in February in the Communications of the ACM.

Min-Yen Kan and Yee Fan Tan (2008) "Record matching in digital library
metadata".  In Communications of the ACM, Technical opinion column,
pp. 91-94, February.

http://doi.acm.org/10.1145/1314215.1314231

We're in the process of releasing a tool/demo for de-dup tasks, as a
java library (jar).  If there's sufficient interest, we might try to
cater some of our string similarity metrics to MARC or other catalog
data.

Cheers,

Min

--
Min-Yen KAN (Dr) :: Assistant Professor :: National University of
Singapore :: School of Computing, AS6 05-12, Law Link, Singapore
117590 :: 65-6516 1885(DID) :: 65-6779 4580 (Fax) ::
[log in to unmask] (E) :: www.comp.nus.edu.sg/~kanmy (W)

Important: This email is confidential and may be privileged. If you
are not the intended recipient, please delete it and notify us
immediately; you should not copy or use it for any purpose, nor
disclose its contents to any other person. Thank you.



On Tue, Oct 21, 2008 at 8:03 AM, Naomi Dushay <[log in to unmask]> wrote:
> I've wondered if standard number matching  (ISBN, LCCN, OCLC, ISSN ...)
> would be a big piece.  Isn't there such a service from OCLC, and another
> flavor of something-or-other from LibraryThing?
>
> - Naomi
>
> On Oct 20, 2008, at 12:21 PM, Jonathan Rochkind wrote:
>
>> To me, "de-duplication" means throwing out some records as duplicates. Are
>> we talking about that, or are we talking about what I call "work set
>> grouping" and others (erroneously in my opinion) call "FRBRization"?
>>
>> If the latter, I don't think there is any mature open source software that
>> addresses that yet. Or for that matter, any proprietary for-purchase
>> software that you could use as a component in your own tools. Various
>> proprietary software includes a work set grouping feature in it's "black
>> box" (AquaBrowser, Primo, I believe the VTLS ILS).  But I don't know of
>> anything available to do it for you in your own tool.
>>
>> I've been just starting to give some thought to how to accomplish this,
>> and it's a bit of a tricky problem on several grounds, including
>> computationally (doing it in a way that performs efficiently). One choice is
>> whether you group records at the indexing stage, or on-demand at the
>> retrieval stage. Both have performance implications--we really don't want to
>> slow down retrieval OR indexing.  Usually if you have the choice, you put
>> the slow down at indexing since it only happens "once" in abstract theory.
>> But in fact, with what we do, when indexing that's already been optmized and
>> does not have this feature can take hours or even days with some of our
>> corpuses, and when in fact we do re-index from time to time (including
>> 'incremental' addition to the index of new and changed records)---we really
>> don't want to slow down indexing either.
>>
>> Jonathan
>>
>> Bess Sadler wrote:
>>>
>>> Hi, Mike.
>>>
>>> I don't know of any off-the-shelf software that does de-duplication of
>>> the kind you're describing, but it would be pretty useful. That would be
>>> awesome if someone wanted to build something like that into marc4j. Has
>>> anyone published any good algorithms for de-duping? As I understand it, if
>>> you have two records that are 100% identical except for holdings
>>> information, that's pretty easy. It gets harder when one record is more
>>> complete than the other, and very hard when one record has even slightly
>>> different information than the other, to tell whether they are the same
>>> record and decide whose information to privilege. Are there any good
>>> de-duping guidelines out there? When a library contracts out the de-duping
>>> of their catalog, what kind of specific guidelines are they expected to
>>> provide? Anyone know?
>>>
>>> I remember the open library folks were very interested in this question.
>>> Any open library folks on this list? Did that effort to de-dupe all those
>>> contributed marc records ever go anywhere?
>>>
>>> Bess
>>>
>>> On Oct 20, 2008, at 1:12 PM, Michael Beccaria wrote:
>>>
>>>> Very cool! I noticed that a feature, MarcDirStreamReader, is capable of
>>>> iterating over all marc record files in a given directory. Does anyone
>>>> know of any de-duplicating efforts done with marc4j? For example,
>>>> libraries that have similar holdings would have their records merged
>>>> into one record with a location tag somewhere. I know places do it
>>>> (consortia etc.) but I haven't been able to find a good open program
>>>> that handles stuff like that.
>>>>
>>>> Mike Beccaria
>>>> Systems Librarian
>>>> Head of Digital Initiatives
>>>> Paul Smith's College
>>>> 518.327.6376
>>>> [log in to unmask]
>>>>
>>>>
>> --
>> Jonathan Rochkind
>> Digital Services Software Engineer
>> The Sheridan Libraries
>> Johns Hopkins University
>> 410.516.8886 rochkind (at) jhu.edu
>
> Naomi Dushay
> [log in to unmask]
>