 |
 |
I'm sure there are plenty of us (say, me for example) who would love to know what algorithms people are using to de-dupe MARC records (using "duplicate" to mean "completely different record describing the exact same resource). Anyone have something that works well-enough for them? On Fri, Mar 30, 2012 at 4:23 AM, Michael Hopwood <[log in to unmask]>wrote: > Hi Peter, Graham, > > I'm interested! Myself and a few friends and colleagues in the UK here are > looking at this and similar questions as part of an ongoing interest in > linked library data, and the schema/data model issues that underlie how > useful it could be. > > What we have here is certainly related to the data model (mostly implicit) > behind MaRC records, which is an interesting blend of expression, > manifestation and maybe item-level data. > > Options for merging, linking and otherwise combining various records is of > interest to us both "theoretically" and practically i.e. technically, so we > would be pleased to contribute to and benefit from the conversation on this. > > Best wishes, > > Michael Hopwood > > -----Original Message----- > From: Code for Libraries [mailto:[log in to unmask]] On Behalf Of > Peter Noerr > Sent: 30 March 2012 01:09 > To: [log in to unmask] > Subject: Re: [CODE4LIB] presenting merged records? > > Hi Graham, > > What we do in our federated search system, and have been doing for some > few years, is basically give the "designer" a choice of what options the > user gets for "de-duped" records. > > Firstly de-duping can be of a number of levels of sophistication, and a > many of them lead to the situation you have - records which are "similar" > rather than identical. On the web search side of things there are a > surprising number of real duplicates (well maybe not surprising if you > study more than one page of web search engine results), and on Twitter the > duplicates well outnumber the original posts (many thanks 're-tweet'). > > Where we get duplicate records the usual options are: 1) keep the first > and just drop all the rest. 2) keep the largest (assumed to have the most > information) and drop the rest. These work well for WSE results where they > are all almost identical (the differences often are just in the advertising > attached to the pages and the results), but not for bibliographic records. > > Less draconian is 3) Mark all the duplicates and keep them in the list (so > you get 1, 2, 3, 4, 5, 5.1, 5.2, 5.3, 6, ...). This groups all the similar > records together under the sort key of the first one, and does enable the > user to easily skip them. > > More user friendly is 4) Mark all duplicates and hide them in a sub-list > attached to the "head" record. This gets them out of the main display, but > allows the user who is interested in that "record" to expand the list and > see the variants. This could be of use to you. > > After that we planned to do what you are proposing and actually merge > record content into a single virtual record, and worked on algorithms to do > it. But nobody was interested. All our partners (who provide systems to > lots of libraries, both public, academic, and special) decided that it > would confuse their users more than it would help. I have my doubts, but > they spoke and we put the development on ice. > > I'm not sure this will help, but it has stood the test of time, and is > well used in its various guises. Since no-one else seems interested in this > topic, you could email me off list and we could discuss what we worked > through in the way of algorithms, etc. > > Peter > > > > -----Original Message----- > > From: Code for Libraries [mailto:[log in to unmask]] On Behalf > > Of graham > > Sent: Wednesday, March 28, 2012 8:05 AM > > To: [log in to unmask] > > Subject: Re: [CODE4LIB] presenting merged records? > > > > Hi Michael > > > > On 03/27/12 11:50, Michael Hopwood wrote: > > > Hi Graham, do I know you from RHUL? > > > > > Yes indeed :-) > > > > > My thoughts on "merged records" would be: > > > > > > 1. don't do it - use separate IDs and just present links between > > > related manifestations; thus > > avoiding potential confusions. > > > > In my case, I can't avoid it as it's a specific requirement: I'm doing > > a federated search across a large number of libraries, and if closely > > similar items aren't merged, the results become excessively large and > > repetitive. I'm merging all the similar items, displaying a summary of > > the merged bibliographic data, and providing links to each of the > > libraries with a copy. So it's not really FRBRization in the normal > > sense, I just thought that FRBRization would lead to similar problems, > so that there might be some well-known discussion of the issues around... > The merger of the records does have advantages, especially if some > libraries have very underpopulated records (eg subject fields). > > > > Cheers > > Graham > > > > > > > > http://www.bic.org.uk/files/pdfs/identification-digibook.pdf > > > > > > possible relationships - see > > > http://www.editeur.org/ONIX/book/codelists/current.html - lists 51 > > (manifestation)and 164 (work). > > > > > > 2. c.f. the way Amazon displays rough and ready categories > > > (paperback, hardback, audiobooks, *ahem* ebooks of some sort...) > > > > > > On dissection and reconstitution of records - there is a lot of talk > > > going on about RDFizing MaRC > > records and re-using in various ways, e.g.: > > > > > > http://www.slideshare.net/JenniferBowen/moving-library-metadata-towa > > > rd -linked-data-opportunities-provided-by-the-extensible-catalog > > > > > > Cheers, > > > > > > Michael > > > > > > -----Original Message----- > > > From: Code for Libraries [mailto:[log in to unmask]] On Behalf > > > Of graham > > > Sent: 27 March 2012 11:06 > > > To: [log in to unmask] > > > Subject: [CODE4LIB] presenting merged records? > > > > > > Hi > > > > > > There seems to be a general trend to presenting merged records to > > > users, as part of the move towards > > FRBRization. If records need merging this generally means they weren't > > totally identical to start with, so you can end up with conflicting > bibliographic data to display. > > > > > > Two examples I've come across with this: Summon can merge > > > print/electronic versions of texts, so uses a new 'merged' material > > > type of 'book/ebook' (it doesn't yet seem to have all the other > > > possible permutations, eg book/audiobook). Pazpar2 (which I'm > > > working with at the > > > moment) has a merge option for publication dates which presents dates > as a period eg 1997-2002. > > > > > > The problem is not with the underlying data (the original unmerged > > > values can still be there in the > > background) but how to present them to the user in an intuitive way. > > With the date example, presenting dates in this format sometimes > > throws people as it looks too much like the author birth/death dates you > might see with a record. > > > > > > I guess people must generally be starting to run into this kind of > > > display problem, so it has maybe > > been discussed to death on ... wherever it is people talk about > > FRBRIzation. Any suggestions? Any mailing lists, blogs etc any can > recommend for me to look at? > > > > > > Thanks for any ideas > > > Graham > -- Bill Dueber Library Systems Programmer University of Michigan Library