 |
 |
I actually love the approach Mark writes about here. It was partly what inspired me to do this work in MarcEdit -- abet, in a light-weight way -- so not to incur any additional dependencies. --tr On Wed, Oct 25, 2017 at 12:23 PM, Phillips, Mark <[log in to unmask]> wrote: > Of possible interest is some work we've done to take the clustering > capabilities of OpenRefine and bake them into our metadata editing > interface for The Portal to Texas History and the UNT Digital Library. > > We've focused a bit on interfaces which might be of interest. I've > written a bit about it in this post. http://vphill.com/journal/post/6173/ > > We are generating the clusters on facets from Solr. > > Mark > ________________________________________ > From: Code for Libraries <[log in to unmask]> on behalf of Péter > Király <[log in to unmask]> > Sent: Wednesday, October 25, 2017 11:18:52 AM > To: [log in to unmask] > Subject: [EXT] Re: [CODE4LIB] clustering techniques for normalizing > bibliographic data > > Hi Eric, > > I am planning to work on detecting such anomalities. What I have > thought about so far the following approaches: > - n-gram analysis > - basket analysis > - similarity detection of Solr > - final state automat > > The tools I will use: Apache Solr and Apache Spark. I haven't started > yet the implementation. > > Best, > Péter > > > 2017-10-25 17:57 GMT+02:00 Eric Lease Morgan <[log in to unmask]>: > > Has anybody here played with any clustering techniques for normalizing > bibliographic data? > > > > My bibliographic data is fraught with inconsistencies. For example, a > publisher’s name may be recorded one way, another way, or a third way. The > same goes for things like publisher place: South Bend; South Bend, IN; > South Bend, Ind. And then there is the ISBD punctuation that is sometimes > applied and sometimes not. All of these inconsistencies make indexing & > faceted browsing more difficult than it needs to be. > > > > OpenRefine is a really good program for finding these inconsistencies > and then normalizing them. OpenRefine calls this process “clustering”, and > it points to a nice page describing the various clustering processes. [1] > Some of the techniques included “fingerprinting” and calculating “nearest > neighbors”. Unfortunately, OpenRefine is not really programable, and I’d > like to automate much of this process. > > > > Does anybody here have any experience automating the process of > normalize bibliographic (MARC) data? > > > > [1] about clustering - http://bit.ly/2izQarE > > > > — > > Eric Morgan > > > > -- > Péter Király > software developer > GWDG, Göttingen - Europeana - eXtensible Catalog - The Code4Lib Journal > http://linkedin.com/in/peterkiraly >