 |
 |
If you have no opposition to Python, I suggest looking at Fuzzywuzzy: https://github.com/seatgeek/fuzzywuzzy M. -- Mark A. Matienzo <[log in to unmask]> Director of Technology, Digital Public Library of America On Fri, Mar 21, 2014 at 2:34 PM, Andrew Gordon <[log in to unmask]> wrote: > Ken, > > A group in Chicago has been working for a few years now on a deduplication > toolkit that might do what you are looking for, they also have a couple > versions that works with an excel file or .csv file. > > https://github.com/datamade/dedupe > https://github.com/datamade/dedupe-web > https://github.com/datamade/csvdedupe > > I have not worked with them extensively, but I have heard others find > these very useful for entity recognition and resolution. > > > > > > > -----Original Message----- > From: Code for Libraries [mailto:[log in to unmask]] On Behalf Of > Ken Irwin > Sent: Friday, March 21, 2014 2:25 PM > To: [log in to unmask] > Subject: [CODE4LIB] tool for finding close matches in vocabular list > > Hi folks, > > I'm looking for a tool that can look at a list of all of subject terms in > a poorly-controlled index as possible candidates for term consolidation. > Our student newspaper index has about 16,000 subject terms and they include > a lot of meaningless typographical and nomenclatural difference, e.g.: > > Irwin, Ken > Irwin, Kenneth > Irwin, Mr. Kenneth > Irwin, Kenneth R. > > Basketball - Women > Basketball - Women's > Basketball-Women > Basketball-Women's > > I would love to have some sort of pattern-matching tool that's smart about > this sort of thing that could go through the list of terms (as a text list, > database, xml file, or whatever structure it wants to ingest) and spit out > some clusters of possible matches. > > Does anyone know of a tool that's good for that sort of thing? > > The index is just a bunch of MySQL tables - there is no real > controlled-vocab system, though I've recently built some systems to suggest > known SH's to reduce this sort of redundancy. > > Any ideas? > > Thanks! > Ken >