Have you considered dumping it into Open Refine?  I haven't used it a
lot, but it is likely a good tool to find similar data and allow you to
globally replace with a canonical entry.
On Fri, Mar 21, 2014 at 11:24 AM, Ken Irwin <[log in to unmask]> wrote:
> Hi folks,
> I'm looking for a tool that can look at a list of all of subject terms in
> a poorly-controlled index as possible candidates for term consolidation.
> Our student newspaper index has about 16,000 subject terms and they include
> a lot of meaningless typographical and nomenclatural difference, e.g.:
> Irwin, Ken
> Irwin, Kenneth
> Irwin, Mr. Kenneth
> Irwin, Kenneth R.
> Basketball - Women
> Basketball - Women's
> I would love to have some sort of pattern-matching tool that's smart about
> this sort of thing that could go through the list of terms (as a text list,
> database, xml file, or whatever structure it wants to ingest) and spit out
> some clusters of possible matches.
> Does anyone know of a tool that's good for that sort of thing?
> The index is just a bunch of MySQL tables - there is no real
> controlled-vocab system, though I've recently built some systems to suggest
> known SH's to reduce this sort of redundancy.
> Any ideas?