On Jan 29, 2018, at 5:30 PM, Tom Hutchinson <[log in to unmask]> wrote:
> My workflow for deduping has been:
>
> -Export a big batch of MARC records
>
> -Load the file into a program
>
> -Have the program process the records one at a time
>
> -For each record, load them into a data structure. Organize the data
> structure so that duplicate records are all stored together. This may
> require additional logic.
>
> -Once all the records are loaded into the data structure, go through
> the data structure and process each cluster of duplicates. Pick one
> record from each cluster and write it to a new data structure used for
> output.
>
> -Go through the output data structure and write those records out to a file
Tom, thank you for sharing, and I’ve taken a different de-duplication tack.
More specifically, to de-duplicate items from a HathiTrust collection, I:
1) create a collection while trying to build it from a smaller number of
libraries with the assumption that each library will hold a
limited number of similar items
2) assume each of the items in my HathTrust collection have more
things in common than differences; assume a given duplicated
item is just as good another duplicated item (which we all know
is not really true)
3) pour my HathiTrust collection (which is a tab-delimited/CSV file)
into OpenRefine [1]
4) using OpenRefine's clustering algorithms, “normalize” as many
titles as possible, meaning, use things like the Levenshtein
Algorithm to “correct” differences [2]
5) delete duplicate items [3]
6) possibly use OpenRefine's global find/replace tools to manually do
more normalization
7) go to Step #4 until tired
8) sort the collection by title, and manually flag duplicate items
9) deleted flagged items
10) done
When and if I learn how to programmatically implement Step #4, then I think I will be able to automate the whole process.
[1] OpenRefine - http://openrefine.org
[2] clustering - https://github.com/OpenRefine/OpenRefine/wiki/Clustering
[3] “Removing duplicate rows when Exact values are found in a column” - https://github.com/OpenRefine/OpenRefine/wiki/Recipes
—
Eric Morgan
