 |
 |
Hi, Eric – I haven't worked with HathiTrust but I have done some de-duping projects. How would you do one by hand? My workflow for deduping has been: -Export a big batch of MARC records -Load the file into a program -Have the program process the records one at a time -For each record, load them into a data structure. Organize the data structure so that duplicate records are all stored together. This may require additional logic. -Once all the records are loaded into the data structure, go through the data structure and process each cluster of duplicates. Pick one record from each cluster and write it to a new data structure used for output. -Go through the output data structure and write those records out to a file I use Java and Marc4J. For one project the records had a common field I could use as an identifier. I put them into a HashMap using that identifier as a key. For another project, I put them into an actual database. I think it was Derby. Sqlite is also a good db for the relatively small number of records libraries commonly work with (100k's to millions). I still used the DB as a relatively simple map. Being able to filter and perform additional processing steps with SQL was helpful. OCLC Classify API can also be thrown into the mix: http://classify.oclc.org/classify2/Classify?oclc=6741810&summary=true http://classify.oclc.org/classify2/Classify?oclc=6741810&summary=false Apologies if this info is too rudimentary for where you are starting from. If it's not rudimentary enough, I'd be happy to write a simple Java script that could be used as a starting point. Regards, Tom On Thu, Jan 25, 2018 at 9:54 AM, Eric Lease Morgan <[log in to unmask]> wrote: > Working with the HathiTrust Research Center data can be fun, and I sincerely believe it is an under-utilized system, but creating collections sans duplicates is difficult. Has anybody here figured out a “kewl” way to remove duplicates. > > Creating HathiTrust collections is easy: do search, select items of interest, and repeat until tired. One can then download a CSV file describing the collection, but upon closer inspection MANY of the titles are repeated. I know why this has happened, alas, but how might I automatically/programmatically resolve this issue? I’ve begun experimenting with OpenRefine. Does anybody else have other suggestions? > > — > Eric Morgan