Hello Code4Lib,

I received a question about deduping from one of our archivists and I'm
wondering if anyone has any experience/recommendations for this sort of
thing.

In short: We received a hard drive that has massive amounts of duplicates,
and they are starting the process of deduping and arranging it. They want
somewhat finer control over which duplicates get retained (currently using
FSlint and Bitcurator), so they can ensure 'complete sets' of files are
retained. But it'd be great to not have to manually select *every* dedup
preference in FSlint.

For example:
1. There is at least one folder that contains numbered audio tracks. When
we ran fslint raw, a few of these got deduped in favor of other copies in
the filesystem. But it would have been preferred to keep these together.
2. If there is a directory in which most of the working files were
originally created together.
3. We'd also generally prefer to keep the copies that will *not* result in,
post-dedup, folders containing only a single file scattered throughout the
directory.

Hopefully some of that makes sense. Has anyone found any helpful workflows
for streamlining the deduping/arranging process?

All I could come up with is logging all of FSlint's decisions, so that any
undesirable dedups could be more easily be tracked/reversed later, but I
really just don't know enough about any of this.

Thank you very much for your time and thoughts.

All the best,
Emily


-- 
Emily Lavins
Associate Systems Librarian
Boston College Libraries