Look at slide 15 here:
http://www.slideshare.net/DuraSpace/sds-cwebinar-1

I think we're worried about the cumulative effect over time of
undetected errors (at least, I am).

On Fri, Oct 03, 2014 at 05:37:14AM -0700, Kyle Banerjee wrote:
> On Thu, Oct 2, 2014 at 3:47 PM, Simon Spero <[log in to unmask]> wrote:
> 
> > Checksums can be kept separate (tripwire style).
> > For JHU archiving, the use of MD5 would give false positives for duplicate
> > detection.
> >
> > There is no reason to use a bad cryptographic hash. Use a fast hash, or use
> > a safe hash.
> >
> 
> I have always been puzzled why so much energy is expended on bit integrity
> in the library and archival communities. Hashing does not accommodate
> modification of internal metadata or compression which do not compromise
> integrity. And if people who can access the files can also access the
> hashes, there is no contribution to security. Also, wholesale hashing of
> repositories scales poorly,  My guess is that the biggest threats are staff
> error or rogue processes (i.e. bad programming). Any malicious
> destruction/modification is likely to be an inside job.
> 
> In reality, using file size alone is probably sufficient for detecting
> changed files -- if dup detection is desired, then hashing the few that dup
> out can be performed. Though if dups are an actual issue, it reflects
> problems elsewhere. Thrashing disks and cooking the CPU for the purposes
> libraries use hashes for seems way overkill, especially given that basic
> interaction with repositories for depositors, maintainers, and users is
> still in a very primitive state.
> 
> kyle
> 

-- 
Charles Blair, Director, Digital Library Development Center, University of Chicago Library
1 773 702 8459 | [log in to unmask] | http://www.lib.uchicago.edu/~chas/