Checksums can be kept separate (tripwire style).
For JHU archiving, the use of MD5 would give false positives for duplicate
detection.

There is no reason to use a bad cryptographic hash. Use a fast hash, or use
a safe hash.

Simon
On Oct 2, 2014 6:34 PM, "Jonathan Rochkind" <[log in to unmask]> wrote:

> For checksums for ensuring archival integrity, are cryptographic flaws
> relavent? I'm not sure, is part of the point of a checksum to ensure
> against _malicious_ changes to files?  I honestly don't know. (But in most
> systems, I'd guess anyone who had access to maliciously change the file
> would also have access to maliciously change the checksum!)
>
> Rot13 is not suitable as a checksum for ensuring archival integrity
> however, because it's output is no smaller than it's input, which is kind
> of what you're looking for.
>
> ________________________________________
> From: Code for Libraries [[log in to unmask]] on behalf of Cary
> Gordon [[log in to unmask]]
> Sent: Thursday, October 02, 2014 5:51 PM
> To: [log in to unmask]
> Subject: Re: [CODE4LIB] What is the real impact of SHA-256? - Updated
>
> +1
>
> MD5 is little better than ROT13. At least with ROT13, you have no
> illusions.
>
> We use SHA 512 for most work. We don't do finance or national security, so
> it is a good fit for us.
>
> Cary
>
> On Oct 2, 2014, at 12:30 PM, Simon Spero <[log in to unmask]> wrote:
>
> > Intel skylake processors have dedicated sha instructions.
> > See: https://software.intel.com/en-us/articles/intel-sha-extensions
> >
> > Using a tree hash approach (which is inherently embarrassingly parallel)
> > will leave io time dominant. This approach is used by Amazon glacier -
> see
> >
> http://docs.aws.amazon.com/amazonglacier/latest/dev/checksum-calculations.html
> >
> > MD5 is broken, and cannot be used for any security purposes. It cannot be
> > used for deduplication if any of the files are in the directories of
> > security researchers!
> >
> > If security is not a concern then there are many faster hashing
> algorithms
> > that avoid the costs imposed by the need to defend against adversaries.
> > See siphash, murmur, cityhash, etc.
> >
> > Simon
> > On Oct 2, 2014 11:18 AM, "Alex Duryee" <[log in to unmask]> wrote:
> >
> >> Despite some of its relative flaws, MD5 is frequently selected over
> SHA-256
> >> in archives as the checksum algorithm of choice. One of the primary
> factors
> >> here is the longer processing time required for SHA-256, though there
> have
> >> been no empirical studies calculating that time difference and its
> overall
> >> impact on checksum generation and verification in a preservation
> >> environment.
> >>
> >> AVPreserve Consultant Alex Duryee recently ran a series of tests
> comparing
> >> the real time and cpu time used by each algorithm. His newly updated
> white
> >> paper "What Is the Real Impact of SHA-256?" presents the results and
> comes
> >> to some interesting conclusions regarding the actual time difference
> >> between the two and what other factors may have a greater impact on your
> >> selection decision and file monitoring workflow. The paper can be
> >> downloaded for free at
> >>
> >>
> http://www.avpreserve.com/papers-and-presentations/whats-the-real-impact-of-sha-256/
> >> .
> >> ______________________________________
> >>
> >> Alex Duryee
> >> *AVPreserve*
> >> 350 7th Ave., Suite 1605
> >> New York, NY 10001
> >>
> >> office: 917-475-9630
> >>
> >> http://www.avpreserve.com
> >> Facebook.com/AVPreserve <http://facebook.com/AVPreserve>
> >> twitter.com/AVPreserve
> >>
>