My concern would be more that given proven weaknesses in MD5, do I want to risk that 1 in a billion chance that the “right” bit error creeps into an archive that manages to not impact the checksum, thus creating the illusion that the archive integrity has not been violated?

-- 
Andrew Anderson, Director of Development, Library and Information Resources Network, Inc.
http://www.lirn.net/ | http://www.twitter.com/LIRNnotes | http://www.facebook.com/LIRNnotes

On Oct 2, 2014, at 18:34, Jonathan Rochkind <[log in to unmask]> wrote:

> For checksums for ensuring archival integrity, are cryptographic flaws relavent? I'm not sure, is part of the point of a checksum to ensure against _malicious_ changes to files?  I honestly don't know. (But in most systems, I'd guess anyone who had access to maliciously change the file would also have access to maliciously change the checksum!)
> 
> Rot13 is not suitable as a checksum for ensuring archival integrity however, because it's output is no smaller than it's input, which is kind of what you're looking for. 
> 
> ________________________________________
> From: Code for Libraries [[log in to unmask]] on behalf of Cary Gordon [[log in to unmask]]
> Sent: Thursday, October 02, 2014 5:51 PM
> To: [log in to unmask]
> Subject: Re: [CODE4LIB] What is the real impact of SHA-256? - Updated
> 
> +1
> 
> MD5 is little better than ROT13. At least with ROT13, you have no illusions.
> 
> We use SHA 512 for most work. We don't do finance or national security, so it is a good fit for us.
> 
> Cary
> 
> On Oct 2, 2014, at 12:30 PM, Simon Spero <[log in to unmask]> wrote:
> 
>> Intel skylake processors have dedicated sha instructions.
>> See: https://software.intel.com/en-us/articles/intel-sha-extensions
>> 
>> Using a tree hash approach (which is inherently embarrassingly parallel)
>> will leave io time dominant. This approach is used by Amazon glacier - see
>> http://docs.aws.amazon.com/amazonglacier/latest/dev/checksum-calculations.html
>> 
>> MD5 is broken, and cannot be used for any security purposes. It cannot be
>> used for deduplication if any of the files are in the directories of
>> security researchers!
>> 
>> If security is not a concern then there are many faster hashing algorithms
>> that avoid the costs imposed by the need to defend against adversaries.
>> See siphash, murmur, cityhash, etc.
>> 
>> Simon
>> On Oct 2, 2014 11:18 AM, "Alex Duryee" <[log in to unmask]> wrote:
>> 
>>> Despite some of its relative flaws, MD5 is frequently selected over SHA-256
>>> in archives as the checksum algorithm of choice. One of the primary factors
>>> here is the longer processing time required for SHA-256, though there have
>>> been no empirical studies calculating that time difference and its overall
>>> impact on checksum generation and verification in a preservation
>>> environment.
>>> 
>>> AVPreserve Consultant Alex Duryee recently ran a series of tests comparing
>>> the real time and cpu time used by each algorithm. His newly updated white
>>> paper "What Is the Real Impact of SHA-256?" presents the results and comes
>>> to some interesting conclusions regarding the actual time difference
>>> between the two and what other factors may have a greater impact on your
>>> selection decision and file monitoring workflow. The paper can be
>>> downloaded for free at
>>> 
>>> http://www.avpreserve.com/papers-and-presentations/whats-the-real-impact-of-sha-256/
>>> .
>>> ______________________________________
>>> 
>>> Alex Duryee
>>> *AVPreserve*
>>> 350 7th Ave., Suite 1605
>>> New York, NY 10001
>>> 
>>> office: 917-475-9630
>>> 
>>> http://www.avpreserve.com
>>> Facebook.com/AVPreserve <http://facebook.com/AVPreserve>
>>> twitter.com/AVPreserve
>>>