Simon - do you have any examples of MD5 collisions in JHU's collections?
The chance of that occurring is vanishingly small (
http://prezi.com/zfyebvaelksh/fixity-20/) so I'm curious what produced the
collision, and how often.

On Fri, Oct 3, 2014 at 12:14 PM, Kyle Banerjee <[log in to unmask]>
wrote:

> On Fri, Oct 3, 2014 at 7:26 AM, Charles Blair <[log in to unmask]> wrote:
>
> > Look at slide 15 here:
> > http://www.slideshare.net/DuraSpace/sds-cwebinar-1
> >
> > I think we're worried about the cumulative effect over time of
> > undetected errors (at least, I am).
>
>
> This slide shows that data loss via drive fault is extremely rare. Note
> that a bit getting flipped is usually harmless. However, I do believe that
> data corruption via other avenues will be considerably more common.
>
> My point is that the use case for libraries is generally weak and the
> solution is very expensive -- don't forget the authenticity checks must
> also be done on the "good" files. As you start dealing with more and more
> data, this system is not sustainable for the simple reason that maintained
> disk space costs a fortune and network capacity is a bottleneck. It's no
> big deal to do this on a few TB since our repositories don't have to worry
> about the integrity of dynamic data, but you eventually get to a point
> where it sucks up too many systems resources and consumes too much
> expertise.
>
> Authoritative files really should be offline but if online access to
> authoritative files is seen as an imperative, it at least makes more sense
> to just do something like dump it all in Glacier and slowly refresh
> everything you own with authoritative copy. Or better yet, just leave the
> stuff there and just make new derivatives when there is any reason to
> believe the existing ones are not good.
>
> While I think integrity is an issue, I think other deficiencies in
> repositories are  more pressing. Except for the simplest use cases, getting
> stuff in or out of them is a hopeless process even with automated
> assistance. Metadata and maintenance aren't very good either. That you
> still need coding skills to get popular platforms that have been in use for
> many years to ingest and serve up things as simple as documents and images
> speaks volumes.
>
> kyle
>