 |
 |
On Fri, Oct 3, 2014 at 7:26 AM, Charles Blair <[log in to unmask]> wrote: > Look at slide 15 here: > http://www.slideshare.net/DuraSpace/sds-cwebinar-1 > > I think we're worried about the cumulative effect over time of > undetected errors (at least, I am). This slide shows that data loss via drive fault is extremely rare. Note that a bit getting flipped is usually harmless. However, I do believe that data corruption via other avenues will be considerably more common. My point is that the use case for libraries is generally weak and the solution is very expensive -- don't forget the authenticity checks must also be done on the "good" files. As you start dealing with more and more data, this system is not sustainable for the simple reason that maintained disk space costs a fortune and network capacity is a bottleneck. It's no big deal to do this on a few TB since our repositories don't have to worry about the integrity of dynamic data, but you eventually get to a point where it sucks up too many systems resources and consumes too much expertise. Authoritative files really should be offline but if online access to authoritative files is seen as an imperative, it at least makes more sense to just do something like dump it all in Glacier and slowly refresh everything you own with authoritative copy. Or better yet, just leave the stuff there and just make new derivatives when there is any reason to believe the existing ones are not good. While I think integrity is an issue, I think other deficiencies in repositories are more pressing. Except for the simplest use cases, getting stuff in or out of them is a hopeless process even with automated assistance. Metadata and maintenance aren't very good either. That you still need coding skills to get popular platforms that have been in use for many years to ingest and serve up things as simple as documents and images speaks volumes. kyle