Print

Print
I've used the Perl Digest::SHA module for setting fixity for the Census 2000 and 2010 data we received, and I'm currently using the same in a project to add fixity setting for the records we receive from the various congressional committees (our ERA CRI instance). It requires a Perl environment on your system, of course; for the projects I'm doing in a Windows environment, I'm using cygwin as the base environment.

One of the issues I have run across with this particular solution in a Windows environment is if you are processing a large number of files in a single script, and those files reside on a network drive, the Digest::SHA module will occasionally kick out an error attempting to calculate the SHA-256 hash code for a file. For example, in our Census 2000 project, I found we were encountering approximate 3 errors for every 100,000 files processed. I had to build in a retry mechanism to work around this problem which worked since the errors were never on contiguous files in the list.

Regards,


Kevin McCarthy
Computer Scientist
National Archives and Records Administration
Electronic Records Archives (ERA)
8601 Adelphi Rd. Room B530
College Park, MD 20740-6001
(301) 837-1657
[log in to unmask]


On Wed, Apr 2, 2014 at 1:16 PM, Shawn Rounds <[log in to unmask]> wrote:
Hello,

Reading over the minutes from the last call and draft of the fixity paper prompts me to pose a question to all of you.  Do you use or know of any stand-alone tools/programs for automated, server-side fixity checks of large files batches (thousands of files)?  Not necessarily looking for open source.  Any pointers to online resource lists on this topic?   

Thanks very much,
Shawn

Shawn Rounds
State Archivist
State Archives Department
Division of Library and Collections
Minnesota Historical Society
345 Kellogg Boulevard West
Saint Paul, Minnesota 55102

MNHS online: www.mnhs.org




On Wed, Mar 26, 2014 at 12:51 PM, Howard, Barrie <[log in to unmask]> wrote:

Good afternoon NDSA Infrastructure Working Group Members,

 

I tidied up the notes from yesterday’s call and have posted them to the wiki, http://www.loc.gov/extranet/wiki/osi/ndiip/ndsa/index.php?title=Tuesday,_Mar_25,_2014 

 

Kind regards,

 

Barrie Howard

NDSA Secretariat

+1-202-707-1830