Kyle --

Although my example doesn't apply for all file formats, it does give an
example of what you're looking for:

BWFMetaEdit (
http://www.digitizationguidelines.gov/guidelines/digitize-embedding.html)
is free tool developed by Federal Agency groups to allow for the
reading/writing of metadata into the BWF and RIFF (BEXT & INFO
respectively) text chunks of WAV audio files. The salient point here is
that this approach was designed with the ability to generate and embed a
checksum of the PCM audio stream within the WAV container so that as new
metadata are added to the container, the audio can be validated against its
specific checksum, not a checksum of the entire container. In this
practice, one can generate a checksum for the audio information (the
content) and for the entire file itself (the content and the metadata).

Take a read through that and maybe it will inspire some ideas.

I know in the moving image field there is also much activity around frame
by frame checksums for moving image material so that when a file is found
to be corrupt, you can even pinpoint which frame has the corruption.

Best --

Bert


Bertram Lyons, CA
AVPreserve | www.avpreserve.com
American Folklife Center | www.loc.gov/folklife
International Association of Sound and Audiovisual Archives |
www.iasa-web.org

On Mon, Jan 26, 2015 at 6:21 AM, Scancella, John <[log in to unmask]> wrote:

> The library of congress has several tools for making and working with
> bagit bags.
>
> Java command line tool and library
> https://github.com/LibraryOfCongress/bagit-java
>
> a python command line tool and library
> https://github.com/LibraryOfCongress/bagit-python
>
> or a standalone java desktop application (GUI based)
> https://github.com/LibraryOfCongress/bagger
>
> -----Original Message-----
> From: Code for Libraries [mailto:[log in to unmask]] On Behalf Of
> Joe Hourcle
> Sent: Saturday, January 24, 2015 10:07 PM
> To: [log in to unmask]
> Subject: Re: [CODE4LIB] Checksums for objects and not embedded metadata
>
> On Jan 23, 2015, at 5:35 PM, Kyle Banerjee wrote:
>
> > Howdy all,
> >
> > I've been toying with the idea of embedding DOI's in all our digital
> > assets and possibly inserting/updating other metadata as well.
> > However, doing this would alter checksums created using normal methods.
> >
> > Is there a practical/easy way to checksum only the objects themselves
> > without the metadata? If the metadata in a tiff or other kind of file
> > is modified, it does nothing to the actual object. Since providing
> > more complete metadata within objects makes them more
> > usable/identifiable and might simplify migrations down the road, it
> > seems like this wouldn't be a bad way to go.
>
>
> The only file format that I'm aware of that has a provision for this is
> FITS (Flexible Image Transport System), which was a concept of a 'CHECKSUM'
> and a 'DATASUM'.  (the 'DATASUM' is the checksum for only the payload
> portion, the 'CHECKSUM' includes the metadata)[1].  It's possible that
> there are others, but I suspect that most consumer file formats won't have
> specific provisions for this.
>
> The problems with 'metadata' in a lot of file formats is that they're just
> arbitrary segments -- you'd have to have a program that knew which segments
> were considered 'headers' vs. not.  It might be easier to have it be able
> to compute a separate checksum for each segment, so that should the
> modifications change their order, they'd still be considered valid.
>
> Of course, I personally don't like changing files if I can help it.
> If it were me, I'd keep the metadata outside the file;  if you're using
> BagIt, you could easily add additional metadata outside of the data
> directory.[2]
>
> If you're just doing this internally, and don't need the DOI to be
> attached to the file when it's served, you could also look into file
> systems that support arbitrary metadata.  Older Macs used to use this,
> where there was a 'data fork' and a 'resource fork', but you had to have a
> service that knew to only send the data fork.
> Other OSes support forks, but some also have 'extended file attributes',
> which allows you to attach a few key/value pairs to the file.  (exact
> limits are dependent upon the OS).
>
> -Joe
>
>
> [1] http://fits.gsfc.nasa.gov/registry/checksum.html
> [2] https://tools.ietf.org/html/draft-kunze-bagit ;
> http://en.wikipedia.org/wiki/BagIt
>