LISTSERV mailing list manager LISTSERV 16.5

Help for NDSA-INFRASTRUCTURE Archives


NDSA-INFRASTRUCTURE Archives

NDSA-INFRASTRUCTURE Archives


NDSA-INFRASTRUCTURE@LISTS.CLIR.ORG


View:

Message:

[

First

|

Previous

|

Next

|

Last

]

By Topic:

[

First

|

Previous

|

Next

|

Last

]

By Author:

[

First

|

Previous

|

Next

|

Last

]

Font:

Monospaced Font

LISTSERV Archives

LISTSERV Archives

NDSA-INFRASTRUCTURE Home

NDSA-INFRASTRUCTURE Home

NDSA-INFRASTRUCTURE  March 2012, Week 1

NDSA-INFRASTRUCTURE March 2012, Week 1

Subject:

compression in preservation storage

From:

Priscilla Caplan <[log in to unmask]>

Reply-To:

The NDSA infrastructure working group list <[log in to unmask]>

Date:

Tue, 6 Mar 2012 16:50:07 -0500

Content-Type:

text/plain

Parts/Attachments:

Parts/Attachments

text/plain (116 lines)

Time to start our next issue-oriented conversation, this time about data
compression.


Data compression can decrease the cost of long-term preservation by
reducing the amount of storage required. There are at least three types
of compression to consider:

-- file compression, using a file compression algorithm suited to the
file type
-- hardware compression, which usually means compression done by a tape
drive as the data is written to tape
-- disk compression, which is performed by many new storage appliances
and uses a combination of compression and deduplication

If there are other kinds of compression, please add that into this
discussion.

*For each of these types of compression:
*

1. Are you currently using this type of compression in your own
archival storage (in the OAIS sense of long-term preservation storage)?

2. How do you feel about using this type of compression for archival
storage? Is this legitimate or something that Best Practice would
discourage?

3. What are the particular risks of this type of compression, if any,
for preservation?

4. Are there any advantages to using this type of compression beyond
reducing storage costs?

5. How do you trade off cost vs. risk?


I looked for best practices or other documents that addressed
compression in the preservation context. Below are some snippets of
what I found, addressing mainly file and hardware compression. *
*

*
Case Western University Archives:*

Compression adds complexity to long-term preservation. Some compression
techniques shed "redundant" information. As an example, JPEG removes
information to reduce file size. The image might look fine on your
current monitor, but as monitors improve, the lower quality of the image
will be more obvious

*Wright, Miller and Addis* in
http://www.prestoprime.org/docs/training/Cost_of_risk_RW.pdf

Not encoding, in particular not using compression, typically results in
files that have minimal sensitivity to corruption. In this way, the
choice not to use compression is a way to mitigate against loss.

*TNA on Image Compression* in
http://www.nationalarchives.gov.uk/documents/image_compression.pdf

It is recommended that algorithms should only be used in the
circumstances for which they are most efficient. It is also strongly
recommended that archival master versions of images should only be
created and stored using lossless algorithms. The Intellectual Property
Rights status of a compression algorithm is primarily an issue for
developers of format specifications, and software encoders/decoders.
However, the use
of open, non-proprietary compression techniques is recommended for the
purposes of sustainability.

*Howard Besser*, quoted in
http://digitalpreservationstrategies.blogspot.com/

Data is often compressed or "scrambled" to assist in its storage and or
protect it's intellectual content. These compression and encryption
algorythms are often developed by private organisations who will one day
cease to support them. If this happens you're stuck between a rock and a
hard place. If you don't want to get into legal trouble you are no
longer able to read your data; and if you go ahead and "do the
unwrapping yourself" it's quite possible you're breaking copyright law.

*NINCH Guide to Good Practice*
http://www.nyu.edu/its/humanities/ninchguide/XIV/

A similar obsolescence problem will have to be addressed with the file
formats and compression techniques you choose. Do not rely on
proprietary file formats and compression techniques, which may not be
supported in the future as the companies which produce them merge, go
out of business or move on to new products. In the cultural heritage
community, thede factostandard formats are uncompressed TIFF for images
and PDF, ACSII (SGML/XML markup) and RTF for text. Migration to future
versions of these formats is likely to be well-supported, since they are
so widely used. Digital objects for preservation should not be stored in
compressed or encrypted formats.

*PRESTO Centre, Threats to Data Integrity from Large-Scale Management
Environments*
http://www.prestocentre.org/library/resources/threats-data-integrity-use-large-scale-management-environments

Compressed formats are in general much more sensitive to data corruption
than uncompressed formats. Due to the 'amplification' effect that
compression has on data corruption, the percentage saving in storage
space is often much less than the percentage increase in the amount of
information that is affected by data corruption.


Priscilla


############################

To unsubscribe from the NDSA-INFRASTRUCTURE list:
write to: mailto:[log in to unmask]
or click the following link:
http://list.digitalpreservation.gov/SCRIPTS/WA-DIGITAL.EXE?SUBED1=NDSA-INFRASTRUCTURE&A=1

Top of Message | Previous Page | Permalink

Advanced Options


Options

Log In

Log In

Get Password

Get Password


Search Archives

Search Archives


Subscribe or Unsubscribe

Subscribe or Unsubscribe


Archives

March 2024
May 2023
March 2023
February 2023
September 2022
July 2022
June 2022
January 2022
December 2021
November 2021
October 2021
September 2021
August 2021
July 2021
June 2021
May 2021
April 2021
March 2021
February 2021
January 2021
December 2020
November 2020
October 2020
September 2020
August 2020
July 2020
June 2020
May 2020
April 2020
March 2020
February 2020
January 2020
December 2019
November 2019
September 2019
August 2019
July 2019
June 2019
May 2019
April 2019
March 2019
February 2019
January 2019
November 2018
October 2018
September 2018
August 2018
July 2018
June 2018
May 2018
April 2018
March 2018
February 2018
January 2018
December 2017
November 2017
October 2017
November 2016
September 2016
August 2016
July 2016
June 2016
May 2016
April 2016
March 2016
February 2016
January 2016
December 2015
November 2015
October 2015
September 2015
August 2015
July 2015
June 2015
May 2015
April 2015
March 2015
February 2015
January 2015
November 2014
October 2014
August 2014
July 2014
June 2014
May 2014
April 2014
March 2014, Week 2
March 2014
February 2014, Week 4
February 2014, Week 3
February 2014, Week 2
February 2014, Week 1
January 2014, Week 5
January 2014, Week 4
January 2014, Week 1
December 2013, Week 2
November 2013, Week 4
November 2013, Week 2
October 2013, Week 5
October 2013, Week 3
October 2013, Week 1
September 2013, Week 3
August 2013, Week 5
August 2013, Week 4
August 2013, Week 1
July 2013, Week 5
June 2013, Week 4
June 2013, Week 3
June 2013, Week 2
May 2013, Week 3
April 2013, Week 5
April 2013, Week 4
April 2013, Week 3
March 2013, Week 4
March 2013, Week 3
March 2013, Week 2
March 2013, Week 1
February 2013, Week 4
February 2013, Week 3
February 2013, Week 2
February 2013, Week 1
January 2013, Week 5
January 2013, Week 4
January 2013, Week 3
January 2013, Week 2
December 2012, Week 2
November 2012, Week 4
November 2012, Week 3
November 2012, Week 2
November 2012, Week 1
October 2012, Week 5
October 2012, Week 3
October 2012, Week 2
October 2012, Week 1
September 2012, Week 4
September 2012, Week 3
August 2012, Week 5
August 2012, Week 4
August 2012, Week 3
August 2012, Week 2
August 2012, Week 1
July 2012, Week 5
July 2012, Week 2
June 2012, Week 4
June 2012, Week 3
May 2012, Week 5
May 2012, Week 3
May 2012, Week 2
May 2012, Week 1
April 2012, Week 5
April 2012, Week 4
April 2012, Week 3
April 2012, Week 2
March 2012, Week 5
March 2012, Week 4
March 2012, Week 2
March 2012, Week 1
February 2012, Week 4
February 2012, Week 3
February 2012, Week 2
February 2012, Week 1
January 2012, Week 5
January 2012, Week 4
January 2012, Week 3
January 2012, Week 1
December 2011, Week 3
December 2011, Week 2
December 2011, Week 1
November 2011, Week 4
November 2011, Week 3
November 2011, Week 2
November 2011, Week 1
October 2011, Week 3
September 2011, Week 4
September 2011, Week 3
September 2011, Week 1
August 2011, Week 5
August 2011, Week 4
August 2011, Week 3
August 2011, Week 1
July 2011, Week 5
July 2011, Week 4
July 2011, Week 1
June 2011, Week 3
June 2011, Week 2
June 2011, Week 1
May 2011, Week 4
May 2011, Week 2
May 2011, Week 1
April 2011, Week 3
April 2011, Week 2
April 2011, Week 1
March 2011, Week 5
March 2011, Week 4
March 2011, Week 3
March 2011, Week 2
February 2011, Week 4
February 2011, Week 3
February 2011, Week 2
February 2011, Week 1
January 2011, Week 4
January 2011, Week 3
January 2011, Week 1
December 2010, Week 3
December 2010, Week 2
October 2010, Week 2
September 2010, Week 3
September 2010, Week 2
September 2010, Week 1
August 2010, Week 5

ATOM RSS1 RSS2



LISTS.CLIR.ORG

CataList Email List Search Powered by the LISTSERV Email List Manager