 |
 |
hi code4lib, if you're archiving web content, please use the WARC format. thanks, [log in to unmask] WARC File Format Published as an International Standard http://netpreserve.org/press/pr20090601.php ISO 28500:2009 specifies the WARC file format: * to store both the payload content and control information from mainstream Internet application layer protocols, such as the Hypertext Transfer Protocol (HTTP), Domain Name System (DNS), and File Transfer Protocol (FTP); * to store arbitrary metadata linked to other stored data (e.g. subject classifier, discovered language, encoding); * to support data compression and maintain data record integrity; * to store all control information from the harvesting protocol (e.g. request headers), not just response information; * to store the results of data transformations linked to other stored data; * to store a duplicate detection event linked to other stored data (to reduce storage in the presence of identical or substantially similar resources); * to be extended without disruption to existing functionality; * to support handling of overly long records by truncation or segmentation, where desired. more info here: http://www.digitalpreservation.gov/formats/fdd/fdd000236.shtml