 |
 |
In response to Michelle's email below, I'm sharing the abstract and conclusion that comes from a paper my co-authors and I are submitting related to the Data Conservancy (https://dataconservancy.org/) provenance and lineage service. The "stack model" mentioned below is described in the attached poster from the most recent IDCC conference. This stack model is also described in the YouTube clip available at https://www.youtube.com/watch?v=F6iYXNvCRO4 Sayeed Data Conservancy Provenance, Context, and Lineage Services: Key Components for Data Preservation and Curation Matthew S. Mayernik, G. Sayeed Choudhury, Tim DiLauro, Ruth Duerr, Elliot Metsger, Anne Thessen For submission to the Data Science Journal: http://www.codata.org/dsj/ Abstract Among the key services that an institutional data management infrastructure must provide is provenance and lineage tracking and the ability to associate data with the contextual information needed to understand and use the data. These functionalities are critical for addressing a number of key issues faced by data creators/collectors and users such as trust in data, results traceability, transparency of data modification or analysis processes, data citation support, etc. In this paper, we describe the support for these services within the Data Conservancy Service (DCS) software. The DCS provenance, context and lineage services provide functionalities that cross the four layers in the DCS data curation stack model: storage, archiving, preservation, and curation. Conclusion In order to ensure that data remain understandable and usable, data infrastructure must manage relationships and processes in addition to the resources themselves. Provenance, context and lineage services provide scalable ways to trace and manage relationships between numerous resources, and to track the processes used to manipulate, transform, and preserve resources. Within the Data Conservancy Service, the provenance functionality documents internal processes, the lineage service establishes linkages between successive versions of submitted resources, and the ability to record relationships between resources allow users to associate provenance and contextual information with a resource. These services reflect the characteristics of provenance ready computational systems noted by Groth, et al. (2012). The results of completed processes are compiled into the provenance and lineage streams as system events by the DCS, reflecting how the processes were executed, and their success or failure. DC provenance and lineage streams are autonomously created by the DCS as system events occur, and are immutable once created. From a service stack perspective, the DCS provenance and lineage services provide functionalities that cross the four layers in the Data Conservancy Service stack model: storage, archiving, preservation, and curation. ● Storage - The DCS begins storing provenance information upon ingest along with the resources themselves. The stream of DCS-generated provenance information can then be leveraged to assess the success and failure of individual processes, and of sets of processes, such as the ingest process. ● Archiving - The DCS creates identifiers for individual entities as they are ingested, and creates a lineage identifier that identifies a set of related resources. The lineage identifier also provides the mechanism by which lineage information can be queried and retrieved. ● Preservation - Capturing system events as provenance and lineage information is one component of a preservation-ready system, in that these streams allow people and machines other than the original data producer to use and interpret the data. Providing users the capability of associating a resource in defined ways with other information irrespective of whether that information is held within the DCS or not; maximizes the chances that all of the information needed by future users will be available. ● Curation - The DCS allows additional features to be built on the lineage service and the ability to define relationships between resources both within and outside of the DCS. The lineage service will underpin user interfaces that allow users to navigate through changes in versions, understand an objects history and context. This flexibility and customizability is a design feature of the software. In the alpha release of the Data Conservancy software, the reference user interface hides the details of the lineage service from the user. Provenance, context and lineage functionalities are essential components of infrastructure for digital research data. The Data Conservancy stack model helps to identify where provenance and lineage functionalities fit within the storage, archiving, preservation, and curation service layers. Future extensions of the work described in this paper will investigate the consequences of allowing users to assert pre-existing provenance events about objects during accession by the DCS archive, in order to, for example, represent format conversions that took place prior to objects being uploaded to the DCS. Other extensions will explore the potential for a unified event model between the business layer and the archive, and develop an understanding of how preservation actions may affect the lineage of archival object graphs. On 6/18/13 8:17 AM, "Gallinger, Michelle" <[log in to unmask]> wrote: >We are interested in learning more about what people are doing in regards >to provenance. > >Are you working on a project that has a provenance component? do you know >of interesting provenance work being done by others? Please share! > >Thanks so much. > >All best, > >Michelle > >Michelle Gallinger >Digital Programs Coordinator >Library of Congress >101 Independence Ave. SE >Washington, DC 20007 >202-707-1619 >[log in to unmask] > > >############################ > >To unsubscribe from the NDSA-ALL list: >write to: mailto:[log in to unmask] >or click the following link: >http://list.digitalpreservation.gov/scripts/wa-DIGITAL.exe?SUBED1=NDSA-ALL >&A=1 ############################ To unsubscribe from the NDSA-ALL list: write to: mailto:[log in to unmask] or click the following link: http://list.digitalpreservation.gov/scripts/wa-DIGITAL.exe?SUBED1=NDSA-ALL&A=1