In response to Michelle's email below, I'm sharing the abstract and
conclusion that comes from a paper my co-authors and I are submitting
related to the Data Conservancy (https://dataconservancy.org/) provenance
and lineage service. The "stack model" mentioned below is described in the
attached poster from the most recent IDCC conference. This stack model is
also described in the YouTube clip available at
https://www.youtube.com/watch?v=F6iYXNvCRO4

Sayeed


Data Conservancy Provenance, Context, and Lineage Services: Key Components
for Data Preservation and Curation

Matthew S. Mayernik, G. Sayeed Choudhury, Tim DiLauro, Ruth Duerr, Elliot
Metsger, Anne Thessen

For submission to the Data Science Journal: http://www.codata.org/dsj/

Abstract
Among the key services that an institutional data management
infrastructure must provide is provenance and lineage tracking and the
ability to associate data with the contextual information needed to
understand and use the data. These functionalities are critical for
addressing a number of key issues faced by data creators/collectors and
users such as trust in data, results traceability, transparency of data
modification or analysis processes, data citation support, etc. In this
paper, we describe the support for these services within the Data
Conservancy Service (DCS) software. The DCS provenance, context and
lineage services provide functionalities that cross the four layers in the
DCS data curation stack model: storage, archiving, preservation, and
curation.


Conclusion
In order to ensure that data remain understandable and usable, data
infrastructure must manage relationships and processes in addition to the
resources themselves. Provenance, context and lineage services provide
scalable ways to trace and manage relationships between numerous
resources, and to track the processes used to manipulate, transform, and
preserve resources. Within the Data Conservancy Service, the provenance
functionality documents internal processes, the lineage service
establishes linkages between successive versions of submitted resources,
and the ability to record relationships between resources allow users to
associate provenance and contextual information with a resource. These
services reflect the characteristics of provenance ready computational
systems noted by Groth, et al. (2012). The results of completed processes
are compiled into the provenance and lineage streams as system events by
the DCS, reflecting how the processes were executed, and their success or
failure. DC provenance and lineage streams are autonomously created by the
DCS as system events occur, and are immutable once created.

From a service stack perspective, the DCS provenance and lineage services
provide functionalities that cross the four layers in the Data Conservancy
Service stack model: storage, archiving, preservation, and curation.

● Storage - The DCS begins storing provenance information upon ingest
along with the resources themselves. The stream of DCS-generated
provenance information can then be leveraged to assess the success and
failure of individual processes, and of sets of processes, such as the
ingest process.

● Archiving - The DCS creates identifiers for individual entities as they
are ingested, and creates a lineage identifier that identifies a set of
related resources. The lineage identifier also provides the
mechanism by which lineage information can be queried and retrieved.

● Preservation - Capturing system events as provenance and lineage
information is one component of a preservation-ready system, in that these
streams allow people and machines other than the original data producer to
use and interpret the data. Providing users the capability of associating
a resource in defined ways with other information irrespective of whether
that information is held within the DCS or not; maximizes the chances that
all of the information needed by future users will be available.

● Curation - The DCS allows additional features to be built on the lineage
service and the ability to define relationships between resources both
within and outside of the DCS. The lineage service will
underpin user interfaces that allow users to navigate through changes in
versions, understand an objects history and context. This flexibility and
customizability is a design feature of the software. In the alpha release
of the Data Conservancy software, the reference user interface hides the
details of the lineage service from the user.

Provenance, context and lineage functionalities are essential components
of infrastructure for digital research data. The Data Conservancy stack
model helps to identify where provenance and lineage functionalities fit
within the storage, archiving, preservation, and curation service layers.

Future extensions of the work described in this paper will investigate the
consequences of allowing users to assert pre-existing provenance events
about objects during accession by the DCS archive, in order to, for
example, represent format conversions that took place prior to objects
being uploaded to the DCS. Other extensions will explore the potential for
a unified event model between the business layer and the archive, and
develop an understanding of how preservation actions may affect the
lineage of archival object graphs.


On 6/18/13 8:17 AM, "Gallinger, Michelle" <[log in to unmask]> wrote:

>We are interested in learning more about what people are doing in regards
>to provenance.
>
>Are you working on a project that has a provenance component? do you know
>of interesting provenance work being done by others? Please share!
>
>Thanks so much.
>
>All best,
>
>Michelle
>
>Michelle Gallinger
>Digital Programs Coordinator
>Library of Congress
>101 Independence Ave. SE
>Washington, DC 20007
>202-707-1619
>[log in to unmask]
>
>
>############################
>
>To unsubscribe from the NDSA-ALL list:
>write to: mailto:[log in to unmask]
>or click the following link:
>http://list.digitalpreservation.gov/scripts/wa-DIGITAL.exe?SUBED1=NDSA-ALL
>&A=1


############################

To unsubscribe from the NDSA-ALL list:
write to: mailto:[log in to unmask]
or click the following link:
http://list.digitalpreservation.gov/scripts/wa-DIGITAL.exe?SUBED1=NDSA-ALL&A=1