# Data Provenance

## Overview

Various elements in XCEDE 2 refer to data or data resources that were generated from other input data by processing tools. Data provenance refers to the art of tracking the origins of the derived data and the processing steps that resulted in the output data. Besides being informative, provenance information could allow a user to recreate the processing stream and replicate the output data using the same tools and inputs. Provenance information could either be written by the software tools themselves, or could be extracted by wrapper scripts that write the provenance information on the tool's behalf (the likely scenario for external or legacy applications).

Some of the following documentation comes from Nicole Aucoin who is leading the Data Provenance effort for the MBIRN.

provenance_t

Data represented by an instance of dataResource_t or `analysis_t` may contain a `<provenance>` element of this type, which contains an ordered sequence of processing steps (processStep_t) representing the processing steps used to generate this data.

processStep_t

This type contains two attributes, ID and parent, indicating a unique ID for the step and its parent step, respectively. It includes a number of child elements that describe the conditions under which an executable was run. All of the child elements are formally optional. The following are strongly recommended:

program

  The name of the executable. Version and build numbers may be specified using the attributes `version` and `build`. See instructions below for inserting version control content. Alternately, you can define it as a user string.

programArguments

  The arguments passed into the executable. Optional attributes `inputs` and `outputs` are used to break down the full list of arguments into input and output arguments.

timeStamp

  The time at which the program started to run.

user

  Who was logged in and running this executable?

hostName

  The unique id of the computer running this executable. Replaces Machine from old schema.

architecture

  The type of hardware that the processing step is run on, `sun`, `x86`, etc.

platform

  The operating system under which the processing step is being run.

These are optional:

cvs

  Revision control repository information. Use the CVS keyword Id, inserted into your source code file as `$Id: chap_provenance.xml,v 1.15 2008-07-29 20:21:31 gadde Exp $` and it will be replaced with the file name, version, date of change and who changed it. Also works with SVN, but see SVN notes for enabling it.

compiler

  What was used to compile this executable? Name and version.

library

  The name and version of a linked library. Can be repeated as many times as necessary. This is dependent on the library in question, examples are given for VTK and Tcl and Tk.

buildTimeStamp

  The time stamp indicating when the executable was built.

package

  This tag gives the overall package version, if the executable is part of a larger unit. For example, mri_convert is part of FreeSurfer, and the whole FreeSurfer package may have a different version number than the mri_convert executable.

repository

  Where can the software be found? Link to a code repository or a web page. Can be extracted automatically if using SVN, via the HeadURL keyword.

## Examples

Provenance elements are children of dataReource and analysis types and describe the sequence of processing steps that were executed to generate the data represented by the parent element. In the example below, the provenance of the NIfTI data file (represented as a binaryDataResource) indicates that it was generated in two processing steps (filter1 and filter2) on the same machine (lablin1) about 30 mintues apart.

Simple provenance example

## Generating provenance XML from applications

The figure below illustrates a C code snippet (written by Nicole Aucoin and implemented in 3DSlicer) that generates provenance XML from an application. [Note: This may not be completely compliant with XCEDE2.]

    void printAllInfo(int argc, char **argv)
    {
      int i;
      struct tm * timeInfo;
      time_t rawtime;
      // yyyy/mm/dd-hh-mm-ss-ms-TZ
      // plus one for 3 char time zone
      char timeStr[27];

      fprintf(stdout, "<ProcessStep>\n");
      fprintf(stdout, "<Program version=\"$Revision: 1.15 $\">%s</ProgramName>\n", argv[0]);
      fprintf(stdout, "<ProgramArguments>");
      for (i = 1; i < argc; i++)
        {
        fprintf(stdout, " %s", argv[i]);
        }
      fprintf(stdout, "</ProgramArguments>\n");
      fprintf(stdout, "<CVS>$Id: chap_provenance.xml,v 1.15 2008-07-29 20:21:31 gadde Exp $</CVS> <TimeStamp>");
      time ( &rawtime );
      timeInfo = localtime (&rawtime);
      strftime (timeStr, 27, "%Y/%m/%d-%H-%M-%S-00-%Z", timeInfo);
      fprintf(stdout, "%s</TimeStamp>\n", timeStr);
      fprintf(stdout, "<User>%s</User>\n", getenv("USER"));

      fprintf(stdout, "<HostName>");
      if (getenv("HOSTNAME") == NULL)
        {
        if (getenv("HOST") == NULL)
          {
          fprintf(stdout, "(unknown)");
          }
        else
          {
          fprintf(stdout, "%s", getenv("HOST"));
          }
        }
      else
        {
        fprintf(stdout, "%s", getenv("HOSTNAME"));
        }
      fprintf(stdout, "</HostName><Platform version=\"");
    #if defined(sun) || defined(__sun)
    #if defined(__SunOS_5_7)
      fprintf(stdout, "2.7");
    #endif
    #if defined(__SunOS_5_8)
      fprintf(stdout, "8");
    #endif
    #endif
    #if defined(linux) || defined(__linux)
      fprintf(stdout, "unknown");
    #endif

      fprintf(stdout, "\">");
      // now the platform name
    #if defined(linux) || defined(__linux)
      fprintf(stdout, "Linux");
    #endif
    #if defined(macintosh) || defined(Macintosh)
      fprintf(stdout, "MAC OS 9");
    #endif
    #ifdef __MACOSX__
      fprintf(stdout, "MAC OS X");
    #endif
    #if defined(sun) || defined(__sun)
    # if defined(__SVR4) || defined(__svr4__)
      fprintf(stdout, "Solaris");
    # else
      fprintf(stdout, "SunOS");
    # endif
    #endif
    #if defined(_WIN32) || defined(__WIN32__)
      fprintf(stdout, "Windows");
    #endif
      fprintf(stdout, "</Platform>\n");

      if (getenv("MACHTYPE") != NULL)
        {
        fprintf(stdout, "<Architecture>%s</Architecture>\n", getenv("MACHTYPE"));
        }

      fprintf(stdout, "<Compiler version=\"");
      #if defined(__GNUC__)
    #if defined(__GNU_PATCHLEVEL__)
      fprintf(stdout, "%d", (__GNUC__ * 10000 + __GNUC_MINOR__ * 100 + __GNUC_PATCHLEVEL__));
    #else
      fprintf(stdout, "%d", (__GNUC__ * 10000 + __GNUC_MINOR__ * 100));
    #endif
    #endif
    #if defined(_MSC_VER)
      fprintf(stdout, "%d", (_MSC_VER));
    #endif
      // now the compiler name
      fprintf(stdout, ">");
    #if defined(__GNUC__)
      fprintf(stdout, "GCC");
    #else
    #if defined(_MSC_VER)
      fprintf(stdout, "MSC");
    #else
      fprintf(stdout, "UKNOWN");
    #endif
    #endif
      fprintf(stdout, "</Compiler>\n");

      fprintf(stdout, "<Library version=\"%s\">VTK</Librarary><Library version=\"unknown\">ITK</Library><Library version=\"%s\">KWWidgets</Library>\n", VTK_VERSION, KWWidgets_VERSION);
      int major, minor, patchLevel;
      Tcl_GetVersion(&major, &minor, &patchLevel, NULL);
      fprintf(stdout, "<Library version=\"%d.%d.%d\">TCL</Library>\n", major, minor, patchLevel);
      //fprintf(stdout, "<Library version=\"%d.%d.%d\">TK</Library>\n", major,
      //minor, patchLevel);
    #ifdef USE_PYTHON
      fprintf(stdout, "<Library version=\"%s\">Python</Library>\n", PY_VERSION);
    #endif
      fprintf(stdout, "<Repository>$HeadURL: http://www.na-mic.org/svn/Slicer3/trunk/Applications/GUI/Slicer3.cxx $</Repository>\n");
      fprintf(stdout, "<ProcessStep>\n");
      fprintf(stdout, "\n");
    }

For the version, you can use the CVS keyword Name, inserted into your source code file as $Name: not supported by cvs2svn $ and it will be replaced with a the version tag when you check out the code with a tag:

    cvs -r tag co [modulename]</programlisting>

If you are using SVN, in order to use CVS style keywords and have them be replaced with the appropriate values, you must enable the SVN repository's properties on a file by file (and keyword by keyword) basis by issuing the following command:

    svn propset svn:keywords "Revision Id Date HeadURL" [filename.ext]

## Reference

provenance_t