https://github.com/common-workflow-language/cwl-v1.2/blob/1.2.1_proposed/CONFORMANCE_TESTS.md

@mr-c will test creation of the CWLProv RO Bundles (fixing any issues found), and then he will help @simleo create his own for testing with cwlprov_to_crate

How long does this workflow take to run?

The actual duration of the represented workflow run can be obtained from endTime - startTime on the CreateAction. Providing an estimate of the typical running time, on the other hand, is a different thing. Can we use totalTime for that? Or a more specific custom property like estimatedRunningTime? And do workflow languages have annotation fields for this?

Workflows can run other workflows as subworkflows. CWLProv outputs separate provenance documents in this case, but such runs are not yet supported in cwlprov_to_crate. Functionally, we need to add the capability to parse the provenance metadata in this scenario. Then there's the issue of adding subworkflow metadata to the RO-Crate. In the relationship graph, subworkflows need to appear in the same place as tool wrappers (what's run by a step). Their type should be the same as the main workflow, minus File since they are stored as sections in packed.cwl:

["SoftwareSourceCode", "ComputationalWorkflow", "HowTo"]

Then we'd need to recursively convert all subworkflows as we did for the main one.

One possibly weird consequence is that some of the workflow components would be SoftwareApplications (the tool wrappers) while others would be of type SoftwareSourceCode (the subworkflows). I guess the reason for the presence of both entities in Schema.org is that the former should model an executable, while the latter should represent code that needs to be compiled. With interpreted languages such as CWL (or Python, etc.), however, the source code is also runnable, so the distinction is not so meaningful.

Was the execution successful?

Can use actionStatus to FailedActionStatus or CompletedActionStatus - can also provide error.

Add a comment to this issue to join the Workflow Run RO-Crate working group. Please indicate your ORCID, if you have one.

We coordinate using the channel #ro-crate on seek4science.slack.com (join) and the RO-Crate mailing list.

Workflow Run RO-Crate is an RO-Crate profile, i.e., a specialization of RO-Crate to a set of use cases. You can join the RO-Crate community at large here.

So we know which requirement/CQ maps to each profile, in order to determine whether everything is covered.

What is the environment/container file used in a specific workflow execution step?

Similar to the configuration file (#11) problem. Need env dump support from workflow engine.

Both in the prospective provenance (what's the name of the variable that a workflow or tool needs?) and the retrospective one (what was the value).

Also, how to hide the value if it's sensitive.

Brought up by @jmfernandez

What is the source code version of the component executed in a workflow step? Is it a script? and executable?

We can use softwareVersion, though getting the version of the actual tool (e.g., grep) that was called by the wrapper might not be easy (related to container image).

What is the script used to wrap up a software component?

We're mapping tool wrappers (e.g., foo.cwl) to SoftwareApplication. Wrappers at lower levels can also be SoftwareApplication, but we need to draw the line somewhere (related to container image).

What are the inputs and outputs of the overall workflow (I don't care about the intermediate results)

We're using object and result on the workflow run action.

What container images (e.g., Docker) were used by the run?

• Source entity: CreateAction
• Target entity? It could be File if the image is a tarball from docker save
• Property? Overload image?

How much memory/cpu/disk was used in run?

• Source entity?
• Property?

What are the configuration files used in a workflow execution step?

ChooseAction? Though maybe the crate generator should just merge the params with the other ones if it can parse the config file. To link to the config file as a black box instead we probably need a new property.

How should we model workflows where one or more steps are executed only if a condition is verified (e.g., when clause in CWL)?

If available, record the application's resource requirements. This is useful to those who want to reproduce the run. In CWL, this is provided via ResourceRequirement. We could use properties like memoryRequirements or storageRequirements for this.

How long does this workflow component take to run? (estimate)

Use totalTime? Allowed on HowTo and HowToDirection but not on HowToStep. Can also get actual duration from endTime - startTime on the action.

Knowing how workflow parameters were passed to individual tools is important to find out how they affected the outputs.

We are currently linking workflow and tool parameters with connectedTo from the source tool / workflow to the target tool / workflow. For instance, in revsort:

we currently have:

{
    "@id": "packed.cwl#revtool.cwl",
    "@type": "SoftwareApplication",
    "input": [
        {"@id": "packed.cwl#revtool.cwl/input"}
    ],
    "output": [
        {"@id": "packed.cwl#revtool.cwl/output"}
    ]
},
{
    "@id": "packed.cwl#sorttool.cwl",
    "@type": "SoftwareApplication",
    "input": [
        {"@id": "packed.cwl#sorttool.cwl/reverse"},
        {"@id": "packed.cwl#sorttool.cwl/input"}
    ],
    "output": [
        {"@id": "packed.cwl#sorttool.cwl/output"}
    ]
},
{
    "@id": "packed.cwl#revtool.cwl/output",
    "@type": "FormalParameter",
    "connectedTo": {"@id": "packed.cwl#sorttool.cwl/input"}
}

but that's inaccurate, since such links only exist within the revsort workflow. packed.cwl#revtool.cwl and packed.cwl#sorttool.cwl represent standalone software tools that happen to be connected this way in revsort, but might be used differently in another workflow.

How to link to secondary files, e.g. CWL's secondaryFile