CQ4 - environment/container file about workflow-run-crate HOT 16 OPEN

Yes but container images are not the only remote environment supported by WMSs. For example, StreamFlow can offload each step of a workflow to a different environment (a Cloud VM, a bare metal node, an HPC Queue manager, a local Docker, etc.). Plus, with the following release it will be possible to stack the things (e.g., a SLURM Queue Manager over an SSH-connected node, a Singularity container over a Queue Manager over SSH, etc.)

I fear that the pure schema.org ontology cannot represent these things effectively enough. I think that if we want to capture these scenarios we have to move to external ontologies. One example is the very recent GAIA-X Ontology. However, it is more provider-oriented than consumer-oriented. Indeed, the link between the resource and a software consumer seems to be missing.

from workflow-run-crate.

This could be a resolved Conda environment (conda export), a resolved Docker image #9 or others. Need sub-types per environment.

Renske have started modelling this for CWLprov.

from workflow-run-crate.

One complication here is how many stack levels to represent. E.g. a Conda environment running in a Docker container running on Kubernetes running on OpenStack running on an HPC cluster.

from workflow-run-crate.

In https://github.com/osoc-es/c2t#ontology-diagram we addressed the basic representation of packages in a container (Docker). But this may be out of the scope of what we intend to do here.

I suggest allowing pointing out to the file that creates the container/environment, or pointing to the id and registry where the container is stored in.

from workflow-run-crate.

Also, the command used to invoke the container/creating the environment is quite useful

from workflow-run-crate.

It's a bit confusing because we have #9 specifically for Docker images. I guess this is more general, maybe it should be split into multiple ones depending on the level of abstraction (minus Docker, since we already have #9).

from workflow-run-crate.

Maybe #9 and this issue should be merged. It is necessary to reference container images; the discussion in #9 has so far been dedicated to this. CQ1/#9 is concerned with compiling a list of all images use by the run. That list can be compiled by collecting the images from all workflow execution steps as described by the spec part which will come out of this issue.

from workflow-run-crate.

Ok. I think we're mixing two orthogonal things though. On the one hand we have the environment in which the process is executed, while on the other we have the method through which the compute infrastructure is accessed to get resources and instantiate the environment.

The first is the essential part. For that the container image, or conda environment, or indeed VM image seems sufficient. Instead, for the latter to me it seems debatable whether it is even interesting enough to be captured in most cases.

Taking containers as an example, my point is that generally I would not care much whether you executed a container over SSH, over batch queue or over k8s; I still only need the container image and the command to reproduce what you did.

from workflow-run-crate.