Comments (6)
To make a CWLProv RO Bundle for each conformance test, do the following
python3 -m venv cwlprov_env
. cwlprov_env/bin/activate
pip install cwltool cwltest
git clone https://github.com/common-workflow-language/cwl-v1.2/
cd cwl-v1.2
mkdir cwlprovs
cd cwlprovs
cwltest --tool cwltool --test ../conformance_tests.yaml --junit-verbose --junit-xml cwlprov_cwl_v1.2_full_results.xml --test-arg short_name==--provenance --verbose
The CWLProv RO Bundles will be in that cwlprovs directory, each directory named using the id
field from https://github.com/common-workflow-language/cwl-v1.2/blob/1.2.1_proposed/conformance_tests.yaml
cwlprov_cwl_v1.2_full_results.xml
will contain a complete log of all the runs
Initial results:
354 tests passed, 10 failures, 0 unsupported features
from workflow-run-crate.
Summary of failures:
legal_symlink
; "symlink to file inside of working directory should be retrieved"modify_directory_content
; missing the side-effect from usingInplaceUpdateRequirement
listing_default_none
Load listing issues. I think we force "deep" Directory listings to capture more provenancelisting_requirement_none
Load listing issues. I think we force "deep" Directory listings to capture more provenancelisting_loadListing_none
Load listing issues. I think we force "deep" Directory listings to capture more provenancelisting_requirement_shallow
Load listing issues. I think we force "deep" Directory listings to capture more provenancelisting_loadListing_shallow
Load listing issues. I think we force "deep" Directory listings to capture more provenancemixed_version_v10_wf
"test v1.0 workflow document that runs other version"mixed_version_v11_wf
another mixed version failureinvalid_syntax_mixed_v12_workflow
(should have failed, but didn't)
from workflow-run-crate.
The CWLProv RO Bundles will be in that cwlprovs directory, each directory named using the
short_name
field from https://github.com/common-workflow-language/cwl-v1.2/blob/1.2.1_proposed/conformance_tests.yaml
I don't see a short_name
field in that file, what am I missing? BTW, the direct link is not working, I had to open the repository page and click on the file (GitHub problem?). After running the above command I got 16 RO bundles, is that expected? Here's the list:
dotproduct-dotproduct-scatter
dotproduct-simple-scatter
flat-crossproduct-flat-crossproduct-scatter
flat-crossproduct-simple-scatter
js-input-record
multiple-input-feature-requirement
nested-crossproduct-nested-crossproduct-scatter
nested-crossproduct-simple-scatter
output_reference_workflow_input
record-order-with-input-bindings
schemadef_types_with_import
simple-dotproduct-scatter
simple-flat-crossproduct-scatter
simple-nested-crossproduct-scatter
simple-simple-scatter
stdout_chained_commands
Regarding the tests, I got only 3 failures, but I don't know which ones they were. cwlprov_cwl_v1.2_full_results.xml
entries are like:
<testcase name="Test ExpressionTool with Javascript engine" time="4.781120" class="inline_javascript, expression_tool" url="cwltest:conformance_tests#22">
<system-out>{
"output": 42
}
</system-out>
</testcase>
I can't see any error dumps or test outcome fields. The console dump does not seem to help much: there are more than 3 error messages there, so I guess some of them are expected or not relevant to test outcomes.
from workflow-run-crate.
I don't see a
short_name
field in that file, what am I missing?
Sorry, that field is called id
in that file
BTW, the direct link is not working, I had to open the repository page and click on the file (GitHub problem?).
That is odd, and I can also confirm!
After running the above command I got 16 RO bundles, is that expected? Here's the list:
Huh, I got 365 RO bundles: ro_test.zip
I can't see any error dumps or test outcome fields.
Yeah, JUnit XML format is weird. Only <testcases>
with <failure type="failure">
inside are failures; the other <testcases>
were successful and can be ignored.
from workflow-run-crate.
I tried again with id==--provenance
instead of short_name==--provenance
and I got 358 RO bundles. The tool tried to save them using full URIs as if they were local paths, so I got everything under a file:<ABSOLUTE_CWD_PATH>
directory. Most ROs have a dir name like conformance_tests.yaml#writable_stagedfiles
, others are under tests/<SUBDIR>/
, e.g. tests/conditionals/test-index.yaml#conditionals_non_boolean_fail
.
from workflow-run-crate.
At the moment the Ro Crate
archive reports a CWL fake folder (e.g. a Directory dictionary generated with an ExpressionTool) as a common Dataset
, together with a File
or Dataset
entry for each element into its deep_listing
hierarchy. CWLProv
used a Proxy
entity to represent this kind of things. But I am not aware of anything like that in Run Crate
As discussed on Slack, reproducing this behaviour 1-to-1 is probably out of the scope of the Run-Crate
provenance, but I leave a reminder here for potential future discussions.
from workflow-run-crate.
Related Issues (20)
- CQ2 - Resource usage HOT 25
- CQ3 - Configuration files HOT 5
- CQ4 - environment/container file HOT 16
- CQ5 - Step running time HOT 2
- CQ6 - Workflow running time HOT 4
- CQ7 - Outcome HOT 5
- CQ8 - Workflow inputs and outputs HOT 2
- CQ9 - Software version
- CQ10 - Tool wrappers HOT 6
- cwlprov_to_crate: support for nested workflows HOT 1
- CQ11 - Parameter connections HOT 4
- Resource requirements HOT 3
- Representing conditional execution HOT 7
- Representing secondary files HOT 2
- Representing environment variables HOT 5
- Content negotiation is not enabled for the profiles HOT 11
- How to represent which compute provider was used? HOT 6
- Remove individual roterms from profile crates
- Citing the profiles HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from workflow-run-crate.