qiime2 / q2cwl Goto Github PK

View Code? Open in Web Editor NEW

8.0 9.0 7.0 310 KB

Prototype interface for automatically generating CWL tools from QIIME 2 actions

License: BSD 3-Clause "New" or "Revised" License

Python 100.00%

hacktoberfest

q2cwl's Introduction

q2cwl

Interface for automatically generating CWL tools from QIIME 2 actions

How it works

QIIME 2's SDK provides information about all registered plugins and actions. Using this information, q2cwl is able to template out CWL tools definitions for use in CWL workflows.

There are currently two ways to template a tool definition, via conda or docker.

Conda Templates

Create a conda environment that contains the QIIME 2 plugins which you would like to template into CWL tools. This environment must include q2cwl in order to work:
```
pip install cwlref-runner
pip install .
```
Identify a directory where the tools should be installed (output-dir/ in this example).
Run the following command within your conda environment:
```
q2cwl template conda output-dir/
```
(You can also run q2cwl template conda --plugin <plugin name> output-dir/ to template only a single plugin)
Test your install with cwl-runner

Docker Templates

Create a Dockerfile that inherits from qiime2/q2cwl-core. This base image will provide miniconda, the QIIME 2 framework, and q2cwl. This step and step 2 can be skipped if an image already exists.
Build (and optionally upload) that docker image for use.
Identify a directory where the tools should be installed (output-dir/ in this example).
Run one of the following commands: If you have a docker image uploaded to docker-hub:
```
docker run -t <YOUR DOCKER IMAGE> -v $PWD/output-dir/:/tmp/cwl-tools/ q2cwl template docker <YOUR DOCKER IMAGE>
```
Otherwise:
```
docker run -t <YOUR DOCKER IMAGE> -v $PWD/output-dir/:/tmp/cwl-tools/ q2cwl template docker --local <YOUR DOCKER IMAGE>
```
(<YOUR DOCKER IMAGE> must be provided twice, once for docker run, and once again at the end so that q2cwl knows to template the image ID into the CWL tool.)
Test your install with cwl-runner

(You may use qiime2/q2cwl-core if you do not have a Dockerfile you would like to use, and do not want to install any extra plugins.)

Examples

Examples are available in examples/. There is a Makefile which will use cwl-runner.

This assumes you have a conda environment with QIIME 2 activated and have run:

pip install .

to install q2cwl.

Run the following to test all examples:

cd examples/
make test

q2cwl's People

Contributors

Stargazers

Watchers

Forkers

ebolyen chriskeefe andrewsanchez oddant1 lizgehret genostack amandabirmingham

q2cwl's Issues

Handle input/output files more idiomatically.

Right now we're just using File for everything, however it would be good to include more information like format. The trouble with that is we don't have any ontological mappings for QIIME 2 types yet.

x-ref: edamontology/edamontology#365

Something that seems super applicable from the spec:

Reasoning about format compatability must be done by checking that an input file format is the same, owl:equivalentClass or rdfs:subClassOf the format required by the input parameter.
owl:equivalentClass is transitive with rdfs:subClassOf, e.g. if <B> owl:equivalentClass <C> and <B> owl:subclassOf <A> then infer <C> owl:subclassOf <A>.

The subclassOf would be an ideal mechanism for describing inputs, however it isn't clear to me how a given input file would be identified as a member of a format? Is this the job of the CWL runner to determine? QIIME 2 artifacts are very easy to introspect so assuming an ontology in RDF/OWL format did exist, how would we perform identification of some arbitrary file?

Relatedly, while composing q2cwl tools should work pretty well within .qza/.qzv files (which is what our end-users will expect) we also need to support exporting/importing (via more generated tools) for composition with other cwl tools. Ideally formats will exist to make verifying workflows possible/discoverable.

I think ultimately what this might all look like is q2cwl generated tools for some plugin will accept an EDAM data type (with subclass-ing mechanisms to describe the semantic subtyping) and q2cwl import/export tools will convert that data type to an EDAM format which will hopefully play well with everything else that is similarly well-described.

cc @mr-c @matuskalas @johnbradley

write out CWL file in a more logical order

Here's how we do it for the CWL upgrader:

https://github.com/common-workflow-language/cwl-upgrader/blob/e06bf4e45cdcf1f2d3bdf1b95ba8bf3476c8cbac/cwlupgrader/main.py#L209

PrecisionFDA Tutorial

Improvement Description
Provide support for users interested in using QIIME 2 through the PrecisionFDA platform. This could take the form of documentation, a tutorial, or some other form of support.

Current Behavior
NA

Proposed Behavior
Offer some kind of handrail to users who want to use QIIME 2 as an app on the precisionFDA platform. PrecisionFDA supports CWL-based importation of applications/app components, so q2cwl and this proposed documentation might allow users to import specific actions rather than building apps from tarballs.

References
forum xref

q2cwl template conda fails on qiime2.plugins.sample_classifier.visualizers.confusion_matrix

Bug Description
q2cwl template conda fails to generate cwl for qiime2.plugins.sample_classifier.visualizers.confusion_matrix

Steps to reproduce the behavior

  535  wget https://data.qiime2.org/distro/core/qiime2-2019.10-py36-osx-conda.yml
  536  conda env create -n qiime2-2019.10 --file qiime2-2019.10-py36-osx-conda.yml
  537  conda activate qiime2-2019.10
  538  pip install cwltool
  539  pip install .
  540  mkdir output
  541  q2cwl template conda output/

Expected behavior
Expected all plugins to produce CWL tools

Screenshots

$ q2cwl template conda output/
output/qiime2.plugins.vsearch.methods.cluster_features_de_novo.cwl : created successfully.
output/qiime2.plugins.vsearch.methods.cluster_features_closed_reference.cwl : created successfully.
output/qiime2.plugins.vsearch.methods.dereplicate_sequences.cwl : created successfully.
output/qiime2.plugins.vsearch.methods.join_pairs.cwl : created successfully.
output/qiime2.plugins.vsearch.methods.uchime_ref.cwl : created successfully.
output/qiime2.plugins.vsearch.methods.uchime_denovo.cwl : created successfully.
output/qiime2.plugins.vsearch.pipelines.cluster_features_open_reference.cwl : created successfully.
output/qiime2.plugins.taxa.methods.collapse.cwl : created successfully.
output/qiime2.plugins.taxa.methods.filter_table.cwl : created successfully.
output/qiime2.plugins.taxa.methods.filter_seqs.cwl : created successfully.
output/qiime2.plugins.taxa.visualizers.barplot.cwl : created successfully.
output/qiime2.plugins.sample_classifier.methods.regress_samples_ncv.cwl : created successfully.
output/qiime2.plugins.sample_classifier.methods.classify_samples_ncv.cwl : created successfully.
output/qiime2.plugins.sample_classifier.methods.fit_classifier.cwl : created successfully.
output/qiime2.plugins.sample_classifier.methods.fit_regressor.cwl : created successfully.
output/qiime2.plugins.sample_classifier.methods.predict_classification.cwl : created successfully.
output/qiime2.plugins.sample_classifier.methods.predict_regression.cwl : created successfully.
output/qiime2.plugins.sample_classifier.methods.split_table.cwl : created successfully.
output/qiime2.plugins.sample_classifier.visualizers.scatterplot.cwl : created successfully.
Traceback (most recent call last):
  File "/Users/dcl9/opt/anaconda3/envs/qiime2-2019.10/lib/python3.6/site-packages/q2cwl/template.py", line 250, in template
    tool = make_tool(plugin, action, directory, extra_req_factory)
  File "/Users/dcl9/opt/anaconda3/envs/qiime2-2019.10/lib/python3.6/site-packages/q2cwl/template.py", line 137, in make_tool
    tool['inputs'].update(template_parameters(name, spec))
  File "/Users/dcl9/opt/anaconda3/envs/qiime2-2019.10/lib/python3.6/site-packages/q2cwl/template.py", line 210, in template_parameters
    raise Exception("Unknown type: %r" % qiime_type)
Exception: Unknown type: Float | Str % Choices('auto')

<visualizer qiime2.plugins.sample_classifier.visualizers.confusion_matrix> : Unknown type: Float | Str % Choices('auto')

See above for error details.

Computation Environment

OS: macOS Mojave 10.14.6
QIIME 2 Release 2019.10

Set up a docker container for q2cwl

References
Should be able to adapt the work in https://github.com/qiime2/vm-playbooks for this.

Use enum instead of string for `Str % Choices({...})` type

From best practices guide:
Use type: enum instead of type: string for elements with a fixed list of valid values.