at https://github.com/broadinstitute/cromwell-tools/blame/master/setup.py#L19
seems a comma is missing? @rexwangcc @samanehsan

• Testing could be significantly simplified by parametrization across, e.g. secrets files vs username + password + url.
• unittest.TestCase objects are not compatible with pytest parametrization.
• This change would require replacing setUp and tearDown with their pytest cognates

We (mint team) should make a pull request to add our version of the cost script (currently in skylab) to cromwell-tools.

It currently lives here:
https://github.com/HumanCellAtlas/skylab/blob/master/smartseq2_single_sample/run_multi_samples/scripts/calculate_cost_workflow.py

Kylee is moving tickets from mint backlog into cromwell-tools issues

A lot of code in cromwell-tools triggers our default patterns.

At this moment, there are only unit test cases for this package. As a user, I'd be more confident to use it against my Cromwell if it has been tested against multiple versions of Cromwell, (e.g. v34-v41). It would be great to provide a list of "fully supported" versions of Cromwell on the main page.

After #13, write automated tests for this package.

• Python 2.7 compatibility prevents the use of some helpful tools (e.g. shutil.which for path checking)
• Python 2.7 is the past

Can we drop support for 2.7 in cromwell-tools?

Right now each time you run the cromwell-tools command, it's required to pass in the auth params, such as --username, --password and --service-account-key, it has been a burden to users, since it is too much to type when you just wanna abort a workflow but you have to:

cromwell-tools abort \
--url xxx \
--username xxx \
---password xxx \
UUID

Some other CLI tools, such as aws, gsutilor kubectl are supporting caching the credentials under ~/.aws or ~/.config, so that you could leverage xxx auth select or xxx config use-credentials to switch between stored credentials/roles. It would somehow make the tool less secure, but incredibly help improve the user-friendliness. This issue serves as a discussion area to talk if we should add that feature in cromwell-tools, which will cover some commands like:

• cromwell-tools auth add
• cromwell-tools auth select
• cromwell-tools auth remove
• cromwell-tools auth list

The OAuth token times out when using the wait command to monitor a long-running workflow. This possibly because the token is not refreshed during the wait polling, which should exist.

Right now it's a pain to type out all of the arguments and commands, it will be much better to have Bash tab completion for cromwell-tools.

We wrote a manager tool (https://github.com/ambrosejcarr/cromwell-manager/) for cromwell to help launch and manage workflows, which aids comp-bio development of pipelines. We can merge this into cromwell-tools.

Can this be renamed to not use the word manager because the job manager also uses that word and it could get confusing?

AC:

• PR of cromwell manager against cromwell-tools

This tool has only been fully tested with JES(Google Pipelines API based) Cromwell instances. It would great to add tests for AWS Batch based Cromwell instances.

• e.g. failure when URL is not passed to start_workflow is not clear.

It came up that the starter currently expects that the wdl you are trying to start is not a single wdl but has imports/subworkflows. @dshiga said it doesn't work for a single wdl file wdl. We should update this so it works for more people.

Use either shutil.which (3.3) or distutils.spawn.find_executable (2.7).

Depends on #26

Github Actions runs much faster than Travis CI nowadays and provides more granular Github events monitoring than latter, it seems to be a good idea to migrate before setting up more tests for cromwell-tools.

When I try to run cromwell-tools without specifying a command I get the following error:

🐋  ~ 1.8 cromwell-tools
Traceback (most recent call last):
  File "/usr/local/bin/cromwell-tools", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/site-packages/cromwell_tools/cli.py", line 258, in main
    command, args = parser(arguments)
  File "/usr/local/lib/python3.7/site-packages/cromwell_tools/cli.py", line 250, in parser
    command = getattr(CromwellAPI, args['command'])
TypeError: getattr(): attribute name must be string

I would expect you'd get usage info instead of a crash.

As a developer, I want to automate the process of publishing this package on PYPI, basically I wish to publish the package by just cutting off a new release on Github.

It's useful for those who don't have Python or have difficulties setting up the right Python evn on the machines to use cromwell-tools. Note: it's still slightly different from the womtool JAR.

Cromwell-on-GCP exposes a runtime option to inject a monitoring script into every task in a workflow, but (as far as I can tell) doesn't expose a way to collect the generated log paths via its API. For users who want to make use of monitoring, I think adding a new collect-monitoring-logs command to cromwell-tools would be useful. The command would:

1. Take a workflow ID,
2. Scrape the corresponding Cromwell execution directory for files named monitoring.log,
3. Output a dict of task ID -> raw monitoring log output.

Users could then bolt their own interpreters onto this command to parse their specific log outputs.

@benjamincarlin has been working on code to do this, and we thought it'd be good to contribute back here instead of setting up yet-another-repo. Would a PR be welcome?

As a user, I'd like to get notified from the command line if there are new versions of cromwell-tools available, so I can update.

Hi, would it be possible to make authentication optional? We run cromwell in a walled environment and would prefer not to deal with authentication. Currently if no form of authentication is passed to start_workflow this exception is raised from `harmonize_credentials:

 File "envs/pipeflow/src/cromwell-tools/cromwell_tools/cromwell_tools.py", line 184, in start_workflow
    auth, headers = _get_auth_credentials(cromwell_user=user, cromwell_password=password, caas_key=caas_key)
  File "envs/pipeflow/src/cromwell-tools/cromwell_tools/cromwell_tools.py", line 59, in _get_auth_credentials
    cromwell_user, cromwell_password = harmonize_credentials(secrets_file, cromwell_user, cromwell_password)
  File "envs/pipeflow/src/cromwell-tools/cromwell_tools/cromwell_tools.py", line 43, in harmonize_credentials
    raise ValueError('One form of cromwell authentication must be provided, please pass '
ValueError: One form of cromwell authentication must be provided, please pass either cromwell_user and cromwell_password or a secrets_file.

Ambrose made a package in https://github.com/ambrosejcarr/cromwell-manager/
with some example notebooks here, including the ones Ambrose demoed: https://github.com/ambrosejcarr/cromwell-manager/tree/master/src/examples

He has said there is some work to be done to make this package hardened and tested enough to be in a broad official repo.

AC:
Harden with code review help

Problem:
I'm trying to use cromwell-tools in a project that is built to support python3.5 (currently the minimum python3 version available on the hosts I'm targeting). When trying to execute there are syntax errors within the cromwell-tools distribution due to the use of f strings, which are a python3.6 feature.

Looking at closed issue tickets, as well as the PR for refactoring to remove support for python2, it looks like python3.5 was mentioned as a minimum supported version. Links to these items here:
#26
#43

Proposed Solutions:

1. Update code to support python3.5 (and possibly other earlier versions) by removing the use of f strings.
2. Update documentation to specify that python3.6 is the minimum supported version as opposed to simply python3. Additionally, include python_requires>=3.6 in the setup call in setup.py

According to https://google-auth.readthedocs.io/en/latest/oauth2client-deprecation.html, the oauth2client lib that we are using in cromwell-tools has been deprecated in favor of google-oauth, for both security and usability concerns, we need to update cromwell-tools with the new dependency.

If using a service account key credentials file, the CromwellAPI.submit function ignores the existing jes_gcs_root attribute in the user-supplied options file. When compose_oauth_options_for_jes_backend_cromwell is called here, no execution_bucket is passed in to override the default of None.

jes_gcs_root then automatically defaults to the gs://%s-cromwell-execution/caas-cromwell-executions bucket each time (as long as you're using a service account key):

It would be great if cromwell-tools can provide a toolkit focusing on diagnostic/analysis toolkit for workflows development, some use cases are:

• I want to know the 5 statistics (avg, max, median...) and the average workflow runtime, given a workflow identifier or during a time window, so that I can have a better understanding of my workflow performance.
• I want to know the 5 statistics (avg, max, median...) and the average of individual tasks and sub-workflows of a workflow, so that I can easily find the bottleneck of my workflow.
• I want to get some serialized/well-parsed formats of the workflow failures in order to debug my workflows.
• I want to have some sort of methods to predict the runtime, resource requirements of my workflow(s), such as Linear Regression, SVM, NNs, etc. based on my historical workflow data, in order to optimize and productionize my workflows.

...

When handling requests.exceptions.HTTPError raised by cromwell_tools.api, it is useful to check the status code of the HTTPError. While this is included in the exception as part of a string, access would be easier if the response object was included on the HTTPError itself as opposed to trying to parse the status code from the string.

The HTTPError inherits from RequestException which accepts a 'response' keyword argument. See source: https://github.com/psf/requests/blob/master/requests/exceptions.py

Incoming pull request with suggested change.

As a user, I want to label the workflow I submit through the CLI of cromwell-tools, e.g. cromwell-tools submit xxxxx -m "my-labels". At the moment, this can only be done by passing in a JSON file that contains the labels e.g. -l labels.json.

The Read the Docs has a link to a quickstart jupyter notebook tutorial and the link is currently broken in both stable and latest.

Validate requires a copy of womtool. To do this, we need to get our docker image into the cloud. @rexwangcc Are you up for this?

On Green (and soon on Monster), CI for WDL development follows the pattern:

1. Run a WDL against some collection of input JSONs
2. Wait for all the runs to complete
3. Feed the outputs of each run into a "validation WDL", along with a corresponding set of "truth" inputs
4. Wait for all the validation runs to complete

The test runner has typically had special knowledge about each pipeline, but it feels like it should be possible to generate the core functionality.

I'm not sure if cromwell-tools would be the best place for that functionality to live (maybe there could be a new cromwell-test-tools package?), but here's an idea for what the command might look like:

$> cromwell-tools validate -h
usage: cromwell-tools validate [-h] [-c CROMWELL_URL] [-u USERNAME] [-p PASSWORD]
                               [-s SECRETS_FILE]
                               --wdl-file WDL_FILE
                               --validation-wdl-file WDL_FILE
                               [--dependencies-json DEPENDENCIES_JSON]
                               --inputs-batch INPUTS_JSON1:TRUTH_OUTPUTS1
                               [--inputs-batch INPUTS_JSON2:TRUTH_OUTPUTS2 ...]
                               --inputs-json INPUTS_JSON
                               [--inputs2-json INPUTS2_JSON]
                               [--options-file OPTIONS_FILE]

The command would do something like:

1. Parse the inputs-jsons out of each inputs-batch arg
2. Submit each input along with wdl-file, dependencies-json, inputs2-json, and options-file using the existing submit logic
3. Wait for each workflow to finish using the existing wait logic
4. Query the outputs of each workflow, getting something like:

    {
      "MyMainWorkflow.out1": "gs://....",
      "MyMainWorkflow.out2": 10
    }

5. Strip the main workflow's name from the outputs, and combine with the truth outputs to build an input JSON for the validation WDL:

   {
     "ValidateMyMainWorkflow.test": {
       "out1": "gs://....",
       "out2": 10
     },
     "ValidateMyMainWorkflow.truth": {
       "out1": "gs://....",
       "out2": 20
     }
   }

6. Run the validation WDL on each of these constructed inputs using the existing submit logic
7. Wait for each validation run to finish using the existing wait logic

Does this sound like logic that'd make sense to live in cromwell-tools? I'd be happy to put a PR together for it if so.

The current tests do not test the CLI, refactoring can break its functionality.

The stack trace is:

Traceback (most recent call last):
 File "/usr/local/bin/cromwell-tools", line 11, in <module>
   sys.exit(main())
 File "/usr/local/lib/python2.7/site-packages/cromwell_tools/cli.py", line 104, in main
   command, args = parser(arguments)
 File "/usr/local/lib/python2.7/site-packages/cromwell_tools/cli.py", line 95, in parser
   auth = CromwellAuth.harmonize_credentials(**args)
TypeError: harmonize_credentials() got an unexpected keyword argument 'inputs_file'

As far as I can tell from the code, I think this will be a problem for all commands that ask for auth, because all the args are getting fed into harmonize_credentials.

Sometimes when you just submit a workflow, the /status endpoint, which is consumed by the wait command, will return 404 instead of the status of the actual workflow, this will break the wait. We need to add retry logic around the wait command!

At this moment, the submit command supports local path and HTTP(s) path for WDLs and JSON files, it would be much more helpful if it could also support reading from gs:// paths!

Adding a contributors section like this would encourage more (external and internal) people to contribute to this tool.

There should also be a "How-to contribute" section along with this.

Using cromwell-tools 1.1.2 (python 3.7.0), I was receiving duplicate log messages after importing CromwellAuth module and calling from_no_authentication function. This seems to be happening because the warning function was called on the logging module (instead of an instance of Logger). See below on how to replicate the issue:

import logging

logger = logging.getLogger("my_module")
logger.setLevel(logging.DEBUG)

formatter = logging.Formatter(
        "[%(asctime)s %(name)s] [%(levelname)s] " +
        "[%(filename)s:%(lineno)s - %(funcName)s()] %(message)s"
)

stream = logging.StreamHandler()
stream.setFormatter(formatter)
stream.setLevel(logging.DEBUG)
logger.addHandler(stream)

logger.debug("test1")

from cromwell_tools.cromwell_auth import CromwellAuth
auth = CromwellAuth.from_no_authentication("http://localhost")

logger.debug("test2")

[2019-04-09 11:27:48,260 my_module] [DEBUG] [test-logger.py:16 - <module>()] test1
WARNING:root:You are not using any authentication with Cromwell. For security purposes, please consider adding authentication in front of your Cromwell instance!
[2019-04-09 11:27:48,547 my_module] [DEBUG] [test-logger.py:21 - <module>()] test2
DEBUG:my_module:test2

I get the error TypeError: harmonize_credentials() got an unexpected keyword argument 'caas_key' when I attempt to run cromwell-tools on Python 3.6:

$ cromwell-tools status --uuid 305cafa1-111c-4ad1-bbae-a4a63c33a018 --url localhost:8000
Traceback (most recent call last):
  File "/usr/local/bin/cromwell-tools", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/dist-packages/cromwell_tools/cli.py", line 108, in main
    command, args = parser(arguments)
  File "/usr/local/lib/python3.6/dist-packages/cromwell_tools/cli.py", line 96, in parser
    auth = CromwellAuth.harmonize_credentials(**auth_arg_dict)
TypeError: harmonize_credentials() got an unexpected keyword argument 'caas_key'