There's a whitespace in handling of dxpy.DXError which raises its own exception when there's an error in uploading a single file from upload_single_file()

We have encountered cases where there is a failed run that has somewhat of a proper run structure but the log file in question is empty. Can we elegantly error out in these cases rather than just throw an exception?

https://github.com/dnanexus-rnd/dx-streaming-upload/blob/8fec7ae047504b6b39996dd52e52dd684306e518/files/dx_sync_directory.py#L299-L300

Currently if we encounter a symlink in the run folder, then the tarfile will just upload it as a symlink. Should we consider dereferencing symlinks so that a user who has symlinks to data in another folder still has valid data uploaded? If so, one could add the dereference=True flag to the tarfile.open.

https://github.com/dnanexus-rnd/dx-streaming-upload/blob/8fec7ae047504b6b39996dd52e52dd684306e518/files/dx_sync_directory.py#L414

Email correspondence:

Can the config file expose parameters for the actual workflow that gets executed post-upload? Or is it only parameters for the uploader machinery that can go there? If you can specify workflow optional parameters, I'd love to have the illumina-demux --sequencing_center option customizable on a per-site basis.

In a few places bare variables are used in this role's playbook. When the playbook is run, Ansible warns that bare variables (ex. monitored_users) are deprecated, and should be replaced with jinja2-style variable syntax (ex. '{{monitored_users}}'). As of Ansible 2.1.0.0 the role playbooks still work, but they may not in the near future unless the bare variables are wrapped.

Currently, we upload the RunInfo.xml and sample sheet in this script. We then call the dx_sync_directory.py. If we encounter an error in that (say it throws an exception) then we will have not uploaded any files except these two. The next time the upload routine occurs, we'll upload these two files again, encounter the error again, an it will carry on ad infinitum. Should we at least consider cleaning up these two files if the call to dx_sync_directory.py fails?

https://github.com/dnanexus-rnd/dx-streaming-upload/blob/8fec7ae047504b6b39996dd52e52dd684306e518/files/incremental_upload.py#L359-L362

Looking over the upload logs (monitor.log) for our install of this role, we've seen connection dropouts (unable to reach api.dnanexus.com) and various other transient issues. It would be helpful to have timestamps in the log file, similar to standard *nix syslog files.

Related to this, the logged progress lines for an active upload seems to use \r characters rather than \n, making it necessary to view the log with replacement of the carriage return characters via something like cat /home/miseq/monitor.log | tr '\r' '\n'. Ultimately, it looks like ua is intended for interactive operation, where a carriage return (\r) would make sense for updating the same line with the latest transfer information, but for headless/logged operation, newlines (\n) should really be used regardless of log verbosity.

Is the fix a simple matter of passing --verbose as part of the cronjob command, or will that result in too much noise in the log file? Would it make sense to add another arg option ("--log-output"?) to ua for non-interactive operation? The arg could be used in the ternary on this line in ua/main.cpp so newline characters are printed.

In case our issues are not due to connection dropouts, here are lines from monitor.log that relate to failed uploads:

Upload throttling is disabled.

Unable to connect to API server. Run 'ua --env' to see the current configuration.

Detailed message (for advanced users only):
DXConnectionError: 'Was unable to make the request: POST 'https://api.dnanexus.com:443/system/greet' . Details: '
*******
Error in using curl_easy_perform.
Error code (CURLcode) = 6
Error Message: 'Could not resolve: api.dnanexus.com (Could not contact DNS servers)'
********
'.', Curl error code = '6'

When the playbook is run to set up a new instance of dx-streaming-upload it automatically creates a lock file in /var/lock/:

https://github.com/dnanexus-rnd/dx-streaming-upload/blob/a14c438db11c77b9c2e8ef5f693fa9b0fead338d/tasks/main.yml#L184-L186

Since this is a sym linked directory to /run/lock which is a tmp file system this gets cleared on every reboot, and as such dx-streaming-upload fails when this file is missing on the next hourly cron job starting. This then requires either manually creating the required lock files again, or restarting dx-streaming-upload by running with the playbook again.

This is not documented in the readme, adding the same touch command from the playbook to /etc/rc.d/rc.local works on RedHat for creating the required lock files on boot. It would be good for this to be documented or for dx-streaming-upload to recreate it's own lock files if they do not already exist.

It would be nice if the retry count in the config file, n_retries, were a template variable rather than a hardcoded value of '3'.

During the ansible set up the config in the playbook is copied to the user config file but the novaseq option is missing from here.

If monitor_runs.py is run via the cmd line and the novaseq key added manually to the config file then this is correctly added and CopyComplete.txt is picked up, but when running via Ansible this is not used and the run never completes uploading.

I'm not sure as to the logic of having a separate 'novaseq' option, could you not just check for either RTAComplete.txt or CopyComplete.txt in check_local_runs() of monitor_runs.py and not need to specify novaseq as an input? This seems simpler and would achieve the same result of handling novaseq runs.

Also this novaseq option is missing from the readme in the playbook options.

From email correspondences:

In monitor_runs.py, the (previously monolithic) class of "unsynced runs" will be sub-divided to 3 classes: The script will attempt to trigger incremental_upload on runs of class 1 then 2; and will not attempt to upload run folders of class 3.

1. Completed runs (has a RunInfo.xml and a RTAComplete.txt/xml file)
2. In-progress run (has a RunInfo.xml file, but not a RTAComplete.txt/xml file)
3. Stale run (has a RunInfo.xml file, which was created more than Y times expected duration of a sequencing run, both the sequencing runtime and Y will be user-specified, with tentative defaults of 3 and 24hrs respectively).

In incremental_upload.py, once an upload has been triggered for an in-progress run (class 2 from above), it will check whether the run goes stale, and timeout if the run is not completed by T times expected duration of run. It will still potentially block the upload process up to a maximum duration of Y times expected run duration, however.

Would there be an advantage for monitor_runs.py to pass --max-size (-M) to incremental_upload.py, based on a value in the monitor_runs.config file? As it is now, if a complete run directory is placed in a path watched by monitor_runs.py, a large tarball of the entire run is created. In such cases, will ua still upload chunks, or will it attempt to upload the entire large tarball?

In use cases like ours were connection instability is an issue, we'd prefer to upload many small chunks so dropouts do not interrupt large file transfers.

Looking over the upload logs (monitor.log) for our install of this role, we've seen connection dropouts (unable to reach api.dnanexus.com) and various other transient issues. It would be helpful to have timestamps in the log file, similar to standard *nix syslog files.

Related to this, the logged progress lines for an active upload seems to use \r characters rather than \n, making it necessary to view the log with replacement of the carriage return characters via something like cat /home/miseq/monitor.log | tr '\r' '\n'. Ultimately, it looks like ua is intended for interactive operation, where a carriage return (\r) would make sense for updating the same line with the latest transfer information, but for headless/logged operation, newlines (\n) should really be used regardless of log verbosity.

Is the fix a simple matter of passing --verbose as part of the cronjob command, or will that result in too much noise in the log file? Would it make sense to add another arg option ("--log-output"?) to ua for non-interactive operation? The arg could be used in the ternary on this line in ua/main.cpp so newline characters are printed.

Looking at the role more, this looks great! A couple initial thoughts:

Currently in the work-in-progress branch, most files are copied to ~/. Perhaps the install location for the uploader should not be relative to the (ansible) user's home directory. Placing the files in /opt as per the Linux FHS might be appropriate if there permissions allow read/execute by the cronjob user.

Additionally, it might be nice to have the user specified for the cron jobs be a role variable (defaulting to root? or ansible_user?) The variable ansible_ssh_user is deprecated in favor of ansible_user. Or maybe there should be an ability to specify multiple users, each with a monitored directory he or she has read access to. Something like this (with singular user as the default in defaults/main.yml):

monitored_users:
  - username: user-one 
    monitored_directories:
      - /path/to/a/run/directory/parent/dir
      - /some/other/directory
  - username: user-two
    monitored_directories:
      - /miseqtwo/run/storage

Having multiple users might complicate dx token usage though? Since the token persists in a user's home directory after logging in via dx login, and the cron user may or may not have logged in? Tokens could be passed in as part of the list of users, and use dx_token if one is not specified:

dx_token: "PROJECTTOKEN"
monitored_users:
  - username: user-one
    dx_user_token: "TOKENVALUE"
    monitored_directories:
      - /path/to/a/run/directory/parent/dir
      - /some/other/directory
  - username: user-two
    monitored_directories:
      - /miseqtwo/run/storage

Using the user list would require something like this:

# Logging into DNAnexus account
- name: Log in to DNAnexus account if token is provided
  command: source ~/dx-toolkit/environment && dx login --token {{ item.dx_user_token if 'dx_user_token' in item else dx_token }} --noprojects
  become:
  become_user: {{ item.username }}
  args:
    executable: /bin/bash
  with_items: "{{monitored_users}}"
  when: dx_token is defined

 #...

- name: set up CRON job to run every hour in deploy mode
  cron: >
    name="DNAnexus monitor runs (deploy)"
    special_time=hourly
    user="{{item.0.username}}"
    job="flock -n ~/dnanexus/cron.lock bash -ex -c 'source ~/dx-toolkit/environment; PATH=$PATH:~/dnanexus-upload-agent; python ~/dnanexus/scripts/monitor_runs.py -c ~/dnanexus/config/monitor_runs.config -p {{ upload_project }} -d {{item.1}} -v > ~/monitor.log 2>&1' > ~/dx-stream_cron.log 2>&1"
  with_subelements:
        - monitored_users
        - monitored_directories
  when: mode == "deploy"

Example debug output:

    - debug: msg="User {{item.0.username}} should monitor {{item.1}}"
      with_subelements:
        - monitored_users
        - monitored_directories

    - debug: msg="User {{ item.username }} has token {{ item.dx_user_token if 'dx_user_token' in item else dx_token }}"
      with_items: "{{monitored_users}}"

ok: [localhost] => (item=({u'username': u'user-one', u'dx_user_token': u'TOKENVALUE'}, u'/path/to/a/run/directory/parent/dir')) => {
    "item": [
        {
            "dx_user_token": "TOKENVALUE",
            "username": "user-one"
        },
        "/path/to/a/run/directory/parent/dir"
    ],
    "msg": "User user-one should monitor /path/to/a/run/directory/parent/dir"
}
ok: [localhost] => (item=({u'username': u'user-one', u'dx_user_token': u'TOKENVALUE'}, u'/some/other/directory')) => {
    "item": [
        {
            "dx_user_token": "TOKENVALUE",
            "username": "user-one"
        },
        "/some/other/directory"
    ],
    "msg": "User user-one should monitor /some/other/directory"
}
ok: [localhost] => (item=({u'username': u'user-two'}, u'/miseqtwo/run/storage')) => {
    "item": [
        {
            "username": "user-two"
        },
        "/miseqtwo/run/storage"
    ],
    "msg": "User user-two should monitor /miseqtwo/run/storage"
}

TASK [debug] *******************************************************************
ok: [localhost] => (item={u'username': u'user-one', u'monitored_directories': [u'/path/to/a/run/directory/parent/dir', u'/some/other/directory'], u'dx_user_token': u'TOKENVALUE'}) => {
    "item": {
        "dx_user_token": "TOKENVALUE",
        "monitored_directories": [
            "/path/to/a/run/directory/parent/dir",
            "/some/other/directory"
        ],
        "username": "user-one"
    },
    "msg": "User user-one has token TOKENVALUE"
}
ok: [localhost] => (item={u'username': u'user-two', u'monitored_directories': [u'/miseqtwo/run/storage']}) => {
    "item": {
        "monitored_directories": [
            "/miseqtwo/run/storage"
        ],
        "username": "user-two"
    },
    "msg": "User user-two has token PROJECTTOKEN"
}

Also, I'm not sure if this is something for the role, or for the DNAnexus project, but it might be nice to organize data on the DNAnexus side into folders based on the ansible_hostname, so we can quickly see which files have been uploaded from a given site. Something like:

  DNAnexus_project/
    node-3/
      runs/
      demux/
    {{some_other_hostname }}/
      runs/
      demux/

The general flow of this script is to monitor a folder for potential new Illumina runs, check to see if there is already a complete run uploaded to DNAnexus that matches those runs, and if not, schedule the runs to be uploaded.

Currently there is logic up front to get a list of potential run folders, and then once we have filtered down to the list of folders that need to be uploaded, we iterate through that list and upload.

During the process of identifying which folders need to be synced, we do some sanity checking. If we fail the sanity checks, the whole process terminates before even uploading any files.

Instead, we should check each folder independently, note folders that are problematic, but continue on with folders that are well formed.

Here are two such checks:
https://github.com/dnanexus-rnd/dx-streaming-upload/blob/8fec7ae047504b6b39996dd52e52dd684306e518/files/monitor_runs.py#L323
https://github.com/dnanexus-rnd/dx-streaming-upload/blob/8fec7ae047504b6b39996dd52e52dd684306e518/files/monitor_runs.py#L323

In incremental_upload.py :: main() it would be useful to also upload the metrics binaries (.bin files) in the args.run_dir + "/InterOp/" directory. These provide metrics about the sequencing run that are important to check before downstream analysis.

1. Is there anything other than the explicit reference to the Ubuntu version of the dx-toolkit tarball in tasks/main.yml that makes this Ubuntu specific?

2. Are the monitored_directories being written by the sequencers using the Run Copy Service?

3. I'm interested in running this in support of a MiniSeq. Any gotcha's I should watch out for?

Thanks