MissingInputException about drop HOT 19 CLOSED

Any update about this bug? I can't run drop until it is fixed. Thanks.

from drop.

Do you have any DNA input for the pipeline? If not, that works explain why MAE does not find any input files and can't be run. The other pipelines should not be affected. If you run snakemake aberrantSplicing separately from the unlock command, it should run.

from drop.

Yes I have DNA input. The issue is not with the aberrantExpression command. The issue is with the unlock command.
If I just do 'snakemake unlock' by itself, I'm getting this error. This error happened after I recently updated drop and then did drop update.

See output below:
[evrong01@bigpurple-ln2 droptest]$ snakemake unlock
check for missing R packages
Structuring dependencies...
Dependencies file generated.

Building DAG of jobs...
Executing subworkflow AE.
Structuring dependencies...
Dependencies file generated.

Building DAG of jobs...
Nothing to be done.
Complete log: /gpfs/scratch/evrong01/droptest/.drop/modules/aberrant-expression-pipeline/.snakemake/log/2020-07-08T153153.882162.snakemake.log
Executing subworkflow AS.
Structuring dependencies...
Dependencies file generated.

Building DAG of jobs...
Nothing to be done.
Complete log: /gpfs/scratch/evrong01/droptest/.drop/modules/aberrant-splicing-pipeline/.snakemake/log/2020-07-08T153154.295884.snakemake.log
Executing subworkflow MAE.
Structuring dependencies...
Dependencies file generated.

Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1
Rules claiming more threads will be scaled down.
Job counts:
count jobs
1 unlock
1

[Wed Jul 8 15:31:54 2020]
rule unlock:
output: /gpfs/scratch/evrong01/droptest/.drop/tmp/MAE/unlock
jobid: 0

Structuring dependencies...
Dependencies file generated.

Building DAG of jobs...
MissingInputException in line 62 of /gpfs/scratch/evrong01/droptest/.drop/modules/mae-pipeline/Snakefile:
Missing input files for rule create_SNVs:
/gpfs/scratch/evrong01/droptest/.drop/modules/mae-pipeline/.drop/modules/mae-pipeline/Scripts/MAE/filterSNVs.sh
/gpfs/scratch/evrong01/droptest/.drop/modules/mae-pipeline/.drop/modules/mae-pipeline/resource/chr_NCBI_UCSC.txt
/gpfs/scratch/evrong01/droptest/.drop/modules/mae-pipeline/.drop/modules/mae-pipeline/resource/chr_UCSC_NCBI.txt
[Wed Jul 8 15:31:56 2020]
Error in rule unlock:
jobid: 0
output: /gpfs/scratch/evrong01/droptest/.drop/tmp/MAE/unlock
shell:
snakemake --unlock --configfile /gpfs/scratch/evrong01/droptest/.drop/tmp/config.yaml
(exited with non-zero exit code)

Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /gpfs/scratch/evrong01/droptest/.drop/modules/mae-pipeline/.snakemake/log/2020-07-08T153154.564192.snakemake.log

from drop.

unlock seemed to be successful for aberrant expression and splicing, but fails for MAE. Calling the first two pipelines should work fine despite the error, as they are independent. As to why the MAE pipeline does not find any input, the only thing that works come to my mind is that the file paths might not be correct, resulting in an empty most of MAE IDs. Anything else would probably break the demo set. Check to see if running drop on the set hiding the updated version still works.

from drop.

The problem is not with my inputs. The problem is with the drop scripts. If I look at the path that the error gives, there is no such folder. See below. It must be a bug in the drop code.

[evrong01@bigpurple-ln2 droptest]$ ls /gpfs/scratch/evrong01/droptest/.drop/modules/mae-pipeline
mae_readme.md resource Scripts Snakefile
[evrong01@bigpurple-ln2 droptest]$ ls /gpfs/scratch/evrong01/droptest/.drop/modules/mae-pipeline/.drop
ls: cannot access /gpfs/scratch/evrong01/droptest/.drop/modules/mae-pipeline/.drop: No such file or directory

from drop.

Also, nothing has changed in my config.yaml and input files. This never gave an error until I updated drop 2 days ago. Something changed in your code.

from drop.

which file are you looking for? the submodule files look complete to me

from drop.

did you call drop update after you pulled new changes? or did that command break the pipeline for you?

from drop.

The error is saying that these files are missing:
MissingInputException in line 62 of /gpfs/scratch/evrong01/droptest/.drop/modules/mae-pipeline/Snakefile:
Missing input files for rule create_SNVs:
/gpfs/scratch/evrong01/droptest/.drop/modules/mae-pipeline/.drop/modules/mae-pipeline/Scripts/MAE/filterSNVs.sh
/gpfs/scratch/evrong01/droptest/.drop/modules/mae-pipeline/.drop/modules/mae-pipeline/resource/chr_NCBI_UCSC.txt
/gpfs/scratch/evrong01/droptest/.drop/modules/mae-pipeline/.drop/modules/mae-pipeline/resource/chr_UCSC_NCBI.txt

[evrong01@bigpurple-ln2 droptest]$ ls /gpfs/scratch/evrong01/droptest/.drop/modules/mae-pipeline/.drop
ls: cannot access /gpfs/scratch/evrong01/droptest/.drop/modules/mae-pipeline/.drop: No such file or directory

These 3 files: filterSNVs.sh, chr_NCBI_UCSC.txt, chr_UCSC_NCBI.txt, are completely missing. They are instead in this folders:
[evrong01@bigpurple-ln2 droptest]$ find . -name "chr_UCSC_NCBI.txt"
./.drop/modules/aberrant-expression-pipeline/resource/chr_UCSC_NCBI.txt
./.drop/modules/mae-pipeline/resource/chr_UCSC_NCBI.txt

However, your script has a bug and it is looking for them in the wrong place.

from drop.

there was a change in the MAE pipeline that was merged recently, however I don't know exactly what changed. @vyepez88 do you know what could cause this?

from drop.

Yes, I did drop update and then everything stopped working.

from drop.

I was able to reproduce your error in the demo dataset when either the aberrant expression or mono-allelic expression modules haven't been run yet. The snakemake unlock command is trying to unlock an empty repository and throws that error. We'll have a look into it.
In any case, that should not prevent you from running any of the 3 modules. Can you try snakemake aberrantSplicing without the unlock command beforehand?

from drop.

Without snakemake unlock it seems to be working. Let me know once the bug is fixed, so that I can also run mono-allelic expression.

from drop.

You should also be able to run mono-allelic expression without the unlock

from drop.

Hi, Has this bug been fixed? I need to rerun the aberrantSplicing pipeline but it won't run because the directory is locked. But I can't unlock the directory due to the unlock bug:

Building DAG of jobs...
MissingInputException in line 38 of /gpfs/home/evrong01/.local/lib/python3.6/site-packages/wbuild/wBuild.snakefile:
Missing input files for rule markdown:
MAE/GTEX100--v32_results.md
[Sat Jul 11 11:36:09 2020]
Error in rule unlock:
jobid: 0
output: /gpfs/scratch/evrong01/droptest/.drop/tmp/MAE/unlock
shell:
snakemake --unlock --configfile /gpfs/scratch/evrong01/droptest/.drop/tmp/config.yaml
(exited with non-zero exit code)

Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /gpfs/scratch/evrong01/droptest/.drop/modules/mae-pipeline/.snakemake/log/2020-07-11T113607.991548.snakemake.log

from drop.

Does the aberrant splicing module remain locked even after the failed unlock command?

from drop.

No it is still locked...
The unlock bug need to be fixed for anyone to be able to use drop.

check for missing R packages
Structuring dependencies...
Dependencies file generated.

Building DAG of jobs...
Error: Directory cannot be locked. Please make sure that no other Snakemake process is trying to create the same files in the following directory:
/gpfs/scratch/evrong01/droptest
If you are sure that no other instances of snakemake are running on this directory, the remaining lock was likely caused by a kill signal or a power loss. It can be removed with the --unlock argument.

from drop.

Try and see if

snakemake --unlock
snakemake aberrantSplicing

works.
If nothing helps, snakemake --nolock should be a quick fix for the problem.

Fixing the bug might take some time, as we are planning to remive the subworkflow structure to make things more accessible. This will be a major upgrade to drop, probably requiring a complete rerun of old projects in order to be sure things work true way they should.

from drop.

Yes snakemake --unlock, followed by snakemake aberrantSplicing worked.
Looking forward to the new upgrade.

from drop.