nanoporetech / dtr-phage-pipeline Goto Github PK

After run:

conda env create -f environment.yml
conda activate DTR-phage-pipeline
snakemake --use-conda -p -r -j 10

I got the error,
-bash: /home1/jialh/tools/miniconda3/envs/DTR-phage-pipeline/bin/snakemake: /home1/jialh/tools/miniconda3/envs/DTR-phage-pipeline/bin/python3.8: bad interpreter: Invalid argument

Hi jmeppley. If I have a long assembled fasta file without summary file, but I want to get the k-mer bins like your pipeline, what should I do? Would it work if I creat a virtual sequencing summary file?

Hi, what's the 'input_summary' in your 'config.yaml'? Is this the 'sequencing_summary.txt' you got after basecalling?

Hi there,

Running the pipeline I run into a recursion error (see part of it below) that is related to scipy. I've tried setting the recursion limit in the conda environment created during the snakemake run to 100000 (sys.setrecursionlimit(100000)) but with no success. Can you provide any ideas to fix this @jmeppley? Let me know what other information I can provide.
Thanks!

  File "/mnt/XIO_4_via_NFS/test_dtr/DTR-phage-pipeline/.snakemake/conda/60ec611e186987fffbc0b287faa5a054/lib/python3.8/site-packages/scipy/cluster/hierarchy.py", line 3614, in _dendrogram_calculate_info
    _dendrogram_calculate_info(
  File "/mnt/XIO_4_via_NFS/test_dtr/DTR-phage-pipeline/.snakemake/conda/60ec611e186987fffbc0b287faa5a054/lib/python3.8/site-packages/scipy/cluster/hierarchy.py", line 3581, in _dendrogram_calculate_info
    _dendrogram_calculate_info(
  File "/mnt/XIO_4_via_NFS/test_dtr/DTR-phage-pipeline/.snakemake/conda/60ec611e186987fffbc0b287faa5a054/lib/python3.8/site-packages/scipy/cluster/hierarchy.py", line 3581, in _dendrogram_calculate_info
    _dendrogram_calculate_info(
  File "/mnt/XIO_4_via_NFS/test_dtr/DTR-phage-pipeline/.snakemake/conda/60ec611e186987fffbc0b287faa5a054/lib/python3.8/site-packages/scipy/cluster/hierarchy.py", line 3511, in _dendrogram_calculate_info
    _append_singleton_leaf_node(Z, p, n, level, lvs, ivl,
  File "/mnt/XIO_4_via_NFS/test_dtr/DTR-phage-pipeline/.snakemake/conda/60ec611e186987fffbc0b287faa5a054/lib/python3.8/site-packages/scipy/cluster/hierarchy.py", line 3385, in _append_singleton_leaf_node
    ivl.append(str(int(i)))
RecursionError: maximum recursion depth exceeded while getting the str of an object
[Tue Jul  6 12:49:37 2021]
Error in rule bin_ava_clustering:
    jobid: 1414
    output: /data6/test_dtr/sample/1D/v1/kmer_binning/refine_bins/alignments/146/146.clust.heatmap.png, /data6/test_dtr/sample/1D/v1/kmer_binning/refine_bins/alignments/146/146.clust.info.csv
    conda-env: /mnt/XIO_4_via_NFS/test_dtr/DTR-phage-pipeline/.snakemake/conda/60ec611e186987fffbc0b287faa5a054
    shell:
        python /mnt/XIO_4_via_NFS/test_dtr/DTR-phage-pipeline/scripts/cluster_ava_alignments.py -p /data6/test_dtr/sample/1D/v1/kmer_binning/refine_bins/alignments/146/146.clust  -s 0.8 -n 5 /data6/test_dtr/sample/1D/v1/kmer_binning/refine_bins/alignments/146/146.ava.paf
        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

If it's of any use, some of the reads on the .paf file start with a number e.g. 3a93a918-197a-42e1-bb8f-c58fdbdb2b16 and I've noticed recursion errors may happen with scipy due to this?

Hi , thanks for your sharing of this DTR pipeline! I was runing this pipeline with my own nanopore data and encountered several errors that might need your help. First of all, I could execute step 1 command, but it seemed to fail to create unmap kmer freq map in "5 of 11 steps", with other 10 steps runing sucessfully. Secondly, I couldn't execute step 3 as it reported that 'NameError: name 'expand_teamplte_from_bins' is not defined'. I am not sure whether this has something to do with the error in step1. Thirdly, in each reason of step 1 log file, there is always "reason: Missing output files:", is this normal? I have upload my log file of step1,2,3 and my config.yml, could you help me check that?
step1.log
step2.log
step3.log
config.yml.txt

Hi all,

We are trying to get this pipeline working with the included control dataset so we can use it to characterize a variety of novel bacteriophages that we think may be circularly permuted with terminal repeats.

However, after cloning the pipeline from git and following the tutorial to test the included control dataset we are running into errors indicating the pipeline may be broken.

Our first error:

snakemake: error: unrecognized arguments: -r

Which seems to be caused by the conda package request for snakemake to the newest version where the "-r" flag is no longer used:

We did find that by editing the environment.yml file to use the originally intended version of snakemake.. changing from ">= 5.14.0" to "= 5.14.0" we were able to get past this error

However, we are currently stuck at the following error:

The log file shows the following:

We are assuming this is an issue with newer packages breaking the pipeline, but any thoughts on how to resolve these issues would be greatly appreciated. Please let me know if I can detail anything further.

Hello and thanks for the promising tool. I've install the pipeline and am trying to run the example using: snakemake --use-conda --conda-frontend mamba -p -r -j 10

Here's the error I'm getting (with the full log attached):

Traceback (most recent call last):
File "/global/homes/s/snayfach/.conda/envs/iontorrent/lib/python3.6/site-packages/snakemake/init.py", line 778, in snakemake
executesubworkflows=execute_subworkflows,
File "/global/homes/s/snayfach/.conda/envs/iontorrent/lib/python3.6/site-packages/snakemake/workflow.py", line 701, in execute
dag.update_checkpoint_dependencies()
File "/global/homes/s/snayfach/.conda/envs/iontorrent/lib/python3.6/site-packages/snakemake/dag.py", line 1296, in update_checkpoint_dependencies
logger.info("Updating job {} ({}).".format(self.jobid(j), j))
File "/global/homes/s/snayfach/.conda/envs/iontorrent/lib/python3.6/site-packages/snakemake/dag.py", line 702, in jobid
return self._jobid[job]
KeyError: align_dtr_seqs_to_polished_refs
[Tue Dec 22 13:13:19 2020]
Finished job 23.
137 of 147 steps (93%) done
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /global/projectb/scratch/snayfach/projects/checkv-db/4_sra/ion_torrent/DTR-phage-pipeline/.snakemake/log/2020-12-22T130253.555676.snakemake.log

log.txt

Hi,

Could you please let me know if you have other polished genomes excepting the uvMED ones, for example, any genomes polished from the three samples (25 m, 117 m, and 250 m).

Thanks in advance.

Lin-Xing Chen
postdoc in The Banfield Lab

I am working through the tutorial to get this pipeline working on a cluster using a conda environment as recommended.

As I go to run the test with default settings:
snakemake --use-conda -p -r -j 10

I run into the following error:
snakemake: error: unrecognized arguments: -r

I assume it might be a missing library, or something with conda, was hoping to clarify what this flag is for and which library it might require?

Thank you!