Comments (10)
iquasere
commented on August 20, 2024
1
Greetings! I'm sorry that you are having problems running MOSCA... Please tell me the output of
sortmerna -v
ls resources_directory/rRNA_databases
head output/Preprocess/SortMeRNA/MIT1_interleaved.fastq
To see if everything is in its right place
from mosca.
iquasere
commented on August 20, 2024
1
I'm going to test this in the latest version of MOSCA, if it is not working it is a big problem, and I'll launch a new version in the next days to rectify it! Your experiments.tsv seems fine, maybe just try to remove the " characters, but does seem to me like it would disturb the parsing of the file
from mosca.
ChrisTrivedi
commented on August 20, 2024
Greetings! I'm sorry that you are having problems running MOSCA... Please tell me the output of
sortmerna -vls resources_directory/rRNA_databaseshead output/Preprocess/SortMeRNA/MIT1_interleaved.fastqTo see if everything is in its right place
Hi, thanks for the prompt reply. Luckily I realized that the program wants to integrate MG and MT data so I added an associated MG samples and it continued through sortmerna this time (although it might be nice if we could run just MT data as there aren't many full pipelines out there for this).
Unfortunately, I did run into another problem at the beginning of the annotation step. I'm not quite sure, but it seems to be an error with downloading the uniprot database? I'll add the traceback here, however, if you prefer I open a new issue I'm happy to as well. Thanks again for the help!
edge base / complete ... finished. [0/1990]
==> SIZE uniprot_sprot.fasta.gz ... 89966396
==> PASV ... ready. ==> RETR uniprot_sprot.fasta.gz ... fert.
Length: 89 966 396 (86M) (not relevant)
uniprot_sprot.fasta 100% [====================>] 85.80M 12.0MB / s in 8.0s
2021-05-03 15:45:05 (10.8 MB / s) - "uniprot_sprot.fasta.gz" belongs to [89966396
]]
Traceback (last call last):
File "/home/ctrivedi/miniconda3/envs/mosca/share/MOSCA/scripts/annotation.py",
Line 218 in <module>
Annotater (). Run ()
File "/home/ctrivedi/miniconda3/envs/mosca/share/MOSCA/scripts/annotation.py",
Line 134, in the run
self.download_uniprot ('/'. join (args.database.split ('/') [: - 1]))
File "/home/ctrivedi/miniconda3/envs/mosca/share/MOSCA/scripts/annotation.py",
Line 81 in download_uniprot
run_command ('zcat uniprot_trembl.fasta.gz uniprot_sprot.fasta.gz', file = '{} /
uniprot.fasta'.format (out_dir))
TypeError: run_command () received an unexpected keyword argument 'file'.
[Monday, May 3, 3:45:05 p.m. 2021]
Error in rule annotation:
jobid: 0
Output: Output / Annotation / Zymo11 / fgs.faa, Output / Annotation / Zymo11 / align.b
latest
RuleException:
CalledProcessError on line 190 of / home / ctrivedi / miniconda3 / envs / mosca / share / MOS
CA / scripts / snakefile:
Command 'set -euo pipefail; python / home / ctrivedi / miniconda3 / envs / mosca / share / M.
OSCA / scripts / annotation.py -i output / Assembly / Zymo11 / contigs.fasta -t 24 -o outp
ut / Annotation / Zymo11 -em illumina_5 -db resources_directory / uniprot.dmnd -mts 1
--download-uniprot --assembled 'returned exit status 1 not equal to zero.
File "/home/ctrivedi/miniconda3/envs/mosca/lib/python3.7/site-packages/snakema
ke / executors / __ init__.py ", line 2352, in run_wrapper
File "/ home / ctrivedi / miniconda3 / envs / mosca / share / MOSCA / scripts / Snakefile", lin
e 190, in __rule_annotation
File "/home/ctrivedi/miniconda3/envs/mosca/lib/python3.7/site-packages/snakema
ke / executors / __ init__.py ", line 569, in _callback
File "/home/ctrivedi/miniconda3/envs/mosca/lib/python3.7/concurrent/futures/t$
read.py ", line 57, in the run
File "/home/ctrivedi/miniconda3/envs/mosca/lib/python3.7/site-packages/snakem$
ke / executors / __ init__.py ", line 555, in cached_or_run
File "/home/ctrivedi/miniconda3/envs/mosca/lib/python3.7/site-packages/snakem$
ke / executors / __ init__.py ", line 2364, in run_wrapper
Quit because a job failed. Look for an error message above
Shutting down may take some time.
Quit because a job failed. Look for an error message above
Full log: / home / ctrivedi / Working / Deep_Purple / GR19_Metatranscriptomes / Mosca / $
snakemake / log / 2021-05-03T113547.475714.snakemake.log
from mosca.
iquasere
commented on August 20, 2024
This is an embarrasing bug, it is a leftover from a function rework that is not catched on testing because the downloading of UniProt is not tested. You can either edit the script directly, and exchange the file= part with output=, or download UniProt manually with the commands
wget ftp://ftp.uniprot.org/pub/databases/uniprot/current_release/knowledgebase/complete/uniprot_trembl.fasta.gz
wget ftp://ftp.uniprot.org/pub/databases/uniprot/current_release/knowledgebase/complete/uniprot_sprot.fasta.gz
zcat uniprot_trembl.fasta.gz uniprot_sprot.fasta.gz > uniprot.fasta
rm uniprot_trembl.fasta.gz uniprot_sprot.fasta.gz
and specify in the config to not download UniProt.
About using only MT data, MOSCA already has that, if there is no MG it should perform MT-only analysis. It is, however, a very different workflow, where annotation with DIAMOND is directly performed on the reads
from mosca.
iquasere
commented on August 20, 2024
This will be rectified in the next version of MOSCA, but may take some time before it comes out, as I am working on finishing the metaproteomics workflow for it
from mosca.
ChrisTrivedi
commented on August 20, 2024
No, problem, I just appreciate the swift replies and the development of the software. I'm not a dev myself, but I can imagine it takes quite a while to update between versions.
I will give your fix a try and see where it gets me.
Another quick question, if you don't mind. Am I correct in assuming that you can run multiple metagenomes and metatranscriptomes at the same time? I've attempted this now and am getting new errors. Maybe I'm organizing my experiment.tsv improperly?
Files Sample Data type Condition Name
"Raw/8MGs_DP_MGs_11_S7_R1_001.fastq.gz,Raw/8MGs_DP_MGs_11_S7_R2_001.fastq.gz" MIT1 dna Ice MIT1_MG
"Raw/RNA_TR_zymo11_S6_R1_001.fastq.gz,Raw/RNA_TR_zymo11_S6_R2_001.fastq.gz" MIT1 mrna Ice MIT1_TR
"Raw/8MGs_DP_MGs_12_S2_R1_001.fastq.gz,Raw/8MGs_DP_MGs_12_S2_R2_001.fastq.gz" MIT2 dna Snow MIT2_MG
"Raw/RNA_TR_zymo12_S7_R1_001.fastq.gz,Raw/RNA_TR_zymo12_S7_R2_001.fastq.gz" MIT2 mrna Snow MIT2_TR
Also, the errors I seem to get are random. One was during Trimmomatic, where this is during Sortmerna, and another just ended without a traceback at all. I have a feeling it might be due to improper naming in my experiment.tsv but I'm not sure. Either way, much appreciated for all the help!
File "/home/ctrivedi/miniconda3/envs/mosca/share/MOSCA/scripts/mosca_tools.py", line 24, in run_command
check=True)
File "/home/ctrivedi/miniconda3/envs/mosca/lib/python3.7/subprocess.py", line 512, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['sortmerna', '--ref', 'resources_directory/rRNA_databases/silva-arc-23s-id98.fasta,resources_directory/rRNA_databases/silva-arc-23s-id98.idx:resources_directory/rRNA_databases/rfam-5s-database-id98.fasta,resources_directory/rRNA_databases/rfam-5s-database-id98.idx:resources_directory/rRNA_databases/silva-euk-18s-id95.fasta,resources_directory/rRNA_databases/silva-euk-18s-id95.idx:resources_directory/rRNA_databases/rfam-5.8s-database-id98.fasta,resources_directory/rRNA_databases/rfam-5.8s-database-id98.idx:resources_directory/rRNA_databases/silva-bac-23s-id98.fasta,resources_directory/rRNA_databases/silva-bac-23s-id98.idx:resources_directory/rRNA_databases/silva-bac-16s-id90.fasta,resources_directory/rRNA_databases/silva-bac-16s-id90.idx:resources_directory/rRNA_databases/silva-euk-28s-id98.fasta,resources_directory/rRNA_databases/silva-euk-28s-id98.idx:resources_directory/rRNA_databases/silva-arc-16s-id95.fasta,resources_directory/rRNA_databases/silva-arc-16s-id95.idx', '--reads', 'output/Preprocess/SortMeRNA/MIT2_TR_interleaved.fastq', '--aligned', 'output/Preprocess/SortMeRNA/MIT2_TR_accepted', '--fastx', '--other', 'output/Preprocess/SortMeRNA/MIT2_TR_rejected', '-a', '24', '--paired_out']' returned non-zero exit status 1.
[Mon May 3 18:48:08 2021]
Error in rule preprocess:
jobid: 0
output: output/Preprocess/Trimmomatic/quality_trimmed_MIT2_TR_forward_paired.fq, output/Preprocess/Trimmomatic/quality_trimmed_MIT2_TR_reverse_paired.fq
RuleException:
CalledProcessError in line 111 of /home/ctrivedi/miniconda3/envs/mosca/share/MOSCA/scripts/Snakefile:
Command 'set -euo pipefail; python /home/ctrivedi/miniconda3/envs/mosca/share/MOSCA/scripts/preprocess.py -i Raw/RNA_TR_zymo12_S7_R1_001.fastq.gz,Raw/RNA_TR_zymo12_S7_R2_001.fastq.gz -t 24 -o output/Preprocess -adaptdir resources_directory/adapters -rrnadbs resources_directory/rRNA_databases -d mrna -rd resources_directory -n MIT2_TR --minlen 100 --avgqual 20' returned non-zero exit status 1.
File "/home/ctrivedi/miniconda3/envs/mosca/lib/python3.7/site-packages/snakemake/executors/__init__.py", line 2352, in run_wrapper
File "/home/ctrivedi/miniconda3/envs/mosca/share/MOSCA/scripts/Snakefile", line 111, in __rule_preprocess
File "/home/ctrivedi/miniconda3/envs/mosca/lib/python3.7/site-packages/snakemake/executors/__init__.py", line 569, in _callback
File "/home/ctrivedi/miniconda3/envs/mosca/lib/python3.7/concurrent/futures/thread.py", line 57, in run
File "/home/ctrivedi/miniconda3/envs/mosca/lib/python3.7/site-packages/snakemake/executors/__init__.py", line 555, in cached_or_run
File "/home/ctrivedi/miniconda3/envs/mosca/lib/python3.7/site-packages/snakemake/executors/__init__.py", line 2364, in run_wrapper
Exiting because a job execution failed. Look above for error message
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /home/ctrivedi/Working/Deep_Purple/GR19_Metatranscriptomes/Mosca/.snakemake/log/2021-05-03T184700.416894.snakemake.log
from mosca.
iquasere
commented on August 20, 2024
The only thing that won't work is the differential expression analysis, as it requires replicates. But all other steps should be fine with that configuration.
from mosca.
ChrisTrivedi
commented on August 20, 2024
I'm going to test this in the latest version of MOSCA, if it is not working it is a big problem, and I'll launch a new version in the next days to rectify it! Your experiments.tsv seems fine, maybe just try to remove the
"characters, but does seem to me like it would disturb the parsing of the file
Good point, I'm not actually sure where the " came from as they are not in my TSV file...maybe something to do with the way I copied/pasted.
The only thing that won't work is the differential expression analysis, as it requires replicates. But all other steps should be fine with that configuration.
This is interesting, is it a limitation of DESeq2? Unfortunately, I don't have replicates sequenced for these samples, but I'm assuming I can try and do DE outside of MOSCA on just two samples, correct?
Thanks again for the speedy replies!
from mosca.
iquasere
commented on August 20, 2024
Sorry for the last reply! The only DE you can do on just two samples is comparing them by eye, as no statistical significance can be obtained without replicates...
from mosca.
ChrisTrivedi
commented on August 20, 2024
Hi again,
I'm testing the latest release of MOSCA! Unfortunately, though, I'm already hindered by another preprocess error, so I thought it only appropriate to append to this closed thread. Let me know if you'd prefer I open a new issue.
Somehow I'm getting stuck on a FastQC step after rRNA removal using Sortmerna. I believe it's because the after_rrna_removal_* files are empty - or at least they are 0 kb on inspection. I haven't been able to determine why this is though.
Here is the full error, maybe you can track it better than I can? Thanks!
FastQC --threads 16 --extract /home/ctrivedi/Working/Deep_Purple/mosca_GR19/output/Preprocess/SortMeRNA/after_rrna_removal_GRIS1_TR_forward.fastq /home/ctrivedi/Working/Deep_Purple/mosca_GR19/output/Preprocess/SortMeRNA/after_rrna_removal_GRIS1_TR_reverse.fastq
Started analysis of after_rrna_removal_GRIS1_TR_forward.fastq
Analysis complete for after_rrna_removal_GRIS1_TR_forward.fastq
Failed to process file after_rrna_removal_GRIS1_TR_forward.fastq
java.lang.ArrayIndexOutOfBoundsException: Index -1 out of bounds for length 0
at uk.ac.babraham.FastQC.Modules.SequenceLengthDistribution.calculateDistribution(SequenceLengthDistribution.java:101)
at uk.ac.babraham.FastQC.Modules.SequenceLengthDistribution.raisesError(SequenceLengthDistribution.java:190)
at uk.ac.babraham.FastQC.Report.HTMLReportArchive.startDocument(HTMLReportArchive.java:336)
at uk.ac.babraham.FastQC.Report.HTMLReportArchive.<init>(HTMLReportArchive.java:84)
at uk.ac.babraham.FastQC.Analysis.OfflineRunner.analysisComplete(OfflineRunner.java:178)
at uk.ac.babraham.FastQC.Analysis.AnalysisRunner.run(AnalysisRunner.java:110)
at java.base/java.lang.Thread.run(Thread.java:834)
Started analysis of after_rrna_removal_GRIS1_TR_reverse.fastq
Analysis complete for after_rrna_removal_GRIS1_TR_reverse.fastq
Failed to process file after_rrna_removal_GRIS1_TR_reverse.fastq
java.lang.ArrayIndexOutOfBoundsException: Index -1 out of bounds for length 0
at uk.ac.babraham.FastQC.Modules.SequenceLengthDistribution.calculateDistribution(SequenceLengthDistribution.java:101)
at uk.ac.babraham.FastQC.Modules.SequenceLengthDistribution.raisesError(SequenceLengthDistribution.java:190)
at uk.ac.babraham.FastQC.Report.HTMLReportArchive.startDocument(HTMLReportArchive.java:336)
at uk.ac.babraham.FastQC.Report.HTMLReportArchive.<init>(HTMLReportArchive.java:84)
at uk.ac.babraham.FastQC.Analysis.OfflineRunner.analysisComplete(OfflineRunner.java:178)
at uk.ac.babraham.FastQC.Analysis.AnalysisRunner.run(AnalysisRunner.java:110)
at java.base/java.lang.Thread.run(Thread.java:834)
Traceback (most recent call last):
File "/home/ctrivedi/miniconda3/envs/mosca/share/MOSCA/scripts/preprocess.py", line 357, in <module>
Preprocesser().run()
File "/home/ctrivedi/miniconda3/envs/mosca/share/MOSCA/scripts/preprocess.py", line 343, in run
minlen=args.minlen, type_of_data=args.data)
File "/home/ctrivedi/miniconda3/envs/mosca/share/MOSCA/scripts/preprocess.py", line 248, in quality_trimming
data = parse_fastqc_report(report)
File "/home/ctrivedi/miniconda3/envs/mosca/share/MOSCA/scripts/mosca_tools.py", line 249, in parse_fastqc_report
file = open(filename).read().split('\n')
FileNotFoundError: [Errno 2] No such file or directory: '/home/ctrivedi/Working/Deep_Purple/mosca_GR19/output/Preprocess/FastQC/after_rrna_removal_GRIS1_TR_forward_fastqc/fastqc_data.txt'
[Thu Jan 13 16:39:23 2022]
Error in rule preprocess:
jobid: 0
output: /home/ctrivedi/Working/Deep_Purple/mosca_GR19/output/Preprocess/Trimmomatic/quality_trimmed_GRIS1_TR_forward_paired.fq, /home/ctrivedi/Working/Deep_Purple/mosca_GR19/output/Preprocess/Trimmomatic/quality_trimmed_GRIS1_TR_reverse_paired.fq
RuleException:
CalledProcessError in line 103 of /home/ctrivedi/miniconda3/envs/mosca/share/MOSCA/scripts/Snakefile:
Command 'set -euo pipefail; python /home/ctrivedi/miniconda3/envs/mosca/share/MOSCA/scripts/preprocess.py -i /home/ctrivedi/Working/Deep_Purple/GR19_Metatranscriptomes/Raw/totalRNA/RNA_TR_zymo7_S4_R1_001.fastq.gz,/home/ctrivedi/Working/Deep_Purple/GR19_Metatranscriptomes/Raw/totalRNA/RNA_TR_zymo7_S4_R2_001.fastq.gz -t 16 -o /home/ctrivedi/Working/Deep_Purple/mosca_GR19/output/Preprocess -adaptdir resources_directory/adapters -rrnadbs resources_directory/rRNA_databases -d mrna -rd resources_directory -n GRIS1_TR --minlen 100 --avgqual 20' returned non-zero exit status 1.
File "/home/ctrivedi/miniconda3/envs/mosca/lib/python3.7/site-packages/snakemake/executors/__init__.py", line 2347, in run_wrapper
File "/home/ctrivedi/miniconda3/envs/mosca/share/MOSCA/scripts/Snakefile", line 103, in __rule_preprocess
File "/home/ctrivedi/miniconda3/envs/mosca/lib/python3.7/site-packages/snakemake/executors/__init__.py", line 568, in _callback
File "/home/ctrivedi/miniconda3/envs/mosca/lib/python3.7/concurrent/futures/thread.py", line 57, in run
File "/home/ctrivedi/miniconda3/envs/mosca/lib/python3.7/site-packages/snakemake/executors/__init__.py", line 554, in cached_or_run
File "/home/ctrivedi/miniconda3/envs/mosca/lib/python3.7/site-packages/snakemake/executors/__init__.py", line 2359, in run_wrapper
Exiting because a job execution failed. Look above for error message
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /home/ctrivedi/Working/Deep_Purple/mosca_GR19/.snakemake/log/2022-01-13T163802.188280.snakemake.log
from mosca.
Related Issues (20)
- Empty Output Directories HOT 9
- Possible to run MT without MG? HOT 42
- typo in mosca_tools.py? HOT 1
- typo in assembly.py HOT 1
- self.bowtie2 not defined in assembly.py HOT 2
- self.input error in annotation step HOT 1
- Versioning / releases HOT 2
- File not found error HOT 5
- New problem encounter HOT 34
- No adapter removal despite Adapter Content Fail in FastQC report HOT 1
- No such file or directory: 'output/Preprocess/SortMeRNA/read1.txt' HOT 5
- join_information issue HOT 8
- Error in rule preprocess - FastQC after Sortmerna HOT 10
- Mismatch between Wiki and actual config requirements HOT 1
- Snakefile KeyError for 'split_gene_calling' HOT 4
- Trouble installing HOT 1
- Issue with metaquast after assembly HOT 2
- Parameter from config.json not recognized. HOT 1
- Error in rule upimapi -Annotation error- HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mosca.