cnr-ibba / nf-resequencing-mem Goto Github PK
View Code? Open in Web Editor NEWNextflow resequencing pipeline with bwa-mem and freebayes
License: MIT License
Nextflow resequencing pipeline with bwa-mem and freebayes
License: MIT License
check_samplesheet.py
can't detect header in certain case, even if it's provided correctly. This could be due to csv.Sniffer.has_header
method, which is known to produce both false positive and negatives: for exaple
sample,fastq_1,fastq_2
200-1-5,/home/ngs/freeclimb_resequencing/IGA_2022-01/delivery_20220110/raw_sequences/1_ID2101_200-1-5-H9H05KWZ-H1_S1_L001_R1_001.fastq.gz,/home/ngs/freeclimb_resequencing/IGA_2022-01/delivery_20220110/raw_sequences/1_ID2101_200-1-5-H9H05KWZ-H1_S1_L001_R2_001.fastq.gz
201-1-9,/home/ngs/freeclimb_resequencing/IGA_2022-01/delivery_20220110/raw_sequences/1_ID2101_201-1-9-H9H05KWZ-A2_S1_L001_R1_001.fastq.gz,/home/ngs/freeclimb_resequencing/IGA_2022-01/delivery_20220110/raw_sequences/1_ID2101_201-1-9-H9H05KWZ-A2_S1_L001_R2_001.fastq.gz
202-1-10,/home/ngs/freeclimb_resequencing/IGA_2022-01/delivery_20220110/raw_sequences/1_ID2101_202-1-10-H9H05KWZ-B2_S1_L001_R1_001.fastq.gz,/home/ngs/freeclimb_resequencing/IGA_2022-01/delivery_20220110/raw_sequences/1_ID2101_202-1-10-H9H05KWZ-B2_S1_L001_R2_001.fastq.gz
has issue in detecting header
Add a step which describe the coverage by sample
According to samtools/markdup documentation, there are additional steps to be done before calling the markdup
steps. This steps are performed by bam_markduplicates_samtools nf-core subworkflow, however they need to be performed before the sort
step (that is done during alignment)
samtools/markdup
modulesamtools/markdup
module to produce reportscram_markduplicates_picard
local subworflowbam_markduplicates_samtools
nf-core workflowpicard/markduplicates
modules and configurationsamtools/markdup
with MultiQC config--save_cram
optionconfig/base.config
config/module.config
If there are duplicate IDs in FASTQ
files, the markduplicate step will fail.
seqkit
step to check for IDs duplicatesamplesheet_check
nf-validation
pluginTry to replace markduplicates with sambamba as described here
sambamba/markdup
modulecram_markduplicates_picard
local subworflowpicard/markduplicates
modules and configuration--save_cram
optionsamtools/depth
samtools/depth
by chromosomessamtools/depth
required timeTry to increase pipeline default parameters to better explot resources
Test this pipeline in AWS
environment
S3
AWS
S3
Freebayes can't run on a compressed genome sequence. Add a new step to generate a FAI
index to be used in freebayes step
follow Using nf-core components outside nf-core tutorial and release pipeline to the public
freebayes_parallel
subworkflowenable_conda
params / or remove conda supportdocker.io
to container URL)-stub
optionTry to optimize the coverage step
upgrade nf-core
modules
use samtools view to convert CRAM into BAM -> CRAM convert with SAMTOOLS/CRAM module:
freebayes
using CRAM filessamtools view
Can't lint pipeline anymore. Something changes in modules? check private modules structure
Test this pipeline in a PBS
environment
singularity
conda
using PBS
singularity
using PBS
freebayes/single
to freebayes/multi
nf-core/tools v2.7.1
cannot lint pipeline anymore: there are some changes which could affect CI test. Pipeline need to be updated and lint test should pass without warnings:
pipeline name has issue: .nf-core.yml
should change like this:
lint:
actions_awsfulltest: False
pipeline_todos: False
files_exist: False
nextflow_config:
- manifest.name
files_unchanged:
- .gitignore
- .github/workflows/linting.yml
actions_ci: False
merge_markers: False
multiqc_config: False
repository_type: pipeline
v2.7.1
has issue in parsing this file, the dev
branch of nf-core/tools
can parse this config file correctly
CI fails because of new templates
use split_ref_by_bai_datasize.py
in freebayes/multi
module
Try to generate a MultiQC report with all supported tools
This pipeline currently cannot deal with single end libraries. Moreover, is difficult to manage single and paired ends samples in the same run. Currently, nextflow
pipelines accept input data from a sample files, where single and paired ends files are clearly stated and managed properly. Adapt this pipeline in order to accept data like this
By visually inspecting the alignments (using samtools tview
, there are some regions which seems to have bad alignments: the reason seems to be that Markduplicates
changes the sequence in aligned file. For example, before markduplicates
we have:
samtools view WT.cram Chr01:60000-60001 --reference Pvulgaris_442_v2.0.fa | head -n1
A01083:294:H3LHWDSXC:2:2553:27751:35556 147 Chr01 59859 60 150M = 59547 -462 CGCCGCGTCTTTTAAGAAAATAGCGGGAGAAGAAACTTCGATTTTCAATAACAATGAAGGTAAATTAAATTGATAAATTTTATATTCAATTGATAGCAATAAATCACGCAAATATGTAAATTGAAATATTTATTTTAAAGTTTCGATAAC FFF:,FFF:F:FFF,FFFFFF:,F,,,:FFFFFFFFFFFFFFFFFFFFFFFFFFFFFF:FFFFFFF:FFFFFFFFFFFFFFFFFFF:FFF:FFFFFFFFFFFFFFFFFFFF:FF:FFFFFFFFFF,FFFFFFFFFFFFFFFFFFFFFFFFMC:Z:150M AS:i:145 XS:i:20 MD:Z:24T125 NM:i:1 RG:Z:WT
And after markduplicates
we have:
samtools view WT.md.cram Chr01:60000-60001 --reference Pvulgaris_442_v2.0.fa | head -n1
A01083:294:H3LHWDSXC:2:2553:27751:35556 147 Chr01 59859 60 150M = 59547 -462 NCNNCNCGNGGGGCCCCCCCGCCNCCCCCCCCCCCNGGNCCGGGGNCCGCCNCCGCCCCCGCCCGGCCCGGCCGCCCGGGGCGCGGNCCGGCCGCCNCCGCCCGNCNCNCCCGCGCGCCCGGCCCCGCGGGCGGGGCCCCGGGNCCGCCN FFF:,FFF:F:FFF,FFFFFF:,F,,,:FFFFFFFFFFFFFFFFFFFFFFFFFFFFFF:FFFFFFF:FFFFFFFFFFFFFFFFFFF:FFF:FFFFFFFFFFFFFFFFFFFF:FF:FFFFFFFFFF,FFFFFFFFFFFFFFFFFFFFFFFFAS:i:145 MC:Z:150M PG:Z:MarkDuplicates XS:i:20 MD:Z:0C0G0C0C0G0C0G0T0C0T0T0T0T0A0A0G0A0A0A0A0T0A0G0C0T0G0G0A0G0A0A0G0A0A0A0C0T0T0C0G0A0T0T0T0T0C0A0A0T0A0A0C0A0A0T0G0A0A0G0G0T0A0A0A0T0T0A0A0A0T0T0G0A0T0A0A0A0T0T0T0T0A0T0A0T0T0C0A0A0T0T0G0A0T0A0G0C0A0A0T0A0A0A0T0C0A0C0G0C0A0A0A0T0A0T0G0T0A0A0A0T0T0G0A0A0A0T0A0T0T0T0A0T0T0T0T0A0A0A0G0T0T0T0C0G0A0T0A0A0C0 NM:i:150 RG:Z:WT
Then number of matches is identical, however markduplicates
add 150 mismatches, and the sequence changed in column 10 is the sequence visualized using samtools tview
. This behaviour does not affect all the genome regions. Is not clear how this affects the calling process. Markduplicates
should be removed as described in #71
Update modules and fix issues with linter/CI
customize MultiQC configuration file
In teory, I will index .bam
files with markduplicates step. Can I remove samtools/index
? or can I skip markduplicates indexing?
Add the possibility to export BAM
(or better CRAM
) as output
remove the bamaddrg
and try to add RG as suggested here or using picard
This pipeline cannot be managed with the latest nf-core
version. Move files and fix stuff to be compliant to DLS2
Add vcfallelicprimitives
step before doing the normalization with bcftools
. Work on each chromosome independently in order to optimize resources
--help
commandTry to update modules to latest version:
$ nf-core modules lint
Should return 0 warnings
Upgrade modules:
Try to normalize VCF in order to filter out SNP as described here and in freebayes documentation
As described in freebayes documentation, we need to call bamaddrg in order to add name to the alignment and then on the VCF header
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.