cnr-ibba / nf-resequencing-mem Goto Github PK

check_samplesheet.py can't detect header in certain case, even if it's provided correctly. This could be due to csv.Sniffer.has_header method, which is known to produce both false positive and negatives: for exaple

sample,fastq_1,fastq_2
200-1-5,/home/ngs/freeclimb_resequencing/IGA_2022-01/delivery_20220110/raw_sequences/1_ID2101_200-1-5-H9H05KWZ-H1_S1_L001_R1_001.fastq.gz,/home/ngs/freeclimb_resequencing/IGA_2022-01/delivery_20220110/raw_sequences/1_ID2101_200-1-5-H9H05KWZ-H1_S1_L001_R2_001.fastq.gz
201-1-9,/home/ngs/freeclimb_resequencing/IGA_2022-01/delivery_20220110/raw_sequences/1_ID2101_201-1-9-H9H05KWZ-A2_S1_L001_R1_001.fastq.gz,/home/ngs/freeclimb_resequencing/IGA_2022-01/delivery_20220110/raw_sequences/1_ID2101_201-1-9-H9H05KWZ-A2_S1_L001_R2_001.fastq.gz
202-1-10,/home/ngs/freeclimb_resequencing/IGA_2022-01/delivery_20220110/raw_sequences/1_ID2101_202-1-10-H9H05KWZ-B2_S1_L001_R1_001.fastq.gz,/home/ngs/freeclimb_resequencing/IGA_2022-01/delivery_20220110/raw_sequences/1_ID2101_202-1-10-H9H05KWZ-B2_S1_L001_R2_001.fastq.gz

has issue in detecting header

Add a step which describe the coverage by sample

According to samtools/markdup documentation, there are additional steps to be done before calling the markdup steps. This steps are performed by bam_markduplicates_samtools nf-core subworkflow, however they need to be performed before the sort step (that is done during alignment)

• install samtools/markdup module
• patch samtools/markdup module to produce reports
• fix and rename cram_markduplicates_picard local subworflow
• install bam_markduplicates_samtools nf-core workflow
• remove picard/markduplicates modules and configuration
• check samtools/markdup with MultiQC config
• test --save_cram option

• add snpeff to the pipeline
• conditional check for snpeff analysis
• snpeff using community databases
• snpeff using custom database
• compress and index VCF file

• add config/base.config
• add config/module.config
• move stuff in config files
• check for local resource availability
• test CI within github workflow
• add a default configuration for core environment

If there are duplicate IDs in FASTQ files, the markduplicate step will fail.

• add seqkit step to check for IDs duplicate
• test stuff

• update or remove samplesheet_check
• add nf-validation plugin
• update assets
• test parameters and validation

Try to replace markduplicates with sambamba as described here

• install sambamba/markdup module
• fix and rename cram_markduplicates_picard local subworflow
• remove picard/markduplicates modules and configuration
• test --save_cram option

• use bgzip with multi-threads when creating samtools/depth
• try to split samtools/depth by chromosomes
• raise samtools/depth required time

Try to increase pipeline default parameters to better explot resources

Test this pipeline in AWS environment

• move test data to S3
• call pipeline on AWS
• collect data from S3

Freebayes can't run on a compressed genome sequence. Add a new step to generate a FAI index to be used in freebayes step

follow Using nf-core components outside nf-core tutorial and release pipeline to the public

• import the freebayes_parallel subworkflow
• change pipeline structure and parameter
• test with real data

• update module dependencies
• update custom module dependencies
• remove the enable_conda params / or remove conda support
• fix changed steps
• fix the default container registry for custom modules (prepend docker.io to container URL)
• test for -stub option

Try to optimize the coverage step

bwa-mem2 seems to be an improved implementation of bwa mem. Replace old bwa modules in pipeline and test data

• replace bwa/index with bwamem2/index
• replace bwa/mem with bwamem2/mem

upgrade nf-core modules

use samtools view to convert CRAM into BAM -> CRAM convert with SAMTOOLS/CRAM module:

• call freebayes using CRAM files
• check alignments with samtools view
• extract info from CRAM
• replace BAM with CRAM
• deal with region overlap

Can't lint pipeline anymore. Something changes in modules? check private modules structure

Test this pipeline in a PBS environment

• install singularity
• test with conda using PBS
• test with singularity using PBS

• move freebayes/single to freebayes/multi
• try to split by chromosomes and calculate freebayes on each chromosome (with all samples)
• chunk chromosomes as described in freebayes parallelization
• collect all data in a unique VCF file
• accept bwa indexes and genome indexes as parameters

nf-core/tools v2.7.1 cannot lint pipeline anymore: there are some changes which could affect CI test. Pipeline need to be updated and lint test should pass without warnings:

Additional information

pipeline name has issue: .nf-core.yml should change like this:

lint:
  actions_awsfulltest: False
  pipeline_todos: False
  files_exist: False
  nextflow_config:
    - manifest.name
  files_unchanged:
    - .gitignore
    - .github/workflows/linting.yml
  actions_ci: False
  merge_markers: False
  multiqc_config: False
repository_type: pipeline

v2.7.1 has issue in parsing this file, the dev branch of nf-core/tools can parse this config file correctly

CI fails because of new templates

use split_ref_by_bai_datasize.py in freebayes/multi module

Try to generate a MultiQC report with all supported tools

This pipeline currently cannot deal with single end libraries. Moreover, is difficult to manage single and paired ends samples in the same run. Currently, nextflow pipelines accept input data from a sample files, where single and paired ends files are clearly stated and managed properly. Adapt this pipeline in order to accept data like this

By visually inspecting the alignments (using samtools tview, there are some regions which seems to have bad alignments: the reason seems to be that Markduplicates changes the sequence in aligned file. For example, before markduplicates we have:

samtools view WT.cram Chr01:60000-60001 --reference Pvulgaris_442_v2.0.fa | head -n1
A01083:294:H3LHWDSXC:2:2553:27751:35556 147     Chr01   59859   60      150M    =       59547   -462    CGCCGCGTCTTTTAAGAAAATAGCGGGAGAAGAAACTTCGATTTTCAATAACAATGAAGGTAAATTAAATTGATAAATTTTATATTCAATTGATAGCAATAAATCACGCAAATATGTAAATTGAAATATTTATTTTAAAGTTTCGATAAC FFF:,FFF:F:FFF,FFFFFF:,F,,,:FFFFFFFFFFFFFFFFFFFFFFFFFFFFFF:FFFFFFF:FFFFFFFFFFFFFFFFFFF:FFF:FFFFFFFFFFFFFFFFFFFF:FF:FFFFFFFFFF,FFFFFFFFFFFFFFFFFFFFFFFFMC:Z:150M       AS:i:145        XS:i:20 MD:Z:24T125     NM:i:1  RG:Z:WT

And after markduplicates we have:

samtools view WT.md.cram Chr01:60000-60001 --reference Pvulgaris_442_v2.0.fa | head -n1
A01083:294:H3LHWDSXC:2:2553:27751:35556 147     Chr01   59859   60      150M    =       59547   -462    NCNNCNCGNGGGGCCCCCCCGCCNCCCCCCCCCCCNGGNCCGGGGNCCGCCNCCGCCCCCGCCCGGCCCGGCCGCCCGGGGCGCGGNCCGGCCGCCNCCGCCCGNCNCNCCCGCGCGCCCGGCCCCGCGGGCGGGGCCCCGGGNCCGCCN FFF:,FFF:F:FFF,FFFFFF:,F,,,:FFFFFFFFFFFFFFFFFFFFFFFFFFFFFF:FFFFFFF:FFFFFFFFFFFFFFFFFFF:FFF:FFFFFFFFFFFFFFFFFFFF:FF:FFFFFFFFFF,FFFFFFFFFFFFFFFFFFFFFFFFAS:i:145        MC:Z:150M       PG:Z:MarkDuplicates     XS:i:20 MD:Z:0C0G0C0C0G0C0G0T0C0T0T0T0T0A0A0G0A0A0A0A0T0A0G0C0T0G0G0A0G0A0A0G0A0A0A0C0T0T0C0G0A0T0T0T0T0C0A0A0T0A0A0C0A0A0T0G0A0A0G0G0T0A0A0A0T0T0A0A0A0T0T0G0A0T0A0A0A0T0T0T0T0A0T0A0T0T0C0A0A0T0T0G0A0T0A0G0C0A0A0T0A0A0A0T0C0A0C0G0C0A0A0A0T0A0T0G0T0A0A0A0T0T0G0A0A0A0T0A0T0T0T0A0T0T0T0T0A0A0A0G0T0T0T0C0G0A0T0A0A0C0    NM:i:150        RG:Z:WT

Then number of matches is identical, however markduplicates add 150 mismatches, and the sequence changed in column 10 is the sequence visualized using samtools tview. This behaviour does not affect all the genome regions. Is not clear how this affects the calling process. Markduplicates should be removed as described in #71

Update modules and fix issues with linter/CI

• add trimming step (Trimmomatic / Trimgalore / cutadapt?)
• add snp calling step

customize MultiQC configuration file

In teory, I will index .bam files with markduplicates step. Can I remove samtools/index? or can I skip markduplicates indexing?

Add the possibility to export BAM (or better CRAM) as output

remove the bamaddrg and try to add RG as suggested here or using picard

This pipeline cannot be managed with the latest nf-core version. Move files and fix stuff to be compliant to DLS2

Add vcfallelicprimitives step before doing the normalization with bcftools. Work on each chromosome independently in order to optimize resources

• support for --help command
• dump parameters used (like other nf pipelines
• check for mandatory parameters and manage errors properly

Try to update modules to latest version:

$ nf-core modules lint 

Should return 0 warnings

Upgrade modules:

• picard/markduplicates
• bwa/mem
• samtools/flagstat
• samtools/sort
• bwa/index
• bwa/mem
• fastqc
• multiqc
• picard/markduplicates
• samtools/flagstat
• trimgalore

Try to normalize VCF in order to filter out SNP as described here and in freebayes documentation

As described in freebayes documentation, we need to call bamaddrg in order to add name to the alignment and then on the VCF header