This project forked from nf-core/chipseq
Nextflow ChIP-seq data analysis pipeline, National Genomics Infrastructure, Science for Life Laboratory in Stockholm
License: MIT License
When running with paired-end input, it could be nice to run deepTools bamPEFragmentSize (and pick this up in the MultiQC report). See http://deeptools.readthedocs.io/en/latest/content/tools/bamPEFragmentSize.html
There is currently no documentation explaining what the pipeline does, or what the output it produces looks like.
Would be great if something like this could be written (doesn't have to be so extensive).
It would be nice with a summary of fragment length etc. from phantompeaktools in multiqc too. Ideally the entire plots, but otherwise the fragment lengths and NCS and RCS numbers.
According to this publication:
"The evaluation of ChIP quality following duplicate removal may therefore underestimate the extent of ChIP enrichment relative to background..." ... "We recommend the assessment of RSC and NSC prior to blacklisting or duplicate removal..."
Thus, perhaps, steps phantompeakqualtools and calculateNSCRSC could be performed on the non-deduplicated data, instead of the deduplicated?
Builds have started failing due to the new v0.27 release of nextflow giving hard failures for missing config variables.
This line has never worked: https://github.com/SciLifeLab/NGI-ChIPseq/blob/master/nextflow.config#L16 - workflow isn't available yet when parsing the config file. This always returned an empty value before giving :latest as expected, so was a silent error.
Now it fails: https://travis-ci.org/SciLifeLab/NGI-ChIPseq/builds/326839056#L648
This is discussed on gitter here: https://gitter.im/nextflow-io/nextflow?at=5a54d682ae53c15903c81509
Maybe squiggly brackets + defining the container tag under the process scope may work...
It would be nice to bring this workflow and the the chipseq_qc pipeline by @orzechoj into line in terms of functionality.
The pipelines are pretty similar, the main thing I can see that's missing here is the bedtools_intersectNeg functionality:
Bedtools intersectNeg
Takes a bam file and a bed file with regions, given by the flag blacklistFile. Filters the bam file, so the only reads not present in the blacklist regions are included in the output. Uses bedtools intesect for this.
Module code is here, but it basically comes down to the following command:
bedtools intersect -abam $INPUT -b $blacklistFile -v > $OUTPUT
The $blacklistFile is a new type of reference file that we'll need to add to the config.
@orzechoj - is there anything else that you think we should add? Docs about current output from this pipeline are here.
Phil
It would be nice to have the mapping stats from BWA in the multiqc report.
Currently, we check that all FastQ input files exist (--reads) and that the macsconfig file exists (--macsconfig). However, we don't check that the sample names defined in the macsconfig file correspond to the FastQ files submitted. If wrong, this results in the pipeline running nearly to completion and then failing with a slightly obscure error message in the MACS peak calling step.
Instead, it would be good to validate that these two correspond with one another before any pipeline tasks are started, exiting with an error message immediately if something is wrong.
It would be nice to collect software versions at run time, as recently implemented in the NGI-MethylSeq and NGI-RNAseq pipelines.
See how this was done in SciLifeLab/NGI-RNAseq#141.
Instead of having embedded Perl and R scripts, make these stand-alone and move them into the pipeline./bin directory. Then call them in the process.
See the NGI-RNAseq pipeline for an example of this. eg. the DupRadar process calls bin/dupRadar.r.
NB: Filenames have to be supplied as command line arguments!! Bit of a pain, but means that scripts can be used by themselves, so easier to maintain and use. Also removes all of the \${} escaping stuff which is a pain in nextflow scripts.
There are several commands in post_peak_calling_processing.r that assumes that we have an internet connection. This can sometimes be a problem.
1.) R modules are installed on the fly (lines 17-34)
Could this be solved by putting these modules in the singularity image?
2.) Genome annotation is downloaded from ensembl using biomart (lines 51, 66, 116).
Not sure what's the best solution to this. Maybe use static annotation files? Or make the annotation step optional??
The jobs run locally (i.e. on the login node) are get_software_versions and multiqc. These fail when I run them on bianca, where I can only get 7Gb memory on the login node. Currently I use the pipeline with base config file like this:
$get_software_versions {
memory = { check_max( 2.GB * task.attempt, 'memory' ) }
executor = 'local'
errorStrategy = 'ignore'
}
$multiqc {
memory = { check_max( 2.GB * task.attempt, 'memory' ) }
executor = 'local'
errorStrategy = { task.exitStatus == 143 ? 'retry' : 'ignore' }
}
The results only included fastqc and trim_galore. They seemed to be finished. The bwa dir was empty and left tasks were undone.
Here is my config for docker version according my poor understanding:
/*
vim: syntax=groovy
-*- mode: groovy;-*-
* -------------------------------------------------
* Nextflow config file for wanglab ChIP-seq QC
* -------------------------------------------------
* Defines reference genomes, using iGenome paths
* Imported under the default 'standard' Nextflow
* profile in nextflow.config
* I change the parameter 2017-06-07
* please see the original template
* at here:
* https://raw.githubusercontent.com/SciLifeLab/NGI-ChIPseq/master/conf/uppmax.config
*
*/
docker {
enabled = true
}
process {
container = 'scilifelab/ngi-chipseq'
executor = 'local'
cpus = 3
memory = 40G
time = 48.h
}
params {
clusterOptions = false
// illumina iGenomes reference file paths on UPPMAX
genomes {
'hg19' { bwa = '/mnt/date/igenomes/Homo_sapiens/UCSC/hg19/Sequence/BWAIndex/' }
'mm10' { bwa = '/mnt/date/igenomes/Mus_musculus/UCSC/mm10/Sequence/BWAIndex/' }
'mm9' { bwa = '/mnt/date/igenomes/Mus_musculus/UCSC/mm9/Sequence/BWAIndex/' }
'GRCm38' { bwa = '/mnt/date/igenomes/Mus_musculus/NCBI/GRCm38/Sequence/BWAIndex/'}
}
}
// nextflow run SciLifeLab/NGI-ChIPseq -profile docker -c /bioware/lib/snake.config --project 'cellHeart' --reads "*.fastq" --genome GRCm38 --macsconfig 'macssetup.config'
Here is the log output:
Jun-08 14:44:04.615 [main] DEBUG nextflow.cli.Launcher - $> /bioware/bin/nextflow run SciLifeLab/NGI-ChIPseq -profile docker -c /bioware/lib/snake.config --project cellHeart --reads *.fastq --genome GRCm38 --macsconfig macssetup.config
Jun-08 14:44:04.789 [main] INFO nextflow.cli.CmdRun - N E X T F L O W ~ version 0.24.1
Jun-08 14:44:05.137 [main] DEBUG nextflow.scm.AssetManager - Git config: /home/zhenyisong/.nextflow/assets/SciLifeLab/NGI-ChIPseq/.git/config; branch: master; remote: origin; url: https://github.com/SciLifeLab/NGI-ChIPseq.git
Jun-08 14:44:05.166 [main] DEBUG nextflow.scm.AssetManager - Git config: /home/zhenyisong/.nextflow/assets/SciLifeLab/NGI-ChIPseq/.git/config; branch: master; remote: origin; url: https://github.com/SciLifeLab/NGI-ChIPseq.git
Jun-08 14:44:05.375 [main] DEBUG nextflow.scm.AssetManager - Git config: /home/zhenyisong/.nextflow/assets/SciLifeLab/NGI-ChIPseq/.git/config; branch: master; remote: origin; url: https://github.com/SciLifeLab/NGI-ChIPseq.git
Jun-08 14:44:05.375 [main] INFO nextflow.cli.CmdRun - Launching `SciLifeLab/NGI-ChIPseq` [sick_blackwell] - revision: c77ec52bb1 [master]
Jun-08 14:44:05.383 [main] DEBUG nextflow.config.ConfigBuilder - Found config base: /home/zhenyisong/.nextflow/assets/SciLifeLab/NGI-ChIPseq/nextflow.config
Jun-08 14:44:05.384 [main] DEBUG nextflow.config.ConfigBuilder - User config file: /bioware/lib/snake.config
Jun-08 14:44:05.385 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /home/zhenyisong/.nextflow/assets/SciLifeLab/NGI-ChIPseq/nextflow.config
Jun-08 14:44:05.385 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /bioware/lib/snake.config
Jun-08 14:44:05.406 [main] DEBUG nextflow.config.ConfigBuilder - Setting config profile: 'docker'
Jun-08 14:44:05.629 [main] DEBUG nextflow.Session - Session uuid: 66ae0ced-cf84-4f80-8f3f-adbd9e404e16
Jun-08 14:44:05.629 [main] DEBUG nextflow.Session - Run name: sick_blackwell
Jun-08 14:44:05.630 [main] DEBUG nextflow.Session - Executor pool size: 64
Jun-08 14:44:05.646 [main] DEBUG nextflow.cli.CmdRun -
Version: 0.24.1 build 4245
Modified: 24-03-2017 19:41 UTC (25-03-2017 03:41 CDT)
System: Linux 3.10.0-327.el7.x86_64
Runtime: Groovy 2.4.10 on OpenJDK 64-Bit Server VM 1.8.0_65-b17
Encoding: UTF-8 (UTF-8)
Process: 47303@wanglab [127.0.0.1]
CPUs: 64 - Mem: 1,007.6 GB (354 GB) - Swap: 128 GB (127.6 GB)
Jun-08 14:44:05.686 [main] DEBUG nextflow.Session - Work-dir: /mnt/date/biodata/data/cellChIP/heart/work [nfs]
Jun-08 14:44:05.974 [main] DEBUG nextflow.Session - Session start invoked
Jun-08 14:44:05.981 [main] DEBUG nextflow.processor.TaskDispatcher - Dispatcher > start
Jun-08 14:44:05.982 [main] DEBUG nextflow.script.ScriptRunner - > Script parsing
Jun-08 14:44:06.323 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution
Jun-08 14:44:06.392 [main] INFO nextflow.Nextflow - ====================================
Jun-08 14:44:06.392 [main] INFO nextflow.Nextflow - ChIP-seq: v1.2
Jun-08 14:44:06.392 [main] INFO nextflow.Nextflow - ====================================
Jun-08 14:44:06.393 [main] INFO nextflow.Nextflow - Reads : *.fastq
Jun-08 14:44:06.393 [main] INFO nextflow.Nextflow - Genome : GRCm38
Jun-08 14:44:06.393 [main] INFO nextflow.Nextflow - BWA Index : /mnt/date/igenomes/Mus_musculus/NCBI/GRCm38/Sequence/BWAIndex/
Jun-08 14:44:06.393 [main] INFO nextflow.Nextflow - MACS Config : macssetup.config
Jun-08 14:44:06.393 [main] INFO nextflow.Nextflow - Extend Reads : 100 bp
Jun-08 14:44:06.394 [main] INFO nextflow.Nextflow - Current home : /home/zhenyisong
Jun-08 14:44:06.394 [main] INFO nextflow.Nextflow - Current user : zhenyisong
Jun-08 14:44:06.394 [main] INFO nextflow.Nextflow - Current path : /home/zhenyisong/biodata/data/cellChIP/heart
Jun-08 14:44:06.394 [main] INFO nextflow.Nextflow - Script dir : /home/zhenyisong/.nextflow/assets/SciLifeLab/NGI-ChIPseq
Jun-08 14:44:06.394 [main] INFO nextflow.Nextflow - Working dir : /mnt/date/biodata/data/cellChIP/heart/work
Jun-08 14:44:06.394 [main] INFO nextflow.Nextflow - Output dir : ./results
Jun-08 14:44:06.394 [main] INFO nextflow.Nextflow - Save Reference : false
Jun-08 14:44:06.394 [main] INFO nextflow.Nextflow - Save Trimmed : false
Jun-08 14:44:06.394 [main] INFO nextflow.Nextflow - Save Intermeds : false
Jun-08 14:44:06.395 [main] INFO nextflow.Nextflow - Config Profile : docker
Jun-08 14:44:06.395 [main] INFO nextflow.Nextflow - Pipeline Commit: c77ec52bb168db24d01240c164e400e84c663554
Jun-08 14:44:06.395 [main] INFO nextflow.Nextflow - ====================================
Jun-08 14:44:06.397 [main] DEBUG nextflow.Channel - files for syntax: glob; folder: ./; pattern: *.fastq; options: [:]
Jun-08 14:44:06.507 [main] DEBUG nextflow.processor.ProcessFactory - << taskConfig executor: local
Jun-08 14:44:06.508 [main] DEBUG nextflow.processor.ProcessFactory - >> processorType: 'local'
Jun-08 14:44:06.513 [main] DEBUG nextflow.executor.Executor - Initializing executor: local
Jun-08 14:44:06.515 [main] INFO nextflow.executor.Executor - [warm up] executor > local
Jun-08 14:44:06.519 [main] DEBUG n.processor.LocalPollingMonitor - Creating local task monitor for executor 'local' > cpus=64; memory=1,007.6 GB; capacity=64; pollInterval=100ms; dumpInterval=5m
Jun-08 14:44:06.522 [main] DEBUG nextflow.processor.TaskDispatcher - Starting monitor: LocalPollingMonitor
Jun-08 14:44:06.522 [main] DEBUG n.processor.TaskPollingMonitor - >>> barrier register (monitor: local)
Jun-08 14:44:06.524 [main] DEBUG nextflow.executor.Executor - Invoke register for executor: local
Jun-08 14:44:06.548 [main] DEBUG nextflow.Session - >>> barrier register (process: fastqc)
Jun-08 14:44:06.559 [main] DEBUG nextflow.processor.TaskProcessor - Creating operator > fastqc -- maxForks: 64
Jun-08 14:44:06.574 [main] DEBUG nextflow.processor.ProcessFactory - << taskConfig executor: local
Jun-08 14:44:06.574 [main] DEBUG nextflow.processor.ProcessFactory - >> processorType: 'local'
Jun-08 14:44:06.574 [main] DEBUG nextflow.executor.Executor - Initializing executor: local
Jun-08 14:44:06.575 [main] DEBUG nextflow.Session - >>> barrier register (process: trim_galore)
Jun-08 14:44:06.582 [main] DEBUG nextflow.processor.TaskProcessor - Creating operator > trim_galore -- maxForks: 64
Jun-08 14:44:06.589 [Actor Thread 1] DEBUG nextflow.processor.TaskProcessor - <fastqc> Poison pill arrived
Jun-08 14:44:06.597 [Actor Thread 5] DEBUG nextflow.processor.StateObj - <fastqc> State before poison: StateObj[submitted: 14; completed: 0; poisoned: false ]
Jun-08 14:44:06.597 [Actor Thread 11] DEBUG nextflow.processor.TaskProcessor - <trim_galore> Poison pill arrived
Jun-08 14:44:06.598 [main] DEBUG nextflow.processor.ProcessFactory - << taskConfig executor: local
Jun-08 14:44:06.598 [Actor Thread 5] DEBUG nextflow.processor.StateObj - <trim_galore> State before poison: StateObj[submitted: 14; completed: 0; poisoned: false ]
Jun-08 14:44:06.598 [main] DEBUG nextflow.processor.ProcessFactory - >> processorType: 'local'
Jun-08 14:44:06.598 [main] DEBUG nextflow.executor.Executor - Initializing executor: local
Jun-08 14:44:06.599 [main] DEBUG nextflow.Session - >>> barrier register (process: bwa)
Jun-08 14:44:06.607 [main] DEBUG nextflow.processor.TaskProcessor - Creating operator > bwa -- maxForks: 64
Jun-08 14:44:06.628 [main] DEBUG nextflow.processor.ProcessFactory - << taskConfig executor: local
Jun-08 14:44:06.629 [main] DEBUG nextflow.processor.ProcessFactory - >> processorType: 'local'
Jun-08 14:44:06.629 [main] DEBUG nextflow.executor.Executor - Initializing executor: local
Jun-08 14:44:06.631 [main] DEBUG nextflow.Session - >>> barrier register (process: samtools)
Jun-08 14:44:06.646 [main] DEBUG nextflow.processor.TaskProcessor - Creating operator > samtools -- maxForks: 64
Jun-08 14:44:06.653 [main] DEBUG nextflow.processor.ProcessFactory - << taskConfig executor: local
Jun-08 14:44:06.653 [main] DEBUG nextflow.processor.ProcessFactory - >> processorType: 'local'
Jun-08 14:44:06.653 [main] DEBUG nextflow.executor.Executor - Initializing executor: local
Jun-08 14:44:06.654 [main] DEBUG nextflow.Session - >>> barrier register (process: picard)
Jun-08 14:44:06.666 [main] DEBUG nextflow.processor.TaskProcessor - Creating operator > picard -- maxForks: 64
Jun-08 14:44:06.703 [main] DEBUG nextflow.processor.ProcessFactory - << taskConfig executor: local
Jun-08 14:44:06.703 [main] DEBUG nextflow.processor.ProcessFactory - >> processorType: 'local'
Jun-08 14:44:06.703 [main] DEBUG nextflow.executor.Executor - Initializing executor: local
Jun-08 14:44:06.704 [main] DEBUG nextflow.Session - >>> barrier register (process: countstat)
Jun-08 14:44:06.712 [main] DEBUG nextflow.processor.TaskProcessor - Creating operator > countstat -- maxForks: 64
Jun-08 14:44:06.717 [main] DEBUG nextflow.processor.ProcessFactory - << taskConfig executor: local
Jun-08 14:44:06.717 [main] DEBUG nextflow.processor.ProcessFactory - >> processorType: 'local'
Jun-08 14:44:06.717 [main] DEBUG nextflow.executor.Executor - Initializing executor: local
Jun-08 14:44:06.718 [main] DEBUG nextflow.Session - >>> barrier register (process: phantompeakqualtools)
Jun-08 14:44:06.725 [main] DEBUG nextflow.processor.TaskProcessor - Creating operator > phantompeakqualtools -- maxForks: 64
Jun-08 14:44:06.739 [main] DEBUG nextflow.processor.ProcessFactory - << taskConfig executor: local
Jun-08 14:44:06.739 [main] DEBUG nextflow.processor.ProcessFactory - >> processorType: 'local'
Jun-08 14:44:06.739 [main] DEBUG nextflow.executor.Executor - Initializing executor: local
Jun-08 14:44:06.741 [main] DEBUG nextflow.Session - >>> barrier register (process: calculateNSCRSC)
Jun-08 14:44:06.749 [main] DEBUG nextflow.processor.TaskProcessor - Creating operator > calculateNSCRSC -- maxForks: 64
Jun-08 14:44:06.755 [main] DEBUG nextflow.processor.ProcessFactory - << taskConfig executor: local
Jun-08 14:44:06.755 [main] DEBUG nextflow.processor.ProcessFactory - >> processorType: 'local'
Jun-08 14:44:06.755 [main] DEBUG nextflow.executor.Executor - Initializing executor: local
Jun-08 14:44:06.756 [main] DEBUG nextflow.Session - >>> barrier register (process: deepTools)
Jun-08 14:44:06.763 [main] DEBUG nextflow.processor.TaskProcessor - Creating operator > deepTools -- maxForks: 64
Jun-08 14:44:06.769 [main] DEBUG nextflow.processor.ProcessFactory - << taskConfig executor: local
Jun-08 14:44:06.770 [main] DEBUG nextflow.processor.ProcessFactory - >> processorType: 'local'
Jun-08 14:44:06.770 [main] DEBUG nextflow.executor.Executor - Initializing executor: local
Jun-08 14:44:06.770 [main] DEBUG nextflow.Session - >>> barrier register (process: ngsplot)
Jun-08 14:44:06.775 [main] DEBUG nextflow.processor.TaskProcessor - Creating operator > ngsplot -- maxForks: 64
Jun-08 14:44:06.779 [main] DEBUG nextflow.processor.ProcessFactory - << taskConfig executor: local
Jun-08 14:44:06.779 [main] DEBUG nextflow.processor.ProcessFactory - >> processorType: 'local'
Jun-08 14:44:06.779 [main] DEBUG nextflow.executor.Executor - Initializing executor: local
Jun-08 14:44:06.892 [main] DEBUG nextflow.Session - >>> barrier register (process: macs)
Jun-08 14:44:06.905 [main] DEBUG nextflow.processor.TaskProcessor - Creating operator > macs -- maxForks: 64
Jun-08 14:44:06.916 [main] DEBUG nextflow.processor.ProcessFactory - << taskConfig executor: local
Jun-08 14:44:06.916 [main] DEBUG nextflow.processor.ProcessFactory - >> processorType: 'local'
Jun-08 14:44:06.917 [main] DEBUG nextflow.executor.Executor - Initializing executor: local
Jun-08 14:44:06.918 [main] DEBUG nextflow.Session - >>> barrier register (process: multiqc)
Jun-08 14:44:06.927 [main] DEBUG nextflow.processor.TaskProcessor - Creating operator > multiqc -- maxForks: 64
Jun-08 14:44:06.929 [main] DEBUG nextflow.script.ScriptRunner - > Await termination
Jun-08 14:44:06.929 [main] DEBUG nextflow.Session - Session await
Jun-08 14:44:07.178 [Actor Thread 10] INFO nextflow.Session - [70/d32341] Submitted process > fastqc (SRR1029883)
Jun-08 14:44:07.202 [Actor Thread 6] INFO nextflow.Session - [a7/088a0a] Submitted process > fastqc (SRR1029879)
Jun-08 14:44:07.203 [Actor Thread 8] INFO nextflow.Session - [a0/28ff1b] Submitted process > fastqc (SRR1029881)
Jun-08 14:44:07.204 [Actor Thread 19] INFO nextflow.Session - [79/364e23] Submitted process > fastqc (SRR1029887)
Jun-08 14:44:07.205 [Actor Thread 3] INFO nextflow.Session - [65/ff76fb] Submitted process > fastqc (SRR1029878)
Jun-08 14:44:07.235 [Actor Thread 2] INFO nextflow.Session - [16/9bfa06] Submitted process > fastqc (SRR1029876)
Jun-08 14:44:07.252 [Actor Thread 15] INFO nextflow.Session - [10/bcca05] Submitted process > fastqc (SRR1029885)
Jun-08 14:44:07.253 [Actor Thread 7] INFO nextflow.Session - [d4/d037f3] Submitted process > fastqc (SRR1029880)
Jun-08 14:44:07.254 [Actor Thread 21] INFO nextflow.Session - [49/14f458] Submitted process > fastqc (SRR1029888)
Jun-08 14:44:07.255 [Actor Thread 17] INFO nextflow.Session - [84/e15b7b] Submitted process > fastqc (SRR1029886)
Jun-08 14:44:07.279 [Actor Thread 12] INFO nextflow.Session - [5d/ef5a8c] Submitted process > fastqc (SRR1029884)
Jun-08 14:44:07.279 [Actor Thread 4] INFO nextflow.Session - [77/4929ce] Submitted process > fastqc (SRR1029877)
Jun-08 14:44:07.280 [Actor Thread 23] INFO nextflow.Session - [c0/51b019] Submitted process > fastqc (SRR1029889)
Jun-08 14:44:07.281 [Actor Thread 29] INFO nextflow.Session - [08/349928] Submitted process > trim_galore (SRR1029886)
Jun-08 14:44:07.282 [Actor Thread 25] INFO nextflow.Session - [f3/0eb386] Submitted process > trim_galore (SRR1029884)
Jun-08 14:44:07.307 [Actor Thread 9] INFO nextflow.Session - [a3/ed353d] Submitted process > fastqc (SRR1029882)
Jun-08 14:44:07.310 [Actor Thread 28] INFO nextflow.Session - [ab/4d0d7a] Submitted process > trim_galore (SRR1029885)
Jun-08 14:44:07.342 [Actor Thread 26] INFO nextflow.Session - [0d/0367e1] Submitted process > trim_galore (SRR1029882)
Jun-08 14:44:07.362 [Actor Thread 1] DEBUG nextflow.processor.TaskProcessor - <fastqc> After stop
Jun-08 14:44:07.440 [Actor Thread 32] INFO nextflow.Session - [3c/628107] Submitted process > trim_galore (SRR1029889)
Jun-08 14:44:07.464 [Actor Thread 16] INFO nextflow.Session - [7c/257023] Submitted process > trim_galore (SRR1029877)
Jun-08 14:44:07.466 [Actor Thread 22] INFO nextflow.Session - [f9/d9551a] Submitted process > trim_galore (SRR1029880)
Jun-08 14:45:44.690 [Actor Thread 30] INFO nextflow.Session - [fa/cd80bd] Submitted process > trim_galore (SRR1029887)
Jun-08 14:45:51.804 [Actor Thread 18] INFO nextflow.Session - [65/a3aff1] Submitted process > trim_galore (SRR1029878)
Jun-08 14:46:04.052 [Actor Thread 24] INFO nextflow.Session - [cb/c95641] Submitted process > trim_galore (SRR1029881)
Jun-08 14:46:07.332 [Actor Thread 13] INFO nextflow.Session - [6a/ebf202] Submitted process > trim_galore (SRR1029876)
Jun-08 14:46:10.474 [Actor Thread 20] INFO nextflow.Session - [69/f94f3e] Submitted process > trim_galore (SRR1029879)
Jun-08 14:46:17.848 [Actor Thread 31] INFO nextflow.Session - [b0/0c5cd7] Submitted process > trim_galore (SRR1029888)
Jun-08 14:46:21.351 [Actor Thread 27] INFO nextflow.Session - [c2/5a5071] Submitted process > trim_galore (SRR1029883)
Jun-08 14:46:21.351 [Actor Thread 11] DEBUG nextflow.processor.TaskProcessor - <trim_galore> After stop
Jun-08 14:49:06.659 [Thread-4] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 15 -- first: TaskHandler[id: 11; name: fastqc (SRR1029885); status: RUNNING; exit: -; workDir: /mnt/date/biodata/data/cellChIP/heart/work/10/bcca05f64448dd8cc9451737dd4f52]
Jun-08 14:50:27.240 [Actor Thread 39] DEBUG nextflow.processor.TaskProcessor - <fastqc> Sending poison pills and terminating process
Jun-08 14:50:27.245 [Actor Thread 39] DEBUG nextflow.Session - <<< barrier arrive (process: fastqc)
Jun-08 14:52:21.356 [Actor Thread 43] INFO nextflow.Session - [a3/d41b33] Submitted process > bwa (SRR1029877)
Jun-08 14:53:25.105 [Actor Thread 46] INFO nextflow.Session - [d6/ba338c] Submitted process > bwa (SRR1029886)
Jun-08 14:53:31.916 [Actor Thread 45] INFO nextflow.Session - [16/d2aca9] Submitted process > bwa (SRR1029884)
Jun-08 14:54:01.685 [Actor Thread 49] INFO nextflow.Session - [d0/9aa776] Submitted process > bwa (SRR1029889)
Jun-08 14:54:06.499 [Actor Thread 48] INFO nextflow.Session - [d8/c97d73] Submitted process > bwa (SRR1029876)
Jun-08 14:54:06.756 [Thread-4] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 14 -- first: TaskHandler[id: 24; name: trim_galore (SRR1029885); status: RUNNING; exit: -; workDir: /mnt/date/biodata/data/cellChIP/heart/work/ab/4d0d7a11574904e18ddd849fa705c1]
Jun-08 14:55:07.840 [Actor Thread 52] INFO nextflow.Session - [51/722bc2] Submitted process > bwa (SRR1029881)
Jun-08 14:56:11.801 [Actor Thread 54] INFO nextflow.Session - [a2/cdb0bc] Submitted process > bwa (SRR1029888)
Jun-08 14:56:42.878 [Actor Thread 57] INFO nextflow.Session - [5d/8d5821] Submitted process > bwa (SRR1029879)
Jun-08 14:58:15.471 [Actor Thread 60] INFO nextflow.Session - [c6/d717d2] Submitted process > bwa (SRR1029887)
Jun-08 14:58:30.396 [Actor Thread 63] INFO nextflow.Session - [27/721071] Submitted process > bwa (SRR1029878)
Jun-08 14:58:40.462 [Actor Thread 66] INFO nextflow.Session - [9e/c3922c] Submitted process > bwa (SRR1029880)
Jun-08 14:59:06.857 [Thread-4] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 14 -- first: TaskHandler[id: 24; name: trim_galore (SRR1029885); status: RUNNING; exit: -; workDir: /mnt/date/biodata/data/cellChIP/heart/work/ab/4d0d7a11574904e18ddd849fa705c1]
Jun-08 15:00:52.654 [Actor Thread 68] INFO nextflow.Session - [02/f85463] Submitted process > bwa (SRR1029882)
Jun-08 15:04:06.929 [Thread-4] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 14 -- first: TaskHandler[id: 24; name: trim_galore (SRR1029885); status: RUNNING; exit: -; workDir: /mnt/date/biodata/data/cellChIP/heart/work/ab/4d0d7a11574904e18ddd849fa705c1]
Jun-08 15:05:39.781 [Actor Thread 71] INFO nextflow.Session - [7e/89c8c5] Submitted process > bwa (SRR1029883)
Jun-08 15:09:07.027 [Thread-4] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 14 -- first: TaskHandler[id: 24; name: trim_galore (SRR1029885); status: RUNNING; exit: -; workDir: /mnt/date/biodata/data/cellChIP/heart/work/ab/4d0d7a11574904e18ddd849fa705c1]
Jun-08 15:13:28.526 [Actor Thread 72] DEBUG nextflow.processor.TaskProcessor - <trim_galore> Sending poison pills and terminating process
Jun-08 15:13:28.527 [Actor Thread 71] DEBUG nextflow.processor.TaskProcessor - <bwa> Poison pill arrived
Jun-08 15:13:28.527 [Actor Thread 72] DEBUG nextflow.Session - <<< barrier arrive (process: trim_galore)
Jun-08 15:13:28.529 [Actor Thread 77] DEBUG nextflow.processor.StateObj - <bwa> State before poison: StateObj[submitted: 14; completed: 0; poisoned: false ]
Jun-08 15:13:28.700 [Actor Thread 74] INFO nextflow.Session - [f1/d71c95] Submitted process > bwa (SRR1029885)
Jun-08 15:13:28.701 [Actor Thread 71] DEBUG nextflow.processor.TaskProcessor - <bwa> After stop
Jun-08 15:14:07.071 [Thread-4] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 14 -- first: TaskHandler[id: 29; name: bwa (SRR1029877); status: RUNNING; exit: -; workDir: /mnt/date/biodata/data/cellChIP/heart/work/a3/d41b33aae2e8f5772807af393961bd]
Jun-08 15:19:07.097 [Thread-4] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 14 -- first: TaskHandler[id: 29; name: bwa (SRR1029877); status: RUNNING; exit: -; workDir: /mnt/date/biodata/data/cellChIP/heart/work/a3/d41b33aae2e8f5772807af393961bd]
Jun-08 15:24:07.102 [Thread-4] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 14 -- first: TaskHandler[id: 29; name: bwa (SRR1029877); status: RUNNING; exit: -; workDir: /mnt/date/biodata/data/cellChIP/heart/work/a3/d41b33aae2e8f5772807af393961bd]
Jun-08 15:29:07.101 [Thread-4] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 14 -- first: TaskHandler[id: 29; name: bwa (SRR1029877); status: RUNNING; exit: -; workDir: /mnt/date/biodata/data/cellChIP/heart/work/a3/d41b33aae2e8f5772807af393961bd]
Jun-08 15:34:07.120 [Thread-4] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 14 -- first: TaskHandler[id: 29; name: bwa (SRR1029877); status: RUNNING; exit: -; workDir: /mnt/date/biodata/data/cellChIP/heart/work/a3/d41b33aae2e8f5772807af393961bd]
Jun-08 15:39:07.118 [Thread-4] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 14 -- first: TaskHandler[id: 29; name: bwa (SRR1029877); status: RUNNING; exit: -; workDir: /mnt/date/biodata/data/cellChIP/heart/work/a3/d41b33aae2e8f5772807af393961bd]
Jun-08 15:44:07.181 [Thread-4] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 14 -- first: TaskHandler[id: 29; name: bwa (SRR1029877); status: RUNNING; exit: -; workDir: /mnt/date/biodata/data/cellChIP/heart/work/a3/d41b33aae2e8f5772807af393961bd]
Jun-08 15:49:07.213 [Thread-4] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 14 -- first: TaskHandler[id: 29; name: bwa (SRR1029877); status: RUNNING; exit: -; workDir: /mnt/date/biodata/data/cellChIP/heart/work/a3/d41b33aae2e8f5772807af393961bd]
Jun-08 15:54:07.226 [Thread-4] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 14 -- first: TaskHandler[id: 29; name: bwa (SRR1029877); status: RUNNING; exit: -; workDir: /mnt/date/biodata/data/cellChIP/heart/work/a3/d41b33aae2e8f5772807af393961bd]
Jun-08 15:59:07.242 [Thread-4] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 14 -- first: TaskHandler[id: 29; name: bwa (SRR1029877); status: RUNNING; exit: -; workDir: /mnt/date/biodata/data/cellChIP/heart/work/a3/d41b33aae2e8f5772807af393961bd]
Jun-08 16:04:07.302 [Thread-4] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 14 -- first: TaskHandler[id: 29; name: bwa (SRR1029877); status: RUNNING; exit: -; workDir: /mnt/date/biodata/data/cellChIP/heart/work/a3/d41b33aae2e8f5772807af393961bd]
Jun-08 16:06:39.654 [Actor Thread 79] INFO nextflow.Session - [b6/45c4f0] Submitted process > samtools (SRR1029884)
Jun-08 16:07:18.296 [Actor Thread 82] INFO nextflow.Session - [44/8d899b] Submitted process > samtools (SRR1029876)
Jun-08 16:09:07.319 [Thread-4] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 14 -- first: TaskHandler[id: 29; name: bwa (SRR1029877); status: RUNNING; exit: -; workDir: /mnt/date/biodata/data/cellChIP/heart/work/a3/d41b33aae2e8f5772807af393961bd]
Jun-08 16:13:19.534 [Actor Thread 85] INFO nextflow.Session - [cd/baf6cf] Submitted process > samtools (SRR1029888)
Jun-08 16:14:07.340 [Thread-4] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 14 -- first: TaskHandler[id: 29; name: bwa (SRR1029877); status: RUNNING; exit: -; workDir: /mnt/date/biodata/data/cellChIP/heart/work/a3/d41b33aae2e8f5772807af393961bd]
Jun-08 16:16:26.939 [Actor Thread 88] INFO nextflow.Session - [87/62247c] Submitted process > picard (SRR1029876)
Jun-08 16:16:29.782 [Thread-4] ERROR nextflow.processor.TaskProcessor - Error executing process > 'picard (SRR1029876)'
Caused by:
Process `picard (SRR1029876)` terminated with an error exit status (1)
Command executed:
set -o pipefail
java -Xmx0g -jar $PICARD_HOME/picard.jar MarkDuplicates \
INPUT=SRR1029876.sorted.bam \
OUTPUT=SRR1029876.dedup.bam \
ASSUME_SORTED=true \
REMOVE_DUPLICATES=true \
METRICS_FILE=SRR1029876.picardDupMetrics.txt \
VALIDATION_STRINGENCY=LENIENT \
PROGRAM_RECORD_ID='null'
samtools sort SRR1029876.dedup.bam -o SRR1029876.dedup.sorted.bam
samtools index SRR1029876.dedup.sorted.bam
bedtools bamtobed -i SRR1029876.dedup.sorted.bam | sort -k 1,1 -k 2,2n -k 3,3n -k 6,6 > SRR1029876.dedup.sorted.bed
Command exit status:
1
Command output:
(empty)
Command error:
Invalid maximum heap size: -Xmx0g
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
Work dir:
/mnt/date/biodata/data/cellChIP/heart/work/87/62247cc953c0ab6ce8db7f47800c43
Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`
Jun-08 16:16:29.785 [Thread-4] DEBUG nextflow.Session - Session aborted -- Cause: Process `picard (SRR1029876)` terminated with an error exit status (1)
Jun-08 16:16:29.789 [main] DEBUG nextflow.Session - Session await > all process finished
Jun-08 16:16:29.789 [Actor Thread 88] DEBUG nextflow.processor.TaskProcessor - <samtools> After stop
Jun-08 16:16:29.789 [main] DEBUG nextflow.Session - Session await > all barriers passed
Jun-08 16:16:29.789 [Actor Thread 85] DEBUG nextflow.processor.TaskProcessor - <picard> After stop
Jun-08 16:16:29.791 [Actor Thread 88] DEBUG nextflow.processor.TaskProcessor - <calculateNSCRSC> After stop
Jun-08 16:16:29.791 [Actor Thread 85] DEBUG nextflow.processor.TaskProcessor - <phantompeakqualtools> After stop
Jun-08 16:16:29.791 [Actor Thread 92] DEBUG nextflow.processor.TaskProcessor - <countstat> After stop
Jun-08 16:16:29.792 [Actor Thread 92] DEBUG nextflow.processor.TaskProcessor - <deepTools> After stop
Jun-08 16:16:29.792 [Actor Thread 88] DEBUG nextflow.processor.TaskProcessor - <macs> After stop
Jun-08 16:16:29.792 [Actor Thread 91] DEBUG nextflow.processor.TaskProcessor - <ngsplot> After stop
Jun-08 16:16:29.792 [Thread-4] DEBUG n.processor.TaskPollingMonitor - <<< barrier arrives (monitor: local)
Jun-08 16:16:29.792 [Actor Thread 87] DEBUG nextflow.processor.TaskProcessor - <multiqc> After stop
Jun-08 16:16:29.801 [main] WARN n.processor.TaskPollingMonitor - Killing pending tasks (13)
Jun-08 16:16:30.035 [main] DEBUG nextflow.CacheDB - Closing CacheDB done
Jun-08 16:16:30.081 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye
It's generally considered best practice to remove multimapping reads for ChIP-seq analysis (eg. the classic bowtie -m 1 for bowtie1). Most modern mappers, including bwa mem that we use here, don't have an equivalent option. However, samtools view can be used to filter out reads with low mapping scores. See biostars posts here and here (and many other places I'm sure).
It would be good to add in an extra filter step like this, possibly making it optional.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.