Comments (5)
I'll update the doc/Internals.md
with regards to that + recommendations. Short answer - pls avoid declaring variables which don't depend on the outputs of any task/workflow in the body. Allocate those things to the top level inputs - this will eliminate unwanted dependencies. I don't envision any immediate changes to the current logic of dxCompiler with regards to that so will probably close the issue shortly. Otherwise, will update here
from dxcompiler.
I think this is the issue discussed in dnanexus/dxWDL#162 where wdl expression evaluations in a WDL workflow gets merged into a fragment for the first following applet.
Referring to the following basic test:
version 1.0
task t1 {
input {
String s
}
command <<<
echo t1 >s
>>>
output {
File out = "s"
}
}
task t2 {
input {
Array[String] s
}
command <<<
echo t1 >s
>>>
output {
File out = "s"
}
}
task dummy {
input {
}
command <<<
>>>
output {
}
}
workflow tinytest3 {
Array[String] chr_num = ["1"]
Array[String] chrs = prefix("chr", chr_num)
call dummy
call t2 {input: s = chrs}
scatter (chr in chrs) {
call t1 { input: s = chr }
}
output {
Array[File] out1 = t1.out
File out2 = t2.out
}
}
With dxCompiler 2.10.0
- The following makes a dnanexus workflow where t2 depends on scatter
workflow tinytest {
Array[String] chr_num = ["1"]
Array[String] chrs = prefix("chr", chr_num)
scatter (chr in chrs) {
call t1 { input: s = chr }
}
call t2 {input: s = chrs}
- The following makes a dnanexus workflow where scatter depends on t2
workflow tinytest2 {
Array[String] chr_num = ["1"]
Array[String] chrs = prefix("chr", chr_num)
call t2 {input: s = chrs}
scatter (chr in chrs) {
call t1 { input: s = chr }
}
- The following makes a dnanexus workflow where scatter and t2 both depend on dummy, but not on each other.
workflow tinytest3 {
Array[String] chr_num = ["1"]
Array[String] chrs = prefix("chr", chr_num)
call dummy
call t2 {input: s = chrs}
scatter (chr in chrs) {
call t1 { input: s = chr }
}
The dependency can be seen by examining the inputs to each stage. E.g. for tinytest3:
$ dx describe workflow-GGGQzJ00Bvbj46PP4gjFyXP9 --json | jq '.stages|.[]|{id: .id,name: .name, input: .input}'
{
"id": "stage-common",
"name": "common",
"input": {}
}
{
"id": "stage-1",
"name": "frag dummy",
"input": {}
}
{
"id": "stage-2",
"name": "t2",
"input": {
"s": {
"$dnanexus_link": {
"outputField": "chrs",
"stage": "stage-1"
}
}
}
}
{
"id": "stage-4",
"name": "scatter (chr in chrs)",
"input": {
"chrs": {
"$dnanexus_link": {
"outputField": "chrs",
"stage": "stage-1"
}
}
}
}
{
"id": "stage-outputs",
"name": "outputs",
"input": {
"t1___out": {
"$dnanexus_link": {
"outputField": "t1___out",
"stage": "stage-4"
}
},
"t2___out": {
"$dnanexus_link": {
"outputField": "out",
"stage": "stage-2"
}
}
}
}
from dxcompiler.
So it's a known behavior that reduces the number of jobs at the expense of potential decrease in concurrency of execution, with a known workaround (see tinytest3).
We can consider making identification of such situations easier or changing the behavior of dxCompiler to place expression evaluation into a separate job when multiple tasks depend on the results of expression evaluation.
from dxcompiler.
@Ben-Habermeyer is there a reason to declare chr_num
and chrs
in the body of the WF, instead of declaring them at the workflow level inputs (with defaults)? This way muse
should depend on a common
stage and not on the scatter.
from dxcompiler.
Hi @Gvaihir thanks for the reply. Yes that is the approach we went with. I think the user was confused why placing the variables in the body of the WF resulted in this behavior
from dxcompiler.
Related Issues (20)
- dxcompiler can not process Pair object HOT 2
- Array[String?] compiles incorrectly to "(hash" and "(array:file)"
- difficulty setting a default docker image HOT 1
- Downloading lots of small files
- Apollo - Spark and cohort files
- Task with stubbed applet as runtime does not obey instance type HOT 5
- slow compilation "unable to parse field default value null" HOT 12
- InvalidInput: i/o value reads needs to be given using DNAnexus links
- Preserve folder hierarchies on the platform HOT 10
- Scatter limit override HOT 3
- dxCompiler possibly incorrect localizes input file expressions HOT 11
- wdl 1.1 `sep()` vs command block `~{sep}` return different values HOT 2
- [error] You must be logged in to compile using mode All HOT 5
- WDL tasks in docker containers are missing DX environment variables HOT 1
- UK Biobank RAP path with spaces in it HOT 3
- CWL workflow complied to include a subworkflow with an `outputs` applet fails to run successfully HOT 1
- [error] org.fusesource.scalate.TemplateException: HOT 1
- java.lang.IllegalArgumentException: Invalid HTTP host: 127.0.0.1:7890/
- Failed to generate Array[File] with optional input
- dxCompiler doesn't seem to work with recent versions of java
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dxcompiler.