Comments (12)
@Ben-Habermeyer, could you please create a DNAnexus project with your build logs, WDL and any other relevant info that might help repro the problem and share it with org-support.
from dxcompiler.
Now it is String = "default" so we don't use optional anymore.
But that's exactly what I'm pointing to. Your previous app had this parameter as an optional. Now you change the type. What happens with the compile time if you keep it optional but still give the default value? It does not go against the conventions of WDL specs. And if you reuse the app the argument types should be the same (i.e. optionals have to remain optionals), otherwise there's a mismatch. Mismatch != reuse. So how will the compile time of special case callers
will be if the argument looks like this?
String? caller_to_emulate = "YOUR_DEFAULT_VALUE"
Second and more important for the comparison purpose - your workflows are very much not of the similar size. Second one is 2-3 layers deeper and has ~2 times more executables. You can verify it by
java -jar dxCompiler.jar describe workflow-XXXX -pretty
Those are not really comparable to benchmark the compile time. What you want to compare is String? caller_to_emulate
vs String caller_to_emulate
in the same workflow structure.
from dxcompiler.
Could you tell me where this warning comes from / some possible solutions to mitigate it?
from dxcompiler.
@Ben-Habermeyer Does this project-G0vbVFj0046589VQPz0kpgFk
contain all the relevant info? I am requesting the access now.
EDIT: could not locate the logs or relevant fixtures in that project.
from dxcompiler.
Hi @Gvaihir and @emiloslavsky thanks for the reply. I shared the project dx invite org-support myelin_parsnips_workflows VIEW
with the zip file of the wdl (in a rendered
directory) is the first file and the build log is the second file
›› dx find data --link workflow-GK2Q9vQ0046Bj77v7xbZk3Qg --all-projects 4:53 Dec19
closed 2022-11-28 13:59:04 69.62 MB /workflows/parsnips_project/1d30b674/5f64837b/parsnips_project-1d30b674-5f64837b.wdl.zip (project-G0vbVFj0046589VQPz0kpgFk:file-GK2Q9x00046PfVjf60876Y39)
closed 2022-11-28 13:59:04 453.82 KB /workflows/parsnips_project/1d30b674/5f64837b/parsnips_project-1d30b674-5f64837b.log (project-G0vbVFj0046589VQPz0kpgFk:file-GK2Q9xj0046J87j987XV65XZ)
From the build log, the command used to compile is
docker run -e DX_SECURITY_CONTEXT \
-v /rendered/rendered_wdl:/rendered/rendered_wdl \
-w /rendered/rendered_wdl \
dnanexus/dxcompiler:2.10.4 compile /rendered/rendered_wdl/sgnips_iseq_batch.wdl \
-extras /rendered_wdl/extras.json \
-defaults /rendered/rendered_wdl/sgnips_iseq_batch_defaults.json \
-archive \
-verbose \
-destination myelin_parsnips_workflows:/workflows/parsnips_project/1d30b674/5f64837b -projectWideReuse
from dxcompiler.
Hi @Ben-Habermeyer!
- The log in file
file-GK2Q9xj0046J87j987XV65XZ
does not contain lines from your original post. Is this a relevant log? - Your note that it "small changes take ~10min longer than they should I estimate" - what is the estimate process? What did you compare?
- Does
-projectWideReuse
withString? = None
compile as usual or slower? Do you see the respective applets reused when usingString = "DEFAULT_STR"
or being rebuilt? What if you keep that input optional (String? = "DEFAULT_STR"
)?
I'm not building this WF anywhere for obvious reasons. Also I don't think a dummy WF will be a good representative. So we need as much info from you as you can give. Or give me permissions to build this workflow on platform
from dxcompiler.
Thanks @Gvaihir sorry if I sent the wrong files I will check and resend different ones today.
for 2) My hunch is that the project wide reuse looking for one particular applet (which previously had default String? = None is problematic and takes longer than it should. I can give more pointed examples today.
for 3) The current WDL does not have String?=None but the applet used to, but we still see these warnings in the logs I believe because of the project wide reuse. Now it is String = "default" so we don't use optional anymore.
Thanks for patience with my error; I'll send some more examples today that might narrow it down a bit more.
from dxcompiler.
ok @Gvaihir I have some more data for you. I compiled 2 workflows of similar size in this same project with reuse. The first was workflow-GKVXjjj00462vX2G6JGB0v8v
(sgnips_iseq_batch.wdl
) and the second one with problematic task search is workflow-GKVY2X000463Vp0k30Kgv7Z3
(special_case_callers.wdl
). The WDL zip should be the same so you can just download one; I just compiled different sub-workflow WDL files.
iseq
›› dx find data --link workflow-GKVXjjj00462vX2G6JGB0v8v --all-projects
closed 2022-12-21 10:55:21 69.62 MB /workflows/parsnips_project/ae30d078/d1a4d4f5/parsnips_project-ae30d078-d1a4d4f5.wdl.zip (project-G0vbVFj0046589VQPz0kpgFk:file-GKVXjk80046Fg0GYFP6Qfz73)
closed 2022-12-21 10:55:20 458.09 KB /workflows/parsnips_project/ae30d078/d1a4d4f5/parsnips_project-ae30d078-d1a4d4f5.log (project-G0vbVFj0046589VQPz0kpgFk:file-GKVXjkj0046551k139kY57J1)
special case callers
›› dx find data --link workflow-GKVY2X000463Vp0k30Kgv7Z3 --all-projects
closed 2022-12-21 11:16:13 69.62 MB /workflows/parsnips_project/ae30d078/e437b83d/parsnips_project-ae30d078-e437b83d.wdl.zip (project-G0vbVFj0046589VQPz0kpgFk:file-GKVY2XQ00465b9Zx4kb5YKYZ)
closed 2022-12-21 11:16:13 307.49 KB /workflows/parsnips_project/ae30d078/e437b83d/parsnips_project-ae30d078-e437b83d.log (project-G0vbVFj0046589VQPz0kpgFk:file-GKVY2Y000464b7BGJfJ8zXJ2)
the iseq build took 2:49 an the special case callers build took 8:14 on the agent. Most of the time is querying for executables, and I see the log lines in the special case callers. Again, I'm not sure if this is a red herring, but we have noticed the compilation taking longer in this specific case
Querying for executables in project DxProject(project-G0vbVFj0046589VQPz0kpgFk)
[warning] unable to parse field caller_to_emulate default value null
[warning] unable to parse field caller_to_emulate default value null
[warning] unable to parse field caller_to_emulate default value null
[warning] unable to parse field caller_to_emulate default value null
[warning] unable to parse field caller_to_emulate default value null
[warning] unable to parse field caller_to_emulate default value null
[warning] unable to parse field caller_to_emulate default value null
[warning] unable to parse field caller_to_emulate default value null
[warning] unable to parse field caller_to_emulate default value null
[warning] unable to parse field caller_to_emulate default value null
[warning] unable to parse field caller_to_emulate default value null
[warning] unable to parse field caller_to_emulate default value null
[warning] unable to parse field caller_to_emulate default value null
[warning] unable to parse field caller_to_emulate default value null
[warning] unable to parse field caller_to_emulate default value null
[warning] unable to parse field caller_to_emulate default value null
[warning] unable to parse field caller_to_emulate default value null
[warning] unable to parse field caller_to_emulate default value null
[warning] unable to parse field caller_to_emulate default value null
[warning] unable to parse field caller_to_emulate default value null
[warning] unable to parse field caller_to_emulate default value null
[warning] unable to parse field caller_to_emulate default value null
[warning] unable to parse field caller_to_emulate default value null
Found 3408 executables in project-G0vbVFj0046589VQPz0kpgFk:None
Found 3408 executables
from dxcompiler.
@Ben-Habermeyer sorry - I was away. I'll check in the coming days
from dxcompiler.
thanks @Gvaihir I will compare those two and get back to you
from dxcompiler.
@Ben-Habermeyer is this still an issue, or should I close it?
from dxcompiler.
Not reproducing
from dxcompiler.
Related Issues (20)
- dxcompiler can not process Pair object HOT 2
- Array[String?] compiles incorrectly to "(hash" and "(array:file)"
- difficulty setting a default docker image HOT 1
- Downloading lots of small files
- Apollo - Spark and cohort files
- Task with stubbed applet as runtime does not obey instance type HOT 5
- InvalidInput: i/o value reads needs to be given using DNAnexus links
- Preserve folder hierarchies on the platform HOT 10
- Scatter limit override HOT 3
- dxCompiler possibly incorrect localizes input file expressions HOT 11
- wdl 1.1 `sep()` vs command block `~{sep}` return different values HOT 2
- [error] You must be logged in to compile using mode All HOT 5
- WDL tasks in docker containers are missing DX environment variables HOT 1
- UK Biobank RAP path with spaces in it HOT 3
- CWL workflow complied to include a subworkflow with an `outputs` applet fails to run successfully HOT 1
- [error] org.fusesource.scalate.TemplateException: HOT 1
- java.lang.IllegalArgumentException: Invalid HTTP host: 127.0.0.1:7890/
- Failed to generate Array[File] with optional input
- dxCompiler doesn't seem to work with recent versions of java
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dxcompiler.