slow compilation "unable to parse field default value null" about dxcompiler HOT 12 CLOSED

@Ben-Habermeyer, could you please create a DNAnexus project with your build logs, WDL and any other relevant info that might help repro the problem and share it with org-support.

from dxcompiler.

@Ben-Habermeyer

Now it is String = "default" so we don't use optional anymore.

But that's exactly what I'm pointing to. Your previous app had this parameter as an optional. Now you change the type. What happens with the compile time if you keep it optional but still give the default value? It does not go against the conventions of WDL specs. And if you reuse the app the argument types should be the same (i.e. optionals have to remain optionals), otherwise there's a mismatch. Mismatch != reuse. So how will the compile time of special case callers will be if the argument looks like this?

String? caller_to_emulate = "YOUR_DEFAULT_VALUE"

Second and more important for the comparison purpose - your workflows are very much not of the similar size. Second one is 2-3 layers deeper and has ~2 times more executables. You can verify it by

java -jar dxCompiler.jar describe workflow-XXXX -pretty

Those are not really comparable to benchmark the compile time. What you want to compare is String? caller_to_emulate vs String caller_to_emulate in the same workflow structure.

from dxcompiler.

Could you tell me where this warning comes from / some possible solutions to mitigate it?

from dxcompiler.

@Ben-Habermeyer Does this project-G0vbVFj0046589VQPz0kpgFk contain all the relevant info? I am requesting the access now.
EDIT: could not locate the logs or relevant fixtures in that project.

from dxcompiler.

Hi @Gvaihir and @emiloslavsky thanks for the reply. I shared the project dx invite org-support myelin_parsnips_workflows VIEW with the zip file of the wdl (in a rendered directory) is the first file and the build log is the second file

›› dx find data --link workflow-GK2Q9vQ0046Bj77v7xbZk3Qg --all-projects                                                        4:53 Dec19
closed  2022-11-28 13:59:04 69.62 MB  /workflows/parsnips_project/1d30b674/5f64837b/parsnips_project-1d30b674-5f64837b.wdl.zip (project-G0vbVFj0046589VQPz0kpgFk:file-GK2Q9x00046PfVjf60876Y39)
closed  2022-11-28 13:59:04 453.82 KB /workflows/parsnips_project/1d30b674/5f64837b/parsnips_project-1d30b674-5f64837b.log (project-G0vbVFj0046589VQPz0kpgFk:file-GK2Q9xj0046J87j987XV65XZ)

From the build log, the command used to compile is

docker run -e DX_SECURITY_CONTEXT  \
  -v /rendered/rendered_wdl:/rendered/rendered_wdl \
  -w /rendered/rendered_wdl \
  dnanexus/dxcompiler:2.10.4 compile /rendered/rendered_wdl/sgnips_iseq_batch.wdl \
  -extras /rendered_wdl/extras.json \
  -defaults /rendered/rendered_wdl/sgnips_iseq_batch_defaults.json \
  -archive \
  -verbose \
  -destination myelin_parsnips_workflows:/workflows/parsnips_project/1d30b674/5f64837b -projectWideReuse

from dxcompiler.

Hi @Ben-Habermeyer!

1. The log in file file-GK2Q9xj0046J87j987XV65XZ does not contain lines from your original post. Is this a relevant log?
2. Your note that it "small changes take ~10min longer than they should I estimate" - what is the estimate process? What did you compare?
3. Does -projectWideReuse with String? = None compile as usual or slower? Do you see the respective applets reused when using String = "DEFAULT_STR" or being rebuilt? What if you keep that input optional (String? = "DEFAULT_STR")?

I'm not building this WF anywhere for obvious reasons. Also I don't think a dummy WF will be a good representative. So we need as much info from you as you can give. Or give me permissions to build this workflow on platform

from dxcompiler.

Thanks @Gvaihir sorry if I sent the wrong files I will check and resend different ones today.

for 2) My hunch is that the project wide reuse looking for one particular applet (which previously had default String? = None is problematic and takes longer than it should. I can give more pointed examples today.

for 3) The current WDL does not have String?=None but the applet used to, but we still see these warnings in the logs I believe because of the project wide reuse. Now it is String = "default" so we don't use optional anymore.

Thanks for patience with my error; I'll send some more examples today that might narrow it down a bit more.

from dxcompiler.

ok @Gvaihir I have some more data for you. I compiled 2 workflows of similar size in this same project with reuse. The first was workflow-GKVXjjj00462vX2G6JGB0v8v (sgnips_iseq_batch.wdl) and the second one with problematic task search is workflow-GKVY2X000463Vp0k30Kgv7Z3 (special_case_callers.wdl). The WDL zip should be the same so you can just download one; I just compiled different sub-workflow WDL files.

iseq

›› dx find data --link workflow-GKVXjjj00462vX2G6JGB0v8v --all-projects
closed  2022-12-21 10:55:21 69.62 MB  /workflows/parsnips_project/ae30d078/d1a4d4f5/parsnips_project-ae30d078-d1a4d4f5.wdl.zip (project-G0vbVFj0046589VQPz0kpgFk:file-GKVXjk80046Fg0GYFP6Qfz73)
closed  2022-12-21 10:55:20 458.09 KB /workflows/parsnips_project/ae30d078/d1a4d4f5/parsnips_project-ae30d078-d1a4d4f5.log (project-G0vbVFj0046589VQPz0kpgFk:file-GKVXjkj0046551k139kY57J1)

special case callers

›› dx find data --link workflow-GKVY2X000463Vp0k30Kgv7Z3 --all-projects
closed  2022-12-21 11:16:13 69.62 MB  /workflows/parsnips_project/ae30d078/e437b83d/parsnips_project-ae30d078-e437b83d.wdl.zip (project-G0vbVFj0046589VQPz0kpgFk:file-GKVY2XQ00465b9Zx4kb5YKYZ)
closed  2022-12-21 11:16:13 307.49 KB /workflows/parsnips_project/ae30d078/e437b83d/parsnips_project-ae30d078-e437b83d.log (project-G0vbVFj0046589VQPz0kpgFk:file-GKVY2Y000464b7BGJfJ8zXJ2)

the iseq build took 2:49 an the special case callers build took 8:14 on the agent. Most of the time is querying for executables, and I see the log lines in the special case callers. Again, I'm not sure if this is a red herring, but we have noticed the compilation taking longer in this specific case

Querying for executables in project DxProject(project-G0vbVFj0046589VQPz0kpgFk)
[warning] unable to parse field caller_to_emulate default value null
[warning] unable to parse field caller_to_emulate default value null
[warning] unable to parse field caller_to_emulate default value null
[warning] unable to parse field caller_to_emulate default value null
[warning] unable to parse field caller_to_emulate default value null
[warning] unable to parse field caller_to_emulate default value null
[warning] unable to parse field caller_to_emulate default value null
[warning] unable to parse field caller_to_emulate default value null
[warning] unable to parse field caller_to_emulate default value null
[warning] unable to parse field caller_to_emulate default value null
[warning] unable to parse field caller_to_emulate default value null
[warning] unable to parse field caller_to_emulate default value null
[warning] unable to parse field caller_to_emulate default value null
[warning] unable to parse field caller_to_emulate default value null
[warning] unable to parse field caller_to_emulate default value null
[warning] unable to parse field caller_to_emulate default value null
[warning] unable to parse field caller_to_emulate default value null
[warning] unable to parse field caller_to_emulate default value null
[warning] unable to parse field caller_to_emulate default value null
[warning] unable to parse field caller_to_emulate default value null
[warning] unable to parse field caller_to_emulate default value null
[warning] unable to parse field caller_to_emulate default value null
[warning] unable to parse field caller_to_emulate default value null
Found 3408 executables in project-G0vbVFj0046589VQPz0kpgFk:None
Found 3408 executables

from dxcompiler.

@Ben-Habermeyer sorry - I was away. I'll check in the coming days

from dxcompiler.

thanks @Gvaihir I will compare those two and get back to you

from dxcompiler.

@Ben-Habermeyer is this still an issue, or should I close it?

from dxcompiler.

Not reproducing

from dxcompiler.