We need a parser that can take as it's input a text file with bash commands and create a Dask HighLevelGraph.
Jobs are normally grouped together by analysis type (QC, filter, preprocess, align) and/or compute resources.
Simple Example with job only dependencies. In this example, all tasks in job_1 must complete before any in job_2 can complete.
#HPC jobname=job_1
#HPC walltime=10:00:00
#HPC mem=32GB
#HPC cpus_per_task=1
job_1_task_1
job_1_task_2
job_1_task_3
job_1_task_4
#HPC jobname=job_2
#HPC walltime=10:00:00
#HPC mem=32GB
#HPC cpus_per_task=1
#HPC deps=job_1
job_2_task_1
job_2_task_2
job_2_task_3
job_2_task_4
Example with Task Dependencies
I think Dharhas would have some really good insight into how this should be done.
When I would submit something like this on SLURM I would use array dependencies. Essentially, instead of submitting a single job N times you submit a job once that is itself a for loop that executes N times.
Jobs can depend on jobs, and tasks within the job can depend upon tasks in another job.
I don't think that this needs to be translated specifically into job arrays, but can be handled by the Dask Scheduler itself.
#HPC jobname=job_1
#HPC walltime=10:00:00
#HPC mem=32GB
#HPC cpus_per_task=1
#HPC task_tags=task_1
job_1_task_1
#HPC task_tags=task_2
job_1_task_2
#HPC task_tags=task_3
job_1_task_3
#HPC task_tags=task_4
job_1_task_4
#HPC jobname=job_2
#HPC deps=job_1
#HPC walltime=10:00:00
#HPC mem=32GB
#HPC cpus_per_task=1
#HPC task_tags=task_1
job_2_task_1
#HPC task_tags=task_2
job_2_task_2
#HPC task_tags=task_3
job_2_task_3
#HPC task_tags=task_1
job_2_task_4
#HPC jobname=job_3
#HPC deps=job_3
#HPC walltime=10:00:00
#HPC mem=32GB
#HPC cpus_per_task=1
job_3_task_1
job_3_task_2
job_3_task_3
job_3_task_4