[FEATURE] Release compute nodes when the workflow is complete about merlin HOT 6 CLOSED

I've seen users have success with this by adding a step at the end of their workflow that looks something like this (for workers run in an sbatch script):

- description: Kill any active workers and cancel all of my Merlin batch jobs
  name: kill
  run:
    cmd: |
        # email me that we're finishing
        echo "All done with simulation run in $(MERLIN_WORKSPACE)" | mail -s "Merlin Cretin Update" [email protected]
        # kill workers
        merlin stop-workers
        # cancel your depending jobs
        scancel -u $USER --name Merlin --state=PENDING
        # cancel your active jobs
        scancel -u $USER --name Merlin --state=RUNNING
    depends: [last_real_step]

Basically what this does is

1. Sends you an email with the name of the ensemble that's done
2. Sends a kill command to all of the workers, which gets them to shut down gracefully
3. Uses slurm to cancel all of your jobs with the name "Merlin" in them

It would be good for us to document this snippet somewhere. It'll need to be battle tested, and perhaps modified for extra features (like only kill the workers associated with the steps in this workflow)

from merlin.

my original reply doesn't work because of this bug #129
... but ...
#182 introduces the new exit code $(MERLIN_STOP_WORKERS) which sends a delayed kill signal to all the workers. $(MERLIN_HARD_FAIL) will alternatively stop the workers connected to that specific step's queue

from merlin.

Yes, this has been addressed with the new monitor.

from merlin.

As an alternative, you can have a process living outside of the run-workers call but in your batch script that checks the status of the workflow periodically and decide when to break out of the job

see the merlin status command

from merlin.

@koning would you say this issue is effectively closed with merlin monitor ?

from merlin.

@srcopela the new command merlin monitor <spec.yaml> ... has this functionality. See the docs for more details: https://merlin.readthedocs.io/en/latest/merlin_commands.html?highlight=merlin%20monitor#monitor-merlin-monitor

from merlin.