Comments (6)
I've seen users have success with this by adding a step at the end of their workflow that looks something like this (for workers run in an sbatch script):
- description: Kill any active workers and cancel all of my Merlin batch jobs
name: kill
run:
cmd: |
# email me that we're finishing
echo "All done with simulation run in $(MERLIN_WORKSPACE)" | mail -s "Merlin Cretin Update" [email protected]
# kill workers
merlin stop-workers
# cancel your depending jobs
scancel -u $USER --name Merlin --state=PENDING
# cancel your active jobs
scancel -u $USER --name Merlin --state=RUNNING
depends: [last_real_step]
Basically what this does is
- Sends you an email with the name of the ensemble that's done
- Sends a kill command to all of the workers, which gets them to shut down gracefully
- Uses slurm to cancel all of your jobs with the name "Merlin" in them
It would be good for us to document this snippet somewhere. It'll need to be battle tested, and perhaps modified for extra features (like only kill the workers associated with the steps in this workflow)
from merlin.
my original reply doesn't work because of this bug #129
... but ...
#182 introduces the new exit code
from merlin.
Yes, this has been addressed with the new monitor.
from merlin.
As an alternative, you can have a process living outside of the run-workers call but in your batch script that checks the status of the workflow periodically and decide when to break out of the job
see the merlin status
command
from merlin.
@koning would you say this issue is effectively closed with merlin monitor
?
from merlin.
@srcopela the new command merlin monitor <spec.yaml> ...
has this functionality. See the docs for more details: https://merlin.readthedocs.io/en/latest/merlin_commands.html?highlight=merlin%20monitor#monitor-merlin-monitor
from merlin.
Related Issues (20)
- [BUG] merlin example list gives an error HOT 4
- [BUG] errors hidden in sample generation
- [FEAT] Adding configuration features to merlin server command. HOT 2
- [FEAT] Unify merlin server config and configfile app.yaml HOT 1
- [FEAT] Need to transition off lgtm to github code-scanning HOT 1
- [FEAT] Launch workers with flux if it is the main scheduler HOT 2
- "merlin stop-workers" with "--spec" and "--queues" flags does not work
- [BUG] JSON Schema Validation causes failures with restarts/resubmits
- Basic Workflow examples requires another service? HOT 29
- Add what is the default concurrency value to the documentation HOT 2
- Walltime in Batch Block HOT 4
- Question: where does singularity openfoam image get pulled? HOT 10
- Step restart may not work with many workers and long run times HOT 2
- [BUG] MERLIN_SPEC variables are not working properly HOT 11
- [BUG] slurm_par example only spins up 1 node instead of 2
- Unable to Place Environment Varibles in field resources.workers.sim_workers.nodes of merlin section HOT 1
- subprocess in check_for_scheduler can also raise PermissionError HOT 1
- [FEAT] Allow for None and empty string in some batch input HOT 2
- [BUG] new “merlin status” on old workflows HOT 3
- 'openfoam_wf_singularity' not found
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from merlin.