Job resubmission with job scheduler doesn't work about veros HOT 10 CLOSED

What’s the content of veros_batch.sh?

from veros.

#!/bin/bash -l
#
#SBATCH -p aegir
#SBATCH -A ocean
#SBATCH --job-name=acc_lr
#SBATCH --time=23:59:59
#SBATCH --constraint=v2
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=32
#SBATCH --exclusive

export BH_STACK=openmp
module load bohrium/05102018 veros/05102018
veros-resubmit -i acc.lowres -n 50 -l 62208000 -c "python acc.py -b bohrium -v debug" --callback "/usr/bin/sbatch /groups/ocean/nutrik/veros_cases/paper/acc/veros_batch.sh"

from veros.

How long (in real time) do the 29 day runs take?

from veros.

11 minutes.
The issue is not about wall clock time limit exceedance. Every model cycle is successfully completed with "Timing summary" information but the case is not resubmitted.

from veros.

And how do you execute the first run?

from veros.

sbatch veros_batch.sh

from veros.

Ok. So the thing is that resubmission has an inherent race condition. The job reschedules itself, waits for a short while, then exits, but if the wait time is too short it might be that the job is killed before the new job has gone through. Unless there is a reliable way to determine whether a new job has been scheduled there's not much we can do about that.

One thing you can try is to add a time.sleep(5) after the following line:

https://github.com/dionhaefner/veros/blob/0079cde8dbf39f9a3944902792292807b20f443c/veros/cli/veros_resubmit.py#L81

If that doesn't help, we'll have to dig deeper.

On a side note, the reason why it works without the --callback argument is that the default behavior for the script is to call itself. In this case, it just runs again on the same node, and is never really rescheduled. It will thus time out after 24h.

from veros.

Yeah, adding time.sleep(5) has solved the issue! Thanks!

from veros.

But, I think the master branch must be fixed for other users.

from veros.

Let’s keep this open for now. I’ll probably introduce some waiting time, but I doubt we can make the problem go away in every case.

from veros.