add a clean up step to the HostedCE shutdown about slate-catalog HOT 6 CLOSED

@LincolnBryant what sort of cleanup task were you thinking?

from slate-catalog.

hi @brianhlin, note this ticket is quite old at this point. however I believe at the time I was thinking of having a preStop hook in Kubernetes that would signal to the pod to remove something. perhaps SSH to the remote side and stop any pilots & delete itself? i guess you could get quite complex with this.

notably I was thinking of the case where we want to completely remove a CE (vs say just pausing it, where in that case we would probably just do a slate instance scale --replicas=0 to stop the running instance but leave all of the config intact)

from slate-catalog.

The deployment already has a preStop that drains the CE, which should also remove the pilots from the remote batch system. So I think this is already done unless you want to do BOSCO/WN client cleanup from the remote host's disk, too.

from slate-catalog.

I have observed that jobs will stay in the remote batch system's queue after the CE is removed. atleast in the case of slurm.

from slate-catalog.

That's curious. Could you try running the drain-ce.sh script manually and taking a look at the GridmanagerLog? I'd expect that the CE would send an scancel to the remote jobs.

from slate-catalog.

This was fixed at some point.

from slate-catalog.