Job/backupjob-XXXXXXXX hangs forever if pvc gets deleted during schedule about k8up HOT 4 CLOSED

Hi @dns2utf8

What storage provider are you using? Some providers (cephfs) don't play nice with volumeMount that are readonly.

The issue here is, that the operator collects a list of all eligible PVCs and uses that to create the jobs. That's why you have to delete the job object, as the mounts are already defined in there. When now a PVC is deleted the job configuration becomes invalid thus the hanging pods.

Unfortunately I currently don't know how to mitigate this easily as the mounts have to be specified by the operator and then a job is created. So there's always a gap where a PVC can be deleted.

If there are temporary PVCs that may get deleted at any time and don't need backups then you can set the appuio.ch/backup=false annotation on the PVC.

If you have any input how that can be solved properly, I'm all ears :)

Best Regards
Simon

from k8up.

Hi Simon

I am using xfs (backed by HPE Nimble CSI).

Thank you for explaining how the race condition comes to be.

My initial reaction was: Some kind of garbage collector could solve that.
Pro GC: it would cleanup stuck jobs without adding complexity to the operator.
Contra GC: Has to maintain the same predictable behaviour as the operator.

After thinking about the problem some more I have another idea:
What if the operator does the GC run itself before creating each Job and after changes to the schedules (all the CRUD operations)?

Best,
Stefan

from k8up.

What exactly would the job of this garbage collector be? Cleaning up stuck pods? Maybe there's already some way to tell k8s to discard pods that are pending for too long.

from k8up.

Closing due to the lack of feedback.

from k8up.