Failed backup stated as succeeded about k8up HOT 5 OPEN

Hi

I checked this problem, and after testing this situation I recognized this problem happened because of restic command.

When the backupcommand annotation is executed, the stdin of the command pipe into restic command, and the restic store stream data in the snapshot

Unfortunately, can't fix this problem because of restic. But k8up has a summary backup for detecting the status of the backup. You can use Webhook or Prometheus to get the status of the backup

Also, I have a solution for fixing this problem:
We can add an annotation to handle errors in the backup command and delete the snapshot when the backup command fails. This backward compatible

@xoxys @Kidswiss

from k8up.

Just wanted to plusone this. I had this problem happen to me with two different workloads for different reasons (xz not being available, and a wrong env syntax for postgres).

If I hadn't checked manually with restic, I wouldn't have noticed this! I think having failed backups marked as such would be a great UX improvement.

from k8up.

Also bumped into this problem, in my case probably due to file permission issues, did spot the following in the logs:

INFO    k8up.restic.restic.backup.progress    restic output    {"msg": "Warning: at least one source file could not be read"}   

It would be great if there would be a way to make the issue more visible. I only noticed this during tests of the restore procedure.

from k8up.

Concerning the backup procedure, I solve the issue by specifying this under the spec of the Backup object (and Schedule object as well).

  podSecurityContext:
    fsGroup: 0
    runAsUser: 0

I wanted to backup my Nextcloud where the data directory is only permitted for www-data:www-data. So I ran my Backup with the user 0 (root) on purpose. I think it only works if you have the permission to execute the k8up jobs as root on the cluster. Otherwise you should try with other user id and group id.

from k8up.

This is a very important issue. Any backup system must inspire trust. If the system reports successful backup completion but actually does not work, there will be no trust to speak of. Without trust, the system will have no users.

Maybe there is a way to wrap restic in another process that will read and parse its logs to determine the success of the operation?

from k8up.