Remediation of `MISSING_LOST` segments. about wal-g HOT 3 OPEN

Hi! You can archive empty file as 0000007200000CBF00000017, 0000007300000CBF00000067 and 0000007400000CBF000000AB.
I understand that this does not sound as a proper solution... Maybe we could add functionality like "mute absance of specific files" or something like that...

from wal-g.

Meanwhile I'd appreciate if you describe how you lost WALs in more details. We should consider if WAL-G could prevent this.

from wal-g.

Hi @x4m!

What happened was we made some adjustments to a Patroni managed cluster in order to band-aid some application problems. Namely we increased max_connections fairly substantially. We also increased the memory allocation but apparently not by a sufficient amount.

What then followed was the k8s pods reached a near OOM state. PostgreSQL then failed to archive segments with this message:

DETAIL:  The failed archive command was: envdir "/run/etc/wal-e.d/env" wal-g wal-push "pg_wal/0000007300000CBF00000067"

Followed by one of the backends geting OOM Killed and then postmaster shutting down and killing the archive task.

After failover I thought these segments would get uploaded by they did not. It does not appear that PostgreSQL even tried to but I am not that familiar with what happens after WAL archiving fails or if the status of not-archived segments is persisted to the secondary.

There was a trace from wal-g but it was because it got a SIGQUIT from postgres, likely during the ungraceful shutdown, other than that nothing else to indicate what went wrong. As far as I can tell wal-g wasn't killed by the OOM killer as I have no logs of it being killed.

from wal-g.