CloudSQL instance gets inexplicably deleted about provider-gcp HOT 3 CLOSED

I have confirmed that cross-namespace ownerReference is the problem. Created 2 PostgreSQLInstance claims and waited until managed resources come up and running. Then manually removed the metadata.ownerReferences from the one of the CloudSQLInstance resources. Restarted the minikube, then did kubectl delete pod --all -n kube-system; the one with owner reference got deleted but the one without owner reference is kept.

It seems like removing the owner reference is a valid workaround. Though this usually helps if you have the chance to remove the owner reference. Also note that without owner reference, when you delete claim the managed resource will not get deleted.

As I linked above, crossplane/crossplane-runtime#21 is the way to go for a permanent fix. It's a little bit complicated since the field name for reclaimPolicy will stay same but its meaning will change, which means breaking public interface which then means we'll probably need to bump the v1beta1 resources to v1beta2 with that change.

from provider-gcp.

The GCP activity log shows a deletion request for the CloudSQL instance at 05:39:43.811Z. The storm of networking issues in the stack-gcp pod ends at 05:38:40.110429, so just about 1 min before the deletion happens.

I'd have to think they are somehow related if the instance is up for multiple hours, this networking storm happens and then within 1 minute the database gets deleted. 🤔

from provider-gcp.

I think kubernetes/kubernetes#65200 is directly related what we're seeing here. By design, we shouldn't be able to have a namespaced-owner(claim) to clusterscoped-dependent(managed resource). The issue tries to address the fact that k8s api accepts requests containing that invalid relation. As far as I see this comment explains why it happens almost randomly. We have seen a similar case in crossplane/crossplane#895 as well.

The actual solution is not to use owner references at all. Actually crossplane/crossplane-runtime#21 fixes this once and for all by following PVC/PV model. I'll prioritize this.

@jbw976 Manually deleting the metadata.ownerReference of the managed resource should probably help as a workaround. I'll test it and let you know.

from provider-gcp.