When deleting a cluster the operator panics because of a nil pointer deference, while trying to remove the steward secret.
It is unclear which steps exactly lead to this.
The general steps where:
The operator panics while trying to handle the deletion of a cluster.
$ vault kv get clusters/kv/t-ancient-morning-1764/c-cold-morning-3608/steward
====== Metadata ======
Key Value
--- -----
created_time 2021-07-27T07:10:58.227801984Z
deletion_time n/a
destroyed false
version 1
==== Data ====
Key Value
--- -----
token <token>
{"level":"info","ts":1627541860.7444935,"logger":"controller_cluster","msg":"Reconciling Cluster","Request.Namespace":"lieutenant-int","Request.Name":"c-cold-morning-3608"}
E0729 06:57:40.976044 1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 2089 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x1551940, 0x223a5e0)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:74 +0xa6
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:48 +0x86
panic(0x1551940, 0x223a5e0)
/usr/local/go/src/runtime/panic.go:965 +0x1b9
github.com/projectsyn/lieutenant-operator/pkg/vault.(*BankVaultClient).removeSecret(0xc000bf6680, 0xc0007350c0, 0x2a, 0x0, 0x0, 0x20, 0x14fb360)
/app/pkg/vault/client.go:178 +0x1d0
github.com/projectsyn/lieutenant-operator/pkg/vault.(*BankVaultClient).RemoveSecrets(0xc000bf6680, 0xc0004af4a0, 0x1, 0x1, 0xc000bf6680, 0x0)
/app/pkg/vault/client.go:153 +0x7e
github.com/projectsyn/lieutenant-operator/pkg/vault.HandleVaultDeletion(0x192d338, 0xc000b396c0, 0xc00047dd40, 0x203000, 0x0, 0x0, 0xc00042de00)
/app/pkg/vault/reconcile_steps.go:99 +0x2b9
github.com/projectsyn/lieutenant-operator/pkg/pipeline.RunPipeline(0x192d338, 0xc000b396c0, 0xc00047dd40, 0xc0006a19a8, 0x8, 0x8, 0x0, 0xc0006a19c0, 0x40e078, 0x30)
/app/pkg/pipeline/pipeline.go:61 +0xb1
github.com/projectsyn/lieutenant-operator/pkg/controller/cluster.clusterSpecificSteps(0x192d338, 0xc000b396c0, 0xc00047dd40, 0x0, 0x0, 0x0, 0x0)
/app/pkg/controller/cluster/cluster_reconcile.go:70 +0x1d8
github.com/projectsyn/lieutenant-operator/pkg/pipeline.RunPipeline(0x192d338, 0xc000b396c0, 0xc00047dd40, 0xc0006a1bd8, 0x6, 0x6, 0x13, 0x18f7b60, 0xc000b396c0, 0x0)
/app/pkg/pipeline/pipeline.go:61 +0xb1
github.com/projectsyn/lieutenant-operator/pkg/controller/cluster.(*ReconcileCluster).Reconcile(0xc000678558, 0xc000c07810, 0xe, 0xc000755170, 0x13, 0x1aeac146c1, 0xc000364480, 0xc00014e788, 0xc00014e750)
/app/pkg/controller/cluster/cluster_reconcile.go:52 +0x570
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000660300, 0x15bb220, 0xc0000ab620, 0x0)
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:256 +0x166
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000660300, 0xc000308500)
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:232 +0xb0
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker(...)
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:211
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc00067a420)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:152 +0x5f
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00067a420, 0x3b9aca00, 0x0, 0x1, 0xc000114420)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:153 +0x105
k8s.io/apimachinery/pkg/util/wait.Until(0xc00067a420, 0x3b9aca00, 0xc000114420)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:88 +0x4d
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:193 +0x32d
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x130e9f0]
goroutine 2089 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:55 +0x109
panic(0x1551940, 0x223a5e0)
/usr/local/go/src/runtime/panic.go:965 +0x1b9
github.com/projectsyn/lieutenant-operator/pkg/vault.(*BankVaultClient).removeSecret(0xc000bf6680, 0xc0007350c0, 0x2a, 0x0, 0x0, 0x20, 0x14fb360)
/app/pkg/vault/client.go:178 +0x1d0
github.com/projectsyn/lieutenant-operator/pkg/vault.(*BankVaultClient).RemoveSecrets(0xc000bf6680, 0xc0004af4a0, 0x1, 0x1, 0xc000bf6680, 0x0)
/app/pkg/vault/client.go:153 +0x7e
github.com/projectsyn/lieutenant-operator/pkg/vault.HandleVaultDeletion(0x192d338, 0xc000b396c0, 0xc00047dd40, 0x203000, 0x0, 0x0, 0xc00042de00)
/app/pkg/vault/reconcile_steps.go:99 +0x2b9
github.com/projectsyn/lieutenant-operator/pkg/pipeline.RunPipeline(0x192d338, 0xc000b396c0, 0xc00047dd40, 0xc0006a19a8, 0x8, 0x8, 0x0, 0xc0006a19c0, 0x40e078, 0x30)
/app/pkg/pipeline/pipeline.go:61 +0xb1
github.com/projectsyn/lieutenant-operator/pkg/controller/cluster.clusterSpecificSteps(0x192d338, 0xc000b396c0, 0xc00047dd40, 0x0, 0x0, 0x0, 0x0)
/app/pkg/controller/cluster/cluster_reconcile.go:70 +0x1d8
github.com/projectsyn/lieutenant-operator/pkg/pipeline.RunPipeline(0x192d338, 0xc000b396c0, 0xc00047dd40, 0xc0006a1bd8, 0x6, 0x6, 0x13, 0x18f7b60, 0xc000b396c0, 0x0)
/app/pkg/pipeline/pipeline.go:61 +0xb1
github.com/projectsyn/lieutenant-operator/pkg/controller/cluster.(*ReconcileCluster).Reconcile(0xc000678558, 0xc000c07810, 0xe, 0xc000755170, 0x13, 0x1aeac146c1, 0xc000364480, 0xc00014e788, 0xc00014e750)
/app/pkg/controller/cluster/cluster_reconcile.go:52 +0x570
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000660300, 0x15bb220, 0xc0000ab620, 0x0)
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:256 +0x166
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000660300, 0xc000308500)
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:232 +0xb0
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker(...)
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:211
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc00067a420)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:152 +0x5f
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00067a420, 0x3b9aca00, 0x0, 0x1, 0xc000114420)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:153 +0x105
k8s.io/apimachinery/pkg/util/wait.Until(0xc00067a420, 0x3b9aca00, 0xc000114420)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:88 +0x4d
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:193 +0x32d
The operator is able to handle the cluster deletion or return an error without crashing if the cluster resource is in an inconsistent state.