Note: Make sure to check out known issues (https://github.com/sparebankenvest/azure-key-vault-to-kubernetes#known-issues) before submitting
Describe the bug
Using environment injection with either the supplied test image or my own results in the container not starting, apparently because the expected volume mounts are not actually created.
AKS with kubernetes 1.17.0
To Reproduce
Steps to reproduce the behavior:
Helm installation as per manual.
apiVersion: v1
kind: Namespace
metadata:
name: akv-test
labels:
azure-key-vault-env-injection: enabled
---
apiVersion: spv.no/v1alpha1
kind: AzureKeyVaultSecret
metadata:
name: db-secret-inject
namespace: akv-test
spec:
vault:
name: akvtest # name of key vault
object:
name: login-secret # name of the akv object
type: secret # akv object type
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: akv2k8s-test-injection
namespace: akv-test
labels:
app: akv2k8s-test-injection
spec:
selector:
matchLabels:
app: akv2k8s-test-injection
template:
metadata:
labels:
app: akv2k8s-test-injection
spec:
containers:
- name: akv2k8s-env-test
image: spvest/akv2k8s-env-test
env:
- name: TEST_SECRET
value: "secret-inject@azurekeyvault"
- name: SP_SECRET
value: "db-secret-inject@azurekeyvault"
- name: ENV_INJECTOR_LOG_LEVEL
value: debug
Expected behavior
SP_SECRET environment variable defined with value from AKV, inside a running container.
Logs
If applicable, add logs to help explain your problem.
The env-injector pod seems to think it did its work:
time="2020-03-11T09:20:02Z" level=info msg="found pod to mutate in namespace 'akv-test'"
time="2020-03-11T09:20:02Z" level=info msg="found container 'akv2k8s-env-test' to mutate"
time="2020-03-11T09:20:02Z" level=info msg="checking for env vars containing '@azurekeyvault' in container akv2k8s-env-test"
time="2020-03-11T09:20:02Z" level=info msg="found env var: secret-inject@azurekeyvault"
time="2020-03-11T09:20:02Z" level=info msg="found env var: db-secret-inject@azurekeyvault"
time="2020-03-11T09:20:02Z" level=info msg="did not find credentials to use with registry 'spvest' - getting default credentials"
time="2020-03-11T09:20:02Z" level=info msg="registry host 'spvest' is not a acr registry"
time="2020-03-11T09:20:02Z" level=info msg="pulling docker image docker.io/spvest/akv2k8s-env-test:latest to get entrypoint and cmd, timeout is 120 seconds"
time="2020-03-11T09:20:04Z" level=info msg="docker image docker.io/spvest/akv2k8s-env-test:latest pulled successfully"
time="2020-03-11T09:20:04Z" level=info msg="inspecting container image docker.io/spvest/akv2k8s-env-test:latest, looking for entrypoint and cmd"
time="2020-03-11T09:20:04Z" level=info msg="using 'entrypoint.sh' as arguments for env-injector"
time="2020-03-11T09:20:04Z" level=info msg="containers mutated and pod updated with init-container and volumes"
But in the pod description:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled <unknown> default-scheduler Successfully assigned akv-test/akv2k8s-test-injection-6c796746df-vqspz to aks-nodepool1-40643695-vmss000001
Normal Pulled 13m kubelet, aks-nodepool1-40643695-vmss000001 Container image "spvest/azure-keyvault-env:1.0.1" already present on machine
Normal Created 13m kubelet, aks-nodepool1-40643695-vmss000001 Created container copy-azurekeyvault-env
Normal Started 13m kubelet, aks-nodepool1-40643695-vmss000001 Started container copy-azurekeyvault-env
Normal Started 13m kubelet, aks-nodepool1-40643695-vmss000001 Started container akv2k8s-env-test
Normal Pulled 12m (x4 over 13m) kubelet, aks-nodepool1-40643695-vmss000001 Successfully pulled image "spvest/akv2k8s-env-test"
Normal Created 12m (x4 over 13m) kubelet, aks-nodepool1-40643695-vmss000001 Created container akv2k8s-env-test
Warning Failed 12m (x3 over 12m) kubelet, aks-nodepool1-40643695-vmss000001 Error: failed to start container "akv2k8s-env-test": Error response from daemon: OCI runtime create failed: container_linux.go:346: starting container process caused "exec: \"/azure-keyvault/azure-keyvault-env\": stat /azure-keyvault/azure-keyvault-env: no such file or directory": unknown
Normal Pulling 11m (x5 over 13m) kubelet, aks-nodepool1-40643695-vmss000001 Pulling image "spvest/akv2k8s-env-test"
Warning BackOff 3m11s (x44 over 12m) kubelet, aks-nodepool1-40643695-vmss000001 Back-off restarting failed container
The init container does not show any error state (exit code 0). Both containers have the /azure-keyvault/ mount listed.
The logs for the init container simply say that the file was copied:
kubectl logs -n akv-test akv2k8s-test-injection-6c796746df-vqspz -c copy-azurekeyvault-env
Copying /azure-keyvault/azure-keyvault-env to /azure-keyvault/
Additional context
This did work yesterday! Nothing has changed in the AKS cluster. I have redeployed the akv2k8s and application deployments a few times to be sure.