Hi,

With typical Kubernetes secrets, mounted Secrets are updated automatically.

I noticed that with the CSI-driver, updating the secret in Key Vault, doesn't make the secret change in the filepath. In my (limited) testing, I was able to get the new version of a secret loaded after recreating the pod (deleting pod and having deployment create a new pod).

This lead to two questions from my side:

1. Is there the ability to disable secret caching?
2. By default, how long are secrets cached before getting re-fetched from Key Vault?

Describe the solution you'd like
It appears that for the secret sync to work, you first need to have a pod that mounts the kv volume to trigger the secret creation. It would be great if the secret creation could be triggered off of the creation of the SecretProviderClass instead. I assume this is just a requirement of the CSI driver itself, so maybe this is outside of the intent of the implementation.

If I am not wrong, we must use the following command to create the secretproviderclasses resource, right?

kubectl apply -f <resource-filename>.yaml

But when I use sample given in the README to create a new secretproviderclasses resource, I get the following error:

error: unable to recognize "resource-filename.yaml": no matches for kind "SecretProviderClass" in version "secrets-store.csi.x-k8s.io/v1alpha1"

Hello,

thank you for this project. Just got few questions:

1. Is kubernetes-keyvault-flexvol being depcreated for this project?
2. Does this project contain all the functionalities of kubernetes-keyvault-flexvol + the following bug fixes/enhancements?
3. Can we already use this project in production? or we have to stick with kubernetes-keyvault-flexvol for a while?

thanks a lot

Describe the solution you'd like
I would like to be able to sync Key Vault secrets and expose them as an environment variables as well. As far as I know it is only possible to expose them via a volume, but a lot of applications still use environment variables (even thought that maybe not best practice).

Anything else you would like to add:
No

Environment:

• Secrets Store CSI Driver version: (use the image tag):
• Azure Key Vault provider version: (use the image tag):
• Kubernetes version: (use kubectl version): <1.16.x>
• Cluster type: (e.g. AKS, aks-engine, etc): AKS

What steps did you take and what happened:

Install CSI driver

helm install csi-secrets-store-provider-azure/csi-secrets-store-provider-azure --generate-name -n $target_namespace
helm ls -n $target_namespace 

Create key-Vault

az group create --name $rg_kv_name --location $location
az keyvault create --name $vault_name --enable-soft-delete true --location $location -g $rg_kv_name
az keyvault show --name $vault_name 
# az keyvault update --name $vault_name --default-action deny -g $rg_kv_name 

kv_id=$(az keyvault show --name $vault_name -g $rg_kv_name --query "id" --output tsv)
echo "KeyVault ID :" $kv_id
nslookup $vault_name.vault.azure.net

az keyvault secret set --name $vault_secret_name --value $vault_secret --description "CSI driver ${appName} Secret" --vault-name $vault_name
az keyvault secret list --vault-name $vault_name
az keyvault secret show --vault-name $vault_name --name $vault_secret_name --output tsv

aks_client_id=$(az aks show -g $rg_name -n $cluster_name --query identityProfile.kubeletidentity.clientId -o tsv)
echo "AKS Cluster Identity Client ID " $aks_client_id
az role assignment create --role Reader --assignee $aks_client_id --scope $kv_id

Create an Azure Identity

export SUBSCRIPTION_ID=$subId
export RESOURCE_GROUP=$rg_name
export IDENTITY_NAME="${appName}-csi-demo-pod-identity" #  must consist of lower case 

# https://github.com/Azure/aad-pod-identity/pull/48
# https://github.com/Azure/aad-pod-identity/issues/38
az identity create -g $rg_name -n $IDENTITY_NAME
export IDENTITY_CLIENT_ID="$(az identity show -g $rg_name -n $IDENTITY_NAME --query clientId -otsv)"
export IDENTITY_RESOURCE_ID="$(az identity show -g $rg_name -n $IDENTITY_NAME --query id -otsv)"
export IDENTITY_ASSIGNMENT_ID="$(az role assignment create --role Reader --assignee $IDENTITY_CLIENT_ID --scope /subscriptions/$subId/resourceGroups/$rg_name --query id -o tsv)"

# set policy to access keys in your keyvault
az keyvault set-policy -n $vault_name --key-permissions get --spn $IDENTITY_CLIENT_ID
# set policy to access secrets in your keyvault
az keyvault set-policy -n $vault_name --secret-permissions get --spn $IDENTITY_CLIENT_ID
# set policy to access certs in your keyvault
az keyvault set-policy -n $vault_name --certificate-permissions get --spn $IDENTITY_CLIENT_ID

export TENANT_ID=$tenantId
export KV_NAME=$vault_name
export SECRET_NAME=$vault_secret_name
export POD_ID=xxxxxx

envsubst < ./cnf/secrets-store-csi-provider-class.yaml > deploy/secrets-store-csi-provider-class.yaml
cat deploy/secrets-store-csi-provider-class.yaml
k apply -f deploy/secrets-store-csi-provider-class.yaml -n $target_namespace
k get secretproviderclasses -n $target_namespace
k describe secretproviderclasses azure-kv-vsegov-xxxxxx -n $target_namespace

envsubst < ./cnf/csi-demo-pod-identity.yaml > deploy/csi-demo-pod-identity.yaml
cat deploy/csi-demo-pod-identity.yaml
k apply -f deploy/csi-demo-pod-identity.yaml -n $target_namespace
k get azureidentity -A
k get azureidentitybindings -A
k get azureassignedidentities -A

Deploy demo app

envsubst < ./cnf/secrets-store-csi-demo-pod.yaml > deploy/secrets-store-csi-demo-pod.yaml
cat deploy/secrets-store-csi-demo-pod.yaml
k apply -f deploy/secrets-store-csi-demo-pod.yaml -n $target_namespace
k get po -n $target_namespace
k get events -n $target_namespace | grep -i "Error" 
k describe pod nginx-secrets-store-inline -n $target_namespace
k logs nginx-secrets-store-inline -n $target_namespace

  Warning  FailedMount  17m (x289 over 13h)  kubelet, aks-vsegovaksnp-21446524-vmss000000  (combined from similar events): MountVolume.SetUp failed for volume "secrets-store-inline" : kubernetes.io/csi: mounter.SetupAt failed: rpc error: code = Unknown desc = error mounting secret time="2020-05-07T08:01:00Z" **level=fatal msg="[error] : failed to get keyvaultClient: failed to get key vault token: nmi response failed with status code: 403"**
 for pod: 44c502cf-d272-4e8c-a972-c1331708d8e3, ns: staging
  Warning  FailedMount  7m52s (x62 over 13h)  kubelet, aks-vsegovaksnp-21446524-vmss000000  Unable to attach or mount volumes: unmounted volumes=[secrets-store-inline], unattached volumes=[default-token-txdsw secrets-store-inline]: timed out waiting for the condition
  Warning  FailedMount  70s (x263 over 14h)   kubelet, aks-vsegovaksnp-21446524-vmss000000  Unable to attach or mount volumes: unmounted volumes=[secrets-store-inline], unattached volumes=[secrets-store-inline default-token-txdsw]: timed out waiting for the condition

Test

k exec -it nginx-secrets-store-inline-podid ls /mnt/secrets-store/

What did you expect to happen:

I would expect demo pod to be up & running

k get po -n $target_namespace
k get events -n $target_namespace | grep -i "Error" 
k describe pod nginx-secrets-store-inline -n $target_namespace
k logs nginx-secrets-store-inline -n $target_namespace

Anything else you would like to add:

Which access mode did you use to access the Azure Key Vault instance:
[e.g. Service Principal, Pod Identity, User Assigned Managed Identity, System Assigned Managed Identity]
Pod Identity

Environment:

• Secrets Store CSI Driver version: (use the image tag): 0.0.5

• Azure Key Vault provider version: (use the image tag):

• Kubernetes version: (use kubectl version):

k version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.2", GitCommit:"c97fe5036ef3df2967d086711e6c0c405941e14b", GitTreeState:"clean", BuildDate:"2019-10-15T19:18:23Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.7", GitCommit:"1fc58d90cc4bc569394d9a882522adcb99fbaa57", GitTreeState:"clean", BuildDate:"2020-04-03T15:20:08Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"}

• Cluster type: (e.g. AKS, aks-engine, etc): AKS 1.16.7

Yaml files attached:
secrets-store-csi-driver-issue88_yaml.zip

This can help contributors working on the Azure Provider locally make sure any changes can be tested effectively in an e2e environment.

Add http userAgent header to all requests with value csi-secrets-store-provider-azure.

Update to Helm v3 to remove dependency on Tiller.

• change the syntax of the helm install inside azure.bats
• upgrade to helm 3 in e2e-bootstrap

System and user MSI https://github.com/Azure/azure-policy-kubernetes/blob/master/pkg/azureauth/azureauth.go#L70

so that we can test Azure-specific features

https://github.com/Azure/secrets-store-csi-driver-provider-azure/blob/master/docs/pod-identity-mode.md

On step 7, we add "usepodidentity" to secretProvider class, which seems logical.
but on step 6 b. it says to had "usepodidentity: true", while we are talking about the deployment. Do we really need to had it to the deployment? If yes, at which level. my different tries make k8s grumpy

• table of content
• demo to show how the solution works
• diagram to show how diff components work together

Between the four modes for accessing a Key Vault instance, what is the best option ?
When to use what ?

The Azure Key Vault Provider offers four modes for accessing a Key Vault instance:

1. Service Principal
2. Pod Identity
3. VMSS User Assigned Managed Identity
4. VMSS System Assigned Managed Identity

The first option for SPN seems to be clearly not recommended.
Pod Identity will give access to Key Vault to only the Pod, not to all the VMs/Nodes in VMSS, right ?
Until today, AKS supports only System Managed Identity, does that mean I cannot use option number 3 ?

What steps did you take and what happened:

1. Install csi-secrets-store-provider-azure
   helm install csi-secrets-store-provider-azure/csi-secrets-store-provider-azure --generate-name --namespace kube-system

2. Deploy secretproviderclass
   kc apply -f secretproviderclasses-sync-k8s-secrets.yaml
   Sample yaml:

apiVersion:` secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
name: test-k8s-secrets
spec:
provider: azure
secretObjects:

• secretName: kvcreds
  type: Opaque
  data:
  • objectName: kvClientId
    key: kvclientid
    parameters:
    usePodIdentity: "false"
    useVMManagedIdentity: "true"
    userAssignedIdentityID: "***********"
    keyvaultName: "test-kv"
    objects: |
    array:
    • |
      objectName: kvClientId
      objectType: secret
      resourceGroup: "test-rg"
      subscriptionId: ""
      tenantId: "**"

3. Deploy k8s secret
   kc apply -f secret-kvcreds.yaml
   Sample yaml:
   apiVersion: v1
   kind: Secret
   metadata:
   name: kvcreds
   type: Opaque
   data:
   kvclientid: ZnJvbV9rZXlfdmF1bHQNCg==

What works as expected?
Deploy a pod containing.....
volumes:
- name: secrets-store-inline
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: "test-k8s-secrets"

kubectl exec -it test-pod -- cat /mnt/secrets-store/kvClientId

• matches the value I set in Key Vault

What does not work as expected?
kc get secret kvcreds -o yaml shows
data:
kvclientid: ZnJvbV9rZXlfdmF1bHQNCg==

• ZnJvbV9rZXlfdmF1bHQNCg== (decoded) does not match the value I set in Key Vault

What did you expect to happen:
I expected the key kvclientid in the k8s secret to be updated with the sync'd value of kvclientid from Key Vault.

Anything else you would like to add:
I tried a few variations of the above repro steps:
I created the k8s secret first, then the secretproviderclass second.
I created the secretproviderclass first, then the k8s secret second.
I did not create a K8s secret in hopes that the secretproviderclass would create the secret for me.
I created the k8s secret with a bogus key in hopes that secretproviderclass would create the key for me.

Which access mode did you use to access the Azure Key Vault instance:
User Assigned Managed Identity

Environment:

• Secrets Store CSI Driver version: (use the image tag): v1.2.0
• Azure Key Vault provider version: (use the image tag): 0.0.5
• Kubernetes version: (use kubectl version): v1.16.7
• Cluster type: (e.g. AKS, aks-engine, etc): AKS

I followed the steps for using Pod Identity and went into this error:
$ kubectl logs mic-76dd75ddf9-8g8k8
E0429 15:46:14.771807 1 vmss.go:73] Failed to update VM with error compute.VirtualMachineScaleSetsClient#Update: Failure sending request: StatusCode=403 -- Original Error: Code="LinkedAuthorizationFailed" Message="The client '00000000-28c9-4944-acc4-a08277f0044b' with object id '00000000-28c9-4944-acc4-a08277f0044b' has permission to perform action 'Microsoft.Compute/virtualMachineScaleSets/write' on scope '/subscriptions/00000000-7ca3-4e76-8d0f-31a2c0bee7a3/resourceGroups/MC_demo-rg_aks-demo_westeurope/providers/Microsoft.Compute/virtualMachineScaleSets/aks-agentpool-51725664-vmss'; however, it does not have permission to perform action 'Microsoft.ManagedIdentity/userAssignedIdentities/assign/action' on the linked scope(s) '/subscriptions/00000000-7ca3-4e76-8d0f-31a2c0bee7a3/resourcegroups/demo-rg/providers/Microsoft.ManagedIdentity/userAssignedIdentities/identity-aks-kv' or the linked scope(s) are invalid."

And when running $ az aks show, I get it tells me there's no identity! When I created this cluster I selected "Service Principal" from the portal.
"identity": null,
"identityProfile": null,

Any help, please ?

• Use distroless base image for pod-identity components

Hi, when I am trying to deploy my app, I wish to mount an "Elliptic Curve" Key in the application pod. But I am getting this error for that:

Events:
  Type     Reason       Age   From                                        Message
  ----     ------       ----  ----                                        -------
  Normal   Scheduled    40s   default-scheduler                           Successfully assigned default/certs-store to aks-agentpool-19909641-vmss000001
  Warning  FailedMount  38s   kubelet, aks-agentpool-19909641-vmss000001  MountVolume.SetUp failed for volume "secrets-store-inline" : kubernetes.io/csi: mounter.SetupAt failed: rpc error: code = Unknown desc = error mounting secret panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x6e4b08]

goroutine 1 [running]:
github.com/Azure/secrets-store-csi-driver-provider-azure/pkg/azure.(*Provider).GetKeyVaultObjectContent(0xc0003b7eb8, 0x831620, 0xc00001a018, 0xc00001ab88, 0x3, 0xc00001ab50, 0xc, 0x0, 0x0, 0xc0003eec00, ...)
  /go/src/github.com/Azure/secrets-store-csi-driver-provider-azure/pkg/azure/provider.go:431 +0x13f8
github.com/Azure/secrets-store-csi-driver-provider-azure/pkg/azure.(*Provider).MountSecretsStoreObjectContent(0xc0003b7eb8, 0x831620, 0xc00001a018, 0xc00005f260, 0xc00005f290, 0x7ffc950fdd55, 0x6f, 0xc0000001a4, 0x7b21b9, 0x7)
  /go/src/github.com/Azure/secrets-store-csi-driver-provider-azure/pkg/azure/provider.go:387 +0xa6d
main.main()
           /go/src/github.com/Azure/secrets-store-csi-driver-provider-azure/cmd/main.go:67 +0x42b
  Warning  FailedMount  36s  kubelet, aks-agentpool-19909641-vmss000001  MountVolume.SetUp failed for volume "secrets-store-inline" : kubernetes.io/csi: mounter.SetupAt failed: rpc error: code = Unknown desc = error mounting secret panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x6e4b08]

I was getting some error with the Flex Volume too. I switched my testing to "secrets-store-csi-driver" for that. But again facing a similar issue.

First of all thanks for this project, I was looking for this exact scenario for my service.

I just followed the guidelines mentioned above and everything works as expected.
Also, the section '[OPTIONAL] Sync with Kubernetes Secrets' works as expected for type 'Opaque'. But, 'Sync with Kubernetes Secrets'' is not working for secret type 'kubernetes.io/tls'.

Refer config below 'SecretProviderClass.yaml'.

• secret1-k8 was created successfully in k8s' secrets.
• cert1pfx-k8 and cert1pem-k8 were not created in k8s' secrets.

Here is the error message from the Pod:
Warning message: 'FailedMount' : MountVolume.SetUp failed for volume "secrets-store-volume-name" : kubernetes.io/csi: mounter.SetupAt failed: rpc error: code = Internal desc = tls key is not supported. Only tls.key and tls.crt are supported

Really appreciate your help. :)

SecretProviderClass.yaml

apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
name: azure-kvname
spec:
provider: azure
parameters:
usePodIdentity: "false" # [OPTIONAL] if not provided, will default to "false"
keyvaultName: "kvname" # the name of the KeyVault
cloudName: "" # [OPTIONAL for Azure] if not provided, azure environment will default to AzurePublicCloud
objects: |
array:
- |
objectName: secret1
objectType: secret # object types: secret, key or cert
objectVersion: "" # [OPTIONAL] object versions, default to latest if empty
- |
objectName: cert1pfx #PFX type secret in KeyVault
objectType: cert
objectVersion: ""
- |
objectName: cert1pem #PEM type secret in KeyVault
objectType: cert
objectVersion: ""
resourceGroup: "testrg" # [REQUIRED for version < 0.0.4] the resource group of the KeyVault
subscriptionId: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" # [REQUIRED for version < 0.0.4] the subscription ID of the KeyVault
tenantId: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" # the tenant ID of the KeyVault
secretObjects:

• secretName: secret1-k8
  type: Opaque
  data:
  • objectName: secret1
    key: secret
• secretName: cert1pfx-k8
  type: kubernetes.io/tls
  data:
  • objectName: cert1pfx
    key: secret
• secretName: cert1pem-k8
  type: kubernetes.io/tls
  data:
  • objectName: cert1pem
    key: secret

For windows, we need to sign the .exe and ps1 files.

What steps did you take and what happened:
[A clear and concise description of what the bug is.]
When I create an AKS that uses Managed Identity instead of SPN, the value for .servicePrincipalProfile.clientId is "msi"

  "servicePrincipalProfile": {
    "clientId": "msi"
  },

Thus, assigning the role doesn't work:

$ az role assignment create --role "Managed Identity Operator" --assignee $aks.servicePrincipalProfile.clientId --scope $identity.id
Cannot find user or service principal in graph database for 'msi'. If the assignee is an appId, make sure the corresponding service principal is created with 'az ad sp create --id msi'.

But there is another section in AKS that have "principalId":

  "identity": {
    "principalId": "00000000-e567-49e7-a579-16e8f4179568",
    "tenantId": "000000-00000-0000000-000000-000000000",
    "type": "SystemAssigned"
  },
  "identityProfile": {
    "kubeletidentity": {
      "clientId": "000000000-ab59-4151-80e4-000000000",
      "objectId": "00000000-40fd-48ac-9ce4-000000000",
      "resourceId": "/subscriptions/000000-0000000000/resourcegroups/MC_rg-demo01_aks-demo01_westeurope/providers/Microsoft.ManagedIdentity/userAssignedIdentities/aks-demo01-agentpool"
    }
  },

Should I use $aks.identity.principalId instead of $aks.servicePrincipalProfile.clientId ?

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Which access mode did you use to access the Azure Key Vault instance:
[e.g. Service Principal, Pod Identity, User Assigned Managed Identity, System Assigned Managed Identity]
Pod Identity

Environment:

• Secrets Store CSI Driver version: (use the image tag):
• Azure Key Vault provider version: (use the image tag):
• Kubernetes version: (use kubectl version):
• Cluster type: (e.g. AKS, aks-engine, etc): AKS 1.17.3

Ability to set cloud name to enable azure provider to work in Sovereign cloud.

What steps did you take and what happened:
Integrate k8s with AzureKeyVault, when fetching a PEM format certificate from the Vault the certificate chain is in the wrong order.
It seems that the server certificate is the last one in the list.
nginx is failed to start with the following msg:
nginx: [emerg] SSL_CTX_use_PrivateKey_file("/etc/nginx/intezer.pem") failed (SSL: error:0B080074:x509 certificate routines:X509_check_private_key:key values mismatch)

What did you expect to happen:
the server certificate is the first one and in the right order.

Anything else you would like to add:
in the Nginx documentation you can see it's the same error. when changing the certificates order it seems to solve the issue.

Which access mode did you use to access the Azure Key Vault instance:
System Assigned Managed Identity

Describe the solution you'd like
Right now when you specify a secret it is always downloaded using utf-8 encoding. Something like objectencoding could be added in order to be able to download correctly other file encodings.

This would be useful to upload/download binary files or jks files which cannot be converted to utf-8 More info

$ kubectl apply -f \
        nginx-pod-secrets-store-inline-volume-secretproviderclass.yaml

The Pod "nginx-keyvault-inline" is invalid:
* spec.volumes[0]: Required value: must specify a volume type
* spec.containers[0].volumeMounts[0].name: Not found: "keyvault"

What did i miss? Not sure how to enable feature gates (--feature-gates=CSIInlineVolume=true) on the api-server in AKS, how is that handled?

k8s version: 1.15.5 (on AKS)

• Image build and push
• run e2e tests

I would like to use this provider so that my ingresses can use certificates stored in KV for TLS. All the examples currently show mounting secrets as volumes, which doesn't help me.

Is k8s secrets of type TLS supported? If so, is there an example?

Configure the Azure Provider Repo to allow remote development through VS Code's Remote Containers extension.

This will allow VS Code to be used as a full-featured development environment for the Azure Provider to ease development and debugging for contributors.

Same issue as in k8s v1.15,
As I've seen in the open incident the solution was to upgrade k8s version.
still with the same issue.

Pod events:
pod has unbound immediate PersistentVolumeClaims (repeated 4 times)

PVC in status Pending

This is the configuration

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-secrets-store
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadOnlyMany
  persistentVolumeReclaimPolicy: Retain
  csi:
    driver: secrets-store.csi.k8s.io
    readOnly: true
    volumeHandle: kv
    volumeAttributes:
      providerName: "azure"
      keyvaultName: <kv_name>
      objects:  |
        array:                                        
          - |
            objectName: <secret_name>
            objectType: cert                 
            objectVersion: ""                
          - |
            objectName: <secret_name>
            objectType: secret
            objectVersion: ""
      resourceGroup: <resource_grp>
      subscriptionId: <subscription_id> 
      tenantId: <tenant-id>    
    nodePublishSecretRef:
      name: secrets-store-creds
      namespace: <ns_name>

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pvc-secrets-store
  namespace: <ns_name>
spec:
  accessModes:
    - ReadOnlyMany
  resources:
    requests:
      storage: 1Ki
  volumeName: pv-secrets-store
  storageClassName: ""

```---

kind: Deployment
apiVersion: apps/v1
metadata:
name: nginx-deployment
namespace: <ns_name>
spec:
replicas: 2
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
selector:
matchLabels:
deploy: example
template:
metadata:
labels:
deploy: example
spec:
containers:
- name: nginx
image: nginx:1.7.9
ports:
- containerPort: 80
name: http
- containerPort: 443
name: https
volumeMounts:
- name: config-volume
mountPath: /etc/nginx
volumes:
- name: kv-store
persistentVolumeClaim:
claimName: pvc-secrets-store
- name: config-volume
configMap:
name: nginx-config

apiVersion: v1
kind: Service
metadata:
name: nginx
namespace: <ns_name>
annotations:
service.beta.kubernetes.io/azure-load-balancer-internal: "true"
spec:
type: LoadBalancer
selector:
deploy: example
ports:

• port: 80
  name: http
• port: 443
  name: https

Followed directions from https://github.com/Azure/secrets-store-csi-driver-provider-azure#usage
Installed fresh today on a new cluster (v1.17.5 via Rancher) with:
helm install csi-secrets-store-provider-azure/csi-secrets-store-provider-azure --generate-name
(also tried:
helm install csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver
kubectl apply -f https://raw.githubusercontent.com/Azure/secrets-store-csi-driver-provider-azure/master/deployment/provider-azure-installer.yaml
Identical results)
All the pods deploy fine
I created Service Principle and installed as secret as directed
Deployed provider class https://github.com/kubernetes-sigs/secrets-store-csi-driver/blob/master/test/bats/tests/azure_synck8s_v1alpha1_secretproviderclass.yaml
(updated appropriate values)
The applied the example https://github.com/kubernetes-sigs/secrets-store-csi-driver/blob/master/test/bats/tests/nginx-deployment-synck8s-azure.yaml
(nginx as container)
nginx pod stuck on ContainerCreating, reporting
Warning FailedMount 7s (x7 over 39s) kubelet, temp2-2 MountVolume.SetUp failed for volume "secrets-store-inline" : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name secrets-store.csi.k8s.io not found in the list of registered CSI drivers

What did you expect to happen:
Create pod and sync secrets as described by docs

Which access mode did you use to access the Azure Key Vault instance:
Service Principal

Environment:
Add defaults as of 5/14/2020

• Secrets Store CSI Driver version: (use the image tag):
• Azure Key Vault provider version: (use the image tag):
• Kubernetes version: (use kubectl version):
• Cluster type: (e.g. AKS, aks-engine, etc):

After following the setup instructions for Azure AD pod identity I was able to confirm the pod is running with the correct identity by inspecting the mic logs.

After setting up the Azure provider, I was unable to attach a volume mount and am receiving the following error:

MountVolume.SetUp failed for volume "secrets-store-inline" : kubernetes.io/csi: mounter.SetupAt failed: rpc error: code = Unknown desc = error mounting secret time="2020-03-23T23:33:37Z" level=fatal msg="[error] : failed to get vault: failed to get management token: nmi response failed with status code: 403"

I have also confirmed that the User Assigned Managed Identity has Read Access to the secret I'm trying to mount.

Add helm charts that consume the driver helm charts as dependency chart. This will enable single helm command installation of the driver and the azure keyvault provider.

The following change -#98 has been done to align the behavior closely with the current az keyvault experience.

az keyvault certificate download provides only the certs
az keyvault secret download provides the cert and private key
az keyvault key download provides the public key

• Document how to get different parts of the cert and key
• Document backward compatibility for sync K8s secret
• Add examples

What steps did you take and what happened:
[A clear and concise description of what the bug is.]
In flexvolume we are able to provide multiple secret names as objectnames as a semicolon seperated where as in csi driver we are not able to provide multiple secrets.

Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="BadParameter" Message="The request URI contains an invalid name:

What did you expect to happen:
Able to load multiple secrets in a comma or semicolon seperated

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Which access mode did you use to access the Azure Key Vault instance:
[e.g. Service Principal, Pod Identity, User Assigned Managed Identity, System Assigned Managed Identity]

Environment:

• Secrets Store CSI Driver version: (use the image tag):
• Azure Key Vault provider version: (use the image tag):
• Kubernetes version: (use kubectl version):
• Cluster type: (e.g. AKS, aks-engine, etc):

Add new --version flag that shows the current provider version, minimum supported driver version.

// providerVersion holds current provider version
type providerVersion struct {
	Version string `json:"version"`
	Date    string `json:"date"`
	// MinDriverVersion is minimum driver version the provider works with
	MinDriverVersion string `json:"minDriverVersion"`
}

Ref: Azure/kubernetes-keyvault-flexvol#128

Add support to inject all secrets with a specific prefix into the same pod

What steps did you take and what happened:
I have configured a persistent volume based on azure-csi-driver.
When fetching a certificate from Azure KeyVault I receive the wrong format.

What did you expect to happen:
receive format suitable for Nginx HTTPS configuration

Anything else you would like to add:
PEM cert type working fine, but still don't know how to convert the cert file after the 51 PR.

Which access mode did you use to access the Azure Key Vault instance:
System Assigned Managed Identity

I am using the below things. However, I need to be able to load these secrets onto env variable. It is not working at all. Could you please help us how to achieve that?

apiVersion: v1
kind: Pod
metadata:
labels:
app: nginx-flex-kv
name: nginx-flex-kv
namespace: ns-gdprhub-kc
spec:
containers:

• name: nginx-flex-kv
  image: nginx
  imagePullPolicy: Always
  command: ["sh", "-c"]
  args: ["export KEYCLOAK_USER=$(cat /kvmnt-kc/gdpr-key-cloak-admin-user)",
  "export KEYCLOAK_PASSWORD=$(cat /kvmnt-kc/gdpr-key-cloak-admin-user-pwd)"]
  volumeMounts:
  • name: secrets-store-inline-gdprhub-kc
    mountPath: /kvmnt-kc
    readOnly: true
    volumes:
• name: secrets-store-inline-gdprhub-kc
  csi:
  driver: secrets-store.csi.k8s.io
  readOnly: true
  volumeAttributes:
  secretProviderClass: "azure-scp-gdprhub-kc"

https://github.com/Azure/secrets-store-csi-driver-provider-azure/blob/master/charts/csi-secrets-store-provider-azure/README.md

There is typo in the Readme which leads to failure of helm install.

Similar to the keyvaultobjectaliases property in the flexvol implementation, please expose property objectAlias in the CSI implementation.

This feature is needed for .NET Core nested configurations where double underscores are used as config key delimiter. Since KeyVault secret names cannot contain underscores, the alias feature is the only way to specify them.

Describe the solution you'd like
The current configuration feels a bit odd with the objects.array which is a mix of secrets & keys.

Example from README:

apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: azure-kvname
spec:
  provider: azure                   
  parameters:
    usePodIdentity: "false"         # [OPTIONAL for Azure] if not provided, will default to "false"
    useVMManagedIdentity: "false"   # [OPTIONAL available for version > 0.0.4] if not provided, will default to "false"
    userAssignedIdentityID: "client_id"  # [OPTIONAL available for version > 0.0.4] use the client id to specify which user assigned managed identity to use. If using a user assigned identity as the VM's managed identity, then specify the identity's client id. If empty, then defaults to use the system assigned identity on the VM
    keyvaultName: "kvname"          # the name of the KeyVault
    cloudName: ""          # [OPTIONAL available for version > 0.0.4] if not provided, azure environment will default to AzurePublicCloud
    objects:  |
      array:
        - |
          objectName: secret1
          objectAlias: SECRET_1     # [OPTIONAL available for version > 0.0.4] object alias
          objectType: secret        # object types: secret, key or cert
          objectVersion: ""         # [OPTIONAL] object versions, default to latest if empty
        - |
          objectName: key1
          objectAlias: ""
          objectType: key
          objectVersion: ""
    resourceGroup: "rg1"            # [REQUIRED for version < 0.0.4] the resource group of the KeyVault
    subscriptionId: "s

While it would feel more natural to be able to use this approach:

apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: azure-kvname
spec:
  provider: azure                   
  parameters:
    usePodIdentity: "false"         # [OPTIONAL for Azure] if not provided, will default to "false"
    useVMManagedIdentity: "false"   # [OPTIONAL available for version > 0.0.4] if not provided, will default to "false"
    userAssignedIdentityID: "client_id"  # [OPTIONAL available for version > 0.0.4] use the client id to specify which user assigned managed identity to use. If using a user assigned identity as the VM's managed identity, then specify the identity's client id. If empty, then defaults to use the system assigned identity on the VM
    keyvaultName: "kvname"          # the name of the KeyVault
    cloudName: ""          # [OPTIONAL available for version > 0.0.4] if not provided, azure environment will default to AzurePublicCloud
    secrets:
    - name: secret1
      alias: SECRET_1
      version: ""
    keys:
    - keys: key1
      alias: KEY_1
      version: ""
    resourceGroup: "rg1"            # [REQUIRED for version < 0.0.4] the resource group of the KeyVault
    subscriptionId: "s

Anything else you would like to add:
I'm not an expert in Kubernetes extensibility so it might be because of a technical limitation, but I would prefer the above proposal which is cleaner for people who are new to it.