azure / kubernetes-kms Goto Github PK

Describe the request

Explain why KMS Plugin for Key Vault needs it

Describe the solution you'd like

Describe alternatives you've considered

Additional context

Describe the bug

Looks like we are automatically inserting quotes to the vault name.

If users specify quotes, they'll get following error:

F0308 19:53:58.845467       1 main.go:63] failed to create server, error: failed to get vault url, error: invalid vault name: "\"kvoqznrrxkxxw56\"", must match [-a-zA-Z0-9]{3,24}

This should be documented or quotes should be ignored.

Steps To Reproduce

Expected behavior

KMS Plugin for Key Vault version

Kubernetes version

Additional context

Follow the hyphen format for flags. Deprecate configFilePath and introduce config-file-path

Hello from reading the documentation, I can see that there will be 3 new flags in the azure.json.

I'm wondering if those will be replacing the SP clientID and secret that we are currently dropping in the json file.

thanks!

Enable metrics endpoint test once v0.0.12 is released.

In the readme.ms, there is a link to the documentation for configuring this. The document link is broken (pasted below).

https://github.com/Azure/aks-engine/blob/master/docs/kubernetes/features.md#azure-key-vault-data-encryption

kube-controller-manager failed to start with the next error:
2018-07-08T14:42:28.82944156Z stderr F E0708 14:42:28.829339 1 tokens_controller.go:261] error synchronizing serviceaccount kube-system/cronjob-controller: Internal error occurred: rpc error: code = Unknown desc = failed to create key, error: storage.AccountsClient#ListKeys: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="ResourceNotFound" Message="The Resource 'Microsoft.Storage/storageAccounts/my-akv' under resource group 'my-rg' was not found."

A week ago everything worked fine. No changes in the deployment playbooks. As you see, it starts looking for a storage account with the name of the key-vault service (exists).

RSA encrypted DEKs are 256 bytes. Using AES KeyWrap to encrypt the DEKs shrinks this to 40 bytes and improves performance.

• Enable soak tests as part of nightly runs with KMS plugin

Describe the request

It would be great to encapsulate all of the bootstrap + application install functionality into a canonical helm chart that any self-managed Kubernetes + Azure environment could consume to make their cluster kms-capable.

We should integrate the above using github actions so we can publish the chart in a repo tightly coupled to this project.

Explain why KMS Plugin for Key Vault needs it

Make testing more generic across the variety of k8s + Azure flavors.

Describe the solution you'd like

Describe alternatives you've considered

Additional context

• validate gRPC connectivity
• validate the caller has encrypt and decrypt permissions on the key

Ref: kubernetes/kubernetes#78540

Describe the request
Switch to using protobuf definitions from k8s.io/apiserver/pkg/storage/value/encrypt/envelope/v1beta1 instead of keeping a local copy in the repo.

Explain why KMS Plugin for Key Vault needs it

Describe the solution you'd like

Describe alternatives you've considered

Additional context

Describe the request

Explain why KMS Plugin for Key Vault needs it

Describe the solution you'd like

Describe alternatives you've considered

Additional context

Describe the request

Explain why KMS Plugin for Key Vault needs it

Describe the solution you'd like

Describe alternatives you've considered

Additional context

Add http userAgent header to all requests with value k8s-kms-keyvault.

https://kubernetes.io/docs/tasks/administer-cluster/kms-provider/#encrypting-your-data-with-the-kms-provider

We got 429 Too Many Requests error from AAD after we restarted API Server. It lasted for quite a while and caused API Server outage since all secrets were unable to be read.

After reading the code, I find that each time KMS makes a request to KeyVault, it calls getKey first. Inside it creates a new KeyVaultClient along with a new AAD token, which means a new call to AAD later. I think that's unnecessary because you only need to create a new key when it doesn't exist. And once it's created, its version is persisted and never changes. We don't have to check it in each encryption or decryption calls.

I'm wondering if we can move the getKey call to KMS starting time and save the KeyVaultClient in KeyManagementServerService for later reuse. In this case, AAD token is also able to be reused. If you can confirm this, I'm happy to submit a PR :)

@ritazh

Describe the request

Explain why KMS Plugin for Key Vault needs it

Describe the solution you'd like

Describe alternatives you've considered

Additional context

Hi,

The documentation here https://github.com/Azure/kubernetes-kms/blob/master/manual-install.md has step-7 to restart "apiserver" but the instructions are missing on how to restart "apiserver" in azure kubernetes.

Thanks

Hi,
Recently we ran into issues with newer versions pf the k8s azure KMS plugin as acs-engine is always pulling the following image "microsoft/k8s-azure-kms:latest" when bringing up the KMS services - https://github.com/Azure/acs-engine/blob/8d7d7202a2ccb57efacb6a7f937980c147785bb8/parts/k8s/artifacts/kubernetesazurekms.service#L16

As you can imagine this is not ideal, I would suggest that we introduce some docker image versioning so that if we deploy using a specific version of acs-engine + k8s Azure KMS then deployment works as expected.

Thanks

use its own configuration

ref: kubernetes/kubernetes#77309

Describe the request

• Investigate how to handle encrypt/decrypt in disconnected scenarios (cluster has no network access)

Explain why KMS Plugin for Key Vault needs it

Describe the solution you'd like

Describe alternatives you've considered

Additional context

Hello!

In order to use KKMS now, we need to change the manager nodes manually (for now).

Is there any plan/eta on when we will be able to use it with AKS? Right now it is not possible as we can't modify the managers since they are self-managed by Azure.

Thanks! Great work!

Best regards,
Gutemberg

Describe the request

• Add metrics using otel

Explain why KMS Plugin for Key Vault needs it

Describe the solution you'd like

Describe alternatives you've considered

Additional context

Hi kubernetes kms team,

I have questions about AKS encryption at rest and hope to get some clarity on that.

We use Azure Key Vault Provider for Secrets Store CSI Driver on AKS to manage our secrets. Recently we noticed that the base64-encoded unencrypted kubernetes secrets can be accessed via kubectl commands or from azure portal. So I have done some reading on the topic and found confusing information.

• k8s official docs (link) says k8s secrets are unencrypted in etcd by default and recommends using EncryptionConfiguration with --encryption-provider-config flag.

• AKS docs (link) says Kubernetes secrets are stored in etcd, a distributed key-value store. Etcd store is fully managed by AKS and data is encrypted at rest within the Azure platform. This contradicts with the unencrypted secrets we saw from kubectl commands or from azure portal.

• This repo's README (link) mentioned AKS does encrypt secrets at rest, but keys are managed by the service and users cannot bring their own.

So my questions are:

1. What does the "encryption at rest" mean if we can see unencrypted secrets from kubectl and portal? Does it mean that secrets are encrypted at the disk level but authorized users to the AKS cluster would still be able to read the unencrypted secrets?
2. Does it mean that so we can’t bring our own encryption keys?

Thank in advance for any help or insights.

Azure/acs-engine#3677

Describe the request

Explain why KMS Plugin for Key Vault needs it

Describe the solution you'd like

Describe alternatives you've considered

Additional context

Describe the request

• We have documentation today on how to rotate the encrypt/decrypt keys for the KMS.
• Investigate if this can be done automatically

Explain why KMS Plugin for Key Vault needs it

Describe the solution you'd like

Describe alternatives you've considered

Additional context

Hi,

We're using a modified version of azure.json. This line in UpdateKMSProvider assumes providerKeyVersion is the last line in JSON as it doesn't end with a comma. This will cause syntax error when azure.json looks like:

  "providerKeyName": "k8s",
  "providerKeyVersion": "xxx"
  "maximumLoadBalancerRuleCount": 240
}

Applying string manipulation to a JSON file is usually buggy. I strongly suggest you using a standard JSON parser rather than assuming it's in an acs-engine generated format.

Azure/acs-engine#4152

Currently we only support creation of a standard sku key vault for this solution. To create HSM-protected keys, we will need to use the Premium service tier: https://docs.microsoft.com/en-us/azure/key-vault/key-vault-get-started#HSM

Describe the request

• kms upgrade test in CI

• cluster/node upgrade test in aks-engine

Explain why KMS Plugin for Key Vault needs it

Describe the solution you'd like

Describe alternatives you've considered

Additional context

Describe the request
We should move base image to distroless/static to avoid security issues
https://github.com/Azure/kubernetes-kms/blob/master/Dockerfile#L1

Will need to run as root to be able to read azure.json.

Explain why KMS Plugin for Key Vault needs it

Describe the solution you'd like

Describe alternatives you've considered

Additional context

• Upgrade k8s version from n, n+1 to n+2 with the KMS plugin enabled

pending go SDK

Describe the issue

(Though I opened this issue as a bug, but I am not sure if it's a bug or a setup issue on my side.)

I followed manual configurations and ran into an issue with Deploy the KMS plugin step.

The azure-kms-provider pod we used is the same as the one from the Deploy the KMS plugin step.

apiVersion: v1
kind: Pod
metadata:
  name: azure-kms-provider
  namespace: kube-system
  labels:
    component: azure-kms-provider
spec:
  priorityClassName: system-node-critical
  hostNetwork: true
  containers:
    - name: azure-kms-provider
      image: mcr.microsoft.com/oss/azure/kms/keyvault:v0.0.11
      imagePullPolicy: IfNotPresent
      args:
        - --listen-addr=unix:///opt/azurekms.socket             # [OPTIONAL] gRPC listen address. Default is unix:///opt/azurekms.socket
        - --keyvault-name=${KV_Name}                 # [REQUIRED] Name of the keyvault
        - --key-name=${KEY_NAME}                             # [REQUIRED] Name of the keyvault key used for encrypt/decrypt
        - --key-version=${KEY_VERSION}        # [REQUIRED] Version of the key to use
        - --log-format-json=false                               # [OPTIONAL] Set log formatter to json. Default is false.
        - --healthz-port=8787                                   # [OPTIONAL] port for health check. Default is 8787
        - --healthz-path=/healthz                               # [OPTIONAL] path for health check. Default is /healthz
        - --healthz-timeout=20s                                 # [OPTIONAL] RPC timeout for health check. Default is 20s
        - -v=1
      ports:
        - containerPort: 8787                                   # Must match the value defined in --healthz-port
          protocol: TCP
      livenessProbe:
        httpGet:
          path: /healthz                                        # Must match the value defined in --healthz-path
          port: 8787                                            # Must match the value defined in --healthz-port
        failureThreshold: 2
        periodSeconds: 10
      resources:
        requests:
          cpu: 100m
          memory: 128Mi
        limits:
          cpu: 4
          memory: 2Gi
      volumeMounts:
        - name: etc-kubernetes
          mountPath: /etc/kubernetes
        - name: etc-ssl
          mountPath: /etc/ssl
          readOnly: true
        - name: sock
          mountPath: /opt
  volumes:
    - name: etc-kubernetes
      hostPath:
        path: /etc/kubernetes
    - name: etc-ssl
      hostPath:
        path: /etc/ssl
    - name: sock
      hostPath:
        path: /opt

We get AKV10022: Invalid audience. Expected https://vault.azure.net, found: https://management.core.windows.net/ error when deploying the pod.

I0525 16:38:05.933121       1 main.go:86] Listening for connections on address: /opt/azurekms.socket
E0525 16:38:14.979721       1 server.go:50] "failed to encrypt" err="failed to encrypt, error: keyvault.BaseClient#Encrypt: Failure responding to request: StatusCode=401 -- Original Error: autorest/azure: Service returned an error. Status=401 Code=\"Unauthorized\" Message=\"AKV10022: Invalid audience. Expected https://vault.azure.net, found: https://management.core.windows.net/.\""
E0525 16:38:25.081137       1 server.go:50] "failed to encrypt" err="failed to encrypt, error: keyvault.BaseClient#Encrypt: Failure responding to request: StatusCode=401 -- Original Error: autorest/azure: Service returned an error. Status=401 Code=\"Unauthorized\" Message=\"AKV10022: Invalid audience. Expected https://vault.azure.net, found: https://management.core.windows.net/.\""
I0525 16:38:25.096147       1 main.go:120] received shutdown signal
I0525 16:38:25.096184       1 main.go:102] terminating the server

I am not sure where did the https://management.core.windows.net audience come from as we don't use that audience in our project. I searched this codebase for https://management.core.windows.net but couldn't find it used anywhere either. Any help is appreciated.

KMS Plugin for Key Vault version

v0.0.11

Kubernetes version

v1.19.7