cert-manager / csi-lib Goto Github PK

View Code? Open in Web Editor NEW

13.0 7.0 13.0 407 KB

A library for building CSI drivers that request certificates from cert-manager

License: Apache License 2.0

Dockerfile 0.46% Go 93.05% Python 4.29% Shell 2.20%

kubernetes

csi-lib's Introduction

cert-manager-csi-lib

A library for building CSI drivers which interact with cert-manager's CertificateRequest API.

Introduction

To provide identity documents and TLS certificates to Kubernetes Pods, a CSI driver can be used which automatically provisions, rotates and exposes certificates at a user-configured path on a filesystem.

This avoids user applications needing to understand how these identities are procured, and allows them to be fetched from any supported cert-manager issuer.

This project is first and foremost presented as a library to better support those wanting to build their own more opinionated identity provisioning drivers whilst still benefiting from the support and adoption of the cert-manager project.

For example, despite the vast configurability of cert-manager's CertificateRequest resource, you may want to restrict/dictate the options used on the certificates (and their corresponding private key). This means your security teams can be confident that these complex identity documents are being handled, configured and procured in a manner which meets the organisational goals you have in place.

Goals

This library makes it easy to create your own, potentially opinionated, CSI drivers.

It takes care of:

Implementing the CSI interface
Communicating with the Kubernetes/cert-manager API via CertificateRequests
Automatically rotating/renewing certificates near expiry
Managing private key & certificate data on disk
Exposing private key & certificate data to pods
Atomically updating written data (to avoid mismatching identity documents)

Usage

An example implementation of the CSI driver can be found in the example/ subdirectory.

This presents a highly configurable CSI driver which allows users to configure the options used when generating private keys and certificate requests using CSI volume attributes (specified in-line on a pod).

If you intend to implement your own CSI driver, the manager/interfaces.go file defines the functions and interfaces you will need to implement.

Contributing

This is a part of the cert-manager project and therefore follows the same contribution workflow.

Pull requests are welcome, however we strongly recommend creating an issue before beginning work on your change else there will likely be additional revisions/changes needed before it can be accepted.

csi-lib's People

Contributors

Stargazers

Watchers

Forkers

munnerz joshvanl sgtcodfish basert wallrj isabella232 7ing davidsbond jsoref irbekrm inteon

csi-lib's Issues

Failed to read metadata: not found

We deployed the csi-driver to two Kubernetes 1.19 clusters. They periodically start spamming errors every second that the csi-volume is not found anymore.
Example, it repeats for every volume every 1 second):

E1127 11:09:00.005373 1 manager.go:485] manager "msg"="Failed to read metadata" "error"="reading metadata file: not found" "volume_id"="csi-c62ccf515ec64bae8f06fdb36d4a2079e3f2c5aecc91c7ef14cda485836c8bb8"
E1127 11:09:00.011191 1 manager.go:485] manager "msg"="Failed to read metadata" "error"="reading metadata file: not found" "volume_id"="csi-437664f63a3af2714f91ee6f7c28ce39bb2810d338a9ec0652d87a1b0ebf7cdf"
E1127 11:09:00.025988 1 manager.go:485] manager "msg"="Failed to read metadata" "error"="reading metadata file: not found" "volume_id"="csi-fbca1dd6a8e7cde2140d3c8b401f0551791d42dd6a84b6536e6dc438928388ea"
E1127 11:09:00.031470 1 manager.go:485] manager "msg"="Failed to read metadata" "error"="reading metadata file: not found" "volume_id"="csi-89ebfc0f44740cf55bcc32dc92121ac7dc9fc9d88c608343c76322fc2c4b5347"
E1127 11:09:00.042389 1 manager.go:485] manager "msg"="Failed to read metadata" "error"="reading metadata file: not found" "volume_id"="csi-40dcbbf685f818a191215fe4b0409d296303dee1c2482a4c6855e1bcf79ad0fc"
E1127 11:09:00.052478 1 manager.go:485] manager "msg"="Failed to read metadata" "error"="reading metadata file: not found" "volume_id"="csi-db647efd61631afe7203e2291292245b01f000ca82d6d88e8f59befd6e05df6d"
E1127 11:09:00.058340 1 manager.go:485] manager "msg"="Failed to read metadata" "error"="reading metadata file: not found" "volume_id"="csi-47485531b1cad209a46179af6f7352e3f9abb2e45fd0de2b73ab5f3c8d9aa543"
E1127 11:09:00.062959 1 manager.go:485] manager "msg"="Failed to read metadata" "error"="reading metadata file: not found" "volume_id"="csi-347ba22bc8bd0a6a447e652b06adb1e47bf441ea03d871471558662e1b30f6a9"
E1127 11:09:00.123721 1 manager.go:485] manager "msg"="Failed to read metadata" "error"="reading metadata file: not found" "volume_id"="csi-82a36d963d482dcfd15293947e5a21478db708abf746c5bb5aab1bfa9bee1f5d"
E1127 11:09:00.135457 1 manager.go:485] manager "msg"="Failed to read metadata" "error"="reading metadata file: not found" "volume_id"="csi-ff9562742144c98ee8cc34d8e2f4f65b4361523b2841f762a32a9f404807725d"
E1127 11:09:00.146480 1 manager.go:485] manager "msg"="Failed to read metadata" "error"="reading metadata file: not found" "volume_id"="csi-494fb192617fe461bc54d6009a53f7b5182a6d234295c224a4cf6b70a890b770"
E1127 11:09:00.147540 1 manager.go:485] manager "msg"="Failed to read metadata" "error"="reading metadata file: not found" "volume_id"="csi-86d15b62ff043b1b46dbaff93959f889a8927006dc41aa24f77aadc1f2744a67"
E1127 11:09:00.156740 1 manager.go:485] manager "msg"="Failed to read metadata" "error"="reading metadata file: not found" "volume_id"="csi-ac8085a7d8cb7cf81d5c2a0f344844d588f170decba4ef6f8316ae428e4be092"
E1127 11:09:00.163012 1 manager.go:485] manager "msg"="Failed to read metadata" "error"="reading metadata file: not found" "volume_id"="csi-cebc70d6bfa48aae23f68b5a7e39281092572d4c62f4570fd4853cba52ee96be"
E1127 11:09:00.207421 1 manager.go:485] manager "msg"="Failed to read metadata" "error"="reading metadata file: not found" "volume_id"="csi-540a55e995ddd32ac13f5317e6af85b4013b96a83a07d0a82808f5b0e951f87f"
E1127 11:09:00.208836 1 manager.go:485] manager "msg"="Failed to read metadata" "error"="reading metadata file: not found" "volume_id"="csi-363f986cf5d931c54c02296a8aada9c0b7d686acce842966e00dce64f9eda5e1"
E1127 11:09:00.210545 1 manager.go:485] manager "msg"="Failed to read metadata" "error"="reading metadata file: not found" "volume_id"="csi-a3e712107ff31e899a44c6bde807d55ab6433f27b7c493a84bc28324451031ba"
E1127 11:09:00.235020 1 manager.go:485] manager "msg"="Failed to read metadata" "error"="reading metadata file: not found" "volume_id"="csi-f7086201ca296691f705ffb8938c8e62f0b01a812b440933c0ce886f964e4ac3"
E1127 11:09:00.235027 1 manager.go:485] manager "msg"="Failed to read metadata" "error"="reading metadata file: not found" "volume_id"="csi-93772e80b03d346265878f48e02aaa4630ed11bdb16d29581157511c86c4e14a"
E1127 11:09:00.238896 1 manager.go:485] manager "msg"="Failed to read metadata" "error"="reading metadata file: not found" "volume_id"="csi-de0ef9825bb0038c962e3ac59854d5e9d3d7df2a10d0e680f6449b84b62d0283"
E1127 11:09:00.252404 1 manager.go:485] manager "msg"="Failed to read metadata" "error"="reading metadata file: not found" "volume_id"="csi-e4fbd00c2f89bc9871053f815e5e7d7e026c33ebdfa07b3028e05c2fb8e28d01"
E1127 11:09:00.269231 1 manager.go:485] manager "msg"="Failed to read metadata" "error"="reading metadata file: not found" "volume_id"="csi-0c3934c55cecd8e037f648c0a1050754df229e444611b6105757a9a66bf94de5"
E1127 11:09:00.275499 1 manager.go:485] manager "msg"="Failed to read metadata" "error"="reading metadata file: not found" "volume_id"="csi-3ae19fa7ef76d689b521e2e1d1efe49c859f09df0a7b7286bb84cdc1d841f8c5"
E1127 11:09:00.276340 1 manager.go:485] manager "msg"="Failed to read metadata" "error"="reading metadata file: not found" "volume_id"="csi-b00704b29c77b11b3478cec51dc4f5d4e55de0023d6ddfd99da14abe2da8d1b5"
E1127 11:09:00.279666 1 manager.go:485] manager "msg"="Failed to read metadata" "error"="reading metadata file: not found" "volume_id"="csi-5a3693f3802878bfa0fb8ec3e05e40091c73af80bc0b33b19595a05c5d359f3e"
E1127 11:09:00.293034 1 manager.go:485] manager "msg"="Failed to read metadata" "error"="reading metadata file: not found" "volume_id"="csi-d36ccbb820903a32f472d693bf3f3450acfe76c02bca74149f5281c14fa854ba"
E1127 11:09:00.297746 1 manager.go:485] manager "msg"="Failed to read metadata" "error"="reading metadata file: not found" "volume_id"="csi-f593159ed5d2d058120e4e619525aa302adc96e26afc9a89e118c2d2f4b04374"
E1127 11:09:00.309292 1 manager.go:485] manager "msg"="Failed to read metadata" "error"="reading metadata file: not found" "volume_id"="csi-d3a864f7401a7c30919f0f38feabf20eb77d0bd45a59867fe4975a4f2cbc21b5"
E1127 11:09:00.311891 1 manager.go:485] manager "msg"="Failed to read metadata" "error"="reading metadata file: not found" "volume_id"="csi-c75c098855b8cc4bd41d1901602221fc3a28b17a37cf75a7e09751852c1fa71b"
E1127 11:09:00.335159 1 manager.go:485] manager "msg"="Failed to read metadata" "error"="reading metadata file: not found" "volume_id"="csi-4e2839b802be2de062704ea22bea70ffc7545c5c256da3f857111d10cf46c373"
E1127 11:09:00.371723 1 manager.go:485] manager "msg"="Failed to read metadata" "error"="reading metadata file: not found" "volume_id"="csi-2abae01a959c965615605d1e137632bf539deb5c97faff2745aa4f068dfb3f9c"
E1127 11:09:00.385678 1 manager.go:485] manager "msg"="Failed to read metadata" "error"="reading metadata file: not found" "volume_id"="csi-0837b8176fe47dee37d0170d82404366154c69b4d3104d7d3fb2859e333aa8ea"
E1127 11:09:00.397648 1 manager.go:485] manager "msg"="Failed to read metadata" "error"="reading metadata file: not found" "volume_id"="csi-9b2aa410fd23f0660d43f63a6d2145cd8357323bb453bcb179c456dbc72c6fc5"
E1127 11:09:00.431602 1 manager.go:485] manager "msg"="Failed to read metadata" "error"="reading metadata file: not found" "volume_id"="csi-92b6762220d76af9b94906a5fb546169a82cb3175b97a6c7fb4ed788b224703a"
E1127 11:09:00.463203 1 manager.go:485] manager "msg"="Failed to read metadata" "error"="reading metadata file: not found" "volume_id"="csi-d0b5dc1da6a63f14cc432aba61d1719f03b5910a369a2abba011e258d9f9d1b0"
E1127 11:09:00.473169 1 manager.go:485] manager "msg"="Failed to read metadata" "error"="reading metadata file: not found" "volume_id"="csi-2116c0023d6e6119ede6db38294598eeab901634280ed79d25489ed7b74ddaee"
E1127 11:09:00.474435 1 manager.go:485] manager "msg"="Failed to read metadata" "error"="reading metadata file: not found" "volume_id"="csi-c1358f0bae71b3f1c232f71c045ed25419c6426969b97a3f536fb4553224b5d4"
E1127 11:09:00.497268 1 manager.go:485] manager "msg"="Failed to read metadata" "error"="reading metadata file: not found" "volume_id"="csi-dd46f10d92085e4bda8569d39df1c0c191ea25208febf8b711bba7b190baa3ab"
E1127 11:09:00.521907 1 manager.go:485] manager "msg"="Failed to read metadata" "error"="reading metadata file: not found" "volume_id"="csi-8ddcc0fa1c558237bbe59998061bb102704451488f34c8f8145673eb20df7994"

Whats interesting: Both clusters start erroring around the same time after they have been restarted. I'm not sure if that's a coincidence, as they are both managed by AWS EKS, or if it's related to the drivers being running for the same amount of time.

Uptime of each driver instance is around 5-10 days.

CertificateRequest should be in terminate state when it is denied by an approver

// Handle cases where the request has been explicitly denied
if apiutil.CertificateRequestIsDenied(updatedReq) {
  cond := apiutil.GetCertificateRequestCondition(updatedReq, cmapi.CertificateRequestConditionDenied)
  return false, fmt.Errorf("request %q has been denied by the approval plugin: %s", updatedReq.Name, cond.Message)
}

Observed behavior:
When CertificateRequest is denied by an approver, the lib will delete it and recreate one. If the approver is auto approve/deny the CSR based on some policies, this would create an infinite loop: create -> denied -> delete -> create again -> ...

Sample logs:

....
I0823 16:00:25.368111       1 nodeserver.go:76] driver "msg"="Registered new volume with storage backend" "pod_name"="csi-64d84b77c-vfblf"
I0823 16:00:25.368202       1 nodeserver.go:86] driver "msg"="Volume registered for management" "pod_name"="csi-64d84b77c-vfblf"
I0823 16:00:25.368376       1 nodeserver.go:97] driver "msg"="Waiting for certificate to be issued..." "pod_name"="csi-64d84b77c-vfblf"
I0823 16:00:26.368756       1 manager.go:514] manager "msg"="Triggering new issuance" "volume_id"="csi-1d842b8307fb94812b128fc0dad4284e119e757ca39bbc164a6a693e8cc2cb80"
I0823 16:00:26.368811       1 manager.go:249] manager "msg"="Processing issuance" "volume_id"="csi-1d842b8307fb94812b128fc0dad4284e119e757ca39bbc164a6a693e8cc2cb80"
I0823 16:00:26.385125       1 manager.go:379] manager "msg"="Deleted CertificateRequest resource" "name"="c7722b4d-cab5-4709-99ed-1576fd4f705e" "namespace"="poc" "volume_id"="csi-1d842b8307fb94812b128fc0dad4284e119e757ca39bbc164a6a693e8cc2cb80"
I0823 16:00:26.503681       1 manager.go:287] manager "msg"="Created new CertificateRequest resource" "volume_id"="csi-1d842b8307fb94812b128fc0dad4284e119e757ca39bbc164a6a693e8cc2cb80"
E0823 16:00:27.512440       1 manager.go:516] manager "msg"="Failed to issue certificate, retrying after applying exponential backoff" "error"="waiting for request: request \"560d3fde-78c0-4d64-9333-3663a672e7a2\" has been denied by the approval plugin: issuer is unauthorized" "volume_id"="csi-1d842b8307fb94812b128fc0dad4284e119e757ca39bbc164a6a693e8cc2cb80"
I0823 16:00:29.711691       1 manager.go:514] manager "msg"="Triggering new issuance" "volume_id"="csi-1d842b8307fb94812b128fc0dad4284e119e757ca39bbc164a6a693e8cc2cb80"
I0823 16:00:29.720993       1 manager.go:249] manager "msg"="Processing issuance" "volume_id"="csi-1d842b8307fb94812b128fc0dad4284e119e757ca39bbc164a6a693e8cc2cb80"
I0823 16:00:29.731453       1 manager.go:379] manager "msg"="Deleted CertificateRequest resource" "name"="560d3fde-78c0-4d64-9333-3663a672e7a2" "namespace"="poc" "volume_id"="csi-1d842b8307fb94812b128fc0dad4284e119e757ca39bbc164a6a693e8cc2cb80"
I0823 16:00:29.807662       1 manager.go:287] manager "msg"="Created new CertificateRequest resource" "volume_id"="csi-1d842b8307fb94812b128fc0dad4284e119e757ca39bbc164a6a693e8cc2cb80"
E0823 16:00:30.808675       1 manager.go:516] manager "msg"="Failed to issue certificate, retrying after applying exponential backoff" "error"="waiting for request: request \"fba62e01-28f9-4644-9f5f-694eaa174084\" has been denied by the approval plugin: issuer is unauthorized" "volume_id"="csi-1d842b8307fb94812b128fc0dad4284e119e757ca39bbc164a6a693e8cc2cb80"
....

Expected behavior:
When CertificateRequest is denied, should be in terminate state. No new object should be created. Or at least has a flag to stop this infinite loop.

A possible solution is to return true in line: https://github.com/cert-manager/csi-lib/blob/main/manager/manager.go#L308

@munnerz

Support prometheus metrics

Similar to cert-manager controllers (https://cert-manager.io/docs/devops-tips/prometheus-metrics/#scraping-metrics), csi-lib should expose some key metrics for promethues to scrap, like:

CertificateRequest status (number issued, approved/denied, failed, etc)
```
certmanager_certificate_ready_status
```
Near expiry CertificateRequests (how many dates left to renew)
```
certmanager_certificate_expiration_timestamp_seconds
```

See related discussion in #51 (comment)

CertificateRequests created by the csi driver are not garbage collected due to incorrect apiVersion

When the pods are deleted, the corresponding CertificateRequest resources are supposed to be cleaned up by the Kubernetes garbage collector controller in the background, based on the ownerReferences. But, the background delete by gc isn't happening due to incorrect apiVersion for Pods here:
https://github.com/cert-manager/csi-lib/blob/v0.3.0/manager/manager.go#L414

Kubernetes version: v1.24.4
csi-lib version: 0.3.0

kube-controller-manager gc controller logs(anonymized) shows that the GC picks up the orphan CertificateRequest, but never deletes it, and below logs keep repeating for such an orphan object:

kube-controller-manager[3877594]: I0820 11:01:22.657996 3877594 garbagecollector.go:496] "Processing object" object="default/aaaaa-d5b687789-vfb57" objectUID=43e8b254-8667-4c3a-b5a7-9ae7329b75a9 kind="Pod" virtual=true
kube-controller-manager[3877594]: I0820 11:18:02.671261 3877594 garbagecollector.go:496] "Processing object" object="default/aaaaa-d5b687789-vfb57" objectUID=43e8b254-8667-4c3a-b5a7-9ae7329b75a9 kind="Pod" virtual=true
kube-controller-manager[3877594]: I0820 11:34:42.684420 3877594 garbagecollector.go:496] "Processing object" object="default/aaaaa-d5b687789-vfb57" objectUID=43e8b254-8667-4c3a-b5a7-9ae7329b75a9 kind="Pod" virtual=true
....

I tested by editing the ownerReferences of an orphan CertificateRequest's apiVersion from core/v1 to v1 and restarting the kube-controller-manager to trigger a gc graph rebuild. that orphan CR resource was cleaned up instantly in the next gc controller sync.

cc @munnerz

Driver won't recover if a volume does not contain a metadata file

Problem Statement

In a rare case, if a volume does not contain a metadata file, the driver won't start up (keep crashing) with manager.NewManagerOrDie. This is because the driver tries to ensure all volumes are accessible. If not, it will return an error that cause a panic.

csi-lib/storage/filesystem.go

Lines 112 to 122 in 9811918

 for _, dir := range dirs { 

 file, err := f.fs.Open(f.metadataPathForVolumeID(dir.Name())) 

 if err != nil { 

 // discovered a volume/directory that does not contain a metadata file 

 // TODO: log this error to allow startup to continue 

 return nil, err 

 } 

 // immediately close the file as we just need to verify it exists 

 file.Close() 

 vols = append(vols, dir.Name()) 

 }

Here is a sample error code we have seen:

panic: failed to start manager: listing existing volumes: open csi-data-dir/inmemfs/csi-04b4520253413b4a3120e28f454cc781c11a759627c25a8bbd3f536e8c1c2020/metadata.json: no such file or directory

goroutine 1 [running]:
github.com/cert-manager/csi-lib/manager.NewManagerOrDie({{0x20d1870, 0xc000490450}, 0x0, {0x20cb308, 0xc0004c5b80}, {0x20e4388, 0x3023568}, 0xc000490588, 0x0, {0x7ffdf920c895, ...}, ...})
	/go/pkg/mod/github.com/cert-manager/[email protected]/manager/manager.go:237 +0xa5

This situation could happen when Pod is created --> NodePublishVolume called --> volume created --> OOM. Thus following defer function would never be called:

csi-lib/driver/nodeserver.go

Lines 56 to 62 in 9811918

 defer func() { 

 if !success { 

 ns.manager.UnmanageVolume(req.GetVolumeId()) 

 _ = ns.mounter.Unmount(req.GetTargetPath()) 

 _ = ns.store.RemoveVolume(req.GetVolumeId()) 

 } 

 }()

The Fix

To fix this issue, we may simply log / delete the volume and proceed the rest of jobs.

[bug] labelSelectorForVolume fail to add labels

We notice there is only one CertificateRequest object exists for a pod with two CSI volumes.
From the watch log, it seems we had two CSRs, but one got purged somehow.

The root cause is:

csi-lib/manager/manager.go

Line 391 in 2f2e205

sel.Add(*req)

sel := labels.NewSelector()
req, err := labels.NewRequirement(...)
sel.Add(*req) // this won't add req to the selectors...

simple fix is like:

sel = sel.Add(*req)

Race condition: CertificateRequests may never be fulfilled if the issuer was overwhelmed

Problem

In a large cluster (1k+ pods, each with multiple CSI volumes for client certs), during initial startup, most of new pods didn't get their certs issued. When checking the driver logs, we observed the driver creating/deleting the requests over and over again. If we scale up the cluster, things are getting worse, as some pods are never up.

Reproduce

We could reproduce it by faking a long waiting work-queue for CSR controller:

Install cert-manager, cert-manager-csi-driver, and example-app
Bring down all pods: kubectl -n sandbox scale --replicas 0 deployment my-csi-app
Disable CSR controller: kubectl -n cert-manager scale --replicas 0 deployment.apps/cert-manager
Bring up one pod: kubectl -n sandbox scale --replicas 1 deployment my-csi-app
- Check associated cert-manager-csi-driver pod logs, we noticed a new CR object will be created every 60s, and the old CR obj will be deleted at the same time.

$ kubectl -n sandbox get cr --watch
NAME                                   APPROVED   DENIED   READY   ISSUER              REQUESTOR                                                    AGE
ca-issuer-mb5mz                        True                True    selfsigned-issuer   system:serviceaccount:cert-manager:cert-manager              7m42s
e21c1ea5-02b5-4628-97f5-d59dc21cada6                               ca-issuer           system:serviceaccount:cert-manager:cert-manager-csi-driver   27s
e21c1ea5-02b5-4628-97f5-d59dc21cada6                               ca-issuer           system:serviceaccount:cert-manager:cert-manager-csi-driver   64s
b1585e95-48a9-4bcf-bc7b-99bb72450362                               ca-issuer           system:serviceaccount:cert-manager:cert-manager-csi-driver   0s
b1585e95-48a9-4bcf-bc7b-99bb72450362                               ca-issuer           system:serviceaccount:cert-manager:cert-manager-csi-driver   68s
4d6bc09b-9ef2-40c3-a4a0-7f6303b1330f                               ca-issuer           system:serviceaccount:cert-manager:cert-manager-csi-driver   0s

Sample logs:

I0225 00:00:56.220170       1 nodeserver.go:83] driver "msg"="Registered new volume with storage backend" "pod_name"="my-csi-app-7777dc4568-plmlk"
I0225 00:00:56.220495       1 manager.go:302] manager "msg"="Processing issuance" "volume_id"="csi-92e6eb2c6bfd04e120305b10fe32322f6aff0537cf747360444d060bff180c0b"
I0225 00:00:56.439959       1 manager.go:340] manager "msg"="Created new CertificateRequest resource" "volume_id"="csi-92e6eb2c6bfd04e120305b10fe32322f6aff0537cf747360444d060bff180c0b"
E0225 00:01:56.229502       1 server.go:109] driver "msg"="failed processing request" "error"="waiting for request: request \"1b56a7f2-a4fc-42c1-bac1-b716249fbd89\" has not yet been approved by approval plugin" "request"={} "rpc_method"="/csi.v1.Node/NodePublishVolume"
I0225 00:01:56.819356       1 nodeserver.go:83] driver "msg"="Registered new volume with storage backend" "pod_name"="my-csi-app-7777dc4568-plmlk"
I0225 00:01:56.819546       1 manager.go:302] manager "msg"="Processing issuance" "volume_id"="csi-92e6eb2c6bfd04e120305b10fe32322f6aff0537cf747360444d060bff180c0b"
I0225 00:01:56.836447       1 manager.go:467] manager "msg"="Deleted CertificateRequest resource" "name"="1b56a7f2-a4fc-42c1-bac1-b716249fbd89" "namespace"="sandbox" "volume_id"="csi-92e6eb2c6bfd04e120305b10fe32322f6aff0537cf747360444d060bff180c0b"
I0225 00:01:57.263983       1 manager.go:340] manager "msg"="Created new CertificateRequest resource" "volume_id"="csi-92e6eb2c6bfd04e120305b10fe32322f6aff0537cf747360444d060bff180c0b"

Root Cause

Because a large amount of CertificateRequest are queued in the controller and never be processed in time (within 60s), the driver is timeout and do a retry, which will delete the old CR. and recreate a new one for another 60s.

This inline with following logic in ManageVolumeImmediate(ctx) method:

call issue() once, wait up to 60s, which is defined in this line:

csi-lib/driver/nodeserver.go

Line 50 in 8ebb8cd

ctx, cancel := context.WithTimeout(ctx, time.Second*60)
unmount volume if failed -> NodePublishVolume() call will be retried -> retry ManageVolumeImmediate(ctx)

To Fix

For this issue, the main problem is CSR controller workers never catch up CertificateRequest object creation/deletion speed (every 60s). An easy solution is, do NOT delete stale requests if they are never proceed by any worker (aka. has no conditions in status field). So it will be picked up by a worker eventually.

To further extend this idea, shall we apply it on all CRs that have transient errors or in pending status ? Especially, if a CR has already been approved manually, but never being issued within 60s (current timeout value) for some reasons (e.g., out band issuer, long worker queue, etc.), this CR will be deleted and a new CR will be created, manual approval might be required again and again.

Expect ManageVolumeImmediate() is a blocking call that waits for certificate to be issued

ManageVolumeImmediate() is introduced in #35. After this PR, we observed inconsistent behaviors in following settings:

Driver's ContinueOnNotReady is set to true
Manager's ReadyToRequestFunc is set as AlwaysReadyToRequest

Before

New pod won't come up until the certificate is issued / written to host path.
Relevant code in v0.3.0:

csi-lib/driver/nodeserver.go

Lines 96 to 102 in 8e344a2

 if isReadyToRequest || !ns.continueOnNotReady { 

 log.Info("Waiting for certificate to be issued...") 

 if err := wait.PollUntil(time.Second, func() (done bool, err error) { 

 return ns.manager.IsVolumeReady(req.GetVolumeId()), nil 

 }, ctx.Done()); err != nil { 

 return nil, err 

 }

Now

New pod can come up before the certificate is issued / written to host path.
Relevant code in main:

meta.NextIssuanceTime is set to epoch time.

csi-lib/driver/nodeserver.go

Lines 71 to 77 in 7dfb05b

 // If continueOnNotReady is enabled, set the NextIssuanceTime in the metadata file to epoch. 

 // This allows the manager to start management for the volume again on restart if the first 

 // issuance did not successfully complete. 

 if meta.NextIssuanceTime == nil && ns.continueOnNotReady { 

 epoch := time.Time{} 

 meta.NextIssuanceTime = &epoch 

 }

ManageVolumeImmediate is called.

csi-lib/driver/nodeserver.go

Lines 95 to 100 in 7dfb05b

 if isReadyToRequest { 

 log.V(4).Info("Waiting for certificate to be issued...") 

 if _, err := ns.manager.ManageVolumeImmediate(ctx, req.GetVolumeId()); err != nil { 

 return nil, err 

 } 

 log.Info("Volume registered for management")

cert is issued asynchronously in startRenewalRoutine call.

csi-lib/manager/manager.go

Lines 559 to 570 in 7dfb05b

 // Only attempt issuance immediately if there isn't already an issued certificate 

 if meta.NextIssuanceTime == nil { 

 // If issuance fails, immediately return without retrying so the caller can decide 

 // how to proceed depending on the context this method was called within. 

 if err := m.issue(ctx, volumeID); err != nil { 

 return true, err 

 } 

 } 

 if !m.startRenewalRoutine(volumeID) { 

 return true, fmt.Errorf("unexpected state: renewal routine not started, please open an issue at https://github.com/cert-manager/csi-lib") 

 }

Thoughts

In most cases, this won't be an issue at all. But this inconsistent behavior may break some deployments that don't have checks on cert files (e.g., using init-container). Sample manifest that would get impact on this:

...
spec:
  containers:
  - name: sample-app
     ...
     volumeMounts:
     - mountPath: /path/to/file/tls.crt
        name: csi-cert
        subPath: tls.crt
  volumes:
  - csi:
        driver: sample.csi.driver
        readOnly: true
        name: csi-cert
...

Since kubelet cannot find the file, it will throw following error and prevent container from starting up:

state:
      waiting:
        message: 'failed to start container "ca79b3d8da42ce7f2d92409d0e52260005b1b02a1231997ad3679db95358128f":
          Error response from daemon: failed to create shim task: OCI runtime create
          failed: runc create failed: unable to start container process: error during
          container init: error mounting "/var/lib/kubelet/pods/449f2b9e-e0c5-4781-8177-626984f324bd/volume-subpaths/csi-cert/sample-app/1"
          to rootfs at "/path/to/file/tls.crt": mount /var/lib/kubelet/pods/449f2b9e-e0c5-4781-8177-626984f324bd/volume-subpaths/csi-cert/sample-app/1:/path/to/file/tls.crt
          (via /proc/self/fd/6), flags: 0x5001: not a directory: unknown: Are you
          trying to mount a directory onto a file (or vice-versa)? Check if the specified
          host path exists and is the expected type'
        reason: RunContainerError

Allow data-root to be an absolute path

Currently, because the data root field on the filesystem/storage struct is plumbed through the new Golang FS interface, it does not support absolute paths.

This makes it confusing for users when configuring their CSI driver deployments, as a relative path must be used (meaning they must have knowledge of the working directory configured in the container).

At the moment, because the working directory is /, this means that users have to strip the leading / from the data directory argument when starting their CSI driver.

/kind bug
/priority important-soon

Exponential backoff handling does not apply to certificate renewal in pending phase

Problem

When a certificate get renewal, the driver will generate a new CertificateRequest and wait for it being ready. If there is an error, the driver will do a retry in an exponential backoff pattern. This feature is brought by #35.
However, if the CertificateRequest is in a pending state (e.g., not being handled by CSR controller, not approved, or approved but not issued yet, etc.), the driver will wait infinitely instead of exponential backoff retry. This behavior is not inline with the above failure situation.

Reproduce

We could reproduce it by faking a long waiting work-queue for CSR controller:

Install cert-manager, cert-manager-csi-driver, and example-app
Add csi.cert-manager.io/duration: "1m" to csi volumeAttribute in the above example-app, to reduce renewal interval.
- Check associated cert-manager-csi-driver pod logs, you may observe "Triggering new issuance" message every 40s (2/3 of duration time).
Now run kubectl -n cert-manager scale --replicas 0 deployment.apps/cert-manager to disable CSR controller.
- Check associated cert-manager-csi-driver pod logs, no additional message after "Triggering new issuance". That's because the driver is in an infinite waiting loop.

Root Cause

In startRenewalRoutine() method, issue() will be called upon renewal time, controlled by RenewalBackoffConfig

each FAILED call will be retried with an interval time set by RenewalBackoffConfig: e.g., 10s, 20s, 40s, ...,
each PENDING call will wait until ctx.Timeout reached

However, in this case, ctx.Timeout is undefined as we use ctx from this line:

csi-lib/manager/manager.go

Lines 608 to 609 in 6284a0d

 // Create a context that will be cancelled when the stopCh is closed 

 ctx, cancel := context.WithCancel(context.Background())

Therefore, if ctx has no timeout, the issue(ctx) call will wait until the CertificateRequest is in Failed or Ready condition. In our example here, since CertificateRequest is never being processed, so it is considered as PENDING, which won't end at all.

To Fix

Each retry in startRenewalRoutine() should have a timeout (configurable), otherwise we won't see next retry until previous one return an error.

Optional auto rotating/renewing certificates

One of the key features of csi-lib is Automatically rotating/renewing certificates near expiry.
But do we consider make this feature optional?

One particular use case is: one-time and short-lived cert for init-container (for mTLS to pull some secrets).
The user container will no longer need the cert after consuming it. Since the pod is still running, csi-lib will continue the renewal logic for this short-lived cert. With least privilege guidance, shall we disable the renewal in this case ?

Upon checking the code, all certificates will be auto renewed once it hits the NextIssuanceTime:
https://github.com/cert-manager/csi-lib/blob/v0.3.0/manager/manager.go#L499
A workaround is to set the NextIssuanceTime much longer than the pod lifetime. But add an option here (pass in as volumeAttribute) would be much cleaner logic.

Helper function to pass token request with specified audience (list)

The reference implementation in util.ClientForMetadataTokenRequestEmptyAud only returns a rest client with an empty audience token request.

May consider add another helper function ClientForMetadataTokenRequestWithAud to support specified audience (list).

Create e2e test to validate CertificateRequest garbage collection

See #31

We should write an e2e test to ensure CertificateRequests are garbage collected by cm when the owning Pod is deleted.

First however, we need to setup our e2e testing framework.

/assign

Drivers can create CertificateRequests for pods that don't exist in very rare edge cases

In the NodePublishVolume call, we have a defer that calls UnmanageVolume (and deletes metadata from the storage backend) if initial issuance fails:

csi-lib/driver/nodeserver.go

Lines 51 to 61 in 031cdcd

 // clean up after ourselves if provisioning fails. 

 // this is required because if publishing never succeeds, unpublish is not 

 // called which leaves files around (and we may continue to renew if so). 

 success := false 

 defer func() { 

 if !success { 

 ns.manager.UnmanageVolume(req.GetVolumeId()) 

 _ = ns.mounter.Unmount(req.GetTargetPath()) 

 _ = ns.store.RemoveVolume(req.GetVolumeId()) 

 } 

 }()

If the driver is stopped during this initial 30s period, and the pod is also deleted whilst the driver is stopped, because the publish step never succeeded, the UnpublishVolume step will never be called in future.

Upon the driver starting up again, it will read the metadata.json file and then attempt to request the certificate for that pod again.

Because the pod no longer exists, the UnpublishVolume step will never be called and therefore the certificate data will never be cleaned up on disk (and the driver will continue to process renewals for the volume indefinitely, until an administrator manually cleans up the metadata file on disk and triggers a restart of the driver).

We should do whatever we can to avoid this state occurring, as it causes excessive churn on nodes, in the apiserver, and for signers.

One option would be, on startup of the driver, if metadata.json files that do not have a nextIssuanceTime set on them are found (which implies the issuance has never succeeded), delete this data/clean it up on disk and await NodePublishVolume being called again (for the case where the pod does still exist and is waiting to startup). There may be some edge cases we have not thought of however, whereby the pod does still exist and is provisioned and for some reason this field is not set. Though I don't think that is a possible state to get into...

	for _, dir := range dirs {
	file, err := f.fs.Open(f.metadataPathForVolumeID(dir.Name()))
	if err != nil {
	// discovered a volume/directory that does not contain a metadata file
	// TODO: log this error to allow startup to continue
	return nil, err
	}
	// immediately close the file as we just need to verify it exists
	file.Close()
	vols = append(vols, dir.Name())
	}

	defer func() {
	if !success {
	ns.manager.UnmanageVolume(req.GetVolumeId())
	_ = ns.mounter.Unmount(req.GetTargetPath())
	_ = ns.store.RemoveVolume(req.GetVolumeId())
	}
	}()

	if isReadyToRequest \|\| !ns.continueOnNotReady {
	log.Info("Waiting for certificate to be issued...")
	if err := wait.PollUntil(time.Second, func() (done bool, err error) {
	return ns.manager.IsVolumeReady(req.GetVolumeId()), nil
	}, ctx.Done()); err != nil {
	return nil, err
	}

	// If continueOnNotReady is enabled, set the NextIssuanceTime in the metadata file to epoch.
	// This allows the manager to start management for the volume again on restart if the first
	// issuance did not successfully complete.
	if meta.NextIssuanceTime == nil && ns.continueOnNotReady {
	epoch := time.Time{}
	meta.NextIssuanceTime = &epoch
	}

	if isReadyToRequest {
	log.V(4).Info("Waiting for certificate to be issued...")
	if _, err := ns.manager.ManageVolumeImmediate(ctx, req.GetVolumeId()); err != nil {
	return nil, err
	}
	log.Info("Volume registered for management")

	// Only attempt issuance immediately if there isn't already an issued certificate
	if meta.NextIssuanceTime == nil {
	// If issuance fails, immediately return without retrying so the caller can decide
	// how to proceed depending on the context this method was called within.
	if err := m.issue(ctx, volumeID); err != nil {
	return true, err
	}
	}

	if !m.startRenewalRoutine(volumeID) {
	return true, fmt.Errorf("unexpected state: renewal routine not started, please open an issue at https://github.com/cert-manager/csi-lib")
	}

	// Create a context that will be cancelled when the stopCh is closed
	ctx, cancel := context.WithCancel(context.Background())

	// clean up after ourselves if provisioning fails.
	// this is required because if publishing never succeeds, unpublish is not
	// called which leaves files around (and we may continue to renew if so).
	success := false
	defer func() {
	if !success {
	ns.manager.UnmanageVolume(req.GetVolumeId())
	_ = ns.mounter.Unmount(req.GetTargetPath())
	_ = ns.store.RemoveVolume(req.GetVolumeId())
	}
	}()

cert-manager / csi-lib Goto Github PK

csi-lib's Introduction

cert-manager-csi-lib

Introduction

Goals

Usage

Contributing

csi-lib's People

Contributors

Stargazers

Watchers

Forkers

csi-lib's Issues

Problem Statement

The Fix

Problem

Reproduce

Root Cause

To Fix

Before

Now

Thoughts

Problem

Reproduce

Root Cause

To Fix

Recommend Projects

Recommend Topics

Recommend Org