Comments (9)
rbren
commented on September 23, 2024
FWIW I deleted the chart, waited 15 minutes 12 hours, and tried again (IIRC the rate limit resets after 10m). Sadly it didn't seem to work - still rate limited. So I'm guessing there's something else going on.
Happy to provide additional logs.
Here are the values I supplied to the budibase chart - that may help to reproduce.
services:
couchdb:
storage: 1Gi
objectStore:
storage: 1Gi
redis:
storage: 1Gi
ingress:
enabled: true
hosts:
- host: $BUDIBASE_HOST
tls: true
tlsSecret: tls-prod-cert-ghost
paths:
- path: /
pathType: Prefix
backend:
service:
name: proxy-service
port:
number: 10000
annotations:
kubernetes.io/ingress.class: nginx
certmanager.k8s.io/cluster-issuer: letsencrypt-prod
kubernetes.io/tls-acme: "true"
nginx.ingress.kubernetes.io/proxy-body-size: 5m
from csi-digitalocean.
timoreimann
commented on September 23, 2024
To my knowledge, the rate limit is not reset entirely at a discrete point in time but rather happens so fluently, with each request having its own timer. See https://docs.digitalocean.com/reference/api/api-reference/#section/Introduction/Rate-Limit for details.
The retry and backoff intervals should be controlled by the sidecars that interact with the DO CSI driver. At this point, I believe all of them should come with reasonable tradeoffs. (I think csi-snapshotter used to have a bug where the retry wasn't honored, but based on your events this should be about provisioning volumes.)
Do I understand correctly that you are using the CSI driver in a self-managed Kubernetes installation (i.e., not DOKS)? If so, could you check the CSI driver logs for how the requests space out? For continuous errors, it should eventually land at a 5 minute waiting period, which should be enough to recover from rate limiting. (Note though that rate limiting applies per API token, so if you reuse the same token then other token consumers may also contribute to hitting the rate limit more easily.)
from csi-digitalocean.
rbren
commented on September 23, 2024
Just tried again - got 429s within a few seconds of installing.
This is the CSI driver in a managed DOKS cluster.
Here are the last few minutes of logs from csi driver. Nothing really suspicious IMO.
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:54:02Z" level=info msg="checking if target is mounted" args="[-o TARGET,PROPAGATION,FSTYPE,OPTIONS -M /var/lib/kubelet/pods/5b5e44c3-5ac8-4f6e-ad02-f79445d86817/volumes/kubernetes.io~csi/pvc-2f34de0b-b85c-4141-b566-c0995f2b311e/mount -J]" cmd=findmnt host_id=275418297 region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:54:02Z" level=info msg="node capacity statistics retrieved" bytes_available=950214656 bytes_total=1023303680 bytes_used=2625536 host_id=275418297 inodes_available=65525 inodes_total=65536 inodes_used=11 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=0468c53e-75ae-11ea-a87f-0a58ac14489e volume_mode=filesystem volume_path="/var/lib/kubelet/pods/5b5e44c3-5ac8-4f6e-ad02-f79445d86817/volumes/kubernetes.io~csi/pvc-2f34de0b-b85c-4141-b566-c0995f2b311e/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:54:06Z" level=info msg="node get capabilities called" host_id=275418297 method=node_get_capabilities node_capabilities="[rpc:<type:STAGE_UNSTAGE_VOLUME > rpc:<type:EXPAND_VOLUME > rpc:<type:GET_VOLUME_STATS > ]" region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:54:06Z" level=info msg="node get volume stats called" host_id=275418297 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=4ced3b86-75c9-11ea-969f-0a58ac144db2 volume_path="/var/lib/kubelet/pods/c5007016-9111-453f-a529-d1ee0513ae93/volumes/kubernetes.io~csi/pvc-e80f4df6-70d1-49d2-bd12-f9653ac62382/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:54:06Z" level=info msg="checking if target is mounted" args="[-o TARGET,PROPAGATION,FSTYPE,OPTIONS -M /var/lib/kubelet/pods/c5007016-9111-453f-a529-d1ee0513ae93/volumes/kubernetes.io~csi/pvc-e80f4df6-70d1-49d2-bd12-f9653ac62382/mount -J]" cmd=findmnt host_id=275418297 region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:54:06Z" level=info msg="node capacity statistics retrieved" bytes_available=3906445312 bytes_total=4160421888 bytes_used=22454272 host_id=275418297 inodes_available=261974 inodes_total=262144 inodes_used=170 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=4ced3b86-75c9-11ea-969f-0a58ac144db2 volume_mode=filesystem volume_path="/var/lib/kubelet/pods/c5007016-9111-453f-a529-d1ee0513ae93/volumes/kubernetes.io~csi/pvc-e80f4df6-70d1-49d2-bd12-f9653ac62382/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:54:09Z" level=info msg="node get capabilities called" host_id=275418297 method=node_get_capabilities node_capabilities="[rpc:<type:STAGE_UNSTAGE_VOLUME > rpc:<type:EXPAND_VOLUME > rpc:<type:GET_VOLUME_STATS > ]" region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:54:09Z" level=info msg="node get volume stats called" host_id=275418297 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=c5c5bbba-1d0b-11eb-a40f-0a58ac14506d volume_path="/var/lib/kubelet/pods/1d1cdb98-329d-41c5-b05e-5040fe216c8d/volumes/kubernetes.io~csi/pvc-8c2ebf45-603f-4e39-89c6-ba58023a9f43/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:54:09Z" level=info msg="checking if target is mounted" args="[-o TARGET,PROPAGATION,FSTYPE,OPTIONS -M /var/lib/kubelet/pods/1d1cdb98-329d-41c5-b05e-5040fe216c8d/volumes/kubernetes.io~csi/pvc-8c2ebf45-603f-4e39-89c6-ba58023a9f43/mount -J]" cmd=findmnt host_id=275418297 region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:54:09Z" level=info msg="node capacity statistics retrieved" bytes_available=7679172608 bytes_total=8388009984 bytes_used=262565888 host_id=275418297 inodes_available=524093 inodes_total=524288 inodes_used=195 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=c5c5bbba-1d0b-11eb-a40f-0a58ac14506d volume_mode=filesystem volume_path="/var/lib/kubelet/pods/1d1cdb98-329d-41c5-b05e-5040fe216c8d/volumes/kubernetes.io~csi/pvc-8c2ebf45-603f-4e39-89c6-ba58023a9f43/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:54:38Z" level=info msg="node get capabilities called" host_id=275418297 method=node_get_capabilities node_capabilities="[rpc:<type:STAGE_UNSTAGE_VOLUME > rpc:<type:EXPAND_VOLUME > rpc:<type:GET_VOLUME_STATS > ]" region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:54:38Z" level=info msg="node get volume stats called" host_id=275418297 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=debe707e-777a-11ea-b3a8-0a58ac144a9e volume_path="/var/lib/kubelet/pods/48faac62-e354-4a20-a4a8-f2f97315a81f/volumes/kubernetes.io~csi/pvc-60b9563e-7b3e-4617-b046-5e04b39f5548/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:54:38Z" level=info msg="checking if target is mounted" args="[-o TARGET,PROPAGATION,FSTYPE,OPTIONS -M /var/lib/kubelet/pods/48faac62-e354-4a20-a4a8-f2f97315a81f/volumes/kubernetes.io~csi/pvc-60b9563e-7b3e-4617-b046-5e04b39f5548/mount -J]" cmd=findmnt host_id=275418297 region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:54:38Z" level=info msg="node capacity statistics retrieved" bytes_available=25775620096 bytes_total=50462347264 bytes_used=22092972032 host_id=275418297 inodes_available=3138861 inodes_total=3145728 inodes_used=6867 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=debe707e-777a-11ea-b3a8-0a58ac144a9e volume_mode=filesystem volume_path="/var/lib/kubelet/pods/48faac62-e354-4a20-a4a8-f2f97315a81f/volumes/kubernetes.io~csi/pvc-60b9563e-7b3e-4617-b046-5e04b39f5548/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:54:44Z" level=info msg="node get capabilities called" host_id=275418297 method=node_get_capabilities node_capabilities="[rpc:<type:STAGE_UNSTAGE_VOLUME > rpc:<type:EXPAND_VOLUME > rpc:<type:GET_VOLUME_STATS > ]" region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:54:44Z" level=info msg="node get volume stats called" host_id=275418297 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=a3d6a386-60b6-11ea-88c9-0a58ac1446ab volume_path="/var/lib/kubelet/pods/da656c76-3a61-4a85-a7bc-e75590dcf929/volumes/kubernetes.io~csi/pvc-a84b3288-0aca-48e5-bfca-22c3986e8187/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:54:44Z" level=info msg="checking if target is mounted" args="[-o TARGET,PROPAGATION,FSTYPE,OPTIONS -M /var/lib/kubelet/pods/da656c76-3a61-4a85-a7bc-e75590dcf929/volumes/kubernetes.io~csi/pvc-a84b3288-0aca-48e5-bfca-22c3986e8187/mount -J]" cmd=findmnt host_id=275418297 region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:54:44Z" level=info msg="node capacity statistics retrieved" bytes_available=7581257728 bytes_total=8388009984 bytes_used=360480768 host_id=275418297 inodes_available=523049 inodes_total=524288 inodes_used=1239 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=a3d6a386-60b6-11ea-88c9-0a58ac1446ab volume_mode=filesystem volume_path="/var/lib/kubelet/pods/da656c76-3a61-4a85-a7bc-e75590dcf929/volumes/kubernetes.io~csi/pvc-a84b3288-0aca-48e5-bfca-22c3986e8187/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:54:46Z" level=info msg="node get capabilities called" host_id=275418297 method=node_get_capabilities node_capabilities="[rpc:<type:STAGE_UNSTAGE_VOLUME > rpc:<type:EXPAND_VOLUME > rpc:<type:GET_VOLUME_STATS > ]" region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:54:46Z" level=info msg="node get volume stats called" host_id=275418297 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=8701ff1c-75ad-11ea-b009-0a58ac144514 volume_path="/var/lib/kubelet/pods/c6ac1ad6-5091-4ed3-b95e-69d0f1821d86/volumes/kubernetes.io~csi/pvc-fcd10d7e-7017-4f63-8e87-48d89a6d93ab/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:54:46Z" level=info msg="checking if target is mounted" args="[-o TARGET,PROPAGATION,FSTYPE,OPTIONS -M /var/lib/kubelet/pods/c6ac1ad6-5091-4ed3-b95e-69d0f1821d86/volumes/kubernetes.io~csi/pvc-fcd10d7e-7017-4f63-8e87-48d89a6d93ab/mount -J]" cmd=findmnt host_id=275418297 region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:54:46Z" level=info msg="node capacity statistics retrieved" bytes_available=1541267456 bytes_total=2046640128 bytes_used=381222912 host_id=275418297 inodes_available=129826 inodes_total=131072 inodes_used=1246 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=8701ff1c-75ad-11ea-b009-0a58ac144514 volume_mode=filesystem volume_path="/var/lib/kubelet/pods/c6ac1ad6-5091-4ed3-b95e-69d0f1821d86/volumes/kubernetes.io~csi/pvc-fcd10d7e-7017-4f63-8e87-48d89a6d93ab/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:55:09Z" level=info msg="node get capabilities called" host_id=275418297 method=node_get_capabilities node_capabilities="[rpc:<type:STAGE_UNSTAGE_VOLUME > rpc:<type:EXPAND_VOLUME > rpc:<type:GET_VOLUME_STATS > ]" region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:55:09Z" level=info msg="node get volume stats called" host_id=275418297 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=c40c49dd-60b6-11ea-b89e-0a58ac14480f volume_path="/var/lib/kubelet/pods/16a28727-423f-47eb-b172-d9738158e85f/volumes/kubernetes.io~csi/pvc-0141174a-de96-409d-8efd-cea6c528893c/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:55:09Z" level=info msg="checking if target is mounted" args="[-o TARGET,PROPAGATION,FSTYPE,OPTIONS -M /var/lib/kubelet/pods/16a28727-423f-47eb-b172-d9738158e85f/volumes/kubernetes.io~csi/pvc-0141174a-de96-409d-8efd-cea6c528893c/mount -J]" cmd=findmnt host_id=275418297 region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:55:09Z" level=info msg="node capacity statistics retrieved" bytes_available=1916198912 bytes_total=2046640128 bytes_used=6291456 host_id=275418297 inodes_available=131061 inodes_total=131072 inodes_used=11 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=c40c49dd-60b6-11ea-b89e-0a58ac14480f volume_mode=filesystem volume_path="/var/lib/kubelet/pods/16a28727-423f-47eb-b172-d9738158e85f/volumes/kubernetes.io~csi/pvc-0141174a-de96-409d-8efd-cea6c528893c/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:55:14Z" level=info msg="node get capabilities called" host_id=275418297 method=node_get_capabilities node_capabilities="[rpc:<type:STAGE_UNSTAGE_VOLUME > rpc:<type:EXPAND_VOLUME > rpc:<type:GET_VOLUME_STATS > ]" region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:55:14Z" level=info msg="node get volume stats called" host_id=275418297 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=c5c5bbba-1d0b-11eb-a40f-0a58ac14506d volume_path="/var/lib/kubelet/pods/1d1cdb98-329d-41c5-b05e-5040fe216c8d/volumes/kubernetes.io~csi/pvc-8c2ebf45-603f-4e39-89c6-ba58023a9f43/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:55:14Z" level=info msg="checking if target is mounted" args="[-o TARGET,PROPAGATION,FSTYPE,OPTIONS -M /var/lib/kubelet/pods/1d1cdb98-329d-41c5-b05e-5040fe216c8d/volumes/kubernetes.io~csi/pvc-8c2ebf45-603f-4e39-89c6-ba58023a9f43/mount -J]" cmd=findmnt host_id=275418297 region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:55:14Z" level=info msg="node capacity statistics retrieved" bytes_available=7679172608 bytes_total=8388009984 bytes_used=262565888 host_id=275418297 inodes_available=524093 inodes_total=524288 inodes_used=195 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=c5c5bbba-1d0b-11eb-a40f-0a58ac14506d volume_mode=filesystem volume_path="/var/lib/kubelet/pods/1d1cdb98-329d-41c5-b05e-5040fe216c8d/volumes/kubernetes.io~csi/pvc-8c2ebf45-603f-4e39-89c6-ba58023a9f43/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:55:31Z" level=info msg="node get capabilities called" host_id=275418297 method=node_get_capabilities node_capabilities="[rpc:<type:STAGE_UNSTAGE_VOLUME > rpc:<type:EXPAND_VOLUME > rpc:<type:GET_VOLUME_STATS > ]" region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:55:31Z" level=info msg="node get volume stats called" host_id=275418297 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=0468c53e-75ae-11ea-a87f-0a58ac14489e volume_path="/var/lib/kubelet/pods/5b5e44c3-5ac8-4f6e-ad02-f79445d86817/volumes/kubernetes.io~csi/pvc-2f34de0b-b85c-4141-b566-c0995f2b311e/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:55:31Z" level=info msg="checking if target is mounted" args="[-o TARGET,PROPAGATION,FSTYPE,OPTIONS -M /var/lib/kubelet/pods/5b5e44c3-5ac8-4f6e-ad02-f79445d86817/volumes/kubernetes.io~csi/pvc-2f34de0b-b85c-4141-b566-c0995f2b311e/mount -J]" cmd=findmnt host_id=275418297 region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:55:31Z" level=info msg="node capacity statistics retrieved" bytes_available=950214656 bytes_total=1023303680 bytes_used=2625536 host_id=275418297 inodes_available=65525 inodes_total=65536 inodes_used=11 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=0468c53e-75ae-11ea-a87f-0a58ac14489e volume_mode=filesystem volume_path="/var/lib/kubelet/pods/5b5e44c3-5ac8-4f6e-ad02-f79445d86817/volumes/kubernetes.io~csi/pvc-2f34de0b-b85c-4141-b566-c0995f2b311e/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:55:42Z" level=info msg="node get capabilities called" host_id=275418297 method=node_get_capabilities node_capabilities="[rpc:<type:STAGE_UNSTAGE_VOLUME > rpc:<type:EXPAND_VOLUME > rpc:<type:GET_VOLUME_STATS > ]" region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:55:42Z" level=info msg="node get volume stats called" host_id=275418297 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=4ced3b86-75c9-11ea-969f-0a58ac144db2 volume_path="/var/lib/kubelet/pods/c5007016-9111-453f-a529-d1ee0513ae93/volumes/kubernetes.io~csi/pvc-e80f4df6-70d1-49d2-bd12-f9653ac62382/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:55:42Z" level=info msg="checking if target is mounted" args="[-o TARGET,PROPAGATION,FSTYPE,OPTIONS -M /var/lib/kubelet/pods/c5007016-9111-453f-a529-d1ee0513ae93/volumes/kubernetes.io~csi/pvc-e80f4df6-70d1-49d2-bd12-f9653ac62382/mount -J]" cmd=findmnt host_id=275418297 region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:55:42Z" level=info msg="node capacity statistics retrieved" bytes_available=3906445312 bytes_total=4160421888 bytes_used=22454272 host_id=275418297 inodes_available=261974 inodes_total=262144 inodes_used=170 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=4ced3b86-75c9-11ea-969f-0a58ac144db2 volume_mode=filesystem volume_path="/var/lib/kubelet/pods/c5007016-9111-453f-a529-d1ee0513ae93/volumes/kubernetes.io~csi/pvc-e80f4df6-70d1-49d2-bd12-f9653ac62382/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:55:57Z" level=info msg="node get capabilities called" host_id=275418297 method=node_get_capabilities node_capabilities="[rpc:<type:STAGE_UNSTAGE_VOLUME > rpc:<type:EXPAND_VOLUME > rpc:<type:GET_VOLUME_STATS > ]" region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:55:57Z" level=info msg="node get volume stats called" host_id=275418297 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=8701ff1c-75ad-11ea-b009-0a58ac144514 volume_path="/var/lib/kubelet/pods/c6ac1ad6-5091-4ed3-b95e-69d0f1821d86/volumes/kubernetes.io~csi/pvc-fcd10d7e-7017-4f63-8e87-48d89a6d93ab/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:55:57Z" level=info msg="checking if target is mounted" args="[-o TARGET,PROPAGATION,FSTYPE,OPTIONS -M /var/lib/kubelet/pods/c6ac1ad6-5091-4ed3-b95e-69d0f1821d86/volumes/kubernetes.io~csi/pvc-fcd10d7e-7017-4f63-8e87-48d89a6d93ab/mount -J]" cmd=findmnt host_id=275418297 region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:55:57Z" level=info msg="node capacity statistics retrieved" bytes_available=1541267456 bytes_total=2046640128 bytes_used=381222912 host_id=275418297 inodes_available=129826 inodes_total=131072 inodes_used=1246 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=8701ff1c-75ad-11ea-b009-0a58ac144514 volume_mode=filesystem volume_path="/var/lib/kubelet/pods/c6ac1ad6-5091-4ed3-b95e-69d0f1821d86/volumes/kubernetes.io~csi/pvc-fcd10d7e-7017-4f63-8e87-48d89a6d93ab/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:56:04Z" level=info msg="node get capabilities called" host_id=275418297 method=node_get_capabilities node_capabilities="[rpc:<type:STAGE_UNSTAGE_VOLUME > rpc:<type:EXPAND_VOLUME > rpc:<type:GET_VOLUME_STATS > ]" region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:56:04Z" level=info msg="node get volume stats called" host_id=275418297 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=a3d6a386-60b6-11ea-88c9-0a58ac1446ab volume_path="/var/lib/kubelet/pods/da656c76-3a61-4a85-a7bc-e75590dcf929/volumes/kubernetes.io~csi/pvc-a84b3288-0aca-48e5-bfca-22c3986e8187/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:56:04Z" level=info msg="checking if target is mounted" args="[-o TARGET,PROPAGATION,FSTYPE,OPTIONS -M /var/lib/kubelet/pods/da656c76-3a61-4a85-a7bc-e75590dcf929/volumes/kubernetes.io~csi/pvc-a84b3288-0aca-48e5-bfca-22c3986e8187/mount -J]" cmd=findmnt host_id=275418297 region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:56:04Z" level=info msg="node capacity statistics retrieved" bytes_available=7581257728 bytes_total=8388009984 bytes_used=360480768 host_id=275418297 inodes_available=523049 inodes_total=524288 inodes_used=1239 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=a3d6a386-60b6-11ea-88c9-0a58ac1446ab volume_mode=filesystem volume_path="/var/lib/kubelet/pods/da656c76-3a61-4a85-a7bc-e75590dcf929/volumes/kubernetes.io~csi/pvc-a84b3288-0aca-48e5-bfca-22c3986e8187/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:56:30Z" level=info msg="node get capabilities called" host_id=275418297 method=node_get_capabilities node_capabilities="[rpc:<type:STAGE_UNSTAGE_VOLUME > rpc:<type:EXPAND_VOLUME > rpc:<type:GET_VOLUME_STATS > ]" region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:56:30Z" level=info msg="node get volume stats called" host_id=275418297 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=debe707e-777a-11ea-b3a8-0a58ac144a9e volume_path="/var/lib/kubelet/pods/48faac62-e354-4a20-a4a8-f2f97315a81f/volumes/kubernetes.io~csi/pvc-60b9563e-7b3e-4617-b046-5e04b39f5548/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:56:30Z" level=info msg="checking if target is mounted" args="[-o TARGET,PROPAGATION,FSTYPE,OPTIONS -M /var/lib/kubelet/pods/48faac62-e354-4a20-a4a8-f2f97315a81f/volumes/kubernetes.io~csi/pvc-60b9563e-7b3e-4617-b046-5e04b39f5548/mount -J]" cmd=findmnt host_id=275418297 region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:56:30Z" level=info msg="node capacity statistics retrieved" bytes_available=25775620096 bytes_total=50462347264 bytes_used=22092972032 host_id=275418297 inodes_available=3138861 inodes_total=3145728 inodes_used=6867 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=debe707e-777a-11ea-b3a8-0a58ac144a9e volume_mode=filesystem volume_path="/var/lib/kubelet/pods/48faac62-e354-4a20-a4a8-f2f97315a81f/volumes/kubernetes.io~csi/pvc-60b9563e-7b3e-4617-b046-5e04b39f5548/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:56:34Z" level=info msg="node get capabilities called" host_id=275418297 method=node_get_capabilities node_capabilities="[rpc:<type:STAGE_UNSTAGE_VOLUME > rpc:<type:EXPAND_VOLUME > rpc:<type:GET_VOLUME_STATS > ]" region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:56:34Z" level=info msg="node get volume stats called" host_id=275418297 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=0468c53e-75ae-11ea-a87f-0a58ac14489e volume_path="/var/lib/kubelet/pods/5b5e44c3-5ac8-4f6e-ad02-f79445d86817/volumes/kubernetes.io~csi/pvc-2f34de0b-b85c-4141-b566-c0995f2b311e/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:56:34Z" level=info msg="checking if target is mounted" args="[-o TARGET,PROPAGATION,FSTYPE,OPTIONS -M /var/lib/kubelet/pods/5b5e44c3-5ac8-4f6e-ad02-f79445d86817/volumes/kubernetes.io~csi/pvc-2f34de0b-b85c-4141-b566-c0995f2b311e/mount -J]" cmd=findmnt host_id=275418297 region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:56:34Z" level=info msg="node capacity statistics retrieved" bytes_available=950214656 bytes_total=1023303680 bytes_used=2625536 host_id=275418297 inodes_available=65525 inodes_total=65536 inodes_used=11 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=0468c53e-75ae-11ea-a87f-0a58ac14489e volume_mode=filesystem volume_path="/var/lib/kubelet/pods/5b5e44c3-5ac8-4f6e-ad02-f79445d86817/volumes/kubernetes.io~csi/pvc-2f34de0b-b85c-4141-b566-c0995f2b311e/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:56:43Z" level=info msg="node get capabilities called" host_id=275418297 method=node_get_capabilities node_capabilities="[rpc:<type:STAGE_UNSTAGE_VOLUME > rpc:<type:EXPAND_VOLUME > rpc:<type:GET_VOLUME_STATS > ]" region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:56:43Z" level=info msg="node get volume stats called" host_id=275418297 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=c5c5bbba-1d0b-11eb-a40f-0a58ac14506d volume_path="/var/lib/kubelet/pods/1d1cdb98-329d-41c5-b05e-5040fe216c8d/volumes/kubernetes.io~csi/pvc-8c2ebf45-603f-4e39-89c6-ba58023a9f43/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:56:43Z" level=info msg="checking if target is mounted" args="[-o TARGET,PROPAGATION,FSTYPE,OPTIONS -M /var/lib/kubelet/pods/1d1cdb98-329d-41c5-b05e-5040fe216c8d/volumes/kubernetes.io~csi/pvc-8c2ebf45-603f-4e39-89c6-ba58023a9f43/mount -J]" cmd=findmnt host_id=275418297 region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:56:43Z" level=info msg="node capacity statistics retrieved" bytes_available=7679172608 bytes_total=8388009984 bytes_used=262565888 host_id=275418297 inodes_available=524093 inodes_total=524288 inodes_used=195 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=c5c5bbba-1d0b-11eb-a40f-0a58ac14506d volume_mode=filesystem volume_path="/var/lib/kubelet/pods/1d1cdb98-329d-41c5-b05e-5040fe216c8d/volumes/kubernetes.io~csi/pvc-8c2ebf45-603f-4e39-89c6-ba58023a9f43/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:56:49Z" level=info msg="node get capabilities called" host_id=275418297 method=node_get_capabilities node_capabilities="[rpc:<type:STAGE_UNSTAGE_VOLUME > rpc:<type:EXPAND_VOLUME > rpc:<type:GET_VOLUME_STATS > ]" region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:56:49Z" level=info msg="node get volume stats called" host_id=275418297 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=4ced3b86-75c9-11ea-969f-0a58ac144db2 volume_path="/var/lib/kubelet/pods/c5007016-9111-453f-a529-d1ee0513ae93/volumes/kubernetes.io~csi/pvc-e80f4df6-70d1-49d2-bd12-f9653ac62382/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:56:49Z" level=info msg="checking if target is mounted" args="[-o TARGET,PROPAGATION,FSTYPE,OPTIONS -M /var/lib/kubelet/pods/c5007016-9111-453f-a529-d1ee0513ae93/volumes/kubernetes.io~csi/pvc-e80f4df6-70d1-49d2-bd12-f9653ac62382/mount -J]" cmd=findmnt host_id=275418297 region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:56:49Z" level=info msg="node capacity statistics retrieved" bytes_available=3906445312 bytes_total=4160421888 bytes_used=22454272 host_id=275418297 inodes_available=261974 inodes_total=262144 inodes_used=170 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=4ced3b86-75c9-11ea-969f-0a58ac144db2 volume_mode=filesystem volume_path="/var/lib/kubelet/pods/c5007016-9111-453f-a529-d1ee0513ae93/volumes/kubernetes.io~csi/pvc-e80f4df6-70d1-49d2-bd12-f9653ac62382/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:56:51Z" level=info msg="node get capabilities called" host_id=275418297 method=node_get_capabilities node_capabilities="[rpc:<type:STAGE_UNSTAGE_VOLUME > rpc:<type:EXPAND_VOLUME > rpc:<type:GET_VOLUME_STATS > ]" region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:56:51Z" level=info msg="node get volume stats called" host_id=275418297 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=c40c49dd-60b6-11ea-b89e-0a58ac14480f volume_path="/var/lib/kubelet/pods/16a28727-423f-47eb-b172-d9738158e85f/volumes/kubernetes.io~csi/pvc-0141174a-de96-409d-8efd-cea6c528893c/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:56:51Z" level=info msg="checking if target is mounted" args="[-o TARGET,PROPAGATION,FSTYPE,OPTIONS -M /var/lib/kubelet/pods/16a28727-423f-47eb-b172-d9738158e85f/volumes/kubernetes.io~csi/pvc-0141174a-de96-409d-8efd-cea6c528893c/mount -J]" cmd=findmnt host_id=275418297 region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:56:51Z" level=info msg="node capacity statistics retrieved" bytes_available=1916198912 bytes_total=2046640128 bytes_used=6291456 host_id=275418297 inodes_available=131061 inodes_total=131072 inodes_used=11 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=c40c49dd-60b6-11ea-b89e-0a58ac14480f volume_mode=filesystem volume_path="/var/lib/kubelet/pods/16a28727-423f-47eb-b172-d9738158e85f/volumes/kubernetes.io~csi/pvc-0141174a-de96-409d-8efd-cea6c528893c/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:57:02Z" level=info msg="node get capabilities called" host_id=275418297 method=node_get_capabilities node_capabilities="[rpc:<type:STAGE_UNSTAGE_VOLUME > rpc:<type:EXPAND_VOLUME > rpc:<type:GET_VOLUME_STATS > ]" region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:57:02Z" level=info msg="node get volume stats called" host_id=275418297 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=8701ff1c-75ad-11ea-b009-0a58ac144514 volume_path="/var/lib/kubelet/pods/c6ac1ad6-5091-4ed3-b95e-69d0f1821d86/volumes/kubernetes.io~csi/pvc-fcd10d7e-7017-4f63-8e87-48d89a6d93ab/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:57:02Z" level=info msg="checking if target is mounted" args="[-o TARGET,PROPAGATION,FSTYPE,OPTIONS -M /var/lib/kubelet/pods/c6ac1ad6-5091-4ed3-b95e-69d0f1821d86/volumes/kubernetes.io~csi/pvc-fcd10d7e-7017-4f63-8e87-48d89a6d93ab/mount -J]" cmd=findmnt host_id=275418297 region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:57:02Z" level=info msg="node capacity statistics retrieved" bytes_available=1541267456 bytes_total=2046640128 bytes_used=381222912 host_id=275418297 inodes_available=129826 inodes_total=131072 inodes_used=1246 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=8701ff1c-75ad-11ea-b009-0a58ac144514 volume_mode=filesystem volume_path="/var/lib/kubelet/pods/c6ac1ad6-5091-4ed3-b95e-69d0f1821d86/volumes/kubernetes.io~csi/pvc-fcd10d7e-7017-4f63-8e87-48d89a6d93ab/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:57:09Z" level=info msg="node get capabilities called" host_id=275418297 method=node_get_capabilities node_capabilities="[rpc:<type:STAGE_UNSTAGE_VOLUME > rpc:<type:EXPAND_VOLUME > rpc:<type:GET_VOLUME_STATS > ]" region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:57:09Z" level=info msg="node get volume stats called" host_id=275418297 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=a3d6a386-60b6-11ea-88c9-0a58ac1446ab volume_path="/var/lib/kubelet/pods/da656c76-3a61-4a85-a7bc-e75590dcf929/volumes/kubernetes.io~csi/pvc-a84b3288-0aca-48e5-bfca-22c3986e8187/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:57:09Z" level=info msg="checking if target is mounted" args="[-o TARGET,PROPAGATION,FSTYPE,OPTIONS -M /var/lib/kubelet/pods/da656c76-3a61-4a85-a7bc-e75590dcf929/volumes/kubernetes.io~csi/pvc-a84b3288-0aca-48e5-bfca-22c3986e8187/mount -J]" cmd=findmnt host_id=275418297 region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:57:09Z" level=info msg="node capacity statistics retrieved" bytes_available=7581257728 bytes_total=8388009984 bytes_used=360480768 host_id=275418297 inodes_available=523049 inodes_total=524288 inodes_used=1239 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=a3d6a386-60b6-11ea-88c9-0a58ac1446ab volume_mode=filesystem volume_path="/var/lib/kubelet/pods/da656c76-3a61-4a85-a7bc-e75590dcf929/volumes/kubernetes.io~csi/pvc-a84b3288-0aca-48e5-bfca-22c3986e8187/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:57:36Z" level=info msg="node get capabilities called" host_id=275418297 method=node_get_capabilities node_capabilities="[rpc:<type:STAGE_UNSTAGE_VOLUME > rpc:<type:EXPAND_VOLUME > rpc:<type:GET_VOLUME_STATS > ]" region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:57:36Z" level=info msg="node get volume stats called" host_id=275418297 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=debe707e-777a-11ea-b3a8-0a58ac144a9e volume_path="/var/lib/kubelet/pods/48faac62-e354-4a20-a4a8-f2f97315a81f/volumes/kubernetes.io~csi/pvc-60b9563e-7b3e-4617-b046-5e04b39f5548/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:57:36Z" level=info msg="checking if target is mounted" args="[-o TARGET,PROPAGATION,FSTYPE,OPTIONS -M /var/lib/kubelet/pods/48faac62-e354-4a20-a4a8-f2f97315a81f/volumes/kubernetes.io~csi/pvc-60b9563e-7b3e-4617-b046-5e04b39f5548/mount -J]" cmd=findmnt host_id=275418297 region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:57:36Z" level=info msg="node capacity statistics retrieved" bytes_available=25775620096 bytes_total=50462347264 bytes_used=22092972032 host_id=275418297 inodes_available=3138861 inodes_total=3145728 inodes_used=6867 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=debe707e-777a-11ea-b3a8-0a58ac144a9e volume_mode=filesystem volume_path="/var/lib/kubelet/pods/48faac62-e354-4a20-a4a8-f2f97315a81f/volumes/kubernetes.io~csi/pvc-60b9563e-7b3e-4617-b046-5e04b39f5548/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:58:02Z" level=info msg="node get capabilities called" host_id=275418297 method=node_get_capabilities node_capabilities="[rpc:<type:STAGE_UNSTAGE_VOLUME > rpc:<type:EXPAND_VOLUME > rpc:<type:GET_VOLUME_STATS > ]" region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:58:02Z" level=info msg="node get volume stats called" host_id=275418297 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=c40c49dd-60b6-11ea-b89e-0a58ac14480f volume_path="/var/lib/kubelet/pods/16a28727-423f-47eb-b172-d9738158e85f/volumes/kubernetes.io~csi/pvc-0141174a-de96-409d-8efd-cea6c528893c/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:58:02Z" level=info msg="checking if target is mounted" args="[-o TARGET,PROPAGATION,FSTYPE,OPTIONS -M /var/lib/kubelet/pods/16a28727-423f-47eb-b172-d9738158e85f/volumes/kubernetes.io~csi/pvc-0141174a-de96-409d-8efd-cea6c528893c/mount -J]" cmd=findmnt host_id=275418297 region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:58:02Z" level=info msg="node capacity statistics retrieved" bytes_available=1916198912 bytes_total=2046640128 bytes_used=6291456 host_id=275418297 inodes_available=131061 inodes_total=131072 inodes_used=11 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=c40c49dd-60b6-11ea-b89e-0a58ac14480f volume_mode=filesystem volume_path="/var/lib/kubelet/pods/16a28727-423f-47eb-b172-d9738158e85f/volumes/kubernetes.io~csi/pvc-0141174a-de96-409d-8efd-cea6c528893c/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:58:05Z" level=info msg="node get capabilities called" host_id=275418297 method=node_get_capabilities node_capabilities="[rpc:<type:STAGE_UNSTAGE_VOLUME > rpc:<type:EXPAND_VOLUME > rpc:<type:GET_VOLUME_STATS > ]" region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:58:05Z" level=info msg="node get volume stats called" host_id=275418297 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=8701ff1c-75ad-11ea-b009-0a58ac144514 volume_path="/var/lib/kubelet/pods/c6ac1ad6-5091-4ed3-b95e-69d0f1821d86/volumes/kubernetes.io~csi/pvc-fcd10d7e-7017-4f63-8e87-48d89a6d93ab/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:58:05Z" level=info msg="checking if target is mounted" args="[-o TARGET,PROPAGATION,FSTYPE,OPTIONS -M /var/lib/kubelet/pods/c6ac1ad6-5091-4ed3-b95e-69d0f1821d86/volumes/kubernetes.io~csi/pvc-fcd10d7e-7017-4f63-8e87-48d89a6d93ab/mount -J]" cmd=findmnt host_id=275418297 region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:58:05Z" level=info msg="node capacity statistics retrieved" bytes_available=1541267456 bytes_total=2046640128 bytes_used=381222912 host_id=275418297 inodes_available=129826 inodes_total=131072 inodes_used=1246 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=8701ff1c-75ad-11ea-b009-0a58ac144514 volume_mode=filesystem volume_path="/var/lib/kubelet/pods/c6ac1ad6-5091-4ed3-b95e-69d0f1821d86/volumes/kubernetes.io~csi/pvc-fcd10d7e-7017-4f63-8e87-48d89a6d93ab/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:58:18Z" level=info msg="node get capabilities called" host_id=275418297 method=node_get_capabilities node_capabilities="[rpc:<type:STAGE_UNSTAGE_VOLUME > rpc:<type:EXPAND_VOLUME > rpc:<type:GET_VOLUME_STATS > ]" region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:58:18Z" level=info msg="node get volume stats called" host_id=275418297 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=c5c5bbba-1d0b-11eb-a40f-0a58ac14506d volume_path="/var/lib/kubelet/pods/1d1cdb98-329d-41c5-b05e-5040fe216c8d/volumes/kubernetes.io~csi/pvc-8c2ebf45-603f-4e39-89c6-ba58023a9f43/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:58:18Z" level=info msg="checking if target is mounted" args="[-o TARGET,PROPAGATION,FSTYPE,OPTIONS -M /var/lib/kubelet/pods/1d1cdb98-329d-41c5-b05e-5040fe216c8d/volumes/kubernetes.io~csi/pvc-8c2ebf45-603f-4e39-89c6-ba58023a9f43/mount -J]" cmd=findmnt host_id=275418297 region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:58:18Z" level=info msg="node capacity statistics retrieved" bytes_available=7679172608 bytes_total=8388009984 bytes_used=262565888 host_id=275418297 inodes_available=524093 inodes_total=524288 inodes_used=195 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=c5c5bbba-1d0b-11eb-a40f-0a58ac14506d volume_mode=filesystem volume_path="/var/lib/kubelet/pods/1d1cdb98-329d-41c5-b05e-5040fe216c8d/volumes/kubernetes.io~csi/pvc-8c2ebf45-603f-4e39-89c6-ba58023a9f43/mount"
csi-do-node-kkxlb csi-do-plugin time="2021-11-27T17:58:23Z" level=info msg="node get capabilities called" host_id=275421176 method=node_get_capabilities node_capabilities="[rpc:<type:STAGE_UNSTAGE_VOLUME > rpc:<type:EXPAND_VOLUME > rpc:<type:GET_VOLUME_STATS > ]" region=nyc1 version=v2.1.2
csi-do-node-kkxlb csi-do-plugin time="2021-11-27T17:58:23Z" level=info msg="node get volume stats called" host_id=275421176 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=c5743bb9-1d0b-11eb-84b4-0a58ac144252 volume_path="/var/lib/kubelet/pods/93e941f7-9448-42e0-8e55-a861370eabc9/volumes/kubernetes.io~csi/pvc-95514ac5-945b-4a6c-a322-77864bfc7c50/mount"
csi-do-node-kkxlb csi-do-plugin time="2021-11-27T17:58:23Z" level=info msg="checking if target is mounted" args="[-o TARGET,PROPAGATION,FSTYPE,OPTIONS -M /var/lib/kubelet/pods/93e941f7-9448-42e0-8e55-a861370eabc9/volumes/kubernetes.io~csi/pvc-95514ac5-945b-4a6c-a322-77864bfc7c50/mount -J]" cmd=findmnt host_id=275421176 region=nyc1 version=v2.1.2
csi-do-node-kkxlb csi-do-plugin time="2021-11-27T17:58:23Z" level=info msg="node capacity statistics retrieved" bytes_available=85047205888 bytes_total=100924641280 bytes_used=10706698240 host_id=275421176 inodes_available=6276744 inodes_total=6291456 inodes_used=14712 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=c5743bb9-1d0b-11eb-84b4-0a58ac144252 volume_mode=filesystem volume_path="/var/lib/kubelet/pods/93e941f7-9448-42e0-8e55-a861370eabc9/volumes/kubernetes.io~csi/pvc-95514ac5-945b-4a6c-a322-77864bfc7c50/mount"
from csi-digitalocean.
rbren
commented on September 23, 2024
Could the issue possibly have to do with the fact that it's trying to provision two volumes at the same time?
from csi-digitalocean.
timoreimann
commented on September 23, 2024
@rbren I took a look at your cluster, and I think I found the problem.
Turns out you are indeed affected by the lack of backoff in csi-snapshotter. Although the feature has been released by now, it is only available in v4 of the csi-snapshotter sidecar while we still have to run v3 in DOKS. (We yet have to migrate from snapshots v1beta1 to v1 which is a requirement for using csi-snapshotter v4.)
The reason why your copy of csi-snapshotter hits the API repeatedly is because you are trying to have more than 25 snapshots per volume which is beyond the limit, so the driver keeps attempting to create snapshots but fails perpetually. Could you delete at least enough snapshots to drop below the 25 snapshots limit (and stop/adjust any processes you may have in place that periodically create new snapshots)? Note that the deletes won't go through right now since I have temporarily disabled your cluster's CSI driver so that your API quota can recover. I will re-enable the driver once you have confirmed that the snapshots deletes were issued. As soon as that happens, the snapshot deletes should complete, and your CSI driver should operate normally again.
Sorry for the inconveniences. Once we're able to upgrade to csi-snapshotter v4 in DOKS, exceeding snapshot limits should not lead to API quota exhaustion anymore. Additionally, there is also no clear Kubernetes event emitted by csi-snapshotter right now to inform about the nature of the problem; if that's still not the case with v4, then we can surely drive an improvement on the error reporting front.
from csi-digitalocean.
rbren
commented on September 23, 2024
@timoreimann sorry for the lack of response!
Just to confirm - that's 25 snapshots per volume, right? I've deleted out a bunch of old snapshots, hopefully that helps. (side note: bulk deletion in the UI would be a very helpful feature)
I don't really want to disable my auto-snapshotter, as that's my backup strategy 😬
Any word on v4?
from csi-digitalocean.
rbren
commented on September 23, 2024
Deleting the old snapshots seemed to work! Hopefully I won't regret it 😅
from csi-digitalocean.
timoreimann
commented on September 23, 2024
My bad, the 25 limit is per volume of course. Glad you figured it out. 👍
v4 support is still on the road map. We are currently sorting out a few details on how to best integrate the validating web hook that's needed to prevent invalid snapshots from becoming inaccessible as customers move from snapshots API beta to v1/GA. Stay tuned. 🙂
from csi-digitalocean.
timoreimann
commented on September 23, 2024
Going to close out the ticket as I believe we've wrapped up the discussion / case.
A new release should make it through once we do support snapshot v4 with the improved retry logic.
from csi-digitalocean.
Related Issues (20)
- Issue creating VolumeSnapshots HOT 3
- Stuck in "Still connecting to unix:///csi/csi.sock" loop
- Problem while existing pod gets updated - multi-attach error. HOT 2
- APIExstension k8s v1beta1 Deprecated
- [Question] privileged vs SYS_ADMIN in the CSI driver deployment
- Deploy as Helm chart HOT 7
- Support for NVMe volumes HOT 10
- PVC - Add do-block-storage-retain storage class with Reclaim policy Retain HOT 4
- Feature request: allow custom snapshot names HOT 1
- Error: waiting for a volume to be created, either by external provisioner "dobs.csi.digitalocean.com" or manually created by system administrator HOT 1
- Round capacity up to minimum supported size HOT 1
- Feature request: Support CSI PV Cloning HOT 5
- Issue with CSI outside of DigitalOcean droplets HOT 1
- Error mounting imported snapshot HOT 5
- Discrepancies with docs about importing snapshots in Kubernetes
- error resizing volume: volume/snapshot capacity limit exceeded HOT 1
- CSI Controller unable to access any API endpoints HOT 1
- Cannot apply csi driver on cis-1.23 hardened rke2 HOT 8
- Feature Request: Support Online PV Resizing
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from csi-digitalocean.