Comments (7)
cofyc
commented on May 18, 2024
hi, you shouldn't have created this directory /var/lib/kubelet/pods/27153f2b-7664-11e9-95b0-000af7854ba0/volumes/kubernetes.io~local-volume/local-pv-3a82f760. /var/lib/kubelet is managed by kubelet.
... Output: mount: special device /mnt/disks/disk1 does not exist
According to the log, please check directory /mnt/disks/disk1.
from sig-storage-local-static-provisioner.
hangchan
commented on May 18, 2024
It works after I manually create the directory and bind mount the filesystem, otherwise it gives me that error.
Here's my fstab entry
UUID=95d3ccf4-9445-484c-a0f9-7b1d62be8e1e /mnt/disks/disk1 xfs defaults,noatime,discard 0 0
Commands to get it to work.
mkdir /var/lib/kubelet/pods/27153f2b-7664-11e9-95b0-000af7854ba0/volumes/kubernetes.io~local-volume/local-pv-3a82f760
mount -o bind /mnt/disks/disk1 /var/lib/kubelet/pods/27153f2b-7664-11e9-95b0-000af7854ba0/volumes/kubernetes.io~local-volume/local-pv-3a82f760
from sig-storage-local-static-provisioner.
cofyc
commented on May 18, 2024
Can you paste the output of mountpoint /mnt/disks/disk1?
from sig-storage-local-static-provisioner.
cofyc
commented on May 18, 2024
kubelet should retry to bind /mnt/disks/disk1 to /var/lib/kubelet/pods/27153f2b-7664-11e9-95b0-000af7854ba0/volumes/kubernetes.io~local-volume/local-pv-3a82f760
from sig-storage-local-static-provisioner.
msau42
commented on May 18, 2024
Also, is your kubelet running in a container?
from sig-storage-local-static-provisioner.
hangchan
commented on May 18, 2024
Yes my kubelet is running in a container.
# mountpoint /mnt/disks/disk1
/mnt/disks/disk1 is a mountpoint
Logs from kubelet
E0516 14:14:44.254598 4210 kubelet.go:1612] Unable to mount volumes for pod "mysql-hybrid-bidder-1_mysql-hybrid-bidder(60d3bb49-77e4-11e9-95b0-000af7854ba0)": timeout expired waiting for volumes to attach or mount for pod "mysql-hybrid-bidder"/"mysql-hybrid-bidder-1". list of unmounted volumes=[data]. list of unattached volumes=[backup data mysql-agent-token-dqqbq]; skipping pod
E0516 14:14:44.254642 4210 pod_workers.go:186] Error syncing pod 60d3bb49-77e4-11e9-95b0-000af7854ba0 ("mysql-hybrid-bidder-1_mysql-hybrid-bidder(60d3bb49-77e4-11e9-95b0-000af7854ba0)"), skipping: timeout expired waiting for volumes to attach or mount for pod "mysql-hybrid-bidder"/"mysql-hybrid-bidder-1". list of unmounted volumes=[data]. list of unattached volumes=[backup data mysql-agent-token-dqqbq]
I0516 14:14:47.593610 4210 reconciler.go:252] operationExecutor.MountVolume started for volume "local-pv-449aa08b" (UniqueName: "kubernetes.io/local-volume/60d3bb49-77e4-11e9-95b0-000af7854ba0-local-pv-449aa08b") pod "mysql-hybrid-bidder-1" (UID: "60d3bb49-77e4-11e9-95b0-000af7854ba0")
E0516 14:14:47.596718 4210 mount_linux.go:152] Mount failed: exit status 32
Mounting command: mount
Mounting arguments: -o bind /mnt/disks/disk1 /var/lib/kubelet/pods/60d3bb49-77e4-11e9-95b0-000af7854ba0/volumes/kubernetes.io~local-volume/local-pv-449aa08b
Output: mount: special device /mnt/disks/disk1 does not exist
E0516 14:14:47.596754 4210 local.go:316] Mount of volume /var/lib/kubelet/pods/60d3bb49-77e4-11e9-95b0-000af7854ba0/volumes/kubernetes.io~local-volume/local-pv-449aa08b failed: mount failed: exit status 32
Mounting command: mount
Mounting arguments: -o bind /mnt/disks/disk1 /var/lib/kubelet/pods/60d3bb49-77e4-11e9-95b0-000af7854ba0/volumes/kubernetes.io~local-volume/local-pv-449aa08b
Output: mount: special device /mnt/disks/disk1 does not exist
E0516 14:14:47.598947 4210 nestedpendingoperations.go:267] Operation for "\"kubernetes.io/local-volume/60d3bb49-77e4-11e9-95b0-000af7854ba0-local-pv-449aa08b\" (\"60d3bb49-77e4-11e9-95b0-000af7854ba0\")" failed. No retries permitted until 2019-05-16 14:16:49.598886503 +0000 UTC m=+843033.722436333 (durationBeforeRetry 2m2s). Error: "MountVolume.SetUp failed for volume \"local-pv-449aa08b\" (UniqueName: \"kubernetes.io/local-volume/60d3bb49-77e4-11e9-95b0-000af7854ba0-local-pv-449aa08b\") pod \"mysql-hybrid-bidder-1\" (UID: \"60d3bb49-77e4-11e9-95b0-000af7854ba0\") : mount failed: exit status 32\nMounting command: mount\nMounting arguments: -o bind /mnt/disks/disk1 /var/lib/kubelet/pods/60d3bb49-77e4-11e9-95b0-000af7854ba0/volumes/kubernetes.io~local-volume/local-pv-449aa08b\nOutput: mount: special device /mnt/disks/disk1 does not exist\n\n
from sig-storage-local-static-provisioner.
hangchan
commented on May 18, 2024
Thanks @msau42. My kubelet was running in a container.
Added additional binds and it's working.
"HostConfig": {
"Binds": [
"/mnt/disks/disk1:/mnt/disks/disk1",
"/mnt/disks/disk2:/mnt/disks/disk2"
],
from sig-storage-local-static-provisioner.
Related Issues (20)
- Move examples to have all of them in a single location HOT 5
- Release Helm chart for v2.5.0 HOT 5
- Transition Windows daemonset to become a HostProcess container HOT 4
- Helm -- nodeSelector not applied to windows daemonset HOT 4
- Remove generation code for helm v2 HOT 1
- Missing doc on 2.5.0 version compatibility with K8s versions in readme.md HOT 1
- v2.5.0 Release breaks helm chart, version, etc. and provides v2.4.0 application HOT 2
- Switch from k8s.gcr.io to registry.k8s.io HOT 1
- Use Helm chart manifests in the e2e tests HOT 6
- Please help me HOT 9
- Can't scrape metrics for more than 1000 nodes HOT 4
- Add the ability to disable the Windows daemonset HOT 6
- Multiple PersistentVolumes from single local device HOT 7
- Take ownership of the Helm repo on Artifact Hub HOT 8
- Deletion of PVCs on statefulsetscale-up due to low intervals HOT 7
- Failed to discover local volumes: failed to get ReclaimPolicy from storage class "fast-disks": storageclass.storage.k8s.io "fast-disks" not found HOT 1
- 0/1 nodes are available: 1 node(s) didn't find available persistent volumes to bind. HOT 2
- Fix Critical CVE-2019-8457 by upgrading base image to debian-base:bookworm-v1.0.1
- node-cleanup deleter process repeatedly deleting unused, but valid PVs
- Any plans to cut a new release soon? HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from sig-storage-local-static-provisioner.