Comments (3)
woodsaj
commented on July 23, 2024
1
looks like most of these issues can be solved by simply passing the "-x" argument to du. This will prevent du from traversing into other fileystems that are mounted on top of the target path.
root@k8s-dev-node1:/# du -s /var/lib/docker
du: cannot access '/var/lib/docker/aufs/mnt/18e46a015e69991737dcdb66591c404ca1071621e701800bb1c36ee2c9727d22/proc_host/21492/task/21492/fd/3': No such file or directory
du: cannot access '/var/lib/docker/aufs/mnt/18e46a015e69991737dcdb66591c404ca1071621e701800bb1c36ee2c9727d22/proc_host/21492/task/21492/fdinfo/3': No such file or directory
du: cannot access '/var/lib/docker/aufs/mnt/18e46a015e69991737dcdb66591c404ca1071621e701800bb1c36ee2c9727d22/proc_host/21492/fd/4': No such file or directory
du: cannot access '/var/lib/docker/aufs/mnt/18e46a015e69991737dcdb66591c404ca1071621e701800bb1c36ee2c9727d22/proc_host/21492/fdinfo/4': No such file or directory
du: cannot access '/var/lib/docker/aufs/mnt/18e46a015e69991737dcdb66591c404ca1071621e701800bb1c36ee2c9727d22/proc/21492/task/21492/fd/3': No such file or directory
du: cannot access '/var/lib/docker/aufs/mnt/18e46a015e69991737dcdb66591c404ca1071621e701800bb1c36ee2c9727d22/proc/21492/task/21492/fdinfo/3': No such file or directory
du: cannot access '/var/lib/docker/aufs/mnt/18e46a015e69991737dcdb66591c404ca1071621e701800bb1c36ee2c9727d22/proc/21492/fd/4': No such file or directory
du: cannot access '/var/lib/docker/aufs/mnt/18e46a015e69991737dcdb66591c404ca1071621e701800bb1c36ee2c9727d22/proc/21492/fdinfo/4': No such file or directory
12153840 /var/lib/docker
root@k8s-dev-node1:/# du -sx /var/lib/docker
11928828 /var/lib/docker
from snap-plugin-collector-docker.
IzabellaRaulin
commented on July 23, 2024
Thank You for reporting this issue. As You mentioned, it might be related with a race condition where a path existed but removed before du -s is executed. I'll add bug label to it.
from snap-plugin-collector-docker.
daniellee
commented on July 23, 2024
This bug is a real problem for us so I investigated some more.
The first problem is that the error from the du -s command is swallowed causing a panic when an error does occur. I made a small change to log the error.
Before: https://github.com/intelsdi-x/snap-plugin-collector-docker/blob/master/fs/fs.go#L370
out, _ := exec.Command("du", "-s", dir).Output()After:
out, err := exec.Command("du", "-s", dir).Output()
if err != nil {
return 0, err
}This prevents the panic but spams the logs with lots of errors:
time="2016-11-17T19:25:22Z" level=debug msg="Cannot get usage for dir=`/var/lib/docker`, err=Disk usage not found for /var/lib/dockerCannot get usage for dir=`/var/lib/docker`, err=Disk usage not found for /var/lib/dockerCannot get usage for dir=`/var/lib/docker`, err=Disk usage not found for /var/lib/dockerWORKER root, ERROR {exit status 1} for /var/lib/docker" _module=plugin-exec io=stderr plugin=snap-plugin-collector-dockerFurther investigation shows the real error (got by running du -s /var/lib/docker inside the container):
root@k8s-sjc-node4:/# du -s /var/lib/docker
du: cannot access '/var/lib/docker/aufs/mnt/d14c83ef98fe0a388bd2350a48dfd90685e7720cfd660768026fbfb5f19f5d4f/proc_host/25381/task/25381/fd/4': No such file or directory
du: cannot access '/var/lib/docker/aufs/mnt/d14c83ef98fe0a388bd2350a48dfd90685e7720cfd660768026fbfb5f19f5d4f/proc_host/25381/task/25381/fdinfo/4': No such file or directory
du: cannot access '/var/lib/docker/aufs/mnt/d14c83ef98fe0a388bd2350a48dfd90685e7720cfd660768026fbfb5f19f5d4f/proc_host/25381/fd/3': No such file or directory
du: cannot access '/var/lib/docker/aufs/mnt/d14c83ef98fe0a388bd2350a48dfd90685e7720cfd660768026fbfb5f19f5d4f/proc_host/25381/fdinfo/3': No such file or directory
du: cannot access '/var/lib/docker/aufs/mnt/d14c83ef98fe0a388bd2350a48dfd90685e7720cfd660768026fbfb5f19f5d4f/proc/25381/task/25381/fd/4': No such file or directory
du: cannot access '/var/lib/docker/aufs/mnt/d14c83ef98fe0a388bd2350a48dfd90685e7720cfd660768026fbfb5f19f5d4f/proc/25381/task/25381/fdinfo/4': No such file or directory
du: cannot access '/var/lib/docker/aufs/mnt/d14c83ef98fe0a388bd2350a48dfd90685e7720cfd660768026fbfb5f19f5d4f/proc/25381/fd/3': No such file or directory
du: cannot access '/var/lib/docker/aufs/mnt/d14c83ef98fe0a388bd2350a48dfd90685e7720cfd660768026fbfb5f19f5d4f/proc/25381/fdinfo/3': No such file or directory
34670904 /var/lib/dockerWe are using this docker image with the following mounts (which is why proc_host is in the list of errors):
volumeMounts:
- mountPath: /sys/fs/cgroup
name: cgroup
- mountPath: /var/run/docker.sock
name: docker-sock
- mountPath: /var/lib/docker
name: fs-stats
- mountPath: /usr/local/bin/docker
name: docker
- mountPath: /proc_host
name: proc
- mountPath: /opt/snap/tasks
name: snap-tasks
I'm not sure what the best solution is, maybe using the exclude argument with du e.g. du -s /var/lib/docker --exclude=proc*
from snap-plugin-collector-docker.
Related Issues (20)
- Unused and incorrect import in tool.go
- Add option to configure procfs path for use inside docker containers
- Wildcard currently results in double the metric collection HOT 1
- Expose cpu.shares specified per container HOT 3
- loading the plugin error
- Cannot build docker images
- Plugin build requirements and patch agnostic scripts HOT 1
- CoreOS Support HOT 11
- Use logrus to logging in docker plugin
- No test for client and fs package
- Add Kubernetes labels as tags to docker metrics HOT 2
- Expose cpuset subsystem metrics per container HOT 1
- Only files in /example/tasks should be Task Manifests HOT 3
- Cgroups driver incorrectly handled on RHEL
- document the docker endpoint config
- Filesystem metrics not available HOT 1
- Support for docker on Windows HOT 1
- Metrics total_usage has been renamed (backward compatibility broken)
- docker-collector hang up few second after startup
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from snap-plugin-collector-docker.