Comments (11)
rjnagal
commented on May 7, 2024
Is this because of the systemd hierarchy?
from cadvisor.
vmarmol
commented on May 7, 2024
@rjnagal we have code that should handle that, but it doesn't look like it is working...
from cadvisor.
vmarmol
commented on May 7, 2024
Fixed by #220
from cadvisor.
bliss
commented on May 7, 2024
Good day! All of a sudden I found out my statistics have gone. After some examination i found that container names in influxdb became like that one
/system.slice/docker-a89ad7021d73e64de54b8f25fec4bf8a3fb23441b661b9b5a2c6cfbcb000b044.scope
instead of more habitual
/k8s_mariadb2562281893.30a8f541_6839bb45-0ff5-4a68-ae88-7a5203814750-k9349_91ad4704-04c1-490c-bed1-ebb750484d4f_c7a0a08a-52e1-11e5-b267-5
2540051caca_0cb6d646
and no network statistics.
How can i fix it?
OS: CentOS7
cat /etc/redhat-release
CentOS Linux release 7.1.1503 (Core)
standalone cAdvisor
cadvisor --version
cAdvisor version 0.14.0
from cadvisor.
vishh
commented on May 7, 2024
k8s_* containers are created by kubernetes. cAdvisor has no control over
the alias of containers.
An issue related to network stats was fixed recently. The next release
should have the required fix.
On Tue, Sep 8, 2015 at 6:44 AM, bliss [email protected] wrote:
Good day! All of a sudden I found out my statistics have gone. After some
examination i found that container names in influxdb become like that one/system.slice/docker-a89ad7021d73e64de54b8f25fec4bf8a3fb23441b661b9b5a2c6cfbcb000b044.scope
instead of more habitual
/k8s_mariadb2562281893.30a8f541_6839bb45-0ff5-4a68-ae88-7a5203814750-k9349_91ad4704-04c1-490c-bed1-ebb750484d4f_c7a0a08a-52e1-11e5-b267-5
2540051caca_0cb6d646and no network statistics.
How can i fix it?
OS: CentOS7cat /etc/redhat-release
CentOS Linux release 7.1.1503 (Core)
—
Reply to this email directly or view it on GitHub
#143 (comment).
from cadvisor.
bliss
commented on May 7, 2024
Thank you for your reply.
"k8s_* containers are created by kubernetes. cAdvisor has no control over the alias of containers." --> But it had, because it definitely fed those aliases to influxdb. Moreover, when polled via API cadvisor returns only system slice statistics, no data for docker containers. Looks like it cannot get access to cgroups or something like that. I turned selinux on and off but no difference. Last time we resolved such issue with cadvisor update: from 0.6.2 to 0.14.
from cadvisor.
vishh
commented on May 7, 2024
Thanks for clarifying the issue! Can you post logs from cAdvisor and the
output of /validate?
On Tue, Sep 8, 2015 at 1:08 PM, bliss [email protected] wrote:
Thank you for your reply.
"k8s_* containers are created by kubernetes. cAdvisor has no control over
the alias of containers." --> But it had, because it definitely fed those
aliases to influxdb. Moreover, when polled via API cadvisor returns only
system slice statistics, no data for docker containers. Looks like it
cannot get access to cgroups or something like that. I turned selinux on
and off but no difference. Last time we resolved such issue with cadvisor
update: from 0.6.2 to 0.14.—
Reply to this email directly or view it on GitHub
#143 (comment).
from cadvisor.
bliss
commented on May 7, 2024
@vishh could please make it some more clear? Even with --logtostderr=true --v=2 I got no log output except start/stop info. And let me please know where can I get the '/validate' ? I tried
curl http://localhost:4194/validate
<a href="/validate/">Moved Permanently</a>.
Thank you!
from cadvisor.
bliss
commented on May 7, 2024
When I started cadvisor from command line as root I got all statistics. Maybe the problem arises when cadvisor started by systemd
from cadvisor.
vishh
commented on May 7, 2024
Can you try curl -L http://localhost:4194/validate?
Do check if your unit files have all the necessary mounts as mentioned in
the docker running instructions in the repo.
On Tue, Sep 8, 2015 at 1:53 PM, bliss [email protected] wrote:
When I started cadvisor from command line as root I got all statistics.
Maybe the problem arises when cadvisor started by systemd—
Reply to this email directly or view it on GitHub
#143 (comment).
from cadvisor.
bliss
commented on May 7, 2024
cAdvisor version: 0.14.0
OS version: CentOS Linux 7 (Core)
Kernel version: [Supported and recommended]
Kernel version is 3.10.0-229.11.1.el7.centos.x86_64. Versions >= 2.6 are supported. 3.0+ are recommended.
Cgroup setup: [Supported and recommended]
Available cgroups: map[cpu:1 memory:1 cpuset:1 devices:1 freezer:1 net_cls:1 blkio:1 perf_event:1 hugetlb:1 cpuacct:1]
Following cgroups are required: [cpu cpuacct]
Following other cgroups are recommended: [memory blkio cpuset devices freezer]
Hierarchical memory accounting enabled. Reported memory usage includes memory used by child containers.
Cgroup mount setup: [Supported and recommended]
Cgroups are mounted at /sys/fs/cgroup.
Cgroup mount directories: blkio cpu cpu,cpuacct cpuacct cpuset devices freezer hugetlb memory net_cls perf_event systemd
Any cgroup mount point that is detectible and accessible is supported. /sys/fs/cgroup is recommended as a standard location.
Cgroup mounts:
cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0
cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpuacct,cpu 0 0
cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0
cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0
cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0
cgroup /sys/fs/cgroup/net_cls cgroup rw,nosuid,nodev,noexec,relatime,net_cls 0 0
cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0
cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0
cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0
Docker version: [Unknown]
Could not parse docker version. Docker version is Unknown. Versions >= 1.0 are supported. 1.2+ are recommended.
Docker driver setup: [Supported and recommended]
Docker exec driver is native-0.2. Storage driver is overlay.
systemd is being used to create cgroups.
Docker container state directory is at "/var/lib/docker/containers" and is accessible.
Block device setup: [Supported, but not recommended]
None of the devices support 'cfq' I/O scheduler. No disk stats can be reported.
Disk "vda" Scheduler type "none".
Disk "vdb" Scheduler type "none".
Looks pretty ridiculous but after I ran cadvisor from under root on one of my nodes it got self-healed and started to return required stats being run as usually by systemd. The 'validate' output above from another node with the same problems. Diff between /validate calls on the 'not-healed' node and 'healed' one below:
diff n1.log n2.log
10c10
< Available cgroups: map[cpu:1 memory:1 cpuset:1 devices:1 freezer:1 net_cls:1 blkio:1 perf_event:1 hugetlb:1 cpuacct:1]
---
> Available cgroups: map[cpu:1 freezer:1 net_cls:1 blkio:1 perf_event:1 hugetlb:1 cpuset:1 cpuacct:1 memory:1 devices:1]
33,34c33,34
< Docker version: [Unknown]
< Could not parse docker version. Docker version is Unknown. Versions >= 1.0 are supported. 1.2+ are recommended.
---
> Docker version: [Supported and recommended]
> Docker version is 1.6.2. Versions >= 1.0 are supported. 1.2+ are recommended.
36d35
<
On all nodes docker is installed, functioning and of the same version:
$ docker --version
Docker version 1.6.2, build ba1f6c3/1.6.2
from cadvisor.
Related Issues (20)
- how to install this on EKS cluster via helm chart
- how to reduce monitor metrics HOT 1
- Container fails at start on Synology with Prometheus HOT 1
- Why does cAdvisor need so many host resources? HOT 6
- in cadvisor process i can't see docker images
- cadvisor print a lot of 'manager.go:1116 Failed to create existing container …' logs. HOT 1
- Docker subcontainers now showing up in webUI when new 'Use containerd for pulling and storing images' beta feature enabled
- Is it possible to blacklist /kubepods.slice ?
- Please fix security vulnerabilities for v0.47.2 HOT 1
- 8080 port HOT 4
- Segmentation violation in podman/factory.go
- Issue launching container with cgroups v2 / AWS Linux 2023 HOT 1
- Memory Usage always zero HOT 2
- Retrieving the PID of a specific container
- Please fix the CVEs for v0.47.2 HOT 3
- Support collecting hugepage info from /sys/devices/system/node
- missing binaries for v0.48 HOT 1
- cadvisor:latest container tag is outdated HOT 1
- libipmctl build dependencies are failing on new alpine image HOT 2
- Latest release on github points to older version. HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cadvisor.