Comments (12)
Thank you Ruiwen, for the cardinality issue I have some comments on it:
This is very different from the existing implementation of pod_start_total_duration_seconds
. Waiting for @dashpole or others from sig-instrumentation to give some advice on the best way to record one-time per-pod metrics like this.
from kubernetes.
cc: @ruiwen-zhao for review.
from kubernetes.
/sig instrumentation
from kubernetes.
/sig node
from kubernetes.
Just to bring up previous discussion around metric cardinality, adding both pod name and node name to metric labels might be too much cardinality. We need to come up with a way to address this.
cc @SergeyKanzhelev @logicalhan @dashpole
from kubernetes.
Thank you Ruiwen, for the cardinality issue I have some comments on it:
-
Kubernetes already has some metrics from scheduler that include the pod name, namespace, and node name as metrics label:
-
kubernetes/pkg/kubelet/metrics/collectors/log_metrics.go
Lines 32 to 34 in 06b813f
Kubenetes has another metric kubelet_container_log_filesystem_used_bytes
that also use pod name and namespace as metrics labels.
- KSM also exports pod metrics in prometheus format and some metrics have pod name and namespace as metrics labels: https://github.com/kubernetes/kube-state-metrics/blob/main/docs/metrics/workload/pod-metrics.md.
from kubernetes.
from kubernetes.
@yujuhong Yes. The pod_start_total_duration_seconds
is a Distribution over all pods in the node, but newly added metric in this feature proposed to add a gauge metric that provides the exact startup time for a single pod to become ready.
@dashpole Hi David, could you please provide some insights here? Thanks!
from kubernetes.
A few questions to get the discussion started:
- Why a gauge instead of a histogram? A gauge is OK when looking at a single stream, or if you want to graph the average. But durations are often best represented by a histogram, as you can graph percentiles, or show a distribution. But if you graph a bunch of gauges, you will just see lots of lines on the graph, which isn't that helpful.
- Does this need to be in the kubelet? IIUC, this metric is produced by watching pods, and emitting a metric when it becomes ready for the first time. It doesn't need any special knowledge that the kubelet has, right?
- How long would the metric exist for? The startup will occur at the very beginning of the pod's life (in a single instant). Most pod-level metrics exist for the lifetime of the pod, but doing that would mean any aggregation would be less meaningful. Averaging the startup time of all currently-running pods in the cluster won't tell you if pod startup is currently slow. We could emit the metric for an arbitrary amount of time (e.g. 5 minutes), but that risks a scraper missing a pod entirely.
Bikeshedding: From the names, kubelet_pod_full_startup_duration_seconds
vs pod_start_total_duration_seconds
, I wouldn't know what the difference is. Would pod_ready_duration_seconds
or pod_first_ready_duration_seconds
be better?
from kubernetes.
@dashpole Hi David thank you for the comment.
Why a gauge instead of a histogram? A gauge is OK when looking at a single stream, or if you want to graph the average. But durations are often best represented by a histogram, as you can graph percentiles, or show a distribution. But if you graph a bunch of gauges, you will just see lots of lines on the graph, which isn't that helpful.
I want to use a gauge because I want to record the exact startup time of the pod, and it will allow users to know the exact time it takes for their pods to become ready to serve. With the pod-level metric, users could also group them together under the workload (e.g. deployment).
Does this need to be in the kubelet? IIUC, this metric is produced by watching pods, and emitting a metric when it becomes ready for the first time. It doesn't need any special knowledge that the kubelet has, right?
I use kubelet as kubelet will track the status of each pod in pod_startup_latency_tracker, and kubelet will watch for the status change of each pod. Also, kubelet is usually the first layer to process the pod status and it's a stable component (compared to other components in the cluster like kube-state-metrics which I usually see out-of-memory issue..) Do you have any recommendation for other places to add such metric?
How long would the metric exist for? The startup will occur at the very beginning of the pod's life (in a single instant). Most pod-level metrics exist for the lifetime of the pod, but doing that would mean any aggregation would be less meaningful. Averaging the startup time of all currently-running pods in the cluster won't tell you if pod startup is currently slow. We could emit the metric for an arbitrary amount of time (e.g. 5 minutes), but that risks a scraper missing a pod entirely.
For "Most pod-level metrics exist for the lifetime of the pod, but doing that would mean any aggregation would be less meaningful", can you provide more context here to help me understand? Thanks!
From the names, kubelet_pod_full_startup_duration_seconds vs pod_start_total_duration_seconds, I wouldn't know what the difference is. Would pod_ready_duration_seconds or pod_first_ready_duration_seconds be better?
pod_first_ready_duration_seconds
looks good to me!
from kubernetes.
Does this need to be in the kubelet? IIUC, this metric is produced by watching pods, and emitting a metric when it becomes ready for the first time. It doesn't need any special knowledge that the kubelet has, right?
Would something like kube-state-metrics more suitable for this?
https://kubernetes.io/docs/concepts/cluster-administration/kube-state-metrics/
from kubernetes.
/assign @JeffLuoo
/assign
/triage accepted
from kubernetes.
Related Issues (20)
- Prometheus allows "/" in label values but kubernetes does not HOT 5
- [KEP-4191] Support case when writeable layer and readable layers are on same mount but in different locations HOT 6
- kubectl logs -f stop after log rotation HOT 7
- PVC Mounts fails when STAGE_UNSTAGE_VOLUME check fails for k8s EFS CSI HOT 7
- Failed: Build failed outside out of test results HOT 1
- RBAC not work for /healthz HOT 5
- PreStop don't work if hostNetwork set to true HOT 3
- Still seeing the issue for endpoints staying out of sync HOT 16
- Using runtime scheme.Convert for Validating Admission Policy version conversion results in error HOT 4
- When setting kubeReserved non-provided values should fallback to config HOT 3
- option to suppress DNSConfigForming warning log messages and events HOT 2
- CVE PLACEHOLDER ISSUE HOT 3
- Protect from Unauthorized access of Volume HOT 18
- Unable to access control plane after Kubernetes worker node reboot HOT 4
- Failure cluster [03a05e59...] `[sig-network] Networking Granular Checks: Services should function for service endpoints using hostNetwork` HOT 6
- Pod Stuck In Terminating State After Kublet Restart HOT 5
- kubectl >= 1.30.0 triggers leak of goroutines in containerd on `kubectl exec` HOT 38
- Add debug_redact to fields in message AuthConfig (image streaming CRI API) HOT 7
- [FG:InPlacePodVerticalScaling] Change in version-skewed behavior in v1.31 HOT 18
- Memory EmptyDir tmpfs size capped at 100% of node RAM size, not SizeLimit as specified HOT 15
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kubernetes.