Comments (14)
You need to provide logs from all dashboard containers first.
from dashboard.
During log collection,i found this line in dashboard-api container:
E0502 11:07:49.844195 1 manager.go:96] Metric client health check failed: the server is currently unable to handle the request (get services kubernetes-dashboard-metrics-scraper). Retrying in 30 seconds.
Can it be the reason?
dashboard-api has these args:
containers:
- args:
- '--namespace=kube-system'
- '--metrics-scraper-service-name=kubernetes-dashboard-metrics-scraper'
env:
- name: CSRF_KEY
valueFrom:
secretKeyRef:
key: private.key
name: kubernetes-dashboard-csrf
image: docker.io/kubernetesui/dashboard-api:1.5.0
dashboard-scraper logs are filled with probes:
10.150.23.46 - - [02/May/2024:11:05:48 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.29+"
10.150.23.46 - - [02/May/2024:11:05:58 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.29+"
I0502 11:06:02.711109 1 main.go:145] Database updated: 12 nodes, 310 pods
10.150.23.46 - - [02/May/2024:11:06:08 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.29+"
10.150.23.46 - - [02/May/2024:11:06:18 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.29+"
10.150.23.46 - - [02/May/2024:11:06:28 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.29+"
10.150.23.46 - - [02/May/2024:11:06:38 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.29+"
10.150.23.46 - - [02/May/2024:11:06:48 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.29+"
10.150.23.46 - - [02/May/2024:11:06:58 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.29+"
I0502 11:07:02.747414 1 main.go:145] Database updated: 12 nodes, 310 pods
10.150.23.46 - - [02/May/2024:11:07:08 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.29+"
10.150.23.46 - - [02/May/2024:11:07:18 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.29+"
10.150.23.46 - - [02/May/2024:11:07:28 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.29+"
10.150.23.46 - - [02/May/2024:11:07:38 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.29+"
10.150.23.46 - - [02/May/2024:11:07:48 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.29+"
10.150.23.46 - - [02/May/2024:11:07:58 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.29+"
I0502 11:08:02.701078 1 main.go:145] Database updated: 12 nodes, 310 pods
10.150.23.46 - - [02/May/2024:11:08:08 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.29+"
10.150.23.46 - - [02/May/2024:11:08:18 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.29+"
10.150.23.46 - - [02/May/2024:11:08:28 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.29+"
10.150.23.46 - - [02/May/2024:11:08:38 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.29+"
10.150.23.46 - - [02/May/2024:11:08:48 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.29+"
metrics-server logs:
I0502 10:12:39.303482 1 serving.go:374] Generated self-signed cert (/tmp/apiserver.crt, /tmp/apiserver.key)
I0502 10:12:39.861123 1 handler.go:275] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager
I0502 10:12:39.979541 1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
I0502 10:12:39.979591 1 shared_informer.go:311] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0502 10:12:39.979654 1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I0502 10:12:39.979740 1 shared_informer.go:311] Waiting for caches to sync for RequestHeaderAuthRequestController
I0502 10:12:39.979763 1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I0502 10:12:39.979852 1 shared_informer.go:311] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0502 10:12:39.980179 1 dynamic_serving_content.go:132] "Starting controller" name="serving-cert::/tmp/apiserver.crt::/tmp/apiserver.key"
I0502 10:12:39.981162 1 secure_serving.go:213] Serving securely on [::]:10250
I0502 10:12:39.981201 1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I0502 10:12:40.080042 1 shared_informer.go:318] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0502 10:12:40.080062 1 shared_informer.go:318] Caches are synced for RequestHeaderAuthRequestController
I0502 10:12:40.080152 1 shared_informer.go:318] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
Metrics-server args:
--secure-port=10250
--cert-dir=/tmp
--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
--kubelet-use-node-status-port
--metric-resolution=15s
--kubelet-insecure-tls
from dashboard.
Ye, for some reason it can't access metrics scraper. Isn't there more detailed error message in the api logs?
from dashboard.
from dashboard.
Check the beginning of logs when the app is being initialized.
from dashboard.
I0502 11:19:08.330608 1 main.go:40] "Starting Kubernetes Dashboard API" version="1.5.0"
I0502 11:19:08.331182 1 init.go:47] Using in-cluster config
I0502 11:19:08.364971 1 main.go:118] "Successful initial request to the apiserver" version="v1.29.3-eks-adc7111"
I0502 11:19:08.365756 1 client.go:268] Creating in-cluster Sidecar client
I0502 11:19:08.381661 1 main.go:95] "Listening and serving on" address="0.0.0.0:8000"
2024/05/02 11:19:08 Getting list of all cron jobs in the cluster
2024/05/02 11:19:09 Getting list of all replica sets in the cluster
2024/05/02 11:19:09 Getting list of all replication controllers in the cluster
2024/05/02 11:19:09 Getting list of ingress classes in the cluster
2024/05/02 11:19:09 Getting list of all services in the cluster
2024/05/02 11:19:09 Getting list of all stateful sets in the cluster
2024/05/02 11:19:09 Getting list of all pods in the cluster
2024/05/02 11:19:09 Getting list of all deployments in the cluster
2024/05/02 11:19:09 Getting list of all jobs in the cluster
2024/05/02 11:19:09 Getting list config maps in the namespace
2024/05/02 11:19:09 Getting list of secrets in &{[]} namespace
2024/05/02 11:19:09 Getting list persistent volumes
2024/05/02 11:19:09 Getting list of RBAC roles
2024/05/02 11:19:09 Getting list of storage classes in the cluster
2024/05/02 11:19:09 Getting list of all roleBindings in the cluster
2024/05/02 11:19:09 Getting list of all roles in the cluster
2024/05/02 11:19:09 Getting list of namespaces
2024/05/02 11:19:09 Getting list of all clusterRoleBindings in the cluster
2024/05/02 11:19:09 Getting list of all deployments in the cluster
2024/05/02 11:19:13 Getting list persistent volumes
2024/05/02 11:19:18 Getting pod metrics
2024/05/02 11:19:39 Getting list of all deployments in the cluster
2024/05/02 11:19:54 Getting list of all deployments in the cluster
2024/05/02 11:19:58 Getting list of namespaces
2024/05/02 11:19:58 Getting list of all cron jobs in the cluster
2024/05/02 11:19:58 Getting list of all pods in the cluster
2024/05/02 11:19:58 Getting list of all replication controllers in the cluster
2024/05/02 11:19:58 Getting list of all deployments in the cluster
2024/05/02 11:19:58 Getting list of all jobs in the cluster
2024/05/02 11:19:58 Getting list of all replica sets in the cluster
2024/05/02 11:19:58 Getting list of all stateful sets in the cluster
2024/05/02 11:19:58 Getting pod metrics
2024/05/02 11:19:59 Getting list of namespaces
2024/05/02 11:19:59 Getting list of all cron jobs in the cluster
2024/05/02 11:19:59 Getting list of all deployments in the cluster
2024/05/02 11:19:59 Getting list of all deployments in the cluster
2024/05/02 11:19:59 Getting list of all jobs in the cluster
2024/05/02 11:19:59 Getting list of all replication controllers in the cluster
2024/05/02 11:19:59 Getting list of all replica sets in the cluster
2024/05/02 11:19:59 Getting list of all pods in the cluster
2024/05/02 11:19:59 Getting list of all stateful sets in the cluster
2024/05/02 11:19:59 Getting pod metrics
2024/05/02 11:20:01 Getting list of all deployments in the cluster
2024/05/02 11:20:02 Getting list of all deployments in the cluster
2024/05/02 11:20:03 Getting list of all jobs in the cluster
2024/05/02 11:20:04 Getting list of all pods in the cluster
2024/05/02 11:20:04 Getting list of all jobs in the cluster
2024/05/02 11:20:04 Getting list of all deployments in the cluster
2024/05/02 11:20:04 Getting list of all cron jobs in the cluster
2024/05/02 11:20:04 Getting list of namespaces
2024/05/02 11:20:04 Getting list of all stateful sets in the cluster
2024/05/02 11:20:04 Getting list of all replication controllers in the cluster
2024/05/02 11:20:04 Getting list of all replica sets in the cluster
2024/05/02 11:20:04 Getting pod metrics
2024/05/02 11:20:04 Getting list of all pods in the cluster
2024/05/02 11:20:06 Getting list of all deployments in the cluster
2024/05/02 11:20:06 Getting list of all cron jobs in the cluster
2024/05/02 11:20:06 Getting list of namespaces
2024/05/02 11:20:06 Getting list of all replication controllers in the cluster
2024/05/02 11:20:06 Getting list of all replica sets in the cluster
2024/05/02 11:20:06 Getting list of all pods in the cluster
2024/05/02 11:20:06 Getting list of all stateful sets in the cluster
2024/05/02 11:20:06 Getting pod metrics
2024/05/02 11:20:06 Getting list of all pods in the cluster
2024/05/02 11:20:06 Getting list of all jobs in the cluster
2024/05/02 11:20:07 Getting list of all deployments in the cluster
2024/05/02 11:20:07 Getting pod metrics
2024/05/02 11:20:09 Getting list of all deployments in the cluster
2024/05/02 11:20:10 Getting pod metrics
2024/05/02 11:20:11 Getting list of namespaces
2024/05/02 11:20:11 Getting list of all pods in the cluster
2024/05/02 11:20:11 Getting pod metrics
2024/05/02 11:20:12 Getting list of all deployments in the cluster
2024/05/02 11:20:13 Getting list of all replica sets in the cluster
2024/05/02 11:20:14 Getting list of namespaces
2024/05/02 11:20:15 Getting list of all replica sets in the cluster
2024/05/02 11:20:17 Getting list of all deployments in the cluster
2024/05/02 11:20:17 Getting list of all deployments in the cluster
2024/05/02 11:20:18 Getting list of namespaces
2024/05/02 11:20:20 Getting details of kubernetes-dashboard-metrics-server deployment in kube-system namespace
2024/05/02 11:20:22 Getting list of all deployments in the cluster
2024/05/02 11:20:24 Getting list of namespaces
2024/05/02 11:20:26 Getting details of kubernetes-dashboard-metrics-server deployment in kube-system namespace
2024/05/02 11:20:26 Getting replication controller kubernetes-dashboard-metrics-server-7c54c778c6 pods in namespace kube-system
2024/05/02 11:20:26 Getting details of kubernetes-dashboard-metrics-server-7c54c778c6 service in kube-system namespace
2024/05/02 11:20:27 Getting list of all deployments in the cluster
2024/05/02 11:20:27 Getting pod metrics
2024/05/02 11:20:28 Getting list of namespaces
2024/05/02 11:20:31 Getting replication controller kubernetes-dashboard-metrics-server-7c54c778c6 pods in namespace kube-system
2024/05/02 11:20:31 Getting details of kubernetes-dashboard-metrics-server-7c54c778c6 service in kube-system namespace
2024/05/02 11:20:31 Getting list of all deployments in the cluster
2024/05/02 11:20:32 Getting pod metrics
2024/05/02 11:20:33 Getting list of namespaces
2024/05/02 11:20:36 Getting list of all deployments in the cluster
2024/05/02 11:20:37 Getting list of all deployments in the cluster
2024/05/02 11:20:38 Getting list of all deployments in the cluster
2024/05/02 11:20:38 Getting list of namespaces
2024/05/02 11:20:43 Getting list of namespaces
2024/05/02 11:20:45 Getting pod metrics
2024/05/02 11:20:48 Getting list of namespaces
2024/05/02 11:20:53 Getting list of namespaces
2024/05/02 11:20:58 Getting list of namespaces
2024/05/02 11:21:03 Getting list of namespaces
2024/05/02 11:21:08 Getting list of namespaces
2024/05/02 11:21:13 Getting list of namespaces
E0502 11:21:18.808202 1 manager.go:96] Metric client health check failed: the server is currently unable to handle the request (get services kubernetes-dashboard-metrics-scraper). Retrying in 30 seconds.
2024/05/02 11:21:18 Getting list of namespaces
2024/05/02 11:21:23 Getting list of namespaces
2024/05/02 11:21:28 Getting list of namespaces
Here is the log from the beginning
from dashboard.
I have used the same configuration and observed similar logs.
I'm encountering the same issue where no graphics are displayed on the Kubernetes dashboard.
from dashboard.
You could check if this helps: #8835 (comment)
from dashboard.
I have tried adding below args but it still does not work for me.
args:
- --metrics-provider=none
- --sidecar-host=kubernetes-dashboard-metrics-scraper.kubernetes-dashboard
This is the API response for request path /api/v1/pod/
that I have received. any insights to why cumulativeMetrics
and metrics
field are set to empty and null respectively
{ "listMeta": { "totalItems": 2 }, "cumulativeMetrics": [], "status": { "running": 2, "pending": 0, "failed": 0, "succeeded": 0, "terminating": 0 }, "pods": [{...other pod fields, "metrics":"null" }], "errors": [] }
from dashboard.
If you set metrics provider to none you are effectively disabling metrics :). Do not override it or use 'sidecar' as value.
from dashboard.
It needs a protocol and port for the sidecar-host
--sidecar-host=http://kubernetes-dashboard-metrics-scraper.kubernetes-dashboard:8000
from dashboard.
@viram99 any progress here?
from dashboard.
/close
from dashboard.
@floreks: Closing this issue.
In response to this:
/close
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.
from dashboard.
Related Issues (20)
- k3s: Make ingress.tls completely optional. HOT 2
- With Ingress to reach the svc , the expose site report error /api/v1/login not found HOT 4
- Ingress not allowed when kong is disabled HOT 4
- auth via kubeconfig HOT 2
- difficulty getting started HOT 4
- Helm repository broken - 404 Not Found HOT 2
- Not able to install kubernetes-dashboard helm chart with default values file HOT 3
- Error installing dashboard on docker for desktop Macos HOT 2
- What happened to the `--enable-skip-login` flag? HOT 4
- Dashboard Helm Chart incompatible with Traefik ingress controller HOT 2
- Helm chart Kong crashes in GKE HOT 5
- Helm values Affinity does not get applied to Kong Deployment HOT 2
- 🚨 Helm `metrics-server` Deployment uses `web` `env` HOT 3
- dashboard login page responds with Unknown error (200): Http failure during parsing for http://10.255.15.136:32260/api/v1/csrftoken/login HOT 14
- Failed to receive reply from UDP server xxx:timeout. HOT 5
- LoadBalancer service type HOT 4
- nginx: [emerg] socket() [::]:8100 failed (97: Address family not supported by protocol) HOT 3
- dashboard does not display CPU/ram metrics HOT 2
- How long is the Kubernetes Dashboard Token valid for HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dashboard.