Comments (1)
huddlesj
commented on August 28, 2024
I did some testing tonight and to reproduce this issue in my lab I deleted a drive behind the worker-5 vm in my lab cluster and powered it off. This put it in a NotReady state. When I ran the healthcheck script it hung checking the bios version on worker-5.
# ./openshift-checks.sh
Using system:admin context
Gathering cluster information as system:admin:
Cluster version:
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.7.0 True False 9d Cluster version is 4.7.0
Cluster operators:
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE
authentication 4.7.0 True False False 2m49s
baremetal 4.7.0 True False False 9d
cloud-credential 4.7.0 True False False 26d
cluster-autoscaler 4.7.0 True False False 26d
config-operator 4.7.0 True False False 26d
console 4.7.0 True False False 9d
csi-snapshot-controller 4.7.0 True False False 9d
dns 4.7.0 True False False 20d
etcd 4.7.0 True False False 26d
image-registry 4.7.0 True False False 26d
ingress 4.7.0 True False False 20d
insights 4.7.0 True False False 26d
kube-apiserver 4.7.0 True False False 26d
kube-controller-manager 4.7.0 True False False 26d
kube-scheduler 4.7.0 True False False 26d
kube-storage-version-migrator 4.7.0 True False False 9d
machine-api 4.7.0 True False False 26d
machine-approver 4.7.0 True False False 26d
machine-config 4.7.0 True False False 9d
marketplace 4.7.0 True False False 9d
monitoring 4.7.0 True False False 9h
network 4.7.0 True True False 9d
node-tuning 4.7.0 True False False 9d
openshift-apiserver 4.7.0 True False False 9d
openshift-controller-manager 4.7.0 True False False 20d
openshift-samples 4.7.0 True False False 9d
operator-lifecycle-manager 4.7.0 True False False 26d
operator-lifecycle-manager-catalog 4.7.0 True False False 26d
operator-lifecycle-manager-packageserver 4.7.0 True False False 9d
service-ca 4.7.0 True False False 26d
storage 4.7.0 True False False 20d
Nodes:
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
master-0 Ready master 26d v1.20.0+ba45583 192.168.51.100 <none> Red Hat Enterprise Linux CoreOS 47.83.202102090044-0 (Ootpa) 4.18.0-240.10.1.el8_3.x86_64 cri-o://1.20.0-0.rhaos4.7.git8921e00.el8.51
master-1 Ready master 26d v1.20.0+ba45583 192.168.51.101 <none> Red Hat Enterprise Linux CoreOS 47.83.202102090044-0 (Ootpa) 4.18.0-240.10.1.el8_3.x86_64 cri-o://1.20.0-0.rhaos4.7.git8921e00.el8.51
master-2 Ready master 26d v1.20.0+ba45583 192.168.51.102 <none> Red Hat Enterprise Linux CoreOS 47.83.202102090044-0 (Ootpa) 4.18.0-240.10.1.el8_3.x86_64 cri-o://1.20.0-0.rhaos4.7.git8921e00.el8.51
worker-0 Ready worker 16d v1.20.0+ba45583 192.168.51.103 <none> Red Hat Enterprise Linux CoreOS 47.83.202102090044-0 (Ootpa) 4.18.0-240.10.1.el8_3.x86_64 cri-o://1.20.0-0.rhaos4.7.git8921e00.el8.51
worker-1 Ready worker 16d v1.20.0+ba45583 192.168.51.104 <none> Red Hat Enterprise Linux CoreOS 47.83.202102090044-0 (Ootpa) 4.18.0-240.10.1.el8_3.x86_64 cri-o://1.20.0-0.rhaos4.7.git8921e00.el8.51
worker-2 Ready worker 16d v1.20.0+ba45583 192.168.51.105 <none> Red Hat Enterprise Linux CoreOS 47.83.202102090044-0 (Ootpa) 4.18.0-240.10.1.el8_3.x86_64 cri-o://1.20.0-0.rhaos4.7.git8921e00.el8.51
worker-3 Ready worker 13d v1.20.0+ba45583 192.168.51.106 <none> Red Hat Enterprise Linux CoreOS 47.83.202102090044-0 (Ootpa) 4.18.0-240.10.1.el8_3.x86_64 cri-o://1.20.0-0.rhaos4.7.git8921e00.el8.51
worker-4 Ready worker 13d v1.20.0+ba45583 192.168.51.107 <none> Red Hat Enterprise Linux CoreOS 47.83.202102090044-0 (Ootpa) 4.18.0-240.10.1.el8_3.x86_64 cri-o://1.20.0-0.rhaos4.7.git8921e00.el8.51
worker-5 NotReady worker 13d v1.20.0+ba45583 192.168.51.108 <none> Red Hat Enterprise Linux CoreOS 47.83.202102090044-0 (Ootpa) 4.18.0-240.10.1.el8_3.x86_64 cri-o://1.20.0-0.rhaos4.7.git8921e00.el8.51
Masters: 3
Workers: 6
Others: 0
Total nodes: 9
Total pods: 357
Checking bios versions (using oc debug, it can take a while)
node/master-0: 1.13.0-2.el8
node/master-1: 1.13.0-2.el8
node/master-2: 1.13.0-2.el8
node/worker-0: 1.13.0-2.el8
node/worker-1: 1.13.0-2.el8
node/worker-2: 1.13.0-2.el8
node/worker-3: 1.13.0-2.el8
node/worker-4: 1.13.0-2.el8
from openshift-checks.
Related Issues (20)
- HTML output for oc debug doesn't work HOT 4
- Intel should report nothing if there are no intel cards (same for others) and return as skipped instead of 'info' HOT 1
- Cleanup some of the codes HOT 1
- Broken link found (/RHsyseng/openshift-checks/blob/main/info/mellanox-firmware-version)
- Broken link found (/RHsyseng/openshift-checks/blob/main/checks/port-thrashing)
- Broken link found (/RHsyseng/openshift-checks/blob/main/info/intel-firmware-version)
- ovn_cleanConntrack.sh hangs waiting for response if a node is in a NotReady status
- Broken link found (/RHsyseng/openshift-checks/blob/main/info/intel-firmware-version)
- Broken link found (/RHsyseng/openshift-checks/blob/main/info/mellanox-firmware-version)
- Implement a check to verify if the etcd database is closer to get full HOT 1
- CLI and HTML report not consistent HOT 4
- Broken link found (https://resources.github.com/devops/)
- [RFE] Check machinset/machine/bmh/nodes matches
- [RFE] Raise an error if Mellanox firmware is an unexpected version
- [RFE] Add a check for bad pod disruption budgets
- Running a single script runs the ssh checks as well
- [RFE] Warn about wrong OVN election timers HOT 1
- Leverage https://github.com/marketplace/actions/actions-k3s for UT
- Option to send all oc debug queries to a specific namespace ( --to-namespace=NAMESPACE ) HOT 1
- [Bug] mellanox-firmware-version doesn't report version properly HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from openshift-checks.