Comments (6)
looks like maybe some more selinux gremlins?
Mar 01 11:16:08 dev-nkjpp-master-1 audit[1063]: AVC avc: denied { read } for pid=1063 comm="nm-dispatcher" name="dispatcher.d" dev="sda4" ino=4207358 scontext=system_u:system_r:NetworkManager_dispatcher_t:s0 tcontext=system_u:object_r:NetworkManager_initrc_exec_t:s0 tclass=dir permissive=0
Mar 01 11:16:08 dev-nkjpp-master-1 audit[1063]: SYSCALL arch=c000003e syscall=257 success=no exit=-13 a0=ffffff9c a1=55fbd8e47ae0 a2=90800 a3=0 items=0 ppid=1 pid=1063 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="nm-dispatcher" exe="/usr/libexec/nm-dispatcher" subj=system_u:system_r:NetworkManager_dispatcher_t:s0 key=(null)
Mar 01 11:16:08 dev-nkjpp-master-1 audit: PROCTITLE proctitle="/usr/libexec/nm-dispatcher"
Mar 01 11:16:08 dev-nkjpp-master-1 nm-dispatcher[1063]: req:22 'connectivity-change': find-scripts: Failed to open dispatcher directory '/etc/NetworkManager/dispatcher.d': Error opening directory “/etc/NetworkManager/dispatcher.d”: Permission denied
Applying the workaround from #1425
[root@dev-nkjpp-master-1 ~]# restorecon -R -v /etc/NetworkManager/dispatcher.d/
Relabeled /etc/NetworkManager/dispatcher.d from system_u:object_r:NetworkManager_initrc_exec_t:s0 to system_u:object_r:NetworkManager_dispatcher_script_t:s0
Relabeled /etc/NetworkManager/dispatcher.d/pre-up.d from system_u:object_r:NetworkManager_initrc_exec_t:s0 to system_u:object_r:NetworkManager_dispatcher_script_t:s0
Relabeled /etc/NetworkManager/dispatcher.d/pre-up.d/10-ofport-request.sh from system_u:object_r:NetworkManager_initrc_exec_t:s0 to system_u:object_r:NetworkManager_dispatcher_script_t:s0
Relabeled /etc/NetworkManager/dispatcher.d/30-resolv-prepender from system_u:object_r:NetworkManager_initrc_exec_t:s0 to system_u:object_r:NetworkManager_dispatcher_script_t:s0
Relabeled /etc/NetworkManager/dispatcher.d/99-vsphere-disable-tx-udp-tnl from system_u:object_r:NetworkManager_initrc_exec_t:s0 to system_u:object_r:NetworkManager_dispatcher_script_t:s0
from okd.
Looks like a dupe of #1475
from okd.
Yeah, i think you may be right. I swear tried this yesterday. Will open another issue if I encounter another problem.
Apologies for the oversight on my part and thank you very much for the eyes and brains @vrutkovs and @melledouwsma.
from okd.
Kubelet seems to be unavailable on both nodes, so the must-gather does not contain much logs from the nodes. Does sudo systemctl status kubelet
report anything useful from the nodes?
$ omg get nodes os-infra-dev-01-zh59r -o json | jq .status.conditions
[
{
"lastHeartbeatTime": "2023-02-28T11:39:56Z",
"lastTransitionTime": "2023-02-28T11:40:37Z",
"message": "Kubelet stopped posting node status.",
"reason": "NodeStatusUnknown",
"status": "Unknown",
"type": "MemoryPressure"
},
{
"lastHeartbeatTime": "2023-02-28T11:39:56Z",
"lastTransitionTime": "2023-02-28T11:40:37Z",
"message": "Kubelet stopped posting node status.",
"reason": "NodeStatusUnknown",
"status": "Unknown",
"type": "DiskPressure"
},
{
"lastHeartbeatTime": "2023-02-28T11:39:56Z",
"lastTransitionTime": "2023-02-28T11:40:37Z",
"message": "Kubelet stopped posting node status.",
"reason": "NodeStatusUnknown",
"status": "Unknown",
"type": "PIDPressure"
},
{
"lastHeartbeatTime": "2023-02-28T11:39:56Z",
"lastTransitionTime": "2023-02-28T11:40:37Z",
"message": "Kubelet stopped posting node status.",
"reason": "NodeStatusUnknown",
"status": "Unknown",
"type": "Ready"
}
]
$ omg get nodes dev-nkjpp-master-1 -o json | jq .status.conditions
[
{
"lastHeartbeatTime": "2023-02-28T11:40:32Z",
"lastTransitionTime": "2023-02-28T11:42:12Z",
"message": "Kubelet stopped posting node status.",
"reason": "NodeStatusUnknown",
"status": "Unknown",
"type": "MemoryPressure"
},
{
"lastHeartbeatTime": "2023-02-28T11:40:32Z",
"lastTransitionTime": "2023-02-28T11:42:12Z",
"message": "Kubelet stopped posting node status.",
"reason": "NodeStatusUnknown",
"status": "Unknown",
"type": "DiskPressure"
},
{
"lastHeartbeatTime": "2023-02-28T11:40:32Z",
"lastTransitionTime": "2023-02-28T11:42:12Z",
"message": "Kubelet stopped posting node status.",
"reason": "NodeStatusUnknown",
"status": "Unknown",
"type": "PIDPressure"
},
{
"lastHeartbeatTime": "2023-02-28T11:40:32Z",
"lastTransitionTime": "2023-02-28T11:42:12Z",
"message": "Kubelet stopped posting node status.",
"reason": "NodeStatusUnknown",
"status": "Unknown",
"type": "Ready"
}
]
from okd.
thanks for taking a look @melledouwsma
Yeah, the kublet isn't running due to the absence of /run/resolv-prepender-kni-conf-done
which seems to be managed by
/etc/NetworkManager/dispatcher.d/30-resolv-prepender
which I am endeavoring to understand this morning to try and understand better where the root of this issue is.
[core@dev-nkjpp-master-1 ~]$ systemctl status kubelet
● kubelet.service - Kubernetes Kubelet
Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; preset: disabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─01-kubens.conf, 10-mco-default-madv.conf, 10-mco-on-prem-wait-resolv.conf, 20-logging.conf, 20-nodenet.conf
Active: activating (auto-restart) (Result: exit-code) since Wed 2023-03-01 11:30:40 UTC; 4s ago
Process: 2578 ExecCondition=/bin/bash -c test -f /run/resolv-prepender-kni-conf-done || exit 255 (code=exited, status=255/EXCEPTION)
CPU: 2ms
[core@dev-nkjpp-master-1 ~]$ stat /run/resolv-prepender-kni-conf-done
stat: cannot statx '/run/resolv-prepender-kni-conf-done': No such file or directory
[core@dev-nkjpp-master-1 ~]$ systemctl status NetworkManager
● NetworkManager.service - Network Manager
Loaded: loaded (/usr/lib/systemd/system/NetworkManager.service; enabled; preset: enabled)
Drop-In: /usr/lib/systemd/system/NetworkManager.service.d
└─NetworkManager-ovs.conf
Active: active (running) since Wed 2023-03-01 11:16:05 UTC; 16min ago
Docs: man:NetworkManager(8)
Main PID: 1052 (NetworkManager)
Tasks: 3 (limit: 38420)
Memory: 8.5M
CPU: 519ms
CGroup: /system.slice/NetworkManager.service
└─1052 /usr/sbin/NetworkManager --no-daemon
from okd.
Check logs on the node for "nm-dispatcher" - this would have logs from 30-resolv-prepender
from okd.
Related Issues (20)
- Release streams failing for 4.10, 4.11 and 4.12 HOT 1
- Bug: failed to sync stale SNATs on node (AWS IPI - OKD 4.13) HOT 5
- Control Plane Node Memory Spikes HOT 7
- 4.6 -> 4.7 upgrade
- [OKD 4.12] Bootstrap failed
- OKD 4.10 automatic update stuck - MCP in degraded state HOT 1
- 4.13.0-0.okd-2023-07-23-051208 install bootstrap problem
- kubelet stopped posting node status
- Unable to start OKD upgrade, the clusterVersionOperator job pod in initcrashloopbackoff
- OKD 4.13.0 IPI build errors
- Outdated pod-identity-webhook version includes a bug that disallows injection of AWS_STS_REGIONAL_ENDPOINTS=regional
- ImageTagMirrorSet - Should they work with Builds?
- Mount a storageclass with a minio external storage server
- [Libvrit] Mirroring of images with imageContentSourcePolicy not working ?
- CI config for latest 4-stable OKD release picked wrong base release HOT 4
- prometheus-k8s route is returning 404 after OKD cluster upgrade to 4.13
- Connot pull from OKD cluster via pod or oc command with error x509: certificate is valid for ***Clusterdomain *** , not *** registry-1.docker.io ***
- Disconnected 4.12 install fails pivot due to missing podman credentials HOT 1
- [OKD 4.13][vSphere 4.13] IPI Install broken. Bootstrap doesn't clean up due to https://github.com/openshift/installer/pull/6770
- [AWS] Cluster bootstrap fails with IngressStateEndpoints_MissingSubsets + operators not initialized HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from okd.