Git Product home page Git Product logo

Comments (6)

nate-duke avatar nate-duke commented on July 17, 2024 1

looks like maybe some more selinux gremlins?

Mar 01 11:16:08 dev-nkjpp-master-1 audit[1063]: AVC avc:  denied  { read } for  pid=1063 comm="nm-dispatcher" name="dispatcher.d" dev="sda4" ino=4207358 scontext=system_u:system_r:NetworkManager_dispatcher_t:s0 tcontext=system_u:object_r:NetworkManager_initrc_exec_t:s0 tclass=dir permissive=0
Mar 01 11:16:08 dev-nkjpp-master-1 audit[1063]: SYSCALL arch=c000003e syscall=257 success=no exit=-13 a0=ffffff9c a1=55fbd8e47ae0 a2=90800 a3=0 items=0 ppid=1 pid=1063 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="nm-dispatcher" exe="/usr/libexec/nm-dispatcher" subj=system_u:system_r:NetworkManager_dispatcher_t:s0 key=(null)
Mar 01 11:16:08 dev-nkjpp-master-1 audit: PROCTITLE proctitle="/usr/libexec/nm-dispatcher"
Mar 01 11:16:08 dev-nkjpp-master-1 nm-dispatcher[1063]: req:22 'connectivity-change': find-scripts: Failed to open dispatcher directory '/etc/NetworkManager/dispatcher.d': Error opening directory “/etc/NetworkManager/dispatcher.d”: Permission denied

Applying the workaround from #1425

[root@dev-nkjpp-master-1 ~]# restorecon -R -v /etc/NetworkManager/dispatcher.d/
Relabeled /etc/NetworkManager/dispatcher.d from system_u:object_r:NetworkManager_initrc_exec_t:s0 to system_u:object_r:NetworkManager_dispatcher_script_t:s0
Relabeled /etc/NetworkManager/dispatcher.d/pre-up.d from system_u:object_r:NetworkManager_initrc_exec_t:s0 to system_u:object_r:NetworkManager_dispatcher_script_t:s0
Relabeled /etc/NetworkManager/dispatcher.d/pre-up.d/10-ofport-request.sh from system_u:object_r:NetworkManager_initrc_exec_t:s0 to system_u:object_r:NetworkManager_dispatcher_script_t:s0
Relabeled /etc/NetworkManager/dispatcher.d/30-resolv-prepender from system_u:object_r:NetworkManager_initrc_exec_t:s0 to system_u:object_r:NetworkManager_dispatcher_script_t:s0
Relabeled /etc/NetworkManager/dispatcher.d/99-vsphere-disable-tx-udp-tnl from system_u:object_r:NetworkManager_initrc_exec_t:s0 to system_u:object_r:NetworkManager_dispatcher_script_t:s0

from okd.

vrutkovs avatar vrutkovs commented on July 17, 2024 1

Looks like a dupe of #1475

from okd.

nate-duke avatar nate-duke commented on July 17, 2024 1

Yeah, i think you may be right. I swear tried this yesterday. Will open another issue if I encounter another problem.

Apologies for the oversight on my part and thank you very much for the eyes and brains @vrutkovs and @melledouwsma.

from okd.

melledouwsma avatar melledouwsma commented on July 17, 2024

Kubelet seems to be unavailable on both nodes, so the must-gather does not contain much logs from the nodes. Does sudo systemctl status kubelet report anything useful from the nodes?

$ omg get nodes os-infra-dev-01-zh59r -o json | jq .status.conditions
[
  {
    "lastHeartbeatTime": "2023-02-28T11:39:56Z",
    "lastTransitionTime": "2023-02-28T11:40:37Z",
    "message": "Kubelet stopped posting node status.",
    "reason": "NodeStatusUnknown",
    "status": "Unknown",
    "type": "MemoryPressure"
  },
  {
    "lastHeartbeatTime": "2023-02-28T11:39:56Z",
    "lastTransitionTime": "2023-02-28T11:40:37Z",
    "message": "Kubelet stopped posting node status.",
    "reason": "NodeStatusUnknown",
    "status": "Unknown",
    "type": "DiskPressure"
  },
  {
    "lastHeartbeatTime": "2023-02-28T11:39:56Z",
    "lastTransitionTime": "2023-02-28T11:40:37Z",
    "message": "Kubelet stopped posting node status.",
    "reason": "NodeStatusUnknown",
    "status": "Unknown",
    "type": "PIDPressure"
  },
  {
    "lastHeartbeatTime": "2023-02-28T11:39:56Z",
    "lastTransitionTime": "2023-02-28T11:40:37Z",
    "message": "Kubelet stopped posting node status.",
    "reason": "NodeStatusUnknown",
    "status": "Unknown",
    "type": "Ready"
  }
]
$ omg get nodes dev-nkjpp-master-1 -o json | jq .status.conditions
[
  {
    "lastHeartbeatTime": "2023-02-28T11:40:32Z",
    "lastTransitionTime": "2023-02-28T11:42:12Z",
    "message": "Kubelet stopped posting node status.",
    "reason": "NodeStatusUnknown",
    "status": "Unknown",
    "type": "MemoryPressure"
  },
  {
    "lastHeartbeatTime": "2023-02-28T11:40:32Z",
    "lastTransitionTime": "2023-02-28T11:42:12Z",
    "message": "Kubelet stopped posting node status.",
    "reason": "NodeStatusUnknown",
    "status": "Unknown",
    "type": "DiskPressure"
  },
  {
    "lastHeartbeatTime": "2023-02-28T11:40:32Z",
    "lastTransitionTime": "2023-02-28T11:42:12Z",
    "message": "Kubelet stopped posting node status.",
    "reason": "NodeStatusUnknown",
    "status": "Unknown",
    "type": "PIDPressure"
  },
  {
    "lastHeartbeatTime": "2023-02-28T11:40:32Z",
    "lastTransitionTime": "2023-02-28T11:42:12Z",
    "message": "Kubelet stopped posting node status.",
    "reason": "NodeStatusUnknown",
    "status": "Unknown",
    "type": "Ready"
  }
]

from okd.

nate-duke avatar nate-duke commented on July 17, 2024

thanks for taking a look @melledouwsma

Yeah, the kublet isn't running due to the absence of /run/resolv-prepender-kni-conf-done which seems to be managed by
/etc/NetworkManager/dispatcher.d/30-resolv-prepender which I am endeavoring to understand this morning to try and understand better where the root of this issue is.

[core@dev-nkjpp-master-1 ~]$ systemctl status kubelet
● kubelet.service - Kubernetes Kubelet
     Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; preset: disabled)
    Drop-In: /etc/systemd/system/kubelet.service.d
             └─01-kubens.conf, 10-mco-default-madv.conf, 10-mco-on-prem-wait-resolv.conf, 20-logging.conf, 20-nodenet.conf
     Active: activating (auto-restart) (Result: exit-code) since Wed 2023-03-01 11:30:40 UTC; 4s ago
    Process: 2578 ExecCondition=/bin/bash -c test -f /run/resolv-prepender-kni-conf-done || exit 255 (code=exited, status=255/EXCEPTION)
        CPU: 2ms
        
[core@dev-nkjpp-master-1 ~]$ stat /run/resolv-prepender-kni-conf-done
stat: cannot statx '/run/resolv-prepender-kni-conf-done': No such file or directory

[core@dev-nkjpp-master-1 ~]$ systemctl status NetworkManager
● NetworkManager.service - Network Manager
     Loaded: loaded (/usr/lib/systemd/system/NetworkManager.service; enabled; preset: enabled)
    Drop-In: /usr/lib/systemd/system/NetworkManager.service.d
             └─NetworkManager-ovs.conf
     Active: active (running) since Wed 2023-03-01 11:16:05 UTC; 16min ago
       Docs: man:NetworkManager(8)
   Main PID: 1052 (NetworkManager)
      Tasks: 3 (limit: 38420)
     Memory: 8.5M
        CPU: 519ms
     CGroup: /system.slice/NetworkManager.service
             └─1052 /usr/sbin/NetworkManager --no-daemon

from okd.

vrutkovs avatar vrutkovs commented on July 17, 2024

Check logs on the node for "nm-dispatcher" - this would have logs from 30-resolv-prepender

from okd.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.