k3s is unable to start after stopped for a while about k3s HOT 4 CLOSED

You didn't mention that you're running longhorn on these nodes, but that appears to be the case? The apiserver isn't coming ready, probably because it can't talk to longhorn to complete aggregated api discovery. Pods appear to be crashlooping because the LH CSI isn't registered:

E0530 12:44:49.589154 32845 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/driver.longhorn.io^pvc-29339ee5-93ea-4b92-b811-1af27fbf8f8c podName:116fd16b-b81c-4750-a0be-682ed21c14f6 nodeName:}" failed. No retries permitted until 2024-05-30 12:44:50.589051758 -0400 EDT m=+12.882424672 (durationBeforeRetry 1s). Error: UnmountVolume.TearDown failed for volume "pvc-29339ee5-93ea-4b92-b811-1af27fbf8f8c" (UniqueName: "kubernetes.io/csi/driver.longhorn.io^pvc-29339ee5-93ea-4b92-b811-1af27fbf8f8c") pod "116fd16b-b81c-4750-a0be-682ed21c14f6" (UID: "116fd16b-b81c-4750-a0be-682ed21c14f6") : kubernetes.io/csi: Unmounter.TearDownAt failed to get CSI client: driver name driver.longhorn.io not found in the list of registered CSI drivers

Maybe check the pod logs on the server node to see what's wrong with the LH CSI deployment? You need to get LH fixed on the server node so that it can finish api discovery and become ready for agents to join.

from k3s.

@brandond Yes, I have longhorn installed, but the longhorn-csi-plugin pod is not starting correctly. It cannot connect to longhorn-backend. I can't seem to restart the pod. Is there anyway to skip api discovery with longhorn?

Also, it seems pods in kube-system namespace is also not starting correctly. How can I get the pod as part of deployment to start? The deployment and replicasets seems fine but related pods are not starting

from k3s.

Have you tried force-deleting the pods on the agents so that they get rescheduled to the server node? I think that the LH pods will need to come up before other things will work properly. You might create an issue in the LH repo to ask if there are any specific cold-start procedures that you need to observe.

from k3s.

This repository uses a bot to automatically label issues which have not had any activity (commit/comment/label) for 45 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the bot can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the bot will automatically close the issue in 14 days. Thank you for your contributions.

from k3s.