TASK [master : Initialize Master v1.14.1] Fails about rak8s HOT 10 OPEN

I'm having the same issue, but for me it occasionally works (maybe 1 out of 10 tries)

from rak8s.

To force the use of systemd instead of cgroupfs edit /lib/systemd/system/docker.service and edit the following line:

ExecStart=/usr/bin/dockerd -H unix://
To be:
ExecStart=/usr/bin/dockerd -H unix:// --exec-opt native.cgroupdriver=systemd

I'm working on a ansible task to do this. I've got the task to edit the line, and added a task to restart the docker service, but it seems you need to reboot the whole pi for it to take effect.

from rak8s.

Here is the output of jornalctl command:"
pi@k8s-master-1:~/rak8s $ sudo journalctl -xeu kubelet Apr 20 11:34:56 k8s-master-1 kubelet[9470]: I0420 11:34:56.026676 9470 server.go:141] Starting to listen on 0.0.0.0:10250 Apr 20 11:34:56 k8s-master-1 kubelet[9470]: I0420 11:34:56.030917 9470 server.go:343] Adding debug handlers to kubelet server. Apr 20 11:34:56 k8s-master-1 kubelet[9470]: I0420 11:34:56.038128 9470 fs_resource_analyzer.go:64] Starting FS ResourceAnalyzer Apr 20 11:34:56 k8s-master-1 kubelet[9470]: I0420 11:34:56.038314 9470 status_manager.go:152] Starting to sync pod status with apiserver Apr 20 11:34:56 k8s-master-1 kubelet[9470]: I0420 11:34:56.038405 9470 kubelet.go:1806] Starting kubelet main sync loop. Apr 20 11:34:56 k8s-master-1 kubelet[9470]: I0420 11:34:56.038509 9470 kubelet.go:1823] skipping pod synchronization - [container runtime status check may not have completed yet., PLEG Apr 20 11:34:56 k8s-master-1 kubelet[9470]: I0420 11:34:56.039270 9470 volume_manager.go:248] Starting Kubelet Volume Manager Apr 20 11:34:56 k8s-master-1 kubelet[9470]: I0420 11:34:56.039341 9470 desired_state_of_world_populator.go:130] Desired state populator starts to run Apr 20 11:34:56 k8s-master-1 kubelet[9470]: E0420 11:34:56.046610 9470 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1beta1.RuntimeClass: Get https://10 Apr 20 11:34:56 k8s-master-1 kubelet[9470]: E0420 11:34:56.047087 9470 controller.go:115] failed to ensure node lease exists, will retry in 200ms, error: Get https://10.0.3.240:6443/ap Apr 20 11:34:56 k8s-master-1 kubelet[9470]: E0420 11:34:56.057409 9470 kubelet.go:2170] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:doc Apr 20 11:34:56 k8s-master-1 kubelet[9470]: I0420 11:34:56.116530 9470 clientconn.go:440] parsed scheme: "unix" Apr 20 11:34:56 k8s-master-1 kubelet[9470]: I0420 11:34:56.116629 9470 clientconn.go:440] scheme "unix" not registered, fallback to default scheme Apr 20 11:34:56 k8s-master-1 kubelet[9470]: I0420 11:34:56.116875 9470 asm_arm.s:868] ccResolverWrapper: sending new addresses to cc: [{unix:///run/containerd/containerd.sock 0 <nil>} Apr 20 11:34:56 k8s-master-1 kubelet[9470]: I0420 11:34:56.116939 9470 clientconn.go:796] ClientConn switching balancer to "pick_first" Apr 20 11:34:56 k8s-master-1 kubelet[9470]: I0420 11:34:56.117100 9470 balancer_conn_wrappers.go:131] pickfirstBalancer: HandleSubConnStateChange: 0x856d0b0, CONNECTING Apr 20 11:34:56 k8s-master-1 kubelet[9470]: I0420 11:34:56.117926 9470 balancer_conn_wrappers.go:131] pickfirstBalancer: HandleSubConnStateChange: 0x856d0b0, READY Apr 20 11:34:56 k8s-master-1 kubelet[9470]: W0420 11:34:56.130476 9470 nvidia.go:66] Error reading "/sys/bus/pci/devices/": open /sys/bus/pci/devices/: no such file or directory Apr 20 11:34:56 k8s-master-1 kubelet[9470]: I0420 11:34:56.144079 9470 kubelet.go:1823] skipping pod synchronization - container runtime status check may not have completed yet. Apr 20 11:34:56 k8s-master-1 kubelet[9470]: E0420 11:34:56.144134 9470 kubelet.go:2244] node "k8s-master-1" not found Apr 20 11:34:56 k8s-master-1 kubelet[9470]: I0420 11:34:56.147640 9470 kubelet_node_status.go:283] Setting node annotation to enable volume controller attach/detach Apr 20 11:34:56 k8s-master-1 kubelet[9470]: I0420 11:34:56.158240 9470 kubelet_node_status.go:72] Attempting to register node k8s-master-1 Apr 20 11:34:56 k8s-master-1 kubelet[9470]: E0420 11:34:56.159527 9470 kubelet_node_status.go:94] Unable to register node "k8s-master-1" with API server: Post https://10.0.3.240:6443/a Apr 20 11:34:56 k8s-master-1 kubelet[9470]: E0420 11:34:56.245247 9470 kubelet.go:2244] node "k8s-master-1" not found Apr 20 11:34:56 k8s-master-1 kubelet[9470]: E0420 11:34:56.252227 9470 controller.go:115] failed to ensure node lease exists, will retry in 400ms, error: Get https://10.0.3.240:6443/ap Apr 20 11:34:56 k8s-master-1 kubelet[9470]: I0420 11:34:56.344428 9470 kubelet.go:1823] skipping pod synchronization - container runtime status check may not have completed yet. Apr 20 11:34:56 k8s-master-1 kubelet[9470]: E0420 11:34:56.345513 9470 kubelet.go:2244] node "k8s-master-1" not found Apr 20 11:34:56 k8s-master-1 kubelet[9470]: I0420 11:34:56.361270 9470 kubelet_node_status.go:283] Setting node annotation to enable volume controller attach/detach Apr 20 11:34:56 k8s-master-1 kubelet[9470]: I0420 11:34:56.364285 9470 kubelet_node_status.go:283] Setting node annotation to enable volume controller attach/detach Apr 20 11:34:56 k8s-master-1 kubelet[9470]: I0420 11:34:56.376494 9470 kubelet_node_status.go:72] Attempting to register node k8s-master-1 Apr 20 11:34:56 k8s-master-1 kubelet[9470]: I0420 11:34:56.378024 9470 cpu_manager.go:155] [cpumanager] starting with none policy Apr 20 11:34:56 k8s-master-1 kubelet[9470]: I0420 11:34:56.378080 9470 cpu_manager.go:156] [cpumanager] reconciling every 10s Apr 20 11:34:56 k8s-master-1 kubelet[9470]: I0420 11:34:56.378120 9470 policy_none.go:42] [cpumanager] none policy: Start Apr 20 11:34:56 k8s-master-1 kubelet[9470]: E0420 11:34:56.378112 9470 kubelet_node_status.go:94] Unable to register node "k8s-master-1" with API server: Post https://10.0.3.240:6443/a Apr 20 11:34:56 k8s-master-1 kubelet[9470]: F0420 11:34:56.380489 9470 kubelet.go:1359] Failed to start ContainerManager failed to initialize top level QOS containers: failed to update Apr 20 11:34:56 k8s-master-1 systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a Apr 20 11:34:56 k8s-master-1 systemd[1]: kubelet.service: Unit entered failed state. Apr 20 11:34:56 k8s-master-1 systemd[1]: kubelet.service: Failed with result 'exit-code'.

from rak8s.

I can reproduce the issue from the command line, so I think it might be a k8s issue, not rak8s.

from rak8s.

To force the use of systemd instead of cgroupfs edit /lib/systemd/system/docker.service and edit the following line:

ExecStart=/usr/bin/dockerd -H unix://
To be:
ExecStart=/usr/bin/dockerd -H unix:// --exec-opt native.cgroupdriver=systemd

I'm working on a ansible task to do this. I've got the task to edit the line, and added a task to restart the docker service, but it seems you need to reboot the whole pi for it to take effect.

Did as you suggested manually.

Now I got this from the journalctl:

Apr 20 18:14:08 rak8s000 systemd[1]: libcontainer-5100-systemd-test-default-dependencies.scope: Scope has no PIDs. Refusing. Apr 20 18:14:09 rak8s000 systemd[1]: libcontainer-5100-systemd-test-default-dependencies.scope: Scope has no PIDs. Refusing. Apr 20 18:14:09 rak8s000 systemd[1]: Created slice libcontainer_5100_systemd_test_default.slice. Apr 20 18:14:09 rak8s000 systemd[1]: Removed slice libcontainer_5100_systemd_test_default.slice. Apr 20 18:14:09 rak8s000 systemd[1]: libcontainer-5108-systemd-test-default-dependencies.scope: Scope has no PIDs. Refusing. Apr 20 18:14:09 rak8s000 systemd[1]: libcontainer-5108-systemd-test-default-dependencies.scope: Scope has no PIDs. Refusing. Apr 20 18:14:09 rak8s000 systemd[1]: Created slice libcontainer_5108_systemd_test_default.slice. Apr 20 18:14:09 rak8s000 kubelet[3360]: W0420 18:14:09.096478 3360 container.go:523] Failed to update stats for container "/libcontainer_5108_systemd_test_default.slice": failed to parse memory.usage_in_bytes - open /sys/fs/cgroup/memory/libcontainer_5108_systemd_test_default.slice/memory.usage_in_bytes: no such file or directory, continuing to push stats Apr 20 18:14:09 rak8s000 systemd[1]: Removed slice libcontainer_5108_systemd_test_default.slice. Apr 20 18:14:12 rak8s000 kubelet[3360]: W0420 18:14:12.903479 3360 cni.go:213] Unable to update cni config: No networks found in /etc/cni/net.d Apr 20 18:14:13 rak8s000 kubelet[3360]: E0420 18:14:13.635837 3360 qos_container_manager_linux.go:329] [ContainerManager]: Failed to update QoS cgroup configuration Apr 20 18:14:13 rak8s000 kubelet[3360]: W0420 18:14:13.635904 3360 qos_container_manager_linux.go:139] [ContainerManager] Failed to reserve QoS requests: failed to set supported cgroup subsystems for cgroup [kubepods burstable]: Failed to find subsystem mount for required subsystem: pids Apr 20 18:14:13 rak8s000 kubelet[3360]: E0420 18:14:13.843958 3360 kubelet.go:2170] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized Apr 20 18:14:17 rak8s000 kubelet[3360]: W0420 18:14:17.903951 3360 cni.go:213] Unable to update cni config: No networks found in /etc/cni/net.d Apr 20 18:14:18 rak8s000 systemd[1]: libcontainer-5126-systemd-test-default-dependencies.scope: Scope has no PIDs. Refusing. Apr 20 18:14:18 rak8s000 systemd[1]: libcontainer-5126-systemd-test-default-dependencies.scope: Scope has no PIDs. Refusing. Apr 20 18:14:18 rak8s000 systemd[1]: Created slice libcontainer_5126_systemd_test_default.slice. Apr 20 18:14:18 rak8s000 systemd[1]: Removed slice libcontainer_5126_systemd_test_default.slice. Apr 20 18:14:18 rak8s000 systemd[1]: libcontainer-5133-systemd-test-default-dependencies.scope: Scope has no PIDs. Refusing. Apr 20 18:14:18 rak8s000 systemd[1]: libcontainer-5133-systemd-test-default-dependencies.scope: Scope has no PIDs. Refusing. Apr 20 18:14:18 rak8s000 systemd[1]: Created slice libcontainer_5133_systemd_test_default.slice. Apr 20 18:14:18 rak8s000 systemd[1]: Removed slice libcontainer_5133_systemd_test_default.slice. Apr 20 18:14:18 rak8s000 systemd[1]: libcontainer-5151-systemd-test-default-dependencies.scope: Scope has no PIDs. Refusing. Apr 20 18:14:18 rak8s000 systemd[1]: libcontainer-5151-systemd-test-default-dependencies.scope: Scope has no PIDs. Refusing. Apr 20 18:14:18 rak8s000 systemd[1]: Created slice libcontainer_5151_systemd_test_default.slice. Apr 20 18:14:18 rak8s000 systemd[1]: Removed slice libcontainer_5151_systemd_test_default.slice. Apr 20 18:14:18 rak8s000 kubelet[3360]: E0420 18:14:18.847741 3360 kubelet.go:2170] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized Apr 20 18:14:18 rak8s000 systemd[1]: libcontainer-5176-systemd-test-default-dependencies.scope: Scope has no PIDs. Refusing. Apr 20 18:14:18 rak8s000 systemd[1]: libcontainer-5176-systemd-test-default-dependencies.scope: Scope has no PIDs. Refusing. Apr 20 18:14:19 rak8s000 systemd[1]: Created slice libcontainer_5176_systemd_test_default.slice. Apr 20 18:14:19 rak8s000 systemd[1]: Removed slice libcontainer_5176_systemd_test_default.slice. Apr 20 18:14:19 rak8s000 systemd[1]: libcontainer-5183-systemd-test-default-dependencies.scope: Scope has no PIDs. Refusing. Apr 20 18:14:19 rak8s000 systemd[1]: libcontainer-5183-systemd-test-default-dependencies.scope: Scope has no PIDs. Refusing. Apr 20 18:14:19 rak8s000 systemd[1]: Created slice libcontainer_5183_systemd_test_default.slice. Apr 20 18:14:19 rak8s000 systemd[1]: Removed slice libcontainer_5183_systemd_test_default.slice. Apr 20 18:14:19 rak8s000 kubelet[3360]: W0420 18:14:19.085643 3360 raw.go:87] Error while processing event ("/sys/fs/cgroup/devices/libcontainer_5183_systemd_test_default.slice": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/devices/libcontainer_5183_systemd_test_default.slice: no such file or directory Apr 20 18:14:19 rak8s000 kubelet[3360]: W0420 18:14:19.087468 3360 container.go:409] Failed to create summary reader for "/libcontainer_5183_systemd_test_default.slice": none of the resources are being tracked. Apr 20 18:14:22 rak8s000 kubelet[3360]: W0420 18:14:22.904522 3360 cni.go:213] Unable to update cni config: No networks found in /etc/cni/net.d Apr 20 18:14:23 rak8s000 kubelet[3360]: E0420 18:14:23.854038 3360 kubelet.go:2170] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized Apr 20 18:14:27 rak8s000 kubelet[3360]: W0420 18:14:27.904971 3360 cni.go:213] Unable to update cni config: No networks found in /etc/cni/net.d Apr 20 18:14:27 rak8s000 systemd[1]: Starting Cleanup of Temporary Directories... Apr 20 18:14:27 rak8s000 systemd[1]: Started Cleanup of Temporary Directories. Apr 20 18:14:28 rak8s000 systemd[1]: libcontainer-5211-systemd-test-default-dependencies.scope: Scope has no PIDs. Refusing. Apr 20 18:14:28 rak8s000 systemd[1]: libcontainer-5211-systemd-test-default-dependencies.scope: Scope has no PIDs. Refusing. Apr 20 18:14:28 rak8s000 systemd[1]: Created slice libcontainer_5211_systemd_test_default.slice. Apr 20 18:14:28 rak8s000 systemd[1]: Removed slice libcontainer_5211_systemd_test_default.slice. Apr 20 18:14:28 rak8s000 systemd[1]: libcontainer-5219-systemd-test-default-dependencies.scope: Scope has no PIDs. Refusing. Apr 20 18:14:28 rak8s000 systemd[1]: libcontainer-5219-systemd-test-default-dependencies.scope: Scope has no PIDs. Refusing. Apr 20 18:14:28 rak8s000 systemd[1]: Created slice libcontainer_5219_systemd_test_default.slice. Apr 20 18:14:28 rak8s000 systemd[1]: Removed slice libcontainer_5219_systemd_test_default.slice. Apr 20 18:14:28 rak8s000 systemd[1]: libcontainer-5236-systemd-test-default-dependencies.scope: Scope has no PIDs. Refusing. Apr 20 18:14:28 rak8s000 systemd[1]: libcontainer-5236-systemd-test-default-dependencies.scope: Scope has no PIDs. Refusing. Apr 20 18:14:28 rak8s000 systemd[1]: Created slice libcontainer_5236_systemd_test_default.slice. Apr 20 18:14:28 rak8s000 systemd[1]: Removed slice libcontainer_5236_systemd_test_default.slice. Apr 20 18:14:28 rak8s000 kubelet[3360]: E0420 18:14:28.857418 3360 kubelet.go:2170] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized Apr 20 18:14:29 rak8s000 systemd[1]: libcontainer-5260-systemd-test-default-dependencies.scope: Scope has no PIDs. Refusing. Apr 20 18:14:29 rak8s000 systemd[1]: libcontainer-5260-systemd-test-default-dependencies.scope: Scope has no PIDs. Refusing. Apr 20 18:14:29 rak8s000 systemd[1]: Created slice libcontainer_5260_systemd_test_default.slice. Apr 20 18:14:29 rak8s000 systemd[1]: Removed slice libcontainer_5260_systemd_test_default.slice. Apr 20 18:14:29 rak8s000 systemd[1]: libcontainer-5267-systemd-test-default-dependencies.scope: Scope has no PIDs. Refusing. Apr 20 18:14:29 rak8s000 systemd[1]: libcontainer-5267-systemd-test-default-dependencies.scope: Scope has no PIDs. Refusing. Apr 20 18:14:29 rak8s000 systemd[1]: Created slice libcontainer_5267_systemd_test_default.slice. Apr 20 18:14:29 rak8s000 systemd[1]: Removed slice libcontainer_5267_systemd_test_default.slice.

from rak8s.

I wiped and completely reinstalled Raspian-Lite on my k8s master. When I run the playbook I got:

TASK [kubeadm : Install k8s 1.14.1-00 Y'all] *********************************** ok: [k8s-node-2] => (item=[u'kubelet=1.14.1-00', u'kubeadm=1.14.1-00', u'kubectl=1.14.1-00']) ok: [k8s-node-4] => (item=[u'kubelet=1.14.1-00', u'kubeadm=1.14.1-00', u'kubectl=1.14.1-00']) ok: [k8s-node-3] => (item=[u'kubelet=1.14.1-00', u'kubeadm=1.14.1-00', u'kubectl=1.14.1-00']) ok: [k8s-node-1] => (item=[u'kubelet=1.14.1-00', u'kubeadm=1.14.1-00', u'kubectl=1.14.1-00']) ok: [k8s-node-5] => (item=[u'kubelet=1.14.1-00', u'kubeadm=1.14.1-00', u'kubectl=1.14.1-00']) ok: [k8s-node-6] => (item=[u'kubelet=1.14.1-00', u'kubeadm=1.14.1-00', u'kubectl=1.14.1-00']) failed: [k8s-master-1] (item=[u'kubelet=1.14.1-00', u'kubeadm=1.14.1-00', u'kubectl=1.14.1-00']) => {"failed": true, "item": ["kubelet=1.14.1-00", "kubeadm=1.14.1-00", "kubectl=1.14.1-00"], "msg": "No package matching 'kubelet' is available"}

It appears that rak8s is not installing dependencies (or removing them when running the cleanup playbook).

So I installed the kubeadm manually with apt-get which installed all the dependencies.

Rerunning the playbook got past the Install master part but is hanging on joining the workers to the cluster.

from rak8s.

My guess is that it is a k8s issue because I get the same error running the kubeadm join command by hand.

At this point I'm going to take a break and then reinstall Raspbian on all my nodes and just follow the manual instructions at here

Good Luck!

from rak8s.

Hit this issue this evening -- still debugging. In the meantime, I just submitted a PR that fixes the "No package matching 'kubelet' is available" issue for me.

from rak8s.

I merged your changes in @PostlMC. Please test on clean installs if you can.

from rak8s.

Any updates here?

from rak8s.