Docker in docker setup constantly logs error about k3s HOT 13 CLOSED

Looks to me like the kube proxy crashing immediately/ not running
There is not log message about the „startup cry“ so at least kube proxy does not finish startup imho.

from k3s.

Hi, thanks for this great job
this bug still present in v0.6.0-rc3

from k3s.

This issue has been around for a while now. Today I tried a clean install and used the latest versions

I used the latest file from:
https://github.com/rancher/k3s/blob/master/docker-compose.yml

I also upgraded the image to image: rancher/k3s:v0.6.1
I also tried removing --disable-agent adding the tmpfs and privileged as suggested elsewhere

I ended up with:

version: '3'
services:
  server:
    image: rancher/k3s:v0.6.1
    tmpfs:
    - /run
    - /var/run
    privileged: true
    command: server
    environment:
    - K3S_CLUSTER_SECRET=somethingtotallyrandom
    - K3S_KUBECONFIG_OUTPUT=/output/kubeconfig.yaml
    - K3S_KUBECONFIG_MODE=666
    volumes:
    - k3s-server:/var/lib/rancher/k3s
    - .:/output
    ports:
    - 6443:6443

  node:
    image: rancher/k3s:v0.6.1
    tmpfs:
    - /run
    - /var/run
    privileged: true
    depends_on:
    - server
    environment:
    - K3S_URL=https://pi99:6443
    - K3S_CLUSTER_SECRET=somethingtotallyrandom

volumes:
  k3s-server: {}

In all cases I continue to get:

server_1  | E0621 11:28:49.435389       1 summary_sys_containers.go:47] Failed to get system container stats for "/docker/7e97344fc37ce943d4558c09426b022113cff8b1ae21f54900c31c6ab6ee6782": failed to get cgroup stats for "/docker/7e97344fc37ce943d4558c09426b022113cff8b1ae21f54900c31c6ab6ee6782": failed to get container info for "/docker/7e97344fc37ce943d4558c09426b022113cff8b1ae21f54900c31c6ab6ee6782": unknown container "/docker/7e97344fc37ce943d4558c09426b022113cff8b1ae21f54900c31c6ab6ee6782"
node_1    | E0621 11:28:50.675223       1 summary_sys_containers.go:47] Failed to get system container stats for "/docker/4ecd9d95b647f3023259911617eb9dd21d3279c5118950faaf8d0dd41bfd62c3": failed to get cgroup stats for "/docker/4ecd9d95b647f3023259911617eb9dd21d3279c5118950faaf8d0dd41bfd62c3": failed to get container info for "/docker/4ecd9d95b647f3023259911617eb9dd21d3279c5118950faaf8d0dd41bfd62c3": unknown container "/docker/4ecd9d95b647f3023259911617eb9dd21d3279c5118950faaf8d0dd41bfd62c3"

from k3s.

same result for me, just tried master on debian under docker 18.09.7 continually scrolling message
node_3 | E0707 12:01:56.602576 1 summary_sys_containers.go:47] Failed to get system container stats for "/docker/c88310cdb7eb512fe54f3dd1476e546eb10d1d024c604046eaa689ba4553071a/kube-proxy": failed to get cgroup stats for "/docker/c88310cdb7eb512fe54f3dd1476e546eb10d1d024c604046eaa689ba4553071a/kube-proxy": failed to get container info for "/docker/c88310cdb7eb512fe54f3dd1476e546eb10d1d024c604046eaa689ba4553071a/kube-proxy": unknown container "/docker/c88310cdb7eb512fe54f3dd1476e546eb10d1d024c604046eaa689ba4553071a/kube-proxy"
node_2 | E0707 12:01:56.643769 1 summary_sys_containers.go:47] Failed to get system container stats for "/docker/25fc1eb7b7e541a81cbc52c27910acb3ef1129dcd9050b35960f298935a5547c/kube-proxy": failed to get cgroup stats for "/docker/25fc1eb7b7e541a81cbc52c27910acb3ef1129dcd9050b35960f298935a5547c/kube-proxy": failed to get container info for "/docker/25fc1eb7b7e541a81cbc52c27910acb3ef1129dcd9050b35960f298935a5547c/kube-proxy": unknown container "/docker/25fc1eb7b7e541a81cbc52c27910acb3ef1129dcd9050b35960f298935a5547c/kube-proxy"
node_1 | E0707 12:01:56.751434 1 summary_sys_containers.go:47] Failed to get system container stats for "/docker/bdd26abd71c35127f48368a8556f3d5d1a461115bd4a8469ab2a877da73dc30d/kube-proxy": failed to get cgroup stats for "/docker/bdd26abd71c35127f48368a8556f3d5d1a461115bd4a8469ab2a877da73dc30d/kube-proxy": failed to get container info for "/docker/bdd26abd71c35127f48368a8556f3d5d1a461115bd4a8469ab2a877da73dc30d/kube-proxy": unknown container "/docker/bdd26abd71c35127f48368a8556f3d5d1a461115bd4a8469ab2a877da73dc30d/kube-proxy"

from k3s.

I've been trying to run k3s in https://labs.play-with-docker.com, which creates 'nodes' with d-in-d and can confirm the issue to loop on the agent nodes.

from k3s.

@tfiduccia can you test with v0.4.0 cause @ibuildthecloud says it might already be fixed

from k3s.

I can still reproduce the problem with 0.4.0

from k3s.

Version - master 4/18
This is not fixed.
node_1 | E0418 19:44:42.171004 1 summary_sys_containers.go:47] Failed to get system container stats for "/docker/7be6fa3ddd65c1059ee34a48321d63899aadfae3922ccd24b129d8972faa5655/kube-proxy": failed to get cgroup stats for "/docker/7be6fa3ddd65c1059ee34a48321d63899aadfae3922ccd24b129d8972faa5655/kube-proxy": failed to get container info for "/docker/7be6fa3ddd65c1059ee34a48321d63899aadfae3922ccd24b129d8972faa5655/kube-proxy": unknown container "/docker/7be6fa3ddd65c1059ee34a48321d63899aadfae3922ccd24b129d8972faa5655/kube-proxy"

from k3s.

@ibuildthecloud Just in case you haven't seen this yet #87 (comment) Maybe a helpful finding to get this solved.

from k3s.

Some further research.

I found this blog post that had said it was working
https://www.mjpitz.com/blog/2019/04/10/k8s-k3s-rpi-oh-my/

• notable that it was an arm64/ubuntu and with v0.3.0 so I tried various combinations

The issue occurs with version 0.3.0 onward, and it occurs with armhf or arm64

It will run in docker if it is run serperately as follows

docker volume create k3s-server-vol

docker run -d \
  --name "k3s-server-ctr" \
  --network "host" \
  --restart "always" \
  -e "K3S_CLUSTER_SECRET=${SECRET}" \
  -v "k3s-server-vol:/var/lib/rancher/k3s" \
  -p "${PORT}:${PORT}" \
  rancher/k3s:${RELEASE} \
  server --disable-agent

docker run -d \
  --name "k3s-agent-ctr" \
  --network "host" \
  --restart "always" \
  -e "K3S_CLUSTER_SECRET=${SECRET}" \
  -e "K3S_URL=https://${MASTER}:${PORT}" \
  --privileged \
  --tmpfs /run \
  --tmpfs /var/run \
  rancher/k3s:${RELEASE}

In conclusion it appears to only be a problem if it is run from the docker-compose.yml but I can't find an obvious mistake in the docker-compose.yml

from k3s.

I'm also getting this on v0.7.0-rc6 (via k3d 1.3.0-dev.0)

E0723 01:54:57.888871       1 summary_sys_containers.go:47] Failed to get system container stats for "/docker/0cb3ec542966f91f0d4f6dd810062ba031dbb76bdc87265e0ba07fdc542be8f2/kube-proxy": failed to get cgroup stats for "/docker/0cb3ec542966f91f0d4f6dd810062ba031dbb76bdc87265e0ba07fdc542be8f2/kube-proxy": failed to get container info for "/docker/0cb3ec542966f91f0d4f6dd810062ba031dbb76bdc87265e0ba07fdc542be8f2/kube-proxy": unknown container "/docker/0cb3ec542966f91f0d4f6dd810062ba031dbb76bdc87265e0ba07fdc542be8f2/kube-proxy"

from k3s.

How's it going?

from k3s.

Fixed with #705

from k3s.