Description
Tried to run kubernetes-in-docker(kind) image under OS version 4.9.137. the kind image can be started-up with "run --privileged". But once I tried to start some other images like etcd and apiserver inside the kind container. It gave me error like:"
level=error msg="stream copy error: read /proc/self/fd/23: file already closed"
Steps to reproduce the issue:
ctr -n moby run --privileged --net-host --rm -d docker.io/kindest/node node /sbin/init
ctr -n moby t exec -t node /bin/bash
docker run -it k8s.gcr.io/etcd:3.2.24 /bin/sh
Describe the results you received:
docker: Error response from daemon: cgroups: cgroup deleted: unknown.
Describe the results you expected:
container should run
Additional information you deem important (e.g. issue happens only occasionally):
journalctl -xeu docker gave me the whole error is:
75e0e07df8f46962ba6 to 4452b1364faad4e687ef6f6a2cdae1ca16b6572315647cca56f9bcefb50a72c5, size: 138521856"
May 18 16:18:20 localhost.localdomain dockerd[295]: time="2019-05-18T16:18:20.496232865Z" level=debug msg="Calling GET /_ping"
May 18 16:18:20 localhost.localdomain dockerd[295]: time="2019-05-18T16:18:20.497650450Z" level=debug msg="Calling GET /v1.38/images/json"
May 18 16:19:10 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:10.852319228Z" level=debug msg="Calling GET /_ping"
May 18 16:19:10 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:10.853942550Z" level=debug msg="Calling POST /v1.38/containers/create"
May 18 16:19:10 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:10.854340730Z" level=debug msg="form data: {"AttachStderr":false,"AttachStdin":false,"AttachStdout":false,"Cmd":["sleep","40"],"Domainname":"","Entrypoint":null,"Env":[],"HostConfig":{"AutoRemove":false,"Binds":null,"BlkioDeviceReadBps":null,"BlkioDeviceReadIOps":null,"BlkioDeviceWriteBps":null,"BlkioDeviceWriteIOps":null,"BlkioWeight":0,"BlkioWeightDevice":[],"CapAdd":null,"CapDrop":null,"Cgroup":"","CgroupParent":"","ConsoleSize":[0,0],"ContainerIDFile":"","CpuCount":0,"CpuPercent":0,"CpuPeriod":0,"CpuQuota":0,"CpuRealtimePeriod":0,"CpuRealtimeRuntime":0,"CpuShares":0,"CpusetCpus":"","CpusetMems":"","DeviceCgroupRules":null,"Devices":[],"DiskQuota":0,"Dns":[],"DnsOptions":[],"DnsSearch":[],"ExtraHosts":null,"GroupAdd":null,"IOMaximumBandwidth":0,"IOMaximumIOps":0,"IpcMode":"","Isolation":"","KernelMemory":0,"Links":null,"LogConfig":{"Config":{},"Type":""},"MaskedPaths":null,"Memory":0,"MemoryReservation":0,"MemorySwap":0,"MemorySwappiness":-1,"NanoCpus":0,"NetworkMode":"default","OomKillDisable":false,"OomScoreAdj":0,"PidMode":"","PidsLimit":0,"PortBindings":{},"Privileged":false,"PublishAllPorts":false,"ReadonlyPaths":null,"ReadonlyRootfs":false,"RestartPolicy":{"MaximumRetryCount":0,"Name":"no"},"SecurityOpt":null,"ShmSize":0,"UTSMode":"","Ulimits":null,"UsernsMode":"","VolumeDriver":"","VolumesFrom":null},"Hostname":"","Image":"k8s.gcr.io/etcd:3.2.24","Labels":{},"NetworkingConfig":{"EndpointsConfig":{}},"OnBuild":null,"OpenStdin":true,"StdinOnce":false,"Tty":false,"User":"","Volumes":{},"WorkingDir":""}"
May 18 16:19:14 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:14.677013914Z" level=debug msg="container mounted via layerStore: &{/var/lib/docker/vfs/dir/bee526de252223e54a56cc37097da4f7c76eb89870c24ecc0177bcff62201368 0x5594ca48fa20 0x5594ca48fa20}"
May 18 16:19:14 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:14.698683026Z" level=debug msg="Calling POST /v1.38/containers/01b5c2d43e17875db13f6a52f72708a5b8b021847eb2f000fd0b94c2f3e694c3/wait?condition=next-exit"
May 18 16:19:14 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:14.704011605Z" level=debug msg="Calling POST /v1.38/containers/01b5c2d43e17875db13f6a52f72708a5b8b021847eb2f000fd0b94c2f3e694c3/start"
May 18 16:19:14 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:14.705103897Z" level=debug msg="container mounted via layerStore: &{/var/lib/docker/vfs/dir/bee526de252223e54a56cc37097da4f7c76eb89870c24ecc0177bcff62201368 0x5594ca48fa20 0x5594ca48fa20}"
May 18 16:19:14 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:14.707311547Z" level=debug msg="Assigning addresses for endpoint hungry_newton's interface on network bridge"
May 18 16:19:14 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:14.707498057Z" level=debug msg="RequestAddress(LocalDefault/172.17.0.0/16, , map[])"
May 18 16:19:14 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:14.707699114Z" level=debug msg="Request address PoolID:172.17.0.0/16 App: ipam/default/data, ID: LocalDefault/172.17.0.0/16, DBIndex: 0x0, Bits: 65536, Unselected: 65533, Sequence: (0xc0000000, 1)->(0x0, 2046)->(0x1, 1)->end Curr:0 Serial:false PrefAddress: "
May 18 16:19:14 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:14.722483835Z" level=debug msg="Assigning addresses for endpoint hungry_newton's interface on network bridge"
May 18 16:19:14 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:14.732705146Z" level=debug msg="Programming external connectivity on endpoint hungry_newton (418e2d7398f8fc0847287937e8df9a274d7dbd4cbb6f3f669169ccb7daa3d544)"
May 18 16:19:14 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:14.734021836Z" level=debug msg="EnableService 01b5c2d43e17875db13f6a52f72708a5b8b021847eb2f000fd0b94c2f3e694c3 START"
May 18 16:19:14 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:14.734118372Z" level=debug msg="EnableService 01b5c2d43e17875db13f6a52f72708a5b8b021847eb2f000fd0b94c2f3e694c3 DONE"
May 18 16:19:14 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:14.739487060Z" level=debug msg="bundle dir created" bundle=/var/run/docker/containerd/01b5c2d43e17875db13f6a52f72708a5b8b021847eb2f000fd0b94c2f3e694c3 module=libcontainerd namespace=moby root=/var/lib/docker/vfs/dir/bee526de252223e54a56cc37097da4f7c76eb89870c24ecc0177bcff62201368
May 18 16:19:14 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:14Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/01b5c2d43e17875db13f6a52f72708a5b8b021847eb2f000fd0b94c2f3e694c3/shim.sock" debug=false pid=6486
May 18 16:19:15 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:15.656935630Z" level=debug msg="sandbox set key processing took 205.566294ms for container 01b5c2d43e17875db13f6a52f72708a5b8b021847eb2f000fd0b94c2f3e694c3"
May 18 16:19:15 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:15.674951623Z" level=debug msg=event module=libcontainerd namespace=moby topic=/tasks/create
May 18 16:19:15 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:15.687541313Z" level=error msg="stream copy error: read /proc/self/fd/23: file already closed"
May 18 16:19:15 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:15.687567973Z" level=error msg="stream copy error: read /proc/self/fd/24: file already closed"
May 18 16:19:15 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:15.693617174Z" level=error msg="failed to delete task after fail start" container=01b5c2d43e17875db13f6a52f72708a5b8b021847eb2f000fd0b94c2f3e694c3 error="task must be stopped before deletion: running: failed precondition" module=libcontainerd namespace=moby
May 18 16:19:15 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:15.697917680Z" level=error msg="failed to delete failed start container" container=01b5c2d43e17875db13f6a52f72708a5b8b021847eb2f000fd0b94c2f3e694c3 error="cannot delete running task 01b5c2d43e17875db13f6a52f72708a5b8b021847eb2f000fd0b94c2f3e694c3: failed precondition"
May 18 16:19:15 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:15.700584348Z" level=debug msg="Revoking external connectivity on endpoint hungry_newton (418e2d7398f8fc0847287937e8df9a274d7dbd4cbb6f3f669169ccb7daa3d544)"
May 18 16:19:15 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:15.700996473Z" level=debug msg="DeleteConntrackEntries purged ipv4:0, ipv6:0"
May 18 16:19:15 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:15.723238773Z" level=debug msg="Releasing addresses for endpoint hungry_newton's interface on network bridge"
May 18 16:19:15 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:15.723375297Z" level=debug msg="ReleaseAddress(LocalDefault/172.17.0.0/16, 172.17.0.2)"
May 18 16:19:15 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:15.723463849Z" level=debug msg="Released address PoolID:LocalDefault/172.17.0.0/16, Address:172.17.0.2 Sequence:App: ipam/default/data, ID: LocalDefault/172.17.0.0/16, DBIndex: 0x0, Bits: 65536, Unselected: 65532, Sequence: (0xe0000000, 1)->(0x0, 2046)->(0x1, 1)->end Curr:3"
May 18 16:19:15 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:15.730333950Z" level=error msg="01b5c2d43e17875db13f6a52f72708a5b8b021847eb2f000fd0b94c2f3e694c3 cleanup: failed to delete container from containerd: cannot delete running task 01b5c2d43e17875db13f6a52f72708a5b8b021847eb2f000fd0b94c2f3e694c3: failed precondition"
May 18 16:19:15 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:15.730608735Z" level=error msg="Handler for POST /v1.38/containers/01b5c2d43e17875db13f6a52f72708a5b8b021847eb2f000fd0b94c2f3e694c3/start returned error: cgroups: cgroup deleted: unknown"
May 18 16:19:55 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:55.775014671Z" level=debug msg=event module=libcontainerd namespace=moby topic=/tasks/exit
May 18 16:19:55 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:55.775100344Z" level=warning msg="Ignoring Exit Event, no such exec command found" container=01b5c2d43e17875db13f6a52f72708a5b8b021847eb2f000fd0b94c2f3e694c3 exec-id=01b5c2d43e17875db13f6a52f72708a5b8b021847eb2f000fd0b94c2f3e694c3 exec-pid=6505
May 18 16:39:53 localhost.localdomain dockerd[295]: time="2019-05-18T16:39:53.200760551Z" level=debug msg="Calling GET /_ping"
May 18 16:39:53 localhost.localdomain dockerd[295]: time="2019-05-18T16:39:53.202086084Z" level=debug msg="Calling GET /v1.38/images/json"
root@localhost:/# docker run -id k8s.gcr.io/etcd:3.2.24 usleep 500
docker run -it k8s.gcr.io/etcd:3.2.24 usleep 500(or /bin/sh, error is the same)
docker: Error response from daemon: cgroups: cgroup deleted: unknown.
Output of docker version:
Inside the kind image:
Client:
Version: 18.06.3-ce
API version: 1.38
Go version: go1.10.3
Git commit: d7080c1
Built: Wed Feb 20 02:28:10 2019
OS/Arch: linux/amd64
Experimental: false
Server:
Engine:
Version: 18.06.3-ce
API version: 1.38 (minimum version 1.12)
Go version: go1.10.3
Git commit: d7080c1
Built: Wed Feb 20 02:26:34 2019
OS/Arch: linux/amd64
Experimental: false
Output of docker info:
docker info inside kind
Containers: 2
Running: 0
Paused: 0
Stopped: 2
Images: 9
Server Version: 18.06.3-ce
Storage Driver: vfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: a592beb5bc4c4092b1b1bac971afed27687340c5
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 4.9.137-1.ph2-esx
Operating System: Ubuntu 18.04.1 LTS (containerized)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 3.86GiB
Name: localhost.localdomain
ID: LBRX:S5U4:2TOD:XT6W:YUNN:DJGH:N4N5:QKK3:GH6I:645R:K6ER:Y2FC
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Additional environment details (AWS, VirtualBox, physical, etc.):
I also tried to do the same thing under CentOS7. But the process works fine.
under systemctl status docker -l, there is:
CGroup: /system.slice/docker.service
295 /usr/bin/dockerd -H fd:// --log-level=debug
302 docker-containerd --config /var/run/docker/containerd/containerd.toml
6486 docker-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/01b5c2d43e17875db13f6a52f72708a5b8b021847eb2f000fd0b94c2f3e694c3 -address /var/run/docker/containerd/docker-containerd.sock -containerd-binary /usr/bin/docker-containerd -runtime-root /var/run/docker/runtime-runc
Maybe the container started up for a very short time and then be killed?