Git Product home page Git Product logo

cgroups's Introduction

cgroups

Build Status codecov GoDoc Go Report Card

Go package for creating, managing, inspecting, and destroying cgroups. The resources format for settings on the cgroup uses the OCI runtime-spec found here.

Examples (v1)

Create a new cgroup

This creates a new cgroup using a static path for all subsystems under /test.

  • /sys/fs/cgroup/cpu/test
  • /sys/fs/cgroup/memory/test
  • etc....

It uses a single hierarchy and specifies cpu shares as a resource constraint and uses the v1 implementation of cgroups.

shares := uint64(100)
control, err := cgroup1.New(cgroup1.StaticPath("/test"), &specs.LinuxResources{
    CPU: &specs.LinuxCPU{
        Shares: &shares,
    },
})
defer control.Delete()

Create with systemd slice support

control, err := cgroup1.New(cgroup1.Systemd, cgroup1.Slice("system.slice", "runc-test"), &specs.LinuxResources{
    CPU: &specs.CPU{
        Shares: &shares,
    },
})

Load an existing cgroup

control, err = cgroup1.Load(cgroup1.Default, cgroups.StaticPath("/test"))

Add a process to the cgroup

if err := control.Add(cgroup1.Process{Pid:1234}); err != nil {
}

Update the cgroup

To update the resources applied in the cgroup

shares = uint64(200)
if err := control.Update(&specs.LinuxResources{
    CPU: &specs.LinuxCPU{
        Shares: &shares,
    },
}); err != nil {
}

Freeze and Thaw the cgroup

if err := control.Freeze(); err != nil {
}
if err := control.Thaw(); err != nil {
}

List all processes in the cgroup or recursively

processes, err := control.Processes(cgroup1.Devices, recursive)

Get Stats on the cgroup

stats, err := control.Stat()

By adding cgroups.IgnoreNotExist all non-existent files will be ignored, e.g. swap memory stats without swap enabled

stats, err := control.Stat(cgroup1.IgnoreNotExist)

Move process across cgroups

This allows you to take processes from one cgroup and move them to another.

err := control.MoveTo(destination)

Create subcgroup

subCgroup, err := control.New("child", resources)

Registering for memory events

This allows you to get notified by an eventfd for v1 memory cgroups events.

event := cgroup1.MemoryThresholdEvent(50 * 1024 * 1024, false)
efd, err := control.RegisterMemoryEvent(event)
event := cgroup1.MemoryPressureEvent(cgroup1.MediumPressure, cgroup1.DefaultMode)
efd, err := control.RegisterMemoryEvent(event)
efd, err := control.OOMEventFD()
// or by using RegisterMemoryEvent
event := cgroup1.OOMEvent()
efd, err := control.RegisterMemoryEvent(event)

Examples (v2/unified)

Check that the current system is running cgroups v2

var cgroupV2 bool
if cgroups.Mode() == cgroups.Unified {
	cgroupV2 = true
}

Create a new cgroup

This creates a new systemd v2 cgroup slice. Systemd slices consider "-" a special character, so the resulting slice would be located here on disk:

  • /sys/fs/cgroup/my.slice/my-cgroup.slice/my-cgroup-abc.slice
import (
    "github.com/containerd/cgroups/v3/cgroup2"
    specs "github.com/opencontainers/runtime-spec/specs-go"
)

res := cgroup2.Resources{}
// dummy PID of -1 is used for creating a "general slice" to be used as a parent cgroup.
// see https://github.com/containerd/cgroups/blob/1df78138f1e1e6ee593db155c6b369466f577651/v2/manager.go#L732-L735
m, err := cgroup2.NewSystemd("/", "my-cgroup-abc.slice", -1, &res)
if err != nil {
	return err
}

Load an existing cgroup

m, err := cgroup2.LoadSystemd("/", "my-cgroup-abc.slice")
if err != nil {
	return err
}

Delete a cgroup

m, err := cgroup2.LoadSystemd("/", "my-cgroup-abc.slice")
if err != nil {
	return err
}
err = m.DeleteSystemd()
if err != nil {
	return err
}

Kill all processes in a cgroup

m, err := cgroup2.LoadSystemd("/", "my-cgroup-abc.slice")
if err != nil {
	return err
}
err = m.Kill()
if err != nil {
	return err
}

Get and set cgroup type

m, err := cgroup2.LoadSystemd("/", "my-cgroup-abc.slice")
if err != nil {
    return err
}

// https://www.kernel.org/doc/html/v5.0/admin-guide/cgroup-v2.html#threads
cgType, err := m.GetType()
if err != nil {
    return err
}
fmt.Println(cgType)

err = m.SetType(cgroup2.Threaded)
if err != nil {
    return err
}

Attention

All static path should not include /sys/fs/cgroup/ prefix, it should start with your own cgroups name

Project details

Cgroups is a containerd sub-project, licensed under the Apache 2.0 license. As a containerd sub-project, you will find the:

information in our containerd/project repository.

cgroups's People

Contributors

akihirosuda avatar austinvazquez avatar blanet avatar chenrui333 avatar crosbymichael avatar dcantah avatar dependabot[bot] avatar dmcgowan avatar dqminh avatar estesp avatar fuweid avatar gliptak avatar jellor avatar jepio avatar jseba avatar kolyshkin avatar kzys avatar metux avatar mlaventure avatar mxpv avatar samuelkarp avatar singholt avatar sparrc avatar stevvooe avatar thajeztah avatar xianlubird avatar yaoyinnan avatar zhsj avatar zouyee avatar zyqsempai avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

cgroups's Issues

v2: ToggleControllers() should toggle ancestors of c.path, not c.path itself

func (c *Manager) ToggleControllers(controllers []string, t ControllerToggle) error {

ToggleControllers() should toggle ancestors of c.path, not c.path itself.

e.g. when c.path is like /foo/bar/baz, the following files need to be written:

  • /sys/fs/cgroup/foo/cgroup.subtree_control
  • /sys/fs/cgroup/foo/bar/cgroup.subtree_control

Note that /sys/fs/cgroup/foo/bar/baz/cgroup.subtree_control does not need to be written.

cgroup2: failed to create a new cgroup

when I create a new cgroup by using cgctl, it returns nil pointer dereference as follows:

./cgctl --mountpoint /sys/fs/cgroup/unified new /testb
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x6e33b4]

goroutine 1 [running]:
github.com/containerd/cgroups/v2.(*Resources).EnabledControllers(...)
	/root/go/src/github.com/containerd/cgroups/v2/manager.go:105
github.com/containerd/cgroups/v2.NewManager(0x7fffa262777d, 0x16, 0x7fffa2627798, 0x6, 0x0, 0x2, 0x2, 0x3a)
	/root/go/src/github.com/containerd/cgroups/v2/manager.go:180 +0x174
main.glob..func1(0xc0000ae6e0, 0x0, 0xc0000869f0)
	/root/go/src/github.com/containerd/cgroups/cmd/cgctl/main.go:81 +0x1b9
github.com/urfave/cli.HandleAction(0x79a8c0, 0x833cd0, 0xc0000ae6e0, 0xc0000ae6e0, 0x0)
	/root/go/pkg/mod/github.com/urfave/[email protected]/app.go:523 +0xbe
github.com/urfave/cli.Command.Run(0x817e6f, 0x3, 0x0, 0x0, 0x0, 0x0, 0x0, 0x820e2e, 0x13, 0x0, ...)
	/root/go/pkg/mod/github.com/urfave/[email protected]/command.go:174 +0x51c
github.com/urfave/cli.(*App).Run(0xc0000b4380, 0xc000096000, 0x5, 0x5, 0x0, 0x0)
	/root/go/pkg/mod/github.com/urfave/[email protected]/app.go:276 +0x725
main.main()
	/root/go/src/github.com/containerd/cgroups/cmd/cgctl/main.go:63 +0x419

https://github.com/containerd/cgroups/blob/master/v2/manager.go#L189
https://github.com/containerd/cgroups/blob/master/cmd/cgctl/main.go#L80

Cgroup deleted: stream copy error:file /proc/self/fd/** already closed in kind

Description
Tried to run kubernetes-in-docker(kind) image under OS version 4.9.137. the kind image can be started-up with "run --privileged". But once I tried to start some other images like etcd and apiserver inside the kind container. It gave me error like:"
level=error msg="stream copy error: read /proc/self/fd/23: file already closed"

Steps to reproduce the issue:

ctr -n moby run --privileged --net-host --rm -d docker.io/kindest/node node /sbin/init
ctr -n moby t exec -t node /bin/bash
docker run -it k8s.gcr.io/etcd:3.2.24 /bin/sh
Describe the results you received:
docker: Error response from daemon: cgroups: cgroup deleted: unknown.

Describe the results you expected:
container should run

Additional information you deem important (e.g. issue happens only occasionally):
journalctl -xeu docker gave me the whole error is:
75e0e07df8f46962ba6 to 4452b1364faad4e687ef6f6a2cdae1ca16b6572315647cca56f9bcefb50a72c5, size: 138521856"
May 18 16:18:20 localhost.localdomain dockerd[295]: time="2019-05-18T16:18:20.496232865Z" level=debug msg="Calling GET /_ping"
May 18 16:18:20 localhost.localdomain dockerd[295]: time="2019-05-18T16:18:20.497650450Z" level=debug msg="Calling GET /v1.38/images/json"
May 18 16:19:10 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:10.852319228Z" level=debug msg="Calling GET /_ping"
May 18 16:19:10 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:10.853942550Z" level=debug msg="Calling POST /v1.38/containers/create"
May 18 16:19:10 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:10.854340730Z" level=debug msg="form data: {"AttachStderr":false,"AttachStdin":false,"AttachStdout":false,"Cmd":["sleep","40"],"Domainname":"","Entrypoint":null,"Env":[],"HostConfig":{"AutoRemove":false,"Binds":null,"BlkioDeviceReadBps":null,"BlkioDeviceReadIOps":null,"BlkioDeviceWriteBps":null,"BlkioDeviceWriteIOps":null,"BlkioWeight":0,"BlkioWeightDevice":[],"CapAdd":null,"CapDrop":null,"Cgroup":"","CgroupParent":"","ConsoleSize":[0,0],"ContainerIDFile":"","CpuCount":0,"CpuPercent":0,"CpuPeriod":0,"CpuQuota":0,"CpuRealtimePeriod":0,"CpuRealtimeRuntime":0,"CpuShares":0,"CpusetCpus":"","CpusetMems":"","DeviceCgroupRules":null,"Devices":[],"DiskQuota":0,"Dns":[],"DnsOptions":[],"DnsSearch":[],"ExtraHosts":null,"GroupAdd":null,"IOMaximumBandwidth":0,"IOMaximumIOps":0,"IpcMode":"","Isolation":"","KernelMemory":0,"Links":null,"LogConfig":{"Config":{},"Type":""},"MaskedPaths":null,"Memory":0,"MemoryReservation":0,"MemorySwap":0,"MemorySwappiness":-1,"NanoCpus":0,"NetworkMode":"default","OomKillDisable":false,"OomScoreAdj":0,"PidMode":"","PidsLimit":0,"PortBindings":{},"Privileged":false,"PublishAllPorts":false,"ReadonlyPaths":null,"ReadonlyRootfs":false,"RestartPolicy":{"MaximumRetryCount":0,"Name":"no"},"SecurityOpt":null,"ShmSize":0,"UTSMode":"","Ulimits":null,"UsernsMode":"","VolumeDriver":"","VolumesFrom":null},"Hostname":"","Image":"k8s.gcr.io/etcd:3.2.24","Labels":{},"NetworkingConfig":{"EndpointsConfig":{}},"OnBuild":null,"OpenStdin":true,"StdinOnce":false,"Tty":false,"User":"","Volumes":{},"WorkingDir":""}"
May 18 16:19:14 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:14.677013914Z" level=debug msg="container mounted via layerStore: &{/var/lib/docker/vfs/dir/bee526de252223e54a56cc37097da4f7c76eb89870c24ecc0177bcff62201368 0x5594ca48fa20 0x5594ca48fa20}"
May 18 16:19:14 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:14.698683026Z" level=debug msg="Calling POST /v1.38/containers/01b5c2d43e17875db13f6a52f72708a5b8b021847eb2f000fd0b94c2f3e694c3/wait?condition=next-exit"
May 18 16:19:14 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:14.704011605Z" level=debug msg="Calling POST /v1.38/containers/01b5c2d43e17875db13f6a52f72708a5b8b021847eb2f000fd0b94c2f3e694c3/start"
May 18 16:19:14 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:14.705103897Z" level=debug msg="container mounted via layerStore: &{/var/lib/docker/vfs/dir/bee526de252223e54a56cc37097da4f7c76eb89870c24ecc0177bcff62201368 0x5594ca48fa20 0x5594ca48fa20}"
May 18 16:19:14 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:14.707311547Z" level=debug msg="Assigning addresses for endpoint hungry_newton's interface on network bridge"
May 18 16:19:14 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:14.707498057Z" level=debug msg="RequestAddress(LocalDefault/172.17.0.0/16, , map[])"
May 18 16:19:14 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:14.707699114Z" level=debug msg="Request address PoolID:172.17.0.0/16 App: ipam/default/data, ID: LocalDefault/172.17.0.0/16, DBIndex: 0x0, Bits: 65536, Unselected: 65533, Sequence: (0xc0000000, 1)->(0x0, 2046)->(0x1, 1)->end Curr:0 Serial:false PrefAddress: "
May 18 16:19:14 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:14.722483835Z" level=debug msg="Assigning addresses for endpoint hungry_newton's interface on network bridge"
May 18 16:19:14 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:14.732705146Z" level=debug msg="Programming external connectivity on endpoint hungry_newton (418e2d7398f8fc0847287937e8df9a274d7dbd4cbb6f3f669169ccb7daa3d544)"
May 18 16:19:14 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:14.734021836Z" level=debug msg="EnableService 01b5c2d43e17875db13f6a52f72708a5b8b021847eb2f000fd0b94c2f3e694c3 START"
May 18 16:19:14 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:14.734118372Z" level=debug msg="EnableService 01b5c2d43e17875db13f6a52f72708a5b8b021847eb2f000fd0b94c2f3e694c3 DONE"
May 18 16:19:14 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:14.739487060Z" level=debug msg="bundle dir created" bundle=/var/run/docker/containerd/01b5c2d43e17875db13f6a52f72708a5b8b021847eb2f000fd0b94c2f3e694c3 module=libcontainerd namespace=moby root=/var/lib/docker/vfs/dir/bee526de252223e54a56cc37097da4f7c76eb89870c24ecc0177bcff62201368
May 18 16:19:14 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:14Z" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/01b5c2d43e17875db13f6a52f72708a5b8b021847eb2f000fd0b94c2f3e694c3/shim.sock" debug=false pid=6486
May 18 16:19:15 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:15.656935630Z" level=debug msg="sandbox set key processing took 205.566294ms for container 01b5c2d43e17875db13f6a52f72708a5b8b021847eb2f000fd0b94c2f3e694c3"
May 18 16:19:15 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:15.674951623Z" level=debug msg=event module=libcontainerd namespace=moby topic=/tasks/create
May 18 16:19:15 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:15.687541313Z" level=error msg="stream copy error: read /proc/self/fd/23: file already closed"
May 18 16:19:15 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:15.687567973Z" level=error msg="stream copy error: read /proc/self/fd/24: file already closed"
May 18 16:19:15 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:15.693617174Z" level=error msg="failed to delete task after fail start" container=01b5c2d43e17875db13f6a52f72708a5b8b021847eb2f000fd0b94c2f3e694c3 error="task must be stopped before deletion: running: failed precondition" module=libcontainerd namespace=moby
May 18 16:19:15 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:15.697917680Z" level=error msg="failed to delete failed start container" container=01b5c2d43e17875db13f6a52f72708a5b8b021847eb2f000fd0b94c2f3e694c3 error="cannot delete running task 01b5c2d43e17875db13f6a52f72708a5b8b021847eb2f000fd0b94c2f3e694c3: failed precondition"
May 18 16:19:15 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:15.700584348Z" level=debug msg="Revoking external connectivity on endpoint hungry_newton (418e2d7398f8fc0847287937e8df9a274d7dbd4cbb6f3f669169ccb7daa3d544)"
May 18 16:19:15 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:15.700996473Z" level=debug msg="DeleteConntrackEntries purged ipv4:0, ipv6:0"
May 18 16:19:15 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:15.723238773Z" level=debug msg="Releasing addresses for endpoint hungry_newton's interface on network bridge"
May 18 16:19:15 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:15.723375297Z" level=debug msg="ReleaseAddress(LocalDefault/172.17.0.0/16, 172.17.0.2)"
May 18 16:19:15 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:15.723463849Z" level=debug msg="Released address PoolID:LocalDefault/172.17.0.0/16, Address:172.17.0.2 Sequence:App: ipam/default/data, ID: LocalDefault/172.17.0.0/16, DBIndex: 0x0, Bits: 65536, Unselected: 65532, Sequence: (0xe0000000, 1)->(0x0, 2046)->(0x1, 1)->end Curr:3"
May 18 16:19:15 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:15.730333950Z" level=error msg="01b5c2d43e17875db13f6a52f72708a5b8b021847eb2f000fd0b94c2f3e694c3 cleanup: failed to delete container from containerd: cannot delete running task 01b5c2d43e17875db13f6a52f72708a5b8b021847eb2f000fd0b94c2f3e694c3: failed precondition"
May 18 16:19:15 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:15.730608735Z" level=error msg="Handler for POST /v1.38/containers/01b5c2d43e17875db13f6a52f72708a5b8b021847eb2f000fd0b94c2f3e694c3/start returned error: cgroups: cgroup deleted: unknown"
May 18 16:19:55 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:55.775014671Z" level=debug msg=event module=libcontainerd namespace=moby topic=/tasks/exit
May 18 16:19:55 localhost.localdomain dockerd[295]: time="2019-05-18T16:19:55.775100344Z" level=warning msg="Ignoring Exit Event, no such exec command found" container=01b5c2d43e17875db13f6a52f72708a5b8b021847eb2f000fd0b94c2f3e694c3 exec-id=01b5c2d43e17875db13f6a52f72708a5b8b021847eb2f000fd0b94c2f3e694c3 exec-pid=6505
May 18 16:39:53 localhost.localdomain dockerd[295]: time="2019-05-18T16:39:53.200760551Z" level=debug msg="Calling GET /_ping"
May 18 16:39:53 localhost.localdomain dockerd[295]: time="2019-05-18T16:39:53.202086084Z" level=debug msg="Calling GET /v1.38/images/json"
root@localhost:/# docker run -id k8s.gcr.io/etcd:3.2.24 usleep 500
docker run -it k8s.gcr.io/etcd:3.2.24 usleep 500(or /bin/sh, error is the same)
docker: Error response from daemon: cgroups: cgroup deleted: unknown.
Output of docker version:

Inside the kind image:
Client:
Version: 18.06.3-ce
API version: 1.38
Go version: go1.10.3
Git commit: d7080c1
Built: Wed Feb 20 02:28:10 2019
OS/Arch: linux/amd64
Experimental: false

Server:
Engine:
Version: 18.06.3-ce
API version: 1.38 (minimum version 1.12)
Go version: go1.10.3
Git commit: d7080c1
Built: Wed Feb 20 02:26:34 2019
OS/Arch: linux/amd64
Experimental: false

Output of docker info:

docker info inside kind
Containers: 2
Running: 0
Paused: 0
Stopped: 2
Images: 9
Server Version: 18.06.3-ce
Storage Driver: vfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: a592beb5bc4c4092b1b1bac971afed27687340c5
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 4.9.137-1.ph2-esx
Operating System: Ubuntu 18.04.1 LTS (containerized)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 3.86GiB
Name: localhost.localdomain
ID: LBRX:S5U4:2TOD:XT6W:YUNN:DJGH:N4N5:QKK3:GH6I:645R:K6ER:Y2FC
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):
I also tried to do the same thing under CentOS7. But the process works fine.
under systemctl status docker -l, there is:
CGroup: /system.slice/docker.service
295 /usr/bin/dockerd -H fd:// --log-level=debug
302 docker-containerd --config /var/run/docker/containerd/containerd.toml
6486 docker-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/01b5c2d43e17875db13f6a52f72708a5b8b021847eb2f000fd0b94c2f3e694c3 -address /var/run/docker/containerd/docker-containerd.sock -containerd-binary /usr/bin/docker-containerd -runtime-root /var/run/docker/runtime-runc
Maybe the container started up for a very short time and then be killed?

provide build metrics.proto guide in README?

I want to add some Stats struct in metrics.proto file and generate a new metrics.proto.go file, however, I can not find Makefile or any guide or steps to introduce how to compile proto file, can anybody provide some guide?

Support `cgroup.kill` in cgroupv2 manager

Kernel 5.14 added support for a cgroup.kill file in cgroupv2; writing "1" to this file will immediately send SIGKILL to all processes in that cgroup hierarchy.

I have a local change to support this feature in this library but I wasn't sure which way would be the preferred method. I'm currently checking the kernel release information from uname(2) to see if the kernel version has the feature, but this is expensive (handled by a sync.Once wrapper) and I can't be sure that no-one will backport this to an older kernel (not likely but not not a possibility).

The other approach would be to just blindly send the write out and if an error is returned, just assume it's not supported and tell the caller that. An improved variant of this approach would be to have a fallback that walks the processes in the cgroup hierarchy and freezes/kills/thaws them (this is the approach used by libcontainer in runc).

go get error

When I run go get github.com/containerd/cgroups, I've got following errors. How do I resolve this?

# github.com/containerd/cgroups
../../containerd/cgroups/blkio.go:301: cannot use st.Rdev (type int32) as type uint64 in argument to major
../../containerd/cgroups/blkio.go:301: cannot use st.Rdev (type int32) as type uint64 in argument to minor
../../containerd/cgroups/memory.go:158: undefined: unix.SYS_EVENTFD2
  • macOS Sierra 10.12.5
  • uname -a
    • Darwin Hiroaki-no-MacBook-Pro-2.local 16.6.0 Darwin Kernel Version 16.6.0: Fri Apr 14 16:21:16 PDT 2017; root:xnu-3789.60.24~6/RELEASE_X86_64 x86_64
  • go version
    • go1.8.3 darwin/amd64

can not install with go modules

λ glados /tmp/broken                                                            
❯ go mod init github.com/fiws/broken-containerd-cgroups
go: creating new go.mod: module github.com/fiws/broken-containerd-cgroups
λ glados /tmp/broken                                                            
❯ go get github.com/containerd/cgroups 
go: finding github.com/containerd/cgroups latest
go: finding github.com/coreos/go-systemd latest
go: finding golang.org/x/sys latest
# github.com/coreos/go-systemd/dbus
/home/fiws/go/pkg/mod/github.com/coreos/[email protected]/dbus/dbus.go:127:33: cannot use dbus.SystemBusPrivate (type func() (*dbus.Conn, error)) as type func(...<T>) (*dbus.Conn, error) in argument to dbusAuthHelloConnection
/home/fiws/go/pkg/mod/github.com/coreos/[email protected]/dbus/dbus.go:136:33: cannot use dbus.SessionBusPrivate (type func() (*dbus.Conn, error)) as type func(...<T>) (*dbus.Conn, error) in argument to dbusAuthHelloConnection
/home/fiws/go/pkg/mod/github.com/coreos/[email protected]/dbus/dbus.go:146:42: undefined: dbus.ConnOption
/home/fiws/go/pkg/mod/github.com/coreos/[email protected]/dbus/dbus.go:204:48: undefined: dbus.ConnOption
/home/fiws/go/pkg/mod/github.com/coreos/[email protected]/dbus/dbus.go:224:53: undefined: dbus.ConnOption

This is with go 1.13

Set $PERIOD in cpu.max

In documentation we have next description for cpu.max

cpu.max
	A read-write two value file which exists on non-root cgroups.
	The default is "max 100000".

	The maximum bandwidth limit.  It's in the following format::

	  $MAX $PERIOD

	which indicates that the group may consume upto $MAX in each
	$PERIOD duration.  "max" for $MAX indicates no limit.  If only
	one number is written, $MAX is updated.

Currently we updating only $MAX, do we care about $PERIOD at all? If so I will open corresponding PR.

How to modify root path /sys/fs/cgroup

I find that the default path is /sys/fs/cgroup

If I run cgroups process in contaienr and mount host cgroups into container ,how should I change the /sys/fs/cgroups to another path like /mycontainer/sys/fs/cgoups

Add network metrics

When we use containerd to run our applications, we can not collect the network metrics about containers. Can we add the network metrics into metrics.proto?

type NetworkInterface struct {
	// Name is the name of the network interface.
	Name string

	RxBytes   uint64
	RxPackets uint64
	RxErrors  uint64
	RxDropped uint64
	TxBytes   uint64
	TxPackets uint64
	TxErrors  uint64
	TxDropped uint64
}

ping @crosbymichael

cgroups: cgroup deleted

Can't load cgroup with this code:
control, err := cgroups.Load(cgroups.Systemd, cgroups.Slice("user.slice", "user-1000"))
("/user.slice/user-1000.slice" exists)

Cannot develop on Mac OS X due to OS dependency

Working with Visual Studio Code on OS X during development leads to the following errors:

# github.com/containerd/cgroups
../../containerd/cgroups/memory.go:179:33: undefined: unix.SYS_EVENTFD2
../../containerd/cgroups/memory.go:179:55: undefined: unix.EFD_CLOEXEC

Would it be possible to make these declarations optional during development on another plattform?

Systemd controller does not work in user space with Delegate=yes services

According to the documentation for Systemd units, User= units should be able to control subhierarchies under its own control group path as long as they are Delegate=. I have been unable to get this functionality to work with this package.

Consider the following unit mattdaemon.service (remember to substitute $USER with a regular non-root user):

[Unit]
Description=User space delegate test

[Service]
Type=simple
Delegate=yes
User=$USER
ExecStart=/path/to/binary/failor

Assume /path/to/binary contains a binary built with go build from the following source code:

package main

import (
        "log"
        "os"

        "github.com/containerd/cgroups"
        specs "github.com/opencontainers/runtime-spec/specs-go"
)

func main() {
        // load the cgroup for "this" process -- since we assume we run in a Delegate=yes user unit,
        // we can only control subhierarchies beneath this control group
        unitGroup, err := cgroups.Load(cgroups.Systemd, cgroups.PidPath(os.Getpid()))
        if err != nil {
                log.Panicln("Load:", err)
        }

        // create a subgroup based on cgroup.Cgroups interface
        control, err := unitGroup.New("subway", &specs.LinuxResources{})
        if err != nil {
                // Interactive authentication required
                log.Panicln("New:", err)
        }
        /* if this were a unit test, we would continue by creating a process and adding it to our sub group */
        log.Println("Success!") // Only gets printed if Unit is run by root
        _ = control             // rest of code stubbed
}

Reproduce the bug by starting the unit:

$ sudo systemctl daemon-reload
$ sudo systemctl start mattdaemon.service
$ journalctl -xafe --unit=mattdaemon.service

Journal Output

Nov 03 11:58:55 devthrostur01 failor[23042]: 2017/11/03 11:58:55 New: Interactive authentication required.
Nov 03 11:58:55 devthrostur01 systemd[1]: mattdaemon.service: main process exited, code=exited, status=2/INVALIDARGUMENT
...

We can enable the functionality by removing the User= line from the systemd unit, but this should not be necessary as per the docs (man 5 systemd.resource-control) since we have enabled Delegate=yes.

Document basic operations for V1, V2 and system cases

In the context of kata containers, I am trying to figure out how to write code that works both for legacy and unified mode. I did not find any good example of that, and I was wondering if it is documented anywhere. If not, maybe it would be nice to update the readme and other examples with code for both cases?

Also, am I correct in understanding that the module does not presently abstract away resources in the two cases, i.e. you need to use Resources for unified and LinuxResources for legacy? Is the plan that everything will switch to Resources over time, or will that dichotomy persist?

Assuming the two interfaces will coexist in the near future, is there any function that converts from one to the other?

(I will try to answer my own questions here, but I would appreciate if someone more knowledgeable with the project can provide some guidance, notably with respect to future intent).

cgroup2: check list for v2 controller implementations

Urgent (required to get CRI and Moby work)

Metrics functions are urgent.
Update() functions are not necessary for CRI and Moby.

Non-urgent

  • NewSystemd() and LoadSystemd(): #125
  • freezer: #103
  • perf_event
  • cpuset: #111
  • rdma: #119
  • hugetlb (to appear in kernel 5.XX)

Copyright is missing

Dear containerd authors,

I would like to package this library for Debian, however I couldn't find a copyright anywhere. This should be mentioned in the LICENSE file as well, right now the copyright line there is just a template.

cgroups/LICENSE

Line 189 in d578e4e

Copyright [yyyy] [name of copyright owner]

Thanks

Upgrade github.com/coreos/go-systemd to v22

This project currently depends on a version of github.com/coreos/go-systemd that didn't use Go modules. This led to the following error when doing a go get -u all ./...:

go: finding github.com/coreos/go-systemd latest
go get all: no matching versions for query "upgrade" (current version is v0.0.0-20190321100706-95778dfbb74e)

I think this could be resolved by updating to v22, which uses Go modules.

compiled error

When I build my example, there are many errors as follow:

../vendor/github.com/crosbymichael/cgroups/hugetlb.go:45: cannot use limit.Limit (type uint64) as type int64 in argument to strconv.FormatInt
../vendor/github.com/crosbymichael/cgroups/memory.go:67: cannot use resources.Memory.Limit (type *uint64) as type *int64 in argument to g
../vendor/github.com/crosbymichael/cgroups/memory.go:67: cannot use resources.Memory.Swap (type *uint64) as type *int64 in argument to g
../vendor/github.com/crosbymichael/cgroups/memory.go:228: cannot use mem.Limit (type *uint64) as type *int64 in field value
../vendor/github.com/crosbymichael/cgroups/memory.go:232: cannot use mem.Swap (type *uint64) as type *int64 in field value
../vendor/github.com/crosbymichael/cgroups/memory.go:236: cannot use mem.Kernel (type *uint64) as type *int64 in field value
../vendor/github.com/crosbymichael/cgroups/memory.go:240: cannot use mem.KernelTCP (type *uint64) as type *int64 in field value

stats/v1: Missing memory.oom_control fields

Hi,

I am playing with containerd package in our environment which is configured with cgroupv1. In our environment, we would like to have oom_kill metric from memory.oom_control to understand the # of times a process under a given cgroup has been killed due to oom. I notice we do have memory.events fields of the memory controller already taken care of in v2/stats. However, if someone is still using v1/stats, then they would not find those fields related to oom. Cgroupv1 has different name of this file (memory.oom_control). Would be great to include those metrics for v1 (although it has gone old now).

Thanks,
Vishal

dep init error for containerd/cgroups/v2

Hello,
I got the error while running dep init -v -no-examples, please suggest how to deal with this

OS VERSION="18.04.4 LTS (Bionic Beaver)"

dep version
dep:
 version     : devel
 build date  : 
 git hash    : 
 go version  : go1.8.3
 go compiler : gc
 platform    : linux/amd64
(27)  ✗   github.com/coreos/go-systemd at master has problem subpkg(s):
(27)    	github.com/coreos/go-systemd/v22/dbus is missing; required by github.com/containerd/cgroups@master.
(26)    ← no more versions of github.com/coreos/go-systemd to try; begin backtrack
(25)  ← backtrack: popped 28 pkgs from google.golang.org/protobuf
(24)  ← backtrack: popped 3 pkgs from github.com/golang/protobuf
(23)  ← backtrack: no more versions of golang.org/x/text to try
(22)  ← backtrack: popped 1 pkgs from github.com/golang/protobuf
(21)  ← backtrack: no more versions of github.com/Nvveen/Gotty to try
(20)  ← backtrack: popped 28 pkgs from google.golang.org/protobuf
(19)  ← backtrack: no more versions of github.com/gogo/protobuf to try
(18)  ← backtrack: popped 30 pkgs from google.golang.org/protobuf
(17)  ← backtrack: popped 4 pkgs from github.com/fsouza/go-dockerclient
(16)  ← backtrack: popped 2 pkgs from golang.org/x/sys
(15)  ← backtrack: no more versions of google.golang.org/protobuf to try
(14)  ← backtrack: popped 2 pkgs from github.com/Azure/go-ansiterm
(13)  ← backtrack: no more versions of github.com/fsouza/go-dockerclient to try
(12)  ← backtrack: popped 1 pkgs from golang.org/x/sys
(11)  ← backtrack: no more versions of github.com/Azure/go-ansiterm to try
(10)  ← backtrack: popped 1 pkgs from golang.org/x/sys
(9)	← backtrack: no more versions of golang.org/x/crypto to try
(8)	← backtrack: no more versions of github.com/containerd/cgroups to try
(8)	  ? continue github.com/containerd/cgroups with 2 pkgs; 2 more versions to try
(8)	    try github.com/containerd/cgroups@release/1.0
(8)	✗   package github.com/containerd/cgroups/v2 does not exist within project github.com/containerd/cgroups
(8)	    try github.com/containerd/cgroups@release/1.1
(8)	✗   package github.com/containerd/cgroups/v2 does not exist within project github.com/containerd/cgroups
panic: runtime error: index out of range

goroutine 1 [running]:
github.com/golang/dep/internal/gps.(*solver).findValidVersion(0xc420177600, 0xc421047220, 0xc42046c8e0, 0x2, 0x2, 0x0, 0xc42046c8e0)
	github.com/golang/dep/internal/gps/solver.go:911 +0x519
github.com/golang/dep/internal/gps.(*solver).backtrack(0xc420177600, 0xc420511fe0)
	github.com/golang/dep/internal/gps/solver.go:1049 +0x9de
github.com/golang/dep/internal/gps.(*solver).solve(0xc420177600, 0x0, 0x0, 0xc4201fa990)
	github.com/golang/dep/internal/gps/solver.go:495 +0x787
github.com/golang/dep/internal/gps.(*solver).Solve(0xc420177600, 0x21, 0xb34500, 0xc4201fa030, 0xc420016047)
	github.com/golang/dep/internal/gps/solver.go:440 +0xb1
main.(*initCommand).Run(0xc42015baca, 0xc42005f8b0, 0xc4200101f0, 0x0, 0x0, 0x0, 0x0)
	github.com/golang/dep/cmd/dep/init.go:186 +0x8a1
main.(*Config).Run(0xc420076fc0, 0xc420076fc0)
	github.com/golang/dep/cmd/dep/main.go:160 +0x8e5
main.main()
	github.com/golang/dep/cmd/dep/main.go:45 +0x253

undefined: specs.LinuxRdma in rdma.go:48

Hi,
When I run go build ., it throw the problem

# github.com/containerd/cgroups
../../../Go/pkg/mod/github.com/containerd/[email protected]/rdma.go:48:45: undefined: specs.LinuxRdma
../../../Go/pkg/mod/github.com/containerd/[email protected]/rdma.go:67:38: resources.Rdma undefined (type *specs.LinuxResources has no field or method Rdma)

Here is my code

        shares := uint64(100)
	config.cgroup, err = cgroups.New(cgroups.V1, cgroups.StaticPath(cgroupFolderPath), &specs.LinuxResources{
		CPU: &specs.LinuxCPU{
			Shares: &shares,
		},
	})

And here is my go env

GOARCH="amd64"
GOBIN=""
GOCACHE="/home/*********/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/*********/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/lib/go-1.12.6"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go-1.12.6/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build734298513=/tmp/go-build -gno-record-gcc-switches"
  • System: Ubuntu 18.04
  • kernel: Linux THENAME 4.15.0-52-generic #56-Ubuntu SMP Tue Jun 4 22:49:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

I use go mod to manage the module.

The package cannot be vendored with `vndr` (regression in #137)

vndr omits the go-systemd package when github.com/coreos/go-systemd is specified in vendor.conf, due to the regression in #137.

vendor.conf:

github.com/containerd/cgroups 918ed86e29ccef2bf3f0c70d76405781afc3bdf5
github.com/coreos/go-systemd v22.0.0
...

vndr output:

...
2020/01/09 16:35:37 WARNING: package github.com/coreos/go-systemd is unused, consider removing it from vendor.conf

If github.com/coreos/go-systemd/v22 is specified in vendor.conf instead, vndr fails as follows:

2020/01/09 16:38:59 WARNING: package github.com/coreos/go-systemd/v22 is not root import, should be github.com/coreos/go-systemd

vndr version: LK4D4/vndr@d87a917

tagged release versions

Would it be possible to have tagged versioned releases for this repo? Is there a reason why we're explicitly choosing to not have versioned releases? LMK if it's just a matter of doing the work, I could make those changes.

I think it would improve ux for at least those projects using dep to manage containerd/cgroups as a dependency.

lock versions.

Good day to you!

Thank you for opensourcing!

I'd like to use your package, but I've got some troubles.

I've checkouted master branch and got:

../../git/go/src/github.com/crosbymichael/cgroups/cpuset.go:49: cannot use resources.CPU.Cpus (type string) as type *string in field value ../../git/go/src/github.com/crosbymichael/cgroups/cpuset.go:53: cannot use resources.CPU.Mems (type string) as type *string in field value ../../git/go/src/github.com/crosbymichael/cgroups/devices.go:62: invalid indirect of device.Type (type string) ../../git/go/src/github.com/crosbymichael/cgroups/devices.go:65: invalid indirect of device.Access (type string)

I see that package is under active development, so couldnt you lock stable working versions with tags?

Hugetlb subsystem can not limit hugetlb reservations

Brief summary of hugetlb subsystem control files:
hugetlb..rsvd.limit_in_bytes # set/show limit of "hugepagesize" hugetlb reservations
hugetlb..rsvd.max_usage_in_bytes # show max "hugepagesize" hugetlb reservations and no-reserve faults
hugetlb..rsvd.usage_in_bytes # show current reservations and no-reserve faults for "hugepagesize" hugetlb
hugetlb..rsvd.failcnt # show the number of allocation failure due to HugeTLB reservation limit
hugetlb..limit_in_bytes # set/show limit of "hugepagesize" hugetlb faults
hugetlb..max_usage_in_bytes # show max "hugepagesize" hugetlb usage recorded
hugetlb..usage_in_bytes # show current usage for "hugepagesize" hugetlb
hugetlb..failcnt # show the number of allocation failure due to HugeTLB usage limit
hugetlb..numa_stat # show the numa information of the hugetlb memory charged to this cgroup

ref: https://docs.kernel.org/admin-guide/cgroup-v1/hugetlb.html

Currently, only hugetlb.hugepagesize.limit_in_bytes is implemented. When it comes to hugepage reservation, hugetlb cannot limit the hugepages occupied by a process.

Do you have any plans to implement hugetlb..rsvd.limit_in_bytes functionality?

containerd fails to detect OOM events in kernel 4.19+

As reported in moby/moby#38352 by @Farrukh-Aftab (thanks!), containerd fails to detect OOM events on kernel 4.19, likely related to the following changes in the kernel; I'll copy the information here;

Description
Hello,

Apologies if the reported bug is a duplicate of another issue. I tried searching through the issues but didn't find anything similar.

There were some improvements made to OOM killer in kernel 4.19 to make it more 'cgroup aware'. You can find the relevant commits under [1] in the additional information section. After the change went in, Docker isn't setting the .State.OOMKilled flag correctly despite OOM killer being invoked.

To reproduce this, I have created a sample image named fakhan/sl7:oom-test. The entrypoint of the image is a program that consumes around 1024MB of memory. Creating this container with anything lower should trigger OOM killer. I have provided more information below on the bug.

BUG REPORT INFORMATION shown below:

Steps to reproduce the issue:

  1. First on an older kernel:
# docker image pull fakhan/sl7:oom-test
oom-test: Pulling from fakhan/sl7
Digest: sha256:6d0f60ad535ba61767073c8414a26416a1304651e3621689a4a8aac621eb39d2
Status: Image is up to date for fakhan/sl7:oom-test

# uname -r
3.10.0-862.11.6.el7.x86_64

# docker container run --network host --name fkhan-test1 --memory 512m --memory-swap 512m fakhan/sl7:oom-test

# docker container inspect fkhan-test1 --format='{{.State.OOMKilled}}'
true
  1. Then on a newer kernel
# docker image pull fakhan/sl7:oom-test
oom-test: Pulling from fakhan/sl7
Digest: sha256:6d0f60ad535ba61767073c8414a26416a1304651e3621689a4a8aac621eb39d2
Status: Image is up to date for fakhan/sl7:oom-test

# uname -r
4.19.4-1.el7.elrepo.x86_64

# docker container run --network host --name fkhan-test2 --memory 512m --memory-swap 512m fakhan/sl7:oom-test

# docker container inspect fkhan-test2 --format='{{.State.OOMKilled}}'
false

I have posted the snippets from /var/log/messages for both these instances under the additional information section.

Describe the results you received:
As shown above, OOMKilled evaluates to 'False' on kernel 4.19

Describe the results you expected:
OOMKilled should have evaluated to 'True' on kernel 4.19 just like the on the previous versions

Additional information you deem important (e.g. issue happens only occasionally):
[1]
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=dc0b58643aff8b378086f25cce6789ccba68cbcb
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=5989ad7b5ede38d605c588981f634c08252abfc3
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=3d8b38eb81cac81395f6a823f6bf401b327268e6

[2]
/var/log/messages snippets are as follows. First for the older kernel

Dec 11 10:48:08 fkhan-test-kernelv3 dockerd: time="2018-12-11T10:48:08-06:00" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/91091d5382d54a44e697e224ebccba7f587c43184bfb51b123cf8
c16920598d9/shim.sock" debug=false pid=11532
Dec 11 10:48:09 fkhan-test-kernelv3 kernel: thrash invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
Dec 11 10:48:09 fkhan-test-kernelv3 kernel: thrash cpuset=91091d5382d54a44e697e224ebccba7f587c43184bfb51b123cf8c16920598d9 mems_allowed=0-1
Dec 11 10:48:09 fkhan-test-kernelv3 kernel: CPU: 2 PID: 11554 Comm: thrash Kdump: loaded Tainted: G          I    ------------ T 3.10.0-862.11.6.el7.x86_64 #1
Dec 11 10:48:09 fkhan-test-kernelv3 kernel: Hardware name: System Manufacturer System Product Name/KFSN4-DRE, BIOS 1013 11/18/2008
Dec 11 10:48:09 fkhan-test-kernelv3 kernel: Call Trace:
Dec 11 10:48:09 fkhan-test-kernelv3 kernel: [<ffffffffa21135d4>] dump_stack+0x19/0x1b
Dec 11 10:48:09 fkhan-test-kernelv3 kernel: [<ffffffffa210e79f>] dump_header+0x90/0x229
Dec 11 10:48:09 fkhan-test-kernelv3 kernel: [<ffffffffa1b9a7b6>] ? find_lock_task_mm+0x56/0xc0
Dec 11 10:48:09 fkhan-test-kernelv3 kernel: [<ffffffffa1c0f5b8>] ? try_get_mem_cgroup_from_mm+0x28/0x60
Dec 11 10:48:09 fkhan-test-kernelv3 kernel: [<ffffffffa1b9ac64>] oom_kill_process+0x254/0x3d0
Dec 11 10:48:09 fkhan-test-kernelv3 kernel: [<ffffffffa1c133c6>] mem_cgroup_oom_synchronize+0x546/0x570
Dec 11 10:48:09 fkhan-test-kernelv3 kernel: [<ffffffffa1c12840>] ? mem_cgroup_charge_common+0xc0/0xc0
Dec 11 10:48:09 fkhan-test-kernelv3 kernel: [<ffffffffa1b9b4f4>] pagefault_out_of_memory+0x14/0x90
Dec 11 10:48:09 fkhan-test-kernelv3 kernel: [<ffffffffa210c941>] mm_fault_error+0x6a/0x157
Dec 11 10:48:09 fkhan-test-kernelv3 kernel: [<ffffffffa2120846>] __do_page_fault+0x496/0x4f0
Dec 11 10:48:09 fkhan-test-kernelv3 kernel: [<ffffffffa21208d5>] do_page_fault+0x35/0x90
Dec 11 10:48:09 fkhan-test-kernelv3 kernel: [<ffffffffa211ca96>] ? error_swapgs+0xa7/0xbd
Dec 11 10:48:09 fkhan-test-kernelv3 kernel: [<ffffffffa211c758>] page_fault+0x28/0x30
Dec 11 10:48:09 fkhan-test-kernelv3 kernel: Task in /docker/91091d5382d54a44e697e224ebccba7f587c43184bfb51b123cf8c16920598d9 killed as a result of limit of /docker/91091d5382d54a44e697e224ebccba7f587c43184bfb51b123c
f8c16920598d9
Dec 11 10:48:09 fkhan-test-kernelv3 kernel: memory: usage 524288kB, limit 524288kB, failcnt 17
Dec 11 10:48:09 fkhan-test-kernelv3 kernel: memory+swap: usage 524288kB, limit 524288kB, failcnt 0
Dec 11 10:48:09 fkhan-test-kernelv3 kernel: kmem: usage 820kB, limit 9007199254740988kB, failcnt 0
Dec 11 10:48:09 fkhan-test-kernelv3 kernel: Memory cgroup stats for /docker/91091d5382d54a44e697e224ebccba7f587c43184bfb51b123cf8c16920598d9: cache:0KB rss:523468KB rss_huge:0KB mapped_file:0KB swap:0KB inactive_anon:46080KB active_anon:477388KB inactive_file:0KB active_file:0KB unevictable:0KB
Dec 11 10:48:09 fkhan-test-kernelv3 kernel: [ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name
Dec 11 10:48:09 fkhan-test-kernelv3 kernel: [11554]     0 11554   263196   130965     263        0             0 thrash
Dec 11 10:48:09 fkhan-test-kernelv3 kernel: Memory cgroup out of memory: Kill process 11554 (thrash) score 1001 or sacrifice child
Dec 11 10:48:09 fkhan-test-kernelv3 kernel: Killed process 11554 (thrash) total-vm:1052784kB, anon-rss:523464kB, file-rss:396kB, shmem-rss:0kB
Dec 11 10:48:10 fkhan-test-kernelv3 dockerd: time="2018-12-11T10:48:10-06:00" level=info msg="shim reaped" id=91091d5382d54a44e697e224ebccba7f587c43184bfb51b123cf8c16920598d9
Dec 11 10:48:10 fkhan-test-kernelv3 dockerd: time="2018-12-11T10:48:10.336767875-06:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"

Then for the newer kernel

Dec 11 10:49:37 fkhan-test-kernelv4 kernel: thrash invoked oom-killer: gfp_mask=0x6000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=0
Dec 11 10:49:37 fkhan-test-kernelv4 kernel: thrash cpuset=cc6b369add7d52448ae406800daa01f4836195e51190d9ee922fd685446f8368 mems_allowed=0-1
Dec 11 10:49:37 fkhan-test-kernelv4 kernel: CPU: 40 PID: 32276 Comm: thrash Not tainted 4.19.4-1.el7.elrepo.x86_64 #1
Dec 11 10:49:37 fkhan-test-kernelv4 kernel: Hardware name: Supermicro Super Server/X10DRL-iT, BIOS 2.0a 01/13/2017
Dec 11 10:49:37 fkhan-test-kernelv4 kernel: Call Trace:
Dec 11 10:49:37 fkhan-test-kernelv4 kernel: dump_stack+0x63/0x88
Dec 11 10:49:37 fkhan-test-kernelv4 kernel: dump_header+0x78/0x2a4
Dec 11 10:49:37 fkhan-test-kernelv4 kernel: ? mem_cgroup_scan_tasks+0x9c/0xf0
Dec 11 10:49:37 fkhan-test-kernelv4 kernel: oom_kill_process+0x262/0x290
Dec 11 10:49:37 fkhan-test-kernelv4 kernel: out_of_memory+0x140/0x4b0
Dec 11 10:49:37 fkhan-test-kernelv4 kernel: mem_cgroup_out_of_memory+0x4b/0x80
Dec 11 10:49:37 fkhan-test-kernelv4 kernel: try_charge+0x667/0x6f0
Dec 11 10:49:37 fkhan-test-kernelv4 kernel: ? __alloc_pages_nodemask+0xf8/0x260
Dec 11 10:49:37 fkhan-test-kernelv4 kernel: mem_cgroup_try_charge+0x8c/0x1e0
Dec 11 10:49:37 fkhan-test-kernelv4 kernel: mem_cgroup_try_charge_delay+0x22/0x50
Dec 11 10:49:37 fkhan-test-kernelv4 kernel: do_anonymous_page+0x11a/0x650
Dec 11 10:49:37 fkhan-test-kernelv4 kernel: __handle_mm_fault+0xc5e/0xef0
Dec 11 10:49:37 fkhan-test-kernelv4 kernel: handle_mm_fault+0x102/0x220
Dec 11 10:49:37 fkhan-test-kernelv4 kernel: __do_page_fault+0x212/0x4e0
Dec 11 10:49:37 fkhan-test-kernelv4 kernel: do_page_fault+0x37/0x140
Dec 11 10:49:37 fkhan-test-kernelv4 kernel: ? page_fault+0x8/0x30
Dec 11 10:49:37 fkhan-test-kernelv4 kernel: page_fault+0x1e/0x30
Dec 11 10:49:37 fkhan-test-kernelv4 kernel: RIP: 0033:0x400728
Dec 11 10:49:37 fkhan-test-kernelv4 kernel: Code: eb 4d 48 c7 45 e8 00 00 00 00 eb 34 48 c7 45 e0 00 00 00 00 eb 1b 48 8b 45 e0 48 c1 e0 0c 48 03 45 e8 48 03 45 d0 48 8b 55 e8 <88> 10 48 83 45 e0 01 48 8b 45 e0 48 3
b 45 f0 7c db 48 83 45 e8 01
Dec 11 10:49:37 fkhan-test-kernelv4 kernel: RSP: 002b:00007ffe86af6ca0 EFLAGS: 00010206
Dec 11 10:49:37 fkhan-test-kernelv4 kernel: RAX: 0000000021b18000 RBX: 0000000000000000 RCX: 0000000000000012
Dec 11 10:49:37 fkhan-test-kernelv4 kernel: RDX: 0000000000000000 RSI: 000000007fffffed RDI: 0000000000000000
Dec 11 10:49:37 fkhan-test-kernelv4 kernel: RBP: 00007ffe86af6ce0 R08: 0000000000000000 R09: 00007fcf1403214d
Dec 11 10:49:37 fkhan-test-kernelv4 kernel: R10: 0000000000000022 R11: 0000000000000000 R12: 0000000000400530
Dec 11 10:49:37 fkhan-test-kernelv4 kernel: R13: 00007ffe86af6de0 R14: 0000000000000000 R15: 0000000000000000
Dec 11 10:49:37 fkhan-test-kernelv4 kernel: Task in /docker/cc6b369add7d52448ae406800daa01f4836195e51190d9ee922fd685446f8368 killed as a result of limit of /docker/cc6b369add7d52448ae406800daa01f4836195e51190d9ee922
fd685446f8368
Dec 11 10:49:37 fkhan-test-kernelv4 kernel: memory: usage 524288kB, limit 524288kB, failcnt 0
Dec 11 10:49:37 fkhan-test-kernelv4 kernel: memory+swap: usage 524288kB, limit 524288kB, failcnt 34
Dec 11 10:49:37 fkhan-test-kernelv4 kernel: kmem: usage 1888kB, limit 9007199254740988kB, failcnt 0
Dec 11 10:49:37 fkhan-test-kernelv4 kernel: Memory cgroup stats for /docker/cc6b369add7d52448ae406800daa01f4836195e51190d9ee922fd685446f8368: cache:0KB rss:522208KB rss_huge:520192KB shmem:0KB mapped_file:0KB dirty:0KB writeback:0KB swap:0KB inactive_anon:226260KB active_anon:296032KB inactive_file:0KB active_file:0KB unevictable:0KB
Dec 11 10:49:37 fkhan-test-kernelv4 kernel: Tasks state (memory values in pages):
Dec 11 10:49:37 fkhan-test-kernelv4 kernel: [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Dec 11 10:49:37 fkhan-test-kernelv4 kernel: [  32276]     0 32276   263199   130814  1097728        0             0 thrash
Dec 11 10:49:37 fkhan-test-kernelv4 kernel: Memory cgroup out of memory: Kill process 32276 (thrash) score 1000 or sacrifice child
Dec 11 10:49:37 fkhan-test-kernelv4 kernel: Killed process 32276 (thrash) total-vm:1052796kB, anon-rss:522108kB, file-rss:1148kB, shmem-rss:0kB
Dec 11 10:49:37 fkhan-test-kernelv4 kernel: oom_reaper: reaped process 32276 (thrash), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Dec 11 10:49:37 fkhan-test-kernelv4 dockerd: time="2018-12-11T10:49:37-06:00" level=info msg="shim reaped" id=cc6b369add7d52448ae406800daa01f4836195e51190d9ee922fd685446f8368
Dec 11 10:49:37 fkhan-test-kernelv4 dockerd: time="2018-12-11T10:49:37.430951185-06:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"

Output of docker version:
Docker version on both machines is the same (18.06)

# uname -r && docker version
3.10.0-862.11.6.el7.x86_64
Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a
 Built:             Tue Aug 21 17:23:03 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       e68fc7a
  Built:            Tue Aug 21 17:25:29 2018
  OS/Arch:          linux/amd64
  Experimental:     false

# uname -r && docker version
4.19.4-1.el7.elrepo.x86_64
Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a
 Built:             Tue Aug 21 17:23:03 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       e68fc7a
  Built:            Tue Aug 21 17:25:29 2018
  OS/Arch:          linux/amd64
  Experimental:     false

Output of docker info:

# docker info
Containers: 1
 Running: 0
 Paused: 0
 Stopped: 1
Images: 1
Server Version: 18.06.1-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
 seccomp
  Profile: default
Kernel Version: 3.10.0-862.11.6.el7.x86_64
Operating System: Scientific Linux 7.5 (Nitrogen)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 23.39GiB
Name: fkhan-test-kernelv3.fnal.gov
ID: RHNY:DX45:JKGJ:EDBY:IENF:E5DT:VX7R:MLGL:M765:43XU:3V2W:NTLP
Docker Root Dir: /storage/local/data1/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
# docker info
Containers: 1
 Running: 0
 Paused: 0
 Stopped: 1
Images: 1
Server Version: 18.06.1-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.19.4-1.el7.elrepo.x86_64
Operating System: Scientific Linux 7.5 (Nitrogen)
OSType: linux
Architecture: x86_64
CPUs: 56
Total Memory: 251.8GiB
Name: fkhan-test-kernelv4.fnal.gov
ID: ITUE:KSPJ:33PN:3NHA:XPJ6:C2K7:DSNY:6WKW:BSR5:5BAM:BQ25:QAFP
Docker Root Dir: /storage/local/data1/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

Additional environment details (AWS, VirtualBox, physical, etc.):
Nothing special about the environment. I am running these commands on two bare metal boxes with different kernels.

Unable to control

#!/bin/sh

echo "start"
echo "scale=50000; 4*a(1)" | bc -l -q
echo "end"

When executing shell scripts, "bc -l -q" is always 100%
image

Is the child process out of control?

Add a new Cgroup interface AddTask()?

Hi Containerd Authors,

In interface Cgroup we only have Add(Process), which operates on cgroup.procs.
Sometimes we want to set tasks (thread ID), but now no interface can be used.

I propose adding a new interface AddTask() to work on it, what's your opinions?

I can work on it if you think it is reasonable :-)

cgroup path is read-only file system

[root@localhost cgctl]# uname -ar
Linux localhost.localdomain 4.18.0-193.19.1.el8_2.x86_64 #1 SMP Mon Sep 14 14:37:00 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

[harold@localhost cgroups]$ sudo mount | grep cgroup
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,seclabel,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,perf_event)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,blkio)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,pids)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,cpu,cpuacct)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,cpuset)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,devices)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,memory)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,freezer)
cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,rdma)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,hugetlb)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,net_cls,net_prio)

[root@localhost cgctl]# ./cgctl new /testaaa
mkdir /sys/fs/cgroup/testaaa: read-only file system

so I run "sudo mount -o remount,rw /sys/fs/cgroup"

and then

[harold@localhost cgroups]$ sudo cmd/cgctl/cgctl new /test
failed to write subtree controllers [] to "/sys/fs/cgroup/cgroup.subtree_control": open /sys/fs/cgroup/cgroup.subtree_control: no such file or directory

Very strict checking of subsystems' existence while loading cgroup

Trying to load the Cgroup of a running container using

func Load(hierarchy Hierarchy, path Path) (Cgroup, error) {
I am getting an error:

failed to load cgroup at path /docker/d840ada2fb82be4259d6875453e735f7ec19fb5ee4205d4d11445a77dce614e8: cgroups: cgroup deleted

The code that produces this error is:

import (
	"fmt"
	cntdcg "github.com/containerd/cgroups"
)

type Cgroup struct {
	cgroup cntdcg.Cgroup
	Cpath  string
}

// Load loads an existing cgroup using the specified cgroup path, returning a new
// Cgroup instance. If the cgroup does not exist, an error will be returned.
func Load(cpath string) (*Cgroup, error) {
	if cpath[0] != '/' {
		return nil, fmt.Errorf("cgroup path %s without leading /", cpath)
	}
	cg, err := cntdcg.Load(cntdcg.V1, cntdcg.StaticPath(cpath))
	if err != nil {
		return nil, fmt.Errorf("failed to load cgroup at path %s: %v", cpath, err)
	}
	return &Cgroup{cgroup: cg, Cpath: cpath}, nil
}

func main() {
	id := "d840ada2fb82be4259d6875453e735f7ec19fb5ee4205d4d11445a77dce614e8"
	path := "/docker/" + id
	cgroup, err := Load(path)
	if err != nil {
		fmt.Println(err)
	} else {
		fmt.Println(cgroup.cgroup)
	}
}

Diving in I notice that the cgroup path actually does not exist under rdma subsystem while on others, like cpu, exists:

$ ls /sys/fs/cgroup/cpu/docker/ | grep d840ada2fb82
d840ada2fb82be4259d6875453e735f7ec19fb5ee4205d4d11445a77dce614e8
$ ls /sys/fs/cgroup/rdma/docker/ | grep d840ada2fb82

Moreover trying with a container that is running for long enough period seems ok:

$ ls /sys/fs/cgroup/rdma/docker/ | grep 1530cdbc2ef5
1530cdbc2ef503601aa5721a05434d3b2bd5f271688bbdf225309056cd65810c
$ go run main.go
&{0x624e80 [0xc42000c240 0xc420010630 0xc420010640 0xc420010650 0xc420010660 0xc420010670 0xc420010680 0xc420010690 0xc4200106a0 0xc4200106b0 0xc4200106c0 0xc4200106d0 0xc4200106e0 0xc42009b020] {0 0} <nil>}

So it looks to me like the rdma subsystem is taking time to be updated for some reason.
The info of the system I'm trying on:

$ uname -r
4.15.0-36-generic
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.1 LTS
Release:        18.04
Codename:       bionic
$ docker --version
Docker version 18.06.1-ce, build e68fc7a

Finally, is it something that Load needs to be stuck on? How about being more lenient on

if _, err := os.Lstat(s.Path(p)); err != nil {

and just bypassing the subsystems that were not confirmed and instead of returning an error just removing the subsystem from the returning subsystems' list of the struct
subsystems: subsystems,
?

Would this be acceptable? Patching this on my local source-code of the library unblocks me, so will a PR with this change be welcomed?

cc: @crosbymichael

bug: memory.memsw.usage_in_bytes : no such file or directory

Hi there!

I have a problems with Stat() function in cgroups package:

Code example(approximately):

control, err = cgroups.New(cgroups.V1, cgroups.StaticPath(cgroupName), &specs.LinuxResources {})
err := control.Add(cgroups.Process{Pid:int(pid)})
stats, err := control.Stat()
usage := stats.Cpu.Usage.Total
fmt.Printf("\n%v", usage)

It's just a part of a code, but its quite simple for seeing the logic I go through.

But when I try to run it:

  • Process that I expect to be moved into cgroupName is moved, it's ok.
  • My code falls into panic with /sys/fs/cgroup/memory/somecgroupname/memory.memsw.usage_in_bytes: no such file or directory

So, what have I found: memory.memsw.usage_in_bytes file isnt always present in system because its optionaly cgroups feature that could be activated by passing swapaccount=1 into kernel (actualy, I'm using Ubunt/Debian systems in my environment, so I have the same situation as I linked you to)

Additional links about kernel parameters that has impackt to file existing factor:

http://cateee.net/lkddb/web-lkddb/MEMCG_SWAP.html
http://cateee.net/lkddb/web-lkddb/MEMCG_SWAP_ENABLED.html.

So, it looks like we need some check about if the file exists or it doesnt to prevent that kind of panic.
May be I will try to figure it out by myself and create PR then, but if you have enough time and see quite easy way to do it, it would be nice.

Anyway, thank you for opensourcing that again!

getting Error: write /sys/fs/cgroup/cpuset/test/cgroup.procs: no space left on device

I am getting following error while adding pids to the cgroup created:

        shares := uint64(100)
        memlimit := int64(1024)
        control, err := cgroups.New(cgroups.V1, cgroups.StaticPath("/test"), &specs.LinuxResources{
        CPU: &specs.LinuxCPU{
                Shares: &shares,
        },
        Memory: &specs.LinuxMemory{
                Limit: &memlimit,
        },
        })

Now while adding the pid to the cgroup it gives error:

       if err := control.Add(cgroups.Process{Pid:pid}); err != nil {
                fmt.Printf("Error while adding process : %d to cgroups: %s\n",pid,err)
        }

Error:

Error while adding process : 19374 to cgroups: write /sys/fs/cgroup/cpuset/test/cgroup.procs: no space left on device

When I checked "/sys/fs/cgroup/cpuset/test/" itself is not created.

Example usage in container

Hi,

sorry for creating an issue but I didn't know where to ask a quesiton.

Can this library be used in a running container to get cgroups memory limits? I would like to get value of /sys/fs/cgroup/memory/memory.limit_in_bytes. Probably it's not safe to read that file directly hence I would like to use this lib to get the memory_limit_in_bytes.

How should be the cgroups created?

control, err := cgroups.Load(cgroups.V1, cgroups.NestedPath("")) //always error

Creating it by PID does not work either as the os.Getpid() returns 1 in the container.

undefined: specs.LinuxRdma

go get -v github.com/containerd/cgroups@master

go: finding github.com/containerd/cgroups master
github.com/containerd/cgroups

github.com/containerd/cgroups

/root/go/pkg/mod/github.com/containerd/[email protected]/rdma.go:49:45: undefined: specs.LinuxRdma
/root/go/pkg/mod/github.com/containerd/[email protected]/rdma.go:68:38: resources.Rdma undefined (type *specs.LinuxResources has no field or method Rdma)

My go.mod file:

module demo1

go 1.13
require (
	github.com/containerd/cgroups v0.0.0-20191011165608-5fbad35c2a7e // indirect
	github.com/opencontainers/runtime-spec v1.0.1 // indirect
)

blkio stats empty for zfs on kernel 5.x

I have two docker instances on the same server, both using:
Docker version 18.09.3, build 774a1f4

and

containerd github.com/containerd/containerd 1.2.4 e6b3f5632f50dbc4e9cb6288d911bf4f5e95b18e

Kernel:
5.0.1-050001-generic

One is using overlay2, the other is using zfs as the graph driver. The instance on overlay2 is running fine, however the zfs instance is not reporting blkio stats:

  "blkio_stats": {
    "io_service_bytes_recursive": [],
    "io_serviced_recursive": [],
    "io_queue_recursive": [],
    "io_service_time_recursive": [],
    "io_wait_time_recursive": [],
    "io_merged_recursive": [],
    "io_time_recursive": [],
    "sectors_recursive": []
  },

I am trying to debug this further, the blkio controller for cgroups seems to have changed, I am only seeing the following stats inside container cgroups:

# ls -la /sys/fs/cgroup/blkio/
total 0
drwxr-xr-x    2 root     root             0 Mar 21 07:48 .
dr-xr-xr-x   14 root     root           360 Mar 21 07:48 ..
-r--r--r--    1 root     root             0 Mar 21 12:29 blkio.bfq.io_service_bytes
-r--r--r--    1 root     root             0 Mar 21 12:29 blkio.bfq.io_service_bytes_recursive
-r--r--r--    1 root     root             0 Mar 21 12:29 blkio.bfq.io_serviced
-r--r--r--    1 root     root             0 Mar 21 12:29 blkio.bfq.io_serviced_recursive
-rw-r--r--    1 root     root             0 Mar 21 12:29 blkio.bfq.weight
--w-------    1 root     root             0 Mar 21 12:29 blkio.reset_stats
-r--r--r--    1 root     root             0 Mar 21 07:48 blkio.throttle.io_service_bytes
-r--r--r--    1 root     root             0 Mar 21 12:29 blkio.throttle.io_service_bytes_recursive
-r--r--r--    1 root     root             0 Mar 21 07:48 blkio.throttle.io_serviced
-r--r--r--    1 root     root             0 Mar 21 12:29 blkio.throttle.io_serviced_recursive
-rw-r--r--    1 root     root             0 Mar 21 12:29 blkio.throttle.read_bps_device
-rw-r--r--    1 root     root             0 Mar 21 12:29 blkio.throttle.read_iops_device
-rw-r--r--    1 root     root             0 Mar 21 12:29 blkio.throttle.write_bps_device
-rw-r--r--    1 root     root             0 Mar 21 12:29 blkio.throttle.write_iops_device
-rw-r--r--    1 root     root             0 Mar 21 12:29 cgroup.clone_children
-rw-r--r--    1 root     root             0 Mar 21 12:29 cgroup.procs
-rw-r--r--    1 root     root             0 Mar 21 12:29 notify_on_release
-rw-r--r--    1 root     root             0 Mar 21 12:29 tasks

So none of the blkio.io* files mentioned here exist:
https://www.kernel.org/doc/Documentation/cgroup-v1/blkio-controller.txt

These are the default kernel config options in 5.0.1 (we're using a default mainline kernel):

CONFIG_BLK_CGROUP=y
CONFIG_CGROUP_SCHED=y
# CONFIG_DEBUG_BLK_CGROUP is not set
# CONFIG_BLK_CGROUP_IOLATENCY is not set

I have tried with both BFQ enabled and disabled (mq-deadline), also tried zfs with and without the zfs_vdev_scheduler=bfq option, but both made no difference.

I believe the relevant code is:
https://github.com/containerd/cgroups/blob/master/blkio.go

However I haven't had a chance to properly dive into the code yet, perhaps I'm missing something obvious, can anyone point me in the right direction? I've also posted in the relevant docker issue for checking kernel 5.x compatibility here moby/moby#38887, however this seems to be a containerd issue.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.