Upgrade to 1.14 breaks some docker container (os: debian) about crun HOT 11 CLOSED

I also need to disable my https://build.opensuse.org/package/show/home:alvistack/containers-crun-1.14, and rolling back all of my Kubernetes nodes on Ubuntu 20.04/22.04 with crun 1.13.

If bpf related should due to 9306457

If apparmor related should due to 5078ce6

from crun.

I am not able to reproduce the issue (I've not tried to configure Docker), could you try if the following patch solves the problem for you?

diff --git a/src/libcrun/ebpf.c b/src/libcrun/ebpf.c
index 5534d47..cb0f8f2 100644
--- a/src/libcrun/ebpf.c
+++ b/src/libcrun/ebpf.c
@@ -483,12 +483,15 @@ libcrun_ebpf_load (struct bpf_program *program, int dirfd, const char *pin, libc
   fd = bpf (BPF_PROG_LOAD, &attr, sizeof (attr));
   if (fd < 0)
     {
-      const size_t log_size = 8192;
-      cleanup_free char *log = xmalloc (log_size);
-
       /* Prior to Linux 5.11, eBPF programs were accounted to the memlock
          prlimit.  Attempt to bump the limit, if possible.  */
       bump_memlock ();
+      fd = bpf (BPF_PROG_LOAD, &attr, sizeof (attr));
+    }
+  if (fd < 0)
+    {
+      const size_t log_size = 8192;
+      cleanup_free char *log = xmalloc (log_size);
 
       log[0] = '\0';
       attr.log_level = 1;

As a workaround, you can try forcing the memory lock ulimit to 4kb, if you are using systemd you can force LimitMEMLOCK=4096 or higher

from crun.

Ciao Giuseppe, e grazie

Actually, in all vms affected by this issue the ulimit is set to unlimited, so I dont' think setting to 4kb can help.
I'll try to rebuild crun with your diff to see if it works.

from crun.

Actually, in all vms affected by this issue the ulimit is set to unlimited, so I dont' think setting to 4kb can help.

could you please check what is the limit for the Docker (or containerd) daemon?

You can grab it from /proc/$(PID DOCKER DAEMON)/limits

from crun.

Sure!

root@norinf01:~# ps aux | grep dockerd
root       26033  2.3  0.8 4653564 136040 ?      Ssl  Jan25  22:26 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
root      473511  0.0  0.0   6240   700 pts/0    S+   15:25   0:00 grep dockerd
root@norinf01:~# cat /proc/26033/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        unlimited            unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             unlimited            unlimited            processes
Max open files            524288               524288               files
Max locked memory         65536                65536                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       63848                63848                signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us

from crun.

thanks, the locked memory ulimit seems fine. I can open a PR with my patch, if you've a possibility to try it out

from crun.

I've configured Docker on Debian 11 with crun, but I am not able to reproduce the issue. What kernel version are you using? How have you created the container?

from crun.

@giuseppe just a guess, could it be related to cgroupns which I set to host? It's the only customization which I have done on docker

Also, I wrote you an email if you wish to test something together, just a proposal ;)

from crun.

that should not really matter, but you never know...

Thanks for the proposal, yes that would help, I'd like to validate my patch in a configuration where we can see the regression

from crun.

Hi @giuseppe, I tried to apply the diff you suggested, and the problem seems to have disappeared

Thanks a lot! Marco

from crun.

thanks for onfirming it! Opened a PR: #1405

from crun.