[Bug] Unable to run container with --runtime=sysbox-runc when Docker data root is on an LVM about sysbox HOT 4 OPEN

Hi @cprevosteau , thanks for giving Sysbox a shot!

1. Can you double check the sysbox-mgr and sysbox-fs daemons are running? (e.g., ps -fu root | grep sysbox)

2. Can you double check the shiftfs module is present in the kernel? (lsmod | grep shiftfs)

For example, in my host these result in:

cesar@focal:~/nestybox/sysbox$ ps -fu root | grep sysbox
root     3137321       1  0 Dec16 pts/0    00:00:08 sysbox-mgr --log /var/log/sysbox-mgr.log
root     3137339       1  0 Dec16 pts/0    00:09:12 sysbox-fs --log /var/log/sysbox-fs.log

cesar@focal:~/nestybox/sysbox$ lsmod | grep shiftfs
shiftfs                28672  0

And thus docker run --runtime=sysbox-runc hello-world works without problem:

cesar@focal:~/nestybox/sysbox$ docker run --runtime=sysbox-runc hello-world
Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

Thanks!

from sysbox.

Hi, thx for the quick answer !
Unfortunately nothing there :
1)

$ ps -fu root | grep sysbox
root      177555    1690  0 19:36 pts/5    00:00:00 sysbox-mgr --log /var/log/sysbox-mgr.log
root      177580    1690  0 19:36 pts/5    00:00:06 sysbox-fs --log /var/log/sysbox-fs.log

$ lsmod | grep shiftfs
shiftfs                28672  0

from sysbox.

Would you mind joining the sysbox slack channel? It's easier to debug that way. We can post the resolution back into this GitHub issue once we find the problem.

The link to the slack channel is here:

Link is here at the bottom of this page: https://github.com/nestybox/sysbox#contact

Thanks!

from sysbox.

After debugging this with @cprevosteau (thanks!), we found out that the problem was that the Docker data-root (which is typically at /var/lib/docker on ext4) was configured to a different directory located on top of an LVM.

As far as I know there is nothing wrong with having the Docker data-root on top of an LVM, but for some reason (yet to be investigated) Sysbox is failing when creating the container in this case.

More specifically, when the docker data-root is on top of an LVM, the container's root filesystem is also on top of that LVM, and this causes sysbox-runc to fail very early when creating the container's init process. Unfortunately sysbox-runc is not providing much info on why the failure occurs. We only see getting the final child's pid from pipe caused , meaning that the container's init process died for some reason very early after it was created.

Interestingly, we also noticed that Docker itself does not like the data-root on an LVM when configured with docker userns-remap. That is, if the Docker data-root is on top of an LVM and "userns-remap": "<some-user>" is added to the /etc/docker/daemon.json file, restarting Docker fails. If the data-root is moved to an ext4 physical partition, restarting Docker works without problem. Thus, there is some incompatibility in Docker itself between userns-remap and LVM.

from sysbox.