Git Product home page Git Product logo

Comments (8)

liulanzheng avatar liulanzheng commented on September 6, 2024 1

@shahzzzam I saw my kernel was update from 5.8.0-1042-azure to 5.8.0-1043-azure during apt upgrade. I think the bug is fixed in new kernel.

from accelerated-container-image.

liulanzheng avatar liulanzheng commented on September 6, 2024

I did not reproduce this failure, you may try to push the converted image to registry and use rpull and then run a container, to see whether it works.
If it does not work, please provide your containerd and os version.

from accelerated-container-image.

shahzzzam avatar shahzzzam commented on September 6, 2024

Btw, the suggested way to push would be ctr image push -u "<creds>" myreg.azurecr.io/test/redis:6.2.1?

I tried to push and rpull as per your suggestion and it fails again with the following error

ctr: failed to attach and mount for snapshot 40: failed to mount /dev/sdc to /var/lib/overlaybd/snapshots/40/block/mountpoint: read-only file system: unknown

When tried to run this image using k8s, I get the following error:

error="failed to create containerd task: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: \"docker-entrypoint.sh\ ": executable file not found in $PATH: unknown

It seems to me that the image that is converted is somehow corrupted.

Note that the same machine was successfully able to run the registry.hub.docker.com/overlaybd/redis:6.2.1_obd. So for whatever reason, converting it fails for me. What do you suggest?

containerd: v1.4.1 ("https://mobyartifacts.azureedge.net/moby/moby-containerd/1.4.1+azure/bionic/linux_amd64/moby-containerd_1.4.1+azure-1_amd64.deb")
OS: Ubuntu 18.04.5 LTS (GNU/Linux 5.4.0-1055-azure x86_64)

from accelerated-container-image.

shahzzzam avatar shahzzzam commented on September 6, 2024

I used a normal containerd v1.4.1 and even just used localhost:5000 example on your page.

VERSION=1.4.1
wget https://github.com/containerd/containerd/releases/download/v${VERSION}/cri-containerd-cni-${VERSION}-linux-amd64.tar.gz

I suspect there might be low level library incompatible with azure linux kernel?
Error:

ctr run --net-host --snapshotter=overlaybd --rm -t localhost:5000/redis:6.2.1_obd demo
ctr: failed to prepare extraction snapshot "extract-802112965-hKE9 sha256:efaff7faaadccc90305dc329108266c0a01483d205e58798ef0413bcaa6f674a": failed to attach and mount for snapshot 10: failed to mount /dev/sde to /var/lib/overlaybd/snapshots/10/block/mountpoint: read-only file system: unknown

Command History:
image

from accelerated-container-image.

liulanzheng avatar liulanzheng commented on September 6, 2024

i will try azure linux, it may take a little longer

from accelerated-container-image.

shahzzzam avatar shahzzzam commented on September 6, 2024

Did you get a chance to try. I used the following commands if you want to reproduce the issue!:

You can provision following VM in Azure:
Operating system: Linux (ubuntu 20.04)
Size :Standard D2ds_v4 (2 vcpus, 8 GiB memory)

Please note, you need to turn anonymous auth for your registry here due to the following auth issue: containerd/overlaybd#58

You can do so by: using az

az acr update --name myregistry --anonymous-pull-enabled false
1  sudo apt update
2  sudo apt install -y pkg-config libgflags-dev libcurl4-openssl-dev libssl-dev libaio-dev libnl-3-dev libnl-genl-3-dev libglib2.0-dev
3  sudo apt install -y make cmake g++ gcc
4  wget https://github.com/google/googletest/archive/refs/tags/release-1.10.0.tar.gz
5  tar -zxvf release-1.10.0.tar.gz
6  cd googletest-release-1.10.0/
7  cmake CMakeLists.txt
8  make
9  sudo make install
10  cd ..
11  git clone https://github.com/alibaba/overlaybd.git
12  cd overlaybd
13  mkdir build
14  cd build
15  cmake .. -DCMAKE_BUILD_TYPE=Debug -DBUILD_TESTING=1
16  make -j8
17  sudo make install
18  sudo systemctl enable /opt/overlaybd/overlaybd-tcmu.service
19  sudo systemctl start overlaybd-tcmu
20  cd ../..
21  curl -OL https://golang.org/dl/go1.17.2.linux-amd64.tar.gz
22  sudo tar -C /usr/local/ -xvf go1.17.2.linux-amd64.tar.gz
23  sudo echo 'export PATH=$PATH:/usr/local/go/bin' >> ~/.profile
24  source ~/.profile
25  moby_runc_package_url=https://packages.microsoft.com/ubuntu/20.04/prod/pool/main/m/moby-runc/moby-runc_1.0.2%2Bazure-1_amd64.deb
26  moby_runc_package_file="./moby-runc.deb"
27  curl -sSL $moby_runc_package_url -o $moby_runc_package_file
28  sudo dpkg --force-all -i $moby_runc_package_file
29  moby_containerd_package_url=https://packages.microsoft.com/ubuntu/20.04/prod/pool/main/m/moby-containerd/moby-containerd_1.5.7%2Bazure-1_amd64.deb
30  moby_containerd_package_file="./moby-containerd.deb"
31  curl -sSL $moby_containerd_package_url -o $moby_containerd_package_file
32  sudo dpkg --force-all -i $moby_containerd_package_file
33  cd accelerated-container-image
34  make
35  sudo mkdir /etc/overlaybd-snapshotter
36  sudo cat <<-EOF | sudo tee /etc/overlaybd-snapshotter/config.json
    {
    "root": "/var/lib/containerd/io.containerd.snapshotter.v1.overlaybd",
    "address": "/run/overlaybd-snapshotter/overlaybd.sock"
    }
EOF

37  sudo mkdir /etc/containerd
38  sudo cat <<-EOF | sudo tee --append /etc/containerd/config.toml
    [proxy_plugins.overlaybd]
    type = "snapshot"
    address = "/run/overlaybd-snapshotter/overlaybd.sock"
EOF

39  sudo bin/overlaybd-snapshotter

(from another terminal)

1  sudo ctr content fetch registry.hub.docker.com/library/redis:6.2.1
2  sudo bin/ctr obdconv registry.hub.docker.com/library/redis:6.2.1 nonpe.azurecr.io/overlaybd/redis:1
3  sudo ctr image push nonpe.azurecr.io/overlaybd/redis:1 -u "nonpe:<password>"
4  sudo ctr image rm nonpe.azurecr.io/overlaybd/redis:1
5  sudo bin/ctr rpull nonpe.azurecr.io/overlaybd/redis:1
6  sudo ctr run --net-host --snapshotter=overlaybd --rm -t  nonpe.azurecr.io/overlaybd/redis:1 demo
ctr: failed to attach and mount for snapshot 14: failed to mount /dev/sdc to /var/lib/containerd/io.containerd.snapshotter.v1.overlaybd/snapshots/14/block/mountpoint: read-only file system: unknown

from accelerated-container-image.

liulanzheng avatar liulanzheng commented on September 6, 2024

@shahzzzam Thank you, i have reproduced the problem. i'm trying to fix it.

from accelerated-container-image.

liulanzheng avatar liulanzheng commented on September 6, 2024

@shahzzzam It is caused by data unsync after calling umount in converting progress, which raised data loss. Errors can be found in dmesg.
I tried to add a sync after umount, converting worked well without errors in dmesg. But this is not a general solution for umount should complete all pending writes. So it maybe a bug under specific environment.
It also can be fixed by apt upgrade, I saw several packages are upgraded but i'm not sure which one fix it.

from accelerated-container-image.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.