Comments (8)
@shahzzzam I saw my kernel was update from 5.8.0-1042-azure to 5.8.0-1043-azure during apt upgrade
. I think the bug is fixed in new kernel.
from accelerated-container-image.
I did not reproduce this failure, you may try to push the converted image to registry and use rpull and then run a container, to see whether it works.
If it does not work, please provide your containerd and os version.
from accelerated-container-image.
Btw, the suggested way to push would be ctr image push -u "<creds>" myreg.azurecr.io/test/redis:6.2.1
?
I tried to push and rpull as per your suggestion and it fails again with the following error
ctr: failed to attach and mount for snapshot 40: failed to mount /dev/sdc to /var/lib/overlaybd/snapshots/40/block/mountpoint: read-only file system: unknown
When tried to run this image using k8s, I get the following error:
error="failed to create containerd task: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: \"docker-entrypoint.sh\ ": executable file not found in $PATH: unknown
It seems to me that the image that is converted is somehow corrupted.
Note that the same machine was successfully able to run the registry.hub.docker.com/overlaybd/redis:6.2.1_obd
. So for whatever reason, converting it fails for me. What do you suggest?
containerd: v1.4.1 ("https://mobyartifacts.azureedge.net/moby/moby-containerd/1.4.1+azure/bionic/linux_amd64/moby-containerd_1.4.1+azure-1_amd64.deb")
OS: Ubuntu 18.04.5 LTS (GNU/Linux 5.4.0-1055-azure x86_64)
from accelerated-container-image.
I used a normal containerd v1.4.1 and even just used localhost:5000
example on your page.
VERSION=1.4.1
wget https://github.com/containerd/containerd/releases/download/v${VERSION}/cri-containerd-cni-${VERSION}-linux-amd64.tar.gz
I suspect there might be low level library incompatible with azure linux kernel?
Error:
ctr run --net-host --snapshotter=overlaybd --rm -t localhost:5000/redis:6.2.1_obd demo
ctr: failed to prepare extraction snapshot "extract-802112965-hKE9 sha256:efaff7faaadccc90305dc329108266c0a01483d205e58798ef0413bcaa6f674a": failed to attach and mount for snapshot 10: failed to mount /dev/sde to /var/lib/overlaybd/snapshots/10/block/mountpoint: read-only file system: unknown
from accelerated-container-image.
i will try azure linux, it may take a little longer
from accelerated-container-image.
Did you get a chance to try. I used the following commands if you want to reproduce the issue!:
You can provision following VM in Azure:
Operating system: Linux (ubuntu 20.04)
Size :Standard D2ds_v4 (2 vcpus, 8 GiB memory)
Please note, you need to turn anonymous auth for your registry here due to the following auth issue: containerd/overlaybd#58
You can do so by: using az
az acr update --name myregistry --anonymous-pull-enabled false
1 sudo apt update
2 sudo apt install -y pkg-config libgflags-dev libcurl4-openssl-dev libssl-dev libaio-dev libnl-3-dev libnl-genl-3-dev libglib2.0-dev
3 sudo apt install -y make cmake g++ gcc
4 wget https://github.com/google/googletest/archive/refs/tags/release-1.10.0.tar.gz
5 tar -zxvf release-1.10.0.tar.gz
6 cd googletest-release-1.10.0/
7 cmake CMakeLists.txt
8 make
9 sudo make install
10 cd ..
11 git clone https://github.com/alibaba/overlaybd.git
12 cd overlaybd
13 mkdir build
14 cd build
15 cmake .. -DCMAKE_BUILD_TYPE=Debug -DBUILD_TESTING=1
16 make -j8
17 sudo make install
18 sudo systemctl enable /opt/overlaybd/overlaybd-tcmu.service
19 sudo systemctl start overlaybd-tcmu
20 cd ../..
21 curl -OL https://golang.org/dl/go1.17.2.linux-amd64.tar.gz
22 sudo tar -C /usr/local/ -xvf go1.17.2.linux-amd64.tar.gz
23 sudo echo 'export PATH=$PATH:/usr/local/go/bin' >> ~/.profile
24 source ~/.profile
25 moby_runc_package_url=https://packages.microsoft.com/ubuntu/20.04/prod/pool/main/m/moby-runc/moby-runc_1.0.2%2Bazure-1_amd64.deb
26 moby_runc_package_file="./moby-runc.deb"
27 curl -sSL $moby_runc_package_url -o $moby_runc_package_file
28 sudo dpkg --force-all -i $moby_runc_package_file
29 moby_containerd_package_url=https://packages.microsoft.com/ubuntu/20.04/prod/pool/main/m/moby-containerd/moby-containerd_1.5.7%2Bazure-1_amd64.deb
30 moby_containerd_package_file="./moby-containerd.deb"
31 curl -sSL $moby_containerd_package_url -o $moby_containerd_package_file
32 sudo dpkg --force-all -i $moby_containerd_package_file
33 cd accelerated-container-image
34 make
35 sudo mkdir /etc/overlaybd-snapshotter
36 sudo cat <<-EOF | sudo tee /etc/overlaybd-snapshotter/config.json
{
"root": "/var/lib/containerd/io.containerd.snapshotter.v1.overlaybd",
"address": "/run/overlaybd-snapshotter/overlaybd.sock"
}
EOF
37 sudo mkdir /etc/containerd
38 sudo cat <<-EOF | sudo tee --append /etc/containerd/config.toml
[proxy_plugins.overlaybd]
type = "snapshot"
address = "/run/overlaybd-snapshotter/overlaybd.sock"
EOF
39 sudo bin/overlaybd-snapshotter
(from another terminal)
1 sudo ctr content fetch registry.hub.docker.com/library/redis:6.2.1
2 sudo bin/ctr obdconv registry.hub.docker.com/library/redis:6.2.1 nonpe.azurecr.io/overlaybd/redis:1
3 sudo ctr image push nonpe.azurecr.io/overlaybd/redis:1 -u "nonpe:<password>"
4 sudo ctr image rm nonpe.azurecr.io/overlaybd/redis:1
5 sudo bin/ctr rpull nonpe.azurecr.io/overlaybd/redis:1
6 sudo ctr run --net-host --snapshotter=overlaybd --rm -t nonpe.azurecr.io/overlaybd/redis:1 demo
ctr: failed to attach and mount for snapshot 14: failed to mount /dev/sdc to /var/lib/containerd/io.containerd.snapshotter.v1.overlaybd/snapshots/14/block/mountpoint: read-only file system: unknown
from accelerated-container-image.
@shahzzzam Thank you, i have reproduced the problem. i'm trying to fix it.
from accelerated-container-image.
@shahzzzam It is caused by data unsync after calling umount in converting progress, which raised data loss. Errors can be found in dmesg
.
I tried to add a sync
after umount, converting worked well without errors in dmesg
. But this is not a general solution for umount should complete all pending writes. So it maybe a bug under specific environment.
It also can be fixed by apt upgrade
, I saw several packages are upgraded but i'm not sure which one fix it.
from accelerated-container-image.
Related Issues (20)
- Resource priority amidst contention: async image pull or on-demand fetch HOT 2
- Support image conversion and push across repositories
- ctr snapshot --snapshotter=overlaybd rm key HOT 1
- Modify Dev's Info HOT 5
- Allow the device size/quota to be adjusted HOT 6
- Regression in v1.0.1: Windows: `pkg/utils/du_unix.go:37:29: undefined: syscall.Stat_t`
- unable to build images `failed to prepare: parent snapshot does not exist: not found` HOT 4
- [Userspace Convertor] Mount or avoid reconverting previously Converted Manifests
- [Userspace Convertor] Overlaybd Layer Versioning for Deduplication HOT 2
- [Userspace Convertor] /usr/bin/ping security capability inconsistency HOT 2
- nerdctl running image and crictl interfere with each other
- Question: multi-platform image support HOT 2
- [Userspace convertor] Unexpectedly high number of disk read/written
- Can't mount 'ext4' overlaybd image when change defaultFsType to 'erofs'
- Was v1.1.4 retracted? HOT 2
- Support loading container images from a secondary disk attached to a kubernetes node HOT 1
- Support digest for input image in userspace converter
- Update doc about EROFS support HOT 3
- Attach overlaybd device will fail if the mountpoint contains a symbol link
- Support flattened convert in userspace convertor HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from accelerated-container-image.