Current extended-base (`quay.io/bioconda/base-glibc-debian-bash:3.0`) for container builds yields docker images unusable on some systems about bioconda-utils HOT 10 CLOSED

My image has been hit by the same problem..., using buildah

I patched the source code of buildah here and the xattr + pull error is gone.

here you can find how I replaced buildah with one build from source in GitHub Actions Hosted Runners.

I have opened a PR against containers/storage, waiting for feedback

from bioconda-utils.

@pontus the base image is built using this Dockerfile. We're in the middle of moving the image-building over into this repo (bioconda-utils), but that dockerfile is not changing.

The building itself is happening here with podman/buildah. That is currently being unified with other image building to happen here in bioconda-utils, but the commands are not changing.

It's not immediately clear to me where the issue is being introduced, whether in the dockerfile, buildah commands, or later on when creating containers with mulled/involucro. Based on the links above is there anything you see that might be causing the issue?

from bioconda-utils.

Thanks, I don't see anything that sticks out to me and can't directly replicate this when I try locally, I'll try to play a bit with that in github actions and see if I can trigger it.

from bioconda-utils.

Sounds promising, thanks @ChristopherHX! Link to PR for others: containers/storage#1847 (took me a while to find because I didn't realise that containers/storage was an actual repo name at first 😅 )

from bioconda-utils.

Hrrm, I haven't yet been able to trigger this issue with buildah, and the commit there is large enough that I can't read it and figure out how to trigger it either.

from bioconda-utils.

Hrrm, I haven't yet been able to trigger this issue with buildah

See containers/storage#1847 for how to reproduce this by pulling an image built by buildah using latest docker:dind-rootless image.
My host system can apply that xattr without an error using a rootful docker 25.x version.

I believe docker buildx doesn't store extended attributes (xattr) in container layers.
Images built with buildah package it if a run step somehow contributes that xattr to the image.

and the commit there is large

I'm just adding

&& !strings.HasPrefix(key, "user.overlay.")

to an if condition in the code that creates an archive of layers built by buildah

The large diff is due to an implicit bump of the patched dependency, since I didn't patch the old revision. See containers/storage#1847 for the actual diff

I never had to deal with any xattr before so I have no idea of implications

from bioconda-utils.

My change seems to got an approval so it might be only a matter of time to get into buildah source tree

from bioconda-utils.

See containers/storage#1847 for how to reproduce this by pulling an image built by buildah using latest docker:dind-rootless image. My host system can apply that xattr without an error using a rootful docker 25.x version.

Yes, I've seen the issue, I just haven't been able to trigger the issue (being the built image having a that extended attribute somewhere) myself with buildah.

from bioconda-utils.

These steps for old buildah releases (like buildah version 1.23.1 (image-spec 1.0.1, runtime-spec 1.0.2-dev) as of ubuntu 22.04 like GitHub Hosted Runners) are enough to have the xattr in several places, tar command for verify, no need for docker

buildah from ubuntu:latest
buildah run ubuntu-working-container apt-get update
buildah run ubuntu-working-container apt-get install -y xattr
buildah run ubuntu-working-container sh -c "echo "TEST" > /etc/test.txt"
buildah commit ubuntu-working-container
buildah images
buildah push <shafromimages> oci:layout:test:test
tar --xattrs --xattrs-include='*' -tvvf layout/blobs/sha256/<shaoflayer-added-by-buildah>

Output of tar

drwxr-xr-x* root/root         0 2024-03-01 17:14 etc/
  x: 0 user.overlay.origin
-rw-r--r--  root/root      1748 2022-01-06 17:26 etc/inputrc
-rw-r--r--  root/root      5991 2024-03-01 17:09 etc/ld.so.cache
-rw-r--r--  root/root     72029 2022-03-21 10:12 etc/mime.types
drwxr-xr-x  root/root         0 2024-03-01 17:09 etc/python3/
-rw-r--r--  root/root        94 2024-03-01 17:09 etc/python3/debian_config
drwxr-xr-x  root/root         0 2024-03-01 17:09 etc/python3.10/
-rw-r--r--  root/root       155 2023-11-20 16:14 etc/python3.10/sitecustomize.py
-rw-r--r--  root/root         5 2024-03-01 17:14 etc/test.txt
drwxr-xr-x* root/root         0 2024-03-01 17:08 run/
  x: 0 user.overlay.origin
-rwxr-xr-x  root/root         0 2024-03-01 17:08 run/.containerenv
drwxrwxrwt* root/root         0 2024-03-01 17:09 tmp/
  x: 0 user.overlay.origin
drwxr-xr-x* root/root         0 2024-02-12 15:02 usr/
  x: 1 user.overlay.impure
  x: 0 user.overlay.origin
drwxr-xr-x* root/root         0 2024-03-01 17:09 usr/bin/
  x: 0 user.overlay.origin
lrwxrwxrwx  root/root         0 2022-08-18 12:39 usr/bin/pdb3 -> pdb3.10
lrwxrwxrwx  root/root         0 2023-11-20 16:14 usr/bin/pdb3.10 -> ../lib/python3.10/pdb.py
-rwxr-xr-x  root/root      7813 2022-08-18 12:39 usr/bin/py3clean
-rwxr-xr-x  root/root     13191 2022-08-18 12:39 usr/bin/py3compile
lrwxrwxrwx  root/root         0 2022-08-18 12:39 usr/bin/py3versions -> ../share/python3/py3versions.py
lrwxrwxrwx  root/root         0 2022-08-18 12:39 usr/bin/pydoc3 -> pydoc3.10
-rwxr-xr-x  root/root        79 2023-11-20 16:14 usr/bin/pydoc3.10
lrwxrwxrwx  root/root         0 2022-08-18 12:39 usr/bin/pygettext3 -> pygettext3.10
-rwxr-xr-x  root/root     24235 2023-06-07 00:30 usr/bin/pygettext3.10
lrwxrwxrwx  root/root         0 2022-08-18 12:39 usr/bin/python3 -> python3.10
-rwxr-xr-x  root/root   5904904 2023-11-20 16:14 usr/bin/python3.10
-rwxr-xr-x  root/root       948 2022-03-25 10:51 usr/bin/xattr
drwxr-xr-x* root/root         0 2024-03-01 17:09 usr/lib/
  x: 1 user.overlay.impure
  x: 0 user.overlay.origin

I just haven't been able to trigger the issue

buildah created a layer with changed files for you? Even an apt install is enough to have many of them... (rootless mode)

My own affected image has been built with docker buildx till 2023-08-01, all these images have no such xattr at all.

Good news, my fix/workaround has been merged in buildah see containers/buildah@662908f
Bad news, ubuntu 22.04 will never pick up this fix.

I wonder if I should create an GitHub Action for setup-buildah from source code and cache the built artifacts via actions/cache, found previously another bug caused by outdated buildah.

from bioconda-utils.

Thanks. I haven't managed to produce anything with these xattrs (and the above example does not) with Ubuntu 23.10 and buildah installed from standard sources with apt.

from bioconda-utils.