ggcr: Docker 25.0 breaks image handling about go-containerregistry HOT 11 CLOSED

I can't reproduce this.

Minimal repro:

$ docker save hello-world@sha256:4bd78111b6914a99dbc560e6a20eab57ff6655aea4a80c50b0c5491968cbc2e6 > hello-world-v25.tar

$ crane validate --tarball hello-world-v25.tar 
FAIL: hello-world-v25.tar: validating layers: unknown blob sha256:3a124446045bb36fb69dc556a1816f475a2c9b0258ae260c7bd517f9031e7d3d
Error: validating layers: unknown blob sha256:3a124446045bb36fb69dc556a1816f475a2c9b0258ae260c7bd517f9031e7d3d

$ tar xvf hello-world-v25.tar 
blobs/
blobs/sha256/
blobs/sha256/ac28800ec8bb38d5c35b49d45a6ac4777544941199075dff8c4eb63e093aa81e
blobs/sha256/b02b598d5d3b77e7c53873d60cb439e05fa547335b5d1a26b66bf0684657d3a8
blobs/sha256/c55c4ab38c73b5b59d76afd01083be207b1897142fee10222e56d1fd708efb62
blobs/sha256/d2c94e258dcb3c5ac2798d32e1249e42ef01cba4841c2234249495f87264ac5a
index.json
manifest.json
oci-layout

$ jq . < manifest.json 
[
  {
    "Config": "blobs/sha256/d2c94e258dcb3c5ac2798d32e1249e42ef01cba4841c2234249495f87264ac5a",
    "RepoTags": null,
    "Layers": [
      "blobs/sha256/ac28800ec8bb38d5c35b49d45a6ac4777544941199075dff8c4eb63e093aa81e"
    ],
    "LayerSources": {
      "sha256:ac28800ec8bb38d5c35b49d45a6ac4777544941199075dff8c4eb63e093aa81e": {
        "mediaType": "application/vnd.oci.image.layer.v1.tar",
        "size": 14848,
        "digest": "sha256:ac28800ec8bb38d5c35b49d45a6ac4777544941199075dff8c4eb63e093aa81e"
      }
    }
  }
]

$ file blobs/sha256/ac28800ec8bb38d5c35b49d45a6ac4777544941199075dff8c4eb63e093aa81e 
blobs/sha256/ac28800ec8bb38d5c35b49d45a6ac4777544941199075dff8c4eb63e093aa81e: POSIX tar archive

Looks like crane is confused by LayerSources.

from go-containerregistry.

The only thing that changed with v25 is that

Not exactly -- they also added an uncompressed local layer to LayerSources that was previously used only (I think) for non-distributable layers. This exposed a bug in how we were handling that.

the archive is also a valid OCI archive.

Not exactly -- the index.json has a null manifests entry, which is not valid.

from go-containerregistry.

I can't reproduce this.

from go-containerregistry.

I can't reproduce this.

Here's minimal standalone code repro:

package main

import (
	"fmt"

	"github.com/google/go-containerregistry/pkg/name"
	"github.com/google/go-containerregistry/pkg/v1/daemon"
)

func main() {
	ref := name.MustParseReference("hello-world")
	img, _ := daemon.Image(ref)
	ls, _ := img.Layers()
	for _, l := range ls {
		d, _ := l.Digest()
		_, err := img.LayerByDigest(d)
		fmt.Println(err)
	}
}

When run in this environment (as reported by docker version):

Client: Docker Engine - Community
 Version:           25.0.0
 API version:       1.44
 Go version:        go1.21.6
 Git commit:        e758fe5
 Built:             Thu Jan 18 17:09:49 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          25.0.0
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       615dfdf
  Built:            Thu Jan 18 17:09:49 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.27
  GitCommit:        a1496014c916f9e62104b33d1bb5bd03b0858e59
 runc:
  Version:          1.1.11
  GitCommit:        v1.1.11-0-g4bccb38
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

the code above outputs unknown blob sha256:ecd863dd28003e46034b63e149a907cb3a5f764255d9eb79515ff36558ce41fa.

When run in this environment (note the downgraded server engine version):

Client: Docker Engine - Community
 Version:           25.0.0
 API version:       1.43 (downgraded from 1.44)
 Go version:        go1.21.6
 Git commit:        e758fe5
 Built:             Thu Jan 18 17:09:49 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.7
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.10
  Git commit:       311b9ff
  Built:            Thu Oct 26 09:07:41 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.27
  GitCommit:        a1496014c916f9e62104b33d1bb5bd03b0858e59
 runc:
  Version:          1.1.11
  GitCommit:        v1.1.11-0-g4bccb38
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

the code outputs <nil>, as expected.

from go-containerregistry.

The problem seems to be that, as used by partial.BlobToDiff(), partial.FSLayers() is supposed to return blob hashes as opposed to DiffIDs. In turn, partial.FSLayers() relies on Manifest.Layers() to hold blob hashes.

The actual code that populates the Manifest is here, and it calls Descriptor() on each *v1.Layer to get the blob hash.

On Docker < 25.0 this type assertion fails, so desc.Digest is calculated and holds the blob hash. On Docker 25.0, however, the same type assertion is successfull, as UncompressedLayer actually is tarball.foreignUncompressedLayer, which results in DiffID hash being returned and the whole partial.BlobToDiff() logic falling apart.

from go-containerregistry.

We also cannot push images exported from Docker 25.0.

➜  crane version
0.18.0
➜  docker version
Client:
 Cloud integration: v1.0.35+desktop.5
 Version:           24.0.7
 API version:       1.43
 Go version:        go1.20.10
 Git commit:        afdd53b
 Built:             Thu Oct 26 09:04:20 2023
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.26.1 (131620)
 Engine:
  Version:          24.0.7
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.10
  Git commit:       311b9ff
  Built:            Thu Oct 26 09:08:15 2023
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.25
  GitCommit:        d8f198a4ed8892c764191ef7b3b06d8a2eeb5c7f
 runc:
  Version:          1.1.10
  GitCommit:        v1.1.10-0-g18a0cb0
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
➜  crane push hello-scratch-docker-24.tar localhost:5000/test-docker:24
2024/01/26 13:39:33 pushed blob: sha256:1bf4ef3c617a6f34a728ec2a5cff1b1dcb926d2d0b93c5bccd830a7918d833da
2024/01/26 13:39:33 pushed blob: sha256:91f3122d7f3d7bf4c65bcfee955064ed51ff651e96f476b7df954aa63cd2ae71
2024/01/26 13:39:33 localhost:5000/test-docker:24: digest: sha256:243d052ba3e857fc4af259630bef9e328d8063478c48fa89bc30b5cd477fd6f8 size: 423
localhost:5000/test-docker@sha256:243d052ba3e857fc4af259630bef9e328d8063478c48fa89bc30b5cd477fd6f8
➜  crane push hello-scratch-docker-25.tar localhost:5000/test-docker:25
2024/01/26 13:39:44 existing blob: sha256:1bf4ef3c617a6f34a728ec2a5cff1b1dcb926d2d0b93c5bccd830a7918d833da
2024/01/26 13:39:44 existing blob: sha256:91f3122d7f3d7bf4c65bcfee955064ed51ff651e96f476b7df954aa63cd2ae71
Error: PUT http://localhost:5000/v2/test-docker/manifests/25: MANIFEST_BLOB_UNKNOWN: blob unknown to registry; sha256:0f0ad350236420631fba749328571f4b364c3c479b46531fc4d07c7535c242a0

The two tar files are straightforward scratch containers with a text file in them. One built and saved with Docker 24.0.7, the other is the same container having been loaded to and saved from a host running Docker 25.0.0. localhost:5000 is running https://hub.docker.com/_/registry version 2.8.3, and we see the same behaviour with ECR.

containers.zip

from go-containerregistry.

Ah, when I tried to upgrade docker, it failed and I didn't notice, so I was on an old version.

I'll look into this.

Edit: Been staring at this for ~10 minutes, I'm not sure if it will ever complete:

Edit2: Yeah seems unlikely:

Edit3: Alright I've reinstalled everything and managed to get an up to date docker desktop version. I'll fix this Monday.

from go-containerregistry.

Turning on the containerd image store fixes everything, as far as I can tell. If you're blocked by this and can turn that on, I'd recommend it.

But without that... this is really interesting behavior.

The image I'm saving (hello-world) looks like this in the registry:

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:ee301c921b8aadc002973b2e0c3da17d701dcd994b606769a7e6eaa100b81d44",
    "size": 581
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:478afc9190022e867bb857b1a25cc5abc7678287af6cb930562ec25be709f1b7",
      "size": 3195
    }
  ],
  "annotations": {
    "org.opencontainers.image.revision": "3fb6ebca4163bf5b9cc496ac3e8f11cb1e754aee",
    "org.opencontainers.image.source": "https://github.com/docker-library/hello-world.git#3fb6ebca4163bf5b9cc496ac3e8f11cb1e754aee:arm64v8/hello-world",
    "org.opencontainers.image.url": "https://hub.docker.com/_/hello-world",
    "org.opencontainers.image.version": "linux"
  }
}

The output of docker save has:

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:ee301c921b8aadc002973b2e0c3da17d701dcd994b606769a7e6eaa100b81d44",
    "size": 581,
    "platform": {
      "architecture": "arm64",
      "os": "linux"
    }
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar",
      "digest": "sha256:12660636fe55438cc3ae7424da7ac56e845cdb52493ff9cf949c47a7f57f8b43",
      "size": 10752
    }
  ]
}

Notable differences:

We've got a platform on the config descriptor.
The layer is uncompressed.
Annotations are missing.

There's a weird index.json too: moby/moby#47210

from go-containerregistry.

Without containerd integration docker pull doesn't really store the manifest from the registry. The manifest is decomposed into the image config and individual uncompressed layers which are stored in the graphdriver-backend image store.

When you do docker save it doesn't have the original manifest/compressed layers, it's created on the fly for push/save. This has been the case before v25 too. The only thing that changed with v25 is that the archive is also a valid OCI archive.

from go-containerregistry.

i also got the weird index.json, but in my case it appeared to be an older image leftover. either way, I think Docker mostly relies on its manifest.json file and the other stuff is there mostly for compatibility (at least, it appears that way the last time i looked :-))

from go-containerregistry.

The manifests: null is a bug when exporting an image via id/digest instead of the reference. It will be fixed in v25.0.3.

from go-containerregistry.