s390x Debian packages have checksum mismatches with the actual files about docker-ce-packaging HOT 4 CLOSED

Ah, there are two issues here, both of which are related to the same event. I wrote a bit about it over here: moby/moby#46470 (comment)

In short, the metadata for IBM packages was out of sync with the package files, and there were limited options to recover; the "least evil" was to re-sign the 24.0.6 binaries for non-IBM platforms (currently, signing and publishing are the same step in the release pipeline).

This situation should happen rarely or not at all; this infrastructure is being actively worked on and I don't think we'll have to repeat this, but I'm also looking at e.g. separating signing and publishing to prevent this scenario more robustly in the future.

I wrote a couple more thoughts on mirrors over at docker/docker-install#379 (comment).

As a final take-away, it seems like we'll really need to figure out a canonical venue to discuss Docker CE/download.docker.com since issues end up opened in so many diverse places (and I also receive emails, Slack messages, etc.); I'll continue to mull over where that should be, as the canonical pipeline that defines this infrastructure is actually 4-5 internal repositories.

from docker-ce-packaging.

We also observed hash sum mismatches on the following packages that we use:

# ARM64
/dists/jammy/pool/stable/arm64/docker-ce-cli_24.0.6-1~ubuntu.22.04~jammy_arm64.deb
/dists/jammy/pool/stable/arm64/docker-ce_24.0.6-1~ubuntu.22.04~jammy_arm64.deb
/dists/jammy/pool/stable/arm64/docker-ce-rootless-extras_24.0.6-1~ubuntu.22.04~jammy_arm64.deb
/dists/jammy/pool/stable/arm64/docker-compose-plugin_2.21.0-1~ubuntu.22.04~jammy_arm64.deb
# AMD64
/dists/jammy/pool/stable/amd64/docker-ce-cli_24.0.6-1~ubuntu.22.04~jammy_amd64.deb
/dists/jammy/pool/stable/amd64/docker-ce_24.0.6-1~ubuntu.22.04~jammy_amd64.deb
/dists/jammy/pool/stable/amd64/docker-ce-rootless-extras_24.0.6-1~ubuntu.22.04~jammy_amd64.deb
/dists/jammy/pool/stable/amd64/docker-compose-plugin_2.21.0-1~ubuntu.22.04~jammy_amd64.deb

Diffing the files confirmed that the same versions had been rebuilt and pushed. For example looking, at part of a binary diff of docker-compose-plugin_2.21.0-1~ubuntu.22.04~jammy_arm64.deb:

640736,640737c640736,640737
< 009c6df0: 4461 7465 3a20 5475 6520 5365 7020 2035  Date: Tue Sep  5
< 009c6e00: 2031 363a 3532 3a33 3220 3230 3233 0a52   16:52:32 2023.R
---
> 009c6df0: 4461 7465 3a20 4672 6920 5365 7020 2038  Date: Fri Sep  8
> 009c6e00: 2031 373a 3031 3a32 3120 3230 3233 0a52   17:01:21 2023.R

There were other minor differences in what looked to be a PGP cert inside the file.

We had to ask our package mirror maintainers to manually delete these files from their cache to fix our builds.

My understanding is that pushing up a different build of the same version is considered bad practice. Is there any reason why these weren't released as new packages with their build number bumped (e.g. 24.0.6-2)?

from docker-ce-packaging.

Okay, so someone's done something because today...

andy:foo$ wget https://download.docker.com/linux/ubuntu/dists/focal/pool/stable/s390x/docker-ce_24.0.3-1~ubuntu.20.04~focal_s390x.deb

...

andy:foo$ md5sum docker-ce_24.0.3-1~ubuntu.20.04~focal_s390x.deb 
df3e70ebf3512c73d2b553c145991652  docker-ce_24.0.3-1~ubuntu.20.04~focal_s390x.deb
andy:foo$ sha1sum docker-ce_24.0.3-1~ubuntu.20.04~focal_s390x.deb 
fd5b0d5a95af5a6459fb120362a84edd3a7ffe38  docker-ce_24.0.3-1~ubuntu.20.04~focal_s390x.deb
andy:foo$ sha256sum docker-ce_24.0.3-1~ubuntu.20.04~focal_s390x.deb 
6d1c1656d82b621fcec013566ddb67537bf53cb73ca3ae839fbbd83804b93796  docker-ce_24.0.3-1~ubuntu.20.04~focal_s390x.deb
andy:foo$ sha512sum docker-ce_24.0.3-1~ubuntu.20.04~focal_s390x.deb 
bf96935b6db864510a95e1842cc54b956c3ed6175a56fa9244dccf00b323b9397f0903216d2497ca2982b7c93f7e46d124801fc190d9640476259a9178c1b403  docker-ce_24.0.3-1~ubuntu.20.04~focal_s390x.deb

So now all the checksums match up with what the Packages files claim

Looking at https://download.docker.com/linux/ubuntu/dists/focal/pool/stable/s390x/, the timestamps on the files are unchanged

from docker-ce-packaging.

Also, closing this for now as resolved.

from docker-ce-packaging.