Git Product home page Git Product logo

Comments (10)

mpermar avatar mpermar commented on May 28, 2024 1

@RasmusGodske definitely a bug. @juamedgod figured out. We are using Crane (go-containerregistry) to relocate the packages, and the way we are doing it actually modifies the format of the manifest. Not the content, but the format.

That does not affect any images that were put into registries using go-containerregistry, like Bitnami's for example. And that's why we hadn't seen this earlier. But your image manifest in DockerHub is pretty-printed. Therefore when we relocate it, go-containerregistry flattens the format and the manifest ends up with a different digest which is what fails the verification.

We will come up with a bug fix soon. But if you try using crane to upload something into DockerHub and then try to wrap/unwrap it, then it should work flawlessly.

from distribution-tooling-for-helm.

mpermar avatar mpermar commented on May 28, 2024

Hi @RasmusGodske. Verification usually will fail when the digests that are in the lock file don't match the digests of the images that the user has asked dt to push.

When running relocate , what dt does is to modify the container image references found at different places like Chart.yaml or values.yaml, so indeed it modifies the Chart. When you then run lock afterwards, that generates a lock from the relocated chart which well, it will cause such inconsistency.

This is probably our bad at documentation. wrap and unwrap are meant to be used together, and not with the "finer grained" commands. Also lock is always the first command you would run after annotate. Only then you should run relocate and not the other way around.

Technically, you can reproduce a wrap -> unwrap with: annotate -> lock -> verify -> pull -> relocate -> push -> verify and then manually pushing the chart to a registry with helm or crane. Wrap and unwrap just try to do all that for you.

from distribution-tooling-for-helm.

RasmusGodske avatar RasmusGodske commented on May 28, 2024

Hello @mpermar. Thanks for the quick response, that makes perfect sense. However I have tried using wrap and unwrap together, which caused the same problem. This led me down the path of using the "fine grained" commands.

This is the modified bash script, using just wrap and unwrap.

#!/bin/bash

helm repo add temp --force-update https://benthosdev.github.io/benthos-helm-chart/
helm repo update

helm pull temp/benthos --version 0.8.0 --untar --untardir .

######### Manually Annotating Chart.yaml #########
images_override=$(cat <<-END
- name: benthos
  image: docker.io/jeffail/benthos:4.11.0
END
)

yq -i e ".annotations.images |= \"$images_override\"" ./benthos/Chart.yaml

helm dt wrap ./benthos
helm dt unwrap  benthos-0.8.0.wrap.tgz oci://localhost:5000 --yes

output:

"temp" has been added to your repositories
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "temp" chart repository
...Successfully got an update from the "stable" chart repository
...Successfully got an update from the "benthos" chart repository
...Successfully got an update from the "openebs-jiva" chart repository
...Successfully got an update from the "vmware-tanzu" chart repository
...Successfully got an update from the "traefik" chart repository
...Successfully got an update from the "grafana" chart repository
...Successfully got an update from the "bitnami" chart repository
Update Complete. ⎈Happy Helming!⎈
 »  Wrapping Helm chart "./benthos"
    ✔  Images.lock file written to "/home/godske/Documents/repositoriesoci-container-registry-storage/test/benthos/Images.lock"                          
    »  Pulling images into "/home/godske/Documents/repositories/oci-container-registry-storage/test/benthos/images"
       ✔  All images pulled successfully                                                                                                                       
    ✔  Compressed into "/home/godske/Documents/repositories/work/oci-container-registry-storage/test/benthos-0.8.0.wrap.tgz"                                   
       
 🎉  Helm chart wrapped into "/home/godske/Documents/repositories/oci-container-registry-storage/test/benthos-0.8.0.wrap.tgz"
 »  Unwrapping Helm chart "benthos-0.8.0.wrap.tgz"
    ✔  Helm chart uncompressed to "/tmp/chart-3226158309/at-wrap3259428596"                                                                                    
    ✔  Helm chart relocated successfully                                                                                                                       
    »  The wrap includes the following 1 images:

       localhost:5000/jeffail/benthos:4.11.0
       
    »  Pushing Images
       ✔  All images pushed successfully                                                                                                                       
    ✘  Failed to push images: failed to verify Helm chart Images.lock: Images.lock does not validate:                                                          
      Helm chart "benthos": image "localhost:5000/jeffail/benthos:4.11.0": digests do not match:
      - sha256:e2fa170d1d0c2e7bf5c89986b6e5de471fe6499f231d6f8e9c1f31c4350353b0
      + sha256:85689f09ddac1449188c9b9c50138027ff476bab81e39d4229f28eb72793314b
      Helm chart "benthos": image "localhost:5000/jeffail/benthos:4.11.0": digests do not match:
      - sha256:244e9c77b90ec18e692acf162c55060156b1d36a820d19a8606f0a98e55d4ee3
      + sha256:f794f3b17252736664244d508c6ffc10f05619cc6d09bf4c4d465d1815e0afa7
Failed to push images: failed to verify Helm chart Images.lock: Images.lock does not validate:
Helm chart "benthos": image "localhost:5000/jeffail/benthos:4.11.0": digests do not match:
- sha256:e2fa170d1d0c2e7bf5c89986b6e5de471fe6499f231d6f8e9c1f31c4350353b0
+ sha256:85689f09ddac1449188c9b9c50138027ff476bab81e39d4229f28eb72793314b
Helm chart "benthos": image "localhost:5000/jeffail/benthos:4.11.0": digests do not match:
- sha256:244e9c77b90ec18e692acf162c55060156b1d36a820d19a8606f0a98e55d4ee3
+ sha256:f794f3b17252736664244d508c6ffc10f05619cc6d09bf4c4d465d1815e0afa7
Error: plugin "dt" exited with error

Worth mentioning that I have tried running it with a redis helm chart from a OCI based registry which runs without any problems.

Example:

#!/bin/bash

helm pull oci://registry-1.docker.io/bitnamicharts/redis --version 18.1.5 --untar --untardir .

helm dt wrap ./redis
helm dt unwrap  redis-18.1.5.wrap.tgz oci://localhost:5000 --yes

Output:

Pulled: registry-1.docker.io/bitnamicharts/redis:18.1.5
Digest: sha256:892a93be15b28f4a03870c31afbaa73578c5bcf1eeb3f32d8d5649d1be9e51ee
 »  Wrapping Helm chart "./redis"
    ✔  Images.lock file written to "/home/godske/Documents/repositories/work/oci-container-registry-storage/test/redis/Images.lock"                                                              
    »  Pulling images into "/home/godske/Documents/repositories/work/oci-container-registry-storage/test/redis/images"
       ✔  All images pulled successfully                                                                                                                                                         
    ✔  Compressed into "/home/godske/Documents/repositories/work/oci-container-registry-storage/test/redis-18.1.5.wrap.tgz"                                                                      
       
 🎉  Helm chart wrapped into "/home/godske/Documents/repositories/work/oci-container-registry-storage/test/redis-18.1.5.wrap.tgz"
 »  Unwrapping Helm chart "redis-18.1.5.wrap.tgz"
    ✔  Helm chart uncompressed to "/tmp/chart-1844294399/at-wrap3814855721"                                                                                                                      
    ✔  Helm chart relocated successfully                                                                                                                                                         
    »  The wrap includes the following 4 images:

       localhost:5000/bitnami/os-shell:11-debian-11-r90
       localhost:5000/bitnami/redis-exporter:1.55.0-debian-11-r0
       localhost:5000/bitnami/redis-sentinel:7.2.1-debian-11-r26
       localhost:5000/bitnami/redis:7.2.1-debian-11-r26
       
    »  Pushing Images
       ✔  All images pushed successfully                                                                                                                                                         
       ✔  Chart "/tmp/chart-1844294399/at-wrap3814855721" lock is valid                                                                                                                          
       
    ✔  Helm chart successfully pushed                                                                                                                                                            
       
 🎉  Helm chart unwrapped successfully: You can use it now by running "helm install oci://localhost:5000/redis --generate-name"

from distribution-tooling-for-helm.

RasmusGodske avatar RasmusGodske commented on May 28, 2024

After some more tinkering and experimenting, I got a working script. This works with the non OCI-based benthos helm chart, which did not work using wrap and unwrap. This time I only made use of the "fine-grade" commands, in the order you mentioned(Much appreciated). I can however not explain why the wrap and unwrap commands did not work for me.

#!/bin/bash

helm repo add temp --force-update https://benthosdev.github.io/benthos-helm-chart/
helm repo update

helm pull temp/benthos --version 0.8.0 --untar --untardir .

images_override=$(cat <<-END
- name: benthos
  image: docker.io/jeffail/benthos:4.11.0
END
)

yq -i e ".annotations.images |= \"$images_override\"" ./benthos/Chart.yaml

helm dt images lock ./benthos

helm dt images verify ./benthos

helm dt images pull ./benthos

helm dt charts relocate ./benthos oci://xragoregistryforgiggles.azurecr.io

helm dt images push ./benthos

helm dt images lock ./benthos

helm dt images verify ./benthos

helm package ./benthos

helm push ./benthos-0.8.0.tgz oci://xragoregistryforgiggles.azurecr.io

from distribution-tooling-for-helm.

mpermar avatar mpermar commented on May 28, 2024

I will try to replicate once back from holidays. I wonder if it might be related to be using a local registry. Have you tried the wrap/unwrap with azurecr?

from distribution-tooling-for-helm.

RasmusGodske avatar RasmusGodske commented on May 28, 2024

No rush this can wait, holidays are important.

I have tried using ACR, however with the same result.

from distribution-tooling-for-helm.

mpermar avatar mpermar commented on May 28, 2024

@RasmusGodske I could reproduce the issue with the exising dt release 0.2.3. Then decided to build from source to debug it and it turned out that I was able to successfully wrap/unwrap the benthos Helm chart. So my theory is that you were hitting a bug that somehow @juamedgod had already fixed but that we hadn't release yet.

Can you try updating to the latest release (v0.3.0) with helm plugin update dt and give it a go again?

By the way, in this release, we have changed how the wraps are structured internally as the way they were it could cause some side-effects with certain Helm charts. This is an improvement but breaks backwards compatibility with older wraps. So you will have to wrap and unwrap for it to work.

Let me know if doing an upgrade fixes it 🙏

from distribution-tooling-for-helm.

RasmusGodske avatar RasmusGodske commented on May 28, 2024

@mpermar I tried to update to the v0.3.0 however the error still persists.

The following script:

#!/bin/bash

helm dt version

helm repo add temp --force-update https://benthosdev.github.io/benthos-helm-chart/
helm repo update

helm pull temp/benthos --version 0.8.0 --untar --untardir .

# This helm chart isn't annotated by default, so I'm manually doing this here
images_override=$(cat <<-END
- name: benthos
  image: docker.io/jeffail/benthos:4.11.0
END
)
yq -i e ".annotations.images |= \"$images_override\"" ./benthos/Chart.yaml

helm dt wrap  ./benthos
helm dt unwrap  benthos-0.8.0.wrap.tgz oci://localhost:5000 --yes

Gives me the following error:

./scripts/test.sh    
Distribution Tooling for Helm v0.3.0
Built on: 2023-12-11T17:44:44Z

"temp" has been added to your repositories
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "benthos" chart repository
...Successfully got an update from the "temp" chart repository
...Successfully got an update from the "stable" chart repository
...Successfully got an update from the "openebs-jiva" chart repository
...Successfully got an update from the "vmware-tanzu" chart repository
...Successfully got an update from the "traefik" chart repository
...Successfully got an update from the "grafana" chart repository
...Successfully got an update from the "bitnami" chart repository
Update Complete. ⎈Happy Helming!⎈
 »  Wrapping Helm chart "./benthos"
    ✔  Images.lock file written to "/tmp/chart-3552518746/wrap/chart/Images.lock"                                                                
    »  Pulling images into "/tmp/chart-3552518746/wrap/chart/images"
       ✔  All images pulled successfully                                                                                                         
    ✔  Compressed into "/home/godske/Documents/repositories/work/oci-container-registry-storage/benthos-0.8.0.wrap.tgz"                          
       
 🎉  Helm chart wrapped into "/home/godske/Documents/repositories/work/oci-container-registry-storage/benthos-0.8.0.wrap.tgz"
 »  Unwrapping Helm chart "benthos-0.8.0.wrap.tgz"
    ✔  Helm chart uncompressed to "/tmp/chart-2496835767/at-wrap2084825586"                                                                      
    ✔  Helm chart relocated successfully                                                                                                         
    »  The wrap includes the following 1 images:

       localhost:5000/jeffail/benthos:4.11.0
       
    »  Pushing Images
       ✔  All images pushed successfully                                                                                                         
    ✘  Failed to push images: failed to verify Helm chart Images.lock: Images.lock does not validate:                                            
      Helm chart "benthos": image "localhost:5000/jeffail/benthos:4.11.0": digests do not match:
      - sha256:e2fa170d1d0c2e7bf5c89986b6e5de471fe6499f231d6f8e9c1f31c4350353b0
      + sha256:85689f09ddac1449188c9b9c50138027ff476bab81e39d4229f28eb72793314b
      Helm chart "benthos": image "localhost:5000/jeffail/benthos:4.11.0": digests do not match:
      - sha256:244e9c77b90ec18e692acf162c55060156b1d36a820d19a8606f0a98e55d4ee3
      + sha256:f794f3b17252736664244d508c6ffc10f05619cc6d09bf4c4d465d1815e0afa7
Failed to push images: failed to verify Helm chart Images.lock: Images.lock does not validate:
Helm chart "benthos": image "localhost:5000/jeffail/benthos:4.11.0": digests do not match:
- sha256:e2fa170d1d0c2e7bf5c89986b6e5de471fe6499f231d6f8e9c1f31c4350353b0
+ sha256:85689f09ddac1449188c9b9c50138027ff476bab81e39d4229f28eb72793314b
Helm chart "benthos": image "localhost:5000/jeffail/benthos:4.11.0": digests do not match:
- sha256:244e9c77b90ec18e692acf162c55060156b1d36a820d19a8606f0a98e55d4ee3
+ sha256:f794f3b17252736664244d508c6ffc10f05619cc6d09bf4c4d465d1815e0afa7
Error: plugin "dt" exited with error

from distribution-tooling-for-helm.

RasmusGodske avatar RasmusGodske commented on May 28, 2024

That's an unexpected error, I'm happy that you manage to find it. I'm off this project soon, however feel free to contact/mention me if you need further testing. I'll be more than happy to help in my spare time.

from distribution-tooling-for-helm.

juamedgod avatar juamedgod commented on May 28, 2024

@RasmusGodske We just merged #30, which should fix your use case. In case you want to give it a try.

from distribution-tooling-for-helm.

Related Issues (3)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.