Git Product home page Git Product logo

Comments (8)

crazy-max avatar crazy-max commented on June 11, 2024 3

Oh my bad didn't catch files: tests/build.hcl in your workflow.

So looking at your workflow and bake definition, you're using the same cache scope for all your targets: https://github.com/leolivier/Mailu/blob/8bdf84bc85fd0f04941568e55c8ca483458f3c84/tests/build.hcl#L15-L35

group "default" {
  targets = [
    "docs",
    "setup",

    "admin",
    "antispam",
    "front",
    "imap",
    "smtp",

    "rainloop",
    "roundcube",

    "antivirus",
    "fetchmail",
    "resolver",
    "traefik-certdumper",
    "webdav"
  ]
}

Which will override each other cache and that's why you have some cache missed on next run.

Suggest to not use grouped targets and instead use a GHA matrix and create cache scoped for each of them. Pretty much like this:

  build:
    name: bake mailu
    runs-on: ubuntu-latest
    needs: setup
    strategy:
      fail-fast: false
      matrix:
        target:
          - docs
          - setup
          - admin
          - antispam
    steps:
      - name: recover setup environment
        shell: bash
        run: |
          echo "BRANCH=${{needs.setup.outputs.BRANCH}}" >> $GITHUB_ENV
          echo "MAILU_VERSION=${{needs.setup.outputs.MAILU_VERSION}}" >> $GITHUB_ENV
          echo "PINNED_MAILU_VERSION=${{needs.setup.outputs.PINNED_MAILU_VERSION}}" >> $GITHUB_ENV
          echo "DOCKER_ORG=${{needs.setup.outputs.DOCKER_ORG}}" >> $GITHUB_ENV
      - uses: actions/checkout@v3
        with:
          # fetch-depth 0 is required to also retrieve all tags.
          fetch-depth: 0
      - name: Set up Docker Buildx
        id: buildx
        uses: docker/setup-buildx-action@v2
      # not using docker_org anymore, 
      - name: Login to DockerHub on release
        uses: docker/login-action@v2
        if: ${{ github.event_name == 'release' }}
        with:
          username: ${{ secrets.Docker_Login }}
          password: ${{ secrets.Docker_Password }}
      - name: Login to GHCR if not release
        uses: docker/login-action@v2
        if: ${{ github.event_name != 'release' }}
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - name: Create version file
        shell: bash
        run: |
          VERSION_FILE="Source: https://github.com/$GITHUB_REPOSITORY Branch: $GITHUB_REF_NAME Tag: $PINNED_MAILU_VERSION Hash: $GITHUB_SHA"
          echo "VERSION_FILE=$VERSION_FILE"
          echo "VERSION_FILE=$VERSION_FILE" >> $GITHUB_ENV
      # needed for gha cache?
      - uses: crazy-max/ghaction-github-runtime@v2
      # depending on login above, will push to GHCR for testing or to Docker hub for releasing
      # Build only arm64 version for now
      - name: Build and push
        uses: docker/[email protected]
        with:
          files: tests/build.hcl
          targets: ${{ matrix.target }}
          push: 'true'
          set: |
            *.args.VERSION=${{ env.VERSION_FILE }}
            *.args.pinned_version=${{ env.VERSION_FILE }}
            *.cache-from=type=gha,scope=build-${{ matrix.target }}
            *.cache-to=type=gha,scope=build-${{ matrix.target }},mode=max
            *.platform=linux/amd64

In strategy.matrix.target, you want each target from default group that you want to build and also in targets, and set input, set the relevant target from the matrix and the cache scoped as well so cache don't collide with each other.

Also instead of defining all targets one by one in your workflow, you can create a dynamic matrix from JSON output like this https://github.com/tonistiigi/xx/pull/68/files#diff-0838c4d4b9b5bbd3feb6c96405933e36901513c0112f0368a07ba1613b327f7aR11-R36

jobs:
  targets:
    runs-on: ubuntu-latest
    outputs:
      matrix: ${{ steps.targets.outputs.matrix }}
    steps:
      -
        name: Checkout
        uses: actions/checkout@v3
      -
        name: Create matrix
        id: targets
        run: |
          echo ::set-output name=matrix::$(docker buildx bake --print | jq -cr '.group.default.targets')
      -
        name: Show matrix
        run: |
          echo ${{ steps.targets.outputs.matrix }}
  build:
    runs-on: ubuntu-latest
    needs:
      - targets
    strategy:
      fail-fast: false
      matrix:
        target: ${{ fromJson(needs.targets.outputs.matrix) }}

from bake-action.

leolivier avatar leolivier commented on June 11, 2024 1

You rock @crazy-max ! Just missing -f ./tests/build.hcl in the docker buildx bake --print line to take my own hcl into account and it now works like a charm!!!

from bake-action.

fubhy avatar fubhy commented on June 11, 2024

We are seeing the same behavior. Is there a problem with caching COPY --from instructions?

from bake-action.

crazy-max avatar crazy-max commented on June 11, 2024

@leolivier Link to your Dockerfile and docker-bake.hcl definition? I don't see them in your repo root.

from bake-action.

leolivier avatar leolivier commented on June 11, 2024

@crazy-max the hcl file in the tests dir: https://github.com/leolivier/Mailu/blob/master/tests/build.hcl as pointed in the action.
There are multiple Dockerfiles actually eg:

from bake-action.

leolivier avatar leolivier commented on June 11, 2024

Thanks a lot for your help and precious advises @crazy-max , I'll try that and keep you updated!

from bake-action.

fubhy avatar fubhy commented on June 11, 2024

Also going to try that to see if it fixes our problem too. Our setup is really similiar so I suspect this is the same issue. Thanks for the quick response!

from bake-action.

csadorf avatar csadorf commented on June 11, 2024

How would one enable caching when targets depend on each other? I understand that in this case, using a matrix would not work, would it?

from bake-action.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.