Comments (8)
Oh my bad didn't catch files: tests/build.hcl
in your workflow.
So looking at your workflow and bake definition, you're using the same cache scope for all your targets: https://github.com/leolivier/Mailu/blob/8bdf84bc85fd0f04941568e55c8ca483458f3c84/tests/build.hcl#L15-L35
group "default" {
targets = [
"docs",
"setup",
"admin",
"antispam",
"front",
"imap",
"smtp",
"rainloop",
"roundcube",
"antivirus",
"fetchmail",
"resolver",
"traefik-certdumper",
"webdav"
]
}
Which will override each other cache and that's why you have some cache missed on next run.
Suggest to not use grouped targets and instead use a GHA matrix and create cache scoped for each of them. Pretty much like this:
build:
name: bake mailu
runs-on: ubuntu-latest
needs: setup
strategy:
fail-fast: false
matrix:
target:
- docs
- setup
- admin
- antispam
steps:
- name: recover setup environment
shell: bash
run: |
echo "BRANCH=${{needs.setup.outputs.BRANCH}}" >> $GITHUB_ENV
echo "MAILU_VERSION=${{needs.setup.outputs.MAILU_VERSION}}" >> $GITHUB_ENV
echo "PINNED_MAILU_VERSION=${{needs.setup.outputs.PINNED_MAILU_VERSION}}" >> $GITHUB_ENV
echo "DOCKER_ORG=${{needs.setup.outputs.DOCKER_ORG}}" >> $GITHUB_ENV
- uses: actions/checkout@v3
with:
# fetch-depth 0 is required to also retrieve all tags.
fetch-depth: 0
- name: Set up Docker Buildx
id: buildx
uses: docker/setup-buildx-action@v2
# not using docker_org anymore,
- name: Login to DockerHub on release
uses: docker/login-action@v2
if: ${{ github.event_name == 'release' }}
with:
username: ${{ secrets.Docker_Login }}
password: ${{ secrets.Docker_Password }}
- name: Login to GHCR if not release
uses: docker/login-action@v2
if: ${{ github.event_name != 'release' }}
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Create version file
shell: bash
run: |
VERSION_FILE="Source: https://github.com/$GITHUB_REPOSITORY Branch: $GITHUB_REF_NAME Tag: $PINNED_MAILU_VERSION Hash: $GITHUB_SHA"
echo "VERSION_FILE=$VERSION_FILE"
echo "VERSION_FILE=$VERSION_FILE" >> $GITHUB_ENV
# needed for gha cache?
- uses: crazy-max/ghaction-github-runtime@v2
# depending on login above, will push to GHCR for testing or to Docker hub for releasing
# Build only arm64 version for now
- name: Build and push
uses: docker/[email protected]
with:
files: tests/build.hcl
targets: ${{ matrix.target }}
push: 'true'
set: |
*.args.VERSION=${{ env.VERSION_FILE }}
*.args.pinned_version=${{ env.VERSION_FILE }}
*.cache-from=type=gha,scope=build-${{ matrix.target }}
*.cache-to=type=gha,scope=build-${{ matrix.target }},mode=max
*.platform=linux/amd64
In strategy.matrix.target
, you want each target from default
group that you want to build and also in targets
, and set
input, set the relevant target from the matrix and the cache scoped as well so cache don't collide with each other.
Also instead of defining all targets one by one in your workflow, you can create a dynamic matrix from JSON output like this https://github.com/tonistiigi/xx/pull/68/files#diff-0838c4d4b9b5bbd3feb6c96405933e36901513c0112f0368a07ba1613b327f7aR11-R36
jobs:
targets:
runs-on: ubuntu-latest
outputs:
matrix: ${{ steps.targets.outputs.matrix }}
steps:
-
name: Checkout
uses: actions/checkout@v3
-
name: Create matrix
id: targets
run: |
echo ::set-output name=matrix::$(docker buildx bake --print | jq -cr '.group.default.targets')
-
name: Show matrix
run: |
echo ${{ steps.targets.outputs.matrix }}
build:
runs-on: ubuntu-latest
needs:
- targets
strategy:
fail-fast: false
matrix:
target: ${{ fromJson(needs.targets.outputs.matrix) }}
from bake-action.
You rock @crazy-max ! Just missing -f ./tests/build.hcl in the docker buildx bake --print line to take my own hcl into account and it now works like a charm!!!
from bake-action.
We are seeing the same behavior. Is there a problem with caching COPY --from
instructions?
from bake-action.
@leolivier Link to your Dockerfile and docker-bake.hcl
definition? I don't see them in your repo root.
from bake-action.
@crazy-max the hcl file in the tests dir: https://github.com/leolivier/Mailu/blob/master/tests/build.hcl as pointed in the action.
There are multiple Dockerfiles actually eg:
- https://github.com/leolivier/Mailu/blob/master/core/admin/Dockerfile
- https://github.com/leolivier/Mailu/blob/master/core/nginx/Dockerfile
- https://github.com/leolivier/Mailu/blob/master/core/dovecot/Dockerfile
- https://github.com/leolivier/Mailu/blob/master/core/postfix/Dockerfile
- https://github.com/leolivier/Mailu/blob/master/core/rspamd/Dockerfile
- https://github.com/leolivier/Mailu/blob/master/core/none/Dockerfile
- and several others you can easily find looking at the build.hcl file
from bake-action.
Thanks a lot for your help and precious advises @crazy-max , I'll try that and keep you updated!
from bake-action.
Also going to try that to see if it fixes our problem too. Our setup is really similiar so I suspect this is the same issue. Thanks for the quick response!
from bake-action.
How would one enable caching when targets depend on each other? I understand that in this case, using a matrix would not work, would it?
from bake-action.
Related Issues (20)
- Builds not using local cache HOT 1
- feature: perform push iff all targets succeed HOT 4
- Q: Proper bake usage withh groups HOT 5
- Additional Metadata Output specifying images generated HOT 1
- v3 breaks build for *.output=type=docker,dest HOT 3
- Action does not respect the Builder mirror configuration HOT 2
- Too large metadata cause " Argument list too long" error HOT 3
- Everything after # gets eaten in the `set` input, with no way to escape it since v3 HOT 3
- Closed connection while exporting cache HOT 1
- Multi level image builds - only base image is cached HOT 5
- `docker buildx bake --metadata-file` as action output
- Building on multiple native nodes HOT 6
- Error: Blocks are not allowed here HOT 8
- Feature Request: Ability to specify working-directory HOT 3
- How to access the bake-action cached image in subsequent steps? HOT 7
- How to access the bake-action cached image in subsequent jobs? HOT 1
- Caching images for use in docker compose between jobs and runs. HOT 3
- Cache not being hit when using TAG workflows HOT 5
- Feat: output grouped digest HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from bake-action.