Comments (19)
liggitt
commented on June 29, 2024
/kind bug
from kubernetes.
MadhavJivrajani
commented on June 29, 2024
/triage accepted
/sig architecture testing
/area code-organization
from kubernetes.
MadhavJivrajani
commented on June 29, 2024
go: download go1.22.3 for linux/amd64: toolchain not available
This is an error I've seen when we use go 1.x and not go 1.x.y as the go directive, some instances: golang/go#62278, dependabot/dependabot-core#7895
Which is why what you said here makes sense: #124922 (comment). Once the actual bump happens, GOTOOLCHAIN env variable that will propagate into the container will match with the toolchain that exists already and go will not try and download it again leading to the error (similar to the issues linked).
This seems to also be the reason the verify job failed here: #122410
When I had started the testing, master was still not on go1.22 leading to the kube-build go toolchain mismatch.
from kubernetes.
MadhavJivrajani
commented on June 29, 2024
I wonder if it would be a good idea to substitute the value of .go-version directly in the kube-build script
But I guess the long term solution would be to incorporate the toolchain directive into our go mods and the tooling.
from kubernetes.
BenTheElder
commented on June 29, 2024
But I guess the long term solution would be to incorporate the toolchain directive into our go mods and the tooling.
This also has the benefit of enabling IDEs to respect it without our makefiles (though your builds still won't match us if you just go build), but we gain additional complexity making these align and we need to continue to respect .go-version / envs properly for now because downstream users depend on this "API"
from kubernetes.
BenTheElder
commented on June 29, 2024
Even if we switch to toolchain in go.mod we should probably make the env / .go-version work properly for at least one minor while putting a release note that downstreams must switch. (I'm sure an argument could be made for longer).
It would probably be preferable to keep them working permanently because nobody wants to carry go.mod patches but it is common to need to force a different go version for security patching etc.
from kubernetes.
BenTheElder
commented on June 29, 2024
Do we know yet if this applies to all branches?
from kubernetes.
liggitt
commented on June 29, 2024
Do we know yet if this applies to all branches?
yes, it applies to all branches (we saw this when trying to just bump .go-version to 1.22.x on release branches)
from kubernetes.
BenTheElder
commented on June 29, 2024
/assign
from kubernetes.
BenTheElder
commented on June 29, 2024
So, interestingly if I do GOTOOLCHAIN=go1.21.9 make verify WHAT=codegen, we get the same failure.
go: downloading go1.21.9 (linux/amd64)
go: download go1.21.9 for linux/amd64: toolchain not available
Seems like something is broken with GOTOOLCHAIN support in this go install or go, rather than our scripts?
from kubernetes.
BenTheElder
commented on June 29, 2024
Maybe something with the kube-build image.
from kubernetes.
BenTheElder
commented on June 29, 2024
Normal go image:
$ docker run --rm -e GOTOOLCHAIN=go1.22.3 golang:1.21.10-bullseye go version
Unable to find image 'golang:1.21.10-bullseye' locally
1.21.10-bullseye: Pulling from library/golang
3d53ef4019fc: Pull complete
08f0bf643eb6: Pull complete
6b037c2b46ab: Pull complete
c9c00c8cd734: Pull complete
a4de642a3616: Pull complete
785d32889ff9: Pull complete
4f4fb700ef54: Pull complete
Digest: sha256:6815d296f20c3cb3afa67bf5dc862a831b6080f5dd8b5b423bb5f862539299de
Status: Downloaded newer image for golang:1.21.10-bullseye
go: downloading go1.22.3 (linux/amd64)
go version go1.22.3 linux/amd64
kube-cross:
docker run --rm -e GOTOOLCHAIN=go1.22.3 registry.k8s.io/build-image/kube-cross:v1.28.0-go1.21.10-bullseye.0 go version
Unable to find image 'registry.k8s.io/build-image/kube-cross:v1.28.0-go1.21.10-bullseye.0' locally
v1.28.0-go1.21.10-bullseye.0: Pulling from build-image/kube-cross
646e886fa3cf: Pull complete
c5a360c5f105: Pull complete
5cbeb8ef1d90: Pull complete
f3054eaa1068: Pull complete
fd19ff5befab: Pull complete
029d7cfe87db: Pull complete
4f4fb700ef54: Pull complete
ab038281f51a: Pull complete
9bc87ef1bf76: Pull complete
eb4d85f62c54: Pull complete
00a4d6140bd2: Pull complete
30d8f173cf20: Pull complete
1903caad7a82: Pull complete
e7f153876917: Pull complete
Digest: sha256:1996e852695eb1d3b48bda278a4584317d5d42eaef226899a331ea37394b6952
Status: Downloaded newer image for registry.k8s.io/build-image/kube-cross:v1.28.0-go1.21.10-bullseye.0
go: downloading go1.22.3 (linux/amd64)
go version go1.22.3 linux/amd64
kube-build:
$ docker run --rm -e GOTOOLCHAIN=go1.22.3 kube-build:build-9a4d0b6c27-5-v1.28.0-go1.21.10-bullseye.0 go version
go: downloading go1.22.3 (linux/amd64)
go version go1.22.3 linux/amd64
but under verify-codegen with GOTOOLCHAIN=go1.22.3 make verify WHAT=codegen:
...
+++ [0520 21:58:12] Building Docker image kube-build:build-17572b7f79-5-v1.28.0-go1.21.10-bullseye.0
+++ [0520 21:58:22] Creating data container kube-build-data-17572b7f79-5-v1.28.0-go1.21.10-bullseye.0
+++ [0520 21:58:23] Syncing sources to container
+++ [0520 21:58:31] Output from this container will be rsynced out upon completion. Set KUBE_RUN_COPY_OUTPUT=n to disable.
+++ [0520 21:58:31] Running build command...
go: downloading go1.22.3 (linux/amd64)
go: download go1.22.3 for linux/amd64: toolchain not available
...
.... hmm
from kubernetes.
BenTheElder
commented on June 29, 2024
narrowed it down to hack/lib/init.sh:
$ GOTOOLCHAIN=go1.22.3 build/run.sh bash -c 'source hack/lib/init.sh && go version && echo $GOTOOLCHAIN'
+++ [0520 22:09:21] Verifying Prerequisites....
+++ [0520 22:09:22] Building Docker image kube-build:build-b9c550bd56-5-v1.28.0-go1.21.10-bullseye.0
+++ [0520 22:09:31] Syncing sources to container
+++ [0520 22:09:33] Output from this container will be rsynced out upon completion. Set KUBE_RUN_COPY_OUTPUT=n to disable.
+++ [0520 22:09:33] Running build command...
go version go1.21.10 linux/amd64
go1.22.3
+++ [0520 22:09:34] Syncing out of container
$ GOTOOLCHAIN=go1.22.3 build/run.sh bash -c 'go version && echo $GOTOOLCHAIN'
+++ [0520 22:09:54] Verifying Prerequisites....
+++ [0520 22:09:54] Building Docker image kube-build:build-b9c550bd56-5-v1.28.0-go1.21.10-bullseye.0
+++ [0520 22:10:04] Syncing sources to container
+++ [0520 22:10:06] Output from this container will be rsynced out upon completion. Set KUBE_RUN_COPY_OUTPUT=n to disable.
+++ [0520 22:10:06] Running build command...
go: downloading go1.22.3 (linux/amd64)
go version go1.22.3 linux/amd64
go1.22.3
+++ [0520 22:10:13] Syncing out of container
from kubernetes.
BenTheElder
commented on June 29, 2024
Talked to @liggitt about this, who found GO111MODULE=off suspect, setting this will seemingly cause go version to ignore GOTOOLCHAIN.
even @ 765e7ef (just merged to master an hour ago) we can see:
$ GOTOOLCHAIN=go1.22.1 build/run.sh bash -c 'GO111MODULE=off go version && echo "GOTOOLCHAIN=$GOTOOLCHAIN"'
+++ [0520 23:39:30] Verifying Prerequisites....
+++ [0520 23:39:31] Building Docker image kube-build:build-3ad27617b0-5-v1.31.0-go1.22.3-bullseye.0
+++ [0520 23:41:10] Deleting image kube-build:build-3ad27617b0-5-v1.30.0-go1.22.2-bullseye.0
+++ [0520 23:41:12] Creating data container kube-build-data-3ad27617b0-5-v1.31.0-go1.22.3-bullseye.0
+++ [0520 23:41:23] Syncing sources to container
+++ [0520 23:41:36] Output from this container will be rsynced out upon completion. Set KUBE_RUN_COPY_OUTPUT=n to disable.
+++ [0520 23:41:36] Running build command...
go version go1.22.3 linux/amd64
GOTOOLCHAIN=go1.22.1
+++ [0520 23:41:37] Syncing out of container
$ GOTOOLCHAIN=go1.22.1 build/run.sh bash -c 'GO111MODULE=on go version && echo "GOTOOLCHAIN=$GOTOOLCHAIN"'
+++ [0520 23:41:54] Verifying Prerequisites....
+++ [0520 23:41:55] Building Docker image kube-build:build-3ad27617b0-5-v1.31.0-go1.22.3-bullseye.0
+++ [0520 23:42:04] Syncing sources to container
+++ [0520 23:42:06] Output from this container will be rsynced out upon completion. Set KUBE_RUN_COPY_OUTPUT=n to disable.
+++ [0520 23:42:06] Running build command...
go: downloading go1.22.1 (linux/amd64)
go version go1.22.1 linux/amd64
GOTOOLCHAIN=go1.22.1
+++ [0520 23:42:14] Syncing out of container
So this is probably related.
On master GOTOOLCHAIN=go1.22.1 make verify WHAT=codege is not broken, presumably because we don't set GO111MODULE=off anymore after @thockin's WORKSPACES work.
I think we can say this is already fixed @ HEAD and is just poorly behaved on the older branches.
We should probably hack in a gimme fallback for this case on older branches while doing our best to respect GOTOOLCHAIN after introducing it (IIRC we put a release note about using GOTOOLCHAIN to control go versions). We already use gimme if go is not available or too old.
from kubernetes.
BenTheElder
commented on June 29, 2024
Might not be worth prioritizing a patch given this is only relevant if you're:
- trying to override the go version
- not overriding the build container image to match
- running on an old release branch
This should be fine in 1.30+ *
*well, there's a smaller bug in #125010
from kubernetes.
MadhavJivrajani
commented on June 29, 2024
Thanks for digging here @BenTheElder!
from kubernetes.
MadhavJivrajani
commented on June 29, 2024
Hmm, this is slightly confusing, on darwin:
❯ go version
go version go1.22.2 darwin/amd64
and with GO111MODULE=off
❯ GOTOOLCHAIN=go1.21.9 GO111MODULE=off go version
go version go1.21.0 darwin/amd64
I'm not sure why its go1.21.0, I don't have that toolchain present locally either...
with GO111MODULE=on:
❯ GOTOOLCHAIN=go1.21.9 GO111MODULE=on go version
go version go1.21.9 darwin/amd64
from kubernetes.
MadhavJivrajani
commented on June 29, 2024
Ah,
I see where the effect of GO111MODULE on GOTOOLCHAIN comes from: https://github.com/golang/go/blob/dddf0ae40fa0c1223aba191d73a44425a08e1035/src/cmd/go/internal/toolchain/select.go#L93
from kubernetes.
BenTheElder
commented on June 29, 2024
Right, so for the newer branches where we have modules on and workspaces adopted, that just leaves #125010
We could try to mitigate this in the older branches ... but it's a fairly niche issue and the workspaces change is too large to backport so we'll have to write something custom for the old branches ... and you can already work around it by just overriding the build image to have the right go versoin.
from kubernetes.
Related Issues (20)
- Node Labeling node.kubernetes.io/out-of-service Taint Label Delay HOT 2
- [FG:InPlacePodVerticalScaling] e2e test does not verify resource update in pod status HOT 3
- cronjob schedule with multiple conditions not working - conflict between day (week) and day (month) HOT 5
- NetPol block self pod trafic using an svc and not direct call HOT 12
- kube-apiserver logs watch requests before they end in 1.30 HOT 9
- Node Lifecycle Controller does not mark pods not ready when node becomes Ready=False HOT 8
- endpoints cannot be changed from notReadyAddresses to addresses HOT 8
- Enhancement: Add vTPM Configuration Fields for Enhanced Container Security HOT 3
- 'kubectl delete istag/$ISTAG --dry-run=server' is unexpectedly deleting the object from the server HOT 5
- [FG:InPlacePodVerticalScaling] resources in pod status are never updated if EventedPLEG is enabled HOT 2
- [Flaking test] ci-kubernetes-e2e-gci-gce.Overall HOT 4
- `kubernetes.io/legacy-token-last-used` label being added to long lived service token secrets HOT 2
- The endpoint status does not update when the pod state changes rapidly. HOT 8
- Pod with exitCode 137， The reason has nothing to do with resources。 HOT 2
- Failure cluster [9afae275...] HOT 2
- finish DRA for 1.31 HOT 4
- [Failing Test] ci-kubernetes-cloud-provider-kind-conformance-parallel-ipv6 (client rate limiter error) HOT 2
- TypeMeta is empty in Type client Apply and Patch responses HOT 3
- Job API: Relax validation enforcing Pod Failure Policy is only compatible with pod restart policy of "Never" HOT 7
- invalid memory address or nil pointer dereference" in wait.JitterUntil HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kubernetes.