Comments (7)
Thanks for your report.
Yes, that would fail, and we need to look into whether/how that should be handled.
OTOH, we should already be asking for the right scope; this happens only on a secondary authentication path. I wonder why the primary path is failing in the first place. Is it possible that your credentials just don’t allow the push at all? Or maybe that your current credentials don’t allow pulling from repos/repo2
(but you did pull from that location with some other credentials previously, triggering a reuse attempt)?
And, those are logged errors on that secondary authentication path, but that path does not actually directly cause anything to fail. I’d expect an error message recording the actual reason to fail to exist (in this case, probably some permission denied with that the secondary authentication was trying to recover from).
Please collect a full podman --log-level=debug push …
output.
from image.
Thanks for your report.
Yes, that would fail, and we need to look into whether/how that should be handled.
OTOH, we should already be asking for the right scope; this happens only on a secondary authentication path. I wonder why the primary path is failing in the first place. Is it possible that your credentials just don’t allow the push at all? Or maybe that your current credentials don’t allow pulling from
repos/repo2
(but you did pull from that location with some other credentials previously, triggering a reuse attempt)?
Thank you for your insight here. After further investigation it turns out that the upgrade from gitlab 15 to 16 broke "podman push" in this deployment! We have noticed a gitlab 15 "podman push" to work as expected for the same container.
And, those are logged errors on that secondary authentication path
, but that path does not actually directly cause anything to fail. I’d expect an error message recording the actual reason to fail to exist (in this case, probably some permission denied with that the secondary authentication was trying to recover from).
Please collect a full
podman --log-level=debug push …
output.
Certainly, please find the log attached: podman-push-gitlab-16.log
from image.
time="2023-06-02T07:19:48Z" level=debug msg="Detected insufficient_scope error, will retry request with updated scope" time="2023-06-02T07:19:48Z" level=debug msg="GET https://gitlab.foo.bar/jwt/auth?scope=repository%3Arepos%2Frepo1%3Apull%2Cpush&scope=repository%3Arepos%2Frepo1%3Apull%2Cpush&service=container_registry"
So we are getting an insufficient_scope
error. In response for that we ask for two scopes: 1. the one we hard-code, and we already got a token for, and 2. the one the registry is complaining about. The URL above shows that — but the two are identical!
Assuming this isn’t an artifact of an inaccurate log redaction, that sounds like something that needs to be diagnosed server-side. Why is the registry reporting “insufficient scope” for a scope that has already been granted?
Error: writing blob: initiating layer upload to /v2/repos/repo1/blobs/uploads/ in registry.gitlab.foo.bar: requested access to the resource is denied
This is the actual upload-terminating failure.
And more details in the error log:
time="2023-06-02T07:19:48Z" level=debug msg="Error initiating layer upload, response http.Response{Status:\"401 Unauthorized\", StatusCode:401, Proto:\"HTTP/1.1\", ProtoMajor:1, ProtoMinor:1, Header:http.Header{\"Connection\":[]string{\"keep-alive\"}, \"Content-Length\":[]string{\"244\"}, \"Content-Type\":[]string{\"application/json\"}, \"Date\":[]string{\"Fri, 02 Jun 2023 07:19:55 GMT\"}, \"Docker-Distribution-Api-Version\":[]string{\"registry/2.0\"}, \"Server\":[]string{\"nginx\"}, \"Www-Authenticate\":[]string{\"Bearer realm=\\\"https://gitlab.foo.bar/jwt/auth\\\",service=\\\"container_registry\\\",scope=\\\"repository:repos/repo1:pull,push\\\",error=\\\"insufficient_scope\\\"\"}, \"X-Content-Type-Options\":[]string{\"nosniff\"}}, Body:(*http.bodyEOFSignal)(0xc000728b80), ContentLength:244, TransferEncoding:[]string(nil), Close:false, Uncompressed:false, Trailer:http.Header(nil), Request:(*http.Request)(0xc000820000), TLS:(*tls.ConnectionState)(0xc000656160)}"
And again it is complaining about a scope that was granted in every earlier /jwt/auth
call; at least I can’t find anything to suggest a /jwt/auth
call has ever failed.
from image.
time="2023-06-02T07:19:48Z" level=debug msg="Detected insufficient_scope error, will retry request with updated scope" time="2023-06-02T07:19:48Z" level=debug msg="GET https://gitlab.foo.bar/jwt/auth?scope=repository%3Arepos%2Frepo1%3Apull%2Cpush&scope=repository%3Arepos%2Frepo1%3Apull%2Cpush&service=container_registry"
So we are getting an
insufficient_scope
error. In response for that we ask for two scopes: 1. the one we hard-code, and we already got a token for, and 2. the one the registry is complaining about. The URL above shows that — but the two are identical!Assuming this isn’t an artifact of an inaccurate log redaction, that sounds like something that needs to be diagnosed server-side. Why is the registry reporting “insufficient scope” for a scope that has already been granted?
Thank you for your help, I have double checked the non-redacted log and the requested repos are indeed the same! I have opened an issue with gitlab registry upstream here: https://gitlab.com/gitlab-org/gitlab/-/issues/414402
from image.
@filippog reading https://gitlab.com/gitlab-org/gitlab/-/issues/414402 suggests that this was caused by a missing podman login
; and in that case the non-debug-log
Error: writing blob: initiating layer upload to /v2/repos/repo1/blobs/uploads/ in registry.gitlab.foo.bar: requested access to the resource is denied
seems basically appropriate.
Is there anything that needs to be fixed in c/image?
from image.
Hello @mtrmac ! That's indeed the case, the root cause was a missing podman login
. Thank you for your help here, I don't think there's anything else actionable here, closing!
from image.
Thanks for the confirmation.
from image.
Related Issues (20)
- Docker client code can no longer talk to the latest verson of the docker daemon 25.0.0 HOT 5
- Allow empty OCI configs for artifacts HOT 9
- policy.json overwrite not honouring $XDG_CONFIG_HOME HOT 3
- Podman cannot pull image from local registry HOT 4
- copy.Options.EnsureCompressionVariantsExist doesn’t detect existing variants with zstd:chunked
- support multiple sigstore keys HOT 6
- How can I copy from a tar file stream HOT 7
- "slices" module only in go 1.21 HOT 1
- Cannot pull sigstore signed image with podman HOT 4
- Error inspecting local manifest-lists HOT 6
- platform.WantedPlatforms is noisy on macOS HOT 7
- Incorrect syntax highlighting in containers-transports.5
- Why do we get the whole image when inspect with docker daemon? HOT 2
- Support sigstore BYO PKI verification HOT 1
- Support more arbitrary credential helper executable names? HOT 4
- OCI image index loose the artifactType property on copy HOT 4
- zstd:chunked and layer encryption don’t make sense together HOT 3
- How to change the policy.json default path? HOT 1
- Mirrors are ignored when registry is blocked HOT 11
- Support Witnessing Sigstore Signing with a Timestamp Authority Server HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from image.