fast_pusher is racy about containerregistry HOT 7 CLOSED

/cc @dekkagaijin do you know who to blame?

from containerregistry.

Not sure, I'm guessing that there's contention between gcloud invocations to generate access tokens. There isn't much to docker-credential-gcloud or gcloud auth docker-helper itself.

@ensonic the command you'd want is echo -n "https://eu.gcr.io" | docker-credential-gcloud get

from containerregistry.

@dekkagaijin: sorry escaping issues with markdown: the command we run is

docker-credential-gcloud get <<<https://eu.gcr.io

The best way we have so far is to take the output for docker-credential-gcloud, store it, then inject a script called docker-credential-gcloud that simply echos it.

In gcloud not crashing if the 'database is locked' and doing retries would be nice. Alternatively is there a way to tell fast_pusher.py to skip the auth check? Probably something related to
https://github.com/google/containerregistry/blob/master/client/docker_creds_.py

from containerregistry.

This'd be a good issue to post internally for the gcloud sdk team, since I'm guessing that the cause is in a common path. iirc there's a component for the docker helpers. I wouldn't call this a google/containerregistry bug, since it most likely affects a whole bunch of tools that'd use docker-credential-gcloud

from containerregistry.

I think this is going through https://github.com/GoogleCloudPlatform/docker-credential-gcr; I wonder if forcing the auth-refresh is not helping when we concurrently push a dozen or so images: https://github.com/GoogleCloudPlatform/docker-credential-gcr/blob/7e55abeb1839689afa77070b523e0057a5c961cd/credhelper/helper.go#L215

I think that means that "preloading" the credentials is also ineffective.

Perhaps we would do better to specify e.g. --min-expiry=300s. We might also benefit from a single retry there, for when the token expires - I'm not sure of the exact semantics.

Is this going through docker-credential-gcr? Should I send a PR there?

from containerregistry.

I ran into something similar: GoogleCloudPlatform/docker-credential-gcr#54

from containerregistry.

This got fixed internally recently (in gcloud). Anecdotally, all my tools are now ~4x faster because getting access tokens from gcloud is much faster. Going to close this, please reopen if you still see it after updating gcloud.

from containerregistry.