Comments (8)
I got past this by passing in --cloud_key
- seems like it should be added to Getting Started, can open a PR if desired.
from caliban.
OK, now I see the documentation: https://caliban.readthedocs.io/en/latest/cloud/service_account.html , and that the documentation just assumes GOOGLE_APPLICATION_CREDENTIALS
is set to a service account (not user account). None of this is a big deal, just some clarification on credentials would be good so people (particularly scientists) don't bounce before getting everything running or at least knowing what to ask for. Some clarification on why credentials are needed to run locally would be good too?
from caliban.
Hey @jordanrule! Thanks for this report - this is definitely a bug if it's on Caliban 0.3.0 or later. Can I ask what version of Caliban you're on? There did indeed used to be a service account credential requirement, before I figured out how I could hijack the normal, gcloud auth login
credentials to do job submissions. That's why I dropped the service account requirements from the main docs.
I think that the code should work the way you expect (with no requirement here) on the latest published version of Caliban.
You've also discovered an internal trick we use. Here's my current understanding of how google SAYS credentials work:
- If you set
GOOGLE_APPLICATION_CREDENTIALS
, it has to point to a service account JSON key, as you've described, not anything else. - That key almost always lives in some directory like
~/.config
, somewhere user-local. - Docker isn't able, by design, to access any files outside of the folder where it's running. So, as a hack, if you HAVE set that environment variable, we create a temporary copy in the project folder, then delete it when the caliban command finishes running. It's not that elegant... I would use a temporary filename, but that would bust the Docker cache. That's why you see the hidden filename
.caliban_default_creds
; that's the static name of the temp file.
What to do?
So how to solve your problem? I think if you upgrade Caliban, this will go away, assuming you've got GOOGLE_APPLICATION_CREDENTIALS
pointed at a valid service account key (and that you delete your copy of the temporary file, so Caliban can write and delete it).
If you've got GOOGLE_APPLICATION_CREDENTIALS
pointed at any OTHER type of creds, like the "application default credentials", then you'll see failures, I expect.
Okay, let me know what you think! Let me know if I've misunderstood anything, and especially if the problem persists after an upgrade. The credentials part of all of this is annoying and we want to make the experience easy for folks.
from caliban.
jordandrule$ caliban --version
caliban 0.3.0
So here's the issue from my perspective: I don't think requiring users to set GOOGLE_APPLICATION_CREDENTIALS
to a service account is a good best practice (yes that's what I ended up doing after running into the same issue with caliban cloud
). In a corporate environment we want to be able to know who is racking up the BQ charges, and we can't do that if all the scientists are using the same service account. Telling them to unset GOOGLE_APPLICATION_CREDENTIALS
after every deployment is a workaround but not ideal.
I might recommend having a different environment variable GOOGLE_APPLICATION_SERVICE_CREDENTIALS
that if set would pull from, and only if it is not set only then default to GOOGLE_APPLICATION_CREDENTIALS
.
Can you provide more detail on hijacking google auth login
and not requiring service credentials? I am on 0.3.0 and caliban run
is not working without service creds. An ideal on-boarding process would allow a scientist to run locally and become comfortable with usability before having to bug engineering for a service account.
from caliban.
Hey @jordanrule ,
I agree with you here. But the GOOGLE_APPLICATION_CREDENTIALS
variable ONLY works when pointing at a service account key: https://cloud.google.com/docs/authentication/getting-started#setting_the_environment_variable
That's the only purpose of that variable, so, if you want to associate billing I think it makes sense to not use this variable at all, and to have folks use gcloud auth login
to authenticate once. Make sense?
The "trick" is that we shell out to gcloud auth print-access-token
and use that token for AI Platform authentication. So, if gcloud auth list
shows that you're authenticated everything should work.
from caliban.
Seeing this above:
and that the documentation just assumes
GOOGLE_APPLICATION_CREDENTIALS
is set to a service account (not user account)
Can you point me to any documentation you find that describes this use of GOOGLE_APPLICATION_CREDENTIALS
? I'm not sure what "pointing to a user account" means here. Thanks!
from caliban.
Aha - you have taught me the distinction between ADC and a service key, and now I see the caliban runs without GOOGLE_APPLICATION_CREDENTIALS
even set.
For some reason I had set GOOGLE_APPLICATION_CREDENTIALS
set to application_default_credentials.json
(who knows why, given the documentation you pointed out states that should never work), which was causing the problem.
Thank you for your patience. I am demoing caliban on Monday so at least you have given me the confidence to explain different Google authentication strategies. Feel free to close this.
from caliban.
It’s all terribly confusing... and to make it more confusing, it DOES work to do what you say for a few random binaries Google produces, I think by accident.
Also, I’m not sure that the docs make this clear, but Caliban will in fact copy application default creds into the container if you have them set. This is nice for getting in-container authorized for buckets etc.
I’ll go ahead and close, but please keep asking questions! Happy to help, and I’m sure you’ll expose some error at some point, which will be quite helpful.
Cheers,
Sam
from caliban.
Related Issues (20)
- HTTP Error 403: Forbidden HOT 3
- ModuleNotFoundError: No module named 'google' HOT 2
- Issue with caliban package with installing using pip HOT 3
- distirbuted training
- Upgrade to modern dependencies [project]
- Caliban should fail more gracefully when Docker isn't available
- Feature request: support REES HOT 1
- "Failed to read the container uri ... Please make sure that CloudML Engine service account has access to it"
- `caliban cloud`: providing project ID through CLI fails HOT 1
- Create base image based on Ubuntu 20.04 LTS HOT 2
- A way to provide my own docker image? HOT 3
- Cannot create cluster HOT 5
- Insufficient quota in GCP free trial account HOT 2
- Missing newlines in generated Dockerfile when using GCP credentials HOT 1
- Docker image is rebuilt for every `cluster job submit` HOT 12
- Google-auth is not installed automatically
- Make caliban fall back to cpu-only gracefully for local or shell commands HOT 1
- Provide Base Docker Image for CUDA 11
- Looking for a strangely-named image HOT 12
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from caliban.