Comments (28)
Here i s an other way :
Create with gcloud a source repository.
https://cloud.google.com/source-repositories/docs/
In the console it is :
https://console.cloud.google.com/code/develop/repo
If your have for example a git repository : [email protected]:org42/billing-service.git
Name the source repository org42-billing-service
in gcloud, connect this source to your github private repository.
Then in your cloudbuilder.yaml :
name: 'gcr.io/cloud-builders/git'
args:
- clone
- https://source.developers.google.com/p/$PROJECT_ID/r/org42-billing-service
from cloud-builders.
@Philmod I think it should, I'm using govendor golang package manager and by default it is https git clone, I'll try to git config --global url."[email protected]:".insteadOf "https://github.com/"
and follow provided tutorial, thank you.
from cloud-builders.
Hey @alexanderGalushka , we now have a tutorial about this topic: https://cloud.google.com/container-builder/docs/access-private-github-repos
Does that help?
from cloud-builders.
So, I ended up writing a little tool which gets temporary access token from Github and use that token in builds.
Code - Fidelius
post explaining more details - link
from cloud-builders.
Did anyone here find a solution ? I have my builds failing because of a go.mod dependency on a private repository. I can clone the repo in my cloud builds but the app deploy
step fails with
fatal: could not read Username for 'https://github.com': terminal prompts disabled
I have tried setting the git config --global
flag as well, but to no avail :(
- name: 'gcr.io/cloud-builders/git'
entrypoint: 'git'
args:
- config
- --global
- --add
- [email protected]:.insteadOf
- https://github.com/
Also, @bendory why was this closed without a resolution ?
from cloud-builders.
Hey,
We don’t have an integrated solution as of today, but we have seen one solution coming from users. The trick is to use a Deploy or User Key that you could store in GCS for example, and pull it as a previous build step.
from cloud-builders.
@bendory I finally seems to have got this working with SSH as well. Thought that was nice because of Githubs deploy keys feature. The setup I use is like this:
cloudbuild.yaml
steps:
- name: 'gcr.io/cloud-builders/gcloud'
args:
- kms
- decrypt
- --ciphertext-file=deployment/id_rsa.enc
- --plaintext-file=deployment/id_rsa
- --location=[location]
- --keyring=[keyring]
- --key=[key]
# Prepare for github cloning
- name: 'gcr.io/cloud-builders/gcloud'
dir: deployment
entrypoint: ./git-prepare.sh
volumes:
- name: 'ssh-setup'
path: /root/.ssh
- name: 'gcr.io/cloud-builders/git'
env:
- GIT_SSH=deployment/git-ssh.sh
args:
- clone
- [email protected]:[org]/[github-reponame]
volumes:
- name: 'ssh-setup'
path: /root/.ssh
deployment/git-prepare.sh
#!/bin/bash
chmod 600 id_rsa
ssh-keyscan github.com > /root/.ssh/known_hosts
deployment/git-ssh.sh
#!/bin/bash
ssh -i /workspace/deployment/id_rsa $1 $2
from cloud-builders.
@Philmod for the record, I wrote up what I did here https://medium.com/@lestrrat/taming-google-container-builder-22a6dded155c
from cloud-builders.
I opened a feature request at https://issuetracker.google.com/issues/150198815 to solve this problem
from cloud-builders.
Is there an example out there of configuring .netrc, or any other way to configure the build environment to hold the git authorization for a service account, so that You wouldn't have to include a secret in the package.json or other package scheme (go)?
from cloud-builders.
We don't have a canonical example at this time. We have seen users do this by storing the secret in GCS and then pulling it into their build using the gsutil
build step.
from cloud-builders.
@bendory, thanks. That works fine if you are scripting bash commands, but it won't work, if you want to use a package manager to pull from git private repos without putting the private token in your package.json file. Since there doesn't seem to be a way to get a username/password for a service account, I think I will just create a fake user email, that I can authenticate at github or google source, then include that in a .netrc that I include in GCS.
from cloud-builders.
This newly-published doc on Using Encrypted Files in your build should be helpful.
from cloud-builders.
@bendory thanks for the tutorial.
Any idea how to actually get the git cloud builder steps to remember the settings from previous steps? I do a git config --global credentials 'store --file [unencrypted credentials file]'
, but in the next git clone
steps, it still tries to use gcloud.sh
. I can confirm that's being used by echoing. Otherwise, I'm not sure how git private repos are being used. Here's what I have so far:
steps:
# Unencrypt the git credentials
- name: 'gcr.io/cloud-builders/gcloud'
args:
- kms
- decrypt
- --ciphertext-file=git-credentials.enc
- --plaintext-file=git-credentials
- --location=global
- --keyring=github-read
- --key=github-read-key
- name: 'gcr.io/cloud-builders/git'
args: ['config', '--global', 'credential.helper', "'store --file /workspace/git-credentials'"]
# Echo git credentials: (Incorrectly outputs `gcloud.sh` instead the file above)
- name: 'gcr.io/cloud-builders/git'
args: ['config', '--global', '--list']
# Pull source repo
# Doesn't work, same problem as above (it's not using the store, as if the command remembered nothing)
- name: 'gcr.io/cloud-builders/git'
args: ['clone', 'https://github.com/myprivate/repo.git']
from cloud-builders.
@imjasonh is working on a better solution for this, but in the meantime, I think the following workaround will help you move forward:
steps:
# Unencrypt the git credentials
# Note that I changed your --plaintext-file arg to .git-credentials
- name: 'gcr.io/cloud-builders/gcloud'
args:
- kms
- decrypt
- --ciphertext-file=git-credentials.enc
- --plaintext-file=.git-credentials
- --location=global
- --keyring=github-read
- --key=github-read-key
# Note that I changed your --file arg to .git-credentials
- name: 'gcr.io/cloud-builders/git'
args: ['config', '--global', 'credential.helper', "'store --file /workspace/.git-credentials'"]
env: ['HOME=/workspace']
# Echo git credentials
- name: 'gcr.io/cloud-builders/git'
args: ['config', '--global', '--list']
env: ['HOME=/workspace']
# Pull source repo
# Doesn't work, same problem as above (it's not using the store, as if the command remembered nothing)
- name: 'gcr.io/cloud-builders/git'
args: ['clone', 'https://github.com/myprivate/repo.git']
env: ['HOME=/workspace']
Full explanation, in case the wizardry isn't clear...
Because the /workspace
directory tree are the only files that persist between build steps, you need your git credentials to live there. git
expects to store config in ~
and read credentials from ~/.git-credentials
, but $HOME
isn't persisted. So what I've done here is persisted the credentials in /workspace/.git-credentials
and then set HOME=/workspace
so that git
's configuration files are persisted across build steps.
Does that make sense? Does it help move you forward?
Disclaimer: I didn't run the above build steps, they may be not quite right, please give a try...
from cloud-builders.
@bendory Thank you for the response and explanation, it did help me move forward :D
from cloud-builders.
I followed this, and I can't seem to get this to work. what's the correct way to do it?
Step #5: git: 'credential-store --file /workspace/.git-credential' is not a git command. See 'git --help'.
from cloud-builders.
@lestrrat it doesn't look like your command is using the right syntax (it's git config credential.helper [...]
: https://git-scm.com/docs/git-credential-store)
from cloud-builders.
@purohit that's not what I typed, that's the git error after executing.
With the quoting as is in #44 (comment), I get
Step #6: 01:22:37.646078 git.c:369 trace: built-in: git 'clone' 'https://github.com/my_project/my_repo.git'
Step #6: Cloning into 'bunto'...
Step #6: 01:22:37.670798 run-command.c:369 trace: run_command: 'git-remote-https' 'origin' 'https://github.com/my_project/my_repo.git'
Step #6: 01:22:38.058218 run-command.c:369 trace: run_command: 'git credential-gcloud.sh get'
Step #6: 01:22:38.059135 run-command.c:228 trace: exec: '/bin/sh' '-c' 'git credential-gcloud.sh get' 'git credential-gcloud.sh get'
Step #6: 01:22:38.081128 git.c:594 trace: exec: 'git-credential-gcloud.sh' 'get'
Step #6: 01:22:38.081269 run-command.c:369 trace: run_command: 'git-credential-gcloud.sh' 'get'
Step #6: 01:22:38.420456 run-command.c:369 trace: run_command: 'git credential-'\''store --file /workspace/.git-credentials'\'' get'
Step #6: 01:22:38.421205 run-command.c:228 trace: exec: '/bin/sh' '-c' 'git credential-'\''store --file /workspace/.git-credentials'\'' get' 'git credential-'\''store --file /workspace/.git-credentials'\'' get'
Step #6: 01:22:38.422737 git.c:594 trace: exec: 'git-credential-store --file /workspace/.git-credentials' 'get'
Step #6: 01:22:38.422808 run-command.c:369 trace: run_command: 'git-credential-store --file /workspace/.git-credentials' 'get'
Step #6: git: 'credential-store --file /workspace/.git-credentials' is not a git command. See 'git --help'.
Step #6: fatal: could not read Username for 'https://github.com': No such device or address
Finished Step #6
With quotes changed to
args:
- 'config'
- '--global'
- 'credential.helper'
- 'store --file /workspace/.git-credentials'
Step #6: 01:24:14.408691 git.c:369 trace: built-in: git 'clone' 'https://github.com/my_project/my_repo.git'
Step #6: Cloning into 'bunto'...
Step #6: 01:24:14.434288 run-command.c:369 trace: run_command: 'git-remote-https' 'origin' 'https://github.com/my_project/my_repo.git'
Step #6: 01:24:14.800590 run-command.c:369 trace: run_command: 'git credential-gcloud.sh get'
Step #6: 01:24:14.801279 run-command.c:228 trace: exec: '/bin/sh' '-c' 'git credential-gcloud.sh get' 'git credential-gcloud.sh get'
Step #6: 01:24:14.818346 git.c:594 trace: exec: 'git-credential-gcloud.sh' 'get'
Step #6: 01:24:14.818435 run-command.c:369 trace: run_command: 'git-credential-gcloud.sh' 'get'
Step #6: 01:24:15.104658 run-command.c:369 trace: run_command: 'git credential-store --file /workspace/.git-credentials get'
Step #6: 01:24:15.105289 run-command.c:228 trace: exec: '/bin/sh' '-c' 'git credential-store --file /workspace/.git-credentials get' 'git credential-store --file /workspace/.git-credentials get'
Step #6: 01:24:15.106617 git.c:594 trace: exec: 'git-credential-store' '--file' '/workspace/.git-credentials' 'get'
Step #6: 01:24:15.106677 run-command.c:369 trace: run_command: 'git-credential-store' '--file' '/workspace/.git-credentials' 'get'
Step #6: fatal: could not read Username for 'https://github.com': No such device or address
So this seems to have been a problem with how the Cloud Builder's interpretation of command arguments when certain quoting rules are applied.
Subsequent errors turned out to be a problem with how I wrote the git-credentials file. I will post a wrap up when everything works.
from cloud-builders.
@biggestT Thanks for the write-up!
from cloud-builders.
@Philmod @bendory has the integration solution been worked on?
@lestrrat I also have a trouble with credential.helper store, have you got it to work?
from cloud-builders.
@Philmod I have tried the tutorial you have specified at
https://cloud.google.com/container-builder/docs/access-private-github-repos
It worked for me but I had to do a little bit of tweaking of the cloudbuild.yaml
, I guess this tutorial was designed for non-triggered builds only. In my case, I needed to use this builder/git for a branch triggered build because the default cloned/mounted repo was a detached repo (it didn't link to the parent and does not retrieve the branches), hence I have to use this builder so I can make use of some git commands, for example checking if a file/dir has changed. Just an FYI.
from cloud-builders.
@pepedocs By default when the build is on a Git repo, we do a "shallow clone" of the repo, which contains only the commit being built. This results in a faster build when the repo is very large, since the shallow clone doesn't have to fetch the full history of the repo, or other branches.
If you want to "unshallow" the clone, you can do so by adding this step:
- steps:
name: gcr.io/cloud-builders/git
args: ['fetch', '--unshallow']
from cloud-builders.
@imjasonh Great. So that just mean I do not have to use the git cloud builder then for my use case. I just have to use the cloned repo (with --unshallow) and then just run git commands from there. Thank you.
---UPDATE---
@imjasonh I guess you were referring to the this git cloud builder when I said that the cloned repo was detached (see my previous post). Just to clarify that the "detached" repo I was referring to was the one cloned and mounted (to the container) by the github triggered build (not the one cloned by the git cloud builder). However, it helps that you mentioned that the git cloud builder clones a shallow repo. I'm still going to use this git cloud builder to properly clone a repo and run git commands with it. Thank you.
from cloud-builders.
@pepedocs
What was your solution for triggered builds ?
from cloud-builders.
@hegdeph For the cloned repo by GCB I ran an unshallow git fetch command. For other repos I used Google kms to clone the repo.
from cloud-builders.
Any update on this problem?
I have a private Go repo sitting on Gitlab which is a dependency of another go app. When I do, app deploy
, build is failing because it cannot access my private repo.
I was able to setup a GCloud Secret Manager to authorize tokens in Gcloud builds and was able to trigger a build successfuly but the app deploy still fails. It seems that the app deploy is not using the tokens I setup in cloud build.
I followed the steps in this document:
https://cloud.google.com/cloud-build/docs/access-private-github-repos
from cloud-builders.
For support with the hosted Google Cloud Build service, please escalate to your Google Cloud Support team or use the public issue tracker at
https://issuetracker.google.com/issues/new?component=190802&template=1162743.
from cloud-builders.
Related Issues (20)
- go:latest uses 1.15 instead of the newer 1.21 images which are present HOT 2
- [BUG] Java Builder doc currently recommends openjdk (DEPRECATED) should recommend Temurin HOT 1
- Docker build squash HOT 1
- [BUG] Bazel image missing `python3-distutils`
- [BUG] kubectl - memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused HOT 3
- [FR] xp.zen.one.zip.MAXFIELD.dev.GT.snapdragon.K3M.oxford.USNRx.fugk.ZoeOzone.ZOZANZI.xia.xiu.BuGzLoPœvzg.opus.hummingbird.circa.VELOTRIX.sexpixel.clarity.verity 609 https://t.co/gadgetgreen
- [BUG] Recent commit for 'latest' tag was a breaking change for many
- [BUG] Latest gcloud image throws (AttributeError): 'bool' object has no attribute 'lower' HOT 2
- [BUG] Cloud Build Error
- [BUG] npm 9.7.0 HOT 2
- Docs refer to stale bazel-provided images, new images aren't compatible with Cloud Build HOT 2
- [BUG] gke-deploy generated files fail deployment HOT 2
- yarn image has outdated node versions
- [FR] Add labels to cloud-builder images
- [BUG] gke-deploy does not respect the declared ApiVersion
- How to pass credentials to GKE-DEPLOY HOT 1
- [FR] Update Bazel to 7.x HOT 2
- [FR] Remove mandatory dependence on gsutil HOT 1
- [BUG] kubectl builder finds wrong context when executed in parallel HOT 1
- [BUG] `/gke-deploy: No such file or directory` HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cloud-builders.