Comments (17)
You can use the source repositories of gcloud instead of github. Read my comment in the #44
I perform a step in my cloudbuild.yaml which rename the repositories :
- id: renameGitHubUrl
name: debian
args:
- sed
- -i
- 's#[email protected]:___GitHubId___/\(.*\)\.git#https://source.developers.google.com/p/___GCLOUD_PROJECT_ID___/r/github-___GitHubId___-\1#g'
- .gitmodules
As soon as your private github repository are well named in gcloud source repositories, it will be correct. After this step you can submodule init and update.
from cloud-builders.
For anyone coming across this: An easier way than creating deploy keys, encrypting them with kms and adding multiple additional steps to your pipeline is to simply have a Google mirror the submodule as well. Then run two commands to temporarily change the submodule config on disk to point to the Google mirror before updating the submodule. Like so:
# Update submodules
- name: 'gcr.io/cloud-builders/git'
entrypoint: 'bash'
args:
- '-c'
- |
git config -f .git/config submodule.one.url https://source.developers.google.com/p/[project_id]/r/[repo_one]
git config -f .gitmodules submodule.one.url https://source.developers.google.com/p/[project_id]/r/[repo_one]
git submodule init
git submodule update
from cloud-builders.
Wondering if this is scheduled to be fixed anytime soon. @imjasonh
from cloud-builders.
For what it's worth, pst's comment saved the day. THANKS!
That said, I must admit I had to poke around to make sense of it, so perhaps these more detailed steps will save the next person time...
To mirror the submodule, go to Cloud Source Repositories. That gave me a repo like:
https://source.cloud.google.com/[my-project]/github_[my-sub-repo-name]
In order for Cloud Build to gain access, I needed to rewrite like:
https://source.developers.google.com/p/[my-project]/r/github_[my-sub-repo-name]
With that new url, I followed the example above... but I did not need to update .git/config
, only .gitmodules
.
So:
- name: 'gcr.io/cloud-builders/git'
entrypoint: 'bash'
args:
- '-c'
- |
git config -f .gitmodules submodule.[my-sub-repo-name].url https://source.developers.google.com/p/[my-project]/r/github_[my-sub-repo-name]
git submodule init
git submodule update
I added this to my cloudbuild.yaml
as the first step. Cloud Build then had no issues cloning the submodule during the build.
Your mileage may vary with a different setup. I do hope Google adds proper support to do this automatically in the near future. What a pain.
from cloud-builders.
For what it's worth, pst's comment saved the day. THANKS!
That said, I must admit I had to poke around to make sense of it, so perhaps these more detailed steps will save the next person time...
To mirror the submodule, go to Cloud Source Repositories. That gave me a repo like:
https://source.cloud.google.com/[my-project]/github_[my-sub-repo-name]
In order for Cloud Build to gain access, I needed to rewrite like:
https://source.developers.google.com/p/[my-project]/r/github_[my-sub-repo-name]
With that new url, I followed the example above... but I did not need to update
.git/config
, only.gitmodules
.So:
- name: 'gcr.io/cloud-builders/git' entrypoint: 'bash' args: - '-c' - | git config -f .gitmodules submodule.[my-sub-repo-name].url https://source.developers.google.com/p/[my-project]/r/github_[my-sub-repo-name] git submodule init git submodule updateI added this to my
cloudbuild.yaml
as the first step. Cloud Build then had no issues cloning the submodule during the build.Your mileage may vary with a different setup. I do hope Google adds proper support to do this automatically in the near future. What a pain.
@pst @youens What did you update in your .gitmodules
file after the change in yaml. I also made the changes you wrote in your comment but getting the following error.
Starting Step #1
Step #1: Already have image (with digest): gcr.io/cloud-builders/git
Step #1: fatal: not a git repository (or any parent up to mount point /)
Step #1: Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
Step #1: fatal: not a git repository (or any parent up to mount point /)
Step #1: Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
Finished Step #1
ERROR
ERROR: build step 1 "gcr.io/cloud-builders/git" failed: exit status 128
from cloud-builders.
Are you sure no subrepo is fetched? We populate the workspace like so:
git config --local credential.helper gcloud.sh && \
(git remote add origin $REMOTE || true) && \
git fetch --depth=1 --recurse-submodules=yes --tags origin $REV && \
git reset --hard $REV
The workspace is initialized once before any build steps run, and it is not cleared between steps. If steps are executed in parallel, normal concurrent disk semantics apply (so be careful about writing the same file from two build steps).
from cloud-builders.
Yes. This is the the output before the BUILD step. There is no mention of attempts to fetch submodules.
FETCHSOURCE
Initialized empty Git repository in /workspace/.git/
From https://source.developers.google.com/p/<my-project>/r/<my-repo>
* branch <my-sha1-long> -> FETCH_HEAD
HEAD is now at <my-sha1-short> use cloudbuild
BUILD
from cloud-builders.
I'm also unable to fetch submodules, which is probably related to using Google Cloud Repository that mirrors a GitHub repo. After running git submodule update
using the cloud-builders/git
image, I just get:
Step #0: Host key verification failed.
Step #0: fatal: Could not read from remote repository.
Is there any workaround? Can I somehow get to the deploy key that was used for the mirroring and convince git to use that to update the submodules?
Nevermind, I forgot that deploy keys are specific to a single repo.
from cloud-builders.
Is this still an issue?
from cloud-builders.
@gmistick Yes, this is still not solved. The issue is that the repo you build from is a mirror that the builder service account has access to, and that account might not have access to private submodules defined in the repo.
/cc @wlynch
from cloud-builders.
Can we have it please? @juancampa @wlynch
from cloud-builders.
We currently use the approach described here: https://cloud.google.com/cloud-build/docs/access-private-github-repos
Instead of the git clone step we do git submodule update.
from cloud-builders.
Unfortunately the advice above didn't seem to work in my case, I just cloned the repo in my cloud build configuration.
from cloud-builders.
@mairh if you triggering the build from Github it will not work since the .git folder will not be present. All repositories need to be fetched from Cloud Source Repositories.
from cloud-builders.
Guys,
I never ever thought this would actually work, but it did :))
- id: git-submodule
name: 'gcr.io/cloud-builders/git'
entrypoint: 'bash'
env: ['GIT_DISCOVERY_ACROSS_FILESYSTEM=1']
args:
- '-c'
- |
git init
git config -f .gitmodules submodule.[my-sub-repo-name].url https://source.developers.google.com/p/[my-project]/r/github_[my-sub-repo-name]
git submodule init
git submodule update
💃
from cloud-builders.
Mine is like this, but with these following conditions
- Original private repo is in Github
- Integration with Cloudbuild Github App
- Parent Github repo must be mirrored to Google Cloud Source Repositories.
- Submodule repos also need to be mirrored to Google Cloud Source Repositories
Here's the Cloudbuild step:
- id: 'initialize_repo'
name: 'gcr.io/cloud-builders/git'
entrypoint: 'bash'
env: ['GIT_DISCOVERY_ACROSS_FILESYSTEM=1']
args:
- '-c'
- |
git init
git clean -d -f . #clean current working directory
git remote add origin https://source.developers.google.com/p/[gcp-project]/r/[parent-repo-name-in-google-cloud-source]
git fetch origin $BRANCH_NAME
git checkout $COMMIT_SHA #checkout at current commit
git config -f .gitmodules submodule.[submodule-repo-name].url https://source.developers.google.com/p/[gcp-project]/r/[submodule-repo-name-in-google-cloud-source]
git submodule init
git submodule update
Notes: well, it's very very very hacky. Basically we re-initialize the repo and discard the source code obtained via Github App. Hopefully Cloudbuild will have better support for git submodule
in the future :)
from cloud-builders.
Note: please report issues with and feature requests for the hosted Google Cloud Build service to your Google Cloud Support team, or use the public issue tracker at
https://issuetracker.google.com/issues/new?component=190802&template=1162743.
I'm closing this issue as "off-topic."
from cloud-builders.
Related Issues (20)
- go:latest uses 1.15 instead of the newer 1.21 images which are present HOT 2
- [BUG] Java Builder doc currently recommends openjdk (DEPRECATED) should recommend Temurin HOT 1
- Docker build squash HOT 1
- [BUG] Bazel image missing `python3-distutils`
- [BUG] kubectl - memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused HOT 3
- [FR] xp.zen.one.zip.MAXFIELD.dev.GT.snapdragon.K3M.oxford.USNRx.fugk.ZoeOzone.ZOZANZI.xia.xiu.BuGzLoPœvzg.opus.hummingbird.circa.VELOTRIX.sexpixel.clarity.verity 609 https://t.co/gadgetgreen
- [BUG] Recent commit for 'latest' tag was a breaking change for many
- [BUG] Latest gcloud image throws (AttributeError): 'bool' object has no attribute 'lower' HOT 2
- [BUG] Cloud Build Error
- [BUG] npm 9.7.0 HOT 2
- Docs refer to stale bazel-provided images, new images aren't compatible with Cloud Build HOT 2
- [BUG] gke-deploy generated files fail deployment HOT 2
- yarn image has outdated node versions
- [FR] Add labels to cloud-builder images
- [BUG] gke-deploy does not respect the declared ApiVersion
- How to pass credentials to GKE-DEPLOY HOT 1
- [FR] Update Bazel to 7.x HOT 2
- [FR] Remove mandatory dependence on gsutil HOT 1
- [BUG] kubectl builder finds wrong context when executed in parallel HOT 1
- [BUG] `/gke-deploy: No such file or directory` HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cloud-builders.