Git Product home page Git Product logo

solutions-modern-cicd-anthos's Introduction

Modern CI/CD with Anthos

CI

Introduction

Kubernetes has given us wonderful abstraction that we can use to optimize the way we develop, deploy, and maintain software projects across multiple environments. In many cases though, Kubernetes is too complex for end users to learn and feel empowered with. To alleviate this learning curve, many teams are looking to build platform abstractions on top of Kubernetes to streamline onboarding and reduce maintenenance for software projects.

In this repository we lay out a prescriptive way to create a multi-team software delivery platform using Anthos. The platform has the following capabilities:

  • Allow platform administrators to create and update best practices for provisioning apps
  • Ensure App Developers can iterate independently in their own "landing zones" without interfering with each other
  • Allow security teams to seamlessly implement and propagate policy across the platform
  • Use GitOps for deployment

For more details, please watch this talk on Youtube.

Architecture Overview

After the Quick Start you will have the following infra:

Anthos Platform Infrastructure

  • GitLab deployed on GKE to host your source code repostitories
  • 1 Dev cluster that can be used for iterative development with tools like Skaffold
  • 1 Staging cluster
  • 2 Production clusters in different GCP regions

Within GitLab you will have the following repo structure: Anthos Platform Repos

Starter repos have examples for:

Pre-requisites

  1. Clone this repo to your local machine.

  2. Install gcloud SDK.

  3. Create a new GCP project.

  4. Enable billing for GCP project

  5. Set the us-central1 region to deploy infrastructure

    export REGION="us-central1"
    gcloud config set compute/region ${REGION}

Build Infrastructure

  1. Run the following commands to setup Cloud Build

    export PROJECT_ID=<INSERT_YOUR_PROJECT_ID>
    gcloud config set core/project ${PROJECT_ID}
    export PROJECT_NUMBER=$(gcloud projects describe ${PROJECT_ID} --format 'value(projectNumber)')
    gcloud services enable cloudbuild.googleapis.com
    gcloud services enable anthos.googleapis.com
    gcloud services enable serviceusage.googleapis.com
    gcloud services enable binaryauthorization.googleapis.com
    gcloud services enable cloudkms.googleapis.com
    gcloud services enable containeranalysis.googleapis.com
    gcloud services enable secretmanager.googleapis.com
    gcloud projects add-iam-policy-binding ${PROJECT_ID} --member serviceAccount:${PROJECT_NUMBER}@cloudbuild.gserviceaccount.com --role roles/owner
    gcloud projects add-iam-policy-binding ${PROJECT_ID} --member serviceAccount:${PROJECT_NUMBER}@cloudbuild.gserviceaccount.com --role roles/containeranalysis.admin
  2. Run Cloud Build to create the necessary resources.

    gcloud builds submit --substitutions=_PROJECT_ID=${PROJECT_ID}

    โš ๏ธ This operation may take up to 30 minutes depending on region. Do not close the console or connection as the operation is NOT idempotent. If a failure occurs, clean up the environment and attempt again.

  3. Log in to your GitLab instance with the URL, username and password printed at the end of the build. Hang on to this password, you will need it for later steps.

  4. Follow the steps in the docs to go through a user journey (add, deploy, and change applications).

Important Variables

  1. Take note and record the Password for your Gitlab account.

  2. URL for Gitlab

    echo "https://gitlab.endpoints.${PROJECT_ID}.cloud.goog"
  3. User and Password for GitLab are stored in the Secrets Manager

export GITLAB_USER=$(gcloud secrets versions access latest --secret="gitlab-user")
export GITLAB_PASSWORD=$(gcloud secrets versions access latest --secret="gitlab-password")

echo "User: ${GITLAB_USER}"
echo "Password: ${GITLAB_PASSWORD}"

Clean Up

  1. Remove infrastructure

    gcloud builds submit --substitutions=_PROJECT_ID=${PROJECT_ID} --config cloudbuild-destroy.yaml
    gcloud endpoints services delete gitlab.endpoints.${PROJECT_ID}.cloud.goog
    gcloud endpoints services delete registry.endpoints.${PROJECT_ID}.cloud.goog
  2. Unset variables (optional)

    unset PROJECT_ID
    unset REGION

Securing the ACM repository

At this stage, you should have a working ACM installation good enough for most demos. If you want to follow production best practices, read Best practices for policy management with Anthos Config Management and GitLab.

Always leave at least one namespace defined in namespaces/managed-apps, otherwise ACM will stop syncing.

solutions-modern-cicd-anthos's People

Contributors

bgood avatar bmenasha avatar cwest avatar danisla avatar dependabot[bot] avatar frankfarzan avatar ghchinoy avatar henrybell avatar kenthua avatar knee-berts avatar michael-intindola avatar mike-ensor avatar mml avatar morgante avatar mrtrustor avatar patflynn avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

solutions-modern-cicd-anthos's Issues

Gitlab password cannot be retrieved from Secret Manager

Behaviour: After running "gcloud builds submit --substitutions=_DOMAIN=${DOMAIN}" the gitlab root user password is not displayed in the shell, see error below.

Step #8 - "output-connect-details": ERROR: (gcloud.secrets.versions.access) INVALID_ARGUMENT: Resource ID [projects/xxxxxxxxxxxxxx/secrets/"gitlab-password"/versions/latest] is not in a valid format.

Expected Behaviour: Gitlab root user password is displayed in the shell at after Cloud Build finished.

Workaround: GCP Console -> Products -> Security -> Secret Manager -> gitlab-password -> Actions -> View Secret Value

Add data layer to a sample app

All the sample applications are stateless, so deployments are simply container pushes. It would be great to add a sample app which is backed by a database and demonstrate how databases can be managed in the CI/CD pipeline.

CLI uses gcloud in the current project

The CLI assumes the current project is the right project. This is confusing at best and dangerous in other cases. Somehow the tool needs to understand which "thing" we are targeting without requiring us to set the correct gcloud project and hand the exact corresponding gitlab parameters.

Builds are Intermittently failing on GitLab install (stage 4)

https://github.com/GoogleCloudPlatform/solutions-modern-cicd-anthos/runs/716929543?check_suite_focus=true and seen locally multiple times

Step #4 - "setup-gitlab": module.gke-gitlab.helm_release.gitlab: Still creating... [8m40s elapsed]
Step #4 - "setup-gitlab": module.gke-gitlab.helm_release.gitlab: Still creating... [8m50s elapsed]
Step #4 - "setup-gitlab": module.gke-gitlab.helm_release.gitlab: Still creating... [9m0s elapsed]
Step #4 - "setup-gitlab": module.gke-gitlab.helm_release.gitlab: Still creating... [9m10s elapsed]
Step #4 - "setup-gitlab": module.gke-gitlab.helm_release.gitlab: Still creating... [9m20s elapsed]
Step #4 - "setup-gitlab": module.gke-gitlab.helm_release.gitlab: Still creating... [9m30s elapsed]
Step #4 - "setup-gitlab": module.gke-gitlab.helm_release.gitlab: Still creating... [9m40s elapsed]
Step #4 - "setup-gitlab": module.gke-gitlab.helm_release.gitlab: Still creating... [9m50s elapsed]
Step #4 - "setup-gitlab": module.gke-gitlab.helm_release.gitlab: Still creating... [10m0s elapsed]
Step #4 - "setup-gitlab": module.gke-gitlab.helm_release.gitlab: Still creating... [10m10s elapsed]
Step #4 - "setup-gitlab": module.gke-gitlab.helm_release.gitlab: Still creating... [10m20s elapsed]
Step #4 - "setup-gitlab": module.gke-gitlab.helm_release.gitlab: Still creating... [10m30s elapsed]
Step #4 - "setup-gitlab": module.gke-gitlab.helm_release.gitlab: Still creating... [10m40s elapsed]
Step #4 - "setup-gitlab": module.gke-gitlab.helm_release.gitlab: Still creating... [10m50s elapsed]
Step #4 - "setup-gitlab": 
Step #4 - "setup-gitlab": Error: rpc error: code = Unknown desc = release gitlab failed: timed out waiting for the condition
Step #4 - "setup-gitlab": 
Step #4 - "setup-gitlab":   on .terraform/modules/gke-gitlab/main.tf line 399, in resource "helm_release" "gitlab":
Step #4 - "setup-gitlab":  399: resource "helm_release" "gitlab" ***
Step #4 - "setup-gitlab": 
Step #4 - "setup-gitlab": 
Finished Step #4 - "setup-gitlab"
ERROR
ERROR: build step 4 "gcr.io/anthos-platform-ci-env/anthos-platform-installer" failed: step exited with non-zero status: 1
--------------------------------------------------------------------------------

##[error]Process completed with exit code 1.

Develop a more comprehensive "wait" for APIs to be available

Using the project-factory project-services module, services are enabled, but the calls to the APIs fail stating the API has not been enabled. It takes a moment (to several moments) for the APIs to be enabled across GCP. A "wait" has been setup inside of "1_clusters/binary-auth.tf" and this needs to be enhanced with some solution that makes and succeeds on a call to the container-analysis API

The original bug is #33 where a quick-fix was added to be dependent on the output of the project-services module

Install fails due to missing containeranalysis API

Here is the error:

Step #3 - "create-clusters": Error: Error creating Note: googleapi: Error 403: Container Analysis API has not been used in project 610065905755 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/containeranalysis.googleapis.com/overview?project=610065905755 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.
Step #3 - "create-clusters": 
Step #3 - "create-clusters":   on .terraform/modules/quality-attestor/modules/binary-authorization/main.tf line 50, in resource "google_container_analysis_note" "build-note":
Step #3 - "create-clusters":   50: resource "google_container_analysis_note" "build-note" {
Step #3 - "create-clusters": 
Step #3 - "create-clusters": 
Step #3 - "create-clusters": ```

Simplifying use of gatekeeper kpt function

https://github.com/GoogleCloudPlatform/solutions-modern-cicd-anthos/blob/master/starter-repos/shared-ci-cd/cd/validate.yaml

Above is a completely valid use of kpt CLI and may be desirable if you actually want separate steps. But we can use a single step as well:

    kpt pkg get https://gitlab-ci-token:${CI_JOB_TOKEN}@${CI_SERVER_HOST}/platform-admins/anthos-config-management.git/constraints/apps app-constraints
    kpt fn source app-constraints/ hydrated-manifests/ | kpt fn run --image gcr.io/kpt-functions/gatekeeper-validate
  artifacts:
    paths:
      - hydrated-manifests/

We can potentially simplify this even further by having kpt fn run to merge multiple directories.

Initial build does not display GitLab password in the console

When running Quickstart and doing initial build, all goes well, except that the output does not show the GitLab password. The error is shown below:

Step #8 - "output-connect-details": Log in to your GitLab instance at: https://gitlab.platform.gcpdemo.org
Step #8 - "output-connect-details": Username: root
Step #8 - "output-connect-details":
Step #8 - "output-connect-details": ERROR: (gcloud.secrets.versions.access) INVALID_ARGUMENT: Resource ID [projects/1065071717895/secrets/"gitlab-password"/versions/latest] is not in a valid format.
Step #8 - "output-connect-details": Retrieve password:
Step #8 - "output-connect-details":
Step #8 - "output-connect-details": Visit the user guide in the docs (docs/index.md) to go through a user journey (add, deploy, and change applications).
Finished Step #8 - "output-connect-details"
PUSH
DONE

The issue appears to be in escaping the double quotes in the main cloudbuild.yaml file line 127:

echo "Retrieve password: $(gcloud secrets versions access latest --secret=\"gitlab-password\")"

I can still get the password by running the command myself:
gcloud secrets versions access latest --secret=gitlab-password

Break GitLab repo creation into separate step

The GitLab repo seeding currently:
a) Is surprisingly embedded inside the GitLab creation directory.
b) Uses semi-imperative scripting and not pure Terraform.

This should be cleaned up. Further discussion here.

Unable to add new app with anthos-platform-cli

This worked for me a week ago on a different project. Now recreated the entire environment and not able to add new application using the command

anthos-platform-cli add app \
    --name $APP_NAME \
    --gitlab-hostname $GITLAB_HOSTNAME \
    --gitlab-token $GITLAB_ROOT_PASSWORD \
    --template-name golang-template
  

Here is the error I am getting:

INFO[0001] Creating AR repository test-app-1 in location us-central1
INFO[0004] Creating service account test-app-1-push
FATA[0006] Unable to add binding for service account test-app-1-push@Your active configuration is: [cicd-anthos]
kharkovski-anthos-cicd.iam.gserviceaccount.com: ERROR: Policy modification failed. For a binding with condition, run "gcloud alpha iam policies lint-condition" to identify issues in condition.
ERROR: (gcloud.beta.artifacts.repositories.add-iam-policy-binding) INVALID_ARGUMENT: Invalid service account (test-app-1-push@Your active configuration is: [cicd-anthos]
kharkovski-anthos-cicd.iam.gserviceaccount.com).

I checked and the IAM service account was created as well as the artifact registry in GCP. Also it seems to me that the name of the Service Account is somehow broken.

CLI uses gcloud in interactive mode

The CLI fails if glcoud beta is not installed:

gcloud components install beta

Should either use --quiet flag or if user action is required, print the gcloud command to run manually.

$ apcli add app --name echo --gitlab-hostname $GITLAB_HOSTNAME --gitlab-token 
       $GITLAB_ROOT_PASSWORD --template-name golang-template

FATA[0002] Unable to list Artifact Registry repos: exit status 1
You do not currently have this command group installed.  Using it
requires the installation of components: [beta]


Your current Cloud SDK version is: 292.0.0
Installing components from version: 292.0.0

+---------------------------------------------+
|     These components will be installed.     |
+----------------------+------------+---------+
|         Name         |  Version   |   Size  |
+----------------------+------------+---------+
| gcloud Beta Commands | 2019.05.17 | < 1 MiB |
+----------------------+------------+---------+

For the latest full release notes, please visit:
  https://cloud.google.com/sdk/release_notes

Do you want to continue (Y/n)?
ERROR: (gcloud) This prompt could not be answered because you are not in an interactive session.  You can re-run the command with the --quiet flag to accept default answers for all prompts.

Document starter repos

The starter repos don't contain much guidance on what they contain or how to use them. Maybe put a top-level README.md and then one in each subdir?

Break primary build into physical stages

The current build implements one large cloud build process. Failures in late stages of the build require a full destroy and rebuild due to stage 4+ not being idempotent.

This issue/story will use a parent/child job structure and in conjunction with remote state to break up the build into modules or stages.

The final solution will need to have a one-click/one-command to "build it all", but will allow each segment to be destroyed and rebuilt independently so a failure in late stage can be build independently without requiring a 30-45 minute destroy/rebuild.

Add public load balancer to the template apps

The golang-template app currently only provisions a ClusterIP service which requires that users set up port-forwarding in order to test.

It would be easier to demonstrate the working service if you could hit a public IP.

Adding an app involves multiple small `Fix templated names` commits

Using the CLI to add an app leads to these commits (newest to oldest):

Fix templated names
Fix templated names
Fix templated names
Fix templated names
Fix templated names
Fix templated names
Initial commit

Why can't the initial commit be the fully customized? If it's desirable to have this separation, why can't there be one commit called Customize app (Fix is misleading as it's not a bug) that includes all the changes. Currently, it seems each commit changes just one file.

hydrate-manifests (Kustomize) stage failing

The image used to run the kustomize stage to hydrate manifests is failing because the image recent got updated.

Error output during the hydrate-manifest stage is:

Error: accumulating resources: accumulateFile "accumulating resources from 'https://<GITLAB-HOSTNAME>/platform-admins/shared-kustomize-bases//golang?ref=master': YAML file [https://<GITLAB-HOSTNAME>/platform-admins/shared-kustomize-bases//golang?ref=master] encounters a format error.\nerror converting YAML to JSON: yaml: line 3: mapping values are not allowed in this context\n", accumulateDirector: "couldn't make target for path '/tmp/kustomize-151693645/repo': unable to find one of 'kustomization.yaml', 'kustomization.yml' or 'Kustomization' in directory '/tmp/kustomize-151693645/repo'"

I think this is probably related to the deprecation of bases or a new line issue in the latest version of Kustomize.

As a work around the kustomize image used for they hydrate-manifests stage in platform-admins/shared-ci-cd/cd/kustomize.yaml can be set to:

gcr.io/cloud-solutions-images/kustomize@sha256:949719059c41085f9eeb89b74c03ccac60e2142eeabeb3ede9614a33433b09eb

Error creating cicd_gitlab blueprint: namespace not found

When applying the cicd_gitlab using kpt, it returns an error:

$ kpt live init cicd_gitlab
Initialized: /*****/cicd_gitlab/inventory-template.yaml
$ kpt live apply cicd_gitlab/
Fatal error: error when creating "generated": namespaces "gitlab" not found

The namespace config exists, but somehow kpt couldn't find it.

$ cat cicd_gitlab/namespace.yaml 
apiVersion: v1
kind: Namespace
metadata:
  name: gitlab # {"$kpt-set":"namespace"}

Using kpt version v0.32.0

@frankfarzan

Setup sometimes fail when creating GitLab Endpoints

Sometimes the setup scripts will fail when creating the Endpoints for GitLab.

Step #5 - "trigger-gitlab": Step #0 - "create-gitlab": module.gke-gitlab.helm_release.gitlab: Creation complete after 3m22s [id=gitlab]
Step #5 - "trigger-gitlab": Step #0 - "create-gitlab":
Step #5 - "trigger-gitlab": Step #0 - "create-gitlab": Error: Error waiting for Performing service rollout.: Error code 13, message: Internal Error
Step #5 - "trigger-gitlab": Step #0 - "create-gitlab":
Step #5 - "trigger-gitlab": Step #0 - "create-gitlab":   on .terraform/modules/cloud-endpoints-dns-registry/main.tf line 58, in resource "google_endpoints_service" "default":
Step #5 - "trigger-gitlab": Step #0 - "create-gitlab":   58: resource "google_endpoints_service" "default" {
Step #5 - "trigger-gitlab": Step #0 - "create-gitlab":
Step #5 - "trigger-gitlab": Step #0 - "create-gitlab":
Step #5 - "trigger-gitlab": Finished Step #0 - "create-gitlab"
Step #5 - "trigger-gitlab": ERROR
Step #5 - "trigger-gitlab": ERROR: build step 0 "gcr.io/pr-anthos-demo/anthos-platform-installer" failed: step exited with non-zero status: 1
Step #5 - "trigger-gitlab": --------------------------------------------------------------------------------
Step #5 - "trigger-gitlab":
Finished Step #5 - "trigger-gitlab"
ERROR
ERROR: build step 5 "gcr.io/cloud-builders/gcloud" failed: step exited with non-zero status: 1
-------------------------------------------------------------------------------------------------------------------

Simply re-running the setup scripts will usually fix the problem.

Create a "pull" model for deploying services to clusters

Be able to push the hydrated YAML manifest updates to the ACM repository allowing ACM to deploy the updates of the service rather than using gke-deploy.

This will add to the CICD pipeline files allowing the project implementing CICD to chose these alternative pipeline jobs over the deploy method

Build process does not build/release CLI

The build automation does not include steps to build and release the CLI either selectively or by default. As a workaround for this, the CLI can be built from within the repo to get an up-to-date version, but this requires a local golang development environment.

Pipelines on non-master branches can't access protected variables

GCP_AR_KEY is a protected variable in a repo. Based on the current default settings, only the master branch has access to protected variables.

 $ [ -z ${GCP_AR_KEY} ] && echo 'Artifact Registry credentials not available' && exit 1
 Artifact Registry credentials not available

For each repo, we would need to add protected branches for specific branches, or wildcard branches.

Repository -> Settings -> Repository -> Protected Branches

Make the CI process better.

Currently the CI for the solution is only runs on merges into the main line and is brittle.

CI would be better if it ran on PRs, nightly and in addition to merges into main.

CLI requires gitlab access tokens stored locally

In order to use the CLI to create apps, I had to generate a gitlab access token. I needed some place to put this thing so I put it on my workstation. The interaction model pretty much encourages a user to create the token once and keep it on the local filesystem. This is only slightly less bad than keeping a password in a file.

From a security perspective, ideally the actuation of gitlab would happen from something running in-cluster and all the user would have to do would be to authenticate to kube-apiserver.

kustomize CI/CD step fails when uploading artifacts

It fails for a new app when uploading artifacts:

$ git config --global url."https://gitlab-ci-token:${CI_JOB_TOKEN}@${CI_SERVER_HOST}".insteadOf "https://${CI_SERVER_HOST}"
$ mkdir -p hydrated-manifests/
$ cd ${KUSTOMIZATION_PATH_STG}
$ kustomize edit set image app="${GCP_AR_REPO}/${CI_PROJECT_NAME}:${CI_COMMIT_SHA}"
$ kustomize build . -o ../../hydrated-manifests/stg.yaml
$ cd ../../
$ cd ${KUSTOMIZATION_PATH_PROD}
$ kustomize edit set image app="${GCP_AR_REPO}/${CI_PROJECT_NAME}:${CI_COMMIT_SHA}"
$ mkdir -p hydrated-manifests/
$ kustomize build . -o ../../hydrated-manifests/prod.yaml
Uploading artifacts...
hydrated-manifests/: found 3 matching files        
WARNING: Uploading artifacts to coordinator... failed  id=156 responseStatus=500 Internal Server Error status=500 Internal Server Error token=GJRGBLv3
WARNING: Retrying...                                error=invalid argument
WARNING: Uploading artifacts to coordinator... failed  id=156 responseStatus=500 Internal Server Error status=500 Internal Server Error token=GJRGBLv3
WARNING: Retrying...                                error=invalid argument
WARNING: Uploading artifacts to coordinator... failed  id=156 responseStatus=500 Internal Server Error status=500 Internal Server Error token=GJRGBLv3
FATAL: invalid argument                            
ERROR: Job failed: command terminated with exit code 1

Docs: Explain more the setup

In the main docs, the biggest step is "run cloud build", which runs with project owner permissions. This is the GCE equivalent of curl foo.sh | bash, and AFAICT we don't remove those permissions after we're done.

I think separately we'll be working to reduce the permissions required, but I really would like there to be more explanation of what happens when I do that. I don't think we should be teaching customers to blindly run a process with project owner permissions, and also it seems important to help them understand the building blocks here. Ultimately, this is just a demo but their world is going to look different than the demo and they'll need to build their own variant of the demo.

Maybe at a high level, we could get away with just saying "after this runs, these APIs are turned on, these resources ..., these permissions ..., these deployments ..., etc."

Add binary attestors to CICD workflow

Use either a CLI or inherited CICD pipeline job approach to add attestations for an attestor at the "build/test", "quality" and "security" or a mix that makes sense to demonstrate a progressive collection of attestations for an image

`setup-gitlab` TF module fails with different symptoms

Initial gcloud build submit is brittle and fails setup-gitlab step with different symptoms

Attempt 1:

Step #4 - "setup-gitlab": module.gke-gitlab.null_resource.sleep_for_cluster_fix_helm_6361: Creation complete after 3m0s [id=5996002542353204672]
Step #4 - "setup-gitlab":
Step #4 - "setup-gitlab": Error: googleapi: Error 403: Service Networking API has not been used in project 553329319679 before or it is disabled. Enable it by visiting https://co
nsole.developers.google.com/apis/api/servicenetworking.googleapis.com/overview?project=553329319679 then retry. If you enabled this API recently, wait a few minutes for the actio
n to propagate to our systems and retry., accessNotConfigured
Step #4 - "setup-gitlab":
Step #4 - "setup-gitlab":   on .terraform/modules/gke-gitlab/main.tf line 148, in resource "google_service_networking_connection" "private_vpc_connection":
Step #4 - "setup-gitlab":  148: resource "google_service_networking_connection" "private_vpc_connection" {
Step #4 - "setup-gitlab":
Step #4 - "setup-gitlab":
Finished Step #4 - "setup-gitlab"
ERROR
ERROR: build step 4 "gcr.io/kpt-fn/anthos-platform-installer" failed: step exited with non-zero status: 1
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
ERROR: (gcloud.builds.submit) build 8d2272a4-28d5-43a4-be3e-e907dbec6343 completed with status "FAILURE"

Attempt 2:

Enumerating objects: 13, done.
Counting objects: 100% (13/13), done.
Delta compression using up to 8 threads
Compressing objects: 100% (6/6), done.
Writing objects: 100% (7/7), 728 bytes | 242.00 KiB/s, done.
Step #4 - "setup-gitlab":
Step #4 - "setup-gitlab": Error: Get http://localhost/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/tiller-admin: dial tcp 127.0.0.1:80: connect: connection refused
Step #4 - "setup-gitlab":
Step #4 - "setup-gitlab":
Step #4 - "setup-gitlab":
Step #4 - "setup-gitlab": Error: Get http://localhost/api/v1/namespaces/default/secrets/gitlab-pg: dial tcp 127.0.0.1:80: connect: connection refused
Step #4 - "setup-gitlab":
Step #4 - "setup-gitlab":
Step #4 - "setup-gitlab":
Step #4 - "setup-gitlab": Error: Error reading Service Account Key "projects/kpt-fn/serviceAccounts/[email protected]/keys/1a9c30cca0d59f30ed2975a5cd186132be0249be": googleapi: Error 403: Permission iam.serviceAccountKeys.get is required to perform this operation on service account key projects/kpt-fn/serviceAccounts/[email protected]/keys/1a9c30cca0d59f30ed2975a5cd186132be0249be., forbidden
Step #4 - "setup-gitlab":
Step #4 - "setup-gitlab":
Finished Step #4 - "setup-gitlab"
ERROR
ERROR: build step 4 "gcr.io/kpt-fn/anthos-platform-installer" failed: step exited with non-zero status: 1

Attempt 3:

Step #4 - "setup-gitlab":   - serviceAccount:[email protected]
Step #4 - "setup-gitlab":   role: roles/storage.admin
Step #4 - "setup-gitlab": etag: BwWmCxB9n80=
Step #4 - "setup-gitlab": version: 1
Step #4 - "setup-gitlab": + for service in ${SERVICES}
Step #4 - "setup-gitlab": + [email protected]
Step #4 - "setup-gitlab": + gcloud projects add-iam-policy-binding kpt-fn --member serviceAccount:[email protected] --role roles/monitoring.metricWriter
Step #4 - "setup-gitlab": ERROR: Policy modification failed. For a binding with condition, run "gcloud alpha iam policies lint-condition" to identify issues in condition.
Step #4 - "setup-gitlab": ERROR: (gcloud.projects.add-iam-policy-binding) INVALID_ARGUMENT: Service account [email protected] does not exist.
Step #4 - "setup-gitlab": + gcloud projects add-iam-policy-binding kpt-fn --member serviceAccount:[email protected] --role roles/cloudtrace.agent
Step #4 - "setup-gitlab": ERROR: Policy modification failed. For a binding with condition, run "gcloud alpha iam policies lint-condition" to identify issues in condition.
Step #4 - "setup-gitlab": ERROR: (gcloud.projects.add-iam-policy-binding) INVALID_ARGUMENT: Service account [email protected] does not exist.
Step #4 - "setup-gitlab": + gcloud projects add-iam-policy-binding kpt-fn --member serviceAccount:[email protected] --role roles/cloudprofiler.agent
Step #4 - "setup-gitlab": ERROR: Policy modification failed. For a binding with condition, run "gcloud alpha iam policies lint-condition" to identify issues in condition.
Step #4 - "setup-gitlab": ERROR: (gcloud.projects.add-iam-policy-binding) INVALID_ARGUMENT: Service account [email protected] does not exist.
Step #4 - "setup-gitlab": + gcloud projects add-iam-policy-binding kpt-fn --member serviceAccount:[email protected] --role roles/clouddebugger.agent
Step #4 - "setup-gitlab": ERROR: Policy modification failed. For a binding with condition, run "gcloud alpha iam policies lint-condition" to identify issues in condition.
Step #4 - "setup-gitlab": ERROR: (gcloud.projects.add-iam-policy-binding) INVALID_ARGUMENT: Service account [email protected] does not exist.
Step #4 - "setup-gitlab": + gcloud projects add-iam-policy-binding kpt-fn --member serviceAccount:[email protected] --role roles/errorreporting.writer
Step #4 - "setup-gitlab": ERROR: Policy modification failed. For a binding with condition, run "gcloud alpha iam policies lint-condition" to identify issues in condition.
Step #4 - "setup-gitlab": ERROR: (gcloud.projects.add-iam-policy-binding) INVALID_ARGUMENT: Service account [email protected] does not exist.
Step #4 - "setup-gitlab": + for service in ${SERVICES}
Step #4 - "setup-gitlab": + [email protected]
Step #4 - "setup-gitlab": + gcloud projects add-iam-policy-binding kpt-fn --member serviceAccount:[email protected] --role roles/monitoring.metricWriter
Step #4 - "setup-gitlab": ERROR: Policy modification failed. For a binding with condition, run "gcloud alpha iam policies lint-condition" to identify issues in condition.
Step #4 - "setup-gitlab": ERROR: (gcloud.projects.add-iam-policy-binding) INVALID_ARGUMENT: Service account [email protected] does not exist.
Step #4 - "setup-gitlab": + gcloud projects add-iam-policy-binding kpt-fn --member serviceAccount:[email protected] --role roles/cloudtrace.agent
Step #4 - "setup-gitlab": ERROR: Policy modification failed. For a binding with condition, run "gcloud alpha iam policies lint-condition" to identify issues in condition.
Step #4 - "setup-gitlab": ERROR: (gcloud.projects.add-iam-policy-binding) INVALID_ARGUMENT: Service account [email protected] does not exist.
Step #4 - "setup-gitlab": + gcloud projects add-iam-policy-binding kpt-fn --member serviceAccount:[email protected] --role roles/cloudprofiler.agent
Step #4 - "setup-gitlab": ERROR: Policy modification failed. For a binding with condition, run "gcloud alpha iam policies lint-condition" to identify issues in condition.
Step #4 - "setup-gitlab": ERROR: (gcloud.projects.add-iam-policy-binding) INVALID_ARGUMENT: Service account [email protected] does not exist.
Step #4 - "setup-gitlab": + gcloud projects add-iam-policy-binding kpt-fn --member serviceAccount:[email protected] --role roles/clouddebugger.agent
Step #4 - "setup-gitlab": ERROR: Policy modification failed. For a binding with condition, run "gcloud alpha iam policies lint-condition" to identify issues in condition.
Step #4 - "setup-gitlab": ERROR: (gcloud.projects.add-iam-policy-binding) INVALID_ARGUMENT: Service account [email protected] does not exist.
Step #4 - "setup-gitlab": + gcloud projects add-iam-policy-binding kpt-fn --member serviceAccount:[email protected] --role roles/errorreporting.writer
Step #4 - "setup-gitlab": ERROR: Policy modification failed. For a binding with condition, run "gcloud alpha iam policies lint-condition" to identify issues in condition.
Step #4 - "setup-gitlab": ERROR: (gcloud.projects.add-iam-policy-binding) INVALID_ARGUMENT: Service account [email protected] does not exist.
Finished Step #4 - "setup-gitlab"
ERROR
ERROR: build step 4 "gcr.io/kpt-fn/anthos-platform-installer" failed: step exited with non-zero status: 1

Finally succeeded on Attempt 4.

Destroy path takes out existing clusters

I got a report from a user that they used our setup and the destroy step took out their non-related clusters.

We should tag our clusters and use those tags to filter out the ones we want to destroy.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.