gke-tf was created to streamline the creation of customized Terraform source files that build GKE clusters and supporting resources in GCP.

License: Apache License 2.0

Python 31.12% Makefile 2.97% Go 62.62% Shell 3.28%

gke gke-helmsman terraform google-cloud kubernetes

gke-terraform-generator's Introduction

Google Kubernetes Engine (GKE) Terraform Generator

Introduction
Architecture
Prerequisites
Usage
Troubleshooting
Relevant Material

Introduction

gke-tf was created to streamline the creation of customized Terraform source files that build GKE clusters and supporting resources in GCP. It follows a similar workflow pattern used by other Kubernetes installation tools like Kops to reduce the difficulty of creating complex GKE deployments in GCP down to manipulating a simple YAML configuration file.

Architecture

From a single YAML configuration file that specifies all the desired configuration settings, the gke-tf binary can validate those settings and combine them with built-in templates to generate Terraform source files in a desired directory. From this point, the user/administrator can use terraform to plan, apply, and then destroy the infrastructure and GKE cluster.

Prerequisites

Tools

gke-tf
Terraform >= 0.12.3
Google Cloud SDK version >= 253.0.0
kubectl matching the latest GKE version
bash or bash-compatible shell
A Google Cloud Platform project where you have Project Owner permissions to create VPC networks, service accounts, IAM Roles, GKE clusters, and more.

Install gke-tf

Download the latest version from the Releases page and move the binary for your platform into your path. We provide OSX, Linux and Windows binaries, but at this point OSX is the tested binary.

Install Cloud SDK

The Google Cloud SDK is used to interact with your GCP resources. Installation instructions for multiple platforms are available online.

Install kubectl CLI

The kubectl CLI is used to interteract with both Kubernetes Engine and kubernetes in general. Installation instructions for multiple platforms are available online.

Install Terraform

Terraform is used to automate the manipulation of cloud infrastructure. Its installation instructions are also available online.

Usage

Authenticate gcloud

Prior to interacting with the Terraform generated by gke-tf, ensure you have authenticated your gcloud client by running the following command:

gcloud auth application-default login

Also, confirm the gcloud configuration is properly pointing at your desired project. Run gcloud config list and make sure that compute/zone, compute/region and core/project are populated with values that work for you. You can set their values with the following commands:

# Where the region is us-east1
gcloud config set compute/region us-east1

Updated property [compute/region].

# Where the zone inside the region is us-east1-c
gcloud config set compute/zone us-east1-c

Updated property [compute/zone].

# Where the project name is my-project-name
gcloud config set project my-project-name

Updated property [core/project].

Setup this project

The Terraform generated by gke-tf will enable the following Google Cloud Service APIs in the target project:

cloudresourcemanager.googleapis.com
container.googleapis.com
compute.googleapis.com
iam.googleapis.com
logging.googleapis.com
monitoring.googleapis.com

Generating the Desired Terraform

Review the YAML files in the examples directory for an understanding of how a GKE cluster can be built using gke-tf. You may use these as a base for customization or one provided by the repository that leverages gke-tf.

With gke-tf in your $PATH, generate the Terraform necessary to build the cluster for this demo. The example command below will send the generated Terraform files to the terraform directory inside this repository and use the examples/example.yaml as the cluster configuration file input. The GCP project is passed to this command as well.

export PROJECT="<my-project-name>"
gke-tf gen -d ./terraform -f examples/example.yaml -o -p ${PROJECT}

Review the generated Terraform files in the terraform directory to understand what will be built inside your GCP project. If anything needs modifying, edit the examples/example.yaml and re-run the gke-tf gen command above. The newly generated Terraform files will reflect your changes. You are then ready to proceed to using Terraform to build the cluster and supporting resources.

Provisioning the Generated Terraform

Next, apply the terraform configuration with:

cd terraform # if not already in this directory
terraform init
terraform plan
terraform apply

When prompted, review the generated plan and enter yes to deploy the environment.

Teardown

To uninstall the resources built using Terraform:

cd terraform # if not already in this directory
terraform destroy

When prompted, review the plan for resource destruction and enter yes to proceed.

Troubleshooting

Error parsing terraform/main.tf: At NN:NN: Unknown token: 20:13 IDENT var. - This is typically caused by using Terraform v0.11 against Terraform files in the v0.12.x format. Check your terraform version to ensure it's v0.12 or higher.

Relevant Material

Terraform

This is not an officially supported Google product

gke-terraform-generator's People

Contributors

Stargazers

Watchers

gke-terraform-generator's Issues

Using a private registry in same project is not working

I have added the missing oauth scope in #21 and for some reason it is not working :(

I am getting docker login issues on the nodes. Here is my yaml in order to recreate:

kind: gke-cluster
metadata:
  name: "test-cluster"
spec:
  region: "us-west2"
  private: "false"
  regional: "false"
  zones: 
    - "us-west2-c"
  addons:
    istio: false
    binaryAuth: false
  network:
    metadata:
      name: my-network
    spec:
      subnetName: my-subnet
      subnetRange: "10.0.0.0/24"
      podSubnetRange: "10.1.0.0/16"
      serviceSubnetRange: "10.2.0.0/20"
      masterIPV4CIDRBlock: "172.16.0.16/28"
  nodePools:
    - metadata:
        name: my-node-pool
      spec:
        minCount: 2
        maxCount: 2
        initialNodeCount: 2
        machineType: n1-standard-2
        preemptible: true

This is a public cluster and needs my other PR merged for the cluster to even launch.

workloadIdentityConfig default value

workloadIdentityConfig:
  identityNamespace: "bgeesaman-gke-demos.svc.id.goog"

We may be able to default the value for identityNamespace. Not certain exactly what it defaults to.

Issue when trying this in us-central1

I have set region/zone to us-central1/us-central1-a, but the demo has failed. By looking at the logs, it seems that the script tried to create a NAT Gateway in us-west1... why ? given that the default region is us-central1.

`google_compute_router.router: Creation complete after 9s [id=us-west1/test-cluster-cloud-router]
google_compute_firewall.bastion-ssh: Still creating... [10s elapsed]
google_compute_subnetwork.subnetwork: Still creating... [10s elapsed]
google_compute_firewall.bastion-ssh: Creation complete after 16s [id=test-cluster-bastion-ssh]
google_compute_subnetwork.subnetwork: Still creating... [20s elapsed]
google_compute_subnetwork.subnetwork: Still creating... [30s elapsed]
google_compute_subnetwork.subnetwork: Creation complete after 39s [id=us-west1/my-subnet]
google_compute_router_nat.nat: Creating...
google_compute_instance.instance: Creating...

Error: Error waiting to patch router us-west1/test-cluster-cloud-router: Error waiting for Patching router: error while retrieving operation: googleapi: Error 401: Invalid Credentials, authError

on network.tf line 109, in resource "google_compute_router_nat" "nat":
109: resource "google_compute_router_nat" "nat" {

Error: Error waiting for instance to create: error while retrieving operation: Get https://www.googleapis.com/compute/v1/projects/my-apigee-project/zones/us-west1-a/operations/operation-1564478751498-58ee2983de116-77b5505e-ecefd8fe?alt=json&prettyPrint=false: oauth2/google: incomplete token received from metadata

on network.tf line 169, in resource "google_compute_instance" "instance":
169: resource "google_compute_instance" "instance" {
`

Istio Enable by default

Do we want istio enabled by default? @bgeesaman @wripley ???

Terraform Remote State

As discussed, I think it would be better to use Terraform remote state with a GCS bucket in this demo to showcase Terraform best practices instead of using local state.
We could add an init script that would create the GCS bucket if needed.

cc @aurelienlegrand

Putty doesn't like -L8888 and needs -L 8888 (a space)

event-exporter-gke crashloopBackoff

├── provider.google
├── provider.google-beta ~> 3.28.0
├── provider.kubernetes
├── provider.vault
└── module.dcp-gke
    ├── provider.google
    ├── provider.google-beta
    ├── provider.kubernetes
    └── provider.random

When i deploy an new cluster by using Terraform providers . every think is OK except this pods : event-exporter-gke

i don't know why he can boot on cluster . I faced this issue:

F0707 13:26:00.772823       1 main.go:123] Failed to get GCE config: error while getting instance (node) name: metadata: GCE metadata "instance/name" not defined

Master version
1.16.9-gke.6
Endpoint
35.205.99.XX

maybe an permission issue ???

any suggestions will help me...

Update CFT templates

In order to use CFT we need to update the various files in https://github.com/GoogleCloudPlatform/gke-terraform-generator/tree/master/pkg/terraform/cft

Each of the files needs to be updated to support the various values set in https://github.com/GoogleCloudPlatform/gke-terraform-generator/blob/master/pkg/api/api.go. Those structs define the values that the user provides in the yaml that is used to define a cluster.

The https://github.com/GoogleCloudPlatform/gke-terraform-generator/tree/master/pkg/terraform/vanilla directory containers the currently working templates for vanilla terraform that can be used as a guide on how to implement the CFT terraform.

The CFT go templates, to my best recollection, where working at one point, but need to be improved and tested.

Tests like https://github.com/GoogleCloudPlatform/gke-terraform-generator/blob/master/pkg/templates/templates_test.go#L82 determine if various values are set in the templates. Further unit testing should be implemented.

Error: Error loading zone 'us-east1-a': googleapi: Error 404: The resource 'projects/[project-id]/zones/us-east1-a' was not found
, notFound

Reviewing the terraform plan I found that the zone was automatically generated.

It looks like the us-east1-a zone doesn't exist, only the zones b, c, d.

Is this an issue or am I missing something?

IntialNode Count MaxCount MinCount Validation

The validation for initialNodeCount: 2 working weird.

This works

  nodePools:
    - metadata:
        name: my-node-pool
      spec:
        minCount: 2
        maxCount: 2
        initialNodeCount: 2

This does not work:

  nodePools:
    - metadata:
        name: my-node-pool
      spec:
        minCount: 2
        initialNodeCount: 2