googlecloudplatform / cloudbuild-integration-testing Goto Github PK

each build should

• create a namespace with the BUILD_ID
• run deployments into this namespace
• delete namespace after integration test

A dash of CSS, a sprinkle of imagery

For two reasons, we don't use a simple snap install microk8s command to install MicroK8s:

1. We want to maintain K8s version parity with GKE. (as of this writing, our GKE cluster is at 1.11.7; upstream k8s--and therefore microk8s--is at 1.14.x)
2. We want to make the Kubernetes API on the microk8s accessible from outside, on a nonsecure port (sure, we ought to connect securely -- and it should only be available via private network [see #22] -- but opening it up makes it easier to configure CI)

So, to get our custom version of MicroK8s, we build from source for a specific version. Prior to building, we patch the defaults for kube-apiserver by changing --insecure-bind-address to 0.0.0.0: allow from anywhere.

All of that takes a kong time to run, so we capture the output as a VM image, and then use that image in our CI.

This is all very silly and fragile.

At a minimum, we should script this nonsense (with e.g. Packer) so it can be reproduced and updated. But much better would be to figure out how to install the MicroK8s snap with a specified version of K8s, and with overrides for the default args.

Another "lightweight k8s that runs nicely on one node..."

https://github.com/rancher/k3s

For portability, the config files in /k8s have a placeholder: PROJECT_ID (see #33). During a build, we can automatically replace these with the ID of the project it's being run in.

Let's try it: what happens if we launch a fresh GKE cluster during the build?

Currently, we tag all images as 'web','db' ... these get pushed to registry as 'web:latest','db:latest'. This happens even if the rest of the build fails. As a result, we have the potential for collision and version drift: one test might pick up the containers built by another. Also, a job deploying to prod might grab a version that's actually no good.

We should tag the images with build-specific tags and push those specific images to the test environment.

(Possibly: at the end of the build—on success—re-tag as "latest")

My current belief is that the VM method works better with NodePort; the GKE method works better with LoadBalancer. Let's dig into that.

• if database is unavailable, unit tests will pass but integration tests fail
• actually these things can't talk to each other

Add database mock to unit tests

Something like this? https://github.com/intuit/karate

(Looks like it’s tailored for rest APIs but might have some use for us? Or other options?)

Zone of cluster should be consistent throughout samples. Currently, some files point to us-central1-c and other points to us-central1-a.

Sometimes the web container can access the db at http://cookieshop_db:3306, sometimes it can't. I think the problem may have something to do with kubedns -- like, if the dns service doesn't come up fast enough, then the web service gets a bad dns resolution (early) and can never get a good one.

Is there another step required to get cloud build to talk to cluster nodes? In my own testing I was unable to get cloud build to talk to a node without whitelisting 0.0.0.0 and after running your setup I'm seeing the same results.

Step #9 - "integration tests": url: http://35.188.172.115:30332
Step #9 - "integration tests": interval: 3
Step #9 - "integration tests": retries: 20
Step #9 - "integration tests": START CONNECTION TESTi

This never seems to complete.

• finish baking image of single-node k8s on GCE
• adapt deploy-to-gke script to work on GCE VM

MiniKube and MicroK8s are both intended as lightweight, single-node K8s implementations. Might they be faster/easier than the full thing? Would they provide appropriate fidelity? (ie. how likely is it that a test against MiniKube will succeed, but then the app fails on Kubernetes? Obviously, that's not supposed to happen.)

There are scattered hard-coded project names, cluster names, zones. Let's pull those out so the build can be run in any project without code changes.

• survival kubectl commands included at beginning of build
• think of max you can do without ssh'ing into pod

Make a cloudbuild configuration that will invoke all of the other cloudbuild configurations. Once it's working, make this a blocking check on PRs to master.

This will probably require using Custom Workers and VPN. Or perhaps lock down via certificate?

When making initial deployment into cluster/namespace, the service's external IP is pending for a short time. Logic should be added into the config to wait for the external IP to be available before storing it, else it will return null and the test will fail.

...this will make it easier to understand which step is which method, when looking at Cloud Build history.

We got acquired! ;-)