Smallstep CA on GCE

Problem Statement

I run Home Assistant in my home network, and wanted to expose that to the internet in order to integrate with a Google Home smart speaker. A sensible choice is to require mTLS client authentication on all inbound conections, but that is hard without sound PKI.

This is where the Smallstep CA comes in.

┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ 
  step-ca project                                                        │
│                                                                         
           ┌────────────────────┐                                        │
│          │ Cloud Run          │                                         
           │    ┌───────────┐   │                                        │
│          │    │           │   │              ┌──────────────────────┐   
        ┌──┼───▶│   NGINX   │───┼──────┐       │ VPC subnet           │  │
│       │  │    │           │   │      │       │                      │   
        │  │    └───────────┘   │      │       │  ┌────────────────┐  │  │
│       │  │          │         │    request   │  │                │  │   
        │  └──────────┼─────────┘    TLS cert  │  │  Smallstep CA  │  │  │
│       │             │                └───────┼─▶│    (GCE VM)    │  │   
        │          proxied                     │  │                │  │  │
│       │          request                     │  └────────────────┘  │   
        │         with added                   │                      │  │
│       │            mTLS                      └──────────────────────┘   
        │             │                                                  │
└ ─ ─ ─ ┼ ─ ─ ─ ─ ─ ─ ┼ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ 
        │             │                                                   
      HTTPS           │       ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─                       
        │             │         Home network       │                      
        │             │       │                                           
┌─ ── ── ── ── ─┐     │            ┌───────────┐   │                      
│   External          │       │    │   Home    │                          
 Service without│     └───────────▶│ Assistant │   │                      
│    CA cert    │             │    │           │                          
└ ── ── ── ── ──                   └───────────┘   │                      
                              │                                           
                               ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘

In this diagram, the "external service" cannot talk to Home Assistant as it doesn't support mTLS connections.

The solution is to run a reverse-proxy which adds the mTLS certificate and forwards the request onto Home Assistant. Cloud Run is used as that cheaply provides a HTTPS endpoint on the web to which the external service can connect.

Basic Setup

Refer to Smallstep's instructions along with the below, as the following will not be up-to-date forever.

Install Smallstep CLI

The CLI is required to setup and interact with the CA from the shell:

curl -o /tmp/step.tgz -L https://github.com/smallstep/cli/releases/download/v0.15.14/step_linux_0.15.14_amd64.tar.gz
tar xzf /tmp/step.tgz --strip-components=1 -C /tmp
mv /tmp/bin/step /usr/local/bin

Install Smallstep CA

Install the actual CA:

curl -o /tmp/step-ca.tgz -L https://github.com/smallstep/certificates/releases/download/v0.15.11/step-certificates_linux_0.15.11_amd64.tar.gz
tar xzf /tmp/step-ca.tgz --strip-components=1 -C /tmp
mv /tmp/bin/step-ca /usr/local/bin

Configure and run Step CA

Example step-ca configuration, adding a JWT provisioner called admin, and SSH cert support. This is copy-pasteable and will create files in /tmp/step for you to poke around:

> export STEPPATH=/tmp/step && mkdir -p $STEPPATH
> step ca init --name="mafro.dev CA" --provisioner=admin --dns=certs.mafro.dev --address=':443' --ssh
✔ What do you want your password to be? [leave empty and we'll generate one]:
✔ Password: ...

Generating root certificate...
all done!

Generating intermediate certificate...

Generating user and host SSH certificate signing keys...
all done!

✔ Root certificate: /tmp/step/certs/root_ca.crt
✔ Root private key: /tmp/step/secrets/root_ca_key
✔ Root fingerprint: c7641ce4f91993dc3f00000000000000000000000f829c626d20fa02d89600e0
✔ Intermediate certificate: /tmp/step/certs/intermediate_ca.crt
✔ Intermediate private key: /tmp/step/secrets/intermediate_ca_key
✔ SSH user root certificate: /tmp/step/certs/ssh_user_ca_key.pub
✔ SSH user root private key: /tmp/step/secrets/ssh_user_ca_key
✔ SSH host root certificate: /tmp/step/certs/ssh_host_ca_key.pub
✔ SSH host root private key: /tmp/step/secrets/ssh_host_ca_key
✔ Database folder: /tmp/step/db
✔ Templates folder: /tmp/step/templates
✔ Default configuration: /tmp/step/config/defaults.json
✔ Certificate Authority configuration: /tmp/step/config/ca.json

Setup GCP

There are a few requirements and manual steps required to make this work. Follow each section below to ensure you get a working result.

Prerequisites

A GCP Project
A DNS zone defined on your project
A service account for this project for running terraform. Save the key file somewhere safe.
Terraform v0.12.x in your $PATH

Set your GCP project ID into an environment variable, so it can be easily used in the below commands and in the Makefile:

export PROJECT_ID=step-ca-a3dd5f

GCP Project and DNS zone

Set the GCP project ID, and DNS zone in a file named terraform.auto.tfvars, in this format:

project_id = "${PROJECT_ID}"
dns_zone   = "ca"

Use Google Cloud KMS to create and host the CA keys

A beta feature in Smallstep allows us to use private keys generated and hosted by Cloud KMS. This changes the security posture considerably, since there is no raw access to the private keys, only IAM-managed access to use the keys for encryption/signing.

Based on the documentation here, run the following:

$ step-cloudkms-init -credentials-file=$GOOGLE_APPLICATION_CREDENTIALS \
    -location=australia-southeast1 \
    -project=${PROJECT_ID} \
    -ring=keyring-name \
    -ssh
Creating PKI ...
✔ Root Key: projects/a3dd5f/locations/global/keyRings/keyring-name/cryptoKeys/root/cryptoKeyVersions/1
✔ Root Certificate: root_ca.crt
✔ Intermediate Key: projects/a3dd5f/locations/global/keyRings/keyring-name/cryptoKeys/intermediate/cryptoKeyVersions/1
✔ Intermediate Certificate: intermediate_ca.crt

Creating SSH Keys ...
✔ SSH User Public Key: ssh_user_ca_key.pub
✔ SSH User Private Key: projects/a3dd5f/locations/global/keyRings/keyring-name/cryptoKeys/ssh-user-key/cryptoKeyVersions/1
✔ SSH Host Public Key: ssh_host_ca_key.pub
✔ SSH Host Private Key: projects/a3dd5f/locations/global/keyRings/keyring-name/cryptoKeys/ssh-host-key/cryptoKeyVersions/1

Edit ca.json, mapping in the Cloud KMS references according to this mapping:

Config key	KMS reference
`key`	Intermediate Key
`hostKey`	SSH Host Private Key
`userKey`	SSH User Private Key

NB: The GCP project ID is now hardcoded in ca.json, so if you delete and recreate your project, you will need to update the configuration.

Webmaster privilege for Terraform service account

The DNS mapping configuration for Cloud Run is not like normal DNS zones and records. The account which creates the mapping must be an Owner of the domain (or subdomain) in Google's webmaster central.

Add the service account which runs the terraform as an owner of your custom domain, before running the terraform.

Push a working docker image

Cloud Run will not start if the docker image is unavailable in GCR. Solve for that ahead of running terraform with:

docker build -t asia.gcr.io/${PROJECT_ID}/step-ca .
docker push asia.gcr.io/${PROJECT_ID}/step-ca

Run the terraform

Finally, run the terraform:

cd infra
make init
terraform apply

Cloud Run service account

The terraform code creates a service account specific to Cloud Run in infra/cloudrun.tf.

This service account has permission to read the KMS keys necessary to start step-ca.

A key for this service account needs to be included in the docker image for the time being. See this line in the Dockerfile. This is all chicken-and-egg and rather hacky, because I expect it will not be needed long term - the container should be able to authenticate to Google's API automatically.

This final -hack-step means downloading a key for this service account, and building a new docker image with the key baked in :/

SSO for SSH

This section is essentially short-form instructions derived from smallstep.com/blog/diy-single-sign-on-for-ssh.

Smallstep CA can issue certs for use with SSH. By configuring Google oAuth as the identity provider, Google does the authentication for us, and step-ca issues the cert.

┌──────────┐            ┌──────────┐           ┌─ ── ── ── ── ─┐
│          │            │          │
│  Client  │────SSH────▶│  Server  │           │    Google     │
│  (macOS) │            │  (locke) │               oAuth app
│          │            │          │           │               │
└──────────┘            └──────────┘
      │                                        └─ ── ── ── ── ─┘
      │                                                ▲
      │                 ┌──────────┐                   │
    request             │          │                   │
      cert─────────────▶│    CA    │────authenticate───┘
                        │ (ringil) │
                        │          │
                        └──────────┘

Note: The naming convention here is to SSH from the client into the host server.

Setup the Google oAuth app

Configure oAuth consent at https://console.developers.google.com/apis/credentials/consent
Create an oAuth app at https://console.cloud.google.com/apis/credentials a. Click Create credentials, choosing OAuth client ID b. Select Desktop app as application type c. Retain your client ID and client secret

Configure the CA to support this OIDC app

Next, we must configure the CA with a new OIDC provisioner (named "Google") using above secrets. The --domain parameter is your Google SSO domain name.

> step ca provisioner add Google --type=OIDC --ssh \
    --client-id "$OIDC_CLIENT_ID" \
    --client-secret "$OIDC_CLIENT_SECRET" \
    --configuration-endpoint 'https://accounts.google.com/.well-known/openid-configuration' \
    --domain mafro.net
Success! Your `step-ca` config has been updated. To pick up the new configuration SIGHUP (kill -1 <pid>) or restart the step-ca
 process.

Create trust relationship between host server and our CA

Next our CA needs to trust an identity document provided by the host system. In the blog post, the host is an AWS EC2 instance which provides its instance identity to the CA server, and is trusted via the Amazon signature of the AWS account ID (see script here).

On the host server, install the Smallstep CLI tools. Next, bootstrap the step client as usual:

> FINGERPRINT=$(step certificate fingerprint root_ca.crt)
> step ca bootstrap --ca-url https://ringil --fingerprint $FINGERPRINT
The root certificate has been saved in $HOME/.step/certs/root_ca.crt.
Your configuration has been saved in $HOME/.step/config/defaults.json.

Generate a certificate and configure sshd to use it. Run the following as root, so it's possible to write /etc/ssh.

In the following example, the host server is named locke. The steps are:

Generate a token with the admin provisioner
Inspect the token for your amusement

> TOKEN=$(step ca token $(hostname) --ssh --host --provisioner admin)
✔ Provisioner: admin (JWK) [kid: ydABxIT07b0000000000000000000000nGYFRfEGmNA]
✔ Please enter the password to decrypt the provisioner key:
> echo $TOKEN | step crypto jwt inspect --insecure
{
  "header": {
    "alg": "ES256",
    "kid": "ydABxIT07bl-G9jSxfCB45pxNylrKitsnGYFRfEGmNA",
    "typ": "JWT"
  },
  "payload": {
    "aud": "https://ringil:8443/1.0/ssh/sign",
    "exp": 1618046362,
    "iat": 1618046062,
    "iss": "admin",
    "jti": "776b2fce13c90b675f0a1f55712eee80f2504f5f6d4723e0a4fd80e5d35fde40",
    "nbf": 1618046062,
    "sha": "b07c800d7bf36422bd7da01fc2db11efebaafdd5b83092ff82136e75a6d033f9",
    "step": {
      "ssh": {
        "certType": "host",
        "keyID": "locke",
        "principals": [],
        "validAfter": "",
        "validBefore": ""
      }
    },
    "sub": "locke"
  },
  "signature": "E-b6SIaN9atMMo-ICdnoUCjQWMLYuJxkVuB5dBDGjxtzKpPyC-ydnLH5qYV9TTss7MgA2tciMNi9ka-PJ0LNqg"
}
> step ssh certificate $(hostname) /etc/ssh/ssh_host_ecdsa_key.pub --host --sign --provisioner admin --principal $(hostname) --token $TOKEN
✔ CA: https://ringil:8443
✔ Would you like to overwrite /etc/ssh/ssh_host_ecdsa_key-cert.pub [y/n]: y
✔ Certificate: /etc/ssh/ssh_host_ecdsa_key-cert.pub
> step ssh config --host --set Certificate=ssh_host_ecdsa_key-cert.pub --set Key=ssh_host_ecdsa_key
✔ /etc/ssh/sshd_config
✔ /etc/ssh/ca.pub
> systemctl restart sshd

Setup the client to use SSH via OIDC

The following steps are run on the client system, which is connecting to the host configured above.

> FINGERPRINT=$(step certificate fingerprint root_ca.crt)
> step ca bootstrap --ca-url https://ringil --fingerprint $FINGERPRINT
The root certificate has been saved in /Users/blackm/.step/certs/root_ca.crt.
Your configuration has been saved in /Users/blackm/.step/config/defaults.json.
> step ssh config
✔ /Users/mafro/.ssh/config
✔ /Users/mafro/.step/ssh/config
✔ /Users/mafro/.step/ssh/known_hosts

Configure your SSH client config such that step is used to generate the SSH certificate on demand:

> cat ~/.ssh/config
Host locke
    User pi
    UserKnownHostsFile /Users/blackm/.step/ssh/known_hosts
    ProxyCommand step ssh proxycommand %r %h %p --provisioner Google

The Google provisioner is the OIDC one created at the beginning.

Now, using this configuration is as simple as ssh locke, and the OIDC flow is triggered:

> ssh locke
✔ Provisioner: Google (OIDC) [client: 824164598483-frmggjqidnm16kjob9ud8a6a6ahvub1v.apps.googleusercontent.com]
Your default web browser has been opened to visit:

https://accounts.google.com/o/oauth2/v2/auth?<snip>

✔ CA: https://ringil:8443
Linux locke 5.10.17-v7l+ #1414 SMP Fri Apr 30 13:20:47 BST 2021 armv7l

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Thu Jun 17 06:07:51 2021 from 192.168.1.139
pi@locke:~ >

If you wanted to have a peek at your SSH certificate, as provisioned by your CA:

> step ssh list --raw | step ssh inspect
-:
    Type: [email protected] user certificate
    Public key: ECDSA-CERT SHA256:1p9Ux0LVclOe3wFH9ISo+eUiqoAi/CoK7bE/VSdf2r0
    Signing CA: ECDSA SHA256:WoobT5Uoi8cddLhcxILd5eLoPiq27iEaVCDV/oL/B6I
    Key ID: "[email protected]"
    Serial: 8826815887645788865
    Valid: from 2021-06-17T05:44:17 to 2021-06-17T21:44:17
    Principals:
        m
        [email protected]
        mafro
        pi
    Critical Options: (none)
    Extensions:
        permit-agent-forwarding
        permit-port-forwarding
        permit-pty
        permit-user-rc
        permit-X11-forwarding

References for oAuth

Service-specific JWK Provisioner

To auto-provision certificates in a service (such as Cloud Run), we can create a unique JWK provisioner dedicated to just that service. An unencrypted private key will need to be made available on the service - secured in this case in Google KMS.

Setup a JWK Provisioner

This step only needs to be done once, on the host running the CA.

Generate a new keypair and decrypt the keypair's password for securing in KMS, then create the JWK provisioner from that keypair:

> step crypto jwk create proxy-jwk.pub proxy-jwk.key
Please enter the password to encrypt the private JWK:
Your public key has been saved in proxy-jwk.pub.
Your private key has been saved in proxy-jwk.key.
> step crypto jwe decrypt < proxy-jwk.key > proxy-jwk.unencrypted
Please enter the password to decrypt the content encryption key:
> step ca provisioner add HomeAssistantProxy proxy-jwk.key --type JWK
Please enter the password to decrypt proxy-jwk.key:
Please enter the password to encrypt the private JWK:
Success! Your `step-ca` config has been updated. To pick up the new configuration SIGHUP (kill -1 <pid>) or restart the step-ca process.

The .unencrypted file should be stored securely in your application (in this case GCP KMS), and then deleted.

Generate a cert using the JWK provisioner

Use this unencrypted private key to generate your own token, and then certificate, without human interaction:

> TOKEN=$(step ca token token-subject --provisioner HomeAssistantProxy --key proxy-jwk.unencrypted)
✔ Provisioner: HomeAssistantProxy (JWK) [kid: nCdmAqcD-LAdEfMW7qCtqqTBO7z50FQdHvKEzAS_EeY]
> step ca certificate proxy-cert /tmp/client.crt /tmp/client.key --token "$TOKEN" --force
✔ CA: https://ringil:8443
✔ Certificate: /tmp/client.crt
✔ Private Key: /tmp/client.key

You can see this in action in the nginx mTLS proxy.

References

https://smallstep.com/blog/step-certificates/#using-certificates-with-tls https://smallstep.com/blog/diy-single-sign-on-for-ssh/ https://gitter.im/smallstep/community

GCE Doco

Some notes and recipes for useful things you can do in GCE.

Running a managed docker image on a VM

GCE can be configured to run a docker container on VM startup - which is a neat way to continue to use docker for development, but target a VM in production.

The terraform-google-container-vm module generates the metadata for a VM instance template, users just need to see how to configure the module by using the examples.

module vm_container {
  source = "github.com/terraform-google-modules/terraform-google-container-vm?ref=v2.0.0"

  container = {
    image = format("asia.gcr.io/%s/step-ca", data.google_project.project.project_id)
  }
  restart_policy = "Always"
}

resource google_compute_instance_template tpl {
  region   = var.region
  project  = data.google_project.project.project_id

  machine_type = "e2-micro"
  metadata     = {
    gce-container-declaration: module.vm_container.metadata_value
  }
}

https://cloud.google.com/compute/docs/containers/deploying-containers

Testing a new docker image without a VM restart

One can quite easily test a new docker image by restarting the konlet service. Assuming the latest docker image has been updated on the registry:

IMAGE_ID=$(docker ps --format '{{.ID}}' --filter 'ancestor=asia.gcr.io/step-ca-a3dd5f/step-ca')
docker rm -f $IMAGE_ID
docker pull asia.gcr.io/step-ca-a3dd5f/step-ca
sudo systemctl restart konlet-startup

Mounting a host volume into the docker image

The terraform-google-container-vm module comes with quite a few useful examples, but the following recipe is missing:

module vm_container {
  source = "github.com/terraform-google-modules/terraform-google-container-vm?ref=v2.0.0"

  container = {
    image = format("asia.gcr.io/%s/step-ca", data.google_project.project.project_id)

    volumeMounts = [
      {
        name      = "db"
        mountPath = "/root/.step/db"
        readOnly  = false
      },
    ]
  }

  volumes = [
    {
      name = "db"
      hostPath = {
        path = "/home/db"
      }
    },
  ]
}

konlet source which helped to figure this out

Quick SSH via IAP

You can use Google's Identity-Aware Proxy to help with managing SSH access to VMs in GCE. Ensure you have the right port open on the firewall:

resource google_compute_firewall iap_ssh {
  project = google_project.project.project_id
  network = google_compute_network.network.self_link
  name    = "allow-ssh-ingress-from-iap"

  allow {
    protocol = "tcp"
    ports    = ["22"]
  }

  source_ranges = ["35.235.240.0/20"]
}

And then simply use gcloud to connect:

gcloud compute ssh ca-x  --tunnel-through-iap --zone australia-southeast1-c

Using toolbox on COS

After logging into a GCE instance in your shell, use the toolbox command to fetch and run a debian-based docker image handy for debugging.

mafro@ca-c3d8 ~ $ toolbox
20200603-00: Pulling from google-containers/toolbox
1c6172af85ee: Pull complete
a4b5cec33934: Pull complete
b7417d4f55be: Pull complete
fed60196983f: Pull complete
8e1533dfae69: Pull complete
112bf8e3d384: Pull complete
1df10c12cc15: Pull complete
b33e020bb38a: Pull complete
938e6be48196: Pull complete
Digest: sha256:36e2f6b8aa40328453aed7917860a8dee746c101dfde4464ce173ed402c1ec57
Status: Downloaded newer image for gcr.io/google-containers/toolbox:20200603-00
gcr.io/google-containers/toolbox:20200603-00
0877997d383a6317d60d0ef76af1f5f914e793f4a65b84094bdec09c284e22c3
mafro-gcr.io_google-containers_toolbox-20200603-00
Please do not use --share-system anymore, use $SYSTEMD_NSPAWN_SHARE_instead.
Spawning container mafro-gcr.io_google-containers_toolbox-20200603-00 on /var/lib/toolbox/mafro-gcr.io_google-containers_toolbox-20200603-00.
Press ^] three times within 1s to kill container.
root@ca-c3d8:~#

https://cloud.google.com/container-optimized-os/docs/how-to/toolbox

Using gsutil on Container-optimised OS

As container-optimised OS does not come with gcloud and friends, the easiest solution is to simply user docker:

docker run --rm google/cloud-sdk:alpine gsutil --help

Configuring a startup/shutdown down script via Terraform

A simple metadata key configures a startup/shutdown script:

resource google_compute_instance_template tpl {
  region   = var.region
  project  = data.google_project.project.project_id

  machine_type = "e2-micro"
  metadata     = {
    gce-container-declaration: module.vm_container.metadata_value
    shutdown-script: file("preempt.sh")
    startup-script:  file("startup.sh")
  }

...

https://cloud.google.com/compute/docs/startupscript

Testing a startup/shutdown down script (in COS)

You can test a startup script in Container-optimised OS with the following command. Substitute shutdown to test the shutdown script.

sudo google_metadata_script_runner --script-type startup --debug

https://cloud.google.com/compute/docs/startupscript#on_container-optimized_os_ubuntu_and_sles_images

Mounting a GCS bucket via fuse

The included docker-entrypoint.sh shows mounting a GCS bucket before a docker application starts up.

The build steps to make gcsfuse binary available are in the Dockerfile.

https://serverfault.com/a/968639/89669

mafrosis / step-ca-on-gce Goto Github PK

step-ca-on-gce's Introduction