For this example we assume a scenario with two clusters: staging and production. The end goal is to leverage Flux and Kustomize to manage both clusters while minimizing duplicated declarations.

We will configure Flux to install, test and upgrade a demo app using HelmRepository and HelmRelease custom resources. Flux will monitor the Helm repository, and it will automatically upgrade the Helm releases to their latest chart version based on semver ranges.

Request a Gloo Edge Enterprise trial license. This is required to enable the enterprise features of the API Gateway, e.g. Authentication, Rate Limiting, Transformation, etc.

You will need a Kubernetes cluster version 1.16 or newer and kubectl version 1.18. For a quick local test, you can use Kubernetes kind. Any other Kubernetes setup will work as well though.

In order to follow the guide you'll need a GitHub account and a personal access token that can create repositories (check all permissions under repo).

Install the Flux CLI on MacOS and Linux using Homebrew:

brew install fluxcd/tap/flux

Or install the CLI by downloading precompiled binaries using a Bash script:

curl -s https://fluxcd.io/install.sh | sudo bash

The Git repository contains the following top directories:

• apps dir contains kubernetes application manifests, which are then customized per cluster.
• infrastructure dir contains common infrastructure services such as datastores, ingress controllers, etc. and Helm repository definitions. In this case Gloo Edge Enterprise
• clusters dir contains the Flux configuration per cluster

├── apps
│   ├── base
│   ├── production 
│   └── staging
├── infrastructure
│   ├── glooedge
│   ├── keycloak
│   └── sources
└── clusters
    ├── production
    └── staging

The apps configuration is structured into:

• apps/base/ dir contains namespaces and Helm release definitions
• apps/production/ dir contains the production Helm release values
• apps/staging/ dir contains the staging values

├── apps
│   ├── base
│   │   └── petclinic
│   │       ├── kustomization.yaml
│   │       ├── namespace.yaml
│   │       ├── service-app.yaml
│   │       ├── service-db.yaml
│   │       ├── statefulset-app.yaml
│   │       ├── statefulset-db.yaml
│   │       └── virtualservice.yaml
│  ├── production
│  │   ├── kustomization.yaml
│   └── staging
│       ├── kustomization.yaml
│       ├── petclinic
│       │   ├── authconfig.yaml
│       │   └── kustomization.yaml
│       └── ratelimit.yaml

In apps/base/petclinic/ dir we have a manifests with common values for both clusters:

apiVersion: gateway.solo.io/v1
kind: VirtualService
metadata:
  name: petclinic
  namespace: gloo-system
spec:
  virtualHost:
    domains:
      - '*'
    routes:
      - matchers:
         - prefix: /
        routeAction:
          single:
            upstream:
              name: default-petclinic-80
              namespace: gloo-system

In apps/staging/ dir we have a Kustomize patch with the staging specific values:

apiVersion: enterprise.gloo.solo.io/v1
kind: AuthConfig
metadata:
  name: oauth
  namespace: gloo-system
spec:
  configs:
  - oauth2:
      oidcAuthorizationCode:
        appUrl: https://192.168.64.51
        callbackPath: /callback
        clientId: 2504c599-fcab-4174-9164-a77d3fb5db14 
        clientSecretRef:
          name: oauth
          namespace: gloo-system
        issuerUrl: "http://192.168.64.50:8080/auth/realms/master/"
        scopes:
        - email
        headers:
          idTokenHeader: jwt

Note: For each environment or cluster you may change the OIDC configuration.

Infrastructure:

./infrastructure/
│   ├── glooedge
│   │   ├── kustomization.yaml
│   │   ├── namespace.yaml
│   │   └── release.yaml
│   ├── keycloak
│   │   ├── deployment.yaml
│   │   ├── keycloak.yaml
│   │   ├── kustomization.yaml
│   │   ├── namespace.yaml
│   │   └── service.yaml
│   ├── kustomization.yaml
│   └── sources
│       ├── glooedge.yaml
│       └── kustomization.yaml

In infrastructure/sources/ dir we have the Gloo Edge Enterprise Helm repositories definitions:

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: HelmRepository
metadata:
  name: glooedge
spec:
  interval: 30m
  url: https://storage.googleapis.com/gloo-public-helm

Note that with interval: 30m we configure Flux to pull the Helm repository index every 30 minutes.

The clusters dir contains the Flux configuration:

./clusters/
├── production
│   ├── apps.yaml
│   └── infrastructure.yaml
└── staging
    ├── apps.yaml
    └── infrastructure.yaml

In clusters/staging/ dir we have the Kustomization definitions:

apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
  name: apps
  namespace: flux-system
spec:
  interval: 10m0s
  dependsOn:
    - name: infrastructure
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./apps/staging
  prune: true
  validation: client
  timeout: 2m
---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
  name: infrastructure
  namespace: flux-system
spec:
  interval: 10m0s
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./infrastructure
  prune: true
  validation: client
  healthChecks:
    - apiVersion: apps/v1
      kind: Deployment
      name: gloo
      namespace: gloo-system
  timeout: 10m

Note that with path: ./apps/staging we configure Flux to sync the staging Kustomize overlay and with dependsOn we tell Flux to create the infrastructure items before deploying the apps.

Fork this repository on your personal GitHub account and export your GitHub access token, username and repo name:

export GITHUB_TOKEN=<your-token>
export GITHUB_USER=<your-username>
export GITHUB_REPO=<repository-name>

Verify that your staging cluster satisfies the prerequisites with:

flux check --pre

Update infrastructure/glooedge/release.yaml license_key with your trial license:

  install:
    remediation:
      retries: 5 
  values:
    license_key: "eyJl.........."
    gloo:
      gloo:
        deployment:
          image:
            tag: 1.8.6

Set the kubectl context to your staging cluster (in this case it's a local microk8s) and bootstrap Flux:

flux bootstrap github \
    --context=microk8s \
    --owner=${GITHUB_USER} \
    --repository=${GITHUB_REPO} \
    --token-auth \
    --branch=main \
    --personal \
    --path=clusters/staging

The bootstrap command commits the manifests for the Flux components in clusters/staging/flux-system dir and creates a deploy key with read-only access on GitHub, so it can pull changes inside the cluster.

Wait for the Gloo Edge, Keycloak and Petclinic application installations to complete on staging:

kubectl get po -n flux-system
NAME                                       READY   STATUS    RESTARTS   AGE
helm-controller-7b58d45956-c6dl9           1/1     Running   0          18m
notification-controller-5f4f4967b9-9v5pw   1/1     Running   0          18m
kustomize-controller-bb6c99658-8pf9k       1/1     Running   0          18m
source-controller-6448c7989-wwgn6          1/1     Running   0          18m

kubectl get po -n gloo-system
NAME                                                  READY   STATUS    RESTARTS   AGE
redis-d7786cb9d-tqtbp                                 1/1     Running   0          8m23s
gloo-58cf777ffb-n86qp                                 1/1     Running   0          8m23s
gateway-proxy-659c5cdc96-8rvl8                        1/1     Running   0          8m22s
discovery-89cc57dbd-rqp6k                             1/1     Running   0          8m23s
observability-6894c76ddb-qgtwg                        1/1     Running   0          8m22s
gloo-fed-7b9ff9c44-m7x46                              1/1     Running   0          8m22s
glooe-prometheus-kube-state-metrics-85c5fcb4c-xp7qb   1/1     Running   0          8m23s
gateway-7dbd774b84-xjdrk                              1/1     Running   0          8m23s
glooe-grafana-75dfc55d4d-tgtrb                        1/1     Running   0          8m23s
extauth-6489bc55f9-6mldc                              1/1     Running   0          8m22s
gloo-fed-console-65859578c7-wqtxz                     3/3     Running   0          8m22s
glooe-prometheus-server-d99d4477b-8vcr9               2/2     Running   0          8m22s
rate-limit-5dfb7b4cbb-b47cm                           1/1     Running   1          8m22s

kubectl get po -n default
NAME                       READY   STATUS    RESTARTS   AGE
keycloak-d68dbc7fb-psn99   1/1     Running   0          16m
petclinic-0                1/1     Running   0          12m
petclinic-db-0             1/1     Running   0          12m

Verify Gloo Edge proxy service EXTERNAL-IP can be accessed, in this case 192.168.64.51:

kubectl -n gloo-system get service/gateway-proxy
NAME            TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                      AGE
gateway-proxy   LoadBalancer   10.152.183.20   192.168.64.51   80:31344/TCP,443:32573/TCP   11m

Clone your personal forked repo locally and cd into the folder. We will now finalise the configuration and start to add TLS encryption, Authentication, Rate Limiting and Transformation capabilties to the Petclinic application via Git commits.

First finalise Keycloak (OIDC) configurtaion by running ./scripts/setup.sh We'll use the APP_URL, KEYCLOAK_URL & client outputs to configure the AuthConfig later.

./scripts/setup.sh
Generating a 2048 bit RSA private key
...............+++
..........+++
writing new private key to 'tls.key'
-----
secret/upstream-tls created
TLS certificate secret created
### Petclinic URL for AuthConfig ###
APP_URL: https://192.168.64.51
### Keycloak endpoint for AuthConfig ###
KEYCLOAK_URL: http://192.168.64.50:8080/auth
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1803  100  1734  100    69   3000    119 --:--:-- --:--:-- --:--:--  3119
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   571  100   542  100    29   4839    258 --:--:-- --:--:-- --:--:--  5053
### Keycloak Client ID for AuthConfig ###
client: 5b6b138b-1e1d-44ea-82e2-081f1d4de695
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1371  100  1317  100    54  23105    947 --:--:-- --:--:-- --:--:-- 24052
Keycloak users created
+--------+-------+
| SECRET | TYPE  |
+--------+-------+
| oauth  | OAuth |
+--------+-------+
Keycloak configuration complete

At this stage you should be able to open the Petclinic application in your browser, http://192.168.64.51 in this case.

• Add TLS Encryption The setup script created the required certificate and kubernetes secret. We edit apps/base/petclinic/virtualservice.yaml and remove the comments for the SSL Config section

spec:
# ---------------- SSL config ---------------------------
  sslConfig:
    secretRef:
      name: upstream-tls
      namespace: gloo-system
# -------------------------------------------------------
  virtualHost:

Note: Leave the rest of the file as is. We'll be returning to this file a couple of times to remove the comments when need be.

Now all we need to do is commit the change to Github and wait for Fluxcd to reconsile the change.

git add -A && git commit -m "staging update" && git push

You will now have to use HTTPS (https://192.168.64.51/) to access Petclinic. It's a self signed certificate, so accept the certificate warnings in your browser.

• Add Authentication The setup script configured Keycloak with users, but we need to update apps/staging/petclinic/authconfig.yaml for your environment. Then we'll update apps/staging/petclinic/kustomization.yaml to apply the configuration.

Find APP_URL, KEYCLOAK_URL & client from the setup.sh output.

### Petclinic URL for AuthConfig ###
APP_URL: https://192.168.64.51
### Keycloak endpoint for AuthConfig ###
KEYCLOAK_URL: http://192.168.64.50:8080/auth
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1803  100  1734  100    69   3000    119 --:--:-- --:--:-- --:--:--  3119
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   571  100   542  100    29   4839    258 --:--:-- --:--:-- --:--:--  5053
### Keycloak Client ID for AuthConfig ###
client: 5b6b138b-1e1d-44ea-82e2-081f1d4de695

In this example apps/staging/petclinic/authconfig.yaml needs to be updated as follows:

apiVersion: enterprise.gloo.solo.io/v1
kind: AuthConfig
metadata:
  name: oauth
  namespace: gloo-system
spec:
  configs:
  - oauth2:
      oidcAuthorizationCode:
        appUrl: https://192.168.64.51
        callbackPath: /callback
        clientId: 5b6b138b-1e1d-44ea-82e2-081f1d4de695
        clientSecretRef:
          name: oauth
          namespace: gloo-system
        issuerUrl: "http://192.168.64.50:8080/auth/realms/master/"
        scopes:
        - email
        headers:
          idTokenHeader: jwt

And then update apps/staging/petclinic/kustomization.yaml by removing the comment in front of authconfig.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base/petclinic
  - authconfig.yaml

Lastly we update the application virtual service to use the OIDC service. Remove the ## commments from apps/base/petclinic/virtualservice.yaml

    routes:
      - matchers:
         - prefix: /
# ------------------- OIDC - Only applied to this matcher -------------------
        options:
          extauth:
            configRef:
              name: oauth
              namespace: gloo-system
# ---------------------------------------------------------------------------
# ---------------- Rate limit config ------------------

❯ git status
On branch main
Your branch is up to date with 'origin/main'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
	modified:   apps/base/petclinic/virtualservice.yaml
	modified:   apps/staging/petclinic/authconfig.yaml
	modified:   apps/staging/petclinic/kustomization.yaml

git add -A && git commit -m "staging update" && git push

Once reconciled the application now has authentication.

Login using Username: user1 and Password: password

• Add Rate Limiting The rate limiting configuration in this example is global, so we'll apply it at the staging cluster level via apps/staging/ratelimit.yaml and apps/staging/kustomization.yaml. Enable the rate limiting configuration via apps/staging/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- petclinic
- ratelimit.yaml

We also edit the virtual service again, to enable the service for the application. Remove the ### commments from apps/base/petclinic/virtualservice.yaml

              namespace: gloo-system
# ---------------------------------------------------------------------------
# ---------------- Rate limit config ------------------
          rateLimitConfigs:
            refs:
            - name: global-limit
              namespace: gloo-system
#------------------------------------------------------
# ---------------- Transformation ------------------

git add -A && git commit -m "staging update" && git push

After a few refreshes of the application in the browser we see rate limiting being applied. As seen in the developer view, HTTP code 429 "Too Many Requests"

• Add a Transformation Lets use the API Gateway to transform the 429 response body. We update apps/base/petclinic/virtualservice.yaml. Remove the #### commments from apps/base/petclinic/virtualservice.yaml

              namespace: gloo-system
#------------------------------------------------------
# ---------------- Transformation ------------------
          transformations:
            responseTransformation:
              transformationTemplate:
                parseBodyBehavior: DontParse
                body:    
                  text: '{% if header(":status") == "429" %}<html><body style="background-color:powderblue;"><h1>Too many Requests!</h1><p>Try again after 1 minute</p></body></html>{% else %}{{ body() }}{% endif %}'
#---------------------------------------------------
        routeAction:

git add -A && git commit -m "staging update" && git push

Once reconciled we now get a more user friendly rate limiting message.

That is it. Eventually the completed virtual service apps/base/petclinic/virtualservice.yaml should look like this.

  namespace: gloo-system
spec:
# ---------------- SSL config ---------------------------
  sslConfig:
    secretRef:
      name: upstream-tls
      namespace: gloo-system
# -------------------------------------------------------
  virtualHost:
    domains:
      - '*'
    routes:
      - matchers:
         - prefix: /
# ------------------- OIDC - Only applied to this matcher -------------------
        options:
          extauth:
            configRef:
              name: oauth
              namespace: gloo-system
# ---------------------------------------------------------------------------
# ---------------- Rate limit config ------------------
          rateLimitConfigs:
            refs:
            - name: global-limit
              namespace: gloo-system
#------------------------------------------------------
# ---------------- Transformation ------------------
          transformations:
           responseTransformation:
              transformationTemplate:
                parseBodyBehavior: DontParse
                body:
                  text: '{% if header(":status") == "429" %}<html><body style="background-color:powderblue;"><h1>Too many Requests!</h1><p>Try again after 1 minute</p></body></html>{% else %}{{ body() }}{% endif %}'
#---------------------------------------------------
        routeAction:
          single:

As you clone the cluster from here, you wont have to go through the staged configuration of the application's API Gateway capabilities. We did that here to illustrate GitOps based configuration of the API Gateway. Simple flux bootstrap with the new cluster context and everything will be provisoned to this state on the new cluster. You will have to run the setup script to configure Keycloak.