This Demo repository shows how to deploy and configure Tekton on Amazon EKS and integrate Tekton with GitLab (especially https://gitlab.com/jonashackt/microservice-api-spring-boot).
Let's simply roll out a AWS EKS cluster with Pulumi:
https://www.pulumi.com/docs/guides/crosswalk/aws/eks/
Our eks-deployment/index.ts looks like this:
import * as eks from "@pulumi/eks";
// Create an EKS cluster with the default configuration.
const cluster = new eks.Cluster("eks-for-tekton");
// Export the cluster's kubeconfig.
export const kubeconfig = cluster.kubeconfig;
export const eksUrl = cluster.eksCluster.endpoint;
To execute our Pulumi program, be sure to be logged into the correct Account at https://app.pulumi.com/your-account-here via pulumi login
using your Pulumi account's token (do a pulumi logout
before, if you're already logged into another Pulumi account).
Now select the correct stack and fire up Pulumi with:
pulumi stack select dev
pulumi up
After your EKS cluster has been setup correctly, use the kubeconfig
const exported inside our Pulumi program to create the kubeconfig.yml
:
pulumi stack output kubeconfig > kubeconfig.yml
To access the cluster be sure to have kubectl
installed. Try accessing it with:
kubectl --kubeconfig kubeconfig.yml get nodes
For merging the new kubeconfig into your systems profile see https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/
For example you could do:
pulumi stack output kubeconfig > ~/.kube/config-eks-for-tekton
export KUBECONFIG=~/.kube/config:~/.kube/config-eks-for-tekton
Now access via kubectx
is also possible.
First we need to create GitHub repository secrets containing our AWS API key id & access key (AWS_ACCESS_KEY_ID
& AWS_SECRET_ACCESS_KEY
) and our Pulumi access token (PULUMI_ACCESS_TOKEN
):
Our provision.yml workflow uses Pulumi like we did locally:
name: provision
on: [push]
jobs:
provision-aws-eks:
runs-on: ubuntu-latest
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}
AWS_DEFAULT_REGION: 'eu-central-1'
# Create an GitHub environment referencing our EKS cluster endpoint
environment:
name: tekton-flux-eks-pulumi-dev
url: ${{ steps.pulumi-up.outputs.eks_url }}
steps:
- name: Checkout
uses: actions/checkout@master
- name: Setup node env
uses: actions/[email protected]
with:
node-version: '14'
- name: Cache node_modules
uses: actions/cache@v2
with:
path: ~/.npm
key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
restore-keys: |
${{ runner.os }}-node-
- name: Install Pulumi dependencies before npm run generate to prevent it from breaking the build
run: npm install
working-directory: ./eks-deployment
- name: Install Pulumi CLI
uses: pulumi/[email protected]
- name: Provision AWS EKS cluster with Pulumi
id: pulumi-up
run: |
pulumi stack select dev
pulumi preview
echo "lets use --suppress-outputs here in order to prevent Pulumi from logging the kubeconfig into public GitHub Action logs"
pulumi up --yes --suppress-outputs
pulumi stack output kubeconfig > kubeconfig.yml
echo "::set-output name=eks_url::$(pulumi stack output eksUrl)/api/hello"
working-directory: ./eks-deployment
- name: Try to connect to our EKS cluster using kubectl
run: kubectl --kubeconfig kubeconfig.yml get nodes
working-directory: ./eks-deployment
Mind to use --suppress-outputs
flag for our pulumi up
to prevent the kubeconfig
from getting logged unmasked.
We also export our eks endpoint url
as an GitHub Environment (as described here).
I got this error (see log):
...
<botocore.awsrequest.AWSRequest object at 0x7f067c580670>
Unable to connect to the server: getting credentials: exec: executable aws failed with exit code 255
Error: Process completed with exit code 1.
Luckily this answer brought me into the right direction: https://stackoverflow.com/a/59184490/4964553
I needed to define the AWS_DEFAULT_REGION: 'eu-central-1'
also solely for kubectl
in GitHub Actions. With this the error was gone, since the other two variables for aws-cli
(which is already installed in the GitHub Actions virtual environment) were defined properly.
https://tekton.dev/docs/getting-started/
Buildpacks: https://buildpacks.io/docs/tools/tekton/
https://tekton.dev/docs/getting-started/#installation
So let's add the installation and wait for Tekton to become available:
...
- name: Install Tekton Pipelines
run: kubectl apply --filename https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml
- name: Wait for Tekton to become ready & show running Tekton pods
run: |
kubectl wait --for=condition=ready pod -l app=tekton-pipelines-controller --namespace tekton-pipelines
kubectl get pods --namespace tekton-pipelines
https://tekton.dev/docs/getting-started/#persistent-volumes
Let's check if our EKS cluster already has a StorageClass
defined with kubectl get storageclasses
:
$ kubectl get storageclasses
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
gp2 (default) kubernetes.io/aws-ebs Delete WaitForFirstConsumer false 179m
Before creating it let's check if there is already a ConfigMap
defined:
kubectl describe configmap config-artifact-pvc -n tekton-pipelines
From the docs:
Your Kubernetes cluster, such as one from Google Kubernetes Engine, may have persistent volumes set up at the time of creation, thus no extra step is required
If there's no Persistens Volume defined, we need to create a ConfigMap
which defines the Persistent Volume Tekton will request:
kubectl create configmap config-artifact-pvc \
--from-literal=size=10Gi \
--from-literal=storageClassName=gp2 \
-o yaml -n tekton-pipelines \
--dry-run=client | kubectl replace -f -
Install the Tekton CLI e.g. via homebrew:
brew tap tektoncd/tools
brew install tektoncd/tools/tektoncd-cli
See the task-hello-world.yaml:
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
name: hello
spec:
steps:
- name: hello
image: ubuntu
command:
- echo
args:
- "Hello World!"
Let's apply it to our cluster:
kubectl apply -f tekton-ci-config/task-hello-world.yaml
Let's show our newly created task:
$ tkn task list
NAMESPACE NAME DESCRIPTION AGE
default hello 24 seconds ago
Now this is only a Tekton Task definition. We need another Tekton object - the TaskRun
- in order to run our Task. Create it with:
tkn task start hello
Follow the logs of the TaskRun with:
tkn taskrun logs --last -f
https://tekton.dev/docs/dashboard/
Install it with:
kubectl apply --filename https://github.com/tektoncd/dashboard/releases/latest/download/tekton-dashboard-release.yaml
Now as we already ran some Tasks let's have a look into the Tekton dashboard:
kubectl proxy --port=8080
Then open your Browser at http://localhost:8080/api/v1/namespaces/tekton-pipelines/services/tekton-dashboard:http/proxy/
The ultra simple (but only PoC grade) solution to expose an public endpoint on a Cloud Provider's managed K8s is to use a Service
with type LoadBalancer
. As the docs state (https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer):
On cloud providers which support external load balancers, setting the type field to LoadBalancer provisions a load balancer for your Service. The actual creation of the load balancer happens asynchronously, and information about the provisioned balancer is published in the Service's .status.loadBalancer field.
And also the AWS EKS docs state this (https://docs.aws.amazon.com/eks/latest/userguide/network-load-balancing.html):
When you create a Kubernetes Service of type LoadBalancer, an AWS Network Load Balancer (NLB) is provisioned that load balances network traffic.
So this would be our ultra simple Service to access our Tekton Dashboard (see tekton-dashboard-service.yml):
apiVersion: v1
kind: Service
metadata:
name: tekton-dashboard-external-svc-manual
spec:
selector:
app: tekton-dashboard
ports:
- protocol: TCP
port: 80
targetPort: 9097
type: LoadBalancer
But it's not that easy to simply apply it and grab the url. Have a look inside our provision.yml for all commands in working order. Here are the steps:
First we apply
our tekton-dashboard-service.yml:
kubectl apply -f tekton-dashboard-service.yml -n tekton-pipelines
Then we have to wait until our AWS ELB LoadBalancer is provisioned. As stated the
...actual creation of the load balancer happens asynchronously, and information about the provisioned balancer is published in the Service's .status.loadBalancer field.
If the LoadBalancer is readily provisioned, a kubectl get service/tekton-dashboard-external-svc-manual --output=yaml
shows us the following:
apiVersion: v1
kind: Service
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"tekton-dashboard-external-svc-manual","namespace":"tekton-pipelines"},"spec":{"ports":[{"port":80,"protocol":"TCP","targetPort":9097}],"selector":{"app":"tekton-dashboard"},"type":"LoadBalancer"}}
creationTimestamp: "2021-11-23T09:07:27Z"
finalizers:
- service.kubernetes.io/load-balancer-cleanup
name: tekton-dashboard-external-svc-manual
namespace: tekton-pipelines
resourceVersion: "677614"
uid: 26431f31-0b27-4df3-a6cd-3d32f825cd5f
spec:
clusterIP: 10.100.42.167
clusterIPs:
- 10.100.42.167
externalTrafficPolicy: Cluster
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
- nodePort: 31306
port: 80
protocol: TCP
targetPort: 9097
selector:
app: tekton-dashboard
sessionAffinity: None
type: LoadBalancer
status:
loadBalancer:
ingress:
- hostname: a26431f310b274df3a6cd3d32f825cd5-1729774979.eu-central-1.elb.amazonaws.com
So we have to somehow wait for the field status.loadBalancer
to contain the ingress[0].hostname
(remember to use the list selector [0]
here, since hostname
is represented as list).
But since the field ingress[0].hostname
isn't present at first, a direct access via kubectl get service tekton-dashboard-external-svc-manual -n tekton-pipelines --output=jsonpath='{.status.loadBalancer.ingress[0].hostname}'
would result in zero results.
As kubectl wait
isn't capable of beeing used without status.conditions
fields present right now (see the issue kubectl wait on arbitrary jsonpath, the solution is beeing merged into v1.23
(see this PR kubernetes/kubernetes#105776)), we need to find an alternative way for now.
For a solution see this so answer https://stackoverflow.com/questions/70108499/kubectl-wait-for-service-on-aws-eks-to-expose-elastic-load-balancer-elb-addres
Therefore the answer "watch" the output of a command until a particular string is observed and then exit comes in handy:
until my_cmd | grep "String Im Looking For"; do : ; done
If we use this approach together with a kubectl get
we can craft a command which will wait until the field ingress
gets populated into the status.loadBalancer
field in our Service:
until kubectl get service/tekton-dashboard-external-svc-manual -n tekton-pipelines --output=jsonpath='{.status.loadBalancer}' | grep "ingress"; do : ; done
Now that our AWS ELB is provisioned we can finally grad it's URL with:
kubectl get service tekton-dashboard-external-svc-manual -n tekton-pipelines --output=jsonpath='{.status.loadBalancer.ingress[0].hostname}'
Then it's easy to use the output and create a GitHub step variable, which is used in the GitHub Environment definition at the top of the job:
echo "--- Create GitHub environment var"
DASHBOARD_HOST=$(kubectl get service tekton-dashboard-external-svc-manual -n tekton-pipelines --output=jsonpath='{.status.loadBalancer.ingress[0].hostname}')
echo "The Tekton dashboard is accessible at $DASHBOARD_HOST - creating GitHub Environment"
echo "::set-output name=dashboard_host::http://$DASHBOARD_HOST"
https://buildpacks.io/docs/tools/tekton/
Install git clone and buildpacks Task:
kubectl apply -f https://raw.githubusercontent.com/tektoncd/catalog/master/task/git-clone/0.4/git-clone.yaml
kubectl apply -f https://raw.githubusercontent.com/tektoncd/catalog/master/task/buildpacks/0.3/buildpacks.yaml
To access the GitLab Container Registry we need to first create a PAT or deploy token (see https://docs.gitlab.com/ee/user/packages/container_registry/#authenticate-with-the-container-registry)
Go to Settings/Repository
inside your GitLab repo - for me this is https://gitlab.com/jonashackt/microservice-api-spring-boot/-/settings/repository
There create a token TektonBuildpacksToken
under Deploy tokens
with a username gitlab-token
and read_registry
& write_registry
access.
Now create GitHub Repository Secrets called GITLAB_CR_USER
and GITLAB_CR_PASSWORD
accordingly with the Tokens username and token.
Finally we can create our Secret inside our GitHub Actions pipeline:
https://buildpacks.io/docs/tools/tekton/#42-authorization
kubectl create secret docker-registry docker-user-pass \
--docker-server=registry.gitlab.com \
--docker-username=${{ secrets.GITLAB_CR_USER }} \
--docker-password=${{ secrets.GITLAB_CR_PASSWORD }} \
--namespace default
After the first successful secret creation, we sadly get the error error: failed to create secret secrets "docker-user-pass" already exists
- which is correct, since the secret already exists.
But there's help (see https://stackoverflow.com/a/45881259/4964553): We add --save-config --dry-run=client -o yaml | kubectl apply -f -
to our command like this:
kubectl create secret docker-registry docker-user-pass \
--docker-server=registry.gitlab.com \
--docker-username=${{ secrets.GITLAB_CR_USER }} \
--docker-password=${{ secrets.GITLAB_CR_PASSWORD }} \
--namespace default \
--save-config --dry-run=client -o yaml | kubectl apply -f -
Now we made an apply
out of our create
kubectl command, which we can use repetitively :)
We also need to create a ServiceAccount
that uses this secret as ghcr-service-account.yml
apiVersion: v1
kind: ServiceAccount
metadata:
name: buildpacks-service-account
secrets:
- name: docker-user-pass
https://buildpacks.io/docs/tools/tekton/#41-pvcs
Create new resources.yml:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: buildpacks-source-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 500Mi
Create pipeline.yml:
apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
name: buildpacks-test-pipeline
spec:
params:
- name: image
type: string
description: image URL to push
workspaces:
- name: source-workspace # Directory where application source is located. (REQUIRED)
- name: cache-workspace # Directory where cache is stored (OPTIONAL)
tasks:
- name: fetch-repository # This task fetches a repository from github, using the `git-clone` task you installed
taskRef:
name: git-clone
workspaces:
- name: output
workspace: source-workspace
params:
- name: url
value: https://github.com/buildpacks/samples
- name: subdirectory
value: ""
- name: deleteExisting
value: "true"
- name: buildpacks # This task uses the `buildpacks` task to build the application
taskRef:
name: buildpacks
runAfter:
- fetch-repository
workspaces:
- name: source
workspace: source-workspace
- name: cache
workspace: cache-workspace
params:
- name: APP_IMAGE
value: "$(params.image)"
- name: SOURCE_SUBPATH
value: "apps/java-maven" # This is the path within the samples repo you want to build (OPTIONAL, default: "")
- name: BUILDER_IMAGE
value: paketobuildpacks/builder:base # This is the builder we want the task to use (REQUIRED)
- name: display-results
runAfter:
- buildpacks
taskSpec:
steps:
- name: print
image: docker.io/library/bash:5.1.4@sha256:b208215a4655538be652b2769d82e576bc4d0a2bb132144c060efc5be8c3f5d6
script: |
#!/usr/bin/env bash
set -e
echo "Digest of created app image: $(params.DIGEST)"
params:
- name: DIGEST
params:
- name: DIGEST
value: $(tasks.buildpacks.results.APP_IMAGE_DIGEST)
And now apply all three configs with:
kubectl apply -f tekton-ci-config/resources.yml -f tekton-ci-config/ghcr-service-account.yml -f tekton-ci-config/pipeline.yml
Create pipeline-run.yml:
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
generateName: buildpacks-test-pipeline-run-
spec:
serviceAccountName: buildpacks-service-account-gitlab # Only needed if you set up authorization
pipelineRef:
name: buildpacks-test-pipeline
workspaces:
- name: source-workspace
subPath: source
persistentVolumeClaim:
claimName: buildpacks-source-pvc
- name: cache-workspace
subPath: cache
persistentVolumeClaim:
claimName: buildpacks-source-pvc
params:
- name: image
value: registry.gitlab.com/jonashackt/microservice-api-spring-boot # This defines the name of output image
A crucial point here is to change the metadata: name: buildpacks-test-pipeline-run
into metadata: generateName: buildpacks-test-pipeline-run-
. Why? Because if we use the name
parameter every kubectl apply
tries to create the PipelineRun
object with the same name which results in errors like this:
Error from server (AlreadyExists): error when creating "tekton-ci-config/pipeline-run.yml": pipelineruns.tekton.dev "buildpacks-test-pipeline-run" already exists
Using the generateName
field fixes our problem (see https://stackoverflow.com/questions/69880096/how-to-restart-tekton-pipelinerun-having-a-pipeline-run-yml-defined-in-git-e-g/69880097#69880097), although we should implement a kind of garbage collection for our PipelineRun objects...
Also mind the params: name: image
and insert an image name containing the correct namespace of your Container Registry you created a Secret for!
Also apply with
kubectl apply -f tekton-ci-config/pipeline-run.yml
Looking into the Tekton dashboard we should now finally see a successful Pipeline run:
It's extremey easy to leverage the Buildpacks cache image feature, that will create a separate cache image for building and will store it alongside the resulting app image in our Container Registry.
Therefore simply add the CACHE_IMAGE
parameter using our $(params.IMAGE)
definition and appending :paketo-build-cache
like this inside our Pipeline:
params:
- name: APP_IMAGE
value: "$(params.IMAGE)"
- name: CACHE_IMAGE
value: "$(params.IMAGE):paketo-build-cache"
- name: BUILDER_IMAGE
value: paketobuildpacks/builder:base # This is the builder we want the task to use (REQUIRED)
Now the next build should produce a separate image inside our Container Registry:
What about extending our Tekton Pipeline with a Maven Task, that initiates a test run before we build our app container using Buildpacks.
Therefore we can simply use the https://hub.tekton.dev/tekton/task/maven Task from Tekton Hub.
First we need to install the Task in our GitHub Actions provision.yml:
kubectl apply -f https://raw.githubusercontent.com/tektoncd/catalog/main/task/maven/0.2/maven.yaml
Now we need to enhance our pipeline.yml with a new workspace maven-settings
and the Task definition maven-test
, which should also be defined as runAfter
target in the buildpacks
Task:
workspaces:
- name: maven-settings # Maven settings, see https://hub.tekton.dev/tekton/task/maven
...
tasks:
...
- name: maven-test
taskRef:
name: maven
runAfter:
- fetch-repository
params:
- name: GOALS
value:
- verify
workspaces:
- name: maven-settings
workspace: maven-settings
- name: source
workspace: source-workspace
- name: buildpacks # This task uses the `buildpacks` task to build the application
taskRef:
name: buildpacks
runAfter:
- maven-test
We also should enhance our pipeline-run.yml and gitlab-push-listener.yml to define the additional workspace:
pipelineRef:
name: buildpacks-test-pipeline
workspaces:
- name: maven-settings
emptyDir: {}
Finally we should also add the new workspace with --workspace name=maven-settings,emptyDir=
to our tkn pipeline start
command in the project's branch https://gitlab.com/jonashackt/microservice-api-spring-boot/-/tree/trigger-tekton-via-gitlabci inside the .gitlab-ci.yml
:
tkn pipeline start buildpacks-test-pipeline \
--serviceaccount buildpacks-service-account-gitlab \
--workspace name=maven-settings,emptyDir= \
--workspace name=source-workspace,subPath=source,claimName=buildpacks-source-pvc \
--workspace name=cache-workspace,subPath=cache,claimName=buildpacks-source-pvc \
--param IMAGE=registry.gitlab.com/jonashackt/microservice-api-spring-boot \
--param SOURCE_URL=https://gitlab.com/jonashackt/microservice-api-spring-boot \
--param REPO_PATH_ONLY=jonashackt/microservice-api-spring-boot \
--param SOURCE_REVISION=main \
--param GITLAB_HOST=gitlab.com \
--param TEKTON_DASHBOARD_HOST="http://abd1c6f235c9642bf9d4cdf632962298-1232135946.eu-central-1.elb.amazonaws.com" \
--timeout 240s \
--showlog
It seems that the Tekton Hub's Maven Task doesn't implement caching for us. Our builds tend to download all Maven dependencies over and over again:
As stated in https://developers.redhat.com/blog/2020/02/26/speed-up-maven-builds-in-tekton-pipelines#maven_task_with_a_workspace we need to define our own Task for that.
As we don't need the settings.xml
configuration (e.g. for Proxy settings), which is the main point of the Tekton Hub's Maven Task, we can simply create our own - see task-maven-with-cache.yml:
apiVersion: tekton.dev/v1alpha1
kind: Task
metadata:
name: maven-with-cache
spec:
workspaces:
- name: source
description: The workspace consisting of maven project.
- name: maven-repo-cache
description: The workspace holding the Maven repository for caching.
params:
- name: GOALS
description: The Maven goals to run
type: array
default:
- "package"
steps:
- name: mvn
image: gcr.io/cloud-builders/mvn
workingDir: $(workspaces.source.path)
command: ["/usr/bin/mvn"]
args:
- -Dmaven.repo.local=$(workspaces.maven-repo-cache.path)
- "$(params.GOALS)"
We need to kubectl apply
our Task maven-with-cache
with:
kubectl apply -f tekton-ci-config/task-maven-with-cache.yml
We also need to use our new Maven Task inside our pipeline.yml:
workspaces:
- name: maven-repo-cache # Maven repository cahce, see https://developers.redhat.com/blog/2020/02/26/speed-up-maven-builds-in-tekton-pipelines#run_a_maven_pipeline
...
- name: maven-test
taskRef:
name: maven-with-cache
runAfter:
- fetch-repository
params:
- name: GOALS
value:
- verify
workspaces:
- name: source
workspace: source-workspace
- name: maven-repo-cache
workspace: maven-repo-cache
Now we could try to also create a new PersistentVolumeClaim
as stated in https://developers.redhat.com/blog/2020/02/26/speed-up-maven-builds-in-tekton-pipelines#run_a_maven_pipeline
- name: maven-repo-cache
persistentVolumeClaim:
claimName: maven-repo-cache-pvc
But we would run into errors like:
completionTime: '2021-12-06T09:13:49Z'
conditions: - lastTransitionTime: '2021-12-06T09:13:49Z'
message: more than one PersistentVolumeClaim is bound
reason: TaskRunValidationFailed
status: 'False'
type:
It seems that Tekton doesn't like to have multiple PVC inside one Task: tektoncd/pipeline#3480 and tektoncd/pipeline#3085
In general, try to only use a single PVC for each task.
So we need to use a separate subPath
inside our already existing PVC buildpacks-source-pvc
(which doesn't have a matching name any more it seems :) ).
As stated in https://buildpacks.io/docs/tools/tekton/#43-pipeline a Tekton workspace could be simply seen as a shared directory (see https://tekton.dev/docs/pipelines/workspaces/, where I didn't get this first).
So we finally simply provide a new workspace using the existing PVC but a different subPath
to our pipeline-run.yml & gitlab-push-listener.yml:
- name: maven-repo-cache
subPath: maven-repo-cache
persistentVolumeClaim:
claimName: buildpacks-source-pvc
And we can even optimize our solution by simply using the maven-settings
workspace definition of the standard Tekton Hub's Maven Task inside our Pipeline
workspaces:
- name: maven-repo-cache # Maven repository cahce, see https://developers.redhat.com/blog/2020/02/26/speed-up-maven-builds-in-tekton-pipelines#run_a_maven_pipeline
...
- name: maven-test
taskRef:
name: maven
runAfter:
- fetch-repository
params:
- name: GOALS
value:
- -Dmaven.repo.local=$(workspaces.maven-settings.path)
- verify
workspaces:
- name: source
workspace: source-workspace
- name: maven-settings
workspace: maven-repo-cache
Now our Pipeline should run faster then before, since the Maven cache is used:
How to trigger Tekton PipelineRun
from GitLab?
The simplest possible solution is to leverage GitLab CI and trigger Tekton from there.
See https://stackoverflow.com/a/69991508/4964553
Have a look at this example gitlab.com project https://gitlab.com/jonashackt/microservice-api-spring-boot/-/tree/trigger-tekton-via-gitlabci
Full getting-started guide: https://github.com/tektoncd/triggers/tree/v0.17.0/docs/getting-started
BUT FIRST Examples are a great inspiration - for GitLab this is especially:
https://github.com/tektoncd/triggers/tree/main/examples/v1beta1/gitlab
https://tekton.dev/docs/triggers/install/
kubectl apply --filename https://storage.googleapis.com/tekton-releases/triggers/latest/release.yaml
kubectl apply --filename https://storage.googleapis.com/tekton-releases/triggers/latest/interceptors.yaml
See https://github.com/tektoncd/triggers/blob/v0.17.0/examples/rbac.yaml
So we also create serviceaccount-rb-crb.yml:
apiVersion: v1
kind: ServiceAccount
metadata:
name: tekton-triggers-example-sa
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: triggers-example-eventlistener-binding
subjects:
- kind: ServiceAccount
name: tekton-triggers-example-sa
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: tekton-triggers-eventlistener-roles
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: triggers-example-eventlistener-clusterbinding
subjects:
- kind: ServiceAccount
name: tekton-triggers-example-sa
namespace: default
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: tekton-triggers-eventlistener-clusterroles
kubectl apply -f tekton-ci-config/triggers/serviceaccount-rb-crb.yml
As our Tekton Trigger API will be setup as a public API in the end, we need to secure our Trigger API somehow.
One way is to create a secret ID the calling JSON must contain. So let's create tekton-trigger-secret.yml:
apiVersion: v1
kind: Secret
metadata:
name: gitlab-secret
type: Opaque
stringData:
secretToken: "1234567"
kubectl apply -f tekton-ci-config/triggers/tekton-trigger-secret.yml
So let's start with the EventListener
. We'll adapt the EventListener
from the example (see https://github.com/tektoncd/triggers/blob/main/examples/v1beta1/gitlab/gitlab-push-listener.yaml) to use our Buildpacks Pipeline defined in pipeline.yml.
Therefore let's create a new file called gitlab-push-listener.yml:
apiVersion: triggers.tekton.dev/v1beta1
kind: EventListener
metadata:
name: gitlab-listener
spec:
serviceAccountName: tekton-triggers-example-sa
triggers:
- name: gitlab-push-events-trigger
interceptors:
- name: "verify-gitlab-payload"
ref:
name: "gitlab"
kind: ClusterInterceptor
params:
- name: secretRef
value:
secretName: "gitlab-secret"
secretKey: "secretToken"
- name: eventTypes
value:
- "Push Hook"
bindings:
- name: gitrevision
value: $(body.checkout_sha)
- name: gitrepositoryurl
value: $(body.repository.git_http_url)
template:
spec:
params:
- name: gitrevision
- name: gitrepositoryurl
- name: message
description: The message to print
default: This is the default message
- name: contenttype
description: The Content-Type of the event
resourcetemplates:
- apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
generateName: buildpacks-test-pipeline-run-
#name: buildpacks-test-pipeline-run
spec:
serviceAccountName: buildpacks-service-account-gitlab # Only needed if you set up authorization
pipelineRef:
name: buildpacks-test-pipeline
workspaces:
- name: source-workspace
subPath: source
persistentVolumeClaim:
claimName: buildpacks-source-pvc
- name: cache-workspace
subPath: cache
persistentVolumeClaim:
claimName: buildpacks-source-pvc
params:
- name: IMAGE
value: registry.gitlab.com/jonashackt/microservice-api-spring-boot # This defines the name of output image
- name: SOURCE_URL
value: https://gitlab.com/jonashackt/microservice-api-spring-boot
- name: SOURCE_REVISION
value: main
Now apply it to our cluster via:
kubectl apply -f tekton-ci-config/triggers/gitlab-push-listener.yml
Tekton Triggers creates a new Deployment and Service for the EventListener
As stated in https://tekton.dev/docs/triggers/eventlisteners/#understanding-the-deployment-of-an-eventlistener
Tekton Triggers uses the EventListener name prefixed with el- to name the Deployment and Service when instantiating them.
this will also deploy a K8s Service
called el-gitlab-listener
and a Deployment
also called el-gitlab-listener
:
$ k get deployment
NAME READY UP-TO-DATE AVAILABLE AGE
el-gitlab-listener 1/1 1 1 25h
$ k get service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
el-gitlab-listener ClusterIP 10.100.101.207 <none> 8080/TCP,9000/TCP 12m
...
What we need here is an example test event as .json
file. For an example see https://github.com/tektoncd/triggers/blob/main/examples/v1beta1/gitlab/gitlab-push-event.json
But we can craft this file ourselves while creating the GitLab Webhook.
Therefore go to your project (in my example here this is https://gitlab.com/jonashackt/microservice-api-spring-boot) and head over to Settings/Webhooks.
Now insert our a fake URL like http://www.google.com
into the URL field (we need a semantically correct url here in order to be able to save the Webhook, but will change it later to the EventListeners Ingress URL).
Also insert our Secret Token 1234567
in the Secret Token field.
Finally choose Push events, deselect SSL verification and scroll down and hit Add webhook.
This should result in the Webhook beeing created and listed right at the bottom of this page (scroll down again) in a list called Project Hooks
.
Click on Test and choose Push events. This will fire a test event (which will produce a HTTP error). Now scroll down again an click on edit. Scroll down inside our WebHook (don't get confused with the UI :)) and you should see the Recent events list:
Choose the latest event and click on View details
. Scroll down again and you should see the Request frame with the generated request json:
Copy the whole part into a local file. In this example project this file is called gitlab-push-test-event.json:
{
"object_kind": "push",
"event_name": "push",
"before": "5bbc8580432fc7a16f50be27eb513db42aad0860",
"after": "c25a74c8f919a72e3f00928917dc4ab2944ab061",
"ref": "refs/heads/main",
"checkout_sha": "c25a74c8f919a72e3f00928917dc4ab2944ab061",
"message": null,
"user_id": 2351133,
"user_name": "Jonas Hecht",
"user_username": "jonashackt",
"user_email": "",
"user_avatar": "https://secure.gravatar.com/avatar/a5c83d481ac20557b775703761aef7dc?s=80&d=identicon",
"project_id": 30444286,
"project": {
"id": 30444286,
"name": "microservice-api-spring-boot",
"description": "Forked from https://github.com/jonashackt/microservice-api-spring-boot",
"web_url": "https://gitlab.com/jonashackt/microservice-api-spring-boot",
"avatar_url": null,
"git_ssh_url": "[email protected]:jonashackt/microservice-api-spring-boot.git",
"git_http_url": "https://gitlab.com/jonashackt/microservice-api-spring-boot.git",
"namespace": "Jonas Hecht",
"visibility_level": 20,
"path_with_namespace": "jonashackt/microservice-api-spring-boot",
"default_branch": "main",
"ci_config_path": "",
"homepage": "https://gitlab.com/jonashackt/microservice-api-spring-boot",
"url": "[email protected]:jonashackt/microservice-api-spring-boot.git",
"ssh_url": "[email protected]:jonashackt/microservice-api-spring-boot.git",
"http_url": "https://gitlab.com/jonashackt/microservice-api-spring-boot.git"
},
"commits": [
{
"id": "c25a74c8f919a72e3f00928917dc4ab2944ab061",
"message": "Fixing cache image naming\n",
"title": "Fixing cache image naming",
"timestamp": "2021-10-19T10:32:58+02:00",
"url": "https://gitlab.com/jonashackt/microservice-api-spring-boot/-/commit/c25a74c8f919a72e3f00928917dc4ab2944ab061",
"author": {
"name": "Jonas Hecht",
"email": "[email protected]"
},
"added": [
],
"modified": [
".gitlab-ci.yml",
"README.md"
],
"removed": [
]
},
{
"id": "06a7f1d2ad646acef149b1aad4600eb2b1268f0c",
"message": "Merge branch 'refactor-ci' into 'main'\n\nRefactor ci\n\nSee merge request jonashackt/microservice-api-spring-boot!2",
"title": "Merge branch 'refactor-ci' into 'main'",
"timestamp": "2021-10-19T08:30:46+00:00",
"url": "https://gitlab.com/jonashackt/microservice-api-spring-boot/-/commit/06a7f1d2ad646acef149b1aad4600eb2b1268f0c",
"author": {
"name": "Jonas Hecht",
"email": "[email protected]"
},
"added": [
],
"modified": [
".gitlab-ci.yml",
"README.md"
],
"removed": [
]
},
{
"id": "5bbc8580432fc7a16f50be27eb513db42aad0860",
"message": "Add cache image\n",
"title": "Add cache image",
"timestamp": "2021-10-19T10:05:31+02:00",
"url": "https://gitlab.com/jonashackt/microservice-api-spring-boot/-/commit/5bbc8580432fc7a16f50be27eb513db42aad0860",
"author": {
"name": "Jonas Hecht",
"email": "[email protected]"
},
"added": [
],
"modified": [
".gitlab-ci.yml",
"README.md"
],
"removed": [
]
}
],
"total_commits_count": 3,
"push_options": {
},
"repository": {
"name": "microservice-api-spring-boot",
"url": "[email protected]:jonashackt/microservice-api-spring-boot.git",
"description": "Forked from https://github.com/jonashackt/microservice-api-spring-boot",
"homepage": "https://gitlab.com/jonashackt/microservice-api-spring-boot",
"git_http_url": "https://gitlab.com/jonashackt/microservice-api-spring-boot.git",
"git_ssh_url": "[email protected]:jonashackt/microservice-api-spring-boot.git",
"visibility_level": 20
}
}
Port forward with a new Service
locally:
kubectl port-forward service/el-gitlab-listener 8080
Now test-drive the trigger via curl:
curl -v \
-H 'X-GitLab-Token: 1234567' \
-H 'X-Gitlab-Event: Push Hook' \
-H 'Content-Type: application/json' \
--data-binary "@tekton-ci-config/triggers/gitlab-push-test-event.json" \
http://localhost:8080
Ingress on EKS
https://www.eksworkshop.com/beginner/130_exposing-service/ingress_controller_alb/
In order for the Ingress resource to work, the cluster must have an ingress controller running. Unlike other types of controllers which run as part of the kube-controller-manager binary, Ingress controllers are not started automatically with a cluster.
The section Expose Tekton Dashboard publicly on EKS describes how to generally expose a web app publicly like the Tekton Dashboard on EKS using a simple Service
of type LoadBalancer
.
But a Tekton Triggers EventListener also creates a Deployment AND Service when installed (https://tekton.dev/docs/triggers/eventlisteners/#exposing-an-eventlistener-outside-of-the-cluster):
By default, ClusterIP services such as EventListeners are only accessible within the cluster on which they are running.
Sometimes the Tekton docs aren't formatted well, but the source GitHub READMEs are!
Here's a simple guide on how to export our Tekton EventListener as Ingress using the Nginx Ingress controller:
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.34.1/deploy/static/provider/cloud/deploy.yaml
The docs have more details for us (https://kubernetes.github.io/ingress-nginx/deploy/#aws):
In AWS we use a Network load balancer (NLB) to expose the NGINX Ingress controller behind a Service of Type=LoadBalancer.
kubectl get eventlistener gitlab-listener -o=jsonpath='{.status.configuration.generatedName}{"\n"}'
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: tekton-eventlistener-ingress
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
rules:
- http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: el-gitlab-listener
port:
number: 8080
kubectl apply -f tekton-ci-config/triggers/tekton-eventlistener-ingress.yml
kubectl get ingress tekton-eventlistener-ingress --output=jsonpath='{.status.loadBalancer.ingress[0].hostname}'
Now let's try our curl
using the predefined gitlab-push-test-event.json:
TEKTON_EVENTLISTENER_INGRESS_HOST="http://$(kubectl get ingress tekton-eventlistener-ingress --output=jsonpath='{.status.loadBalancer.ingress[0].hostname}')"
curl -v \
-H 'X-GitLab-Token: 1234567' \
-H 'X-Gitlab-Event: Push Hook' \
-H 'Content-Type: application/json' \
--data-binary "@tekton-ci-config/triggers/gitlab-push-test-event.json" \
$TEKTON_EVENTLISTENER_INGRESS_HOST
Finally we can implement all this inside our GitHub Action workflow .github/workflows/provision.yml:
- name: Expose Tekton Triggers EventListener via Ingress & testdrive Trigger
run: |
echo "--- Deploy Nginx Ingress Controller"
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.34.1/deploy/static/provider/cloud/deploy.yaml
echo "--- Apply Tekton EventListener Ingress"
kubectl apply -f tekton-ci-config/triggers/tekton-eventlistener-ingress.yml
echo "--- Get Ingress host name"
TEKTON_EVENTLISTENER_INGRESS_HOST="http://$(kubectl get ingress tekton-eventlistener-ingress --output=jsonpath='{.status.loadBalancer.ingress[0].hostname}')"
echo "Our EventListener's hostname is $TEKTON_EVENTLISTENER_INGRESS_HOST"
echo "--- Testdrive Trigger via curl"
curl -v \
-H 'X-GitLab-Token: 1234567' \
-H 'X-Gitlab-Event: Push Hook' \
-H 'Content-Type: application/json' \
--data-binary "@tekton-ci-config/triggers/gitlab-push-test-event.json" \
$TEKTON_EVENTLISTENER_INGRESS_HOST
We now should extend our gitlab-push-listener.yml to use the values send by the GitLab Webhook via json.
Our file now looks like this:
apiVersion: triggers.tekton.dev/v1beta1
kind: EventListener
metadata:
name: gitlab-listener
spec:
serviceAccountName: tekton-triggers-example-sa
triggers:
- name: gitlab-push-events-trigger
interceptors:
- name: "verify-gitlab-payload"
ref:
name: "gitlab"
kind: ClusterInterceptor
params:
- name: secretRef
value:
secretName: "gitlab-secret"
secretKey: "secretToken"
- name: eventTypes
value:
- "Push Hook"
bindings:
- name: gitrevision
value: $(body.checkout_sha)
- name: gitrepositoryurl
value: $(body.repository.git_http_url)
- name: gitrepository_pathonly
value: $(body.project.path_with_namespace)
template:
spec:
params:
- name: gitrevision
- name: gitrepositoryurl
- name: gitrepository_pathonly
- name: message
description: The message to print
default: This is the default message
- name: contenttype
description: The Content-Type of the event
resourcetemplates:
- apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
generateName: buildpacks-test-pipeline-run-
spec:
serviceAccountName: buildpacks-service-account-gitlab
pipelineRef:
name: buildpacks-test-pipeline
workspaces:
- name: source-workspace
subPath: source
persistentVolumeClaim:
claimName: buildpacks-source-pvc
- name: cache-workspace
subPath: cache
persistentVolumeClaim:
claimName: buildpacks-source-pvc
params:
- name: IMAGE
value: "registry.gitlab.com/$(tt.params.gitrepository_pathonly)" #here our GitLab's registry url must be configured
- name: SOURCE_URL
value: $(tt.params.gitrepositoryurl)
- name: SOURCE_REVISION
value: $(tt.params.gitrevision)
There's not so much that changes here. The param SOURCE_URL
will be provided the git repo url via $(tt.params.gitrepositoryurl)
- this is done using the tt.params
notation as the docs state.
The same is used for SOURCE_REVISION
. Both parameters must be defined in the template:spec:params
section also.
And they must all be defined in the TriggerBinding inside the bindings
section. Here we read the values from the parsed json request from GitLab using the $(body.key1)
notation - see the docs on TriggerBindings json payload access.
Finally we also read another value in the bindings
section: the gitrepository_pathonly
value will be obtained from $(body.project.path_with_namespace)
- which represents our GitLab repo's group and repo names. In this example this is jonashackt/microservice-api-spring-boot
.
But what do we need this value for? You'll see it inside the IMAGE
parameter of the PipelineRun
definition inside the TriggerTemplate.
With value: "registry.gitlab.com/$(tt.params.gitrepository_pathonly)"
we use the gitrepository_pathonly
to craft the correct GitLab Container Registry URL with the predefined GitLab CR domain name and the appended group and repo name.
Go to your project (in my example here this is https://gitlab.com/jonashackt/microservice-api-spring-boot) and head over to Settings/Webhooks.
Now insert our Tekton Triggers EventListener URL into the already created Webhook's URL field (). Remember you can obtain the URL via
kubectl get ingress tekton-eventlistener-ingress --output=jsonpath='{.status.loadBalancer.ingress[0].hostname}'
You need to scroll down to Project Hooks and edit your existing Webhook.
Finally Test-drive the Webhook again choosing Test and Push event from the drop down.
Switch to the Tekton Dashboard in your Browser and you should see the PipelineRun triggered:
The last step in our journey of integrating GitLab with Tekton is to report the status of our Tekton Pipelines back to GitLab.
There are multiple options. Let's first start simple using the Tekton Hub task https://hub.tekton.dev/tekton/task/gitlab-set-status
Inside our provision.yml workflow we need to install the gitlab-set-status Task:
kubectl apply -f https://raw.githubusercontent.com/tektoncd/catalog/main/task/gitlab-set-status/0.1/gitlab-set-status.yaml
To access the GitLab commit API (see https://docs.gitlab.com/ee/api/commits.html#post-the-build-status-to-a-commit) using the gitlab-set-status
task we need to create an access token as stated in https://docs.gitlab.com/ee/api/index.html#authentication
On self-managed GitLab instances you can create project access tokens for example.
Using gitlab.com we cannot use project access tokens, but can create personal access tokens instead: https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html#create-a-personal-access-token
Therefore head over to Edit profile and choose Access Tokens on the left. Then create a token called gitlab-api-token
for example:
Now since we're using GitHub Actions to provision our EKS cluster and install Tekton, we need to enhance our provision.yml workflow.
First create a new GitHub repository secret GITLAB_API_TOKEN
containing the personal access token which we just created:
Now we can use these GitHub repo secrets to create the actual Kubernetes secret in our provision.yml workflow:
kubectl create secret generic gitlab-api-secret \
--from-literal=token=${{ secrets.GITLAB_API_TOKEN }} \
--namespace default \
--save-config --dry-run=client -o yaml | kubectl apply -f -
Using the Tekton Hub task https://hub.tekton.dev/tekton/task/gitlab-set-status we can create a new step inside our Tekton Pipeline. But first ne need to create some new parameters for our Pipeline:
params:
...
- name: REPO_PATH_ONLY
type: string
description: GitLab group & repo name only (e.g. jonashackt/microservice-api-spring-boot)
...
- name: GITLAB_HOST
type: string
description: Your GitLabs host only (e.g. gitlab.com)
- name: TEKTON_DASHBOARD_HOST
type: string
description: The Tekton dashboard host name only
Now we can implement the task:
- name: report-pipeline-end-to-gitlab
taskRef:
name: "gitlab-set-status"
runAfter:
- buildpacks
params:
- name: "STATE"
value: "success"
- name: "GITLAB_HOST_URL"
value: "$(params.GITLAB_HOST)"
- name: "REPO_FULL_NAME"
value: "$(params.REPO_PATH_ONLY)"
- name: "GITLAB_TOKEN_SECRET_NAME"
value: "gitlab-api-secret"
- name: "GITLAB_TOKEN_SECRET_KEY"
value: "token"
- name: "SHA"
value: "$(params.SOURCE_REVISION)"
- name: "TARGET_URL"
value: "$(params.TEKTON_DASHBOARD_HOST)/#/namespaces/default/pipelineruns/$(context.pipelineRun.name)"
- name: "CONTEXT"
value: "tekton-pipeline"
- name: "DESCRIPTION"
value: "Finished building your commit in Tekton"
First we should make sure this task runs only after the buildpacks
task using runAfter
.
Then we need to provide the GITLAB_HOST_URL
and REPO_FULL_NAME
.
Also the GITLAB_TOKEN_SECRET_NAME
needs to refer to the Kubernetes secret gitlab-api-secret
we created leveraging a Personal Access Token. The GITLAB_TOKEN_SECRET_KEY
must reference to the key
name inside the kubectl create secret
command, where we used --from-literal=token=${{ secrets.GITLAB_API_TOKEN }}
. So the GITLAB_TOKEN_SECRET_KEY
is token
here (which is also the default).
The TARGET_URL
is a bit more tricky and needs to be crafted with care:
- name: "TARGET_URL"
value: "$(params.TEKTON_DASHBOARD_HOST)/#/namespaces/default/pipelineruns/$(context.pipelineRun.name)"
It consists of the Tekton dashboard host $(params.TEKTON_DASHBOARD_HOST)
and the Pipeline details prefix /#/namespaces/default/pipelineruns/
. The $(context.pipelineRun.name)
finally gives us the current PipelineRun's name we need to be able to reference the actual PipelineRun from the GitLab UI.
Finally the CONTEXT
and DESCRIPTION
should contain useful information to be displayed in the GitLab UI later:
Our pipeline-run.yml (for manual triggering):
...
params:
- name: IMAGE
value: registry.gitlab.com/jonashackt/microservice-api-spring-boot # This defines the name of output image
- name: SOURCE_URL
value: https://gitlab.com/jonashackt/microservice-api-spring-boot
- name: REPO_PATH_ONLY
value: jonashackt/microservice-api-spring-boot
- name: SOURCE_REVISION
value: main
- name: GITLAB_HOST
value: gitlab.com
- name: TEKTON_DASHBOARD_HOST
value: http://abd1c6f235c9642bf9d4cdf632962298-1232135946.eu-central-1.elb.amazonaws.com
and the EventListener (for automatic triggering by our gitlab.com projects) need to pass some new parameters in order to get the gitlab-set-status
task working:
params:
- name: IMAGE
value: "registry.gitlab.com/$(tt.params.gitrepository_pathonly)" #here our GitLab's registry url must be configured
- name: SOURCE_URL
value: $(tt.params.gitrepositoryurl)
- name: REPO_PATH_ONLY
value: $(tt.params.gitrepository_pathonly)
- name: SOURCE_REVISION
value: $(tt.params.gitrevision)
- name: GITLAB_HOST
value: gitlab.com
- name: TEKTON_DASHBOARD_HOST
value: {{TEKTON_DASHBOARD_HOST}}
Adding the REPO_PATH_ONLY
is no problem, since we alread used $(tt.params.gitrepository_pathonly)
inside the IMAGE
parameter. As with the GitLab registry url we can also "hard code" the GITLAB_HOST
here for now.
The TEKTON_DASHBOARD_HOST
is the trickiest part, since we need to substitute this value from outside of the Tekton Trigger process, which doesn't know about the Tekton dashboard url.
But luckily inside our GitHub Actions provision.yml workflow we can use https://stackoverflow.com/questions/48296082/how-to-set-dynamic-values-with-kubernetes-yaml-file/70152914#70152914:
echo "--- Insert Tekton dashboard url into EventListener config and apply it (see https://stackoverflow.com/a/70152914/4964553)"
TEKTON_DASHBOARD_HOST="${{ steps.dashboard-expose.outputs.dashboard_host }}"
sed "s#{{TEKTON_DASHBOARD_HOST}}#$TEKTON_DASHBOARD_HOST#g" tekton-ci-config/triggers/gitlab-push-listener.yml | kubectl apply -f -
Using sed we simply replace {{TEKTON_DASHBOARD_HOST}}
with the already defined GitHub Actions variable ${{ steps.dashboard-expose.outputs.dashboard_host }}
.
Testing our full workflow is simple pushing a change to our repo using a branch without GitLab CI: https://gitlab.com/jonashackt/microservice-api-spring-boot/-/commits/trigger-tekton-via-webhook
Now our GitLab Pipelines view gets filled with a Tekton Pipelines status (success
only for now):
Now that we generally know how to use the gitlab-set-status
Task, we could also use another Task definition to report the starting of a Tekton Pipeline run to GitLab UI.
Therefore we enhance our Tekton Pipeline with a new Task starting the whole Pipeline called report-pipeline-start-to-gitlab
:
tasks:
- name: report-pipeline-start-to-gitlab
taskRef:
name: gitlab-set-status
params:
- name: "STATE"
value: "running"
- name: "GITLAB_HOST_URL"
value: "$(params.GITLAB_HOST)"
- name: "REPO_FULL_NAME"
value: "$(params.REPO_PATH_ONLY)"
- name: "GITLAB_TOKEN_SECRET_NAME"
value: "gitlab-api-secret"
- name: "GITLAB_TOKEN_SECRET_KEY"
value: "token"
- name: "SHA"
value: "$(params.SOURCE_REVISION)"
- name: "TARGET_URL"
value: "$(params.TEKTON_DASHBOARD_HOST)/#/namespaces/default/pipelineruns/$(context.pipelineRun.name)"
- name: "CONTEXT"
value: "tekton-pipeline"
- name: "DESCRIPTION"
value: "Building your commit in Tekton"
- name: fetch-repository # This task fetches a repository from github, using the `git-clone` task you installed
taskRef:
name: git-clone
runAfter:
- report-pipeline-start-to-gitlab
And the fetch-repository
only starts after the status is reported to GitLab. With this new Task in place a push to our repository https://gitlab.com/jonashackt/microservice-api-spring-boot directly presents a running
Pipeline inside the GitLab UI:
And in the details view we can directly access our running Tekton Pipeline via a correct Tekton dashboard link:
How do we catch all status from our Tekton pipeline and then report based on that to GitLab?
In v0.14 Tekton introduced finally Tasks, which run at the end of every PipelineRun - regardless which Task failed or succeeded. As the docs state:
finally tasks are guaranteed to be executed in parallel after all PipelineTasks under tasks have completed regardless of success or error.
Finally tasks look like this:
spec:
tasks:
- name: tests
taskRef:
name: integration-test
finally:
- name: cleanup-test
taskRef:
name: cleanup
With Guard[ing] finally Task execution using when expressions we can enhance this even further.
Because using when
expressions we can run Tasks based on the overall Pipeline status (or Aggregate Pipeline status) - see https://tekton.dev/docs/pipelines/pipelines/#when-expressions-using-aggregate-execution-status-of-tasks-in-finally-tasks
finally:
- name: notify-any-failure # executed only when one or more tasks fail
when:
- input: $(tasks.status)
operator: in
values: ["Failed"]
taskRef:
name: notify-failure
The Aggregate Execution Status we can grap using $(tasks.status)
is stated to have those 4 possible status:
Succeeded
("all tasks have succeeded") or Completed
("all tasks completed successfully including one or more skipped tasks"), which could be translated into the gitlab-set-status
Tasks STATE
value success
.
And Failed
("one ore more tasks failed") or None
("no aggregate execution status available (i.e. none of the above), one or more tasks could be pending/running/cancelled/timedout"), which could both be translated into the gitlab-set-status
Tasks STATE
value failed
. For None
this is only valid, since we're in a finally task
, since pending/running
could otherwise also mean that a Pipeline is in a good state.
Luckily the when
expressions
So we're able to do
when:
- input: $(tasks.status)
operator: in
values: [ "Failed", "None" ]
and
when:
- input: $(tasks.status)
operator: in
values: [ "Succeeded", "Completed" ]
In the end this results in our Tekton Pipeline's finally
block locking like this:
...
finally:
- name: report-pipeline-failed-to-gitlab
when:
- input: $(tasks.status)
operator: in
values: [ "Failed", "None" ] # see aggregated status https://tekton.dev/docs/pipelines/pipelines/#using-aggregate-execution-status-of-all-tasks
taskRef:
name: "gitlab-set-status"
params:
- name: "STATE"
value: "failed"
- name: "GITLAB_HOST_URL"
value: "$(params.GITLAB_HOST)"
- name: "REPO_FULL_NAME"
value: "$(params.REPO_PATH_ONLY)"
- name: "GITLAB_TOKEN_SECRET_NAME"
value: "gitlab-api-secret"
- name: "GITLAB_TOKEN_SECRET_KEY"
value: "token"
- name: "SHA"
value: "$(params.SOURCE_REVISION)"
- name: "TARGET_URL"
value: "$(params.TEKTON_DASHBOARD_HOST)/#/namespaces/default/pipelineruns/$(context.pipelineRun.name)"
- name: "CONTEXT"
value: "tekton-pipeline"
- name: "DESCRIPTION"
value: "An error occurred building your commit in Tekton"
- name: report-pipeline-success-to-gitlab
when:
- input: $(tasks.status)
operator: in
values: [ "Succeeded", "Completed" ] # see aggregated status https://tekton.dev/docs/pipelines/pipelines/#using-aggregate-execution-status-of-all-tasks
taskRef:
name: "gitlab-set-status"
params:
- name: "STATE"
value: "success"
- name: "GITLAB_HOST_URL"
value: "$(params.GITLAB_HOST)"
- name: "REPO_FULL_NAME"
value: "$(params.REPO_PATH_ONLY)"
- name: "GITLAB_TOKEN_SECRET_NAME"
value: "gitlab-api-secret"
- name: "GITLAB_TOKEN_SECRET_KEY"
value: "token"
- name: "SHA"
value: "$(params.SOURCE_REVISION)"
- name: "TARGET_URL"
value: "$(params.TEKTON_DASHBOARD_HOST)/#/namespaces/default/pipelineruns/$(context.pipelineRun.name)"
- name: "CONTEXT"
value: "tekton-pipeline"
- name: "DESCRIPTION"
value: "Finished building your commit in Tekton"
Executing our Tekton Pipeline should now be reported correctly to our GitLab. Failures look like this:
and in the detail view:
The solution is based on https://stackoverflow.com/questions/70156006/report-tekton-pipeline-status-to-gitlab-regardless-if-pipeline-failed-or-succeed/70156007#70156007.
https://hub.tekton.dev/tekton/task/create-gitlab-release
https://gitlab.com/gitlab-org/gitlab/-/issues/213360
https://gitlab.com/gitlab-org/gitlab-runner/-/issues/28051
https://www.youtube.com/watch?v=skcLi9-WTkA
https://cloud.google.com/tekton
Why not use a Chat bot to do the job?
-> If I push, a bot asks me to assign another person for the pull request
- see Kubernetes github project: prow comments on every pull request
https://archive.fosdem.org/2021/schedule/event/ci_on_gitlab_ringing_gitlab_tekton_and_prow_together/
prow is GitHub specific, so what's the alternative?
rather than being GitHub specific Lighthouse uses jenkins-x/go-scm so it can support any Git provider (while Lighthouse reuses the Prow plugin source code and a bunch of plugins from Prow)
https://jenkins-x.io/v3/about/what/#chatops
With the ever growing number of microservices needing automation, Jenkins X provides the ability to interact with pipelines via comments on pull requests. Lighthouse has evolved from Prow which is used heavily in the Kubernetes ecosystem to provide a consistent developer workflow for triggering tests, approvals, hold and other common commands developers use in their everyday activities
https://github.com/jenkins-x/lighthouse
Lighthouse is a lightweight ChatOps based webhook handler which can trigger Jenkins X Pipelines, Tekton Pipelines or Jenkins Jobs based on webhooks from multiple git providers such as GitHub, GitHub Enterprise, BitBucket Server and GitLab.
https://github.com/jenkins-x/lighthouse/blob/main/docs/install_lighthouse_with_tekton.md
lighthouse-foghorn: watches execution of Tekton PipelineRun
triggered by lighthouse and updates AND BLOCKS pull requests from beeing merged until Tekton pipelines succeeded
And more:
--> Lighthouse Tekton integration misses JSON-payload proceeding to Tekton Pipelines/Tasks!
GitLab Webhooks --> trigger Tekton Triggers
https://github.com/tektoncd/triggers
https://github.com/tektoncd/experimental/tree/main/commit-status-tracker
The AWS ALB Ingress Controller has been rebranded to AWS Load Balancer Controller.
https://aws.amazon.com/blogs/opensource/kubernetes-ingress-aws-alb-ingress-controller/
Pulumi AWS Load Balancer Controller support: pulumi/pulumi-eks#29
https://pulumi.awsworkshop.io/50_eks_platform/30_deploy_ingress_controller.html