This is a concourse task which uses kind to start a kubernetes cluster locally for users to run their tests against.
Users can choose to run whichever kubernetes version kind uses by default, or use a specific kubernetes version by providing the kubernetes source tree checked out at a specific version.
It is heavily based on concourse-dcind and of course uses kind. This is really just plumbing ...
jobs:
- name: kind
plan:
- in_parallel:
- get: kind-on-c
- get: kind-release
params:
globs:
- kind-linux-amd64
- task: run-kind
privileged: true
file: kind-on-c/kind.yaml
params:
KIND_TESTS: |
# your actual tests go here!
kubectl get nodes -o wide
resources:
- name: kind-release
type: github-release
source:
owner: kubernetes-sigs
repository: kind
access_token: <some github token>
pre_release: true
- name: kind-on-c
type: git
source:
uri: https://github.com/pivotal-k8s/kind-on-cThis uses the kind binary, provided by the kind-release resource, to create
a kubernetes cluster. It will use the node image from the kind team.
The version of kind in use and thus, indirectly, the kubernetes version
deployed can be controlled by pinning the kind-release resource to a specific
version.
When the cluster is up and running, any commands configured in $KIND_TESTS
will run. The environment is setup with $KUBECONFIG which points to the
client configuration for the deployed kubernetes cluster. Also the most recent
(stable) version of kubectl will be downloaded and put into the $PATH,
therefore kubectl <something> will just work™️.
jobs:
- name: kind
plan:
- in_parallel:
- get: k8s-git
- get: kind-on-c
- get: kind-release
params:
globs:
- kind-linux-amd64
- task: run-kind
privileged: true
file: kind-on-c/kind.yaml
params:
KIND_TESTS: |
# your actual tests go here!
kubectl get nodes -o wide
resources:
- name: k8s-git
type: git
source:
uri: https://github.com/kubernetes/kubernetes
- name: kind-release
type: github-release
source:
owner: kubernetes-sigs
repository: kind
access_token: <some github token>
pre_release: true
- name: kind-on-c
type: git
source:
uri: https://github.com/pivotal-k8s/kind-on-cIf the task finds an task input named k8s-git it treats that as a
kubernetes source tree and tells kind to create a node image off of that. You
can just use a git resource, and pin it to a specific commit if need be, if you
want to run a specific kubernetes version.
In this case, also kubectl is compiled on the fly and therefore exactly
matches the version of kubernetes deployed.
KIND_TESTS
... this is the important piece: configure, what YOU actually want to run against the cluster! E.g.:Note: The tests will run in bash with the optionsKIND_TESTS: | kubectl get nodes -o jsonpath="{..name}" | grep -q 'worker2' || { echo >&2 "Expected node 'worker2' not found" return 1 }
errexit&nounsetKIND_CONFIG
... is the config kind when creating the cluster. Optional. If not specified, the default config is used, but can be overwritten with something like:Note: potentially the config you specify will be patched with things that are essential for kind-on-c to work properly.KIND_CONFIG: | kind: Cluster apiVersion: kind.x-k8s.io/v1alpha4 nodes: - role: control-plane - role: worker - role: worker
-
EXPORT_NODE_ROOTFS
If this parameter is set, the node image built by kind is made available as an output in the format of a rootfs. This output can be consumed by other steps and used to run a task off of.
See also the ouputexported-node-rootfs.
Note: If the task is configured to use upstream kind's default node image, this feature is not available. A warning is printed and the output is kept empty. -
EXPORT_NODE_IMAGE
If this parameter is set, the node image built by kind is made available as an output in the format of an OCI image. This output can be consumed by other steps and e.g. pushed to a registry. This might come in handy if you want to run multiple tests against the same version of kubernetes in parallel. You could have one step which builds the node image, exports it as and OCI image, and pushes it to a registry and then some more parallel jobs that consume that image. With that approach you don't need to compile the smae node image over and over.
See also the ouputexported-node-image.
Note: If the task is configured to use upstream kind's default node image, this feature is not available. A warning is printed and the output is kept empty. -
INSTALL_METALLB
If this parameter is set, metallb is installed onto the cluster. This allows users to deploy services of typeLoadBalancerand get anEXTERNAL-IPfor those. This external IP can be used to connect to a exposed service from the task container, thus from code running inKIND_TESTS. KIND_PRE_START
... if you want or need to run something just before the kind cluster is startedDOCKERD_OPTS
... if you need to add some configs when starting the docker daemonDOCKERD_TIMEOUT
... how long do you want to wait for docker to come up?KINDONC_DEBUG
... if you want to see all the ugly things that are happening to bring up docker and to run kindKIND_LOGLEVEL
... make kind more or less verbose when it is doing its business.
For kind<= 0.5.1this value is used for the--logleveloption and needs to be one ofpanic,fatal,error,warning,info,debug.
For kind> 0.5.1this value is used for the--verbosityoption and needs to be and integer.KIND_CLUSTER_NAME
... in case you want to change kind's cluster name -- you actually should not need to do that ...
kind-on-c, mandatory
The file tree of this repo. At least the task file and the entrypoint will (or should) be used from this input, in future potentially more. This input is typically provided by a git resource that is pulling this very git repo.kind-release, mandatory Must prrovide the kind binary, namedkind-linux-amd64. This should either be backed by a github-release resource or a earlier task that compiles kind from the source.k8s-git, optional
Must provide a git source tree of kubernetes. When configured, the node image will be built off of the checked-out revision of kubernetes. This is typically a git resource, pointing to (a fork of) k/k.node-image, optional Must provide an OCI imageimage.tarthat will be used as a node image. This can be an image generated viaEXPORT_NODE_IMAGE/exported-node-imageor any other OCI image that can be used as node image for kind.inputs, optional
If this task depends on other resources, aggregate them into theinputsinput. More about that in Aggregated inputs & outputs.
The task generates all the outputs in this list, however depending on the task's configuration it may leave certain (or even all) outputs empty.
-
exported-node-image
If the task is configured to output the node image as an OCI image, this output will be populated. A following step can consume this output and e.g. theregistry-imageresource could be used to push the node image into a registry.jobs: - name: kind plan: - task: run-kind privileged: true file: kind-on-c/kind.yaml params: EXPORT_NODE_IMAGE: 1 - put: image-repo # this is a registry-image resource params: image: exported-node-image/image.tar
-
exported-node-rootfs
If the task is configured to output the node image as a rootfs, this output will be populated with a rootfs and a minimalmetadata.json. With that it is possible to use the node image as a task image for a following concourse task.jobs: - name: kind plan: - task: run-kind privileged: true file: kind-on-c/kind.yaml params: EXPORT_NODE_ROOTFS: 1 - task: do something with the node rootfs image: exported-node-rootfs config: platform: linux run: path: bash args: - -xeuc - | echo "Concourse is now running the image we built with kind"
outputs
If users want to create further outputs, they can do so by aggregating them in theouputsoutput. More about that in Aggregated inputs & outputs.
Chances are good, that you need some additional tools to run your
KIND_TESTS. To do that, you can use a custom task image.
You can e.g. use the registry-image resource to pull in your custom image and
then override kind-on-c's default image by providing its rootfs to the
kind-on-c task via the image directive. This overrides the image that is
configured in the task config.
When it comes to creating the image, you can base it on the default kind-on-c
image and add your additional tools and things as another layer. For that you
can follow the image's ci-latest tag, which gets updated every time we push a
new task image. Or you can extract all information (repo and digest) of the
default task image from the task config file.
When you want to bring a entirely different image which is not based on kind-on-c's default task image, you need to check which dependencies we need in the Dockerfile.
For e.g. static binaries you could, instead of providing a custom image, have
concourse pull down all the needed things and provide them to the task via the
inputs input. Also, of course, nothing stand in the way to use both a custom
image and provide certain things via inputs.
jobs:
- name: kind
plan:
- in_parallel:
- get: custom-kind-on-c-image
- get: kind-on-c
- get: kind-release
- get: other-custom-things
- task: run-kind
privileged: true
file: kind-on-c/kind.yaml
image: custom-kind-on-c-image # This overrides the image from the task config file
input_mapping:
inputs: other-custom-things
params:
KIND_TESTS: # use custom things either from the image or from inputs
resources:
- name: custom-kind-on-c-image
type: registry-image
source: {repository: my-custom-image, tag: "1.13"}
- name: other-custom-things
type: some-resource-type
source: {...}Currently this task only allows for a fixed set of inputs and outputs (some inputs are optional, some output might be kept empty). This set cannot be changed by users of kind-on-c.
Still users need to be able to provide one or more of their own resources to be consumed by this task as inputs or generate one or more outputs by this task.
A workaround for this are (what I call now) aggregated inputs & outputs:
Users can use as many input resources they want, they however need to aggregate
them into the one input inputs for this task. Likewise, if that is something
a user needs to do, they can place all their outputs into the single outputs
output and have a later task split them apart again, into separate, individual
outputs.
After all, inputs and outputs are eventually "just directories" on disk and therefore relatively easy to manipulate if need be.
The following (contrived) example shows how this could be used:
- aggregates the resources
input-res-1,input-res-2, andinput-res-3into the inputinputs - the kind-on-c task uses the input
inputs - the kind-on-c task uses the output
outputsand placesartifacts,metrics, andlogsin there - the last task consumes the
outputsoutput of the kind-on-c task, splits those up and makes them available as separate outputsartifacts,metrics, andlogs - all of the
putresources just care about one of the individual outputs
plan:
- in_parallel:
- get: kind-on-c
- get: input-res-1
trigger: true
- get: input-res-2
trigger: true
- get: input-res-3
- task: aggregate inputs for kind-on-c
config:
platform: linux
image_resource:
type: registry-image
source: { repository: bash }
inputs:
- name: input-res-1
- name: input-res-2
- name: input-res-3
outputs:
- name: inputs # this will be the aggregated inputs for kind-on-c
run:
path: bash
args:
- -xeuc
- |
cp -a input-res-1 inputs/
cp -a input-res-2 inputs/
cp -a input-res-3 inputs/
- task: kind-on-c
privileged: true
file: kind-on-c/kind.yaml
params:
KIND_TESTS: |
# your actual tests go here!
cp -a some/artifacts outputs
cp -a some/metrics outputs
cp -a some/logs outputs
- task: split aggregated outputs of kind-on-c
config:
platform: linux
image_resource:
type: registry-image
source: { repository: bash }
inputs:
- name: outputs
outputs:
- name: artifacts
- name: metrics
- name: logs
run:
path: bash
args:
- -xeuc
- |
cp -a outputs/artifacts/* artifacts
cp -a outputs/metrics/* metrics
cp -a outputs/logs/* logs
- in_parallel:
- put: artifactory # using only the artifacts output of the last task
params:
folder: artifacts
- put: lftp-log-dump # using only the log output of the previous task
params:
files: "logs/*.log"
- put: swift-metrics-store # using only the metrics output of the previous task
params:
from: "metrics/(.*)"kind-on-c made the design decision to use a task file. This is mostly to have an easy way to specify the specific version of the kind-on-c image, have the pipeline automatically update it, and track which commit has been tested with which version of the image.
concourse however does not allow to mix settings (e.g. inputs) from a task file with inline configuration in the task config.
Therefore, for now at least, kind-on-c is opting for specifying the set of allowed/available inputs/outputs, which might mean a bit of overhead for the users.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent