Is this a request for help?:
No.
Is this a BUG REPORT or FEATURE REQUEST? (choose one):
FEATURE REQUEST
Provide a nicer way to deploy multiple agents with different configuration into the same namespace where the app
label and selector can be set to unique names but without producing awkwardly named pods while doing so.
I'm raising this feature request as I plan to raise a pull request and want something to reference as to why I'm creating it.
Version of Helm and Kubernetes:
Not Applicable. But Helm v3.11.2 and Kubernetes 1.26.3.
Which chart:
buildkite/agent
What happened:
I wish to be able to deploy multiple agents with a slightly different configuration into the same namespace in our cluster.
For example I use helmfile and I define deployments like so:
releases:
- name: buildkite-agent
chart: buildkite/agent
namespace: buildkite
values:
- helm/vars/values.yaml.gotmpl
- name: buildkite-large-agent
chart: buildkite/agent
namespace: buildkite
values:
- helm/vars/values.yaml.gotmpl
- helm/vars/large.yaml.gotmpl
Where helm/vars/values.yaml.gotmpl
contains the values for both of the agents and helm/vars/large.yaml.gotmpl
overrides some values (like the queue
name in agent.tags
) for the buildkite-large-agent
deployment only.
If I do that with the version of the chart as it is now, then both Deployment
resources end up with a label
of app: agent
and the selector
will match against that.
The two deployments will have no way of distinguishing which pods belong to which deployment.
In the case of us using KEDA to scale these agents it can't because it produces HPAs that end up with errors like:
Warning AmbiguousSelector 4m39s (x26 over 23m) horizontal-pod-autoscaler pods by selector app=agent are controlled by multiple HPAs: [buildkite/keda-hpa-buildkite-large-agent buildkite/keda-hpa-buildkite-agent]
I can set nameOverride
to make this app
label unique, however it combined with the .Release.Name
variable leads to some awkwardly named agents.
E.g. if I set nameOverride: large
for the 2nd agent I end up with a Deployment name like buildkite-large-agent-large
for example.
What you expected to happen:
I'd like a way to have more control over the app
label and selector that gets generated while still having control over the names of the deployments.
I plan to submit a PR with the following change:
diff --git a/stable/agent/templates/_helpers.tpl b/stable/agent/templates/_helpers.tpl
index 559791e..724c5b2 100644
--- a/stable/agent/templates/_helpers.tpl
+++ b/stable/agent/templates/_helpers.tpl
@@ -11,9 +11,17 @@ Create a default fully qualified app name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
*/}}
{{- define "fullname" -}}
+{{- if .Values.fullnameOverride -}}
+{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}}
+{{- else -}}
{{- $name := default .Chart.Name .Values.nameOverride -}}
+{{- if contains $name .Release.Name -}}
+{{- .Release.Name | trunc 63 | trimSuffix "-" -}}
+{{- else -}}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
{{- end -}}
+{{- end -}}
+{{- end -}}
{{/*
Set the Deployment API version based on the version of Kubernetes the chart is being deployed into.
With the code change above I can set nameOverride: buildkite-large-agent
in helm/vars/large.yaml.gotmpl
to match the release name and it'll produce resources called buildkite-large-agent
with the label and selector set to the same.
E.g. part of the output from Helm
buildkite, buildkite-large-agent, Deployment (apps) has been added:
+ # Source: agent/templates/deployment.yaml
+ apiVersion: apps/v1
+ kind: Deployment
+ metadata:
+ name: buildkite-large-agent
+ labels:
+ app: buildkite-large-agent
+ chart: agent-0.6.2
+ release: buildkite-large-agent
+ heritage: Helm
+ spec:
+ replicas: 1
+ selector:
+ matchLabels:
+ app: buildkite-large-agent
+ template:
In the code above I've also catered for a fullnameOverride
variable to completely override the names of resources independently of .Release.Name
and .nameOverride
if you like since that is a common pattern I see in Helm charts.
How to reproduce it (as minimally and precisely as possible):
I've shown a way to produce it with helmfile.
Anything else we need to know:
I plan to raise a PR to address this.