Document how to run multiple agents with different config files about docs HOT 9 OPEN

Hi folks! 👋

The systemd way to do this is probably using a template instance specific drop-in file.

[email protected] is a service template. [email protected] is an instance with the instance name "custom".

[email protected] inherits the config from the default at /lib/systemd/system/[email protected]. We can add instance specific config in a drop-in (describe in the systemd.unit man page) named after the template instance at /etc/systemd/system/[email protected]/custom-config.conf with contents like:

[Service]
Environment=BUILDKITE_AGENT_CONFIG=/etc/buildkite-agent/buildkite-agent-custom.cfg

This will override the config location for this one particular service instance.

You can verify this is working using systemctl cat ...:

root@468ee23db373:/# systemctl cat [email protected]
# /lib/systemd/system/[email protected]
[Unit]
Description=Buildkite Agent (%i)
Documentation=https://buildkite.com/agent
After=syslog.target
After=network.target

[Service]
Type=simple
User=buildkite-agent
Environment=HOME=/var/lib/buildkite-agent
ExecStart=/usr/bin/buildkite-agent start
RestartSec=5
Restart=on-failure
TimeoutStartSec=10
TimeoutStopSec=5min
KillMode=mixed

[Install]
WantedBy=multi-user.target
DefaultInstance=1

# /etc/systemd/system/[email protected]/custom-config.conf
[Service]
Environment=BUILDKITE_AGENT_CONFIG=/etc/buildkite-agent/buildkite-agent-custom.cfg

from docs.

What about a scenario where you want to run a few agents per host, plus an agent dedicated to pipeline uploads on a separate queue?

$ systemctl edit [email protected]
# opens /etc/systemd/system/[email protected]/override.conf in $EDITOR

[Service]
Environment=BUILDKITE_AGENT_NAME=%%hostname-pipeline-uploader
Environment=BUILDKITE_AGENT_TAGS=queue=pipeline-uploader

$ systemctl cat [email protected]
# /lib/systemd/system/[email protected]
[Unit]
Description=Buildkite Agent (%i)
Documentation=https://buildkite.com/agent
After=syslog.target
After=network.target

[Service]
Type=simple
User=buildkite-agent
Environment=HOME=/var/lib/buildkite-agent
ExecStart=/usr/bin/buildkite-agent start
RestartSec=5
Restart=on-failure
TimeoutStartSec=10
TimeoutStopSec=5min
KillMode=mixed

[Install]
WantedBy=multi-user.target
DefaultInstance=1

# /etc/systemd/system/[email protected]/override.conf
[Service]
Environment=BUILDKITE_AGENT_NAME=%%hostname-pipeline-uploader
Environment=BUILDKITE_AGENT_TAGS=queue=pipeline-uploader

$ systemctl enable --now buildkite-agent@pipeline-uploader
Created symlink from /etc/systemd/system/multi-user.target.wants/[email protected] to /lib/systemd/system/[email protected].

$ systemctl enable --now buildkite-agent@1
Created symlink from /etc/systemd/system/multi-user.target.wants/[email protected] to /lib/systemd/system/[email protected].

$ systemctl enable --now buildkite-agent@2
Created symlink from /etc/systemd/system/multi-user.target.wants/[email protected] to /lib/systemd/system/[email protected].

$ systemctl enable --now buildkite-agent@3
Created symlink from /etc/systemd/system/multi-user.target.wants/[email protected] to /lib/systemd/system/[email protected].

$ systemctl enable --now buildkite-agent@4
Created symlink from /etc/systemd/system/multi-user.target.wants/[email protected] to /lib/systemd/system/[email protected].

The default systemd description makes sure the instance name is interpolated, so it looks nice:

from docs.

@toolmantim is this still the only way to have multiple config files? Does the config path env var/flag not allow you to specify different files?

from docs.

I'm not exactly sure of the steps… I think there's a few different ways to do it. At a base level the agent let's you customise the config path, and create separate configuration files, but the Ubuntu-specific instructions here https://buildkite.com/docs/agent/v3/ubuntu#running-multiple-agents don't tell you how to do it — I believe it only shows how to run multiple agents from the same config file, because it's using a systemd template: https://www.digitalocean.com/community/tutorials/understanding-systemd-units-and-unit-files#creating-instance-units-from-template-unit-files

@sj26 is our go-to systemd expert!

from docs.

Looks like this is related to #186!

@sj26 is there a 'recommended' way to do separate config files with systemd templates?

from docs.

It'd be great to figure out a real life example, so we could use that instead of "custom". If you're doing a custom agent config, it's probably also on a separate queue? Or perhaps a separate organization?

Would you also probably want to override the unit’s Description=Buildkite Agent (%i) value too?

from docs.

What if I wanted to make let's say 5 copies of a custom service? I could use BUILDKITE_AGENT_SPAWN but then they all have the same name and can't run at the same time because they all want to work from the same directory. I can't seem to do [email protected] or anything of the sort because it just launches the default [email protected]. Do I have to make N custom services? Can I somehow get the names to be decorated by the spawn count?

from docs.

@hofstee ah, in that case you might like to add %n to the agent name which will be substituted by a number which ensures the agent name is unique across all your agents (docs). So for a custom service drop-in, it's as above but with:

[Service]
Environment=BUILDKITE_AGENT_NAME=%%hostname-pipeline-uploader-%%n
Environment=BUILDKITE_AGENT_TAGS=queue=pipeline-uploader
Environment=BUILDKITE_AGENT_SPAWN=5

You could probably also use systemd's %i and friends.

from docs.

(The default buildkite agent configuration uses the name %hostname-%n.)

from docs.