buildkite / docs Goto Github PK
View Code? Open in Web Editor NEWThe source files for the Buildkite documentation
Home Page: https://buildkite.com/docs
License: MIT License
The source files for the Buildkite documentation
Home Page: https://buildkite.com/docs
License: MIT License
Hi,
Loving Buildkite. Originally provided the below info in a ticket, but @keithpitt asked me to open an issue here, so here it is:
I had some confusion trying to understand exactly how to properly get an environment hook set up in the docker agent, to ensure my github ssh deploy key and other secrets were properly set for a pipeline which builds and publishes a docker image.
It would be great if there were more of a "step by step" example in the docker agent README for these cases.
At least, it would be good to explain exactly where to stick the environment hook (e.g. ensuring you have mounted /buildkite/hooks
with your environment
hook, which can also contain a mechanism to copy your private key to the right place).
Also, I was trying to mount the entire /buildkite
volume while using the 'stable' image, which didn't work because it blows away the bootstrap. So, you have to either mount only /buildkite/hooks
or ensure you use the beta image.
I've documented the steps I took here, feel free to steal from them - below are links to two "Buildkite-specific" sections I have in the notes I took during my setup. They are a little specific to what I did (on Rancher), but should be a good start on a more detailed step-by-step walkthrough. They are also to specific commits, so the line numbers don't get stale, but you can look at the latest raw version on master for easier copy-paste.
Thanks,
-- Chad
We have recently provisioned a brand new Mac mini for iOS builds, and have found that the instructions to start the service in the background per https://buildkite.com/docs/agent/osx don't work:
buildkite-mac-1:~$ brew info buildkite-agent
buildkite/buildkite/buildkite-agent: stable 2.6.6, devel 3.0-beta.34
Build runner for use with Buildkite
https://buildkite.com/docs/agent
/usr/local/Cellar/buildkite-agent/2.6.6 (5 files, 9.9MB) *
Built from source on 2017-10-26 at 07:52:25 with: --token=
From: https://github.com/buildkite/homebrew-buildkite/blob/master/buildkite-agent.rb
==> Options
--token=
Your account's agent token to add to the config on install
--devel
Install development version 3.0-beta.34
==> Caveats
buildkite-agent is now installed!
Configuration file (to configure agent meta-data, priority, name, etc):
/usr/local/etc/buildkite-agent/buildkite-agent.cfg
Hooks directory (for customising the agent):
/usr/local/etc/buildkite-agent/hooks
Builds directory:
/usr/local/var/buildkite-agent/builds
Log paths:
/usr/local/var/log/buildkite-agent.log
/usr/local/var/log/buildkite-agent.error.log
If you set up the LaunchAgent, set your machine to auto-login as
your current user. It's also recommended to install Caffeine
(http://lightheadsw.com/caffeine/) to prevent your machine from going to
sleep or logging out.
To run multiple agents simply run the buildkite-agent start command
multiple times, or duplicate the LaunchAgent plist to create another
that starts on login.
To have launchd start buildkite/buildkite/buildkite-agent now and restart at login:
brew services start buildkite/buildkite/buildkite-agent
Or, if you don't want/need a background service you can just run:
buildkite-agent start
Following the instructions yields:
buildkite-mac-1:~$ brew services start buildkite/buildkite/buildkite-agent
Could not find domain for
Error: Failure while executing: /bin/launchctl enable gui/502/homebrew.mxcl.buildkite-agent
The docs refer to BUILDKITE_NO_AUTOMATIC_SSH_FINGERPRINT_VERIFICATION
, which is only in Agent v3. In Agent v2 it is BUILDKITE_AUTO_SSH_FINGERPRINT_VERIFICATION
. See https://github.com/buildkite/agent/blob/2b8f1d569b659e07de346c0e3ae7090cb98e49ba/templates/bootstrap.sh#L226
I'm trying to configure a docker agent using a config file, but it seems the documented config file location is incorrect. In the documentation it says to map your config file to /buildkite/buildkite-agent.cfg
So I've got the following config file:
# file: /devops/buildkite/buildkite-agent.cfg
token="<secret>"
name="docker-test"
meta-data="ci=true,docker=true"
debug=true
And spin up a container with:
docker run -it -v "$HOME/buildkite/buildkite-agent.cfg:/buildkite/buildkite-agent.cfg:ro" buildkite/agent
It tells me: 2018-03-15 08:29:37 FATAL Missing token. See: buildkite-agent start --help
However if I run it detached and docker exec into the container to check the config file, it is correctly mounted, so that doesn't appear to be the problem.
I've also tried running buildkite/agent:3 but the issue persists
I'm personally lacking of list of all possible states for Jobs.
Maybe something else isn't documented too
Hi BK team!
We're currently evaluating BuildKite to see if it's a good fit for our build-deploy process long-term. It's been a good experience so far, but right now we're in the following situation we need to account for:
Right now our only option is to force-stop / start agents en masse, but this runs the risk of killing inflight jobs.
Is there a mechanism to either
Thank you!
There really should be only one place for official documentation.
For Docker it exists:
https://buildkite.com/docs/agent/v3/docker
https://github.com/buildkite/docker-buildkite-agent
Maybe just move Github docs into your official one ?
On this page here:
https://buildkite.com/docs/agent/v3/docker
It is has a link to "Ubuntu Dockerfile" which is broken.
Hi,
Just to be sure. We've had a member in our organization who set up the most of our buildkite environment. Now that he has left the organization I want to delete him from the users. What are the consequences of this? I previously had some bad experiences where deleting a user, automatically deleted all the things that he had set up for us.
Bye, Peter
Currently this results in no external network connectivity, because there is no NAT gateway. This either needs to be documented or a NAT gateway conditionally provisioned.
We need to update this for Agent v3:
https://buildkite.com/docs/builds/docker-containerized-builds
Its unclear how to install v3 from the Manual installation instructions.
Specifically, where does the bootstratp.sh
come from? Or is this relevant to v3
Referring to this page:
https://buildkite.com/docs/agent/v3/cli-artifact#uploading-artifacts
Within the "Uploading artifacts" section it shows this:
$ export BUILDKITE_S3_ACCESS_KEY_ID=xxx $ export BUILDKITE_S3_SECRET_ACCESS_KEY=yyy $ export BUILDKITE_S3_DEFAULT_REGION=eu-central-1 # default is us-east-1 $ export BUILDKITE_S3_ACL=private # default is public-read $ buildkite-agent artifact upload "log/**/*.log" s3://name-of-your-s3-bucket/$BUILDKITE_JOB_ID
But then further down in the "Using your own private AWS S3 bucket" section it says:
"To do so, you’ll need to export the following environment variables using an environment agent hook (this can not be set via the Buildkite web interface, API, or during pipeline upload):"
export BUILDKITE_ARTIFACT_UPLOAD_DESTINATION="s3://name-of-your-s3-bucket/$BUILDKITE_JOB_ID" export BUILDKITE_S3_DEFAULT_REGION="eu-central-1" # default: us-east-1
Do I need to use an environment agent hook or can I just export variables from the pipeline code ?
It’d be helpful if the trigger step documentation included some mention that the user who starts the triggering build must have access to build in the triggered build.
The https://buildkite.com/docs/pipelines/branch-configuration page should also provide some sample pipeline.yml
with step-level branch filtering.
When I export a pipeline it says "label:" on the steps. https://buildkite.com/docs/guides/uploading-pipelines has something that says "name:", but never mentions "label:". Are they the same? Is there reference documentation for this somewhere?
There is a missing image at https://buildkite.com/docs/quickstart/getting-started
Specifically email-address.png
is missing, the one reference here:
<%= image "email-address.png", size: "251x52", alt: "Screenshot of email address section of Buildkite settings" %>
https://github.com/buildkite/docs/blob/master/pages/quickstart/getting_started.md.erb#L25
It seems parts of the docs are scattered all over the place with big chunks missing
I can see definition of the main keys in a step, then followed by the main plugins and what they can do, and what the different parameters are.
I've noticed that many of the environment variables listed when doing a buildkite-agent start --help
are not documented here: https://buildkite.com/docs/guides/environment-variables, although the ones I've tried can be queried in the build step.
A useful one is BUILDKITE_BUILD_PATH
BUILDKITE_AGENT_CONFIG
BUILDKITE_AGENT_DEBUG
BUILDKITE_AGENT_DEBUG_HTTP
BUILDKITE_AGENT_ENDPOINT
BUILDKITE_AGENT_META_DATA
BUILDKITE_AGENT_NO_COLOR
BUILDKITE_AGENT_PRIORITY
BUILDKITE_AGENT_TOKEN
BUILDKITE_BOOTSTRAP_SCRIPT_PATH
BUILDKITE_BUILD_PATH
BUILDKITE_HOOKS_PATH
BUILDKITE_NO_AUTOMATIC_SSH_FINGERPRINT_VERIFICATION
BUILDKITE_NO_COMMAND_EVAL
BUILDKITE_NO_PTY
In the docs we show how to run multiple agents but we don't show any way to customise the agent configuration files for each agent… say, for giving them different tags/meta-data.
Someone just submitted the process it took for them:
To run multiple agents, on the same host, with different configurations, and behind a proxy server.
Once you have installed and tested an agent as per the installation instructions.
## Kill the single agent.
Stop the service and disable it "sudo systemctl stop buildkite-agent" and "sudo systemctl disable buildkite-agent"
## Config Files.
In the buildkite-agent home directory, `/ect/buildkite-agent` make a copy of buildkite-agent.config for each different config you want, e.g. `buildkite-agent-1.cfg`, `buildkite-agent-2.cfg`, etc. The name is irrelevant so call it what you like.
Then edit the new config files and set whatever configuration you want - I just set different queue names in the meta-data/tags.
## System launch file.
This file is `/lib/system/system/buildkite-agent.service`
Make a copy for each agent you want to run, i.e. `[email protected]`, `[email protected]`, etc., and edit this file.
There is an ExecStart line that launches the service, add to the end of this, in each file, ` -config /etc/buildkite-agent/buildkite-agent-1.cfg` or whichever conf you want to use.
Now you can enable the service, `sudo systemctl enable buildkite-agent@1`, and it should run.
## Repeat for however many agent you have setup.
This creates a symlink called `/etc/system/system/mutli-user.target.wants/[email protected]` that should link to the system launch file `/lib/sytemd/system/[email protected]` that was created earlier.
## Start the agents
Finally start the agents: `sudo systemctl start buildkite-agent@1`
My company is tarring up the node_modules and uploading this as an artifact. We then have to check in each build step if a node_modules exists and if not to download the artifact & untar it. None of the examples presented are even close to real world, since 1 step isn't close to how people use CI, which makes it quite difficult to understand whether this is the proper way to handle it.
The directions here https://buildkite.com/docs/builds/parallel-builds where it talks about running multiple agents is not accurate for newer Ubuntu systems that use systemd
based configurations.
Are there updates or other documentation available?
The documentation for global/build hooks mentions that:
"Each agent installer comes with a hooks directory containing a sample set of hooks"
https://buildkite.com/docs/agent/v3/hooks
But there are no sample hook files with the Docker image:
https://github.com/buildkite/docker-buildkite-agent
I'm looking at:
https://buildkite.com/docs/agent/hooks
An example hook is given as:
#!/bin/bash
set -eu
echo '--- :house_with_garden: Setting up the environment'
export GITHUB_RELEASE_ACCESS_KEY='xxx'
The first line "#!/bin/bash" implies this is a script which will get executed. If it is executed, the export won't do anything. A similar issue exists with the pre-command hook.
It's a small picky issue, but caused me some confusion and extra work when someone asked me to create an env hook.
BUILDKITE_DISABLE_GIT_SUBMODULES
buildkite/agent#168 isn’t listed on:
There are a few broken images on the parallel builds page.
Hi!
The section on Google Cloud Storage [1] is missing documentation on the below two environment variables (which I found through the agent source code):
Also, it doesn't mention that GCS is only supported with agent 3.x.
Thanks!
[1] https://buildkite.com/docs/agent/gcloud#uploading-artifacts-to-google-cloud-storage
Just signed up for https://githost.io/, trying to get it integrated with BK. Have run into some issues, would certainly be good to get it mentioned on here https://buildkite.com/docs/how-tos/gitlab
https://github.com/buildkite/docs/blob/master/pages/agent/aws.md.erb
parallizing
should be parallelizing
or parallelising
depending on your regional bias...
A customer reported that the GitLab docs are out of date:
As a side note, the links in the Gitlab setup
settings/setup/gitlab?setup=true
for automatic builds are invalid. After the recent gitlab UI changes, both services and webhooks are under/settings/integrations
I'm looking at the API docs and trying to work out how to create a pipeline in a "team"
https://buildkite.com/docs/rest-api/pipelines#create-a-pipeline
It's unclear how to do this.
Hi peeps
I'm a little confused about the environment variables & interpolation logic. For example -
env:
A: 'somevar'
steps;
- name: 'test'
env:
A: 'anothervar'
and I fire a build and set an environment within the build of
A=foobar
What does $A
result in?
It appears that the result is A=foobar
.
It seems the logic is the environment within a step, and then the global environment set within a pipeline. Any pass through environment variables that are set are ignored if there is a global environment set within the pipeline.yml
The AWS SNS guide is outdated and the linked document no longer gives enough detail to follow through with!
This agent install doc: https://buildkite.com/docs/agent/v3/redhat
References: https://yum.buildkite.com/buildkite-agent/unstable/x86_64/
But I assume that since 3.0 is now stable, that should be updated. Same thing seem to be the case for most other distros.
I run a Builtkite agent behind an HTTP proxy. The proxy only allows connections to whitelisted domains. Which domains should I have whitelisted for the Buildkite agent to work correctly?
https://agent.buildkite.com
... anything else?
We don't document how to configure buildkite-agent’s HTTP proxy settings. This was just submitted by a user who went through the pain of figuring it out:
## Proxy.
If you are behind a proxy server you need to create a folder "/etc/system/system/buildkite-agent.service.d" and in there create a file called proxy.conf with your proxy settings inside it. You'll need to stop and start the service for the changes to take affect.
Hi Team,
wondering how to schedule build using pipeline.yml file.
Thanks,
Aparna
Does BUILDKITE_BUILD_NUMBER
ever wrap around or does it increase monotonically? It would be great if this could be documented. Thanks :-)
On the following pages:
https://buildkite.com/docs/builds/artifacts
https://buildkite.com/docs/agent/v2/cli-artifact
https://buildkite.com/docs/pipelines/defining-steps
There is either no mention (first 2 pages) or little mention (last page) of the artifact_paths property for pipelines or really automatic uploading in general.
Old
The stack connects to the Buildkite API to retrieve the autoscaling metrics to spins up and down EC2 instances based on the build activity. You’ll need to create a Buildkite API Access Token for the Elastic CI Stack to use:
New
The stack connects to the Buildkite API to gather autoscaling metrics and build activity to scale up or scale down the number of EC2 instances. You’ll need to create a Buildkite API Access Token for the Elastic CI Stack to use:
Stumbled upon the incredibly useful --default flag in the source. I wouldn't have been able to successfully use meta-data without it. Would be great to see it reflected in the documentation for others to benefit from it.
https://buildkite.com/docs/graphql-api#relay-compatibility mentions:
The Buildkite GraphQL API adheres to the Relay Specification, which defines standards for querying paginated collections ("Connections" and "Edges")
The pagination doesn't seem to work though. I can see that the number of builds returned is limited to 500, however the pageInfo
element never sets hasNextPage
to true. Even though BuildConnection.count
is higher than the number of returned edges.
This may be either a lack of documentation about how to handle paging results, or a bug in the implementation.
We are often asked how to restart the agent to pick up new configuration or how to shutdown agents without killing running tasks. This needs to be documented somewhere.
https://github.com/buildkite/docs/blob/master/pages/guides/docker_containerized_builds.md.erb#L9
to define it’s own
"its" is the possessive.
In regards to this section:
https://buildkite.com/docs/agent/v3/cli-artifact#downloading-artifacts-outside-a-running-build
It isn't clear what this means. Does it mean:
(1) Downloading artifacts outside the current build
(2) Downloading artifacts outside any build
I was specifically looking to convert a CircleCI build over, and needed some guidelines for mapping over the various bits. Especially caching things like asdf
and yarn
etc.
The docs are very useful, but would be even more useful if they were searchable.
Please make it happen!
https://buildkite.com/docs/pipelines/wait-step
- command: "command.sh"
- wait: ~
continue_on_failure: true
- command: "echo This runs regardless of the success or failure"
- wait
- command: "echo The command passed"
is listed as an example of a wait step-including-pipeline that continues. Is the wait: ~
mandatory? What if we attached a blank key like
- command: "command.sh"
- wait:
continue_on_failure: true
- command: "echo This runs regardless of the success or failure"
- wait
- command: "echo The command passed"
Instead? Should that still continue? It's not clear according to the docs what role ~
has here.
Buildkite team can check this pipeline: https://buildkite.com/gusto/buildkite-superpull/builds/95#772d22ee-e9f1-44b7-a63b-f89577f91624 for an example of a wait:
key with continue_on_failure
set that doesn't have~
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.