buildkite-plugins / docker-buildkite-plugin Goto Github PK

We've had reports that long running scripts run in the docker plugin leave the container running when the job is cancelled.

I verified this with this gist: https://gist.github.com/lox/045a4b56a0c1e1c815fd011657c34b46/708574c0b0ff9ec8748d2cd736de31951da555cb

The output looks like:

For some context, the agent initiates cancellation and sends a SIGTERM to the process group of buildkite-agent bootstrap that is executing the job.

It appears that docker run doesn't like being SIGTERM'd and terminates without stopping the container. My previous understanding was that docker run would proxy signals through to the container, and whilst there were some caveats around how pid 1 operates, it should be ok with the --init flag or a tini entrypoint, however, that doesn't seem to be accurate.

Heya, I'm trying to setup a pipeline to run inside windows containers, and am getting some unexpected results. As far as I can tell, the commands aren't ran at all.

I'm running the build agent on a Windows Server version 1803 Datacenter Core for Containers image on Google Container Engine, on which I installed Git for Windows, NSSM and the build agent, as per https://buildkite.com/docs/agent/v3/windows.

My pipeline.yml looks like this

steps:    
  - label: windows-steps
    command: "echo hello"
    plugins:
      - docker#v2.0.0:
          image: "microsoft/dotnet:latest"
    agents:
      windows: true

And the output log looks like this:

Running plugin github.com/buildkite-plugins/docker-buildkite-plugin#v2.0.0 command hook
Running CMD.EXE /c 'echo hello' in microsoft/dotnet:latest | 2s
  | Microsoft Windows [Version 10.0.17134.345]
  | (c) 2018 Microsoft Corporation. All rights reserved.
  |  
  | C:\workdir>> cd C:\buildkite-agent\builds\gce-buildkite-agent-windows-1-1\angular\testsetup\c\buildkite-agent\builds\gce-buildkite-agent-windows-1-1\angular\testsetup

I thought the command might be running but the output not shown, so I tried something that should error out (command: "not-a-binary"). The build still passed, same message, no error anywhere.

I'm running a Kubernetes cluster with BuildKite running as a pod
Builds are running great, except when I'm using this plugin, which causes the following error (ANSI chars removed):

/buildkite/plugins/github-com-buildkite-plugins-docker-buildkite-plugin-v1-1-1/hooks/command
--- :docker: Running - yum install -y https://centos7.iuscommunity.org/ius-release.rpm
- yum install -y python36u python36u-pip git
- pip3.6 --disable-pip-version-check --no-cache-dir install -r requirements.txt
- COVERALLS_REPO_TOKEN="$(cat /buildkite/secrets/coveralls-token.txt)" ./test.sh
 in centos:7.4.1708
docker: Error response from daemon: error while creating mount source path '/buildkite/builds/buildkite-agent-7857746466-fcbd9/kfirz/deployster': mkdir /buildkite: read-only file system.
Error: The plugin github.com/buildkite-plugins/docker-buildkite-plugin#v1.1.1 command hook exited with status 125

The BuildKite has Docker binary & socket mounted as volumes from the hosting node.

This pattern will never work when running the build-kite agent from the official container.
Supposing the buildkite agent is run from buildkite/agent like docker installation and the docker.sock is passed in from the host using a host mount /var/run/docker.sock:/var/run/docker.sock. In this situation the agent and the plugin are both using the host docker installation however the workdir folder only exists in the agent container and not on the host. This causes the workdir bind mount for the plugin to attempt to bind to a non existant host folder. The only correct way to do this is to create a docker volume and copy the agents build context into the docker volume. Once a docker volume exists that would then be mounted to the docker plugin container. Additionally the agent container would have to be responsible for cleaning up volumes of failed or completed builds.

i'm trying to run a buildkite agent on OSX to run a CentOS docker image to run some jobs. Since i can't mount the OSX darwin build in a linux container, I am shipping the container with the agent baked in and setting BUILDKITE_PLUGIN_DOCKER_MOUNT_BUILDKITE_AGENT=0. That gets me up and running but i'm unable to use the buildkite-agent to upload or download artifacts because BUILD_ID is not passed along because of the logic behind the mount - https://github.com/buildkite-plugins/docker-buildkite-plugin/blob/master/hooks/command#L36

for now i think i can workaround by explicitly passing the needed 3 ENV vars, but it would be nice if they were passed through automatically, always, or at least with an option.

We've run into situations where the directory mounted in the container gets into a bad state, and the buildkite-agent user isn't privileged enough to delete the contents of the directory.

We had a couple solutions in mind, one of the most reasonable of which we thought would be using the docker cp command to put the checked-out directory into the container. Another way we could accomplish this is by injecting a few commands to mount read-only volumes and copying them through the container to avoid any changes propogating, but docker cp seems like a cleaner solution.

Would be willing to put in a PR to mirror the usage of mount-checkout, but for docker cp'ing a directory into the container/image at runtime. Would like to hear your thoughts!

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Location: renovate.json
Error type: The renovate configuration file contains some invalid settings
Message: Invalid configuration option: label

When using this plugin, all the builds for the same repository are checked out in the same directory:

Preparing working directory | 46s
-- | --
  | > cd C:\buildkite-agent\builds\gce-buildkite-windows-1-1\angular\angular
  | > git remote set-url origin https://github.com/angular/angular
  | > git clean -fxdq

After checkout, the repository root will be mounted on a Docker container using the --volume flag, which shares file changes between the container and the host file systems.

If there is a build already running, and a second build is triggered, the new commit will be checked out on the same directory. The already running build will have its code changed in mid-build.

At best it will test the wrong code, and at worst it will crash the build or other unexpected behaviour.

Even with Cancel Intermediate Builds turned on, this can cause odd behaviour as files can still be locked while the previous build is being cancelled. This is especially noticeable on windows where locked files/directories cannot be deleted.

Regardless, multiple running builds should not interfere with each other. It is common to have multiple builds running at any given time (e.g. two PRs opened close together).

Right now the Docker plugin only supports specifying an existing docker image. It would be really awesome to be able to specify a local dockerfile and have that built and used. The docker compose plugin supports this now and I am using that for now.

I'm not sure if I'm doing something wrong, or if this is an actual bug, so apologies if it's the former.

Whatever I do, I get the error docker: command not found. Even if my step is something very simple, like an echo statement:

Running plugin github.com/buildkite-plugins/docker-buildkite-plugin#v1.1.1 command hook | 0s
-- | --
  | # Executing "/home/buildkite/.buildkite-agent/plugins/github-com-buildkite-plugins-docker-buildkite-plugin-v1-1-1/hooks/command"
  |   | Running echo 'Does this work?' . in docker:latest | 0s
  | /home/buildkite/.buildkite-agent/plugins/github-com-buildkite-plugins-docker-buildkite-plugin-v1-1-1/hooks/command: line 62: docker: command not found

Here is my pipeline.yml:

steps: 
  - label: 'test plugin'
    command: echo 'Does this work?'
    plugins: 
      docker#v1.1.1:
        image: "docker:latest"
        mounts: 
          - /var/run/docker.sock:/var/run/docker.sock

I took a look at the hook script and the failing line (62) is:

docker run "${args[@]}" "${BUILDKITE_PLUGIN_DOCKER_IMAGE}" bash -c "${BUILDKITE_COMMAND}"

The command is run like this:

docker run "${args[@]}" "${BUILDKITE_PLUGIN_DOCKER_IMAGE}" bash -c "${BUILDKITE_COMMAND}"

When using Windows Server containers this doesn't work without some messing around. In my environment bash -c could be safely dropped. I would have expected that when adding the buildkite command to my pipeline file it would be the entire cmd section, rather than wrapped.

bash also isn't present in some minimal Linux containers IIRC (I think Alpine just has 'sh', or at least it used to)

Personally I'd prefer this to drop the bash -c entirely and assume that the pipeline author would add this if required. If there's a reason for this being there that I'm missing, perhaps you could provide a mechanism for overriding the interpreter in the config. For example:

  - command:  build-windows.ps1
    name: Windows build
    plugins:
      docker:
        image: microsoft/windowsservercore:1709
        always-pull: true
        interpreter: powershell -Command
    agents:
      windows: true

My manifest file is like this:

docker#v1.1.1:
  image: "path-to-image:$BUILDKITE_COMMIT"

and running it under buildkite agent 3.1.0 gave error:

docker: invalid reference format.

It's seems to me UNIX env var interpolation is not supported. I would like to double-check to make sure. cc @toolmantim Much thanks in adv

Reading the Readme:

and the required environment variables set

lead me to believe that all the BUILDKITE_* env vars would be passed into the docker container but it turns out after some debugging that is not to be the case, and only 3 vars are actually passed in:
BUILDKITE_JOB_ID, BUILDKITE_BUILD_ID and BUILDKITE_AGENT_ACCESS_TOKEN.

Is that the intention or as per the code in https://github.com/buildkite-plugins/docker-buildkite-plugin/blob/master/hooks/command#L44 that only the very specifically names env vars are passed into the container along with the 3 above? If thats the case it would be good to be explicit about this in the Readme.

I have a WORKDIR set in the Docker image I'm wanting to run, and wasn't expecting to have this overwritten

It happens every now and then that a build fails because there was a (transient) issue to retrieve an image.

It would be nice to retry the image download, e.g., have three tries (maybe with a 5 sec and 30 sec back-off respectively) and only fail if the image is still not available then. I'm not sure about what docker reports, but we are seeing 5xx errors if we cannot get an image, if that could be distinguished from 4xx then those could fail immediately, no point in retrying once we got a definite answer from the other side.

This Fawkes build #73 shows the problem. A simple retry of the step a little later "fixed" it. Even though this was with 1.4.0, the current code has not changed and still tries only once.

In v3.3.0, the entrypoint documentation specifies:

Set it to false to disable the default entrypoint for the image.

However, doing so doesn't disable the entrypoint and I see no --entrypoint flag in the logged docker run command.

steps:
  - command: ls
    plugins:
      - docker#v3.3.0:
          image: hashicorp/terraform:light
          entrypoint: false

docker run -it --rm --init --volume /var/lib/buildkite-agent/builds/xxxx:/workdir --workdir /workdir --env BUILDKITE_JOB_ID --env BUILDKITE_BUILD_ID --env BUILDKITE_AGENT_ACCESS_TOKEN --volume /usr/bin/buildkite-agent:/usr/bin/buildkite-agent --label com.buildkite.job-id=6b701e83-acb8-4834-a129-ccb3266577a7 hashicorp/terraform:light /bin/sh -e -c ls

Here the default entrypoint is being used and for this image ends up running terraform /bin/sh -e -c ls. This fails because /bin/sh is an invalid subcommand for terraform.

Work-around

Setting this attribute to an empty string seems to produce the desired behaviour.

entrypoint: ""

docker run -it --rm --init --volume /var/lib/buildkite-agent/builds/xxxx:/workdir --workdir /workdir --env BUILDKITE_JOB_ID --env BUILDKITE_BUILD_ID --env BUILDKITE_AGENT_ACCESS_TOKEN --volume /usr/bin/buildkite-agent:/usr/bin/buildkite-agent --entrypoint '' --label com.buildkite.job-id=107582c8-92e5-4a95-8e34-93523f5c6a82 hashicorp/terraform:light ls

Suggestion

I think we should update the plugin behaviour to match the documentation (alternatively, update the documentation to match the plugin behaviour).

e.g.: I can't disable quoting even though the command option of the docker plugin should allow me to pass things verbatim.

A step like this:

  - label: ':bash: Lint: SH'
    command: ''
    artifact_paths:
      - web/target/shellcheck-*.xml
    plugins:
      - 'docker#v3.3.0':
          image: 'koalaman/shellcheck:v0.7.0'
          debug: false
          workdir: /workdir
          command:
            - '-x'
            - a.sh
            - '--format'
            - checkstyle
            - '>'
            - web/target/shellcheck-42820572-83ee-4588-abf2-4f761754dba9.xml

results in:

koalaman/shellcheck:v0.7.0 -x x.sh --format checkstyle \> web/target/shellcheck-42820572-83ee-4588-abf2-4f761754dba9.xml

which means the output of the docker image is not redirected and thus not picked up as an artifact.

Hey

Installed the agent on Ubuntu 16.04, and installed docker. Getting the following:

docker: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.37/containers/create: dial unix /var/run/docker.sock: connect: permission denied.

steps:
  - command: docker build . -t image:tag
    plugins:
      docker#v1.1.1:
        image: "docker:latest"
        mounts:
          - /var/run/docker.sock:/var/run/docker.sock

I traditionally use /app for where the app in the container lives, so having this mounted for the buildkite-agent path lead to some debugging confusion

Right now there's no way to tell buildkite what method it should use to communicate with the docker daemon (it assumes the default socket perms), which forces users to add the executing user to the docker group, which can be unsafe when people are executing arbitrary code inside the containers.

With a DOCKER_HOST (or whatever var), the script could be adjusted such that it propagated the -H option to the docker command, and then non-privileged users could route their traffic over the 127.0.0.1 loopback or similar to avoid having to elevate the permissions of the buildkite-agent user to what is essentially root.

Our deploy pipeline is basically a docker run using this plugin. For other pipelines it wouldn't matter, but deploys... you do open it and want to see how it's going while it runs.

At the moment every time I open this pipeline I need to go and click on the tiny line that says "Running command in " so I can see the output. Would it be possible to have this block of logs open as default? Or configurable? I guess it'd mean changing this to a +++... no?

What do you all think?

steps:
  - label: ':hammer: Build image'
    command: 
      - >
        docker build --pull
        --build-arg PIP_INDEX_URL=${PIP_INDEX_URL:-https://pip.example.net/root/dev/+simple/}
        --build-arg PIP_TRUSTED_HOST=${PIP_TRUSTED_HOST}
        -t acme/app:${BUILDKITE_BRANCH}.latest .
    plugins:
      docker#v1.4.0:
        image: "docker:stable"
        debug: true
        mounts:
          - /var/run/docker.sock:/var/run/docker.sock

Running docker build --pull --build-arg 
     PIP_INDEX_URL=${PIP_INDEX_URL:-https://pip.example.net/root/dev/+simple/}
     --build-arg PIP_TRUSTED_HOST=${PIP_TRUSTED_HOST} -t acme/app:citest.latest .
--|--
  | in docker:stable
  | $ docker run -it --rm --volume /buildkite/builds/abcdef123-1/acme/app:/workdir 
      --workdir /workdir --env BUILDKITE_JOB_ID --env BUILDKITE_BUILD_ID --volume
      /var/run/docker.sock:/var/run/docker.sock docker:stable bash -e -c docker build --pull 
      --build-arg PIP_INDEX_URL=${PIP_INDEX_URL:-https://pip.example.net/root/dev/+simple/} 
      --build-arg PIP_TRUSTED_HOST=${PIP_TRUSTED_HOST} -t acme/app:citest.latest .
  |  
  | /usr/local/bin/docker-entrypoint.sh: exec: line 35: bash: not found
  | 🚨 Error: The command exited with status 127

The docker images have busybox as sh, and no bash.

shell (optional)
Set the shell to use for the command. Set it to false to pass the command directly to the docker run command. The default is bash -e -c.

For my example setting shell: false works fine.

I wonder whether it should be sh -e -c to try and catch as many image variants as possible? The most common ${FOO:-default} etc stuff is all handled relatively consistently, even if some of the super-crazy bash'isms aren't. Or add a note to the docker build doc to say that if you need any complex stuff use shell: "sh -ec"?

I tried using the shell option , but unfortunately the constructed command is still invalid:

[...] --label com.buildkite.job-id=c691aa38-56e7-4abb-8acc-8546b863420d my/image0 -p 80:8080 /bin/sh -e -c my/script.sh

it needs to be:

[...] --label com.buildkite.job-id=c691aa38-56e7-4abb-8acc-8546b863420d -p 80:8080 my/image0 /bin/sh -e -c my/script.sh

Trying to use this plugin but my agent is giving:

🚨 Error: Failed to parse a plugin definition: Configuration for "github.com/buildkite-plugins/image-buildkite-plugin" is not a hash

Using latest docker image.

pipeline.yml looks like:

steps:
- label: 'go env'
  command: go env
  plugins:
  - docker#v3.0.1:
    image: "golang:latest"

I am running the following very basic pipeline using the Docker plugin:

steps:
  - command: "go version"
    plugins:
      - docker#v3.3.0:
          image: "golang:1.11"

The agent is running in GKE with the following configuration (I tried both buildkite/agent:3-ubuntu and buildkite/agent:3 images):

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: buildkite-agent
  name: buildkite-agent
spec:
  replicas: 1
  selector:
    matchLabels:
      app: buildkite-agent
  template:
    metadata:
      labels:
        app: buildkite-agent
    spec:
      containers:
      - env:
        - name: BUILDKITE_AGENT_TOKEN
          valueFrom:
            secretKeyRef:
              key: token
              name: buildkite-agent
        image: buildkite/agent:3-ubuntu
        imagePullPolicy: Always
        name: buildkite-agent
        securityContext:
          privileged: true
        volumeMounts:
        - mountPath: /usr/bin/docker
          name: docker-binary
        - mountPath: /var/run/docker.sock
          name: docker-socket
        - mountPath: /root/.ssh/id_rsa
          name: ssh-keys
          subPath: id_rsa
        - mountPath: /root/.ssh/id_rsa.pub
          name: ssh-keys
          subPath: id_rsa.pub
      restartPolicy: Always
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - hostPath:
          path: /usr/bin/docker
        name: docker-binary
      - hostPath:
          path: /var/run/docker.sock
        name: docker-socket
      - name: ssh-keys
        secret:
          defaultMode: 256
          secretName: buildkite-agent-ssh

The logs from the build show that the following command is being executed (I beautified the command for more readability):

docker run -it
	--rm
	--init
	--volume /buildkite/builds/buildkite-agent-xxxxxxxx-xxxx-x/xxxx/xxxxx:/workdir
	--workdir /workdir
	--env BUILDKITE_JOB_ID
	--env BUILDKITE_BUILD_ID
	--env BUILDKITE_AGENT_ACCESS_TOKEN
	--volume /usr/local/bin/buildkite-agent:/usr/bin/buildkite-agent
	--label com.buildkite.job-id=xxxxxxxx-xxxxxx-xxxxxx-xxxxx-xxxxxxxx
		golang:1.11 /bin/sh -e -c go\ version

The command results in the following error no matter which configuration I try to use:

 /buildkite/plugins/github-com-buildkite-plugins-docker-buildkite-plugin-v3-3-0/hooks/command: line 385:   168 Segmentation fault      (core dumped) docker run "${args[@]}"

It looks like something is going wrong with the Docker instance, as additional information I am running the GKE version of Kubernetes 1.13.7-gke.8 with preemptible nodes enabled in europe-west3-a.

Hi there,

I would like to be able to connect a build container to a 'custom' docker network when using the plugin.

Ideally it should create the network if it does not already exist.

Will have a play around and submit a pull request later today.

Cheers

Rob

Instead of just an image name.

https://github.com/buildkite-plugins/docker-buildkite-plugin/blob/master/hooks/command#L82 checks for a non-empty string in entrypoint, but it should allow "" so you can reset the entrypoint without having to do weird things like entrypoint: "/bin/sh"; shell: "-ec"

expand_relative_volume_path: Doesn't expand all relative volumes. When putting in .cache:/path/inside/container it doesn't expand to the expected $PWD/.cache.

I"m not so good at reading bash or regexes, but it doesn't seem to expand the way I'd expect for directories that don't start with ./.

I think it's safe to assume that if the path doesn't start with / (or w/e the equivalent in windows is), then it's a relative path no?

If you define multiple steps in a standard BuildKite pipeline are defined, the pipeline will only pass if all of the commands return a 0 exit status. When running in the Docker plugin, we are noticing this behaviour is missing. Here is a sample pipeline.yml for reproducing the issue:

steps:
  - name: "multiple command first fail bash"
    command:
      - /bin/bash -c 'exit 1'
      - /bin/bash -c 'exit 0'

  - name: "multiple command first fail docker"
    plugins:
      docker:
        always-pull: true
        image: 'ubuntu'
    command:
      - /bin/bash -c 'exit 1'
      - /bin/bash -c 'exit 0'

Both pipelines should fail but only the non-Docker pipeline fails.

I noticed that commands are being added into one big string. Shouldn't a list of commands run one by one in their own docker run command?

The anka-buildkite-plugin does it this way: https://github.com/chef/anka-buildkite-plugin/blob/7fc658b99133d5c3e11c60a2c3d47dd9d335fc2d/hooks/command#L125

This means we have to have slightly different commands executed between mac and linux builds and complicates managing our pipeline.yml

Hi,

This plugin is really basic, which pretty sucks for people moving from Gitlab CI.

If I make some pull request, would you merge it?

I have been using a private fork that has

• custom volume mounts (useful for docker-beside-docker and other stuff like mounting secrets)
• always_pull options
• custom entrypoint (instead of always locking it to bash, which is pretty useless in alpine images)

It's not really cleaned but I can take a few hours to clean it up and make some pull requests.

Let me know.

Cheers

I am just testing out buildkite running it in google kubernetes engine, following the docs and I am running into issues when wanting to mount in secrets as a file via kubernetes. This is my yaml:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
 name: buildkite-agent
 namespace: ci
spec:
 replicas: 1
 template:
   metadata:
     labels:
       app: buildkite-agent
   spec:
     containers:
       - name: buildkite-agent
         image: buildkite/agent:3
         imagePullPolicy: Always
         securityContext:
           privileged: true
         env:
           - name: BUILDKITE_AGENT_TOKEN
             valueFrom: {secretKeyRef: {name: buildkite-agent, key: token}}
           - name: BUILDKITE_BUILD_PATH
             value: "/home/kubernetes/flexvolume/buildkite/builds"
         volumeMounts:
           - name: docker-binary
             mountPath: /usr/bin/docker
           - name: docker-socket
             mountPath: /var/run/docker.sock
           - name: ssh-keys
             mountPath: /root/.ssh/id_rsa
             subPath: id_rsa
           - name: ssh-keys
             mountPath: /root/.ssh/id_rsa.pub
             subPath: id_rsa.pub
           - name: secrets
             mountPath: /home/kubernetes/flexvolume/secrets
           - name: buildkite-path
             mountPath: /home/kubernetes/flexvolume/buildkite/builds
     volumes:
       - name: docker-binary
         hostPath: {path: /usr/bin/docker}
       - name: docker-socket
         hostPath: {path: /var/run/docker.sock}
       - name: buildkite-path
         hostPath: {path: /home/kubernetes/flexvolume/buildkite/builds}
       - name: ssh-keys
         secret:
           secretName: buildkite-agent-ssh
           defaultMode: 0400
       - name: secrets
         secret:
           secretName: buildkite-agent-secrets
           defaultMode: 0400

Then my pipeline.yml:

steps:
  - label: Build
    command: ls -lah /home/kubernetes/flexvolume/secrets && cat /home/kubernetes/flexvolume/secrets/secrets.libsonnet
    plugins:
      - docker#v3.0.1:
          image: "ecosiadev/bazel-circleci:latest"
          mount-buildkite-agent: false
          propagate-environment: true
          additional-groups: ['root']
          user: 'root'
          volumes:
            - "/home/kubernetes/flexvolume/secrets:/home/kubernetes/flexvolume/secrets"

I can see that the secrets file exists in the first ls but for the cat command I am getting the error cat: /home/kubernetes/flexvolume/secrets/secrets.libsonnet: Is a directory. I can see that the kubernetes pod has the secrets file and I can read it but I have no idea why it is turned into a directory through this volume mount. Could it be in any way related to #61?

Commands passed to docker run should be unfurled, otherwise they do not work as intended, for example the node snipped from the readme:

  - command:
      - "yarn install"
      - "yarn run test"
    plugins:
      - docker#v3.1.0:
          image: "node:7"
          always-pull: true
          environment:
            - "MY_SECRET_KEY"
            - "MY_SPECIAL_BUT_PUBLIC_VALUE=kittens"

would output:

yarn run test' in node:7
--
  | docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "exec: \"yarn install\\nyarn run test\": executable file not found in $PATH": unknown.
  | 🚨 Error: The command exited with status 127

This is because its like doing:
docker run --rm node:7 "yarn install" instead of docker run --rm node:7 yarn install

Line:

should probably change to

[...] docker run ${args[@]}

I've got the following

  - label: ":centos: [CentOS] 7 Ensure Dependency Image Exists"
    command:
      - "$APT_PREP"
      # - "mkdir -p ~/.ssh && chmod 600 ~/.ssh && ssh-keyscan -H github.com >> ~/.ssh/known_hosts"
      - "$ENSURE_LINUX_IMAGE centos:7 $IMAGE_TAG_VERSION"
    agents:
      queue: "automation-eos-builder-fleet"
    plugins:
      - docker#v3.2.0:
          image: "ubuntu:18.04"
          shell: ["/bin/bash", "-i", "-e", "-c"]
          debug: true
    timeout: 120
    skip: $SKIP_CENTOS_7

Buildkite is showing that running as

docker run -it --rm --init --volume /home/kubernetes/flexvolume/buildkite/builds/buildkite-k8s-agents-eos-builder-588d697f99-wgml2-1/EOSIO/base-images-test:/workdir --workdir /workdir --env BUILDKITE_JOB_ID --env BUILDKITE_BUILD_ID --env BUILDKITE_AGENT_ACCESS_TOKEN --volume /usr/local/bin/buildkite-agent:/usr/bin/buildkite-agent ubuntu:18.04 /bin/bash -i -e -c apt-get update && apt-get upgrade -y && apt-get install -y git sudo jq

However, I ls -laht /usr/bin/buildkite-agent and see an empty directory.

Any ideas what's wrong?

When the buildkite-agent runs a second time it cannot cleanup the git directory because it doesn't own the files created by the previous build in the docker container.

Is there a recommended way of dealing with this? Adding a hook that runs another container and does a chown of some kind would probably work, but I was hoping for something a little more clean.

I'm trying to use shellcheck to check multiple scripts in a repository, using a glob like **/*.sh to avoid to have an error-prone manual list of all the scripts (the same applies to scripts/*.sh or similar). Per the README (and #74), this needs to be done via command:

steps:
  - plugins:
      docker#2d93e90397c1497a8e15a5be2bdaf7aca59af3f3:
        image: "koalaman/shellcheck"
        command: ["**/*.sh"]

However this fails, because the glob isn't expanded:

Running **/*.sh in koalaman/shellcheck:latest
**/*.sh: **/*.sh: openBinaryFile: does not exist (No such file or directory)

I might be misreading the documentation, but there doesn't seem to be a way to force the expansion, nor does there seem to be a way to more directly with an image like this one without just dropping the plugin entirely.

I created a pipeline that was (almost exactly) copied from this plugin's README:

steps:
  - plugins:
      docker#v1.4.0:
        image: "koalaman/shellcheck"
        command: ["./scripts/env.sh"]

This failed with:

--- :docker: Running  in koalaman/shellcheck
Unable to find image 'koalaman/shellcheck:latest' locally
latest: Pulling from koalaman/shellcheck
4ce50f5e1cc9: Pull complete 
72c15ff5b56b: Pull complete 
Digest: sha256:efc19e81d1a17fad9cbf4b6630d6bdd50839e6573402063f454459764c3538b0
Status: Downloaded newer image for koalaman/shellcheck:latest
Invalid number: -c
🚨 Error: The command exited with status 3

I get the same error message when I run docker run -it koalaman/shellcheck /bin/sh -e -c locally, suggesting that command isn't defaulting the shell option to false as documented.

# Plugin "github.com/buildkite-plugins/docker-buildkite-plugin" will be checked out to "/var/lib/buildkite-agent/plugins/github-com-buildkite-plugins-docker-buildkite-plugin-v1-4-0"
$ cd /var/lib/buildkite-agent/plugins/github-com-buildkite-plugins-docker-buildkite-plugin-v1-4-0
# Switching to the plugin directory
$ git clone -v -- https://github.com/buildkite-plugins/docker-buildkite-plugin .
Cloning into '.'...
POST git-upload-pack (765 bytes)
remote: Enumerating objects: 77, done.
remote: Counting objects: 100% (77/77), done.
remote: Compressing objects: 100% (28/28), done.
remote: Total 579 (delta 38), reused 66 (delta 33), pack-reused 502
Receiving objects: 100% (579/579), 151.18 KiB | 726.00 KiB/s, done.
Resolving deltas: 100% (255/255), done.
# Checking out `v1.4.0`
$ git checkout -f v1.4.0
Note: checking out 'v1.4.0'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at e6a11d4... Merge pull request #55 from buildkite-plugins/make-sure-to-error-on-multiple-command-failure
$ cd /
... snip ...
~~~ Running global pre-command hook
$ /etc/buildkite-agent/hooks/pre-command
~~~ Authenticating with AWS ECR
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Login Succeeded
~~~ :docker: Logging into docker registry hub.docker.com
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Login Succeeded
# BUILDKITE_PLUGIN_ECR_LOGIN changed
# BUILDKITE_PLUGIN_DOCKER_LOGIN_USERNAME changed
# BUILDKITE_PLUGIN_DOCKER_LOGIN_PASSWORD changed
~~~ Running plugin github.com/buildkite-plugins/docker-buildkite-plugin#v1.4.0 command hook
$ /var/lib/buildkite-agent/plugins/github-com-buildkite-plugins-docker-buildkite-plugin-v1-4-0/hooks/command
--- :docker: Running  in koalaman/shellcheck
Unable to find image 'koalaman/shellcheck:latest' locally
latest: Pulling from koalaman/shellcheck
4ce50f5e1cc9: Pull complete 
72c15ff5b56b: Pull complete 
Digest: sha256:efc19e81d1a17fad9cbf4b6630d6bdd50839e6573402063f454459764c3538b0
Status: Downloaded newer image for koalaman/shellcheck:latest
Invalid number: -c
🚨 Error: The command exited with status 3
^^^ +++
^^^ +++
~~~ Running global pre-exit hook
$ /etc/buildkite-agent/hooks/pre-exit
~~~ Stopping ssh-agent 4792
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 4792 killed;

I want to be able to clone private git repositories within my ci builds and tried to set mount-ssh-agent: true but all I am seeing is:
/buildkite/plugins/github-com-buildkite-plugins-docker-buildkite-plugin-v3-0-1/hooks/command: line 169: SSH_AUTH_SOCK: unbound variable, is there anything I need to set up in the agent for this to work?

in the readme it says:

Also see the Docker Compose Buildkite Plugin which supports docker-compose.yml, multiple containers, and overriding many of Docker’s defaults.

Should also mention it allows building images which I think would be a massive benefit.

This is a feature request

Enable buildkit to improve performance

https://docs.docker.com/develop/develop-images/build_enhancements/#to-enable-buildkit-builds

raised here too seek-oss/docker-ecr-cache-buildkite-plugin#12

Docker registry seems to have had a ton of issues in the last few days. It would be nice if the Docker plugin itself had retries and could be reliable.

I know we can retry steps based on exit codes, but trying to keep our scripts and plugin setup exit codes non-overlapping seems like a tough challenge.

Examples:

• https://buildkite.com/batfish/pybatfish-pre-commit/builds/849#52d71847-31a6-447f-94c4-8499abe5d154
• https://buildkite.com/batfish/pybatfish-pre-commit/builds/814#1f25c187-4d59-4c4c-9c95-847f13151d19
• https://buildkite.com/batfish/batfish-pre-commit/builds/2407#5f82824c-d3db-416f-a144-eb5eb3c8cc40

(The containers in question are public, so 503 seems like an invalid response code).

We're using the docker-buildkite-plugin to achieve a basic level of build isolation to make it safer to run pull requests from third-party forked repositories.

However, for this to be effective, we need to make sure we set mount-buildkite-agent: false for each invocation of the docker-buildkite-plugin, as otherwise the agent token is leaked into the docker container, allowing a malicious PR author to run arbitrary pipelines on our build agents.

I'd like to suggest that mount-buildkite-agent should be false by default.

As stated in (#53) buildkite env vars are not passed as env vars inside the container. This is making the parallel execution functionality unusable due to the fact that BUILDKITE_PARALLEL_JOB and BUILDKITE_PARALLEL_JOB_COUNT are not present inside the container.

How can this be addressed?

The documentation isn't explicit about not accepting a list of commands so I assumed it would work, as it does with non-Docker builds. Maybe a feature request? It should help with readability.

Tested with buildkite-agent 3.2.0, docker plugin 1.3.0

pipeline.yml:

steps:
  - command:
      - "apk add --no-cache python make gcc"
      - "npm install"
      - "npm test"
    plugins:
      docker:
        image: "node:8-alpine"
        always-pull: true

Error:

~~~ Setting up plugins
# Plugin "github.com/buildkite-plugins/docker-buildkite-plugin" already checked out (5773f6a)
~~~ Preparing working directory
$ cd /home/buildkite-agent-4/builds/hostname/fluid-project/kettle
$ git remote set-url origin https://github.com/fluid-project/kettle.git
$ git clean -fqd
# Fetch and checkout pull request head from GitHub
$ git fetch -v origin refs/pull/43/head
From https://github.com/fluid-project/kettle
 * branch            refs/pull/43/head -> FETCH_HEAD
# FETCH_HEAD is now `30cb8b81e13b5ae7ea936edb5f32d5475d9d8c51`
$ git checkout -f 30cb8b81e13b5ae7ea936edb5f32d5475d9d8c51
HEAD is now at 30cb8b8... KETTLE-70 - Switch to Alpine image
# Cleaning again to catch any post-checkout changes
$ git clean -fqd
# Checking to see if Git data needs to be sent to Buildkite
$ buildkite-agent meta-data exists buildkite:git:commit
~~~ Running plugin github.com/buildkite-plugins/docker-buildkite-plugin command hook
$ /etc/buildkite-agent/plugins/github-com-buildkite-plugins-docker-buildkite-plugin/hooks/command
--- :docker: Pulling node:8-alpine
8-alpine: Pulling from library/node
605ce1bd3f31: Already exists 
45d835fcee17: Pull complete 
f2b4bf87c277: Pull complete 
Digest: sha256:01e564c3a2f1c29b62aeafd21611712f9973b953ad9f595b9e112260160e1ae7
Status: Downloaded newer image for node:8-alpine
--- :docker: Running apk add --no-cache python make gcc
npm install
npm test in node:8-alpine
docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "exec: \"bash\": executable file not found in $PATH": unknown.
🚨 Error: The command exited with status 127
^^^ +++
^^^ +++

Thank you!

Is it required to use buildkite-agent artifact upload versus using the yaml artifact_paths field?

Because of a misconfiguration in our pipeline, we recently discovered that a failure to pull the configured docker image does not result in a failed build (screenshot attached).

I believe the problem is here

https://github.com/buildkite-plugins/docker-buildkite-plugin/blob/master/hooks/command#L221-L226

I /think/ the issue is that combining if ! and $? doesn't work as that code expects. Consider:

> cat ./test.sh 
#!/bin/bash

set -euo pipefail

if ! false; then
    rv=$?
    echo "RV is $rv"
fi
> ./test.sh 
RV is 0

-it is being passed as an argument by default. The command output isn't interactive so I'm not sure why either i or t is required.

In my Windows environment, it's leading to errors such as:

the input device is not a TTY. If you are using mintty, try prefixing the command with 'winpty'

I would assume that these params could be safely dropped.

Per the Elastic CI for AWS guidance (https://github.com/buildkite/elastic-ci-stack-for-aws#what-type-of-builds-does-this-support):

Each project should encapsulate it's dependencies via Docker and Docker Compose

This results in using this plugin for setting up build dependencies. It would be nice to add an option to propagate all environment variables to the container, e.g.

steps:
  - command: "echo \$BUILDKITE_COMMIT"
    plugins:
      - docker#v2.0.0:
         image: bash 

Rather than having to specify that BUILDKITE_COMMIT propagates to the container (with environment)

Feature request

Allow the consumer to pass in values to be applied with --add-host="host:IP to append records to /etc/hosts.

As described in Docker Network settings

Suggested syntax

  docker#v1.4.0:
    image: 'node:10.15'
    hosts:
     - 'example.com.local:127.0.0.1'
# OR
       example.com.local: '127.0.0.1'