Learn to use https://concourse.ci with this linear sequence of tutorials. Learn each concept that builds on the previous concept.
- 01 - Hello World task
- 02 - Task inputs
- 03 - Task scripts
- 04 - Basic pipeline
- 05 - Tasks extracted into Resources
- 06 - View job output in terminal
- 07 - Trigger jobs with fly
- 08 - Triggering jobs with resources
- 09 - Destroying pipelines
- 10 - Using resource inputs in job tasks
- 11 - Passing task outputs to another task
- 12 - Publishing outputs
- 13 - Actual pipeline - passing resources between jobs
- 14 - Parameterized pipelines
The following sections are found in subfolders of this repository and the tutorial continues in their README:
Thanks to Alex Suraci for inventing Concourse CI, and to Pivotal for sponsoring him and a team of developers to work on it for over a year (2014 onwards).
At Stark & Wayne we started this tutorial as we were learning Concourse in early 2015, and we've been using Concourse in production since mid-2015 internally and at nearly all client projects.
Thanks to everyone who has worked through this tutorial and found it useful. I love learning that you're enjoying the tutorial and enjoying Concourse.
Thanks for all the pull requests to help fix regressions with some Concourse versions that came out with "backwards incompatible change".
Install Vagrant/Virtualbox.
Fetch this tutorial and start a local Concourse server:
git clone https://github.com/starkandwayne/concourse-tutorial.git
cd concourse-tutorial
vagrant box add concourse/lite --box-version $(cat VERSION)
vagrant up
Open http://192.168.100.4:8080/ in your browser:
Once the page loads in your browser, click to download the fly CLI appropriate for your operating system:
Once downloaded, copy the fly binary into your path ($PATH), such as /usr/local/bin or ~/bin. Don't forget to also make it executable. For example,
sudo mkdir -p /usr/local/bin
sudo mv ~/Downloads/fly /usr/local/bin
sudo chmod 0755 /usr/local/bin/fly
In the spirit of declaring absolutely everything you do to get absolutely the same result every time, the fly CLI requires that you specify the target API for every fly request.
First, alias it with a name tutorial (this name is used by all the tutorial task scripts):
fly --target tutorial login --concourse-url http://192.168.100.4:8080
fly --target tutorial sync
You can now see this saved target Concourse API in a local file:
cat ~/.flyrc
Shows a simple YAML file with the API, credentials etc:
targets:
tutorial:
api: http://192.168.100.4:8080
token:
type: ""
value: ""When we use the fly command we will target this Concourse API using fly --target tutorial.
@alexsuraci: I promise you'll end up liking it more than having an implicit target state :) Makes reusing commands from shell history much less dangerous (rogue fly configure can be bad)
The central concept of Concourse is to run tasks. You can run them directly from the command line as below, or from within pipeline jobs (as per every other section of the tutorial).
cd 01_task_hello_world
fly --target tutorial execute --config task_hello_world.yml
The output starts with
executing build 1
initializing
Every task in Concourse runs within a "container" (as best available on the target platform). The task_hello_world.yml configuration shows that we are running on a linux platform using the busybox container image.
Within this container it will run the command echo hello world:
---
platform: linux
image_resource:
type: docker-image
source: {repository: busybox}
run:
path: echo
args: [hello world]At this point in the output above it is downloading a Docker image busybox. It will only need to do this once; though will recheck every time that it has the latest busybox image.
Eventually it will continue and invoke the command echo hello world successfully:
running echo hello world
hello world
succeeded
Try changing the image_resource: and the run: and run a different task:
---
platform: linux
image_resource:
type: docker-image
source: {repository: ubuntu, tag: "14.04"}
run:
path: uname
args: [-a]This task file is provided for convenience:
fly --target tutorial execute --config task_ubuntu_uname.yml
The output looks like:
executing build 2
initializing
running uname -a
Linux 96f1e0d6-ffdb-40e1-5e8b-5ae9a3b36a58 4.2.0-42-generic #49~14.04.1-Ubuntu SMP Wed Jun 29 20:22:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
succeeded
The reason that you can select any base image (or image_resource when configuring a task) is that this allows your task to have any prepared dependencies that it needs to run. Instead of installing dependencies each time during a task you might choose to pre-bake them into an image to make your tasks much faster.
In the previous section the only inputs to the task container were the image used. Base images, such as Docker images, are relatively static and relatively big, slow things to create. So Concourse supports inputs into tasks to pass in files/folders for processing.
Consider the working directory of a task that explicitly has no inputs:
cd ../02_task_inputs
fly --target tutorial execute --config no_inputs.yml
The task runs ls -al to show the (empty) contents of the working folder inside the container:
running ls -al
total 8
drwxr-xr-x 2 root root 4096 Feb 27 07:23 .
drwxr-xr-x 3 root root 4096 Feb 27 07:23 ..
In the example task inputs_required.yml we add a single input:
inputs:
- name: some-important-inputWhen we try to execute the task:
fly --target tutorial execute --config inputs_required.yml
It will fail:
error: missing required input `some-important-input`
Commonly if wanting to run fly execute we will want to pass in the local folder (.). Use -i name=path option to configure each of the required inputs:
fly --target tutorial execute --config inputs_required.yml --input some-important-input=.
Now the fly execute command will upload the . directory as an input to the container. It will be made available at the path some-important-input:
running ls -alR
.:
total 8
drwxr-xr-x 3 root root 4096 Feb 27 07:27 .
drwxr-xr-x 3 root root 4096 Feb 27 07:27 ..
drwxr-xr-x 1 501 20 64 Feb 27 07:27 some-important-input
./some-important-input:
total 12
drwxr-xr-x 1 501 20 64 Feb 27 07:27 .
drwxr-xr-x 3 root root 4096 Feb 27 07:27 ..
-rw-r--r-- 1 501 20 112 Feb 27 07:30 input_parent_dir.yml
-rw-r--r-- 1 501 20 118 Feb 27 07:27 inputs_required.yml
-rw-r--r-- 1 501 20 79 Feb 27 07:18 no_inputs.yml
To pass in a different directory as an input, provide its absolute or relative path:
fly --target tutorial execute --config inputs_required.yml --input some-important-input=../01_task_hello_world
The fly execute --input option can be removed if the current directory is the same name as the required input.
The task input_parent_dir.yml contains an input 02_task_inputs which is also the current directory. So the following command will work and return the same results as above:
fly --target tutorial execute --config input_parent_dir.yml
The inputs feature of a task allows us to pass in two types of inputs:
- requirements/dependencies to be processed/tested/compiled
- task scripts to be executed to perform complex behavior
A common pattern is for Concourse tasks to run: complex shell scripts rather than directly invoking commands as we did above (we ran uname command with arguments -a).
Let's refactor 01_task_hello_world/task_ubuntu_uname.yml into a new task 03_task_scripts/task_show_uname.yml with a separated task script 03_task_scripts/task_show_uname.sh
cd ../03_task_scripts
fly --target tutorial execute --config task_show_uname.yml
The former specifies the latter as its task script:
run:
path: ./03_task_scripts/task_show_uname.shWhere does the ./03_task_scripts/task_show_uname.sh file come from?
From section 2 we learned that we could pass inputs into the task. The task configuration 03_task_scripts/task_show_uname.yml specifies one input:
inputs:
- name: 03_task_scripts
Since input 03_task_scripts matches the current directory 03_task_scripts we did not need to specify fly execute --input 03_task_scripts=..
The current directory was uploaded to the Concourse task container and placed inside the 03_task_scripts directory.
Therefore its file task_show_uname.sh is available within the Concourse task container at 03_task_scripts/task_show_uname.sh.
The only further requirement is that task_show_uname.sh is an executable script.
1% of tasks that Concourse runs are via fly execute. 99% of tasks that Concourse runs are within "pipelines".
cd ../04_basic_pipeline
fly --target tutorial set-pipeline --config pipeline.yml --pipeline helloworld
It will display the concourse pipeline (or any changes) and request confirmation:
jobs:
job job-hello-world has been added:
name: job-hello-world
public: true
plan:
- task: hello-world
config:
platform: linux
image_resource:
type: docker-image
source: {repository: busybox}
run:
path: echo
args:
- hello world
dir: ""You will be prompted to apply any configuration changes each time you run fly set-pipeline (or its alias fly sp)
apply configuration? [yN]:
Press y.
You should see:
pipeline created!
you can view your pipeline here: http://192.168.100.4:8080/pipelines/helloworld
the pipeline is currently paused. to unpause, either:
- run the unpause-pipeline command
- click play next to the pipeline in the web ui
As suggested, un-pause a pipeline from the fly CLI:
fly --target tutorial unpause-pipeline --pipeline helloworld
Next, as suggested, visit the web UI http://192.168.100.4:8080/pipelines/helloworld.
Your first pipeline is unimpressive - a single job job-hello-world with no inputs from the left and no outputs to its right, no jobs feeding into it, nor jobs feeding from it. It is the most basic pipeline. The job is gray colour because it has never been run before.
Click on job-hello-world and then click on the large + in the top right corner. Your job will run.
Clicking the top-left "Home" icon will show the status of our pipeline. The job job-hello-world is now green. This means that the last time the job ran it completed successfully.
It is very fast to iterate on a job's tasks by configuring them in the pipeline.yml as above. You edit the pipeline.yml, run fly set-pipeline, and the entire pipeline is updated atomically.
But, as per section 3, if a task becomes complex then its run: command can be extracted into a task script, and the task itself can be extracted into a yml task file.
In section 3 we uploaded the task file and task script from our local computer with the fly execute command.
Unlike section 3, with a pipeline we now need to store the task file and task script somewhere outside of Concourse.
Concourse offers no services for storing/retrieving your data. No git repositories. No blobstores. No build numbers. Every input and output must be provided externally. Concourse calls them "Resources". Example resources are git, s3 and semver respectively.
See the section "Available concourse resources" below for the list of available built-in resources and how to find community resources. Send messages to Slack. Bump a version number from 0.5.6 to 1.0.0. Create a ticket on Pivotal Tracker. It is all possible with Concourse resources.
The most common resource to store our task files and task scripts is the git resource.
This tutorial's source repository is a Git repo, and it contains many task files (and their task scripts). For example, the original 01_task_hello_world/task_hello_world.yml.
The following pipeline will load this task file and run it. We will update the previous helloworld pipeline:
cd ../05_pipeline_task_hello_world
fly set-pipeline --target tutorial --config pipeline.yml --pipeline helloworld
The output will show the delta between the two pipelines and request confirmation. Type y. If successful, it will show:
apply configuration? [yN]: y
configuration updated
The helloworld pipeline now shows an input resource resource-tutorial feeding into the job job-hello-world.
This tutorial verbosely prefixes resource- to resource names, and job- to job names, to help you identify one versus the other whilst learning. Eventually you will know one from the other and can remove the extraneous text.
After manually triggering the job via the UI, the output will look like:
The in-progress or newly-completed job-hello-world job UI has three sections:
- Preparation for running the job - collecting inputs and dependencies
resource-tutorialresource is fetchedhello-worldtask is executed
The latter two are "steps" in the job's build plan. A build plan is a sequence of steps to execute. These steps may fetch down or update Resources, or execute Tasks.
The first build plan step fetches down (note the down arrow to the left) a git repository for these training materials and tutorials. The pipeline named this resource resource-tutorial.
The pipeline.yml documents this single resource:
resources:
- name: resource-tutorial
type: git
source:
uri: https://github.com/starkandwayne/concourse-tutorial.gitThe resource name resource-tutorial is then used in the build plan for the job:
jobs:
- name: job-hello-world
public: true
plan:
- get: resource-tutorialAny fetched resource can now be an input to any task in the job build plan. As discussed in section 3 & section 4, task inputs can be used as task scripts.
The second step runs a user-defined task. The pipeline named the task hello-world. The task itself is not described in the pipeline. Instead it is described in a file 01_task_hello_world/task_hello_world.yml from the resource-tutorial input.
The completed job looks like:
jobs:
- name: job-hello-world
public: true
plan:
- get: resource-tutorial
- task: hello-world
file: resource-tutorial/01_task_hello_world/task_hello_world.ymlThe task {task: hello-world, file: resource-tutorial/...} has access to all fetched resources (and later, to the outputs from tasks).
The name of resources, and later the name of task outputs, determines the name used to access them by other tasks (and later, by updated resources).
So, hello-world can access anything from resource-tutorial (this tutorial's git repository) under the resource-tutorial/ path. Since the relative path of task_hello_world.yml task file inside this repo is 01_task_hello_world/task_hello_world.yml, the task: hello-world references it by joining the two: file: resource-tutorial/01_task_hello_world/task_hello_world.yml
There are benefits and downsides to abstracting tasks into YAML files outside of the pipeline.
One benefit is that the behavior of the task can be kept in sync with the primary input resource (for example, a software project with tasks for running tests, building binaries, etc).
Another benefit of extracting inline tasks into task files is that pipeline.yml files can get long and it can be hard to read and comprehend all the YAML. When extracting tasks into separate files we can give them long names so that readers can understand the purpose and expectation of the tasks at a glance.
One downside is that the pipeline.yml no longer explains exactly what commands will be invoked. Comprehension of pipeline behavior is potentially reduced.
Also, when we extract inline tasks into files, fly set-pipeline is no longer the only step to updating a pipeline.
From now onwards, any change to your pipeline might require you to do one or both:
fly set-pipelineto update Concourse on a change to the job build plan and/or input/output resourcesgit commitandgit pushyour primary resource that contains the task files and task scripts
If the new behavior you intended is not showing up in the pipeline then you may have skipped one of the two steps above.
The job-hello-world had terminal output from its resource fetch of a git repo and of the hello-world task running.
In addition to the Concourse web ui you can also view this output from the terminal with fly:
fly --target tutorial watch --job helloworld/job-hello-world
The output will be similar to:
using version of resource found in cache
initializing
running echo hello world
hello world
succeeded
The --build NUM option allows you to see the output of a specific build number, rather than the latest build output.
You can see the results of recent builds across all pipelines with fly builds:
fly --target tutorial builds
The output will look like:
id pipeline/job build status start end duration
9 helloworld/job-hello-world 2 succeeded 2017-01-09@10:42:52-0800 2017-01-09@10:42:56-0800 4s
8 helloworld/job-hello-world 1 succeeded 2017-01-09@10:42:26-0800 2017-01-09@10:42:28-0800 2s
7 one-off n/a succeeded 2017-01-09@10:41:25-0800 2017-01-09@10:41:33-0800 8s
6 one-off n/a succeeded 2017-01-09@10:41:14-0800 2017-01-09@10:41:17-0800 3s
5 one-off n/a succeeded 2017-01-09@10:41:08-0800 2017-01-09@10:41:11-0800 3s
4 one-off n/a succeeded 2017-01-09@10:40:57-0800 2017-01-09@10:41:00-0800 3s
3 one-off n/a succeeded 2017-01-09@10:40:40-0800 2017-01-09@10:40:43-0800 3s
2 one-off n/a succeeded 2017-01-09@10:40:00-0800 2017-01-09@10:40:28-0800 28s
1 one-off n/a succeeded 2017-01-09@10:39:41-0800 2017-01-09@10:39:51-0800 10s
There are three ways for a job to be triggered:
- Clicking the
+button on the web UI of a job (as we did in previous sections) - Input resource triggering a job (see section 8 below)
fly --target target trigger-job --job pipeline/jobnamecommand
fly --target tutorial trigger-job --job helloworld/job-hello-world
Whilst the job is running, and after it has completed, you can then watch the output in your terminal using fly watch:
fly --target tutorial watch --job helloworld/job-hello-world
The primary way that Concourse jobs will be triggered to run will be by resources changing. A git repo has a new commit? Run a job to test it. A GitHub project cuts a new release? Run a job to pull down its attached files and do something with them.
Triggering resources are defined the same as non-triggering resources, such as the resource-tutorial defined earlier.
The difference is in the job build plan where triggering is desired.
By default, including get: my-resource in a build plan does not trigger its job.
To mark a fetched resource as a trigger add trigger: true
jobs:
- name: job-demo
plan:
- get: resource-tutorial
trigger: trueIn the above example the job-demo job would trigger anytime the remote resource-tutorial had a new version. For a git resource this would be new git commits.
The time resource has express purpose of triggering jobs.
If you want a job to trigger every few minutes then there is the time resource.
resources:
- name: my-timer
type: time
source:
interval: 2mNow upgrade the helloworld pipeline with the time trigger.
cd ../08_triggers
fly set-pipeline --target tutorial --config pipeline.yml --pipeline helloworld
This adds a new resource named my-timer which triggers job-hello-world approximately every 2 minutes.
Visit the pipeline dashboard http://192.168.100.4:8080/pipelines/helloworld and wait a few minutes and eventually the job will start running automatically.
The dashboard UI makes non-triggering resources distinct with a hyphenated line connecting them into the job. Triggering resources have a full line.
Why does time resource configured with interval: 2m trigger "approximately" every 2 minutes?
"resources are checked every minute, but there's a shorter (10sec) interval for determining when a build should run; time resource is to just ensure a build runs on some rough periodicity; we use it to e.g. continuously run integration/acceptance tests to weed out flakiness" - alex
The net result is that a timer of 2m will trigger every 2 to 3 minutes.
The current helloworld pipeline will now keep triggering every 2-3 minutes for ever. If you want to destroy a pipeline - and lose all its build history - then may the power be granted to you.
You can delete the helloworld pipeline:
fly destroy-pipeline --target tutorial --pipeline helloworld
Note, the topic of running unit tests in your pipeline will be covered in more detail in future sections.
Consider a simple application that has unit tests. In order to run those tests inside a pipeline we need:
- a task
imagethat contains any dependencies - an input
resourcecontaining the task script that knows how to run the tests - an input
resourcecontaining the application source code
For the example Go application simple-go-web-app, the task image needs to include the Go programming language. We will use the golang:1.6-alpine image from:
https://hub.docker.com/_/golang/ (see https://imagelayers.io/?images=golang:1.6-alpine for size of layers)
The task file task_run_tests.yml includes:
image_resource:
type: docker-image
source: {repository: golang, tag: 1.6-alpine}
inputs:
- name: resource-tutorial
- name: resource-app
path: gopath/src/github.com/cloudfoundry-community/simple-go-web-appThe resource-app resource will place the inbound files for the input into an alternate path. By default we have seen that inputs store their contents in a folder of the same name. The reason for using an alternate path in this example is specific to building & testing Go language applications and is outside the scope of the section.
To run this task within a pipeline:
cd ../10_job_inputs
fly set-pipeline --target tutorial --config pipeline.yml --pipeline simple-app --non-interactive
fly unpause-pipeline --target tutorial --pipeline simple-app
View the pipeline UI http://192.168.100.4:8080/pipelines/simple-app and notice that the job automatically starts.
The job will pause on the first run at web-app-tests task because it is downloading the golang:1.6-alpine image for the first time.
The web-app-tests output below corresponds to the Go language test output (in case you've not seen it before):
ok github.com/cloudfoundry-community/simple-go-web-app 0.003s
In section 10 our task web-app-tests consumed an input resource and ran a script that ran some unit tests. The task did not create anything new. Some tasks will want to create something that is then passed to another task for further processing (this section); and some tasks will create something that is pushed back out to the external world (next section).
So far our pipelines' tasks' inputs have only come from resources using get: resource-tutorial build plan steps.
A task's inputs can also come from the outputs of previous tasks. All a task needs to do is declare that it publishes outputs, and subsequent steps can consume those as inputs by the same name.
A task file declares it will publish outputs with the outputs section:
outputs:
- name: some-files
If a task included the above outputs section then its run: command would be responsible for putting interesting files in the some-files directory.
Subsequent tasks (discussed in this section) or resources (discussed in the next section) could reference these interesting files within the some-files/ directory.
cd ../11_task_outputs_to_inputs
fly set-pipeline --target tutorial --config pipeline.yml --pipeline pass-files --non-interactive
fly unpause-pipeline --target tutorial --pipeline pass-files
Open http://192.168.100.4:8080/teams/main/pipelines/pass-files in your browser and trigger job-pass-files.
In this pipeline's job-pass-files there are two task steps create-some-files and show-some-files:
The former creates 4 files into its own some-files/ directory. The latter gets a copy of these files placed in its own task container filesystem at the path some-files/.
The pipeline build plan only shows that two tasks are to be run in a specific order. It does not indicate that show-files/ is an output of one task and used as an input into the next task.
jobs:
- name: job-pass-files
public: true
plan:
- get: resource-tutorial
- task: create-some-files
file: resource-tutorial/11_task_outputs_to_inputs/create_some_files.yml
- task: show-some-files
file: resource-tutorial/11_task_outputs_to_inputs/show_files.ymlNote, task create-some-files build output includes the following error:
mkdir: can't create directory 'some-files': File exists
This is a demonstration that if a task includes outputs then those output directories are pre-created and do not need creating.
So far we have used the git resource to fetch down a git repository, and used git & time resources as triggers. The git resource can also be used to push a modified git repository to a remote endpoint (possibly different than where the git repo was originally cloned from).
cd ../12_publishing_outputs
fly set-pipeline --target tutorial --config pipeline.yml --pipeline publishing-outputs --non-interactive
fly unpause-pipeline --target tutorial --pipeline publishing-outputs
Pipeline dashboard http://192.168.100.4:8080/pipelines/publishing-outputs shows that the input resource is erroring (see orange in key):
The pipeline.yml does not yet have a git repo nor its write-access private key credentials.
Create a Github Gist with a single file bumpme, and press "Create public gist":
Copy the "SSH" git URL:
And paste it into the pipeline.yml file:
---
resources:
- name: resource-gist
type: git
source:
uri: [email protected]:0c2e172346cb8b0197a9.git
branch: master
private_key: |
-----BEGIN RSA PRIVATE KEY-----
MIIEpQIBAAKCAQEAuvUl9YU...
...
HBstYQubAQy4oAEHu8osRhH...
-----END RSA PRIVATE KEY-----
Also paste in your ~/.ssh/id_rsa private key (or which ever you have registered with github) into the private_key section.
Be careful, If you don't include the pipe symbol after private_key: you will get an error like Private keys with passphrases are not supported.
Update the pipeline:
fly set-pipeline --target tutorial --config pipeline.yml --pipeline publishing-outputs --non-interactive
Revisit the dashboard UI and the orange resource will change to black if it can successfully fetch the new [email protected]:XXXX.git repo.
After running the job-bump-date job, refresh your gist:
This pipeline is an example of updating a resource. It has pushed up new git commits to the git repo (your github gist).
Where did the new commit come from?
The bump-timestamp-file.yml task file describes a single output updated-gist:
outputs:
- name: updated-gistThe bump-timestamp-file task runs the following bump-timestamp-file.sh script:
git clone resource-gist updated-gist
cd updated-gist
echo $(date) > bumpme
git config --global user.email "[email protected]"
git config --global user.name "Concourse"
git add .
git commit -m "Bumped date"First, it copied the input resource resource-gist into the output resource updated-gist (using git clone as the preferred git way to do this). The modifications are subsequently made to the updated-gist directory, including a git commit. It is this updated-gist and its additional git commit that is subsequently pushed back to the gist by the pipeline step:
- put: resource-gist
params: {repository: updated-gist}The updated-gist output from the bump-timestamp-file task becomes the updated-gist input to the resource-gist resource (see the git resource for additional configuration) because their names match.
The bump-timestamp-file.sh script needed the git CLI tool. It could have installed it each time via apt-get install git or similar, but this would have made the task very slow. Instead bump-timestamp-file.yml task file uses a different base image concourse/concourse-ci that contains git CLI (and many other pre-installed packages). The contents of this Docker image are described at https://github.com/concourse/concourse/blob/master/ci/dockerfiles/bosh-cli/Dockerfile
Finally, it is time to make an actual pipeline - one job passing results to another job upon success.
In all previous sections our pipelines have only had a single job. For all their wonderfulness, they haven't yet felt like actual pipelines. Jobs passing results between jobs. This is where Concourse shines even brighter.
Update the publishing-outputs pipeline with a second job job-show-date which will run whenever the first job successfully completes:
- name: job-show-date
plan:
- get: resource-tutorial
- get: resource-gist
passed: [job-bump-date]
trigger: true
- task: show-date
config:
platform: linux
image_resource:
type: docker-image
source: {repository: busybox}
inputs:
- name: resource-gist
run:
path: cat
args: [resource-gist/bumpme]Update the pipeline:
cd ../13_pipeline_jobs
fly set-pipeline --target tutorial --config pipeline.yml --pipeline publishing-outputs --non-interactive
The dashboard UI displays the additional job and its trigger/non-trigger resources. Importantly, it shows our first pipeline:
The latest resource-gist commit fetched down in job-show-date will be the exact commit used in the last successful job-bump-date job. If you manually created a new git commit in your gist and manually ran the job-show-date job it would continue to use the previous commit it used, and ignore your new commit. This is the power of pipelines.
In the preceding sections you were asked to private credentials and personal git URLs into the pipeline.yml files. This would make it difficult to share your pipeline.yml with anyone who had access to the repository. Not everyone needs nor should have access to the shared secrets.
Concourse pipelines can include {{parameter}} parameters for any value in the pipeline YAML file.
Parameters are all mandatory:
cd ../14_parameters
fly set-pipeline --target tutorial --config pipeline.yml --pipeline publishing-outputs --non-interactive
The error output will be like:
failed to evaluate variables into template: 2 error(s) occurred:
* unbound variable in template: 'gist-url'
* unbound variable in template: 'github-private-key'
Somewhere secret on laptop create a credentials.yml file with keys gist-url and github-private-key. The values come from your previous pipeline.yml files:
gist-url: [email protected]:xxxxxxx.git
github-private-key: |
-----BEGIN RSA PRIVATE KEY-----
MIIEpQIBAAKCAQEAuvUl9YUlDHWBMVcuu0FH9u2gSi83PkL4o9TS+F185qDTlfUY
fGLxDo/bn8ws8B88oNbRKBZR6yig9anIB4Hym2mSwuMOUAg5qsA9zm5ArXQBGoAr
...
iSHcGbKdWqpObR7oau2LIR6UtLvevUXNu80XNy+jaXltqo7MSSBYJjbnLTmdUFwp
HBstYQubAQy4oAEHu8osRhH1VX8AR/atewdHHTm48DN74M/FX3/HeJo=
-----END RSA PRIVATE KEY-----
To pass in your credentials.yml file use the --load-vars-from or -l options:
fly set-pipeline --target tutorial --config pipeline.yml --pipeline publishing-outputs --non-interactive --load-vars-from ../credentials.yml
This tutorial now leaves this README and goes out to the dozens of subfolders. Each has its own README.
Why are there numerical gaps between the section numbers? Because renumbering is hard and so I just left gaps. If we had a cool way to renumber the sections then perhaps wouldn't need the gaps. Sorry :)
In addition to the git resource and the time resource there are many ways to interact with the external world: getting objects/data or putting objects/data.
Two ideas for finding them:
- https://github.com/concourse?query=resource - the resources bundled with every concourse installation
- https://github.com/search?q=concourse+resource - additional resources shared by other concourse users
Some of the bundled resources include:
- bosh-deployment-resource - deploy bosh releases as part of your pipeline
- semver-resource - automated semantic version bumping
- bosh-io-release-resource - Tracks the versions of a release on bosh.io
- s3-resource - Concourse resource for interacting with AWS S3
- git-resource - Tracks the commits in a git repository.
- bosh-io-stemcell-resource - Tracks the versions of a stemcell on bosh.io.
- vagrant-cloud-resource - manages boxes in vagrant cloud, by provider
- docker-image-resource - a resource for docker images
- archive-resource - downloads and extracts an archive (currently tgz) from a uri
- github-release-resource - a resource for github releases
- tracker-resource - pivotal tracker output resource
- time-resource - a resource for triggering on an interval
- cf-resource - Concourse resource for interacting with Cloud Foundry
To find out which resources are available on your target Concourse you can ask the API endpoint /api/v1/workers:
$ curl -s http://192.168.100.4:8080/api/v1/workers | jq -r ".[0].resource_types[].type" | sort
archive
bosh-deployment
bosh-io-release
bosh-io-stemcell
cf
docker-image
git
github-release
pool
s3
semver
time
tracker
vagrant-cloud
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent