This is a demo Concourse pipeline developed by SDC. Its purpose is to serve as a reference on how to develop pipelines which support the ways in which SDC work (see ONS Software Engineer Community's Workflow).

The pipeline deploys all the way to the prod environment (Cloud Foundry space) via the ci, latest and preprod environments. Deployments to ci and latest are continuous and trigger on every merge to master, deployments to preprod and prod require manual triggers.

This pipeline is made up of the following:

• A Java Microservice
• A Python Microservice
• The Acceptance Tests
• This Pipeline Repository
• The Deployment Trigger Repository

When artifacts are created, the repository resource is no longer needed in later jobs in the pipeline (because the artifact is used for deployment instead). Even so, we have decided to carry the resource through the entire pipeline so that commit hashes are visible (along side the artifact version number) in the jobs. This is intended to help they team isolate the commit causing the error when a job fails.

In order to deploy the pipeline you need access to the secrets. A template for the secrets config file can be found in secrets.example.yml. It is recommended that you copy this template to a file called secrets.yml and fill in the details (be careful not to commit your secrets.yml to git!)

The pipline is deployed with Concourse's fly command. After you have logged with fly -t ons login, you will need to run the following command:

fly -t ons set-pipeline --pipeline concourse-demo --config pipeline.yml -l secrets.yml

For more information on what is considered secret data, see ONS Software Engineer Community's Secrets Standards.

Here are the process requirements that the pipeline had to meet and the technical solutions that we came up with to satisfy them.

Pull requests are required by the teams in order to moderate the code quality of the services. All service level tests are run by TravisCI before pull requests are merged. This happens before the Concourse pipeline is triggered - which happens every time a merge is made to the master branch.

The acceptance tests run in the ci environment once all services have been successfully built and deployed to the environment. These check the functionality of the application as a whole. Any failures should be investigated and resolved as soon as possible so that the pipeline doesn't block other work from being deployed.

Concourse supports this natively, so this is one of the benefits of using it.

The reason we are using manual triggers for preprod and prod is to allow for manual testing by preventing services being continuously deployed to these environments.

This is achieved by using a trigger repo in Github. The Concourse job is triggered when a user pushes a commit to this repo. To prevent the jobs being started manually via the Concourse UI, the manual trigger is disabled via the disable_manual_trigger flag in the pipeline yml.

Access is limited by restricting who has permissions to commit to the trigger repository.

Using a Github trigger repo allows all deployments to preprod and prod to be audited. The users that have access to run preprod and prod deployments can be managed via permissions to push to the trigger repo.

Running the shell script in the repository creates a blank file named with the current date and time. Github will show which user created the file and therefore ran the release.

The commit history in the trigger repository serves as an audit trail of all deployments to preprod and prod.

Before a service is deployed to preprod, the current version is incremented using semantic versioning. i.e. 1.0.1 becomes 1.0.2.

This happens in the release-* jobs after the preprod-trigger.

In order to ensure that the correct commit is tagged, the pipeline has to pass the service repository through the entire pipeline with passed flag set in each job. If this isn't done, then we risk tagging the wrong commit.

Java artifacts are required because Maven doesn't guarentee that the dependencies will be identical each time is is run.

For Java services, an artifact is built in the first step of the pipeline and pushed to Artifactory. This artifact is then used when deploying to both the ci and latest environments.

In the release-* job, a copy of the artifact is then created with the version number in the name, and then pushed to Artifactory. This is then used for the deployments to preprod and prod.

This is currently not demonstated in this pipeline but should be achieved by the following mechanisms:

1. Create all Cloud Foundry services (e.g. databases) as idempotent processes in the deployment job of every services which uses them.

2. Use database migrations in the services to ensure that the databases are updated to he required state.