fidelityinternational / cic Goto Github PK

Brief

Ansible like anything else is code. Code has behaviour and as a result should have tests around it if we are going to make changes to it with confidence.

An exercise teaching participants how to functionally test the Ansible they are writing is essential

Learning Objectives

• Understand the importance of automated functional testing
• Introduction to functional testing framework for infrastructure

Acceptance Criteria:

Introduction

Give a written explanation of the purpose of functional testing and how it can be used to test that our Ansible is working as intended.

Tutorial

Give an tutorial type guided walk through, demonstrating with the code snippets how to construct a test against the supplied Ansible

Exercise

Provide an exercise that gets the participant to do the following:

• extend the test produced during the Tutorial
  • One passing test
  • One failing test - That the participant should fix and in the process see in real life the value of automated tests.

It's possible that templates can be amended and submitted as part of a pull request without the corresponding Readme having been generated.

Get the CI pipeline to verify that Readme reflect the latest version of their templates.

Currently the top level README.md has this:

5. setup the courseware: ./bin/setup

But running this doesn't expose the required environment, I suspect this should be sourced with . bin/setup instead (as is documented in some of the exercises.

If that is the case, the set -e should be removed from bin/setup too as it is causing the running shell to exit when the next failed command is executed.

CIC provides a content framework to help contributors write exercises. Documentation is required in order to tell contributors how to use it.

Brief

It's envitable that usernames, passwords and secrets will be required when managing infrastructure. It's important that these are not stored in the open, easily readable by any unauthorised person who might gain access to the codebase. Ansible's Vault feature allows sensitive data to be encrypted.

Learning Objectives

• Learn how to use Vault to:
  • Create
  • Edit
  • Encrypt
  • Decrypt
  • View
• Use encrypt_string to create encrypted variables to embed in yaml
• Understand Vault Ids and Multiple Vault Passwords.

Acceptance Criteria:

Introduction

Give a written explanation of the purpose the purpose of encrypting sensitive data and using Vault.

Tutorial/Exercise

This section should give a guided exercise showing participants how to use Vault. It might be useful to provide some Ansible to which encrypted config is added to as part of the tutorial.

Verification test

Provide an automated test that will only pass when encrypted files (from vault) are provided to the supplied Ansible.

Different branches may use different versions of our docker container, however all are configured to use the 'latest' version of the course image. Make this configurable, so that the work done in one branch does not break the builds in another.

Whilst going through setting up an enviroment with @TheWonn, we found that whilst @jimconner had supplied a working docker installation, @TheWonn's user had not been added to the correct docker users group. Our setup script didn't add the user to the group because Docker was already installed.

This kind of thing is likely to happen quite a bit so I think we should have something like a checksum command that can be used to verify that a user's environment is setup correctly before they proceed.

Before getting to deep in to the guts of Ansible it's useful for candidates to familiarise themselves with their environment and perform some simple operations to get themselves going. This will also help users to validate that they have setup their environment correctly.

At this stage we don't expect participants to have to 'create' anything as such, rather just give them a easy run in to the material and ansible which will highlight the available commands that ansible uses and give a quick guide on how to work with the ansible-console.

Acceptance Criteria:

Learning Objectives

• Introduce Ansible (possibly via links to some reference documentation)
• Introduce the following commands:
  • ansible
  • ansible-playbook
  • ansible-doc
  • ansible-console

Exercise Verification test

We want participants to get used to the style of our exercises. I.e. There will be a test that they can execute that verifies that they have completed the exercise correctly. As part of the first exercise we should supply them a test.

As the number of things that need to be installed goes up, the bash script that we have is becoming less appropriate. It seems the right time to flip over to Ansible for the configuring of the host machine.

@jimconner opened #5. He found the version of ruby that he had installed on his machine was not the same version as we have been using on this project. To avoid confusion we have agreed that a particular version of ruby and bundler should be installed on the behalf of users.

Acceptance Criteria:
as part of the setup script install:

• Install RVM
• Ruby 2.4.3
• bundler

Implement cic list subcommand to display a filtered list of running containers.

As with all projects, we should be able to verify that code changes do not break the current courseware.

Acceptance Criteria:

• Pipeline should be integrated with Github so that pull requests etc... can be validated
• Should run overall project build target.

It would be useful to have a task that can be called by a contributor to create a new exercise. This will:

• create a new exercise directory in the right place
• create the necessary skeleton files
• etc...

Brief

Even simple changes to a code base can cause serious bugs. As the number of lines of code go up so do the chances of making a mistake. Chuck in multiple team members all making changes to the same code base and the chances of broken code making its way to production becomes almost a certainty.

Continuous Integration (CI) is a development practice that requires developers to integrate code into a shared repository several times a day. Each check-in is then verified by an automated build, allowing teams to detect problems early.
By integrating regularly, you can detect errors quickly, and locate them more easily. ThoughtWorks

CI is a vital discipline that we want to be able to teach to anyone whom intends to release code.

This is the first exercise in a Module on CI. The purpose of this exercise to introduce the topic and get participants to see and fix a failing build.

Learning Objectives

• Introduce topic of CI
• Introduce CI Server and its outputs

Suggested Exercise

Participants could start a pre-canned CI Server which is already running a failing set of tests. The Participant could fix the failing test and see that reflected in the CI server.

Acceptance Criteria:

The readme to the exercise should include the following:

Introduction

# Required - Give a written explanation of the purpose of the topic this exercise is based on

Required prerequisite knowledge

Having some experience of testing is useful but should not be essential.

Useful Terminology

# Optional - any key terms you intend to use in your exercise, if appropriate, could be listed here

Tutorial

# Ideal but optional - This section should provided a guided exercise showing participants how to use the subject of the exercise

Exercise

# required - Following the tutorial, provide an exercise that requires the participant use the knowledge that they have gained through the intro and tutorial

Verification test

This will be seeing the build go green having fixed and comited the failing test.

Other

# Optional - add any other requirements here

#32 made a change to the cic up and cic down commands. These commands now look for a docker-compose file under .cic directory for the current path. The scaffold used by the exercise create command needs to be updated to reflect this.

Our 2 existing exercises are now part of the ansible track.

Write preamble for them so that participants have a nice exercise intro in the cards we create for them on their project board.

Playbooks are the construct provided by Ansible for users to group their 'plays' which in combination result in the required system automation.

This exercise is required to guide participants in writing their first, very simple, playbook using ansible. A ready made set of tests should be provided so that the participant will know that they have created a successful playbook.

Suggested Exercise

Guide the user in writing a playbook that will result in Apache:

• Installed
• With a particular module enabled

Acceptance Criteria:

Introduction

Give a written explanation of the purpose of playbooks in Ansible

Tutorial

Participants should then be able to go through a guided tutorial in which they will put together their first simple playbool

Exercise

Following the tutorial, provide an exercise that requires the participant to write a new playbook from scratch.

Verification test

We want participants to get used to the style of our exercises. I.e. There will be a test that they can execute that verifies that they have completed the exercise correctly. As part of the first exercise we should supply them a test.

Other

• Make use of the dockerised ansible runner that has been implemented to support participants taking these exercises.

As the complexity of the required test infrastructure goes up for exercises such as #18. The cic start command is likely to get more complicated. Maybe even to the point where it starts to mirror the docker run command.

Some spiking has shown that the docker compose command is the likely answer to this problem.

Docker compose allows us to specify the test infrastructure and how it connects together in one place. This can then be wrapped in a very simple command that does not require the user to pass through the detail that is instead contained within the docker-compose.yml file.

I was having problems getting bundler to be happy on linux mint 18.2 with the system default ruby.
Installed rbenv and ruby 2.5.1, but found that .ruby-version is set to 2.4.3
We should improve the setup checks around ruby/bundler availability.

The exercises provided in this course can be used in different numbers/combinations to emphasise different topics.
e.g:

• Ansible
• CI
• CD
• CI/CD
• TDD

Participants need guidance as to which exercises they should complete in order to achieve their learning goals. This is where Tracks come in.

Participants could be given a command as part of the courseware that they receive that against their fork:

• creates a project
• creates issues for each of the exercises they should complete
• assigns those issues in the right order to the project.

This will provide them prioritised backlog for them to work through. They can then raise pull request against each ticket that they complete so that someone can review how they have done.

Playbooks are one of the features provided by Ansible to allow users to encapsulate their automation.

Learning objectives:

• Create new playbook from scratch
• Pitfalls

The README for #39, guides user in starting learning tracks. Writing instructions as to how to navigate to the project created by the track command would be much easier if his command returned the project url directly.

Brief

Servers quite often require identical config with the exception with a few environment specific values. An exercise is required to show participants how templates can help them to keep their config files at a manageable level.

Learning Objectives

• Ansible Templates
• Jinja2
• Pitfalls to be avoided.

Acceptance Criteria:

The readme to the exercise should include the following:

Introduction

Give a written explanation of the purpose of the topic this exercise is based on

Tutorial

This section should guided exercise showing participants how to use the subject of the exercise.

Exercise

Following the tutorial, provide an exercise that requires the participant use the knowledge that they have gained through the intro and tutorial

Verification test

If possible, supply an acceptance test that participants can execute to validate that they have completed the exercise

Brief

Inventory represents the systems that are visible to Ansible.

There are a number of different ways to provide the inventory to Ansible. Inventory has been touched on during the introductory exercise produced by #7, however deeper insight is needed in to how Inventory can be supplied to Ansible.

Learning Objectives

• Static methods of providing inventory
• Dynamic inventory
• Writing your own inventory provider
• Pros and cons of static and dynamic approaches to inventory.
• Pitfalls to avoid

Acceptance Criteria:

The readme to the exercise should include the following:

Introduction

Give a written explanation of the purpose of inventory within the context of Ansible.

Tutorial

This section should guided exercise showing participants how to use the subject of the exercise

Exercise

Following the tutorial, provide an exercise that requires the participant use the knowledge that they have gained through the intro and tutorial.

Note: Building a inventory provider will require knowledge of Python and some kind of stub to represent an external source from which inventory is going to be retrieved from.

Verification test

If possible, supply an acceptance test that participants can execute to validate that they have completed the exercise

Each exercise should be accompanied by notes for tutors/reviews.

These notes guide them on:

• What specifically to focus on when asked for help.
• What they are look out for when reviewing a participants exercise submission.

Docker is used to provide access to course tooling. Therefore it is a dependency of this project and should be installed along with the rest of the courseware.

Acceptance Criteria:

• Enhance the setup scripts to install Docker on behalf of the user.
• Local user should have privileges to use docker. I.e. be in the correct group to use Docker without the need for sudo

#61 implemented the cic up and cic down commands. These commands wrapped docker-compose and provide a much easier means for standing up test infrastructure on behalf of the participants as we will no longer require them to run such complex commands.

Update the existing exercises to use cic up and cic down instead of cic start and cic stop

Note: both #318 and #32 are using this new feature so it is worth waiting till they are finished before playing this story just incase they identify any changes to the interface of these commands etc...

Brief

Even simple changes to a code base can cause serious bugs. As the number of lines of code go up so do the chances of making a mistake. Chuck in multiple team members all making changes to the same code base and the chances of broken code making its way to production becomes almost a certainty.
CI is a vital discipline that we want to be able to teach to anyone whom intends to release code.

Having been introduced to CI in #32, an exercise is required to teach participants to configure a CI server with a build that runs the tests that their teams are writing. At this stage a single stage build pipeline is all that is required. Continuous delivery pipelines will be covered in detail in #30 and #31

Learning Objectives

• Enable participants to configure a CI server to run their tests automatically when code changes are detected.

Suggested Exercise

The exercise should not require participants to write lots of tests so consider supplying these as part of the exercise materials.

Acceptance Criteria:

The readme to the exercise should include the following:

Introduction

# Required - Give a written explanation of the purpose of the topic this exercise is based on

Required prerequisite knowledge

Having some experience of testing is useful but should not be essential.

Useful Terminology

# Optional - any key terms you intend to use in your exercise, if appropriate, could be listed here

Tutorial

# Ideal but optional - This section should provided a guided exercise showing participants how to use the subject of the exercise

Exercise

# required - Following the tutorial, provide an exercise that requires the participant use the knowledge that they have gained through the intro and tutorial

Verification test

This will be seeing the build

Brief

Roles are a good way of removing duplication from Ansible Playbooks and promoting reuse. An exercise is needed to show participants the value of Roles and how to create Roles for themselves.

Learning Objectives

• Understand the value of Roles in Ansible
• How to write an Ansible Role
• Pitfalls to watch out for.

Suggested Exercise

One idea is to give the participant a ready made playbook that is crying out for a Role(s) to be written for it and then guide them through the process of doing so.

Acceptance Criteria:

The readme to the exercise should include the following:

Introduction

Give a written explanation of the purpose of the topic this exercise is based.

Learning objectives

As stated above and any others that seem relevant.

Tutorial

This section should guided exercise showing participants how to use the subject of the exercise

Exercise

Following the tutorial, provide an exercise/extension that requires the participant use the knowledge that they have gained through the intro and tutorial.

Verification test

Supply an acceptance test that participants can execute to validate that they have completed the exercise. In this case it could be a test that passed prior to the extracting Roles from the Playbook that should continue to pass once the roles have been extracted. An additional test scenario should be supplied for any extension exercises.

Since merging #16 the setup script now very slowly. This is not an issue when running it for the first because it is installing dependencies that are required by the courseware.

However, users are required to call bin/setup subsequently when they return to the project in a new terminal to setup the PATH variable. In this use case the time it take to source the script is to long (25 seconds)

I think it is time to split out the one time prereqs portion of the setup script.

#39 introduced a template for the participant readme. Right now there isn't a task that is building this automatically. Therefore making updates to the generated README.md requires 'special' knowledge. I.e. you have to run exercise generate yourself.

My ancient version of docker doesn't support the container subcommand:

[dev] a587850@ukx02380 ~/dev/ci-cd-training/exercises/IaC/ansible/inventory $ docker --version
Docker version 1.12.6, build c4618fb/1.12.6

[dev] a587850@ukx02380 ~/dev/ci-cd-training/exercises/IaC/ansible/inventory $ docker container
docker: 'container' is not a docker command.
See 'docker --help'.

Are we enforcing a minimum version of docker/docker-compose?

Note: Remove unrequired sections and any comments that are in this template before submitting

Brief

# required - A description of the topic and the need for an exercise

Learning Objectives

# required - the learning objectives for participants that take this exercise

• Pitfalls to be avoided.

Suggested Exercise

# Optional - If clear direction on the content of the exercise is required then put it in here.

Acceptance Criteria:

The readme to the exercise should include the following:

Introduction

# Required - Give a written explanation of the purpose of the topic this exercise is based on

Tutorial

# Ideal but optional - This section should provided a guided exercise showing participants how to use the subject of the exercise

Exercise

# required - Following the tutorial, provide an exercise that requires the participant use the knowledge that they have gained through the intro and tutorial

Verification test

# Ideal but optional - Supply an acceptance test that participants can execute to validate that they have completed the exercise

Other

# Optional - add any other requirements here

Brief

Ansible uses variables to enable more flexibility in playbooks and roles. They can be used to loop through a set of given vales, access various information like the hostname of a system and replace certain strings in templates by system-specific values.

We need to explain the different ways that variables can by used through:

• the inventory
• by variable files
• on the command line
• retrieved from the vault

Suggested Exercise

Variables in loops
Variables in templates
Variables in conditions (when)
Getting variables from the system (facts)
Getting variables from the command line
Setting variables in playbooks
Setting variables in the inventory
Setting variables on a system (local facts)
Results of tasks (registered variables)
Accessing variables of other hosts

Acceptance Criteria:

Introduction

Give an explanation of the different ways in which variables can be used (see list above in Suggested Exercise)

Tutorial

Variable types:

• k/v pairs
• dictionaries

Exercise(s)

Verification test

Define a test that confirms that the participant has correctly used variables in the exercise.

Other

Good doc here: https://liquidat.wordpress.com/2016/01/26/howto-introduction-to-ansible-variables/

It would be good to have a command that participants can run to identify the learning tracks that are available to them.

This has come to light whilst writing the README as part of #39

Note: Remove unrequired sections and any comments that are in this template before submitting

Brief

# required - A description of the topic and the need for an exercise

Learning Objectives

# required - the learning objectives for participants that take this exercise

• Pitfalls to be avoided.

Suggested Exercise

# Optional - If clear direction on the content of the exercise is required then put it in here.

Acceptance Criteria:

The readme to the exercise should include the following:

Introduction

# Required - Give a written explanation of the purpose of the topic this exercise is based on

Tutorial

# Ideal but optional - This section should provided a guided exercise showing participants how to use the subject of the exercise

Exercise

# required - Following the tutorial, provide an exercise that requires the participant use the knowledge that they have gained through the intro and tutorial

Verification test

# Ideal but optional - Supply an acceptance test that participants can execute to validate that they have completed the exercise

Other

# Optional - add any other requirements here

The exercise readme's have a note in them telling participants to make sure that they have changed directory in their prompt to the path of the current exercise. The note also includes the path that of the exercise itself.

Whenever the directory containing the exercise changes name, or location, it invalidates the literal path that has been put in to the readme. A helper method for inserting the exercise path would be very easy to write and very useful.

Note: Remove unrequired sections and any comments that are in this template before submitting

Brief

# required - A description of the topic and the need for an exercise

Learning Objectives

# required - the learning objectives for participants that take this exercise

• Pitfalls to be avoided.

Suggested Exercise

# Optional - If clear direction on the content of the exercise is required then put it in here.

Acceptance Criteria:

The readme to the exercise should include the following:

Introduction

# Required - Give a written explanation of the purpose of the topic this exercise is based on

Tutorial

# Ideal but optional - This section should provided a guided exercise showing participants how to use the subject of the exercise

Exercise

# required - Following the tutorial, provide an exercise that requires the participant use the knowledge that they have gained through the intro and tutorial

Verification test

# Ideal but optional - Supply an acceptance test that participants can execute to validate that they have completed the exercise

Other

# Optional - add any other requirements here

The top level readme is yet to be defined.

The readme should:

• give participants crystal clear instructions for getting
• instructions as to how to operate the courseware
• Suggested best practices for going through material and getting the most from the exercises
• etc...

Brief

Continuous integration vital for helping to ensuring the quality of code. LINTing is another way of helping to do this.

Learning Objectives

• Introduce the value of LINTing
• Introducing LINTing in to CI pipelines

Exercise Suggestions

• The exercise could supply code that fails LINTIng checks. On setting up LINTing participants could fix the reported issues.
• Ask participant to implement code coverage to existing pipeline and add missing coverage that is identified.

Acceptance Criteria:

The readme to the exercise should include the following:

Introduction

# Required - Give a written explanation of the purpose of the topic this exercise is based on

Required prerequisite knowledge

Having some experience of testing is useful but should not be essential.

Useful Terminology

# Optional - any key terms you intend to use in your exercise, if appropriate, could be listed here

Tutorial

# Ideal but optional - This section should provided a guided exercise showing participants how to use the subject of the exercise

Exercise

# required - Following the tutorial, provide an exercise that requires the participant use the knowledge that they have gained through the intro and tutorial

Verification test

This will be seeing the build go "green"

Brief

Pipelines in concourse are written in YAML. Like in any medium, duplication makes code maintenance difficult.

Concourse provides a number of abstractions to allow users to refactor their pipelines and keep them lean.

An exercise is required to introduce some of the constructs that concourse provides.

Learning Objectives

Introduce concourse constructs such as:

• Tasks
• Intermediary artifacts to promote reuse. Eg. docker containers?
• Useful resource types
• Add others that you consider relevant.

Exercise Suggestions

It might be a good idea to provide a concourse pipeline that exhibits various code smells that participants can then fix.

Acceptance Criteria:

The readme to the exercise should include the following:

Introduction

# Required - Give a written explanation of the purpose of the topic this exercise is based on

Required prerequisite knowledge

Having some experience of testing is useful but should not be essential.

Useful Terminology

# Optional - any key terms you intend to use in your exercise, if appropriate, could be listed here

Tutorial

# Ideal but optional - This section should provided a guided exercise showing participants how to use the subject of the exercise

Exercise

# required - Following the tutorial, provide an exercise that requires the participant use the knowledge that they have gained through the intro and tutorial

Verification test

provide some means of verifying that participants have completed the exercise correctly. E.g. someway that verifies that pipeline deployment/outputs continue to be produced

Brief

Ansible provides a 'handlers' feature which can be used to notify Ansible to trigger certain tasks when a particular event happens. For example to restart a service when its config has changed.

An exercise is needed to teach participants how to manage services with the use of handlers.

Learning Objectives

• Handlers
• Pitfalls to be avoided.
  • Name clash
  • Order dependant

Suggested Exercise

In #11 participants wrote a playbook to install a service. This exercise could act as a follow on from that exercise. I.e. introduce handlers to manage the lifecycle of that service. In any case this exercise should not require a participant to have complete the exercise set out in #11 and so should supply the required Ansible as a starting point.

Acceptance Criteria:

The readme to the exercise should include the following:

Introduction

Give a written explanation of the purpose of the topic this exercise is based on.

Tutorial

This section should provide a guided exercise showing participants how to use the subject of the exercise.

Exercise

Following the tutorial, provide an exercise that requires the participant use the knowledge that they have gained through the intro and tutorial

Verification test

Supply an acceptance test that participants can execute to validate that they have completed the exercise

When things like CIC command change the course docker image should be rebuilt.

Note: This could also be mean regenerating the exercises if the output of the cic command changes...

Brief

There comes a time where logic and looping become useful. Ansible provides conditional and looping constructs to cater for this.

Learning Objectives

Learn about the following constructs within Ansible:

• when conditional statement
• loop statement

Acceptance Criteria:

Introduction

Give a written explanation of the purpose of the topic this exercise is based on

Tutorial

Participants should be given a guided exercise showing participants how to use the subject of the exercise. If convenient, this tutorial should be backed by ansible that should be supplied along with this exercise.

Exercise

Following the tutorial, provide an exercise that requires the participant use the knowledge that they have gained through the intro and tutorial.

Verification test

Supply an acceptance test that participants can execute to validate that they have completed the exercise

Brief

Getting to code to production quickly on demand requires discipline. Automated Continuous Delivery Pipelines (CDPs) are the way to do this.

There are a number of concerns a CDP can address:

• CI
• Integration
• Performance
• Auditing
• Approval

Learning Objectives

• Introduce the concept of fully automated continuous delivery
• Enable participants to configure multi stage pipelines within a CI server
• Introduce techniques such as tagging to mark successful releases/candidates.

Acceptance Criteria:

The readme to the exercise should include the following:

Introduction

# Required - Give a written explanation of the purpose of the topic this exercise is based on

Required prerequisite knowledge

Having some experience of testing is useful but should not be essential.

Useful Terminology

# Optional - any key terms you intend to use in your exercise, if appropriate, could be listed here

Tutorial

# Ideal but optional - This section should provided a guided exercise showing participants how to use the subject of the exercise

Exercise

# required - Following the tutorial, provide an exercise that requires the participant use the knowledge that they have gained through the intro and tutorial

Verification test

provide some means of verifying that participants have completed the exercise correctly

Code coverage tooling should be run all all framework code where possible.

Acceptance Criteria:

• Should run as part of overall build target.