Git Product home page Git Product logo

docker-ci's Introduction

Build Status codecov PS Gallery

docker-ci

PowerShell Core module to build and test Docker images. The module comes with CmdLets to perform the most commonly used tasks with regards to building docker images:

  • Build
  • Tests
  • Lint
  • Login
  • Pull
  • Push
  • Tag
  • Inspect (not implemented yet)

For each tasks there is a corresponding CmdLet:

  • Invoke-DockerBuild
  • Invoke-DockerTests
  • Invoke-DockerLint
  • Invoke-DockerLogin
  • Invoke-DockerPull
  • Invoke-DockerPush
  • Invoke-DockerTag

To run, just do

PS C:\docker> Invoke-DockerBuild .

and so on.

Prerequisite

  • PowerShell core >= 6.0 is needed.
  • Docker Engine >= 20.10.0 or Docker Desktop >= 3.0.0 is needed

Installation

PS C:\docker> Install-Module Docker-ci -Repository PSGallery

Examples

In the following section we'll cover how to use the module to add testing and linting to an existing Dockerfile. I am assuming you repository looks like this at the moment:

PS C:\docker> Get-Content .\Dockerfile
FROM ubuntu:18.04

SHELL ["/bin/bash", "-o", "pipefail", "-c"]

RUN apt-get update \
    && apt-get install -y --no-install-recommends curl=7.* ca-certificates=* \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*

RUN update-ca-certificates

RUN curl -sL https://get.docker.com/ | sh

RUN curl -LO https://storage.googleapis.com/container-structure-test/v1.8.0/container-structure-test-linux-amd64 \
    && chmod +x container-structure-test-linux-amd64 \
    && mv container-structure-test-linux-amd64 /usr/local/bin/container-structure-test

VOLUME /configs

ENTRYPOINT [ "/usr/local/bin/container-structure-test" ]
Listing 1: Example Dockerfile

Building an image from Dockerfile

To build an image based on a Dockerfile, use the Invoke-DockerBuild CmdLet, like so:

PS C:\docker> Invoke-DockerBuild . -ImageName structure
Sending build context to Docker daemon  4.608kB

Step 1/8 : FROM ubuntu:18.04
 ---> 775349758637
Step 2/8 : SHELL ["/bin/bash", "-o", "pipefail", "-c"]
 ---> Using cache
 ---> b3d8c49615a7
Step 3/8 : RUN apt-get update     && apt-get install -y --no-install-recommends curl=7.* ca-certificates=*     && apt-get clean     && rm -rf /var/lib/apt/lists/*
 ---> Using cache
 ---> 9470f9c9ecea
Step 4/8 : RUN update-ca-certificates
 ---> Using cache
 ---> 80853c222946
Step 5/8 : RUN curl -sL https://get.docker.com/ | sh
 ---> Using cache
 ---> af17b9b8fb1b
Step 6/8 : RUN curl -LO https://storage.googleapis.com/container-structure-test/v1.8.0/container-structure-test-linux-amd64     && chmod +x container-structure-test-linux-amd64     && mv container-structure-test-linux-amd64 /usr/local/bin/container-structure-test
 ---> Using cache
 ---> bea1dca8e10e
Step 7/8 : VOLUME /configs
 ---> Using cache
 ---> 97e90bf8481b
Step 8/8 : ENTRYPOINT [ "/usr/local/bin/container-structure-test" ]
 ---> Using cache
 ---> 6b9746ab76d8
Successfully built 6b9746ab76d8
Successfully tagged structure:latest
SECURITY WARNING: You are building a Docker image from Windows against a non-Windows Docker host. All files and directories added to build context will have '-rwxr-xr-x' permissions. It is recommended to double check and reset permissions for sensitive files and directories.

Dockerfile    : Dockerfile
ImageName     : structure
Registry      :
Tag           : latest
CommandResult : CommandResult
Listing 2: Output from building a docker image

In this scenario, you will see the both the output from Docker and the result of the execution which is a PSCustomObject that holds:

  • The path to the Dockerfile being used as the basis for the image.
  • The name of the image being produced.
  • The registry (if unset defaults to Docker's default registry)
  • The command result object which has more detailed information about the execution.

In most cases you will want to store the result in a variable for further processing or output to a CI/CD pipeline, like so:

PS C:\docker> Invoke-DockerBuild . -ImageName structure
Sending build context to Docker daemon  4.608kB

Step 1/8 : FROM ubuntu:18.04
 ---> 775349758637
Step 2/8 : SHELL ["/bin/bash", "-o", "pipefail", "-c"]
 ---> Using cache
 ---> b3d8c49615a7
Step 3/8 : RUN apt-get update     && apt-get install -y --no-install-recommends curl=7.* ca-certificates=*     && apt-get clean     && rm -rf /var/lib/apt/lists/*
 ---> Using cache
 ---> 9470f9c9ecea
Step 4/8 : RUN update-ca-certificates
 ---> Using cache
 ---> 80853c222946
Step 5/8 : RUN curl -sL https://get.docker.com/ | sh
 ---> Using cache
 ---> af17b9b8fb1b
Step 6/8 : RUN curl -LO https://storage.googleapis.com/container-structure-test/v1.8.0/container-structure-test-linux-amd64     && chmod +x container-structure-test-linux-amd64     && mv container-structure-test-linux-amd64 /usr/local/bin/container-structure-test
 ---> Using cache
 ---> bea1dca8e10e
Step 7/8 : VOLUME /configs
 ---> Using cache
 ---> 97e90bf8481b
Step 8/8 : ENTRYPOINT [ "/usr/local/bin/container-structure-test" ]
 ---> Using cache
 ---> 6b9746ab76d8
Successfully built 6b9746ab76d8
Successfully tagged structure:latest
SECURITY WARNING: You are building a Docker image from Windows against a non-Windows Docker host. All files and directories added to build context will have '-rwxr-xr-x' permissions. It is recommended to double check and reset permissions for sensitive files and directories.
Listing 3: Build docker image and store result in variable.

in which case you will only see the output from Docker, the result object is stored in $result.

You can verify the existence of the image you just created using docker images

PS C:\docker> docker images

If you want less output, use -Quiet switch to output only the final result of the command. Combined with storing the result in a variable, this will give a completely silent execution of the CmdLet.

Disabling verbose output

The -Quiet setting for CmdLets that support it, defaults to the value of the enviromenment variable DOCKER_CI_QUIET_MODE. So you can set this environment variable to the desired setting for the -Quiet switch so you don't have to set it for each invocation of a cmdlet that supports it.

Linting a Dockerfile

An important aspect of writing quality Docker images is to try and learn from the best in the community. To this end, we provide a convenient way to run hadolint against a Dockerfile. Hadolint is a 3rd party component that scans a dockerfile and produces linted output. You can find the hadolint project here: https://github.com/hadolint/hadolint

Here's how to use the linter via a CmdLet:

PS C:\docker> $result = Invoke-DockerLint .\Dockerfile
1: FROM ubuntu:18.04
2:
3: SHELL ["/bin/bash", "-o", "pipefail", "-c"]
4:
5: RUN apt-get update \
6:     && apt-get install -y --no-install-recommends curl=7.* ca-certificates=* \
7:     && apt-get clean \
8:     && rm -rf /var/lib/apt/lists/*
9:
10: RUN update-ca-certificates
11:
12: RUN curl -sL https://get.docker.com/ | sh
13:
14: RUN curl -LO https://storage.googleapis.com/container-structure-test/v1.8.0/container-structure-test-linux-amd64 \
15:     && chmod +x container-structure-test-linux-amd64 \
16:     && mv container-structure-test-linux-amd64 /usr/local/bin/container-structure-test
17:
18: VOLUME /configs
19:
20: ENTRYPOINT [ "/usr/local/bin/container-structure-test" ]
Listing 4: Build docker image and store result in variable.

This Dockerfile in particular has no linting remarks, so it is just output in its entirety with line numbers. Imagine I omitted the instruction in line 3 on how to deal with commands that fail in a piped execution and run the linting again:

PS C:\docker> $result = Invoke-DockerLint .\Dockerfile
1: FROM ubuntu:18.04
2:
3: RUN apt-get update \
4:     && apt-get install -y --no-install-recommends curl=7.* ca-certificates=* \
5:     && apt-get clean \
6:     && rm -rf /var/lib/apt/lists/*
7:
8: RUN update-ca-certificates
9:
DL4006 Set the SHELL option -o pipefail before RUN with a pipe in it
10: RUN curl -sL https://get.docker.com/ | sh
11:
12: RUN curl -LO https://storage.googleapis.com/container-structure-test/v1.8.0/container-structure-test-linux-amd64 \
13:     && chmod +x container-structure-test-linux-amd64 \
14:     && mv container-structure-test-linux-amd64 /usr/local/bin/container-structure-test
15:
16: VOLUME /configs
17:
18: ENTRYPOINT [ "/usr/local/bin/container-structure-test" ]
Listing 5: Lint docker file and store result in variable

Now, the linter is no longer happy and it has added a remark just above line 10 instructing us on how to fix the problem. The first part of the message is a unique lint rule id that can be used to find the lint rationale and in-depth explanation on https://github.com/hadolint/hadolint.

Failing the build when there are lint remarks

In CI/CD contexts, you might want to fail a build or similar, if there are linting errors. Raising an exception when there are lint remarks found is straigt-forward:

Invoke-DockerLint -TreatLintRemarksFoundAsException

Going from the example Dockerfile in listing 5, we achieve an error:

PS C:\docker> $result = Invoke-DockerLint -TreatLintRemarksFoundAsException
Assert-ExitCodeOk : The command 'Get-Content "C:\docker\Dockerfile" | docker run -i hadolint/hadolint:v1.17.2' failed with exit code: 1.
Command output:
/dev/stdin:10 DL4006 Set the SHELL option -o pipefail before RUN with a pipe in it
At C:\Users\Rasmus Jelsgaard\Documents\PowerShell\Modules\Docker-CI\0.4.8\Public\Invoke-DockerLint.ps1:31 char:9
+         Assert-ExitCodeOk $commandResult
+         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo          : InvalidResult: (:) [Assert-ExitCodeOk], Exception
+ FullyQualifiedErrorId : 1000,Assert-ExitCodeOk

PS C:\docker> $LASTEXITCODE
1
Listing 6: Fail execution if there are lint remarks.

If you would rather only fail once a certain threshold is met, you can do something like this to that effect (again based on the Dockerfile in listing 5):

PS C:\docker> $result = Invoke-DockerLint -Quiet
PS C:\docker> $result.LintRemarks.Length -gt 0
True

This concludes the examples on linting. Whilst linting can help you improve parts of your Docker-style and quality of the images, it is no substitute for real testing.

Testing a docker image

We provide testing of docker images using Google's Container Structure framework (https://github.com/GoogleContainerTools/container-structure-test).

To run tests, you first define them in .yml configs. Then you build the image the image you want to test, and then finally you run the tests.

So let's start by building an image called structure

PS C:\docker> Invoke-DockerBuild . -Quiet -ImageName structure

Dockerfile    : Dockerfile
ImageName     : structure
Registry      :
Tag           : latest
CommandResult : CommandResult

Then, we define a test to check if the correct binary was produced and placed in accordance with the documentation at https://github.com/GoogleContainerTools/container-structure-test.

The config file gcs-commands.yml looks like this:

schemaVersion: "2.0.0"

commandTests:
  - name: "say hello world"
    command: "bash"
    args:
      - -c
      - |
        echo hello &&
        echo world
    exitCode: 0
    expectedOutput: ["hello", "world"]

And you run the tests like this:

PS C:\docker> $result = Invoke-DockerTests -ImageName 3shape/containerized-structure-test -ConfigFiles gcs-commands.yml
@{Pass=1; Fail=0; Total=1; Results=System.Object[]}
PS C:\docker> $result

TestResult                                          TestReportPath            CommandResult ImageName
----------                                          --------------            ------------- ---------
@{Pass=1; Fail=0; Total=1; Results=System.Object[]} C:\docker\testreport.json CommandResult 3shape/containerized-structure-test

This concludes the section with examples. Let us know if there is something missing, that is not clear from the documentation.

Development environment setup

  • Install PowerShell Core 6.x or newer.
  • Run .\Install-Prerequisites.ps1

docker-ci's People

Contributors

jetersen avatar lee-at-work avatar mindaugaslaganeckas avatar rasmusjelsgaard avatar romankarnaukh avatar viktor-kaydalov avatar zionyx avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

docker-ci's Issues

Docs: CI/CD pipeline examples

Improve the documentation by adding examples that show how the module can be easy to make work in a pipeline simple and easy.

The CI/CD solution chosen is not important, the optimal solution would be to even have a couple of examples so it's easy to see how it is done using different technologies and show the idioms of this module (pipelining and so on).

Auto-label in Invoke-DockerBuild

The Invoke-DockerBuild cmdlet adds relevant labels to the images produced automatically on a best-effort basis. This is on top of whatever labels are added already in the Dockerfile.

The primary use case for this feature is from within CI/CD where the labels can be deduced.

It should be possible to manually override the labels and to add custom labels. The ability to add labels from a label file as well would be very handy in a CI/CD context.

https://github.com/opencontainers/image-spec/blob/master/annotations.md#pre-defined-annotation-keys provides a reasonable starting point. We should be able to deduce much of the information needed to provide these labels automatically.

Improve usability of cmdlets by eliminating redundant parameters

In the world of docker, the image name comprises of the full text that includes these elements:

  1. registry (optional)
  2. image name
  3. tag / digest

As <registry>/<image name>/<tag / digest>.
Eg: docker.io/library/ubuntu:latest or just ubuntu:latest

But the parameters the cmdlets ask for, are 3 separate parameters as stated above.

To improve usability, I think we should let the users formulate the complete image names themselves, than to have this logic added in the cmdlets.

This will eliminate the need for 3 parameters, and streamline them into 1, similar to docker cli.

In docker tag scenario, it doubles the number of parameters.

Current scenario:

Invoke-DockerTag -Registry localregistry -ImageName myubuntu -Tag latest -NewRegistry localregistry -NewImageName myubuntu -NewTag rc1

Proposed scenario:

Invoke-DockerTag -ImageName localregistry/myubuntu:latest -NewImageName localregistry/myubuntu:rc1

In short, merge these parameters below into just -ImageName:

  • -Registry
  • -Tag

Additional Prerequisite to call out: Docker installed.

It may see obvious, but I'm hoping to find a tool to allow me to pull, tag, and push images without having the docker daemon running locally. I found (understandably) that this module requires docker to be installed and running. May want to add that to README.

Invoke-DockerTag does not log anything

When running Invoke-DockerTag no output is given.

Invoke-DockerTag -Registry artifactorydk.3shape.local/threeshapedocker-features -ImageName threeshape.dotnet.framework.runtime -Tag 4.8-servercore-amd64-72cff51b -NewRegistry artifactorydk.3shape.local/threeshapedocker -NewImageName threeshape.dotnet.framework.runtime -NewTag 4.8-servercore-amd64

Cross-product support for PS Core 6 and newer major versions

Currently we "just" test on different platforms using the same version of PowerShell core.

It would be great if we could add checks for backwards compatibility of the major version of pscore. At the time of writing it would mean testing on PSCore 6 and 7.

I suggest starting by reading through this to get started and learn the current status of azure pipelines in terms of supporting this.

Clean up publish pollution

Our Travis CI job includes way too much in the packages we publish to PSGallery.

Investigate why this is happening and fix it.

Integration tests for all public functions

Some errors can slip through the cracks of the current tests because there is no integration testing of individual functions.

We should make sure there is at least one extensive integration test for each public function.
This probably requires refactoring the code in the Integration.Tests.ps1 file and extract the setup code there into a new function in the Docker-CI.Tests.psm1 module. This code would then be a common setup used to setup and teardown a local docker registry for use in all the integration tests.

Add generic paramter for `Invoke-DockerBuild` to support all other possibilities of docker build

I don't think we should keep expanding the parameters for Invoke-DockerBuild to cover all possibilities supported by docker build command.

The combinations are close to endless. See https://docs.docker.com/v17.09/engine/reference/commandline/build/#options

Alternatively, provide a string parameter to append the string to the end of docker build command.

This way allows us to quickly use the parameter for:

  • --memory → memory limit to boost performance when building
  • --label → Set metadata for an image
  • --platform → Set platform if server is multi-platform capable (very nice feature to have)
  • --no-cache → Sometimes useful to rebuild from scratch
  • --build-arg → Set build-time variables

Quiet is not

Not all commands seem to respect the Quiet switch.
E.g. Invoke-DockerPull seems not to pass the Quiet switch to the Invoke-DockerCommand.
Also the docker process output is written using Write-Information -InformationAction 'Continue', which means, that the caller can't properly control the information stream. E.g. using Invoke-DockerPull -InformationAction Ignore will not work as expected (the information output will still be produced).
This means, that the only way of not having the Invoke-DockerCommand produce any output to the information channel, is by using the DOCKER_CI_QUIET_MODE environment variable, which is not very convenient, when you're writing a more complex automation/script where some output should be produced and some not.

Caching of docker os type info

Implement caching of the results of calls to Find-DockerOsType. It's an expensive call we use in multiple places in the codebase.

Successful login spits `System.Management.Automation.RemoteException`

[2019-11-13T13:49:57.735Z] WARNING! Your password will be stored unencrypted in /home/jenkins/.docker/config.json.
[2019-11-13T13:49:57.735Z] Configure a credential helper to remove this warning. See
[2019-11-13T13:49:57.735Z] https://docs.docker.com/engine/reference/commandline/login/#credentials-store
[2019-11-13T13:49:57.735Z] System.Management.Automation.RemoteException
[2019-11-13T13:49:57.735Z] Login Succeeded

Need to fix this exception even if login succeeds.

Add publish test to pipeline

so that we can test, that we can:

Grab the latest (just published) version of the module. Install it. Then verify that it exposes the expected CmdLets.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.