Git Product home page Git Product logo

gocd-json-config-plugin's Introduction

Join the chat at https://gitter.im/gocd/gocd

GoCD JSON configuration plugin

Build Status

GoCD pipeline configuration as code

This is a GoCD server plugin which allows to keep pipelines and environments configuration in version control systems supported by GoCD (git, svn, mercurial, etc.). See this document to find out what GoCD's configuration repositories are.

Resources:

Quickstart

Make sure you are running at least 16.8.0 GoCD server.

Installation

If you're using a GoCD version older than 17.8.0, you need to install the plugin in the GoCD server.

You'll have to drop .jar to plugins/external directory in your server installation. Plugin jars can be downloaded from releases page.

Add configuration repository

There is no UI to add configuration repositories so you'll have to edit the config XML. You will need to add config-repo section within config-repos. If config-repos does not exist yet then you add it above first <pipelines /> and <artifactStores/>.

To add all configurations from git repository https://github.com/tomzo/gocd-json-config-example.git a section like this should be added:

...
<config-repos>
   <config-repo pluginId="json.config.plugin" id="repo1">
     <git url="https://github.com/tomzo/gocd-json-config-example.git" />
   </config-repo>
</config-repos>
...
<artifactStores />
<pipelines />
...

Configuration files

Using this plugin you can store any number of pipeline or environment configurations without a versioned repository like git.

By default pipelines should be stored in *.gopipeline.json files and environments should be stored in *.goenvironment.json files.

The file name pattern can be changed on plugin configuration page.

Format

The pipeline configuration files should be stored in format similar to one exposed by GoCD API.

The format of environment configuration files is much simpler, you can find examples of correct environments at the bottom.

Format version

Please note that it is now recommended to declare format_version in each *.gopipeline.json or *.goenvironment.json file. Version 3 will be introduced in GoCD 18.7.0 Currently it is recommended to declare consistent version in all your files:

{
  "format_version" : 3
}

Syntax checking

Since 0.3.1 plugin is an executable and supports basic syntax checking.

Usage with java installed

You need to download the jar from releases page and place it somewhere convenient. For example /usr/lib/gocd-json-plugin/json-config-plugin.jar. Then to validate your gopipeline.json file run something like:

java -jar /usr/lib/gocd-json-plugin/json-config-plugin.jar syntax mypipe.gopipeline.json

Usage with IDE and docker

IDE is a bash script, a cli wrapper around docker to help with running development tasks in docker. You can install the ide script so that it is available on the PATH with:

sudo bash -c "`curl -L https://raw.githubusercontent.com/ai-traders/ide/master/install.sh`"

Add Idefile in your project with following content

IDE_DOCKER_IMAGE=tomzo/gocd-json-ide:0.3.1

To validate files run:

ide gocd-json syntax mypipe.gopipeline.json

Personally, I recommend the following project structure:

  • gocd/ directory for all your GoCD configuration files.
  • gocd/Idefile file pointing which docker image can be used to validate configuration.

Then when working with gocd pipelines config, you can run from the root of your project

cd gocd
ide      # will open interactive shell
watch gocd-json syntax mypipe.gopipeline.json

Usage with docker only

docker run -ti --rm --volume $(pwd):/ide/work tomzo/gocd-json-ide:0.3.1 bash

Then you have an interactive shell as above.

Implementation note

This plugin leverages JSON message format used internally for GoCD server and plugin communication.

GoCD pipeline and environment configuration has very deep structure. So instead of reading a very long schema, below you can find examples of all configuration elements.

It is exactly like documented here

It is close to official xml schema and also official JSONs in pipeline configuration API

JSON Configuration objects

  1. Environment
  2. Environment variables
  3. Pipeline
  4. Stage
  5. Job
  6. Tasks
  7. Materials

Environment

Configures a GoCD environment

{
  "name": "dev",
  "environment_variables": [
    {
      "name": "key1",
      "value": "value1"
    }
  ],
  "agents": [
    "123"
  ],
  "pipelines": [
    "mypipeline1"
  ]
}

Environment variables

Environment variables is a JSON array that can be declared in Environments, Pipelines, Stages and Jobs.

Any variable must contain name and value or encrypted_value.

{
  "environment_variables": [
    {
      "name": "key1",
      "value": "value1"
    },
    {
      "name": "keyd",
      "encrypted_value": "v12@SDfwez"
    }
  ]
}

Pipeline

{
  "format_version" : 1,
  "group": "group1",
  "name": "pipe2",
  "label_template": "foo-1.0-${COUNT}",
  "enable_pipeline_locking" : false,
  "parameters": [
    {
      "name": "param",
      "value": "parameter"
    }
  ],
  "mingle": {
    "base_url": "http://mingle.example.com",
    "project_identifier": "my_project"
  },
  "tracking_tool": null,
  "timer": {
    "spec": "0 15 10 * * ? *"
  },
  "environment_variables": [],
  "materials": [
    ...
  ],
  "stages": [
    ...
  ]
}

Referencing an existing template in a config repo:

{
  "format_version" : 1,
  "group": "group1",
  "name": "pipe-with-template",
  "label_template": "foo-1.0-${COUNT}",
  "enable_pipeline_locking" : false,
  "template": "template1",
  "parameters": [
    {
      "name": "param",
      "value": "parameter"
    }
  ],
  "materials": [
    ...
  ]
}

Please note:

  • Pipeline declares a group to which it belongs

Pipeline locking

Expected since GoCD v17.12, you need to use lock_behavior rather than enable_pipeline_locking.

"lock_behavior" : "none"

lock_behavior can be one of:

  • lockOnFailure - same as enable_pipeline_locking: true
  • unlockWhenFinished -
  • none - same enable_pipeline_locking: false

Mingle

{
    "base_url": "https://mingle.example.com",
    "project_identifier": "foobar_widgets",
    "mql_grouping_conditions": "status > 'In Dev'"
}

Tracking tool

{
  "link": "http://your-trackingtool/yourproject/${ID}",
  "regex": "evo-(\\d+)"
}

Timer

{
    "spec": "0 0 22 ? * MON-FRI",
    "only_on_changes": true
}

Stage

{
  "name": "test",
  "fetch_materials": true,
  "never_cleanup_artifacts": false,
  "clean_working_directory": false,
  "approval" : null,
  "environment_variables": [
    {
      "name": "TEST_NUM",
      "value": "1"
    }
  ],
  "jobs": [
    ...
  ]
}

Approval

{
  "type": "manual",
  "users": [],
  "roles": [
    "manager"
  ]
}

Job

{
  "name": "test",
  "run_instance_count" : null,
  "environment_variables": [],
  "timeout": 180,
  "elastic_profile_id": "docker-big-image-1",
  "tabs": [
     {
       "name": "test",
       "path": "results.xml"
     }
   ],
   "resources": [
    "linux"
  ],
  "artifacts": [
    {
      "source": "src",
      "destination": "dest",
      "type": "test"
    },
    {
      "type": "external",
      "id": "docker-release-candidate",
      "store_id": "dockerhub",
      "configuration": [
        {
          "key": "Image",
          "value": "gocd/gocd-demo"
        },
        {
          "key": "Tag",
          "value": "${GO_PIPELINE_COUNTER}"
        },
        {
          "key": "some_secure_property",
          "encrypted_value": "!@ESsdD323#sdu"
        }
      ]
    }
  ],
  "properties": [
    {
      "name": "perf",
      "source": "test.xml",
      "xpath": "substring-before(//report/data/all/coverage[starts-with(@type,\u0027class\u0027)]/@value, \u0027%\u0027)"
    }
  ],
  "tasks": [
    ...
  ]
}

Artifacts

There are 3 types of artifacts recognized by GoCD. Build and Test artifacts are stored on the GoCD server. The source and the destination of the artifact that should be stored on the GoCD server must be specified.

Build

{
  "source": "src",
  "destination": "dest",
  "type": "build"
}

Test

{
  "source": "src",
  "destination": "dest",
  "type": "test"
}

External

Artifacts of type external are stored in an artifact store outside of GoCD. The external artifact store's configuration must be created in the main GoCD config. Support for external artifact store config to be checked in as yaml is not available. The external artifact store is referenced by the store_id. The build specific artifact details that the artifact plugin needs to publish the artifact is provided as configuration.

{
  "type": "external",
  "id": "docker-release-candidate",
  "store_id": "dockerhub",
  "configuration": [
    {
      "key": "Image",
      "value": "gocd/gocd-demo"
    },
    {
      "key": "Tag",
      "value": "${GO_PIPELINE_COUNTER}"
    },
    {
      "key": "some_secure_property",
      "encrypted_value": "!@ESsdD323#sdu"
    }
  ]
}

Property

{
  "name": "coverage.class",
  "source": "target/emma/coverage.xml",
  "xpath": "substring-before(//report/data/all/coverage[starts-with(@type,'class')]/@value, '%')"
}

Tab

{
      "name": "cobertura",
      "path": "target/site/cobertura/index.html"
}

Run many instances

Part of job object can be number of job to runs

"run_instance_count" : 6

Or to run on all agents

"run_instance_count" : "all"

Default is null which runs just one job.

Materials

All materials:

  • must have type - git, svn, hg, p4, tfs, dependency, package, plugin.
  • can have name and must have name when there is more than one material in pipeline

SCM-related materials have destination field.

Filter - blacklist and whitelist

All scm materials can have filter object:

  • for blacklisting:
"filter": {
  "ignore": [
    "externals",
    "tools"
  ]
}
  • for whitelisting (since Go >=16.7.0):
"filter": {
 "whitelist": [
   "moduleA"
 ]
}

Git

{
  "url": "http://my.git.repository.com",
  "branch": "feature12",
  "filter": {
    "ignore": [
      "externals",
      "tools"
    ]
  },
  "destination": "dir1",
  "auto_update": false,
  "name": "gitMaterial1",
  "type": "git",
  "shallow_clone": true
}

Svn

{
  "url": "http://svn",
  "username": "user1",
  "password": "pass1",
  "check_externals": true,
  "filter": {
    "ignore": [
      "tools",
      "lib"
    ]
  },
  "destination": "destDir1",
  "auto_update": false,
  "name": "svnMaterial1",
  "type": "svn"
}

Instead of plain password you may specify encrypted_password with encrypted content which usually makes more sense considering that value is stored in SCM.

Hg

{
  "url": "repos/myhg",
  "filter": {
    "ignore": [
      "externals",
      "tools"
    ]
  },
  "destination": "dir1",
  "auto_update": false,
  "name": "hgMaterial1",
  "type": "hg"
}

Perforce

{
  "port": "10.18.3.102:1666",
  "username": "user1",
  "password": "pass1",
  "use_tickets": false,
  "view": "//depot/dev/src...          //anything/src/...",
  "filter": {
    "ignore": [
      "lib",
      "tools"
    ]
  },
  "destination": "dir1",
  "auto_update": false,
  "name": "p4materialName",
  "type": "p4"
}

Instead of plain password you may specify encrypted_password with encrypted content which usually makes more sense considering that value is stored in SCM.

Tfs

{
  "url": "url3",
  "username": "user4",
  "domain": "example.com",
  "password": "pass",
  "project": "projectDir",
  "filter": {
    "ignore": [
      "tools",
      "externals"
    ]
  },
  "destination": "dir1",
  "auto_update": false,
  "name": "tfsMaterialName",
  "type": "tfs"
}

Instead of plain password you may specify encrypted_password with encrypted content which usually makes more sense considering that value is stored in SCM.

Dependency

{
  "pipeline": "pipeline2",
  "stage": "build",
  "name": "pipe2",
  "type": "dependency"
}

Package

{
  "package_id": "apt-repo-id",
  "name": "myapt",
  "type": "package"
}

Pluggable SCM

{
  "scm_id": "someScmGitRepositoryId",
  "destination": "destinationDir",
  "filter": {
    "ignore": [
      "dir1",
      "dir2"
    ]
  },
  "name": "myPluggableGit",
  "type": "plugin"
}

Configrepo

This is a convenience for shorter and more consistent material declaration. When configuration repository is the same as one of pipeline materials, then you usually need to repeat definitions in XML and in JSON, for example:

...
  "materials": [
    {
      "url": "https://github.com/tomzo/gocd-json-config-example.git",
      "branch" : "ci",
      "type": "git",
      "name" : "mygit"
    }
  ],
...

And in server XML:

<config-repos>
   <config-repo pluginId="json.config.plugin" id="repo1">
     <git url="https://github.com/tomzo/gocd-json-config-example.git" branch="ci" />
   </config-repo>
</config-repos>

Notice that url and branch is repeated. This is inconvenient in case when you move repository, because it requires 2 updates, in code and in server XML.

Using configrepo material type, above repetition can be avoided, last example can be refactored into:

...
  "materials": [
    {
      "type": "configrepo",
      "name" : "mygit"
    }
  ],
...

Server interprets configrepo material in this way:

Clone the material configuration of the repository we are parsing as is in XML and replace name, destination and filters (whitelist/blacklist), then use the modified clone in place of configrepo material.

Tasks

Every task object must have type field. Which can be exec, ant, nant, rake, fetch, plugin

Optionally any task can have run_if and on_cancel.

  • run_if is a string. Valid values are passed, failed, any
  • on_cancel is a task object. Same rules apply as to tasks described on this page.

Exec

{
    "type": "exec",
    "run_if": "passed",
    "on_cancel" : null,
    "command": "make",
    "arguments": [
      "-j3",
      "docs",
      "install"
    ],
    "working_directory": null      
}

Ant

{
  "build_file": "mybuild.xml",
  "target": "compile",
  "type": "ant",
  "run_if": "any",
  "on_cancel" : null,
}

Nant

{
  "type": "nant",
  "run_if": "passed",
  "working_directory": "script/build/123",
  "build_file": null,
  "target": null,
  "nant_path": null
}

Rake

{
  "type": "rake",
  "run_if": "passed",
  "working_directory": "sample-project",
  "build_file": null,
  "target": null
}

Fetch

Fetch artifact from the GoCD server

{
   "type": "fetch",
   "artifact_origin": "gocd",
   "run_if": "any",
   "pipeline": "upstream",
   "stage": "upstream_stage",
   "job": "upstream_job",
   "is_source_a_file": false,
   "source": "result",
   "destination": "test"
 }

Fetch artifact from an external artifact store

{
   "type": "fetch",
   "artifact_origin": "external",
   "run_if": "any",
   "pipeline": "upstream",
   "stage": "upstream_stage",
   "job": "upstream_job",
   "artifact_id": "upstream_external_artifactid",
   "configuration": [
     {
       "key": "DestOnAgent",
       "value": "foo"
     },
     {
       "key": "some_secure_property",
       "encrypted_value": "ssd#%fFS*!Esx"
     }
   ]
 }

Plugin

{
  "type": "plugin",
  "configuration": [
    {
      "key": "ConverterType",
      "value": "jsunit"
    },
    {
      "key": "password",
      "encrypted_value": "ssd#%fFS*!Esx"
    }
  ],
  "run_if": "passed",
  "plugin_configuration": {
    "id": "xunit.converter.task.plugin",
    "version": "1"
  },
  "on_cancel": null
}

Contributing

Create issues and PRs if

  • something does not work as you expect it,
  • documentation is not good enough
  • you have questions about GoCD behavior with remote configuration

There has been a long effort to make it possible to store configuration in SCMs, so obviously there will be some errors in lots of new code. Please file issues here or ask on gocd gitter chat.

License and Authors

License: Apache 2.0

Authors:

gocd-json-config-plugin's People

Contributors

tomzo avatar arvindsv avatar marques-work avatar ibnc avatar varshavaradarajan avatar ganeshspatil avatar hveiga avatar aerostitch avatar ketan avatar

Watchers

James Cloos avatar avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.