boxboat / dockhand-jenkins Goto Github PK
View Code? Open in Web Editor NEWUpstream Jenkins Shared Library to Build, Promote, and Deploy Docker images
License: MIT License
Upstream Jenkins Shared Library to Build, Promote, and Deploy Docker images
License: MIT License
Running promote when a tag has been written to build-versions but not tagged on the repo (with promote.gitTagDisable=false
) causes all future promotions to fail to promote with: No version changes detected
.
Proposed solution
These checks also need a null check to allow continued operations if a tag has not been written back (or was deleted): https://github.com/boxboat/dockhand/blob/master/src/com/boxboat/jenkins/pipeline/promote/BoxPromote.groovy#L138-L139
Steps to reproduce:
promote.gitTagDisable=false
)promote.gitTagDisable=true
)promote.gitTagDisable=true
Logs:
+ cat /home/jenkins/agent/workspace/jca_docker_scratch-prmote_master/build-versions/image-versions/commit/master/scratch-jenkins-kubectl.txt
[Pipeline] sh
+ cd /home/jenkins/agent/workspace/jca_docker_scratch-prmote_master
+ git config remote.origin.url
[Pipeline] sh
+ '[' -f /home/jenkins/agent/workspace/jca_docker_scratch-prmote_master/build-versions/repo-versions/<>/scratch-pipeline/tag-release.txt ]
+ cat /home/jenkins/agent/workspace/jca_docker_scratch-prmote_master/build-versions/repo-versions/<>/scratch-pipeline/tag-release.txt
[Pipeline] sh
+ cd /home/jenkins/agent/workspace/jca_docker_scratch-prmote_master
+ git show-ref '--hash=12' -s 0.1.10
+ :
[Pipeline] sh
+ cd /home/jenkins/agent/workspace/jca_docker_scratch-prmote_master
+ git show-ref '--hash=12' -s 0.1.10
+ :
[Pipeline] sh
+ cd /home/jenkins/agent/workspace/jca_docker_scratch-prmote_master
+ git show-ref '--hash=12' -s null
+ :
[Pipeline] echo
No version changes detected
Rename getImageTagsMetadata - get/set methods with no parameters can lead to unexpected behavior with groovy properties
Currently when using Utils.withTmpDir
or Utils.withTmpFile
paths are set in an environment variable, relative to the current location (ie. DIR_PATH=./tmp-${UUID}
). This becomes a problem if a script changes directories after the file/directory is created. Temporary files should be created in the workspace folder so they have a consistent reference point
A user may wish to patch and promote an old version of their software that they still support while they continue to work on a newer version. If the major/minor version has advanced (ie. want to patch 1.0.0
, current released version is 3.0.0
) the library does not support this.
Proposed solution:
Create another event type in promote (promote.promoteToEvent
) of format imageTag/<version>
and allow users to input this field and a commit hash to apply this tag. When the pipeline is run with this parameter, promote that hash with that tag and push. Since this is a one-off, do not write back to build-versions
Users could then cut a support branch, patch, promote and merge back in for old versions of their software
Dockhand expects credentials to be pulled from Jenkins credentials store for configuration defined in the library's config.yaml
.
Anywhere the credential
key is used, I would like to be able to reference a path and a vault (from the vaultMap
) and read that credential from Hashicorp Vault. Furthermore, I would like to be able to specify the env var names that credentials are set to as I can with Jenkin's Credentials Plugin.
Add dockcmd
support to allow for advanced secrets management for Vault and AWS Secrets Manager
As a user I would like to be able to define my own configuration to use with my own library functions and pipelines. An example of what this might look like is:
common:
userConfigMap:
toolConfig:
configLocation: "/path/to/config"
additionalOptions: "--option value"
During image promotion pipelines extract commit log between SemVer releases and store in BoxPromote
. Allow downstream clients the ability to add arbitrary metadata object to image-versions and repo-versions for promotion pipelines. Users may wish to associate issues to a release For example, you can now leverage the commits between semvers to scrape issues and store them with the build-version:
Override BoxPromote.addPromotionMetadata
:
@Override
def addPromotionMetadata() {
Pattern pattern = Pattern.compile("[A-Z]{2,}-\\d+")
Set<String> issues = new LinkedHashSet<>()
commits.each {
Matcher matcher = pattern.matcher(it.subject)
while(matcher.find()) {
issues.add(matcher.group(0))
}
}
metadata.put("issues", issues.toArray())
}
Results in a build-versions entry like:
version: 0.3.18
metadata:
issues: [ABC-1, XY-123, XYZ-1234]
BoxDeploy
has a new method getImageTagsMetadata
that will allow you to retrieve the metadata associated with images you are about to deploy returning something like:
demo/dockhand-demo-app:
version: 0.3.18
metadata:
issues: [ABC-1, XY-123, XYZ-1234]
This change does not break backwards compatibility but will migrate build-versions .txt
files that previously contained just the SemVer text to a yaml format minimally containing:
version: <version-string>
This conversion will occur as images are build and promoted by jenkins-dockhand
At present the shared library tags with a latest equivalent tag (commit-branch
) and build-${SHORT_COMMIT_HASH}
. For users who have already defined a build-tag structure or wish to key off of certain build tags, it would be useful to allow a custom set of tags to be applied to all images in a build.
For example, if using fluxcd it you may wish to have ${BRANCH}-${SHORT_COMMIT_HASH}
or ${BRANCH}-${BUILD_NUMBER}
in addition to the build tag so different environments can upgrade their images independently.
This could be done with BoxBuild.push(List<String> tagList = [])
and appending tags in that list to the tags dockhand generates
Test the following:
Dockhand should allow for configuration of deployment to multiple Kubernetes clusters for a deployment. i.e. Scenario where customer has production environment spanning two kubernetes clusters and wants to replicate an application deployment to both clusters from one Jenkins pipeline
If running the shared library against PRs sometimes it will fail to determine that the PR is at the tip of its respective branch and it will thus fail to write to build versions. This occurs when the Jenkins merges the target branch in to the PR and thus gets a different commit hash (Ie. branch is no longer ahead of master)
It is not transparent to the user that this is happening and that is the cause of build versions not being updated. Building without this merge seems unwise since build results may differ before and after the merge.
There is not a clear way to fix this. We could disable the setting in Jenkins and make the library do the merge but that means that setup could be finicky. The merge should be detectable -- GIT_COMMIT
will not match GIT_COMMIT_SHORT_HASH
Would love to hear other people's thoughts
Jenkins settings:
Behaviors:
Discover pull requests from origin:
Merging the pull request with the current branch target revision
jenkins.yaml
:
common:
eventRegistryKeys:
- registryKey: commit/PR-(.*)
registry: default
I would expect that defining a configuration in a jenkins.yaml
would merge with, instead of replace configuration defined in config.yaml
. The documentation does not state which should happen. Experimentation points to replacement instead of merging. Is this the desired behavior?
config.yaml
:
repo:
common:
eventRegistryKeys:
- event: commit/develop
registryKey: develop
- event: commit/master
registryKey: master
- eventRegex: commit/release/(.*)
registryKey: release
jenkins.yaml
:
common:
eventRegistryKeys:
- event: commit/develop
registryKey: develop-alternative
build:
eventRegistryKeys:
- event: commit/master
registryKey: master-alternative
I would expect a common pipeline to have the config available to it:
eventRegistryKeys:
- event: commit/develop
registryKey: develop-alternative
- event: commit/master
registryKey: master
- eventRegex: commit/release/(.*)
registryKey: release
Instead it can only accessy:
eventRegistryKeys:
- event: commit/develop
registryKey: develop-alternative
I would expect my build pipeline to have the config available to it:
eventRegistryKeys:
- event: commit/develop
registryKey: develop-alternative
- event: commit/master
registryKey: master-alternative
- eventRegex: commit/release/(.*)
registryKey: release
Instead it can only access:
eventRegistryKeys:
- event: commit/master
registryKey: master-alternative
Currently, dockhand
has a DTRClean
. Need to add a HarborClean
to provide similar functionality
For promotion pipelines, it would be useful for the promote
to use modular components so that items like the currSemVers can be accessed and used to promote non-docker artifacts (ie. npm, nuget, python, etc) when an image is promoted without duplicating this code.
In the simplest sense this could be done like:
class BoxPromote {
def getCurrentSemanticVersion(){ ... }
def getNextSemanticVersion(){ ... }
def hasVersionChanged(){ ... }
}
with a subclass like this using it as it likes:
class MyPromote extends BoxPromote {
@Override
def init(){
def currVer = getCurrentSemanticVersion()
def nextVer = getNextSemanticVersion()
Config.pipeline.env['VERSION'] = (hasVersionChanged()) ? currVer : nextVer
}
}
Support Google Cloud Services:
GCloudRegistry
GCloudRegistryClean
GCloudGKEDeployTarget
After the initial promotion tag is set from promote.baseVersion
it appears that updates to this field do not cause the tag to update.
To reproduce:
0.1.0
to 0.1.50
-- build versions only up to 0.1.10
) run another promotion.Job triggers are not processed properly if someone introduces a jenkins job name containing a space
Add BuildKit support to Compose
Add regctl for image promotion. This will accelerate image promotion pipelines
Add support for Azure Key Vault, which will be in dockhand
1.1.0
Currently when a promotion pipeline runs it will always increment the semantic version. The promotion logic should determine whether or not the image has actually changed and do nothing if the image hasn't been updated
Image pushes are currently done sequentially. With many images, this can slow down the build process. I would like a push that takes an image name as a parameter and pushes all the tags for that image. This would allow push to be run in a parallel{}
block. Alternatively refactoring to make it easier to extending the push functionality (ie. splitting the actual image push to its own function) would allow user libraries to provide this functionality for themselves
Registry clean logic currently only removes build-<hash>
tags from the registry, but overtime your commit-<branch>
tags will continue to grow - requiring manual image tag removal. Update build-version
repo store git repo url and branch metadata - allowing the cleanup routines to cleanup stale commit-<branch>
tags
I would like to run Dockhand with multiple Jenkins instances. This may cause collisions with build-versions
because the write-lock to that repo is controlled by a Jenkins instance. To get around this, I would like to be able to configure multiple build-versions
repos in the library for different Jenkins instances and select between them from my code repo (ie. in the jenkins.yaml
)
Presently, gitRepo.getBranch()
returns the name of the PR, such as PR-1
for pull requests
Example setup:
feature/my-feature
commit/feature/my-feature
master
- PR-1
is builtgitRepo.getBranch()
returns PR-2
as the branch name, which does not match commit/feature/my-feature
My inclination is to have gitRepo.getBranch()
return the CHANGE_BRANCH
environment variable, if set:
About the only downside that I can see is that if pull requests from forks in an untrusted environment (such as the public) are allowed, this would be a security issue:
I am not aware of any projects using this library that allow this, so it may not be an issue.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.