In order to build an operator that turns CRDs into a configmap the api needs to support reloading the configmap on changes on-the-fly.

In order to control deployments without requiring a GUI integration with Slack might be the fastest way to securely disclose these actions.

Slash commands seem most suitable for quickly implementing these.

Considerations:

• lock per manifest?
• lock per pipeline?

• support incremental version numbers
• support storing secrets (encrypted in manifest?)
• make build logs accessible
• deployments in manifest
• reference to deployment in pipelines section to automatically do integration deployments
• slack integration for starting, rolling back deployment
• allow choice of nightly, alpha, beta, stable agent for build
• google auth to access logs?
• triggers: docker update, npm update, nuget update, github push, github release, bitbucket push, cron

https://api.slack.com/slash-commands#validating_the_command

Currently only public registry containers are available for use in the build manifest.

Add support for private images from

• Docker Hub
• Google Container Registry (gcr.io)

In order to figure out why broken builds are broken the logs need to be viewable.

Things to take into consideration:

• can we postpone a gui or use a gui that only deeplinks to build logs as long as possible?
• can logs be viewable via slack integration instead?
• or is deeplinking into GCE stackdriver logs good enough for now?
• if stored how to control retention? server-wide, per build manifest?

In order to support semver or another type of custom version number depending on the build iteration a build counter is required.

Things to take into considerations

• counter per branch?
• label per branch but not for the stable release branch (usually master)?

Instead of relying on a shared integration environment it's hugely beneficial to be able to run sandboxed service containers during steps of your pipeline. This could run a mysql database or anything else that's containerized.

Right now any track can be set but it results in failed image pull in the created job.

If the Pub/Sub subscription and topic in a different project, the Pub/Sub trigger won't work

Builds usually run on preemptible vms, which can be killed at any moment. For long-running tasks - let's say more than an hour - it might be useful to be able to specify that it's long-running so it can be scheduled on a regular vm.

For example for viewing logs, starting deployments, rolling back, etc.

I'd say it's feature request.
It would be useful to configure release policies inside estafette-ci for specific projects/pipelines.

We have few projects with specific release restrictions.
For example in the gcp-infra-* projects we trying to follow gitflow and have following release policies:

1. Per project/pipeline limit release branch name to allow release only from specific branch ( gcp-infra-* release only from ‘master’ )
2. Prevent release(rollback) of old build, allow only releases(deployment) of currently released build version or newer ( for example initiating rollback by releasing of old/historical builds in gcp-infra-* may apply outdated terraform manifests and delete GCP resources, we want to force people to move forward and even in case if rollback make merge changes to the master and only then releasing new build)

While these policies can be implemented in individual extensions it feels role of enforcing such policies belongs to CI/CD system.
By having release logic in extension we mixing core functionality of extension with release logic (checking branch names, providing mechanism of semversioning to track deployed builds). Also all configuration provided through pipeline/project .estafette file and can be altered in a branch by user so branch with removed restrictions can be pushed and released ignoring master branch configuration.
Also with policy logic implemented in extension there's no clear way to prevent release from running in estafette - now when estafette release initiated we apply logic inside extension and then fail release with log message describing policy violation - ideally I'd like to prevent release running at all.

I think having such functionality might help other projects trying to implement gitflow and enforce conventions in uniform way.

Possible implementation might be additional section in estafette-ci config 'release_policies'. So we configure it outside of pipeline projects and have validation and enforcement done by estafette-ci.

release_policies:
  gcp-infra-prd:
    pipeline_name: gcp-infra-prd
    restrict_branch: master
    strategy: move_forward

https://developer.github.com/webhooks/securing/

In order to push containers to registries, deploy containers to any environment, etc a secret is usually required.

To benefit from version controlling the secrets it would be nice to store them encrypted in plain sight in the manifest, but storing them in a database could be an easier alternative.

The extension uses the ESTAFETTE_BUILD_MAX_COUNTER_CURRENT_BRANCH and ESTAFETTE_BUILD_CURRENT_COUNTER to ensure running the latest build for the branch. But the value of ESTAFETTE_BUILD_MAX_COUNTER_CURRENT_BRANCH seems not right.

To start builds or deployments on a schedule a cron trigger is required.

• which revision to trigger?
• what cron syntax to use? user-friendly, powerful or best of both worlds?

https://developer.github.com/changes/2020-04-15-replacing-create-installation-access-token-endpoint/

Considerations:

• merge multiple manifests into one and then run ci-builder?
• link them via triggers, for example when a new version of a container is available it can be deployed with deployment manifest in separate repo?

Storing the logs in the database works pretty well to get started, but doesn't scale very well. Make storing logs in GCS configurable instead.

To trigger builds when a docker image is updated the following is required:

• possibility to list triggers in manifest
• integration with docker hub (https://docs.docker.com/docker-hub/webhooks/)
• starting a job for building or deploying an application (which revision to use?)

The execution of the .estafette.yaml manifest completely depends on Docker container images. As the ci-builder runs docker-in-docker and starts a fresh Kubernetes job for every build the used Docker container images are never available locally and have to be fetched from Docker Hub.

To improve performance a local cache would help. Preferably with background refreshes of the dev, beta and stable track tags.

And publish to https://helm.estafette.io

In order to elevate deployments to a special status compared to any build steps having a deployments section in the manifest makes sense.

Things to take into consideration:

• separate deployment targets with separate environment variables in the deployments section
• either manually or automatically triggered
• to be able to deploy during builds - for example to an integration environment - and run tests after as part of the build a reference from the pipelines section to one of the deployment targets is required
• with possibly many separate steps during a deployment is it sensible to be able to define one or more deployment templates within the manifest and refer to those from each deployment target?
• to make deploying any revision to any environment via cli / chatops / gui is a deployment template required?
• define actions that can be used to manipulate a deployment? deploy, rollback, canary, percentage, etc

In order to preview new features or test newer revisions of the builder it is useful to be able to select which builder you want to use if you don't want to use the default stable track.

For this to work the builder builds need a way to promote a revision to one of the tracks; possibly by just merging a revision into a similarly named branch and a container push step that uses the branch name as tag.

Build/ release steps are already migrated to GCS but the JSON blobs are still
stored in steps column of table which cause these three tables to take more than
90% of database space. Queries on these tables are also slowed due to this

Currently the following cpu/memory request/limit values are used for all jobs:

It might be useful to be able to set this via each manifest, although we would still probably like to be able to set upper limits.

cpuRequest := "1.0"
cpuLimit := "3.0"
memoryRequest := "2.0Gi"
memoryLimit := "20.0Gi"

The following error happens when config is updated:

panic: duplicate metrics collector registration attempted
estafette-ci-api-canary-74fddc5d7c-l78lp estafette-ci-api
goroutine 144 [running]:
github.com/prometheus/client_golang/prometheus.(*Registry).MustRegister(0xc0001473b0, 0xc000fabe90, 0x1, 0x1)
	/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/registry.go:400 +0xad
github.com/prometheus/client_golang/prometheus.MustRegister(...)
	/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/registry.go:177
github.com/go-kit/kit/metrics/prometheus.NewCounterFrom(0x13e9d88, 0x3, 0x13f5523, 0xf, 0x13f3c59, 0xd, 0x140443f, 0x1c, 0x0, 0xc000fabe50, ...)
	/go/pkg/mod/github.com/go-kit/[email protected]/metrics/prometheus/prometheus.go:24 +0xf7
github.com/estafette/estafette-ci-api/helpers.NewRequestCounter(...)
	/estafette-work/helpers/metrics.go:20
main.getInstances(0x15f6860, 0xc000128000, 0x34, 0xc001161e88, 0x1, 0x1, 0xb, 0x2b, 0xc0002f7d20)
	/estafette-work/main.go:179 +0x590
main.initRequestHandlers.func1(0xc000329a00, 0x13, 0x8)
	/estafette-work/main.go:122 +0x14f
github.com/estafette/estafette-foundation.WatchForFileChanges.func1.1(0xc000080190, 0xc000693580, 0x13fa455, 0x14, 0x13fa455, 0x14, 0xc0002e8450, 0xc0000800f0)
	/go/pkg/mod/github.com/estafette/[email protected]/foundation.go:118 +0x1f4
created by github.com/estafette/estafette-foundation.WatchForFileChanges.func1
	/go/pkg/mod/github.com/estafette/[email protected]/foundation.go:100 +0x1dd

In order to test a build before submitting it running ci-builder locally runs the build steps identically to being run by the Estafette CI system, besides subtle differences like Docker version, Linux kernel version, etc.

Depends on #19 and #9.

Secrets can be stored encrypted in the manifest, but there should be the ability to make it extensible and retrieve secrets from

• Vault
• Google Cloud KMS
• AWS KMS

Currently the builder can decrypt secrets, but this isn't very safe. Better to let the api exclusively handle this. Depends on implementation of OAuth (or another form of authentication) in the API layer. See #9