appuio / openshift-upgrade-controller Goto Github PK

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Repository problems

These problems occurred while renovating this repository. View logs.

• WARN: Package lookup failures

Rate-Limited

These updates are currently rate-limited. Click on a checkbox below to force their creation now.

• Update actions/cache action to v4

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• Update github.com/openshift/api digest to 1606e0e
• Update golang.org/x/exp digest to 1b97071
• Update docker.io/library/alpine Docker tag to v3.19
• Update kubernetes packages to v0.29.1 (k8s.io/api, k8s.io/apimachinery, k8s.io/client-go)
• Update module github.com/prometheus/client_golang to v1.18.0
• Update module golang.org/x/mod to v0.14.0
• Update module sigs.k8s.io/controller-runtime to v0.17.0
• Update module sigs.k8s.io/controller-tools to v0.14.0
• Update module sigs.k8s.io/kustomize/kustomize/v5 to v5.3.0
• Update actions/setup-go action to v5
• Click on this checkbox to rebase all open PRs at once

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

Summary

As a maintenance engineer
I want to execute code, in a maintenance window, that does not conflict with upgrades
So that I don't have to intervene during out-of-office hours.

Context

We added support for running arbitrary code before and after upgrades, we now not only use this for communication with external systems but also to rollout changes or drain nodes.

Some of those tasks need to be run in every maintenance window. We could create a cronjob for that, but the cronjob has no insight if the cluster is upgrading. We could block the upgrade process or introduce conditions where the cluster upgrade fails by accident.

Out of Scope

No response

Further links

• https://kb.vshn.ch/oc4/references/architecture/upgrade_controller.html#_upgradejobhook

Acceptance Criteria

• Hooks can be run on the upgrade schedule:
  • immediately if no upgrade
  • in sync with the upgrade if one exists

Implementation Ideas

UpgradeConfig creates UpgradeJob with empty desiredVersion. Upgrade job runs health checks and jobs as usual, but does not touch the desiredVersion.

          This does not always seem to be true. If the update gets stuck - seen on the "play" customer cluster, broken ceph node - it gives up at some point. Needs more research.

Originally posted by @bastjan in #4 (comment)

Summary

As cluster operator
I want to mark UpgradeJobHooks as possibly disruptive
So that I can decide if i should set up any alert silences

As cluster operator
I want to mark UpgradeJobHooks as possibly disruptive
So that I can have metrics and mark the jobs in dashboards

Context

At VSHN we use UpgradeJobHooks to set silences and to do various disruptive and non-disruptive tasks.

We only want to set silences if there is a possibly disruptive job or an upgrade.
We want to distinguished "risky" upgrades from "noop" upgrades on our dashboards.

Out of Scope

No response

Further links

• https://github.com/appuio/component-openshift-upgrade-controller/blob/master/component/scripts/silence.sh
• https://insights.appuio.net/d/bfdba027-3b4b-4b24-8ca4-6522130b3545/openshift-upgrades?orgId=1

Acceptance Criteria

• UpgradeJobHook CRD has a new field allowing to mark jobs as disruptive.
• UpgradeJob has a metric if it matches any disruptive UpgradeJobHook
• UpgradeJobHook pods have an environment variable passed in with the disruptive status of the current hook and of the disruptive status of all matching hooks.

Implementation Ideas

No response