Comments (12)
Sounds good. If not strictly required, I would like to avoid having handling for specific kubeadm feature gates in CAPI (eventually the feature gate will go away anyway)
from cluster-api.
/assign
from cluster-api.
If we change the webhook to make the field mutable, what is then the current rollout behavior without further changes?
I don't know if a rollout will be triggered by the change. I'll check the code and maybe also test it with our fork.
In our case with the EtcdLearnerMode
we don't even need a rollout. The next rollout will enable the feature gate and it is also only necessary for new nodes.
from cluster-api.
Sounds good to me.
Also it is not necessary to rollout the KCP immediately after the change.
If we change the webhook to make the field mutable, what is then the current rollout behavior without further changes?
from cluster-api.
cc @fabriziopandini maybe you have some thoughts on this 🙂
from cluster-api.
@sbueringer so I just tested the mutation of the kubeadm feature gates by deleting the validation webhook capi-kubeadm-control-plane-validating-webhook-configuration
.
After adding a kubeadm feature gate the KCP controller will trigger a rolling update immediately. So I think this is exactly what we are trying to achieve, right?
Example
Patch does not work
❯ k -n c42pc004 get kcp
NAME CLUSTER INITIALIZED API SERVER AVAILABLE REPLICAS READY UPDATED UNAVAILABLE AGE VERSION
c42pc004-control-plane c42pc004 true true 1 1 1 0 13h v1.27.9
❯ kubectl -n c42pc004 patch kcp c42pc004-control-plane --type='merge' -p '{"spec":{"kubeadmConfigSpec":{"clusterConfiguration":{"featureGates":{"EtcdLearnerMode": true}}}}}'
The KubeadmControlPlane "c42pc004-control-plane" is invalid: spec.kubeadmConfigSpec.clusterConfiguration.featureGates.EtcdLearnerMode: Forbidden: cannot be modified
Remove the validation webhook
❯ k delete validatingwebhookconfigurations capi-kubeadm-control-plane-validating-webhook-configuration
validatingwebhookconfiguration.admissionregistration.k8s.io "capi-kubeadm-control-plane-validating-webhook-configuration" deleted
Patch the KCP
❯ kubectl -n c42pc004 patch kcp c42pc004-control-plane --type='merge' -p '{"spec":{"kubeadmConfigSpec":{"clusterConfiguration":{"featureGates":{"EtcdLearnerMode": true}}}}}'
kubeadmcontrolplane.controlplane.cluster.x-k8s.io/c42pc004-control-plane patched
❯ k -n c42pc004 get kcp
NAME CLUSTER INITIALIZED API SERVER AVAILABLE REPLICAS READY UPDATED UNAVAILABLE AGE VERSION
c42pc004-control-plane c42pc004 true true 2 1 1 1 13h v1.27.9
And to verify that the feature gate is working:
Jan 24 17:05:34 c42pc004-control-plane-7646d458b6-fjkcn cloud-init[658]: [2024-01-24 17:05:34] I0124 17:05:34.688405 3391 etcd.go:416] [etcd] Adding etcd member as learner: 68747470733a2f2f3139322e3136382e302e37393a32333830
Jan 24 17:05:34 c42pc004-control-plane-7646d458b6-fjkcn cloud-init[658]: [2024-01-24 17:05:34] I0124 17:05:34.781981 3391 etcd.go:506] [etcd] Promoting a learner as a voting member: 8c3d38ccb8d86480
from cluster-api.
trying to understand what are the implications.
does this mean that the KCP controller will persist an FG in kube-system/kubeadm-config.ClusterConfiguration ?
on upgrade in a later version if kubeadm join
then downloads the ClusterConfiguration and finds a legacy FG, we need to make sure it does not fail. i forgot what's the existing behavior in kubeadm / need to check.
from cluster-api.
Hm not sure. The proposed change here is mostly about changing the webhook. Not sure what the implications are regarding the ConfigMap
from cluster-api.
kube-system/kubeadm-config.ClusterConfiguration is a versioned API.
thus when converting to internal type a validation will trigger:
https://github.com/kubernetes/kubernetes/blob/0e945e5cec50028bf5ad00c19a710dfc50ddec9f/cmd/kubeadm/app/apis/kubeadm/validation/validation.go#L612
i don't recall instances where in kubeadm we manually removed a FG from the ClusterConfiguration CM, during upgrade.
which means we likely left this action to the user.
in CAPI, on kubeadm join
the CM is downloaded, converted to internal type and kubeadm join can fail. this means the CAPI admin must manually remove a legacy FG from kubeadm's ClusterConfiguration if they want the cluster to allow upgrade or machine scale.
from cluster-api.
/triage approved
+1 for the change
As @neolit123 noted out, if we make some field mutable and they are expected to be persisted in the kubeadm config map (or in some other kubeadm managed resource), we should make sure this happens as well.
We already have several examples of this in https://github.com/fabriziopandini/cluster-api/blob/67c222eaaddbee7c48f5514d9f1d86c3c086069d/controlplane/kubeadm/internal/controllers/upgrade.go#L57-L123 (as well as the framework for managing kubeadm config versions)
from cluster-api.
@fabriziopandini: The label(s) triage/approved
cannot be applied, because the repository doesn't have them.
In response to this:
/triage approved
+1 for the change
As @neolit123 noted out, if we make some field mutable and they are expected to be persisted in the kubeadm config map (or in some other kubeadm managed resource), we should make sure this happens as well.We already have several examples of this in https://github.com/fabriziopandini/cluster-api/blob/67c222eaaddbee7c48f5514d9f1d86c3c086069d/controlplane/kubeadm/internal/controllers/upgrade.go#L57-L123 (as well as the framework for managing kubeadm config versions)
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
from cluster-api.
/triage accepted
from cluster-api.
Related Issues (20)
- Test framework: clusterProxy.CreateOrUpdate does not accept namespace argument HOT 7
- Cleanup the hard coded String HOT 6
- Release notes should not include the detail about release PR HOT 6
- Provide an option in the clusterctl library to suppress API server warnings HOT 17
- kube-apiserver does not gracefully terminate when a control plane machine is scaled down HOT 2
- Remove the deprecate metrics-bind-addr during 1.9 release cycle HOT 3
- BootstrapConfig should be optional for MachinePool based ClusterClass HOT 5
- Add constraint type for developer makes provider HOT 3
- Configurable machine replacement HOT 5
- Add new feature to renew certification HOT 4
- Consider if to add more on KCP checks on machine status HOT 3
- Add e2e test job for main branch that uses Kubernetes latest HOT 8
- Improve validation during upgrade e2e tests HOT 3
- Bump cert-manager to 1.15.2 HOT 1
- CAPI cluster stuck in failed after infrastructure cluster has failureMessage HOT 7
- MachineDeployment rolls out Unhealthy Machines with Ready Nodes HOT 7
- Add NodeDrainPodFilters to filter out certain pods when draining nodes HOT 12
- Scale down specific node is not working HOT 4
- conditions Utility to let specify the entire condition HOT 13
- Instance in machine pool failed to join cluster withe error bootstrap token not found HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cluster-api.