Comments (5)
The KMS provider itself runs as a pod on the cluster.
I'm not familiar with this deployment pattern for KMS providers - why are you trying to do this? It suffers from the obvious chicken-and-egg problem you're running into here, where the cluster can't start because it needs access to something that won't be available until after it's up.
You're trying to figure out how to lock your keys in the car but still open the door. I don't think there's a good way to make this work.
from k3s.
The KMS provider itself runs as a pod on the cluster.
This is not an uncommon pattern for KMS deployment.
Arguably k3s has a circular dependency on kubernetes secrets. It is unfortunate that this is not part of the conformance tests, at least as far as I can tell.
https://github.com/kubernetes-sigs/aws-encryption-provider
https://github.com/Azure/kubernetes-kms?tab=readme-ov-file
https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/barbican-kms-plugin/using-barbican-kms-plugin.md
https://github.com/Tencent/tke-kms-plugin/blob/90b71a5c7d78a564567040ebe1ce7135afe99ce5/deployment/tke-kms-plugin.yaml#L4
from k3s.
K3s uses secrets for a couple things internally:
- Node password verification
- Supervisor/apiserver certificate sync
Both of these should soft-fail and retry until secrets can be read. Where exactly does k3s startup stall?
I see that https://github.com/kubernetes-sigs/aws-encryption-provider for example suggests running the KMS as a static pod - are you doing that by placing the pod spec in a file in /var/lib/rancher/k3s/agent/pod-manifests/
, or are you trying to deploy it via kubectl apply
?
from k3s.
suggests running the KMS as a static pod
Yes. Static pods have the same issue.
Both of these should soft-fail and retry until secrets can be read. Where exactly does k3s startup stall?
I don't know. I attached the logs from the systemd service in the issue where it's trying to access /registry/secrets/kube-system/k3s-serving
. Does that answer your question? Why does it hard fail on this secret? I can get more logs if you share instructions.
from k3s.
This repository uses a bot to automatically label issues which have not had any activity (commit/comment/label) for 45 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the bot can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the bot will automatically close the issue in 14 days. Thank you for your contributions.
from k3s.
Related Issues (20)
- [Release-1.29] - Agent loadbalancer may deadlock when servers are removed HOT 1
- [Release-1.28] - Agent loadbalancer may deadlock when servers are removed
- [Release-1.27] - Agent loadbalancer may deadlock when servers are removed HOT 1
- [Release-1.29] - `k3s etcd-snapshot` commands run against server specified in config file, instead of local server HOT 1
- [Release-1.28] - `k3s etcd-snapshot` commands run against server specified in config file, instead of local server HOT 1
- [Release-1.27] - `k3s etcd-snapshot` commands run against server specified in config file, instead of local server HOT 1
- kube-proxy with ipvs and lc does not work as expected (perhaps a conflict with flannel rules) HOT 3
- [Release-1.30] - Agent loadbalancer may deadlock when servers are removed HOT 1
- [Release-1.30] - `k3s etcd-snapshot` commands run against server specified in config file, instead of local server HOT 1
- Update the Traefik chart HOT 3
- Respect XDG base directory HOT 3
- Install script fails without sudo due to missing $SUDO prefix in transactional-update commands
- Logging of kube-scheduler HOT 5
- NVIDIA GPU detection doesn't work with all the drivers & toolkits installed HOT 2
- Containerd not installed by startup script - invalid capacity 0 on image filesystem warning when starting k3s node HOT 3
- SyncLoadBalancerFailed when using a very long (yet valid) service name
- "Section 3 Control Plane Configuration" in CIS Kubernetes Benchmark v1.8.0 is not applicable for K3s hardening ?
- K3s Multus + Whereabouts doesn't work
- v1.30.2+k3s2: 502 bad gateway when trying to get pod logs HOT 2
- Certificate rotation has no affect on client if server directory is present HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from k3s.