Comments (3)
It sounds like your etcd datastore was corrupted. Did the k3s process crash, or did the OS experience a kernel panic and reboot? I've only ever seen anything like this when the node is unexpectedly restarted or powered off, and files are corrupted on disk. I will also note that raspberry pis are highly likely to experience filesystem corruption when using SD cards; we do not recommend using etcd with sd cards under any circumstances.
Removing the files from disk and restoring from a snapshot is the proper way to address datastore corruption. Since the errors are from etcd, and were triggered by a crash, I'm not sure we have anything to fix here in K3s.
from k3s.
It sounds like your etcd datastore was corrupted. Did the k3s process crash, or did the OS experience a kernel panic and reboot? I've only ever seen anything like this when the node is unexpectedly restarted or powered off, and files are corrupted on disk. I will also note that raspberry pis are highly likely to experience filesystem corruption when using SD cards; we do not recommend using etcd with sd cards under any circumstances.
Removing the files from disk and restoring from a snapshot is the proper way to address datastore corruption. Since the errors are from etcd, and were triggered by a crash, I'm not sure we have anything to fix here in K3s.
@brandond external usb/sata SSD, no corruption, system didn't reboot or panic, hence the issue.
from k3s.
Can you provide the complete logs from journald covering the time period before and after the crash? We don't have any way to handle etcd database file corruption in k3s, but if there is a preventable crash occurring we can take a look at that.
from k3s.
Related Issues (20)
- I can't configure the docker.io image repository HOT 1
- containerd does not fall back to default endpoint when using wildcard registry entry (airgap env) HOT 11
- K3S Logging is not working in Alma 9.3 environment HOT 7
- /etc/rancher/k3s/registries.yaml is ignored HOT 1
- Procedure for changing server one of a etcd cluster HOT 2
- Etcd bootstrap conflict HOT 1
- Issue with Embedded Registry Mirror Usage HOT 4
- Validate SUSE Liberty Linux 8.9 HOT 1
- Enhance install script to support Suse Liberty Linux HOT 2
- [v1.29.3+k3s1] Certificate rotation on k3s-agent fails to include sans addressing, causing api requests via kubectl to potentially fail if kube-vip is in use HOT 1
- Add etcd snapshot s3 config secret support
- Multiple simultaneous snapshots result in silent failure and/or corruption of at least one snapshot HOT 2
- [Release-1.29] - Multiple simultaneous snapshots result in silent failure and/or corruption of at least one snapshot HOT 1
- [Release-1.28] - Multiple simultaneous snapshots result in silent failure and/or corruption of at least one snapshot HOT 1
- [Release-1.27] - Multiple simultaneous snapshots result in silent failure and/or corruption of at least one snapshot HOT 1
- kubectl logs selectively timing out based on the host CIDR range HOT 4
- v1.30.2-rc3 Images missing HOT 4
- rpm.rancher.io RHEL 9 packages missing/removed HOT 3
- k3s crashes completely with "Observed a panic: "integer divide by zero"" HOT 10
- Allow configuration of Rootlesskit's CopyUpDirs through an environment variable
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from k3s.