etcd keeps crashing about okd HOT 14 CLOSED

I should note we also upgraded the drives in our ctl nodes to SSDs which seemed like it did the trick for a bit, but etcd still fell over. I would expect if this was a hardware or network issue we'd have seen it in the previous version too. I've tried restarting all the nodes multiple times and keep seeing the same behavior: it's fine for a while, and then eventually etcd gets overwhelmed and dies

from okd.

Oh and oddly around the time of the issue we get loads of 'peer netns reference is invalid' messages from the garbage-collector namespace
extract-2024-01-04T21_29_11.613Z.csv

from okd.

{"msg":"apply request took too long","took":"2.00022516s","request":"key:"/kubernetes.io/health" ","caller":"etcdserver/util.go:170","level":"warn","prefix":"read-only range ","response":"","error":"context canceled","expected-duration":"200ms","ts":"2024-01-04T20:52:04.900678Z"}

It takes etcd 2 seconds to do a read operation, disk is too slow for it. SSD or NVME disks are a requirement for OKD

from okd.

{"msg":"apply request took too long","took":"2.00022516s","request":"key:"/kubernetes.io/health" ","caller":"etcdserver/util.go:170","level":"warn","prefix":"read-only range ","response":"","error":"context canceled","expected-duration":"200ms","ts":"2024-01-04T20:52:04.900678Z"}

It takes etcd 2 seconds to do a read operation, disk is too slow for it. SSD or NVME disks are a requirement for OKD

We're using SSDs, and before etcd falls over, the request latency is totally normal, check my "before" logs

from okd.

Also prior to this version we had totally acceptable etcd performance

from okd.

I mean, here's our etcd metrics:

it's possible I'm running this query wrong, but the latency looks pretty normal?

from okd.

I have no idea what the anomalies at 2 PM and just now are

from okd.

And that's the thing that's so weird about this: if I reboot the control plane nodes, OKD seems to behave for a reasonable amount of time with no signs of high latency:

from okd.

but at some point it just gets a storm of reads or something from the apiserver which topples it

from okd.

{"level":"info","ts":"2024-01-05T01:04:41.203378Z","caller":"traceutil/trace.go:171","msg":"trace[1574369963] range","detail":"{range_begin:/kubernetes.io/events/; range_end:/kubernetes.io/events0; response_count:250; response_revision:1138218247; }","duration":"105.575164ms","start":"2024-01-05T01:04:41.097779Z","end":"2024-01-05T01:04:41.203354Z","steps":["trace[1574369963] 'range keys from in-memory index tree'  (duration: 103.958977ms)"],"step_count":1}
{"level":"info","ts":"2024-01-05T01:05:40.560993Z","caller":"traceutil/trace.go:171","msg":"trace[1657201551] range","detail":"{range_begin:/kubernetes.io/events/hoby-feedback/exposer-ongsovhjcos7k8j79ni71h29962l557g8a6q8due9n0p3sr5fj5g.17a6f94efeedff69\u0000; range_end:/kubernetes.io/events0; response_count:10000; response_revision:1138219153; }","duration":"113.96833ms","start":"2024-01-05T01:05:40.447007Z","end":"2024-01-05T01:05:40.560975Z","steps":["trace[1657201551] 'range keys from in-memory index tree'  (duration: 86.5626ms)","trace[1657201551] 'range keys from bolt db'  (duration: 26.781581ms)"],"step_count":2}
{"level":"info","ts":"2024-01-05T01:07:19.046712Z","caller":"etcdserver/server.go:1404","msg":"triggering snapshot","local-member-id":"26c01424b8ce265f","local-member-applied-index":1302462815,"local-member-snapshot-index":1302452814,"local-member-snapshot-count":10000}
{"level":"info","ts":"2024-01-05T01:07:19.051514Z","caller":"etcdserver/server.go:2422","msg":"saved snapshot","snapshot-index":1302462815}
{"level":"info","ts":"2024-01-05T01:07:19.051698Z","caller":"etcdserver/server.go:2452","msg":"compacted Raft logs","compact-index":1302457815}

Like, our latencies are fairly normal almost all the time, but suddenly we get a bunch of requests or something which causes etcd to get overwhelmed.

from okd.

We're using these SSDs on our control plane, they back etcd:

[core@okd4-ctl02-nrh ~]$ cat /sys/block/sdb/device/model
VK000960GWJPF

from okd.

So, sometimes the disks are okay and you can open a console - and sometimes the disk is slow and so etcd is timing out? This is clearly a hardware / IO usage problem, not sure what OKD can do about this.

from okd.

@vrutkovs turns out the new version seems to have changed network handling: our control plane nodes have statically configured addresses and the generated default networkmanager config wasn't limited to any particular interface, so we were having intermittent networking issues caused by having two default routes and two interfaces with the same IP. I think this is a regression or something, because we didn't have this problem on older versions.

from okd.

Manually editing the networkmanager configs fixed it

from okd.