HA setup over Tailscale about k3s HOT 6 CLOSED

See https://docs.k3s.io/networking/distributed-multicloud:

Embedded etcd is not supported in this type of deployment. If using embedded etcd, all server nodes must be reachable to each other via their private IPs. Agents may be distributed over multiple networks, but all servers should be in the same location.

All etcd nodes must be on the same private network.

from k3s.

Hi @brandond! They are.

from k3s.

OK, but can they reach each other at their private IPs? It appears they cannot based on your logs:

May 22 10:34:08 control-cax21-nbg1 k3s[210139]: {"level":"warn","ts":"2024-05-22T10:34:08.562614Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x40007cb880/142.132.176.81:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
May 22 10:34:08 control-cax21-nbg1 k3s[210139]: time="2024-05-22T10:34:08Z" level=fatal msg="etcd cluster join failed: context deadline exceeded"
May 22 10:34:08 control-cax21-nbg1 systemd[1]: k3s.service: Main process exited, code=exited, status=1/FAILURE

Are you using public IPs as the nodes private addresses?

from k3s.

Nope, that's the thing. I'm setting the node-external-ip and not the node-ip as the logs say that value is overridden by VPN configuration. Do I need to set the node-ip as well?

from k3s.

Do you have any idea why the nodes wouldn't be able to reach each other at the selected addresses? Do you have firewall rules or something else in place that is blocking the etcd traffic?

from k3s.

Yes I do have an idea. The nodes are trying to use the external address to comunicate and yes, that traffic is not allowed by firewall rules! Setting the node-ip in the previous sessions I tried didn't seem to help, but I would have to check it out again. The idea as I understand would be to

• Set the node ip to the private subnet one (not the vpn). K3s will not override it for etcd communication
• Use the vpn to allow agent nodes to join the cluster. Use the vpn ip to join
• I expect to be able to set the external ip as well and it won't be used to do any inter-node communication (i think this is the point were there is something not working as expected)

from k3s.