Trying to connect agent to server in two different AWS regions: SSL certificate problem: self-signed certificate in certificate chain about k3s HOT 6 CLOSED

curl -vs https://18.157.186.193:6443/cacerts on agent exposes the following problem:

This is not a problem, the cluster does use its own self-signed root CA.

May 21 17:44:03 ip-172-31-46-96 k3s[589]: time="2024-05-21T17:44:03Z" level=error msg="failed to get CA certs: Get "https://127.0.0.1:6444/cacerts\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"

Check the logs prior to this - I suspect that you have not set --node-external-ip properly on the server, and the agent load-balancer is trying to connect to the server's internal IP, which the agent cannot connect to because they are not on the same private network.

from k3s.

The security groups are open on ingress for port 6443, 6444, and 443.

Also, this is not sufficient for a functional cluster, see https://docs.k3s.io/installation/requirements#inbound-rules-for-k3s-nodes

from k3s.

The security groups are open on ingress for port 6443, 6444, and 443.

Also, this is not sufficient for a functional cluster, see https://docs.k3s.io/installation/requirements#inbound-rules-for-k3s-nodes

Are you referring to the Flannel ports and the kubelet metrics (port 10250) or more? When I've read this, I got the impression that the other ports are only optional. TCP ports 5001 and 6443 are needed by Spegel, TCP 2379-2380 for HA, and UDP port 8472, 51820 and 51821 are needed by Flannel.

from k3s.

443 and 6444 are not needed. The ports required for CNI traffic depend on which flannel backend you're using; if you don't open those then pods and services hosted on different nodes will not be able to communicate.

from k3s.

Thank you very much! I could now connect the agent with the server using the following commands and the mentioned ports.

# K3s Server
export MASTER_IP=<PUBLIC_IP>
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--node-external-ip $MASTER_IP" sh -s -
sudo cat /var/lib/rancher/k3s/server/node-token

and using the above TOKEN and MASTER_IP from server in the agent node:

# K3s Agent
export K3S_URL=https://<MASTER_IP>:6443
export K3S_TOKEN=<TOKEN>
curl -sfL https://get.k3s.io | K3S_URL=$K3S_URL K3S_TOKEN=$K3S_TOKEN sh -

The only problem I now have is that I can't connect to the Internet from the within the pods. It looks like a problem caused by coreDNS or Flannel as described in issue 5349 but it does not look like a problem with the outbound IP range of the AWS security group (I opened "All Traffic" and "All TCP" ports from 0 - 65535).

I've checked the following:

• ping 8.8.8.8 works
• ping google.com:80 does not work (bad address)
• ping google.com:443 does not work (bad address)
• curl google.com or wget https://google.com do not work either (Could not resolve host)

But using the above commands in kubectl run -i -t busybox --image=osexp2000/ubuntu-with-utils --overrides='{"kind":"Pod", "apiVersion":"v1", "spec": {"hostNetwork": true}}' --command -- sh - works

And the potential firewalls are disabled:

$ sudo systemctl status firewalld
Unit firewalld.service could not be found.

$ sudo ufw status
Status: inactive

Should I open a new issue or do you see some easy solution?

from k3s.

As I mentioned above, you've not opened the CNI ports. Also, the default flannel backend (vxlan) is not suitable for use over the internet; it is insecure and does not tolerate NAT well. I suspect that the vxlan CNI traffic between pods and the coredns service is being dropped, which is why your DNS lookups fail. See the docs at https://docs.k3s.io/networking/distributed-multicloud

from k3s.