Comments (2)
aganesh-suse
commented on May 27, 2024
Validated on release-1.28 branch with commit aa3a18b
Environment Details
Infrastructure
- Cloud
- Hosted
Node(s) CPU architecture, OS, and Version:
$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.2 LTS"
$ uname -m
x86_64
Cluster Configuration:
HA: 3 server/ 1 agent
Config.yaml:
token: xxxx
cluster-cidr: 2001:cafe:42:0::/56
service-cidr: 2001:cafe:43:0::/112
cluster-init: true
write-kubeconfig-mode: "0644"
node-label:
- k3s-upgrade=server
debug: true
Testing Steps
- Copy config.yaml
$ sudo mkdir -p /etc/rancher/k3s && sudo cp config.yaml /etc/rancher/k3s
- Install k3s
curl -sfL https://get.k3s.io | sudo INSTALL_K3S_COMMIT='aa3a18ba9b7a02de75b69412769e14e057661631' sh -s - server
- Verify Cluster Status:
kubectl get nodes -o wide
kubectl get pods -A
- Check node annotations for ipv6 entries first and not ipv4
kubectl get nodes -o yaml | grep 'etcd.k3s.cattle.io/node-address'
kubectl get nodes -o yaml | grep 'k3s.io/internal-ip'
kubectl get nodes -o yaml | grep 'flannel.alpha.coreos.com/public-ip'
- Check metrics-server pod logs for scraper.go lines - if there are failures.
POD_NAME=$(kubectl get pods -n kube-system | grep 'metrics-server' | awk '{ print $1 }')
kubectl logs pod/${POD_NAME} -n kube-system | grep scraper.go
Replication Results:
- k3s version used for replication:
$ k3s -v
k3s version v1.28.7+k3s1 (051b14b2)
go version go1.21.7
$ sudo /usr/local/bin/kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml get nodes -o yaml | grep 'etcd.k3s.cattle.io/node-address'
etcd.k3s.cattle.io/node-address: 192.x.x.x
$ sudo /usr/local/bin/kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml get nodes -o yaml | grep 'k3s.io/internal-ip'
k3s.io/internal-ip: 192.x.x.x,<ipv6_ip>
$ sudo /usr/local/bin/kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml get nodes -o yaml | grep 'flannel.alpha.coreos.com/public-ip'
flannel.alpha.coreos.com/public-ipv6: <ipv6_2>
$ sudo /usr/local/bin/kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml logs pod/metrics-server-67c658944b-447mj -n kube-system | grep scraper.go
E0313 21:38:28.029124 1 scraper.go:140] "Failed to scrape node" err="Get \"https://192.x.x.x:10250/metrics/resource\": dial tcp 192.x.x.x:10250: connect: network is unreachable" node="ip-192.x.x.x"
E0313 21:38:43.023757 1 scraper.go:140] "Failed to scrape node" err="Get \"https://192.x.x.x:10250/metrics/resource\": dial tcp 192.x.x.x:10250: connect: network is unreachable" node="ip-192.x.x.x"
Validation Results:
- k3s version used for validation:
$ k3s -v
k3s version v1.28.7+k3s-aa3a18ba (aa3a18ba)
go version go1.21.7
$ sudo /usr/local/bin/kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml get nodes -o yaml | grep 'etcd.k3s.cattle.io/node-address'
etcd.k3s.cattle.io/node-address: 192.x.x.x
etcd.k3s.cattle.io/node-address: 192.2.2.2
etcd.k3s.cattle.io/node-address: 192.3.3.3
$ sudo /usr/local/bin/kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml get nodes -o yaml | grep 'k3s.io/internal-ip'
k3s.io/internal-ip: <ipv6_1>,192.x.x.x
k3s.io/internal-ip: <ipv6_2>,192.2.2.2
k3s.io/internal-ip: <ipv6_3>,192.3.3.3
$ sudo /usr/local/bin/kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml get nodes -o yaml | grep 'flannel.alpha.coreos.com/public-ip'
flannel.alpha.coreos.com/public-ipv6: <ipv6_1>
flannel.alpha.coreos.com/public-ipv6: <ipv6_2>
flannel.alpha.coreos.com/public-ipv6: <ipv6_3>
$ sudo /usr/local/bin/kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml logs pod/metrics-server-54fd9b65b-xzpz7 -n kube-system | grep scraper.go
E0314 20:25:27.733232 1 scraper.go:147] "Failed to scrape node, timeout to access kubelet" err="Get \"https://[<ipv6_3>]:10250/metrics/resource\": context deadline exceeded" node="ip-192-x-x-x" timeout="10s"
E0314 20:25:42.738548 1 scraper.go:147] "Failed to scrape node, timeout to access kubelet" err="Get \"https://[<ipv6_3>]:10250/metrics/resource\": context deadline exceeded" node="ip-192-x-x-x" timeout="10s"
Metrics server still fails with the ipv6 address now.
from k3s.
aganesh-suse
commented on May 27, 2024
Validated on release-1.28 branch with commit aa3a18b
Environment Details
Infrastructure
- Cloud
- Hosted
Node(s) CPU architecture, OS, and Version:
$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.2 LTS"
$ uname -m
x86_64
Cluster Configuration:
HA: 3 server/ 1 agent
Config.yaml:
token: xxxx
cluster-cidr: 2001:cafe:42:0::/56
service-cidr: 2001:cafe:43:0::/112
flannel-ipv6-masq: true
cluster-init: true
write-kubeconfig-mode: "0644"
node-label:
- k3s-upgrade=server
debug: true
Testing Steps
- Copy config.yaml
$ sudo mkdir -p /etc/rancher/k3s && sudo cp config.yaml /etc/rancher/k3s
- Install k3s
curl -sfL https://get.k3s.io | sudo INSTALL_K3S_COMMIT='aa3a18ba9b7a02de75b69412769e14e057661631' sh -s - server
- Verify Cluster Status:
kubectl get nodes -o wide
kubectl get pods -A
- Check node annotations for ipv6 entries first and not ipv4
kubectl get nodes -o yaml | grep 'etcd.k3s.cattle.io/node-address'
kubectl get nodes -o yaml | grep 'k3s.io/internal-ip'
kubectl get nodes -o yaml | grep 'flannel.alpha.coreos.com/public-ip'
- Check metrics-server pod logs for scraper.go lines - if there are failures.
POD_NAME=$(kubectl get pods -n kube-system | grep 'metrics-server' | awk '{ print $1 }')
kubectl logs pod/${POD_NAME} -n kube-system | grep scraper.go
Validation Results:
- k3s version used for validation:
$ k3s -v
k3s version v1.28.7+k3s-aa3a18ba (aa3a18ba)
go version go1.21.7
$ sudo /usr/local/bin/kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml get nodes -o yaml | grep 'etcd.k3s.cattle.io/node-address'
etcd.k3s.cattle.io/node-address: 192.x.x.x
etcd.k3s.cattle.io/node-address: 192.2.2.2
etcd.k3s.cattle.io/node-address: 192.3.3.3
$ sudo /usr/local/bin/kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml get nodes -o yaml | grep 'k3s.io/internal-ip'
k3s.io/internal-ip: <ipv6_1>,192.x.x.x
k3s.io/internal-ip: <ipv6_2>,192.2.2.2
k3s.io/internal-ip: <ipv6_3>,192.3.3.3
k3s.io/internal-ip: <ipv6_4>,192.4.4.4
$ sudo /usr/local/bin/kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml get nodes -o yaml | grep 'flannel.alpha.coreos.com/public-ip'
flannel.alpha.coreos.com/public-ipv6: <ipv6_1>
flannel.alpha.coreos.com/public-ipv6: <ipv6_2>
flannel.alpha.coreos.com/public-ipv6: <ipv6_3>
flannel.alpha.coreos.com/public-ipv6: <ipv6_4>
$ sudo /usr/local/bin/kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml logs pod/metrics-server-54fd9b65b-xzpz7 -n kube-system | grep scraper.go
did not find any errors for metrics server. closing the issue.
from k3s.
Related Issues (20)
- K3s docker image is missing user info for root HOT 1
- [Release-1.28] - Unrecoverable error when joining node attempts to retrieve etcd member list from itself
- [Release-1.27] - Unrecoverable error when joining node attempts to retrieve etcd member list from itself
- [Release-1.28] - [cloud-provider] only pretend that instances from the current "cloud" exist HOT 1
- [Release-1.27] - [cloud-provider] only pretend that instances from the current "cloud" exist HOT 1
- [Release-1.28] - Errors logged when pulling images from text file HOT 1
- [Release-1.27] - Errors logged when pulling images from text file
- embedded registry for k3s agent.. HOT 2
- ICMPv6 Neighbour Solicitation packets dropped preventing IPv6 from working HOT 1
- Rename master node HOT 1
- [Release-1.28] - K3s should not disable etcd and apiserver when used with kine HOT 1
- [Release-1.27] - K3s should not disable etcd and apiserver when used with kine HOT 1
- Can't run `k3s server`
- [Release-1.28] - K3s docker image is missing user info for root HOT 1
- [Release-1.27] - K3s docker image is missing user info for root
- [Release-1.28] - etcd snapshot controller thrashes on etcdsnapshotfile management when server is run with `--disable-agent` HOT 1
- [Release-1.27] - etcd snapshot controller thrashes on etcdsnapshotfile management when server is run with `--disable-agent` HOT 1
- remove k3s node by node annotation? HOT 1
- Remove deprecated `"k8s.io/utils/pointer"` library HOT 1
- Reealuate the current pinned Golang dependencies
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from k3s.