Routing issue in IPv6-Only cluster about kube-router HOT 6 CLOSED

Unfortunately, I'm not able to reproduce this result on any of my clusters. I have tested with a cluster peering with a Juniper router that is controlling the next-hop routing. I have also tested with a cluster that is using FRR to peer with kube-router and controlling the Linux ip routing table much the way that bird is doing for you. Neither of them exhibit this issue.

If I had to guess, I would say that it is most likely an artifact of how bird is configured? If you can somehow dig into it more to provide more information, update this issue and I'll take a look again.

from kube-router.

Routing table on FRR node:

# ip -6 r show 
::1 dev lo proto kernel metric 256 pref medium
2001:db8:42:1000::/64 nhid 16 via 2600:1f18:477d:6900:e20c:7eee:81e1:9014 dev ens5 proto bgp metric 20 onlink pref medium
2001:db8:42:1001::/64 nhid 18 via 2600:1f18:477d:6900:e5e9:d1cd:2e60:6d11 dev ens5 proto bgp metric 20 onlink pref medium
2001:db8:42:1200:: nhid 32 proto bgp metric 20 pref medium
        nexthop via 2600:1f18:477d:6900:e20c:7eee:81e1:9014 dev ens5 weight 1 onlink 
        nexthop via 2600:1f18:477d:6900:e5e9:d1cd:2e60:6d11 dev ens5 weight 1 onlink 
2600:1f18:477d:6900::/64 dev ens5 proto ra metric 100 pref medium
fe80::/64 dev ens5 proto kernel metric 256 pref medium
default via fe80::8ff:f7ff:fe8a:e375 dev ens5 proto ra metric 100 expires 1798sec pref medium

mtr results:

# mtr -T -P 5000 2001:db8:42:1200:: -m 10 -c 1 --report
Start: 2024-05-07T15:30:13+0000
HOST: aws-bgp                     Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 2001:db8:42:1200::         0.0%     1    0.4   0.4   0.4   0.4   0.0

Relevant FRR details:

# vtysh -c "show bgp all"                              
...
   Network          Next Hop            Metric LocPrf Weight Path
*>i2001:db8:42:1000::/64
                    2600:1f18:477d:6900:e20c:7eee:81e1:9014
                                            10    100      0 i
*>i2001:db8:42:1001::/64
                    2600:1f18:477d:6900:e5e9:d1cd:2e60:6d11
                                            10    100      0 i
*>i2001:db8:42:1200::/128
                    2600:1f18:477d:6900:e20c:7eee:81e1:9014
                                            10    100      0 i
*=i                 2600:1f18:477d:6900:e5e9:d1cd:2e60:6d11
                                            10    100      0 i

Displayed  3 routes and 4 total paths

from kube-router.

Hey, thanks for your effort. I just tried to use Quagga BGP instead of Bird to rule this issue out, but I was just unable to get a session established between kube-router and Quagga, so I rather want to stay with Bird.

I don't really know where I can investigate further.
In my eyes everyone is sending packets to the router (as expected), the router routes the packets correctly to the nodes and the nodes don't know where to go with them so they throw them down the default route, back to the router.
So in my eyes the issue is simply, that the nodes do not have routes set up to handle the LoadBalancer/Service traffic. This is, at least in my head, completely unrelated to the BGP.

So maybe you can give me a hint in which direction I have to look now

from kube-router.

I noticed that you're running --service-proxy=false it is normally this functionality in kube-router that is responsible for routing packets for services. I would assume that if you're running that way then you have something like kube-proxy in the loop that is handling routing service traffic? Maybe something is wrong with kube-proxy or whatever is handling service proxy for you?

from kube-router.

Ohh no. I used the non-Service Proxy config in combination with removing the kube-proxy.
I enabled the service-proxy and while I still had issues out-of-the-box with the firewall blocking everything.
After adding a rule to accept all incoming traffic from the server's internal IP, everything seems to work.

So, I'm sorry for making such trouble and big thanks for your help!

from kube-router.

No worries. BTW in regards to your other mention of how workers without the workload were advertising VIPs I would recommend that you look into Kubernetes Traffic Policies: https://kubernetes.io/docs/reference/networking/virtual-ips/#traffic-policies

There is also a service.local annotation that kube-router provides for controlling this as well (https://www.kube-router.io/docs/user-guide/#controlling-service-locality-traffic-policies), but I think that its better and more portable to use the upstream service traffic policy definition.

However, either mechanism will allow you to control how kube-router advertises BGP VIPs.

from kube-router.