Wrong source IP about kube-router HOT 3 CLOSED

I found below entry in my iptables

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
88949 5341K KUBE-POSTROUTING  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes postrouting rules */
    4   240 MASQUERADE  all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set kube-router-pod-subnets src ! match-set kube-router-pod-subnets dst ! match-set kube-router-node-ips dst random-fully
   14   784 SNAT       all  --  *      *      !10.0.13.0/24        !10.0.13.0/24         vdir ORIGINAL vmethod MASQ /*  */ to:167.235.70.254 random-fully
    0     0 SNAT       all  --  *      *      !10.0.13.0/24        !10.0.13.0/24         vdir ORIGINAL vmethod MASQ /*  */ to:10.103.0.4 random-fully

Why external interface is above internal? After clearing and restarting kube-router everything starts work as expected and iptables changes to original state

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
89065 5348K KUBE-POSTROUTING  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes postrouting rules */
    0     0 SNAT       all  --  *      *      !10.0.13.0/24        !10.0.13.0/24         vdir ORIGINAL vmethod MASQ /*  */ to:10.103.0.4 random-fully
    0     0 MASQUERADE  all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set kube-router-pod-subnets src ! match-set kube-router-pod-subnets dst ! match-set kube-router-node-ips dst random-fully

from kube-router.

Hmm... I've definitely not seen this before.

What I am able to tell you is the following:

• That IP address is only ever written to when kube-router is first starting and it first instantiates the NetworkServiceController struct:

  kube-router/pkg/controllers/proxy/network_services_controller.go

  Line 2348 in 1d1ff05

  NodeIP, err = utils.GetNodeIP(node)

• That means that in order for that to change, kube-router would have had to have been restarted, which isn't impossible if it were to be upgraded, or it failed a health-check, the node was rebooted, etc. but it certainly doesn't happen all the time. Maybe that's why you only see it sometimes happen this way?
• The way that it gets that address is from the node definition in Kubernetes. It looks at the node's status.addresses array and chooses the first type: InternalIP that it sees, and if it doesn't see one of that type, then it will choose the first type: ExternalIP
• Normally, this address structure is very stable, and is set by either the Kubelet or your Cloud Provider

I would guess that Hetzner or whatever you're using to orchestrate your Kubernetes cluster is occasionally mixing this structure on the node. I can say with confidence, that as long as your internal IP address, in this case 10.103.0.4, is consistently typed with InternalIP and your external IP is consistently typed with ExternalIP that you would not see this issue.

I would follow up with either your cloud provider or your Kubernetes orchestrator to see if you can get these to be labeled and arranged consistently. Maybe do a long-term watch on your node object to see what might be modifying it and when? You could also turn on audit logging in Kubernetes for node events as well to see what is changing this.

As an aside, the log message that you posted shows that you're running in a Kubernetes cluster with dual-stack enabled. kube-router is just barely beginning to support dual stack as of our v2.0.0 release candidate line and the --run-service-proxy option is still the last controller that we have not yet converted. Keep in mind that your mileage may vary widely if you use IPv6 in your cluster right now with kube-router, and I would highly recommend upgrading to one of the v2.0.0 release candidates as at least --run-firewall and --run-router are fixed to work with a dual-stack cluster.

from kube-router.

Thanks @aauren I could confirm that if kube-router starts too early, before cloud controller updates node object with correct external and internal IPs, it chooses external IP which breaks installation. Now I only need to find out how to delay kube-router start time. Is there a flag to wait until node is in initialized stage?

from kube-router.