Hairpin nat does not work about kube-router HOT 9 CLOSED

I've gotten this to work by adding a rule like this for every active service endpoint:

iptables -t nat -A POSTROUTING -s $ENDPOINT_IP -d $ENDPOINT_IP -m ipvs --vaddr $SVC_IP -j SNAT --to-source $SVC_IP

The result is that loopback traffic back to the pod is seen as coming from the service IP -- a possible drawback? Also, assuming lots and lots of services and/or endpoints, would the number of rules be too many?

If this looks reasonable I will submit a patch to implement it.

from kube-router.

@bzub seems reasonable solution.

"The result is that loopback traffic back to the pod is seen as coming from the service IP -- a possible drawback?"

Yeah, its has to be either node ip or service ip. Either of which will have problem with network policies if its permitting traffic only form certian set of pod IP's. But there is nothing much can be done.

"Assuming lots and lots of services and/or endpoints, would the number of rules be too many?"
We will only be adding rules for the pods running on the node. still it can lead to lots of rule.

We might need to check Kube-proxy how they are addressing this problem. Please take a look at kube-proxy. If there is no better alternative please go ahead with the PR.

from kube-router.

I believe kube-proxy and kubelet delegates this to the CNI plugin when CNI is used. I was able to get the "bridge" CNI plugin to set the sysctl hairpin_mode to 1 for all the veth devices, but that wasn't enough. I will see about what kubernetes does with "kubenet" mode which is the only other option other than CNI. That I believe sets hairpin-mode to "promiscuous bridge" but when I tested setting the kube-bridge interface to promiscuous mode it broke things. So I'll have to look further.

from kube-router.

@bzub yes, it seems its offloaded to CNI plugin. Not sure merit of this use case.

Please go ahed with your current fix, but keep flag '--hairpin-mode' like kubelet does (perhaps for kubenet). So if some one wants they can take any perf hit (if there is any) and enable this flag.

from kube-router.

OK, so the SNAT rule should only affect hairpin traffic going back to the endpoint that used its own service, so network policy issues would only affect that communication. I'm thinking I'll make a new flag like kube-router --hairpin-mode=true to enable the rule for all endpoints, and also allow the user to set a service annotation like service-proxy.kube-router.io/hairpin-mode=true if they only want this enabled on a subset of services.

from kube-router.

Yes please keep the flag 'kube-router --hairpin-mode=true' like i mentioned in above comment kubelet has this as well.

from kube-router.

Update: I am pursuing a solution I've tested that manipulates the network stack namespace within the pods that need hairpin. Being able to manipulate ip interfaces, routes, and even ipvs within pods is extremely powerful and could be a way to provide other features in kube-router. The biggest drawback is that we must access /proc on the host, which could be a security issue if kube-router pod is compromised. There are ways to minimize what accesses /proc. For example running a program/script dedicated to only exposing what we need from /proc and having kube-router consume from that.

For this issue (hairpin) the result will be that a pod traffic to its own service/port will never leave the pod, and the source IP will be preserved. This can be enhanced later to load-balance between all service endpoints for hairpin traffic.

Basic command that will be implemented per hairpin-enabled pod:

nsenter --net=/host/proc/$PID_OF_PAUSE_CONTAINER/ns/net iptables -t nat -D OUTPUT -d $SERVICE_IP -j REDIRECT

from kube-router.

@bzub, "manipulates the network stack namespace within the pods that need hairpin.", how do we know pods that need hairpin?

"This can be enhanced later to load-balance between all service endpoints for hairpin traffic."

I am not clear on this as well. Please elobrate.

from kube-router.

@bzub, "manipulates the network stack namespace within the pods that need hairpin.", how do we know pods that need hairpin?

We can still use a --hairpin-mode=true flag and then all pods that are service endpoints will get this iptables rule inside the pod. Or if the service has the annotation, same result for only those endpoints.

"This can be enhanced later to load-balance between all service endpoints for hairpin traffic."

I figured the best goal would be that hairpin traffic gets loadbalanced the same as outside client traffic. The current solution I have only gotten hairpin traffic to stay inside the pod, so it doesn't go to other endpoints. In the future it should balance between itself and other endpoints.

I will get a this implemented today most likely so we can try it out.

from kube-router.