Explore using IPVS direct routing mode about kube-router HOT 8 CLOSED

This is most efficient mode to use, but unfortunately there are constraints due to which its very non-trivial to achieve.

Please see http://www.linuxvirtualserver.org/VS-DRouting.html
https://www.centos.org/docs/5/html/Virtual_Server_Administration/s2-lvs-directrouting-VSA.html
http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/LVS-HOWTO.arp_problem.html

We need L2 adjacency between the IPVS/LVS director and real server. In the context of kubernetes it means each node need to have L2 adjacency to pods running on same or different nodes. As i understand we need to do some trickery (not sure even its possible) as container network is overlaid on node network.

Please use ipvsadm to try the scenario to see even its possible. I could not figure a way.

Also please see https://www.youtube.com/watch?v=KJ-A8LYriGI&t=204s, though it seems more like for north-south traffic.

Current container orchestration platforms (Docker swarm, Meso Minuteman) that uses IPVS does not support DR mode either. Please see if there are any efforts in those project to get DR mode.

from kube-router.

Adding a WIP design done by @thoro

masquerading with asymetric routing

from kube-router.

Elegent solution for DSR(Direct Server Return) in Microsoft Ananta

Take a look at Fig 7. It can be applied to kube-router as below.

Packet flow for the first packet

step1: client->service vip
step2: Node select an endpoint performs IP-IP encap with same source, and destination as pod IP
step3: Destination node intercept perform decapsulation send packet to pod
step4: pod gets the packet, respond directly to the client IP.

Each node to need to be aware of the flows and need to be consistent hashing for subsequent packets as detailed in the paper.

from kube-router.

Figure 2 of google's maglev they use GRE for DSR

from kube-router.

How Facebook does DSR with IP-IP 11.00 to 16.00 https://www.usenix.org/conference/srecon15europe/program/presentation/shuff

from kube-router.

https://www.slideshare.net/jschauma/l3dsr-overcoming-layer-2-limitations-of-direct-server-return-load-balancing

Use of DSCP to achive DSR.

from kube-router.

This post also describes Facebook's solution: https://www.facebook.com/notes/facebook-engineering/under-the-hood-network-implementation-for-world-ipv6-launch/10150873176303920/

from kube-router.

IPVS Configuration for FWMARK service with DSR

iptables:

iptables -t mangle -A PREROUTING -d 185.208.132.3 -m tcp -p tcp --dport 333 -j MARK --set-mark 5
iptables -t mangle -A OUTPUT -m mark --mark 5 -d 10.253.10.12 -j DSCP --set-dscp 3

ip rule add iif eth0 fwmark 0x5 lookup 100
ip route add local default dev lo table 100

<-- here it's necessary to put the ingress interface, or the fwmark is deleted on output, otherwise the packet is routed again to lo, and be careful with the route, to really add it to the table, otherwise you kill the IP connectivity ;)

ipvsadm:

ipvsadm -A --fwmark-service 5
ipvsadm -a --fwmark-service 5 -r 10.253.10.12:333 --masquerading

<-- here it's important that a port is defined, otherwise 0 is used and the packet is dropped

pod iptables:

iptables -A INPUT -m dscp --dscp 3 -j DSTWRITE --to-destination 185.208.132.3

if you leave out the two rules with dscp the current ipvsadm behaviour stays.

from kube-router.