Running maz-nat in userspace about click HOT 9 CLOSED

Hi Murad,

This is most likely not a click bug, but segmentation offloading built into the NIC. You can use something like CheckLength -> Print to confirm. The driver aggregates packets resulting in packets >2000 bytes causing the invalid header checks and message too long errors. Its not helpful anyway for your router implementation.

Use ethtool to disable segmentation offloading e.g.

ethtool --offload eth0 tso off
ethtool --offload eth0 gso off
ethtool --offload eth0 gro off

from click.

Can you post an example click config? You probably need to add an IPClassifier to filter out ARP if the Host is responding or to divert those packets to Elements that can if the Host is not responding. You're right about there being differences between userland and linuxmodule, and one of those differences is the way ARP is handled. I'm surprised you didn't have to implement an ARPTable + ARPQuerier + ARPResponder before but without a click config it is hard to say.

from click.

Thanks for the reply. Attached is the config file in the second comment (I didn't know how to attach it here).
Some clarifications about the setup:
1- The machine running click has two interfaces br1 and br2 with each in a different network 10.15.0.0/24 and 10.17.0.0/24 respectively. Click machine is connected directly to the two end hosts. There is no gateway or extern-network as in original maz-nat.

2- Traffic should be forwarded from 10.15.0.0(internal network) to 10.17.0.0 (external network).

3- That said, EtherEncap(0x0800, b8:2a:72:dd:5a:c9, 00:30:1b:bd:6b:6f) in line 80 encapsulate src MAC with br2 MAC and dst MAC with the external machine running iperf -s.

4- tap0 is created by running
sudo ip tuntap add dev tap0 mode tap user murad
But it is not attached to anything. I wasn't sure to where I should attach it.

Thanks again,
Muad

from click.

// ADDRESS INFORMATION

AddressInfo(
intern 10.15.0.1 10.15.0.0/24 b8:2a:72:dd:5a:c8,
extern 10.17.0.1 10.17.0.0/24 b8:2a:72:dd:5a:c9,
intern_server 10.15.0.3
);

// DEVICE SETUP

elementclass GatewayDevice {
$device |
from :: FromDevice($device, SNIFFER false)
-> output;
input -> q :: Queue(1024)
-> to :: ToDevice($device);
ScheduleInfo(from .1, to 1);
}

FromHost(tap0) -> Discard;

//extern_dev :: GatewayDevice(extern:eth);
//intern_dev :: GatewayDevice(intern:eth);

extern_dev :: GatewayDevice(br2);
intern_dev :: GatewayDevice(br1);

ip_to_host :: EtherEncap(0x0800, 1:1:1:1:1:1, intern) // -> IPPrint(to_host)
-> ToHost(tap0);

// ARP MACHINERY

extern_arp_class, intern_arp_class
:: Classifier(12/0806 20/0001, 12/0806 20/0002, 12/0800, -);
intern_arpq :: ARPQuerier(intern);

extern_dev -> extern_arp_class;
extern_arp_class[0] -> ARPResponder(extern) // ARP queries
-> extern_dev;
extern_arp_class[1] -> ToHost(tap0); // ARP responses
extern_arp_class[3] -> Discard;

intern_dev -> intern_arp_class;
intern_arp_class[0] -> ARPResponder(intern) // ARP queries
-> intern_dev;
intern_arp_class[1] -> intern_arpr_t :: Tee;
intern_arpr_t[0] -> ToHost(tap0);
intern_arpr_t[1] -> [1]intern_arpq;
intern_arp_class[3] -> Discard;

// REWRITERS

IPRewriterPatterns(to_world_pat extern 65533 - -,
to_server_pat intern 50000-65535 intern_server -);

rw :: IPRewriter(// internal traffic to outside world
pattern to_world_pat 0 1,
// external traffic redirected to 'intern_server'
pattern to_server_pat 1 0,
// internal traffic redirected to 'intern_server'
pattern to_server_pat 1 1,
// virtual wire to output 0 if no mapping
pattern to_world_pat 0 1,
// virtual wire to output 2 if no mapping
pass 2);

tcp_rw :: TCPRewriter(// internal traffic to outside world
pattern to_world_pat 0 1,
// everything else is dropped
drop);

// OUTPUT PATH

ip_to_extern :: GetIPAddress(16)
-> CheckIPHeader
-> EtherEncap(0x0800, b8:2a:72:dd:5a:c9, 00:30:1b:bd:6b:6f)
//-> IPPrint(To-ExternDev-AfterCheck-and-Encap)
-> extern_dev;
ip_to_intern :: GetIPAddress(16)
-> CheckIPHeader
-> [0]intern_arpq
//-> IPPrint(To-InternDev-AfterCheck)
-> intern_dev;

// to outside world or gateway from inside network
rw[0] -> ip_to_extern_class :: IPClassifier(dst host intern, -);
ip_to_extern_class[0] -> ip_to_host;
ip_to_extern_class[1] -> ip_to_extern;
// to server
rw[1] -> ip_to_intern;
// only accept packets from outside world to gateway
rw[2] -> IPClassifier(dst host extern) //-> IPPrint(from-rw2-to-ip_to_host)
-> ip_to_host;

// tcp_rw is used only for FTP control traffic
tcp_rw[0] -> ip_to_extern;
tcp_rw[1] -> ip_to_intern;

// FILTER & REWRITE IP PACKETS FROM OUTSIDE

ip_from_extern :: IPClassifier(dst host extern,
-);
my_ip_from_extern :: IPClassifier(dst tcp ssh,
dst tcp www or https,
src tcp port ftp,
tcp or udp,
-);

extern_arp_class[2] -> Strip(14)
-> CheckIPHeader
-> ip_from_extern;
ip_from_extern[0] -> my_ip_from_extern;
my_ip_from_extern[0] -> [1]rw; // SSH traffic (rewrite to server)
my_ip_from_extern[1] -> [1]rw; // HTTP(S) traffic (rewrite to server)
my_ip_from_extern[2] -> [1]tcp_rw; // FTP control traffic, rewrite w/tcp_rw
my_ip_from_extern[3] -> [4]rw; // other TCP or UDP traffic, rewrite or to gw
my_ip_from_extern[4] -> Discard; // non TCP or UDP traffic is dropped
ip_from_extern[1] -> Discard; // stuff for other people

// FILTER & REWRITE IP PACKETS FROM INSIDE

ip_from_intern :: IPClassifier(dst host intern,
dst net intern,
dst tcp port ftp,
-);
my_ip_from_intern :: IPClassifier(dst tcp ssh,
dst tcp www or https,
src or dst port dns,
dst tcp port auth,
tcp or udp,
-);

intern_arp_class[2] -> Strip(14)
-> CheckIPHeader
-> ip_from_intern;
ip_from_intern[0] -> my_ip_from_intern; // stuff for 10.0.0.1 from inside
my_ip_from_intern[0] -> ip_to_host; // SSH traffic to gw
my_ip_from_intern[1] -> [2]rw; // HTTP(S) traffic, redirect to server instead
my_ip_from_intern[2] -> Discard; // DNS (no DNS allowed yet)
my_ip_from_intern[3] -> ip_to_host; // auth traffic, gw will reject it
my_ip_from_intern[4] -> [3]rw; // other TCP or UDP traffic, send to linux
// but pass it thru rw in case it is the
// returning redirect HTTP traffic from server
my_ip_from_intern[5] -> ip_to_host; // non TCP or UDP traffic, to linux
ip_from_intern[1] -> ip_to_host; // other net 10 stuff, like broadcasts
ip_from_intern[2] -> Print(FTP-pckt-to-be-discarded) //FTPPortMapper(tcp_rw, rw, 0)
-> [0]tcp_rw; // FTP traffic for outside needs special
// treatment
ip_from_intern[3] -> [0]rw; // stuff for outside

from click.

Updates
We have been working on running tcp iperf and cleaned up the config file to narrow down the problem. We can run tcp connection through Click NAT and stream small packets. I cleaned up the ARP handling and I can see that Click successfully responses to ARP guiries with the right MAC address.

We believe the problem raises when it comes to stream packers larger than 400 Bytes through Click (we tested with iperf, netperf, and i-dtg). Click in userspace seems to have a bug/unable to handle packets above 400 bytes and misses up checksum and IP header check. Resulting with packets have actual length that is different than what its stated in the header, thus resulting Click to drop the packets when its checks IP header and produce the previous error mentioned "CheckIPHeader@37: IP header check failed: bad IP length".

We tried to change the behavior to not drop the packets even if the checksome failed. But ToDevice element did not like the packet when it arrived to it and produced "Message too long" error.

from click.

Thanks for your reply,
Just to clarify, when you say "driver aggregates packets". Do you mean at the sender machine (generator of traffic)? or the machine running click?

The two end hosts connected to machine running click in a small setup with 1G links. With UDP we can get maximum of 812Mbit throughput. Do you have an estimate what it the maximum BW we can achieve with Click in userspace with the above issue in NIC?

Thanks again.

from click.

On the interfaces used in the click config.

The segmentation offloading will not severely affect throughput, but once you can process 1500 byte packets, the overhead per packet will be less and throughput will be closer to 1Gbps.

from click.

Awesome!
I made the changes you mentioned above with ethtool and I can get 896Mbit.

Thank you so much :)
Murad

from click.

Great, pleasure.

from click.