IPv6 expansion of AS-NORDUNET consistently fails about bgpq3 HOT 6 CLOSED

Have you tried buffer size tuning on your host ? Looks like AS-NORDUNET is big enough
to not fit into 128k. From the documentation:

NOTES ON ULTRA-LARGE PREFIX-LISTS

When using bgpq3 to expand extra-large AS-SETs, bgpq3 may stuck due to lacking tcp buffer size. To avoid this, tune your OS. FreeBSD can be tuned in the following way:

sysctl -w net.inet.tcp.sendbuf_max=16777216
Linux can be tuned in the following way:

sysctl -w net.ipv4.tcp_window_scaling=1
sysctl -w net.core.rmem_max=16777216
sysctl -w net.core.wmem_max=16777216
sysctl -w net.ipv4.tcp_rmem="4096 87380 16777216"
sysctl -w net.ipv4.tcp_wmem="4096 65536 16777216"

PS: unable to reproduce on FreeBSD with net.inet.tcp.sendbuf_max: 2097152

from bgpq3.

Thanks. sysctl -w net.core.wmem_max=16777216 did indeed do the trick on my lab node (I didn't have to touch the other sysctls). That said, isn't this situation something that bgpq3 would ideally be able to avoid with the default settings? After all if you don't have super-user access on the machine you're running bgpq3 on you'll be unable to change any sysctls.

Another strange thing is that the IPv6 expansion of AS-NORDUNET only yields 2351 routes. The set of IPv4 routes is much, much larger (77809 routes), yet there was never any problems with that one.

from bgpq3.

On Thu, Mar 26, 2015 at 03:51:41AM -0700, Tore Anderson wrote:

Thanks. sysctl -w net.core.wmem_max=16777216 did indeed do the trick on my lab
node (I didn't have to touch the other sysctls. That said, isn't this situation
something that bgpq3 would ideally be able to avoid with the default settings?

bgpq3 can handle this situation with default settings. You just have to
disable request pipelining mode (-T switch) reverting bgpq3 to 'wait for
reply before sending next request' operations.

Not sure about your latency to RADB, here with my 128ms (pretty normal
for transeurope+transatlantic+15ms in USA) it makes bgpq3 painfully slow,
resolving AS-NORDUNET takes 12min without pipelining:

snar@fri:~/compile/bgpq3>time ./bgpq3 -T6 AS-NORDUNET
no ipv6 prefix-list NN
ipv6 prefix-list NN permit 2001::/32
[...]
real 12m43.395s
user 0m0.105s
sys 0m0.096s

snar@fri:~/compile/bgpq3>ping whois.radb.net
PING whois.radb.net (198.108.0.18): 56 data bytes
64 bytes from 198.108.0.18: icmp_seq=0 ttl=58 time=128.485 ms
64 bytes from 198.108.0.18: icmp_seq=1 ttl=58 time=128.574 ms
64 bytes from 198.108.0.18: icmp_seq=2 ttl=58 time=128.430 ms

(AS-NORDUNET is 5220 asn's 128ms each, even with ideal world
5220 * 0.128 = 668.160sec).

and that's why pipelining is turned on by default. Well, pipelining
implemented in the simplest way, by extending socket send buffer to
maximum allowed by OS and then just writing to that buffer, that's why
you have to tune OS. May be someday I'll add some 'internal buffering'
too, but as workaround is as simple as OS tuning - I do not think that
it will be implemented anytime soon.

After all if you don't have super-user access on the machine you're running
bgpq3 on you'll be unable to change any sysctls.

bgpq3 designed for network engineers, who either have their own netwokr
management servers or can easily talk sysadmins to increase any parameter
they want..

Another strange thing is that the IPv6 expansion of AS-NORDUNET only yields
2351 routes. The set of IPv4 routes is much, much larger (77809 routes), yet
there was never any problems with that one.

That's caused by query size: to expand IPv4 routes belonging to some as,
all bgpq3 sending to radb is '!gasNNNNN\n' (9-11 bytes), for IPv6 the same
query is '-T route6 -i origin asNNNNN\n' (29-31 bytes, three times more).

As a result, expanding AS-NORDUNET for IPv4 prefixes means 'sending 52k bytes
of query':

snar@fri:~/compile/bgpq3>./bgpq3 -d AS-NORDUNET 2>&1 | grep -- ': sending ' | sed "s/.*'//" | wc
5221 5221 52179

and for IPv6 it becomes 151k:

snar@fri:~/compile/bgpq3>./bgpq3 -d6 AS-NORDUNET 2>&1 | grep -- '-T route6' | sed "s/.*'-T/-T/" | wc
5220 26100 151343

(that suggests that you have send buffer of 128k by default, which is indeed
too small in modern internet).

from bgpq3.

Looks like bgpq3 doesn't perform pipelining correctly, i.e. it tries to send all commands first and only then reads the responses.

However, the server starts sending responses as soon as it processes the first pipelined commands and bgpq3 is not reading them - thus the server stops accepting further commands. Increasing of the buffer size just hides this bad behaviour.

To fix: bgpq3 must read responses from the network socket as soon as they start arriving, i.e. between writing commands it should poll the socket for incoming data and read them if present.

Pipelining requires bi-directional flow of data otherwise the server considers the socket to be dead.

from bgpq3.

On Sat, May 09, 2015 at 02:35:58PM -0700, hroudny wrote:

Looks like bgpq3 doesn't perform pipelining correcly, i.e. it tries to
send all comands first and only then reads the responses.

Yes, that's how it implemented. You can disable this behaviour by using
-T commandline switch.

However, the server starts sending responses as soon as it processes the first
pipelined commands and bgpq3 is not reading them - thus the server stops
accepting further commands. Increasing of the buffer size just hides this bad
behaviour.

Increasing buffer size means that bgpq3 is able to queue all the commands
(it takes microseconds) and is ready to read replies from RADB just in time
for the first reply to arrive: well, this may be not true if you are within
microseconds to RADB, but in most locations in Europe RTT to RADB are about
50-150 _milli_seconds:

snar@fri:~>ping whois.radb.net
PING whois.radb.net (198.108.0.18): 56 data bytes
64 bytes from 198.108.0.18: icmp_seq=0 ttl=58 time=128.978 ms
64 bytes from 198.108.0.18: icmp_seq=1 ttl=58 time=128.711 ms

(orders of magnitude more than it takes to buffer requests).

So, if bgpq3 have enough buffer size to queue all queries, it is ready
to receive replies before they arrive..

Ok, if bgpq3 have not enough buffer size - it will stuck in write'ing
requests, and, as it is indeed does not reading while writing, most
possible outcome is that it will fail.. That's why increasing buffer
size is the simplest workaround for this problem.
The other workaround, as stated above, is to disable pipelining using
-T switch.

To fix: bgpq3 must read responses from the network socket as soon as they
start arriving, i.e. between writing commands it should poll the socket
for incoming data.

You are welcome to implement this behaviour.

from bgpq3.

Increasing buffer size means that bgpq3 is able to queue all the commands
(it takes microseconds) and is ready to read replies from RADB just in time
for the first reply to arrive

If that's the case, then the fix is very simple: bgpq3 should create a temporary memory buffer using mmap(), write all the commands there, call fork() and let the child process copy the commands from memory buffer to socket, while parent starts reading responses and processing them.

from bgpq3.