Comments (9)
ctz
commented on May 26, 2024
1
Due to io-uring nature packets order is not guaranteed
This would imply that io-uring doesn't work for TCP (an ordered byte stream) which is certainly not true.
Could you elaborate on short writes?
See the docs for write(2):
The number of bytes written may be less than count if, for example, there is insufficient space on the underlying physical medium
In the case of TCP, the "underlying physical medium" will be the peer's receive window. This can happen irrespective of whether you submit the system call directly or via io_uring.
All problems started when i begin to send >64kb of data at once
Very likely you are writing >64KB, which succeeds and tells you <64KB was written. Now your application must: retain the unwritten data, wait for the socket to become writable (either because the peer's window has cleared, or grown), and then write the remainder.
Note that 64KB is the maximum window size if window scaling hasn't been negotiated.
from rustls.
ctz
commented on May 26, 2024
Could you attach the pcap? Is this the same as #1710 or a different issue?
from rustls.
serzhiio
commented on May 26, 2024
Could you attach the pcap?
I could, but not sure i want to share IP addresses
Is this the same as #1710 or a different issue?
I think, yes, same
from rustls.
ctz
commented on May 26, 2024
Could you repro it in a test network with RFC1918 addresses, or otherwise post a repro code sample?
But I'm pretty sure that somewhere you will be mishandling short writes, and therefore dropping data.
from rustls.
serzhiio
commented on May 26, 2024
Could you repro it in a test network with RFC1918 addresses...
This is very hard as this server is working with 30+ QUIC clients and several WS UI clients. One of WS client under Fedora Linux is started to show this errors recently, other clients is working.
...or otherwise post a repro code sample?
Will try!
But I'm pretty sure that somewhere you will be mishandling short writes, and therefore dropping data.
The question worrying me is why other clients working with same server data normally within same network while this specific client is not.
Could you elaborate on short writes?
And here is log (with my comments) from server side when affected client trying to connect:
[2024-01-25T07:15:18.224Z INFO server::app::ui] UI:0 Hello!
[2024-01-25T07:15:18.225Z TRACE tcp_req] Sending data(85775b) to client:0... <- 85775b of plaintext data is ready
[2024-01-25T07:15:18.225Z TRACE tcp_uni] * process data(65652bytes) for sending | Identity(type:ServerTcpClient,flag:3,slave:6,master:0), zero_copy:false <- TLS data written to internal buffer
[2024-01-25T07:15:18.225Z TRACE tcp_uni] * process data(20297bytes) for sending | Identity(type:ServerTcpClient,flag:3,slave:7,master:0), zero_copy:false <- TLS data written to internal buffer
[2024-01-25T07:15:18.226Z TRACE tcp] TCP incoming_data Identity(type:ServerTcpClient,flag:3,slave:6,master:0), res=65652 | is_more: false <-- all 65652bytes has been sent, this kind of data means io_uring send result of data send
[2024-01-25T07:15:18.226Z TRACE tcp] TCP incoming_data Identity(type:ServerTcpClient,flag:3,slave:7,master:0), res=11348 | is_more: false <-- only 11348/20297 bytes has been sent
___ total of 77000 bytes were sent (using two buffers) out of required 85775b ?
[2024-01-25T07:15:19.225Z TRACE tcp] TCP incoming_data Identity(type:ServerTcpClient,flag:0,slave:0,master:0), res=44 | is_more: true <- this is recv
[2024-01-25T07:15:19.225Z INFO server::app::ui] UI:0 First ping recvd!
[2024-01-25T07:15:19.225Z TRACE tcp_req] Sending data(11b) to client:0...
[2024-01-25T07:15:19.225Z TRACE tcp_uni] * process data(40bytes) for sending | Identity(type:ServerTcpClient,flag:3,slave:9,master:0), zero_copy:false
<- 40b sent
[2024-01-25T07:15:19.225Z TRACE tcp] TCP incoming_data Identity(type:ServerTcpClient,flag:3,slave:9,master:0), res=40 | is_more: false
<- 40 ok
[2024-01-25T07:15:19.236Z TRACE tcp_req] Sending data(983b) to client:0...
[2024-01-25T07:15:19.236Z TRACE tcp_uni] * process data(1012bytes) for sending | Identity(type:ServerTcpClient,flag:3,slave:11,master:0), zero_copy:false
<- 1012b sent
[2024-01-25T07:15:19.236Z TRACE tcp] TCP incoming_data Identity(type:ServerTcpClient,flag:3,slave:11,master:0), res=1012 | is_more: false
<- 1012 ok
[2024-01-25T07:15:19.250Z TRACE tcp_req] Sending data(989b) to client:0...
[2024-01-25T07:15:19.250Z TRACE tcp_uni] * process data(1018bytes) for sending | Identity(type:ServerTcpClient,flag:3,slave:13,master:0), zero_copy:false
[2024-01-25T07:15:19.236Z TRACE tcp] TCP incoming_data Identity(type:ServerTcpClient,flag:3,slave:11,master:0), res=1012 | is_more: false
<- 1012/1018b sent ?
[2024-01-25T07:15:19.250Z TRACE tcp_req] Sending data(989b) to client:0...
[2024-01-25T07:15:19.250Z TRACE tcp_uni] * process data(1018bytes) for sending | Identity(type:ServerTcpClient,flag:3,slave:13,master:0), zero_copy:false
<- 1018b sent
[2024-01-25T07:15:19.250Z TRACE tcp] TCP incoming_data Identity(type:ServerTcpClient,flag:3,slave:13,master:0), res=1018 | is_more: false
<- 1018 ok
Serverside's tcpdump log(with comments) on affected client connect:
tcpdump: data link type LINUX_SLL2
tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
07:15:17.378196 enp3s0f0 In IP (tos 0x0, ttl 50, id 53177, offset 0, flags [DF], proto TCP (6), length 60) 94.204.80.81.58396 > 51.195.104.xxx.54475: Flags [S], cksum 0xba3c (correct), seq 3270659376, win 21900, options [mss 1412,sackOK,TS val 1338168314 ecr 0,nop,wscale 7], length 0
07:15:17.378244 enp3s0f0 Out IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60) 51.195.104.xxx.54475 > 94.204.80.81.58396: Flags [S.], cksum 0x4bcd (incorrect -> 0x680f), seq 2681053197, ack 3270659377, win 43440, options [mss 1460,sackOK,TS val 2232039649 ecr 1338168314,nop,wscale 8], length 0
07:15:17.511970 enp3s0f0 In IP (tos 0x0, ttl 50, id 53178, offset 0, flags [DF], proto TCP (6), length 52) 94.204.80.81.58396 > 51.195.104.xxx.54475: Flags [.], cksum 0x3f5b (correct), ack 1, win 172, options [nop,nop,TS val 1338168448 ecr 2232039649], length 0
07:15:17.512269 enp3s0f0 In IP (tos 0x0, ttl 50, id 53179, offset 0, flags [DF], proto TCP (6), length 569) 94.204.80.81.58396 > 51.195.104.xxx.54475: Flags [P.], cksum 0x9169 (correct), seq 1:518, ack 1, win 172, options [nop,nop,TS val 1338168449 ecr 2232039649], length 517
07:15:17.512294 enp3s0f0 Out IP (tos 0x0, ttl 64, id 4328, offset 0, flags [DF], proto TCP (6), length 52) 51.195.104.xxx.54475 > 94.204.80.81.58396: Flags [.], cksum 0x4bc5 (incorrect -> 0x3cd3), ack 518, win 168, options [nop,nop,TS val 2232039783 ecr 1338168449], length 0
07:15:17.515153 enp3s0f0 Out IP (tos 0x0, ttl 64, id 4329, offset 0, flags [DF], proto TCP (6), length 2248) 51.195.104.xxx.54475 > 94.204.80.81.58396: Flags [P.], cksum 0x5459 (incorrect -> 0x7e4e), seq 1:2197, ack 518, win 168, options [nop,nop,TS val 2232039785 ecr 1338168449], length 2196
07:15:17.649163 enp3s0f0 In IP (tos 0x0, ttl 50, id 53180, offset 0, flags [DF], proto TCP (6), length 52) 94.204.80.81.58396 > 51.195.104.xxx.54475: Flags [.], cksum 0x33ba (correct), ack 2197, win 163, options [nop,nop,TS val 1338168585 ecr 2232039785], length 0
07:15:17.649164 enp3s0f0 In IP (tos 0x0, ttl 50, id 53181, offset 0, flags [DF], proto TCP (6), length 145) 94.204.80.81.58396 > 51.195.104.xxx.54475: Flags [P.], cksum 0xf046 (correct), seq 518:611, ack 2197, win 163, options [nop,nop,TS val 1338168586 ecr 2232039785], length 93
07:15:17.649613 enp3s0f0 Out IP (tos 0x0, ttl 64, id 4331, offset 0, flags [DF], proto TCP (6), length 103) 51.195.104.xxx.54475 > 94.204.80.81.58396: Flags [P.], cksum 0x4bf8 (incorrect -> 0xceee), seq 2197:2248, ack 611, win 168, options [nop,nop,TS val 2232039920 ecr 1338168586], length 51
07:15:17.783227 enp3s0f0 In IP (tos 0x0, ttl 50, id 53182, offset 0, flags [DF], proto TCP (6), length 625) 94.204.80.81.58396 > 51.195.104.xxx.54475: Flags [P.], cksum 0x077e (correct), seq 611:1184, ack 2248, win 163, options [nop,nop,TS val 1338168720 ecr 2232039920], length 573
07:15:17.783571 enp3s0f0 Out IP (tos 0x0, ttl 64, id 4332, offset 0, flags [DF], proto TCP (6), length 210) 51.195.104.xxx.54475 > 94.204.80.81.58396: Flags [P.], cksum 0x4c63 (incorrect -> 0x5610), seq 2248:2406, ack 1184, win 166, options [nop,nop,TS val 2232040054 ecr 1338168720], length 158
07:15:17.957998 enp3s0f0 In IP (tos 0x0, ttl 50, id 53183, offset 0, flags [DF], proto TCP (6), length 52) 94.204.80.81.58396 > 51.195.104.xxx.54475: Flags [.], cksum 0x2e0c (correct), ack 2406, win 163, options [nop,nop,TS val 1338168895 ecr 2232040054], length 0
07:15:18.224132 enp3s0f0 In IP (tos 0x0, ttl 50, id 53184, offset 0, flags [DF], proto TCP (6), length 88) 94.204.80.81.58396 > 51.195.104.xxx.54475: Flags [P.], cksum 0xd8df (correct), seq 1184:1220, ack 2406, win 163, options [nop,nop,TS val 1338169161 ecr 2232040054], length 36
07:15:18.226019 enp3s0f0 Out IP (tos 0x0, ttl 64, id 4333, offset 0, flags [DF], proto TCP (6), length 2852) 51.195.104.xxx.54475 > 94.204.80.81.58396: Flags [P.], cksum 0x56b5 (incorrect -> 0xd96c), seq 2406:5206, ack 1220, win 166, options [nop,nop,TS val 2232040496 ecr 1338169161], length 2800
07:15:18.226036 enp3s0f0 Out IP (tos 0x0, ttl 64, id 4335, offset 0, flags [DF], proto TCP (6), length 2852) 51.195.104.xxx.54475 > 94.204.80.81.58396: Flags [P.], cksum 0x56b5 (incorrect -> 0x8541), seq 5206:8006, ack 1220, win 166, options [nop,nop,TS val 2232040496 ecr 1338169161], length 2800
07:15:18.226071 enp3s0f0 Out IP (tos 0x0, ttl 64, id 4337, offset 0, flags [DF], proto TCP (6), length 2852) 51.195.104.xxx.54475 > 94.204.80.81.58396: Flags [P.], cksum 0x56b5 (incorrect -> 0xe8fe), seq 8006:10806, ack 1220, win 166, options [nop,nop,TS val 2232040496 ecr 1338169161], length 2800
07:15:18.226080 enp3s0f0 Out IP (tos 0x0, ttl 64, id 4339, offset 0, flags [DF], proto TCP (6), length 2852) 51.195.104.xxx.54475 > 94.204.80.81.58396: Flags [P.], cksum 0x56b5 (incorrect -> 0xa82f), seq 10806:13606, ack 1220, win 166, options [nop,nop,TS val 2232040496 ecr 1338169161], length 2800
07:15:18.226102 enp3s0f0 Out IP (tos 0x0, ttl 64, id 4341, offset 0, flags [DF], proto TCP (6), length 2852) 51.195.104.xxx.54475 > 94.204.80.81.58396: Flags [P.], cksum 0x56b5 (incorrect -> 0xb522), seq 13606:16406, ack 1220, win 166, options [nop,nop,TS val 2232040496 ecr 1338169161], length 2800 <- 14000 bytes sent
07:15:18.359021 enp3s0f0 In IP (tos 0x0, ttl 50, id 53185, offset 0, flags [DF], proto TCP (6), length 52) 94.204.80.81.58396 > 51.195.104.xxx.54475: Flags [.], cksum 0x1fad (correct), ack 5206, win 163, options [nop,nop,TS val 1338169296 ecr 2232040496], length 0
07:15:18.359054 enp3s0f0 Out IP (tos 0x0, ttl 64, id 4343, offset 0, flags [DF], proto TCP (6), length 2852) 51.195.104.xxx.54475 > 94.204.80.81.58396: Flags [P.], cksum 0x56b5 (incorrect -> 0x263e), seq 16406:19206, ack 1220, win 166, options [nop,nop,TS val 2232040629 ecr 1338169296], length 2800
07:15:18.359065 enp3s0f0 Out IP (tos 0x0, ttl 64, id 4345, offset 0, flags [DF], proto TCP (6), length 1452) 51.195.104.xxx.54475 > 94.204.80.81.58396: Flags [P.], cksum 0x513d (incorrect -> 0x7b2b), seq 19206:20606, ack 1220, win 166, options [nop,nop,TS val 2232040629 ecr 1338169296], length 1400 <- 18200(+4200) bytes sent
07:15:18.359086 enp3s0f0 In IP (tos 0x0, ttl 50, id 53186, offset 0, flags [DF], proto TCP (6), length 52) 94.204.80.81.58396 > 51.195.104.xxx.54475: Flags [.], cksum 0x14bd (correct), ack 8006, win 163, options [nop,nop,TS val 1338169296 ecr 2232040496], length 0
07:15:18.359087 enp3s0f0 In IP (tos 0x0, ttl 50, id 53187, offset 0, flags [DF], proto TCP (6), length 52) 94.204.80.81.58396 > 51.195.104.xxx.54475: Flags [.], cksum 0x09cd (correct), ack 10806, win 163, options [nop,nop,TS val 1338169296 ecr 2232040496], length 0
07:15:18.359088 enp3s0f0 In IP (tos 0x0, ttl 50, id 53188, offset 0, flags [DF], proto TCP (6), length 52) 94.204.80.81.58396 > 51.195.104.xxx.54475: Flags [.], cksum 0xfedc (correct), ack 13606, win 163, options [nop,nop,TS val 1338169296 ecr 2232040496], length 0
07:15:18.359089 enp3s0f0 In IP (tos 0x0, ttl 50, id 53189, offset 0, flags [DF], proto TCP (6), length 52) 94.204.80.81.58396 > 51.195.104.xxx.54475: Flags [.], cksum 0xf3ec (correct), ack 16406, win 163, options [nop,nop,TS val 1338169296 ecr 2232040496], length 0
07:15:18.359112 enp3s0f0 Out IP (tos 0x0, ttl 64, id 4346, offset 0, flags [DF], proto TCP (6), length 2852) 51.195.104.xxx.54475 > 94.204.80.81.58396: Flags [P.], cksum 0x56b5 (incorrect -> 0xa6c6), seq 20606:23406, ack 1220, win 166, options [nop,nop,TS val 2232040629 ecr 1338169296], length 2800
07:15:18.359122 enp3s0f0 Out IP (tos 0x0, ttl 64, id 4348, offset 0, flags [DF], proto TCP (6), length 2852) 51.195.104.xxx.54475 > 94.204.80.81.58396: Flags [P.], cksum 0x56b5 (incorrect -> 0x3579), seq 23406:26206, ack 1220, win 166, options [nop,nop,TS val 2232040629 ecr 1338169296], length 2800
07:15:18.359138 enp3s0f0 Out IP (tos 0x0, ttl 64, id 4350, offset 0, flags [DF], proto TCP (6), length 2852) 51.195.104.xxx.54475 > 94.204.80.81.58396: Flags [P.], cksum 0x56b5 (incorrect -> 0xa7a8), seq 26206:29006, ack 1220, win 166, options [nop,nop,TS val 2232040629 ecr 1338169296], length 2800
07:15:18.359148 enp3s0f0 Out IP (tos 0x0, ttl 64, id 4352, offset 0, flags [DF], proto TCP (6), length 1452) 51.195.104.xxx.54475 > 94.204.80.81.58396: Flags [P.], cksum 0x513d (incorrect -> 0xc4ad), seq 29006:30406, ack 1220, win 166, options [nop,nop,TS val 2232040629 ecr 1338169296], length 1400
07:15:18.359164 enp3s0f0 Out IP (tos 0x0, ttl 64, id 4353, offset 0, flags [DF], proto TCP (6), length 2852) 51.195.104.xxx.54475 > 94.204.80.81.58396: Flags [P.], cksum 0x56b5 (incorrect -> 0x8250), seq 30406:33206, ack 1220, win 166, options [nop,nop,TS val 2232040629 ecr 1338169296], length 2800
07:15:18.359179 enp3s0f0 Out IP (tos 0x0, ttl 64, id 4355, offset 0, flags [DF], proto TCP (6), length 2852) 51.195.104.xxx.54475 > 94.204.80.81.58396: Flags [P.], cksum 0x56b5 (incorrect -> 0xc8ad), seq 33206:36006, ack 1220, win 166, options [nop,nop,TS val 2232040629 ecr 1338169296], length 2800
<- 33600(+15400) bytes sent
07:15:18.492124 enp3s0f0 In IP (tos 0x0, ttl 50, id 53190, offset 0, flags [DF], proto TCP (6), length 52) 94.204.80.81.58396 > 51.195.104.xxx.54475: Flags [.], cksum 0xe7f2 (correct), ack 19206, win 163, options [nop,nop,TS val 1338169429 ecr 2232040629], length 0
07:15:18.492154 enp3s0f0 Out IP (tos 0x0, ttl 64, id 4357, offset 0, flags [DF], proto TCP (6), length 2852) 51.195.104.xxx.54475 > 94.204.80.81.58396: Flags [P.], cksum 0x56b5 (incorrect -> 0xf544), seq 36006:38806, ack 1220, win 166, options [nop,nop,TS val 2232040762 ecr 1338169429], length 2800
<- 36400(+2800) bytes sent
07:15:18.492986 enp3s0f0 In IP (tos 0x0, ttl 50, id 53191, offset 0, flags [DF], proto TCP (6), length 52) 94.204.80.81.58396 > 51.195.104.xxx.54475: Flags [.], cksum 0xe27a (correct), ack 20606, win 163, options [nop,nop,TS val 1338169429 ecr 2232040629], length 0
07:15:18.492987 enp3s0f0 In IP (tos 0x0, ttl 50, id 53192, offset 0, flags [DF], proto TCP (6), length 52) 94.204.80.81.58396 > 51.195.104.xxx.54475: Flags [.], cksum 0xd78a (correct), ack 23406, win 163, options [nop,nop,TS val 1338169429 ecr 2232040629], length 0
07:15:18.492988 enp3s0f0 In IP (tos 0x0, ttl 50, id 53193, offset 0, flags [DF], proto TCP (6), length 52) 94.204.80.81.58396 > 51.195.104.xxx.54475: Flags [.], cksum 0xcc9a (correct), ack 26206, win 163, options [nop,nop,TS val 1338169429 ecr 2232040629], length 0
07:15:18.492989 enp3s0f0 In IP (tos 0x0, ttl 50, id 53194, offset 0, flags [DF], proto TCP (6), length 52) 94.204.80.81.58396 > 51.195.104.xxx.54475: Flags [.], cksum 0xc1aa (correct), ack 29006, win 163, options [nop,nop,TS val 1338169429 ecr 2232040629], length 0
07:15:18.492990 enp3s0f0 In IP (tos 0x0, ttl 50, id 53196, offset 0, flags [DF], proto TCP (6), length 52) 94.204.80.81.58396 > 51.195.104.xxx.54475: Flags [.], cksum 0xa652 (correct), ack 36006, win 163, options [nop,nop,TS val 1338169429 ecr 2232040629], length 0
07:15:18.492991 enp3s0f0 In IP (tos 0x0, ttl 50, id 53195, offset 0, flags [DF], proto TCP (6), length 52) 94.204.80.81.58396 > 51.195.104.xxx.54475: Flags [.], cksum 0xb142 (correct), ack 33206, win 163, options [nop,nop,TS val 1338169429 ecr 2232040629], length 0
07:15:18.493013 enp3s0f0 Out IP (tos 0x0, ttl 64, id 4359, offset 0, flags [DF], proto TCP (6), length 1452) 51.195.104.xxx.54475 > 94.204.80.81.58396: Flags [.], cksum 0x513d (incorrect -> 0x5d6e), seq 38806:40206, ack 1220, win 166, options [nop,nop,TS val 2232040763 ecr 1338169429], length 1400
07:15:18.493030 enp3s0f0 Out IP (tos 0x0, ttl 64, id 4360, offset 0, flags [DF], proto TCP (6), length 2852) 51.195.104.xxx.54475 > 94.204.80.81.58396: Flags [P.], cksum 0x56b5 (incorrect -> 0x70db), seq 40206:43006, ack 1220, win 166, options [nop,nop,TS val 2232040763 ecr 1338169429], length 2800
07:15:18.493045 enp3s0f0 Out IP (tos 0x0, ttl 64, id 4362, offset 0, flags [DF], proto TCP (6), length 2852) 51.195.104.xxx.54475 > 94.204.80.81.58396: Flags [P.], cksum 0x56b5 (incorrect -> 0xfe02), seq 43006:45806, ack 1220, win 166, options [nop,nop,TS val 2232040763 ecr 1338169429], length 2800
07:15:18.493060 enp3s0f0 Out IP (tos 0x0, ttl 64, id 4364, offset 0, flags [DF], proto TCP (6), length 2852) 51.195.104.xxx.54475 > 94.204.80.81.58396: Flags [P.], cksum 0x56b5 (incorrect -> 0xa73e), seq 45806:48606, ack 1220, win 166, options [nop,nop,TS val 2232040763 ecr 1338169429], length 2800
07:15:18.493078 enp3s0f0 Out IP (tos 0x0, ttl 64, id 4366, offset 0, flags [DF], proto TCP (6), length 1452) 51.195.104.xxx.54475 > 94.204.80.81.58396: Flags [P.], cksum 0x513d (incorrect -> 0xcb4c), seq 48606:50006, ack 1220, win 166, options [nop,nop,TS val 2232040763 ecr 1338169429], length 1400
07:15:18.493088 enp3s0f0 Out IP (tos 0x0, ttl 64, id 4367, offset 0, flags [DF], proto TCP (6), length 2852) 51.195.104.xxx.54475 > 94.204.80.81.58396: Flags [P.], cksum 0x56b5 (incorrect -> 0xb363), seq 50006:52806, ack 1220, win 166, options [nop,nop,TS val 2232040763 ecr 1338169429], length 2800
07:15:18.493096 enp3s0f0 Out IP (tos 0x0, ttl 64, id 4369, offset 0, flags [DF], proto TCP (6), length 2852) 51.195.104.xxx.54475 > 94.204.80.81.58396: Flags [P.], cksum 0x56b5 (incorrect -> 0x87ba), seq 52806:55606, ack 1220, win 166, options [nop,nop,TS val 2232040763 ecr 1338169429], length 2800
<- 53200(+16800) bytes sent
07:15:18.626286 enp3s0f0 In IP (tos 0x0, ttl 50, id 53197, offset 0, flags [DF], proto TCP (6), length 52) 94.204.80.81.58396 > 51.195.104.xxx.54475: Flags [.], cksum 0x9a58 (correct), ack 38806, win 163, options [nop,nop,TS val 1338169562 ecr 2232040762], length 0
07:15:18.626288 enp3s0f0 In IP (tos 0x0, ttl 50, id 53198, offset 0, flags [DF], proto TCP (6), length 52) 94.204.80.81.58396 > 51.195.104.xxx.54475: Flags [.], cksum 0x89ad (correct), ack 43006, win 228, options [nop,nop,TS val 1338169563 ecr 2232040763], length 0
07:15:18.626289 enp3s0f0 In IP (tos 0x0, ttl 50, id 53199, offset 0, flags [DF], proto TCP (6), length 52) 94.204.80.81.58396 > 51.195.104.xxx.54475: Flags [.], cksum 0x7e91 (correct), ack 45806, win 272, options [nop,nop,TS val 1338169563 ecr 2232040763], length 0
07:15:18.626290 enp3s0f0 In IP (tos 0x0, ttl 50, id 53200, offset 0, flags [DF], proto TCP (6), length 52) 94.204.80.81.58396 > 51.195.104.xxx.54475: Flags [.], cksum 0x7375 (correct), ack 48606, win 316, options [nop,nop,TS val 1338169563 ecr 2232040763], length 0
07:15:18.626291 enp3s0f0 In IP (tos 0x0, ttl 50, id 53201, offset 0, flags [DF], proto TCP (6), length 52) 94.204.80.81.58396 > 51.195.104.xxx.54475: Flags [.], cksum 0x6de7 (correct), ack 50006, win 338, options [nop,nop,TS val 1338169563 ecr 2232040763], length 0
07:15:18.626291 enp3s0f0 In IP (tos 0x0, ttl 50, id 53202, offset 0, flags [DF], proto TCP (6), length 52) 94.204.80.81.58396 > 51.195.104.xxx.54475: Flags [.], cksum 0x62cb (correct), ack 52806, win 382, options [nop,nop,TS val 1338169563 ecr 2232040763], length 0
07:15:18.626292 enp3s0f0 In IP (tos 0x0, ttl 50, id 53203, offset 0, flags [DF], proto TCP (6), length 52) 94.204.80.81.58396 > 51.195.104.xxx.54475: Flags [.], cksum 0x57af (correct), ack 55606, win 426, options [nop,nop,TS val 1338169563 ecr 2232040763], length 0
07:15:18.626329 enp3s0f0 Out IP (tos 0x0, ttl 64, id 4371, offset 0, flags [DF], proto TCP (6), length 2852) 51.195.104.xxx.54475 > 94.204.80.81.58396: Flags [P.], cksum 0x56b5 (incorrect -> 0xe57c), seq 55606:58406, ack 1220, win 166, options [nop,nop,TS val 2232040897 ecr 1338169562], length 2800
07:15:18.626349 enp3s0f0 Out IP (tos 0x0, ttl 64, id 4373, offset 0, flags [DF], proto TCP (6), length 1452) 51.195.104.xxx.54475 > 94.204.80.81.58396: Flags [.], cksum 0x513d (incorrect -> 0x1a1d), seq 58406:59806, ack 1220, win 166, options [nop,nop,TS val 2232040897 ecr 1338169563], length 1400
07:15:18.626359 enp3s0f0 Out IP (tos 0x0, ttl 64, id 4374, offset 0, flags [DF], proto TCP (6), length 2852) 51.195.104.xxx.54475 > 94.204.80.81.58396: Flags [P.], cksum 0x56b5 (incorrect -> 0xc61a), seq 59806:62606, ack 1220, win 166, options [nop,nop,TS val 2232040897 ecr 1338169563], length 2800
07:15:18.626367 enp3s0f0 Out IP (tos 0x0, ttl 64, id 4376, offset 0, flags [DF], proto TCP (6), length 2852) 51.195.104.xxx.54475 > 94.204.80.81.58396: Flags [P.], cksum 0x56b5 (incorrect -> 0x328e), seq 62606:65406, ack 1220, win 166, options [nop,nop,TS val 2232040897 ecr 1338169563], length 2800
07:15:18.626428 enp3s0f0 Out IP (tos 0x0, ttl 64, id 4378, offset 0, flags [DF], proto TCP (6), length 2852) 51.195.104.xxx.54475 > 94.204.80.81.58396: Flags [P.], cksum 0x56b5 (incorrect -> 0xb059), seq 65406:68206, ack 1220, win 166, options [nop,nop,TS val 2232040897 ecr 1338169563], length 2800
07:15:18.626504 enp3s0f0 Out IP (tos 0x0, ttl 64, id 4380, offset 0, flags [DF], proto TCP (6), length 1452) 51.195.104.xxx.54475 > 94.204.80.81.58396: Flags [.], cksum 0x513d (incorrect -> 0x9839), seq 68206:69606, ack 1220, win 166, options [nop,nop,TS val 2232040897 ecr 1338169563], length 1400
07:15:18.626510 enp3s0f0 Out IP (tos 0x0, ttl 64, id 4381, offset 0, flags [DF], proto TCP (6), length 2852) 51.195.104.xxx.54475 > 94.204.80.81.58396: Flags [P.], cksum 0x56b5 (incorrect -> 0x39ed), seq 69606:72406, ack 1220, win 166, options [nop,nop,TS val 2232040897 ecr 1338169563], length 2800
07:15:18.626525 enp3s0f0 Out IP (tos 0x0, ttl 64, id 4383, offset 0, flags [DF], proto TCP (6), length 2852) 51.195.104.xxx.54475 > 94.204.80.81.58396: Flags [P.], cksum 0x56b5 (incorrect -> 0x6088), seq 72406:75206, ack 1220, win 166, options [nop,nop,TS val 2232040897 ecr 1338169563], length 2800
07:15:18.626559 enp3s0f0 Out IP (tos 0x0, ttl 64, id 4385, offset 0, flags [DF], proto TCP (6), length 2852) 51.195.104.xxx.54475 > 94.204.80.81.58396: Flags [P.], cksum 0x56b5 (incorrect -> 0x6ac5), seq 75206:78006, ack 1220, win 166, options [nop,nop,TS val 2232040897 ecr 1338169563], length 2800
07:15:18.626567 enp3s0f0 Out IP (tos 0x0, ttl 64, id 4387, offset 0, flags [DF], proto TCP (6), length 1452) 51.195.104.xxx.54475 > 94.204.80.81.58396: Flags [P.], cksum 0x513d (incorrect -> 0x663d), seq 78006:79406, ack 1220, win 166, options [nop,nop,TS val 2232040897 ecr 1338169563], length 1400
<- 77000(+23800) bytes sent
07:15:18.760117 enp3s0f0 In IP (tos 0x0, ttl 50, id 53204, offset 0, flags [DF], proto TCP (6), length 52) 94.204.80.81.58396 > 51.195.104.xxx.54475: Flags [.], cksum 0x4b88 (correct), ack 58406, win 470, options [nop,nop,TS val 1338169696 ecr 2232040897], length 0
07:15:18.760118 enp3s0f0 In IP (tos 0x0, ttl 50, id 53206, offset 0, flags [DF], proto TCP (6), length 52) 94.204.80.81.58396 > 51.195.104.xxx.54475: Flags [.], cksum 0x2fc3 (correct), ack 65406, win 579, options [nop,nop,TS val 1338169696 ecr 2232040897], length 0
07:15:18.760119 enp3s0f0 In IP (tos 0x0, ttl 50, id 53205, offset 0, flags [DF], proto TCP (6), length 52) 94.204.80.81.58396 > 51.195.104.xxx.54475: Flags [.], cksum 0x3adf (correct), ack 62606, win 535, options [nop,nop,TS val 1338169696 ecr 2232040897], length 0
07:15:18.760120 enp3s0f0 In IP (tos 0x0, ttl 50, id 53207, offset 0, flags [DF], proto TCP (6), length 52) 94.204.80.81.58396 > 51.195.104.xxx.54475: Flags [.], cksum 0x24a7 (correct), ack 68206, win 623, options [nop,nop,TS val 1338169696 ecr 2232040897], length 0
07:15:18.760121 enp3s0f0 In IP (tos 0x0, ttl 50, id 53208, offset 0, flags [DF], proto TCP (6), length 52) 94.204.80.81.58396 > 51.195.104.xxx.54475: Flags [.], cksum 0x13fe (correct), ack 72406, win 688, options [nop,nop,TS val 1338169696 ecr 2232040897], length 0
07:15:18.760122 enp3s0f0 In IP (tos 0x0, ttl 50, id 53209, offset 0, flags [DF], proto TCP (6), length 52) 94.204.80.81.58396 > 51.195.104.xxx.54475: Flags [.], cksum 0x08e2 (correct), ack 75206, win 732, options [nop,nop,TS val 1338169696 ecr 2232040897], length 0
07:15:18.760123 enp3s0f0 In IP (tos 0x0, ttl 50, id 53210, offset 0, flags [DF], proto TCP (6), length 52) 94.204.80.81.58396 > 51.195.104.xxx.54475: Flags [.], cksum 0xfdc5 (correct), ack 78006, win 776, options [nop,nop,TS val 1338169696 ecr 2232040897], length 0
07:15:18.760124 enp3s0f0 In IP (tos 0x0, ttl 50, id 53211, offset 0, flags [DF], proto TCP (6), length 52) 94.204.80.81.58396 > 51.195.104.xxx.54475: Flags [.], cksum 0xf837 (correct), ack 79406, win 798, options [nop,nop,TS val 1338169696 ecr 2232040897], length 0
07:15:19.225059 enp3s0f0 In IP (tos 0x0, ttl 50, id 53212, offset 0, flags [DF], proto TCP (6), length 96) 94.204.80.81.58396 > 51.195.104.xxx.54475: Flags [P.], cksum 0x5745 (correct), seq 1220:1264, ack 79406, win 798, options [nop,nop,TS val 1338170161 ecr 2232040897], length 44
-> 44b recvd
07:15:19.225229 enp3s0f0 Out IP (tos 0x0, ttl 64, id 4388, offset 0, flags [DF], proto TCP (6), length 92) 51.195.104.xxx.54475 > 94.204.80.81.58396: Flags [P.], cksum 0x4bed (incorrect -> 0x8256), seq 79406:79446, ack 1264, win 166, options [nop,nop,TS val 2232041496 ecr 1338170161], length 40
<- 40b sent
07:15:19.250958 enp3s0f0 Out IP (tos 0x0, ttl 64, id 4389, offset 0, flags [DF], proto TCP (6), length 1452) 51.195.104.xxx.54475 > 94.204.80.81.58396: Flags [.], cksum 0x513d (incorrect -> 0x3228), seq 79446:80846, ack 1264, win 166, options [nop,nop,TS val 2232041521 ecr 1338170161], length 1400 <- ?
07:15:19.252839 enp3s0f0 Out IP (tos 0x0, ttl 64, id 4390, offset 0, flags [DF], proto TCP (6), length 1452) 51.195.104.xxx.54475 > 94.204.80.81.58396: Flags [.], cksum 0x513d (incorrect -> 0x792d), seq 80846:82246, ack 1264, win 166, options [nop,nop,TS val 2232041523 ecr 1338170161], length 1400 <- ?
07:15:19.358007 enp3s0f0 In IP (tos 0x0, ttl 50, id 53213, offset 0, flags [DF], proto TCP (6), length 52) 94.204.80.81.58396 > 51.195.104.xxx.54475: Flags [.], cksum 0xf335 (correct), ack 79446, win 798, options [nop,nop,TS val 1338170295 ecr 2232041496], length 0
07:15:19.358035 enp3s0f0 Out IP (tos 0x0, ttl 64, id 4391, offset 0, flags [DF], proto TCP (6), length 294) 51.195.104.xxx.54475 > 94.204.80.81.58396: Flags [P.], cksum 0x4cb7 (incorrect -> 0xc03c), seq 82246:82488, ack 1264, win 166, options [nop,nop,TS val 2232041628 ecr 1338170295], length 242 <- ?
07:15:19.384891 enp3s0f0 In IP (tos 0x0, ttl 50, id 53214, offset 0, flags [DF], proto TCP (6), length 52) 94.204.80.81.58396 > 51.195.104.xxx.54475: Flags [.], cksum 0xed73 (correct), ack 80846, win 821, options [nop,nop,TS val 1338170321 ecr 2232041521], length 0
07:15:19.386000 enp3s0f0 In IP (tos 0x0, ttl 50, id 53215, offset 0, flags [DF], proto TCP (6), length 52) 94.204.80.81.58396 > 51.195.104.xxx.54475: Flags [.], cksum 0xe7e0 (correct), ack 82246, win 844, options [nop,nop,TS val 1338170323 ecr 2232041523], length 0
07:15:19.391144 enp3s0f0 Out IP (tos 0x0, ttl 64, id 4392, offset 0, flags [DF], proto TCP (6), length 1452) 51.195.104.xxx.54475 > 94.204.80.81.58396: Flags [.], cksum 0x513d (incorrect -> 0x7b6e), seq 82488:83888, ack 1264, win 166, options [nop,nop,TS val 2232041661 ecr 1338170323], length 1400 <- ?
07:15:19.491024 enp3s0f0 In IP (tos 0x0, ttl 50, id 53216, offset 0, flags [DF], proto TCP (6), length 52) 94.204.80.81.58396 > 51.195.104.xxx.54475: Flags [.], cksum 0xe606 (correct), ack 82488, win 866, options [nop,nop,TS val 1338170428 ecr 2232041628], length 0
07:15:19.491051 enp3s0f0 Out IP (tos 0x0, ttl 64, id 4393, offset 0, flags [DF], proto TCP (6), length 1079) 51.195.104.xxx.54475 > 94.204.80.81.58396: Flags [P.], cksum 0x4fc8 (incorrect -> 0x1cca), seq 83888:84915, ack 1264, win 166, options [nop,nop,TS val 2232041761 ecr 1338170428], length 1027 <- ?
07:15:19.524212 enp3s0f0 In IP (tos 0x0, ttl 50, id 53217, offset 0, flags [DF], proto TCP (6), length 52) 94.204.80.81.58396 > 51.195.104.xxx.54475: Flags [.], cksum 0xe036 (correct), ack 83888, win 888, options [nop,nop,TS val 1338170461 ecr 2232041661], length 0
07:15:19.527877 enp3s0f0 Out IP (tos 0x0, ttl 64, id 4394, offset 0, flags [DF], proto TCP (6), length 1452) 51.195.104.xxx.54475 > 94.204.80.81.58396: Flags [.], cksum 0x513d (incorrect -> 0xd0c8), seq 84915:86315, ack 1264, win 166, options [nop,nop,TS val 2232041798 ecr 1338170461], length 1400
07:15:19.528290 enp3s0f0 Out IP (tos 0x0, ttl 64, id 4395, offset 0, flags [DF], proto TCP (6), length 1452) 51.195.104.xxx.54475 > 94.204.80.81.58396: Flags [.], cksum 0x513d (incorrect -> 0x5456), seq 86315:87715, ack 1264, win 166, options [nop,nop,TS val 2232041799 ecr 1338170461], length 1400
07:15:19.529080 enp3s0f0 Out IP (tos 0x0, ttl 64, id 4396, offset 0, flags [DF], proto TCP (6), length 1452) 51.195.104.xxx.54475 > 94.204.80.81.58396: Flags [.], cksum 0x513d (incorrect -> 0x6cef), seq 87715:89115, ack 1264, win 166, options [nop,nop,TS val 2232041799 ecr 1338170461], length 1400
07:15:19.581807 enp3s0f0 Out IP (tos 0x0, ttl 64, id 4397, offset 0, flags [DF], proto TCP (6), length 1452) 51.195.104.xxx.54475 > 94.204.80.81.58396: Flags [.], cksum 0x513d (incorrect -> 0x73f7), seq 89115:90515, ack 1264, win 166, options [nop,nop,TS val 2232041852 ecr 1338170461], length 1400
07:15:19.624072 enp3s0f0 In IP (tos 0x0, ttl 50, id 53218, offset 0, flags [DF], proto TCP (6), length 52) 94.204.80.81.58396 > 51.195.104.xxx.54475: Flags [.], cksum 0xdb55 (correct), ack 84915, win 910, options [nop,nop,TS val 1338170561 ecr 2232041761], length 0
07:15:19.624100 enp3s0f0 Out IP (tos 0x0, ttl 64, id 4398, offset 0, flags [DF], proto TCP (6), length 353) 51.195.104.xxx.54475 > 94.204.80.81.58396: Flags [P.], cksum 0x4cf2 (incorrect -> 0x8e7a), seq 90515:90816, ack 1264, win 166, options [nop,nop,TS val 2232041894 ecr 1338170561], length 301
07:15:19.624235 enp3s0f0 In IP (tos 0x0, ttl 50, id 53219, offset 0, flags [DF], proto TCP (6), length 83) 94.204.80.81.58396 > 51.195.104.xxx.54475: Flags [P.], cksum 0x6139 (correct), seq 1264:1295, ack 84915, win 910, options [nop,nop,TS val 1338170561 ecr 2232041761], length 31
07:15:19.624769 enp3s0f0 Out IP (tos 0x0, ttl 64, id 4399, offset 0, flags [DF], proto TCP (6), length 52) 51.195.104.xxx.54475 > 94.204.80.81.58396: Flags [F.], cksum 0x4bc5 (incorrect -> 0xc68a), seq 90816, ack 1295, win 166, options [nop,nop,TS val 2232041895 ecr 1338170561], length 0
07:15:19.625115 enp3s0f0 In IP (tos 0x0, ttl 50, id 53220, offset 0, flags [DF], proto TCP (6), length 52) 94.204.80.81.58396 > 51.195.104.xxx.54475: Flags [F.], cksum 0xdb35 (correct), seq 1295, ack 84915, win 910, options [nop,nop,TS val 1338170561 ecr 2232041761], length 0
07:15:19.625132 enp3s0f0 Out IP (tos 0x0, ttl 64, id 4400, offset 0, flags [DF], proto TCP (6), length 52) 51.195.104.xxx.54475 > 94.204.80.81.58396: Flags [.], cksum 0x4bc5 (incorrect -> 0xc689), ack 1296, win 166, options [nop,nop,TS val 2232041895 ecr 1338170561], length 0
07:15:19.662110 enp3s0f0 In IP (tos 0x0, ttl 50, id 0, offset 0, flags [DF], proto TCP (6), length 40) 94.204.80.81.58396 > 51.195.104.xxx.54475: Flags [R], cksum 0x9646 (correct), seq 3270660640, win 0, length 0
07:15:19.662112 enp3s0f0 In IP (tos 0x0, ttl 50, id 0, offset 0, flags [DF], proto TCP (6), length 40) 94.204.80.81.58396 > 51.195.104.xxx.54475: Flags [R], cksum 0x9646 (correct), seq 3270660640, win 0, length 0
07:15:19.662993 enp3s0f0 In IP (tos 0x0, ttl 50, id 0, offset 0, flags [DF], proto TCP (6), length 40) 94.204.80.81.58396 > 51.195.104.xxx.54475: Flags [R], cksum 0x9646 (correct), seq 3270660640, win 0, length 0
07:15:19.715962 enp3s0f0 In IP (tos 0x0, ttl 50, id 0, offset 0, flags [DF], proto TCP (6), length 40) 94.204.80.81.58396 > 51.195.104.xxx.54475: Flags [R], cksum 0x9646 (correct), seq 3270660640, win 0, length 0
07:15:19.758056 enp3s0f0 In IP (tos 0x0, ttl 50, id 0, offset 0, flags [DF], proto TCP (6), length 40) 94.204.80.81.58396 > 51.195.104.xxx.54475: Flags [R], cksum 0x9646 (correct), seq 3270660640, win 0, length 0
07:15:19.759036 enp3s0f0 In IP (tos 0x0, ttl 50, id 0, offset 0, flags [DF], proto TCP (6), length 40) 94.204.80.81.58396 > 51.195.104.xxx.54475: Flags [R], cksum 0x9627 (correct), seq 3270660671, win 0, length 0
07:15:19.759037 enp3s0f0 In IP (tos 0x0, ttl 50, id 0, offset 0, flags [DF], proto TCP (6), length 40) 94.204.80.81.58396 > 51.195.104.xxx.54475: Flags [R], cksum 0x9626 (correct), seq 3270660672, win 0, length 0
Log from serverside on normal client connection:
[2024-01-25T07:59:16.861Z INFO server::app::ui] UI:2 Hello!
[2024-01-25T07:59:16.863Z TRACE tcp_req] Sending data(85800b) to client:2...
[2024-01-25T07:59:16.863Z TRACE tcp_uni] * process data(65652bytes) for sending | Identity(type:ServerTcpClient,flag:3,slave:6,master:2), zero_copy:false
[2024-01-25T07:59:16.863Z TRACE tcp_uni] * process data(20322bytes) for sending | Identity(type:ServerTcpClient,flag:3,slave:7,master:2), zero_copy:false
[2024-01-25T07:59:16.863Z TRACE tcp] TCP incoming_data Identity(type:ServerTcpClient,flag:3,slave:6,master:2), res=65652 | is_more: false
<- 65652 ok
[2024-01-25T07:59:16.863Z TRACE tcp] TCP incoming_data Identity(type:ServerTcpClient,flag:3,slave:7,master:2), res=20322 | is_more: false
<- 20322 ok
[2024-01-25T07:59:17.867Z TRACE tcp] TCP incoming_data Identity(type:ServerTcpClient,flag:0,slave:0,master:2), res=44 | is_more: true
-> 44b recvd
[2024-01-25T07:59:17.867Z INFO server::app::ui] UI:2 First ping recvd!
[2024-01-25T07:59:17.867Z TRACE tcp_req] Sending data(11b) to client:2...
[2024-01-25T07:59:17.867Z TRACE tcp_uni] * process data(40bytes) for sending | Identity(type:ServerTcpClient,flag:3,slave:9,master:2), zero_copy:false
[2024-01-25T07:59:17.867Z TRACE tcp] TCP incoming_data Identity(type:ServerTcpClient,flag:3,slave:9,master:2), res=40 | is_more: false
<- 40 ok
[2024-01-25T07:59:17.952Z TRACE tcp_req] Sending data(937b) to client:2...
[2024-01-25T07:59:17.952Z TRACE tcp_uni] * process data(966bytes) for sending | Identity(type:ServerTcpClient,flag:3,slave:11,master:2), zero_copy:false
[2024-01-25T07:59:17.952Z TRACE tcp] TCP incoming_data Identity(type:ServerTcpClient,flag:3,slave:11,master:2), res=966 | is_more: false
<- 966 ok
[2024-01-25T07:59:17.953Z TRACE tcp_req] Sending data(2027b) to client:2...
[2024-01-25T07:59:17.953Z TRACE tcp_uni] * process data(2056bytes) for sending | Identity(type:ServerTcpClient,flag:3,slave:13,master:2), zero_copy:false
[2024-01-25T07:59:17.953Z TRACE tcp] TCP incoming_data Identity(type:ServerTcpClient,flag:3,slave:13,master:2), res=2056 | is_more: false
<- 2056 ok
Serverside tcpdump log on normal client connection (very nice packets segmentation):
tcpdump: data link type LINUX_SLL2
tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
07:59:15.871821 enp3s0f0 In IP (tos 0x0, ttl 50, id 0, offset 0, flags [DF], proto TCP (6), length 64) 94.204.80.81.56482 > 51.195.104.xxx.54475: Flags [S], cksum 0x47f6 (correct), seq 465251429, win 65535, options [mss 1412,nop,wscale 6,nop,nop,TS val 3014456923 ecr 0,sackOK,eol], length 0
07:59:15.871869 enp3s0f0 Out IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60) 51.195.104.xxx.54475 > 94.204.80.81.56482: Flags [S.], cksum 0x4bcd (incorrect -> 0x9d7b), seq 2976341875, ack 465251430, win 43440, options [mss 1460,sackOK,TS val 2234678142 ecr 3014456923,nop,wscale 8], length 0
07:59:16.059666 enp3s0f0 In IP (tos 0x0, ttl 50, id 0, offset 0, flags [DF], proto TCP (6), length 52) 94.204.80.81.56482 > 51.195.104.xxx.54475: Flags [.], cksum 0x6d33 (correct), ack 1, win 2056, options [nop,nop,TS val 3014457113 ecr 2234678142], length 0
07:59:16.059924 enp3s0f0 In IP (tos 0x0, ttl 50, id 0, offset 0, flags [DF], proto TCP (6), length 569) 94.204.80.81.56482 > 51.195.104.xxx.54475: Flags [P.], cksum 0x2069 (correct), seq 1:518, ack 1, win 2056, options [nop,nop,TS val 3014457113 ecr 2234678142], length 517
07:59:16.059948 enp3s0f0 Out IP (tos 0x0, ttl 64, id 40388, offset 0, flags [DF], proto TCP (6), length 52) 51.195.104.xxx.54475 > 94.204.80.81.56482: Flags [.], cksum 0x4bc5 (incorrect -> 0x71d2), ack 518, win 168, options [nop,nop,TS val 2234678330 ecr 3014457113], length 0
07:59:16.063308 enp3s0f0 Out IP (tos 0x0, ttl 64, id 40389, offset 0, flags [DF], proto TCP (6), length 2248) 51.195.104.xxx.54475 > 94.204.80.81.56482: Flags [P.], cksum 0x5459 (incorrect -> 0x403f), seq 1:2197, ack 518, win 168, options [nop,nop,TS val 2234678334 ecr 3014457113], length 2196
07:59:16.286715 enp3s0f0 In IP (tos 0x0, ttl 50, id 0, offset 0, flags [DF], proto TCP (6), length 52) 94.204.80.81.56482 > 51.195.104.xxx.54475: Flags [.], cksum 0x612f (correct), ack 2197, win 2021, options [nop,nop,TS val 3014457319 ecr 2234678334], length 0
07:59:16.286716 enp3s0f0 In IP (tos 0x0, ttl 50, id 0, offset 0, flags [DF], proto TCP (6), length 145) 94.204.80.81.56482 > 51.195.104.xxx.54475: Flags [P.], cksum 0xff25 (correct), seq 518:611, ack 2197, win 2048, options [nop,nop,TS val 3014457328 ecr 2234678334], length 93
07:59:16.287174 enp3s0f0 Out IP (tos 0x0, ttl 64, id 40391, offset 0, flags [DF], proto TCP (6), length 103) 51.195.104.xxx.54475 > 94.204.80.81.56482: Flags [P.], cksum 0x4bf8 (incorrect -> 0xc764), seq 2197:2248, ack 611, win 168, options [nop,nop,TS val 2234678557 ecr 3014457328], length 51
07:59:16.424585 enp3s0f0 In IP (tos 0x0, ttl 50, id 0, offset 0, flags [DF], proto TCP (6), length 52) 94.204.80.81.56482 > 51.195.104.xxx.54475: Flags [.], cksum 0x5f06 (correct), ack 2248, win 2047, options [nop,nop,TS val 3014457479 ecr 2234678557], length 0
07:59:16.425708 enp3s0f0 In IP (tos 0x0, ttl 50, id 0, offset 0, flags [DF], proto TCP (6), length 672) 94.204.80.81.56482 > 51.195.104.xxx.54475: Flags [P.], cksum 0x5626 (correct), seq 611:1231, ack 2248, win 2048, options [nop,nop,TS val 3014457481 ecr 2234678557], length 620
07:59:16.426005 enp3s0f0 Out IP (tos 0x0, ttl 64, id 40392, offset 0, flags [DF], proto TCP (6), length 210) 51.195.104.xxx.54475 > 94.204.80.81.56482: Flags [P.], cksum 0x4c63 (incorrect -> 0x1387), seq 2248:2406, ack 1231, win 166, options [nop,nop,TS val 2234678696 ecr 3014457481], length 158
07:59:16.562611 enp3s0f0 In IP (tos 0x0, ttl 50, id 0, offset 0, flags [DF], proto TCP (6), length 52) 94.204.80.81.56482 > 51.195.104.xxx.54475: Flags [.], cksum 0x5ae8 (correct), ack 2406, win 2045, options [nop,nop,TS val 3014457618 ecr 2234678696], length 0
07:59:16.861629 enp3s0f0 In IP (tos 0x0, ttl 50, id 0, offset 0, flags [DF], proto TCP (6), length 88) 94.204.80.81.56482 > 51.195.104.xxx.54475: Flags [P.], cksum 0x2db1 (correct), seq 1231:1267, ack 2406, win 2048, options [nop,nop,TS val 3014457917 ecr 2234678696], length 36
07:59:16.863568 enp3s0f0 Out IP (tos 0x0, ttl 64, id 40393, offset 0, flags [DF], proto TCP (6), length 2852) 51.195.104.xxx.54475 > 94.204.80.81.56482: Flags [P.], cksum 0x56b5 (incorrect -> 0x1c51), seq 2406:5206, ack 1267, win 166, options [nop,nop,TS val 2234679134 ecr 3014457917], length 2800
07:59:16.863610 enp3s0f0 Out IP (tos 0x0, ttl 64, id 40395, offset 0, flags [DF], proto TCP (6), length 2852) 51.195.104.xxx.54475 > 94.204.80.81.56482: Flags [P.], cksum 0x56b5 (incorrect -> 0x0173), seq 5206:8006, ack 1267, win 166, options [nop,nop,TS val 2234679134 ecr 3014457917], length 2800
07:59:16.863619 enp3s0f0 Out IP (tos 0x0, ttl 64, id 40397, offset 0, flags [DF], proto TCP (6), length 2852) 51.195.104.xxx.54475 > 94.204.80.81.56482: Flags [P.], cksum 0x56b5 (incorrect -> 0x9edb), seq 8006:10806, ack 1267, win 166, options [nop,nop,TS val 2234679134 ecr 3014457917], length 2800
07:59:16.863652 enp3s0f0 Out IP (tos 0x0, ttl 64, id 40399, offset 0, flags [DF], proto TCP (6), length 2852) 51.195.104.xxx.54475 > 94.204.80.81.56482: Flags [P.], cksum 0x56b5 (incorrect -> 0x28aa), seq 10806:13606, ack 1267, win 166, options [nop,nop,TS val 2234679134 ecr 3014457917], length 2800
07:59:16.863685 enp3s0f0 Out IP (tos 0x0, ttl 64, id 40401, offset 0, flags [DF], proto TCP (6), length 2852) 51.195.104.xxx.54475 > 94.204.80.81.56482: Flags [P.], cksum 0x56b5 (incorrect -> 0xd62b), seq 13606:16406, ack 1267, win 166, options [nop,nop,TS val 2234679134 ecr 3014457917], length 2800
07:59:17.069621 enp3s0f0 In IP (tos 0x0, ttl 50, id 0, offset 0, flags [DF], proto TCP (6), length 40) 94.204.80.81.56482 > 51.195.104.xxx.54475: Flags [.], cksum 0x4398 (correct), ack 2406, win 2048, length 0
07:59:17.069642 enp3s0f0 Out IP (tos 0x0, ttl 64, id 40403, offset 0, flags [DF], proto TCP (6), length 52) 51.195.104.xxx.54475 > 94.204.80.81.56482: Flags [.], cksum 0x4bc5 (incorrect -> 0x27bc), ack 1267, win 166, options [nop,nop,TS val 2234679340 ecr 3014457917], length 0
07:59:17.075647 enp3s0f0 In IP (tos 0x0, ttl 50, id 0, offset 0, flags [DF], proto TCP (6), length 52) 94.204.80.81.56482 > 51.195.104.xxx.54475: Flags [.], cksum 0x205a (correct), ack 16406, win 2048, options [nop,nop,TS val 3014458131 ecr 2234679134], length 0
07:59:17.075672 enp3s0f0 Out IP (tos 0x0, ttl 64, id 40404, offset 0, flags [DF], proto TCP (6), length 2852) 51.195.104.xxx.54475 > 94.204.80.81.56482: Flags [P.], cksum 0x56b5 (incorrect -> 0xa219), seq 16406:19206, ack 1267, win 166, options [nop,nop,TS val 2234679346 ecr 3014458131], length 2800
07:59:17.075710 enp3s0f0 Out IP (tos 0x0, ttl 64, id 40406, offset 0, flags [DF], proto TCP (6), length 2852) 51.195.104.xxx.54475 > 94.204.80.81.56482: Flags [P.], cksum 0x56b5 (incorrect -> 0xacdd), seq 19206:22006, ack 1267, win 166, options [nop,nop,TS val 2234679346 ecr 3014458131], length 2800
07:59:17.075732 enp3s0f0 Out IP (tos 0x0, ttl 64, id 40408, offset 0, flags [DF], proto TCP (6), length 2852) 51.195.104.xxx.54475 > 94.204.80.81.56482: Flags [P.], cksum 0x56b5 (incorrect -> 0x4338), seq 22006:24806, ack 1267, win 166, options [nop,nop,TS val 2234679346 ecr 3014458131], length 2800
07:59:17.075752 enp3s0f0 Out IP (tos 0x0, ttl 64, id 40410, offset 0, flags [DF], proto TCP (6), length 2852) 51.195.104.xxx.54475 > 94.204.80.81.56482: Flags [P.], cksum 0x56b5 (incorrect -> 0xc688), seq 24806:27606, ack 1267, win 166, options [nop,nop,TS val 2234679346 ecr 3014458131], length 2800
07:59:17.075770 enp3s0f0 Out IP (tos 0x0, ttl 64, id 40412, offset 0, flags [DF], proto TCP (6), length 2852) 51.195.104.xxx.54475 > 94.204.80.81.56482: Flags [P.], cksum 0x56b5 (incorrect -> 0x7c72), seq 27606:30406, ack 1267, win 166, options [nop,nop,TS val 2234679346 ecr 3014458131], length 2800
07:59:17.075784 enp3s0f0 Out IP (tos 0x0, ttl 64, id 40414, offset 0, flags [DF], proto TCP (6), length 2852) 51.195.104.xxx.54475 > 94.204.80.81.56482: Flags [P.], cksum 0x56b5 (incorrect -> 0x63b9), seq 30406:33206, ack 1267, win 166, options [nop,nop,TS val 2234679346 ecr 3014458131], length 2800
07:59:17.075799 enp3s0f0 Out IP (tos 0x0, ttl 64, id 40416, offset 0, flags [DF], proto TCP (6), length 2852) 51.195.104.xxx.54475 > 94.204.80.81.56482: Flags [P.], cksum 0x56b5 (incorrect -> 0x3665), seq 33206:36006, ack 1267, win 166, options [nop,nop,TS val 2234679346 ecr 3014458131], length 2800
07:59:17.075843 enp3s0f0 Out IP (tos 0x0, ttl 64, id 40418, offset 0, flags [DF], proto TCP (6), length 2852) 51.195.104.xxx.54475 > 94.204.80.81.56482: Flags [P.], cksum 0x56b5 (incorrect -> 0x0127), seq 36006:38806, ack 1267, win 166, options [nop,nop,TS val 2234679346 ecr 3014458131], length 2800
07:59:17.075870 enp3s0f0 Out IP (tos 0x0, ttl 64, id 40420, offset 0, flags [DF], proto TCP (6), length 2852) 51.195.104.xxx.54475 > 94.204.80.81.56482: Flags [P.], cksum 0x56b5 (incorrect -> 0x87fa), seq 38806:41606, ack 1267, win 166, options [nop,nop,TS val 2234679346 ecr 3014458131], length 2800
07:59:17.075890 enp3s0f0 Out IP (tos 0x0, ttl 64, id 40422, offset 0, flags [DF], proto TCP (6), length 2852) 51.195.104.xxx.54475 > 94.204.80.81.56482: Flags [P.], cksum 0x56b5 (incorrect -> 0x1f79), seq 41606:44406, ack 1267, win 166, options [nop,nop,TS val 2234679346 ecr 3014458131], length 2800
07:59:17.289548 enp3s0f0 In IP (tos 0x0, ttl 50, id 0, offset 0, flags [DF], proto TCP (6), length 52) 94.204.80.81.56482 > 51.195.104.xxx.54475: Flags [.], cksum 0xb150 (correct), ack 44406, win 2048, options [nop,nop,TS val 3014458344 ecr 2234679346], length 0
07:59:17.289591 enp3s0f0 Out IP (tos 0x0, ttl 64, id 40424, offset 0, flags [DF], proto TCP (6), length 2852) 51.195.104.xxx.54475 > 94.204.80.81.56482: Flags [P.], cksum 0x56b5 (incorrect -> 0xace9), seq 44406:47206, ack 1267, win 166, options [nop,nop,TS val 2234679560 ecr 3014458344], length 2800
07:59:17.289604 enp3s0f0 Out IP (tos 0x0, ttl 64, id 40426, offset 0, flags [DF], proto TCP (6), length 2852) 51.195.104.xxx.54475 > 94.204.80.81.56482: Flags [P.], cksum 0x56b5 (incorrect -> 0x7a2e), seq 47206:50006, ack 1267, win 166, options [nop,nop,TS val 2234679560 ecr 3014458344], length 2800
07:59:17.289639 enp3s0f0 Out IP (tos 0x0, ttl 64, id 40428, offset 0, flags [DF], proto TCP (6), length 2852) 51.195.104.xxx.54475 > 94.204.80.81.56482: Flags [P.], cksum 0x56b5 (incorrect -> 0x50ca), seq 50006:52806, ack 1267, win 166, options [nop,nop,TS val 2234679560 ecr 3014458344], length 2800
07:59:17.289658 enp3s0f0 Out IP (tos 0x0, ttl 64, id 40430, offset 0, flags [DF], proto TCP (6), length 2852) 51.195.104.xxx.54475 > 94.204.80.81.56482: Flags [P.], cksum 0x56b5 (incorrect -> 0x822a), seq 52806:55606, ack 1267, win 166, options [nop,nop,TS val 2234679560 ecr 3014458344], length 2800
07:59:17.289670 enp3s0f0 Out IP (tos 0x0, ttl 64, id 40432, offset 0, flags [DF], proto TCP (6), length 2852) 51.195.104.xxx.54475 > 94.204.80.81.56482: Flags [P.], cksum 0x56b5 (incorrect -> 0xa748), seq 55606:58406, ack 1267, win 166, options [nop,nop,TS val 2234679560 ecr 3014458344], length 2800
07:59:17.289685 enp3s0f0 Out IP (tos 0x0, ttl 64, id 40434, offset 0, flags [DF], proto TCP (6), length 2852) 51.195.104.xxx.54475 > 94.204.80.81.56482: Flags [P.], cksum 0x56b5 (incorrect -> 0x5bbc), seq 58406:61206, ack 1267, win 166, options [nop,nop,TS val 2234679560 ecr 3014458344], length 2800
07:59:17.289699 enp3s0f0 Out IP (tos 0x0, ttl 64, id 40436, offset 0, flags [DF], proto TCP (6), length 2852) 51.195.104.xxx.54475 > 94.204.80.81.56482: Flags [P.], cksum 0x56b5 (incorrect -> 0x21c1), seq 61206:64006, ack 1267, win 166, options [nop,nop,TS val 2234679560 ecr 3014458344], length 2800
07:59:17.289742 enp3s0f0 Out IP (tos 0x0, ttl 64, id 40438, offset 0, flags [DF], proto TCP (6), length 2852) 51.195.104.xxx.54475 > 94.204.80.81.56482: Flags [P.], cksum 0x56b5 (incorrect -> 0x2cbc), seq 64006:66806, ack 1267, win 166, options [nop,nop,TS val 2234679560 ecr 3014458344], length 2800
07:59:17.289753 enp3s0f0 Out IP (tos 0x0, ttl 64, id 40440, offset 0, flags [DF], proto TCP (6), length 2852) 51.195.104.xxx.54475 > 94.204.80.81.56482: Flags [P.], cksum 0x56b5 (incorrect -> 0x99fb), seq 66806:69606, ack 1267, win 166, options [nop,nop,TS val 2234679560 ecr 3014458344], length 2800
07:59:17.289783 enp3s0f0 Out IP (tos 0x0, ttl 64, id 40442, offset 0, flags [DF], proto TCP (6), length 2852) 51.195.104.xxx.54475 > 94.204.80.81.56482: Flags [P.], cksum 0x56b5 (incorrect -> 0xdfb8), seq 69606:72406, ack 1267, win 166, options [nop,nop,TS val 2234679560 ecr 3014458344], length 2800
07:59:17.289800 enp3s0f0 Out IP (tos 0x0, ttl 64, id 40444, offset 0, flags [DF], proto TCP (6), length 2852) 51.195.104.xxx.54475 > 94.204.80.81.56482: Flags [P.], cksum 0x56b5 (incorrect -> 0x86d9), seq 72406:75206, ack 1267, win 166, options [nop,nop,TS val 2234679560 ecr 3014458344], length 2800
07:59:17.289816 enp3s0f0 Out IP (tos 0x0, ttl 64, id 40446, offset 0, flags [DF], proto TCP (6), length 2852) 51.195.104.xxx.54475 > 94.204.80.81.56482: Flags [P.], cksum 0x56b5 (incorrect -> 0x0463), seq 75206:78006, ack 1267, win 166, options [nop,nop,TS val 2234679560 ecr 3014458344], length 2800
07:59:17.289831 enp3s0f0 Out IP (tos 0x0, ttl 64, id 40448, offset 0, flags [DF], proto TCP (6), length 2852) 51.195.104.xxx.54475 > 94.204.80.81.56482: Flags [P.], cksum 0x56b5 (incorrect -> 0x3cf0), seq 78006:80806, ack 1267, win 166, options [nop,nop,TS val 2234679560 ecr 3014458344], length 2800
07:59:17.289846 enp3s0f0 Out IP (tos 0x0, ttl 64, id 40450, offset 0, flags [DF], proto TCP (6), length 2852) 51.195.104.xxx.54475 > 94.204.80.81.56482: Flags [P.], cksum 0x56b5 (incorrect -> 0x5fc0), seq 80806:83606, ack 1267, win 166, options [nop,nop,TS val 2234679560 ecr 3014458344], length 2800
07:59:17.289889 enp3s0f0 Out IP (tos 0x0, ttl 64, id 40452, offset 0, flags [DF], proto TCP (6), length 2852) 51.195.104.xxx.54475 > 94.204.80.81.56482: Flags [P.], cksum 0x56b5 (incorrect -> 0x9e3d), seq 83606:86406, ack 1267, win 166, options [nop,nop,TS val 2234679560 ecr 3014458344], length 2800
07:59:17.289897 enp3s0f0 Out IP (tos 0x0, ttl 64, id 40454, offset 0, flags [DF], proto TCP (6), length 2026) 51.195.104.xxx.54475 > 94.204.80.81.56482: Flags [P.], cksum 0x537b (incorrect -> 0xb68f), seq 86406:88380, ack 1267, win 166, options [nop,nop,TS val 2234679560 ecr 3014458344], length 1974
--- here we recvd all data
07:59:17.492640 enp3s0f0 In IP (tos 0x0, ttl 50, id 0, offset 0, flags [DF], proto TCP (6), length 52) 94.204.80.81.56482 > 51.195.104.xxx.54475: Flags [.], cksum 0x03e8 (correct), ack 88380, win 2048, options [nop,nop,TS val 3014458548 ecr 2234679560], length 0
07:59:17.867515 enp3s0f0 In IP (tos 0x0, ttl 50, id 0, offset 0, flags [DF], proto TCP (6), length 96) 94.204.80.81.56482 > 51.195.104.xxx.54475: Flags [P.], cksum 0xdd6e (correct), seq 1267:1311, ack 88380, win 2048, options [nop,nop,TS val 3014458918 ecr 2234679560], length 44
--- and 44b recvd
07:59:17.867692 enp3s0f0 Out IP (tos 0x0, ttl 64, id 40456, offset 0, flags [DF], proto TCP (6), length 92) 51.195.104.xxx.54475 > 94.204.80.81.56482: Flags [P.], cksum 0x4bed (incorrect -> 0x9880), seq 88380:88420, ack 1311, win 166, options [nop,nop,TS val 2234680138 ecr 3014458918], length 40
07:59:17.953225 enp3s0f0 Out IP (tos 0x0, ttl 64, id 40457, offset 0, flags [DF], proto TCP (6), length 2852) 51.195.104.xxx.54475 > 94.204.80.81.56482: Flags [P.], cksum 0x56b5 (incorrect -> 0x7798), seq 88420:91220, ack 1311, win 166, options [nop,nop,TS val 2234680224 ecr 3014458918], length 2800
07:59:18.056641 enp3s0f0 In IP (tos 0x0, ttl 50, id 0, offset 0, flags [DF], proto TCP (6), length 52) 94.204.80.81.56482 > 51.195.104.xxx.54475: Flags [.], cksum 0xff1e (correct), ack 88420, win 2047, options [nop,nop,TS val 3014459112 ecr 2234680138], length 0
07:59:18.056671 enp3s0f0 Out IP (tos 0x0, ttl 64, id 40459, offset 0, flags [DF], proto TCP (6), length 274) 51.195.104.xxx.54475 > 94.204.80.81.56482: Flags [P.], cksum 0x4ca3 (incorrect -> 0xa8a4), seq 91220:91442, ack 1311, win 166, options [nop,nop,TS val 2234680327 ecr 3014459112], length 222
07:59:18.089755 enp3s0f0 In IP (tos 0x0, ttl 50, id 0, offset 0, flags [DF], proto TCP (6), length 52) 94.204.80.81.56482 > 51.195.104.xxx.54475: Flags [.], cksum 0xf3b5 (correct), ack 91220, win 2048, options [nop,nop,TS val 3014459146 ecr 2234680224], length 0
07:59:18.144460 enp3s0f0 Out IP (tos 0x0, ttl 64, id 40460, offset 0, flags [DF], proto TCP (6), length 1452) 51.195.104.xxx.54475 > 94.204.80.81.56482: Flags [.], cksum 0x513d (incorrect -> 0xb57d), seq 91442:92842, ack 1311, win 166, options [nop,nop,TS val 2234680415 ecr 3014459146], length 1400
07:59:18.144966 enp3s0f0 Out IP (tos 0x0, ttl 64, id 40461, offset 0, flags [DF], proto TCP (6), length 1452) 51.195.104.xxx.54475 > 94.204.80.81.56482: Flags [.], cksum 0x513d (incorrect -> 0xd733), seq 92842:94242, ack 1311, win 166, options [nop,nop,TS val 2234680415 ecr 3014459146], length 1400
07:59:18.192525 enp3s0f0 In IP (tos 0x0, ttl 50, id 0, offset 0, flags [DF], proto TCP (6), length 52) 94.204.80.81.56482 > 51.195.104.xxx.54475: Flags [.], cksum 0xf20e (correct), ack 91442, win 2044, options [nop,nop,TS val 3014459248 ecr 2234680327], length 0
from rustls.
serzhiio
commented on May 26, 2024
Due to io-uring nature packets order is not guaranteed, and it seems the reason of all this
from rustls.
djc
commented on May 26, 2024
Right -- TLS operates on a byte stream, so that won't work. I guess that explains why the QUIC clients work.
from rustls.
serzhiio
commented on May 26, 2024
Right -- TLS operates on a byte stream, so that won't work. I guess that explains why the QUIC clients work.
It is not 100% true that it won't work (and it depends on design), i see that IO_HARDLINK flag do the right thing to keep submission queue ordered. At least after all this time i see that it is working.
P.S.: All problems started when i begin to send >64kb of data at once
from rustls.
serzhiio
commented on May 26, 2024
Very likely you are writing >64KB, which succeeds and tells you <64KB was written. Now your application must: retain the unwritten data, wait for the socket to become writable (either because the peer's window has cleared, or grown), and then write the remainder.
Thank you very much, i have two ways to eliminate this problem now!
from rustls.
Related Issues (20)
- 0.23 docs build is broken HOT 2
- optimize receiving data with TLS 1.2 and aes-128-gcm HOT 1
- optimize receiving data with TLS 1.3 and aes-256-gcm
- optimize server-side full handshakes for TLS 1.2 and 1.3 HOT 1
- Connection::dangerous_extract_secrets returns ConnectionTrafficSecrets::Aes128Gcm even when AES-256-GCM is negotiated
- Error: badRecordMac HOT 4
- Cipher suites configured through WebPkiServerVerifier::builder_with_provider is not working. Client hello contains more cipher suites then it configured. HOT 6
- rand_core::RngCore & CryptoRng support for CryptoProvider HOT 7
- expose more information in ClientHello HOT 4
- No common ciphersuit when FFDHE and ECDHE ciphersuites are available on server and client using TLS 1.2 HOT 4
- US Export control information HOT 4
- doc: AcceptedAlert::write doesn't necessarily write all bytes HOT 7
- Support using rustls without using specific ring or aws-lc-rs apis HOT 2
- Feature request: a way to set/get default ticketer dynamically HOT 6
- Feature Request: Avoid panicking when ring and aws_lc_rs are both specified HOT 12
- Option to Relax SNI Host Name Validation for IP Addresses HOT 7
- unbuffered: B: `CapacityBuffer` for `output_tls.capacity()` HOT 9
- The support for "mipsel-unknown-linux-musl" has failed. HOT 2
- Io(Custom { kind: InvalidData, error: AlertReceived(HandshakeFailure) }) HOT 6
- Linux compilation is slow and seems unable to store compilation results HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from rustls.