Comments (34)
KeithWiles
commented on July 28, 2024
I believe this works, but the person that can help is out of the office 27-28 and most likely will not reply till next week.
I am sure he tested this type of setup. Make sure the MAC addresses are correct for the Linux server or place the Linux server interface into promiscuous mode e.g. 'ifconfig eth0 promisc' this step is not normally required, but just to eliminate the MAC addresses.
from cndp.
liuzhen111
commented on July 28, 2024
debug:
2023-06-27T14:48:04Z DEBUG wireguard_rs::wireguard::workers] TUN worker, IP packet of 84 bytes (MTU = 1280)
[2023-06-27T14:48:04Z DEBUG wireguard_rs::wireguard::workers] TUN worker, router returned Err(NoCryptoKeyRoute)
[2023-06-27T14:48:05Z DEBUG wireguard_rs::wireguard::workers] TUN worker, IP packet of 84 bytes (MTU = 1280)
[2023-06-27T14:48:05Z DEBUG wireguard_rs::wireguard::workers] TUN worker, router returned Err(NoCryptoKeyRoute)
from cndp.
manojgop
commented on July 28, 2024
Hi @liuzhen111 - I verified that it works wireguard cndp <--> linux normal wireguard. Does wireguard default (without cndp) <--> linux normal wireguard and wireguard cndp <--> wireguard cndp work for you ? I ran wireguard cndp using ./src/platform/linux/cndp/cndprustwg.sh. Did you set ethtool rules for wireguard cndp for the required wg endpoint interface to send/recv packets to a specific qid ?
from cndp.
liuzhen111
commented on July 28, 2024
@manojgop Hello, the network topology is:
Server A(with cndp) <--> Server B(without cndp)
I have used ethtool to queue the specified traffic as follows on Server A:
ethtool -U ens6f1 flow-type udp4 src-ip 192.168.60.2 dst-ip 192.168.60.1 action 1 loc 1
ethtool -U ens6f1 flow-type udp4 src-ip 192.168.60.1 dst-ip 192.168.60.2 action 2 loc 2
192.168.60.2 is an intel x710 network adapter on ServerA (ubuntu 20).
192.168.60.1 is the ip address of a remote server B using a kvm VM (ubuntu 20).
from cndp.
manojgop
commented on July 28, 2024
Need more information. Can you share output of sudo wg on both systems . Also the contents of fwd.jsonc and wg_cndp.toml file used with CNDP
from cndp.
liuzhen111
commented on July 28, 2024
@manojgop Hello brother, I have sent relevant configuration files to your email address: [email protected]
sudo wg output information display is normal.
from cndp.
manojgop
commented on July 28, 2024
@manojgop Hello brother, I have sent relevant configuration files to your email address: [email protected] sudo wg output information display is normal.
Some email attachment got blocked. Just to confirm - Did you set the private keys in local wg interface ? Below is the configuration in my setup with CNDP wireguard and Linux kernel wireguard
Endpoint 1 (CNDP Wireguard)
=============================
$ sudo wg
interface: rsuwg0
public key: Gj/rmlfLyAtS/4RK9oq6WB6gDyKcQmgiKYwosIqJ9HA=
private key: (hidden)
peer: DV/f0jntATrrng/D7jVmqBz6IFlHWjPT/QsmhDut5Es=
endpoint: 48.0.0.155:5500
allowed ips: 10.0.4.2/32
$ip addr show enp134s0
5: enp134s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 xdp/id:654 qdisc mq state UP group default qlen 1000
link/ether 40:a6:b7:78:3f:ec brd ff:ff:ff:ff:ff:ff
inet 48.0.0.154/24 scope global enp134s0
valid_lft forever preferred_lft forever
inet6 fe80::42a6:b7ff:fe78:3fec/64 scope link
valid_lft forever preferred_lft forever
ethtool rules
sudo ethtool -N enp134s0 flow-type udp4 src-ip 48.0.0.155 dst-ip 48.0.0.154 action 23
sudo ethtool -N enp134s0 flow-type udp4 src-ip 48.0.0.154 dst-ip 48.0.0.155 action 33
Endpoint 2 (Linux Kernel Wireguard)
=====================================
$ sudo wg
interface: wg0
public key: DV/f0jntATrrng/D7jVmqBz6IFlHWjPT/QsmhDut5Es=
private key: (hidden)
listening port: 5500
peer: Gj/rmlfLyAtS/4RK9oq6WB6gDyKcQmgiKYwosIqJ9HA=
endpoint: 48.0.0.154:5400
allowed ips: 10.0.4.1/32
$ip addr show enp134s0
136: enp134s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 40:a6:b7:68:95:d4 brd ff:ff:ff:ff:ff:ff
inet 48.0.0.155/24 scope global enp134s0
valid_lft forever preferred_lft forever
ethtool rules
sudo ethtool -N enp134s0 flow-type udp4 src-ip 48.0.0.154 dst-ip 48.0.0.155 action 23
sudo ethtool -N enp134s0 flow-type udp4 src-ip 48.0.0.155 dst-ip 48.0.0.154 action 33
from cndp.
manojgop
commented on July 28, 2024
@manojgop The problem has been resolved, the server port cannot be modified, and only 5500 can be used
You could modify the script cndprustwg.sh to use the required port
from cndp.
liuzhen111
commented on July 28, 2024
@manojgop I have compressed the relevant configuration files into a zip file. Could you please help me check if there are any issues? Thank you
from cndp.
manojgop
commented on July 28, 2024
Could you check if the traffic is hitting the required queue --> watch -n1 -d "sudo ethtool -S ens6f1 | grep packets | column"
from cndp.
liuzhen111
commented on July 28, 2024
@manojgop
ethtool -U ens6f1 flow-type udp4 src-ip 192.168.60.2 dst-ip 192.168.60.1 action 1 loc 1
ethtool -U ens6f1 flow-type udp4 src-ip 192.168.60.1 dst-ip 192.168.60.2 action 2 loc 2
tx-1.packets: 0 And rx-1.packets: 0
from cndp.
manojgop
commented on July 28, 2024
So looks like no packets reaching queue 1. Based on your ethtool rules, if traffic is flowing in both directions (for example ping ICMP request and response packets ) then wireguard packets should be seen in queue 1 and 2
from cndp.
liuzhen111
commented on July 28, 2024
@manojgop Hello, what is the operating system version and kernel version you used for testing?
from cndp.
manojgop
commented on July 28, 2024
Once you ping remote wireguard interface IP, it'll request a new key. As you can see in following logs you can see "no key encryption key available" and then "request new key".
Finished dev [optimized + debuginfo] target(s) in 45.61s
Running `target/debug/wireguard-rs --disable-drop-privileges rsuwg0`
[2023-07-07T05:35:23Z INFO wireguard_rs] Starting rsuwg0 WireGuard device.
[2023-07-07T05:35:23Z DEBUG wireguard_rs::wireguard::workers] wireguard(757399cf) : handshake worker, started
[2023-07-07T05:35:23Z DEBUG wireguard_rs::wireguard::workers] wireguard(757399cf) : handshake worker, started
[2023-07-07T05:35:23Z DEBUG wireguard_rs::wireguard::workers] wireguard(757399cf) : handshake worker, started
[2023-07-07T05:35:23Z DEBUG wireguard_rs::wireguard::workers] wireguard(757399cf) : handshake worker, started
[2023-07-07T05:35:23Z DEBUG wireguard_rs::platform::linux::tun] netlink, fetch event (fd = 5)
[2023-07-07T05:35:24Z DEBUG wireguard_rs::platform::linux::tun] netlink, received message (80 bytes)
[2023-07-07T05:35:24Z DEBUG wireguard_rs::configuration::uapi] UAPI, Result of operation: Err(IOError)
[2023-07-07T05:35:24Z DEBUG wireguard_rs::configuration::uapi] UAPI, Set operation
[2023-07-07T05:35:24Z INFO wireguard_rs::configuration::config] configuration, set private key
[2023-07-07T05:35:24Z DEBUG wireguard_rs::wireguard::router::device] Clear sending keys
[2023-07-07T05:35:24Z DEBUG wireguard_rs::configuration::uapi] UAPI, Result of operation: Ok(())
[2023-07-07T05:35:24Z DEBUG wireguard_rs::platform::linux::tun] netlink, received message (1372 bytes)
[2023-07-07T05:35:24Z INFO wireguard_rs] Tun down
[2023-07-07T05:35:24Z INFO wireguard_rs::configuration::config] configuration, set device down
[2023-07-07T05:35:24Z DEBUG wireguard_rs::platform::linux::tun] netlink, fetch event (fd = 5)
[2023-07-07T05:35:25Z DEBUG wireguard_rs::configuration::uapi] UAPI, Result of operation: Err(IOError)
[2023-07-07T05:35:25Z DEBUG wireguard_rs::configuration::uapi] UAPI, Set operation
[2023-07-07T05:35:25Z DEBUG wireguard_rs::configuration::uapi] UAPI, Result of operation: Ok(())
[2023-07-07T05:35:25Z DEBUG wireguard_rs::configuration::uapi] UAPI, Result of operation: Err(IOError)
[2023-07-07T05:35:25Z DEBUG wireguard_rs::configuration::uapi] UAPI, Set operation
[2023-07-07T05:35:25Z DEBUG wireguard_rs::configuration::uapi] UAPI, Result of operation: Ok(())
[2023-07-07T05:35:25Z DEBUG wireguard_rs::platform::linux::tun] netlink, received message (1376 bytes)
[2023-07-07T05:35:25Z INFO wireguard_rs] Tun up (mtu = 1420)
[2023-07-07T05:35:25Z INFO wireguard_rs::configuration::config] configuration, set device up
[2023-07-07T05:35:25Z INFO wireguard_rs::platform::linux::cndp::udp] Wireguard called bind at port 5400
[2023-07-07T05:35:25Z DEBUG wireguard_rs::wireguard::workers] TUN worker, IP packet of 1 bytes (MTU = 0)
manoj@qct-4:~/wireguard/cndp/wireguard-rs$ [2023-07-07T05:35:26Z DEBUG wireguard_rs::platform::linux::cndp::udp] netdev = enp134s0
[2023-07-07T05:35:26Z DEBUG wireguard_rs::platform::linux::cndp::udp] ipv4_addr = Some(48.0.0.154)
[2023-07-07T05:35:26Z DEBUG wireguard_rs::platform::linux::cndp::udp] src_mac = Some(40:a6:b7:78:3f:ec)
[2023-07-07T05:35:26Z DEBUG wireguard_rs::platform::linux::tun] netlink, fetch event (fd = 5)
[2023-07-07T05:35:26Z DEBUG wireguard_rs::platform::linux::tun] netlink, received message (72 bytes)
[2023-07-07T05:35:26Z DEBUG wireguard_rs::platform::linux::tun] netlink, received message (1396 bytes)
[2023-07-07T05:35:26Z DEBUG wireguard_rs::platform::linux::tun] netlink, received message (1408 bytes)
[2023-07-07T05:35:29Z DEBUG wireguard_rs::wireguard::workers] TUN worker, IP packet of 48 bytes (MTU = 1420)
[2023-07-07T05:35:29Z DEBUG wireguard_rs::wireguard::workers] TUN worker, router returned Err(NoCryptoKeyRoute)
[2023-07-07T05:35:38Z DEBUG wireguard_rs::wireguard::workers] TUN worker, IP packet of 48 bytes (MTU = 1420)
[2023-07-07T05:35:38Z DEBUG wireguard_rs::wireguard::workers] TUN worker, router returned Err(NoCryptoKeyRoute)
[2023-07-07T05:35:48Z DEBUG wireguard_rs::wireguard::workers] TUN worker, IP packet of 84 bytes (MTU = 1420)
[2023-07-07T05:35:48Z DEBUG wireguard_rs::wireguard::router::peer] no key encryption key available
[2023-07-07T05:35:48Z DEBUG wireguard_rs::wireguard::router::peer] request new key
[2023-07-07T05:35:48Z DEBUG wireguard_rs::wireguard::workers] TUN worker, router returned Ok(())
[2023-07-07T05:35:48Z DEBUG wireguard_rs::wireguard::workers] wireguard(757399cf) : handshake worker, new handshake requested for PeerHandle(format: TODO)
[2023-07-07T05:35:48Z DEBUG wireguard_rs::wireguard::handshake::noise] create initiation
[2023-07-07T05:35:48Z DEBUG wireguard_rs::platform::linux::cndp::udp] iface_name: enp134s0, dst_ipv4 = 48.0.0.155
[2023-07-07T05:35:48Z DEBUG wireguard_rs::platform::linux::tun] netlink, received message (1396 bytes)
[2023-07-07T05:35:48Z DEBUG wireguard_rs::platform::linux::cndp::packet] Sent ARP request
[2023-07-07T05:35:48Z DEBUG wireguard_rs::platform::linux::cndp::packet] Received ARP reply
[2023-07-07T05:35:48Z DEBUG wireguard_rs::platform::linux::tun] netlink, received message (1396 bytes)
[2023-07-07T05:35:48Z DEBUG wireguard_rs::platform::linux::cndp::udp] dst mac from ARP = 40:a6:b7:68:95:d4
[2023-07-07T05:35:48Z DEBUG wireguard_rs::platform::linux::cndp::udp] stored dst mac in map
[2023-07-07T05:35:48Z DEBUG wireguard_rs::platform::linux::cndp::udp] Read bulk 1 UDP Packets at time 1688708148096391
[2023-07-07T05:35:48Z DEBUG wireguard_rs::platform::linux::cndp::udp] Read cndp_ethernet_packet = CndpPacket [
src_mac: Some(40:a6:b7:68:95:d4), dst_mac: Some(40:a6:b7:78:3f:ec),
src_ip: Some(48.0.0.155), dst_ip: Some(48.0.0.154),
src_port: Some(5500), dst_port: Some(5400)
]
from cndp.
manojgop
commented on July 28, 2024
Both ends will derive session keys from public keys using wireguard handshake protocol.
from cndp.
manojgop
commented on July 28, 2024
My system has Ubuntu 22.04 and 5.11.0-18-generic kernel
from cndp.
manojgop
commented on July 28, 2024
You could try running basic af_xdp sample app in both systems. @KeithWiles Do you have any suggestions ?
from cndp.
manojgop
commented on July 28, 2024
Can you send me a copy of the configuration file you are currently using. Including: cndprustwg.sh, fwd.jsonc, wg_ Cndp.toml My email is [email protected] thank you
from cndp.
KeithWiles
commented on July 28, 2024
The errors above are not really errors but information messages. I am assuming these messages appear each time you start the application, if not then maybe we have an issue. If they are happening while running the application then you may not have enough buffers allocated.
Buffers can be consumed for Rx and Tx AF_XDP rings, plus they can be on the CQ/FQ rings and mempool cache, plus the application could be holding onto a large number of buffers as well. Find out how big the Rx/Tx/FQ/CQ rings times the number of AF_XDP sockets are in size than double or triple the number of mbufs.
But if these message only appear at startup, but not while running then you do not have a problem with the number of buffers. Running the AF_XDP sample application could help determine if things are really working, but the WG example is not.
from cndp.
liuzhen111
commented on July 28, 2024
I switched the ethtool -N rule to:
sudo ethtool -U ens6f1 flow-type udp4 src-ip 192.168.60.2 dst-ip 192.168.60.1 action 2 loc 1
sudo ethtool -U ens6f1 flow-type udp4 src-ip 192.168.60.1 dst-ip 192.168.60.2 action 1 loc 2
but:
Still only queue 2 has packets on it, which is very surprising
from cndp.
KeithWiles
commented on July 28, 2024
Please make sure you have removed all of the old rules, which I assume you have done already.
I have had some problems in the past with drivers not really supporting all of the ethtool configuration options, but normally I would get an error when setting the rules and that does not seem to be the problem here.
If you configure ethtool to only send packets to 1 using something like sudo ethtool -N ens6f1 flow-type udp4 dst-port 5678 action 1 pick a dst-port that works or just redirect all udp4 packets to 1 and see if they are being sent to 1.
I did not see the CNDP .jsonc file you are using, please post it here as sending things via email can get blocked.
from cndp.
liuzhen111
commented on July 28, 2024
from cndp.
liuzhen111
commented on July 28, 2024
I reset ethtool as follows:
sudo ethtool -K ens6f1 ntuple off
sudo ethtool -K ens6f1 ntuple on
sudo ethtool -N ens6f1 flow-type udp4 action 1 loc 1
from cndp.
KeithWiles
commented on July 28, 2024
I would change to the following, just to increase the number of mbufs for each port. It may not be this problem, but I think it needs to be more.
"umems": {
"umem0": {
"bufcnt": 64, // <--
"bufsz": 2,
"mtype": "2MB",
"regions": [
32, // <--
32 // <--
],
"rxdesc": 0,
"txdesc": 0,
"description": "UMEM Description 0"
}
}
from cndp.
KeithWiles
commented on July 28, 2024
In the above ethtool output you have packets on RX-1 and TX-2. Meaning a unidirectional traffic are being received on rx-1 and sent to tx-2. Are you sending traffic in the other direction?
from cndp.
liuzhen111
commented on July 28, 2024
@KeithWiles Server B is linux normal wireguard(without cndp)
from cndp.
KeithWiles
commented on July 28, 2024
I assume ARP resolves to the correct IP and MAC addresses on both machines after the ping command?
All of the non-udp4 frames should be ending up on queue 0 the default and processed by the linux kernel, which includes ARP and ICMP echo requests. Both machines should be able to ping each other.
from cndp.
KeithWiles
commented on July 28, 2024
Setup a ping between the two machines to run continuously, then try WG and see if that helps. Assuming the two machines are ping correctly. Ping from both machines at the same time. Just having one should work, but just to verify traffic is moving between the two machines while trying the WG application.
from cndp.
liuzhen111
commented on July 28, 2024
After about 20 packets, they can't ping each other again.
Re-execute./cndprustwg.sh and ping each other again for a short time.from cndp.
KeithWiles
commented on July 28, 2024
Does the ARP entries age out after it stops? arp -n
from cndp.
liuzhen111
commented on July 28, 2024
yes ，
When ping is aborted, it is displayed here
192.168.60.1 (incomplete) ens6f1
from cndp.
KeithWiles
commented on July 28, 2024
I see, this means something is deleting the ARP entries or they are arging out some how.
from cndp.
liuzhen111
commented on July 28, 2024
I see, this means something is deleting the ARP entries or they are arging out some how.
The problem has been resolved with a replacement of the peer server and now ping is fine.
from cndp.
liuzhen111
commented on July 28, 2024
@KeithWiles thank you very much!
from cndp.
Related Issues (20)
- remove privileged lport flag from rust implementation
- LPORT_UNPRIVILEGED for UDS should be configured under the hood rather than the CNDP app HOT 1
- update cndp af_xdp plugin integration notes to remove ClusterType
- ip_forward node fib lookup tightly coupled with arp entry HOT 1
- cnet: ip4_forward node fib lookup tightly coupled with arp entry HOT 7
- Difficulties in using the wireguard cndp on the public network
- quicly example looks no more working/building HOT 45
- cndp performance reducing when both ingress and egress traffic is there. HOT 15
- Support loading custom bpf programs through the jsonc configuration HOT 1
- Long network device names HOT 4
- Golang: fixup uds to be on a per port basis
- Rust: fixup uds to be on a per port basis
- Documentation for AF_XDP deployment with CNDP it out of date - containerd
- Documentation for AF_XDP deployment with CNDP it out of date - crio
- Add a param to allow one to specify the queue to the jsonc configuration file
- PMD Constructors Priority should be less than max. HOT 2
- Add CNDP to Fedora Copr
- `pktdev_close_all` is counter-intuitive HOT 1
- Run Memory Profiling using Valgrind
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cndp.