xilinx / open-nic Goto Github PK

AMD OpenNIC Project Overview

License: Apache License 2.0

Shell 100.00%

fpga networking network-acceleration smartnic datacenter

open-nic's Introduction

AMD OpenNIC Project

The OpenNIC project provides an FPGA-based NIC platform for the open source community. It consists of multiple components: a NIC shell, a Linux kernel driver, and a DPDK driver. The NIC shell contains the RTL sources and design files for targetting several of the AMD-Xilinx Alveo boards featuring UltraScale+ FPGAs. It delivers a NIC implementation supporting up to four PCI-e physical functions (PFs) and two 100Gbps Ethernet ports. The shell is equipped with well-defined data and control interfaces and is designed to enable easy integration of user logic into the shell. A block diagram of the OpenNIC shell follows:

The Linux kernel driver implements the device driver for the NIC shell. It supports multiple PFs and multiple TX/RX queues in each PF. The RX queues are selected through a receive-side scaling (RSS) implementation in the shell.

The goal of OpenNIC is to enable fast prototyping of hardware-accelerated network-attached applications. It is not a fully-fledged SmartNIC solution.

The latest version of OpenNIC is 1.0, which uses OpenNIC shell version 1.0 and OpenNIC driver version 1.0 or OpenNIC DPDK driver version 1.0.

Repo Structure

This repository serves as the release point for the OpenNIC project, which consists of three components:

A released version of OpenNIC pins to a commit in the master branch of each component repository.

A Bash script script/checkout.sh is provided to checkout a specific version of OpenNIC. It takes two arguments, the root directory for the cloned repositories and optionally, a version number. By default, it will checkout the latest version. The correspondence between OpenNIC versions and component repository tags are tracked in script/version.yaml.

Technical Reference Guide

A technical reference guide (PDF version or MS Word version) provides details of the design of the OpenNIC. The document primarily covers the hardware architecture and its related implementation. It also briefly describes the organization of the Linux kernel driver for OpenNIC.

FAQ

A set of frequently asked questions has been prepared to help in answering questions regarding this project. The FAQ has sections for: (a) general questions, (b) feature set questions, (c) hardware questions, (d) software questions, and (e) operation questions. Please contact us to submit any additional questions that you feel would help others.

Copyright Notice and Disclaimer

This file contains confidential and proprietary information of Advanced Micro Devices-Xilinx and is protected under U.S. and international copyright and other intellectual property laws.

DISCLAIMER

This disclaimer is not a license and does not grant any rights to the materials distributed herewith. Except as otherwise provided in a valid license issued to you by Xilinx, and to the maximum extent permitted by applicable law: (1) THESE MATERIALS ARE MADE AVAILABLE "AS IS" AND WITH ALL FAULTS, AND XILINX HEREBY DISCLAIMS ALL WARRANTIES AND CONDITIONS, EXPRESS, IMPLIED, OR STATUTORY, INCLUDING BUT NOT LIMITED TO WARRANTIES OF MERCHANTABILITY, NONINFRINGEMENT, OR FITNESS FOR ANY PARTICULAR PURPOSE; and (2) Xilinx shall not be liable (whether in contract or tort, including negligence, or under any other theory of liability) for any loss or damage of any kind or nature related to, arising under or in connection with these materials, including for any direct, or any indirect, special, incidental, or consequential loss or damage (including loss of data, profits, goodwill, or any type of loss or damage suffered as a result of any action brought by a third party) even if such damage or loss was reasonably foreseeable or Xilinx had been advised of the possibility of the same.

CRITICAL APPLICATIONS

Xilinx products are not designed or intended to be fail-safe, or for use in any application requiring failsafe performance, such as life-support or safety devices or systems, Class III medical devices, nuclear facilities, applications related to the deployment of airbags, or any other applications that could lead to death, personal injury, or severe property or environmental damage (individually and collectively, "Critical Applications"). Customer assumes the sole risk and liability of any use of Xilinx products in Critical Applications, subject only to applicable laws and regulations governing limitations on product liability.

THIS COPYRIGHT NOTICE AND DISCLAIMER MUST BE RETAINED AS PART OF THIS FILE AT ALL TIMES.

open-nic's People

Contributors

Stargazers

Watchers

open-nic's Issues

Dual port CMAC drops

Hello,

We are having trouble with the dual port configuration of Open Nic, as we are hoping to receive 50 Gbps per port with tx disabled on both cmacs. ONIC is configured with dual cmacs and a single QDMA instance on the U55 using the DPDK drivers, where each port is configured as their own physical function sharing a single QDMA instance. Further, we have the vfio-pci driver bound to each individual port.

Currently, we can get 80+ Gbps on either port with next to zero packet drops when we send traffic on only a single port. If we send traffic to both ports simultaneously, we drop almost 20% of packets when the speed is higher than 5 Gbps per port. Are we right to assume that we should be able to receive packets on both ports simultaneously? If so, do any modifications need to be made to the opennic dpdk driver register configuration to reduce the contention between ports?

Vitis HLS streaming IP

Hi,

What does the following FAQ means? Vitis HLS can generate AXI Stream interfaces for an IP according to the standard stream protocol. Why specifically we need a wrapper?

### Yes, you can use HLS to design modules that fit within the user logic boxes in the OpenNIC. However, HLS does not generate the streaming interfaces that the OpenNIC expects, so you will have to create your own RTL wrappers to connect the HLS module to these interfaces.

Where can I get DPDK driver?

FAQ says

What SW support can I expect to use, for example, DPDK, eBPF, VM, etc.?

Xilinx provides the linux open-nic-driver sources, and an excellent DPDK driver has been developed by the community.

Where can I get it ?
DPDK can be used for open-nic?

Card shown in lspci but not in lshw or /sys/devices/pci0000:xx

TLDR;

I was trying to do the loopback test as mentioned in open-nic-driver. But the card (Alveo U50 with open nic shell) does not show up in /sys/devices, it does show up in ifconfig and lspci. Unsure what might be going on here.

Steps I followed:

I was trying out open nic shell and driver (both main branches and not the v1.0). I am using a Supermicro SYS2029GP-TR server with Ubuntu 18.04, Vivado 2020.2, and Linux kernel 4.15.0-161-generic.
I loaded open nic shell bitstream onto an Alveo U50. On loading the bitstream (direct program, i.e, non MCS method) the first time (when the Alveo card had factory gold image), the server crashed and rebooted automatically. After this there wasn't any Xilinx devices shown in lspci. JTAG was still accessible, I re-loaded the open nic shell bitstream (non MCS method) (no crash this time) and did a warm reboot. After this the card does show up in lspci as a memory controller:
3b:00.0 Memory controller: Xilinx Corporation Device 903f

Next I loaded the onic.ko kernel module. No errors (apart from kernel module verification) were seen in dmesg. A new interface shows in ifconfig -a as enp59s0. I do sudo ifconfig enp59s0 10.1.212.190 netmask 255.255.255.0 up to bring it up and assign it an IP address.

Then I tried to test loopback following instructions on the open nic kernel driver page.
It mentions following command to enable loopback:

sudo ./pcimem /sys/devices/pci0000:d7/0000:d7:00.0/0000:d8:00.0/resource2 0x8090 w 0x1

lspci showed bus as 0000:3b:00.0. This is also shown in sudo ethtool -i enp59s0 as:

driver: onic
version: 0.21
firmware-version:
expansion-rom-version:
bus-info: 0000:3b:00.0
supports-statistics: no
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

However, there is no directory for this pcie device (e.g., pci0000:3b) in /sys/devices/. ls /sys/devices | grep pci0000 shows:

pci0000:00
pci0000:17
pci0000:3a
pci0000:5d
pci0000:80
pci0000:85
pci0000:ae
pci0000:d7

Thus the pcimem command won't work for this the device's bus.
The Alveo board also does not show up in sudo lshw -businfo.

Can it support sn1000?

From some command in driver, it seems do. But in shell did not show it in list.

Any help?

Any performance data avaliable?

what's the performance of this OPEN-NIC?
any experiments conducted?

RX not aligned on U50

Environments

Ubuntu 20.04.5 LTS
kernel 5.15.0-53-generic
vivdao 2020.2
Alveo U50

What did i do?

I generate bitstream and loaded it onto an Alveo U50 and reboot the machine. After that I try to insmod onic.ko.

What happend?

After reboot I can see the 01:00.0 Memory controller: Xilinx Corporation Device 903f on lspci result. But when I try to insmod I got CMAC 0 RX not aligned after waiting in the dmesg log and I cannot ping another machine it connected to.

The full log is following:

[  213.858893] OpenNIC Linux Kernel Driver 0.21
[  213.859161] onic 0000:01:00.0 onic1s0f0 (uninitialized): Set MAC address to 00:0a:35:9d:56:f1
[  213.859169] onic 0000:01:00.0: device is a master PF
[  213.859380] onic 0000:01:00.0: Allocated 8 queue vectors
[  213.859440] onic 0000:01:00.0: CMAC 0 RX_STATUS_REG value: 0x000000C0
[  218.832614] onic 0000:01:00.0: CMAC 0 RX_STATUS_REG value: 0x000000C0
[  218.832620] onic 0000:01:00.0: CMAC 0 RX not aligned after waiting
[  218.832627] onic 0000:01:00.0: Number of CMAC instances = 1
[  218.832689] onic 0000:01:00.0: Setup IRQ vector 146 with name onic1s0f0-0
[  218.832701] onic 0000:01:00.0: Setup IRQ vector 147 with name onic1s0f0-1
[  218.832713] onic 0000:01:00.0: Setup IRQ vector 148 with name onic1s0f0-2
[  218.832725] onic 0000:01:00.0: Setup IRQ vector 149 with name onic1s0f0-3
[  218.832736] onic 0000:01:00.0: Setup IRQ vector 150 with name onic1s0f0-4
[  218.832746] onic 0000:01:00.0: Setup IRQ vector 151 with name onic1s0f0-5
[  218.832758] onic 0000:01:00.0: Setup IRQ vector 152 with name onic1s0f0-6
[  218.832768] onic 0000:01:00.0: Setup IRQ vector 153 with name onic1s0f0-7
[  218.838351] onic 0000:01:00.0 enp1s0: renamed from onic1s0f0
[  218.906384] onic 0000:01:00.0 enp1s0: ethtool: onic_get_link port: 0   carrier ok: 1 -- rx status ok: 0
[  218.910071] onic 0000:01:00.0 enp1s0: ethtool: onic_get_link port: 0   carrier ok: 1 -- rx status ok: 0
[  219.840626] IPv6: ADDRCONF(NETDEV_CHANGE): enp1s0: link becomes ready
[  220.041218] userif-3: sent link down event.
[  220.041225] userif-3: sent link up event.

I checked the CMAC document and learned that the RX_STATUS_REG 0xC0 means "rx_internal_local_fault". Could you please give me some suggestions for troubleshooting or some ideas of potential problems?

10 Gbps support

Hello everyone,

can the OpenNIC support 10 Gbps speed?

Thank you, best regards.

Hi, is there a whole HDL project of the OpenNIC that we can download?

Alveo u200 "Packet Length Mismatch" during DPDK pktgen and testpmd application execution

Hi Team
I am working on OpenNIC design for au200, and I have diligently followed the steps provided in https://github.com/Xilinx/open-nic-dpdk.

However, I am encountering an issue while executing the pktgen application, specifically a "Packet Length Mismatch" error. This discrepancy is resulting in lower-than-expected Tx/Rx values and packet drops. Furthermore, the RX functionality appears to stop altogether, resulting in a throughput of only 9Gbps, significantly lower than the target of 100Gbps.

Here is the command I am using to run pktgen:

./pktgen-dpdk-pktgen-20.11.3/usr/local/bin/pktgen -a 08:00.0 -a 08:00.1 -d librte_net_qdma.so -l 4-10 -n 4 -a 03:00.0 -a 03:00.0 -- -m [6:7].0 -m [8:9].1

Error Message
"Timeout on request to dma internal csr register", "Packet length mismatch error" and "Detected Fatal length mismatch"

Error generated,
C2H_STAT_S_AXIS_C2H_ACCEPTED 0xa88 0x110cb 69835
C2H_STAT_S_AXIS_WRB_ACCEPTED 0xa8c 0x10ed0 69328
C2H_STAT_DESC_RSP_PKT_ACCEPTED 0xa90 0x10ed1 69329
C2H_STAT_AXIS_PKG_CMP 0xa94 0x10ed1 69329
C2H_STAT_DBG_DMA_ENG_0 0xb1c 0x48e00304 1222640388
C2H_STAT_DBG_DMA_ENG_1 0xb20 0xe7e40000 -404488192
C2H_STAT_DBG_DMA_ENG_2 0xb24 0x80000000 -2147483648
C2H_STAT_DBG_DMA_ENG_3 0xb28 0x80020813 -2147350509
C2H_STAT_DESC_RSP_DROP_ACCEPTED 0xb10 0x1c8e1 116961
C2H_STAT_DESC_RSP_ERR_ACCEPTED 0xb14 0 0
eqdma_hw_error_process detected Fatal Len mismatch error

I have raised queries in git under below links, no reply yet
Xilinx/open-nic-dpdk#2 (comment)

Please let me know how to resolve this issue.

Can open-nic support XCVP1202?

I'am going to write a net driver based on XCVP1202 and DCMAC refer to open-nic.
I want to know if open-nic can support XCVP1202. And I think it is very likely to support. The reason is as follows.

According to the source code of open-nic and dma_ip_drivers( https://github.com/Xilinx/dma_ip_drivers ), The supported device ids of open-nic are completely contained by dma_ip_drivers. Because dma_ip_drivers supports XCVP1202, So I think open-nic also supports XCVP1202. Of course, some register configurations may need to be adjusted.And I should know the differences of register between to devices.

Is it possible that i think so?