p4lang / p4app-switchml Goto Github PK

View Code? Open in Web Editor NEW

166.0 21.0 45.0 355 KB

Switch ML Application

Home Page: https://switchml.readthedocs.io/

License: Apache License 2.0

Makefile 4.13% C++ 51.08% NASL 3.62% C 0.15% P4 14.37% Shell 0.24% Python 24.27% Dockerfile 2.14%

p4 p4lang tofino tna machine-learning in-network-compute dpdk rdma collectives

p4app-switchml's People

Contributors

Stargazers

Watchers

p4app-switchml's Issues

batch_size

hi，i'm using DPDK.
I have got 3517s when i run pytorch without switchml and 1912s with p4app-switchml ，it speedup to 46%。
However ,when i adjust batch_size=16 to batch_size=256 in ‘main.py’ which is my AI program，i just got 924s without switchml and 916s with p4app-switchml , it just speedup 0.8%。why batch_size become so importance？ i do not know if my 'Tesla V100-PCIE' card reach bottleneck. how about your batch_size .

#1 ./hello_world: error while loading shared libraries: libprotobuf.so.3.11.2.0: cannot open shared object file: No such file or directory #2 Cannot allocate memory [12]

Hi,
I ran make RDMA=1 under p4app-switchML/dev_root/client_lib/, then make RDMA=1 under p4app-switchML/dev_root/examples/, and finally ran . /hello_world under p4app-switchML/dev_root/ build/bin/ . But it shows: "./hello_world: error while loading shared libraries: libprotobuf.so.3.11.2.0: cannot open shared object file: No such file or directory".

Environment:

Ubuntu16.04
Python 3.8.3
ConnectX-6 NIC

The acceleration effect does not meet the requirements。

i can run switchml completely。however ,i can not get the ideal results。
i got 8785 seconds without switchml and got 9067 seconds with p4app-switchml ，why it have not acceleration effect at all , what can i do to adjust parameter。
use switchml

without switchml

switchml.cfg

which parameter should i try ?

Problem using SwitchML with PCI NIC virtual function

Hi,

I am running SwitchML allreduce_benchmarks on a cluster of nodes with a mix of MLX5 NICs and some with Intel 82599 ES 10G NICs thus I'm using DPDK as the communication backend. I need to share the NIC on each host with other traffic so I'm virtualizing it by creating a virtual function of the PCI device in order to use the original device for general purpose traffic and the virtual device to run the SwitchML app. However, when I try to run SwitchML with the virtual device, I'm getting the following error:

Submitting 5 warmup jobs.
EAL: Detected 20 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: No available hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:01:10.1 on NUMA socket 0
EAL:   probe driver: 8086:10ed net_ixgbe_vf
EAL:   using IOMMU type 1 (Type 1)
E1116 19:08:30.807562 74629 dpdk_master_thread_utils.inc:277] Flow isolated mode failed: 1 Function not implemented
F1116 19:08:31.361418 74629 dpdk_master_thread_utils.inc:154] Flow rule can't be added: 1Function not implemented
*** Check failure stack trace: ***
    @     0x7f2291e280cd  google::LogMessage::Fail()
    @     0x7f2291e29f33  google::LogMessage::SendToLog()
    @     0x7f2291e27c28  google::LogMessage::Flush()
    @     0x7f2291e2a999  google::LogMessageFatal::~LogMessageFatal()
    @     0x564504203fb2  switchml::InsertFlowRule()
    @     0x5645042048ca  switchml::InitPort()
    @     0x564504205ae6  switchml::DpdkMasterThread::operator()()
    @     0x7f2291ae34c0  (unknown)
    @     0x7f22915766db  start_thread
    @     0x7f229015761f  clone

which seems to be caused by struct rte_flow_error error; LOG_IF(FATAL, rte_flow_validate(port_id, &attr, pattern, action, &error) != 0) << "Flow rule can't be added: " << error.type << (error.message ? error.message : "(no stated reason)"); in InsertFlowRule function. Any ideas on why this is happening and whether I can overcome this? Much appreciate it.

Thank you.

Using multiple pipes on the same job

Hi,

This issue is related to #33. I am using a BF2556-1T switch with two pipes on the Tofino chip and I'm trying to use its QSFP front panel ports which are unfortunately split between pipe 1 and pipe 3 of the switch. Since I'm not using the folded pipe option, I've changed the parser logic in order to not reserve pipes 1-3 for recirculation as follows:

    state parse_port_metadata {
        // parse port metadata
        ig_md.port_metadata = port_metadata_unpack<port_metadata_t>(pkt);

        transition select(ig_intr_md.ingress_port) {
            64: parse_recirculate; // pipe 0 CPU Eth port
            68: parse_recirculate; // pipe 0 recirc port
            320: parse_ethernet;   // pipe 2 CPU PCIe port
            // 0x080 &&& 0x180: parse_recirculate; // all pipe 1 ports
            // 0x100 &&& 0x180: parse_recirculate; // all pipe 2 ports
            // 0x180 &&& 0x180: parse_recirculate; // all pipe 3 ports
            default:  parse_ethernet;
        }
    }

However, when I execute a job including workers on different pipes, the execution hangs with packets arriving in the switch but no packet is being sent back to the workers as can be seen in the following:

and this is what the bitmap looks like (it seems to be stuck in these values under repeated queries to it):

Is concurrent execution on both pipes feasible? If so, what modifications are required? Is perhaps the fact that registers(for CONSUME and HARVEST) are pipe local somehow related to this? Thank you.

nccl_plugin

Can I use nccl_plugin or switchml_nccl.patch in current p4app-SwitchML ?
I don't konw how it works， if node send Roce packets to P4 switch instead of another node when i use nccl_plugin or switchml_nccl.patch ？

Controller configuration & port loopback error

Hi,

I am trying to benchmark switchML on a Tofino switch(APS Networks BF2556X-1T-A1F) and I'm having some difficulty running the controller. I think I have managed to compile the P4 program and deploy it on the switch. Then when I'm trying to execute the controller, I'm getting the following output from the python switchml.py script:

$ python switchml.py 
Subscribe attempt #1
Subscribe response received 0
Binding with p4_name SwitchML
Binding with p4_name SwitchML successful!!
Received SwitchML on GetForwarding on client 0, device 0
Setting 49 front panel ports in loopback mode
Unexpected error. Stopping controller.
Terminated

and the bf_switchd throws the following errors:

2022-03-09 16:33:56.086991 BF_PLTFM ERROR - bf_pm_pre_port_add_cfg_set: Unsupported speed dev : 0 : front port : 41/0 port-cfg-nlanes:4 invalid channel:1 
2022-03-09 16:33:56.087064 BF_PM ERROR - bf_pm_port_add_with_lanes:4897 Error in setting pltfm pre-port-add cfg for dev : 0 : front port : 41/0 (128) :  cfg-speed:100g n-lanes:4 err: Invalid arguments (3)
2022-03-09 16:33:56.087094 BF_PAL ERROR - Unable to add port for dev : 0 : front port : 41/0 (128) : Invalid arguments (3)
2022-03-09 16:33:56.087122 BF_BFRT ERROR - BF_RT_SERVER:write_entry:3111 Table Add failed table:$PORT Invalid arguments
2022-03-09 16:33:56.087210 BF_PLTFM ERROR - bf_pm_pre_port_add_cfg_set: Unsupported speed dev : 0 : front port : 45/0 port-cfg-nlanes:4 invalid channel:1 
2022-03-09 16:33:56.087237 BF_PM ERROR - bf_pm_port_add_with_lanes:4897 Error in setting pltfm pre-port-add cfg for dev : 0 : front port : 45/0 (132) :  cfg-speed:100g n-lanes:4 err: Invalid arguments (3)
2022-03-09 16:33:56.087262 BF_PAL ERROR - Unable to add port for dev : 0 : front port : 45/0 (132) : Invalid arguments (3)
2022-03-09 16:33:56.087286 BF_BFRT ERROR - BF_RT_SERVER:write_entry:3111 Table Add failed table:$PORT Invalid arguments
2022-03-09 16:33:56.087364 BF_PLTFM ERROR - bf_pm_pre_port_add_cfg_set: Unsupported speed dev : 0 : front port : 37/0 port-cfg-nlanes:4 invalid channel:1 
2022-03-09 16:33:56.087389 BF_PM ERROR - bf_pm_port_add_with_lanes:4897 Error in setting pltfm pre-port-add cfg for dev : 0 : front port : 37/0 (136) :  cfg-speed:100g n-lanes:4 err: Invalid arguments (3)
2022-03-09 16:33:56.087412 BF_PAL ERROR - Unable to add port for dev : 0 : front port : 37/0 (136) : Invalid arguments (3)
2022-03-09 16:33:56.087434 BF_BFRT ERROR - BF_RT_SERVER:write_entry:3111 Table Add failed table:$PORT Invalid arguments
2022-03-09 16:33:56.343103 BF_PLTFM ERROR - bf_pm_pre_port_add_cfg_set: Unsupported speed dev : 0 : front port : 33/0 port-cfg-nlanes:4 invalid channel:1 
2022-03-09 16:33:56.343144 BF_PM ERROR - bf_pm_port_add_with_lanes:4897 Error in setting pltfm pre-port-add cfg for dev : 0 : front port : 33/0 (144) :  cfg-speed:100g n-lanes:4 err: Invalid arguments (3)
2022-03-09 16:33:56.343174 BF_PAL ERROR - Unable to add port for dev : 0 : front port : 33/0 (144) : Invalid arguments (3)
2022-03-09 16:33:56.343201 BF_BFRT ERROR - BF_RT_SERVER:write_entry:3111 Table Add failed table:$PORT Invalid arguments
2022-03-09 16:33:56.343252 BF_PAL ERROR - Unable to get the front port handle for dev : 0 : dev_port : 148 : Invalid arguments (3)
2022-03-09 16:33:56.343283 BF_BFRT ERROR - BF_RT_SERVER:write_entry:3111 Table Add failed table:$PORT Invalid arguments
2022-03-09 16:33:56.343341 BF_PAL ERROR - Unable to get the front port handle for dev : 0 : dev_port : 152 : Invalid arguments (3)
2022-03-09 16:33:56.343370 BF_BFRT ERROR - BF_RT_SERVER:write_entry:3111 Table Add failed table:$PORT Invalid arguments
2022-03-09 16:33:56.599107 BF_PLTFM ERROR - bf_pm_pre_port_add_cfg_set: Unsupported speed dev : 0 : front port : 29/0 port-cfg-nlanes:4 invalid channel:1 
2022-03-09 16:33:56.599154 BF_PM ERROR - bf_pm_port_add_with_lanes:4897 Error in setting pltfm pre-port-add cfg for dev : 0 : front port : 29/0 (160) :  cfg-speed:100g n-lanes:4 err: Invalid arguments (3)

....

(Continues similarly for the other ports)

I expect this to have something to do with the exact configuration of my switch maybe? I have also included the switchml.log file.

Controller Log:
switchml.log

Environment:

Tofino Switch: APS Networks BF2556X-1T-A1F
SDE: 9.6.0
Python: 3.8.10

thank you,

make compilaton error

p4@p4:~/intel/p4app-switchML/dev_root/examples$ make
DEBUG is not set. Enabling all optimizations.
g++ -std=c++17 -DNDEBUG -O3 -I/home/p4/intel/p4app-switchML/dev_root/build/include /home/p4/intel/p4app-switchML/dev_root/examples/hello_world/main.cc -L/home/p4/intel/p4app-switchML/dev_root/build/lib -lswitchml-client -lglog -lstdc++ -lboost_program_options -lpthread -o /home/p4/intel/p4app-switchML/dev_root/build/bin/hello_world
/home/p4/intel/p4app-switchML/dev_root/examples/hello_world/main.cc:23:10: fatal error: switchml/context.h: No such file or directory
23 | #include <switchml/context.h>
| ^~~~~~~~~~~~~~~~~~~~
compilation terminated.
make: *** [Makefile:103: /home/p4/intel/p4app-switchML/dev_root/build/bin/hello_world] Error 1

No Ethernet ports

here are my nic info:

Device #1:

Device Type: ConnectX4
Part Number: MCX415A-CCA_Ax
Description: ConnectX-4 EN network interface card; 100GbE single-port QSFP28; PCIe3.0 x16; ROHS R6
PSID: MT_2140110033
PCI Device Name: d8:00.0
Base GUID: 506b4b03000fb5c4
Base MAC: 506b4b0fb5c4
Versions: Current Available
FW 12.28.2006 12.28.2006
PXE 3.6.0102 3.6.0102
UEFI 14.21.0017 14.21.0017
Status: Up to date
When i use it for switchml train， got errors like this：

EAL: Selected IOVA mode 'PA'
EAL: No available hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: WARNING! Base virtual address hint (0x100a94000 != 0x7fe4d3e00000) not respected!
EAL: This may cause issues with mapping memory into secondary processes
EAL: WARNING! Base virtual address hint (0x1016f5000 != 0x7fe0d3c00000) not respected!
EAL: This may cause issues with mapping memory into secondary processes
EAL: WARNING! Base virtual address hint (0x102356000 != 0x7fdcd3a00000) not respected!
EAL: This may cause issues with mapping memory into secondary processes
EAL: WARNING! Base virtual address hint (0x102fb7000 != 0x7fd8d3800000) not respected!
EAL: This may cause issues with mapping memory into secondary processes
EAL: WARNING! Base virtual address hint (0x103c18000 != 0x7fd4d3600000) not respected!
EAL: This may cause issues with mapping memory into secondary processes
EAL: WARNING! Base virtual address hint (0x104879000 != 0x7fd0d3400000) not respected!
EAL: This may cause issues with mapping memory into secondary processes
EAL: WARNING! Base virtual address hint (0x1054da000 != 0x7fccd3200000) not respected!
EAL: This may cause issues with mapping memory into secondary processes
EAL: WARNING! Base virtual address hint (0x10613b000 != 0x7fc8d3000000) not respected!
EAL: This may cause issues with mapping memory into secondary processes
F0707 03:14:46.742125 28820 dpdk_master_thread_utils.inc:190] No Ethernet ports
*** Check failure stack trace: ***
@ 0x7fe93c7d40cd google::LogMessage::Fail()
@ 0x7fe93c7d5f33 google::LogMessage::SendToLog()
@ 0x7fe93c7d3c28 google::LogMessage::Flush()
@ 0x7fe93c7d6999 google::LogMessageFatal::~LogMessageFatal()
@ 0x7fe94ebd8f01 switchml::InitPort()
@ 0x7fe94ebda811 switchml::DpdkMasterThread::operator()()
@ 0x7fe95b65dbf4 execute_native_thread_routine
@ 0x7fe974ce36db start_thread
@ 0x7fe974a0c61f clone
switchml cann't find Ethernet ports ，if it has requirements for network card type ？what can i do for this .

when i run switchml program，should i put nic into dpdk status manually ?

(base) root@work4:/p4app-switchML/dev_root/third_party/dpdk/usertools# ./dpdk-devbind.py -s

Network devices using DPDK-compatible driver

0000:86:00.0 'Ethernet Controller X710 for 10GbE backplane 1581' drv=igb_uio unused=i40e
0000:d8:00.0 'MT27700 Family [ConnectX-4] 1013' drv=igb_uio unused=mlx5_core

Unable to run multiple SwitchML instances on same machine

Hi,

I'm trying to utilize both of my MLX5 NIC's ports for an experiment and I'm having trouble running the allreduce_benchmark when two instances are executed on the same machine. I have programmatically set the configuration for each instance by overloading the client library constructor to accept a configuration file path. I'm thus explicitly telling one instance to use mlx5_0 and the other mlx5_1. However, the execution seems to hang after Submitting 5 warmup jobs., so jobs never finish. Is there perhaps a specific reason why two instances of SwitchML jobs cannot start on the same machine? Is somehow the client lib a singleton and cannot support multiple SwitchML jobs on different NIC ports?

Thanks

RDMA helloworld got stuck at "receive and send loop" and no traffic sent

Hi!
I am trying to run the helloworld program with RDMA client. I am using a mallenox ConnectX-5 NIC and connecting it directly to a tofino-32D. I build the docker image with rdma_pytorch.dockerfile, which is modified to use predownloaded source code and use system NCCL. Then I start the container with the following command:

docker run -itd --gpus all --net=host --cap-add=IPC_LOCK --device=/dev/infiniband/uverbs0 --name sml_rdma_env --privileged switchml_rdma

After that, i run helloworld with that NIC and get stuck at "Worker thread '0' is starting the receive and send loop", and the controller with 'show_rdma_workers' command shows correct IP address and MAC address with all 0 sent and recv.
Here is my configuration:

[general]
rank = 0
num_workers = 1
num_worker_threads = 1
max_outstanding_packets = 256
packet_numel = 64
backend = rdma
scheduler = fifo
prepostprocessor = cpu_exponent_quantizer
instant_job_completion = 0
controller_ip_str = 192.168.10.3
controller_port = 50099
timeout = 10
timeout_threshold = 100
timeout_threshold_increment = 100
--(derived)--
max_outstanding_packets_per_worker_thread = 256
[backend.rdma]
msg_numel = 256
device_name = mlx5_0
device_port_id = 1
gid_index = 3
--(derived)--
num_pkts_per_msg = 4
max_outstanding_msgs = 64
max_outstanding_msgs_per_worker_thread = 64

I have also run disable-icrc.sh, althought i dont think it is related.
Any advice on where the problem lies? Or what can i do to get more infomation?

Connection refused error on starting SwitchML reference driver on Tofino Model

I was able to compile the P4 program with P4Studio SDE 9.9.0 on Tofino model (not the real h/w, just the tofino model)
However, when I run driver switchd, I recurrently get the following error: connect failed. Error: Connection refused.

Any idea what could be causing this? Let me know if you need me to share any specific logs. Thanks!

How to get MLX5_0 REGS？

vectorclass.h

hi,I can't find the code of "vectorclass.h"

Errors when building Pytorch patch and plugin

When I built the pytorch patchand plugin, I got the following errors but no errors reported before them:

Error message when building the patch:

(base) sora@fengkeyleaf-rog:~/pytorch$ git apply ~/switchml/dev_root/build/switchml_pytorch.patch
/home/sora/switchml/dev_root/build/switchml_pytorch.patch:50: trailing whitespace.
#The following lines should be filled automatically using the pytorch patch makefile to
error: patch failed: torch/csrc/distributed/c10d/init.cpp:1
error: torch/csrc/distributed/c10d/init.cpp: patch does not apply
error: patch failed: torch/lib/c10d/CMakeLists.txt:1
error: torch/lib/c10d/CMakeLists.txt: patch does not apply
error: patch failed: torch/lib/c10d/ProcessGroupGloo.cpp:1
error: torch/lib/c10d/ProcessGroupGloo.cpp: patch does not apply

Error message when building the plugin:

(/home/sora/switchml/dev_root/frameworks_integration/pytorch_plugin/path_to_env) sora@fengkeyleaf-rog:~/switchml/dev_root/frameworks_integration/pytorch_plugin$ python setup.py install
Traceback (most recent call last):
File "setup.py", line 37, in
from torch.utils import cpp_extension
File "/home/sora/switchml/dev_root/frameworks_integration/pytorch_plugin/path_to_env/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 23, in
from pkg_resources import packaging # type: ignore[attr-defined]
ImportError: cannot import name 'packaging' from 'pkg_resources' (/home/sora/switchml/dev_root/frameworks_integration/pytorch_plugin/path_to_env/lib/python3.8/site-packages/pkg_resources/init.py)

I wonder if there are some ways to solve the problem or anyone else has the same error before? By the way, my OS info:

No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.6 LTS
Release: 20.04
Codename: focal

client_lib/ABI_VERSION: No such file or directory

when I exec make DPDK=1 in path dev_root, it shows
cat: /p4app-switchML/dev_root/client_lib/ABI_VERSION: No such file or directory ,
Does it matter if this prompt is left alone?

No module named 'switchml_pb2'

Hello,
I try to run SwitchML as a benchmark, but encountered a problem when trying to run the python controller as below:

root@localhost:~/Projects/git/p4app-switchML/dev_root/controller# python switchml.py 
Traceback (most recent call last):
  File "switchml.py", line 45, in <module>
    from grpc_server import GRPCServer
  File "/root/Projects/git/p4app-switchML/dev_root/controller/grpc_server.py", line 20, in <module>
    import switchml_pb2
ModuleNotFoundError: No module named 'switchml_pb2'

Environment:

SDE version: 9.3.0
Python version: 3.8.11

Looking forward to your reply.

Performance discussion.

Hi，
I'm a little confused about packets and folding pipes. One pipeline processes 64 elements data at a time. Is this the limitation of Tofino chip? If yes, can the data of 128 elements or 256 elements only be processed through the 2-fold pipeline and the 4-fold pipeline, or can the data of 128 elements and 256 elements not be processed on the switch with only one pipe.

ERROR - '$PKTGEN_PORT_CFG'

Hi，
I'm trying to benchmark SwitchML on a Tofino switch (Wedge100BF-32X-O-AC-F), but I'm having some difficulty running the controller. I think I have successfully compiled the P4 program and deployed it on the switch. When I run the controller under "/dev_root/controller" on the switch, it gives me the following error.

Controller Log:

Environment:

Tofino Switch: Wedge100BF-32X-O-AC-F
SDE: 9.8.0
Python:3.6.9

Thank you.

hello_world example not sending any traffic

Hi,

I am trying to benchmark SwitchML and I have managed to compile the P4 program and deploy it on the switch. I am also running the controller on the switch directly and I have connected two different machines on the switch and I'm trying to run the hello_world example. I have modified the switchml.cfg file accordingly by setting the backend to rdma and the relevant values for the rest (num_workers=2). However, when I start the example from the two different machines, they manage to connect with the controller running on the switch but they seem to hang afterwards waiting for the jobs to finish. Specifically, the output of the program is the following:

Hello world!. Starting the switchml context
Allocating data
Initializing data
Submiting all reduce jobs
Waiting for all jobs to finish

On the controller side, I am able to query with show_rdma_workers for which I do get two entries for the workers with the correct worker ids and MACs (the IPs are all 0 but I assume this is because testing over RDMA). However, the counters for received/sent packets are all zero. Furthermore, on the client machines, the TX packets on the corresponding interface do not increase (at least in the short time after the start of execution). This makes me question whether traffic is indeed generated and whether it reaches the switch. Any thoughts perhaps on what could cause this or potential misconfiguration ? Thanks.

Environment:

Tofino Switch: APS Networks BF2556X-1T-A1F
SDE: 9.6.0
Python: 3.8.10
NICs:ConnectX-5

Thanks

Port traffic redirection with logic bypass

Hi,

I would like to benchmark SwitchML on an emulated topology bigger than that of a single rack. To do so, I plan to reuse the same Tofino switch(APS Networks BF2556X-1T) with SwitchML deployed on it and redirect all traffic coming in from a specific port into another port without any processing done. That port would then be physically connected to yet another port which will consume the traffic regularly via the SwitchML logic.

I was wondering whether this is something that is already possible with the existing P4 model and SwitchML controller program? If not, is there a way I could modify the Forwarder table in forwarder.p4 (populated via the forwarder.py on the control plane) to accommodate this functionality without messing the underlying SwitchML logic? Practically, I need the traffic from that specific port to bypass the SwitchML aggregation logic in the switch and just be redirected out of another port (and vice versa for the return journey). From what I understand, currently, the forwarder table matches on ethernet addresses. Could this be somehow extended to accommodate the case without jeopardizing the underlying functionality of the P4 program?

Thank you,

Could not start the context due to missing configuration.

Hello. I ran make RDMA=1 under p4app-switchML/dev_root/client_lib/, then ran make RDMA=1 under p4app-switchML/dev_root/examples/, finally ran ./hello_world under p4app-switchML/dev_root/build/bin/. I found Line 32: ctx.Start() in p4app-switchML/dev_root/examples/hello_world/main.cc could not run correctly and the process aborted as the following fig. shows.

So how to run the hello_world example correctly? Thanks in advance for any help!

Programs stucked with "submitting 5 warmup jobs" printed.

Hello. I ran allreduce_benchmark on two servers and found that both programs stucked with "submitting 5 warmup jobs" printed on the console. I had checked that in the switchml.cfgs on both servers, the num_workers was 2, and the rank was 0 for one while 1 for the other. What should I do next to debug it? Thanks in advance for any help!

Failed to integrate switchml into Pytorch by applying pytorch patch.

I followed the README in dev_root/frameworks_integration/pytorch_patch/README.md step by step. In the third step "Apply the switchml pytorch patch to pytorch", I make with no variable flags. And finally, I met the following problem:

/home/frank/Mpytorch/torch/lib/c10d/ProcessGroupGloo.cpp:49:10: fatal error: switchml/context.h: No such file or directory
#include <switchml/context.h>

Please help me!

No traffic is communicated due to timeout

I have setup SwitchML P4 app and controller successfully and compiled the client library for RDMA backend but I'm having a similar issue as #8 that no communication happens after worker setup. I have two workers with disabled ICRC and have tried this both for hello_world example and allreduce benchmark. Here is the output of GLOG_logtostderr=1 GLOG_v=2 ./allreduce_benchmark for one of the workers (the other worker has a similar output and the behavior is the same for hello_world as well):

 ../third_party/stdarg.h ?? ../../switchml_bkp.cfg
I0902 11:34:34.264950 18461 context.cc:64] Starting switchml context.
I0902 11:34:34.265347 18461 config.cc:139] Using this configuration file 'switchml.cfg'.
I0902 11:34:34.265789 18461 config.cc:216] Printing configuration
I0902 11:34:34.265799 18461 config.cc:219] 
[general]
    rank = 1
    num_workers = 2
    num_worker_threads = 4
    max_outstanding_packets = 4
    packet_numel = 64
    backend = rdma
    scheduler = fifo
    prepostprocessor = cpu_exponent_quantizer
    instant_job_completion = 0
    controller_ip_str = 10.0.0.1
    controller_port = 50099
    timeout = 10000
    timeout_threshold = 100
    timeout_threshold_increment = 100
    --(derived)--
    max_outstanding_packets_per_worker_thread = 1
I0902 11:34:34.265834 18461 config.cc:270] 
[backend.rdma]
    msg_numel = 64
    device_name = mlx5_0
    device_port_id = 1
    gid_index = 3
    --(derived)--
    num_pkts_per_msg = 1
    max_outstanding_msgs = 4
    max_outstanding_msgs_per_worker_thread = 1
I0902 11:34:34.265851 18461 rdma_backend.cc:42] Setting up worker.
I0902 11:34:34.266697 18461 rdma_endpoint.cc:65] Found Verbs device mlx5_0 with guid 0x98039b03008e0d50
I0902 11:34:34.266713 18461 rdma_endpoint.cc:65] Found Verbs device mlx5_1 with guid 0x98039b03008e0d51
I0902 11:34:34.266721 18461 rdma_endpoint.cc:79] Using Verbs device mlx5_0 gid index 3
I0902 11:34:34.290702 18461 rdma_endpoint.cc:116] GID 0 is 0x80fe 0x500d8efeff9b039a
I0902 11:34:34.290791 18461 rdma_endpoint.cc:116] GID 1 is 0x80fe 0x500d8efeff9b039a
I0902 11:34:34.290869 18461 rdma_endpoint.cc:116] GID 2 is 0 0x401a8c0ffff0000
I0902 11:34:34.290951 18461 rdma_endpoint.cc:116] GID 3 is 0 0x401a8c0ffff0000
I0902 11:34:39.579232 18461 context.cc:99] Switchml context started successfully.
Submitting 5 warmup jobs.
I0902 11:34:39.766185 18467 rdma_utils.h:193] Worker 0 bound to core 0 on NUMA node 0
I0902 11:34:39.766194 18469 rdma_utils.h:193] Worker 2 bound to core 2 on NUMA node 0
I0902 11:34:39.766386 18469 rdma_worker_thread.cc:129] Worker 2 QP 0:0x519 using rkey 5 for remote rkey 63210
I0902 11:34:39.766402 18467 rdma_worker_thread.cc:129] Worker 0 QP 0:0x517 using rkey 1 for remote rkey 63210
I0902 11:34:39.773722 18471 rdma_utils.h:193] Worker 3 bound to core 3 on NUMA node 0
I0902 11:34:39.773824 18471 rdma_worker_thread.cc:129] Worker 3 QP 0:0x51a using rkey 7 for remote rkey 63210
I0902 11:34:39.803228 18468 rdma_utils.h:193] Worker 1 bound to core 1 on NUMA node 0
I0902 11:34:39.803387 18468 rdma_worker_thread.cc:129] Worker 1 QP 0:0x518 using rkey 3 for remote rkey 63210

After no progress when I exit the process I get:

^CSignal 2 received, preparing to exit...
I0902 11:37:39.248157 18462 context.cc:105] Stopping switchml context
I0902 11:37:39.248188 18462 scheduler.cc:48] Waking up waiting threads
I0902 11:37:39.248227 18462 rdma_backend.cc:56] Cleaning up worker.
I0902 11:37:39.248417 18462 stats.cc:97] Stats: 
    Submitted jobs: #5#
    Submitted jobs sizes: #[268435456,268435456,268435456,268435456,268435456,]#
    Submitted jobs sizes distribution: #Sum: 1342177280 Mean: 268435456.0000 Max: 268435456  Min: 268435456  Median: 268435456  Stdev: 0.0000    #
    Finished jobs: #0#
    Worker thread: #0#
        Total packets sent: #18#
        Total packets received: #0#
        Wrong packets received: #0#
        Correct packets received: #0#
        Number of timeouts: #17#
    Worker thread: #1#
        Total packets sent: #18#
        Total packets received: #0#
        Wrong packets received: #0#
        Correct packets received: #0#
        Number of timeouts: #17#
    Worker thread: #2#
        Total packets sent: #18#
        Total packets received: #0#
        Wrong packets received: #0#
        Correct packets received: #0#
        Number of timeouts: #17#
    Worker thread: #3#
        Total packets sent: #18#
        Total packets received: #0#
        Wrong packets received: #0#
        Correct packets received: #0#
        Number of timeouts: #17#
I0902 11:37:39.248509 18462 context.cc:130] Stopped switchml context
Warmup finished.
Submitting 10 jobs.
Signal handler thread is exiting

Here is the outputs on controller side:

SwitchML>show_switch_address

Switch MAC: 00:11:22:33:44:55 IP: 192.168.1.100


SwitchML>show_rdma_workers

                                                   Received              Sent        
 Worker ID     Worker MAC        Worker IP     Packets  /    Bytes      Packets  /    Bytes    
     0      98:03:9b:83:1a:b2   192.168.1.2       0     /      0           0     /      0      
     1      98:03:9b:8e:3d:ac   192.168.1.4       0     /      0           0     /      0      


SwitchML>show_ports

  Port Up Valid Enabled Speed  FEC    Tx Packets        Tx Bytes        Rx Packets        Rx Bytes        Rx Errors        Tx Errors        FCS Errors   
  1/0  1    1      1    100G  NONE       219             66345              68             10710              0                0                0        
  2/0  1    1      1    100G  NONE       181             60116             106              8959              0                0                0        
  3/0  1    1      1    100G  NONE       265             83132              33              6105              0                0                0        
  4/0  1    1      1    100G  NONE       178             59180             133             12990              0                0                0        


SwitchML>show_statistics

             Broadcasted         Recirculated       Retransmitted          Dropped       
  Index    Set 0     Set 1     Set 0     Set 1     Set 0     Set 1     Set 0     Set 1   
    0        0         0         0         0         0         0         0         0     
    1        0         0         0         0         0         0         0         0     
    2        0         0         0         0         0         0         0         0     
    3        0         0         0         0         0         0         0         0     
    4        0         0         0         0         0         0         0         0     
    5        0         0         0         0         0         0         0         0     
    6        0         0         0         0         0         0         0         0     
    7        0         0         0         0         0         0         0         0

And on the switch side I get:

bf-sde.pm> show
-----+----+---+----+-------+----+--+--+---+---+---+--------+----------------+----------------+-
PORT |MAC |D_P|P/PT|SPEED  |FEC |AN|KR|RDY|ADM|OPR|LPBK    |FRAMES RX       |FRAMES TX       |E
-----+----+---+----+-------+----+--+--+---+---+---+--------+----------------+----------------+-
1/0  |23/0|132|2/ 4|100G   |NONE|Ds|Au|YES|ENB|UP |  NONE  |              68|             219|
2/0  |22/0|140|2/12|100G   |NONE|Ds|Au|YES|ENB|UP |  NONE  |             106|             181|
3/0  |21/0|148|2/20|100G   |NONE|Ds|Au|YES|ENB|UP |  NONE  |              33|             265|
4/0  |20/0|156|2/28|100G   |NONE|Ds|Au|YES|ENB|UP |  NONE  |             133|             178|

My environment is:

Switch: Wedge BF100-32x
SDE: 9.9.0
Python: 3.8
NICs: ConnectX-5

My ports.yaml has:

ports:
    1/0 : {speed: "100G", fec: "none", autoneg: "disable", mac: "98:03:9b:8e:82:98"}
    2/0 : {speed: "100G", fec: "none", autoneg: "disable", mac: "98:03:9b:83:1a:b2"}
    3/0 : {speed: "100G", fec: "none", autoneg: "disable", mac: "98:03:9b:83:34:d2"}
    4/0 : {speed: "100G", fec: "none", autoneg: "disable", mac: "98:03:9b:8e:3d:ac"}

And finally my config file is here:

switchml.cfg.txt

My guess is that the switch data plane as an end point is unreachable for some reason (but only one packet does not timeout so I'm not sure). Is there a way to ensure connectivity between

Thank you!

Error building DPDK for client lib

Hi,

I'm trying to build the client lib with DPDK as the communication backend but I'm getting the following compilation error when trying to build the specific DPDK version from the source in the third_party/dpdk folder

== Build drivers/compress
  Building modules, stage 2.
  MODPOST 1 modules
  Building modules, stage 2.
  MODPOST 1 modules
  CC [M]  p4app-switchML/dev_root/third_party/dpdk/build/build/kernel/linux/igb_uio/igb_uio.mod.o
  CC [M]  p4app-switchML/dev_root/third_party/dpdk/build/build/kernel/linux/kni/rte_kni.mod.o
cc1: error: code model kernel does not support PIC mode
cc1: error: code model kernel does not support PIC mode
make[8]: *** [scripts/Makefile.modfinal:28: p4app-switchML/dev_root/third_party/dpdk/build/build/kernel/linux/igb_uio/igb_uio.mod.o] Error 1
make[8]: *** [scripts/Makefile.modfinal:28: p4app-switchML/dev_root/third_party/dpdk/build/build/kernel/linux/kni/rte_kni.mod.o] Error 1
make[7]: *** [scripts/Makefile.modpost:95: __modpost] Error 2
make[7]: *** [scripts/Makefile.modpost:95: __modpost] Error 2

I see that in the makefile the flag -fPIC is included but researching online about the specific error, people propose avoiding PIE by including -fno-pie. I would appreciate any help on this.

Thanks,

Confusion about the intergration of pytorch.

Hi there,
I read your PyTorch patch, and was confused about the modification, your modification was based on Gloo backend code, which would synchronize the stream every time the procedure call the AllReduce function. Isn't it inefficient? It seems that NCCL backend would put all the AllReduce operations into an async work queue which does not require synchronizing the stream before every AllReduce call.
Best regards

Undefined reference to rte functions when compiling allreduce_benchmark with RDMA=1.

Hello. Sorry to bother again. I try to compile allreduce_benchmark with RDMA=1. However, when I compile it with the command "make allreduce_benchmark RDMA=1 MLX5=1 VCL=0 CUDA=1", the console prints something as follows. It shows that many rte functions used in dpdk_master_thread.cc are not defined. I'm puzzled that why dpdk_master_thread.cc is included in the compiling process for RDMA, and I wonder how to solve these undefined references. Whould you like to give some help? Sincerely hope to receive your reply. Thanks in advance!

DEBUG is not set. Enabling all optimizations.
CUDA is set. Compiling cuda code.
Makefile:72: CUDA is set. But GPU memory is not yet handled by any of the prepostprocessors in the client library.
RDMA is set.
mkdir -p /home/zxy/dpdk/p4app-switchML/dev_root/build/obj/allreduce_benchmark
# Compiling source
g++ --compile -std=c++17 -DNDEBUG -O3 -DCUDA -I/home/zxy/dpdk/p4app-switchML/dev_root/build/include -I /usr/local/cuda/include /home/zxy/dpdk/p4app-switchML/dev_root/benchmarks/allreduce_benchmark/main.cc -o /home/zxy/dpdk/p4app-switchML/dev_root/build/obj/allreduce_benchmark/cpu_main.o
# Linking object
nvcc -dlink /home/zxy/dpdk/p4app-switchML/dev_root/build/obj/allreduce_benchmark/cpu_main.o -o /home/zxy/dpdk/p4app-switchML/dev_root/build/obj/allreduce_benchmark/gpu_main.o
g++ -std=c++17 -DNDEBUG -O3 -DCUDA /home/zxy/dpdk/p4app-switchML/dev_root/build/obj/allreduce_benchmark/gpu_main.o /home/zxy/dpdk/p4app-switchML/dev_root/build/obj/allreduce_benchmark/cpu_main.o -L/home/zxy/dpdk/p4app-switchML/dev_root/build/lib -lswitchml-client -lglog -lstdc++ -lboost_program_options -lpthread -L /usr/local/cuda/lib64 -lcudart -libverbs -lhugetlbfs `pkg-config --libs protobuf grpc++` -o /home/zxy/dpdk/p4app-switchML/dev_root/build/bin/allreduce_benchmark
/home/zxy/dpdk/p4app-switchML/dev_root/build/lib/libswitchml-client.a(grpc_client_cc.o): In function `switchml::operator<<(std::ostream&, switchml_proto::BarrierRequest const&)':
grpc_client.cc:(.text+0x180): multiple definition of `switchml::operator<<(std::ostream&, switchml_proto::BarrierRequest const&)'
/home/zxy/dpdk/p4app-switchML/dev_root/build/lib/libswitchml-client.a(grpc_client.o):grpc_client.cc:(.text+0x170): first defined here
/home/zxy/dpdk/p4app-switchML/dev_root/build/lib/libswitchml-client.a(grpc_client_cc.o): In function `switchml::operator<<(std::ostream&, switchml_proto::BarrierResponse const&)':
grpc_client.cc:(.text+0x210): multiple definition of `switchml::operator<<(std::ostream&, switchml_proto::BarrierResponse const&)'
/home/zxy/dpdk/p4app-switchML/dev_root/build/lib/libswitchml-client.a(grpc_client.o):grpc_client.cc:(.text+0x200): first defined here
/home/zxy/dpdk/p4app-switchML/dev_root/build/lib/libswitchml-client.a(grpc_client_cc.o): In function `switchml::operator<<(std::ostream&, switchml_proto::BroadcastRequest const&)':
grpc_client.cc:(.text+0x230): multiple definition of `switchml::operator<<(std::ostream&, switchml_proto::BroadcastRequest const&)'
/home/zxy/dpdk/p4app-switchML/dev_root/build/lib/libswitchml-client.a(grpc_client.o):grpc_client.cc:(.text+0x220): first defined here
/home/zxy/dpdk/p4app-switchML/dev_root/build/lib/libswitchml-client.a(grpc_client_cc.o): In function `switchml::operator<<(std::ostream&, switchml_proto::BroadcastResponse const&)':
grpc_client.cc:(.text+0x330): multiple definition of `switchml::operator<<(std::ostream&, switchml_proto::BroadcastResponse const&)'
/home/zxy/dpdk/p4app-switchML/dev_root/build/lib/libswitchml-client.a(grpc_client.o):grpc_client.cc:(.text+0x320): first defined here
/home/zxy/dpdk/p4app-switchML/dev_root/build/lib/libswitchml-client.a(grpc_client_cc.o): In function `switchml::GrpcClient::Barrier(switchml_proto::BarrierRequest const&, switchml_proto::BarrierResponse*)':
grpc_client.cc:(.text+0x600): multiple definition of `switchml::GrpcClient::Barrier(switchml_proto::BarrierRequest const&, switchml_proto::BarrierResponse*)'
/home/zxy/dpdk/p4app-switchML/dev_root/build/lib/libswitchml-client.a(grpc_client.o):grpc_client.cc:(.text+0x760): first defined here
/home/zxy/dpdk/p4app-switchML/dev_root/build/lib/libswitchml-client.a(grpc_client_cc.o): In function `switchml::GrpcClient::Broadcast(switchml_proto::BroadcastRequest const&, switchml_proto::BroadcastResponse*)':
grpc_client.cc:(.text+0x870): multiple definition of `switchml::GrpcClient::Broadcast(switchml_proto::BroadcastRequest const&, switchml_proto::BroadcastResponse*)'
/home/zxy/dpdk/p4app-switchML/dev_root/build/lib/libswitchml-client.a(grpc_client.o):grpc_client.cc:(.text+0x9d0): first defined here
/home/zxy/dpdk/p4app-switchML/dev_root/build/lib/libswitchml-client.a(grpc_client_cc.o): In function `switchml::GrpcClient::GrpcClient(switchml::Config&)':
grpc_client.cc:(.text+0xd50): multiple definition of `switchml::GrpcClient::GrpcClient(switchml::Config&)'
/home/zxy/dpdk/p4app-switchML/dev_root/build/lib/libswitchml-client.a(grpc_client.o):grpc_client.cc:(.text+0xeb0): first defined here
/home/zxy/dpdk/p4app-switchML/dev_root/build/lib/libswitchml-client.a(grpc_client_cc.o): In function `switchml::GrpcClient::GrpcClient(switchml::Config&)':
grpc_client.cc:(.text+0xd50): multiple definition of `switchml::GrpcClient::GrpcClient(switchml::Config&)'
/home/zxy/dpdk/p4app-switchML/dev_root/build/lib/libswitchml-client.a(grpc_client.o):grpc_client.cc:(.text+0xeb0): first defined here
/home/zxy/dpdk/p4app-switchML/dev_root/build/lib/libswitchml-client.a(dpdk_master_thread_cc.o): In function `switchml::CheckPortLinkStatus(unsigned short)':
dpdk_master_thread.cc:(.text+0x21b): undefined reference to `rte_delay_us'
dpdk_master_thread.cc:(.text+0x23e): undefined reference to `rte_eth_link_get_nowait'
/home/zxy/dpdk/p4app-switchML/dev_root/build/lib/libswitchml-client.a(dpdk_master_thread_cc.o): In function `switchml::InsertFlowRule(unsigned char, unsigned short, unsigned short)':
dpdk_master_thread.cc:(.text+0xa68): undefined reference to `rte_flow_validate'
dpdk_master_thread.cc:(.text+0xb07): undefined reference to `rte_flow_create'
/home/zxy/dpdk/p4app-switchML/dev_root/build/lib/libswitchml-client.a(dpdk_master_thread_cc.o): In function `switchml::InitPort(switchml::DpdkBackendConfig, unsigned short)':
dpdk_master_thread.cc:(.text+0xb6b): undefined reference to `rte_eth_dev_count_avail'
dpdk_master_thread.cc:(.text+0xb89): undefined reference to `rte_eth_find_next_owned_by'
dpdk_master_thread.cc:(.text+0xbc2): undefined reference to `rte_eth_find_next_owned_by'
dpdk_master_thread.cc:(.text+0xcd8): undefined reference to `rte_eth_dev_info_get'
dpdk_master_thread.cc:(.text+0x105e): undefined reference to `rte_flow_isolate'
dpdk_master_thread.cc:(.text+0x107c): undefined reference to `rte_eth_dev_configure'
dpdk_master_thread.cc:(.text+0x10c4): undefined reference to `rte_eth_dev_adjust_nb_rx_tx_desc'
dpdk_master_thread.cc:(.text+0x10d8): undefined reference to `rte_eth_macaddr_get'
dpdk_master_thread.cc:(.text+0x11f3): undefined reference to `rte_eth_dev_socket_id'
dpdk_master_thread.cc:(.text+0x1209): undefined reference to `rte_eth_tx_queue_setup'
dpdk_master_thread.cc:(.text+0x13e4): undefined reference to `rte_socket_id'
dpdk_master_thread.cc:(.text+0x1404): undefined reference to `rte_pktmbuf_pool_create'
dpdk_master_thread.cc:(.text+0x1417): undefined reference to `rte_eth_dev_socket_id'
dpdk_master_thread.cc:(.text+0x1432): undefined reference to `rte_eth_rx_queue_setup'
dpdk_master_thread.cc:(.text+0x1441): undefined reference to `rte_strerror'
dpdk_master_thread.cc:(.text+0x14b3): undefined reference to `rte_strerror'
dpdk_master_thread.cc:(.text+0x155c): undefined reference to `per_lcore__rte_errno'
dpdk_master_thread.cc:(.text+0x156b): undefined reference to `rte_strerror'
dpdk_master_thread.cc:(.text+0x1603): undefined reference to `rte_eth_dev_start'
dpdk_master_thread.cc:(.text+0x18b6): undefined reference to `rte_strerror'
dpdk_master_thread.cc:(.text+0x191c): undefined reference to `rte_strerror'
dpdk_master_thread.cc:(.text+0x1982): undefined reference to `rte_strerror'
/home/zxy/dpdk/p4app-switchML/dev_root/build/lib/libswitchml-client.a(dpdk_master_thread_cc.o): In function `switchml::DpdkMasterThread::operator()()':
dpdk_master_thread.cc:(.text+0x262c): undefined reference to `rte_eal_init'
dpdk_master_thread.cc:(.text+0x2645): undefined reference to `rte_get_next_lcore'
dpdk_master_thread.cc:(.text+0x266b): undefined reference to `rte_get_next_lcore'
dpdk_master_thread.cc:(.text+0x28f7): undefined reference to `rte_eth_macaddr_get'
dpdk_master_thread.cc:(.text+0x2935): undefined reference to `rte_get_next_lcore'
dpdk_master_thread.cc:(.text+0x294b): undefined reference to `rte_get_next_lcore'
dpdk_master_thread.cc:(.text+0x2964): undefined reference to `rte_eal_get_lcore_state'
dpdk_master_thread.cc:(.text+0x2970): undefined reference to `rte_eal_get_lcore_state'
dpdk_master_thread.cc:(.text+0x2a3d): undefined reference to `rte_get_next_lcore'
dpdk_master_thread.cc:(.text+0x2a82): undefined reference to `rte_get_next_lcore'
dpdk_master_thread.cc:(.text+0x2b85): undefined reference to `rte_eal_remote_launch'
dpdk_master_thread.cc:(.text+0x2d87): undefined reference to `rte_get_next_lcore'
dpdk_master_thread.cc:(.text+0x2d9b): undefined reference to `rte_get_next_lcore'
dpdk_master_thread.cc:(.text+0x2db4): undefined reference to `rte_eal_wait_lcore'
dpdk_master_thread.cc:(.text+0x2e94): undefined reference to `rte_flow_flush'
dpdk_master_thread.cc:(.text+0x2ea9): undefined reference to `rte_eth_dev_stop'
dpdk_master_thread.cc:(.text+0x2eb0): undefined reference to `rte_eth_dev_close'
dpdk_master_thread.cc:(.text+0x33e2): undefined reference to `per_lcore__rte_errno'
dpdk_master_thread.cc:(.text+0x33f1): undefined reference to `rte_strerror'
/home/zxy/dpdk/p4app-switchML/dev_root/build/lib/libswitchml-client.a(dpdk_worker_thread_cc.o): In function `switchml::TxBufferCallback(rte_mbuf**, unsigned short, void*)':
dpdk_worker_thread.cc:(.text+0x1c0): undefined reference to `rte_eth_devices'
dpdk_worker_thread.cc:(.text+0x1f3): undefined reference to `rte_eth_devices'
/home/zxy/dpdk/p4app-switchML/dev_root/build/lib/libswitchml-client.a(dpdk_worker_thread_cc.o): In function `rte_pktmbuf_free':
dpdk_worker_thread.cc:(.text+0x2ec): undefined reference to `per_lcore__lcore_id'
dpdk_worker_thread.cc:(.text+0x3c2): undefined reference to `per_lcore__lcore_id'
dpdk_worker_thread.cc:(.text+0x45d): undefined reference to `rte_mempool_ops_table'
dpdk_worker_thread.cc:(.text+0x4fb): undefined reference to `per_lcore__lcore_id'
dpdk_worker_thread.cc:(.text+0x577): undefined reference to `rte_mempool_ops_table'
dpdk_worker_thread.cc:(.text+0x5cd): undefined reference to `per_lcore__lcore_id'
dpdk_worker_thread.cc:(.text+0x630): undefined reference to `rte_mempool_ops_table'
dpdk_worker_thread.cc:(.text+0x66c): undefined reference to `__rte_panic'
dpdk_worker_thread.cc:(.text+0x680): undefined reference to `rte_mempool_ops_table'
dpdk_worker_thread.cc:(.text+0x698): undefined reference to `per_lcore__lcore_id'
dpdk_worker_thread.cc:(.text+0x6e8): undefined reference to `per_lcore__lcore_id'
dpdk_worker_thread.cc:(.text+0x789): undefined reference to `per_lcore__lcore_id'
dpdk_worker_thread.cc:(.text+0x7cc): undefined reference to `rte_mempool_ops_table'
dpdk_worker_thread.cc:(.text+0x7e1): undefined reference to `per_lcore__lcore_id'
dpdk_worker_thread.cc:(.text+0x8d9): undefined reference to `rte_mempool_ops_table'
/home/zxy/dpdk/p4app-switchML/dev_root/build/lib/libswitchml-client.a(dpdk_worker_thread_cc.o): In function `switchml::DpdkWorkerThread::operator()()':
dpdk_worker_thread.cc:(.text+0xa1b): undefined reference to `per_lcore__lcore_id'
dpdk_worker_thread.cc:(.text+0xcd6): undefined reference to `rte_socket_id'
dpdk_worker_thread.cc:(.text+0xd00): undefined reference to `rte_pktmbuf_pool_create'
dpdk_worker_thread.cc:(.text+0xd11): undefined reference to `rte_socket_id'
dpdk_worker_thread.cc:(.text+0xd31): undefined reference to `rte_malloc_socket'
dpdk_worker_thread.cc:(.text+0xd49): undefined reference to `per_lcore__lcore_id'
dpdk_worker_thread.cc:(.text+0xebc): undefined reference to `rte_mempool_ops_table'
dpdk_worker_thread.cc:(.text+0xee2): undefined reference to `rte_socket_id'
dpdk_worker_thread.cc:(.text+0xf0a): undefined reference to `rte_zmalloc_socket'
dpdk_worker_thread.cc:(.text+0xf30): undefined reference to `rte_eth_tx_buffer_init'
dpdk_worker_thread.cc:(.text+0xf46): undefined reference to `rte_eth_tx_buffer_set_err_callback'
dpdk_worker_thread.cc:(.text+0xf53): undefined reference to `rte_get_tsc_hz'
dpdk_worker_thread.cc:(.text+0xf8b): undefined reference to `rte_socket_id'
dpdk_worker_thread.cc:(.text+0xf9c): undefined reference to `rte_malloc_socket'
dpdk_worker_thread.cc:(.text+0xfb1): undefined reference to `rte_timer_subsystem_init'
dpdk_worker_thread.cc:(.text+0xfb6): undefined reference to `rte_get_tsc_hz'
dpdk_worker_thread.cc:(.text+0x10f3): undefined reference to `rte_timer_init'
dpdk_worker_thread.cc:(.text+0x129b): undefined reference to `rte_mempool_ops_table'
dpdk_worker_thread.cc:(.text+0x1661): undefined reference to `rte_socket_id'
dpdk_worker_thread.cc:(.text+0x167a): undefined reference to `rte_zmalloc_socket'
dpdk_worker_thread.cc:(.text+0x18b2): undefined reference to `rte_timer_reset_sync'
dpdk_worker_thread.cc:(.text+0x1b68): undefined reference to `rte_eth_devices'
dpdk_worker_thread.cc:(.text+0x1be5): undefined reference to `rte_eth_devices'
dpdk_worker_thread.cc:(.text+0x1cb7): undefined reference to `rte_eth_devices'
dpdk_worker_thread.cc:(.text+0x1cdb): undefined reference to `rte_eth_devices'
dpdk_worker_thread.cc:(.text+0x1f2e): undefined reference to `rte_eth_devices'
/home/zxy/dpdk/p4app-switchML/dev_root/build/lib/libswitchml-client.a(dpdk_worker_thread_cc.o):dpdk_worker_thread.cc:(.text+0x1f64): more undefined references to `rte_eth_devices' follow
/home/zxy/dpdk/p4app-switchML/dev_root/build/lib/libswitchml-client.a(dpdk_worker_thread_cc.o): In function `switchml::DpdkWorkerThread::operator()()':
dpdk_worker_thread.cc:(.text+0x1ffa): undefined reference to `rte_timer_reset_sync'
dpdk_worker_thread.cc:(.text+0x2121): undefined reference to `rte_timer_stop_sync'
dpdk_worker_thread.cc:(.text+0x224c): undefined reference to `per_lcore__lcore_id'
dpdk_worker_thread.cc:(.text+0x232c): undefined reference to `per_lcore__lcore_id'
dpdk_worker_thread.cc:(.text+0x23cf): undefined reference to `rte_mempool_ops_table'
dpdk_worker_thread.cc:(.text+0x245d): undefined reference to `per_lcore__lcore_id'
dpdk_worker_thread.cc:(.text+0x24a0): undefined reference to `rte_mempool_ops_table'
dpdk_worker_thread.cc:(.text+0x24f4): undefined reference to `per_lcore__lcore_id'
dpdk_worker_thread.cc:(.text+0x25d2): undefined reference to `per_lcore__lcore_id'
dpdk_worker_thread.cc:(.text+0x266f): undefined reference to `rte_mempool_ops_table'
dpdk_worker_thread.cc:(.text+0x26e5): undefined reference to `per_lcore__lcore_id'
dpdk_worker_thread.cc:(.text+0x2728): undefined reference to `rte_mempool_ops_table'
dpdk_worker_thread.cc:(.text+0x277e): undefined reference to `per_lcore__lcore_id'
dpdk_worker_thread.cc:(.text+0x27d4): undefined reference to `rte_mempool_ops_table'
dpdk_worker_thread.cc:(.text+0x2830): undefined reference to `per_lcore__lcore_id'
dpdk_worker_thread.cc:(.text+0x2864): undefined reference to `rte_mempool_ops_table'
dpdk_worker_thread.cc:(.text+0x292d): undefined reference to `__rte_panic'
dpdk_worker_thread.cc:(.text+0x29df): undefined reference to `rte_mempool_ops_table'
dpdk_worker_thread.cc:(.text+0x29f6): undefined reference to `per_lcore__lcore_id'
dpdk_worker_thread.cc:(.text+0x2a33): undefined reference to `__rte_panic'
dpdk_worker_thread.cc:(.text+0x2a3f): undefined reference to `rte_free'
dpdk_worker_thread.cc:(.text+0x2bdf): undefined reference to `rte_mempool_ops_table'
dpdk_worker_thread.cc:(.text+0x2bf6): undefined reference to `per_lcore__lcore_id'
dpdk_worker_thread.cc:(.text+0x2c13): undefined reference to `per_lcore__rte_errno'
dpdk_worker_thread.cc:(.text+0x2c22): undefined reference to `rte_strerror'
dpdk_worker_thread.cc:(.text+0x2dfc): undefined reference to `per_lcore__lcore_id'
dpdk_worker_thread.cc:(.text+0x2eef): undefined reference to `rte_cycles_vmware_tsc_map'
dpdk_worker_thread.cc:(.text+0x2f3e): undefined reference to `rte_timer_manage'
dpdk_worker_thread.cc:(.text+0x2f7c): undefined reference to `__rte_panic'
dpdk_worker_thread.cc:(.text+0x3160): undefined reference to `__rte_panic'
dpdk_worker_thread.cc:(.text+0x3251): undefined reference to `per_lcore__lcore_id'
dpdk_worker_thread.cc:(.text+0x32d8): undefined reference to `rte_mempool_ops_table'
dpdk_worker_thread.cc:(.text+0x3323): undefined reference to `__rte_panic'
dpdk_worker_thread.cc:(.text+0x33ad): undefined reference to `__rte_panic'
dpdk_worker_thread.cc:(.text+0x33d5): undefined reference to `__rte_panic'
dpdk_worker_thread.cc:(.text+0x33e1): undefined reference to `rte_free'
dpdk_worker_thread.cc:(.text+0x33f6): undefined reference to `rte_pktmbuf_free_bulk'
dpdk_worker_thread.cc:(.text+0x33fe): undefined reference to `rte_free'
dpdk_worker_thread.cc:(.text+0x340a): undefined reference to `rte_free'
dpdk_worker_thread.cc:(.text+0x35c2): undefined reference to `__rte_panic'
dpdk_worker_thread.cc:(.text+0x37e7): undefined reference to `rte_mempool_ops_table'
dpdk_worker_thread.cc:(.text+0x3820): undefined reference to `__rte_panic'
dpdk_worker_thread.cc:(.text+0x3848): undefined reference to `__rte_panic'
dpdk_worker_thread.cc:(.text+0x38bb): undefined reference to `rte_eth_devices'
dpdk_worker_thread.cc:(.text+0x3908): undefined reference to `rte_eth_devices'
dpdk_worker_thread.cc:(.text+0x3968): undefined reference to `__rte_panic'
dpdk_worker_thread.cc:(.text+0x3981): undefined reference to `rte_mempool_ops_table'
dpdk_worker_thread.cc:(.text+0x39ce): undefined reference to `__rte_panic'
dpdk_worker_thread.cc:(.text+0x3b81): undefined reference to `__rte_panic'
dpdk_worker_thread.cc:(.text+0x3b95): undefined reference to `rte_mempool_ops_table'
dpdk_worker_thread.cc:(.text+0x3be2): undefined reference to `__rte_panic'
/home/zxy/dpdk/p4app-switchML/dev_root/build/lib/libswitchml-client.a(dpdk_worker_thread_cc.o): In function `switchml::ResendPacketCallback(rte_timer*, void*)':
dpdk_worker_thread.cc:(.text+0x3d25): undefined reference to `per_lcore__lcore_id'
dpdk_worker_thread.cc:(.text+0x4002): undefined reference to `rte_eth_devices'
dpdk_worker_thread.cc:(.text+0x4021): undefined reference to `rte_eth_devices'
dpdk_worker_thread.cc:(.text+0x40a9): undefined reference to `rte_timer_reset_sync'
dpdk_worker_thread.cc:(.text+0x4130): undefined reference to `rte_mempool_ops_table'
dpdk_worker_thread.cc:(.text+0x4164): undefined reference to `rte_mempool_ops_table'
dpdk_worker_thread.cc:(.text+0x42c0): undefined reference to `__rte_panic'
collect2: error: ld returned 1 exit status

make DPDK=1 MLX5=1 error

When I compile and install client_lib with command 'make DPDK=1 MLX5=1', I encounter four problems:

error: unknown subcommand：get-url，which is in dev_root/client_lib/Makefile:154
fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.
Package absl_base was not found in the pkg-config search path. Perhaps you should add the directory containing `absl_base.pc' but I have compiled and installed absl-cpp in dev_root/third_party/grpc/third_party
/dev_root/build/lib/libswitchml-client.a: Index cannot be added: no symbol can be archived; Run ranlib to add one

How can I fix them?

segment

Hello. I recently try to run allreduce_benchmark with GPU. However, when I run it, the console prints "Segmentation fault (core dumped)" after it printed "Submitting 5 warmup jobs.". I find that when I compile the allreduce_benchmark using the command "make allreduce_benchmark DPDK=1 MLX5=1 VCL=0 CUDA=1" under the /p4app-switchML/dev_root/benchmarks/, the console prints that "Makefile:72: CUDA is set. But GPU memory is not yet handled by any of the prepostprocessors in the client library." I think this problem causes segment fault but don't understand why it happens. Would you like to give some advice? Thanks in advance!

DPDK flow isolated mode failed

Hi,

I'm trying to run an experiment with workers having a mix of MLX5 NICs and some workers have Intel 82599 ES 10G NICs. However, when the allreduce_benchmark is started in the worker with the intel NIC, an error is generated from DPDK saying Flow isolated mode failed: 1 Function not implemented. Any insights on this would be much appreciated.

Thanks.

Question about folded_pipe.

Hi, I notice that the current project only supports folded_pipe1 and folded_pipe 4. Can I modify the code to support folded_pipe 2？

About GDR support

Hello,
I noticed that you have removed the options related to GDR in the most recent commit b5c74a0. May I ask if you have abandoned support for GDR?
I believe GDR is crucial for GPU training. Without specific optimizations, the copy overhead from device to host can severely impact the performance of AllReduce between multiple machines. So I really hope to see GDR being supported by SwitchML.

How to disable ICRC on RNICs?

Hi i am very interested in the RDMA-based implementation.
Could u tell me how to disable ICRC on RNICs (referred in README)?

How can I get the register value of the ConnectX-6 NIC ?

Hi，
I'm using a ConnectX-6 NIC, so how can I get the register values?

p4app-switchML/dev_root/scripts/disable-icrc.sh

Line 13 in 6933744

REGS="0x5361c.12:1 0x5363c.12:1 0x53614.29:1 0x53634.29:1"

Invalid pattern mask error with Intel i40e driver and possible fix

Hi,
I would like to report a bug found in #12, which is related to Intel i40e driver, and a possible fix.
I am using Intel X722 10GbE card with the dpdk backend, and running switchML helloworld will throw an error at the function switchml:: InsertFlowRule():
F0721 16:21:16.853484 108 dpdk_master_thread_utils.inc:154] Flow rule can't be added: 13Invalid pattern mask.

I believe this is a compatibility issue with NICs that use Intel i40e driver. So I commented out this line in dev_root/client_lib/src/backends/dpdk/dpdk_master_thread_utils.inc:
134 ip_mask.hdr.next_proto_id = 0xFF;
After that, I can run switchML helloworld and allreduce benchmark with multi-thread. If all incoming packets are UDP, this line should not be necessary and maybe it should be removed for compatibility.

Help with compiling switchml P4 code：internal compile error

I am currently compiling switchml.p4 on a programmable switch using the command "p4_build-9.x.y.sh switchml.p4". However, I encountered some issues while compiling the code. The console printed the following messages: The number of P4 compiler-proper Errors is 0, and the number of P4 compiler-proper WARNINGs is 28. The crucial part is that it shows "Internal compiler error. Please submit a bug with your code."

I searched online and found that this could be due to insufficient memory. However, I did not observe any lack of available memory during the actual compilation (e.g., available memory less than 1G). I would like to know how to resolve this issue. Could you please provide some assistance? I sincerely hope to receive your reply. Thanks in advance!@AmedeoSapio

My environment is:

Platform:X308P-T

Switch Unit: intel Tofino 1 BFN-T10-032D

SDE:9.7.4

GCC:debian 6.3.0-18+deb9u1

Build issue: RDMAConnectRequest in namespace ‘SwitchML’ does not name a type

Hi, I have tried to build your project, after solving some problems that had not been mentioned in the document, I ran "make RDMA=1" in ./dev_root/clinet_lib, I encountered following errors:

In file included from /home/tt/p4app-switchML/dev_root/client_lib/./src/backends/rdma/rdma_connection.h:18:0,
                 from /home/tt/p4app-switchML/dev_root/client_lib/./src/backends/rdma/rdma_backend.h:18,
                 from /home/tt/p4app-switchML/dev_root/client_lib/./src/backend.cc:16:
/home/tt/p4app-switchML/dev_root/client_lib/./src/backends/rdma/rdma_grpc_client.h:51:38: error: ‘RDMAConnectRequest’ in namespace ‘SwitchML’ does not name a type
     void RDMAConnect(const SwitchML::RDMAConnectRequest& request,
                                      ^~~~~~~~~~~~~~~~~~
/home/tt/p4app-switchML/dev_root/client_lib/./src/backends/rdma/rdma_grpc_client.h:52:22: error: ‘SwitchML::RDMAConnectResponse’ has not been declared
                      SwitchML::RDMAConnectResponse* response);
                      ^~~~~~~~
/home/tt/p4app-switchML/dev_root/client_lib/./src/backends/rdma/rdma_grpc_client.h:60:32: error: ‘BcastRequest’ in namespace ‘SwitchML’ does not name a type
     void Bcast(const SwitchML::BcastRequest& request,
                                ^~~~~~~~~~~~
/home/tt/p4app-switchML/dev_root/client_lib/./src/backends/rdma/rdma_grpc_client.h:61:16: error: ‘SwitchML::BcastResponse’ has not been declared
                SwitchML::BcastResponse* response);
                ^~~~~~~~
/home/tt/p4app-switchML/dev_root/client_lib/./src/backends/rdma/rdma_grpc_client.h:77:31: error: ‘RDMAServer’ is not a member of ‘SwitchML’
     std::unique_ptr<SwitchML::RDMAServer::Stub> rdmaserver_stub_;
                               ^~~~~~~~~~
/home/tt/p4app-switchML/dev_root/client_lib/./src/backends/rdma/rdma_grpc_client.h:77:47: error: template argument 1 is invalid
     std::unique_ptr<SwitchML::RDMAServer::Stub> rdmaserver_stub_;
                                               ^
/home/tt/p4app-switchML/dev_root/client_lib/./src/backends/rdma/rdma_grpc_client.h:77:47: error: template argument 2 is invalid
Makefile:171: recipe for target '/home/tt/p4app-switchML/dev_root/client_lib/./../build/obj/backend.o' failed
make: *** [/home/tt/p4app-switchML/dev_root/client_lib/./../build/obj/backend.o] Error 1

And I can't find these classes' definition in any file of the project.

Failed to install the Pytorch Patch

When I run "git apply /home/antl/wjk/p4app-switchML/dev_root/build/" in the "/home/antl/wjk/p4app-switchML/dev_root/build/ switchml_pytorch.patch" from https://github.com/p4lang/p4app-switchML/tree/main/dev_root/frameworks_integration/pytorch_patch#3-apply-the-switchml-pytorch-patch-to-pytorch, it has the following problem.

But I see in the "rdma_pytorch.dockerfile"

p4app-switchML/dev_root/dockerfiles/rdma_pytorch.dockerfile

Line 92 in b5c74a0

RUN cd /home/switchml/dev_root/ && \

that it runs this command in the "p4app-switchML/dev_root" directory, so I follow this and I don't get any output, I feel it should work. But when I finished compiling pytorch, "import torch_switchml" did not find sml, which means that I failed to install the patch.

I don't know what the reason is.

My conda environment is installed with：
environment.txt

Could not start the context due to missing configuration.

Hi, I have installed pytorch with "pytorch_patch" as required, but when I run "allreduce_pytorch_benchmark.py" from the benchmark, it gives the following error：

Configuring port recirculation

Hi,

I'm running some SwitchML benchmarks on a BF2556-1T switch and I would like to use some of its QSFP ports in the experiments. However, these ports correspond to the pipe num.3 which are all statically reserved for packet recirculation by the SwitchML P4 code here:

p4app-switchML/dev_root/p4/parsers.p4

Lines 46 to 59 in 7ded628

 state parse_port_metadata { 

 // parse port metadata 

 ig_md.port_metadata = port_metadata_unpack<port_metadata_t>(pkt); 

 transition select(ig_intr_md.ingress_port) { 

 64: parse_recirculate; // pipe 0 CPU Eth port 

 68: parse_recirculate; // pipe 0 recirc port 

 320: parse_ethernet; // pipe 2 CPU PCIe port 

 0x080 &&& 0x180: parse_recirculate; // all pipe 1 ports 

 0x100 &&& 0x180: parse_recirculate; // all pipe 2 ports 

 0x180 &&& 0x180: parse_recirculate; // all pipe 3 ports 

 default: parse_ethernet; 

 } 

 }

Could you let me know what changes I would need to make in order to make use of these ports? Could I perhaps substitute the target ports with some other free ports from pipe 0 for performing packet recirculation? Is there a specific number of ports that need to be in recirculation mode in order for SwitchML to operate normally?

Thank you in advance,

	state parse_port_metadata {
	// parse port metadata
	ig_md.port_metadata = port_metadata_unpack<port_metadata_t>(pkt);

	transition select(ig_intr_md.ingress_port) {
	64: parse_recirculate; // pipe 0 CPU Eth port
	68: parse_recirculate; // pipe 0 recirc port
	320: parse_ethernet; // pipe 2 CPU PCIe port
	0x080 &&& 0x180: parse_recirculate; // all pipe 1 ports
	0x100 &&& 0x180: parse_recirculate; // all pipe 2 ports
	0x180 &&& 0x180: parse_recirculate; // all pipe 3 ports
	default: parse_ethernet;
	}
	}

p4lang / p4app-switchml Goto Github PK

p4app-switchml's People

Contributors

Stargazers

Watchers

Forkers

p4app-switchml's Issues

Device #1:

Network devices using DPDK-compatible driver

Recommend Projects

Recommend Topics

Recommend Org