Git Product home page Git Product logo

rdma-tutorial's Introduction

RDMA-Tutorial

This project presents an example based tutorial on RDMA based programming. A more detailed discussion can be found on the Wiki page.

Hardware and software requirement

  • Mellanox HCAs
  • GNU make
  • gcc-4.4
  • Mellanox OFED 3.3

How to use

build project

Simply use make to build the release version or make debug to build the debug version.

navigate through examples

The project contains 4 examples. Details of the examples can be found on the Wiki page. The code of the examples are organized as git commits. Simply do git log --oneline to find the commit version number corresponding to the examples you are looking for.

Contact

Jiachen Xue ([email protected])

rdma-tutorial's People

Contributors

gareging avatar jcxue avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

rdma-tutorial's Issues

An error for handling ring-buffer

I'm trying example 1 : git checkout 65893ec as the wiki stated.
If I tune up the concurrency config_info.num_concurr_msgs = 2; (main.c:38) , the program crashed with local protection error.

The error is caused by the wrong implementation of the ring buffer.
i.e. (client.c:53)

        buf_offset = (buf_offset + msg_size) % buf_size;
        buf_ptr +=  buf_offset;

should be changed to

        buf_offset = (buf_offset + msg_size) % buf_size;
        buf_ptr = ib_res.ib_buf + buf_offset;

Otherwise, the buffer is never wrap around and causes an overflow.

a question about mutli thread

In example2, i want to use multi-thread transporting, so i modify the value of the num_threads,but when i run the project in the command line, nothing output in the command line, i forced the process to exit, i found in the log,there is some information about IB config, so the qp connect is success,but nothing tranported.

Problems with running the first example on Mellanox ConnectX-5

Hi,
Thanks for the great tutorial.
I have an issue when running the first example on both server and client.

================ IB Echo Server ================

************ Configuraion ************

is_server = true
msg_size = 64
num_concurr_msgs = 1
sock_port = 8080

************ End of Configuraion ************

[ERROR] (setup_ib.c:184:setup_ib: errno: Cannot allocate memory) Failed to create qp
[ERROR] (main.c:39:main: errno: None) Failed to setup IB

================ Run Finished ================

gcc 8.0 warning for strncpy

the output is as the following:


gcc -Wall -Werror -O2 -c -o config.o config.c
config.c: In function ‘get_rank’:
config.c:105:5: error: ‘strncpy’ output may be truncated copying 64 bytes from a string of length 64 [-Werror=stringop-truncation]
strncpy (hostname, utsname_buf.nodename, sizeof(hostname));
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cc1: all warnings being treated as errors


gcc does this for security reason, and it seems that many project have got this warning.[1]
I think we can use strlcpy[2] instead:)
[1]openwall/john#3127 (comment)
[2]https://linux.die.net/man/3/strlcpy
https://stackoverflow.com/questions/2114896/why-are-strlcpy-and-strlcat-considered-insecure

你好,能否请教一下perftest工具包的延迟测试里的参数的含义?

两块ConnectX-3的板子直连,用ib_write_lat测试时延,结果出来的数据没看懂什么意思。
比如1024bytes的t_avg竟然有8299usec,就是8ms,是不是太慢了,而如果去掉-C选项,又变成了2usec,又太快了,这是怎么一回事?从源码里来看,为什么不加-C要除CPU的主频,加了又不除了。
那么从哪里能得到正常的数据呢?还是我看的方法不对?

[root@localhost perftest-master]# ib_write_lat -a -n 5 -F -C


  • Waiting for client to connect... *

Requested SQ size might be too big. Try reducing TX depth and/or inline size.
Current TX depth is 1 and inline size is 220 .

                RDMA_Write Latency Test

Dual-port : OFF Device : mlx4_0
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
Mtu : 1024[B]
Link type : Ethernet
GID index : 1
Max inline data : 220[B]
rdma_cm QPs : OFF
Data ex. method : Ethernet

local address: LID 0000 QPN 0x0248 PSN 0xe6be8a RKey 0xe0010101 VAddr 0x007f360b400000
GID: 00:00:00:00:00:00:00:00:00:00:255:255:12:12:12:01
remote address: LID 0000 QPN 0x0253 PSN 0x158390 RKey 0x60010102 VAddr 0x007fe0a4e10000
GID: 00:00:00:00:00:00:00:00:00:00:255:255:12:12:12:02

#bytes #iterations t_min[usec] t_max[usec] t_typical[usec] t_avg[usec] t_stdev[usec] 99% percentile[usec] 99.9% percentile[usec]
2 5 2808.00 4636.00 2809.00 2809.00 1.00 4636.00 4636.00
4 5 2775.50 2780.50 2776.50 2776.50 1.00 2780.50 2780.50
8 5 2674.50 2848.50 2722.50 2722.50 48.00 2848.50 2848.50
16 5 2795.50 2841.50 2810.00 2810.25 14.75 2841.50 2841.50
32 5 2856.50 3028.00 2912.00 2912.25 55.75 3028.00 3028.00
64 5 3091.50 3191.50 3141.00 3141.25 49.75 3191.50 3191.50
128 5 3534.50 3601.50 3557.00 3557.00 22.50 3601.50 3601.50
256 5 5689.50 8023.00 5913.50 5913.50 224.00 8023.00 8023.00
512 5 5727.00 6357.00 5826.50 5826.75 99.75 6357.00 6357.00
1024 5 7834.50 8299.00 8040.00 8040.25 205.75 8299.00 8299.00
2048 5 10110.50 10364.50 10195.00 10195.25 84.75 10364.50 10364.50
4096 5 15141.00 15210.50 15152.50 15152.75 11.75 15210.50 15210.50
8192 5 24677.50 24892.50 24784.50 24784.75 107.25 24892.50 24892.50
16384 5 44256.00 44398.00 44310.50 44310.75 54.75 44398.00 44398.00
32768 5 83104.50 83235.50 83160.00 83160.00 55.50 83235.50 83235.50
65536 5 160902.00 161911.00 160909.00 160909.00 7.00 161911.00 161911.00
131072 5 316419.00 316811.50 316430.50 316430.75 11.75 316811.50 316811.50
262144 5 627500.00 627571.50 627504.50 627504.75 4.75 627571.50 627571.50
524288 5 1249507.00 1249553.00 1249521.00 1249521.25 14.25 1249553.00 1249553.00
1048576 5 2493749.50 2493809.00 2493766.00 2493766.25 16.75 2493809.00 2493809.00
2097152 5 4981971.50 4982092.00 4981986.50 4981986.50 15.00 4982092.00 4982092.00
4194304 5 9958520.50 9959012.50 9958554.50 9958554.50 34.00 9959012.50 9959012.50
8388608 5 19911566.50 19911583.50 19911573.00 19911573.25 6.75 19911583.50 19911583.50

这是加了-C选项,出来的时延是不是太大了。而下面不加的又太小了。。

[root@localhost perftest-master]# ib_write_lat -a -n 5 -F


  • Waiting for client to connect... *

Requested SQ size might be too big. Try reducing TX depth and/or inline size.
Current TX depth is 1 and inline size is 220 .

                RDMA_Write Latency Test

Dual-port : OFF Device : mlx4_0
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
Mtu : 1024[B]
Link type : Ethernet
GID index : 1
Max inline data : 220[B]
rdma_cm QPs : OFF
Data ex. method : Ethernet

local address: LID 0000 QPN 0x0246 PSN 0xe0bd4b RKey 0xd0010101 VAddr 0x007f7c0cc00000
GID: 00:00:00:00:00:00:00:00:00:00:255:255:12:12:12:01
remote address: LID 0000 QPN 0x0251 PSN 0xf35b27 RKey 0x50010102 VAddr 0x007f27bc793000
GID: 00:00:00:00:00:00:00:00:00:00:255:255:12:12:12:02

#bytes #iterations t_min[usec] t_max[usec] t_typical[usec] t_avg[usec] t_stdev[usec] 99% percentile[usec] 99.9% percentile[usec]
2 5 1.03 1.49 1.09 1.09 0.06 1.49 1.49
4 5 1.03 1.80 1.04 1.04 0.01 1.80 1.80
8 5 0.99 1.05 1.00 1.00 0.01 1.05 1.05
16 5 1.02 1.24 1.03 1.03 0.00 1.24 1.24
32 5 1.07 1.16 1.09 1.09 0.02 1.16 1.16
64 5 1.14 1.18 1.15 1.15 0.01 1.18 1.18
128 5 1.57 1.96 1.62 1.62 0.05 1.96 1.96
256 5 1.75 1.81 1.77 1.77 0.02 1.81 1.81
512 5 2.11 2.15 2.11 2.11 0.00 2.15 2.15
1024 5 2.82 2.84 2.83 2.83 0.01 2.84 2.84
2048 5 3.72 3.78 3.74 3.74 0.01 3.78 3.78
4096 5 5.54 5.59 5.56 5.56 0.01 5.59 5.59
8192 5 9.13 9.35 9.14 9.14 0.01 9.35 9.35
16384 5 16.27 16.34 16.30 16.30 0.03 16.34 16.34
32768 5 30.67 30.73 30.68 30.68 0.00 30.73 30.73
65536 5 59.41 60.78 59.84 59.84 0.43 60.78 60.78
131072 5 116.64 116.70 116.66 116.66 0.02 116.70 116.70
262144 5 231.34 231.42 231.37 231.37 0.02 231.42 231.42
524288 5 460.72 460.87 460.73 460.73 0.01 460.87 460.87
1048576 5 919.49 919.67 919.49 919.49 0.00 919.67 919.67
2097152 5 1836.98 1837.59 1836.99 1836.99 0.01 1837.59 1837.59
4194304 5 3671.97 3672.00 3671.98 3671.98 0.01 3672.00 3672.00
8388608 5 7341.96 7342.20 7342.08 7342.08 0.11 7342.20 7342.20

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.