Does Accelio support Multicast for RDMA transport? If so, are there any examples on how to use it?

If mempool A is running in thread A context, another thread can use this xio_mempool_add_slab to create additional slab meaningwhile?

Hi @eyalsol

What we're seeing is that after reconnect succeeds while there's more than one connected transport - calling ibv_post_recv results in some tasks (and their rdma handles) pointing to a transport handle that doesn't match the actual physical transport on which the task has been received. This of course causes issues as the session can't be found by upper layers in order to handle the task as expected.

Your help would be much appreciated.

Regards,
Barak

On server side, run xio_server as follows:

./xio_server 172.16.10.203 -n 32 -w 32 -r tcp

On the client side, run xio_client as follows:

./xio_client 172.16.10.203 -f 1 -r tcp

it will print error log as follows after normal exit:
......
exit signaled
[2015/07/09-09:41:29.148082] xio_tcp_management.c:477 [ERROR] - shutting context while tcp_hndl=0x1f13f00 state is INIT?
exit complete

I think the workaround should be removing line 477 of src/usr/transport/tcp/xio_tcp_management.c, to make it fall through to lin 487. ERROR_LOG is not necessary at all.

in my test case, i create two connections, A to B, and B to A; then send msg A to B, and B to A, but sometime my process is coredump, or msg send failed;
only on RDMA, tcp is ok.

1. coredump:
   Program terminated with signal 11, Segmentation fault.
   #0 xio_rdma_on_recv_rsp (rdma_hndl=0x6a7dd0, task=0x7f7e62365290) at transport/rdma/xio_rdma_datapath.c:2955

2955 osgtbl_ops = (struct xio_sg_table_ops *)
(gdb) bt
#0 xio_rdma_on_recv_rsp (rdma_hndl=0x6a7dd0, task=0x7f7e62365290) at transport/rdma/xio_rdma_datapath.c:2955
#1 0x00007f7e61b185f9 in xio_rdma_rx_handler (task=, rdma_hndl=)

at transport/rdma/xio_rdma_datapath.c:801

#2 xio_handle_wc (task=, rdma_hndl=) at transport/rdma/xio_rdma_datapath.c:1110
#3 xio_poll_cq (task=, rdma_hndl=) at transport/rdma/xio_rdma_datapath.c:1216
#4 0x00007f7e61b18de9 in xio_poll_cq_armable (tcq=0x1) at transport/rdma/xio_rdma_datapath.c:1263
#5 0x00007f7e61b18e95 in xio_cq_event_handler (fd=, events=, data=0x673c10)

at transport/rdma/xio_rdma_datapath.c:1363

#6 0x00007f7e61b02be5 in xio_ev_loop_run_helper (loop_hndl=0x67d640, timeout=-1) at xio/xio_ev_loop.c:448
#7 0x00007f7e61b06658 in xio_context_run_loop (ctx=0x67d430, timeout_ms=575) at xio/xio_context.c:487

ERROR log: "ERROR: expected sn:1, arrived sn:0"

1. msg send failed:
   ERROR log:xio_mbuf_first_read_tlv failed. tlv.head:0x7ff9ba424000, len:-1, buf.tail:0x7ff9ba426400

and tlv is "magic 0 type 0,len 0"

The possible reasons is ？？？？

thanks！

Hi Eyal,

While trying the reconnect feature in accelio, I ran a simple test which took the transport down and then back up during IOs - and encountered some issues. I'd appreciate some words about how the reconnect feature is designed - specifically with regards to how the nexus swapping and the transport duplication take action during the reconnect (on the client and server side).

Your help would be much appreciated.
Thank you,
Barak

after creating four sessions on the client side, i received the following errors:

lino

[ 350.826908] ERROR: /home/lino/git/mellanox/accelio/src/kernel/transport/rdma/xio_rdma_memory.c:744::xio_create_frwr_pool(): Failed to allocate ib_fast_reg_mr err=-12
[ 350.827099] ERROR: /home/lino/git/mellanox/accelio/src/kernel/transport/rdma/xio_rdma_memory.c:744::xio_create_frwr_pool(): Failed to allocate ib_fast_reg_mr err=-12
[ 351.047437] ERROR: /home/lino/git/mellanox/accelio/src/kernel/transport/rdma/xio_rdma_management.c:1466::xio_rdma_primary_pool_post_create(): fast reg init failed
[ 351.052491] ERROR: /home/lino/git/mellanox/accelio/src/kernel/transport/rdma/xio_rdma_management.c:1466::xio_rdma_primary_pool_post_create(): fast reg init failed

hi guys,

do we have any plan to package libaccelio, so user is able to cook an rpm or deb package before the packages are accept by distros?

thanks,

1. when I send msg by RDMA SEND operation, the remote side how to select recv MR? will select mr which application register?
2. when recv RDMA SEND msg(mr), can application operate it without dereg? and this mr will recv next msg ?
   or how does rdma know this mr can recv msg, may be it is operated by application?

Creating connection would be costly, especially if we use TCP for underlying transport (it must consume time for 3 way handshake).

Is it possible to reusing a single connection for different sessions for avoiding the cost? For example,

1. worker thread A creates a new context, session, and connection for RPC
2. A ends its RPC, destory the context and session, and cache the connection for other worker thread
3. B creates a new context, session, and reuse the connection created and cached by A

i have seen and experience problems with and without using the keep-alive mechanism. without the keep-alive, a connection currently takes >5 min to go from a close event to a tear-down event. i should be able to request a connection to be torn-down without such a wait Why is this needed in the current implementation?

when i use the keep alive; the connection properly moves from one step to the other; however, when i try to close the session and destroy it; it complains that connections are still active and does not close it.

first the connection timeout should be a configurable parameter, not a #define; next, i should have the option to destroy a connection and be able to close a session quickly as my application see fit. i would appreciate an explanation for the current behavior

lino

I want to send a huge message by using single iovector. I get a error shows send queue overflow. I change XIO_OPTNAME_SND_QUEUE_DEPTH_BYTES and found the source code here.

It uses int32_t to make sure the value not less than 1, but assigned the value as uint64_t. Is it some magic design? Or can I just change the int32_t to uint64_t?

I'm trying to implement RPC with accelio like examples/usr/hello_world_iov. My code follows the implementation of xio_server.c in the example. However, I'm facing the below error:

[2015/08/20-19:06:43.356188] xio_tcp_datapath.c:1753 [ERROR] - partial completion of request due to missing, response buffer
[2015/08/20-19:06:43.356230] xio_tcp_datapath.c:1812 [ERROR] - xio_tcp_send_msg failed
[2015/08/20-19:06:43.356251] xio_nexus.c:2200 [ERROR] - transport send failed err:1247689306

Seems that construction of reply is missing something. Could you describe a cause which can produce the above error?

Sorry for my confusing question :(

The code can be found here: https://github.com/mitake/sheepdog/blob/accelio-v0/sheep/xio.c#L161

I'm trying to use accelio in an existing application which uses epoll() for its main event loop. For minimizing cost of coding, I'd like to use xio_context_get_poll_fd() and add the fd of accelio context to the epoll loop as a first step.

I have a question for such usage. Should I create one context for one connection?

It seems that if the server has only one global xio_context and calls xio_context_run_loop() with the context, the xio_context_run_loop() causes infinite blocking (I'm passing XIO_INFINITE) and there's no chance of stopping the loop with xio_context_stop_loop().

For avoiding the above situation, I'd like to know the assumed usage of context fds.

xio_context_create() seems to unregister signal handlers of program (e.g. a handler of SIGUSR1 seems to be overwritten when program calls xio_context_create()).
Is my understanding correct? If it is true, can I change this behavior for preserving default signal handlers?

We tested reconnect over bond interface following the instruction here:

https://community.mellanox.com/docs/DOC-2158

But the reconnect didn't work. Was anyone able to make the above test scenario work? Below is our console output by running the hello_ow_test as mentioned in the above link:

server:

**** message 132000000 - hello world request data
**** message 136000000 - hello world request data
[2016/03/17-08:37:24.286883] xio_rdma_management.c:168 [ERROR] - ibv_get_async_event: dev:mlx4_0 evt: port error
[2016/03/17-08:37:24.638430] xio_rdma_management.c:2386 [DEBUG] - cm event: [RDMA_CM_EVENT_ADDR_CHANGE], hndl:0x20f02b0, status:0
[2016/03/17-08:37:24.638438] xio_rdma_management.c:2241 [DEBUG] - on_cm_disconnected. rdma_hndl:0x20f02b0, state:LISTEN
[2016/03/17-08:37:24.650001] xio_rdma_management.c:2386 [DEBUG] - cm event: [RDMA_CM_EVENT_ADDR_CHANGE], hndl:0x20f15e0, status:0
[2016/03/17-08:37:24.650005] xio_rdma_management.c:2241 [DEBUG] - on_cm_disconnected. rdma_hndl:0x20f15e0, state:CONNECTED
[2016/03/17-08:37:24.650044] xio_rdma_management.c:168 [ERROR] - ibv_get_async_event: dev:mlx4_0 evt: GID table change
[2016/03/17-08:37:24.650690] xio_rdma_management.c:168 [ERROR] - ibv_get_async_event: dev:mlx4_1 evt: GID table change
[2016/03/17-08:37:24.650977] xio_rdma_management.c:168 [ERROR] - ibv_get_async_event: dev:mlx4_1 evt: GID table change
[2016/03/17-08:37:24.651395] xio_rdma_management.c:168 [ERROR] - ibv_get_async_event: dev:mlx4_1 evt: GID table change
[2016/03/17-08:37:24.651904] xio_rdma_management.c:168 [ERROR] - ibv_get_async_event: dev:mlx4_1 evt: GID table change
[2016/03/17-08:37:24.652172] xio_rdma_management.c:168 [ERROR] - ibv_get_async_event: dev:mlx4_1 evt: GID table change
[2016/03/17-08:37:24.652441] xio_rdma_management.c:168 [ERROR] - ibv_get_async_event: dev:mlx4_1 evt: GID table change
[2016/03/17-08:37:24.652870] xio_rdma_management.c:168 [ERROR] - ibv_get_async_event: dev:mlx4_0 evt: GID table change
[2016/03/17-08:37:24.653159] xio_rdma_management.c:168 [ERROR] - ibv_get_async_event: dev:mlx4_1 evt: GID table change
[2016/03/17-08:37:24.653369] xio_rdma_management.c:168 [ERROR] - ibv_get_async_event: dev:mlx4_1 evt: GID table change
[2016/03/17-08:37:24.653759] xio_rdma_management.c:168 [ERROR] - ibv_get_async_event: dev:mlx4_1 evt: GID table change
[2016/03/17-08:37:24.654159] xio_rdma_management.c:168 [ERROR] - ibv_get_async_event: dev:mlx4_1 evt: GID table change
[2016/03/17-08:37:38.209444] xio_connection.c:2926 [DEBUG] - send keepalive request. session:0x2101420, connection:0x2101660
[2016/03/17-08:37:58.209451] xio_connection.c:3093 [WARN ] - connection keepalive timeout. connection:0x2101660 probes:[1]
[2016/03/17-08:38:18.209471] xio_connection.c:3093 [WARN ] - connection keepalive timeout. connection:0x2101660 probes:[2]
[2016/03/17-08:38:24.650386] xio_nexus.c:1637 [DEBUG] - nexus: [notification] - transport disconnected. nexus:0x20f9c10, transport:0x20f15e0
[2016/03/17-08:38:38.209487] xio_connection.c:3072 [ERROR] - connection keepalive timeout. connection:0x2101660 probes:[3]
session event: connection error. session:0x2101420, connection:0x2101660, reason: Timeout
[2016/03/17-08:40:10.650408] xio_session_server.c:641 [DEBUG] - session: [notification] - connection disconnected session:0x2101420, nexus:0x20f9c10
[2016/03/17-08:40:10.650418] xio_session.c:934 [DEBUG] - xio_session_on_nexus_disconnected. session:0x2101420, nexus:0x20f9c10
[2016/03/17-08:40:10.650422] xio_connection.c:2274 [DEBUG] - connection disconnected: connection:0x2101660
session event: connection disconnected. session:0x2101420, connection:0x2101660, reason: Session disconnected session event: connection teardown. session:0x2101420, connection:0x2101660, reason: Session disconnected last recv:139453507
[2016/03/17-08:40:10.650447] xio_connection.c:2218 [DEBUG] - xio_connection_destroy. session:0x2101420, connection:0x2101660 nexus:0x20f9c10 nr:1, state:DISCONNECTED
[2016/03/17-08:40:10.650451] xio_connection.c:2143 [DEBUG] - xio_connection_post_destroy. session:0x2101420, connection:0x2101660 conn:0x20f9c10 nr:1
[2016/03/17-08:40:10.650454] xio_session_server.c:651 [DEBUG] - session: [notification] - connection closed. session:0x2101420, nexus:0x20f9c10
[2016/03/17-08:40:10.650459] xio_rdma_management.c:2697 [DEBUG] - xio_rmda_close: [close] handle:0x20f15e0, qp:0x20f3890 state:DISCONNECTED
[2016/03/17-08:40:10.650463] xio_nexus.c:1643 [DEBUG] - nexus: [notification] - transport closed. nexus:0x20f9c10, transport:0x20f15e0
[2016/03/17-08:40:10.650466] xio_nexus.c:1697 [DEBUG] - nexus:0x20f9c10 - close complete
session event: session teardown. session:0x2101420, connection:(nil), reason: Session disconnected exit signaled
[2016/03/17-08:40:10.651655] xio_server.c:383 [DEBUG] - xio_server_destroy - server:0x20f0040
[2016/03/17-08:40:10.651659] xio_rdma_management.c:2697 [DEBUG] - xio_rmda_close: [close] handle:0x20f02b0, qp:(nil) state:LISTEN
[2016/03/17-08:40:10.651662] xio_nexus.c:1643 [DEBUG] - nexus: [notification] - transport closed. nexus:0x20f0100, transport:0x20f02b0
[2016/03/17-08:40:10.651664] xio_nexus.c:1697 [DEBUG] - nexus:0x20f0100 - close complete

client:

transactions per second: 3036700, bandwidth: TX 2965.53 MB/s, length: TX: 1024 B transactions per second: 3036772, bandwidth: TX 2965.60 MB/s, length: TX: 1024 B
[2016/03/17-08:37:38.209487] xio_connection.c:2926 [DEBUG] - send keepalive request. session:0x714dd0, connection:0x7166a0
[2016/03/17-08:37:58.209495] xio_connection.c:3093 [WARN ] - connection keepalive timeout. connection:0x7166a0 probes:[1]
[2016/03/17-08:38:18.209509] xio_connection.c:3093 [WARN ] - connection keepalive timeout. connection:0x7166a0 probes:[2]
[2016/03/17-08:38:38.209524] xio_connection.c:3072 [ERROR] - connection keepalive timeout. connection:0x7166a0 probes:[3]
session event: connection error. reason: Timeout
[2016/03/17-08:40:38.209548] xio_connection.c:2926 [DEBUG] - send keepalive request. session:0x714dd0, connection:0x7166a0
[2016/03/17-08:40:58.209555] xio_connection.c:3093 [WARN ] - connection keepalive timeout. connection:0x7166a0 probes:[1]
[2016/03/17-08:41:18.209570] xio_connection.c:3093 [WARN ] - connection keepalive timeout. connection:0x7166a0 probes:[2]
[2016/03/17-08:41:38.209583] xio_connection.c:3072 [ERROR] - connection keepalive timeout. connection:0x7166a0 probes:[3]
session event: connection error. reason: Timeout

It happens sometime in my code that the response is already recieved but the send_msg_complete event is not invoked. So I set all my response flag to XIO_MSG_FLAG_IMM_SEND_COMP. The problem seems solved.

But, why does this problem happen? How does accelio call send_msg_complete in detail?

We can get a fd of xio_context with xio_context_get_poll_fd(). Like this, can I get a fd of xio_session?

IIUC, xio_session is something like TCP connection. If I can get a fd of this object, integrating accelio to an event loop of existing applications would be easier (e.g. registering fd which is returned by accept(2) to epoll loop would be very common pattern of server software).

I'm very new to accelio, so my question and request would be conflict with its principle.

On newer 18 core processor with 2 socket there are 72 threads aka vcpus. cpumask currently being a 64bit type will overflow on these machines.

I got the following compile error with v1.4:

CCLD xio_rdma_client
xio_rdma_common_usr.o: In function publish_our_buffer': /root/accelio-1.4/tests/usr/direct_rdma_test/../../../tests/portable/direct_rdma_test/xio_rdma_common.c:89: undefined reference toxio_lookup_rkey_by_response'
collect2: error: ld returned 1 exit status
make[1]: *** [xio_rdma_client] Error 1
make[1]: Leaving directory `/root/accelio-1.4/tests/usr/direct_rdma_test'
make: *** [all-recursive] Error 1

Why does XIO_SGL_TYPE_IOV for tcp transport only support first element in output vector?
I changed hello_word example to send two elements, but xio_server only sees
first element in its in vector.

diff --git a/examples/usr/hello_world/xio_client.c b/examples/usr/hello_world/xio_client.c
index 33934b2..ac52998 100644
--- a/examples/usr/hello_world/xio_client.c
+++ b/examples/usr/hello_world/xio_client.c
@@ -239,7 +239,15 @@ int main(int argc, char *argv[])
              req->out.data_iov.sglist[0].iov_base)
               + 1;

-       req->out.data_iov.nents = 1;
+       req->out.data_iov.sglist[1].iov_base =
+           strdup("hello world data request 2");
+
+       req->out.data_iov.sglist[1].iov_len =
+           strlen((const char *)
+             req->out.data_iov.sglist[1].iov_base)
+              + 1;
+
+       req->out.data_iov.nents = 2;
        req++;
    }
    /* send first message */
==========================

and xio_server only print first element:

librdmacm: couldn't read ABI version.
librdmacm: assuming: 4
CMA: unable to get RDMA device list
librdmacm: couldn't read ABI version.
librdmacm: assuming: 4
CMA: unable to get RDMA device list
[2015/09/10-21:24:35.577572] xio_rdma_management.c:758    [ERROR] - Failed to get IB devices list
[2015/09/10-21:24:35.577666] xio_rdma_management.c:3289   [ERROR] - Failed to initialize device list
listen to tcp://127.0.0.1:8008
new session event. session:0x2321f40
session event: new connection. session:0x2321f40, connection:0x2322140, reason: Success
message header : [4000000] - hello world header request
message data: [4000000][0][52] - hello world data request
message header : [8000000] - hello world header request
message data: [8000000][0][52] - hello world data request
message header : [12000000] - hello world header request
message data: [12000000][0][52] - hello world data request

whether or not msg->in.header.iov_base has been received, when assign data buf? done on RDMA? tcp not?

xio_session_ops->on_msg() is called when a new request arrives. The first parameter of the callback is the session which the newly arrived request belongs to. Like this session, I want to obtain the connection which the request belongs to. I read the document of accelio but couldn't find a way of doing it. Is it possible in the current accelio?

I want to know the protocol packet format of XIO,such as XIO_MSG_TYPE_RSP.
thank you.

Accelio defines two constants as the maximum allowed work requests for the RDMA HW. These are MAX_SEND_WR and MAX_RECV_WR. When a client initiates an RDMA request which requires an sg-list with more than one entry (i.e. when the remote or local data-resident memories are fragmented), accelio splits the request into several WRs, creating a list of concatenated WRs (called phantom tasks) and only accounting for the "root" request in the reqs_in_flight_nr or rsps_in_flight_nr counters. It is notable that phantom requests are moved to the in_flight_list although they were not accounted for.
The lack of accountability for phantom tasks makes verify_req_send_limits skip the actual check for WRs usage and may lead to a depletion of the maximum allowed WRs (both on the receiving and the sending sides).
Our question is - shouldn't the phantom tasks be accounted and checked in verify_req_send_limits as well?

We have two questions regarding handling of error messages.

1. We get error callbacks (on_msg_error) on internal accelio messages (which accelio initiates, e.g keep_alive messages). Our expectation is that accelio should internally handle theses errors and our code should not be notified (We've also seen that in impl of on_msg_error in accelio samples, the sample code assumes all messages belong to it.). I suppose that's not intentional?
2. On disconnection, we have encountered some cases when keep_alive messages are sent in an infinite loop. It happens as xio_nexus_xmit returns -ENOMSG as the transport send fails (with transport error as expected), and xio_connection_xmit_inl does not remove the message from the list for this error message (with xio_msg_list_remove) so it is sent again with the same repeated behavior again. Shouldn't the message be removed from the list in this case to avoid the loop.

On the server side, run xio_server:
xio_server 10.45.25.203 -n 32 -w 9088 -r tcp
On the client side, run xio_client:
xio_client 10.45.25.203 -f 1 -w 9088 -r tcp
Then all of sides exit abnormally with error on the server side:
**** [0x25f8920] on_new_session :10.45.25.204:43030
session event: new connection. session:0x25f8920, connection:0x25f8b00, reason: Success
[2015/07/09-17:23:42.330884] xio_tcp_datapath.c:1758 [ERROR] - partial completion of request due to missing, response buffer
[2015/07/09-17:23:42.330991] xio_tcp_datapath.c:1817 [ERROR] - xio_tcp_send_msg failed
[2015/07/09-17:23:42.331003] xio_nexus.c:2200 [ERROR] - transport send failed err:1247689306
**** [0x25f8920] message [0] failed. reason: Message too long
**** [0x25f8920] message [1] failed. reason: Message discarded
......

Besides, xio_lat_server/client happens to the same bug as xio_server/client when message length is larger than 9119 byte.

I am trying to get a key value sample application working with Accelio, but can't seem to get multiple messages to be sent per session by looking at the hello world example.

I would like to be able to send a "PUT" request then a "GET" request, but currently only the first message I send goes through. My code sits here: https://github.com/EntilZha/accelio/tree/master/examples/usr/keyvalue

Last few commits/comments might show what I am trying to do, but in summary:

1. Tried issuing multiple xio_send_request, then running the context loop with xio_context_run_loop. The result is that only the last message I sent goes through, so it looks like the message is getting overwritten. My expected behavior was that this would queue up messages.
2. Tried sticking xio_send_request in the on_request portion of the request reply cycle, this doesn't seem to do anything
3. Tried making a array of xio_msg and issuing multiple xio_send_request like in 1., however it complains that it is not a valid message (looks like xio_connect initializes somethign in cparams.conn_user_context)
4. Tried issuing pairs of xio_send_request with xio_context_run_loop first waiting for zero events with code below, then just waiting 1000 seconds. I get an error that user object leaked on xio_connection and xio_session.

while(req->in.data_iov.nents == 0) {
xio_context_run_loop(session_data.ctx, 0);
}

My output right now looks something like this

server

-bash-4.2$ ./run_server.sh rdma53.csel.io 9999
listen to rdma://rdma53.csel.io:9999
new session event. session:0x6610e0
Leaving on_new_session
session event: new connection. session:0x6610e0, connection:0x6612c0, reason: Success
Header: PUT
Received request data:
Key: KEY1
value: VALUE1
Send response..
session event: connection disconnected. session:0x6610e0, connection:0x6612c0, reason: Session disconnected
session event: connection teardown. session:0x6610e0, connection:0x6612c0, reason: Session disconnected
Tear down the connection...
[2015/12/02-11:17:47.491158] xio_connection.c:2203        [ERROR] - tasks still pending. connection:0x6612c0
session event: session teardown. session:0x6610e0, connection:(nil), reason: Session disconnected
Destroy the session...
exit signaled
[2015/12/02-11:17:47.494171] xio_task.h:331               [ERROR] - tasks inventory: 311/312 = missing:1
[2015/12/02-11:17:47.494195] xio_task.c:341               [ERROR] - pool_name:ctx:0x62e450 - primary_pool_rdma: in use: task:0x7f14fdcd60e0, type:0x12

client

-bash-4.2$ ./run_client.sh rdma53.csel.io 9999
BEGIN Message Send
Header: PUT
Sending: KEY1 and VALUE1
session event: connection established. reason: Success
Received response to : 0x7f50b0012a08, message #140574389561864d
Response header: (0x7f50abba8064) PUT
response received is : (0x7f50abba8069) SUCCESS
BEGIN Message Send
Header: GET
Sending: KEY2 and VALUE2
exit signaled
[2015/12/02-11:19:04.258733] xio_idr.c:176                [ERROR] - user object leaked: 0x13736a0, type:struct xio_connection
[2015/12/02-11:19:04.258872] xio_idr.c:176                [ERROR] - user object leaked: 0x136b0b0, type:struct xio_session

Any pointers would be great, only so much I can get from looking at the hello world example. Does accelio have an IRC channel?

I just send a xio_msg as response, which sgl_type is XIO_SGL_TYPE_IOV_PTR, the nents = 5, the max_nents = 5, the xio shows "invalid out message". Is there any way to send one message with larger nents, instead of split this message into pieces?

I'm interested in constructing a simple server that allows clients to run operations with IBV_WR_ATOMIC_FETCH_AND_ADD. Is this something that will be possible within the Accelio framework?

Hi, All:
Recently I am interested in accelio and focusing on its performace over RDMA with InfiniBand devices. I found there are many examples and tests such as xio_s/c, xio_lat_s/c, xio_mt_s/c, xio_bidi_s/c, xio_oneway_s/c, xio_ow_s/c, etc, which shared the libxio or libraio.
Since accelio supports pluggable transport level, which seamlessly use RDMA to replace TCP, besides, the performace of RDMA transport is much better than TCP transport (with IPoIB), so how to use accelio simply to replace traditional TCP socket with RDMA interface?
Thanks for your attention!

In the example usr/hello_world_daemon/xiosrvd.c, xio_init() is called after daemon(3). Should this order preserved in any programs which use accelio?

i found this page https://community.mellanox.com/docs/DOC-2158 and have a question - does reconnect work only with mellanox ofed or it works with openfabrics ofed too?

When I use one way xio_send_msg api, I can free the message by on_ow_msg_send_complete callback. I can free response message by on_msg_send_complete callback. But how to free the request message which is already sent to other side?

Do I need free request message by myself?

Accelio library will fail when invoke make install command when not root user. The '/sbin/ldconfig' will be invoked in normal user and cause no permission error.

It is invoked by src/usr/Makefile install-exec-hook section.

At first, create a test file on server, here is /tmp/test_256m, whose size is 256 MB.
Then run an raio server process as follows:

./raio_server -a 172.16.10.212 -p 2015

Run raio client process with following options:

./raio_client -a 172.16.10.212 -p 2015 -f /tmp/test_256m -l 1 -b 65536

reading started .
Then it hangs in dead loop until you press CTRL+C to exit.
The normal result is as follows with the other block size, e.g, 32768 as follows:

./raio_client -a 172.16.10.212 -p 2015 -f /tmp/test_256m -l 1 -b 32768

reading started .
reading completed
done [8192/8192] time: 215.21 msecs, size: 256.00 MB, rate: 1247.30 MBps, pps: 39914, block size: 32768 B
good bye

does accelio support IB、iWARP、RoCE？
Thanks!

It seems that xio_send_request() is nonblocking and xio_context_run_loop() seems to cause blocking request sending. Is this always correct?
(I just want to check my understanding is correct or not)

Is xio_mempool functions thread safe? Can I alloc and free memory from same pool in different thread? Can I add_slab in different thread to same mempool?

Hi,

In nbdx_connect_work context, request will be sent in stack on_response->xio_release_response->xio_connection_xmit.

In nbdx_request context, request will be sent in stack
nbdx_transfer->xio_send_request->xio_connection_xmit.

Obviously, the two contexts are different, nbdx_connect_work
is xio connection context, nbdx_request is application context,
how to ensure xio send safely in this condition.

Thanks.

--haiwei.

Hi,

I'm using a node whose ip addr output is like below:

$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:1e:67:24:49:8a brd ff:ff:ff:ff:ff:ff
    inet 192.168.12.237/24 brd 192.168.12.255 scope global eth0
    inet6 fe80::21e:67ff:fe24:498a/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000
    link/ether 00:1e:67:24:49:8b brd ff:ff:ff:ff:ff:ff
4: ib0: <BROADCAST,MULTICAST,UP> mtu 4092 qdisc mq state UNKNOWN qlen 1024
    link/infiniband a0:00:02:20:fe:80:00:00:00:00:00:00:00:1e:67:03:00:24:49:8f brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
5: ib1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 qdisc pfifo_fast state UP qlen 1024
    link/infiniband a0:00:00:2a:fe:80:00:00:00:00:00:00:00:02:c9:03:00:e6:98:40 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
    inet6 fe80::202:c903:e6:9840/64 scope link 
       valid_lft forever preferred_lft forever
6: ib2: <BROADCAST,MULTICAST,UP> mtu 65520 qdisc pfifo_fast state UNKNOWN qlen 1024
    link/infiniband a0:00:00:2c:fe:80:00:00:00:00:00:00:00:02:c9:03:00:e6:98:48 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff

For using the ib interfaces, how should I specify a URL as a parameter of xio_session_create() and xio_bind()? Based on the doc, a format of rdma://... is used. Can I specify rdma:// ? How many ports can I use per interface?

Sorry for my disorganized question. I'm completely new to infiniband and know virtually nothing about it :(

Lunchpad issue #12299984
In function kmem_cache_create "name" will be assigned to cache's member ptr.but next loop it will be refilled by sprintf

I could find this statement in the accelio doc (http://www.accelio.org/wp-admin/accelio_doc/index.html) :

In order to destroy the context, xio_context_destroy method should be called. At this point, all resources associated with this context (session, connection, etc.) should already be destroyed.

What happen if resources of context aren't destroyed before destroying context? In addition, what does the "etc" mean? I want to know that applications would be corrupted or not if some resources are not destroyed.

In libxio the function "int xio_redirect()" can redirect a connection to a different node or portals or process, so what is the concrete application scenario and how to redirect an xio connection which has been established?
Thanks!

Hi!

I'm doing rpm packaging for accelio and I see, that there is make install difference in build accelio as a kmod and as a user space library:

kmod:

configure command:

./configure --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --disable-static --enable-kernel-module --with-kernel=/usr/src/kernels/3.10.0-123.el7.x86_64 --enable-stat-counters=no --enable-extra-checks=no

make install (part of output):

/usr/bin/install -c -m 644 xio_core.ko /lib/modules/3.10.0-123.el7.x86_64/extra/net/xio/xio_core.ko
/usr/bin/install -c -m 644 `pwd`/../../../include/libxio.h /opt/xio/include
/usr/bin/install -c -m 644 `pwd`/../../../include/xio_base.h /opt/xio/include
/usr/bin/install -c -m 644 `pwd`/../../../include/xio_kernel.h /opt/xio/include

user space:

configure command:

./configure --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --disable-static --enable-stat-counters=no --enable-extra-checks=no

make install:

 /usr/bin/install -c -m 644 ../../include/libxio.h ../../include/xio_base.h ../../include/xio_user.h ../../include/xio_predefs.h '/usr/local/include/'

So, in case --enable-kernel-module the dest dir is: /opt/xio/include and in user space lib it is: /usr/local/include/
Is it an expected behaviour? Should I write if condition in rpm spec?

Also, make install ignores flag DESTDIR:

+ make DESTDIR=/home/ec2-user/rpmbuild/BUILDROOT/libxio-1.6-1.el7.centos.x86_64 install
Making install in src/kernel/xio
make[1]: Entering directory `/home/ec2-user/rpmbuild/BUILD/accelio-1.6/src/kernel/xio'
export NOSTDINC_FLAGS
make -C /usr/src/kernels/3.10.0-123.el7.x86_64 SUBDIRS=`pwd`  KBUILD_EXTRA_SYMBOLS=/usr/src/ofa_kernel/default/Module.symvers modules
make[2]: Entering directory `/usr/src/kernels/3.10.0-123.el7.x86_64'
  Building modules, stage 2.
  MODPOST 1 modules
make[2]: Leaving directory `/usr/src/kernels/3.10.0-123.el7.x86_64'
mkdir -p /home/ec2-user/rpmbuild/BUILDROOT/libxio-1.6-1.el7.centos.x86_64/lib/modules/3.10.0-123.el7.x86_64/extra/net/xio
mkdir -p /opt/xio/include
/usr/bin/install -c -m 644 xio_core.ko /home/ec2-user/rpmbuild/BUILDROOT/libxio-1.6-1.el7.centos.x86_64/lib/modules/3.10.0-123.el7.x86_64/extra/net/xio/xio_core.ko
/usr/bin/install -c -m 644 `pwd`/../../../include/libxio.h /opt/xio/include
/usr/bin/install: cannot remove '/opt/xio/include/libxio.h': Permission denied
make[1]: *** [install-y] Error 1
make[1]: Leaving directory `/home/ec2-user/rpmbuild/BUILD/accelio-1.6/src/kernel/xio'
make: *** [install-recursive] Error 1
error: Bad exit status from /var/tmp/rpm-tmp.kauXdh (%install)

Is it a special situation, or it's a bug?
Thanks for any help.

After installing RDMA development dependencies I am able to compile Accelio, but when running with TCP transport options, the examples still fail as it appears they are trying to query for RDMA hardware.

For example

[nwatkins@kyoto hello_world_daemon]$ ./xiosrvd -r tcp
librdmacm: Warning: couldn't read ABI version.
librdmacm: Warning: assuming: 4
librdmacm: Fatal: unable to get RDMA device list

Hi, eyal:
It is easy to create a 1:1 peer model with libxio, but if there has been an xio_connection between a server and a client, how to create a 1:many peers model with xio? In other word, on the server side, we use a parent process to listen on a port for waiting a remote client's connection, and then use a child process to listen on another port to wait another remote client's connection. After calling xio_init() on the second client, the xio_connection status is 3 but the status of IB WC happens to "local protection error" in xio_rdma_datapath.c. So what is the correct flow of creating a 1:many peers xio_connection?
Thanks a lot!

Hi accelio team,

I found an unexpected (may not bug) behavior of accelio in my implementation: even a server sets nents of response with vmsg_sglist_set_nents() (e.g. vmsg_sglist_set_nents(pomsg, 1); ), a client sees a response with 2 nents (vmsg_sglist_nents(pimsg) == 2).

I expect maximum nents of response is always 1 because the server uses a header member and first one entry of xio_vmsg.

(sgl_type is XIO_SGL_TYPE_IOV_PTR)

Is this ordinal behavior of accelio? If so, how can I obtain the above expected behavior?

Thanks,
Hitoshi

When sending messages during reconnect xio returns synchronous error (ENOTCONN) rather than inserting the message back to the connection queue for a later retry (to be issued after reconnect succeeds). Is this behaviour intentional? what is being expected from the application during accelio reconnect attempts? shouldn't reconnect attempts be totally hidden from the user?

xio_accept() accepts user context with its 4th and 5th parameters. Can this user context be obtained in succeeding call of on_msg() callback?