Hi,

rpma_write_atomic() just passes 8 bytes of data to rpma_mr_write(), so does the atomic write operation depend on RDMA write with 8 bytes? If wrong, How to ensure atomic write operation by specifying 8 bytes of data?

Best Regards,
Xiao Yang

Make get_timestamp_prefix() a bit more thread-safe by
using localtime_r() instead of localtime() inside get_timestamp_prefix().

Please verify if all of valgrind (helgrind and drd) suppressions are still needed.

See: #990

I run make doc on current master (a38e166) and it does not work.

Error: current list of manuals(3) does match the file: /tmp/rpma/doc/manuals_3.txt
1d0
< librpma.7
Error: current list of manuals(7) does match the file: /tmp/rpma/doc/manuals_7.txt
In order to fix it, run 'make doc-fix'

I'm working on clean, just downloaded repo: see full_log.txt

I cloned codes in the master branch and tried to build them on ubuntu 16, but encountered some errors.

cmake ..

-- Checking for module 'libibverbs'
--   No package 'libibverbs' found
-- Checking for module 'libibverbs' w/o PkgConfig
--   Found libibverbs w/o PkgConfig
-- Checking for module 'librdmacm'
--   No package 'librdmacm' found
-- Checking for module 'librdmacm' w/o PkgConfig
--   Found librdmacm w/o PkgConfig
-- Checking for module 'libpmem>=1.6'
--   No package 'libpmem' found
-- On-Demand Paging (ODP) in libibverbs supported - Success
-- Using a new signature of rdma_getaddrinfo()
CMake Warning at tests/CMakeLists.txt:31 (message):


  WARNING: The RPMA_SOFT_ROCE_IP environment variable is not set! It should
  contain an IP address of a SoftRoCE-configured network interface.

  127.0.0.1 will be used instead, the tests requiring SoftRoCE can fail!

  There are possible two alternative actions:

  a) Run 'make config_softroce' to configure SoftRoCE and to get the IP of a
  SoftRoCE-configured network interface and set the RPMA_SOFT_ROCE_IP
  environment variable to silent this message.

  or:

  b) Set the TESTS_SOFT_ROCE cmake variable to OFF in order to disable tests
  that require a SoftRoCE-configured network interface.


-- Checking for executable 'pylint'
--   Found pylint
-- Configuring done
-- Generating done
-- Build files have been written to: /home/gray/PMDK/pmdk/rpma/cmake

make  # remake

[  0%] Built target checkers
[  2%] Built target rpma
[  2%] Built target test_backtrace
[  2%] Building C object tests/unit/conn/CMakeFiles/conn-flush.dir/__/common/mocks-rdma_cm.c.o
/home/gray/PMDK/pmdk/rpma/tests/unit/common/mocks-rdma_cm.c:303:1: error: conflicting types for ‘rdma_getaddrinfo’
 rdma_getaddrinfo(const char *node, const char *port,
 ^~~~~~~~~~~~~~~~
In file included from /home/gray/PMDK/pmdk/rpma/tests/unit/common/mocks-rdma_cm.c:8:0:
/usr/include/rdma/rdma_cma.h:706:5: note: previous declaration of ‘rdma_getaddrinfo’ was here
 int rdma_getaddrinfo(char *node, char *service,
     ^~~~~~~~~~~~~~~~
tests/unit/conn/CMakeFiles/conn-flush.dir/build.make:117: recipe for target 'tests/unit/conn/CMakeFiles/conn-flush.dir/__/common/mocks-rdma_cm.c.o' failed
make[2]: *** [tests/unit/conn/CMakeFiles/conn-flush.dir/__/common/mocks-rdma_cm.c.o] Error 1
CMakeFiles/Makefile2:1446: recipe for target 'tests/unit/conn/CMakeFiles/conn-flush.dir/all' failed
make[1]: *** [tests/unit/conn/CMakeFiles/conn-flush.dir/all] Error 2
Makefile:145: recipe for target 'all' failed
make: *** [all] Error 2

thx!

In the light of #958 it might be required to add yet another field to struct rpma_completion which would allow to track down the connection from which the completion comes from. Since after making CQ for receives shared by multiple connections it is no longer obvious from which connection the completion comes from.

It is another situation where this public structure might be extended (you might recall an immediate data extension a few months ago). It is not a good sign since, we hope, we are heading to the first stable release of the library. It might be a necessity to hide this structure internals and allow accessing its values only via getters but it would require allocating at runtime which is a drawback.

A possible rework might also include reconsideration of using enum rpma_op. At this point, it is a simple translation of enum ibv_wc_opcode which may be unnecessary.

To sum up, all things to consider:

• extending struct rpma_completion with a connection which the completion comes from
• banning direct access to struct rpma_completion fields and provide getters/setters
• the value of having enum rpma_op

Currently, struct rpma_conn_cfg (in the near future probably also struct rpma_srq_cfg) provides some defaults for parameters which values should be known by the software developer e.g. size of CQ, SQ, RQ. Without defaults, these parameters will become required. It is very important to make this decision consciously because introducing a required parameter is a not backward compatible change. On the other hand, knowing the exact values of these parameters is required to write a non-trivial application using librpma.

Let's discuss.

Your RPMA build may fail with the following or similar output:

-- Checking for module 'libibverbs>=1.0'
--   Package 'libibverbs', required by 'virtual:world', not found
CMake Error at /usr/local/share/cmake-3.18/Modules/FindPkgConfig.cmake:545 (message):
  A required package was not found
Call Stack (most recent call first):
  /usr/local/share/cmake-3.18/Modules/FindPkgConfig.cmake:733 (_pkg_check_modules_internal)
  CMakeLists.txt:105 (pkg_check_modules)

It may be caused by two possible situations:

1. The dev(el) package is not installed in the system
2. The dev(el) package does not provide pkg-config files

Solution 1.

You can verify if the dev(el) package is installed using this or similar command:

$ dnf list installed | grep 'rdma\|verbs'
libibverbs.x86_64                            16.2-3.fc28                     @fedora
librdmacm.x86_64                             16.2-3.fc28                     @fedora
rdma-core.x86_64                             16.2-3.fc28                     @fedora
rdma-core-devel.x86_64                       16.2-3.fc28                     @fedora

If you have libibverbs-devel or rdma-core-devel on the list above you can jump right to Solution 2.
Otherwise, you have to install one of the following packages:

• rdma-core-dev(el) OR
• libibverbs-dev(el) OR
• OFED provided by your RNIC manufacturer

Solution 2.

You can verify if the dev(el) package is installed as follows:

$ ls -l /usr/lib64/pkgconfig/ | grep verbs
-rw-r--r--. 1 root root 379 08-17 16:38 libibverbs.pc

If you have on the list above libibverbs.pc file but still experiencing the issue please contact us.
Otherwise, your system is not yet supported by the RPMA build system. The RPMA team have identified the problem and we will prepare the required enhancements soon. Please stay tuned for the updates.

In the meantime, a workaround for the issue is to manually copy the libibverbs.pc and librdmacm.pc files to /usr/lib64/pkgconfig on the system. This workaround is not seamless. May not work for all systems or may require manual changes to the contents of the *.pc files.

pc_issue.zip

Note

For the simplicity sake, I have called development packages as dev(el). Whereas some Linux distributions provide them as dev packages (e.g. libibverbs-dev) and other as devel packages (e.g. libibverbs-devel).

Acknowledgement

Thank you @graceleeis for reporting this issue.

References

• linux-rdma/rdma-core#373 - Generate and install pkgconfig files for the RDMA libraries (the *.pc files are available from rdma-core v20.1)
• ofiwg/libfabric#2070 - verbs provider needs to strengthen libibverbs link check
• arn314/libfabric@f3ba382 - verbs: Strengthen configure check to workaround libtool behavior

Hi, as far as I know, the IETF and IBTA are both planed to extend RDMA protocol to support persistent memory. Three new verbs including flush/verify/atomic will be introduced in the future. I want to know what is the difference between rpma and RDMA with PMEM extensions. If they are for the same purpose, will rpma be obsoleted after the RDMA with PMEM extensions is mature. Thanks.

It failed to build 09-flush-to-persistent-GPSPM example with the below method. What's the right way to build it out?

$ git clone https://github.com/pmem/rpma.git
$ cd rpma
$ git rev-parse HEAD
d8f4fed0b72060bf42a08d2961dd03dd878057dc
$ mkdir build
$ cd build
$ cmake .. -DBUILD_DOC=OFF
$ ls examples/
01-connection        03-read-to-persistent   05-flush-to-persistent   07-atomic-write        CMakeFiles           CTestTestfile.cmake  Makefile
02-read-to-volatile  04-write-to-persistent  06-multiple-connections  08-messages-ping-pong  cmake_install.cmake  log                  template

If you are just curious what librpma API call is thread-safe and which is not I strongly recommend reading the documentation: https://pmem.io/rpma/manpages/v0.9.0/librpma.7.html#thread-safety

But let's assume you are a good egg wanting to help us prove the thread-safety of the librpma API calls. If it is the case I want to collect here all required information that may be useful for guiding you through the process.

Adding missing suppressions

It is expected that libibverbs and librdmacm may require providing additional suppressions. The role of these suppressions in the whole process is yet to be described.

To add the missing suppressions, follow these steps:

1. Push a commit with the change in file helpers.cmake:
   • Add the --gen-suppressions=all command to these lines.
     • https://github.com/pmem/rpma/blob/master/tests/cmake/helpers.cmake#L108
     • https://github.com/pmem/rpma/blob/master/tests/cmake/helpers.cmake#L111
   • As follows:

   set(TRACE valgrind --error-exitcode=99 --tool=helgrind --gen-suppressions=all --suppressions=${TEST_ROOT_DIR}/helgrind.supp)
   set(ENV{LIBRPMA_TRACER_HELGRIND} 1)
elseif(${TRACER} STREQUAL drd)
   set(TRACE valgrind --error-exitcode=99 --tool=drd --gen-suppressions=all --suppressions=${TEST_ROOT_DIR}/drd.supp)

2. On Circle Ci open the build log.
3. Modify the resulting suppressions and add them to the appropriate files:
   • https://github.com/pmem/rpma/blob/master/tests/drd.supp
   • https://github.com/pmem/rpma/blob/master/tests/helgrind.supp

A suppression example

Suppression fragment obtained from Circle CI:

{
   <insert_a_suppression_name_here>
   drd:ConflictingAccess
   fun:strlen
   fun:__tzstring
   fun:__tzfile_compute
   fun:__tz_convert
   fun:__vsyslog_internal
   fun:syslog
   fun:rpma_log_default_function
   fun:rpma_ep_listen
   fun:thread
   fun:mtt_thread_main
   obj:/usr/lib/x86_64-linux-gnu/valgrind/vgpreload_drd-amd64-linux.so
   fun:start_thread
}

A ready suppression:

{
   Race while reading the name of time zone ("GMT").
   drd:ConflictingAccess
   ...
   fun:__tzfile_compute
   fun:__tz_convert
   fun:__vsyslog_internal
   fun:syslog
   fun:rpma_log_default_function
   ...
}

Stay tuned for more.

07-atomic-write example sometimes fails:
See: https://app.circleci.com/pipelines/github/pmem/rpma/67/workflows/702155b0-afd7-424b-9a86-8e89cea578a8/jobs/67

*** Running example: 07-atomic-write 
Starting the server ...
Apr 19 06:44:09.005263 [36257] *NOTE*  ep.c:  92: rpma_ep_listen: Waiting for incoming connection on 10.142.1.161:7204
Starting the client ...
Apr 19 06:44:10.006313 [36262] *NOTE*  conn_req.c: 369: rpma_conn_req_new: Requesting a connection to 10.142.1.161:7204
Apr 19 06:44:10.007670 [36262] *NOTE*  conn.c: 190: rpma_conn_next_event: Connection established
Apr 19 06:44:10.007723 [36262] *INFO*  mr.c: 457: rpma_mr_remote_from_descriptor: new rpma_mr_remote(raddr=0x55d6258d2000, size=1040, rkey=0x1911, usage=0x19)
Apr 19 06:44:10.007832 [36257] *NOTE*  conn.c: 190: rpma_conn_next_event: Connection established
used value: 16
Apr 19 06:44:10.007953 [36262] *NOTE*  conn.c: 230: rpma_conn_disconnect: Requesting for disconnection
Apr 19 06:44:10.008022 [36257] *NOTE*  conn.c: 190: rpma_conn_next_event: Connection closed
Apr 19 06:44:10.008062 [36257] *NOTE*  conn.c: 230: rpma_conn_disconnect: Requesting for disconnection
Apr 19 06:44:10.008167 [36262] *NOTE*  conn.c: 190: rpma_conn_next_event: Connection closed
Error: example 07-atomic-write FAILED!

As a library user, I would like to have access to the raw memory pointer and size of registered memory so I do not need to pass them separately from function to function but could pass only mr_local pointer.

W: librpma-dev: groff-message usr/share/man/man3/rpma_log_set_function.3.gz 44: warning: macro '..' not defined

And indeed, this is not a benign warning: the man page contains an ugly non-indented bold (in "man") or blue-background (in "mc") ...); on the left side of the screen.

Apparently, src2man fails to cope with lines starting with a dot. Not sure how to fix this problem — perhaps moving the dots elsewhere or using a non-ASCII dot would be a good enough workaround?

A full build spams stderr with:

cat: ../../utils/mans_header.md: No such file or directory

This won't work unless you build from inside the tree, in a manually created subdir exactly one level deep. Typical build scripts use a completely different dir.

Also, failure of the cat fails to fail the build.

#include <infiniband/verbs.h>

int main (int argc, char *argv[]) {
    if (argc < 3) {
        fprintf(stderr, USAGE_STR, argv[0]);
        exit(-1);
    }
    ibv_fork_init(); //to support fork

    char *addr = argv[1];
    char *port = argv[2];
    int ret;

    struct rpma_peer *peer = NULL;
    void *dst_ptr;
    struct rpma_mr_local *dst_mr;
    struct ibv_context *dev = NULL;

    ret = rpma_utils_get_ibv_context(addr, RPMA_UTIL_IBV_CONTEXT_LOCAL, &dev);
    if (ret)
        return ret;

    rpma_peer_new(dev, &peer);

    size_t dst_size = 320;
    dst_ptr = malloc_aligned(dst_size);
    rpma_mr_reg(peer, dst_ptr, dst_size, RPMA_MR_USAGE_READ_DST,
                      &dst_mr);

    int pid = fork();
    if (pid == 0) {
        FILE *fp = fopen("tmpfile", "w");
        fclose(fp);  //crash here
        _exit(0);
    } else if (pid < 0) {
        _exit(0);
    } else {
        int status;
        waitpid(pid, &status, 0);
    }

    return 0;
}

the up code invoke ibv_fork_init to support fork , after fork clild process will crash in fclose. crash will not happen if ibv_fork_init or rpma_mr_reg not invoked. I also find it's safe if ibv_fork_init and rpma_mr_reg invoke in different thread.

the backtrace like this:
#0 0x00007f9f0d2dd81d in _int_free () from /lib64/libc.so.6
#1 0x00007f9f0d2ca047 in fclose@@GLIBC_2.2.5 () from /lib64/libc.so.6
#2 0x0000000000402198 in main (argc=-1, argv=0x800076)

It seems that normal fio command(from fio package) cannot parse the fio job of RPMA, for example:

# fio /dev/shm/librpma-server-20-12-29-133029.fio
fio: engine librpma_server not loadable
fio: failed to load engine
Bad option <bindname=${bindname}>
Did you mean filename?
Bad option <port=7204>
fio: job global dropped

Could you tell me which fio can be used in rpma_fio_bench.sh.
In another word, how to run rpma_fio_bench.sh correctly?

(This is a duplicate of the issue opened in FIO)

Description of the bug:

In FIO, when the librpma_apm server is setup to write in a file in a PMEM module formated in ext4 and mounted with DAX option, RNR NACKs packets appear during the transfer (meaning that the server is "not ready" to receive data).
Each time a RNR NACK is sent, the client waits a moment to let the server become ready, and retries, which ruins performances.
In our tests, each 10KB transfer was sent twice, following this pattern :

• Client sends 10KB
• Server returns RNR NACK
• Client waits...
• Client sends 10KB
• Server returns ACK

The exact same job file writing in our PMEM in DAX mode (/dev/dax1.0) works perfectly.

Environment:

• Client and Server: Ubuntu 20.04, Mellanox ConnectX-5
• Server: 128GB Optane DCPMM, in fsdax mode (/dev/pmem0), with ext4, mounted in DAX mode. DDIO is deactivated, so we can enable "direct_write_to_pmem".

fio version: 3.27

Reproduction steps

• Format the NVMM module in ext4, and mount it with mount -o dax /dev/pmem0 /mnt/pmem0
• Create an empty file ( /mnt/pmem0/my_file )
• Start the server, start a wireshark capture, start the client

client.fio

[global]
ioengine=librpma_apm_client
create_serialize=0
serverip=<--replaceme-->
port=7204
thread

[client]
numjobs=1
group_reporting=1
sync=1
iodepth=2
iodepth_batch_submit=1
rw=write
blocksize=10KB
size=20KB

server.fio

[global]
ioengine=librpma_apm_server
create_serialize=0
kb_base=1000
serverip=<--replaceme-->
port=7204
thread

[server]
direct_write_to_pmem=1
numjobs=1
size=100MiB
filename=/mnt/pmem0/my_file

Partial workaround

We had to convert our PMEM namespace mode from "fsdax" to "dax", which transforms /dev/pmem1 into /dev/dax1.0 wit the following command :
sudo ndctl create-namespace -f -e namespace1.0 --mode=dax

Then replace filename=/mnt/pmem0/my_file by filename=/dev/dax1.0 in server.fio

This example will show it is possible to make data persistent single-sidedly using RECV buffers carved out from PMEM and following rpma_send() with rpma_flush().

SERVER

prepare memory (recv_mr) 
create a new peer 
listening endpoint 
register memory (rpma_mr_reg(USAGE_RECV && USAGE_FLUSH)) 
receive an incoming connection request 
initial receive to be prepared for the first message (rpma_conn_req_recv) 
prepare private_data (recv_mr) 
accept the connection 
wait for the connection to be established 
while (1) { 
        do { 
                prepare completions, get one and validate it 
        } while (!recv_cmpl); 
        if (*recv == I_M_DONE) 
                break; 
        rpma_recv 
} (!recv_cmpl); 
disconnect 

CLIENT

prepare memory (send_mr) 
register memory(RPMA_MR_USAGE_SEND) 
establish a new connection to a server listening at addr:port 
unpack private date 
while (--rounds) { //rounds -argument provided by user 
        rpma_send 
        rpma_flush (mr_region from private_data) 
        do { 
                prepare completions, get one and validate it 
        } while (!send_cmpl ); 
} 
rpma_send (I_M_DONE) 
disconnect 

Having an SRQ on its way (#958 #685 #783 #737) we are close to a situation when it will be no longer obvious from which struct rpma_conn a particular struct rpma_completion comes from. It will be handy to allow tracking it down.

struct ibv_wc has a field named uint32_t qp_num which allegedly is:

Local QP number of completed WR. Relevant for Receive Work Completions that are associated with an SRQ

Indeed, struct ibv_qp also has the same field which is:

The number of this Queue Pair. A 24 bits value, which is unique per RDMA device. As QPs are destroyed and created, QP numbers may be reused. However, at a given point in time, only a single QP in the RDMA device will exist with the given number. The user cannot control or influence this value

Roughly speaking we can imagine two possible ways of solving this issue:

• allow querying the uint32_t qp_num value for struct rpma_conn and struct rpma_completion which will allow the end-user to match these two entities or
• allow getting directly struct rpma_conn related to a particular struct rpma_completion

Let's discuss this fascinating issue. :-)

Ref: https://www.rdmamojo.com/2013/02/15/ibv_poll_cq/
Ref: https://www.rdmamojo.com/2012/12/21/ibv_create_qp/

ib_read.sh needs to call some ib tools(ib_read_lat, ib_read_bw, etc), but I cannot find them in RPMA.
Are they going to be added in future(e.g. next month)?

Currently, the application running a server side (ep) does not have access to a connection private data until a connection is established.
In this case, the application does not have the possibility to tune the connection environment (memory regions, etc.) based on the peer application's data.
rpma_conn_private_data is already a part of rpma_con_req. But no interface makes them available for the application.

The proposal of the solution has been drafted in #646.
The final solution should cover UT and an example.

Use the libpmem2 library.
As an example, see https://github.com/pmem/pmdk/blob/master/src/examples/libpmem2/advanced/advanced.c
Using libpmem2 we can map exactly the area we need:

• pmem2_config_set_offset()
• pmem2_config_set_length()

Requires changes in Makefile and configure.

For changes related to librpma_apm_* and librpma_gpspm_* engines please use branch: https://github.com/pmem/fio/tree/rpma
For changes related to librpma_aof_* engine please use this branch: https://github.com/pmem/fio/tree/aof

Note: This example runs properly only on HW.

CLIENT

Perform all steps of Example 7 with changes:

parameters: path to the file being rewritten, drop_connection

/* defined log structure */ 
struct log { 
        char signature[LOG_SIGNATURE_SIZE]; 
        uint8_t lock; /* 1 - client writes, 0 - log free */ 
        /* last written data (aligned to RPMA_ATOMIC_WRITE_ALIGNMENT) */ 
        uint64_t used; 
        char data[LOG_DATA_SIZE]; 
}; 

function reconnect() 
{ 
        disconnect
        establish a new connection (several attempts may be needed, like FIO engines
                                    https://github.com/pmem/fio/blob/aof/engines/librpma_fio.c#L209 )
        rpma_read(used) 
        continue writing from reading used 
} 

rpma_read(lock) 
if (lock == 1) { 
        fprintf(stderr, “busy”); 
        disconnecting and closing the program 
} 
rpma_write_atomic (lock=1) 
… (example 07) 
if (drop_connection) { /* user specified parameter */ 
        reconnect() 
} 

ret = rpma_conn_completion_get();
if (ret && ret != RPMA_E_NO_COMPLETION)
	goto err;
if (ret == RPMA_E_NO_COMPLETION) {
        rpma_conn_completion_wait() 
        rpma_conn_completion_get() 
}
if (cmpl.op_status == IBV_WC_RETRY_EXC_ERR) { 
        ret = rpma_conn_next_event() 
        if (!ret && conn_event != RPMA_CONN_CLOSED) { 
                fprintf(stderr, 
                 	 "rpma_conn_next_event returned an unexpected event: %s\n", 
                	 rpma_utils_conn_event_2str(conn_event)); 
                disconnecting and closing the program 
         } 
         reconnect() 
} 
rpma_write_atomic (lock = 0) 
rpma_flush 
disconnect 

SERVER

Perform all steps of Example 7 with changes:
parameters: drop_connection

after accept connection: 
if (drop_connection) {/* user specified parameter */ 
        sleep (5); 
        disconnect 
        if (lock == 1) { 
                create new connection 
        } 
} 

do { 
        wait for RPMA_CONN_CLOSED 
        disconnect 
        if (lock == 1) { 
                create new connection 
        } 
} while (lock != 0) 

print saved data 

• Please add parameters descriptions of all functions to man pages.

i.e. it's pretty hard to figure out if rpma_conn_private_data pointer passed to rpma_conn_get_private_data() may be NULL or not.

• Error 404:
  http://pmem.io/rpma/master/rpma_peer_cfg_get_persistent_flush_supported.3
  http://pmem.io/rpma/master/rpma_peer_cfg_set_auto_flush.3
  http://pmem.io/rpma/master/rpma_peer_cfg_set_ddio.3

• XXX in error list http://pmem.io/rpma/master/rpma_conn_apply_remote_peer_cfg.3

• No description in some methods. i.e. http://pmem.io/rpma/master/rpma_conn_cfg_get_cq_size.3

There are workflows in which processing incoming messages might be blocked by unfinished processing of previously sent messages (send queue slot is not available until completion is read from completion queue).
That forces the application to buffer receiving messages until sending message completions are collected to have a free slot in RDMA send queue.

The solution would be to have optionally a separate completion queue for receiving messages.
The receiving completion queue should have a separate completion channel and separate API to get the completion channel file descriptor. A separate API to collect completions is also required.
In case, a receiving completion queue is created main API rpma_conn_completion_get should only return the sending completions (send/read/write).

When rpma_info_new() and rdma_create_id() are swapped in rpma_ep_listen():

https://github.com/ldorau/rpma/tree/rpma-swap-rpma_info_new-and-rdma_create_id-in-rpma_ep_listen

ep-test fails:

[ RUN      ] ep_listen_test_info_new_E_NOMEM
[  ERROR   ] --- rdma_create_id: '%s' parameter still has values that haven't been checked.
/home/ldorau/work/rpma/tests/ep-test/ep-test.c:112: note: remaining item was declared here

on:

expect_value(rdma_create_id, channel, evch);

The package has just been rejected from Debian because of missing license text of CDDL.

The only file under CDDL, utils/cstyle, includes the following header:

# The contents of this file are subject to the terms of the
# Common Development and Distribution License (the "License").
# You may not use this file except in compliance with the License.
#
# You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
# or http://www.opensolaris.org/os/licensing.
# See the License for the specific language governing permissions
# and limitations under the License.
#
# When distributing Covered Code, include this CDDL HEADER in each
# file and include the License file at usr/src/OPENSOLARIS.LICENSE.
# If applicable, add the following below this CDDL HEADER, with the
# fields enclosed by brackets "[]" replaced with your own identifying
# information: Portions Copyright [yyyy] [name of copyright owner]

Both places listed in the license are gone:

• usr/src/OPENSOLARIS.LICENSE is not there on any Linux distro, and OpenSolaris itself is dead since 2010 (you can still obtain a fork, or the original in some dark corner of the Internet)
• www.opensolaris.org is down (and pretty obviously 410-compliant)

If we want to continue to ship this file, we'd need to obtain a copy of OPENSOLARIS.LICENSE and include it in our tree.

If no private data provided for rdma_accept(3) the client may find it anyway.
During our experiments, it was a chunk of zeroed-out memory.

Thanks @osalyk for finding this issue! :-)

Rationale

Currently, there are two ways of obtaining private data:

• rpma_conn_req_get_private_data() and
• rpma_conn_get_private_data().

Because of the limitations of the current implementation the private data obtained from the rpma_conn_req_get_private_data() is invalidated after accepting the connection request which is not an inherit, irremovable feature of the private data itself.

Aim

Improve the private data handling implementation in a way to remove the invalidation of the private data pointer during the transition from the connection request to the connection.

Ref: #911

Issue

The config_softroce.sh script assumes that if rdma command is available in the system it has implemented rdma link add command. Which is not true for all rdma releases. e.g. rdma binary from rdma-core-devel-22.4-4.el7_8 from CentOS 7.8 does not provide this command. It only implements rdma link show command.

A possible solution

Use good old rxe_cfg as a fallback in case rdma is not available in the system or rdma link add is not implemented.

Shall we move install-cmocka.sh into utils?
CMake should check cmoka availability and either install it automatically (after user's confirmation) or ask the user to install it using this particular script.

Shall we also add the uninstall option to this script?

Hi,

It seems that two flush types(RPMA_FLUSH_TYPE_PERSISTENT/RPMA_FLUSH_TYPE_VISIBILITY) specified by rpma_flush() are not passed into rpma_mr_read() and rpma_flush() just does RDMA read operation. So I wonder how rpma_flush() makes these types take effect?

Best Regards,
Xiao Yang

MT-test failed:
https://app.circleci.com/pipelines/github/pmem/rpma/194/workflows/74f049b3-b477-4ccc-8d1d-82883a9fed66/jobs/199

Maybe a synchro issue in the MT-framework?

• build system
• examples at startup

Just a place to discuss the API @ultra-nexus is working on.

Ref: #685
Ref: #783

Add both values to the struct ibv_post_send_mock_args and improve all tests.

Hi

There needs rpma_conn_completion_wait and rpma_conn_completion_get when I use rpma_write with RPMA_F_COMPLETION_ALWAYS. But When I use RPMA_F_COMPLETION_ON_ERROR, I don't need to wait the completion status. So how I know the operation is success or failed?

And a condition:
After I continuously perform the rma_write operation with RPMA_F_COMPLETION_ON_ERROR, and then use rma_read with RPMA_F_COMPLETION_ALWAYS, then if rma_read returns success, it means that the previous rma_write is also successful?
And is the order of these operations strictly executed in the order in which I called them? In other words, the rpma_read operation must be executed after the rpma_write is completely completed?

Thanks

Hi, as librpmem has some pain points, I think librpma will be the core library to use RDMA with NVDIMM. I have glaced this repository and have found that it only implements several functions currently. Does this library have a roadmap or something like design documentation so I can follow its development or contribute to it. Thanks.

Currently, all mocks requires for a certain unit test is delivered in the same C-file. This approach introduces some unneeded redundancy. Please, investigate the possibility of making at least some of the mocks common for all unit tests which are using them.

1. rai->ai_src_canonname in rpma_info_bind_addr() is always NULL.
2. rai->ai_src_canonname and rai->ai_dst_canonname in rpma_info_resolve_addr() are always NULL.

I am not sure how to make them show the correc IP address.
@janekmi @ldorau do you have any suggestion?

There's an (annoying) block against in-source builds, that has false positives:

[~/c/rpma](master)$ cmake .
-- The C compiler identification is GNU 10.2.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
CMake Error at CMakeLists.txt:12 (message):
  In-source build is not allowed.


-- Configuring incomplete, errors occurred!
See also "/home/kilobyte/c/rpma/CMakeFiles/CMakeOutput.log".
[~/c/rpma](master)$ cd /tmp/
[/tmp]$ mkdir r
[/tmp]$ cd r
[/tmp/r]$ cmake ~/c/rpma/
CMake Error at CMakeLists.txt:12 (message):
  In-source build is not allowed.


-- Configuring incomplete, errors occurred!
See also "/home/kilobyte/c/rpma/CMakeFiles/CMakeOutput.log".

Probably it is a problem with pkgconfig file.

Reference:
#29 (review)

Repro steps:

# with librpma installed in the system
$ cd rpma/src/examples/connect
$ mkdir build
$ cd build
$  cmake ..
$ make -j

Note: We have to add building examples from standalone cmake recipes in our CI.

When running tests build with -O2 -D_FORTIFY_SOURCE=2 (included in Debian's default flags for building packages), sprintf/syslog/stderr tests fail with errors such as:

• %s() has remaining non-returned values.
• '%s' parameter still has values that haven't been checked.

Doesn't reproduce with -O0.

-- Stderr:
[  ERROR   ] --- %s() has remaining non-returned values.
/home/kilobyte/c/rpma/tests/unit/log_default/function.c:107: note: remaining item was declared here

[  ERROR   ] --- %s() has remaining non-returned values.
/home/kilobyte/c/rpma/tests/unit/log_default/function.c:122: note: remaining item was declared here
%s() has remaining non-returned values.
/home/kilobyte/c/rpma/tests/unit/log_default/function.c:123: note: remaining item was declared here
%s() has remaining non-returned values.
/home/kilobyte/c/rpma/tests/unit/log_default/function.c:124: note: remaining item was declared here
syslog: '%s' parameter still has values that haven't been checked.
/home/kilobyte/c/rpma/tests/unit/log_default/function.c:125: note: remaining item was declared here
'%s' parameter still has values that haven't been checked.
/home/kilobyte/c/rpma/tests/unit/log_default/function.c:132: note: remaining item was declared here

plus many more of the above.

As you use a weird mocking library with many layers of indirection, it wasn't obvious to me where's the actual call, thus I'll let you take a look first before digging deeper.

It will be very useful to have a separate file to describe how to prepare development environment for librpma.
An example is here: https://gitlab.com/cmocka/cmocka/-/blob/master/INSTALL.md

• cmake ver ??? installation
• libibverbs-dev installation
• librdmacm-dev installation
• cmocka installation -> insyall_cmocka.sh
• libunwind-dev installation
• valigrind installation -> install_valgrind.sh
• txt2man installation -> install_txt2man.sh
• sudo apt install groff

In the examples that use libpmem providing the mapping may be improved using the libpmem2 library.
As an example, see https://github.com/pmem/pmdk/blob/master/src/examples/libpmem2/advanced/advanced.c
Using libpmem2 we can map exactly the area we need:

• pmem2_config_set_offset()
• pmem2_config_set_length()

Ceph is in C++, and RPMA is in C. And can RPMA provide the same mechanism as PMDK so that we can call RPMA interfaces directly?
https://github.com/pmem/pmdk/blob/master/src/include/libpmemlog.h#L46

Hello.

I tried to make install rpma on my server(with pm) and client(no pm). In server, everything runs OK. But in client, when I tried use make -j(under root), it will stop because of errors. Detailed information as follows:

# make -j
[0%] Built target test_backtrace
[0%] Built target doc_snippets_template-snippet
[3%] Built target rpma
[0%] Built target checkers
src2man: errors found in the "/root/2020/rpma-0.9.0/src/common.h" file:
awk: command line: 64: Warning: The escape sequence "\;" in the regular expression is not a valid operator
make[2]: *** [doc/CMakeFiles/doc.dir/build.make:57:doc/CMakeFiles/doc] Error 1
make[1]: *** [CMakeFiles/Makefile2:3151: doc/CMakeFiles/doc.dir/all] Error 2
make[1]: *** Waiting for unfinished tasks...
[7%] Built target conn-recv
[7%] Built target conn-completion_wait
[9%] Built target conn-apply_remote_peer_cfg
[11%] Built target conn-get_completion_fd
[13%] Built target conn-write_atomic
[14%] Built target conn-write
[16%] Built target conn-read
[20%] Built target conn-new
[20%] Built target conn-completion_get
[22%] Built target conn-flush
make: *** [Makefile:141:all] Error 2

By the way, client and server's distributions are all Ubuntu 20.04.

Thanks!

The rpma_cq_get_fd()'s manual is too short to inform the end-user what they can do with this file descriptor.