Plan to implement rpma flush & atomic_write based on new ibverbs flush & atomic_write APIs about rpma HOT 6 CLOSED

After thinking about the implementation of rpma_flush based on new ibverbs flush, I wonder how to make old API and new API works together. For example, we need to finish the remote write by a) and b):
a) write data into remote node by rpma_write()
b) flush data into remote node by rpma_flush()
In this case, rpma_write() uses the qp created by old rdma/ibv_create_qp() but rpma_flush() needs to use the qp created by new rdma/ibv_create_qp_ex(), how to convert one to the other one? or is it ok to use the unified rdma/ibv_create_qp_ex() by replacing old rdma/ibv_create_qp()?

@grom72 @ldorau do you have any suggestion?

from rpma.

We should not change public API (if possible).
To get access to new (MPE) verbs we should add a build flag (e.g. BUILD_WITH_MPE_SUPPORT), that forces alternative implementation based on ibv_cq_ex for all RDMA services.
qp_ex should be later obtain via ibv_qp_to_qp_ex (conn->id->qp) or alternatively stored in rpma_conn_req/rpma_conn
To create ibv_cq_ex let's use rdma_create_qp_ex instead of rdma_create_qp
Later we could abandon this BUILD_WITH_MPE_SUPPORT flag when we check that all supported OS's support ibv_qp_ex

from rpma.

I tried to replace rdma_create_qp() with rdma_create_qp_ex() by #2084.
This change is to prepare for new libibverbs flush and atomic write.

from rpma.

@grom72 @ldorau

I have created a PR to introduce libibverbs atomic write so we can discuss based on it.
#2085

from rpma.

@grom72 @ldorau

I have created a PR to introduce libibverbs flush so we can discuss the logic based on it.
#2104

from rpma.

Both #2085 and #2104 have been merged so close the issue.

from rpma.