"`Failed to modify UD QP to INIT on mlx5_0: Operation not permitted`" error with OpenMPI after updating to OFED 23.10 about easybuild-easyconfigs HOT 5 OPEN

We've pinned this down to the libfabric dependency we include with OpenMPI: when libfabric is removed as a dependency, the problem no longer occurs.

This requires rebuilding OpenMPI, so that's painful on a production system.

As a workaround, you can instruct OpenMPI to not use libfabric by passing the following options to mpirun:

mpirun -mca pml ucx -mca btl '^uct,ofi' -mca mtl '^ofi'

Or equivalently, you can set the following environment variables:

export OMPI_MCA_btl='^uct,ofi'
export OMPI_MCA_pml='ucx'
export OMPI_MCA_mtl='^ofi'

from easybuild-easyconfigs.

SURF (@casparvl) also saw a very similar issue, they worked around it by setting $FI_PROVIDER to verbs...

The error there was Invalid argument though, more similar to what was reported in openucx/ucx#9468

from easybuild-easyconfigs.

I've not seen this, but we added the following in December 2021 because of another issue we'd seen

            # avoid libfabric warning "unknown link width 0x10"
            # see https://github.com/ComputeCanada/software-stack-config/pull/19
            setenv OMPI_MCA_mtl "^ofi"
            setenv OMPI_MCA_btl "^openib,ofi"

from easybuild-easyconfigs.

Yep, we get these:

avoid libfabric warning "unknown link width 0x10"

As well regularly. We also solved it by setting those envrionment variables (I believe, would need to check what we set exactly)

from easybuild-easyconfigs.

More detail on what I hit here btw

from easybuild-easyconfigs.