HDF5: infinite loop error on Setonix (using singularity/3.8.6-mpi) about underworld2 HOT 5 OPEN

Hey Gilly,

Yeah this is an on going issue we have raise with setonix on several occasions. For now we are stuck with build bare metal builds on setonix. I will upload some instructions for it later today.

from underworld2.

After digging further into the Pawsey doco I found this https://pawsey.org.au/technical-newsletter/ (see 13 March 2023 entry):

Parallel IO within Containers
Currently there are issues running MPI-enabled software that makes use of parallel IO from within a container being run by the Singularity container engine. The error message seen will be similar to:

Example of error message

Assertion failed in file ../../../../src/mpi/romio/adio/ad_cray/ad_cray_adio_open.c at line 520: liblustreapi != NULL
/opt/cray/pe/mpich/default/ofi/gnu/9.1/lib-abi-mpich/libmpi.so.12(MPL_backtrace_show+0x26) [0x14ac6c37cc4b]
/opt/cray/pe/mpich/default/ofi/gnu/9.1/lib-abi-mpich/libmpi.so.12(+0x1ff3684) [0x14ac6bd2e684]
/opt/cray/pe/mpich/default/ofi/gnu/9.1/lib-abi-mpich/libmpi.so.12(+0x2672775) [0x14ac6c3ad775]
/opt/cray/pe/mpich/default/ofi/gnu/9.1/lib-abi-mpich/libmpi.so.12(+0x26ae1c1) [0x14ac6c3e91c1]
/opt/cray/pe/mpich/default/ofi/gnu/9.1/lib-abi-mpich/libmpi.so.12(MPI_File_open+0x205) [0x14ac6c38e625]

Currently it is unclear exactly what is causing this issue. Investigations are ongoing.

Workaround:

There is no workaround that does not require a change in the workflow. Either the container needs to be rebuilt to not make use of parallel IO libraries (e.g. the container was built using parallel HDF5) or if that is not possible, the software stack must be built “bare-metal” on Setonix itself (see How to Install Software).

I guess I'm about to install UW2 from source on Setonix... Would you have any step-by-step recipe at hands for this specific Cray machine? I found the one you put together for Magnus a few years back.

from underworld2.

Hey Gilly,
To update you on this.
Setonix's permission setup means I can't install things for a project I'm not a user in. So I'm trying to put together bare metal instructions for you that make things as smooth as possible from your end.
I'm testing some instructions I have put together this afternoon and if things work out I'll send them though later.

from underworld2.

Hi Jules,

I have been off grid for the past couple weeks and back in the office now. If you have a recipe at hand for the install I would love to give it to!
Cheers
Gilly

from underworld2.

Hi Gilly,

https://support.pawsey.org.au/documentation/display/US/Containers+changes
I'm going to rebuild the docker image and try singularity again on setonix. I'll keep you posted.
cheers,
J

from underworld2.