VR does not run in MPI+hydro mode about velociraptor-stf HOT 6 CLOSED

@MatthieuSchaller thanks for the report. I don't think I'll be able to look into this until early next week though. In the meanwhile, could you please share the data, or push for my cosma application to proceed? I tried to reproduce locally with some of the inputs I had received in the past but they seem to lack the fields needed by the hydro-enabled code.

from velociraptor-stf.

If it helps, there is smaller test case here:

• /snap7/scratch/dp004/jlvc76/SWIFT/EoS_tests/swiftsim/examples/EAGLE_low_z/EAGLE_6/eagle_0000.hdf5
• /snap7/scratch/dp004/jlvc76/SWIFT/EoS_tests/swiftsim/examples/EAGLE_low_z/EAGLE_6/vrconfig_3dfof_subhalos_SO_hydro.cfg

This one crashes about 20s after start so might be easier.

Config is: cmake ../ -DVR_USE_HYDRO=ON -DCMAKE_BUILD_TYPE=Debug
Run command line is: stf -C vrconfig_3dfof_subhalos_SO_hydro.cfg -i eagle_0000 -o halos_0000 -I 2

• Problem happens with gcc or ICC,
• Running with -DVR_OPENMP=OFF also crashes in the same way,
• Running without VR_MPI_REDUCE crashes in a different way. There the crash happens when reading in stuff.

from velociraptor-stf.

Thanks @MatthieuSchaller for the extra information. I managed to reproduce this locally with the data you pointed above, I hope I can post an update soon with some information

from velociraptor-stf.

Running under valgrind revealed an invalid write:

==14074==    by 0x2D2CC6: MPISendReceiveFOFHydroInfoBetweenThreads(Options&, fofid_in*, std::vector<long long, std::allocator<long long> >&, std::vector<float, std::allocator<float> >&, int, int, ompi_communicator_t*&) (mpiroutines.cxx:2697)

The in gdb the reason became clear: the proprecvbuff vector needed to be properly sized before receiving data. Its size should be the same that propsendbuff, which in turn should be the size of indicessend times numextrafields.

On a new issue-54 branch I fixed the resizing of proprecvbuff to accommodate for all incoming data, and also added an assertion to make sure the sizes of the input vectors are related to each other as expected. After these changes I can run the code to completion. @MatthieuSchaller please confirm that this is working for you too.

I also tried out running without VR_MPI_REDUCE and also got a crash. That looks like a separate issue though, so I'll create a separate ticket to keep track of it separately.

from velociraptor-stf.

I confirm this works both with and without OpenMP (as expected since unrelated). Thanks for tracking this down!

(I am never brave enough to fire up valgrind inside an mpirun call...)

from velociraptor-stf.

Great! I merged this into the master branch now, thanks for testing!

from velociraptor-stf.