Crash in latest master with full hydro run over MPI about velociraptor-stf HOT 9 CLOSED

I can confirm that the current master works fine with and without MPI if I comment out this section:

# Compute the total luminosity in the 9 GAMA bands                                                                                                                                                                                                                                                                                                                        
Star_internal_property_names=Luminosities,Luminosities,Luminosities,Luminosities,Luminosities,Luminosities,Luminosities,Luminosities,Luminosities,                                                                                                                                                                                                                       
Star_internal_property_index_in_file=0,1,2,3,4,5,6,7,8,                                                                                                                                                                                                                                                                                                                  
Star_internal_property_input_output_unit_conversion_factors=1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,                                                                                                                                                                                                                                                                         
Star_internal_property_calculation_type=aperture_total,aperture_total,aperture_total,aperture_total,aperture_total,aperture_total,aperture_total,aperture_total,aperture_total,                                                                                                                                                                                          
Star_internal_property_output_units=unitless,unitless,unitless,unitless,unitless,unitless,unitless,unitless,unitless,       

from velociraptor-stf.

I've reproduced this problem with the latest master. Different runs give different problems though (double-free, invalid pointer, corrupted double-linked list, etc) which points to some type of memory corruption.

After a small debugging session I think I spotted what the problem could be. In MPISendReceiveFOFStarInfoBetweenThreads the receive buffer proprecvbuff is resized to numrecv, but then it receives numrecv * numextrafields values:

Similar code is correctly coded in MPISendReceiveFOFHydroInfoBetweenThreads though:

Then it's wrong again in MPISendReceiveFOFBHInfoBetweenThreads and MPISendReceiveFOFExtraDMInfoBetweenThreads:

From the git history it seems like this problem has always been present since these routines were introduced (April 2020). The routine that behaves correctly does so because I fixed it on an earlier commit 082ff68 for a similar problem reported in #54. Back then I didn't realise this affected more than one routine, and now that it became clear I'll obviously go and fix them all. I'll try to unify a bit the code as well, but without being too disruptive.

from velociraptor-stf.

That all makes sense. Thanks.

Indeed, not related to the latest master. I tried to go back in time to find a version that works yesterday but couldn't. That got me really confused so I took a break.

But, one thing that is different, and that your test reveals, is that we are now making use of the extra star properties in the config file. That is something we have not used much.
I added these config options recently in our EAGLE setup and tested it without MPI. So the problem went unnoticed.

Josh then tried that same setup but in MPI-only mode (as he still gets hit by the negative density bug when using OMP) and that when it all went wrong.

Hopefully the solution you had worked out for the hydro case can be relatively easily transplanted here.

from velociraptor-stf.

@MatthieuSchaller I pushed the relevant changes to the issue-87 branch. I tested them against the dataset/config you provided and now the code gets past the original problem, so it seems like the issue is gone.

However, at dataset writing time there is a new crash due to some invalid size passed to the HDF5 routines. I'll look into that separately on a different issue, but it appears to be specific to parallel HDF5 writing, so deactivating that option might be a workaround. Given this new, additional problem we can't really confirm the issue in this ticket is really gone until we have a successful execution, so I'll refrain from merging for the time being until we have a fix for the second issue.

from velociraptor-stf.

Great. Let me know whether there is anything I can help with or test. I suppose there isn't much point before the i/o-time crash is solved but nevertheless can try if needed.

from velociraptor-stf.

@MatthieuSchaller a test that you can try, if the resulting files are useful, is to run with the latest issue-87 branch but without parallel HDF5 writing, thus hopefully avoiding #88.

from velociraptor-stf.

Yes, it all works smoothly.

from velociraptor-stf.

The version with parallel-hdf5 hangs while writing.

from velociraptor-stf.

Thanks @MatthieuSchaller for confirming the fix works. Since both you and I have seen the fix working separately I've merged it now the master branch, so I'm closing this issue. I'll try to focus now on #88.

from velociraptor-stf.