Comments (17)
EvgeniiChaikin
commented on August 26, 2024
1
I re-created the same test as in the description.
I ran it for two versions of the VR
8f380fcd4b53c4b1063eaa294e270e30610daf4b(May 26)64de17bff6925f47f3ebe8f8195108801d661d95(Mar 24)
In both cases, VR was compiled with MPI.
As before, the older version would produce the metallicities of star-formation gas equal to zero. However, that is not the case for the newer version (May 26). To confirm this result, I used 5 different snapshots for this test and never found the star-forming gas to have zero metallicities in the catalogues produced by the newer version of VR.
I thus conclude that the bug has been fixed and close this thread.
Thanks everyone for the help!
from velociraptor-stf.
EvgeniiChaikin
commented on August 26, 2024
Update: the masses of star-forming gas are also zero if VR is run with config 2.
from velociraptor-stf.
rtobar
commented on August 26, 2024
This sounds like a similar problem to that from #72, but @EvgeniiChaikin clearly tested using the latest fixes introduced to fix #72. That plus the fact that @MatthieuSchaller was mentioned in the original description point out that this is a different problem, although probably closely related.
In #72 (comment) there is a reference to some values not being zero though, so again, I suppose this is slightly different.
from velociraptor-stf.
rtobar
commented on August 26, 2024
I tried to reproduce this in comsa6 but it actually worked for me. My build has both MPI and OpenMP and HYDRO all ON. What were your compilation flags?
$> d=/cosma7/data/dp004/dc-chai1/test_VR
$> # Running against the two configurations
$> builds/71/stf -C $d/vrconfig_3dfof_subhalos_SO_hydro_1.cfg -i $d/colibre_0023 -I 2 -o halo_v1_0023 > /dev/null
$> builds/71/stf -C $d/vrconfig_3dfof_subhalos_SO_hydro_2.cfg -i $d/colibre_0023 -I 2 -o halo_v2_0023 > /dev/null
$> # First numbers from original output for Aperture_Zmet_gas_100_kpc
$> h5dump -d Aperture_Zmet_gas_100_kpc halo_v1_0023.properties.0 | head --lines 10
HDF5 "halo_v1_0023.properties.0" {
DATASET "Aperture_Zmet_gas_100_kpc" {
DATATYPE H5T_IEEE_F64LE
DATASPACE SIMPLE { ( 823 ) / ( 823 ) }
DATA {
(0): 0.00119531, 0.00411753, 0.000572314, 0.000104163, 0.000112529,
(5): 5.33842e-05, 0.000107015, 0.000100115, 1.05497e-05, 9.58872e-06,
(10): 8.96096e-06, 2.11294e-05, 1.57962e-05, 9.3072e-06, 4.0905e-06,
(15): 2.95208e-05, 1.55529e-05, 8.73288e-06, 3.35401e-06, 2.06113e-05,
(20): 6.30811e-06, 5.24397e-06, 0, 1.0879e-05, 5.93846e-06, 3.55144e-06,
$> # Full diff between first and second run for Aperture_Zmet_gas_100_kpc (i.e.: all numbers are equal)
$> diff -Naur <(h5dump -d Aperture_Zmet_gas_100_kpc halo_v1_0023.properties.0) <(h5dump -d Aperture_Zmet_gas_100_kpc halo_v2_0023.properties.0)
--- /dev/fd/63 2021-04-06 09:46:29.416308901 +0100
+++ /dev/fd/62 2021-04-06 09:46:29.417308911 +0100
@@ -1,4 +1,4 @@
-HDF5 "halo_v1_0023.properties.0" {
+HDF5 "halo_v2_0023.properties.0" {
DATASET "Aperture_Zmet_gas_100_kpc" {
DATATYPE H5T_IEEE_F64LE
DATASPACE SIMPLE { ( 823 ) / ( 823 ) }
$> # The actual numbers you requested
$> h5dump -d Aperture_IronOverHydrogenMasses_aperture_total_gas_10_kpc halo_v2_0023.properties.0 | head --lines 10
HDF5 "halo_v2_0023.properties.0" {
DATASET "Aperture_IronOverHydrogenMasses_aperture_total_gas_10_kpc" {
DATATYPE H5T_IEEE_F64LE
DATASPACE SIMPLE { ( 823 ) / ( 823 ) }
DATA {
(0): 1.24204e-05, 0.000109075, 1.9672e-05, 6.68841e-06, 2.95815e-06,
(5): 1.57079e-06, 3.03353e-06, 1.79867e-06, 5.43869e-07, 5.85223e-07,
(10): 8.79976e-07, 7.09117e-07, 9.37009e-07, 0, 3.45762e-07, 1.26667e-06,
(16): 3.49815e-08, 4.42882e-07, 1.32802e-08, 1.03888e-06, 2.37694e-07,
(21): 2.18679e-07, 0, 5.73632e-07, 1.18798e-07, 1.8089e-07, 2.28771e-08,
from velociraptor-stf.
rtobar
commented on August 26, 2024
I just tried again with -DVR_MPI=OFF and obtained the exact same results as in my previous comment, so I'm still unable to reproduce this problem.
And actually I just compared the two original output files pointed at by @EvgeniiChaikin and their Aperture_Zmet_gas_100_kpc also look identical:
$> d=/cosma7/data/dp004/dc-chai1/test_VR
$> diff -Naur <(h5dump -d Aperture_Zmet_gas_100_kpc $d/halo_v1_0023.properties.0) <(h5dump -d Aperture_Zmet_gas_100_kpc $d/halo_v2_0023.properties.0)
--- /dev/fd/63 2021-04-07 09:55:22.992640634 +0100
+++ /dev/fd/62 2021-04-07 09:55:22.993640643 +0100
@@ -1,4 +1,4 @@
-HDF5 "/cosma7/data/dp004/dc-chai1/test_VR/halo_v1_0023.properties.0" {
+HDF5 "/cosma7/data/dp004/dc-chai1/test_VR/halo_v2_0023.properties.0" {
DATASET "Aperture_Zmet_gas_100_kpc" {
DATATYPE H5T_IEEE_F64LE
DATASPACE SIMPLE { ( 823 ) / ( 823 ) }
@EvgeniiChaikin maybe there's a problem with the python tools you are using to read these files? Or maybe I'm missing something obvious from the original description of the problem?
from velociraptor-stf.
EvgeniiChaikin
commented on August 26, 2024
Hi @rtobar,
Thank you for helping me out!
I complied VR with
cmake -DVR_USE_GAS=ON -DVR_USE_STAR=ON -DVR_USE_BH=ON
I saved the output from the compilation process into
/cosma7/data/dp004/dc-chai1/VR_ICRAR_27_March/VELOCIraptor-STF/compile_output.txt
This bug is probably not related to the python tools I am using because I was also able to reproduce it using h5dump
[dc-chai1@login7b [cosma7] test_VR]$ h5dump -d Aperture_Zmet_gas_sf_100_kpc halo_v1_0023.properties.0 | head --lines 10
HDF5 "halo_v1_0023.properties.0" {
DATASET "Aperture_Zmet_gas_sf_100_kpc" {
DATATYPE H5T_IEEE_F64LE
DATASPACE SIMPLE { ( 823 ) / ( 823 ) }
DATA {
(0): 0.00428788, 0.012751, 0.00263531, 0.00478277, 0, 0.00130498,
(6): 0.000917185, 0, 0, 0, 0, 0, 0.00342874, 0, 0.000424479, 0.00703778,
(16): 0, 0, 0, 0.000731791, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.00384946,
(32): 0, 0, 0, 0, 0, 0, 0, 0.000778238, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
(51): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
[dc-chai1@login7b [cosma7] test_VR]$ h5dump -d Aperture_Zmet_gas_sf_100_kpc halo_v2_0023.properties.0 | head --lines 10
HDF5 "halo_v2_0023.properties.0" {
DATASET "Aperture_Zmet_gas_sf_100_kpc" {
DATATYPE H5T_IEEE_F64LE
DATASPACE SIMPLE { ( 823 ) / ( 823 ) }
DATA {
(0): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
(23): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
(46): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
(69): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
(92): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
I presume the reason why you were not able to reproduce the bug so far is that you have been using apertures.zmet_gas_100_kpc whereas I take apertures.zmet_gas_sf_100_kpc. That is, the problem seems to only occur in the fields related to the star-forming gas.
from velociraptor-stf.
rtobar
commented on August 26, 2024
@EvgeniiChaikin ahhhhh what a silly mistake on my side! I lost the sf bit in the name of the dataset, I'll re-run things to double-check again and confirm.
from velociraptor-stf.
rtobar
commented on August 26, 2024
I was now able to reproduce this, sorry it took so long!
This is a problem specific to MPI builds it seems. I tried to reproduce the problem with an MPI enabled build, then with an MPI disabled one, and only the former produced the problem:
$> # build with and without MPI:
$> for mpi in ON OFF; do builddir=builds/75-mpi-$mpi; cmake -B $builddir -DVR_USE_GAS=ON -DVR_USE_STAR=ON -DVR_USE_BH=ON -DVR_MPI=$mpi; cmake --build $builddir -j 16; done
$> # Run both executables against the two config files
$> for mpi in ON OFF; do builddir=builds/75-mpi-$mpi; for config in 1 2; do $builddir/stf -C $d/vrconfig_3dfof_subhalos_SO_hydro_$config.cfg -i $d/colibre_0023 -I 2 -o mpi-$mpi-config-$config > /dev/null; done; done
$> Compare things now
$> compare_dataset() { diff -Naur <(h5dump -d $1 $2) <(h5dump -d $1 $3) | grep 'DATA {' -A 100; }
# MPI=ON has differences
$> compare_dataset Aperture_Zmet_gas_sf_100_kpc mpi-ON-config-{1,2}.properties.0 | head
DATA {
- (0): 0.00428788, 0.012751, 0.00263531, 0.00478277, 0, 0.00130498,
- (6): 0.000917185, 0, 0, 0, 0, 0, 0.00342874, 0, 0.000424479, 0.00703778,
- (16): 0, 0, 0, 0.000731791, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.00384946,
- (32): 0, 0, 0, 0, 0, 0, 0, 0.000778238, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
- (51): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
- (74): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
- (97): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
- (120): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
- (142): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
# MPI=OFF has no differences; and MPI ON/OFF are the same for config 1
$> compare_dataset Aperture_Zmet_gas_sf_100_kpc mpi-OFF-config-{1,2}.properties
$> compare_dataset Aperture_Zmet_gas_sf_100_kpc mpi-OFF-config-1.properties mpi-ON-config-1.properties.0
$>
This will hopefully help constraining the search for the underlying problem, plus it indicates there is a workaround for small runs that can happen on a single node.
from velociraptor-stf.
EvgeniiChaikin
commented on August 26, 2024
Thanks a lot @rtobar !
I mostly feed VR with small cosmological boxes so your fix is perfect for me!
from velociraptor-stf.
MatthieuSchaller
commented on August 26, 2024
Could the recent fix to the negative densities over MPI have magically fixed this?
from velociraptor-stf.
MatthieuSchaller
commented on August 26, 2024
@JBorrow is that not something we see with the EAGLE runs?
from velociraptor-stf.
JBorrow
commented on August 26, 2024
Master from 2021-04-13 works for us; this plot shows Z_{gas, sf} against M_*, using the MPI-only version of the code.
from velociraptor-stf.
MatthieuSchaller
commented on August 26, 2024
Is that version then different from the one @EvgeniiChaikin uses?
from velociraptor-stf.
MatthieuSchaller
commented on August 26, 2024
@EvgeniiChaikin (when you have time) the fix (#79) that is now in master may have helped here.
from velociraptor-stf.
rtobar
commented on August 26, 2024
I wanted to try this out with the original config/inputs reported in this ticket but they were gone. Hopefully the fix in #79 has some positive effect on this problem -- the fact that MPI builds work in some cases and not in others could be indicative that this was an uninitialised variable problem indeed.
from velociraptor-stf.
MatthieuSchaller
commented on August 26, 2024
Agreed, that seems plausible.
from velociraptor-stf.
rtobar
commented on August 26, 2024
Great news @EvgeniiChaikin, thanks for retrying this!
from velociraptor-stf.
Related Issues (20)
- Incorrectly sized buffer given for MPI_Bcast reception HOT 1
- Writing parallel properties file in hydro builds is broken HOT 6
- SO list offsets are wrong/counterintuitive HOT 6
- Inconsistent array names between properties files HOT 6
- SO list output too large and possibly wrong HOT 11
- Error in writing HDF5 outputs HOT 5
- Improve VR's memory usage for extra data in Particles
- DMO Zoom on-the-fly with SWIFT segfault HOT 4
- OpenMP bug in temperature calculations. HOT 5
- Memory usage blowing up in large DMO runs HOT 22
- Differences in halo masses when switching on/off substructure search HOT 2
- Mistakes in metallity calculations HOT 1
- Apparently wrong output when using Star_internal_property options
- HIGHRES needs undocumented Extensive_interloper_properties_output config option
- Crash in latest master with full hydro run over MPI HOT 9
- Can't write more than 2GB HDF5 datasets in parallel HOT 28
- More potential issues hidden in MPISendReceive*InfoBetweenThreads functions
- Uninitialised variables in PropData class
- Compiling with DVR_USE_GAS=ON but no other options doesn't work HOT 5
- Buffer overflow in PotentialTree with OpenMP HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from velociraptor-stf.