Comments (8)
rtobar
commented on July 23, 2024
Yes, this should be possible I think. It would also alter probably the same portion of code touched by #27, I'll do this first though.
from velociraptor-stf.
MatthieuSchaller
commented on July 23, 2024
I should have given you some more context, sorry. It looks like compressing the .properties file saves a factor of 3 in disk space. We would like to see whether that does not come at a huge penalty in writing time. For instance, it could be that writing the compressed id catalogue file is very slow for little benefit but the .properties is fast and compresses well.
from velociraptor-stf.
rtobar
commented on July 23, 2024
Thanks @MatthieuSchaller for the extra context, it definitely helps in deciding how far to go with this. I'm doing something quite simple to begin with, and it sounds like it should be enough to do the comparisons you are describing.
from velociraptor-stf.
rtobar
commented on July 23, 2024
How does this look as a first iteration? (output trimmed to final lines)
[...]
saving SO particle lists to lala.catalog_SOlist.0
Wrote lala.catalog_SOlist.0 in 3.104 [s]
Done SO masses for field objects in 20.098
Memory report, func = SortAccordingtoBindingEnergy--line--4719 task = 0 : Average = 1.746150 GB, Data = 1.640755 GB, Dirty = 0.000000 GB, Library = 0.000000 GB, Peak = 2.915272 GB, Resident = 1.612049 GB, Shared = 0.017147 GB, Size = 1.838261 GB, Text = 0.001534 GB,
Done
saving property data to lala.properties.0
Wrote lala.properties.0 in 51 [ms]
saving group catalog to lala.catalog_groups.0
saving particle catalog to lala.catalog_particles.0
Wrote catalogues in 2.816 [s]
saving hierarchy data to lala.catalog_groups.0
saving hierarchy data to lala.hierarchy.0
Wrote lala.hierarchy.0 in 3 [ms]
saving particle type info to lala.catalog_parttypes.0
Wrote particle type info in 202 [ms]
saving profiles lala.profiles.0
Wrote lala.profiles.0 in 10 [ms]
TIME::0 took 304.657 in all
0 : finished running VR
from velociraptor-stf.
MatthieuSchaller
commented on July 23, 2024
Yes, perfect. Exactly what I was hoping for.
from velociraptor-stf.
rtobar
commented on July 23, 2024
I've pushed these changes to the io-timing branch, could you give them a try? If you're happy with how they look I'll merge to master.
from velociraptor-stf.
MatthieuSchaller
commented on July 23, 2024
saving property data to halos_0036.compressed.properties.0
Wrote halos_0036.compressed.properties.0 in 3.354 [s]
saving group catalog to halos_0036.compressed.catalog_groups.0
saving particle catalog to halos_0036.compressed.catalog_particles.0
Wrote catalogues in 1.198 [s]
saving hierarchy data to halos_0036.compressed.catalog_groups.0
saving hierarchy data to halos_0036.compressed.hierarchy.0
Wrote halos_0036.compressed.hierarchy.0 in 10 [ms]
saving particle type info to halos_0036.compressed.catalog_parttypes.0
Wrote particle type info in 762 [ms]
saving profiles halos_0036.compressed.profiles.0
Wrote halos_0036.compressed.profiles.0 in 307 [ms]
TIME::0 took 1391.607 in all
All looks good to me, thanks.
from velociraptor-stf.
rtobar
commented on July 23, 2024
Merged to master now.
from velociraptor-stf.
Related Issues (20)
- Incorrectly sized buffer given for MPI_Bcast reception HOT 1
- Writing parallel properties file in hydro builds is broken HOT 6
- SO list offsets are wrong/counterintuitive HOT 6
- Inconsistent array names between properties files HOT 6
- SO list output too large and possibly wrong HOT 11
- Error in writing HDF5 outputs HOT 5
- Improve VR's memory usage for extra data in Particles
- DMO Zoom on-the-fly with SWIFT segfault HOT 4
- OpenMP bug in temperature calculations. HOT 5
- Memory usage blowing up in large DMO runs HOT 22
- Differences in halo masses when switching on/off substructure search HOT 2
- Mistakes in metallity calculations HOT 1
- Apparently wrong output when using Star_internal_property options
- HIGHRES needs undocumented Extensive_interloper_properties_output config option
- Error on compiling HOT 9
- Particle_type_for_reference_frames value not listed in the output HOT 3
- More potential issues hidden in MPISendReceive*InfoBetweenThreads functions
- Uninitialised variables in PropData class
- Compiling with DVR_USE_GAS=ON but no other options doesn't work HOT 5
- Buffer overflow in PotentialTree with OpenMP HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from velociraptor-stf.