Git Product home page Git Product logo

Comments (4)

moustakas avatar moustakas commented on June 27, 2024

@Chanuntorn the following procedure worked for me just now, at least at NERSC:

  1. Log into NERSC. Following the instructions here, create a dedicated (local) conda environment and install Revolver's dependencies (plus a couple more handy libraries like matplotlib):
module load python
conda create -n desivoids python=3.8
conda activate desivoids
conda install -y numpy scipy astropy cython ipython matplotlib
pip install healpy pyfftw

  1. A few notes regarding the preceding step:

    • I called my environment desivoids but you can call it whatever you want.
    • pyfftw doesn't work with python v3.9, so you have to use v3.8 at least for now.
    • After you've created your conda environment (or if you have one from previous projects or work) then you just need to type conda activate desivoids into your terminal without doing module load python. (In detail, conda modifies your .bashrc to run conda init every time you log in, which is not great practice, but here we are.)

  2. Next, create a working directory (obviously choose your own path) and then clone the Revolver code base into that directory. Now, before running make I had to hand-edit line 14 of the Makefile to change make -C src all to make -C src all_nompi, as instructed in the README if MPI wasn't available. I did try module load openmpi as instructed here first, but Revolver couldn't find the appropriate library and header files. It would be great to try to figure this out in future. In any case, do:

mkdir /global/cscratch1/sd/ioannis/voids
cd /global/cscratch1/sd/ioannis/voids
git clone https://github.com/seshnadathur/Revolver.git
cd Revolver
[edit Makefile]
make
  1. Next, to reproduce the SDSS/CMASS results, download the large-scale structure galaxy and random catalogs:
cd /global/cscratch1/sd/ioannis/voids
wget https://data.sdss.org/sas/dr12/boss/lss/galaxy_DR12v5_CMASS_South.fits.gz
wget https://data.sdss.org/sas/dr12/boss/lss/random0_DR12v5_CMASS_South.fits.gz
gunzip *.fits.gz
  1. Next, you need to modify the /global/cscratch1/sd/ioannis/voids/Revolver/parameters/params.py file to point to these catalogs. Use a text editor to update the paths and might as well use 32 threads. Note that the trailing slash on the output_folder directory name is crucial! Here's what my params.py file looks like
handle = 'DR12_CMASS_South'
output_folder = '/global/cscratch1/sd/ioannis/voids/out/'
tracer_file = '/global/cscratch1/sd/ioannis/voids/galaxy_DR12v5_CMASS_South.fits'
random_file = '/global/cscratch1/sd/ioannis/voids/random0_DR12v5_CMASS_South.fits'
mask_file = 'masks/DR12v5_CMASS_South_mask.fits'
void_prefix = 'recon-voids'
verbose = True
do_recon = True
use_barycentres = False
run_voxelvoids = True
do_tessellation = True
nthreads = 32
  1. Finally you can run Revolver, which for me took about 4 minutes on a login node (with 32 threads)!
cd /global/cscratch1/sd/ioannis/voids/Revolver
time python revolver.py --par parameters/params.py
Loading parameters from parameters/params.py

 ==== Running reconstruction for real-space positions ====
Loading galaxy data from file...
Loading randoms data from file...

 ==== Starting the reconstruction ====
Using values of growth rate f = 0.780 and bias b = 2.000
Smoothing scale [Mpc/h]: 10.0
Number of bins: 512
Box size [Mpc/h]: 2904.55
Bin size [Mpc/h]: 5.67
Loop 0
Reading wisdom from  wisdom.512.32
Creating FFTW objects...
Allocating randoms in cells...
Smoothing...
Allocating galaxies in cells...
Smoothing galaxy density field ...
Computing density fluctuations, delta...
Fourier transforming delta field...
Inverse Fourier transforming to get psi...
Calculating shifts...
Loop 1
Allocating galaxies in cells...
Smoothing galaxy density field ...
Computing density fluctuations, delta...
Fourier transforming delta field...
Inverse Fourier transforming to get psi...
Calculating shifts...
Loop 2
Allocating galaxies in cells...
Smoothing galaxy density field ...
Computing density fluctuations, delta...
Fourier transforming delta field...
Inverse Fourier transforming to get psi...
Calculating shifts...
 ==== Done reconstruction ====

Reconstruction took 44.655 seconds
Loading galaxy data from file...

 ==== Starting the void-finding with voxel-based method ====
217780 tracers found
Box size [Mpc/h]: 2544.546
Initial bin size [Mpc/h]: 12.79, nbins = 199
Final bin size [Mpc/h]: 5.68, nbins = 448
Smoothing scale [Mpc/h]: 18.29
Allocating galaxies in cells...
Allocating randoms in cells...
Smoothing density fields ...
Finding density minima
Total number of voxels: 89915392
Reading in the dens data from file ...
Setting the voxel adjacencies ...
Finding jumper for each voxel
About to jump ...
Post-jump ...
10979 zones found...
Finding zone borders
Allocating zone adjacencies and links
Finding weakest links
Found zone adjacencies
delta ranges from -9.945806e-01 to 4.009290e+00.
Writing the zone memberships
Post-processing voids
Total 2613 voids pass all cuts
 ==== Finished with voxel-based method ====
Voxel voids took 39.823 seconds

 ==== Starting the void-finding with ZOBOV ====
Loading galaxy data from file...
Cutting galaxies outside the redshift limits provided
WARNING: AstropyDeprecationWarning: "verbose" was deprecated in version 1.15.0 and will be removed in a future version.  [python_tools.zobov]
Kept 208491 tracers after all cuts
Determining survey redshift selection function ...
Generating buffer mocks around survey edges ...
	buffer mocks will have 10.0 x the galaxy number density
WARNING: AstropyDeprecationWarning: "verbose" was deprecated in version 1.15.0 and will be removed in a future version.  [python_tools.zobov]
	placed 135214 buffer mocks at high-redshift cap
	placed 55543 buffer mocks at low-redshift cap
WARNING: AstropyDeprecationWarning: "verbose" was deprecated in version 1.15.0 and will be removed in a future version.  [python_tools.zobov]
	placed 354366 buffer mocks along the survey boundary edges
	Using box length 2774.22
	added 24292 guards to stabilize the tessellation
Buffer mocks written to file /global/cscratch1/sd/ioannis/voids/out/DR12_CMASS_South_mocks.npy
Calling vozisol to do the tessellation...
Tessellation done.
Post-processing voids ...
Identified 2138 potential voids. Now extracting circumcentres ...
WARNING: AstropyDeprecationWarning: "verbose" was deprecated in version 1.15.0 and will be removed in a future version.  [python_tools.zobov]
Removed 0 edge-contaminated voids
 ==== Finished with ZOBOV-based method ====
ZOBOV took 132.518 seconds

real	4m8.975s
user	6m8.724s
sys	0m58.498s

from revolver.

moustakas avatar moustakas commented on June 27, 2024

For the record I did try to do all of this in a Docker/shifter container (including MPI), but the way the Revolver package is set up made it challenging to add the Python code to PYTHONPATH and to also use a parameter file in a non-standard location. But that's a different ticket!

from revolver.

seshnadathur avatar seshnadathur commented on June 27, 2024

Thanks for looking into this and suggesting the workaround – I have not had time to test Revolver out on NERSC yet. A few questions/comments:

  1. What is the error you get when trying to compile the MPI version? When running this on a (non-NERSC) HPI system I have only needed to do the equivalent of your module load openmpi and things have worked fine.
  2. If you are not compiling with MPI then the only part of the code that is not running single-thread is the reconstruction prior to void-finding. As you can see from the timings above, reconstruction is not the slowest step. So lots of threads then make for rather an inefficient use of resources, and in the interest of budgeting your NERSC allocation it would be better to use fewer.
  3. Having said that, some parts of the code (especially reconstruction and the voxel void-finding) can require a lot of memory because of needing an N^3 grid. On some HPC systems it might be necessary to increase the number of threads to get sufficient memory. This is an efficiency issue for Revolver that needs work.
  4. I don't understand the issue with using a parameter file in a non-standard location, could you elaborate (maybe on another ticket?) This should be a quick and easy fix.

from revolver.

seshnadathur avatar seshnadathur commented on June 27, 2024

I should also mention that as soon as I have the time, I am planning a major rewrite of all the Revolver code, including transitioning to using pyrecon to provide the reconstruction, and using YAML input in place of the current system. This will change some of the workflow, but hopefully in a positive way, and making it more consistent with other DESI code.

from revolver.

Related Issues (9)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.