seshnadathur / revolver Goto Github PK

Real-space void locations from survey reconstruction

License: GNU General Public License v3.0

Makefile 0.35% Python 6.21% CMake 0.05% C++ 18.17% Batchfile 1.21% Shell 1.18% HTML 25.56% Roff 1.66% QMake 0.77% C 44.54% Cython 0.31%

revolver's People

Contributors

Stargazers

Watchers

Forkers

broukema alexwoodfinden dforero0896 pychristine hawinther marianajbr mrhossen38 chanuntorn gracospa epaillas

revolver's Issues

Can I look up what tracers belong to what cluster/void?

Thanks for creating Revolver, I am using it now in my cosmology PhD and with its help I will publish 2 papers, if things continue at the current pace. Right now I wonder if Revolver produces an out-put file that tells me what tracers belong to what cluster found by Revolver? Or analogously, what tracers are contained within a particular void or form the margin of the void? If this functionality is not present can you give tips on how to alter Revolver so that it produces this output? Thanks a lot!

AbacusSummit Cubic box catalogue out of bounds error right on boundary

```Traceback (most recent call last):
  File "/project/rrg-wperciva/tsfraser/Revolver/Revolver/revolver_modified.py", line 191, in <module>
    voidcat.run_voidfinder()
  File "/lustre06/project/6033532/tsfraser/Revolver/Revolver/python_tools/voxelvoids.py", line 164, in run_voidfinder
    fastmodules.allocate_gal_cic(rhog, self.cat.x, self.cat.y, self.cat.z, self.cat.weight, self.cat.size,
  File "python_tools/fastmodules.pyx", line 143, in fastmodules.allocate_gal_cic
IndexError: Out of bounds on buffer access (axis 2)```

Triggers when running AbacusSummit mocks, especially for the 1000 HOD mocks per cosmology. Axis in the error refers to the specific column in the coordinate array that's passed to allocate_gal_cic (i.e. in this case z has a single value at 2000.0 instead of 1999.9, which for a boxsize of 2000 is enough to trigger this). Will also flag this with Abacus, since their parameter space is 0<xyz<=2000 instead of 0<xyz<2000 since I'm assuming several codes are built around the latter assumption?

Error running revolver.py

An error occurs when running python revolver.py --par parameters/params.py .

==== Starting the void-finding with ZOBOV ==== 
Loading galaxy data from file...
Cutting galaxies outside the redshift limits provided
WARNING: AstropyDeprecationWarning: "verbose" was deprecated in version 1.15.0 and will be removed in a future version.  [python_tools.zobov]
Traceback (most recent call last):
  File "revolver.py", line 163, in <module>
    voidcat = ZobovVoids(parms)
  File "/global/u2/c/chanun/Revolver/python_tools/zobov.py", line 107, in __init__
    mask = hp.read_map(parms.mask_file, verbose=False)
  File "/global/homes/c/chanun/.conda/envs/revenv/lib/python3.8/site-packages/astropy/utils/decorators.py", line 547, in wrapper
    return function(*args, **kwargs)
  File "/global/homes/c/chanun/.conda/envs/revenv/lib/python3.8/site-packages/healpy/fitsfunc.py", line 379, in read_map
    nside = pixelfunc.npix2nside(pix.size)
  File "/global/homes/c/chanun/.conda/envs/revenv/lib/python3.8/site-packages/healpy/pixelfunc.py", line 1121, in npix2nside
    raise ValueError("Wrong pixel number (it is not 12*nside**2)")
ValueError: Wrong pixel number (it is not 12*nside**2)

Steps I completed after logging into jupyter.nersc

git clone https://github.com/seshnadathur/Revolver.git 
module load python
conda create --name revenv python=3.8
source activate revenv
conda install astropy scipy cython
pip install pyfftw healpy
conda install ipython

Then I edited parameters/params.py to match my file paths after downloading the SDSS files. In the Makefile, I changed the line make -C src all to make -C src all_nompi . I proceed with the following

export PYTHONPATH=/global/homes/c/chanun/Revolver:$PYTHONPATH
cd Revolver
make clean
make
python revolver.py --par parameters/params.py

modifying void_prefix in zobov.py

Is there a specific line where i can reliably modify "void_prefix" in zobov.py?

I'm currently working on a branch of revolver that allows you to alter the filenames with command line arguments, and I'm trying to add the "handle" to the void catalogue naming system. I've gotten it to work so far with voxelvoids.py (and can provide the source code for it), but I was wondering which line in zobov should be changed (if it even is a single line)? Since I've modified several parts but to no avail.

Mask has to fit tightly to the data, else code breaks

@seshnadathur, thx for making revolver public!
Maybe I am making some mistake, but I have been using revolver for months and I noticed that if I provide a mask that doesn't fit tightly to the spatial distribution of the sources in source_file the execution breaks. Example: if I provide as a source file galaxies on the full sky or within a quadrant, but I provide a mask that covers only an octant then I will get the print-out
Galaxies exist where mask=0. Maybe check the input?
and later the execution breaks with message
Error message: “ File "ckdtree.pyx", line 592, in scipy.spatial.ckdtree.cKDTree.__init__ ValueError: Negative input data are outside of the periodic box.”
See also the attached image of the error message

This is annoying, because frequently I would like to compare the void distribution in different patches of my source population just using masks. Instead, I have also to alter my source file, subsample it and save it in new files, too.

So how about a fix that ignores the data outside the mask and lets the code complete? Thx

all files moved by zobov if handle is empty

SUMMARY: An empty value of handle in parameters/params.py causes
zobov to shift all files from the Revolver/ main directory into a
subsubdirectory and then crash.

ENVIRONMENT:

python2.7
Debian GNU/Linux
Revolver commit 792322e Tue May 21 12:13:49 2019 +0100

REPRODUCE:

To reproduce the bug: compile Revolver, provide an ascii simulation
file simfile (an N=32^3, XYZ T^3 simulation file of unit fundamental domain size),
create a directory 'outputs/', and modify
parameters.py as per the following patch, in order to analyse an example
with zobov only:

diff --git a/parameters/params.py b/parameters/params.py
index 88fc4fd..b9ef372 100644
--- a/parameters/params.py
+++ b/parameters/params.py
@@ -3,14 +3,14 @@
 # distance units in this code are calculated in Mpc/h by default

 # ======= runtime options ======== #
-verbose = False  # True for more informative output statements
+verbose = True  # True for more informative output statements
 debug = False    # True for output checks during reconstruction
 nthreads = 1    # set to the number of CPUs you have
 # ================================ #

 # ========= file handling options ========= #
 handle = ''         # string; used to identify the sample and set filenames
-output_folder = ''  # /path/to/folder/ where output should be placed
+output_folder = 'outputs/'  # /path/to/folder/ where output should be placed
 # ========================================= #

 # ========== cosmology ============ #
@@ -18,7 +18,7 @@ omega_m = 0.308  # used for reconstruction and to convert redshifts to distances
 # ================================= #

 # ======= reconstruction options ========== #
-do_recon = True     # if False, no reconstruction is performed and other recon options are ignored
+do_recon = False     # if False, no reconstruction is performed and other recon options are ignored
 nbins = 256     # the number of grid cells per side of the box
 padding = 200.  # for survey data, the extra 'padding' for the cubic box, in Mpc/h
 smooth = 10.    # smoothing scale in Mpc/h
@@ -28,10 +28,10 @@ niter = 3       # number of iterations in the FFT reconstruction method, 3 is su
 # ========================================= #

 # ======= input tracer data options =========== #
-tracer_file = ''    # /path/to/file with input data
-is_box = False      # True if tracers cover a cubic simulation box with periodic boundaries; False for survey data
-box_length = 1500.  # if is_box, the box side length in Mpc/h; else ignored
-boss_like = True    # True if the input data file is in FITS format with same data fields as BOSS data
+tracer_file = 'simfile'    # /path/to/file with input data
+is_box = True      # True if tracers cover a cubic simulation box with periodic boundaries; False for survey data
+box_length = 1.  # if is_box, the box side length in Mpc/h; else ignored
+boss_like = False    # True if the input data file is in FITS format with same data fields as BOSS data
 special_patchy = False  # set True if input array is in the special PATCHY format provided by Hector
 # if not boss_like or special_patchy, data file must contain array data in ASCII or NPY format
 posn_cols = [0, 1, 2]  # columns of tracer input array containing 3D position information
@@ -65,7 +65,7 @@ randoms_file = ''   # /path/to/file containing randoms data: must be formatted s
 # =========================== #

 # ========== void-finding choices ============= #
-run_voxelvoids = True  # watershed void-finding based on particle-mesh density field interpolation in voxels
+run_voxelvoids = False  # watershed void-finding based on particle-mesh density field interpolation in voxels
 run_zobov = True   # watershed void-finding (using ZOBOV) based on Voronoi tessellation
 # note that these two options are not mutually exclusive - 2 sets of voids can be produced if desired

EXPECTED OUTPUT:

Using the command

python revolver.py --par=parameters/params.py

you should get

Loading parameters from parameters/params.py

 ==== Starting the void-finding with ZOBOV ====
Loading tracer positions from file simfile
32768 tracers found
MPI run: calling voz1b1 and voztie to do the tessellation...
Tessellation done.
mv: cannot move './outputs' to a subdirectory of itself, 'outputs/rawZOBOV/./outputs'
Post-processing voids ...
Traceback (most recent call last):
  File "revolver.py", line 202, in <module>
  File "/scratch/Revolver/python_tools/zobov.py", line 859, in postprocess_voids
  File "/scratch/.local/lib/python2.7/site-packages/numpy/lib/npyio.py", line 962, in loadtxt
    fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
  File "/scratch/.local/lib/python2.7/site-packages/numpy/lib/_datasource.py", line 266, in open
    return ds.open(path, mode, encoding=encoding, newline=newline)
  File "/scratch/.local/lib/python2.7/site-packages/numpy/lib/_datasource.py", line 624, in open
    raise IOError("%s not found." % path)
IOError: outputs/rawZOBOV/.txt not found.

ANALYSIS:

The error can be understood by checking what's in the working directory,
in outputs/, and in outputs/rawZOBOV:

boud:/scratch/Revolver$ ls -l
total 4
drwxr-xr-x 4 boud boud 4096 Aug  8 17:59 outputs/
boud:/scratch/Revolver$ ls -l outputs/
total 788
drwxr-xr-x 2 boud boud   4096 Aug  8 17:58 log/
-rw-r--r-- 1 boud boud 786436 Aug  8 17:58 _pos.dat
drwxr-xr-x 9 boud boud   4096 Aug  8 17:59 rawZOBOV/
-rw-r--r-- 1 boud boud    129 Aug  8 17:58 sample_info.txt
boud:/scratch/Revolver$ ls -l outputs/rawZOBOV/
total 108
drwxr-xr-x 2 boud boud   4096 Aug  8 17:50 bin/
drwxr-xr-x 3 boud boud   4096 Aug  8 17:50 build/
-rwxr-xr-x 1 boud boud     27 Aug  8 17:50 __init__.py*
-rw-r--r-- 1 boud boud  35149 Aug  8 17:49 LICENSE
-rwxr-xr-x 1 boud boud    456 Aug  8 17:50 Makefile*
drwxr-xr-x 2 boud boud   4096 Aug  8 17:50 masks/
drwxr-xr-x 2 boud boud   4096 Aug  8 17:58 parameters/
drwxr-xr-x 2 boud boud   4096 Aug  8 17:51 python_tools/
drwxr-xr-x 7 boud boud   4096 Aug  8 17:50 qhull/
-rw-r--r-- 1 boud boud   3599 Aug  8 17:50 README.md
-rwxr-xr-x 1 boud boud  10894 Aug  8 17:50 revolver.py*
lrwxrwxrwx 1 boud boud 884590 Aug  8 17:53 simfile
drwxr-xr-x 2 boud boud   4096 Aug  8 17:50 src/
-rw-r--r-- 1 boud boud   4969 Aug  8 17:55 wisdom.256.1

The problem seems to be related to handle being empty.

SUGGESTED FIX:

One solution is to put a default handle if handle is left
as an empty string; another could be to warn the user and give
a fatal error.

bug: accessing the array orig at invalid indices

SUMMARY: In src/voz1b1.c the array orig can be accessed at
an invalid index, yielding what to the user appears to be a programming
error but in reality is an unwise choice of analysis parameters.

ENVIRONMENT:

python2.7
Debian GNU/Linux
This bug is present in Revolver commit 792322e

REPRODUCE the bug:

Obtain Revolver commit 0af80af in branch debug_orig_array and compile:

git clone https://github.com/broukema/Revolver.git
cd Revolver
git checkout 0af80af
make

In your Revolver distribution directory, obtain this N=32^3 T^3 EdS
late-time XYZ ascii snapshot and call it inputfile.ascii:

wget -O inputfile.ascii.bz2 https://cosmo.torun.pl/~boud/example.EdS.sim.ascii.no.header.norm1.bz2
bunzip2 inputfile.ascii.bz2

Run revolver.

python revolver.py --par=parameters/params.py

EXPECTED OUTPUT:

Loading parameters from parameters/params.py

 ==== Starting the void-finding with ZOBOV ====
Loading tracer positions from file inputfile.ascii
32768 tracers found
MPI run: calling voz1b1 and voztie to do the tessellation...
Tessellation done.
Post-processing voids ...
Identified 268 potential voids. Now extracting circumcentres ...
Traceback (most recent call last):
  File "revolver.py", line 202, in <module>
    voidcat.postprocess_voids()
  File "/scratch/tmp/Revolver/python_tools/zobov.py", line 1009, in postprocess_voids
    edge_flag = self.find_void_circumcentres(num_acc, wtd_avg_dens, edge_flag)
  File "/scratch/tmp/Revolver/python_tools/zobov.py", line 1078, in find_void_circumcentres
    partadjs[index].append(i)
IndexError: list index out of range

The full set of output files (despite the error) is:

outputs/:
total 808
-rw-r--r-- 1 boud boud   4162 Aug  9 16:39 Voids.void
-rw-r--r-- 1 boud boud  12257 Aug  9 16:39 Voids_list.txt
drwxr-xr-x 2 boud boud   4096 Aug  9 16:36 log
drwxr-xr-x 2 boud boud   4096 Aug  9 16:39 rawZOBOV
-rw-r--r-- 1 boud boud    133 Aug  9 16:36 sample_info.txt
-rw-r--r-- 1 boud boud 786436 Aug  9 16:36 test_pos.dat

outputs/log:
total 3788
-rw-r--r-- 1 boud boud 3871578 Aug  9 16:39 test-mpirun.out

outputs/rawZOBOV:
total 1888
-rw-r--r-- 1 boud boud 1227144 Aug  9 16:39 test.adj
-rw-r--r-- 1 boud boud  262148 Aug  9 16:39 test.trvol
-rw-r--r-- 1 boud boud   20868 Aug  9 16:39 test.txt
-rw-r--r-- 1 boud boud    8870 Aug  9 16:39 test.void
-rw-r--r-- 1 boud boud  262148 Aug  9 16:39 test.vol
-rw-r--r-- 1 boud boud  118409 Aug  9 16:39 test.zone

ANALYSIS:

The error appears to come from line 348 of voz1b1.c in commit 5954583 on Sep 4, 2018
https://github.com/seshnadathur/Revolver/blob/master/src/voz1b1.c
where orig is assumed to have already been assigned an "original" value
for adjacent particle index adjs[i].adj[j]. But if the analysis parameters
such as zobov_box_div and zobov_buffer are not sufficient for this particular
dataset then it's possible for orig[adjs[i].adj[j]] to have not been evaluated
at all.

COMMENT:

Adjusting the analysis parameters by setting them to

zobov_box_div = 2   # tessellation will be divided into (zobov_box_div)^3 chunks run in parallel
zobov_buffer = 0.1  # fraction of box length overlap between sub-boxes

is sufficient to avoid the error and get reasonable results, including a
reasonable looking outputs/barycentres/Voids_baryC_cat.txt . But evaluating
an array at an unallocated place in memory is still a bug - the user should be
stopped (and preferably warned) before that happens.

SUGGESTED FIX:

One solution is to record which orig[.] members are evaluated, and throw a fatal
error, giving an appropriate warning, if access to an unevaluated member (line 348
of voz1b1.c as stated above) is attempted.

seshnadathur / revolver Goto Github PK

revolver's People

Contributors

Stargazers

Watchers

Forkers

revolver's Issues

Can I look up what tracers belong to what cluster/void?

AbacusSummit Cubic box catalogue out of bounds error right on boundary

Error running revolver.py

modifying void_prefix in zobov.py

Mask has to fit tightly to the data, else code breaks

all files moved by zobov if handle is empty

bug: accessing the array orig at invalid indices

Zobov void-finding broken for simulation data running in serial

Issue with 0 shifts running recon code

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent