Git Product home page Git Product logo

photozdc1's Introduction

Photo-z Task Force Simulations Generator

Project aim

The photo-z Task Force simulations will be the starting point of simulating systematic effects in the LSST photometry that impact the quality of the photo-z. This includes (eventually) simulating galaxy properties realistically enough that we can mimic the kinds of spectroscopic training and calibration samples that will be obtainable.

We will use baseline properties of galaxies from mock galaxy catalogs produced by the DESC simulation working group, and these will contain LSST photometry with no systematics. Using these simulations we will be able to perturb the spectral energy distributions (SEDs) of galaxies according to a variety of different systematics, e.g. dust, IGM absorption, blended galaxies, etc --- to determine how LSST photometry is affected. These simulations should be considered a secondary "layer" on top of the cosmological galaxy simulations where we can generate many observed galaxy catalogs with differing properties very quickly. We will have the option to turn on and off these various sources of systematic perturbations, and can determine which are the most important, and characterize their impact.

The resulting galaxy catalogs will not only be used to test the quality of the photo-z due to a variety of systematic effects, but also to test the robustness of photo-z algorithms in producing robust probability of redshifts p(z).

Brief Project Summary

Using mock catalogs developed in the Simulations group, the Task Force will determine a way of populating the sims with galaxy SEDs (e.g. the Brown templates or a continuous parameterization of them), connecting physical parameters such as stellar mass to the SEDs. The goal is to construct more realistic obtainable training and calibration sets from these catalogs (e.g. via a stellar mass complete sample). As well as generating large photo-z catalogs, this will enable us to study training set incompleteness and its effect on the cosmological analysis.

Tasks

To be implemented!

  • define a continuous parametrization of realistic SEDs, see sedGenerator.py
  • construct the mapping from SEDs onto simulated photometry and physical galaxy parameters, see sedMapper.py
  • add prescriptions for incorporating the impact of emission lines, see emLineGenerator.py
  • add prescriptions for including the variation of the line of sight IGM absorption for high redshift galaxies, see igmModel.py

Deliverables

  • Catalogs of p(z)

Code installation

  1. python 2.X distribution with all the usual (numpy, scipy, matplotlib, etc) and the following packages: cython (optional), scikit-learn, pandas, astropy. Anaconda is a good choice.
  2. clone repository to local directory
  3. add src/ directory to PYTHONPATH environment variable
  4. try running some examples!

Code organisation/directory structure

  • src/ contains all the important code, hopefully docstrings make everything clear-ish
  • examples/ demonstrates some current calculations the code can do and demonstrates how to construct programs
  • sed_data/ contains the SED data, see README file for more information
  • filter_data/ contains filter transmission functions, see README file for more information
  • igm_data/ currently empty, will contain line of sight transmission data for the IGM
  • tests/ contains unit tests

Warnings

This code is currently under extreme development!

If using python < 2.7.6 and/or scipy < 0.14 you may get a lot of printing of: IntegrationWarning: The maximum number of subdivisions (50) has been achieved.' from scipy.integrate.quad`

The integration method is under development because scipy.quad is too slow.

photozdc1's People

Contributors

connolly avatar enourbakhsh avatar johannct avatar mgckind avatar sjs86 avatar sschmidt23 avatar

Stargazers

 avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

photozdc1's Issues

the spline interpolator should be forbidden to extrapolate

In my work, I have faced several times issues with extrapolations beyond the data boundaries, especially with splines. See http://docs.scipy.org/doc/scipy-0.16.0/reference/generated/scipy.interpolate.InterpolatedUnivariateSpline.html : I believe that using ext=2 (raise if beyond data bounds) is safest, but it might not work with the assumed cutoff behavior of filters, in which case I would use ext=1.
sedFilter.py:l.66 and l.183 uses the default, ext=0, allowing for extrapolation....

Integration speed

The integration of SEDs within filters needs to be sped up considerably. I made a sad attempt at doing this via cython calling gsl routines (see cGslInteg.pxd and cphotometry.pyx) but it's a bit of a mess and not much of a speed up.

Reading FITS into pandas dataframe

See comments in interface.py ReadCosmoSim._read_fits()

Reading in some FITS files via astropy.io.fits, e.g.

hdulist = fits.open(file_to_read)
self._data = pd.DataFrame(np.asarray(hdulist[nhdu].data), columns=col_names)

Resulted in the following error:

ValueError: Big-endian buffer not supported on little-endian compiler

To avoid it, read in FITS via astropy.table

table = Table.read(file_to_read)
data = table._data # will be deprecated but Table.as_array doesn't work????
# Fix byte order.
# See https://github.com/astropy/astropy/issues/1156
data = data.byteswap().newbyteorder()
self._data = pd.DataFrame.from_records(data)

But current versions of astropy have deprecated table._data and this method will be broken

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.