mwvgroup / wiyn_lsst Goto Github PK

Processing of SweetSpot WIYN+WHIRC images using LSST DM Stack.

License: Other

Shell 22.60% Python 66.44% Jupyter Notebook 10.95%

wiyn_lsst's Issues

The filename mapper is not figuring out what to call the file in subtraction routine

The subtraction routine is not producing the final diffexp.fits file. It is creating a tmp/test_dr2/diff directory and adding subdirectories for supernovae that we calibrated, but there are never the final images. Here is a snippet of error from SN2011iu. The stacks are all good for this one and it is calibrated.

It makes the temporary fits files that it deposits in /tmp/test_dr2/, but never deletes them or creates the /tmp/test_dr2/diff/$supernova$/diffexp.fits files from them.

From MWV: " There's something wrong with filename mapper. It's not figuring out what to call the file. Will have to think about this."

Error from Cori:

[cori03 wiyn_lsst] python sub_wiyn_dr2.py $SCRATCH/tmp/test_dr2              
Processing SN2011iu                                                          
Running ImageDifferenceTask.parseAndrun with args:                           
['/global/cscratch1/sd/kap146/tmp/test_dr2', '--id', 'fileroot=SN2011iu_A_H_20120108', '--templateId', 'fileroot=SN2011iu_A_H_20120925', '--output', '/global/cscratch1/sd/kap146/tmp/test_dr2', '--configfile', 'diffimconfig.py', '--clobber-config', '--clobber-versions']                                                 
root INFO: Running: lsst.pipe.base.cmdLineTask.ImageDifferenceTask.parseAndRun('/global/cscratch1/sd/kap146/tmp/test_dr2', '--id', 'fileroot=SN2011iu_A_H_20120108', '--templateId', 'fileroot=SN2011iu_A_H_20120925', '--output', '/global/cscratch1/sd/kap146/tmp/test_dr2', '--configfile', 'diffimconfig.py', '--clobber-config', '--clobber-versions')                                                                             
fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.                       
Use '--' to separate paths from revisions, like this:                                                     
'git <command> [<revision>...] -- [<file>...]'                                                            
fatal: Not a git repository: '/global/homes/w/wmwv/local/lsst/obs_file/.git'                              
imageDifference INFO: Processing DataId(initialdata={'fileroot': 'SN2011iu_A_H_20120108'}, tag=set([]))   
imageDifference.getTemplate INFO: Fetching calexp (DataId(initialdata={'fileroot': 'SN2011iu_A_H_20120925'}, tag=set([]))) as template.                                                                            
imageDifference INFO: Source selection via src product                                                    
/global/common/cori/contrib/lsst/lsstDM/dc1/Linux64/meas_algorithms/12.1-17-g13cfda1+4/python/lsst/meas/algorithms/objectSizeStarSelector.py:399: RuntimeWarning: invalid value encountered in less                 
  bad = numpy.logical_or(bad, width < self.config.widthMin)                                               
/global/common/cori/contrib/lsst/lsstDM/dc1/Linux64/meas_algorithms/12.1-17-g13cfda1+4/python/lsst/meas/algorithms/objectSizeStarSelector.py:400: RuntimeWarning: invalid value encountered in greater              
  bad = numpy.logical_or(bad, width > self.config.widthMax)                                               
imageDifference INFO: Selected 32 / 534 sources for Psf matching (8 for control sample)                   
imageDifference INFO: Subtracting images                                                                  
imageDifference.subtract INFO: Template Wcs : 6.245210,0.816067 -> 6.243156,0.814774                      
imageDifference.subtract INFO: Science Wcs : 6.245101,0.816153 -> 6.243204,0.814880                       
imageDifference.subtract INFO: Astrometrically registering template to science image                      
imageDifference.subtract INFO: templateFwhmPix: 4.73901486328                                             
imageDifference.subtract INFO: scienceFwhmPix: 11.8430608219                                              
imageDifference.subtract INFO: Growing 32 kernel candidate stars by 35 pixels                             
imageDifference.subtract INFO: Selected 32 / 32 sources for KernelCandidacy                               
imageDifference.subtract INFO: Matching Psf FWHM 4.74 -> 11.84 pix                                        
imageDifference.subtract INFO: Final spatial kernel sum 1.120                                             
imageDifference.subtract INFO: Spatial model condition number 3.566e+07                                   
imageDifference.subtract INFO: Doing stats of kernel candidates used in the spatial fit.                  
imageDifference.subtract INFO: 32 candidates total, 18 rejected, 14 used                                  
imageDifference.subtract INFO: Spatial kernel model well constrained; 14 candidates, 3 terms, 27 bases    
imageDifference.subtract INFO: Spatial background model appears well constrained; 14 candidates, 1 terms  
imageDifference FATAL: Failed on dataId=DataId(initialdata={'fileroot': 'SN2011iu_A_H_20120108'}, tag=set([])): [Errno 20] Not a directory                                                                         
Traceback (most recent call last):                                                                        
  File "/global/common/cori/contrib/lsst/lsstDM/dc1/Linux64/pipe_base/12.1-5-g06c326c+4/python/lsst/pipe/base/cmdLineTask.py", line 347, in __call__                                                                
    result = task.run(dataRef, **kwargs)                                                                  
  File "/global/common/cori/contrib/lsst/lsstDM/dc1/Linux64/pipe_base/12.1-5-g06c326c+4/python/lsst/pipe/base/timer.py", line 121, in wrapper                                                                      
    res = func(self, *args, **keyArgs)                                                                    
  File "/global/common/cori/contrib/lsst/lsstDM/dc1/Linux64/pipe_tasks/12.1-20-g324f6d3+4/python/lsst/pipe/tasks/imageDifference.py", line 521, in run                                                              
    sensorRef.put(subtractRes.matchedExposure, self.config.coaddName + "Diff_matchedExp")                 
  File "/global/common/cori/contrib/lsst/lsstDM/dc1/Linux64/daf_persistence/12.1-17-g9654cba+4/python/lsst/daf/persistence/butlerSubset.py", line 216, in put                                                       
    self.butlerSubset.butler.put(obj, datasetType, self.dataId, doBackup=doBackup, **rest)                
  File "/global/common/cori/contrib/lsst/lsstDM/dc1/Linux64/daf_persistence/12.1-17-g9654cba+4/python/lsst/daf/persistence/butler.py", line 784, in put                                                             
    location.getRepository().write(location, obj)                                                         
  File "/global/common/cori/contrib/lsst/lsstDM/dc1/Linux64/daf_persistence/12.1-17-g9654cba+4/python/lsst/daf/persistence/repository.py", line 144, in write                                                       
    return self._storage.write(butlerLocation, obj)                                                       
  File "/global/common/cori/contrib/lsst/lsstDM/dc1/Linux64/daf_persistence/12.1-17-g9654cba+4/python/lsst/daf/persistence/posixStorage.py", line 229, in write                                                    
    self.persistence.persist(obj, storageList, additionalData)                                            
  File "/global/common/cori/contrib/lsst/lsstDM/dc1/Linux64/miniconda2/3.19.0.lsst4/lib/python2.7/contextlib.py", line 24, in __exit__                                                                              
    self.gen.next()                                                                                       
  File "/global/common/cori/contrib/lsst/lsstDM/dc1/Linux64/daf_persistence/12.1-17-g9654cba+4/python/lsst/daf/persistence/safeFileIo.py", line 126, in SafeFilename                                               
    os.rename(tempName, name)                                                                             
OSError: [Errno 20] Not a directory

Do forced-photometry on calexp; diffim for given RA, Dec

Write code that will do forced-photometry on either calibrated images (calexp) or subtracted image (diffim) for a given set of RA, Dec.

See

https://community.lsst.org/t/how-do-i-do-forced-photometry-on-a-set-of-ra-dec/1074/4

Calibrate all DR2 images.

processCcd is not calibrating all DR2 images. Need to figure out why and fix this.

Solve "Too Many CR pixels" errors; presumably by setting smaller initial PSF guess

Identify a test case with 3 images that fails.
Learn how to submit smaller initial PSF guess
Test to see if smaller initial PSF solves problem

Figure out why I need to specify `expnum` in subtractions.

expnum still needs to be specified in subtractions or the wrong matching expnum is found. I think the same expnum is used for all images in a given subtractions.

Run subtraction for one field.

Run subtraction for one field

PTF11mty

Create individual-image lightcurves for 33 DR1 supernova

Run forced photometry on calexp and diffim

calexp
forced photometry

there are some bookkeping changes to be made from the twinkles_subtractions repo version.

Do forced-photometry on stacks based on coadd detections for all objects

Run calexp forced photometry for non-host-contaminated SNe

Currently only running photometry only for SNe with difference images, which are those with host template available in DR1.

Should run calexp forced photometry on all DR1 SNe that don't need host template subtractions.

Enable astrometric and photometric calibration

Run subtractions for DR1 fields with references.

Run subtraction for SN~2011hb (Host images taken 2013B)
Run subtraction for SN~2011iu (Host images taken 2012B)
Run subtraction for SN~2011gy (Host images taken 2012A+B)
Run subtraction for PTF~11owc (Host images taken 2012A+B)

#Name                    semester    RA            Dec      z       approxTmax n_J n_H n_K Host_J  Host_H  Host_K Super-LOTIS comments
PTF11mty                 2011B  21:34:05.20   +10:25:24.6   0.077    20111001   2   2   0  2012B     2012B     NA        ...  "ATEl 3668"
PTF11owc                 2011B  09:16:10.05   +49:37:30.7   0.07     20111102   3   3   0  2012A     2012A+B   NA        ...  "ATEL 3739; Can't find SN"
SN2011gy                 2011B  03:29:35.319  +40:52:02.93  0.017    20111031   4   4   0  2012B     2012A+B   NA        ...  "CBET 2871; Updated Position"
SN2011iu                 2011B  23:51:02.342  +46:43:21.55  0.046    20111129   2   2   0  2012B     2012B     NA        ...  "CBET 2939; Updated position"
LSQ12fuk                 2012B  04:58:15.927  -16:17:58.14  0.02     20121101   4   4   0  2013B     2013B     NA        ...  "ATEL 4537; Updated position"
PSNJ07250042+2347030     2012B  07:25:00.408  +23:47:03.15  0.018    20121018   5   5   0  2013B     2013B     NA        ...  "ATEL 4497; Updated position"
PTF11qzq                 2011B  07:19:27.311  +54:13:48.84  0.06     20111210   2   2   0  2013B     2012B     NA        ...  "ATEL 3798; Updated position"
SN2011hb                 2011B  23:27:55.508  +08:46:45.45  0.029    20111107   3   3   0  2013B     2013B     NA        ...  "CBET 2880, 2011hb, PTF11pbp; Updated Position"
SN2012fm                 2012B  06:56:13.884  +84:04:50.26  0.014    20121028   5   5   0  2013B     2013B     NA        ...  "CBET 3272; Updated position; Images needed 1 deg CCW rotation - perhaps extreme Dec?"

Run processCcd for all DR1 stacks

Presently only 150 of 700 stacks successfully make it through processCcd.py. Get this up to at least 80%.

Debug astrometric matching
Debug reference catalogs
Identify additional failure modes.

Update to work with 12.1 and WIYN+WHIRC data

Errors that happen are

python/lsst/meas/base/baseLib_wrap.cc:44109:53: error: no member named 'logging' in namespace 'lsst::pex'

Identify why some fields have multiple patches in the tract.

Identify why PTF11qzq and SN2012go have 0,1 patches in addition to 0,0 patches.
Address multiple patches for a tract if there are fields where that's necessary.
Identify field size. Currently default of 4000x4000 pixels is too big for most fields.

Create per field/filter coadds for all DR1 fields

Create coadd for one field
Do photometry for one field
Create coadds for all fields. -> #17 Blocked on #16.
Created photometry for all fields. -> #17 Blocked on #16

Extract lightcurve for one field: LSQ13cwp

Investigate LSQ13cwp, LSQ13cmt, iPTF13dkl calexp forced-photometry lightcurves

The lightcurves of LSQ13cwp, LSQ13cmt, iPTF13dkl are 0.5 mag discrepant between the DM Science Pipelines analysis and the published R! lightcurves.

Generate lightcurves of the DR1 calibration 2MASS star in these fields
Compare 3 additional 2MASS stars in each field to the DR1 2MASS lightcurves
Compare zeropoints between DM Science Pipelines and DR1 published data+calibrations for each image in these fields.

Extract lightcurve from subtractions for isolated SN

Extract lightcurve from subtractions for isolated SN: LSQ13cwp

Extract lightcurve from subtractions for LSQ13cwp
Compare to individual-image photometry.

Make sn_lightcurve.py use same Butler object

Currently a new butler is instantiated for each make_lc call. RE-factor to only use one.

Consider creating an object view with state.

Switch to including `expnum` in filenames

Include expnum in stack, claexp, and related filenames. This way one doesn't have to know the expnum to look up an image. The Butler will complete a partial dataId to include the expnum information, which is redundant with field+filter.

Switch to repo/test_dr1 instead of tmp/test_dr1 to indicate progress.
Remove expnum from scripts calling stack code.
Verify processCcd still works
Verify coaddition still works
Verify subtraction still works.

This is unfortunately going to get mixed in with getting this to work in new w_2018_14, because I need to update to deal with an int size issue that was fixed as of d_2018_04_06.