keck-datareductionpipelines / kcwidrp Goto Github PK
View Code? Open in Web Editor NEWKCWI data reduction pipeline. Developers: Don Neill, Matt Matuszewski, Chris Martin (Caltech)
License: GNU General Public License v2.0
KCWI data reduction pipeline. Developers: Don Neill, Matt Matuszewski, Chris Martin (Caltech)
License: GNU General Public License v2.0
Hi, we are wanting to skip sky subtraction because of a complicated background field, but I don't quite understand the output files. From the documentation (DATA_PRODUCTS), there should be no difference between the "_ocubes" files with or without sky-subtraction; the difference should be in the "_icubes" files; is this correct?
Also, when I do sky-subtraction, the algorithm doesn't seem as simple as "icube=ocube-scube"; I get residuals of about 1 part in 10^7 (but definitely not zero). Can you please explain?
BTW, we are NOT using N&S.
Hello, I am a little confused about how exactly or whether to run the Stage5sky step. I have observed three difference science targets whose extended emission warranted separate sky frames for each. According to the AAAREADME file, I would need to input said sky frames as the sky master in the kcwi.proc file. (I presume these would be the *_intf.fits versions, or the *_sky.fits files? Would the latter be generated if stage5 wasn't run?) But more importantly, the readme file also said that this step should be skipped if the sky is not well sampled in the frame or is mixed with a large fraction of object light. So should stage5 be run with the separate sky frames specified in the kcwi.proc, or not? Or would stage6cube just generate separate sky cubes if they're correctly identified in the kcwi.proc file (after stage 5 is skipped)? Thank you for any pointer in advance.
Kcwi_prep works once the file root is provided explicitly but everything beyond stage 1 of the pipeline runs into problems. Specifically, the DRP does not appear to create / find the necessary calibration files:
41/ 41 88 2 2 TUP 0 10 000 Med BM KBlu 5000.0 2458573.138 10.0 object 88 308.59116667 25.06380556 90.00 1.370 wolf1346
KCWI_STAGE2DARK [INFO 1 ] input reduced image: /Volumes/T01/d1/keck/kcwi/raw/mar19/redux/kb190330_00088_int.fits
KCWI_DO_DARK [INFO 1 ] DS Test (drk?, msig, tsig): NO 0.1864 0.2841
KCWI_STAGE2DARK [WARNING] cannot associate with any master dark: kb190330_00088_int.fits
KCWI_STAGE2DARK [INFO 1 ] scattered light subtracted
KCWI_WRITE_IMAGE [INFO 1 ] wrote image file: /Volumes/T01/d1/keck/kcwi/raw/mar19/redux/kb190330_00088_mskd.fits
KCWI_WRITE_IMAGE [INFO 1 ] wrote image file: /Volumes/T01/d1/keck/kcwi/raw/mar19/redux/kb190330_00088_vard.fits
KCWI_WRITE_IMAGE [INFO 1 ] wrote image file: /Volumes/T01/d1/keck/kcwi/raw/mar19/redux/kb190330_00088_intd.fits
KCWI_STAGE2DARK [INFO 1 ] run time in seconds: 117.51911
KCWI_STAGE2DARK [INFO 1 ] finished on Fri Apr 12 11:57:07 2019
IDL> kcwi_stage3geom
% Compiled module: KCWI_STAGE3GEOM.
fatal: Not a git repository (or any of the parent directories): .git
KCWI_READ_PROC [INFO 1 ] Master proc file: /Volumes/T01/d1/keck/kcwi/raw/mar19/redux/kcwi.proc
KCWI_READ_PROC [INFO 1 ] Fri Apr 12 11:57:15 2019
KCWI_STAGE3GEOM [INFO 1 ] Number of input images: 41
1/ 41 27 2 2 TUP 0 10 000 Med BM KBlu 5000.0 2458572.590 36.7 dflat 27 58.00000000 45.00000000 0.00 1.410
KCWI_STAGE3GEOM [INFO 1 ] input 2-D image: /Volumes/T01/d1/keck/kcwi/raw/mar19/redux/kb190330_00027_intd.fits
KCWI_STAGE3GEOM [WARNING] missing calibration file(s) for: /Volumes/T01/d1/keck/kcwi/raw/mar19/redux/kb190330_00027_intd.fits
The AAAREADME file provides no information that would help me solve these problems - or did I miss it? All my original FITS files are in the same directory, and I specified "." as the "calibration directory" during the kcwi_prep phase.
Thanks in advance!
Harald
We were observing He3 in the standard star list and it appears to have a proper motion moving it off the slicer from the listed position. Could the coordinates be updated? It may also be useful to check other standards for proper motions.
To Whom It May Concern,
I am running into over subtraction issues, thus I am trying to run the kcwi_stage5sky with the display=2.
However, an error is popping up kctv.pro at line 344. There appears to be a missing function called
"list".
Any help is appreciated, thanks!
Hi again - On line 89 of MakeMasterSky.py
the code appears to be reading in the original file name (but it's possible that I'm missing something):
groot = tab['OFNAME'][0].split('.fits')[0]
Is there a workaround for dealing with data downloaded from the archive (that have a different filename convention)? Here is the full traceback:
2021-06-11 11:23:35:KCWI:INFO: Looking for ARCLAMP frames
2021-06-11 11:23:35:KCWI:INFO: Reading image: kb191219_00036_wavemap.fits
[Errno 2] No such file or directory: 'redux/kb191219_00036_wavemap.fits'
2021-06-11 11:23:35:DRPF:ERROR: Exception [Errno 2] No such file or directory: 'redux/kb191219_00036_wavemap.fits' while invoking MakeMasterSky
Traceback (most recent call last):
File "/Users/rcooke/anaconda3/lib/python3.7/site-packages/keckdrpframework/core/framework.py", line 229, in execute
action_output = pipeline.get_action(action_name)(action, context)
File "/Users/rcooke/anaconda3/lib/python3.7/site-packages/keckdrpframework/pipelines/base_pipeline.py", line 84, in f
return obj.apply()
File "/Users/rcooke/anaconda3/lib/python3.7/site-packages/keckdrpframework/primitives/base_primitive.py", line 57, in apply
self.output = self._perform()
File "/Users/rcooke/anaconda3/lib/python3.7/site-packages/kcwidrp/primitives/MakeMasterSky.py", line 96, in _perform
wmf))[0]
File "/Users/rcooke/anaconda3/lib/python3.7/site-packages/kcwidrp/primitives/kcwi_file_primitives.py", line 559, in kcwi_fits_reader
raise e
File "/Users/rcooke/anaconda3/lib/python3.7/site-packages/kcwidrp/primitives/kcwi_file_primitives.py", line 556, in kcwi_fits_reader
hdul = fits.open(file)
File "/Users/rcooke/anaconda3/lib/python3.7/site-packages/astropy/io/fits/hdu/hdulist.py", line 164, in fitsopen
lazy_load_hdus, **kwargs)
File "/Users/rcooke/anaconda3/lib/python3.7/site-packages/astropy/io/fits/hdu/hdulist.py", line 402, in fromfile
lazy_load_hdus=lazy_load_hdus, **kwargs)
File "/Users/rcooke/anaconda3/lib/python3.7/site-packages/astropy/io/fits/hdu/hdulist.py", line 1051, in _readfrom
fileobj = _File(fileobj, mode=mode, memmap=memmap, cache=cache)
File "/Users/rcooke/anaconda3/lib/python3.7/site-packages/astropy/utils/decorators.py", line 535, in wrapper
return function(*args, **kwargs)
File "/Users/rcooke/anaconda3/lib/python3.7/site-packages/astropy/io/fits/file.py", line 175, in __init__
self._open_filename(fileobj, mode, overwrite)
File "/Users/rcooke/anaconda3/lib/python3.7/site-packages/astropy/io/fits/file.py", line 564, in _open_filename
self._file = fileobj_open(self.name, IO_FITS_MODES[mode])
File "/Users/rcooke/anaconda3/lib/python3.7/site-packages/astropy/io/fits/util.py", line 397, in fileobj_open
return open(filename, mode, buffering=0)
FileNotFoundError: [Errno 2] No such file or directory: 'redux/kb191219_00036_wavemap.fits'
2021-06-11 11:23:35:DRPF:INFO: Event failed: name object_make_sky, action MakeMasterSky, arg name KB.20191219.40218.fits.gz, recurr False
Note: kcwi_stage6drr is not listed in the code flow in AAAReadme. should it be there?
Hi there - I was running the standard pipeline command reduce_kcwi -f KB*.fits.gz
, and I received an error that _lacosmicx
is not defined. Looking at the RemoveCosmicRays.py
file, it seems that _lacosmicx
is imported in the init
method, rather than at the top of the file. I do have lacosmicx
installed, and it seems to be imported fine in the init method. I was able to fix the bug by moving the import to the top of the file (where the other imports are located).
Here is the full traceback:
2021-06-11 11:12:29:DRPF:ERROR: Exception name '_lacosmicx' is not defined while invoking RemoveCosmicRays
Traceback (most recent call last):
File "/Users/rcooke/anaconda3/lib/python3.7/site-packages/keckdrpframework/core/framework.py", line 229, in execute
action_output = pipeline.get_action(action_name)(action, context)
File "/Users/rcooke/anaconda3/lib/python3.7/site-packages/keckdrpframework/pipelines/base_pipeline.py", line 84, in f
return obj.apply()
File "/Users/rcooke/anaconda3/lib/python3.7/site-packages/keckdrpframework/primitives/base_primitive.py", line 57, in apply
self.output = self._perform()
File "/Users/rcooke/anaconda3/lib/python3.7/site-packages/kcwidrp/primitives/RemoveCosmicRays.py", line 54, in _perform
mask, clean = _lacosmicx.lacosmicx(
NameError: name '_lacosmicx' is not defined
I recently ran the kcwi_drp for the archive data. I found standard stars that are not in the database (data/stds) of the kcwi_drp.
These standard star are observed at 2019/12/19 named BB0837p5427, BB1245p4238, and BB1417p4941.
Since these standard stars are not within the kcwi_drp database, I cannot run the final step which is kcwi_stage8std.
Is there any way to solve this problem?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.