As titled.

Generate routines (and provide a settings option) to automatically identify the type of frame, so that we don't have to rely on headers.

To better keep track of changes in behavior due to dependency updates, we should include the version of each dependency in some output. This could be in the log, or in the header of some fits outputs.

In arsort.match_science, the time difference is calculated between each frame and a target science frame (line 437). This value should be less than that specified in spect['fits']['calwin']. The next line gets the indices satisfying that criterion, but then we should set the indices of interest to n[w] rather than the intersection of n (the indices corresponding to the frames) and w (the indices of n which match the time window). The time_window_fix branch makes that switch, but someone should check this to make sure it makes sense.

Line 783:
warnings.warn(msg, pu.RankWarning)
pu is not defined

Line 199:
tpixt, num = arcyarc.detections_sigma(yval,np.array([rms]*mid_row.shape[0]),np.zeros(yval.shape[0],dtype=np.int),siglev/2.0,siglev)

mid_row is not defined

At the moment only a ".red" file is allowed to be read in. But this might be a confusing extension if there's a red and a blue arm of the instrument, and red/blue is in the filename.

Please propose your favourite alternatives! Here are some suggestions:

.reduce
.pypit
.input
.in

I quite like ".pypit", but I'm not married to it...

Consider reducing trace orders polyorder to 2 (or 1) for amrlsd

I don't think any of the standard instruments need polyorder=3.

As titled ;-)

I use python3.5, and retrieved and installed all the most updated packages on Github.

When I run "./pypit_test lris" in test_suite, it says:

"No such file or directory:
[BUG] :: There appears to be a bug on Line 514 of arload.py with error:
[Errno 2] No such file or directory: '/Users/zhengcai/software/PYPIT/build/lib.macosx-10.5-x86_64-3.5/pypit/settings.lris_blue'
---> please contact the authors"

    I checked and found settings.* is actually in here: 

/Users/zhengcai/software/PYPIT/pypit/

    How can let PYPIT find this directory "/Users/zhengcai/software/PYPIT/pypit/" as default? 

I already added the following lines into ~/.cshrc

setenv TST_PYPIT /Users/zhengcai/software/PYPIT/test_suite/
setenv PYPIT /Users/zhengcai/software/PYPIT/pypit
setenv PYPIT_CALIBS /Users/zhengcai/software/PYPIT-development-suite/PYPIT_CALIBS

A cookbook needs to be written (likely by me) for reducing APF data with PYPIT. Use the test case - after ingested - for cookbook.

Consider whether this line in arproc.flatnorm:

            for i in xrange(recmean.size):
                recframe[i,:] /= recmean[i]

should be

            for i in xrange(recmean.size):
                recframe[:,i] /= recmean[i]

Running python setup.py install fails with
error: can't copy 'pypit/data/arc_lines/NIST': doesn't exist or not a regular file

Didn't notice it since I was installing through symlinks. Can probably just specify the directory in the keyword, trying with branch install_fix.

There are NAN's entering in the calculation of wtilt
in artrace.py at line 1087:

        if wmask.size != 0:
            wtilt[xint:xint+2*sz+1, j] = np.sqrt(2.0) / (1.4826 * np.median(np.abs(ytfit-model)[wmask]))

This leads to NANs for the tilts.
Not sure why, nor how to mitigate.

This is occurring in the Kast blue test suite.

I'm working on a branch for developing the DEIMOS settings. It might be a good idea to have a separate branch for developing MOS reductions in general, which can then be merged into instrument-specific development.

At least spec1d

con: Developed and provided by Anaconda
pro: super-easy otherwise and can be huge speedup

Am starting a new branch for reducing WHT ISIS data

It is possible to create ipython notebooks (.ipynb files)
that one can then run (automatically or manually).

Consider having some/all QA in this mode.

I propose that throughout the code we use following debug import statement:

try:
from xastropy.xutils import xdebug as xdb
except:
import pdb as xdb

It would save us the trouble of changing xdb to pdb inline (and vice versa), especially if someone does not have xastropy installed. Then, throughout the code we consistently use: xdb.set_trace()

Any objections to me implementing this in my coming PR?

We should attempt to set the linelist to match the lamps
that were indicated to be on in the header (when possible).

The APF contains an image of the pupil in the red part of the spectrum. This will likely need to be addressed when both reducing science and calibration images.

I include an example in the image below. This is a science frame so the tertiary hole is visible.

Hey Ryan,
GSL, if installed by anaconda, causes issues in the test suite due to some relative paths (@rpath) that appear in .so files. This is an issue with the El Capitan operating system in particular (which introduced system integrity protection).
If you were able to change those relative paths to full paths (every installer should have defined the GSL_PATH and LD_LIBRARY_PATH environment variables in their rc files (see the new installation documentation I added), this issue would go away and people who installed GSL with anaconda could avoid some ugly hacks.

Thanks!
Camille

Should add a scaling option to comb_frames.

Distinct from weight [which was intended to only work with weightmean, I think]

It appears that Python3's range is the same as Python2's xrange.
There is no equivalent of Python2's range in Python3.

To be Python3 compliant we should replace all xrange instances with range
and I leave it to @rcooke-ast to decide whether to rename range as xrange throughout the package.

To have a 'permanent' record of how the data were reduced.

See diff for [https://github.com/PYPIT/PYPIT/commit/f2918b22b8a91052e0f4e34dd49226cccba27ee8]

If canbe is a single string instead of a list of strings, the current version will attempt to loop through that string, .e.g, if canbe = 'pixflat', then cb = 'p'

1. PDF file of QA plots
2. FITS file of science output
   JXP tasked to provide this

The line below replaces spaces with an empty string, and so any header entries with strings fail to be matched.

[https://github.com/PYPIT/PYPIT/commit/2718b76bc795ec008daf667315233e6f5152d402#diff-73e36695d52ea8945cfe778bacd5530fR578]

Is this addition necessary for the WHT settings? If not, the branch iss111 reverts the change.

Unidentified CRs are causing poor results in local
sky subtraction:

https://www.dropbox.com/s/hujnvi0quvy0icb/Screenshot%202016-05-02%2005.34.51.png?dl=0

Consider adding a clipping step to arcyutils.func2d_fit_val

Be able to get from dosctring documentation what kind of modes (e.g. armed, armlsd etc.), spectrographs (e.g. kast_blue, kast_red, lris_blue, etc.) are available.

We should write the arflags as a JSON file (or YAML)
as part of the standard outputs. This would really save
the input parameters (i.e. including the defaults).

Here are the relevant lines in LowRedux:

;;Begin - MF Jan 2011: Original line
; imagecol = ncol-namps_(prepix+postpix)/xbin
; Updated line:
IF (strcmp(instrument, 'LRIS')) and (strmid(sxpar(hdr, 'DATE'), 0, 10) GT '2010-12-10') THEN BEGIN
IF xbin EQ 2 THEN imagecol = ncol-namps_(prepix+postpix-1)/xbin $
ELSE IF xbin EQ 1 THEN imagecol = ncol-namps_(prepix+postpix)/xbin
;;
;; JFH 12-29-2013: the prepix + postpix = 87 for the new chip, so we need this -1
;; for xbin =2
ENDIF ELSE imagecol = ncol-namps_(prepix+postpix)/xbin
;Note that this might be a problem even before 2010-12-10 for binned data.
;;End - MF Jan 2011

Consider making it possible to derive these sections
from the Header instead of setting in the settings file.

add Kr and Xe

remove .py from scripts
do some updates to handle the refactoring

I've started a branch on this

Two things as regards flats:

1. I think rej_cosmicray = 20.0 allows in way too many CRs
2. Should consider adding a grow option (as defualt even). 1 pixel would probably be enough.

Deal with saturated arc lines (remove from
wavelength analysis).

Decide if MK_SATMASK is fine for long slit

Once APF code ingested, need to build a full test suite for the software.

Mauna Kea sky can be very dark.
Likely requires its own sky spectrum for LRISb.

May be causing issues on unix.

I don't think the code is properly making use of internally stored master frames,
i.e. the calibrations are being re-made for each exposure. Am going to investigate
in a branch named reuse_master

I've opened another branch for docs more_docs.

Please use it.

When we generate a FITS file with PYPIT, let's
be sure to add metadata describing it, e.g. "SPEC_1D"
and not rely on the filename.

The repository needs the current draft of the APF code and the corresponding test case.

Need to develop at least a semi-automated wavelength
calibration algorithm.

Consider developing a combined method that uses both bias images
and the overscan region.

This will be a band-aid for now, but I am going to open a branch
to calculate the offset input id_pix and those calculated.

trace_tilt() method
2 errors

det not defined
arfitbase not defined

Need to examine the variance image and related
products in PYPIT.

Current results are markedly different from LowRedux