It seems importing the package fails with Python 3.7. The __path__ of puil data is not subscriptable.There is likely a simple fix to work around this. I will look into it ASAP.

Error reported by Katherine:

PUPIL_DIR = pupil_data.__path__[0]
TypeError: '_NamespacePath' object is not subscriptable

Currently, the pipeline and its various steps are similar to the JWST pipeline from the user's perspective, but they don't follow the same code structure and inherit the same classes. Following the JWST structure should remove some complexity from the code and make it easier to eventually make this an official JWST pipeline.

There is documentation here writing a pipeline and a step.

I'm not self-assigning for now in case someone else wants to work on this in the short term, but I'm happy to look at this in the next few weeks.

Currently, the filter properties are hardcoded at the beginning of the KPI3Pipeline script. In the future, they should be downloaded on the fly from the Spanish Virtual Observatory Filter Profile Service. I have code that does that, I just need to find the time to implement it here.

It would be nice to be able to skip saving intermediate products from the pipeline. We might even want to make this the default option, given the size that intermediate FITS files can take.

I would see two complementary solutions here:

• Option to skip saving the HDUL for each file (I think this would be fixed automatically by solving #11, as implemented in #17)
• Option to save a "preprocessing" cube, where each HDU is a step of the pipeline. This might be an interesting solution if people want easy access to the different image pre-processing while reducing the output size (the required size is ~ 10% of what it is now from quick tests I did)

The Fourier plane bad pixel fixing algorithm has been implemented by the NIRISS Commissioning Team but it has not yet been propagated into the Kpi3Pipeline.

Currently, the default pupil model is an old square grid one (used during commissioning). We can replace the default models by the updated hexagonal grid from @kammerje, and also upgrade the pupil model generation script to show how the model was created.

I think there is a loop over vectorized code in the trim step:

The loop variable is not used and the code does the operation on all frames with : I believe.

SVO seems down currently, which made me realize it might be good to have a "fallback" option for when SVO cannot be queried (either by saving the SVO list for JWST when users run a query or by including a static version with the package.

@kammerje has an updated version of calwebb_kpi3.py with his latest changes. The formatting and structure changed slightly in this repo compared to the old one. I can take care of merging the updates into this repo.

Currently, files are overwritten by default. It might be good to have an option to fully skip the processing if the output already exist. Or maybe still run the processing, but skip saving the file if it exists.

Currently, the pupil model is used directly from the jwst subdirectory and the path is hardcoded. Two things that could add more flexbility would be:

• Allow users to specify a path to a pupil model instead of using the one from the repo
• Integrate a step in the pipeline to genreate a pupil model with specific settings

@kammerje let me know what you think. The first item is probably simple to implement. The second one is a bit more involved.

Whe bad pixels are NaN (e.g. with NIRCam default reductions), the scipy medfilt does not behave well in my experience. Astropy has workarounds for this, so it might be worth switching to astropy for median filtering operations in the pipeline.

This page expalins the difference between astropy and scipy for NaN handling: https://docs.astropy.org/en/stable/convolution/index.html.

What do you think @kammerje ?

Once kammerje/xara#13 is merged, we can update the dependency back to his fork. Also, once this repo is transferred to @kammerje, we can update the install instructions to the new link.