Tyler Desjardins mentions that we should consider moving emails from help[at]stsci.edu to point to the web portal where possible and appropriate. For HST (or any non-JWST), it is https://hsthelp.stsci.edu . For JWST, it is https://jwsthelp.stsci.edu . Please update info in setup.py, setup.cfg, documentation, etc as appropriate.

Please close this issue if it is irrelevant to your repository. This is an automated issue. If this is opened in error, please let pllim know!

xref spacetelescope/hstcal#317

Hello!

I was running updatewcs on some ACS images of 47 Tuc (ex. ja9bw1xhq_flc.fits) with a new IDCTAB when updatewcs crashed. I took a screenshot of the traceback to help you with debugging see attached), but you can see that it was stwcs 1.5.1 with python 3.7.3. It ran perfectly when I used version 1.4.2 of stwcs that I had in an older environment with python 3.5.6, so I know the problem is not the IDCTAB, but something to do with updatewcs. Also, this is data that was downloaded awhile ago (header says 2018-06-11) so it doesn't have the new astrometry. I'm putting the two lines of code that I used to get this error below.

import stwcs
stwcs.updatewcs.updatewcs('j*flc.fits', use_db=False)

Please let me know if you need any other information.

Cheers,
Samantha

I think this code has the opposite effect of what is desired:

CC: @nden

As reported in astropy/astropy#8777, Dr. Peter Maksym traced a crash in the astrodrizzle to apply_headerlet appending new D2IM1.* and D2IM2.* keywords to the header even if they already exist. Therefore, when a WCS object is being constructed, astropy.wcs.WCS() crashes because of missing D2IM1.NAXES. PR astropy/astropy#8777 improves robustness of astropy.wcs.WCS(). However, IMO stwcs should not create duplicate D2IM values in the first place.

Example (using a WFPC2 image)

Essentially, perform the following operations:

1. Run updatewcs()
2. make sure we can load the WCS from extension 1. It should work.
3. create a headerlet from existing WCS
4. apply the headerlet to the image from which it was created. Inspecting the header will reveal two instances (duplicates) of D2IM keywords.
5. attempt to create a WCS from updated (with headerlet WCS) image. It will crash.

Python 3.7.3 (default, Mar 27 2019, 16:54:48)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.5.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from stwcs.updatewcs import updatewcs

In [2]: from stwcs.wcsutil import HSTWCS, headerlet

In [3]: import astropy

In [4]: import stwcs

In [5]: print(astropy.__version__)
3.1.2

In [6]: print(stwcs.__version__)
1.4.2

In [7]: updatewcs('ua3w6209m_c0m.fits', use_db=False)
<class 'str'>
('ua3w6209m_c0m.fits', None)
WARNING: AstropyDeprecationWarning:
Private attributes "_naxis1" and "_naxis2" have been deprecated since v3.1.
Instead use the "pixel_shape" property which returns a list of NAXISj keyword values.
 [astropy.wcs.wcs]
WARNING: AstropyDeprecationWarning:
Private attributes "_naxis1" and "_naxis2" have been deprecated since v3.1.
Instead use the "pixel_shape" property which returns a list of NAXISj keyword values.
 [astropy.wcs.wcs]
- IDCTAB: Distortion model from row 421 for chip 3 : F606W,POLQ
- OFFTAB: Offset interpolated from rows 31 and 35
WARNING: VerifyWarning: Card is too long, comment will be truncated. [astropy.io.fits.card]
WARNING: AstropyDeprecationWarning:
Private attributes "_naxis1" and "_naxis2" have been deprecated since v3.1.
Instead use the "pixel_shape" property which returns a list of NAXISj keyword values.
 [astropy.wcs.wcs]
WARNING: AstropyDeprecationWarning:
Private attributes "_naxis1" and "_naxis2" have been deprecated since v3.1.
Instead use the "pixel_shape" property which returns a list of NAXISj keyword values.
 [astropy.wcs.wcs]
- IDCTAB: Distortion model from row 89 for chip 1 : F606W,POLQ
- OFFTAB: Offset interpolated from rows 29 and 33
- IDCTAB: Distortion model from row 255 for chip 2 : F606W,POLQ
- OFFTAB: Offset interpolated from rows 30 and 34
- IDCTAB: Distortion model from row 421 for chip 3 : F606W,POLQ
- OFFTAB: Offset interpolated from rows 31 and 35
- IDCTAB: Distortion model from row 587 for chip 4 : F606W,POLQ
- OFFTAB: Offset interpolated from rows 32 and 36
Out[7]: ['ua3w6209m_c0m.fits']

In [8]: w = HSTWCS('ua3w6209m_c0m.fits', ext=1)
WARNING: AstropyDeprecationWarning:
Private attributes "_naxis1" and "_naxis2" have been deprecated since v3.1.
Instead use the "pixel_shape" property which returns a list of NAXISj keyword values.
 [astropy.wcs.wcs]
WARNING: AstropyDeprecationWarning:
Private attributes "_naxis1" and "_naxis2" have been deprecated since v3.1.
Instead use the "pixel_shape" property which returns a list of NAXISj keyword values.
 [astropy.wcs.wcs]

In [9]: headerlet.create_headerlet('ua3w6209m_c0m.fits').writeto('ua3w6209m_hlet.fits', overwrite=True)
WARNING: AstropyDeprecationWarning:
Private attributes "_naxis1" and "_naxis2" have been deprecated since v3.1.
Instead use the "pixel_shape" property which returns a list of NAXISj keyword values.
 [astropy.wcs.wcs]
WARNING: AstropyDeprecationWarning:
Private attributes "_naxis1" and "_naxis2" have been deprecated since v3.1.
Instead use the "pixel_shape" property which returns a list of NAXISj keyword values.
 [astropy.wcs.wcs]
WARNING: VerifyWarning: Card is too long, comment will be truncated. [astropy.io.fits.card]

In [10]: headerlet.apply_headerlet_as_primary('ua3w6209m_c0m.fits', 'ua3w6209m_hlet.fits')
Applying ua3w6209m_hlet.fits as Primary WCS to ua3w6209m_c0m.fits
Initializing new WCSCORR table for  ua3w6209m_c0m.fits
WARNING: AstropyDeprecationWarning:
Private attributes "_naxis1" and "_naxis2" have been deprecated since v3.1.
Instead use the "pixel_shape" property which returns a list of NAXISj keyword values.
 [astropy.wcs.wcs]
WARNING: AstropyDeprecationWarning:
Private attributes "_naxis1" and "_naxis2" have been deprecated since v3.1.
Instead use the "pixel_shape" property which returns a list of NAXISj keyword values.
 [astropy.wcs.wcs]
WARNING: AstropyDeprecationWarning:
Private attributes "_naxis1" and "_naxis2" have been deprecated since v3.1.
Instead use the "pixel_shape" property which returns a list of NAXISj keyword values.
 [astropy.wcs.wcs]
WARNING: AstropyDeprecationWarning:
Private attributes "_naxis1" and "_naxis2" have been deprecated since v3.1.
Instead use the "pixel_shape" property which returns a list of NAXISj keyword values.
 [astropy.wcs.wcs]
WARNING: VerifyWarning: Keyword name 'IDCSCALEO' is greater than 8 characters or contains characters not allowed by the FITS standard; a HIERARCH card will be created. [astropy.io.fits.card]
WARNING: AstropyDeprecationWarning:
Private attributes "_naxis1" and "_naxis2" have been deprecated since v3.1.
Instead use the "pixel_shape" property which returns a list of NAXISj keyword values.
 [astropy.wcs.wcs]
WARNING: AstropyDeprecationWarning:
Private attributes "_naxis1" and "_naxis2" have been deprecated since v3.1.
Instead use the "pixel_shape" property which returns a list of NAXISj keyword values.
 [astropy.wcs.wcs]
WARNING: VerifyWarning: Keyword name 'IDCSCALEO' is greater than 8 characters or contains characters not allowed by the FITS standard; a HIERARCH card will be created. [astropy.io.fits.card]
WARNING: AstropyDeprecationWarning:
Private attributes "_naxis1" and "_naxis2" have been deprecated since v3.1.
Instead use the "pixel_shape" property which returns a list of NAXISj keyword values.
 [astropy.wcs.wcs]
WARNING: AstropyDeprecationWarning:
Private attributes "_naxis1" and "_naxis2" have been deprecated since v3.1.
Instead use the "pixel_shape" property which returns a list of NAXISj keyword values.
 [astropy.wcs.wcs]
WARNING: VerifyWarning: Keyword name 'IDCSCALEO' is greater than 8 characters or contains characters not allowed by the FITS standard; a HIERARCH card will be created. [astropy.io.fits.card]
WARNING: AstropyDeprecationWarning:
Private attributes "_naxis1" and "_naxis2" have been deprecated since v3.1.
Instead use the "pixel_shape" property which returns a list of NAXISj keyword values.
 [astropy.wcs.wcs]
WARNING: AstropyDeprecationWarning:
Private attributes "_naxis1" and "_naxis2" have been deprecated since v3.1.
Instead use the "pixel_shape" property which returns a list of NAXISj keyword values.
 [astropy.wcs.wcs]
WARNING: VerifyWarning: Keyword name 'IDCSCALEO' is greater than 8 characters or contains characters not allowed by the FITS standard; a HIERARCH card will be created. [astropy.io.fits.card]
WARNING: AstropyDeprecationWarning:
Private attributes "_naxis1" and "_naxis2" have been deprecated since v3.1.
Instead use the "pixel_shape" property which returns a list of NAXISj keyword values.
 [astropy.wcs.wcs]
WARNING: AstropyDeprecationWarning:
Private attributes "_naxis1" and "_naxis2" have been deprecated since v3.1.
Instead use the "pixel_shape" property which returns a list of NAXISj keyword values.
 [astropy.wcs.wcs]
WARNING: AstropyDeprecationWarning:
Private attributes "_naxis1" and "_naxis2" have been deprecated since v3.1.
Instead use the "pixel_shape" property which returns a list of NAXISj keyword values.
 [astropy.wcs.wcs]
WARNING: AstropyDeprecationWarning:
Private attributes "_naxis1" and "_naxis2" have been deprecated since v3.1.
Instead use the "pixel_shape" property which returns a list of NAXISj keyword values.
 [astropy.wcs.wcs]
WARNING: AstropyDeprecationWarning:
Private attributes "_naxis1" and "_naxis2" have been deprecated since v3.1.
Instead use the "pixel_shape" property which returns a list of NAXISj keyword values.
 [astropy.wcs.wcs]
WARNING: AstropyDeprecationWarning:
Private attributes "_naxis1" and "_naxis2" have been deprecated since v3.1.
Instead use the "pixel_shape" property which returns a list of NAXISj keyword values.
 [astropy.wcs.wcs]
WARNING: AstropyDeprecationWarning:
Private attributes "_naxis1" and "_naxis2" have been deprecated since v3.1.
Instead use the "pixel_shape" property which returns a list of NAXISj keyword values.
 [astropy.wcs.wcs]
WARNING: AstropyDeprecationWarning:
Private attributes "_naxis1" and "_naxis2" have been deprecated since v3.1.
Instead use the "pixel_shape" property which returns a list of NAXISj keyword values.
 [astropy.wcs.wcs]
WARNING: AstropyDeprecationWarning:
Private attributes "_naxis1" and "_naxis2" have been deprecated since v3.1.
Instead use the "pixel_shape" property which returns a list of NAXISj keyword values.
 [astropy.wcs.wcs]
WARNING: AstropyDeprecationWarning:
Private attributes "_naxis1" and "_naxis2" have been deprecated since v3.1.
Instead use the "pixel_shape" property which returns a list of NAXISj keyword values.
 [astropy.wcs.wcs]
WARNING: VerifyWarning: Card is too long, comment will be truncated. [astropy.io.fits.card]
WARNING: FITSFixedWarning: The WCS transformation has more axes (2) than the image it is associated with (0) [astropy.wcs.wcs]
/.../lib/python3.7/site-packages/stwcs/wcsutil/hstwcs.py:468: RuntimeWarning: cdelt will be ignored since cd is present
  self.wcs.cd = self.wcs.pc * self.wcs.cdelt[1]
WARNING: FITSFixedWarning: The WCS transformation has more axes (2) than the image it is associated with (0) [astropy.wcs.wcs]
/.../lib/python3.7/site-packages/stwcs/wcsutil/hstwcs.py:468: RuntimeWarning: cdelt will be ignored since cd is present
  self.wcs.cd = self.wcs.pc * self.wcs.cdelt[1]
WARNING: AstropyDeprecationWarning:
Private attributes "_naxis1" and "_naxis2" have been deprecated since v3.1.
Instead use the "pixel_shape" property which returns a list of NAXISj keyword values.
 [astropy.wcs.wcs]
WARNING: AstropyDeprecationWarning:
Private attributes "_naxis1" and "_naxis2" have been deprecated since v3.1.
Instead use the "pixel_shape" property which returns a list of NAXISj keyword values.
 [astropy.wcs.wcs]
WARNING: FITSFixedWarning: The WCS transformation has more axes (2) than the image it is associated with (0) [astropy.wcs.wcs]
/.../lib/python3.7/site-packages/stwcs/wcsutil/hstwcs.py:468: RuntimeWarning: cdelt will be ignored since cd is present
  self.wcs.cd = self.wcs.pc * self.wcs.cdelt[1]
WARNING: AstropyDeprecationWarning:
Private attributes "_naxis1" and "_naxis2" have been deprecated since v3.1.
Instead use the "pixel_shape" property which returns a list of NAXISj keyword values.
 [astropy.wcs.wcs]
WARNING: AstropyDeprecationWarning:
Private attributes "_naxis1" and "_naxis2" have been deprecated since v3.1.
Instead use the "pixel_shape" property which returns a list of NAXISj keyword values.
 [astropy.wcs.wcs]
WARNING: FITSFixedWarning: The WCS transformation has more axes (2) than the image it is associated with (0) [astropy.wcs.wcs]
/.../lib/python3.7/site-packages/stwcs/wcsutil/hstwcs.py:468: RuntimeWarning: cdelt will be ignored since cd is present
  self.wcs.cd = self.wcs.pc * self.wcs.cdelt[1]
WARNING: AstropyDeprecationWarning:
Private attributes "_naxis1" and "_naxis2" have been deprecated since v3.1.
Instead use the "pixel_shape" property which returns a list of NAXISj keyword values.
 [astropy.wcs.wcs]
WARNING: AstropyDeprecationWarning:
Private attributes "_naxis1" and "_naxis2" have been deprecated since v3.1.
Instead use the "pixel_shape" property which returns a list of NAXISj keyword values.
 [astropy.wcs.wcs]
WARNING: FITSFixedWarning: The WCS transformation has more axes (2) than the image it is associated with (0) [astropy.wcs.wcs]
/.../lib/python3.7/site-packages/stwcs/wcsutil/hstwcs.py:468: RuntimeWarning: cdelt will be ignored since cd is present
  self.wcs.cd = self.wcs.pc * self.wcs.cdelt[1]
WARNING: AstropyDeprecationWarning:
Private attributes "_naxis1" and "_naxis2" have been deprecated since v3.1.
Instead use the "pixel_shape" property which returns a list of NAXISj keyword values.
 [astropy.wcs.wcs]
WARNING: AstropyDeprecationWarning:
Private attributes "_naxis1" and "_naxis2" have been deprecated since v3.1.
Instead use the "pixel_shape" property which returns a list of NAXISj keyword values.
 [astropy.wcs.wcs]
attach_to_file CRITICAL: Headerlet IDC_ta81040lu cannot be attached toobservation ua3w6209m_c0m.fits * Image ua3w6209m_c0m.fits already has headerlet with HDRNAME='IDC_ta81040lu'

WARNING: VerifyWarning: Card is too long, comment will be truncated. [astropy.io.fits.card]

In [11]: w = HSTWCS('ua3w6209m_c0m.fits', ext=1)
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-11-4d013fa0c6de> in <module>
----> 1 w = HSTWCS('ua3w6209m_c0m.fits', ext=1)

.../lib/python3.7/site-packages/stwcs/wcsutil/hstwcs.py in __init__(self, fobj, ext, minerr, wcskey)
    145
    146             WCS.__init__(self, ehdr, fobj=phdu, minerr=self.minerr,
--> 147                          key=self.wcskey)
    148             if self.instrument == 'DEFAULT':
    149                 self.pc2cd()

.../lib/python3.7/site-packages/astropy/wcs/wcs.py in __init__(self, header, fobj, key, minerr, relax, naxis, keysel, colsel, fix, translate_units, _do_set)
    455             self.naxis = est_naxis
    456
--> 457             det2im = self._read_det2im_kw(header, fobj, err=minerr)
    458             cpdis = self._read_distortion_kw(
    459                 header, fobj, dist='CPDIS', err=minerr)

.../lib/python3.7/site-packages/astropy/wcs/wcs.py in _read_det2im_kw(self, header, fobj, err)
    793                 for key in list(header):
    794                     if key.startswith(dp + str('.')):
--> 795                         del header[key]
    796             else:
    797                 tables[i] = None

.../lib/python3.7/site-packages/astropy/io/fits/header.py in __delitem__(self, key)
    219                 # To delete keyword without caring if they were present,
    220                 # Header.remove(Keyword) can be used with optional argument ignore_missing as True
--> 221                 raise KeyError("Keyword '{}' not found.".format(key))
    222
    223             for idx in reversed(indices[key]):

KeyError: "Keyword 'D2IM2.NAXES' not found."

Testing of drizzled products gets complicated by the fact that the order of the WCSDVARR extensions containing the WCS solutions can change from one build to the next. This occurs due to the use of a simple Python dict to manage the apriori solutions returned by the database. This should be updated to use an OrderedDict instead to preserve whatever (arbitrary and still potentially volatile) order the WCSs have as returned by the astrometry database. Comparisons with fitsdiff would then be easier to interpret.

Tracked with JIRA HLA-290.

Applying updatewcs(filename, use_db=True) multiple times results in multiple copies of primary WCS backed up as Alt WCS. WCSNAME of alternate WCSes repeat after two runs of updatewcs with use_db=True.

For example, if original primary WCSNAME was 'IDC_2731450pi', after first run of updatewcs(..., use_db=True) we get:

Run 1:

WCSNAME = 'IDC_2731450pi-FIT_REL_GAIADR2'                                       
WCSNAMEA= 'IDC_2731450pi'                                                       
HDRNAME = 'ie6d07ujq_flt_IDC_2731450pi-FIT_REL_GAIADR2'                         

Run 2:

WCSNAME = 'IDC_2731450pi-FIT_REL_GAIADR2'                                       
WCSNAMEA= 'IDC_2731450pi'                                                       
WCSNAMEB= 'ie6d07ujq_flt_IDC_2731450pi-FIT_REL_GAIADR2'                         
HDRNAME = 'ie6d07ujq_flt_IDC_2731450pi-FIT_REL_GAIADR2'                         

Run 2:

WCSNAME = 'IDC_2731450pi-FIT_REL_GAIADR2'                                       
WCSNAMEA= 'IDC_2731450pi'                                                       
WCSNAMEB= 'ie6d07ujq_flt_IDC_2731450pi-FIT_REL_GAIADR2'                         
WCSNAMEC= 'ie6d07ujq_flt_IDC_2731450pi-FIT_REL_GAIADR2'                         
HDRNAME = 'ie6d07ujq_flt_IDC_2731450pi-FIT_REL_GAIADR2'                         

I think if WCSNAME is the same and/or WCS parameters (CRPIX, CRVAL, CDi_j, etc.) are the same, then the primary WCS should not be saved again as an AltWCS.

IMPORTANT NOTE
These results are performed using master 4.2.dev259+g1a065d5ce after astropy/astropy#10482 was merged. Therefore the problem is independent from astropy/astropy#10479

Currently, tests are still running in pandokia using the following files, separate from those in this repo (xref https://ssb.stsci.edu/pandokia/pandokia.cgi?query=summary&qid=10553):

• testaltwcs.py
• testheaderlet.py
• test.py

However, I also see legit tests in the tests sub-directory in this repo. Why can't we just activate Travis CI here and deactivate those in pandokia? These tests don't see data intensive, so I don't even think we need Jenkins and Artifactory here.

The original WCS for HST observations, with the WCSNAME=OPUS gets defined by the HST calibration pipeline using information from a project level database that contains the aperture definitions for each instrument. This OPUS WCS, however, was created using a distortion model which was included in the project level database and most likely does NOT correspond to any currently used or updated distortion models provided by the instrument teams for use in the calibration pipeline.

Treating this WCS consistent with the FITS standards on alternate WCSs requires, then, that it get written and restored as a separate headerlet solution. This WCS should never be included in the image header at the same time as a SIP-based WCS, and should never have '-SIP' attached to the CTYPE keywords.

When restored to the image header from a headerlet, it should still be applied as the 'O' (capital o) alternate WCS to maintain at least a minimum amount of consistency with the historic use of this alternate WCS.

I've been told by the STIS team that updates they have made to the idctable file are not being properly accounted for. We are having a meeting today to discuss this with their team.
@stscicrawford @stscirij @stsci-hack

Python 2 will not be maintained past Jan 1, 2020 (see https://pythonclock.org/). Please remove all Python 2 compatibility and move this package to Python 3 only.

For conda recipe (including astroconda-contrib), please include the following to prevent packaging it for Python 2 (https://conda.io/docs/user-guide/tasks/build-packages/define-metadata.html?preprocessing-selectors#skipping-builds):

build: 
  skip: true  # [py2k] 

Please close this issue if it is irrelevant to your repository. This is an automated issue. If this is opened in error, please let pllim know!

If you look at https://readthedocs.org/projects/stwcs/builds/ , everytime a commit is pushed to master branch, two builds are kicked off: "master" and "latest". Someone with RTD admin access for this project should disable the "master" build, as just the "latest" should be sufficient.

This issue will be used to track the implementation of an interface to a new astrometry database which will contain new astrometrically-accurate solutions for the pointing for HST observations.

The initial requirements for this interface will be:

• it will use a web-interface to query the database and retrieve any new WCS solutions available as headerlets for the exposure
• if access to the web-interface can not be established, the user will be notified, either through an Exception that stops all processing or a warning in the log which allows processing to continue unhindered
• the code will rely on an environment variable to determine whether failure to access or failure in applying the new solutions will result in Exceptions (as requested by the pipeline) or simple warnings (which will support users better)
• new solutions retrieved as headerlets from the database will simply be appended to the observation
• this update will be implemented (at least initially) as part of the updatewcs step

and fix doc build.

@justincely It appears you are the only owner of this account. Can you add me as a maintainer?

Hello. I think I found some bugs in headerlet.restore_from_headerlet. I was working with some ACS images that I downloaded from MAST today. I needed to restore the WCS solution to those produced by the pipeline using only the IDCTAB and I am not sure that the WCS is completely restored. I ran one of the FLC through updatewcs with use_db=False (I am considering this the truth version). On a different copy, I ran headerlet.restore_from_headerlet('jbpb65q4q_flc.fits',hdrname='IDC_0461802dj').
I then checked some of the WCS related keywords to make sure they were restored properly. The ORIENTAT, CRVAL1, and CRVAL2 keywords look different from the truth.

filename hdu keyword value
restore/jbpb65q4q_flc.fits 1 CRVAL1 173.25250163464
updatewcs/jbpb65q4q_flc.fits 1 CRVAL1 173.252501634641

filename hdu keyword value
restore/jbpb65q4q_flc.fits 1 CRVAL2 -0.52163118590471
updatewcs/jbpb65q4q_flc.fits 1 CRVAL2 -0.5216311859047115

filename hdu keyword value
restore/jbpb65q4q_flc.fits 1 ORIENTAT -67.24843074506964
updatewcs/jbpb65q4q_flc.fits 1 ORIENTAT -67.21933245016795

Finally, in using both methods, the HDRNAME doesn't get properly restored.

filename hdu keyword value
restore/jbpb65q4q_flc.fits 1 HDRNAME jbpb65q4q_flc_IDC_0461802dj-FIT_REL_GAIADR2
updatewcs/jbpb65q4q_flc.fits 1 HDRNAME jbpb65q4q_flc_IDC_0461802dj-FIT_REL_GAIADR2

I am using STWCS version 1.5.3.

Is there a public access point for the AstrometryDB service? The default URL appears to point to a development server that it is not accessible outside of STScI.

from stwcs.updatewcs import astrometry_utils as stwcsau
db = stwcsau.AstrometryDB()
WARNING : AstrometryDB service inaccessible!
AstrometryDB called: https://mastdev.stsci.edu/portal/astrometryDB/

astroy.WCS _naxis1 and _naxis2 were deprecated in favor of pixel_shape astropy/astropy#7973.

Recent testing of the ACS distortion model by the ACS INS team uncovered a problem with the TDD correction. It appears that the correction is being applied with the wrong sign to the coefficient. This only came to light after they gained the ability to process several hundred datasets that span a 10 year observing period.

Drizzlepac logs often contain numerous messages about inconsistent use of SIP in HST images, see, e.g., https://ssb.stsci.edu/pandokia/pandokia.cgi?query=detail&key_id=40222839 such as:

INFO: 
                Inconsistent SIP distortion information is present in the FITS header and the WCS object:
                SIP coefficients were detected, but CTYPE is missing a "-SIP" suffix.
                astropy.wcs is using the SIP distortion coefficients,
                therefore the coordinates calculated here might be incorrect.

                If you do not want to apply the SIP distortion coefficients,
                please remove the SIP coefficients from the FITS header or the
                WCS object.  As an example, if the image is already distortion-corrected
                (e.g., drizzled) then distortion components should not apply and the SIP
                coefficients should be removed.

                While the SIP distortion coefficients are being applied here, if that was indeed the intent,
                for consistency please append "-SIP" to the CTYPE in the FITS header or the WCS object.

                 [astropy.wcs.wcs]

Often this happens when drizzlepac code attempts to load the "original" WCS (the one that relies on IDCTAB instead of SIP to describe distortions). stwcs should be modified to remove SIP from image headers before loading alternative WCS.

It looks to me that the WCS being archived after >= 2 runs of updatewcs() is the wrong one. This issue might be related to #128. It seems that updatewcs() keeps archiving AltWCS instead of the primary WCS.

To illustrate, first I deleted all WCSes except for original WCS, then I run updatewcs(use_db=False), and then I run updatewcs(use_db=True) several times. Here is a summary of the WCSes in the header:

Note (Common part):

All WCSes have the following parameters in common so they will not be repeated:

CRPIX1  =               2048.0 / Pixel coordinate of reference point            
CRPIX2  =               1026.0 / Pixel coordinate of reference point            
CUNIT1  = 'deg'                / Units of coordinate increment and value        
CUNIT2  = 'deg'                / Units of coordinate increment and value        
CTYPE1  = 'RA---TAN-SIP'       / TAN (gnomonic) projection + SIP distortions    
CTYPE2  = 'DEC--TAN-SIP'       / TAN (gnomonic) projection + SIP distortions    

Run 0 (updatewcs(use_db=False)):

WCSNAME = 'IDC_2731450pi'      / Coordinate system title                        
CRVAL1  =       83.19300506082 / [deg] Coordinate value at reference point      
CRVAL2  =   -67.73222548108998 / [deg] Coordinate value at reference point      
CD1_1   = 3.92029060251604E-06 / partial of first axis coordinate w.r.t. x      
CD1_2   = -1.0082434475667E-05 / partial of first axis coordinate w.r.t. y      
CD2_1   = -1.0351930440257E-05 / partial of second axis coordinate w.r.t. x     
CD2_2   = -4.5644540153863E-06 / partial of second axis coordinate w.r.t. y     

Run 1 (updatewcs(use_db=True)):

WCSNAMEA= 'IDC_2731450pi'                                                       
CRVAL1A =       83.19300506082 / [deg] Coordinate value at reference point      
CRVAL2A =      -67.73222548109 / [deg] Coordinate value at reference point      
LONPOLEA=                180.0 / [deg] Native longitude of celestial pole       
LATPOLEA=      -67.73222548109 / [deg] Native latitude of celestial pole        
CD1_1A  =   3.920290602516E-06 / partial of first axis coordinate w.r.t. x      
CD1_2A  = -1.0082434475667E-05 / partial of first axis coordinate w.r.t. y      
CD2_1A  = -1.0351930440257E-05 / partial of second axis coordinate w.r.t. x     
CD2_2A  = -4.5644540153863E-06 / partial of second axis coordinate w.r.t. y     

WCSNAME = 'IDC_2731450pi-FIT_REL_GAIADR2'                                       
CRVAL1  =      83.192882353876 / [deg] Coordinate value at reference point      
CRVAL2  =     -67.732242935857 / [deg] Coordinate value at reference point      
LONPOLE =                180.0 / [deg] Native longitude of celestial pole       
LATPOLE =     -67.732242935857 / [deg] Native latitude of celestial pole        
CD1_1   =  3.9224079009662E-06 / partial of first axis coordinate w.r.t. x      
CD1_2   = -1.0086154113768E-05 / partial of first axis coordinate w.r.t. y      
CD2_1   = -1.0362353698318E-05 / partial of second axis coordinate w.r.t. x     
CD2_2   = -4.5651680786243E-06 / partial of second axis coordinate w.r.t. y     

WCS 'A' has (almost) the same values as primary WCS after updatewcs(use_db=False) except that some values seem to be rounded (I wish this would not happen). So, everything appears to be OK.

Run 2 (updatewcs(use_db=True)):

WCSNAMEA= 'IDC_2731450pi'                                                       
CRVAL1A =       83.19300506082 / [deg] Coordinate value at reference point      
CRVAL2A =      -67.73222548109 / [deg] Coordinate value at reference point      
LONPOLEA=                180.0 / [deg] Native longitude of celestial pole       
LATPOLEA=      -67.73222548109 / [deg] Native latitude of celestial pole        
CD1_1A  =   3.920290602516E-06 / partial of first axis coordinate w.r.t. x      
CD1_2A  = -1.0082434475667E-05 / partial of first axis coordinate w.r.t. y      
CD2_1A  = -1.0351930440257E-05 / partial of second axis coordinate w.r.t. x     
CD2_2A  = -4.5644540153863E-06 / partial of second axis coordinate w.r.t. y     

WCSNAMEB= 'ie6d07ujq_flt_IDC_2731450pi-FIT_REL_GAIADR2'                         
CRVAL1B =       83.19300506082 / [deg] Coordinate value at reference point      
CRVAL2B =      -67.73222548109 / [deg] Coordinate value at reference point      
LONPOLEB=                180.0 / [deg] Native longitude of celestial pole       
LATPOLEB=      -67.73222548109 / [deg] Native latitude of celestial pole        
CD1_1B  =   3.920290602516E-06 / partial of first axis coordinate w.r.t. x      
CD1_2B  = -1.0082434475667E-05 / partial of first axis coordinate w.r.t. y      
CD2_1B  = -1.0351930440257E-05 / partial of second axis coordinate w.r.t. x     
CD2_2B  = -4.5644540153863E-06 / partial of second axis coordinate w.r.t. y     

WCSNAME = 'IDC_2731450pi-FIT_REL_GAIADR2'                                       
CRVAL1  =      83.192882353876 / [deg] Coordinate value at reference point      
CRVAL2  =     -67.732242935857 / [deg] Coordinate value at reference point      
LONPOLE =                180.0 / [deg] Native longitude of celestial pole       
LATPOLE =     -67.732242935857 / [deg] Native latitude of celestial pole        
CD1_1   =  3.9224079009662E-06 / partial of first axis coordinate w.r.t. x      
CD1_2   = -1.0086154113768E-05 / partial of first axis coordinate w.r.t. y      
CD2_1   = -1.0362353698318E-05 / partial of second axis coordinate w.r.t. x     
CD2_2   = -4.5651680786243E-06 / partial of second axis coordinate w.r.t. y     

WCS 'B' is a copy of the WCS 'A' instead of the primary WCS. Subsequent runs of updatewcs(use_db=True) keep making copies of WCS 'B' instead of the primary WCS.

First of all, I think there should be detection if a WCS was already archived, no new copies should be created. Secondly, it seems that the wrong WCS is archived: instead of archiving the primary WCS, updatewcs archives an alternate WCS! This does not seem right.

CC: @stsci-hack @nden

A user reported and I confirmed that the code records the zero-point TDD coefficients in the header, not the coefficients which were computed for the observation date and used in the correction. @stsci-hack Was this intentional? I'd assume the actual correction should be recorded.

Note that the code correctly computes and applies the TDD correction.

Some change after commit b7c7aaa has resulted in the pipeline software 'runastrodriz' to crash with the following traceback:

WARNING: wcs_from_key: Could not read WCS with key A [stwcs.wcsutil.altwcs]
##############################################################################
#                                                                            #
# ERROR:                                                                     #
# AstroDrizzle Version svm0.3.1.dev9+g8b7ab5a5 encountered a problem!        #
# Processing terminated at 13:49:08.882 (02/07/2020).                        #
#                                                                            #
##############################################################################
Traceback (most recent call last):
  File "/user/hack/dev/release/stsci_python/pypi/stwcs/git/stwcs/stwcs/wcsutil/altwcs.py", line 525, in wcs_from_key
    w = pywcs.WCS(hdr, fobj=fobj, key=from_key)
  File "/user/hack/dev/miniconda3/envs/rthstdev/lib/python3.6/site-packages/astropy/wcs/wcs.py", line 449, in __init__
    header, fobj, dist='CPDIS', err=minerr)
  File "/user/hack/dev/miniconda3/envs/rthstdev/lib/python3.6/site-packages/astropy/wcs/wcs.py", line 946, in _read_distortion_kw
    d_data = fobj['WCSDVARR', d_extver].data
  File "/user/hack/dev/miniconda3/envs/rthstdev/lib/python3.6/site-packages/astropy/io/fits/hdu/hdulist.py", line 307, in __getitem__
    self._positive_index_of(key))
  File "/user/hack/dev/miniconda3/envs/rthstdev/lib/python3.6/site-packages/astropy/io/fits/hdu/hdulist.py", line 761, in _positive_index_of
    index = self.index_of(key)
  File "/user/hack/dev/miniconda3/envs/rthstdev/lib/python3.6/site-packages/astropy/io/fits/hdu/hdulist.py", line 742, in index_of
    raise KeyError(f'Extension {key!r} not found.')
KeyError: "Extension ('WCSDVARR', 1.0) not found."

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/user/hack/dev/release/stsci_python/pypi/drizzlepac/git/drizzlepac/drizzlepac/util.py", line 225, in wrapper
    result = func(*args, **kwargs)
  File "/user/hack/dev/release/stsci_python/pypi/drizzlepac/git/drizzlepac/drizzlepac/astrodrizzle.py", line 219, in run
    procSteps=procSteps)
  File "/user/hack/dev/release/stsci_python/pypi/drizzlepac/git/drizzlepac/drizzlepac/ablot.py", line 186, in runBlot
    wcsmap=wcsmap)
  File "/user/hack/dev/release/stsci_python/pypi/drizzlepac/git/drizzlepac/drizzlepac/ablot.py", line 318, in run_blot
    virtual=img.inmemory)
  File "/user/hack/dev/release/stsci_python/pypi/drizzlepac/git/drizzlepac/drizzlepac/outputimage.py", line 455, in writeFITS
    wcs_functions.removeAllAltWCS(fo,wcs_ext)
  File "/user/hack/dev/release/stsci_python/pypi/drizzlepac/git/drizzlepac/drizzlepac/wcs_functions.py", line 497, in removeAllAltWCS
    altwcs.deleteWCS(hdulist,extn,wkey)
  File "/user/hack/dev/release/stsci_python/pypi/stwcs/git/stwcs/stwcs/wcsutil/altwcs.py", line 358, in deleteWCS
    hwcs = wcs_from_key(fobj, i, from_key=wkey)
  File "/user/hack/dev/release/stsci_python/pypi/stwcs/git/stwcs/stwcs/wcsutil/altwcs.py", line 528, in wcs_from_key
    log.warning(f'              Skipping {fobj.filename():s}[{ext}]')
TypeError: unsupported format string passed to NoneType.__format__
ERROR: Could not complete astrodrizzle processing of j8c107010_pipeline_asn.fits.
unsupported format string passed to NoneType.__format__
ERROR: Cannot run astrodrizzle on -id.
Traceback (most recent call last):
  File "/user/hack/dev/release/stsci_python/pypi/stwcs/git/stwcs/stwcs/wcsutil/altwcs.py", line 525, in wcs_from_key
    w = pywcs.WCS(hdr, fobj=fobj, key=from_key)
  File "/user/hack/dev/miniconda3/envs/rthstdev/lib/python3.6/site-packages/astropy/wcs/wcs.py", line 449, in __init__
    header, fobj, dist='CPDIS', err=minerr)
  File "/user/hack/dev/miniconda3/envs/rthstdev/lib/python3.6/site-packages/astropy/wcs/wcs.py", line 946, in _read_distortion_kw
    d_data = fobj['WCSDVARR', d_extver].data
  File "/user/hack/dev/miniconda3/envs/rthstdev/lib/python3.6/site-packages/astropy/io/fits/hdu/hdulist.py", line 307, in __getitem__
    self._positive_index_of(key))
  File "/user/hack/dev/miniconda3/envs/rthstdev/lib/python3.6/site-packages/astropy/io/fits/hdu/hdulist.py", line 761, in _positive_index_of
    index = self.index_of(key)
  File "/user/hack/dev/miniconda3/envs/rthstdev/lib/python3.6/site-packages/astropy/io/fits/hdu/hdulist.py", line 742, in index_of
    raise KeyError(f'Extension {key!r} not found.')
KeyError: "Extension ('WCSDVARR', 1.0) not found."

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/user/hack/dev/release/stsci_python/pypi/drizzlepac/git/drizzlepac/drizzlepac/runastrodriz.py", line 716, in run_driz
    **pipeline_pars)
  File "/user/hack/dev/release/stsci_python/pypi/drizzlepac/git/drizzlepac/drizzlepac/astrodrizzle.py", line 118, in AstroDrizzle
    run(configObj, wcsmap=wcsmap)
  File "/user/hack/dev/release/stsci_python/pypi/drizzlepac/git/drizzlepac/drizzlepac/util.py", line 239, in wrapper
    raise errorobj
  File "/user/hack/dev/release/stsci_python/pypi/drizzlepac/git/drizzlepac/drizzlepac/util.py", line 225, in wrapper
    result = func(*args, **kwargs)
  File "/user/hack/dev/release/stsci_python/pypi/drizzlepac/git/drizzlepac/drizzlepac/astrodrizzle.py", line 219, in run
    procSteps=procSteps)
  File "/user/hack/dev/release/stsci_python/pypi/drizzlepac/git/drizzlepac/drizzlepac/ablot.py", line 186, in runBlot
    wcsmap=wcsmap)
  File "/user/hack/dev/release/stsci_python/pypi/drizzlepac/git/drizzlepac/drizzlepac/ablot.py", line 318, in run_blot
    virtual=img.inmemory)
  File "/user/hack/dev/release/stsci_python/pypi/drizzlepac/git/drizzlepac/drizzlepac/outputimage.py", line 455, in writeFITS
    wcs_functions.removeAllAltWCS(fo,wcs_ext)
  File "/user/hack/dev/release/stsci_python/pypi/drizzlepac/git/drizzlepac/drizzlepac/wcs_functions.py", line 497, in removeAllAltWCS
    altwcs.deleteWCS(hdulist,extn,wkey)
  File "/user/hack/dev/release/stsci_python/pypi/stwcs/git/stwcs/stwcs/wcsutil/altwcs.py", line 358, in deleteWCS
    hwcs = wcs_from_key(fobj, i, from_key=wkey)
  File "/user/hack/dev/release/stsci_python/pypi/stwcs/git/stwcs/stwcs/wcsutil/altwcs.py", line 528, in wcs_from_key
    log.warning(f'              Skipping {fobj.filename():s}[{ext}]')
TypeError: unsupported format string passed to NoneType.__format__

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/user/hack/dev/release/stsci_python/pypi/drizzlepac/git/drizzlepac/drizzlepac/runastrodriz.py", line 1356, in main
    align_to_gaia=align_to_gaia, force_alignment=force_alignment, debug=debug)
  File "/user/hack/dev/release/stsci_python/pypi/drizzlepac/git/drizzlepac/drizzlepac/runastrodriz.py", line 438, in process
    find_crs=True, **adriz_pars)
  File "/user/hack/dev/release/stsci_python/pypi/drizzlepac/git/drizzlepac/drizzlepac/runastrodriz.py", line 942, in verify_alignment
    good_bits=good_bits, **pipeline_pars)
  File "/user/hack/dev/release/stsci_python/pypi/drizzlepac/git/drizzlepac/drizzlepac/runastrodriz.py", line 729, in run_driz
    raise Exception(str(errorobj))
Exception: unsupported format string passed to NoneType.__format__

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/user/hack/dev/miniconda3/envs/rthstdev/bin/runastrodriz", line 11, in <module>
    load_entry_point('drizzlepac', 'console_scripts', 'runastrodriz')()
  File "/user/hack/dev/release/stsci_python/pypi/drizzlepac/git/drizzlepac/drizzlepac/runastrodriz.py", line 1361, in main
    raise Exception(str(errorobj))
Exception: unsupported format string passed to NoneType.__format__

This problem can be reproduced with the following commands in an ipython session:

from drizzlepac.haputils import astroquery_utils as aqutils
files = aqutils.retrieve_observation(['j8c107010'], suffix=['FLT', 'FLC', 'RAW', 'ASN'], clobber=True) 
!runastrodriz -id j8c107010_asn.fits

Updating the file with apriori astrometry solutions from the astrometry database currently will not overwrite solutions with the same WCSNAME. We have already encountered situations where those solutions were incorrect or otherwise corrupted. This led to either invalid, incomplete or spurious WCS solutions to be applied to data. Efforts are underway to prevent this as much as possible in the database. However, due to the rule to not overwrite a solution with the same name, 'updatewcs' with 'use_db=True' can NOT be used to clean up these solutions they are corrected in the database since the new correct solutions will have the same WCSNAME as the corrupted ones already written to the file.

The logic needs to be changed to allow a user to 'force' an update or 'force' the removal of all solutions from the astrometry database so that spurious solutions can be replaced by corrected solutions.

The reprocessing pipeline would benefit, when applying new astrometric solutions, from minimizing the amount of file I/O is required when processing the files. In particular, populating the astrometric database only requires being able to read in data from the online cache, then using that in-memory to develop the headerlets which get stored in the database.

Therefore, it has been requested that the updatewcs step be modified to support FITS HDUList objects directly as inputs.

I see a doc sub-directory but there is no link to rendered HTML doc.

Oh, wait... I see there is a http://stwcs.readthedocs.io but no badge. Just adding a badge should be enough.

When running AstroDrizzle, it crashes crashes at line 353 of altwcs.py saying it can't find keyword 'MJD-OBS', which is not a header keyword, but I believe one of the WCS keywords.

Starting in line 347 in altwcs.py:

    for i in ext:
        hdr = fobj[i].header
        hwcs = readAltWCS(fobj,i,wcskey=wkey)
        if hwcs is None:
            continue
        for k in hwcs:
            del hdr[k]

Drizzle runs fine if those lines are changed to:

    for i in ext:
        hdr = fobj[i].header
        hwcs = readAltWCS(fobj,i,wcskey=wkey)
        if hwcs is None:
            continue
        for k in hwcs.keys():
            del hwcs[k]

EDIT: After looking at the code more, I see that this does not 'fix' the behavior, rather just avoids a currently broken behavior.

It seems that hwcs somehow gets extra keys that are not part of the hdr object. Perhaps readAltWCS is getting more keywords somehow?

A compatibility problem with Python 3 as well as the solution has been reported by @cmccully: in Python 3 string.uppercase needs to be replaced with string.ascii_uppercase.

Code

is never really executed unless one of WCS keywords starts with 'O' and has length 2 and it is the last keyword in the previous loop. This is because key is a loop variable in the loop before this code.

What I do not know is what would be the consequences of this bug. Maybe it is harmless.

CC: @nden

The clobber parameter in astropy.io.fits write functions has been deprecated and it has been replaced with overwrite: see astropy/astropy#5171.

This has caused our code (see, e.g., drizzlepac - spacetelescope/drizzlepac#15) to issue numerous deprecation warnings.

The code in stwcs needs to be updated so that it uses overwrite for versions of astropy >= 1.3 and clobber for verions < 1.3 (for backward compatibility).

We keep getting help tickets describing errors when running updatewcs.

updatewcs.updatewcs('*flc.fits')
TypeError: Can't convert 'NoneType' object to str implicitly

To solve this, we are telling users to set a new parameter which is not documented in any of our drizzling examples:

updatewcs.updatewcs('*flc.fits', use_db=False)

Can we simply change the default to False so that people trying to reprocess their data don't keep tripping over this?

Mihai pointed me to some open isseus:
#38
#40

He suggests: 'Maybe the code should be fixed to automatically ignore astrometry_db errors and proceed as if 'use_db=False' ?

RADESYS is set to ICRS or FK5 based on the value of REFFRAME.
However the APT template allows also a value of OTHER and in small number of observations this is chosen.
What should the value of RADESYS be in this case? Given the lack of information we could:

• not set RADESYS and let astropy.wcs decide what to set it to.
• set it to ICRS (which is not going to be more incorrect than OTHER).

@stsci-hack ?

Currently stwcs code may write header keywords containing trailing space character, particularly for the primary WCS. For example, see

This is a problem since, in principle, space character is not permitted in FITS keyword names - see section 4.1.2.1 of the FITS standard:

The keyword name shall be a left justified, 8-character, space-filled, ASCII string with no embedded spaces. All digits 0 through 9 (decimal ASCII codes 48 to 57, or hexadecimal 30
to 39) and upper case Latin alphabetic characters 'A' through 'Z' (decimal 65 to 90 or hexadecimal 41 to 5A) are permitted; lower case characters shall not be used. The underscore ('_', decimal 95 or hexadecimal 5F) and hyphen ('-', decimal 45 or hexadecimal 2D) are also permitted. No other characters are permitted.

This may result in multiple versions of, i.e., CD1_1 keywords being stored in the FITS header - see astropy/astropy#10479

wcsutil.altwcs.wcskeys returns a list of keys identifying alternate WCSs in the header. To find those it looks at WCSNAME keysowrds. It should also look at CTYPE and possibly other WCS keywords.

This module defines a FileHandler for the logging in the current working directory for the user when this module first gets imported. If that directory is removed later by the user, it will cause an Exception when performing headerlet operations later in another directory.

By itself, this is a relatively rare occurrence during use of this code. However, this code relies on use of a decorator which may not be necessary given what Python's logging module can do.

The WFC3 team does not want to have the astrometric updates pushed into the pipeline without accompanying documentation. Since the a posteriori astrometry isn't going live with the build, this probably doesn't need to be anything exhaustive. It should, however show users how to switch between solutions, or at the very least, just restore the regular solution (the IDC_XXXXXXXX solution). It should probably describe (or link to) how the a priori solution is different than the current solution. A brief description of headerlets (the purpose of them, rather than the technical details) would likely be helpful as well.

@rjavila Does the ACS team have any thoughts on this?

@stsci-hack and I discovered some bugs on how .travis.yml is configured.

1. Do not use python directive, instead use PYTHON_VERSION. See astropy file as an example.
2. One of the job should be using ASTROPY_VERSION=stable to avoid breaking backward compatibility in your package.
3. Use ASTROPY_VERSION=development to pick up the dev version. There is no need to hardcode the git command.

Because no parameters are passed to the initialization, a log file is not written out.
One way to do this is to check the verbose parameter to updatewcs.updatewcs and if not False initialize AstrometryDB(write_log=True).

Saw this in regression test, please fix. @mcara or @nden ?

https://ssb.stsci.edu/steuermann/steuermann.cgi?action=log&name=conda_tests_2017-07-30-10:30:02/nott:conda_testing/dev_tests

run_proc: pdkrun --dir --environment_already_set /srv/rt/stsci_python/stwcs 
directory /srv/rt/stsci_python/stwcs
COMMAND : 'pdknose --pdk --with-doctest --doctest-tests test.py' (for file /srv/rt/stsci_python/stwcs/test.py) 2017-07-30 10:37:00.812892
WARNING: VerifyWarning: Card is too long, comment will be truncated. [astropy.io.fits.card]
WARNING: AstropyDeprecationWarning: The use of ``AXISCORR`` for D2IM correction has been deprecated.`~astropy.wcs` will read in files with ``AXISCORR`` but ``to_fits()`` will write out files without it. [astropy.wcs.wcs]
WARNING: FITSFixedWarning: CPERROR1= 0.0 / Maximum error of dgeo correction for axis 1 
keyword looks very much like CPERRja but isn't. [astropy.wcs.wcs]
WARNING: FITSFixedWarning: CPERROR2= 0.0 / Maximum error of dgeo correction for axis 2 
keyword looks very much like CPERRja but isn't. [astropy.wcs.wcs]
..../srv/rt/stsci_python/stwcs/test.py:108: RuntimeWarning: cdelt will be ignored since cd is present
  utils.assert_almost_equal(self.w1.wcs.cdelt, self.w1r.wcs.cdelt)
/srv/rt/stsci_python/stwcs/test.py:115: RuntimeWarning: cdelt will be ignored since cd is present
  utils.assert_almost_equal(self.w4.wcs.cdelt, self.w4r.wcs.cdelt)

If relic becomes a submodule, there is no need to do git clone everytime you run setup.

In pc2cd() function in hstwcs.py, conversion from PC to CD matrix is performed by multiplying PC matrix by cdelt[1], which is not what should be done mathematically:

What is the reason for this?

CC: @nden @stsci-hack

In distortion/utils.py, the function make_orthogonal_cd()...

The following lines check if wcs.idcscale exists, but does not check if the value is set to None:

    124         # find pixel scale:
    125         if hasattr(wcs, 'idcscale'):
--> 126             scale = (wcs.idcscale) / 3600.  # HST pixel scale provided
    127         else:
    128             scale = np.sqrt(np.abs(det))  # find as sqrt(pixel area)

Proposed solution to add a conditional, similar to the following wfc3 code:

    636     if hasattr(wcs, 'idcscale'):
    637         if wcs.idcscale is None:
    638                 wcs.idcscale = np.sqrt(np.sum(wcs.wcs.cd[0,:]**2))*3600.
    639     else:
    640         wcs.idcscale = np.sqrt(np.sum(wcs.wcs.cd[0,:]**2))*3600.

hello, the stack trace below highlights an error when importing stwcs.

>>> import process_main
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "./process_main.py", line 31, in <module>
    from stwcs.wcsutil import headerlet, altwcs
  File "./wcsutil/headerlet.py", line 30, in <module>
    from stwcs.updatewcs import utils
  File "./stwcs/updatewcs/__init__.py", line 8, in <module>
    from .. import __version__
ImportError: cannot import name '__version__' from 'stwcs' (./stwcs/__init__.py)

The hangup happens when __version__ isn't defined when its passed in the try/catch block
https://github.com/spacetelescope/stwcs/blob/master/stwcs/__init__.py#L26

It looks like this is intentional logic, but I'm uncertain how to move forward with limited error output. Which package should be installed?

Would have probably caught undeclared variable in #114.

Just installed the most recent astroconda build. This includes stwcs version 1.2.5, running with python 2.7.13. In the python terminal I try to run the following:

     import stwcs
    from stwcs import updatewcs
    updatewcs.updatewcs(['fltname.fits'])

I get the following error:

LookupError: 
Problem finding row in IDCTAB! Could not find row matching:
        CHIP: 1
     FILTERS: F140W 

I have checked that my $iref and $jref values are set correctly and that the IDCTAB table listed in my flt header is in the $iref directory.

Note that this problem does not occur when running the version with my old ureka installation, which had stwcs Version: 1.2.1.dev39127. With this older version I get the following output after running the same code as above:
- IDCTAB: Distortion model from row 4 for chip 1 : F140W
- IDCTAB: Distortion model from row 4 for chip 1 : F140W
['fltname.fits']

Which I assume means that everything worked.

Currently the flow in updatewcs is:

• run updatewcs and populated the WCS based on eference files and distortion
• check for better WCS solutions for this file in the apriori astrometry database
  • if available - update the WCS with it, saving the computed one as an alternate WCS.
  • if not available attempt to injst the computed WCS in the astrometry database.

Consider splitting the functionality and not updating the database by default.

convertAltWCS either pre-#132 or post-#138 incorrectly handles the case when a WCS contains header keywords that are not AltWCS-specific. For example MJDREF is supposed to be a single keyword in the header "shared" by all WCSes in the header (primary and Alt). These keywords should not be "archived" or "converted" by convertAltWCS or archiveWCS.

The original code (pre-#132) -

NOTE: This is not limited to MJDREF(I,F). Looking at https://github.com/astropy/astropy/blob/master/cextern/wcslib/C/wcshdr.c we could identify many other keywords that are not supposed to depend on the "key", such as 'TIMESYS', 'TIMEUNIT', 'PLEPHEM', 'DATEREF', etc.

CC: @nden

There is a very strange line of code in stwcs.wcsutil.altwcs.py:

It is a loop that loops only over the last element:

>>> for k in ['a', 'b', 'c'][-1::]:
>>>    print(k)
c

CC: @stsci-hack @nden

If I run updatewcs() more than once on a WFC3 FLT time, things get really confused: The first time it runs, if alters the WCS solution in a way that I can understand. But if you run updatewcs a second time, you get the same result as if not running it.

Here is an example:
Original 500 500
ra,dec and back to x,y in original: 500.00000000027416 499.9999999998585
Running updatewcs
ra,dec and back to x,y in updatewcs(orginal): 498.6298533528224 499.43186222970786
Running updatewcs
ra,dec and back to x,y in updatewcs(updatewcs(orginal)): 499.9999999994957 499.9999999953178

The setup file has an install_requires section which lists numpy, astropy and stsci.tools. Technically these are not installation requirements. Is there a reason they are listed here and can they be removed?
@jhunkeler

Use of the restore_from_headerlet() function results in an Exception being thrown as it tries to identify what extension needs to be updated in the science file. The exact call and Exception is given below.

This error needs to be corrected in order to allow users to successfully choose what WCS they want to use after updating with astrometrically-corrected solutions from the astrometry database (for example).

`In [57]: headerlet.restore_from_headerlet(m35names[0], hdrname='ic4001vjq_IDC_yas1621ai-GSC-GAIAdr1', force=True)

KeyError Traceback (most recent call last)
in ()
----> 1 headerlet.restore_from_headerlet(m35names[0], hdrname='ic4001vjq_IDC_yas1621ai-GSC-GAIAdr1', force=True)

/user/hack/dev/release/anaconda3/envs/hstrt-2018.1a/lib/python3.6/site-packages/stwcs/wcsutil/headerlet.py in wrapped(*args, **kw)
128
129 init_logging(func.name, level, mode, **func_args)
--> 130 return func(*args, **kw)
131 return wrapped
132

/user/hack/dev/release/anaconda3/envs/hstrt-2018.1a/lib/python3.6/site-packages/stwcs/wcsutil/headerlet.py in restore_from_headerlet(filename, hdrname, hdrext, archive, force, logging, logmode)
1485 if 'extname' in ext.header and ext.header['extname'] == 'SIPWCS':
1486 # convert from string to tuple or int
-> 1487 sciext = eval(ext.header['sciext'])
1488 extlist.append(fobj[sciext])
1489 # determine whether distortion is the same

/user/hack/dev/release/anaconda3/envs/hstrt-2018.1a/lib/python3.6/site-packages/astropy/io/fits/header.py in getitem(self, key)
143 else:
144 keyword = key
--> 145 card = self._cards[self._cardindex(key)]
146 if card.field_specifier is not None and keyword == card.rawkeyword:
147 # This is RVKC; if only the top-level keyword was specified return

/user/hack/dev/release/anaconda3/envs/hstrt-2018.1a/lib/python3.6/site-packages/astropy/io/fits/header.py in _cardindex(self, key)
1652
1653 if not indices:
-> 1654 raise KeyError("Keyword {!r} not found.".format(keyword))
1655
1656 try:

KeyError: "Keyword 'SCIEXT' not found."
`

Looking at

Is this desired behavior or is it just acceptable or should we remove this code?

CC: @stsci-hack @nden