Git Product home page Git Product logo

rad's Introduction

Roman Attribute Dictionary

CI Documentation Status Weekly CI Powered by STScI Badge

This is a repository for the Roman Attribute Dictionary

Installation

The easiest way to install the latest rad release into a fresh virtualenv or conda environment is

pip install rad

Detailed Installation

The rad package can be installed into a virtualenv or conda environment via pip. We recommend that for each installation you start by creating a fresh environment that only has Python installed and then install the rad package and its dependencies into that bare environment. If using conda environments, first make sure you have a recent version of Anaconda or Miniconda installed. If desired, you can create multiple environments to allow for switching between different versions of the rad package (e.g. a released version versus the current development version).

In all cases, the installation is generally a 3-step process:

  • Create a conda environment
  • Activate that environment
  • Install the desired version of the rad package into that environment

Details are given below on how to do this for different types of installations, including tagged releases, DMS builds used in operations, and development versions. Remember that all conda operations must be done from within a bash shell.

Installing latest releases

You can install the latest released version via pip. From a bash shell:

conda create -n <env_name> python
conda activate <env_name>
pip install rad

Note
Alternatively, you can also use virtualenv to create an environment; however, this installation method is not supported by STScI if you encounter issues.

You can also install a specific version (from rad 0.1.0 onward):

conda create -n <env_name> python
conda activate <env_name>
pip install rad==0.5.0

Installing the development version from Github

You can install the latest development version (not as well tested) from the Github main branch:

conda create -n <env_name> python
conda activate <env_name>
pip install git+https://github.com/spacetelescope/rad

Installing for Developers

If you want to be able to work on and test the source code with the rad package, the high-level procedure to do this is to first create a conda environment using the same procedures outlined above, but then install your personal copy of the code overtop of the original code in that environment. Again, this should be done in a separate conda environment from any existing environments that you may have already installed with released versions of the rad package.

As usual, the first two steps are to create and activate an environment:

conda create -n <env_name> python
conda activate <env_name>

To install your own copy of the code into that environment, you first need to fork and clone the rad repo:

cd <where you want to put the repo>
git clone https://github.com/spacetelescope/rad
cd rad

Note
Installing via setup.py (python setup.py install, python setup.py develop, etc.) is deprecated and does not work.

Install from your local checked-out copy as an "editable" install:

pip install -e .

If you want to run the unit or regression tests and/or build the docs, you can make sure those dependencies are installed too:

pip install -e ".[test]"
pip install -e ".[docs]"
pip install -e ".[test,docs]"

Need other useful packages in your development environment?

pip install ipython pytest-xdist

rad's People

Stargazers

 avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

rad's Issues

flat schema specifies a float32 dq array

In rad, the schema defines dq like this:

dq: 
  tag: tag:stsci.edu:asdf/core/ndarray-1.0.0 
  datatype: float32
  ndim: 2

but in romancal:

dq:
  title: Data quality array
  default: 0
  datatype: uint16

reference files should not require most of the information in basic-1.0.0

Most of the information in this schema is irrelevant for reference files, and can be very misleading. CRDS context, for example, should not be in the file since it may be used in several contexts. Likewise for calibration_software_version, crds_software_version, prd_software_version. Telescope is already required in another schema. If file date and such are needed, they should go in the ref_common schema and basic-1.0.0 removed from ref_common.

Create L1 schema for guide windows

Issue RAD-48 was created on JIRA by Nadia Dencheva:

Create a L1 schema for guide windows.

  • metadata - TBD
  •  an array for the guide window pixels in the science array, these would be a cube for each full frame science image
  • an array for the Amp33 reference output pixels

 

 **Note if these changes are to be reflected in the archive, the archive_catalog section should be populated and that work will be done subsequent to this being completed.

Changes to schemas for reference pixel correction

Issue RAD-46 was created on JIRA by Nadia Dencheva:

Update Feb 7, 2022, INS and DMS agreed:

A Level1 file should contain 

  • the science array and the border reference pixels - 4096 x 4096 x nframes
  • an array for the Amp33  reference output pixels

A Level2 file should contain:

  • the science array only, the border reference pixels are trimmed- 2D array, 4088 x 4088
  • four 3D arrays, one for each of the border reference pixels rows/columns
  • four 2D DQ arrays with border pixels
  • 1 array for the amp33 reference pixels

RampModel

  • dq_init will initialize a RampModel as before 
  • dq_init will copy the reference pixels into 4 cubes, duplicating corners
  • dq_init will copy the border pixels from the pixeldq arrays into 4 2D arrays
  • RampModel will not change through the detector level steps
  • The border pixels will be cut at the end of ramp_fitting when an ImageModel (L2) file is created

 

Initial discussion and decision are documented at
https://innerspace.stsci.edu/display/SCSB/Data+models+support+for+refpix+step

 

Add attribute for compression in originating packets

Issue RAD-10 was created on JIRA by Megan Sosey:

The WFI packets will contain the compression flag that’s appropriate for the data, those are part of the L0 data, and not all L0 data will be compressed.

DMS will use the decompression software to decompress as appropriate, and the level 1 file we create will contain uncompressed data.

 We should carry through some metadata into the L1 file that says whether the packets that were used to construct it arrived compressed or not, in case it’s useful for downstream error analysis – for the case that folks looking at the level 1+ data are trying to diagnose a general issue. I would expect errors with the actual decompression to be addressed immediately.

**Note if these changes are to be reflected in the archive, the archive_catalog section should be populated and that work will be done subsequent to this being completed. 

Add Mode To Reference Schemas

Issue RAD-38 was created on JIRA by Paul Huwe:

The following reference files are viewing mode dependent and thus require the exposure type as a CRDS selector.

The schemas (list from INS):

Dark

Readnoise

 

Add Optical_Element to Reference File Schemas

Issue RAD-39 was created on JIRA by Paul Huwe:

The following reference files depend upon optical element, and thus require this keyword as a CRDS selector. Optical_element keyword will be added by importing optical_element from wfi_mode.

The schemas (as listed by INS):

Dark

Flat

Area

 

Update guide star attribute

Issue RAD-45 was created on JIRA by Megan Sosey:

https://github.com/spacetelescope/rad/blob/main/src/rad/resources/schemas/guidestar-1.0.0.yaml#L102

 The attribute linked above gs_pcs_mode should probably be changed to gs_fgs_mode

The guide windows and pointing on Roman are functionally different. The guide window modes that go with this would be:

  • WIM
  • WSM
  • DEFO_M
  • DEFO_L
  • CGI

see https://innerspace.stsci.edu/display/ROMAN/WFI+Guide+Windows  and the PRDS DFCD for more information. 

 

There may be multiple guide window specifications for the same mode. Standby modes are not transmitted to the ground

**Note if these changes are to be reflected in the archive, the archive_catalog section should be populated and that work will be done subsequent to this being completed.

Adding more information for clarity:

Roman guide mode is not stored in PSS, and not transmitted in a direct way to DMS, it has to be derived from other information we receive. 

The page linked above shows the different guide windows that are currently planned, those are the effective "modes" for roman, it also give more information about when full frame science data and guide window data are transmitted to the ground:

"WIM-ACQ, WSM-ACQ-1, and WSM-ACQ-2 will not transmit full-frame science arrays to the ground, just guide window data."

That means the derivable modes for full frame science data that we receive and would populate this field for the WFI images are:

  • WIM, WSM, DEFOCUSED_MODERATE, DEFOCUSED_LARGE

We haven't made schemas for the guide window files yet, but it's likely that the guide star schema would be included in them, which means it might make sense to include a more complete enumerated list of possibilities since we will get guide window data down without science frame data:

  • WIM-ACQ, WIM-TRACK, WSM-ACQ-1, WSM-ACQ-2, WSM-TRACK, DEFOCUSED-MODERATE, DEFOCUSED-LARGE, CGI

 

CGI -> Not sure if this is needed, when CGI data is being taken, we should get WIM-TRACK guide windows down, no decision on parallel full-frame WFI exposures at the same time. So WIM-TRACK can go into the guide window metadata, likely another metadata for prime SI could be included

 

 

Change archive_catalog.datatype bit fields to nchar(1)

Issue RAD-42 was created on JIRA by Lisa Gardner:

There are 3 fields defined in archive_catalog.datatype as bit.  These are coming in the asdf files as true and false.  Instead of adding extra code to the Ingest software we are going to treat them like JWST does which is just take the first letter t/f.  We do that by having the stored procedure input parameter defined as nchar(1).

The 3 fields are:

exposure-1.0.0.yaml: **data_problem
visit-1.0.0.yaml: target_of_opportunity
visit-1.0.0.yaml: internal_target

Probably have to change rad_schema-1.0.0.yaml to include nchar.  Ask Paul.

Corrections to core schema

After discussing with @ddavis-stsci, @PaulHuwe, and Andrea Bellini today, we found some corrections to the core schema that should be implemented. These are mostly just clarifications to the metadata descriptions:

  • exposure:
    • start_time, mid_time, end_time should be of data type "string"
  • guidestar:
    • Add units to descriptions of gs_ra, gs_dec, etc. (should be decimal degrees)
    • Remove all references to FGS in descriptions.
  • observation:
    • Add examples to descriptions of _number keys (e.g., program_number 'PPPPP')

datamodels-1.0 needs a real extension URI

Currently we only have this in datamodels-1.0.yaml:

extension_uri: http://stsci.edu/datamodels/extensions/

This is the URI that will be included in the ASDF file metadata to show that this extension was used when writing the file.

Manifest URI does not match id field in the YAML

The datamodels-1.0.yaml file is being registered with asdf at this URI:

http://stsci.edu/asdf/datamodels/roman/manifests/datamodels-1.0

but the document's id field has a different URI:

http://stsci.edu/datamodels/manifests/datamodels-1.0

Invalid $ref in wfi_image-1.0.0

The $ref to pixelarea-1.0.0 in wfi_image-1.0.0 is invalid syntax. I'm not sure what the best solution is, it depends on whether we want pixelarea to be its owned tagged object or if we really just want an array named "area" in the image schema.

Add Remaining Calsteps to Required

Issue RAD-43 was created on JIRA by Paul Huwe:

Only two of the cal_steps are listed in the required list at present. This list should be updated to include them all [assign_wcs, flat_field, dark, dq_init, jump, linearity, ramp_fit, saturation].

**Note if these changes are to be reflected in the archive, the archive_catalog section should be populated and that work will be done subsequent to this being completed.

Add dq_def File

Needs to be implemented once table styles are decided upon.

Decide On Table Types

Decide on the table type support needed for RAD schema yamls (dq_def, group, ref/wfi_img_photom, etc.). Astropy tables aren't presently supported, but may be expected downstream, so some R&D and decision malign required.

Astropy implementation example (from group):
group:
title: Group parameters table
tag: tag:astropy.org:astropy/table/table-1.0.0
colnames: [ integration_number, group_number, end_day, end_milliseconds, end_submilliseconds,
group_end_time, number_of_columns, number_of_rows, number_of_gaps,
completion_code_number, completion_code_text, bary_end_time, helio_end_time ]
columns:
- name: integration_number
datatype: int16
- name: group_number
datatype: int16
- name: end_day
datatype: int16
- name: end_milliseconds
datatype: int32
- name: end_submilliseconds
datatype: int16
- name: group_end_time
datatype: [ ascii, 26 ]
- name: number_of_columns
datatype: int16
- name: number_of_rows
datatype: int16
- name: number_of_gaps
datatype: int16
- name: completion_code_number
datatype: int16
- name: completion_code_text
datatype: [ ascii, 36 ]
- name: bary_end_time
datatype: float64
- name: helio_end_time
datatype: float64

Add p_keywords to readnoise and dark

Issue RAD-44 was created on JIRA by Nadia Dencheva:

Add 
meta.exposure.p_exptype
to dark and readnoise schemas to enable automatic rmap generation.

**Note if these changes are to be reflected in the archive, the archive_catalog section should be populated and that work will be done subsequent to this being completed.

Add style test of array spacing

Add a test that requires schema arrays to be formatted like this:

some_array: [some, array, values]

instead of this:

some_array: [ some, array, values ]

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.