rbeyer / hiproc Goto Github PK

• hiproc version: 0.9.0 (current head)
• Python version: 3.8
• Operating System: ubuntu

Description

I updated Hiproc to v 0.9.0 recently and attempted to run 'PSP_002930_1800' through the pipeline hiproc -vv -j ./*.IMG.

but hiproc just seems to stop without issuing any warning or errors that I can see right after HiCal and does not attempt to run any later streps like HiStitch.

• pyrise version: latest main branch
• Python version: 3.9
• Operating System: ubuntu

Description

I found that for one image that hiproc would fail somewhere in the pipeline for an image ESP_065856_1890 on a KeyError for 'Average' but it wasn't clear with verbose logging which function/area the pipeline was breaking. After some trial and error running things I found that EDR_Stats was failing on a single ccd channel (in this case ESP_065856_1890_RED2_1) because of a missing json file.

Diving deeper in I found the error was specifically with the "CAL_BUFFER" return for this ccd, where there is no average value/other statistical data.

Below is the output for the bad ccd:

PVLGroup([
  ('TotalPixels', 372)
  ('ValidPixels', 0)
  ('NullPixels', 0)
  ('LisPixels', 372)
  ('LrsPixels', 0)
  ('HisPixels', 0)
  ('HrsPixels', 0)
])

and the next is the "CAL_BUFFER" for ESP_065856_1890_RED2_0:

PVLGroup([
  ('Average', 1075.9247311828)
  ('StandardDeviation', 22.093155656939)
  ('Variance', 488.10752688172)
  ('Minimum', 1035.0)
  ('Maximum', 1104.0)
  ('TotalPixels', 372)
  ('ValidPixels', 372)
  ('NullPixels', 0)
  ('LisPixels', 0)
  ('LrsPixels', 0)
  ('HisPixels', 0)
  ('HrsPixels', 0)
])

There was nothing in the stderr for the histat kalasiris function, so I am not sure what went wrong, maybe something internal to ISIS?

As a longer term suggestion, it would be helpful if verbose logging was more verbose in terms of saying which step of the pipeline it is in, which image is being operated on etc.

What I Did

note: requires my moody tool to download the edr files, simply pip install moody

moody hirise_edr ESP_065856_1890
EDR_Stats ./*.IMG -v -k

• pyrise version: current master
• Python version: 3.9.5
• Operating System: ubuntu

Description

When testing the full hiproc pipeline with the hiproc command with hijack force enabled, two empty (0B) files 'sys' and 're' are created by hiproc. I am not sure where these files are being created as nothing obvious in the code is making them and they share the names of parts of the python standard library so it's not clear if they are some intentional intermediate product that keep wants to retain, all I know is they were created near the final 3 minutes of the pipeline so that points to one of the later processing steps being responsible.

What I Did

hiproc -k -p --logfile hiproc.log ./ESP_016657_1890*.IMG

• pyrise version:
• Python version: 3.9.5
• Operating System: Centos7

Description

Installation via conda-forge is not installing the data directory. Perhaps because it does not have an __init__.py file and/or it is not specified in the setup.py.

What I Did

conda create -n hiproc
conda activate hiproc
mamba install -c conda-forge hiproc
condo activate --stack isis5.0.0
hiproc

> `FileNotFoundError: [Errno 2] No such file or directory: '/home/jlaura/anaconda3/envs/hiproc/lib/python3.9/site-packages/hiproc/data/HiJitReg.conf'`

• hiproc version: 0.10.0
• Python version: 3.9.6
• Operating System: CentOS

Description

Seeing differences in the ephemeris information between UA dejittered, noproj data, and hiproc dejittered noproj data. Seeking input on the differences.

What I Did

Ran the full pipeline on ESP_018957_2655

using: hiproc --keep --verbose --precision --max_workers 8 "$@"

This ran HiJACK and generated a RED NOPROJ mosaic as expected (awesome). I then compared a UA dejittered, NOPROJ RED cube and the hiproc generated cube, looking at the ephemeris information.

Ephemeris times and look vectors are identical between the two data products. Again, awesome! The spacecraft XYZ positions differ though. isisA is the UA dejittered RED NOPROJ cube and isisB is the hiproc dejittered RED NOPROJ cube. Differences in spacecraft position are measures at 100 points over the image. I have truncated the results below. The interesting thing is that the differences in x, y, and z are relatively consistent along and across track. I say relatively because the magnitude of the difference is monotonically increasing across and along track.

Is this expected behavior? Where might this systematic offset be coming from? Operator error?

@rbeyer If you want the UA cub and the notebook I am using for comparisons just let me know an I can get you a URL / gist.