juliahci / adi.jl Goto Github PK

View Code? Open in Web Editor NEW

6.0 2.0 5.0 68.92 MB

Algorithms for performing angular differential imaging (ADI)

Home Page: https://juliahci.github.io/ADI.jl/dev/

License: MIT License

Julia 100.00%

astronomy hci imaging reduction adi

adi.jl's People

Stargazers

Watchers

Forkers

siddharthlal25 giordano semaphorep zeta1999 ptiede

adi.jl's Issues

Error during benchmarks

Since I couldn't install the Python environment on my system (multiple errors for psutil and scipy), I created the following Docker container to reproduce them:

FROM python:3.8

# Install Julia
RUN curl -q -o - 'https://julialang-s3.julialang.org/bin/linux/x64/1.5/julia-1.5.3-linux-x86_64.tar.gz' | tar xzf -
ENV PATH="/julia-1.5.3/bin:${PATH}"

# Install ADI
RUN julia -e 'import Pkg; Pkg.add(name="ADI", version="0.5.2")'

# Instantiate Julia environment for benchmark
WORKDIR /root/.julia/packages/ADI/OCUJ4/bench
# We need to add ADI because the current Manifest file hard-codes a local path.
RUN julia --project=. -e 'import Pkg; Pkg.add(name="ADI", version="0.5.2"); Pkg.instantiate(); Pkg.precompile()'

# Prepare Python environment
RUN apt-get update -qq
RUN DEBIAN_FRONTEND=noninteractive apt-get install -qq -y gfortran libblas-dev liblapack-dev < /dev/null > /dev/null
# Cleanup apt cache
RUN rm -rf /var/lib/apt/lists/*
RUN pip install -r requirements.txt

I had also a problem with the Julia environment, but the solution was to not hard-code the local path of ADI (see #31).

However, when finally running the script adi.jl I get the following error:

/usr/local/lib/python3.8/site-packages/hciplot/hciplot.py:28: MatplotlibDeprecationWarning: Passing raw data via parameters data and lut to register_cmap() is deprecated since 3.3 and will become an error two minor releases later. Instead use: register_cmap(cmap=LinearSegmentedColormap(name, data, lut))
  register_cmap('ds9cool', data=ds9cool)
/usr/local/lib/python3.8/site-packages/hciplot/hciplot.py:29: MatplotlibDeprecationWarning: Passing raw data via parameters data and lut to register_cmap() is deprecated since 3.3 and will become an error two minor releases later. Instead use: register_cmap(cmap=LinearSegmentedColormap(name, data, lut))
  register_cmap('ds9heat', data=ds9heat)
ERROR: LoadError: KeyError: key :medsub not found
Stacktrace:
 [1] __getproperty(::PyObject, ::Symbol) at /root/.julia/packages/PyCall/BcTLp/src/PyCall.jl:307
 [2] getproperty(::PyObject, ::Symbol) at /root/.julia/packages/PyCall/BcTLp/src/PyCall.jl:312
 [3] top-level scope at /root/.julia/packages/ADI/OCUJ4/bench/adi.jl:33
 [4] include(::String) at ./client.jl:457
 [5] top-level scope at REPL[3]:1
in expression starting at /root/.julia/packages/ADI/OCUJ4/bench/adi.jl:33

Update PSFModels and fix column-orientation

ADI.jl needs desperately updated to use [email protected], [email protected], [email protected], and [email protected]. These are not trivial changes- the new PSF models and injection code will require lots of line-by-line changes, I'm not sure how much are already here. Fixing the column-orientation is going to be tedious, partly because the "convenient" helpers and abstractions get really messy for LOCI and the annulus views. In addition, the documentation will need to be carefully revisited to make sure the orientation convention is consistent.

"raw" contrast, or single-frame contrast

It would be nice to have a method for creating contrast curves from a single frame. This is essentially just a radial profile of the inverse noise, but it's a convenient shorthand since the throughput doesn't have to be measured and the interface becomes much simpler. Right now I've used

cube = view(frame, :, :, 1)
alg(cube, args...; kwargs...) = cube[:, :, 1]
cc = contrast_curve(alg, cube, zeros(1), psf; kwargs...)

TagBot trigger issue

This issue is used to trigger TagBot; feel free to unsubscribe.

If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.

Another error duing benchmarks

Following up from #32, this is my new Dockerfile:

FROM python:3.8

# Install Julia
RUN curl -q -o - 'https://julialang-s3.julialang.org/bin/linux/x64/1.5/julia-1.5.3-linux-x86_64.tar.gz' | tar xzf -
ENV PATH="/julia-1.5.3/bin:${PATH}"

# Install ADI
RUN julia -e 'using Pkg; pkg"dev ADI"'

# Instantiate Julia environment for benchmark
WORKDIR /root/.julia/dev/ADI/bench
# We need to dev ADI because the current Manifest file hard-codes a local path.
RUN julia --project=. -e 'using Pkg; pkg"dev ADI"; Pkg.instantiate(); Pkg.precompile()'

# Prepare Python environment
RUN apt-get update -qq
RUN DEBIAN_FRONTEND=noninteractive apt-get install -qq -y gfortran libblas-dev liblapack-dev < /dev/null > /dev/null
# Cleanup apt cache
RUN rm -rf /var/lib/apt/lists/*
RUN pip install -r requirements.txt

# Prefetch HCI data sets
ENV DATADEPS_ALWAYS_ACCEPT=true
RUN julia --project=. -e 'using HCIDatasets: BetaPictoris, HR8799; BetaPictoris[:cube]; BetaPictoris[:pa]; HR8799[:cube]; HR8799[:pa]'

Now when running bench/adi.jl I get:

[ Info: Benchmarking - HCIDatasets.BetaPictorisDataset() ADI Reduction
[ Info: Python - #6
ERROR: LoadError: PyError ($(Expr(:escape, :(ccall(#= /root/.julia/packages/PyCall/BcTLp/src/pyfncall.jl:43 =# @pysym(:PyObject_Call), PyPtr, (PyPtr, PyPtr, PyPtr), o, pyargsptr, kw))))) <class 'RuntimeError'>
RuntimeError('Opencv python bindings cannot be imported. Install opencv or set imlib to skimage')
  File "/usr/local/lib/python3.8/site-packages/vip_hci/medsub/medsub_source.py", line 168, in median_sub
    cube_der = cube_derotate(cube_out, angle_list, imlib=imlib,
  File "/usr/local/lib/python3.8/site-packages/vip_hci/preproc/derotation.py", line 199, in cube_derotate
    array_der[i] = frame_rotate(array[i], -angle_list[i], imlib=imlib,
  File "/usr/local/lib/python3.8/site-packages/vip_hci/preproc/derotation.py", line 118, in frame_rotate
    raise RuntimeError(msg)

Stacktrace:
 [1] pyerr_check at /root/.julia/packages/PyCall/BcTLp/src/exception.jl:62 [inlined]
 [2] pyerr_check at /root/.julia/packages/PyCall/BcTLp/src/exception.jl:66 [inlined]
 [3] _handle_error(::String) at /root/.julia/packages/PyCall/BcTLp/src/exception.jl:83
 [4] macro expansion at /root/.julia/packages/PyCall/BcTLp/src/exception.jl:97 [inlined]
 [5] #110 at /root/.julia/packages/PyCall/BcTLp/src/pyfncall.jl:43 [inlined]
 [6] disable_sigint at ./c.jl:446 [inlined]
 [7] __pycall! at /root/.julia/packages/PyCall/BcTLp/src/pyfncall.jl:42 [inlined]
 [8] _pycall!(::PyObject, ::PyObject, ::Tuple{Array{Float32,3},Array{Float32,1}}, ::Int64, ::PyObject) at /root/.julia/packages/PyCall/BcTLp/src/pyfncall.jl:29
 [9] _pycall!(::PyObject, ::PyObject, ::Tuple{Array{Float32,3},Array{Float32,1}}, ::Base.Iterators.Pairs{Symbol,Bool,Tuple{Symbol},NamedTuple{(:verbose,),Tuple{Bool}}}) at /root/.julia/packages/PyCall/BcTLp/src/pyfncall.jl:11
 [10] #_#117 at /root/.julia/packages/PyCall/BcTLp/src/pyfncall.jl:86 [inlined]
 [11] (::var"#6#12")(::Array{Float32,3}, ::Vararg{Any,N} where N; kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /root/.julia/dev/ADI/bench/adi.jl:34
 [12] #6 at /root/.julia/dev/ADI/bench/adi.jl:34 [inlined]
 [13] ##core#295(::var"#6#12", ::Array{Float32,3}, ::Array{Float32,1}) at /root/.julia/packages/BenchmarkTools/eCEpo/src/execution.jl:371
 [14] ##sample#296(::BenchmarkTools.Parameters) at /root/.julia/packages/BenchmarkTools/eCEpo/src/execution.jl:377
 [15] _run(::BenchmarkTools.Benchmark{Symbol("##benchmark#294")}, ::BenchmarkTools.Parameters; verbose::Bool, pad::String, kwargs::Base.Iterators.Pairs{Symbol,Integer,NTuple{4,Symbol},NamedTuple{(:samples, :evals, :gctrial, :gcsample),Tuple{Int64,Int64,Bool,Bool}}}) at /root/.julia/packages/BenchmarkTools/eCEpo/src/execution.jl:405
 [16] (::Base.var"#inner#2"{Base.Iterators.Pairs{Symbol,Integer,NTuple{5,Symbol},NamedTuple{(:verbose, :samples, :evals, :gctrial, :gcsample),Tuple{Bool,Int64,Int64,Bool,Bool}}},typeof(BenchmarkTools._run),Tuple{BenchmarkTools.Benchmark{Symbol("##benchmark#294")},BenchmarkTools.Parameters}})() at ./essentials.jl:713
 [17] #invokelatest#1 at ./essentials.jl:714 [inlined]
 [18] #run_result#37 at /root/.julia/packages/BenchmarkTools/eCEpo/src/execution.jl:32 [inlined]
 [19] run(::BenchmarkTools.Benchmark{Symbol("##benchmark#294")}, ::BenchmarkTools.Parameters; progressid::Nothing, nleaves::Float64, ndone::Float64, kwargs::Base.Iterators.Pairs{Symbol,Integer,NTuple{5,Symbol},NamedTuple{(:verbose, :samples, :evals, :gctrial, :gcsample),Tuple{Bool,Int64,Int64,Bool,Bool}}}) at /root/.julia/packages/BenchmarkTools/eCEpo/src/execution.jl:94
 [20] #warmup#45 at /root/.julia/packages/BenchmarkTools/eCEpo/src/execution.jl:141 [inlined]
 [21] warmup(::BenchmarkTools.Benchmark{Symbol("##benchmark#294")}) at /root/.julia/packages/BenchmarkTools/eCEpo/src/execution.jl:141
 [22] top-level scope at /root/.julia/packages/BenchmarkTools/eCEpo/src/execution.jl:287
 [23] top-level scope at /root/.julia/dev/ADI/bench/adi.jl:47
 [24] include(::String) at ./client.jl:457
 [25] top-level scope at REPL[1]:1
in expression starting at /root/.julia/dev/ADI/bench/adi.jl:39

Not sure what I'm supposed to do, whether to install opencv or set imlib to skimage.

Documentation suggestions.

Going through your documentation, I think it would be helpful to be more explicit in the requirements for the input data format. In particular:

Mentioning image registration requirements. Is the star at the center of the image (what about for even-sized arrays) for all images?
The format of the angles vector. Also, whether this is CCW or CW rotation, and whether it is in degrees or radians.

For the SDI feature, I understand it is still under development but here are some suggestions:

Typically, this is done in IFS images where multiple wavelengths are acquired at the same time. I'd suggest leading with the nf dimension rather than the nlam dimension
How is the scales argument defined? I think there is a 1/x degeneracy without knowing its definition (or an example computing this term from the wavelength).

`UndefVarError: head not defined` in the documentation

While following the example of the reduction of β Pictoris in the documentation, in the section about the contrast curve I ran into the error

julia> head(filter(row -> isfinite(row.contrast_corr), cc))
ERROR: UndefVarError: head not defined
Stacktrace:
 [1] top-level scope
   @ REPL[41]:1

This is my environment:

(tmp) pkg> status
Status `/tmp/Project.toml`
  [904a6c7d] ADI v0.5.2
  [3b1b4be9] CFITSIO v1.0.0
  [a93c6f00] DataFrames v0.22.0
  [1893dc6a] HCIDatasets v0.2.0
  [b6cd55e5] HCIToolbox v0.4.2
  [91a5bcdd] Plots v1.9.1

Make `ADIAlgorithm <: Any`?

I might be wrong, but I don't think the fact ADIAlgorithm <: Function is used anywhere. The Function type shouldn't really be used to restrict function arguments to functions: there are a lot of "callable" objects which aren't Functions, e.g. all type constructors and Python functions from PyCall.

I don't have a strong feeling about ADIAlgorithm <: Function, but I'd like to hear your opinion about doing simply

abstract type ADIAlgorithm end

Paper comments

Overall, the paper looks great. The one thing I found confusing is the distinction between "posterior analysis" and "forward modeling". I agree there is a distinction here, but I feel like "posterior analysis" is quite vague, and forward modeling may also fall into that category since the forward models are compared to the residual images in similar ways. It seems like everything under the category of "posterior analysis" currently could be more precisely categorized as "detection metrics".

The table comparing existing packages is very useful for the community and overall I found this summary excellent. As the maintainer of pyklip, I would note that pyklip also supports nmf and weighted PCA algorithms, although they are not available in the forward modeling framework.

Possible improvements to documentation

After going through the documentation, I can give some recommendations:

the introduction to HCI is very interesting, it gives a nice overview of the topic without going in too many details. As mentioned at the beginning, it isn't meant to be an exhaustive lecture note, but it may be good to leave some references at the end for further reading
the first snippets in the getting started page don't show using ADI, which can confuse newcomers. The first using ADI in the whole documentation is in the Reduction Process section of the same page. Maybe the home page might mention that you need to do using ADI to load it?
the Decomposition section presents a formula without introducing the terms, especially w and A. The page is geared towards people familiar with ADI, but briefly mentioning what these terms are could improve understanding of newcomers, too
the reduction of β Pictoris example requires a bunch of packages which aren't necessarily present in the user's environment. Maybe quickly mention that they might need to install them?
despite what is written in the data reduction section, mask_circle doesn't seem to be re-exported by ADI.
```
julia> mask_cube = mask_circle(cube, 10)
ERROR: UndefVarError: mask_circle not defined
Stacktrace:
 [1] top-level scope
   @ REPL[29]:1
```
Same for Kernels and construct in the contrast curve section
decompose docstring only shows the syntax of the function without further details. Is it supposed to not give more details?
the documentation lacks community guidelines, like instructions about how to contribute, report issues, seek support

juliahci / adi.jl Goto Github PK

adi.jl's People

Stargazers

Watchers

Forkers

adi.jl's Issues

Error during benchmarks

Update PSFModels and fix column-orientation

"raw" contrast, or single-frame contrast

TagBot trigger issue

Another error duing benchmarks

Documentation suggestions.

`UndefVarError: head not defined` in the documentation

Make `ADIAlgorithm <: Any`?

Paper comments

Possible improvements to documentation

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent