tonyketcham / depth-mapper Goto Github PK

Uses a Laplace transform on each image in the stack to map the images' second derivatives. The algorithm then iterates through each corresponding pixel of the entire stack to find the image with the highest rate of change at that pixel. This pixel is plotted in the final image. Then the algorithm loops until the entire final image is filled.

Scaling (lens-breathing) occurs from shifting the focus distance of the lens when recording the image stack. This results in ghosted textures in the final images, among other artifacts by comparing misaligned pixels between images.

Implementing a feature-based alignment algorithm should automatically fix these issues without any extra user input or lens profiles.

Possible solutions:
[1] https://www.learnopencv.com/image-alignment-ecc-in-opencv-c-python/
[2] https://www.learnopencv.com/image-alignment-feature-based-using-opencv-c-python/

Issue

The Video.findTransformECC() method should not require an imageMask arg, so null should be applicable in this case. However a null pointer exception is thrown when using a mask is not desired. Attempting to pass in a zeros matrix does not seem to remedy this issue either.

Potential Solution

Allegedly an issue with OpenCV 4.1. Downgrading to an earlier version may fix this issue, but doing so may have the side effect of worsening the running time of the overall program.

UMats utilize GPU when applicable to drastically decrease processing time over Mats which are bound to CPU processing: see more...

JavaCV code proves syntactically cleaner and more true to the native C++/Python style given in the OpenCV documentation. C++ modules that OpenCV has not yet wrapped for Java are also included in JavaCV.

Crucially, JavaCV has gone where stock Java OpenCV hasn't and wrapped Transparent API, which is useful for instantiating UMats through OpenCL that play nicely with OpenCV's Mat methods. This takes advantage of GPU hardware acceleration (as opposed to Mats which are CPU-bound) and can speed up processes by orders of magnitude. In the case of image alignment on a stack of 20MP+ images, this kind of speed is imperative.

Problem

The ECC image alignment implementation takes ~10 minutes to align a stack of 15 large (20MP) images. Although a noticeable improvement, the results are not perfect either. Some ghosting lines are still present from image misalignment.

Potential Solution

Bitmap alignment may prove much faster than ECC. Perhaps finding a rough homography first may even be faster.

Each image in the stack is assumed to be ordered: first is closest, last is furthest from the camera. A linear transformation is then done between the number of input images onto the range of 255-0 (white to black), mapping white to the closest image, and so on.

The depth mapping procedure then piggybacks off of the focus stacking algorithm. After the focus stacking decides which pixel to place in a given position of the final focus stacked image, the grayscale value that is mapped to the source image of that respective pixel is grabbed and set to the same pixel position in a depth map image.

The process results in a black and white depth map of the scene.