manurare / 360monodepth Goto Github PK

hi,
when i study the 360monodepth source code, I can not find the 'configuration' file or can not install the module 'configuration' by 'pip install configuration'.
need you help .
thankyou

Hi,
thanks for sharing the code.

Based on my understanding, the output erp_dispmap_blend of the depth_estimation() function is the disparity maps not the depth map.

If so, how can I convert disparity to depth and store them as pfm.

Thanks.

I have simplify and extent this repository in this fork so I can easily use it as a pipeline to extract depth from 360 photos taken with a Ricoh camera. Can't made poisson blending to work (#18) but I had success using frustum blending.
Here the results of the test image on midas2 and midas3 models


When extending the models to support ZoeDepth motivated by their spatial consistency the results seams to have blending issues.

Wonder if the approach I'm using is the correct, and if there is something else I should be doing?

Hi, the main.py python script crashes at this line part of compute_linear_system_matrices() as part of the poisson blending process.

    self.eigen_solver.A = self.A.transpose().dot(self.A)

Any idea how should I debug or solve it?

Hi, nice to see your interesting work

I have problems to run your docker environment on Ubuntu 20.04 and Windows 10

Can you help to test whether the docker environment is valid or not

Thanks a lot !

Hi I m gonna make a test with your nice library, but a quick question about scale, if I would provide 2 images of similar location would I get a consistent scale in both results? Is there anyway to get into the right scale or just like midas it s an avarage scale ?

Thanks

Hello,

Thank you very much for this nice library. Could you please let me know how I can access the output depth map? When I run your code on an unseen equirectangular RGB image, I get five images as follows:
000_360monodepth_midas2_frustum.png, 000_360monodepth_midas2_mean.png, 000_360monodepth_midas2_nn.png, 000_360monodepth_midas2_poisson.png, 000_360monodepth_midas2_radial.png

However, I would like to generate the point cloud from the output of your program to see how the depth prediction works visually.

Hi 360monodepth team, awesome work!
I follow the method to produce my customed replica360 dataset as mentioned in your work. The panoramic rgb image looks ok, but the depth image is likely been truncated cause of inappropriate depth scale or depth image format.
Specificly, I use matryodshka-replica360 code to render panoramic depth images, I find the depth image can be naturally unprojected to a unit sphere, the we can get the hole room pointcloud at once.

./ReplicaSDK/ReplicaRendererDataset /media/ziqianbai/BACKPACK_DATA1/Replica_all/replica_v1/room_0/mesh.ply /media/ziqianbai/BACKPACK_DATA1/Replica_all/replica_v1/room_0/textures/ /media/ziqianbai/BACKPACK_DATA1/Replica_all/replica_v1/room_0/glass.sur ../glob/train/room_0_6dof.txt y /media/ziqianbai/BACKPACK_DATA1/Replica_all/replica_for_panonerf/room_0/ 1024 512 ../glob/pro2pos.txt

But the pointcloud seems imprecise, the depth is likely been truncated cause of inappropriate depth scale or depth image format:

The code I used to unproject the equirect_depth_image :

def test_spherical_depth():
    def get_unit_spherical_map():
        h = 512
        w = 1024
        Theta = np.arange(h).reshape(h, 1) * np.pi / h + np.pi / h / 2
        Theta = np.repeat(Theta, w, axis=1)
        Phi = np.arange(w).reshape(1, w) * 2 * np.pi / w + np.pi / w - np.pi
        Phi = -np.repeat(Phi, h, axis=0)

        X = np.expand_dims(np.sin(Theta) * np.sin(Phi),2)
        Y =  np.expand_dims(np.cos(Theta),2)
        Z = np.expand_dims(np.sin(Theta) * np.cos(Phi),2)
        unit_map = np.concatenate([X,Z,Y],axis=2)

        return unit_map

    depth_img_filepath = '/media/ziqianbai/BACKPACK_DATA1/Replica_all/replica_for_panonerf/room_0/room_0_0000_pos12.png'
    raw_depth_img = Image.open(depth_img_filepath)
    depth_img = ImageOps.grayscale(raw_depth_img)
    depth_img = np.asarray(depth_img)
    # depth_img = np.asarray(Image.open(depth_img_filepath))
    depth_img=np.expand_dims((depth_img*16.0),axis=2)
    pointcloud = depth_img * get_unit_spherical_map()

    o3d_pointcloud = o3d.geometry.PointCloud()
    o3d_pointcloud.points = o3d.utility.Vector3dVector(pointcloud.reshape(-1,3))
    o3d.io.write_point_cloud('/media/ziqianbai/BACKPACK_DATA1/Replica_all/replica_for_panonerf/room_0/room_0_0000_pcl_2.ply', o3d_pointcloud)

I'm sure you probably have similar problems. Could u help me figure out the reason the resulting phenomenon? Thanks in advance! @cr333 @manurare

Hi @manurare, currently the model takes 20 images from the faces of an icosahedron.
Could you recommend me ways in which I can increase the number of images (close to 100 or 200) and the code to do the same?

I am aware that project_icosahedron.py implements the required projections. More details will be super helpful.
Thanks!

Hi,

I see that the function disparity2depth() has 2 parameters baseline and focal.

Do you know how to find these two parameters?

Since I use only a single 360 camera with Lidar to capture depths, there is no baseline between the two cameras.

Could you please suggest some approach to determine the baseline and focal?

Hello,

When running Docker I run into this error:

[12/12] RUN cd /monodepth/python/src/test && python3 ./test_depthmapAlign_module.py --task 1:
#16 0.278 /bin/sh: 1: cd: can't cd to /monodepth/python/src/test

I'm not well versed in Docker, but seems like it's referencing a directory that's not there. What can I do?

Hello,

Thanks to the very nice 360 RGBD datasets! I'm struggling with using the .dpt files...Don't know how to open it and get the depth numbers.

Could you give an example code for reading in the .dpt file?

Thank you.

Thank you for this amazing work. I successfully installed everything with the help of Dockerfile. However, when I ran the main.py code, I got following error everytime. I tried recompiling everything but still got same error.

root@55d28786ec5a:/monodepth/code/python/src# python3 main.py 
Adding '/monodepth/code/python/src' to sys.path
2023-02-10 08:01:56,106 - utility.depthmap_align - INFO - depthmapAlign python module installed!
Traceback (most recent call last):
  File "/monodepth/code/python/src/main.py", line 17, in <module>
    from utility import blending
  File "/monodepth/code/python/src/utility/blending.py", line 6, in <module>
    from EigenSolvers import LinearSolver
ImportError: /monodepth/code/cpp/lib/EigenSolvers.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN6google10LogMessage6streamEv
root@55d28786ec5a:/monodepth/code/python/src#

Can you help me with this? Thank you.

Is there an easy way to get a rotation matrix for each tangent image? i.e. the rotation matrix between the tangent image and the panorama's coordinate system.

Hi,
I want to fine-tune or train the model on the custom dataset.

Could you please give some guidance?

Thanks,
Hieu

Hello,

The result of your program looks amazing, I am trying to run the code myself to take a live look but when I try to run the code after the docker builds successfully it provides me this error:
docker: Error response from daemon: Unknown runtime specified nvidia.

I then modified the docker run command to use the --gpus all instead of the --runtime=nvidia command, this got me a little further however I still get a NVIDIA GeForce RTX 3070 with CUDA capability sm_86 is not compatible with the current PyTorch installation error.

Am I doing something wrong, do you have any advice?

Thank you.