Depth map and intrinsic about co3d HOT 9 CLOSED

The vaule of the depth map car/106_12650_23736/depths/frame000001.jpg.geometric.png range from [16490-22649], which indicates the camera center is far from the object.

The color map of the depth (scaled by 1e-3):

from co3d.

Hi, we provide tooling for operating with the depth maps loaded using the Co3DDataset object.
Specifically, this function

from co3d.

Thanks. I will follow the scripts and let you know how it turns out.

from co3d.

@colesbury Sorry to bother you agiain. I want to know some details about the depth maps.

Do the depth maps directly come from the COLMAP? Have you made any post-processing modifications to the depth maps?

from co3d.

I guess i found the resaon why the depth map is not working.

The depth value in the depth maps should be loaded as 16 bit float number. The same as the _load_16bit_depth

This is how it looks:

from co3d.

I guess i found the resaon why the depth map is not working.

The depth value in the depth maps should be loaded as 16 bit float number. The same as the _load_16bit_depth

co3d/dataset/co3d_dataset.py

Line 728 in b22f145

def _load_16big_png_depth(depth_png):

This is how it looks:

Hello, this projection is exactly what I need in my current project. However I didn't fully understand how you obtained it, I tried quite a few provided functions in this repository but it's a bit unclear how to use them. Please, could you tell us what steps you followed and which functions you used to get this nice point cloud?

For example, loading depth as 16 bit had a similar result as your first projection.

from co3d.

@AlexisStdp Did you use the provided function to load depth? Please note the files are not a standard 16-bit PNG; the function reinterprets binary 16-bit values as floats.

from co3d.

@shapovalov Thank you very much for your answer! I indeed used some of the provided functions to load depth, and to load RGB image. And some functions to get the frame_annotations.
Ideally if I could have the following setting it would be perfect: given an RGB image and a depth image (and the .jgz annotations), we load RGB and depth, we get the intrinsic (and maybe scale adjustment), and we backproject to get the partial point cloud.

In the provided code, it seems that the function "get_rgbd_point_cloud" is exactly that, but I'm unfortunately unable to make it work because I'm not sure how to get the argument "camera: CamerasBase" (is there a simple way I could get it?).
There is a function "get_co3d_sequence_pointcloud" which I tried to use in order to get this "camera" (because it itself uses "get_rgdp_point_cloud"), but inside of it the Dataloader didn't work for various reasons.

Is there maybe an alternative approach? - I tried using open3d for example but this gives me a "flat" point cloud (where all points seem to lie on a 2D plane).

Thanks in advance, and happy new year!

from co3d.

@AlexisStdp Co3dDataset should load the data in the required format. In particular, frame_data.camera is in PyTorch3D format, which should be compatible with get_rgbd_point_cloud. If that does not work for you, could you share more details?

If you want to dig deeper, this is the code that reads the Cameras object: https://github.com/facebookresearch/co3d/blob/main/dataset/co3d_dataset.py#L490

Note that the camera is in NDC coordinate system. Please refer to PyTorch3D documentation for details: https://pytorch3d.org/docs/cameras

from co3d.