zju3dv / im4d Goto Github PK

Thanks for your awesome work!

I am re-implement this repo. But I don't know what is the correct data format. Could you please provide an example to run your code, like DNA-Rendering?

Thank you very much!

Excellent work. I saw the results of DyNerf and outdoor datasets on the project page, but I did not find the relevant config and data processing files in the project. Will they be added in the future?

command is

python train_net.py --cfg_file configs/exps/im4d/nhr/sport1.yaml

error:

Traceback (most recent call last):
  File "/home/haima/im4d/train_net.py", line 108, in <module>
    main()
  File "/home/haima/im4d/train_net.py", line 100, in main
    train(cfg, network)
  File "/home/haima/im4d/train_net.py", line 22, in train
    train_loader = make_data_loader(cfg,
  File "/home/haima/im4d/lib/datasets/make_dataset.py", line 90, in make_data_loader
    dataset = make_dataset(cfg, is_train)
  File "/home/haima/im4d/lib/datasets/make_dataset.py", line 39, in make_dataset
    dataset = dataset(**args)
  File "/home/haima/im4d/lib/datasets/volcap/base_dataset.py", line 35, in __init__
    self.prepare_camera() # prepare camera intrinsics and extrinsics
  File "/home/haima/im4d/lib/datasets/volcap/base_dataset.py", line 68, in prepare_camera
    cams = easy_utils.read_camera(intri_path, extri_path, read_rvec=self.cfg.get('read_rvec', True))
  File "/home/haima/im4d/lib/utils/easyvv/easy_utils.py", line 73, in read_camera
    assert os.path.exists(intri_path), intri_path
AssertionError: /mnt/e/im4d/NHR/sport_1_easymocap/intri.yml

question:

1. NHR sport1 download from mlp_maps , but miss intri.yml
   from google: https://drive.google.com/file/d/1BnqbXXzswcDh_3VzkYz1PW6LMWVf_HNv/view

Hi,

Thanks for the great work. May I ask whether there is a per-scene breakdown of the quantitative results for the DyNeRF dataset. The paper says it will be included in the supplementary, but I can't seem to find it? May I ask whether the result for the DyNeRF dataset is calculated on the flame_salmon scene or all the scenes together?

Thanks!

Thanks for your wonderful work!

I am trying trying to training with theDNA_Rendering data, but we lack of the .ply file in pointcloud_denoise folder, could you please tell us how to get these point cloud files?

Thanks again, looking forward for your reply!!!

Thanks for your great work, I would like to ask what is the meaning of m_c in formula (4) in section 3.2 of the paper?

python run.py --type evaluate --cfg_file configs/exps/im4d/renbody/0013_01.yaml save_result True After running the command, I am prompted that there is no file renbody\0013_01\ intr.yml, I would like to ask if this file is missing

Hello, can you please share the config file for the DyNeRF dataset? I couldn't find it in the open-source code. Thank you!

Hello, i test do it follow this guide, but i can't get have color image or video, just 4d no color bgr.I can't sure this is right.If is right,how can i get the color bgr.Please reply. Thank you!

For custom dataset, there is no smc file like DNA-Rendering.

And this tutorial https://github.com/zju3dv/im4d/blob/main/docs/custom_data_tutorial.md only introduces the way using smc file.

Hi!
Thanks for the great work.
I've noticed that the base_dataset.py seems to expect a 4x4 transform matrix from the dataset_cfg to align the camera extrinsic Z-axis to the up direction. base_dataset.py

I've attempted to visualize this transform and apply it to the camera extrinsics using the following code:

import open3d as o3d # 0.16.0
import numpy as np

if __name__ == "__main__":

    # Step 0 - Init
    WIDTH = 1280
    HEIGHT = 720

    # Step 1 - Get scene objects
    meshFrame = o3d.geometry.TriangleMesh.create_coordinate_frame(size=2.0, origin=[0, 0, 0])
    
    # Step 2 - Create visualizer object
    vizualizer = o3d.visualization.Visualizer()
    vizualizer.create_window()
    vizualizer.create_window(width=WIDTH, height=HEIGHT)

    # Step 3 - Add objects to visualizer
    vizualizer.add_geometry(meshFrame)
    
    # NHR Baskball Dataset Camera Params
    dataset_trans = np.array([[ 0.90630779 ,-0.07338689,  0.41619774 , 2.        ],
 [ 0.42261826  ,0.1573787,  -0.89253894, -5.13      ],
 [ 0.         , 0.98480775 , 0.17364818 , 2.85      ],
 [ 0.          ,0.        ,  0.        ,  1.        ]])
    intrinsic = np.array([[ 2.3717e+03, -1.0060e+00,  5.2414e+02],
        [ 0.0000e+00,  2.3718e+03,  4.1167e+02],
        [ 0.0000e+00,  0.0000e+00,  1.0000e+00]])
    Rt = np.array([[-0.6087, -0.2077,  0.7657,  3.3700],
        [ 0.0279, -0.9701, -0.2410, -2.1150],
        [ 0.7929, -0.1253,  0.5963,  6.6570],
        [ 0.0000,  0.0000,  0.0000,  1.0000]]) # extrinsic, world2cam
    
    # Rt = Rt @ np.linalg.inv(dataset_trans) # whether to transform the camera extrinsic
    cam2world = np.linalg.inv(Rt)
    camFrame = o3d.geometry.TriangleMesh.create_coordinate_frame(size=1.0)
    camFrame.transform(cam2world)
    vizualizer.add_geometry(camFrame)

    # Visulize camera
    camera_in = o3d.camera.PinholeCameraIntrinsic()
    camera_in.intrinsic_matrix = intrinsic
    camera = o3d.camera.PinholeCameraParameters()
    camera.intrinsic = camera_in
    camera.extrinsic = Rt
    cameraLines = o3d.geometry.LineSet.create_camera_visualization(intrinsic=camera.intrinsic, extrinsic=camera.extrinsic)
    vizualizer.add_geometry(cameraLines)
    vizualizer.run()
    

However, my results seem to indicate that this operation is primarily rotating the camera coordinate system along the Z-axis, as in the following figure:

I would greatly appreciate your help in clarifying the usage of the dataset transform. Specifically, I'm interested in understanding the following:

• What is the intended purpose of the dataset_cfg transform in aligning the camera extrinsic Z-axis to the up direction?
• How should I create this transform for my custom dataset? Are there specific guidelines or steps to follow?

Thanks in advance!

hello, it's a really nice work! but some error occurs to me when I try to do testing(Reproduce the quantitative results in the paper.） what I input is [python run.py --type evaluate --cfg_file configs/exps/im4d/renbody/0013_09.yaml save_result True] , but it shows two strange error :
1.[pretrained model does not exist: /data/trained_model/0013_09/im4d/0013_09]
2.AssertionError: /data/renbody/0013_09/intri.yml
but I check the path I set is just right , could you please give me any advice?