Difference From 4DGS about 4k4d HOT 6 CLOSED

1. Yes, this is a good starting point since the whole EasyVolcap framework uses such an arrangement. However, for the specific algorithm of 4K4D, you still need to do some pre-processing to generate the initialization point clouds (using space carving).
2. Yes, in a way we do, just need to symlink the files to the EasyVolcap/EasyMoCap format using the script: https://github.com/zju3dv/4K4D/blob/main/scripts/preprocess/neural3dv_to_easyvolcap.py. There're also lots of scripts for conversion from other data formats in that folder as well.
3. Typically, you need a 10-degree angle between cameras to get good results (18 cameras for 180 degrees) and frame count should be no longer than 300 to avoid memory issues.

from 4k4d.

Hi, thanks for the interest!
Which 4DGS are you referring to? The deformable 4DGS (deformation field + canonical space https://github.com/hustvl/4DGaussians), or the per-frame ones (like the first Dynamic 3DGS for tracking https://github.com/JonathonLuiten/Dynamic3DGaussians) or the 4D Gaussian ones (like https://github.com/fudan-zvg/4d-gaussian-splatting).
Currently, it's not (fully) possible for the native 4K4D model to be directly converted to ply files since the bell curve for the opacity is a little bit different and IBR doesn't correspond one-to-one to SHs. However, if you're willing to sacrifice some quality, it should be possible to just use the radius as the isometric scale for Gaussians (with a constant multiplied) and fit a SH function to the IBR colors.

from 4k4d.

Hello, firstly, thank you for your quick response and for your splendid job. I'm referring to 4DGS which is https://github.com/hustvl/4DGaussians especially.

Oh I see. On the other hand there are multiple scripts with several names on scripts folder, can I train with different backbones or methods for dynamic scene rendering on these with 4K4D or there are a strict method that I can not change for training ?

Cheers

from 4k4d.

1. Difference from deformable Gaussians: We use a fully implicit method for modeling dynamics (per-frame xyzs + kplanes encoded geometry and appearance properties), thus large motions can be trivially handled like training a per-frame 3DGS, while the deformable methods might struggle to follow the motion. The downside is also obvious, you don't get the tracking results for the dynamic scene.
2. You can certainly try to train with different backbones, for example, you can simply replace our bell curve for opacity with a Gaussian. Notably the CUDA backend I implemented is tailored for the 4K4D bell curve. To use the original Gaussian curve, just use the original CUDA-based Gaussian rasterizer.

from 4k4d.

I see, can't wait to try then :) Here are my last questions on your repo with custom dataset relation.

1. Importing extri.yml and intri.yml files with EasyMocap and extracting video frames is enough to train a multiview custom dataset ?
2. Is there any support DyNerf dataset type (as I see, there is not) ? (It'd be much more easier than generating a poses file with COLMAP rather than extri-intri with EasyMocap and ChessMat)
3. Additionally what is the suggested frame and camera number for getting best results ?

Cheers

from 4k4d.

Thank you for detailed answers!

from 4k4d.