Comments (6)
- Yes, this is a good starting point since the whole EasyVolcap framework uses such an arrangement. However, for the specific algorithm of 4K4D, you still need to do some pre-processing to generate the initialization point clouds (using space carving).
- Yes, in a way we do, just need to symlink the files to the EasyVolcap/EasyMoCap format using the script: https://github.com/zju3dv/4K4D/blob/main/scripts/preprocess/neural3dv_to_easyvolcap.py. There're also lots of scripts for conversion from other data formats in that folder as well.
- Typically, you need a 10-degree angle between cameras to get good results (18 cameras for 180 degrees) and frame count should be no longer than 300 to avoid memory issues.
from 4k4d.
Hi, thanks for the interest!
Which 4DGS are you referring to? The deformable 4DGS (deformation field + canonical space https://github.com/hustvl/4DGaussians), or the per-frame ones (like the first Dynamic 3DGS for tracking https://github.com/JonathonLuiten/Dynamic3DGaussians) or the 4D Gaussian ones (like https://github.com/fudan-zvg/4d-gaussian-splatting).
Currently, it's not (fully) possible for the native 4K4D model to be directly converted to ply files since the bell curve for the opacity is a little bit different and IBR doesn't correspond one-to-one to SHs. However, if you're willing to sacrifice some quality, it should be possible to just use the radius as the isometric scale for Gaussians (with a constant multiplied) and fit a SH function to the IBR colors.
from 4k4d.
Hello, firstly, thank you for your quick response and for your splendid job. I'm referring to 4DGS which is https://github.com/hustvl/4DGaussians especially.
Oh I see. On the other hand there are multiple scripts with several names on scripts
folder, can I train with different backbones or methods for dynamic scene rendering on these with 4K4D or there are a strict method that I can not change for training ?
Cheers
from 4k4d.
- Difference from deformable Gaussians: We use a fully implicit method for modeling dynamics (per-frame xyzs + kplanes encoded geometry and appearance properties), thus large motions can be trivially handled like training a per-frame 3DGS, while the deformable methods might struggle to follow the motion. The downside is also obvious, you don't get the tracking results for the dynamic scene.
- You can certainly try to train with different backbones, for example, you can simply replace our bell curve for opacity with a Gaussian. Notably the CUDA backend I implemented is tailored for the 4K4D bell curve. To use the original Gaussian curve, just use the original CUDA-based Gaussian rasterizer.
from 4k4d.
I see, can't wait to try then :) Here are my last questions on your repo with custom dataset relation.
- Importing
extri.yml
andintri.yml
files with EasyMocap and extracting video frames is enough to train a multiview custom dataset ? - Is there any support DyNerf dataset type (as I see, there is not) ? (It'd be much more easier than generating a poses file with COLMAP rather than extri-intri with EasyMocap and ChessMat)
- Additionally what is the suggested frame and camera number for getting best results ?
Cheers
from 4k4d.
Thank you for detailed answers!
from 4k4d.
Related Issues (20)
- Are there any detailed steps for training our own data? HOT 10
- error in imgui_bundle HOT 5
- version `GLIBC_2.33' not found HOT 2
- Extraction of depth map and normal map HOT 1
- training speed HOT 1
- Runtime exception: Could not find {optimized/intri.yml,optimized/extri.yml} HOT 1
- There was a problem rendering the smallest data set HOT 1
- Bad result on custom dataset HOT 5
- Render my trained model HOT 1
- Training Error: launcher() got an unexpected keyword argument 'viewer_cfg' HOT 2
- error in evc-gui -c configs/projects/realtime4dv/rendering/4k4d_0008_01.yaml,configs/specs/vf0.yaml HOT 2
- error in python scripts/realtime4dv/extract_images.py
- error in: evc-train HOT 2
- An error occurred during the joint training model conversion. HOT 3
- Got bad rendering quality than expected after training 4K4D on NHR dataset HOT 4
- Question about the pretrained models HOT 1
- Evaluation: rendering the test views HOT 3
- Training on NHR HOT 1
- eglQueryDeviceAttribEXT is not available.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from 4k4d.