Original reference implementation of "3D Gaussian Splatting for Real-Time Radiance Field Rendering"
Home Page: https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/
License: Other
Greetings! I am trying to run this wonderful tool on Windows 11 x64. For the other tools like the viewer, I didnt build them but used the setups you have kindly created for Windows, so thanks for that too! For this repo I ran the commands for creating environment.yml file but in the middle of things it says "crtdefs.h not found", among other errors.
I am trying to use this but I am not an expert in build tools. I have some doubts regarding the following requirements set forth in README.MD:
- C++ Compiler for PyTorch extensions (we used Visual Studio 2019 for Windows)
- CUDA SDK 11 for PyTorch extensions (we used 11.8, known issues with 11.6)
- C++ Compiler and CUDA SDK must be compatible
q1. I have Visual Studio 2022 x64 installed. Is having it installed enough?
q2. I have noticed that creating the environment the first time automatically installs and sets up cuda for py, etc. I still downloaded and setup CUDA Toolkit 11.8 from official developer.nvidia.com. All are x64 bit.
q3. I suppose I could build the submodules by opening them in Visual Studio myself but I would prefer not to.. So I tried to look up for online solutions for where the missing libraries are and adding their folders to path. At first activating the environment said "Cant find ninja, falling back to slow old tools.. Cannot find cl.exe"
I found the file at: %ProgramFiles%\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.35.32215\bin\Hostx64\x64 and added it to path, thereafter I believe it has fixed the issue.
Visual Studio proceeded to the next issue: "crtdefs.h". There is one big difference that In my case the Visual Studio directory has a different structure than most solutions given online for missing libraries and utils, probably because MS keeps shifting headers around.
I had crtdefs.h in both these directories, I added the first one to path, but hasnt worked:
%ProgramFiles%\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.35.32215\include\
%ProgramFiles%\Microsoft Visual Studio\2022\Community\SDK\ScopeCppSDK\vc15\VC\include\
Online solutions that are using older versions of VS have this header present at the following path:
%ProgramFiles%\Microsoft Visual Studio\2022\Community\VC\include\
What I also tried was opening the submodules like diff-gaussian-rasterization on Visual Studio and building them from there assuming Visual Studio would know where to get the missing headers, and that building suceeded. but nothing to help with this issue. I therefore cannot proceed with the setup. Kindly help. Heres the full trace:
D:\gaussian-splatting>conda env create --file environment.yml
Retrieving notices: ...working... done
Collecting package metadata (repodata.json): done
Solving environment: done
Downloading and Extracting Packages
Preparing transaction: done
Verifying transaction: done
Executing transaction: | "By downloading and using the CUDA Toolkit conda packages, you accept the terms and conditions of the CUDA End User License Agreement (EULA): https://docs.nvidia.com/cuda/eula/index.html"
done
Installing pip dependencies: / Ran pip subprocess with arguments:
['C:\\Users\\User\\miniconda3\\envs\\gaussian_splatting\\python.exe', '-m', 'pip', 'install', '-U', '-r', 'D:\\gaussian-splatting\\condaenv.wui8rdg4.requirements.txt', '--exists-action=b']
Pip subprocess output:
Processing D:\gaussian-splatting\submodules\diff-gaussian-rasterization
Preparing metadata (setup.py): started
Preparing metadata (setup.py): finished with status 'done'
Processing D:\gaussian-splatting\submodules\simple-knn
Preparing metadata (setup.py): started
Preparing metadata (setup.py): finished with status 'done'
Building wheels for collected packages: diff-gaussian-rasterization, simple-knn
Building wheel for diff-gaussian-rasterization (setup.py): started
Building wheel for diff-gaussian-rasterization (setup.py): finished with status 'error'
Running setup.py clean for diff-gaussian-rasterization
Building wheel for simple-knn (setup.py): started
Building wheel for simple-knn (setup.py): finished with status 'error'
Running setup.py clean for simple-knn
Failed to build diff-gaussian-rasterization simple-knn
Installing collected packages: simple-knn, diff-gaussian-rasterization
Running setup.py install for simple-knn: started
Running setup.py install for simple-knn: finished with status 'error'
Pip subprocess error:
error: subprocess-exited-with-error
× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> [20 lines of output]
running bdist_wheel
C:\Users\User\miniconda3\envs\gaussian_splatting\lib\site-packages\torch\utils\cpp_extension.py:411: UserWarning: Attempted to use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the slow distutils backend.
warnings.warn(msg.format('we could not find ninja.'))
running build
running build_py
creating build
creating build\lib.win-amd64-cpython-37
creating build\lib.win-amd64-cpython-37\diff_gaussian_rasterization
copying diff_gaussian_rasterization\__init__.py -> build\lib.win-amd64-cpython-37\diff_gaussian_rasterization
running build_ext
C:\Users\User\miniconda3\envs\gaussian_splatting\lib\site-packages\torch\utils\cpp_extension.py:813: UserWarning: The detected CUDA version (11.8) has a minor version mismatch with the version that was used to compile PyTorch (11.6). Most likely this shouldn't be a problem.
warnings.warn(CUDA_MISMATCH_WARN.format(cuda_str_version, torch.version.cuda))
building 'diff_gaussian_rasterization._C' extension
creating build\temp.win-amd64-cpython-37
creating build\temp.win-amd64-cpython-37\Release
creating build\temp.win-amd64-cpython-37\Release\cuda_rasterizer
"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\bin\nvcc" -c cuda_rasterizer/backward.cu -o build\temp.win-amd64-cpython-37\Release\cuda_rasterizer/backward.obj -IC:\Users\User\miniconda3\envs\gaussian_splatting\lib\site-packages\torch\include -IC:\Users\User\miniconda3\envs\gaussian_splatting\lib\site-packages\torch\include\torch\csrc\api\include -IC:\Users\User\miniconda3\envs\gaussian_splatting\lib\site-packages\torch\include\TH -IC:\Users\User\miniconda3\envs\gaussian_splatting\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\include" -IC:\Users\User\miniconda3\envs\gaussian_splatting\include -IC:\Users\User\miniconda3\envs\gaussian_splatting\Include -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -ID:\gaussian-splatting\submodules\diff-gaussian-rasterization\third_party/glm/ -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_61,code=compute_61 -gencode=arch=compute_61,code=sm_61 --use-local-env
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\include\crt/host_config.h(231): fatal error C1083: Cannot open include file: 'crtdefs.h': No such file or directory
backward.cu
error: command 'C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v11.8\\bin\\nvcc.exe' failed with exit code 2
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for diff-gaussian-rasterization
error: subprocess-exited-with-error
× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> [12 lines of output]
running bdist_wheel
C:\Users\User\miniconda3\envs\gaussian_splatting\lib\site-packages\torch\utils\cpp_extension.py:411: UserWarning: Attempted to use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the slow distutils backend.
warnings.warn(msg.format('we could not find ninja.'))
running build
running build_ext
C:\Users\User\miniconda3\envs\gaussian_splatting\lib\site-packages\torch\utils\cpp_extension.py:813: UserWarning: The detected CUDA version (11.8) has a minor version mismatch with the version that was used to compile PyTorch (11.6). Most likely this shouldn't be a problem.
warnings.warn(CUDA_MISMATCH_WARN.format(cuda_str_version, torch.version.cuda))
building 'simple_knn._C' extension
"C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.35.32215\bin\Hostx64\x64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD -IC:\Users\User\miniconda3\envs\gaussian_splatting\lib\site-packages\torch\include -IC:\Users\User\miniconda3\envs\gaussian_splatting\lib\site-packages\torch\include\torch\csrc\api\include -IC:\Users\User\miniconda3\envs\gaussian_splatting\lib\site-packages\torch\include\TH -IC:\Users\User\miniconda3\envs\gaussian_splatting\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\include" -IC:\Users\User\miniconda3\envs\gaussian_splatting\include -IC:\Users\User\miniconda3\envs\gaussian_splatting\Include /EHsc /Tpext.cpp /Fobuild\temp.win-amd64-cpython-37\Release\ext.obj /MD /wd4819 /wd4251 /wd4244 /wd4267 /wd4275 /wd4018 /wd4190 /EHsc /wd4624 -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=_C -D_GLIBCXX_USE_CXX11_ABI=0
ext.cpp
C:\Users\User\miniconda3\envs\gaussian_splatting\lib\site-packages\torch\include\torch\csrc\api\include\torch/cuda.h(5): fatal error C1083: Cannot open include file: 'cstdint': No such file or directory
error: command 'C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.35.32215\\bin\\Hostx64\\x64\\cl.exe' failed with exit code 2
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for simple-knn
error: subprocess-exited-with-error
× Running setup.py install for simple-knn did not run successfully.
│ exit code: 1
╰─> [28 lines of output]
running install
C:\Users\User\miniconda3\envs\gaussian_splatting\lib\site-packages\setuptools\_distutils\cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.
!!
********************************************************************************
Please avoid running ``setup.py`` directly.
Instead, use pypa/build, pypa/installer or other
standards-based tools.
See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
********************************************************************************
!!
self.initialize_options()
running build
running build_ext
C:\Users\User\miniconda3\envs\gaussian_splatting\lib\site-packages\torch\utils\cpp_extension.py:411: UserWarning: Attempted to use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the slow distutils backend.
warnings.warn(msg.format('we could not find ninja.'))
C:\Users\User\miniconda3\envs\gaussian_splatting\lib\site-packages\torch\utils\cpp_extension.py:813: UserWarning: The detected CUDA version (11.8) has a minor version mismatch with the version that was used to compile PyTorch (11.6). Most likely this shouldn't be a problem.
warnings.warn(CUDA_MISMATCH_WARN.format(cuda_str_version, torch.version.cuda))
building 'simple_knn._C' extension
creating build
creating build\temp.win-amd64-cpython-37
creating build\temp.win-amd64-cpython-37\Release
"C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.35.32215\bin\Hostx64\x64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD -IC:\Users\User\miniconda3\envs\gaussian_splatting\lib\site-packages\torch\include -IC:\Users\User\miniconda3\envs\gaussian_splatting\lib\site-packages\torch\include\torch\csrc\api\include -IC:\Users\User\miniconda3\envs\gaussian_splatting\lib\site-packages\torch\include\TH -IC:\Users\User\miniconda3\envs\gaussian_splatting\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\include" -IC:\Users\User\miniconda3\envs\gaussian_splatting\include -IC:\Users\User\miniconda3\envs\gaussian_splatting\Include /EHsc /Tpext.cpp /Fobuild\temp.win-amd64-cpython-37\Release\ext.obj /MD /wd4819 /wd4251 /wd4244 /wd4267 /wd4275 /wd4018 /wd4190 /EHsc /wd4624 -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=_C -D_GLIBCXX_USE_CXX11_ABI=0
ext.cpp
C:\Users\User\miniconda3\envs\gaussian_splatting\lib\site-packages\torch\include\torch\csrc\api\include\torch/cuda.h(5): fatal error C1083: Cannot open include file: 'cstdint': No such file or directory
error: command 'C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.35.32215\\bin\\Hostx64\\x64\\cl.exe' failed with exit code 2
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure
× Encountered error while trying to install package.
╰─> simple-knn
note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.
failed
CondaEnvException: Pip failed
If we move our views up and down, the color pattern will become strange. We could notice there is a plane seems like cut the whole scene half. The color becomes normal when our camera pose was reset to training view poses.
This phenomenon happens globally. This plane is located at certain height.
Besides, we noticed when rendering the initial point clouds, some points are disappeared when the camera moves up and down.
I am wandering if the issues were connected to raster? Are they actually the same problems?
Hello, I was surprised by your work and tried to reproduce it with the code you've provided.
However, every time I tried to run the code, it always failed to run with the runtime error i mentioned on the title.
Traceback (most recent call last):
File "train.py", line 213, in
training(lp.extract(args), op.extract(args), pp.extract(args), args.test_iterations, args.save_iterations, args.checkpoint_iterations, args.start_checkpoint)
File "train.py", line 87, in training
loss = (1.0 - opt.lambda_dssim) * Ll1 + opt.lambda_dssim * (1.0 - ssim(image, gt_image))
File "/home/seohoiki/Research/NeRF/gaussian-splatting/utils/loss_utils.py", line 38, in ssim
window = window.cuda(img1.get_device())
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Training progress: 0%| | 0/30000 [00:00<?, ?it/s]
I tried all the methods you've told in other issues, but failed.
My system & settings:
RTX4090
Ubuntu 22.04 LTS
Exact environment with given .yml file
Strangely, my colleague who has system with RTX 3090 / Ubuntu 20.04 runs the code without any problem.(Except them, all the settings are exactly the same including CUDA SDK version)
I hope I can get some solution for this problem!
Thank you.
=====================================
Results with cuda-memcheck
========= CUDA-MEMCHECK
========= This tool is deprecated and will be removed in a future release of the CUDA toolkit
========= Please use the compute-sanitizer tool as a drop-in replacement
Optimizing
Output folder: ./output/54877260-0 [17/07 19:21:51]
Tensorboard not available: not logging progress [17/07 19:21:51]
Found transforms_train.json file, assuming Blender data set! [17/07 19:21:51]
Reading Training Transforms [17/07 19:21:51]
Reading Test Transforms [17/07 19:21:53]
Loading Training Cameras [17/07 19:21:56]
Loading Test Cameras [17/07 19:21:57]
Number of points at initialisation : 100000 [17/07 19:21:57]
Training progress: 0%| | 0/30000 [00:00<?, ?it/s]Traceback (most recent call last):
File "train.py", line 213, in
training(lp.extract(args), op.extract(args), pp.extract(args), args.test_iterations, args.save_iterations, args.checkpoint_iterations, args.start_checkpoint)
File "train.py", line 87, in training
loss = (1.0 - opt.lambda_dssim) * Ll1 + opt.lambda_dssim * (1.0 - ssim(image, gt_image))
File "/home/seohoiki/Research/NeRF/gaussian-splatting/utils/loss_utils.py", line 38, in ssim
window = window.cuda(img1.get_device())
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Training progress: 0%| | 0/30000 [00:00<?, ?it/s]
========= ERROR SUMMARY: 0 errors
I saw fast culling option in SIBR_gaussianViewer_app.exe, it can speed up 2X the FPS in my scene (50fps -> 100fps), I wonder if I can also use this option in render.py?
Hi! I'm wondering if I can save density png sequence from the render process?
Something like this:
this is the conventional fox scene exported from instant ngp.exe
SIBR_remoteGaussian_app can be used to connect to training process on remote machine, but how can I use the remote machine to render remotely after rendering?
Hi, thanks for making this code available, it works very well.
I am looking to port the rendering code into a game engine, can you please explain how the rendering works with your model?
Which parts of SIBR are used?
Thanks!
Hi, I went through the setup of installing the code on new machine yesterday and have noticed the results now have a major drop in visual quality.
The test was the same dataset trained with the same settings, viewing the 7000 iteration.
My dataset is 59 source images at 3008 x 4112 and ran with the command -r 4
I tested copying my original gaussian-splatting project folder over to the new PC and ran the same training and the result was the same good quality as the original.
So my questions are;
Great works!
I tried to use the steps in the SIBR document to generate video and save frame, but I can't save anything. And the log shows that it does not seem to run the step of saving the video.
I would like to ask how to solve this problem？
Thank you very much for sharing the amazing work.
Is there any easy way to try random point initialization instead of SfM points as shown in Table 3 in the paper?
I don't find the argument options in train.py.
Sincerely,
After training my data, I use the SIBR_gaussianViewer_app.exe to render the trained model, but its black
Tensorboard not available: not logging progress [13/07 01:10:31]
Reading camera 56/56 [13/07 01:10:31]
Loading Training Cameras [13/07 01:10:31]
Loading Test Cameras [13/07 01:11:08]
Number of points at initialisation : 26558 [13/07 01:11:08]
Training progress: 23%|█████████████████████▏ | 7000/30000 [09:00<34:06, 11.24it/s, Loss=0.0516592]Traceback (most recent call last):
File "train.py", line 204, in <module>
training(lp.extract(args), op.extract(args), pp.extract(args), args.test_iterations, args.save_iterations)
File "train.py", line 99, in training
training_report(tb_writer, iteration, Ll1, loss, l1_loss, iter_start.elapsed_time(iter_end), testing_iterations, scene, render, (pipe, background))
File "train.py", line 162, in training_report
images = torch.cat((images, image.unsqueeze(0)), dim=0)
RuntimeError: Sizes of tensors must match except in dimension 0. Expected size 2588 but got size 2589 for tensor number 1 in the list.
Training progress: 23%|█████████████████████▏ | 7000/30000 [09:01<29:39, 12.93it/s, Loss=0.0516592]
Images are 56 of around 3451 x 5178, I tried running with the auto 1.6k scaling and also with -r 2 and still get same type of error
I think the subfolders ours_70000 in the image.zip are named incorrectly and should be ours_7000 ( 7k )
Can you confirm?
Error message;
(gaussian_splatting) C:\gaussian-splatting>python train.py -s C:\gaussian-splatting\data\test
Optimizing
Output folder: ./output/15709e19-a [10/07 13:19:45]
Tensorboard not available: not logging progress [10/07 13:19:45]
Reading camera 1/31Traceback (most recent call last):
File "train.py", line 204, in <module>
training(lp.extract(args), op.extract(args), pp.extract(args), args.test_iterations, args.save_iterations)
File "train.py", line 35, in training
scene = Scene(dataset, gaussians)
File "C:\gaussian-splatting\scene\__init__.py", line 44, in __init__
scene_info = sceneLoadTypeCallbacks["Colmap"](args.source_path, args.images, args.eval)
File "C:\gaussian-splatting\scene\dataset_readers.py", line 145, in readColmapSceneInfo
cam_infos_unsorted = readColmapCameras(cam_extrinsics=cam_extrinsics, cam_intrinsics=cam_intrinsics, images_folder=os.path.join(path, reading_dir))
File "C:\gaussian-splatting\scene\dataset_readers.py", line 95, in readColmapCameras
assert False, "Colmap camera model not handled!"
AssertionError: Colmap camera model not handled!
Not sure what to do here, its a camera with 24mm lens, colmap 3.8 aligned the 31 image loop fine, any suggestions?
Why is the scene_extent computed as the radius of camera center rather than for example something related to the initial pointcloud? In your implement, the cameras with the same trajectories but one set of camera facing inward and one set of cameras facing outward will have exactly the same scene_extent, but the former one may corresponds with a smaller scene.
Also the resulting scene_extent is used to scale up the position learning rate. I still quite not understand why the learning rate should be proportional to the radius of camera center.
Thank you very much for releasing the source code!
I see that the model is fully implemented in CUDA. However, for experimentation purposes, pure PyTorch implementation would be better.
Is there a plan to release the PyTorch model implementation?
Thank you!
During Training
anaconda3/envs/gaussian_splatting/lib/python3.7/site-packages/diff_gaussian_rasterization/__init__.py", line 78, in forward
num_rendered, color, radii, geomBuffer, binningBuffer, imgBuffer = _C.rasterize_gaussians(*args)
RuntimeError: numel: integer multiplication overflow
Training progress: 18%|██████████ | 17940/100000 [1:36:57<7:23:31, 3.08it/s, Loss=0.0742834]
Any idea how to solve this?
Current render on initialization (without training) (13th 4pm):
PSNR: 17.6
whereas yesterday (12th) at 4.30pm, the same render at initialization (without training) looked like this:
PSNR: 24.6
Did anything change when I repulled things? I don't think I did anything to make this change.. (but maybe I did)?
HI, i'm trying to run the provided trained images that has been available.
but isn't clear to me what commands to run it. It's possible to just writes the steps to run the available on images.zip file?
Can observe many floating foggy artifact training my data with default param, any idea how to reduce them?
With CUDA 12.0 and pytorch 2.0, it seems that simple-knn has the following build failure:
/home/dllu/builds/gaussian-splatting/submodules/simple-knn/simple_knn.cu(196): error: namespace "thrust" has no member "sequence"
It seems it is likely that will fail on other versions of CUDA as well but I haven't tried...
This is fixed with the following change:
diff --git a/simple_knn.cu b/simple_knn.cu
index 4828d2b..f99fee4 100644
--- a/simple_knn.cu
+++ b/simple_knn.cu
@@ -8,6 +8,7 @@
#include <vector>
#include <cuda_runtime_api.h>
#include <thrust/device_vector.h>
+#include <thrust/sequence.h>
#define __CUDACC__
#include <cooperative_groups.h>
@@ -207,4 +208,4 @@ void SimpleKNN::knn(int P, float3* points, float* meanDists)
boxMeanDist << <num_boxes, BOX_SIZE >> > (P, points, indices_sorted.data().get(), boxes.data().get(), meanDists);
cudaFree(result);
-}
\ No newline at end of file
+}(malheureusement je ne sais pas comment créer un PR sur https://gitlab.inria.fr/bkerbl/simple-knn)
Hi,
Thanks for providing the source code;
As I can see inside the scene class, you can only load Blender and Colmap datasets. Do you have any code for loading the Tank and Temple dataset?
Thanks.
In the software requirements of the setup is says;
CUDA SDK 11 for PyTorch extensions (we used 11.8, known issues with 11.6)
yet in the environment.yml is says;
dependencies:
can some one explain this please?
Hi ! Your work is truly amazing.
I have one question about the renderer, i would like to know where do you cull the 3D Gaussian on the tiles.
Thanks
Hi Thanks for sharing this great work. I wondering is there any ways to parse the geomBuffer, binningBuffer, imgBuffer returned by rasterize_gaussians? I tried but didn't figure out a correct ways to parse those buffers.
The tools for adding cameras to make a camera path and rendering it doesn't seem to work in the pre-built viewer, am I missing something?
I use SIBR remote viewer to connect to remote training server, and training server indicates the viewer is connected but the viewer app cannot find the model path, even i use --path to give the right model path in training server
could you please explain the following instructions, and that would be better if you can give me an example.
due to them running on different (virtual) machines), you may specify an override location to the viewer by using -s
Thanks
Error running;
conda env create --file environment.yml
message;
Installing pip dependencies: - Ran pip subprocess with arguments:
['C:\\Users\\admin\\.conda\\envs\\gaussian_splatting\\python.exe', '-m', 'pip', 'install', '-U', '-r', 'C:\\gaussian-splatting\\condaenv.5o1tubvd.requirements.txt', '--exists-action=b']
Pip subprocess output:
Pip subprocess error:
ERROR: Directory 'submodules/diff-gaussian-rasterization' is not installable. Neither 'setup.py' nor 'pyproject.toml' found.
failed
CondaEnvException: Pip failed
Because these modules are not installed I cannot run train.py
(gaussian_splatting) C:\gaussian-splatting>python train.py -s C:\gaussian-splatting\data\test
Traceback (most recent call last):
File "train.py", line 16, in <module>
from gaussian_renderer import render, network_gui
File "C:\gaussian-splatting\gaussian_renderer\__init__.py", line 14, in <module>
from diff_gaussian_rasterization import GaussianRasterizationSettings, GaussianRasterizer
ModuleNotFoundError: No module named 'diff_gaussian_rasterization'
These folders are empty, should they contain anything.. any suggestions?
Using the same model, it cost 6GB-16GB VRAM when use render.py to render on my A10 card. But on my 2080 card with only 8GB it can run the viewer smoothly. How could that happen?
Any idea to reduce the VRAM requirment when use render.py
Hi, the work is great! I'm trying to migrate the Algorithm to some other platform.
During reading your code, I notice that you implement a GPU memory pool in your rasterizer and use an alignment of 128. I'm curious about the idea behind this design decision. I'm wondering if the GPU memory pool help. Is there any data point/experiment result comparing using/not using the GPU memory pool?
Thanks!
I get the following error upon running "cmake -Bbuild ." while trying to install SIBR:
CMake Error at cmake/linux/dependencies.cmake:248 (find_package):
Could not find a configuration file for package "OpenCV" that is compatible
with requested version "4.5".
The following configuration files were considered but not accepted:
/usr/lib/x86_64-linux-gnu/cmake/opencv4/OpenCVConfig.cmake, version: 4.2.0
/lib/x86_64-linux-gnu/cmake/opencv4/OpenCVConfig.cmake, version: 4.2.0
Call Stack (most recent call first):
cmake/linux/include_once.cmake:20 (include)
src/CMakeLists.txt:46 (include_once)
I am not very familiar with cmake, so although I am sure there is a quick and easy fix, I am not able to easily find it.
hi I'm running it like so:
python train.py --ip 0.0.0.0 --port 13379 -s $colmap -m $outdir
python render.py -m $outdir
The logs from train.py look like this:
Optimizing /home/danlu/debug/gaussian-splatting-2023-07-10
Output folder: /home/danlu/debug/gaussian-splatting-2023-07-10 [11/07 13:56:37]
Reading camera 1130/1130 [11/07 13:56:39]
Loading Training Cameras [11/07 13:56:39]
Loading Test Cameras [11/07 13:56:52]
Number of points at initialisation : 532404 [11/07 13:56:52]
Training progress: 23%|████████████████▌ | 7000/30000 [01:45<04:24, 86.86it/s, Loss=0.2195082]
[ITER 7000] Evaluating train: L1 0.2186211347579956 PSNR 12.067031860351562 [11/07 13:58:39]
[ITER 7000] Saving Gaussians [11/07 13:58:39]
Training progress: 100%|█████████████████████████████████████████████████████████████████████▉| 30000/30000 [06:19<00:00, 78.99it/s, Loss=0.2308751]
[ITER 30000] Saving Gaussians [11/07 14:03:11]
Training complete. [11/07 14:03:13]
But the output looks very bad, almost all the frames in `$outdir/ are like this:
or like this:
and only a few frames (around 5 out of over 1000 frames) have any sort of recognizable structure in them.
On the other hand, using taichi gaussian splatting with the exact same dataset, the output looks very reasonable.
Does anyone have any ideas for the parameters that I might have to change?
Hi I'm using the windows Viewer and this command;
C:/gaussian-splatting/SIBR_viewers_win/bin/SIBR_gaussianViewer_app.exe -m C:/gaussian-splatting/output/truck --iteration 7000
And getting the error;
[SIBR] -- INFOS --: Initialization of GLFW
[SIBR] -- INFOS --: OpenGL Version: 4.6.0 NVIDIA 528.95[major: 4, minor: 6]
Number of input Images to read: 59
Number of Cameras set up: 59
LOADSFM: Try to open C:\\gaussian-splatting\\data\\truck/sparse/0/points3D.bin
Num 3D pts 47989
[SIBR] -- INFOS --: SfM Mesh 'C:\\gaussian-splatting\\data\\truck/sparse/0/points3d.bin successfully loaded. (47989) vertices detected. Init GL ...
[SIBR] -- INFOS --: Init GL mesh complete
[SIBR] ## ERROR ##: FILE C:\projects\gauss2\SIBR_viewers\src\projects\gaussianviewer\renderer\GaussianView.cpp
LINE 75, FUNC loadPly
Unable to find model's PLY file, attempted:
C:/gaussian-splatting/output/truck/point_cloud/iteration_7000 / point_cloud.ply
It looks like it is referencing where the project was created C:\projects\gauss2\SIBR_viewers Does it need compiling a different way so it doesn't need to reference these files?
Hi, datasets I am testing do not align well in colmap, is there an alternative format?
Hi,
Thanks for your work. I just want to know whether your code support the "restore from specific iteration steps"? For example, I have stored the point clould for 10K stpes. Now, I want to load this point cloud information and restore training from it. Does your code support this?
Thanks in advance.
Hi, is it possible to edit the noise and floaters from the trained .ply file?
I have opened the file in CloudCompare and can see that there are many headers with layers of information. Is there a way to edit the clouds in one go, crop them to a bounding box and save them back to the trained file type?
The subject in the bounding box is what I would like to crop to, as you can see there are a lot of floaters way out of the main scene area.
Alternatively, is there a way to constrain the splatting much closer to the input point cloud? I have tried swapping out the input cloud to one that has been edited so that just the subject is isolated but training still creates noise and floaters outside of this area, is this expected? We would also prefer to avoid masking.
And finally, is there an aabb_crop in the viewer it self, so the noise is still in the file but not rendered?
traceback (most recent call last):
File "train.py", line 204, in <module>
training(lp.extract(args), op.extract(args), pp.extract(args), args.test_iterations, args.save_iterations)
File "train.py", line 75, in training
render_pkg = render(viewpoint_cam, gaussians, pipe, background)
File "C:\gaussian-splatting\gaussian_renderer\__init__.py", line 92, in render
cov3D_precomp = cov3D_precomp)
File "C:\Users\admin\.conda\envs\gaussian_splatting\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "C:\Users\admin\.conda\envs\gaussian_splatting\lib\site-packages\diff_gaussian_rasterization\__init__.py", line 194, in forward
raster_settings,
File "C:\Users\admin\.conda\envs\gaussian_splatting\lib\site-packages\diff_gaussian_rasterization\__init__.py", line 37, in rasterize_gaussians
raster_settings,
File "C:\Users\admin\.conda\envs\gaussian_splatting\lib\site-packages\diff_gaussian_rasterization\__init__.py", line 78, in forward
num_rendered, color, radii, geomBuffer, binningBuffer, imgBuffer = _C.rasterize_gaussians(*args)
RuntimeError: Storage size calculation overflowed with sizes=[-2139761793]
Training progress: 20%|████████ | 6020/30000 [30:05<1:59:53, 3.33it/s, Loss=0.0507467]
Running 56 of 3450 x 5180 images with -r 1.
Using a6000 with 48gb, is there anyway to calculate the limit of whats possible to run to which iteration?
Hi, I'm getting these types of artifacts . Does anyone know what causes it and how to reduce it ?
Hallo, Thanks for the great work!
I saw the remote Gaussian visualizer return the message do_shs_python, and the assigned params is not used anywhere.
Line 57 in add0ada
and the param is assigned to pipe the config for pipeline, and there has been a param convert_SHs_python.
gaussian-splatting/arguments/__init__.py
Lines 64 to 69 in add0ada
Is there any relation between the 2 params? what is the logic behind it?
I used viewer downloaded in 07/12, I all worked fine. After downloading the latest version of viewer yesterday, the viewer just could not open the scene (a blank viewer pop out and disappear after a while). The logs are pasted below, I am using RTX2080, any idea what could cause this?
CMD:
.\bin\SIBR_gaussianViewer_app.exe -m .\gaussian-splatting\output\shop\ -s .\gs_dataset\dataset\shop\ --rendering-size 1920 1080
latest:
[SIBR] -- INFOS --: Initialization of GLFW
[SIBR] -- INFOS --: OpenGL Version: 4.6.0 NVIDIA 516.59[major: 4, minor: 6]
previous:
[SIBR] -- INFOS --: Initialization of GLFW
[SIBR] -- INFOS --: OpenGL Version: 4.6.0 NVIDIA 516.59[major: 4, minor: 6]
Number of input Images to read: 802
Number of Cameras set up: 802
LOADSFM: Try to open D:\workspace\SotaNeRFs\gs_dataset\dataset\shop_800\/sparse/0/points3D.bin
Num 3D pts 99869
[SIBR] -- INFOS --: SfM Mesh 'D:\workspace\SotaNeRFs\gs_dataset\dataset\shop_800\/sparse/0/points3d.bin successfully loaded. (99869) vertices detected. Init GL ...
[SIBR] -- INFOS --: Init GL mesh complete
[SIBR] -- INFOS --: Loading 1193241 Gaussian splats
[SIBR] -- INFOS --: Initializing Raycaster
[SIBR] -- INFOS --: Interactive camera using (0.009,1100) near/far planes.
Switched to trackball mode.
[SIBR] -- INFOS --: Deinitialization of GLFW
see this line:
this used to be set based on self.spatial_lr_scale = spatial_lr_scale, and thus setting it to the initial radius calculated in data preproc.
Now it is hard coded to 5.
Is this better?
Hey, thanks a lot for releasing the code for such an outstanding paper! Couldn't wait to try it out on my data.
Unfortunately, I bumped into this:
Anyway, maybe anyone has any idea how to fix the following problem during the compilation of SIBR?
[ 89%] Linking CXX executable SIBR_texturedMesh_app
/usr/bin/ld: ../../../../core/system/libsibr_system.so: undefined reference to `gtk_file_chooser_set_do_overwrite_confirmation'
/usr/bin/ld: ../../../../core/system/libsibr_system.so: undefined reference to `gtk_main_iteration'
/usr/bin/ld: ../../../../core/system/libsibr_system.so: undefined reference to `gtk_widget_destroy'
/usr/bin/ld: ../../../../core/system/libsibr_system.so: undefined reference to `gtk_dialog_run'
/usr/bin/ld: ../../../../core/system/libsibr_system.so: undefined reference to `gtk_file_chooser_set_select_multiple'
/usr/bin/ld: ../../../../core/system/libsibr_system.so: undefined reference to `gtk_file_chooser_set_current_folder'
/usr/bin/ld: ../../../../core/system/libsibr_system.so: undefined reference to `gtk_file_chooser_add_filter'
/usr/bin/ld: ../../../../core/system/libsibr_system.so: undefined reference to `gtk_file_filter_new'
/usr/bin/ld: ../../../../core/system/libsibr_system.so: undefined reference to `gtk_file_chooser_get_filenames'
/usr/bin/ld: ../../../../core/system/libsibr_system.so: undefined reference to `gtk_dialog_get_type'
/usr/bin/ld: ../../../../core/system/libsibr_system.so: undefined reference to `gtk_file_filter_add_pattern'
/usr/bin/ld: ../../../../core/system/libsibr_system.so: undefined reference to `gtk_file_chooser_get_type'
/usr/bin/ld: ../../../../core/system/libsibr_system.so: undefined reference to `gtk_file_chooser_get_filename'
/usr/bin/ld: ../../../../core/system/libsibr_system.so: undefined reference to `gtk_init_check'
/usr/bin/ld: ../../../../core/system/libsibr_system.so: undefined reference to `gtk_file_filter_set_name'
/usr/bin/ld: ../../../../core/system/libsibr_system.so: undefined reference to `gtk_events_pending'
/usr/bin/ld: ../../../../core/system/libsibr_system.so: undefined reference to `gtk_file_chooser_dialog_new'
collect2: error: ld returned 1 exit status
make[2]: *** [src/projects/basic/apps/texturedMesh/CMakeFiles/SIBR_texturedMesh_app.dir/build.make:202: src/projects/basic/apps/texturedMesh/SIBR_texturedMesh_app] Error 1
make[1]: *** [CMakeFiles/Makefile2:1335: src/projects/basic/apps/texturedMesh/CMakeFiles/SIBR_texturedMesh_app.dir/all] Error 2
make: *** [Makefile:136: all] Error 2
The thing is, I have libgtk3-dev correctly installed
dpkg -l libgtk-3-dev
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==================-==================-============-=====================================
ii libgtk-3-dev:amd64 3.24.20-0ubuntu1.1 amd64 development files for the GTK library
and cmake also finds it during configuration:
-- Checking for module 'gtk+-3.0'
-- Found gtk+-3.0, version 3.24.20
Great work! When I train the model using different sh degrees like 1 or 2 and use the viewer to visualize the result. It seems that the screen all turns black. When the sh degree of the training model is 3, it works well. Did I mix up something? Or is the viewer only supporting the sh degree at 3? Thank you very much!
After set up folder as /input with images in there (tested with jpg and png files), and running: python .\convert.py -s <location> --resize, the script fails with the error stated on the title:
Traceback (most recent call last):
File ".\convert.py", line 63, in <module>
files = os.listdir(args.source_path + "/sparse")
FileNotFoundError: [WinError 3] The system cannot find the path specified: '<location>/sparse'
If I create the folder '<location>/sparse' manually, then it creates images_N folders and fails with the following error:
Copying and resizing...
Traceback (most recent call last):
File ".\convert.py", line 81, in <module>
files = os.listdir(args.source_path + "/images")
FileNotFoundError: [WinError 3] The system cannot find the path specified: '<location>/images'
Finally, If again <location>/images folder is created, the script doesn't fail but it doesn't do anything as well:
Copying and resizing...
Done.
Tested on latest main branch, on windows 11.
BTW thank you for the great work, and for sharing the source code, amaizing!
Hi,
Thanks a lot for sharing your great work. I've built the environment properly and while running the tanks and temples dataset, getting following error:
Training progress: 0%| | 0/30000 [00:00<?, ?it/s]Traceback (most recent call last):
File "train.py", line 204, in
training(lp.extract(args), op.extract(args), pp.extract(args), args.test_iterations, args.save_iterations)
File "train.py", line 81, in training
loss = (1.0 - opt.lambda_dssim) * Ll1 + opt.lambda_dssim * (1.0 - ssim(image, gt_image))
File "/media/sdc/merf_research/gaussian_mixture/gaussian-splatting/utils/loss_utils.py", line 38, in ssim
window = window.cuda(img1.get_device())
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Any idea how to resolve? My environment matches exactly with your repo
Hi. I have the SIBR Viewer setup on an Ubuntu 22.04 machine which fully satisfies the mentioned requirements (hardware and software). The required dependencies for the SIBR viewer are there and the build finished successfully.
In order to run the GUI-based Remote Gaussian SIBR Viewer, I SSH'ed into my server using X11 on MacOS and XMing (with PuTTY) on Windows with the -X flag in the ssh command and ensured that the DISPLAY env variable is properly configured.
Upon running the following command: ./SIBR_remoteGaussian_app command, I get the following output:
MacOS
[SIBR] -- INFOS --: Initialization of GLFW
libGL error: No matching fbConfigs or visuals found
libGL error: failed to load driver: swrast
[SIBR] ## ERROR ##: FILE /home/paperspace/gaussian-splatting/SIBR_viewers/src/core/graphics/Window.cpp
LINE 30, FUNC glfwErrorCallback
GLX: An OpenGL profile requested but GLX_ARB_create_context_profile is unavailable
terminate called after throwing an instance of 'std::runtime_error'
Window
[SIBR] -- INFOS --: Initialization of GLFW
[SIBR] ## ERROR ##: FILE /home/paperspace/gaussian-splatting/SIBR_viewers/src/core/graphics/Window.cpp
LINE 30, FUNC glfwErrorCallback
GLX: GLX version 1.3 is required
terminate called after throwing an instance of 'std::runtime_error'
Server Specifications:
OS: Ubuntu 22.04
GPU: A100
Hi,
Thanks for making a great contribution to the area. I am wondering do you have Python rasterization code, which is easier to read and redevelop. Thank you.
the red part is the only valid part, the rest seems out side of camera. In my opinion, your method shoud produce a more valid pointcloud since using splatting as rendering method?
Any idea how to generate a more valid point cloud in future work?
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.