This Project aims to provide an executable application for rendering Neural Radiance Fields in realtime. This is achieved by reimplementing the NeRF functionalities of Nvidias Instant-NGP and rendering the resulting images in realtime using the 3D-Library FUSEE. Since FUSEE uses the .NET Framework, an interface for working with matrices stored on the GPU and managed C# Code is needed. Therefore, this project makes use of TorchSharp and follows other reimplementations of Instant-NGP such as xrnerf and torch-ngp. Tiny-cuda-nn's networks are used to further increase the performance.

• Open the .sln file in MS Visual Studio
• specify the launch parameters in Config/config.json, add a path to the dataset
• Instant-NGPs dataset and the original NeRF synthetic datasets are supported, although most testing has been done on the synthetic lego dataset

{
    "dataPath": "D:\\path\\to\\data folder\\containing\\transforms.json file",
    "trainDataFilename": "transforms_train",
    "evalDataFilename": "transforms_val",
    "datasetType": "synthetic",
    "aabbMin": 0.0,
    "aabbMax": 1.0,
    "aabbScale": 0.33,
    "offset": [ 0.5, 0.5, 0.5 ],
    "bgColor": [ 1.0, 1.0, 1.0 ],
    "useRandomBgColor":  true,
    "imageDownscale": 4.0,
    "nRays": 4096,
    "stepsToTrain":  3000,
    "gradScale": 1.0,
    "optimizerCfg": {
        "otype": "Ema",
        "decay": 0.95,
        "nested": {
            "otype": "ExponentialDecay",
            "decay_start": 20000,
            "decay_interval": 10000,
            "decay_base": 0.33,
            "nested": {
                "otype": "Adam",
                "learning_rate": 1e-2,
                "beta1": 0.9,
                "beta2": 0.99,
                "epsilon": 1e-15,
                "l2_reg": 1e-6
            }
        }
    },
    "posEncodingCfg": {
        "otype": "HashGrid",
        "n_levels": 8,
        "n_features_per_level": 4,
        "log2_hashmap_size": 19,
        "base_resolution": 16
    },
    "dirEncodingCfg": {
        "n_dims_to_encode": 3,
        "otype": "SphericalHarmonics",
        "degree": 4
    },
    "sigmaNetCfg": {
        "otype": "FullyFusedMLP",
        "activation": "ReLU",
        "output_activation": "None",
        "n_neurons": 64,
        "n_hidden_layers": 1
    },
    "colorNetCfg": {
        "otype": "FullyFusedMLP",
        "activation": "ReLU",
        "output_activation": "None",
        "n_neurons": 64,
        "n_hidden_layers": 2
    }
}

• rebuild the Solution or just the FuseeNeRF project for Release x64 and launch \FuseeNeRF\Build\x64\Release\net7.0-windows\FuseeApp.exe

• Windows x64
• An Nvidia GPU, currently only GPUs with compute capability >= 70 are supported (tested on RTX 3070)
• A valid installation of CUDA 11.7
  • Add \NVIDIA GPU Computing Toolkit\CUDA\v11.7 to a CUDA_PATH environment variable
• Requires libtorch 2.0.1: download libtorch or pytorch with libtorch 2.0.1 backend here
  • Add \libtorch directory to a new TORCH_DIR environment variable
• A valid installation of MS Visual Studio (tested with Visual Studio 2022 Community, other versions may work too)
• A valid installation of .NET 7 and the latest C++ Redistributables (can be installed together with MS Visual Studio)

• Build Nvidias tiny-cuda-nn project by following their instructions and copy the resulting tiny-cuda-nn.lib to FuseeNeRF/TcnnApi/dependencies/Libraries
• Open the project in Visual Studio to install all C# dependencies
• Build the project and launch FuseeApp.exe

This application is designed to be a real-time 3D Viewer for Neural Radiance Fields in C#, extending the capabilities of the game engine FUSEE. The project reimplements some features of Instant-NGP and makes them usable in the .NET framework. Rendering Neural Radiance Fields in real-time requires different optimaztion steps. The Hash grid encoding provided by instant-ngp, in combination with their optimized rendering algorithms and the optimized networks tiny-cuda-nn allow for high performant NeRF rendering. The three optimizations are accessed with the help of CUDA APis, which can be called in .NET thanks to the interoperability between C# and C++/CUDA. As an interface for using the data in the context of C# and CUDA, TorchSharp is used. The training data is loaded in C#, and passed as rays via an API to a CUDA kernel to generate training samples and update a density grid and bitfield. These samples are passed to an API using tiny-cuda-nn in combination with instant-ngp's NeRF implementation to query a neural network with a highly optimized input encoding. The generated results can be used in volume rendering kernels to calculate an image, or comapare the data to ground truth values to calculate a loss and gradients. The gradients are propagated backward and finally passed to the tiny-cuda-nn API to be used by their Adam implementation to update the networks parameters.

FuseeNeRF
├── FuseeApp
│   ├── FuseeNeRF
│   ├── Program
├── InstantNeRF
│   ├── Autograd & Context
│   ├── Config
│   ├── DataProvider
│   ├── GridSampler
│   ├── MLP & Tcnn Nerf Apis
│   ├── Network
│   ├── Optimizer
│   ├── VolumeRenderer
│   ├── Trainer
│   ├── Utils
├── RaymarchApi
│   ├── CUDA kernels & headers
│   ├── API
├── TcnnNerfApi
│   ├── Module
│   ├── NerfNetwork
│   ├── Optimizer
│   ├── API
...
├── Build
│   ├── x64\Release\net7.0-windows\FuseeApp.exe
├── Config
│   ├── config.json

The viewer is able to render images at 200x200 pixels up to 800x800 pixels at ~7 frames pare second during live training and images witha resolution of 100x100 at ~60 frames per second during inference after the model stopped training. This seems related to a bug caused by the camera pose, ray generation/marching, the density bitfield or a combination of these issues. At resolutions > 100x100, more and more rays seem to be skipped, by either not hitting the bounding box or by the bitfield being calculated wrongly due to a wrong camera pose transformation. Further testing has to be done. For reference, consider reading the following files:

• matrixToNGP() in Utils.cs & nerf_loader.cu, nerf_loader.h, matrix_nerf2ngp for pose transformation
• getRaysFromPose() in Utils.cs & get_rays.py for ray generation

This issue also causes the controls of the viewer to work incorrectly, so a default pose is set.

• Fixing the issues related to the pose
• User Interface

This project uses code from various codebases and is limited by their restrictions. The licenses can found in the corresponding subdirectories

Credits and thanks to:

• Instant-NGP
• tiny-cuda-nn
• xrnerf
• torch-ngp
• TorchSharp
• FUSEE