Git Product home page Git Product logo

x-kanerf's Introduction

X-KANeRF [KANeRF-benchmarking]: KAN-based NeRF with Various Basis Functions

Is there any basis function can explain the NeRF formula?!

$$\mathbf{c}, \sigma = F_{\Theta}(\mathbf{x}, \mathbf{d}),$$ where $\mathbf{c}=(r,g,b)$ is RGB color, $\sigma$ is density, $\mathbf{x}$ is 3D position, $\mathbf{d}$ is the direction.

To explore this issue, I used Kolmogorov-Arnold Networks (KAN) with various basis functions to fit the NeRF equation based on nerfstudio.

The code might be a bit COARSE, any suggestions and criticisms are welcome!

X-KAN Models (Here are various KANs!)

TODO Basis Functions Mathtype Acknowledgement
1 B-Spline $$S_i(x) = a_i + b_i(x - x_i) + c_i(x - x_i)^2 + d_i(x - x_i)^3$$ Efficient-Kan
2 Fourier $$\phi_k(x) = \sin(2\pi kx), \phi_k(x) = \cos(2\pi kx)$$ FourierKAN
3 Gaussian RBF $$\phi(x, c) = e^{-\frac{|x - c|^2}{2\sigma^2}}$$ FastKAN
4 Radial Basis Function $$\phi(x, c) = f(|x - c|)$$ RBFKAN
5 FCN - FCN-KAN
6 FCN-Interpolation - FCN-KAN
7 1st Chebyshev Polynomials $$T_n(x) = \cos(n \cos^{-1}(x))$$ ChebyKAN
8 2nd-Chebyshev Polynomials $$U_n(x) = \frac{\sin((n+1)\cos^{-1}(x))}{\sin(\cos^{-1}(x))}$$ OrthogPolyKANs
9 Jacobi polynomials $$P_n^{(\alpha, \beta)}(x) = \frac{1}{2^n n!} \frac{d^n}{dx^n} \left[ (1-x)^{\alpha+n} (1+x)^{\beta+n} \right]$$ JacobiKAN
10 Hermite polynomials $$H_n(x) = (-1)^n e^{x^2} \frac{d^n}{dx^n}(e^{-x^2})$$ OrthogPolyKANs
11 Gegenbauer polynomials $$C_{n+1}^{(\lambda)}(x) = \frac{2(n+\lambda)}{n+1}x C_n^{(\lambda)}(x) - \frac{(n+2\lambda-1)}{n+1}C_{n-1}^{(\lambda)}(x)$$ OrthogPolyKANs
12 Legendre polynomials $$P_n(x) = \frac{1}{2^n n!} \frac{d^n}{dx^n} \left( x^2 - 1 \right)^n$$ OrthogPolyKANs
13 Laguerre polynomials $$L_n(x) = \frac{e^x}{n!} \frac{d^n}{dx^n} \left( x^n e^{-x} \right)$$ OrthogPolyKANs
14 Bessel polynomials $$J_n(x) = \sum_{k=0}^{\infty} \frac{(-1)^k}{k!(n+k)!} \left( \frac{x}{2} \right)^{2k+n}$$ OrthogPolyKANs
15 Fibonacci polynomials $$F_n(x) = xF_{n-1}(x) + F_{n-2}(x), \quad \text{for } n \geq 2.$$ OrthogPolyKANs
16 Lucas polynomials $$L_n(x) = xL_{n-1}(x) + L_{n-2}(x)$$ OrthogPolyKANs
17 Mexican hat wavelet $$\psi(x) = \frac{2}{\sqrt{3a}\pi^{\frac{1}{4}}} \left(1 - \frac{x^2}{a^2}\right) e^{-\frac{x^2}{2a^2}}$$ Wav-KAN
18 Morlet wavelet (Gabor wavelet) $$\psi(t) = \pi^{-\frac{1}{4}} e^{i\omega_0 t} e^{-\frac{t^2}{2}}$$ Wav-KAN
19 Difference of Gaussians(DoG) $$\text{DoG}(x, y) = \frac{1}{\sqrt{2\pi}\sigma_1} e^{-\frac{x^2 + y^2}{2\sigma_1^2}} - \frac{1}{\sqrt{2\pi}\sigma_2} e^{-\frac{x^2 + y^2}{2\sigma_2^2}}$$ Wav-KAN
20 Meyer wavelet $$\psi(x) = \sqrt{\frac{2}{T}} \sum_{k=1}^{N} \left(1 - \left(\frac{k}{N}\right)^2\right) \left[ \cos\left(\frac{2\pi x k}{T}\right) - \frac{\sin(\pi x k / T)}{\pi x k / T}\right]$$ Wav-KAN
21 Shannon wavelet $$\psi(t) = \frac{\sin(\pi t) - \sin\left(\frac{\pi t}{2}\right)}{\pi t}$$ Wav-KAN
22 Bump wavelet $$\psi(t) = e^{-\frac{1}{1 - t^2}}$$ Wav-KAN
More and More!!! - - -

Performance Comparision on RTX-3090

Model hidden_dim hidden_dim_color num_layers num_layers_color geo_feat_dim appearance_embed_dim
Nefacto-MLP-A 32 32 2 2 7 8
Nefacto-MLP-B 8 8 8 8 7 8
Others 8 8 1 1 7 8
  • nerf_synthetic: lego / 30k

Note that the current Train Rays/Sec and Train Time(ETA Time) are not accurate, they are the values ​​when the number of iterations reaches 100.

Model Layer Params $\downarrow$ Train Rays/Sec $\uparrow$ Train Time $\downarrow$ FPS $\uparrow$ PSNR $\uparrow$ SSIM $\uparrow$ LPIPS $\downarrow$
Nerfacto-MLP-A 9902 ~170K ~14m 0.71 32.53 0.968 0.0167
Nerfacto-MLP-B 3382 ~165K ~14m 0.75 27.11 0.915 0.0621
Nerfacto-MLP 1118 ~190K ~13m 0.99 28.60 0.952 0.0346
BSplines-KAN 8092 ~37K ~54 m 0.19 32.33 0.965 0.0174
GRBF-KAN 3748 ~115K ~19 m 0.50 32.39 0.967 0.0172
RBF-KAN 3512 ~140K ~15m 0.71 32.57 0.966 0.0177
Fourier-KAN 5222 ~80K ~25 m 0.42 31.72 0.956 0.0241
FCN-KAN(Iters: 4k) 5184 ~4K ~90m 0.02 29.67 0.938 0.0401
FCN-Interpolation-KAN 6912 ~52K ~40m 0.21 32.67 0.965 0.0187
1st Chebyshev-KAN 4396 ~53K ~40m 0.34 28.56 0.924 0.0523
1st Chebyshev-KAN(faster) 4426 ~86K ~23m 0.45 28.54 0.924 0.0514
Jacobi-KAN 3532 ~72K ~30m 0.37 27.88 0.915 0.0553
Bessel-KAN 3532 ~76K ~28m 0.33 25.79 0.878 0.1156
2nd Chebyshev-KAN 4396 ~55K ~39m 0.33 28.53 0.924 0.0500
Fibonacci-KAN 4396 ~65K ~32m 0.34 28.30 0.922 0.0521
Gegenbauer-KAN 4396 ~53K ~40m 0.32 28.39 0.922 0.0514
Hermite-KAN 4396 ~55K ~38m 0.37 27.58 0.913 0.0591
Legendre-KAN 4396 ~55K ~38m 0.33 26.64 0.893 0.0986
Lucas-KAN 3532 ~75K ~28m 0.42 27.95 0.916 0.0550
Laguerre-KAN 3532 ~74K ~28m 0.39 27.39 0.912 0.0593
MexicanHat-KAN 3614 ~66K ~32m 0.35 31.23 0.961 0.0221
Morlet-KAN 3614 ~67K ~31m 0.38 13.06 0.686 0.2583
DoG-KAN 3614 ~75K ~28m 0.41 32.59 0.966 0.0174
Meyer-KAN 3614 ~36K ~55m 0.17 11.91 0.728 0.2991
Shannon-KAN 3614 ~73K ~28m 0.49 9.15 0.738 0.4434
Bump-KAN 3614 ~70K ~28m 0.33 1.79 0.147 0.9072
# create python env
conda create --name nerfstudio -y python=3.8
conda activate nerfstudio
python -m pip install --upgrade pip

# install torch
pip install torch==2.0.1+cu117 torchvision==0.15.2+cu117 --extra-index-url https://download.pytorch.org/whl/cu117
conda install -c "nvidia/label/cuda-11.7.1" cuda-toolkit

# install tinycudann
pip install ninja git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch

# install nerfstudio
pip install nerfstudio==0.3.4

# pip install torchmetrics==0.11.4

# Tab command
ns-install-cli

# !!! If you use `ns-process-data`, please install this version opencv
pip install opencv-python==4.3.0.36

Run

# Train
############# kan_basis_type #############
# mlp, bspline, grbf, rbf, fourier,
# fcn, fcn_inter, chebyshev, jacobi
# bessel, chebyshev2, finonacci, hermite
# legendre, gegenbauer, lucas, laguerre
# mexican_hat, morlet, dog, meyer, shannon, bump
bash train_blender.sh [kan_basis_type]

# eval
bash run_eval.sh [exp_path]

# render RGB & Depth
bash run_render.sh [exp_path]

Docs

Citation

If you use this benchmark in your research, please cite this project.

@misc{xkanerf,
	title={X-KANeRF: KAN-based NeRF with Various Basis Functions},
	author={Linfei Li},
	howpublished = {\url{https://github.com/lif314/X-KANeRF}},
	year={2024}
}

Acknowledgement

  • KANeRF, A big thank you for this awesome work!
    @Manual{kanerf,
    	title = {Hands-On NeRF with KAN},
    	author = {Delin Qu, Qizhi Chen},
    	year = {2024},
    	url = {https://github.com/Tavish9/KANeRF},
    }
  • nerfstudio
     @inproceedings{nerfstudio,
     	title = {Nerfstudio: A Modular Framework for Neural Radiance Field Development},
     	author = {
     		Tancik, Matthew and Weber, Ethan and Ng, Evonne and Li, Ruilong and Yi, Brent
     		and Kerr, Justin and Wang, Terrance and Kristoffersen, Alexander and Austin,
     		Jake and Salahi, Kamyar and Ahuja, Abhik and McAllister, David and Kanazawa,
     		Angjoo
     	},
     	year = 2023,
     	booktitle = {ACM SIGGRAPH 2023 Conference Proceedings},
     	series = {SIGGRAPH '23}
     }

x-kanerf's People

Contributors

lif314 avatar

Stargazers

YiLiu avatar  avatar MinBLi avatar Xiaobing Han avatar Yining Jiao avatar  avatar Ruihan Xu avatar Hangwei Zhang avatar Mingyue Tang avatar Nam Le avatar Chenze avatar zzz avatar vignesh avatar Xiangkun Zhao avatar Zongyue Yang avatar Rekkles avatar  avatar Alireza Hosseini avatar Amirhossein Kazerouni avatar Adrian Azzarelli avatar  avatar  avatar jingfanwang avatar Matan Mizrachi avatar  avatar  avatar samwang avatar  avatar Will_Chen avatar  avatar wincent avatar  avatar yuzy avatar Zhiqi Li avatar Jaeah Lee avatar Jaerin Lee avatar  avatar  avatar  avatar  avatar  avatar  avatar Charles Li avatar Du Guanghao avatar  avatar  avatar Yufei Wang avatar kyouhimesama avatar  avatar Dong Sui (隋栋) avatar  avatar huangfeng avatar Qingtian Zhu avatar 爱可可-爱生活 avatar Jan avatar Zhuoqian Yang avatar  avatar Ruichen Geng avatar Blue-Giant avatar  avatar  avatar  avatar  avatar  avatar zzccchen avatar huangzhouzhou250 avatar KaiyuanTang avatar  avatar  avatar  avatar Ryan Missel avatar zzzzzz avatar  avatar Aryan avatar  avatar  avatar  avatar Tanora Vic avatar Haoqin Hong avatar AnYouCai avatar  avatar  avatar Julien avatar  avatar  avatar  avatar Jaehyeon Lee avatar oxyhexagen avatar Rocky avatar santis_yang avatar  avatar  avatar ITRookie avatar  avatar  avatar Zhang Tony avatar llhhtt7788 avatar yuno avatar  avatar Zhen Liu avatar

Watchers

Dong Sui (隋栋) avatar  avatar Ligen avatar  avatar  avatar

x-kanerf's Issues

Nice job + Any insights you can share?

Thank you for the deeper model comparisons and results!
I was doing some playing about with KANeRF architectures (on my own) and was struggling to get decent results and low training time, so this is super helpful for knowing whether they can be applied.

I was wondering if we can expect other tests in the paper and/or whether there are any other cool insights you could share about KAN-based architectures? For instance, KAN seems to be good at overfitting scenes - might this work better for sparse-view scenes?

~ I acknowledge that you probably want to keep some details for the paper, so I hope I'm not being too nosy ~

Update ChebyKAN

ChebyKAN has a faster vectorized implemention not using the recursive form you have in your implementation

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.