Silhouette Rendering about splatam HOT 4 CLOSED

Hi, Thanks for your interest in our work!

The Silhouette is basically the cumulative opacity along a ray. This is naturally obtained by the alpha-compositing in the rasterization of 3D Gaussians.

Instead of implementing the gradient for the alpha-compositing in CUDA. Here, we render the Silhouette by considering it one channel of dummy color, where the other two channels are depth and depth squared. We essentially assign a color of 1 to each Gaussian. Let's say there are 3 Gaussians along a ray; the cumulative opacity will be given by (Opacity_Gaussian_1 * 1 + Opacity_Gaussian_2 * 1 + Opacity_Gaussian_3 * 1). On the other hand, in the original 3D Gaussian Splatting color rendering, the following would happen: (Opacity_Gaussian_1 * (r, g, b) + Opacity_Gaussian_2 * (r, g, b) + Opacity_Gaussian_3 * (r, g, b)).

In this fashion, for each pixel, you will directly get a cumulative opacity along the ray, which is the Silhouette. For empty space, the value will be 0 because no Gaussians exist along the ray.

from splatam.

Thank u for your explanation. I have understood the principle by thinking in terms of the α-compositing.

from splatam.

Hi, thanks for your question and the author`s explanation.
I want to get the detailed formula of Silhouette as equation(5) furthermore in the SplaTAM paper.
Firstly, equation(1) in the paper tells us each Gaussian in the 3D space $\mathbf{x} \in \mathbb{R}^{3}$ :

$$f(\mathbf{x})=o \exp \left(-\frac{|\mathbf{x}-\boldsymbol{\mu}|^{2}}{2 r^{2}}\right)$$

, and the Silhouette for each pixel $\mathbf{p}=(u, v)$ is defined by

$$S(\mathbf{p})=\sum_{i=1}^{n} f_{i}(\mathbf{p}) \prod_{j=1}^{i-1}\left(1-f_{j}(\mathbf{p})\right)$$

, where $f_{i}(\mathbf{p})$ is computed as in equation (1) but with $\boldsymbol{\mu}$ and $r$ of the splatted 2D Gaussians in pixel-space:

$$\boldsymbol{\mu}^{2 \mathrm{D}}=K \frac{E_{t} \boldsymbol{\mu}}{d}$$

$$r^{2 \mathrm{D}}=\frac{f r}{d}, \quad \text { where } \quad d=\left(E_{t} \boldsymbol{\mu}\right)_{z}$$

Above all are the descriptions in the SplaTAM paper. However, I don`t know the exact $f(p)$ formula. So at the beginning of my personal understanding, I think it may look like this:

$$f(p)=o \exp \left(-\frac{\left|p-\mu^{2 D}\right|^{2}}{2\left(r^{2 D}\right)^{2}}\right)$$

$$f(u, v)=o \exp \left(-\frac{\left(u-\mu_{u}^{2 D}\right)^{2}+\left(v-\mu_{v}^{2 D}\right)^{2}}{2\left(\frac{f r}{d}\right)^{2}}\right)$$

However, I found it a little weird to directly exchange the $\mathbf{x} \in \mathbb{R}^{3} $ to $\mathbf{p}=(u, v)$ for equation (1).
Since the $S(p)$ is in the perspective of a rendered pixel on 2D image plane, it may relate to many 3D Gaussians.
On the contrary, $\mathbf{x} \in \mathbb{R}^{3}$ refers to one Gaussian model in the Space. So I wonder there might be a more accurate description like:

$$ g_{i}(\mathbf{p}) = w(\mathbf{x}_i, p) f(\mathbf{x}_i) $$

, the function $w(\mathbf{x}_i, p)$ denotes the contribution coefficient of the Gaussian model at the position $\mathbf{x}_i$ towards the rendering of the pixel $p$, so the Silhouette should be rewritten as

$$S(\mathbf{p})=\sum_{i=1}^{n} g_{i}(\mathbf{p}) \prod_{j=1}^{i-1}\left(1-g_{j}(\mathbf{p})\right)$$

The views expressed above are my preliminary thoughts on the matter, and I would be grateful for any critiques or corrections from yours. Thanks very much!!

from splatam.

Hi @lvmingzhe, Thanks for sharing your views!

Yes, you are correct. The detailed equation would be similar to what you outlined. One small addition is that the contribution coefficient of the Gaussian is the opacity (alpha) of the Gaussian. Hence, the Silhouette is the output of the alpha-compositing process in the rasterization of 3D Gaussians.

from splatam.