Questions about the implementation of calculation of Low-Rank Covariance Matrix about swa_gaussian HOT 2 CLOSED

Sorry for such meaningless question as a newbee.
I have understood the covariance estimation in SWAG. And I will note it here for other new boys,
(1) The inference do not need the calculation of covariance matrix, instead, during training, sqrt of covariance matrix is saved, which can be used for inference.
$$\widetilde{\theta}=\theta_{\mathrm{SWA}}+\frac{1}{\sqrt{2}} \cdot \Sigma_{\mathrm{diag}}^{\frac{1}{2}} z_1+\frac{1}{\sqrt{2(K-1)}} \widehat{D} z_2, \quad \text { where } z_1 \sim \mathcal{N}\left(0, I_d\right), z_2 \sim \mathcal{N}\left(0, I_K\right)$$

(2) In Section 4, the eigvector of covariance matrix is calculated by SVD. The mathmetical foundation is that: If $C = A^{T}A$, the SVD of $A$ is denoted by $A = U \Sigma V^T$, the $U$ here is composed of the eigvectors of $C$.

So there is no need for the calculation of covariance matrix. It's a fantastic trick to aviod the large computation budget.

from swa_gaussian.

@Codefmeister Can you eloprate on how you solved the issue. I am having the same issue. From my understanding, you still have to calculate the covarience as you are trying to store the sqrt of the convarience matrix and you will hit the same wall?

from swa_gaussian.