Variance of gradient noise and optimal learning rate about swa_gaussian HOT 5 CLOSED

Hi,

Thanks for your quick and detailed reply! I should mention that actually my supervisor Fredrik Lindsten spotted that error first and brought it up in our discussion. 🙂

David

from swa_gaussian.

Hi,

Thanks for carefully reading our paper! We just went through and determined out that your understanding of the formula is correct. We'll post a correction to both the code and arxiv soon (actually the code doesn't include the momentum correction and we had to correct that manually ourselves).

One point is that the minimum optimal learning rate we report should be 30,000 from the Figure 4 in the Appendix (a small typo due to the momentum scaling), and so the corrected version of the minimum optimal learning rate is actually 30,000 / 128^2 ≈ 1.8. It's also probably important to note that VGG is a bit higher based on the plot - close to 3 here.

Here is the corrected version of Appendix Figure 4:

Again, thanks for catching the error and please let us know if there are any more errors.

Wesley

from swa_gaussian.

Thanks again for noticing that issue - I put your name in the acknowledgements in the final version on arxiv.

Cheers,
Wesley

from swa_gaussian.

I'm closing this now that I've updated the code: b97b40c

Thanks again for bringing it to our attention.

from swa_gaussian.

Thanks!

from swa_gaussian.