reliability diagrams about swa_gaussian HOT 7 OPEN

Hmm.. I'll have to look into it. Maybe @izmailovpavel can be of help?

from swa_gaussian.

Hi Kirk,

You're reading the plot incorrectly - beneath the blue line shows that both SGD and SWAG are overconfident in that situation (confidence > accuracy). With that being said, I'm not sure if we ever checked calibration of the CIFAR5+5 task - will get back to you on that.

from swa_gaussian.

Thank you, Wesley!
Appreciate the prompt reply and clarification.

from swa_gaussian.

Just following up, I checked and we never seem to have run calibration on CIFAR 5+5, but it's not terribly surprising that both SGD and SWAG (somewhat less so) are overconfident here as well.

from swa_gaussian.

Hi Wesley,
thanks a lot for the follow up.
May I ask an additional question if you could clarify that for me please?
Why is the split on cifar10 (5+5) deterministic, (i.e. predefined as 0<---first half of the classes and 1<---the remaining, where 0 = [0, 1, 2, 8, 9] and 1 = [3, 2, 4, 8, 1] <--- labels)
Have you noticed that if you train on 1 yields better results than on 0 for out of distribution?

from swa_gaussian.

I believe we sampled those randomly at one point, so it's a holdover from that.

No, I haven't noticed that.

from swa_gaussian.

Thank you, Wesley!

Here's an example of the difference between sgd vs swag if you train on 1 vs 0. Basically swag seems to perform worse than sgd when trained on 0. Left plots are trained on 1 and right ones on 0.

from swa_gaussian.