Replicating results of transfer learning and out-of-domain image detection about swa_gaussian HOT 3 CLOSED

Sure, first, you'll have to train models as described in the README here. Then, using a saved checkpoint, you can compute the uncertainties overall in this README. Running those scripts will save .npz files of all predictions.

For all of those models, we trained using the first half of the CIFAR classes, which is the --split_classes=0 flag throughout; that is, you'll have to add that flag into whatever training and evaluation scripts you use.

Hope this helps,

from swa_gaussian.

Hi,
Thanks for your reply and apologies for very late reply.
I tried to replicate transfer learning experiment. The models were trained as you mentioned here and then predictions were done on STL-10 dataset.

Example Command:
python3.6 ./experiments/uncertainty/uncertainty.py --data_path=<stl_dataPath> --dataset=STL10 --model=VGG16 --use_test --cov_mat --method=SWAG --scale=0.5 --file=<vgg16_swagModelPath> --save_path=<save_path>

Results: (Averaged over three runs with different random seed)

NLL Results:

Accuracy Results

I have used exact parameter values to train the models as mentioned so not sure why NLL numbers are different than once mentioned in paper. Can you please give some feedback on this ?

Thanks,
Amit

from swa_gaussian.

Hmm... so those are all numbers that are pretty similar to what we are reporting and in all but accuracy on WRN, slightly better than the numbers we reported (lower NLL is better, higher accuracy is better). I'm pretty confident that this is just random seeds (don't know what @izmailovpavel used 18 mos ago) and/or slight changes in pytorch since we last ran the experiments.

from swa_gaussian.