Loading SWAG Checkpoint and Continue SWAG Training about swa_gaussian HOT 7 CLOSED

Hey @FloLins so I think in order to actually resume training, you need to also load the non-swag checkpoint with the resume flag: https://github.com/wjmaddox/swa_gaussian/blob/master/experiments/train/run_swag.py#L243-L248 this part loads the optimizer and the base model parameters. Do you have a regular checkpoint or just the SWAG one?

from swa_gaussian.

Can you just remove that flag in the code?

from swa_gaussian.

Thanks for your reply.
I changed the code to:
swag_model = SWAG(
model_cfg.base,
no_cov_mat=args.no_cov_mat,
max_num_models=args.max_num_models,
#loading=True, #Now the Flag is removed
*model_cfg.args,
num_classes=num_classes,
**model_cfg.kwargs
)

This does omit the error, but the training just started from epoch one and the SWAG checkpoint is not loaded.
Loading a non SWAG training Checkpoint is still possible.

from swa_gaussian.

Hey @izmailovpavel I already tried that.
In your paper you use 300 epochs of training. (160 till local minima and 140 SWAG training)
I loaded the normal checkpoint from 160 and from 200 and observed the accuracy and final results.
The run loaded from 160 has a higher accuracy as the run loaded from 200.
It seems that the algorithm just starts SWAG training from beginning instead of loading SWAG parameters and continuing SWAG training.

Does this load the SWAG Parameters too?
https://github.com/wjmaddox/swa_gaussian/blob/master/experiments/train/run_swag.py#L243-L248

from swa_gaussian.

Hey @FloLins you have to use both resume and swa_resume to load both the base model / optimizer and the swag model. Sorry for the confusion.

from swa_gaussian.

Hey @izmailovpavel I tried that too before opening this issue, and it didn't work due to the "loading=True" flag
If I comment this flag out (line 261 in run_swag.py) like @wjmaddox suggested and use both parameters "resume" and "swa_resume" it works fine!

Will any other functionality broke down, if I keep this flag commented out?

Thanks for all your help and have a nice day,

Yours sincerely,
Florian Linsner

from swa_gaussian.

Hi @FloLins, glad that it worked!

I think you could even completely remove these lines: https://github.com/wjmaddox/swa_gaussian/blob/master/experiments/train/run_swag.py#L252-L260.

I don't remember why we had the loading parameter, but I don't think it's used anywhere.

from swa_gaussian.