Training After Switch to SGD is Flat about swats HOT 4 OPEN

Sorry for the late response, I haven't received notification of this issue. SGD generally requires some kind of lr scheduling, which is not included in this implementation. I tried adding it, however I couldnt find a nice solution to handle the SGD and Adam phase differently, which is desirable for any kind of scheduling.

from swats.

Oh I see. So in my case, what do you think should be done?

I'm facing similar issue (flat accuracy after switching)

from swats.

You could try writing some kind of custom lr scheduling, since the optimizer stores the current phase in the param group with the key phase and apply different behavior based on that.

def custom_schedule(optimizer, compute_adam_lr, compute_sgd_lr):
    for param_group in optimizer.param_groups:
        if param_group['phase'] == 'SGD':
            param_group['lr'] = compute_sgd_lr()
        elif param_group['phase'] == 'ADAM':
            param_group['lr'] = compute_adam_lr()

from swats.

I see. Thanks a lot, will try that and hopefully the problem does not occur again. Thank you @Mrpatekful

from swats.