Thanks so much for releasing the code. I have several questions about the implementati

About LPF-SGD implementation about lpf-sgd HOT 5 CLOSED

lucasliunju commented on June 20, 2024

About LPF-SGD implementation

from lpf-sgd.

Comments (5)

lucasliunju commented on June 20, 2024 1

Dear Devansh,
Thank you very much for your reply. I have reproduced the results based on cosine decay, which is also the same as the results in the paper. Thanks again!

Best,
Lucas

from lpf-sgd.

devansh20la commented on June 20, 2024

I think you might be looking at the wrong file. This was from another optimizer.

EDIT: I had a naming error. The code you are looking for starts here:

LPF-SGD/codes/resnets_nodataaug/lpf_train.py

Lines 57 to 80 in 9a35c5f

 with torch.no_grad(): 

 noise = [] 

 for mp in model.parameters(): 

 if len(mp.shape) > 1: 

 sh = mp.shape 

 sh_mul = np.prod(sh[1:]) 

 temp = mp.view(sh[0], -1).norm(dim=1, keepdim=True).repeat(1, sh_mul).view(mp.shape) 

 temp = torch.normal(0, args.std*temp).to(mp.data.device) 

 else: 

 temp = torch.empty_like(mp, device=mp.data.device) 

 temp.normal_(0, args.std*(mp.view(-1).norm().item() + 1e-16)) 

 noise.append(temp) 

 mp.data.add_(noise[-1]) 

 # single sample convolution approximation 

 with torch.set_grad_enabled(True): 

 outputs = model(inputs) 

 batch_loss = criterion(outputs, targets) / args.M 

 batch_loss.backward() 

 # going back to without theta 

 with torch.no_grad(): 

 for mp, n in zip(model.parameters(), noise): 

 mp.data.sub_(n)

from lpf-sgd.

lucasliunju commented on June 20, 2024

Thank you very much for your reply!

from lpf-sgd.

lucasliunju commented on June 20, 2024

Dear Devansh,

This is a great work for me and LPF-SGD can significantly improve the performance compared with vanilla momentum sgd. I would like to ask whether cosine learning rate decay can also work for LPF-SGD on WRN and ResNet. Since I find the most learning rate decay is StepLR

Thank you very much!

Best,
Lucas

from lpf-sgd.

devansh20la commented on June 20, 2024

I did not try the cosine learning rate decay. I can't think of any reason it shouldn't work, it might just require finetuning to find the best hyperparameters.

from lpf-sgd.

About LPF-SGD implementation about lpf-sgd HOT 5 CLOSED

Comments (5)

Related Issues (3)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent

	with torch.no_grad():
	noise = []
	for mp in model.parameters():
	if len(mp.shape) > 1:
	sh = mp.shape
	sh_mul = np.prod(sh[1:])
	temp = mp.view(sh[0], -1).norm(dim=1, keepdim=True).repeat(1, sh_mul).view(mp.shape)
	temp = torch.normal(0, args.std*temp).to(mp.data.device)
	else:
	temp = torch.empty_like(mp, device=mp.data.device)
	temp.normal_(0, args.std*(mp.view(-1).norm().item() + 1e-16))
	noise.append(temp)
	mp.data.add_(noise[-1])

	# single sample convolution approximation
	with torch.set_grad_enabled(True):
	outputs = model(inputs)
	batch_loss = criterion(outputs, targets) / args.M
	batch_loss.backward()

	# going back to without theta
	with torch.no_grad():
	for mp, n in zip(model.parameters(), noise):
	mp.data.sub_(n)