jywa / fednova Goto Github PK

Hello,

I am not sure about this, but it seems there is a slight mismatch between the implementation of FedNova, and it's description from the paper. It seems that in the current implementation, you compute the cumulative sum of gradient, without multiplying by weights a_{i}, even when etamu is not zero (see below)

# update accumalated local updates
if 'cum_grad' not in param_state:
      param_state['cum_grad'] = torch.clone(d_p).detach()
      param_state['cum_grad'].mul_(local_lr)
else:
      param_state['cum_grad'].add_(local_lr, d_p)

p.data.add_(-local_lr, d_p)

Probably it is needed to add a line in order to multiply d_p by a_i (this variable should be tracked as well), before updating param_state['cum_grad'].

Is what I said correct, or is there anything I am missing?

Thanks in advance for your help

I'm sorry to bother you, but can you provide me with the source code for this paper?

Hi,
Congratulation for your great work. I've a question.
In the case that the local epoch is fixed for each user, vector ai and it's norm ||ai|| is unchanged across rounds?

For example, consider SGDMomentum-Nova, vector ai is as following:
[1+m+m2+m3+...m^(ti-1), ..., 1+m, 1]
where ti is the number of round's steps. When ti is fixed, ai is always equal to [1+m+m2+m3+...m^(ti-1), ..., 1+m, 1]

So, the value ||ai|| is known in advance as a certain function f(number of steps). It can be calculated once and reused for every round?

Thank you in advance for your feedback.