Algorithm 1 in paper about hypergrad HOT 5 CLOSED

Yes, this is a typo - the hypergrad code has the correct implementation, as you noticed. Thanks for pointing it out! Algorithm 2 also has a couple of typos in the paper but not in the code.

from hypergrad.

Great -- thanks! Will use the code as a reference, in that case.

from hypergrad.

A related question -- the version of SGD w/ momentum implemented in hypergrad is different from the standard implementation. hypergrad has:

velocity = momentum * velocity - (1.0 - momentum) * g
x = x + learning_rate * velocity

vs other place (eg the NAG paper, CS231 from Stanford) have

velocity = momentum * velocity - learning_rate * g
x = x + velocity

@duvenaud Do you have any insight on why the hypergrad implementation differs?

~ Ben

from hypergrad.

Correct me if I'm wrong but I think your latter question was answered in autograd#289. Maybe this issue can be closed?

from hypergrad.

It was -- thanks. Apologies for the double post.

from hypergrad.