Preserving gradients with copy()? about tch-rs HOT 2 CLOSED

copy uses PyTorch copy_ operator behind the hood and from a quick experiment on the python side this seems to properly propagate the gradient through. Did you try it out in rust and that wasn't the case?

>>> t = torch.tensor([0.1], requires_grad=True)
>>> u = torch.tensor([0.])
>>> u.copy_(t + 1)
tensor([1.1000], grad_fn=<CopyBackwards>)
>>> u.backward()
>>> t.grad
tensor([1.])

Re iterating on the trainable variables, this sounds perfectly fine, weight decay is already handled by most optimizers if that's what you're after but for another regularization loss that would probably be the way to go.

from tch-rs.

Thanks. I didn't notice any gradient issues with copy() but I had heard that some inplace operations don't propagate gradients in PyTorch so I was concerned that copy() might be problematic in some scenarios.

from tch-rs.