Future LRScheduler implementation with Tensor support about pytorch HOT 2 OPEN

This is a bit related to #68332 and #110636. In the sense that there is a more general problem that a pytorch function currently doesn't work nicely with Python scalars. In #110636 there was a proposal to support them as arguments to more pytorch functions (like torch.sqrt(x) currently does not accept Python scalars, but x ** 0.5 does), but make them always return tensors (I think this actually is what NumPy does, so this complicates partly the lanscape). Originally I proposed that many simple PyTorch functions should also support Python scalars but preserve tensorness and return a Python scalar if needed.

So in a way, I wonder does this preserve-tensorness-idea bring closer or further the purely functional scheduler formulas

from pytorch.

Personally, I think either way would be fine, but supporting both tensor and pure scalar in the functional scheduler would be more challenging. Seems like fully converting to torch.Tensor before passing to functional scheduler would be one of the solution that solve this issue, and keep LRScheduler from being too complex.

from pytorch.