Improve debugability of warnings/errors "Triggered internally at" about pytorch HOT 7 OPEN

Anomaly mode is supposed to always work for errors that are triggered from Variable._execution_engine.run_backward. But actually, it seems like the problem here is warnings, rather than errors, and it does seem likely to me that we are not annotating the traceback of the forward that caused it with warnings. This is compounded by the fact that we are typically not setting stacklevel correctly when we warn, so the single line warning printout doesn't even say what the relevant user code is.

It feels like it should be possible to install a temporary warning handler when we run backwards which augments warnings as well with user stacks. But we... probably don't want to print the full stacks? So we need some way of abbreviating it to one filename:lineno by default?? Not trivial. If anyone wants to try their hand at it I'd be happy tor eview.

@albanD wdyt

from pytorch.

It might also be a good time, in the age of PT2 supremacy, to consider turning anomaly mode error tracking on by default.

from pytorch.

Anomaly mode is supposed to always work for errors that are triggered from Variable._execution_engine.run_backward. But actually, it seems like the problem here is warnings, rather than errors

To be clear, I am not talking about anomaly mode, but warnings (that are printed from C++) in general. I actually meant that anomaly mode is an example of relatively GOOD warning messages. What I'd like to see is for all other C++ warnings in pytorch to have the same debugability.

But we... probably don't want to print the full stacks? So we need some way of abbreviating it to one filename:lineno by default?? Not trivial.

Actually, I think that full tracebacks are required for debugging. Rather than trying to guess the correct stack level, I'd prefer to have a special "verbose warnings" mode where the full tracebacks are printed.

I suspect that the hard part would be to record/identify the correct tracebacks during the forward pass. Without this, it doesn't matter if you are able to guess the stack level correctly, all of the warnings will just point to the my_loss.backwards() line in the user's code (which isn't that much of an improvement compared to return Variable._execution_engine.run_backward).

from pytorch.

Oh, I am reminded of #72948 which we eventually decided not to do because passing C++ log messages to Python was just... not a great idea. @kurtamohler did we ever take a closer look at the warning only piece of the puzzle?

from pytorch.

I don't think I ever looked into improving the traceability of warnings

from pytorch.

Actionable to augment warnings during backward with user stacks.
Prior to executing each node during backward, we already enter a warning recording context - when anomaly mode is enabled we should be able to include more information there. See: ehttps://github.com//pull/66235.

from pytorch.

Reserving this as an internal onboarding task

from pytorch.