Running `aten._foreach_addcdiv_.ScalarList` with `FakeTensor`s uses high CPU memory and is slow about pytorch HOT 7 OPEN

We could add it to torch logs. I don't want to add to warning spam, especially for something that is not actionable to users.

But in either case we should get to full meta coverage of foreach kernels. As issue #105105 here shows there are a number of foreach without meta.

from pytorch.

cc @mlazos ?

from pytorch.

The fix should be to add a meta registration (#123463). I have confirmed that fixes the issue locally. I am not familiar with the test infra to adjust the expected failing tests, as it seems like some change is needed for the sample inputs.

Something for discussion though is if we can warn when the fallback behavior happens. It looks like the op was silently being run on CPU despite being under FakeTensorMode.

from pytorch.

From offline discussion: @janeyx99 will figure out how to land the PR with the appropriate testing changes.

from pytorch.

While the PR #123486 "fixes" this issue for this specific use case, it'd be good to confirm why this happened in the first place/if the behavior is expected.

from pytorch.

@janeyx99 yes it is expected that if you don't have a meta registration we fallback to the eager implementation. This was originally for bootstrapping before we had large meta kernel support. @zou3519 turned the default to not fallback for non-aten kernels but could not switch the default for aten kernels due to long-tail ops.

For foreach kernels that we are running with parameters IMO it would makes sense to prioritize meta coverage here since fallback can be memory intensive.

cc @mlazos

from pytorch.

@eellison What is your opinion on adding a warning when falling back to the eager implementation? Would it be too noisy?

For this particular case, debugging this CPU OOM was non-trivial (first seen on MAST and having to iteratively narrow down the culprit through the training loop to DTensor to the FakeTensorMode part of DTensor sharding propagation), and it would have saved a lot of time if we knew that the fallback was materializing CPU tensors.

from pytorch.