torch.compile() throws RuntimeError on FP8 Triton matmuls about pytorch HOT 6 OPEN

Seems like there are a number of functions that are not yet implemented for float8 types on either cpu or cuda:

1. .item() for cuda
2. unique for cpu and cuda
3. torch.index_select for cpu and cuda
4. torch.equal for cpu
5. fill_ for cpu
6. torch.where for cpu and cuda
   That is all I have found so far.
   Would following the same approach as #116806 be sufficient to add float8 support for the above?

from pytorch.

Seems like item() on cuda just doesn't support the float8 dtypes yet (which the graph passes in inductor happen to rely on):

# this works for me on cpu, but not on cuda
>>> x = torch.tensor(2, dtype=torch.float8_e4m3fn, device='cuda')
>>> y = x.item()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: "_local_scalar_dense_cuda" not implemented for 'Float8_e4m3fn'

cc @drisspg @vkuzo, should we just add .item() support for float8 cuda?

from pytorch.

yes we should add it, I can take a look but not sure on the timeline, if anyone can get to it sooner happy to help / review the code

from pytorch.

Can I give this a shot?

from pytorch.

will let @ajbrent and @vkuzo figure it out

from pytorch.

Would following the same approach as #116806 be sufficient to add float8 support for the above?

Thanks for offering to help! Yes, that approach is reasonable. I'd also say there is no expectation for every op to work for float8. The only numerics changing op with hardware support is a gemm (torch._scaled_mm), and there are other ops which are useful such as torch.cat, item(), etc which do not change numerics. We don't want to add support for things like torch.add with float8 inputs. The ops you listed in the comment above all sound good to me, although I'm not sure I'd prioritize things like unique unless there is a use case.

If you are up for also adding fill_out that would help close #128257!

from pytorch.