[inductor][perf] Inductor/Triton softmax kernel is slower than eager about pytorch HOT 4 OPEN

If we run with either TORCHINDUCTOR_COORDINATE_DESCENT_TUNING or force persistent reductions to occur it beats eager. Funny enough the coordinate descent tuning it discovers is essentially a persistent reduction because it makes the rblock equal to the length of the loop:

:XBLOCK: 1, RBLOCK: 8192, num_warps: 16, num_ctas: 1, num_stages: 1

If we force the rblock to be equal to the length of the loop is there any difference between persistent reduction and normal codegen 🤔 assuming that triton can dedup loads.

I can look into improving the heuristics for this case.

from pytorch.

If rnumel is even larger, we would not be able to use persistent reduction.

I think this online softmax PR (#127011 ) would help here.

from pytorch.

I think this is pretty similar to some of the experiments I did previously on softmax perf - block size tuning is the main gap here.

from pytorch.

Interesting, perhaps some more evidence of suboptimal heuristics. cc @AlnisM @Chillee

from pytorch.