[inductor][perf] Suboptimal codegen for horizontally fused softmax about pytorch HOT 4 OPEN

yea. I think maybe doing more things in a loop results in more register pressures and cause perf problem. Will need verify with some profiling though

It's very likely. With two separated loops, the intermediate result _tmp8 will be stored to SMEM via tl.sum to free some regsiters, which isn't too bad (much better than tl.store).

from pytorch.

this codegen might be something we should try to fix on the inductor side. Since splitting loops up resulted in speedups, maybe we should explore doing some kernels in more loops.

yea. I think maybe doing more things in a loop results in more register pressures and cause perf problem. Will need verify with some profiling though

from pytorch.

Maybe we should actually do 5 loops here:

1. max_a -- evict last
2. sum_a (in reverse order to maximize cache hits) -- evict first
3. max_b -- evict last
4. sum_b (in reverse order to maximize cache hits) -- evict first
5. output

from pytorch.

Maybe we should actually do 5 loops here:

1. max_a
2. sum_a (in reverse order to maximize cache hits)
3. max_b
4. sum_b (in reverse order to maximize cache hits)
5. output

This sounds reasonable to me. In general I would not fuse loops that do not have data dependencies into one. When they have data dependencies, fusion can be helpful in that the intermediate results from the first loop can be pipelined directly to the second loop in the same loop iteration. When there are no dependencies, loop fission (loop distribution) would be the way to go usually.

from pytorch.