`StackCompressor` goes into infinite loop after `SSATransform` on inline assembly with very long chains of dependent assignments about solidity HOT 2 OPEN

Hi @hedgar2017, I can confirm that this is an issue, so thank for the report.

from solidity.

I also managed to reproduce it earlier today and it looks like it's specifically the SSATransform step in the optimizer sequence that triggers this.

Repro on the CLI (after extracting PoseidonT3.sol)

solc PoseidonT3.sol --via-ir --optimize --yul-optimizations a

I let it run for > 20 min and it ended up eating all memory on a machine with 32 GB RAM and being killed by the OOM killer.

It only happens with Yul optimizer enabled and is reproducible down to v0.8.10 (with --experimental-via-ir), which seems to be the version that introduced the optimized EVM transform and also included various changes to data flow analyzer. Before that it compiles fine.

I also took a quick look at the call stack in a debugger and the relevant bits look like this:

• OptimiserSuite::run()
• StackCompressor::run()
• eliminateVariablesOptimizedCodegen()
• Rematerialiser::run()
• DataFlowAnalyzer::operator(Block)
• Then lots of alternating between calls to Rematerialiser, ASTModifier and DataFlowAnalyzer
• Then deep chains of calls into the CodeCost metric

The stack is generally pretty deep and getting a little deeper each time I pause the execution. Got up to something like ~575 calls deep after ~10 min (in a debug build, so not necessarily comparable with the ~20 I mentioned above). The call chains measured by the metric seem weirdly long. Much longer than what I'd expect from nesting visible in unoptimized IR and there are no recursive calls there.

What it looks like to me is that probably StackCompressor keeps repeatedly using Rematerialiser to inline all those intermediate variables and ending up with very deep nesting, which is then very expensive to measure and analyze. Still not sure if it is actually doing infinite recursion or if it would finish eventually but is just pathologically costly to run.

from solidity.