aryamanarora / causalgym Goto Github PK

• store the data table as parquet?

• don't reload model weights every time in test_all
• ignore obviously non-causally-relevant spans
• cache source activations for interventions

This is a bit messy now. Here is what needs to be done to make this a usable and reproducible benchmark.

Data generation

• We shouldn't need to regenerate the data to get new intervention positions.
• Each thing should be an object with associated helper functions, not the current mess of free-floating functions.
• No mixing of train and eval.
• It should be simple to filter for fixed token length inputs of various kinds.
• Allow interventions between differing length inputs. Need to figure out the alignment stuff.