Non-causal implementation of language model for synthetic datasets about safari HOT 5 OPEN

For the M2-BERT synthetics, we ran a non-causal form of associative recall to fine-tune the architecture.

For induction head - the same analogy applies for what we implemented here, but you're right that the setup is a bit different.

from safari.

from safari.

Thanks for your quick response. Regarding this dataset, if the goal is to optimize the test accuracy, reported in the paper which is evaluated on the last token, it seems that we don't need to model the training as a causal language model and we can simply design the training loss to match the test loss by evaluation only the last token. I understand that the goal was to design a benchmark to evaluate the autoregressive (causal) behaviour of the LM but it seems that in this dataset, this different training procedure is indeed working as a regularizer by forcing the model to predict next token while the final goal is the accuracy of the last token only.

from safari.

My intuition is that the next-token prediction actually makes it easier for the model to learn the circuit it needs to do associative recall, since there's more tokens it needs to predict correctly during training (roughly the second half of the tokens, once it's seen each example once). You can see this when the training accuracy jumps around the same time that the test accuracy jumps - it's learned a generalizable behavior, not just a shortcut.

In any case - the synthetic is mostly useful insofar as it predicts downstream performance on language modeling. There's a lot of techniques that can solve associative recall (e.g., Python function) - the interesting bit is when we can use the synthetic to guess how well a layer will perform downstream.

from safari.

Thanks for your clarification. For Monarch Mixer that you mentioned, did you use the causal form of M2 for associative recall dataset?

I also have another question regarding the induction head dataset: does the same analogy (causality and next token prediction) apply to it as well? I am asking this since, in contrast to associative recall, in this dataset it only need to recall content after a special token that has happened at the middle of the sequence not all the tokens.

from safari.