Comments (5)
For the M2-BERT synthetics, we ran a non-causal form of associative recall to fine-tune the architecture.
For induction head - the same analogy applies for what we implemented here, but you're right that the setup is a bit different.
from safari.
from safari.
Thanks for your quick response. Regarding this dataset, if the goal is to optimize the test accuracy, reported in the paper which is evaluated on the last token, it seems that we don't need to model the training as a causal language model and we can simply design the training loss to match the test loss by evaluation only the last token. I understand that the goal was to design a benchmark to evaluate the autoregressive (causal) behaviour of the LM but it seems that in this dataset, this different training procedure is indeed working as a regularizer by forcing the model to predict next token while the final goal is the accuracy of the last token only.
from safari.
My intuition is that the next-token prediction actually makes it easier for the model to learn the circuit it needs to do associative recall, since there's more tokens it needs to predict correctly during training (roughly the second half of the tokens, once it's seen each example once). You can see this when the training accuracy jumps around the same time that the test accuracy jumps - it's learned a generalizable behavior, not just a shortcut.
In any case - the synthetic is mostly useful insofar as it predicts downstream performance on language modeling. There's a lot of techniques that can solve associative recall (e.g., Python function) - the interesting bit is when we can use the synthetic to guess how well a layer will perform downstream.
from safari.
Thanks for your clarification. For Monarch Mixer that you mentioned, did you use the causal form of M2 for associative recall dataset?
I also have another question regarding the induction head dataset: does the same analogy (causality and next token prediction) apply to it as well? I am asking this since, in contrast to associative recall, in this dataset it only need to recall content after a special token that has happened at the middle of the sequence not all the tokens.
from safari.
Related Issues (20)
- RuntimeError: u must have shape (batch_size, H, L) HOT 3
- fftconv compiling issue HOT 3
- configs for Hyena Wikitext103 experiments HOT 10
- Downstream evaluation on SuperGLUE HOT 2
- Inconsistent between implementation and paper descriptions HOT 1
- Encoder decoder
- Does Hyena support BERT style LLM? HOT 3
- Bidirectional Hyena HOT 1
- low train accuracy (10% / 40%) on synthetic language modeling tasks for H3 HOT 2
- What is the suggested config for running LRA exps with Hyena?
- RWKV
- learn_ifft in long_conv.py HOT 1
- dropout_add_layer_norm is not installed HOT 1
- Hyena seems forward leakage? HOT 9
- Accuracy on CIFAR is not similar to that in the paper HOT 4
- Hyena
- About the squash operator of long convolution HOT 2
- How to visualize Hyena Matrix H(u) from the Paper HOT 1
- How to reproduce the Hyena-Imagenet experiment result simillar with the paper? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from safari.