Comments (6)
This small change was to accommodate checkpoints of models trained on a different codebase, the order of these two ops does not make a huge difference as long as you perform both (see how there is an additional gating at the output, so you can view the sequence of ops as pre-gate followed by the regular conv-gate order described in the paper).
from safari.
Thanks for the clarification!
from safari.
Sorry but I have one more question. Doesn't current implementation reduces the number of time domain conv?
For example, in the paper order-2 hyena (which is corresponds to H3) has two frequency domain conv and two time domain conv. But for current implementation, the order-2 hyena has two frequency domain conv and a single conv time domain conv. Should I just think that order-2 hyena in current code is equivalent to order-1 hyena in the paper?
from safari.
that's correct there is one less long convolution in the current implementation. The number of total time-domain convolutions is still 2 if you count the short input convolution (which replaces the shift SSM in H3). The changes were made to carry out fair ablations (in terms of total FLOPs and model depth) between H3 and order-2 Hyenas.
from safari.
The number of total time-domain convolutions is still 2 if you count the short input convolution (which replaces the shift SSM in H3).
Which means that order-2 hyena with short conv + long conv is used for all evaluations in the paper?
The changes were made to carry out fair ablations (in terms of total FLOPs and model depth) between H3 and order-2 Hyenas.
When just comparing model quality without consideration of FLOPs, is order-2 hyena with two long conv better than a short-conv and a long-conv? or is this short-conv essential for model quality?
from safari.
Which means that order-2 hyena with short conv + long conv is used for all evaluations in the paper?
Yes, anywhere we state Hyena-2 in the manuscript. There are some experiments with other orders (e.g. wikitext).
When just comparing model quality without consideration of FLOPs, is order-2 hyena with two long conv better than a short-conv and a long-conv? or is this short-conv essential for model quality?
Higher orders can be more performant depending on the task, but you should always keep at least one short convolution.
from safari.
Related Issues (20)
- RuntimeError: u must have shape (batch_size, H, L) HOT 3
- fftconv compiling issue HOT 3
- configs for Hyena Wikitext103 experiments HOT 10
- Downstream evaluation on SuperGLUE HOT 2
- Inconsistent between implementation and paper descriptions HOT 1
- Encoder decoder
- Does Hyena support BERT style LLM? HOT 3
- Bidirectional Hyena HOT 1
- low train accuracy (10% / 40%) on synthetic language modeling tasks for H3 HOT 2
- What is the suggested config for running LRA exps with Hyena?
- RWKV
- learn_ifft in long_conv.py HOT 1
- dropout_add_layer_norm is not installed HOT 1
- Hyena seems forward leakage? HOT 9
- Accuracy on CIFAR is not similar to that in the paper HOT 4
- Non-causal implementation of language model for synthetic datasets HOT 5
- Hyena
- About the squash operator of long convolution HOT 2
- How to visualize Hyena Matrix H(u) from the Paper HOT 1
- How to reproduce the Hyena-Imagenet experiment result simillar with the paper? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from safari.