Comments (4)
Hi, thanks for the issue.
It looks like the config and code in this version is quite different from the one used to run CIFAR for the paper, @exnx might be able to help a bit more with recreating the exact config.
One thing I can tell is different is the order of the filter - try setting model.layer.filter_order=128
in the config.
Probably more important, I think the scheduler in the hyena-vit-cifar
config may not be set correctly, probably a copy-paste problem. Try changing the values at the top:
# @package _global_
defaults:
- /pipeline: cifar-2d
- /model: vit
- override /model/layer: hyena
- override /scheduler: cosine_warmup_timm
to:
# @package _global_
defaults:
- /pipeline: cifar-2d
- /model: vit
- override /model/layer: hyena
- override /scheduler: cosine_warmup
scheduler:
num_training_steps: 100000
(following this config for the scheduler: https://github.com/HazyResearch/safari/blob/main/configs/experiment/cifar/s4-simple-cifar.yaml)
EDIT: nevermind, I think this config is not supposed to be used... Eric will comment with a correction soon.
from safari.
Hi, thanks for the issue.
It looks like the config and code in this version is quite different from the one used to run CIFAR for the paper, @exnx might be able to help a bit more with recreating the exact config.
One thing I can tell is different is the order of the filter - try setting
model.layer.filter_order=128
in the config.Probably more important, I think the scheduler in the
hyena-vit-cifar
config may not be set correctly, probably a copy-paste problem. Try changing the values at the top:# @package _global_ defaults: - /pipeline: cifar-2d - /model: vit - override /model/layer: hyena - override /scheduler: cosine_warmup_timm
to:
# @package _global_ defaults: - /pipeline: cifar-2d - /model: vit - override /model/layer: hyena - override /scheduler: cosine_warmup scheduler: num_training_steps: 100000
(following this config for the scheduler: https://github.com/HazyResearch/safari/blob/main/configs/experiment/cifar/s4-simple-cifar.yaml)
EDIT: nevermind, I think this config is not supposed to be used... Eric will comment with a correction soon.
Thanks for your response. With the current setup, I got accuracy of 87% using the following setting:
scheduler:
t_in_epochs: False
t_initial: ${eval:${div_up:50000, ${train.global_batch_size}} * ${trainer.max_epochs}}
warmup_lr_init: 1e-6
warmup_t: 500
lr_min: ${eval:0.1 * ${optimizer.lr}}
model:
_name_: vit_b_16
d_model: 128
from safari.
I also tried it with python -m train experiment=cifar/hyena-vit-cifar model.d_model=128
to match the description in Appendix of Hyena paper. As a result, the performance improved to
from safari.
Hello!
So the current vit-cifar config was only meant for testing the pipeline, it's not meant for comparing cifar results. (eg, testing ViT pipeline on cifar is faster than running it on ImageNet).
In the Hyena paper, the cifar results are for a * 2D * version of Hyena (notably not a ViT model).
We still need to port the 2D version over (for cifar), which we'll do at some point soon!
from safari.
Related Issues (20)
- RuntimeError: u must have shape (batch_size, H, L) HOT 3
- fftconv compiling issue HOT 3
- configs for Hyena Wikitext103 experiments HOT 10
- Downstream evaluation on SuperGLUE HOT 2
- Inconsistent between implementation and paper descriptions HOT 1
- Encoder decoder
- Does Hyena support BERT style LLM? HOT 3
- Bidirectional Hyena HOT 1
- low train accuracy (10% / 40%) on synthetic language modeling tasks for H3 HOT 2
- What is the suggested config for running LRA exps with Hyena?
- RWKV
- learn_ifft in long_conv.py HOT 1
- dropout_add_layer_norm is not installed HOT 1
- Hyena seems forward leakage? HOT 9
- Non-causal implementation of language model for synthetic datasets HOT 5
- Hyena
- About the squash operator of long convolution HOT 2
- How to visualize Hyena Matrix H(u) from the Paper HOT 1
- How to reproduce the Hyena-Imagenet experiment result simillar with the paper? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from safari.