ALR Transformer that replaces the original transformer implementation of an joint encoder + decoder block with a feedforward/alr block with a decoder block
pip install alr-transformer
import torch
from alr_transformer import ALRTransformer
x = torch.randint(0, 100000, (1, 2048))
model = ALRTransformer(
dim = 512,
depth = 6,
num_tokens = 100000,
dim_head = 64,
heads = 8,
ff_mult = 4
)
out = model(x)
print(out)
print(out.shape)- First git clone the repo then download and then run the following
python3 train.py
@misc{bozic2023rethinking,
title={Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as an Alternative to Attention Layers in Transformers},
author={Vukasin Bozic and Danilo Dordervic and Daniele Coppola and Joseph Thommes},
year={2023},
eprint={2311.10642},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent