awf / functional-transformer Goto Github PK
View Code? Open in Web Editor NEWA pure-functional implementation of a machine learning transformer model in Python/JAX
License: MIT License
A pure-functional implementation of a machine learning transformer model in Python/JAX
License: MIT License
From #4 (comment)_
See how we might include the sin(t)
terms, rather than just 'learned' encodings.
Mistake 1. The attention heads are all summed together; they should be concatenated (see code and image of attention equation).
Mistake 2. After the missing concatenation there should have been another linear layer (see code and image of attention equation).
Note. Using a for-loop for heads doesn't seem to be efficiently compiled by Jax.
Note. The weight initialization is different to minGPT.
# Multi-head self-attention
for head in layer.heads:
# Project into this head's query/key space
query = linear(head.query, t1) # L x Dk
key = linear(head.key, t1) # L x Dk
# Compute L x L attention matrix
score = query @ key.T + mask # L x L
attn = jax.nn.softmax(cfg.tau * score, axis=1) # L x L
value = linear(head.value, t1) # L x Dm
self_attn = attn @ value # L x Dm
# Add this head's contribution into embeddings
embeddings += self_attn # L x Dm <---- sum instead of concatenate
# <-- after concatenating all attention heads there should be another linear layer here.
As noted in #6 the model does not match the original code, or indeed the original transformer paper. I therefore consider this a "transformer variant", but of course it would be sensible to make it match and check if that improves/disimproves performance.
WandB loss curves (e.g. here) show a sawtooth form, correlated with batch ID.
Batches are randomized and this occurs even with 1-bit gradients, so it's not Adam...
Hi there. Great work!
I shared this with a colleague and they were concerned that your example does not seem to initialize the positional encodings beyond zeros.
Should there be a comment or an implementation of setting up the positional encoding?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.