jostmey / nakedattention Goto Github PK
View Code? Open in Web Editor NEWRevealing example of self-attention, the building block of transformer AI models
License: GNU General Public License v3.0
Revealing example of self-attention, the building block of transformer AI models
License: GNU General Public License v3.0
I tried running your code in a Kaggle notebook, but quickly ran out of RAM. I think the problem is that you are effectively building a computational graph that gets bigger every time you run e_train += e_batch
, even though you only need the numerical value of e_train
and you don't plan to backprop it. I think you can fix this by running .detach()
on each pytorch object before you treat it as a plain numerical value. That way, the computational graph only contains a single batch, not the whole epoch. In my tests, this brings the memory usage back into the Kaggle bounds.
Here is the change I made:
- e_train += e_batch/len(loader_train) # Accumulate average loss for this epoch
- a_train += a_batch/len(loader_train) # Accumulate average accuracy for this epoch
+ e_train += e_batch.detach()/len(loader_train) # Accumulate average loss for this epoch
+ a_train += a_batch.detach()/len(loader_train) # Accumulate average accuracy for this epoch
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.