This code repository contains the code used for my "Optimizing Memory Usage for Training LLMs and Vision Transformers in PyTorch" blog post.

You can install the dependencies via

pip install -r requirements.txt

The scripts are all standalone scripts and can be run by executing

python 01_pytorch-vit.py

and so forth. The only requirement is to have the local_utilities.py in the same folder as the script as it contains some data loading utilities that are reused across all the scripts.

I tracked the script outputs in the logs.md file.