kumarkrishna / fastssl Goto Github PK
View Code? Open in Web Editor NEWFast SSL
Fast SSL
Merge the precaching and linear classification steps to be run from the same python file. Steps would potentially include:
Installing a working environment with ffcv (FFCV-SSL) has been painful, due to changing version of dependencies etc. So, keeping track of the instructions that worked.
echo "Creating new conda environment ======"
conda create --prefix $SCRATCH/conda_envs/ffcv_ssl_fastssl -y python=3.9 conda conda-libmamba-solver -c conda-forge
conda activate $SCRATCH/conda_envs/ffcv_ssl_fastssl
echo "Setting up conda env variables and config options =========="
export CONDA_EXE="$(hash -r; which conda)"
conda config --set solver libmamba
conda config --set channel_priority flexible
echo "Installing torch and other important packages+dependencies ============"
conda install -y cupy pkg-config compilers libjpeg-turbo opencv=4.7.0 pytorch=2.1.2 torchvision=0.16.2 pytorch-cuda=12.1 torchmetrics numba=0.56.2 terminaltables matplotlib scikit-learn pandas assertpy pytz -c pytorch -c nvidia -c conda-forge
cd <FFCV-SSL folder>
pip install -e .
pip install wandb tqdm
Notes: Not sure if we need numba=0.56.2
but it was required to install original ffcv library
Reinstall ffcv using FFCV-SSL codebase and incorporate functionality to generate multiple patches
Aim to do around Feb 16
Title. As I saw that you speed SSL by FFCV, e.g. https://github.com/kumarkrishna/fastssl/blob/8f96d16d2a/write_ffcv_double_datasets.py
Have you tried the same thing on MoCo v2? Since the issue mentioned in another SSL speeding library implies that adapting FFCV to MoCo v2 is non-trivial.
Currently the projector is thrown away after BarlowTwins pretraining -- check proper arguments are passed to the function when reloading network (under if conditions of algorithm 'linear') and while doing forward pass!
New todos:
FFCV is integrated and trains reasonably well to ~74% accuracy. The same config with PyTorch data loaders gives a ~84% model.
Dears,
I have been recently following your work "Assessing Representation Quality in Self-Supervised Learning by measuring eigenspectrum decay" and I was trying to apply to compute alpha in order to assess the representation quality of Self-Supervised Model.
I computed the covariance matrix of the model features then I applied torch.linalg.eigvals() to calculate the eigenspectrum using the following code
cov = torch.zeros(768, 768)
N = len(val_data_loader)
for i,x in enumerate(val_data_loader):
inputs,batchLabels = x
features = backbone.features(inputs.to('cuda'))
features = features.detach().cpu()
cov += torch.mm(features.T, features)/N
eigenspectrum = torch.linalg.eigvals(cov).detach().cpu().numpy()
Now to calculate alpha from functions fit_powerlaw() and stringer_get_powerlaw(), from your code , I can see that both functions take eigenspectrum and other argument that implies the range.
My question is: What does this range imply? and what is the suitable range I should use to calculate alpha properly to assess the model's features' quality?
Thanks!
Nader
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.