This repository contains an implementation for 8 self-supervised instance-level (image-level) representation learning methods:

• Contrastive:
  • MoCo (v2, v3)
  • SimCLR
  • NNCLR
  • Decoupled Contrastive Learning (named DCCLR in this repository)
• Redundancy-reducing:
  • BarlowTwins
  • Hilbert-Schmidt Independence Criterion Maximization (named HSICTwins in this repository)
• Negative-free:
  • DINO
  • TWIST

Try it out in a Colab Notebook (one training takes 40-60 minutes):

The codebase follows modern Tensorflow2 + Keras best practices and the implementation seeks to be as concise and readable as possible. This implementation is intended to be used as an easy-to-use baseline instead of as a line-by-line reproduction of the papers.

The image augmentation pipeline is a crucial component of all these methods. You can find implementations of other custom Keras image augmentation layers in this repository.

• simple and easy-to-read implementation over accuracy to the finest implementation details
• simple feedforward convolutional architecture:
  • the methods have to be robust enough to work on simple architectures as well
  • this enables the usage of larger batch sizes
  • the shorter training time enables more thorough hyperparameter tuning so that the comparison is fairer
• no batchnorm layers used for benchmarking: as reported in CPCv2 and MoCo, it introduces an intra-batch dependency between samples, which can hurt performance
• only the most important image augmentations are used, to avoid having too much hyperparameters:
  • random resized crop: forces the model to encode different parts of the same image similarly
  • random color jitter: prevents a trivial color histogram-based solution to the task by distorting color histograms
  • random horizontal flip: introduces the prior that the horizontal directions are more interchangeable than the vertical ones
• dataset: STL10, a semi-supervised dataset with 100.000 unlabeled + 5000 labeled images, well suited for self-supervised learning experiments