This repository contains materials from the lectures.
-
Wiki course page: link
-
Course page at hse.ru: link
-
Lecture recordings on YouTube (in Russian): YouTube
-
[week1] Introduction to Course, classical pretext tasks for SSL. Recordings: lecture, seminar
-
[week2] What to do with a pretrained model? Probing, Linear classifier, Fine-tuning. Recordings: lecture, seminar
-
[week3] Contrastive learning for images. Mutual information, SimCLR, MoCo, BYOL, SimSiam, SwAV. Recordings: lecture, seminar
-
[week4] Masked Image Modeling. DINO, BEiT, MAE, MaskFeat. Different approaches to improving contrastive learning. Recordings: lecture, seminar
-
[week5] Overview of Transformer-based language models. GPT, BERT, XLNet, RoBERTa, ALBERT, MASS, BART, ELECTRA. Recordings: lecture, seminar
-
[week6] Model pre-training for the source code domain. code2vec, code2seq, CodeBERT, GraphCodeBERT, CodeT5, Codex. Recordings: lecture
-
[week7] Diffusion models for NLP. Theory introduction, Multinomial Diffusion, D3PM, Diffusion-LM, DiffuSeq. Recordings: lecture, seminar
-
[week8] Self-supervised learning for audio. Introduction to audio processing. CPC, Wav2Vec 2.0, HUBERT, Multi-format contrastive learning, BYOL-A. Recordings: lecture
-
[week9] Self-supervised learning for graphs. Intro to representation learning on graphs. Review of approaches. Recordings: lecture
- [hw1] SSL with pretext tasks. Task description, Kaggle competition
- [hw2] Contrastive learning for images. Task notebook
- [hw3] Pre-training of language models. Task notebook
- [hw4] Pre-training of audio models. Task notebook