knyazzzzz / pytorch-lifestream Goto Github PK

pytorch-lifestream a library built upon PyTorch for building embeddings on discrete event sequences using self-supervision. It can process terabyte-size volumes of raw events like game history events, clickstream data, purchase history or card transactions.

It supports various methods of self-supervised training, adapted for event sequences:

• Contrastive Learning for Event Sequences (CoLES)
• Contrastive Predictive Coding (CPC)
• Replaced Token Detection (RTD) from ELECTRA
• Next Sequence Prediction (NSP) from BERT
• Sequences Order Prediction (SOP) from ALBERT
• Masked Language Model (MLM) from ROBERTA

It supports several types of encoders, including Transformer and RNN. It also supports many types of self-supervised losses.

The following variants of the contrastive losses are supported:

• Contrastive loss (paper)
• Triplet loss (paper)
• Binomial deviance loss (paper)
• Histogramm loss (paper)
• Margin loss (paper)
• VICReg loss (paper)

pip install pytorch-lifestream

# Ubuntu 20.04

sudo apt install python3.8 python3-venv
pip3 install pipenv

pipenv sync  --dev # install packages exactly as specified in Pipfile.lock
pipenv shell
pytest

• Supervised model training notebook
• Self-supervided training and embeddings for downstream task notebook
• Self-supervided embeddings in CatBoost notebook
• Self-supervided training and fine-tuning notebook
• Self-supervised TrxEncoder only training with Masked Language Model task and fine-tuning notebook
• Pandas data preprocessing options notebook
• PySpark and Parquet for data preprocessing notebook
• Fast inference on large dataset notebook

Documentation

Library description index

pytorch-lifestream usage experiments on several public event datasets are available in the separate repo