nvidia-merlin / transformers4rec Goto Github PK

Even wants Tf4Rec Tensorflow version to be created before RecSys conference.

Related issues on NVTabular side: NVIDIA-Merlin/NVTabular#741

Keras propagates static shape information. This allows things like a Dense layer where you can just specify the hidden-dimension, and then we it builds the layer with the shapes in can init the weights. We would like to have something similar in the torch-side of the library so that we could enable things like a MLPBlock.

Create an end-2-end inference example on Triton with NVT preproc Transformers4Rec model

Extending the embedding tables of categorical features for new values seen on incremental training.
P.s. requires incremental preprocessing ( NVIDIA-Merlin/NVTabular#798 )

Benchmark the new PyT data loader with the REES46 ecommerce dataset, using multiple GPUs

Train set: All train.parquet files for 31 days (1 parquet file by week). P.s. Set row group size accordingly
Eval set: All valid.parquet files concatenated

• Create a recsys_main.py variation for non-incremental training
• Train with 3 weeks and evaluate on the last week
• Run experiments varying the number of GPUs: Single GPU, Multi-GPU Data Parallel, Multi-GPU Distributed DataParallel

reformer, xlnet, transfoxl, electra and albert have specific arguments that need to be set with respect to training data and LM objective. These arguments are specified in extra_config variable found here: https://github.com/NVIDIA-Merlin/Transformers4Rec/blob/main/transformers4rec/meta_model_api/tower_model.py

Currently blocked by #53.

Currently blocked by #49.

Test a simple pipeline with Preproc grouping sessions and input to a PyTorch model to probe the support to list columns in the output

Currently blocked by #52.

Should include some basic tests as well

We currently have a custom DataLoader (that inherits from the NVT PyT Data loader) that converts the offsets representation of list columns to sparse tensors.
We need to test and adapt our code to check if Julio's PR fully replaces our custom Data loader.

This includes:

• SequentialBlock
• BlockWithHead
• MLPBlock

Features-API here means the API that's part of torch.features.

Tasks:

• #52
• #53
• Do the refactoring

Currently we use PyTorch DataParallel, which is not optimal. Move to DistributedParallel (recommended) using the Multi-GPU support of NVT Data loaders and check which Transformer architectures work

References:
Context on DataParallel / Distributed Parallel: https://github.com/rapidsai/recsys/tree/main/transformers4recsys/codes#2-run-the-training--evaluation-script
NVT with Horovod: https://nbviewer.jupyter.org/github/NVIDIA/NVTabular/blob/main/examples/multi-gpu-movielens/01-03b-MultiGPU-Download-Convert-ETL-with-NVTabular-Training-with-PyTorch.ipynb"

Test Transformers4Rec pipeline with multi-GPU PyTorch NVT data loader.
Check if NVT supports DataParallel or DistributedDataparallel and if HF supports Horovod

• Better modularize our Meta-Architecture, make it more flexible and testable

This requires the PyTorch modules in here to be enhanced to generate NVT, PyTorch and ensemble model config files with multi-hot input inference support.

Ability to save/ load the model checkpoints and optimizer states

Since it's a core building-block of the library, I would aim for a minimum of 80% test-coverage.

Currently blocked by #51.

When mf_constrained_embeddings is set to True in SequentialPrediction task, the output layer is tied to the embedding table of item-id. the prediction head should therefore get this table from the model block.

• Preproc with NVTabular
• Training with Transformers4Rec
• Inference with Triton

This aggregator needs to be able to aggregate sequential features with tabular features.

Fix and test the torch implementations of:

• CausalLanguageModeling
• MaskedLanguageModeling
• PermutationLanguageModeling
• ReplacementLanguageModeling

Needs to happen after Marc finishes the NVTabular schema/column tagging API

Create an API for the Transformers4Rec modules (Meta-Architecture, Evaluation, Logging), allowing users to define their own training and evaluation pipeline (not necessarily using our recsys_main.py script)

Documentation of Transformers4Rec: how to install with docker and conda, example commands to run preproc and training, API documentation

Adjust Docker files to use the nvcr.io/nvidia/merlin/merlin-pytorch-training:0.5 as base image

• The current version returns 2D tensor where sequential inputs are combined using EmbeddingBag with 'mean' combiner. Instead, we should support returning a 3-D tensor to build item interaction embedding for session-based task.

• Requires fixing the issue #67

The related files are :

• https://github.com/NVIDIA-Merlin/Transformers4Rec/blob/main/transformers4rec/torch/tabular.py
• https://github.com/NVIDIA-Merlin/Transformers4Rec/blob/main/transformers4rec/torch/features/embedding.py
• https://github.com/NVIDIA-Merlin/Transformers4Rec/blob/main/transformers4rec/torch/features/continuous.py

Migrate the preprocessing notebook for the REES46 ecommerce from PySpark to NVT

This ticket includes the 3 basic aggregators we support for non-sequential data:

• ConcatFeatures
• StackFeatures
• ElementwiseSum

Upgrade Transformers4Rec code (currently supporting HF v4.1.1) to support the latest HF Transformers release (v4.6.0) https://github.com/huggingface/transformers/releases

Create a notebook for training a Transformer model with pre-processed yoochoose dataset using NVTabular and Transformers4Rec API. The steps are :

• #111
• Load features from the Tagging schema
• Define the sequential aggregator to create interaction embeddings
• Instantiate the masking class for a given language modelling task
• Instantiate the sequential model : Load tf4rec config and init the related HF class
• Instantiate the sequential prediction task with related metrics and loss
• Run training using two first day
• Run evaluation on the third day

N.B: This notebook is the simpler version of the Transformers4Rec tasks.

For multi-task learning, head class defines add methods related to binary and classification tasks. We need to add a method for item_prediction_task. The labels could be retrieved either from: ColumnGroup or MaskedSequence.

Create automated tests for the Transformers4Rec project

This is a requirement for RecSys21 tutorial. Once we load the models to the Triton server, we then should be able to create a client.py or an example code in jp notebook to send request to the server, and generate final prediction results for the next item to be clicked.

Load categorical features from a config (dict or yaml file) instead of ColumnGroup

Check if any required common session-based recommendation op is missing for CPU