Official implementation of: Object-Centric Video Prediction via Decoupling of Object Dynamics and Interactions by Villar-Corrales et al. ICIP 2023. [Paper] [Project Page]
We refer to docs/INSTALL.md for detailed installation and preparation instructions.
We refer to docs/TRAIN.md for detailed instructions for training your own Object-Centric Video Decomposition model. Additonally, we report the required training time for both the SAVi scene decomposition, as well as the OCVP-Seq predictor module.
To reproduce the results provided in our paper, you can download our pretrained models, including checkpoints for the SAVi decomposition and prediction modules, by running the download_pretrained bash script:
chmod +x download_pretrained.sh
./download_pretrained.sh
You can evaluate a SAVi video decomposition model using the src/03_evaluate_savi_noMasks.py and src/03_evaluate_savi.py scripts. The former measures the quality of the reconstructed frames, whereas the latter measures the fidelity of the object masks.
Example:
python src/03_evaluate_savi_noMasks.py \
-d experiments/MOViA/ \
--checkpoint savi_movia.pth
python src/03_evaluate_savi.py \
-d experiments/MOViA/ \
--checkpoint savi_movia.pth
To evaluate an object-centric video predictor module (i.e. LSTM, Transformer, OCVP-Seq or OCVP-Par), you can use the src/05_evaluate_predictor.py script.
usage: 05_evaluate_predictor.py [-h] -d EXP_DIRECTORY -m SAVI_MODEL --name_predictor_experiment NAME_PREDICTOR_EXPERIMENT --checkpoint CHECKPOINT [--num_preds NUM_PREDS]
arguments:
-d EXP_DIRECTORY, --exp_directory EXP_DIRECTORY
Path to the father exp. directory
-m SAVI_MODEL, --savi_model SAVI_MODEL
Name of the SAVi checkpoint to load
--name_predictor_experiment NAME_PREDICTOR_EXPERIMENT
Name to the directory inside the exp_directory corresponding to a predictor experiment.
--checkpoint CHECKPOINT
Checkpoint with predictor pre-trained parameters to load for evaluation
--num_preds NUM_PREDS
Number of rollout frames to predict for
Example 1: Reproduce LSTM predictor results on the Obj3D dataset:
python src/05_evaluate_predictor.py \
-d experiments/Obj3D/ \
--savi_model savi_obj3d.pth \
--name_predictor_experiment Predictor_LSTM \
--checkpoint lstm_obj3d.pth \
--num_preds 25
Example 2: Reproduce OCVP-Seq predictor results on the MOVi-A dataset:
python src/05_evaluate_predictor.py \
-d experiments/MOViA/ \
--savi_model savi_movia.pth \
--name_predictor_experiment Predictor_OCVPSeq \
--checkpoint OCVPSeq_movia.pth \
--num_preds 18
To generate video prediction, object prediction and segmentation figures and animations, you can use the
src/06_generate_figs_pred.py script.
Example:
python src/06_generate_figs_pred.py \
-d experiments/Obj3D/ \
--savi_model savi_obj3d.pth \
--name_predictor_experiment Predictor_OCVPSeq \
--checkpoint OCVPSeq_obj3d.pth \
--num_seqs 10 \
--num_preds 25
Our work is inspired and uses resources from the following repositories:
This repository is maintained by Angel Villar-Corrales.
Please consider citing our paper if you find our work or our repository helpful.
@inproceedings{villar_ObjectCentricVideoPrediction_2023,
title={Object-Centric Video Prediction via Decoupling of Object Dynamics and Interactions},
author={Villar-Corrales, Angel and Wahdan, Ismail and Behnke, Sven},
booktitle={Internation Conference on Image Processing (ICIP)},
year={2023}
}
In case of any questions or problems regarding the project or repository, do not hesitate to contact the authors at [email protected].
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent