Code release for our paper
Understanding 3D Object Articulation in Internet Videos
Shengyi Qian, Linyi Jin, Chris Rockwell, Siyi Chen, David Fouhey
CVPR 2022
Please check the project page for more details and consider citing our paper if it is helpful:
@inproceedings{Qian22,
author = {Shengyi Qian and Linyi Jin and Chris Rockwell and Siyi Chen and David F. Fouhey},
title = {Understanding 3D Object Articulation in Internet Videos},
booktitle = {CVPR},
year = 2022
}
We are using pyenv to set up the anaconda environment. It is tested on pytorch 1.7.1, detectron2 0.4, and pytorch3d 0.4.0.
VERSION_ALIAS="articulation3d" PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install anaconda3-2020.11
# pytorch and pytorch3d
conda install -c pytorch pytorch=1.7.1 torchvision cudatoolkit=10.2
conda install -c fvcore -c iopath -c conda-forge fvcore iopath
conda install -c bottler nvidiacub
conda install pytorch3d -c pytorch3d
# detectron2 with pytorch 1.7, cuda 10.2
python -m pip install detectron2 -f https://dl.fbaipublicfiles.com/detectron2/wheels/cu102/torch1.7/index.htmlAlternatively, we have tested the anaconda virtual environment. It is tested on pytorch 1.12.1, detectron2 0.6, and pytorch3d 0.7.0.
conda create -n articulation3d python=3.8
conda activate articulation3d
conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch
pip install 'git+https://github.com/facebookresearch/pytorch3d.git@stable'
pip install 'git+https://github.com/facebookresearch/detectron2.git'To install python packages,
# other packages
pip install scikit-image matplotlib imageio plotly opencv-python
pip install mapbox-earcut
pip install numpy-quaternion
pip install imageio-ffmpeg
pip install scikit-learn
# install articulation3d
cd articulation3d
pip install -e .
If necessary, download our pretrained model and put it at exps/model_0059999.pth
mkdir exps
cd exps
wget https://www.dropbox.com/s/50uderl5ynan2yt/model_0059999.pth?dl=0To run the model and temporal optimization on a video,
python tools/inference.py --config config/config.yaml --input example.mp4 --output outputTo save the 3d model, add --save-obj and --webvis flags,
python tools/inference.py --config config/config.yaml --input example.mp4 --output output --save-obj --webvisOur training consists of three stages.
In the first stage, we train the bounding box on Internet videos.
python tools/train_net.py --config-file config/step1_bbox.yamlIn the second stage, we train articulation axis on Internet videos while freezing the backbone.
python tools/train_net.py --config-file config/step2_axis.yamlIn the final stage, we train the plane head on ScanNet images.
python tools/train_net.py --config-file config/step3_plane.yamlWe reuse the codebase of SparsePlanes and Mesh R-CNN.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent