This codebase implements the system described in the paper:

Unsupervised Scale-consistent Depth and Ego-motion Learning from Monocular Video

Jia-Wang Bian, Zhichao Li, Naiyan Wang, Huangying Zhan, Chunhua Shen, Ming-Ming Cheng, Ian Reid

NeurIPS 2019

See the paper on [arXiv] and the [project webpage] for more details.

1. A geometry consistency loss for enforcing the scale-consistency of predictions between consecutive frames.
2. A self-discovered mask for detecting moving objects and occlusions.
3. Enabling the unsupervised estimator (learned from monocular videos) to do visual odometry on a long video.

This codebase was developed and tested with python 3.6, Pytorch 1.0.1, and CUDA 10.0 on Ubuntu 16.04. It is based on Clement Pinard's SfMLearner implementation, in which we make little modification and add our proposed losses.

pip3 install -r requirements.txt

or install manually the following packages :

torch >= 1.0.1
imageio
matplotlib
scipy
argparse
tensorboardX
blessings
progressbar2
path.py
evo

It is also advised to have python3 bindings for opencv for tensorboard visualizations

See "scripts/run_prepare_data.sh" for examples, including KITTI Raw, Cityscapes, and KITTI Odometry.

For KITTI Raw dataset, download the dataset using this script provided on the official website.

For Cityscapes, download the following packages: 1) leftImg8bit_sequence_trainvaltest.zip, 2) camera_trainvaltest.zip. You will probably need to contact the administrators to be able to get it.

For KITTI Odometry dataset download the dataset with color images.

The "scripts" folder provides several examples for training and testing.

You can train the depth model on KITTI Raw by running

sh scripts/train_resnet_256.sh

or train the pose model on KITTI Odometry by running

sh scripts/train_posenet_256.sh

Then you can start a tensorboard session in this folder by

tensorboard --logdir=checkpoints/

and visualize the training progress by opening https://localhost:6006 on your browser.

You can evaluate depth using Eigen's split by running

sh scripts/run_depth_test.sh

and evaluate visual odometry by running

sh scripts/run_vo_test.sh

Also, you can evaluate 5-frame pose as SfMLearner by running

sh scripts/run_pose_test.sh

Avalaible here

Note that depth models are trained on KITTI Raw dataset, and pose models are trained on KITTI Odometry dataset, respectively. They are not coupled.

@inproceedings{bian2019depth,
  title={Unsupervised Scale-consistent Depth and Ego-motion Learning from Monocular Video},
  author={Bian, Jia-Wang and Li, Zhichao and Wang, Naiyan and Zhan, Huangying and Shen, Chunhua and Cheng, Ming-Ming and Reid, Ian},
  booktitle= {Thirty-third Conference on Neural Information Processing Systems (NeurIPS)},
  year={2019}
}