• See our new implementation in https://github.com/JiawangBian/sc_depth_pl

This codebase implements the system described in the paper:

Unsupervised Scale-consistent Depth Learning from Video

Jia-Wang Bian, Huangying Zhan, Naiyan Wang, Zhichao Li, Le Zhang, Chunhua Shen, Ming-Ming Cheng, Ian Reid

IJCV 2021 [PDF]

This is an extended version of NeurIPS 2019 [PDF] [Project webpage]

1. A geometry consistency loss, which makes the predicted depths to be globally scale consistent.
2. A self-discovered mask, which detects moving objects and occlusions for boosting accuracy.
3. Scale-consistent predictions, which can be used in the Monocular Visual SLAM system.

@article{bian2021ijcv, 
  title={Unsupervised Scale-consistent Depth Learning from Video}, 
  author={Bian, Jia-Wang and Zhan, Huangying and Wang, Naiyan and Li, Zhichao and Zhang, Le and Shen, Chunhua and Cheng, Ming-Ming and Reid, Ian}, 
  journal= {International Journal of Computer Vision (IJCV)}, 
  year={2021} 
}

Note that this is an improved version, and you can find the NeurIPS version in 'Release / NeurIPS Version' for reproducing the results reported in paper. Compared with NeurIPS version, we (1) Change networks by using Resnet18 and Resnet50 pretrained model (on ImageNet) for depth and pose encoders. (2) Add 'auto_mask' by Monodepth2 to remove stationary points. (3) Integrate the depth and pose prediction into the ORB-SLAM system. (4) Add training and testing on NYUv2 indoor dataset. See Unsupervised-Indoor-Depth for details.

This codebase was developed and tested with python 3.6, Pytorch 1.0.1, and CUDA 10.0 on Ubuntu 16.04. It is based on Clement Pinard's SfMLearner implementation.

pip3 install -r requirements.txt

See "scripts/run_prepare_data.sh".

For KITTI Raw dataset, download the dataset using this script http://www.cvlibs.net/download.php?file=raw_data_downloader.zip) provided on the official website.

For KITTI Odometry dataset, download the dataset with color images.

Or you can download our pre-processed dataset from the following link

kitti_256 (for kitti raw) | kitti_vo_256 (for kitti odom) | kitti_depth_test (eigen split) | kitti_vo_test (seqs 09-10)

The "scripts" folder provides several examples for training and testing.

You can train the depth model on KITTI Raw by running

sh scripts/train_resnet18_depth_256.sh

or train the pose model on KITTI Odometry by running

sh scripts/train_resnet50_pose_256.sh

Then you can start a tensorboard session in this folder by

tensorboard --logdir=checkpoints/

and visualize the training progress by opening https://localhost:6006 on your browser.

You can evaluate depth on Eigen's split by running

sh scripts/test_kitti_depth.sh

evaluate visual odometry by running

sh scripts/test_kitti_vo.sh

and visualize depth by running

sh scripts/run_inference.sh

Latest Models

To evaluate the NeurIPS models, please download the code from 'Release/NeurIPS version'.

• SfMLearner-Pytorch (CVPR 2017, our baseline framework.)

• Depth-VO-Feat (CVPR 2018, trained on stereo videos for depth and visual odometry)

• DF-VO (ICRA 2020, use scale-consistent depth with optical flow for more accurate visual odometry)

• Kitti-Odom-Eval-Python (python code for kitti odometry evaluation)

• Unsupervised-Indoor-Depth (Using SC-SfMLearner in NYUv2 dataset)