This is the implementation of the paper Structure-Aware Residual Pyramid Network for Monocular Depth Estimation, IJCAI 2019, Xiaotian Chen, Xuejin Chen, and Zheng-Jun Zha.
@inproceedings{Chen2019structure-aware,
title = {Structure-Aware Residual Pyramid Network for Monocular Depth Estimation},
author = {Chen, Xiaotian and Chen , Xuejin and Zha, Zheng-Jun},
conference={International Joint Conferences on Artificial Intelligence},
year = {2019}
}
@article{Chen2020LapNet,
title={Laplacian Pyramid Neural Network for Dense Continuous-Value Regression for Complex Scenes},
author={Chen, Xuejin and Chen, Xiaotian and Zhang, Yiteng and Fu, Xueyang and Zha, Zheng-Jun},
journal={IEEE TNNLS},
year={2020}
}
Monocular depth estimation is an essential task for scene understanding. The underlying structure of objects and stuff in a complex scene is critical to recovering accurate and visually-pleasing depth maps. Global structure conveys scene layouts,while local structure reflects shape details. Recently developed approaches based on convolutional neural networks (CNNs) significantly improve the performance of depth estimation. However, few of them take into account multi-scale structures in complex scenes. In this paper, we propose a Structure-Aware Residual Pyramid Network (SARPN) to exploit multi-scale structures for accurate depth prediction. We propose a Residual Pyramid Decoder (RPD) which expresses global scene structure in upper levels to represent layouts, and local structure in lower levels to present shape details. At each level, we propose Residual Refinement Modules (RRM) that predict residual maps to progressively add finer structures on the coarser structure predicted at the upper level. In order to fully exploit multi-scale image features, an Adaptive Dense Feature Fusion (ADFF) module, which adaptively fuses effective features from all scales for inferring structures of each scale, is introduced.
As an example, use the following command to train SARPN on NYUDV2.
CUDA_VISIBLE_DEVICES="0,1,2,3" python train.py --trainlist_path (the path of trainlist(nyu2_train.csv))\
--checkpoint_dir (the directory to save the checkpoints)\
--root_path (the root path of dataset)\
--logdir (the directory to save logs and checkpoints)\
--pretrained_dir (the path of pretrained models)\
--do_summary
Use the following command to evaluate the trained SARPN on NYUDV2 test data.
CUDA_VISIBLE_DEVICES="0" python evaluate.py --testlist_path (the path of testlist(nyu2_test.csv))\
--root_path (the root path of dataset)\
--loadckpt (the path of the loaded model)\
--pretrained_dir (the path of pretrained models)\
--threshold (threshold of the pixels on edges)
You can download the pretrained model:
NOTE: You don't need to decompress our pre-trained model. Please use torch.load() to load it.
NYUDV2 KITTI and Height estimation
You can download the pre-processed data from this link, which is shared by Junjie Hu.
Indoor scene
Outdoor scene
Thanks to Junjie Hu for opening source of his excellent work. Our work is inspired by this work and part of codes.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent