This repository contains MATLAB codes and data for sparse depth sensing, the problem of dense depth image reconstruction from very limited amount of measurements. Please refer to our paper Sparse Depth Sensing for Resource-Constrained Robots and click on the YouTube video below for more details.

• Matlab R2015a or later versions, including
  • the Computer Vision System Toolbox
  • the Robotics System Toolbox

• The code is self-contained. No installation required.
• However, if you are interested in trying out a differnet solver CVX, please download it from this link and follow the installation instructions there. CVX is a Matlab-based modeling system for convex optimization and it has slightly higher accuracy than our fast solver NESTA. We recommend obtaining an academic license for the most optimized performance.

• run demo_single_frame.m for a demo of the reconstruction algorithm on samples from each individual frame of depth images.
• run demo_multi_frame.m for a demo of the reconstruction algorithm on samples collected across multiple frames, given odometry information.
• run demo_middlebury.m for a demo of the reconstruction algorithm on the Middlebury dataset.

The code is structured as follows.

• lib/algorithm contains the core code, i.e., the formulation of our optimization problem (in l1ReconstructionOnImage.m).
• lib/nesta_solver is the implementation of the NESTA fast solver tailored to our problem.
• lib/geometry handles all geometry related implementations (e.g., rigid body transformation, image projection).
• lib/sampling provides functions for create a small set of measurements from the ground truth depth image.
• lib/utility contains other helper functions.

The data folder contains two datasets, including

• ZED: rgb images and depth images collected from the ZED Stereo Camera.
• lids_floor6: rgb and depth images collected from the Kinect sensor, along with odometry information obtained from a odometer.

Tested on following hosts

• Matlab R2017a on Windows 7 Enterprise
• Matlab R2016b on Ubuntu 16.04 LTS
• Matlab R2017a on macOS Sierra (Version 10.12.5)

Future work:

• add CUDA code for parallelization on GPU

If you use our code in your research, please consider citing:

@article{Ma2017SparseDepthSensing,
  title={Sparse Depth Sensing for Resource-Constrained Robots},
  author={Ma, Fangchang and Carlone, Luca and Ayaz, Ulas and Karaman, Sertac},
  journal={arXiv preprint arXiv:1703.01398},
  year={2017}
}

@inproceedings{ma2016sparse,
  title={Sparse sensing for resource-constrained depth reconstruction},
  author={Ma, Fangchang and Carlone, Luca and Ayaz, Ulas and Karaman, Sertac},
  booktitle={Intelligent Robots and Systems (IROS), 2016 IEEE/RSJ International Conference on},
  pages={96--103},
  year={2016},
  organization={IEEE}
}

Please email Fangchang Ma ([email protected]) for problems and bugs. Thanks!