Our repository provides:
1) a time line of the development in the related research community 2) a taxonomy of the survey 3) a public and holistic benchmark 4) collects deep learning-based camera calibration Publication, Objective, Network 5) dataset summary
More content and details can be found in our Survey Paper: Deep Learning for Camera Calibration and Beyond: A Survey.
- 2024.01.05: The benchmark is released. Please refer to the dataset link and more details in Benchmark.
- 2023.03.19: The survey of the arXiv version is online.
- Multiple view geometry in computer vision - Hartley, R., & Zisserman, A. (2004)
- A Flexible New Technique for Camera Calibration - Zhengyou Zhang. (2000)
Year | Publication | Title | Abbreviation | Objective | Platform | Network |
---|---|---|---|---|---|---|
2015 | ICIP | Deepfocal: A method for direct focal length estimation | DeepFocal | Intrinsics | Caffe | AlexNet |
2015 | ICCV | Posenet: A convolutional network for real-time 6-dof camera relocalization | PoseNet | Extrinsics | Caffe | GoogLeNet |
2016 | BMVC | Horizon lines in the wild | DeepHorizon | Extrinsics | Caffe | GoogLeNet |
2016 | CVPR | Detecting vanishing points using global image context in a non-manhattan world | DeepVP | Extrinsics | Caffe | AlexNet |
2016 | ACCV | Radial lens distortion correction using convolutional neural networks trained with synthesized images | Rong et al. | Distortion coefficients | Caffe | AlexNet |
2016 | RSSW | Deep image homography estimation | DHN | Projection matrixs | Caffe | VGG |
2017 | CVPR | Clkn: Cascaded lucas-kanade networks for image alignment | CLKN | Projection matrixs | Torch | CNN + Lucas-Kanade layer |
2017 | ICCVW | Homography estimation from image pairs with hierarchical convolutional networks | HierarchicalNet | Projection matrixs | TensorFlow | VGG |
2017 | CVPR | Unrolling the Shutter: CNN to Correct Motion Distortions | URS-CNN | Undistortion | Torch | CNNs |
2017 | IV | RegNet: Multimodal sensor registration using deep neural networks | RegNet | Camera + LiDAR | Caffe | CNNs |
2018 | CVPR | A perceptual measure for deep single image camera calibration | Hold-Geoffroy et al. | Intrinsics + Extrinsics | DenseNet | |
2018 | CVMP | DeepCalib: a deep learning approach for automatic intrinsic calibration of wide field-of-view cameras | DeepCalib | Intrinsics + Distortion coefficients | TensorFlow | Inception-V3 |
2018 | ECCV | Fisheyerecnet: A multi-context collaborative deep network for fisheye image rectification | FishEyeRecNet | Distortion coefficients | Caffe | VGG |
2018 | ICPR | Radial lens distortion correction by adding a weight layer with inverted foveal models to convolutional neural networks | Shi et al. | Distortion coefficients | PyTorch | ResNet |
2018 | ECCV | Deep fundamental matrix estimation | DeepFM | Projection matrixs | PyTorch | ResNet |
2018 | ECCVW | Deep fundamental matrix estimation without correspondences | Poursaeed et al. | Projection matrixs | CNNs | |
2018 | RAL | Unsupervised deep homography: A fast and robust homography estimation model | UDHN | Projection matrixs | TensorFlow | VGG |
2018 | ACCV | Rethinking planar homography estimation using perspective fields | PFNet | Projection matrixs | TensorFlow | FCN |
2018 | IROS | CalibNet: Geometrically Supervised Extrinsic Calibration using 3D Spatial Transformer Networks | CalibNet | Camera + LiDAR | TensorFlow | ResNet |
2018 | ICRA | DeepVP: Deep Learning for Vanishing Point Detection on 1 Million Street View Images | Chang et al. | Standard | Matconvnet | AlexNet |
2019 | CVPR | Deep single image camera calibration with radial distortion | Lopez et al. | Intrinsics + Extrinsics + Distortion coefficients | PyTorch | DenseNet |
2019 | ICCV | UprightNet: geometry-aware camera orientation estimation from single images | UprightNet | Extrinsics | PyTorch | U-Net |
2019 | IROS | Degeneracy in self-calibration revisited and a deep learning solution for uncalibrated slam | Zhuang et al. | Intrinsics + Distortion coefficients | PyTorch | ResNet |
2019 | PRL | Self-Supervised deep homography estimation with invertibility constraints | SSR-Net | Projection matrixs | PyTorch | ResNet |
2019 | ICCVW | A geometric approach to obtain a bird's eye view from an image | Abbas et al. | Projection matrixs | TensorFlow | CNNs |
2019 | TCSVT | DR-GAN: Automatic radial distortion rectification using conditional GAN in real-time | DR-GAN | Undistortion | TensorFlow | GANs |
2019 | TCSVT | Distortion rectification from static to dynamic: A distortion sequence construction perspective | STD | Undistortion | TensorFlow | GANs |
2019 | VR | Deep360Up: A deep learning-based approach for automatic VR image upright adjustment | Deep360Up | Extrinsics | DenseNet | |
2019 | JVCIR | Unsupervised fisheye image correction through bidirectional loss with geometric prior | UnFishCor | Distortion coefficients | TensorFlow | VGG |
2019 | CVPR | Blind geometric distortion correction on images through deep learning | BlindCor | Undistortion | PyTorch | U-Net |
2019 | CVPR | Learning structure-and-motion-aware rolling shutter correction | RSC-Net | Undistortion | PyTorch | ResNet |
2019 | CVPR | Learning to calibrate straight lines for fisheye image rectification | Xue et al. | Distortion coefficients | PyTorch | ResNet |
2019 | ICCV | Learning perspective undistortion of portraits | Zhao et al. | Intrinsics + Undistortion | VGG + U-Net | |
2019 | NeurIPS | NeurVPS: Neural Vanishing Point Scanning via Conic Convolution | NeurVPS | Standard | PyTorch | CNNs |
2020 | CVPR | End-to-end camera calibration for broadcast videos | Sha et al. | Projection matrixs | TensorFlow | Siamese-Net + U-Net |
2020 | ECCV | Neural geometric parser for single image camera calibration | Lee et al. | Intrinsics + Extrinsics | PointNet + CNNs | |
2020 | ICRA | Learning camera miscalibration detection | MisCaliDet | Average pixel position difference | TensorFlow | CNNs |
2020 | WACV | DeepPTZ: deep self-calibration for PTZ cameras | DeepPTZ | Intrinsics + Extrinsics + Distortion coefficients | PyTorch | Inception-V3 |
2020 | CVPR | Deep homography estimation for dynamic scenes | MHN | Projection matrixs | TensorFlow | VGG |
2020 | ACMMM | SRHEN: Stepwise-Refining Homography Estimation Network via Parsing Geometric Correspondences in Deep Latent Space | SRHEN | Projection matrixs | CNNs | |
2020 | ECCV | 360∘ camera alignment via segmentation | Davidson et al. | Extrinsics | FCN | |
2020 | ECCV | Content-aware unsupervised deep homography estimation | CA-UDHN | Projection matrixs | PyTorch | FCN + ResNet |
2020 | IROS | Deep keypoint-based camera pose estimation with geometric constraints | DeepFEPE | Extrinsics | PyTorch | VGG + PointNet |
2020 | TIP | Model-free distortion rectification framework bridged by distortion distribution map | DDM | Undistortion | Tensorflow | GANs |
2020 | TIP | Deep face rectification for 360° dual-fisheye cameras | Li et al. | Undistortion | CNNs | |
2020 | ICPR | Position-aware and symmetry enhanced GAN for radial distortion correction | PSE-GAN | Undistortion | GANs | |
2020 | ICIP | A simple yet effective pipeline for radial distortion correction | RDC-Net | Undistortion | PyTorch | ResNet |
2020 | ICASSP | Self-supervised deep learning for fisheye image rectification | FE-GAN | Undistortion | PyTorch | GANs |
2020 | CVPR | RDCFace: radial distortion correction for face recognition | RDCFace | Undistortion | ResNet | |
2020 | arXiv | Fisheye distortion rectification from deep straight lines | LaRecNet | Distortion coefficients | PyTorch | ResNet |
2020 | CVPR | Height and uprightness invariance for 3d prediction from a single view | Baradad et al. | Intrinsics + Extrinsics | PyTorch | CNNs |
2020 | CVPR | What does plate glass reveal about camera calibration? | Zheng et al. | Intrinsics + Extrinsics | CNNs | |
2020 | ECCV | Single view metrology in the wild | Zhu et al. | Intrinsics + Extrinsics | PyTorch | CNNs + PointNet |
2020 | CVPR | Deep Shutter Unrolling Network | DeepUnrollNet | Undistortion | PyTorch | FCN |
2020 | RAL | RGGNet: Tolerance Aware LiDAR-Camera Online Calibration With Geometric Deep Learning and Generative Model | RGGNet | Camera + LiDAR | Tensorflow | ResNet |
2020 | IROS | CalibRCNN: Calibrating Camera and LiDAR by Recurrent Convolutional Neural Network and Geometric Constraints | CalibRCNN | Camera + LiDAR | Tensorflow | RNN |
2020 | ICRA | Online Camera-LiDAR Calibration with Sensor Semantic Information | SSI-Calib | Camera-LiDAR | Tensorflow | CNNs |
2020 | arXiv | SOIC: Semantic Online Initialization and Calibration for LiDAR and Camera | SOIC | Camera-LiDAR | - | ResNet+PointRCNN |
2020 | ICPR | NetCalib: A Novel Approach for LiDAR-Camera Auto-calibration Based on Deep Learning | NetCalib | Camera-LiDAR | PyTorch | CNNs |
2021 | TCI | Online training of stereo self-calibration using monocular depth estimation | StereoCaliNet | Extrinsics | PyTorch | U-Net |
2021 | ICCV | CTRL-C: Camera calibration TRansformer with Line-Classification | CTRL-C | Intrinsics + Extrinsics | PyTorch | Transformer |
2021 | ICCVW | Deep single fisheye image camera calibration for over 180-degree projection of field of view | Wakai et al. | Intrinsics + Extrinsics | DenseNet | |
2021 | TIP | A deep ordinal distortion estimation approach for distortion rectification | OrdianlDistortion | Distortion coefficients | TensorFlow | CNNs |
2021 | TCSVT | Revisiting radial distortion rectification in polar-coordinates: A new and efficient learning perspective | PolarRecNet | Undistortion | PyTorch | VGG + U-Net |
2021 | PRL | DQN-based gradual fisheye image rectification | DQN-RecNet | Undistortion | PyTorch | VGG |
2021 | CVPR | Practical wide-angle portraits correction with deep structured models | Tan et al. | Undistortion | PyTorch | U-Net |
2021 | CVPR | Progressively complementary network for fisheye image rectification using appearance flow | PCN | Undistortion | PyTorch | U-Net |
2021 | ICCV | Multi-level curriculum for training a distortion-aware barrel distortion rectification model | DaRecNet | Undistortion | TensorFlow | U-Net |
2021 | CVPR | Deep Lucas-Kanade homography for multimodal image alignment | DLKFM | Projection matrixs | TensorFlow | Siamese-Net |
2021 | ICCV | LocalTrans: A multiscale local transformer network for cross-resolution homography estimation | LocalTrans | Projection matrixs | PyTorch | Transformer |
2021 | ICCV | Motion basis learning for unsupervised deep homography estimation with subspace projection | BasesHomo | Projection matrixs | PyTorch | ResNet |
2021 | ICIP | Fast and accurate homography estimation using extendable compression network | ShuffleHomoNet | Projection matrixs | TensorFlow | ShuffleNet |
2021 | TCSVT | Depth-aware multi-grid deep homography estimation with contextual correlation | DAMG-Homo | Projection matrixs | TensorFlow | CNNs |
2021 | BMVC | A simple approach to image tilt correction with self-attention MobileNet for smartphones | SA-MobileNet | Extrinsics | TensorFlow | MobileNet |
2021 | ICCV | SPEC: Seeing people in the wild with an estimated camera | SPEC | Intrinsics + Extrinsics | PyTorch | ResNet |
2021 | CVPR | Wide-Baseline Relative Camera Pose Estimation with Directional Learning | DirectionNet | Extrinsics | TensorFlow | U-Net |
2021 | CVPR | Towards Rolling Shutter Correction and Deblurring in Dynamic Scenes | JCD | Undistortion | PyTorch | FCN |
2021 | CVPRW | LCCNet: LiDAR and Camera Self-Calibration using Cost Volume Network | LCCNet | Camera + LiDAR | PyTorch | CNNs |
2021 | Sensors | CFNet: LiDAR-Camera Registration Using Calibration Flow Network | CFNet | Camera + LiDAR | PyTorch | FCN |
2021 | ICCV | Inverting a Rolling Shutter Camera: Bring Rolling Shutter Images to High Framerate Global Shutter Video | Fan et al. | Distortion | PyTorch | U-Net |
2021 | ICCV | SUNet: Symmetric Undistortion Network for Rolling Shutter Correction | SUNet | Distortion | PyTorch | DenseNet+ResNet |
2021 | IROS | SemAlign: Annotation-Free Camera-LiDAR Calibration with Semantic Alignment Loss | SemAlign | Camera-LiDAR | PyTorch | CNNs |
2022 | CVPR | Deep vanishing point detection: Geometric priors make dataset variations vanish | DVPD | Extrinsics | PyTorch | CNNs |
2022 | ICRA | Self-supervised camera self-calibration from video | Fang et al. | Intrinsics + Extrinsics | PyTorch | CNNs |
2022 | ICASSP | Camera calibration through camera projection loss | CPL | Intrinsics + Extrinsics | TensorFlow | Inception-V3 |
2022 | CVPR | Iterative Deep Homography Estimation | IHN | Projection matrixs | PyTorch | Siamese-Net |
2022 | CVPR | Unsupervised Homography Estimation with Coplanarity-Aware GAN | HomoGAN | Projection matrixs | PyTorch | GANs |
2022 | CVPR | Semi-Supervised Wide-Angle Portraits Correction by Multi-Scale Transformer | SS-WPC | Undistortion | PyTorch | Transformer |
2022 | CVPR | Learning Adaptive Warping for Real-World Rolling Shutter Correction | AW-RSC | Undistortion | CNNs | |
2022 | CVPR | EvUnroll: Neuromorphic Events based Rolling Shutter Image Correction | EvUnroll | Undistortion | PyTorch | U-Net |
2022 | CVPR | Learning to Detect Scene Landmarks for Camera Localization | Do et al. | Extrinsics | PyTorch | ResNet |
2022 | CVPR | DiffPoseNet: Direct Differentiable Camera Pose Estimation | DiffPoseNet | Extrinsics | PyTorch | CNNs + LSTM |
2022 | CVPR | SceneSqueezer: Learning to Compress Scene for Camera Relocalization | SceneSqueezer | Extrinsics | PyTorch | Transformer |
2022 | arXiv | FishFormer: Annulus Slicing-based Transformer for Fisheye Rectification with Efficacy Domain Exploration | FishFormer | Undistortion | PyTorch | Transformer |
2022 | CVPR | Focal Length and Object Pose Estimation via Render and Compare | FocalPose | Intrinsics + Extrinsics | PyTorch | CNNs |
2022 | arXiv | DXQ-Net: Differentiable LiDAR-Camera Extrinsic Calibration Using Quality-aware Flow | DXQ-Net | Camera + LiDAR | PyTorch | CNNs + RNNs |
2022 | ITSC | SST-Calib: Simultaneous Spatial-Temporal Parameter Calibration between LIDAR and Camera | SST-Calib | Camera + LiDAR | PyTorch | CNNs |
2022 | IROS | Learning-Based Framework for Camera Calibration with Distortion Correction and High Precision Feature Detection | CCS-Net | Undistortion | PyTorch | UNet |
2022 | TIP | SIR: Self-supervised image rectification via seeing the same scene from multiple different lenses | SIR | Undistortion | PyTorch | ResNet |
2022 | TIV | ATOP: An Attention-to-Optimization Approach for Automatic LiDAR-Camera Calibration Via Cross-Modal Object Matching | ATOP | Camera + LiDAR | CNNs | |
2022 | ICRA | FusionNet: Coarse-to-Fine Extrinsic Calibration Network of LiDAR and Camera with Hierarchical Point-pixel Fusion | FusionNet | Camera + LiDAR | PyTorch | CNNs+PointNet |
2022 | TIM | Keypoint-Based LiDAR-Camera Online Calibration With Robust Geometric Network | RGKCNet | Camera + LiDAR | PyTorch | CNNs+PointNet |
2022 | ECCV | Rethinking generic camera models for deep single image camera calibration to recover rotation and fisheye distortion | GenCaliNet | Intrinsics + Extrinsics + Distortion coefficients | DenseNet | |
2022 | PAMI | Content-Aware Unsupervised Deep Homography Estimation and Beyond | Liu et al. | Projection matrixs | PyTorch | ResNet |
Name | Publication | Real/Synthetic | Image/Video | Objectives | Dataset |
---|---|---|---|---|---|
KITTI | CVPR | Real | Video | Base | Dataset |
MS-COCO | ECCV | Real | Image | Base | Dataset |
SUN360 | CVPR | Real | Image | Base | Dataset |
Places2 | PAMI | Real | Image | Base | Dataset |
CelebA | ICCV | Real | Image | Base | Dataset |
1DSfM | ECCV | Real | Image | Focal Length | Dataset |
Cambridge Landmarks | ICCV | Real | Video | Extrinsics | Dataset |
HLW | BMVC | Real | Image | Horizon Line | Dataset |
YUD | ECCV | Real | Image | Vanishing Point | Dataset |
ECD | ECCV | Real | Image | Vanishing Point | Dataset |
SU3 Wireframe | ICCV | Synthetic | Image | Vanishing Point | Dataset |
ScanNet | CVPR | Real | Video | Extrinsics | Dataset |
Indoor-6 | CVPR | Real | Image | Extrinsics | Dataset |
DeepVP | ICRA | Real | Image | Vanishing Point | Dataset |
CAHomo | ECCV | Real | Video | Homography | Dataset |
MHN | CVPR | Real | Video | Homography | Dataset |
UDIS | TIP | Real | Video | Homography | Dataset |
Carla-RS | CVPR | Synthetic | Video | RS-Distortion | Dataset |
Fastec-RS | CVPR | Synthetic | Video | RS-Distortion | Dataset |
BS-RSC | CVPR | Real | Video | RS-Distortion | Dataset |
GEV-RS | CVPR | Real | Video | RS-Distortion | Dataset |
LMS | ICASSP | Both | Video | Radial Distortion | Dataset |
SS-WPC | CVPR | Real | Image | Radial Distortion | Dataset |
The survey and benchmark are only made available for academic research purpose.
@article{kang2023deep,
Author = {Kang Liao and Lang Nie and Shujuan Huang and Chunyu Lin and Jing Zhang and Yao Zhao and Moncef Gabbouj and Dacheng Tao},
Title = {Deep Learning for Camera Calibration and Beyond: A Survey},
Year = {2023},
Journal = {arXiv:2303.10559}
}