If you find our codes and method useful to your work, please consider citing this work:

@Article{YIN2020-camera-lidar-calibration,
AUTHOR = {Lu Yin, Bin Luo, Wei Wang, Huan Yu, Chenjie Wang and Chengyuan Li},
TITLE = {CoMask: Corresponding Mask-Based End-to-End Extrinsic Calibration of the Camera and LiDAR},
JOURNAL = {Remote Sensing},
VOLUME = {12},
YEAR = {2020},
NUMBER = {12},
ARTICLE-NUMBER = {1925},
ISSN = {2072-4292},
DOI = {10.3390/rs12121925}
}

This is a matlab-GUI based toobox to calibrate the camera and lidar using the CoMask-based end-to-end method.

We propose a fully automatic end-to-end method based on the 3D-2D corresponding masks (CoMask) to directly estimate the extrinsic parameters with high precision. In particular, a simple subtraction is applied to extract the candidate point cluster from the complex background, then the 3D LiDAR points located on the checkerboard are selected and refined by spatial growth clustering. Once the distance transform of the 2D checkerboard mask is generated, the extrinsic calibration of the two sensors can be converted to the 3D-2D mask correspondences alignment. A simple but efficient strategy combining genetic algorithm with Levenberg-Marquardt method is used to solve the optimization problem without initial guess but can still converge to accurate result. Both simulated and real experiments show that the proposed method can get accurate result without manual intervention, special setup, and approximate initial guess in advance. Compared with the state-of-the-arts, our method has obvious advantages in accuracy, robustness and noise resistance.

The package is used to calibrate a Lidar with a camera. Specially, the csi camera has been successfully calibrated against Velodyne VLP-16 and Hesai P40P using chessboard_camera_lidar_calibration_toolbox. We show the reprojection error by projecting the plane Lidar points to the image plane, which has high accuracy. You can also load other image-lidar pairs captured by the same setup to see the calibration accuracy.

The package finds a rotation and translation that transform all the points in the LiDAR frame to the (monocular) camera frame.

• Setup
  • matlab
  • g2o
  • sophus
  • mex-build
• Getting Started
  • data preparation
• Usage
• A Simple Demo
• Parameters Description
• Future Improvements

The toolbox is so easy to setup due to the depenence only on matlab.

1. matlab
   For more support for ROS-interface, we strongly suggest you upgrade your matlab version at least after 2017b.

2. g2o
   G2O is used to solve the LM optimization problem.
   cd ./thirdParty/g2o/
mkdir build
cd build
cmake ..
make & sudo make install

3. sophus
   sophus is used to convert the problem from euler space to se3 space.
   cd ./thirdParty/Sophus-1.0.0/
mkdir build
cd build
cmake ..
make & sudo make install

4. mex-build
   cd ./utils/cloudSegmentation
build.m
cd ./utils/optimization
build.m

The toolbox depends on the image-points-pairs with chessboard. The quality of the data has some relationship of the calibration results.

1. Data Preparation
   You should prepare the time-sychronized image-points pairs with chessboard exsiting within the overlapped field view of both the camera and Lidar. You do not need too much observations and the distance and orientation of the checkerboard do not has too much limiatation. You can get the sample data with 25 observation from here

   When you are intend to prepare your data, you should keep these key points in mind:

   • The LiDAR points and camera images should be time sychronised.
   • If your system does not have hard synchronization, then the time synchronization between your image and the laser point cloud pair cannot have a very good synchronization effect. But you can use some methods to alleviate the asynchrony of time.
     • If you are using your data packets recorded by the ROS system, you can use the system timestamp as the timestamp of the data, so that you can get the data pair with the closest time through the correlation of the timestamp.
     • If your data is generated and recorded when there is a data transmission delay or storage delay, this will make the effect of synchronizing the point cloud and image data only through the system time stamp worse. In other words, there may be a systematic time deviation between the image and the point cloud data. At this time, you can add this deviation compensation when generating the data pair. The matlab-GUI framework we provide has a toolkit to deal with such problems, which will be further introduced later.
     • If the time synchronization of your system cannot be guaranteed, you need to be as careful as possible when recording data. You can keep the scene still while collecting data at the observation point you think is better, so that you can eliminate the data asynchrony caused by time asynchrony to a certain extent. Our MATLAB-GUI provides a tool interface for manually selecting observation data.
       
   • The image of the checkerboard target and the laser point cloud pair data need to be recorded and placed in the first frame of the data.
     • The core of our algorithm is to accurately and robustly extract the checkerboard pairs in the image and laser point cloud. In order to avoid using too many parameters and guidelines to filter out non-checkerboard targets in the laser point cloud, we adopted the point cloud subtraction method to obtain the new point cloud in the scene, that is, we need to record a background point cloud , And then obtain the remaining point cloud containing the main target by making the difference between the point cloud with the target and the background point cloud.
     • When extracting checkerboard pairs, our algorithm will default to storing the static background in the first frame, so the frame numbers of the static background data are taken as "000001.pcd" and "000001.png". Of course, our code framework also supports point clouds and images in other formats, but we recommend that you use the pcd format to store point clouds and the png format to store images, and please note that the number of our data has been named and is similar to the KITTI data set .
       

You should first run multimodal_data_studio.m to launch the GUI as the above. When the matlab ask you if you want to change your path or add to the path, you should choose change your path to focus on the current path.
There is a button on the main interface, corresponding to the toolkit for data visualization and data synchronization segmentation, the toolkit for lidar and image calibration, and the toolkit for camera internal parameter calibration.

The graphical interface provides the convenience of operation, but objectively brings the complexity of the operation, especially the problem of mutual influence of the operation. This description focuses on how to use lidar and camera calibration, so only a general description of the tool.

When you click the Lidar and camera calibration button on the main GUI, you can enter the camera and lidar calibration toolbox. This toolbox has two GUI interfaces, one is the interface that mainly contains operations, and the other is the main and view The relevant interface is shown in the above two figures.

Below we will explain the general function of these operation buttons:

The relevant points of data preparation have been mentioned above. Here is a sample data provided as an example to illustrate the composition form of the data that needs to be prepared.

You can get the sample data with 25 observation from sample data

The basic parameter settings are completed through the content of the setting button 9, including the point cloud type, the point cloud extraction checkerboard method, and the point cloud range. The default values of these parameters are stored in the "camera_lidar_calibration_config.yml" file. For your special point cloud, the parameters may be slightly different. You can modify the above configuration file, or you can modify it online on the tool. The part of the point cloud containing the checkerboard is filtered out by the range. We only process the point cloud within the filtering range, so you need to pay attention to whether your parameter settings are correct.

The checkerboard is extracted by calling button 10. When you click button 10, you will see a progress bar for processing the image, and the processing information will be displayed when the image is processed. Click "ok" to proceed to the next step.

After clicking "ok", the point cloud data will continue to be processed, a processing progress bar will appear, and the relevant logs of the extracted point cloud can be seen in the "command" window. After this process is complete, you can see the results of the processing.

After the point cloud processing is completed, click "ok", the platform will extract the processed effective point cloud pair, and refresh the data column on the left. Invalid data pairs will no longer be displayed.

You can set button 6 to modify the rendering effect. For example, select "extracted" to display the result of checkerboard corner extraction in the image window, and display the result of checkerboard extraction in the point cloud window. As shown in the figure, in the image window, the checkerboard corner points are displayed with blue crosshairs, and in the point cloud window, the checkerboard point cloud is rendered in red.

Start external parameter calibration by calling button 11. Using the "comask-ga-lm" method will first use the genetic algorithm to obtain a roughly correct initial parameter, and then use the LM method to obtain an accurate result based on this. During the calibration process, the corresponding log information will be output in the "command" window of matlab. After the calibration is completed, you can see the external parameters of the calibration output in the data window of the laser point cloud and in the area 23 through the refresh button 24.

Preview the accuracy of external parameter calibration by rendering the laser point cloud onto the image. You can set different rendering modes, or render all point clouds on the image, or just render the checkerboard point clouds on the image.

The quality of the rendering can be judged by rendering the point cloud, especially the checkerboard point cloud, onto the image. Possibly due to the adjustment of parameters and the degradation of data quality, the extraction accuracy of the checkerboard is poor. For example, the point cloud that is not a checkerboard is extracted into a checkerboard or the extraction of the checkerboard point cloud is incomplete. At this time, you can uncheck the frame data to remove the influence of the frame data on the results, and click the 11th button to recalibrate.

Press button 12 to save the calibration result as data in yml format.

The core of the algorithm in this paper is divided into two parts, one is the extraction of the corresponding checkerboard in the image and point cloud, and the other is the calibration of the external parameters based on the corresponding mask. These links all involve some parameters, and the selection of some parameters has a certain influence on the accuracy of the calibration results, so additional attention is needed.
Our parameters are mainly initialized by default in the "camera_lidar_calibration_config.yml" file. Now we will briefly explain the default values, meanings and setting opinions of the parameters in this file. Regarding the further adjustment of these parameters, you can put forward comments in the issue, and the author will explain and reply to comments as soon as possible.

You are welcome to submit questions and bug reports as GitHub Issues.

• Ethan: [email protected]