youkely / infrascal Goto Github PK

View Code? Open in Web Editor NEW

218.0 218.0 55.0 62.45 MB

InfrasCal: Automatic Infrastructure-based Intrinsic and Extrinsic Calibration of a Multi-camera System

License: Other

CMake 7.11% C 0.34% C++ 90.26% Cuda 2.22% MATLAB 0.07%

infrascal's People

Stargazers

Watchers

infrascal's Issues

Error: Linear Solver Failure

Hello @youkely

Thank you for the amazing contrition to science.

I am trying to use your work to calibrate 8 camera (6 Simple-Radial and 2 Simple-Radial-Fisheye) on a custom "rig", and while doing so I am getting the following error.

W1020 12:22:03.798044 10037 levenberg_marquardt_strategy.cc:115] Linear solver failure. Failed to compute a step: CHOLMOD warning: Matrix not positive definite.

Due to this the results I am getting are not correct.

Also, I am attaching last few lines of the output.

# INFO: Reprojection error: avg = 25.2254226694 px | max = 10203.9231710676 px | count = 11817
# INFO: Optimize camera poses, odometry and scene points.
# INFO: Optimize camera intrinsics as well.
# INFO: Reprojection error: avg = 25.2254226694 px | max = 10203.9231710676 px | count = 11817
# INFO: Optimization took 20.3128063679 s.
# INFO: Reprojection error: avg = 6.0066607393 px | max = 10246.7209746549 px | count = 11817
# INFO: Initial reprojection error: avg = 6.0066607393 px | max = 10246.7209746549 px | count = 11817
# INFO: Reprojection error after pruning: avg = 0.9323049211 px | max = 19.2428337728 px | count = 7508
# INFO: Optimize camera poses, odometry and scene points.
# INFO: Optimize camera intrinsics as well.
# INFO: Reprojection error: avg = 0.9323049211 px | max = 19.2428337728 px | count = 7508
WARNING: Logging before InitGoogleLogging() is written to STDERR
W1020 12:32:21.544764 14604 levenberg_marquardt_strategy.cc:115] Linear solver failure. Failed to compute a step: CHOLMOD warning: Matrix not positive definite.
W1020 12:32:21.552954 14604 levenberg_marquardt_strategy.cc:115] Linear solver failure. Failed to compute a step: CHOLMOD warning: Matrix not positive definite.
W1020 12:32:21.666002 14604 levenberg_marquardt_strategy.cc:115] Linear solver failure. Failed to compute a step: CHOLMOD warning: Matrix not positive definite.
W1020 12:32:21.871100 14604 levenberg_marquardt_strategy.cc:115] Linear solver failure. Failed to compute a step: CHOLMOD warning: Matrix not positive definite.
W1020 12:32:21.914731 14604 levenberg_marquardt_strategy.cc:115] Linear solver failure. Failed to compute a step: CHOLMOD warning: Matrix not positive definite.
W1020 12:32:22.341405 14604 levenberg_marquardt_strategy.cc:115] Linear solver failure. Failed to compute a step: CHOLMOD warning: Matrix not positive definite.
W1020 12:32:22.392978 14604 levenberg_marquardt_strategy.cc:115] Linear solver failure. Failed to compute a step: CHOLMOD warning: Matrix not positive definite.
W1020 12:32:22.646796 14604 levenberg_marquardt_strategy.cc:115] Linear solver failure. Failed to compute a step: CHOLMOD warning: Matrix not positive definite.
W1020 12:32:22.953632 14604 levenberg_marquardt_strategy.cc:115] Linear solver failure. Failed to compute a step: CHOLMOD warning: Matrix not positive definite.
W1020 12:32:23.227234 14604 levenberg_marquardt_strategy.cc:115] Linear solver failure. Failed to compute a step: CHOLMOD warning: Matrix not positive definite.
W1020 12:32:23.283560 14604 levenberg_marquardt_strategy.cc:115] Linear solver failure. Failed to compute a step: CHOLMOD warning: Matrix not positive definite.
W1020 12:32:23.601552 14604 levenberg_marquardt_strategy.cc:115] Linear solver failure. Failed to compute a step: CHOLMOD warning: Matrix not positive definite.
W1020 12:32:23.953294 14604 levenberg_marquardt_strategy.cc:115] Linear solver failure. Failed to compute a step: CHOLMOD warning: Matrix not positive definite.
W1020 12:32:24.191643 14604 levenberg_marquardt_strategy.cc:115] Linear solver failure. Failed to compute a step: CHOLMOD warning: Matrix not positive definite.
W1020 12:32:24.269335 14604 levenberg_marquardt_strategy.cc:115] Linear solver failure. Failed to compute a step: CHOLMOD warning: Matrix not positive definite.
W1020 12:32:24.589169 14604 levenberg_marquardt_strategy.cc:115] Linear solver failure. Failed to compute a step: CHOLMOD warning: Matrix not positive definite.
W1020 12:32:24.855190 14604 levenberg_marquardt_strategy.cc:115] Linear solver failure. Failed to compute a step: CHOLMOD warning: Matrix not positive definite.
W1020 12:32:25.059864 14604 levenberg_marquardt_strategy.cc:115] Linear solver failure. Failed to compute a step: CHOLMOD warning: Matrix not positive definite.
W1020 12:32:25.281478 14604 levenberg_marquardt_strategy.cc:115] Linear solver failure. Failed to compute a step: CHOLMOD warning: Matrix not positive definite.
W1020 12:32:25.332710 14604 levenberg_marquardt_strategy.cc:115] Linear solver failure. Failed to compute a step: CHOLMOD warning: Matrix not positive definite.
W1020 12:32:25.678844 14604 levenberg_marquardt_strategy.cc:115] Linear solver failure. Failed to compute a step: CHOLMOD warning: Matrix not positive definite.
# INFO: Optimization took 8.1711330414 s.
# INFO: Reprojection error: avg = 0.9079871059 px | max = 29.6271476861 px | count = 7508
# INFO: Odometry distance: 1657821.3820502316 m
final result
camera_0
1.0000000000 -0.0000000000 0.0000000000 0.0000000000
-0.0000000000 1.0000000000 -0.0000000000 0.0000000000
-0.0000000000 -0.0000000000 1.0000000000 0.0000000000

camera_1
0.9995074477 -0.0024515332 0.0312866111 -0.0704844395
0.0024583481 0.9999969622 -0.0001793579 -0.0016178855
-0.0312860763 0.0002561830 0.9995104381 0.0052558259

camera_2
0.3663087674 0.0116839439 -0.9304199979 -0.5965381791
0.0100420079 0.9998132892 0.0165089292 0.0264367677
0.9304391678 -0.0153906505 0.3661230433 -0.6143894830

camera_3
-0.8158658774 -0.0116768131 -0.5781232758 -0.7385022426
0.0027641080 0.9997059088 -0.0240926478 0.4566484789
0.5782345802 -0.0212543644 -0.8155936625 0.5508691692

camera_4
0.3259205190 0.0089380631 0.9453549208 0.5042611884
-0.0154632047 0.9998719393 -0.0041224123 0.0027628541
-0.9452707044 -0.0132746380 0.3260169924 -0.6592093227

camera_5
-0.8504673351 -0.0037030096 0.5260148283 0.7129544226
-0.0156972548 0.9997085443 -0.0183418255 0.4543444331
-0.5257935984 -0.0238561122 -0.8502775887 0.4671475996

camera_6
0.9999871102 -0.0005675861 -0.0050455170 -0.0357885765
0.0006039335 0.9999738592 0.0072052830 0.0025069919
0.0050412955 -0.0072082372 0.9999613126 -0.0305635123

camera_7
-0.9793493558 0.0397355452 -0.1982319998 -0.1814422937
-0.0796478660 0.8253740943 0.5589399090 0.3061714322
0.1858253393 0.5631861956 -0.8051647362 -2.0746009898

# INFO: Calibration took 72.6876826286 s.

Please let me know if you have faced a similar issue.

Thank you for your help.

Regards
PJ

How to work by using the results of Camodocal?

I'm very interested in your work.It's wonderful.
First, I ran it in camodocal and saved the calculation results, including 2D and 3D features and map information. The results are then used as input to infrascal for multi camera calibration. Can infrascal work properly in this situation? If it works, what other parameters need to be modified?

about the solvePnPRadialRansac funciton

in your code :

InfrasCal/src/infrastr_calib/InfrastructureCalibration.cc

Line 2455 in b36103b

for (size_t k = 0; k < corr2D3Ds.size(); ++k)

there have a clip:

    for (size_t k = 0; k < corr2D3Ds.size(); ++k)
        {
            const std::pair<Point2DFeaturePtr, Point3DFeaturePtr> corr2D3D = corr2D3Ds.at(k);
            Eigen::Vector3d P2 = corr2D3D.second->point();
            Eigen::Vector3d P1 = transformPoint(H_inv, P2);
            Eigen::Vector2d p1_pred(P1[0]/P1[2], P1[1]/P1[2]);
            double p1_pred_norm = p1_pred.norm();
            p1_pred(0) = p1_pred(0)/p1_pred_norm;
            p1_pred(1) = p1_pred(1)/p1_pred_norm;
            const Point2DFeatureConstPtr& f1 = corr2D3D.first;
            Eigen::Vector2d p1_pixel(f1->keypoint().pt.x - cxcy[0],
                                     f1->keypoint().pt.y - cxcy[1]);
            if(p1_pixel(0) * P1[0] + p1_pixel(1) * P1[1] < 0)
                continue;
            Eigen::Vector2d p1_pred_proj;
            p1_pred_proj(0) = p1_pred(0)*p1_pred(0)*p1_pixel(0)
                              + p1_pred(0)*p1_pred(1)*p1_pixel(1);
            p1_pred_proj(1) = p1_pred(1)*p1_pred(1)*p1_pixel(1)
                              + p1_pred(0)*p1_pred(1)*p1_pixel(0);
            double err = hypot(p1_pixel(0) - p1_pred_proj(0),
                               p1_pixel(1) - p1_pred_proj(1));
            if (!isnormal(err) || err > k_reprojErrorThresh)
            {
                continue;
            }

            inliersIds.push_back(k);
        }

I dont konw why what p1_pred(0) = p1_pred(0)/p1_pred_norm; and p1_pred_proj(0) = p1_pred(0)*p1_pred(0)*p1_pixel(0) + p1_pred(0)*p1_pred(1)*p1_pixel(1); doing? can you explain to me or share some paper link?
@youkely
@tsattler

Using Radial and Fisheye Camera Calibration

Hello @youkely

Is it possible to run the calibration with a set of camera that include both RADIAL and FISHEYE modes?

 ./build/bin/infrastr_calib --camera-count 5 \
 --output ./data/demo/results \
 --map ./data/demo/map \
 --database ./data/demo/map/database.db \
 --input ./data/demo/ \
 --vocab ./data/vocabulary/sift128.bin \
 -v --camera-model pinhole-radtan --save

The above command given in the dataset shows that it can run of a dataset if all the camera are either RADIAL or FISHEYE. Is it possible to run the command for a dataset that contains both radial and fisheye?

Looking forward to your reply.

Regards
PJ

Why do you need 3d coordinate values that match the SIFT feature points？ this is hard to get in practice.

this is a very good job , thanks to publish it ! I read this codes, It may be that there is a problem with my understanding.I think the only regret is that this method requires 3d coordinate values that match the SIFT features, which limits the practical application of this method. how to improve this ? I'm looking forward to the replay, thanks !

how to get the test dataset on the paper?

great work!
What are the test data in the paper? Is it open source? Can you share the link?
i would like to know how to record the dataset for surround view camera

How to get the map at the required format?

Hello,

I am interested in using your package, thanks for working on something that is really missing. My question is how to get the map at the required format, I have a point cloud made out of the frames from agisoft, how should I export/parse it in the proper way?

Average radial reprojection error over all frames: -nan px

i test the kitti dataset "2011_09_26_drive_0020_sync"
and get the 3d map by colmap.

when i use the map to get the extrinsic between the camera_0 and camera_1

my test data also come from the train data,

Docker image

Hello,
I would like to ask if there is a docker image dedicated for this repo? It seems impossible to make all of the packages work with each other as there are a lot of dependencies problems right now.
Regards

Reproducing map with demo data for InfrasCal

Hello @youkely

I am trying to reproduce your results using the data under demo directory.

I wanted to create a map using your data and COLMAP. However, COLMAP recognizes each image as an independent camera. So 5camera*4Images=20Images is treated as 20Cameras by COLMAP. Were there any particular setting you used for COLMAP?

Also, if possible, can you share the steps you used to produce the map using COLMAP?

Thank you for your help.

Regards
Jayam Patel

Result Visualization

Thanks for your great job!

Is there any visualization part inside this project? To check the extrinsic calibration process and the final result?

If no, is there any hint about how to implement a visualization?

what is "Rig Poses" mean？

In your paper there have "Rig Poses" and "Camera-Rig Extrinsics" ，I want to konw what is this mean?

[Table1.] Evaluation with a scale

Hi, cool work! And thank you for sharing the code!

Btw, I have a question about the published paper; How did you evaluate the extrinsics calibration performance with the "centimeter" scale?
Since this work is based on the COLMAP, if I’m not wrong, the model can only take a pose on non-scaled coordinates.

Best regards,

How to use the RobotCar datasets for evaluation as chapter 5.5 in the paper

Hi, thank you for great work!
I got the map from RobotCar Season dataset which is of NVM format, and I turned it to the ColMap(use the script from hloc). then I downloaded the 2014/12/16 datasets in the morning as input framesets. as the paper mentioned, I choose the framesets from No.500 to frame No.900. But I can't get a good result.

Here is the result:

$ ./build/bin/infrastr_calib --camera-count 3 --output 74747B25747AE96E/loc_dataset/robotcar/2014_12_16/2014-12-16-09-14-09 --map 74747B25747AE96E/loc_dataset/overcast/demo/all_map --database 74747B25747AE96E/loc_dataset/overcast/demo/all_map/overcast-reference.db --input 74747B25747AE96E/loc_dataset/robotcar/2014_12_16/2014-12-16-09-14-09 --vocab ./data/vocabulary/sift128.bin -v --camera-model pinhole-radtan --save

# INFO: Calibration mode InRaSU
# INFO: Using radial pose absolute estimator with unknown intrinsics.
# INFO: Initializing...
Get images from input directory
# INFO: Number of frames: 400
# INFO: Loading map... Finished.
# INFO: Setting up location recognition... Finished.
# INFO: Initialization finished!
# INFO: Wrote 0 frame sets to74747B25747AE96E/loc_dataset/robotcar/2014_12_16/2014-12-16-09-14-09/frames.sg
# INFO: Number of framesets as input:400
# INFO: Pose estimation took 265.522 s.
# INFO: Average radial reprojection error over all frames: nan px
# INFO: Average number of frames per set: nan
# ERROR: No complete frame sets were found.
# INFO: Calibration took 265.522 s.
terminate called after throwing an instance of 'std::out_of_range'
what(): vector::_M_range_check: __n (which is 281473647845376) >= this->size() (which is 187650136359000)
Aborted (core dumped)

Maybe I use the wrong vocab or framesets because I can run the demo well. But I can't figure it out. Could you offer more information for evaluation as chapter 5.5 in the paper?
Thank you!

Question about result file.

The result file is showed above.
My questions are:

what's the frames.sg and how can i visualize it?
Why the subfolder "frames" is empty?

Thanks!

Extrinsic Parameter Reference

Hello,

I have 6 Camera configuration and I am running your package to generate extrinsic configuration for them.

Are all the output extrinsic parameters in reference to cam0? or are they in reference to the previous camera? For example, came1 in reference to cam0, cam2 in reference to cam1 and so on.

I really appreciate your help.

Regards
PJ

Run calib without map folder

Hi,

Can i run calib without map folder?

./bin/infrastr_calib --camera-count 3 --output /media/ubuntu/data/1639127956941/results --input /media/ubuntu/data/1639127956941 -v --camera-model pinhole-radtan --save

[Build failed] OPENCV_XFEATURES2D_NONFREE NOT PRESENT

Hi,

Thanks for your great work! I'm trying to compile InfrasCal, however some errors occurs (see attached file "make_error.log").

I installed the OpenCV in the directory /data1/home/slam/Documents/sources/installed using -DCMAKE_INSTALL_PREFIX=/data1/home/slam/Documents/sources/installed.
Accordingly, I changed the LibraryConfig.cmake as follows:

The cmake outputs are attached to "cmake.log"
cmake.log.
It prints "OPENCV_XFEATURES2D_NONFREE NOT PRESENT". However, I DO have the 'opencv2/xfeatures2d/nonfree.hpp' header file in the directory:

make_error.log

Is there any solution? Many thanks!

Best wishes
Qin Shi

youkely / infrascal Goto Github PK

infrascal's People

Stargazers

Watchers

Forkers

infrascal's Issues

Recommend Projects

Recommend Topics

Recommend Org