tum-vision / tandem Goto Github PK

View Code? Open in Web Editor NEW

904.0 39.0 146.0 63.57 MB

[CoRL 21'] TANDEM: Tracking and Dense Mapping in Real-time using Deep Multi-view Stereo

Python 15.40% Shell 0.30% CMake 1.28% C++ 76.39% Cuda 6.52% C 0.10%

3d-reconstruction slam visual-odometry multiview-stereo

tandem's Introduction

TANDEM: Tracking and Dense Mapping
in Real-time using Deep Multi-view Stereo

Lukas Koestler^1* Nan Yang^1,2*,† Niclas Zeller^2,3 Daniel Cremers^1,2

^*equal contribution ^†corresponding author

¹Technical University of Munich ²Artisense
³Karlsruhe University of Applied Sciences

Conference on Robot Learning (CoRL) 2021, London, UK

3DV 2021 Best Demo Award

arXiv | Video | OpenReview | Project Page

Code and Data

📣 C++ code released before Christmas! Please check tandem/.
📣 CVA-MVSNet released! Please check cva_mvsnet/.
📣 Replica training data released! Please check replica/.
Minor improvements throughout January. Contributions are highly welcomed!
Release of the ScanNet-trained model
Docker image for TANDEM. Contributions are highly welcomed!

Abstract

In this paper, we present TANDEM a real-time monocular tracking and dense mapping framework. For pose estimation, TANDEM performs photometric bundle adjustment based on a sliding window of keyframes. To increase the robustness, we propose a novel tracking front-end that performs dense direct image alignment using depth maps rendered from a global model that is built incrementally from dense depth predictions. To predict the dense depth maps, we propose Cascade View-Aggregation MVSNet (CVA-MVSNet) that utilizes the entire active keyframe window by hierarchically constructing 3D cost volumes with adaptive view aggregation to balance the different stereo baselines between the keyframes. Finally, the predicted depth maps are fused into a consistent global map represented as a truncated signed distance function (TSDF) voxel grid. Our experimental results show that TANDEM outperforms other state-of-the-art traditional and learning-based monocular visual odometry (VO) methods in terms of camera tracking. Moreover, TANDEM shows state-of-the-art real-time 3D reconstruction performance.

Poster

tandem's People

Contributors

Stargazers

Watchers

Forkers

dronetdsp linshu1994 be997398715 fcntes simontu yonghuicai 911wb liuqinglong110 zhongyinbiao liudongdong1 zhangsongdmk mfkiwl cuijianzhu arenas7307979 xgenietony shenckens icameling liuguoyou peterzhousz jingwenwang95 quantum-entropy hiyyg yangzhengshi minxuanjun yuancaimaiyi h312h edwinstudy asyr625 yuangao2019 key-word jsyangcode lqql2012 wbl1997 pandinosaurus super-iron-man yuhuangying bingyingl metavai liuxin-sh lutieyuan dlxxj chengwei920412 aipanpan nrupatunga arctanbell rainleong tony2cmu bigjohnn behindyou0-0 fengzhang123 93yh davidxupiaopiao qiqzhang li-creator15 pankhurivanjani zmf2022 xanxuscrypto grf-sunomikp31 rickkcir ecstasygu kongan zhouqun92 echosystemmark harveyliufly swpsgithub fightmoney francescoli teaxus aiprogrammer youken-tang yohannes-taye huaifeng1993 simmeryson luyu94 doomzzju jackliuday bccwai lawlawcodes xinfushe louzq16 altria1122 likojack infinite-dreamland swarmt mysterycver billy-liu-12 graysoncv tarmy3124 zichaotong mrjohnrobert baituhuangyu laplacekorea thzhfei lt-xiang slamplus rancheng bruinxiong jaychen1996 prograguo guo253

tandem's Issues

error related to pangolin when run make -j

Does any one know how to solve it ? Is there something I have not install?

My pangolin version is 0.5.

Saving sparse reconstruction

Hi. I wanted to ask how we can save the Sparse reconstruction instead of normal one. I nthe GUI we have option show sparse but I am not sure how to save that sparse map.

RUNNING --- results/tracking/dense/euroc/V1_01_easy/0 [W BinaryOps.cpp:467] Warning: floor_divide is deprecated, and will be removed in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor'). (function operator())

Dear @imoinuddin:
I am writing for your help,When I enter and execute"bash scripts/tracking_euroc.bash",it display an warning ,and pass half hour,it exited,the warning is :
[W BinaryOps.cpp:467] Warning: floor_divide is deprecated, and will be removed in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values.
To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor'). (function operator())
In the result folder,I can find the those files such as evaluate_ate.png ,mesh.obj,poses_dso.txt,keyframes_dso.txt out.txt ,result.txt,please help me!!!!!!!

make error about Sophus

When I fellow the readme in the Sophus folder to make Sophus, something that looks likes make fail appeared.

Does anyone know how to solve it ?
Thanks!

Publish date?

Hi, Thank you for the awesome work. I'm curious as to when would you be open-sourcing source code of underline method? Thanks!

cnpy error when i install

when i run make -j i got :

what should i do to fix it?


/home/tandem/tandem-master/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker.cpp: In member function 'Vec6 CudaCoarseTracker::calcRes(const Eigen::Matrix&, float, const Vector2d&, float)':
/home/tandem/tandem-master/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker.cpp:218:64: error: no matching function for call to 'npy_save(const char [25], float [6], )'
218 | cnpy::npy_save("cct_data/nfxfyaffLLb.npy", nfxfyaffLLb, {6});
| ^
In file included from /home/tandem/tandem-master/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker.cpp:9:
/usr/local/include/cnpy.h:64:31: note: candidate: 'template void cnpy::npy_save(std::string, const T*, const unsigned int*, unsigned int, std::string, std::string)'
64 | template void npy_save(std::string fname, const T* data, const unsigned int* shape, const unsigned int ndims, std::string mode = "w", std::string order = "C") {
| ^~~~~~~~
/usr/local/include/cnpy.h:64:31: note: template argument deduction/substitution failed:
/home/tandem/tandem-master/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker.cpp:218:64: note: candidate expects 6 arguments, 3 provided
218 | cnpy::npy_save("cct_data/nfxfyaffLLb.npy", nfxfyaffLLb, {6});
| ^
/home/tandem/tandem-master/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker.cpp:221:87: error: no matching function for call to 'npy_save(const char [48], float [6], )'
221 | cnpy::npy_save("cct_data/setting_huberTH_maxEnergy_cutoffTH.npy", nfxfyaffLLb, {3});
| ^
In file included from /home/tandem/tandem-master/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker.cpp:9:
/usr/local/include/cnpy.h:64:31: note: candidate: 'template void cnpy::npy_save(std::string, const T*, const unsigned int*, unsigned int, std::string, std::string)'
64 | template void npy_save(std::string fname, const T* data, const unsigned int* shape, const unsigned int ndims, std::string mode = "w", std::string order = "C") {
| ^~~~~~~~
/usr/local/include/cnpy.h:64:31: note: template argument deduction/substitution failed:
/home/tandem/tandem-master/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker.cpp:221:87: note: candidate expects 6 arguments, 3 provided
221 | cnpy::npy_save("cct_data/setting_huberTH_maxEnergy_cutoffTH.npy", nfxfyaffLLb, {3});
| ^
/home/tandem/tandem-master/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker.cpp:223:79: error: no matching function for call to 'npy_save(const char [30], float*&, )'
223 | cnpy::npy_save("cct_data/refToNew_Ki_host.npy", refToNew_Ki_host, {16 + 9});
| ^
In file included from /home/tandem/tandem-master/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker.cpp:9:
/usr/local/include/cnpy.h:64:31: note: candidate: 'template void cnpy::npy_save(std::string, const T*, const unsigned int*, unsigned int, std::string, std::string)'
64 | template void npy_save(std::string fname, const T* data, const unsigned int* shape, const unsigned int ndims, std::string mode = "w", std::string order = "C") {
| ^~~~~~~~
/usr/local/include/cnpy.h:64:31: note: template argument deduction/substitution failed:
/home/tandem/tandem-master/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker.cpp:223:79: note: candidate expects 6 arguments, 3 provided
223 | cnpy::npy_save("cct_data/refToNew_Ki_host.npy", refToNew_Ki_host, {16 + 9});
| ^
/home/tandem/tandem-master/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker.cpp:227:65: error: no matching function for call to 'npy_save(const char [18], float*&, )'
227 | cnpy::npy_save("cct_data/pc_u.npy", buf, {(unsigned long) n});
| ^
In file included from /home/tandem/tandem-master/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker.cpp:9:
/usr/local/include/cnpy.h:64:31: note: candidate: 'template void cnpy::npy_save(std::string, const T*, const unsigned int*, unsigned int, std::string, std::string)'
64 | template void npy_save(std::string fname, const T* data, const unsigned int* shape, const unsigned int ndims, std::string mode = "w", std::string order = "C") {
| ^~~~~~~~
/usr/local/include/cnpy.h:64:31: note: template argument deduction/substitution failed:
/home/tandem/tandem-master/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker.cpp:227:65: note: candidate expects 6 arguments, 3 provided
227 | cnpy::npy_save("cct_data/pc_u.npy", buf, {(unsigned long) n});
| ^
/home/tandem/tandem-master/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker.cpp:230:65: error: no matching function for call to 'npy_save(const char [18], float*&, )'
230 | cnpy::npy_save("cct_data/pc_v.npy", buf, {(unsigned long) n});
| ^
In file included from /home/tandem/tandem-master/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker.cpp:9:
/usr/local/include/cnpy.h:64:31: note: candidate: 'template void cnpy::npy_save(std::string, const T*, const unsigned int*, unsigned int, std::string, std::string)'
64 | template void npy_save(std::string fname, const T* data, const unsigned int* shape, const unsigned int ndims, std::string mode = "w", std::string order = "C") {
| ^~~~~~~~
/usr/local/include/cnpy.h:64:31: note: template argument deduction/substitution failed:
/home/tandem/tandem-master/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker.cpp:230:65: note: candidate expects 6 arguments, 3 provided
230 | cnpy::npy_save("cct_data/pc_v.npy", buf, {(unsigned long) n});
| ^
/home/tandem/tandem-master/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker.cpp:233:70: error: no matching function for call to 'npy_save(const char [23], float*&, )'
233 | cnpy::npy_save("cct_data/pc_idepth.npy", buf, {(unsigned long) n});
| ^
In file included from /home/tandem/tandem-master/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker.cpp:9:
/usr/local/include/cnpy.h:64:31: note: candidate: 'template void cnpy::npy_save(std::string, const T*, const unsigned int*, unsigned int, std::string, std::string)'
64 | template void npy_save(std::string fname, const T* data, const unsigned int* shape, const unsigned int ndims, std::string mode = "w", std::string order = "C") {
| ^~~~~~~~
/usr/local/include/cnpy.h:64:31: note: template argument deduction/substitution failed:
/home/tandem/tandem-master/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker.cpp:233:70: note: candidate expects 6 arguments, 3 provided
233 | cnpy::npy_save("cct_data/pc_idepth.npy", buf, {(unsigned long) n});
| ^
/home/tandem/tandem-master/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker.cpp:236:69: error: no matching function for call to 'npy_save(const char [22], float*&, )'
236 | cnpy::npy_save("cct_data/pc_color.npy", buf, {(unsigned long) n});
| ^
In file included from /home/tandem/tandem-master/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker.cpp:9:
/usr/local/include/cnpy.h:64:31: note: candidate: 'template void cnpy::npy_save(std::string, const T*, const unsigned int*, unsigned int, std::string, std::string)'
64 | template void npy_save(std::string fname, const T* data, const unsigned int* shape, const unsigned int ndims, std::string mode = "w", std::string order = "C") {
| ^~~~~~~~
/usr/local/include/cnpy.h:64:31: note: template argument deduction/substitution failed:
/home/tandem/tandem-master/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker.cpp:236:69: note: candidate expects 6 arguments, 3 provided
236 | cnpy::npy_save("cct_data/pc_color.npy", buf, {(unsigned long) n});
| ^
/home/tandem/tandem-master/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker.cpp:241:74: error: no matching function for call to 'npy_save(const char [19], float*&, )'
241 | cnpy::npy_save("cct_data/dInew.npy", buf, {(unsigned long) 3 * w * h});
| ^
In file included from /home/tandem/tandem-master/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker.cpp:9:
/usr/local/include/cnpy.h:64:31: note: candidate: 'template void cnpy::npy_save(std::string, const T*, const unsigned int*, unsigned int, std::string, std::string)'
64 | template void npy_save(std::string fname, const T* data, const unsigned int* shape, const unsigned int ndims, std::string mode = "w", std::string order = "C") {
| ^~~~~~~~
/usr/local/include/cnpy.h:64:31: note: template argument deduction/substitution failed:
/home/tandem/tandem-master/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker.cpp:241:74: note: candidate expects 6 arguments, 3 provided
241 | cnpy::npy_save("cct_data/dInew.npy", buf, {(unsigned long) 3 * w * h});
| ^
/home/tandem/tandem-master/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker.cpp: In member function 'void CudaCoarseTracker::calcG(Eigen::Matrix&, Eigen::Matrix&, float, const Vector2d&)':
/home/tandem/tandem-master/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker.cpp:290:64: error: no matching function for call to 'npy_save(const char [25], float [6], )'
290 | cnpy::npy_save("cct_data/nfxfyaffLLb.npy", nfxfyaffLLb, {6});
| ^
In file included from /home/tandem/tandem-master/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker.cpp:9:
/usr/local/include/cnpy.h:64:31: note: candidate: 'template void cnpy::npy_save(std::string, const T*, const unsigned int*, unsigned int, std::string, std::string)'
64 | template void npy_save(std::string fname, const T* data, const unsigned int* shape, const unsigned int ndims, std::string mode = "w", std::string order = "C") {
| ^~~~~~~~
/usr/local/include/cnpy.h:64:31: note: template argument deduction/substitution failed:
/home/tandem/tandem-master/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker.cpp:290:64: note: candidate expects 6 arguments, 3 provided
290 | cnpy::npy_save("cct_data/nfxfyaffLLb.npy", nfxfyaffLLb, {6});
| ^
/home/tandem/tandem-master/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker.cpp:294:69: error: no matching function for call to 'npy_save(const char [22], float*&, )'
294 | cnpy::npy_save("cct_data/pc_color.npy", buf, {(unsigned long) n});
| ^
In file included from /home/tandem/tandem-master/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker.cpp:9:
/usr/local/include/cnpy.h:64:31: note: candidate: 'template void cnpy::npy_save(std::string, const T*, const unsigned int*, unsigned int, std::string, std::string)'
64 | template void npy_save(std::string fname, const T* data, const unsigned int* shape, const unsigned int ndims, std::string mode = "w", std::string order = "C") {
| ^~~~~~~~
/usr/local/include/cnpy.h:64:31: note: template argument deduction/substitution failed:
/home/tandem/tandem-master/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker.cpp:294:69: note: candidate expects 6 arguments, 3 provided
294 | cnpy::npy_save("cct_data/pc_color.npy", buf, {(unsigned long) n});
| ^
/home/tandem/tandem-master/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker.cpp:297:69: error: no matching function for call to 'npy_save(const char [22], float*&, )'
297 | cnpy::npy_save("cct_data/warped_u.npy", buf, {(unsigned long) n});
| ^
In file included from /home/tandem/tandem-master/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker.cpp:9:
/usr/local/include/cnpy.h:64:31: note: candidate: 'template void cnpy::npy_save(std::string, const T*, const unsigned int*, unsigned int, std::string, std::string)'
64 | template void npy_save(std::string fname, const T* data, const unsigned int* shape, const unsigned int ndims, std::string mode = "w", std::string order = "C") {
| ^~~~~~~~
/usr/local/include/cnpy.h:64:31: note: template argument deduction/substitution failed:
/home/tandem/tandem-master/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker.cpp:297:69: note: candidate expects 6 arguments, 3 provided
297 | cnpy::npy_save("cct_data/warped_u.npy", buf, {(unsigned long) n});
| ^
/home/tandem/tandem-master/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker.cpp:300:69: error: no matching function for call to 'npy_save(const char [22], float*&, )'
300 | cnpy::npy_save("cct_data/warped_v.npy", buf, {(unsigned long) n});
| ^
In file included from /home/tandem/tandem-master/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker.cpp:9:
/usr/local/include/cnpy.h:64:31: note: candidate: 'template void cnpy::npy_save(std::string, const T*, const unsigned int*, unsigned int, std::string, std::string)'
64 | template void npy_save(std::string fname, const T* data, const unsigned int* shape, const unsigned int ndims, std::string mode = "w", std::string order = "C") {
| ^~~~~~~~
/usr/local/include/cnpy.h:64:31: note: template argument deduction/substitution failed:
/home/tandem/tandem-master/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker.cpp:300:69: note: candidate expects 6 arguments, 3 provided
300 | cnpy::npy_save("cct_data/warped_v.npy", buf, {(unsigned long) n});
| ^
/home/tandem/tandem-master/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker.cpp:303:70: error: no matching function for call to 'npy_save(const char [23], float*&, )'
303 | cnpy::npy_save("cct_data/warped_dx.npy", buf, {(unsigned long) n});
| ^
In file included from /home/tandem/tandem-master/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker.cpp:9:
/usr/local/include/cnpy.h:64:31: note: candidate: 'template void cnpy::npy_save(std::string, const T*, const unsigned int*, unsigned int, std::string, std::string)'
64 | template void npy_save(std::string fname, const T* data, const unsigned int* shape, const unsigned int ndims, std::string mode = "w", std::string order = "C") {
| ^~~~~~~~
/usr/local/include/cnpy.h:64:31: note: template argument deduction/substitution failed:
/home/tandem/tandem-master/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker.cpp:303:70: note: candidate expects 6 arguments, 3 provided
303 | cnpy::npy_save("cct_data/warped_dx.npy", buf, {(unsigned long) n});
| ^
/home/tandem/tandem-master/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker.cpp:306:70: error: no matching function for call to 'npy_save(const char [23], float*&, )'
306 | cnpy::npy_save("cct_data/warped_dy.npy", buf, {(unsigned long) n});
| ^
In file included from /home/tandem/tandem-master/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker.cpp:9:
/usr/local/include/cnpy.h:64:31: note: candidate: 'template void cnpy::npy_save(std::string, const T*, const unsigned int*, unsigned int, std::string, std::string)'
64 | template void npy_save(std::string fname, const T* data, const unsigned int* shape, const unsigned int ndims, std::string mode = "w", std::string order = "C") {
| ^~~~~~~~
/usr/local/include/cnpy.h:64:31: note: template argument deduction/substitution failed:
/home/tandem/tandem-master/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker.cpp:306:70: note: candidate expects 6 arguments, 3 provided
306 | cnpy::npy_save("cct_data/warped_dy.npy", buf, {(unsigned long) n});
| ^
/home/tandem/tandem-master/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker.cpp:309:74: error: no matching function for call to 'npy_save(const char [27], float*&, )'
309 | cnpy::npy_save("cct_data/warped_idepth.npy", buf, {(unsigned long) n});
| ^
In file included from /home/tandem/tandem-master/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker.cpp:9:
/usr/local/include/cnpy.h:64:31: note: candidate: 'template void cnpy::npy_save(std::string, const T*, const unsigned int*, unsigned int, std::string, std::string)'
64 | template void npy_save(std::string fname, const T* data, const unsigned int* shape, const unsigned int ndims, std::string mode = "w", std::string order = "C") {
| ^~~~~~~~
/usr/local/include/cnpy.h:64:31: note: template argument deduction/substitution failed:
/home/tandem/tandem-master/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker.cpp:309:74: note: candidate expects 6 arguments, 3 provided
309 | cnpy::npy_save("cct_data/warped_idepth.npy", buf, {(unsigned long) n});
| ^
/home/tandem/tandem-master/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker.cpp:312:76: error: no matching function for call to 'npy_save(const char [29], float*&, )'
312 | cnpy::npy_save("cct_data/warped_residual.npy", buf, {(unsigned long) n});
| ^
In file included from /home/tandem/tandem-master/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker.cpp:9:
/usr/local/include/cnpy.h:64:31: note: candidate: 'template void cnpy::npy_save(std::string, const T*, const unsigned int*, unsigned int, std::string, std::string)'
64 | template void npy_save(std::string fname, const T* data, const unsigned int* shape, const unsigned int ndims, std::string mode = "w", std::string order = "C") {
| ^~~~~~~~
/usr/local/include/cnpy.h:64:31: note: template argument deduction/substitution failed:
/home/tandem/tandem-master/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker.cpp:312:76: note: candidate expects 6 arguments, 3 provided
312 | cnpy::npy_save("cct_data/warped_residual.npy", buf, {(unsigned long) n});
| ^
/home/tandem/tandem-master/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker.cpp:315:74: error: no matching function for call to 'npy_save(const char [27], float*&, )'
315 | cnpy::npy_save("cct_data/warped_weight.npy", buf, {(unsigned long) n});
| ^
In file included from /home/tandem/tandem-master/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker.cpp:9:
/usr/local/include/cnpy.h:64:31: note: candidate: 'template void cnpy::npy_save(std::string, const T*, const unsigned int*, unsigned int, std::string, std::string)'
64 | template void npy_save(std::string fname, const T* data, const unsigned int* shape, const unsigned int ndims, std::string mode = "w", std::string order = "C") {
| ^~~~~~~~
/usr/local/include/cnpy.h:64:31: note: template argument deduction/substitution failed:
/home/tandem/tandem-master/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker.cpp:315:74: note: candidate expects 6 arguments, 3 provided
315 | cnpy::npy_save("cct_data/warped_weight.npy", buf, {(unsigned long) n});
| ^
make[2]: *** [libdr/cuda_coarse_tracker/CMakeFiles/cuda_coarse_tracker.dir/build.make:76: libdr/cuda_coarse_tracker/CMakeFiles/cuda_coarse_tracker.dir/src/cuda_coarse_tracker.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [CMakeFiles/Makefile2:275: libdr/cuda_coarse_tracker/CMakeFiles/cuda_coarse_tracker.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....

Program crashes during initialization

It's great, thanks for your code.

I try to run the 'tandem_dem' with my realsense435i, but it can not initialize. I move the camera very slow and only have translation almost, I used the gdb to find out it crash, it is crash in refusion::tsdfvh::TsdfVolume::IntegrateScanAsync(refusion::RgbdSensor const&, unsigned char*, float*, refusion::float4x4 const&), here is the output of GDB:

#0  0x00005555556cb837 in refusion::tsdfvh::TsdfVolume::IntegrateScanAsync(refusion::RgbdSensor const&, unsigned char*, float*, refusion::float4x4 const&) ()
#1  0x00005555556c702f in DrFusion::IntegrateScanAsync(unsigned char*, float*, float const*) ()
#2  0x000055555564ed73 in TandemBackendImpl::CallSequential() (this=this@entry=0x5555972c7180) at /home/hjx/tandem/tandem/src/tandem/tandem_backend.cpp:166
#3  0x000055555564f51e in TandemBackendImpl::Loop() (this=0x5555972c7180) at /home/hjx/tandem/tandem/src/tandem/tandem_backend.cpp:129
#4  0x00007ffff79babcd in  () at /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.65.1
#5  0x00007fffec4546db in start_thread (arg=0x7ffdfbfff700) at pthread_create.c:463
#6  0x00007fffd9aec71f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Linking error to pangolin

Hi,

Thanks for making this great work open source.

My compilation env: Ubuntu18.04 + cmake 3.22.1 + cuda 11.3 + Pangolin (0.6)

The compilation was done, but when linking to pangolin, the following error was reported:
[100%] Linking CXX executable bin/tandem_dataset CMakeFiles/tandem_dataset.dir/src/main_tandem_pangolin.cpp.o: In functionmain::{lambda()#1}::operator()() const':
/home/sky/project/tandem_workspace/tandem/tandem/src/main_tandem_pangolin.cpp:299: undefined reference to DrFusion::SaveMeshToFile(std::string const&, float*, float*)' lib/libtandem.a(PangolinDSOViewer.cpp.o): In function dso::IOWrap::PangolinDSOViewer::run()':
/home/sky/project/tandem_workspace/tandem/tandem/src/IOWrapper/Pangolin/PangolinDSOViewer.cpp:111: undefined reference to pangolin::CreateWindowAndBind(std::string, int, int, pangolin::Params const&)' /home/sky/project/tandem_workspace/tandem/tandem/src/IOWrapper/Pangolin/PangolinDSOViewer.cpp:132: undefined reference to pangolin::Display(std::string const&)'
/home/sky/project/tandem_workspace/tandem/tandem/src/IOWrapper/Pangolin/PangolinDSOViewer.cpp:140: undefined reference to pangolin::Display(std::string const&)' /home/sky/project/tandem_workspace/tandem/tandem/src/IOWrapper/Pangolin/PangolinDSOViewer.cpp:143: undefined reference to pangolin::Display(std::string const&)'
/home/sky/project/tandem_workspace/tandem/tandem/src/IOWrapper/Pangolin/PangolinDSOViewer.cpp:185: undefined reference to pangolin::CreatePanel(std::string const&)' lib/libtandem.a(PangolinDSOViewer.cpp.o): In function void pangolin::InitialiseNewVarMeta(pangolin::VarValue&, std::string const&, double, double, int, bool)':
/usr/local/include/pangolin/var/var.h:66: undefined reference to pangolin::Split(std::string const&, char)' lib/libtandem.a(PangolinDSOViewer.cpp.o): In function void pangolin::InitialiseNewVarMeta(pangolin::VarValue&, std::string const&, double, double, int, bool)':
/usr/local/include/pangolin/var/var.h:66: undefined reference to pangolin::Split(std::string const&, char)' lib/libtandem.a(PangolinDSOViewer.cpp.o): In function void pangolin::InitialiseNewVarMeta(pangolin::VarValue&, std::string const&, double, double, int, bool)':
/usr/local/include/pangolin/var/var.h:66: undefined reference to pangolin::Split(std::string const&, char)' collect2: error: ld returned 1 exit status CMakeFiles/tandem_dataset.dir/build.make:176: recipe for target 'bin/tandem_dataset' failed make[2]: *** [bin/tandem_dataset] Error 1 CMakeFiles/Makefile2:170: recipe for target 'CMakeFiles/tandem_dataset.dir/all' failed make[1]: *** [CMakeFiles/tandem_dataset.dir/all] Error 2 Makefile:90: recipe for target 'all' failed make: *** [all] Error 2

I can compile dso successfully, so I compare the CMakeLists.txt between dso and tandem, but nothing was found. And I tried different version of pangolin, the above error reported.

I have no idea about this error, any comments will be appreciated.

About the rendered depth of the last keyframe.

Very appreciate your great work! I have some confusion about dense tracking with the rendered depth of the last keyframe.

During the dense tracking, a new frame is tracked against the last keyframe n with the rendered depth of n.

However, when the last keyframe is tracked, you already predicted a dense depth map of the last keyframe with CVA-MVSNet. And the predicted depth of the last keyframe is used for constructing the TSDF volume. Therefore, is there so much difference between the predicted depth or the rendered depth? I'm curious about how much the difference is.

Error with c++11 compiler

Hello!

I am trying to run the code, while building tandem I am getting 2 major errors from cuda_coarse_tracker side. I have cuda 11.1, and same cudNN and libtorch version as mentioned in readme.

(tandem-exp) vanjani@port-2118:~/tandem/tandem/build$ make -jConsolidate compiler generated dependencies of target dr-mvsnet
[  1%] Building CXX object libdr/cuda_coarse_tracker/CMakeFiles/cuda_coarse_tracker.dir/src/cuda_coarse_tracker.cpp.o
[  3%] Building CUDA object libdr/cuda_coarse_tracker/CMakeFiles/cuda_coarse_tracker.dir/src/cuda_coarse_tracker_private.cu.o
Consolidate compiler generated dependencies of target dr-fusion
[  6%] Built target dr-mvsnet
[ 22%] Built target dr-fusion
Consolidate compiler generated dependencies of target dr_mvsnet_test
[ 25%] Built target dr_mvsnet_test
/home/vanjani/tandem/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker.cpp:9:18: fatal error: cnpy.h: No such file or directory
compilation terminated.
libdr/cuda_coarse_tracker/CMakeFiles/cuda_coarse_tracker.dir/build.make:75: recipe for target 'libdr/cuda_coarse_tracker/CMakeFiles/cuda_coarse_tracker.dir/src/cuda_coarse_tracker.cpp.o' failed
make[2]: *** [libdr/cuda_coarse_tracker/CMakeFiles/cuda_coarse_tracker.dir/src/cuda_coarse_tracker.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
In file included from /usr/include/c++/5/type_traits:35:0,
                 from /home/vanjani/tandem/tandem/libdr/cuda_coarse_tracker/include/private/utils.h:11,
                 from /home/vanjani/tandem/tandem/libdr/cuda_coarse_tracker/include/private/numeric_cuda.h:16,
                 from /home/vanjani/tandem/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker_private.cu:4:
/usr/include/c++/5/bits/c++0x_warning.h:32:2: error: #error This file requires compiler and library support for the ISO C++ 2011 standard. This support must be enabled with the -std=c++11 or -std=gnu++11 compiler options.
 #error This file requires compiler and library support \
  ^
libdr/cuda_coarse_tracker/CMakeFiles/cuda_coarse_tracker.dir/build.make:89: recipe for target 'libdr/cuda_coarse_tracker/CMakeFiles/cuda_coarse_tracker.dir/src/cuda_coarse_tracker_private.cu.o' failed
make[2]: *** [libdr/cuda_coarse_tracker/CMakeFiles/cuda_coarse_tracker.dir/src/cuda_coarse_tracker_private.cu.o] Error 1
CMakeFiles/Makefile2:274: recipe for target 'libdr/cuda_coarse_tracker/CMakeFiles/cuda_coarse_tracker.dir/all' failed
make[1]: *** [libdr/cuda_coarse_tracker/CMakeFiles/cuda_coarse_tracker.dir/all] Error 2
Makefile:90: recipe for target 'all' failed
make: *** [all] Error 2
(tandem-exp) vanjani@port-2118:~/tandem/tandem/build$

Could you suggest anything to solve these?

I am using SET(CMAKE_CXX_FLAGS "-std=c++0x") in CMakeLists.txt

Thanks,

Update1: I fixed the cnpy error by cloning the cnpy repo, cmake, make and make install for future reference

C++11 Compiler error still remains

(tandem-exp) vanjani@port-2118:~/tandem/tandem/build$ cmake ..     -DCMAKE_BUILD_TYPE=Release     -DCMAKE_PREFIX_PATH=$TANDEM_LIBTORCH_DIR     -DCUDNN_LIBRARY=$TANDEM_CUDNN_LIBRARY     -DCUDNN_INCLUDE_PATH=$TANDEM_CUDNN_INCLUDE_PATH
-- Caffe2: CUDA detected: 11.1
-- Caffe2: CUDA nvcc is: /usr/local/cuda-11.1/bin/nvcc
-- Caffe2: CUDA toolkit directory: /usr/local/cuda-11.1
-- Caffe2: Header version is: 11.1
-- Found cuDNN: v8.0.5  (include: /usr/local/cuda/include, library: /usr/local/cuda-11.1/lib64/libcudnn.so)
-- /usr/local/cuda-11.1/lib64/libnvrtc.so shorthash is 3a20f2b6
-- Autodetected CUDA architecture(s):  6.1
-- Added CUDA NVCC flags for: -gencode;arch=compute_61,code=sm_61
-- Found Eigen headers in: /usr/local/include/eigen3
-- Release mode (!= not Debug). Adding -O3;-lineinfo to NVCC FLAGS
--- found PANGOLIN, compiling tandem_pangolin library.
--- found OpenCV (/usr/local/include;/usr/local/include/opencv), compiling tandem_opencv library.
--- compiling tandem_dataset.
--- not building tandem_demo, since either don't have openCV, Pangolin, or Realsense.
--- not building realsense_calib_recorder, since either don't have openCV, Pangolin, or Realsense.
-- Configuring done
-- Generating done
-- Build files have been written to: /home/vanjani/tandem/tandem/build
(tandem-exp) vanjani@port-2118:~/tandem/tandem/build$ make -jConsolidate compiler generated dependencies of target dr-mvsnet
Consolidate compiler generated dependencies of target dr-fusion
[  1%] Building CXX object libdr/cuda_coarse_tracker/CMakeFiles/cuda_coarse_tracker.dir/src/cuda_coarse_tracker.cpp.o
[  3%] Building CUDA object libdr/cuda_coarse_tracker/CMakeFiles/cuda_coarse_tracker.dir/src/cuda_coarse_tracker_private.cu.o
[ 18%] Built target dr-fusion
[ 22%] Built target dr-mvsnet
Consolidate compiler generated dependencies of target dr_mvsnet_test
[ 25%] Built target dr_mvsnet_test
In file included from /usr/include/c++/5/type_traits:35:0,
                 from /home/vanjani/tandem/tandem/libdr/cuda_coarse_tracker/include/private/utils.h:11,
                 from /home/vanjani/tandem/tandem/libdr/cuda_coarse_tracker/include/private/numeric_cuda.h:16,
                 from /home/vanjani/tandem/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker_private.cu:4:
/usr/include/c++/5/bits/c++0x_warning.h:32:2: error: #error This file requires compiler and library support for the ISO C++ 2011 standard. This support must be enabled with the -std=c++11 or -std=gnu++11 compiler options.
 #error This file requires compiler and library support \
  ^
libdr/cuda_coarse_tracker/CMakeFiles/cuda_coarse_tracker.dir/build.make:89: recipe for target 'libdr/cuda_coarse_tracker/CMakeFiles/cuda_coarse_tracker.dir/src/cuda_coarse_tracker_private.cu.o' failed
make[2]: *** [libdr/cuda_coarse_tracker/CMakeFiles/cuda_coarse_tracker.dir/src/cuda_coarse_tracker_private.cu.o] Error 1
make[2]: *** Waiting for unfinished jobs....
CMakeFiles/Makefile2:274: recipe for target 'libdr/cuda_coarse_tracker/CMakeFiles/cuda_coarse_tracker.dir/all' failed
make[1]: *** [libdr/cuda_coarse_tracker/CMakeFiles/cuda_coarse_tracker.dir/all] Error 2
Makefile:90: recipe for target 'all' failed
make: *** [all] Error 2
(tandem-exp) vanjani@port-2118:~/tandem/tandem/build$

This error was coming in dr_mvsnet too but using SET(CMAKE_CXX_FLAGS "-std=c++0x") in Cmakelists.txt solved the issue but somehow it is not helping for cuda_coarse_tracler

Can be used for measuring distance,if can, how long range can detect ?

Very cool project, thanks for sharing.
And I wander this project can be used for measuring distance,if can,I want to use for detecting object and measuring distance, how long range can detect ?
And any general monocular camera can be usable for your project ?
Thanks very much.

RTX 3090

Thank you for sharing this great paper and excellent code.
RTX 3090 only supports cuda11 and above. Cuda version affects pytorch and other package versions. Therefore, this project can’t run on 30 series graphics cards, what a pity! TUT
When I run this project on 3090,there is a error: 'CUDA error: no kernel image is available for execution on the device'

Install Sophus from thirdparty folder,but compile error

hello,I installed the sophus in thirdparty folder ,but I met a problem like this picture when I use make -j10 to build the project.Do you know how to solve it ?

got error when i make the project,the error log as follows

Is it a version issue with g++/gcc?I've commented out the path to conda's environment variables.

About TSDF Fusion

Regarding tsdf map fusion, there is no detailed explanation in the paper. I would like to ask a few questions, which I would be grateful for.

When building a map in real time, how is the size of TSDF map initialized?
Traditional TSDF algorithm can't deal with the occluded surface. Is there any special treatment for TSDF fusion in Tandem? What's the difference between it and conventional TSDF fusion?
Thank you very much for your guidance！

TypeError when train CVA-MVSNet

Thank you for sharing this great paper and excellent code.
I followed the README file in CVA-MVSNet. When I tried to train the model, there was a TypeError. Could you please help me? Thank you for your time.

Traceback (most recent call last):
File "train.py", line 127, in
main(*parse_args(parser))
File "train.py", line 123, in main
trainer.fit(model)
File "/media/dai/NEWSMY/anaconda3/envs/sfm/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 859, in fit
self.single_gpu_train(model)
File "/media/dai/NEWSMY/anaconda3/envs/sfm/lib/python3.8/site-packages/pytorch_lightning/trainer/distrib_parts.py", line 503, in single_gpu_train
self.run_pretrain_routine(model)
File "/media/dai/NEWSMY/anaconda3/envs/sfm/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1015, in run_pretrain_routine
self.train()
File "/media/dai/NEWSMY/anaconda3/envs/sfm/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 347, in train
self.run_training_epoch()
File "/media/dai/NEWSMY/anaconda3/envs/sfm/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 405, in run_training_epoch
for batch_idx, (batch, is_last_batch) in self.profiler.profile_iterable(
File "/media/dai/NEWSMY/anaconda3/envs/sfm/lib/python3.8/site-packages/pytorch_lightning/profiler/profilers.py", line 64, in profile_iterable
value = next(iterator)
File "/media/dai/NEWSMY/anaconda3/envs/sfm/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 801, in _with_is_last
last = next(it)
File "/media/dai/NEWSMY/anaconda3/envs/sfm/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 521, in next
data = self._next_data()
File "/media/dai/NEWSMY/anaconda3/envs/sfm/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 561, in _next_data
data = self._dataset_fetcher.fetch(index) # may raise StopIteration
File "/media/dai/NEWSMY/anaconda3/envs/sfm/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/media/dai/NEWSMY/anaconda3/envs/sfm/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/media/dai/NEWSMY/depth_estimate/tandem-master/cva_mvsnet/models/datasets.py", line 583, in getitem
item = self.dataset.getitem(idx)
File "/media/dai/NEWSMY/depth_estimate/tandem-master/cva_mvsnet/models/datasets.py", line 562, in getitem
data = self.transform(data)
File "/media/dai/NEWSMY/depth_estimate/tandem-master/cva_mvsnet/models/datasets.py", line 97, in preprocess
img_aug = color_trans(img_ori)
File "/media/dai/NEWSMY/anaconda3/envs/sfm/lib/python3.8/site-packages/torchvision/transforms/transforms.py", line 61, in call
img = t(img)
TypeError: 'tuple' object is not callable

Depth map

Can you tell me how to get the depth map from RGB pictures?@lkskstlr

No such file or directory

When I make the project, I get

fatal error: cub/block/block_reduce.cuh: No such file or directory
 #include <cub/block/block_reduce.cuh>

but I find this .cuh file in my computer. I don't know how to do next, does anyone have the same question?
My environment: Ubuntu18.04, CUDA10.2

compilation terminated. Related to sophus/se3.hpp when run make -j

$ cmake ..
Proceeded without errors.

From what I've checked,
the CMakeLists.txt file has include_directories - ${PROJECT_SOURCE_DIR}/thirdparty/Sophus, so it seems to be sophus/se3.hpp with relative path, is that correct?

It doesn't seem to be finding the path, how can I fix this?

Segmentation fault

Hi and thank you for the repo.
I keep getting the following error on 80% of the videos.
I have tried different devices like gopro, iphone, drone footage. I also performed checkerboard calibration.

There has been an error. Do not use the model.
Couldn't load MVSNet successfully.INITIALIZE FROM INITIALIZER (1961 pts)!
I THINK INITIALIZATINO FAILED! Resetting.
RESETTING!
destroyed ThreadReduce
INITIALIZE FROM INITIALIZER (1961 pts)!
Segmentation fault (core dumped)

cmake error

编译cmake失败，报错找不到CUDA库，是由于cmake的问题吗？使用的是CUDA 11-1和CMAKE 3-16

error: no instance of overloaded function "atomicAdd" matches the argument list

Hi ,
I am tring to build the code with the cuda and cudann.
The installation of cuda and cudnn is good and when build the tandem code I got followings:
Any suggestions are appreciated.

Scanning dependencies of target cuda_coarse_tracker
[ 1%] Building CXX object libdr/cuda_coarse_tracker/CMakeFiles/cuda_coarse_tracker.dir/src/cuda_coarse_tracker.cpp.o
[ 3%] Building CUDA object libdr/cuda_coarse_tracker/CMakeFiles/cuda_coarse_tracker.dir/src/cuda_coarse_tracker_private.cu.o
/tDisk-1/workspace/bigtree/workspace/slam/tandem/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker_private.cu(355): error: no instance of overloaded function "atomicAdd" matches the argument list
argument types are: (double *, const double)
detected during instantiation of "void calcGKernel<TPB,Accum>(float, float, float2, float, int, int, const float *, const float *, const float *, const float *, const float *, const float *, const float *, const float *, Accum *) [with TPB=128, Accum=double]"
(429): here

/tDisk-1/workspace/bigtree/workspace/slam/tandem/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker_private.cu(355): error: no instance of overloaded function "atomicAdd" matches the argument list
argument types are: (double *, const double)
detected during instantiation of "void calcGKernel<TPB,Accum>(float, float, float2, float, int, int, const float *, const float *, const float *, const float *, const float *, const float *, const float *, const float *, Accum *) [with TPB=4, Accum=double]"
(445): here

2 errors detected in the compilation of "/tDisk-1/workspace/bigtree/workspace/slam/tandem/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker_private.cu".
libdr/cuda_coarse_tracker/CMakeFiles/cuda_coarse_tracker.dir/build.make:92: recipe for target 'libdr/cuda_coarse_tracker/CMakeFiles/cuda_coarse_tracker.dir/src/cuda_coarse_tracker_private.cu.o' failed
make[2]: *** [libdr/cuda_coarse_tracker/CMakeFiles/cuda_coarse_tracker.dir/src/cuda_coarse_tracker_private.cu.o] Error 1
CMakeFiles/Makefile2:298: recipe for target 'libdr/cuda_coarse_tracker/CMakeFiles/cuda_coarse_tracker.dir/all' failed
make[1]: *** [libdr/cuda_coarse_tracker/CMakeFiles/cuda_coarse_tracker.dir/all] Error 2
Makefile:100: recipe for target 'all' failed
make: *** [all] Error 2

How to convert ICL-NUIM to tandem format?

Hello, Thank you for your work.
I have download the ICL-NUIM dataset from the official website, and I tried to convert it to tandem format for depth-estimation on the CVA-MVSNET. I use the abl04_fewer_depth_planes.ckpt.
I have processed the data in the TUM-RGBD format, converted the quaternions to pose matrix and run DSO to record the optimization window. But it does not work.
Could you tell me how to convert ICL-NUIM to tandem format?

tandem_dataset preset=gui got Segmentation fault

Thanks for opensource the work. It's very amazing!
I am tryinig to reproduce what descripted in the work. I could run with tandem_dataset preset=dataset... on the euroc dataset.
But when I set preset as gui I got the following issue:

bigtrrssss:~/project/newproject/tandem/tandem$ scripts/tracking_euroc.bash

RUNNING --- results/tracking/dense/euroc/V1_01_easy/0
Framebuffer with requested attributes not available. Using available framebuffer. You may see visual artifacts.[W BinaryOps.cpp:607] Warning: floor_divide is deprecated, and will be removed in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values.
To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor'). (function operator())
[W TensorShape.cpp:2081] Warning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (function operator())
Pass 'Combine redundant instructions' is not initialized.
Verify if there is a pass dependency cycle.
Required Passes:
scripts/tracking_euroc.bash: line 14: 2268508 Segmentation fault (core dumped) build/bin/tandem_dataset preset=gui result_folder=$result_folder files=$scene/images calib=$scene/camera.txt mvsnet_folder=$mvsnet_folder exit_when_done=1 mode=1 > $result_folder/out.txt

Please let me know if I missed anthing.
Thanks.

cmake error

-- The C compiler identification is GNU 8.4.0
-- The CXX compiler identification is GNU 8.4.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Eigen3: /usr/local/include/eigen3 (Required is at least version "2.91.0")
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Found Boost: /usr/include (found version "1.67.0") found components: system thread chrono date_time atomic
-- Found CUDA: /usr/local/cuda-11.1 (found version "11.1")
-- Caffe2: CUDA detected: 11.1
-- Caffe2: CUDA nvcc is: /usr/local/cuda-11.1/bin/nvcc
-- Caffe2: CUDA toolkit directory: /usr/local/cuda-11.1
-- Caffe2: Header version is: 11.1
-- Found CUDNN: /usr/local/cuda-11.1/lib64/libcudnn.so
-- Found cuDNN: v7.6.5 (include: /usr/local/cuda-11.1/include, library: /usr/local/cuda-11.1/lib64/libcudnn.so)
-- /usr/local/cuda-11.1/lib64/libnvrtc.so shorthash is 3a20f2b6
-- Autodetected CUDA architecture(s): 7.5
-- Added CUDA NVCC flags for: -gencode;arch=compute_75,code=sm_75
-- Found Torch: /home/haimou/文档/libtorch-cxx11-abi-shared-with-deps-1.9.0+cu111/libtorch/lib/libtorch.so
-- Found OpenCV: /usr/local (found version "4.2.0")
-- The CUDA compiler identification is NVIDIA 11.1.105
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - done
-- Check for working CUDA compiler: /usr/local/cuda-11.1/bin/nvcc - skipped
-- Detecting CUDA compile features
-- Detecting CUDA compile features - done
-- Found CUDAToolkit: /usr/local/cuda-11.1/include (found version "11.1.105")
-- Found Eigen headers in: /usr/local/include/eigen3
-- Release mode (!= not Debug). Adding -O3;-lineinfo to NVCC FLAGS
--- found PANGOLIN, compiling tandem_pangolin library.
--- found OpenCV (/usr/local/include/opencv4), compiling tandem_opencv library.
--- compiling tandem_dataset.
--- not building tandem_demo, since either don't have openCV, Pangolin, or Realsense.
--- not building realsense_calib_recorder, since either don't have openCV, Pangolin, or Realsense.
-- Configuring done
CMake Error at CMakeLists.txt:131 (add_executable):
Target "tandem_dataset" links to target "Eigen3::Eigen" but the target was
not found. Perhaps a find_package() call is missing for an IMPORTED
target, or an ALIAS target is missing?

CMake Error at CMakeLists.txt:131 (add_executable):
Target "tandem_dataset" links to target "Eigen3::Eigen" but the target was
not found. Perhaps a find_package() call is missing for an IMPORTED
target, or an ALIAS target is missing?

-- Generating done
CMake Generate step failed. Build files cannot be regenerated correctly.

NO PHOTOMETRIC Calibration!

When I run tracking_euroc.bash, I get an error below.

ERROR: dont't have photometric calibation. Need to use commandline options mode=1 or mode=2 The run took 0 seconds which includes warm-up and loading.

Why there are differences between EuRoC dataset and EuRoC_Tandem_format's camera intrinsic?

I download the euroc_tandem_format_1.1.beta from here，and there is a camera.txt in in V1_01_easy directory, in V101_easy :
Pinhole 436.2250208688045 436.2250208688045 310.1627625810339 256.94604110717773 0
640 480
crop
640 480
but I downloaded EuRoC dataset from https://projects.asl.ethz.ch/datasets/doku.php?id=kmavvisualinertialdatasets
and I found the camera intrinsics in EuRoC Dataset is different from Tandem format, I want to ask why and how the camera intrinsics computed in tandem format
sensor.yaml of origin EuRoC dataset:
cam0
intrinsics: [458.654, 457.296, 367.215, 248.375] #fu, fv, cu, cv
cam1
intrinsics: [457.587, 456.134, 379.999, 255.238] #fu, fv, cu, cv

RuntimeError: CUDA out of memory

I will like to start by saying thank you for the amazing project and paper! After I cloned the repository I followed the instructions to set up the environment as described in the read me file. Finally, I downloaded the replica dataset and started the training, but before it even starts it exists with the CUDA out of memory error message. I'm using NVIDIA GeForce RTX 2080 Ti. I will put the full error message below. Any help would be appreciated . Thank you !!


GPU available: True, used: True
INFO:lightning:GPU available: True, used: True
No environment variable for node rank defined. Set as 0.
WARNING:lightning:No environment variable for node rank defined. Set as 0.
CUDA_VISIBLE_DEVICES: [0]
INFO:lightning:CUDA_VISIBLE_DEVICES: [0]
Epoch 1:   0%|                                                                                                                                                                     | 0/2714 [00:00<?, ?it/s]Traceback (most recent call last):
  File "train.py", line 128, in <module>
    main(*parse_args(parser))
  File "train.py", line 124, in main
    trainer.fit(model)
  File "/home/tmc/miniconda3/envs/tandem/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 859, in fit
    self.single_gpu_train(model)
  File "/home/tmc/miniconda3/envs/tandem/lib/python3.7/site-packages/pytorch_lightning/trainer/distrib_parts.py", line 503, in single_gpu_train
    self.run_pretrain_routine(model)
  File "/home/tmc/miniconda3/envs/tandem/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1015, in run_pretrain_routine
    self.train()
  File "/home/tmc/miniconda3/envs/tandem/lib/python3.7/site-packages/pytorch_lightning/trainer/training_loop.py", line 347, in train
    self.run_training_epoch()
  File "/home/tmc/miniconda3/envs/tandem/lib/python3.7/site-packages/pytorch_lightning/trainer/training_loop.py", line 419, in run_training_epoch
    _outputs = self.run_training_batch(batch, batch_idx)
  File "/home/tmc/miniconda3/envs/tandem/lib/python3.7/site-packages/pytorch_lightning/trainer/training_loop.py", line 597, in run_training_batch
    loss, batch_output = optimizer_closure()
  File "/home/tmc/miniconda3/envs/tandem/lib/python3.7/site-packages/pytorch_lightning/trainer/training_loop.py", line 561, in optimizer_closure
    output_dict = self.training_forward(split_batch, batch_idx, opt_idx, self.hiddens)
  File "/home/tmc/miniconda3/envs/tandem/lib/python3.7/site-packages/pytorch_lightning/trainer/training_loop.py", line 727, in training_forward
    output = self.model.training_step(*args)
  File "/home/tmc/tandem/cva_mvsnet/models/tandem.py", line 101, in training_step
    outputs = self(batch)
  File "/home/tmc/miniconda3/envs/tandem/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/tmc/tandem/cva_mvsnet/models/tandem.py", line 43, in forward
    depth_max=batch['depth_max'],
  File "/home/tmc/miniconda3/envs/tandem/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/tmc/tandem/cva_mvsnet/models/cva_mvsnet.py", line 162, in forward
    volume_gate=self.volume_gates[stage] if self.view_aggregation else None,
  File "/home/tmc/tandem/cva_mvsnet/models/module.py", line 1085, in depth_prediction
    half_pixel_centers=half_pixel_centers
  File "/home/tmc/tandem/cva_mvsnet/models/module.py", line 891, in homo_warping
    warped_src_fea[torch.isnan(warped_src_fea)] = 0
RuntimeError: CUDA out of memory. Tried to allocate 2.34 GiB (GPU 0; 10.76 GiB total capacity; 7.43 GiB already allocated; 740.06 MiB free; 8.94 GiB reserved in total by PyTorch)
Epoch 1:   0%|          | 0/2714 [00:04<?, ?it/s]

make error

Consolidate compiler generated dependencies of target dr-mvsnet
[ 1%] Building CUDA object libdr/cuda_coarse_tracker/CMakeFiles/cuda_coarse_tracker.dir/src/cuda_coarse_tracker_private.cu.o
[ 3%] Building CXX object libdr/cuda_coarse_tracker/CMakeFiles/cuda_coarse_tracker.dir/src/cuda_coarse_tracker.cpp.o
Consolidate compiler generated dependencies of target dr-fusion
[ 7%] Built target dr-mvsnet
[ 8%] Linking CXX static library lib/libdr-fusion.a
[ 10%] Building CXX object libdr/dr_mvsnet/CMakeFiles/dr_mvsnet_test.dir/src/dr_mvsnet_test.cpp.o
[ 25%] Built target dr-fusion
/home/tongwu/Desktop/tandem-master/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker.cpp:6:10: fatal error: sophus/se3.hpp: No such file or directory
#include <sophus/se3.hpp>
^~~~~~~~~~~~~~~~
compilation terminated.
libdr/cuda_coarse_tracker/CMakeFiles/cuda_coarse_tracker.dir/build.make:75: recipe for target 'libdr/cuda_coarse_tracker/CMakeFiles/cuda_coarse_tracker.dir/src/cuda_coarse_tracker.cpp.o' failed
make[2]: *** [libdr/cuda_coarse_tracker/CMakeFiles/cuda_coarse_tracker.dir/src/cuda_coarse_tracker.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
/home/tongwu/Desktop/tandem-master/tandem/libdr/cuda_coarse_tracker/src/cuda_coarse_tracker_private.cu:5:10: fatal error: cub/block/block_reduce.cuh: No such file or directory
#include <cub/block/block_reduce.cuh>
^~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
libdr/cuda_coarse_tracker/CMakeFiles/cuda_coarse_tracker.dir/build.make:89: recipe for target 'libdr/cuda_coarse_tracker/CMakeFiles/cuda_coarse_tracker.dir/src/cuda_coarse_tracker_private.cu.o' failed
make[2]: *** [libdr/cuda_coarse_tracker/CMakeFiles/cuda_coarse_tracker.dir/src/cuda_coarse_tracker_private.cu.o] Error 1
CMakeFiles/Makefile2:274: recipe for target 'libdr/cuda_coarse_tracker/CMakeFiles/cuda_coarse_tracker.dir/all' failed
make[1]: *** [libdr/cuda_coarse_tracker/CMakeFiles/cuda_coarse_tracker.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[ 26%] Linking CXX executable bin/dr_mvsnet_test
[ 26%] Built target dr_mvsnet_test
Makefile:90: recipe for target 'all' failed
make: *** [all] Error 2

it looks like cmake does not recognize the location of cub/block/block_reduce.cuh and sophus/se3.hpp
Any idea how I can solve this problem?

Can run without any visual window

(https://user-images.githubusercontent.com/91367508/176985320-60ae95da-2216-4fe6-8f83-53faa045151d.png)

Validation accuracy is lower than paper

This is the result on replica of abl01_baseline.ckpt :

STAGE3 a1 0.983 a2 0.729 a3 0.137
Eigen et. al : 0.9927 0.9971 0.99865

which is significantly lower than that in paper, and the other ablation ckpts are the same situation.

Have you test your method on Scannet Dataset?

You mentioned it in your paper you trained a model with scannet dataset, right? I'm curious if you test this method on Scannet and its performance. Does CVA-MVSNet achieve better results on Scannet with the model trained on Replica?

Custom Dataset Issue

Team,

Thanks for the great and exciting work!

All,

I've been trying to get the code to work with my own custom dataset (images exported from video) but have been running into issues.

My setup is as follows:
camera.txt

Pinhole 928.36203652 929.19195421 943.47243438 544.41549704 0
1080 1920
crop
1080 1920

Dataset PNG image format details:
000001.png: PNG image data, 1080 x 1920, 8-bit grayscale, non-interlaced

When running tandem_dataset, I get the following error:

RUNNING --- results/tracking/dense/custom/custom_01/0
[W BinaryOps.cpp:467] Warning: floor_divide is deprecated, and will be removed in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values.
To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor'). (function operator())
terminate called after throwing an instance of 'std::runtime_error'
  what():  The following operation failed in the TorchScript interpreter.
Traceback of TorchScript, serialized code (most recent call last):
  File "code/__torch__/models/cas_mvsnet/___torch_mangle_296.py", line 515, in forward
    input29 = torch._convolution(input28, CONSTANTS.c54, None, [2, 2, 2], [1, 1, 1], [1, 1, 1], True, [1, 1, 1], 1, False, False, True, True)
    input30 = torch.batch_norm(input29, CONSTANTS.c55, CONSTANTS.c56, CONSTANTS.c57, CONSTANTS.c58, False, 0.10000000000000001, 1.0000000000000001e-05, True)
    input31 = torch.add(_264, torch.relu_(input30))
              ~~~~~~~~~ <--- HERE
    input32 = torch._convolution(input31, CONSTANTS.c59, None, [2, 2, 2], [1, 1, 1], [1, 1, 1], True, [1, 1, 1], 1, False, False, True, True)
    input33 = torch.batch_norm(input32, CONSTANTS.c60, CONSTANTS.c61, CONSTANTS.c62, CONSTANTS.c63, False, 0.10000000000000001, 1.0000000000000001e-05, True)

Traceback of TorchScript, original code (most recent call last):
/home/lukas/dr_mvsnet_new/models/module.py(633): forward
/home/lukas/miniconda3/envs/t1.9/lib/python3.8/site-packages/torch/nn/modules/module.py(1039): _slow_forward
/home/lukas/miniconda3/envs/t1.9/lib/python3.8/site-packages/torch/nn/modules/module.py(1051): _call_impl
/home/lukas/dr_mvsnet_new/models/module.py(1291): depth_prediction
/home/lukas/dr_mvsnet_new/models/cas_mvsnet.py(376): forward
/home/lukas/miniconda3/envs/t1.9/lib/python3.8/site-packages/torch/nn/modules/module.py(1039): _slow_forward
/home/lukas/miniconda3/envs/t1.9/lib/python3.8/site-packages/torch/nn/modules/module.py(1051): _call_impl
/home/lukas/miniconda3/envs/t1.9/lib/python3.8/site-packages/torch/jit/_trace.py(952): trace_module
/home/lukas/miniconda3/envs/t1.9/lib/python3.8/site-packages/torch/jit/_trace.py(735): trace
export_model.py(204): main
export_model.py(230): <module>
RuntimeError: The size of tensor a (135) must match the size of tensor b (136) at non-singleton dimension 4

Is there something obvious that I might be missing?

Thanks!

Use preset=gui didn't get mesh

Hi,
I can successfully generate the mesh obj on the Euroc dataset.
My problem is when I set preset=gui I can get some frames showing on a window but didn't see the mesh in the same window.
Could you let me know the culprit?
Thanks.

About the backbone

Excellent work!
But I find the backbone, which is used to extract feature map, was implemented by yourselves instead of the pretrained resnet18. I wonder why this is happening, as we know pretrained model usually performs better than un-pretrained one, so whether the Imagenet feature is seriously not suitable for MVS?

not able to open "tandem/cva_mvsnet/model.pt"

Thanks for the opensourcing code. It seems the model.pt is not provided for running your system on euroc dataset.

After running the command for the given dataset euroc, the system seems to initialize successfully but is not able to open the "model.pt" using cva_mvsnet.
The flag "mvsnet_folder=Folder that contains the exported MVSNet." is not very clearly explained. Following is my log from the terminal:

~/tandem/tandem$ ./build/bin/tandem_dataset preset=dataset result_folder=./results files=/home/shu/Downloads/euroc_tandem_format_1.1.beta/euroc_tandem_format/V1_01_easy/images calib=/home/shu/Downloads/euroc_tandem_format_1.1.beta/euroc_tandem_format/V1_01_easy/camera.txt mvsnet_folder=/home/shu/Examples/tandem/cva_mvsnet mode=1

loading data from /home/shu/Downloads/euroc_tandem_format_1.1.beta/euroc_tandem_format/V1_01_easy/images!
loading calibration from /home/shu/Downloads/euroc_tandem_format_1.1.beta/euroc_tandem_format/V1_01_easy/camera.txt!
Loading MVSNet from /home/shu/Examples/tandem/cva_mvsnet/!
PHOTOMETRIC MODE WITHOUT CALIBRATION!

=============== TANDEM Settings: ===============
Setting 'dataset':
- no real-time enforcing
- 2000 active points
- 5-7 active frames
- 1-6 LM iteration each KF
- TSDF fusion: yes
- dense tracking on cpu (step=1)
- Pangolin
- Fullscreen: 0
- Mesh: 1
- Smaller Images: 1

Reading Calibration from file /home/shu/Downloads/euroc_tandem_format_1.1.beta/euroc_tandem_format/V1_01_easy/camera.txt ... found!
Input resolution: 640 480
In: 436.225021 436.225021 310.162763 256.946041 0.000000
Out: Rectify Crop
Output resolution: 640 480
finding CROP optimal new model!
initial range: x: -0.7181 - 0.7613; y: -0.5949 - 0.5141!
iteration 00001: range: x: -0.7145 - 0.7575; y: -0.5949 - 0.5141!
iteration 00002: range: x: -0.7109 - 0.7537; y: -0.5949 - 0.5141!
iteration 00003: range: x: -0.7109 - 0.7537; y: -0.5919 - 0.5115!
iteration 00004: range: x: -0.7109 - 0.7537; y: -0.5890 - 0.5090!
iteration 00005: range: x: -0.7109 - 0.7537; y: -0.5890 - 0.5090!

Rectified Kamera Matrix:
436.27 0 310.165
0 436.28 256.95
0 0 1

NO PHOTOMETRIC Calibration!
Reading Photometric Calibration from file
PhotometricUndistorter: Could not open file!
set timestamps and exposures to zero!
set EXPOSURES to zero!
got 2778 images and 0 timestamps and 0 exposures.!
ImageFolderReader: got 2778 files in /home/shu/Downloads/euroc_tandem_format_1.1.beta/euroc_tandem_format/V1_01_easy/images!
using pyramid levels 0 to 3. coarsest resolution: 80 x 60!

----DRMVSNET Initalizing fusion----

----DRMVSNET Initalizing fusion done----
DrMvsnet torch::cuda::is_vailable == true --> seems good
terminate called after throwing an instance of 'c10::Error'
what(): open file failed, file path: /home/shu/Examples/tandem/cva_mvsnet/model.pt
Exception raised from FileAdapter at ../caffe2/serialize/file_adapter.cc:11 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x69 (0x7f8ea507c1d9 in /home/shu/Examples/libtorch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) + 0xd2 (0x7f8ea5078812 in /home/shu/Examples/libtorch/lib/libc10.so)
frame #2: caffe2::serialize::FileAdapter::FileAdapter(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) + 0x1ff (0x7f8ea7ddbc0f in /home/shu/Examples/libtorch/lib/libtorch_cpu.so)
frame #3: torch::jit::load(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, c10::optionalc10::Device, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::hash<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator > const, std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > >&) + 0x40 (0x7f8ea92e4b50 in /home/shu/Examples/libtorch/lib/libtorch_cpu.so)
frame #4: torch::jit::load(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, c10::optionalc10::Device) + 0x6f (0x7f8ea92e4cdf in /home/shu/Examples/libtorch/lib/libtorch_cpu.so)
frame #5: + 0x16b827 (0x55a76f9e3827 in ./build/bin/tandem_dataset)
frame #6: + 0x163482 (0x55a76f9db482 in ./build/bin/tandem_dataset)
frame #7: + 0x3a6bc (0x55a76f8b26bc in ./build/bin/tandem_dataset)
frame #8: + 0x2980d (0x55a76f8a180d in ./build/bin/tandem_dataset)
frame #9: + 0xbd6df (0x7f8e526046df in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
frame #10: + 0x76db (0x7f8ea55006db in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #11: clone + 0x3f (0x7f8e51cc171f in /lib/x86_64-linux-gnu/libc.so.6)

Aborted (core dumped)

make error related to Sophus

Hi,
I meet some problem related to sophus while building the project,
the log is :
[ 81%] Building CXX object CMakeFiles/tandem.dir/src/dvo-core/src/dense_tracking_config.cpp.o
[ 83%] Building CXX object CMakeFiles/tandem.dir/src/dvo-core/src/util/histogram.cpp.o
[ 84%] Building CXX object CMakeFiles/tandem.dir/src/dvo-core/src/dense_tracking_impl.cpp.o
[ 86%] Building CXX object CMakeFiles/tandem.dir/src/IOWrapper/OpenCV/ImageDisplay_OpenCV.cpp.o
[ 88%] Building CXX object CMakeFiles/tandem.dir/src/IOWrapper/Pangolin/KeyFrameDisplay.cpp.o
[ 89%] Building CXX object CMakeFiles/tandem.dir/src/dvo-core/src/util/id_generator.cpp.o
[ 91%] Building CXX object CMakeFiles/tandem.dir/src/IOWrapper/Pangolin/DrFrameDisplay.cpp.o
[ 93%] Building CXX object CMakeFiles/tandem.dir/src/IOWrapper/OpenCV/ImageRW_OpenCV.cpp.o
[ 94%] Building CXX object CMakeFiles/tandem.dir/src/IOWrapper/Pangolin/PangolinDSOViewer.cpp.o
In file included from /home/hutiancheng/project/tandem/tandem/src/util/settings.cpp:27:0:
/usr/local/include/boost/bind.hpp:41:1: note: #pragma message: The practice of declaring the Bind placeholders (1, 2, ...) in the global namespace is deprecated. Please use <boost/bind/bind.hpp> + using namespace boost::placeholders, or define BOOST_BIND_GLOBAL_PLACEHOLDERS to retain the current behavior.
)
^
[ 94%] Built target dr_mvsnet_test
/home/hutiancheng/project/tandem/tandem/src/FullSystem/HessianBlocks.cpp: In member function ‘void dso::FrameHessian::setStateZero(const Vec10&)’:
/home/hutiancheng/project/tandem/tandem/src/FullSystem/HessianBlocks.cpp:84:23: error: ‘template<class Scalar, int Options> class Sophus::SE3’ used without template parameters
SE3 EepsP = Sophus::SE3::exp(eps);
^~~
/home/hutiancheng/project/tandem/tandem/src/FullSystem/HessianBlocks.cpp:85:23: error: ‘template<class Scalar, int Options> class Sophus::SE3’ used without template parameters
SE3 EepsM = Sophus::SE3::exp(-eps);
^~~
/home/hutiancheng/project/tandem/tandem/src/IOWrapper/Pangolin/PangolinDSOViewer.cpp: In member function ‘void dso::IOWrap::PangolinDSOViewer::run()’:
/home/hutiancheng/project/tandem/tandem/src/IOWrapper/Pangolin/PangolinDSOViewer.cpp:280:27: error: ‘glDrawColoredVertices’ is not a member of ‘pangolin’
pangolin::glDrawColoredVertices(
^~~~~~~~~~~~~~~~~~~~~
/home/hutiancheng/project/tandem/tandem/src/IOWrapper/Pangolin/PangolinDSOViewer.cpp:280:27: note: suggested alternative: ‘glDrawColouredCube’
pangolin::glDrawColoredVertices(
^~~~~~~~~~~~~~~~~~~~~
glDrawColouredCube
/home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp: In member function ‘dso::Vec4 dso::FullSystem::trackNewCoarse(dso::FrameHessian*)’:
/home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:452:46: error: ‘Quaterniond’ is not a member of ‘Sophus’
SE3(Sophus::Quaterniond(1, rotDelta, 0, 0),
^~~~~~~~~~~
/home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:452:46: note: suggested alternative:
In file included from /usr/local/include/eigen3/Eigen/Geometry:42:0,
from /usr/local/include/eigen3/Eigen/Dense:6,
from /usr/local/include/sophus/rotation_matrix.hpp:7,
from /usr/local/include/sophus/so3.hpp:7,
from /usr/local/include/sophus/rxso3.hpp:7,
from /usr/local/include/sophus/sim3.hpp:7,
from /home/hutiancheng/project/tandem/tandem/src/util/NumType.h:28,
from /home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.h:29,
from /home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:32:
/usr/local/include/eigen3/Eigen/src/Geometry/Quaternion.h:310:28: note: ‘Eigen::Quaterniond’
typedef Quaternion Quaterniond;
^~~~~~~~~~~
/home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:455:46: error: ‘Quaterniond’ is not a member of ‘Sophus’
SE3(Sophus::Quaterniond(1, 0, rotDelta, 0),
^~~~~~~~~~~
/home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:455:46: note: suggested alternative:
In file included from /usr/local/include/eigen3/Eigen/Geometry:42:0,
from /usr/local/include/eigen3/Eigen/Dense:6,
from /usr/local/include/sophus/rotation_matrix.hpp:7,
from /usr/local/include/sophus/so3.hpp:7,
from /usr/local/include/sophus/rxso3.hpp:7,
from /usr/local/include/sophus/sim3.hpp:7,
from /home/hutiancheng/project/tandem/tandem/src/util/NumType.h:28,
from /home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.h:29,
from /home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:32:
/usr/local/include/eigen3/Eigen/src/Geometry/Quaternion.h:310:28: note: ‘Eigen::Quaterniond’
typedef Quaternion Quaterniond;
^~~~~~~~~~~
/home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:458:46: error: ‘Quaterniond’ is not a member of ‘Sophus’
SE3(Sophus::Quaterniond(1, 0, 0, rotDelta),
^~~~~~~~~~~
/home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:458:46: note: suggested alternative:
In file included from /usr/local/include/eigen3/Eigen/Geometry:42:0,
from /usr/local/include/eigen3/Eigen/Dense:6,
from /usr/local/include/sophus/rotation_matrix.hpp:7,
from /usr/local/include/sophus/so3.hpp:7,
from /usr/local/include/sophus/rxso3.hpp:7,
from /usr/local/include/sophus/sim3.hpp:7,
from /home/hutiancheng/project/tandem/tandem/src/util/NumType.h:28,
from /home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.h:29,
from /home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:32:
/usr/local/include/eigen3/Eigen/src/Geometry/Quaternion.h:310:28: note: ‘Eigen::Quaterniond’
typedef Quaternion Quaterniond;
^~~~~~~~~~~
/home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:461:46: error: ‘Quaterniond’ is not a member of ‘Sophus’
SE3(Sophus::Quaterniond(1, -rotDelta, 0, 0),
^~~~~~~~~~~
/home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:461:46: note: suggested alternative:
In file included from /usr/local/include/eigen3/Eigen/Geometry:42:0,
from /usr/local/include/eigen3/Eigen/Dense:6,
from /usr/local/include/sophus/rotation_matrix.hpp:7,
from /usr/local/include/sophus/so3.hpp:7,
from /usr/local/include/sophus/rxso3.hpp:7,
from /usr/local/include/sophus/sim3.hpp:7,
from /home/hutiancheng/project/tandem/tandem/src/util/NumType.h:28,
from /home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.h:29,
from /home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:32:
/usr/local/include/eigen3/Eigen/src/Geometry/Quaternion.h:310:28: note: ‘Eigen::Quaterniond’
typedef Quaternion Quaterniond;
^~~~~~~~~~~
/home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:464:46: error: ‘Quaterniond’ is not a member of ‘Sophus’
SE3(Sophus::Quaterniond(1, 0, -rotDelta, 0),
^~~~~~~~~~~
/home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:464:46: note: suggested alternative:
In file included from /usr/local/include/eigen3/Eigen/Geometry:42:0,
from /usr/local/include/eigen3/Eigen/Dense:6,
from /usr/local/include/sophus/rotation_matrix.hpp:7,
from /usr/local/include/sophus/so3.hpp:7,
from /usr/local/include/sophus/rxso3.hpp:7,
from /usr/local/include/sophus/sim3.hpp:7,
from /home/hutiancheng/project/tandem/tandem/src/util/NumType.h:28,
from /home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.h:29,
from /home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:32:
/usr/local/include/eigen3/Eigen/src/Geometry/Quaternion.h:310:28: note: ‘Eigen::Quaterniond’
typedef Quaternion Quaterniond;
^~~~~~~~~~~
/home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:467:46: error: ‘Quaterniond’ is not a member of ‘Sophus’
SE3(Sophus::Quaterniond(1, 0, 0, -rotDelta),
^~~~~~~~~~~
/home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:467:46: note: suggested alternative:
In file included from /usr/local/include/eigen3/Eigen/Geometry:42:0,
from /usr/local/include/eigen3/Eigen/Dense:6,
from /usr/local/include/sophus/rotation_matrix.hpp:7,
from /usr/local/include/sophus/so3.hpp:7,
from /usr/local/include/sophus/rxso3.hpp:7,
from /usr/local/include/sophus/sim3.hpp:7,
from /home/hutiancheng/project/tandem/tandem/src/util/NumType.h:28,
from /home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.h:29,
from /home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:32:
/usr/local/include/eigen3/Eigen/src/Geometry/Quaternion.h:310:28: note: ‘Eigen::Quaterniond’
typedef Quaternion Quaterniond;
^~~~~~~~~~~
/home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:470:46: error: ‘Quaterniond’ is not a member of ‘Sophus’
SE3(Sophus::Quaterniond(1, rotDelta, rotDelta, 0),
^~~~~~~~~~~
/home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:470:46: note: suggested alternative:
In file included from /usr/local/include/eigen3/Eigen/Geometry:42:0,
from /usr/local/include/eigen3/Eigen/Dense:6,
from /usr/local/include/sophus/rotation_matrix.hpp:7,
from /usr/local/include/sophus/so3.hpp:7,
from /usr/local/include/sophus/rxso3.hpp:7,
from /usr/local/include/sophus/sim3.hpp:7,
from /home/hutiancheng/project/tandem/tandem/src/util/NumType.h:28,
from /home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.h:29,
from /home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:32:
/usr/local/include/eigen3/Eigen/src/Geometry/Quaternion.h:310:28: note: ‘Eigen::Quaterniond’
typedef Quaternion Quaterniond;
^~~~~~~~~~~
/home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:473:46: error: ‘Quaterniond’ is not a member of ‘Sophus’
SE3(Sophus::Quaterniond(1, 0, rotDelta, rotDelta),
^~~~~~~~~~~
/home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:473:46: note: suggested alternative:
In file included from /usr/local/include/eigen3/Eigen/Geometry:42:0,
from /usr/local/include/eigen3/Eigen/Dense:6,
from /usr/local/include/sophus/rotation_matrix.hpp:7,
from /usr/local/include/sophus/so3.hpp:7,
from /usr/local/include/sophus/rxso3.hpp:7,
from /usr/local/include/sophus/sim3.hpp:7,
from /home/hutiancheng/project/tandem/tandem/src/util/NumType.h:28,
from /home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.h:29,
from /home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:32:
/usr/local/include/eigen3/Eigen/src/Geometry/Quaternion.h:310:28: note: ‘Eigen::Quaterniond’
typedef Quaternion Quaterniond;
^~~~~~~~~~~
/home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:476:46: error: ‘Quaterniond’ is not a member of ‘Sophus’
SE3(Sophus::Quaterniond(1, rotDelta, 0, rotDelta),
^~~~~~~~~~~
/home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:476:46: note: suggested alternative:
In file included from /usr/local/include/eigen3/Eigen/Geometry:42:0,
from /usr/local/include/eigen3/Eigen/Dense:6,
from /usr/local/include/sophus/rotation_matrix.hpp:7,
from /usr/local/include/sophus/so3.hpp:7,
from /usr/local/include/sophus/rxso3.hpp:7,
from /usr/local/include/sophus/sim3.hpp:7,
from /home/hutiancheng/project/tandem/tandem/src/util/NumType.h:28,
from /home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.h:29,
from /home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:32:
/usr/local/include/eigen3/Eigen/src/Geometry/Quaternion.h:310:28: note: ‘Eigen::Quaterniond’
typedef Quaternion Quaterniond;
^~~~~~~~~~~
/home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:479:46: error: ‘Quaterniond’ is not a member of ‘Sophus’
SE3(Sophus::Quaterniond(1, -rotDelta, rotDelta, 0),
^~~~~~~~~~~
/home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:479:46: note: suggested alternative:
In file included from /usr/local/include/eigen3/Eigen/Geometry:42:0,
from /usr/local/include/eigen3/Eigen/Dense:6,
from /usr/local/include/sophus/rotation_matrix.hpp:7,
from /usr/local/include/sophus/so3.hpp:7,
from /usr/local/include/sophus/rxso3.hpp:7,
from /usr/local/include/sophus/sim3.hpp:7,
from /home/hutiancheng/project/tandem/tandem/src/util/NumType.h:28,
from /home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.h:29,
from /home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:32:
/usr/local/include/eigen3/Eigen/src/Geometry/Quaternion.h:310:28: note: ‘Eigen::Quaterniond’
typedef Quaternion Quaterniond;
^~~~~~~~~~~
/home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:482:46: error: ‘Quaterniond’ is not a member of ‘Sophus’
SE3(Sophus::Quaterniond(1, 0, -rotDelta, rotDelta),
^~~~~~~~~~~
/home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:482:46: note: suggested alternative:
In file included from /usr/local/include/eigen3/Eigen/Geometry:42:0,
from /usr/local/include/eigen3/Eigen/Dense:6,
from /usr/local/include/sophus/rotation_matrix.hpp:7,
from /usr/local/include/sophus/so3.hpp:7,
from /usr/local/include/sophus/rxso3.hpp:7,
from /usr/local/include/sophus/sim3.hpp:7,
from /home/hutiancheng/project/tandem/tandem/src/util/NumType.h:28,
from /home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.h:29,
from /home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:32:
/usr/local/include/eigen3/Eigen/src/Geometry/Quaternion.h:310:28: note: ‘Eigen::Quaterniond’
typedef Quaternion Quaterniond;
^~~~~~~~~~~
/home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:485:46: error: ‘Quaterniond’ is not a member of ‘Sophus’
SE3(Sophus::Quaterniond(1, -rotDelta, 0, rotDelta),
^~~~~~~~~~~
/home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:485:46: note: suggested alternative:
In file included from /usr/local/include/eigen3/Eigen/Geometry:42:0,
from /usr/local/include/eigen3/Eigen/Dense:6,
from /usr/local/include/sophus/rotation_matrix.hpp:7,
from /usr/local/include/sophus/so3.hpp:7,
from /usr/local/include/sophus/rxso3.hpp:7,
from /usr/local/include/sophus/sim3.hpp:7,
from /home/hutiancheng/project/tandem/tandem/src/util/NumType.h:28,
from /home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.h:29,
from /home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:32:
/usr/local/include/eigen3/Eigen/src/Geometry/Quaternion.h:310:28: note: ‘Eigen::Quaterniond’
typedef Quaternion Quaterniond;
^~~~~~~~~~~
/home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:488:46: error: ‘Quaterniond’ is not a member of ‘Sophus’
SE3(Sophus::Quaterniond(1, rotDelta, -rotDelta, 0),
^~~~~~~~~~~
/home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:488:46: note: suggested alternative:
In file included from /usr/local/include/eigen3/Eigen/Geometry:42:0,
from /usr/local/include/eigen3/Eigen/Dense:6,
from /usr/local/include/sophus/rotation_matrix.hpp:7,
from /usr/local/include/sophus/so3.hpp:7,
from /usr/local/include/sophus/rxso3.hpp:7,
from /usr/local/include/sophus/sim3.hpp:7,
from /home/hutiancheng/project/tandem/tandem/src/util/NumType.h:28,
from /home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.h:29,
from /home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:32:
/usr/local/include/eigen3/Eigen/src/Geometry/Quaternion.h:310:28: note: ‘Eigen::Quaterniond’
typedef Quaternion Quaterniond;
^~~~~~~~~~~
/home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:491:46: error: ‘Quaterniond’ is not a member of ‘Sophus’
SE3(Sophus::Quaterniond(1, 0, rotDelta, -rotDelta),
^~~~~~~~~~~
/home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:491:46: note: suggested alternative:
In file included from /usr/local/include/eigen3/Eigen/Geometry:42:0,
from /usr/local/include/eigen3/Eigen/Dense:6,
from /usr/local/include/sophus/rotation_matrix.hpp:7,
from /usr/local/include/sophus/so3.hpp:7,
from /usr/local/include/sophus/rxso3.hpp:7,
from /usr/local/include/sophus/sim3.hpp:7,
from /home/hutiancheng/project/tandem/tandem/src/util/NumType.h:28,
from /home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.h:29,
from /home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:32:
/usr/local/include/eigen3/Eigen/src/Geometry/Quaternion.h:310:28: note: ‘Eigen::Quaterniond’
typedef Quaternion Quaterniond;
^~~~~~~~~~~
/home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:494:46: error: ‘Quaterniond’ is not a member of ‘Sophus’
SE3(Sophus::Quaterniond(1, rotDelta, 0, -rotDelta),
^~~~~~~~~~~
/home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:494:46: note: suggested alternative:
In file included from /usr/local/include/eigen3/Eigen/Geometry:42:0,
from /usr/local/include/eigen3/Eigen/Dense:6,
from /usr/local/include/sophus/rotation_matrix.hpp:7,
from /usr/local/include/sophus/so3.hpp:7,
from /usr/local/include/sophus/rxso3.hpp:7,
from /usr/local/include/sophus/sim3.hpp:7,
from /home/hutiancheng/project/tandem/tandem/src/util/NumType.h:28,
from /home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.h:29,
from /home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:32:
/usr/local/include/eigen3/Eigen/src/Geometry/Quaternion.h:310:28: note: ‘Eigen::Quaterniond’
typedef Quaternion Quaterniond;
^~~~~~~~~~~
/home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:497:46: error: ‘Quaterniond’ is not a member of ‘Sophus’
SE3(Sophus::Quaterniond(1, -rotDelta, -rotDelta, 0),
^~~~~~~~~~~
/home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:497:46: note: suggested alternative:
In file included from /usr/local/include/eigen3/Eigen/Geometry:42:0,
from /usr/local/include/eigen3/Eigen/Dense:6,
from /usr/local/include/sophus/rotation_matrix.hpp:7,
from /usr/local/include/sophus/so3.hpp:7,
from /usr/local/include/sophus/rxso3.hpp:7,
from /usr/local/include/sophus/sim3.hpp:7,
from /home/hutiancheng/project/tandem/tandem/src/util/NumType.h:28,
from /home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.h:29,
from /home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:32:
/usr/local/include/eigen3/Eigen/src/Geometry/Quaternion.h:310:28: note: ‘Eigen::Quaterniond’
typedef Quaternion Quaterniond;
^~~~~~~~~~~
/home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:500:46: error: ‘Quaterniond’ is not a member of ‘Sophus’
SE3(Sophus::Quaterniond(1, 0, -rotDelta, -rotDelta),
^~~~~~~~~~~
/home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:500:46: note: suggested alternative:
In file included from /usr/local/include/eigen3/Eigen/Geometry:42:0,
from /usr/local/include/eigen3/Eigen/Dense:6,
from /usr/local/include/sophus/rotation_matrix.hpp:7,
from /usr/local/include/sophus/so3.hpp:7,
from /usr/local/include/sophus/rxso3.hpp:7,
from /usr/local/include/sophus/sim3.hpp:7,
from /home/hutiancheng/project/tandem/tandem/src/util/NumType.h:28,
from /home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.h:29,
from /home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:32:
/usr/local/include/eigen3/Eigen/src/Geometry/Quaternion.h:310:28: note: ‘Eigen::Quaterniond’
typedef Quaternion Quaterniond;
^~~~~~~~~~~
/home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:503:46: error: ‘Quaterniond’ is not a member of ‘Sophus’
SE3(Sophus::Quaterniond(1, -rotDelta, 0, -rotDelta),
^~~~~~~~~~~
/home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:503:46: note: suggested alternative:
In file included from /usr/local/include/eigen3/Eigen/Geometry:42:0,
from /usr/local/include/eigen3/Eigen/Dense:6,
from /usr/local/include/sophus/rotation_matrix.hpp:7,
from /usr/local/include/sophus/so3.hpp:7,
from /usr/local/include/sophus/rxso3.hpp:7,
from /usr/local/include/sophus/sim3.hpp:7,
from /home/hutiancheng/project/tandem/tandem/src/util/NumType.h:28,
from /home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.h:29,
from /home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:32:
/usr/local/include/eigen3/Eigen/src/Geometry/Quaternion.h:310:28: note: ‘Eigen::Quaterniond’
typedef Quaternion Quaterniond;
^~~~~~~~~~~
/home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:506:46: error: ‘Quaterniond’ is not a member of ‘Sophus’
SE3(Sophus::Quaterniond(1, -rotDelta, -rotDelta, -rotDelta),
^~~~~~~~~~~
/home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:506:46: note: suggested alternative:
In file included from /usr/local/include/eigen3/Eigen/Geometry:42:0,
from /usr/local/include/eigen3/Eigen/Dense:6,
from /usr/local/include/sophus/rotation_matrix.hpp:7,
from /usr/local/include/sophus/so3.hpp:7,
from /usr/local/include/sophus/rxso3.hpp:7,
from /usr/local/include/sophus/sim3.hpp:7,
from /home/hutiancheng/project/tandem/tandem/src/util/NumType.h:28,
from /home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.h:29,
from /home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:32:
/usr/local/include/eigen3/Eigen/src/Geometry/Quaternion.h:310:28: note: ‘Eigen::Quaterniond’
typedef Quaternion Quaterniond;
^~~~~~~~~~~
/home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:509:46: error: ‘Quaterniond’ is not a member of ‘Sophus’
SE3(Sophus::Quaterniond(1, -rotDelta, -rotDelta, rotDelta),
^~~~~~~~~~~
/home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:509:46: note: suggested alternative:
In file included from /usr/local/include/eigen3/Eigen/Geometry:42:0,
from /usr/local/include/eigen3/Eigen/Dense:6,
from /usr/local/include/sophus/rotation_matrix.hpp:7,
from /usr/local/include/sophus/so3.hpp:7,
from /usr/local/include/sophus/rxso3.hpp:7,
from /usr/local/include/sophus/sim3.hpp:7,
from /home/hutiancheng/project/tandem/tandem/src/util/NumType.h:28,
from /home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.h:29,
from /home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:32:
/usr/local/include/eigen3/Eigen/src/Geometry/Quaternion.h:310:28: note: ‘Eigen::Quaterniond’
typedef Quaternion Quaterniond;
^~~~~~~~~~~
/home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:512:46: error: ‘Quaterniond’ is not a member of ‘Sophus’
SE3(Sophus::Quaterniond(1, -rotDelta, rotDelta, -rotDelta),
^~~~~~~~~~~
/home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:512:46: note: suggested alternative:
In file included from /usr/local/include/eigen3/Eigen/Geometry:42:0,
from /usr/local/include/eigen3/Eigen/Dense:6,
from /usr/local/include/sophus/rotation_matrix.hpp:7,
from /usr/local/include/sophus/so3.hpp:7,
from /usr/local/include/sophus/rxso3.hpp:7,
from /usr/local/include/sophus/sim3.hpp:7,
from /home/hutiancheng/project/tandem/tandem/src/util/NumType.h:28,
from /home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.h:29,
from /home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:32:
/usr/local/include/eigen3/Eigen/src/Geometry/Quaternion.h:310:28: note: ‘Eigen::Quaterniond’
typedef Quaternion Quaterniond;
^~~~~~~~~~~
/home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:515:46: error: ‘Quaterniond’ is not a member of ‘Sophus’
SE3(Sophus::Quaterniond(1, -rotDelta, rotDelta, rotDelta),
^~~~~~~~~~~
/home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:515:46: note: suggested alternative:
In file included from /usr/local/include/eigen3/Eigen/Geometry:42:0,
from /usr/local/include/eigen3/Eigen/Dense:6,
from /usr/local/include/sophus/rotation_matrix.hpp:7,
from /usr/local/include/sophus/so3.hpp:7,
from /usr/local/include/sophus/rxso3.hpp:7,
from /usr/local/include/sophus/sim3.hpp:7,
from /home/hutiancheng/project/tandem/tandem/src/util/NumType.h:28,
from /home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.h:29,
from /home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:32:
/usr/local/include/eigen3/Eigen/src/Geometry/Quaternion.h:310:28: note: ‘Eigen::Quaterniond’
typedef Quaternion Quaterniond;
^~~~~~~~~~~
/home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:518:46: error: ‘Quaterniond’ is not a member of ‘Sophus’
SE3(Sophus::Quaterniond(1, rotDelta, -rotDelta, -rotDelta),
^~~~~~~~~~~
/home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:518:46: note: suggested alternative:
In file included from /usr/local/include/eigen3/Eigen/Geometry:42:0,
from /usr/local/include/eigen3/Eigen/Dense:6,
from /usr/local/include/sophus/rotation_matrix.hpp:7,
from /usr/local/include/sophus/so3.hpp:7,
from /usr/local/include/sophus/rxso3.hpp:7,
from /usr/local/include/sophus/sim3.hpp:7,
from /home/hutiancheng/project/tandem/tandem/src/util/NumType.h:28,
from /home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.h:29,
from /home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:32:
/usr/local/include/eigen3/Eigen/src/Geometry/Quaternion.h:310:28: note: ‘Eigen::Quaterniond’
typedef Quaternion Quaterniond;
^~~~~~~~~~~
/home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:521:46: error: ‘Quaterniond’ is not a member of ‘Sophus’
SE3(Sophus::Quaterniond(1, rotDelta, -rotDelta, rotDelta),
^~~~~~~~~~~
/home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:521:46: note: suggested alternative:
In file included from /usr/local/include/eigen3/Eigen/Geometry:42:0,
from /usr/local/include/eigen3/Eigen/Dense:6,
from /usr/local/include/sophus/rotation_matrix.hpp:7,
from /usr/local/include/sophus/so3.hpp:7,
from /usr/local/include/sophus/rxso3.hpp:7,
from /usr/local/include/sophus/sim3.hpp:7,
from /home/hutiancheng/project/tandem/tandem/src/util/NumType.h:28,
from /home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.h:29,
from /home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:32:
/usr/local/include/eigen3/Eigen/src/Geometry/Quaternion.h:310:28: note: ‘Eigen::Quaterniond’
typedef Quaternion Quaterniond;
^~~~~~~~~~~
/home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:524:46: error: ‘Quaterniond’ is not a member of ‘Sophus’
SE3(Sophus::Quaterniond(1, rotDelta, rotDelta, -rotDelta),
^~~~~~~~~~~
/home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:524:46: note: suggested alternative:
In file included from /usr/local/include/eigen3/Eigen/Geometry:42:0,
from /usr/local/include/eigen3/Eigen/Dense:6,
from /usr/local/include/sophus/rotation_matrix.hpp:7,
from /usr/local/include/sophus/so3.hpp:7,
from /usr/local/include/sophus/rxso3.hpp:7,
from /usr/local/include/sophus/sim3.hpp:7,
from /home/hutiancheng/project/tandem/tandem/src/util/NumType.h:28,
from /home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.h:29,
from /home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:32:
/usr/local/include/eigen3/Eigen/src/Geometry/Quaternion.h:310:28: note: ‘Eigen::Quaterniond’
typedef Quaternion Quaterniond;
^~~~~~~~~~~
/home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:527:46: error: ‘Quaterniond’ is not a member of ‘Sophus’
SE3(Sophus::Quaterniond(1, rotDelta, rotDelta, rotDelta),
^~~~~~~~~~~
/home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:527:46: note: suggested alternative:
In file included from /usr/local/include/eigen3/Eigen/Geometry:42:0,
from /usr/local/include/eigen3/Eigen/Dense:6,
from /usr/local/include/sophus/rotation_matrix.hpp:7,
from /usr/local/include/sophus/so3.hpp:7,
from /usr/local/include/sophus/rxso3.hpp:7,
from /usr/local/include/sophus/sim3.hpp:7,
from /home/hutiancheng/project/tandem/tandem/src/util/NumType.h:28,
from /home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.h:29,
from /home/hutiancheng/project/tandem/tandem/src/FullSystem/FullSystem.cpp:32:
/usr/local/include/eigen3/Eigen/src/Geometry/Quaternion.h:310:28: note: ‘Eigen::Quaterniond’
typedef Quaternion Quaterniond;
^~~~~~~~~~~
CMakeFiles/tandem.dir/build.make:209: recipe for target 'CMakeFiles/tandem.dir/src/FullSystem/HessianBlocks.cpp.o' failed
make[2]: *** [CMakeFiles/tandem.dir/src/FullSystem/HessianBlocks.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
CMakeFiles/tandem.dir/build.make:573: recipe for target 'CMakeFiles/tandem.dir/src/IOWrapper/Pangolin/PangolinDSOViewer.cpp.o' failed
make[2]: *** [CMakeFiles/tandem.dir/src/IOWrapper/Pangolin/PangolinDSOViewer.cpp.o] Error 1
CMakeFiles/tandem.dir/build.make:79: recipe for target 'CMakeFiles/tandem.dir/src/FullSystem/FullSystem.cpp.o' failed
make[2]: *** [CMakeFiles/tandem.dir/src/FullSystem/FullSystem.cpp.o] Error 1
CMakeFiles/Makefile2:190: recipe for target 'CMakeFiles/tandem.dir/all' failed
make[1]: *** [CMakeFiles/tandem.dir/all] Error 2
Makefile:100: recipe for target 'all' failed
make: *** [all] Error 2

could you give me some device?
Thanks a lot!

Segmentation fault when testing tandem on a custom synthetic data

Hello,

I am trying to test Tandem on a custom dataset.

When it loads the model the program crashes and I get a segmentation fault like this

(tandem) pankhuri@pankhuri-GE76-Raider-11UH:~/tandem/tandem$ ./build/bin/tandem_dataset preset=gui result_folder=/home/pankhuri/tandem-dataset/results files=/home/pankhuri/tandem-dataset/images calib=/home/pankhuri/tandem-dataset/camera.txt mvsnet_folder=/home/pankhuri/tandem/tandem/exported/tandem mode=1
loading data from /home/pankhuri/tandem-dataset/images!
loading calibration from /home/pankhuri/tandem-dataset/camera.txt!
Loading MVSNet from /home/pankhuri/tandem/tandem/exported/tandem/!
PHOTOMETRIC MODE WITHOUT CALIBRATION!

=============== TANDEM Settings: ===============
	Setting 'gui':
	- no real-time enforcing
	- 2000 active points
	- 5-7 active frames
	- 1-6 LM iteration each KF
	- TSDF fusion: yes
	- dense tracking on cpu (step=1)
	- Pangolin
	  - Fullscreen: 0
	  - Mesh: 1
	  - Smaller Images: 1

Reading Calibration from file /home/pankhuri/tandem-dataset/camera.txt ... found!
Input resolution: 640 480
In: 300.000000 300.000000 480.000000 270.000000 0.000000
Out: Rectify Crop
Output resolution: 640 480
finding CROP optimal new model!
initial range: x: -1.6159 - 0.5352; y: -0.9090 - 0.7036!
iteration 00001: range: x: -1.6078 - 0.5325; y: -0.9090 - 0.7036!
iteration 00002: range: x: -1.5998 - 0.5299; y: -0.9090 - 0.7036!
iteration 00003: range: x: -1.5998 - 0.5299; y: -0.9045 - 0.7000!
iteration 00004: range: x: -1.5998 - 0.5299; y: -0.8999 - 0.6965!
iteration 00005: range: x: -1.5998 - 0.5299; y: -0.8999 - 0.6965!

Rectified Kamera Matrix:
300.051       0 480.015
      0 300.035 270.011
      0       0       1

NO PHOTOMETRIC Calibration!
Reading Photometric Calibration from file 
PhotometricUndistorter: Could not open file!
got 0 images and 0 timestamps and 0 exposures.!
ImageFolderReader: got 0 files in /home/pankhuri/tandem-dataset/images!
using pyramid levels 0 to 3. coarsest resolution: 80 x 60!
START PANGOLIN!

----DRMVSNET Initalizing fusion----

----DRMVSNET Initalizing fusion done----
DrMvsnet torch::cuda::is_vailable == true --> seems good
View Num: 7, ref index: 5
[W BinaryOps.cpp:467] Warning: floor_divide is deprecated, and will be removed in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values.
To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor'). (function operator())
Correctness:
	Depth correct     : 0, error: 0.0106286
	Confidence correct: 1, error: 0.00296127
Correctness:
	Depth correct     : 0, error: 0.0106182
	Confidence correct: 1, error: 0.00295481
Correctness:
	Depth correct     : 0, error: 0.0106182
	Confidence correct: 1, error: 0.00295481
Correctness:
	Depth correct     : 0, error: 0.0106182
	Confidence correct: 1, error: 0.00295482
Correctness:
	Depth correct     : 0, error: 0.0106182
	Confidence correct: 1, error: 0.00295482
Correctness:
	Depth correct     : 0, error: 0.0106182
	Confidence correct: 1, error: 0.00295482
Correctness:
	Depth correct     : 0, error: 0.0106182
	Confidence correct: 1, error: 0.0029548
Correctness:
	Depth correct     : 0, error: 0.0106182
	Confidence correct: 1, error: 0.00295482
Correctness:
	Depth correct     : 0, error: 0.0106182
	Confidence correct: 1, error: 0.00295482
Performance:
	CallAsync     : 6.832 ms
	Ready         : 0 ms
	GetResult     : 116.539 ms
There has been an error. Do not use the model.
Segmentation fault (core dumped)
(tandem) pankhuri@pankhuri-GE76-Raider-11UH:~/tandem/tandem$

What could be the reason behind this segmentation fault? Is it dependent on geometric calibration?

Note:

It works fine with other dataset like tum rgbd freiburg sequence, although there comes a warning of not loading MVSNet successfully

(tandem) pankhuri@pankhuri-GE76-Raider-11UH:~/tandem/tandem$ ./build/bin/tandem_dataset preset=gui result_folder=/home/pankhuri/tandem-dataset/results files=/home/pankhuri/rgbd_dataset_freiburg2_xyz/rgb calib=/home/pankhuri/rgbd_dataset_freiburg2_xyz/camera.txt mvsnet_folder=/home/pankhuri/tandem/tandem/exported/tandem mode=1
loading data from /home/pankhuri/rgbd_dataset_freiburg2_xyz/rgb!
loading calibration from /home/pankhuri/rgbd_dataset_freiburg2_xyz/camera.txt!
Loading MVSNet from /home/pankhuri/tandem/tandem/exported/tandem/!
PHOTOMETRIC MODE WITHOUT CALIBRATION!

=============== TANDEM Settings: ===============
	Setting 'gui':
	- no real-time enforcing
	- 2000 active points
	- 5-7 active frames
	- 1-6 LM iteration each KF
	- TSDF fusion: yes
	- dense tracking on cpu (step=1)
	- Pangolin
	  - Fullscreen: 0
	  - Mesh: 1
	  - Smaller Images: 1

Reading Calibration from file /home/pankhuri/rgbd_dataset_freiburg2_xyz/camera.txt ... found!
Input resolution: 640 480
In: 517.300000 516.500000 318.600000 255.300000 0.000000
Out: Rectify Crop
Output resolution: 640 480
finding CROP optimal new model!
initial range: x: -0.6220 - 0.6255; y: -0.4991 - 0.4374!
iteration 00001: range: x: -0.6188 - 0.6224; y: -0.4991 - 0.4374!
iteration 00002: range: x: -0.6158 - 0.6193; y: -0.4991 - 0.4374!
iteration 00003: range: x: -0.6158 - 0.6193; y: -0.4966 - 0.4352!
iteration 00004: range: x: -0.6158 - 0.6193; y: -0.4942 - 0.4331!
iteration 00005: range: x: -0.6158 - 0.6193; y: -0.4942 - 0.4331!

Rectified Kamera Matrix:
517.406       0 318.595
      0 516.592 255.281
      0       0       1

NO PHOTOMETRIC Calibration!
Reading Photometric Calibration from file 
PhotometricUndistorter: Could not open file!
set EXPOSURES to zero!
got 3669 images and 3669 timestamps and 0 exposures.!
ImageFolderReader: got 3669 files in /home/pankhuri/rgbd_dataset_freiburg2_xyz/rgb!
using pyramid levels 0 to 3. coarsest resolution: 80 x 60!
START PANGOLIN!

----DRMVSNET Initalizing fusion----

----DRMVSNET Initalizing fusion done----
DrMvsnet torch::cuda::is_vailable == true --> seems good
View Num: 7, ref index: 5
[W BinaryOps.cpp:467] Warning: floor_divide is deprecated, and will be removed in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values.
To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor'). (function operator())
Correctness:
	Depth correct     : 0, error: 0.0106286
	Confidence correct: 1, error: 0.00296125
Correctness:
	Depth correct     : 0, error: 0.0106182
	Confidence correct: 1, error: 0.00295482
Correctness:
	Depth correct     : 0, error: 0.0106182
	Confidence correct: 1, error: 0.00295482
Correctness:
	Depth correct     : 0, error: 0.0106182
	Confidence correct: 1, error: 0.00295481
Correctness:
	Depth correct     : 0, error: 0.0106182
	Confidence correct: 1, error: 0.00295481
Correctness:
	Depth correct     : 0, error: 0.0106182
	Confidence correct: 1, error: 0.00295481
Correctness:
	Depth correct     : 0, error: 0.0106182
	Confidence correct: 1, error: 0.0029548
Correctness:
	Depth correct     : 0, error: 0.0106182
	Confidence correct: 1, error: 0.00295482
Correctness:
	Depth correct     : 0, error: 0.0106182
	Confidence correct: 1, error: 0.00295482
Performance:
	CallAsync     : 6.80075 ms
	Ready         : 0 ms
	GetResult     : 116.749 ms
There has been an error. Do not use the model.
Couldn't load MVSNet successfully.INITIALIZE FROM INITIALIZER (2048 pts)!

TANDEM TIMING: ==================
3668 Frames (32.8 fps)
30.48ms per frame; 
111.79s total time; 
======================

DrFusion::SaveMeshToFile volume_->ExtractMesh (485 ms)
DrFusion::SaveMeshToFile mesh.SaveToFile (3587 ms)
Mesh Saving done!
^CCaught signal 2

links to target "CUDA::cublas" but the target was not found.

Hi，
when I run

cmake .. \
   -DCMAKE_BUILD_TYPE=Release \
   -DCMAKE_PREFIX_PATH=$TANDEM_LIBTORCH_DIR \
   -DCUDNN_LIBRARY=$TANDEM_CUDNN_LIBRARY \
   -DCUDNN_INCLUDE_PATH=$TANDEM_CUDNN_INCLUDE_PATH

there are some errors:
-- The C compiler identification is GNU 7.5.0
-- The CXX compiler identification is GNU 7.5.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Boost version: 1.65.1
-- Found the following Boost libraries:
-- system
-- thread
-- chrono
-- date_time
-- atomic
-- Found CUDA: /usr/local/cuda (found version "11.1")
-- Caffe2: CUDA detected: 11.1
-- Caffe2: CUDA nvcc is: /usr/local/cuda/bin/nvcc
-- Caffe2: CUDA toolkit directory: /usr/local/cuda
-- Caffe2: Header version is: 11.1
-- Found CUDNN: /usr/local/cuda/lib64/libcudnn.so
-- Found cuDNN: v8.2.1 (include: /usr/local/cuda/include, library: /usr/local/cuda/lib64/libcudnn.so)
-- /usr/local/cuda/lib64/libnvrtc.so shorthash is 3a20f2b6
-- Autodetected CUDA architecture(s): 8.6
-- Added CUDA NVCC flags for: -gencode;arch=compute_86,code=sm_86
-- Found Torch: /home/libtorch-cxx11-abi-shared-with-deps-1.9.0+cu111/libtorch/lib/libtorch.so
-- Found OpenCV: /usr/local (found version "4.5.4")
-- Boost version: 1.65.1
-- Found the following Boost libraries:
-- system
-- thread
-- chrono
-- date_time
-- atomic
-- The CUDA compiler identification is NVIDIA 11.1.105
-- Check for working CUDA compiler: /usr/local/cuda/bin/nvcc
-- Check for working CUDA compiler: /usr/local/cuda/bin/nvcc -- works
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - done
-- Found Eigen headers in: /usr/include/eigen3
-- Release mode (!= not Debug). Adding -O3;-lineinfo to NVCC FLAGS
--- found PANGOLIN, compiling tandem_pangolin library.
--- found OpenCV (/usr/local/include/opencv4), compiling tandem_opencv library.
--- compiling tandem_dataset.
-- TANDEM DEMO will be compiled
-- Realsense Calib Recorded Will be compiled
-- Configuring done
CMake Error at CMakeLists.txt:115 (add_library):
Target "tandem" links to target "CUDA::cublas" but the target was not
found. Perhaps a find_package() call is missing for an IMPORTED target, or
an ALIAS target is missing?

CMake Error at CMakeLists.txt:142 (add_executable):
Target "tandem_demo" links to target "CUDA::cublas" but the target was not
found. Perhaps a find_package() call is missing for an IMPORTED target, or
an ALIAS target is missing?

CMake Error at CMakeLists.txt:131 (add_executable):
Target "tandem_dataset" links to target "CUDA::cublas" but the target was
not found. Perhaps a find_package() call is missing for an IMPORTED
target, or an ALIAS target is missing?

CMake Error at libdr/cuda_coarse_tracker/CMakeLists.txt:27 (add_library):
Target "cuda_coarse_tracker" links to target "CUDA::cublas" but the target
was not found. Perhaps a find_package() call is missing for an IMPORTED
target, or an ALIAS target is missing?

CMake Error: Error required internal CMake variable not set, cmake may not be built correctly.
Missing variable is:
CMAKE_CUDA_DEVICE_LINK_EXECUTABLE
CMake Error: Error required internal CMake variable not set, cmake may not be built correctly.
Missing variable is:
CMAKE_CUDA_DEVICE_LINK_EXECUTABLE
-- Generating done

how to fix it ? Thanks!

About generating realistic trajectories for visual SLAM.

Hi,

I am interested in the way how to generate trajectories for visual SLAM in the Replica dataset.

Can you share the way you do in the Replica dataset?

Thanks!

hardware_reset() will cause devices can not be detected!

Thanks for your excellent open sources work!
I am following the calibration instructions to build realsense_calib_recorder, and I found that the hardware_reset() function in
tandem/tandem/src/realsense/realsense.cpp,line77
location
will cause the camera can not be connected to the host system. and some exception like

terminate called after throwing an instance of 'rs2::error'
  what():  No device connected

will be throwed out. Is there any big problem to comment this line?
My OS: Deepin 20.05

scannet cpp exported model

Thank you for sharing the great work!
Could you please release the exported cpp model of scannet_tandem.ckpt?

RuntimeError: Freezing is currently only implemented for modules for modules in eval mode

Thank you for sharing this great paper and excellent code.
I followed the README file in CVA-MMVSNET.When I tried to run the code to export to C++, there was a runtimeerroe.Could you please help me?
Thank you for your time.

python export_model.py --data_dir tandem_replica --out_dir cpp_exported_models --model pretrained/ablation/abl04_fewer_depth_planes.ckpt --height 480 --width 640 --view_num 7 --jit_freeze --jit_run_frozen_optimizations

/mnt/b7074bde-e07b-4bc7-8b24-a7b759401d85/Ji/tandem-master/cva_mvsnet/models/module.py:1498: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
assert (depth > 0).all(), f"Depth min = {torch.min(depth)}"
/mnt/b7074bde-e07b-4bc7-8b24-a7b759401d85/Ji/tandem-master/cva_mvsnet/models/module.py:24: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if nans.any() or infs.any():
/mnt/b7074bde-e07b-4bc7-8b24-a7b759401d85/Ji/tandem-master/cva_mvsnet/models/module.py:27: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
assert not (nans.any() or infs.any()), msg
/mnt/b7074bde-e07b-4bc7-8b24-a7b759401d85/Ji/tandem-master/cva_mvsnet/models/module.py:1561: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
assert (depth_range > 0).all(
--- jit_freeze
Traceback (most recent call last):
File "export_model.py", line 230, in
main(parser.parse_args())
File "export_model.py", line 196, in main
traced_script_module = torch.jit.freeze(traced_script_module)
File "/home/u/anaconda3/envs/tandem/lib/python3.7/site-packages/torch/jit/_freeze.py", line 92, in freeze
"Freezing is currently only implemented for modules in eval mode. "
RuntimeError: Freezing is currently only implemented for modules in eval mode. Please call .eval() on your module before freezing.

segment fault(core dumped) without any error

this is the log of the terminal:

loading data from /mnt/dataset/SLAM/TUM/TUM_dataset.zip/sequence_07/images!
loading calibration from /mnt/dataset/SLAM/TUM/TUM_dataset.zip/sequence_07/camera.txt!
Loading MVSNet from /home/nuc/Documents/SLAM/DenseReconstruct/tandem-master/tandem/exported/tandem/!
PHOTOMETRIC MODE WITHOUT CALIBRATION!

Reading Calibration from file /mnt/dataset/SLAM/TUM/TUM_dataset.zip/sequence_07/camera.txt ... found!
found ATAN camera model, building rectifier.
Creating FOV undistorter
Input resolution: 1280 1024
In: 0.535719 0.669567 0.493249 0.500409 0.897966

Found fx=0.535719, fy=0.669567, cx=0.493249, cy=0.500409.
I'm assuming this is the "relative" calibration file format,and will rescale this by image width / height to fx=685.720714, fy=685.636463, cx=630.858138, cy=511.918472.

Out: Rectify Crop
Output resolution: 640 480
finding CROP optimal new model!
initial range: x: -1.1372 - 1.1901; y: -0.8311 - 0.8292!
iteration 00001: range: x: -1.1315 - 1.1841; y: -0.8311 - 0.8292!
iteration 00002: range: x: -1.1258 - 1.1782; y: -0.8311 - 0.8292!
iteration 00003: range: x: -1.1258 - 1.1782; y: -0.8270 - 0.8251!
iteration 00004: range: x: -1.1258 - 1.1782; y: -0.8228 - 0.8209!
iteration 00005: range: x: -1.1258 - 1.1782; y: -0.8228 - 0.8209!

Rectified Kamera Matrix:
277.34 0 312.234
0 291.402 239.777
0 0 1

NO PHOTOMETRIC Calibration!
Reading Photometric Calibration from file
PhotometricUndistorter: Could not open file!
got 0 images and 0 timestamps and 0 exposures.!
ImageFolderReader: got 0 files in /mnt/dataset/SLAM/TUM/TUM_dataset.zip/sequence_07/images!
using pyramid levels 0 to 3. coarsest resolution: 80 x 60!

----DRMVSNET Initalizing fusion----

----DRMVSNET Initalizing fusion done----
DrMvsnet torch::cuda::is_vailable == true --> seems good
View Num: 7, ref index: 5
[W BinaryOps.cpp:467] Warning: floor_divide is deprecated, and will be removed in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values.
To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor'). (function operator())
Correctness:
Depth correct : 1, error: 0.000665128
Confidence correct: 1, error: 0.000441837
Correctness:
Depth correct : 1, error: 0.000665203
Confidence correct: 1, error: 0.000441824
Correctness:
Depth correct : 1, error: 0.000665201
Confidence correct: 1, error: 0.000441813
Correctness:
Depth correct : 1, error: 0.000665201
Confidence correct: 1, error: 0.000441825
Correctness:
Depth correct : 1, error: 0.000665194
Confidence correct: 1, error: 0.000441814
Correctness:
Depth correct : 1, error: 0.000665206
Confidence correct: 1, error: 0.000441811
Correctness:
Depth correct : 1, error: 0.000665201
Confidence correct: 1, error: 0.000441811
Correctness:
Depth correct : 1, error: 0.000665191
Confidence correct: 1, error: 0.000441814
Correctness:
Depth correct : 1, error: 0.000665207
Confidence correct: 1, error: 0.00044181
Performance:
CallAsync : 15.0173 ms
Ready : 0 ms
GetResult : 224.087 ms
All looks good!
Segmentation fault (core dumped)

ROS TANDEM

Team,

Thanks for the great and exciting work!

All,

I want to run Tandem live on external cameras like dso_ros.

what should I do？

Thanks!

How "Abs" is calculated in Table2?

Hi, I do not understand how's "Abs" defined in table 2. I guess it is the mean value of differences between predicted depth map and ground truth( at each pixel)? am I right?

Couldn't load MVSNet successfully.INITIALIZE FROM INITIALIZER

Hello,

I get this in logs while running tandem for any dataset,

(tandem) pankhuri@pankhuri-GE76-Raider-11UH:~/tandem/tandem$ ./build/bin/tandem_dataset preset=gui result_folder=/home/pankhuri/tandem-dataset/results files=/home/pankhuri/tandem-dataset/euroc_tandem_format/V1_01_easy/images calib=/home/pankhuri/tandem-dataset/euroc_tandem_format/V1_01_easy/camera.txt mvsnet_folder=/home/pankhuri/tandem/tandem/exported/tandem mode=1
loading data from /home/pankhuri/tandem-dataset/euroc_tandem_format/V1_01_easy/images!
loading calibration from /home/pankhuri/tandem-dataset/euroc_tandem_format/V1_01_easy/camera.txt!
Loading MVSNet from /home/pankhuri/tandem/tandem/exported/tandem/!
PHOTOMETRIC MODE WITHOUT CALIBRATION!

=============== TANDEM Settings: ===============
	Setting 'gui':
	- no real-time enforcing
	- 2000 active points
	- 5-7 active frames
	- 1-6 LM iteration each KF
	- TSDF fusion: yes
	- dense tracking on cpu (step=1)
	- Pangolin
	  - Fullscreen: 0
	  - Mesh: 1
	  - Smaller Images: 1

Reading Calibration from file /home/pankhuri/tandem-dataset/euroc_tandem_format/V1_01_easy/camera.txt ... found!
Input resolution: 640 480
In: 436.225021 436.225021 310.162763 256.946041 0.000000
Out: Rectify Crop
Output resolution: 640 480
finding CROP optimal new model!
initial range: x: -0.7181 - 0.7613; y: -0.5949 - 0.5141!
iteration 00001: range: x: -0.7145 - 0.7575; y: -0.5949 - 0.5141!
iteration 00002: range: x: -0.7109 - 0.7537; y: -0.5949 - 0.5141!
iteration 00003: range: x: -0.7109 - 0.7537; y: -0.5919 - 0.5115!
iteration 00004: range: x: -0.7109 - 0.7537; y: -0.5890 - 0.5090!
iteration 00005: range: x: -0.7109 - 0.7537; y: -0.5890 - 0.5090!

Rectified Kamera Matrix:
 436.27       0 310.165
      0  436.28  256.95
      0       0       1

NO PHOTOMETRIC Calibration!
Reading Photometric Calibration from file 
PhotometricUndistorter: Could not open file!
set timestamps and exposures to zero!
set EXPOSURES to zero!
got 2778 images and 0 timestamps and 0 exposures.!
ImageFolderReader: got 2778 files in /home/pankhuri/tandem-dataset/euroc_tandem_format/V1_01_easy/images!
using pyramid levels 0 to 3. coarsest resolution: 80 x 60!
START PANGOLIN!

----DRMVSNET Initalizing fusion----

----DRMVSNET Initalizing fusion done----
DrMvsnet torch::cuda::is_vailable == true --> seems good
View Num: 7, ref index: 5
[W BinaryOps.cpp:467] Warning: floor_divide is deprecated, and will be removed in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values.
To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor'). (function operator())
Correctness:
	Depth correct     : 0, error: 0.0106286
	Confidence correct: 1, error: 0.00296125
Correctness:
	Depth correct     : 0, error: 0.0106182
	Confidence correct: 1, error: 0.00295481
Correctness:
	Depth correct     : 0, error: 0.0106182
	Confidence correct: 1, error: 0.0029548
Correctness:
	Depth correct     : 0, error: 0.0106182
	Confidence correct: 1, error: 0.00295482
Correctness:
	Depth correct     : 0, error: 0.0106182
	Confidence correct: 1, error: 0.00295482
Correctness:
	Depth correct     : 0, error: 0.0106182
	Confidence correct: 1, error: 0.00295482
Correctness:
	Depth correct     : 0, error: 0.0106182
	Confidence correct: 1, error: 0.00295482
Correctness:
	Depth correct     : 0, error: 0.0106182
	Confidence correct: 1, error: 0.0029548
Correctness:
	Depth correct     : 0, error: 0.0106182
	Confidence correct: 1, error: 0.00295482
Performance:
	CallAsync     : 6.655 ms
	Ready         : 0 ms
	GetResult     : 116.791 ms
There has been an error. Do not use the model.
Couldn't load MVSNet successfully.INITIALIZE FROM INITIALIZER (2043 pts)!

TANDEM TIMING: ==================
2777 Frames (11.8 fps)
84.59ms per frame; 
234.91s total time; 
======================

DrFusion::SaveMeshToFile volume_->ExtractMesh (482 ms)
DrFusion::SaveMeshToFile mesh.SaveToFile (17763 ms)
Mesh Saving done!

I can see the 3D reconstruction but this error still exists.

I wish to ask:

Is anyone else also getting the same error? How do we solve this ?
Is it related to the warning that floor_divide is deprecated?
I am not getting good results on other custom datasets with tandem, Can this error affect the 3D reconstruction on a custom synthetic dataset?

Thank you,

Cudatoolkit couldn't find

Hi,

I installed CUDAToolkit 11.1.0 as per the nvidia official website. When I ran nvcc -V, the terminal displayed the text in the first

image. I believe It means the installation is complete. However, when I ran 'cmake ..', an error happened(the second image).

Can anyone help me solve this problem?

Thank you!

about the playbackSpeed

Hi, this's a great work!
I am trying to run the command:
build/bin/tandem_dataset
preset=gui
result_folder=./results
files=euroc_tandem_format/V1_01_easy/images
calib=euroc_tandem_format/V1_01_easy/camera.txt
mvsnet_folder=exported/tandem
mode=1
playbackSpeed=1

The speed is faster than the program that PlaybackSpeed is set to 0. But it doesn't work very well.
The error like this:
BIG ERROR! tracking failed entirely. Take predictred pose and hope we may somehow recover.
Initial Tracking failed: LOST!
LOST!!

TypeError when train CVA-MVSNet

GPU available: True, used: True
INFO:lightning:GPU available: True, used: True
No environment variable for node rank defined. Set as 0.
WARNING:lightning:No environment variable for node rank defined. Set as 0.
CUDA_VISIBLE_DEVICES: [0]
INFO:lightning:CUDA_VISIBLE_DEVICES: [0]
Validation sanity check: 0it [00:00, ?it/s]/home/hyx/anaconda3/envs/tandem/lib/python3.7/site-packages/torch/functional.py:568: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:2228.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
Epoch 1: 0%| | 0/2714 [00:00<?, ?it/s]Traceback (most recent call last):
File "train.py", line 126, in
main(*parse_args(parser))
File "train.py", line 122, in main
trainer.fit(model)
File "/home/hyx/anaconda3/envs/tandem/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 859, in fit
self.single_gpu_train(model)
File "/home/hyx/anaconda3/envs/tandem/lib/python3.7/site-packages/pytorch_lightning/trainer/distrib_parts.py", line 503, in single_gpu_train
self.run_pretrain_routine(model)
File "/home/hyx/anaconda3/envs/tandem/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1015, in run_pretrain_routine
self.train()
File "/home/hyx/anaconda3/envs/tandem/lib/python3.7/site-packages/pytorch_lightning/trainer/training_loop.py", line 347, in train
self.run_training_epoch()
File "/home/hyx/anaconda3/envs/tandem/lib/python3.7/site-packages/pytorch_lightning/trainer/training_loop.py", line 406, in run_training_epoch
enumerate(_with_is_last(train_dataloader)), "get_train_batch"
File "/home/hyx/anaconda3/envs/tandem/lib/python3.7/site-packages/pytorch_lightning/profiler/profilers.py", line 64, in profile_iterable
value = next(iterator)
File "/home/hyx/anaconda3/envs/tandem/lib/python3.7/site-packages/pytorch_lightning/trainer/training_loop.py", line 801, in _with_is_last
last = next(it)
File "/home/hyx/anaconda3/envs/tandem/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 530, in next
data = self._next_data()
File "/home/hyx/anaconda3/envs/tandem/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1224, in _next_data
return self._process_data(data)
File "/home/hyx/anaconda3/envs/tandem/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1250, in _process_data
data.reraise()
File "/home/hyx/anaconda3/envs/tandem/lib/python3.7/site-packages/torch/_utils.py", line 457, in reraise
raise exception
TypeError: Caught TypeError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/home/hyx/anaconda3/envs/tandem/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
data = fetcher.fetch(index)
File "/home/hyx/anaconda3/envs/tandem/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/hyx/anaconda3/envs/tandem/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 49, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/hyx/tandem/cva_mvsnet/models/datasets.py", line 583, in getitem
item = self.dataset.getitem(idx)
File "/home/hyx/tandem/cva_mvsnet/models/datasets.py", line 562, in getitem
data = self.transform(data)
File "/home/hyx/tandem/cva_mvsnet/models/datasets.py", line 97, in preprocess
img_aug = color_trans(img_ori)
File "/home/hyx/anaconda3/envs/tandem/lib/python3.7/site-packages/torchvision/transforms/transforms.py", line 95, in call
img = t(img)
TypeError: 'tuple' object is not callable

tum-vision / tandem Goto Github PK

tandem's Introduction

TANDEM: Tracking and Dense Mappingin Real-time using Deep Multi-view Stereo

Code and Data

Abstract

Poster

tandem's People

Contributors

Stargazers

Watchers

Forkers

tandem's Issues

when i run make -j i got :

Recommend Projects

Recommend Topics

Recommend Org

TANDEM: Tracking and Dense Mapping
in Real-time using Deep Multi-view Stereo