princeton-vl / dpvo Goto Github PK

View Code? Open in Web Editor NEW

458.0 458.0 56.0 1.79 MB

Deep Patch Visual Odometry

License: MIT License

CMake 5.95% Python 19.09% C++ 69.31% Cuda 5.21% C 0.29% Shell 0.15%

dpvo's People

Contributors

Stargazers

Watchers

dpvo's Issues

Points not visible in DPViewer

Hello,

Thanks for your excellent work, and for making it open source! I am able to build the repository and run the examples, however I notice that the DPViewer visualizer does not show any 3D points, which I expect to see, since 3D points are visible in the Youtube videos and other demo videos.

what part of the tartan dataset being used for training?

Hi! Thanks for the great work!
wondering on what part of the TartanAir dataset the pre-trained model was trained? and what part of the dataset was used to get the result in the paper? Thanks!

Running in c++?

Has anyone experimented with porting this code to c++ to fit into other slam systems?

metric depth estimation

Hi, Could the model be fine tune to pose with metric depth estimated?

Run demo.py, while only loops twice and ends

The result was that the file was not saved and there were no error messages. What is the reason?
Tracked to, second loop, \dpvo\dpvo.py: if self.n>0 and not self.is_ Initialized: Print out: n=1, is_ Initialized=False, motion_probe()=NULL, The return has ended.
@lahavlipson

Cannot use dpviewer on Jetson AGX Orin

Hi, thanks for your great work!
I run demo.py on Orin with Ubuntu20.04 and met a error on dpviewer. Here is the output:

$ python demo.py --imagedir=movies/IMG_0492.MOV --calib=calib/iphone.txt --plot --save_reconstruction --save_trajectory --viz
Running with config...
BUFFER_SIZE: 2048
GRADIENT_BIAS: False
KEYFRAME_INDEX: 4
KEYFRAME_THRESH: 15.0
MIXED_PRECISION: True
MOTION_DAMPING: 0.5
MOTION_MODEL: DAMPED_LINEAR
OPTIMIZATION_WINDOW: 10
PATCHES_PER_FRAME: 96
PATCH_LIFETIME: 13
REMOVAL_WINDOW: 22
Traceback (most recent call last):
File "demo.py", line 93, in
pred_traj = run(cfg, args.network, args.imagedir, args.calib, args.stride, args.skip, args.viz, args.timeit, args.save_reconstruction)
File "/home/mi/miniforge3/envs/hloc/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "demo.py", line 45, in run
slam = DPVO(cfg, network, ht=image.shape[1], wd=image.shape[2], viz=viz)
File "/home/mi/dev/DPVO/dpvo/dpvo.py", line 87, in init
self.start_viewer()
File "/home/mi/dev/DPVO/dpvo/dpvo.py", line 118, in start_viewer
from dpviewer import Viewer
File "/home/mi/miniforge3/envs/hloc/lib/python3.8/site-packages/dpviewer/init.py", line 1, in
from dpviewerx import Viewer
ImportError: /home/mi/miniforge3/envs/hloc/lib/python3.8/site-packages/dpviewerx.cpython-38-aarch64-linux-gnu.so: undefined symbol: _ZN8pangolin19CreateWindowAndBindENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEiiRKNS_6ParamsE

Could you help me? Thanks!

Is running in CPU only possible?

Hello guys, does anyone make it as the title. I have no GPU to work with CUDA, so that is it possible to run with CPU only?

Does it still require the intrinsic parameter of the camera and

Hello Lahav,

I have two questions:

the same as the title.
What shoud I do with your approach if I simply want to train a model for pose estimation without others of the pipe line such as correlation tracking, etc.

What is the purpose of scaling the input image to -0.5 to 1.5

image = 2 * (image[None,None] / 255.0) - 0.5

Trouble with installing DPViewer

ERROR ❌ installing DPViewer

Hi,
After following the procedure up to the point in which we have to install dpviewer
pip3 install ./DPViewer
I fall in the following error...

seems to start having troubles with bdist_wheel that are correctly installed like that:

pip install wheel
python setup.py bdist_wheel

Do you have any clues to why is not installing ? is there a workaround to get the viewer anyways?

`Processing ./DPViewer
Preparing metadata (setup.py) ... done
Building wheels for collected packages: dpviewer
Building wheel for dpviewer (setup.py) ... error
error: subprocess-exited-with-error

× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> [116 lines of output]
running bdist_wheel
running build
running build_py
running build_ext
-- Caffe2: CUDA detected: 11.1
-- Caffe2: CUDA nvcc is: /usr/local/cuda-11.1/bin/nvcc
-- Caffe2: CUDA toolkit directory: /usr/local/cuda-11.1
-- Caffe2: Header version is: 11.1
-- Found cuDNN: v7.6.5 (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libcudnn.so)
-- /usr/local/cuda-11.1/lib64/libnvrtc.so shorthash is 3a20f2b6
-- Autodetected CUDA architecture(s): 8.6 7.5 7.5 7.5 7.5 7.5 7.5
-- Added CUDA NVCC flags for: -gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_75,code=sm_75
-- MKL_ARCH: intel64
-- MKL_ROOT /opt/intel/oneapi/mkl/2022.0.2
-- MKL_LINK: dynamic
-- MKL_INTERFACE_FULL: intel_ilp64
-- MKL_THREADING: intel_thread
-- MKL_MPI: intelmpi
CMake Warning at /home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:22 (message):
static library kineto_LIBRARY-NOTFOUND not found.
Call Stack (most recent call first):
/home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:127 (append_torchlib_if_found)
CMakeLists.txt:24 (find_package)

  -- pybind11 v2.10.0 dev1
  -- Configuring done
  CMake Warning at pybind11/tools/pybind11Tools.cmake:173 (add_library):
    Cannot generate a safe runtime search path for target dpviewerx because
    files in some directories may conflict with libraries in implicit
    directories:
  
      runtime library [libnvToolsExt.so.1] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
        /usr/local/cuda-11.1/lib64
  
    Some of these libraries may not be found correctly.
  Call Stack (most recent call first):
    dpviewer/CMakeLists.txt:8 (pybind11_add_module)
  
  
  -- Generating done
  -- Build files have been written to: /home/pellerito/DPVO/DPViewer/build/temp.linux-x86_64-cpython-310
  /home/pellerito/anaconda3/envs/dpvo/bin/cmake -S/home/pellerito/DPVO/DPViewer -B/home/pellerito/DPVO/DPViewer/build/temp.linux-x86_64-cpython-310 --check-build-system CMakeFiles/Makefile.cmake 0
  /home/pellerito/anaconda3/envs/dpvo/bin/cmake -E cmake_progress_start /home/pellerito/DPVO/DPViewer/build/temp.linux-x86_64-cpython-310/CMakeFiles /home/pellerito/DPVO/DPViewer/build/temp.linux-x86_64-cpython-310//CMakeFiles/progress.marks
  /usr/bin/make  -f CMakeFiles/Makefile2 all
  make[1]: Entering directory '/home/pellerito/DPVO/DPViewer/build/temp.linux-x86_64-cpython-310'
  /usr/bin/make  -f dpviewer/CMakeFiles/dpviewerx.dir/build.make dpviewer/CMakeFiles/dpviewerx.dir/depend
  make[2]: Entering directory '/home/pellerito/DPVO/DPViewer/build/temp.linux-x86_64-cpython-310'
  cd /home/pellerito/DPVO/DPViewer/build/temp.linux-x86_64-cpython-310 && /home/pellerito/anaconda3/envs/dpvo/bin/cmake -E cmake_depends "Unix Makefiles" /home/pellerito/DPVO/DPViewer /home/pellerito/DPVO/DPViewer/dpviewer /home/pellerito/DPVO/DPViewer/build/temp.linux-x86_64-cpython-310 /home/pellerito/DPVO/DPViewer/build/temp.linux-x86_64-cpython-310/dpviewer /home/pellerito/DPVO/DPViewer/build/temp.linux-x86_64-cpython-310/dpviewer/CMakeFiles/dpviewerx.dir/DependInfo.cmake --color=
  Dependencies file "dpviewer/CMakeFiles/dpviewerx.dir/viewer_cuda.cu.o.d" is newer than depends file "/home/pellerito/DPVO/DPViewer/build/temp.linux-x86_64-cpython-310/dpviewer/CMakeFiles/dpviewerx.dir/compiler_depend.internal".
  Dependencies file "dpviewer/CMakeFiles/dpviewerx.dir/viewer.cpp.o.d" is newer than depends file "/home/pellerito/DPVO/DPViewer/build/temp.linux-x86_64-cpython-310/dpviewer/CMakeFiles/dpviewerx.dir/compiler_depend.internal".
  Consolidate compiler generated dependencies of target dpviewerx
  make[2]: Leaving directory '/home/pellerito/DPVO/DPViewer/build/temp.linux-x86_64-cpython-310'
  /usr/bin/make  -f dpviewer/CMakeFiles/dpviewerx.dir/build.make dpviewer/CMakeFiles/dpviewerx.dir/build
  make[2]: Entering directory '/home/pellerito/DPVO/DPViewer/build/temp.linux-x86_64-cpython-310'
  [ 33%] Linking CXX shared module ../../lib.linux-x86_64-cpython-310/dpviewerx.cpython-310-x86_64-linux-gnu.so
  cd /home/pellerito/DPVO/DPViewer/build/temp.linux-x86_64-cpython-310/dpviewer && /home/pellerito/anaconda3/envs/dpvo/bin/cmake -E cmake_link_script CMakeFiles/dpviewerx.dir/link.txt --verbose=1
  /usr/bin/c++ -fPIC -O3 -DNDEBUG -flto -shared  -o ../../lib.linux-x86_64-cpython-310/dpviewerx.cpython-310-x86_64-linux-gnu.so CMakeFiles/dpviewerx.dir/viewer.cpp.o CMakeFiles/dpviewerx.dir/viewer_cuda.cu.o   -L/usr/local/cuda-11.1/targets/x86_64-linux/lib/stubs  -L/usr/local/cuda-11.1/targets/x86_64-linux/lib  -Wl,-rpath,/home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/torch/lib:/usr/local/cuda-11.1/lib64/stubs:/usr/local/cuda-11.1/lib64:/home/pellerito/DPVO/Pangolin/build /home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/torch/lib/libtorch.so /home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/torch/lib/libc10.so /usr/local/cuda-11.1/lib64/stubs/libcuda.so /usr/local/cuda-11.1/lib64/libnvrtc.so /usr/lib/x86_64-linux-gnu/libnvToolsExt.so /usr/local/cuda-11.1/lib64/libcudart.so /home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/torch/lib/libc10_cuda.so /home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/torch/lib/libtorch_python.so /home/pellerito/DPVO/Pangolin/build/libpango_glgeometry.so /home/pellerito/DPVO/Pangolin/build/libpango_python.so /home/pellerito/DPVO/Pangolin/build/libpango_scene.so /home/pellerito/DPVO/Pangolin/build/libpango_tools.so /home/pellerito/DPVO/Pangolin/build/libpango_video.so -Wl,--no-as-needed,"/home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so" -Wl,--as-needed -Wl,--no-as-needed,"/home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cpp.so" -Wl,--as-needed -Wl,--no-as-needed,"/home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so" -Wl,--as-needed -lmkl_intel_ilp64 -lmkl_core -lmkl_intel_thread /home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/torch/lib/libc10_cuda.so /home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/torch/lib/libc10.so /usr/local/cuda-11.1/lib64/libcufft.so /usr/local/cuda-11.1/lib64/libcurand.so /usr/local/cuda-11.1/lib64/libcublas.so /usr/lib/x86_64-linux-gnu/libcudnn.so -Wl,--no-as-needed,"/home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cu.so" -Wl,--as-needed -Wl,--no-as-needed,"/home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/torch/lib/libtorch.so" -Wl,--as-needed /usr/lib/x86_64-linux-gnu/libnvToolsExt.so /usr/local/cuda-11.1/lib64/libcudart.so /home/pellerito/DPVO/Pangolin/build/libpango_geometry.so /home/pellerito/DPVO/Pangolin/build/libtinyobj.so /home/pellerito/DPVO/Pangolin/build/libpango_plot.so /home/pellerito/DPVO/Pangolin/build/libpango_display.so /home/pellerito/DPVO/Pangolin/build/libpango_vars.so /home/pellerito/DPVO/Pangolin/build/libpango_windowing.so /home/pellerito/DPVO/Pangolin/build/libpango_opengl.so /usr/lib/x86_64-linux-gnu/libGLEW.so /usr/lib/x86_64-linux-gnu/libOpenGL.so /usr/lib/x86_64-linux-gnu/libGLX.so /usr/lib/x86_64-linux-gnu/libGLU.so /home/pellerito/DPVO/Pangolin/build/libpango_image.so /home/pellerito/DPVO/Pangolin/build/libpango_packetstream.so /home/pellerito/DPVO/Pangolin/build/libpango_core.so -lrt -lpthread -lcudadevrt -lcudart_static -lrt -lpthread -ldl
  /usr/bin/ld: cannot find -lmkl_intel_ilp64
  /usr/bin/ld: cannot find -lmkl_core
  /usr/bin/ld: cannot find -lmkl_intel_thread
  collect2: error: ld returned 1 exit status
  dpviewer/CMakeFiles/dpviewerx.dir/build.make:147: recipe for target '../lib.linux-x86_64-cpython-310/dpviewerx.cpython-310-x86_64-linux-gnu.so' failed
  make[2]: *** [../lib.linux-x86_64-cpython-310/dpviewerx.cpython-310-x86_64-linux-gnu.so] Error 1
  make[2]: Leaving directory '/home/pellerito/DPVO/DPViewer/build/temp.linux-x86_64-cpython-310'
  CMakeFiles/Makefile2:115: recipe for target 'dpviewer/CMakeFiles/dpviewerx.dir/all' failed
  make[1]: *** [dpviewer/CMakeFiles/dpviewerx.dir/all] Error 2
  make[1]: Leaving directory '/home/pellerito/DPVO/DPViewer/build/temp.linux-x86_64-cpython-310'
  Makefile:90: recipe for target 'all' failed
  make: *** [all] Error 2
  Traceback (most recent call last):
    File "<string>", line 2, in <module>
    File "<pip-setuptools-caller>", line 34, in <module>
    File "/home/pellerito/DPVO/DPViewer/setup.py", line 123, in <module>
      setup(
    File "/home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/setuptools/__init__.py", line 87, in setup
      return distutils.core.setup(**attrs)
    File "/home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 185, in setup
      return run_commands(dist)
    File "/home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
      dist.run_commands()
    File "/home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
      self.run_command(cmd)
    File "/home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/setuptools/dist.py", line 1208, in run_command
      super().run_command(command)
    File "/home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/wheel/bdist_wheel.py", line 299, in run
      self.run_command('build')
    File "/home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
      self.distribution.run_command(command)
    File "/home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/setuptools/dist.py", line 1208, in run_command
      super().run_command(command)
    File "/home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/setuptools/_distutils/command/build.py", line 132, in run
      self.run_command(cmd_name)
    File "/home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
      self.distribution.run_command(command)
    File "/home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/setuptools/dist.py", line 1208, in run_command
      super().run_command(command)
    File "/home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/setuptools/command/build_ext.py", line 84, in run
      _build_ext.run(self)
    File "/home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 346, in run
      self.build_extensions()
    File "/home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 468, in build_extensions
      self._build_extensions_serial()
    File "/home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 494, in _build_extensions_serial
      self.build_extension(ext)
    File "/home/pellerito/DPVO/DPViewer/setup.py", line 116, in build_extension
      subprocess.check_call(
    File "/home/pellerito/anaconda3/envs/dpvo/lib/python3.10/subprocess.py", line 369, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--verbose', '-j']' returned non-zero exit status 2.
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for dpviewer
Running setup.py clean for dpviewer
Failed to build dpviewer
Installing collected packages: dpviewer
Running setup.py install for dpviewer ... error
error: subprocess-exited-with-error

× Running setup.py install for dpviewer did not run successfully.
│ exit code: 1
╰─> [158 lines of output]
running install
/home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
running build
running build_py
creating build/lib.linux-x86_64-cpython-310
creating build/lib.linux-x86_64-cpython-310/dpviewer
copying dpviewer/init.py -> build/lib.linux-x86_64-cpython-310/dpviewer
running build_ext
-- The C compiler identification is GNU 9.4.0
-- The CXX compiler identification is GNU 9.4.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- The CUDA compiler identification is NVIDIA 11.1.105
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - done
-- Check for working CUDA compiler: /usr/local/cuda-11.1/bin/nvcc - skipped
-- Detecting CUDA compile features
-- Detecting CUDA compile features - done
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Found CUDA: /usr/local/cuda-11.1 (found version "11.1")
-- Caffe2: CUDA detected: 11.1
-- Caffe2: CUDA nvcc is: /usr/local/cuda-11.1/bin/nvcc
-- Caffe2: CUDA toolkit directory: /usr/local/cuda-11.1
-- Caffe2: Header version is: 11.1
-- Found CUDNN: /usr/lib/x86_64-linux-gnu/libcudnn.so
-- Found cuDNN: v7.6.5 (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libcudnn.so)
-- /usr/local/cuda-11.1/lib64/libnvrtc.so shorthash is 3a20f2b6
-- Autodetected CUDA architecture(s): 8.6 7.5 7.5 7.5 7.5 7.5 7.5
-- Added CUDA NVCC flags for: -gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_75,code=sm_75
-- MKL_ARCH: None, set to intel64 by default
-- MKL_ROOT /opt/intel/oneapi/mkl/2022.0.2
-- MKL_LINK: None, set to dynamic by default
-- MKL_INTERFACE_FULL: None, set to intel_ilp64 by default
-- MKL_THREADING: None, set to intel_thread by default
-- MKL_MPI: None, set to intelmpi by default
CMake Warning at /home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:22 (message):
static library kineto_LIBRARY-NOTFOUND not found.
Call Stack (most recent call first):
/home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:127 (append_torchlib_if_found)
CMakeLists.txt:24 (find_package)

  -- Found Torch: /home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/torch/lib/libtorch.so
  -- pybind11 v2.10.0 dev1
  -- Found PythonInterp: /home/pellerito/anaconda3/envs/dpvo/bin/python (found suitable version "3.10", minimum required is "3.6")
  -- Found PythonLibs: /home/pellerito/anaconda3/envs/dpvo/lib/libpython3.10.so
  -- Performing Test HAS_FLTO
  -- Performing Test HAS_FLTO - Success
  -- Configuring done
  CMake Warning at pybind11/tools/pybind11Tools.cmake:173 (add_library):
    Cannot generate a safe runtime search path for target dpviewerx because
    files in some directories may conflict with libraries in implicit
    directories:
  
      runtime library [libnvToolsExt.so.1] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
        /usr/local/cuda-11.1/lib64
  
    Some of these libraries may not be found correctly.
  Call Stack (most recent call first):
    dpviewer/CMakeLists.txt:8 (pybind11_add_module)
  
  
  -- Generating done
  -- Build files have been written to: /home/pellerito/DPVO/DPViewer/build/temp.linux-x86_64-cpython-310
  /home/pellerito/anaconda3/envs/dpvo/bin/cmake -S/home/pellerito/DPVO/DPViewer -B/home/pellerito/DPVO/DPViewer/build/temp.linux-x86_64-cpython-310 --check-build-system CMakeFiles/Makefile.cmake 0
  /home/pellerito/anaconda3/envs/dpvo/bin/cmake -E cmake_progress_start /home/pellerito/DPVO/DPViewer/build/temp.linux-x86_64-cpython-310/CMakeFiles /home/pellerito/DPVO/DPViewer/build/temp.linux-x86_64-cpython-310//CMakeFiles/progress.marks
  /usr/bin/make  -f CMakeFiles/Makefile2 all
  make[1]: Entering directory '/home/pellerito/DPVO/DPViewer/build/temp.linux-x86_64-cpython-310'
  /usr/bin/make  -f dpviewer/CMakeFiles/dpviewerx.dir/build.make dpviewer/CMakeFiles/dpviewerx.dir/depend
  make[2]: Entering directory '/home/pellerito/DPVO/DPViewer/build/temp.linux-x86_64-cpython-310'
  cd /home/pellerito/DPVO/DPViewer/build/temp.linux-x86_64-cpython-310 && /home/pellerito/anaconda3/envs/dpvo/bin/cmake -E cmake_depends "Unix Makefiles" /home/pellerito/DPVO/DPViewer /home/pellerito/DPVO/DPViewer/dpviewer /home/pellerito/DPVO/DPViewer/build/temp.linux-x86_64-cpython-310 /home/pellerito/DPVO/DPViewer/build/temp.linux-x86_64-cpython-310/dpviewer /home/pellerito/DPVO/DPViewer/build/temp.linux-x86_64-cpython-310/dpviewer/CMakeFiles/dpviewerx.dir/DependInfo.cmake --color=
  make[2]: Leaving directory '/home/pellerito/DPVO/DPViewer/build/temp.linux-x86_64-cpython-310'
  /usr/bin/make  -f dpviewer/CMakeFiles/dpviewerx.dir/build.make dpviewer/CMakeFiles/dpviewerx.dir/build
  make[2]: Entering directory '/home/pellerito/DPVO/DPViewer/build/temp.linux-x86_64-cpython-310'
  [ 33%] Building CUDA object dpviewer/CMakeFiles/dpviewerx.dir/viewer_cuda.cu.o
  [ 66%] Building CXX object dpviewer/CMakeFiles/dpviewerx.dir/viewer.cpp.o
  cd /home/pellerito/DPVO/DPViewer/build/temp.linux-x86_64-cpython-310/dpviewer && /usr/local/cuda-11.1/bin/nvcc -forward-unknown-to-host-compiler -DHAVE_EIGEN -DHAVE_GLEW -DHAVE_PYTHON -DPANGO_DEFAULT_WIN_URI=\"wayland\" -DUSE_C10D_GLOO -DUSE_C10D_NCCL -DUSE_DISTRIBUTED -DUSE_RPC -DUSE_TENSORPIPE -DVERSION_INFO=0.0.1 -D_GLIBCXX_USE_CXX11_ABI=0 -D_LINUX_ -Ddpviewerx_EXPORTS -isystem=/home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/torch/include -isystem=/home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem=/home/pellerito/DPVO/DPViewer/pybind11/include -isystem=/home/pellerito/anaconda3/envs/dpvo/include/python3.10 -isystem=/usr/local/cuda-11.1/include -isystem=/home/pellerito/DPVO/Pangolin/components/pango_core/include -isystem=/home/pellerito/DPVO/Pangolin/components/pango_display/include -isystem=/home/pellerito/DPVO/Pangolin/components/pango_opengl/include -isystem=/home/pellerito/DPVO/Pangolin/components/pango_image/include -isystem=/usr/include/eigen3 -isystem=/home/pellerito/DPVO/Pangolin/components/pango_windowing/include -isystem=/home/pellerito/DPVO/Pangolin/build/include -isystem=/home/pellerito/DPVO/Pangolin/components/pango_vars/include -isystem=/home/pellerito/DPVO/Pangolin/components/pango_geometry/include -isystem=/home/pellerito/DPVO/Pangolin/components/tinyobj/include -isystem=/home/pellerito/DPVO/Pangolin/components/pango_glgeometry/include -isystem=/home/pellerito/DPVO/Pangolin/components/pango_packetstream/include -isystem=/home/pellerito/DPVO/Pangolin/components/pango_plot/include -isystem=/home/pellerito/DPVO/Pangolin/components/pango_python/include -isystem=/home/pellerito/DPVO/Pangolin/components/pango_video/include -isystem=/home/pellerito/DPVO/Pangolin/components/pango_scene/include -isystem=/home/pellerito/DPVO/Pangolin/components/pango_tools/include -DONNX_NAMESPACE=onnx_c2 -gencode arch=compute_86,code=sm_86 -gencode arch=compute_75,code=sm_75 -Xcudafe --diag_suppress=cc_clobber_ignored,--diag_suppress=integer_sign_change,--diag_suppress=useless_using_declaration,--diag_suppress=set_but_not_used,--diag_suppress=field_without_dll_interface,--diag_suppress=base_class_has_different_dll_interface,--diag_suppress=dll_interface_conflict_none_assumed,--diag_suppress=dll_interface_conflict_dllexport_assumed,--diag_suppress=implicit_return_from_non_void_function,--diag_suppress=unsigned_compare_with_zero,--diag_suppress=declared_but_not_referenced,--diag_suppress=bad_friend_decl --expt-relaxed-constexpr --expt-extended-lambda -O3 -DNDEBUG -Xcompiler=-fPIC -Xcompiler=-fvisibility=hidden -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++17 -MD -MT dpviewer/CMakeFiles/dpviewerx.dir/viewer_cuda.cu.o -MF CMakeFiles/dpviewerx.dir/viewer_cuda.cu.o.d -x cu -c /home/pellerito/DPVO/DPViewer/dpviewer/viewer_cuda.cu -o CMakeFiles/dpviewerx.dir/viewer_cuda.cu.o
  cd /home/pellerito/DPVO/DPViewer/build/temp.linux-x86_64-cpython-310/dpviewer && /usr/bin/c++ -DHAVE_EIGEN -DHAVE_GLEW -DHAVE_PYTHON -DPANGO_DEFAULT_WIN_URI=\"wayland\" -DUSE_C10D_GLOO -DUSE_C10D_NCCL -DUSE_DISTRIBUTED -DUSE_RPC -DUSE_TENSORPIPE -DVERSION_INFO=0.0.1 -D_GLIBCXX_USE_CXX11_ABI=0 -D_LINUX_ -Ddpviewerx_EXPORTS -isystem /home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/torch/include -isystem /home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /home/pellerito/DPVO/DPViewer/pybind11/include -isystem /home/pellerito/anaconda3/envs/dpvo/include/python3.10 -isystem /usr/local/cuda-11.1/include -isystem /home/pellerito/DPVO/Pangolin/components/pango_core/include -isystem /home/pellerito/DPVO/Pangolin/components/pango_display/include -isystem /home/pellerito/DPVO/Pangolin/components/pango_opengl/include -isystem /home/pellerito/DPVO/Pangolin/components/pango_image/include -isystem /usr/include/eigen3 -isystem /home/pellerito/DPVO/Pangolin/components/pango_windowing/include -isystem /home/pellerito/DPVO/Pangolin/build/include -isystem /home/pellerito/DPVO/Pangolin/components/pango_vars/include -isystem /home/pellerito/DPVO/Pangolin/components/pango_geometry/include -isystem /home/pellerito/DPVO/Pangolin/components/tinyobj/include -isystem /home/pellerito/DPVO/Pangolin/components/pango_glgeometry/include -isystem /home/pellerito/DPVO/Pangolin/components/pango_packetstream/include -isystem /home/pellerito/DPVO/Pangolin/components/pango_plot/include -isystem /home/pellerito/DPVO/Pangolin/components/pango_python/include -isystem /home/pellerito/DPVO/Pangolin/components/pango_video/include -isystem /home/pellerito/DPVO/Pangolin/components/pango_scene/include -isystem /home/pellerito/DPVO/Pangolin/components/pango_tools/include -O3 -DNDEBUG -fPIC -fvisibility=hidden -flto -fno-fat-lto-objects -D_GLIBCXX_USE_CXX11_ABI=0 -std=gnu++17 -MD -MT dpviewer/CMakeFiles/dpviewerx.dir/viewer.cpp.o -MF CMakeFiles/dpviewerx.dir/viewer.cpp.o.d -o CMakeFiles/dpviewerx.dir/viewer.cpp.o -c /home/pellerito/DPVO/DPViewer/dpviewer/viewer.cpp
  [100%] Linking CXX shared module ../../lib.linux-x86_64-cpython-310/dpviewerx.cpython-310-x86_64-linux-gnu.so
  cd /home/pellerito/DPVO/DPViewer/build/temp.linux-x86_64-cpython-310/dpviewer && /home/pellerito/anaconda3/envs/dpvo/bin/cmake -E cmake_link_script CMakeFiles/dpviewerx.dir/link.txt --verbose=1
  /usr/bin/c++ -fPIC -O3 -DNDEBUG -flto -shared  -o ../../lib.linux-x86_64-cpython-310/dpviewerx.cpython-310-x86_64-linux-gnu.so CMakeFiles/dpviewerx.dir/viewer.cpp.o CMakeFiles/dpviewerx.dir/viewer_cuda.cu.o   -L/usr/local/cuda-11.1/targets/x86_64-linux/lib/stubs  -L/usr/local/cuda-11.1/targets/x86_64-linux/lib  -Wl,-rpath,/home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/torch/lib:/usr/local/cuda-11.1/lib64/stubs:/usr/local/cuda-11.1/lib64:/home/pellerito/DPVO/Pangolin/build /home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/torch/lib/libtorch.so /home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/torch/lib/libc10.so /usr/local/cuda-11.1/lib64/stubs/libcuda.so /usr/local/cuda-11.1/lib64/libnvrtc.so /usr/lib/x86_64-linux-gnu/libnvToolsExt.so /usr/local/cuda-11.1/lib64/libcudart.so /home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/torch/lib/libc10_cuda.so /home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/torch/lib/libtorch_python.so /home/pellerito/DPVO/Pangolin/build/libpango_glgeometry.so /home/pellerito/DPVO/Pangolin/build/libpango_python.so /home/pellerito/DPVO/Pangolin/build/libpango_scene.so /home/pellerito/DPVO/Pangolin/build/libpango_tools.so /home/pellerito/DPVO/Pangolin/build/libpango_video.so -Wl,--no-as-needed,"/home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so" -Wl,--as-needed -Wl,--no-as-needed,"/home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cpp.so" -Wl,--as-needed -Wl,--no-as-needed,"/home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so" -Wl,--as-needed -lmkl_intel_ilp64 -lmkl_core -lmkl_intel_thread /home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/torch/lib/libc10_cuda.so /home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/torch/lib/libc10.so /usr/local/cuda-11.1/lib64/libcufft.so /usr/local/cuda-11.1/lib64/libcurand.so /usr/local/cuda-11.1/lib64/libcublas.so /usr/lib/x86_64-linux-gnu/libcudnn.so -Wl,--no-as-needed,"/home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cu.so" -Wl,--as-needed -Wl,--no-as-needed,"/home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/torch/lib/libtorch.so" -Wl,--as-needed /usr/lib/x86_64-linux-gnu/libnvToolsExt.so /usr/local/cuda-11.1/lib64/libcudart.so /home/pellerito/DPVO/Pangolin/build/libpango_geometry.so /home/pellerito/DPVO/Pangolin/build/libtinyobj.so /home/pellerito/DPVO/Pangolin/build/libpango_plot.so /home/pellerito/DPVO/Pangolin/build/libpango_display.so /home/pellerito/DPVO/Pangolin/build/libpango_vars.so /home/pellerito/DPVO/Pangolin/build/libpango_windowing.so /home/pellerito/DPVO/Pangolin/build/libpango_opengl.so /usr/lib/x86_64-linux-gnu/libGLEW.so /usr/lib/x86_64-linux-gnu/libOpenGL.so /usr/lib/x86_64-linux-gnu/libGLX.so /usr/lib/x86_64-linux-gnu/libGLU.so /home/pellerito/DPVO/Pangolin/build/libpango_image.so /home/pellerito/DPVO/Pangolin/build/libpango_packetstream.so /home/pellerito/DPVO/Pangolin/build/libpango_core.so -lrt -lpthread -lcudadevrt -lcudart_static -lrt -lpthread -ldl
  /usr/bin/ld: cannot find -lmkl_intel_ilp64
  /usr/bin/ld: cannot find -lmkl_core
  /usr/bin/ld: cannot find -lmkl_intel_thread
  collect2: error: ld returned 1 exit status
  dpviewer/CMakeFiles/dpviewerx.dir/build.make:147: recipe for target '../lib.linux-x86_64-cpython-310/dpviewerx.cpython-310-x86_64-linux-gnu.so' failed
  make[2]: *** [../lib.linux-x86_64-cpython-310/dpviewerx.cpython-310-x86_64-linux-gnu.so] Error 1
  make[2]: Leaving directory '/home/pellerito/DPVO/DPViewer/build/temp.linux-x86_64-cpython-310'
  CMakeFiles/Makefile2:115: recipe for target 'dpviewer/CMakeFiles/dpviewerx.dir/all' failed
  make[1]: *** [dpviewer/CMakeFiles/dpviewerx.dir/all] Error 2
  make[1]: Leaving directory '/home/pellerito/DPVO/DPViewer/build/temp.linux-x86_64-cpython-310'
  Makefile:90: recipe for target 'all' failed
  make: *** [all] Error 2
  Traceback (most recent call last):
    File "<string>", line 2, in <module>
    File "<pip-setuptools-caller>", line 34, in <module>
    File "/home/pellerito/DPVO/DPViewer/setup.py", line 123, in <module>
      setup(
    File "/home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/setuptools/__init__.py", line 87, in setup
      return distutils.core.setup(**attrs)
    File "/home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 185, in setup
      return run_commands(dist)
    File "/home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
      dist.run_commands()
    File "/home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
      self.run_command(cmd)
    File "/home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/setuptools/dist.py", line 1208, in run_command
      super().run_command(command)
    File "/home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/setuptools/command/install.py", line 68, in run
      return orig.install.run(self)
    File "/home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/setuptools/_distutils/command/install.py", line 698, in run
      self.run_command('build')
    File "/home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
      self.distribution.run_command(command)
    File "/home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/setuptools/dist.py", line 1208, in run_command
      super().run_command(command)
    File "/home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/setuptools/_distutils/command/build.py", line 132, in run
      self.run_command(cmd_name)
    File "/home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
      self.distribution.run_command(command)
    File "/home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/setuptools/dist.py", line 1208, in run_command
      super().run_command(command)
    File "/home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/setuptools/command/build_ext.py", line 84, in run
      _build_ext.run(self)
    File "/home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 346, in run
      self.build_extensions()
    File "/home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 468, in build_extensions
      self._build_extensions_serial()
    File "/home/pellerito/anaconda3/envs/dpvo/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 494, in _build_extensions_serial
      self.build_extension(ext)
    File "/home/pellerito/DPVO/DPViewer/setup.py", line 116, in build_extension
      subprocess.check_call(
    File "/home/pellerito/anaconda3/envs/dpvo/lib/python3.10/subprocess.py", line 369, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--verbose', '-j']' returned non-zero exit status 2.
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> dpviewer

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.`

How to add our own training data?

Dear author:
thanks for your wonderful work. when I want to add my own data to train dpvo, however I don't known how to generate .pickle file for my own dataset? （for example TartanAir.pickle file for tartanair dataset）

Segmentation fault when run demo and some fix

OS: ubuntu 20.04LTS
Gcc and g++: 10.3.0
Conda environment: same as environment.yml
Cudnn: 8.3.2
Driver Version: 510.47.03
CUDA Version: 11.6

After installing the dependencies and downloading the model according to the README, run the README
demo command, and it run failed. Without and error message.

$ CUDA_LAUNCH_BLOCKING=1 python demo.py --imagedir=movies/IMG_0494.MOV --calib=calib/iphone.txt --stride=5 --viz
Running with config...
BUFFER_SIZE: 2048
GRADIENT_BIAS: False
KEYFRAME_INDEX: 4
KEYFRAME_THRESH: 15.0
MIXED_PRECISION: True
MOTION_DAMPING: 0.5
MOTION_MODEL: DAMPED_LINEAR
OPTIMIZATION_WINDOW: 10
PATCHES_PER_FRAME: 96
PATCH_LIFETIME: 13
REMOVAL_WINDOW: 22
[1]    310720 segmentation fault (core dumped)  CUDA_LAUNCH_BLOCKING=1 python demo.py --imagedir=movies/IMG_0494.MOV   --viz

I found some method to fix it:
in DPViewer/dpviewer/viewer.cpp

and

Then I can run without error(sometimes).
And I found the following change will be helpful, before the change, there was a certain probability of segmentation fault, after the change, no more segmentation fault. Presumably this is due to pointer initialization.

Since I commented out the code to calculate the transformMatrix, the camera will not be updated.
And I found that whenever I called the calculated code in the loop of this thread, it would definitely error out. Therefore, I create a member function specifically for calculating and updating the transformMatrix in main thread.

New update transformMatrix function (Don't forget to write the function definition and mutex definition):
Lock when render thread draw points and poses:
Call it in python:

Now, in the visualization view, the camera can render the pose normally.

But I observed that point cloud data was used, but my visualization view only had the camera pose and video, not any points.

I'm not very familiar with gl programming, so just a simple replace of the render point function.

Finally, I can now run the visualization interface properly on my computer and update the poses and point cloud dynamically!

Here is all changes: dicarne@5f17684 dicarne@677b056

Some unanswered questions: Why, even with the same environment configuration as the author claims, I was unable to run the code of this project until I modified some of the code? I changed several computers and operating systems, and tried in docker, but all failed. I'm curious what the environment configuration is like to run it directly.

Is it possible to support panoramic images?

Hi, is it possible for DPVO to support panoramic images? The camera model of panoramic image is not like pinhole camera but equirectangular camera model, see here https://wiki.panotools.org/Equirectangular_Projection

Missing Code: pops.induced_flow

Hi, when I read the code, I find that there maybe some code which may be not uploadded, such as the pops.induced_flow which is used in dpvo/data_readers/rgbd_utils.py. May I ask your help to upload these realtive codes?

Version of CMAKE

Dear author,
I think the specific version of CMAKE should be told. A higher version of CMAKE will cause problem for dpviewer installation.

Call for help. It dosent work when I run demo.py with iphone datasets.

I think I run the setup stuff well. But when I run demo, it dosent work.I realy dont know why.Call for help.

BUFFER_SIZE not large enough

Traceback (most recent call last):                                                                                                                                                                                                                                                                                                                                                  
  File "/home/relh/DPVO/demo.py", line 90, in <module>                                                                                                                                                                                                                                                                                                                              
    pred_traj = run(cfg, args.network, args.imagedir, args.calib, args.stride, args.skip, args.viz, args.timeit, args.save_reconstruction)                                                                                                                                                                                                                                          
  File "/home/relh/anaconda3/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context                                                                                                                                                                                                                                                                
    return func(*args, **kwargs)                                                                                                                                                                                                                                                                                                                                                    
  File "/home/relh/DPVO/demo.py", line 51, in run                                                                                                                                                                                                                                                                                                                                   
    slam(t, image, intrinsics)                                                              
  File "/home/relh/DPVO/dpvo/dpvo.py", line 342, in __call__                                                                                                                              
    self.index_[self.n + 1] = self.n + 1                                                                                                                                                  
IndexError: index 2048 is out of bounds for dimension 0 with size 2048

I have an error from BUFFER_SIZE in the cfg not being large enough. Is the right way to address this to increase the buffer size or is it indicative of a different problem?

How is TarTanAir dataset downloaded?

Hi, I wonder how is TarTanAir downloaded on my machine? In

only a .pickle file is loaded, what about those .npy files?

Thanks!

About camera pose for Replica

Thanks for your great work!

I tried to use your trained model on a scene of Replica, which is used in iMAP and NICE-SLAM. I noticed that the estimated camera pose is not on a real scale. I also visualized the sparse point cloud, which is smaller than the ground truth mesh. I would like to know if there are any mistakes I made in using your code.

Thanks for your help!

sparse point cloud

ground truth mesh

support RGB-D Cameras?

Hi Authors,

"We propose Deep Patch Visual Odometry (DPVO), a new deep learning system for monocular Visual Odometry (VO)....", I wan to ask that DPVO supports RGB-D Cameras just like DROID-SLAM ?

Thanks!

Why does the program get stuck when calling ii jj kk in the forward function of net.py

In the forward function of net.py, using ii jj kk will cause the program to freeze, so the code will get stuck in this line of "ix, jx = fastba.neighbors(kk, jj)"
Even getting stuck after run "print(ii)"
Could you please help me solve the problem?

model generalization issues

Hello, I reimplement the model exactly following the training method introduced in the paper.The results of my implementation are almost equal to those on the tartan dataset (within 10%).But my model's generalization ability is worse than yours.Why there is a gap between our models？Could you please give me some suggestions please?Thank you!

Compiling in Windows

Hello, I'm trying to compile DPVO to use on WHAM project. but I'm not having success, the erorr that shows me is

      correlation_kernel.obj : error LNK2001: s¡mbolo externo nÆo resolvido "public: long * __cdecl at::TensorBase::data_ptr<long>(void)const " (??$data_ptr@J@TensorBase@at@@QEBAPEAJXZ)
      build\lib.win-amd64-3.9\cuda_corr.cp39-win_amd64.pyd : fatal error LNK1120: 1 externo nÆo resolvidos
      error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.29.30133\\bin\\HostX64\\x64\\link.exe' failed with exit code 1120
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for dpvo
  Running setup.py clean for dpvo
Failed to build dpvo

Can you guys give some hint on what I should do to make it work?

thanks a lot

bad result on kitti

is there any way to solve this scale problem?

Some problems when running demo.py

Hi, Thanks for opening source your code.

I follow your instructions to configure the environment successfully.
But when running the demo.py, it gives me some errors like this:

Framebuffer with requested attributes not available. Using available framebuffer. You may see visual artifacts.Aborted (core dumped)

I have searched for it on the internet, but their solutions are not working.
Can you give me some advice?

Thanks.

error running `pip install ./DPViewer`

I'm a little stuck at this point in the setup:

pip install ./DPViewer

I get the following message:

pip install ./DPViewer
Processing ./DPViewer
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: dpviewer
  Building wheel for dpviewer (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [102 lines of output]
      running bdist_wheel
      running build
      running build_py
      running build_ext
      -- Caffe2: CUDA detected: 11.3
      -- Caffe2: CUDA nvcc is: /usr/local/cuda-11.3/bin/nvcc
      -- Caffe2: CUDA toolkit directory: /usr/local/cuda-11.3
      -- Caffe2: Header version is: 11.3
      -- Found cuDNN: v8.5.0  (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libcudnn.so)
      -- /usr/local/cuda-11.3/lib64/libnvrtc.so shorthash is 8aa72235
      -- Autodetected CUDA architecture(s):  7.5
      -- Added CUDA NVCC flags for: -gencode;arch=compute_75,code=sm_75
      CMake Warning at /home/ddetone/anaconda3/envs/dpvo/lib/python3.10/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:22 (message):
        static library kineto_LIBRARY-NOTFOUND not found.
      Call Stack (most recent call first):
        /home/ddetone/anaconda3/envs/dpvo/lib/python3.10/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:127 (append_torchlib_if_found)
        CMakeLists.txt:24 (find_package)
      
      
      -- pybind11 v2.10.0 dev1
      -- Configuring done
      CMake Warning at pybind11/tools/pybind11Tools.cmake:173 (add_library):
        Cannot generate a safe runtime search path for target dpviewerx because
        files in some directories may conflict with libraries in implicit
        directories:
      
          runtime library [libcudnn.so.8] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
            /home/ddetone/anaconda3/envs/dpvo/lib/python3.10/site-packages/torch/lib
      
        Some of these libraries may not be found correctly.
      Call Stack (most recent call first):
        dpviewer/CMakeLists.txt:8 (pybind11_add_module)
      
      
      -- Generating done
      -- Build files have been written to: /mnt/share/code/DPVO/DPViewer/build/temp.linux-x86_64-3.10
      /home/ddetone/anaconda3/envs/dpvo/lib/python3.10/site-packages/cmake/data/bin/cmake -S/mnt/share/code/DPVO/DPViewer -B/mnt/share/code/DPVO/DPViewer/build/temp.linux-x86_64-3.10 --check-build-system CMakeFiles/Makefile.cmake 0
      /home/ddetone/anaconda3/envs/dpvo/lib/python3.10/site-packages/cmake/data/bin/cmake -E cmake_progress_start /mnt/share/code/DPVO/DPViewer/build/temp.linux-x86_64-3.10/CMakeFiles /mnt/share/code/DPVO/DPViewer/build/temp.linux-x86_64-3.10//CMakeFiles/progress.marks
      /usr/bin/make  -f CMakeFiles/Makefile2 all
      make[1]: Entering directory '/mnt/share/code/DPVO/DPViewer/build/temp.linux-x86_64-3.10'
      /usr/bin/make  -f dpviewer/CMakeFiles/dpviewerx.dir/build.make dpviewer/CMakeFiles/dpviewerx.dir/depend
      make[2]: Entering directory '/mnt/share/code/DPVO/DPViewer/build/temp.linux-x86_64-3.10'
      cd /mnt/share/code/DPVO/DPViewer/build/temp.linux-x86_64-3.10 && /home/ddetone/anaconda3/envs/dpvo/lib/python3.10/site-packages/cmake/data/bin/cmake -E cmake_depends "Unix Makefiles" /mnt/share/code/DPVO/DPViewer /mnt/share/code/DPVO/DPViewer/dpviewer /mnt/share/code/DPVO/DPViewer/build/temp.linux-x86_64-3.10 /mnt/share/code/DPVO/DPViewer/build/temp.linux-x86_64-3.10/dpviewer /mnt/share/code/DPVO/DPViewer/build/temp.linux-x86_64-3.10/dpviewer/CMakeFiles/dpviewerx.dir/DependInfo.cmake --color=
      Dependencies file "dpviewer/CMakeFiles/dpviewerx.dir/viewer_cuda.cu.o.d" is newer than depends file "/mnt/share/code/DPVO/DPViewer/build/temp.linux-x86_64-3.10/dpviewer/CMakeFiles/dpviewerx.dir/compiler_depend.internal".
      Dependencies file "dpviewer/CMakeFiles/dpviewerx.dir/viewer.cpp.o.d" is newer than depends file "/mnt/share/code/DPVO/DPViewer/build/temp.linux-x86_64-3.10/dpviewer/CMakeFiles/dpviewerx.dir/compiler_depend.internal".
      Consolidate compiler generated dependencies of target dpviewerx
      make[2]: Leaving directory '/mnt/share/code/DPVO/DPViewer/build/temp.linux-x86_64-3.10'
      /usr/bin/make  -f dpviewer/CMakeFiles/dpviewerx.dir/build.make dpviewer/CMakeFiles/dpviewerx.dir/build
      make[2]: Entering directory '/mnt/share/code/DPVO/DPViewer/build/temp.linux-x86_64-3.10'
      make[2]: *** No rule to make target '/usr/lib/x86_64-linux-gnu/libSM.so', needed by '../lib.linux-x86_64-3.10/dpviewerx.cpython-310-x86_64-linux-gnu.so'.  Stop.
      make[2]: Leaving directory '/mnt/share/code/DPVO/DPViewer/build/temp.linux-x86_64-3.10'
      CMakeFiles/Makefile2:115: recipe for target 'dpviewer/CMakeFiles/dpviewerx.dir/all' failed
      make[1]: *** [dpviewer/CMakeFiles/dpviewerx.dir/all] Error 2
      make[1]: Leaving directory '/mnt/share/code/DPVO/DPViewer/build/temp.linux-x86_64-3.10'
      Makefile:90: recipe for target 'all' failed
      make: *** [all] Error 2
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/mnt/share/code/DPVO/DPViewer/setup.py", line 123, in <module>
          setup(
        File "/home/ddetone/anaconda3/envs/dpvo/lib/python3.10/site-packages/setuptools/__init__.py", line 87, in setup
          return distutils.core.setup(**attrs)
        File "/home/ddetone/anaconda3/envs/dpvo/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 148, in setup
          return run_commands(dist)
        File "/home/ddetone/anaconda3/envs/dpvo/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 163, in run_commands
          dist.run_commands()
        File "/home/ddetone/anaconda3/envs/dpvo/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 967, in run_commands
          self.run_command(cmd)
        File "/home/ddetone/anaconda3/envs/dpvo/lib/python3.10/site-packages/setuptools/dist.py", line 1214, in run_command
          super().run_command(command)
        File "/home/ddetone/anaconda3/envs/dpvo/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
          cmd_obj.run()
        File "/home/ddetone/anaconda3/envs/dpvo/lib/python3.10/site-packages/wheel/bdist_wheel.py", line 299, in run
          self.run_command('build')
        File "/home/ddetone/anaconda3/envs/dpvo/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/home/ddetone/anaconda3/envs/dpvo/lib/python3.10/site-packages/setuptools/dist.py", line 1214, in run_command
          super().run_command(command)
        File "/home/ddetone/anaconda3/envs/dpvo/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
          cmd_obj.run()
        File "/home/ddetone/anaconda3/envs/dpvo/lib/python3.10/site-packages/setuptools/_distutils/command/build.py", line 135, in run
          self.run_command(cmd_name)
        File "/home/ddetone/anaconda3/envs/dpvo/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/home/ddetone/anaconda3/envs/dpvo/lib/python3.10/site-packages/setuptools/dist.py", line 1214, in run_command
          super().run_command(command)
        File "/home/ddetone/anaconda3/envs/dpvo/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
          cmd_obj.run()
        File "/home/ddetone/anaconda3/envs/dpvo/lib/python3.10/site-packages/setuptools/command/build_ext.py", line 79, in run
          _build_ext.run(self)
        File "/home/ddetone/anaconda3/envs/dpvo/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 339, in run
          self.build_extensions()
        File "/home/ddetone/anaconda3/envs/dpvo/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 448, in build_extensions
          self._build_extensions_serial()
        File "/home/ddetone/anaconda3/envs/dpvo/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 473, in _build_extensions_serial
          self.build_extension(ext)
        File "/mnt/share/code/DPVO/DPViewer/setup.py", line 116, in build_extension
          subprocess.check_call(
        File "/home/ddetone/anaconda3/envs/dpvo/lib/python3.10/subprocess.py", line 369, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--verbose', '-j']' returned non-zero exit status 2.
      [end of output]

Any ideas? Maybe something wrong with my cudnn install?

Error when implementing demo

Hi, @zachteed @lahavlipson, sorry to bother you again. After almost one-week-long, I made some breakthroughs. After I created the virtual environment on my Ubunutu20.04 and succeeded in installing CUDA 11.8 and the corresponding cudnn (8.50 for CUDA 11.X), I finally solved the problem of the failure to download DPViewer, which is not that easy as it sounded, especially for me who might had no background of how to implement learning on Ubuntu before. I still encountered some problems when testing the demo given. After I inputted the python demo.py --imagedir=movies/IMG_0494.MOV --calib=calib/iphone.txt --stride=5 --viz, the outcome is given below. The error is the failure to locate the file "dpvo.pth", I wonder if I miss something or not. I would be very grateful if you could do me a favor.
`(dpvo) ➜ DPVO git:(main) ✗ python demo.py --imagedir=movies/IMG_0494.MOV --calib=calib/iphone.txt --stride=5 --viz
Running with config...
BUFFER_SIZE: 2048
GRADIENT_BIAS: False
KEYFRAME_INDEX: 4
KEYFRAME_THRESH: 15.0
MIXED_PRECISION: True
MOTION_DAMPING: 0.5
MOTION_MODEL: DAMPED_LINEAR
OPTIMIZATION_WINDOW: 10
PATCHES_PER_FRAME: 96
PATCH_LIFETIME: 13
REMOVAL_WINDOW: 22
Traceback (most recent call last):
File "/home/rui/workspace/DPVO/demo.py", line 78, in
run(cfg, args.network, args.imagedir, args.calib, args.stride, args.skip, args.viz, args.timeit)
File "/home/rui/anaconda3/envs/dpvo/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/home/rui/workspace/DPVO/demo.py", line 42, in run
slam = DPVO(cfg, network, ht=image.shape[1], wd=image.shape[2], viz=viz)
File "/home/rui/workspace/DPVO/dpvo/dpvo.py", line 21, in init
self.load_weights(network)
File "/home/rui/workspace/DPVO/dpvo/dpvo.py", line 93, in load_weights
state_dict = torch.load(network)
File "/home/rui/anaconda3/envs/dpvo/lib/python3.10/site-packages/torch/serialization.py", line 699, in load
with _open_file_like(f, 'rb') as opened_file:
File "/home/rui/anaconda3/envs/dpvo/lib/python3.10/site-packages/torch/serialization.py", line 230, in _open_file_like
return _open_file(name_or_buffer, mode)
File "/home/rui/anaconda3/envs/dpvo/lib/python3.10/site-packages/torch/serialization.py", line 211, in init
super(_open_file, self).init(open(name, mode))
FileNotFoundError: [Errno 2] No such file or directory: 'dpvo.pth'

Regarding the issue of deep initialization

Author, hello. I noticed that DPVO samples on the feature map and initializes the deep values randomly. If I want to use sensor depth values for initialization, how should I proceed? (I tried multiplying the coordinates of the sample points by 4 and retrieving the depth values from the depth map, but found that the read depth values were incorrect.)

TensorRT export

Did anyone try TensorRT export for real time performance?

Possibility of obtaining per-frame camera localization confidences

Hi @lahavlipson @zachteed!

Great work, thanks a lot for releasing such a neat implementation!

I wonder whether it is possible to obtain confidences in a way that is correlated with the camera pose estimation quality. In an ideal scenario, I need the per-frame confidences which captures the uncertainty of camera localization to design an objective. I noticed that there are confidence weights estimated for each edge in the factor graph. But they were not quite helpful when I compare them across frames in a video. Also between two videos (one with bad, another with good camera localization results), the average confidence weights of the video with bad camera localization was higher. Do you have any idea how I can capture such uncertainty from the model?

Issue related to paper published and implementation

Hi @lahavlipson ,
I was able to run demo on my dataset and the results were amazing. Thanks for this wornderful work . I was trying to understand the codebase and the paper. I came across some i,j,k conventions . In paper it is mentioned as i = patch index, k= other frame, and j = source frame whereas in code k = patch index, i = source frame and j = other frame. I have added below snippet of code and paper reference. Can you please clarify whether I have understood wrong or there is some difference.

Thanks,
Vinay

Code Snippet

def transform(poses, patches, intrinsics, ii, jj, kk, depth=False, valid=False, jacobian=False, tonly=False):
    """ projective transform """

    # backproject
    X0 = iproj(patches[:,kk], intrinsics[:,ii])

    # transform
    Gij = poses[:, jj] * poses[:, ii].inv()

    if tonly:
        Gij[...,3:] = torch.as_tensor([0,0,0,1], device=Gij.device)

    X1 = Gij[:,:,None,None] * X0

    # project
    x1 = proj(X1, intrinsics[:,jj], depth)


    if jacobian:
        p = X1.shape[2]
        X, Y, Z, H = X1[...,p//2,p//2,:].unbind(dim=-1)
        o = torch.zeros_like(H)
        i = torch.zeros_like(H)

        fx, fy, cx, cy = intrinsics[:,jj].unbind(dim=-1)

        d = torch.zeros_like(Z)
        d[Z.abs() > 0.2] = 1.0 / Z[Z.abs() > 0.2]

        Ja = torch.stack([
            H,  o,  o,  o,  Z, -Y,
            o,  H,  o, -Z,  o,  X, 
            o,  o,  H,  Y, -X,  o,
            o,  o,  o,  o,  o,  o,
        ], dim=-1).view(1, len(ii), 4, 6)
        
        Jp = torch.stack([
             fx*d,     o, -fx*X*d*d,  o,
                o,  fy*d, -fy*Y*d*d,  o,
        ], dim=-1).view(1, len(ii), 2, 4)

        Jj = torch.matmul(Jp, Ja)
        Ji = -Gij[:,:,None].adjT(Jj)
        
        Jz = torch.matmul(Jp, Gij.matrix()[...,:,3:])

        return x1, (Z > 0.2).float(), (Ji, Jj, Jz)

    if valid:
        return x1, (X1[...,2] > 0.2).float()
        
    return

Paper ref :

Real-time pose estimation

Hi, I'm trying to use the DPVO to get the pose for each frame in a real-time fashion, for example, printing xyz and quaternion. Do you have any hints for this? Looking at the function

DPVO/dpvo/dpvo.py

Line 153 in 4f2f0cc

def get_pose(self, t):

I'm trying to print lietorch.SE3(slam.poses_[-1]).data.cpu().numpy(), but I always get value [0. 0. 0. 0. 0. 0. 1.].

Here is the modified loop inside the run function of demo.py:

while 1:
        (t, image, intrinsics) = queue.get()
        if t < 0: break

        image = torch.from_numpy(image).permute(2,0,1).cuda()
        intrinsics = torch.from_numpy(intrinsics).cuda()

        if slam is None:
            slam = DPVO(cfg, network, ht=image.shape[1], wd=image.shape[2], viz=viz)

        image = image.cuda()
        intrinsics = intrinsics.cuda()

        with Timer("SLAM", enabled=timeit):
            slam(t, image, intrinsics)
        
        print(lietorch.SE3(slam.poses_[-1]).data.cpu().numpy())

What am I doing wrong?
Thanks in advance

Missing calib/icl_nuim.txt

Hi, it seems like the camera intrinsics file required by the evaluate_icl_nuim.py script is missing. When I use values from the dataset paper (481.20 480.0 319.50 239.50) I get higher ATE values than reported.

Could you share the values used in your paper?

Illegal memory access while using custom images

I am trying to use DPVO to estimate poses for my object. I have continuous images surrounding the object from all directions. When I run the code on my images using demo.py code. The directory contains about 115 images in all.

File "/DPVO/demo.py", line 92, in
pred_traj = run(cfg, args.network, args.imagedir, args.calib, args.stride, args.skip, args.viz, args.timeit, args.save_reconstruction)
File "/root/miniconda3/envs/dpvo/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/DPVO/demo.py", line 51, in run
slam(t, image, intrinsics)
File "/DPVO/dpvo/dpvo.py", line 394, in call
self.update()
File "/DPVO/dpvo/dpvo.py", line 278, in update
self.network.update(self.net, ctx, corr, None, self.ii, self.jj, self.kk)
File "/root/miniconda3/envs/dpvo/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/DPVO/dpvo/net.py", line 80, in forward
ix, jx = fastba.neighbors(kk, jj)
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1

when I run the code as CUDA_LAUNCH_BLOCKING=1 python demo.py --save_reconstruction --save_trajectory --imagedir=images3_jpg/ --calib=custom_calib.txt --stride=1
The code runs without any errors, but the the saved trajectory file contains pose values as nans beyond index 15
I have observed the same with multiple other custom image directories as well.

Question: is it applicable to do camera movement classification?

Hi, first, thanks for your work!)

I would like to ask you for advice: I am completely new to this field (actually I came from NLP), and now playing with pet project where I want to classify camera movement

For example, here camera is moving around the "main" scene.

While here camera stands in the same place, while rotating to the right.

If I will use optical flow, I will get more or less the same output for both videos.

So, what are your thoughts about it? Does your model applicable to this case? What about generalization?

LieTorch Error: Cannot reshape tensor of 0 elements into shape ... When running TartanAir Evaluation

Hi, I'm installing DPVO using the provided docker image with CUDA version of 11.7.

When I'm running the evaluation for TartanAir dataset, the following exception is raised:

(dpvo) root@docker-desktop:/DPVO# python evaluate_tartan.py --trials=5 --split=validation --plot --save_trajectory
Running with config...
BUFFER_SIZE: 2048
GRADIENT_BIAS: False
KEYFRAME_INDEX: 4
KEYFRAME_THRESH: 15.0
MIXED_PRECISION: True
MOTION_DAMPING: 0.5
MOTION_MODEL: DAMPED_LINEAR
OPTIMIZATION_WINDOW: 10
PATCHES_PER_FRAME: 96
PATCH_LIFETIME: 13
REMOVAL_WINDOW: 22
Traceback (most recent call last):
  File "/DPVO/evaluate_tartan.py", line 182, in <module>
    results = evaluate(cfg, args.weights, split=args.split, trials=args.trials, plot=args.plot, save=args.save_trajectory)
  File "/root/miniconda3/envs/dpvo/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/DPVO/evaluate_tartan.py", line 110, in evaluate
    traj_est, tstamps = run(scene_path, config, net)
  File "/root/miniconda3/envs/dpvo/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/DPVO/evaluate_tartan.py", line 54, in run
    slam.update()
  File "/DPVO/dpvo/dpvo.py", line 272, in update
    coords = self.reproject()
  File "/DPVO/dpvo/dpvo.py", line 188, in reproject
    coords = pops.transform(SE3(self.poses), self.patches, self.intrinsics, ii, jj, kk)
  File "/DPVO/dpvo/projective_ops.py", line 60, in transform
    Gij = poses[:, jj] * poses[:, ii].inv()
  File "/DPVO/dpvo/lietorch/groups.py", line 147, in inv
    return self.__class__(self.apply_op(Inv, self.data))
  File "/DPVO/dpvo/lietorch/groups.py", line 130, in apply_op
    return data.view(out_shape + (-1,))
RuntimeError: cannot reshape tensor of 0 elements into shape [1, 0, -1] because the unspecified dimension size -1 can be any value and is ambiguous

Does anyone know how to solve this problem?

More Environment Information

$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Jun__8_16:49:14_PDT_2022
Cuda compilation tools, release 11.7, V11.7.99
Build cuda_11.7.r11.7/compiler.31442593_0

$ nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 515.57       Driver Version: 516.59       CUDA Version: 11.7     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  On   | 00000000:01:00.0  On |                  N/A |
| N/A   50C    P5    25W /  N/A |   1289MiB /  6144MiB |      7%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A        20      G   /Xwayland                       N/A      |
|    0   N/A  N/A        22      G   /Xwayland                       N/A      |
|    0   N/A  N/A        33      G   /Xwayland                       N/A      |
+-----------------------------------------------------------------------------+

$ python --version
Python 3.10.0

Question about the channels of patch tensors

Hi, thanks for your remarkable work!
How, i am trying to understand the code of DPVO, but some problems confuse me.
In the transform function of projective_ops.py, we can get X0 based on patches and intrinsics, and X0 is constructed as [X Y 1 d]. However, after transforming using Gij, i saw that transformed results X1 donot have any channel equals to 1, which seems confusing, and its construction is like [X Y Z H] in the following code. What's the meaning of H?
Thanks at advances.

Unable to find cuda_ba in the site packages folder after running pip install .

Before running the pip install . , running demo.py will prompt no module named cuda_ba
However, after running the pip install . the program will get stuck in the file dpvo/fastba/ba.py
The IDE marked with a red line in the line "import cuda_ba"
but the program will not report an error, it will only get stuck in the line "ix, jx=fastba. highbors (kk, jj)" in the devo/net.py
Could you please tell me how to solve the problem

Extracting full image depth maps

Hi,

Thanks for the great work! Is there a way to extract complete frame depth maps?

RuntimeError: CUDA illegal memory access

Hi,

Thank you for your work and code, this is really cool!

I've been running some experiments with your code and I noticed the gradient_bias option inside the Patchifier network to bias patch selection towards regions with high gradient.

However when I enable this option the training exits with an error:

RuntimeError: CUDA error: an illegal memory access was encountered

I think this is happening because of a shape mismatch inside the patchify_forward_kernel cuda kernel coming from the gradient images, but I wanted to check if you are aware of this error?

randomly selecting patch centroids vs. 2D keypoint locations

The paper mentions that Randomly selecting patch centroids works better than 2D keypoint locations produced via SIFT [22], ORB [29], Superpoint [9], or pixels with high image gradient, I have two questions:

Do you use the classical method to select feature points and resize the location to the feature map size as the patch center centroids?
Is the model retrained when the classical keypoint methods are used?

Looking forward to your reply. Thx a lot!

How is the structure of the Tartan datasets arranged during training?

Thank you for open sourcing your excellent work. I encountered some problems during training. For one of the sequences, my path is as follows：
～/datasets/TartanAir/abandonedfactory/Easy/depth_left/abandonedfactory/EasyP000/depth_left/xxxx.npy
~/datasets/TartanAir/abandonedfactory/Easy/image_left/abandonedfactory/Easy/P000/image_left/xxxx.png

How can I rearrange the file structure?
Is there any preprocessing after downloading the data set?
Thank you very much and look forward to your answer.

how to make calib file for our own camera

Dear author,

how to make calib file for our own camera ? Can you interprete each number of the calib file so that we can make our own calib file ?

Thanks

How to understand the softagg for frame aggregation?

Hi, I've a problem understanding the index of edges when you perform the softagg for the frame aggregation:
net = net + self.agg_ij(net, ii*12345 + jj)

DPVO/dpvo/net.py

Line 88 in 3f65b72

net = net + self.agg_ij(net, ii*12345 + jj)

In the paper, it is said that frame aggregation is performed on the edges sharing the same source and destination frames. How can it be "ii*12345 + jj"? It will be appreciated if you can explain it a little bit.

scale drift on demo

Hi, Thanks for the great work!
I'm running the demo on a video captured from my iphone. I've found the intrinsic params of my iphone's camera. I'm encountering scale drift as the video progress.
Any idea what can cause that?

GCC Version

Not an issue as such, but I had difficulty building this on Ubuntu 22.04. What ended up working was using gcc-10 / g++-10 as the CC and CXX compilers respectively. Seems something about GCC-11 wasn't letting the extensions be built.

ERROR: Failed building wheel for dpvo

I keep getting this error when I run: pip install . how can I resolve this error ?

What's the best way to fine-tune DPVO?

Hello,

I have a custom dataset. I want to fine-tune DPVO on this.

What are the layers those I can un-freeze for fine-tuning? And also should I change any training parameters? Can you please suggest regarding the same.

Thank you
Gopi

Image gradient bias

I see that in the code, is present a function to bias selection toward high image gradients.

 g = self.__image_gradient(images)
  x = torch.randint(1, w-1, size=[n, 3*patches_per_image], device="cuda")
  y = torch.randint(1, h-1, size=[n, 3*patches_per_image], device="cuda")
  coords = torch.stack([x, y], dim=-1).float()
  g = altcorr.patchify(g[0,:,None], coords, 0).view(n, 3 * patches_per_image)
  ix = torch.argsort(g, dim=1)
  x = torch.gather(x, 1, ix[:, -patches_per_image:])
  y = torch.gather(y, 1, ix[:, -patches_per_image:])

Activating this part makes the training unstable and in general worser.
Did you have an explanation to why image gradient bias doesn’t work? from the paper I haven’t found any decisive conclusion.

princeton-vl / dpvo Goto Github PK

dpvo's People

Contributors

Stargazers

Watchers

Forkers

dpvo's Issues

Recommend Projects

Recommend Topics

Recommend Org