Git Product home page Git Product logo

isl-org / open3d Goto Github PK

View Code? Open in Web Editor NEW
10.3K 196.0 2.2K 590.47 MB

Open3D: A Modern Library for 3D Data Processing

Home Page: http://www.open3d.org

License: Other

Shell 0.50% CMake 1.50% C++ 81.97% C 0.54% GLSL 0.09% Python 7.74% JavaScript 0.85% Cuda 6.45% Objective-C++ 0.22% HTML 0.05% CSS 0.08%
mesh-processing computer-graphics opengl cpp python reconstruction odometry visualization registration machine-learning

open3d's Introduction

Open3D: A Modern Library for 3D Data Processing

Open3D is an open-source library that supports rapid development of software that deals with 3D data. The Open3D frontend exposes a set of carefully selected data structures and algorithms in both C++ and Python. The backend is highly optimized and is set up for parallelization. We welcome contributions from the open-source community.

Ubuntu CI macOS CI Windows CI

Core features of Open3D include:

  • 3D data structures
  • 3D data processing algorithms
  • Scene reconstruction
  • Surface alignment
  • 3D visualization
  • Physically based rendering (PBR)
  • 3D machine learning support with PyTorch and TensorFlow
  • GPU acceleration for core 3D operations
  • Available in C++ and Python

Here's a brief overview of the different components of Open3D and how they fit together to enable full end to end pipelines:

Open3D_layers

For more, please visit the Open3D documentation.

Python quick start

Pre-built pip packages support Ubuntu 18.04+, macOS 10.15+ and Windows 10+ (64-bit) with Python 3.8-3.11.

# Install
pip install open3d       # or
pip install open3d-cpu   # Smaller CPU only wheel on x86_64 Linux (v0.17+)

# Verify installation
python -c "import open3d as o3d; print(o3d.__version__)"

# Python API
python -c "import open3d as o3d; \
           mesh = o3d.geometry.TriangleMesh.create_sphere(); \
           mesh.compute_vertex_normals(); \
           o3d.visualization.draw(mesh, raw_mode=True)"

# Open3D CLI
open3d example visualization/draw

To get the latest features in Open3D, install the development pip package. To compile Open3D from source, refer to compiling from source.

C++ quick start

Checkout the following links to get started with Open3D C++ API

To use Open3D in your C++ project, checkout the following examples

Open3D-Viewer app

Open3D-Viewer is a standalone 3D viewer app available on Debian (Ubuntu), macOS and Windows. Download Open3D Viewer from the release page.

Open3D-ML

Open3D-ML is an extension of Open3D for 3D machine learning tasks. It builds on top of the Open3D core library and extends it with machine learning tools for 3D data processing. To try it out, install Open3D with PyTorch or TensorFlow and check out Open3D-ML.

Communication channels

  • GitHub Issue: bug reports, feature requests, etc.
  • Forum: discussion on the usage of Open3D.
  • Discord Chat: online chats, discussions, and collaboration with other users and developers.

Citation

Please cite our work if you use Open3D.

@article{Zhou2018,
    author    = {Qian-Yi Zhou and Jaesik Park and Vladlen Koltun},
    title     = {{Open3D}: {A} Modern Library for {3D} Data Processing},
    journal   = {arXiv:1801.09847},
    year      = {2018},
}

open3d's People

Contributors

benjaminum avatar csukuangfj avatar dudulle avatar errissa avatar forrestjgq avatar germanros1987 avatar griegler avatar junha-l avatar kongsgard avatar nachovizzo avatar nobuotsukamoto avatar prewettg avatar pvarvak avatar qianyizh avatar reyanshsolis avatar roehling avatar rohanrathi avatar sammo2828 avatar sanskar107 avatar saurabheights avatar scimad avatar shreyanshdarshan avatar shubhamagarwal12 avatar ssheorey avatar stotko avatar syncle avatar takanokage avatar thended avatar yuecideng avatar yxlao avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

open3d's Issues

Read PCD file RGB channel bug

When read PCD files, the RGB color seems to be BGR.
I have tested reading PLY file. Seems to be fine.

pcd0 = ReadPointCloud("TestData/ICP/cloud_bin_0.pcd")
DrawGeometries([pcd0])

Will create the following image.

screen shot 2017-06-15 at 9 04 19 pm

Pybind issue

I notice that cmake 3.6 tool-chain generates py3d instead of py3d.so after compilation.
This affects python scripts. They don't recognize py3d library.

For example:

> python
>>> from py3d import *
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named py3d

But if I change the name, it works

> mv py3d py3d.so
> python
>>> from py3d import *
(no error)

This issue comes with MacOSX Sierra, Cmake 3.6, Mac Native Python 2.7.
Windows + VisualStudio2015 seems to fine.
We may need to change CMakeList.txt file or get some information from Pybind to fix this issue.

PEP8 style for python bindings

Hi guys,

I just wanted to let you know that I just started to seriously work with Open3D and it is absolutely awesome. I especially like the python bindings!

That said I wanted to point out that CamelCase for function names is very unusual in the python world. The standard style guide https://www.python.org/dev/peps/pep-0008/#function-names suggest lower case with underscores. AFAIK most python packages follow this convention, and I could imagine users asking for this pretty quickly after the initial release.

Registration evaluation on Redwood benchmark

I have discussed with Jaesik.
We need to demonstrate that we have the state-of-the-art registration results.
It also helps us debugging the reconstruction system.

  • Python helper function to download and unzip the redwood benchmark
  • Python script to run global registration
  • Benchmarking accuracy. It should match or outperform PCL and Choi et al
  • Benchmarking performance. It should be much faster than PCL and Choi et al
  • Fix bugs and submit PR

Compile failure due to Libpng

I encountered compile error with up-to-date master branch. It is due to libpng.

Cmake says there is libpng in my Mac, but build tool-chain is bit broken somehow. The tool-chain cannot link libpng to the objects.

-- Checking for one of the modules 'libjpeg'
-- Build libjpeg from source
-- Use native build of libpng
-- Use native build of glfw
:
[ 74%] Linking CXX executable ../bin/Test/TestFileDialog
ld: library not found for -lpng16
clang: error: linker command failed with exit code 1 (use -v to see invocation)

I am using -DOpen3D_USE_NATIVE_DEPENDENCY_BUILD=OFF for now to avoid this failure case.

Point cloud registration need to output information matrix.

As title says, point cloud registration module does not outputs information matrix. Information matrix is necessary for global graph optimization.

I am not trying to output information matrix for every case as computing information matrix would add additional computational burden, and not used for normal cases. I am going to introduce additional option for registration class, and registration result class will include information matrix if it is specified. How do you think?

Follow up work of #41

#41 covers basic data structure and functions for RGBD odometry application.
There are few issues additionally need to be handled.

  • OpenMP boosted version is not that fast enough: I expect 4~5 times faster when OpenMP is used, but the performance gain is not obvious. Should be handled.
  • Some example that shows Hybrid odometry is better than old odometry.

Headless OpenGL rendering

This can be useful in scenarios where users want to render a (long) video.
It can also be useful when Open3D is deployed in a remote server and window creation is a problem.

Some information regarding GLFW: glfw/glfw#648

PPM/PGM file support

Looks like the current image IO can handle jpg and png image format. However, some of RGBD dataset (such as NYU dataset) is captured with ppm/pgm format. Do we need to extend OpenCV support for these formats as well?

Follow up work of #46

Follow up work of #46

  1. Write a Python application tutorial to download and test on the Redwood benchmark.
  2. Make sure the test matches the accuracy of Choi et al, and is faster for an order of magnitude.
  3. Optimize code if necessary.

These will be the next pull request.

Additionally, the next step is FastGlobalRegistration. It is another PR.

Using OpenMP on OSX

OpenMP conveniently parallelize many algorithms and accelerates them greatly. However, the default Xcode LLVM does not support OpenMP.

The current solution is to use homebrew-gcc on OSX, see
https://github.com/IntelVCL/Open3D/blob/master/docs/compilation.rst

It however has some compatibility issues with some external libraries (glfw). It also introduces conflicts between libc++ and libstdc++, see #8

We have experimented a homebrew-llvm. It is supposed to support OpenMP. However, it is hard to configure it right. This should be resolved in the future.

Basic image processing module

During the process of implementing OdometryRGBD, we found a need of basic image processing modules (e.g., pixel pointer, downsampling, filtering, creating pyramid image) that are generally useful for OdometryRGBD, texturing meshes, and so on.

Plans:

  • Put image processing functions into core/geometry/image.cpp/h
  • Some useful image functions are distributed in the individual projects, we are going to move them too after discussion.
  • I am not going to introduce heavy library for image processing. Instead, I will implement reasonably efficient functions: such as separable 1D filtering instead of 2D filtering.
  • I will add TestImageProcess.cpp/h in Test module for debugging and demonstrating implemented functions.
  • If OpenCV is available, CMake tool-chain will link it for utilizing higher level functions such as ORB.

Jaesik

Custom principal point

ConvertFromPinholeCameraParameters function in Open3D/src/Visualization/Visualizer/ViewControl.cpp is designed for deterministic pinhole camera model: its principal point should be central position of an image domain.

For example, ViewGeometry terminates if below condition does not meet:

intrinsic.intrinsic_matrix_(0, 2) != (double)window_width_ / 2.0 - 0.5 ||
intrinsic.intrinsic_matrix_(1, 2) != (double)window_height_ / 2.0 - 0.5)

Can we extend current OpenGL interface to allow principle point of non-centered position?

Wrapper of DrawGeometriesWithAnimationCallback

Using Visualizer in Python is still quite tedious.
This should be fixed.

def custom_draw_geometry_with_rotation(pcd):
	def rotate_view(vis):
		ctr = vis.GetViewControl()
		ctr.Rotate(10.0, 0.0)
		return False
	vis = Visualizer()
	vis.CreateWindow()
	vis.AddGeometry(pcd)
	vis.RegisterAnimationCallback(rotate_view)
	vis.Run()
	vis.DestroyWindow()	

Implement the odometry module

  • RGB-D odometry
  • ICP odometry (KinectFusion odometry)
  • Hybrid odometry (new)
  • Odometry initialization based on feature matching (new)
  • Test data and test programs
  • Python binding and python tests
  • Document and tutorial

Add functions to accept more RGB-D input formats (different resolution, different view, etc)

As discussed in #42 , in the future, we should add conversion function to covert different size and different view point RGB-D image pairs. Quote:

There are some minor formatting issues. Others are good.

I have an additional question: what if the input color image is of different size of depth image? There are some use cases:

  1. We use Primesense to capture a higher resolution image stream (my color mapping paper).
  2. We use RealSense DS4 to capture a higher resolution image stream.
  3. We use occipital sensor with an iPhone camera.

We don't need to handle them immediately. But I think we should leave a note at the class definition of RGBDImage, saying that it is for a pair of registered color and depth images, viewed from the same view, of the same resolution. If you have other format, convert it first.

In the future, we can provide similar helper functions for other kinds of input (as long as the input resolution and the calibrated extrinsic matrix is given). But it is future work.

Implement the registration module

  • Point-to-point ICP
  • Point-to-plane ICP
  • Colored ICP (new)
  • FPFH
  • RANSAC based on given correspondences
  • RANSAC based on feature matching
  • Fast global registration (new)
  • Test data and test program
  • Python binding and test python script
  • Document and tutorial

Protected function in Core\Utility\Timer.h

I was trying to add timer to my graph optimization loop. I want to just retrieve the time duration.
For example, I would like to add time : xx.xxx sec. to the last in the following line.

PrintDebug("[Iteration %02d] residual : %e, lambda : %e, valid edges : %d/%d\n, time : %2.3f",
	iter, current_residual, current_lambda, 
	valid_edges, n_edges - (n_nodes - 1), /* get time duration here */);

I tried to use GetDuration(); in Core\Utility\Timer.h but it is protected function. Can I just get time of duration from Timer? This would be useful if I want to customize timing messages.

Add file dialog / message dialog support to Visualization

So far the best file dialog library is tinyfiledialogs: https://sourceforge.net/projects/tinyfiledialogs/
It is small, clean, cross-platform. It has all functionalities I need.

With it, we shall have Ctrl/Cmd+O/S, ...

Roadblock: I need to do it.

Problems: On Windows, the input box is ugly (I checked, there seem to be no better way. Windows does not have native input box support). On Windows, the file boxes are slightly blurred on 4K monitors (there seem to be no easy workarounds).

Other options (I considered and decided against them):

Native File Dialog
https://github.com/mlabbe/nativefiledialog
It does not have message box etc.

Qt/GTK3+
Too complicated to use/include in External.

Ubuntu compiling error

Travis CI detects some compile errors in Ubuntu trusty.

:
[ 19%] Building CXX object IO/CMakeFiles/IO.dir/ClassIO/PinholeCameraTrajectoryIO.cpp.o
[ 20%] Building CXX object IO/CMakeFiles/IO.dir/FileFormat/FileBIN.cpp.o
[ 21%] Building CXX object IO/CMakeFiles/IO.dir/FileFormat/FileJPG.cpp.o
[ 21%] Building CXX object IO/CMakeFiles/IO.dir/FileFormat/FilePLY.cpp.o
[ 22%] Building CXX object IO/CMakeFiles/IO.dir/FileFormat/FileXYZN.cpp.o
[ 23%] Building CXX object IO/CMakeFiles/IO.dir/FileFormat/FileLOG.cpp.o
[ 24%] Building CXX object IO/CMakeFiles/IO.dir/FileFormat/FileXYZRGB.cpp.o
[ 25%] Building CXX object IO/CMakeFiles/IO.dir/FileFormat/FilePCD.cpp.o
[ 25%] Building CXX object IO/CMakeFiles/IO.dir/FileFormat/FilePNG.cpp.o
/home/travis/build/syncle/Open3D/src/IO/FileFormat/FilePNG.cpp:36:47: error: ‘png_image’ has not been declared
 void SetPNGImageFromImage(const Image &image, png_image &pngimage)
                                               ^
/home/travis/build/syncle/Open3D/src/IO/FileFormat/FilePNG.cpp: In function ‘void three::{anonymous}::SetPNGImageFromImage(const three::Image&, int&)’:
/home/travis/build/syncle/Open3D/src/IO/FileFormat/FilePNG.cpp:38:11: error: request for member ‘width’ in ‘pngimage’, which is of non-class type ‘int’
  pngimage.width = image.width_;
           ^
/home/travis/build/syncle/Open3D/src/IO/FileFormat/FilePNG.cpp:39:11: error: request for member ‘height’ in ‘pngimage’, which is of non-class type ‘int’
  pngimage.height = image.height_;
           ^
/home/travis/build/syncle/Open3D/src/IO/FileFormat/FilePNG.cpp:40:11: error: request for member ‘format’ in ‘pngimage’, which is of non-class type ‘int’
:

This is because png.h is not correctly included by Linux cmake build chain.
I looked /src/External/CMakeLists.txt. If cmake detects libpng, PNG_INCLUDE_DIRS in /IO/CMakelists.txt is weirdly set to as /usr/include/usr/include. If libpng is built from source, the correct header is included and this issue is gone.

I was also regenerate the same error using my Linux workstation with CMake 3.8.2

Beautify function names

Things like:

  • SelectionPolygonVolume::CropGeometry
  • SelectDownSample, VoxelDownSample, UniformDownSample, ...
  • ComputePointCloudToPointCloudDistance, ...

Move the output to the return part of functions. For all Geometry type, return std::share_ptr because Visualizer needs to share the ownership. But for std::vector, use the move and trivial copy mechanics to directly return.

Performance issue from GetRegistrationResultAndCorrespondences

This function has for-loop with OpenMP parallelization. However, it actually uses single core on run time. This is A bottleneck for RegistrationRANSACBasedOnFeatureMatching. I found this by using Visual Studio Profiler.

Let me put code snippet for a reference:

#ifdef _OPENMP
#pragma omp parallel for schedule(static)
#endif
	for (int i = 0; i < (int)source.points_.size(); i++) {
		std::vector<int> indices(1);
		std::vector<double> dists(1);
		const auto &point = source.points_[i];
		if (target_kdtree.SearchHybrid(point, max_correspondence_distance, 1,
				indices, dists) > 0) {
#ifdef _OPENMP
#pragma omp critical
#endif
			{
				result.correspondence_set_.push_back(
						Eigen::Vector2i(i, indices[0]));
				error2 += dists[0];
			}
		}
	}

I think something is happening to target_kdtree.SearchHybrid. Here are a few suspicious things:

  • Replacing this function with dummy operation, the app utilizes full CPU cores.
  • When I simply replace it with target_kdtree.SearchKNN, the app crashes on run-time.
  • In addition, there is no OpenMP + SearchHybrid example except for this in entire Open3D codes.

Is the code is reliable? Is the issue related to low level KDTree operations?

Visual Studio 2017 migration

Need to add Visual Studio 2017 into the tool chain.
Later on, we should setup travis-ci or similar testing systems to test the build of Visual Studio.

Python interface for PoseGraph

We have fancy class PoseGraph and its derived class: PoseGraphNode and PoseGraphEdge.

A question arises If I want to make PoseGraph and grow their members in python. For example, I get odometry and want to save the trajectories into .jsonfile. I would need to use PoseGraph as it is intended for this case.

So. I want to add new variable. the hope is:

# in python
pose_graph = PoseGraph()
pose_graph.nodes.append(PoseGraphNode(T))
pose_graph.edges.append(PoseGraphEdge(s,t,T,...))

what I am not clear is that whether we can use .append or std::vector<> push_back() operator in python.

In our c++ implementation the member value nodes_ and edges_ in PoseGraph are in std::vector format so I think this would be possible. Is there natural way to append new items to PoseGraph?

Python interface for GUI

I have been wanting to it for a long time, but have not got time for it.

The idea is to use the lambda function hooks to play customized animation in Python using GUI. This will solve lots of issues.

E.g. we want to access raw data (in numpy format) of a screen capture stream in an animation, etc.

JsonCpp warning

There is a warning when I compiling Open3D with OSX-El capitan.

/Users/travis/build/IntelVCL/Open3D/src/IO/FileFormat/FileJSON.cpp:42:8: warning: 'Reader' is deprecated: Use CharReader and CharReaderBuilder instead [-Wdeprecated-declarations]
        Json::Reader reader;
              ^
/usr/local/Cellar/jsoncpp/1.8.3/include/json/reader.h:35:83: note: 'Reader' has been explicitly marked deprecated here
class JSONCPP_DEPRECATED("Use CharReader and CharReaderBuilder instead") JSON_API Reader {
                                                                                  ^
/Users/travis/build/IntelVCL/Open3D/src/IO/FileFormat/FileJSON.cpp:59:8: warning: 'StyledStreamWriter' is deprecated: Use StreamWriterBuilder instead [-Wdeprecated-declarations]
        Json::StyledStreamWriter writer;
              ^
/usr/local/Cellar/jsoncpp/1.8.3/include/json/writer.h:273:70: note: 'StyledStreamWriter' has been explicitly marked deprecated here
class JSONCPP_DEPRECATED("Use StreamWriterBuilder instead") JSON_API StyledStreamWriter {
                                                                     ^

Can we replace Reader as suggested, or upgrade jsoncpp?

Python binding

This is a must have for Open3D.
I looked around, the current best solution seems to be pybind11.
https://github.com/pybind/pybind11
It is neat, header only, cross platform, and supports any non-ancient C++ compiler.

Main roadblock: I need to find time to do it.

Desired platform for Travis CI testing

#33 only handles Ubuntu trusty (14.04). We need more. Some candidates below.

  • OS X El Capitan
  • OS X Sierra
  • Ubuntu Xenial (16.04)
  • Window is not supported by Travis CI. However, I can easily check compatibility as I am mostly using VC2015 for developing.

Implement the reconstruction module

This module is built based on the odometry and registration module.

  • Graph optimization
  • Robust graph optimization
  • Local volumetric fusion
  • Large scale volumetric fusion
  • Pipeline code that integrates the functions
  • Test data and test programs
  • Python binding and python tests
  • Document and tutorial

libc++ and libstdc++ conflicts

When using g++-6 on OSX (for the sake of OpenMP), there are compilation errors linking Open3D against brew-installed jsoncpp. After digging it for a while, it is due to conflict between libc++ (used by llvm by default) and libstdc++ (used by g++ by default).

See this post for details:
open-source-parsers/jsoncpp#597

The current workaround is 6eef4a9
We force a compilation of jsoncpp from source when g++ compiler is used on OSX.

This however is a temporary solution. In the long run, we should switch to:
http://libcxx.llvm.org/docs/UsingLibcxx.html#using-libc-with-gcc
Link everything to libc++ on OSX.

This issue is related to #1

Make zlib/jpeg/png/... auto link to native libraries on Ubuntu/OSX

On Windows, all packages are included in External, and linked with Open3D.
On Ubuntu/OSX, this should be done automatically with native libraries.
This should be done with findpackage in CMake.

Roadblock: Need to do it and thoroughly test if these dependencies (a weird version, e.g.) break Open3D. In that case, we should fall back to the code included in External.

Colored point cloud registration & Multi-scale approach

I am trying to implant our ICCV code to Open3D.

The files I am editing are core/registration/TransformationEstimation.h/cpp and Registration.cpp. The question is the following virtual function:

	Eigen::Matrix4d ComputeTransformation(const PointCloud &source,
			const PointCloud &target,
			const CorrespondenceSet &corres) const override;

for colored points registration, we need additional argument e.g. point-wise color gradient for computing Jacobian. Any good idea for this?

In addition, we know the multi-scale registration really helps registration fidelity. Our ICCV implementation code is based on this idea, but I would like to make a new wrapper class so all the ICP methods in Open3D can get benefits. How do you think?

Upgrade pybind11 to 2.2+

The current pybind11 is modified from 2.0 dev.
The main modifications are:

  1. Direct (reference) access of Eigen matrices, using this patch pybind/pybind11#610
  2. Change of std::vector binding style

The first modification is now merged into pybind11 master branch.
The pybind11 in Open3D should be upgraded to mainstream stable pybind11 version.

Segfault from std::vector<Eigen::Vector6d>

We have flexible Jacobian evaluation for RGBDOdometry in Core/Odometry/RGBDOdometryJacobian.cpp. However, using std::vector<Eigen::Vector6d> has memory alignment issue. According to Eigen Issue, it says:

Otherwise, without knowing the specialization the compiler will compile that particular instance with the default std::allocator and you program is most likely to crash.

Actually this is happening to my Ubuntu 16.04. The odometry module sometimes crashes with segfault due to this issue.

Option 1
I tried two remedies described in the Eigen Issue, but could not resolve the problem. This also depends on how to interpret assignment operator by C++11 compilers (see the answer in this link.) To my understanding at this moment, this option seems to be not valid.

Option 2
Another way is not using std::vector<Eigen::Vector6d>, but this will break the structure of virtual class ComputeJacobianAndResidual as its inherited classes: RGBDOdometryJacobianFromColorTerm and RGBDOdometryJacobianFromHybridTerm as they take different size of Jacobian matrix (6x1 matrix and 6x2 matrix, respectively).

Do you have any suggestion on this?

Python interface

In python

pcd_1 = pcd_0

does not hard copy. Instead, pcd_1 gets the pointer of pcd_0. I think It is not intuitive for python. Maybe need to define = operators for basic data types?

Benchmark for building Jacobian matrix

This issue is derived from #41

I had benchmarked several ways for building Jacobian matrix.
Please check the code in Option 1 and Option 2.

	// data generation
	const int NCORR = 100000000;
	time_t tstart, tend;
	std::vector<Eigen::Vector3d> data;
	data.resize(NCORR);
	for (size_t i = 0; i < NCORR; i++) {
		data[i] = Eigen::Vector3d::Random();
	}

	// Option #1
	// Directly Building JtJ and Jtr
	tstart = time(0);
	Eigen::Matrix6d ATA;
	Eigen::Vector6d ATb;
	Eigen::Vector6d A_r;
	double r;
	ATA.setZero();
	ATb.setZero();
	for (size_t i = 0; i < NCORR; i++) {
		const Eigen::Vector3d &vs = data[i];
		const Eigen::Vector3d &vt = data[i];
		const Eigen::Vector3d &nt = data[i];
		r = (vs - vt).dot(nt);
		A_r.block<3, 1>(0, 0) = vs.cross(nt);
		A_r.block<3, 1>(3, 0) = nt;
		ATA += A_r * A_r.transpose();
		ATb += A_r * r;
	}
	tend = time(0);
	std::cout << "Option1 took " << difftime(tend, tstart) << " second(s)." << std::endl;

	// Option #2
	// Seperately build J and r
	tstart = time(0);
	Eigen::MatrixXd A(NCORR,6);
	Eigen::VectorXd b(NCORR);
	A.setZero();
	b.setZero();
	for (size_t i = 0; i < NCORR; i++) {
		const Eigen::Vector3d &vs = data[i];
		const Eigen::Vector3d &vt = data[i];
		const Eigen::Vector3d &nt = data[i];
		r = (vs - vt).dot(nt);
		A_r.block<3, 1>(0, 0) = vs.cross(nt);
		A_r.block<3, 1>(3, 0) = nt;
		A.block<1, 6>(i, 0) = A_r;
		b(i) = r;
	}
	Eigen::MatrixXd AT = A.transpose();
	ATA = AT * A;
	ATb = AT * b;
	tend = time(0);
	std::cout << "Option2 took " << difftime(tend, tstart) << " second(s)." << std::endl;

The outputs are

Option1 took 6 second(s).
Option2 took 12 second(s).

It is obvious Option 1 is faster than Option 2. It is presumably due to better cash hit.
Option 1 consumes much smaller amount of memory as well.

However, the question arises when the for loop is paralleled using OpenMP.

	// Option #3
	// Directly Building JtJ and Jtr
	tstart = time(0);
	
	ATA.setZero();
	ATb.setZero();
#ifdef _OPENMP
#pragma omp parallel for schedule(static)
#endif
	for (int i = 0; i < NCORR; i++) {
		const Eigen::Vector3d &vs = data[i];
		const Eigen::Vector3d &vt = data[i];
		const Eigen::Vector3d &nt = data[i];
		double r = (vs - vt).dot(nt);
		Eigen::Vector6d A_r;
		A_r.block<3, 1>(0, 0) = vs.cross(nt);
		A_r.block<3, 1>(3, 0) = nt;
#ifdef _OPENMP
#pragma omp critical
#endif
		{
			ATA += A_r * A_r.transpose();
			ATb += A_r * r;
		}		
	}
	tend = time(0);
	std::cout << "Option3 took " << difftime(tend, tstart) << " second(s)." << std::endl;

	// Option #4
	// Seperately build J and r
	tstart = time(0);
	A.setZero();
	b.setZero();
#ifdef _OPENMP
#pragma omp parallel for schedule(static)
#endif
	for (int i = 0; i < NCORR; i++) {
		const Eigen::Vector3d &vs = data[i];
		const Eigen::Vector3d &vt = data[i];
		const Eigen::Vector3d &nt = data[i];
		r = (vs - vt).dot(nt);
		Eigen::Vector6d A_r;
		A_r.block<3, 1>(0, 0) = vs.cross(nt);
		A_r.block<3, 1>(3, 0) = nt;
		A.block<1, 6>(i, 0) = A_r;
		b(i) = r;
	}
	AT = A.transpose();
	ATA = AT * A;
	ATb = AT * b;
	tend = time(0);
	std::cout << "Option4 took " << difftime(tend, tstart) << " second(s)." << std::endl;

The output is

Option3 took 43 second(s).
Option4 took 6 second(s).

I had to put omp critical for Option 3 because ATA and ATb should not be accumulated by multiple threads at the same time. Without omp critical, it makes unreliable sum. omp critical actually mess up because only single thread gets writing authorization and other threads need to wait until the writing business is done.

This experiment indicates that how we write parallel code using Eigen::MatrixXd, and indicates why parallel implementation of RGBD odometry is not that fast (related to #51) compared to original code. The Jacobian module in RGBD Odometry is similar to Option 4, and the original implementation is similar to Option 1. As we observed, Option 1 and Option 4 takes 6 seconds.

Do you have any idea how we can write more effective code? I am individually investigating now. The goal is to design 'option 5' that takes just 1~2 seconds.

Reading log files of dense matching

Open3D has PinholeCameraIntrinsic and PinholeCameraTrajectory that are good to read camera information from file. However, we don't have a file IO support for dense matching point cloud pairs. This is essential to GraphOptimization module.

Compilation failure from libpng and libjpeg

There is an compilation failure from Travis CI. Related to #92.

/home/travis/build/IntelVCL/Open3D/src/IO/FileFormat/FilePNG.cpp:36:47: error: ‘png_image’ has not been declared
 void SetPNGImageFromImage(const Image &image, png_image &pngimage)
                                               ^
/home/travis/build/IntelVCL/Open3D/src/IO/FileFormat/FilePNG.cpp: In function ‘void three::{anonymous}::SetPNGImageFromImage(const three::Image&, int&)’:
/home/travis/build/IntelVCL/Open3D/src/IO/FileFormat/FilePNG.cpp:38:11: error: request for member ‘width’ in ‘pngimage’, which is of non-class type ‘int’
  pngimage.width = image.width_;
           ^
/home/travis/build/IntelVCL/Open3D/src/IO/FileFormat/FilePNG.cpp:39:11: error: request for member ‘height’ in ‘pngimage’, which is of non-class type ‘int’
  pngimage.height = image.height_;

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.