Hello! Thank you for your open source! I benefited a lot from your excellent work!

However, when I use offline_runner.py to pre segment the sequence:TUM fr3_walking_xyz or fr3_walking_halfsphere, it's output mapping seems not as perfect as showed in vedio. You can see the pictures listed below : the system performs well at begining as in peciture 1, but in the mid of the sequence, some overlaps appear in the map, you can see 2 computers on desk in picture 2, I guess the moving person influences the RGBD odometery. How can I make it perform just like the vedio? Where i need to make some changes ? I am looking forward to your reply!!!!

Thank you again.
pic1:

pic2:

Hi,

I'd like to know if RGB is used for recognition

So that for example, it can differentiate between two Different Brands of Cereal Boxes which actually shapes the same

Thanks!

I am having issues building maskfusion. Everything has built properly until the final step, after calling cmake in the maskfusion/build directory and then calling make.

I am getting an error in the file maskfusion/Core/Shaders/FeedbackBuffer.cpp

Here is the error message:

maskfusion/Core/Shaders/FeedbackBuffer.cpp: In member function 'void FeedbackBuffer::render(pangolin::OpenGlMatrix, const Matrix4f&, bool, bool)':
maskfusion/Core/Shaders/FeedbackBuffer.cpp:133:45: error: no matching function for call to 'Uniform::Uniform(const char[4], pangolin::OpenGlMatrix&)' drawProgram->setUniform(Uniform("MVP", mvp));

It looks like a fix I can apply is by editing FeedbackBuffer.cpp to manually convert mvp into an Eigen matrix, but I don't know what dimensions it needs to be.

In Core/Shaders/Uniform.h there is no matching function that takes in an OpenGlMatrix as a parameter.

Hi, thank you so much for sharing the code!
I have a question about your code.
In RGBDOdometry.cpp,
lastDepth[i] and nextDepth[i] are updated with

verticesToDepth(vmaps_tmp, destDepths[0], maxDepthRGB);

function which is called from initRGBModel or initRGB function, right?

However, vmaps_tmp is updated in only initICPModel function and not in initICP function, which means vmaps_tmp contains only the vertex information of the projected global model.
That means, both of the lastDepth[i] and nextDepth[i] contains the same information?

Thank you in advance!

Hi, I am learning your project. For I have no fast GPU with enough memory, I use a container service to run maskfusion.
I have created the executable file MaskFusion, when I run the followed command line:
"./MaskFusion -run -l /mnt/home/maskfusion/test/teddy-handover.klg"
show error:
Calibration set to resolution: 640x480, [fx: 528 fy: 528, cx: 320 cy: 240] Reading log file: /mnt/home/maskfusion/test/teddy-handover.klg which has 528 frames. terminate called after throwing an instance of 'std::runtime_error' what(): Pangolin X11: Failed to open X display Aborted (core dumped)
I think the cause of error is the container service can not support GUI display. But I don't know how to modify the code to avoid to display the real-time 3D reconstruction but to save the result.
If you know how to solve my problem, I will be greatful for your response.

ubuntu 16.04 cuda10.0 anaconda tensorflow-gpu 1.4
i7 gtx1080
without freetype-gl-cpp-master
error:
`./MaskFusion -run -l ../teddy-handover.klg
Calibration set to resolution: 640x480, [fx: 528 fy: 528, cx: 320 cy: 240]
Reading log file: ../teddy-handover.klg which has 528 frames.
Framebuffer with requested attributes not available. Using available framebuffer. You may see visual artifacts.GLSL Shader compilation failed: /home/finch/maskfusion-master/Core/Shaders/draw_global_surface.vert:
0:20(10): error: GLSL 4.30 is not supported. Supported versions are: 1.10, 1.20, 1.30, 1.00 ES, and 3.00 ES
0:19(10): error: GLSL 3.30 is not supported. Supported versions are: 1.10, 1.20, 1.30, 1.00 ES, and 3.00 ES

GLSL Shader compilation failed: /home/finch/maskfusion-master/Core/Shaders/sample.geom:
0:19(10): error: GLSL 3.30 is not supported. Sup
段错误 (核心已转储)
`
any suggestion?

The python of ubuntu16.04 is 3.5，it will make error when use "virtualenv python-environment".
So i install python3.6 ,and build this code.
When i run it with"./Maskfusion -l teddy.klg -run",the error comes:"
Instructions for updating:
Use tf.cast instead.
Python error indicator is set:
Traceback (most recent call last):
File "/home/finch/fusion/maskfusion-master/build/GUI/MaskRCNN.py", line 96, in
model.load_weights(model_path, by_name=True)
File "/home/finch/fusion/maskfusion-master/deps/Mask_RCNN/mrcnn/model.py", line 2130, in load_weights
saving.load_weights_from_hdf5_group_by_name(f, layers)
File "/home/finch/fusion/maskfusion-master/python-environment/lib/python3.6/site-packages/keras/engine/saving.py", line 1224, in load_weights_from_hdf5_group_by_name
original_keras_version = f.attrs['keras_version'].decode('utf8')
AttributeError: 'str' object has no attribute 'decode'
terminate called after throwing an instance of 'std::runtime_error'
what(): Could not open MaskRCNN module.
已放弃 (核心已转储)"

I am trying to run MaskFusion with the teddy-handover.klg sequence from Co-Fusion available at http://visual.cs.ucl.ac.uk/pubs/cofusion/data/teddy-handover.klg. In order to run it, I needed to change line 52 of maskfusion/build/GUI/MaskRCNN.py as following:

from keras.backend.tensorflow_backend import set_session, clear_session

When I run it with the command "./MaskFusion -l ../../teddy-handover.klg", a window appears, but when I click the "Pause" option to start tracking, the following error occurs:

/home/jpsml/maskfusion/Core/Cuda/cudafuncs.cu(646) : cudaSafeCall() Runtime API error : invalid texture reference.
/home/jpsml/maskfusion/Core/GPUTexture.cpp(73) : cudaSafeCall() Runtime API error : invalid texture reference.

I am using Ubuntu 19.04, CUDA 10.0, TensorFlow 1.14, Keras 2.3.

You can find below the complete log:

jpsml@jpsml-desktop:~/maskfusion/build/GUI$ ./MaskFusion -l ../../teddy-handover.klg
Calibration set to resolution: 640x480, [fx: 528 fy: 528, cx: 320 cy: 240]
Reading log file: ../../teddy-handover.klg which has 528 frames.
Initialised MainController. Frame resolution is set to: 640x480
Exporting results to: ../../teddy-handover.klg-export//

• Initialising MaskRCNN (thread: 139722768774912) ...
• Loading module...
  /home/jpsml/maskfusion/python-environment/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
  /home/jpsml/maskfusion/python-environment/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
  /home/jpsml/maskfusion/python-environment/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
  /home/jpsml/maskfusion/python-environment/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
  /home/jpsml/maskfusion/python-environment/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
  /home/jpsml/maskfusion/python-environment/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
  Created model with max number of vertices: 9437184
  Initialised multi-object fusion (main-thread: 139723769482624)

• The background model can have up to 9437184 surfel (3072x3072)
• Object models can have up to 1048576 surfel (1024x1024)
• Using GPU 0 for SLAM system and GPU 0 for MaskRCNN
• Using frame-queue of size: 30
  /home/jpsml/maskfusion/python-environment/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
  /home/jpsml/maskfusion/python-environment/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
  /home/jpsml/maskfusion/python-environment/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
  /home/jpsml/maskfusion/python-environment/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
  /home/jpsml/maskfusion/python-environment/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
  /home/jpsml/maskfusion/python-environment/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
  WARNING:tensorflow:From /home/jpsml/maskfusion/build/GUI/MaskRCNN.py:47: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

WARNING:tensorflow:From /home/jpsml/maskfusion/build/GUI/MaskRCNN.py:51: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2019-09-20 11:11:39.509898: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-09-20 11:11:39.533194: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3696000000 Hz
2019-09-20 11:11:39.534004: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f13b9042da0 executing computations on platform Host. Devices:
2019-09-20 11:11:39.534020: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): ,
2019-09-20 11:11:39.534089: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2019-09-20 11:11:39.534157: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-09-20 11:11:39.534591: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.635
pciBusID: 0000:01:00.0
2019-09-20 11:11:39.534777: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2019-09-20 11:11:39.535622: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
2019-09-20 11:11:39.536360: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0
2019-09-20 11:11:39.536554: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0
2019-09-20 11:11:39.537570: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0
2019-09-20 11:11:39.538342: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0
2019-09-20 11:11:39.538395: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcudnn.so.7'; dlerror: libcudnn.so.7: cannot open shared object file: No such file or directory
2019-09-20 11:11:39.538401: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1663] Cannot dlopen some GPU libraries. Skipping registering GPU devices...
2019-09-20 11:11:39.538463: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-20 11:11:39.538471: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187] 0
2019-09-20 11:11:39.538474: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0: N
2019-09-20 11:11:39.539481: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-09-20 11:11:39.539973: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f13ba99a3c0 executing computations on platform CUDA. Devices:
2019-09-20 11:11:39.539985: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): GeForce RTX 2080 Ti, Compute Capability 7.5
Using TensorFlow backend.

Configurations:
BACKBONE resnet101
BACKBONE_STRIDES [4, 8, 16, 32, 64]
BATCH_SIZE 1
BBOX_STD_DEV [0.1 0.1 0.2 0.2]
COMPUTE_BACKBONE_SHAPE None
DETECTION_MAX_INSTANCES 100
DETECTION_MIN_CONFIDENCE 0.7
DETECTION_NMS_THRESHOLD 0.3
FPN_CLASSIF_FC_LAYERS_SIZE 1024
GPU_COUNT 1
GRADIENT_CLIP_NORM 5.0
IMAGES_PER_GPU 1
IMAGE_CHANNEL_COUNT 3
IMAGE_MAX_DIM 1024
IMAGE_META_SIZE 93
IMAGE_MIN_DIM 800
IMAGE_MIN_SCALE 0
IMAGE_RESIZE_MODE square
IMAGE_SHAPE [1024 1024 3]
LEARNING_MOMENTUM 0.9
LEARNING_RATE 0.001
LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0}
MASK_POOL_SIZE 14
MASK_SHAPE [28, 28]
MAX_GT_INSTANCES 100
MEAN_PIXEL [123.7 116.8 103.9]
MINI_MASK_SHAPE (56, 56)
NAME coco
NUM_CLASSES 81
POOL_SIZE 7
POST_NMS_ROIS_INFERENCE 1000
POST_NMS_ROIS_TRAINING 2000
PRE_NMS_LIMIT 6000
ROI_POSITIVE_RATIO 0.33
RPN_ANCHOR_RATIOS [0.5, 1, 2]
RPN_ANCHOR_SCALES (32, 64, 128, 256, 512)
RPN_ANCHOR_STRIDE 1
RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2]
RPN_NMS_THRESHOLD 0.7
RPN_TRAIN_ANCHORS_PER_IMAGE 256
STEPS_PER_EPOCH 1000
TOP_DOWN_PYRAMID_SIZE 256
TRAIN_BN False
TRAIN_ROIS_PER_IMAGE 200
USE_MINI_MASK True
USE_RPN_ROIS True
VALIDATION_STEPS 50
WEIGHT_DECAY 0.0001

WARNING:tensorflow:From /home/jpsml/maskfusion/python-environment/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:4070: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead.

WARNING:tensorflow:From /home/jpsml/maskfusion/deps/Mask_RCNN/mrcnn/model.py:341: The name tf.log is deprecated. Please use tf.math.log instead.

WARNING:tensorflow:From /home/jpsml/maskfusion/deps/Mask_RCNN/mrcnn/model.py:399: add_dispatch_support..wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
WARNING:tensorflow:From /home/jpsml/maskfusion/deps/Mask_RCNN/mrcnn/model.py:423: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version.
Instructions for updating:
box_ind is deprecated, use box_indices instead
WARNING:tensorflow:From /home/jpsml/maskfusion/deps/Mask_RCNN/mrcnn/model.py:720: The name tf.sets.set_intersection is deprecated. Please use tf.sets.intersection instead.

WARNING:tensorflow:From /home/jpsml/maskfusion/deps/Mask_RCNN/mrcnn/model.py:722: The name tf.sparse_tensor_to_dense is deprecated. Please use tf.sparse.to_dense instead.

WARNING:tensorflow:From /home/jpsml/maskfusion/deps/Mask_RCNN/mrcnn/model.py:772: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
2019-09-20 11:11:44.478694: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set. If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU. To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.

• Initialised MaskRCNN
• MaskRCNN got first data -- starting loop.
  WARNING:tensorflow:From /home/jpsml/maskfusion/python-environment/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:422: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.

/home/jpsml/maskfusion/Core/Cuda/cudafuncs.cu(646) : cudaSafeCall() Runtime API error : invalid texture reference.
/home/jpsml/maskfusion/Core/GPUTexture.cpp(73) : cudaSafeCall() Runtime API error : invalid texture reference.

Do you know how should I proceed in order to run it? Thank you.

Recently I tried to run the maskfusion s according to the author's steps.But I have But I was entangled in a problem which was showed in the picture below.

My cuda and cudnn versions are respectively 9.0 and v7 as i showed in the following picture

However when I Input nvidia-smi ，it showed that I had a cuda diver 10.1 and I didn't know if this would affect.

the following is my tensorflow version

By the way ,my maskrcnn can run successfully !

I checked it online, the segment fault seems to be an error caused by illegal access to memory, but I have been following the steps.The process of make is also very smooth.

I have been stuck in this place for a long time, I hope I can get help from the big guys.

Thank you very much!
@martinruenz @adrianknobloch

Hello guys,

MaskFusion is really awesome according to your homepage video. Would you mind merging MaskFusion with some other famous 3D repositories? Just like OpenMVG, OpenMVS and Open3D. Open3D has a mature maintain team and initialized from RealSense cameras. It has a great framework but its functionality is not so wonderful as MaskFusion. While OpenMVG and OpenMVS are good at utilizing RGB images to make 3D model. They are all open on github.

MaskFusion has too few stars compared to its functionality. It can shock the world.

Best Wishes,
Wenyin

I am learning your code,could you let me know which function deal the geometric segmentation

error: GLSL 3.30 is not supported. Supported versions are: 1.10, 1.20, 1.30, 1.40, 1.00 ES, and 3.00 ES
my environment: ubuntu18.04 cuda-10.0 tensorflow-1.13.2 kears-2.2 OpenGL3.1 and 3.3

How can I solve my problem? Please help me

I used the tensorflow1.9.0-gpu cuda9.0 and python2. Does the code support it?

When I execute the last make step of ''make -j8'' , I had meet the error that /home/sheng/maskfusion-master/Core/Segmentation/../Utils/BoundingBox.h:46:50: error: no matching function for call to ‘BoundingBox::BoundingBox()’
...........
...........
...........
[ 46%] Building CXX object Core/CMakeFiles/libmaskfusion.dir/Utils/Gnuplot.cpp.o
[ 48%] Building CXX object Core/CMakeFiles/libmaskfusion.dir/Utils/Parse.cpp.o
[ 51%] Building CXX object Core/CMakeFiles/libmaskfusion.dir/Utils/RGBDOdometry.cpp.o
[ 53%] Building CXX object Core/CMakeFiles/libmaskfusion.dir/Shaders/ComputePack.cpp.o
Core/CMakeFiles/libmaskfusion.dir/build.make:155: recipe for target 'Core/CMakeFiles/libmaskfusion.dir/MaskFusion.cpp.o' failed
make[2]: *** [Core/CMakeFiles/libmaskfusion.dir/MaskFusion.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
CMakeFiles/Makefile2:117: recipe for target 'Core/CMakeFiles/libmaskfusion.dir/all' failed
make[1]: *** [Core/CMakeFiles/libmaskfusion.dir/all] Error 2
Makefile:83: recipe for target 'all' failed
make: *** [all] Error 2

How can I solve it and have a correct build ?
Thank you for your enthusiastic answer !

I'm getting a ton of errors like these for different VarValueT:

/home/user/repos/maskfusion/maskfusion/GUI/Tools/GUI.h: In lambda function:
/home/user/repos/maskfusion/maskfusion/GUI/Tools/GUI.h:241:70: error: ‘class std::shared_ptr<pangolin::VarValueT<bool> >’ has no member named ‘Set’; did you mean ‘get’?
  241 |     pangolin::RegisterKeyPressCallback('w', [&]() { saveCloud->Ref().Set(true); });

Probably Pangolin was changed/refactored? Could you say from which commit you pulled Pangolin in your working version, so I can downgrade Pangolin to a compatible one.

Hi, I tried the latest version of maskfusion and got this message:
[ 77%] Built target MaskFusionTools
Scanning dependencies of target MaskFusion
[ 79%] Building CXX object GUI/CMakeFiles/MaskFusion.dir/Main.cpp.o
[ 81%] Building CXX object GUI/CMakeFiles/MaskFusion.dir/MainController.cpp.o
/home/titan/maskfusion/GUI/MainController.cpp: In constructor ‘MainController::MainController(int, char**)’:
/home/titan/maskfusion/GUI/MainController.cpp:277:45: error: conversion from ‘toml::basic_value<toml::discard_comments, std::unordered_map, std::vector>’ to non-scalar type ‘toml::table {aka std::unordered_map<std::__cxx11::basic_string, toml::basic_value<toml::discard_comments, std::unordered_map, std::vector> >}’ requested
toml::table tomlConfig = toml::parse("config.toml");
^
/home/titan/maskfusion/GUI/MainController.cpp:278:45: error: ‘Table’ is not a member of ‘toml’
const auto tomlMaskRCNN = toml::gettoml::Table(tomlConfig.at("MaskRCNN"));
^
/home/titan/maskfusion/GUI/MainController.cpp:278:45: error: ‘Table’ is not a member of ‘toml’
/home/titan/maskfusion/GUI/MainController.cpp:278:83: error: no matching function for call to ‘get(std::unordered_map<std::__cxx11::basic_string, toml::basic_value<toml::discard_comments, std::unordered_map, std::vector> >::mapped_type&)’
const auto tomlMaskRCNN = toml::gettoml::Table(tomlConfig.at("MaskRCNN"));
^
In file included from /home/titan/maskfusion/deps/toml11/toml.hpp:39:0,
from /home/titan/maskfusion/GUI/MainController.cpp:32:
/home/titan/maskfusion/deps/toml11/toml/get.hpp:19:1: note: candidate: template<class T, class C, template<class ...> class M, template<class ...> class V> toml::detail::enable_if_t<toml::detail::is_exact_toml_type<T, toml::basic_value<C, Tb, A> >::value, T>& toml::get(toml::basic_value<C, Tb, A>&)
get(basic_value<C, M, V>& v)
^
/home/titan/maskfusion/deps/toml11/toml/get.hpp:19:1: note: template argument deduction/substitution failed:
/home/titan/maskfusion/GUI/MainController.cpp:278:83: error: template argument 1 is invalid
const auto tomlMaskRCNN = toml::gettoml::Table(tomlConfig.at("MaskRCNN"));
^
In file included from /home/titan/maskfusion/deps/toml11/toml.hpp:39:0,
from /home/titan/maskfusion/GUI/MainController.cpp:32:
/home/titan/maskfusion/deps/toml11/toml/get.hpp:27:1: note: candidate: template<class T, class C, template<class ...> class M, template<class ...> class V> toml::detail::enable_if_t<toml::detail::is_exact_toml_type<T, toml::basic_value<C, Tb, A> >::value, T>& toml::get(const toml::basic_value<C, Tb, A>&)
get(const basic_value<C, M, V>& v)
^
/home/titan/maskfusion/deps/toml11/toml/get.hpp:27:1: note: template argument deduction/substitution failed:
/home/titan/maskfusion/GUI/MainController.cpp:278:83: error: template argument 1 is invalid
const auto tomlMaskRCNN = toml::gettoml::Table(tomlConfig.at("MaskRCNN"));
^
In file included from /home/titan/maskfusion/deps/toml11/toml.hpp:39:0,
from /home/titan/maskfusion/GUI/MainController.cpp:32:
/home/titan/maskfusion/deps/toml11/toml/get.hpp:35:1: note: candidate: template<class T, class C, template<class ...> class M, template<class ...> class V> toml::detail::enable_if_t<toml::detail::is_exact_toml_type<T, toml::basic_value<C, Tb, A> >::value, T>&& toml::get(toml::basic_value<C, Tb, A>&&)
get(basic_value<C, M, V>&& v)
^
/home/titan/maskfusion/deps/toml11/toml/get.hpp:35:1: note: template argument deduction/substitution failed:
/home/titan/maskfusion/GUI/MainController.cpp:278:83: error: template argument 1 is invalid
const auto tomlMaskRCNN = toml::gettoml::Table(tomlConfig.at("MaskRCNN"));
^
In file included from /home/titan/maskfusion/deps/toml11/toml.hpp:39:0,
from /home/titan/maskfusion/GUI/MainController.cpp:32:
/home/titan/maskfusion/deps/toml11/toml/get.hpp:46:1: note: candidate: template<class T, class C, template<class ...> class M, template<class ...> class V> toml::detail::enable_if_t<std::is_same<T, toml::basic_value<C, Tb, A> >::value, T>& toml::get(toml::basic_value<C, Tb, A>&)
get(basic_value<C, M, V>& v)
^
/home/titan/maskfusion/deps/toml11/toml/get.hpp:46:1: note: template argument deduction/substitution failed:
/home/titan/maskfusion/GUI/MainController.cpp:278:83: error: template argument 1 is invalid
const auto tomlMaskRCNN = toml::gettoml::Table(tomlConfig.at("MaskRCNN"));
^
In file included from /home/titan/maskfusion/deps/toml11/toml.hpp:39:0,
from /home/titan/maskfusion/GUI/MainController.cpp:32:
/home/titan/maskfusion/deps/toml11/toml/get.hpp:54:1: note: candidate: template<class T, class C, template<class ...> class M, template<class ...> class V> toml::detail::enable_if_t<std::is_same<T, toml::basic_value<C, Tb, A> >::value, T>& toml::get(const toml::basic_value<C, Tb, A>&)
get(const basic_value<C, M, V>& v)
^
/home/titan/maskfusion/deps/toml11/toml/get.hpp:54:1: note: template argument deduction/substitution failed:
/home/titan/maskfusion/GUI/MainController.cpp:278:83: error: template argument 1 is invalid
const auto tomlMaskRCNN = toml::gettoml::Table(tomlConfig.at("MaskRCNN"));
^
In file included from /home/titan/maskfusion/deps/toml11/toml.hpp:39:0,
from /home/titan/maskfusion/GUI/MainController.cpp:32:
/home/titan/maskfusion/deps/toml11/toml/get.hpp:62:1: note: candidate: template<class T, class C, template<class ...> class M, template<class ...> class V> toml::detail::enable_if_t<std::is_same<T, toml::basic_value<C, Tb, A> >::value, T>&& toml::get(toml::basic_value<C, Tb, A>&&)
get(basic_value<C, M, V>&& v)
^
/home/titan/maskfusion/deps/toml11/toml/get.hpp:62:1: note: template argument deduction/substitution failed:
/home/titan/maskfusion/GUI/MainController.cpp:278:83: error: template argument 1 is invalid
const auto tomlMaskRCNN = toml::gettoml::Table(tomlConfig.at("MaskRCNN"));
^
In file included from /home/titan/maskfusion/deps/toml11/toml.hpp:39:0,
from /home/titan/maskfusion/GUI/MainController.cpp:32:
/home/titan/maskfusion/deps/toml11/toml/get.hpp:75:1: note: candidate: template<class T, class C, template<class ...> class M, template<class ...> class V> toml::detail::enable_if_t<toml::detail::conjunction<toml::detail::is_basic_value, toml::detail::negation<std::is_same<T, toml::basic_value<C, Tb, A> > > >::value, T> toml::get(const toml::basic_value<C, Tb, A>&)
get(const basic_value<C, M, V>& v)
^
/home/titan/maskfusion/deps/toml11/toml/get.hpp:75:1: note: template argument deduction/substitution failed:
/home/titan/maskfusion/GUI/MainController.cpp:278:83: error: template argument 1 is invalid
const auto tomlMaskRCNN = toml::gettoml::Table(tomlConfig.at("MaskRCNN"));
^
In file included from /home/titan/maskfusion/deps/toml11/toml.hpp:39:0,
from /home/titan/maskfusion/GUI/MainController.cpp:32:
/home/titan/maskfusion/deps/toml11/toml/get.hpp:91:1: note: candidate: template<class T, class C, template<class ...> class M, template<class ...> class V> toml::detail::enable_if_t<toml::detail::conjunction<std::is_integral<_Tp>, toml::detail::negation<std::is_same<T, bool> >, toml::detail::negation<toml::detail::is_exact_toml_type<T, toml::basic_value<C, Tb, A> > > >::value, T> toml::get(const toml::basic_value<C, Tb, A>&)
get(const basic_value<C, M, V>& v)
^
/home/titan/maskfusion/deps/toml11/toml/get.hpp:91:1: note: template argument deduction/substitution failed:
/home/titan/maskfusion/GUI/MainController.cpp:278:83: error: template argument 1 is invalid
const auto tomlMaskRCNN = toml::gettoml::Table(tomlConfig.at("MaskRCNN"));
^
In file included from /home/titan/maskfusion/deps/toml11/toml.hpp:39:0,
from /home/titan/maskfusion/GUI/MainController.cpp:32:
/home/titan/maskfusion/deps/toml11/toml/get.hpp:106:1: note: candidate: template<class T, class C, template<class ...> class M, template<class ...> class V> toml::detail::enable_if_t<toml::detail::conjunction<std::is_floating_point<_Tp>, toml::detail::negation<toml::detail::is_exact_toml_type<T, toml::basic_value<C, Tb, A> > > >::value, T> toml::get(const toml::basic_value<C, Tb, A>&)
get(const basic_value<C, M, V>& v)
^
/home/titan/maskfusion/deps/toml11/toml/get.hpp:106:1: note: template argument deduction/substitution failed:
/home/titan/maskfusion/GUI/MainController.cpp:278:83: error: template argument 1 is invalid
const auto tomlMaskRCNN = toml::gettoml::Table(tomlConfig.at("MaskRCNN"));
^
In file included from /home/titan/maskfusion/deps/toml11/toml.hpp:39:0,
from /home/titan/maskfusion/GUI/MainController.cpp:32:
/home/titan/maskfusion/deps/toml11/toml/get.hpp:118:1: note: candidate: template<class T, class C, template<class ...> class M, template<class ...> class V> toml::detail::enable_if_t<std::is_same<T, std::__cxx11::basic_string >::value, std::__cxx11::basic_string >& toml::get(toml::basic_value<C, Tb, A>&)
get(basic_value<C, M, V>& v)
^
/home/titan/maskfusion/deps/toml11/toml/get.hpp:118:1: note: template argument deduction/substitution failed:
/home/titan/maskfusion/GUI/MainController.cpp:278:83: error: template argument 1 is invalid
const auto tomlMaskRCNN = toml::gettoml::Table(tomlConfig.at("MaskRCNN"));
^
In file included from /home/titan/maskfusion/deps/toml11/toml.hpp:39:0,
from /home/titan/maskfusion/GUI/MainController.cpp:32:
/home/titan/maskfusion/deps/toml11/toml/get.hpp:126:1: note: candidate: template<class T, class C, template<class ...> class M, template<class ...> class V> toml::detail::enable_if_t<std::is_same<T, std::__cxx11::basic_string >::value, std::__cxx11::basic_string >& toml::get(const toml::basic_value<C, Tb, A>&)
get(const basic_value<C, M, V>& v)
^
/home/titan/maskfusion/deps/toml11/toml/get.hpp:126:1: note: template argument deduction/substitution failed:
/home/titan/maskfusion/GUI/MainController.cpp:278:83: error: template argument 1 is invalid
const auto tomlMaskRCNN = toml::gettoml::Table(tomlConfig.at("MaskRCNN"));
^
In file included from /home/titan/maskfusion/deps/toml11/toml.hpp:39:0,
from /home/titan/maskfusion/GUI/MainController.cpp:32:
/home/titan/maskfusion/deps/toml11/toml/get.hpp:134:1: note: candidate: template<class T, class C, template<class ...> class M, template<class ...> class V> toml::detail::enable_if_t<std::is_same<T, std::__cxx11::basic_string >::value, std::__cxx11::basic_string >& toml::get(toml::basic_value<C, Tb, A>&&)
get(basic_value<C, M, V>&& v)
^
/home/titan/maskfusion/deps/toml11/toml/get.hpp:134:1: note: template argument deduction/substitution failed:
/home/titan/maskfusion/GUI/MainController.cpp:278:83: error: template argument 1 is invalid
const auto tomlMaskRCNN = toml::gettoml::Table(tomlConfig.at("MaskRCNN"));
^
In file included from /home/titan/maskfusion/deps/toml11/toml.hpp:39:0,
from /home/titan/maskfusion/GUI/MainController.cpp:32:
/home/titan/maskfusion/deps/toml11/toml/get.hpp:158:1: note: candidate: template<class T, class C, template<class ...> class M, template<class ...> class V> toml::detail::enable_if_t<toml::detail::is_chrono_duration::value, T> toml::get(const toml::basic_value<C, Tb, A>&)
get(const basic_value<C, M, V>& v)
^
/home/titan/maskfusion/deps/toml11/toml/get.hpp:158:1: note: template argument deduction/substitution failed:
/home/titan/maskfusion/GUI/MainController.cpp:278:83: error: template argument 1 is invalid
const auto tomlMaskRCNN = toml::gettoml::Table(tomlConfig.at("MaskRCNN"));
^
In file included from /home/titan/maskfusion/deps/toml11/toml.hpp:39:0,
from /home/titan/maskfusion/GUI/MainController.cpp:32:
/home/titan/maskfusion/deps/toml11/toml/get.hpp:171:1: note: candidate: template<class T, class C, template<class ...> class M, template<class ...> class V> toml::detail::enable_if_t<std::is_same<std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long int, std::ratio<1l, 1000000000l> > >, T>::value, T> toml::get(const toml::basic_value<C, Tb, A>&)
get(const basic_value<C, M, V>& v)
^
/home/titan/maskfusion/deps/toml11/toml/get.hpp:171:1: note: template argument deduction/substitution failed:
/home/titan/maskfusion/GUI/MainController.cpp:278:83: error: template argument 1 is invalid
const auto tomlMaskRCNN = toml::gettoml::Table(tomlConfig.at("MaskRCNN"));
^
In file included from /home/titan/maskfusion/deps/toml11/toml.hpp:39:0,
from /home/titan/maskfusion/GUI/MainController.cpp:32:
/home/titan/maskfusion/deps/toml11/toml/get.hpp:276:1: note: candidate: template<class T, class C, template<class ...> class M, template<class ...> class V> toml::detail::enable_if_t<toml::detail::conjunction<toml::detail::is_container, toml::detail::has_resize_method, toml::detail::negation<toml::detail::is_exact_toml_type<T, toml::basic_value<C, Tb, A> > > >::value, T> toml::get(const toml::basic_value<C, Tb, A>&)
get(const basic_value<C, M, V>& v)
^
/home/titan/maskfusion/deps/toml11/toml/get.hpp:276:1: note: template argument deduction/substitution failed:
/home/titan/maskfusion/GUI/MainController.cpp:278:83: error: template argument 1 is invalid
const auto tomlMaskRCNN = toml::gettoml::Table(tomlConfig.at("MaskRCNN"));
^
In file included from /home/titan/maskfusion/deps/toml11/toml.hpp:39:0,
from /home/titan/maskfusion/GUI/MainController.cpp:32:
/home/titan/maskfusion/deps/toml11/toml/get.hpp:298:1: note: candidate: template<class T, class C, template<class ...> class M, template<class ...> class V> toml::detail::enable_if_t<toml::detail::conjunction<toml::detail::is_container, toml::detail::negation<toml::detail::has_resize_method >, toml::detail::negation<toml::detail::is_exact_toml_type<T, toml::basic_value<C, Tb, A> > > >::value, T> toml::get(const toml::basic_value<C, Tb, A>&)
get(const basic_value<C, M, V>& v)
^
/home/titan/maskfusion/deps/toml11/toml/get.hpp:298:1: note: template argument deduction/substitution failed:
/home/titan/maskfusion/GUI/MainController.cpp:278:83: error: template argument 1 is invalid
const auto tomlMaskRCNN = toml::gettoml::Table(tomlConfig.at("MaskRCNN"));
^
In file included from /home/titan/maskfusion/deps/toml11/toml.hpp:39:0,
from /home/titan/maskfusion/GUI/MainController.cpp:32:
/home/titan/maskfusion/deps/toml11/toml/get.hpp:323:1: note: candidate: template<class T, class C, template<class ...> class M, template<class ...> class V> toml::detail::enable_if_t<toml::detail::is_std_pair::value, T> toml::get(const toml::basic_value<C, Tb, A>&)
get(const basic_value<C, M, V>& v)
^
/home/titan/maskfusion/deps/toml11/toml/get.hpp:323:1: note: template argument deduction/substitution failed:
/home/titan/maskfusion/GUI/MainController.cpp:278:83: error: template argument 1 is invalid
const auto tomlMaskRCNN = toml::gettoml::Table(tomlConfig.at("MaskRCNN"));
^
In file included from /home/titan/maskfusion/deps/toml11/toml.hpp:39:0,
from /home/titan/maskfusion/GUI/MainController.cpp:32:
/home/titan/maskfusion/deps/toml11/toml/get.hpp:357:1: note: candidate: template<class T, class C, template<class ...> class M, template<class ...> class V> toml::detail::enable_if_t<toml::detail::is_std_tuple::value, T> toml::get(const toml::basic_value<C, Tb, A>&)
get(const basic_value<C, M, V>& v)
^
/home/titan/maskfusion/deps/toml11/toml/get.hpp:357:1: note: template argument deduction/substitution failed:
/home/titan/maskfusion/GUI/MainController.cpp:278:83: error: template argument 1 is invalid
const auto tomlMaskRCNN = toml::gettoml::Table(tomlConfig.at("MaskRCNN"));
^
In file included from /home/titan/maskfusion/deps/toml11/toml.hpp:39:0,
from /home/titan/maskfusion/GUI/MainController.cpp:32:
/home/titan/maskfusion/deps/toml11/toml/get.hpp:383:1: note: candidate: template<class T, class C, template<class ...> class M, template<class ...> class V> toml::detail::enable_if_t<toml::detail::conjunction<toml::detail::is_map, toml::detail::negation<toml::detail::is_exact_toml_type<T, toml::basic_value<C, Tb, A> > > >::value, T> toml::get(const toml::basic_value<C, Tb, A>&)
get(const basic_value<C, M, V>& v)
^
/home/titan/maskfusion/deps/toml11/toml/get.hpp:383:1: note: template argument deduction/substitution failed:
/home/titan/maskfusion/GUI/MainController.cpp:278:83: error: template argument 1 is invalid
const auto tomlMaskRCNN = toml::gettoml::Table(tomlConfig.at("MaskRCNN"));
^
In file included from /home/titan/maskfusion/deps/toml11/toml.hpp:39:0,
from /home/titan/maskfusion/GUI/MainController.cpp:32:
/home/titan/maskfusion/deps/toml11/toml/get.hpp:409:1: note: candidate: template<class T, class C, template<class ...> class M, template<class ...> class V> toml::detail::enable_if_t<toml::detail::conjunction<toml::detail::negation<toml::detail::is_exact_toml_type<T, toml::basic_value<C, Tb, A> > >, toml::detail::has_from_toml_method<T, C, M, V>, std::is_default_constructible<_Tp> >::value, T> toml::get(const toml::basic_value<C, Tb, A>&)
get(const basic_value<C, M, V>& v)
^
/home/titan/maskfusion/deps/toml11/toml/get.hpp:409:1: note: template argument deduction/substitution failed:
/home/titan/maskfusion/GUI/MainController.cpp:278:83: error: template argument 1 is invalid
const auto tomlMaskRCNN = toml::gettoml::Table(tomlConfig.at("MaskRCNN"));
^
In file included from /home/titan/maskfusion/deps/toml11/toml.hpp:39:0,
from /home/titan/maskfusion/GUI/MainController.cpp:32:
/home/titan/maskfusion/deps/toml11/toml/get.hpp:418:3: note: candidate: template<class T, class C, template<class ...> class M, template<class ...> class V, long unsigned int S> T toml::get(const toml::basic_value<C, Tb, A>&)
T get(const basic_value<C, M, V>& v)
^
/home/titan/maskfusion/deps/toml11/toml/get.hpp:418:3: note: template argument deduction/substitution failed:
/home/titan/maskfusion/GUI/MainController.cpp:278:83: error: template argument 1 is invalid
const auto tomlMaskRCNN = toml::gettoml::Table(tomlConfig.at("MaskRCNN"));
^
GUI/CMakeFiles/MaskFusion.dir/build.make:86: recipe for target 'GUI/CMakeFiles/MaskFusion.dir/MainController.cpp.o' failed
make[2]: *** [GUI/CMakeFiles/MaskFusion.dir/MainController.cpp.o] Error 1
CMakeFiles/Makefile2:211: recipe for target 'GUI/CMakeFiles/MaskFusion.dir/all' failed
make[1]: *** [GUI/CMakeFiles/MaskFusion.dir/all] Error 2
Makefile:83: recipe for target 'all' failed
make: *** [all] Error 2

I git the latest toml at https://github.com/ToruNiina/toml11, is it because of the wrong version of toml or something else?

Best Wishes!

I use

• kinect v2
• ./MaskFusion -v2

I saw this error when quitting my program, and the Kinect camera still running after quitting the program.

Error:
Exception ignored in: <module 'threading' from '/usr/lib/python3.5/threading.py'> Traceback (most recent call last): File "/usr/lib/python3.5/threading.py", line 1290, in _shutdown _main_thread._delete() File "/usr/lib/python3.5/threading.py", line 1013, in _delete del _active[get_ident()] KeyError: 139809078914176 terminate called without an active exception Aborted (core dumped)
Also, how can I set PYTHON_VE_PATH when I use Anaconda.

Thanks

segmentation fault (core dumped)

Have anyone meet this problem before?

Hello,

Till now, I had built the whole project and make a test video data for running the MaskFusion. But It takes place a problem of FrameBuffer the error infor is as follows:

startBufferLoop()
Started data-buffering thread with id: 139846288672512
Framebuffer with requested attributes not available. Using available framebuffer. You may see visual artifacts.
X11 Error: GLXBadFBConfig
Pangolin X11: Indirect GLX rendering context obtained

unfortunately, I had not found some solutions which will work on the problem. Should I change some configurations?

By the way, in the readme.md it is mentioned some CMAKE options and how to set the value of each value?

Thanks for your reply

@martinruenz Hi Martin, thanks for the great work and clear instructions, I successfully build the system. However, when I ran the command:
./MaskFusion -run -l /home/charlie/Downloads/teddy-handover.klg

There are no segmentation results displayed on the screen. Could you help to point out the potential cause of this issue?

• output of the command

Calibration set to resolution: 640x480, [fx: 528 fy: 528, cx: 320 cy: 240]
Reading log file: /home/charlie/Downloads/teddy-handover.klg which has 528 frames. 
Initialised MainController. Frame resolution is set to: 640x480
Exporting results to: /home/charlie/Downloads/teddy-handover.klg-export//
* Initialising MaskRCNN (thread: 140646186743552) ...
 * Loading module...
/home/charlie/python-environment/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/charlie/python-environment/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/charlie/python-environment/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:521: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/charlie/python-environment/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:522: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/charlie/python-environment/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:523: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/charlie/python-environment/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:528: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
Created model with max number of vertices: 9437184
Initialised multi-object fusion (main-thread: 140647066766720)
- The background model can have up to 9437184 surfel (3072x3072)
- Object models can have up to 1048576 surfel (1024x1024)
- Using GPU unspecified for SLAM system and GPU 0 for MaskRCNN
- Using frame-queue of size: 30
2020-06-27 01:31:23.576753: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
Using TensorFlow backend.
Your GPU "GeForce RTX 2060" isn't in the ICP Step performance database, please add it
Your GPU "GeForce RTX 2060" isn't in the RGB Step performance database, please add it
Your GPU "GeForce RTX 2060" isn't in the RGB Res performance database, please add it
Your GPU "GeForce RTX 2060" isn't in the SO3 Step performance database, please add it

Configurations:
BACKBONE                       resnet101
BACKBONE_STRIDES               [4, 8, 16, 32, 64]
BATCH_SIZE                     1
BBOX_STD_DEV                   [0.1 0.1 0.2 0.2]
COMPUTE_BACKBONE_SHAPE         None
DETECTION_MAX_INSTANCES        100
DETECTION_MIN_CONFIDENCE       0.7
DETECTION_NMS_THRESHOLD        0.3
FPN_CLASSIF_FC_LAYERS_SIZE     1024
GPU_COUNT                      1
GRADIENT_CLIP_NORM             5.0
IMAGES_PER_GPU                 1
IMAGE_CHANNEL_COUNT            3
IMAGE_MAX_DIM                  1024
IMAGE_META_SIZE                93
IMAGE_MIN_DIM                  800
IMAGE_MIN_SCALE                0
IMAGE_RESIZE_MODE              square
IMAGE_SHAPE                    [1024 1024    3]
LEARNING_MOMENTUM              0.9
LEARNING_RATE                  0.001
LOSS_WEIGHTS                   {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0}
MASK_POOL_SIZE                 14
MASK_SHAPE                     [28, 28]
MAX_GT_INSTANCES               100
MEAN_PIXEL                     [123.7 116.8 103.9]
MINI_MASK_SHAPE                (56, 56)
NAME                           coco
NUM_CLASSES                    81
POOL_SIZE                      7
POST_NMS_ROIS_INFERENCE        1000
POST_NMS_ROIS_TRAINING         2000
PRE_NMS_LIMIT                  6000
ROI_POSITIVE_RATIO             0.33
RPN_ANCHOR_RATIOS              [0.5, 1, 2]
RPN_ANCHOR_SCALES              (32, 64, 128, 256, 512)
RPN_ANCHOR_STRIDE              1
RPN_BBOX_STD_DEV               [0.1 0.1 0.2 0.2]
RPN_NMS_THRESHOLD              0.7
RPN_TRAIN_ANCHORS_PER_IMAGE    256
STEPS_PER_EPOCH                1000
TOP_DOWN_PYRAMID_SIZE          256
TRAIN_BN                       False
TRAIN_ROIS_PER_IMAGE           200
USE_MINI_MASK                  True
USE_RPN_ROIS                   True
VALIDATION_STEPS               50
WEIGHT_DECAY                   0.0001


* Initialised MaskRCNN
* MaskRCNN got first data -- starting loop.
Segmentation fault (core dumped)

I also tried the Core/Segmentation/MaskRCNN/offline_runner.py script, and it worked pretty well. So the issue might not be related to mask rcnn and tf.

Btw, I would appreciate it if you could tell me a way to generate a semantic point cloud just like the one in your impressive video?

Hi,
Could you give me some more details on how to obtain and save the 3D global semantic map as you shown in the video (if I do not concerned about running online or offline)? I haven't found the corresponding parameter to do this, so is it done any other way?

Besides, I am not quite clear about some points. Here's what I think and please correct me if I'm wrong:
After the 3D global semantic map is reconstructed and a new frame comes in, Mask-RCNN is applied to it and with the help of depth map, the segmentation of this frame is obtained. The multiple objects recognized in this frame will try to find their corresponding 3D models in the 3D global map individually (I'm not sure about this process), and then "If the association is successful the 3D model is updated, otherwise yet another 3D model might be created" as you answered in another question.
So, if two adjacent frames comes in, the segmentation information flows as follows:
segmentation in frame 1 → corresponding 3D model in reconstructed map (if exists)
segmentation in frame 2 → corresponding 3D model in reconstructed map (if exists)
And if the corresponding 3D model is the same, the object in two adjacent frames is tracked and mapped. Is that how it works? It seems require heavy computation and I'm quite not sure about this. Will the traditional feature points help tracking and mapping?

Thanks a lot :-D
Have a wonderful day~

As soon as I try to do a "Step" in the GUI I get the following error:

Loading camera parameters from file: /home/user/data/2i_test1/calibration.txt
Calibration set to resolution: 640x480, [fx: 1431.95 fy: 949.361, cx: 1431.95 cy: 542.731]
Found 438 mask images.
Opened dataset with 438 frames, starting with index: 0 The dataset also provides masks.
startBufferLoop()
Trying to create window
Started data-buffering thread with id: 139982274987776
Created Window
Initialised MainController. Frame resolution is set to: 640x480
Exporting results to: /home/user/data/2i_test1/-export//
Created model with max number of vertices: 9437184
Initialised multi-object fusion (main-thread: 139982703030976)
- The background model can have up to 9437184 surfel (3072x3072)
- Object models can have up to 1048576 surfel (1024x1024)
- Using GPU 1 for SLAM system and GPU 1 for MaskRCNN
- Using frame-queue of size: 0
/home/user/repos/maskfusion/maskfusion/Core/Cuda/cudafuncs.cu(646) : cudaSafeCall() Runtime API error : context is destroyed.
/home/user/repos/maskfusion/maskfusion/Core/GPUTexture.cpp(73) : cudaSafeCall() Runtime API error : context is destroyed.
Segmentation fault (core dumped)

Ideas what causes it?

Hi, thanks for sharing your work. I have a problem when I built the whole project, actually in building freetype-gl-cpp.

When I run "cmake -DBUILD_EXAMPLES=OFF -DCMAKE_INSTALL_PREFIX="pwd/../install" -DCMAKE_BUILD_TYPE=Release .."

It output the followings:
CMake Warning at CMakeLists.txt:18 (find_package):
By not providing "FindFontConfig.cmake" in CMAKE_MODULE_PATH this project
has asked CMake to find a package configuration file provided by
"FontConfig", but CMake did not find one.

Could not find a package configuration file provided by "FontConfig" with
any of the following names:

FontConfigConfig.cmake
fontconfig-config.cmake

Add the installation prefix of "FontConfig" to CMAKE_PREFIX_PATH or set
"FontConfig_DIR" to a directory containing one of the above files. If
"FontConfig" provides a separate development package or SDK, be sure it has been installed.

CMake Error at CMakeLists.txt:37 (add_subdirectory):
add_subdirectory given source "external/freetype-gl" which is not an
existing directory.

CMake Error at CMakeLists.txt:24 (file):
file failed to open for reading (No such file or directory):

/home/zh/Documents/maskfusion/deps/freetype-gl-cpp/external/freetype-gl/shaders/text.vert

Call Stack (most recent call first):
CMakeLists.txt:40 (make_includable)

CMake Error at CMakeLists.txt:24 (file):
file failed to open for reading (No such file or directory):

/home/zh/Documents/maskfusion/deps/freetype-gl-cpp/external/freetype-gl/shaders/text.frag

Call Stack (most recent call first):
CMakeLists.txt:41 (make_includable)

I am using Ubuntu16.04 and the version of cmake is 3.16
I really hope you can help me. Thanks a lot!

Hello, @martinruenz. Have you met this problem?
When I ran MaskFusion with two GPUs, there had this problem.
First, it will stop before the end.(This data is in the bottom-left of the GUI )

Log 723/755

Second, when I closed the GUI, in the terminal, there had this error.

Exception ignored in: <module 'threading' from '/usr/lib/python3.5/threading.py'>
Traceback (most recent call last):
File "/usr/lib/python3.5/threading.py", line 1290, in _shutdown
_main_thread._delete()
File "/usr/lib/python3.5/threading.py", line 1013, in _delete
del _active[get_ident()]
KeyError: 140619725679744
terminate called without an active exception
已放弃 (核心已转储)

I think maybe the thread of MaskRCNN is still running, so the thread of MaskRCNN can't be closed. But I have no idea to solve it.
What should I do to solve this problem?
Thank you!

Hi,

Finally managed to get MaskFusion to open.

Turns out Kinect v1 & v2 is supported, althrough v2 requries a additional installation of libfreenect2

Asus Xtion Live (with RGB) is also supported and taken as v1

To run Asus Xtion Live or Kinect v1: (tested with Xtion)

./MaskFusion -run -v1

To run Kinect v2: (need to set WITH_FREENECT2 during build) (not tested yet)

./MaskFusion -run -v2

Hi, thanks for the open source project.

It seems that we cannot reproduce results on the tum fr3_household dataset with the default parameters. Using less strict thresholds can give us a reasonable trajectory without object tracking module, but nothing seems to work with object tracking module turned on.

Would you like to kindly share the parameter you were using for this sequence for your ISMAR paper? Thanks.

Hi there,
Thanks for your contribution.
Could you explain how did you get annotated data and train RGBD benchmark dataset?
As I know, TUM dataset only contains trajectory ground truth camera pose without labelled frames.

I run it "./MaskFusion -l /home/finch/Downloads/teddy-handover.klg -exportdir ./res -run"
it can 3d reconstruct and output point cloud but it cannot distinguished any thing.The third window is always black.
"MaskRCNN.py.in" files 98lines:
filter_classes = [class_names.index(x) for x in filter_classes]
#filter_classes = ['person', 'skateboard', 'teddy bear']

I tried those two code,they all failed.

BYW,how to use offline_runner.py file?
what is /path/to/rgb/frames ?where can i download ?

Hi, thanks so much for sharing such a great project.

I've built MaskFusion following all steps but I didn't manage to run it with my D435i camera. Is D435i supported by MaskFusion?

Your help is really appreciated,
Thanks.

/home/titan/maskfusion/GUI/MainController.cpp: In constructor ‘MainController::MainController(int, char**)’:
/home/titan/maskfusion/GUI/MainController.cpp:277:45: error: conversion from ‘toml::basic_value<toml::discard_comments, std::unordered_map, std::vector>’ to non-scalar type ‘toml::table {aka std::unordered_map<std::__cxx11::basic_string, toml::basic_value<toml::discard_comments, std::unordered_map, std::vector> >}’ requested
toml::table tomlConfig = toml::parse("config.toml");
^
/home/titan/maskfusion/GUI/MainController.cpp:278:45: error: ‘Table’ is not a member of ‘toml’
const auto tomlMaskRCNN = toml::gettoml::Table(tomlConfig.at("MaskRCNN"));
^
/home/titan/maskfusion/GUI/MainController.cpp:278:45: error: ‘Table’ is not a member of ‘toml’
/home/titan/maskfusion/GUI/MainController.cpp:278:83: error: no matching function for call to ‘get(std::unordered_map<std::__cxx11::basic_string, toml::basic_value<toml::discard_comments, std::unordered_map, std::vector> >::mapped_type&)’
const auto tomlMaskRCNN = toml::gettoml::Table(tomlConfig.at("MaskRCNN"));
^
In file included from /home/titan/maskfusion/deps/toml11/toml.hpp:39:0,
from /home/titan/maskfusion/GUI/MainController.cpp:32:
/home/titan/maskfusion/deps/toml11/toml/get.hpp:19:1: note: candidate: template<class T, class C, template<class ...> class M, template<class ...> class V> toml::detail::enable_if_t<toml::detail::is_exact_toml_type<T, toml::basic_value<C, Tb, A> >::value, T>& toml::get(toml::basic_value<C, Tb, A>&)
get(basic_value<C, M, V>& v)
^
/home/titan/maskfusion/deps/toml11/toml/get.hpp:19:1: note: template argument deduction/substitution failed:
/home/titan/maskfusion/GUI/MainController.cpp:278:83: error: template argument 1 is invalid
const auto tomlMaskRCNN = toml::gettoml::Table(tomlConfig.at("MaskRCNN"));
^
In file included from /home/titan/maskfusion/deps/toml11/toml.hpp:39:0,
from /home/titan/maskfusion/GUI/MainController.cpp:32:
/home/titan/maskfusion/deps/toml11/toml/get.hpp:27:1: note: candidate: template<class T, class C, template<class ...> class M, template<class ...> class V> toml::detail::enable_if_t<toml::detail::is_exact_toml_type<T, toml::basic_value<C, Tb, A> >::value, T>& toml::get(const toml::basic_value<C, Tb, A>&)
get(const basic_value<C, M, V>& v)
^
/home/titan/maskfusion/deps/toml11/toml/get.hpp:27:1: note: template argument deduction/substitution failed:
/home/titan/maskfusion/GUI/MainController.cpp:278:83: error: template argument 1 is invalid
const auto tomlMaskRCNN = toml::gettoml::Table(tomlConfig.at("MaskRCNN"));
^
In file included from /home/titan/maskfusion/deps/toml11/toml.hpp:39:0,
from /home/titan/maskfusion/GUI/MainController.cpp:32:
/home/titan/maskfusion/deps/toml11/toml/get.hpp:35:1: note: candidate: template<class T, class C, template<class ...> class M, template<class ...> class V> toml::detail::enable_if_t<toml::detail::is_exact_toml_type<T, toml::basic_value<C, Tb, A> >::value, T>&& toml::get(toml::basic_value<C, Tb, A>&&)
get(basic_value<C, M, V>&& v)
^
/home/titan/maskfusion/deps/toml11/toml/get.hpp:35:1: note: template argument deduction/substitution failed:
/home/titan/maskfusion/GUI/MainController.cpp:278:83: error: template argument 1 is invalid
const auto tomlMaskRCNN = toml::gettoml::Table(tomlConfig.at("MaskRCNN"));
^
In file included from /home/titan/maskfusion/deps/toml11/toml.hpp:39:0,
from /home/titan/maskfusion/GUI/MainController.cpp:32:
/home/titan/maskfusion/deps/toml11/toml/get.hpp:46:1: note: candidate: template<class T, class C, template<class ...> class M, template<class ...> class V> toml::detail::enable_if_t<std::is_same<T, toml::basic_value<C, Tb, A> >::value, T>& toml::get(toml::basic_value<C, Tb, A>&)
get(basic_value<C, M, V>& v)
^
/home/titan/maskfusion/deps/toml11/toml/get.hpp:46:1: note: template argument deduction/substitution failed:
/home/titan/maskfusion/GUI/MainController.cpp:278:83: error: template argument 1 is invalid
const auto tomlMaskRCNN = toml::gettoml::Table(tomlConfig.at("MaskRCNN"));
^
In file included from /home/titan/maskfusion/deps/toml11/toml.hpp:39:0,
from /home/titan/maskfusion/GUI/MainController.cpp:32:
/home/titan/maskfusion/deps/toml11/toml/get.hpp:54:1: note: candidate: template<class T, class C, template<class ...> class M, template<class ...> class V> toml::detail::enable_if_t<std::is_same<T, toml::basic_value<C, Tb, A> >::value, T>& toml::get(const toml::basic_value<C, Tb, A>&)
get(const basic_value<C, M, V>& v)
^
/home/titan/maskfusion/deps/toml11/toml/get.hpp:54:1: note: template argument deduction/substitution failed:
/home/titan/maskfusion/GUI/MainController.cpp:278:83: error: template argument 1 is invalid
const auto tomlMaskRCNN = toml::gettoml::Table(tomlConfig.at("MaskRCNN"));
^
In file included from /home/titan/maskfusion/deps/toml11/toml.hpp:39:0,
from /home/titan/maskfusion/GUI/MainController.cpp:32:
/home/titan/maskfusion/deps/toml11/toml/get.hpp:62:1: note: candidate: template<class T, class C, template<class ...> class M, template<class ...> class V> toml::detail::enable_if_t<std::is_same<T, toml::basic_value<C, Tb, A> >::value, T>&& toml::get(toml::basic_value<C, Tb, A>&&)
get(basic_value<C, M, V>&& v)
^
/home/titan/maskfusion/deps/toml11/toml/get.hpp:62:1: note: template argument deduction/substitution failed:
/home/titan/maskfusion/GUI/MainController.cpp:278:83: error: template argument 1 is invalid
const auto tomlMaskRCNN = toml::gettoml::Table(tomlConfig.at("MaskRCNN"));
^
In file included from /home/titan/maskfusion/deps/toml11/toml.hpp:39:0,
from /home/titan/maskfusion/GUI/MainController.cpp:32:
/home/titan/maskfusion/deps/toml11/toml/get.hpp:75:1: note: candidate: template<class T, class C, template<class ...> class M, template<class ...> class V> toml::detail::enable_if_t<toml::detail::conjunction<toml::detail::is_basic_value, toml::detail::negation<std::is_same<T, toml::basic_value<C, Tb, A> > > >::value, T> toml::get(const toml::basic_value<C, Tb, A>&)
get(const basic_value<C, M, V>& v)
^
/home/titan/maskfusion/deps/toml11/toml/get.hpp:75:1: note: template argument deduction/substitution failed:
/home/titan/maskfusion/GUI/MainController.cpp:278:83: error: template argument 1 is invalid
const auto tomlMaskRCNN = toml::gettoml::Table(tomlConfig.at("MaskRCNN"));
^
In file included from /home/titan/maskfusion/deps/toml11/toml.hpp:39:0,
from /home/titan/maskfusion/GUI/MainController.cpp:32:
/home/titan/maskfusion/deps/toml11/toml/get.hpp:91:1: note: candidate: template<class T, class C, template<class ...> class M, template<class ...> class V> toml::detail::enable_if_t<toml::detail::conjunction<std::is_integral<_Tp>, toml::detail::negation<std::is_same<T, bool> >, toml::detail::negation<toml::detail::is_exact_toml_type<T, toml::basic_value<C, Tb, A> > > >::value, T> toml::get(const toml::basic_value<C, Tb, A>&)
get(const basic_value<C, M, V>& v)
^
/home/titan/maskfusion/deps/toml11/toml/get.hpp:91:1: note: template argument deduction/substitution failed:
/home/titan/maskfusion/GUI/MainController.cpp:278:83: error: template argument 1 is invalid
const auto tomlMaskRCNN = toml::gettoml::Table(tomlConfig.at("MaskRCNN"));
^
In file included from /home/titan/maskfusion/deps/toml11/toml.hpp:39:0,
from /home/titan/maskfusion/GUI/MainController.cpp:32:
/home/titan/maskfusion/deps/toml11/toml/get.hpp:106:1: note: candidate: template<class T, class C, template<class ...> class M, template<class ...> class V> toml::detail::enable_if_t<toml::detail::conjunction<std::is_floating_point<_Tp>, toml::detail::negation<toml::detail::is_exact_toml_type<T, toml::basic_value<C, Tb, A> > > >::value, T> toml::get(const toml::basic_value<C, Tb, A>&)
get(const basic_value<C, M, V>& v)
^
/home/titan/maskfusion/deps/toml11/toml/get.hpp:106:1: note: template argument deduction/substitution failed:
/home/titan/maskfusion/GUI/MainController.cpp:278:83: error: template argument 1 is invalid
const auto tomlMaskRCNN = toml::gettoml::Table(tomlConfig.at("MaskRCNN"));
^
In file included from /home/titan/maskfusion/deps/toml11/toml.hpp:39:0,
from /home/titan/maskfusion/GUI/MainController.cpp:32:
/home/titan/maskfusion/deps/toml11/toml/get.hpp:118:1: note: candidate: template<class T, class C, template<class ...> class M, template<class ...> class V> toml::detail::enable_if_t<std::is_same<T, std::__cxx11::basic_string >::value, std::__cxx11::basic_string >& toml::get(toml::basic_value<C, Tb, A>&)
get(basic_value<C, M, V>& v)
^
/home/titan/maskfusion/deps/toml11/toml/get.hpp:118:1: note: template argument deduction/substitution failed:
/home/titan/maskfusion/GUI/MainController.cpp:278:83: error: template argument 1 is invalid
const auto tomlMaskRCNN = toml::gettoml::Table(tomlConfig.at("MaskRCNN"));
^
In file included from /home/titan/maskfusion/deps/toml11/toml.hpp:39:0,
from /home/titan/maskfusion/GUI/MainController.cpp:32:
/home/titan/maskfusion/deps/toml11/toml/get.hpp:126:1: note: candidate: template<class T, class C, template<class ...> class M, template<class ...> class V> toml::detail::enable_if_t<std::is_same<T, std::__cxx11::basic_string >::value, std::__cxx11::basic_string >& toml::get(const toml::basic_value<C, Tb, A>&)
get(const basic_value<C, M, V>& v)
^
/home/titan/maskfusion/deps/toml11/toml/get.hpp:126:1: note: template argument deduction/substitution failed:
/home/titan/maskfusion/GUI/MainController.cpp:278:83: error: template argument 1 is invalid
const auto tomlMaskRCNN = toml::gettoml::Table(tomlConfig.at("MaskRCNN"));
^
In file included from /home/titan/maskfusion/deps/toml11/toml.hpp:39:0,
from /home/titan/maskfusion/GUI/MainController.cpp:32:
/home/titan/maskfusion/deps/toml11/toml/get.hpp:134:1: note: candidate: template<class T, class C, template<class ...> class M, template<class ...> class V> toml::detail::enable_if_t<std::is_same<T, std::__cxx11::basic_string >::value, std::__cxx11::basic_string >& toml::get(toml::basic_value<C, Tb, A>&&)
get(basic_value<C, M, V>&& v)
^
/home/titan/maskfusion/deps/toml11/toml/get.hpp:134:1: note: template argument deduction/substitution failed:
/home/titan/maskfusion/GUI/MainController.cpp:278:83: error: template argument 1 is invalid
const auto tomlMaskRCNN = toml::gettoml::Table(tomlConfig.at("MaskRCNN"));
^
In file included from /home/titan/maskfusion/deps/toml11/toml.hpp:39:0,
from /home/titan/maskfusion/GUI/MainController.cpp:32:
/home/titan/maskfusion/deps/toml11/toml/get.hpp:158:1: note: candidate: template<class T, class C, template<class ...> class M, template<class ...> class V> toml::detail::enable_if_t<toml::detail::is_chrono_duration::value, T> toml::get(const toml::basic_value<C, Tb, A>&)
get(const basic_value<C, M, V>& v)
^
/home/titan/maskfusion/deps/toml11/toml/get.hpp:158:1: note: template argument deduction/substitution failed:
/home/titan/maskfusion/GUI/MainController.cpp:278:83: error: template argument 1 is invalid
const auto tomlMaskRCNN = toml::gettoml::Table(tomlConfig.at("MaskRCNN"));
^
In file included from /home/titan/maskfusion/deps/toml11/toml.hpp:39:0,
from /home/titan/maskfusion/GUI/MainController.cpp:32:
/home/titan/maskfusion/deps/toml11/toml/get.hpp:171:1: note: candidate: template<class T, class C, template<class ...> class M, template<class ...> class V> toml::detail::enable_if_t<std::is_same<std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long int, std::ratio<1l, 1000000000l> > >, T>::value, T> toml::get(const toml::basic_value<C, Tb, A>&)
get(const basic_value<C, M, V>& v)
^
/home/titan/maskfusion/deps/toml11/toml/get.hpp:171:1: note: template argument deduction/substitution failed:
/home/titan/maskfusion/GUI/MainController.cpp:278:83: error: template argument 1 is invalid
const auto tomlMaskRCNN = toml::gettoml::Table(tomlConfig.at("MaskRCNN"));
^
In file included from /home/titan/maskfusion/deps/toml11/toml.hpp:39:0,
from /home/titan/maskfusion/GUI/MainController.cpp:32:
/home/titan/maskfusion/deps/toml11/toml/get.hpp:276:1: note: candidate: template<class T, class C, template<class ...> class M, template<class ...> class V> toml::detail::enable_if_t<toml::detail::conjunction<toml::detail::is_container, toml::detail::has_resize_method, toml::detail::negation<toml::detail::is_exact_toml_type<T, toml::basic_value<C, Tb, A> > > >::value, T> toml::get(const toml::basic_value<C, Tb, A>&)
get(const basic_value<C, M, V>& v)
^
/home/titan/maskfusion/deps/toml11/toml/get.hpp:276:1: note: template argument deduction/substitution failed:
/home/titan/maskfusion/GUI/MainController.cpp:278:83: error: template argument 1 is invalid
const auto tomlMaskRCNN = toml::gettoml::Table(tomlConfig.at("MaskRCNN"));
^
In file included from /home/titan/maskfusion/deps/toml11/toml.hpp:39:0,
from /home/titan/maskfusion/GUI/MainController.cpp:32:
/home/titan/maskfusion/deps/toml11/toml/get.hpp:298:1: note: candidate: template<class T, class C, template<class ...> class M, template<class ...> class V> toml::detail::enable_if_t<toml::detail::conjunction<toml::detail::is_container, toml::detail::negation<toml::detail::has_resize_method >, toml::detail::negation<toml::detail::is_exact_toml_type<T, toml::basic_value<C, Tb, A> > > >::value, T> toml::get(const toml::basic_value<C, Tb, A>&)
get(const basic_value<C, M, V>& v)
^
/home/titan/maskfusion/deps/toml11/toml/get.hpp:298:1: note: template argument deduction/substitution failed:
/home/titan/maskfusion/GUI/MainController.cpp:278:83: error: template argument 1 is invalid
const auto tomlMaskRCNN = toml::gettoml::Table(tomlConfig.at("MaskRCNN"));
^
In file included from /home/titan/maskfusion/deps/toml11/toml.hpp:39:0,
from /home/titan/maskfusion/GUI/MainController.cpp:32:
/home/titan/maskfusion/deps/toml11/toml/get.hpp:323:1: note: candidate: template<class T, class C, template<class ...> class M, template<class ...> class V> toml::detail::enable_if_t<toml::detail::is_std_pair::value, T> toml::get(const toml::basic_value<C, Tb, A>&)
get(const basic_value<C, M, V>& v)
^
/home/titan/maskfusion/deps/toml11/toml/get.hpp:323:1: note: template argument deduction/substitution failed:
/home/titan/maskfusion/GUI/MainController.cpp:278:83: error: template argument 1 is invalid
const auto tomlMaskRCNN = toml::gettoml::Table(tomlConfig.at("MaskRCNN"));
^
In file included from /home/titan/maskfusion/deps/toml11/toml.hpp:39:0,
from /home/titan/maskfusion/GUI/MainController.cpp:32:
/home/titan/maskfusion/deps/toml11/toml/get.hpp:357:1: note: candidate: template<class T, class C, template<class ...> class M, template<class ...> class V> toml::detail::enable_if_t<toml::detail::is_std_tuple::value, T> toml::get(const toml::basic_value<C, Tb, A>&)
get(const basic_value<C, M, V>& v)
^
/home/titan/maskfusion/deps/toml11/toml/get.hpp:357:1: note: template argument deduction/substitution failed:
/home/titan/maskfusion/GUI/MainController.cpp:278:83: error: template argument 1 is invalid
const auto tomlMaskRCNN = toml::gettoml::Table(tomlConfig.at("MaskRCNN"));
^
In file included from /home/titan/maskfusion/deps/toml11/toml.hpp:39:0,
from /home/titan/maskfusion/GUI/MainController.cpp:32:
/home/titan/maskfusion/deps/toml11/toml/get.hpp:383:1: note: candidate: template<class T, class C, template<class ...> class M, template<class ...> class V> toml::detail::enable_if_t<toml::detail::conjunction<toml::detail::is_map, toml::detail::negation<toml::detail::is_exact_toml_type<T, toml::basic_value<C, Tb, A> > > >::value, T> toml::get(const toml::basic_value<C, Tb, A>&)
get(const basic_value<C, M, V>& v)
^
/home/titan/maskfusion/deps/toml11/toml/get.hpp:383:1: note: template argument deduction/substitution failed:
/home/titan/maskfusion/GUI/MainController.cpp:278:83: error: template argument 1 is invalid
const auto tomlMaskRCNN = toml::gettoml::Table(tomlConfig.at("MaskRCNN"));
^
In file included from /home/titan/maskfusion/deps/toml11/toml.hpp:39:0,
from /home/titan/maskfusion/GUI/MainController.cpp:32:
/home/titan/maskfusion/deps/toml11/toml/get.hpp:409:1: note: candidate: template<class T, class C, template<class ...> class M, template<class ...> class V> toml::detail::enable_if_t<toml::detail::conjunction<toml::detail::negation<toml::detail::is_exact_toml_type<T, toml::basic_value<C, Tb, A> > >, toml::detail::has_from_toml_method<T, C, M, V>, std::is_default_constructible<_Tp> >::value, T> toml::get(const toml::basic_value<C, Tb, A>&)
get(const basic_value<C, M, V>& v)
^
/home/titan/maskfusion/deps/toml11/toml/get.hpp:409:1: note: template argument deduction/substitution failed:
/home/titan/maskfusion/GUI/MainController.cpp:278:83: error: template argument 1 is invalid
const auto tomlMaskRCNN = toml::gettoml::Table(tomlConfig.at("MaskRCNN"));
^
In file included from /home/titan/maskfusion/deps/toml11/toml.hpp:39:0,
from /home/titan/maskfusion/GUI/MainController.cpp:32:
/home/titan/maskfusion/deps/toml11/toml/get.hpp:418:3: note: candidate: template<class T, class C, template<class ...> class M, template<class ...> class V, long unsigned int S> T toml::get(const toml::basic_value<C, Tb, A>&)
T get(const basic_value<C, M, V>& v)
^
/home/titan/maskfusion/deps/toml11/toml/get.hpp:418:3: note: template argument deduction/substitution failed:
/home/titan/maskfusion/GUI/MainController.cpp:278:83: error: template argument 1 is invalid
const auto tomlMaskRCNN = toml::gettoml::Table(tomlConfig.at("MaskRCNN"));
^
GUI/CMakeFiles/MaskFusion.dir/build.make:86: recipe for target 'GUI/CMakeFiles/MaskFusion.dir/MainController.cpp.o' failed
make[2]: *** [GUI/CMakeFiles/MaskFusion.dir/MainController.cpp.o] Error 1
CMakeFiles/Makefile2:211: recipe for target 'GUI/CMakeFiles/MaskFusion.dir/all' failed
make[1]: *** [GUI/CMakeFiles/MaskFusion.dir/all] Error 2
Makefile:83: recipe for target 'all' failed
make: *** [all] Error 2

how to use toml?

Hi there,

Thanks for this toolbox.

Can I stick my point clouds together using this toolbox when I have the x, y, z, and quaternion orientation of each point cloud?

Regards,
Yaqub

Hi,

I built MaskFusion following the build.sh script (but with CUDA 9.0).
I am trying to run it on the fr3_walking_halfsphere sequence. The only class I enabled for the segmentation is "person", which, if I understand correctly, should be filtered out for the registration. However, the software ignores completely the semantic segmentation. To confirm this I tried to run it with the "-static" flag and it behaves exactly the same. I also tried to precompute the masks (using offline_runner.py and passing them with "-maskdir"). The masks are computed correctly, but again MaskFusion completely ignores them. The terminal does not show anything unusual, except for the following messages:

Your GPU "GeForce GTX 1080" isn't in the ICP Step performance database, please add it
Your GPU "GeForce GTX 1080" isn't in the RGB Step performance database, please add it
Your GPU "GeForce GTX 1080" isn't in the RGB Res performance database, please add it
Your GPU "GeForce GTX 1080" isn't in the SO3 Step performance database, please add it

What am I missing?

What should I do if THE following error occurs when I make：
nvcc fatal : Unsupported gpu architecture 'compute_86'
CMake Error at cuda_compile_1_generated_reduce.cu.o.cmake:219 (message):
Error generating
/home/lemons/maskfusion/build/Core/CMakeFiles/cuda_compile_1.dir/Cuda/./cuda_compile_1_generated_reduce.cu.o

nvcc fatal : Unsupported gpu architecture 'compute_86'
CMake Error at cuda_compile_1_generated_segmentation.cu.o.cmake:219 (message):
Error generating
/home/lemons/maskfusion/build/Core/CMakeFiles/cuda_compile_1.dir/Cuda/./cuda_compile_1_generated_segmentation.cu.o

Hello ,Firstly , thank you for your contribution about Maskfusion's method and code ! When I build the environment of Maskfusion as the build.sh described and execute the command: sudo update-java-alternatives -s java-1.8.0-openjdk-amd64, I had meet a error that It is update-alternatives: error: no alternatives for mozilla-javaplugin.so , I can't solve this error . Can you help me ? Thank you for your help sincerely !

Hi, I was able to run MaskFusion with the real input sequences from Co-Fusion, but the quality of the reconstruction obtained for the dynamic objects did not seem to be so good.

Results for teddy-handover.klg:

Results for sliding-clock.klg:

Results for place-items.klg:

Is this expected? Is there something that I should change in order to get better results?

I am getting the following error after running the build scripts

In file included from /home/ssg1002/dev/maskfusion/Core/Model/Model.h:25, from /home/ssg1002/dev/maskfusion/Core/Model/GlobalProjection.cpp:20: /home/ssg1002/dev/maskfusion/Core/Model/../Shaders/FeedbackBuffer.h:28:10: fatal error: pangolin/display/opengl_render_state.h: No such file or directory 28 | #include <pangolin/display/opengl_render_state.h> | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ compilation terminated.

File "/home/lemons/maskfusion/build/GUI/MaskRCNN.py", line 28, in
exec(open(ve_path).read(), {'file': ve_path})
FileNotFoundError: [Errno 2] No such file or directory: '/home/lemons/maskfusion/build/../python-environment/bin/activate_this.py'
Could you please tell me how to solve this problem！！

hi, thanks for the fantastic work.
I met the similar problem like #21.

I did according to the REAME.md, the only difference is that I added two CMake parameters as below, so that there would not be "out of memory" error on my GPU ( Nvidia 840M, yes... an old one)

-DMASKFUSION_NUM_GSURFELS=921600 -DMASKFUSION_NUM_OSURFELS=102400

and checked my toml file to keep it unchanged:

# If not empty, ignore every class not listed here. Example: ['teddy bear']
filter_classes = []

# If not empty, only track classes listed here (otherwise apply BG motion)
trackable_classes = []

# Reject predictions with lower score
score_threshold = 0.55

However, when running on the dataset from Co-Fusion, it seems the label "clock" has been filtered.

so in the final result, the trajectory of the clock makes the reconstruction not so good:

The offline result from MaskRCNN shows good:

I'm wondering why the dynamic parts are ignored and not tracked. Is there some size filter somewhere else? My output shows like below:

Initialised multi-object fusion (main-thread: 140037120672128)
- The background model can have up to 921600 surfel (960x960)
- Object models can have up to 102400 surfel (320x320)
- Using GPU 0 for SLAM system and GPU 0 for MaskRCNN
- Using frame-queue of size: 0
New label detected (**2147483647,2147483647 -2147483648,-2147483648**) - try relocating...
Found new model.
Created model with max number of vertices: 102400
New label detected (2147483647,2147483647 -2147483648,-2147483648) - try relocating...
Found new model.
Created model with max number of vertices: 102400

"2147483647,2147483647 -2147483648,-2147483648" is a little weird, however I checked the codes and they are defined as initial values for bounding boxes.

Thanks so much. I'll keep on digging into the codes to find the answer.

Hello,

I think there are two methods of accessing the surfelmap.

1. downloadMap in model.h which copies the memory from the GPU
2. access the graphic buffer directly

downloadMap is too slowly for real time. It copies stable and unstable surfels.
Can I copy the surfelmap to normal memory in real-time?
What is the fastest way to copy a surfelmap with only stable/confident surfels to normal memory? Can this be done in real-time?

I hope this makes sense. Thank you very much.

I can only download the dataset used in the paper co-fusion, where can I download the dataset used in maskfusion , such as skateboards and teddy bears

In file included from /home/sheng/maskfusion-master/Core/Segmentation/Segmentation.h:21:0,
from /home/sheng/maskfusion-master/Core/MaskFusion.h:38,
from /home/sheng/maskfusion-master/Core/MaskFusion.cpp:19:
/home/sheng/maskfusion-master/Core/Segmentation/Slic.h:26:20: fatal error: gSLICr.h: No such file or directory
#include <gSLICr.h>
^
compilation terminated.
Core/CMakeFiles/libmaskfusion.dir/build.make:155: recipe for target 'Core/CMakeFiles/libmaskfusion.dir/MaskFusion.cpp.o' failed
make[2]: *** [Core/CMakeFiles/libmaskfusion.dir/MaskFusion.cpp.o] Error 1
CMakeFiles/Makefile2:117: recipe for target 'Core/CMakeFiles/libmaskfusion.dir/all' failed
make[1]: *** [Core/CMakeFiles/libmaskfusion.dir/all] Error 2
Makefile:83: recipe for target 'all' failed
make: *** [all] Error 2