yang7879 / 3d-bonet Goto Github PK

Hello, I am very interesting in your great work, and I want to use scannet to train your net, and I follewed #6 to process the dataset, but i still have some questions, *_vh_clean_2.ply is the raw point cloud? and when I use pointnet for processing, what are the classes should I set in indoor3d_util.py?

Hi, @Yang7879

Thanks for sharing this amazing work,
Would you also release the code about NYU dataset ?

Hi Bo, thanks for sharing the code. May I ask what's the gap between the performance between the val and test sets, so I can know the which position I am in. Another question is what if using pointnet++ for segmentation rather than sparse conv? Is this have a big influence on the final results?

11

Hello,thanks for your reply,how to visualize the semantic segmentation on s3dis data?can you provide some codes about visualization?looking forward your reply.

您好，我在使用自己数据集测试时，想要只显示其中指定类别实例的时候不太明白BlockMerging中volume和volume_seg的含义，可以解释一下吗

Hi @Yang7879,

How would you prepare outdoor datasets for BoNet and how do you think BoNet will perform on them? I think we would need to change quite a few things in the pre-processing steps. From the top of my head, increasing the block size from 1mX1m to something bigger would be a must since most outdoor instances are bigger than that.

你好，假如我的dataset是室外的。我的pointcloud里面我只关注的object有3类，总共就5个我需要实例化的object，那么我应该怎么对数据标注？是不是我只需要从pointcloud里分别提取这5个object的x,y,x,r,g,b信息分别保存为对应的txt或者别的格式。提取过后的the remaining points的x,y,z,r,g,b信息单独保存为一个txt，作为另外一个类别类似于你代码中定义的clutter。可是，这些remaining points 空间上可能是不连续的，后面对其bbox进行编码的时候，是获取这些remaining points的[x_min,y_min,z_min,x_max,y_max,z_max].那么这样的话，这个bbox可能就会把我需要的5个object也包含在内。所以，我应该怎么标注自己的数据集？我是不是应该把这些remaining points 按照空间信息并且以不包含object在内的原则分块保存？另外，针对block merging 还有什么好的算法吗？非常期待你的解答！感谢！

Hello! I want to know how to carry 6-fold CV on S3DIS?
Thank you very much!

I am interested in visualizing the point masks for an entire scene, but I am a little confused by the format of the .mat file. Do you have any code that would out put the .mat file in res_by_scene to a .txt file or other point cloud type file? I am able to visualize point cloud masks for each batch but am having trouble doing the same for the entire scene. I am looking for a visualization similar to Fig. 7 in your paper.
Thanks in advance

Hi @Yang7879
Thank you very much for sharing the great project!
I am new to tensorflow. However, I found it quite difficult to debug on tensorflow, which means that with the sess.run(), I can not see the real training data transform in the model.
Could you please share me some experience on how to build the model correctly, and how to debug with this static graph framework effectively?
Waiting for your reply~~~

Thanks for your nice work. I have a question related to block-merging function.
In Blockmerging I don't quite understand the hard code at L#52:
if maxoverlapgroupcounts[i] < 7 and groupcounts[i] > 12.
What is 7 and 12 stand for? Could you please give me some intuition about this if statement.

Hi @Yang7879
Thank for sharing the experiments of s3dis!
Could you please release the code about the training and evaluation process of Scannet_v2 dataset ?

Respected sir, can you share a pre-trained model or weights with me. I have not a Cuda based System, that is why I am not able to test this code. Object localization is much important for my research. But due to the unavailability of resources, I have struggled to complete my research. it is my honor request, if possible please share with me the trained models or CPU based code.

Hi @Yang7879 , thx for your released code. The paper is easy understanding, but I don't find any reinforcement learning algorithms in the code as Appendix D wrote. I'm not familiar with DPG actually. Do I miss something? Is it necessary to propagate gradient along the assignment path?

In your evaluation code, you removed small instances. I would like to ask about the Scannet dataset. How did you handle these small instances when uploading benchmark?
Thank you very much.

Good morning,
I am recoding your work using pytorch to use it as a baseline in my work. I am at the end of the loss part trying to implement the Hugarian function. I am trying my code with a simple toy example with a maximum number of predicted bounding boxes of 10 and a groundtruth of 5 bounding boxes and pass to the Hungarian function a cost matrix of shape (H*T) where H is the number of predicted BB and T the number of boxes in the ground truth. However in the following part :
unmapped = num_instances - ins_count

if unmapped > 0:
   rest = np.array(range(ins_count, num_instances))
   row_ind = np.concatenate([row_ind, rest])
   unmapped_ind = np.array(list(set(range(num_instances)) - set(col_ind)))
   col_ind = np.concatenate([col_ind, unmapped_ind])

col_ind and row_ind are transformed to 10 of length arrays and on the line :
loss_total += cost[idx][row_ind, col_ind].sum()
I get out of bound error as cost is a (105) matrix and not (1010).

Am I missing something?

Yours faithfully
J

Hi @Yang7879 ,thx for your released code. In your paper, Table2 describes the testing result on S3DIS dataset. I want to ask whether it is tested on sampled scenes (4096 for each cube) or full scenes? Furthermore, if it is tested on full scenes, can you share your test code or give some suggestions on how to test on full scene?

Best regards.

Hi @Yang7879 , thanks for sharing the code, now I am reading the helper_net.py, but I have some confusion about the contents. I don't understand the meaning of these lines :
tp_pred = 100 * tp1_pred * tp2_pred
tp_pred = tf.maximum(tf.minimum(tp_pred, 20.0), -20.0)

Could you please explain my confusion? Thank you.

@Yang7879 Thanks for your amaizng work, but I got trouble in downloading ScanNetv2 dataset. When I used the script "python2 download-scannet.py -o ScanNetDatav2/", it warned me that "WARNING: You are downloading the entire ScanNet v2 release which requires 1.2TB of space." I have limited space, so how do you download part of ScanNet for 3D instance segmentation?

var Y_bbox_helper in helper_net.py line 253 is used for finding valid gt boxes. After sum operation, Y_bbox_helper maybe less than 0, so adding an abs operation before greater operation maybe a better solution.

Thanks for sharing the code and congratulations on your NeurIPS spotlight!!!

If I understand correctly, when testing, you still test on each block (1m*1m) and then merge? My question is: if one instance, e.g. long table that is 3m long, I would imagine that this instance can span two or more blocks. How do you merge the same instance from multiple blocks?

您好，请问有没有S3DIS数据百度网盘的下载方式啊？感谢感谢！

你好，这里提供的h5格式的S3DIS数据包括coords，labels和points三种量，请问coords和points分别代表什么呢？如果想要从包含实例真值的txt格式数据得到这样的h5需要怎么做呢？

Hello, thanks for sharing your code! Recently I want to train bonet with scannet database, and I followesd the step in #6 to process scannet database, firstly I ran helper_data_scannet_prepare.py to get the xyzrgbsemins; secondly, I used pointnet to divide them into 1*1 block. when training the net, I changed the sem_names and sem_ids, but it occurs “>= 2 sem for an ins: /home/data/bonet-scannet/scannet/ply_data_all_0.h5”. I don't know where is the problem when preprocessing the data, can you please help me?

Hi, @Yang7879 ,

Thanks for sharing the code. For the setup packages, it seems a bit old. As current version for tensorflow is 1.13+, and open3d is 0.6+. It's not so easy to install older version for them. Would you help to update the code to adapt with recent version on tensorflow and open3d.

THX!

@Yang7879 thanks for open sourcing the wonderfull work , i had few queries
Q1 have you trained the architecture on the available other dataset like semanttic Kitti and 3D dataset
Q2 If not trained can we follow the same training pipeline , if trained can you please share the pre-trained model
Q3 can we use the currently pre-trained model to test on custom dataset which less number of point cloud density

Thanks in advance

Thank you for your sharing! Errors occurred when I ran main_train.py.
It seems that there are something wrong with the compiling. But I can't find what happened.
My environment is: python3.6+tensorflow1.4+gcc7.3 +cuda8.0

Thx for your working.
I've trained the module with 50 epochs and finnaly finish the evalution on Area5. But the results are quiet different from your paper table 3. The Area 5 mPrec and mRecall are 57.5 and 40.2 but the mPrec I got is 52%. And then I use the realeased module. The mPrec is 60 which is close enogh. I'm wondering if the released module was trained on different hyperparameters or is there other reasons?

Hi @Yang7879 , thanks for sharing this great work!

I didn't see any code related to nms or duplicate bbox removal. Does it mean that duplicate bbox removal are not necessary because the network is trained with 1-to-1 matched gt box and pred box?

Looking forward to your reply~

Hi, thanks for your great work! Can you explain the meaning of 'ins_labels' and 'sem_labels' in helper_data_s3dis.py line 44 and 45 . I am not sure if I understand correctly. Looking forward your reply!

Dear Bo,

Thanks for sharing the code. It was pretty easy to run, but I have a few questions regarding the evaluation of the S3DIS.

I just trained your network from scratch simply by running main_train.py on S3DIS, but, I found out that the final evaluations on Area 5 were different from the ones reported on the paper, and wanted to know the cause of the discrepancy.

In Table 3 of your paper https://arxiv.org/pdf/1906.01140.pdf, the Area 5 mPrec and mRecall are 57.5 and 40.2, but the final results that I got were 53.36 and 40.55 respectively.

Is the default variable different from the one you reported on the paper?

Second question is why did you use mPrec and mRecall separately? It is quite standard to use mAP which measures the unweighted class-wise average of the areas under the precision-recall curve for all classes. What're your thoughts on the evaluation metrics?

Finally, would it be possible for you to share the class-wise results of the reported baselines? I would prefer to compute mAP on all baselines and it would be nice if you could share the results that you got on S3DIS.

Thanks!
Chris

Hi,Yang7879.I am learning your project now. You really make a remarkably job.
When I configure the environment, like the tensorflow, if I use a higher version of the tensorflow(tensorflow 1.13.1), what will happen? So, is your project backwards compatible?

By the way, does the project need very fast GPU? For I have a GTX 730 GPU,it has poor computing power.
Thanks.

杨博士你好，我尝试用你的网络训练自己的数据集，训练的环境是：
tensorflow1.15
cuda10
cudnn7.0.4
可以正常编译和读数据，但是训练到第30多个epoch时训练中断，训练日志如下，希望杨博士能够解答
epoch 32 end time is : 2021-01-08 14:21:29.075227
train files shuffled!
is training ep : 33
total train batch num: 100
ep 33 i 0 psemce 0.0 bbvert -0.23279694 l2 0.060497742 ce 0.26028115 siou -0.5535758 bbscore 0.0038582273 pmask 0.6467816
ep 33 i 0 test psem 0.0 bbvert 1.9851223 l2 0.059121773 ce 2.2534707 siou -0.3274702 bbscore 0.0048341155 pmask 3.4519947
test pred bborder [[2 1 0]]
ep 33 i 20 psemce 0.0 bbvert -0.44303733 l2 0.030050844 ce 0.16678412 siou -0.6398723 bbscore 0.0026201883 pmask 0.63400114
ep 33 i 20 test psem 0.0 bbvert -0.4450612 l2 0.04898257 ce 0.23810822 siou -0.732152 bbscore 0.0016184862 pmask 0.47476012
test pred bborder [[2 0 1]]
ep 33 i 40 psemce 0.0 bbvert 0.43996847 l2 0.08725857 ce 0.8109299 siou -0.45822 bbscore 0.0034747643 pmask 0.9270937
ep 33 i 40 test psem 0.0 bbvert -0.07955924 l2 0.040802542 ce 0.4581066 siou -0.5784684 bbscore 0.0087565 pmask 1.0110209
test pred bborder [[2 0 1]]
ep 33 i 60 psemce 0.0 bbvert 0.057684183 l2 0.071036406 ce 0.5709717 siou -0.58432394 bbscore 0.00046247765 pmask 0.48829234
ep 33 i 60 test psem 0.0 bbvert -0.3817188 l2 0.03684793 ce 0.2770622 siou -0.69562894 bbscore 0.0019779946 pmask 0.54431504
test pred bborder [[2 0 1]]
ep 33 i 80 psemce 0.0 bbvert 0.038050413 l2 0.034902867 ce 0.6071266 siou -0.60397905 bbscore 0.015431552 pmask 0.8978894
ep 33 i 80 test psem 0.0 bbvert 1.1076844 l2 0.07785928 ce 1.3761435 siou -0.34631833 bbscore 0.033992507 pmask 1.7343999
test pred bborder [[2 0 1]]
model saved in : ./log/train_mod/model033.cptk
epoch 33 end time is : 2021-01-08 14:21:44.245053
train files shuffled!
is training ep : 34
total train batch num: 100
ep 34 i 0 psemce 0.0 bbvert -0.41581324 l2 0.057975773 ce 0.28829214 siou -0.76208115 bbscore 0.0003172583 pmask 0.34254307
ep 34 i 0 test psem 0.0 bbvert 1.7912706 l2 0.08668331 ce 2.017744 siou -0.31315675 bbscore 0.00576146 pmask 2.1253805
test pred bborder [[0 2 1]]
ep 34 i 20 psemce 0.0 bbvert -0.14073128 l2 0.034625944 ce 0.4937689 siou -0.6691261 bbscore 0.0047056335 pmask 0.7088615
ep 34 i 20 test psem 0.0 bbvert 1.9534252 l2 0.0907397 ce 2.1705353 siou -0.3078499 bbscore 0.0019757028 pmask 2.682012
test pred bborder [[0 2 1]]
ep 34 i 40 psemce 0.0 bbvert 0.27091432 l2 0.053299602 ce 0.7194822 siou -0.5018675 bbscore 0.001409175 pmask 0.60204554
ep 34 i 40 test psem 0.0 bbvert -0.28416353 l2 0.09286666 ce 0.19349718 siou -0.5705274 bbscore 0.003192804 pmask 0.21589296
test pred bborder [[1 0 2]]
2021-01-08 14:21:52.432564: W tensorflow/core/framework/op_kernel.cc:1639] Invalid argument: ValueError: matrix contains invalid numeric entries
Traceback (most recent call last):

File "/home/liu/anaconda3/envs/tf1.15/lib/python3.6/site-packages/tensorflow_core/python/ops/script_ops.py", line 235, in call
ret = func(*args)

File "/home/liu/disk1/Life/3DBoNetPoint818a(linux)/helper_net.py", line 115, in assign_mappings_valid_only
row_ind, col_ind = linear_sum_assignment(valid_cost)

File "/home/liu/anaconda3/envs/tf1.15/lib/python3.6/site-packages/scipy/optimize/_hungarian.py", line 93, in linear_sum_assignment
raise ValueError("matrix contains invalid numeric entries")

ValueError: matrix contains invalid numeric entries

Traceback (most recent call last):
File "/home/liu/anaconda3/envs/tf1.15/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call
return fn(*args)
File "/home/liu/anaconda3/envs/tf1.15/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn
target_list, run_metadata)
File "/home/liu/anaconda3/envs/tf1.15/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
(0) Invalid argument: ValueError: matrix contains invalid numeric entries
Traceback (most recent call last):

File "/home/liu/anaconda3/envs/tf1.15/lib/python3.6/site-packages/tensorflow_core/python/ops/script_ops.py", line 235, in call
ret = func(*args)

File "/home/liu/disk1/Life/3DBoNetPoint818a(linux)/helper_net.py", line 115, in assign_mappings_valid_only
row_ind, col_ind = linear_sum_assignment(valid_cost)

File "/home/liu/anaconda3/envs/tf1.15/lib/python3.6/site-packages/scipy/optimize/_hungarian.py", line 93, in linear_sum_assignment
raise ValueError("matrix contains invalid numeric entries")

ValueError: matrix contains invalid numeric entries

 [[{{node bbox/PyFunc}}]]
 [[gradients/backbone/fa_layer1/ThreeInterpolate_grad/ThreeInterpolateGrad/_425]]

(1) Invalid argument: ValueError: matrix contains invalid numeric entries
Traceback (most recent call last):

File "/home/liu/anaconda3/envs/tf1.15/lib/python3.6/site-packages/tensorflow_core/python/ops/script_ops.py", line 235, in call
ret = func(*args)

File "/home/liu/disk1/Life/3DBoNetPoint818a(linux)/helper_net.py", line 115, in assign_mappings_valid_only
row_ind, col_ind = linear_sum_assignment(valid_cost)

File "/home/liu/anaconda3/envs/tf1.15/lib/python3.6/site-packages/scipy/optimize/_hungarian.py", line 93, in linear_sum_assignment
raise ValueError("matrix contains invalid numeric entries")

ValueError: matrix contains invalid numeric entries

I am training a dental pointcloud dataset that has already been tested on ASIS with quite good results, but now testing on 3D-BoNet and having used the recommended h5 files generation script the algorithm crashes after few iterations:

  File "/home/ubuntu/3D-BoNet/helper_net.py", line 120, in assign_mappings_valid_only
    row_ind, col_ind=linear_sum_assignment(valid_cost)

  File "/home/ubuntu/miniconda3/envs/tf_gpu114_p36_source/lib/python3.6/site-packages/scipy/optimize/_hungarian.py", line 93, in linear_sum_assignment
    raise ValueError("matrix contains invalid numeric entries")

ValueError: matrix contains invalid numeric entries

Have you any idea for why could this be happening? A feature of the dataset is that each block has very few instances, and that every mandible (room) has an instance present in all blocks (mouth gum).

Hi @Yang7879,

Thanks for your excellent work. I am trying to extend Bo-Net and now go well on the ScanNet eval dataset. However, when I try to check the performance on the test dataset, the ScanNet benchmark does not respond.

I have successfully created an account and am trying to submit my 3D instance segmentation results. After filling the information required in the 'Create New Submission' page and clicking on the 'Save' button, the website always returns "This page isn't working; kaldir.vc.in.tum.de is currently unable to handle this request. Http Error 500".

I was wondering if you met this problem during submission. It would be great if any hint is available.

Best Regards,
Jianyuan

@Yang7879 Could you share the code to make dataset with scanNet (v2) or describe how to do it? Thanks

After testing, I got some .mat files. And there are many blocks in them. But I don't know how to deal with them to get the visualization of them. How should I put them together?

感谢杨博士能抽时间看看我的问题,我在尝试做同样的数据的格式，但是对instance label的ID编码规则不是很清楚。语义分割是按照13个类别的顺序按照0-12的规则对应，那么实例分割是按怎样的顺序呢？我看到实例的ID到40多了，希望能得到你的讲解。

Hi, @Yang7879 ,

When will the code be released?
THX!

Hi, @Yang7879

Thanks for sharing this amazing work,
do you have some experience result on KITTI ?

In scannet, the _vh_clean_2.ply is mesh file, I don't know how to get their xyzrgb attribute, can you please tell me how to get its xyzrgb value from mesh. thanks

Thanks for your nice work. But I could not train s3dis... I saved the downloaded *.h5 s in a folder (as you explained). But I got this error when I run "main_train":

1811670
model not found, all weights are initilized
check area: Area_1
check area: Area_2
check area: Area_3
check area: Area_4
check area: Area_6
check area: Area_5
train files: 0
test files: 0
train files shuffled!
total train batch num: 0
train files shuffled!
total train batch num: 0
train files shuffled!
total train batch num: 0
train files shuffled!
total train batch num: 0
train files shuffled!
total train batch num: 0
train files shuffled!
total train batch num: 0
train files shuffled!
...

Do you have any idea why I got this error and it couldn't read my data? Thanks a lot for your help.

Hi,Yang. I am reading the helper_data_s3dis.py, I have some confusion about the data contents.
Here are the code:
fin = h5py.File(file_path, 'r') coords = fin['coords'][block_id] points = fin['points'][block_id] semIns_labels = fin['labels'][block_id] pc = np.concatenate([coords, points[:, 3:9]], axis=-1)
I can not understand the ' pc = np.concatenate([coords, points[:, 3:9]], axis=-1)'
The coords mean the coordinates of the point clouds, and the semIns_labels are the semantices and instance labels of the point clouds, while the points means what? I know it's shape is (4096, 9).

could you please explain my confusion? Thanks

Hi, I'm Lawrence. I'm an undergraduate student studying applied mathematics at UC Berkeley. I saw your paper posted on /r/machinelearning and found it super interesting! I'm trying to re-implement the network myself right now in order to try and improve my programming / machine learning implementation skills.

I was looking at the point mask prediction part of your model, and I was wondering if instead of doing a binary classification, you could add on an additional level of detail with a multi-class label assuming the existing data set had classified labels.

I was thinking that this theoretically shouldn't have much effect on the loss since in the worst case, the classification should basically just degrade into the existing binary classification. But I'm finding that my loss shoots up dramatically when I try to implement it into the model. Is there something that I'm failing to account for here, that I should be thinking about or need to add in addition?

Thank you for your sharing of the codes. And I would like to ask can I use open3d for windows to visualize the results. I faced the problem below. I only use python for windows to visualize the results because it seems my server can't show the images, just see the second picture.

Thanks for sharing your great work. Would you please tell us about your result on ScanNet val set at 0.25AP with different class? That's very helpful for us to compare the results.

Thank you!