Question regarding evaluation on S3DIS about 3d-bonet HOT 1 OPEN

Hi @chrischoy , thanks for your interest in our paper.
(1) The results can indeed be different from the paper if the model is trained again. It can be better or worse. For example, the released model (trained after NeurIPS submission) for Area 5 is better than the paper, but it may also be worse as you reported. I guess the primary reason is the Hungarian algorithm which may bring about instability during training. It seems a more stable back-prop algorithm is worthwhile to be explored. (Btw, all network configurations are the same.)

(2) I do agree mAP is more general to measure the results of obj detection or ins segmentation. However, the reported mAP scores of the first paper SGPN are incorrect according to their released code, which is also pointed out in GSPN. For a fair comparison with the ASIS which was SoTA on S3DIS, we simply follow their mPrec/mRec protocol. For the benefit of the community, I strongly believe a standard mAP protocol and the correct implementation are quite important.

(3) Here are the per-category pre/rec scores of 3D-BoNet (6 fold cross-validation), but unfortunately the results for other baselines are no longer available.

-----------pre/rec-------
ceiling: 0.8852/0.6180
floor: 0.8989/0.7464
wall: 0.6487/0.4999
beam: 0.4230/0.4217
column: 0.4801/0.2716
window: 0.9301/0.6242
door: 0.6676/0.5845
table: 0.5539/0.4861
chair: 0.7198/0.6158
sofa: 0.4972/0.2876
bookcase: 0.5830/0.2843
board: 0.8074/0.4648
clutter: 0.4762/0.2860

from 3d-bonet.