samsunglabs / td3d Goto Github PK

View Code? Open in Web Editor NEW

125.0 6.0 7.0 3.59 MB

[WACV'24] TD3D: Top-Down Beats Bottom-Up in 3D Instance Segmentation

License: Other

Python 99.66% Dockerfile 0.07% Shell 0.09% MATLAB 0.17%

3d-instance-segmentation pytorch s3dis scannet td3d

td3d's Introduction

TD3D: Top-Down Beats Bottom-Up in 3D Instance Segmentation

News:

🔥 February 6, 2023. We achieved SOTA results on the ScanNet test subset (mAP@25).
🔥 February 2023. The source code has been published.

This repository contains an implementation of TD3D, a 3D instance segmentation method introduced in our paper:

Top-Down Beats Bottom-Up in 3D Instance Segmentation
Maksim Kolodiazhnyi, Danila Rukhovich, Anna Vorontsova, Anton Konushin
Samsung Research
https://arxiv.org/abs/2302.02871

Installation

For convenience, we provide a Dockerfile.

Alternatively, you can install all required packages manually. This implementation is based on mmdetection3d framework.

Please refer to the original installation guide getting_started.md, including MinkowskiEngine installation, replacing open-mmlab/mmdetection3d with samsunglabs/td3d.

Most of the TD3D-related code locates in the following files: detectors/td3d_instance_segmentor.py, necks/ngfc_neck.py, decode_heads/td3d_instance_head.py.

Getting Started

Please see getting_started.md for basic usage examples. We follow the mmdetection3d data preparation protocol described in s3dis for S3DIS and in scannet for ScanNet and ScanNet200.

Training

To start training, run train with TD3D configs. To avoid gpu memory problems during validation callback, set score_thr to 0.15 and nms_pre to 100 in configs before training (then return them to their original values during testing):

python tools/train.py configs/td3d_is/td3d_is_scannet-3d-18class.py

For training on S3DIS with pretrained on ScanNet weights, download ScanNet model and put it into your working directory. Then use configs/td3d_is/td3d_is_s3dis-3d-5class_pretrain.py according to the previous instructions.

Testing

Test pre-trained model using test with TD3D configs:

python tools/test.py configs/td3d_is/td3d_is_scannet-3d-18class.py \
    work_dirs/td3d_is_scannet-3d-18class/latest.pth --eval mAP

Visualization

Visualizations can be created with test script. For better visualizations, you may set score_thr to 0.20 and nms_pre to 200 in configs:

python tools/test.py configs/td3d_is/td3d_is_scannet-3d-18class.py \
    work_dirs/td3d_is_scannet-3d-18class/latest.pth --eval mAP --show \
    --show-dir work_dirs/td3d_is_scannet-3d-18class

Models (quality on validation subset)

Dataset	[email protected]	[email protected]	mAP	Download
ScanNet	81.9	71.2	47.3	model \| config
S3DIS (5 area)	73.8	65.1	48.6	model \| config
S3DIS (5 area) (ScanNet pretrain)	75.0	67.2	52.1	model \| config
ScanNet200	40.4	34.8	23.1	model \| config
STPLS3D	74.0	69.6	54.5	model \| config

Examples

Citation

If you find this work useful for your research, please cite our paper:

@misc{kolodiazhnyi2023topdown,
  doi = {10.48550/ARXIV.2302.02871},
  url = {https://arxiv.org/abs/2302.02871},
  author = {Kolodiazhnyi, Maksim and Rukhovich, Danila and Vorontsova, Anna and Konushin, Anton},
  title = {Top-Down Beats Bottom-Up in 3D Instance Segmentation},
  publisher = {arXiv},
  year = {2023}
}

td3d's People

Contributors

Stargazers

Watchers

Forkers

guttappa1238 cv-seg wen-binghui arndz standardgalactic liuxinren456852

td3d's Issues

KeyError: 'TD3DInstanceSegmentor is not in the models registry'

Hello, thank you for publishing this work, we are trying to use your netwok te sepatare our own point cloud objects, first we tried to use the demo data in TR3D network. We installed the required libraries as explained.

We are using the demo script to segment the demo data, here is the script we use:
'''
from argparse import ArgumentParser
from mmdet3d.apis import inference_segmentor, init_model, show_result_meshlab

parser = ArgumentParser()
parser.add_argument('--pcd', default='/media/navlab/GNSS/work/instance_segmentation/td3d-main/demo/data/scannet/scene0000_00.bin', help='Point cloud file')
parser.add_argument('--config', default='/media/navlab/GNSS/work/instance_segmentation/td3d-main/configs/td3d_is/td3d_is_scannet-3d-18class.py', help='Config file')
parser.add_argument('--checkpoint', default='/media/navlab/GNSS/work/instance_segmentation/td3d-main/pretrained_models/td3d_scannet.pth', help='Checkpoint file')
parser.add_argument('--device', default='cuda:0', help='Device used for inference')
parser.add_argument('--out-dir', type=str, default='/media/navlab/GNSS/work/instance_segmentation/td3d-main/evaluate_results/try1', help='dir to save results')
parser.add_argument('--show',action='store_true',help='show online visualization results')
parser.add_argument('--snapshot', action='store_true', help='whether to save online visualization results')
args = parser.parse_args()

model = init_model(args.config, args.checkpoint, device=args.device)
result, data = inference_segmentor(model, args.pcd)
show_result_meshlab(data,result,args.out_dir,show=args.show,snapshot=args.snapshot,task='seg',
palette=model.PALETTE)
'''

But we got this error:
(td3d1) navlab@navlab-ProLiant-DL380-Gen10:/media/navlab/GNSS/work/instance_segmentation/td3d-main$ /home/navlab/anaconda3/envs/td3d1/bin/python /media/navlab/GNSS/work/instance_segmentation/td3d-main/demo/pc_seg_demo_ok.py
/home/navlab/anaconda3/envs/td3d1/lib/python3.7/site-packages/MinkowskiEngine/init.py:42: UserWarning: The environment variable OMP_NUM_THREADS not set. MinkowskiEngine will automatically set OMP_NUM_THREADS=16. If you want to set OMP_NUM_THREADS manually, please export it on the command line before running a python script. e.g. export OMP_NUM_THREADS=12; python your_program.py. It is recommended to set it below 24.
"It is recommended to set it below 24.",
Traceback (most recent call last):
File "/media/navlab/GNSS/work/instance_segmentation/td3d-main/demo/pc_seg_demo_ok.py", line 17, in
model = init_model(args.config, args.checkpoint, device=args.device)
File "/home/navlab/anaconda3/envs/td3d1/lib/python3.7/site-packages/mmdet3d/apis/inference.py", line 60, in init_model
model = build_model(config.model, test_cfg=config.get('test_cfg'))
File "/home/navlab/anaconda3/envs/td3d1/lib/python3.7/site-packages/mmdet3d/models/builder.py", line 122, in build_model
return build_detector(cfg, train_cfg=train_cfg, test_cfg=test_cfg)
File "/home/navlab/anaconda3/envs/td3d1/lib/python3.7/site-packages/mmdet3d/models/builder.py", line 96, in build_detector
cfg, default_args=dict(train_cfg=train_cfg, test_cfg=test_cfg))
File "/home/navlab/anaconda3/envs/td3d1/lib/python3.7/site-packages/mmcv/utils/registry.py", line 237, in build
return self.build_func(*args, **kwargs, registry=self)
File "/home/navlab/anaconda3/envs/td3d1/lib/python3.7/site-packages/mmcv/cnn/builder.py", line 27, in build_model_from_cfg
return build_from_cfg(cfg, registry, default_args)
File "/home/navlab/anaconda3/envs/td3d1/lib/python3.7/site-packages/mmcv/utils/registry.py", line 62, in build_from_cfg
f'{obj_type} is not in the {registry.name} registry')
KeyError: 'TD3DInstanceSegmentor is not in the models registry'

Althgouh, we installed all these libraries:
Package Version
absl-py 2.0.0
addict 2.4.0
albumentations 1.3.0
aliyun-python-sdk-core 2.14.0
aliyun-python-sdk-kms 2.16.2
ansi2html 1.8.0
anyio 3.7.1
argon2-cffi 23.1.0
argon2-cffi-bindings 21.2.0
attrs 23.1.0
backcall 0.2.0
beautifulsoup4 4.12.2
black 23.3.0
bleach 6.0.0
cachetools 5.3.2
certifi 2022.12.7
cffi 1.15.1
chamfer 2.0.0
charset-normalizer 3.3.2
click 8.1.7
colorama 0.4.6
comm 0.1.4
ConfigArgParse 1.7
crcmod 1.7
cryptography 41.0.5
cycler 0.11.0
dash 2.14.1
dash-core-components 2.0.0
dash-html-components 2.0.0
dash-table 5.0.0
debugpy 1.7.0
decorator 5.1.1
defusedxml 0.7.1
descartes 1.1.0
emd-ext 0.0.0
entrypoints 0.4
exceptiongroup 1.1.3
fastjsonschema 2.18.1
fire 0.5.0
flake8 3.9.2
Flask 2.2.5
fonttools 4.38.0
google-auth 2.23.4
google-auth-oauthlib 0.4.6
grpcio 1.59.2
idna 3.4
imageio 2.31.2
importlib-metadata 6.7.0
importlib-resources 5.12.0
iniconfig 2.0.0
ipykernel 6.16.2
ipython 7.34.0
ipython-genutils 0.2.0
ipywidgets 8.1.1
itsdangerous 2.1.2
jedi 0.19.1
Jinja2 3.1.2
jmespath 0.10.0
joblib 1.3.2
jsonschema 4.17.3
jupyter 1.0.0
jupyter_client 7.4.9
jupyter-console 6.6.3
jupyter_core 4.12.0
jupyter-server 1.24.0
jupyterlab-pygments 0.2.2
jupyterlab-widgets 3.0.9
kiwisolver 1.4.5
llvmlite 0.36.0
lyft-dataset-sdk 0.0.8
Markdown 3.4.4
markdown-it-py 2.2.0
MarkupSafe 2.1.3
matplotlib 3.5.3
matplotlib-inline 0.1.6
mccabe 0.6.1
mdurl 0.1.2
MinkowskiEngine 0.5.4
mistune 3.0.2
mmcls 0.25.0
mmcv-full 1.6.0
mmdet 2.24.1
mmdet3d 1.0.0rc3
mmsegmentation 0.24.1
model-index 0.1.11
mypy-extensions 1.0.0
nbclassic 1.0.0
nbclient 0.7.4
nbconvert 7.6.0
nbformat 5.7.0
nest-asyncio 1.5.8
networkx 2.2
notebook 6.5.6
notebook_shim 0.2.3
numba 0.53.0
numpy 1.21.5
nuscenes-devkit 1.1.9
oauthlib 3.2.2
open3d 0.17.0
opencv-python 4.8.1.78
opencv-python-headless 4.8.1.78
opendatalab 0.0.10
openmim 0.3.9
openxlab 0.0.10
ordered-set 4.1.0
oss2 2.17.0
packaging 23.2
pandas 1.1.5
pandocfilters 1.5.0
parso 0.8.3
pathspec 0.11.2
pexpect 4.8.0
pickleshare 0.7.5
Pillow 9.5.0
pip 22.3.1
pkgutil_resolve_name 1.3.10
platformdirs 3.11.0
plotly 5.18.0
pluggy 1.2.0
plyfile 0.7.4
prettytable 3.7.0
prometheus-client 0.17.1
prompt-toolkit 3.0.39
protobuf 3.20.3
psutil 5.9.6
ptyprocess 0.7.0
pyasn1 0.5.0
pyasn1-modules 0.3.0
pycocotools 2.0.7
pycodestyle 2.7.0
pycparser 2.21
pycryptodome 3.19.0
pyflakes 2.3.1
Pygments 2.16.1
pyparsing 3.1.1
pyquaternion 0.9.9
pyrsistent 0.19.3
pytest 7.4.3
python-dateutil 2.8.2
pytz 2023.3.post1
PyWavelets 1.3.0
PyYAML 6.0.1
pyzmq 24.0.1
qtconsole 5.4.4
QtPy 2.4.1
qudida 0.0.4
requests 2.28.2
requests-oauthlib 1.3.1
retrying 1.3.4
rich 13.6.0
rsa 4.9
scikit-image 0.19.3
scikit-learn 1.0.2
scipy 1.7.3
Send2Trash 1.8.2
setuptools 65.6.3
shapely 2.0.2
six 1.16.0
sniffio 1.3.0
soupsieve 2.4.1
tabulate 0.9.0
tenacity 8.2.3
tensorboard 2.11.2
tensorboard-data-server 0.6.1
tensorboard-plugin-wit 1.8.1
termcolor 2.3.0
terminado 0.17.1
terminaltables 3.1.10
threadpoolctl 3.1.0
tifffile 2021.11.2
tinycss2 1.2.1
tomli 2.0.1
torch 1.8.1+cu101
torchaudio 0.8.1
torchvision 0.9.1+cu101
tornado 6.2
tqdm 4.65.2
traitlets 5.9.0
trimesh 2.35.39
typed-ast 1.5.5
typing_extensions 4.7.1
urllib3 1.26.18
wcwidth 0.2.9
webencodings 0.5.1
websocket-client 1.6.1
Werkzeug 2.2.3
wheel 0.38.4
widgetsnbextension 4.0.9
yapf 0.40.2
zipp 3.15.0

Could you please help us to overcome this error? Thank you so much.
Regards

Unet not passed and unused parameters error sometimes

td3d/mmdet3d/models/decode_heads/td3d_instance_head.py

Lines 210 to 223 in 33d8073

 if tensors[0].features.shape[0] == 0: 

 return (targets.new_zeros((0, 1)), 

 targets.new_zeros((0, 1)), 

 targets.new_zeros(0), 

 targets.new_zeros(0), 

 [targets.new_zeros((0, 7)) for i in range(len(bbox_list))], 

 [targets.new_zeros(0) for i in range(len(bbox_list))], 

 [targets.new_zeros(0) for i in range(len(bbox_list))]) 

 feats = ME.SparseTensor(tensors[0].features[:, :-2], tensors[0].coordinates) 

 targets = tensors[0].features[:, -2:] 

 preds = self.unet(feats).features 

 return preds, targets, feats.coordinates[:, 0].long(), ids[0], rois[0], scores[0], labels[0]

In this If branch, if we have detection result so poor that no GT is assgined to any rois, it will return a empty result with proper shapes.

However, if we trigger this branch and return without passing through the Unet, pytorch will find parameters of Unet unused and raises error, which may happen in a few iterations after training starts.

Adding find_unused_parameters=True in the config file would solve this preblem. However this may be brutal and cost more time for pytorch will check all parameters each iteration. Even if find_unused_parameters can be turned off after some training, like 1 epoch.

Another solution is to set Unet not requiring grad when going in this branch and vice versa. Unfortunately I failed trying setting it.

Not running

Thank you for your wonderful work!

I am running td3d on my dataset, which only includes one class. I have add necessary modification to the original code, to my best knowledge.

It seems to run normally. However, the training just won't start, showed as the following. The GPU memory is occupied but the GPU is not working (CUDA is available). Instead, the CPU is running. Could you please help me to find the problem? Thank you!

2023-08-03 05:56:47,461 - mmdet - INFO - Start running, host: root@autodl-container-8ce5118fae-93ac1f8e, work_dir: /root/autodl-tmp/td3d/work_dirs/td3d_is_s3dis-3d-5class
2023-08-03 05:56:47,461 - mmdet - INFO - Hooks will be executed in the following order:
before_run:
(VERY_HIGH ) StepLrUpdaterHook
(NORMAL ) CheckpointHook
(LOW ) EvalHook
(VERY_LOW ) TextLoggerHook

before_train_epoch:
(VERY_HIGH ) StepLrUpdaterHook
(NORMAL ) EmptyCacheHook
(LOW ) IterTimerHook
(LOW ) EvalHook
(VERY_LOW ) TextLoggerHook

before_train_iter:
(VERY_HIGH ) StepLrUpdaterHook
(LOW ) IterTimerHook
(LOW ) EvalHook

after_train_iter:
(ABOVE_NORMAL) OptimizerHook
(NORMAL ) CheckpointHook
(NORMAL ) EmptyCacheHook
(LOW ) IterTimerHook
(LOW ) EvalHook
(VERY_LOW ) TextLoggerHook

after_train_epoch:
(NORMAL ) CheckpointHook
(NORMAL ) EmptyCacheHook
(LOW ) EvalHook
(VERY_LOW ) TextLoggerHook

before_val_epoch:
(NORMAL ) EmptyCacheHook
(LOW ) IterTimerHook
(VERY_LOW ) TextLoggerHook

before_val_iter:
(LOW ) IterTimerHook

after_val_iter:
(NORMAL ) EmptyCacheHook
(LOW ) IterTimerHook

after_val_epoch:
(NORMAL ) EmptyCacheHook
(VERY_LOW ) TextLoggerHook

after_run:
(VERY_LOW ) TextLoggerHook

2023-08-03 05:56:47,461 - mmdet - INFO - workflow: [('train', 1)], max: 33 epochs
2023-08-03 05:56:47,461 - mmdet - INFO - Checkpoints will be saved to /root/autodl-tmp/td3d/work_dirs/td3d_is_s3dis-3d-5class by HardDiskBackend.

(I waited for half an hour and there is no further log.)

Training aborts when saving checkpoint after epoch 1

Hi, I am currently trying to train the network on the S3DIS dataset using the td3d_is_s3dis-3d-5class config.

The training works fine for all training steps in epoch 1. At the end of the epoch when saving the checkpoint, the memory usage on the GPU suddenly jumps from ~8/9 GB to 18 GB and eventually failing when reaching the limit of 24 GB.

Is this an known issue?

2023-05-16 14:05:55,895 - mmdet - INFO - workflow: [('train', 1)], max: 33 epochs
2023-05-16 14:05:55,895 - mmdet - INFO - Checkpoints will be saved to /mmdetection3d/tools/work_dirs/td3d_is_s3dis-3d-5class by HardDiskBackend.
/usr/local/lib/python3.7/dist-packages/MinkowskiEngine/MinkowskiSparseTensor.py:298: UserWarning: coordinates implicitly converted to torch.IntTensor. To remove this warning, use `.int()` to convert the coords into an torch.IntTensor
  + "coords into an torch.IntTensor"
2023-05-16 14:06:42,654 - mmdet - INFO - Epoch [1][50/663]	lr: 1.000e-03, eta: 5:40:12, time: 0.935, data_time: 0.290, memory: 7883, bbox_loss: 0.8248, cls_loss: 0.7217, inst_loss: 0.7607, loss: 2.3071, grad_norm: 1.5957
2023-05-16 14:07:15,048 - mmdet - INFO - Epoch [1][100/663]	lr: 1.000e-03, eta: 4:47:17, time: 0.648, data_time: 0.016, memory: 7883, bbox_loss: 0.7341, cls_loss: 0.3994, inst_loss: 0.6373, loss: 1.7708, grad_norm: 0.9642
2023-05-16 14:07:50,540 - mmdet - INFO - Epoch [1][150/663]	lr: 1.000e-03, eta: 4:36:46, time: 0.710, data_time: 0.035, memory: 8027, bbox_loss: 0.7062, cls_loss: 0.3581, inst_loss: 0.6261, loss: 1.6904, grad_norm: 1.1923
2023-05-16 14:08:25,190 - mmdet - INFO - Epoch [1][200/663]	lr: 1.000e-03, eta: 4:29:42, time: 0.693, data_time: 0.014, memory: 8027, bbox_loss: 0.6692, cls_loss: 0.3358, inst_loss: 0.6145, loss: 1.6194, grad_norm: 1.0767
2023-05-16 14:09:01,773 - mmdet - INFO - Epoch [1][250/663]	lr: 1.000e-03, eta: 4:28:00, time: 0.732, data_time: 0.023, memory: 8027, bbox_loss: 0.6513, cls_loss: 0.3226, inst_loss: 0.6042, loss: 1.5781, grad_norm: 1.2070
2023-05-16 14:09:39,756 - mmdet - INFO - Epoch [1][300/663]	lr: 1.000e-03, eta: 4:28:21, time: 0.760, data_time: 0.015, memory: 8027, bbox_loss: 0.6300, cls_loss: 0.3100, inst_loss: 0.5524, loss: 1.4923, grad_norm: 1.2423
2023-05-16 14:10:18,196 - mmdet - INFO - Epoch [1][350/663]	lr: 1.000e-03, eta: 4:28:53, time: 0.769, data_time: 0.015, memory: 8027, bbox_loss: 0.6168, cls_loss: 0.3033, inst_loss: 0.5165, loss: 1.4367, grad_norm: 1.2490
2023-05-16 14:11:00,874 - mmdet - INFO - Epoch [1][400/663]	lr: 1.000e-03, eta: 4:32:56, time: 0.854, data_time: 0.056, memory: 8638, bbox_loss: 0.6106, cls_loss: 0.2944, inst_loss: 0.5128, loss: 1.4178, grad_norm: 1.3136
2023-05-16 14:11:40,923 - mmdet - INFO - Epoch [1][450/663]	lr: 1.000e-03, eta: 4:33:49, time: 0.801, data_time: 0.017, memory: 8638, bbox_loss: 0.6041, cls_loss: 0.2857, inst_loss: 0.4876, loss: 1.3774, grad_norm: 1.3142
2023-05-16 14:12:23,333 - mmdet - INFO - Epoch [1][500/663]	lr: 1.000e-03, eta: 4:36:05, time: 0.848, data_time: 0.021, memory: 8638, bbox_loss: 0.5784, cls_loss: 0.2747, inst_loss: 0.4711, loss: 1.3242, grad_norm: 1.2854
2023-05-16 14:13:04,558 - mmdet - INFO - Epoch [1][550/663]	lr: 1.000e-03, eta: 4:37:03, time: 0.824, data_time: 0.014, memory: 8638, bbox_loss: 0.5704, cls_loss: 0.2632, inst_loss: 0.4488, loss: 1.2824, grad_norm: 1.2698
2023-05-16 14:13:47,635 - mmdet - INFO - Epoch [1][600/663]	lr: 1.000e-03, eta: 4:38:49, time: 0.862, data_time: 0.025, memory: 8638, bbox_loss: 0.5713, cls_loss: 0.2618, inst_loss: 0.4385, loss: 1.2715, grad_norm: 1.3437
2023-05-16 14:14:31,222 - mmdet - INFO - Epoch [1][650/663]	lr: 1.000e-03, eta: 4:40:30, time: 0.872, data_time: 0.055, memory: 8638, bbox_loss: 0.5565, cls_loss: 0.2557, inst_loss: 0.4479, loss: 1.2601, grad_norm: 1.3006
2023-05-16 14:14:41,589 - mmdet - INFO - Saving checkpoint at 1 epochs
[>>>>>>                                            ] 9/68, 1.0 task/s, elapsed: 9s, ETA:    60sTraceback (most recent call last):
  File "train.py", line 263, in <module>
    main()
  File "train.py", line 259, in main
    meta=meta)
  File "/usr/local/lib/python3.7/dist-packages/mmdet3d/apis/train.py", line 351, in train_model
    meta=meta)
  File "/usr/local/lib/python3.7/dist-packages/mmdet3d/apis/train.py", line 319, in train_detector
    runner.run(data_loaders, cfg.workflow)
  File "/usr/local/lib/python3.7/dist-packages/mmcv/runner/epoch_based_runner.py", line 136, in run
    epoch_runner(data_loaders[i], **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/mmcv/runner/epoch_based_runner.py", line 58, in train
    self.call_hook('after_train_epoch')
  File "/usr/local/lib/python3.7/dist-packages/mmcv/runner/base_runner.py", line 317, in call_hook
    getattr(hook, fn_name)(self)
  File "/usr/local/lib/python3.7/dist-packages/mmcv/runner/hooks/evaluation.py", line 271, in after_train_epoch
    self._do_evaluate(runner)
  File "/usr/local/lib/python3.7/dist-packages/mmdet/core/evaluation/eval_hooks.py", line 56, in _do_evaluate
    results = single_gpu_test(runner.model, self.dataloader, show=False)
  File "/usr/local/lib/python3.7/dist-packages/mmdet/apis/test.py", line 29, in single_gpu_test
    result = model(return_loss=False, rescale=True, **data)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/mmcv/parallel/data_parallel.py", line 51, in forward
    return super().forward(*inputs, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/parallel/data_parallel.py", line 166, in forward
    return self.module(*inputs[0], **kwargs[0])
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/mmcv/runner/fp16_utils.py", line 116, in new_func
    return old_func(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/mmdet3d/models/detectors/base.py", line 62, in forward
    return self.forward_test(**kwargs)
  File "/usr/local/lib/python3.7/dist-packages/mmdet3d/models/detectors/base.py", line 43, in forward_test
    return self.simple_test(points[0], img_metas[0], img[0], **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/mmdet3d/models/detectors/td3d_instance_segmentor.py", line 122, in simple_test
    instances = self.head.forward_test(x, field, img_metas)
  File "/usr/local/lib/python3.7/dist-packages/mmdet3d/models/decode_heads/td3d_instance_head.py", line 556, in forward_test
    cls_preds, idxs, v2r, r2scene, rois, scores, labels = self._forward_second(x[0], src_idxs, bbox_list)
  File "/usr/local/lib/python3.7/dist-packages/mmdet3d/models/decode_heads/td3d_instance_head.py", line 222, in _forward_second
    preds = self.unet(feats).features
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/mmdet3d/models/backbones/mink_unet.py", line 225, in forward
    out = self.conv0p1s1(x)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/MinkowskiEngine/MinkowskiConvolution.py", line 321, in forward
    input._manager,
  File "/usr/local/lib/python3.7/dist-packages/MinkowskiEngine/MinkowskiConvolution.py", line 84, in forward
    coordinate_manager._manager,
MemoryError: std::bad_alloc: cudaErrorMemoryAllocation: out of memory

where `data_infos` is defined?

Could anyone tell me where data_infos is defined?
I would appreciate if you could reply to me.

Issue with mmcv version

Hi,
when I preprocess S3dis data with file "tools/create_data.py", I run into problem:

File "tools/create_data.py", line 5, in <module>
    from tools.data_converter import indoor_converter as indoor
ModuleNotFoundError: No module named 'tools.data_converter'

so I run with command "python -m tools.create_data.py s3dis ***", but I run into weird mmcv version problem:
If I run with version 1.6.0, i get:

Traceback (most recent call last):
  File "/home/jerry/anaconda3/envs/openmmlab/lib/python3.8/runpy.py", line 185, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/home/jerry/anaconda3/envs/openmmlab/lib/python3.8/runpy.py", line 111, in _get_module_details
    __import__(pkg_name)
  File "/home/jerry/3D/3d_seg/td3d/td3d/tools/create_data.py", line 6, in <module>
    from tools.data_converter import kitti_converter as kitti
  File "/home/jerry/3D/3d_seg/td3d/td3d/tools/data_converter/kitti_converter.py", line 9, in <module>
    from mmdet3d.core.bbox import box_np_ops, points_cam2img
  File "/home/jerry/3D/3d_seg/td3d/td3d/mmdet3d/__init__.py", line 4, in <module>
    import mmdet
  File "/home/jerry/anaconda3/envs/openmmlab/lib/python3.8/site-packages/mmdet/__init__.py", line 16, in <module>
    assert (mmcv_version >= digit_version(mmcv_minimum_version)
AssertionError: MMCV==1.6.0 is used but incompatible. Please install mmcv>=2.0.0rc4, <2.1.0.

But if I change the version to 2.0.0, I get this error instead:

Traceback (most recent call last):
  File "/home/jerry/anaconda3/envs/openmmlab/lib/python3.8/runpy.py", line 185, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/home/jerry/anaconda3/envs/openmmlab/lib/python3.8/runpy.py", line 111, in _get_module_details
    __import__(pkg_name)
  File "/home/jerry/3D/3d_seg/td3d/td3d/tools/create_data.py", line 6, in <module>
    from tools.data_converter import kitti_converter as kitti
  File "/home/jerry/3D/3d_seg/td3d/td3d/tools/data_converter/kitti_converter.py", line 9, in <module>
    from mmdet3d.core.bbox import box_np_ops, points_cam2img
  File "/home/jerry/3D/3d_seg/td3d/td3d/mmdet3d/__init__.py", line 26, in <module>
    assert (mmcv_version >= digit_version(mmcv_minimum_version)
AssertionError: MMCV==2.0.0 is used but incompatible. Please install mmcv>=1.4.8, <=1.6.0.

It seems that the newest openmmlab does not agree with this code.
How can we fix this?

Thanks

question

Whether to consider upgrading to rotation detection?

How long does it take for training.

Hi, thanks for your great job! In your paper, you mentioned that you train for 330 epochs on a single NVidia 3090 GPU, using Adam optimizer. May I ask how long does it take? How many days/hours? Look forward to your reply. Thank you in advance!

mmdet - WARNING - The model and loaded state dict do not match exactly

When I run python "tools/train.pyconfigs/td3d _ is/td3d _ is _ s3dis-3d-13class _ pretrain.py", the following warning appears. I wonder if there is something wrong with the model.

2024-04-25 07:07:05,302 - mmdet - INFO - load checkpoint from local path: td3d_scannet.pth
2024-04-25 07:07:05,798 - mmdet - WARNING - The model and loaded state dict do not match exactly

CUDA Out of memory during training on S3DIS

Hi, @filaPro

Can I ask for the detailed hardware specs you are using for training? We are using an RTX 3090ti with the default setup, but this causes the CUDA out of memory. Thank you for your time.

Welcome update to OpenMMLab 2.0

I am Vansin, the technical operator of OpenMMLab. In September of last year, we announced the release of OpenMMLab 2.0 at the World Artificial Intelligence Conference in Shanghai. We invite you to upgrade your algorithm library to OpenMMLab 2.0 using MMEngine, which can be used for both research and commercial purposes. If you have any questions, please feel free to join us on the OpenMMLab Discord at https://discord.gg/A9dCpjHPfE or add me on WeChat (ID: van-sin) and I will invite you to the OpenMMLab WeChat group.

Here are the OpenMMLab 2.0 repos branches:

	OpenMMLab 1.0 branch	OpenMMLab 2.0 branch
MMEngine		0.x
MMCV	1.x	2.x
MMDetection	0.x 、1.x、2.x	3.x
MMAction2	0.x	1.x
MMClassification	0.x	1.x
MMSegmentation	0.x	1.x
MMDetection3D	0.x	1.x
MMEditing	0.x	1.x
MMPose	0.x	1.x
MMDeploy	0.x	1.x
MMTracking	0.x	1.x
MMOCR	0.x	1.x
MMRazor	0.x	1.x
MMSelfSup	0.x	1.x
MMRotate	0.x	1.x
MMYOLO		0.x

Attention: please create a new virtual environment for OpenMMLab 2.0.

"td3d_is_s3dis-3d-5class_pretrain.py" does not exist

I can't find the file "td3d _ is _ s3dis-3d-5 class _ pretrain.py".

Key 'metainfo' does not contain 'pts_path'

When I solved the problem of #22, I found :
key' meta info' does not contain' pts _ path，Key 'data_list' does not contain 'pts_path.
Moreover, there are only two key values "metainfo" and "data_list" in self.data_infos. This means that
pts _ filename = OSP. Join (self. data _ root, info ['pts _ path']) ，Input _ dict = dict (pts _ filename = pts _ filename),cannot be performed. how to solve this problem?

About the result of wall.

Hi,
It's a nice work. I found that the current methods rarely deal with the boundary.How does your method compare to Mask3D about results on the wall boundary?
I'm looking forward to hearing back from you.
Best.

Outdoor or Indoor

Thanks for your great job！ Congratulation！
I have some questions：

Can it use for outdoor scene？ Like school，street. In three config files, I do not find outdoor classes.
What size scene does it work on? Also considering the outdoor scene.
Looking forward to your reply.

Using -1 to represent background causes error

td3d/mmdet3d/datasets/pipelines/transforms_3d.py

Lines 524 to 535 in 33d8073

 num_pts_per_inst_src = pts_instance_mask_one_hot.sum(axis=0) 

 pts_instance_mask = torch.cat((torch.tensor(input_dict['pts_instance_mask'][new_idxs]), inst_idxs)) 

 idxs_sum = (pts_instance_mask == -1).sum() 

 pts_instance_mask[pts_instance_mask == -1] = torch.max(pts_instance_mask) + 1 

 pts_instance_mask_one_hot = torch.nn.functional.one_hot(pts_instance_mask) 

 if idxs_sum > 0: 

 pts_instance_mask_one_hot = pts_instance_mask_one_hot[:, :-1] 

 pts_instance_mask_one_hot = pts_instance_mask_one_hot[:-len(inst_idxs), :] 

 num_pts_per_inst = pts_instance_mask_one_hot.sum(axis=0) 

 good_insts = num_pts_per_inst / num_pts_per_inst_src > self.remaining_points_thr

Here, if one uses -1 to represent background, the calculation of good_insts will raises error for the shapes of these two tensors does not fit. This is because pts_instance_mask_one_hot is cut in line 531 and 'num_pts_per_inst' will have an extra element.

The solution is to move line 524 to 532, after 'if'.

Caught KeyError in DataLoader worker process 0.

When I was running "python tools/train.pyconfigs/td3d _ is/td3d _ is _ s3dis-3d-13class _ pretrain.py"in docker, the following error appeared.

Traceback (most recent call last):
  File "tools/train.py", line 263, in <module>
    main()
  File "tools/train.py", line 259, in main
    meta=meta)
  File "/mmdetection3d/mmdet3d/apis/train.py", line 351, in train_model
    meta=meta)
  File "/mmdetection3d/mmdet3d/apis/train.py", line 319, in train_detector
    runner.run(data_loaders, cfg.workflow)
  File "/opt/conda/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 136, in run
    epoch_runner(data_loaders[i], **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 49, in train
    for i, data_batch in enumerate(self.data_loader):
  File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 521, in __next__
    data = self._next_data()
  File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1203, in _next_data
    return self._process_data(data)
  File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1229, in _process_data
    data.reraise()
  File "/opt/conda/lib/python3.7/site-packages/torch/_utils.py", line 434, in reraise
    raise exception
KeyError: Caught KeyError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
    data = fetcher.fetch(index)
  File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 49, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/opt/conda/lib/python3.7/site-packages/mmdet/datasets/dataset_wrappers.py", line 178, in __getitem__
    return self.dataset[idx % self._ori_len]
  File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataset.py", line 308, in __getitem__
    return self.datasets[dataset_idx][sample_idx]
  File "/mmdetection3d/mmdet3d/datasets/custom_3d.py", line 435, in __getitem__
    data = self.prepare_train_data(idx)
  File "/mmdetection3d/mmdet3d/datasets/custom_3d.py", line 225, in prepare_train_data
    input_dict = self.get_data_info(index)
  File "/mmdetection3d/mmdet3d/datasets/s3dis_dataset.py", line 128, in get_data_info
    info = self.data_infos[index]
KeyError: 0

How to finalize and utilize demo test code

Dear all, thank you for this network. I finally installed all the required libraries and solved all the errors that appeared.
I used the demo test code to instancely segment the point cloud file for some chairs. This is the snippet used:

from argparse import ArgumentParser
from mmdet3d.apis import inference_segmentor, init_model
import numpy as np
import os
# Convert from txt file to a 32 float bin file  
input_txt_file = '/media/navlab/GNSS/work/instance_segmentation/td3d-main/demo/tested_data/twochair.txt'
filename = os.path.splitext(os.path.basename(input_txt_file))[0]
output_bin_file = f'{input_txt_file}_ok.bin'

# Load the text file and save the bin file
point_cloud = np.loadtxt(input_txt_file, delimiter=',')
point_cloud.astype(np.float32).tofile(output_bin_file)
parser = ArgumentParser()
parser.add_argument('--pcd', default=f'{output_bin_file}', help='Point cloud file')  
parser.add_argument('--config', default='/media/navlab/GNSS/work/instance_segmentation/td3d-main/configs/td3d_is/td3d_is_scannet-3d-18class.py', help='Config file')
parser.add_argument('--checkpoint', default='/media/navlab/GNSS/work/instance_segmentation/td3d-main/pretrained_models/td3d_scannet.pth', help='Checkpoint file')
parser.add_argument('--device', default='cuda:0', help='Device used for inference')
parser.add_argument('--out-dir', type=str, default=f'/media/navlab/GNSS/work/instance_segmentation/td3d-main/demo/extracted_info/{filename}', help='Directory to save results')
parser.add_argument('--show', action='store_true', help='Show online visualization results')
parser.add_argument('--snapshot', action='store_true', help='Whether to save online visualization results')
args = parser.parse_args()

# Build the model from a config file and a checkpoint file
model = init_model(args.config, args.checkpoint, device=args.device)
result, data = inference_segmentor(model, args.pcd)
print(result)
print(data)

And this is the output of the snippet:

[{'instance_mask': tensor([[False, False, False,  ...,  True,  True,  True],
        [ True,  True,  True,  ..., False, False, False],
        [False, False, False,  ...,  True,  True,  True]]), 'instance_label': tensor([2, 2, 3]), 'instance_score': tensor([0.5066, 0.4710, 0.2660])}]
{'img_metas': [[{'flip': False, 'pcd_horizontal_flip': False, 'pcd_vertical_flip': False, 'pcd_scale_factor': 1.0, 'pts_filename': '/media/navlab/GNSS/work/instance_segmentation/td3d-main/demo/tested_data/twochair.txt_ok.bin', 'box_type_3d': <class 'mmdet3d.core.bbox.structures.depth_box3d.DepthInstance3DBoxes'>}]], 'points': [[tensor([[ -1.9100, -22.3000,   0.4300,   0.0000,   0.0000,   0.0000],
        [ -1.8800, -22.3000,   0.4300,   0.0000,   0.0000,   0.0000],
        [ -1.8400, -22.3000,   0.4300,   0.0000,   0.0000,   0.0000],
        ...,
        [ -2.2600, -20.2300,   0.5300,   0.0000,   0.0000,   0.0000],
        [ -2.2700, -20.2100,   0.5400,   0.0000,   0.0000,   0.0000],
        [ -2.2600, -20.2200,   0.5300,   0.0000,   0.0000,   0.0000]],
       device='cuda:0')]]}

Could you tell me how to extract a point cloud file for each chair?
Thank you.

	if tensors[0].features.shape[0] == 0:
	return (targets.new_zeros((0, 1)),
	targets.new_zeros((0, 1)),
	targets.new_zeros(0),
	targets.new_zeros(0),
	[targets.new_zeros((0, 7)) for i in range(len(bbox_list))],
	[targets.new_zeros(0) for i in range(len(bbox_list))],
	[targets.new_zeros(0) for i in range(len(bbox_list))])

	feats = ME.SparseTensor(tensors[0].features[:, :-2], tensors[0].coordinates)
	targets = tensors[0].features[:, -2:]

	preds = self.unet(feats).features
	return preds, targets, feats.coordinates[:, 0].long(), ids[0], rois[0], scores[0], labels[0]

	num_pts_per_inst_src = pts_instance_mask_one_hot.sum(axis=0)

	pts_instance_mask = torch.cat((torch.tensor(input_dict['pts_instance_mask'][new_idxs]), inst_idxs))
	idxs_sum = (pts_instance_mask == -1).sum()
	pts_instance_mask[pts_instance_mask == -1] = torch.max(pts_instance_mask) + 1
	pts_instance_mask_one_hot = torch.nn.functional.one_hot(pts_instance_mask)
	if idxs_sum > 0:
	pts_instance_mask_one_hot = pts_instance_mask_one_hot[:, :-1]

	pts_instance_mask_one_hot = pts_instance_mask_one_hot[:-len(inst_idxs), :]
	num_pts_per_inst = pts_instance_mask_one_hot.sum(axis=0)
	good_insts = num_pts_per_inst / num_pts_per_inst_src > self.remaining_points_thr

samsunglabs / td3d Goto Github PK

td3d's Introduction

TD3D: Top-Down Beats Bottom-Up in 3D Instance Segmentation

Installation

Getting Started

Models (quality on validation subset)

Examples

Citation

td3d's People

Contributors

Stargazers

Watchers

Forkers

td3d's Issues

before_train_epoch: (VERY_HIGH ) StepLrUpdaterHook (NORMAL ) EmptyCacheHook (LOW ) IterTimerHook (LOW ) EvalHook (VERY_LOW ) TextLoggerHook

before_train_iter: (VERY_HIGH ) StepLrUpdaterHook (LOW ) IterTimerHook (LOW ) EvalHook

after_train_iter: (ABOVE_NORMAL) OptimizerHook (NORMAL ) CheckpointHook (NORMAL ) EmptyCacheHook (LOW ) IterTimerHook (LOW ) EvalHook (VERY_LOW ) TextLoggerHook

after_train_epoch: (NORMAL ) CheckpointHook (NORMAL ) EmptyCacheHook (LOW ) EvalHook (VERY_LOW ) TextLoggerHook

before_val_epoch: (NORMAL ) EmptyCacheHook (LOW ) IterTimerHook (VERY_LOW ) TextLoggerHook

before_val_iter: (LOW ) IterTimerHook

after_val_iter: (NORMAL ) EmptyCacheHook (LOW ) IterTimerHook

after_val_epoch: (NORMAL ) EmptyCacheHook (VERY_LOW ) TextLoggerHook

after_run: (VERY_LOW ) TextLoggerHook