gap-lab-cuhk-sz / to-scene Goto Github PK

View Code? Open in Web Editor NEW

41.0 3.0 5.0 33.66 MB

(ECCV 2022 Oral) TO-Scene: A Large-scale Dataset for Understanding 3D Tabletop Scenes

License: MIT License

Shell 0.30% C++ 1.87% Python 94.26% Cuda 2.67% MATLAB 0.91%

to-scene's Introduction

TO-Scene: A Large-scale Dataset for Understanding 3D Tabletop Scenes

By Mutian Xu*, Pei Chen*, Haolin Liu, and Xiaoguang Han

Introduction

This repository is built for:

TO-Scene: A Large-scale Dataset for Understanding 3D Tabletop Scenes (ECCV2022 Oral) [arXiv]

If you find our work useful in your research, please consider citing:

@inproceedings{xu2022toscene,
  title={TO-Scene: A Large-scale Dataset for Understanding 3D Tabletop Scenes},
  author={Xu, Mutian and Chen, Pei and Liu, Haolin and Han, Xiaoguang},
  booktitle={ECCV},
  year={2022}
}

Dataset

TO-Scene contains 20,740 scenes with three different variants which are TO_Vanilla, TO_Crowd, TO_ScanNet.

1. Full TO-Scene data:

You can download our dataset with the corresponding variants from Google Drive:

Format	TO_Vanilla	TO_Crowd	TO_ScanNet
ply (point cloud)	Download (4.3GB)	Download (2.1GB)	Download (4.3GB)
npz (xyz, color, semantic_label, instance_label, bbox)	Download (6.2GB)	Download (2.8GB)	Download (6.5GB)

Alternatively, for mainland China users, we also provide Baiduyun link:

Format	TO_Vanilla	TO_Crowd	TO_ScanNet
ply (point cloud)	Download (4.3GB)	Download (2.1GB)	Download (4.3GB)
npz (xyz, color, semantic_label, instance_label, bbox)	Download (6.2GB)	Download (2.8GB)	Download (6.5GB)

Additionally, you can download our TO-Real data we mentioned in the paper,

2. Meta data:

You may also need to download meta_data at Google Drive or Baiduyun, including train/val/test split in format of .txt and pre-calculated mean_size_arr (mean size of objects of different classes).

3. Build your own dataset:

We have released the original CAD placement file (json file, describing the position, scale, and orientation of the small objects). You can run your own rendering and reconstruction based on this original annotation file, with parameters (e.g. point density) in your control.

Benchmark on TO-Scene

We have provided the code implementations for running 3D semantic segmentation and 3D object detection on our dataset, with the corresponding instructions.

Note that TO-Scene dataset contains 60,174 tabletop object instances from 52 common classes. For reference, we show the classes of these small tabletop objects below, which can be downloaded here as well.

Big furniture

Class	Semantic	Class	Semantic	Class	Semantic
3	cabinet	9	window	24	refrigerator
4	bed	10	bookshelf	28	showercurtain
5	chair	11	picture	33	toilet
6	sofa	12	counter	34	sink
7	table	14	desk	36	bathtub
8	door	16	curtain	39	garbagebin

Small tabletop objects

Class	Semantic	Class	Semantic	Class	Semantic	Class	Semantic
41	bag	54	laptop	67	chessboard	80	mirror
42	bottle	55	microphone	68	coffee_machine	81	notebook
43	bowl	56	microwave	69	comb	82	pencil
44	camera	57	mug	70	cutting_board	83	plant
45	can	58	printer	71	dishes	84	plate
46	cap	59	remote_control	72	doll	85	radio
47	clock	60	phone	73	eraser	86	ruler
48	keyboard	61	alarm	74	eye_glasses	87	saucepan
49	display	62	book	75	file_box	88	spoon
50	earphone	63	cake	76	fork	89	tea_pot
51	jar	64	calculator	77	fruit	90	toaster
52	knife	65	candle	78	globe	92	vase
53	lamp	66	charger	79	hat	93	vegetables

Contact

If you have any questions, please contact Mutian Xu ([email protected]) or Pei Chen ([email protected]).

News

The dataset challenge will be released soon, via a complete website!

to-scene's People

Contributors

Stargazers

Watchers

Forkers

liuxinren456852 bbbbubble jlqzzz peterzs ryan1288

to-scene's Issues

When will you release all the datasets variants

Question for data representation(type)

I think the TO-scene presented in the paper constructed mesh representations.
In github, but dataset is constructed by point cloud.
Is there any plan to release mesh dataset?

Thanks

The accuracy of the TO-Real test set is poor

  I used the pointnet++ model to train the dataset, and tested the val dataset very well. However, when testing the To-real dataset, the accuracy was very low. Can you tell me why ?
  thanks!

Why is there no labels in test set

Recently, when reproducing this paper, I found that there are no labels in the test set , so I would like to ask about this situation.

unable to find To-real test dataset in readme.md

Can you tell me where it is?

thank you !

TO-Real broken download link

Thank you for making this dataset available!
Could you please update the TO-Real download link?

run error

Hello, I followed the step of 3d object detection, but when I am running python main.py --mode train --config ./configs/train_heatmap.yaml, I encountered follow problem:

Loading configuration
{'resume': False, 'method': 'heatmap', 'exp_name': 'TO-crowrd_heatmap_1124', 'device': {'use_gpu': True, 'gpu_ids': '3'}, 'data': {'dataset': 'TOS_desk', 'data_dir': './data/TO-crowd-wHM', 'use_color': False, 'use_height': True, 'use_aug': True, 'batch_size': 8, 'num_workers': 8, 'ap_iou_thresh': 0.25, 'npoints': 40000}, 'model': {'input_feature_dim': 1}, 'optimizer': {'type': 'Adam', 'lr': 0.0001, 'beta1': 0.9, 'beta2': 0.999, 'eps': None, 'weight_decay': None}, 'scheduler': {'type': 'MultiStepLR', 'milestone': [50, 80], 'gamma': 0.2}, 'other': {'nepoch': 100, 'model_save_interval': 1, 'model_save_dir': './checkpoints', 'dump_result': True, 'dump_interval': 1000, 'test_interval': 10, 'log_interval': 100}, 'log': {'path': './checkpoints/TO-crowrd_heatmap_1124'}, 'config': './configs/train_heatmap.yaml', 'mode': 'train'}
Data save path: ./checkpoints/TO-crowrd_heatmap_1124
Loading device settings.
CPU mode is on.
Loading dataset.
Traceback (most recent call last):
File "/home/ray/workspace/HRI_Project/TO-Scene/obj_det/main.py", line 25, in
train.run(cfg)
File "/home/ray/workspace/HRI_Project/TO-Scene/obj_det/train.py", line 15, in run
train_loader = get_dataloader(cfg.config, mode='train')
File "/home/ray/workspace/HRI_Project/TO-Scene/obj_det/utils/train_test_utils.py", line 297, in get_dataloader
from dataset.TOS_desk_dataset import TOS_Desk_Dataloader
File "/home/ray/workspace/HRI_Project/TO-Scene/obj_det/dataset/TOS_desk_dataset.py", line 7, in
DC=DOS_desk_config()
File "/home/ray/workspace/HRI_Project/TO-Scene/obj_det/data/model_utils_DOS.py", line 32, in init
self.mean_size_arr = np.load(os.path.join('./data/doscannet_means_desklevel-axisalign-5e6.npz'))['arr_0']
File "/home/ray/anaconda3/envs/pytorch/lib/python3.9/site-packages/numpy/lib/npyio.py", line 405, in load
fid = stack.enter_context(open(os_fspath(file), "rb"))
FileNotFoundError: [Errno 2] No such file or directory: './data/doscannet_means_desklevel-axisalign-5e6.npz'

It seems that it can't find file doscannet_means_desklevel-axisalign-5e6.npz, however I don't know where the file is.
Could you please tell me how to deal with this?
Thanks very much.

The model training effect is poor

  I'm sorry to bother you, I used the source code which provided for training. At the 50th round of training, the loss value was still large and had no convergence trend, and the accuracy was also very low. Can you tell me why? Alternatively, could you share your checkpoint
  Thanks!
  ![screenshot](https://github.com/GAP-LAB-CUHK-SZ/TO-Scene/assets/115631053/9b94c836-073c-4ecb-b1ff-5b3ebce97147)