This repo implements the paper FUTR3D: A Unified Sensor Fusion Framework for 3D Detection. Paper - project page

We built our implementation upon MMdetection3D 0.17.3. The major part of the code is in the directory plugin/futr3d.

1. mmcv-full>=1.3.8, <=1.4.0
2. mmdet>=2.14.0, <=3.0.0
3. mmseg>=0.14.1, <=1.0.0
4. nuscenes-devkit

For cameras with Radar setting, you should generate a meta file or say .pkl file including Radar infos.

python3 tools/data_converter/nuscenes_converter_radar.py

For others, please follow the mmdet3d to process the data. https://mmdetection3d.readthedocs.io/en/stable/datasets/nuscenes_det.html

For example, to train FUTR3D with LiDAR only on 8 GPUs, please use

bash tools/dist_train.sh plugin/futr3d/configs/lidar_only/01voxel_q6_step_38e.py 8

For LiDAR-Cam and Cam-Radar version, we need pre-trained model.

The Cam-Radar uses DETR3D model as pre-trained model, please check DETR3D.

The LiDAR-Cam uses fused LiDAR-only and Cam-only model as pre-trained model. You can use

python tools/fuse_model.py --img <cam checkpoint path> --lidar <lidar checkpoint path> --out <out model path>

to fuse cam-only and lidar-only models.

For example, to evalaute FUTR3D with LiDAR-cam on 8 GPUs, please use

bash tools/dist_train.sh plugin/futr3d/configs/lidar_cam/res101_01voxel_step_3e.py ../lidar_cam.pth 8 --eval bbox

The camera-only version of FUTR3D is the same as DETR3D. Please check DETR3D for detail implementation.

For the implementation, we rely heavily on MMCV, MMDetection, MMDetection3D, and DETR3D

1. DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries
2. MUTR3D: A Multi-camera Tracking Framework via 3D-to-2D Queries
3. For more projects on Autonomous Driving, check out our Visual-Centric Autonomous Driving (VCAD) project page webpage

@article{chen2022futr3d,
  title={FUTR3D: A Unified Sensor Fusion Framework for 3D Detection},
  author={Chen, Xuanyao and Zhang, Tianyuan and Wang, Yue and Wang, Yilun and Zhao, Hang},
  journal={arXiv preprint arXiv:2203.10642},
  year={2022}
}

Contact: Xuanyao Chen at: [email protected] or [email protected]