orangesodahub / crlfnet Goto Github PK

View Code? Open in Web Editor NEW

94.0 2.0 24.0 1.56 GB

Camera-Radar-Lidar Fusion detection net based on ROS, YOLO, OpenPCDet.

Home Page: https://crlf-net.vercel.app

License: MIT License

CMake 1.68% C++ 13.19% Python 59.82% Shell 0.30% Dockerfile 0.20% Cuda 5.91% C 0.23% HTML 1.62% JavaScript 17.05%

fusion-detection openpcdet ros yolo

crlfnet's Introduction

CRLFnet

The source code of the CRLFnet.

INSTALL & BUILD

Env: Ubuntu20.04 + ROS(Noetic) + Python3.x

If using Google-colab, there is a recommanded environment: CUDA10.2+PyTorch1.6.
Refer to INSTALL.md for the installation of OpenPCDet.
Install ros_numpy package mannually: [Source code][Install]

Absolute paths may need your mind:

file path	Line(s)
src/camera_info/get_cam_info.cpp	26,64,102,140,178,216,254,292,330,368,
src/LidCamFusion/OpenPCDet/tools/cfgs/custom_models/pointrcnn.yaml	4,5
src/LidCamFusion/OpenPCDet/tools/cfgs/custom_models/pv_rcnn.yaml	5,6

Docker

Build project from Dockerfile:

docker build -t [name]:tag /docker/

or pull image directly:

docker pull gzzyyxy/crlfnet:yxy

Launch the Site

This needs ROS to be installed.

    cd /ROOT
    
    # launch the site
    roslaunch site_model spwan.launch
    
    # launch the vehicles (optional)
    woslaunch pkg racecar.launch

Rad-Cam Fusion

This part integrates the Kalman-Filter to real-time radar data.

Necessary Configurations on GPU and model data

Set use_cuda to True in src/site_model/config/config.yaml to use GPU.
Download yolo_weights.pth from jbox, and move to src/site_model/src/utils/yolo/model_data.

Run The Rad-Cam Fusion Model

The steps to run the radar-camera fusion is listed as follows.

For the last command, an optional parameter --save or -s is available if you need to save the track of vehicles as images. The --mode or -m parameter has three options, which are normal, off-yolo and from-save. The off-yolo and from-save modes enable the user to run YOLO seprately to simulate a higher FPS.

    #--- AFTER THE SITE LAUNCHED ---#
    # run the radar message filter
    rosrun site_model radar_listener.py
    
    # run the rad-cam fusion program
    cd src/site_model
    python -m src.RadCamFusion.fusion [-m MODE] [-s]

Camera Calibration

The calibration parameters are needed in related camera-data transformation. Once the physical models are modified, update the camera calibration parameters:

    #--- AFTER THE SITE LAUNCHED ---#
    # get physical parameters of cameras
    rosrun site_model get_cam_info

    # generate calibration formula according to parameters of cameras
    python src/site_model/src/utils/generate_calib.py

Lid-Cam Fusion

This part integrates OpenPCDet to real-time lidar object detection, refer to CustomDataset.md to find how to proceed with self-product dataset using only raw lidar data.

Config Files

Configurations for model and dataset need to be specified:

Model Configs tools/cfgs/custom_models/XXX.yaml
Dataset Configs tools/cfgs/dataset_configs/custom_dataset.yaml

Now pointrcnn.yaml and pv_rcnn.yaml are supported.

Datasets

Create dataset infos before training:

    cd OpenPCDet/
    python -m pcdet.datasets.custom.custom_dataset create_custom_infos tools/cfgs/dataset_configs/custom_dataset.yaml

File custom_infos_train.pkl, custom_dbinfos_train.pkl and custom_infos_test.pkl will be saved to data/custom.

Train

Specify the model using YAML files defined above.

    cd tools/
    python train.py --cfg_file path/to/config/file/

For example, if using PV_RCNN for training:

    cd tools/
    python train.py --cfg_file cfgs/custom_models/pv_rcnn.yaml --batch_size 2 --workers 4 --epochs 80

Pretrained Model

Download pretrained model through these links:

model	time cost	URL
PointRCNN	~3h	Google drive / Jbox
PV_RCNN	~6h	Google drive / Jbox

Predict (Local)

Prediction on local dataset help to check the result of trainin. Prepare the input properly.

python pred.py --cfg_file path/to/config/file/ --ckpt path/to/checkpoint/ --data_path path/to/dataset/

For example:

python pred.py --cfg_file cfgs/custom_models/pv_rcnn.yaml --ckpt ../output/custom_models/pv_rcnn/default/ckpt/checkpoint_epoch_80.pth --data_path ../data/custom/testing/velodyne/

Visualize the results in rviz, white boxes represents the vehicles.

Lid-Cam Fusion

Follow these steps for only lidar-camera fusion. Some of them need different bash terminals. For the last command, additional parameter --save_result is required if need to save the results of fusion in the form of image.

    #--- AFTER THE SITE LAUNCHED --#
    # cameras around lidars start working
    python src/site_model/src/LidCamFusion/camera_listener.py 

    # lidars start working
    python src/site_model/src/LidCamFusion/pointcloud_listener.py

    # combine all the point clouds and fix their coords
    rosrun site_model pointcloud_combiner

    # start camera-lidar fusion
    cd src/site_model/
    python -m src.LidCamFusion.fusion [--config] [--eval] [--re] [--disp] [--printl] [--printm]

TODO...

Issues

Some problems may occurred during debugging.

Confused: set the batch_size=1 and still out of memory: open-mmlab/OpenPCDet#140
段错误(核心已转储) when run dem.py: open-mmlab/OpenPCDet#846
N > 0 assert faild. CUDA kernel launch blocks must be positive, but got N= 0 when training: open-mmlab/OpenPCDet#945
raise NotImplementedError, NaN or Inf found in input tensor when training: open-mmlab/OpenPCDet#280
fix recall calculation bug for empty scene: open-mmlab/OpenPCDet#908
installation Error " fatal error: THC/THC.h: No such file or directory #include <THC/THC.h> ": open-mmlab/OpenPCDet#1014
...
Welcome to report more issues!

Star History

## Star History

crlfnet's People

Contributors

Stargazers

Watchers

crlfnet's Issues

Error using custom data while training in OpenPCDet

I am using a custom data,
my pointcloud details(from labelcloud) are

Number of Points: 13441
Number of Colors: 13441
Point Cloud Center: [ 5.46 -0.05 -1.04]
Point Cloud Minimums: [ 0. -8.05 -2.02]
Point Cloud Maximums: [14.69 8.32 -0.08]
Initial Translation: [ -5.46 0.05 -21.04]

i have created labels according to given format

example
6.4 -0.0 -1.25 0.85 0.75 1.35 -0.38 Pedestrian

currently i have only 1 class pedestrian

i made following changes in
custom_dataset.yaml

POINT_CLOUD_RANGE: [0.0, -8.05, -2.02, 14.69, 8.32, -0.08]

MAP_CLASS_TO_KITTI: {
'Pedestrian': 'Pedestrian',
}

POINT_FEATURE_ENCODING: {
encoding_type: absolute_coordinates_encoding,
used_feature_list: ['x', 'y', 'z', 'intensity'],
src_feature_list: ['x', 'y', 'z', 'intensity'],
}

PREPARE: {
filter_by_min_points: ['Pedestrian:5'],
}

SAMPLE_GROUPS: ['Pedestrian:15']
VOXEL_SIZE: [0.1, 0.1, 0.15]

After making changes i ran
python -m pcdet.datasets.custom.custom_dataset create_custom_infos tools/cfgs/dataset_configs/custom_dataset.yaml

now i want to implement PartA2_free model

so i made changes in PartA2_free.yaml

CLASS_NAMES: ['Robot']

DATA_CONFIG:
BASE_CONFIG: cfgs/dataset_configs/custom_dataset.yaml

Finally im hitting
python3 train.py --cfg_file /home/robot/OpenPCDet/tools/cfgs/kitti_models/PartA2_free.yaml

But i am getting error:

Traceback (most recent call last):
File "train.py", line 230, in
main()
File "train.py", line 175, in main
train_model(
File "/home/robot/OpenPCDet/tools/train_utils/train_utils.py", line 180, in train_model
accumulated_iter = train_one_epoch(
File "/home/robot/OpenPCDet/tools/train_utils/train_utils.py", line 33, in train_one_epoch
batch = next(dataloader_iter)
File "/home/robot/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 628, in next
data = self._next_data()
File "/home/robot/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1333, in _next_data
return self._process_data(data)
File "/home/robot/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1359, in _process_data
data.reraise()
File "/home/robot/.local/lib/python3.8/site-packages/torch/_utils.py", line 543, in reraise
raise exception
ValueError: Caught ValueError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/home/robot/.local/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 302, in _worker_loop
data = fetcher.fetch(index)
File "/home/robot/.local/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 58, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/robot/.local/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 58, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/robot/OpenPCDet/tools/../pcdet/datasets/custom/custom_dataset.py", line 109, in getitem
data_dict = self.prepare_data(data_dict=input_dict)
File "/home/robot/OpenPCDet/tools/../pcdet/datasets/dataset.py", line 185, in prepare_data
data_dict = self.data_augmentor.forward(
File "/home/robot/OpenPCDet/tools/../pcdet/datasets/augmentor/data_augmentor.py", line 302, in forward
data_dict = cur_augmentor(data_dict=data_dict)
File "/home/robot/OpenPCDet/tools/../pcdet/datasets/augmentor/database_sampler.py", line 497, in call
data_dict = self.add_sampled_boxes_to_scene(
File "/home/robot/OpenPCDet/tools/../pcdet/datasets/augmentor/database_sampler.py", line 398, in add_sampled_boxes_to_scene
obj_points = np.fromfile(str(file_path), dtype=np.float64).reshape(-1, self.sampler_cfg.NUM_POINT_FEATURES)
ValueError: cannot reshape array of size 246 into shape (4)

Now, is there something which i am missing can you help ?

error occured during training custom dataset

Hi，thanks for your great job.
Recently，I have generated my custom dataset‘s pkl file successfully，but when I tried to use "python /root/OpenPCDet/tools/train.py --cfg_file /root/OpenPCDet/tools/cfgs/custom_models/pointpillar.yaml --batch_size=2 --epochs=40" to train my dataset, it occured error like "EasyDict has not attribute to "ANCHOR_GENERATOR_CONFIG" "
it bothered me a few days.
The details are in the pictures.

How to add evaluation?

Thanks a lot!
I find "#no evaluation" in custom_dataset.py in line 357. That means this is a no evaluation version.And I want to know how to fix it to add the evaluation?

some question about angle in custom dataset

Sorry to bother you, I am trying to complete 3d object detection under my dataset under OpenPCDet. And I found some relative links to this repository. I want to know if the label file processed by custom_dataset.py become information in the OpenPCDet coordinate. For example, I label a car going straight ahead whose rotation is 90 in labelcloud GUI. And it will export a txt label file describing its rotation is 0(kitti-format). But the py file says
gt_boxes_lidar = np.concatenate([loc_lidar, l, w, h, (np.pi / 2 - rots[..., np.newaxis])], axis=1) the box liadr 's angle will be 90.
But it's rotation is 0 in the OpenPCDet coordinate. It makes me confused.

OpenPCDet training dataset

Excuse me, if the annotation tool is "point_cloud_annotation_tool", how should the "get_calib" method be written?

custom_dataset.py问题

Hello，In your file：CRLFnet/src/site_model/src/LidCamFusion/OpenPCDet/pcdet/datasets/custom/custom_dataset.py，it seems exist a problem:
you replace "pred_boxes_camera = box_utils.boxes3d_lidar_to_kitti_camera(pred_boxes,calib)"with
"pred_boxes_camera = box_utils.boxes3d_lidar_to_kitti_camera[pred_boxes]",but boxes3d_lidar_to_kitti_camera is a
function,can it work?

Label dataset with CloudCompare

I found that the data labeled with cloudcompare is different from labelcloud.

I used CloudCompare to directly get the length, width and height of the 3D box+geometric center (x, y, z). However, the geometric center of labelcloud undergoes a series of transformations, and the new geometric center(x1, y1, z1) after transformation is saved. Can my data run your program directly?

test.py

How to use test.py？When I test it, it will have the following problems

2022-11-04 19:53:07,216 INFO Start evaluation cfgs/custom_models/pointrcnn_person(default)
2022-11-04 19:53:07,216 INFO Loading Custom dataset.
2022-11-04 19:53:07,217 INFO Total samples for CUSTOM dataset: 100
2022-11-04 19:53:07,218 INFO ==> Loading parameters from checkpoint /home/qiu/OpenPCDet/output/cfgs/custom_models/pointrcnn_person/default/ckpt/checkpoint_epoch_10.pth to GPU
2022-11-04 19:53:07,272 INFO ==> Checkpoint trained from version: pcdet+0.5.2+0000000
2022-11-04 19:53:07,284 INFO ==> Done (loaded 309/309)
2022-11-04 19:53:07,285 INFO *************** EPOCH 10 EVALUATION *****************
eval: 100%|███| 100/100 [00:12<00:00, 7.93it/s, recall_0.3=(0, 0) / 0]
2022-11-04 19:53:19,902 INFO *************** Performance of EPOCH 10 *****************
2022-11-04 19:53:19,902 INFO Generate label finished(sec_per_example: 0.1262 second).
2022-11-04 19:53:19,902 INFO recall_roi_0.3: 0.000000
2022-11-04 19:53:19,902 INFO recall_rcnn_0.3: 0.000000
2022-11-04 19:53:19,902 INFO recall_roi_0.5: 0.000000
2022-11-04 19:53:19,902 INFO recall_rcnn_0.5: 0.000000
2022-11-04 19:53:19,903 INFO recall_roi_0.7: 0.000000
2022-11-04 19:53:19,903 INFO recall_rcnn_0.7: 0.000000
2022-11-04 19:53:19,903 INFO Average predicted number of objects(100 samples): 5.950
Traceback (most recent call last):
File "tools/train.py", line 207, in
main()
File "tools/train.py", line 200, in main
dist_test=dist_train
File "/home/qiu/OpenPCDet/tools/test.py", line 122, in repeat_eval_ckpt
result_dir=cur_result_dir, save_to_file=args.save_to_file
File "/home/qiu/OpenPCDet/tools/eval_utils/eval_utils.py", line 110, in eval_one_epoch
result_str, result_dict = dataset.evaluation(
File "/home/qiu/anaconda3/envs/openDet2/lib/python3.7/site-packages/torch/utils/data/dataset.py", line 83, in getattr
raise AttributeError
AttributeError

orangesodahub / crlfnet Goto Github PK

crlfnet's Introduction

CRLFnet

INSTALL & BUILD

Docker

Launch the Site

Rad-Cam Fusion

Necessary Configurations on GPU and model data

Run The Rad-Cam Fusion Model

Camera Calibration

Lid-Cam Fusion

Config Files

Datasets

Train

Pretrained Model

Predict (Local)

Lid-Cam Fusion

TODO...

Issues

crlfnet's People

Contributors

Stargazers

Watchers

Forkers

crlfnet's Issues

Recommend Projects

Recommend Topics

Recommend Org