Issue with mAP function about nuscenes-devkit HOT 4 CLOSED

This is an interesting corner case.

How did this happen that there are no gt labels but there are ones that are predicted?

I can modify the code for the metric so that it would throw an exception if there are classes that are not in ground truth.

from nuscenes-devkit.

We may have such a situation if we have an unstratified validation set, like in this Kaggle kernel: https://www.kaggle.com/gzuidhof/reference-model
there's no ground truth box for emergency_vehicle in the validation set, and because of which we get a divide-by-zero error in mAP evaluation code.

from nuscenes-devkit.

I would say the this is the issue of the data preparation. We should have all the classes that we want to evaluate in the validation file.

The overall metric should be calculated over all classes. Number of classes for metric calculation is defined by the number of classes in the ground truth file.

=> if we have fewer classes there => mAP will be misleading.

If someone wants to have a metric for a subset of classes, they should probably call

            recalls, precisions, average_precision = recall_precision(
                gt_by_class_name[class_name], pred_by_class_name[class_name], iou_threshold
            )

for the classes of interest.

from nuscenes-devkit.

Makes sense.

from nuscenes-devkit.