jonathonluiten / trackeval Goto Github PK

Thank you very much for the code and data packages, I was wondering if the Test data for the MOT Benchmarks are available ? And how to adapt the code to run only one sequence of one benchmark of one tracker ?

Hello,

thank you for your great work and contribution to MOT community.

As stated in the title, evaluator parses the name of the sequences by using csv library. Is there a valid assumption that there is no comma in the name of the sequence? If it is stated somewhere in the README or project, I am sorry for the hassle.

Thanks,

Hi,

Thanks for the amazing repo. It helps a lot. I have been using the code here for 2D box tracking evaluation in KITTI format dataset. However, it seems there is no implmentation for 3D box tracking in pointcloud (Although such data was given in KITTI). Did I miss something?

Hi. Thank you for the great work.
Do you have a plan to provide official HOTA results when we submit to the MOT Challenge Benchmark?

Hi, Thanks for the code, it works like a charm for mono-camera.

I would like to compute HOTA for multi camera dataset (let's say we have 2 seqs corresponding to camera 1 and camera 2 sequence). What should I do?

I had some ideas like :

• each camera sequence as an independant sequence (like 2 mono camera sequence)
• do one sequence which is the concatenation of both sequence ?
• Simultaneously sequence 1 and sequence 2 on the same .txt. Like 1 ID appears on two frames.

I think option 3 would definetely messus up with HOTA computation.
I think that option 1 and 2 would work but would it take in consideration that one ID is the same per frame accross all cameras. I mean if I have an ID_1 on frame 1 on both camera then it takes in consideration it is ID_1 on both camera and not ID_1 and ID_2 for exemple.
We could have a situation like : ID_1 and ID_2 is perfectly tracked in each camera sequence but my tracking advice would still label two differents IDs for the same ID across camera ...

That would be to use HOTA for Person-Reidentification task as Market1501 dataset and so on. In the article it says that HOTA can be easily extended for multi camera, I am curious about this extension and I am sorry if the answer is trivial.

I hope this is clearn and thanks again for your amazing work.

Hi,

Thank you for your great work with the HOTA metrics.
I've a related issue but this is not about the difference between 9 and 10 values per line (even though I tried to remove one to see if it helps, but it does not).

I tried your script with a GT file from the AICity challenge 2021 and compared it with a pred file they provide, both following the MOT format (attached to this message), but for some unknown reason, the script does not read the GT file, same as if it was empty:

Count: DeepSORT-SSD512-pedestrian  Dets      GT_Dets   IDs       GT_IDs
s01-c003                           14151     0         219       0
COMBINED                           14151     0         219       0

I tried to change some value, indexed by obj_id instead of frame_id, remove one '-1' at the end of each line, checked the encoding and also checked the entire hierarchy of 'trackers' and 'gt' folders.

Finally, I copied a GT file from a MOT sequence (MOT16-train\MOT16-02) and pasted it as it was my own GT file, even if it doesn't match at all... and it reads the GT file correctly:

Count: DeepSORT-SSD512-pedestrian  Dets      GT_Dets   IDs       GT_IDs
s01-c003                           14151     17833     219       54
COMBINED                           14151     17833     219       54

Here's the original GT file of the sequence I want to test:
gt.txt

Here's one of the prediction file:
s01-c003.txt

Thank you in advance for your help.

Originally posted by @JonathanSamelson in #23 (comment)

Hi,

Thank you for your great work. Its very helpful

I have evaluated People detection and tracking solution based on PeopleNet + KLT (nvtracker) using TrackEval on MOT Challenge in CLEAR metrics for data MOT20 training

Tracker outputs
MOT20-01.txt
MOT20-02.txt
MOT20-03.txt
MOT20-05.txt

Metric results
pedestrian_detailed.csv
pedestrian_summary.txt

As can be seen in the attached results. I don’t understand why i have sometimes ‘-’ or ‘+’ in MOTA There are also 0 values of MOTP, which signals an error in calculations (?)

nuScenes:https://www.nuscenes.org/tracking/?externalData=all&mapData=all&modalities=Any
waymo:https://waymo.com/open/challenges/2020/3d-tracking/

I want to annotate my custom data to make a similar format to the MOT data format.
Are there any open source/free annotation tools?

Thanks

Hi, I found a bug here:

if bboxes1 and bboxes2 both have whole numbers, then ioas will have np.int dtype, and so at the end all values will be zeros (or 1 if a particular box is 100% inside the other box)

To fix this, let's do this:
ioas = np.zeros_like(intersection).astype(np.float)
OR (maybe)
convert bboxes1 and bboxes2 to float at the very beginning.

The paper mentions OHOTA for online tracking. Is it implemented in TrackEval? If so, is there a flag or command-line option for computing this instead of HOTA?

Hi,
Can I evalutate MODA metric using this code?
MODA (Multi-Object Detection Accuracy): the metric for object detection. (MOT17Det, MOT20Det)

Hello,

I have a question regarding the optimisation of HOTA for matchings with the Hungarian algorithm.
Aside from MOT performance, I would like to use TrackEval to obtain raw framewise detection scores (DetRe and DetPr). I do read that matching is built "such that as a first objective the number of TPs is maximised" but I would also like to remove the influence of the association mean to obtain a true "track independant" detection-score before running TrackEval the normal way.

Could you point me to the code modifications I can make for that purpose ? In hota.py, I found the following line but I am not sure what to change:
score_mat = global_alignment_score[gt_ids_t[:, np.newaxis], tracker_ids_t[np.newaxis, :]] * similarity

Thank you very much iin advance !

Mathis

I received the following question via email. I thought I would share the question and my answer here as an issue in case anyone is wondering the same. - Jono

hi, I am reading your paper of hota tracking metric. I have one question about how AssA is calculated.

As the pic below shown, A(c) is ass_a in your code. The paper says AssA equals the sum of all A(c) / TP, but an extra matches_count is timed to every A(c) before sum, which is different from what in paper.

Could you tell me why? thanks.

Hi, I thought I found a bug in TrackEval/trackeval/metrics/identity.py.

Current Version
The fn_mat[:num_gt_ids, :num_tracker_ids] and fp_mat[:num_gt_ids, :num_tracker_ids] entries are not right.

Currently,

fn_mat[:num_gt_ids, :num_tracker_ids] = number_of_gt_ids - potential_matches_count
fp_mat[:num_gt_ids, :num_tracker_ids] = number_of_tracker_ids - potential_matches_count

However, this is not what the original paper explains (click here). This does not handle the case when there are no potential matches.

Two regular nodes are connected with an edge e ∈ E if their trajectories overlap in time.

What I think It Should Be

There should be an if-else statement which checks whether trajectories overlap in time.

# Assume 1e10 is infinity
fn_mat[:num_gt_ids, :num_tracker_ids] = (number_of_gt_ids - potential_matches_count) if (potential_matches_count > 0) else 1e10
fp_mat[:num_gt_ids, :num_tracker_ids] = (number_of_tracker_ids - potential_matches_count) if (potential_matches_count > 0) else 1e10

I have been quite handwavy with the code notation, but hopefully it still conveys the meaning.

Is my understanding right?

First, thanks for your great job! I have a question for kitti mots datasets, I don't the content of the directory structure, such as the name, content and position of seqmap file.

Hi guys,
Currently using this code to evaluate my MOT system and really liking the results!
I have a dataset where a lot of occlusions happen which have been labelled.

I was wondering if there is a system in place to correctly handle that information in the Ground-Truth.
Any advice would be wonderful!
Regards,
Max

Hi!

Within the context of PR #36, I ran every scripts in scripts folder. I found out some minor errors in comment to run the script from the data folder. What I understood is those lines at the beginning of each script are indications how to run evaluation scripts for each dataset. So I believe it would be better for new users of trackeval if these errors could be fixed.

• run_bdd.py:
  Line 5: run_bdd.py --USE_PARALLEL False --METRICS Hota --TRACKERS_TO_EVAL qdtrack -> run_bdd.py --USE_PARALLEL False --METRICS HOTA --TRACKERS_TO_EVAL qdtrack.
  • Hota is mistyped, must be HOTA.
• run_kitty.py:
  Line 5: run_kitti.py --USE_PARALLEL False --METRICS Hota --TRACKERS_TO_EVAL CIWT -> run_kitti.py --USE_PARALLEL False --METRICS HOTA --TRACKERS_TO_EVAL CIWT.
  • Hota is mistyped, must be HOTA.
• run_kitty_mots.py:
  Line 5: run_kitti_mots.py --USE_PARALLEL False --METRICS HOTA --TRACKERS_TO_EVAL trackrcnn -> run_kitti_mots.py --USE_PARALLEL False --METRICS HOTA --TRACKERS_TO_EVAL track_rcnn --GT_FOLDER data/gt/kitti/kitti_mots_train.
  • --GT_FOLDER must match data folder.
  • --TRACKERS_TO_EVAL has a typo.
• run_mot_challenge.py:
  Line 5: run_mot_challenge.py --USE_PARALLEL False --METRICS Hota --TRACKERS_TO_EVAL Lif_T -> run_mot_challenge.py --USE_PARALLEL False --METRICS HOTA --TRACKERS_TO_EVAL MPNTrack.
  • --TRACKERS_TO_EVAL Lif_T doesn't match tracker in data folder.
  • Hota is mistyped, must be HOTA.
• run_mots_challenge.py:
  Line 4: run_mots.py --USE_PARALLEL False --METRICS Hota --TRACKERS_TO_EVAL TrackRCNN -> run_mots_challenge.py --USE_PARALLEL False --METRICS HOTA --TRACKERS_TO_EVAL track_rcnn.
  • --TRACKERS_TO_EVAL Lif_T doesn't match tracker in data folder.
  • Hota is mistyped, must be HOTA.
  • Python script name is mistyped.

Have a great day!

Mathias.

Hi!

I'm currently trying to use this great repository to evaluate the performance of my own tracker on a dataset i made myself.

The results from the tracker is in this format (which i believe is correct):
A snippet from my results.txt file is shown below.

2,1,0,227,107,129,-1,-1,-1,-1
2,2,2,0,631,359,-1,-1,-1,-1
2,3,367,211,75,28,-1,-1,-1,-1
3,1,0,226,108,130,-1,-1,-1,-1
3,2,4,0,630,358,-1,-1,-1,-1
3,3,365,211,75,28,-1,-1,-1,-1

As far as i understand the ground truth file is supposed to have the same format (which is has).

However, i'm having troubles with running the run_mot_challenge.py script:
raise TrackEvalException('ini file does not exist: ' + seq + '/' + os.path.basename(ini_file))
trackeval.utils.TrackEvalException: ini file does not exist: ShipSeq-01/seqinfo.ini

It seems like it can't locate the seqinfo.ini file even though it is there. It might be because im very confused about the seqmaps and seqinfo files and how i'm supposed to modify them to my own dataset. Do you have any ideas of what i should do?

Regards

Hi, Thank you for sharing your code, it is an amazing work!
Yet when I run the code, I got the following error:
Traceback (most recent call last): File ".\eval_code\Scripts\run_MOTChallenge.py", line 59, in <module> from eval_code.Datasets.MOTChallenge_Dataset import load_raw_MOTCha_seq, preproc ModuleNotFoundError: No module named 'eval_code'
May I ask the reason?

Hey,

First and foremost, thank you for this work to facilitate analysis of multiobject trackers. After having a look at the readme-files, code, and provided data I have a couple questions.
i) The RobMOTS data, train_data.zip, seems to contain detections, ground-truth, and the predictions made by the simplest possible tracker. The detections are missing on BDD100k and waymo. The ground-truth seems to be missing for waymo. Are the missing detections and ground-truth incoming?
ii) Are the corresponding val_data.zip and test_data.zip missing? Will at least the validation set be provided with ground-truth?
iii) I noticed that there are no corresponding images provided. For the RobMOTS challenge, are we expected to rely only on the detections? For tracker evaluation and analysis in general, do you plan to supply also matching images?
iv) On that note, it seems straightforward to download images myself and match them to the provided detections, as the detection files are named after the sequence names. However, how do I match the image frames to the time steps? Specifically, the YouTubeVIS detection times do not seem to correspond to the YouTubeVIS dataset times.
v) Both the TAO-dataset and BDD100k are utilized. Is this not problematic as TAO contains some sequences from BDD100k?

Best regards,
Joakim

I am trying to understand what format my tracker output should be:

in 1. its confusing for me, conf and what is det.txt and in 2. here conf value is -1.

my question, in my tracker output do i print conf as -1 ? and what does this statement mean? "The conf value contains the detection confidence in the det.txt files." i am aware that my detector can produce txt files with confidence value. so in the tracker output i don't print conf value and make it -1 and let my detector print conf value?

frame, id, bb_left, bb_top, bb_width, bb_height, conf, x, y, z
The conf value contains the detection confidence in the det.txt files.

Tracking with bounding boxes
(MOT15, MOT16, MOT17, MOT20)
1, 3, 794.27, 247.59, 71.245, 174.88, -1, -1, -1, -1
1, 6, 1648.1, 119.61, 66.504, 163.24, -1, -1, -1, -1
1, 8, 875.49, 399.98, 95.303, 233.93, -1, -1, -1, -1
...

Hi
I want to use one of the MOTChallenge videos randomly e.g. MOT17-11-SDP to test my tracker's performance. ( i use videos that are for training, so I can get the evaluation metrics like HOTA since the ground truth files are available). I test some trackers like ByteTrack on custom data and want to use TrackEval for evaluation. I'm sure that the output file of the ByteTrack is compatible with what TrackEval requires.
As I use their pretrained models and just test the model on MOTChallenge training videos, I expect to get results almost the same as what they said in their paper (MOTA = 80, IDF1 = 77.3, and HOTA=63.1). but after evaluation using this repository, I get strange results as shown in the following image.
I searched a lot for this problem but couldn't find any helpful advice. is there any step or preprocessing which I forgot to handle?? or the problem is related to the TrackEval??
thanks

It is easy to calculate hota metrics after detection/gt match. But it is quite confusing about matching rule.
From the paper, hota is calculated independently under different iou threshold for 2d bbox tracking. And the Eq15 is to do Hungarian algorithm on Amax add weighted iou.
But it is some kind of Amax multiply iou in this repo.
It is hard to understand why it's guaranteed to meet maximum hota on all possible detection/gt match, both on paper and code.
For a tiny example on iou 0.5 threshold:

data['gt_ids'] = [np.array([0])] * 9 + [np.array([0,1])]

data['tracker_ids'] = [np.array([0])] * 10

data['similarity_scores'] = [np.array([[1]])] * 9 + [np.array([0.49,0.51]).reshape(2,1)]

data['num_tracker_ids'] = 1

data['num_gt_ids'] = 2

In the example, the code will discard last frame detection under iou 0.5 threshold, DetA 0.75 AssA 0.818 HOTA 0.783
But just let last frame detection to match gt id 1: DetA 0.909 AssA 0.746 HOTA 0.8237

Just do linear_sum_assignment(-similarity) to reproduce.

Hello,

Is there an easy workaround to use this kit for a tracking method which only detects 2D points ?

From what I see, given bounding boxes coordinates it's easy to rewrite the function computing similarity scores to use Euclidean distance between the centers, but other than that it seems that I will have to specify "fake" bounding box width and heights if I want to comply with any of the dataset formats.

Thanks in advance.

Hello,

I have been experiencing difficulties to reproduce MOTChallenge scores of SORT tracker for MOT15 dataset. I used det.txt provided in git repo of the paper (frcnn detections). I have attached my results in .csv and .txt formats. Any help would be appreciated. Thanks.

pedestrian_detailed.csv
pedestrian_summary.txt

https://arxiv.org/pdf/1602.00763.pdf

Fatih

Hi, this is a great job!

However, I found some parameters not familiar for me, could you please explain what files they correspond to, like in the MOT dataset, thanks!

They are seqmap_file and seq_info in the code here.

i try to use run_mot_challenge.py tool to evaluate my tracker on my custom dataset.
Before evalutation, i have change default_dataset_config['CLASS_TO_EVAL'], self.valid_classes, self.class_name_to_class_id, distractor_class_name etc in mot_challenge_2d_box.py.
But the output MOTA is negative, and i dont know how to solve it. Can you help me？thks

Hey there,

I've been using your repo for quite a while for MOT challenge. However, I'm facing problems with MOTS challenge.

I've downloaded the data folder to run some simple examples.

When I run (which is a slight fix of what you've mentioned in the comments in run_mots_challenge.py)
python scripts/run_mots_challenge.py --USE_PARALLEL False --METRICS HOTA --TRACKERS_TO_EVAL track_rcnn

I'm getting the following error:

ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject

Hello, thanks for your work!

Can I run the evaluation having results stored as txt files? I need something like: HOTA = (results.txt, ground_truth.txt). Thanks in advance!

Hi Jonathon,

Great work! I was just wondering if there would be any support for evaluating MOTS when the ground truths are in MOTS PNG format?
I have a custom dataset I would like to evaluate with HOTA, but it is annotated in the MOTS PNG format just like KITTI MOTS, and I don't have the MOTS txt's handy with me because the annotation tool doesn't offer exports to that format. Unless there is a way to convert the MOTS PNG to MOTS txt, I don't know how to evaluate my dataset with HOTA.

Thanks a lot!

I created a clean virtual environment with python3, installed numpy and scipy. I also downloaded the data.zip and placed it in the main directory of this repository (same level as docs, scripts, tests, trackeval).
When I run the script from the main folder this happens:

./venv/bin/python3 scripts/run_kitti.py

Eval Config:
USE_PARALLEL : False
NUM_PARALLEL_CORES : 8
BREAK_ON_ERROR : True
RETURN_ON_ERROR : False
LOG_ON_ERROR : /home/ias/git/TrackEval/error_log.txt
PRINT_RESULTS : True
PRINT_ONLY_COMBINED : False
PRINT_CONFIG : True
TIME_PROGRESS : True
DISPLAY_LESS_PROGRESS : True
OUTPUT_SUMMARY : True
OUTPUT_EMPTY_CLASSES : True
OUTPUT_DETAILED : True
PLOT_CURVES : True

Kitti2DBox Config:
PRINT_CONFIG : True
GT_FOLDER : /home/ias/git/TrackEval/data/gt/kitti/kitti_2d_box_train
TRACKERS_FOLDER : /home/ias/git/TrackEval/data/trackers/kitti/kitti_2d_box_train/
OUTPUT_FOLDER : None
TRACKERS_TO_EVAL : None
CLASSES_TO_EVAL : ['car', 'pedestrian']
SPLIT_TO_EVAL : training
INPUT_AS_ZIP : False
TRACKER_SUB_FOLDER : data
OUTPUT_SUB_FOLDER :
TRACKER_DISPLAY_NAMES : None

Evaluating 1 tracker(s) on 21 sequence(s) for 2 class(es) on Kitti2DBox dataset using the following metrics: HOTA, CLEAR, Identity, Count

Evaluating CIWT

Tracker CIWT was unable to be evaluated.
Ground-truth data contains the following invalid timesteps in seq 0000: 0,
Traceback (most recent call last):
File "/home/ias/git/TrackEval/trackeval/eval.py", line 87, in evaluate
metric_names)
File "/home/ias/git/TrackEval/trackeval/_timing.py", line 16, in wrap
result = f(*args, **kw)
File "/home/ias/git/TrackEval/trackeval/eval.py", line 187, in eval_sequence
raw_data = dataset.get_raw_seq_data(tracker, seq)
File "/home/ias/git/TrackEval/trackeval/_timing.py", line 16, in wrap
result = f(*args, **kw)
File "/home/ias/git/TrackEval/trackeval/datasets/_base_dataset.py", line 87, in get_raw_seq_data
raw_gt_data = self._load_raw_file(tracker, seq, is_gt=True)
File "/home/ias/git/TrackEval/trackeval/datasets/kitti_2d_box.py", line 187, in _load_raw_file
[str(x) + ', ' for x in extra_time_keys]))
trackeval.utils.TrackEvalException: Ground-truth data contains the following invalid timesteps in seq 0000: 0,
Traceback (most recent call last):
File "scripts/run_kitti.py", line 86, in
evaluator.evaluate(dataset_list, metrics_list)
File "/home/ias/git/TrackEval/trackeval/_timing.py", line 16, in wrap
result = f(*args, **kw)
File "/home/ias/git/TrackEval/trackeval/eval.py", line 177, in evaluate
raise err
File "/home/ias/git/TrackEval/trackeval/eval.py", line 87, in evaluate
metric_names)
File "/home/ias/git/TrackEval/trackeval/_timing.py", line 16, in wrap
result = f(*args, **kw)
File "/home/ias/git/TrackEval/trackeval/eval.py", line 187, in eval_sequence
raw_data = dataset.get_raw_seq_data(tracker, seq)
File "/home/ias/git/TrackEval/trackeval/_timing.py", line 16, in wrap
result = f(*args, **kw)
File "/home/ias/git/TrackEval/trackeval/datasets/_base_dataset.py", line 87, in get_raw_seq_data
raw_gt_data = self._load_raw_file(tracker, seq, is_gt=True)
File "/home/ias/git/TrackEval/trackeval/datasets/kitti_2d_box.py", line 187, in _load_raw_file
[str(x) + ', ' for x in extra_time_keys]))
trackeval.utils.TrackEvalException: Ground-truth data contains the following invalid timesteps in seq 0000: 0,

Hi,

Thanks for the excellent work! I found HOTA had become one of the most popular metrics for MOT nowadays, everyone in the tracking community is using it.

I am recently trying to test trackers on the TAO dataset. When I run TrackEval and the official TAO evaluation script, I find their outputs of track mAP are inconsistent. I am not sure if this is due to my mistake, or if the two implementations of track mAP are actually different? Should I expect the two results to match perfectly?

By the way, I used the off-the-shelf results from AOA, and test on the val split. The output of the official eval code is around 29 mAP, while the output of this codebase is around 16 mAP.

I am using the structure mentioned in comment number #29 with following command to run

!python TrackEval/scripts/run_mot_challenge.py --DO_PREPROC False --BENCHMARK cam1 --SPLIT_TO_EVAL all --TRACKERS_TO_EVAL bytetrack

this is the error i get :

trackeval.utils.TrackEvalException: Ground-truth data contains the following invalid timesteps in seq seq-01: 430, , 431, , 432, , 433, , 434, , 435, , 436, , 437, , 438, , 439, , 440, , 441, , 442, , 443, , 444, , 445, , 446, , 447, , 448, , 449, , 450, , 451, , 452, , 453, , 454, , 455, , 456, , 457, , 458,

these numbers are the frame ids

my gt file looks like this
372,1,1224.45,414.41,55.549999999999955,71.69999999999999,1,1,1.0
373,1,1208.7,414.41,71.29999999999995,71.69999999999999,1,1,1.0
374,1,1178.3,414.41,101.70000000000005,71.69999999999999,1,1,1.0
375,1,1141.8,414.41,138.20000000000005,71.69999999999999,1,1,1.0
376,1,1102.9,402.26,177.0999999999999,71.69,1,1,1.0
377,1,1085.28,400.44,177.10000000000014,71.69,1,1,1.0
378,1,1067.66,398.61,177.0999999999999,71.69999999999999,1,1,1.0
379,1,1034.85,392.54,177.10000000000014,71.69,1,1,1.0
380,1,1003.25,377.95,177.0999999999999,71.69999999999999,1,1,1.0

my tracker out put looks like this
372,1,1002,138,25,23,1,1,1
372,2,367,219,87,41,1,1,1
372,3,683,283,25,24,1,1,1
372,4,1196,419,83,58,1,1,1
372,5,261,57,53,25,1,1,1
372,6,368,219,88,42,1,1,1
372,7,808,177,44,24,1,1,1
372,8,598,206,60,27,1,1,1
372,9,882,170,39,22,1,1,1

any help is appreciated :)

Hi Jonathon,

I tried use ground truth files as tracker prediction result to test the code. For most metrics, including HOTA, they are 100%, which is reasonable. But for locA score under HOTA_AUC section, it's over 99%, but not 100%.

The ground truth file I use is MOT17-04-DPM, and I remove all other classes, except pedestrain class as the tracker's prediction file. I use earlier version code before current version. Could you explain it, thanks!

In the README Quickly evaluate on supported benchmarks , the gt data (~150mb) is provided such as YouTube-2019

data/gt/youtube_vis/youtube_vis_train_sub_split/train_sub_split.json.

When I look into this gt file, I find the length of videos is 302, same as the YouTube-VIS 2019 validation set provided in CodaLab. And, after a detailed checking, I find they are two different video ID sets. So, Does that mean the data you provied is just part of training set (sample 302 videos from training 2238 videos) ? @JonathonLuiten

Also, what about the STEm_Seg tracker results?

data/trackers/youtube_vis/youtube_vis_train_sub_split/STEm_Seg/data/results.json

Does that mean the above prediction results just are inferenced by the default model provied in STEm_Seg repo or a new model trained by remaining set (2238-302=1936 videos) ?

Thanks for your contribution for Tracking research.

Hi, thanks for your amazing work.

But I got different MOTA score when using evaluation code in ByteTrack and TrackEval (72.5 vs. 73.275).
Do you know the reason of that? Or there are some parameters to set?

In the 48th line of test_all_quick.py, you writetest_data_loc = os.path.join(os.path.dirname(__file__), '..', 'data', 'tests', file_loc), but under the \data directory, there are no \tests but only \trackers, so an FileNotFoundError occurred.

Hi!

Within the context of PR #36 I found the following problem. I found out that I can't execute scripts/run_davis.py, I have the following ouput error:

Eval Config:
USE_PARALLEL         : False                         
NUM_PARALLEL_CORES   : 8                             
BREAK_ON_ERROR       : True                          
RETURN_ON_ERROR      : False                         
LOG_ON_ERROR         : /home/muffie/code/TrackEval-master/error_log.txt
PRINT_RESULTS        : True                          
PRINT_ONLY_COMBINED  : False                         
PRINT_CONFIG         : True                          
TIME_PROGRESS        : True                          
DISPLAY_LESS_PROGRESS : True                          
OUTPUT_SUMMARY       : True                          
OUTPUT_EMPTY_CLASSES : True                          
OUTPUT_DETAILED      : True                          
PLOT_CURVES          : False                         

DAVIS Config:
PRINT_CONFIG         : True                          
GT_FOLDER            : /home/muffie/code/TrackEval-master/data/gt/davis/davis_unsupervised_val/
TRACKERS_FOLDER      : /home/muffie/code/TrackEval-master/data/trackers/davis/davis_unsupervised_val/
OUTPUT_FOLDER        : None                          
TRACKERS_TO_EVAL     : ['ags']                       
SPLIT_TO_EVAL        : val                           
CLASSES_TO_EVAL      : ['general']                   
TRACKER_SUB_FOLDER   : data                          
OUTPUT_SUB_FOLDER    :                               
TRACKER_DISPLAY_NAMES : None                          
SEQMAP_FILE          : None                          
SEQ_INFO             : None                          
MAX_DETECTIONS       : 0                             

Evaluating 1 tracker(s) on 30 sequence(s) for 1 class(es) on DAVIS dataset using the following metrics: HOTA, Count


Evaluating ags

Tracker ags was unable to be evaluated.
numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject
Traceback (most recent call last):
  File "/home/muffie/code/TrackEval-master/trackeval/eval.py", line 86, in evaluate
    res[curr_seq] = eval_sequence(curr_seq, dataset, tracker, class_list, metrics_list,
  File "/home/muffie/code/TrackEval-master/trackeval/_timing.py", line 16, in wrap
    result = f(*args, **kw)
  File "/home/muffie/code/TrackEval-master/trackeval/eval.py", line 187, in eval_sequence
    raw_data = dataset.get_raw_seq_data(tracker, seq)
  File "/home/muffie/code/TrackEval-master/trackeval/_timing.py", line 16, in wrap
    result = f(*args, **kw)
  File "/home/muffie/code/TrackEval-master/trackeval/datasets/_base_dataset.py", line 87, in get_raw_seq_data
    raw_gt_data = self._load_raw_file(tracker, seq, is_gt=True)
  File "/home/muffie/code/TrackEval-master/trackeval/datasets/davis.py", line 123, in _load_raw_file
    from pycocotools import mask as mask_utils
  File "/home/muffie/.local/lib/python3.8/site-packages/pycocotools/mask.py", line 3, in <module>
    import pycocotools._mask as _mask
  File "pycocotools/_mask.pyx", line 1, in init pycocotools._mask
ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject
Traceback (most recent call last):
  File "scripts/run_davis.py", line 90, in <module>
    evaluator.evaluate(dataset_list, metrics_list)
  File "/home/muffie/code/TrackEval-master/trackeval/_timing.py", line 16, in wrap
    result = f(*args, **kw)
  File "/home/muffie/code/TrackEval-master/trackeval/eval.py", line 177, in evaluate
    raise err
  File "/home/muffie/code/TrackEval-master/trackeval/eval.py", line 86, in evaluate
    res[curr_seq] = eval_sequence(curr_seq, dataset, tracker, class_list, metrics_list,
  File "/home/muffie/code/TrackEval-master/trackeval/_timing.py", line 16, in wrap
    result = f(*args, **kw)
  File "/home/muffie/code/TrackEval-master/trackeval/eval.py", line 187, in eval_sequence
    raw_data = dataset.get_raw_seq_data(tracker, seq)
  File "/home/muffie/code/TrackEval-master/trackeval/_timing.py", line 16, in wrap
    result = f(*args, **kw)
  File "/home/muffie/code/TrackEval-master/trackeval/datasets/_base_dataset.py", line 87, in get_raw_seq_data
    raw_gt_data = self._load_raw_file(tracker, seq, is_gt=True)
  File "/home/muffie/code/TrackEval-master/trackeval/datasets/davis.py", line 123, in _load_raw_file
    from pycocotools import mask as mask_utils
  File "/home/muffie/.local/lib/python3.8/site-packages/pycocotools/mask.py", line 3, in <module>
    import pycocotools._mask as _mask
  File "pycocotools/_mask.pyx", line 1, in init pycocotools._mask
ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject

Steps to reproduce:

• git clone https://github.com/JonathonLuiten/TrackEval
• add data folder to root dir of the repo folder
• install local venv with depencies : numpy==1.18.1, scipy==1.4.1, pycocotools==2.0.2, matplotlib==3.2.1, Pillow==8.1.2 according to requirements.txt
• python scripts/run_davis.py --USE_PARALLEL False --METRICS HOTA --TRACKERS_TO_EVAL ags --PLOT_CURVES False

Also, DAVIS evaluation needs pycocotools which is not in minimal_requirements.txt. Also pycocotools module relies on matplotlib + pillow. According to this README, I think these depencies should be added to minimal requirements in order to run on DAVIS dataset as intended

Update: Same problem with KITTI MOTS, MOTS Challenge and Youtube VIS. They both needs pycocotools that seems a core package for segmentation evaluation. And it got the exact same error.

For some frames when there are no tracks (tracker_dets_t empty), _calculate_similarities calls either _calculate_box_ious or _calculate_mask_ious assuming both contains objects

@JonathonLuiten

Hey, hope all is fine. Well I am trying to evaluate my online deepsort tracker. The method uses a separate object detector and thus cannot use premade detections for feature extraction and tracking, but rather live input from a detector were my input video is the MOT16-02 Video. Therefore, some frames are missed and thus removed these frames from the Gt file not to account for missed frames, but the missed detections are not removed from the Gt.

The evaluation output values are so low such that the HOTA is 1.6, MOTA is negative 26.23 and IDF1/IDR/IDP/IDTP values are 0. Is this because I am not using the same pre-generated detections ? And is there another evaluation method that doesnt need any precomputed Detections ?

Hello Dear Authors,
Thank you for your effort in creating the dataset.

I am trying to use MOT17 data for my project and I was wondering if you have segmentation masks for the frames along with tracking/detection annotations?

Also, I see that MOTS has the segmentation masks but not sure if it has tracking and detection annotations?

Could you please clarify on both the fronts?
Thank you in advance.

Hello ,
Im getting following error with this command
!python scripts/run_mot_challenge.py --DO_PREPROC False --BENCHMARK MOT20-train

raise TrackEvalException('no seqmap found: ' + os.path.basename(seqmap_file))
trackeval.utils.TrackEvalException: no seqmap found: MOT20-train-train.txt

Looking forward to your respond

In the description file, it's written that the ground truth/ prediction files should contain 10 values per row for MOT sequences (15, 16, 17, 20): link.

However, in the data.zip mentioned in the README, GT files data/gt/mot_challenge/ MOT16-train, MOT17-train, MOT20-train contain only 9 values per row.

Is it an intended thing or can be considered as an issue?

The MOT challenge format description for tracking with bounding boxes on the MOT challenge website as well as here: https://github.com/JonathonLuiten/TrackEval/blob/master/docs/MOTChallenge-format.txt say that the columns are:

<frame>, <id>, <bb_left>, <bb_top>, <bb_width>, <bb_height>, <conf>, <x>, <y>, <z>
...
The world coordinates x,y,z are ignored for the 2D challenge and can be filled with -1.

However, putting -1 like the example in the column of ground truth data causes TrackEval to throw an exception. On inspection of the code, it seems that this column is actually being interpreted as object class (lines 249-250 in datasets/mot_challenge_2d_box.py):

               if time_data.shape[1] >= 8:
                    raw_data['classes'][t] = np.atleast_1d(time_data[:, 7]).astype(int)

I wonder if the MOT format changes for ground truth data, or if I am missing something.

Hi
I'm trying to evaluate tracking results on a custom dataset which has been converted to the KITTI 2d format.
I downloaded the example data for some clarity on the structure the data should have but I can't figure out what the evaluate_tracking.seqmap.test file is representing and how I should configure it for my custom dataset.
Hope anyone can help on this,
thanks in advance

Thank you for your great work! I wonder whether your metrics can measure the detection and tracking performance for MOT17Det and MOT20Det? Thank you so much!

how to evaluate mutil class dataset?

Like Kitti dashboard, How do we get graphs such as 'AssA bs DetA', 'AssPr vs AssRe', 'DetPr vs DetRe', and 'HOTA(0) vs LocA(0)'?
please, let me know a python script.