Comments (7)
What is it that I can exactly help you with here?
Is there any issue with the evaluation code? Or just that your reimplementation of SORT doesn't achieve identical results to the original paper?
It is common for reimplementations to not achieve exactly the same results due to a large number of different hyper-parameters to set.
Furthermore, it seems as though the results you posted (MOTA = 37.682) are actually BETTER than those from the original paper (MOTA = 34.0), so that is good isn't it? (Isn't that what you want?)
One reason your results are likely better is because you are probably using a different detector (there are many different version of FRCNN detections and not all are the same). I don't know what one the original SORT used, but it is very likely not the same as in the det.txt file provided (although I am not sure of this).
Overall though, I don't see a problem that I can help you with here.
Let me know if there is something else that I have missed.
For now, I will close this, but feel free to re-open it if I haven't answered well enough,.
from trackeval.
Hello again,
let me clarify my situation, i did not re-implement SORT, I used it as is. I followed their git repo (https://github.com/abewley/sort) and paper (https://arxiv.org/pdf/1602.00763.pdf). I did not use a frcnn detector at all, I just used the detections provided in the git repo of SORT. I have not repeated the same evaluation in the depreciated version of mot metric calculator (https://github.com/dendorferpatrick/MOTChallengeEvalKit) yet so I cannot compare the results with your evaluator now. I hope I am clearer than before. Thank you.
Fatih.
from trackeval.
Okay. Makes sense, but without more info I can't know why there is a difference to the paper.
It would be interesting if you ran the depreciated version like you said and see if there is the same score as for this repo.
If it's the same (or very similar, there are some known very minor differences), then there is no problem with this repo, and maybe this is an issue for the SORT repo instead??
Jono
from trackeval.
I evaluated by using py-motmetrics repo (https://github.com/cheind/py-motmetrics) and the results are pretty close to the results of HOTA TrackEval except MOTP score. I am going to investigate the inconsistency between the paper and TrackEval results on SORT repo by opening an issue as you suggested. Thank you for your guidence and great work by serving this great repo to the MOT community.
Fatih.
from trackeval.
Yeah the MOTP is defined here as 1-MOTP (for historical reasons).
Cool, I guess this is closed then
from trackeval.
Now everything is clear! The results presented on the Paper are for the TEST sequences of MOT15, evaluated on MOT benchmark test server. My results are for the TRAINING sequences. I was comparing apple and pear and questioning why they are not the same. Sorry for my mistake!
Fatih.
from trackeval.
Glad to hear it's all cleared up. :)
from trackeval.
Related Issues (20)
- How to evaluate test data without ground truth? HOT 1
- How to evalute test set of kitti 2D HOT 2
- Implementation of ai city challenge?
- a method to modify the result file to csv
- How can I further understand the meaning of "Metric arguments"?
- regarding track id matching and tracker results
- Tracker evaluate on KITTI dataset but has format error HOT 1
- seqinfo.ini error HOT 1
- About the confused relationship between DetA and predict IDs. HOT 2
- Evaluation of class "cyclist"
- How are 2d bounding boxes labeled in kitti ? HOT 1
- The difference between MOT and MOTS evaluation on HOTA metrcis HOT 1
- how to calculate HOTA for DeepSORT? HOT 1
- 3D MOT code
- I encountered an error and don't know how to solve it HOT 1
- Evaluation of tracks where not a single object was tracked HOT 7
- Is there any difference in the IDSW metric between TrackEval and motmetrics? HOT 1
- Numpy version HOT 1
- Too Low HOTA and IDF1 HOT 1
- Negative metrics value HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from trackeval.