machinevisionbeans / sort_of Goto Github PK

Extending SORT's Kalman filter measurement model to add a velocity component derived from Optical Flow. This was part of my Bachelor's Thesis for Computer Science in collaboration with the German Aerospace Center. Although not used for pedestrian tracking, the method can be evaluated and used with the MOT datasets.

Basic idea can be summarized as:

• Get a region of interest in form of a bounding box and detect feature points in this region.
• Calculate the optical flow for the detected feature points to the frame before, providing a velocity.
• Choose the velocity for updating based upon the lowest Mahalanobis distance.
• Update the Kalman filter of the track with the detection and/or the velocity.

For the detection of feature points I use OpenCV's ORB, calculation of the sparse optical flow is done by OpenCV's implementation of the Robust Local Optical Flow (RLOF).

This implementation runs at ~50FPS on the 2D MOT 2015 Dataset on an AMD Ryzen7 3700X with 64GB RAM with up to 50 feature points per detection, parametrization for RLOF is set to:

useIlluminationModel = false;
useInitialFlow = false;
maxIteration = 5;
setUseMEstimator(false);
setUseGlobalMotionPrior(true);
setSupportRegionType(cv::optflow::SR_FIXED);

The following external libraries are needed:

• OpenCV for the Kalman filter, the bounding box representation, calculation of feature points and calculation of optical flow.
• dlib for the linear assignment problem.

If you are on an ArchLinux based system like me, install OpenCV from the official repository:

$ sudo pacman -S opencv glu glew hdf5 vtk

dlib can be obtained from the aur.

On Ubuntu based distributions simply install them with:

$ sudo apt-get install libopencv-contrib-dev libdlib-dev

I use cmake for the build process.

$ git clone https://github.com/tylernewnoise/SORT_OF
$ cd SORT_OF
$ mkdir build && cd build
$ cmake .. -DCMAKE_BUILD_TYPE=Release
$ make

Put your data in this folder structure:

SORT_OF
  └─ data
    └─ Sequence
      ├─ det
      └─ gt

Run

$./SORT_OF -d

inside the SORT_OF directory to get a visualization with neat bounding boxes. Omit the -d to run the tracker offline. Add -o to write results to output/SequenceName.txt in mot challenge format. One can use py-motmetrics to evaluate the results.

I tested and wrote this on AchLinux, for other operating system changes may have to be made.

Functionality is written in a single header file for simple usage in other projects, just import the header with:

#import sort_of.h

Detections and the corresponding image have to be passed as a struct DetectionsAndImg{} which is defined as:

struct DetectionsAndImg {
  std::vector<BBox> detections;
  cv::Mat img;
};

Tracks are returned as std::vector<struct Track>, whereat the struct Track{} is defined as:

struct Track {
  BBox bbox;
  std::size_t id{};
};

See the demo.cpp for an example.

I used Alex's detections and py-motetrics for evaluation. These are the data sets from a static camera only. Results can be improved by increasing the value of max_age.

SORT paper, describing the original Simple Online and Realtime Tracking method:

@inproceedings{sort2016,
    author = {Bewley, Alex and Ge, Zongyuan and Ott, Lionel and Ramos, Fabio and Upcroft, Ben},
    booktitle = {2016 IEEE International Conference on Image Processing (ICIP)},
    title = {Simple online and realtime tracking},
    year = {2016},
    pages = {3464-3468}
}

Also checkout nwojke 's DeepSORT implementation.

ORB feature point detection is explained on OpenCV's site.

Papers for the Robust Optical Flow can be found here.

Other SORT C++ implementations:

• samuelmurray
• yasenh
• mcxmiming
• my own