At the moment there are some issues with time estimates in vot_test function we should fix this and also provide some way of predicting time during the experiments.

Pre-built mex files have issues with raising exceptions on Windows because of issues with raising exceptions across dll boundaries. The only solution at the moment is to compile mex files yourself.

Improve progress reporting when the experiment is performed. This includes: expected time to finish and scores output (remove deprecated code from calculate_scores.m).

if you use octave, you may get the error:
regexprep: \ at end of pattern at position 1 of expression

The remedy is:

print_text.m
line.10: text = regexprep(text, '', '\');

When calculating the overlap between the polygons there is an error in calculating the height (or width).

In the function compute_polygon_overlap the boundaries are calculated as follows:
float x = MIN(b1.left, b2.left);
float y = MIN(b1.top, b2.top);

int width = (int) ((MAX(b1.right, b2.right)) - x);
int height = (int) ((MAX(b1.bottom, b2.bottom)) - y);

These calculations depend on the macros MAX and MIN defined in the file region.h (trax/lib/region.h)
However because of the missing brackets around MAX (and MIN) the results of the rasterisiation function is not always as desired. This can be verified by printing the value of for example the calculated height.

The macro is currently defined as :

define MAX(a,b) ((a) > (b)) ? (a) : (b)

but should be defined as:

define MAX(a,b) (((a) > (b)) ? (a) : (b))

If required more details can be provided.

As mentioned in #12 the CamShift example behaves strangely. We should replace it with a simple NCC template matching.

Provide a Docker image that installs Octave and VOT toolkit.

We could gather more information about the system, like CPU type, disk type, etc.

When running the script: "run_analysis.m" of the master branch, I get the following error:

Error using set
Bad property value found.
Object Name :  axes
Property Name : 'XLim'
Values must be increasing and non-NaN.

Error in xlim (line 44)
        set(ax,'xlim',val);

Error in generate_ranking_plot (line 25)
    xlim([1, plot_limit]);

Error in generate_ranking_report (line 120)
    h = generate_ranking_plot(trackers, average_accuracy_ranks,
    average_robustness_ranks, ...

Error in ranking_analysis (line 84)
    report_file = generate_ranking_report(context, trackers(available),
    experiment, accuracy, robustness, ...

Error in run_analysis (line 15)
ranking_index = ranking_analysis(context, trackers, sequences, ...

TraX support should be detected automatically for a tracker and then cached.

It seems there is some failure with the include files / syntax error.

error message is:
ERROR: Unable to compile MEX function: "/home/gustav/Documents/MATLAB/vot/vot-toolkit/sequence/region_overlap.cpp:7:1: error: ‘region_container’ does not name a type
region_container* get_polygon(const mxArray * input) {
^
/home/gustav/Documents/MATLAB/vot/vot-toolkit/sequence/region_overlap.cpp: In function ‘void mexFunction(int, mxArray**, int, const mxArray**)’:
/home/gustav/Documents/MATLAB/vot/vot-toolkit/sequence/region_overlap.cpp:45:2: error: ‘region_container’ was not declared in this scope
region_container* p1;
^
/home/gustav/Documents/MATLAB/vot/vot-toolkit/sequence/region_overlap.cpp:45:20: error: ‘p1’ was not declared in this scope
region_container* p1;
^
/home/gustav/Documents/MATLAB/vot/vot-toolkit/sequence/region_overlap.cpp:46:20: error: ‘p2’ was not declared in this scope
region_container* p2;
^
/home/gustav/Documents/MATLAB/vot/vot-toolkit/sequence/region_overlap.cpp:60:26: error: ‘get_polygon’ was not declared in this scope
p1 = get_polygon(prhs[0]);
^
/home/gustav/Documents/MATLAB/vot/vot-toolkit/sequence/region_overlap.cpp:68:9: error: ‘region_overlap’ was not declared in this scope
region_overlap overlap = region_compute_overlap(p1, p2);
^
/home/gustav/Documents/MATLAB/vot/vot-toolkit/sequence/region_overlap.cpp:68:24: error: expected ‘;’ before ‘overlap’
region_overlap overlap = region_compute_overlap(p1, p2);
^
/home/gustav/Documents/MATLAB/vot/vot-toolkit/sequence/region_overlap.cpp:70:21: error: ‘overlap’ was not declared in this scope
result[0] = overlap.overlap;

Separate code for result bundle generation into a separate script, it can be done separately from the experiments now that the toolkit is more general.

Merge development changes and add a VOT2014 tag.

The initialization rectangle for the vot2013 region_noise
experiment is in 4 corners format, instead of top_left x y and width height
as it is for the baseline expetiment.
This is probably some kind of unintentional behaviour, since vot2013 should
use the old format as all the sequences are annotated with axis-align bboxes and the trackers, that are evaluated on that benchmark, expect the old format.

Hi,

Here is a bug (if you are using the old file protocol):
vot-toolkit/blob/master/tracker/examples/c/vot.h

line 87: float* pointbuf = (float*) malloc(sizeof(float) * 4);

it should have been:
float* pointbuf = (float*) malloc(sizeof(float) * 8); //there are four (x,y) values for the corners of the rotated rectangle.

Thank you very much.

With best Regards
HCK

Move burnin and repeat variables to individual experiment instead of storing them in the global scope.

Probably missing check if the gt is valid when creating the region.txt file for sub-sequence after tracker failure (for gt with multiple consecutive NaN entries).

Is the tracker failure because of gt is NaN count as a failure in the evaluation ?

The toolkit would be easier to set up if the native parts of it could be simply downloaded in a precompiled form.

Support trackers written in Python. Related to votchallenge/trax#5.

Currently when generating the legend for the report, if the name of a tracker is to long it will overlap with the symbol/name of some tracker in the next column.

This could be fixed by either making the columns wider (is this even possible?) or making the figure larger before generating the pdf/png output image. (this is very easy and what i have done so far)

I suppose it is also possible to truncate the name of the tracker, if it is known how wide a column is. The width varies with the size of the figure window though.

The term label is confusing, replace it with tag (which is more appropriate and a bit broader that attribute).

We will be slowly dropping support for the file-based legacy integration protocol. As a first step the toolkit should issue a deprecation warning every time a tracker that does not use TraX is used.

When running a tracker on a vot2014-final version the performance profile does not get computed, therefore speed normalization is not performed during speed analysis.

Dear all,
I have tested my tracker using the toolkit but the obtained results evaluated using @trax_wrapper and @system_wrapper are quite different. I have spent a lot of time to find the reason for this issue but still cannot figure it out. Have anyone meet a similar issue for this? If have, could you tell me the reason? Or how can I disable @trax_wrapper during evaluation of my tracker?
Thank you in advance!

I am using windows xp and octave 3.6.4 (VS 2010 built).
I have the error:

error: imdilate: image and structuring element must have the same class
error: called from:
error: C:\Program Files (x86)\Octave-3.6.4\share\octave\packages\image-2.0.0\imdilate.m at line 47, column 5

I fix it forcing "se" to logical at:
estimate_accuracy.m

line.17: if is_octave()
line.18: se = logical ([zeros(burnin - 1, 1); ones(burnin, 1)]);
line.19: else

Basic use cases like workspace initialization, execution of experiments, analysis generation need unit tests so that support for Octave and Matlab can be tested regularly.

Hi,

This is not a bug.
But I find that the tracking time computed include the time to read the image file.
It may be better if file i/o can be excluded, e.g. in xxx.cpp

for(f = 1;; f++)
{
    // *****************************************
    // VOT: Call vot_frame to get path of the 
    //      current image frame. If the result is
    //      null, the sequence is over.
    // *****************************************
    const char* imagefile = vot_frame();
    if (!imagefile) break;

    IplImage *imgColor=cvLoadImage(imagefile,1);

     //start timer here
     ....   

    //start tracking
if(f==1){
       //initialise tracker here!  (first frame)
          ...
}else{
         //run tracker
         ...
} 

   //end timer
   ....  

    // *****************************************
    // VOT: Report the position of the object 
    //      every frame using vot_report function.
    // *****************************************
    vot_report(selection);  
}

// *************************************
// VOT: Call vot_deinitialize at the end
// *************************************
vot_deinitialize(); //total time can be written to a output file

By the way, will the organization will be running the binary submission on a common machine to compare timings?

Currently, http://votchallenge.net/howto/workspace.html
states that you have to add at least

addpath('<vot-toolkit-dir>/workspace'); 

to your environment to use 'workspace_create'. If I do that I get notified about missing functions/m-files. ('is_octave.m' is for example not on the path then).

However,

addpath(genpath('<vot-toolkit-dir>'));

works for 'workspace_create'.
This is a minor issue, but maybe the howto could be updated.

Improve the look and functionality of generated reports.

I am experiencing a very strange behavior: I have run the experiments on a tracker with run_experiments. The report and the results are saved as normally in their folders.

Now when I try to run_experiments.m, the script just calculates a new score using the results in the folder working_directory/results/trackerName. The problem is that each time it prints different scores (accuracy/failures)

Example:

>> run_experiments
 Outputting scores:
    Sequence "ball" - Accuracy: 0.470, Failures: 3.000, Speed: 57.177
    Sequence "basketball" - Accuracy: 0.449, Failures: 4.000, Speed: 70.487

>> run_experiments
 Outputting scores:
    Sequence "ball" - Accuracy: 0.467, Failures: 2.667, Speed: 57.177
    Sequence "basketball" - Accuracy: 0.445, Failures: 5.333, Speed: 70.487

>> run_experiments
 Outputting scores:
    Sequence "ball" - Accuracy: 0.469, Failures: 2.333, Speed: 57.177
    Sequence "basketball" - Accuracy: 0.445, Failures: 6.333, Speed: 70.487

As you can see, especially the failures, I got different scores given the same results. The only thing that doesn't change is the Speed value.

This potentially could be a very important bug.

In my tracker_mytracker.m file I put:

tracker_trax = true

and then in configuration.m I put:

set_global_variable('trax_client', 'traxclient');

I've also compiled trax with cmake and ran make install. It seems to compile fine and I've got the library and client tool in /usr/local/bin and /usr/local/lib. I also compiled the c++ ncc_trax example and set tracker_command to reference it. However, it doesn't seem to work. In particular, when I try to do run_test this is what happens (I'm using Octave to run the scripts):

octave:3> run_test
Initializing VOT environment ...
Compiling MEX files ...

***************************************************************************

Welcome to the VOT sandbox!
This process will help you prepare your tracker for the evaluation.
When beginning with the integration it is recommended to follow the steps 
a, b, c to verify the execution and the output data.

***************************************************************************

Choose action:
  a - Generate a directory with input data for manual test
  b - Run tracker once within the evaluation
  d - Enable debug output
  e - Exit
Choose action: a
Choose a sequence:
  1 - "ball"
  2 - "basketball"
  3 - "bicycle"
  4 - "bolt"
  5 - "car"
  6 - "david"
  7 - "diving"
  8 - "drunk"
  9 - "fernando"
  10 - "fish1"
  11 - "fish2"
  12 - "gymnastics"
  13 - "hand1"
  14 - "hand2"
  15 - "jogging"
  16 - "motocross"
  17 - "polarbear"
  18 - "skating"
  19 - "sphere"
  20 - "sunshade"
  21 - "surfing"
  22 - "torus"
  23 - "trellis"
  24 - "tunnel"
  25 - "woman"
Selected sequence: 4
Input data generated in directory "/tmp/oct-XWI5M5"
Open the directory in a terminal and manually execute the tracker command.
The current command as defined in the environment is: traxclient  -r 1 -e "MATLAB_ROOT=/usr" -I "/tmp/oct-XWI5M5/images.txt" -G "/home/davis/temp/new-vot-toolkit/working_dir/sequences/bolt/groundtruth.txt" -O "/tmp/oct-XWI5M5/output.txt" -S "/tmp/oct-XWI5M5/initialization.txt" -T "/tmp/oct-XWI5M5/timing.txt" -- /home/davis/temp/new-vot-toolkit/working_dir/../examples/cpp/ncc_trax
Once the tracker is working as expected, delete the directory.

Ok, so I run this in a terminal window and it hangs for maybe 30 seconds and then eventually prints the unable to establish connection error:

/tmp/oct-XWI5M5$ traxclient  -r 1 -e "MATLAB_ROOT=/usr" -I "/tmp/oct-XWI5M5/images.txt" -G "/home/davis/temp/new-vot-toolkit/working_dir/sequences/bolt/groundtruth.txt" -O "/tmp/oct-XWI5M5/output.txt" -S "/tmp/oct-XWI5M5/initialization.txt" -T "/tmp/oct-XWI5M5/timing.txt" -- /home/davis/temp/new-vot-toolkit/working_dir/../examples/cpp/ncc_trax

Error: Unable to establish connection

I get the same error if I try to run it through Octave as well:

Choose action:
  a - Generate a directory with input data for manual test
  b - Run tracker once within the evaluation
  d - Enable debug output
  e - Exit
Choose action: b
Choose a sequence:
  1 - "ball"
  2 - "basketball"
  3 - "bicycle"
  4 - "bolt"
  5 - "car"
  6 - "david"
  7 - "diving"
  8 - "drunk"
  9 - "fernando"
  10 - "fish1"
  11 - "fish2"
  12 - "gymnastics"
  13 - "hand1"
  14 - "hand2"
  15 - "jogging"
  16 - "motocross"
  17 - "polarbear"
  18 - "skating"
  19 - "sphere"
  20 - "sunshade"
  21 - "surfing"
  22 - "torus"
  23 - "trellis"
  24 - "tunnel"
  25 - "woman"
Selected sequence: 4
Sequence "bolt"
Error: Unable to establish connection

I've looked into the code and it seems that overlap measure assumes that the boxes being compared are rectangles, while ground truth is provided in rotated rectangles. For trackers which output rotated rectangles the measure cannot find true Jaccard distance. I would like to know if it is the case?

The traxclient application supports setting the timeout, but this option is not exposed by the toolkit. For slower trackers this may be a problem (the current default is 30 seconds).

I downloaded a fresh copy of the repository, went into the root folder, ran octave and then typed the following:
octave:1> addpath(pwd)
octave:2> cd working_dir/
octave:3> vot_initialize
error: 'initialize_defaults' undefined near line 9 column 1
error: called from:
error: /home/davis/temp/new-vot-toolkit/vot_initialize.m at line 9, column 1

I get an error that it's trying to call initialize_defaults but it apparently isn't anywhere in the repository.

On some platforms (OSX, Windows) some part of Matlab interface remains visible for Matlab-based trackers. This has been partially solved by using TraX and mwrapper, but for the old file-system based integration we should still investigate other options. One option is to use -minimize flag (Windows). The other is to use an undocumented call to:

com.mathworks.mde.desk.MLDesktop.getInstance.getMainFrame.hide

There may also be other options.

Hi,
I have been using VOT 2014 and observed that if a tracker fails it does not re-start immediately. I have also read the new performance metroc of VOT 2015 which uses the accuracy outputs of the VOT 2014.
So my question is: why the framework does not detect the failure (zero overlap) and re-start the tracker from the frame where it should start ? Since it doesn't do so, the framework waits till the end of the sequence although the tracker failed at the very beginning of the sequence. It is really time consuming if your tracker fails mostly in long sequences.

Thanks.
Erhan

Evaluation stuck when testing for trax protocol. Fix: setting tracking_interpreter = 'matlab' in tracker script.

If the resutls folder is added to the matlab path the create_trackers function will fail because the check if a file with a specific name exists on the disk is too vague. The function should only check if a file exists relative to the active workspace.

IR vot sub challenge integration, needs an experiment file, and some extension to read .png files

Hi,

I'm trying to follow the instructions on http://www.votchallenge.net/howto/workspace.html and http://www.votchallenge.net/howto/integration.html. Unfortunately, there is no tutorial yet on how to get it working with Octave. I'm guessing that functions like generate_matlab_command() perhaps need an Octave equivalent also?

Could you please test that the current toolkit works with the latest Octave and provide a tutorial on getting it to work?

Thanks in advance!

p.s. I've gotten as far as setting up the workspace, but can't get run_test() to work even on the provided ncc.m tracker. The following error always pops up:

error: 'nargin' undefined near line 36 column 7
error: called from
unimplemented at line 36 column 3
create_sequence at line 51 column 17
load_sequences at line 67 column 21
workspace_load at line 117 column 15
run_test.m at line 6 column 24

I'm using Octave 4.0.0 by the way.

The constructor of the VOT class calls the method getNextImage() and is not consisent with the 2013 version of the example provided in cpp tracker folder:

    VOT vot_io("region.txt", "images.txt", "output.txt");     // Calls getNextImage

    //img = firts frame, initPos = initial position in the first frame
    VOTPolygon initPos = vot_io.getInitPolygon();
    vot_io.getNextImage(img);                                 // Calls getNextImage

    //output init also bbox
    vot_io.outputPolygon(initPos);

    //tracker initialization
    tracker.init(img, initPos);                              // Initialize the algorithm with the second frame

The problem is that with the example provided, the algorithm is initialized with the second frame.

I suggest to remove the getNextImage in the VOT constructor (it's useless there).
I.e. just comment these two lines:

//read dummy image
//cv::Mat mat;
//getNextImage(mat);

Of the file: \tracker\examples\cpp\vot.hpp

I encounter the following error while running workspace_create. Indeed, the function tracker_create is not contained in the dev branch.

Select one of the available experiment stacks:
1 - test
2 - vot2013
3 - vot2013_extra
4 - vot2014
Selection: 1
Input an unique identifier for your tracker: DSST
Undefined function 'tracker_create' for input arguments of type 'char'.

Error in workspace_create (line 110)
tracker_create('identifier', tracker_identifier, 'directory', directory);

in line 362 to 367

    for (int j = 0; j < _region->count; j++) {
        top = MIN(top, _region->y[i]);
        bottom = MAX(bottom, _region->y[i]);
        left = MIN(left, _region->x[i]);
        right = MAX(right, _region->x[i]);
    }

there is no i. It should be j or change for loop.

I get the error of getline function which works only for linux gcc.
I use Visual Studio 2012 on Win7.

line 44 of file calculate_scores.m

function normalize_speed() takes 5 arguments but the caller passes only 4, skipping one in the middle (3rd), resulting in wrong arguments order.

instead of tracker structure, the function get sequence structure resulting in the error in "performance" field test
if ~isfield(tracker, 'performance')
error('Tracker %s has no performance profile, unable to normalize speed.', tracker.identifier);
end;

Similarly to LaTeX table export we can also have simple CSV export so that data can be imported to spreadsheet editor.

When I integrate the ncc tracker in the VOT2014 evaluation kit by using TraX protocol on Windows, occur the error "Unable to start the tracker process". Who can tell me how to write the "tracker-comand=<TODO: set a tracker executable command>" in the tracker-ncc.m?

The updated toolkit for VOT2015 apparently has some issues with robustness ranking which results in all trackers being ranked equally.

Octave's version of unique() doesn't support the stable option. So when you run run_analysis you get this output:

octave:6> run_analysis
Initializing VOT environment ...
Compiling MEX files ...
error: unique: invalid option
error: called from:
error:   /usr/share/octave/3.8.1/m/set/unique.m at line 58, column 9
error:   /home/davis/temp/new-vot-toolkit/tracker/create_trackers.m at line 23, column 13
error:   /home/davis/temp/new-vot-toolkit/working_dir/run_analysis.m at line 10, column 10

At the moment the entire dataset is downloaded as one zip file. This can be problematic and no progress report is available. It would be better to have individual sequence bundles and a list file available as separate resources.

An output log of the tracker should be saved to some meaningful location in the workspace if a crash is detected to facilitate easier debugging.