Mentioned in the paper "We use multiples of three in the range [3,30] to form the grid, based on which δw = 12 and δb = 15 are selected in the final implementation." What do the two parameters correspond to in the code？

""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
def add_node(self, frame_index, recorder, node):
# iou judge
if len(self.nodes) > 0:
n = self.nodes[-1]
iou = n.get_iou(frame_index, recorder, node.id)
delta_frame = frame_index - n.frame_index
if delta_frame in TrackerConfig.min_iou_frame_gap:
iou_index = TrackerConfig.min_iou_frame_gap.index(delta_frame)
# if iou < TrackerConfig.min_iou[iou_index]:
if iou < TrackerConfig.min_iou[-1]:
return False
self.nodes.append(node)
self.reset_age()
return True
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
I am confused abut ’‘self.reset_age()’‘ .
My confusion is when to grow the age of this track because everytime when executing add_node , the age of this track would be reset to 0 ?

Is line 296 in tracker.py "self.all_similarity[frame_index][pre_index] = iou" a type error ?

Should it be "self.all_iou[frame_index][pre_index] = iou" instead of "self.all_similarity[frame_index][pre_index] = iou" ?

Thanks for reply.

very nice work, but I'd like to say there are some mistakes with your code. In your original code, the images and the bounding boxes are not corresponding.
mot.py, line 113: current = self.recorder[frame_index]
should be: current = self.recorder[frame_index-1]
mot.py, line 152: next = self.recorder[next_frame_index]
should be: next = self.recorder[next_frame_index-1]
OR:
mot.py, line 99:
image_path = os.path.join(self.folder, 'img1/{0:06}.jpg'.format(frame_index))
modified version:
image_path = os.path.join(self.folder, 'img1/{0:06}.jpg'.format(frame_index+1))

@shijieS
Thanks for your great work. I have not modified any parameters when training the model on MOT17. After training, I obtain the model "sst300_0712_79680.pth". However, the predicted trajectories are wrong, but the model provided by the authors can produce the precise trajectories. The problem is same with 51 @hanshong . Hope to your reply, thanks.

Hello, does the SST can be implemented with caffe?

Hi,

Thanks for sharing the code.
In your code, the output features of selector are with shape (80, 520) and the inputs to affinity net are with shape(1040, 80, 80).
Why not set the output features of selector as (n_obj, 520), and then the inputs to affinity net will be with shape (1040, n_pre, n_cur). It seems okay with variable input size to FCN. And it will save time for training and inference.

HI
thank you for provide the code .
when i do the 'test' on UA,i can not find 'config['ua_detection_root'] = '/media/ssm/seagate/dataset/UA-DETRAC/EB' ' EB.how to get it?thank you a lot.

UserWarning: indexing with dtype torch.uint8 is now deprecated, please use a dtype torch.bool instead
请问怎样处理？

Hello,

Thank you for providing open-source code for SST tracker. I am trying to retrain this tracker on MOT17 and I am running into issues. I get this traceback:

Traceback (most recent call last):
File "train_mot17.py", line 234, in
train()
File "train_mot17.py", line 142, in train
batch_iterator = iter(data_loader)
File "/home/shashank/anaconda3/envs/SST/lib/python3.5/site-packages/torch/utils/data/dataloader.py", line 417, in iter
return DataLoaderIter(self)
File "/home/shashank/anaconda3/envs/SST/lib/python3.5/site-packages/torch/utils/data/dataloader.py", line 242, in init
self._put_indices()
File "/home/shashank/anaconda3/envs/SST/lib/python3.5/site-packages/torch/utils/data/dataloader.py", line 290, in _put_indices
indices = next(self.sample_iter, None)
File "/home/shashank/anaconda3/envs/SST/lib/python3.5/site-packages/torch/utils/data/sampler.py", line 119, in iter
for idx in self.sampler:
File "/home/shashank/anaconda3/envs/SST/lib/python3.5/site-packages/torch/utils/data/sampler.py", line 50, in iter
return iter(torch.randperm(len(self.data_source)).long())
RuntimeError: invalid argument 1: must be strictly positive at /pytorch/torch/lib/TH/generic/THTensorMath.c:2247

I changed the configurations appropriately in the config file before trying to train and I made sure that the dataset paths is configured correctly.

For training on MOT17, I have not modified any parameters except batch_size=2 and num_workers=0. And I am not sure whether the learning rate(1e-6) is too low that the loss is non-convergent and always between 1.4-2.6 in the training process. In the generated images, the connection lines between the front and back frame's objects disappears, or all lines points to the same object in the back frame.
For testing on MOT17, the predict trajectories are scattered and wrong straight lines while sst300_0712_83000 can generate great and precise trajectories.

Are there some parameters that need to be adjusted when retraining on MOT17? Hope for response, thanks.

As I had compared original VGG 16 weights size with given Dropbox weights provided in readme. I observed size of given weights are lesser than original.

Hello, I want to test on my own video.

I write the code like this to select two ROIs for query:

from tracker import SSTTracker, TrackerConfig, Track
# from sst_tracker import TrackSet as SSTTracker
import cv2
from data.mot_data_reader import MOTDataReader
import numpy as np
from config.config import config
from utils.timer import Timer
import argparse
import os

parser = argparse.ArgumentParser(description='Demo images creating')
parser.add_argument('--version', default='v1', help='current version')
parser.add_argument('--mot_root', default=config['mot_root'], help='MOT ROOT')
parser.add_argument('--type', default=config['type'], help='train/test')
parser.add_argument('--show_image', default=True, help='show image if true, or hidden')
parser.add_argument('--log_folder', default=config['log_folder'], help='video saving or result saving folder')
parser.add_argument('--mot_version', default=17, help='mot version')
parser.add_argument('--video', required=True)
args = parser.parse_args()


selected_frames = [100+5*i for i in range(6)]
#image_folder = os.path.join(args.mot_root, 'train/MOT17-09-FRCNN/img1')
#detection_file_name = os.path.join(args.mot_root, 'train/MOT17-09-FRCNN/det/det.txt')

def create(c,video_path):
    if not os.path.exists(args.log_folder):
        os.mkdir(args.log_folder)

    tracker = SSTTracker()
    '''
    reader = MOTDataReader(image_folder=image_folder,
                  detection_file_name=detection_file_name,
                           min_confidence=0.0)
    
    select_squences = [402, 404, 410, 422]
    '''
    cap = cv2.VideoCapture(args.video)
    frame_index = 0
    new_bboxes=[]
    bbox = [0]*4
    while True:
        ok, img = cap.read()
        if img is None or os is False:
            continue
        frame_index = frame_index+1
        if frame_index < 200:
            continue
        if frame_index == 200:
            x1, y1, w1, h1 = cv2.selectROI("SST", img, fromCenter=False)
            x2, y2, w2, h2 = cv2.selectROI("SST", img, fromCenter=False)
            bbox[0] = x1
            bbox[1] = y1
            bbox[2] = x1 + w1
            bbox[3] = y1 + h1
            new_bboxes.append(bbox)
            bbox[0] = x2
            bbox[1] = y2
            bbox[2] = x2 + w2
            bbox[3] = y2 + h2
            new_bboxes.append(bbox)

        image_org = tracker.update(img, np.array(new_bboxes), args.show_image, frame_index)
        print("OK")
        frame_index += 1

        if args.show_image and not image_org is None:
            # det[:, [2, 4]] *= float(w)
            # det[:, [3, 5]] *= float(h)
            # boxes = det[:, 2:6].astype(int)
            # for bid, b in enumerate(boxes):
            #     image_org = cv2.putText(image_org, '{}'.format(bid), tuple(b[:2]), cv2.FONT_HERSHEY_SIMPLEX, 1,
            #                             (0, 0, 0), 2)
            cv2.imshow('res', image_org)
            cv2.imwrite(os.path.join(args.log_folder, '{0:06}.jpg'.format(i)), image_org)
            # cv2.waitKey(0)
            print('frame: {}'.format(i))



if __name__ == '__main__':
    c = (0, 0, 4, -1, 5, 4)
    TrackerConfig.set_configure(c)

    create(c, args.video)

However, I cannot launch the tracker query and quit with failure:

use configure: init_test_mot17
Select a ROI and then press SPACE or ENTER button!
Cancel the selection process by pressing c button!
Select a ROI and then press SPACE or ENTER button!
Cancel the selection process by pressing c button!
OK
ASSERT: "false" in file qasciikey.cpp, line 501
Aborted

My question is what is the root cause of my failure?

How can I fix the problem?

Thanks for sharing your idea.

Regards!

Can you help me, I am new to pytorch.

**Traceback (most recent call last):
File "SST/train_ua.py", line 246, in
train()
File "SST/train_ua.py", line 184, in train
out = net(img_pre, img_next, boxes_pre, boxes_next, valid_pre, valid_next)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/data_parallel.py", line 150, in forward
return self.module(*inputs[0], **kwargs[0])
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/content/SST/layer/sst.py", line 68, in forward
x_next = self.forward_vgg(x_next, self.vgg, sources_next)
File "/content/SST/layer/sst.py", line 179, in forward_vgg
x = vggk
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, kwargs)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py", line 338, in forward
self.padding, self.dilation, self.groups)
RuntimeError: CUDA out of memory. Tried to allocate 1.55 GiB (GPU 0; 11.17 GiB total capacity; 9.28 GiB already allocated; 1.12 GiB free; 448.72 MiB cached)

can you tell me make the 'det.txt'?
or how do I change the MOTDataReader?
thanks

Should parametres in BatchNorm not be initialized ? I just read the way of initializing for the params in Conv2d like follows :
"""
def weights_init(m):
if isinstance(m, nn.Conv2d):
xavier(m.weight.data)
m.bias.data.zero_()
"""
What about the initialization for the params in BN ？ But I did read the net construction with
"def add_extras(cfg, i, batch_norm=True):"
and
"def selector(vgg, extra_layers, batch_normal=True):"
and corresponding calling to them
"return SST(phase, *selector(vgg(base[str(size)], 3),add_extras(extras[str(size)],
1024)),add_final(final[str(size)]), use_gpu )",
or is there something that I mistook ? Welcom to leave your perspectives , thanks a lot.

Hi,
in your paper you say regarding MOT15 that

The provided ground truth detections do not consider object occlusions and a bounding box for a completely occluded object of interest is provided for training anyway

But this is also true for MOT17, as you can see in this frame here (frame number 421 of MOT17-02-DPM) where I have superimposed the ground truth over the image:

This is also described in the MOT16 dataset paper, look at TABLE 1, that describes the criteria for the dataset annotation:

Under "Occlusions" they clearly state:

Always annotate during occlusions if the position can be determined unambiguously.

Thus, your model trains fine even though bounding boxes for completely occluded objects are provided :)

I have retried your project on the MOT17 evaluation metric,but I got a little lower score than you,why?

I'm trying to write code for object tracking using yolo detection results in a .txt file. But, I got stuck at writing code for DataReader. Can some help me out?

the yolo output looks kinda similar to the below line:

/home/anil/yolo/yolo_test/22_march_4f00910.jpg, Motorcycle, 0.548347, 0.713307, 0.199746, 0.2855
(path to the image directory, predicted class, x,y,w, h)

Hello @yanteng @shijieS @ZhangBin1993

1. I have a video of people walking and crossing each other I am trying to give a unique id to them even if they cross each other.
2. Is it possible to give the same id for the object which was available in camera 1 and camera 2.
   If not do anyone knows how to solve these problems

As is illustrated in the paper, in training phase two affinity matrices are processd by row-wise or column-wise softmax. In test phase, how's the final affinity matrix computed since only one matrix is needed?

Hey,

Thanks for your pretty job.

I find the function get_all_choices_max_track_node in tracker.py/TrackerConfig is same with get_all_choices_decay(). Is that wrong? Maybe it should be:

def get_all_choices_max_track_node():
        # ???
        return [(1, 0, 0, 0, i5, 2) for i5 in range(11)]

Or is there something I missed?

Hello!
First of all thanks for uploading the code of the project. I'm trying to understand the code while reading the paper and I have two questions.

1- In the line bellow, you add the layer from the extension network to sources if the index%6 = 3. From what I understand, sources is a list with the nine layers you keep for the dimensionality reduction but in the paper you are keeping layers with index%6 = 5. Why is there this index difference?

2- When training in train_mot17 you have two variables net and sst_net. It seems like you are using net for training and sst_net for saving and loading the model. What is the reason for that?

@shijieS Hi, I find the implementation of sst loss may exist mistake
in sst_loss.py, the forward function looks like this:

def forward(self, input, target, mask0, mask1):
        set_trace()
        mask_pre = mask0[:, :, :]
        mask_next = mask1[:, :, :]
        mask0 = mask0.unsqueeze(3).repeat(1, 1, 1, self.max_object+1)
        mask1 = mask1.unsqueeze(2).repeat(1, 1, self.max_object+1, 1)
        mask0 = Variable(mask0.data)
        mask1 = Variable(mask1.data)
        target = Variable(target.byte().data)

        if self.use_gpu:
            mask0 = mask0.cuda()
            mask1 = mask1.cuda()

        mask_region = (mask0 * mask1).float() # the valid position mask
        mask_region_pre = mask_region.clone() #note: should use clone (fix this bug)
        mask_region_pre[:, :, self.max_object, :] = 0 
        mask_region_next = mask_region.clone() #note: should use clone (fix this bug)
        mask_region_next[:, :, :, self.max_object] = 0 
        mask_region_union = mask_region_pre*mask_region_next

        input_pre = nn.Softmax(dim=3)(mask_region_pre*input)
        input_next = nn.Softmax(dim=2)(mask_region_next*input)
        input_all = input_pre.clone()

here, mask_region_pre and mask_region_next correspond to M1 and M2 matrixes in the paper respectively, so input_pre should be computed by applying softmax along dim=2 rather than 3, and input_next should be computed by applying softmax along dim=3 rather 2.
Please correct me if I'm wrong

Thanks for your great work . I don't make sense the process of converting reducted features to concatenated ones. For example ,red dot on the feature map to gain the red line feature. Specially，the last layer 3320 gain Nm*20.Could you help me understand it ?

Failed to download test weights from both dropbox and BaiYunPan. Can you update the weights?

Hi，
I have retried your project on the MOT17 dataset，but found that the processing efficiency is too low , can you tell me you final FPS?
Best，

Hello shijieS:
Thanks for your sharing. I am wondering if I can refactor the DAN code framework into my repo. I will cite your repo and paper.

I review the code of augmentation in utils/augmentation.py, and I want to ask: does augmentation generates more pictures? or just process the picture and the number of picture remains the same?

i changed the config['sst_dim'] from 900 to 600, is this ok?

I want to test tracker with my own video. Can u explain steps of doing this?

Hi, thanks for your great work in MOT field.

After reading your paper, I'm confused about the IDSW score of DAN's. In the table 6, we can find that the IDSW of DAN is 8431, that is the largest one among all offline and online methods.

I understand the main contribution of DAN is to improve the tracking association results by using deep learning technique, in that case, can't we get lower IDSW number due to good association results?

I'm not sure whether I missed something.

Why do I think there is a tiny inaccurate place in Fuction ToPercentCoords in utils/augmentations.py in line 85-96 ?

It feels like the following lines
"""
boxes_pre[:, 0] /= width
boxes_pre[:, 2] /= width
boxes_pre[:, 1] /= height
boxes_pre[:, 3] /= height
boxes_next[:, 0] /= width
boxes_next[:, 2] /= width
boxes_next[:, 1] /= height
boxes_next[:, 3] /= height
"""
should be corrected as:
"""
boxes_pre[:, 0] /= width-1
boxes_pre[:, 2] /= width-1
boxes_pre[:, 1] /= height-1
boxes_pre[:, 3] /= height-1
boxes_next[:, 0] /= width-1
boxes_next[:, 2] /= width-1
boxes_next[:, 1] /= height-1
boxes_next[:, 3] /= height-1
"""
cause when I used the original codes as an example to extract a specific number in a tensor by means of function torch.nn.functional.grid_sample, it wouldn't get the expected number, but when I changed width/height to width-1/height-1 , it came out the expected number.
Does anyone have the same confusion?

Hello，may I ask you a question？ In your algorithm，a center point presents a person and then extract its feature by using the point. I wonder if I can use a feature map of of one person to generate its one dimension feature without using the center point information. If it may work ,how could it be done?

Hi,

If there is any instructions on training/testing UA datasets?

I find UA-DETRAC/Insight-MVT_Annotation_Train and UA-DETRAC/gt in your config.py, but don't know what exactly it is.

Best,

As there are more popular and high-efficient base networks, i.e. ResNetX, why select VGG as the feature extractor? As we know, it enjoys tremendous parameters that slow down the computation speed and requires increasing memory.

Hello,

i want to perform tracking on my own video. Do I need to transform the image or do I have to pass the image without transformation as it is read with cv2.imread() ?

Moreover: which are the ways to improve the performances? Are there some parameters one may tune to improve overall performance and avoid ID switches between people detected?

Thanks for this repository and for the support

When run test_mot17.py
cd
python test_mot17.py
The result is shown as follows. The title of each detected boxes represents (track id, detection id). However, it seems that in the results all ID assignments are incorrect as track id and detection id don't match. Do the results indicate failures of tracking?

Why the input is " It + Ct-n " and not " It + Ct " ?

FileNotFoundError: File b'/datanew/hwb/data/MOT17/test/MOT17-01-FRCNN/gt/gt.txt' does not exist
Anyone helps me ?I can not find this files.

It is great work to open source code in MOT field. I really appreciate it. However I find that the tracking results with DPM is same with that with SDP in MOT17 ranking list about your tracker. Are there something wrong with the system?

I have an computer which doesn't have an valid gpu for cuda. Is it possible to test on my own data without installing cuda?

I use your model 'sst300_0712_83000.pth',test MOT17 and send the result to MOT Challenge,but I got a little lower score than you,can you tell why?This is the result bellow:

MOTA | IDF1 | MT | ML | FP | FN | ID Sw. | Frag | HZ
43.2±14.8 | 44.3 | 13.6% | 40.0% | 30,178 | 283,542 | 6,869 (138.1) | 14,822 (297.9) | 1.2 |  

Hi,I run the test_mot17.py in RTX2080Ti, and the FPS is 1.82.But in your paper,the FPS is 6.3.Can you tell me the reason and your computer configuration,thank you.

Hi Shijie,

I want to use your pre-trained MOT17 tracker on another dataset, what should I prepare?

Hi,

I have managed to install and run the network on the MOT17 dataset.
I tried submitting the results on the evaluation server (the results won't appear in the leaderboard, of course) to see if I have configured everything correctly.
But I got only 44.3 MOTA, while the paper claims a 52.4 MOTA score.
Is there some config I need to change to reproduce the paper results?

I am confused about center coordinates of boxes ,

in utils/augmentations.py/FormatBoxes , center = boxes[：,：2] + boxes[：, 2：] -1

but in tracker.py/TrackUtil/convert_detection, center = 2 * boxes[：,：2] + boxes[：, 2：] -1

can someone explain why it is like this or it's just a type error ?