why not using size (n_pre, n_cur) as the input to the Affinity Net? about sst HOT 15 CLOSED

Thank you for your suggestions. It will be great to output features of selector as (n_obj, 520). It does save time for training. However, as far as I know, the input size of FCN cannot get changed. If not, could you provide me with some examples?

from sst.

Thanks for your reply. Actually I am not that familiar with Pytorch. I will try and report the results.

from sst.

Look forward to your good results :) .

from sst.

Thank you for your suggestions. It will be great to output features of selector as (n_obj, 520). It does save time for training. However, as far as I know, the input size of FCN cannot get changed. If not, could you provide me with some examples?

In semantic segmentation, it seems that the FCN network could deal with any size input image.

from sst.

Thank you for your reply, @SpyderXu . Now, I understand that. @nejyeah , Your idea is really great. I think this network can be greatly improved.

from sst.

Actually, the feature extraction process can be put in the detector (such as SSD), which will also improve this framework. I need to do that.

from sst.

Hi, the problem of varying size seems to locate at the trainning batch. In a Batch, we need to make the features to have the same size. The training loss may be modified to discard the padding value when using a fixed input size. And then the inference can take advantage of the varying size for just one frame each time.

from sst.

In the training process, the row and col of the labels have been shuffled so as the input feature(1040 x 80 x 80) before feeding into the compression network. Because the compression network just consists of 1x1 conv, the shuffle of the order(80 x 80) seems to be useless.

I am a beginner to the tracking. So I am grateful to you for sharing the nice project. I have learned a lot from it.

from sst.

Actually, the feature extraction process can be put in the detector (such as SSD), which will also improve this framework. I need to do that.

Good idea! For practical, the detect net and MOT net should be merged into one.

from sst.

In the training process, the row and col of the labels have been shuffled so as the input feature(1040 x 80 x 80) before feeding into the compression network. Because the compression network just consists of 1x1 conv, the shuffle of the order(80 x 80) seems to be useless.

Yes. as you say, the shuffle of the order is useless. @nejyeah . Thank you for your appreciation of my work. It's also my first work on tracking. Thank you for your suggestions again.

from sst.

Hi, the problem of varying size seems to locate at the trainning batch. In a Batch, we need to make the features to have the same size. The training loss may be modified to discard the padding value when using a fixed input size. And then the inference can take advantage of the varying size for just one frame each time.

I add dummy box into the MOTTrainDataset. So it seems that you need start from here and it should be lots of work. The training loss uses mask in order to remove the negative affect of dummy box. All the work is based on my first thinking which is not that perfect.

from sst.

I have tested the varying size for inference on GTX1080Ti for MOT17-01-FRCNN. For the original version, the inference time is about 0.266 s/frame. For the varying size, the inference time is about 0.243. And It saves 20 ms. And the results seems to make little difference from eyes based on the output vedio.

from sst.

That's great. By the way, different input boxes may take different time. Is this the average time. If possible, could you push a branch to this repository?

from sst.

Yes, it is the average time in your code for tracking test.
I just modified two functions in your code.
Modify layers/sst.py

126     def forward_stacker_features(self, xp, xn, fill_up_column=True):
127         pre_rest_num = self.max_object - xp.shape[1]
128         next_rest_num = self.max_object - xn.shape[1]
129         pre_num = xp.shape[1]
130         next_num = xn.shape[1]
131         ### x = self.forward_stacker2(
132         ###     self.resize_dim(xp, pre_rest_num, dim=1),
133         ###     self.resize_dim(xn, next_rest_num, dim=1)
134         ### )
            x = self.forward_stacker2_inference(xp, xn)
135       
136         x = self.final_dp(x)
137         # [B, N, N, 1]
138         x = self.forward_final(x, self.final_net)
139         x = x.contiguous()
140         # add zero
141         ### if next_num < self.max_object:
142         ###     x[0, 0, :, next_num:] = 0
143         ### if pre_num < self.max_object:
144         ###     x[0, 0, pre_num:, :] = 0
145         x = x[0, 0, :]
146         # add false unmatched row and column
147         x = self.resize_dim(x, 1, dim=0, constant=self.false_constant)
148         x = self.resize_dim(x, 1, dim=1, constant=self.false_constant)
149 
150         x_f = F.softmax(x, dim=1)
151         x_t = F.softmax(x, dim=0)
152         # slice
153         last_row, last_col = x_f.shape
154         row_slice = list(range(pre_num)) + [last_row-1]
155         col_slice = list(range(next_num)) + [last_col-1]
156         ### x_f = x_f[row_slice, :]
157         ### x_f = x_f[:, col_slice]
158         ### x_t = x_t[row_slice, :]
159         ### x_t = x_t[:, col_slice]

Add function forward_stacker2_inference(), copy and revised from forward_statcker2 for not affecting training

223     def forward_stacker2_inference(self, stacker1_pre_output, stacker1_next_output):
224         ### stacker1_pre_output = stacker1_pre_output.unsqueeze(2).repeat(1, 1, self.max_object, 1).permute(0, 3, 1, 2)
225         ### stacker1_next_output = stacker1_next_output.unsqueeze(1).repeat(1, self.max_object, 1, 1).permute(0, 3, 1, 2)
            n_pre = stacker1_pre_output.shape[1]
            n_next = stacker1_next_output.shape[1]
            stacker1_pre_output = stacker1_pre_output.unsqueeze(2).repeat(1, 1, n_next, 1).permute(0, 3, 1, 2)
            stacker1_next_output = stacker1_next_output.unsqueeze(1).repeat(1, n_pre, 1, 1).permute(0, 3, 1, 2)
226 
227         stacker1_pre_output = self.stacker2_bn(stacker1_pre_output.contiguous()    )
228         stacker1_next_output = self.stacker2_bn(stacker1_next_output.contiguous    ())
229 
230         output = torch.cat(
231             [stacker1_pre_output, stacker1_next_output],
232             1
233         )
234 
235         return output

from sst.

Great, Thank you.

from sst.