Thank you for your inspiring work!
I was wondering the function of 'debug' and 'postfix' in the config file.

We observe a consistent performance lag when training AdaMixer with mmcv_full==1.3.5, especially with the longer training scheme. This phenomenon may be also widespread with mmcv_full>1.3.3.

For right reproduction, please use mmcv_full==1.3.3. We are actively investigating the reason behind. More information will be updated in this issue.

Hi,
Thanks for your work, I am doing some work to improve detection performance.
I wonder if adamixer suitable for cascade-mask-rcnn?
Wishing for your reply.

Thanks for your impressive work.

Could you provide the training log with the validation mAP result for each epoch?

Thanks in advance!

发错了

Thanks for your work!
The visualization of sampling points for object is very cool.
Are there any code about it in the project? +. .+
0

When I train on my custom data, I have met this:
File "/home/zqy/compare/AdaMixer/mmdet/models/detectors/base.py", line 233, in train_step losses = self(**data) File "/home/zqy/anaconda3/envs/mmlab/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(*input, **kwargs) File "/home/zqy/anaconda3/envs/mmlab/lib/python3.7/site-packages/mmcv/runner/fp16_utils.py", line 116, in new_func return old_func(*args, **kwargs) File "/home/zqy/compare/AdaMixer/mmdet/models/detectors/base.py", line 167, in forward return self.forward_train(img, img_metas, **kwargs) File "/home/zqy/compare/AdaMixer/mmdet/models/detectors/sparse_rcnn.py", line 63, in forward_train imgs_whwh=imgs_whwh) File "/home/zqy/compare/AdaMixer/mmdet/models/roi_heads/adamixer_decoder.py", line 125, in forward_train gt_labels[i], img_metas[i]) File "/home/zqy/compare/AdaMixer/mmdet/core/bbox/assigners/hungarian_assigner.py", line 132, in assign matched_row_inds, matched_col_inds = linear_sum_assignment(cost) File "/home/zqy/anaconda3/envs/mmlab/lib/python3.7/site-packages/scipy/optimize/_lsap.py", line 104, in linear_sum_assignment return _lsap_module.calculate_assignment(cost_matrix) ValueError: matrix contains invalid numeric entries
It seems that the loss is inf, I have no idea ,can anyone help?

Hi!
I have a few questions to ask you and look forward to your answers.
Because mmdetection has been updated to 2.25.0, and adamixer needs mmdetection2.12.0, so I did the following：
1、git clone adamixer-main
2、 cd adamixer-main
3、pip install -r requirements/build.txt and pip install -v -e .
finally，I successfully install mmdetection2.12.0. （mmcv-full=1.3.3）
I just modified the path of the dataset, and the results could be show when i experimenting with "adamixer_r50_1x_coco.py", but the error(ValueError: matrix contains invalid numeric entries) was shown when experimenting with “adamixer_r50_300_query_crop_mstrain_480-800_3x_coco.py”and others.
there are no images in the dataset that do not contain any objects.
Thank you for taking the time to read this question and look forward to your answer!

D:\conda3\envs\ada\lib\site-packages\mmcv\runner\hooks\optimizer.py:31: FutureWarning: Non-finite norm encountered in torch.nn.utils.clip_grad_norm_; continuing anyway. Note that the default behavior will change in a future release to error out if a non-finite total norm is encountered. At that point, setting error_if_nonfinite=false will be required to retain the old behavior.
return clip_grad.clip_grad_norm_(params, **self.grad_clip)
Traceback (most recent call last):
File "D:/code/AdaMixer-main/tools/train.py", line 203, in
main()
File "D:/code/AdaMixer-main/tools/train.py", line 199, in main
meta=meta)
File "d:\code\adamixer-main\mmdet\apis\train.py", line 170, in train_detector
runner.run(data_loaders, cfg.workflow)
File "D:\conda3\envs\ada\lib\site-packages\mmcv\runner\epoch_based_runner.py", line 125, in run
epoch_runner(data_loaders[i], **kwargs)
File "D:\conda3\envs\ada\lib\site-packages\mmcv\runner\epoch_based_runner.py", line 50, in train
self.run_iter(data_batch, train_mode=True, **kwargs)
File "D:\conda3\envs\ada\lib\site-packages\mmcv\runner\epoch_based_runner.py", line 30, in run_iter
**kwargs)
File "D:\conda3\envs\ada\lib\site-packages\mmcv\parallel\data_parallel.py", line 67, in train_step
return self.module.train_step(*inputs[0], **kwargs[0])
File "d:\code\adamixer-main\mmdet\models\detectors\base.py", line 233, in train_step
losses = self(**data)
File "D:\conda3\envs\ada\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "D:\conda3\envs\ada\lib\site-packages\mmcv\runner\fp16_utils.py", line 95, in new_func
return old_func(*args, **kwargs)
File "d:\code\adamixer-main\mmdet\models\detectors\base.py", line 167, in forward
return self.forward_train(img, img_metas, **kwargs)
File "d:\code\adamixer-main\mmdet\models\detectors\sparse_rcnn.py", line 63, in forward_train
imgs_whwh=imgs_whwh)
File "d:\code\adamixer-main\mmdet\models\roi_heads\adamixer_decoder.py", line 125, in forward_train
gt_labels[i], img_metas[i])
File "d:\code\adamixer-main\mmdet\core\bbox\assigners\hungarian_assigner.py", line 132, in assign
matched_row_inds, matched_col_inds = linear_sum_assignment(cost)
File "D:\conda3\envs\ada\lib\site-packages\scipy\optimize_lsap.py", line 100, in linear_sum_assignment
return _lsap_module.calculate_assignment(cost_matrix)
ValueError: matrix contains invalid numeric entries

Welcome update to OpenMMLab 2.0

I am Vansin, the technical operator of OpenMMLab. In September of last year, we announced the release of OpenMMLab 2.0 at the World Artificial Intelligence Conference in Shanghai. We invite you to upgrade your algorithm library to OpenMMLab 2.0 using MMEngine, which can be used for both research and commercial purposes. If you have any questions, please feel free to join us on the OpenMMLab Discord at https://discord.gg/amFNsyUBvm or add me on WeChat (van-sin) and I will invite you to the OpenMMLab WeChat group.

Here are the OpenMMLab 2.0 repos branches:

Attention: please create a new virtual environment for OpenMMLab 2.0.

hi,this is a very good job,can you share the code of the visual sampling point?

您好，请问有纯pytorch版本吗？

Tanks for your great job and code!

    def refine_xyzr(self, xyzr, xyzr_delta, return_bbox=True):
        ##  z==(h*w).sqrt().log2()
        z = xyzr[..., 2:3]
        new_xy = xyzr[..., 0:2] + xyzr_delta[..., 0:2] * (2 ** z)
        new_zr = xyzr[..., 2:4] + xyzr_delta[..., 2:4]
        xyzr = torch.cat([new_xy, new_zr], dim=-1)
        if return_bbox:
            return xyzr, decode_box(xyzr)
        else:
            return xyzr

请问下论文中的公式4和公式5中都是 z+r 和 z-r 进行计算，为啥这里没有z±r了，直接用z进行计算？
按论文公式可以理解xyzr_delta的预测意义，但按代码中的写法该如何理解学到的xyzr_delta的意义？
按代码看，是否可以这样理解预测变量xyzr_delta的意义 xyzr_delta[..., 0:2]=((dx/dy).sqrt(), (dy/dx).sqrt()) ?

def _decode_init_proposals(self, imgs, img_metas):
      ............
      xy = 0.5 * (proposals[..., 0:2] + proposals[..., 2:4])
      wh = proposals[..., 2:4] - proposals[..., 0:2]
      z = (wh).prod(-1, keepdim=True).sqrt().log2()
      r = (wh[..., 1:2]/wh[..., 0:1]).log2()
      ..............

另外在编码 xyzr的时候，为啥 r不直接编码成 r = (wh[..., 1:2]/wh[..., 0:1]).sqrt().log2()，
这样2**(z-r)=w, 2**(z+r)=h，感觉跟论文描述直观上更一致一点，解码也更省事。

多谢 多谢！

Hello,
I have read your work AdamMixer and very interested in it. Due to resource constraints, I don't have much time to retrain Adamixer models in your paper. Is it convenient for you to release your trained models on MS-COCO dataset. Thank you!

Thanks for your work!! but I cannot find the Adamixer...

Is my mmdetetion version wrong? (mmdet=2.25.3 and mmcv-full=1.7.0)

I use the tools/analysis_tools/get_flops.py to calculate AdaMixer's FLOPs and get the following results which is not aligned with the number reported in the paper. And it seems that the SRShadowForFlops doesn't work.

  (5): AdaMixerDecoderStage(
    18.116 M, 13.462% Params, 1.812 GFLOPs, 1.762% FLOPs, 
    (loss_cls): FocalLoss(0.0 M, 0.000% Params, 0.0 GFLOPs, 0.000% FLOPs, )
    (loss_bbox): L1Loss(0.0 M, 0.000% Params, 0.0 GFLOPs, 0.000% FLOPs, )
    (fc_cls): Linear(0.021 M, 0.015% Params, 0.002 GFLOPs, 0.002% FLOPs, in_features=256, out_features=80, bias=True)
    (fc_reg): Linear(0.001 M, 0.001% Params, 0.0 GFLOPs, 0.000% FLOPs, in_features=256, out_features=4, bias=True)
    (loss_iou): GIoULoss(0.0 M, 0.000% Params, 0.0 GFLOPs, 0.000% FLOPs, )
    (attention): MultiheadAttention(
      0.0 M, 0.000% Params, 0.0 GFLOPs, 0.000% FLOPs, 
      (attn): MultiheadAttention(
        0.0 M, 0.000% Params, 0.0 GFLOPs, 0.000% FLOPs, 
        (out_proj): NonDynamicallyQuantizableLinear(0.0 M, 0.000% Params, 0.0 GFLOPs, 0.000% FLOPs, in_features=256, out_features=256, bias=True)
      )
      (proj_drop): Dropout(0.0 M, 0.000% Params, 0.0 GFLOPs, 0.000% FLOPs, p=0.0, inplace=False)
      (dropout_layer): Dropout(0.0 M, 0.000% Params, 0.0 GFLOPs, 0.000% FLOPs, p=0.0, inplace=False)
    )
    (attention_norm): LayerNorm(0.001 M, 0.000% Params, 0.0 GFLOPs, 0.000% FLOPs, (256,), eps=1e-05, elementwise_affine=True)
    (instance_interactive_conv_dropout): Dropout(0.0 M, 0.000% Params, 0.0 GFLOPs, 0.000% FLOPs, p=0.0, inplace=False)
    (instance_interactive_conv_norm): LayerNorm(0.001 M, 0.000% Params, 0.0 GFLOPs, 0.000% FLOPs, (256,), eps=1e-05, elementwise_affine=True)
    (ffn): FFN(
      1.051 M, 0.781% Params, 0.105 GFLOPs, 0.102% FLOPs, 
      (activate): ReLU(0.0 M, 0.000% Params, 0.0 GFLOPs, 0.000% FLOPs, inplace=True)
      (layers): Sequential(
        1.051 M, 0.781% Params, 0.105 GFLOPs, 0.102% FLOPs, 
        (0): Sequential(
          0.526 M, 0.391% Params, 0.053 GFLOPs, 0.051% FLOPs, 
          (0): Linear(0.526 M, 0.391% Params, 0.052 GFLOPs, 0.051% FLOPs, in_features=256, out_features=2048, bias=True)
          (1): ReLU(0.0 M, 0.000% Params, 0.0 GFLOPs, 0.000% FLOPs, inplace=True)
          (2): Dropout(0.0 M, 0.000% Params, 0.0 GFLOPs, 0.000% FLOPs, p=0.0, inplace=False)
        )
        (1): Linear(0.525 M, 0.390% Params, 0.052 GFLOPs, 0.051% FLOPs, in_features=2048, out_features=256, bias=True)
        (2): Dropout(0.0 M, 0.000% Params, 0.0 GFLOPs, 0.000% FLOPs, p=0.0, inplace=False)
      )
      (dropout_layer): Identity(0.0 M, 0.000% Params, 0.0 GFLOPs, 0.000% FLOPs, )
    )
    (ffn_norm): LayerNorm(0.001 M, 0.000% Params, 0.0 GFLOPs, 0.000% FLOPs, (256,), eps=1e-05, elementwise_affine=True)
    (cls_fcs): ModuleList(
      0.066 M, 0.049% Params, 0.007 GFLOPs, 0.006% FLOPs, 
      (0): Linear(0.066 M, 0.049% Params, 0.007 GFLOPs, 0.006% FLOPs, in_features=256, out_features=256, bias=True)
      (1): LayerNorm(0.001 M, 0.000% Params, 0.0 GFLOPs, 0.000% FLOPs, (256,), eps=1e-05, elementwise_affine=True)
      (2): ReLU(0.0 M, 0.000% Params, 0.0 GFLOPs, 0.000% FLOPs, inplace=True)
    )
    (reg_fcs): ModuleList(
      0.066 M, 0.049% Params, 0.007 GFLOPs, 0.006% FLOPs, 
      (0): Linear(0.066 M, 0.049% Params, 0.007 GFLOPs, 0.006% FLOPs, in_features=256, out_features=256, bias=True)
      (1): LayerNorm(0.001 M, 0.000% Params, 0.0 GFLOPs, 0.000% FLOPs, (256,), eps=1e-05, elementwise_affine=True)
      (2): ReLU(0.0 M, 0.000% Params, 0.0 GFLOPs, 0.000% FLOPs, inplace=True)
    )
    (sampling_n_mixing): AdaptiveSamplingMixing(
      16.909 M, 12.566% Params, 1.692 GFLOPs, 1.644% FLOPs, 
      (sampling_offset_generator): Sequential(
        0.099 M, 0.073% Params, 0.01 GFLOPs, 0.010% FLOPs, 
        (0): Linear(0.099 M, 0.073% Params, 0.01 GFLOPs, 0.010% FLOPs, in_features=256, out_features=384, bias=True)
      )
      (norm): LayerNorm(0.001 M, 0.000% Params, 0.0 GFLOPs, 0.000% FLOPs, (256,), eps=1e-05, elementwise_affine=True)
      (adaptive_mixing): AdaptiveMixing(
        16.81 M, 12.492% Params, 1.682 GFLOPs, 1.635% FLOPs, 
        (parameter_generator): Sequential(
          8.421 M, 6.258% Params, 0.839 GFLOPs, 0.815% FLOPs, 
          (0): Linear(8.421 M, 6.258% Params, 0.839 GFLOPs, 0.815% FLOPs, in_features=256, out_features=32768, bias=True)
        )
        (out_proj): Linear(8.389 M, 6.234% Params, 0.839 GFLOPs, 0.815% FLOPs, in_features=32768, out_features=256, bias=True)
        (act): ReLU(0.0 M, 0.000% Params, 0.004 GFLOPs, 0.004% FLOPs, inplace=True)
        (shadow): SRShadowForFlops(0.0 M, 0.000% Params, 0.0 GFLOPs, 0.000% FLOPs, )
      )
    )
  )
  init_cfg=[{'type': 'Normal', 'std': 0.01, 'override': {'name': 'fc_cls'}}, {'type': 'Normal', 'std': 0.001, 'override': {'name': 'fc_reg'}}]
)

/==============================
Input shape: (3, 1280, 800)
Flops: 102.88 GFLOPs
Params: 134.57 M
/==============================

Could you tell me how to get the exact FLOPs number?

In the code 'mmdet/models/roi_heads/bbox_heads/adaptive_mixing_operator.py'
[line69] self.m_parameters = self.eff_in_dim * self.eff_out_dim
[line116-117] M, S = params.split([self.m_parameters, self.s_parameters], 2)
M got a dim [..., self.eff_in_dim * self.eff_out_dim].

However, M is reshaped to [..., self.eff_in_dim, self.eff_in_dim] in
[line140-141] M = M.reshape(B*N, G, self.eff_in_dim, self.eff_in_dim)

Is that a bug or feature?

Thanks for your great works!

I have a question about the design of positional vector (x, y, z, r). In most of DETR-like detectors, the positional vector are learnable, such as DAB-DETR, Anchor-DETR. In your design, you detach it from computation graph. Dose learnable positional vector hurt the performance?

This is important to me and thank you very much for your answer！

What is the function of the weight for the j-th feature map wj

resnet50，同等条件下，能刷的过dn-deformable DETR 吗

Thanks for your impressive work!

It seems that you don't use residual connection here and use content query with positional embedding as values. Do you find it helpful for performance？

I don't know if it is a problem with my environment, I just changed the dataset path to my own, but the error is reported. My torch==1.3.1.
error:
File "/home/ubuntu/.conda/envs/mmlab/lib/python3.7/site-packages/torch/nn/functional.py", line 3352, in multi_head_attention_forward attn_output_weights += attn_mask
RuntimeError: output with shape [16, 300, 300] doesn't match the broadcast shape [1, 16, 300, 300]

In file mmdet/models/roi_heads/bbox_heads/adamixer_decoder_stage.py line 353， avg_factor = reduce_mean(num_pos) should be modified as 'avg_factor = max(reduce_mean(num_pos), 1)' to support training pure background images.

Hello author! When I'm training, work_dirs will generate all the configs and mmdet files, is this necessary? If it is not necessary, how can I turn off this setting?

Hi I am running the training code and I meet the following error; I am using pytorch1.3.1 torchvision0.4.2 cuda10.1

File "AdaMixer/mmdet/models/roi_heads/adamixer_decoder.py", line 60, in _bbox_forward
featmap_strides=self.featmap_strides)
File "/home/python3.7/site-packages/torch/nn/modules/module.py", line 541, in call
result = self.forward(*input, **kwargs)
File "/home/python3.7/site-packages/mmcv/runner/fp16_utils.py", line 95, in new_func
return old_func(*args, **kwargs)
File "AdaMixer/mmdet/models/roi_heads/bbox_heads/adamixer_decoder_stage.py", line 302, in forward
attn_mask=attn_bias,
File "/home/python3.7/site-packages/torch/nn/modules/module.py", line 541, in call
result = self.forward(*input, **kwargs)
File "/home/python3.7/site-packages/mmcv/cnn/bricks/transformer.py", line 137, in forward
key_padding_mask=key_padding_mask)[0]
File "/home/python3.7/site-packages/torch/nn/modules/module.py", line 541, in call
result = self.forward(*input, **kwargs)
File "/home/python3.7/site-packages/torch/nn/modules/activation.py", line 783, in forward
attn_mask=attn_mask)
File "/home/python3.7/site-packages/torch/nn/functional.py", line 3352, in multi_head_attention_forward
attn_output_weights += attn_mask
RuntimeError: output with shape [16, 100, 100] doesn't match the broadcast shape [1, 16, 100, 100]

Looking forward to your reply and many thanks. And can you please specify your pytorch and cuda version?