why the speed so slow compare with maskformer first version? about mask2former HOT 3 CLOSED

The decrease in speed is mainly due to using MSDeformAttn as the pixel decoder (compared to multi-scale features and masked attention). Using BiFPN as the pixel decoder can improve speed by (approximately) 25% and using FPN can probably improve it by 40%.

Mask2Former is highly modularized (you can make independent change to backbone, pixel decoder or transformer decoder), if you care about speed, I would suggest making changes in the following order:

1. pixel decoder (using more efficient FPN instantiation)
2. backbone (using more efficient backbone design)
3. Transformer decoder (reduce the number of decoder layers, using single-scale low-resolution features)

Masked attention does not increase computation much and it is the most important component, so I would not suggest removing it to increase speed.

from mask2former.

@bowenc0221 thanks, that's helpful. you mean if remove the origin pixel decoder, which is maskatten based, althought speed up, but accuracymight drop a lot?

from mask2former.

No, the Masked Attention is only used to replace Cross Attention in the Transformer decoder (it does not matter what pixel decoder you use). I mean compared to cross-attention, masked attention is not much slower.

from mask2former.