Question about H-swish activation and SE layer about oqa HOT 2 OPEN

Thanks for your appreciation!

We have noticed that quantizing activation into positive numbers with HSwish is not optimal. And there are also papers like LSQ+ investigate this problem. It can be solved with an asymmetric quantization function for activation which improves the final quantization accuracy. However, some previous quantization works also follow the same setting with only positive numbers. Therefore, we did this for fair comparison.

We didn't quantize SE Layers either for the same purpose of fair comparison. We believe that it could decrease the quantization accuracy. However the SE module is lightweight and as a result, the quantization of SE modules may not affect the inference time substantially.

from oqa.

Thanks for the quick response! The clarification helps a lot. One minor follow-up, the point_linear layer in MBConv has a linear activation, which would mean that the input to the next MobileInvertedResidualBlock is likely to be distributed somewhat evenly around 0. Maybe I'm missing something, but in this case, do we still quantize it with an unsigned quantizer?

from oqa.