Does this framework support "what you serve is what you train" for weight only quantization? about aqt HOT 5 CLOSED

Yes, in the config, you can select it individually.

from aqt.

Thanks for the speedy reply. Would you mind pointing me to the config field that toggles this?

from aqt.

https://github.com/google/aqt/blob/main/aqt/common/aqt_config.py#L362

As you see lhs and rhs have completly separate quantization configs.

from aqt.

Yes, but from what i see in the underlying dot product code, if activation is not quantized, a float * float dot product is used? https://github.com/google/aqt/blob/main/aqt/jax/aqt_dot_general.py#L91 Is this a fake quantization or is this also the arithmetic in serving time?

from aqt.

Sorry that I missed your comment.
For the dot product to be accelerated, both sides need to have the same type.
You are probably most concerned about weight loading.
We will implement that in AQTv2 within a month.

from aqt.