Quantized ONNX Model Still Has Float32 Input/Output Tensors about onnxruntime HOT 4 OPEN

This is the QDQ representation of ONNX model. In order to perform integer-arithmetic only, you have to quantize your model to QOperator representation. For mor detail, follow https://onnxruntime.ai/docs/performance/model-optimizations/quantization.html.
I'm curious what is the method you used to quantized your model? Is it Post-quantize training in ONNXRuntime?
If yes, you just have to change quant_format from QDQ to QOperator

def quantize_static(
    model_input: Union[str, Path, onnx.ModelProto],
    model_output: Union[str, Path],
    calibration_data_reader: CalibrationDataReader,
    quant_format=QuantFormat.QDQ, # Change this to QuantFormat.QOperator
    op_types_to_quantize=None,
    per_channel=False,
    reduce_range=False,
    activation_type=QuantType.QInt8,
    weight_type=QuantType.QInt8,
    nodes_to_quantize=None,
    nodes_to_exclude=None,
    use_external_data_format=False,
    calibrate_method=CalibrationMethod.MinMax,
    extra_options=None,
):
...

from onnxruntime.

@hoangtv2000 Thank you for the comment!
The following are the details for my quantization strategy

• Post-training quantization: Yes
• Method selection: Static
• Representation format: QDQ
• Data type selection: Activations: uint8, Weights: int8 (U8S8)

I tried your suggestion to perform quantization using QOperator. However, the quantized model's input and output remain float32.
I also tried different data types such as U8U8, S8S8, and U8S8, but the results were almost identical.
Although the quantized model with float32 input/output runs 2-3x as fast as the non-quantized model, I still do not understand why the quantized output is not in int8 format.

Do you have any idea regarding this?

from onnxruntime.

This issue has been automatically marked as stale due to inactivity and will be closed in 30 days if no further activity occurs. If further support is needed, please provide an update and/or more details.

from onnxruntime.

I'm facing the same problem, have you solved that?

from onnxruntime.