Comments (4)
This is the QDQ representation of ONNX model. In order to perform integer-arithmetic only, you have to quantize your model to QOperator representation. For mor detail, follow https://onnxruntime.ai/docs/performance/model-optimizations/quantization.html.
I'm curious what is the method you used to quantized your model? Is it Post-quantize training in ONNXRuntime?
If yes, you just have to change quant_format from QDQ to QOperator
def quantize_static(
model_input: Union[str, Path, onnx.ModelProto],
model_output: Union[str, Path],
calibration_data_reader: CalibrationDataReader,
quant_format=QuantFormat.QDQ, # Change this to QuantFormat.QOperator
op_types_to_quantize=None,
per_channel=False,
reduce_range=False,
activation_type=QuantType.QInt8,
weight_type=QuantType.QInt8,
nodes_to_quantize=None,
nodes_to_exclude=None,
use_external_data_format=False,
calibrate_method=CalibrationMethod.MinMax,
extra_options=None,
):
...
from onnxruntime.
@hoangtv2000 Thank you for the comment!
The following are the details for my quantization strategy
- Post-training quantization: Yes
- Method selection: Static
- Representation format: QDQ
- Data type selection: Activations: uint8, Weights: int8 (U8S8)
I tried your suggestion to perform quantization using QOperator. However, the quantized model's input and output remain float32.
I also tried different data types such as U8U8, S8S8, and U8S8, but the results were almost identical.
Although the quantized model with float32 input/output runs 2-3x as fast as the non-quantized model, I still do not understand why the quantized output is not in int8 format.
Do you have any idea regarding this?
from onnxruntime.
This issue has been automatically marked as stale due to inactivity and will be closed in 30 days if no further activity occurs. If further support is needed, please provide an update and/or more details.
from onnxruntime.
I'm facing the same problem, have you solved that?
from onnxruntime.
Related Issues (20)
- [Build] cuda dockerfile build error HOT 3
- [Mobile] QNN-EP graph preparation failed
- DirectML failed with invalid command
- [Web] [SD 1.5][SD Turbo][Whipser Base] Browser page out of memory - WebGPU and WebNN GPU EP HOT 3
- [Web] Previously used .wasm binaries missing in 1.19.0 (using web bundler with CopyPlugin)
- [Mobile][Android][Bug] SIGSEGV if ortSession.close() is called before ortSession.run() finish HOT 4
- [Bug] [onnxruntime-node] Error: no available backend found. ERR: [wasm] backend not found. HOT 2
- [Documentation] the example does not define test_data variable
- Cant get GPU to work with ONNX Runtime 1.19 Cuda 12.6 CuDNN 9 RTXA4000 HOT 6
- Release version of onnxruntime.dll has a dependency on dbghelp.dll HOT 3
- [Training] No Op registered for SimplifiedLayerNormalization with domain_version of 14 when using generate_artifatcs HOT 2
- [Build] Use Protobuf External Problem
- [Build] HOT 1
- [QNN] Initialization failure in particular use case. HOT 1
- [Mobile] Will provide supported 16KB-page-size prebuilt Android .so in the future?
- [Bug] [ONNXRuntimeError] : 7 : INVALID_PROTOBUF : Load model from yolov8n.enginefailed:Protobuf parsing failed.
- [Performance] Why does genai run 2x as fast as vanilla managed onnxruntime? HOT 1
- [Mobile] Inference error using QNN on Android phone. HOT 1
- [Build] CUDA Provider Loading Failure: ONNX Runtime 1.19.0 incompatible with CUDA 12.6 (cublas64_11.dll missing) HOT 3
- [Feature Request] EP context cache options don't seem to work with onnxruntime perf test
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from onnxruntime.