Comments (7)
Hi @MiroPsota Thanks for bringing up this issue!
Does the model with NMS op could run on previous version of ONNXRuntime+TRT ?
Also could you share the standard model (without nms) that you tested on CPU/GPU/OpenVINO EP?
from onnxruntime.
Zip with all the models and updated run.py
.
1.18.0 and 1.18.1 - the mentioned problem occurs.
I used TensorRT 8.6.1.6 from here for 1.17.1 tests (ORT gpu pypi package).
OOM doesn't occur, but another error occurs (op not implemented) and it adds not wanted copy operations to a host and back. See the log. I will investigate further.
The ONNX model for TensorRT can be run without problems with mmdeploy, which uses TensorRT directly. One difference is that TRTBatchedNMS
had originally a different domain, which I changed to trt.plugins
according to docs, so it can be run in ORT.
from onnxruntime.
I think the problem is with TensorRT 10.0, probably the same as mentioned here.
I solved it with compiling ORT 1.18.1 with TRT 8.6.1 and --use_tensorrt_oss_parser
. More info here.
from onnxruntime.
What your tested with following combinations are expected:
- ORT 1.18.1 + TRT 10.0 (use built-in parser by default) ---> OOM
- ORT 1.17.1 + TRT 8.6 (use built-in parser by default) ---> TRTBatchedNMS can't be recognized by TRT
- ORT 1.18.1 + TRT 8.6 + TRT OSS parser ---> Can successfully run the inference
For TRT 10.0, i tested with trtexec and got following error, but i didn't see the error with TRT 8.6 trtexec:
(It seems TRT 10 has some issues and i'm reporting this to Nvidia. )
...
[07/09/2024-17:06:31] [I] [TRT] ----------------------------------------------------------------
[07/09/2024-17:06:31] [I] [TRT] No checker registered for op: TRTBatchedNMS. Attempting to check as plugin.
[07/09/2024-17:06:31] [I] [TRT] No importer registered for op: TRTBatchedNMS. Attempting to import as plugin.
[07/09/2024-17:06:31] [I] [TRT] Searching for plugin: TRTBatchedNMS, plugin_version: 1, plugin_namespace:
[07/09/2024-17:06:31] [I] [TRT] Successfully created plugin: TRTBatchedNMS
[07/09/2024-17:06:31] [W] [TRT] IElementWiseLayer with inputs /TRTBatchedNMS_output_2 and /Mul_output_0: first input has type Int32 but second input has type Int64.
[07/09/2024-17:06:31] [E] Error[4]: ITensor::getDimensions: Error Code 4: Internal Error (/Add_2: IElementWiseLayer with SUM operation has incompatible input types Int32 and Int64 type.)
[07/09/2024-17:06:31] [E] [TRT] ModelImporter.cpp:949: While parsing node number 434 [Add -> "/Add_2_output_0"]:
[07/09/2024-17:06:31] [E] [TRT] ModelImporter.cpp:950: --- Begin node ---
input: "/TRTBatchedNMS_output_2"
input: "/Mul_output_0"
output: "/Add_2_output_0"
name: "/Add_2"
op_type: "Add"
[07/09/2024-17:06:31] [E] [TRT] ModelImporter.cpp:951: --- End node ---
[07/09/2024-17:06:31] [E] [TRT] ModelImporter.cpp:954: ERROR: ModelImporter.cpp:195 In function parseNode:
[6] Invalid Node - /Add_2
ITensor::getDimensions: Error Code 4: Internal Error (/Add_2: IElementWiseLayer with SUM operation has incompatible input types Int32 and Int64 type.)
...
As for TRT 8.6, user can only use ORT TRT with TRT OSS parser to successfully register and run the customed TRT plugins (in this case, the TRTBatchedNMS op). But for TRT 10, ORT + both built-in parser and OSS parser can successfully register the customed TRT plugins.
from onnxruntime.
it adds not wanted copy operations to a host and back. See the log. I will investigate further.
The copy ops being added is expected. It's because, TRTBatchedNMS op is not recognized by TRT and ORT will try to put TRTBatchedNMS op on other EPs which will result in the copy ops being added. But still, no other EPs can run TRTBatchedNMS ops, so it end up with the error you saw
from onnxruntime.
The OOM you saw with TRT 10 is because TRT EP is calling TRT parser to check the TRT eligible nodes in the model where onnxruntime::TensorrtExecutionProvider::GetSupportedList() is being called many times recursively and hit OOM.
I'm also checking with Nvidia regarding this.
from onnxruntime.
Nvidia helped identify the issue.
TRT 10.0 added support for INT64 types, whereas TensorRT 8.6 treated all INT64-typed tensors as INT32. The TRTBatchedNMS plugin internally always produces INT32 output tensors, even though in the ONNX graph they’re marked as INT64:
This causes the build issues in standalone TensorRT 10, as the subsequent operations (i.e. the Add) assumes INT64-typed operands when in reality there is a INT32 + INT64 mix. This issue doesn’t exist in TensorRT 8.6 as everything is casted to INT32, and the types match.
Update the TRTBatchedNMS plugin ( inlibmmdeploy_op.so) to produce proper INT64 output tensors should fix the issue.
from onnxruntime.
Related Issues (20)
- QDQ removal around Resize (mode=linear) causes wrong numeric values
- [Build] RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running MatMul node. Name:'/MatMul_7' Status Message: /onnxruntime_src/onnxruntime/core/framework/op_kernel.cc:83 virtual OrtValue* onnxruntime::OpKernelContext::OutputMLValue(int, const onnxruntime::TensorShape&) status.IsOK() was false. Shape mismatch attempting to re-use buffer. {1,1,512} != {1,32,512}. Validate usage of dim_value (values should be > 0) and dim_param (all values with the same string should equate to the same size) in shapes in the model. HOT 7
- Not able to load onnx model multilingual-e5-large HOT 3
- [Crash] Crash while loading AlibabaNLP/gte-base ONNX model HOT 5
- Model saved with offline basic optimizations will not load - ShapeInferenceError HOT 1
- [Training] [ShapeInferenceError] Dimension could not be inferred: incompatible shapes
- [Build] How can I quantize the llama3 model activation to int4 ?
- [Feature Request] ORT-Profiler: Include timestamps for tensor allocations and deallocations. HOT 2
- header files path not recognized or unable to read header file HOT 1
- [Build] AllocatorTest.CUDAAllocatorFallbackTest failed HOT 1
- [Performance] Get nan value when I block all the node in fp16 conversion HOT 16
- [Bug] The per_tensor quantized weight type of matmul is wrong HOT 1
- ONNX Runtime 1.18.1 CUDA 12.4 cuDNN 9.2 breaks inference with repeated inputs when enable_mem_reuse is enabled
- Latest Release(1.18.1) Java Artifacts Unavailable HOT 1
- [Build] C++ API cannot be reliably linked with an program using CMake
- [BUG] CANN: onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError]
- [Build] Cross compilation of the library for ARMv7 32bit target with gcc 8.3 HOT 4
- CUDA 12 and session.get_providers() not showing CUDAExecutionProvider HOT 9
- [Web] Memory access out of bounds / alignment fault
- An error occurred when I installed onnxruntime-qnn in an Arm environment HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from onnxruntime.