[DO NOT UNPIN] ORT 1.18.0 Release Candidates available for testing about onnxruntime HOT 19 CLOSED

OrtApi last entry from 1.18 is not protected by the usual static_assert

@Hopobcn @sophies927 A fix is here #20671

from onnxruntime.

@JulienTheron can you provide us a repro test case or point us to a public model that can be used to investigate further? Are you also updating TensorRT version from 8.6 to 10.0 along with the ORT version update? +@chilo-ms

Yes I've used the default TensorRT version for each ORT release, so 8.6 for 1.17.3 and 10 for 1.18.0.
I will see if I can find a public model to repro the issue with.

from onnxruntime.

Is there GPU python package for CUDA 12?

There will be for the release. We had to make some updates to make the CUDA 12 package publication a bit easier (see this PR), which is why the packages weren't included in the first round release candidates.

from onnxruntime.

Here's a follow-up to the TensorRT issue I mentioned.
I edited

The issue is now gone. I haven't quite figured why ScatterND is not placed on the TensorRT EP with TRT 10, but I guess the issue has nothing to do with the ONNX Runtime?

from onnxruntime.

Is there GPU python package for CUDA 12?

from onnxruntime.

OrtApi last entry from 1.18 is not protected by the usual static_assert

https://github.com/microsoft/onnxruntime/blob/rel-1.18.0/onnxruntime/core/session/onnxruntime_c_api.cc#L2729
https://github.com/microsoft/onnxruntime/blob/rel-1.18.0/onnxruntime/core/session/onnxruntime_c_api.cc#L2758C32-L2758C54

from onnxruntime.

OrtApi last entry from 1.18 is not protected by the usual static_assert

Thanks for catching that!

@yihonglyu would you mind looking into this?

from onnxruntime.

Hi, I just found out that ScatterND nodes are now placed on CPU. They were placed on GPU before.

See below:

2024-05-13 15:19:36.112  [16248]  DEBUG  Node placements
2024-05-13 15:19:36.112  [16248]  DEBUG   Node(s) placed on [CPUExecutionProvider]. Number of nodes: 6
2024-05-13 15:19:36.112  [16248]  DEBUG    ScatterND (/ScatterND)
2024-05-13 15:19:36.112  [16248]  DEBUG    ScatterND (/ScatterND_1)
2024-05-13 15:19:36.112  [16248]  DEBUG    ScatterND (/ScatterND_2)
2024-05-13 15:19:36.112  [16248]  DEBUG    ScatterND (/ScatterND_3)
2024-05-13 15:19:36.112  [16248]  DEBUG    ScatterND (/ScatterND_4)
2024-05-13 15:19:36.112  [16248]  DEBUG    ScatterND (/ScatterND_5)
2024-05-13 15:19:36.112  [16248]  DEBUG   Node(s) placed on [TensorrtExecutionProvider]. Number of nodes: 23

With ORT 1.17.3:

2024-05-13 15:38:07.967  [  576]  DEBUG  Node placements
2024-05-13 15:38:07.967  [  576]  DEBUG   All nodes placed on [TensorrtExecutionProvider]. Number of nodes: 1

Is it because of the upgrade to TensorRT 10?

from onnxruntime.

Hi, I just found out that ScatterND nodes are now placed on CPU. They were placed on GPU before.

See below:

2024-05-13 15:19:36.112  [16248]  DEBUG  Node placements
2024-05-13 15:19:36.112  [16248]  DEBUG   Node(s) placed on [CPUExecutionProvider]. Number of nodes: 6
2024-05-13 15:19:36.112  [16248]  DEBUG    ScatterND (/ScatterND)
2024-05-13 15:19:36.112  [16248]  DEBUG    ScatterND (/ScatterND_1)
2024-05-13 15:19:36.112  [16248]  DEBUG    ScatterND (/ScatterND_2)
2024-05-13 15:19:36.112  [16248]  DEBUG    ScatterND (/ScatterND_3)
2024-05-13 15:19:36.112  [16248]  DEBUG    ScatterND (/ScatterND_4)
2024-05-13 15:19:36.112  [16248]  DEBUG    ScatterND (/ScatterND_5)
2024-05-13 15:19:36.112  [16248]  DEBUG   Node(s) placed on [TensorrtExecutionProvider]. Number of nodes: 23

With ORT 1.17.3:

2024-05-13 15:38:07.967  [  576]  DEBUG  Node placements
2024-05-13 15:38:07.967  [  576]  DEBUG   All nodes placed on [TensorrtExecutionProvider]. Number of nodes: 1

Is it because of the upgrade to TensorRT 10?

@jywu-msft

from onnxruntime.

Hi, I just found out that ScatterND nodes are now placed on CPU. They were placed on GPU before.

See below:

2024-05-13 15:19:36.112  [16248]  DEBUG  Node placements
2024-05-13 15:19:36.112  [16248]  DEBUG   Node(s) placed on [CPUExecutionProvider]. Number of nodes: 6
2024-05-13 15:19:36.112  [16248]  DEBUG    ScatterND (/ScatterND)
2024-05-13 15:19:36.112  [16248]  DEBUG    ScatterND (/ScatterND_1)
2024-05-13 15:19:36.112  [16248]  DEBUG    ScatterND (/ScatterND_2)
2024-05-13 15:19:36.112  [16248]  DEBUG    ScatterND (/ScatterND_3)
2024-05-13 15:19:36.112  [16248]  DEBUG    ScatterND (/ScatterND_4)
2024-05-13 15:19:36.112  [16248]  DEBUG    ScatterND (/ScatterND_5)
2024-05-13 15:19:36.112  [16248]  DEBUG   Node(s) placed on [TensorrtExecutionProvider]. Number of nodes: 23

With ORT 1.17.3:

2024-05-13 15:38:07.967  [  576]  DEBUG  Node placements
2024-05-13 15:38:07.967  [  576]  DEBUG   All nodes placed on [TensorrtExecutionProvider]. Number of nodes: 1

Is it because of the upgrade to TensorRT 10?

+@chilo-ms , @yf711

from onnxruntime.

Hi, I just found out that ScatterND nodes are now placed on CPU. They were placed on GPU before.

See below:

2024-05-13 15:19:36.112  [16248]  DEBUG  Node placements
2024-05-13 15:19:36.112  [16248]  DEBUG   Node(s) placed on [CPUExecutionProvider]. Number of nodes: 6
2024-05-13 15:19:36.112  [16248]  DEBUG    ScatterND (/ScatterND)
2024-05-13 15:19:36.112  [16248]  DEBUG    ScatterND (/ScatterND_1)
2024-05-13 15:19:36.112  [16248]  DEBUG    ScatterND (/ScatterND_2)
2024-05-13 15:19:36.112  [16248]  DEBUG    ScatterND (/ScatterND_3)
2024-05-13 15:19:36.112  [16248]  DEBUG    ScatterND (/ScatterND_4)
2024-05-13 15:19:36.112  [16248]  DEBUG    ScatterND (/ScatterND_5)
2024-05-13 15:19:36.112  [16248]  DEBUG   Node(s) placed on [TensorrtExecutionProvider]. Number of nodes: 23

With ORT 1.17.3:

2024-05-13 15:38:07.967  [  576]  DEBUG  Node placements
2024-05-13 15:38:07.967  [  576]  DEBUG   All nodes placed on [TensorrtExecutionProvider]. Number of nodes: 1

Is it because of the upgrade to TensorRT 10?

@JulienTheron can you provide us a repro test case or point us to a public model that can be used to investigate further? Are you also updating TensorRT version from 8.6 to 10.0 along with the ORT version update? +@chilo-ms

from onnxruntime.

Here is a version of our model that we can share:
model.zip

To repro, just create a TensorRT session like this:

const Ort::Env env(ORT_LOGGING_LEVEL_VERBOSE);

Ort::SessionOptions options;
OrtTensorRTProviderOptionsV2* trtOptions;
Ort::GetApi().CreateTensorRTProviderOptions(&trtOptions);
options.AppendExecutionProvider_TensorRT_V2(*trtOptions);
Ort::GetApi().ReleaseTensorRTProviderOptions(trtOptions);

Ort::Session session(env, L"model.onnx", options);

Thanks.

from onnxruntime.

#20715

from onnxruntime.

20713

from onnxruntime.

#20715

Thanks for calling this out! Appears to be an issue with the model input caused by an ONNX update, rather than an issue with ORT 1.18.

from onnxruntime.

@sophies927, Regarding to #20737, shall we achieve older pypi package before release (also publish new packages to aiinfra.pkgs.visualstudio.com as a backup)? It seems that every new release will cause an older version be deleted.

from onnxruntime.

@sophies927, Regarding to #20737, shall we achieve older pypi package before release (also publish new packages to aiinfra.pkgs.visualstudio.com as a backup)? It seems that every new release will cause an older version be deleted.

Yes, I think that should definitely be the plan moving forward

from onnxruntime.

The ORT 1.18.0 release is now available here: https://github.com/microsoft/onnxruntime/releases

from onnxruntime.

@sophies927 Thank you for the release! Unfortunately, Python's onnxruntime-gpu package built for CUDA 12 is missing here https://aiinfra.visualstudio.com/PublicPackages/_artifacts/feed/onnxruntime-cuda-12 . Could you please upload it? Thank you very much.

from onnxruntime.