Comments (5)
(1) Cuda graph requires inputs binded to a fixed buffer in GPU (so you will need copy input to the same address in GPU memory for every run). In your case, the input is binded to CPU:
Ort::MemoryInfo memory_info = Ort::MemoryInfo::CreateCpu(OrtDeviceAllocator, OrtMemTypeCPU);
(2)
Memory is deleted before reading the data to a vector:
cudaFree(output_data_ptr); //suhail
...
std::vector output(rawOutput, rawOutput + count);
Please follow the following examples to use IO/Binding and CUDA Graph:
from onnxruntime.
Where do you populate input/output names?
from onnxruntime.
Hi yuslepukhin,
please check below snippet. i got the input and output name from there
Ort::Session session(env, model_path.c_str(), session_options);
// Create an allocator object based on default options to provide memory allocation functions for subsequent operations
Ort::AllocatorWithDefaultOptions allocator;
// Get the number of input nodes
size_t num_input_nodes = session.GetInputCount();
// Get the number of output nodes
size_t num_output_nodes = session.GetOutputCount();
// Get input node name and dimensions
for (int i = 0; i < num_input_nodes; i++) {
auto input_name = session.GetInputNameAllocated(i, allocator);
input_node_names.push_back(input_name.get());
Ort::TypeInfo input_typeinfo = session.GetInputTypeInfo(i);
auto input_tensorinfo = input_typeinfo.GetTensorTypeAndShapeInfo();
auto input_dims = input_tensorinfo.GetShape();
ONNXTensorElementDataType inputType = input_tensorinfo.GetElementType();
if (input_dims.at(0) == IMR_ERROR)
{
std::cout << "[Warning] Got dynamic batch size. Setting output batch size to "
<< BATCH_SIZE << "." << std::endl;
input_dims.at(0) = BATCH_SIZE;
}
input_node_dims.push_back(input_dims);
std::cout << "[INFO] Input name and shape is: " << input_name.get() << " [";
for (size_t j = 0; j < input_dims.size(); j++) {
std::cout << input_dims[j];
if (j != input_dims.size()-1) {
std::cout << ",";
}
}
std::cout << ']' << std::endl;
}
// Get output node name
std::vector <vector <int64_t>> output_node_dims;
for (int i = 0; i < num_output_nodes; i++) {
auto output_name = session.GetOutputNameAllocated(i, allocator);
output_node_names.push_back(output_name.get());
Ort::TypeInfo output_typeinfo = session.GetOutputTypeInfo(i);
auto output_tensorinfo = output_typeinfo.GetTensorTypeAndShapeInfo();
auto output_dims = output_tensorinfo.GetShape();
if (output_dims.at(0) == IMR_ERROR)
{
std::cout << "[Warning] Got dynamic batch size. Setting output batch size to "
<< BATCH_SIZE << "." << std::endl;
output_dims.at(0) = BATCH_SIZE;
}
output_node_dims.push_back(output_dims);
std::cout << "[INFO] Output name and shape is: " << output_name.get() << " [";
for (size_t j = 0; j < output_dims.size(); j++) {
std::cout << output_dims[j];
if (j != output_dims.size()-1) {
std::cout << ",";
}
}
std::cout << ']' << std::endl;
}
from onnxruntime.
I refer the code from git. due to my project needs CUDA performance i need to add I/O Binding. so i gone through the reference from the [https://onnxruntime.ai/docs/performance/tune-performance/iobinding.html] ONNX Runtime site.
is there anything i miss from the I/O Binding logic?
from onnxruntime.
Hi Tianlei Wu,
Thank u so much. I resolved my issue with the reference which you shared.
from onnxruntime.
Related Issues (20)
- Issue with performing shape inference using symbolic_shape_infer.py with Phi-3 ONNX Models HOT 4
- [Performance] Mapfile support for certain external data files is not working HOT 2
- [Documentation] Setup the CUDA Environment is not detailed enough HOT 2
- [Documentation] phi-3 vision tutorial lacks samples for languages that are actually used for desktop development. HOT 1
- [Build] Build python interface for Onnxruntime-qnn on aarch64 Linux HOT 11
- [Build] ‘struct onnxruntime::ProviderHostCPU’ has no member named ‘UpsampleBase__AdjustOutputSizeAsPolicy’ when CONTRIB ops are disabled. HOT 15
- Initialization crash using OnnxRuntime 17.0 (previously working on 16.3)
- [Web] `Error: using ceil() in shape computation is not yet supported for AveragePool` HOT 4
- ORT 1.18 crashes on exit after using Cuda EP to run inference on a specific model HOT 2
- [Transformers Optimizer] CLIP-ViT encoder attention not getting fused HOT 4
- [Documentation] How Configure CUDA 12.* and cuDNN for GPU with ONNX Runtime and C# on Windows 11 HOT 6
- Onnx Model run failed in a loop HOT 2
- [Mobile] QNN HTP Backend Setup on Android Device HOT 8
- onnxruntime-gpu not working with my gpu / setup HOT 9
- onnxruntime::python::CreateExecutionProviderInstance CUDA_PATH is set but CUDA wasnt able to be loaded. HOT 2
- [TensorRT EP] OOM (RAM) when loading ONNX model HOT 7
- Inference result different between cuda and cpu HOT 2
- [Mobile] QNN failed to finalize QNN graph for attention layer
- SIGSEGV on CoreMLExecutionProvider when using dynamic batch
- [Mobile] Pixel 7a (Google edge tpu) NNAPI runtime exception unordered_map::at: key not found when setting USE_FP16 HOT 10
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from onnxruntime.