Comments (10)
Yes, that's the correct place to specify the feature flags to enable the execution providers in Ort, however the correct libonnxruntime also needs to be built to support the execution providers you provide. Currently it's only cuda
and tensorrt
because those are available in the precompiled library that is downloaded from Microsoft's releases.
Long term I'd like to have our own precompiled libonnxruntime(s) with more EPs built out, then pull those for the download rather than the Microsoft releases. For now, you'd need to compile onnxruntime yourself and then point to that with ORT_LIB_LOCATION
in addition to adding the feature flag in config.exs
to enable it for Ort.
from ortex.
Ohh I see, I have read ort readme where it says so.
Another question, how can I make ortex to use my nvida gpu? I think it is not using it (nvtop doesn't says the opossite)
I'm running over docker with nvidia-container-runtime, inside the container I see that it is correctly detected the card (with nvidia-smi)
from ortex.
Without knowing the entire setup I can't say for certain why it isn't using your GPU. Are you passing the cuda
option through as the first EP when the model is loaded?
model = Ortex.load("./path/to/model", [:cuda])
If you are and it's still not utilizing the GPU, ensure LD_LIBRARY_PATH
has your cuda version linked and that you have the correct version of cuda and cudnn installed for your driver version.
from ortex.
@mortont it seems that the EP are not enabled in ort
, this is why it's not using the GPU.
I have a Jetson Nano Orin, I compiled onnxruntime
with support for cuda
and tensorrt
. After loading the model I have this output.
Ortex.load("model.onnx", [:tensorrt, :cuda])
The warnings
2023-07-19T19:58:28.807552Z DEBUG ort::environment: Environment not yet initialized, creating a new one
2023-07-19T19:58:28.850619Z DEBUG ort::environment: Environment created env_ptr="0xffff1042c910"
2023-07-19T19:58:28.851381Z WARN apply_execution_providers: ort::execution_providers: `ort` was not compiled to enable TensorrtExecutionProvider - you may be missing a Cargo feature to enable it.
2023-07-19T19:58:28.851403Z WARN apply_execution_providers: ort::execution_providers: `ort` was not compiled to enable CUDAExecutionProvider - you may be missing a Cargo feature to enable it.
2023-07-19T19:58:28.851695Z DEBUG ort: Flush-to-zero and denormal-as-zero are off 2023-07-19T19:58:28.851729Z DEBUG ort: Creating and using per session threadpools since use_per_session_threads_ is true
2023-07-19T19:58:28.851754Z DEBUG ort: Dynamic block base set to 0
...
After enabling: cuda
and tensorrt
in default features in Cargo
[features]
default = ["download-binaries", "cuda", "tensorrt"]
Now everything seems good
2023-07-19T20:03:58.617038Z DEBUG ort::environment: Environment not yet initialized, creating a new one
2023-07-19T20:03:58.654983Z DEBUG ort::environment: Environment created env_ptr="0xfffefc42c910" 2023-07-19T20:03:58.655141Z INFO apply_execution_providers: ort::execution_providers: TensorRT execution provider registered successfully
2023-07-19T20:04:01.109383Z INFO apply_execution_providers: ort::execution_providers: TensorRT execution provider registered successfully
2023-07-19T20:04:01.109456Z INFO apply_execution_providers: ort::execution_providers: TensorRT execution provider registered successfully
2023-07-19T20:04:01.109473Z INFO apply_execution_providers: ort::execution_providers: CUDA execution provider registered successfully
2023-07-19T20:04:01.355042Z INFO apply_execution_providers: ort::execution_providers: CUDA execution provider registered successfully
....
from ortex.
Those feature flags should be enabled here. We don't set them in the cargo features so that elixir can control the features depending on platform.
It sounds like :os.type()
is not returning an expected result on the Jetson Nano Orin, could you run that and let me know what it returns @gBillal?
from ortex.
actually I suspected that at first and so I run that command, however everything is fine
from ortex.
Interesting. I'll see if I can reproduce this on my jetson and work from there.
from ortex.
Something seems off here, when pulling ortex
via hex
there's no config folder in deps/ortex
. So adding a configuration in config.exs
on my project seems to work
import Config
config :ortex, Ortex.Native, features: ["cuda", "tensorrt"]
I can see this
==> ortex
Compiling 6 files (.ex)
Compiling crate ortex in release mode (native/ortex)
["cargo", "rustc", "--features", "cuda,tensorrt", "--release"]
Compiling libc v0.2.139
Compiling cc v1.0.79
Compiling log v0.4.17
Compiling memchr v2.5.0
...
But without the configuration above, this is what I have
==> ortex
Compiling 6 files (.ex)
Compiling crate ortex in release mode (native/ortex)
["cargo", "rustc", "--release"]
Compiling libc v0.2.139
Compiling cc v1.0.79
...
Does this mean we need to add this config in our project. This should be specified in the documentation.
from ortex.
Ah, thank you. Yes, looks like I forgot to include config.exs
in the hex package. I'll add that now. This way, if you want to add different execution providers you can override them in your own config but still have a sane default. I'll add a section in the docs to highlight this as well.
from ortex.
Ortex v0.1.7 is now live on Hex, which includes the default config.exs
as well as some more docs around adding different EPs. I'm going to close this for now, but feel free to reopen if this does not resolve your issue.
from ortex.
Related Issues (20)
- Building on macOS M2 HOT 3
- Docs tidy up suggestion HOT 3
- Implement slice on Ortex.Backend
- Implement reshape, squeeze, and concatenate for Ortex.Backend HOT 3
- Update to rustler 0.29.0 for use with OTP 26 HOT 2
- Error loading onnx model HOT 6
- Bump Nx version HOT 1
- Update to run on Nx ~> 0.6 HOT 2
- Implement argmax
- Is it possible to update ort library to 1.16.3? HOT 3
- Unable to use ortex from main HOT 8
- Plans for Rustler precompiled? HOT 1
- [Feature request] with_session_from_memory support HOT 3
- Add example for BERT sequence classification HOT 1
- Missing ONNX Runtime Contrib Operators?
- More flexible rustler dependency HOT 2
- Missing onnxruntime on Mac stops build HOT 3
- Upgrade to 1.16 HOT 2
- On main, using silero VAD, output shapes are inconsistent HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ortex.