elixir-nx / ortex Goto Github PK

When sending Nx.Serving.batched_run batches from multiple processes to an Ortex.Serving process, the run fails with:

    ** (RuntimeError) operation slice is not yet supported on Ortex.Backend.
        (ortex 0.1.3) lib/ortex/backend.ex:106: Ortex.Backend.slice/5
        (nx 0.5.3) lib/nx/defn/composite.ex:101: anonymous fn/3 in Nx.Defn.Composite.traverse/2
        (elixir 1.14.0) lib/enum.ex:1780: Enum."-map_reduce/3-lists^mapfoldl/2-0-"/3
        (nx 0.5.3) lib/nx/container.ex:110: Nx.Container.Tuple.traverse/3
        (nx 0.5.3) lib/nx/defn/composite.ex:101: Nx.Defn.Composite.traverse/2
        (nx 0.5.3) lib/nx/serving.ex:682: Nx.Serving.receive_batched/6
        (nx 0.5.3) lib/nx/serving.ex:648: Nx.Serving.local_batched_run/3
        (nx 0.5.3) lib/nx/serving.ex:618: Nx.Serving.local_batched_run!/3

This is due to Ortex.Backend not implementing slice. This will need to be implemented for Ortex.Serving to be complete.

Hey,
I am trying to use silero_vad and for that it seem to me, I need an up to date version of ort and therefore the current versin of "ortex" in main.

I am doing this on window11.

I tried the following. In my dependencies I added ortex:

      {:ortex, github: "elixir-nx/ortex"}

And then I did:

mix deps.get
mix deps.compile ortex

Resulting in:

==> ortex
Compiling 6 files (.ex)
Compiling crate ortex in release mode (native/ortex)
...
Compiling lib/ortex/native.ex (it's taking more than 10s)
   Compiling ortex v0.1.0 (<project_dir>\deps\ortex\native\ortex)
   = note: `#[warn(unused_imports)]` on by default

warning: `ortex` (lib) generated 1 warning (run `cargo fix --lib -p ortex` to apply 1 suggestion)
    Finished release [optimized] target(s) in 19.43s

== Compilation error in file lib/ortex/native.ex ==
** (File.CopyError) could not copy from "<project_dir>/_build/dev/lib/ortex/priv/native/libonnxruntime.so" to "<project_dir>/_build/dev/lib/ortex/priv/native/libonnxruntime.so.1.17.0": no such file or directory
    (elixir 1.16.1) lib/file.ex:864: File.cp!/3
    lib/ortex/native.ex:9: (module)
could not compile dependency :ortex, "mix compile" failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile ortex --force", update it with "mix deps.update ortex" or clean it with "mix deps.clean ortex"

Now, when I look into <project_dir>/_build/dev/lib/ortex/priv/native/
I find a file libortex.dll but not the wanted lionnxruntime.so.

I will look further into this, but if anyone has a hint ...

Thanks,
Nathan

There is a way of specify the execution providers for the ort build?

I think you do it here https://github.com/elixir-nx/ortex/blob/841ee6bebd842f43a15b2bb684e4e0691f69a790/config/config.exs#L11C9-L11C9 but I'm not sure.

Would be nice to allow onednn as it allows more performance on AWS arm instances (graviton2, 3...), to allow specifying the providers for compile time (rustler not elixir)

Hi @mortont!

I took another look at the library and I am loving it. Just minor nitpicks on the docs:

• I think Util and Native should be @moduledoc false
• Ortex.Model and Ortex have duplicated functions. I assume we want to make the functions in Ortex.Model @doc false?
• Shall we add a quick example of how to use Ortex.Serving?

Having these will reduce the number of Nx.backend_transfer calls needed when using Ortex with autoregressive models. There are likely others, similar to #8 that will make this more ergonomic.

Hello! ❤️ the library, I'm using it right now to run inference on a BERT fine-tuned for sequence classification.

@mortont do you think it would be helpful to add an example on how to set up the pipeline for inference similar to the StableLM example? The main thing that I did not find documented anywhere (maybe I didn't looked hard enough) was getting the id2label mappings from the generated config.json. If you think it will add value, I'll prepare the PR.

   Compiling ort v1.14.6
The following warnings were emitted during compilation:

warning: Provider not available for this strategy and/or target: CUDA
warning: Provider not available for this strategy and/or target: TENSORRT
warning: Provider not available for this strategy and/or target: CUDA
warning: Provider not available for this strategy and/or target: TENSORRT
warning: Provider not available for this strategy and/or target: CUDA
warning: Provider not available for this strategy and/or target: TENSORRT

error: failed to run custom build command for `ort v1.14.6`

Caused by:
  process didn't exit successfully: `/Users/jstimpson/dev/elixir/ortex/_build/dev/lib/ortex/native/ortex/debug/build/ort-c81d1256a1916c1c/build-script-build` (exit status: 101)
  --- stdout
  [ort] strategy: "unknown"
  cargo:warning=Provider not available for this strategy and/or target: CUDA
  cargo:warning=Provider not available for this strategy and/or target: TENSORRT
  cargo:warning=Provider not available for this strategy and/or target: CUDA
  cargo:warning=Provider not available for this strategy and/or target: TENSORRT
  cargo:rerun-if-env-changed=ORT_STRATEGY
  cargo:warning=Provider not available for this strategy and/or target: CUDA
  cargo:warning=Provider not available for this strategy and/or target: TENSORRT

  --- stderr
  thread 'main' panicked at 'Microsoft does not provide ONNX Runtime downloads for triplet: osx-arm64-gpu; you may have to use the `system` strategy instead', /Users/jstimpson/.asdf/installs/rust/1.69.0/registry/src/github.com-1ecc6299db9ec823/ort-1.14.6/build.rs:164:18
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

== Compilation error in file lib/ortex/native.ex ==
** (RuntimeError) Rust NIF compile error (rustc exit code 101)
    (rustler 0.26.0) lib/rustler/compiler.ex:41: Rustler.Compiler.compile_crate/2
    lib/ortex/native.ex:7: (module)

Workaround

index 635c517..c349099 100644
--- a/native/ortex/Cargo.toml
+++ b/native/ortex/Cargo.toml
@@ -11,7 +11,7 @@ crate-type = ["cdylib"]

 [dependencies]
 rustler = "0.26.0"
-ort = {version = "1.14.6", features = ["cuda", "tensorrt"]}
+ort = {version = "1.14.6", features = []}
 ndarray = "0.15.6"
 half = "2.2.1"
 tracing-subscriber = { version = "0.3", features = [ "env-filter", "fmt" ] }

In working on a segment-anthing livebook, based upon this forum example, I am downloading the onnx model from hugging face. The benefit is that the Livebook can be readily shared with no dependencies.

Currently it needs to be saved to a file system before calling Ortext.load/1 since that function ultimately delegates to the following in model.rs:

    let session = Session::builder()?
        .with_execution_providers(&eps)?
        .with_optimization_level(map_opt_level(opt))?
        .with_model_from_file(model_path)?;

A version or an option which calls .with_model_from_memory(binary) would be a valuable, and hopefully a simple addition.

I think Livebooks is a likely a common use case so hopefully something you might consider. I would happily send a PR if I had any relevant skill, but in Rust I do not.

Hey there! Any plans to add CI workflows w/ Rustler Precompiled? Would definitely help with deployment.

Hey team, not really an issue but I was wondering if it would be possible to update the Ort library to 1.16.3 so that we can use it with the last onnxruntime stable version which is 1.16.1.
To be honest I tried doing it myself, but I figured it was not a simple task as it seems from version 1.14.3 to version 1.15.0 lot of things seems to have change.

Thank you !

I am using the Ortex lib to do object detection using the Yolov8 model.
I need to use Nx.vectorize/2 but it is only available from Nx 0.6 onwards.
And Ortex still uses Nx 0.5

Hi, I'm trying to load a onnx model and I'm getting this error:

(RuntimeError) Failed to create ONNX Runtime session: Load model from /home/sonic182/work/myproyect/models/e5-small-v2-onnx/model.onnx failed:This is an invalid model. In Node, ("MaskReduceSum_0", ReduceSum, "", -1) : ("attention_mask_int32": tensor(int32),) -> ("mask_index_0",) , Error Unrecognized attribute: axes for operator ReduceSum
(ortex 0.1.6) lib/ortex/model.ex:28: Ortex.Model.load/3
/home/sonic182/sandbox/livebooks/embeddings.livemd#cell:s35av5wcbejnldtu5ekss352elzjd74y:3: (file)

The model was generated with this two code lines in python 3.10.12 (using colab, moving results to my drive and downloading it)

!pip install transformers "onnxruntime<1.15" optimum[onnxruntime,exporters] torch
!optimum-cli export onnx -m intfloat/e5-small-v2 --optimize O2 e5-small-v2-onnx --opset 18

using huggingface optimum cli, the model is e5-small-v2

This kind of model operations are not supported?

Can we update the rustler dependency to 0.29.0 to make it work with OTP 26 as per rusterlium/rustler#544 (comment)

Allows use of newer Nx functions like Nx.vectorize from v0.6

Having Nx.argmax implemented will help to reduce backend_transfer calls when recursively calling Ortex for applications that use greedy search, beam search, etc.