jetbrains-research / kinference Goto Github PK

Hi,

I'm trying to run a CLIP model (https://models.attentionai.uk/69977135045d97639861ef8c1af7b751d86e0f20.onnx), which uses convolution.

It works great in the KI backend (I had to implment the new Softmax operator version locally, think I'll make a PR soonish).

My project is multi platform including a web target, and seeing that tfjs is suggested for web targets for performance reasons, I am trying to use this.

However, the TFJS backend doesn't currently support Conv.

I am wondering if it is coherent and sensible to have a backend that uses TFJS where possible, and falls back to the KI backend for operations that aren't supported yet, but still using the fast TFJS backend for future supported operations. I can see that the KI backend is in a commainMain module, so it seems not straight forward to directly try but am curious for your thoughts.

I have been studying the Python demo code for llama.onnx, found here:
https://github.com/tpoisonooo/llama.onnx/blob/main/demo_llama.py#L184

I have looked through all the examples we currently have for kinference, but nothing is doing tokenisation yet. You would sort of expect an example like POSTagger to be doing tokenisation, but it seems to skip the hard part and load the end result directly in as the input.(Unless I'm misreading the code?)

How do I go from a string prompt, into an ONNXData object that would be accepted by this model?

Hello

• I am trying to load a MobileNet model using KIEngine.loadModel(model)
• I get Unsupported Operation error and I could see QuantizeLinear is not supported yet from KIOperatorFactory

May I know the list of supported models and support for other unsupported operators and models in the near future?

If defined the repository and dependency coordinates as stated in the README, I get the following:

Execution failed for task ':compileKotlinJvm'.

Error while evaluating property 'filteredArgumentsMap' of task ':compileKotlinJvm'
Could not resolve all files for configuration ':jvmCompileClasspath'.
> Could not find io.kinference:inference:0.1.12.
Searched in the following locations:
- https://packages.jetbrains.team/maven/p/ki/maven/io/kinference/inference/0.1.12/inference-0.1.12.pom
- https://repo.maven.apache.org/maven2/io/kinference/inference/0.1.12/inference-0.1.12.pom

Should I use inference-api instead? ("io.kinference:inference-api:0.1.12")

Needs docs around how to use, features, performance, limitations, and comparisons with other products [such as Tribuo from Oracle]

Hello, I was wondering if KInference is multiplatform or when it plans on being multiplatform if not.

This would be huge for us as we plan on making a Multiplateform library for our offline users that wish to make inferences on our models.

We have a C++ desktop app we'd like to support therefore we most likely will need native support.

Considering this project seems still in early development I thought that if it's not multiplatform yet then perhaps it would be important to think about it earlier than later.

Can this be setup on maven for easy distribution?

See link to registry: https://packages.jetbrains.team/maven/p/ki/maven/io/kinference/inference/

Thanks for the initiative!

It seems to look for a specific patched version of onnx
Could not find com.microsoft.onnxruntime:onnxruntime:1.13.1.patched

Specifying an explicit version will find it but I'm not sure it's compatible.
implementation("com.microsoft.onnxruntime:onnxruntime:1.13.1")

I run through a lot of errors then
java.lang.ClassNotFoundException: javaslang.λ

primary use case: inference of ML models on end-user devices, for example, with neural coprocessors like M1 has

Hi,

I am quite new to Java and barely know nothing about Kotlin. I am currently attempting to use the Kinference package in my Java project. While exploring the example codes in the public repo (kinference-examples) and it seems I could use Engine.loadmodel(path) to get a model, therefore I have the following code to achieve this:

import io.kinference.ort.model.ORTModel;
import io.kinference.ort.ORTEngine;

public class ORTEngineExample {
    public static void main(String modelPath) {
        // Create an instance of ORTEngine
        ORTEngine ENGINE = new ORTEngine();
        // Load model
        ORTModel ortModel = ENGINE.loadModel(modelPath);
    }
}

yet I get

error: ORTEngine() has private access in ORTEngine
        ORTEngine ENGINE = new ORTEngine();
                           ^

As a newbee, I am struggling to understand how to resolve this issue so I am reaching out for assistance. Could someone maybe share an example code demonstrating how to load an ONNX model and run inferencing effectively in a Java project? Thanks in advance!