iwatake2222 / inferencehelper Goto Github PK

Environment (Hardware)

• Hardware: iMX8 Plus SOC with NPU.
• Software: yocto, qt, cmake

Information

I have a Qt6 application (cmake based) and included the Inference helper with Tensorflow Lite support.
I include this Qt project into a Yocto project, generating a Linux image for iMX8 Plus platform.

The project compile all well and the Inference helper runs with the INFERENCE_HELPER_ENABLE_TFLITE_DELEGATE_XNNPACK settings.

external_delegate_path

In Yocto tensorflow-lite and tensorflow-lite-vx-delegate for iMX8P is integrated.
If I use the following command from the installed examples

USE_GPU_INFERENCE=0 ./label_image -m mobilenet_v1_1.0_224_quant.tflite -i grace_hopper.bmp -l labels.txt --external_delegate_path=/usr/lib/libvx_delegate.so

Tensorflow use the NPU hardware acceleration. Important for this are

• USE_GPU_INFERENCE=0
• --external_delegate_path=/usr/lib/libvx_delegate.so

Question

Is it possible to include USE_GPU_INFERENCE and --external_delegate_path=/usr/lib/libvx_delegate.so into the XNNPACK settings?
Or need I create a completely custom delegate pack?

https://www.tensorflow.org/lite/performance/nnapi

Add kNcnnVulkan

https://developer.qualcomm.com/software/qualcomm-neural-processing-sdk
https://developer.qualcomm.com/docs/snpe/overview.html

How to choose gpu1 and gpu2 device if using embedded platform and two gpus deployed via tflite's nnapi？
e.g. Dual GPU for embedded platform NXP i.MX8 QuadMax

https://github.com/Arm-software/armnn
https://developer.arm.com/ip-products/processors/machine-learning/arm-nn

In the loop in PreProcess() function, GetWidth()/GetHeight()/GetChannel() is used as end condition. It makes processing speed slow because there are some logic in these getter functions.

Just to store some generated files temporary

• Converted model (mobilenet_v2_1.0_224.nnp)
• Library for Jetson Xavier NX

mobilenet_v2_1.0_224.zip
nnabla_prebuilt_jetson_xavier_nx.zip

https://www.tensorflow.org/lite/performance/hexagon_delegate

Thanks for your open source, I would like to extend this project to support speech or NLP models. I found the TensorInfo does not support nlp or voice .

• Issue
  • After updating TensorFlow Lite to v2.6.0, build with NNAPI delegate in Android fails

Now that ncnn provides pre-built libraries for major OS/CPU, it's better to use them, rather than using "my" pre-built libraries.
https://github.com/Tencent/ncnn/releases/tag/20211208