JPEG-AI is a learning-based image coding standard, and it includes verification models as part of the standard. This repository aims to offer the verification models in ONNX format, along with our optimized backend support known as JAIOPT. These models consist of five pairs of Deep Neural Network (DNN) models used within the JPEG-AI framework, including single_encode, hyper_encoder, hyper_decoder, hyper_scale_decoder, and single_decoder. Each of these pairs includes two models, one for luminance (_y) and one for chrominance (_uv) encoding or decoding. The first four pairs of models are employed in the encoding phase, while the last three pairs are used in the decoding phase.
It is essential to note that the ONNX models, as well as our inference framework found in this repository, are primarily intended for performance evaluation rather than end-to-end inference. Additionally, we provide performance comparisons between our framework and others.
In summary, our inference framework demonstrates significant speed improvements, approximately 5.5-6.5 times faster than NNAPI and about 3 times faster than QNN (Qualcomm Neural Network).`

The backend in this repo was tested on Qualcomm Snapdragon 8 device (Example: Samsung Galaxy S21+).

This repository comprises two directories:

1. JAIOPT
   Has our inference framework including compiler and inference engine.
2. Verification_Models
   Contains the models discussed in the Introduction.

.
├── config.yml
├── Copyright.md
├── images
├── JAIOPT               // Our inference framework
├── LICENSE-APACHE
├── LICENSE-MIT
├── README.md
└── Verification_Models  // Models

Table 1 provides performance comparisons of decoder models between our inference framework and others on Snapdragon SoC (especially GPU) with Android OS. The focus is on execution latency. The testing configurations are as follows:

• Input image size:
  1024x1024
• Benchmark platform:
  Snapdragon 8 Gen 2, GPU
• Inference framework:
  JAIOPT (i.e., our framework)
  NNAPI (with TFLite and ONNX-Runtime, e.g., ORT as front ends)
  QNN
• Data type:
  FP16

Table 1. DECODER Performance Comparisons

Table 2 provides performance comparisons of the encoder model (single_encoder_y) between our inference framework and others under different input sizes. The testing configurations are the same as we mentioned above:

Table 2. single_encoder_y Performance Comparisons

We conducted a comparison between the computed results of the exported (ONNX) models and the original PyTorch version on a pixel level, setting tolerance thresholds ranging from 0.5% to 1%. The results indicate that the exported ONNX models consistently yield accurate results when compared to the PyTorch version.

Diagram 1 illustrates the verification procedure's underlying mechanism.

Diagram 1. Result Verification Procedure

To perform result verification for the hyper_decoder_y model, follow the example below:

1. Provide three validation-related configurations, as illustrated in Figure 1. These configurations include validation_inputs_data and validation_outputs_data to specify the inputs and outputs from the PyTorch model, along with validation_threshold indicating the tolerance for similarity between the PyTorch and ONNX results.

.

Figure 1. Configure for results verification

2. Once the configuration file is in place, execute the following commands for validation:

• Convert the model:

python ./lothar/controller.py convert --config=/Path/To/Project/JPEG-AI-OPT/Verification_Models/encoder/hyper_decoder_y.yml

• Run the model and validate the results:

python ./lothar/controller.py run --validate --warmup_round=5 --config=/Path/To/Project/JPEG-AI-OPT/Verification_Models/encoder/hyper_decoder_y.yml

Figure 2 presents the results verification for the hyper_decoder_y model.

Figure 2. Results verfication for model hyper_decoder_y

Note1: PyTorch models' inputs and outputs are available at: /Path/To/Project/JPEG-AI-OPT/Verification_Models/pytorch_io

Note2: Further details on running the models will be explained later.

Table 2 displays the operator types that are supported and have been utilized in the verification models.

Table 2. Supported Operator Types

Clone this repository to your workspace. (Our workspace in this example is ~/Documents/GitHub/demo)

mkdir ~/Documents/GitHub/demo
cd ~/Documents/GitHub/demo
git clone https://github.com/SmartHarmony/JPEG-AI-OPT
cd JPEG-AI-OPT

First, configure the Python path to align with your specific environment.

Note: This instruction uses Python 3.7 as an example. However, Python versions later than 3.7 should also be compatible. Make sure to use the path corresponding to the Python version you are working with.

Navigate to the JPEG-AI-OPT/JAIOPT/repo/opencl-kernel directory:

cd ~/Documents/GitHub/demo/JPEG-AI-OPT/JAIOPT/repo/opencl-kernel

Open the file named opencl_kernel_configure.bzl and find line 40 in the file. It should look like this:

python_bin_path = "/usr/local/bin/python3.7"

Replace /usr/local/bin/python3.7 with the path to your Python 3.7 installation.

To find your Python path, run the following command in your terminal:

which python

Executing the above commands will display your Python path as shown in the following image:

After configuring your custom Python path, the next step is to install the required libraries.
Navigate to the JAIOPT directory:

cd ~/Documents/GitHub/demo/JPEG-AI-OPT/JAIOPT/

There, you can install the necessary Python libraries by executing the following command:

pip install -r requirements.txt

Note: Whenever you see any importing error, you may need to install the missing package by running

pip install missing_package_name

On Linux, run:

sudo apt-get install cmake gcc g++ libboost-all-dev libncurses5

On MacOS, run:

brew install cmake gcc boost ncurses

For MacOS users, if Homebrew is not installed on your system, you can install it by the instructions on the Homebrew website.

• Install Bazel (5.0.0) from Bazel Documentation.

• Download and install Android SDK. It can be downloaded either from Android Studio or from the Android SDK command line tool. Detailed steps for installation can be referred to Installation through Android Studio and Installation through command line tool

• Make sure platform-tools will be installed after the installation of Android SDK. Please refer to the steps from the above links.

• Download Android NDK version r16b or r17c (later versions may be supported but have not been tested).

• Export the directory of Android SDK and Android NDK to the environment path.

  If you are in zsh or bash environments, add the following lines to your .bashrc file:

  export ANDROID_SDK_HOME=~/path/to/Android/sdk
  export ANDROID_NDK_HOME=~/path/to/android-ndk-r17c
  export PATH=~/path/to/Android/sdk/tools:~/path/to/Android/sdk/platform-tools:$PATH

  If you are in cshell enviornments, add the following lines to your .cshrc file:

  setenv ANDROID_SDK_HOME ~/path/to/Android/sdk
  setenv ANDROID_NDK_HOME ~/path/to/android-ndk-r17c
  setenv PATH ~/path/to/Android/sdk/tools:~/path/to/Android/sdk/platform-tools:$PATH`
  

Navigate to the JAIOPT directory:

cd ~/Documents/GitHub/demo/JPEG-AI-OPT/JAIOPT/

Execute the following commands:

bazel build --config android --config optimization //deepvan/executor:libexecutor_shared.so --config symbol_hidden --define neon=true --define openmp=true --define opencl=true --cpu=arm64-v8a

Note: In this step, if you encounter the build error:

 error: invalid value 'c++17' in '-std=c++17'

please proceed with the modifications outlined below:

• Navigate to the JAIOPT directory and open the .bazelrc file.
• Locate lines 8 and 9, which should appear as follows:

 build --cxxopt=-std=c++17
 build --host_cxxopt=-std=c++17

• Update these lines to use the C++1z standard instead of C++17:

 build --cxxopt=-std=c++1z
 build --host_cxxopt=-std=c++1z

In this part, we are using the decoder_uv model as an example for performance measurement.

Navigate to the JAIOPT directory:

cd ~/Documents/GitHub/demo/JPEG-AI-OPT/JAIOPT/

Run the build.sh script under the JAIOPT directory to initiate the build process:

After compiling JAIOPT successfully, proceed with the configuration steps for the decoder_uv model.

Navigate to the Verification_Models/decoder directory:

cd ~/Documents/GitHub/demo/JPEG-AI-OPT/Verification_Models/decoder/

Locate the configuration file decoder_uv.yml, which contains pre-defined parameters like model input/output shape, data type, and runtime.
Update the model_file_path in this file to reflect the correct path by modifying line 6 to:

 model_file_path: path/to/model/decoder_uv.onnx

Note: This example is specific to the decoder_uv model. For a general .yml template, please refer to the config.yml.

Here is a more detailed example:

Our model path is:

model_file_path: /home/phoenix/Documents/GitHub/demo/JPEG-AI-OPT/Verification_Models/decoder/decoder_uv.onnx

The modification is like this:

If you need to validate the model's results, make sure to update the paths for validation_inputs_data and validation_outputs_data as well.

Navigate to your JAIOPT directory:

cd ~/Documents/GitHub/demo/JPEG-AI-OPT/JAIOPT/

In your JAIOPT folder, make sure you can find a folder named lothar, then execute the command to convert that ONNX to our internal computational graph:

python lothar/controller.py convert --config=path/to/decoder_uv.yml --model_path=path/to/decoder_uv.onnx

Note: In the given command, the decoder_uv model is used as a representative example. For testing with a different ONNX model, replace decoder_uv with the name of your target model and update the corresponding configuration .yml file accordingly. Refer to the contents of the Verification_Models directory for examples of model configurations.

Here is a more detailed example:

Enter the command:

python /home/phoenix/Documents/GitHub/demo/JPEG-AI-OPT/JAIOPT/lothar/controller.py convert --config=/home/phoenix/Documents/GitHub/demo/JPEG-AI-OPT/Verification_Models/decoder/decoder_uv.yml --model_path=/home/phoenix/Documents/GitHub/demo/JPEG-AI-OPT/Verification_Models/decoder/decoder_uv.onnx

The result of the conversion should be like this:

To run the model, connect your Android phone to this computer, enable Developer mode, and turn on USB Debugging in the Developer Options within your phone's Settings.

Navigate to your JAIOPT directory:

cd ~/Documents/GitHub/demo/JPEG-AI-OPT/JAIOPT/

In your JAIOPT folder, make sure you can find a folder named lothar, afterward, run the following command in the root directory of this tool on your computer:

python lothar/controller.py run --config=path/to/decoder_uv.yml --model_path=path/to/decoder_uv.onnx

Note: In the given command, the decoder_uv model is used as a representative example. For testing with a different ONNX model, replace decoder_uv with the name of your target model and update the corresponding configuration .yml file accordingly. Refer to the contents of the Verification_Models directory for examples of model configurations.

Here is a more detailed example:

Enter the command:

python /home/phoenix/Documents/GitHub/demo/JPEG-AI-OPT/JAIOPT/lothar/controller.py run --config=/home/phoenix/Documents/GitHub/demo/JPEG-AI-OPT/Verification_Models/decoder/decoder_uv.yml --model_path=/home/phoenix/Documents/GitHub/demo/JPEG-AI-OPT/Verification_Models/decoder/decoder_uv.onnx

The decoder_uv model will then be executed on the smartphone on random inputs created by the script. The running result should be like this:

Source code uses Apache License 2.0

Other materials uses Creative Commons Attribution (CC BY 4.0)