We should implement cl-mlperf:program:image-classification-onnx similar to
ck-tensorflow:program:image-classification-tflite and friends to support MLPerf image classification models in the ONNX format (e.g. ResNet50). Perhaps main.py already does all we need, just needs to have its inputs and outputs unified with program:image-classification-tflite.

I've created a copy of the script we used for benchmarking Image Classification models for the Closed division of MLPerf Inference v0.5. Below are some suggestions for updating this script for MLPerf Inference v0.7.

• On a platform like Firefly-RK3399, we support 3 workflow variants (implementations/backends): TFLite, ArmNN with Neon support, ArmNN with OpenCL support. For v0.5, we modified the script to specify which variant is "active" and relaunched. Near the submission deadline, we introduced iteration over implementations and backends into the audit.sh script. Now we should do similarly for the run.sh script. Bonus points for spotting other improvements based on the audit.sh script or the run.sh script for the Open division!

• In addition to rpi4, add other platforms that do not support ArmNN with OpenCL such as Jetson TX1 (tx1). In fact, perhaps write a function that checks whether the given platform is in an OpenCL "blacklist"?

• MobileNet has been dropped, so this leaves ResNet only. For simplicity, we will only support one preprocessing method (using OpenCV). Therefore, we should change this:

models=( "mobilenet" "resnet" )
models_tags=( "model,tflite,mobilenet-v1-1.0-224,non-quantized" "model,tflite,resnet,no-argmax" )
# Preferred preprocessing methods per model.
models_preprocessing_tags=( "full,side.224,preprocessed,using-opencv" "full,side.224,preprocessed,using-tensorflow" )

to

models=( "resnet" )
models_tags=( "model,tflite,resnet,no-argmax" )
# Preferred preprocessing methods per model.
models_preprocessing_tags=( "full,side.224,preprocessed,using-opencv" )

• We should update the library versions: currently, ArmNN v20.05 and TFLite v1.15.3 are the latest supported (although we may be able to support TFLite 2.x?)

• In principle, the difference between Closed and Open division submissions will be primarily in the list of models to use: ResNet only for the Closed; all MobileNet variants for the Open. Having a unified script would be great!

• Another idea for unification. We typically have two variants of a workflow: with LoadGen and without LoadGen support e.g. program:image-classification-tflite-loadgen and program:image-classification-tflite. For testing purposes, we should be able to specify which program variants to use as e.g. elsewhere. This choice can manifest itself e.g. here.

The MLPerf ResNet50 reference by @guschmue supports weights both in the TensorFlow and ONNX formats.

@guschmue notes:

The benchmark is a reference implementation that is not meant to be the fastest implementation possible. It is written in python which might make it less suitable for lite models like mobilenet or large number of cpu's. We are thinking to provide a c++ implementation with identical functionality in the near future.

In fact, we already have such an implementation wrapped in CK: ck-tensorflow:program:image-classification-tf-cpp. Let's check it works with the ResNet50 weights in the TensorFlow format by providing a CK package.

Packages such as:

• model-tf-mlperf-resnet50
• model-tf-mlperf-mobilenet
  already have the model graphs deposited on Zenodo. We should identify all model graphs that currently reside elsewhere: Amazon S3, TensorFlow.org, private DropBox accounts, etc. - and deposit them on Zenodo too.

TFLite models

• model-tflite-mlperf-resnet - DropBox (converted by Xored)
• model-tflite-mlperf-ssd-mobilenet - DropBox (converted by Xored)

TF models

• model-tf-mlperf-mobilenet-quantized - TensorFlow.org
• model-tf-mlperf-ssd-mobilenet - TensorFlow.org
• model-tf-mlperf-ssd-mobilenet-quantized - TensorFlow.org (does not work so not committed yet)
• model-tf-mlperf-ssd-mobilenet-quantized-finetuned - DropBox (courtesy of Habana)

I tried to build TFLite 1.13 for android28-arm64 with Android NDK r18b but it fails. I used the following commands:

$ ck detect soft:compiler.llvm.android.ndk --target_os=android28-arm64
$ ck install package --tags=lib,tflite,v1.13 --env.CK_HOST_CPU_NUMBER_OF_PROCESSORS=2 --target_os=android28-arm64

Here is the output:
output.log

For v0.5, we only used quantized MobileNet-v1 models, but not to quantized MobileNet-v2 ones. Google have since released quantized MobileNet-v2 checkpoints. We should investigate if they can be converted to quantized TFLite models relatively easily (if they are not already).

@bellycat77 has managed to convert the TF ResNet50 v1.5 model used in MLPerf Inference to TFLite with the following script:

import tensorflow as tf

graph_def_file = "resnet50_v1.pb"
input_arrays = ["input_tensor"]
output_arrays = ["softmax_tensor"]

converter = tf.contrib.lite.TFLiteConverter.from_frozen_graph(
  graph_def_file, input_arrays, output_arrays)
tflite_model = converter.convert()
open("resnet50_v1.tflite", "wb").write(tflite_model)

We should generalise and automate this via a CK script. For example, the input file can come from a dependency on a TF model, whereas the input and output arrays can be specified in the model's metadata.

Similarly to multi-variation packages for MobileNet-v1 and MobileNet-v2, we should create one or more such packages for MobileNet-v3, minimalistic and EdgeTPU models. It's probably best to test them with program:image-classification-tflite.

For v0.5, we only had time to create a TFLite workflow with LoadGen for Image Classification out of the existing TFLite workflow for Image Classification.

For v0.7, we would like to create a similar TFLite workflow with LoadGen for Object Detection out of the exiting TFLite workflow for Object Detection.

The RNN-T CmdGen is work-in-progress. We started it for the v0.7 submission round, but eventually did not submit due to a belatedly discovered postprocessing issue. Not surprisingly, it needs more love to get into shape.

Currently, the usage is:

$ ck run cmdgen:benchmark.speech-recognition-loadgen --model=rnnt \
--scenario=singlestream --mode=accuracy \
--sut=aws-g4dn.4xlarge

Future improvements:

• The --model parameter should be optional. (We only support one model after all.)
• The --sut parameter should allow any SUT name. At the moment, it is restricted to handful, and alternatives result in an error (CK error: [cmdgen] build_map[sut] is missing both 'aws-g4dn.4xsmall' and '###' values!.)
• The record name (e.g. mlperf-closed-aws-g4dn.4xlarge-pytorch-v1.15.1-rnnt-singlestream) must include the --mode to allow keeping both performance and accuracy experiment entries simultaneously. At the moment, a previously recorded experiment entry for one mode must be removed (ck rm local:experiment:mlperf-closed*rnnt* -f) to allow for the other mode.
• The record name should not include a bogus inference engine version (v1.15.1). The default inference engine name (pytorch) should be customizable according to the plugins used.

With the closing of the public download links for ImageNet (see discussion at pytorch/vision#1453), the ck install package --tags=image-classification,dataset,imagenet,val,original,full command now fails.

Hi,
I am trying to reproduce the resnet50 benchmark result shows on mlperf.org (edged/closed division row 4). Currently I can achieve 2.4ms latency on Jetson Xavier AGX SingleStream scenario . However, I use trtexec to generate the int8 engine file from resnet50_v1.onnx and trtexec does not support calibration (it uses random weight instead). Could you tell me how did you convert resnet50_v1.onnx to int8 engine file (or plan file) you use in this this repo?

The script can search for all experiments with specific tags (e.g. mlperf,image-classification,mobilenet-v1-1.0-224) and offer to select a pair of them for comparison (e.g. one from a reference implementation and another from a vendor optimised one; or one from an floating-point implementation and another from a quantised one).

For example, the reference MobileNet implementation can be benchmarked as follows:

$ ck benchmark program:image-classification-tflite \
--repetitions=10 --env.CK_BATCH_SIZE=1 --env.CK_BATCH_COUNT=2 \
--record --record_repo=local --record_uoa=mlperf-mobilenet-v1-1.00-224-tflite-0.1.7-performance \
--tags=mlperf,image-classification,mobilenet-v1-1.0-224,tflite-0.1.7,performance \
--skip_print_timers --skip_stat_analysis --process_multi_keys

We should update all image classification clients we support for MLPerf to use new preprocessing (in the order of priority - high to low):

• ck-tensorflow:program:image-classification-tflite
• ck-tensorflow:program:image-classification-tf-cpp
• ck-tensorflow:program:image-classification-tf-py
• ck-mlperf:program:image-classification-onnx-py

Time-permitting, we should also update non-MLPerf clients:

• ck-tensorflow:program:image-classification-tf-frozen-py
• ck-tensorflow:program:image-classification-tflite-py

At the same time, we should also:

• retire preprocess.py from ck-mlperf:script:image-classification.
• make the conversion script in ck-env:package:dataset-imagenet-preprocessed usable without CK.
• fully document ck-env:package:dataset-imagenet-preprocessed for use with and without CK.

Hi

Thank you for the latest docker file. The build is failing on my system with the following error at imageNet inference stage

  -----------------------------------
Installing to /home/dvdt/CK_TOOLS/dataset-imagenet-preprocessed
From: /home/dvdt/CK_TOOLS/dataset-imagenet-ilsvrc2012-val-min , To: /home/dvdt/CK_TOOLS/dataset-imagenet-preprocessed , Size: 224 , Crop: 87.5 , InterSize: 0 , 2GU: 0,  2BGR: False, OFF: 0, VOL: '', FOF: image_list
.txt, DTYPE: uint8, EXT: rgb8, IMG: 
Traceback (most recent call last):
  File "/home/dvdt/CK_REPOS/ck-env/package/dataset-imagenet-preprocessed/preprocess_image_dataset.py", line 186, in <module>
    output_filenames = preprocess_files(selected_filenames, source_dir, destination_dir, crop_percentage, square_side, inter_size, guentherization_mode, convert_to_bgr, data_type, new_file_extension)
  File "/home/dvdt/CK_REPOS/ck-env/package/dataset-imagenet-preprocessed/preprocess_image_dataset.py", line 134, in preprocess_files
    convert_to_bgr = convert_to_bgr)
  File "/home/dvdt/CK_REPOS/ck-env/package/dataset-imagenet-preprocessed/preprocess_image_dataset.py", line 87, in load_image
    img = scipy.misc.imread(image_path)
**AttributeError: module 'scipy' has no attribute 'misc'**

Details
CK version: 1.9.7
Python version used by CK: 3.7.3 (default, Mar 27 2019, 22:11:17) 
[GCC 7.3.0]

docker --version
Docker version 18.09.6, build 481bc77156

OS - Centos7 on GCP

Running MobileNet quantized with TFLite on images preprocessed with OpenCV for MobileNet into rgbf32 fails:

$ ck benchmark program:image-classification-tflite --speed --skip_print_timers \
--repetitions=1 --env.CK_BATCH_SIZE=1 --env.CK_BATCH_COUNT=10 \
--dep_add_tags.images=preprocessed,using-opencv,normalized-for.mobilenet \
--dep_add_tags.weights=mobilenet,quantized
...
Summary:
-------------------------------
Graph loaded in 0.001913s
All images loaded in 0.002327s
All images classified in 0.460941s
Average classification time: 0.043231s
Accuracy top 1: 0.0 (0 of 10)
Accuracy top 5: 0.0 (0 of 10)
--------------------------------

The same works fine with TF-C++:

$ ck benchmark program:image-classification-tf-cpp --speed --skip_print_timers \
--repetitions=1 --env.CK_BATCH_SIZE=1 --env.CK_BATCH_COUNT=10 \
--dep_add_tags.images=preprocessed,using-opencv,normalized-for.mobilenet \
--dep_add_tags.weights=mobilenet,quantized
...
Summary:
-------------------------------
Graph loaded in 0.081466s
All images loaded in 0.002623s
All images classified in 0.528722s
Average classification time: 0.011366s
Accuracy top 1: 0.7 (7 of 10)
Accuracy top 5: 0.9 (9 of 10)
--------------------------------

or with TFLite on images preprocessed using OpenCV into rgb8:

$ ck benchmark program:image-classification-tflite --speed --skip_print_timers \
--repetitions=1 --env.CK_BATCH_SIZE=1 --env.CK_BATCH_COUNT=10 \
--dep_add_tags.images=preprocessed,using-opencv,rgb8 \
--dep_add_tags.weights=mobilenet,quantized
...
Summary:
-------------------------------
Graph loaded in 0.001789s
All images loaded in 0.000663s
All images classified in 0.477427s
Average classification time: 0.043434s
Accuracy top 1: 0.7 (7 of 10)
Accuracy top 5: 0.9 (9 of 10)
--------------------------------

Due to warm up effects, we usually suggest to run our image classification clients with --env.CK_BATCH_COUNT=2. As we explain:

When using the batch count of N, the program classifies N images, but the slow first run is not taken into account when computing the average classification time

This works a treat for the average classification time, but leads to an inconsistency when reporting the total classification time: tflite and tf-cpp truthfully include the time of the first batch into the total, while tf-py and onnx do not:

• tflite

($ ck benchmark program:image-classification-tflite \
--repetitions=10 --env.CK_BATCH_SIZE=1 --env.CK_BATCH_COUNT=2
...
Processing batches...

Batch 1 of 2

Batch loaded in 0.00802251 s
Batch classified in 0.16831 s

Batch 2 of 2

Batch loaded in 0.00776105 s
Batch classified in 0.0762354 s
...
Summary:
-------------------------------
Graph loaded in 0.000663s
All images loaded in 0.015784s
All images classified in 0.244545s
Average classification time: 0.076235s
Accuracy top 1: 0.5 (1 of 2)
Accuracy top 5: 1.0 (2 of 2)
--------------------------------

• tf-cpp

$ ck benchmark program:image-classification-tf-cpp \
--repetitions=10 --env.CK_BATCH_SIZE=1 --env.CK_BATCH_COUNT=2
...
Processing batches...

Batch 1 of 2
Batch loaded in 0.00341696 s
Batch classified in 0.355268 s

Batch 2 of 2
Batch loaded in 0.00335902 s
Batch classified in 0.0108837 s
...
Summary:
-------------------------------
Graph loaded in 0.053440s
All images loaded in 0.006776s
All images classified in 0.366151s
Average classification time: 0.010884s
Accuracy top 1: 0.5 (1 of 2)
Accuracy top 5: 1.0 (2 of 2)
--------------------------------

• tf-py

$ ck benchmark program:image-classification-tf-py \
--repetitions=10 --env.CK_BATCH_SIZE=1 --env.CK_BATCH_COUNT=2
...
Weights loaded in 0.293122s

Batch 1 of 2
Batch loaded in 0.001036s
Batch classified in 0.121501s

Batch 2 of 2
Batch loaded in 0.001257s
Batch classified in 0.013995s
...
Summary:
-------------------------------
Graph loaded in 1.115745s
All images loaded in 0.002293s
All images classified in 0.013995s
Average classification time: 0.013995s
Accuracy top 1: 0.5 (1 of 2)
Accuracy top 5: 1.0 (2 of 2)
--------------------------------

• onnx

$ ck benchmark program:image-classification-onnx-py --cmd_key=preprocessed \
--repetitions=10 --env.CK_BATCH_SIZE=1 --env.CK_BATCH_COUNT=2
...
Batch 1 of 2
Batch loaded in 0.001307s
Batch classified in 0.186297s

Batch 2 of 2
Batch loaded in 0.000721s
Batch classified in 0.029533s
...
Summary:
-------------------------------
Graph loaded in 0.018409s
All images loaded in 0.002028s
All images classified in 0.029533s
Average classification time: 0.029533s
Accuracy top 1: 0.5 (1 of 2)
Accuracy top 5: 1.0 (2 of 2)
--------------------------------