TFlite support for tensorflow java about java HOT 6 CLOSED

Does Tensorflow Java API support .tflite file format?

AFAIK, it does not and I don't know what would be the amount of work to support it.

Ideally, you would still have access to the original .pb model that was used to generate the TF Lite version and use that one instead, as there are no real gain running a .tflite model if your application has access to the complete set of TF ops, as provided by TF Java. But if you are planning to use TF Lite models distributed on the public hub, it is possible that the original .pb file is "lost", unless you reach and ask the author directly.

from java.

Sorry to resurrect a dead issue, however...

as there are no real gain running a .tflite model if your application has access to the complete set of TF ops, as provided by TF Java

I was under the assumption that post training quantization required use of the tflite model. In our scenario, we're interested in taking a SavedModel with (a full .pb file) and quantizing it before using it with tensorflow/java bindings. Is that doable at all?

from java.

I think the TF implementation of it emits a tflite model, but there's no reason you couldn't do the ops by hand and save out a new SavedModel.

from java.

@Craigacp that sounds interesting! Sadly, I'm rather new to using non-python interfaces to tensorflow. Could you expand, or link to any resources for what "do the ops by hand" mean?

Currently, we're trying to specifically utilize this TF1 model: universal sentence encoder version 3. I'm able to get it into a suitable SavedModel format with some python code as follows:

# using tensorflow version 1.15.4

def save_module(url, save_path, input_type):
    with tf.Graph().as_default():
        module = hub.Module(url)

        model_input = tf.compat.v1.placeholder(input_type, name="input")
        model_output = tf.identity(module(model_input), name="output")
        with tf.compat.v1.Session() as session:
            session.run(tf.compat.v1.global_variables_initializer())
            tf.compat.v1.saved_model.simple_save(
                session,
                save_path,
                inputs={"input": model_input},
                outputs={"output": model_output},
                legacy_init_op=tf.compat.v1.initializers.tables_initializer(
                    name="init_all_tables"
                ),
            )

output_path = "/opt/model"
url = "https://tfhub.dev/google/universal-sentence-encoder-large/3"

save_module(
    url,
    output_path,
    tf.string,
)

This is because the version of the the hub module didn't have a tag and I had issues loading it with the tag-set value of '':

❯ saved_model_cli show --all --dir .

MetaGraphDef with tag-set: '' contains the following SignatureDefs:

signature_def['default']:
  The given SavedModel SignatureDef contains the following input(s):
    inputs['text'] tensor_info:
        dtype: DT_STRING
        shape: (-1)
        name: Placeholder:0
  The given SavedModel SignatureDef contains the following output(s):
    outputs['default'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 512)
        name: Encoder_en/hidden_layers/l2_normalize:0
  Method name is:

After resaving it though, it works. My hope was I'd be able to just quantize the model too (its around 750mb). I'd be interested in any resources or examples of what you're suggesting - I apologize in advance for my cluelessness 🙇

from java.

I had assumed that tflite would perform the transformation operations in python and so we could apply the same operations in Java (similar to the graph freezing). Unfortunately that's not the case, and so while you could naively replicate the model structure using int8 tensors and quantize the fp32 weights manually I'm not sure it would work that well (especially on a complex model like USE).

You might be able to use the graph transform tool to perform the quantization. It looks like it supports it, and should be able to write it back out to a frozen graph or a saved model, both of which are supported in Java.

Otherwise it looks like the options are limited for a pure TensorFlow-Java solution.

from java.

Great! I'll investigate that approach. Thanks for the pointer!

from java.