Passing custom/additional data to kernels about tflite-micro HOT 11 OPEN

it might be related to #619

from tflite-micro.

Thanks for filing the issue. This is indeed an interesting problem. The typical approach for using accelerators with tflite-micro has been to use a custom op. I'd recommend taking a look at the Ethos-U example in-tree, but the gist of it is to run an AOT tool that takes a TFLite model, identify which layers can be accelerated, and merge/replace those with an Ethos-U custom op that includes a command stream in the custom data payload. This approach does not require the model to be re-trained and looks at the model as a whole, so multiple layers may get combined into a single accelerated node.
As you point out, this does specialize the TFLite model for a given hardware and is no longer runnable on other devices.

While your option 2 addresses that, I think there might be a less invasive way to add extra information. The Model schema allows for Metadata to be specified. This is essentially a key-value store mapping strings to opaque data blobs. You could add an entry specific to the accelerator that encoded additional information. In your op implementation, you could then check for that metadata key and check for additional node information. Reference kernels would ignore this metadata. This approach would also allow you to describe it in whichever way would make sense for your accelerator, as opposed to having it annotated on every operator.

Does that help?

from tflite-micro.