Question: Does the 'unpack' accept the object types in spec.json about playtorch HOT 7 CLOSED

@JohnZcp, the YOLOv5 model is not supported out of the box. IIRC, there is 1 additional native data processing step missing at the moment. Let me know if you are interested in adding the step, and we can help/advice!

from playtorch.

I checked the output through Colab. At this moment, the prediction result I got from the ptl file is a tuple of list of tensor (Tensor,[Tensor, Tensor, Tensor]). Generally, the prediction result of yolo_v5 is a object (before I convert the model to ScriptModule). The object should able to return the following info - xmin, ymin, xmax, ymax, confidence, class, name, by using method object.xyxy(0). I guess the first tensor is equivalent to the bounding box(xmin, ymin, xmax, ymax) and the tensors in the list are the confidence, class, and name. However, the tuple and list type of are not supported on iOS for "unpack" section in the spec.json. If I understand correctly, I think the biggest problem are the structure of "unpack", and how to pre-process the first tensor of the output in the spec.json.

from playtorch.

@raedle Can you explain more about the native data processing step you mentioned above?

from playtorch.

@JohnZcp, the pack and unpack data processing steps are in the following folders:

Android

• Packer implementations on Android
• Unpacker implementations on Android
• Packer/Unpacker registration on Android

iOS

• Packer implementations on iOS
• Unpacker implementations on iOS
• Packer/Unpacker registration on iOS

A good example to look at would be the bounding_boxes unpacker that's used for the DETR model.

We are working on a solution that will make the integration easier, but it's probably going to take a bit until it's stable. That said, we are looking for feedback from the community, so let us know if you want to experiment with this API

from playtorch.

@raedle Yes. And I wonder what should I do if I want to design a unpacker for yolo_v5. I want to have a try, but not sure how.

from playtorch.

@JohnZcp, do you have a torchscripted YOLOv5 model that runs in the lite interpreter runtime, and can you share the model with me with example input/output processing?

Feel free to DM me on the PyTorch Live Discord channel to work out a solution for YOLOv5

from playtorch.

Closing this issue out since the packer / unpacker approach was removed with 0.2.2. Check out the PyTorch-inspired API in the README as an alternative that avoids these issues: https://github.com/facebookresearch/playtorch/tree/main/react-native-pytorch-core#example-usage

As an additional note, see the YOLOv5 example for how to unpack using the current APIs: https://playtorch.dev/docs/tutorials/snacks/yolov5/

from playtorch.