How to detect objects Real Time about playtorch HOT 8 CLOSED

Hey @dongdv95,

Take a look at the following example: ObjectDetectionExample.tsx. In this example, you can learn how to perform object detection with DETR. The problem is that it's a transformer so it's not meant to work on real-time because it has too many parameters.

You can use a "lighter" model (with less parameters) like the Faster R-CNN (for the moment, you may experiment some issues if you try to use YOLO, that's why I recommend you that model). You can learn how to import this custom model in the following tutorial: PyTorch Live: Prepare Custom Model.

To import the model in the make_models.py file you can use torchvision:

torchvision.models.detection.faster_rcnn

You can see more details about this model in the following link: torchvision faster_rcnn

Finally, to adapt the example so it works in real-time you can follow the Image Classification tutorial which covers how to perform image classification in real-time.

Just to summarise, if you only want to see an example of a real-time use of "onFrame" you can just take a look at the Image Classification tutorial and you will see. But I also wanted to show you an example that implements the drawing of the bounding boxes and how you can import a custom model as I think that it will speed up your creation of the app.

Best of luck!

from playtorch.

Hey @dongdv95,

Take a look at the following example: ObjectDetectionExample.tsx. In this example, you can learn how to perform object detection with DETR. The problem is that it's a transformer so it's not meant to work on real-time because it has too many parameters.

You can use a "lighter" model (with less parameters) like the Faster R-CNN (for the moment, you may experiment some issues if you try to use YOLO, that's why I recommend you that model). You can learn how to import this custom model in the following tutorial: PyTorch Live: Prepare Custom Model.

To import the model in the make_models.py file you can use torchvision:

torchvision.models.detection.faster_rcnn

You can see more details about this model in the following link: torchvision faster_rcnn

Finally, to adapt the example so it works in real-time you can follow the Image Classification tutorial which covers how to perform image classification in real-time.

Just to summarise, if you only want to see an example of a real-time use of "onFrame" you can just take a look at the Image Classification tutorial and you will see. But I also wanted to show you an example that implements the drawing of the bounding boxes and how you can import a custom model as I think that it will speed up your creation of the app.

Best of luck!

Thanks for answer @aaronespasa , I 'm try Image Classification with resnet18, 34 with custom model , but with ObjectDetection
How to create live.spec.json ?
I tried with model torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True) have issue here :

this code make_model

from pathlib import Path
import torch
import torchvision
from torch.utils.mobile_optimizer import optimize_for_mobile


def dict_to_tuple(out_dict):
    if "masks" in out_dict.keys():
        return out_dict["boxes"], out_dict["scores"], out_dict["labels"], out_dict["masks"]
    return out_dict["boxes"], out_dict["scores"], out_dict["labels"]

    
class TraceWrapper(torch.nn.Module):
    def __init__(self, model):
        super().__init__()
        self.model = model

    def forward(self, inp):
        out = self.model(inp)
        return dict_to_tuple(out[0])
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)
model = TraceWrapper(model)
model.eval()

name = "fasterrcnn_resnet50_fpn"
    MODEL_EXTENSION = "ptl"
    traced_script_module = torch.jit.trace(model, (torch.rand(2,3, 224, 224)), strict=False)
    spec = Path("live.spec.json").read_text()
    extra_files = {"model/live.spec.json": spec}
    optimized_model = optimize_for_mobile(traced_script_module)
    optimized_model._save_for_lite_interpreter(
        f"{name}.{MODEL_EXTENSION}", _extra_files=extra_files
    )

This is live.spec.json

{
  "pack": {
    "type": "tensor_from_image",
    "image": "image",
    "transforms": [
      {
        "type": "image_to_image",
        "name": "scale",
        "width": 800,
        "height": 800
      },
      {
        "type": "image_to_tensor",
        "name": "rgb_norm",
        "mean": [0.485, 0.456, 0.406],
        "std": [0.229, 0.224, 0.225]
      }
    ]
  },
  "unpack": {
    "type": "tuple",
    "items": ["labels", "boxes", "scores"]
  }
}

and Error

TracingCheckError: Tracing failed sanity checks!
ERROR: Graphs differed across invocations!
	Graph diff:
		  graph(%self.1 : __torch__.TraceWrapper,
		        %inp : Tensor):
		    %model : __torch__.torchvision.models.detection.faster_rcnn.FasterRCNN = prim::GetAttr[name="model"](%self.1)
		    %8 : float = prim::Constant[value=0.03125](), scope: __module.model/__module.model.roi_heads/__module.model.roi_heads.box_roi_pool # /media/D/DONG/React/react_native/pro1/project1/models/venv/lib/python3.7/site-packages/torchvision/ops/roi_align.py:58:0
		    %9 : float = prim::Constant[value=0.0625](), scope: __module.model/__module.model.roi_heads/__module.model.roi_heads.box_roi_pool # /media/D/DONG/React/react_native/pro1/project1/models/venv/lib/python3.7/site-packages/torchvision/ops/roi_align.py:58:0
		    %10 : float = prim::Constant[value=0.125](), scope: __module.model/__module.model.roi_heads/__module.model.roi_heads.box_roi_pool # /media/D/DONG/React/react_native/pro1/project1/models/venv/lib/python3.7/site-packages/torchvision/ops/roi_align.py:58:0
		    %11 : int = prim::Constant[value=7](), scope: __module.model/__module.model.roi_heads/__module.model.roi_heads.box_roi_pool # /media/D/DONG/React/react_native/pro1/project1/models/venv/lib/python3.7/site-packages/torchvision/ops/roi_align.py:58:0
		    %12 : float = prim::Constant[value=0.25](), scope: __module.model/__module.model.roi_heads/__module.model.roi_heads.box_roi_pool # /media/D/DONG/React/react_native/pro1/project1/models/venv/lib/python3.7/site-packages/torchvision/ops/roi_align.py:58:0
		    %13 : Tensor = prim::Constant[value={2}](), scope: __module.model/__module.model.roi_heads/__module.model.roi_heads.box_roi_pool # /media/D/DONG/React/react_native/pro1/project1/models/venv/lib/python3.7/site-packages/torchvision/ops/poolers.py:80:0
		    %14 : int = prim::Constant[value=5](), scope: __module.model/__module.model.roi_heads/__module.model.roi_heads.box_roi_pool # /media/D/DONG/React/react_native/pro1/project1/models/venv/lib/python3.7/site-packages/torchvision/ops/poolers.py:79:0
...
...
v/lib/python3.7/site-packages/torchvision/models/detection/transform.py:285:0
		?      ^^^^                         ^^
		-   %3196 : (Tensor, Tensor, Tensor) = prim::TupleConstruct(%3195, %3169, %3170)
		?      ^^                                                      ^^      ^     ^^
		+   %3188 : (Tensor, Tensor, Tensor) = prim::TupleConstruct(%3187, %3161, %3162)
		?      ^^                                                      ^^      ^     ^^
		-   %4 : Tensor, %5 : Tensor, %6 : Tensor = prim::TupleUnpack(%3196)
		?                                                                ^^
		+   %4 : Tensor, %5 : Tensor, %6 : Tensor = prim::TupleUnpack(%3188)
		?                                                                ^^
		    %7 : (Tensor, Tensor, Tensor) = prim::TupleConstruct(%4, %5, %6)
		    return (%7)
	First diverging operator:
	Node diff:
		- %model : __torch__.torchvision.models.detection.faster_rcnn.___torch_mangle_7858.FasterRCNN = prim::GetAttr[name="model"](%self.1)
		?                                                                              ---
		+ %model : __torch__.torchvision.models.detection.faster_rcnn.___torch_mangle_8037.FasterRCNN = prim::GetAttr[name="model"](%self.1)
		?                                                                             +++

I have : torch version 1.10.1 , os: linux-ubuntu18.04, python 3.7.10, torch version 0.11.2

from playtorch.

@dongdv95 when do you get this error? When trying to run the model in a PyTorch Live app?

from playtorch.

@dongdv95 when do you get this error? When trying to run the model in a PyTorch Live app?

Hi @clarksandholtz, When I make model (model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)) , I get this error ,

I'm not sure my live.spec.json is correct.
I have output of model with example

x = torch.rand(1,3, 224, 224)
predictions = model(x)
print(predictions)

output

[{'boxes': tensor([], size=(0, 4), grad_fn=<StackBackward0>), 
'labels': tensor([], dtype=torch.int64),
'scores': tensor([], grad_fn=<IndexBackward0>)}]

my live.spec.json:

{
  "pack": {
    "type": "tensor_from_image",
    "image": "image",
    "transforms": [
      {
        "type": "image_to_image",
        "name": "scale",
        "width": 800,
        "height": 800
      },
      {
        "type": "image_to_tensor",
        "name": "rgb_norm",
        "mean": [0.485, 0.456, 0.406],
        "std": [0.229, 0.224, 0.225]
      }
    ]
  },
  "unpack": {
    "type": "tuple",
    "items": ["labels", "boxes", "scores"]
  }
}

from playtorch.

@dongdv95 if this is happening when you are making the model in Python, then the live.spec.json isn't being used at all. The live.spec.json is only used by PyTorch Live in mobile apps. If you remove all of the code that references the live.spec.json you still get the same error.

Looks like there is an issue with the tracing, but I'm not sure what it is.

from playtorch.

Thanks @clarksandholtz , I test with another model
( torchvision.models.detection.ssd300_vgg16(pretrained=True) ) and
(FasterRCNN(backbone= torchvision.models.mobilenet_v2(pretrained=True).features)), I make model success
but when run inference on Android with error

Error: Following ops cannot be found. Check fburl.com/missing_ops for the fix.{torchvision::nms, } ()
Exception raised from print_unsupported_ops_and_throw at ../torch/csrc/jit/mobile/parse_operators.cpp:65 (most recent call first):
(no backtrace available)

I’m grateful for your assistance

from playtorch.

Closing this issue out since the packer / unpacker approach was removed with 0.2.2. Check out the PyTorch-inspired API in the README as an alternative that avoids these issues: https://github.com/facebookresearch/playtorch/tree/main/react-native-pytorch-core#example-usage

As for the unsupported operation, I'd try it again. We have updated the versions of PyTorch and Torchvision used in the PlayTorch app (side note: we have rebranded from PyTorch Live to PlayTorch), and what you need may now be supported. Here is a starting-off point for using the PlayTorch app: https://playtorch.dev/docs/tutorials/get-started/

from playtorch.

Hello @aaronespasa, Where can i find example of a real-time use of "onFrame"?
This Image Classification tutorial link is unavailable and I can't find real-time example at image classification example link of playtorch tutorial.

from playtorch.