Model Spec for MobileNet Classes about playtorch HOT 6 CLOSED

Thanks for reaching out @awcchungster. PyTorch Live currently has a limited set of ops that are supported in the processing specification, and it doesn't support topk. It is possible to add additional ops to the PyTorch Live model specification. Besides topk, are there any other ops that you are considering to use?

Also, we will add functionality to PyTorch Live that will eventually remove these constraints entirely (e.g., by providing a JavaScript API that's backed by libTorch C++). At the moment, we are hashing out the API details and haven't pushed the code to GitHub yet, but here is an example of what the API currently looks like for retrieving top 5 results of a MobileNetV3 model inference result:

let imageTensor = torchvision.tensorFromImage(frame, 480, 640);
imageTensor = imageTensor.permute([2, 0, 1]);
imageTensor = torch.unsqueeze(imageTensor, 0);

let output = module.forward(imageTensor);

let result = torch.squeeze(output);
const softmaxResult = torch.softmax(result, 1);
const [_values, indices] = torch.topk(softmaxResult, 5);

indices.data.forEach((idx: number) => {
  const cls = IMAGE_CLASSES[idx];
  console.log(cls);
});

(API subject to change)

Please let us know if you have feedback on this approach!

from playtorch.

Hi @raedle! Thanks for responding timely manner. I found a bounding box example for mobilenet that I plan on testing next. After that I plan to do a segmentation model.

Are there examples of how to take a mobile net segmentation model results and apply it to PTL?

from playtorch.

Also, the APIs change a little too frequently. I understand this is all pre v1, so at a minimum putting this into a change log seems like a reasonable request.

I ran into a number of occurrences. For example, the NPM documentation didn't work. I had to dig through the Github to understand the class name changes that were done.

from playtorch.

@awcchungster, we have an example for DETR object detection but not MobileNet. If you have a model that you can share with expected inputs/outputs, I can look into it and probably provide an example.

Also, the APIs change a little too frequently. I understand this is all pre v1, so at a minimum putting this into a change log seems like a reasonable request.

Absolutely agreed that the API needs to be and will be stable, and we'll provide migration guides and documentation when it is ready!

Thanks for flagging the issue on the NPM doc!

from playtorch.

The NPM doc is fixed and shows in the nightly builds (e.g., 0.0.0-03182022-1202-65c9a6e25)

from playtorch.

Closing this issue out since the "Live Spec" approach was removed with 0.2.2. Check out the PyTorch-inspired API in the README as an alternative that avoids these issues: https://github.com/facebookresearch/playtorch/tree/main/react-native-pytorch-core#example-usage

As an additional note, topk is available in the new API: https://playtorch.dev/docs/api/core/interfaces/torchlive_torch.tensor/#topk

from playtorch.