Comments (19)
Hello @bairock, thank you for your interest in PyTorch Live! As far as I’m aware, there isn’t yet a Live Spec defined by the community for YOLOv5. Are you planning to create one?
You can follow the tutorial here for instructions about how to adapt existing models to PyTorch Live (https://pytorch.org/live/docs/tutorials/prepare-custom-model), which includes creating a Live Spec. A Live Spec defines how to convert from an input in JavaScript to the tensor the model expects (this is the
pack
step) and how to convert from the model’s output to what will be returned to JavaScript (theunpack
step). Existing operations for use in thepack
andunpack
steps can be found in the API for Model Specification (https://pytorch.org/live/docs/api/model-spec/). Examples of existing Live Specs that ship with the PyTorch Live example app can be found at https://github.com/pytorch/live/tree/main/react-native-pytorch-core/example/modelsWhen you are successful, please do share a code pointer to the Live Spec you created! If you find that the Live Spec does not offer the operations you require, please let us know.
Is there an example of unpuck type tuple? found no example anywhere.
from playtorch.
Hi, what the structure of the items in the tuple unpack should be, here is an example:
# | xmin | ymin | xmax | ymax | confidence | class | name |
---|---|---|---|---|---|---|---|
0 | 749.50 | 43.50 | 1148.0 | 704.5 | 0.874023 | 0 | person |
1 | 433.50 | 433.50 | 517.5 | 714.5 | 0.687988 | 27 | tie |
2 | 114.75 | 195.75 | 1095.0 | 708.0 | 0.624512 | 0 | person |
3 | 986.00 | 304.00 | 1028.0 | 420.0 | 0.286865 | 0 | tie |
from playtorch.
Just FYI,
The official torchscript exported by yolov5 only contains the general model inferencing part, one must implement the pre-preprocess (letterbox
) and post-proprocess (mainly the nms
op) if they want to deploy this torchscript.
We did some experiments on yolort to embed the pre-processing and post-processing into the torchscript graph following the strategy of TorchVision's object detection banks, as such we don't need to write the pre-process and post-process within torchlive. So it will be very easy to deploy YOLOv5 with yolort if we could deploy the models in TorchVision on torchlive.
from playtorch.
@JohnZcp, @Aisenapio, @dongdv95, @bairock, @jslok, and @zhiqwang, the latest release 0.2.0
of PyTorch Live has a JavaScript API to pre-/post-process data. The live.spec.json
is no longer needed.
I create a YOLOv5 example app with the new PyTorch Live API.
It would be fantastic if this could also work with the YOLOv5 Runtime Stack!
from playtorch.
Hello @bairock, thank you for your interest in PyTorch Live! As far as I’m aware, there isn’t yet a Live Spec defined by the community for YOLOv5. Are you planning to create one?
You can follow the tutorial here for instructions about how to adapt existing models to PyTorch Live (https://pytorch.org/live/docs/tutorials/prepare-custom-model), which includes creating a Live Spec. A Live Spec defines how to convert from an input in JavaScript to the tensor the model expects (this is the pack
step) and how to convert from the model’s output to what will be returned to JavaScript (the unpack
step). Existing operations for use in the pack
and unpack
steps can be found in the API for Model Specification (https://pytorch.org/live/docs/api/model-spec/). Examples of existing Live Specs that ship with the PyTorch Live example app can be found at https://github.com/pytorch/live/tree/main/react-native-pytorch-core/example/models
When you are successful, please do share a code pointer to the Live Spec you created! If you find that the Live Spec does not offer the operations you require, please let us know.
from playtorch.
Hi, what the structure of the items in the tuple unpack should be, here is an example:
xmin ymin xmax ymax confidence class name
0 749.50 43.50 1148.0 704.5 0.874023 0 person
1 433.50 433.50 517.5 714.5 0.687988 27 tie
2 114.75 195.75 1095.0 708.0 0.624512 0 person
3 986.00 304.00 1028.0 420.0 0.286865 0 tie
Hi @Aisenapio , have you created Live Specs of yolov5 ? or an example of unpuck type tuple.
from playtorch.
Hello, I am also trying to create the spec.json for yolo_v5. How is your progress? @bairock
from playtorch.
Hi @raedle , I export a YOLOv5 torchscript model that accepts an image tensor as the input with yolort, and I upload the torchscript and optimized model at
- https://huggingface.co/spaces/zhiqwang/assets/blob/main/yolov5s_scripted.pt
- https://huggingface.co/spaces/zhiqwang/assets/blob/main/yolov5s_scriptmodule.ptl
I only embed the post-processing (nms
) into the torchscript graph, and the model is same with YOLOv5, check out this docs for more details, and feel free to pin me if anyone have any questions about how to export the torchscript-ed model or the relationship between YOLOv5 and yolort.
And you can use the following scripts to test the inference output using this torchscript-ed models (install yolort with pip install yolort==0.6.3
fist):
import cv2
import torch
import torchvision
from torch.utils.mobile_optimizer import optimize_for_mobile
from yolort.utils import Visualizer, read_image_to_tensor
from yolort.v5 import letterbox, scale_coords, attempt_download
img_size = 640
stride = 32
device = torch.device('cpu')
export_scripted_source = "https://huggingface.co/spaces/zhiqwang/assets/resolve/main/yolov5s_scripted.pt"
export_scripted_path = attempt_download(export_scripted_source)
export_optimized_path = "yolov5s_scriptmodule.ptl"
### Load the TorchScript model
scripted_model = torch.jit.load(export_scripted_path)
optimized_model = optimize_for_mobile(scripted_model)
# Or download the optimized model from https://huggingface.co/spaces/zhiqwang/assets/blob/main/yolov5s_scriptmodule.ptl
optimized_model._save_for_lite_interpreter(export_optimized_path)
scripted_model = scripted_model.eval()
scripted_model = scripted_model.to(device)
### Load the image
img_source = "https://huggingface.co/spaces/zhiqwang/assets/resolve/main/bus.jpg"
# img_source = "https://huggingface.co/spaces/zhiqwang/assets/resolve/main/zidane.jpg"
img_path = attempt_download(img_source)
img_raw = cv2.imread(img_path)
image = letterbox(img_raw, new_shape=(img_size, img_size), stride=stride)[0]
image = read_image_to_tensor(image)
image = image.to(device)
image = image[None]
with torch.no_grad():
out_script = scripted_model(image)
# Don't forget to rescale the coordinates back to original image scale
scale_coords(image.shape[2:], out_script[1][0]['boxes'], img_raw.shape[:-1])
print(out_script)
And the out_script[1][0]
is the inference result of this image.
{'scores': tensor([0.89617, 0.87025, 0.85156, 0.84933, 0.53494]),
'labels': tensor([0, 0, 0, 5, 0]),
'boxes': tensor([[6.71788e+02, 3.95372e+02, 8.10000e+02, 8.78361e+02],
[2.20657e+02, 4.08141e+02, 3.46167e+02, 8.67381e+02],
[4.92508e+01, 3.89991e+02, 2.48078e+02, 9.12459e+02],
[1.26507e+01, 2.23378e+02, 8.09707e+02, 7.88516e+02],
[4.53959e-02, 5.52411e+02, 6.78823e+01, 8.75375e+02]])}
from playtorch.
Thanks for sharing the inference code, @zhiqwang! That was incredibly helpful!
Here is a screencast of an early working version. I will need to finalize the changes and then make it available for everyone as soon as possible.
Thanks again for the model and the example!
yolov5s.mp4
from playtorch.
@zhiqwang Yes. And I think we still need to provide the live.spec.json as the extra file for generating ptl version. But, I don't think missing this json file will cause the error I mentioned above. It only effects library's data processing for input and output.
from playtorch.
Hello, I am also trying to create the spec.json for yolo_v5. How is your progress? @bairock
hachiko mode
from playtorch.
Do you have any idea about the unpack? Currently, I guess the output type should be set as object "models.common.Detections". About I don't think torch live recognizes this type as output according to tutorial.
from playtorch.
Any progress on exporting yolov5 to .ptl?
from playtorch.
We can take a look, but would need a working torchscript-ed model.
Does anyone have a working torchscript-ed version of a YOLOv5 model that accepts an image tensor as input and can share it here for download?
from playtorch.
Hi @raedle , It would be my pleasure and I can add a more detailed scripts for exporting the torchscript-ed module of YOLOv5 if you need it.
-
First, we can use the following script to verify the inference output between YOLOv5 and yolort.
Click to show the verification details
# Prepare environment, image and model weights to test import cv2 import torch from yolort.models.yolo import YOLO from yolort.utils import Visualizer, read_image_to_tensor from yolort.v5 import load_yolov5_model, letterbox, non_max_suppression, scale_coords, attempt_download device = torch.device('cpu') img_source = "https://huggingface.co/spaces/zhiqwang/assets/resolve/main/bus.jpg" # img_source = "https://huggingface.co/spaces/zhiqwang/assets/resolve/main/zidane.jpg" img_path = attempt_download(img_source) img_raw = cv2.imread(img_path) # Downloaded from 'https://github.com/ultralytics/yolov5/releases/download/v6.1/yolov5s.pt' model_path = 'yolov5s.pt' checkpoint_path = attempt_download(model_path) img_size = 640 stride = 32 score_thresh = 0.25 nms_thresh = 0.45 # Here we use the letterbox of YOLOv5 to do the image pre-processing. image = letterbox(img_raw, new_shape=(img_size, img_size), stride=stride)[0] image = read_image_to_tensor(image) image = image.to(device) image = image[None] ## Load model as ultralytics and inference model_yolov5 = load_yolov5_model(checkpoint_path, fuse=True) model_yolov5 = model_yolov5.to(device) model_yolov5 = model_yolov5.eval() with torch.no_grad(): yolov5_dets = model_yolov5(image)[0] yolov5_dets = non_max_suppression(yolov5_dets, score_thresh, nms_thresh, agnostic=False)[0] ## Updating model weights from ultralytics to yolort and inference model_yolort = YOLO.load_from_yolov5( checkpoint_path, score_thresh=score_thresh, nms_thresh=nms_thresh, version="r6.0", ) model_yolort = model_yolort.eval() model_yolort = model_yolort.to(device) with torch.no_grad(): yolort_dets = model_yolort(image) print(f"Detection boxes with yolort:\n{yolort_dets[0]['boxes']}") print(f"Detection scores with yolort:\n{yolort_dets[0]['scores']}") print(f"Detection labels with yolort:\n{yolort_dets[0]['labels']}") ## Verify the detection results between yolort and ultralytics # Testing boxes torch.testing.assert_allclose(yolort_dets[0]['boxes'], yolov5_dets[:, :4]) # Testing scores torch.testing.assert_allclose(yolort_dets[0]['scores'], yolov5_dets[:, 4]) # Testing labels torch.testing.assert_allclose(yolort_dets[0]['labels'], yolov5_dets[:, 5].to(dtype=torch.int64)) print("Exported model has been tested, and the result looks good!") # Detection output visualisation # Hah, that's the trick to rescale the box correctly. We need to scale the inference results back to the original scale of the image. scale_coords(image.shape[2:], yolort_dets[0]['boxes'], img_raw.shape[:-1]) # This only works on Jupyter now v = Visualizer(img_raw, model_yolov5.names) # Prepare the prediction labels for the Visualizer v.draw_instance_predictions(yolort_dets[0]) v.imshow(scale=0.5)
-
Then we could export the torchscript-ed module of YOLOv5 (containing the
nms
op)## Export the TorchScript-ed model import torch import torchvision from torch.utils.mobile_optimizer import optimize_for_mobile from yolort.models import YOLO from yolort.v5 import attempt_download # Prepare some parameters for the exported torchscript-ed module score_thresh = 0.25 nms_thresh = 0.45 device = torch.device("cpu") # Downloaded from 'https://github.com/ultralytics/yolov5/releases/download/v6.1/yolov5s.pt' model_path = "yolov5s.pt" checkpoint_path = attempt_download(model_path) model = YOLO.load_from_yolov5(checkpoint_path, score_thresh=score_thresh, nms_thresh=nms_thresh) model = model.eval() model = model.to(device) export_scripted_path = "yolov5s_scripted.pt" export_optimized_path = "yolov5s_scriptmodule.ptl" scripted_model = torch.jit.script(model) scripted_model.save(export_scripted_path) optimized_model = optimize_for_mobile(scripted_model) optimized_model._save_for_lite_interpreter(export_optimized_path)
-
Last, let's verify the inference result between the PyTorch and TorchScript
Click to show the verification details
scripted_model = torch.jit.load(export_scripted_path) scripted_model = scripted_model.eval() scripted_model = scripted_model.to(device) ## Verify the detection results between PyTorch and TorchScript with torch.no_grad(): out_script = scripted_model(image) scale_coords(image.shape[2:], out_script[1][0]['boxes'], img_raw.shape[:-1]) for k, v in yolort_dets[0].items(): torch.testing.assert_allclose(out_script[1][0][k], v, rtol=1e-07, atol=1e-09) print("Exported model has been tested with libtorch, and the result looks good!")
from playtorch.
Hi, do anyone know how to deploy a custom model into PyTorch live? I am encountering some bugs.
from playtorch.
@raedle I also used the yolot generate the .ptl file for yolov5, but received the error "Could not convert downloaded file into Torch Module". I compared my model to the example of MobileNetV3, and they are all RecursiveScriptModule. Since yolot and yolov5 have the same model structure, I am not sure the reason for this error. I checked the code of the package, and it said "TorchModule.load will set an empty string if the model file is not bundled inside the model file". Can you provide more explanation?
from playtorch.
Hi @JohnZcp ,
and it said "TorchModule.load will set an empty string if the model file is not bundled inside the model file".
May I ask do you use the yolort model with the pre-processing (letterbox
) part? I also encountered some problems when including pre-processing, I'm not quite sure why this is happening. But everything works fine when exporting yolov5 with only post-processing (nms
) as above (and I copied the detailed conversion script below).
## Export the TorchScript-ed model
import torch
import torchvision
from torch.utils.mobile_optimizer import optimize_for_mobile
from yolort.models import YOLO
from yolort.v5 import attempt_download
# Prepare some parameters for the exported torchscript-ed module
score_thresh = 0.25
nms_thresh = 0.45
device = torch.device("cpu")
# Downloaded from 'https://github.com/ultralytics/yolov5/releases/download/v6.1/yolov5s.pt'
model_path = "yolov5s.pt"
checkpoint_path = attempt_download(model_path)
model = YOLO.load_from_yolov5(checkpoint_path, score_thresh=score_thresh, nms_thresh=nms_thresh)
model = model.eval()
model = model.to(device)
export_scripted_path = "yolov5s_scripted.pt"
export_optimized_path = "yolov5s_scriptmodule.ptl"
scripted_model = torch.jit.script(model)
scripted_model.save(export_scripted_path)
optimized_model = optimize_for_mobile(scripted_model)
optimized_model._save_for_lite_interpreter(export_optimized_path)
from playtorch.
Closing this issue because it appears to be resolved. Feel free to re-open if more info is needed. Also note, there is now a YOLOv5 tutorial available: https://playtorch.dev/docs/tutorials/snacks/yolov5/
from playtorch.
Related Issues (20)
- PlayTorchJSIModule not found - ios & android m2 macbook pro HOT 2
- It is not possible to convert audio float32 tensor to playable uint8 audio expected by .audioFromBytes() HOT 1
- How to specify interpolation in transforms.resize?
- AudioUtil is not working on Android
- How to convert my yolov5s.pt to yolov5s.ptl HOT 1
- resize function gives error: 'Factory function resize expects 1 argument but 4 are given.'
- Cannot scan QR code from the PlayTorch app HOT 2
- playtorchjsimodule not found
- Mac M1 tutorial failure HOT 2
- unable to load local .ptl model HOT 2
- How to export a Huggingface transfromer model? HOT 3
- react_native_pytorch_core error during gradle build HOT 15
- image data fetch
- Is there a processing step im missing when using a custom trained yolo model?
- Is this project still maintained?
- Unable to work
- CMake Error
- Android build fails | Expo app using prebuild HOT 4
- 'react_native_pytorch_core-Swift.h' file not found during ios build HOT 1
- Add how to use EAS (Expo Application Service) build within this tutorial
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from playtorch.