tucan9389 / semanticsegmentation-coreml Goto Github PK
View Code? Open in Web Editor NEWThe example project of inferencing Semantic Segementation using Core ML
Home Page: https://github.com/motlabs/awesome-ml-demos-with-ios
License: MIT License
The example project of inferencing Semantic Segementation using Core ML
Home Page: https://github.com/motlabs/awesome-ml-demos-with-ios
License: MIT License
In the performance comparison chart, what is total time (cpu) and how is it measured?
is there any way to extract image from segmentation map or how to convert MLMultiArray to image?
Hi !
I wonder if the metal workflow you are implementing in the repo is compatible with ARKIT ?
https://github.com/zllrunning/face-parsing.PyTorch
['background', 'skin', 'l_brow', 'r_brow', 'l_eye', 'r_eye', 'eye_g', 'l_ear', 'r_ear', 'ear_r', 'nose', 'mouth', 'u_lip', 'l_lip', 'neck', 'neck_l', 'cloth', 'hair', 'hat']
import torch
import os.path as osp
import json
from PIL import Image
import torchvision.transforms as transforms
from model import BiSeNet
import coremltools as ct
dspth = 'res/test-img'
cp = '79999_iter.pth'
device = torch.device('cpu')
output_mlmodel_path = "FaceParsing.mlmodel"
labels = ['background', 'skin', 'l_brow', 'r_brow', 'l_eye', 'r_eye', 'eye_g', 'l_ear', 'r_ear', 'ear_r',
'nose', 'mouth', 'u_lip', 'l_lip', 'neck', 'neck_l', 'cloth', 'hair', 'hat']
n_classes = len(labels)
print("n_classes:", n_classes)
class MyBiSeNet(torch.nn.Module):
def __init__(self, n_classes, pretrained_model_path):
super(MyBiSeNet, self).__init__()
self.model = BiSeNet(n_classes=n_classes)
self.model.load_state_dict(torch.load(pretrained_model_path, map_location=device))
self.model.eval()
def forward(self, x):
x = self.model(x)
x = x[0]
x = torch.argmax(x, dim=1)
x = torch.squeeze(x)
return x
pretrained_model_path = osp.join('res/cp', cp)
model = MyBiSeNet(n_classes=n_classes, pretrained_model_path=pretrained_model_path)
model.eval()
example_input = torch.rand(1, 3, 512, 512) # after test, will get 'size mismatch' error message with size 256x256
preprocess = transforms.Compose([
transforms.ToTensor(),
transforms.Normalize(
mean=[0.485, 0.456, 0.406],
std=[0.229, 0.224, 0.225],
),
])
traced_model = torch.jit.trace(model, example_input)
# Convert to Core ML using the Unified Conversion API
print(example_input.shape)
scale = 1.0 / (0.226 * 255.0)
red_bias = -0.485 / 0.226
green_bias = -0.456 / 0.226
blue_bias = -0.406 / 0.226
mlmodel = ct.convert(
traced_model,
inputs=[ct.ImageType(name="input",
shape=example_input.shape,
scale=scale,
color_layout="BGR",
bias=[blue_bias, green_bias, red_bias])], #name "input_1" is used in 'quickstart'
)
labels_json = {"labels": labels}
mlmodel.user_defined_metadata["com.apple.coreml.model.preview.type"] = "imageSegmenter"
mlmodel.user_defined_metadata['com.apple.coreml.model.preview.params'] = json.dumps(labels_json)
mlmodel.save(output_mlmodel_path)
import coremltools.proto.FeatureTypes_pb2 as ft
spec = ct.utils.load_spec(output_mlmodel_path)
for feature in spec.description.output:
if feature.type.HasField("multiArrayType"):
feature.type.multiArrayType.dataType = ft.ArrayFeatureType.INT32
ct.utils.save_spec(spec, output_mlmodel_path)
Model | Size | Minimum iOS Version |
---|---|---|
DeepLabV3 | 8.6 | iOS12 |
DeepLabV3FP16 | 4.3 | iOS12 |
DeepLabV3Int8LUT | 2.3 | iOS12 |
Model vs. Device | XS | X |
---|---|---|
DeepLabV3 | 135 | 177 |
DeepLabV3FP16 | 136 | 177 |
DeepLabV3Int8LUT | 135 | 177 |
Model vs. Device | XS | X |
---|---|---|
DeepLabV3 | 409 | 531 |
DeepLabV3FP16 | 403 | 530 |
DeepLabV3Int8LUT | 412 | 517 |
Model vs. Device | XS | X |
---|---|---|
DeepLabV3 | 2 | 1 |
DeepLabV3FP16 | 2 | 1 |
DeepLabV3Int8LUT | 2 | 1 |
Hi there - first off, great work on this repo! :D
I wonder if there's a way to segment the whole image by padding the sides - as it stands, since imageCropAndScaleOption
is .centerCrop
, we only get the center.
I'm searching for this issue slowly. https://stackoverflow.com/questions/61154192/are-there-recommended-ways-for-drawing-2d-array-in-fast-time
Now:
Inference -> convert to primitive array from mlarray -> draw the converted array
Suggest:
Inference -> draw the mlarray
Hi, sorry the this issue to ask for help.
The input image is 1920 * 1080 pixel, and the result is 513 * 513 array, renderView is 300 * 300 pt.
Why the renderView is rendered normal even the input is not suitable?
And how can I add mosaic on the original image depend on the segmentation?
Hi, Thanks for the great repo!
Let's imagine the coreml network needs a cropped image (only a subpart of the camera feed). No matter how the crop is done (it can be hard coded for testing purpose), I wonder if there is a way to change the DrawingSegmentationView
for example, to achieve this. Right now, if the input image is cropped, the output view resize the image to the viewport and the result isn't well registered.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.