Git Product home page Git Product logo

vladih / flutter_vision Goto Github PK

View Code? Open in Web Editor NEW
66.0 10.0 24.0 99.16 MB

A Flutter plugin for managing both Yolov5 model and Tesseract v4, accessing with TensorFlow Lite 2.x. Support object detection, segmentation and OCR on both iOS and Android.

Home Page: https://pub.dev/packages/flutter_vision

License: MIT License

Java 60.99% Swift 0.79% Objective-C 0.62% Dart 36.86% Ruby 0.74%
detection-model ocr-recognition segmentation-models yolov5 yolov8

flutter_vision's Introduction

flutter_vision

A Flutter plugin for managing Yolov5, Yolov8 and Tesseract v5 accessing with TensorFlow Lite 2.x. Support object detection, segmentation and OCR on Android. iOS not updated, working in progress.

Installation

Add flutter_vision as a dependency in your pubspec.yaml file.

Android

In android/app/build.gradle, add the following setting in android block.

    android{
        aaptOptions {
            noCompress 'tflite'
            noCompress 'lite'
        }
    }

iOS

Comming soon ...

Usage

For YoloV5 and YoloV8 MODEL

  1. Create a assets folder and place your labels file and model file in it. In pubspec.yaml add:
  assets:
   - assets/labels.txt
   - assets/yolovx.tflite
  1. Import the library:
import 'package:flutter_vision/flutter_vision.dart';
  1. Initialized the flutter_vision library:
 FlutterVision vision = FlutterVision();
  1. Load the model and labels: modelVersion: yolov5 or yolov8 or yolov8seg
await vision.loadYoloModel(
        labels: 'assets/labelss.txt',
        modelPath: 'assets/yolov5n.tflite',
        modelVersion: "yolov5",
        quantization: false,
        numThreads: 1,
        useGpu: false);

For camera live feed

  1. Make your first detection: confThreshold work with yolov5 other case it is omited.

Make use of camera plugin

final result = await vision.yoloOnFrame(
        bytesList: cameraImage.planes.map((plane) => plane.bytes).toList(),
        imageHeight: cameraImage.height,
        imageWidth: cameraImage.width,
        iouThreshold: 0.4,
        confThreshold: 0.4,
        classThreshold: 0.5);

For static image

  1. Make your first detection or segmentation:
final result = await vision.yoloOnImage(
        bytesList: byte,
        imageHeight: image.height,
        imageWidth: image.width,
        iouThreshold: 0.8,
        confThreshold: 0.4,
        classThreshold: 0.7);
  1. Release resources:
await vision.closeYoloModel();

For Tesseract 5.0.0 MODEL

  1. Create an assets folder, then create a tessdata directory and tessdata_config.json file and place them into it. Download trained data for tesseract from here and place it into tessdata directory. Then, modifie tessdata_config.json as follow.
{
    "files": [
      "spa.traineddata"
    ]
}
  1. In pubspec.yaml add:
assets:
    - assets/
    - assets/tessdata/
  1. Import the library:
import 'package:flutter_vision/flutter_vision.dart';
  1. Initialized the flutter_vision library:
 FlutterVision vision = FlutterVision();
  1. Load the model:
await vision.loadTesseractModel(
      args: {
        'psm': '11',
        'oem': '1',
        'preserve_interword_spaces': '1',
      },
      language: 'spa',
    );

For static image

  1. Get Text from static image:
    final XFile? photo = await picker.pickImage(source: ImageSource.gallery);
    if (photo != null) {
      final result = await vision.tesseractOnImage(bytesList: (await photo.readAsBytes()));
    }
  1. Release resources:
await vision.closeTesseractModel();

About results

For Yolo v5 or v8 in detection task

result is a List<Map<String,dynamic>> where Map have the following keys:

   Map<String, dynamic>:{
    "box": [x1:left, y1:top, x2:right, y2:bottom, class_confidence]
    "tag": String: detected class
   }

For YoloV8 in segmentation task

result is a List<Map<String,dynamic>> where Map have the following keys:

   Map<String, dynamic>:{
    "box": [x1:left, y1:top, x2:right, y2:bottom, class_confidence]
    "tag": String: detected class
    "polygons": List<Map<String, double>>: [{x:coordx, y:coordy}]
   }

For Tesseract

result is a List<Map<String,dynamic>> where Map have the following keys:

    Map<String, dynamic>:{
      "text": String
      "word_conf": List:int
      "mean_conf": int}

Example

Screenshot_2022-04-08-23-59-05-652_com vladih dni_scanner_example Home Detection Segmentation

Contact

For flutter_vision bug reports and feature requests please visit GitHub Issues


flutter_vision's People

Contributors

alvarocda avatar vladih avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

flutter_vision's Issues

Confidence always show 100%

Trained my model using colab and tested it. Never 100%. I tried to print(${result['box'][4]}) and it always show 0.999.. What do you think is the issue?

App size increase

"After adding the flutter_vision: ^1.1.4 plugin, the APK size increased significantly.
For instance, the previous Android APK size was 51MB, but after the addition, it exceeded 100MB. Can you suggest a solution to reduce the size?"

No "detection" on some phone

Hi, I've some issue to detect on some specific phone and maybe someone here have an idea why, and how to resolve the issue.
I'm doing Live Detection of vehicle with the Camera package + flutter_vision.

Both phone load the GPU delegate correctly (I had to upgrade flutter vision internal package for that - can see here if needed :
https://github.com/JobiJoba/flutter_vision
)

In this case I'm using a Oppo F11 Pro and here are the "debug print" of the detection:

image

As we can see, the width is 480 and the height 640 (From Camera Preview) and he resize to 640,640 and detect correctly.

Now with an Oppo Reno 7Z (newer phone)

image

the base size is 480,720 and he resize it to 640,640.
But here no detection (result at least)

In both case with have a result in model output, but he "can't" translate them into a correct result.

Here is the debug of both model_outputs variable:

image

The F11 pro also had some result on row 14 while Reno 7Z only on 3 first top.


I also tried different libraries:

flutter pytorch: Everything works, but no gpu acceleration so fps quite low ...
flutter_tflite: Same issue as here, can't detect anything with their base model (and they don't have yolov8 support).

If someone see something I don't see or an idea on what to look at :)

Unexpected change in YOLOv8 output format

Hi!

I used this package earlier this summer, and it worked just fine!
But ever since, any YOLOv8 model I try to export using Ultralytics exports will give an output format that is unrecognizable to the program as it previously was. The most recent updates to this package don't seem to make a difference, unfortunately.

Any ideas as to why this might be occurring, or how to address the issue? Let me know if you need more info for debuggin!

Example previous output (working):
[[1587.7335205078125, 693.9750366210938, 1921.768798828125, 1119.3153076171875, 0.9181058406829834], tag: obj},
{box: [2221.90576171875, 754.1986694335938, 2613.1064453125, 1153.677001953125, 0.3026130199432373], tag: obj},
{box: [2323.805419921875, 1268.9488525390625, 2681.66845703125, 1696.794677734375, 0.9503433704376221], tag: obj},
{box: [1531.5927734375, 1640.84130859375, 2018.4903564453125, 1948.836669921875, 0.6532635688781738], tag: obj}]

Example new output (broken):
[[1.239598274230957, 0.5401819348335266, 1.4939693212509155, 0.8794116377830505, 0.9612507224082947], tag: obj},
{box: [1.7374616861343384, 0.588816225528717, 2.0423240661621094, 0.9065394401550293, 0.9636867642402649], tag: obj},
{box: [1.8174190521240234, 0.9895595908164978, 2.099705457687378, 1.3236058950424194, 0.965178370475769], tag: obj},
{box: [1.195393681526184, 1.2803641557693481, 1.5751936435699463, 1.5251888036727905, 0.969660222530365], tag: obj}]

Slow image processing

Hello, I have trained a model and it is quite large
It works on old phones, but it's slow, and I appreciate this problem because the phone I'm using is old, and this is normal
But when I try the application on a very modern phone, the application is very slow in processing images
Although it should be much faster than the old phones

Yolov8 segmentation

Hi,

first of all I'd like to thank you for your work. I tried various ways (pytorch mobile via torchscript and tflite_flutter library) to get Yolov8 running in flutter but your library was the only one which worked for me with the default Yolov8 model.

So currently I am trying to get the Yolov8 segmentation running (using official yolov8-seg models). I figured out that the output shape of yolov8-seg differs from detection task's output.

Yolov8 CLI export to tflite prints: PyTorch: starting from yolov8n-seg.pt with input shape (1, 3, 640, 640) BCHW and output shape(s) ((1, 116, 8400), (1, 32, 160, 160)) (6.7 MB)

I added a logging statement to your load model method to see the shape in tflite files: Shapes retrieved: [Input tensor #0 has shape: [1, 640, 640, 3], Output tensor #0 has shape: [1, 116, 8400], Output tensor #1 has shape: [1, 160, 160, 32]].

This gives following output for default yolov8 model:
PyTorch: starting from yolov8n.pt with input shape (1, 3, 640, 640) BCHW and output shape(s) (1, 84, 8400) (6.2 MB)
Shapes retrieved: [Input tensor #0 has shape: [1, 224, 224, 3], Output tensor #0 has shape: [1, 84, 1029]].

As I am pretty new to machine learning I am a little bit confused why the loaded model has a lower image size and different output shape. Can you give me a hint on this?

So I went on and tried to modify the output output array in Yolo.java. Now the detection task doesn't give a message like java.lang.IllegalArgumentException: Cannot copy to a TensorFlowLite tensor (inputs_0) with 4915200 bytes from a Java Buffer with 602112 bytes. anymore but causes always an OutOfMemory error.

I'd really appreciate your support to get segmentation running!
Thanks in advance :)

How to I use customer tensorflow model with flutter_vision package ?

I am working on a flutter project that requires some machine learning capabilities. I have my model already trained and ready to be used.

I'm stuck at the level of integrating it to flutter. I have tried couple of packages that I found on pub.dev, but unfortunately I did not worked. Most of these packages are no longer maintained.

How do I use this package to interact with customer tensorflow lite model ?

The README was not clear about that!

How to use yolo8x

I converted it to tflite. But it gives an error like this

Cannot copy to a TensorFlowLite tensor (serving_default_images:0) with 4915200 bytes from a Java Buffer with 23040 bytes.

Bounding boxes are not centered on the object; they start halfway through the object.

The bounding boxes in the detection output are incorrectly positioned. Instead of centering around the detected object, they start halfway through it, resulting in inaccurate representations of the object’s location.

Click here to view the image

Code:

  List<Widget> displayBoxesAroundRecognizedObjects(Size screen) {
    if (yoloResults.isEmpty) return [];
    double factorX = screen.width / (cameraImage?.height ?? 1);
    double factorY = screen.height / (cameraImage?.width ?? 1);

    return yoloResults.map((result) {
      final label = result['tag'];
      final confidence = (result['box'][4] * 100).toInt(); // Remove decimal places and use integer value
      final color = Colors.pink; // You can use dynamic colors if needed

      double objectX = result["box"][0] * factorX;
      double objectY = result["box"][1] * factorY;
      double objectWidth = (result["box"][2] - result["box"][0]) * factorX;
      double objectHeight = (result["box"][3] - result["box"][1]) * factorY;

      return Positioned(
        left: objectX,
        top: objectY,
        width: objectWidth,
        height: objectHeight,
        child: Stack(
          children: [
            Container(
              decoration: BoxDecoration(
                borderRadius: const BorderRadius.all(Radius.circular(4.0)),
                border: Border.all(color: color, width: 3.0), // Thicker border similar to YOLOv design
              ),
            ),
            Positioned(
              bottom: 0,
              child: Container(
                padding: const EdgeInsets.symmetric(horizontal: 4.0, vertical: 2.0),
                color: color.withOpacity(0.7), // Background color for label
                child: Text(
                  "$label $confidence%",
                  style: TextStyle(
                    color: Colors.white,
                    fontSize: 14.0,
                    fontWeight: FontWeight.bold,
                  ),
                ),
              ),
            ),
          ],
        ),
      );
    }).toList();
  }

How to convert camera image to use for model using uint8 instead of float32

Hi,

if I run the example app from flutter_vision, it works fine. However I replaced the yolov8n model with a pretrained model (it detects licens plates) and then I get the following exception:

E/flutter (19641): [ERROR:flutter/runtime/dart_vm_initializer.cc(41)] Unhandled Exception: PlatformException(100, Detection Error, java.lang.IllegalArgumentException: Cannot copy to a TensorFlowLite tensor (normalized_input_image_tensor) with **270000** bytes from a Java Buffer with **1080000** bytes.
E/flutter (19641): 	at org.tensorflow.lite.TensorImpl.throwIfSrcShapeIsIncompatible(TensorImpl.java:416)
E/flutter (19641): 	at org.tensorflow.lite.TensorImpl.setTo(TensorImpl.java:140)
E/flutter (19641): 	at org.tensorflow.lite.NativeInterpreterWrapper.run(NativeInterpreterWrapper.java:246)
E/flutter (19641): 	at org.tensorflow.lite.InterpreterImpl.runForMultipleInputsOutputs(InterpreterImpl.java:133)
E/flutter (19641): 	at org.tensorflow.lite.Interpreter.runForMultipleInputsOutputs(Interpreter.java:80)
E/flutter (19641): 	at org.tensorflow.lite.InterpreterImpl.run(InterpreterImpl.java:126)
E/flutter (19641): 	at org.tensorflow.lite.Interpreter.run(Interpreter.java:80)
E/flutter (19641): 	at com.vladih.computer_vision.flutter_vision.models.Yolov8.detect_task(Yolov8.java:63)
E/flutter (19641): 	at com.vladih.computer_vision.flutter_vision.FlutterVisionPlugin$DetectionTask.run(FlutterVisionPlugin.java:273)
E/flutter (19641): 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:487)
E/flutter (19641): 	at java.util.concurrent.FutureTask.run(FutureTask.java:264)
E/flutter (19641): 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
E/flutter (19641): 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:644)
E/flutter (19641): 	at java.lang.Thread.run(Thread.java:1012)
E/flutter (19641): , null)

If I investigate the model with Netron, I see the model expects the input in the format uint8, as opposed to model from the example, which expects float32. That would explain the factor 4 when comparing 270000 with 1080000 (see Exception above).

Input format from example:

Bildschirmfoto 2024-06-03 um 06 53 04

Input format from my pretrained license_plate model (which throws exception above)

Bildschirmfoto 2024-06-03 um 08 42 23

However I cant figure out how to convert the YUV420 Camera image from android to the expected byte buffer.

Can somebody help me, I know this is not an issue with the library, Im just basically begging for help because Im stuck.

Thanks a lot,
Daniel

get_polygons_from_bitmap method, where a MatOfPoint object is expected but is null.

I got this error for some image.

E/flutter (12120): [ERROR:flutter/runtime/dart_vm_initializer.cc(41)] Unhandled Exception: PlatformException(100, Detection Error, java.lang.NullPointerException: Attempt to invoke virtual method 'java.util.List org.opencv.core.MatOfPoint.toList()' on a null object reference
E/flutter (12120):      at com.vladih.computer_vision.flutter_vision.models.Yolov8Seg.get_polygons_from_bitmap(Yolov8Seg.java:197)
E/flutter (12120):      at com.vladih.computer_vision.flutter_vision.models.Yolov8Seg.restore_seg_mask_size(Yolov8Seg.java:165)
E/flutter (12120):      at com.vladih.computer_vision.flutter_vision.models.Yolov8Seg.detect_task(Yolov8Seg.java:99)       
E/flutter (12120):      at com.vladih.computer_vision.flutter_vision.FlutterVisionPlugin$DetectionTask.run(FlutterVisionPlugin.java:273)
E/flutter (12120):      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:457)
E/flutter (12120):      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
E/flutter (12120):      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1162)
E/flutter (12120):      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:636)
E/flutter (12120):      at java.lang.Thread.run(Thread.java:764)
E/flutter (12120): , null)
E/flutter (12120): #0      StandardMethodCodec.decodeEnvelope (package:flutter/src/services/message_codecs.dart:652:7)     
E/flutter (12120): #1      MethodChannel._invokeMethod (package:flutter/src/services/platform_channel.dart:310:18)
E/flutter (12120): <asynchronous suspension>
E/flutter (12120): #2      AndroidFlutterVision._yoloOnImage (package:flutter_vision/src/plugin/android.dart:189:17)       
E/flutter (12120): <asynchronous suspension>
E/flutter (12120): #3      AndroidFlutterVision.yoloOnImage (package:flutter_vision/src/plugin/android.dart:168:14)        
E/flutter (12120): <asynchronous suspension>
E/flutter (12120): #4      _PotholeDetectionPageState.yoloOnImage (package:pothole_detection_app/nav_pages/pothole_detection_page_view.dart:286:20)
E/flutter (12120): <asynchronous suspension>
E/flutter (12120):

How to handle potrait image?

List displayBoxesAround(Size screen, {required bool isPotrait}) {
if (yoloResults.isEmpty) return [];

// Calculate scaling factors based on image orientation
double factorX, factorY, imgRatio, newWidth, newHeight;
if (isPotrait) {
  factorX = screen.height / (imageHeight);
  imgRatio = imageWidth / imageHeight;
  newWidth = imageWidth * factorX;
  newHeight = newWidth / imgRatio;
  factorY = newHeight / (imageHeight);
} else {
  // Handle landscape image
  factorX = screen.width / (imageWidth);
  imgRatio = imageWidth / imageHeight;
  newWidth = imageWidth * factorX;
  newHeight = newWidth / imgRatio;
  factorY = newHeight / (imageHeight);
}

// Calculate padding for centered positioning
double paddingX = (screen.width - factorX * imageWidth) / 2;
double paddingY = (screen.height - factorY * imageHeight) / 2;

Color colorPick = const Color.fromARGB(255, 50, 233, 30);
return yoloResults.map((result) {
  return Positioned(
    left: paddingX + result["box"][0] * factorX,
    top: paddingY + result["box"][1] * factorY,
    width: (result["box"][2] - result["box"][0]) * factorX,
    height: (result["box"][3] - result["box"][1]) * factorY,
    child: Container(
      decoration: BoxDecoration(
        borderRadius: const BorderRadius.all(Radius.circular(10.0)),
        border: Border.all(color: Colors.pink, width: 2.0),
      ),
      child: Text(
        "${result['tag']} ${(result['box'][4] * 100).toStringAsFixed(0)}%",
        style: TextStyle(
          background: Paint()..color = colorPick,
          color: Colors.white,
          fontSize: 18.0,
        ),
      ),
    ),
  );
}).toList();

}

OCR accuracy is very low.

Image:

tt

Flutter vision tesseract:

tt2

Pytesseract:

| мы ааеоог лоно о четоотео м пол олам а на
| курица №
7у9214-003-50301584-2014 :
изделие кулинарное из мяса птицы с гарниром охлаждённое:
шницель куриный в сырной панировке с макаронами отварными
масса нет го: 230 г
состав: шницель куриниий в сырной панировуе (куриное филе грудки, сухари панировочные (мука!
пшеничная хлебопекаг нгя высшего сорта, вла питьевая, дрожжи хлебопекарные прессованные), масло!
подсолнечное ргфини ›с занное лезодорир в энное, меланж яичный жидкий пастеризованный, сыр твёрдый
«пармезана м. д. ж.в г уз ом вещ:стве 40 % ‚ элоко нормализованное пастеризованное, закваска кульур
молочнокислых микросргани: мов, молоко = ртывающий ферментный препарат микробного. др
происхождения, сель иищев я, уплотните: › - хлорид кальция), мука пшеничная хлебопекар шего
сорта, вода пить. е ая, лук г ‘г чатый свежу ^ соль пищевая, перец белый молотый), макароны отварные
(макаронные у ›/ лия гру | а высшего ‚ ‚рта (мука из твёрдой пшеницы для макаронных изделий, вода
питьевая), мг оливкс ›" г рафиниров: ное, соль пищевая). кзводится на предпц
используют 7 арахис, соло поро ола ия родукты их пер
пищевая це - ность 100. г продукта (средние значения): белки - 12,4 г; жиры - 13.0 квт.ч,
кислоты - 1,6 вт ч.трасизомеры жирных кислот =, г; углеводы = 24, квт, саха
добавленный сахар = 0 г; соль = 12 г. ‘зе
энергетическая ценность; 263,4 ккал / 1103,2 кдж. маи
хранить при температуре от +2 до +6 °с. употребить сразу после вск]
в газомодифицированной атмосфере (упаковочный газ: 70 % = №,
раноцаця по приди ооо ‹роволновой п
© открытой плёнкой. продукт готов к употреблению,
изготовитель: общество с ограниченной о 0
московская область, город химки, проезд северный, кор!
по претензиям и пожеланиям пишите и звоните нам;
мляки$ у! ги, +7 (495) 663-86-02, |
срок годности: 4 суток, ‚й
дата изготовления и упаковывания: |

pytesseract preprocessing: Only grayscale

tesseract configuration: psm 6, oem 1

yoloOnFrame using static image

Hello, I would like to know if it is possible to identify objects from a static image (gallery image for example).
As far as I've seen, apparently it's only possible to use the camera feed.

If possible, can you provide an example?

Please help me with getting coordinates on image

Here is the code which I provided to detect some text on images then, I want to crop the image and display only the region of the image which is detected by yolov8 model, but it does not work properly, and crop somewhere else:
`
yoloOnImage() async {
yoloResults.clear();
Uint8List byte = await imageFile!.readAsBytes();
img.Image? originalImage = img.decodeImage(byte);

final result = await widget.vision.yoloOnImage(
    bytesList: byte,
    imageHeight: originalImage!.height,
    imageWidth: originalImage.width,
    iouThreshold: 0.2,
    confThreshold: 0.2,
    classThreshold: 0.2);
if (result.isNotEmpty) {
  // Sort the results by confidence score in descending order
  result.sort((a, b) => b["confidence"].compareTo(a["confidence"]));

  // Keep only the result with the highest confidence score
  var bestResult = result[0];
  print(bestResult);
  // Assuming you have the bounding box coordinates
  double x = bestResult["box"]
      [0]; // x coordinate of the top left corner of the bounding box
  double y = bestResult["box"]
      [1]; // y coordinate of the top left corner of the bounding box
  double w = bestResult["box"][2]; // width of the bounding box
  double h = bestResult["box"][3]; // height of the bounding box

  // Convert normalized coordinates to pixel coordinates
  int left = (x * originalImage.width).round();
  int top = (y * originalImage.height).round();
  int width = (w * originalImage.width).round();
  int height = (h * originalImage.height).round();

  img.Image cropped = img.copyCrop(
    originalImage,
    x: left,
    y: top,
    width: width,
    height: height,
  );

  // Get the temporary directory path.
  final tempDir = await getTemporaryDirectory();

  // Create a temporary file with a unique name.
  final file =
      await File('${tempDir.path}/${DateTime.now().toIso8601String()}.png')
          .create();

  // Write the bytes of the cropped image to the file.
  await file.writeAsBytes(img.encodePng(cropped));

  setState(() {
    yoloResults = [bestResult];
    imageFile = file;
  });
}

}
}
`

Android API 34 problem

I have notice, that if Android emulator or real device have higher API than 33, that flutter app fails, not possible to use flutter_vision. Have You also noticed this issue?

ios support

Do you have a timeframe for when this package will support ios?

in last update 1.1.4

All readings from the model are the same

[{box: [720.0, 1280.0, 720.0, 1280.0, 0.5922492742538452], tag: 5}

Costome yolo model not work

[ERROR:flutter/runtime/dart_vm_initializer.cc(41)] Unhandled Exception: PlatformException(100, Error on load Yolov5 model, java.lang.IllegalArgumentException: Internal error: Cannot create interpreter: Following operations are not supported by GPU delegate:
E/flutter (12733): DELEGATE TfLiteFlexDelegate: Operation is not supported.
E/flutter (12733): DEQUANTIZE:
E/flutter (12733): 35 operations will run on the GPU, and the remaining 340 operations will run on the CPU.
E/flutter (12733): Can not open OpenCL library on this device - dlopen failed: library "libOpenCL.so" not found
E/flutter (12733): Falling back to OpenGL
E/flutter (12733): TfLiteGpuDelegate Init: No shader implementation for transpose
E/flutter (12733): TfLiteGpuDelegate Prepare: delegate is not initialized

IOS version ?

Hi @vladiH , thanks for this wonderful plugin. I want to know when this package is going to support the IOS version. ?

Live detection example not detecting anything

Hi,

I'm trying to run the example in this project and when using the live camera, he does not detect anything.
Using the picture mode works correctly.

Any idea what could be wrong? Does it works on your side?

Thanks

two models at once

hey, because of budget consideration I'm forced to run two yolo models at the same time (instead of training one that combines them).

it it possible with flutterVision?

from what i've seen for some reason it only inference the second model loaded

Future<void> loadYoloModel() async {
    await vision1.loadYoloModel(
      labels: 'assets/labels1.txt',
      modelPath: 'assets/model1.tflite',
      modelVersion: "yolov5",
      numThreads: 4,
      useGpu: true,
    );

    await vision2.loadYoloModel(
      labels: 'assets/labels2.txt',
      modelPath: 'assets/model2.tflite',
      modelVersion: "yolov5",
      numThreads: 4,
      useGpu: true,
    );
  }

YOLO Results documentation in README

Hi @vladiH,
Kindly have a look at the coordinate representation of the box.
I think this should be the correct representation [x1:left, y1:top, x2:right, y2:bottom].
I don't know if this is right with respect to your package, but this is what I have found to work.
I hope you consider this request and make the possible changes needed.
Many thanks

   Map<String, dynamic>:{
    "box": [x1:top, y1:left, x2:bottom, y2:right, class_confidence]
    "tag": String: detected class
   }

Please help me with getting the right coordinates of some detected region on images

I am sorry that I have asked for this several times, but I could not solve this issue. Bellow is the code which is taken from flutter_vision package example code, please help me to crop the image after detecting the region of interest with yolov8 model.
yoloOnImage() async {
yoloResults.clear();

Uint8List byte = await imageFile!.readAsBytes();
final image = await decodeImageFromList(byte);
imageHeight = image.height;
imageWidth = image.width;
final result = await widget.vision.yoloOnImage(
    bytesList: byte,
    imageHeight: image.height,
    imageWidth: image.width,
    iouThreshold: 0.8,
    confThreshold: 0.4,
    classThreshold: 0.5);
if (result.isNotEmpty) {
  setState(() {
    yoloResults = result;
  });
}

}
`

YoloV9 support

I'm trying to add support to yolov9, if anyone can help and wants to join forces I'm here.

Switching Between Models

When I call a function to change my detection model in UI, (for example from yolov8 to yolov5) I get the error:
[ERROR:flutter/runtime/dart_vm_initializer.cc(41)] Unhandled Exception: PlatformException(100, Detection Error, java.lang.IllegalArgumentException: Cannot copy to a TensorFlowLite tensor (serving_default_input_1:0) with 4915200 bytes from a Java Buffer with 602112 bytes.

Error remains even though I load the model from scratch. Also tried to create different FlutterVision instances for each model but the problem remains. Is there any way to switch between models ?

Thank you!

Pixel support

I tried out the example screen on my phone.
I had some issues getting it started. The tflite model you've provided was working for me only when I set the resolution to low. However, when I used a yolov5n model which I had converted to tflite myself, it was working with medium resolution. It was not smooth however, phone started heating up quite fast and after around 1min it was stuttering. I have a pixel 7 pro. I also noticed these lines in the log:

W/libc    (21736): Access denied finding property "ro.mediatek.platform"
W/libc    (21736): Access denied finding property "ro.chipname"
W/libc    (21736): Access denied finding property "ro.hardware.chipname"
I/tflite  (21736): Created TensorFlow Lite XNNPACK delegate for CPU.

Is my problem because of the app using CPU instead of the GPU?
I am looking for some clarity on this. Hope I get a response soon!

Realtime object detection use camera plugin and best-fp16.tflite model

Hello, I'm currently trying to learn to use flutter_vision and the yolov5 model (best-fp16.tflite) that is in the example in this package and follow the steps and connect it to the camera. I've tried to point the camera at some of the objects listed in labels.txt (fingerprint/card), but the detection results give me null.

Is the way I connect yolov5 (best-fp16.tflite) with the camera wrong? or because the object that I pointed at the camera is not right? or maybe because of something else?

I don't have a yolov5 model, I'd like to learn how to build a model if it connects successfully.

image

image

Cropping the detected region

Please help me to crop the region of interest in flutter(Only the part of image where the model detected).
I have trained a yolov8 model that detect some specific text on drug image, now I want to crop the image and display only the detected region not all image, but right now I can not crop properly the region, and I have trouble. It crop but other region not the region of interest

The application shows a crash message, but if you wait, it continues to work.

app

Log:

I/.tess.tesserac(23210): Thread[5,tid=24331,WaitingInMainSignalCatcherLoop,Thread*=0x79b0db6c00,peer=0x13280250,"Signal Catcher"]: reacting to signal 3
I/.tess.tesserac(23210): 
I/.tess.tesserac(23210): Wrote stack traces to tombstoned

YOLO version: 5
Phone: Samsung Galaxy M31
How to reproduce the bug: Make photo of image with many text

Please help me with cropping images

Please update the bellow code to crop the detected region with yolov8 model, because right now it draws bounding box around the detected object, but I need only the detected region, and I want to display it on screen instead of the whole image.

`
List displayBoxesAroundRecognizedObjects(Size screen) {
if (yoloResults.isEmpty) return [];

double factorX = screen.width / (imageWidth);
double imgRatio = imageWidth / imageHeight;
double newWidth = imageWidth * factorX;
double newHeight = newWidth / imgRatio;
double factorY = newHeight / (imageHeight);

double pady = (screen.height - newHeight) / 2;

Color colorPick = const Color.fromARGB(255, 50, 233, 30);
return yoloResults.map((result) {
  return Positioned(
    left: result["box"][0] * factorX,
    top: result["box"][1] * factorY + pady,
    width: (result["box"][2] - result["box"][0]) * factorX,
    height: (result["box"][3] - result["box"][1]) * factorY,
    child: Container(
      decoration: BoxDecoration(
        borderRadius: const BorderRadius.all(Radius.circular(10.0)),
        border: Border.all(color: Colors.pink, width: 2.0),
      ),
      child: Text(
        "${result['tag']} ${(result['box'][4] * 100).toStringAsFixed(0)}%",
        style: TextStyle(
          background: Paint()..color = colorPick,
          color: Colors.white,
          fontSize: 18.0,
        ),
      ),
    ),
  );
}).toList();

}
}
`

Bounding Boxes are off

Followed readme and the example provided. The resulting bounding boxes are out of place for my custom trained yolov5 model (and yolov8 model). Also the confidence scores are always very high.
` List displayBoxesAroundRecognizedObjects(Size screen) {
if (yoloResults.isEmpty) return [];

double factorX = screen.width / (cameraImage?.height ?? 1);
double factorY = screen.height / (cameraImage?.width ?? 1);

// print("screen width"+ screen.width.toString());
// print("screen height"+ screen.height.toString());
// print("camera image width");
// print(cameraImage?.width);
// print("camera image height");
// print(cameraImage?.height);
//
// print("factor X"+ factorX.toString());
// print("factor Y"+ factorY.toString());

Color colorPick = const Color.fromARGB(255, 50, 233, 30);

return yoloResults.map((result) {
  return Positioned(
    left: result["box"][0] * factorX,
    top: result["box"][1] * factorY,
    right: result["box"][2] * factorX,
    bottom: result["box"][3] * factorY,
    child: Container(
      decoration: BoxDecoration(
        borderRadius: const BorderRadius.all(Radius.circular(10.0)),
        border: Border.all(color: Colors.pink, width: 2.0),
      ),
      child: Text(
        "${result['tag']} ${(result['box'][4] * 100).toStringAsFixed(0)}%",
        style: TextStyle(
          background: Paint()..color = colorPick,
          color: Colors.white,
          fontSize: 18.0,
        ),
      ),
    ),
  );
}).toList();

}
`

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.