Git Product home page Git Product logo

peculiarventures / gammacv Goto Github PK

View Code? Open in Web Editor NEW
171.0 17.0 21.0 27.25 MB

GammaCV is a WebGL accelerated Computer Vision library for browser

Home Page: https://gammacv.com

License: MIT License

JavaScript 70.90% GLSL 10.84% Shell 0.18% Sass 3.78% TypeScript 14.30%
computer-vision machine-learning webgl image-processing image-analysis object-detection feature-extraction gpu-acceleration gpu machine-vision

gammacv's Introduction

GammaCV Logo

Lines Extraction Example

GammaCV is a WebGL accelerated Computer Vision library for modern web applications.

We created GammaCV to make it easy to integrate Computer Vision in modern web applications. GammaCV was built with a similar architecture to TensorFlow, in particular, it also uses a data flow paradigm to create and run graphs on GPU, this enables the robust image processing and feature extraction capability of the library.

CircleCI

Installation

To install the stable version:

npm install --save gammacv

This assumes you are using npm as your package manager.

Examples

Documentation

You can find the GammaCV documentation on the website.

Want to help?

Want to file a bug, contribute some code, or improve documentation? Excellent! Read up on our guidelines for contribution.

Core Contributors

Arkadiy Pilguk, Mihail Zachepilo, Dmitriy Donskov, Ihor Pchelnik, Rostyslav Kravchenko, Andrii Omelianenko

gammacv's People

Contributors

adamdottv avatar andromelianenko avatar apilguk avatar bonn2018 avatar dependabot[bot] avatar donskov avatar jamesmfriedman avatar kharya1337 avatar mariktar avatar p1nta avatar rmhrisk avatar rostyslavkravchenko avatar silvertracer avatar snyk-bot avatar worldthirteen avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

gammacv's Issues

[demo help] GammaCV for webcam

Hello,

Earlier I asked about some assistance on a demo using the webcam, in particular I'm trying to use the the pixelwise_math "sub" example in a project. I'm using a single file vue component.

I was trying to piece it together from pieces of these two files:

Any assistance would be great! Here's the script part of the component I tossed together quickly but its obviously not functional. I seem to get valid instances from gm.Tensor and gm.Session and gm.CaptureVideo works fine but my updateVideoCanvas function throws an error when it hits the lines around gm.tensorClone or gm.canvasFromTensor.

<script>
let gm;
if (process.browser) {
  gm = require("gammacv");
}

export default {
  data() {
    return {
      currFrameTensor: undefined,
      prevFrameTensor: undefined,
      outputTensor: undefined,
      modifiedCanvas: undefined,
      stream: undefined,
      session: undefined,
      operation: undefined
    };
  },
  mounted() {
    this.modifiedCanvas = this.$refs.modifiedCanvasEl;

    // create the tensors with the dimensions of the canvas the output will go to
    this.currFrameTensor = new gm.Tensor("uint8", [
      this.modifiedCanvas.clientWidth,
      this.modifiedCanvas.clientHeight,
      4
    ]);

    this.prevFrameTensor = new gm.Tensor("uint8", [
      this.modifiedCanvas.clientWidth,
      this.modifiedCanvas.clientHeight,
      4
    ]);
  },
  methods: {
    startCam() {
      this.session = new gm.Session();
      this.stream = new gm.CaptureVideo(
        this.modifiedCanvas.clientWidth,
        this.modifiedCanvas.clientHeight
      );
      this.operation = gm.sub(this.currFrameTensor, this.prevFrameTensor);

      // start video from CaptureVideo() and run RAF loop
      this.stream
        .start()
        .then(() => {
          this.outputTensor = gm.tensorFrom(this.operation);
          this.session.init(this.operation);
          this.stream.getImageBuffer(this.currFrameTensor);
          this.updateVideoCanvas();
        })
        .catch(() => {
          this.stop(true);
        });
    },
    updateVideoCanvas() {
      gm.clearCanvas(this.modifiedCanvas);
      gm.tensorClone(this.currFrameTensor, this.prevFrameTensor);
      this.stream.getImageBuffer(this.currFrameTensor);
      gm.tensorClone(this.currFrameTensor, this.prevFrameTensor);
      gm.canvasFromTensor(this.modifiedCanvas, this.currFrameTensor);
      requestAnimationFrame(this.updateVideoCanvas);
    }
  }
};
</script>

Error handling for uninitialized operation

Sometimes happens to me I forgetting to initialize operation for a specific session sess.init(op) and then trying to run it sess.runOp(op, frame, out). Currently, we have no any message and error is really unclear what is happening

HOG demo broken in desktop safari

Hello, I tested the HOG demo on the gammacv website on Safari on my 2016 Macbook Pro. On chrome it works great, same with mobile safari. On desktop safari though, it struggles to show 1 frame every 5-10 seconds because of this WebGL error.

Attached is a screenshot of the WebGL error:
Screen Shot 2019-03-13 at 10 32 02 AM

Error with pcLines

I have this code

const sess = new gm.Session();
const input = await gm.imageTensorFromURL(
  '/image.jpg',
  'uint8',
  [width, height, 4],
  true
);

let operation = gm.grayscale(input);
operation = gm.downsample(input, 2, 'max');
operation = gm.gaussianBlur(input, 30, 1);
operation = gm.dilate(input, [3, 3]);
operation = gm.adaptiveThreshold(input, 3, 5);
operation = gm.sobelOperator(input);
operation = gm.cannyEdges(input, 0.8, 1);
operation = gm.pcLines(input, 10, 2, 2);

const output = gm.tensorFrom(operation);
sess.init(operation);
sess.runOp(operation, 0, output);

if I run this code I get the error Tensor: Shape is not valid.

If I change the output to be const output = gm.tensorFrom(input); I get the error KernelConstructor: Constant W, has invalid type "number".

None of these errors happen if I remove the pcLines call. What can be the cause?

Floating Point operations on iOS / Safari

Your library is my last hope in solving my problems for this app.

Basically, I'm trying to do an extraction of a card using hough lines and finding the intersection of the points. A few things.

  • I'm still very new to computer vision, so some of the more intensive topics are lost on me.
  • The best I can deduce is that there is a floating point issue with WebGL on safari.
  • You can see from my screenshots, chrome is "mostly" detecting the appropriate lines. I am able to filter the bad ones out to get to the result I need.
  • Safari is all over the place and doesn't seem to work consistently.
  • The same behavior can be seen on examples page if you open chrome vs safari. https://gammacv.com/examples/pc_lines

Good Example (Chrome), I'm able to extract the license using the pclines.
image

Bad Example (Safari). Same exact code, wildly different result. You'll notice the lines (in the top right image) don't seem to correspond to anything.

image

Any help you could possibly give would be amazing. I wasted several weeks trying to get openCV js working before finally giving up. Found your library after, was able to get to the same point I was at with openCV in half a day, and the performance is incredible.

Local tests does't work on Chrome >= 96

While tests successfully pass on CI, I wasn't able to run them correctly on my local machine.
GL context is null in headless chrome mode.

Device info:

  • MacOS 13.1 (22C65)
  • CPU: M1 Pro
  • ChromeVersion 109.0.5414.119 (Official Build) (arm64)

Steps to reproduce:

  • install dependencies
  • run tests with npm run test
  • Get 17+ tests failed with error (sample)
Chrome Headless 109.0.5414.119 (Mac OS 10.15.7) Adaptive Threshold "before each" hook for "binarize grayscale image" FAILED
        Error: Session: WebGL not supported.

Seems related to:

The use of eval causes problems for bundlers and minification, use new Function instead.

In the functions tensorInvert and flipTensor you are using the eval function. The way you are using it could, I think, be replaced by a new Function. The use of eval can cause, in particular, issues for Rollup (which throws a few warnings if gammaCV is imported) - see for more information here. Given that you are actually using eval to create a function it would also seem better practice to use new Function.

Cropping

I'd like to use GammaCV to crop an imag on the GPU. So far I only found perspectiveProjection which I can use for cropping. My guess is that this wouldn't be as fast as a simpler cropping operation.
Is there a better way to do this? I'm new to computer vision & not very familiar with graphics programming.

Types mistake

Current Behavior

Method start of CaptureVideo class returns type void.

export class CaptureVideo {
  ...
  start(deviceID?: string, exactFacingMode?: string): void;
}

https://github.com/PeculiarVentures/GammaCV/blob/master/lib/index.d.ts#L11

Expected Behavior

Method start of CaptureVideo returns type Promise<void>.

Steps to Reproduce

const stream = new gm.CaptureVideo(100, 100);

console.og(stream.start() instanceof Promise) // => true

Question: Crash on capture

I'm not sure this is related to GammaCV, hence put in as a question rather than an issue report. On executing the following:

  const im_mat = new gm.MediaInput(video, [video.videoHeight, video.videoWidth, 4])
  const operation = gm.upsample(im_mat, 1)
  const tensor = gm.tensorFrom(im_mat)
  const sess = new gm.Session()
  sess.init(operation)
  sess.runOp(operation, 0, tensor)

I'll get the following crash very intermittently:
I've seen a few reports, it seems to be related to attempting to lock the video stream itself. It happens more often on slower devices. This specifically happened on mobile firefox on a Pixel 7, but it also happens on Chrome.

Abort message: 'FORTIFY: pthread_mutex_lock called on a destroyed mutex (0x7565f75238)
02-13 10:27:17.908 23727 23765 F libc    : Fatal signal 6 (SIGABRT), code -1 (SI_QUEUE) in tid 23765 (binder:23727_3),
pid 23727 (main)
02-13 10:27:18.617 30061 30061 F DEBUG   : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
02-13 10:27:18.617 30061 30061 F DEBUG   : Build fingerprint: 'google/panther/panther:14/UQ1A.240105.004/11206848:user/
release-keys'
02-13 10:27:18.617 30061 30061 F DEBUG   : Revision: 'MP1.0'
02-13 10:27:18.617 30061 30061 F DEBUG   : ABI: 'arm64'
02-13 10:27:18.617 30061 30061 F DEBUG   : Timestamp: 2024-02-13 10:27:18.023913426-0800
02-13 10:27:18.617 30061 30061 F DEBUG   : Process uptime: 445s
02-13 10:27:18.617 30061 30061 F DEBUG   : Cmdline: app_process /data/local/tmp/.studio com.android.tools.screensharing
.Main --socket=screen-sharing-agent-63068 --max_size=1750,3118 --codec=vp8
02-13 10:27:18.617 30061 30061 F DEBUG   : pid: 23727, tid: 23765, name: binder:23727_3  >>> app_process <<<
02-13 10:27:18.617 30061 30061 F DEBUG   : uid: 2000
02-13 10:27:18.617 30061 30061 F DEBUG   : tagged_addr_ctrl: 0000000000000001 (PR_TAGGED_ADDR_ENABLE)
02-13 10:27:18.617 30061 30061 F DEBUG   : signal 6 (SIGABRT), code -1 (SI_QUEUE), fault addr --------
02-13 10:27:18.617 30061 30061 F DEBUG   : Abort message: 'FORTIFY: pthread_mutex_lock called on a destroyed mutex (0x7
565f75238)'
02-13 10:27:18.617 30061 30061 F DEBUG   :     x0  0000000000000000  x1  0000000000005cd5  x2  0000000000000006  x3  00
00007574327290
02-13 10:27:18.617 30061 30061 F DEBUG   :     x4  6b6b60621f6a626e  x5  6b6b60621f6a626e  x6  6b6b60621f6a626e  x7  7f
7f7f7f7f7f7f7f
02-13 10:27:18.617 30061 30061 F DEBUG   :     x8  00000000000000f0  x9  00000078ef691050  x10 0000000000000001  x11 00
000078ef6dab60
02-13 10:27:18.617 30061 30061 F DEBUG   :     x12 00000078f7e40020  x13 000000007fffffff  x14 0000000003e8c534  x15 00
00097a8ebec5f6
02-13 10:27:18.617 30061 30061 F DEBUG   :     x16 00000078ef746cf8  x17 00000078ef723470  x18 000000756cb60000  x19 00
00000000005caf
02-13 10:27:18.617 30061 30061 F DEBUG   :     x20 0000000000005cd5  x21 00000000ffffffff  x22 0000000003141592  x23 00
00007574329000
02-13 10:27:18.617 30061 30061 F DEBUG   :     x24 0000007574327600  x25 0000000000000001  x26 0000000000000960  x27 00
00000000000438
02-13 10:27:18.617 30061 30061 F DEBUG   :     x28 0000007574329000  x29 0000007574327310
02-13 10:27:18.617 30061 30061 F DEBUG   :     lr  00000078ef6cb178  sp  0000007574327270  pc  00000078ef6cb1a4  pst 00
00000000001000
02-13 10:27:18.617 30061 30061 F DEBUG   : 31 total frames
02-13 10:27:18.617 30061 30061 F DEBUG   : backtrace:
02-13 10:27:18.617 30061 30061 F DEBUG   :       #00 pc 000000000005c1a4  /apex/com.android.runtime/lib64/bionic/libc.s
o (abort+164) (BuildId: 19c32900d9d702c303d2b4164fbba76c)
02-13 10:27:18.617 30061 30061 F DEBUG   :       #01 pc 000000000005e10c  /apex/com.android.runtime/lib64/bionic/libc.s
o (__fortify_fatal(char const*, ...)+124) (BuildId: 19c32900d9d702c303d2b4164fbba76c)
02-13 10:27:18.617 30061 30061 F DEBUG   :       #02 pc 00000000000cb08c  /apex/com.android.runtime/lib64/bionic/libc.s
o (HandleUsingDestroyedMutex(pthread_mutex_t*, char const*)+60) (BuildId: 19c32900d9d702c303d2b4164fbba76c)
02-13 10:27:18.617 30061 30061 F DEBUG   :       #03 pc 00000000000caf20  /apex/com.android.runtime/lib64/bionic/libc.s
o (pthread_mutex_lock+208) (BuildId: 19c32900d9d702c303d2b4164fbba76c)
02-13 10:27:18.617 30061 30061 F DEBUG   :       #04 pc 000000000004e0a8  /apex/com.android.vndk.v34/lib64/libc++.so (s
td::__1::mutex::lock()+8) (BuildId: b943576b13bab3840a43769b64a94e88)
02-13 10:27:18.617 30061 30061 F DEBUG   :       #05 pc 0000000000022144  /vendor/lib64/hw/android.hardware.graphics.ma
[email protected] (BufferManager::retain(native_handle const*)+84) (BuildId: e65795567f90475a14dff55aed4c7dd6)
02-13 10:27:18.617 30061 30061 F DEBUG   :       #06 pc 0000000000012b08  /vendor/lib64/hw/android.hardware.graphics.ma
[email protected] (arm::mapper::common::importBuffer(android::hardware::hidl_handle const&, std::__1::function<void (and
roid::hardware::graphics::mapper::V4_0::Error, void*)>)+120) (BuildId: e65795567f90475a14dff55aed4c7dd6)
02-13 10:27:18.617 30061 30061 F DEBUG   :       #07 pc 00000000000113ac  /vendor/lib64/hw/android.hardware.graphics.ma
[email protected] (arm::mapper::GrallocMapper::importBuffer(android::hardware::hidl_handle const&, std::__1::function<vo
id (android::hardware::graphics::mapper::V4_0::Error, void*)>)+124) (BuildId: e65795567f90475a14dff55aed4c7dd6)
02-13 10:27:18.617 30061 30061 F DEBUG   :       #08 pc 0000000000016634  /system/lib64/android.hardware.graphics.mappe
[email protected] (android::hardware::graphics::mapper::V4_0::BsMapper::importBuffer(android::hardware::hidl_handle const&, std:
:__1::function<void (android::hardware::graphics::mapper::V4_0::Error, void*)>)+132) (BuildId: 5b7788f34d654d544830e5fb
59ec2e63)
02-13 10:27:18.617 30061 30061 F DEBUG   :       #09 pc 000000000003e8c8  /system/lib64/libui.so (android::Gralloc4Mapp
er::importBuffer(native_handle const*, native_handle const**) const+120) (BuildId: 7572b8a958e72a3ff726aeef7f8f37b4)
02-13 10:27:18.617 30061 30061 F DEBUG   :       #10 pc 00000000000318ec  /system/lib64/libui.so (android::GraphicBuffe
rMapper::importBuffer(native_handle const*, unsigned int, unsigned int, unsigned int, int, unsigned long, unsigned int,
 native_handle const**)+108) (BuildId: 7572b8a958e72a3ff726aeef7f8f37b4)
02-13 10:27:18.617 30061 30061 F DEBUG   :       #11 pc 0000000000034d3c  /system/lib64/libui.so (android::GraphicBuffe
r::unflatten(void const*&, unsigned long&, int const*&, unsigned long&)+444) (BuildId: 7572b8a958e72a3ff726aeef7f8f37b4
)
02-13 10:27:18.617 30061 30061 F DEBUG   :       #12 pc 0000000000026128  /system/lib64/libmedia_omx.so (android::hardw
are::media::omx::V1_0::utils::convertTo(android::GraphicBuffer*, android::hardware::media::V1_0::AnwBuffer const&)+376)
 (BuildId: c541d41a23e7ff4ec3ce17cd40c544c8)
02-13 10:27:18.617 30061 30061 F DEBUG   :       #13 pc 0000000000025e00  /system/lib64/libmedia_omx.so (android::hardw
are::media::omx::V1_0::utils::convertTo(android::OMXBuffer*, android::hardware::media::omx::V1_0::CodecBuffer const&)+4
00) (BuildId: c541d41a23e7ff4ec3ce17cd40c544c8)
02-13 10:27:18.617 30061 30061 F DEBUG   :       #14 pc 00000000000255e4  /system/lib64/libmedia_omx.so (android::hardw
are::media::omx::V1_0::utils::TWOmxNode::emptyBuffer(unsigned int, android::hardware::media::omx::V1_0::CodecBuffer con
st&, unsigned int, unsigned long, android::hardware::hidl_handle const&)+100) (BuildId: c541d41a23e7ff4ec3ce17cd40c544c
8)
02-13 10:27:18.617 30061 30061 F DEBUG   :       #15 pc 000000000003dd78  /system/lib64/libstagefright_omx.so (android:
:hardware::media::omx::V1_0::implementation::TWGraphicBufferSource::TWOmxNodeWrapper::emptyBuffer(int, unsigned int, an
droid::sp<android::GraphicBuffer> const&, long, int)+312) (BuildId: 8d716a2cf486804a50b1fe56fc45686a)
02-13 10:27:18.617 30061 30061 F DEBUG   :       #16 pc 000000000000c260  /system/lib64/libstagefright_bufferqueue_help
er.so (android::GraphicBufferSource::submitBuffer_l(android::GraphicBufferSource::VideoBuffer const&)+752) (BuildId: 46
a728db4640bd335560d48cbec1b27e)
02-13 10:27:18.617 30061 30061 F DEBUG   :       #17 pc 000000000000b758  /system/lib64/libstagefright_bufferqueue_help
er.so (android::GraphicBufferSource::fillCodecBuffer_l()+968) (BuildId: 46a728db4640bd335560d48cbec1b27e)
02-13 10:27:18.617 30061 30061 F DEBUG   :       #18 pc 000000000000a9d4  /system/lib64/libstagefright_bufferqueue_help
er.so (android::GraphicBufferSource::onFrameAvailable(android::BufferItem const&)+292) (BuildId: 46a728db4640bd335560d4
8cbec1b27e)
02-13 10:27:18.617 30061 30061 F DEBUG   :       #19 pc 0000000000009d58  /system/lib64/libstagefright_bufferqueue_help
er.so (android::GraphicBufferSource::ConsumerProxy::onFrameAvailable(android::BufferItem const&)+104) (BuildId: 46a728d
b4640bd335560d48cbec1b27e)
02-13 10:27:18.617 30061 30061 F DEBUG   :       #20 pc 00000000000ef4ec  /system/lib64/libgui.so (android::BufferQueue
::ProxyConsumerListener::onFrameAvailable(android::BufferItem const&)+92) (BuildId: c76ca08a7f5b2af5ba32a74bee6ae2e9)
02-13 10:27:18.617 30061 30061 F DEBUG   :       #21 pc 00000000000df17c  /system/lib64/libgui.so (android::BufferQueue
Producer::queueBuffer(int, android::IGraphicBufferProducer::QueueBufferInput const&, android::IGraphicBufferProducer::Q
ueueBufferOutput*)+2316) (BuildId: c76ca08a7f5b2af5ba32a74bee6ae2e9)
02-13 10:27:18.617 30061 30061 F DEBUG   :       #22 pc 00000000001169c4  /system/lib64/libgui.so (android::BnGraphicBu
fferProducer::onTransact(unsigned int, android::Parcel const&, android::Parcel*, unsigned int)+612) (BuildId: c76ca08a7
f5b2af5ba32a74bee6ae2e9)
02-13 10:27:18.617 30061 30061 F DEBUG   :       #23 pc 0000000000068088  /system/lib64/libbinder.so (android::BBinder:
:transact(unsigned int, android::Parcel const&, android::Parcel*, unsigned int)+248) (BuildId: 66b89443e02390ede8efed54
005902da)
02-13 10:27:18.617 30061 30061 F DEBUG   :       #24 pc 0000000000066d34  /system/lib64/libbinder.so (android::IPCThrea
dState::executeCommand(int)+500) (BuildId: 66b89443e02390ede8efed54005902da)
02-13 10:27:18.617 30061 30061 F DEBUG   :       #25 pc 0000000000085df4  /system/lib64/libbinder.so (android::IPCThrea
dState::joinThreadPool(bool)+596) (BuildId: 66b89443e02390ede8efed54005902da)
02-13 10:27:18.617 30061 30061 F DEBUG   :       #26 pc 0000000000085b88  /system/lib64/libbinder.so (android::PoolThre
ad::threadLoop()+24) (BuildId: 66b89443e02390ede8efed54005902da)
02-13 10:27:18.617 30061 30061 F DEBUG   :       #27 pc 000000000000fd70  /system/lib64/libutils.so (android::Thread::_
threadLoop(void*)+480) (BuildId: 4ce5249405dff1a8480d738901c940d4)
02-13 10:27:18.617 30061 30061 F DEBUG   :       #28 pc 00000000000e719c  /system/lib64/libandroid_runtime.so (android:
:AndroidRuntime::javaThreadShell(void*)+140) (BuildId: 151eb211a9e62bdd846be20679c2a65c)
02-13 10:27:18.617 30061 30061 F DEBUG   :       #29 pc 00000000000c9ccc  /apex/com.android.runtime/lib64/bionic/libc.s
o (__pthread_start(void*)+204) (BuildId: 19c32900d9d702c303d2b4164fbba76c)
02-13 10:27:18.617 30061 30061 F DEBUG   :       #30 pc 000000000005db00  /apex/com.android.runtime/lib64/bionic/libc.s
o (__start_thread+64) (BuildId: 19c32900d9d702c303d2b4164fbba76c)
02-13 12:46:29.855 32182 19402 E AndroidRuntime: FATAL EXCEPTION: Thread-650
02-13 12:46:29.855 32182 19402 E AndroidRuntime: Process: org.mozilla.firefox, PID: 32182
02-13 12:46:29.855 32182 19402 E AndroidRuntime: android.opengl.GLException: Failed to create EGL context: 0x3000
02-13 12:46:29.855 32182 19402 E AndroidRuntime:        at org.webrtc.EglBase14Impl$EglConnection.<init>(EglBase14Impl.
java:20)
02-13 12:46:29.855 32182 19402 E AndroidRuntime:        at org.webrtc.EglBase14Impl.<init>(EglBase14Impl.java:10)
02-13 12:46:29.855 32182 19402 E AndroidRuntime:        at org.webrtc.EglBase$-CC.create(EglBase.java:6)
02-13 12:46:29.855 32182 19402 E AndroidRuntime:        at org.webrtc.videoengine.VideoCaptureAndroid.<init>(VideoCaptu
reAndroid.java:94)

Enable gammaCV to work in webworkers

gammaCV can work fine in webworkers, however because a webgl canvas is generated using document.createElement, a fairly ugly hack is needed to get it to work.

self.document={
  createElement(tagName) {
    if (tagName === 'canvas') return new OffscreenCanvas(1,1);
    throw new Error('cannot create a '+tagName)
  }
}

(initialising with a size of 1,1 was a guess which seems to work)
The use of webworkers can enable much more complex image processing pipelines to still effectively run in the browser (especially on older machines) without blocking the UI thread and impacting user experience.

This could be solved easily in a few different ways, depending on your preference:

  1. A configuration option for the consumer of the library to provide a function which returns a new canvas.
  2. Auto-detection of the context and behaving differently depending on if in a webworker or the main browser thread.
  3. Using OffscreenCanvas by default and only falling back to using document.createElement if that is not available.

OES_texture_float is not supported on Android Mali GPU's

Mali GPU's, which is around 40% of Android smartphone market share (including all Samsung Galaxy series) and 80% of Android TV's, does not support OES_texture_float.

This is a huge number of devices not supported.

Consider implement WebGL2, which has floating point operations and is fully supported by all smartphone GPU's, or apply a float pack / unpack fallback like this: CesiumGS/cesium#4700

Fix documentation.

In Create Operation page this line of code
const tSrc = new gm.Tensor('uint8', [2, 2, 4], [ 200, 2, 3, 34, 5, 6, 7, 125, 9, 6, 7, 0, 3, 4, 5, 0, ]);

results in the following error messag.

Error: Different dtypes assigned:
expected - uint8
actual - array

This should be changed to
const tSrc = new gm.Tensor('uint8', [2, 2, 4], new Uint8Array([ 200, 2, 3, 34, 5, 6, 7, 125, 9, 6, 7, 0, 3, 4, 5, 0, ]));

Adaptive threshold produces poor image on Android

Testing our application that does image capture on a Google Pixel 7.
The code is basically straight out of the examples:
` let pipeline = gm.grayscale(im_mat);

pipeline = gm.adaptiveThreshold(

  pipeline,

  DEFAULT_ADAPTIVE_BOXSIZE_VALUE,    // 5

  DEFAULT_ADAPTIVE_THRESHOLD_VALUE,  // 15

  0 // grayscale channel only

)

const grayScale = gm.tensorFrom(pipeline)

const sess = new gm.Session()

sess.init(pipeline)

sess.runOp(pipeline, 0, grayScale)

sess.destroy()`

On desktop web (Chrome 119.0.6045.123), both our application and the example have the expected results.
However, on the Pixel, the image looks very fuzzy and full of artifacts. I tried the example for your page and saw the same results.

How to run canny edge detection in gammacv as a webpage?

Hey there, im trying to figure out how to do some basic shape detection using gammacv, but its not working for me.
I downloaded the latest gammacv.js and put it in the same folder as my html file.
Probably an easy fix, or my lack of understanding. Can I run gammacv this way?

<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8" />
    <title>Rectangle Detection</title>
  </head>
  <body>
    <input type="file" accept="image/*" onchange="detectRectangle(this.files[0])" />
    <br />
    <img id="image" />
    <br />
    <canvas id="canvas"></canvas>
    <script src="gammacv.js"></script>
    <script>
      // Define detectRectangle as a global function
      async function detectRectangle(file) {
        // Read the image file as a data URL
        const image = document.getElementById('image');
        const reader = new FileReader();
        reader.onload = () => {
          image.onload = async () => {
            // Create a canvas element to draw the image
            const canvas = document.getElementById('canvas');
            canvas.width = image.width;
            canvas.height = image.height;
            const ctx = canvas.getContext('2d');
            ctx.drawImage(image, 0, 0);
        // Convert the image to grayscale
        const imageData = ctx.getImageData(0, 0, canvas.width, canvas.height);
        const gray = GammaCV.Image.fromImageData(imageData).convert(GammaCV.CV_8U, { alpha: 0.5 }).cvtColor(GammaCV.COLOR_BGR2GRAY);

        // Detect edges in the image using the Canny algorithm
        const edges = await GammaCV.cannyEdges(gray, {
          lowThreshold: 50,
          highThreshold: 150,
          blur: 3,
        });

        // Find contours in the edge image
        const contours = edges.findContours(GammaCV.RETR_EXTERNAL, GammaCV.CHAIN_APPROX_SIMPLE);

        // Find the largest rectangle contour
        let largestArea = 0;
        let largestContour = null;
        for (const contour of contours) {
          const area = contour.area();
          if (area > largestArea) {
            largestArea = area;
            largestContour = contour;
          }
        }

        // Draw the rectangle contour on the canvas
        if (largestContour) {
          const rect = largestContour.minAreaRect();
          ctx.strokeStyle = 'green';
          ctx.lineWidth = 2;
          ctx.beginPath();
          ctx.moveTo(rect.points[0].x, rect.points[0].y);
          ctx.lineTo(rect.points[1].x, rect.points[1].y);
          ctx.lineTo(rect.points[2].x, rect.points[2].y);
          ctx.lineTo(rect.points[3].x, rect.points[3].y);
          ctx.closePath();
          ctx.stroke();
        }
      };
      image.src = reader.result;
    };
    reader.readAsDataURL(file);
  }

  window.onload = function() {
    // Do any other initialization here
  };
</script>

Typescript Support

I'm still using this lib for quite a few things. I'm currently adding types for my own purposes, but I wanted to guage if there was any interest in converting this lib to Typescript. Given the space and subject matter, strong types seem like they would be a must.

Not getting correct lines.

I have used the below code to get the Hough lines. But I am not getting the expected output.
Here is the code:

const params = {
    PROCESSING: {
        name: 'PROCESSING',
        dCoef: {
            name: 'Downsample',
            type: 'constant',
            min: 1,
            max: 4,
            step: 1,
            default: 2,
        },
    },
    PCLINES: {
        name: 'PC LINES',
        count: {
            name: 'Lines Count',
            type: 'uniform',
            min: 1,
            max: 100,
            step: 1,
            default: 10,
        },
        layers: {
            name: 'Layers Count',
            type: 'constant',
            min: 1,
            max: 5,
            step: 1,
            default: 2,
        },
    },
};
const width = 500;
const heigth = 400;
// initialize WebRTC stream and session for runing operations on GPU
const stream = new gm.CaptureVideo(width, heigth);
const sess = new gm.Session();
const canvasProcessed = gm.canvasCreate(width, heigth);

// session uses a context for optimize calculations and prevent recalculations
// context actually a number which help algorythm to run operation efficiently  
let frameNumber = 0;
// allocate memeory for storing a frame and calculations output
const input = new gm.Tensor('uint8', [heigth, width, 4]);
const context = { line: new gm.Line() }
// construct operation grap which is actially a Canny Edge Detector
let pipeline = gm.grayscale(input);
pipeline = gm.downsample(pipeline, 2, 'max');
pipeline = gm.gaussianBlur(pipeline, 3, 1);
pipeline = gm.dilate(pipeline, [3, 3]);
pipeline = gm.sobelOperator(pipeline);
pipeline = gm.cannyEdges(pipeline, 0.25, 0.75);
pipeline = gm.pcLinesTransform(pipeline, 3, 2, 2);

// initialize graph
sess.init(pipeline);

// allocate output
const output = gm.tensorFrom(pipeline);

// create loop
const tick = () => {
    requestAnimationFrame(tick);
    // Read current in to the tensor
    stream.getImageBuffer(input);
    //
    const maxP = Math.max(input.shape[0], input.shape[1]);
    let lines = [];

    // session.runOp(operation, frame, output);
    sess.runOp(pipeline, frameNumber, output);
    gm.canvasFromTensor(canvasProcessed, input);


    for (let i = 0; i < output.size / 4; i += 1) {
        const y = ~~(i / output.shape[1]);
        const x = i - (y * output.shape[1]);
        const value = output.get(y, x, 0);
        const x0 = output.get(y, x, 1);
        const y0 = output.get(y, x, 2);

        if (value > 0.0) {
            lines.push([value, x0, y0]);
        }
    }

    lines = lines.sort((b, a) => a[0] - b[0]);
    // console.log(lines.length)
    lines = lines.slice(0, 50);
    // console.log(lines.length)
    for (let i = 0; i < lines.length; i += 1) {
        context.line.fromParallelCoords(
            lines[i][1] * 1, lines[i][2] * 1,
            input.shape[1], input.shape[0], maxP, maxP / 2,
        );
        // console.log(lines[i][1], lines[i][2]);

        gm.canvasDrawLine(canvasProcessed, context.line, 'rgba(0, 255, 0, 1.0)');
    }

    frameNumber += 1;
}

function main() {
    // start capturing a camera and run loop
    stream.start();
    tick();

    document.body.children[0].appendChild(canvasProcessed);
}
main()

This is the expected output:

Screenshot from 2020-06-09 14-27-15

And this is what I am getting:

Screenshot from 2020-06-09 14-26-59

BTW what is the format of the output from PClinesTransform. My goal is to get the lines obtained from applying hough transform which lie in a certain angle range.
Thanks in advance.

JSDoc description links parsing error

I use link https://en.wikipedia.org/wiki/Dilation_(morphology) in a JSDoc description

/**
 * @description
 *  [Wiki](https://en.wikipedia.org/wiki/Erosion_(morphology))
 */

and after my docs is builded I have
image
and link truncated to
image

seems parsing doesn't work well if line ending by )

Maybe just a dumb doubt

Hi I'm quite new to web development and I have a doubt: As I understand GammaCV works on front-end, using WebGL, WASM, etc. but tutorial says that I must install it using node's package manager npm?! I though node.js was a back-end technology.
Should I use a node-based local server for installing it and then loading the result on my webpage?
I'm quite confused as my closest experience is using OpenCv.js which I could include into my page via their nightly release at their CDN or compiling the files myself using Emscripten and then uploading it to the server.

How can I include your library on my project?

Sorry if this question is dumb but I'm quite confused, however maybe you could help some noobs like me on starting using your library if this kind of obvious steps are shown on the tutorial.

Thank you for your work!

Perspective transform broken on some iOS devices

I've ran into an issue with using perspective transform on some iOS devices using Ionic and Capacitor (i.e. WKWebView) - thus far an iPhone 6 and a iPad Air 2. I've put together a Github project demonstrating the issue.

I've tested on iPhone 7 and iPhone 8 and it's worked fine. I upgraded the iPad Air 2's OS to iOS 12.4; the issue persists.

EDIT: I decided a much smaller image made my life easier as regards getting the tensor data

Desired

alt text

Undesired (iPad Air 2 iOS 12.4)

alt text

Original

alt text

Does this work in Vue/Nuxt?

Hello!

GammaCV is amazing. Your demos work very well on desktop and mobile safari. If I try to use GammaCV in Nuxt/Vue it breaks at import. I'd love to use this in our projects at work. If you have any advice I'd be happy to try and help make this work in this context.

Screen Shot 2019-03-12 at 7 58 51 PM

StrokeWidthTransform (swt) example

Hi,

sorry if I'm posting this in the wrong place but do you maybe have an example of a call for .swt() function?
I'm struggling to set it up and I can't find a reference to its implementation in any given example.

Thank you in advance and sorry for troubling.

Feature Description Algorithms

I'm interested in feature description and in particular ORB. Is implementing FAST, BRIEF, and ORB something that's feasible within GammaCV, or is there any reason why this would be particularly difficult within this library?

I'm interested in diving into this, just wanting to do a sanity check first!

Sry for distrub

, how i full setup of canny image, plz help me, i download the labrary , canny images example aslo copy on server,

Documenting perspectiveProjection

I will happily add this to the documentation, but I need help figuring out how to use it ;).

Basically I want to extract a quadrilateral region from an image and perspective transform it to be a rectangle. Per my other open issue, I've been pointed to the undocumented gm.perspectiveProjection.

Per this operation

export default (tSrc, tTransform, shape = [10, 10, 4], dtype = tSrc.dtype) => new RegisterOperation('PerspectiveProjection')

I'm doing a lot of guessing here, but I'm assuming you use the perspective transform util to pass the transform into the perspectiveProjection op.

const input = await gm.imageTensorFromURL(...); // 1024 x 1024 input image

// I get my quadrilateral by some method... TL, TR, BL, BR.
const pts = [85.03672790527344, 228.44911193847656, 893.9627685546875, 234.818603515625, 49.670997619628906, 758.93505859375, 982.0530395507812, 724.961669921875];

// The Rect TL, TR, BR, BL
const rect = new gm.Rect(
	pts[0][0],
	pts[0][1],
	pts[1][0],
	pts[1][1],
	pts[3][0],
	pts[3][1],
	pts[2][0],
	pts[2][1]
);

const transform = gm.generateTransformMatrix(
	rect, // pass the rect
	[1024, 1024], // this is bounds? I am assuming an x and y of maxWidth and maxHeight
  new gm.Tensor('uint8', input.shape) // this I am unsure of. It's called transformMatrix, but it appears to just need an empty Tensor to save the transform date into. Not sure about type or shape
);

const operation = gm.perspectiveProjection(
	input, // pretty sure about this.
	transform, // also pretty sure about this/
	transform.shape // not so sure about this
);

const output = gm.tensorFrom(operation);
sess.init(operation);
sess.runOp(operation, 0, output);
gm.canvasFromTensor(document.getElementById('my-canvas'), output);

The good news is I'm getting some output. The bad news is it is just a giant single color canvas. Any help would be greatly appreciated.

Explanation needed about Line

I am working on detecting vertical and horizontal lines in an image. The gm.pcLines works well on finding those lines. But I got confused when I was trying to get the middle point of the line.

As far as already known, the x1,y1,x2,y2 coordinates can be obtained by line.x1, line.y1 etc.

This did work on horizontal lines and I have got something like [0, 522, 1910, 522, 955, 522, 0, 0]. But for a vertical line, I have got a line like [0, -972190, 1910, 849950, 1, 1020, 0, 89.93994140625].

The y coordinates were big numbers, while the image itself is of 1910 * 1030. But this line indeed can be drawn by gm.canvasDrawLine().

So could you please give some explanation on the gm.Line object? What I want is the middle point coordinate of a vertical line.

Question: Speeding up imageTensorFromURL?

I have a pipeline that's basically

  1. Capture Image From video object
  2. gm.imageTensorFromURL
  3. Processing
    From profiling, it looks like imageTensorFromURL takes up over half the time of the processing pipeline, a typical run is (milliseconds)
    image read 505 pipeline 676 hough Features 845 contours 850
    For images around 3072x2140 on a Google Pixel 7. Is there any way to speed this up, aside from reducing the resolution? We need fairly high resolution images for upload.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.