🐛 Camera is not aligned properly when using `useSkiaFrameProcessor` about react-native-vision-camera HOT 48 CLOSED

I will never understand why people ignore the fact that I need log output. I will write a bot in the future to auto-close issues that don't have valid log output.

then you will have zero issues in vc 😅

from react-native-vision-camera.

@mrousavy MR created :)

from react-native-vision-camera.

Try playing around with the code from useSkiaFrameProcessor:

Maybe we're doing something wrong.

Also, the ImageProxy (which is what the native Frame type holds) contains a getSensorToBufferTransformMatrix() method - maybe we can use this Matrix to transform the Frame safely to view dimensions? cc @rodgomesc if you wanna look into this.

from react-native-vision-camera.

then it's probably the sensor to buffer transform matrix.

from react-native-vision-camera.

i think that i found a fix for this, should send a pr some time tomorrow

from react-native-vision-camera.

oh lol after posting this I just realized I used the wrong function.... lmao see 38c6983

now that should work

from react-native-vision-camera.

aha interesting... can you send a PR?

from react-native-vision-camera.

Not yet.

from react-native-vision-camera.

+1

from react-native-vision-camera.

I will never understand why people ignore the fact that I need log output. I will write a bot in the future to auto-close issues that don't have valid log output.

from react-native-vision-camera.

Maybe play around with useSkiaFrameProcessor hook, specifically the render() function in there. If you can figure out what's wrong, please submit a PR.
I'm on vacation this week

from react-native-vision-camera.

This happened when you using resizeMode={'contain'}

Try to delete this from Camera props

from react-native-vision-camera.

@Space6188 does it properly display without the resizeMode?

from react-native-vision-camera.

I can also reproduce this when using resizeMode="cover"...

from react-native-vision-camera.

i tried all resizeMode possibilities, also all fit possibilities, the problem is not there

from react-native-vision-camera.

@Space6188правильно ли он отображается без resizeMode?

No,my bad,sorry, it is not working too

from react-native-vision-camera.

In case it helps - I am able to reproduce this by passing in the format prop with a custom videoResolution to the camera. Removing the format prop seems to fix the issue.

Edit: Actually, removing format only fixes the white bar that appears to the left of the preview, and moves it down to the bottom. Which is less noticeable as it looks like that's just where the camera preview ends. But in my case, the preview should take up the entire screen, but doesn't when using the Skia frame processor.

from react-native-vision-camera.

it seems that canvas.drawImage draws the image with a default clip and matrix that are messed by the canvas rotation somehow? that’s confusing since if we use canvas.drawImageRect with the same src/dst props it draws correctly! in my tests it seem to solve the issue

diff --git a/package/src/skia/useSkiaFrameProcessor.ts b/package/src/skia/useSkiaFrameProcessor.ts
index 3feb336..fd69ca1 100644
--- a/package/src/skia/useSkiaFrameProcessor.ts
+++ b/package/src/skia/useSkiaFrameProcessor.ts
@@ -6,8 +6,34 @@ import { VisionCameraProxy, wrapFrameProcessorWithRefCounting } from '../FramePr
 import type { DrawableFrameProcessor } from '../types/CameraProps'
 import type { ISharedValue, IWorkletNativeApi } from 'react-native-worklets-core'
 import { WorkletsProxy } from '../dependencies/WorkletsProxy'
-import type { SkCanvas, SkPaint, SkImage, SkSurface } from '@shopify/react-native-skia'
+import { type SkCanvas, type SkPaint, type SkImage, type SkSurface, ColorType, AlphaType, Skia } from '@shopify/react-native-skia'
 import { SkiaProxy } from '../dependencies/SkiaProxy'
+import { InteractionManager } from 'react-native'
+
+// Mocked Frame Data for testings
+const mockedImageWidth = 1920
+const mockedImageHeight = 1080
+
+const pixels = new Uint8Array(mockedImageWidth * mockedImageHeight * 4)
+pixels.fill(255)
+let i = 0
+for (let x = 0; x < mockedImageWidth; x++) {
+  for (let y = 0; y < mockedImageHeight; y++) {
+    pixels[i++] = (x * y) % 255
+  }
+}
+
+const data = Skia.Data.fromBytes(pixels)
+const image = Skia.Image.MakeImage(
+  {
+    width: mockedImageWidth,
+    height: mockedImageHeight,
+    alphaType: AlphaType.Opaque,
+    colorType: ColorType.RGBA_8888,
+  },
+  data,
+  mockedImageWidth * 4,
+)!
 
 /**
  * Represents a Camera Frame that can be directly drawn to using Skia.
@@ -155,7 +181,7 @@ export function createSkiaFrameProcessor(
 
     // Convert Frame to SkImage/Texture
     const nativeBuffer = (frame as FrameInternal).getNativeBuffer()
-    const image = Skia.Image.MakeImageFromNativeBuffer(nativeBuffer.pointer)
+    // const image = Skia.Image.MakeImageFromNativeBuffer(nativeBuffer.pointer)
 
     return new Proxy(frame as DrawableFrame, {
       get: (_, property: keyof DrawableFrame) => {
@@ -165,11 +191,36 @@ export function createSkiaFrameProcessor(
             'worklet'
             // rotate canvas to properly account for Frame orientation
             canvas.save()
-            const rotation = getRotationDegrees(frame.orientation)
-            canvas.rotate(rotation, frame.width / 2, frame.height / 2)
-            // render the Camera Frame to the Canvas
-            if (paint != null) canvas.drawImage(image, 0, 0, paint)
-            else canvas.drawImage(image, 0, 0)
+
+            const rotationAngle = 0 //getRotationDegrees('portrait')
+            const { width: frameWidth, height: frameHeight } = getPortraitSize(frame)
+
+            // center of the canvas
+            const centerX = frameWidth / 2
+            const centerY = frameHeight / 2
+
+            const currentPaint = paint ?? Skia.Paint()
+
+            // only draw the rect by 40% of the frame size so we can vizualize the whole thing to debug :)
+            const reductionFactor = 0.4
+            const rectWidth = frameWidth * reductionFactor
+            const rectHeight = frameHeight * reductionFactor
+
+            // apply rotation around the center of the canvas
+            canvas.rotate(rotationAngle, centerX, centerY)
+
+            const rectX = centerX - rectWidth / 2
+            const rectY = centerY - rectHeight / 2
+
+            const srcRect = Skia.XYWHRect(0, 0, image.width(), image.height())
+            // Define the destination rectangle on the canvas
+            const dstRect = Skia.XYWHRect(rectX, rectY, rectWidth, rectHeight)
+            // Draw the image on the canvas
+
+            canvas.drawImageRect(image, srcRect, dstRect, currentPaint)
 
             // restore transforms/rotations again
             canvas.restore()
@@ -185,7 +236,7 @@ export function createSkiaFrameProcessor(
           return () => {
             'worklet'
             // dispose the Frame and the SkImage/Texture
-            image.dispose()
+            // image.dispose() // we do not dispose our mocked image :)
             nativeBuffer.delete()
           }
         }

const srcRect = Skia.XYWHRect(0, 0, frameWidth, frameHeight);
const dstRect = Skia.XYWHRect(0, 0, frameWidth, frameHeight);
canvas.drawImageRect(image, srcRect, dstRect, currentPaint);

from react-native-vision-camera.

Oh wait, this solves the misalignment of the frame inside the canvas; however, it causes another issue: the image buffer is now rotated 90 degrees clockwise inside a (0 deg portrait) preview for me. I think this is expected since we are lacking orientation support for Android, @mrousavy?

from react-native-vision-camera.

I am applying the orientation inside the render(..) function, so actually it is expected that the Preview looks correct.

from react-native-vision-camera.

I am applying the orientation inside the render(..) function, so actually it is expected that the Preview looks correct.

i mean, frame.orientation will always returns landscape-right since orientation support not being complete on android, and in render you are just doing getRotationDegrees(frame.orientation)
so i can't visualize the full picture here, of how did you got the image buffers always in portrait without a counter-clockwise rotation of -90deg in the current code that's running in render() {...}

from react-native-vision-camera.

i mean, frame.orientation will always returns landscape-right since orientation support not being complete on android,

no, frame.orientation is an exception here - this is the sensor relative orientation of the ImageProxy, so this is correct as it is.

from react-native-vision-camera.

Any news on this? I'm also experiencing this misplacement. I observed that it depends on format and video resolution:

1. Without passing format props to Camera:

2. Default format got from useCasmeraFormat():

  const format = useCameraFormat(device, []);
  console.log(format);

Log:

{"autoFocusSystem": "contrast-detection", "fieldOfView": 80.98175244802063, "maxFps": 30, "maxISO": 3200, "minFps": 7, "minISO": 100, "photoHeight": 2160, "photoWidth": 4096, "supportsDepthCapture": false, "supportsPhotoHdr": false, "supportsVideoHdr": false, "videoHeight": 2160, "videoStabilizationModes": ["off", "cinematic"], "videoWidth": 3840}

const format = useCameraFormat(device, [
    { videoResolution: { width: 600, height: 400 } },
  ]);
  console.log(format);

Log:

{"autoFocusSystem": "contrast-detection", "fieldOfView": 80.98175244802063, "maxFps": 30, "maxISO": 3200, "minFps": 7, "minISO": 100, "photoHeight": 2160, "photoWidth": 4096, "supportsDepthCapture": false, "supportsPhotoHdr": false, "supportsVideoHdr": false, "videoHeight": 480, "videoStabilizationModes": ["off", "cinematic"], "videoWidth": 640}

const format = useCameraFormat(device, [
    { videoResolution: { width: 1200, height: 800 } },
  ]);
  console.log(format);

Log:

{"autoFocusSystem": "contrast-detection", "fieldOfView": 80.98175244802063, "maxFps": 30, "maxISO": 3200, "minFps": 7, "minISO": 100, "photoHeight": 2160, "photoWidth": 4096, "supportsDepthCapture": false, "supportsPhotoHdr": false, "supportsVideoHdr": false, "videoHeight": 720, "videoStabilizationModes": ["off", "cinematic"], "videoWidth": 1280}

So in case of lower resolutions (default or set manually) only bottom part is cut off, while in higher resolutions entire frame is misplacement. Maybe that somehow helps you

from react-native-vision-camera.

Є новини з цього приводу? Я також відчуваю це неправильне розташування. Я помітив, що це залежить від формату та роздільної здатності відео:

1. Без передачі параметрів формату в камеру:

2. Формат за замовчуванням отримано з `useCasmeraFormat()`:

  const format = useCameraFormat(device, []);
  console.log(format);

Журнал:

{"autoFocusSystem": "contrast-detection", "fieldOfView": 80.98175244802063, "maxFps": 30, "maxISO": 3200, "minFps": 7, "minISO": 100, "photoHeight": 2160, "photoWidth": 4096, "supportsDepthCapture": false, "supportsPhotoHdr": false, "supportsVideoHdr": false, "videoHeight": 2160, "videoStabilizationModes": ["off", "cinematic"], "videoWidth": 3840}

const format = useCameraFormat(device, [
    { videoResolution: { width: 600, height: 400 } },
  ]);
  console.log(format);

Журнал:

{"autoFocusSystem": "contrast-detection", "fieldOfView": 80.98175244802063, "maxFps": 30, "maxISO": 3200, "minFps": 7, "minISO": 100, "photoHeight": 2160, "photoWidth": 4096, "supportsDepthCapture": false, "supportsPhotoHdr": false, "supportsVideoHdr": false, "videoHeight": 480, "videoStabilizationModes": ["off", "cinematic"], "videoWidth": 640}

const format = useCameraFormat(device, [
    { videoResolution: { width: 1200, height: 800 } },
  ]);
  console.log(format);

Журнал:

{"autoFocusSystem": "contrast-detection", "fieldOfView": 80.98175244802063, "maxFps": 30, "maxISO": 3200, "minFps": 7, "minISO": 100, "photoHeight": 2160, "photoWidth": 4096, "supportsDepthCapture": false, "supportsPhotoHdr": false, "supportsVideoHdr": false, "videoHeight": 720, "videoStabilizationModes": ["off", "cinematic"], "videoWidth": 1280}

Таким чином, у разі нижчої роздільної здатності (за замовчуванням або встановленої вручну) обрізається лише нижня частина, тоді як у вищій роздільній здатності зміщується весь кадр. Можливо, це якось тобі допоможе

For some reason, the format parameters don't change anything for me, the behavior is the same as yours in the second example. And the same problem with useSkiaFrameProcessor. On the emulator, the screen is completely black, and on real devices, part of it is. Another strange thing is that the frame itself processes frames rotated 90 degrees, I noticed this when my model started drawing rectangle around object on a black screen turned to the horizontal position. In IOS all ok.

from react-native-vision-camera.

Another strange thing is that the frame itself processes frames rotated 90 degrees

That's because of how Cameras work. Sensors are in 90deg orientation. See frame.orientation.

from react-native-vision-camera.

I also encountered this when using useSkiaFrameProcessor:

but useFrameProcessor works normally.

from react-native-vision-camera.

I just changed this line

-              canvas.rotate(rotation, frame.width / 2, frame.height / 2)
+              canvas.rotate(rotation, frame.width / 2, frame.height / 1.5)

on useSkiaFrameProcessor.ts
Not it's working as expected

from react-native-vision-camera.

Created a fix for this here; #2931

Can you guys test if this works for you? :)

from react-native-vision-camera.

Created a fix for this here; #2931

Can you guys test if this works for you? :)

@mrousavy
Hey, for me, it's even worse
I'm using Moto G84

from react-native-vision-camera.

@mrousavy
I was using this format before

const format = useCameraFormat(device, [
    {
      videoResolution: {
        width: 480,
        height: Platform.OS === 'ios' ? 640 : 720,
      },
    },
  ]);

Using this format, even the colors are changing

from react-native-vision-camera.

@EnzoDomingues can you leave the format, aspect ration configurations at the default or use them exactly like in this example app and send a new screenshot?

from react-native-vision-camera.

@EnzoDomingues can you leave the format, aspect ration configurations at the default or use them exactly like in this example app and send a new screenshot?

The first screenshot is default, without any format

from react-native-vision-camera.

Hm. Matrixes are really hard to get right.

from react-native-vision-camera.

@EnzoDomingues can you leave the format, aspect ration configurations at the default or use them exactly like in this example app and send a new screenshot?

The first screenshot is default, without any format

yup, i can confirm, it's worse for me as well

from react-native-vision-camera.

@EnzoDomingues can you confirm if applying this patch #2762 (comment) the frame keeps centered for you?

from react-native-vision-camera.

Try again now - I think it now works :)

from react-native-vision-camera.

@mrousavy I added your changes locally and the video is not showing anymore

from react-native-vision-camera.

@EnzoDomingues can you confirm if applying this patch #2762 (comment) the frame keeps centered for you?

@rodgomesc, with your patch

from react-native-vision-camera.

@EnzoDomingues can you log frame.orientation?

from react-native-vision-camera.

@mrousavy sure

 LOG  landscape-right
 LOG  landscape-right
 LOG  landscape-right

from react-native-vision-camera.

Ah that's why! Can you try to adjust this code here:

I don't have a phone where it's sensor orientation is landscape-right, so I cannot test this myself. Maybe just switch out the translations, use 0 or whatever - if you find a fix pls let me know.

from react-native-vision-camera.

@mrousavy

   // rotate two flips on (0,0) origin and move X + Y into view again
        canvas.translate(0, frame.width)
        canvas.rotate(270, 0, 0)

This works for me instead canvas.translate(frame.height, frame.width)
Could you update it?

from react-native-vision-camera.

Is there a release containing this fix?

from react-native-vision-camera.

Not yet.

Waiting on this for prod release

from react-native-vision-camera.

I wonder @mrousavy why your sample app does not seem to have this issue? My app is in Portrait mode and OS is locked in rotation too but the SKIA preview is a zoomed rotated one.

from react-native-vision-camera.

@pke are you using the current main branch? As I just said above; there is no release for this yet. It's on main, but not on npm.

from react-native-vision-camera.

Tried to use "react-native-vision-camera": "mrousavy/react-native-vision-camera#bd955b5" in package file, which removes the POD when running bundle exec pod install :/

from react-native-vision-camera.

What's the plan to release this one? @mrousavy

from react-native-vision-camera.