react-native-pytorch-core camera is showing blank screen on Windows environment React Native App about playtorch HOT 9 CLOSED

First of all, thanks for the reply, I'll try to make it work as soon as I can, but right now I'm working on an emergency in another project, when I manage to work this out, I'll reply.

from playtorch.

@ademirmochi, what style is applied to the camera? It may be the camera component mounts but renders with 0 width and height. Please try giving the camera a fixed width/height or apply the following style.

import {StyleSheet} from 'react-native';
// ...
<Camera style={StyleSheet.absoluteFill} />

EDIT: The camera should render based on the example in your first comment, so there might be something else happening. Are there any errors in the terminal/log? If not, can you try Flipper and see if there is an error logged but swallowed by the console?

So I did it, it didn't help, but I did manage to go over this particular problem, now, I've created a custom ptl model from scratch, but the thing is, it was based in Yolo, not MobileNet, like in the tutorial, how can I read it properly, I mean, I did like in the tutorial, but, since it expects a MobileNet model, of course that generated an error, so basically my question is, and keep in mind I very recent with AI development with image recognition, so I know very little:

Does PyTorchLive for react native, or react-native-pytorch-core, offer support for ptl files generated from Yolo data, and if so, how can I do it?

To further explain my problem, pretty much, I have to config the image specs in the input and the output, and that is the problem, the image interface from react-native-pytorch-core, does not help me with that...

In this case, I'm expecting a image with this specs from the camera: [1,640,640,3], but when I tried to mess with the image resolution, by using the targetResolution, according with the tutorial for the Camera component, but it doesn't put in the resolution I want, it even raised it to 1024/768, why is this happening?

same for the output, that I should expect a float32[1,25200,9] matrix, so how can I do that with react-native-pytorch-core?

BTW this is the error after trying with the tutorial from Metro:

Possible Unhandled Promise Rejection (id: 1):
Error: End of input at character 0 of
promiseMethodWrapper@http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=com.test_pytorch&modulesOnly=false&runModule=true:2552:45
_callee3$@http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=com.test_pytorch&modulesOnly=false&runModule=true:110988:76
tryCatch@http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=com.test_pytorch&modulesOnly=false&runModule=true:24227:23
invoke@http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=com.test_pytorch&modulesOnly=false&runModule=true:24397:32
tryCatch@http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=com.test_pytorch&modulesOnly=false&runModule=true:24227:23
invoke@http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=com.test_pytorch&modulesOnly=false&runModule=true:24299:30
http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=com.test_pytorch&modulesOnly=false&runModule=true:24329:19
tryCallTwo@http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=com.test_pytorch&modulesOnly=false&runModule=true:28913:9
doResolve@http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=com.test_pytorch&modulesOnly=false&runModule=true:29077:25
Promise@http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=com.test_pytorch&modulesOnly=false&runModule=true:28936:14
callInvokeWithMethodAndArg@http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=com.test_pytorch&modulesOnly=false&runModule=true:24328:33
enqueue@http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=com.test_pytorch&modulesOnly=false&runModule=true:24333:157
http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=com.test_pytorch&modulesOnly=false&runModule=true:24348:69
_callee3@http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=com.test_pytorch&modulesOnly=false&runModule=true:110982:42
execute@http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=com.test_pytorch&modulesOnly=false&runModule=true:110999:8
handleImage$@http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=com.test_pytorch&modulesOnly=false&runModule=true:109911:128
tryCatch@http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=com.test_pytorch&modulesOnly=false&runModule=true:24227:23
invoke@http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=com.test_pytorch&modulesOnly=false&runModule=true:24397:32
tryCatch@http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=com.test_pytorch&modulesOnly=false&runModule=true:24227:23
invoke@http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=com.test_pytorch&modulesOnly=false&runModule=true:24299:30
http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=com.test_pytorch&modulesOnly=false&runModule=true:24329:19
tryCallTwo@http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=com.test_pytorch&modulesOnly=false&runModule=true:28913:9
doResolve@http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=com.test_pytorch&modulesOnly=false&runModule=true:29077:25
Promise@http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=com.test_pytorch&modulesOnly=false&runModule=true:28936:14
callInvokeWithMethodAndArg@http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=com.test_pytorch&modulesOnly=false&runModule=true:24328:33
enqueue@http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=com.test_pytorch&modulesOnly=false&runModule=true:24333:157
http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=com.test_pytorch&modulesOnly=false&runModule=true:24348:69
handleImage@http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=com.test_pytorch&modulesOnly=false&runModule=true:109906:40
http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=com.test_pytorch&modulesOnly=false&runModule=true:110155:39
invokeGuardedCallbackProd@http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=com.test_pytorch&modulesOnly=false&runModule=true:3879:21
invokeGuardedCallback@http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=com.test_pytorch&modulesOnly=false&runModule=true:3983:42
invokeGuardedCallbackAndCatchFirstError@http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=com.test_pytorch&modulesOnly=false&runModule=true:3987:36
executeDispatch@http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=com.test_pytorch&modulesOnly=false&runModule=true:4065:48
executeDispatchesInOrder@http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=com.test_pytorch&modulesOnly=false&runModule=true:4085:26
executeDispatchesAndRelease@http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=com.test_pytorch&modulesOnly=false&runModule=true:5345:35
forEachAccumulated@http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=com.test_pytorch&modulesOnly=false&runModule=true:4618:18
runEventsInBatch@http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=com.test_pytorch&modulesOnly=false&runModule=true:5369:27
runExtractedPluginEventsInBatch@http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=com.test_pytorch&modulesOnly=false&runModule=true:5449:25
http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=com.test_pytorch&modulesOnly=false&runModule=true:5424:42
batchedUpdates$1@http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=com.test_pytorch&modulesOnly=false&runModule=true:17486:20
batchedUpdates@http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=com.test_pytorch&modulesOnly=false&runModule=true:5331:36
_receiveRootNodeIDEvent@http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=com.test_pytorch&modulesOnly=false&runModule=true:5423:23
receiveEvent@http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=com.test_pytorch&modulesOnly=false&runModule=true:5453:32
__callFunction@http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=com.test_pytorch&modulesOnly=false&runModule=true:3056:36
http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=com.test_pytorch&modulesOnly=false&runModule=true:2780:31
__guard@http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=com.test_pytorch&modulesOnly=false&runModule=true:3007:15
callFunctionReturnFlushedQueue@http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=com.test_pytorch&modulesOnly=false&runModule=true:2779:21
callFunctionReturnFlushedQueue@[native code]

from playtorch.

So, first a update, i've started using a mac to eliminate any problems with the SO, next, about the last question, i simply messed with the targetResolution parameter in the Camera component, to 640/640, but on metro server console.log it appeared as 1024/768, just that, nothing more.

But from my understanding, Yolo is not supported, but i've tried to change the input of the image, to change the resolution, height, width, and the output object from the ptl model, but I have no access to the image frame taken from the camera, nor the parameters of said model undergoing your component, at least, that's what it seems, only the parameters the Camera component gives me to change it accordingly, such as targetResolution.

btw, i should mention that i'm testing with my own android cellphone, don't know if that is a issue.

In the end, what I need to know is, the react-native-pytorch-core dependency and its modules and components, does any of them offer me a solution to customize the input frame image taken from the camera, and the output model object given by the ptl model?

from playtorch.

Hi @ademirmochi thanks for using PTL and provide the feedback. We are happy to follow up on this -- if I am understanding correctly, you are developing (and building the app) with a Windows machine. May I ask where is the app ran? Is the camera NOT running on a physical device, simulator, or both? And are you building an Android App or iOS app? And are you trying to integrate the react-native-pytorch-core package to an existing app, which might require adjustment of the permission?

This would help us reproduce the issue and see how we can help! Thanks!

from playtorch.

Hi @ademirmochi thanks for using PTL and provide the feedback. We are happy to follow up on this -- if I am understanding correctly, you are developing (and building the app) with a Windows machine. May I ask where is the app ran? Is the camera NOT running on a physical device, simulator, or both? And are you building an Android App or iOS app? And are you trying to integrate the react-native-pytorch-core package to an existing app, which might require adjustment of the permission?

This would help us reproduce the issue and see how we can help! Thanks!

thanks for the reply, BOTH, I've used in a Windows machine, then, after seeing that there was a lot of problems with windows and/or Linux, I've tried with a m1 Mac machine, I'm trying to use pytorch-live-core sdk and integrating in an already existing app, and I've tried with both IOS and Android phones, and with the emulator as well, none worked, but since there is a camera involved, I'm trying more with my android device, so I can see the camera working, but it only shows a blank screen, it appears the text "Image classification" on top, like the tutorial, and even asks for permission, but the area where is supposed to be a camera rolling, there is just a blank screen.

BTW I'm trying to make it work regardless of the phone SO, so I'm trying for both Android and iOS system

from playtorch.

@ademirmochi, what style is applied to the camera? It may be the camera component mounts but renders with 0 width and height. Please try giving the camera a fixed width/height or apply the following style.

import {StyleSheet} from 'react-native';
// ...
<Camera style={StyleSheet.absoluteFill} />

EDIT: The camera should render based on the example in your first comment, so there might be something else happening. Are there any errors in the terminal/log? If not, can you try Flipper and see if there is an error logged but swallowed by the console?

from playtorch.

Hi @ademirmochi
To your questions

Does PyTorchLive for react native, or react-native-pytorch-core, offer support for ptl files generated from Yolo data, and if so, how can I do it?

So far YOLO is not supported by react-native-pytorch-core package. We are currently working on a Data Processing API to make the data preprocessing/postprocessing more flexible for users.

In this case, I'm expecting a image with this specs from the camera: [1,640,640,3], but when I tried to mess with the image resolution, by using the targetResolution, according with the tutorial for the Camera component, but it doesn't put in the resolution I want, it even raised it to 1024/768, why is this happening?

May I ask how you are configuring the camera? Though the main support would still focus on Mac OS, we are trying to gain more experience on PTL with Windows (and your experience are valuable! It would be great if you could provide codes/ steps to reproduce the issue.

Thanks!

from playtorch.

Hi @ademirmochi congrat on your new mac (but pls let us know if you still need support for windows!)

about the last question, i simply messed with the targetResolution parameter in the Camera component, to 640/640, but on metro server console.log it appeared as 1024/768, just that, nothing more.

for the set resolution, that's expected. As mentioned in (https://pytorch.org/live/docs/api/core/interfaces/cameraview.cameraprops/#targetresolution)

Camera target resolution. It is not guaranteed that the camera runs at the set target resolution, and it might pick the closest available resolution.

Android implementation are based on CameraX, and iOS's implementation are here (https://github.com/pytorch/live/blob/main/react-native-pytorch-core/ios/Camera/CameraView.swift#L77-L91). (so the output image resolution might be platform-specific as well)

thanks @raedle for providing the pointer!

In the end, what I need to know is, the react-native-pytorch-core dependency and its modules and components, does any of them offer me a solution to customize the input frame image taken from the camera, and the output model object given by the ptl model?

at this moment, the data pre/post process support is limited to the pack/unpack json (we are woking on a more flexible one)

(as an example: https://github.com/pytorch/live/blob/423caa394f43e0a615c287071494c94697eee853/react-native-pytorch-core/example/models/resnet18.ptl.live.spec.json#L11-L14).

(and FYI the Android Native implementaion: https://github.com/pytorch/live/blob/2282cd52946f9133551e152200cd06202be54653/react-native-pytorch-core/android/src/main/java/org/pytorch/rn/core/ml/processing/BaseIValuePacker.java#L661)

let me know does that answer your questions? and what else I can help!

from playtorch.

@ademirmochi I will close the issue for now. But feel free to re-open it / create another issue once you are back with us!

from playtorch.