Comments (7)
Anyone? having any idea about this issue and any possible fix?
from react-native-vision-camera.
Hey!
I just spent a few days on thinking about a battleproof timestamp synchronization solution, and I came up with a great idea.
I built a TrackTimeline
helper class which represents a video or audio track - it can be started & stopped, paused & resumed, and even supports nesting pauses without issues.
- The total duration of the video is summed up from the difference between the first and the last actually written timestamps, minus the total duration of all pauses between a video. No more incorrect
video.duration
! 🥳 - Whereas before I just had a 4 second timeout if no frames arrive, I now just wait twice the frame latency (a few milliseconds) to ensure no frames are left out at maximum! 🎉
- A video can be stopped while it is paused without any issues, as a pause call is taken into consideration before stopping 💪
- A video file's session now exactly starts at the start() timestamp, and ends at the exact timestamp of the last video frame - this ensures there can never be any blank frames in the video, even if the audio track is longer 🤩
This was really complex to built as I had to synchronize timestamps between capture sessions, and the entire thing is a producer model - a video buffer can come like a second or so later than the audio buffer, but I need to make sure the video track starts before the audio track starts, and ends after the audio track ends - that's a huge brainf*ck! 🤯😅
There's also no helper APIs for this on iOS, and it looks like no other Camera framework (not even native Swift/ObjC iOS Camera libraries) support this - they all break when timestamps have a delay (e.g. video stabilization enabled) (or dont even support delays at all) ; so I had to build the thing myself.
Check out this PR and try if it fixes the issue for you; #2948
Thanks! ❤️
from react-native-vision-camera.
I just re-read what you said and it sounds actually intentional - there are situations where a few more frames are being encoded into the video.
This is to ensure the video is longer than the audio, but the video metadata has a flag that specifies the actual duration of the track session - this might cut off a few frames in the start or end.
See AVAssetWriter.startSession(atSourceTime:)
/ AVAssetWriter.endSession(atSourceTime:)
from react-native-vision-camera.
I think this is either fixed now in 4.2.0, or intentional (depending on the comment above). https://github.com/mrousavy/react-native-vision-camera/releases/tag/v4.2.0
from react-native-vision-camera.
Thank you @mrousavy for the detailed comments and the new logic for accurate duration calculation. However, I'm still encountering the issue I mentioned earlier. Let me provide a more detailed explanation:
- Video recording starts.
- Each call to
let successful = assetWriterInput.append(buffer)
returns success, and I am counting the frames written to the file at this point. - Video recording stops.
For example, the count is 184 because assetWriterInput.append(buffer)
was called successfully 184 times. The video file's metadata also reflects 184 frames. However, when I decode the recorded file using the Python script I mentioned in my first comment, it shows 183 frames or sometimes 182 frames. The decoded frame count is always less than the number of frames actually written to the file.
Could you suggest a way to fix this discrepancy? I have even tried excluding frames that are before the video starting timestamp by returning false in the start case of the events (if timestamp < event.timestamp
).
I need the metadata file frame count to match the decoded frame count because I am recording the timestamp of each frame for later video analysis. The timestamps are recorded in a separate JSON file. So, when 184 buffers are appended, the timestamp count is also 184. But with only 183 or 182 decoded frames, there is a mismatch, and it's unclear which frame was dropped or skipped (whether at the start, middle, or end).
Any assistance to resolve this issue would be greatly appreciated. Thanks!
from react-native-vision-camera.
I think this shouldn't be changed in VisionCamera, but rather in your Python script.
VisionCamera just does add a few frames before or after the video to make sure there are no blanks (because if an audio sample comes after a video sample, it will be a blank frame in the resulting video).
So I guess you just need to make sure to decode only the Frames that are actually within the time range of the track duration.
from react-native-vision-camera.
The decoded frames are always fewer than the frames added by the Vision Camera, never more. I have tried using the ffmpeg -i video.mp4 thumb%04d.jpg -hide_banner
command, and the result is the same as with the Python script.
Additionally, the video recorded by Expo Camera does not exhibit any frame discrepancies. I have also tested some random recorded videos from other sources, and none of them show any frame differences.
Do you think this issue can be fixed on the Vision Camera side? Any help would be greatly appreciated.
from react-native-vision-camera.
Related Issues (20)
- 💭 Code scans only in the top right corner HOT 2
- 🔧 I had issue building !! HOT 1
- 🔧 cannot build!! HOT 1
- 🐛 Whats the meaning of life HOT 1
- 💭 Hans Pie recipe HOT 2
- 💭 Possible to zoom with useSkiaFrameProcessor? HOT 6
- 💭 How to set Low video resolution HOT 3
- 🐛 In production, runAsync does not work inside a useSkiaFrameProcessor HOT 1
- 🐛 Cannot zoom on Android when using useSkiaFrameProcessor HOT 5
- ✨ Request for Adding Additional Cinematic Modes HOT 1
- 🐛 The picture looks blurry ( out of focus) in Camera View but comes out focused after clicking on Ipad HOT 11
- 🐛 ERROR Frame Processor Error: Failed to lock HardwareBuffer for reading!, js engine: VisionCamera HOT 5
- 🐛 [capture/invalid-path] Unable to take photos or videos with custom path with Android using expo file system. HOT 5
- 💭 How to wrap an overlay around the preview and render a square output HOT 2
- 💭 Issue with FrameProcessorsUnavailableError HOT 3
- 💭 Lanscape mode issue. HOT 3
- 🐛 iOS front Camera landscape mode issue. HOT 1
- 🔧 Build fails after upgrading to targetSdkVeersion 34 HOT 7
- 🐛 Wrong Orientation on `takePhoto` (`takeSnapshot` is working properly) HOT 6
- 💭 Camera format attribute supportsPhotoHdr is always 'false' on iPhone 12 (that does support HDR) HOT 8
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from react-native-vision-camera.