Git Product home page Git Product logo

Comments (8)

Breakthrough avatar Breakthrough commented on September 9, 2024 1

@mmokrejs sorry for the delay, thank you for the samples I did receive them. I'll leave this open for now and re-word it as an investigation into improved logging on frame corruption. I haven't had any time to experiment with the videos you sent yet, but will let you know if I can come up with any improvements. In the meantime happy to accept any PRs if folks have similar fixes.

Many thanks for the test material.

from dvr-scan.

Breakthrough avatar Breakthrough commented on September 9, 2024

If you are able, can you test running the video with another related project as suggested in the linked issue? It entails running PySceneDetect on the failing video with the following commands:

scenedetect -i video.mp4 --verbosity debug --backend opencv detect-content
scenedetect -i video.mp4 --verbosity debug --backend pyav detect-content

I would like to know if using a different library to handle video IO would resolve the situation, or at least be able to return an error that can be acted on. Will also try this once I have a sample, sent you an email just now. I might be missing some debug logging from PySceneDetect so will look into making sure that gets logged in DVR-Scan.

With OpenCV we try to read at least 5 more frames after a failure to decode (see https://github.com/Breakthrough/PySceneDetect/blob/main/scenedetect/backends/opencv.py#L282). Unfortunately there is no error that gets returned from OpenCV in the case of a decode failure.

Would be nice if the List of motion events: table contained also the frame starts and ends, so that one could see if the input was processed up to the very last frame or not.

As long as you are not seeking in the input, you should be able to tell how many frames were processed from the log output:

[DVR-Scan] Processed 387014 frames read in 4459.8 secs (avg 86.8 FPS).

Thank you for the detailed report!

from dvr-scan.

mmokrejs avatar mmokrejs commented on September 9, 2024

Here I ran the command on just the problematic 51MB file (unlike merged HEVC file for whole day):

$ scenedetect -i input.3gp --verbosity debug --backend opencv detect-content
INFO: context.handle_options(): PySceneDetect 0.6.3
DEBUG: context.handle_options(): User config file not found.
DEBUG: context.handle_options(): Parsing program options.
DEBUG: __init__.open_video(): Opening video with opencv...
DEBUG: context._open_video_stream(): Video opened using backend VideoStreamCv2
DEBUG: context.handle_options(): Initializing SceneManager.
DEBUG: __init__.detect_content_command(): Adding detector: ContentDetector({'weights': Components(delta_hue=1.0, delta_sat=1.0, delta_lum=1.0, delta_edges=0.0), 'kernel_size': None, 'luma_only': False, 'min_scene_len': 10, 'threshold': 27.0})
DEBUG: controller.run_scenedetect(): Running controller.
INFO: scene_manager.detect_scenes(): Downscale factor set to 10, effective resolution: 256 x 192
INFO: scene_manager.detect_scenes(): Detecting scenes...                                                                                                                                                               
  Detected: 0 | Progress:  37%|█████████████████████████████████████████████████████▍                                                                                           | 295/801 [00:02<00:04, 111.83frames/s][hevc @ 0x5641b3d58d80] Duplicate POC in a sequence: 13.
DEBUG: opencv.read(): Frame failed to decode.                                                                                                                                                                          
  Detected: 0 | Progress: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊| 800/801 [00:07<00:00, 103.77frames/s]
INFO: controller._detect(): Processed 800 frames in 7.7 seconds (average 103.76 FPS).
INFO: controller.run_scenedetect(): Detected 1 scenes, average shot length 46.9 seconds.
$ scenedetect -i input.3gp --verbosity debug --backend pyav detect-content
INFO: context.handle_options(): PySceneDetect 0.6.3
DEBUG: context.handle_options(): User config file not found.
DEBUG: context.handle_options(): Parsing program options.
Usage: scenedetect [OPTIONS] COMMAND1 [ARGS]... [COMMAND2 [ARGS]...]...
Try 'scenedetect -h' for help.

Error: Invalid value for -b/--backend: Specified backend pyav is not available on this system!
$

Hmm, I have dvr-scan and scenedetect from pip. I have opencv on Gentoo system-wide.

OK, I installed https://pypi.org/project/av/ into same environemnt like dvr-scan and scenedetect.

$ pip install av
Collecting av
  Downloading av-12.0.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.6 kB)
Downloading av-12.0.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.7 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 34.7/34.7 MB 5.9 MB/s eta 0:00:00
Installing collected packages: av
Successfully installed av-12.0.0
$

Now I have:

$ scenedetect -i input.3gp --verbosity debug --backend pyav detect-content
INFO: context.handle_options(): PySceneDetect 0.6.3
DEBUG: context.handle_options(): User config file not found.
DEBUG: context.handle_options(): Parsing program options.
DEBUG: __init__.open_video(): Opening video with pyav...
DEBUG: pyav.__init__(): Restoring default ffmpeg log callbacks.
DEBUG: pyav.__init__(): Threading mode set: AUTO
DEBUG: context._open_video_stream(): Video opened using backend VideoStreamAv
DEBUG: context.handle_options(): Initializing SceneManager.
DEBUG: __init__.detect_content_command(): Adding detector: ContentDetector({'weights': Components(delta_hue=1.0, delta_sat=1.0, delta_lum=1.0, delta_edges=0.0), 'kernel_size': None, 'luma_only': False, 'min_scene_len': 15, 'threshold': 27.0})
DEBUG: controller.run_scenedetect(): Running controller.
INFO: scene_manager.detect_scenes(): Downscale factor set to 10, effective resolution: 256 x 192
INFO: scene_manager.detect_scenes(): Detecting scenes...                                                                                                                                                                                    
  Detected: 0 | Progress:   0%|                                                                                                                                                                                 | 0/801 [00:00<?, ?frames/s][swscaler @ 0x7f631cd76f80] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cdae880] deprecated pixel format used, make sure you did set range correctly
  Detected: 0 | Progress:   0%|▏                                                                                                                                                                        | 1/801 [00:00<01:45,  7.56frames/s][swscaler @ 0x7f631cd87800] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cd947c0] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cdd2840] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cd87800] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cd947c0] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cdae880] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cd87800] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cd947c0] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cdae880] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cdbb840] deprecated pixel format used, make sure you did set range correctly
  Detected: 0 | Progress:   1%|██▌                                                                                                                                                                     | 12/801 [00:00<00:13, 59.30frames/s][swscaler @ 0x7f631cdae880] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cdbb840] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cdae880] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cdbb840] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cdae880] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cdbb840] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cdae880] deprecated pixel format used, make sure you did set range correctly
  Detected: 0 | Progress:   2%|███▉                                                                                                                                                                    | 19/801 [00:00<00:13, 59.98frames/s][swscaler @ 0x7f631cdbb840] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cdae880] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cdbb840] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cdae880] deprecated pixel format used, make sure you did set range correctly

...

[swscaler @ 0x7f631ce4b380] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631ce93940] deprecated pixel format used, make sure you did set range correctly
  Detected: 0 | Progress:  37%|████████████████████████████████████████████████████████████▋                                                                                                         | 293/801 [00:02<00:04, 116.16frames/s][swscaler @ 0x7f631cdbd080] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cef3040] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631ce76ac0] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cea1980] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631ce84b40] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631ced4600] deprecated pixel format used, make sure you did set range correctly
[hevc @ 0x7f636ced2480] Duplicate POC in a sequence: 13.
[hevc @ 0x7f636ced2480] Error parsing NAL unit #0.
[swscaler @ 0x7f631ce84b40] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cee3b40] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631ce0c700] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631ce4df80] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631ce68a80] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631ce4df80] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631ce1a1c0] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631ce4df80] deprecated pixel format used, make sure you did set range correctly
  Detected: 0 | Progress:  38%|███████████████████████████████████████████████████████████████▍                                                                                                      | 306/801 [00:02<00:04, 113.30frames/s][swscaler @ 0x7f631cebff00] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631ce4df80] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631ce3f0c0] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631ce935c0] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631ce3f0c0] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cdede40] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631ce3f0c0] deprecated pixel format used, make sure you did set range correctly
CRITICAL: scene_manager._decode_thread(): Fatal error: Exception raised in decode thread.                                                                                                                                                   
  Detected: 0 | Progress:  39%|█████████████████████████████████████████████████████████████████                                                                                                     | 314/801 [00:02<00:04, 105.20frames/s]
Traceback (most recent call last):
  File "/home/mmokrejs/virtualenv/dvr-scan/bin/scenedetect", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/mmokrejs/virtualenv/dvr-scan/lib/python3.11/site-packages/scenedetect/__main__.py", line 47, in main
    run_scenedetect(cli_ctx)
  File "/home/mmokrejs/virtualenv/dvr-scan/lib/python3.11/site-packages/scenedetect/_cli/controller.py", line 58, in run_scenedetect
    scene_list, cut_list = _detect(context)
                           ^^^^^^^^^^^^^^^^
  File "/home/mmokrejs/virtualenv/dvr-scan/lib/python3.11/site-packages/scenedetect/_cli/controller.py", line 101, in _detect
    num_frames = context.scene_manager.detect_scenes(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mmokrejs/virtualenv/dvr-scan/lib/python3.11/site-packages/scenedetect/scene_manager.py", line 895, in detect_scenes
    raise self._exception_info[1].with_traceback(self._exception_info[2])
  File "/home/mmokrejs/virtualenv/dvr-scan/lib/python3.11/site-packages/scenedetect/scene_manager.py", line 918, in _decode_thread
    frame_im = video.read()
               ^^^^^^^^^^^^
  File "/home/mmokrejs/virtualenv/dvr-scan/lib/python3.11/site-packages/scenedetect/backends/pyav.py", line 279, in read
    self._frame = next(self._container.decode(video=0))
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "av/container/input.pyx", line 208, in decode
  File "av/packet.pyx", line 80, in av.packet.Packet.decode
  File "av/video/stream.pyx", line 35, in av.video.stream.VideoStream.decode
  File "av/video/stream.pyx", line 44, in av.video.stream.VideoStream.decode
  File "av/codec/context.pyx", line 507, in av.codec.context.CodecContext.decode
  File "av/codec/context.pyx", line 404, in av.codec.context.CodecContext._send_packet_and_recv
  File "av/error.pyx", line 328, in av.error.err_check
av.error.InvalidDataError: [Errno 1094995529] Invalid data found when processing input
$

from dvr-scan.

mmokrejs avatar mmokrejs commented on September 9, 2024

Would be nice if the List of motion events: table contained also the frame starts and ends, so that one could see if the input was processed up to the very last frame or not.

As long as you are not seeking in the input, you should be able to tell how many frames were processed from the log output:

[DVR-Scan] Processed 387014 frames read in 4459.8 secs (avg 86.8 FPS).

So how many total frames were in the input file, I mean how many ffmpeg would find otherwise regardless of eventual failure)?

Or show column with percentages of the total input?

from dvr-scan.

Breakthrough avatar Breakthrough commented on September 9, 2024

So how many total frames were in the input file, I mean how many ffmpeg would find otherwise regardless of eventual failure)?

Or show column with percentages of the total input?

There's no easy way to show the expected total, some containers don't report it accurately and it is often incorrect. Using other libraries like PyAV allows ensuring that you have exhausted the full stream, but using OpenCV like this library currently does, it's not possible to do things like that. I think PyAV allows capturing those warnings, although I recall running into some deadlocks when doing that... I was recently made aware of some progress on the issue at PyAV-Org/PyAV#751 though so it probably is worth looking into to see if it's possible.

In the meantime this is similar enough to #118 and has been a huge help in the debugging process. I'll mark this as a duplicate of that, but feel free to continue the discussion there.

from dvr-scan.

mmokrejs avatar mmokrejs commented on September 9, 2024

Thank you for your efforts so far. I tend to believe the [ffmpeg/video] hevc: Error parsing NAL unit #0 is not always criticial and maybe the testcase cannt be processed due to the Error while decoding frame (hardware decoding)! instead. I see multiple [ffmpeg/video] hevc: Error parsing NAL unit #0 messages in a row while playng the movies.

I have several more testcases of mpv unplayable HEVC files (and causing dvr-scan to skip the remainder), but with different errors:

attempt 1
[ffmpeg] Failed to reallocate parser buffer to -2147482773
[ffmpeg] Failed to reallocate parser buffer to -2147483584
[ffmpeg] Failed to reallocate parser buffer to -2147483584
[ffmpeg] NULL: missing picture in access unit with size 579234816
(Buffering) V: 00:00:38 / unknown (71%) Dropped: 262

attempt 2
[ffmpeg] Failed to reallocate parser buffer to -2147483292
[ffmpeg] Failed to reallocate parser buffer to -2147483584
[ffmpeg] Failed to reallocate parser buffer to -2147483584
[ffmpeg] Failed to reallocate parser buffer to -2147483584
[ffmpeg] Failed to reallocate parser buffer to -2147483584
[ffmpeg] Failed to reallocate parser buffer to -2147483584
[ffmpeg] Failed to reallocate parser buffer to -2147483584
[ffmpeg] NULL: missing picture in access unit with size 1418189824
(Buffering) V: 00:00:24 / unknown (71%) Dropped: 129

https://trac.ffmpeg.org/ticket/8732
mpv-player/mpv#5893
mpv-player/mpv#5755

My CPU is Intel i-13900K chip so I use Intel QSV for hardware speedup (i915 kernel module).

My assumption is that all of this happens due to flakey ONVIF cameras and a switch dropping packets. Because of that, sometimes h264 video playing slows down or even stops, possibly while scanning for next full frame. Probably some combination of long series of events triggers a bug somewhere. It seems I will really need to re-encode the original 3gp files into different stream (h265) to ensure the incomplete frames are discarded. This also means the FPS (originally 15 or 25 fp) will appear variable frame rate (VFR) instead of fixed because many of the frames were randomly lost along the way to NVR. But then I will run into a different issue with dvr-scan as it does not support variable frame rate.

I think the design could be completely changed. A recording should be analyzed, regions of repeated movement (trees blowing in the wind) detected as glossy regions and their median color (a car may show up in front of them later so do not ask this region completely). In another sweep dvr-scan should look for motions in other places, even in a single frame. Further, dvr-scan detects some kernel size during it run. Provided I merge 3gp per 24-hrs into a single HEVC container (no re-encoding so far) all the videos start in night, in b/w color. I do not know what this kernel size is but certainly dvr-scan should split the video into ranges. This would be the very first pass of reading the file. Notably, the recordings at about the switch of IR cur filter are more grainy with distorted color and should be treated as a separate. Comparing individual/adjacent frames with each other does not make sense. The scene is largely static, sources of errors (small windblow movements can be learned) and then the real scene changes be detected more easily, even if recorded in a single frame.

Manual does not describe expectations so this is how I evolved my scripts to handle thousands of files per day and what I think is meaningful. Merging all videos per day into a single file is asking for a trouble but technically is the only way for me not to get lost in the zillions of files. It already sucks enough that ffmpeg -i list-of-files.txt ... -o merged.hevc dies due to the moov atom not found issue which means I have to edit the list-of-files.txt file and drop from it the problematic file and re-run ffmpeg.

from dvr-scan.

Breakthrough avatar Breakthrough commented on September 9, 2024

It seems I will really need to re-encode the original 3gp files into different stream (h265) to ensure the incomplete frames are discarded. This also means the FPS (originally 15 or 25 fp) will appear variable frame rate (VFR) instead of fixed because many of the frames were randomly lost along the way to NVR. But then I will run into a different issue with dvr-scan as it does not support variable frame rate.

The video output should be correct, as long as you are using the default OpenCV output mode. VFR video is only an issue for the accuracy of timestamps, which is why the ffmpeg ouptut mode won't work with it currently. That being said, there should be a filter for ffmpeg that will allow you to convert the VFR video into a fixed framerate one - you may want to consider that as a workaround.

A recording should be analyzed, regions of repeated movement (trees blowing in the wind) detected as glossy regions and their median color (a car may show up in front of them later so do not ask this region completely). In another sweep dvr-scan should look for motions in other places, even in a single frame

This sounds possible and like a good iea, but am lacking a lot of test videos which makes it difficult to develop these kinds of features. That being said, you may have some luck experimenting with the kernel size setting, which should help reduce small movements due to trees for example, and using the region editor to mask out problematic areas that aren't of interest.

from dvr-scan.

mmokrejs avatar mmokrejs commented on September 9, 2024

@Breakthrough Did you get my emails with links to the testcases causing dvr-scan to stop processing input files?

from dvr-scan.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.