Merging fails due to duplicated MOOV Atoms about livestream_saver HOT 9 CLOSED

I am confused. Did you use the download script from this repository to get the segments, or some other script?

the merged file only has one segment in it.

Not sure what you mean by that.

The errors you get might be because you are missing some segments, or maybe it's not a DASH format.

from livestream_saver.

I'm still looking into this, but I am very confused as to what happens exactly.

The script you linked is actually a git submodule in the /archived path of this here repo (although we don't use it at all, it's just there for reference). From my experience, merge_v1() never worked, that's why I only re-implemented merge_v2().
But this issue forced me to re-implement v1 here anyway. It still does not work: the resulting concatenated .ts file has a messed up duration (probably due to timestamps adding up, instead of simply updating the total duration of the stream).

The solution was to merge the video segments manually as described in the linked stackoverflow question.

Did you use the method described in the question or in the answer? The answer uses the concat demuxer, which is essentially the same as merge_v1(). I'm trying to understand why it would work for you in this case.

If you still have those segments available, would you mind sharing a subset of them for me to investigate?

Also I'd like to know which version of ffmpeg you are using.

I also forgot to mention that I pushed a commit to log ffmpeg's output even during the concatenation phase, perhaps that will show something.

from livestream_saver.

I have removed the use of concat protocol since it does not seem very reliable and may crash if the argument list overflows. I'll keep the code around in case it becomes useful again. 1a08482

I doubt the Found duplicated MOOV Atom. Skipped it warnings is of any real concern.
The Failed to add index entry error however is probably why you were missing concatenated segments.
It might also be due to an outdated version of ffmpeg (I currently use version 4.3.2).

Either way, the concat demuxer is currently used as a fallback in case the duration of the file doesn't match what would be reasonably expected. Hopefully this will be enough to work around any future problem.

from livestream_saver.

Thanks for the added details. It seems that ffmpeg fails in some way during the first merge phase (when it concatenates the .ts segments).

If you have not yet deleted your segment files, perhaps you could run the merge script again with --log DEBUG and check the logs for any ffmpeg related error message.

I should write more checks after that phase to make sure the intermediate concatenated file is the right length.

from livestream_saver.

I have pushed a possible fix for this but this needs more testing.
It should now try to detect if the resulting concatenated file has an abnormal duration, in which case it falls back to using the concat demuxer as the first fallback method, then the concat protocol (which works, despite showing the "duplicated MOOV Atom" warnings).

from livestream_saver.

Sorry for the lack of detail. I used the stream_downloader.py from the repository.
The errors appeared during the intermediate merging of the video segments and the resulting concatenated .ts file only had the length of one single segment after being processed by ffmpeg (apparently because ffmpeg simply skipped over the rest).
The solution was to merge the video segments manually as described in the linked stackoverflow question.
After this, the merge.py script from the repository was able to pick up from the intermediate file and create a fully working mp4 file. So I'm sure no segments were missing.

from livestream_saver.

Well, running with --log DEBUG doesn't crank up the log-level of the first ffmpeg run, so this doesn't show any errors. If I interrupt merge.py with CTRL+C and run ffmpeg on the concatenated .ts file like that

ffmpeg -hide_banner -y -i concat_xxx_video.ts -c copy -movflags faststart xxx_video_v2_ffmpeg.mp4

I get a lot of those messages already pasted above:

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x55b154b89440] Found duplicated MOOV Atom. Skipped it
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x55b154b89440] Failed to add index entry
    Last message repeated 149 times

By the way, I've found another script with the same purpose as yours, which seems to implement two different merge methods and tries both. I assume, because the one method implemented in your script doesn't always work:
https://github.com/mrwnwttk/youtube_stream_capture/blob/main/merge.py

from livestream_saver.

Indeed, I've used the method described in the answer.

This is my ffmpeg:

ffmpeg version 3.4.8 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 7 (SUSE Linux)
  configuration: --prefix=/usr --libdir=/usr/lib64 --shlibdir=/usr/lib64 --incdir=/usr/include/ffmpeg --extra-cflags='-fmessage-length=0 -grecord-gcc-switches -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables -fstack-clash-protection -g' --optflags='-fmessage-length=0 -grecord-gcc-switches -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables -fstack-clash-protection -g' --disable-htmlpages --enable-pic --disable-stripping --enable-shared --disable-static --enable-gpl --disable-openssl --enable-avresample --enable-libcdio --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libcelt --enable-libcdio --enable-libdc1394 --enable-libfreetype --enable-libgsm --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libzimg --enable-libzvbi --enable-vaapi --enable-vdpau --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-version3 --enable-libx264 --enable-libx265 --enable-libxvid
  libavutil      55. 78.100 / 55. 78.100
  libavcodec     57.107.100 / 57.107.100
  libavformat    57. 83.100 / 57. 83.100
  libavdevice    57. 10.100 / 57. 10.100
  libavfilter     6.107.100 /  6.107.100
  libavresample   3.  7.  0 /  3.  7.  0
  libswscale      4.  8.100 /  4.  8.100
  libswresample   2.  9.100 /  2.  9.100
  libpostproc    54.  7.100 / 54.  7.100
Hyper fast Audio and Video encoder
usage: ffmpeg [options] [[infile options] -i infile]... {[outfile options] outfile}...

Use -h to get full help or, even better, run 'man ffmpeg'

Unfortunately, I don't have the segments anymore and the stream was only available temporarily.

from livestream_saver.

That sounds great. Thanks! I will try this out once the next opportunity comes up.

I've just noticed that there is a separate ffmpeg-4 package available for OpenSUSE, so I will install that if any further problems occur.

from livestream_saver.