Metadata issue leading to out-of-sync audio after recording on server about haishinkit.swift HOT 16 CLOSED

@shogo4405 your PR #1446 fixes the issue in my testing.

Thank you for your patience, diligence, and perseverance in getting this bug fixed. 🙏 It certainly was a tricky one to reproduce.

from haishinkit.swift.

I now have a way to reliably reproduce this issue. I've created a simple project that runs nginx-rtmp-module inside of a Docker container and a simple Node server to validate RTMP stream requests: https://github.com/alexmck/docker-nginx-rtmp

There are two RTMP end points in this project, /live which does not send a request to authorise a stream, and /delay which adds an artificial 2 second delay while it authorises a stream.

@zachsimone was able to confirm my findings. Recordings made to the /live RTMP endpoint had no audio and video sync issues, while some recordings to the /delay RTMP endpoint did exhibit audio and video sync issues. I should note that in our testing, it wasn't every stream to the /delay endpoint. In my testing with the sample HaishinKit app, I found that a second stream would often have a recording that would exhibit the audio and video sync issues. Zach said in his testing it was anywhere from the 2nd to 6th test stream.

The significance here is that many live streaming platforms authorise an RTMP stream before allowing the user to stream, while users who are streaming with 4G and poor latency may also exhibit the same problem even without authorisation from the server.

Please let me know if there is anything else I can do to help resolve this issue or if you have any questions about my testing methodology.

from haishinkit.swift.

Thanks for your attempted fix. I have tested this branch but unfortunately the bug still exists. Zach was also still able to reproduce this issue with this branch.

To delay the stream in nginx, I am using the on_publish directive in my nginx.conf. This on_publish method sends an HTTP request to a simple Node HTTP server I set up which adds a 2 second delay before allowing the stream to continue. My server.js file runs the simple HTTP server that adds this 2 second delay.

Please let me know if I can provide you with any more details or answer anymore questions.

from haishinkit.swift.

Thank you for your confirmation and for providing the reproduction steps. I have merged the revised version.

from haishinkit.swift.

I tried analyzing the received FLV file. It seems that there is one audio header information before and after the metadata. There seem to be cases where it works and cases where it doesn't. Could you please provide the FLV file that worked when it was successful?

from haishinkit.swift.

What will happen with specifying fcPublishName pattern in this branch? Will it be similar? https://github.com/shogo4405/HaishinKit.swift/pull/1414/files

from haishinkit.swift.

I confirmed it with VideoLAN and ffplay, but I couldn't notice any delay between the video and audio. How are you playing it?

from haishinkit.swift.

Hi @shogo4405, thank you for the quick reply.

What will happen with specifying fcPublishName pattern in this branch? Will it be similar? https://github.com/shogo4405/HaishinKit.swift/pull/1414/files

Unfortunately I still see the issue when using this newer version.

I couldn't notice any delay between the video and audio. How are you playing it?

My apologies, I didn't check before uploading and uploaded the wrong files. Attaching new recordings below. One is definitely out of sync, and the other is working well. The problematic one is noted in the file name. I'm playing them back using VLC on macOS.

Please let me know if I can provide any more information that might be helpful here :-)

Recordings.zip

from haishinkit.swift.

This is an image showing a dump of the contents of an FLV file. It can be observed that the timestamp of the data starts from 579, indicating a misalignment. Applications like VideoLAN tend to adhere to this timestamp, resulting in a discrepancy between the video and audio. It's implemented in https://github.com/shogo4405/HaishinKit.swift/tree/main/Examples/macOS.

from haishinkit.swift.

Unfortunately, I'm struggling to reproduce it on my end.
I'm testing it in the following environment. Would the same issue occur in the Example iOS?

• iPhone15 Pro max + AirPods Pro + 44100.0 khz
• main branch: 1c24433
• Example iOS

% brew info rtmp-nginx-module
==> marcqualie/nginx/rtmp-nginx-module: stable 1.2.2-r1
NGINX-based Media Streaming Server
https://github.com/sergey-dryabzhinsky/nginx-rtmp-module
/opt/homebrew/Cellar/rtmp-nginx-module/1.2.2-r1_3 (94 files, 1.4MB) *
  Built from source on 2023-11-02 at 01:54:03
From: https://github.com/marcqualie/homebrew-nginx/blob/HEAD/Formula/rtmp-nginx-module.rb

from haishinkit.swift.

Thanks @shogo4405. Just a quick note to let you know I haven't forgotten about this and I'm trying to find a good way to reproduce that is decoupled from our streaming infrastructure. I'll let you know when I have more info.

from haishinkit.swift.

Just a thought after looking at the change, before the change FCPublish was called right after cleaning the timestamps with dataTimestamps.removeAll(). Now it's triggered from from createStream by handling NetStream.Connect.Success event. Maybe if there's a longer network delay before the createStream is triggered, the timestamps will accumulate the delay? Manually delaying createStream call would confirm that.

from haishinkit.swift.

Thank you @levs42. That gives me something to test to confirm this theory, and actually makes some sense as to why we might be seeing this issue.

from haishinkit.swift.

I believe I now have a method to consistently reproduce this error. I'm going to sanity check it with Zach and we will come back.

from haishinkit.swift.

If dataTimestamps is the cause, I believe I can improve it with this commit. There is a race condition because lockQueue is used within send(), but I believe we have a 100% reset understanding here. Please confirm.
main...feature/fix-outof-sync
https://github.com/shogo4405/HaishinKit.swift/tree/feature/fix-outof-sync

If possible, could you provide the code to delay nginx? My email is [email protected].

from haishinkit.swift.

@alexmck I was able to reproduce the issue using the method you introduced. Thank you. #1410 (comment)

I have confirmed that the issue is improved with this PR on my end. Please check if it works the same in your environment.#1446

from haishinkit.swift.