non906 / fftools_lib Goto Github PK
View Code? Open in Web Editor NEWLicense: Other
License: Other
Unity Crashes whenever you stop and start a stream.
Options used: -rtsp_transport tcp -probesize 32 -flags low_delay -vf setpts=0
To reproduce:
It usually crashes with this stack trace:
========== OUTPUTTING STACK TRACE ==================
0x00007FFEAAE9B7C4 (avfilter-8) avfilter_graph_dump
0x00007FFEAAE9CB9E (avfilter-8) avfilter_graph_parse_ptr
0x00007FFEE5BC96C0 (libffmpegDll) ffplay_set_audio_spec_size
0x00007FFEE5BCA0E3 (libffmpegDll) ffplay_set_audio_spec_size
0x00007FFEE5BCAF9D (libffmpegDll) ffplay_set_audio_spec_size
0x00007FFF1B2B4D53 (libwinpthread-1) pthread_create_wrapper
0x00007FFF22CBE634 (msvcrt) beginthreadex
0x00007FFF22CBE70C (msvcrt) endthreadex
0x00007FFF23A3269D (KERNEL32) BaseThreadInitThunk
0x00007FFF24ECA9F8 (ntdll) RtlUserThreadStart
========== END OF STACKTRACE ===========
This happens even with no Audio Component attached.
Using the profiler, we can see that we get two threads when a stream is running:
After closing and opening the stream again, two new threads are created but the threads from the previous stream are still present.
To fix the crash I added these to the Dispose method:
protected virtual void Dispose()
{
_source.Cancel();
if (_initCoroutine != null)
{
StopCoroutine(_initCoroutine);
}
//...
}
I also changed Thread and CallEventThread to tasks and they run on the threadpool and gave them a cancellation token.
This may have fixed the crashes but theres a bigger problem. Whenever I stop and start a stream, the CPU usage goes up by a lot and stays like that until Unity is closed (even not in playmode).
For example:
Start a stream, close the stream, open the same stream. Repeat 4+ times.
This results in 100% CPU usage and it won't clear even when exiting playmode and Unity has to be closed via task manager.
When the stream is stopped, the CPU usage goes down, but once it's restarted, the CPU usage a lot higher, and continues like this for each stop and start.
Possible problems:
Here is two problems:
Is there a good way to Optimize this issue?
First of all, these two mistakes are accidental. I run my program in Linux standalone build. I debug for days but not find reason.
Once the error happens, the program crashed.
Once the error happens, if I call FfmpegCommand.StopFfmpeg() function the program and editor will stuck forever. I debug in Windows editor and found that program running is stuck at ffmpegThread_.Join() at line 159 in FfmpegCommandImpWinLib.cs. In ffmpegThread_, running is stuck at line 99 ffmpeg_start() function.
This error is start with NullReferenceException: Object reference not set to an instance of an object
, following log is in the picture.
As say in title, the FFmpeg for Unity v2.7 add libopenh264.so in Linux build, which result the issue. The previous versions (<=2.5) work well. Could you please provide a libopenh264.so works on GLIBC_2.30 and before?
========== OUTPUTTING STACK TRACE ==================
0x00007FF9B71CD669 (msvcrt) strncmp
0.00 A-V: nan fd= 0 aq= 21KB vq= 639KB sq= 0B f=0/0
0x00007FF94B76D7C8 (libffmpegDll) ffplay_set_audio_spec_size
0x00007FF99AC34D53 (libwinpthread-1) pthread_create_wrapper
0x00007FF9B71AAF5A (msvcrt) beginthreadex
0x00007FF9B71AB02C (msvcrt) endthreadex
0x00007FF9B7397614 (KERNEL32) BaseThreadInitThunk
0x00007FF9B81A26A1 (ntdll) RtlUserThreadStart
========== END OF STACKTRACE ===========
I use FfplayCommand and FfmpegCaptureCommand as follow:
using FfmpegUnity;
using System;
using UnityEngine;
using System.IO;
using OwnRequestStruct;
public class FFmpegManager : MonoBehaviour
{
GameObject captureObject;
FfmpegCaptureCommand capture;
FfplayCommand videoPlayer;
FfmpegCommand videoConverter;
private void Awake()
{
// 初始化Ffmpeg for unity,用于音视频录制
captureObject = new GameObject("FFmpeg");
// 使用Ffmpeg for unity播放视频
videoPlayer = captureObject.AddComponent<FfplayCommand>();
videoPlayer.ExecuteOnStart = false;
videoPlayer.DefaultPath = FfmpegPath.DefaultPath.NONE;
var videoTexture = gameObject.AddComponent<FfmpegPlayerVideoTexture>();
videoTexture.VideoTexture = GetComponent<InitBaseScense>().imageShow.texture;
videoPlayer.VideoTexture = videoTexture;
var audioSourceVideo = captureObject.AddComponent<AudioSource>();
videoPlayer.AudioSourceComponent = audioSourceVideo;
// 使用Ffmpeg for unity调整音轨时间
videoConverter = captureObject.AddComponent<FfmpegCommand>();
videoConverter.ExecuteOnStart = false;
}
#region FFmpeg相关函数
/// <summary>
/// 开始录制,在这里初始化录制脚本,避免修改分辨率以后画面错位问题。
/// 以“.mp4”结尾录制为文件,否则推流,流地址为rtmp://ip:1935/live/{savePath}
/// </summary>
/// <param name="savePath"></param>
public void StartRecord(string savePath, RequestStruct req)
{
// 设置相关参数
capture = captureObject.AddComponent<FfmpegCaptureCommand>(); // 视频录制脚本
capture.ExecuteOnStart = false;
capture.req = req;
capture.callback = GetComponent<Callback>();
var video = capture.CaptureSources[0];
video.Type = FfmpegCaptureCommand.CaptureSource.SourceType.Video_Camera;
video.SourceCamera = Camera.main;
video.Width = Screen.width;
video.Height = Screen.height;
video.FrameRate = 30;
// 录制为文件或推流
string recordArgs;
string msg;
var ext = Path.GetExtension(savePath);
if (ext == "")
{
string rtmp = $"rtmp://127.0.0.1:1935/live/{savePath}";
recordArgs = $"-c:v libopenh264 -c:a aac -map 0:v:0 -map 1:a:0 -g 30 -vsync 1 -async 96000 -f flv {rtmp}";
msg = $"Vidoe will stream to: {rtmp}";
}
else
{
if (ext == ".webm") recordArgs = $"-c:v libvpx-vp9 -c:a libvorbis -map 0:v:0 -map 1:a:0 -crf 30 -b:v 0 -deadline realtime -cpu-used 8 {savePath}";
else if (ext == ".mp4") recordArgs = $"-c:v libopenh264 -c:a aac -map 0:v:0 -map 1:a:0 {savePath}";
else throw new NotImplementedException($"Recording video to {ext} is unsupport");
msg = $"Vidoe will save to: {savePath}";
}
// 开始FFmpeg录制线程
capture.CaptureOptions = recordArgs;
capture.StartFfmpeg();
Debug.Log($" [ - FFmpeg - ] Start recording. Res: {video.Width} x {video.Height}. {msg}");
}
public void StopRecord()
{
// 停止FFmpeg线程
if (capture.IsRunning)
{
capture.StopFfmpeg();
Destroy(capture);
Destroy(Camera.main.GetComponent<FfmpegCaptureCamera>());
Debug.Log(" [ - FFmpeg - ] Stop recording.");
}
}
public enum VideoExt
{
mp4,
avi,
flv,
m4v,
mkv,
mov,
webm,
wmv
}
/// <summary>
/// 播放视频文件,经过测试,支持的后缀为:mp4,avi,flv,m4v,mkv,mov,webm,wmv
/// </summary>
/// <param name="videoPath"></param>
/// <param name="volume"></param>
/// <exception cref="Exception"></exception>
public void PlayVideoFile(string videoPath, float volume=1.0f)
{
if (!File.Exists(videoPath))
throw new Exception("Video file not exist: " + videoPath);
GetComponent<AudioPlayer>().StopPlay(); // 关闭音频
if (videoPlayer.IsRunning)
{
videoPlayer.Dispose();
}
videoPlayer.InputPath = videoPath;
videoPlayer.Volume = volume;
videoPlayer.Play();
GetComponent<InitBaseScense>().canvasShow.enabled = true;
Debug.Log($" [ - FFmpeg - ] Start playing: {videoPath}. volume={volume}");
}
public void StopPlay()
{
if (videoPlayer.IsRunning)
{
videoPlayer.Dispose();
GetComponent<InitBaseScense>().canvasShow.enabled = false;
Debug.Log(" [ - FFmpeg - ] Stop playing");
}
}
private void Update()
{
if (!videoPlayer.IsRunning)
return;
// 视频播放完毕时IsRunning=true,Paused=true
if (videoPlayer.Paused)
{
StopPlay();
}
}
public void Convert(string inputPath, string outputPath, float audioDelaySec)
{
videoConverter.Options = string.Join(
"\n",
"-y",
$"-i {inputPath}",
$"-itsoffset {audioDelaySec}",
$"-i {inputPath}",
"-map 0:v",
"-map 1:a",
"-c copy",
outputPath
);
videoConverter.ExecuteFfmpeg();
Debug.Log($"Convert {inputPath} to {outputPath} by delaying audio {audioDelaySec} s");
}
#endregion
}
In windows editor, if you call PlayVideoFile() and StopPlay() or StartRecord() and StopRecord() many times, there is no memory leak.
In linux build, invoking 10 times StartRecord(), PlayVideoFile(), StopPlay() and StopPlay() will result in about 1GB of memory leakage. Here is the error code. Onec the issue occurres, the errors will loop to print many times :
OutOfMemoryException: Out of memory
at (wrapper managed-to-native) System.Object.__icall_wrapper_ves_icall_array_new_specific(intptr,int)
at System.Collections.Generic.List`1[T].set_Capacity (System.Int32 value) [0x00021] in <0bfb382d99114c52bcae2561abca6423>:0
at System.Collections.Generic.List`1[T].EnsureCapacity (System.Int32 min) [0x00036] in <0bfb382d99114c52bcae2561abca6423>:0
at System.Collections.Generic.List`1[T].InsertRange (System.Int32 index, System.Collections.Generic.IEnumerable`1[T] collection) [0x00032] in <0bfb382d99114c52bcae2561abca6423>:0
at System.Collections.Generic.List`1[T].AddRange (System.Collections.Generic.IEnumerable`1[T] collection) [0x00000] in <0bfb382d99114c52bcae2561abca6423>:0
at FfmpegUnity.FfmpegCaptureCommand.OnAudioFilterWriteToCaptureAudio (System.Single[] data, System.Int32 channels, System.Int32 streamId) [0x000b6] in <9a6ccb49780e4cc8976535d3f6ffa12c>:0
at FfmpegUnity.FfmpegCaptureAudio.OnAudioFilterRead (System.Single[] data, System.Int32 channels) [0x0000e] in <9a6ccb49780e4cc8976535d3f6ffa12c>:0
i try use -ast audio_stream_specifier, but it does not work
I have a project that mush run in URP(Universal Render Pipeline). I try to record video but no files generated. We can find in Unity Doc with this: "OnRenderImage is not supported in the Scriptable Render Pipeline. "
More and more projects are using URP, and I think this tool also needs to be upgraded to support it.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.