Git Product home page Git Product logo

fftools_lib's People

Contributors

non906 avatar

Stargazers

 avatar  avatar  avatar

Watchers

 avatar

fftools_lib's Issues

Add libvpx support for windows Library

  • I bought the FfmpegUnity tool from the unity store. Thank you so much for developing this tool. It works very well.
  • I want to record video in webm format, but it only works with linux build. The windows editor will prompt a parameter error. I opened Print Error and found that windows library compiled without libvpx support on. I also used the windows Build in Binary you provided, which could record but the video frame rate was abnormal. The recorded video flashed away, and the video that was supposed to last 10 seconds played for only two seconds. Recording as mp4 is fine.
  • In addition, I tried using Ffmpeg-windows-build-helpers compile ffmpeg Library myself, but I haven't been able to do that because of network problems.
  • Therefore, I hope you can update the windows Library by enabling libvpx, after all, windows and linux libraries should be as consistent as possible. And unity supports webm natively.

Unity crashes when RTSP is closed and opened a few times.

Unity Crashes whenever you stop and start a stream.

Options used: -rtsp_transport tcp -probesize 32 -flags low_delay -vf setpts=0

To reproduce:

  1. Start a RTSP stream in FfplayRenderTexture/FfplayTexture
  2. Stop the stream
  3. Start the same stream
  4. Unity crash

It usually crashes with this stack trace:

========== OUTPUTTING STACK TRACE ==================

0x00007FFEAAE9B7C4 (avfilter-8) avfilter_graph_dump
0x00007FFEAAE9CB9E (avfilter-8) avfilter_graph_parse_ptr
0x00007FFEE5BC96C0 (libffmpegDll) ffplay_set_audio_spec_size
0x00007FFEE5BCA0E3 (libffmpegDll) ffplay_set_audio_spec_size
0x00007FFEE5BCAF9D (libffmpegDll) ffplay_set_audio_spec_size
0x00007FFF1B2B4D53 (libwinpthread-1) pthread_create_wrapper
0x00007FFF22CBE634 (msvcrt) beginthreadex
0x00007FFF22CBE70C (msvcrt) endthreadex
0x00007FFF23A3269D (KERNEL32) BaseThreadInitThunk
0x00007FFF24ECA9F8 (ntdll) RtlUserThreadStart

========== END OF STACKTRACE ===========

This happens even with no Audio Component attached.

Using the profiler, we can see that we get two threads when a stream is running:
image

After closing and opening the stream again, two new threads are created but the threads from the previous stream are still present.

To fix the crash I added these to the Dispose method:

 protected virtual void Dispose()
        {
            _source.Cancel();

            if (_initCoroutine != null)
            {
                StopCoroutine(_initCoroutine);
            }
           //...
        }

I also changed Thread and CallEventThread to tasks and they run on the threadpool and gave them a cancellation token.

This may have fixed the crashes but theres a bigger problem. Whenever I stop and start a stream, the CPU usage goes up by a lot and stays like that until Unity is closed (even not in playmode).
For example:
Start a stream, close the stream, open the same stream. Repeat 4+ times.
This results in 100% CPU usage and it won't clear even when exiting playmode and Unity has to be closed via task manager.
When the stream is stopped, the CPU usage goes down, but once it's restarted, the CPU usage a lot higher, and continues like this for each stop and start.

Possible problems:

  1. When a stream stops, it doesn't close the stream and once start is called again, it is assigned a new ID and starts a new stream while the old one is still in usage.
  2. The native wrapper for FFMPEG has a memory leak somewhere.
  3. Calling ffplay_stop is not enough to clear the stream.

The audio and image are out of sync

Here is two problems:

  • The audio and image are out of sync in recorded video. The latency is about 0.2s to 1s at random, and the audio is always faster than image.
  • Video image is stuck sometimes, but audio is fine. Perhaps this issue led to the previous one.

Is there a good way to Optimize this issue?

FFplay segmentation fault and FfmpegCaputure writeVideo BUG

First of all, these two mistakes are accidental. I run my program in Linux standalone build. I debug for days but not find reason.

1. FFplay segmentation fault

Once the error happens, the program crashed.

image

2. FfmpegCaputure writeVideo Bug

Once the error happens, if I call FfmpegCommand.StopFfmpeg() function the program and editor will stuck forever. I debug in Windows editor and found that program running is stuck at ffmpegThread_.Join() at line 159 in FfmpegCommandImpWinLib.cs. In ffmpegThread_, running is stuck at line 99 ffmpeg_start() function.
This error is start with NullReferenceException: Object reference not set to an instance of an object, following log is in the picture.

image

I Got a Crash when play some videos with four ffplayCommands

========== OUTPUTTING STACK TRACE ==================

0x00007FF9B71CD669 (msvcrt) strncmp
0.00 A-V: nan fd= 0 aq= 21KB vq= 639KB sq= 0B f=0/0
0x00007FF94B76D7C8 (libffmpegDll) ffplay_set_audio_spec_size
0x00007FF99AC34D53 (libwinpthread-1) pthread_create_wrapper
0x00007FF9B71AAF5A (msvcrt) beginthreadex
0x00007FF9B71AB02C (msvcrt) endthreadex
0x00007FF9B7397614 (KERNEL32) BaseThreadInitThunk
0x00007FF9B81A26A1 (ntdll) RtlUserThreadStart

========== END OF STACKTRACE ===========

Using FfmpegCaptureCommand will cause Memory leak in Linux build

I use FfplayCommand and FfmpegCaptureCommand as follow:

using FfmpegUnity;
using System;
using UnityEngine;
using System.IO;
using OwnRequestStruct;

public class FFmpegManager : MonoBehaviour
{
    GameObject captureObject;
    FfmpegCaptureCommand capture;
    FfplayCommand videoPlayer;
    FfmpegCommand videoConverter;


    private void Awake()
    {

        // 初始化Ffmpeg for unity,用于音视频录制
        captureObject = new GameObject("FFmpeg");

        // 使用Ffmpeg for unity播放视频
        videoPlayer = captureObject.AddComponent<FfplayCommand>();
        videoPlayer.ExecuteOnStart = false;
        videoPlayer.DefaultPath = FfmpegPath.DefaultPath.NONE;
        var videoTexture = gameObject.AddComponent<FfmpegPlayerVideoTexture>();
        videoTexture.VideoTexture = GetComponent<InitBaseScense>().imageShow.texture;
        videoPlayer.VideoTexture = videoTexture;
        var audioSourceVideo = captureObject.AddComponent<AudioSource>();
        videoPlayer.AudioSourceComponent = audioSourceVideo;

        // 使用Ffmpeg for unity调整音轨时间
        videoConverter = captureObject.AddComponent<FfmpegCommand>();
        videoConverter.ExecuteOnStart = false;
    }

    #region FFmpeg相关函数
    /// <summary>
    /// 开始录制,在这里初始化录制脚本,避免修改分辨率以后画面错位问题。
    /// 以“.mp4”结尾录制为文件,否则推流,流地址为rtmp://ip:1935/live/{savePath}
    /// </summary>
    /// <param name="savePath"></param>
    public void StartRecord(string savePath, RequestStruct req)
    {
        // 设置相关参数
        capture = captureObject.AddComponent<FfmpegCaptureCommand>();   // 视频录制脚本
        capture.ExecuteOnStart = false;
        capture.req = req;
        capture.callback = GetComponent<Callback>();
        var video = capture.CaptureSources[0];
        video.Type = FfmpegCaptureCommand.CaptureSource.SourceType.Video_Camera;
        video.SourceCamera = Camera.main;
        video.Width = Screen.width;
        video.Height = Screen.height;
        video.FrameRate = 30;

        // 录制为文件或推流
        string recordArgs;
        string msg;

        var ext = Path.GetExtension(savePath);
        if (ext == "")
        {
            string rtmp = $"rtmp://127.0.0.1:1935/live/{savePath}";
            recordArgs = $"-c:v libopenh264 -c:a aac -map 0:v:0 -map 1:a:0 -g 30 -vsync 1 -async 96000 -f flv {rtmp}";
            msg = $"Vidoe will stream to: {rtmp}";
        }
        else
        {
            if (ext == ".webm") recordArgs = $"-c:v libvpx-vp9 -c:a libvorbis -map 0:v:0 -map 1:a:0 -crf 30 -b:v 0 -deadline realtime -cpu-used 8 {savePath}";
            else if (ext == ".mp4") recordArgs = $"-c:v libopenh264 -c:a aac -map 0:v:0 -map 1:a:0 {savePath}";
            else throw new NotImplementedException($"Recording video to {ext} is unsupport");
            msg = $"Vidoe will save to: {savePath}";
        }

        // 开始FFmpeg录制线程
        capture.CaptureOptions = recordArgs;
        capture.StartFfmpeg();
        Debug.Log($" [ - FFmpeg - ] Start recording. Res: {video.Width} x {video.Height}. {msg}");
    }


    public void StopRecord()
    {
        // 停止FFmpeg线程
        if (capture.IsRunning)
        {
            capture.StopFfmpeg();
            Destroy(capture);
            Destroy(Camera.main.GetComponent<FfmpegCaptureCamera>());
            Debug.Log(" [ - FFmpeg - ] Stop recording.");
        }
    }


    public enum VideoExt
    {
        mp4,
        avi,
        flv,
        m4v,
        mkv,
        mov,
        webm,
        wmv
    }
    
    /// <summary>
    /// 播放视频文件,经过测试,支持的后缀为:mp4,avi,flv,m4v,mkv,mov,webm,wmv
    /// </summary>
    /// <param name="videoPath"></param>
    /// <param name="volume"></param>
    /// <exception cref="Exception"></exception>
    public void PlayVideoFile(string videoPath, float volume=1.0f)
    {
        if (!File.Exists(videoPath))
            throw new Exception("Video file not exist: " + videoPath);
        GetComponent<AudioPlayer>().StopPlay(); // 关闭音频
        if (videoPlayer.IsRunning)
        {
            videoPlayer.Dispose();
        }
        videoPlayer.InputPath = videoPath;
        videoPlayer.Volume = volume;
        videoPlayer.Play();
        GetComponent<InitBaseScense>().canvasShow.enabled = true;
        Debug.Log($" [ - FFmpeg - ] Start playing: {videoPath}. volume={volume}");
    }

    public void StopPlay()
    {
        if (videoPlayer.IsRunning)
        {
            videoPlayer.Dispose();
            GetComponent<InitBaseScense>().canvasShow.enabled = false;
            Debug.Log(" [ - FFmpeg - ] Stop playing");
        }
    }

    private void Update()
    {
        if (!videoPlayer.IsRunning)
            return;

        // 视频播放完毕时IsRunning=true,Paused=true
        if (videoPlayer.Paused)
        {
            StopPlay();
        }
    }

    public void Convert(string inputPath, string outputPath, float audioDelaySec)
    {
        videoConverter.Options = string.Join(
          "\n",
          "-y",
          $"-i {inputPath}",
          $"-itsoffset {audioDelaySec}",
          $"-i {inputPath}",
          "-map 0:v",
          "-map 1:a",
          "-c copy",
          outputPath
          );
        videoConverter.ExecuteFfmpeg();
        Debug.Log($"Convert {inputPath} to {outputPath} by delaying audio {audioDelaySec} s");
    }
#endregion
}

In windows editor, if you call PlayVideoFile() and StopPlay() or StartRecord() and StopRecord() many times, there is no memory leak.
In linux build, invoking 10 times StartRecord(), PlayVideoFile(), StopPlay() and StopPlay() will result in about 1GB of memory leakage. Here is the error code. Onec the issue occurres, the errors will loop to print many times :

OutOfMemoryException: Out of memory
  at (wrapper managed-to-native) System.Object.__icall_wrapper_ves_icall_array_new_specific(intptr,int)
  at System.Collections.Generic.List`1[T].set_Capacity (System.Int32 value) [0x00021] in <0bfb382d99114c52bcae2561abca6423>:0
  at System.Collections.Generic.List`1[T].EnsureCapacity (System.Int32 min) [0x00036] in <0bfb382d99114c52bcae2561abca6423>:0
  at System.Collections.Generic.List`1[T].InsertRange (System.Int32 index, System.Collections.Generic.IEnumerable`1[T] collection) [0x00032] in <0bfb382d99114c52bcae2561abca6423>:0
  at System.Collections.Generic.List`1[T].AddRange (System.Collections.Generic.IEnumerable`1[T] collection) [0x00000] in <0bfb382d99114c52bcae2561abca6423>:0
  at FfmpegUnity.FfmpegCaptureCommand.OnAudioFilterWriteToCaptureAudio (System.Single[] data, System.Int32 channels, System.Int32 streamId) [0x000b6] in <9a6ccb49780e4cc8976535d3f6ffa12c>:0
  at FfmpegUnity.FfmpegCaptureAudio.OnAudioFilterRead (System.Single[] data, System.Int32 channels) [0x0000e] in <9a6ccb49780e4cc8976535d3f6ffa12c>:0

URP support required

I have a project that mush run in URP(Universal Render Pipeline). I try to record video but no files generated. We can find in Unity Doc with this: "OnRenderImage is not supported in the Scriptable Render Pipeline. "

More and more projects are using URP, and I think this tool also needs to be upgraded to support it.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.