Please Add Support for Android and IOS

Hi Keijiro. Nice job on this plugin! However I noticed that there is a drop in quality for line objects once recording starts. I was able to fix this by adding this line after you define _tempTarget in CameraCapture.cs:
_tempTarget.antiAliasing = 8;

I hope this helps!

is H.264 (default) the MPEG-4 /AVC format? Are H.264 variants provided in the code exploit x264 libraries for encoding?

How to encode a single rendered texture or png frame at a given point in time using this package?

Hi I am trying to read the video -- ie stream to other "clients" while it is being "written to", is this a windows permission issue or is something to do with the code, ie it seems like the only way I can actually read the file is after its closed.

Okay I tried it on mac and the same issue. Is the video not a video until after it closes and does whatever it needs to, is there any easy way to change this?

We are currently developing an application using HDRP and using UI setting of Screen Space Overlay.
Although the UI is not included in the render texture under this setting, we have to use this setting in order for the UI to work in HDRP.
We also need to record the UI, but it is not recorded.
We are also using this recording feature in a build, so I suppose Unity Recorder would not be an option.
Does anyone have a easy way to work around this issue or this is a dead-end?

Had issues with changed API for RenderPipeline.beginCameraRendering so changed it to RenderPipelineManager.beginCameraRendering , adjusted the call and commented out legacy line

// FFmpegOut - FFmpeg video encoding plugin for Unity
// https://github.com/keijiro/KlakNDI

using UnityEngine;
using UnityEngine.Rendering;

namespace FFmpegOut
{
    sealed class Blitter : MonoBehaviour
    {
        #region Factory method

        static System.Type[] _initialComponents =
            { typeof(Camera), typeof(Blitter) };

        public static GameObject CreateInstance(Camera source)
        {
            var go = new GameObject("Blitter", _initialComponents);
            go.hideFlags = HideFlags.HideInHierarchy;

            var camera = go.GetComponent<Camera>();
            camera.cullingMask = 1 << UILayer;
            camera.targetDisplay = source.targetDisplay;

            var blitter = go.GetComponent<Blitter>();
            blitter._sourceTexture = source.targetTexture;

            return go;
        }

        #endregion

        #region Private members

        // Assuming that the 5th layer is "UI". #badcode
        const int UILayer = 5;

        Texture _sourceTexture;
        Mesh _mesh;
        Material _material;

        void OnBeginCameraRendering(ScriptableRenderContext context, Camera camera)
        {
            if (_mesh == null || camera != GetComponent<Camera>()) return;

            Graphics.DrawMesh(
                _mesh, transform.localToWorldMatrix,
                _material, UILayer, camera
            );
        }

        #endregion

        #region MonoBehaviour implementation

        void Update()
        {
            if (_mesh == null)
            {
                // Index-only triangle mesh
                _mesh = new Mesh();
                _mesh.vertices = new Vector3[3];
                _mesh.triangles = new int [] { 0, 1, 2 };
                _mesh.bounds = new Bounds(Vector3.zero, Vector3.one);
                _mesh.UploadMeshData(true);

                // Blitter shader material
                var shader = Shader.Find("Hidden/FFmpegOut/Blitter");
                _material = new Material(shader);
                _material.SetTexture("_MainTex", _sourceTexture);

                // Register the camera render callback.
                UnityEngine.Rendering.RenderPipelineManager.
                    beginCameraRendering += OnBeginCameraRendering; // SRP
				//	Camera.onPreCull += OnBeginCameraRendering; // Legacy
            }
        }

        void OnDisable()
        {
            if (_mesh != null)
            {
                // Unregister the camera render callback.
                UnityEngine.Rendering.RenderPipelineManager.
                    beginCameraRendering -= OnBeginCameraRendering; // SRP
				//	Camera.onPreCull += OnBeginCameraRendering; // Legacy

                // Destroy temporary objects.
                Destroy(_mesh);
                Destroy(_material);
                _mesh = null;
                _material = null;
            }
        }

        #endregion
    }
}

I attached the script CameraCapture from Runtime to my main camera gameobject, and I assigned it values as shown below:

However, I run it in editor and then stop running it and I don't know where the final video is stored, if it is being created at all. Please let me know any solutions to this.

Hi, is there a way to capture videos from multiple cameras in the scene at the same time?
Thanks

Hi!

I found a problem with that the camera view will flip and lose anti aliasing after recording with Set resolution is set to true. It works correctly if Set resolution is set to false.

I have tested this in a new project with only this added and it stills happens.
If I disable the CaptureCamera script after recording it flips back correctly but the anti aliasing is still missing.

The recent versions of FFmpeg supports hardware acceleration with NVidia GPUs. I'm sill not sure that it actually accelerate the encoding process.

Hi @keijiro, first thank you for your wonderful work!

I have some strange result when rendering with the Camera-Capture : it looks like the Z-buffer is corrupted.
Here is a sample : on the left, normal rendering, on the right when I activate the Camera-Capture script.

It seems back-faces are sometimes renderer after and in front of other faces.

Just let me know if you need to have access to my project.

Regards,
Adrien

When I have both FFmpegOut + KinoBloom, there is some weird things happening. It looks like a feedback effect and horizontal mirror.

Original scene (with KinoBloom):

With FFmpegOut:

Unity 5.6.0
Windows 10

While recording in HDRP, the game view only shows the skybox, and the recorded file is 1kb. Will there be HDRP support?

Aparently, the UI Canvases (World Space) are not captured/visible in the build app (Windows Standalone Player) but works fine in the Editor (on the right).

Any ideas why?

请问“CameraCapture.cs”中的“[SerializeField, HideInInspector] Shader _shader;”这句话中，这个_shader是如何初始化的？我都没有在脚本中看到初始化的语句，但运行起来确实是正常赋值的。请教一下，这个是怎么实现的呢？

Unity crashes after recording for more than ~20 seconds.

How to reproduce:

• Open Test scene.
• Set Record Length on camera to 60.
• Unity crashes after around 20 seconds.

Unity version: 5.5.2f1

The ability to include audio from Unity would be a perfect addition. I might work on this and submit a PR if no one else can.

Every time I record, the video output is half in length, if I say 50 seconds, the video is 25 seconds, and also it is accelerated. I am not sure what causing this? Any idea?

I have plan for build and run on linux server for without GPU card, I trying with currently code not working. Do you have any idea for it?, I'm programmer and I can support

I would like to manually start and stop a recording session in different output paths set within a script. Is it possible to do it? thank you!

Adding multithreading support to avoid hiccups during capture.

I'd like to use this to save the last X amount of seconds of footage i.e. record continuously and on demand, save only the last x seconds instead of saving the whole thing?

Another repo that can do this is https://github.com/Chman/Moments. It records continuously and when you press space, it saves the last few seconds as a gif.

Running the test scene after copying the FFmpeg binary executables in the right folders, makes the console scream...
Win32Exception: ApplicationName='/Users/neutral90/Desktop/FFmpegOut/Assets/StreamingAssets/FFmpegOut/OSX/ffmpeg', CommandLine='-y -f rawvideo -vcodec rawvideo -pixel_format rgb24 -video_size 1280x720 -framerate 30 -loglevel warning -i - -c:v prores_ks -pix_fmt yuv422p10le Main_Camera_2017_0412_121855.mov', CurrentDirectory='' System.Diagnostics.Process.Start_noshell (System.Diagnostics.ProcessStartInfo startInfo, System.Diagnostics.Process process) System.Diagnostics.Process.Start_common (System.Diagnostics.ProcessStartInfo startInfo, System.Diagnostics.Process process) System.Diagnostics.Process.Start (System.Diagnostics.ProcessStartInfo startInfo) FFmpegOut.FFmpegPipe..ctor (System.String name, Int32 width, Int32 height, Int32 framerate, Codec codec) (at Assets/FFmpegOut/FFmpegPipe.cs:39) FFmpegOut.CameraCapture.OpenPipe () (at Assets/FFmpegOut/CameraCapture.cs:133) FFmpegOut.CameraCapture.Update () (at Assets/FFmpegOut/CameraCapture.cs:79)
...about 177 times.

Unity 5.6.0f3, OS X 10.11.6

Hi,
with Unity 2018. LTS i have no issues but with Unity 2019.2.6 and HDRP (6.9 + 6.9.2) i can't see what the camera actual renders while recording. The recording works and the rendered video is like expected, but my gameview only shows the sky so i can't see what i'm actualy recording.

I have the problem that as soon as I build the player and perform the recording from the player, the generated video seems to have wrong color information. I suspected the -pixel_format for the raw video for ffmpeg would be wrong, but none of the combinations of rgb and a resulted in correct colors.

The green part is supposed to be gray (#6c6c6c) and the blue part is supposed to be brown.

When running from the editor, everything works as expected. I have tried on both macOS catalina (metal) and Ubuntu (Vulkan). The mac has intel integrated graphics, the ubuntu has NVIDIA card with official nvidia drivers.

Is it possible to use android binary's with FFmpegOut?

I attempted to detect if platform is android then changed the executable path to binary's that i found elsewhere. This did not work, but maybe it can work and i'm not doing something right.

I made a Unity demo to output an short animation with FFmpegout. Each game instance would output one file then close byitself. Now when I open three instances, the 3rd one opens but FFmpeg attach to it would close by itself. The 3rd file would be empty?

Could it be that FFmpeg refuse to run more than two instances?

Hi,
For internal scientific research (so no direct issues with licenses) I want to use your runtime option of the recorder. In a default project it works fine, but using VR (legacy XR because of using OpenVR with HTC Vive Pro) in 2019.4.1 with HDRP 7.4.1 the camera stops reacting to the headset. So there is an image, no errors, just the tracking stops. There is also no video saved in the project's root folder.
Is this recording solution compatible with the legacy XR in Unity? If so, are there any special settings?

Cheers!

Marco

Failed to initialize an FFmpeg session due to missing executable file. Please check FFmpeg installation.
Mac

So this isn't really an issue, it's more of an experiment and very long question about what the plugin is capable of.

I am trying to push realtime video encoding with the Nvidia HEVC codec to it's absolute max.

I'm having the Unity application run as fast as possible and I'm encoding frames at a constant rate using an "isDirty" flag (I'm using this modification #39 from a previous question I posted).

On an RTX 2080 I am able to capture either [email protected], 3392x2880@15fps, or 2560x2160@30fps. By my estimate this pipeline is pushing and encoding about 140-160 million pixels per second.

In editor the application is running at about 400 fps and I am able to record without dropping a single frame. So far so good.

If i try to raise it to a 3392x2880@30fps, or a 5120x4320@15 fps (about 290-330 million pixels per second) the pipeline begins to stall.

In editor the application framerate drops to around 10-20 fps and it's dropping about every other frame. When not recording the scene runs at 700fps when rendering at 5120x4320.

Specifically it's line 56 of FFmpegPipe.cs that is causing the stall:
while (_pipeQueue.Count > 4) _pipePong.WaitOne();

Nvidia claims that the RTX GPUs are capable of encoding 8k video, but I'm not sure if they specifically say it can encode 8k video in realtime.

Encoding 160 million pixels in a second is already pretty mind boggling, so I wouldn't be surprised if the hardware encoder is limited to around 200 million pixels per second.

I realize this is an absurd experiment and that this plugin wasn't meant for interactive or realtime capture. I just wanted to double check and make absolutely sure there wasn't something else in the pipeline that could be causing this slow down.

Thanks again for your time and help with the previous question.

This is probably because the complete path where the video is saved isn't in between " characters, so spaces break the arguments in the command line.

Hello,

I wanted to know if there is any way that the script won´t start directly after pressing play, cause in my case i need it to start when i press a button.

Thanks in advance!

Here is two picture, one is from game, the other is from FFmpegOut video:

The in game one look much more better than the output video, why??

Hi Keijiro! Huge fan of all of your work. I've been using FFmpegOut for a year or so, and I've been really happy with it. I much prefer it to every other solution I've found!

I upgraded to Unity 2018.3.0b8 to try out the newest iteration of FFmpegOut. It works very well, but the only issue is there are a few black frames at the beginning of all of my recordings. It's almost as if the CameraCapture begins recording before it is fully ready to render the scene or something. I tried starting the scene with CameraCapture component disabled, then enabled automatically via script after a second, but that didn't alleviate the issue.

I'm trying to make perfectly looping generative art videos, so the black frames are a bit of a bother.

I'm using CameraCapture with 1920x1920 resolution, H.264 NVIDIA (MP4) preset, and 60 fps. I'm also using FrameRateController with Offline mode enabled and the Frame Rate set to 60 fps.

Let me know if you need any more info!

RenderTexture.Create: Depth|ShadowMap RenderTexture requested without a depth buffer. Changing to a 16 bit depth buffer.

i do use post processing stack on the camera as well the capturing looks really weird

It's nice to have alpha channel support when using the ProRes codec.

Hi Keijiro,

As I have said before, amazing work as always!!
I have been running some tests, now with VR. I am using the built in OpenVR API to run my HTC Vive.
The recording blocks the HMD position and rotation tracking as well as displaying back the image to the headset. Currently I am using the script directly on the camera of the HMD, but I ll get back to you with a test where I put another camera only for the recording, that might be a good temporary solution.

I know this very much likely is a unity bug, but I'll report it here anyways.
Tested with an empty scene. If you start recording. You will notice your memory gets filled up quickly.
And even if you stop recording, the memory does not get released. This is true in both editor and build. Interestingly, if you are in the editor and stop playmode, the memory still does not get freed. Change scene does nothing, force GC collect or unload assets does nothing. In memory profiler, it seems total allocated momery shows huge amount of memory is allocated, but you don't see that in any thing in your game. Strange.
My suspect is GPUAsync request and getdata allocated persistant memory, and they never get disposed. I could be wrong. But my evidences point to it. It could be just unity and has nothing to do HDRP, but I lack testing on other SRPs.

I build a simple scene in Unity2017.1.0f3 to test capture something ,but when capture started,the camera renderering nothing except skybox,finally i only have a skybox vedio.What happened?What should i do?
The attachment is the simple scene,please take a look at it when you have free time,thank you!
CaptureTestScene.zip

HI, i want to resample a mp4 file in unity in order to reduce the mp4 file size and make speed up play effect, do you have any idea about the result ?

Hi there,
thanks for sharing the code. It was really helpful for my project. I tried to make it stream to an IP but I couldn't do it. Do you think you can add streaming support?
Cheers,
Nick

Is it possible to get audio alongside video? If so, how?

Where does the file get output?
I can't seem to see if it has recorded anything or not

Hi Keijiro!

This plugin is extremely helpful in my current project, so thank you very much for all of your work on it.

I am trying to encode a video at a framerate lower than my application framerate. As an example I want to be able to have the game run at 60fps, but record the video at 15fps.

So I only want to send a frame to the encoder every 4th update. However it seems that CameraCapture.cs is written to push a frame every single update, since the project was intended to capture video at the same framerate as the application.

I tried rewriting the update function of CameraCapture.cs to this:
`void Update()
{
var camera = GetComponent();

// Lazy initialization
if (_session == null)
{
    // Give a newly created temporary render texture to the camera
    // if it's set to render to a screen. Also create a blitter
    // object to keep frames presented on the screen.
    if (camera.targetTexture == null)
    {
        _tempRT = new RenderTexture(_width, _height, 24, GetTargetFormat(camera));
        _tempRT.antiAliasing = GetAntiAliasingLevel(camera);
        camera.targetTexture = _tempRT;
        _blitter = Blitter.CreateInstance(camera);
    }

    // Start an FFmpeg session.
    _session = FFmpegSession.Create(
        gameObject.name,
        camera.targetTexture.width,
        camera.targetTexture.height,
        _frameRate, preset
    );

    _startTime = Time.time;
    _frameCount = 0;
    _frameDropCount = 0;

}

if (isDirty)
{
    _session.PushFrame(camera.targetTexture);
    _frameCount++;
    isDirty = false;
}

}`

Where another script is setting the "isDirty" variable every 4th frame.

However I get this warning when trying to record:
GPU readback error was detected. UnityEngine.Debug:LogWarning(Object) FFmpegOut.FFmpegSession:ProcessQueue() (at Assets/FFmpegOut/Runtime/FFmpegSession.cs:186) FFmpegOut.FFmpegSession:PushFrame(Texture) (at Assets/FFmpegOut/Runtime/FFmpegSession.cs:55) FFmpegOut.CameraCapture:Update() (at Assets/FFmpegOut/Runtime/CameraCapture.cs:180)

and I get a 1kb video file.

Do you know what could be causing this issue? Is there any way to send a render texture to the encoder on demand, where it does not have to be every frame?

Hi @keijiro , first of all thank you for this great script, it's working nicely except below issue.

Resolutions higher than full-hd (1920x1080) causes error with H.264. It exports the .mp4 to the folder, size looks reasonable but video can't be played and resolution of video is blank. I tried enable/disable 'Set Resolution' tickbox or change it via script but nothing worked (Everything works okay with ProRes). Is that a technical inability of the codec, thus can't be overcome?

Here are 2 different errors i get on different captures:

ffmpeg returned with a warning or an error message. See the following lines for details:
[swscaler @ 000000000035cfa0] Warning: data is not aligned! This can lead to a speedloss

UnityEngine.Debug:LogWarning(Object)
FFmpegOut.CameraCapture:ClosePipe() (at Assets/FFmpegOut/CameraCapture.cs:177)
FFmpegOut.CameraCapture:Update() (at Assets/FFmpegOut/CameraCapture.cs:83)

ffmpeg returned with a warning or an error message. See the following lines for details:
[libx264 @ 000000000071cb40] frame MB size (320x180) > level limit (36864)
[libx264 @ 000000000071cb40] DPB size (4 frames, 230400 mbs) > level limit (3 frames, 184320 mbs)

UnityEngine.Debug:LogWarning(Object)
FFmpegOut.CameraCapture:ClosePipe() (at Assets/FFmpegOut/CameraCapture.cs:177)
FFmpegOut.CameraCapture:Update() (at Assets/FFmpegOut/CameraCapture.cs:83)

Hi, I have multiple camera scene + UI screen space canvas
I was wondering if I can use this awesome tool to capture the screen and not a specific camera

Maybe to use new thing:
https://docs.unity3d.com/ScriptReference/ScreenCapture.CaptureScreenshotIntoRenderTexture.html

can I get your advice on this? (is it possible for you can show us how to make the adjustment?)
Thanks!

Hello,

First of all thank you for your FFmpeg integration plugin, it's been really helpful to handle in app video encoding and is quite easy to use. I successfully managed to integrate it in my application (on both OSX and Windows). I'm only struggling after code signing my Application on OSX.

I suspect the issue is linked to the App SandBoxing made mandatory by Apple on all Mac App store App.

com.apple.security.app-sandbox

The plugin works well when codesigning without Sandboxing. I tried multiple keys in entitlement including a solution using an app-sandbox inheritance but without any success.

com.apple.security.inherit

Details on my codesign process and my entitlement configuration can be found on a stack overflow post i made.

Do you have any clues or hint on the matter ? Do i need a specific configuration on my entitlements file or my FFmpeg file structure ?

Thanks again for your plugin and your time,
Regards,
Alexis.