FPS drop and lags during streaming about streampack HOT 20 CLOSED

Hi!
The first thing I can think of is that your CPU is overload. What is your device and Android version?
Do you have the same issue with the example application?
Do you have the logcat to see if anything wrong?

The preview has nothing to do with the live stream. Your camera device send frames to the preview and to the encoder. Changing something on the preview should not change anything on the live stream.

from streampack.

Thank you for the answer! I have Redmi Note 8 pro:
Android version - 11
MIUI - 12.5.3
CPU - MediaTek Helio G90T
RAM - 8GB

Yes, with your example application has the same issue.
Logs attached, also a screenshot Profiler. The application was launched with this SurfaceHolder.Callback::

    private val surfaceViewCallback = object : SurfaceHolder.Callback {
        var nbOnSurfaceChange = 0

        override fun surfaceCreated(holder: SurfaceHolder) {
            nbOnSurfaceChange = 0
        }

        override fun surfaceChanged(holder: SurfaceHolder, format: Int, width: Int, height: Int) {
            require(context != null)

            try {
                nbOnSurfaceChange++
                if (nbOnSurfaceChange == 2) {
                    viewModel.startPreview(holder.surface)

                } else {

                    val choices = context!!.getCameraOutputSizes(
                        SurfaceHolder::class.java,
                        viewModel.cameraId
                    )

                    chooseBigEnoughSize(choices, width, height)?.let { size ->
                        holder.setFixedSize(size.width, size.height)
                    }
                }
            } catch (e: Exception) {
                Log.e(TAG, "!!!Can't start preview due to ${e.message}!!!", e)
            }
        }

        override fun surfaceDestroyed(holder: SurfaceHolder) {
            viewModel.stopPreview()
            binding.surfaceView.holder.removeCallback(this)
        }
    }

LogsAfterStartingStream.txt
LogsAfterStoppingStream.txt

from streampack.

Hi,

Ok, everything seems alright. Your CPU is really low which is really good.
Anyway, I found this: https://stackoverflow.com/questions/57485050/how-to-increase-frame-rate-with-android-camerax-imageanalysis/57490987#57490987. It is for CameraX but the same settings apply to camera2.
As I only test on Nexus and Pixel, I might have miss something on other constructors. I want to be sure what setting really work on your phone.
It might take a while to test. Can you test on the sample of https://github.com/ThibaultBee/StreamPack/tree/bugfix/framerate?

from streampack.

I tested your example, fps is still low, but another problem has been added =).
The camera capture has become dark, although the room is light, there is also a problem with the aspect ratio, but it was there before too. I have attached screenshots.

I didn't change the video settings. They stood by default:
Encoder - AVC/H.264
Resolution - 1280x720
Framerate - 30
Video start bitrate - 2000

LogsAfterStartingStream.txt
LogsAfterStoppingStream.txt

Is there anything else I can be useful?

from streampack.

Ok.
The dark effect is because AE is disabled.
Anyway, I have changed https://github.com/ThibaultBee/StreamPack/tree/bugfix/framerate.
Can you retest it?

from streampack.

Hi Thibault!

Dark effect is gone, CPU does not overload, problems with fps remain, but I noticed that when switching to the front camera, fps does not drop.

I also wanted to clarify with you why call holder.setFixedSize () in surfaceChanged? Considering what you wrote: "The preview has nothing to do with the live stream. Your camera device send frames to the preview and to the encoder. Changing something on the preview should not change anything on the live stream". If it is needed for the correct aspect ratio of the surfaceView, then this doesn't work. I found this example: https://github.com/android/camera-samples/blob/main/Camera2Video/utils/src/main/java/com/example/android/camera/utils/AutoFitSurfaceView.kt on google and it works well with aspect ratio. I don't have much development experience, so any advice from your side would be useful to me =).

LogsAfterStartingApp.txt
LogsAfterStartingStream.txt
LogsAfterStoppingStream.txt

from streampack.

The code in surfaceChanged comes from the former Android camera sample.
I replace this dirty mechanism by an AutoFitSurfaceView. Can you test again the sample of https://github.com/ThibaultBee/StreamPack/tree/bugfix/framerate?

from streampack.

Now there are no problems with fps, at least in SurfaceView, but the stream does not play. The same situation occurred when I called holder.setFixedSize () only in surfaceCreated (), which I wrote about earlier. As our backend developer explained to me, this happens because the fps information does not reach the server, which breaks the transcoding and leads to errors.

LogsAfterStartingApp.txt
LogsAfterStartingStream.txt
LogsAfterStoppingStream.txt

from streampack.

Now there are no problems with fps, at least in SurfaceView, but the stream does not play. The same situation occurred when I called holder.setFixedSize () only in surfaceCreated ()

Glad to hear. It's a good start :) but I don't understand why :(

As our backend developer explained to me, this happens because the fps information does not reach the server, which breaks the transcoding and leads to errors.

Which fps information are you talking about? Have you try with ffplay (ffplay -fflags nobuffer srt://0.0.0.0:9998?mode=listener)?

from streampack.

Which fps information are you talking about?

For example, if start stream using your library, the following information comes to the server:

• Stream #0:0[0x100]: Video: hevc (Main) (HEVC / 0x43564548), yuv420p(tv), 720x1280, 30 tbr, 90k tbn, 90k tbc
• Stream #0:1[0x101]: Audio: aac (LC) ([15][0][0][0] / 0x000F), 48000 Hz, stereo, fltp, 130 kb/s

But the following information should come:

• Stream #0:0[0x100]: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p(tv, progressive), 1920x1080 [SAR 1:1 DAR 16:9], 50 fps, 50 tbr, 90k tbn, 100 tbc
• Stream #0:1[0x101]: Audio: aac (LC) ([15][0][0][0] / 0x000F), 48000 Hz, stereo, fltp, 122 kb/s

Have you try with ffplay (ffplay -fflags nobuffer srt://0.0.0.0:9998?mode=listener)?

Unfortunately not yet, but I want to try. As soon as I do this, I will inform you of the results :).

from streampack.

Oh, I guess your server expects the VUI in the SPS (especially the timing_info).
In H.264, VUI is not a mandatory field. vui_parameters_present_flag indicates if VUI is present in the SPS.
Both SPS and PPS are generated by the Android encoder API (aka MediaCodec) and MediaCodec chooses to not add VUI in the SPS.
You should consider to support incoming stream that does not have VUI.

from streampack.

Hi @BakhromMuydinov,
Have you fixed your issue?

By the way, I added AutoFitSurfaceView in the SDK instead of the sample.

from streampack.

Hi @ThibaultBee! I apologize for not getting in touch for a long time.

Yes, we fixed it, thank you very much!
But there is still a problem with the correct aspect ratio of the received stream. I also launched a stream on ffplay, there is the same problem.

What can be done about it?

Stream #0:0[0x20]: Video: hevc (Main) ([36][0][0][0] / 0x0024), yuv420p(tv, bt470bg/bt470bg/smpte170m), 720x1280, 29.58 tbr, 90k tbn
Stream #0:1[0x21]: Audio: aac (LC) ([15][0][0][0] / 0x000F), 44100 Hz, stereo, fltp, 130 kb/s

from streampack.

Hi,

Glad you fixed your issue!
You are using StreamPack in portrait. Did you notice the same effect in landscape?

Regards,
Thibault

from streampack.

Hi!

The situation is better in landscape orientation, but still this effect, in my opinion, is slightly present.

Stream #0:0[0x20]: Video: hevc (Main) ([36][0][0][0] / 0x0024), yuv420p(tv, bt470bg/bt470bg/smpte170m), 1280x720, 29.58 tbr, 90k tbn
Stream #0:1[0x21]: Audio: aac (LC) ([15][0][0][0] / 0x000F), 44100 Hz, stereo, fltp, 131 kb/s

from streampack.

Hi!
Sorry for the delay, I have been really busy, + covid,...
Landscape seems alright on my side if you want to be sure, you should film a square and check if it is still a square on ffplay display.
Also, there is an issue in portrait but I try to fix it. Can you test https://github.com/ThibaultBee/StreamPack/tree/bugfix/portrait ?

Regards,
Thibault

from streampack.

Hi!
Ohh.. I wish you a speedy recovery!!
Now everything looks correct in portrait, landscape, and square =).

Thank you very much!!

from streampack.

Hello,

I merged the branch.
I am waiting for a last development before releasing 1.4.0. Are you okay to wait few days?

Also, could we have a chat on what you are going to do with this library, the features you are using, not using or missing? I am pretty sure you can find me on linkedin with my name.

Regards,
Thibaultt

from streampack.

I am waiting for a last development before releasing 1.4.0. Are you okay to wait few days?

No problem! I'm not in a hurry =).

Unfortunately, I'm not on LinkedIn, but if you want, we can chat on Telegram.

from streampack.

Hi,

These issues are fixed. Fixes will be shipped with next release.

Best regards,
Thibault :)

from streampack.