Preview video is delayed and interlaced about v4l2test HOT 32 CLOSED

your input video signal is CVBS ? using a video decoder convert to ccir656 yuv422, right?

if cvbs, it means the video is odd/even interlaced video. the v4l2test treat every frame is progressive. so you need do deinterlace when preview on screen.

for the delayed issue, I think the video rendering spend much most time, the actual preview frame rate is slowed down.

check the code in camdev.cpp

#define DEF_WIN_PIX_FMT HAL_PIXEL_FORMAT_YCrCb_420_SP

the DEF_WIN_PIX_FMT is the pixel format used for android native window to rendering video.
for specific platform a specific pixel format may support hardware acceleration. mean hardware yuv rendering without color convert. so you need make sure which format is supported and the rendering speed is fast.

in function:

static void render_v4l2(CAMDEV *cam,

add ALOGD at the begin and at the end. and then using
logcat -v time
to measure the time rendering a frame need.
normally for 720x480 video, only 1ms - 5ms

on A33 platform the HAL_PIXEL_FORMAT_YCrCb_420_SP rendering format is support for hardware acceleration, and the camera data format is also HAL_PIXEL_FORMAT_YCrCb_420_SP. so there is no need to do any color convert and scale. a memcpy is used to copy data from camera to screen. so it is fast.

if not, below code will be executed:

    //++ do sws scale
    if (cam->win_w != dstw || cam->win_h != dsth) {
        if (cam->swsctxt) {
            sws_freeContext(cam->swsctxt);
        }
        cam->swsctxt = sws_getContext(srcw, srch, sws_src_fmt, dstw, dsth, sws_dst_fmt, SWS_POINT, 0, 0, 0);
        cam->win_w   = dstw;
        cam->win_h   = dsth;
    }
    sws_scale(cam->swsctxt, src_data, src_linesize, 0, srch, dst_data, dst_linesize);
    //-- do sws scale

it will take long time, and consume much cpu resource.

from v4l2test.

check the code in ffrecoder.cpp

static FFRECORDER_PARAMS DEF_FFRECORDER_PARAMS =
{
    // micdev input params
    44100,                      // mic_sample_rate
#ifdef USE_MICDEV_ANDROID
    1,                          // mic_channel_num
#else
    2,                          // mic_channel_num
#endif

    // camdev input params
    (char*)"/dev/video0",       // cam_dev_name_0
    0,                          // cam_sub_src_0
    1920,                       // cam_frame_width_0
    1080,                       // cam_frame_height_0
    25,                         // cam_frame_rate_0

change 1920x1080 to 720x480

from v4l2test.

Yes, my input video signal is CVBS and i'm using video decoder (isl79985 from Intersil) to convert to YUV422. I have enabled VDI (video de-interlaced) in Linux Kernel.
In camdev.cpp, i use "#define DEF_WIN_PIX_FMT HAL_PIXEL_FORMAT_YCrCb_420_SP".
In function "render_v4l2", this condition "if(sws_src_fmt == sws_dst_fmt && srcw == dstw && srch == dsth) is not true because sws_src_fmt != sws_dst_fmt, so it runs block code:
//++ do sws scale
if (cam->win_w != dstw || cam->win_h != dsth) {
if (cam->swsctxt) {
sws_freeContext(cam->swsctxt);
}
cam->swsctxt = sws_getContext(srcw, srch, sws_src_fmt, dstw, dsth, sws_dst_fmt, SWS_POINT, 0, 0, 0);
cam->win_w = dstw;
cam->win_h = dsth;
}
sws_scale(cam->swsctxt, src_data, src_linesize, 0, srch, dst_data, dst_linesize);
//-- do sws scale

So it renders video slowly.

In ffrecoder.cpp, i have changed 1920x1080 to 720x480 already.

Now my problem is chosen proper "#define DEF_WIN_PIX_FMT ..."?

from v4l2test.

as I know android native window dose not support YUV422 format.

on A33 HAL_PIXEL_FORMAT_YCrCb_420_SP is fast, but on you platform may not.
if on your platform HAL_PIXEL_FORMAT_YCrCb_420_SP is not hardware accelerated, it may using software algorithm to do conversion by android hal. it is slow.

try

    HAL_PIXEL_FORMAT_RGBA_8888          = 1,
    HAL_PIXEL_FORMAT_RGBX_8888          = 2,
    HAL_PIXEL_FORMAT_RGB_888            = 3,
    HAL_PIXEL_FORMAT_RGB_565            = 4,
    HAL_PIXEL_FORMAT_BGRA_8888          = 5,

from v4l2test.

With other format, how can i calculate dstlen?
// dst len
if (dstlen == -1) {
switch (dstfmt) {
case HAL_PIXEL_FORMAT_RGB_565: dstlen = dstw * dsth * 2; break;
case HAL_PIXEL_FORMAT_RGBX_8888: dstlen = dstw * dsth * 4; break;
case HAL_PIXEL_FORMAT_YV12: dstlen = dstw * dsth + dstw * dsth / 2; break;
case HAL_PIXEL_FORMAT_YCrCb_420_SP: dstlen = dstw * dsth + dstw * dsth / 2; break;
default: dstlen = 0; break;
}
}

from v4l2test.

dstlen is the android native windows buffer length, it is related to the pixel format, width and height.

from v4l2test.

Yes, and in "ffutils.h", how i can modify (add) new pixel format in this function:
inline int android_pixfmt_to_ffmpeg_pixfmt(int srcfmt)
{
// dst fmt
int dst_fmt = 0;
switch (srcfmt) {
case HAL_PIXEL_FORMAT_RGB_565: dst_fmt = AV_PIX_FMT_RGB565; break;
case HAL_PIXEL_FORMAT_RGBX_8888: dst_fmt = AV_PIX_FMT_BGR32; break;
case HAL_PIXEL_FORMAT_YV12: dst_fmt = AV_PIX_FMT_YUV420P; break;
case HAL_PIXEL_FORMAT_YCrCb_420_SP: dst_fmt = AV_PIX_FMT_NV12; break;
//Add...
case HAL_PIXEL_FORMAT_RGB_888: dst_fmt = ?; break;
case HAL_PIXEL_FORMAT_RGBA_8888: dst_fmt = ?; break;
...
}
return dst_fmt;
}

from v4l2test.

you can check the header file pixfmt.h in ffmpeg.

from v4l2test.

    if (sws_src_fmt == sws_dst_fmt && srcw == dstw && srch == dsth) {
//      ALOGD("===ck=== sws_src_fmt = 0x%0x, sws_dst_fmt = 0x%0x", sws_src_fmt, sws_dst_fmt);
//      memcpy(dstbuf, (uint8_t*)srcbuf + 0, (dstlen < srclen ? dstlen : srclen) - 0);
        memcpy(dstbuf, (uint8_t*)srcbuf + 1, (dstlen < srclen ? dstlen : srclen) - 1);
        return;
    }

need change to

    if (sws_src_fmt == sws_dst_fmt && srcw == dstw && srch == dsth) {
//      ALOGD("===ck=== sws_src_fmt = 0x%0x, sws_dst_fmt = 0x%0x", sws_src_fmt, sws_dst_fmt);
        memcpy(dstbuf, (uint8_t*)srcbuf + 0, (dstlen < srclen ? dstlen : srclen) - 0);
//      memcpy(dstbuf, (uint8_t*)srcbuf + 1, (dstlen < srclen ? dstlen : srclen) - 1);
        return;
    }

from v4l2test.

What is cam pixel format did you use? In render_v4l2 call 2 function:
// src fmt
AVPixelFormat sws_src_fmt = (AVPixelFormat)v4l2dev_pixfmt_to_ffmpeg_pixfmt(srcfmt);
// dst fmt
AVPixelFormat sws_dst_fmt = (AVPixelFormat)android_pixfmt_to_ffmpeg_pixfmt(dstfmt);

and only use cam->cam_pixfmt = V4L2_PIX_FMT_NV12;
then sws_src_fmt = sws_dst_fmt = AV_PIX_FMT_NV12; and this block code run:
// memcpy if same fmt and size
if (sws_src_fmt == sws_dst_fmt && srcw == dstw && srch == dsth) {
memcpy(dstbuf, (uint8_t*)srcbuf + 1, (dstlen < srclen ? dstlen : srclen) - 1);
return;
}

from v4l2test.

    if (0 == v4l2_try_fmt_size(cam->fd, V4L2_PIX_FMT_MJPEG, &w, &h)) {
        cam->cam_pixfmt = V4L2_PIX_FMT_MJPEG;
        cam->cam_w      = w;
        cam->cam_h      = h;
    }
    else if (0 == v4l2_try_fmt_size(cam->fd, V4L2_PIX_FMT_NV12, &w, &h)) {
//      ALOGD("===ck=== V4L2_PIX_FMT_NV12");
        cam->cam_pixfmt = V4L2_PIX_FMT_NV12;
        cam->cam_w      = w;
        cam->cam_h      = h;
    }
    else if (0 == v4l2_try_fmt_size(cam->fd, V4L2_PIX_FMT_NV21, &w, &h)) {
//      ALOGD("===ck=== V4L2_PIX_FMT_NV21");
        cam->cam_pixfmt = V4L2_PIX_FMT_NV21;
        cam->cam_w      = w;
        cam->cam_h      = h;
    }
    else if (0 == v4l2_try_fmt_size(cam->fd, V4L2_PIX_FMT_YUYV, &w, &h)) {
//      ALOGD("===ck=== V4L2_PIX_FMT_YUYV");
        cam->cam_pixfmt = V4L2_PIX_FMT_YUYV;
        cam->cam_w      = w;
        cam->cam_h      = h;
    }

in camdev_init function, it will try to set camera pixel format.

on A33 platform the NV21 of camera equals to HAL_PIXEL_FORMAT_YCrCb_420_SP android native window pixel format.

from v4l2test.

On my video decoder, it supports 2 output format: YUV422 or RGB565

from v4l2test.

try both video decoder and android native window use RGB565 format

from v4l2test.

Yes, i try use cam pixel format cam->cam_pixfmt = V4L2_PIX_FMT_RGB565, it always "D/camdev: failed to de-queue buffer !" and nothing on screen

from v4l2test.

Hi CK,
i delete two lines, in Android.mk
-DENABLE_H264_HWENC
-DUSE_CEDARX_H264ENC
to record video using x264 software encoding. During recording, i see many warning logs: "video frame dropped by encoder !", after Stop record, i saw .mp4 file was created, but i can't play it on both android and computer.

Could you help me see how are them!
link video: https://www.mediafire.com/?4iugty1puyhpi49

Thanks!

from v4l2test.

it means the cpu could not encoding video in time, for you device, the video input is 720x480 30/60 fps. but the x264 software encoding is very slowly. so many video frame was dropped.

I can not download the file. how is the file size you recorded. even dropping frame happened, at least is could play. It can't be played back, it is a problem.

    // ffencoder output
    16000,                      // out_audio_bitrate_1
    AV_CH_LAYOUT_MONO,          // out_audio_chlayout_1
    16000,                      // out_audio_samprate_1
    5000000,                    // out_video_bitrate_1
    1280,                       // out_video_width_1
    720,                        // out_video_height_1
    25,                         // out_video_frate_1

try change the recording video size, frame rate and audio/video bitrate.

and how dose the camera preview issue now, did you fix the delayed and interlace issue ?

from v4l2test.

You can copy this link and paste to Browser, double click on it not ok.
https://www.mediafire.com/?4iugty1puyhpi49
the file size is about 10MB.

The delayed and interlace issue, i tried some ways to fix but not successful.
In "ffrecorder.cpp", if i use camdev input params (/dev/video0) 720,240, stream video has a litle latency, not "comb" effect, but can only capture to "720*240" .jpg file.
If i config to 720,480, video has more latency, and "comb" effect.

I also try to test render video on background screen using IPU or G2D, it's very good.

mxc_v4l2_tvin.txt

from v4l2test.

Hi CK,
i'm still stuck at recording video. I can't to play .mp4 file. When recording, i saw android log:
"W/audio_hw_primary: out_set_parameters routing=2, ret 0, out -1242765568"
"W/audio_hw_primary: no standby"
and many:
"W/AudioFlinger: RecordThread: buffer overflow"

I tested record audio by another recorder app and it works well.

Can i record video without record audio in my DVR project?

And this is log when app save .mp4 file:

V/MediaSaver: Current video URI: content://media/external/video/media/144
I/FslInspector: TryMp4Type SUCCESS
D/FslExtractor: FslExtractor::FslExtractor mime=video/mp4
D/FslExtractor: FslExtractor::Init BEGIN
D/FslExtractor: GetLibraryName lib_mp4_parser_arm11_elinux.3.0.so
D/FslExtractor: load parser name lib_mp4_parser_arm11_elinux.3.0.so
D/FslExtractor: FslExtractor::CreateParserInterface success
I/FslExtractor: Core parser MPEG4PARSER_06.09.31 build on Apr 8 2016 15:52:36
D/FslExtractor: createParser2 flag=1c,err=-120
E/FslExtractor: fail to create the parser: -120
D/FslExtractor: FslExtractor::Init ret=-2147483648
E/MetadataRetrieverClient: failed to capture a video frame
596-700/android.process.media E/MediaMetadataRetrieverJNI: getFrameAtTime: videoFrame is a NULL pointer
D/FslExtractor: FslExtractor::~FslExtractor
596-700/android.process.media W/MediaThumbRequest: Can't create mini thumbnail for /storage/emulated/0/DCIM/DVR_Video/A_VID_19700101_025625.mp4

from v4l2test.

how to without record audio ?

recorder->audio_source[0] = 0;
recorder->audio_source[1] = 0;
recorder->audio_source[2] =-1;

-->

recorder->audio_source[0] =-1;
recorder->audio_source[1] =-1;
recorder->audio_source[2] =-1;

from v4l2test.

or call setAudioSource method:

setAudioSource(0, -1);
setAudioSource(1, -1);

from v4l2test.

I tried record without audio but still error, i think caused by android.
(D/FslExtractor: createParser2 flag=1c,err=-120
E/FslExtractor: fail to create the parser: -120
D/FslExtractor: FslExtractor::Init ret=-2147483648
E/MetadataRetrieverClient: failed to capture a video frame)

I tried run "recordertest" then it can records video (w/o audio) and save to .mp4 file, and i can playback them, if record with audio, Error! maybe it failed by tinyalsa.

Can you point me how to stop record when running "recordertest"?
i see in code "recordertest.cpp" using "property_get("sys.ffrecorder.test.exit", exit, "0") but i don't know how to use, so i use "Ctrl-C" to stop record.

from v4l2test.

adb shell
try command
setprop sys.ffrecorder.test.exit 1

from v4l2test.

when record video into mp4 file using ffmpeg, these is a known issue, if

av_write_trailer(encoder->ofctxt);

was not be executed. the mp4 file would not be playback, even if video and audio were encoded, muxed and stored into mp4 file.

so if mp4 file unable to playback, maybe the av_write_trailer not be executed.

you'd better add logs and debug on these.

from v4l2test.

You are alright! av_write_trailer not be executed.
I press Stop record on android app, these code in "ffrecorder.cpp" can be reached:

ALOGD("Recorder enclosed...\n");
recorder->enclose[encidx] = recorder->encoder[encidx];
recorder->encoder[encidx] = NULL;
recorder->state &=~(FRF_RECORDING);

but this code in "ffencorder.cpp" not be executed:
void ffencoder_free(void *ctxt)
{
//................

/* write the trailer, if any. The trailer must be written before you
* close the CodecContexts open when you wrote the header; otherwise
* av_write_trailer() may try to use memory that was freed on
* av_codec_close(). */
ALOGD("av_write_trailer starting...\n");
av_write_trailer(encoder->ofctxt);
ALOGD("av_write_trailer DONE...\n");
.....
}

from v4l2test.

it works fine when i debugging on allwinner a33 soc.
so you'd fix your issue by yourself.

from v4l2test.

I fixed it!
Thank you!

from v4l2test.

what is the problem.
how did you fixed it ?

from v4l2test.

I added this code to "ffrecorder_record_stop" function:
void ffrecorder_record_stop(void *ctxt, int encidx)
{
...........
//Added
for (int i=0; i<MAX_ENCODER_NUM; i++) {
ffencoder_free(recorder->enclose[i]);
}
}

Now my issue is audio record enabled but when playback it's silence!

from v4l2test.

Hello CK,
i have issue with capture image in 720x480 size. Now, i can only encode to JPEG in 720x240, so it looks like stretch.
And now i want to SCALE (or Resizing) image during capture (when encode to JPEG) from 720x240 to 720x480, how can i modify code in function "ffjpeg_encode_thread_proc" (ffjpeg.c file)?
I tried to use sws_scale before "avcodec_encode_video2" but not successfull.

Please help me!

Thank you!

from v4l2test.

did you pass correct size to function ffjpeg_encoder_encode ?
you should pass 720 and 480

from v4l2test.

Hi Ck,
i modify a little in "ffjpeg_encoder_encode" function (assign h = 480), and now i can capture with 720x480!
I have one more question: can we adjust compression ratio when encode to JPEG?

Thanks!

from v4l2test.

sorry I also don't know how to adjust the compression ratio.

if you find the way, please tell me.

from v4l2test.