linux-kernel - Re: [PATCH] media: v4l-ctrl: add control for variable frame rate

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Date:   Sat, 29 Jun 2019 10:54:35 +0200
From:   Paul Kocialkowski <paul.kocialkowski@...tlin.com>
To:     Fish Lin <linfish@...gle.com>
Cc:     Hans Verkuil <hverkuil-cisco@...all.nl>,
        Mauro Carvalho Chehab <mchehab@...nel.org>,
        Philipp Zabel <p.zabel@...gutronix.de>,
        Keiichi Watanabe <keiichiw@...omium.org>,
        Will Deacon <will.deacon@....com>,
        Sakari Ailus <sakari.ailus@...ux.intel.com>,
        Tomasz Figa <tfiga@...omium.org>,
        Smitha T Murthy <smitha.t@...sung.com>,
        linux-media@...r.kernel.org, linux-kernel@...r.kernel.org,
        Maxime Ripard <maxime.ripard@...tlin.com>,
        Nicolas Dufresne <nicolas@...fresne.ca>,
        Boris Brezillon <boris.brezillon@...labora.com>,
        Ezequiel Garcia <ezequiel@...labora.com>,
        Jonas Karlman <jonas@...boo.se>,
        Jernej Škrabec <jernej.skrabec@...l.net>
Subject: Re: [PATCH] media: v4l-ctrl: add control for variable frame rate

Hi,

On Fri 29 Mar 19, 10:46, Fish Lin wrote:
> The idea for variable framerate is to reduce bitrate by encode less frames.
> Sometimes the motion is too much and encoder can encode in really low
> bitrate, will be able to dynamic drop some frames to archive
> requirement.

This looks like an interesting optimization to support!

> Codec like H.264 and HEVC can support VFR, and lots of containers also support.
> https://en.wikipedia.org/wiki/Comparison_of_video_container_formats
> 
> Yes, the decoder need to know if video is CFR or VFR, not all decoder
> support VFR, especially for video editing.
> There are also some tools to convert VFR video to CFR video.

Do you have some idea of what using this feature requires in terms of bitstream
elements? We are currently working on the H.264 stateless decoding spec for
hardware decoders and it would be useful to have a somewhat precise idea of how
we could fit support for this feature into it.

Cheers,

Paul

> Some reference:
> https://en.wikipedia.org/wiki/Variable_frame_rate
> https://www.bandicam.com/support/tips/vfr-cfr/
> https://pro-av.panasonic.net/manual/html/AJ-PX270P_PJ_PX_EN_AN(VQT5J83A-8(E))/chapter04_08.htm
> 
> Hans Verkuil <hverkuil-cisco@...all.nl> 於 2019年3月28日 週四 下午10:57寫道：
> >
> > On 3/20/19 4:10 AM, Fish Lin wrote:
> > > These flag are added
> > > * V4L2_CID_MPEG_VIDEO_VFR_ENABLE
> > > * V4L2_CID_MPEG_VIDEO_VFR_MIN_FRAMERATE
> > > One is to set variable frame rate enable or not, another is to control
> > > minimal frame rate that video encoder should keep per second.
> > >
> > > Signed-off-by: Fish Lin <linfish@...gle.com>
> > > ---
> > >  Documentation/media/uapi/v4l/ext-ctrls-codec.rst | 9 +++++++++
> > >  drivers/media/v4l2-core/v4l2-ctrls.c             | 3 +++
> > >  include/uapi/linux/v4l2-controls.h               | 3 +++
> > >  3 files changed, 15 insertions(+)
> > >
> > > diff --git a/Documentation/media/uapi/v4l/ext-ctrls-codec.rst b/Documentation/media/uapi/v4l/ext-ctrls-codec.rst
> > > index c97fb7923be5..4bc013d71b67 100644
> > > --- a/Documentation/media/uapi/v4l/ext-ctrls-codec.rst
> > > +++ b/Documentation/media/uapi/v4l/ext-ctrls-codec.rst
> > > @@ -620,7 +620,16 @@ enum v4l2_mpeg_video_bitrate_mode -
> > >      * - Bit 24:31
> > >        - Must be zero.
> > >
> > > +.. _v4l2-mpeg-video-variable-framerate:
> > >
> > > +``V4L2_CID_MPEG_VIDEO_VFR_ENABLE (boolean)``
> > > +    Set variable framerate enable or not. When enabled, video encoder is able
> > > +    to drop frames based on some mechanism, like frame similarity or bitrate
> > > +    control (default disable).
> > > +
> > > +``V4L2_CID_MPEG_VIDEO_VFR_MIN_FRAMERATE (integer)``
> > > +    Minimal kept frame per second when variable framerate is enabled (default
> > > +    is input framerate).
> >
> > You don't really have a concept of frames per second when encoding. What you
> > are probably trying to say is something like: "don't drop more than one frame for
> > every X frames" or something like that.
> >
> > Which codecs support this? A decoder needs to know that a frame is missing, right?
> >
> > What's the use-case for this?
> >
> > Sorry, just trying to understand this better.
> >
> > Regards,
> >
> >         Hans
> >
> > >
> > >  .. _v4l2-mpeg-video-dec-pts:
> > >
> > > diff --git a/drivers/media/v4l2-core/v4l2-ctrls.c b/drivers/media/v4l2-core/v4l2-ctrls.c
> > > index b79d3bbd8350..566c6552c9a0 100644
> > > --- a/drivers/media/v4l2-core/v4l2-ctrls.c
> > > +++ b/drivers/media/v4l2-core/v4l2-ctrls.c
> > > @@ -786,6 +786,8 @@ const char *v4l2_ctrl_get_name(u32 id)
> > >       case V4L2_CID_MPEG_VIDEO_MB_RC_ENABLE:                  return "H264 MB Level Rate Control";
> > >       case V4L2_CID_MPEG_VIDEO_HEADER_MODE:                   return "Sequence Header Mode";
> > >       case V4L2_CID_MPEG_VIDEO_MAX_REF_PIC:                   return "Max Number of Reference Pics";
> > > +     case V4L2_CID_MPEG_VIDEO_VFR_ENABLE:                    return "Variable Frame Rate Enable";
> > > +     case V4L2_CID_MPEG_VIDEO_VFR_MIN_FRAMERATE:             return "VFR Minimal Frame Rate";
> > >       case V4L2_CID_MPEG_VIDEO_H263_I_FRAME_QP:               return "H263 I-Frame QP Value";
> > >       case V4L2_CID_MPEG_VIDEO_H263_P_FRAME_QP:               return "H263 P-Frame QP Value";
> > >       case V4L2_CID_MPEG_VIDEO_H263_B_FRAME_QP:               return "H263 B-Frame QP Value";
> > > @@ -1108,6 +1110,7 @@ void v4l2_ctrl_fill(u32 id, const char **name, enum v4l2_ctrl_type *type,
> > >       case V4L2_CID_MPEG_VIDEO_DECODER_SLICE_INTERFACE:
> > >       case V4L2_CID_MPEG_VIDEO_FRAME_RC_ENABLE:
> > >       case V4L2_CID_MPEG_VIDEO_MB_RC_ENABLE:
> > > +     case V4L2_CID_MPEG_VIDEO_VFR_ENABLE:
> > >       case V4L2_CID_MPEG_VIDEO_H264_8X8_TRANSFORM:
> > >       case V4L2_CID_MPEG_VIDEO_H264_VUI_SAR_ENABLE:
> > >       case V4L2_CID_MPEG_VIDEO_MPEG4_QPEL:
> > > diff --git a/include/uapi/linux/v4l2-controls.h b/include/uapi/linux/v4l2-controls.h
> > > index 06479f2fb3ae..f1bf52eb0152 100644
> > > --- a/include/uapi/linux/v4l2-controls.h
> > > +++ b/include/uapi/linux/v4l2-controls.h
> > > @@ -404,6 +404,9 @@ enum v4l2_mpeg_video_multi_slice_mode {
> > >  #define V4L2_CID_MPEG_VIDEO_MV_V_SEARCH_RANGE                (V4L2_CID_MPEG_BASE+228)
> > >  #define V4L2_CID_MPEG_VIDEO_FORCE_KEY_FRAME          (V4L2_CID_MPEG_BASE+229)
> > >
> > > +#define V4L2_CID_MPEG_VIDEO_VFR_ENABLE (V4L2_CID_MPEG_BASE + 250)
> > > +#define V4L2_CID_MPEG_VIDEO_VFR_MIN_FRAMERATE (V4L2_CID_MPEG_BASE + 251)
> > > +
> > >  #define V4L2_CID_MPEG_VIDEO_H263_I_FRAME_QP          (V4L2_CID_MPEG_BASE+300)
> > >  #define V4L2_CID_MPEG_VIDEO_H263_P_FRAME_QP          (V4L2_CID_MPEG_BASE+301)
> > >  #define V4L2_CID_MPEG_VIDEO_H263_B_FRAME_QP          (V4L2_CID_MPEG_BASE+302)
> > >
> >

-- 
Paul Kocialkowski, Bootlin
Embedded Linux and kernel engineering
https://bootlin.com