linux-kernel - Re: [PATCH] v4l2-ctrl: add control for thumnails

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <de40cdf1-0b84-a252-0137-a550f078d28b@linaro.org>
Date:   Tue, 13 Oct 2020 15:52:11 +0300
From:   Stanimir Varbanov <stanimir.varbanov@...aro.org>
To:     Tomasz Figa <tfiga@...omium.org>,
        Hans Verkuil <hverkuil-cisco@...all.nl>
Cc:     Stanimir Varbanov <stanimir.varbanov@...aro.org>,
        Linux Media Mailing List <linux-media@...r.kernel.org>,
        Linux Kernel Mailing List <linux-kernel@...r.kernel.org>,
        Philipp Zabel <p.zabel@...gutronix.de>,
        Ezequiel Garcia <ezequiel@...labora.com>,
        Paul Kocialkowski <paul.kocialkowski@...tlin.com>,
        Nicolas Dufresne <nicolas@...fresne.ca>
Subject: Re: [PATCH] v4l2-ctrl: add control for thumnails

Hi,

On 6/4/20 3:57 PM, Tomasz Figa wrote:
> On Thu, Jun 4, 2020 at 2:56 PM Hans Verkuil <hverkuil-cisco@...all.nl> wrote:
>>
>> On 04/06/2020 14:34, Stanimir Varbanov wrote:
>>> Hi Hans,
>>>
>>> On 6/4/20 12:08 PM, Hans Verkuil wrote:
>>>> On 04/06/2020 11:02, Stanimir Varbanov wrote:
>>>>> Hi Hans,
>>>>>
>>>>> On 5/27/20 12:53 AM, Stanimir Varbanov wrote:
>>>>>> Hi Hans,
>>>>>>
>>>>>> On 5/26/20 3:04 PM, Hans Verkuil wrote:
>>>>>>> On 26/05/2020 10:54, Stanimir Varbanov wrote:
>>>>>>>> Add v4l2 control for decoder thumbnail.
>>>>>>>>
>>>>>>>> Signed-off-by: Stanimir Varbanov <stanimir.varbanov@...aro.org>
>>>>>>>> ---
>>>>>>>>  Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst | 7 +++++++
>>>>>>>>  drivers/media/v4l2-core/v4l2-ctrls.c                      | 2 ++
>>>>>>>>  include/uapi/linux/v4l2-controls.h                        | 2 ++
>>>>>>>>  3 files changed, 11 insertions(+)
>>>>>>>>
>>>>>>>> diff --git a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
>>>>>>>> index d0d506a444b1..e838e410651b 100644
>>>>>>>> --- a/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
>>>>>>>> +++ b/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst
>>>>>>>> @@ -3726,6 +3726,13 @@ enum v4l2_mpeg_video_hevc_size_of_length_field -
>>>>>>>>      disables generating SPS and PPS at every IDR. Setting it to one enables
>>>>>>>>      generating SPS and PPS at every IDR.
>>>>>>>>
>>>>>>>> +``V4L2_CID_MPEG_VIDEO_DECODER_THUMBNAIL (button)``
>>>>>>>> +    Instructs the decoder to produce immediate output. The decoder should
>>>>>>>> +    consume first input buffer for progressive stream (or first two buffers
>>>>>>>> +    for interlace). Decoder should not allocate more output buffers that it
>>>>>>>> +    is required to consume one input frame. Usually the decoder input
>>>>>>>> +    buffers will contain only I/IDR frames but it is not mandatory.
>>>>>>>
>>>>>>> This is very vague. It doesn't explain why the control is called 'THUMBNAIL',
>>>>>>> but more importantly it doesn't explain how this relates to normal decoding.
>>>>>>
>>>>>> If in the normal decode the capture queue buffers are 5, in the
>>>>>> thumbnail mode the number of buffers will be only 1 (if the bitstream is
>>>>>> progressive) and this will guarantee low memory usage. The other
>>>>>> difference is that the decoder will produce decoded frames (without
>>>>>> errors) only for I/IDR (sync frames).
>>>>
>>>> Isn't this really a "DECODE_SYNC_FRAMES_ONLY" control? That's what it does,
>>>> right? Skip any B/P frames and only decode sync frames.
>>>
>>> Yes, it is.
>>> To me V4L2_CID_MPEG_VIDEO_DECODE_SYNC_FRAMES sounds better. If you are
>>> fine I can send a new patch.
>>>
>>> The definition of "sync frames" is a bit difficult for codec-agnostic
>>> controls. Is it sound better "INTRA", DECODE_INTRA_FRAMES (ONLY)?
>>
>> INTRA is better. DECODE_INTRA_FRAMES_ONLY is a good name, I think.
>>
>> Thumbnail creation can be given as an example in the description of the
>> control, but that's just a use-case.
> 
> How about the other aspect of the behavior?
> 
> "Decoder should not allocate more output buffers that it
> is required to consume one input frame."
> 

In fact I have to refine this; It looks like that picture type decode vs
thumbnail are two different modes.

Thumbnail mode - first frame decode without additional memory (one input
buffer and one output buffer). The first frame can be even non-intra
frame as well. Also no matter frame parser is sending, the decoder will
try to produce output for thumbnail generation.

> Best regards,
> Tomasz
> 
>>
>> Regards,
>>
>>         Hans
>>
>>>
>>>>
>>>> That this is useful for creating thumbnails is just a specific use-case.
>>>>
>>>> Regards,
>>>>
>>>>      Hans
>>>>
>>>>>>
>>>>>>>
>>>>>>> I.e. if you are decoding and 'press' this control, what happens then?
>>>>>>
>>>>>> Might be the button type wasn't great idea. In fact the control should
>>>>>> be set before streamon so that the driver returns min_capture_bufs 1.
>>>>>>
>>>>>>>
>>>>>>> What exactly is the use-case?
>>>>>>
>>>>>> It could be used to generate thumbnails of all video clips in a folder
>>>>>> or when you open a Gallery application on your phone.
>>>>>>
>>>>>
>>>>> What is your opinion on that control? I could consider to make it Venus
>>>>> custom control but from the use-case it looks other drivers also can
>>>>> benefit of it.
>>>>>
>>>>> I tried to make more generic one [1] but it looks it will be too difficult.
>>>>>
>>>>
>>>
>>

-- 
regards,
Stan