Message-ID: <fda6a365-5fda-503a-0c33-306deac8193d@quicinc.com>
Date: Fri, 15 Mar 2024 19:21:03 +0530
From: Vikash Garodia <quic_vgarodia@...cinc.com>
To: Hans Verkuil <hverkuil@...all.nl>, Hans Verkuil <hans.verkuil@...co.com>,
        <mchehab@...nel.org>, <stanimir.k.varbanov@...il.com>,
        <andersson@...nel.org>, <konrad.dybcio@...aro.org>,
        <bryan.odonoghue@...aro.org>, <agross@...nel.org>,
        <linux-media@...r.kernel.org>, <linux-kernel@...r.kernel.org>
CC: <linux-arm-msm@...r.kernel.org>, <quic_abhinavk@...cinc.com>,
        "Dikshita
 Agarwal" <quic_dikshita@...cinc.com>
Subject: Re: [PATCH v2 00/34] Qualcomm video encoder and decoder driver

Hi Hans,

On 3/12/2024 4:07 PM, Hans Verkuil wrote:
> Hi Vikash,
> 
> On 2/29/24 16:09, Vikash Garodia wrote:
>> Hello All,
>>
>> On 12/18/2023 5:01 PM, Dikshita Agarwal wrote:
>>> This patch series introduces support for Qualcomm new video acceleration
>>> hardware architecture, used for video stream decoding/encoding. This driver
>>> is based on new communication protocol between video hardware and application
>>> processor.
>>> This driver comes with below capabilities:
>>> - V4L2 complaint video driver with M2M and STREAMING capability.
>>> - Supports H264, H265, VP9 decoders.
>>> - Supports H264, H265 encoders.
>>> This driver comes with below features:
>>> - Centralized resource and memory management.
>>> - Centralized management of core and instance states.
>>> - Defines platform specific capabilities and features. As a results, it provides
>>>   a single point of control to enable/disable a given feature depending on 
>>>   specific platform capabilities.
>>> - Handles hardware interdependent configurations. For a given configuration from
>>>   client, the driver checks for hardware dependent configuration/s and updates
>>>   the same.
>>> - Handles multiple complex video scenarios involving state transitions - DRC,
>>>   Drain, Seek, back to back DRC, DRC during Drain sequence, DRC during Seek
>>>   sequence.
>>> - Introduces a flexible way for driver to subscribe for any property with
>>>   hardware. Hardware would inform driver with those subscribed property with any
>>>   change in value.
>>> - Introduces performance (clock and bus) model based on new hardware
>>>   architecture.
>>> - Introduces multi thread safe design to handle communication between client and
>>>   hardware.
>>> - Adapts encoder quality improvements available in new hardware architecture.
>>> - Implements asynchronous communication with hardware to achieve better
>>>   experience in low latency usecases.
>>> - Supports multi stage hardware architecture for encode/decode.
>>> - Output and capture planes are controlled independently. Thereby providing a
>>>   way to reconfigure individual plane.
>>> - Hardware packetization layer supports synchronization between configuration
>>>   packet and data packet.
>>> - Introduces a flexibility to receive a hardware response for a given command
>>>   packet.
>>> - Native hardware support of LAST flag which is mandatory to align with port
>>>   reconfiguration and DRAIN sequence as per V4L guidelines.
>>> - Native hardware support for drain sequence.
>>
>> 1. We considered enhancing the venus driver to support iris functionality but
>> concluded that the result would essentially be two parallel drivers implemented
>> in the same folder.
>> 2. Although the underlying hardware for venus and iris are quite similar, but
>> there are other factors which brings the need of new driver:
>>    a. the host firmware interface (HFI) changed between the two. Venus supports
>> HFI protocol 1.0 while iris supports HFI protocol 2.0.
>>    b. unlike HFI protocol 1.0, HFI protocol 2.0 is better aligned to V4L2 APIs.
>>    c. iris driver modularize platforms, hardware variants, and states to extend
>> it for any upcoming SOC. Thereby more extendable to newer SOCs in future.
>>    d. Iris also supports many advanced features.
>> 3. Based on the comments received on posted iris driver [1], we evaluated it
>> further and to better align with the upstream standard and practices, we
>> acknowledge that even though iris driver is the way forward, it would be ideal
>> to bring in the Venus hardwares enabled in this driver.
>> 4. Hence, we decided to rework iris driver to add support of HFI protocol 1.0 as
>> well.
>> 5. Initially we would start with adding support for one HFI protocol 1.0 based
>> platform which would be SM8250.
>> 6. SM8250 enablement on iris driver would be incremental, starting with basic
>> decode for H264 codec.
>> 7. In long-term, all venus supported platforms would be migrated to iris.
>> 8. Iris driver and venus driver will co-exist till remaining supported targets
>> also gets migrated to iris driver.
>> 9. We would continue to support and maintain venus driver during this phased out
>> approach.
>>
>> Please share your thoughts on the above proposal.
> 
> I think this is a reasonable approach: the venus driver is quite old and it was
> created when we were at more-or-less the same time also developing better codec
> frameworks. So it is not quite up-to-date with all the latest requirements.
> 
> Starting with a clean slate for the iris driver and then add support for venus
> platforms to the iris driver makes sense.
Appreciate your time to review the proposal.
> 
> Just one serious concern: who will do the venus platform migration? Are there resources
> available to do that work? Or is this just wishful thinking?
To initiate the venus platform migration, iris driver would be enabled with all
SOCs under HFI_V6XX which are currently supported in venus driver. In doing so,
iris driver, posted earlier [1] is being reworked to have cleaner abstraction to
invoke HFI 1.0 protocol (venus variants) and would be applicable for all venus
platforms. With this, remaining platform migration would be done with some
incremental efforts (platform specific handling).
We would like to get help from community and co-work in enabling the migration
for remaining venus platforms in a phased out approach.

Regards,
Vikash

[1]
https://patchwork.kernel.org/project/linux-media/cover/1702899149-21321-1-git-send-email-quic_dikshita@quicinc.com/#25643473

Powered by blists - more mailing lists