Message-ID: <be621717ee62c2ffda40d531dda15e0c9f28a858.camel@ndufresne.ca>
Date: Wed, 03 Sep 2025 09:07:57 -0400
From: Nicolas Dufresne <nicolas@...fresne.ca>
To: DEEPA GUTHYAPPA MADIVALARA <deepa.madivalara@....qualcomm.com>, Mauro
 Carvalho Chehab <mchehab@...nel.org>, Vikash Garodia
 <quic_vgarodia@...cinc.com>, Dikshita Agarwal	 <quic_dikshita@...cinc.com>,
 Abhinav Kumar <abhinav.kumar@...ux.dev>, Bryan O'Donoghue
 <bryan.odonoghue@...aro.org>
Cc: linux-media@...r.kernel.org, linux-kernel@...r.kernel.org, 
	linux-arm-msm@...r.kernel.org
Subject: Re: [PATCH RFC 0/5] Enable support for AV1 stateful decoder

Le mardi 02 septembre 2025 à 16:00 -0700, DEEPA GUTHYAPPA MADIVALARA a écrit :
> Hi all,
> 
> This patch series adds initial support for the AV1 stateful
> decoder codecs in iris decoder. Also it adds support for AV1
> stateful decoder in V4l2. The objective of this work is to
> extend the Iris decoder's capabilities to handle AV1 format
> codec streams, including necessary format handling and buffer
> management. I'm sharing this series as an RFC because conformance
> testing and gstreamer testing are still in progress. While initial
> functional tests show positive results, I would appreciate early
> feedback on the design, implementation, and fixes before moving to
> a formal submission. I plan to submit a formal patch series after
> completing all the compliance checks. Meanwhile, any feedback or
> suggestion to improve this work are very welcome and will be of
> great help.
> 
> Gstreamer testing:
> Gstreamer MR for enabling AV1 stateful decoder:
> https://gitlab.freedesktop.org/dmadival/gstreamer/-/merge_requests/1
> 
> Thanks to Nicolas Dufresne for proving the MR 
> https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/9469
> 
> However, Gst testing with parsebin is not resolving to av1parser as
> below:
> Ex:  With the following command parsebin is unable to resolve to
> av1parser.
> GST_DEBUG=*:2,parsebin:6 gst-launch-1.0 --no-fault 
> filesrc
> location=/media/sd/fluster/fluster/resources/AV1-ARGON-PROFILE0-CORE-ANNEX-B/
> argon_coveragetool_av1_base_and_extended_profiles_v2.1/profile0_core/streams/test10220.obu
> ! parsebin ! v4l2av1dec ! video/x-raw ! videoconvert dither=none !
> video/x-raw,format=I420
> ! filesink location=gst_decoder_output.yuv

Be aware that Argon integration with fluster/GStreamer still needs some work,
see issue tracker:

https://github.com/fluendo/fluster/issues/222

It is fine to proceed with testing the other (IVF or MKV based) test suites.
Feel free to report all wrongly identified files onto that issue, this will be
addressed in GStreamer (and eventually in FFMPEG too if need be).

> 
> 0:00:00.051674896   400 0xffff8c000b90 DEBUG
> parsebin gstparsebin.c:2439:type_found:<parsebin0> typefind found caps
> video/x-h263, variant=(string)itu
> 
> The same test with the av1parse command parses correctly:
> GST_DEBUG=*:2,av1parse:6 gst-launch-1.0 --no-fault
> filesrc
> location=/media/sd/fluster/fluster/resources/AV1-ARGON-PROFILE0-CORE-ANNEX-B/
> argon_coveragetool_av1_base_and_extended_profiles_v2.1/profile0_core/streams/test10220.obu
> ! av1parse ! v4l2av1dec ! video/x-raw ! videoconvert dither=none !
> video/x-raw,format=I420
> ! filesink location=/tmp/gst_decoder_output.yuv
> 
> Got EOS from element "pipeline0".
> Execution ended after 0:00:01.599088176
> Setting pipeline to NULL ...
> 0:00:03.580831249  1075     0x3354f960 DEBUG
> av1parse gstav1parse.c:435:gst_av1_parse_stop:<av1parse0> stop
> 
> Fluster testing:
> As fluster.py is using parsebin for gstreamer, seeing the same issue as
> described above for the following testsuites.
> AV1-ARGON-PROFILE0-CORE-ANNEX-B
> AV1-ARGON-PROFILE0-NON-ANNEX-B
> AV1-ARGON-PROFILE0-NON-ANNEX-B
> 
> Test suite: AV1-TEST-VECTORS 
> The result of fluster test on SM8550:
> 134/242 testcases passed while testing AV1-TEST-VECTORS with
> GStreamer-AV1-V4L2-Gst1.0
> unsupported content, bit depth: a000a (66 tests)
> Iris hardware decoder supports only 8bit NV12
> av1-1-b10-00-quantizer-*          

That is interesting, I believe there was no profile without 10bit, which would
mean this is non-compliant hardware. Not a blocker, just a remark, and me being
surprised 10bit isn't supported on modern codec in 2025.

> 
> Unsupported resolution (36 tests).
> Iris hardware decoder supports min resolution of 96x96
> av1-1-b8-01-size-*
> 
> Unsupported colorformat (1 test)
> av1-1-b8-24-monochrome

Fair enough.

> 
> Crc mismatch: debug in progress (5tests)
> av1-1-b8-03-sizeup                       
> av1-1-b8-03-sizedown                     

These should be basic dynamic resolution changes cases, let me know what you
find.

> av1-1-b8-16-intra_only-intrabc-extreme-dv

No idea about this one.

> av1-1-b8-22-svc-L2T1                     
> av1-1-b8-22-svc-L2T2  

These two are spatial SVC. That means that some decode only frames are going to
be smaller in dimensions. Only the last frame of a TU is displayed. Both do have
the optional sequence header announcing the maximum resolution though. On
stateless side, these usually works by simply decoding the hidden frames into
the much larger buffers. Dealing with firmware can be harder, since firmware may
silently drop the decode only frames, which leads to wrong timestamp matching,
which could have side effect in frameworks.

On stateful decoding, SVC have never been done, some documentation will likely
be needed. Strictly unsupported is the case the sequence header is not there,
which results in resolution change on non-keyframe. We now have the ability to
allocate frames of different dimension at run-time, and free old unused frames,
but in stateful, decoders streamoff the capture queue, which currently results
in lost of references.

Let us know your progress, inter-frame resolution change is not a strict
requirement, it is quite rare in real life.

looking forward, there is nothing particularly alarming in this report. Thanks
for the transparency.

Nicolas

> 
> Testsuite: CHROMIUM-8bit-AV1-TEST-VECTORS
> 12/13 testcases passed while testing CHROMIUM-8bit-AV1-TEST-VECTORS with
> GStreamer-AV1-V4L2-Gst1.0
> Crc mismatch: debug in progress
> av1-1-b8-03-sizeup.ivf  
> 
> Unsupported test suites:
> Iris Hardware Decoder supports only
> PROFILE0/V4L2_MPEG_VIDEO_AV1_PROFILE_MAIN
> and 8 bit, 420 only
> AV1-ARGON-PROFILE1-CORE-ANNEX-B
> AV1-ARGON-PROFILE1-NON-ANNEX-B
> AV1-ARGON-PROFILE1-STRESS-ANNEX-B
> AV1-ARGON-PROFILE2-CORE-ANNEX-B
> AV1-ARGON-PROFILE2-NON-ANNEX-B
> AV1-ARGON-PROFILE2-STRESS-ANNEX-B
> CHROMIUM-10bit-AV1-TEST-VECTORS
> 
> Compliance test for iris_driver device /dev/video0:
> 
> Driver Info:
>         Driver name      : iris_driver
>         Card type        : Iris Decoder
>         Bus info         : platform:aa00000.video-codec
>         Driver version   : 6.16.0
>         Capabilities     : 0x84204000
>                 Video Memory-to-Memory Multiplanar
>                 Streaming
>                 Extended Pix Format
>                 Device Capabilities
>         Device Caps      : 0x04204000
>                 Video Memory-to-Memory Multiplanar
>                 Streaming
>                 Extended Pix Format
>         Detected Stateful Decoder
> 
> Required ioctls:
>         test VIDIOC_QUERYCAP: OK
>         test invalid ioctls: OK
> 
> Allow for multiple opens:
>         test second /dev/video0 open: OK
>         test VIDIOC_QUERYCAP: OK
>         test VIDIOC_G/S_PRIORITY: OK
>         test for unlimited opens: OK
> 
> Debug ioctls:
>         test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported)
>         test VIDIOC_LOG_STATUS: OK (Not Supported)
> 
> Input ioctls:
>         test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
>         test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
>         test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
>         test VIDIOC_ENUMAUDIO: OK (Not Supported)
>         test VIDIOC_G/S/ENUMINPUT: OK (Not Supported)
>         test VIDIOC_G/S_AUDIO: OK (Not Supported)
>         Inputs: 0 Audio Inputs: 0 Tuners: 0
> 
> Output ioctls:
>         test VIDIOC_G/S_MODULATOR: OK (Not Supported)
>         test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
>         test VIDIOC_ENUMAUDOUT: OK (Not Supported)
>         test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
>         test VIDIOC_G/S_AUDOUT: OK (Not Supported)
>         Outputs: 0 Audio Outputs: 0 Modulators: 0
> 
> Input/Output configuration ioctls:
>         test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
>         test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
>         test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
>         test VIDIOC_G/S_EDID: OK (Not Supported)
> 
> Control ioctls:
>         test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK
>         test VIDIOC_QUERYCTRL: OK
>         test VIDIOC_G/S_CTRL: OK
>         test VIDIOC_G/S/TRY_EXT_CTRLS: OK
>         test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK
>         test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
>         Standard Controls: 12 Private Controls: 0
> 
> Format ioctls:
>         test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
>         test VIDIOC_G/S_PARM: OK (Not Supported)
>         test VIDIOC_G_FBUF: OK (Not Supported)
>         test VIDIOC_G_FMT: OK
>         test VIDIOC_TRY_FMT: OK
>         test VIDIOC_S_FMT: OK
>         test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
>         test Cropping: OK
>         test Composing: OK
>         test Scaling: OK (Not Supported)
> 
> Codec ioctls:
>         test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
>         test VIDIOC_G_ENC_INDEX: OK (Not Supported)
>         test VIDIOC_(TRY_)DECODER_CMD: OK
> 
> Buffer ioctls:
>         test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK
>         test CREATE_BUFS maximum buffers: OK
>         test VIDIOC_REMOVE_BUFS: OK
>         test VIDIOC_EXPBUF: OK
>         test Requests: OK (Not Supported)
> [65391.311675] qcom-iris aa00000.video-codec: invalid plane
> [65395.340586] qcom-iris aa00000.video-codec: invalid plane
>         test blocking wait: OK
> 
> Total for iris_driver device /dev/video0: 48, Succeeded: 48, Failed: 0,
> Warnings: 0
> 
> V4l2-ctl Test verified for 2 streams as well.
> 
> Thanks,
> Deepa
> 
> Signed-off-by: Deepa Guthyappa Madivalara <deepa.madivalara@....qualcomm.com>
> ---
> DEEPA GUTHYAPPA MADIVALARA (5):
>       media: uapi: videodev2: Add support for AV1 stateful decoder
>       media: v4l2: Add description for V4L2_PIX_FMT_AV1 in v4l_fill_fmtdesc()
>       media: iris: Add support for AV1 format in iris decoder
>       media: iris: Add internal buffer calculation for AV1 decoder
>       media: iris: Define AV1-specific platform capabilities and properties
> 
>  drivers/media/platform/qcom/iris/iris_buffer.h     |   2 +
>  drivers/media/platform/qcom/iris/iris_ctrls.c      |   8 +
>  drivers/media/platform/qcom/iris/iris_hfi_common.h |   3 +
>  .../platform/qcom/iris/iris_hfi_gen2_command.c     | 109 ++++++++-
>  .../platform/qcom/iris/iris_hfi_gen2_defines.h     |  10 +
>  .../platform/qcom/iris/iris_hfi_gen2_response.c    |  22 ++
>  drivers/media/platform/qcom/iris/iris_instance.h   |   1 +
>  .../platform/qcom/iris/iris_platform_common.h      |  15 ++
>  .../media/platform/qcom/iris/iris_platform_gen2.c  | 156 ++++++++++++-
>  .../platform/qcom/iris/iris_platform_sm8250.c      |  17 ++
>  drivers/media/platform/qcom/iris/iris_vdec.c       |  25 +-
>  drivers/media/platform/qcom/iris/iris_vidc.c       |   1 +
>  drivers/media/platform/qcom/iris/iris_vpu_buffer.c | 255 ++++++++++++++++++++-
>  drivers/media/platform/qcom/iris/iris_vpu_buffer.h | 105 +++++++++
>  drivers/media/v4l2-core/v4l2-ioctl.c               |   1 +
>  include/uapi/linux/videodev2.h                     |   1 +
>  16 files changed, 703 insertions(+), 28 deletions(-)
> ---
> base-commit: 88a6b4187eacb700a678296afb0c610eb3781e2f
> change-id: 20250821-rfc_split-c3ff834bb2c9
> prerequisite-change-id: 20250704-iris-video-encoder-b193350b487a:v3
> prerequisite-patch-id: 8a566690da276da34430c10dbc2fe64c1d623a9c
> prerequisite-patch-id: 1430a33603b425d0b142aab98befcda771fb885e
> prerequisite-patch-id: 32024cd49d2445ff396e31f40739b32597be59a4
> prerequisite-patch-id: 65b569952650647174e8221dc7adde9b000a7ae3
> prerequisite-patch-id: da128980fab8538bf668f19016c5121fb03759c2
> prerequisite-patch-id: 079823dffbe8b89990797bf7f7640b754382d8ce
> prerequisite-patch-id: 6ce10e03d7b3b96b2391e26cda703b650bde7cd0
> prerequisite-patch-id: b5950670ac5068a0c5b26651ebd433f7d3bbe6ca
> prerequisite-patch-id: 4de7a934f6bdfe28c84e461f70495925aa98365e
> prerequisite-patch-id: 07682a6d2530b5796122bf8763f94b5bc92949ec
> prerequisite-patch-id: 72b7eba20f1a222908d41323f28be3ba84106759
> prerequisite-patch-id: fd9e2e1b157112c39c69486799493ee99e6033a7
> prerequisite-patch-id: ae0ad8a04a04dd3434a092d4c2bb3f493417c6e1
> prerequisite-patch-id: 52631eec348735d1dc5f5804b573e3cf942550a0
> prerequisite-patch-id: 4109c59edb1b757162db46297914c8f7c14408dc
> prerequisite-patch-id: fc0b713eb4822047e8172d11fd4cd5a097ef23a5
> prerequisite-patch-id: 20ac8e7307f1f852b2a43268b2474178fbc0b94c
> prerequisite-patch-id: e8419d716573beb64ad89968f0074d6bddfa86d3
> prerequisite-patch-id: bdc72f5876ceb2e981d594c86a45cb21a6264af3
> prerequisite-patch-id: fc5d26d01cab94d229a00eab819ae80196f3f5d5
> prerequisite-patch-id: 32a9fe1371fffc9abd9a862b2814050a144d1968
> prerequisite-patch-id: 25184583b5de886f78ee0444a4a59d5f3c271ce5
> prerequisite-patch-id: df3376b9de27b23ae81a4c7b7a8fe4b429c32423
> prerequisite-patch-id: e68fbf7c82567d2e9f3fdd0fdf2e2911329d5ccd
> prerequisite-patch-id: 580fa40de01a81a8685e56420f562d299bfc60fa
> prerequisite-patch-id: 89548da6690681854ee1de992a491bed73202b83
> 
> Best regards,

Download attachment "signature.asc" of type "application/pgp-signature" (229 bytes)

Powered by blists - more mailing lists