linux-kernel - Re: [PATCH v2 2/4] media: rockchip: Introduce the rkvdec2 driver

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date: Thu, 20 Jun 2024 01:46:23 +0800
From: Jianfeng Liu <liujianfeng1994@...il.com>
To: detlev.casanova@...labora.com
Cc: andy.yan@...k-chips.com,
	benjamin.gaignard@...labora.com,
	boris.brezillon@...labora.com,
	conor+dt@...nel.org,
	daniel.almeida@...labora.com,
	devicetree@...r.kernel.org,
	didi.debian@...ow.org,
	dsimic@...jaro.org,
	ezequiel@...guardiasur.com.ar,
	gregkh@...uxfoundation.org,
	heiko@...ech.de,
	hverkuil-cisco@...all.nl,
	jonas@...boo.se,
	knaerzche@...il.com,
	krzk+dt@...nel.org,
	linux-arm-kernel@...ts.infradead.org,
	linux-kernel@...r.kernel.org,
	linux-media@...r.kernel.org,
	linux-rockchip@...ts.infradead.org,
	linux-staging@...ts.linux.dev,
	mchehab@...nel.org,
	nicolas.dufresne@...labora.com,
	paul.kocialkowski@...tlin.com,
	robh@...nel.org,
	sebastian.reichel@...labora.com
Subject: Re: [PATCH v2 2/4] media: rockchip: Introduce the rkvdec2 driver

Hi Detlev,

On Wed, 19 Jun 2024 10:57:19 -0400, Detlev Casanova wrote:
>+	if (!(sps->flags & V4L2_H264_SPS_FLAG_FRAME_MBS_ONLY))
>+		height *= 2;
>+
>+	if (width > ctx->coded_fmt.fmt.pix_mp.width ||
>+	    height > ctx->coded_fmt.fmt.pix_mp.height)
>+		return -EINVAL;

I did further invesatigation on chromium. I find that before real video
is decoded, chromium will call VIDIOC_STREAMON twice with value of
sps->flags 0:

At the first time width and height are 16; ctx->coded_fmt.fmt.pix_mp.width
and coded_fmt.fmt.pix_mp.height are 16, which are the min size of decoder;
At the second time width and height are still 16; while
coded_fmt.fmt.pix_mp.width is 1920 and coded_fmt.fmt.pix_mp.height is
1088, which are the real size of video.

So VIDIOC_STREAMON will fall at the first time call because sps->flags is
0 so V4L2_H264_SPS_FLAG_FRAME_MBS_ONLY is not set, and then height is
doubled to 32 which is larger than 16.

What do you think if we skip doubling height if sps->flags is 0 and at the
same time V4L2_H264_SPS_FLAG_FRAME_MBS_ONLY is not set? The following hack
did fix my chromium:

--- a/drivers/staging/media/rkvdec2/rkvdec2-h264.c
+++ b/drivers/staging/media/rkvdec2/rkvdec2-h264.c
@@ -767,7 +767,7 @@ static int rkvdec2_h264_validate_sps(struct rkvdec2_ctx *ctx,
         * which is half the final height (see (7-18) in the
         * specification)
         */
-       if (!(sps->flags & V4L2_H264_SPS_FLAG_FRAME_MBS_ONLY))
+       if (!(sps->flags & V4L2_H264_SPS_FLAG_FRAME_MBS_ONLY) && sps->flags)
                height *= 2;

        if (width > ctx->coded_fmt.fmt.pix_mp.width ||

Best regards
Jianfeng