linux-kernel - [PATCH] hantro: Fix JPEG encoder ENUM

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-Id: <20221223141644.703088-1-nicolas.dufresne@collabora.com>
Date:   Fri, 23 Dec 2022 09:16:44 -0500
From:   Nicolas Dufresne <nicolas.dufresne@...labora.com>
To:     Ezequiel Garcia <ezequiel@...guardiasur.com.ar>,
        Philipp Zabel <p.zabel@...gutronix.de>,
        Mauro Carvalho Chehab <mchehab@...nel.org>,
        Heiko Stuebner <heiko@...ech.de>,
        Hans Verkuil <hverkuil-cisco@...all.nl>
Cc:     kernel@...labora.com,
        Nicolas Dufresne <nicolas.dufresne@...labora.com>,
        Robert Mader <robert.mader@...labora.com>,
        Mauro Carvalho Chehab <mchehab+samsung@...nel.org>,
        linux-media@...r.kernel.org, linux-rockchip@...ts.infradead.org,
        linux-arm-kernel@...ts.infradead.org, linux-kernel@...r.kernel.org
Subject: [PATCH] hantro: Fix JPEG encoder ENUM_FRAMESIZE on RK3399

The frmsize structure was left initialize to 0, as side effect, the driver was
reporting an invalid frmsize.

  Size: Stepwise 0x0 - 0x0 with step 0/0

Fix this by replicating the constraints in the raw formats too. This fixes
taking picture in Gnome Cheese Software, or any software using GSteamer
encodebin feature.

Fixes: 775fec69008d30 ("media: add Rockchip VPU JPEG encoder driver")
Reported-by: Robert Mader <robert.mader@...labora.com>
Signed-off-by: Nicolas Dufresne <nicolas.dufresne@...labora.com>
---
 .../platform/verisilicon/rockchip_vpu_hw.c    | 22 ++++++++++++-------
 1 file changed, 14 insertions(+), 8 deletions(-)

diff --git a/drivers/media/platform/verisilicon/rockchip_vpu_hw.c b/drivers/media/platform/verisilicon/rockchip_vpu_hw.c
index 8de6fd2e8eefa..7e65b5c3aa03c 100644
--- a/drivers/media/platform/verisilicon/rockchip_vpu_hw.c
+++ b/drivers/media/platform/verisilicon/rockchip_vpu_hw.c
@@ -21,40 +21,46 @@
  * Supported formats.
  */
 
+#define ROCKCHIP_VPU_JPEG_ENC_FRMSIZE {\
+                        .min_width = 96, \
+                        .max_width = 8192,\
+                        .step_width = MB_DIM, \
+                        .min_height = 32, \
+                        .max_height = 8192,\
+                        .step_height = MB_DIM, \
+		     }
+
 static const struct hantro_fmt rockchip_vpu_enc_fmts[] = {
 	{
 		.fourcc = V4L2_PIX_FMT_YUV420M,
 		.codec_mode = HANTRO_MODE_NONE,
 		.enc_fmt = ROCKCHIP_VPU_ENC_FMT_YUV420P,
+		.frmsize = ROCKCHIP_VPU_JPEG_ENC_FRMSIZE,
 	},
 	{
 		.fourcc = V4L2_PIX_FMT_NV12M,
 		.codec_mode = HANTRO_MODE_NONE,
 		.enc_fmt = ROCKCHIP_VPU_ENC_FMT_YUV420SP,
+		.frmsize = ROCKCHIP_VPU_JPEG_ENC_FRMSIZE,
 	},
 	{
 		.fourcc = V4L2_PIX_FMT_YUYV,
 		.codec_mode = HANTRO_MODE_NONE,
 		.enc_fmt = ROCKCHIP_VPU_ENC_FMT_YUYV422,
+		.frmsize = ROCKCHIP_VPU_JPEG_ENC_FRMSIZE,
 	},
 	{
 		.fourcc = V4L2_PIX_FMT_UYVY,
 		.codec_mode = HANTRO_MODE_NONE,
 		.enc_fmt = ROCKCHIP_VPU_ENC_FMT_UYVY422,
+		.frmsize = ROCKCHIP_VPU_JPEG_ENC_FRMSIZE,
 	},
 	{
 		.fourcc = V4L2_PIX_FMT_JPEG,
 		.codec_mode = HANTRO_MODE_JPEG_ENC,
 		.max_depth = 2,
 		.header_size = JPEG_HEADER_SIZE,
-		.frmsize = {
-			.min_width = 96,
-			.max_width = 8192,
-			.step_width = MB_DIM,
-			.min_height = 32,
-			.max_height = 8192,
-			.step_height = MB_DIM,
-		},
+		.frmsize = ROCKCHIP_VPU_JPEG_ENC_FRMSIZE,
 	},
 };
 
-- 
2.38.1