linux-kernel - [PATCH 3.12 67/98] saa7134: Fix bytesperline not being set correctly for planar formats

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-Id: <d49823c0cf1ae2ed9ca14e75875f9a1e5f66536b.1460380917.git.jslaby@suse.cz>
Date:	Mon, 11 Apr 2016 15:23:09 +0200
From:	Jiri Slaby <jslaby@...e.cz>
To:	stable@...r.kernel.org
Cc:	linux-kernel@...r.kernel.org, Hans de Goede <hdegoede@...hat.com>,
	Hans Verkuil <hans.verkuil@...co.com>,
	Mauro Carvalho Chehab <mchehab@....samsung.com>,
	Jiri Slaby <jslaby@...e.cz>
Subject: [PATCH 3.12 67/98] saa7134: Fix bytesperline not being set correctly for planar formats

From: Hans de Goede <hdegoede@...hat.com>

3.12-stable review patch.  If anyone has any objections, please let me know.

===============

commit 3e71da19f9dc22e39a755d6ae9678661abb66adc upstream.

bytesperline should be the bytesperline for the first plane for planar
formats, not that of all planes combined.

This fixes a crash in xawtv caused by the wrong bpl.

BugLink: https://bugzilla.redhat.com/show_bug.cgi?id=1305389
Reported-and-tested-by: Stas Sergeev <stsp@...t.ru>

Signed-off-by: Hans de Goede <hdegoede@...hat.com>
Signed-off-by: Hans Verkuil <hans.verkuil@...co.com>
Signed-off-by: Mauro Carvalho Chehab <mchehab@....samsung.com>
Signed-off-by: Jiri Slaby <jslaby@...e.cz>
---
 drivers/media/pci/saa7134/saa7134-video.c | 18 ++++++++++++------
 1 file changed, 12 insertions(+), 6 deletions(-)

diff --git a/drivers/media/pci/saa7134/saa7134-video.c b/drivers/media/pci/saa7134/saa7134-video.c
index fb60da85bc2c..d11fdfbac275 100644
--- a/drivers/media/pci/saa7134/saa7134-video.c
+++ b/drivers/media/pci/saa7134/saa7134-video.c
@@ -1562,10 +1562,13 @@ static int saa7134_g_fmt_vid_cap(struct file *file, void *priv,
 	f->fmt.pix.height       = dev->height;
 	f->fmt.pix.field        = fh->cap.field;
 	f->fmt.pix.pixelformat  = dev->fmt->fourcc;
-	f->fmt.pix.bytesperline =
-		(f->fmt.pix.width * dev->fmt->depth) >> 3;
+	if (dev->fmt->planar)
+		f->fmt.pix.bytesperline = f->fmt.pix.width;
+	else
+		f->fmt.pix.bytesperline =
+			(f->fmt.pix.width * dev->fmt->depth) / 8;
 	f->fmt.pix.sizeimage =
-		f->fmt.pix.height * f->fmt.pix.bytesperline;
+		(f->fmt.pix.height * f->fmt.pix.width * dev->fmt->depth) / 8;
 	f->fmt.pix.colorspace   = V4L2_COLORSPACE_SMPTE170M;
 	f->fmt.pix.priv = 0;
 	return 0;
@@ -1646,10 +1649,13 @@ static int saa7134_try_fmt_vid_cap(struct file *file, void *priv,
 	if (f->fmt.pix.height > maxh)
 		f->fmt.pix.height = maxh;
 	f->fmt.pix.width &= ~0x03;
-	f->fmt.pix.bytesperline =
-		(f->fmt.pix.width * fmt->depth) >> 3;
+	if (fmt->planar)
+		f->fmt.pix.bytesperline = f->fmt.pix.width;
+	else
+		f->fmt.pix.bytesperline =
+			(f->fmt.pix.width * fmt->depth) / 8;
 	f->fmt.pix.sizeimage =
-		f->fmt.pix.height * f->fmt.pix.bytesperline;
+		(f->fmt.pix.height * f->fmt.pix.width * fmt->depth) / 8;
 	f->fmt.pix.colorspace   = V4L2_COLORSPACE_SMPTE170M;
 	f->fmt.pix.priv = 0;
 
-- 
2.8.1