linux-kernel - [PATCH 3/6] drm: fourcc byteorder: add bigendian support to drm_mode_legacy_fb

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Date:   Mon, 24 Apr 2017 08:25:29 +0200
From:   Gerd Hoffmann <kraxel@...hat.com>
To:     dri-devel@...ts.freedesktop.org
Cc:     Ville Syrjälä 
        <ville.syrjala@...ux.intel.com>,
        Daniel Vetter <daniel.vetter@...el.com>,
        Pekka Paalanen <ppaalanen@...il.com>,
        Ilia Mirkin <imirkin@...m.mit.edu>,
        Michel Dänzer <michel@...nzer.net>,
        Alex Deucher <alexdeucher@...il.com>,
        amd-gfx@...ts.freedesktop.org,
        Jani Nikula <jani.nikula@...ux.intel.com>,
        Sean Paul <seanpaul@...omium.org>,
        David Airlie <airlied@...ux.ie>,
        Gerd Hoffmann <kraxel@...hat.com>,
        linux-kernel@...r.kernel.org (open list)
Subject: [PATCH 3/6] drm: fourcc byteorder: add bigendian support to drm_mode_legacy_fb_format

Return correct fourcc codes on bigendian.  Drivers must be adapted to
this change.

Signed-off-by: Gerd Hoffmann <kraxel@...hat.com>
---
 drivers/gpu/drm/drm_fourcc.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/drivers/gpu/drm/drm_fourcc.c b/drivers/gpu/drm/drm_fourcc.c
index adb3ff59a4..28401d3745 100644
--- a/drivers/gpu/drm/drm_fourcc.c
+++ b/drivers/gpu/drm/drm_fourcc.c
@@ -42,11 +42,34 @@ static char printable_char(int c)
  *
  * Computes a drm fourcc pixel format code for the given @bpp/@...th values.
  * Useful in fbdev emulation code, since that deals in those values.
+ *
+ * DRM_FORMAT_* are little endian, we'll pick cpu endian here, therefore we
+ * results differ depending on byte order.
  */
 uint32_t drm_mode_legacy_fb_format(uint32_t bpp, uint32_t depth)
 {
 	uint32_t fmt;
 
+#ifdef __BIG_ENDIAN
+	switch (bpp) {
+	case 8:
+		fmt = DRM_FORMAT_C8;
+		break;
+	case 24:
+		fmt = DRM_FORMAT_BGR888;
+		break;
+	case 32:
+		if (depth == 24)
+			fmt = DRM_FORMAT_BGRX8888;
+		else
+			fmt = DRM_FORMAT_BGRA8888;
+		break;
+	default:
+		DRM_ERROR("bad bpp, assuming b8g8r8x8 pixel format\n");
+		fmt = DRM_FORMAT_BGRX8888;
+		break;
+	}
+#else
 	switch (bpp) {
 	case 8:
 		fmt = DRM_FORMAT_C8;
@@ -73,6 +96,7 @@ uint32_t drm_mode_legacy_fb_format(uint32_t bpp, uint32_t depth)
 		fmt = DRM_FORMAT_XRGB8888;
 		break;
 	}
+#endif
 
 	return fmt;
 }
-- 
2.9.3