linux-kernel - [PATCH 3/3] drm: fourcc byteorder: add drm_mode_legacy_fb_format

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [day] [month] [year] [list]

Message-Id: <20170502133404.15354-4-kraxel@redhat.com>
Date:   Tue,  2 May 2017 15:34:04 +0200
From:   Gerd Hoffmann <kraxel@...hat.com>
To:     dri-devel@...ts.freedesktop.org
Cc:     Ville Syrjälä 
        <ville.syrjala@...ux.intel.com>,
        Daniel Vetter <daniel.vetter@...el.com>,
        Pekka Paalanen <ppaalanen@...il.com>,
        Ilia Mirkin <imirkin@...m.mit.edu>,
        Michel Dänzer <michel@...nzer.net>,
        Alex Deucher <alexdeucher@...il.com>,
        amd-gfx@...ts.freedesktop.org,
        Jani Nikula <jani.nikula@...ux.intel.com>,
        Sean Paul <seanpaul@...omium.org>,
        David Airlie <airlied@...ux.ie>,
        Gerd Hoffmann <kraxel@...hat.com>,
        linux-kernel@...r.kernel.org (open list)
Subject: [PATCH 3/3] drm: fourcc byteorder: add drm_mode_legacy_fb_format_he

Add drm_mode_legacy_fb_format variant which returns fourcc codes
for framebuffer format in host byte order.

Signed-off-by: Gerd Hoffmann <kraxel@...hat.com>
---
 include/drm/drm_fourcc.h     |  3 +++
 drivers/gpu/drm/drm_fourcc.c | 54 +++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 56 insertions(+), 1 deletion(-)

diff --git a/include/drm/drm_fourcc.h b/include/drm/drm_fourcc.h
index cae05153e8..729e9d1b11 100644
--- a/include/drm/drm_fourcc.h
+++ b/include/drm/drm_fourcc.h
@@ -73,7 +73,10 @@ const struct drm_format_info *drm_format_info(u32 format);
 const struct drm_format_info *
 drm_get_format_info(struct drm_device *dev,
 		    const struct drm_mode_fb_cmd2 *mode_cmd);
+
 uint32_t drm_mode_legacy_fb_format(uint32_t bpp, uint32_t depth);
+uint32_t drm_mode_legacy_fb_format_he(uint32_t bpp, uint32_t depth);
+
 int drm_format_num_planes(uint32_t format);
 int drm_format_plane_cpp(uint32_t format, int plane);
 int drm_format_horz_chroma_subsampling(uint32_t format);
diff --git a/drivers/gpu/drm/drm_fourcc.c b/drivers/gpu/drm/drm_fourcc.c
index adb3ff59a4..97ff2cdf4e 100644
--- a/drivers/gpu/drm/drm_fourcc.c
+++ b/drivers/gpu/drm/drm_fourcc.c
@@ -36,12 +36,24 @@ static char printable_char(int c)
 }
 
 /**
- * drm_mode_legacy_fb_format - compute drm fourcc code from legacy description
+ * drm_mode_legacy_fb_format - compute drm fourcc code from legacy description,
+ *                             little endian.
  * @bpp: bits per pixels
  * @depth: bit depth per pixel
  *
+ * Deprecated, use drm_mode_legacy_fb_format_he instead.
+ *
  * Computes a drm fourcc pixel format code for the given @bpp/@...th values.
  * Useful in fbdev emulation code, since that deals in those values.
+ *
+ * Note that drm_mode_addfb (DRM_IOCTL_MODE_ADDFB implementation) uses this
+ * too.
+ *
+ * For historical reasons, this function returns fourcc codes for framebuffer
+ * formats in little endian byte order unconditinally, even though fbdev
+ * emulation expects the framebuffer in host byte order (i.e. big endian on
+ * big endian machines).  Ideally we would simply fix this function, but that
+ * would break drivers expecting the broken behavior ...
  */
 uint32_t drm_mode_legacy_fb_format(uint32_t bpp, uint32_t depth)
 {
@@ -79,6 +91,46 @@ uint32_t drm_mode_legacy_fb_format(uint32_t bpp, uint32_t depth)
 EXPORT_SYMBOL(drm_mode_legacy_fb_format);
 
 /**
+ * drm_mode_legacy_fb_format_he - compute drm fourcc code from legacy
+ *                                description, host endian.
+ * @bpp: bits per pixels
+ * @depth: bit depth per pixel
+ *
+ * Computes a drm fourcc pixel format code for the given @bpp/@...th values.
+ * Useful in fbdev emulation code, since that deals in those values.
+ */
+uint32_t drm_mode_legacy_fb_format_he(uint32_t bpp, uint32_t depth)
+{
+#ifdef __BIG_ENDIAN
+	uint32_t fmt;
+
+	switch (bpp) {
+	case 8:
+		fmt = DRM_FORMAT_C8;
+		break;
+	case 24:
+		fmt = DRM_FORMAT_BGR888;
+		break;
+	case 32:
+		if (depth == 24)
+			fmt = DRM_FORMAT_BGRX8888;
+		else
+			fmt = DRM_FORMAT_BGRA8888;
+		break;
+	default:
+		DRM_ERROR("bad bpp, assuming b8g8r8x8 pixel format\n");
+		fmt = DRM_FORMAT_BGRX8888;
+		break;
+	}
+
+	return fmt;
+#else
+	return drm_mode_legacy_fb_format(bpp, depth);
+#endif
+}
+EXPORT_SYMBOL(drm_mode_legacy_fb_format_he);
+
+/**
  * drm_get_format_name - fill a string with a drm fourcc format's name
  * @format: format to compute name of
  * @buf: caller-supplied buffer
-- 
2.9.3