linux-kernel - Re: [PATCH 3/6] drm: fourcc byteorder: add bigendian support to drm_mode_legacy_fb

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:   Wed, 26 Apr 2017 14:28:01 +0100
From:   Eric Engestrom <eric.engestrom@...tec.com>
To:     Gerd Hoffmann <kraxel@...hat.com>
CC:     Michel Dänzer <michel@...nzer.net>,
        Daniel Vetter <daniel.vetter@...el.com>,
        <amd-gfx@...ts.freedesktop.org>,
        open list <linux-kernel@...r.kernel.org>,
        <dri-devel@...ts.freedesktop.org>
Subject: Re: [PATCH 3/6] drm: fourcc byteorder: add bigendian support to
 drm_mode_legacy_fb_format

On Wednesday, 2017-04-26 07:53:10 +0200, Gerd Hoffmann wrote:
> On Di, 2017-04-25 at 12:18 +0900, Michel Dänzer wrote:
> > On 24/04/17 03:25 PM, Gerd Hoffmann wrote:
> > > Return correct fourcc codes on bigendian.  Drivers must be adapted to
> > > this change.
> > > 
> > > Signed-off-by: Gerd Hoffmann <kraxel@...hat.com>
> > 
> > Just to reiterate, this won't work for the radeon driver, which programs
> > the GPU to use (effectively, per the current definition that these are
> > little endian GPU formats) DRM_FORMAT_XRGB8888 with pre-R600 and
> > DRM_FORMAT_BGRX8888 with >= R600.
> 
> Hmm, ok, how does bigendian fbdev emulation work on pre-R600 then?
> 
> > > +#ifdef __BIG_ENDIAN
> > > +	switch (bpp) {
> > > +	case 8:
> > > +		fmt = DRM_FORMAT_C8;
> > > +		break;
> > > +	case 24:
> > > +		fmt = DRM_FORMAT_BGR888;
> > > +		break;
> > 
> > BTW, endianness as a concept cannot apply to 8 or 24 bpp formats.
> 
> I could move the 8 bpp case out of the #ifdef somehow, but code
> readability will suffer then I think ...

How about something like this?

	uint32_t drm_mode_legacy_fb_format(uint32_t bpp, uint32_t depth)
	{
		uint32_t fmt;
	#ifdef __BIG_ENDIAN
		enum { LITTLE_ENDIAN = 0 };
	#else
		enum { LITTLE_ENDIAN = 1 };
	#endif
	/* ... */

(using an enum for compile-time constness)

and then
	fmt = DRM_FORMAT_ARGB8888;
becomes
	fmt = LITTLE_ENDIAN ? DRM_FORMAT_ARGB8888 : DRM_FORMAT_BGRA8888;

Might be easier to read than duplicating the whole switch?

> 
> For 24 we have different byte orderings, but yes, you can't switch from
> one to the other with byteswapping.  Probably one of the reasons why
> this format is pretty much out of fashion these days ...
> 
> cheers,
>   Gerd
>