linux-kernel - Re: [RfC PATCH] drm: fourcc byteorder: brings header file comments in line with reality.

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <1492605289.6853.32.camel@redhat.com>
Date:   Wed, 19 Apr 2017 14:34:49 +0200
From:   Gerd Hoffmann <kraxel@...hat.com>
To:     Pekka Paalanen <ppaalanen@...il.com>
Cc:     Michel Dänzer <michel@...nzer.net>,
        Daniel Vetter <daniel.vetter@...el.com>,
        amd-gfx@...ts.freedesktop.org,
        "dri-devel@...ts.freedesktop.org" <dri-devel@...ts.freedesktop.org>,
        open list <linux-kernel@...r.kernel.org>
Subject: Re: [RfC PATCH] drm: fourcc byteorder: brings header file comments
 in line with reality.

  Hi,

> > >> BTW, this supports Gerd's patch, since the KMS fbdev emulation code uses
> > >> e.g. DRM_FORMAT_XRGB8888 for depth/bpp 24/32, and the fbdev API uses
> > >> native endian packed colour values.  
> > > 
> > > Same is true for DRM_IOCTL_MODE_ADDFB, with depth/bpp 24/32 you'll get
> > > DRM_FORMAT_XRGB8888 (only DRM_IOCTL_MODE_ADDFB2 allows userspace specify
> > > fourcc formats directly).  
> > 
> > Right, and since all major Xorg drivers use DRM_IOCTL_MODE_ADDFB,
> > they're effectively using DRM_FORMAT_XRGB8888 as native endianness as well.
> 
> I sincerely hope this doesn't actually force us into a place where we
> have XRGB8888 (and ARGB8888?) as native-endian, but the other format
> codes - since being used explicitly - must be kept as little-endian
> because they were used like that honouring the documentation we have
> atm.

My expectation is that the other formats are (almost) unused in
practice.  cairo for example supports XRGB8888 + ARGB8888 (native
endian) only from all depth/bpp 24/32 formats.

IIRC there was a brief discussion how we should handle endianness in
qemu stdvga / bochsdrm.ko before we've added the new (virtual) hardware
register to switch endianness.  The idea to simply run with fixed
endianness (framebuffer is always little endian) was shot down quickly
with the argument that this isn't going to fly due to lack of support
for XRGB8888 in non-native byte order in the whole graphics stack.

> It's starting to resemble the wl_shm format codes problem we have
> on Wayland for BE.
> 
> Has this now turned into a question of what the kernel drivers do
> with the DRM pixel format codes?
> 
> Hmm, I suppose that has been the question all along...

Yep, basically.  I have the impression that drivers are either consider
those formats being native endian or simply don't care because they are
never used in systems with bigendian (-capable) cpus.

Anyone aware of anything else?

Guess I'll go prepare a new version of the patch, declaring all rgb
formats as native endian and putting a bunch of points from this thread
into the commit message.

cheers,
  Gerd