linux-kernel - Re: [PATCH] drm/ast: Fix big-endian support

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-Id: <A0A92AB8-FA61-4AAC-96C9-00BE93E3F6D6@exactco.de>
Date: Fri, 5 Dec 2025 20:50:06 +0100
From: René Rebe <rene@...ctco.de>
To: Thomas Zimmermann <tzimmermann@...e.de>
Cc: Timothy Pearson <tpearson@...torengineering.com>,
 dri-devel <dri-devel@...ts.freedesktop.org>,
 linux-kernel <linux-kernel@...r.kernel.org>,
 Dave Airlie <airlied@...hat.com>
Subject: Re: [PATCH] drm/ast: Fix big-endian support

Hi,

> On 5. Dec 2025, at 20:46, Thomas Zimmermann <tzimmermann@...e.de> wrote:
> 
> Hi
> 
> Am 05.12.25 um 19:31 schrieb Timothy Pearson:
>> 
>> ----- Original Message -----
>>> From: "René Rebe" <rene@...ctco.de>
>>> To: tzimmermann@...e.de
>>> Cc: "dri-devel" <dri-devel@...ts.freedesktop.org>, "linux-kernel" <linux-kernel@...r.kernel.org>, "Dave Airlie"
>>> <airlied@...hat.com>, "Timothy Pearson" <tpearson@...torengineering.com>
>>> Sent: Friday, December 5, 2025 9:14:59 AM
>>> Subject: Re: [PATCH] drm/ast: Fix big-endian support
>>> Hello Thomas,
>>> 
>>> On Wed, 3 Dec 2025 10:40:17 +0100, Thomas Zimmermann <tzimmermann@...e.de>
>>> wrote:
>>> 
>>>> [2]
>>>> https://gitlab.freedesktop.org/drm/misc/kernel/-/blob/drm-misc-next-2025-12-01-1/drivers/gpu/drm/ast/ast_mode.c?ref_type=tags#L559
>>>> [3]
>>>> https://gitlab.freedesktop.org/drm/misc/kernel/-/blob/drm-misc-next-2025-12-01-1/drivers/gpu/drm/ast/ast_cursor.c?ref_type=tags#L209
>>>> 
>>>>> + case DRM_FORMAT_RGB565:
>>>>> + ast_set_index_reg_mask(ast, AST_IO_VGACRI, AST_IO_VGACRA2, 0x3f,
>>>>> 0x40);
>>>>> + break;
>>>>> + case DRM_FORMAT_XRGB8888
>>> While working on it I discovered that the Big-Endian byte-swapping
>>> bits do apparently not just-work on a newer AST2400 in our Power 8
>>> while my initial patch did work as tested with an AST2200 in the Sun
>>> T4-1 :-/
> 
> In the upcoming v6.19-rc1, ast will support per-chip quirks. So we can control this by chip version, if necessary
> 
>>> 
>>> Maybe that is what Timothy meant with "This is due to a ppc64 hardware
>>> quirk, which when combined with a hardware design fault in the AST2500
>>> VGA controller results in a need to use software-based red-blue
>>> channel swapping." [1]
>>> 
>>> Is there a way to simply specify the frame-buffer as BGRX8888? In a
>>> quick test the drm layer complaint about "not supported" and "no
>>> compatible format found"?
>> I've been all around that loop.  You can't do that -- the fb code has no idea how to drive such a framebuffer, and elsewhere in the kernel it's made clear that the GPU driver *must* provide a RGBX8888 linear framebuffer if the Linux fb code is going to be able to display a console.
>> 
>> Does the Sun T4 CPU perform automatic byte swapping on PCI[e] data transactions?  That might be the difference; POWER performs the byte swapping, and since the ASpeed device is broken in BE mode we can't swap back by setting the BE register bit in the AST GPU hardware.
>> 
>> Fun fact -- it'll sorta work on the framebuffer side, but we lose the entire control BAR in the process.  ASpeed seems OK with this, they just say something along the lines of "oh, BE is not supported despite our documentation" :facepalm:
> 
> On the 2400-and-onwards models, ast could set drm_device.mode_config.quirk_addfb_prefer_host_byte_order. If set, the format lookup will select a different format on BE machines. [1] For example requesting XRGB8888 returns BGRX8888 instead. If ast later sees such a format in the atomic_update code, it could transparently swap bytes when writing out pixels to the video memory.  IIRC this works transparently to DRM clients and fbcon.  I think this would be the preferred way of fixing the issue.

Uff, I get better than nothing ;-)

> [1] https://elixir.bootlin.com/linux/v6.18/source/drivers/gpu/drm/drm_fourcc.c#L123
> 
> For the pre-2400 chips, I suggest to fix this problem with the hardware byte swapping if possible. That seems like the correct approach.

I had re-done the code as you suggested, should I send a v2 as tested on the sparc64 t4-1 and we quirk later, non working chips or ppc64 later?

	René

> Best regards
> Thomas

-- 
https://exactco.de • https://t2linux.com • https://patreon.com/renerebe