linux-kernel - Re: [PATCH] drm/ast: Fix big-endian support

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <34cce8e3-51c5-4d44-8f6e-592a5943aec8@suse.de>
Date: Thu, 11 Dec 2025 08:22:12 +0100
From: Thomas Zimmermann <tzimmermann@...e.de>
To: René Rebe <rene@...ctco.de>
Cc: tpearson@...torengineering.com, dri-devel@...ts.freedesktop.org,
 linux-kernel@...r.kernel.org, airlied@...hat.com
Subject: Re: [PATCH] drm/ast: Fix big-endian support

Hi

Am 10.12.25 um 17:56 schrieb René Rebe:
> Hi,
>
> On Wed, 10 Dec 2025 16:41:50 +0100, Thomas Zimmermann <tzimmermann@...e.de> wrote:
>
>> Hi
>>
>> Am 10.12.25 um 16:33 schrieb René Rebe:
>>> Hallo,
>>>
>>> On Wed, 10 Dec 2025 09:55:50 +0100, Thomas Zimmermann
>>> <tzimmermann@...e.de> wrote:
>>> ...
>>>>>> Does that fix the color corruption?
>>>>> Following your suggestions conversion does not want to just work:
>>>>>
>>>>> root@...DE_SPARC_T4_1:~# dmesg  | tail
>>>>> [  105.444761] ast 0000:0a:00.0: AST 2200 detected
>>>>> [ 105.444947] ast 0000:0a:00.0: [drm] dram MCLK=266 Mhz type=2
>>>>> bus_width=32
>>>>> [  105.444963] ast 0000:0a:00.0: [drm] Using analog VGA
>>>>> [  105.445470] [drm] Initialized ast 0.1.0 for 0000:0a:00.0 on minor 0
>>>>> [ 105.673289] ast 0000:0a:00.0: [drm] format BX24 little-endian
>>>>> (0x34325842) not supported
>>>>> [  105.673302] ast 0000:0a:00.0: [drm] No compatible format found
>>>>> [ 105.673348] ast 0000:0a:00.0: [drm] *ERROR* fbdev: Failed to setup
>>>>> emulation (ret=-22)
>>>>> [ 105.901306] ast 0000:0a:00.0: [drm] format BX24 little-endian
>>>>> (0x34325842) not supported
>>>>> [  105.901319] ast 0000:0a:00.0: [drm] No compatible format found
>>>>> [ 105.901350] ast 0000:0a:00.0: [drm] *ERROR* fbdev: Failed to setup
>>>>> emulation (ret=-22)
>>>> Oh well...
>>>>
>>>> There's a very simple patch attach. Does it fix the problem?
>>> Yes, only leaving the hardcoded swapping from my patch liek this fixes
>>> the byte-swapped output as expected on the sparc64 Sun T4.
>> Great.
>>
>>> How would you like me to go from here? Just use the chip_id to force
>>> swapping and enable hw swapper for pre-AST2400 chips or fix the
>>> generic format selection to work as you had suggested?
>>>
>>> Does the ast_primary_plane_formats need to byte swapped formats for it
>>> to work?
>> I'll send out a full patch that implements the byte swapping. Once
>> reviewed, it can be merged quickly. Can I add your Tested-by tag to
>> the patch?
> I'd be happy to finish my work.

Of course, no problem.

The code for the primary plane should be fine now. But we also need 
something for the cursor plane as well. There's a ast_set_cursor_image() 
with a memcpy_toio() [1] and several additional writes. IIUC they all 
have to be swapped as well.

[1] 
https://gitlab.freedesktop.org/drm/misc/kernel/-/blob/drm-misc-next/drivers/gpu/drm/ast/ast_cursor.c#L101

Best regards
Thomas

>   But if you want to put the last touch
> on it now you can add Co-developed-by, too ... and I'll test the final
> version.
>
>     René
>

-- 
--
Thomas Zimmermann
Graphics Driver Developer
SUSE Software Solutions Germany GmbH
Frankenstr. 146, 90461 Nürnberg, Germany, www.suse.com
GF: Jochen Jaser, Andrew McDonald, Werner Knoblich, (HRB 36809, AG Nürnberg)