linux-kernel - Re: [PATCH] drm/ast: Fix big-endian support

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <48b1bcea-05ff-469c-95d8-de094e68b4fc@suse.de>
Date: Mon, 15 Dec 2025 08:27:56 +0100
From: Thomas Zimmermann <tzimmermann@...e.de>
To: René Rebe <rene@...ctco.de>
Cc: tpearson@...torengineering.com, dri-devel@...ts.freedesktop.org,
 linux-kernel@...r.kernel.org, airlied@...hat.com
Subject: Re: [PATCH] drm/ast: Fix big-endian support

Hi

Am 12.12.25 um 21:15 schrieb René Rebe:
> Hi,
>
> On Thu, 11 Dec 2025 15:31:01 +0100 (CET), René Rebe <rene@...ctco.de> wrote:
>
>>>   	/* write checksum + signature */
>>> +	writel(swab32(csum), dst);
>>> +	writel(swab32(width), dst + AST_HWC_SIGNATURE_SizeX);
>>> +	writel(swab32(height), dst + AST_HWC_SIGNATURE_SizeY);
>>> +	writel(swab32(0), dst + AST_HWC_SIGNATURE_HOTSPOTX);
>>> +	writel(swab32(0), dst + AST_HWC_SIGNATURE_HOTSPOTY);
>>> +#else
>>> +	memcpy_toio(dst, src, AST_HWC_SIZE);
>>>   	dst += AST_HWC_SIZE;
>>> +
>>> +	/* write checksum + signature */
>>>   	writel(csum, dst);
>>>   	writel(width, dst + AST_HWC_SIGNATURE_SizeX);
>>>   	writel(height, dst + AST_HWC_SIGNATURE_SizeY);
>>>   	writel(0, dst + AST_HWC_SIGNATURE_HOTSPOTX);
>>>   	writel(0, dst + AST_HWC_SIGNATURE_HOTSPOTY);
>>> +#endif
>> I'm pretty sure this will break the cursor, as the position was
>> working correctly and I only had to swap the cursor image data. The
>> csum will also not be identical anyway, as the checksum function
>> computes it in native byte order. Theoretically that would have to be
>> changed. However, I do not see where it is really used, maybe only
>> some special remote desktop vendor protocol that I'm not using. Maybe
>> the exact checksum does not even matter and is only used as
>> optimization to not resend an unchanged cursor image.
>>
>> I'll send a final version after validating it w/ HW later.
> I just sent a more minimally tested V4 removing the superflous unused
> fmt_cnv_state to ast_set_cursor_image you somehow had added.

That is a leftover form earlier drm_fb_swab() code.

>
> As the additional writen cursor RDP service writes are untested, I
> left them out.
>
> There is still something suspect or buggy: the 2-bit X11 cursor is
> filled with transparency, while the ARGB RGB channel work now. Modern
> ARGB Xcursor theme also look strange. Not sure if that is just due to
> loosing 4-bit precision and thus half of the dynamic range with all
> the shadows. To me at least the 2-bit transparent X cursor looks like
> a fmt conversion bug in some layer that we would need to continue
> debugging another day. At least the framebuffer / installer text would
> be more readble upstream now, too ;-)

The kernel's ast driver doesn't support 2-bit cursors. It's likely a 
problem somewhere in Xorg or the rendering libraries.

Best regards
Thomas

>
> Thanks,
> 	René
>

-- 
--
Thomas Zimmermann
Graphics Driver Developer
SUSE Software Solutions Germany GmbH
Frankenstr. 146, 90461 Nürnberg, Germany, www.suse.com
GF: Jochen Jaser, Andrew McDonald, Werner Knoblich, (HRB 36809, AG Nürnberg)