linux-kernel - Re: [PATCH] drm/fbdev-generic: Fix framebuffer on big endian devices

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date: Fri, 28 Jun 2024 09:03:50 +0200
From: Geert Uytterhoeven <geert@...ux-m68k.org>
To: Thomas Huth <thuth@...hat.com>
Cc: dri-devel@...ts.freedesktop.org, 
	Maarten Lankhorst <maarten.lankhorst@...ux.intel.com>, Maxime Ripard <mripard@...nel.org>, 
	Thomas Zimmermann <tzimmermann@...e.de>, David Airlie <airlied@...il.com>, Daniel Vetter <daniel@...ll.ch>, 
	linux-kernel@...r.kernel.org, Javier Martinez Canillas <javierm@...hat.com>, 
	Hamza Mahfooz <hamza.mahfooz@....com>, Christian Zigotzky <chzigotzky@...osoft.de>
Subject: Re: [PATCH] drm/fbdev-generic: Fix framebuffer on big endian devices

Hi Thomas,

CC Christian

On Thu, Jun 27, 2024 at 7:35 PM Thomas Huth <thuth@...hat.com> wrote:
> Starting with kernel 6.7, the framebuffer text console is not working
> anymore with the virtio-gpu device on s390x hosts. Such big endian fb
> devices are usinga different pixel ordering than little endian devices,
> e.g. DRM_FORMAT_BGRX8888 instead of DRM_FORMAT_XRGB8888.
>
> This used to work fine as long as drm_client_buffer_addfb() was still
> calling drm_mode_addfb() which called drm_driver_legacy_fb_format()
> internally to get the right format. But drm_client_buffer_addfb() has
> recently been reworked to call drm_mode_addfb2() instead with the
> format value that has been passed to it as a parameter (see commit
> 6ae2ff23aa43 ("drm/client: Convert drm_client_buffer_addfb() to drm_mode_addfb2()").
>
> That format parameter is determined in drm_fbdev_generic_helper_fb_probe()
> via the drm_mode_legacy_fb_format() function - which only generates
> formats suitable for little endian devices. So to fix this issue
> switch to drm_driver_legacy_fb_format() here instead to take the
> device endianness into consideration.
>
> Fixes: 6ae2ff23aa43 ("drm/client: Convert drm_client_buffer_addfb() to drm_mode_addfb2()")
> Closes: https://issues.redhat.com/browse/RHEL-45158
> Signed-off-by: Thomas Huth <thuth@...hat.com>

Reviewed-by: Geert Uytterhoeven <geert@...ux-m68k.org>
works fine on m68k-virt, so:
Tested-by: Geert Uytterhoeven <geert@...ux-m68k.org>

Christian had reported a similar issue before[1].
I submitted a different solution fixing virtio[2] instead, but that
caused issues with virtio-mouse-pci cursor...

> --- a/drivers/gpu/drm/drm_fbdev_generic.c
> +++ b/drivers/gpu/drm/drm_fbdev_generic.c
> @@ -84,7 +84,8 @@ static int drm_fbdev_generic_helper_fb_probe(struct drm_fb_helper *fb_helper,
>                     sizes->surface_width, sizes->surface_height,
>                     sizes->surface_bpp);
>
> -       format = drm_mode_legacy_fb_format(sizes->surface_bpp, sizes->surface_depth);
> +       format = drm_driver_legacy_fb_format(dev, sizes->surface_bpp,
> +                                            sizes->surface_depth);
>         buffer = drm_client_framebuffer_create(client, sizes->surface_width,
>                                                sizes->surface_height, format);
>         if (IS_ERR(buffer))

[1] https://lore.kernel.org/6530cea3-4507-454e-bc36-a6970c8e7578@xenosoft.de/
[2] "[PATCH v2] drm/virtio: Add suppport for non-native buffer formats"
    https://lore.kernel.org/47a81d2e0e47b1715718779b6978a8b595cc7c5d.1700140609.git.geert@linux-m68k.org

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@...ux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds