linux-kernel - Re: 6.11/regression/bisected - after commit 1b04dcca4fb1, launching some RenPy games causes computer hang

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <eeab54b4-c055-4992-9ca4-f9e382db68c4@amd.com>
Date: Tue, 10 Sep 2024 11:47:48 -0400
From: Leo Li <sunpeng.li@....com>
To: Mikhail Gavrilov <mikhail.v.gavrilov@...il.com>
Cc: Harry Wentland <harry.wentland@....com>, zaeem.mohamed@....com,
 pekka.paalanen@...labora.com, "Wheeler, Daniel" <daniel.wheeler@....com>,
 "Deucher, Alexander" <alexander.deucher@....com>,
 amd-gfx list <amd-gfx@...ts.freedesktop.org>,
 dri-devel <dri-devel@...ts.freedesktop.org>,
 Linux List Kernel Mailing <linux-kernel@...r.kernel.org>,
 Linux regressions mailing list <regressions@...ts.linux.dev>
Subject: Re: 6.11/regression/bisected - after commit 1b04dcca4fb1, launching
 some RenPy games causes computer hang

On 2024-09-08 19:30, Mikhail Gavrilov wrote:
> I have done additional tests:
> 1. The computer does not hang with 6900XT instead the screen flickers
> when moving the cursor.
> 2. The computer does not hang with 7900XTX if I turn off VRR. But the
> screen flickers when moving the cursor, as on 6900XT.
> To enable VRR, please set 'variable-refresh-rate' in
> experimental-features, and in the Display setting, enable Variable
> Refresh Rate.
> $ gsettings set org.gnome.mutter experimental-features
> "['variable-refresh-rate', 'scale-monitor-framebuffer']"
> https://postimg.cc/PvXYdvGR

Thanks Mikhail, I think I know what's going on now.

The `scale-monitor-framebuffer` experimental setting is what puts us down the
bad code path. It seems VRR has nothing to do with this issue, just setting
`scale-monitor-framebuffer` is enough to reproduce.

It seems that mutter with this setting is opting for HW scaling rather than GPU
scaling. I see that "Find the Orange Narwhal" sends out a 1080p buffer,
which with this setting, gets directly scanned out and scaled by DCN HW to 4k in
full screen.

An oddity with current gen DCN hardware is that the cursor inherits the scaling
of the HW plane underneath. So if mutter requests a hw cursor with a different
scaling than the game's plane, amdgpu will reject that, and likely force mutter
into SW cursor.

My offending patch changed this behavior by rerouting DCN HW pipes to
accommodate such a configuration. It essentially takes a full-fledged DCN
overlay plane, and uses that just for the cursor, and thereby freeing it from
inheriting things from the underlying hw plane.

My guess is this causes flickering due to how DC (display core driver) handles
updates; it needs all enabled planes in it's update state. However, a KMS cursor
update will only include the cursor plane. It's likely that amdgpu_dm only adds
the dedicated cursor plane to DC's update state, leaving the game's plane out.

The fix isn't exactly trivial. If I don't get anywhere before the fixes window,
I'll send out a revert.

Cheers,
Leo