Message-ID: 
 <CAGwozwErcHdA9UD4wr6xDow5C5+sh-D=1Y74fG-p=0E3en+BHg@mail.gmail.com>
Date: Tue, 26 Aug 2025 21:21:16 +0200
From: Antheas Kapenekakis <lkml@...heas.dev>
To: Mario Limonciello <superm1@...nel.org>
Cc: Alex Deucher <alexdeucher@...il.com>, amd-gfx@...ts.freedesktop.org,
	dri-devel@...ts.freedesktop.org, linux-kernel@...r.kernel.org,
	Alex Deucher <alexander.deucher@....com>,
 Christian König <christian.koenig@....com>,
	David Airlie <airlied@...il.com>, Simona Vetter <simona@...ll.ch>,
	Harry Wentland <harry.wentland@....com>,
 Rodrigo Siqueira <siqueira@...lia.com>,
	Mario Limonciello <mario.limonciello@....com>, Peyton Lee <peytolee@....com>,
 Lang Yu <lang.yu@....com>
Subject: Re: [PATCH v1 1/2] drm/amdgpu/vpe: increase VPE_IDLE_TIMEOUT to fix
 hang on Strix Halo

On Tue, 26 Aug 2025 at 21:19, Mario Limonciello <superm1@...nel.org> wrote:
>
> On 8/26/2025 8:41 AM, Alex Deucher wrote:
> > On Tue, Aug 26, 2025 at 3:49 AM Antheas Kapenekakis <lkml@...heas.dev> wrote:
> >>
> >> On Mon, 25 Aug 2025 at 03:38, Mario Limonciello <superm1@...nel.org> wrote:
> >>>
> >>>
> >>>
> >>> On 8/24/25 3:46 PM, Antheas Kapenekakis wrote:
> >>>> On Sun, 24 Aug 2025 at 22:16, Mario Limonciello <superm1@...nel.org> wrote:
> >>>>>
> >>>>>
> >>>>>
> >>>>> On 8/24/25 3:53 AM, Antheas Kapenekakis wrote:
> >>>>>> On the Asus Z13 2025, which uses a Strix Halo platform, around 8% of the
> >>>>>> suspend resumes result in a soft lock around 1 second after the screen
> >>>>>> turns on (it freezes). This happens due to power gating VPE when it is
> >>>>>> not used, which happens 1 second after inactivity.
> >>>>>>
> >>>>>> Specifically, the VPE gating after resume is as follows: an initial
> >>>>>> ungate, followed by a gate in the resume process. Then,
> >>>>>> amdgpu_device_delayed_init_work_handler with a delay of 2s is scheduled
> >>>>>> to run tests, one of which is testing VPE in vpe_ring_test_ib. This
> >>>>>> causes an ungate, After that test, vpe_idle_work_handler is scheduled
> >>>>>> with VPE_IDLE_TIMEOUT (1s).
> >>>>>>
> >>>>>> When vpe_idle_work_handler runs and tries to gate VPE, it causes the
> >>>>>> SMU to hang and partially freezes half of the GPU IPs, with the thread
> >>>>>> that called the command being stuck processing it.
> >>>>>>
> >>>>>> Specifically, after that SMU command tries to run, we get the following:
> >>>>>>
> >>>>>> snd_hda_intel 0000:c4:00.1: Refused to change power state from D0 to D3hot
> >>>>>> ...
> >>>>>> xhci_hcd 0000:c4:00.4: Refused to change power state from D0 to D3hot
> >>>>>> ...
> >>>>>> amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000
> >>>>>> amdgpu 0000:c4:00.0: amdgpu: Failed to power gate VPE!
> >>>>>> [drm:vpe_set_powergating_state [amdgpu]] *ERROR* Dpm disable vpe failed, ret = -62.
> >>>>>> amdgpu 0000:c4:00.0: [drm] *ERROR* [CRTC:93:crtc-0] flip_done timed out
> >>>>>> amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000
> >>>>>> amdgpu 0000:c4:00.0: amdgpu: Failed to power gate JPEG!
> >>>>>> [drm:jpeg_v4_0_5_set_powergating_state [amdgpu]] *ERROR* Dpm disable jpeg failed, ret = -62.
> >>>>>> amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000
> >>>>>> amdgpu 0000:c4:00.0: amdgpu: Failed to power gate VCN instance 0!
> >>>>>> [drm:vcn_v4_0_5_stop [amdgpu]] *ERROR* Dpm disable uvd failed, ret = -62.
> >>>>>> thunderbolt 0000:c6:00.5: 0: timeout reading config space 1 from 0xd3
> >>>>>> thunderbolt 0000:c6:00.5: 0: timeout reading config space 2 from 0x5
> >>>>>> thunderbolt 0000:c6:00.5: Refused to change power state from D0 to D3hot
> >>>>>> amdgpu 0000:c4:00.0: [drm] *ERROR* [CRTC:97:crtc-1] flip_done timed out
> >>>>>> amdgpu 0000:c4:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000032 SMN_C2PMSG_82:0x00000000
> >>>>>> amdgpu 0000:c4:00.0: amdgpu: Failed to power gate VCN instance 1!
> >>>>>>
> >>>>>> In addition to e.g., kwin errors in journalctl. 0000:c4.00.0 is the GPU.
> >>>>>> Interestingly, 0000:c4.00.6, which is another HDA block, 0000:c4.00.5,
> >>>>>> a PCI controller, and 0000:c4.00.2, resume normally. 0x00000032 is the
> >>>>>> PowerDownVpe(50) command which is the common failure point in all
> >>>>>> failed resumes.
> >>>>>>
> >>>>>> On a normal resume, we should get the following power gates:
> >>>>>> amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownVpe(50) param: 0x00000000, resp: 0x00000001
> >>>>>> amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownJpeg0(33) param: 0x00000000, resp: 0x00000001
> >>>>>> amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownJpeg1(38) param: 0x00010000, resp: 0x00000001
> >>>>>> amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownVcn1(4) param: 0x00010000, resp: 0x00000001
> >>>>>> amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerDownVcn0(6) param: 0x00000000, resp: 0x00000001
> >>>>>> amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpVcn0(7) param: 0x00000000, resp: 0x00000001
> >>>>>> amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpVcn1(5) param: 0x00010000, resp: 0x00000001
> >>>>>> amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpJpeg0(34) param: 0x00000000, resp: 0x00000001
> >>>>>> amdgpu 0000:c4:00.0: amdgpu: smu send message: PowerUpJpeg1(39) param: 0x00010000, resp: 0x00000001
> >>>>>>
> >>>>>> To fix this, increase VPE_IDLE_TIMEOUT to 2 seconds. This increases
> >>>>>> reliability from 4-25 suspends to 200+ (tested) suspends with a cycle
> >>>>>> time of 12s sleep, 8s resume.
> >>>>>
> >>>>> When you say you reproduced with 12s sleep and 8s resume, was that
> >>>>> 'amd-s2idle --duration 12 --wait 8'?
> >>>>
> >>>> I did not use amd-s2idle. I essentially used the script below with a
> >>>> 12 on the wake alarm and 12 on the for loop. I also used pstore for
> >>>> this testing.
> >>>>
> >>>> for i in {1..200}; do
> >>>>     echo "Suspend attempt $i"
> >>>>     echo `date '+%s' -d '+ 60 seconds'` | sudo tee /sys/class/rtc/rtc0/wakealarm
> >>>>     sudo sh -c 'echo mem > /sys/power/state'
> >>>>
> >>>>     for j in {1..50}; do
> >>>>       # Use repeating sleep in case echo mem returns early
> >>>>       sleep 1
> >>>>     done
> >>>> done
> >>>
> >>> 👍
> >>>
> >>>>
> >>>>>> The suspected reason here is that 1s that
> >>>>>> when VPE is used, it needs a bit of time before it can be gated and
> >>>>>> there was a borderline delay before, which is not enough for Strix Halo.
> >>>>>> When the VPE is not used, such as on resume, gating it instantly does
> >>>>>> not seem to cause issues.
> >>>>>>
> >>>>>> Fixes: 5f82a0c90cca ("drm/amdgpu/vpe: enable vpe dpm")
> >>>>>> Signed-off-by: Antheas Kapenekakis <lkml@...heas.dev>
> >>>>>> ---
> >>>>>>     drivers/gpu/drm/amd/amdgpu/amdgpu_vpe.c | 4 ++--
> >>>>>>     1 file changed, 2 insertions(+), 2 deletions(-)
> >>>>>>
> >>>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vpe.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vpe.c
> >>>>>> index 121ee17b522b..24f09e457352 100644
> >>>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vpe.c
> >>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vpe.c
> >>>>>> @@ -34,8 +34,8 @@
> >>>>>>     /* VPE CSA resides in the 4th page of CSA */
> >>>>>>     #define AMDGPU_CSA_VPE_OFFSET       (4096 * 3)
> >>>>>>
> >>>>>> -/* 1 second timeout */
> >>>>>> -#define VPE_IDLE_TIMEOUT     msecs_to_jiffies(1000)
> >>>>>> +/* 2 second timeout */
> >>>>>> +#define VPE_IDLE_TIMEOUT     msecs_to_jiffies(2000)
> >>>>>>
> >>>>>>     #define VPE_MAX_DPM_LEVEL                   4
> >>>>>>     #define FIXED1_8_BITS_PER_FRACTIONAL_PART   8
> >>>>>>
> >>>>>> base-commit: c17b750b3ad9f45f2b6f7e6f7f4679844244f0b9
> >>>>>
> >>>>> 1s idle timeout has been used by other IPs for a long time.
> >>>>> For example JPEG, UVD, VCN all use 1s.
> >>>>>
> >>>>> Can you please confirm both your AGESA and your SMU firmware version?
> >>>>> In case you're not aware; you can get AGESA version from SMBIOS string
> >>>>> (DMI type 40).
> >>>>>
> >>>>> ❯ sudo dmidecode | grep AGESA
> >>>>
> >>>> String: AGESA!V9 StrixHaloPI-FP11 1.0.0.0c
> >>>>
> >>>>> You can get SMU firmware version from this:
> >>>>>
> >>>>> ❯ grep . /sys/bus/platform/drivers/amd_pmc/*/smu_*
> >>>>
> >>>> grep . /sys/bus/platform/drivers/amd_pmc/*/smu_*
> >>>> /sys/bus/platform/drivers/amd_pmc/AMDI000B:00/smu_fw_version:100.112.0
> >>>> /sys/bus/platform/drivers/amd_pmc/AMDI000B:00/smu_program:0
> >>>>
> >>>
> >>> Thanks, I'll get some folks to see if we match this AGESA version if we
> >>> can also reproduce it on reference hardware the same way you did.
> >>>
> >>>>> Are you on the most up to date firmware for your system from the
> >>>>> manufacturer?
> >>>>
> >>>> I updated my bios, pd firmware, and USB device firmware early August,
> >>>> when I was doing this testing.
> >>>>
> >>>>> We haven't seen anything like this reported on Strix Halo thus far and
> >>>>> we do internal stress testing on s0i3 on reference hardware.
> >>>>
> >>>> Cant find a reference for it on the bug tracker. I have four bug
> >>>> reports on the bazzite issue tracker, 2 about sleep wake crashes and 2
> >>>> for runtime crashes, where the culprit would be this. IE runtime gates
> >>>> VPE and causes a crash.
> >>>
> >>> All on Strix Halo and all tied to VPE?  At runtime was VPE in use?  By
> >>> what software?
> >>>
> >>> BTW - Strix and Kraken also have VPE.
> >>
> >> All on the Z13. Not tied to VPE necessarily. I just know that I get
> >> reports of crashes on the Z13, and with this patch they are fixed for
> >> me. It will be part of the next bazzite version so I will get feedback
> >> about it.
> >>
> >> I don't think software that is using the VPE is relevant. Perhaps for
> >> the runtime crashes it is and this patch helps in that case as well.
> >> But in my case, the crash is caused after the ungate that runs the
> >> tests on resume on the delayed handler.
> >>
> >> The Z13 also has some other quirks with spurious wakeups when
> >> connected to a charger. So, if systemd is configured to e.g., sleep
> >> after 20m, combined with this crash if it stays plugged in overnight
> >> in the morning it has crashed.
> >>
> >>>>
> >>>>> To me this seems likely to be a platform firmware bug; but I would like
> >>>>> to understand the timing of the gate vs ungate on good vs bad.
> >>>>
> >>>> Perhaps it is. It is either something like that or silicon quality.
> >>>>
> >>>>> IE is it possible the delayed work handler
> >>>>> amdgpu_device_delayed_init_work_handler() is causing a race with
> >>>>> vpe_ring_begin_use()?
> >>>>
> >>>> I don't think so. There is only a single ungate. Also, the crash
> >>>> happens on the gate. So what happens is the device wakes up, the
> >>>> screen turns on, kde clock works, then after a second it freezes,
> >>>> there is a softlock, and the device hangs.
> >>>>
> >>>> The failed command is always the VPE gate that is triggered after 1s in idle.
> >>>>
> >>>>> This should be possible to check without extra instrumentation by using
> >>>>> ftrace and looking at the timing of the 2 ring functions and the init
> >>>>> work handler and checking good vs bad cycles.
> >>>>
> >>>> I do not know how to use ftrace. I should also note that after the
> >>>> device freezes around 1/5 cycles will sync the fs, so it is also not a
> >>>> very easy thing to diagnose. The device just stops working. A lot of
> >>>> the logs I got were in pstore by forcing a kernel panic.
> >>>
> >>> Here's how you capture the timing of functions.  Each time the function
> >>> is called there will be an event in the trace buffer.
> >>>
> >>> ❯ sudo trace-cmd record -p function -l
> >>> amdgpu_device_delayed_init_work_handler -l vpe_idle_work_handler -l
> >>> vpe_ring_begin_use -l vpe_ring_end_use -l amdgpu_pmops_suspend -l
> >>> amdgpu_pmops_resume
> >>>
> >>> Here's how you would review the report:
> >>>
> >>> ❯ trace-cmd report
> >>> cpus=24
> >>>     kworker/u97:37-18051 [001] ..... 13655.970108: function:
> >>> amdgpu_pmops_suspend <-- pci_pm_suspend
> >>>     kworker/u97:21-18036 [002] ..... 13666.290715: function:
> >>> amdgpu_pmops_resume <-- dpm_run_callback
> >>>     kworker/u97:21-18036 [015] ..... 13666.308295: function:
> >>> vpe_ring_begin_use <-- amdgpu_ring_alloc
> >>>     kworker/u97:21-18036 [015] ..... 13666.308298: function:
> >>> vpe_ring_end_use <-- vpe_ring_test_ring
> >>>       kworker/15:1-12285 [015] ..... 13666.960191: function:
> >>> amdgpu_device_delayed_init_work_handler <-- process_one_work
> >>>       kworker/15:1-12285 [015] ..... 13666.963970: function:
> >>> vpe_ring_begin_use <-- amdgpu_ring_alloc
> >>>       kworker/15:1-12285 [015] ..... 13666.965481: function:
> >>> vpe_ring_end_use <-- amdgpu_ib_schedule
> >>>       kworker/15:4-16354 [015] ..... 13667.981394: function:
> >>> vpe_idle_work_handler <-- process_one_work
> >>>
> >>> I did this on a Strix system just now to capture that.
> >>>
> >>> You can see that basically the ring gets used before the delayed init
> >>> work handler, and then again from the ring tests.  My concern is if the
> >>> sequence ever looks different than the above.  If it does; we do have a
> >>> driver race condition.
> >>>
> >>> It would also be helpful to look at the function_graph tracer.
> >>>
> >>> Here's some more documentation about ftrace and trace-cmd.
> >>> https://www.kernel.org/doc/html/latest/trace/ftrace.html
> >>> https://lwn.net/Articles/410200/
> >>>
> >>> You can probably also get an LLM to help you with building commands if
> >>> you're not familiar with it.
> >>>
> >>> But if you're hung so bad you can't flush to disk that's going to be a
> >>> problem without a UART.  A few ideas:
> >>
> >> Some times it flushes to disk
> >>
> >>> 1) You can use CONFIG_PSTORE_FTRACE
> >>
> >> I can look into that
> >>
> >>> 2) If you add "tp_printk" to the kernel command line it should make the
> >>> trace ring buffer flush to kernel log ring buffer.  But be warned this
> >>> is going to change the timing, the issue might go away entirely or have
> >>> a different failure rate.  So hopefully <1> works.
> >>>>
> >>>> If you say that all IP blocks use 1s, perhaps an alternative solution
> >>>> would be to desync the idle times so they do not happen
> >>>> simultaneously. So 1000, 1200, 1400, etc.
> >>>>
> >>>> Antheas
> >>>>
> >>>
> >>> I don't dobut your your proposal of changing the timing works.  I just
> >>> want to make sure it's the right solution because otherwise we might
> >>> change the timing or sequence elsewhere in the driver two years from now
> >>> and re-introduce the problem unintentionally.
> >>
> >> If there are other idle timers and only this one changes to 2s, I will
> >> agree and say that it would be peculiar. Although 1s seems arbitrary
> >> in any case.
> >
> > All of these timers are arbitrary.  Their point is just to provide a
> > future point where we can check if the engine is idle.  The idle work
> > handler will either power down the IP if it is idle or re-schedule in
> > the future and try again if there is still work.  Making the value
> > longer will use more power as it will wait longer before checking if
> > the engine is idle.  Making it shorter will save more power, but adds
> > extra overhead in that the engine will be powered up/down more often.
> > In most cases, the jobs should complete in a few ms.  The timer is
> > there to avoid the overhead of powering up/down the block too
> > frequently when applications are using the engine.
> >
> > Alex
>
> We had a try internally with both 6.17-rc2 and 6.17-rc3 and 1001b or
> 1001c AGESA on reference system but unfortunately didn't reproduce the
> issue with a 200 cycle attempt on either kernel or either BIOS (so we
> had 800 cycles total).

I think I did 6.12, 6.15, and a 6.16rc stock. I will have to come back
to you with 6.17-rc3.

> Was your base a bazzite kernel or was it an upstream kernel?  I know
> there are some other patches in bazzite especially relevant to suspend,
> so I wonder if they could be influencing the timing.
>
> Can you repo on 6.17-rc3?
>

Powered by blists - more mailing lists