Message-ID: <aefe451c31d693cfe8f3cc157fd0009040fd48c7.camel@infradead.org>
Date: Thu, 24 Jul 2025 15:48:51 +0200
From: David Woodhouse <dwmw2@...radead.org>
To: "Rafael J. Wysocki" <rafael@...nel.org>
Cc: Pavel Machek <pavel@...nel.org>, linux-pm <linux-pm@...r.kernel.org>, 
 Marc Zyngier <maz@...nel.org>, linux-arm-kernel@...ts.infradead.org,
 "Saidi, Ali" <alisaidi@...zon.com>,  "oliver.upton"
 <oliver.upton@...ux.dev>, Joey Gouly <joey.gouly@....com>, Suzuki K Poulose
 <suzuki.poulose@....com>, Zenghui Yu <yuzenghui@...wei.com>, Catalin
 Marinas <catalin.marinas@....com>, Will Deacon <will@...nel.org>,
 linux-kernel <linux-kernel@...r.kernel.org>, "Heyne, Maximilian"
 <mheyne@...zon.de>,  Alexander Graf <graf@...zon.com>, "Stamatis, Ilias"
 <ilstam@...zon.com>
Subject: Re: Memory corruption after resume from hibernate with Arm GICv3 ITS

On Thu, 2025-07-24 at 11:51 +0200, Rafael J. Wysocki wrote:
> 
> > So the hibernated kernel seems to be doing the right thing in both
> > suspend and resume phases but it looks like the *boot* kernel doesn't
> > call the suspend method before transitioning;
> 
> No, it does this, but the messages are missing from the log.
> 
> The last message you see from the boot/restore kernel is about loading
> the image; a lot of stuff happens afterwards.
> 
> This message:
> 
> [    1.871617] PM: hibernation: Read 462616 kbytes in 0.47 seconds (984.28 MB/s)
> 
> is printed by load_compressed_image() which gets called by
> swsusp_read(), which is invoked by load_image_and_restore().
> 
> It is successful, so hibernation_restore() gets called and it does
> quite a bit of work, including calling resume_target_kernel(), which
> among other things calls syscore_suspend(), from where your messages
> should be printed if I'm not mistaken.
> 
> I have no idea why those messages don't get into the log (that would
> happen if your boot kernel were different from the image kernel and it
> didn't actually print them).

This is serial console output (stdout from 'qemu -serial mon:stdio'). I
guess the missing messages were in the printk buffer of the boot kernel
but just didn't get flushed? I added some --trace arguments to qemu to
see what's actually happening.

So when resuming, the boot looks like this:

gicv3_its_process_command GICv3 ITS: processing command at offset 0x4: 0x8
gicv3_its_cmd_mapd GICv3 ITS: command MAPD DeviceID 0x10 Size 0x6 ITT_addr 0x10f0a20 V 1
gicv3_its_dte_write GICv3 ITS: Device Table write for DeviceID 0x10: valid 1 size 0x6 ITTaddr 0x10f0a20
[   27.440351] its_build_mapd_cmd dev 0x10 valid 1 addr 0x10f0a2000

And then the transition goes:

[   47.668973] PM: Image loading progress:  90%
[   48.030462] PM: Image loading progress: 100%
[   48.031307] PM: Image loading done
[   48.031773] PM: hibernation: Read 460728 kbytes in 13.11 seconds (35.14 MB/s)
gicv3_its_cmd_mapd GICv3 ITS: command MAPD DeviceID 0x10 Size 0x6 ITT_addr 0x10f0a20 V 0
gicv3_its_dte_write GICv3 ITS: Device Table write for DeviceID 0x10: valid 0 size 0x6 ITTaddr 0x10f0a20
gicv3_its_cmd_mapd GICv3 ITS: command MAPD DeviceID 0x10 Size 0x6 ITT_addr 0x10e3130 V 1
gicv3_its_dte_write GICv3 ITS: Device Table write for DeviceID 0x10: valid 1 size 0x6 ITTaddr 0x10e3130
[  178.261284] Disabling non-boot CPUs ...
[  178.261674] its_save_disable
[  178.261674] its_build_mapd_cmd dev 0x10 valid 0 addr 0x10e313000
[  178.261674] PM: hibernation: Creating image:
[  178.261674] PM: hibernation: Need to copy 152532 pages
[  178.261674] hibernate: Restored 0 MTE pages
[  178.261674] its_restore_enable
[  178.261674] its_build_mapd_cmd dev 0x10 valid 1 addr 0x10e313000
[  178.831481] OOM killer enabled.
[  178.831614] Restarting tasks: Starting

So we don't see the *printk* from the boot kernel, as you said. But it
*is* unmapping from the old address (MAPD, ITT_addr 0x10f0a20, Valid 0)
before the resumed kernel does the map at the address *it* was using
(MAPD, ITT_addr 0x10e3130, Valid 1). Looking just at the MAPD traces:

gicv3_its_cmd_mapd GICv3 ITS: command MAPD DeviceID 0x10 Size 0x6 ITT_addr 0x10e3130 V 1  ← Original clean boot
gicv3_its_cmd_mapd GICv3 ITS: command MAPD DeviceID 0x10 Size 0x6 ITT_addr 0x10e3130 V 0  ← Prior to generating hibernate image
gicv3_its_cmd_mapd GICv3 ITS: command MAPD DeviceID 0x10 Size 0x6 ITT_addr 0x10e3130 V 1  ← Before *writing* hibernate image and powering down (actually reboot in this case)
gicv3_its_cmd_mapd GICv3 ITS: command MAPD DeviceID 0x10 Size 0x6 ITT_addr 0x10f0a20 V 1  ← Boot kernel starting up prior to resume
gicv3_its_cmd_mapd GICv3 ITS: command MAPD DeviceID 0x10 Size 0x6 ITT_addr 0x10f0a20 V 0  ← Boot kernel unmapping when we don't see its printk
gicv3_its_cmd_mapd GICv3 ITS: command MAPD DeviceID 0x10 Size 0x6 ITT_addr 0x10e3130 V 1  ← Hibernated kernel remapping the ITT

So it looks like my test patch is doing the right thing, at least for
hibernation? I'm not sure about kexec?

There are also *other* tables where the GIC scribbles on memory, for
pending interrupts for KVM guests (vLPI pending tables). We've had
problems with those too¹, causing machines to crash on kexec because
the GIC scribbles on pages which are *actually* now the new kernel's
text. I'm not sure if we should try to come up with a unified solution
for that or deal with them separately... the solution there seems to
involve iterating ∀ kvm ∀ vCPU so I suspect it does need to live in
KVM.

¹ https://lore.kernel.org/all/20250623132714.965474-2-dwmw2@infradead.org/

Download attachment "smime.p7s" of type "application/pkcs7-signature" (5069 bytes)

Powered by blists - more mailing lists