Message-ID: <14cf0b83-7030-4ec5-97b1-40c90424fe3f@zhaoxin.com>
Date: Thu, 25 Jul 2024 10:31:23 +0800
From: Tony W Wang-oc <TonyWWang-oc@...oxin.com>
To: <rafael@...nel.org>, <len.brown@...el.com>, <pavel@....cz>,
	<linux-pm@...r.kernel.org>, <akpm@...ux-foundation.org>,
	<dianders@...omium.org>, <tglx@...utronix.de>, <song@...nel.org>,
	<liusong@...ux.alibaba.com>, <yaoma@...ux.alibaba.com>,
	<kjlx@...pleofstupid.com>, <lizhe.67@...edance.com>, <linux@...ssschuh.net>,
	<j.granados@...sung.com>, <linux-kernel@...r.kernel.org>
CC: "CobeChen@...oxin.com" <CobeChen@...oxin.com>, "TimGuo@...oxin.com"
	<TimGuo@...oxin.com>, <SilviaZhao@...oxin.com>, "Linda Chai(BJ-RD)"
	<LindaChai@...oxin.com>, <Felixzhang@...oxin.com>
Subject: Unknown NMI after S4 resume

Hi,

When running S4 test on Zhaoxin platform with Ubuntu22.04 kernel-6.10 we 
got unknown NMI messages after S4 resumed:

[  115.792224] Uhhuh. NMI received for unknown reason 2d on CPU 0.
[  115.792226] Do you have a strange power saving mode enabled?
[  115.792228] Dazed and confused, but trying to continue

And reproduced on Intel platform.

After tracing, we find that the reason for this Unknown NMI occurs is as 
follows:
a, 1st kernel starts normally and NMI watchdog is enabled on all cores;
b, NMI watchdog is disabled on all cores through the sys interface, then 
variable active_events goto zero;
c, Start hibernate, create & save hibernation image, then go hibernated;
d, S4 resume event happened, 2nd kernel starts normally and NMI watchdog 
is enabled on All cores;
e, 2nd kernel find S4 image and try to restore S4 image;
f, 2nd kernel disable non-boot CPUs, which would disable NMI watchdog 
for APs;
g, Restore S4 image saved at step c;
h, 1st-hibernated kernel restore, re-enable non-boot CPUs, as NMI 
watchdog is disabled in step b, this which would keep APs NMI watchdog 
disabled; Besides, the variable active_events is restored to zero;

But BSP NMI watchdog is still enabled, and the PMC will trigger NMI 
interrupt periodically.
If PMC NMI triggered, perf_event_nmi_handler will be called, but it 
would see active_events is zero, so it goes out directly and return 
NMI_DONE;
This then leads to unknown NMI messages.

static int
perf_event_nmi_handler(unsigned int cmd, struct pt_regs *regs)
{
        u64 start_clock;
        u64 finish_clock;
        int ret;

        /*
        * All PMUs/events that share this PMI handler should make sure to
        * increment active_events for their events.
        */
        if (!atomic_read(&active_events))
               return NMI_DONE;
......

It seems that the BSP does not refer to the settings of the NMI watchdog 
sys interface previously saved to the S4 image to configure the NMI 
watchdog when doing S4 resume.
Should consider this situation and patch it?

---
Zhaoxin platform /proc/cpuinfo
processor	: 0
vendor_id	: CentaurHauls
cpu family	: 7
model		: 59
model name	: ZHAOXIN KaiXian KX-U6780A@...GHz
stepping	: 0
cpu MHz		: 2694.594
cache size	: 4096 KB
physical id	: 0
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov 
pat pse36 clflush acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb 
rdtscp lm constant_tsc arch_perfmon rep_good nopl nonstop_tsc cpuid 
aperfmperf pni pclmulqdq monitor vmx smx est tm2 ssse3 cx16 xtpr pcid 
sse4_1 sse4_2 movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand 
rng rng_en ace ace_en ace2 phe phe_en pmm pmm_en lahf_lm abm 
3dnowprefetch ibrs ibpb tpr_shadow ept vpid fsgsbase tsc_adjust bmi1 
smep bmi2 invpcid rdseed adx smap sha_ni xsaveopt dtherm vnmi umip pku 
ospke rdpid arch_capabilities
vmx flags	: vnmi invvpid ept_x_only ept_1gb tsc_offset vtpr mtf ept vpid 
unrestricted_guest ple
bugs		: spectre_v1
bogomips	: 5389.21
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

Intel platform /proc/cpuinfo
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 151
model name	: 12th Gen Intel(R) Core(TM) i7-12700K
stepping	: 2
microcode	: 0x35
cpu MHz		: 4653.842
cache size	: 25600 KB
physical id	: 0
siblings	: 20
core id		: 0
cpu cores	: 12
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 32
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov 
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx 
pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl 
xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq 
dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm 
sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c 
rdrand lahf_lm abm 3dnowprefetch cpuid_fault cat_l2 cdp_l2 ssbd ibrs 
ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad 
fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdt_a rdseed adx 
smap clflushopt clwb intel_pt sha_ni xsaveopt xsavec xgetbv1 xsaves 
split_lock_detect avx_vnni dtherm ida arat pln pts hwp hwp_notify 
hwp_act_window hwp_epp hwp_pkg_req hfi vnmi umip pku ospke waitpkg gfni 
vaes vpclmulqdq rdpid movdiri movdir64b fsrm md_clear serialize pconfig 
arch_lbr ibt flush_l1d arch_capabilities
vmx flags	: vnmi preemption_timer posted_intr invvpid ept_x_only ept_ad 
ept_1gb flexpriority apicv tsc_offset vtpr mtf vapic ept vpid 
unrestricted_guest vapic_reg vid ple shadow_vmcs ept_violation_ve 
ept_mode_based_exec tsc_scaling usr_wait_pause
bugs		: spectre_v1 spectre_v2 spec_store_bypass swapgs eibrs_pbrsb rfds bhi
bogomips	: 7219.20
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:

BRs!
TonyWWang-oc

Powered by blists - more mailing lists