Message-Id: <20260204-macbook-pro-mtrr-resume-fix-v1-1-25fc4326d06a@paulson-ellis.org>
Date: Wed, 04 Feb 2026 16:15:18 +0000
From: Chris Paulson-Ellis <chris@...lson-ellis.org>
To: Thomas Gleixner <tglx@...nel.org>, Ingo Molnar <mingo@...hat.com>, 
 Borislav Petkov <bp@...en8.de>, Dave Hansen <dave.hansen@...ux.intel.com>, 
 x86@...nel.org, "H. Peter Anvin" <hpa@...or.com>
Cc: linux-kernel@...r.kernel.org, Juergen Gross <jgross@...e.com>, 
 Chris Paulson-Ellis <chris@...lson-ellis.org>
Subject: [PATCH] x86/mtrr: Expedite cache init if MTRRs change on resume

An Apple MacBook Pro mid-2015 almost always takes ~20s to resume from
suspend, measured from lid open to display of the desktop lock screen.

This is due to the MTRRs being different between suspend and resume on 4
out of the 8 CPUs, including the boot CPU. The CPUs execute very slowly
until arch_thaw_secondary_cpus_end() calls cache_aps_init(), which
restores the MTRR settings to their pre-suspend values.

To obtain a reasonable resume time, we need to minimise the time the
CPUs execute with inconsistent MTRR settings.

We do this by detecting the unexpected restore of the MTRRs on the boot
CPU in cache_bp_restore(), and use this to override the delay of cache
initialisation in the cache_ap_online() CPU hotplug handler, and
skip the delayed cache initialisation in cache_aps_init().

With this fix, the system in question resumes in ~3s.

---
As per the commit message, this patch is about making my mid-2015
MacBook Pro resume from suspend in a reasonable amount of time (3s
rather that 20s).

My whole investigative journey is documented in [1]. There is a lot
there that will not be of interest, but some if the details may be, such
as the observed MTRR values on resume [2].

The Apple BIOS seems to be buggy with respect to CPU MTRR settings. Here
is what I've observed:

 * The MTRRs are always sane & consistent on boot.
 * The MTRRs are unchanged by any of the Linux software I run.
 * After suspending, on resume the boot processor MTRRs have usually
   been modified [2] - I'm assuming by the BIOS.
 * The MTRR modifications are the same on every suspend/resume cycle.
 * Both the fixed & variable MTRRs change.
 * When Linux brings up the secondary processors on resume, 3 of them
   with have the same modified MTRRs as the BP, and 4 will have the
   original MTRRs. There are 8 CPUs in total, so half are OK, and half
   have changed.
 * The AP MTRRs that are modified have the same fixed & variable MTRR
   changes as on the BP.
 * On the BP only, the default MTTR type is changed from writeback to
   uncachable.
 * For a single boot, every suspend/resume cycle will see the same MTRR
   modifications on the same list of APs.
 * After rebooting, the list of which specific 3/7 APs have the MTRR
   modifications may change.
 * I've never observed modified MTRRs on APs without also observing it
   on the BP.
 * On some (rare) boots the problem is not observed at all, regardless
   of how many suspend/resume cycles are performed.
 * FWIW I never observed extended resume times running MacOS X for many
   years. I only switched to running Linux natively when it went EOL.

The original MTRRs settings are restored by Linux and the system is
fine, but this doesn't happen until after all the CPUs are brought up.
While this is happening the CPUs have inconsistent MTRR settings, and
the system runs very slowly, causing the long resume times.

I have observed variation in the extended resume times [3], which is
probaby caused not by which Linux version is running (which is what I
was looking for), but by the (apparently random) differences in which
specific APs have modified MTRRs, and therefore how long the system runs
with inconsistent MTRR settings.

My solution to the problem is to detect the MTRR modifications on the
BP, and then expedite restoration of the MTRRs on the APs.

A downside of this solution is that many more synchronised MTRR checks
are performed [4], which could impact systems with many CPUs.

I therefore have some questions about my fix:

Are there systems were is it normal for the BP to have different MTRR
settings between suspend and resume? If so, the new behaviour would be
triggered on these systems, and resume times could be affected if they
have many CPUs.

Should I therefore add some additional condition on the new behaviour?
Such as:

 * Somehow identifying my specific BIOS.
 * Limiting it to systems with <= n CPUs (where n is 8 for my system).
 * Requiring some runtime config to enable it, such as a boot parameter.

[1] https://macbook-pro-resume-delay.pages.dev/
[2] https://macbook-pro-resume-delay.pages.dev/journey#changes-after-resume
[3] https://macbook-pro-resume-delay.pages.dev/journey#id3
[4] https://macbook-pro-resume-delay.pages.dev/journey#cost-of-extra-checks

Signed-off-by: Chris Paulson-Ellis <chris@...lson-ellis.org>
---
 arch/x86/include/asm/mtrr.h        |  7 +++++--
 arch/x86/kernel/cpu/cacheinfo.c    | 33 ++++++++++++++++++++++++++-------
 arch/x86/kernel/cpu/mtrr/generic.c |  6 +++++-
 3 files changed, 36 insertions(+), 10 deletions(-)

diff --git a/arch/x86/include/asm/mtrr.h b/arch/x86/include/asm/mtrr.h
index 76b95bd1a405..bc5872e57f3c 100644
--- a/arch/x86/include/asm/mtrr.h
+++ b/arch/x86/include/asm/mtrr.h
@@ -60,7 +60,7 @@ extern int mtrr_trim_uncached_memory(unsigned long end_pfn);
 extern int amd_special_default_mtrr(void);
 void mtrr_disable(void);
 void mtrr_enable(void);
-void mtrr_generic_set_state(void);
+bool mtrr_generic_set_state(void);
 #  else
 static inline void guest_force_mtrr_state(struct mtrr_var_range *var,
 					  unsigned int num_var,
@@ -105,7 +105,10 @@ static inline int mtrr_trim_uncached_memory(unsigned long end_pfn)
 #define mtrr_bp_init() do {} while (0)
 #define mtrr_disable() do {} while (0)
 #define mtrr_enable() do {} while (0)
-#define mtrr_generic_set_state() do {} while (0)
+static inline bool mtrr_generic_set_state(void)
+{
+	return false;
+}
 #  endif
 
 #ifdef CONFIG_COMPAT
diff --git a/arch/x86/kernel/cpu/cacheinfo.c b/arch/x86/kernel/cpu/cacheinfo.c
index 51a95b07831f..7911beffc339 100644
--- a/arch/x86/kernel/cpu/cacheinfo.c
+++ b/arch/x86/kernel/cpu/cacheinfo.c
@@ -710,15 +710,16 @@ void cache_enable(void) __releases(cache_disable_lock)
 	raw_spin_unlock(&cache_disable_lock);
 }
 
-static void cache_cpu_init(void)
+static bool cache_cpu_init(void)
 {
 	unsigned long flags;
+	bool changed = false;
 
 	local_irq_save(flags);
 
 	if (memory_caching_control & CACHE_MTRR) {
 		cache_disable();
-		mtrr_generic_set_state();
+		changed = mtrr_generic_set_state();
 		cache_enable();
 	}
 
@@ -726,6 +727,8 @@ static void cache_cpu_init(void)
 		pat_cpu_init();
 
 	local_irq_restore(flags);
+
+	return changed;
 }
 
 static bool cache_aps_delayed_init = true;
@@ -743,7 +746,7 @@ bool get_cache_aps_delayed_init(void)
 static int cache_rendezvous_handler(void *unused)
 {
 	if (get_cache_aps_delayed_init() || !cpu_online(smp_processor_id()))
-		cache_cpu_init();
+		(void)cache_cpu_init();
 
 	return 0;
 }
@@ -754,20 +757,31 @@ void __init cache_bp_init(void)
 	pat_bp_init();
 
 	if (memory_caching_control)
-		cache_cpu_init();
+		(void)cache_cpu_init();
 }
 
+static bool cache_bp_changed_on_restore;
+
 void cache_bp_restore(void)
 {
 	if (memory_caching_control)
-		cache_cpu_init();
+		cache_bp_changed_on_restore = cache_cpu_init();
 }
 
 static int cache_ap_online(unsigned int cpu)
 {
 	cpumask_set_cpu(cpu, cpu_cacheinfo_mask);
 
-	if (!memory_caching_control || get_cache_aps_delayed_init())
+	if (!memory_caching_control)
+		return 0;
+
+	/*
+	 * Normally we delay MTRR (and PAT) init until cache_aps_init(), but if
+	 * the MTRRs had to be restored on the boot processor on resume, then
+	 * delaying any required MTTR restore on the APs can lead to very slow
+	 * CPU execution during the period when the MTRRs are inconsistent.
+	 */
+	if (get_cache_aps_delayed_init() && !cache_bp_changed_on_restore)
 		return 0;
 
 	/*
@@ -803,8 +817,13 @@ void cache_aps_init(void)
 	if (!memory_caching_control || !get_cache_aps_delayed_init())
 		return;
 
-	stop_machine(cache_rendezvous_handler, NULL, cpu_online_mask);
+	if (cache_bp_changed_on_restore)
+		pr_warn("mtrr: your CPUs had unexpected MTRR settings on resume\n");
+	else
+		stop_machine(cache_rendezvous_handler, NULL, cpu_online_mask);
+
 	set_cache_aps_delayed_init(false);
+	cache_bp_changed_on_restore = false;
 }
 
 static int __init cache_ap_register(void)
diff --git a/arch/x86/kernel/cpu/mtrr/generic.c b/arch/x86/kernel/cpu/mtrr/generic.c
index 0863733858dc..6d96fde84275 100644
--- a/arch/x86/kernel/cpu/mtrr/generic.c
+++ b/arch/x86/kernel/cpu/mtrr/generic.c
@@ -960,12 +960,14 @@ void mtrr_enable(void)
 	mtrr_wrmsr(MSR_MTRRdefType, deftype_lo, deftype_hi);
 }
 
-void mtrr_generic_set_state(void)
+bool mtrr_generic_set_state(void)
 {
 	unsigned long mask, count;
+	bool changed;
 
 	/* Actually set the state */
 	mask = set_mtrr_state();
+	changed = mask != 0;
 
 	/* Use the atomic bitops to update the global mask */
 	for (count = 0; count < sizeof(mask) * 8; ++count) {
@@ -973,6 +975,8 @@ void mtrr_generic_set_state(void)
 			set_bit(count, &smp_changes_mask);
 		mask >>= 1;
 	}
+
+	return changed;
 }
 
 /**

---
base-commit: 18f7fcd5e69a04df57b563360b88be72471d6b62
change-id: 20260204-macbook-pro-mtrr-resume-fix-23c892257d80

Best regards,
-- 
Chris Paulson-Ellis <chris@...lson-ellis.org>

Powered by blists - more mailing lists