Message-ID: <bee6b93d-aa2e-4335-9801-89f02eb3eccc@arm.com>
Date: Sun, 2 Nov 2025 12:11:11 +0000
From: Ryan Roberts <ryan.roberts@....com>
To: Guenter Roeck <linux@...ck-us.net>, Yang Shi <yang@...amperecomputing.com>
Cc: catalin.marinas@....com, will@...nel.org, akpm@...ux-foundation.org,
 david@...hat.com, lorenzo.stoakes@...cle.com, ardb@...nel.org,
 dev.jain@....com, scott@...amperecomputing.com, cl@...two.org,
 linux-arm-kernel@...ts.infradead.org, linux-kernel@...r.kernel.org,
 linux-mm@...ck.org, nd@....com
Subject: Re: [PATCH v8 3/5] arm64: mm: support large block mapping when
 rodata=full

On 02/11/2025 10:31, Ryan Roberts wrote:
> On 01/11/2025 16:14, Guenter Roeck wrote:
>> Hi,
>>
>> On Wed, Sep 17, 2025 at 12:02:09PM -0700, Yang Shi wrote:
>>> When rodata=full is specified, kernel linear mapping has to be mapped at
>>> PTE level since large page table can't be split due to break-before-make
>>> rule on ARM64.
>>>
>>> This resulted in a couple of problems:
>>>   - performance degradation
>>>   - more TLB pressure
>>>   - memory waste for kernel page table
>>>
>>> With FEAT_BBM level 2 support, splitting large block page table to
>>> smaller ones doesn't need to make the page table entry invalid anymore.
>>> This allows kernel split large block mapping on the fly.
>>>
>>> Add kernel page table split support and use large block mapping by
>>> default when FEAT_BBM level 2 is supported for rodata=full.  When
>>> changing permissions for kernel linear mapping, the page table will be
>>> split to smaller size.
>>>
>>> The machine without FEAT_BBM level 2 will fallback to have kernel linear
>>> mapping PTE-mapped when rodata=full.
>>>
>>> With this we saw significant performance boost with some benchmarks and
>>> much less memory consumption on my AmpereOne machine (192 cores, 1P)
>>> with 256GB memory.
>>>
>>> * Memory use after boot
>>> Before:
>>> MemTotal:       258988984 kB
>>> MemFree:        254821700 kB
>>>
>>> After:
>>> MemTotal:       259505132 kB
>>> MemFree:        255410264 kB
>>>
>>> Around 500MB more memory are free to use.  The larger the machine, the
>>> more memory saved.
>>>
>>> * Memcached
>>> We saw performance degradation when running Memcached benchmark with
>>> rodata=full vs rodata=on.  Our profiling pointed to kernel TLB pressure.
>>> With this patchset we saw ops/sec is increased by around 3.5%, P99
>>> latency is reduced by around 9.6%.
>>> The gain mainly came from reduced kernel TLB misses.  The kernel TLB
>>> MPKI is reduced by 28.5%.
>>>
>>> The benchmark data is now on par with rodata=on too.
>>>
>>> * Disk encryption (dm-crypt) benchmark
>>> Ran fio benchmark with the below command on a 128G ramdisk (ext4) with
>>> disk encryption (by dm-crypt).
>>> fio --directory=/data --random_generator=lfsr --norandommap            \
>>>     --randrepeat 1 --status-interval=999 --rw=write --bs=4k --loops=1  \
>>>     --ioengine=sync --iodepth=1 --numjobs=1 --fsync_on_close=1         \
>>>     --group_reporting --thread --name=iops-test-job --eta-newline=1    \
>>>     --size 100G
>>>
>>> The IOPS is increased by 90% - 150% (the variance is high, but the worst
>>> number of good case is around 90% more than the best number of bad
>>> case). The bandwidth is increased and the avg clat is reduced
>>> proportionally.
>>>
>>> * Sequential file read
>>> Read 100G file sequentially on XFS (xfs_io read with page cache
>>> populated). The bandwidth is increased by 150%.
>>>
>>
>> With lock debugging enabled, we see a large number of "BUG: sleeping
>> function called from invalid context at kernel/locking/mutex.c:580"
>> and "BUG: Invalid wait context:" backtraces when running v6.18-rc3.
>> Please see example below.
>>
>> Bisect points to this patch.
>>
>> Please let me know if there is anything I can do to help tracking
>> down the problem.
> 
> Thanks for the report - ouch!
> 
> I expect you're running on a system that supports BBML2_NOABORT, based on the
> stack trace, I expect you have CONFIG_DEBUG_PAGEALLOC enabled? That will cause
> permission tricks to be played on the linear map at page allocation and free
> time, which can happen in non-sleepable contexts. And with this patch we are
> taking pgtable_split_lock (a mutex) in split_kernel_leaf_mapping(), which is
> called as a result of the permission change request.
> 
> However, when CONFIG_DEBUG_PAGEALLOC enabled we always force-map the linear map
> by PTE so split_kernel_leaf_mapping() is actually unneccessary and will return
> without actually having to split anything. So we could add an early "if
> (force_pte_mapping()) return 0;" to bypass the function entirely in this case,
> and I *think* that should solve it.
> 
> But I'm also concerned about KFENCE. I can't remember it's exact semantics off
> the top of my head, so I'm concerned we could see similar problems there (where
> we only force pte mapping for the KFENCE pool).
> 
> I'll investigate fully tomorrow and hopefully provide a fix.

Here's a proposed fix, although I can't get access to a system with BBML2 until 
tomorrow at the earliest. Guenter, I wonder if you could check that this 
resolves your issue?

---8<---
commit 602ec2db74e5abfb058bd03934475ead8558eb72
Author: Ryan Roberts <ryan.roberts@....com>
Date:   Sun Nov 2 11:45:18 2025 +0000

    arm64: mm: Don't attempt to split known pte-mapped regions
    
    It has been reported that split_kernel_leaf_mapping() is trying to sleep
    in non-sleepable context. It does this when acquiring the
    pgtable_split_lock mutex, when either CONFIG_DEBUG_ALLOC or
    CONFIG_KFENCE are enabled, which change linear map permissions within
    softirq context during memory allocation and/or freeing.
    
    But it turns out that the memory for which these features may attempt to
    modify the permissions is always mapped by pte, so there is no need to
    attempt to split the mapping. So let's exit early in these cases and
    avoid attempting to take the mutex.
    
    Closes: https://lore.kernel.org/all/f24b9032-0ec9-47b1-8b95-c0eeac7a31c5@roeck-us.net/
    Fixes: a166563e7ec3 ("arm64: mm: support large block mapping when rodata=full")
    Signed-off-by: Ryan Roberts <ryan.roberts@....com>

diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index b8d37eb037fc..6e26f070bb49 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -708,6 +708,16 @@ static int split_kernel_leaf_mapping_locked(unsigned long addr)
 	return ret;
 }
 
+static inline bool force_pte_mapping(void)
+{
+	bool bbml2 = system_capabilities_finalized() ?
+		system_supports_bbml2_noabort() : cpu_supports_bbml2_noabort();
+
+	return (!bbml2 && (rodata_full || arm64_kfence_can_set_direct_map() ||
+			   is_realm_world())) ||
+		debug_pagealloc_enabled();
+}
+
 static DEFINE_MUTEX(pgtable_split_lock);
 
 int split_kernel_leaf_mapping(unsigned long start, unsigned long end)
@@ -723,6 +733,16 @@ int split_kernel_leaf_mapping(unsigned long start, unsigned long end)
 	if (!system_supports_bbml2_noabort())
 		return 0;
 
+	/*
+	 * If the region is within a pte-mapped area, there is no need to try to
+	 * split. Additionally, CONFIG_DEBUG_ALLOC and CONFIG_KFENCE may change
+	 * permissions from softirq context so for those cases (which are always
+	 * pte-mapped), we must not go any further because taking the mutex
+	 * below may sleep.
+	 */
+	if (force_pte_mapping() || is_kfence_address((void *)start))
+		return 0;
+
 	/*
 	 * Ensure start and end are at least page-aligned since this is the
 	 * finest granularity we can split to.
@@ -1009,16 +1029,6 @@ static inline void arm64_kfence_map_pool(phys_addr_t kfence_pool, pgd_t *pgdp) {
 
 #endif /* CONFIG_KFENCE */
 
-static inline bool force_pte_mapping(void)
-{
-	bool bbml2 = system_capabilities_finalized() ?
-		system_supports_bbml2_noabort() : cpu_supports_bbml2_noabort();
-
-	return (!bbml2 && (rodata_full || arm64_kfence_can_set_direct_map() ||
-			   is_realm_world())) ||
-		debug_pagealloc_enabled();
-}
-
 static void __init map_mem(pgd_t *pgdp)
 {
 	static const u64 direct_map_end = _PAGE_END(VA_BITS_MIN);
---8<---

Thanks,
Ryan

> 
> Yang Shi, Do you have any additional thoughts?
> 
> Thanks,
> Ryan
> 
>>
>> Thanks,
>> Guenter
>>
>> ---
>> Example log:
>>
>> [    0.537499] BUG: sleeping function called from invalid context at kernel/locking/mutex.c:580
>> [    0.537501] in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 1, name: swapper/0
>> [    0.537502] preempt_count: 1, expected: 0
>> [    0.537504] 2 locks held by swapper/0/1:
>> [    0.537505]  #0: ffffb60b01211960 (sched_domains_mutex){+.+.}-{4:4}, at: sched_domains_mutex_lock+0x24/0x38
>> [    0.537510]  #1: ffffb60b01595838 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire+0x0/0x40
>> [    0.537516] CPU: 0 UID: 0 PID: 1 Comm: swapper/0 Not tainted 6.18.0-dbg-DEV #1 NONE
>> [    0.537517] Call trace:
>> [    0.537518]  show_stack+0x20/0x38 (C)
>> [    0.537520]  __dump_stack+0x28/0x38
>> [    0.537522]  dump_stack_lvl+0xac/0xf0
>> [    0.537525]  dump_stack+0x18/0x3c
>> [    0.537527]  __might_resched+0x248/0x2a0
>> [    0.537529]  __might_sleep+0x40/0x90
>> [    0.537531]  __mutex_lock_common+0x70/0x1818
>> [    0.537533]  mutex_lock_nested+0x34/0x48
>> [    0.537534]  split_kernel_leaf_mapping+0x74/0x1a0
>> [    0.537536]  update_range_prot+0x40/0x150
>> [    0.537537]  __change_memory_common+0x30/0x148
>> [    0.537538]  __kernel_map_pages+0x70/0x88
>> [    0.537540]  __free_frozen_pages+0x6e4/0x7b8
>> [    0.537542]  free_frozen_pages+0x1c/0x30
>> [    0.537544]  __free_slab+0xf0/0x168
>> [    0.537547]  free_slab+0x2c/0xf8
>> [    0.537549]  free_to_partial_list+0x4e0/0x620
>> [    0.537551]  __slab_free+0x228/0x250
>> [    0.537553]  kfree+0x3c4/0x4c0
>> [    0.537555]  destroy_sched_domain+0xf8/0x140
>> [    0.537557]  cpu_attach_domain+0x17c/0x610
>> [    0.537558]  build_sched_domains+0x15a4/0x1718
>> [    0.537560]  sched_init_domains+0xbc/0xf8
>> [    0.537561]  sched_init_smp+0x30/0x98
>> [    0.537562]  kernel_init_freeable+0x148/0x230
>> [    0.537564]  kernel_init+0x28/0x148
>> [    0.537566]  ret_from_fork+0x10/0x20
>> [    0.537569] =============================
>> [    0.537569] [ BUG: Invalid wait context ]
>> [    0.537571] 6.18.0-dbg-DEV #1 Tainted: G        W
>> [    0.537572] -----------------------------
>> [    0.537572] swapper/0/1 is trying to lock:
>> [    0.537573] ffffb60b011f3830 (pgtable_split_lock){+.+.}-{4:4}, at: split_kernel_leaf_mapping+0x74/0x1a0
>> [    0.537576] other info that might help us debug this:
>> [    0.537577] context-{5:5}
>> [    0.537578] 2 locks held by swapper/0/1:
>> [    0.537579]  #0: ffffb60b01211960 (sched_domains_mutex){+.+.}-{4:4}, at: sched_domains_mutex_lock+0x24/0x38
>> [    0.537582]  #1: ffffb60b01595838 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire+0x0/0x40
>> [    0.537585] stack backtrace:
>> [    0.537585] CPU: 0 UID: 0 PID: 1 Comm: swapper/0 Tainted: G        W           6.18.0-dbg-DEV #1 NONE
>> [    0.537587] Tainted: [W]=WARN
>> [    0.537588] Call trace:
>> [    0.537589]  show_stack+0x20/0x38 (C)
>> [    0.537591]  __dump_stack+0x28/0x38
>> [    0.537593]  dump_stack_lvl+0xac/0xf0
>> [    0.537596]  dump_stack+0x18/0x3c
>> [    0.537598]  __lock_acquire+0x980/0x2a20
>> [    0.537600]  lock_acquire+0x124/0x2b8
>> [    0.537602]  __mutex_lock_common+0xd8/0x1818
>> [    0.537604]  mutex_lock_nested+0x34/0x48
>> [    0.537605]  split_kernel_leaf_mapping+0x74/0x1a0
>> [    0.537607]  update_range_prot+0x40/0x150
>> [    0.537608]  __change_memory_common+0x30/0x148
>> [    0.537609]  __kernel_map_pages+0x70/0x88
>> [    0.537610]  __free_frozen_pages+0x6e4/0x7b8
>> [    0.537613]  free_frozen_pages+0x1c/0x30
>> [    0.537615]  __free_slab+0xf0/0x168
>> [    0.537617]  free_slab+0x2c/0xf8
>> [    0.537619]  free_to_partial_list+0x4e0/0x620
>> [    0.537621]  __slab_free+0x228/0x250
>> [    0.537623]  kfree+0x3c4/0x4c0
>> [    0.537625]  destroy_sched_domain+0xf8/0x140
>> [    0.537627]  cpu_attach_domain+0x17c/0x610
>> [    0.537628]  build_sched_domains+0x15a4/0x1718
>> [    0.537630]  sched_init_domains+0xbc/0xf8
>> [    0.537631]  sched_init_smp+0x30/0x98
>> [    0.537632]  kernel_init_freeable+0x148/0x230
>> [    0.537633]  kernel_init+0x28/0x148
>> [    0.537635]  ret_from_fork+0x10/0x20
>>
>> ---
>> bisect:
>>
>> # bad: [3a8660878839faadb4f1a6dd72c3179c1df56787] Linux 6.18-rc1
>> # good: [e5f0a698b34ed76002dc5cff3804a61c80233a7a] Linux 6.17
>> git bisect start 'v6.18-rc1' 'v6.17'
>> # bad: [58809f614e0e3f4e12b489bddf680bfeb31c0a20] Merge tag 'drm-next-2025-10-01' of https://gitlab.freedesktop.org/drm/kernel
>> git bisect bad 58809f614e0e3f4e12b489bddf680bfeb31c0a20
>> # bad: [a8253f807760e9c80eada9e5354e1240ccf325f9] Merge tag 'soc-newsoc-6.18' of git://git.kernel.org/pub/scm/linux/kernel/git/soc/soc
>> git bisect bad a8253f807760e9c80eada9e5354e1240ccf325f9
>> # bad: [4b81e2eb9e4db8f6094c077d0c8b27c264901c1b] Merge tag 'timers-vdso-2025-09-29' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
>> git bisect bad 4b81e2eb9e4db8f6094c077d0c8b27c264901c1b
>> # bad: [f1004b2f19d7e9add9d707f64d9fcbc50f67921b] Merge tag 'm68k-for-v6.18-tag1' of git://git.kernel.org/pub/scm/linux/kernel/git/geert/linux-m68k
>> git bisect bad f1004b2f19d7e9add9d707f64d9fcbc50f67921b
>> # good: [a9401710a5f5681abd2a6f21f9e76bc9f2e81891] Merge tag 'v6.18-rc-part1-smb3-common' of git://git.samba.org/ksmbd
>> git bisect good a9401710a5f5681abd2a6f21f9e76bc9f2e81891
>> # good: [fe68bb2861808ed5c48d399bd7e670ab76829d55] Merge tag 'microblaze-v6.18' of git://git.monstr.eu/linux-2.6-microblaze
>> git bisect good fe68bb2861808ed5c48d399bd7e670ab76829d55
>> # bad: [f2d64a22faeeecff385b4c91fab5fe036ab00162] Merge branch 'for-next/perf' into for-next/core
>> git bisect bad f2d64a22faeeecff385b4c91fab5fe036ab00162
>> # good: [30f9386820cddbba59b48ae0670c3a1646dd440e] Merge branch 'for-next/misc' into for-next/core
>> git bisect good 30f9386820cddbba59b48ae0670c3a1646dd440e
>> # good: [43de0ac332b815cf56dbdce63687de9acfd35d49] drivers/perf: hisi: Relax the event ID check in the framework
>> git bisect good 43de0ac332b815cf56dbdce63687de9acfd35d49
>> # good: [5973a62efa34c80c9a4e5eac1fca6f6209b902af] arm64: map [_text, _stext) virtual address range non-executable+read-only
>> git bisect good 5973a62efa34c80c9a4e5eac1fca6f6209b902af
>> # good: [b3abb08d6f628a76c36bf7da9508e1a67bf186a0] drivers/perf: hisi: Refactor the event configuration of L3C PMU
>> git bisect good b3abb08d6f628a76c36bf7da9508e1a67bf186a0
>> # good: [6d2f913fda5683fbd4c3580262e10386c1263dfb] Documentation: hisi-pmu: Add introduction to HiSilicon V3 PMU
>> git bisect good 6d2f913fda5683fbd4c3580262e10386c1263dfb
>> # good: [2084660ad288c998b6f0c885e266deb364f65fba] perf/dwc_pcie: Fix use of uninitialized variable
>> git bisect good 2084660ad288c998b6f0c885e266deb364f65fba
>> # bad: [77dfca70baefcb988318a72fe69eb99f6dabbbb1] Merge branch 'for-next/mm' into for-next/core
>> git bisect bad 77dfca70baefcb988318a72fe69eb99f6dabbbb1
>> # first bad commit: [77dfca70baefcb988318a72fe69eb99f6dabbbb1] Merge branch 'for-next/mm' into for-next/core
>>
>> ---
>> bisect into branch:
>>
>> - git checkout -b testing 77dfca70baefcb988318a72fe69eb99f6dabbbb1
>> - git rebase 77dfca70baefcb988318a72fe69eb99f6dabbbb1~1
>>   [ fix minor conflict similar to the conflict resolution in 77dfca70baefc]
>> - git diff 77dfca70baefcb988318a72fe69eb99f6dabbbb1
>>   [ confirmed that there are no differences ]
>> - confirm that the problem is still seen at the tip of the rebase
>> - git bisect start HEAD 77dfca70baefcb988318a72fe69eb99f6dabbbb1~1
>> - run bisect
>>
>> Results:
>>
>> # bad: [47fc25df1ae3ae8412f1b812fb586c714d04a5e6] arm64: map [_text, _stext) virtual address range non-executable+read-only
>> # good: [30f9386820cddbba59b48ae0670c3a1646dd440e] Merge branch 'for-next/misc' into for-next/core
>> git bisect start 'HEAD' '77dfca70baefcb988318a72fe69eb99f6dabbbb1~1'
>> # good: [805491d19fc21271b5c27f4602f8f66b625c110f] arm64/Kconfig: Remove CONFIG_RODATA_FULL_DEFAULT_ENABLED
>> git bisect good 805491d19fc21271b5c27f4602f8f66b625c110f
>> # bad: [13c7d7426232cc4489df7cd2e1f646a22d3f6172] arm64: mm: support large block mapping when rodata=full
>> git bisect bad 13c7d7426232cc4489df7cd2e1f646a22d3f6172
>> # good: [a4d9c67e503f2b73c2d89d8e8209dfd241bdc8d8] arm64: Enable permission change on arm64 kernel block mappings
>> git bisect good a4d9c67e503f2b73c2d89d8e8209dfd241bdc8d8
>> # first bad commit: [13c7d7426232cc4489df7cd2e1f646a22d3f6172] arm64: mm: support large block mapping when rodata=full
> 

Powered by blists - more mailing lists