linux-kernel - [RFC PATCH] mm/page_alloc: ?nommu? Deadlock in drain_pages

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [day] [month] [year] [list]

Message-ID: <20251211102607.2538595-1-daniel@thingy.jp>
Date: Thu, 11 Dec 2025 19:26:07 +0900
From: Daniel Palmer <daniel@...ngy.jp>
To: linux-mm@...ck.org
Cc: linux-kernel@...r.kernel.org,
	Daniel Palmer <daniel@...ngy.jp>
Subject: [RFC PATCH] mm/page_alloc: ?nommu? Deadlock in drain_pages_zone()?

per_cpu_pages_init() sets pcp->batch to 1 (BOOT_PAGESET_BATCH) at boot,
then later it is set to 0 (from zone_batchsize()) via the following path:

init_per_zone_wmark_min()
 ->setup_per_zone_wmarks()
   ->zone_pcp_update()
     ->zone_set_pageset_high_and_batch()

Once this happens if an OOM happens drain_pages_zone() will never return
because count never gets to zero.

Signed-off-by: Daniel Palmer <daniel@...ngy.jp>
---

Background: I am running a nommu kernel on 68000 with 8MB of RAM,
I messed up my userland config and the resulting binaries became
too big to fit in to the free blocks of memory, OOM should have
happened and I should have seen the kernel trying to free memory
but instead it dead locked. Since this is nommu I thought maybe
I am corrupting memory but I checked with gdb and it doesn't seem
like it. I have some local patches to make this weird setup work
too but I don't think its anything to do with that either.

This change stops the dead lock and I see OOM messages instead as
expected. I have no idea if it is correct. Patch is simply to show
what I am seeing.

 mm/page_alloc.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 822e05f1a964..83c9ce6f93ad 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -2618,8 +2618,11 @@ static void drain_pages_zone(unsigned int cpu, struct zone *zone)
 		spin_lock(&pcp->lock);
 		count = pcp->count;
 		if (count) {
-			int to_drain = min(count,
-				pcp->batch << CONFIG_PCP_BATCH_SCALE_MAX);
+			int to_drain = count;
+
+			/* if pcp->batch is zero this loop will never exit, on nommu pcp->batch is always 0 */
+			if (likely(pcp->batch))
+				to_drain = min(count, pcp->batch << CONFIG_PCP_BATCH_SCALE_MAX);

 			free_pcppages_bulk(zone, to_drain, pcp, 0);
 			count -= to_drain;
-- 
2.51.0