linux-kernel - [PATCH]: OOM can panic due to processes stuck in __alloc

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <4558C3EB.1070400@openvz.org>
Date:	Mon, 13 Nov 2006 22:13:47 +0300
From:	Kirill Korotaev <dev@...nvz.org>
To:	Andrew Morton <akpm@...l.org>,
	Linux Kernel Mailing List <linux-kernel@...r.kernel.org>,
	devel@...nvz.org
Subject: [PATCH]: OOM can panic due to processes stuck in __alloc_pages()

OOM can panic due to the processes stuck in __alloc_pages()
doing infinite rebalance loop while no memory can be reclaimed.
OOM killer tries to kill some processes, but unfortunetaly,
rebalance label was moved by someone below the TIF_MEMDIE check,
so buddy allocator doesn't see that process is OOM-killed
and it can simply fail the allocation :/

Observed in reality on RHEL4(2.6.9)+OpenVZ kernel when a user doing
some memory allocation tricks triggered OOM panic.

Signed-Off-By: Denis Lunev <den@...ru>
Signed-Off-By: Kirill Korotaev <dev@...nvz.org>

--- ./mm/page_alloc.c.oomx	2006-11-08 17:44:16.000000000 +0300
+++ ./mm/page_alloc.c	2006-11-13 21:57:33.000000000 +0300
@@ -1251,6 +1251,7 @@ restart:

 	/* This allocation should allow future memory freeing. */

+rebalance:
 	if (((p->flags & PF_MEMALLOC) || unlikely(test_thread_flag(TIF_MEMDIE)))
 			&& !in_interrupt()) {
 		if (!(gfp_mask & __GFP_NOMEMALLOC)) {
@@ -1272,7 +1273,6 @@ nofail_alloc:
 	if (!wait)
 		goto nopage;

-rebalance:
 	cond_resched();

 	/* We now go into synchronous reclaim */
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/