linux-kernel - [PATCH] oom killer: break from infinite loop

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Date:	Thu, 25 Mar 2010 00:25:05 +0800
From:	Anfei Zhou <anfei.zhou@...il.com>
To:	akpm@...ux-foundation.org, rientjes@...gle.com,
	kosaki.motohiro@...fujitsu.com, nishimura@....nes.nec.co.jp,
	kamezawa.hiroyu@...fujitsu.com
Cc:	linux-mm@...ck.org, linux-kernel@...r.kernel.org
Subject: [PATCH] oom killer: break from infinite loop

In multi-threading environment, if the current task(A) have got
the mm->mmap_sem semaphore, and the thread(B) in the same process
is selected to be oom killed, because they shares the same semaphore,
thread B can not really be killed.  So __alloc_pages_slowpath turns
to be a infinite loop.  Here set all the threads in the group to
TIF_MEMDIE, it gets a chance to break and exit.

Signed-off-by: Anfei Zhou <anfei.zhou@...il.com>
---
 mm/oom_kill.c |    4 ++++
 1 files changed, 4 insertions(+), 0 deletions(-)

diff --git a/mm/oom_kill.c b/mm/oom_kill.c
index 9b223af..aab9892 100644
--- a/mm/oom_kill.c
+++ b/mm/oom_kill.c
@@ -381,6 +381,8 @@ static void dump_header(struct task_struct *p, gfp_t gfp_mask, int order,
  */
 static void __oom_kill_task(struct task_struct *p, int verbose)
 {
+	struct task_struct *t;
+
 	if (is_global_init(p)) {
 		WARN_ON(1);
 		printk(KERN_WARNING "tried to kill init!\n");
@@ -412,6 +414,8 @@ static void __oom_kill_task(struct task_struct *p, int verbose)
 	 */
 	p->rt.time_slice = HZ;
 	set_tsk_thread_flag(p, TIF_MEMDIE);
+	for (t = next_thread(p); t != p; t = next_thread(t))
+		set_tsk_thread_flag(t, TIF_MEMDIE);
 
 	force_sig(SIGKILL, p);
 }
-- 
1.6.4.rc1

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/