linux-kernel - [PATCH] oom-killer kills more than needed

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [day] [month] [year] [list]

Date:	Fri, 19 Sep 2008 11:06:45 -0700
From:	"Chad Zanonie" <chad.zanonie@...il.com>
To:	linux-kernel@...r.kernel.org
Subject: [PATCH] oom-killer kills more than needed

Signed-off-by: Chad Zanonie <chad.zanonie@...il.com>

Possibility exists for an exiting application to be in between
marking its mm NULL and calling mmput when out_of_memory is
invoked. select_bad_process() will continue past this process as
opposed to returning -1UL due to its mm being NULL. This causes
the oom killer in certain scenarios to not only kill the memory
culprit, but also kill the runner up.

EXIT_DEAD seems to be the only flag that guarantees that mmput()
has finished. Checking for PF_KTHREAD should replace p->mm
regardless. Adding EXIT_DEAD to the check seems to prevent
unnecessary kills in local testing.

diff --git a/mm/oom_kill.c b/mm/oom_kill.c
index 64e5b4b..ad3928a 100644
--- a/mm/oom_kill.c
+++ b/mm/oom_kill.c
@@ -216,7 +216,7 @@ static struct task_struct
*select_bad_process(unsigned long *ppoints,
                * skip kernel threads and tasks which have already
released
                * their mm.
                */
-               if (!p->mm)
+               if (p->flags & PF_KTHREAD || p->flags & EXIT_DEAD)
                       continue;
               /* skip the init task */
               if (is_global_init(p))
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/