linux-kernel - Re: [patch for-3.5-rc1] mm, oom: fix badness score underflow

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Sat, 09 Jun 2012 18:25:46 -0400
From:	KOSAKI Motohiro <kosaki.motohiro@...il.com>
To:	David Rientjes <rientjes@...gle.com>
CC:	Linus Torvalds <torvalds@...ux-foundation.org>,
	Andrew Morton <akpm@...ux-foundation.org>,
	Oleg Nesterov <oleg@...hat.com>, Dave Jones <davej@...hat.com>,
	Hugh Dickins <hughd@...gle.com>, Ingo Molnar <mingo@...e.hu>,
	Peter Zijlstra <peterz@...radead.org>,
	Srikar Dronamraju <srikar@...ux.vnet.ibm.com>,
	KOSAKI Motohiro <kosaki.motohiro@...fujitsu.com>,
	Ananth N Mavinakayanahalli <ananth@...ibm.com>,
	Anton Arapov <anton@...hat.com>,
	Masami Hiramatsu <masami.hiramatsu.pt@...achi.com>,
	linux-kernel@...r.kernel.org, kosaki.motohiro@...il.com
Subject: Re: [patch for-3.5-rc1] mm, oom: fix badness score underflow

(6/8/12 4:21 PM), David Rientjes wrote:
> If the privileges given to root threads (3% of allowable memory) or a
> negative value of /proc/pid/oom_score_adj happen to exceed the amount of
> rss of a thread, its badness score overflows as a result of a7f638f999ff
> ("mm, oom: normalize oom scores to oom_score_adj scale only for
> userspace").
>
> Fix this by making the type signed and return 1, meaning the thread is
> still eligible for kill, if the value is negative.
>
> Reported-by: Dave Jones<davej@...hat.com>
> Acked-by: Oleg Nesterov<oleg@...hat.com>
> Signed-off-by: David Rientjes<rientjes@...gle.com>
> ---
>   mm/oom_kill.c |    4 ++--
>   1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/mm/oom_kill.c b/mm/oom_kill.c
> --- a/mm/oom_kill.c
> +++ b/mm/oom_kill.c
> @@ -183,7 +183,7 @@ static bool oom_unkillable_task(struct task_struct *p,
>   unsigned long oom_badness(struct task_struct *p, struct mem_cgroup *memcg,
>   			  const nodemask_t *nodemask, unsigned long totalpages)
>   {
> -	unsigned long points;
> +	long points;
>
>   	if (oom_unkillable_task(p, memcg, nodemask))
>   		return 0;
> @@ -223,7 +223,7 @@ unsigned long oom_badness(struct task_struct *p, struct mem_cgroup *memcg,
>   	 * Never return 0 for an eligible task regardless of the root bonus and
>   	 * oom_score_adj (oom_score_adj can't be OOM_SCORE_ADJ_MIN here).
>   	 */
> -	return points ? points : 1;
> +	return points>  0 ? points : 1;
>   }

Use long long. following line is dangerous.

         points += p->signal->oom_score_adj * totalpages / 1000;

maximum oom_score_adj is 1000. then if system has >8G memory on 32bit
(i.e. LONG_MAX [pages] * 4096 [pagesize] / 1000), it might get an overflow.

Or, don't use normalized oom_score_adj. i.e, oom_score_adj_write() convert
oom_score_adj into rss based modifier.

This is oom-killer code. A micro optimization don't bring us a performance benefit.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/