linux-kernel - Re: [PATCH v2] mm,oom: don't abort on exiting processes when selecting a victim.

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Date:	Wed, 17 Feb 2016 15:39:17 +0100
From:	Michal Hocko <mhocko@...nel.org>
To:	Tetsuo Handa <penguin-kernel@...ove.SAKURA.ne.jp>
Cc:	akpm@...ux-foundation.org, rientjes@...gle.com, mgorman@...e.de,
	oleg@...hat.com, torvalds@...ux-foundation.org, hughd@...gle.com,
	andrea@...nel.org, riel@...hat.com, linux-mm@...ck.org,
	linux-kernel@...r.kernel.org, Johannes Weiner <hannes@...xchg.org>
Subject: Re: [PATCH v2] mm,oom: don't abort on exiting processes when
 selecting a victim.

On Wed 17-02-16 23:31:25, Tetsuo Handa wrote:
> Currently, oom_scan_process_thread() returns OOM_SCAN_ABORT when there is
> a thread which is exiting. But it is possible that that thread is blocked
> at down_read(&mm->mmap_sem) in exit_mm() called from do_exit() whereas
> one of threads sharing that memory is doing a GFP_KERNEL allocation
> between down_write(&mm->mmap_sem) and up_write(&mm->mmap_sem)
> (e.g. mmap()).
> 
> ----------
> T1                  T2
>                     Calls mmap()
> Calls _exit(0)
>                     Arrives at vm_mmap_pgoff()
> Arrives at do_exit()
> Gets PF_EXITING via exit_signals()
>                     Calls down_write(&mm->mmap_sem)
>                     Calls do_mmap_pgoff()
> Calls down_read(&mm->mmap_sem) from exit_mm()
>                     Calls out of memory via a GFP_KERNEL allocation but
>                     oom_scan_process_thread(T1) returns OOM_SCAN_ABORT
> ----------
> 
> down_read(&mm->mmap_sem) by T1 is waiting for up_write(&mm->mmap_sem) by
> T2 while oom_scan_process_thread() by T2 is waiting for T1 to set
> T1->mm = NULL. Under such situation, the OOM killer does not choose
> a victim, which results in silent OOM livelock problem.
> 
> This patch changes oom_scan_process_thread() not to return OOM_SCAN_ABORT
> when there is a thread which is exiting.

Thank you for the updated changelog. This makes much more sense now.
This problem exists for quite some time but I would be hesitant to
mark it for stable because the side effects are quite hard to evaluate.
We could e.g. see a premature OOM killer invocation while the currently
exiting task just didn't get to finish and release its mm.

> Signed-off-by: Tetsuo Handa <penguin-kernel@...ove.SAKURA.ne.jp>
> Signed-off-by: Johannes Weiner <hannes@...xchg.org>

Acked-by: Michal Hocko <mhocko@...e.com>

> ---
>  mm/oom_kill.c | 3 ---
>  1 file changed, 3 deletions(-)
> 
> diff --git a/mm/oom_kill.c b/mm/oom_kill.c
> index cf87153..6e6abaf 100644
> --- a/mm/oom_kill.c
> +++ b/mm/oom_kill.c
> @@ -292,9 +292,6 @@ enum oom_scan_t oom_scan_process_thread(struct oom_control *oc,
>  	if (oom_task_origin(task))
>  		return OOM_SCAN_SELECT;
>  
> -	if (task_will_free_mem(task) && !is_sysrq_oom(oc))
> -		return OOM_SCAN_ABORT;
> -
>  	return OOM_SCAN_OK;
>  }
>  
> -- 
> 1.8.3.1

-- 
Michal Hocko
SUSE Labs