linux-kernel - Re: 4.8.8 kernel trigger OOM killer repeatedly when I have lots of RAM that should be free

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20161128072315.GC14788@dhcp22.suse.cz>
Date:   Mon, 28 Nov 2016 08:23:15 +0100
From:   Michal Hocko <mhocko@...nel.org>
To:     Marc MERLIN <marc@...lins.org>
Cc:     Vlastimil Babka <vbabka@...e.cz>,
        Linus Torvalds <torvalds@...ux-foundation.org>,
        linux-mm <linux-mm@...ck.org>,
        LKML <linux-kernel@...r.kernel.org>,
        Joonsoo Kim <iamjoonsoo.kim@....com>,
        Tejun Heo <tj@...nel.org>,
        Greg Kroah-Hartman <gregkh@...uxfoundation.org>
Subject: Re: 4.8.8 kernel trigger OOM killer repeatedly when I have lots of
 RAM that should be free

Marc, could you try this patch please? I think it should be pretty clear
it should help you but running it through your use case would be more
than welcome before I ask Greg to take this to the 4.8 stable tree.

Thanks!

On Wed 23-11-16 07:34:10, Michal Hocko wrote:
[...]
> commit b2ccdcb731b666aa28f86483656c39c5e53828c7
> Author: Michal Hocko <mhocko@...e.com>
> Date:   Wed Nov 23 07:26:30 2016 +0100
> 
>     mm, oom: stop pre-mature high-order OOM killer invocations
>     
>     31e49bfda184 ("mm, oom: protect !costly allocations some more for
>     !CONFIG_COMPACTION") was an attempt to reduce chances of pre-mature OOM
>     killer invocation for high order requests. It seemed to work for most
>     users just fine but it is far from bullet proof and obviously not
>     sufficient for Marc who has reported pre-mature OOM killer invocations
>     with 4.8 based kernels. 4.9 will all the compaction improvements seems
>     to be behaving much better but that would be too intrusive to backport
>     to 4.8 stable kernels. Instead this patch simply never declares OOM for
>     !costly high order requests. We rely on order-0 requests to do that in
>     case we are really out of memory. Order-0 requests are much more common
>     and so a risk of a livelock without any way forward is highly unlikely.
>     
>     Reported-by: Marc MERLIN <marc@...lins.org>
>     Signed-off-by: Michal Hocko <mhocko@...e.com>
> 
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index a2214c64ed3c..7401e996009a 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -3161,6 +3161,16 @@ should_compact_retry(struct alloc_context *ac, unsigned int order, int alloc_fla
>  	if (!order || order > PAGE_ALLOC_COSTLY_ORDER)
>  		return false;
>  
> +#ifdef CONFIG_COMPACTION
> +	/*
> +	 * This is a gross workaround to compensate a lack of reliable compaction
> +	 * operation. We cannot simply go OOM with the current state of the compaction
> +	 * code because this can lead to pre mature OOM declaration.
> +	 */
> +	if (order <= PAGE_ALLOC_COSTLY_ORDER)
> +		return true;
> +#endif
> +
>  	/*
>  	 * There are setups with compaction disabled which would prefer to loop
>  	 * inside the allocator rather than hit the oom killer prematurely.
> -- 
> Michal Hocko
> SUSE Labs

-- 
Michal Hocko
SUSE Labs