linux-kernel - Re: Possible mem cgroup bug in kernels between 4.18.0 and 5.3-rc1.

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:   Mon, 5 Aug 2019 10:42:28 +0200
From:   Michal Hocko <mhocko@...nel.org>
To:     Tetsuo Handa <penguin-kernel@...ove.SAKURA.ne.jp>
Cc:     Masoud Sharbiani <msharbiani@...le.com>,
        Greg KH <gregkh@...uxfoundation.org>, hannes@...xchg.org,
        vdavydov.dev@...il.com, linux-mm@...ck.org,
        cgroups@...r.kernel.org, linux-kernel@...r.kernel.org
Subject: Re: Possible mem cgroup bug in kernels between 4.18.0 and 5.3-rc1.

On Sun 04-08-19 00:51:18, Tetsuo Handa wrote:
> Masoud, will you try this patch?
> 
> By the way, is /sys/fs/cgroup/memory/leaker/memory.usage_in_bytes remains non-zero
> despite /sys/fs/cgroup/memory/leaker/tasks became empty due to memcg OOM killer expected?
> Deleting big-data-file.bin after memcg OOM killer reduces some, but still remains
> non-zero.
> 
> ----------------------------------------
> >From 2f92c70f390f42185c6e2abb8dda98b1b7d02fa9 Mon Sep 17 00:00:00 2001
> From: Tetsuo Handa <penguin-kernel@...ove.SAKURA.ne.jp>
> Date: Sun, 4 Aug 2019 00:41:30 +0900
> Subject: [PATCH] memcg, oom: don't require __GFP_FS when invoking memcg OOM killer
> 
> Masoud Sharbiani noticed that commit 29ef680ae7c21110 ("memcg, oom: move
> out_of_memory back to the charge path") broke memcg OOM called from
> __xfs_filemap_fault() path.

This is very well spotted! I really didn't think of GFP_NOFS although
xfs in the mix could give me some clue.

> It turned out that try_chage() is retrying
> forever without making forward progress because mem_cgroup_oom(GFP_NOFS)
> cannot invoke the OOM killer due to commit 3da88fb3bacfaa33 ("mm, oom:
> move GFP_NOFS check to out_of_memory"). Regarding memcg OOM, we need to
> bypass GFP_NOFS check in order to guarantee forward progress.

This deserves more information about the fix. Why is it OK to trigger
OOM for GFP_NOFS allocations? Doesn't this lead to pre-mature OOM killer
invocation?

You can argue that memcg charges have ignored GFP_NOFS without seeing a
lot of problems. But please document that in the changelog.

It is 3da88fb3bacfaa33 that has introduced this heuristic and I have to
confess I haven't realized the side effect on the memcg side because
OOM was triggered only from the GFP_KERNEL context. So I would point
to 3da88fb3bacfaa33 as introducing the regression albeit silent at the
time.

> Signed-off-by: Tetsuo Handa <penguin-kernel@...ove.SAKURA.ne.jp>
> Reported-by: Masoud Sharbiani <msharbiani@...le.com>
> Bisected-by: Masoud Sharbiani <msharbiani@...le.com>
> Fixes: 29ef680ae7c21110 ("memcg, oom: move out_of_memory back to the charge path")

I would say
Fixes: 3da88fb3bacfaa33 # necessary after 29ef680ae7c21110

Other than that I am not really sure about a better fix. Let's see
whether we see some pre-mature memcg OOM reports and think where to get
from there.

With updated changelog
Acked-by: Michal Hocko <mhocko@...e.com>

Thanks!

> ---
>  mm/oom_kill.c | 5 +++--
>  1 file changed, 3 insertions(+), 2 deletions(-)
> 
> diff --git a/mm/oom_kill.c b/mm/oom_kill.c
> index eda2e2a..26804ab 100644
> --- a/mm/oom_kill.c
> +++ b/mm/oom_kill.c
> @@ -1068,9 +1068,10 @@ bool out_of_memory(struct oom_control *oc)
>  	 * The OOM killer does not compensate for IO-less reclaim.
>  	 * pagefault_out_of_memory lost its gfp context so we have to
>  	 * make sure exclude 0 mask - all other users should have at least
> -	 * ___GFP_DIRECT_RECLAIM to get here.
> +	 * ___GFP_DIRECT_RECLAIM to get here. But mem_cgroup_oom() has to
> +	 * invoke the OOM killer even if it is a GFP_NOFS allocation.
>  	 */
> -	if (oc->gfp_mask && !(oc->gfp_mask & __GFP_FS))
> +	if (oc->gfp_mask && !(oc->gfp_mask & __GFP_FS) && !is_memcg_oom(oc))
>  		return true;
>  
>  	/*
> -- 
> 1.8.3.1
> 

-- 
Michal Hocko
SUSE Labs