Message-ID: <ZIgl9Zl5D+Y1RGM0@dhcp22.suse.cz>
Date:   Tue, 13 Jun 2023 10:16:53 +0200
From:   Michal Hocko <mhocko@...e.com>
To:     程垲涛 Chengkaitao Cheng 
        <chengkaitao@...iglobal.com>
Cc:     "tj@...nel.org" <tj@...nel.org>,
        "lizefan.x@...edance.com" <lizefan.x@...edance.com>,
        "hannes@...xchg.org" <hannes@...xchg.org>,
        "corbet@....net" <corbet@....net>,
        "roman.gushchin@...ux.dev" <roman.gushchin@...ux.dev>,
        "shakeelb@...gle.com" <shakeelb@...gle.com>,
        "akpm@...ux-foundation.org" <akpm@...ux-foundation.org>,
        "brauner@...nel.org" <brauner@...nel.org>,
        "muchun.song@...ux.dev" <muchun.song@...ux.dev>,
        "viro@...iv.linux.org.uk" <viro@...iv.linux.org.uk>,
        "zhengqi.arch@...edance.com" <zhengqi.arch@...edance.com>,
        "ebiederm@...ssion.com" <ebiederm@...ssion.com>,
        "Liam.Howlett@...cle.com" <Liam.Howlett@...cle.com>,
        "chengzhihao1@...wei.com" <chengzhihao1@...wei.com>,
        "pilgrimtao@...il.com" <pilgrimtao@...il.com>,
        "haolee.swjtu@...il.com" <haolee.swjtu@...il.com>,
        "yuzhao@...gle.com" <yuzhao@...gle.com>,
        "willy@...radead.org" <willy@...radead.org>,
        "vasily.averin@...ux.dev" <vasily.averin@...ux.dev>,
        "vbabka@...e.cz" <vbabka@...e.cz>,
        "surenb@...gle.com" <surenb@...gle.com>,
        "sfr@...b.auug.org.au" <sfr@...b.auug.org.au>,
        "mcgrof@...nel.org" <mcgrof@...nel.org>,
        "feng.tang@...el.com" <feng.tang@...el.com>,
        "cgroups@...r.kernel.org" <cgroups@...r.kernel.org>,
        "linux-doc@...r.kernel.org" <linux-doc@...r.kernel.org>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        "linux-fsdevel@...r.kernel.org" <linux-fsdevel@...r.kernel.org>,
        "linux-mm@...ck.org" <linux-mm@...ck.org>
Subject: Re: [PATCH v3 0/2] memcontrol: support cgroup level OOM protection

On Sun 04-06-23 08:05:53, 程垲涛 Chengkaitao Cheng wrote:
> At 2023-05-29 22:02:47, "Michal Hocko" <mhocko@...e.com> wrote:
> >On Thu 25-05-23 07:35:41, 程垲涛 Chengkaitao Cheng wrote:
> >> At 2023-05-22 21:03:50, "Michal Hocko" <mhocko@...e.com> wrote:
> >[...]
> >> >> I have created a new indicator oom_kill_inherit that maintains a negative correlation 
> >> >> with memory.oom.protect, so we have a ruler to measure the optimal value of 
> >> >> memory.oom.protect.
> >> >
> >> >An example might help here.
> >> 
> >> In my testing case, by adjusting memory.oom.protect, I was able to significantly 
> >> reduce the oom_kill_inherit of the corresponding cgroup. In a physical machine 
> >> with severely oversold memory, I divided all cgroups into three categories and 
> >> controlled their probability of being selected by the oom-killer to 0%,% 20, 
> >> and 80%, respectively.
> >
> >I might be just dense but I am lost. Can we focus on the barebone
> >semantic of the group oom selection and killing first. No magic
> >auto-tuning at this stage please.
> >
> >> >> >> about the semantics of non-leaf memcgs protection,
> >> >> >> If a non-leaf memcg's oom_protect quota is set, its leaf memcg will proportionally 
> >> >> >> calculate the new effective oom_protect quota based on non-leaf memcg's quota.
> >> >> >
> >> >> >So the non-leaf memcg is never used as a target? What if the workload is
> >> >> >distributed over several sub-groups? Our current oom.group
> >> >> >implementation traverses the tree to find a common ancestor in the oom
> >> >> >domain with the oom.group.
> >> >> 
> >> >> If the oom_protect quota of the parent non-leaf memcg is less than the sum of 
> >> >> sub-groups oom_protect quota, the oom_protect quota of each sub-group will 
> >> >> be proportionally reduced
> >> >> If the oom_protect quota of the parent non-leaf memcg is greater than the sum 
> >> >> of sub-groups oom_protect quota, the oom_protect quota of each sub-group 
> >> >> will be proportionally increased
> >> >> The purpose of doing so is that users can set oom_protect quota according to 
> >> >> their own needs, and the system management process can set appropriate 
> >> >> oom_protect quota on the parent non-leaf memcg as the final cover, so that 
> >> >> the system management process can indirectly manage all user processes.
> >> >
> >> >I guess that you are trying to say that the oom protection has a
> >> >standard hierarchical behavior. And that is fine, well, in fact it is
> >> >mandatory for any control knob to have a sane hierarchical properties.
> >> >But that doesn't address my above question. Let me try again. When is a
> >> >non-leaf memcg potentially selected as the oom victim? It doesn't have
> >> >any tasks directly but it might be a suitable target to kill a multi
> >> >memcg based workload (e.g. a full container).
> >> 
> >> If nonleaf memcg have the higher memory usage and the smaller 
> >> memory.oom.protect, it will have the higher the probability being 
> >> selected by the killer. If the non-leaf memcg is selected as the oom 
> >> victim, OOM-killer will continue to select the appropriate child 
> >> memcg downwards until the leaf memcg is selected.
> >
> >Parent memcg has more or equal memory charged than its child(ren) by
> >definition. Let me try to ask differently. Say you have the following
> >hierarchy
> >
> >		  root
> >		/     \
> >       container_A     container_B
> >     (oom.prot=100M)   (oom.prot=200M)
> >     (usage=120M)      (usage=180M)
> >     /     |     \
> >    A      B      C
> >                 / \
> >		C1  C2
> >
> >
> >container_B is protected so it should be excluded. Correct? So we are at
> >container_A to chose from. There are multiple ways the system and
> >continer admin might want to achieve.
> >1) system admin might want to shut down the whole container.
> >2) continer admin might want to shut the whole container down
> >3) cont. admin might want to shut down a whole sub group (e.g. C as it
> >   is a self contained workload and killing portion of it will put it into
> >   inconsistent state).
> >4) cont. admin might want to kill the most excess cgroup with tasks (i.e. a
> >   leaf memcg).
> >5) admin might want to kill a process in the most excess memcg.
> >
> >Now we already have oom.group thingy that can drive the group killing
> >policy but it is not really clear how you want to incorporate that to
> >the protection.
> >
> >Again, I think that an oom.protection makes sense but the semantic has
> >to be very carefully thought through because it is quite easy to create
> >corner cases and weird behavior. I also think that oom.group has to be
> >consistent with the protection.
> 
> The barebone semantic of the function implemented by my patch are 
> summarized as follows:
> Memcg only allows processes in the memcg to be selected by their 
> ancestor's OOM killer when the memory usage exceeds "oom.protect"

I am sure you would need to break this expectation if there is no such
memcg with tasks available or do you panic the system in that case in
the global case and retry for ever for the memcg oom?

> It should be noted that "oom.protect" and "oom.group" are completely 
> different things, and kneading them together may make the explanation 
> more confusing.

I am not suggesting to tight those two together by any means. I am
merely saying that those two have to be mutually cooperative and still
represent a reasonable semantic. Please have a look at above example
usecases and try to explain how the memory protection fits in here as
you have defined and implemented it.

-- 
Michal Hocko
SUSE Labs

Powered by blists - more mailing lists