Message-ID: <C5E5137F-8754-40CC-9F0C-0EB3D8AC1EC2@didiglobal.com>
Date:   Sun, 4 Jun 2023 08:05:53 +0000
From:   程垲涛 Chengkaitao Cheng 
        <chengkaitao@...iglobal.com>
To:     Michal Hocko <mhocko@...e.com>
CC:     "tj@...nel.org" <tj@...nel.org>,
        "lizefan.x@...edance.com" <lizefan.x@...edance.com>,
        "hannes@...xchg.org" <hannes@...xchg.org>,
        "corbet@....net" <corbet@....net>,
        "roman.gushchin@...ux.dev" <roman.gushchin@...ux.dev>,
        "shakeelb@...gle.com" <shakeelb@...gle.com>,
        "akpm@...ux-foundation.org" <akpm@...ux-foundation.org>,
        "brauner@...nel.org" <brauner@...nel.org>,
        "muchun.song@...ux.dev" <muchun.song@...ux.dev>,
        "viro@...iv.linux.org.uk" <viro@...iv.linux.org.uk>,
        "zhengqi.arch@...edance.com" <zhengqi.arch@...edance.com>,
        "ebiederm@...ssion.com" <ebiederm@...ssion.com>,
        "Liam.Howlett@...cle.com" <Liam.Howlett@...cle.com>,
        "chengzhihao1@...wei.com" <chengzhihao1@...wei.com>,
        "pilgrimtao@...il.com" <pilgrimtao@...il.com>,
        "haolee.swjtu@...il.com" <haolee.swjtu@...il.com>,
        "yuzhao@...gle.com" <yuzhao@...gle.com>,
        "willy@...radead.org" <willy@...radead.org>,
        "vasily.averin@...ux.dev" <vasily.averin@...ux.dev>,
        "vbabka@...e.cz" <vbabka@...e.cz>,
        "surenb@...gle.com" <surenb@...gle.com>,
        "sfr@...b.auug.org.au" <sfr@...b.auug.org.au>,
        "mcgrof@...nel.org" <mcgrof@...nel.org>,
        "feng.tang@...el.com" <feng.tang@...el.com>,
        "cgroups@...r.kernel.org" <cgroups@...r.kernel.org>,
        "linux-doc@...r.kernel.org" <linux-doc@...r.kernel.org>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        "linux-fsdevel@...r.kernel.org" <linux-fsdevel@...r.kernel.org>,
        "linux-mm@...ck.org" <linux-mm@...ck.org>
Subject: Re: [PATCH v3 0/2] memcontrol: support cgroup level OOM protection

At 2023-05-29 22:02:47, "Michal Hocko" <mhocko@...e.com> wrote:
>On Thu 25-05-23 07:35:41, 程垲涛 Chengkaitao Cheng wrote:
>> At 2023-05-22 21:03:50, "Michal Hocko" <mhocko@...e.com> wrote:
>[...]
>> >> I have created a new indicator oom_kill_inherit that maintains a negative correlation 
>> >> with memory.oom.protect, so we have a ruler to measure the optimal value of 
>> >> memory.oom.protect.
>> >
>> >An example might help here.
>> 
>> In my testing case, by adjusting memory.oom.protect, I was able to significantly 
>> reduce the oom_kill_inherit of the corresponding cgroup. In a physical machine 
>> with severely oversold memory, I divided all cgroups into three categories and 
>> controlled their probability of being selected by the oom-killer to 0%,% 20, 
>> and 80%, respectively.
>
>I might be just dense but I am lost. Can we focus on the barebone
>semantic of the group oom selection and killing first. No magic
>auto-tuning at this stage please.
>
>> >> >> about the semantics of non-leaf memcgs protection,
>> >> >> If a non-leaf memcg's oom_protect quota is set, its leaf memcg will proportionally 
>> >> >> calculate the new effective oom_protect quota based on non-leaf memcg's quota.
>> >> >
>> >> >So the non-leaf memcg is never used as a target? What if the workload is
>> >> >distributed over several sub-groups? Our current oom.group
>> >> >implementation traverses the tree to find a common ancestor in the oom
>> >> >domain with the oom.group.
>> >> 
>> >> If the oom_protect quota of the parent non-leaf memcg is less than the sum of 
>> >> sub-groups oom_protect quota, the oom_protect quota of each sub-group will 
>> >> be proportionally reduced
>> >> If the oom_protect quota of the parent non-leaf memcg is greater than the sum 
>> >> of sub-groups oom_protect quota, the oom_protect quota of each sub-group 
>> >> will be proportionally increased
>> >> The purpose of doing so is that users can set oom_protect quota according to 
>> >> their own needs, and the system management process can set appropriate 
>> >> oom_protect quota on the parent non-leaf memcg as the final cover, so that 
>> >> the system management process can indirectly manage all user processes.
>> >
>> >I guess that you are trying to say that the oom protection has a
>> >standard hierarchical behavior. And that is fine, well, in fact it is
>> >mandatory for any control knob to have a sane hierarchical properties.
>> >But that doesn't address my above question. Let me try again. When is a
>> >non-leaf memcg potentially selected as the oom victim? It doesn't have
>> >any tasks directly but it might be a suitable target to kill a multi
>> >memcg based workload (e.g. a full container).
>> 
>> If nonleaf memcg have the higher memory usage and the smaller 
>> memory.oom.protect, it will have the higher the probability being 
>> selected by the killer. If the non-leaf memcg is selected as the oom 
>> victim, OOM-killer will continue to select the appropriate child 
>> memcg downwards until the leaf memcg is selected.
>
>Parent memcg has more or equal memory charged than its child(ren) by
>definition. Let me try to ask differently. Say you have the following
>hierarchy
>
>		  root
>		/     \
>       container_A     container_B
>     (oom.prot=100M)   (oom.prot=200M)
>     (usage=120M)      (usage=180M)
>     /     |     \
>    A      B      C
>                 / \
>		C1  C2
>
>
>container_B is protected so it should be excluded. Correct? So we are at
>container_A to chose from. There are multiple ways the system and
>continer admin might want to achieve.
>1) system admin might want to shut down the whole container.
>2) continer admin might want to shut the whole container down
>3) cont. admin might want to shut down a whole sub group (e.g. C as it
>   is a self contained workload and killing portion of it will put it into
>   inconsistent state).
>4) cont. admin might want to kill the most excess cgroup with tasks (i.e. a
>   leaf memcg).
>5) admin might want to kill a process in the most excess memcg.
>
>Now we already have oom.group thingy that can drive the group killing
>policy but it is not really clear how you want to incorporate that to
>the protection.
>
>Again, I think that an oom.protection makes sense but the semantic has
>to be very carefully thought through because it is quite easy to create
>corner cases and weird behavior. I also think that oom.group has to be
>consistent with the protection.

The barebone semantic of the function implemented by my patch are 
summarized as follows:
Memcg only allows processes in the memcg to be selected by their 
ancestor's OOM killer when the memory usage exceeds "oom.protect"

It should be noted that "oom.protect" and "oom.group" are completely 
different things, and kneading them together may make the explanation 
more confusing.

>> >> >All that being said and with the usecase described more specifically. I
>> >> >can see that memcg based oom victim selection makes some sense. That
>> >> >menas that it is always a memcg selected and all tasks withing killed.
>> >> >Memcg based protection can be used to evaluate which memcg to choose and
>> >> >the overall scheme should be still manageable. It would indeed resemble
>> >> >memory protection for the regular reclaim.
>> >> >
>> >> >One thing that is still not really clear to me is to how group vs.
>> >> >non-group ooms could be handled gracefully. Right now we can handle that
>> >> >because the oom selection is still process based but with the protection
>> >> >this will become more problematic as explained previously. Essentially
>> >> >we would need to enforce the oom selection to be memcg based for all
>> >> >memcgs. Maybe a mount knob? What do you think?
>> >> 
>> >> There is a function in the patch to determine whether the oom_protect 
>> >> mechanism is enabled. All memory.oom.protect nodes default to 0, so the function 
>> >> <is_root_oom_protect> returns 0 by default.
>> >
>> >How can an admin determine what is the current oom detection logic?
>> 
>> The memory.oom.protect are set by the administrator themselves, and they 
>> must know what the current OOM policy is. Reading the memory.oom.protect 
>> of the first level cgroup directory and observing whether it is 0 can also 
>> determine whether the oom.protect policy is enabled.
>
>How do you achieve that from withing a container which doesn't have a
>full visibility to the cgroup tree?

The container does not need to have full visibility to the cgroup tree, function 
"oom.protect" requires it to only focus on local cgroup and subgroup trees.

--
Thanks for your comment!
chengkaitao

Powered by blists - more mailing lists