linux-kernel - Re: [PATCH] mm: add config option to select the initial overcommit mode

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <64a74ddc-c11b-75b9-c5f6-7e46be6f2122@gmail.com>
Date:	Tue, 17 May 2016 13:29:31 -0400
From:	"Austin S. Hemmelgarn" <ahferroin7@...il.com>
To:	Sebastian Frias <sf84@...oste.net>,
	Michal Hocko <mhocko@...nel.org>
Cc:	One Thousand Gnomes <gnomes@...rguk.ukuu.org.uk>,
	Mason <slash.tmp@...e.fr>, linux-mm@...ck.org,
	Andrew Morton <akpm@...ux-foundation.org>,
	Linus Torvalds <torvalds@...ux-foundation.org>,
	LKML <linux-kernel@...r.kernel.org>, bsingharora@...il.com
Subject: Re: [PATCH] mm: add config option to select the initial overcommit
 mode

On 2016-05-17 12:16, Sebastian Frias wrote:
> Hi Michal,
>
> On 05/17/2016 10:57 AM, Michal Hocko wrote:
>> On Tue 17-05-16 10:24:20, Sebastian Frias wrote:
>> [...]
>>>>> Also, under what conditions would copy-on-write fail?
>>>>
>>>> When you have no memory or swap pages free and you touch a COW page that
>>>> is currently shared. At that point there is no resource to back to the
>>>> copy so something must die - either the process doing the copy or
>>>> something else.
>>>
>>> Exactly, and why does "killing something else" makes more sense (or
>>> was chosen over) "killing the process doing the copy"?
>>
>> Because that "something else" is usually a memory hog and so chances are
>> that the out of memory situation will get resolved. If you kill "process
>> doing the copy" then you might end up just not getting any memory back
>> because that might be a little forked process which doesn't own all that
>> much memory on its own. That would leave you in the oom situation for a
>> long time until somebody actually sitting on some memory happens to ask
>> for CoW... See the difference?
>>
>
> I see the difference, your answer seems a bit like the one from Austin, basically:
> - killing a process is a sort of kernel protection attempting to deal "automatically" with some situation, like deciding what is a 'memory hog', or what is 'in infinite loop', "usually" in a correct way.
> It seems there's people who think its better to avoid having to take such decisions and/or they should be decided by the user, because "usually" != "always".
FWIW, it's really easy to see what's using a lot of memory, it's 
impossible to tell if something is stuck in an infinite loop without 
looking deep into the process state and possibly even at the source code 
(and even then it can be almost impossible to be certain).  This is why 
we have a OOM-Killer, and not a infinite-loop-killer.

Again I reiterate, if a system is properly provisioned (that is, if you 
have put in enough RAM and possibly swap space to do what you want to 
use it for), the only reason the OOM-killer should be invoked is due to 
a bug.  The non-default overcommit options still have the same issues 
they just change how and when they happen (overcommit=never will fire 
sooner, overcommit=always will fire later), and also can impact memory 
allocation performance (I have numbers somewhere that I can't find right 
now that demonstrated that overcommit=never gave more deterministic and 
(on average) marginally better malloc() performance, and simple logic 
would suggest that overcommit=always would make malloc() perform better 
too).
> And people who see that as a nice thing but complex thing to do.
> In this thread we've tried to explain why this heuristic (and/or OOM-killer) is/was needed and/or its history, which has been very enlightening by the way.
>
> From reading Documentation/cgroup-v1/memory.txt (and from a few replies here talking about cgroups), it looks like the OOM-killer is still being actively discussed, well, there's also "cgroup-v2".
> My understanding is that cgroup's memory control will pause processes in a given cgroup until the OOM situation is solved for that cgroup, right?
> If that is right, it means that there is indeed a way to deal with an OOM situation (stack expansion, COW failure, 'memory hog', etc.) in a better way than the OOM-killer, right?
> In which case, do you guys know if there is a way to make the whole system behave as if it was inside a cgroup? (*)
No, not with the process freeze behavior, because getting the group 
running again requires input from an external part of the system, which 
by definition doesn't exist if the group is the entire system; and, 
because our GUI isn't built into the kernel, we can't pause things and 
pop up a little dialog asking the user what to do to resolve the issue.