linux-kernel - Re: [RFC] Per file OOM badness

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:   Fri, 19 Jan 2018 12:37:51 +0100
From:   Christian König <christian.koenig@....com>
To:     Michal Hocko <mhocko@...nel.org>
Cc:     Eric Anholt <eric@...olt.net>,
        Andrey Grodzovsky <andrey.grodzovsky@....com>,
        linux-kernel@...r.kernel.org, linux-mm@...ck.org,
        dri-devel@...ts.freedesktop.org, amd-gfx@...ts.freedesktop.org
Subject: Re: [RFC] Per file OOM badness

Am 19.01.2018 um 11:40 schrieb Michal Hocko:
> On Fri 19-01-18 09:39:03, Christian König wrote:
>> Am 19.01.2018 um 09:20 schrieb Michal Hocko:
> [...]
>>> OK, in that case I would propose a different approach. We already
>>> have rss_stat. So why do not we simply add a new counter there
>>> MM_KERNELPAGES and consider those in oom_badness? The rule would be
>>> that such a memory is bound to the process life time. I guess we will
>>> find more users for this later.
>> I already tried that and the problem with that approach is that some buffers
>> are not created by the application which actually uses them.
>>
>> For example X/Wayland is creating and handing out render buffers to
>> application which want to use OpenGL.
>>
>> So the result is when you always account the application who created the
>> buffer the OOM killer will certainly reap X/Wayland first. And that is
>> exactly what we want to avoid here.
> Then you have to find the target allocation context at the time of the
> allocation and account it.

And exactly that's the root of the problem: The target allocation 
context isn't known at the time of the allocation.

We could add callbacks so that when the memory is passed from the 
allocator to the actual user of the memory. In other words when the 
memory is passed from the X server to the client the driver would need 
to decrement the X servers accounting and increment the clients accounting.

But I think that would go deep into the file descriptor handling (we 
would at least need to handle dup/dup2 and passing the fd using unix 
domain sockets) and most likely would be rather error prone.

The per file descriptor badness is/was just the much easier approach to 
solve the issue, because the drivers already knew which client is 
currently using which buffer objects.

I of course agree that file descriptors can be shared between processes 
and are by themselves not killable. But at least for our graphics driven 
use case I don't see much of a problem killing all processes when a file 
descriptor is used by more than one at the same time.

Regards,
Christian.

> As follow up emails show, implementations
> might differ and any robust oom solution have to rely on the common
> counters.