linux-kernel - Re: [PATCH] Repeated fork() causes SLAB to grow without bound

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Mon, 24 Nov 2014 11:09:40 +0400
From:	Konstantin Khlebnikov <koct9i@...il.com>
To:	Rik van Riel <riel@...hat.com>
Cc:	Michel Lespinasse <walken@...gle.com>,
	Vlastimil Babka <vbabka@...e.cz>,
	Andrew Morton <akpm@...ux-foundation.org>,
	Hugh Dickins <hughd@...gle.com>,
	Andrea Arcangeli <aarcange@...hat.com>,
	Linux Kernel Mailing List <linux-kernel@...r.kernel.org>,
	"linux-mm@...ck.org" <linux-mm@...ck.org>,
	Tim Hartrick <tim@...ecast.com>, Michal Hocko <mhocko@...e.cz>
Subject: Re: [PATCH] Repeated fork() causes SLAB to grow without bound

On Thu, Nov 20, 2014 at 6:03 PM, Konstantin Khlebnikov <koct9i@...il.com> wrote:
> On Thu, Nov 20, 2014 at 5:50 PM, Rik van Riel <riel@...hat.com> wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> On 11/20/2014 09:42 AM, Konstantin Khlebnikov wrote:
>>
>>> I'm thinking about limitation for reusing anon_vmas which might
>>> increase performance without breaking asymptotic estimation of
>>> count anon_vma in the worst case. For example this heuristic: allow
>>> to reuse only anon_vma with single direct descendant. It seems
>>> there will be arount up to two times more anon_vmas but
>>> false-aliasing must be much lower.

Done. RFC patch in attachment.

This patch adds heuristic which decides to reuse existing anon_vma instead
of forking new one. It counts vmas and direct descendants for each anon_vma.
Anon_vma with degree lower than two will be reused at next fork.
As a result each anon_vma has either alive vma or at least two descendants,
endless chains are no longer possible and count of anon_vmas is no more than
two times more than count of vmas.


>>
>> It may even be possible to not create a child anon_vma for the
>> first child a parent forks, but only create a new anon_vma once
>> the parent clones a second child (alive at the same time as the
>> first child).
>>
>> That still takes care of things like apache or sendmail, but
>> would not create infinite anon_vmas for a task that keeps forking
>> itself to infinite depth without calling exec...
>
> But this scheme is still exploitable. Malicious software easily could create
> sequence of forks and exits which leads to infinite chain of anon_vmas.
>
>>
>> - --
>> All rights reversed
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG v1
>>
>> iQEcBAEBAgAGBQJUbf+hAAoJEM553pKExN6DxhQH/1QL+9GdhaSx7EQnRcbDRcHi
>> GuEfMU0g9Kv4ad+oPSQnH/L7vJMJAYeh5ZJGH+rOykWHp3sGReqDZOnzpXRAe11z
>> 1cSC1BJsndzrv9wX8niFpuKpYbF0IP+ckv3qaEzWtm5yCRyhHVZfr6b794Y4K9jF
>> z2EPPu1vAAldbkx1VlYTwofBA5lESL5UmrFvH4ouI7BeWYSEe6BgVCbvK+K5fANT
>> ketdA5R08xyUAcXDa+28qpBYkdWnxNhwqseDoXCW8SOFNwWbLDI6GRfrsCNku13i
>> Gi41h3uEuIAGDf+AU/GMjiymgwutCOGq+cfZlszELaRvHmDpNGYdPv1llghNg7Q=
>> =Vk+H
>> -----END PGP SIGNATURE-----

Download attachment "mm-prevent-endless-growth-of-anon_vma-hierarchy" of type "application/octet-stream" (5437 bytes)