linux-kernel - Re: [patch] mm, madvise: fail with ENOMEM when splitting vma will hit max_map

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Date:   Wed, 25 Jan 2017 14:14:55 -0800 (PST)
From:   David Rientjes <rientjes@...gle.com>
To:     Anshuman Khandual <khandual@...ux.vnet.ibm.com>
cc:     Andrew Morton <akpm@...ux-foundation.org>,
        Jonathan Corbet <corbet@....net>,
        Johannes Weiner <hannes@...xchg.org>, mtk.manpages@...il.com,
        Jerome Marchand <jmarchan@...hat.com>,
        "Kirill A. Shutemov" <kirill.shutemov@...ux.intel.com>,
        linux-man@...r.kernel.org, linux-kernel@...r.kernel.org,
        linux-mm@...ck.org
Subject: Re: [patch] mm, madvise: fail with ENOMEM when splitting vma will
 hit max_map_count

On Wed, 25 Jan 2017, Anshuman Khandual wrote:

> But in the due course there might be other changes in number of VMAs of
> the process because of unmap() or merge() which could reduce the total
> number of VMAs and hence this condition may not exist afterwards. In
> that case EAGAIN still makes sense.
> 

Imagine a singlethreaded process that is operating on its own privately 
mapped memory.  Attempting to split an existing vma and meeting 
vm.max_map_count is not something that will be fixed by trying again, i.e. 
it is not helpful to loop when madvise() returns -1 with errno EAGAIN if 
vm.max_map_count will always be encountered.  The other cases where ENOMEM 
is blindly converted to EAGAIN is when slab allocation fails which can 
encounter external freeing, the meaning of "kernel resource is temporarily 
unavailable."  There is no such guarantee for vm.max_map_count, so ENOMEM 
clearly indicates the failure.

After this, it makes sense for userspace to loop for advice such as 
MADV_DONTNEED because we are actively freeing memory when EAGAIN is 
returned.  If we are meeting vm.max_map_count, this will infinitely loop.  
This is the case in tcmalloc and this patch addresses the issue when 
vm.max_map_count is low.