linux-kernel - Re: [PATCH 0/3] mm/mmap.c: don't unmap the overlapping VMA(s)

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Thu, 7 Apr 2016 18:11:29 +0200
From:	Piotr Kwapulinski <kwapulinski.piotr@...il.com>
To:	Vlastimil Babka <vbabka@...e.cz>
Cc:	Michal Hocko <mhocko@...nel.org>, akpm@...ux-foundation.org,
	mtk.manpages@...il.com, cmetcalf@...lanox.com, arnd@...db.de,
	viro@...iv.linux.org.uk, mszeredi@...e.cz, dave@...olabs.net,
	kirill.shutemov@...ux.intel.com, mingo@...nel.org,
	dan.j.williams@...el.com, dave.hansen@...ux.intel.com,
	koct9i@...il.com, hannes@...xchg.org, jack@...e.cz,
	xiexiuqi@...wei.com, iamjoonsoo.kim@....com, oleg@...hat.com,
	gang.chen.5i5j@...il.com, aarcange@...hat.com,
	aryabinin@...tuozzo.com, rientjes@...gle.com, denc716@...il.com,
	toshi.kani@....com, ldufour@...ux.vnet.ibm.com,
	kuleshovmail@...il.com, linux-kernel@...r.kernel.org,
	linux-mm@...ck.org, linux-arch@...r.kernel.org
Subject: Re: [PATCH 0/3] mm/mmap.c: don't unmap the overlapping VMA(s)

On Mon, Apr 04, 2016 at 05:26:43PM +0200, Vlastimil Babka wrote:
> On 04/04/2016 09:31 AM, Michal Hocko wrote:
> >On Sat 02-04-16 21:17:31, Piotr Kwapulinski wrote:
> >>Currently the mmap(MAP_FIXED) discards the overlapping part of the
> >>existing VMA(s).
> >>Introduce the new MAP_DONTUNMAP flag which forces the mmap to fail
> >>with ENOMEM whenever the overlapping occurs and MAP_FIXED is set.
> >>No existing mapping(s) is discarded.
> >
> >You forgot to tell us what is the use case for this new flag.
> 
> Exactly. Also, returning ENOMEM is strange, EINVAL might be a better match,
> otherwise how would you distinguish a "geunine" ENOMEM from passing a wrong
> address?
> 
> 

Thanks to all for suggestions. I'll fix them.

The example use case:
#include <stdio.h>
#include <string.h>
#include <sys/mman.h>

void main(void)
{
  void* addr = (void*)0x1000000;
  size_t size = 0x600000;
  void* start = 0;
  start = mmap(addr,
               size,
               PROT_WRITE,
               MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED,
               -1, 0);

  strcpy(start, "PPPP");
  printf("%s\n", start);        // == PPPP

  addr = (void*)0x1000000;
  size = 0x9000;
  start = mmap(addr,
               size,
               PROT_READ | PROT_WRITE,
               MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED,
               -1, 0);
  
  printf("%s\n", start);        // != PPPP
}

Another use case, this time with huge pages in action.
The limit configured in proc's nr_hugepages is exceeded.
mmap unmaps the area and fails. No new mapping is created.
The program segfaults.

echo 0 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages

#include <stdio.h>
#include <string.h>
#include <sys/mman.h>
#include <unistd.h>

void main(void)
{
  void* addr = (void*)0x1000000;
  size_t size = 0x600000;
  void* start = 0;
  start = mmap(addr,
               size,
               PROT_WRITE,
               MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED,
               -1, 0);

  strcpy(start, "PPPP");
  printf("%s\n", start);        // == PPPP

  addr = (void*)0x1000000;
  size = 0x400000;
  start = mmap(addr,
               size,
               PROT_READ | PROT_WRITE,
               MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED | MAP_HUGETLB,
               -1, 0);         // mmap fails but unmaps the area

  printf("%s\n", start);       // segfault
}

Piotr