linux-kernel - Re: mprotect() failed: Cannot allocate memory

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Wed, 21 Apr 2010 10:22:19 +0200
From:	Yann Droneaud <yann@...neaud.fr>
To:	Paweł Sikora <pluto@...k.net>
Cc:	libc-help@...rceware.org, Mike Frysinger <vapier@...too.org>,
	linux-kernel@...r.kernel.org
Subject: Re: mprotect() failed: Cannot allocate memory

Le mercredi 21 avril 2010 à 01:44 +0200, Paweł Sikora a écrit :
> On Wednesday 21 April 2010 01:17:22 Mike Frysinger wrote:
> > On Tuesday 20 April 2010 19:05:20 Paweł Sikora wrote:
> > > i'm trying to debug an ugly application with ElectricFence.
> > 
> > electricfence does a lot of ugly memory tricks to do its thing, including,
> > but not limited to, overriding memory related symbols.  best to seek help
> > from the electricfence authors.
> 
> so, let's avoid EF and run following test:
> 
> #include <stdio.h>
> #include <stdlib.h>
> #include <unistd.h>
> #include <sys/mman.h>
> 
> void* my_alloc( size_t n )
> {
>         size_t ps = getpagesize();
>         printf( "request for %Zd bytes => ", n );
>         /* alloc PAGE_SIZE + n */
>         char* p = mmap( 0, ps + n, PROT_READ | PROT_WRITE, MAP_SHARED | 
> MAP_ANONYMOUS, -1, 0 );
>         if ( p == MAP_FAILED )
>                 __builtin_abort();
>         /* block guard page */
>         int rc = mprotect( p, ps, PROT_NONE );
>         if ( rc != 0 )
>                 __builtin_abort();
>         char* q = p + ps;
>         printf( "guard page @ %p, allocated region @ %p\n", p, q );
>         return q;
> }
> 
> int main()
> {
>         #define N 100  
>         size_t NN = 4*100*100;
>         size_t kmax = 100;
>         int i;
> 
>         double **bm = (double **)my_alloc( NN * sizeof( double* ) );
>         for( i = 0; i < NN; ++i )
>         {
>                 bm[ i ] = (double*)my_alloc( kmax * sizeof( double ) );
>         }
>         // leak...
>         return 0;
> }
> 
> and the result is...
> 
> (...)
> mmap(NULL, 4896, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_ANONYMOUS, -1, 0) = 
> 0x7f5fd97df000
> mprotect(0x7f5fd97df000, 4096, PROT_NONE) = -1 ENOMEM (Cannot allocate memory)

Have you checked available memory on your system ? Or user limit ?

You test program is going to allocate 
 79 + 1 pages for bm
 1 + 1 for each double arrays (x 40000)

So in the end your program is allocating 80080 pages, so about
312MBytes.

It not that big for a 64bits system.

Check limits such as 
-d	the maximum size of a process's data segment
-l	the maximum size a process may lock into memory
-m	the maximum resident set size

Regards.

-- 
Yann Droneaud


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/