linux-kernel - Re: mmap'ed memory in core files ?

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Wed, 2 Jul 2008 16:35:22 +1000
From:	"Rob Mueller" <robm@...tmail.fm>
To:	"Michael Kerrisk" <mtk.manpages@...glemail.com>,
	"Bron Gondwana" <brong@...tmail.fm>
Cc:	"Philippe De Muyter" <phdm@...qel.be>,
	<linux-kernel@...r.kernel.org>
Subject: Re: mmap'ed memory in core files ?

>> This is particularly interesting on a 64 bit kernel where a bug in
>> your code causes you to try to read something about 2Gb into your
>> alleged mmaped file (actual size ~500 bytes) and the segfault causes
>> a coredump.
>
> Do you have a ssimple example program for this?

Trying to reproduce the problem, I think it's actually related to sparse 
files.

$ cat a.c
#include <stdlib.h>
int main() {
  unsigned int i;
  char * a = malloc(3000000000u);
  *(int *)0 = 0;
}
$ gcc a.c
$ ulimit -c 10240 && ./a.out
$ ls -l
-rw------- 1 root root 3000082432 Jul  2 02:23 core.7761
$

It's clearly sparse, but slightly unintuitive that the ulimit doesn't 
actually limit the filesize, just the size of the data written to the file.

If I change the code to include this line after the malloc():

  for (i = 0; i < 3000000000u; i++) a[i] = i % 256;

I get:

-rw------- 1 root root   10485760 Jul  2 02:25 core.8992

More what you'd expect.

One interesting side effect of running a 64-bit kernel + 32-bit userland is 
that previously bugs that might have previously caused malloc() to fail (eg 
underflowing integer to a huge value), now succeed and allocate a huge chunk 
of memory rather than failing and causing the program to bailout/crash on 
dereference.

Rob

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/