| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <040c0443-00ee-4622-bc8f-41ffb07f7fec@zmail15.collab.prod.int.phx2.redhat.com>
Date: Wed, 21 Dec 2011 10:58:01 -0500 (EST)
From: Ramon de C Valle <rcvalle@...hat.com>
To: linux-kernel@...r.kernel.org
Subject: brk() system call, RLIMIT_AS, and RLIMIT_DATA in 32-bit compatibility mode
Hi,
Today I came across an interesting behaviour and would like some enlightenment of what I'm probably missing.
Considering the following program:
.globl _start
_start:
xorl %ebx,%ebx
pushl $0xc0000000
popl %ebx
continue:
addl $0x01000000,%ebx
xorl %eax,%eax
movb $0x2d,%al
int $0x80
jmp continue
This is the strace output of the above program in a 32-bit machine:
[...]
brk(0xb9000000) = 0xb9000000
brk(0xba000000) = 0xba000000
brk(0xbb000000) = 0xbb000000
brk(0xbc000000) = 0xbc000000
brk(0xbd000000) = 0xbd000000
brk(0xbe000000) = 0xbe000000
brk(0xbf000000) = 0xbf000000
brk(0xc0000000) = 0xbf000000
brk(0xc1000000) = 0xbf000000
brk(0xc2000000) = 0xbf000000
brk(0xc3000000) = 0xbf000000
brk(0xc4000000) = 0xbf000000
brk(0xc5000000) = 0xbf000000
[...]
brk(0xf9000000) = 0xbf000000
brk(0xfa000000) = 0xbf000000
brk(0xfb000000) = 0xbf000000
brk(0xfc000000) = 0xbf000000
brk(0xfd000000) = 0xbf000000
brk(0xfe000000) = 0xbf000000
brk(0xff000000) = 0xbf000000
brk(0) = 0xbf000000
brk(0x1000000) = 0xbf000000
brk(0x2000000) = 0xbf000000
brk(0x3000000) = 0xbf000000
brk(0x4000000) = 0xbf000000
brk(0x5000000) = 0xbf000000
brk(0x6000000) = 0xbf000000
brk(0x7000000) = 0xbf000000
brk(0x8000000) = 0xbf000000
brk(0x9000000) = 0xbf000000
brk(0xa000000) = 0xa000000
[...]
Notice that from 0xc0000000 upwards it fails and returns the current break as expected. However, the following is the strace output of the same program in a 64-bit machine running in 32-bit compatibility mode:
[...]
brk(0xb9000000) = 0xffffffffb9000000
brk(0xba000000) = 0xffffffffba000000
brk(0xbb000000) = 0xffffffffbb000000
brk(0xbc000000) = 0xffffffffbc000000
brk(0xbd000000) = 0xffffffffbd000000
brk(0xbe000000) = 0xffffffffbe000000
brk(0xbf000000) = 0xffffffffbf000000
brk(0xc0000000) = 0xffffffffc0000000
brk(0xc1000000) = 0xffffffffc1000000
brk(0xc2000000) = 0xffffffffc2000000
brk(0xc3000000) = 0xffffffffc3000000
brk(0xc4000000) = 0xffffffffc4000000
brk(0xc5000000) = 0xffffffffc5000000
[...]
brk(0xf9000000) = 0xfffffffff9000000
brk(0xfa000000) = 0xfffffffffa000000
brk(0xfb000000) = 0xfffffffffb000000
brk(0xfc000000) = 0xfffffffffc000000
brk(0xfd000000) = 0xfffffffffd000000
brk(0xfe000000) = 0xfffffffffe000000
brk(0xff000000) = 0xffffffffff000000
brk(0) = 0xffffffffff000000
brk(0x1000000) = 0xffffffffff000000
brk(0x2000000) = 0xffffffffff000000
brk(0x3000000) = 0xffffffffff000000
brk(0x4000000) = 0xffffffffff000000
brk(0x5000000) = 0xffffffffff000000
brk(0x6000000) = 0xffffffffff000000
brk(0x7000000) = 0xffffffffff000000
brk(0x8000000) = 0xffffffffff000000
brk(0x9000000) = 0x9000000
brk(0xa000000) = 0xa000000
[...]
Notice that from 0xc0000000 upwards it continues expanding the program break above 0xc0000000 up to 0xff000000 where it also starts failing and returning the current break. Is this an expected behaviour when running an application in 64-bit machine in 32-bit compatibility mode? Is RLIMIT_AS and RLIMIT_DATA being honoured?
Thanks,
--
Ramon de C Valle / Red Hat Security Response Team
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Powered by blists - more mailing lists