linux-kernel - inconsistent behavior with ptrace(TRACEME) and fork/exec

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [day] [month] [year] [list]

Message-Id: <200810270455.32247.vapier@gentoo.org>
Date:	Mon, 27 Oct 2008 04:55:29 -0400
From:	Mike Frysinger <vapier@...too.org>
To:	linux-kernel@...r.kernel.org
Subject: inconsistent behavior with ptrace(TRACEME) and fork/exec

i'm hoping my understanding of ptrace is correct and that the attached test 
case (reduced from LTP) isnt just completely broken ... my understanding is 
that if a parent forks and the child does a ptrace(TRACEME) right before 
doing an exec(), the kernel should always halt it and wait indefinitely for 
the parent to start ptracing it.

unfortunately, this behavior seems to be unreliable.  most of the time it 
works, but sometimes the kernel does not halt the child and it gladly does 
the exec() that it set out to do.  i dont believe this to be a race in the 
user space parent component as forcing it to delay via judicious usage of 
sleep() shows the same behavior.

if  all goes well, we should only ever see "SUCCESS!" from the test case:
$ gcc -Wall ptrace-vfork-traceme.c -o ptrace-vfork-traceme
$ while ./ptrace-vfork-traceme ; do :; done
SUCCESS! :D
SUCCESS! :D
SUCCESS! :D
SUCCESS! :D
SUCCESS! :D
SUCCESS! :D
SUCCESS! :D
SUCCESS! :D
SUCCESS! :D
failure, child exited with 17: Child exited
wait() = 12275
status = 1407
        WIFEXITED = 0
        WEXITSTATUS = 5
        WIFSIGNALED = 0
        WTERMSIG = 127 (Unknown signal 127)

i'm testing 2.6.26/2.6.27 atm.  both x86_64 and ppc64 seem to behave the same.  
while gcc-4.3.2 / glibc-2.8 is in use in both places, i dont think that 
matters.

also, while the attached test uses vfork(), same behavior can be observed with 
fork().  vfork() is used because i like my test cases to work on both MMU and 
no-MMU systems.
-mike

Download attachment "signature.asc " of type "application/pgp-signature" (836 bytes)

View attachment "ptrace-vfork-traceme.c" of type "text/x-csrc" (1569 bytes)