linux-kernel - Circular debugging using ptrace results in deadlock due to race condition?

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [day] [month] [year] [list]

Date:   Thu, 08 Jun 2017 16:08:54 +0200
From:   Ilja Nevolin <Ilja.Nevolin@...nt.be>
To:     linux-kernel@...r.kernel.org
Subject: Circular debugging using ptrace results in deadlock due to race
 condition?

Hi guys,

As part of my master's thesis I am facing a challenging problem.
I am trying to let two processes be each other's debuggers using the  
ptrace syscall. However, my proof-of-concept implementation always  
results in a deadlock state (both processes get stuck in 't+' state as  
shown by 'ps aux').

Here is my code, it's pretty simple:
https://pastebin.com/A1iBA3nh

I have compiled and run this on an ARMv7 developer board with kernel  
version 3.0.35 (Linaro 13.08).
The output of the above code is this:

   A    waiting to continue...
   B    attachTo: 0
   B    setOptions: 0
   B    setVarData: 0
   B    setVarData: 0
   B    cont: 0
   B    waiting to continue...
   B    waiting to continue...
   A    attachTo: 0

As you can see it never reaches the "finished" printf code, and gets  
stuck as soon as the other process attempts to attach to the debugger.

I have done a similar experiment for 3 processes, such that each one  
attempts to attach to the other in a circular fashion: A -> B -> C -> A
The result in this case was exactly the same. However, here I was able  
to detect a race condition, because sometimes the code executed  
properly without getting stuck in a deadlock (but it's hard to  
reproduce).

If you wish, you can test this by using a lightweight debugger I've  
developed and three console terminals. Here's the code:
https://pastebin.com/fPJb8ZNb

Once you've compiled the above code, you simply run the binary on each  
console and enter the PID of another process to establish a 3-way  
circle.

I am far from an expert on the kernel, but I did have a look at the  
ARM specific kernel implementation which left me puzzled. I couldn't  
find where/how/why this code does not work.
Now I'm wondering if it's possible at all to make this work without a  
deadlock occurring? Does anyone have any experience with this, or can  
provide some clues/feedback?

Thank you greatly for your time, attention and effort!
Ilya Nevolin
ilja.nevolin@...nt.be