If a task T holds rtmutex L and has pending waiters on that lock then an attempt of task T to recursivly lock L can escape the deadlock detection if T itself does not end up being the top most waiter on that lock. So it happily enqueues itself in the waiter list. This was exposed by Dave Jones trinity syscall fuzzer: http://lkml.kernel.org/r/20140429151655.GA14277@redhat.com The fix for the issue at hand is simple and more comment than actual code: Test whether the new waiter task owns one of the locks in the lock chain and handle it the same way as the other deadlock sites. The problem has been in the rtmutex code forever. I would have added a testcase for such a scenario to the rtmutex tester, but the tester got wreckaged with commit 8161239a8 (rtmutex: Simplify PI algorithm and make highest prio task get lock). So that becomes a separate issue. Sigh! Signed-off-by: Thomas Gleixner Cc: stable@vger.kernel.org --- kernel/locking/rtmutex.c | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) Index: linux-2.6/kernel/locking/rtmutex.c =================================================================== --- linux-2.6.orig/kernel/locking/rtmutex.c +++ linux-2.6/kernel/locking/rtmutex.c @@ -284,7 +284,7 @@ static int rt_mutex_adjust_prio_chain(st struct rt_mutex_waiter *orig_waiter, struct task_struct *top_task) { - struct rt_mutex *lock; + struct rt_mutex *lock = orig_lock; struct rt_mutex_waiter *waiter, *top_waiter = orig_waiter; int detect_deadlock, ret = 0, depth = 0; unsigned long flags; @@ -339,6 +339,22 @@ static int rt_mutex_adjust_prio_chain(st goto out_unlock_pi; /* + * Deadlock check for the following scenario: + * + * T holds lock L and has waiters + * T locks L again, but does not end up as it's own top waiter + * + * So we would drop out at the next check without noticing. + * + * Note, we need to check for orig_waiter as it might be NULL + * when deboosting! + */ + if (orig_waiter && orig_waiter->task == rt_mutex_owner(lock)) { + ret = deadlock_detect ? -EDEADLK : 0; + goto out_unlock_pi; + } + + /* * Drop out, when the task has no waiters. Note, * top_waiter can be NULL, when we are in the deboosting * mode! -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/