linux-kernel - Re: [BUG -rt] Priority inversion deadlock caused by condvars

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <20080915092135.GA7725@in.ibm.com>
Date:	Mon, 15 Sep 2008 14:51:35 +0530
From:	Ankita Garg <ankita@...ibm.com>
To:	john stultz <johnstul@...ibm.com>
Cc:	drepper@...hat.com, Thomas Gleixner <tglx@...utronix.de>,
	mingo <mingo@...hat.com>, Steven Rostedt <rostedt@...dmis.org>,
	Dinakar Guniguntala <dino@...ibm.com>,
	Darren Hart <dvhltc@...ibm.com>,
	Sripathi Kodi <sripathi@...ibm.com>,
	lkml <linux-kernel@...r.kernel.org>
Subject: Re: [BUG -rt] Priority inversion deadlock caused by condvars

Hi All,

> On Fri, 2008-09-12 at 15:01 -0700, john stultz wrote:
> > 	So we've been seeing application hangs with a very threaded (~8k
> > threads) realtime java test. After a fair amount of debugging we found
> > most of the SCHED_FIFO threads are blocked in futex_wait(). This raised
> > some alarm, since futex_wait isn't priority-inheritance aware.
> > 
> > After seeing what was going on, Dino came up with a possible deadlock
> > case in the pthread_cond_wait() code.
> > 
> > The problem, as I understand it, assuming there is only one cpu, is if a
> > low priority thread is going to call pthread_cond_wait(), it takes the
> > associated PI mutex, and calls the function. The glibc implementation
> > acquires the condvar's internal non-PI lock, releases the PI mutex and
> > tries to block on futex_wait().
> > 
> > However if a medium priority cpu hog, and a high priority start up while
> > the low priority thread holds the mutex, the low priority thread will be
> > boosted until it releases that mutex, but not long enough for it to
> > release the condvar's internal lock (since the internal lock is not
> > priority inherited). 
> > 
> > Then the high priority thread will aquire the mutex, and try to acquire
> > the condvar's internal lock (which is still held). However, since we
> > also have a medium prio cpu hog, it will block the low priority thread
> > from running, and thus block it from releasing the lock.
> > 
> > And then we're deadlocked.
> > 
> > Thomas mentioned this is a known problem, but I wanted to send this
> > example out so maybe others might become aware.

Looks like a similar issue was raised sometime back.

http://sourceware.org/bugzilla/show_bug.cgi?id=5192

> > 
> > The attached test illustrates this hang as described above when bound to
> > a single cpu. I believe its correct, but these sorts of tests often have
> > their own bugs that create false positives, so please forgive me and let
> > me know if you see any problems. :)
> > 
> > Many thanks to Dino, Ankita and Sripathi for helping to sort out this
> > issue.
> > 
> > To run:
> > 	./pthread_cond_hang               => will PASS (on SMP)
> > 	taskset -c 0 ./pthread_cond_hang  => will HANG

-- 
Regards,
Ankita Garg (ankita@...ibm.com)
Linux Technology Center
IBM India Systems & Technology Labs, 
Bangalore, India   
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/