linux-kernel - Re: Futex hang/lockup problem in 2.6.30+ on AMD64

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20100112145213.GB3925@hack>
Date:	Tue, 12 Jan 2010 22:52:13 +0800
From:	Américo Wang <xiyou.wangcong@...il.com>
To:	Andrew Athan <linux_kernel_aathan@...eplex.com>
Cc:	linux-kernel@...r.kernel.org, Darren Hart <dvhltc@...ibm.com>,
	Peter Zijlstra <peterz@...radead.org>,
	Thomas Gleixner <tglx@...utronix.de>,
	Ingo Molnar <mingo@...e.hu>
Subject: Re: Futex hang/lockup problem in 2.6.30+ on AMD64

On Tue, Jan 12, 2010 at 04:18:07AM -0500, Andrew Athan wrote:
>
> After some investigation I believe I am experiencing a problem similar
> to the one described in this posting:
> http://sourceware.org/ml/libc-help/2009-10/msg00026.html, in that the
> poster suspects a problem in the futex implementation in 2.6.30 and
> above kernels.  In my case, the problem is not a soft lockup in the
> kernel, but it does result in an application lock up due to all threads
> waiting for futex's.
>
> For me this problem began to appear once I upgraded my Debian
> squeeze/testing x86_64 installation (AMD) to a new kernel.  I'm not
> sure what the prior kernel version was.  The same software running on
> different machines with earlier kernels (lenny) does not seem to
> experience the problem.
>
> I'm really not sure if this is a libc or kernel problem, but due to
> the stack trace, which shows what appears to be a hang on the internal
> __lock of the condition variable, it appears likely this is not an
> application bug.  Memory does not appear to be corrupt (I store
> sentinels around the mutexes, and they have retained their values).
>
> It appears that the cond var's __lock indicates there are waiters
> even though there are/should-be none (assuming I'm interpreting the
> __lock value of 2 correctly).  Since the __lock in question is a futex
> primitive, and it must be held regardless of other libc/nptl state  
> variables,
> I don't believe this is a libc problem.
>
> The problem occurs rarely, but innevitably, and sometimes only after
> several hours of normal program operation.  I have not yet
> successfully created a reduced test program that can faithfully
> reproduce the hang in a short timeframe.
>
> The application contains a thread pool where threads perform many
> operations between pthread calls but can be summarized as one of three
> cases below.  Due to the design of the thread pool, threads
> round-robbin or at least are randomly assigned a workload (in contrast
> to having one constant broadcast thread).
>
> case 1:  while(1){ *A* pthread_lock();pthread_unlock();}
> case 2:  pthread_lock();pthread_cond_wait();pthread_unlock();
> case 3:  pthread_lock(); *B* pthread_cond_broadcast();pthread_unlock();
>
> The application becomes hung with all threads but one stuck at *A*,
> and one thread at *B*.
>
> The stack trace and other details appear below.  I've saved the core
> file in case I can provide additional information.
>
>
> $ uname -a
> Linux UK22 2.6.30-2-amd64 #1 SMP Fri Sep 25 22:16:56 UTC 2009 x86_64  
> GNU/Linux

Hmm, thanks for reporting this here.

Adding futex experters into Cc...


-- 
Live like a child, think like the god.
 
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/