linux-kernel - Re: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks on v3.6

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Tue, 07 Aug 2012 12:24:50 +0200
From:	Sasha Levin <levinsasha928@...il.com>
To:	John Stultz <johnstul@...ibm.com>
CC:	Avi Kivity <avi@...hat.com>, paulmck@...ux.vnet.ibm.com,
	"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
	mingo@...nel.org, a.p.zijlstra@...llo.nl, prarit@...hat.com,
	tglx@...utronix.de, Dave Jones <davej@...hat.com>,
	Michael Wang <wangyun@...ux.vnet.ibm.com>
Subject: Re: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks on v3.6

On 08/07/2012 07:40 AM, John Stultz wrote:
> On 08/06/2012 11:28 AM, Sasha Levin wrote:
>> On 08/06/2012 08:20 PM, John Stultz wrote:
>>> On 08/06/2012 10:21 AM, John Stultz wrote:
>>>> On 08/05/2012 09:55 AM, Sasha Levin wrote:
>>>>> On 07/30/2012 03:17 PM, Avi Kivity wrote:
>>>>>> Possible causes:
>>>>>>    - the APIC calibration in the guest failed, so it is programming too
>>>>>> low values into the timer
>>>>>>    - it actually needs 1 us wakeups and then can't keep up (esp. as kvm
>>>>>> interrupt injection is slowing it down)
>>>>>>
>>>>>> You can try to find out by changing
>>>>>> arch/x86/kvm/lapic.c:start_lapic_timer() to impose a minimum wakeup of
>>>>>> (say) 20 microseconds which will let the guest live long enough for you
>>>>>> to ftrace it and see what kind of timers it is programming.
>>>>> I've kept trying to narrow it down, and found out It's triggerable using adjtimex().
>>> Sorry, one more question: Could you provide details on how is it trigger-able using adjtimex?
>> It triggers after a while of fuzzing using trinity of just adjtimex ('./trinity --quiet -l off -cadjtimex').
>>
>> Trinity is available here: http://git.codemonkey.org.uk/?p=trinity.git .
>>
>> Let me know if I can help further with reproducing this, I can probably copy over my testing environment to some other host if you'd like.
> 
> Ok. Finally I *think* got it reproduced.  (Had some trouble initially, as I think since the first time I ran it as a normal user, the socket cache isn't the same as if you run it the first time as root? Anyway, after doing a make clean and rebuilding it started to trigger).
> 
> I'm not seeing the rcu stall message, but I do manage to trigger two other behaviors: a hard hang and a sort of zombie state where memory isn't properly being freed & everything starts segfaulting.   So this may not be the exact same issue, but it triggers quickly as you described (within a few seconds of running trinity as root).
> 
> It looks like both of these issues are caused by adjtimex(ADJ_SETOFFSET), which  adds or subtracts a huge offset and that either goes negative or gets clamped to a ktime_t at KTIME_MAX (if you get clamped the system hangs, if it goes negative, the system barely functions, but sort of drags along).
> 
> An updated version of my KTIME_MAX sanity checking patch to handle both of these conditions is below.
> 
> Would you mind giving this patch a shot and letting me know if you still see problems?

This fixes the problem for me. Thanks!
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/