linux-kernel - Re: rcu_preempt self-detected stall on CPU from 4.5-rc3, since 3.17

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CANfgCY0SaXWkCFq=dHGG38AnFd3Rd+wvVGQ6TH9DYow881YUWA@mail.gmail.com>
Date:	Mon, 22 Feb 2016 05:15:56 +1100
From:	Ross Green <rgkernel@...il.com>
To:	Paul McKenney <paulmck@...ux.vnet.ibm.com>
Cc:	John Stultz <john.stultz@...aro.org>,
	Mathieu Desnoyers <mathieu.desnoyers@...icios.com>,
	Thomas Gleixner <tglx@...utronix.de>,
	Peter Zijlstra <peterz@...radead.org>,
	lkml <linux-kernel@...r.kernel.org>,
	Ingo Molnar <mingo@...nel.org>,
	Lai Jiangshan <jiangshanlai@...il.com>, dipankar@...ibm.com,
	Andrew Morton <akpm@...ux-foundation.org>,
	Josh Triplett <josh@...htriplett.org>,
	rostedt <rostedt@...dmis.org>,
	David Howells <dhowells@...hat.com>,
	Eric Dumazet <edumazet@...gle.com>,
	Darren Hart <dvhart@...ux.intel.com>,
	Frédéric Weisbecker <fweisbec@...il.com>,
	Oleg Nesterov <oleg@...hat.com>,
	pranith kumar <bobby.prani@...il.com>
Subject: Re: rcu_preempt self-detected stall on CPU from 4.5-rc3, since 3.17

On Sun, Feb 21, 2016 at 4:04 PM, Ross Green <rgkernel@...il.com> wrote:
> On Sat, Feb 20, 2016 at 5:32 PM, Paul E. McKenney
> <paulmck@...ux.vnet.ibm.com> wrote:
>> On Sat, Feb 20, 2016 at 03:34:30PM +1100, Ross Green wrote:
>>> On Sat, Feb 20, 2016 at 4:33 AM, Paul E. McKenney
>>> <paulmck@...ux.vnet.ibm.com> wrote:
>>> > On Thu, Feb 18, 2016 at 08:13:18PM -0800, John Stultz wrote:
>>> >> On Thu, Feb 18, 2016 at 7:56 PM, Ross Green <rgkernel@...il.com> wrote:
>>> >> > Well a bonus extra!
>>> >> > Kept everything running and there was another stall.
>>> >> > So i have included the demsg output for perusal.
>>> >> >
>>> >> > Just to clear things up there is no hotplug involved in this system.
>>> >> > It is a standard Pandaboard ES Ti4460 two processor system.
>>> >> > I use this for testing as a generic armv7 processor, plus can keep it
>>> >> > just running along for testing for a long time. the system has a total
>>> >> > of 23-25 process running on average. Mainly standard daemons. There is
>>> >> > certainly no heavy processing going on. I run a series of benchmarks
>>> >> > that are cpu intensive for the first 20 miinutes after boot and then
>>> >> > just leave it idle away. checking every so often to see how it has
>>> >> > gone.
>>> >> > As mentioned I have observed these stalls going back to 3.17 kernel.
>>> >> > It will often take up to a week to record such a stall. I will
>>> >> > typically test every new release kernel, so the -rc? series will get
>>> >> > around a weeks testing.
>>> >>
>>> >> Sorry. Kind of hopping in a bit late here. Is this always happening
>>> >> with just the pandaboard? Or are you seeing this on different
>>> >> machines?
>>> >>
>>> >> Have you tried enabling CONFIG_DEBUG_TIMEKEEPING just in case
>>> >> something is going awry there?
>>> >
>>> > Excellent point -- timekeeping issues have caused this sort of issue
>>> > in the past.
>>> >
>>> > Ross, on your next test, could you please enable CONFIG_DEBUG_TIMEKEEPING
>>> > as John suggests?
>>> >
>>> >                                                         Thanx, Paul
>>> >
>>> As John has suggested have already enabled CONFIG_DEBUG_TIMEKEEPING.
>>>
>>> So far just on 1 day running.
>>>
>>> Sigh...!! Nothing to report as yet, only one day on the clock.
>>> Its like watching grass grow!
>>
>> I hear you!  Though I was thinking in terms of watching paint dry...
>>
>>                                                         Thanx, Paul
>>
> Yes,
>
> but with paint drying there is an end point!
> Grass just keeps on growing ...
>
> More like the children in the back of the car ...
> Are we there yet? ...
>
> Well still nothing .. to report. I have just built a 4.5-rc5, but will
> wait till I get some outcome from the previous test. That can't be too
> much longer!
>
> In hope,
>
> Ross Green
Patience little ones ...

Well after 2 days plus running pulled another stall.
This is with 4.5-rc4 and CONFIG_DEBUG_TIMEKEEPING set.

Can't see anything related to the TIMEKEEPING.

Anyway here is the dmesg output for people to look at.

Paul, I was going to move onto 4.5-rc5 kernel, is there something else
that you want me to test with that, Anyone else have any suggestions
or ideas?

Regards,

Ross Green

Download attachment "dmesg-4.5-rc4-3" of type "application/octet-stream" (25170 bytes)