linux-kernel - Re: rcu_preempt self-detected stall on CPU from 4.5-rc3, since 3.17

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20160217192817.GA21818@linux.vnet.ibm.com>
Date:	Wed, 17 Feb 2016 11:28:17 -0800
From:	"Paul E. McKenney" <paulmck@...ux.vnet.ibm.com>
To:	Ross Green <rgkernel@...il.com>
Cc:	linux-kernel@...r.kernel.org, mingo@...nel.org,
	jiangshanlai@...il.com, dipankar@...ibm.com,
	akpm@...ux-foundation.org,
	Mathieu Desnoyers <mathieu.desnoyers@...icios.com>,
	josh@...htriplett.org, tglx@...utronix.de, peterz@...radead.org,
	rostedt@...dmis.org, dhowells@...hat.com,
	Eric Dumazet <edumazet@...gle.com>, dvhart@...ux.intel.com,
	Frédéric Weisbecker <fweisbec@...il.com>,
	oleg@...hat.com, pranith kumar <bobby.prani@...il.com>
Subject: Re: rcu_preempt self-detected stall on CPU from 4.5-rc3, since 3.17

On Tue, Feb 16, 2016 at 09:45:49PM -0800, Paul E. McKenney wrote:
> On Tue, Feb 09, 2016 at 09:11:55PM +1100, Ross Green wrote:
> > Continued testing with the latest linux-4.5-rc3 release.
> > 
> > Please find attached a copy of traces from dmesg:
> > 
> > There is a lot more debug and trace data so hopefully this will shed
> > some light on what might be happening here.
> > 
> > My testing remains run a series of simple benchmarks, let that run to
> > completion and then leave the system idle away with just a few daemons
> > running.
> > 
> > the self detected stalls in this instance turned up after a days run time.
> > There were  NO heavy artificial computational loads on the machine.
> 
> It does indeed look quiet on that dmesg for a good long time.
> 
> The following insanely crude not-for-mainline hack -might- be producing
> good results in my testing.  It will take some time before I can claim
> statistically different results.  But please feel free to give it a go
> in the meantime.  (Thanks to Al Viro for pointing me in this direction.)

No joy, just a statistical anomaly.  :-(

							Thanx, Paul

> ------------------------------------------------------------------------
> 
> commit 0c2c8d9fd1641809830a7a75f84dcad69936ef56
> Author: Paul E. McKenney <paulmck@...ux.vnet.ibm.com>
> Date:   Tue Feb 16 15:42:36 2016 -0800
> 
>     rcu: Crude exploratory hack
>     
>     Signed-off-by: Paul E. McKenney <paulmck@...ux.vnet.ibm.com>
> 
> diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
> index 507d0ed48b97..5928e084620d 100644
> --- a/kernel/rcu/tree.c
> +++ b/kernel/rcu/tree.c
> @@ -2194,8 +2194,10 @@ static int __noreturn rcu_gp_kthread(void *arg)
>  					       READ_ONCE(rsp->gpnum),
>  					       TPS("fqswait"));
>  			rsp->gp_state = RCU_GP_WAIT_FQS;
> -			ret = wait_event_interruptible_timeout(rsp->gp_wq,
> -					rcu_gp_fqs_check_wake(rsp, &gf), j);
> +			ret = schedule_timeout_interruptible(j > 0 ? j : 1);
> +			rcu_gp_fqs_check_wake(rsp, &gf);
> +			// ret = wait_event_interruptible_timeout(rsp->gp_wq,
> +			//		rcu_gp_fqs_check_wake(rsp, &gf), j);
>  			rsp->gp_state = RCU_GP_DOING_FQS;
>  			/* Locking provides needed memory barriers. */
>  			/* If grace period done, leave loop. */