linux-kernel - Re: Question concerning RCU

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20150112194412.GE9719@linux.vnet.ibm.com>
Date:	Mon, 12 Jan 2015 11:44:12 -0800
From:	"Paul E. McKenney" <paulmck@...ux.vnet.ibm.com>
To:	"Stoidner, Christoph" <c.stoidner@...ero.de>
Cc:	"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>
Subject: Re: Question concerning RCU

On Mon, Jan 12, 2015 at 11:48:28AM +0000, Stoidner, Christoph wrote:
> Hi Paul,
> 
> > You got stack traces with the stall warnings, correct?  If so, please look
> > at them and at Documentation/RCU/stallwarn.txt and see if the kernel is
> > looping somewhere inappropriate.
> 
> Yes and no. I have a stack trace, but it is not generated by a stall warning. More
> precise: I can never see any stall warning. The reason is that the system freezes 
> when it is about to output such a warning. Instead the stack trace is generated 
> by gdb and JTAG hardware debugging, when freezing has occurred.
> 
> So I am not sure if there is really a CPU-stall condition or it is just a misrepresented
> stall detection. However, outputting a stall warning leads to system freeze. The 
> warning is never seen.

Two things to try:

1.	alt-sysreq-t to get all tasks' stacks, or
2.	disable RCU CPU stall warnings and see if the hangs go away.

Hmmm...  Are you by chance pushing all dmesg through a serial console?

> > I am not familiar with the low-level ARM kernel code, but the stack below
> > leads me to suspect that your kernel is interrupting itself to death or
> > is improperly handling interrupts.
> 
> The stack trace must be read from bottom to top. The repetitive occurrence of
> "__irq_svc () at arch/arm/kernel/entry-armv.S:202" on bottom of stack trace is 
> caused by the stack frame of the interrupt context. This is completely legal and 
> also the case in normal situations. Instead the problem is on the top of the stack 
> trace, in function rcu_print_task_stall(). The loop rcutree_plugin.h in line 528 
> never ends:
> 
> static int rcu_print_task_stall(struct rcu_node *rnp)
> {
> 	...
> 	...
> 
> 	list_for_each_entry_continue(t, &rnp->blkd_tasks, rcu_node_entry) {
> 		printk(KERN_CONT " P%d", t->pid);
> 		ndetected++;
> 	}
> 
> 	...    
> 	...
> }        
> 
> That means list_for_each_entry_continue () never ends since rcu_node_entry.next 
> seems to point to it-self but not to rnp->blkd_tasks. I have no idea how this can
> happen.

It is not supposed to happen, and I haven't heard of it happening
anywhere else.  I do hold the appropriate lock across that code.

One thing to try would be to add a counter and break out of the loop
after (say) 10 iterations.  Is that a change you are comfortable making?

> One more thing: Just for testing I have now enabled CONFIG_TINY_PREEMPT_RCU. 
> Until now the problem has not occurred anymore. Do you have any idea what makes
> the differences here?

Any number of things, including that I am not sure that your version
of CONFIG_TINY_PREEMPT_RCU correctly detects RCU CPU stalls.  ;-)
Please note that CONFIG_TINY_PREEMPT_RCU was removed a few versions ago.

							Thanx, Paul

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/