netdev - Re: RCU stalls when running out of memory on 3.14-rc4 w/ NFS and kernel threads priorities changed

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20140305053440.GD3334@linux.vnet.ibm.com>
Date:	Tue, 4 Mar 2014 21:34:40 -0800
From:	"Paul E. McKenney" <paulmck@...ux.vnet.ibm.com>
To:	Florian Fainelli <f.fainelli@...il.com>
Cc:	Eric Dumazet <eric.dumazet@...il.com>,
	"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
	linux-mm <linux-mm@...ck.org>,
	linux-nfs <linux-nfs@...r.kernel.org>,
	"trond.myklebust" <trond.myklebust@...marydata.com>,
	netdev <netdev@...r.kernel.org>
Subject: Re: RCU stalls when running out of memory on 3.14-rc4 w/ NFS and
 kernel threads priorities changed

On Tue, Mar 04, 2014 at 07:55:03PM -0800, Florian Fainelli wrote:
> 2014-03-04 17:43 GMT-08:00 Paul E. McKenney <paulmck@...ux.vnet.ibm.com>:
> > On Tue, Mar 04, 2014 at 05:16:27PM -0800, Florian Fainelli wrote:
> >> 2014-03-04 17:03 GMT-08:00 Florian Fainelli <f.fainelli@...il.com>:
> >> > 2014-03-04 16:48 GMT-08:00 Eric Dumazet <eric.dumazet@...il.com>:
> >> >> On Tue, 2014-03-04 at 15:55 -0800, Florian Fainelli wrote:
> >> >>> Hi all,
> >> >>>
> >> >>> I am seeing the following RCU stalls messages appearing on an ARMv7
> >> >>> 4xCPUs system running 3.14-rc4:
> >> >>>
> >> >>> [   42.974327] INFO: rcu_sched detected stalls on CPUs/tasks:
> >> >>> [   42.979839]  (detected by 0, t=2102 jiffies, g=4294967082,
> >> >>> c=4294967081, q=516)
> >> >>> [   42.987169] INFO: Stall ended before state dump start
> >> >>>
> >> >>> this is happening under the following conditions:
> >> >>>
> >> >>> - the attached bumper.c binary alters various kernel thread priorities
> >> >>> based on the contents of bumpup.cfg and
> >> >>> - malloc_crazy is running from a NFS share
> >> >>> - malloc_crazy.c is running in a loop allocating chunks of memory but
> >> >>> never freeing it
> >> >>>
> >> >>> when the priorities are altered, instead of getting the OOM killer to
> >> >>> be invoked, the RCU stalls are happening. Taking NFS out of the
> >> >>> equation does not allow me to reproduce the problem even with the
> >> >>> priorities altered.
> >> >>>
> >> >>> This "problem" seems to have been there for quite a while now since I
> >> >>> was able to get 3.8.13 to trigger that bug as well, with a slightly
> >> >>> more detailed RCU debugging trace which points the finger at kswapd0.
> >> >>>
> >> >>> You should be able to get that reproduced under QEMU with the
> >> >>> Versatile Express platform emulating a Cortex A15 CPU and the attached
> >> >>> files.
> >> >>>
> >> >>> Any help or suggestions would be greatly appreciated. Thanks!
> >> >>
> >> >> Do you have a more complete trace, including stack traces ?
> >> >
> >> > Attatched is what I get out of SysRq-t, which is the only thing I have
> >> > (note that the kernel is built with CONFIG_RCU_CPU_STALL_INFO=y):
> >>
> >> QEMU for Versatile Express w/ 2 CPUs yields something slightly
> >> different than the real HW platform this is happening with, but it
> >> does produce the RCU stall anyway:
> >>
> >> [  125.762946] BUG: soft lockup - CPU#1 stuck for 53s! [malloc_crazy:91]
> >
> > This soft-lockup condition can result in RCU CPU stall warnings.  Fix
> > the problem causing the soft lockup, and I bet that your RCU CPU stall
> > warnings go away.
> 
> I definitively agree, which is why I was asking for help, as I think
> the kernel thread priority change is what is causing the soft lockup
> to appear, but nothing obvious jumps to mind when looking at the
> trace.

Is your hardware able to make the malloc_crazy CPU periodically dump
its stack, perhaps in response to an NMI?  If not, another approach is
to use ftrace -- though this will require a very high-priority task to
turn tracing on and off reasonably quickly, unless you happen to have
a very large amount of storage to hold the trace.

What happens if you malloc() less intensively?  Does that avoid this
problem?  The reason I ask is that you mentioned that avoiding NFS helped,
and it is possible that NFS is increasing storage-access latencies and
thus triggering another problem.  It is quite possible that slowing
down the malloc()s would also help, and might allow you to observe what
is happening more easily than when the system is driven fully to the
lockup condition.

Finally, what are you trying to achieve with this workload?  Does your
production workload behave in this way?  Or is this an experimental
investigation of some sort?

							Thanx, Paul

--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html