linux-kernel - Re: RCU stalls when running out of memory on 3.14-rc4 w/ NFS and kernel threads priorities changed

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CAGVrzcae0XPXpue_+n-O+EBzK92JqXHNftTPGt+5SRzroTSF3Q@mail.gmail.com>
Date:	Tue, 4 Mar 2014 19:55:03 -0800
From:	Florian Fainelli <f.fainelli@...il.com>
To:	Paul McKenney <paulmck@...ux.vnet.ibm.com>
Cc:	Eric Dumazet <eric.dumazet@...il.com>,
	"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
	linux-mm <linux-mm@...ck.org>,
	linux-nfs <linux-nfs@...r.kernel.org>,
	"trond.myklebust" <trond.myklebust@...marydata.com>,
	netdev <netdev@...r.kernel.org>
Subject: Re: RCU stalls when running out of memory on 3.14-rc4 w/ NFS and
 kernel threads priorities changed

2014-03-04 17:43 GMT-08:00 Paul E. McKenney <paulmck@...ux.vnet.ibm.com>:
> On Tue, Mar 04, 2014 at 05:16:27PM -0800, Florian Fainelli wrote:
>> 2014-03-04 17:03 GMT-08:00 Florian Fainelli <f.fainelli@...il.com>:
>> > 2014-03-04 16:48 GMT-08:00 Eric Dumazet <eric.dumazet@...il.com>:
>> >> On Tue, 2014-03-04 at 15:55 -0800, Florian Fainelli wrote:
>> >>> Hi all,
>> >>>
>> >>> I am seeing the following RCU stalls messages appearing on an ARMv7
>> >>> 4xCPUs system running 3.14-rc4:
>> >>>
>> >>> [   42.974327] INFO: rcu_sched detected stalls on CPUs/tasks:
>> >>> [   42.979839]  (detected by 0, t=2102 jiffies, g=4294967082,
>> >>> c=4294967081, q=516)
>> >>> [   42.987169] INFO: Stall ended before state dump start
>> >>>
>> >>> this is happening under the following conditions:
>> >>>
>> >>> - the attached bumper.c binary alters various kernel thread priorities
>> >>> based on the contents of bumpup.cfg and
>> >>> - malloc_crazy is running from a NFS share
>> >>> - malloc_crazy.c is running in a loop allocating chunks of memory but
>> >>> never freeing it
>> >>>
>> >>> when the priorities are altered, instead of getting the OOM killer to
>> >>> be invoked, the RCU stalls are happening. Taking NFS out of the
>> >>> equation does not allow me to reproduce the problem even with the
>> >>> priorities altered.
>> >>>
>> >>> This "problem" seems to have been there for quite a while now since I
>> >>> was able to get 3.8.13 to trigger that bug as well, with a slightly
>> >>> more detailed RCU debugging trace which points the finger at kswapd0.
>> >>>
>> >>> You should be able to get that reproduced under QEMU with the
>> >>> Versatile Express platform emulating a Cortex A15 CPU and the attached
>> >>> files.
>> >>>
>> >>> Any help or suggestions would be greatly appreciated. Thanks!
>> >>
>> >> Do you have a more complete trace, including stack traces ?
>> >
>> > Attatched is what I get out of SysRq-t, which is the only thing I have
>> > (note that the kernel is built with CONFIG_RCU_CPU_STALL_INFO=y):
>>
>> QEMU for Versatile Express w/ 2 CPUs yields something slightly
>> different than the real HW platform this is happening with, but it
>> does produce the RCU stall anyway:
>>
>> [  125.762946] BUG: soft lockup - CPU#1 stuck for 53s! [malloc_crazy:91]
>
> This soft-lockup condition can result in RCU CPU stall warnings.  Fix
> the problem causing the soft lockup, and I bet that your RCU CPU stall
> warnings go away.

I definitively agree, which is why I was asking for help, as I think
the kernel thread priority change is what is causing the soft lockup
to appear, but nothing obvious jumps to mind when looking at the
trace.

Thanks!
-- 
Florian
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/