netdev - Re: RCU stalls when running out of memory on 3.14-rc4 w/ NFS and kernel threads priorities changed

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Tue, 4 Mar 2014 17:03:24 -0800
From:	Florian Fainelli <f.fainelli@...il.com>
To:	Eric Dumazet <eric.dumazet@...il.com>
Cc:	"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
	linux-mm <linux-mm@...ck.org>,
	paulmck <paulmck@...ux.vnet.ibm.com>,
	linux-nfs <linux-nfs@...r.kernel.org>,
	"trond.myklebust" <trond.myklebust@...marydata.com>,
	netdev <netdev@...r.kernel.org>
Subject: Re: RCU stalls when running out of memory on 3.14-rc4 w/ NFS and
 kernel threads priorities changed

2014-03-04 16:48 GMT-08:00 Eric Dumazet <eric.dumazet@...il.com>:
> On Tue, 2014-03-04 at 15:55 -0800, Florian Fainelli wrote:
>> Hi all,
>>
>> I am seeing the following RCU stalls messages appearing on an ARMv7
>> 4xCPUs system running 3.14-rc4:
>>
>> [   42.974327] INFO: rcu_sched detected stalls on CPUs/tasks:
>> [   42.979839]  (detected by 0, t=2102 jiffies, g=4294967082,
>> c=4294967081, q=516)
>> [   42.987169] INFO: Stall ended before state dump start
>>
>> this is happening under the following conditions:
>>
>> - the attached bumper.c binary alters various kernel thread priorities
>> based on the contents of bumpup.cfg and
>> - malloc_crazy is running from a NFS share
>> - malloc_crazy.c is running in a loop allocating chunks of memory but
>> never freeing it
>>
>> when the priorities are altered, instead of getting the OOM killer to
>> be invoked, the RCU stalls are happening. Taking NFS out of the
>> equation does not allow me to reproduce the problem even with the
>> priorities altered.
>>
>> This "problem" seems to have been there for quite a while now since I
>> was able to get 3.8.13 to trigger that bug as well, with a slightly
>> more detailed RCU debugging trace which points the finger at kswapd0.
>>
>> You should be able to get that reproduced under QEMU with the
>> Versatile Express platform emulating a Cortex A15 CPU and the attached
>> files.
>>
>> Any help or suggestions would be greatly appreciated. Thanks!
>
> Do you have a more complete trace, including stack traces ?

Attatched is what I get out of SysRq-t, which is the only thing I have
(note that the kernel is built with CONFIG_RCU_CPU_STALL_INFO=y):

Thanks!
-- 
Florian

View attachment "rcu_stall_3.14-rc4_arm_sysrq_t.txt" of type "text/plain" (40969 bytes)