linux-kernel - Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:   Sun, 27 Nov 2016 00:32:52 -0500
From:   "Christopher S. Aker" <caker@...shore.net>
To:     Michal Hocko <mhocko@...nel.org>
Cc:     Donald Buczek <buczek@...gen.mpg.de>, dvteam@...gen.mpg.de,
        Paul Menzel <pmenzel@...gen.mpg.de>, linux-xfs@...r.kernel.org,
        linux-kernel@...r.kernel.org, Josh Triplett <josh@...htriplett.org>
Subject: Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`


> On Nov 24, 2016, at 5:15 AM, Michal Hocko <mhocko@...nel.org> wrote:
> 
>> * No rcu_* warnings on that machine with 4.7.2, but with 4.8.4 , 4.8.6 ,
>> 4.8.8 and now 4.9.0-rc5+Pauls patch
> 
> I assume you haven't tried the Linus 4.8 kernel without any further
> stable patches? Just to be sure we are not talking about some later
> regression which found its way to the stable tree.

We are also seeing this frequently on our fleet since moving from 4.7.x to 4.8. This is from a machine running vanilla 4.8.6 just a few moments ago:

INFO: rcu_sched detected stalls on CPUs/tasks:
	13-...: (420 ticks this GP) idle=ce1/140000000000000/0 softirq=225550784/225550904 fqs=87105 
	(detected by 26, t=600030 jiffies, g=68185325, c=68185324, q=344996)
Task dump for CPU 13:
kswapd1         R  running task    12200  1840      2 0x00000808
 0000000000000001 0000000000000034 000000000000012b 0000000000003139
 ffff8b643fffb000 ffff8b028cee7cf8 ffff8b028cee7cf8 ffff8b028cee7d08
 ffff8b028cee7d08 ffff8b028cee7d18 ffff8b028cee7d18 ffff8b0200000000
Call Trace:
 [] ? shrink_node+0xcd/0x2f0
 [] ? kswapd+0x304/0x710
 [] ? mem_cgroup_shrink_node+0x160/0x160
 [] ? kthread+0xc4/0xe0
 [] ? ret_from_fork+0x1f/0x40
 [] ? kthread_worker_fn+0x140/0x140

The machine will lag terribly during these occurrences .. some will eventually recover, some will spiral down and require a reboot.

-Chris