linux-kernel - Re: sched/core warning triggers on rcu torture test

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20180628163323.GB19886@lerouge>
Date:   Thu, 28 Jun 2018 18:33:24 +0200
From:   Frederic Weisbecker <frederic@...nel.org>
To:     "Paul E. McKenney" <paulmck@...ux.vnet.ibm.com>
Cc:     Peter Zijlstra <peterz@...radead.org>,
        Anna-Maria Gleixner <anna-maria@...utronix.de>,
        linux-kernel@...r.kernel.org, Thomas Gleixner <tglx@...utronix.de>,
        Frederic Weisbecker <fweisbec@...il.com>
Subject: Re: sched/core warning triggers on rcu torture test

On Wed, Jun 27, 2018 at 07:25:29AM -0700, Paul E. McKenney wrote:
> On Wed, Jun 27, 2018 at 12:40:15PM +0200, Frederic Weisbecker wrote:
> > On Tue, Jun 26, 2018 at 10:48:26AM -0700, Paul E. McKenney wrote:
> > > On Tue, Jun 26, 2018 at 06:32:55PM +0200, Peter Zijlstra wrote:
> > > > On Tue, Jun 26, 2018 at 06:16:04PM +0200, Anna-Maria Gleixner wrote:
> > > > > Hi,
> > > > > 
> > > > > during rcu torture tests (TREE04 and TREE07) I noticed, that a
> > > > > WARN_ON_ONCE() in sched core triggers on a recent 4.18-rc2 based
> > > > > kernel (6f0d349d922b ("Merge
> > > > > git://git.kernel.org/pub/scm/linux/kernel/git/davem/net")) as well as
> > > > > on a 4.17.3.
> > > 
> > > First, I am very glad that I am not the only one running rcutorture!  ;-)
> > > 
> > > > > I'm running the tests on a machine with 144 cores:
> > > > > 
> > > > >   tools/testing/selftests/rcutorture/bin/kvm.sh --cpus 144 --duration 120 --configs "9*TREE07"
> > > > >   tools/testing/selftests/rcutorture/bin/kvm.sh --cpus 144 --duration 120 --configs "18*TREE04"
> > > > > 
> > > > > 
> > > > > The warning was introduced by commit d84b31313ef8 ("sched/isolation:
> > > > > Offload residual 1Hz scheduler tick").
> > > > > 
> > > > > 
> > > > > Output looks similar for all tests I did (this one is the output of
> > > > > the 4.18-rc2 based kernel):
> > > > > 
> > > > > WARNING: CPU: 11 PID: 906 at kernel/sched/core.c:3138 sched_tick_remote+0xb6/0xc0
> > > > 
> > > > That's nohz_full stuff, is that a normal part of rcutorture? In any
> > > > case, is the one housekeeping CPU getting seriously overloaded or
> > > > something?
> > > 
> > > Yes, nohz_full is a normal part for rcutorture because RCU has to deal
> > > differently with userspace execution in the nohz_full case.
> > > 
> > > I do see this splat (at least when I don't comment it out), but I
> > > do share my system with others, so I could easily be overloading the
> > > housekeeping vCPUs due to hypervisor preemption.  I was intending to
> > > dig into this one once I got done consolidating RCU-bh, RCU-preempt,
> > > and RCU-sched at Linus's behest.
> > > 
> > > On overloading the housekeeping CPU without outside load, let's look at
> > > TREE04 and TREE07 separately.
> > > 
> > > TREE04 uses eight CPUs, and seven of them ("nohz_full=1-7") are nohz_full
> > > CPUs, and rcutorture doesn't generate all that large of a callback load.
> > > It looks like all 144 CPUs are used in this case (18*8), though RCU
> > > enforces idle periods in order to test idle/non-idle transitions.
> > > But was there anything else running on the machine at the time?
> > > 
> > > TREE07 uses 16 CPUs, and eight of them ("nohz_full=2-9") are nohz_full
> > > CPUs.  Again, it looks like all 144 CPUs are used (9*8).
> > > 
> > > I sometimes see this on TASKS03 as well, which uses two CPUs, and one of
> > > them ("nohz_full=1") is a nohz_full CPU.
> > > 
> > > If your system is otherwise idle, would it make sense to trace context
> > > switches on CPU 0 to see what it is up to?  And to do an ftrace_dump()
> > > and turn tracing off when the warning triggers as well?
> > 
> > Yeah you guys reported me this warning a few times ago. I didn't manage to reproduce
> > it because I fought and failed with a high NR_CPUS machine. But apparently 8 CPUs
> > are enough. Let me try that with TREE04.
> 
> Looking forward to hearing what you find!

Please check "[PATCH] sched/nohz: Skip remote tick on idle task entirely" which I
just posted. In the hope that the warning didn't trigger for another reason on
your testings.

Thanks.