linux-kernel - Re: [PATCH v1] sched: fix nohz idle load balancer issues

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <1317167376.11592.53.camel@sbsiddha-desk.sc.intel.com>
Date:	Tue, 27 Sep 2011 16:49:36 -0700
From:	Suresh Siddha <suresh.b.siddha@...el.com>
To:	Venki Pallipadi <venki@...gle.com>
Cc:	Srivatsa Vaddagiri <vatsa@...ux.vnet.ibm.com>,
	Peter Zijlstra <a.p.zijlstra@...llo.nl>,
	Paul Turner <pjt@...gle.com>, Ingo Molnar <mingo@...e.hu>,
	Vaidyanathan Srinivasan <svaidy@...ux.vnet.ibm.com>,
	Kamalesh Babulal <kamalesh@...ux.vnet.ibm.com>,
	"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>
Subject: Re: [PATCH v1] sched: fix nohz idle load balancer issues

On Tue, 2011-09-27 at 12:53 -0700, Venki Pallipadi wrote:
> On Mon, Sep 26, 2011 at 4:50 AM, Srivatsa Vaddagiri
> <vatsa@...ux.vnet.ibm.com> wrote:
> >
> > Reviewing idle load balancer code and testing it with some
> > trace_printk(), I observed the following:
> >
> > 1. I had put a trace_printk() in nohz_idle_balance() as below:
> >
> >        nohz_idle_balance()
> >        {
> >
> >        if (idle != CPU_IDLE || !this_rq->nohz_balance_kick)
> >                return;
> >
> >        ..
> >
> >                trace_printk("Running rebalance for %d\n", balance_cpu);
> >
> >                rebalance_domains(balance_cpu, CPU_IDLE);
> >        }
> >
> >    I *never* got that printed during the test. Further investigation
> >    revealed that ilb_cpu was bailing out early because idle =
> >    CPU_NOT_IDLE i.e ilb_cpu was no longer idle_at_tick by the time it
> >    got around to handle the kick. As a result, no one was truly
> >    doing a load balance on behalf of sleeping idle cpus.

One of the reasons why we saw lib_cpu not idle is probably because that
info was stale.

Consider this scenario.

a. got a tick when the cpu was busy, so idle_at_tick was not set
b. cpu went idle
c. same cpu got the kick IPI from other busy cpu
d. and as it has idle_at_tick not set, it couldn't proceed with the nohz
idle balance.

> > This patch is an attempt to solve above issues observed with idle load
> > balancer.
> >
> > - The patch causes a ilb_cpu (that was kicked) to kick another idle
> >  cpu in case it was found to be !idle_at_tick (so that another idle cpu
> >  can do load balance on behalf of idle cpus). This fixes issue #1
> 
> Some comments:
> 
> Another potential change here is to
> - either reverse the order of rebalance_domains() and
> nohz_idle_balance() in run_rebalance_domains()
> - or to kick another idle CPU in case of need_resched() in nohz_idle_balance.
> This should help with idle balance of tickless CPUs when ilb CPU gets
> a new task through load balance and hence aborts ilb.

I think we are mostly likely seeing the above mentioned scenario.

Also Vatsa, there is a deadlock associated by using
__smp_call_funciton_single() in the nohz_balancer_kick(). So I am
planning to remove the IPI that is used to kick the nohz balancer and
instead use the resched_cpu logic to kick the nohz balancer.

I will post this patch mostly tomorrow. That patch will not use the
idle_at_tick check in the nohz_idle_balance(). So that should address
your issue in some cases if not most.

thanks,
suresh

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/