linux-kernel - Re: [regression] cpuidle_get_cpu

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <2092099.bR8HA2X8Pe@vostro.rjw.lan>
Date:	Tue, 18 Dec 2012 00:33:15 +0100
From:	"Rafael J. Wysocki" <rjw@...k.pl>
To:	Daniel Lezcano <daniel.lezcano@...aro.org>,
	Sivaram Nair <sivaramn@...dia.com>
Cc:	Russ Anderson <rja@....com>,
	Peter De Schrijver <pdeschrijver@...dia.com>,
	"akpm@...ux-foundation.org" <akpm@...ux-foundation.org>,
	"shuox.liu@...el.com" <shuox.liu@...el.com>,
	"yanmin_zhang@...el.com" <yanmin_zhang@...el.com>,
	"linux-pm@...r.kernel.org" <linux-pm@...r.kernel.org>,
	"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>
Subject: Re: [regression] cpuidle_get_cpu_driver livelocks idle system

On Monday, December 17, 2012 10:59:09 PM Daniel Lezcano wrote:
> On 12/17/2012 08:36 PM, Russ Anderson wrote:
> > The 3.7 kernel grinds to a halt on boot of a system with
> > 2048 cpus.  NMI showed most of the cpus in
> > _raw_spin_lock in cpuidle_get_cpu_driver().  (backtrace below)
> > 
> > A quick look at cpuidle_get_cpu_driver() shows the hot lock.
> > 
> > In drivers/cpuidle/driver.c:
> > --------------------------------------------------------
> > /**
> >  * cpuidle_get_cpu_driver - return the driver tied with a cpu
> >  */
> > struct cpuidle_driver *cpuidle_get_cpu_driver(struct cpuidle_device *dev)
> > {
> >         struct cpuidle_driver *drv;
> > 
> >         if (!dev)
> >                 return NULL;
> > 
> >         spin_lock(&cpuidle_driver_lock);
> >         drv = __cpuidle_get_cpu_driver(dev->cpu);
> >         spin_unlock(&cpuidle_driver_lock);
> > 
> >         return drv;
> > }
> > --------------------------------------------------------
> 
> Hi Russ,
> 
> thanks for investigating the problem. You are right, there is a
> bottleneck here.
> 
> Regarding how is used the cpuidle code, I think it is safe to remove the
> locks.

OK, a patch would be appreciated. :-)

If you prepare one, please explain in the changelog why it is safe to drop the
locks.

Thanks,
Rafael


> > This change was added in on Nov 14th, 2012.
> > http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=commit;h=bf4d1b5ddb78f86078ac6ae0415802d5f0c68f92
> > 
> > The patch says it adds support for cpus with different characteristics,
> > but adds a big global lock.  The comment claims "no impact for the other
> > platforms if the option is disabled", which leads me to believe the
> > spin_lock was added inadvertently.  CPU_IDLE_MULTIPLE_DRIVERS is off
> > in my config file.
> > 
> > linux$ grep CPU_IDLE_MULTIPLE_DRIVERS .config
> > # CONFIG_CPU_IDLE_MULTIPLE_DRIVERS is not set
> > 
> > As more cpus become idle, more cpus fight over the lock until
> > the system livelocks on the crushing weight of idle.
> > 
> > The fix may be to move the spin_lock into __cpuidle_get_cpu_driver,
> > which has different versions for CONFIG_CPU_IDLE_MULTIPLE_DRIVERS,
> > to avoid impacting the disabled case, or get rid of the spin_lock
> > all together.
> > 
> > 
> > --------------------------------------------------------
> > == UV NMI process trace cpu 12: ==
> > CPU 12
> > Pid: 0, comm: swapper/12 Tainted: G           O 3.7.0.rja-sgi+ #38
> > RIP: 0010:[<ffffffff81614e45>]  [<ffffffff81614e45>] _raw_spin_lock+0x25/0x30
> > [...]
> > Call Trace:
> >  [<ffffffff814c891c>] cpuidle_get_cpu_driver+0x1c/0x30
> >  [<ffffffff814c871d>] cpuidle_idle_call+0x7d/0x1b0
> >  [<ffffffff8101d08d>] cpu_idle+0xdd/0x130
> >  [<ffffffff8160a3ea>] start_secondary+0xc6/0xcc
> > --------------------------------------------------------
> > 
> 
> 
> 
-- 
I speak only for myself.
Rafael J. Wysocki, Intel Open Source Technology Center.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/