linux-kernel - Re: [Bug #12650] Strange load average and ksoftirqd behavior with 2.6.29-rc2-git1

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20090217140130.GA6761@linux.vnet.ibm.com>
Date:	Tue, 17 Feb 2009 06:01:30 -0800
From:	"Paul E. McKenney" <paulmck@...ux.vnet.ibm.com>
To:	Ingo Molnar <mingo@...e.hu>
Cc:	Damien Wyart <damien.wyart@...e.fr>,
	Peter Zijlstra <a.p.zijlstra@...llo.nl>,
	Mike Galbraith <efault@....de>,
	Frédéric Weisbecker <fweisbec@...il.com>,
	"Rafael J. Wysocki" <rjw@...k.pl>,
	Linux Kernel Mailing List <linux-kernel@...r.kernel.org>,
	Kernel Testers List <kernel-testers@...r.kernel.org>
Subject: Re: [Bug #12650] Strange load average and ksoftirqd behavior with
	2.6.29-rc2-git1

On Tue, Feb 17, 2009 at 10:46:57AM +0100, Ingo Molnar wrote:
> 
> * Paul E. McKenney <paulmck@...ux.vnet.ibm.com> wrote:
> 
> > On Mon, Feb 16, 2009 at 02:39:44PM -0800, Paul E. McKenney wrote:
> > > On Mon, Feb 16, 2009 at 09:09:23PM +0100, Ingo Molnar wrote:
> > > > 
> > > > * Paul E. McKenney <paulmck@...ux.vnet.ibm.com> wrote:
> > > > 
> > > > > Here the calls to rcu_process_callbacks() are only 75 
> > > > > microseconds apart, so that this function is consuming more 
> > > > > than 10% of a CPU.  The strange thing is that I don't see a 
> > > > > raise_softirq() in between, though perhaps it gets inlined or 
> > > > > something that makes it invisible to ftrace.
> > > > 
> > > > look at the latest trace please, that has even the most inline 
> > > > raise-softirq method instrumented, so all the raising is 
> > > > visible.
> > > 
> > > Ah, my apologies!  This time looking at:
> > > 
> > > http://damien.wyart.free.fr/ksoftirqd_pb/trace_tip_2009.02.16_ksoftirqd_pb_abstime_proc.txt.gz
> > > 
> > > 
> > >   799.521187 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.521371 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.521555 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.521738 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.521934 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.522068 |   1)  ksoftir-2324  |               |                rcu_check_callbacks() {
> > >   799.522208 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.522392 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.522575 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.522759 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.522956 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.523074 |   1)  ksoftir-2324  |               |                  rcu_check_callbacks() {
> > >   799.523214 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.523397 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.523579 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.523762 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.523960 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.524079 |   1)  ksoftir-2324  |               |                  rcu_check_callbacks() {
> > >   799.524220 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.524403 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.524587 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > >   799.524770 |   1)    <idle>-0    |               |  rcu_check_callbacks() {
> > > [ . . . ]
> > > 
> > > Yikes!!!
> > > 
> > > Why is rcu_check_callbacks() being invoked so often?  It should be called
> > > but once per jiffy, and here it is called no less than 22 times in about
> > > 3.5 milliseconds, meaning one call every 160 microseconds or so.
> > 
> > BTW, the other question I have is "why do we need to call 
> > rcu_pending() and rcu_check_callbacks() from the idle loop of 
> > 32-bit x86, especially given that no other architecture does 
> > this?".  Don't get me wrong, it would be good to get rcutree's 
> > rcu_pending() to avoid spuriously saying that 
> > rcu_check_callbacks() should be invoked, so I would still like 
> > the trace with my patch, but...
> 
> There's no strong reason - we've been back and forth about RCU 
> in the dynticks code. Mind sending a test patch for Damien to 
> try?

But of course!  ;-)

The following patch removes the call to rcu_pending() and
rcu_check_callbacks() from the x86 32-bit idle loop in order to
reduce the softirq load on idle systems.

Signed-off-by: Paul E. McKenney <paulmck@...ux.vnet.ibm.com>
---

 process_32.c |    3 ---
 1 file changed, 3 deletions(-)

diff --git a/arch/x86/kernel/process_32.c b/arch/x86/kernel/process_32.c
index a546f55..bd4da2a 100644
--- a/arch/x86/kernel/process_32.c
+++ b/arch/x86/kernel/process_32.c
@@ -104,9 +104,6 @@ void cpu_idle(void)
 			check_pgt_cache();
 			rmb();
 
-			if (rcu_pending(cpu))
-				rcu_check_callbacks(cpu, 0);
-
 			if (cpu_is_offline(cpu))
 				play_dead();
 
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/