linux-kernel - Re: rcu: performance regression

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20110614125612.GE2264@linux.vnet.ibm.com>
Date:	Tue, 14 Jun 2011 05:56:12 -0700
From:	"Paul E. McKenney" <paulmck@...ux.vnet.ibm.com>
To:	Ingo Molnar <mingo@...e.hu>
Cc:	Shaohua Li <shaohua.li@...el.com>,
	lkml <linux-kernel@...r.kernel.org>,
	"Chen, Tim C" <tim.c.chen@...el.com>,
	"Shi, Alex" <alex.shi@...el.com>,
	Linus Torvalds <torvalds@...ux-foundation.org>,
	Peter Zijlstra <a.p.zijlstra@...llo.nl>,
	Thomas Gleixner <tglx@...utronix.de>
Subject: Re: rcu: performance regression

On Tue, Jun 14, 2011 at 10:13:15AM +0200, Ingo Molnar wrote:
> 
> * Shaohua Li <shaohua.li@...el.com> wrote:
> 
> > Commit a26ac2455ffcf3(rcu: move TREE_RCU from softirq to kthread)
> > introduced performance regression. In our AIM7 test, this commit caused
> > about 40% regression.
> 
> Sigh, this commit is somewhat of a train-wreck.
> 
> > The commit runs rcu callbacks in a kthread instead of softirq. We 
> > observed high rate of context switch which is caused by this. Out 
> > test system has 64 CPUs and HZ is 1000, so we saw more than 64k 
> > context switch per second which is caused by the rcu thread.
> >
> > I also did trace and found when rcy thread is woken up, most time 
> > the thread doesn't handle any callbacks actually, it just 
> > initializes new gp or end one gp or similar.
> >
> > From my understanding, the purpose to make rcu runs in kthread is 
> > to speed up rcu callbacks run (with help of rtmutex PI), not for 
> > end gp and so on, which runs pretty fast actually and doesn't need 
> > boost. To verify my findings, I had below debug patch applied. It 
> > still handles rcu callbacks in kthread if there is any pending 
> > callbacks, but other things are still running in softirq. this 
> > completely solved our regression. I thought this can still boost 
> > callbacks run. but I'm not expert in the area, so please help.
> > 
> > Thanks,
> > Shaohua
> > ---
> >  Documentation/filesystems/proc.txt  |    1 +
> >  include/linux/interrupt.h           |    1 +
> >  include/trace/events/irq.h          |    3 ++-
> >  kernel/rcutree.c                    |   23 +++++++++++++++++++----
> >  kernel/rcutree.h                    |    1 +
> >  kernel/rcutree_plugin.h             |    9 +++++++++
> >  kernel/softirq.c                    |    2 +-
> >  tools/perf/util/trace-event-parse.c |    1 +
> >  8 files changed, 35 insertions(+), 6 deletions(-)
> 
> Paul? Unless this patch is the obviously correct solution everyone 
> wants to have, the other obviously correct solution is to do the 
> revert ...

I will look Shaohua's patch over.  Of course, given that mid-90s
could do well in excess of 100,000 context switches per second
per CPU, I am having a hard time seeing how 1,000 context switches
per second per CPU is by itself resulting in a 40% regression.

Nevertheless, fewer context switches per second should speed things
up, and so again, I will look at Shaohua's patch.

							Thanx, Paul
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/