linux-kernel - Re: Mainline kernel OLTP performance update

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-Id: <1241028413.4529.9247.camel@localhost.localdomain>
Date:	Wed, 29 Apr 2009 11:06:53 -0700
From:	"Pallipadi, Venkatesh" <venkatesh.pallipadi@...el.com>
To:	Chris Mason <chris.mason@...cle.com>
Cc:	Peter Zijlstra <a.p.zijlstra@...llo.nl>,
	Andrew Morton <akpm@...ux-foundation.org>,
	"Styner, Douglas W" <douglas.w.styner@...el.com>,
	"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
	"Tripathi, Sharad C" <sharad.c.tripathi@...el.com>,
	"arjan@...ux.intel.com" <arjan@...ux.intel.com>,
	"Wilcox, Matthew R" <matthew.r.wilcox@...el.com>,
	"Kleen, Andi" <andi.kleen@...el.com>,
	"Siddha, Suresh B" <suresh.b.siddha@...el.com>,
	"Ma, Chinang" <chinang.ma@...el.com>,
	"Wang, Peter Xihong" <peter.xihong.wang@...el.com>,
	"Nueckel, Hubert" <hubert.nueckel@...el.com>,
	"Recalde, Luis F" <luis.f.recalde@...el.com>,
	"Nelson, Doug" <doug.nelson@...el.com>,
	"Cheng, Wu-sun" <wu-sun.cheng@...el.com>,
	"Prickett, Terry O" <terry.o.prickett@...el.com>,
	"Shunmuganathan, Rajalakshmi" <rajalakshmi.shunmuganathan@...el.com>,
	"Garg, Anil K" <anil.k.garg@...el.com>,
	"Chilukuri, Harita" <harita.chilukuri@...el.com>,
	Ingo Molnar <mingo@...e.hu>
Subject: Re: Mainline kernel OLTP performance update

On Wed, 2009-04-29 at 10:46 -0700, Chris Mason wrote:
> On Wed, 2009-04-29 at 18:25 +0200, Peter Zijlstra wrote:
> > On Wed, 2009-04-29 at 09:07 -0700, Andrew Morton wrote:
> > > On Wed, 29 Apr 2009 08:48:19 -0700 "Styner, Douglas W" <douglas.w.styner@...el.com> wrote:
> > > 
> > > > >-----Original Message-----
> > > > >From: Andrew Morton [mailto:akpm@...ux-foundation.org]
> > > > >Sent: Wednesday, April 29, 2009 12:30 AM
> > > > >To: Styner, Douglas W
> > > > >Cc: linux-kernel@...r.kernel.org; Tripathi, Sharad C;
> > > > >arjan@...ux.intel.com; Wilcox, Matthew R; Kleen, Andi; Siddha, Suresh B;
> > > > >Ma, Chinang; Wang, Peter Xihong; Nueckel, Hubert; Recalde, Luis F; Nelson,
> > > > >Doug; Cheng, Wu-sun; Prickett, Terry O; Shunmuganathan, Rajalakshmi; Garg,
> > > > >Anil K; Chilukuri, Harita; chris.mason@...cle.com
> > > > >Subject: Re: Mainline kernel OLTP performance update
> > > > >
> > > > >On Tue, 28 Apr 2009 10:08:22 -0700 "Styner, Douglas W"
> > > > ><douglas.w.styner@...el.com> wrote:
> > > > >
> > > > >> Summary: Measured the mainline kernel from kernel.org (2.6.30-rc3).
> > > > >>
> > > > >> The regression for 2.6.30-rc3 against the baseline, 2.6.24.2 is 1.91%.
> > > > >Oprofile reports 71.1626% user, 28.8295% system.
> > > > >>
> > > > >> Linux OLTP Performance summary
> > > > >> Kernel#            Speedup(x)   Intr/s  CtxSw/s us%     sys%    idle%
> > > > >iowait%
> > > > >> 2.6.24.2                1.000   22106   43709   75      24      0       0
> > > > >> 2.6.30-rc3              0.981   30645   43027   75      25      0       0
> > > > >
> > > > >The main difference there is the interrupt frequency.  Do we know which
> > > > >interrupt source(s) caused this?
> > > >
> > > > Our analysis of the interrupts shows that rescheduling interrupts are
> > > > up 2.2x from 2.6.24.2 --> 2.6.30-rc3.  Qla2xxx interrupts are roughly
> > > > the same.
> > > 
> > > (top-posting repaired)
> > > 
> > > OK, thanks.  Seems odd that the rescheduling interrupt rate increased
> > > while the context-switch rate actually fell a couple of percent.
> > > 
> > > This came up a few weeks ago and iirc Peter was mainly involved, and I
> > > don't believe that anything conclusive ended up happening.  Peter,
> > > could you please remind us of (and summarise) the story here?
> > 
> > I've had several reports about the resched-ipi going in overdrive, but
> > nobody bothered to bisect it, nor have I yet done so -- no clear ideas
> > on why it is doing so.
> > 
> > I'll put it somewhere higher on the todo list.
> > 
> 
> One cause of them in the past was the ondemand cpufreq module.  It got
> fixed up for my laptop workload at least starting w/2.6.29, but it might
> make sense to try without ondemand if you're running it.
> 

Output of
# grep . /sys/devices/system/cpu/cpu0/cpufreq/*
can tell us whether P-state software coordination is the reason behind
excessive resched IPIs. Look for ondemand being the current_governor and
affected_cpus containing more than one CPU in it.

Thanks,
Venki

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/