linux-kernel - Re: Performance regression from switching lock to rw-sem for anon-vma tree

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Wed, 26 Jun 2013 17:25:01 -0700
From:	Tim Chen <tim.c.chen@...ux.intel.com>
To:	Ingo Molnar <mingo@...nel.org>
Cc:	Ingo Molnar <mingo@...e.hu>,
	Andrea Arcangeli <aarcange@...hat.com>,
	Mel Gorman <mgorman@...e.de>, "Shi, Alex" <alex.shi@...el.com>,
	Andi Kleen <andi@...stfloor.org>,
	Andrew Morton <akpm@...ux-foundation.org>,
	Michel Lespinasse <walken@...gle.com>,
	Davidlohr Bueso <davidlohr.bueso@...com>,
	"Wilcox, Matthew R" <matthew.r.wilcox@...el.com>,
	Dave Hansen <dave.hansen@...el.com>,
	Peter Zijlstra <a.p.zijlstra@...llo.nl>,
	Rik van Riel <riel@...hat.com>, linux-kernel@...r.kernel.org,
	linux-mm <linux-mm@...ck.org>
Subject: Re: Performance regression from switching lock to rw-sem for
 anon-vma tree

On Wed, 2013-06-26 at 14:36 -0700, Tim Chen wrote:
> On Wed, 2013-06-26 at 11:51 +0200, Ingo Molnar wrote: 
> > * Tim Chen <tim.c.chen@...ux.intel.com> wrote:
> > 
> > > On Wed, 2013-06-19 at 09:53 -0700, Tim Chen wrote: 
> > > > On Wed, 2013-06-19 at 15:16 +0200, Ingo Molnar wrote:
> > > > 
> > > > > > vmstat for mutex implementation: 
> > > > > > procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
> > > > > >  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
> > > > > > 38  0      0 130957920  47860 199956    0    0     0    56 236342 476975 14 72 14  0  0
> > > > > > 41  0      0 130938560  47860 219900    0    0     0     0 236816 479676 14 72 14  0  0
> > > > > > 
> > > > > > vmstat for rw-sem implementation (3.10-rc4)
> > > > > > procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
> > > > > >  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
> > > > > > 40  0      0 130933984  43232 202584    0    0     0     0 321817 690741 13 71 16  0  0
> > > > > > 39  0      0 130913904  43232 224812    0    0     0     0 322193 692949 13 71 16  0  0
> > > > > 
> > > > > It appears the main difference is that the rwsem variant context-switches 
> > > > > about 36% more than the mutex version, right?
> > > > > 
> > > > > I'm wondering how that's possible - the lock is mostly write-locked, 
> > > > > correct? So the lock-stealing from Davidlohr Bueso and Michel Lespinasse 
> > > > > ought to have brought roughly the same lock-stealing behavior as mutexes 
> > > > > do, right?
> > > > > 
> > > > > So the next analytical step would be to figure out why rwsem lock-stealing 
> > > > > is not behaving in an equivalent fashion on this workload. Do readers come 
> > > > > in frequently enough to disrupt write-lock-stealing perhaps?
> > > 
> > > Ingo, 
> > > 
> > > I did some instrumentation on the write lock failure path.  I found that
> > > for the exim workload, there are no readers blocking for the rwsem when
> > > write locking failed.  The lock stealing is successful for 9.1% of the
> > > time and the rest of the write lock failure caused the writer to go to
> > > sleep.  About 1.4% of the writers sleep more than once. Majority of the
> > > writers sleep once.
> > > 
> > > It is weird that lock stealing is not successful more often.
> > 
> > For this to be comparable to the mutex scalability numbers you'd have to 
> > compare wlock-stealing _and_ adaptive spinning for failed-wlock rwsems.
> > 
> > Are both techniques applied in the kernel you are running your tests on?
> > 
> 
> Ingo,
> 
> The previous experiment was done on a kernel without spinning. 
> I've redone the testing on two kernel for a 15 sec stretch of the
> workload run.  One with the adaptive (or optimistic) 
> spinning and the other without.  Both have the patches from Alex to avoid 
> cmpxchg induced cache bouncing.
> 
> With the spinning, I sleep much less for lock acquisition (18.6% vs 91.58%).
> However, I've got doubling of write lock acquisition getting
> blocked.  So that offset the gain from spinning which may be why
> I didn't see gain for this particular workload.
> 
> 						No Opt Spin	Opt Spin
> Writer acquisition blocked count		3448946		7359040
> Blocked by reader				0.00%		0.55%
> Lock acquired first attempt (lock stealing)	8.42%		16.92%
> Lock acquired second attempt (1 sleep)		90.26%		17.60%
> Lock acquired after more than 1 sleep		1.32%		1.00%
> Lock acquired with optimistic spin		N/A		64.48%
> 

Adding also the mutex statistics for the 3.10-rc4 kernel with mutex
implemenation of lock for anon_vma tree.  Wonder if Ingo has any
insight on why mutex performs better from these stats.

Mutex acquisition blocked count			14380340
Lock acquired in slowpath (no sleep)		0.06%
Lock acquired in slowpath (1 sleep)		0.24%
Lock acquired in slowpath more than 1 sleep	0.98%
Lock acquired with optimistic spin		99.6%

Thanks.

Tim

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/