[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <tip-c0fcb6c2d332041256dc55d8a1ec3c0a2d0befb8@git.kernel.org>
Date: Fri, 3 Jun 2016 03:55:45 -0700
From: tip-bot for Jason Low <tipbot@...or.com>
To: linux-tip-commits@...r.kernel.org
Cc: heiko.carstens@...ibm.com, tony.luck@...el.com,
tim.c.chen@...ux.intel.com, paulmck@...ux.vnet.ibm.com,
dave@...olabs.net, akpm@...ux-foundation.org,
torvalds@...ux-foundation.org, ink@...assic.park.msu.ru,
hpa@...or.com, arnd@...db.de, schwidefsky@...ibm.com,
peterz@...radead.org, cl@...ux.com, linux-kernel@...r.kernel.org,
jason.low2@....com, tglx@...utronix.de, fenghua.yu@...el.com,
peter@...leysoftware.com, jason.low2@...com, rth@...ddle.net,
mattst88@...il.com, Waiman.Long@....com, terry.rudd@....com,
mingo@...nel.org
Subject: [tip:locking/core] locking/rwsem: Optimize write lock by reducing
operations in slowpath
Commit-ID: c0fcb6c2d332041256dc55d8a1ec3c0a2d0befb8
Gitweb: http://git.kernel.org/tip/c0fcb6c2d332041256dc55d8a1ec3c0a2d0befb8
Author: Jason Low <jason.low2@....com>
AuthorDate: Mon, 16 May 2016 17:38:00 -0700
Committer: Ingo Molnar <mingo@...nel.org>
CommitDate: Fri, 3 Jun 2016 09:47:13 +0200
locking/rwsem: Optimize write lock by reducing operations in slowpath
When acquiring the rwsem write lock in the slowpath, we first try
to set count to RWSEM_WAITING_BIAS. When that is successful,
we then atomically add the RWSEM_WAITING_BIAS in cases where
there are other tasks on the wait list. This causes write lock
operations to often issue multiple atomic operations.
We can instead make the list_is_singular() check first, and then
set the count accordingly, so that we issue at most 1 atomic
operation when acquiring the write lock and reduce unnecessary
cacheline contention.
Signed-off-by: Jason Low <jason.low2@....com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@...radead.org>
Acked-by: Waiman Long<Waiman.Long@....com>
Acked-by: Davidlohr Bueso <dave@...olabs.net>
Cc: Andrew Morton <akpm@...ux-foundation.org>
Cc: Arnd Bergmann <arnd@...db.de>
Cc: Christoph Lameter <cl@...ux.com>
Cc: Fenghua Yu <fenghua.yu@...el.com>
Cc: Heiko Carstens <heiko.carstens@...ibm.com>
Cc: Ivan Kokshaysky <ink@...assic.park.msu.ru>
Cc: Jason Low <jason.low2@...com>
Cc: Linus Torvalds <torvalds@...ux-foundation.org>
Cc: Martin Schwidefsky <schwidefsky@...ibm.com>
Cc: Matt Turner <mattst88@...il.com>
Cc: Paul E. McKenney <paulmck@...ux.vnet.ibm.com>
Cc: Peter Hurley <peter@...leysoftware.com>
Cc: Peter Zijlstra <peterz@...radead.org>
Cc: Richard Henderson <rth@...ddle.net>
Cc: Terry Rudd <terry.rudd@....com>
Cc: Thomas Gleixner <tglx@...utronix.de>
Cc: Tim Chen <tim.c.chen@...ux.intel.com>
Cc: Tony Luck <tony.luck@...el.com>
Link: http://lkml.kernel.org/r/1463445486-16078-2-git-send-email-jason.low2@hpe.com
Signed-off-by: Ingo Molnar <mingo@...nel.org>
---
kernel/locking/rwsem-xadd.c | 25 ++++++++++++++++++-------
1 file changed, 18 insertions(+), 7 deletions(-)
diff --git a/kernel/locking/rwsem-xadd.c b/kernel/locking/rwsem-xadd.c
index fcbf75a..b957da7 100644
--- a/kernel/locking/rwsem-xadd.c
+++ b/kernel/locking/rwsem-xadd.c
@@ -261,17 +261,28 @@ struct rw_semaphore __sched *rwsem_down_read_failed(struct rw_semaphore *sem)
}
EXPORT_SYMBOL(rwsem_down_read_failed);
+/*
+ * This function must be called with the sem->wait_lock held to prevent
+ * race conditions between checking the rwsem wait list and setting the
+ * sem->count accordingly.
+ */
static inline bool rwsem_try_write_lock(long count, struct rw_semaphore *sem)
{
/*
- * Try acquiring the write lock. Check count first in order
- * to reduce unnecessary expensive cmpxchg() operations.
+ * Avoid trying to acquire write lock if count isn't RWSEM_WAITING_BIAS.
*/
- if (count == RWSEM_WAITING_BIAS &&
- cmpxchg_acquire(&sem->count, RWSEM_WAITING_BIAS,
- RWSEM_ACTIVE_WRITE_BIAS) == RWSEM_WAITING_BIAS) {
- if (!list_is_singular(&sem->wait_list))
- rwsem_atomic_update(RWSEM_WAITING_BIAS, sem);
+ if (count != RWSEM_WAITING_BIAS)
+ return false;
+
+ /*
+ * Acquire the lock by trying to set it to ACTIVE_WRITE_BIAS. If there
+ * are other tasks on the wait list, we need to add on WAITING_BIAS.
+ */
+ count = list_is_singular(&sem->wait_list) ?
+ RWSEM_ACTIVE_WRITE_BIAS :
+ RWSEM_ACTIVE_WRITE_BIAS + RWSEM_WAITING_BIAS;
+
+ if (cmpxchg_acquire(&sem->count, RWSEM_WAITING_BIAS, count) == RWSEM_WAITING_BIAS) {
rwsem_set_owner(sem);
return true;
}
Powered by blists - more mailing lists