linux-kernel - [PATCH-tip 1/3] locking/rwsem: Check wait

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:   Wed, 22 Feb 2017 13:03:52 -0500
From:   Waiman Long <longman@...hat.com>
To:     Ingo Molnar <mingo@...hat.com>,
        Peter Zijlstra <peterz@...radead.org>
Cc:     linux-kernel@...r.kernel.org, Davidlohr Bueso <dave@...olabs.net>,
        Waiman Long <longman@...hat.com>
Subject: [PATCH-tip 1/3] locking/rwsem: Check wait_list without lock if spinner present

We can safely check the wait_list to see if waiters are present without
lock when there are spinners to fall back on in case we miss a waiter.
The advantage is that we can save a pair of spin_lock/unlock calls
when the wait_list is empty. This translates to a reduction in latency
and hence slightly better performance.

On a 2-socket 36-core 72-thread x86-64 E5-2699 v3 system, a rwsem
microbenchmark was run with 36 locking threads (one/core) doing 1
million writer lock/unlock operations each, the resulting locking
rates (avg of 4 runs) on a 4.10 kernel were 7,755 Mop/s and 8,276
Mop/s without and with the patch respectively. That was an increase
of about 7%.

Signed-off-by: Waiman Long <longman@...hat.com>
---
 kernel/locking/rwsem-xadd.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/kernel/locking/rwsem-xadd.c b/kernel/locking/rwsem-xadd.c
index 34e727f..b5d7055 100644
--- a/kernel/locking/rwsem-xadd.c
+++ b/kernel/locking/rwsem-xadd.c
@@ -611,6 +611,17 @@ struct rw_semaphore *rwsem_wake(struct rw_semaphore *sem)
 		 * state is consulted before reading the wait_lock.
 		 */
 		smp_rmb();
+
+		/*
+		 * Normally checking wait_list without wait_lock isn't safe
+		 * as we may miss an incoming waiter. With spinners present,
+		 * however, we have someone to fall back on in case that
+		 * happens. This can save a pair of spin_lock/unlock calls
+		 * when there is no waiter.
+		 */
+		if (list_empty(&sem->wait_list))
+			return sem;
+
 		if (!raw_spin_trylock_irqsave(&sem->wait_lock, flags))
 			return sem;
 		goto locked;
-- 
1.8.3.1