linux-kernel - [PATCH 5/6] locking/rwsem: Optimize slowpath/sleeping

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-Id: <1422257769-14083-6-git-send-email-dave@stgolabs.net>
Date:	Sun, 25 Jan 2015 23:36:08 -0800
From:	Davidlohr Bueso <dave@...olabs.net>
To:	Peter Zijlstra <peterz@...radead.org>,
	Ingo Molnar <mingo@...nel.org>
Cc:	"Paul E. McKenney" <paulmck@...ux.vnet.ibm.com>,
	Jason Low <jason.low2@...com>,
	Michel Lespinasse <walken@...gle.com>,
	Tim Chen <tim.c.chen@...ux.intel.com>,
	linux-kernel@...r.kernel.org, Davidlohr Bueso <dave@...olabs.net>,
	Davidlohr Bueso <dbueso@...e.de>
Subject: [PATCH 5/6] locking/rwsem: Optimize slowpath/sleeping

When blocking , we incur in multiple barriers when setting the
task's uninterruptable state. This is particularly bad when the
lock keeps getting stolen from the task trying to acquire the sem.
These changes propose delaying setting the task's new state until
we are sure that calling schedule is inevitable.

This implies that we do the trylock and active check (both basically
->counter checks) as TASK_RUNNING. For the trylock we hold the wait
lock with interrupts disabled, so no risk there. And for the active
check, the window for which we could get interrupted is quite small
and makes no tangible difference.

This patch increases Unixbench's 'execl' throughput by 25% on a 40
core machine.

Signed-off-by: Davidlohr Bueso <dbueso@...e.de>
---
 kernel/locking/rwsem-xadd.c | 20 +++++++++++++++-----
 1 file changed, 15 insertions(+), 5 deletions(-)

diff --git a/kernel/locking/rwsem-xadd.c b/kernel/locking/rwsem-xadd.c
index 18a50da..88b3468 100644
--- a/kernel/locking/rwsem-xadd.c
+++ b/kernel/locking/rwsem-xadd.c
@@ -459,17 +459,27 @@ struct rw_semaphore __sched *rwsem_down_write_failed(struct rw_semaphore *sem)
 		count = rwsem_atomic_update(RWSEM_WAITING_BIAS, sem);
 
 	/* wait until we successfully acquire the lock */
-	set_current_state(TASK_UNINTERRUPTIBLE);
 	while (true) {
 		if (rwsem_try_write_lock(count, sem))
 			break;
+
+		__set_current_state(TASK_UNINTERRUPTIBLE);
 		raw_spin_unlock_irq(&sem->wait_lock);
 
-		/* Block until there are no active lockers. */
-		do {
+		/*
+		 * When there are active locks after we wake up,
+		 * the lock was probably stolen from us. Thus,
+		 * go immediately back to sleep and avoid taking
+		 * the wait_lock.
+		 */
+		while (true) {
 			schedule();
-			set_current_state(TASK_UNINTERRUPTIBLE);
-		} while ((count = sem->count) & RWSEM_ACTIVE_MASK);
+
+			count = READ_ONCE(sem->count);
+			if (!(count & RWSEM_ACTIVE_MASK))
+				break;
+			__set_current_state(TASK_UNINTERRUPTIBLE);
+		}
 
 		raw_spin_lock_irq(&sem->wait_lock);
 	}
-- 
2.1.2

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/