linux-kernel - [PATCH] locking/rwsem: Synchronize task state & waiter->task of readers

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Date:   Tue, 10 Apr 2018 13:22:18 -0400
From:   Waiman Long <longman@...hat.com>
To:     Peter Zijlstra <peterz@...radead.org>,
        Ingo Molnar <mingo@...hat.com>
Cc:     linux-kernel@...r.kernel.org, Dave Chinner <david@...morbit.com>,
        Eric Sandeen <sandeen@...hat.com>,
        "Paul E. McKenney" <paulmck@...ux.vnet.ibm.com>,
        Waiman Long <longman@...hat.com>
Subject: [PATCH] locking/rwsem: Synchronize task state & waiter->task of readers

It was observed occasionally in PowerPC systems that there was reader
who had not been woken up but that its waiter->task had been cleared.

One probable cause of this missed wakeup may be the fact that the
waiter->task and the task state have not been properly synchronized as
the lock release-acquire pair of different locks in the wakeup code path
does not provide a full memory barrier guarantee. So smp_store_mb()
is now used to set waiter->task to NULL to provide a proper memory
barrier for synchronization.

Signed-off-by: Waiman Long <longman@...hat.com>
---
 kernel/locking/rwsem-xadd.c | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/kernel/locking/rwsem-xadd.c b/kernel/locking/rwsem-xadd.c
index e795908..b3c588c 100644
--- a/kernel/locking/rwsem-xadd.c
+++ b/kernel/locking/rwsem-xadd.c
@@ -209,6 +209,23 @@ static void __rwsem_mark_wake(struct rw_semaphore *sem,
 		smp_store_release(&waiter->task, NULL);
 	}
 
+	/*
+	 * To avoid missed wakeup of reader, we need to make sure
+	 * that task state and waiter->task are properly synchronized.
+	 *
+	 *     wakeup		      sleep
+	 *     ------		      -----
+	 * __rwsem_mark_wake:	rwsem_down_read_failed*:
+	 *   [S] waiter->task	  [S] set_current_state(state)
+	 *	 MB		      MB
+	 * try_to_wake_up:
+	 *   [L] state		  [L] waiter->task
+	 *
+	 * For the wakeup path, the original lock release-acquire pair
+	 * does not provide enough guarantee of proper synchronization.
+	 */
+	smp_mb();
+
 	adjustment = woken * RWSEM_ACTIVE_READ_BIAS - adjustment;
 	if (list_empty(&sem->wait_list)) {
 		/* hit end of list above */
-- 
1.8.3.1