linux-kernel - [tip:locking/core] locking/pvqspinlock: Order pv_unhash() after cmpxchg() on unlock slowpath

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [day] [month] [year] [list]

Message-ID: <tip-3b3fdf10a8add87ef0050138d51bfee9ab4983df@git.kernel.org>
Date:	Mon, 3 Aug 2015 09:59:46 -0700
From:	tip-bot for Will Deacon <tipbot@...or.com>
To:	linux-tip-commits@...r.kernel.org
Cc:	tglx@...utronix.de, paulmck@...ux.vnet.ibm.com,
	will.deacon@....com, akpm@...ux-foundation.org, hpa@...or.com,
	torvalds@...ux-foundation.org, Waiman.Long@...com,
	peterz@...radead.org, mingo@...nel.org, Steve.Capper@....com,
	linux-kernel@...r.kernel.org
Subject: [tip:locking/core] locking/pvqspinlock: Order pv_unhash()
  after cmpxchg() on unlock slowpath

Commit-ID:  3b3fdf10a8add87ef0050138d51bfee9ab4983df
Gitweb:     http://git.kernel.org/tip/3b3fdf10a8add87ef0050138d51bfee9ab4983df
Author:     Will Deacon <will.deacon@....com>
AuthorDate: Mon, 13 Jul 2015 16:58:30 +0100
Committer:  Ingo Molnar <mingo@...nel.org>
CommitDate: Mon, 3 Aug 2015 10:57:09 +0200

locking/pvqspinlock: Order pv_unhash() after cmpxchg() on unlock slowpath

When we unlock in __pv_queued_spin_unlock(), a failed cmpxchg() on the lock
value indicates that we need to take the slow-path and unhash the
corresponding node blocked on the lock.

Since a failed cmpxchg() does not provide any memory-ordering guarantees,
it is possible that the node data could be read before the cmpxchg() on
weakly-ordered architectures and therefore return a stale value, leading
to hash corruption and/or a BUG().

This patch adds an smb_rmb() following the failed cmpxchg operation, so
that the unhashing is ordered after the lock has been checked.

Reported-by: Peter Zijlstra <peterz@...radead.org>
Signed-off-by: Will Deacon <will.deacon@....com>
[ Added more comments]
Signed-off-by: Peter Zijlstra (Intel) <peterz@...radead.org>
Acked-by:  Waiman Long <Waiman.Long@...com>
Cc: Andrew Morton <akpm@...ux-foundation.org>
Cc: Linus Torvalds <torvalds@...ux-foundation.org>
Cc: Paul E. McKenney <paulmck@...ux.vnet.ibm.com>
Cc: Paul McKenney <paulmck@...ux.vnet.ibm.com>
Cc: Steve Capper <Steve.Capper@....com>
Cc: Thomas Gleixner <tglx@...utronix.de>
Link: http://lkml.kernel.org/r/20150713155830.GL2632@arm.com
Signed-off-by: Ingo Molnar <mingo@...nel.org>
---
 kernel/locking/qspinlock_paravirt.h | 23 ++++++++++++++++++-----
 1 file changed, 18 insertions(+), 5 deletions(-)

diff --git a/kernel/locking/qspinlock_paravirt.h b/kernel/locking/qspinlock_paravirt.h
index 489a878..ab8b1bb 100644
--- a/kernel/locking/qspinlock_paravirt.h
+++ b/kernel/locking/qspinlock_paravirt.h
@@ -244,13 +244,17 @@ static void pv_wait_head(struct qspinlock *lock, struct mcs_spinlock *node)
 		if (!lp) { /* ONCE */
 			lp = pv_hash(lock, pn);
 			/*
-			 * lp must be set before setting _Q_SLOW_VAL
+			 * We must hash before setting _Q_SLOW_VAL, such that
+			 * when we observe _Q_SLOW_VAL in __pv_queued_spin_unlock()
+			 * we'll be sure to be able to observe our hash entry.
 			 *
-			 * [S] lp = lock                [RmW] l = l->locked = 0
-			 *     MB                             MB
-			 * [S] l->locked = _Q_SLOW_VAL  [L]   lp
+			 *   [S] pn->state
+			 *   [S] <hash>                 [Rmw] l->locked == _Q_SLOW_VAL
+			 *       MB                           RMB
+			 * [RmW] l->locked = _Q_SLOW_VAL  [L] <unhash>
+			 *                                [L] pn->state
 			 *
-			 * Matches the cmpxchg() in __pv_queued_spin_unlock().
+			 * Matches the smp_rmb() in __pv_queued_spin_unlock().
 			 */
 			if (!cmpxchg(&l->locked, _Q_LOCKED_VAL, _Q_SLOW_VAL)) {
 				/*
@@ -306,6 +310,15 @@ __visible void __pv_queued_spin_unlock(struct qspinlock *lock)
 	}
 
 	/*
+	 * A failed cmpxchg doesn't provide any memory-ordering guarantees,
+	 * so we need a barrier to order the read of the node data in
+	 * pv_unhash *after* we've read the lock being _Q_SLOW_VAL.
+	 *
+	 * Matches the cmpxchg() in pv_wait_head() setting _Q_SLOW_VAL.
+	 */
+	smp_rmb();
+
+	/*
 	 * Since the above failed to release, this must be the SLOW path.
 	 * Therefore start by looking up the blocked node and unhashing it.
 	 */
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/