linux-kernel - Re: [RFC] locking/rwsem: Avoid issuing wakeup before setting the reader waiter to nil

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:   Thu, 29 Nov 2018 13:30:17 -0800
From:   Davidlohr Bueso <dave@...olabs.net>
To:     unlisted-recipients:; (no To-header on input)
Cc:     Peter Zijlstra <peterz@...radead.org>,
        Yongji Xie <elohimes@...il.com>, mingo@...hat.com,
        will.deacon@....com, linux-kernel@...r.kernel.org,
        xieyongji@...du.com, zhangyu31@...du.com, liuqi16@...du.com,
        yuanlinsi01@...du.com, nixun@...du.com, lilin24@...du.com
Subject: Re: [RFC] locking/rwsem: Avoid issuing wakeup before setting the
 reader waiter to nil

On Thu, 29 Nov 2018, Waiman Long wrote:

>That can be costly for x86 which will now have 2 locked instructions.

Yeah, and when used as an actual queue we should really start to notice.
Some users just have a single task in the wake_q because avoiding the cost
of wake_up_process() with locks held is significant.

How about instead of adding the barrier before the cmpxchg, we do it
in the failed branch, right before we return. This is the uncommon
path.

Thanks,
Davidlohr

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 091e089063be..0d844a18a9dc 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -408,8 +408,14 @@ void wake_q_add(struct wake_q_head *head, struct task_struct *task)
 	 * This cmpxchg() executes a full barrier, which pairs with the full
 	 * barrier executed by the wakeup in wake_up_q().
 	 */
-	if (cmpxchg(&node->next, NULL, WAKE_Q_TAIL))
+	if (cmpxchg(&node->next, NULL, WAKE_Q_TAIL)) {
+		/*
+		 * Ensure, that when the cmpxchg() fails, the corresponding
+		 * wake_up_q() will observe our prior state.
+		 */
+		smp_mb__after_atomic();
 		return;
+	}
 
 	get_task_struct(task);