linux-kernel - [tip:locking/core] locking/osq: Break out of spin-wait busy waiting loop for a preempted vCPU in osq

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <tip-5aff60a191e579ae00ae5ca6ce16c13b687bc8a3@git.kernel.org>
Date:   Tue, 22 Nov 2016 04:36:16 -0800
From:   tip-bot for Pan Xinhui <tipbot@...or.com>
To:     linux-tip-commits@...r.kernel.org
Cc:     xinhui.pan@...ux.vnet.ibm.com, pbonzini@...hat.com,
        borntraeger@...ibm.com, mingo@...nel.org,
        linux-kernel@...r.kernel.org, jgross@...e.com,
        boqun.feng@...il.com, peterz@...radead.org, hpa@...or.com,
        torvalds@...ux-foundation.org, tglx@...utronix.de
Subject: [tip:locking/core] locking/osq: Break out of spin-wait busy waiting
 loop for a preempted vCPU in osq_lock()

Commit-ID:  5aff60a191e579ae00ae5ca6ce16c13b687bc8a3
Gitweb:     http://git.kernel.org/tip/5aff60a191e579ae00ae5ca6ce16c13b687bc8a3
Author:     Pan Xinhui <xinhui.pan@...ux.vnet.ibm.com>
AuthorDate: Wed, 2 Nov 2016 05:08:29 -0400
Committer:  Ingo Molnar <mingo@...nel.org>
CommitDate: Tue, 22 Nov 2016 12:48:10 +0100

locking/osq: Break out of spin-wait busy waiting loop for a preempted vCPU in osq_lock()

An over-committed guest with more vCPUs than pCPUs has a heavy overload
in osq_lock().

This is because if vCPU-A holds the osq lock and yields out, vCPU-B ends
up waiting for per_cpu node->locked to be set. IOW, vCPU-B waits for
vCPU-A to run and unlock the osq lock.

Use the new vcpu_is_preempted(cpu) interface to detect if a vCPU is
currently running or not, and break out of the spin-loop if so.

test case:

 $ perf record -a perf bench sched messaging -g 400 -p && perf report

 before patch:
 18.09%  sched-messaging  [kernel.vmlinux]  [k] osq_lock
 12.28%  sched-messaging  [kernel.vmlinux]  [k] rwsem_spin_on_owner
  5.27%  sched-messaging  [kernel.vmlinux]  [k] mutex_unlock
  3.89%  sched-messaging  [kernel.vmlinux]  [k] wait_consider_task
  3.64%  sched-messaging  [kernel.vmlinux]  [k] _raw_write_lock_irq
  3.41%  sched-messaging  [kernel.vmlinux]  [k] mutex_spin_on_owner.is
  2.49%  sched-messaging  [kernel.vmlinux]  [k] system_call

 after patch:
 20.68%  sched-messaging  [kernel.vmlinux]  [k] mutex_spin_on_owner
  8.45%  sched-messaging  [kernel.vmlinux]  [k] mutex_unlock
  4.12%  sched-messaging  [kernel.vmlinux]  [k] system_call
  3.01%  sched-messaging  [kernel.vmlinux]  [k] system_call_common
  2.83%  sched-messaging  [kernel.vmlinux]  [k] copypage_power7
  2.64%  sched-messaging  [kernel.vmlinux]  [k] rwsem_spin_on_owner
  2.00%  sched-messaging  [kernel.vmlinux]  [k] osq_lock

Suggested-by: Boqun Feng <boqun.feng@...il.com>
Tested-by: Juergen Gross <jgross@...e.com>
Signed-off-by: Pan Xinhui <xinhui.pan@...ux.vnet.ibm.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@...radead.org>
Acked-by: Christian Borntraeger <borntraeger@...ibm.com>
Acked-by: Paolo Bonzini <pbonzini@...hat.com>
Cc: David.Laight@...LAB.COM
Cc: Linus Torvalds <torvalds@...ux-foundation.org>
Cc: Peter Zijlstra <peterz@...radead.org>
Cc: Thomas Gleixner <tglx@...utronix.de>
Cc: benh@...nel.crashing.org
Cc: bsingharora@...il.com
Cc: dave@...olabs.net
Cc: kernellwp@...il.com
Cc: konrad.wilk@...cle.com
Cc: linuxppc-dev@...ts.ozlabs.org
Cc: mpe@...erman.id.au
Cc: paulmck@...ux.vnet.ibm.com
Cc: paulus@...ba.org
Cc: rkrcmar@...hat.com
Cc: virtualization@...ts.linux-foundation.org
Cc: will.deacon@....com
Cc: xen-devel-request@...ts.xenproject.org
Cc: xen-devel@...ts.xenproject.org
Link: http://lkml.kernel.org/r/1478077718-37424-3-git-send-email-xinhui.pan@linux.vnet.ibm.com
[ Translated to English. ]
Signed-off-by: Ingo Molnar <mingo@...nel.org>
---
 kernel/locking/osq_lock.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/kernel/locking/osq_lock.c b/kernel/locking/osq_lock.c
index 4ea2710..a316794 100644
--- a/kernel/locking/osq_lock.c
+++ b/kernel/locking/osq_lock.c
@@ -21,6 +21,11 @@ static inline int encode_cpu(int cpu_nr)
 	return cpu_nr + 1;
 }
 
+static inline int node_cpu(struct optimistic_spin_node *node)
+{
+	return node->cpu - 1;
+}
+
 static inline struct optimistic_spin_node *decode_cpu(int encoded_cpu_val)
 {
 	int cpu_nr = encoded_cpu_val - 1;
@@ -118,8 +123,10 @@ bool osq_lock(struct optimistic_spin_queue *lock)
 	while (!READ_ONCE(node->locked)) {
 		/*
 		 * If we need to reschedule bail... so we can block.
+		 * Use vcpu_is_preempted() to avoid waiting for a preempted
+		 * lock holder:
 		 */
-		if (need_resched())
+		if (need_resched() || vcpu_is_preempted(node_cpu(node->prev)))
 			goto unqueue;
 
 		cpu_relax();