linux-kernel - [tip:sched/core] sched/idle/x86: Optimize unnecessary mwait

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Mon, 16 Mar 2015 05:11:25 -0700
From:	tip-bot for Mike Galbraith <tipbot@...or.com>
To:	linux-tip-commits@...r.kernel.org
Cc:	linux-kernel@...r.kernel.org, efault@....de, mingo@...nel.org,
	lenb@...nel.org, tglx@...utronix.de, jwboyer@...hat.com,
	peterz@...radead.org, bitbucket@...ine.de,
	torvalds@...ux-foundation.org, hpa@...or.com, ibmalone@...il.com,
	bp@...en8.de, len.brown@...el.com
Subject: [tip:sched/core] sched/idle/x86: Optimize unnecessary mwait_idle(
 ) resched IPIs

Commit-ID:  f8e617f4582995f7c25ef25b4167213120ad122b
Gitweb:     http://git.kernel.org/tip/f8e617f4582995f7c25ef25b4167213120ad122b
Author:     Mike Galbraith <bitbucket@...ine.de>
AuthorDate: Sat, 18 Jan 2014 17:14:44 +0100
Committer:  Ingo Molnar <mingo@...nel.org>
CommitDate: Mon, 16 Mar 2015 11:14:22 +0100

sched/idle/x86: Optimize unnecessary mwait_idle() resched IPIs

To fully take advantage of MWAIT, apparently the CLFLUSH instruction needs
another quirk on certain CPUs: proper barriers around it on certain machines.

On a Q6600 SMP system, pipe-test scheduling performance, cross core,
improves significantly:

  3.8.13                   487.2 KHz    1.000
  3.13.0-master            415.5 KHz     .852
  3.13.0-master+           415.2 KHz     .852     + restore mwait_idle
  3.13.0-master++          488.5 KHz    1.002     + restore mwait_idle + IPI fix

Since X86_BUG_CLFLUSH_MONITOR is already a quirk, don't create a separate
quirk for the extra smp_mb()s.

Signed-off-by: Mike Galbraith <bitbucket@...ine.de>
Cc: <stable@...r.kernel.org> # 3.10+
Cc: Borislav Petkov <bp@...en8.de>
Cc: H. Peter Anvin <hpa@...or.com>
Cc: Ian Malone <ibmalone@...il.com>
Cc: Josh Boyer <jwboyer@...hat.com>
Cc: Len Brown <len.brown@...el.com>
Cc: Len Brown <lenb@...nel.org>
Cc: Linus Torvalds <torvalds@...ux-foundation.org>
Cc: Mike Galbraith <efault@....de>
Cc: Peter Zijlstra <peterz@...radead.org>
Cc: Thomas Gleixner <tglx@...utronix.de>
Link: http://lkml.kernel.org/r/1390061684.5566.4.camel@marge.simpson.net
[ Ported to recent kernel, added comments about the quirk. ]
Signed-off-by: Ingo Molnar <mingo@...nel.org>
---
 arch/x86/kernel/process.c | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c
index da06f74..6ad8a63 100644
--- a/arch/x86/kernel/process.c
+++ b/arch/x86/kernel/process.c
@@ -428,18 +428,22 @@ static int prefer_mwait_c1_over_halt(const struct cpuinfo_x86 *c)
 
 static void mwait_idle(void)
 {
-	if (!need_resched()) {
-		if (this_cpu_has(X86_BUG_CLFLUSH_MONITOR))
+	if (!current_set_polling_and_test()) {
+		if (this_cpu_has(X86_BUG_CLFLUSH_MONITOR)) {
+			smp_mb(); /* quirk */
 			clflush((void *)&current_thread_info()->flags);
+			smp_mb(); /* quirk */
+		}
 
 		__monitor((void *)&current_thread_info()->flags, 0, 0);
-		smp_mb();
 		if (!need_resched())
 			__sti_mwait(0, 0);
 		else
 			local_irq_enable();
-	} else
+	} else {
 		local_irq_enable();
+	}
+	__current_clr_polling();
 }
 
 void select_idle_routine(const struct cpuinfo_x86 *c)
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/