linux-kernel - [PATCH] x86: Optimize variable_test

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <20150501151630.GH5029@twins.programming.kicks-ass.net>
Date:	Fri, 1 May 2015 17:16:30 +0200
From:	Peter Zijlstra <peterz@...radead.org>
To:	Ingo Molnar <mingo@...nel.org>, "H. Peter Anvin" <hpa@...or.com>,
	Thomas Gleixner <tglx@...utronix.de>,
	Linus Torvalds <torvalds@...ux-foundation.org>
Cc:	linux-kernel@...r.kernel.org, Borislav Petkov <bp@...en8.de>,
	Jakub Jelinek <jakub@...hat.com>
Subject: [PATCH] x86: Optimize variable_test_bit()

While looking at some asm I noticed we produce the most horrid code for
test_bit():

    1a5e:       49 0f a3 30             bt     %rsi,(%r8)
    1a62:       45 19 c0                sbb    %r8d,%r8d
    1a65:       45 85 c0                test   %r8d,%r8d
    1a68:       75 a6                   jne    1a10 <x86_schedule_events+0xc0>

Since test_bit() doesn't actually have any output variables, we can use
asm goto without having to add a memory clobber. This reduces the code
to something sensible:

    1a12:       49 0f a3 30             bt     %rsi,(%r8)
    1a16:       72 68                   jb     1a80 <x86_schedule_events+0x130>

Signed-off-by: Peter Zijlstra (Intel) <peterz@...radead.org>
---
PS. should we kill the memory clobber for __test_and_change_bit()? It
    seems inconsistent and out of place.

PPS. Jakub, I see gcc5.1 still hasn't got output operands for asm goto;
     is this something we can get 'fixed' ?

 arch/x86/include/asm/bitops.h | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/arch/x86/include/asm/bitops.h b/arch/x86/include/asm/bitops.h
index cfe3b954d5e4..bcf4fa77c04f 100644
--- a/arch/x86/include/asm/bitops.h
+++ b/arch/x86/include/asm/bitops.h
@@ -313,6 +313,15 @@ static __always_inline int constant_test_bit(long nr, const volatile unsigned lo
 
 static inline int variable_test_bit(long nr, volatile const unsigned long *addr)
 {
+#ifdef CC_HAVE_ASM_GOTO
+	asm_volatile_goto ("bt %1, %0\n\t"
+			   "jc %l[cc_label]"
+			   : : "m" (*(unsigned long *)addr), "Ir" (nr)
+			   : : cc_label);
+	return 0;
+cc_label:
+	return 1;
+#else
 	int oldbit;
 
 	asm volatile("bt %2,%1\n\t"
@@ -321,6 +330,7 @@ static inline int variable_test_bit(long nr, volatile const unsigned long *addr)
 		     : "m" (*(unsigned long *)addr), "Ir" (nr));
 
 	return oldbit;
+#endif
 }
 
 #if 0 /* Fool kernel-doc since it doesn't do macros yet */
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/