linux-kernel - [PATCH -tip 2/2] x86/hweight: Use POPCNT when available with X86_NATIVE

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20250325164854.199420-2-ubizjak@gmail.com>
Date: Tue, 25 Mar 2025 17:48:38 +0100
From: Uros Bizjak <ubizjak@...il.com>
To: x86@...nel.org,
	linux-kernel@...r.kernel.org
Cc: Uros Bizjak <ubizjak@...il.com>,
	Thomas Gleixner <tglx@...utronix.de>,
	Ingo Molnar <mingo@...nel.org>,
	Borislav Petkov <bp@...en8.de>,
	Dave Hansen <dave.hansen@...ux.intel.com>,
	"H. Peter Anvin" <hpa@...or.com>
Subject: [PATCH -tip 2/2] x86/hweight: Use POPCNT when available with X86_NATIVE_CPU option

Emit naked POPCNT instruction when available with X86_NATIVE_CPU
option. The compiler is not bound by ABI when emitting the instruction
without the fallback call to __sw_hweight{32,64}() library function
and has much more freedom to allocate input and output operands,
including memory input operand.

The code size of x86_64 defconfig (with X86_NATIVE_CPU option)
shrinks by 599 bytes:

  add/remove: 0/0 grow/shrink: 45/197 up/down: 843/-1442 (-599)
  Total: Before=22710531, After=22709932, chg -0.00%

The asm changes from e.g.:

	   3bf9c:	48 8b 3d 00 00 00 00 	mov    0x0(%rip),%rdi
	   3bfa3:	e8 00 00 00 00       	call   3bfa8 <...>
	   3bfa8:	90                   	nop
	   3bfa9:	90                   	nop

with:

	     34b:	31 c0                	xor    %eax,%eax
	     34d:	f3 48 0f b8 c7       	popcnt %rdi,%rax

in the .altinstr_replacement section

to:

	   3bfdc:	31 c0                	xor    %eax,%eax
	   3bfde:	f3 48 0f b8 05 00 00 	popcnt 0x0(%rip),%rax
	   3bfe5:	00 00

where there is no need for an entry in the .altinstr_replacement
section, shrinking all text sections by 9476 bytes:

	    text           data     bss      dec            hex filename
	27267068        4643047  814852 32724967        1f357e7 vmlinux-old.o
	27257592        4643047  814852 32715491        1f332e3 vmlinux-new.o

Signed-off-by: Uros Bizjak <ubizjak@...il.com>
Cc: Thomas Gleixner <tglx@...utronix.de>
Cc: Ingo Molnar <mingo@...nel.org>
Cc: Borislav Petkov <bp@...en8.de>
Cc: Dave Hansen <dave.hansen@...ux.intel.com>
Cc: "H. Peter Anvin" <hpa@...or.com>
---
 arch/x86/include/asm/arch_hweight.h | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)

diff --git a/arch/x86/include/asm/arch_hweight.h b/arch/x86/include/asm/arch_hweight.h
index aa0b3bd309fc..d39710e57531 100644
--- a/arch/x86/include/asm/arch_hweight.h
+++ b/arch/x86/include/asm/arch_hweight.h
@@ -25,13 +25,18 @@ static __always_inline unsigned int __arch_hweight32(unsigned int w)
 {
 	unsigned int res;
 
+#ifdef __POPCNT__
+	asm_inline (ASM_FORCE_CLR "popcntl %[val], %[cnt]"
+		    : [cnt] "=&r" (res)
+		    : [val] ASM_INPUT_RM (w));
+#else
 	asm_inline (ALTERNATIVE(ANNOTATE_IGNORE_ALTERNATIVE
 				"call __sw_hweight32",
 				ASM_CLR "popcntl %[val], %[cnt]",
 				X86_FEATURE_POPCNT)
 			 : [cnt] "=a" (res), ASM_CALL_CONSTRAINT
 			 : [val] REG_IN (w));
-
+#endif
 	return res;
 }
 
@@ -56,13 +61,18 @@ static __always_inline unsigned long __arch_hweight64(__u64 w)
 {
 	unsigned long res;
 
+#ifdef __POPCNT__
+	asm_inline (ASM_FORCE_CLR "popcntq %[val], %[cnt]"
+		    : [cnt] "=&r" (res)
+		    : [val] ASM_INPUT_RM (w));
+#else
 	asm_inline (ALTERNATIVE(ANNOTATE_IGNORE_ALTERNATIVE
 				"call __sw_hweight64",
 				ASM_CLR "popcntq %[val], %[cnt]",
 				X86_FEATURE_POPCNT)
 			 : [cnt] "=a" (res), ASM_CALL_CONSTRAINT
 			 : [val] REG_IN (w));
-
+#endif
 	return res;
 }
 #endif /* CONFIG_X86_32 */
-- 
2.42.0