[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <20241014213342.1480681-6-xur@google.com>
Date: Mon, 14 Oct 2024 14:33:39 -0700
From: Rong Xu <xur@...gle.com>
To: Alice Ryhl <aliceryhl@...gle.com>, Andrew Morton <akpm@...ux-foundation.org>,
Arnd Bergmann <arnd@...db.de>, Bill Wendling <morbo@...gle.com>, Borislav Petkov <bp@...en8.de>,
Breno Leitao <leitao@...ian.org>, Brian Gerst <brgerst@...il.com>,
Dave Hansen <dave.hansen@...ux.intel.com>, David Li <davidxl@...gle.com>,
Han Shen <shenhan@...gle.com>, Heiko Carstens <hca@...ux.ibm.com>, "H. Peter Anvin" <hpa@...or.com>,
Ingo Molnar <mingo@...hat.com>, Jann Horn <jannh@...gle.com>, Jonathan Corbet <corbet@....net>,
Josh Poimboeuf <jpoimboe@...nel.org>, Juergen Gross <jgross@...e.com>,
Justin Stitt <justinstitt@...gle.com>, Kees Cook <kees@...nel.org>,
Masahiro Yamada <masahiroy@...nel.org>, "Mike Rapoport (IBM)" <rppt@...nel.org>,
Nathan Chancellor <nathan@...nel.org>, Nick Desaulniers <ndesaulniers@...gle.com>,
Nicolas Schier <nicolas@...sle.eu>, "Paul E. McKenney" <paulmck@...nel.org>,
Peter Zijlstra <peterz@...radead.org>, Rong Xu <xur@...gle.com>,
Sami Tolvanen <samitolvanen@...gle.com>, Thomas Gleixner <tglx@...utronix.de>,
Wei Yang <richard.weiyang@...il.com>, workflows@...r.kernel.org,
Miguel Ojeda <miguel.ojeda.sandonis@...il.com>, Maksim Panchenko <max4bolt@...il.com>
Cc: x86@...nel.org, linux-arch@...r.kernel.org, linux-doc@...r.kernel.org,
linux-kbuild@...r.kernel.org, linux-kernel@...r.kernel.org,
llvm@...ts.linux.dev, Sriraman Tallam <tmsriram@...gle.com>,
Krzysztof Pszeniczny <kpszeniczny@...gle.com>
Subject: [PATCH v4 5/6] AutoFDO: Enable machine function split optimization
for AutoFDO
Enable the machine function split optimization for AutoFDO in Clang.
Machine function split (MFS) is a pass in the Clang compiler that
splits a function into hot and cold parts. The linker groups all
cold blocks across functions together. This decreases hot code
fragmentation and improves iCache and iTLB utilization.
MFS requires a profile so this is enabled only for the AutoFDO builds.
Co-developed-by: Han Shen <shenhan@...gle.com>
Signed-off-by: Han Shen <shenhan@...gle.com>
Signed-off-by: Rong Xu <xur@...gle.com>
Suggested-by: Sriraman Tallam <tmsriram@...gle.com>
Suggested-by: Krzysztof Pszeniczny <kpszeniczny@...gle.com>
---
include/asm-generic/vmlinux.lds.h | 6 ++++++
scripts/Makefile.autofdo | 2 ++
2 files changed, 8 insertions(+)
diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
index ace617d1af9b..20e46c0917db 100644
--- a/include/asm-generic/vmlinux.lds.h
+++ b/include/asm-generic/vmlinux.lds.h
@@ -565,9 +565,14 @@ defined(CONFIG_AUTOFDO_CLANG)
__unlikely_text_start = .; \
*(.text.unlikely .text.unlikely.*) \
__unlikely_text_end = .;
+#define TEXT_SPLIT \
+ __split_text_start = .; \
+ *(.text.split .text.split.[0-9a-zA-Z_]*) \
+ __split_text_end = .;
#else
#define TEXT_HOT *(.text.hot .text.hot.*)
#define TEXT_UNLIKELY *(.text.unlikely .text.unlikely.*)
+#define TEXT_SPLIT
#endif
/*
@@ -584,6 +589,7 @@ defined(CONFIG_AUTOFDO_CLANG)
ALIGN_FUNCTION(); \
*(.text.asan.* .text.tsan.*) \
*(.text.unknown .text.unknown.*) \
+ TEXT_SPLIT \
TEXT_UNLIKELY \
. = ALIGN(PAGE_SIZE); \
TEXT_HOT \
diff --git a/scripts/Makefile.autofdo b/scripts/Makefile.autofdo
index 9c9a530ef090..380042a301cc 100644
--- a/scripts/Makefile.autofdo
+++ b/scripts/Makefile.autofdo
@@ -11,6 +11,7 @@ endif
ifdef CLANG_AUTOFDO_PROFILE
CFLAGS_AUTOFDO_CLANG += -fprofile-sample-use=$(CLANG_AUTOFDO_PROFILE) -ffunction-sections
+ CFLAGS_AUTOFDO_CLANG += -fsplit-machine-functions
endif
ifdef CONFIG_LTO_CLANG_THIN
@@ -18,6 +19,7 @@ ifdef CONFIG_LTO_CLANG_THIN
KBUILD_LDFLAGS += --lto-sample-profile=$(CLANG_AUTOFDO_PROFILE)
endif
KBUILD_LDFLAGS += --mllvm=-enable-fs-discriminator=true --mllvm=-improved-fs-discriminator=true -plugin-opt=thinlto
+ KBUILD_LDFLAGS += -plugin-opt=-split-machine-functions
endif
export CFLAGS_AUTOFDO_CLANG
--
2.47.0.rc1.288.g06298d1525-goog
Powered by blists - more mailing lists