Message-ID: <67c07d2f-fb1f-4b7d-96e2-fb5ceb8fc692@cachyos.org>
Date: Mon, 4 Nov 2024 18:30:57 +0100
From: Peter Jung <ptr1337@...hyos.org>
To: Han Shen <shenhan@...gle.com>
Cc: Rong Xu <xur@...gle.com>, Alice Ryhl <aliceryhl@...gle.com>,
 Andrew Morton <akpm@...ux-foundation.org>, Arnd Bergmann <arnd@...db.de>,
 Bill Wendling <morbo@...gle.com>, Borislav Petkov <bp@...en8.de>,
 Breno Leitao <leitao@...ian.org>, Brian Gerst <brgerst@...il.com>,
 Dave Hansen <dave.hansen@...ux.intel.com>, David Li <davidxl@...gle.com>,
 Heiko Carstens <hca@...ux.ibm.com>, "H. Peter Anvin" <hpa@...or.com>,
 Ingo Molnar <mingo@...hat.com>, Jann Horn <jannh@...gle.com>,
 Jonathan Corbet <corbet@....net>, Josh Poimboeuf <jpoimboe@...nel.org>,
 Juergen Gross <jgross@...e.com>, Justin Stitt <justinstitt@...gle.com>,
 Kees Cook <kees@...nel.org>, Masahiro Yamada <masahiroy@...nel.org>,
 "Mike Rapoport (IBM)" <rppt@...nel.org>,
 Nathan Chancellor <nathan@...nel.org>,
 Nick Desaulniers <ndesaulniers@...gle.com>,
 Nicolas Schier <nicolas@...sle.eu>, "Paul E. McKenney" <paulmck@...nel.org>,
 Peter Zijlstra <peterz@...radead.org>,
 Sami Tolvanen <samitolvanen@...gle.com>, Thomas Gleixner
 <tglx@...utronix.de>, Wei Yang <richard.weiyang@...il.com>,
 workflows@...r.kernel.org, Miguel Ojeda <miguel.ojeda.sandonis@...il.com>,
 Maksim Panchenko <max4bolt@...il.com>, "David S. Miller"
 <davem@...emloft.net>, Andreas Larsson <andreas@...sler.com>,
 Yonghong Song <yonghong.song@...ux.dev>, Yabin Cui <yabinc@...gle.com>,
 Krzysztof Pszeniczny <kpszeniczny@...gle.com>,
 Sriraman Tallam <tmsriram@...gle.com>, Stephane Eranian
 <eranian@...gle.com>, x86@...nel.org, linux-arch@...r.kernel.org,
 sparclinux@...r.kernel.org, linux-doc@...r.kernel.org,
 linux-kbuild@...r.kernel.org, linux-kernel@...r.kernel.org,
 llvm@...ts.linux.dev
Subject: Re: [PATCH v7 1/7] Add AutoFDO support for Clang build

Hi Han,

I have tested your provided method, but the AutoFDO profile (lld does 
not get lto-sample-profile=$pathtoprofile passed)  nor Clang as compiler 
gets used.
Please replace following PKGBUILD and config from linux-mainline with 
the provided one in the gist. The patch is also included there.

https://gist.github.com/ptr1337/c92728bb273f7dbc2817db75eedec9ed

The main change I am doing here, is passing following to the build array 
and replacing "make all":

make LLVM=1 LLVM_IAS=1 CLANG_AUTOFDO_PROFILE=${srcdir}/perf.afdo all

When compiling the kernel with makepkg, this results at the packaging to 
following issue and can be reliable reproduced.

Regards,

Peter


On 04.11.24 05:50, Han Shen wrote:
> Hi Peter, thanks for reporting the issue. I am trying to reproduce it
> in the up-to-date archlinux environment. Below is what I have:
>    0. pacman -Syu
>    1. cloned archlinux build files from
> https://aur.archlinux.org/linux-mainline.git the newest mainline
> version is 6.12rc5-1.
>    2. changed the PKGBUILD file to include the patches series
>    3. changed the "config" to turn on clang autofdo
>    4. collected afdo profiles
>    5. MAKEFLAGS="-j48 V=1 LLVM=1 CLANG_AUTOFDO_PROFILE=$(pwd)/perf.afdo" \
>          makepkg -s --skipinteg --skippgp
>    6. install and reboot
> The above steps succeeded.
> You mentioned the error happens at "module_install", can you instruct
> me how to execute the "module_install" step?
> 
> Thanks,
> Han
> 
> On Sat, Nov 2, 2024 at 12:53 PM Peter Jung<ptr1337@...hyos.org> wrote:
>>
>>
>> On 02.11.24 20:46, Peter Jung wrote:
>>>
>>> On 02.11.24 18:51, Rong Xu wrote:
>>>> Add the build support for using Clang's AutoFDO. Building the kernel
>>>> with AutoFDO does not reduce the optimization level from the
>>>> compiler. AutoFDO uses hardware sampling to gather information about
>>>> the frequency of execution of different code paths within a binary.
>>>> This information is then used to guide the compiler's optimization
>>>> decisions, resulting in a more efficient binary. Experiments
>>>> showed that the kernel can improve up to 10% in latency.
>>>>
>>>> The support requires a Clang compiler after LLVM 17. This submission
>>>> is limited to x86 platforms that support PMU features like LBR on
>>>> Intel machines and AMD Zen3 BRS. Support for SPE on ARM 1,
>>>>    and BRBE on ARM 1 is part of planned future work.
>>>>
>>>> Here is an example workflow for AutoFDO kernel:
>>>>
>>>> 1) Build the kernel on the host machine with LLVM enabled, for example,
>>>>          $ make menuconfig LLVM=1
>>>>       Turn on AutoFDO build config:
>>>>         CONFIG_AUTOFDO_CLANG=y
>>>>       With a configuration that has LLVM enabled, use the following
>>>>       command:
>>>>          scripts/config -e AUTOFDO_CLANG
>>>>       After getting the config, build with
>>>>         $ make LLVM=1
>>>>
>>>> 2) Install the kernel on the test machine.
>>>>
>>>> 3) Run the load tests. The '-c' option in perf specifies the sample
>>>>      event period. We suggest     using a suitable prime number,
>>>>      like 500009, for this purpose.
>>>>      For Intel platforms:
>>>>         $ perf record -e BR_INST_RETIRED.NEAR_TAKEN:k -a -N -b -c
>>>> <count> \
>>>>           -o <perf_file> -- <loadtest>
>>>>      For AMD platforms:
>>>>         The supported system are: Zen3 with BRS, or Zen4 with amd_lbr_v2
>>>>        For Zen3:
>>>>         $ cat proc/cpuinfo | grep " brs"
>>>>         For Zen4:
>>>>         $ cat proc/cpuinfo | grep amd_lbr_v2
>>>>         $ perf record --pfm-events RETIRED_TAKEN_BRANCH_INSTRUCTIONS:k
>>>> -a \
>>>>           -N -b -c <count> -o <perf_file> -- <loadtest>
>>>>
>>>> 4) (Optional) Download the raw perf file to the host machine.
>>>>
>>>> 5) To generate an AutoFDO profile, two offline tools are available:
>>>>      create_llvm_prof and llvm_profgen. The create_llvm_prof tool is part
>>>>      of the AutoFDO project and can be found on GitHub
>>>>      (https://github.com/google/autofdo), version v0.30.1 or later. The
>>>>      llvm_profgen tool is included in the LLVM compiler itself. It's
>>>>      important to note that the version of llvm_profgen doesn't need to
>>>>      match the version of Clang. It needs to be the LLVM 19 release or
>>>>      later, or from the LLVM trunk.
>>>>         $ llvm-profgen --kernel --binary=<vmlinux> --
>>>> perfdata=<perf_file> \
>>>>           -o <profile_file>
>>>>      or
>>>>         $ create_llvm_prof --binary=<vmlinux> --profile=<perf_file> \
>>>>           --format=extbinary --out=<profile_file>
>>>>
>>>>      Note that multiple AutoFDO profile files can be merged into one via:
>>>>         $ llvm-profdata merge -o <profile_file>  <profile_1> ...
>>>> <profile_n>
>>>>
>>>> 6) Rebuild the kernel using the AutoFDO profile file with the same config
>>>>      as step 1, (Note CONFIG_AUTOFDO_CLANG needs to be enabled):
>>>>         $ make LLVM=1 CLANG_AUTOFDO_PROFILE=<profile_file>
>>>>
>>>> Co-developed-by: Han Shen<shenhan@...gle.com>
>>>> Signed-off-by: Han Shen<shenhan@...gle.com>
>>>> Signed-off-by: Rong Xu<xur@...gle.com>
>>>> Suggested-by: Sriraman Tallam<tmsriram@...gle.com>
>>>> Suggested-by: Krzysztof Pszeniczny<kpszeniczny@...gle.com>
>>>> Suggested-by: Nick Desaulniers<ndesaulniers@...gle.com>
>>>> Suggested-by: Stephane Eranian<eranian@...gle.com>
>>>> Tested-by: Yonghong Song<yonghong.song@...ux.dev>
>>>> Tested-by: Yabin Cui<yabinc@...gle.com>
>>>> Tested-by: Nathan Chancellor<nathan@...nel.org>
>>>> Reviewed-by: Kees Cook<kees@...nel.org>
>>> Tested-by: Peter Jung<ptr1337@...hyos.org>
>>>
>> The compilations and testing with the "make pacman-pkg" function from
>> the kernel worked fine.
>>
>> One problem I do face:
>> When I apply a AutoFDO profile together with the PKGBUILD [1] from
>> archlinux im running into issues at "module_install" at the packaging.
>>
>> See following log:
>> ```
>> make[2]: *** [scripts/Makefile.modinst:125:
>> /tmp/makepkg/linux-cachyos-rc-autofdo/pkg/linux-cachyos-rc-autofdo/usr/lib/modules/6.12.0-rc5-5-cachyos-rc-autofdo/kernel/arch/x86/kvm/kvm.ko]
>> Error 1
>> make[2]: *** Deleting file
>> '/tmp/makepkg/linux-cachyos-rc-autofdo/pkg/linux-cachyos-rc-autofdo/usr/lib/modules/6.12.0-rc5-5-cachyos-rc-autofdo/kernel/arch/x86/kvm/kvm.ko'
>>     INSTALL
>> /tmp/makepkg/linux-cachyos-rc-autofdo/pkg/linux-cachyos-rc-autofdo/usr/lib/modules/6.12.0-rc5-5-cachyos-rc-autofdo/kernel/crypto/cryptd.ko
>> make[2]: *** Waiting for unfinished jobs....
>> ```
>>
>>
>> This can be fixed with removed "INSTALL_MOD_STRIP=1" to the passed
>> parameters of module_install.
>>
>> This explicitly only happens, if a profile is passed - otherwise the
>> packaging works without problems.
>>
>> Regards,
>>
>> Peter Jung
>>

Powered by blists - more mailing lists