[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <e9889ff1-053a-4acf-bb45-ee31d255da2a@cachyos.org>
Date: Tue, 5 Nov 2024 18:19:40 +0100
From: Peter Jung <ptr1337@...hyos.org>
To: Rong Xu <xur@...gle.com>, Han Shen <shenhan@...gle.com>
Cc: Alice Ryhl <aliceryhl@...gle.com>,
Andrew Morton <akpm@...ux-foundation.org>, Arnd Bergmann <arnd@...db.de>,
Bill Wendling <morbo@...gle.com>, Borislav Petkov <bp@...en8.de>,
Breno Leitao <leitao@...ian.org>, Brian Gerst <brgerst@...il.com>,
Dave Hansen <dave.hansen@...ux.intel.com>, David Li <davidxl@...gle.com>,
Heiko Carstens <hca@...ux.ibm.com>, "H. Peter Anvin" <hpa@...or.com>,
Ingo Molnar <mingo@...hat.com>, Jann Horn <jannh@...gle.com>,
Jonathan Corbet <corbet@....net>, Josh Poimboeuf <jpoimboe@...nel.org>,
Juergen Gross <jgross@...e.com>, Justin Stitt <justinstitt@...gle.com>,
Kees Cook <kees@...nel.org>, Masahiro Yamada <masahiroy@...nel.org>,
"Mike Rapoport (IBM)" <rppt@...nel.org>,
Nathan Chancellor <nathan@...nel.org>,
Nick Desaulniers <ndesaulniers@...gle.com>,
Nicolas Schier <nicolas@...sle.eu>, "Paul E. McKenney" <paulmck@...nel.org>,
Peter Zijlstra <peterz@...radead.org>,
Sami Tolvanen <samitolvanen@...gle.com>, Thomas Gleixner
<tglx@...utronix.de>, Wei Yang <richard.weiyang@...il.com>,
workflows@...r.kernel.org, Miguel Ojeda <miguel.ojeda.sandonis@...il.com>,
Maksim Panchenko <max4bolt@...il.com>, "David S. Miller"
<davem@...emloft.net>, Andreas Larsson <andreas@...sler.com>,
Yonghong Song <yonghong.song@...ux.dev>, Yabin Cui <yabinc@...gle.com>,
Krzysztof Pszeniczny <kpszeniczny@...gle.com>,
Sriraman Tallam <tmsriram@...gle.com>, Stephane Eranian
<eranian@...gle.com>, x86@...nel.org, linux-arch@...r.kernel.org,
sparclinux@...r.kernel.org, linux-doc@...r.kernel.org,
linux-kbuild@...r.kernel.org, linux-kernel@...r.kernel.org,
llvm@...ts.linux.dev
Subject: Re: [PATCH v7 1/7] Add AutoFDO support for Clang build
Here the bugreport, in case someone wants to track it:
https://sourceware.org/bugzilla/show_bug.cgi?id=32340
On 05.11.24 15:56, Peter Jung wrote:
> You were right - reverting commit:
> https://github.com/bminor/binutils-gdb/commit/
> b20ab53f81db7eefa0db00d14f06c04527ac324c from the 2.43 branch does fix
> the packaging.
>
> I will forward this to an issue at their bugzilla.
>
> On 05.11.24 15:33, Peter Jung wrote:
>> Hi Rong,
>>
>> Glad that you were able to reproduce the issue!
>> Thanks for finding the root cause as well as the part of the code.
>> This really helps.
>>
>> I was able to do a successful packaging with binutils 2.42.
>> Lets forward this to the binutils tracker and hope this will be soon
>> solved. 🙂
>>
>> I have tested this also on the latest commit
>> (e1e4078ac59740a79cd709d61872abe15aba0087) and the issue is also
>> reproducible there.
>>
>> Thanks for your time! I dont see this as blocker. 🙂
>> It gets time to get this series merged :P
>>
>> Best regards,
>>
>> Peter
>>
>>
>>
>> On 05.11.24 08:25, Rong Xu wrote:
>>> We debugged this issue and we found the failure seems to only happen
>>> with strip (version 2.43) in binutil.
>>>
>>> For a profile-use compilation, either with -fprofile-use (PGO or
>>> iFDO), or -fprofile-sample-use (AutoFDO),
>>> an ELF section of .llvm.call-graph-profile is created for the object.
>>> For some reasons (like to save space?),
>>> the relocations in this section are of type "rel', rather the more
>>> common "rela" type.
>>>
>>> In this case,
>>> $ readelf -r kvm.ko |grep llvm.call-graph-profile
>>> Relocation section '.rel.llvm.call-graph-profile' at offset 0xf62a00
>>> contains 4 entries:
>>>
>>> strip (v2.43.0) has difficulty handling the relocations in
>>> .rel.llvm.call-graph-profile -- it silently failed with --strip-debug.
>>> But strip (v.2.42) has no issue with kvm.ko. The strip in llvm (i.e.
>>> llvm-strip) also passes with kvm.ko
>>>
>>> I compared binutil/strip source code for version v2.43.0 and v2.42.
>>> The different is around here:
>>> In v2.42 of bfd/elfcode.h
>>> Â Â Â 1618Â Â Â Â Â Â if ((entsize == sizeof (Elf_External_Rela)
>>> Â Â Â 1619Â Â Â Â Â Â Â Â Â Â Â && ebd->elf_info_to_howto != NULL)
>>> Â Â Â 1620Â Â Â Â Â Â Â Â Â Â || ebd->elf_info_to_howto_rel == NULL)
>>> Â Â Â 1621Â Â Â Â Â Â Â Â res = ebd->elf_info_to_howto (abfd, relent, &rela);
>>> Â Â Â 1622Â Â Â Â Â Â else
>>> Â Â Â 1623Â Â Â Â Â Â Â Â res = ebd->elf_info_to_howto_rel (abfd, relent, &rela);
>>>
>>> In v2.43.0 of bfd/elfcode.h
>>> Â Â Â 1618Â Â Â Â Â Â if (entsize == sizeof (Elf_External_Rela)
>>> Â Â Â 1619Â Â Â Â Â Â Â Â Â Â && ebd->elf_info_to_howto != NULL)
>>> Â Â Â 1620Â Â Â Â Â Â Â Â res = ebd->elf_info_to_howto (abfd, relent, &rela);
>>> Â Â Â 1621Â Â Â Â Â Â else if (ebd->elf_info_to_howto_rel != NULL)
>>> Â Â Â 1622Â Â Â Â Â Â Â Â res = ebd->elf_info_to_howto_rel (abfd, relent, &rela);
>>>
>>> In the 2.43 strip, line 1618 is false and line 1621 is also false.
>>> "res" is returned as false and the program exits with -1.
>>>
>>> While in 2.42, line 1620 is true and we get "res" from line 1621 and
>>> program functions correctly.
>>>
>>> I'm not familiar with binutil code base and don't know the reason for
>>> removing line 1620.
>>> I can file a bug for binutil for people to further investigate this.
>>>
>>> It seems to me that this issue should not be a blocker for our patch.
>>>
>>> Regards,
>>>
>>> -Rong
>>>
>>>
>>>
>>>
>>>
>>> On Mon, Nov 4, 2024 at 12:24 PM Han Shen<shenhan@...gle.com> wrote:
>>>> Hi Peter,
>>>> Thanks for providing the detailed reproduce.
>>>> Now I can see the error (after I synced to 6.12.0-rc6, I was using
>>>> rc5).
>>>> I'll look into that and report back.
>>>>
>>>>> I have tested your provided method, but the AutoFDO profile (lld does
>>>> not get lto-sample-profile=$pathtoprofile passed)
>>>>
>>>> I see. You also turned on ThinLTO, which I didn't, so the profile was
>>>> only used during compilation, not passed to lld.
>>>>
>>>> Thanks,
>>>> Han
>>>>
>>>> On Mon, Nov 4, 2024 at 9:31 AM Peter Jung<ptr1337@...hyos.org> wrote:
>>>>> Hi Han,
>>>>>
>>>>> I have tested your provided method, but the AutoFDO profile (lld does
>>>>> not get lto-sample-profile=$pathtoprofile passed)Â nor Clang as
>>>>> compiler
>>>>> gets used.
>>>>> Please replace following PKGBUILD and config from linux-mainline with
>>>>> the provided one in the gist. The patch is also included there.
>>>>>
>>>>> https://gist.github.com/ptr1337/c92728bb273f7dbc2817db75eedec9ed
>>>>>
>>>>> The main change I am doing here, is passing following to the build
>>>>> array
>>>>> and replacing "make all":
>>>>>
>>>>> make LLVM=1 LLVM_IAS=1 CLANG_AUTOFDO_PROFILE=${srcdir}/perf.afdo all
>>>>>
>>>>> When compiling the kernel with makepkg, this results at the
>>>>> packaging to
>>>>> following issue and can be reliable reproduced.
>>>>>
>>>>> Regards,
>>>>>
>>>>> Peter
>>>>>
>>>>>
>>>>> On 04.11.24 05:50, Han Shen wrote:
>>>>>> Hi Peter, thanks for reporting the issue. I am trying to reproduce it
>>>>>> in the up-to-date archlinux environment. Below is what I have:
>>>>>> Â Â Â 0. pacman -Syu
>>>>>> Â Â Â 1. cloned archlinux build files from
>>>>>> https://aur.archlinux.org/linux-mainline.git the newest mainline
>>>>>> version is 6.12rc5-1.
>>>>>> Â Â Â 2. changed the PKGBUILD file to include the patches series
>>>>>> Â Â Â 3. changed the "config" to turn on clang autofdo
>>>>>> Â Â Â 4. collected afdo profiles
>>>>>> Â Â Â 5. MAKEFLAGS="-j48 V=1 LLVM=1 CLANG_AUTOFDO_PROFILE=$(pwd)/
>>>>>> perf.afdo" \
>>>>>> Â Â Â Â Â Â Â Â Â makepkg -s --skipinteg --skippgp
>>>>>> Â Â Â 6. install and reboot
>>>>>> The above steps succeeded.
>>>>>> You mentioned the error happens at "module_install", can you instruct
>>>>>> me how to execute the "module_install" step?
>>>>>>
>>>>>> Thanks,
>>>>>> Han
>>>>>>
>>>>>> On Sat, Nov 2, 2024 at 12:53 PM Peter Jung<ptr1337@...hyos.org>
>>>>>> wrote:
>>>>>>>
>>>>>>> On 02.11.24 20:46, Peter Jung wrote:
>>>>>>>> On 02.11.24 18:51, Rong Xu wrote:
>>>>>>>>> Add the build support for using Clang's AutoFDO. Building the
>>>>>>>>> kernel
>>>>>>>>> with AutoFDO does not reduce the optimization level from the
>>>>>>>>> compiler. AutoFDO uses hardware sampling to gather information
>>>>>>>>> about
>>>>>>>>> the frequency of execution of different code paths within a
>>>>>>>>> binary.
>>>>>>>>> This information is then used to guide the compiler's optimization
>>>>>>>>> decisions, resulting in a more efficient binary. Experiments
>>>>>>>>> showed that the kernel can improve up to 10% in latency.
>>>>>>>>>
>>>>>>>>> The support requires a Clang compiler after LLVM 17. This
>>>>>>>>> submission
>>>>>>>>> is limited to x86 platforms that support PMU features like LBR on
>>>>>>>>> Intel machines and AMD Zen3 BRS. Support for SPE on ARM 1,
>>>>>>>>> Â Â Â and BRBE on ARM 1 is part of planned future work.
>>>>>>>>>
>>>>>>>>> Here is an example workflow for AutoFDO kernel:
>>>>>>>>>
>>>>>>>>> 1) Build the kernel on the host machine with LLVM enabled, for
>>>>>>>>> example,
>>>>>>>>> Â Â Â Â Â Â Â Â Â $ make menuconfig LLVM=1
>>>>>>>>> Â Â Â Â Â Â Turn on AutoFDO build config:
>>>>>>>>> Â Â Â Â Â Â Â Â CONFIG_AUTOFDO_CLANG=y
>>>>>>>>> Â Â Â Â Â Â With a configuration that has LLVM enabled, use the
>>>>>>>>> following
>>>>>>>>> Â Â Â Â Â Â command:
>>>>>>>>> Â Â Â Â Â Â Â Â Â scripts/config -e AUTOFDO_CLANG
>>>>>>>>> Â Â Â Â Â Â After getting the config, build with
>>>>>>>>> Â Â Â Â Â Â Â Â $ make LLVM=1
>>>>>>>>>
>>>>>>>>> 2) Install the kernel on the test machine.
>>>>>>>>>
>>>>>>>>> 3) Run the load tests. The '-c' option in perf specifies the
>>>>>>>>> sample
>>>>>>>>>      event period. We suggest    using a suitable prime number,
>>>>>>>>> Â Â Â Â Â like 500009, for this purpose.
>>>>>>>>> Â Â Â Â Â For Intel platforms:
>>>>>>>>> Â Â Â Â Â Â Â Â $ perf record -e BR_INST_RETIRED.NEAR_TAKEN:k -a -N -b -c
>>>>>>>>> <count> \
>>>>>>>>> Â Â Â Â Â Â Â Â Â Â -o <perf_file> -- <loadtest>
>>>>>>>>> Â Â Â Â Â For AMD platforms:
>>>>>>>>> Â Â Â Â Â Â Â Â The supported system are: Zen3 with BRS, or Zen4 with
>>>>>>>>> amd_lbr_v2
>>>>>>>>> Â Â Â Â Â Â Â For Zen3:
>>>>>>>>> Â Â Â Â Â Â Â Â $ cat proc/cpuinfo | grep " brs"
>>>>>>>>> Â Â Â Â Â Â Â Â For Zen4:
>>>>>>>>> Â Â Â Â Â Â Â Â $ cat proc/cpuinfo | grep amd_lbr_v2
>>>>>>>>> Â Â Â Â Â Â Â Â $ perf record --pfm-events
>>>>>>>>> RETIRED_TAKEN_BRANCH_INSTRUCTIONS:k
>>>>>>>>> -a \
>>>>>>>>> Â Â Â Â Â Â Â Â Â Â -N -b -c <count> -o <perf_file> -- <loadtest>
>>>>>>>>>
>>>>>>>>> 4) (Optional) Download the raw perf file to the host machine.
>>>>>>>>>
>>>>>>>>> 5) To generate an AutoFDO profile, two offline tools are
>>>>>>>>> available:
>>>>>>>>> Â Â Â Â Â create_llvm_prof and llvm_profgen. The create_llvm_prof
>>>>>>>>> tool is part
>>>>>>>>> Â Â Â Â Â of the AutoFDO project and can be found on GitHub
>>>>>>>>> Â Â Â Â Â (https://github.com/google/autofdo), version v0.30.1 or
>>>>>>>>> later. The
>>>>>>>>> Â Â Â Â Â llvm_profgen tool is included in the LLVM compiler
>>>>>>>>> itself. It's
>>>>>>>>> Â Â Â Â Â important to note that the version of llvm_profgen
>>>>>>>>> doesn't need to
>>>>>>>>> Â Â Â Â Â match the version of Clang. It needs to be the LLVM 19
>>>>>>>>> release or
>>>>>>>>> Â Â Â Â Â later, or from the LLVM trunk.
>>>>>>>>> Â Â Â Â Â Â Â Â $ llvm-profgen --kernel --binary=<vmlinux> --
>>>>>>>>> perfdata=<perf_file> \
>>>>>>>>> Â Â Â Â Â Â Â Â Â Â -o <profile_file>
>>>>>>>>> Â Â Â Â Â or
>>>>>>>>> Â Â Â Â Â Â Â Â $ create_llvm_prof --binary=<vmlinux> --
>>>>>>>>> profile=<perf_file> \
>>>>>>>>> Â Â Â Â Â Â Â Â Â Â --format=extbinary --out=<profile_file>
>>>>>>>>>
>>>>>>>>> Â Â Â Â Â Note that multiple AutoFDO profile files can be merged
>>>>>>>>> into one via:
>>>>>>>>> Â Â Â Â Â Â Â Â $ llvm-profdata merge -o <profile_file>Â <profile_1> ...
>>>>>>>>> <profile_n>
>>>>>>>>>
>>>>>>>>> 6) Rebuild the kernel using the AutoFDO profile file with the
>>>>>>>>> same config
>>>>>>>>> Â Â Â Â Â as step 1, (Note CONFIG_AUTOFDO_CLANG needs to be enabled):
>>>>>>>>> Â Â Â Â Â Â Â Â $ make LLVM=1 CLANG_AUTOFDO_PROFILE=<profile_file>
>>>>>>>>>
>>>>>>>>> Co-developed-by: Han Shen<shenhan@...gle.com>
>>>>>>>>> Signed-off-by: Han Shen<shenhan@...gle.com>
>>>>>>>>> Signed-off-by: Rong Xu<xur@...gle.com>
>>>>>>>>> Suggested-by: Sriraman Tallam<tmsriram@...gle.com>
>>>>>>>>> Suggested-by: Krzysztof Pszeniczny<kpszeniczny@...gle.com>
>>>>>>>>> Suggested-by: Nick Desaulniers<ndesaulniers@...gle.com>
>>>>>>>>> Suggested-by: Stephane Eranian<eranian@...gle.com>
>>>>>>>>> Tested-by: Yonghong Song<yonghong.song@...ux.dev>
>>>>>>>>> Tested-by: Yabin Cui<yabinc@...gle.com>
>>>>>>>>> Tested-by: Nathan Chancellor<nathan@...nel.org>
>>>>>>>>> Reviewed-by: Kees Cook<kees@...nel.org>
>>>>>>>> Tested-by: Peter Jung<ptr1337@...hyos.org>
>>>>>>>>
>>>>>>> The compilations and testing with the "make pacman-pkg" function
>>>>>>> from
>>>>>>> the kernel worked fine.
>>>>>>>
>>>>>>> One problem I do face:
>>>>>>> When I apply a AutoFDO profile together with the PKGBUILD [1] from
>>>>>>> archlinux im running into issues at "module_install" at the
>>>>>>> packaging.
>>>>>>>
>>>>>>> See following log:
>>>>>>> ```
>>>>>>> make[2]: *** [scripts/Makefile.modinst:125:
>>>>>>> /tmp/makepkg/linux-cachyos-rc-autofdo/pkg/linux-cachyos-rc-
>>>>>>> autofdo/usr/lib/modules/6.12.0-rc5-5-cachyos-rc-autofdo/kernel/
>>>>>>> arch/x86/kvm/kvm.ko]
>>>>>>> Error 1
>>>>>>> make[2]: *** Deleting file
>>>>>>> '/tmp/makepkg/linux-cachyos-rc-autofdo/pkg/linux-cachyos-rc-
>>>>>>> autofdo/usr/lib/modules/6.12.0-rc5-5-cachyos-rc-autofdo/kernel/
>>>>>>> arch/x86/kvm/kvm.ko'
>>>>>>> Â Â Â Â INSTALL
>>>>>>> /tmp/makepkg/linux-cachyos-rc-autofdo/pkg/linux-cachyos-rc-
>>>>>>> autofdo/usr/lib/modules/6.12.0-rc5-5-cachyos-rc-autofdo/kernel/
>>>>>>> crypto/cryptd.ko
>>>>>>> make[2]: *** Waiting for unfinished jobs....
>>>>>>> ```
>>>>>>>
>>>>>>>
>>>>>>> This can be fixed with removed "INSTALL_MOD_STRIP=1" to the passed
>>>>>>> parameters of module_install.
>>>>>>>
>>>>>>> This explicitly only happens, if a profile is passed - otherwise the
>>>>>>> packaging works without problems.
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Peter Jung
>>>>>>>
>>
>
Powered by blists - more mailing lists