Date:   Mon, 18 Jan 2021 01:26:56 +0100
From:   Sedat Dilek <sedat.dilek@...il.com>
To:     Bill Wendling <morbo@...gle.com>
Cc:     Jonathan Corbet <corbet@....net>,
        Masahiro Yamada <masahiroy@...nel.org>,
        Linux Doc Mailing List <linux-doc@...r.kernel.org>,
        LKML <linux-kernel@...r.kernel.org>,
        Linux Kbuild mailing list <linux-kbuild@...r.kernel.org>,
        Clang-Built-Linux ML <clang-built-linux@...glegroups.com>,
        Andrew Morton <akpm@...ux-foundation.org>,
        Nathan Chancellor <natechancellor@...il.com>,
        Nick Desaulniers <ndesaulniers@...gle.com>,
        Sami Tolvanen <samitolvanen@...gle.com>,
        Fangrui Song <maskray@...gle.com>
Subject: Re: [PATCH v5] pgo: add clang's Profile Guided Optimization infrastructure

On Mon, Jan 18, 2021 at 12:33 AM Sedat Dilek <sedat.dilek@...il.com> wrote:

[ big snip ]

> > > > > > > > > > > > +Workflow
> > > > > > > > > > > > +========
> > > > > > > > > > > > +
> > > > > > > > > > > > +The PGO kernel can be run on the host or test machines. The data though should
> > > > > > > > > > > > +be analyzed with Clang's tools from the same Clang version as the kernel was
> > > > > > > > > > > > +compiled. Clang's tolerant of version skew, but it's easier to use the same
> > > > > > > > > > > > +Clang version.
> > > > > > > > > > > > +
> > > > > > > > > > > > +The profiling data is useful for optimizing the kernel, analyzing coverage,
> > > > > > > > > > > > +etc. Clang offers tools to perform these tasks.
> > > > > > > > > > > > +
> > > > > > > > > > > > +Here is an example workflow for profiling an instrumented kernel with PGO and
> > > > > > > > > > > > +using the result to optimize the kernel:
> > > > > > > > > > > > +
> > > > > > > > > > > > +1) Install the kernel on the TEST machine.
> > > > > > > > > > > > +
> > > > > > > > > > > > +2) Reset the data counters right before running the load tests
> > > > > > > > > > > > +
> > > > > > > > > > > > +   .. code-block:: sh
> > > > > > > > > > > > +
> > > > > > > > > > > > +      $ echo 1 > /sys/kernel/debug/pgo/reset
> > > > > > > > > > > > +
> > > > > > > > > > >
> > > > > > > > > > > I do not get this...
> > > > > > > > > > >
> > > > > > > > > > > # mount | grep debugfs
> > > > > > > > > > > debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
> > > > > > > > > > >
> > > > > > > > > > > After the load-test...?
> > > > > > > > > > >
> > > > > > > > > > > echo 0 > /sys/kernel/debug/pgo/reset
> > > > > > > > > > >
> > > > > > > > > > Writing anything to /sys/kernel/debug/pgo/reset will cause it to reset
> > > > > > > > > > the profiling counters. I picked 1 (one) semi-randomly, but it could
> > > > > > > > > > be any number, letter, your favorite short story, etc. You don't want
> > > > > > > > > > to reset it before collecting the profiling data from your load tests
> > > > > > > > > > though.
> > > > > > > > > >
> > > > > > > > > > > > +3) Run the load tests.
> > > > > > > > > > > > +
> > > > > > > > > > > > +4) Collect the raw profile data
> > > > > > > > > > > > +
> > > > > > > > > > > > +   .. code-block:: sh
> > > > > > > > > > > > +
> > > > > > > > > > > > +      $ cp -a /sys/kernel/debug/pgo/profraw /tmp/vmlinux.profraw
> > > > > > > > > > > > +
> > > > > > > > > > >
> > > > > > > > > > > This is only 4,9M small and seen from the date 5mins before I run the
> > > > > > > > > > > echo-1 line.
> > > > > > > > > > >
> > > > > > > > > > > # ll /sys/kernel/debug/pgo
> > > > > > > > > > > insgesamt 0
> > > > > > > > > > > drwxr-xr-x  2 root root 0 16. Jan 17:29 .
> > > > > > > > > > > drwx------ 41 root root 0 16. Jan 17:29 ..
> > > > > > > > > > > -rw-------  1 root root 0 16. Jan 17:29 profraw
> > > > > > > > > > > --w-------  1 root root 0 16. Jan 18:19 reset
> > > > > > > > > > >
> > > > > > > > > > > # cp -a /sys/kernel/debug/pgo/profraw /tmp/vmlinux.profraw
> > > > > > > > > > >
> > > > > > > > > > > # ll /tmp/vmlinux.profraw
> > > > > > > > > > > -rw------- 1 root root 4,9M 16. Jan 17:29 /tmp/vmlinux.profraw
> > > > > > > > > > >
> > > > > > > > > > > For me there was no prof-data collected from my defconfig kernel-build.
> > > > > > > > > > >
> > > > > > > > > > The /sys/kernel/debug/pgo/profraw file is read-only. Nothing writes to
> > > > > > > > > > it, not even the kernel. All it does is serialize the profiling
> > > > > > > > > > counters from a memory location in the kernel into a format that
> > > > > > > > > > LLVM's tools can understand.
> > > > > > > > > >
> > > > > > > > > > > > +5) (Optional) Download the raw profile data to the HOST machine.
> > > > > > > > > > > > +
> > > > > > > > > > > > +6) Process the raw profile data
> > > > > > > > > > > > +
> > > > > > > > > > > > +   .. code-block:: sh
> > > > > > > > > > > > +
> > > > > > > > > > > > +      $ llvm-profdata merge --output=vmlinux.profdata vmlinux.profraw
> > > > > > > > > > > > +
> > > > > > > > > > >
> > > > > > > > > > > Is that executed in /path/to/linux/git?
> > > > > > > > > > >
> > > > > > > > > > The llvm-profdata tool is not in the linux source tree. You need to
> > > > > > > > > > grab it from a clang distribution (or built from clang's git repo).
> > > > > > > > > >
> > > > > > > > > > > > +   Note that multiple raw profile data files can be merged during this step.
> > > > > > > > > > > > +
> > > > > > > > > > > > +7) Rebuild the kernel using the profile data (PGO disabled)
> > > > > > > > > > > > +
> > > > > > > > > > > > +   .. code-block:: sh
> > > > > > > > > > > > +
> > > > > > > > > > > > +      $ make LLVM=1 KCFLAGS=-fprofile-use=vmlinux.profdata ...
> > > > > > > > > > >
> > > > > > > > > > > How big is vmlinux.profdata (make defconfig)?
> > > > > > > > > > >
> > > > > > > > > > I don't have numbers for this, but from what you listed here, it's ~5M
> > > > > > > > > > in size. The size is proportional to the number of counters
> > > > > > > > > > instrumented in the kernel.
> > > > > > > > > >
> > > > > > > > > > > Do I need to do a full defconfig build or can I stop the build after
> > > > > > > > > > > let me say 10mins?
> > > > > > > > > > >
> > > > > > > > > > You should do a full rebuild. Make sure that PGO is disabled during the rebuild.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Thanks Bill for all the information.
> > > > > > > > >
> > > > > > > > > And sorry if I am so pedantic.
> > > > > > > > >
> > > > > > > > > I have installed my Debian system with Legacy-BIOS enabled.
> > > > > > > > >
> > > > > > > > > When I rebuild with KCFLAGS=-fprofile-use=vmlinux.profdata (LLVM=1 I
> > > > > > > > > have as a default) my system hangs on reboot.
> > > > > > > > >
> > > > > > > > > [ diffconfig ]
> > > > > > > > > $ scripts/diffconfig /boot/config-5.11.0-rc3-8-amd64-clang12-pgo
> > > > > > > > > /boot/config-5.11.0-rc3-9-amd64-clang12-pgo
> > > > > > > > > BUILD_SALT "5.11.0-rc3-8-amd64-clang12-pgo" -> "5.11.0-rc3-9-amd64-clang12-pgo"
> > > > > > > > > PGO_CLANG y -> n
> > > > > > > > >
> > > > > > > > > [ my make line ]
> > > > > > > > > $ cat ../start-build_5.11.0-rc3-9-amd64-clang12-pgo.txt
> > > > > > > > > dileks     63120   63095  0 06:47 pts/2    00:00:00 /usr/bin/perf_5.10
> > > > > > > > > stat make V=1 -j4 HOSTCC=clang HOSTCXX=clang++ HOSTLD=ld.lld CC=clang
> > > > > > > > > LD=ld.lld LLVM=1 LLVM_IAS=1 PAHOLE=/opt/pahole/bin/pahole
> > > > > > > > > LOCALVERSION=-9-amd64-clang12-pgo KBUILD_VERBOSE=1
> > > > > > > > > KBUILD_BUILD_HOST=iniza KBUILD_BUILD_USER=sedat.dilek@...il.com
> > > > > > > > > KBUILD_BUILD_TIMESTAMP=2021-01-17 bindeb-pkg
> > > > > > > > > KDEB_PKGVERSION=5.11.0~rc3-9~bullseye+dileks1
> > > > > > > > > KCFLAGS=-fprofile-use=vmlinux.profdata
> > > > > > > > >
> > > > > > > > > ( Yes, 06:47 a.m. in the morning :-). )
> > > > > > > > >
> > > > > > > > > When I boot with the rebuild Linux-kernel I see:
> > > > > > > > >
> > > > > > > > > Wrong EFI loader signature
> > > > > > > > > ...
> > > > > > > > > Decompressing
> > > > > > > > > Parsing EFI
> > > > > > > > > Performing Relocations done.
> > > > > > > > > Booting the Kernel.
> > > > > > > > >
> > > > > > > > > *** SYSTEM HANGS ***
> > > > > > > > > ( I waited for approx 1 min )
> > > > > > > > >
> > > > > > > > > I tried to turn UEFI support ON and OFF.
> > > > > > > > > No success.
> > > > > > > > >
> > > > > > > > > Does Clang-PGO support Legacy-BIOS or is something different wrong?
> > > > > > > > >
> > > > > > > > > Thanks.
> > > > > > > > >
> > > > > > > >
> > > > > > > > My bootloader is GRUB.
> > > > > > > >
> > > > > > > > In UEFI-BIOS settings there is no secure-boot disable option.
> > > > > > > > Just simple "Use UEFI BIOS" enabled|disabled.
> > > > > > > >
> > > > > > > > Installed Debian packages:
> > > > > > > >
> > > > > > > > ii grub-common 2.04-12
> > > > > > > > ii grub-pc 2.04-12
> > > > > > > > ii grub-pc-bin 2.04-12
> > > > > > > > ii grub2-common 2.04-12
> > > > > > > >
> > > > > > > > I found in the below link to do in grub-shell:
> > > > > > > >
> > > > > > > > set check_signatures=no
> > > > > > > >
> > > > > > > > But this is when grub-efi is installed.
> > > > > > > >
> > > > > > > > - Sedat -
> > > > > > > >
> > > > > > > > Link: https://unix.stackexchange.com/questions/126286/grub-efi-disable-signature-check
> > > > > > >
> > > > > > > Forget about that "Wrong EFI bootloader" - I see this with all other
> > > > > > > kernels (all boot fine).
> > > > > > >
> > > > > > > I tried in QEMU with and without KASLR:
> > > > > > >
> > > > > > > [ run_qemu.sh ]
> > > > > > > KPATH=$(pwd)
> > > > > > >
> > > > > > > APPEND="root=/dev/ram0 console=ttyS0 hung_task_panic=1 earlyprintk=ttyS0,115200"
> > > > > > > APPEND="$APPEND nokaslr"
> > > > > > >
> > > > > > > qemu-system-x86_64 -enable-kvm -M pc -kernel $KPATH/bzImage -initrd
> > > > > > > $KPATH/initrd.img -m 512 -net none -serial stdio -append "${APPEND}"
> > > > > > > [ /run_qemu.sh ]
> > > > > > >
> > > > > > > $ ./run_qemu.sh
> > > > > > > Probing EDD (edd=off to disable)... ok
> > > > > > > Wrong EFI loader signature.
> > > > > > > early console in extract_kernel
> > > > > > > input_data: 0x000000000289940d
> > > > > > > input_len: 0x000000000069804a
> > > > > > > output: 0x0000000001000000
> > > > > > > output_len: 0x0000000001ef2010
> > > > > > > kernel_total_size: 0x0000000001c2c000
> > > > > > > needed_size: 0x0000000002000000
> > > > > > > trampoline_32bit: 0x000000000009d000
> > > > > > >
> > > > > > >
> > > > > > > KASLR disabled: 'nokaslr' on cmdline.
> > > > > > >
> > > > > > >
> > > > > > > Decompressing Linux... Parsing ELF... No relocation needed... done.
> > > > > > > Booting the kernel.
> > > > > > >
> > > > > > > QEMU run stops, too.
> > > > > > >
> > > > > >
> > > > > > I re-generated my initrd.img with GZIP as compressor (my default is ZSTD).
> > > > > >
> > > > > > --- /etc/initramfs-tools/initramfs.conf 2021-01-17 12:35:30.823818501 +0100
> > > > > > +++ /etc/initramfs-tools/initramfs.conf.zstd    2020-09-21
> > > > > > 23:55:43.121735427 +0200
> > > > > > @@ -41,7 +41,7 @@ KEYMAP=n
> > > > > > # COMPRESS: [ gzip | bzip2 | lz4 | lzma | lzop | xz | zstd ]
> > > > > > #
> > > > > >
> > > > > > -COMPRESS=gzip
> > > > > > +COMPRESS=zstd
> > > > > >
> > > > > > #
> > > > > > # DEVICE: ...
> > > > > >
> > > > > > root# KVER="5.11.0-rc3-9-amd64-clang12-pgo" ; update-initramfs -c -k $KVER
> > > > > >
> > > > > > QEMU boot stops at the same stage.
> > > > > >
> > > > > > Now, my head is empty...
> > > > > >
> > > > > > Any comments?
> > > > > >
> > > > >
> > > > > ( Just as a side note I have Nick's DWARF-v5 support enabled. )
> > > > >
> > > > > There is one EFI related warning in my build-log:
> > > > >
> > > > > $ grep warning: build-log_5.11.0-rc3-9-amd64-clang12-pgo.txt
> > > > > dpkg-architecture: warning: specified GNU system type x86_64-linux-gnu
> > > > > does not match CC system type x86_64-pc-linux-gnu, try setting a
> > > > > correct CC environment variable
> > > > > warning: arch/x86/platform/efi/quirks.c: Function control flow change
> > > > > detected (hash mismatch) efi_arch_mem_reserve Hash =
> > > > > 391331300655996873 [-Wbackend-plugin]
> > > > > warning: arch/x86/platform/efi/efi.c: Function control flow change
> > > > > detected (hash mismatch) efi_attr_is_visible Hash = 567185240781730690
> > > > > [-Wbackend-plugin]
> > > > > arch/x86/crypto/aegis128-aesni-glue.c:265:30: warning: unused variable
> > > > > 'simd_alg' [-Wunused-variable]
> > > > > warning: lib/crypto/sha256.c: Function control flow change detected
> > > > > (hash mismatch) sha256_update Hash = 744640996947387358
> > > > > [-Wbackend-plugin]
> > > > > warning: arch/x86/boot/compressed/string.c: Function control flow
> > > > > change detected (hash mismatch) memcmp Hash = 742261418966908927
> > > > > [-Wbackend-plugin]
> > > > > warning: arch/x86/boot/compressed/string.c: Function control flow
> > > > > change detected (hash mismatch) bcmp Hash = 742261418966908927
> > > > > [-Wbackend-plugin]
> > > > > warning: arch/x86/boot/compressed/string.c: Function control flow
> > > > > change detected (hash mismatch) strcmp Hash = 536873291001348520
> > > > > [-Wbackend-plugin]
> > > > > warning: arch/x86/boot/compressed/string.c: Function control flow
> > > > > change detected (hash mismatch) strnlen Hash = 146835646621254984
> > > > > [-Wbackend-plugin]
> > > > > warning: arch/x86/boot/compressed/string.c: Function control flow
> > > > > change detected (hash mismatch) simple_strtoull Hash =
> > > > > 252792765950587360 [-Wbackend-plugin]
> > > > > warning: arch/x86/boot/compressed/string.c: Function control flow
> > > > > change detected (hash mismatch) strstr Hash = 391331303349076211
> > > > > [-Wbackend-plugin]
> > > > > warning: arch/x86/boot/compressed/string.c: Function control flow
> > > > > change detected (hash mismatch) strchr Hash = 1063705159280644635
> > > > > [-Wbackend-plugin]
> > > > > warning: arch/x86/boot/compressed/string.c: Function control flow
> > > > > change detected (hash mismatch) kstrtoull Hash = 758414239132790022
> > > > > [-Wbackend-plugin]
> > > > > drivers/infiniband/hw/hfi1/platform.o: warning: objtool: tune_serdes()
> > > > > falls through to next function apply_tx_lanes()
> > > > >
> > > > > Cannot say if this information is helpful.
> > > > >
> > > >
> > > > My LLVM/Clang v12 is from <apt.llvm.org>:
> > > >
> > > > clang-12 version 1:12~++20210115111113+45ef053bd709-1~exp1~20210115101809.3724
> > > >
> > > > My kernel-config is attached.
> > > >
> > >
> > > I dropped "LLVM_IAS=1" from my make line and did for my next build:
> > >
> > > $ scripts/diffconfig /boot/config-5.11.0-rc3-8-amd64-clang12-pgo .config
> > > BUILD_SALT "5.11.0-rc3-8-amd64-clang12-pgo" -> "5.11.0-rc3-10-amd64-clang12-pgo"
> > > DEBUG_INFO_DWARF2 n -> y
> > > DEBUG_INFO_DWARF5 y -> n
> > > PGO_CLANG y -> n
> > >
> > > Means dropped DWARF5 support.
> > >
> > Hi Sedat,
> >
> > Using PGO just improves optimizations. So unless there's miscompile,
> > or some other nefarious thing, it shouldn't affect how the boot loader
> > runs.
> >
> > As a sanity check, does the same Linux source and compiler version
> > generate a bootable kernel when PGO isn't used?
> >
>
> Yes, I can boot with the same code base without PGO.
>
> With the attached kernel-config.
>
> I remember there is a fix in CBL issue tracker for...
>
> ( https://github.com/ClangBuiltLinux/linux/issues/1250 )
>
> Loading, please wait...
> Starting version 247.2-4
> [    2.157223] floppy: module verification failed: signature and/or
> required key missing - tainting kernel
> [    2.179326] i2c_piix4: Unknown symbol _GLOBAL_OFFSET_TABLE_ (err -2)
> [    2.183558] scsi_mod: Unknown symbol _GLOBAL_OFFSET_TABLE_ (err -2)
> [    2.187991] floppy: Unknown symbol _GLOBAL_OFFSET_TABLE_ (err -2)
> [    2.195047] psmouse: Unknown symbol _GLOBAL_OFFSET_TABLE_ (err -2)
> [    2.210404] scsi_mod: Unknown symbol _GLOBAL_OFFSET_TABLE_ (err -2)
> [    2.231055] scsi_mod: Unknown symbol _GLOBAL_OFFSET_TABLE_ (err -2)
>

[ CC Fangrui ]

With the attached...

   [PATCH v3] module: Ignore _GLOBAL_OFFSET_TABLE_ when warning for
undefined symbols

...I was finally able to boot into a rebuild PGO-optimized Linux-kernel.
For details see ClangBuiltLinux issue #1250 "Unknown symbol
_GLOBAL_OFFSET_TABLE_ loading kernel modules".

@ Bill Nick Sami Nathan

1, Can you say something of the impact passing "LLVM_IAS=1" to make?
2. Can you please try Nick's DWARF v5 support patchset v5 and
CONFIG_DEBUG_INFO_DWARF5=y (see attachments)?

I would like to know what the impact of the Clang's Integrated
Assembler and DWARF v5 are.

I dropped both means...

1. Do not pass "LLVM_IAS=1" to make.
2. Use default DWARF v2 (with Nick's patchset: CONFIG_DEBUG_INFO_DWARF2=y).

...for a successfull build and boot on bare metal.

Thanks.

- Sedat -

[1] https://github.com/ClangBuiltLinux/linux/issues/1250

View attachment "dmesg-T_5.11.0-rc3-10-amd64-clang12-pgo.txt" of type "text/plain" (74353 bytes)

Download attachment "v3_20210115_maskray_module_ignore__global_offset_table__when_warning_for_undefined_symbols.mbx" of type "application/octet-stream" (3875 bytes)

Download attachment "config-5.11.0-rc3-10-amd64-clang12-pgo" of type "application/octet-stream" (238239 bytes)

Download attachment "v5_20210115_ndesaulniers_kbuild_dwarf_v5_support.mbx" of type "application/octet-stream" (14129 bytes)

Powered by blists - more mailing lists