Message-ID: <YbyIJYzqtHPKRMFt@zn.tnic>
Date:   Fri, 17 Dec 2021 13:52:53 +0100
From:   Borislav Petkov <bp@...e.de>
To:     Yin Fengwei <fengwei.yin@...el.com>
Cc:     Carel Si <beibei.si@...el.com>, Joerg Roedel <jroedel@...e.de>,
        LKML <linux-kernel@...r.kernel.org>, x86@...nel.org,
        lkp@...ts.01.org, lkp@...el.com, bfields@...ldses.org,
        llvm@...ts.linux.dev
Subject: Re: [LKP] Re: [x86/mm/64] f154f29085:
 BUG:kernel_reboot-without-warning_in_boot_stage - clang KCOV?

Add Bruce and clang folks to the party.

On Thu, Dec 16, 2021 at 08:21:15PM +0800, Yin Fengwei wrote:
> Hi Boris,
> 
> On 12/16/2021 7:58 PM, Carel Si wrote:
> > Hi Boris,
> > 
> > On Thu, Dec 16, 2021 at 11:06:59AM +0100, Borislav Petkov wrote:
> >> On Thu, Dec 16, 2021 at 03:04:16PM +0800, Yin Fengwei wrote:
> >>> The testing was with Qemu.
> >>
> >> This is hardly what I asked for.
> >>
> >>> And we found that the hang is related with clang-14.
> >>
> >> I saw that already.
> >>
> >>> The original report showed the kernel is built with clang-14:
> >>>         # build kernel
> >>> 	cd linux
> >>> 	cp config-5.16.0-rc3-00003-gf154f290855b .config
> >>> 	make HOSTCC=clang-14 CC=clang-14 ARCH=x86_64 olddefconfig prepare modules_prepare bzImage modules
> >>> 	make HOSTCC=clang-14 CC=clang-14 ARCH=x86_64 INSTALL_MOD_PATH=<mod-install-dir> modules_install
> >>
> >> I saw that too.
> >>
> >>> Looks like KASAN related stub generated by clang-14 (KASAN_SHADOW_OFFSET and asan_report).
> >>> This function is early function called before kasan_init.
> >>>
> >>> Looks like we need to disable KASAN_SANITIZE for arch/x86/kernel/cpu/common.c. So clang-14 will
> >>> be happy with this kind of early TLB flush? Thanks.
> >>
> >> Ok, I don't understand: I asked for how exactly to reproduce and whether
> >> you can send me your vmlinux you built with your clang-14. What I get is
> >> some possible explanation about what might be happening.
> >>
> >> So what do you expect me to do? Say, "oh, sure, you're right, send me a
> >> patch" without even being able to see for myself what the root cause is?
> >>
> >> What if it is not the kernel's fault but clang-14 is miscompiling crap
> >> as in so many other cases?
> >>
> >> I built clang-14 and built with your .config and it works here fine. So
> >> why does yours fail?
> >>
> >> Or what's the point of all this?
> 
> I had concern that our report is an invalid report because you can't reproduce
> it in your side. If that's the case, it could waste more your time. That's why
> I did check and shared what I got. I am very sorry if I did it wrong.

Sure, you can always add your analysis but I'd like to reproduce myself
too. So, in the future, please answer the questions and then feel free
to add your analysis - I'll gladly have a look.

Which wasn't that far from the truth, btw.

But it isn't KASAN but GCOV profiling. Or is it KCOV profiling which
clang does.

That thing adds some counting glue to native_write_cr4():

(my comments from the actual singlestepping in qemu start with '##' below)

	movq	$__llvm_gcov_ctr.48+8, %rbx				##  mov    $0xffffffff8837d3c0,%rbx
.LBB8_1:                                # %set_register
                                        # =>This Inner Loop Header: Depth=1

	jmp	.Ltmp42
	...

.Ltmp42:                                # Block address taken
.LBB8_7:                                # %if.end79
        movq    %rbx, %rax						## 0xffffffff8837d3c0
        shrq    $3, %rax						## 0x1ffffffff106fa78
        movabsq $-2305847407260205056, %rcx     # imm = 0xDFFFFC0000000000	## 0xdffffc0000000000
        cmpb    $0, (%rax,%rcx)
        je      .LBB8_9

so the memory address CMP accesses is something as nonsensical as

  0xfffffbfff106fa78

so I'm guessing we need to setup something for that __llvm_gcov_ctr to
deref properly but I haven't dug deeper.

The important thing is that this triggers with clang-13 and -14. gcc is
fine with the same config but that probably is because gcc does other
profiling - gcov - I guess. Looking at the resulting asm, it has a bunch
of those counter increments:

        incq    __gcov0.native_write_cr4+88(%rip)       # __gcov0.native_write_cr4[11]

but no weird memory references.

So, clang folks, what's up?

The fix is simple but I'd like to understand first why does this fail
only with clang, 13 and newer.

(I mean, melver pointed me to

  380d53c45ff2 ("compiler_attributes.h: define __no_profile, add to noinstr")

which explains why 13 and newer).

Btw, joro, that second hunk is I think needed too because a couple of
lines earlier we set up the cr4 shadow so I think you should use it
instead of touching the hw CR4.

---
diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c
index 0083464de5e3..79b3d67addcc 100644
--- a/arch/x86/kernel/cpu/common.c
+++ b/arch/x86/kernel/cpu/common.c
@@ -384,7 +384,7 @@ void native_write_cr0(unsigned long val)
 }
 EXPORT_SYMBOL(native_write_cr0);
 
-void native_write_cr4(unsigned long val)
+void __no_profile native_write_cr4(unsigned long val)
 {
 	unsigned long bits_changed = 0;
 
diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c
index 75acb6027a87..68d2b7f9a913 100644
--- a/arch/x86/kernel/head64.c
+++ b/arch/x86/kernel/head64.c
@@ -483,7 +483,7 @@ asmlinkage __visible void __init x86_64_start_kernel(char * real_mode_data)
 	/* Kill off the identity-map trampoline */
 	reset_early_page_tables();
 
-	__native_tlb_flush_global(native_read_cr4());
+	__native_tlb_flush_global(this_cpu_read(cpu_tlbstate.cr4));
 
 	clear_bss();
 


Leaving in the rest for the newly added folks.

> If you don't want to use lkp tool to reproduce the issue, following command
> could be used as well:
> 
> Use Qemu command so only kernel image need be downloaded:
> qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 16G -s -S -kernel vmlinuz-5.16.0-rc3-00003-gf154f290855b -nographic -append "console=ttyS0 earlyprintk=ttyS0,115200"
> to reproduce it.
> 
> 
> 
> Regards
> Yin, Fengwei
> 
> 
> 
> >>
> >> I mean, if you cannot send me what I ask for, you can say so. Then I can
> >> ignore this whole report altogether and waste my time somewhere else.
> > 
> > We have uploaded vmlinuz, modules.cgz, config as well as other related file to:
> > https://download.01.org/0day-ci/lkp-qemu/pkg/linux/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/
> > 
> > Machine types can refer to:
> > https://zerobin.net/?e107cf7b56495d80#MQLh14wUT9Osv1tWCwiQx/okkAN48Nq+drVPE0PiNPw=
> > 
> > If there's any other msg needed, pls feel free to propose, thanks.
> > 
> > Below are our full steps to reproduce the issue:
> > 
> > # download lkp-tests
> > $ git clone https://github.com/intel/lkp-tests.git
> > 
> > $ cd lkp-tests/
> > 
> > # download vmlinuz
> > $ wget https://download.01.org/0day-ci/lkp-qemu/pkg/linux/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/vmlinuz-5.16.0-rc3-00003-gf154f290855b
> > 
> > # dowmload modules.cgz
> > $ wget https://download.01.org/0day-ci/lkp-qemu/pkg/linux/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/modules.cgz
> > 
> > # download job-script which is attached
> > 
> > # run lkp qemu
> > lkp-tests$ sudo bin/lkp qemu -k vmlinuz-5.16.0-rc3-00003-gf154f290855b -m modules.cgz job-script
> > 
> > ~/lkp-tests/pkg/lkp-src ~/lkp-tests
> > x86_64
> > ==> Making package: lkp-src 0-1 (Thu 16 Dec 2021 07:26:22 PM CST)
> > ==> Checking runtime dependencies...
> > ==> Checking buildtime dependencies...
> > ==> WARNING: Using existing $srcdir/ tree
> > ==> Removing existing $pkgdir/ directory...
> > ==> Starting build()...
> > make: Entering directory '/home/carel/lkp-tests/bin/event'
> > klcc  -D_FORTIFY_SOURCE=2  -c -o wakeup.o wakeup.c
> > klcc  -Wl,-O1,--sort-common,--as-needed,-z,relro -static -o wakeup wakeup.o
> > rm -f wakeup.o
> > strip wakeup
> > make: Leaving directory '/home/carel/lkp-tests/bin/event'
> > ==> Entering fakeroot environment...
> > x86_64
> > ==> Starting package()...
> > ==> Creating package "lkp-src"...
> > 103987 blocks
> > renamed '/home/carel/.lkp/cache/lkp-x86_64.cgz.tmp' -> '/home/carel/.lkp/cache/lkp-x86_64.cgz'
> > ==> Leaving fakeroot environment.
> > ==> Finished making: lkp-src 0-1 (Thu 16 Dec 2021 07:26:24 PM CST)
> > ~/lkp-tests
> > 12 blocks
> > result_root: /home/carel/.lkp//result/boot/1/vm-snb/debian-10.4-x86_64-20200603.cgz/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/0
> > downloading initrds ...
> > use local modules: /home/carel/.lkp/cache/modules.cgz
> > /usr/bin/wget -q --timeout=1800 --tries=1 --local-encoding=UTF-8 http://0day.sh.intel.com:80/~lkp/osimage/debian/debian-10.4-x86_64-20200603.cgz -N -P /home/carel/.lkp/cache/osimage/debian
> > 440459 blocks
> > /usr/bin/wget -q --timeout=1800 --tries=1 --local-encoding=UTF-8 http://0day.sh.intel.com:80/~lkp/osimage/deps/debian-10.4-x86_64-20200603.cgz/run-ipconfig_20200608.cgz -N -P /home/carel/.lkp/cache/osimage/deps/debian-10.4-x86_64-20200603.cgz
> > 1773 blocks
> > /usr/bin/wget -q --timeout=1800 --tries=1 --local-encoding=UTF-8 http://0day.sh.intel.com:80/~lkp/osimage/deps/debian-10.4-x86_64-20200603.cgz/lkp_20210707.cgz -N -P /home/carel/.lkp/cache/osimage/deps/debian-10.4-x86_64-20200603.cgz
> > 2321 blocks
> > /usr/bin/wget -q --timeout=1800 --tries=1 --local-encoding=UTF-8 http://0day.sh.intel.com:80/~lkp/osimage/deps/debian-10.4-x86_64-20200603.cgz/rsync-rootfs_20200608.cgz -N -P /home/carel/.lkp/cache/osimage/deps/debian-10.4-x86_64-20200603.cgz
> > 6856 blocks
> > exec command: qemu-system-x86_64 -enable-kvm -fsdev local,id=test_dev,path=/home/carel/.lkp//result/boot/1/vm-snb/debian-10.4-x86_64-20200603.cgz/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/0,security_model=none -device virtio-9p-pci,fsdev=test_dev,mount_tag=9p/virtfs_mount -kernel vmlinuz-5.16.0-rc3-00003-gf154f290855b -append root=/dev/ram0 user=lkp job=/lkp/jobs/scheduled/vm-snb-192/boot-1-debian-10.4-x86_64-20200603.cgz-f154f290855b070cc94dd44ad253c0ef8a9337bb-20211208-23538-lnvkeg-5.yaml ARCH=x86_64 kconfig=x86_64-randconfig-a013-20211207 branch=tip/x86/mm commit=f154f290855b070cc94dd44ad253c0ef8a9337bb BOOT_IMAGE=/pkg/linux/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/vmlinuz-5.16.0-rc3-00003-gf154f290855b vmalloc=128M initramfs_async=0 page_owner=on max_uptime=600 RESULT_ROOT=/result/boot/1/vm-snb/debian-10.4-x86_64-20200603.cgz/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/3 LKP_LOCAL_RUN=1 selinux=0 debug apic=debug sysrq_always_enabled rcupdate.rcu_cpu_stall_timeout=100 net.ifnames=0 printk.devkmsg=on panic=-1 softlockup_panic=1 nmi_watchdog=panic oops=panic load_ramdisk=2 prompt_ramdisk=0 drbd.minor_count=8 systemd.log_level=err ignore_loglevel console=tty0 earlyprintk=ttyS0,115200 console=ttyS0,115200 vga=normal rw  ip=dhcp result_service=9p/virtfs_mount -initrd /home/carel/.lkp/cache/final_initrd -smp 2 -m 3144M -no-reboot -watchdog i6300esb -rtc base=localtime -device e1000,netdev=net0 -netdev user,id=net0 -display none -monitor null -serial stdio
> > early console in setup code
> > early console in extract_kernel
> > input_data: 0x0000000006ffc2e0
> > input_len: 0x000000000260cb2b
> > output: 0x0000000001000000
> > output_len: 0x00000000079e7da4
> > kernel_total_size: 0x0000000008a2c000
> > needed_size: 0x0000000008c00000
> > trampoline_32bit: 0x000000000009d000
> > Physical KASLR using RDTSC...
> > Virtual KASLR using RDTSC...
> > 
> > Decompressing Linux... Parsing ELF... Performing relocations... done.
> > Booting the kernel.
> > 
> >>
> >> -- 
> >> Regards/Gruss,
> >>     Boris.
> >>
> >> SUSE Software Solutions Germany GmbH, GF: Ivo Totev, HRB 36809, AG Nürnberg

-- 
Regards/Gruss,
    Boris.

SUSE Software Solutions Germany GmbH, GF: Ivo Totev, HRB 36809, AG Nürnberg

Powered by blists - more mailing lists