Message-ID: <b11156b7-e912-5b65-3b2f-4fd940727bc9@intel.com>
Date:   Sat, 18 Dec 2021 18:39:46 +0800
From:   Yin Fengwei <fengwei.yin@...el.com>
To:     Borislav Petkov <bp@...e.de>
CC:     Carel Si <beibei.si@...el.com>, Joerg Roedel <jroedel@...e.de>,
        LKML <linux-kernel@...r.kernel.org>, <x86@...nel.org>,
        <lkp@...ts.01.org>, <lkp@...el.com>, <bfields@...ldses.org>,
        <llvm@...ts.linux.dev>
Subject: Re: [LKP] Re: [x86/mm/64] f154f29085:
 BUG:kernel_reboot-without-warning_in_boot_stage - clang KCOV?


Hi Boris,

On 12/17/2021 8:52 PM, Borislav Petkov wrote:
> Add Bruce and clang folks to the party.
> 
> On Thu, Dec 16, 2021 at 08:21:15PM +0800, Yin Fengwei wrote:
>> Hi Boris,
>>
>> On 12/16/2021 7:58 PM, Carel Si wrote:
>>> Hi Boris,
>>>
>>> On Thu, Dec 16, 2021 at 11:06:59AM +0100, Borislav Petkov wrote:
>>>> On Thu, Dec 16, 2021 at 03:04:16PM +0800, Yin Fengwei wrote:
>>>>> The testing was with Qemu.
>>>>
>>>> This is hardly what I asked for.
>>>>
>>>>> And we found that the hang is related with clang-14.
>>>>
>>>> I saw that already.
>>>>
>>>>> The original report showed the kernel is built with clang-14:
>>>>>         # build kernel
>>>>> 	cd linux
>>>>> 	cp config-5.16.0-rc3-00003-gf154f290855b .config
>>>>> 	make HOSTCC=clang-14 CC=clang-14 ARCH=x86_64 olddefconfig prepare modules_prepare bzImage modules
>>>>> 	make HOSTCC=clang-14 CC=clang-14 ARCH=x86_64 INSTALL_MOD_PATH=<mod-install-dir> modules_install
>>>>
>>>> I saw that too.
>>>>
>>>>> Looks like KASAN related stub generated by clang-14 (KASAN_SHADOW_OFFSET and asan_report).
>>>>> This function is early function called before kasan_init.
>>>>>
>>>>> Looks like we need to disable KASAN_SANITIZE for arch/x86/kernel/cpu/common.c. So clang-14 will
>>>>> be happy with this kind of early TLB flush? Thanks.
>>>>
>>>> Ok, I don't understand: I asked for how exactly to reproduce and whether
>>>> you can send me your vmlinux you built with your clang-14. What I get is
>>>> some possible explanation about what might be happening.
>>>>
>>>> So what do you expect me to do? Say, "oh, sure, you're right, send me a
>>>> patch" without even being able to see for myself what the root cause is?
>>>>
>>>> What if it is not the kernel's fault but clang-14 is miscompiling crap
>>>> as in so many other cases?
>>>>
>>>> I built clang-14 and built with your .config and it works here fine. So
>>>> why does yours fail?
>>>>
>>>> Or what's the point of all this?
>>
>> I had concern that our report is an invalid report because you can't reproduce
>> it in your side. If that's the case, it could waste more your time. That's why
>> I did check and shared what I got. I am very sorry if I did it wrong.
> 
> Sure, you can always add your analysis but I'd like to reproduce myself
> too. So, in the future, please answer the questions and then feel free
> to add your analysis - I'll gladly have a look.
Thanks a lot for sharing this. Lessons learnt here. Will follow this
rule exactly in the future.

> 
> Which wasn't that far from the truth, btw.
> 
> But it isn't KASAN but GCOV profiling. Or is it KCOV profiling which
> clang does.
> 
> That thing adds some counting glue to native_write_cr4():
> 
> (my comments from the actual singlestepping in qemu start with '##' below)
> 
> 	movq	$__llvm_gcov_ctr.48+8, %rbx				##  mov    $0xffffffff8837d3c0,%rbx
> .LBB8_1:                                # %set_register
>                                         # =>This Inner Loop Header: Depth=1
> 
> 	jmp	.Ltmp42
> 	...
> 
> .Ltmp42:                                # Block address taken
> .LBB8_7:                                # %if.end79
>         movq    %rbx, %rax						## 0xffffffff8837d3c0
>         shrq    $3, %rax						## 0x1ffffffff106fa78
>         movabsq $-2305847407260205056, %rcx     # imm = 0xDFFFFC0000000000	## 0xdffffc0000000000
>         cmpb    $0, (%rax,%rcx)
>         je      .LBB8_9
> 
> so the memory address CMP accesses is something as nonsensical as
> 
>   0xfffffbfff106fa78
> 
> so I'm guessing we need to setup something for that __llvm_gcov_ctr to
> deref properly but I haven't dug deeper.
> 
> The important thing is that this triggers with clang-13 and -14. gcc is
> fine with the same config but that probably is because gcc does other
> profiling - gcov - I guess. Looking at the resulting asm, it has a bunch
> of those counter increments:
> 
>         incq    __gcov0.native_write_cr4+88(%rip)       # __gcov0.native_write_cr4[11]
> 
> but no weird memory references.
> 
> So, clang folks, what's up?
> 
> The fix is simple but I'd like to understand first why does this fail
> only with clang, 13 and newer.
> 
> (I mean, melver pointed me to
> 
>   380d53c45ff2 ("compiler_attributes.h: define __no_profile, add to noinstr")
> 
> which explains why 13 and newer).
> 
> Btw, joro, that second hunk is I think needed too because a couple of
> lines earlier we set up the cr4 shadow so I think you should use it
> instead of touching the hw CR4.
> 
> ---
> diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c
> index 0083464de5e3..79b3d67addcc 100644
> --- a/arch/x86/kernel/cpu/common.c
> +++ b/arch/x86/kernel/cpu/common.c
> @@ -384,7 +384,7 @@ void native_write_cr0(unsigned long val)
>  }
>  EXPORT_SYMBOL(native_write_cr0);
>  
> -void native_write_cr4(unsigned long val)
> +void __no_profile native_write_cr4(unsigned long val)
>  {
>  	unsigned long bits_changed = 0;
>  
> diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c
> index 75acb6027a87..68d2b7f9a913 100644
> --- a/arch/x86/kernel/head64.c
> +++ b/arch/x86/kernel/head64.c
> @@ -483,7 +483,7 @@ asmlinkage __visible void __init x86_64_start_kernel(char * real_mode_data)
>  	/* Kill off the identity-map trampoline */
>  	reset_early_page_tables();
>  
> -	__native_tlb_flush_global(native_read_cr4());
> +	__native_tlb_flush_global(this_cpu_read(cpu_tlbstate.cr4));
>  
>  	clear_bss();
>  
> 
> 
> Leaving in the rest for the newly added folks.
I tried this fix and it works fine in my local env. Will test it also in our
test box once we back to office. Thanks.

Regards
Yin, Fengwei

> 
>> If you don't want to use lkp tool to reproduce the issue, following command
>> could be used as well:
>>
>> Use Qemu command so only kernel image need be downloaded:
>> qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 16G -s -S -kernel vmlinuz-5.16.0-rc3-00003-gf154f290855b -nographic -append "console=ttyS0 earlyprintk=ttyS0,115200"
>> to reproduce it.
>>
>>
>>
>> Regards
>> Yin, Fengwei
>>
>>
>>
>>>>
>>>> I mean, if you cannot send me what I ask for, you can say so. Then I can
>>>> ignore this whole report altogether and waste my time somewhere else.
>>>
>>> We have uploaded vmlinuz, modules.cgz, config as well as other related file to:
>>> https://download.01.org/0day-ci/lkp-qemu/pkg/linux/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/
>>>
>>> Machine types can refer to:
>>> https://zerobin.net/?e107cf7b56495d80#MQLh14wUT9Osv1tWCwiQx/okkAN48Nq+drVPE0PiNPw=
>>>
>>> If there's any other msg needed, pls feel free to propose, thanks.
>>>
>>> Below are our full steps to reproduce the issue:
>>>
>>> # download lkp-tests
>>> $ git clone https://github.com/intel/lkp-tests.git
>>>
>>> $ cd lkp-tests/
>>>
>>> # download vmlinuz
>>> $ wget https://download.01.org/0day-ci/lkp-qemu/pkg/linux/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/vmlinuz-5.16.0-rc3-00003-gf154f290855b
>>>
>>> # dowmload modules.cgz
>>> $ wget https://download.01.org/0day-ci/lkp-qemu/pkg/linux/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/modules.cgz
>>>
>>> # download job-script which is attached
>>>
>>> # run lkp qemu
>>> lkp-tests$ sudo bin/lkp qemu -k vmlinuz-5.16.0-rc3-00003-gf154f290855b -m modules.cgz job-script
>>>
>>> ~/lkp-tests/pkg/lkp-src ~/lkp-tests
>>> x86_64
>>> ==> Making package: lkp-src 0-1 (Thu 16 Dec 2021 07:26:22 PM CST)
>>> ==> Checking runtime dependencies...
>>> ==> Checking buildtime dependencies...
>>> ==> WARNING: Using existing $srcdir/ tree
>>> ==> Removing existing $pkgdir/ directory...
>>> ==> Starting build()...
>>> make: Entering directory '/home/carel/lkp-tests/bin/event'
>>> klcc  -D_FORTIFY_SOURCE=2  -c -o wakeup.o wakeup.c
>>> klcc  -Wl,-O1,--sort-common,--as-needed,-z,relro -static -o wakeup wakeup.o
>>> rm -f wakeup.o
>>> strip wakeup
>>> make: Leaving directory '/home/carel/lkp-tests/bin/event'
>>> ==> Entering fakeroot environment...
>>> x86_64
>>> ==> Starting package()...
>>> ==> Creating package "lkp-src"...
>>> 103987 blocks
>>> renamed '/home/carel/.lkp/cache/lkp-x86_64.cgz.tmp' -> '/home/carel/.lkp/cache/lkp-x86_64.cgz'
>>> ==> Leaving fakeroot environment.
>>> ==> Finished making: lkp-src 0-1 (Thu 16 Dec 2021 07:26:24 PM CST)
>>> ~/lkp-tests
>>> 12 blocks
>>> result_root: /home/carel/.lkp//result/boot/1/vm-snb/debian-10.4-x86_64-20200603.cgz/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/0
>>> downloading initrds ...
>>> use local modules: /home/carel/.lkp/cache/modules.cgz
>>> /usr/bin/wget -q --timeout=1800 --tries=1 --local-encoding=UTF-8 http://0day.sh.intel.com:80/~lkp/osimage/debian/debian-10.4-x86_64-20200603.cgz -N -P /home/carel/.lkp/cache/osimage/debian
>>> 440459 blocks
>>> /usr/bin/wget -q --timeout=1800 --tries=1 --local-encoding=UTF-8 http://0day.sh.intel.com:80/~lkp/osimage/deps/debian-10.4-x86_64-20200603.cgz/run-ipconfig_20200608.cgz -N -P /home/carel/.lkp/cache/osimage/deps/debian-10.4-x86_64-20200603.cgz
>>> 1773 blocks
>>> /usr/bin/wget -q --timeout=1800 --tries=1 --local-encoding=UTF-8 http://0day.sh.intel.com:80/~lkp/osimage/deps/debian-10.4-x86_64-20200603.cgz/lkp_20210707.cgz -N -P /home/carel/.lkp/cache/osimage/deps/debian-10.4-x86_64-20200603.cgz
>>> 2321 blocks
>>> /usr/bin/wget -q --timeout=1800 --tries=1 --local-encoding=UTF-8 http://0day.sh.intel.com:80/~lkp/osimage/deps/debian-10.4-x86_64-20200603.cgz/rsync-rootfs_20200608.cgz -N -P /home/carel/.lkp/cache/osimage/deps/debian-10.4-x86_64-20200603.cgz
>>> 6856 blocks
>>> exec command: qemu-system-x86_64 -enable-kvm -fsdev local,id=test_dev,path=/home/carel/.lkp//result/boot/1/vm-snb/debian-10.4-x86_64-20200603.cgz/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/0,security_model=none -device virtio-9p-pci,fsdev=test_dev,mount_tag=9p/virtfs_mount -kernel vmlinuz-5.16.0-rc3-00003-gf154f290855b -append root=/dev/ram0 user=lkp job=/lkp/jobs/scheduled/vm-snb-192/boot-1-debian-10.4-x86_64-20200603.cgz-f154f290855b070cc94dd44ad253c0ef8a9337bb-20211208-23538-lnvkeg-5.yaml ARCH=x86_64 kconfig=x86_64-randconfig-a013-20211207 branch=tip/x86/mm commit=f154f290855b070cc94dd44ad253c0ef8a9337bb BOOT_IMAGE=/pkg/linux/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/vmlinuz-5.16.0-rc3-00003-gf154f290855b vmalloc=128M initramfs_async=0 page_owner=on max_uptime=600 RESULT_ROOT=/result/boot/1/vm-snb/debian-10.4-x86_64-20200603.cgz/x86_64-randconfig-a013-20211207/clang-14/f154f290855b070cc94dd44ad253c0ef8a9337bb/3 LKP_LOCAL_RUN=1 selinux=0 debug apic=debug sysrq_always_enabled rcupdate.rcu_cpu_stall_timeout=100 net.ifnames=0 printk.devkmsg=on panic=-1 softlockup_panic=1 nmi_watchdog=panic oops=panic load_ramdisk=2 prompt_ramdisk=0 drbd.minor_count=8 systemd.log_level=err ignore_loglevel console=tty0 earlyprintk=ttyS0,115200 console=ttyS0,115200 vga=normal rw  ip=dhcp result_service=9p/virtfs_mount -initrd /home/carel/.lkp/cache/final_initrd -smp 2 -m 3144M -no-reboot -watchdog i6300esb -rtc base=localtime -device e1000,netdev=net0 -netdev user,id=net0 -display none -monitor null -serial stdio
>>> early console in setup code
>>> early console in extract_kernel
>>> input_data: 0x0000000006ffc2e0
>>> input_len: 0x000000000260cb2b
>>> output: 0x0000000001000000
>>> output_len: 0x00000000079e7da4
>>> kernel_total_size: 0x0000000008a2c000
>>> needed_size: 0x0000000008c00000
>>> trampoline_32bit: 0x000000000009d000
>>> Physical KASLR using RDTSC...
>>> Virtual KASLR using RDTSC...
>>>
>>> Decompressing Linux... Parsing ELF... Performing relocations... done.
>>> Booting the kernel.
>>>
>>>>
>>>> -- 
>>>> Regards/Gruss,
>>>>     Boris.
>>>>
>>>> SUSE Software Solutions Germany GmbH, GF: Ivo Totev, HRB 36809, AG Nürnberg
> 

Powered by blists - more mailing lists