Date:   Mon, 2 Jan 2023 20:17:46 +0900
From:   Hyeonggon Yoo <42.hyeyoo@...il.com>
To:     Maxim Levitsky <mlevitsk@...hat.com>
Cc:     kernel test robot <oliver.sang@...el.com>,
        Vlastimil Babka <vbabka@...e.cz>, oe-lkp@...ts.linux.dev,
        lkp@...el.com, Mike Rapoport <rppt@...ux.ibm.com>,
        Christoph Lameter <cl@...ux.com>, linux-kernel@...r.kernel.org,
        linux-mm@...ck.org, Paolo Bonzini <pbonzini@...hat.com>,
        Wanpeng Li <wanpengli@...cent.com>,
        Vitaly Kuznetsov <vkuznets@...hat.com>,
        Juergen Gross <jgross@...e.com>,
        "Srivatsa S. Bhat" <srivatsa@...il.mit.edu>,
        Alexey Makhalov <amakhalov@...are.com>,
        VMware PV-Drivers Reviewers <pv-drivers@...are.com>,
        kvm@...r.kernel.org, Sean Christopherson <seanjc@...gle.com>
Subject: Re: supervisor write access in kernel mode in
 __pv_queued_spin_unlock_slowpath

On Sun, Jan 01, 2023 at 01:08:07PM +0200, Maxim Levitsky wrote:
> On Sun, 2023-01-01 at 16:37 +0900, Hyeonggon Yoo wrote:
> > On Sun, Jan 01, 2023 at 03:50:28PM +0900, Hyeonggon Yoo wrote:
> > > On Sat, Dec 31, 2022 at 11:26:25PM +0800, kernel test robot wrote:
> > > > Greeting,
> > > > 
> > > > FYI, we noticed kernel_BUG_at_include/linux/mm.h due to commit (built with gcc-11):
> > > > 
> > > > commit: 0af8489b0216fa1dd83e264bef8063f2632633d7 ("mm, slub: remove percpu slabs with CONFIG_SLUB_TINY")
> > > > https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
> > > > 
> > > > [test failed on linux-next/master c76083fac3bae1a87ae3d005b5cb1cbc761e31d5]
> > > > 
> > > > in testcase: rcutorture
> > > > version: 
> > > > with following parameters:
> > > > 
> > > > 	runtime: 300s
> > > > 	test: default
> > > > 	torture_type: tasks-tracing
> > > > 
> > > > test-description: rcutorture is rcutorture kernel module load/unload test.
> > > > test-url: https://www.kernel.org/doc/Documentation/RCU/torture.txt
> > > > 
> > > > 
> > > > on test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 16G
> > > > 
> > > > caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace):
> > > > 
> > > > 
> > > > If you fix the issue, kindly add following tag
> > > > > Reported-by: kernel test robot <oliver.sang@...el.com>
> > > > > Link: https://lore.kernel.org/oe-lkp/202212312021.bc1efe86-oliver.sang@intel.com
> > > 
> > > <snip>
> > > 
> > > > 
> > > > To reproduce:
> > > > 
> > > >         # build kernel
> > > > 	cd linux
> > > > 	cp config-6.1.0-rc2-00014-g0af8489b0216 .config
> > > > 	make HOSTCC=gcc-11 CC=gcc-11 ARCH=i386 olddefconfig prepare modules_prepare bzImage modules
> > > > 	make HOSTCC=gcc-11 CC=gcc-11 ARCH=i386 INSTALL_MOD_PATH=<mod-install-dir> modules_install
> > > > 	cd <mod-install-dir>
> > > > 	find lib/ | cpio -o -H newc --quiet | gzip > modules.cgz
> > > > 
> > > > 
> > > >         git clone https://github.com/intel/lkp-tests.git
> > > >         cd lkp-tests
> > > >         bin/lkp qemu -k <bzImage> -m modules.cgz job-script # job-script is attached in this email
> > > > 
> > > >         # if come across any failure that blocks the test,
> > > >         # please remove ~/.lkp and /lkp dir to run from a clean state.
> > > 
> > > I was unable to reproduce in the same way as described above
> > > because some files referenced in job-script couldn't be downloaded from
> > > download.01.org/0day :(
> > > 
> > > So I just built rcutorture module as builtin
> > > and I got weird spinlock bug on commit: 0af8489b0216
> > > ("mm, slub: remove percpu slabs with CONFIG_SLUB_TINY")
> > 
> > (+Cc KVM/Paravirt experts)
> > 
> > > full dmesg added as attachment
> > > 
> > > [ 1387.564837][   T57] BUG: unable to handle page fault for address: c108f5f4
> > > [ 1387.566649][   T57] #PF: supervisor write access in kernel mode
> > > [ 1387.567965][   T57] #PF: error_code(0x0003) - permissions violation
> > > [ 1387.569439][   T57] *pde = 010001e1 
> > > [ 1387.570276][   T57] Oops: 0003 [#1] SMP
> > > [ 1387.571149][   T57] CPU: 2 PID: 57 Comm: rcu_torture_rea Tainted: G S                 6.1.0-rc2-00010-g0af8489b0216 #2130 63d19ac2b985fca570c354d8750f489755de37ed
> > > [ 1387.574673][   T57] EIP: kvm_kick_cpu+0x54/0x90
> > > [ 1387.575802][   T57] Code: 2f c5 01 8b 04 9d e0 d4 4e c4 83 15 14 7b 2f c5 00 83 05 08 6d 2f c5 01 0f b7 0c 30 b8 05 00 00 00 83 15 0c 6d 2f c5 00 31 db <0f> 01 c1 83 05 10 6d 2f c5 01 8b 5d f8 8b 75 fc 83 15 14 6d 2f c5
> 
> 																				^^^^^^
> Yes this is the unfamous hypercall patching bug....
> 
> > > 
> 
> So what is happening is that Intel and AMD has a *slightly* different instruction reserved for hypercalls
> (paravirt calls from guest to host hypervisor).
> 
> KVM developers made a mistake to be 'nice' to the guests and if the guest uses the wrong hypercall instruction
> the KVM attempts to rewrite it with the right instruction.
> 
> That can fail, because to avoid security issues, KVM uses the exact same security context as the instruction itself
> (it is as if the instruction was defined such as it overwrote itself)
> This means that is the guest memory is marked read-only in the guest paging, then the write will fail and #PF
> will happen on the wrong hypercall instruction.
> 
> Here we have the Intel's instruction (VMCALL, 0f 01 C1), and the host machine is likely AMD which uses VMMCALL instruction
> which is (0F 01 D9)

Oh, right. my host machine is AMD Ryzen and seems I built kernel that does not
correctly support the machine.

> Now any recent Linux guest is supposed to use a right instruction using the alternatives mechanism, but it can if
> the hypervisor passes 'non native' vendor id, like GenunineIntel on AMD machine.

[    0.000000][    T0] KERNEL supported cpus:
[    0.000000][    T0]   Intel GenuineIntel
[    0.000000][    T0]   Vortex Vortex86 SoC
[    0.000000][    T0] CPU: vendor_id 'AuthenticAMD' unknown, using generic init.
[    0.000000][    T0] CPU: Your system may be unstable.
 
> In my testing using named CPU models like you do '-cpu SandyBridge' still passes through host vendor ID (that is the guest
> will see Intel's cpu but with vendor='AutheticAMD') but nobody confirmed me that this is a bug or a feature and I am not
> sure if older qemu versions also did this.
>
> Assuming that your host machine is AMD,
> your best bet to check if my theory is right is to boot the guest without triggering the bug, 
> and check in /proc/cpuinfo if the vendor string is 'GenuineIntel'

Same here. the vendor string is AuthenticAMD no matter if I pass -cpu SandyBridge
or -cpu host.

I didn't even imagine this could happen when using configuration the bot
passed without thinking and running it on CPU with different vendor :)

Thank you for such a kind explanation!

Phew, so this bug was totally unrelated the issue bot reported
and I have no clue why the original bug happened.

-- 
Regards,
Hyeonggon

Powered by blists - more mailing lists