| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <20151211130258.GA4815@pd.tnic>
Date: Fri, 11 Dec 2015 14:02:58 +0100
From: Borislav Petkov <bp@...en8.de>
To: Paolo Bonzini <pbonzini@...hat.com>
Cc: linux-kernel@...r.kernel.org, kvm@...r.kernel.org,
Jörg Rödel <joro@...tes.org>
Subject: Re: [PATCH] kvm: x86: move tracepoints outside extended quiescent
state
On Fri, Dec 11, 2015 at 01:15:04PM +0100, Paolo Bonzini wrote:
> My wild guess is that RSP is getting corrupted, but I guess I'll have to try
> to reproduce to figure out what happens.
Yeah, something's scribbling over stuff where it shouldn't.
> The last thing I need from you (hopefully) is a Kconfig. If you have some
Attached.
> time, it would be great to check if you can reproduce it with an older kernel
> version---trying 4.4-rc1 and 4.3 would be great.
Just did 4.3 and got a much cleaner splat, see below. According to Code:, we're
trapping at:
2b:* ff 50 78 callq *0x78(%rax) <-- trapping instruction
in kvm_arch_vcpu_put() which is this:
movq kvm_x86_ops(%rip), %rax # kvm_x86_ops, kvm_x86_ops
...
call *120(%rax) # _2->vcpu_put
and RAX is 0. So ->vcpu_put() is 0?! I don't think so. So it must be
something corrupted kvm_x86_ops.
Hmmm.
...
[ 133.387161] kvm: zapping shadow pages for mmio generation wraparound
[ 135.998430] kvm [3717]: vcpu0 unhandled rdmsr: 0xc0011021
[ 136.359458] kvm [3717]: vcpu1 unhandled rdmsr: 0xc0011021
[ 136.466257] kvm [3717]: vcpu2 unhandled rdmsr: 0xc0011021
[ 136.563658] kvm [3717]: vcpu3 unhandled rdmsr: 0xc0011021
[ 136.663534] kvm [3717]: vcpu4 unhandled rdmsr: 0xc0011021
[ 136.763708] kvm [3717]: vcpu5 unhandled rdmsr: 0xc0011021
[ 136.869131] kvm [3717]: vcpu6 unhandled rdmsr: 0xc0011021
[ 136.967479] kvm [3717]: vcpu7 unhandled rdmsr: 0xc0011021
[ 245.585109] kvm: zapping shadow pages for mmio generation wraparound
[ 247.482552] kvm [3781]: vcpu0 unhandled rdmsr: 0xc0011021
[ 247.810769] kvm [3781]: vcpu1 unhandled rdmsr: 0xc0011021
[ 247.909217] kvm [3781]: vcpu2 unhandled rdmsr: 0xc0011021
[ 248.009079] kvm [3781]: vcpu3 unhandled rdmsr: 0xc0011021
[ 248.111232] kvm [3781]: vcpu4 unhandled rdmsr: 0xc0011021
[ 248.214389] kvm [3781]: vcpu5 unhandled rdmsr: 0xc0011021
[ 248.318640] kvm [3781]: vcpu6 unhandled rdmsr: 0xc0011021
[ 248.422611] kvm [3781]: vcpu7 unhandled rdmsr: 0xc0011021
[ 266.489947] qemu-system-x86[3783]: segfault at ffffffff816c90b5 ip ffffffff816c90b5 sp 00007f1f3b50ca48 error 15
[ 273.715102] kvm: zapping shadow pages for mmio generation wraparound
[ 275.742117] kvm [3831]: vcpu0 unhandled rdmsr: 0xc0011021
[ 276.119764] kvm [3831]: vcpu1 unhandled rdmsr: 0xc0011021
[ 276.220739] kvm [3831]: vcpu2 unhandled rdmsr: 0xc0011021
[ 276.325400] kvm [3831]: vcpu3 unhandled rdmsr: 0xc0011021
[ 276.426287] kvm [3831]: vcpu4 unhandled rdmsr: 0xc0011021
[ 276.529345] kvm [3831]: vcpu5 unhandled rdmsr: 0xc0011021
[ 276.634471] kvm [3831]: vcpu6 unhandled rdmsr: 0xc0011021
[ 276.733205] kvm [3831]: vcpu7 unhandled rdmsr: 0xc0011021
[ 667.184231] kvm: zapping shadow pages for mmio generation wraparound
[ 669.043526] kvm [3889]: vcpu0 unhandled rdmsr: 0xc0011021
[ 669.363923] kvm [3889]: vcpu1 unhandled rdmsr: 0xc0011021
[ 669.463309] kvm [3889]: vcpu2 unhandled rdmsr: 0xc0011021
[ 669.568626] kvm [3889]: vcpu3 unhandled rdmsr: 0xc0011021
[ 669.672595] kvm [3889]: vcpu4 unhandled rdmsr: 0xc0011021
[ 669.771024] kvm [3889]: vcpu5 unhandled rdmsr: 0xc0011021
[ 669.871012] kvm [3889]: vcpu6 unhandled rdmsr: 0xc0011021
[ 669.970899] kvm [3889]: vcpu7 unhandled rdmsr: 0xc0011021
[ 687.205933] BUG: unable to handle kernel NULL pointer dereference at 0000000000000078
[ 687.213883] IP: [<ffffffffa02b6b94>] kvm_arch_vcpu_put+0x14/0x40 [kvm]
[ 687.220484] PGD 0
[ 687.222529] Oops: 0000 [#1] PREEMPT SMP
[ 687.226526] Modules linked in: tun sha256_ssse3 sha256_generic drbg binfmt_misc ipv6 vfat fat fuse dm_crypt dm_mod kvm_amd kvm crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd amd64_edac_mod fam15h_power k10temp edac_mce_amd amdkfd amd_iommu_v2 radeon acpi_cpufreq
[ 687.226549] CPU: 6 PID: 3891 Comm: qemu-system-x86 Not tainted 4.3.0 #1
[ 687.226550] Hardware name: To be filled by O.E.M. To be filled by O.E.M./M5A97 EVO R2.0, BIOS 1503 01/16/2013
[ 687.226552] task: ffff88042986c740 ti: ffff880415164000 task.ti: ffff880415164000
[ 687.226553] RIP: 0010:[<ffffffffa02b6b94>]
[ 687.226570] [<ffffffffa02b6b94>] kvm_arch_vcpu_put+0x14/0x40 [kvm]
[ 687.226572] RSP: 0018:ffff880415167dd0 EFLAGS: 00010256
[ 687.226573] RAX: 0000000000000000 RBX: ffff8804157f8000 RCX: 00000000c0000102
[ 687.226574] RDX: 0000000000000000 RSI: ffffffffa029fa1b RDI: ffff8804157f8000
[ 687.226575] RBP: ffff880415167dd8 R08: 0000000000000001 R09: 0000000000000000
[ 687.226576] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000000
[ 687.226577] R13: ffff8804157f8000 R14: 0000000000000000 R15: 0000000000000000
[ 687.226579] FS: 00007ffff6353700(0000) GS:ffff88042d200000(0000) knlGS:0000000000000000
[ 687.226580] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 687.226581] CR2: 0000000000000078 CR3: 00000003f75e9000 CR4: 00000000000406e0
[ 687.226582] Stack:
[ 687.226583] ffff8804157f8000
[ 687.226584] ffff880415167df0
[ 687.226584] ffffffffa02b6b97
[ 687.226585] ffff8804157f8000
[ 687.226586] ffff880415167e08
[ 687.226587] ffffffffa029f8bf
[ 687.226587] 0000000000000000
[ 687.226588] ffff880415167e98
[ 687.226589] ffffffffa029fa1b
[ 687.226589] ffff880427df1a00
[ 687.226590] 0000000000000070
[ 687.226591] 0000000000004000
[ 687.226592] Call Trace:
[ 687.226610] [<ffffffffa02b6b97>] kvm_arch_vcpu_put+0x17/0x40 [kvm]
[ 687.226623] [<ffffffffa029f8bf>] vcpu_put+0x1f/0x60 [kvm]
[ 687.226636] [<ffffffffa029fa1b>] kvm_vcpu_ioctl+0x11b/0x6f0 [kvm]
[ 687.226640] [<ffffffff811932c7>] do_vfs_ioctl+0x2d7/0x530
[ 687.226643] [<ffffffff8119f179>] ? __fget_light+0x29/0x90
[ 687.226646] [<ffffffff8119356c>] SyS_ioctl+0x4c/0x90
[ 687.226650] [<ffffffff816c905b>] entry_SYSCALL_64_fastpath+0x16/0x73
[ 687.226652] Code: 83 6b de e0 e9 2a ff ff ff 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 8b 05 63 75 04 00 48 89 e5 53 48 89 fb <ff> 50 78 48 89 df e8 a1 fd ff ff 0f 31 48 c1 e2 20 48 09 d0 48
[ 687.226691] RIP [<ffffffffa02b6b94>] kvm_arch_vcpu_put+0x14/0x40 [kvm]
[ 687.226708] RSP <ffff880415167dd0>
[ 687.226710] CR2: 0000000000000078
[ 687.242688] ---[ end trace 41cdf7a208af97a1 ]---
[ 687.242746] note: qemu-system-x86[3891] exited with preempt_count 1
[ 687.226652] Code: 83 6b de e0 e9 2a ff ff ff 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 8b 05 63 75 04 00 48 89 e5 53 48 89 fb <ff> 50 78 48 89 df e8 a1 fd ff ff 0f 31 48 c1 e2 20 48 09 d0 48
All code
========
0: 83 6b de e0 subl $0xffffffe0,-0x22(%rbx)
4: e9 2a ff ff ff jmpq 0xffffffffffffff33
9: 0f 1f 40 00 nopl 0x0(%rax)
d: 66 2e 0f 1f 84 00 00 nopw %cs:0x0(%rax,%rax,1)
14: 00 00 00
17: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1)
1c: 55 push %rbp
1d: 48 8b 05 63 75 04 00 mov 0x47563(%rip),%rax # 0x47587
24: 48 89 e5 mov %rsp,%rbp
27: 53 push %rbx
28: 48 89 fb mov %rdi,%rbx
2b:* ff 50 78 callq *0x78(%rax) <-- trapping instruction
2e: 48 89 df mov %rbx,%rdi
31: e8 a1 fd ff ff callq 0xfffffffffffffdd7
36: 0f 31 rdtsc
38: 48 c1 e2 20 shl $0x20,%rdx
3c: 48 09 d0 or %rdx,%rax
3f: 48 rex.W
Code starting with the faulting instruction
===========================================
0: ff 50 78 callq *0x78(%rax)
3: 48 89 df mov %rbx,%rdi
6: e8 a1 fd ff ff callq 0xfffffffffffffdac
b: 0f 31 rdtsc
d: 48 c1 e2 20 shl $0x20,%rdx
11: 48 09 d0 or %rdx,%rax
14: 48 rex.W
00000000000120a0 <kvm_arch_vcpu_put>:
120a0: e8 00 00 00 00 callq 120a5 <kvm_arch_vcpu_put+0x5>
120a5: 55 push %rbp
120a6: 48 8b 05 00 00 00 00 mov 0x0(%rip),%rax # 120ad <kvm_arch_vcpu_put+0xd>
120ad: 48 89 e5 mov %rsp,%rbp
120b0: 53 push %rbx
120b1: 48 89 fb mov %rdi,%rbx
120b4: ff 50 78 callq *0x78(%rax)
120b7: 48 89 df mov %rbx,%rdi
120ba: e8 00 00 00 00 callq 120bf <kvm_arch_vcpu_put+0x1f>
120bf: 0f 31 rdtsc
120c1: 48 c1 e2 20 shl $0x20,%rdx
120c5: 48 09 d0 or %rdx,%rax
120c8: 48 89 83 e8 32 00 00 mov %rax,0x32e8(%rbx)
120cf: 5b pop %rbx
120d0: 5d pop %rbp
120d1: c3 retq
120d2: 0f 1f 40 00 nopl 0x0(%rax)
120d6: 66 2e 0f 1f 84 00 00 nopw %cs:0x0(%rax,%rax,1)
120dd: 00 00 00
.globl kvm_arch_vcpu_put
.type kvm_arch_vcpu_put, @function
kvm_arch_vcpu_put:
.LFB3643:
.loc 1 2742 0
.cfi_startproc
1: call __fentry__
pushq %rbp #
.cfi_def_cfa_offset 16
.cfi_offset 6, -16
.loc 1 2743 0
movq kvm_x86_ops(%rip), %rax # kvm_x86_ops, kvm_x86_ops
.loc 1 2742 0
movq %rsp, %rbp #,
.cfi_def_cfa_register 6
pushq %rbx #
.cfi_offset 3, -24
movq %rdi, %rbx # vcpu, vcpu
.loc 1 2743 0
call *120(%rax) # _2->vcpu_put
.loc 1 2744 0
movq %rbx, %rdi # vcpu,
call kvm_put_guest_fpu #
--
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.
View attachment ".config" of type "text/plain" (95234 bytes)
Powered by blists - more mailing lists