linux-kernel - Re: vhost: linux-next: kernel crash at vhost_dev

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <CACGkMEu+Qn06bmp6br699Gk6SxTbSDZCW8mQ0qFUfDZUpAx62A@mail.gmail.com>
Date: Tue, 29 Jul 2025 15:44:09 +0800
From: Jason Wang <jasowang@...hat.com>
To: JAEHOON KIM <jhkim@...ux.ibm.com>
Cc: mst@...hat.com, kvm@...r.kernel.org, virtualization@...ts.linux.dev, 
	netdev@...r.kernel.org, linux-kernel@...r.kernel.org, jonah.palmer@...cle.com, 
	Eric Farman <farman@...ux.ibm.com>
Subject: Re: vhost: linux-next: kernel crash at vhost_dev_cleanup/kfree

On Tue, Jul 29, 2025 at 5:25 AM JAEHOON KIM <jhkim@...ux.ibm.com> wrote:
>
>
> Dear Jason Wang,
>
> I would like to kindly report a kernel crash issue on our s390x server
> which seems to be related to the following patch.
> --------------------------------------------------------------------------------------------------------------------------
>    commit 7918bb2d19c9 ("vhost: basic in order support")
> https://git.kernel.org/pub/scm/linux/kernel/git/mst/vhost.git/commit/?id=7918bb2d19c9
> --------------------------------------------------------------------------------------------------------------------------
>
> This patch landed in linux-next between July 16th and 17th. Since then,
> kernel crash have been observed during stress testing.
> The issue can be confirmed using the following command:
> -------------------------------------------
>    stress-ng --dev 1 -t 10s
> -------------------------------------------

Right, I forgot to initialize vq->nheads in vhost_dev_init().

I've posted a fix here:

https://lore.kernel.org/virtualization/20250729073916.80647-1-jasowang@redhat.com/T/#u

Thanks

>
> Crash log and call stack are as follows.
> Additionally, this crash appears similar to the issue discussed in the
> following thread:
> https://lore.kernel.org/kvm/bvjomrplpsjklglped5pmwttzmljigasdafjiizt2sfmytc5rr@ljpu455kx52j/
>
> [ 5413.029569] Unable to handle kernel pointer dereference in virtual
> kernel address space
> [ 5413.029573] Failing address: 00000328856e8000 TEID: 00000328856e8803
> [ 5413.029576] Fault in home space mode while using kernel ASCE.
> [ 5413.029580] AS:0000000371fdc007 R3:0000000000000024
> [ 5413.029607] Oops: 003b ilc:3 [#1]SMP
>    .......
> [ 5413.029655] CPU: 23 UID: 0 PID: 2339 Comm: stress-ng-dev Not tainted
> 6.16.0-rc6-10099-g60a66ed35d6b #63 NONE
> [ 5413.029659] Hardware name: IBM 3906 M05 780 (LPAR)
> [ 5413.029662] Krnl PSW : 0704e00180000000 0000032714b9f156
> (kfree+0x66/0x340)
> [ 5413.029673]            R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:2
> PM:0 RI:0 EA:3
> [ 5413.029677] Krnl GPRS: 0000000000000002 0000008c056e8000
> 0000262500000000 0000000085bf4610
> [ 5413.029681]            0000000085bf4660 0000000085bf4618
> 0000032716402270 0000032694e0391a
> [ 5413.029683]            0000032716402290 0000032714720000
> 00000328856e8000 0000262500000000
> [ 5413.029685]            000003ff8312cfa8 0000000000000000
> 000023015ba00000 000002a71e8d3ba8
> [ 5413.029697] Krnl Code: 0000032714b9f146: e3e060080008 ag %r14,8(%r6)
> [ 5413.029697]            0000032714b9f14c: ec1e06b93a59 risbgn
> %r1,%r14,6,185,58
> [ 5413.029697]           #0000032714b9f152: b90800a1 agr %r10,%r1
> [ 5413.029697]           >0000032714b9f156: e320a0080004 lg      %r2,8(%r10)
> [ 5413.029697]            0000032714b9f15c: a7210001 tmll    %r2,1
> [ 5413.029697]            0000032714b9f160: a77400e0 brc 7,0000032714b9f320
> [ 5413.029697]            0000032714b9f164: c004000000ca brcl
> 0,0000032714b9f2f8
> [ 5413.029697]            0000032714b9f16a: 95f5a030 cli 48(%r10),245
> [ 5413.029738] Call Trace:
> [ 5413.029741]  [<0000032714b9f156>] kfree+0x66/0x340
> [ 5413.029747]  [<0000032694e0391a>] vhost_dev_free_iovecs+0x9a/0xc0 [vhost]
> [ 5413.029757]  [<0000032694e05406>] vhost_dev_cleanup+0xb6/0x210 [vhost]
> [ 5413.029763]  [<000003269507000a>] vhost_vsock_dev_release+0x1aa/0x1e0
> [vhost_vsock]
> [ 5413.029768]  [<0000032714c16ece>] __fput+0xee/0x2e0
> [ 5413.029774]  [<00000327148c0488>] task_work_run+0x88/0xd0
> [ 5413.029783]  [<00000327148977aa>] do_exit+0x18a/0x4e0
> [ 5413.029786]  [<0000032714897cf0>] do_group_exit+0x40/0xc0
> [ 5413.029789]  [<0000032714897dce>] __s390x_sys_exit_group+0x2e/0x30
> [ 5413.029792]  [<00000327156519c6>] __do_syscall+0x136/0x340
> [ 5413.029797]  [<000003271565d5de>] system_call+0x6e/0x90
> [ 5413.029802] Last Breaking-Event-Address:
> [ 5413.029803]  [<0000032694e03914>] vhost_dev_free_iovecs+0x94/0xc0 [vhost]
> [ 5413.029811] Kernel panic - not syncing: Fatal exception: panic_on_oops
>
>
> Best regards,
> Jaehoon Kim
>