| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <4hr3olmtvvqccfb4xvqfs3zisozqjlvjbpdgk4tgbhhhv7vra3@haamqnpqtivp>
Date: Tue, 29 Jul 2025 11:57:14 +0200
From: Stefano Garzarella <sgarzare@...hat.com>
To: Jason Wang <jasowang@...hat.com>
Cc: Breno Leitao <leitao@...ian.org>, Will Deacon <will@...nel.org>,
"Michael S. Tsirkin" <mst@...hat.com>, eperezma@...hat.com, linux-arm-kernel@...ts.infradead.org,
kvm@...r.kernel.org, Stefan Hajnoczi <stefanha@...hat.com>, netdev@...r.kernel.org
Subject: Re: vhost: linux-next: crash at vhost_dev_cleanup()
On Tue, Jul 29, 2025 at 03:44:49PM +0800, Jason Wang wrote:
>On Thu, Jul 24, 2025 at 9:50 PM Breno Leitao <leitao@...ian.org> wrote:
>>
>> On Thu, Jul 24, 2025 at 02:52:08PM +0200, Stefano Garzarella wrote:
>> > On Thu, 24 Jul 2025 at 14:48, Breno Leitao <leitao@...ian.org> wrote:
>> > >
>> > > On Thu, Jul 24, 2025 at 09:44:38AM +0100, Will Deacon wrote:
>> > > > > > On Thu, 24 Jul 2025 at 09:48, Michael S. Tsirkin <mst@...hat.com> wrote:
>> > > > > > >
>> > > > > > > On Wed, Jul 23, 2025 at 08:04:42AM -0700, Breno Leitao wrote:
>> > > > > > > > Hello,
>> > > > > > > >
>> > > > > > > > I've seen a crash in linux-next for a while on my arm64 server, and
>> > > > > > > > I decided to report.
>> > > > > > > >
>> > > > > > > > While running stress-ng on linux-next, I see the crash below.
>> > > > > > > >
>> > > > > > > > This is happening in a kernel configure with some debug options (KASAN,
>> > > > > > > > LOCKDEP and KMEMLEAK).
>> > > > > > > >
>> > > > > > > > Basically running stress-ng in a loop would crash the host in 15-20
>> > > > > > > > minutes:
>> > > > > > > > # while (true); do stress-ng -r 10 -t 10; done
>> > > > > > > >
>> > > > > > > > >From the early warning "virt_to_phys used for non-linear address",
>> > > > > >
>> > > > > > mmm, we recently added nonlinear SKBs support in vhost-vsock [1],
>> > > > > > @Will can this issue be related?
>> > > > >
>> > > > > Good point.
>> > > > >
>> > > > > Breno, if bisecting is too much trouble, would you mind testing the commits
>> > > > > c76f3c4364fe523cd2782269eab92529c86217aa
>> > > > > and
>> > > > > c7991b44d7b44f9270dec63acd0b2965d29aab43
>> > > > > and telling us if this reproduces?
>> > > >
>> > > > That's definitely worth doing, but we should be careful not to confuse
>> > > > the "non-linear address" from the warning (which refers to virtual
>> > > > addresses that lie outside of the linear mapping of memory, e.g. in the
>> > > > vmalloc space) and "non-linear SKBs" which refer to SKBs with fragment
>> > > > pages.
>> > >
>> > > I've tested both commits above, and I see the crash on both commits
>> > > above, thus, the problem reproduces in both cases. The only difference
>> > > I noted is the fact that I haven't seen the warning before the crash.
>> > >
>> > >
>> > > Log against c76f3c4364fe ("vhost/vsock: Avoid allocating
>> > > arbitrarily-sized SKBs")
>> > >
>> > > Unable to handle kernel paging request at virtual address 0000001fc0000048
>> > > Mem abort info:
>> > > ESR = 0x0000000096000005
>> > > EC = 0x25: DABT (current EL), IL = 32 bits
>> > > SET = 0, FnV = 0
>> > > EA = 0, S1PTW = 0
>> > > FSC = 0x05: level 1 translation fault
>> > > Data abort info:
>> > > ISV = 0, ISS = 0x00000005, ISS2 = 0x00000000
>> > > CM = 0, WnR = 0, TnD = 0, TagAccess = 0
>> > > GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
>> > > user pgtable: 64k pages, 48-bit VAs, pgdp=0000000cdcf2da00
>> > > [0000001fc0000048] pgd=0000000000000000, p4d=0000000000000000, pud=0000000000000000
>> > > Internal error: Oops: 0000000096000005 [#1] SMP
>> > > Modules linked in: vfio_iommu_type1 vfio md4 crc32_cryptoapi ghash_generic unix_diag vhost_net tun vhost vhost_iotlb tap mpls_gso mpls_iptunnel mpls_router fou sch_fq ghes_edac tls tcp_diag inet_diag act_gact cls_bpf nvidia_c
>> > > CPU: 34 UID: 0 PID: 1727297 Comm: stress-ng-dev Kdump: loaded Not tainted 6.16.0-rc6-upstream-00027-gc76f3c4364fe #19 NONE
>> > > pstate: 23401009 (nzCv daif +PAN -UAO +TCO +DIT +SSBS BTYPE=--)
>> > > pc : kfree+0x48/0x2a8
>> > > lr : vhost_dev_cleanup+0x138/0x2b8 [vhost]
>> > > sp : ffff80013a0cfcd0
>> > > x29: ffff80013a0cfcd0 x28: ffff0008fd0b6240 x27: 0000000000000000
>> > > x26: 0000000000000000 x25: 0000000000000000 x24: 0000000000000000
>> > > x23: 00000000040e001f x22: ffffffffffffffff x21: ffff00014f1d4ac0
>> > > x20: 0000000000000001 x19: ffff00014f1d0000 x18: 0000000000000000
>> > > x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000
>> > > x14: 000000000000001f x13: 000000000000000f x12: 0000000000000001
>> > > x11: 0000000000000000 x10: 0000000000000402 x9 : ffffffdfc0000000
>> > > x8 : 0000001fc0000040 x7 : 0000000000000000 x6 : 0000000000000000
>> > > x5 : ffff000141931840 x4 : 0000000000000000 x3 : 0000000000000008
>> > > x2 : ffffffffffffffff x1 : ffffffffffffffff x0 : 0000000000010000
>> > > Call trace:
>> > > kfree+0x48/0x2a8 (P)
>> > > vhost_dev_cleanup+0x138/0x2b8 [vhost]
>> > > vhost_net_release+0xa0/0x1a8 [vhost_net]
>> >
>> > But here is the vhost_net, so I'm confused now.
>> > Do you see the same (vhost_net) also on 9798752 ("Add linux-next
>> > specific files for 20250721") ?
>>
>> I will need to reproduce, but, looking at my logs, I see the following
>> against: c76f3c4364fe ("vhost/vsock: Avoid allocating arbitrarily-sized SKBs").
>> The logs are a bit intermixed, probably there were multiple CPUs hitting
>> the same code path.
>>
>> virt_to_phys used for non-linear address: 000000001b662678 (0xffe61984a460)
>> WARNING: CPU: 15 PID: 112846 at arch/arm64/mm/physaddr.c:15 __virt_to_phys+0x80/0xa8
>> Modules linked in: vhost_vsock(E) vhost(E) vhost_iotlb(E) ghes_edac(E) tls(E) act_gact(E) cls_bpf(E) tcp_diag(E) inet_diag(E) ipmi_ssif(E) ipmi_devintf(E) ipmi_msghandler(E) sch_fq_codel(E) drm(E) backlight(E) drm_panel_orientation_quirks(E) acpi_power_meter(E) loop(E) efivarfs(E) autofs4(E)
>> CPU: 15 UID: 0 PID: 112846 Comm: stress-ng-dev Kdump: loaded Tainted: G W E N 6.16.0-rc6-upstream-00027-gc76f3c4364fe #16 PREEMPT(none)
>> Tainted: [W]=WARN, [E]=UNSIGNED_MODULE, [N]=TEST
>> pstate: 63401009 (nZCv daif +PAN -UAO +TCO +DIT +SSBS BTYPE=--)
>> pc : __virt_to_phys+0x80/0xa8
>> lr : __virt_to_phys+0x7c/0xa8
>> sp : ffff8001184d7a30
>> x29: ffff8001184d7a30 x28: 00000000000045d8 x27: 1fffe000e7e88014
>> x26: 1fffe000e7e888f7 x25: ffff0007e578bf00 x24: 1fffe000e7e88013
>> x23: 0000000000000000 x22: 0000ffe61984a460 x21: ffff00073f440098
>> x20: ffffff1000080000 x19: 0000ffe61984a460 x18: 0000000000000002
>> x17: 6666783028203837 x16: 3632363662313030 x15: 0000000000000001
>> x14: 1fffe006d52e90f2 x13: 0000000000000000 x12: 0000000000000000
>> x11: ffff6006d52e90f3 x10: 0000000000000002 x9 : cfc659a21c727d00
>> x8 : ffff800083c19000 x7 : 0000000000000001 x6 : 0000000000000001
>> x5 : ffff8001184d7398 x4 : ffff800084866d60 x3 : ffff8000805fdd94
>> x2 : 0000000000000001 x1 : 0000000000000004 x0 : 000000000000004b
>> Call trace:
>> __virt_to_phys+0x80/0xa8 (P)
>> kfree+0xac/0x4b0
>> vhost_dev_cleanup+0x484/0x8b0 [vhost]
>> vhost_vsock_dev_release+0x2f4/0x358 [vhost_vsock]
>> __fput+0x2b4/0x608
>> fput_close_sync+0xe8/0x1e0
>> __arm64_sys_close+0x84/0xd0
>> invoke_syscall+0x8c/0x208
>> do_el0_svc+0x128/0x1a0
>> el0_svc+0x58/0x160
>> el0t_64_sync_handler+0x78/0x108
>> el0t_64_sync+0x198/0x1a0
>> irq event stamp: 0
>> hardirqs last enabled at (0): [<0000000000000000>] 0x0
>> hardirqs last disabled at (0): [<ffff8000801d876c>] copy_process+0xd5c/0x29f8
>> softirqs last enabled at (0): [<ffff8000801d879c>] copy_process+0xd8c/0x29f8
>> softirqs last disabled at (0): [<0000000000000000>] 0x0
>> ---[ end trace 0000000000000000 ]---
>> Unable to handle kernel paging request at virtual address 0000040053791288
>> ------------[ cut here ]------------
>> lr : kfree+0xac/0x4b0
>> virt_to_phys used for non-linear address: 00000000290839fd (0x2500000000)
>> WARNING: CPU: 41 PID: 112845 at arch/arm64/mm/physaddr.c:15 __virt_to_phys+0x80/0xa8
>> Modules linked in: vhost_vsock(E) vhost(E) vhost_iotlb(E) ghes_edac(E) tls(E) act_gact(E) cls_bpf(E) tcp_diag(E) inet_diag(E) ipmi_ssif(E) ipmi_devintf(E) ipmi_msghandler(E) sch_fq_codel(E) drm(E) backlight(E) drm_panel_orientation_quirks(E) acpi_power_meter(E) loop(E) efivarfs(E) autofs4(E)
>> CPU: 41 UID: 0 PID: 112845 Comm: stress-ng-dev Kdump: loaded Tainted: G W E N 6.16.0-rc6-upstream-00027-gc76f3c4364fe #16 PREEMPT(none)
>> x23: 0000000000000001
>> Tainted: [W]=WARN, [E]=UNSIGNED_MODULE, [N]=TEST
>> pstate: 63401009 (nZCv daif +PAN -UAO +TCO +DIT +SSBS BTYPE=--)
>> pc : __virt_to_phys+0x80/0xa8
>> lr : __virt_to_phys+0x7c/0xa8
>> sp : ffff8001a8277a30
>> x29: ffff8001a8277a30 x28: 00000000000045d8 x27: 1fffe000e7e3c014
>> x26: 1fffe000e7e3c8f7 x25: ffff0007bcff0000 x24: 1fffe000e7e3c013
>> x23: 0000000000000000 x22: 0000002500000000 x21: ffff00073f1e0098
>> x20: ffffff1000080000 x19: 0000002500000000 x18: 0000000000000004
>> x17: 00000000ffffffff x16: 0000000000000001 x15: 0000000000000001
>> x14: 1fffe006d53920f2 x13: 0000000000000000 x12: 0000000000000000
>> sp : ffff8001184d7a50
>> x29: ffff8001184d7a60 x28: 00000000000045d8 x27: 1fffe000e7e88014
>> x26: 1fffe000e7e888f7 x25: ffff0007e578bf00 x24: 1fffe000e7e88013
>> x23: 0000000000000000 x22: 0000ffe61984a460 x21: ffff00073f440098
>> x20: 0000040053791280 x19: ffff80000d0b8bbc x18: 0000000000000002
>> x17: 6666783028203837 x16: 3632363662313030 x15: 0000000000000001
>> x22: 0000ffe6199a459d
>> x14: 1fffe006d52e90f2
>> x21: ffff00073f7f0098
>> x20: ffffff1000080000 x19: 0000ffe6199a459d x18: 0000000000000004
>> x17: 54455320203b2d2c x16: 0000000000000011 x15: 0000000000000001
>> x14: 1fffe006d52bb8f2 x13: 0000000000000000 x12: 0000000000000000
>> x11: ffff6006d52bb8f3 x10: dfff800000000000 x9 : 77521a2bd3e0be00
>> x8 : ffff800083c19000 x7 : 0000000000000000 x6 : ffff80008036bc2c
>> x5 : 0000000000000000 x4 : 0000000000000001 x3 : ffff8000805fdd94
>> x2 : 0000000000000001 x1 : 0000000000000004 x0 : 000000000000004b
>> Call trace:
>> __virt_to_phys+0x80/0xa8 (P)
>> kfree+0xac/0x4b0
>> vhost_dev_cleanup+0x484/0x8b0 [vhost]
>> vhost_vsock_dev_release+0x2f4/0x358 [vhost_vsock]
>> __fput+0x2b4/0x608
>> x11: ffff6006d53920f3
>> fput_close_sync+0xe8/0x1e0
>> __arm64_sys_close+0x84/0xd0
>> invoke_syscall+0x8c/0x208
>> do_el0_svc+0x128/0x1a0
>> el0_svc+0x58/0x160
>> el0t_64_sync_handler+0x78/0x108
>> el0t_64_sync+0x198/0x1a0
>> irq event stamp: 0
>> x13: 0000000000000000
>> hardirqs last enabled at (0): [<0000000000000000>] 0x0
>> hardirqs last disabled at (0): [<ffff8000801d876c>] copy_process+0xd5c/0x29f8
>>
>>
>> > The initial report contained only vhost_vsock traces IIUC, so I'm
>> > suspecting something in the vhost core.
>>
>> Right, it seems we are hitting the same code path, on on both
>> vhost_vsock and vhost_net.
>>
>
>I've posted a fix here:
>
>https://lore.kernel.org/virtualization/20250729073916.80647-1-jasowang@redhat.com/T/#u
>
>I think it should address this issue.
Thanks for the fix!
Stefano
Powered by blists - more mailing lists