| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID:
<SN6PR02MB41579F60E39CA2A3CA8A5A75D467A@SN6PR02MB4157.namprd02.prod.outlook.com>
Date: Sat, 7 Feb 2026 01:30:11 +0000
From: Michael Kelley <mhklinux@...look.com>
To: Jan Kiszka <jan.kiszka@...mens.com>, "K. Y. Srinivasan"
<kys@...rosoft.com>, Haiyang Zhang <haiyangz@...rosoft.com>, Wei Liu
<wei.liu@...nel.org>, Dexuan Cui <decui@...rosoft.com>, Long Li
<longli@...rosoft.com>, Thomas Gleixner <tglx@...nel.org>, Ingo Molnar
<mingo@...hat.com>, Borislav Petkov <bp@...en8.de>, Dave Hansen
<dave.hansen@...ux.intel.com>, "x86@...nel.org" <x86@...nel.org>
CC: "linux-hyperv@...r.kernel.org" <linux-hyperv@...r.kernel.org>,
"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>, Florian
Bezdeka <florian.bezdeka@...mens.com>, RT <linux-rt-users@...r.kernel.org>,
Mitchell Levy <levymitchell0@...il.com>, "skinsburskii@...ux.microsoft.com"
<skinsburskii@...ux.microsoft.com>, "mrathor@...ux.microsoft.com"
<mrathor@...ux.microsoft.com>, "anirudh@...rudhrb.com"
<anirudh@...rudhrb.com>, "schakrabarti@...ux.microsoft.com"
<schakrabarti@...ux.microsoft.com>, "ssengar@...ux.microsoft.com"
<ssengar@...ux.microsoft.com>
Subject: RE: [PATCH] x86: mshyperv: Use kthread for vmbus interrupts on
PREEMPT_RT
From: Jan Kiszka <jan.kiszka@...mens.com> Sent: Thursday, February 5, 2026 10:41 PM
>
> On 05.02.26 19:55, Michael Kelley wrote:
> > From: Jan Kiszka <jan.kiszka@...mens.com> Sent: Tuesday, February 3, 2026 8:02 AM
> >>
> >> Resolves the following lockdep report when booting PREEMPT_RT on Hyper-V
> >> with related guest support enabled:
> >>
> >> [ 1.127941] hv_vmbus: registering driver hyperv_drm
> >>
> >> [ 1.132518] =============================
> >> [ 1.132519] [ BUG: Invalid wait context ]
> >> [ 1.132521] 6.19.0-rc8+ #9 Not tainted
> >> [ 1.132524] -----------------------------
> >> [ 1.132525] swapper/0/0 is trying to lock:
> >> [ 1.132526] ffff8b9381bb3c90 (&channel->sched_lock){....}-{3:3}, at: vmbus_chan_sched+0xc4/0x2b0
> >> [ 1.132543] other info that might help us debug this:
> >> [ 1.132544] context-{2:2}
> >> [ 1.132545] 1 lock held by swapper/0/0:
> >> [ 1.132547] #0: ffffffffa010c4c0 (rcu_read_lock){....}-{1:3}, at: vmbus_chan_sched+0x31/0x2b0
> >> [ 1.132557] stack backtrace:
> >> [ 1.132560] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 6.19.0-rc8+ #9 PREEMPT_{RT,(lazy)}
> >> [ 1.132565] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS Hyper-V UEFI Release v4.1 09/25/2025
> >> [ 1.132567] Call Trace:
> >> [ 1.132570] <IRQ>
> >> [ 1.132573] dump_stack_lvl+0x6e/0xa0
> >> [ 1.132581] __lock_acquire+0xee0/0x21b0
> >> [ 1.132592] lock_acquire+0xd5/0x2d0
> >> [ 1.132598] ? vmbus_chan_sched+0xc4/0x2b0
> >> [ 1.132606] ? lock_acquire+0xd5/0x2d0
> >> [ 1.132613] ? vmbus_chan_sched+0x31/0x2b0
> >> [ 1.132619] rt_spin_lock+0x3f/0x1f0
> >> [ 1.132623] ? vmbus_chan_sched+0xc4/0x2b0
> >> [ 1.132629] ? vmbus_chan_sched+0x31/0x2b0
> >> [ 1.132634] vmbus_chan_sched+0xc4/0x2b0
> >> [ 1.132641] vmbus_isr+0x2c/0x150
> >> [ 1.132648] __sysvec_hyperv_callback+0x5f/0xa0
> >> [ 1.132654] sysvec_hyperv_callback+0x88/0xb0
> >> [ 1.132658] </IRQ>
> >> [ 1.132659] <TASK>
> >> [ 1.132660] asm_sysvec_hyperv_callback+0x1a/0x20
> >>
> >> As code paths that handle vmbus IRQs use sleepy locks under PREEMPT_RT,
> >> the complete vmbus_handler execution needs to be moved into thread
> >> context. Open-coding this allows to skip the IPI that irq_work would
> >> additionally bring and which we do not need, being an IRQ, never an NMI.
> >>
> >> Signed-off-by: Jan Kiszka <jan.kiszka@...mens.com>
> >> ---
[snip]
> >>
> >> arch/x86/kernel/cpu/mshyperv.c | 52 ++++++++++++++++++++++++++++++++--
> >
> > You've added this code under arch/x86. But isn't it architecture independent? I
> > think it should also work on arm64. If that's the case, the code should probably
> > be added to drivers/hv/vmbus_drv.c instead.
> >
>
> I checked that before: arm64 uses normal IRQs, not over-optimized APIC
> vectors. And those IRQs are auto-threaded.
Just to clarify, with CONFIG_PREEMPT_RT=y, you expect the normal Linux
IRQ handling mechanism to offload a per-CPU interrupt handler from interrupt
level onto a thread?
>
> That said, someone with an arm64 Hyper-V deployment should still try to
> run things there once (PREEMPT_RT + PROVE_LOCKING). I don't have such a
> setup.
>
I've run your suggested experiment on an arm64 VM in the Azure cloud. My
kernel was linux-next 20260128. I set CONFIG_PREEMPT_RT=y and
CONFIG_PROVE_LOCKING=y, but did not add either of your two patches
(neither the storvsc driver patch nor the x86 VMBus interrupt handling patch).
The VM comes up and runs, but with this warning during boot:
[ 3.075604] hv_utils: Registering HyperV Utility Driver
[ 3.075636] hv_vmbus: registering driver hv_utils
[ 3.085920] =============================
[ 3.088128] hv_vmbus: registering driver hv_netvsc
[ 3.091180] [ BUG: Invalid wait context ]
[ 3.093544] 6.19.0-rc7-next-20260128+ #3 Tainted: G E
[ 3.097582] -----------------------------
[ 3.099899] systemd-udevd/284 is trying to lock:
[ 3.102568] ffff000100e24490 (&channel->sched_lock){....}-{3:3}, at: vmbus_chan_sched+0x128/0x3b8 [hv_vmbus]
[ 3.108208] other info that might help us debug this:
[ 3.111454] context-{2:2}
[ 3.112987] 1 lock held by systemd-udevd/284:
[ 3.115626] #0: ffffd5cfc20bcc80 (rcu_read_lock){....}-{1:3}, at: vmbus_chan_sched+0xcc/0x3b8 [hv_vmbus]
[ 3.121224] stack backtrace:
[ 3.122897] CPU: 0 UID: 0 PID: 284 Comm: systemd-udevd Tainted: G E 6.19.0-rc7-next-20260128+ #3 PREEMPT_RT
[ 3.129631] Tainted: [E]=UNSIGNED_MODULE
[ 3.131946] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS Hyper-V UEFI Release v4.1 06/10/2025
[ 3.138553] Call trace:
[ 3.140015] show_stack+0x20/0x38 (C)
[ 3.142137] dump_stack_lvl+0x9c/0x158
[ 3.144340] dump_stack+0x18/0x28
[ 3.146290] __lock_acquire+0x488/0x1e20
[ 3.148569] lock_acquire+0x11c/0x388
[ 3.150703] rt_spin_lock+0x54/0x230
[ 3.152785] vmbus_chan_sched+0x128/0x3b8 [hv_vmbus]
[ 3.155611] vmbus_isr+0x34/0x80 [hv_vmbus]
[ 3.158093] vmbus_percpu_isr+0x18/0x30 [hv_vmbus]
[ 3.160848] handle_percpu_devid_irq+0xdc/0x348
[ 3.163495] handle_irq_desc+0x48/0x68
[ 3.165851] generic_handle_domain_irq+0x20/0x38
[ 3.168664] gic_handle_irq+0x1dc/0x430
[ 3.170868] call_on_irq_stack+0x30/0x70
[ 3.173161] do_interrupt_handler+0x88/0xa0
[ 3.175724] el1_interrupt+0x4c/0xb0
[ 3.177855] el1h_64_irq_handler+0x18/0x28
[ 3.180332] el1h_64_irq+0x84/0x88
[ 3.182378] _raw_spin_unlock_irqrestore+0x4c/0xb0 (P)
[ 3.185493] rt_mutex_slowunlock+0x404/0x440
[ 3.187951] rt_spin_unlock+0xb8/0x178
[ 3.190394] kmem_cache_alloc_noprof+0xf0/0x4f8
[ 3.193100] alloc_empty_file+0x64/0x148
[ 3.195461] path_openat+0x58/0xaa0
[ 3.197658] do_file_open+0xa0/0x140
[ 3.199752] do_sys_openat2+0x190/0x278
[ 3.202124] do_sys_open+0x60/0xb8
[ 3.204047] __arm64_sys_openat+0x2c/0x48
[ 3.206433] invoke_syscall+0x6c/0xf8
[ 3.208519] el0_svc_common.constprop.0+0x48/0xf0
[ 3.211050] do_el0_svc+0x24/0x38
[ 3.212990] el0_svc+0x164/0x3c8
[ 3.214842] el0t_64_sync_handler+0xd0/0xe8
[ 3.217251] el0t_64_sync+0x1b0/0x1b8
[ 3.219450] hv_utils: Heartbeat IC version 3.0
[ 3.219471] hv_utils: Shutdown IC version 3.2
[ 3.219844] hv_utils: TimeSync IC version 4.0
I don't see an indication that vmbus_isr() has been offloaded from
interrupt level onto a thread. The stack starting with el1h_64_irq()
and going forward is the stack for normal per-cpu interrupt handling.
Maybe arm64 with PREEMPT_RT does the offload to a thread only
for SPIs and LPIs, but not for PPIs? I haven't looked at the source code
for how PREEMPT_RT affects arm64 interrupt handling.
Also, I had expected to see a problem with storvsc because I did
not apply your storvsc patch. But there was no such problem, even
with some disk I/O load (read only). arm64 VMs in Azure use exactly
the same virtual SCSI devices that are used with x86 VMs in Azure or
on local Hyper-V. I don't have an explanation. Will think about it.
Michael
Powered by blists - more mailing lists