linux-kernel - RE: hyper_bf soft lockup on Azure Gen2 VM when taking kdump or executing kexec

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID:
 <BLAPR10MB521780F7C93DC013E2E031BDFDE92@BLAPR10MB5217.namprd10.prod.outlook.com>
Date: Thu, 30 Jan 2025 18:49:56 +0000
From: Thomas Tai <thomas.tai@...cle.com>
To: "mhkelley58@...il.com" <mhkelley58@...il.com>,
        "haiyangz@...rosoft.com"
	<haiyangz@...rosoft.com>,
        "wei.liu@...nel.org" <wei.liu@...nel.org>,
        "decui@...rosoft.com" <decui@...rosoft.com>,
        "drawat.floss@...il.com"
	<drawat.floss@...il.com>,
        "javierm@...hat.com" <javierm@...hat.com>,
        Helge
 Deller <deller@....de>, "daniel@...ll.ch" <daniel@...ll.ch>,
        "airlied@...il.com" <airlied@...il.com>,
        "tzimmermann@...e.de"
	<tzimmermann@...e.de>
CC: "dri-devel@...ts.freedesktop.org" <dri-devel@...ts.freedesktop.org>,
        "linux-fbdev@...r.kernel.org" <linux-fbdev@...r.kernel.org>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        "linux-hyperv@...r.kernel.org" <linux-hyperv@...r.kernel.org>
Subject: RE: hyper_bf soft lockup on Azure Gen2 VM when taking kdump or
 executing kexec

> -----Original Message-----
> From: Thomas Tai <thomas.tai@...cle.com>
> Sent: Thursday, January 30, 2025 1:37 PM
> To: mhkelley58@...il.com; haiyangz@...rosoft.com; wei.liu@...nel.org;
> decui@...rosoft.com; drawat.floss@...il.com; javierm@...hat.com; Helge
> Deller <deller@....de>; daniel@...ll.ch; airlied@...il.com;
> tzimmermann@...e.de
> Cc: dri-devel@...ts.freedesktop.org; linux-fbdev@...r.kernel.org; linux-
> kernel@...r.kernel.org; linux-hyperv@...r.kernel.org; Thomas Tai
> <thomas.tai@...cle.com>
> Subject: hyper_bf soft lockup on Azure Gen2 VM when taking kdump or
> executing kexec

Sorry for the typo in the subject title. It should have been 'hyperv_fb soft lockup on Azure Gen2 VM when taking kdump or executing kexec'

Thomas

> 
> Hi Michael,
> 
> We see an issue with the mainline kernel on the Azure Gen 2 VM when
> trying to induce a kernel panic with sysrq commands. The VM would hang
> with soft lockup. A similar issue happens when executing kexec on the VM.
> This issue is seen only with Gen2 VMs(with UEFI boot). Gen1 VMs with bios
> boot are fine.
> 
> git bisect identifies the issue is cased by the commit 20ee2ae8c5899
> ("fbdev/hyperv_fb: Fix logic error for Gen2 VMs in hvfb_getmem()" ).
> However, reverting the commit would cause the frame buffer not to work
> on the Gen2 VM.
> 
> Do you have any hints on what caused this issue?
> 
> To reproduce the issue with kdump:
> - Install mainline kernel on an Azure Gen 2 VM and trigger a kdump
> - echo 1 > /proc/sys/kernel/sysrq
> - echo c > /proc/sysrq-trigger
> 
> To reproduce the issue with executing kexec:
> - Install mainline kernel on Azure Gen 2 VM and use kexec
> - sudo kexec -l /boot/vmlinuz --initrd=/boot/initramfs.img --command-
> line="$( cat /proc/cmdline )"
> - sudo kexec -e
> 
> Thank you,
> Thomas
> 
> --- soft lockup log---
> [    1.690032] efifb: probing for efifb
> [    1.693989] efifb: framebuffer at 0x40000000, using 3072k, total 3072k
> [    1.700996] efifb: mode is 1024x768x32, linelength=4096, pages=1
> [    1.706999] efifb: scrolling: redraw
> [    1.710981] efifb: Truecolor: size=8:8:8:8, shift=24:16:8:0
> [    1.716806] Console: switching to colour frame buffer device 128x48
> [   29.176649] watchdog: BUG: soft lockup - CPU#0 stuck for 26s!
> [swapper/0:1]
> [   29.176655] Modules linked in:
> [   29.176658] CPU: 0 UID: 0 PID: 1 Comm: swapper/0 Not tainted 6.13.0+ #1
> [   29.176661] Hardware name: Microsoft Corporation Virtual
> Machine/Virtual Machine, BIOS Hyper-V UEFI Release v4.1 03/08/2024
> [   29.176663] RIP: 0010:fast_imageblit.isra.0+0x2de/0x460
> [   29.176669] Code: 07 49 21 f1 41 83 e1 01 46 8b 4c 8c 40 44 89 09 41 89 d1
> 4c 8d 79 08 41 c0 e9 06 49 21 f1 41 83 e1 03 46 8b 4c 8c 40 45 89 0e <41> 89
> d1 4c 8d 71 0c 41 c0 e9 05 49 21 f1 41 83 e1 07 46 8b 4c 8c
> [   29.176671] RSP: 0018:ffffc900000437a8 EFLAGS: 00010246
> [   29.176673] RAX: 0000000000000400 RBX: 0000000000000005 RCX:
> ffffc900006bb140
> [   29.176674] RDX: 0000000000000000 RSI: 0000000000000001 RDI:
> ffff888012146600
> [   29.176675] RBP: 0000000000000080 R08: ffffc900006bb000 R09:
> 0000000000000000
> [   29.176676] R10: 0000000000000001 R11: ffff888012146580 R12:
> 0000000000001000
> [   29.176677] R13: ffff88801214658a R14: ffffc900006bb144 R15:
> ffffc900006bb148
> [   29.176678] FS:  0000000000000000(0000) GS:ffff888026a00000(0000)
> knlGS:0000000000000000
> [   29.176680] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [   29.176681] CR2: 00007f8bc160c238 CR3: 000000002ac38002 CR4:
> 00000000003706b0
> [   29.176683] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> 0000000000000000
> [   29.176684] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> 0000000000000400
> [   29.176685] Call Trace:
> [   29.176687]  <IRQ>
> [   29.176690]  ? watchdog_timer_fn+0x220/0x2a0
> [   29.176695]  ? __pfx_watchdog_timer_fn+0x10/0x10
> [   29.176697]  ? __hrtimer_run_queues+0x112/0x2b0
> [   29.176702]  ? hrtimer_interrupt+0x108/0x270
> [   29.176704]  ? sched_clock_cpu+0x60/0x1a0
> [   29.176708]  ? __sysvec_hyperv_stimer0+0x32/0x60
> [   29.176712]  ? sysvec_hyperv_stimer0+0x70/0x90
> [   29.176715]  </IRQ>
> [   29.176716]  <TASK>
> [   29.176716]  ? asm_sysvec_hyperv_stimer0+0x1a/0x20
> [   29.176722]  ? fast_imageblit.isra.0+0x2de/0x460
> [   29.176724]  cfb_imageblit+0x433/0x470
> [   29.176726]  bit_putcs+0x291/0x570
> [   29.176731]  ? __pfx_bit_putcs+0x10/0x10
> [   29.176733]  fbcon_putcs+0x139/0x1a0
> [   29.176735]  do_update_region+0xf1/0x110
> [   29.176740]  redraw_screen+0x22f/0x290
> [   29.176743]  do_bind_con_driver.isra.0+0x2ab/0x3d0
> [   29.176745]  do_take_over_console+0x3a/0x50
> [   29.176747]  do_fbcon_takeover+0x5c/0xe0
> [   29.176749]  fbcon_fb_registered+0x4f/0x70
> [   29.176751]  do_register_framebuffer+0x1bc/0x2a0
> [   29.176755]  devm_register_framebuffer+0x28/0x90
> [   29.176757]  efifb_probe+0x544/0x720
> [   29.176760]  platform_probe+0x43/0xb0
> [   29.176763]  really_probe+0xd9/0x390
> [   29.176767]  ? __pfx___device_attach_driver+0x10/0x10
> [   29.176769]  __driver_probe_device+0x78/0x160
> [   29.176771]  driver_probe_device+0x1e/0xa0
> [   29.176773]  __device_attach_driver+0x99/0x130
> [   29.176775]  bus_for_each_drv+0x98/0xf0
> [   29.176777]  __device_attach+0xbc/0x1f0
> [   29.176779]  bus_probe_device+0x8d/0xb0
> [   29.176781]  device_add+0x4f1/0x6e0
> [   29.176785]  platform_device_add+0xfa/0x260
> [   29.176787]  sysfb_init+0x109/0x120
> [   29.176791]  ? __pfx_sysfb_init+0x10/0x10
> [   29.176793]  do_one_initcall+0x5b/0x330
> [   29.176796]  do_initcalls+0xac/0x130
> [   29.176800]  kernel_init_freeable+0x134/0x1e0
> [   29.176802]  ? __pfx_kernel_init+0x10/0x10
> [   29.176806]  kernel_init+0x1a/0x1d0
> [   29.176808]  ret_from_fork+0x34/0x50
> [   29.176813]  ? __pfx_kernel_init+0x10/0x10
> [   29.176815]  ret_from_fork_asm+0x1a/0x30
> [   29.176819]  </TASK>
> [  432.306986] fb0: EFI VGA frame buffer device