Message-ID:
 <SN6PR02MB41578AC54B7C0B7386B8ED00D4F62@SN6PR02MB4157.namprd02.prod.outlook.com>
Date: Thu, 6 Feb 2025 21:00:15 +0000
From: Michael Kelley <mhklinux@...look.com>
To: Michael Kelley <mhklinux@...look.com>, Thomas Tai <thomas.tai@...cle.com>,
	"mhkelley58@...il.com" <mhkelley58@...il.com>, "haiyangz@...rosoft.com"
	<haiyangz@...rosoft.com>, "wei.liu@...nel.org" <wei.liu@...nel.org>,
	"decui@...rosoft.com" <decui@...rosoft.com>, "drawat.floss@...il.com"
	<drawat.floss@...il.com>, "javierm@...hat.com" <javierm@...hat.com>, Helge
 Deller <deller@....de>, "daniel@...ll.ch" <daniel@...ll.ch>,
	"airlied@...il.com" <airlied@...il.com>, "tzimmermann@...e.de"
	<tzimmermann@...e.de>
CC: "dri-devel@...ts.freedesktop.org" <dri-devel@...ts.freedesktop.org>,
	"linux-fbdev@...r.kernel.org" <linux-fbdev@...r.kernel.org>,
	"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
	"linux-hyperv@...r.kernel.org" <linux-hyperv@...r.kernel.org>
Subject: RE: hyper_bf soft lockup on Azure Gen2 VM when taking kdump or
 executing kexec

From: Michael Kelley <mhklinux@...look.com>
> 
> From: Thomas Tai <thomas.tai@...cle.com> Sent: Thursday, January 30, 2025 12:44 PM
> >
> > > -----Original Message-----
> > > From: Michael Kelley <mhklinux@...look.com> Sent: Thursday, January 30, 2025 3:20 PM
> > >
> > > From: Thomas Tai <thomas.tai@...cle.com> Sent: Thursday, January 30,
> > > 2025 10:50 AM
> > > >
> > > > Sorry for the typo in the subject title. It should have been 'hyperv_fb soft lockup on
> > > > Azure Gen2 VM when taking kdump or executing kexec'
> > > >
> > > > Thomas
> > > >
> > > > >
> > > > > Hi Michael,
> > > > >
> > > > > We see an issue with the mainline kernel on the Azure Gen 2 VM when
> > > > > trying to induce a kernel panic with sysrq commands. The VM would hang
> > > > > with soft lockup. A similar issue happens when executing kexec on the VM.
> > > > > This issue is seen only with Gen2 VMs(with UEFI boot). Gen1 VMs with bios
> > > > > boot are fine.
> > > > >
> > > > > git bisect identifies the issue is cased by the commit 20ee2ae8c5899
> > > > > ("fbdev/hyperv_fb: Fix logic error for Gen2 VMs in hvfb_getmem()" ).
> > > > > However, reverting the commit would cause the frame buffer not to work
> > > > > on the Gen2 VM.
> > > > >
> > > > > Do you have any hints on what caused this issue?
> > > > >
> > > > > To reproduce the issue with kdump:
> > > > > - Install mainline kernel on an Azure Gen 2 VM and trigger a kdump
> > > > > - echo 1 > /proc/sys/kernel/sysrq
> > > > > - echo c > /proc/sysrq-trigger
> > > > >
> > > > > To reproduce the issue with executing kexec:
> > > > > - Install mainline kernel on Azure Gen 2 VM and use kexec
> > > > > - sudo kexec -l /boot/vmlinuz --initrd=/boot/initramfs.img --command-
> > > > > line="$( cat /proc/cmdline )"
> > > > > - sudo kexec -e
> > > > >
> > > > > Thank you,
> > > > > Thomas
> > >
> > > I will take a look, but it might be early next week before I can do so.
> > >
> >
> > Thank you, Michael for your help!
> >
> > > It looks like your soft lockup log below is from the kdump kernel (or the newly
> > > kexec'ed kernel). Can you confirm? Also, this looks like a subset of the full log.
> >
> > Yes, the soft lockup log below is from the kdump kernel.
> >
> > > Do you have the full serial console log that you could email to me?  Seeing
> > > everything might be helpful. Of course, I'll try to repro the problem myself
> > > as well.
> >
> > I have attached the complete bootup and kdump kernel log.
> >
> > File: bootup_and_kdump.log
> > Line 1 ... 984 (bootup log)
> > Line 990       (kdump kernel booting up)
> > Line 1351      (soft lockup)
> >
> > Thank you,
> > Thomas
> >
> 
> I have reproduced the problem in an Azure VM running Oracle Linux
> 9.4 with the 6.13.0 kernel. Interestingly, the problem does not occur
> in a VM running on a locally installed Hyper-V with Ubuntu 20.04 and
> the 6.13.0 kernel. There are several differences in the two
> environments:  the version of Hyper-V, the VM configuration, the Linux
> distro, and the .config file used to build the 6.13.0 kernel. I'll try to
> figure out what make the difference, and then the root cause.
> 

This has been a real bear to investigate. :-(  The key observation
is that with older kernel versions, the efifb driver does *not* try
to load when running in the kdump kernel, and everything works.
In newer kernels, the efifb driver *does* try to load, and it appears
to hang.  (Actually, it is causing the VM to run very slowly. More on
that in a minute.)

I've bisected the kernel again, compensating for the fact that commit
20ee2ae8c5899 is needed to make the Hyper-V frame buffer work. With
that compensation, the actual problematic commit is 2bebc3cd4870
(Revert "firmware/sysfb: Clear screen_info state after consuming it").
Doing the revert causes screen_info.orig_video_isVGA to retain its value
of 0x70 (VIDEO_TYPE_EFI), which the kdump kernel picks up, causing it
to load the efifb driver.

Then the question is why the efifb driver doesn't work in the kdump
kernel. Actually, it *does* work in many cases. I built the 6.13.0 kernel
on the Oracle Linux 9.4 system, and transferred the kernel image binary
and module binaries to an Ubuntu 20.04 VM in Azure. In that VM, the
efifb driver is loaded as part of the kdump kernel, and it doesn't cause
any problems. But there's an interesting difference. In the Oracle Linux
9.4 VM, the efifb driver finds the framebuffer at 0x40000000, while on
the Ubuntu 20.04 VM, it finds the framebuffer at 0x40900000. This
difference is due to differences in how the screen_info variable gets
setup in the two VMs.

When the normal kernel starts in a freshly booted VM, Hyper-V provides
the EFI framebuffer at 0x40000000, and it works. But after the Hyper-V
FB driver or Hyper-V DRM driver has initialized, Linux has picked a
different MMIO address range and told Hyper-V to use the new
address range (which often starts at 0x40900000). A kexec does *not*
reset Hyper-V's transition to the new range, so when the efifb driver
tries to use the framebuffer at 0x40000000, the accesses trap to
Hyper-V and probably fail or timeout (I'm not sure of the details). After
the guest does some number of these bad references, Hyper-V considers
itself to be under attack from an ill-behaving guest, and throttles the
guest so that it doesn't run for a few seconds. The throttling repeats,
and results in extremely slow running in the kdump kernel.

Somehow in the Ubuntu 20.04 VM, the location of the frame buffer
as stored in screen_info.lfb_base gets updated to be 0x40900000. I
haven't fully debugged how that happens. But with that update, the
efifb driver is using the updated framebuffer address and it works. On
the Oracle Linux 9.4 system, that update doesn't appear to happen,
and the problem occurs.

This in an interim update on the problem. I'm still investigating how
screen_info.lfb_base is set in the kdump kernel, and why it is different
in the Ubuntu 20.04 VM vs. in the Oracle Linux 9.4 VM. Once that is
well understood, we can contemplate how to fix the problem. Undoing
the revert that is commit 2bebc3cd4870 doesn't seem like the solution
since the original code there was reported to cause many other issues.
The solution focus will likely be on how to ensure the kdump kernel gets
the correct framebuffer address so the efifb driver works, since the
framebuffer address changing is a quirk of Hyper-V behavior.

If anyone else has insight into what's going on here, please chime in.
What I've learned so far is still somewhat tentative.

Michael

Powered by blists - more mailing lists