Message-ID: <3c06cc1bb206f1ff9925dc4c7cf5a23f3f6c3505.camel@redhat.com>
Date: Mon, 10 Feb 2025 18:56:30 -0500
From: Maxim Levitsky <mlevitsk@...hat.com>
To: Michael Kelley <mhklinux@...look.com>, "thomas.tai@...cle.com"
	 <thomas.tai@...cle.com>, "mhkelley58@...il.com" <mhkelley58@...il.com>, 
	"haiyangz@...rosoft.com"
	 <haiyangz@...rosoft.com>, "wei.liu@...nel.org" <wei.liu@...nel.org>, 
	"decui@...rosoft.com"
	 <decui@...rosoft.com>, "drawat.floss@...il.com" <drawat.floss@...il.com>, 
	"javierm@...hat.com"
	 <javierm@...hat.com>, Helge Deller <deller@....de>, "daniel@...ll.ch"
	 <daniel@...ll.ch>, "airlied@...il.com" <airlied@...il.com>, 
	"tzimmermann@...e.de"
	 <tzimmermann@...e.de>
Cc: "dri-devel@...ts.freedesktop.org" <dri-devel@...ts.freedesktop.org>, 
	"linux-fbdev@...r.kernel.org"
	 <linux-fbdev@...r.kernel.org>, "linux-kernel@...r.kernel.org"
	 <linux-kernel@...r.kernel.org>, "linux-hyperv@...r.kernel.org"
	 <linux-hyperv@...r.kernel.org>
Subject: Re: hyper_bf soft lockup on Azure Gen2 VM when taking kdump or
 executing kexec

On Mon, 2025-02-10 at 21:35 +0000, Michael Kelley wrote:
> From: thomas.tai@...cle.com <thomas.tai@...cle.com> Sent: Monday, February 10, 2025 7:08 AM
> > <snip>
> > 
> > > > Then the question is why the efifb driver doesn't work in the kdump
> > > > kernel. Actually, it *does* work in many cases. I built the 6.13.0 kernel
> > > > on the Oracle Linux 9.4 system, and transferred the kernel image binary
> > > > and module binaries to an Ubuntu 20.04 VM in Azure. In that VM, the
> > > > efifb driver is loaded as part of the kdump kernel, and it doesn't cause
> > > > any problems. But there's an interesting difference. In the Oracle Linux
> > > > 9.4 VM, the efifb driver finds the framebuffer at 0x40000000, while on
> > > > the Ubuntu 20.04 VM, it finds the framebuffer at 0x40900000. This
> > > > difference is due to differences in how the screen_info variable gets
> > > > setup in the two VMs.
> > > > 
> > > > When the normal kernel starts in a freshly booted VM, Hyper-V provides
> > > > the EFI framebuffer at 0x40000000, and it works. But after the Hyper-V
> > > > FB driver or Hyper-V DRM driver has initialized, Linux has picked a
> > > > different MMIO address range and told Hyper-V to use the new
> > > > address range (which often starts at 0x40900000). A kexec does *not*
> > > > reset Hyper-V's transition to the new range, so when the efifb driver
> > > > tries to use the framebuffer at 0x40000000, the accesses trap to
> > > > Hyper-V and probably fail or timeout (I'm not sure of the details). After
> > > > the guest does some number of these bad references, Hyper-V considers
> > > > itself to be under attack from an ill-behaving guest, and throttles the
> > > > guest so that it doesn't run for a few seconds. The throttling repeats,
> > > > and results in extremely slow running in the kdump kernel.
> > > > 
> > > > Somehow in the Ubuntu 20.04 VM, the location of the frame buffer
> > > > as stored in screen_info.lfb_base gets updated to be 0x40900000. I
> > > > haven't fully debugged how that happens. But with that update, the
> > > > efifb driver is using the updated framebuffer address and it works. On
> > > > the Oracle Linux 9.4 system, that update doesn't appear to happen,
> > > > and the problem occurs.
> > > > 
> > > > This in an interim update on the problem. I'm still investigating how
> > > > screen_info.lfb_base is set in the kdump kernel, and why it is different
> > > > in the Ubuntu 20.04 VM vs. in the Oracle Linux 9.4 VM. Once that is
> > > > well understood, we can contemplate how to fix the problem. Undoing
> > > > the revert that is commit 2bebc3cd4870 doesn't seem like the solution
> > > > since the original code there was reported to cause many other issues.
> > > > The solution focus will likely be on how to ensure the kdump kernel gets
> > > > the correct framebuffer address so the efifb driver works, since the
> > > > framebuffer address changing is a quirk of Hyper-V behavior.
> > > > 
> > > > If anyone else has insight into what's going on here, please chime in.
> > > > What I've learned so far is still somewhat tentative.
> > > > 
> > > Here's what is happening. On Ubuntu 20.04, the kdump image is
> > > loaded into crash memory using the kexec command. Ubuntu 20.04
> > > has kexec from the kexec-tools package version 2.0.18-1ubuntu1.1,
> > > and per the kexec man page, it defaults to using the older kexec_load()
> > > system call. When using kexec_load(), the contents to be loaded into
> > > crash memory is constructed in user space by the kexec command.
> > > The kexec command gets the "screen_info" settings, including the
> > > physical address of the frame buffer, via the FBIOGET_FSCREENINFO
> > > ioctl against /dev/fb0. The Hyper-V FB or DRM driver registers itself
> > > with the fbdev subsystem so that it is /dev/fb0, and the ioctl returns
> > > the updated framebuffer address. So the efifb driver loads and runs
> > > correctly.
> > > 
> > > On Oracle Linux 9.4, the kdump image is also loaded with the
> > > kexec command, but from kexec-tools package version
> > > kexec-tools-2.0.28-1.0.10.el9_5.x86_64, which is slightly later than
> > > the version on Ubuntu 20.04. This newer kexec defaults to using the
> > > newer kexec_file_load() system call. This system call gets the
> > > framebuffer address from the screen_info variable in the kernel, which
> > > has not been updated to reflect the new framebuffer address. Hence
> > > in the kdump kernel, the efifb driver uses the old framebuffer address,
> > > and hence the problem.
> > > 
> > > To further complicate matters, the kexec on Oracle Linux 9.4 seems to
> > > have a bug when the -c option forces the use of kexec_load() instead
> > > of kexec_file_load(). As an experiment, I modified the kdumpctl shell
> > > script to add the "-c" option to kexec, but in that case the value "0x0"
> > > is passed as the framebuffer address, which is wrong. Furthermore,
> > > the " screen_info.orig_video_isVGA" value (which I mentioned earlier
> > > in connection with commit 2bebc3cd4870) is also set to 0, so the
> > > kdump kernel no longer thinks it has an EFI framebuffer. Hence the
> > > efifb driver isn't loaded, and the kdump works, though for the wrong
> > > reasons. If kexec 2.0.18 from Ubuntu is copied onto the Oracle Linux 9.4
> > > VM, then kdump works as expected, with the efifb driver being loaded
> > > and using the correct framebuffer address. So something is going wrong
> > > with kexec 2.0.28 in how it sets up the screen_info when the -c option
> > > is used. I'll leave the debugging of the kexec bug to someone else.
> > 
> > Hi Michael,
> > 
> > Do you think we need to handle Azure Gen2 VM differently in the kexec?
> > 
> > Or should we change the kexec_file_load() system call to retrieve the correct
> > framebuffer address?
> 
> I'm thinking there may be a fix in the Hyper-V FB and Hyper-V DRM drivers.
> Commit c25a19afb81c may also be a cause of the problem -- see precursor
> commit 3cb73bc3fa2a, which describes exactly the problem. I still need to
> do some testing, but without that commit, kdump won't detect that it has
> an EFI framebuffer, won't load the efifb driver, and so won't encounter the
> problem. But we probably need to get Thomas Zimmerman to weigh in on
> the implications of reverting c25a19afb81c.
> 
> There's one additional variation of the problem. Assume the Hyper-V FB
> driver is loaded (for example) during boot and moves the framebuffer. Then
> system runs kexec as part of arming kdump during the boot sequence.
> The most recent location of the framebuffer (and whether it is an EFI framebuffer)
> gets picked at the time kexec runs, and is stored in the crash kernel memory area.
> But what if the framebuffer later moves, perhaps because the Hyper-V FB driver
> is unbound? The crash kernel memory area doesn’t get updated and kdump
> could still have the wrong framebuffer address. This anomaly argues for the
> commit 3cb73bc3fa2a approach of just ensuring that the efifb driver doesn't
> load. Of course that approach means that the kdump kernel *must* contain
> either the Hyper-V FB or Hyper-V DRM driver in order to work on a system
> with only a framebuffer for text output. The efifb driver won't work. But
> perhaps that's OK.
> 
> Changing kexec (or the invoking script) to special case Hyper-V Gen 2 VMs and
> always use kexec_load() instead of kexec_file_load() sounds like a big hack
> to me.  And with that approach, you give up the ability to enforce loading only
> properly signed kdump images. This is something kexec_file_load() provides
> that kexec_load() doesn't, and is one of the main reasons that kexec_file_load()
> was added.
> 
> Whether the kexec_file_load() system call could be enhanced to get the
> frame buffer information from the /dev/fb0 device, I'm not sure. That might
> be a reasonable approach, though it still has the problem that the framebuffer
> address could change *after* kexec_file_load() runs.
> 
> Anyway, that's a dump of my current thoughts. I haven't reached a final
> conclusion or recommendation yet. Comments from others on the
> thread are welcome.

Hi!

Asking because I also had to do some digging in this area:

Do you think that the kernel can *ask* the hypervisor where the framebuffer is instead
of relying on bios, the bootloader and/or kexec to somehow provide this information?

If hyperv doesn't provide this API, how hard it would be in your opinion to provide it?

I am asking because, I also had to debug a RHEL downstream issue where a slightly botched backport
ensured that the first stage of the compressed uefi boot image, stopped passing the 'screen_info'
to the second stage (the kernel itself), and as a result of this, the second stage stopped
loading simplefb, and as a result of *this*, the PCI driver started to try to use the framebuffer
range for its own use which failed and resulted in a cryptic error.

If the kernel was to just issue some form of a hypercall to ask the hypervisor where the framebuffer currently is,
we could avoid a whole class of bugs similar to this.
What do you think?

Best regards,
	Maxim Levitsky


> 
> Michael
> 
> > Thank you,
> > Thomas
> > 
> > > I'm still thinking about alternatives to fix this mess. Please chime
> > > in if you have suggestions.
> > > 
> > > Michael

Powered by blists - more mailing lists