linux-kernel - fbdev deferred I/O broken in some scenarios

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID:
 <SN6PR02MB4157227300E59ACB3B0DABD0D4DE2@SN6PR02MB4157.namprd02.prod.outlook.com>
Date: Tue, 18 Mar 2025 02:05:11 +0000
From: Michael Kelley <mhklinux@...look.com>
To: "linux-fbdev@...r.kernel.org" <linux-fbdev@...r.kernel.org>
CC: "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
	"linux-hyperv@...r.kernel.org" <linux-hyperv@...r.kernel.org>
Subject: fbdev deferred I/O broken in some scenarios

I've been trying to get mmap() working with the hyperv_fb.c fbdev driver, which
is for Linux guests running on Microsoft's Hyper-V hypervisor. The hyperv_fb driver
uses fbdev deferred I/O for performance reasons. But it looks to me like fbdev
deferred I/O is fundamentally broken when the underlying framebuffer memory
is allocated from kernel memory (alloc_pages or dma_alloc_coherent).

The hyperv_fb.c driver may allocate the framebuffer memory in several ways,
depending on the size of the framebuffer specified by the Hyper-V host and the VM
"Generation".  For a Generation 2 VM, the framebuffer memory is allocated by the
Hyper-V host and is assigned to guest MMIO space. The hyperv_fb driver does a
vmalloc() allocation for deferred I/O to work against. This combination handles mmap()
of /dev/fb<n> correctly and the performance benefits of deferred I/O are substantial.

But for a Generation 1 VM, the hyperv_fb driver allocates the framebuffer memory in
contiguous guest physical memory using alloc_pages() or dma_alloc_coherent(), and
informs the Hyper-V host of the location. In this case, mmap() with deferred I/O does
not work. The mmap() succeeds, and user space updates to the mmap'ed memory are
correctly reflected to the framebuffer. But when the user space program does munmap()
or terminates, the Linux kernel free lists become scrambled and the kernel eventually
panics. The problem is that when munmap() is done, the PTEs in the VMA are cleaned
up, and the corresponding struct page refcounts are decremented. If the refcount goes
to zero (which it typically will), the page is immediately freed. In this way, some or all
of the framebuffer memory gets erroneously freed. From what I see, the VMA should
be marked VM_PFNMAP when allocated memory kernel is being used as the
framebuffer with deferred I/O, but that's not happening. The handling of deferred I/O
page faults would also need updating to make this work.

The fbdev deferred I/O support was originally added to the hyperv_fb driver in the
5.6 kernel, and based on my recent experiments, it has never worked correctly when
the framebuffer is allocated from kernel memory. fbdev deferred I/O support for using
kernel memory as the framebuffer was originally added in commit 37b4837959cb9
back in 2008 in Linux 2.6.29. But I don't see how it ever worked properly, unless
changes in generic memory management somehow broke it in the intervening years.

I think I know how to fix all this. But before working on a patch, I wanted to check
with the fbdev community to see if this might be a known issue and whether there
is any additional insight someone might offer. Thanks for any comments or help.

Michael Kelley