Message-ID: <20250401111159.7632a0fa@gandalf.local.home>
Date: Tue, 1 Apr 2025 11:11:59 -0400
From: Steven Rostedt <rostedt@...dmis.org>
To: Mike Rapoport <rppt@...nel.org>
Cc: Linus Torvalds <torvalds@...ux-foundation.org>,
 linux-kernel@...r.kernel.org, linux-trace-kernel@...r.kernel.org, Masami
 Hiramatsu <mhiramat@...nel.org>, Mark Rutland <mark.rutland@....com>,
 Mathieu Desnoyers <mathieu.desnoyers@...icios.com>, Andrew Morton
 <akpm@...ux-foundation.org>, Vincent Donnefort <vdonnefort@...gle.com>,
 Vlastimil Babka <vbabka@...e.cz>, Kees Cook <kees@...nel.org>, Tony Luck
 <tony.luck@...el.com>, "Guilherme G. Piccoli" <gpiccoli@...lia.com>,
 linux-hardening@...r.kernel.org
Subject: Re: [PATCH v2 1/2] tracing: ring-buffer: Have the ring buffer code
 do the vmap of physical memory

On Tue, 1 Apr 2025 12:56:31 +0300
Mike Rapoport <rppt@...nel.org> wrote:

> > For example, using "mem=" on the kernel command line will literally
> > limit the amount of RAM the kernel will use, and in doing so will
> > limit the page allocations too.  
> 
> And using memmap=m$n on x86 creates a hole in System RAM that does not have
> neither struct page nor kernel mappings and it is never considered RAM
> anywhere in mm.

Hmm, when that is used, then we had better not "free" the buffer.

>  
> > IOW, all of these kernel command line things are *subtle*.
> > 
> > Don't mis-use them by then making assumptions about how they work
> > today (or how they will work tomorrow).  
> 
> I'd say it's better not to use them at all. They cause weirdness in memory
> layout and also they are inconsistent in how architectures implement them.
>  
> > > Mike can correct me if I'm wrong, but the memory that was stolen was actual
> > > memory returned by the system (E820 in x86). It reserves the memory before
> > > the memory allocation reserves this memory. So what reserve_mem returns is
> > > valid memory that can be used by memory allocator, but is currently just
> > > "reserved" which means it wants to prevent the allocator from using it.  
> > 
> > That may indeed be true of reserve_mem.  
> 
> The reserve_mem behaves like any other early allocation, it has proper
> struct pages (PG_Reserved) and it is mapped in the direct map so
> phys_to_virt() will work on it.
> 
> As for mapping it to userspace, vm_iomap_memory() seems the best API to
> use. It has all the alignment checks and will refuse to map ranges that are
> not properly aligned and it will use vma information to create the right
> mappings.
>  

When using vmap() to get the virtual addresses (via the kmalloc_array() of
struct pages), the vunmap() gives the memory back to the memory allocator:

~# free
               total        used        free      shared  buff/cache   available
Mem:         8185928      296676     7840576         920      148280     7889252
Swap:        7812092           0     7812092
~# rmdir /sys/kernel/tracing/instances/boot_mapped/
~# free
               total        used        free      shared  buff/cache   available
Mem:         8206404      290868     7866772         920      148384     7915536
Swap:        7812092           0     7812092

With no issues.

But if I use vmap_page_range(), how do I give that back to the memory allocator?

Calling vunmap() on that memory gives me:

 1779.832484] ------------[ cut here ]------------
[ 1779.834076] Trying to vunmap() nonexistent vm area (000000027c000000)
[ 1779.835941] WARNING: CPU: 6 PID: 956 at mm/vmalloc.c:3413 vunmap+0x5a/0x60
[ 1779.837587] Modules linked in:
[ 1779.838455] CPU: 6 UID: 0 PID: 956 Comm: rmdir Not tainted 6.14.0-rc4-test-00019-ga9c509c0c8e7-dirty #379
[ 1779.840597] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[ 1779.842576] RIP: 0010:vunmap+0x5a/0x60
[ 1779.843553] Code: 89 c7 48 85 c0 74 12 e8 94 e1 01 00 48 8b 5d f8 c9 c3 cc cc cc cc 90 0f 0b 90 48 c7 c7 78 b6 c4 9d 48 89 de e8 57 b4 cd ff 90 <0f> 0b 90 90 eb dc 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90
[ 1779.847159] RSP: 0018:ffffbfdb7ec93da0 EFLAGS: 00010282
[ 1779.848256] RAX: 0000000000000000 RBX: 000000027c000000 RCX: 0000000000000000
[ 1779.849623] RDX: ffff9f0efdfab108 RSI: ffff9f0efdf9cbc0 RDI: 0000000000000001
[ 1779.851079] RBP: ffffbfdb7ec93da8 R08: 00000000ffffdfff R09: ffffffff9e7652c8
[ 1779.852447] R10: ffffffff9e6b5320 R11: 0000000000000000 R12: ffff9f0d80226e00
[ 1779.853746] R13: 0000000000000001 R14: ffff9f0d806294c0 R15: ffff9f0d80629190
[ 1779.855121] FS:  00007f414dea6740(0000) GS:ffff9f0efdf80000(0000) knlGS:0000000000000000
[ 1779.856524] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1779.857548] CR2: 00007f28a0a4d350 CR3: 000000010f950002 CR4: 0000000000172ef0
[ 1779.858916] Call Trace:
[ 1779.859467]  <TASK>
[ 1779.859986]  ? show_regs.cold+0x19/0x24
[ 1779.860708]  ? vunmap+0x5a/0x60
[ 1779.861387]  ? __warn.cold+0xc2/0x157
[ 1779.862141]  ? vunmap+0x5a/0x60
[ 1779.862752]  ? report_bug+0x10a/0x150
[ 1779.865787]  ? handle_bug+0x5c/0xa0
[ 1779.866684]  ? exc_invalid_op+0x1c/0x80
[ 1779.867455]  ? asm_exc_invalid_op+0x1f/0x30
[ 1779.868256]  ? vunmap+0x5a/0x60
[ 1779.868908]  ring_buffer_free+0xac/0xc0
[ 1779.869595]  __remove_instance.part.0.constprop.0+0xeb/0x1f0
[ 1779.870590]  instance_rmdir+0xe1/0xf0
[ 1779.871342]  tracefs_syscall_rmdir+0x5c/0xa0
[ 1779.872198]  vfs_rmdir+0xa0/0x220
[ 1779.872806]  do_rmdir+0x146/0x190
[ 1779.873477]  __x64_sys_rmdir+0x43/0x70
[ 1779.874188]  x64_sys_call+0x114f/0x1d70
[ 1779.874944]  do_syscall_64+0xbb/0x1d0
[ 1779.875605]  entry_SYSCALL_64_after_hwframe+0x77/0x7f

What's the proper way to say: "I no longer need this physical memory I
reserved, the kernel can now use it"?

-- Steve

Powered by blists - more mailing lists