linux-kernel - RE: vmemmap alloc failure in hot_add

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <MWHPR21MB159397B915AFE4EF1FEEF301D70D9@MWHPR21MB1593.namprd21.prod.outlook.com>
Date:   Fri, 18 Jun 2021 00:16:53 +0000
From:   Michael Kelley <mikelley@...rosoft.com>
To:     David Hildenbrand <david@...hat.com>,
        Nathan Chancellor <nathan@...nel.org>
CC:     KY Srinivasan <kys@...rosoft.com>,
        Haiyang Zhang <haiyangz@...rosoft.com>,
        Stephen Hemminger <sthemmin@...rosoft.com>,
        Wei Liu <wei.liu@...nel.org>, Dexuan Cui <decui@...rosoft.com>,
        "linux-hyperv@...r.kernel.org" <linux-hyperv@...r.kernel.org>,
        "linux-mm@...ck.org" <linux-mm@...ck.org>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        Oscar Salvador <osalvador@...e.de>,
        Hillf Danton <hdanton@...a.com>
Subject: RE: vmemmap alloc failure in hot_add_req()

From: David Hildenbrand <david@...hat.com> Sent: Thursday, June 17, 2021 1:43 AM
> 
> > It does look like this kernel configuration has
> > CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y.
> 
> Okay, so then it's most likely really more of an issue with fragmented
> physical memory -- which is suboptimal but not a show blocker in your setup.
> 
> (there are still cases where memory onlining can fail, especially with
> kasan running, but these are rather corner cases)
> 
> >
> >> If it's not getting onlined, you easily sport after hotplug e.g., via
> >> "lsmem" that there are quite some offline memory blocks.
> >>
> >> Note that x86_64 code will fallback from populating huge pages to
> >> populating base pages for the vmemmap; this can happen easily when under
> >> memory pressure.
> >
> > Not sure if it is relevant or not but this warning can show up within a
> > minute of startup without me doing anything in particular.
> 
> I remember that Hyper-V will start with a certain (configured) boot VM
> memory size and once the guest is up and running, use memory stats of
> the guest to decide whether to add (hotplug) or remove (balloon inflate)
> memory from the VM.
> 
> So this could just be Hyper-V trying to apply its heuristics.

Nathan --

Could you clarify if your VM is running in the context of the Windows
Subsystem for Linux (WSL) v2 feature in Windows 10?  Or are you
running a "traditional" VM created using the Hyper-V Manager UI
or Powershell?

If the latter, how do you have the memory configuration set up?  In
the UI, first you can specify the RAM allocated to the VM.  Then
separately, you can enable the "Dynamic Memory" feature, in which
case you also specify a "Minimum RAM" and "Maximum RAM".  It
looks like you must have the "Dynamic Memory" feature enabled
since the original stack trace includes the hot_add_req() function
from the hv_balloon driver.

The Dynamic Memory feature is generally used only when you
need to allow Hyper-V to manage the allocation of physical memory
across multiple VMs.  Dynamic Memory is essentially Hyper-V's way of
allowing memory overcommit.  If you don't need that capability,
turning off Dynamic Memory and just specifying the amount of
memory you want to assign to the VM is the best course of action.

With Dynamic Memory enabled, you may have encountered a
situation where the memory needs of the VM grew very quickly,
and Hyper-V balloon driver got into a situation where it needed
to allocate memory in order to add memory, and it couldn't.  If
you want to continue to use the Dynamic Memory feature, then
you probably need to increase the initial amount of RAM assigned
to the VM (the "RAM" setting in the Hyper-V Manager UI).

Michael

> 
> >
> >> If adding memory would fail completely, you'd see another "hot_add
> >> memory failed error is ..." error message from hyper-v in the kernel
> >> log. If that doesn't show up, it's simply suboptimal, but hotplugging
> >> memory still succeeded.
> >
> > I did notice that from the code in hv_balloon.c but I do not think I
> > have ever seen that message in my logs.
> 
> Okay, so at least hotplugging memory is working.
> 
> --
> Thanks,
> 
> David / dhildenb