Message-ID: <fd6e84d5-9dba-47fb-a39e-1f7f0995fdf5@fujitsu.com>
Date: Mon, 29 Jul 2024 00:37:26 +0000
From: "Zhijian Li (Fujitsu)" <lizhijian@...itsu.com>
To: Michal Hocko <mhocko@...e.com>
CC: "linux-mm@...ck.org" <linux-mm@...ck.org>, Andrew Morton
	<akpm@...ux-foundation.org>, David Hildenbrand <david@...hat.com>, Oscar
 Salvador <osalvador@...e.de>, "linux-kernel@...r.kernel.org"
	<linux-kernel@...r.kernel.org>, "Yasunori Gotou (Fujitsu)"
	<y-goto@...itsu.com>
Subject: Re: [PATCH RFC] mm: Avoid triggering oom-killer during memory
 hot-remove operations

Michal,

Sorry to the late reply.


On 26/07/2024 17:17, Michal Hocko wrote:
> On Fri 26-07-24 16:44:56, Li Zhijian wrote:
>> When a process is bound to a node that is being hot-removed, any memory
>> allocation attempts from that node should fail gracefully without
>> triggering the OOM-killer. However, the current behavior can cause the
>> oom-killer to be invoked, leading to the termination of processes on other
>> nodes, even when there is sufficient memory available in the system.
> 
> But you said they are bound to the node that is offlined.
>   
>> Prevent the oom-killer from being triggered by processes bound to a
>> node undergoing hot-remove operations. Instead, the allocation attempts
>> from the offlining node will simply fail, allowing the process to handle
>> the failure appropriately without causing disruption to the system.
> 
> NAK.
> 
> Also it is not really clear why process of offlining should behave any
> different from after the node is offlined. Could you describe an actual
> problem you are facing with much more details please?

We encountered that some processes(including some system critical services, for example sshd, rsyslogd, login)
were killed during our memory hot-remove testing. Our test program are described previous mail[1]

In short, we have 3 memory nodes, node0 and node1 are DRAM, while node2 is CXL volatile memory that is onlined
to ZONE_MOVABLE. When we attempted to remove the node2, oom-killed was invoked to kill other processes
(sshd, rsyslogd, login) even though there is enough memory on node0+node1.

This oom-killed was triggered by allocating memory path of our own testing process which was bound to node2.

So I expect,
- our own tes process failed to allocate memory from node2 which is being hot-removed is acceptable.
- oom-killer should not be invoked to kill processes other than running on node2.


[1] https://lore.kernel.org/linux-mm/6a07125f-e720-404c-b2f9-e55f3f166e85@fujitsu.com/


>   
>> Signed-off-by: Li Zhijian <lizhijian@...itsu.com>
>> ---
>>   include/linux/memory_hotplug.h |  6 ++++++
>>   mm/memory_hotplug.c            | 21 +++++++++++++++++++++
>>   mm/page_alloc.c                |  6 ++++++
>>   3 files changed, 33 insertions(+)
>>
>> diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h
>> index 7a9ff464608d..0ca804215e11 100644
>> --- a/include/linux/memory_hotplug.h
>> +++ b/include/linux/memory_hotplug.h
>> @@ -332,6 +332,7 @@ extern int offline_pages(unsigned long start_pfn, unsigned long nr_pages,
>>   extern int remove_memory(u64 start, u64 size);
>>   extern void __remove_memory(u64 start, u64 size);
>>   extern int offline_and_remove_memory(u64 start, u64 size);
>> +bool is_offlining_node(nodemask_t nodes);
>>   
>>   #else
>>   static inline void try_offline_node(int nid) {}
>> @@ -348,6 +349,11 @@ static inline int remove_memory(u64 start, u64 size)
>>   }
>>   
>>   static inline void __remove_memory(u64 start, u64 size) {}
>> +
>> +static inline bool is_offlining_node(nodemask_t nodes)
>> +{
>> +	return false;
>> +}
>>   #endif /* CONFIG_MEMORY_HOTREMOVE */
>>   
>>   #ifdef CONFIG_MEMORY_HOTPLUG
>> diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
>> index 431b1f6753c0..da3982751ba9 100644
>> --- a/mm/memory_hotplug.c
>> +++ b/mm/memory_hotplug.c
>> @@ -1938,6 +1938,22 @@ static int count_system_ram_pages_cb(unsigned long start_pfn,
>>   	return 0;
>>   }
>>   
>> +static nodemask_t offlining_node = NODE_MASK_NONE;
>> +
>> +bool is_offlining_node(nodemask_t nodes)
>> +{
>> +	return nodes_equal(offlining_node, nodes);
>> +}
>> +
>> +static void offline_pages_start(int node)
>> +{
>> +	node_set(node, offlining_node);
>> +}
>> +
>> +static void offline_pages_end(void)
>> +{
>> +	offlining_node = NODE_MASK_NONE;
>> +}
>>   /*
>>    * Must be called with mem_hotplug_lock in write mode.
>>    */
>> @@ -1991,6 +2007,7 @@ int __ref offline_pages(unsigned long start_pfn, unsigned long nr_pages,
>>   		goto failed_removal;
>>   	}
>>   
>> +	offline_pages_start(node);
>>   	/*
>>   	 * Disable pcplists so that page isolation cannot race with freeing
>>   	 * in a way that pages from isolated pageblock are left on pcplists.
>> @@ -2107,6 +2124,8 @@ int __ref offline_pages(unsigned long start_pfn, unsigned long nr_pages,
>>   
>>   	memory_notify(MEM_OFFLINE, &arg);
>>   	remove_pfn_range_from_zone(zone, start_pfn, nr_pages);
>> +	offline_pages_end();
>> +
>>   	return 0;
>>   
>>   failed_removal_isolated:
>> @@ -2121,6 +2140,8 @@ int __ref offline_pages(unsigned long start_pfn, unsigned long nr_pages,
>>   		 (unsigned long long) start_pfn << PAGE_SHIFT,
>>   		 ((unsigned long long) end_pfn << PAGE_SHIFT) - 1,
>>   		 reason);
>> +
>> +	offline_pages_end();
>>   	return ret;
>>   }
>>   
>> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
>> index 1780df31d5f5..acdab6b114a5 100644
>> --- a/mm/page_alloc.c
>> +++ b/mm/page_alloc.c
>> @@ -3563,6 +3563,12 @@ __alloc_pages_may_oom(gfp_t gfp_mask, unsigned int order,
>>   	if (page)
>>   		goto out;
>>   
>> +	/* hot-remove is on-going, it generally fails to allocate memory from
>> +	 * the being removed memory node. Leave it alone.
>> +	 */
>> +	if (is_offlining_node(*ac->nodemask))
>> +		goto out;
>> +
>>   	/* Coredumps can quickly deplete all memory reserves */
>>   	if (current->flags & PF_DUMPCORE)
>>   		goto out;
>> -- 
>> 2.29.2
>>
> 

Powered by blists - more mailing lists