Message-ID: <5146E521.1040708@cn.fujitsu.com>
Date:	Mon, 18 Mar 2013 17:57:53 +0800
From:	Tang Chen <tangchen@...fujitsu.com>
To:	Will Huck <will.huckk@...il.com>
CC:	rob@...dley.net, tglx@...utronix.de, mingo@...hat.com,
	hpa@...or.com, yinghai@...nel.org, akpm@...ux-foundation.org,
	wency@...fujitsu.com, trenn@...e.de, liwanp@...ux.vnet.ibm.com,
	mgorman@...e.de, walken@...gle.com, riel@...hat.com,
	khlebnikov@...nvz.org, tj@...nel.org, minchan@...nel.org,
	m.szyprowski@...sung.com, mina86@...a86.com, laijs@...fujitsu.com,
	isimatu.yasuaki@...fujitsu.com, linfeng@...fujitsu.com,
	jiang.liu@...wei.com, kosaki.motohiro@...fujitsu.com,
	guz.fnst@...fujitsu.com, x86@...nel.org, linux-doc@...r.kernel.org,
	linux-kernel@...r.kernel.org, linux-mm@...ck.org
Subject: Re: [PATCH v1 part1 0/9] Introduce movablemem_map boot option.

Hi Will,

On 03/17/2013 08:25 AM, Will Huck wrote:
>
> http://marc.info/?l=linux-mm&m=136014458829566&w=2
>
> It seems that Mel don't like this idea.
>

Thank you for reminding me this.

And yes, I have read that email. :)

And about this boot option, we have had a long discussion before.
Please refer to: https://lkml.org/lkml/2012/11/29/190

The situation is:

For now, Linux kernel cannot migrate kernel direct mapping memory. And
there is no way to ensure that ZONE_NORMAL has no kernel memory. So we
can only use ZONE_MOVABLE to ensure the memory device could be removed.

For now, I have the following reasons that movablemem_map boot option is
necessary. Some may be mentioned before, but here, I think I need to say
them again:

1) If we want to hot-remove a memory device, the device should only have
    memory of two types:
    - kernel memory whose life cycle is the same as the memory device.
      such as pagetables, vmemmap
    - user memory that could be migrated.

    For type1: we can allocate it on local node, just like Yinghai's work,
               and free it when hot-removing.
    For type2: we can migrate it at run time. But it must be in ZONE_MOVABLE
               because we cannot ensure ZONE_NORMAL has no kernel memory.

    So we need a way to limit hotpluggable memory in ZONE_MOVABLE.

2) We have the following ways to do it:
    a) use SRAT, which I have already implemented
    b) specify physical address ranges, which I have implemented too, but
       obviously very few guys like it.
    c) specify node id. But nid could be changed on some platform by 
firmware.

    Because of c), we chose to use physical address ranges. To satisfy all
    users, I also implemented a).

3) Even if we don't specify physical address in command line, we use SRAT,
    we still need the logic in this patch-set to achieve the same goal.

4) Since setting a whole node as movable will cause NUMA performance down,
    no matter which way we use, we always need an interface to open or close
    this functionality.
    The boot option itself is an interface. If users don't specify it in
    command line, the kernel will work as before.

So I do want to try again to push this boot option.  :)

With this boot option, memory hotplug will work now.


It's true that if we reimplement the whole mm in Linux to make kernel
memory migratable, but we need to handle a lot of problems. I agree with 
Mel.
But it is a long way to go in the future.

And the work in the near future:
1) Allocate pagetables and vmemmap on local node, as Yinghai said.
2) Do the proper modification for hot-add and hot-remove.
    - Reserve memory for pagetables and vmemmap when hot-add, maybe use
      memblock.
    - Free all pagetables and vmemmap before hot-remove.
3) And about Mel's advice, modify memory management in Linux to migrate
    kernel pages, it is a long way to go in the future. I think we can
    discuss more.

Thanks. :)


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Powered by blists - more mailing lists