Message-Id: <1363430142-14563-1-git-send-email-tangchen@cn.fujitsu.com>
Date:	Sat, 16 Mar 2013 18:35:33 +0800
From:	Tang Chen <tangchen@...fujitsu.com>
To:	rob@...dley.net, tglx@...utronix.de, mingo@...hat.com,
	hpa@...or.com, yinghai@...nel.org, akpm@...ux-foundation.org,
	wency@...fujitsu.com, trenn@...e.de, liwanp@...ux.vnet.ibm.com,
	mgorman@...e.de, walken@...gle.com, riel@...hat.com,
	khlebnikov@...nvz.org, tj@...nel.org, minchan@...nel.org,
	m.szyprowski@...sung.com, mina86@...a86.com, laijs@...fujitsu.com,
	isimatu.yasuaki@...fujitsu.com, linfeng@...fujitsu.com,
	jiang.liu@...wei.com, kosaki.motohiro@...fujitsu.com,
	guz.fnst@...fujitsu.com
Cc:	x86@...nel.org, linux-doc@...r.kernel.org,
	linux-kernel@...r.kernel.org, linux-mm@...ck.org
Subject: [PATCH v1 part1 0/9] Introduce movablemem_map boot option.

Hi Yinghai, all,

As Yinghai have implemented parsing numa info early more considerately,
I think we can introduce the movablemem_map boot option again.

This patch-set is based on Linux 3.9 rc-2, but need to apply Yinghai's
"x86, ACPI, numa: Parse numa info early" patch-set first.
Please refer to:
v1: https://lkml.org/lkml/2013/3/7/642
v2: https://lkml.org/lkml/2013/3/10/47


In this part1 patch-set, we reimplemented movablemem_map boot option
based on Yinghai's SRAT work. The path is like this:
1) parse SRAT, fill only existing memory into numa_meminfo, like:
   numa_cleanup_meminfo() {
 251         const u64 low = 0;
 252         const u64 high = PFN_PHYS(max_pfn);
......
 255         /* first, trim all entries */
 256         for (i = 0; i < mi->nr_blks; i++) {
 257                 struct numa_memblk *bi = &mi->blk[i];
 258 
 259                 /* make sure all blocks are inside the limits */
 260                 bi->start = max(bi->start, low);
 261                 bi->end = min(bi->end, high);
 262 
 263                 /* and there's no empty block */
 264                 if (bi->start >= bi->end)
 265                         numa_remove_memblk_from(i--, mi);
 266         }
......
   }

   Those non-existing memory, such as memory not added yet, won't be
   stored in numa_meminfo.

2) initialize memory mapping for the existing memory, putting pagetables
   and vmemmap on local node.

Since not all memory info is kept, we have to sanitize movablemem_map.map[]
when we parse SRAT, so we may prevent allocating pagetables or vmemmap on
local node if user specified the whole node as movable.

To avoid this problem, here is my idea:
1) Store not only existing memory ranges in numa_mem_info, but all the
   memory info from SRAT; 
2) Map only existing memory as before;
3) Do memblock limitation after memory mapping initialization using
   numa_meminfo, so that movablemem_map will be able to exclude pagetables
   and vmemmap ranges on local node.

This will be done in part2 soon.

How do you think?

Part2 of this patch-set is under development.

========================================================================
[What we are doing]
This patchset introduces a boot option for user to specify ZONE_MOVABLE
memory map for each node in the system. Users can use it in two ways:

1. movablecore_map=nn[KMG]@ss[KMG]
   In this way, the kernel will make sure memory range from ss to ss+nn is
   on ZONE_MOVABLE. The hotplug info provided by SRAT will be ignored.

2. movablecore_map=acpi
   In this way, the kernel will use memory hotplug info in SRAT to determine
   ZONE_MOVABLE for each node. All the ranges user has specified will be
   ignored.


[Why we do this]
If we hot remove a memroy device, it cannot have kernel memory,
because Linux cannot migrate kernel memory currently. Therefore,
we have to guarantee that the hot removed memory has only movable
memoroy.
(Here is an exception: When we implement the node hotplug functionality,
for those kernel memory whose life cycle is the same as the node, such as
pagetables, vmemmap and so on, although the kernel cannot migrate them,
we can still put them on local node because we can free them before we
hot-remove the node. This is not implemented yet.)

Linux has two boot options, kernelcore= and movablecore=, for
creating movable memory. These boot options can specify the amount
of memory use as kernel or movable memory. Using them, we can
create ZONE_MOVABLE which has only movable memory.
(NOTE: doing this will cause NUMA performance because the kernel won't
 be able to distribute kernel memory evenly to each node.)

But it does not fulfill a requirement of memory hot remove, because
even if we specify the boot options, movable memory is distributed
in each node evenly. So when we want to hot remove memory which
memory range is 0x80000000-0c0000000, we have no way to specify
the memory as movable memory.

Furthermore, even if we can use SRAT, users still need an interface
to enable/disable this functionality if they don't want to lose their
NUMA performance.  So I think, a user interface is always needed.

So we proposed this new feature which specifies memory range to use as
movable memory.


[Ways to do this]
There may be 2 ways to specify movable memory.
1. use firmware information
2. use boot option

1. use firmware information
  According to ACPI spec 5.0, SRAT table has memory affinity structure
  and the structure has Hot Pluggable Filed. See "5.2.16.2 Memory
  Affinity Structure". If we use the information, we might be able to
  specify movable memory by firmware. For example, if Hot Pluggable
  Filed is enabled, Linux sets the memory as movable memory.

2. use boot option
  This is our proposal. New boot option can specify memory range to use
  as movable memory.


[How we do this]
We chose second way, because if we use first way, users cannot change
memory range to use as movable memory easily. We think if we create
movable memory, performance regression may occur by NUMA. In this case,
user can turn off the feature easily if we prepare the boot option.
And if we prepare the boot optino, the user can select which memory
to use as movable memory easily. 


[How to use]
1. For movablecore_map=nn[KMG]@ss[KMG]:
         *
         * SRAT:                |_____| |_____| |_________| |_________| ......
         * node id:                0       1         1           2
         * user specified:                |__|                 |___|
         * ZONE_MOVABLE:                  |___| |_________|    |______| ......
         *
   NOTE: 1) User can specify this option more than once, but at most MAX_NUMNODES
            times. The extra options will be ignored.
         2) In this case, SRAT info will be ingored.

2. For movablemem_map=acpi:
         *
         * SRAT:                |_____| |_____| |_________| |_________| ......
         * node id:                0       1         1           2
         * hotpluggable:           n       y         y           n
         * ZONE_MOVABLE:                |_____| |_________|
         *
   NOTE: 1) Before parsing SRAT, memblock has already reserve some memory ranges
            for other purposes, such as for kernel image. We cannot prevent
            kernel from using these memory, so we need to exclude these memory
            even if it is hotpluggable.
            Furthermore, to ensure the kernel has enough memory to boot, we make
            all the memory on the node which the kernel resides in should be
            un-hotpluggable.
         2) In this case, all the user specified memory ranges will be ingored.

We also need to consider the following points:
1) Using this boot option could cause NUMA performance down because the kernel
   memory will not be distributed on each node evenly. So for users who don't
   want to lose their NUMA performance, just don't use it.
2) If kernelcore or movablecore is also specified, movablecore_map will have
   higher priority to be satisfied.
3) This option has no conflict with memmap option.


Tang Chen (8):
  acpi: Print hotplug info in SRAT.
  x86, mm, numa, acpi: Add movable_memmap boot option.
  x86, mm, numa, acpi: Introduce zone_movable_limit[] to store start
    pfn of ZONE_MOVABLE.
  x86, mm, numa, acpi: Extend movablemem_map to the end of each node.
  x86, mm, numa, acpi: Support getting hotplug info from SRAT.
  x86, mm, numa, acpi: Sanitize zone_movable_limit[].
  x86, mm, numa, acpi: make movablemem_map have higher priority
  x86, mm, numa, acpi: Memblock limit with movablemem_map

Yasuaki Ishimatsu (1):
  x86: get pg_data_t's memory from other node

 Documentation/kernel-parameters.txt |   36 +++++
 arch/x86/mm/numa.c                  |    5 +-
 arch/x86/mm/srat.c                  |  130 +++++++++++++++++-
 include/linux/memblock.h            |    2 +
 include/linux/mm.h                  |   22 +++
 mm/memblock.c                       |   50 +++++++
 mm/page_alloc.c                     |  265 ++++++++++++++++++++++++++++++++++-
 7 files changed, 500 insertions(+), 10 deletions(-)

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Powered by blists - more mailing lists