Date:   Sun, 24 Apr 2022 11:02:47 +0800
From:   "ying.huang@...el.com" <ying.huang@...el.com>
To:     Jagdish Gediya <jvgediya@...ux.ibm.com>,
        Wei Xu <weixugc@...gle.com>, Yang Shi <shy828301@...il.com>,
        Dave Hansen <dave.hansen@...ux.intel.com>,
        Dan Williams <dan.j.williams@...el.com>,
        Davidlohr Bueso <dave@...olabs.net>
Cc:     Linux MM <linux-mm@...ck.org>,
        Linux Kernel Mailing List <linux-kernel@...r.kernel.org>,
        Andrew Morton <akpm@...ux-foundation.org>,
        "Aneesh Kumar K.V" <aneesh.kumar@...ux.ibm.com>,
        Baolin Wang <baolin.wang@...ux.alibaba.com>,
        Greg Thelen <gthelen@...gle.com>,
        MichalHocko <mhocko@...nel.org>,
        Brice Goglin <brice.goglin@...il.com>
Subject: Re: [PATCH v2 0/5] mm: demotion: Introduce new node state
 N_DEMOTION_TARGETS

Hi, All,

On Fri, 2022-04-22 at 16:30 +0530, Jagdish Gediya wrote:

[snip]

> I think it is necessary to either have per node demotion targets
> configuration or the user space interface supported by this patch
> series. As we don't have clear consensus on how the user interface
> should look like, we can defer the per node demotion target set
> interface to future until the real need arises.
> 
> Current patch series sets N_DEMOTION_TARGET from dax device kmem
> driver, it may be possible that some memory node desired as demotion
> target is not detected in the system from dax-device kmem probe path.
> 
> It is also possible that some of the dax-devices are not preferred as
> demotion target e.g. HBM, for such devices, node shouldn't be set to
> N_DEMOTION_TARGETS. In future, Support should be added to distinguish
> such dax-devices and not mark them as N_DEMOTION_TARGETS from the
> kernel, but for now this user space interface will be useful to avoid
> such devices as demotion targets.
> 
> We can add read only interface to view per node demotion targets
> from /sys/devices/system/node/nodeX/demotion_targets, remove
> duplicated /sys/kernel/mm/numa/demotion_target interface and instead
> make /sys/devices/system/node/demotion_targets writable.
> 
> Huang, Wei, Yang,
> What do you suggest?

We cannot remove a kernel ABI in practice.  So we need to make it right
at the first time.  Let's try to collect some information for the kernel
ABI definitation.

The below is just a starting point, please add your requirements.

1. Jagdish has some machines with DRAM only NUMA nodes, but they don't
want to use that as the demotion targets.  But I don't think this is a
issue in practice for now, because demote-in-reclaim is disabled by
default.

2. For machines with PMEM installed in only 1 of 2 sockets, for example,

Node 0 & 2 are cpu + dram nodes and node 1 are slow
memory node near node 0,

available: 3 nodes (0-2)
node 0 cpus: 0 1
node 0 size: n MB
node 0 free: n MB
node 1 cpus:
node 1 size: n MB
node 1 free: n MB
node 2 cpus: 2 3
node 2 size: n MB
node 2 free: n MB
node distances:
node   0   1   2
  0:  10  40  20
  1:  40  10  80
  2:  20  80  10

We have 2 choices,

a)
node	demotion targets
0	1
2	1

b)
node	demotion targets
0	1
2	X

a) is good to take advantage of PMEM.  b) is good to reduce cross-socket
traffic.  Both are OK as defualt configuration.  But some users may
prefer the other one.  So we need a user space ABI to override the
default configuration.

3. For machines with HBM (High Bandwidth Memory), as in

https://lore.kernel.org/lkml/39cbe02a-d309-443d-54c9-678a0799342d@gmail.com/

> [1] local DDR = 10, remote DDR = 20, local HBM = 31, remote HBM = 41

Although HBM has better performance than DDR, in ACPI SLIT, their
distance to CPU is longer.  We need to provide a way to fix this.  The
user space ABI is one way.  The desired result will be to use local DDR
as demotion targets of local HBM.

Best Regards,
Huang, Ying

Powered by blists - more mailing lists