Message-ID: <CAAPL-u_ZtCsuNNu2SoqCeqQqrGQxjcsjrbu0ooP3y5Zw802daA@mail.gmail.com>
Date:   Tue, 24 May 2022 22:27:39 -0700
From:   Wei Xu <weixugc@...gle.com>
To:     "Aneesh Kumar K.V" <aneesh.kumar@...ux.ibm.com>
Cc:     Jonathan Cameron <Jonathan.Cameron@...wei.com>,
        Dave Hansen <dave.hansen@...el.com>,
        Alistair Popple <apopple@...dia.com>,
        Huang Ying <ying.huang@...el.com>,
        Andrew Morton <akpm@...ux-foundation.org>,
        Greg Thelen <gthelen@...gle.com>,
        Yang Shi <shy828301@...il.com>,
        Linux Kernel Mailing List <linux-kernel@...r.kernel.org>,
        Jagdish Gediya <jvgediya@...ux.ibm.com>,
        Michal Hocko <mhocko@...nel.org>,
        Tim C Chen <tim.c.chen@...el.com>,
        Baolin Wang <baolin.wang@...ux.alibaba.com>,
        Feng Tang <feng.tang@...el.com>,
        Davidlohr Bueso <dave@...olabs.net>,
        Dan Williams <dan.j.williams@...el.com>,
        David Rientjes <rientjes@...gle.com>,
        Linux MM <linux-mm@...ck.org>,
        Brice Goglin <brice.goglin@...il.com>,
        Hesham Almatary <hesham.almatary@...wei.com>
Subject: Re: RFC: Memory Tiering Kernel Interfaces (v2)

On Tue, May 24, 2022 at 6:27 AM Aneesh Kumar K.V
<aneesh.kumar@...ux.ibm.com> wrote:
>
> Wei Xu <weixugc@...gle.com> writes:
>
> > On Wed, May 18, 2022 at 5:00 AM Jonathan Cameron
> > <Jonathan.Cameron@...wei.com> wrote:
> >>
> >> On Wed, 18 May 2022 00:09:48 -0700
> >> Wei Xu <weixugc@...gle.com> wrote:
>
> ...
>
> > Nice :)
> >>
> >> Initially I thought this was over complicated when compared to just leaving space, but
> >> after a chat with Hesham just now you have us both convinced that this is an elegant solution.
> >>
> >> Few corners probably need fleshing out:
> >> *  Use of an allocator for new tiers. Flat number at startup, or new one on write of unique
> >>    value to set_memtier perhaps?  Also whether to allow drivers to allocate (I think
> >>    we should).
> >> *  Multiple tiers with same rank.  My assumption is from demotion path point of view you
> >>    fuse them (treat them as if they were a single tier), but keep them expressed
> >>    separately in the sysfs interface so that the rank can be changed independently.
> >> *  Some guidance on what values make sense for given rank default that might be set by
> >>    a driver. If we have multiple GPU vendors, and someone mixes them in a system we
> >>    probably don't want the default values they use to result in demotion between them.
> >>    This might well be a guidance DOC or appropriate set of #define
> >
> > All of these are good ideas, though I am afraid that these can make
> > tier management too complex for what it's worth.
> >
> > How about an alternative tier numbering scheme that uses major.minor
> > device IDs?  For simplicity, we can just start with 3 major tiers.
> > New tiers can be inserted in-between using minor tier IDs.
>
>
> What drives the creation of a new memory tier here?  Jonathan was
> suggesting we could do something similar to writing to set_memtier for
> creating a new memory tier.
>
> $ echo "memtier128" > sys/devices/system/node/node1/set_memtier
>
> But I am wondering whether we should implement that now. If we keep
> "rank" concept and detach tier index (memtier0 is the memory tier with
> index 0) separate from rank, I assume we have enough flexibility for a
> future extension that will allow us to create a memory tier from userspace
> and assigning it a rank value that helps the device to be placed before or
> after DRAM in demotion order.
>
> ie, For now we will only have memtier0, memtier1, memtier2. We won't add
> dynamic creation of memory tiers and the above memory tiers will have
> rank value 0, 1, 2 according with demotion order 0 -> 1 -> 2.

Great. So the consensus is to go with the "rank" approach.  The above
sounds good to me as a starting point.

> -aneesh

Powered by blists - more mailing lists