Message-ID: <20250306054532.221138-1-bharata@amd.com>
Date: Thu, 6 Mar 2025 11:15:28 +0530
From: Bharata B Rao <bharata@....com>
To: <linux-kernel@...r.kernel.org>, <linux-mm@...ck.org>
CC: <AneeshKumar.KizhakeVeetil@....com>, <Hasan.Maruf@....com>,
	<Jonathan.Cameron@...wei.com>, <Michael.Day@....com>,
	<akpm@...ux-foundation.org>, <dave.hansen@...el.com>, <david@...hat.com>,
	<feng.tang@...el.com>, <gourry@...rry.net>, <hannes@...xchg.org>,
	<honggyu.kim@...com>, <hughd@...gle.com>, <jhubbard@...dia.com>,
	<k.shutemov@...il.com>, <kbusch@...a.com>, <kmanaouil.dev@...il.com>,
	<leesuyeon0506@...il.com>, <leillc@...gle.com>, <liam.howlett@...cle.com>,
	<mgorman@...hsingularity.net>, <mingo@...hat.com>, <nadav.amit@...il.com>,
	<nphamcs@...il.com>, <peterz@...radead.org>, <raghavendra.kt@....com>,
	<riel@...riel.com>, <rientjes@...gle.com>, <rppt@...nel.org>,
	<shivankg@....com>, <shy828301@...il.com>, <sj@...nel.org>, <vbabka@...e.cz>,
	<weixugc@...gle.com>, <willy@...radead.org>, <ying.huang@...ux.alibaba.com>,
	<ziy@...dia.com>, <dave@...olabs.net>, <yuanchu@...gle.com>,
	<hyeonggon.yoo@...com>, Bharata B Rao <bharata@....com>
Subject: [RFC PATCH 0/4] Kernel daemon for detecting and promoting hot pages

Hi,

This is an attempt towards having a single subsystem that accumulates
hot page information from lower memory tiers and does hot page
promotion.

At the heart of this subsystem is a kernel daemon named kpromoted that
does the following:

1. Exposes an API that other subsystems which detect/generate memory
   access information can use to inform the daemon about memory
   accesses from lower memory tiers.
2. Maintains the list of hot pages and attempts to promote them to
   toptiers.

Currently I have added AMD IBS driver as one source that provides
page access information as an example. This driver feeds info to
kpromoted in this RFC patchset. More sources were discussed in a
similar context here at [1].

This is just an early attempt to check what it takes to maintain
a single source of page hotness info and also separate hot page
detection mechanisms from the promotion mechanism. There are too
many open ends right now and I have listed a few of them below.

- The API that is provided to register memory access expects
  the PFN, NID and time of access at the minimum. This is
  described more in patch 2/4. This API currently can be called
  only from contexts that allow sleeping and hence this rules
  out using it from PTE scanning paths. The API needs to be
  more flexible with respect to this.
- Some sources like PTE A bit scanning can't provide the precise
  time of access or the NID that is accessing the page. The latter
  has been an open problem to which I haven't come across a good
  and acceptable solution.
- The way the hot page information is maintained is pretty
  primitive right now. Ideally we would like to store hotness info
  in such a way that it should be easily possible to lookup say N
  most hot pages.
- If PTE A bit scanners are considered as hotness sources, we will
  be bombarded with accesses. Do we want to accomodate all those
  accesses or just go with hotness info for fixed number of pages
  (possibly as a ratio of lower tier memory capacity)?
- Undoubtedly the mechanism to classify a page as hot and subsequent
  promotion needs to be more sophisticated than what I have right now.

This is just an early RFC posted now to ignite some discussion
in the context of LSFMM [2].

I am also working with Raghu to integrate his kmmdscan [3] as the
hotness source and use kpromoted for migration.

Also, I had posted the IBS driver ealier as an alternative to
hint faults based NUMA Balancing [4]. However here I am using
it as generic page hotness source.

[1] https://lore.kernel.org/linux-mm/de31971e-98fc-4baf-8f4f-09d153902e2e@amd.com/
[2] https://lore.kernel.org/linux-mm/20250123105721.424117-1-raghavendra.kt@amd.com/
[3] https://lore.kernel.org/all/20241201153818.2633616-1-raghavendra.kt@amd.com/
[3] https://lore.kernel.org/lkml/20230208073533.715-2-bharata@amd.com/

Regards,
Bharata.

Bharata B Rao (4):
  mm: migrate: Allow misplaced migration without VMA too
  mm: kpromoted: Hot page info collection and promotion daemon
  x86: ibs: In-kernel IBS driver for memory access profiling
  x86: ibs: Enable IBS profiling for memory accesses

 arch/x86/events/amd/ibs.c           |  11 +
 arch/x86/include/asm/entry-common.h |   3 +
 arch/x86/include/asm/hardirq.h      |   2 +
 arch/x86/include/asm/ibs.h          |   9 +
 arch/x86/include/asm/msr-index.h    |  16 ++
 arch/x86/mm/Makefile                |   3 +-
 arch/x86/mm/ibs.c                   | 344 ++++++++++++++++++++++++++++
 include/linux/kpromoted.h           |  54 +++++
 include/linux/mmzone.h              |   4 +
 include/linux/vm_event_item.h       |  30 +++
 mm/Kconfig                          |   7 +
 mm/Makefile                         |   1 +
 mm/kpromoted.c                      | 305 ++++++++++++++++++++++++
 mm/migrate.c                        |   5 +-
 mm/mm_init.c                        |  10 +
 mm/vmstat.c                         |  30 +++
 16 files changed, 831 insertions(+), 3 deletions(-)
 create mode 100644 arch/x86/include/asm/ibs.h
 create mode 100644 arch/x86/mm/ibs.c
 create mode 100644 include/linux/kpromoted.h
 create mode 100644 mm/kpromoted.c

-- 
2.34.1

Powered by blists - more mailing lists