Message-ID: <20250411221111.493193-1-gourry@gourry.net>
Date: Fri, 11 Apr 2025 18:11:05 -0400
From: Gregory Price <gourry@...rry.net>
To: linux-mm@...ck.org
Cc: cgroups@...r.kernel.org,
	linux-kernel@...r.kernel.org,
	kernel-team@...a.com,
	akpm@...ux-foundation.org,
	mingo@...hat.com,
	peterz@...radead.org,
	juri.lelli@...hat.com,
	vincent.guittot@...aro.org,
	hannes@...xchg.org,
	mhocko@...nel.org,
	roman.gushchin@...ux.dev,
	shakeel.butt@...ux.dev,
	donettom@...ux.ibm.com,
	Huang Ying <ying.huang@...ux.alibaba.com>,
	Keith Busch <kbusch@...a.com>,
	Feng Tang <feng.tang@...el.com>,
	Neha Gholkar <nehagholkar@...a.com>
Subject: [RFC PATCH v4 0/6] Promotion of Unmapped Page Cache Folios.

Unmapped page cache pages can be demoted to low-tier memory, but
they can presently only be promoted in two conditions:
    1) The page is fully swapped out and re-faulted
    2) The page becomes mapped (and exposed to NUMA hint faults)

This RFC proposes promoting unmapped page cache pages by using
folio_mark_accessed as a hotness hint for unmapped pages.

We show in a microbenchmark that this mechanism can increase
performance up to 23.5% compared to leaving page cache on the
low tier - when that page cache becomes excessively hot.

When disabled (NUMA tiering off), overhead in folio_mark_accessed
was limited to <1% in a worst case scenario (all work is file_read()).

Patches 1-2
	allow NULL as valid input to migration prep interfaces
	for vmf/vma - which is not present in unmapped folios.
Patch 3
	adds NUMA_HINT_PAGE_CACHE to vmstat
Patch 4
	Implement migrate_misplaced_folio_batch
Patch 5
	add the promotion mechanism, along with a sysfs
	extension which defaults the behavior to off.
	/sys/kernel/mm/numa/pagecache_promotion_enabled
Patch 6
	add MGLRU implementation by Donet Tom

v4 Notes
===
- Add MGLRU implementation
- dropped ifdef change patch after build testing
- Worst-case scenario analysis (thrashing)
- FIO Test analysis

Test Environment
================
    1.5-3.7GHz CPU, ~4000 BogoMIPS, 
    1TB Machine with 768GB DRAM and 256GB CXL

FIO Test:
=========
We evaluated this with FIO with the page-cache pre-loaded

Step 1: Load 128GB file into page cache with a mempolicy
        (dram, cxl, and cxl to promote)
Step 2: Run FIO with 4 reading jobs
        Config does not invalidate pagecache between runs
Step 3: Repeat with a 900GB file that spills into CXL and
        creates thrashing to show its impact.

Configuration:
[global]
bs=1M
size=128G  # 900G
time_based=1
numjobs=4
rw=randread,randwrite
filename=test.data
direct=0
invalidate=0 # Do not drop the cache between runs
[test1]
ioengine=psync
iodepth=64
runtime=240s # Also did 480s, didn't show much difference

On promotion runs, vmstat reported the entire file is promoted
	numa_pages_migrated 33554772   (128.01GB)

DRAM (128GB):
  lat (usec)   : 50=98.34%, 100=1.61%, 250=0.05%, 500=0.01%
  cpu          : usr=1.42%, sys=98.35%, ctx=213, majf=0, minf=264
  bw=83.5GiB/s (89.7GB/s)

Remote (128GB)
  lat (usec)   : 100=91.78%, 250=8.22%, 500=0.01%
  cpu          : usr=0.66%, sys=99.13%, ctx=449, majf=0, minf=263
  bw=41.4GiB/s (44.4GB/s)

Promo (128GB)
  lat (usec)   : 50=88.02%, 100=10.65%, 250=1.05%, 500=0.20%, 750=0.06%
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=0.01%
  cpu          : usr=1.44%, sys=97.72%, ctx=1679, majf=0, minf=265
  bw=79.2GiB/s (85.0GB/s)

900GB Hot - No Promotion (~150GB spills to remote node via demotion)
  lat (usec)   : 50=69.26%, 100=13.79%, 250=16.95%, 500=0.01%
  bw=64.5GiB/s (69.3GB/s)

900GB Hot - Promotion (Causes thrashing between demotion/promotion)
  lat (usec)   : 50=47.79%, 100=29.59%, 250=21.16%, 500=1.24%, 750=0.04%
  lat (usec)   : 1000=0.03%
  lat (msec)   : 2=0.15%, 4=0.01%
  bw=47.6GiB/s (51.1GB/s)

900GB Hot - No remote memory (Fault-in/page-out of read-only data)
  lat (usec)   : 50=39.73%, 100=31.34%, 250=4.71%, 500=0.01%, 750=0.01%
  lat (usec)   : 1000=1.78%
  lat (msec)   : 2=21.63%, 4=0.81%, 10=0.01%, 20=0.01%
  bw=10.2GiB/s (10.9GB/s)

Obviously some portion of the overhead comes from migration, but the
results here are pretty dramatic.  We can regain nearly all of the
performance in a degenerate scenario (demoted page cache becomes hot)
by turning on promotion - even temporarily.

In the scenario where the entire workload is hot, turning on promotion
causes thrashing, and we hurt performance.

So this feature is useful in one of two scenarios:
1) Headroom on DRAM is available and we opportunistically move page
   cache to the higher tier as it's accessed, or
2) A lower performance node becomes un-evenly pressured.
   It doesn't help us if the higher node is pressured.

For example, it would be nice for a userland daemon to detect DRAM-tier
memory becomes available, and to flip the bit to migrate any hotter page
cache up a level until DRAM becomes pressured again. Cold pagecache
stays put and new allocations still occur on the fast tier.


Worst Case Scenario Test (Micro-benchmark)
==========================================

Goal:
   Generate promotions and demonstrate upper-bound on performance
   overhead and gain/loss.

System Settings:
   CXL Memory in ZONE_MOVABLE (no fallback alloc, demotion-only use)
   echo 2 > /proc/sys/kernel/numa_balancing
   echo 1 > /sys/kernel/mm/numa/pagecache_promotion_enabled
   echo 1 > /sys/kernel/mm/numa/demotion_enabled
   
Test process:
   In each test, we do a linear read of a 128GB file into a buffer
   in a loop.  To allocate the pagecache into CXL, we use mbind prior
   to the CXL test runs and read the file.  We omit the overhead of
   allocating the buffer and initializing the memory into CXL from the
   test runs.

   1) file allocated in DRAM with mechanisms off
   2) file allocated in DRAM with balancing on but promotion off
   3) file allocated in DRAM with balancing and promotion on
      (promotion check is negative because all pages are top tier)
   4) file allocated in CXL with mechanisms off
   5) file allocated in CXL with mechanisms on

Each test was run with 50 read cycles and averaged (where relevant)
to account for system noise.  This number of cycles gives the promotion
mechanism time to promote the vast majority of memory (usually <1MB
remaining in worst case).

Tests 2 and 3 test the upper bound on overhead of the new checks when
there are no pages to migrate but work is dominated by file_read().

|     1     |    2     |     3       |    4     |      5         |
| DRAM Base | Promo On | TopTier Chk | CXL Base | Post-Promotion |
|  7.5804   |  7.7586  |   7.9726    |   9.75   |    7.8941      |

Baseline DRAM vs Baseline CXL shows a ~28% overhead just allowing the
file to remain on CXL, while after promotion, we see the performance
trend back towards the overhead of the TopTier check time - a total
overhead reduction of ~84% (or ~5% overhead down from ~23.5%).

During promotion, we do see overhead which eventually tapers off over
time.  Here is a sample of the first 10 cycles during which promotion
is the most aggressive, which shows overhead drops off dramatically
as the majority of memory is migrated to the top tier.

12.79, 12.52, 12.33, 12.03, 11.81, 11.58, 11.36, 11.1, 8, 7.96

After promotion, turning the mechanism off via sysfs increased the
overall performance back to the DRAM baseline. The slight (~1%)
increase between post-migration performance and the baseline mechanism
overhead check appears to be general variance as similar times were
observed during the baseline checks on subsequent runs.

The mechanism itself represents a ~2.5% overhead in a worst case
scenario (all work is file_read(), all pages are in DRAM, all pages are
hot - which is highly unrealistic). This is inclusive of any overhead 

Development History and Notes
=======================================
During development, we explored the following proposals:

1) directly promoting within folio_mark_accessed (FMA)
   Originally suggested by Johannes Weiner
   https://lore.kernel.org/all/20240803094715.23900-1-gourry@gourry.net/

   This caused deadlocks due to the fact that the PTL was held
   in a variety of cases - but in particular during task exit.
   It also is incredibly inflexible and causes promotion-on-fault.
   It was discussed that a deferral mechanism was preferred.


2) promoting in filemap.c locations (callers of FMA)
   Originally proposed by Feng Tang and Ying Huang
   https://git.kernel.org/pub/scm/linux/kernel/git/vishal/tiering.git/patch/?id=5f2e64ce75c0322602c2ec8c70b64bb69b1f1329

   First, we saw this as less problematic than directly hooking FMA,
   but we realized this has the potential to miss data in a variety of
   locations: swap.c, memory.c, gup.c, ksm.c, paddr.c - etc.

   Second, we discovered that the lock state of pages is very subtle,
   and that these locations in filemap.c can be called in an atomic
   context.  Prototypes lead to a variety of stalls and lockups.


3) a new LRU - originally proposed by Keith Busch
   https://git.kernel.org/pub/scm/linux/kernel/git/kbusch/linux.git/patch/?id=6616afe9a722f6ebedbb27ade3848cf07b9a3af7

   There are two issues with this approach: PG_promotable and reclaim.

   First - PG_promotable has generally be discouraged.

   Second - Attach this mechanism to an LRU is both backwards and
   counter-intutive.  A promotable list is better served by a MOST
   recently used list, and since LRUs are generally only shrank when
   exposed to pressure it would require implementing a new promotion
   list shrinker that runs separate from the existing reclaim logic.


4) Adding a separate kthread - suggested by many

   This is - to an extent - a more general version of the LRU proposal.
   We still have to track the folios - which likely requires the
   addition of a page flag.  Additionally, this method would actually
   contend pretty heavily with LRU behavior - i.e. we'd want to
   throttle addition to the promotion candidate list in some scenarios.


5) Doing it in task work

   This seemed to be the most realistic after considering the above.

   We observe the following:
    - FMA is an ideal hook for this and isolation is safe here
    - the new promotion_candidate function is an ideal hook for new
      filter logic (throttling, fairness, etc).
    - isolated folios are either promoted or putback on task resume,
      there are no additional concurrency mechanics to worry about
    - The mechanic can be made optional via a sysfs hook to avoid
      overhead in degenerate scenarios (thrashing).


Suggested-by: Huang Ying <ying.huang@...ux.alibaba.com>
Suggested-by: Johannes Weiner <hannes@...xchg.org>
Suggested-by: Keith Busch <kbusch@...a.com>
Suggested-by: Feng Tang <feng.tang@...el.com>
Tested-by: Neha Gholkar <nehagholkar@...a.com>
Signed-off-by: Gregory Price <gourry@...rry.net>
Co-developed-by: Donet Tom <donettom@...ux.ibm.com>

Donet Tom (1):
  mm/swap.c: Enable promotion of unmapped MGLRU page cache pages

Gregory Price (5):
  migrate: Allow migrate_misplaced_folio_prepare() to accept a NULL VMA.
  memory: allow non-fault migration in numa_migrate_check path
  vmstat: add page-cache numa hints
  migrate: implement migrate_misplaced_folio_batch
  migrate,sysfs: add pagecache promotion

 .../ABI/testing/sysfs-kernel-mm-numa          | 20 +++++
 include/linux/memory-tiers.h                  |  2 +
 include/linux/migrate.h                       | 11 +++
 include/linux/sched.h                         |  4 +
 include/linux/sched/sysctl.h                  |  1 +
 include/linux/vm_event_item.h                 |  8 ++
 init/init_task.c                              |  2 +
 kernel/sched/fair.c                           | 24 ++++-
 mm/memcontrol.c                               |  1 +
 mm/memory-tiers.c                             | 27 ++++++
 mm/memory.c                                   | 30 ++++---
 mm/mempolicy.c                                | 25 ++++--
 mm/migrate.c                                  | 88 ++++++++++++++++++-
 mm/swap.c                                     | 15 +++-
 mm/vmstat.c                                   |  2 +
 15 files changed, 236 insertions(+), 24 deletions(-)

-- 
2.49.0

Powered by blists - more mailing lists