linux-kernel - [PATCH] mm-unstable: Multi-gen LRU: Fix workingset accounting

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <20230523205922.3852731-1-kaleshsingh@google.com>
Date:   Tue, 23 May 2023 13:59:21 -0700
From:   Kalesh Singh <kaleshsingh@...gle.com>
To:     yuzhao@...gle.com, akpm@...ux-foundation.org
Cc:     surenb@...gle.com, android-mm@...gle.com, kernel-team@...roid.com,
        Kalesh Singh <kaleshsingh@...gle.com>,
        Charan Teja Kalla <quic_charante@...cinc.com>,
        Oleksandr Natalenko <oleksandr@...alenko.name>,
        "Jan Alexander Steffens (heftig)" <heftig@...hlinux.org>,
        Brian Geffon <bgeffon@...gle.com>, linux-mm@...ck.org,
        linux-kernel@...r.kernel.org
Subject: [PATCH] mm-unstable: Multi-gen LRU: Fix workingset accounting

On Android app cycle workloads, MGLRU showed a significant reduction
in workingset refaults although pgpgin/pswpin remained relatively
unchanged. This indicated MGLRU may be undercounting workingset
refaults.

This has impact on userspace programs, like Android's LMKD, that
monitor workingset refault statistics to detect thrashing.

It was found that refaults were only accounted if the MGLRU shadow
entry was for a recently evicted folio. However, recently evicted
folios should be accounted as workingset activation, and refaults
should be accounted regardless of recency.

Fix MGLRU's workingset refault and activation accounting to more
closely match that of the conventional active/inactive LRU.

Fixes: ac35a4902374 ("mm: multi-gen LRU: minimal implementation")
Cc: Yu Zhao <yuzhao@...gle.com>
Cc: Andrew Morton <akpm@...ux-foundation.org> 
Reported-by: Charan Teja Kalla <quic_charante@...cinc.com>
Signed-off-by: Kalesh Singh <kaleshsingh@...gle.com>
---
 mm/vmscan.c     |  1 -
 mm/workingset.c | 10 +++++++---
 2 files changed, 7 insertions(+), 4 deletions(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index eeca83e28c9b..ef687f9be13c 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -4925,7 +4925,6 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, int tier_idx)
 
 		WRITE_ONCE(lrugen->protected[hist][type][tier - 1],
 			   lrugen->protected[hist][type][tier - 1] + delta);
-		__mod_lruvec_state(lruvec, WORKINGSET_ACTIVATE_BASE + type, delta);
 		return true;
 	}
 
diff --git a/mm/workingset.c b/mm/workingset.c
index 5796e927e6d7..4686ae363000 100644
--- a/mm/workingset.c
+++ b/mm/workingset.c
@@ -278,6 +278,7 @@ static bool lru_gen_test_recent(void *shadow, bool file, struct lruvec **lruvec,
 
 static void lru_gen_refault(struct folio *folio, void *shadow)
 {
+	bool recent;
 	int hist, tier, refs;
 	bool workingset;
 	unsigned long token;
@@ -288,10 +289,13 @@ static void lru_gen_refault(struct folio *folio, void *shadow)
 
 	rcu_read_lock();
 
-	if (!lru_gen_test_recent(shadow, type, &lruvec, &token, &workingset))
+	recent = lru_gen_test_recent(shadow, type, &lruvec, &token, &workingset);
+	if (lruvec != folio_lruvec(folio))
 		goto unlock;
 
-	if (lruvec != folio_lruvec(folio))
+	mod_lruvec_state(lruvec, WORKINGSET_REFAULT_BASE + type, delta);
+
+	if (!recent)
 		goto unlock;
 
 	lrugen = &lruvec->lrugen;
@@ -302,7 +306,7 @@ static void lru_gen_refault(struct folio *folio, void *shadow)
 	tier = lru_tier_from_refs(refs);
 
 	atomic_long_add(delta, &lrugen->refaulted[hist][type][tier]);
-	mod_lruvec_state(lruvec, WORKINGSET_REFAULT_BASE + type, delta);
+	mod_lruvec_state(lruvec, WORKINGSET_ACTIVATE_BASE + type, delta);
 
 	/*
 	 * Count the following two cases as stalls:

base-commit: cd255dae35f51ea33cb85d29720d6b4e5fa16365
-- 
2.40.1.698.g37aff9b760-goog