linux-kernel - [PATCH] memcg: no refill for offlined objcg

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <20250410210535.1005312-1-shakeel.butt@linux.dev>
Date: Thu, 10 Apr 2025 14:05:35 -0700
From: Shakeel Butt <shakeel.butt@...ux.dev>
To: Andrew Morton <akpm@...ux-foundation.org>
Cc: Johannes Weiner <hannes@...xchg.org>,
	Michal Hocko <mhocko@...nel.org>,
	Roman Gushchin <roman.gushchin@...ux.dev>,
	Muchun Song <muchun.song@...ux.dev>,
	Yosry Ahmed <yosry.ahmed@...ux.dev>,
	Waiman Long <llong@...hat.com>,
	linux-mm@...ck.org,
	cgroups@...r.kernel.org,
	linux-kernel@...r.kernel.org,
	Meta kernel team <kernel-team@...a.com>
Subject: [PATCH] memcg: no refill for offlined objcg

In our fleet, we are observing refill_obj_stock() spending a lot of cpu
in obj_cgroup_get() and on further inspection it seems like the given
objcg is offlined and the kernel has to take the slow path i.e. atomic
operations for objcg reference counting.

Other than expensive atomic operations, refilling stock of an offlined
objcg is a waster as there will not be new allocations for the offlined
objcg. In addition, refilling triggers flush of the previous objcg which
might be used in future. So, let's just avoid refilling the stock with
the offlined objcg.

Signed-off-by: Shakeel Butt <shakeel.butt@...ux.dev>
---
 mm/memcontrol.c | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 2178a051bd09..23c62ae6a8c6 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -2474,6 +2474,17 @@ static inline void __mod_objcg_mlstate(struct obj_cgroup *objcg,
 	rcu_read_unlock();
 }
 
+static inline void mod_objcg_mlstate(struct obj_cgroup *objcg,
+				     struct pglist_data *pgdat,
+				     enum node_stat_item idx, int nr)
+{
+	unsigned long flags;
+
+	local_irq_save(flags);
+	__mod_objcg_mlstate(objcg, pgdat, idx, nr);
+	local_irq_restore(flags);
+}
+
 static __always_inline
 struct mem_cgroup *mem_cgroup_from_obj_folio(struct folio *folio, void *p)
 {
@@ -2925,6 +2936,13 @@ static void refill_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes,
 	unsigned long flags;
 	unsigned int nr_pages = 0;
 
+	if (unlikely(percpu_ref_is_dying(&objcg->refcnt))) {
+		atomic_add(nr_bytes, &objcg->nr_charged_bytes);
+		if (pgdat)
+			mod_objcg_mlstate(objcg, pgdat, idx, nr_acct);
+		return;
+	}
+
 	local_lock_irqsave(&memcg_stock.stock_lock, flags);
 
 	stock = this_cpu_ptr(&memcg_stock);
-- 
2.47.1