Message-ID: <f4uoxrjr4xer3w4mgwpxypdfdopynwqi4mwc6yskvotbd4ty2f@y4bqddqxoamw>
Date: Wed, 30 Apr 2025 08:05:22 -0700
From: Shakeel Butt <shakeel.butt@...ux.dev>
To: Vlastimil Babka <vbabka@...e.cz>
Cc: Andrew Morton <akpm@...ux-foundation.org>, 
	Johannes Weiner <hannes@...xchg.org>, Michal Hocko <mhocko@...nel.org>, 
	Roman Gushchin <roman.gushchin@...ux.dev>, Muchun Song <muchun.song@...ux.dev>, 
	Jakub Kicinski <kuba@...nel.org>, Eric Dumazet <edumazet@...gle.com>, 
	Soheil Hassas Yeganeh <soheil@...gle.com>, linux-mm@...ck.org, cgroups@...r.kernel.org, 
	netdev@...r.kernel.org, linux-kernel@...r.kernel.org, 
	Meta kernel team <kernel-team@...a.com>
Subject: Re: [PATCH] memcg: multi-memcg percpu charge cache

On Wed, Apr 30, 2025 at 11:57:13AM +0200, Vlastimil Babka wrote:
> On 4/16/25 20:02, Shakeel Butt wrote:
> > Memory cgroup accounting is expensive and to reduce the cost, the kernel
> > maintains per-cpu charge cache for a single memcg. So, if a charge
> > request comes for a different memcg, the kernel will flush the old
> > memcg's charge cache and then charge the newer memcg a fixed amount (64
> > pages), subtracts the charge request amount and stores the remaining in
> > the per-cpu charge cache for the newer memcg.
> > 
> > This mechanism is based on the assumption that the kernel, for locality,
> > keep a process on a CPU for long period of time and most of the charge
> > requests from that process will be served by that CPU's local charge
> > cache.
> > 
> > However this assumption breaks down for incoming network traffic in a
> > multi-tenant machine. We are in the process of running multiple
> > workloads on a single machine and if such workloads are network heavy,
> > we are seeing very high network memory accounting cost. We have observed
> > multiple CPUs spending almost 100% of their time in net_rx_action and
> > almost all of that time is spent in memcg accounting of the network
> > traffic.
> > 
> > More precisely, net_rx_action is serving packets from multiple workloads
> > and is observing/serving mix of packets of these workloads. The memcg
> > switch of per-cpu cache is very expensive and we are observing a lot of
> > memcg switches on the machine. Almost all the time is being spent on
> > charging new memcg and flushing older memcg cache. So, definitely we
> > need per-cpu cache that support multiple memcgs for this scenario.
> > 
> > This patch implements a simple (and dumb) multiple memcg percpu charge
> > cache. Actually we started with more sophisticated LRU based approach but
> > the dumb one was always better than the sophisticated one by 1% to 3%,
> > so going with the simple approach.
> > 
> > Some of the design choices are:
> > 
> > 1. Fit all caches memcgs in a single cacheline.
> > 2. The cache array can be mix of empty slots or memcg charged slots, so
> >    the kernel has to traverse the full array.
> > 3. The cache drain from the reclaim will drain all cached memcgs to keep
> >    things simple.
> > 
> > To evaluate the impact of this optimization, on a 72 CPUs machine, we
> > ran the following workload where each netperf client runs in a different
> > cgroup. The next-20250415 kernel is used as base.
> > 
> >  $ netserver -6
> >  $ netperf -6 -H ::1 -l 60 -t TCP_SENDFILE -- -m 10K
> > 
> > number of clients | Without patch | With patch
> >   6               | 42584.1 Mbps  | 48603.4 Mbps (14.13% improvement)
> >   12              | 30617.1 Mbps  | 47919.7 Mbps (56.51% improvement)
> >   18              | 25305.2 Mbps  | 45497.3 Mbps (79.79% improvement)
> >   24              | 20104.1 Mbps  | 37907.7 Mbps (88.55% improvement)
> >   30              | 14702.4 Mbps  | 30746.5 Mbps (109.12% improvement)
> >   36              | 10801.5 Mbps  | 26476.3 Mbps (145.11% improvement)
> > 
> > The results show drastic improvement for network intensive workloads.
> > 
> > Signed-off-by: Shakeel Butt <shakeel.butt@...ux.dev>
> 
> Acked-by: Vlastimil Babka <vbabka@...e.cz>
> 
> See below
> 
> > ---
> >  mm/memcontrol.c | 128 ++++++++++++++++++++++++++++++++++--------------
> >  1 file changed, 91 insertions(+), 37 deletions(-)
> > 
> > diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> > index 1ad326e871c1..0a02ba07561e 100644
> > --- a/mm/memcontrol.c
> > +++ b/mm/memcontrol.c
> > @@ -1769,10 +1769,11 @@ void mem_cgroup_print_oom_group(struct mem_cgroup *memcg)
> >  	pr_cont(" are going to be killed due to memory.oom.group set\n");
> >  }
> >  
> > +#define NR_MEMCG_STOCK 7
> >  struct memcg_stock_pcp {
> >  	local_trylock_t stock_lock;
> > -	struct mem_cgroup *cached; /* this never be root cgroup */
> > -	unsigned int nr_pages;
> > +	uint8_t nr_pages[NR_MEMCG_STOCK];
> > +	struct mem_cgroup *cached[NR_MEMCG_STOCK];
> 
> I have noticed memcg_stock is a DEFINE_PER_CPU and not
> DEFINE_PER_CPU_ALIGNED so I think that the intended cacheline usage isn't
> guaranteed now.
> 
> Actually tried compiling and got in objdump -t vmlinux:
> 
> ffffffff83a26e60 l     O .data..percpu  0000000000000088 memcg_stock
> 
> AFAICS that's aligned to 32 bytes only (0x60 is 96) bytes, not 64.
> 
> changing to _ALIGNED gives me:
> 
> ffffffff83a2c5c0 l     O .data..percpu  0000000000000088 memcg_stock
> 
> 0xc0 is 192 so multiple of 64, so seems to work as intended and indeed
> necessary. So you should change it too while adding the comment.
> 

Wow I didn't notice this at all. Thanks a lot. I will fix this in the
next fix diff.

Powered by blists - more mailing lists