linux-kernel - Re: [PATCH] cgroup: rstat: use LOCK CMPXCHG in css_rstat

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <5i4ei2rbszdwlezpi63h5ksmckry27ffx6kfcg74qbvgjk22ao@2y2jeadg43y3>
Date: Fri, 5 Dec 2025 09:59:39 -0800
From: Shakeel Butt <shakeel.butt@...ux.dev>
To: Tejun Heo <tj@...nel.org>
Cc: Johannes Weiner <hannes@...xchg.org>, 
	Michal Koutný <mkoutny@...e.com>, "Paul E . McKenney" <paulmck@...nel.org>, 
	JP Kobryn <inwardvessel@...il.com>, Yosry Ahmed <yosry.ahmed@...ux.dev>, linux-mm@...ck.org, 
	cgroups@...r.kernel.org, linux-kernel@...r.kernel.org, 
	Meta kernel team <kernel-team@...a.com>
Subject: Re: [PATCH] cgroup: rstat: use LOCK CMPXCHG in css_rstat_updated

On Fri, Dec 05, 2025 at 07:35:30AM -1000, Tejun Heo wrote:
> Hello,
> 
> On Thu, Dec 04, 2025 at 06:24:37PM -0800, Shakeel Butt wrote:
> ...
> > In Meta's fleet running the kernel with the commit 36df6e3dbd7e, we are
> > observing on some machines the memcg stats are getting skewed by more
> > than the actual memory on the system. On close inspection, we noticed
> > that lockless node for a workload for specific CPU was in the bad state
> > and thus all the updates on that CPU for that cgroup was being lost. At
> > the moment, we are not sure if this CMPXCHG without LOCK is the cause of
> > that but this needs to be fixed irrespective.
> 
> Is there a plausible theory of events that can explain the skew with the use
> of this_cpu_cmpxchg()? lnode.next being set to self but this_cpu_cmpxchg()
> returning something else? It may be useful to write a targeted repro for the
> particular combination - this_cpu_cmpxchg() vs. remote NULL clearing and see
> whether this_cpu_cmpxchg() can return a value that doesn't agree with what
> gets written in the memory.

Yes, I am working on creating a repro for this and will share the
results.

> 
> > @@ -113,9 +112,8 @@ __bpf_kfunc void css_rstat_updated(struct cgroup_subsys_state *css, int cpu)
> >  	 * successful and the winner will eventually add the per-cpu lnode to
> >  	 * the llist.
> >  	 */
> > -	self = &rstatc->lnode;
> > -	rstatc_pcpu = css->rstat_cpu;
> > -	if (this_cpu_cmpxchg(rstatc_pcpu->lnode.next, self, NULL) != self)
> > +	expected = &rstatc->lnode;
> > +	if (!try_cmpxchg(&rstatc->lnode.next, &expected, NULL))
> 
> Given that this is a relatively cold path, I don't see a problem with using
> locked op here even if this wasn't necessarily the culprit; however, can you
> please update the comment right above accordingly and explain why the locked
> op is used? After this patch, the commend and code disagree.

Thanks and yes I will update the comment in the next version.