Message-ID: <7b76f0e4-6148-4290-a295-7ceeb1ef2feb@efficios.com>
Date: Fri, 7 Nov 2025 10:53:34 -0500
From: Mathieu Desnoyers <mathieu.desnoyers@...icios.com>
To: Shakeel Butt <shakeel.butt@...ux.dev>,
 kernel test robot <oliver.sang@...el.com>
Cc: oe-lkp@...ts.linux.dev, lkp@...el.com,
 Andrew Morton <akpm@...ux-foundation.org>,
 "Paul E. McKenney" <paulmck@...nel.org>, Steven Rostedt
 <rostedt@...dmis.org>, Masami Hiramatsu <mhiramat@...nel.org>,
 Dennis Zhou <dennis@...nel.org>, Tejun Heo <tj@...nel.org>,
 Christoph Lameter <cl@...ux.com>, Martin Liu <liumartin@...gle.com>,
 David Rientjes <rientjes@...gle.com>, SeongJae Park <sj@...nel.org>,
 Michal Hocko <mhocko@...e.com>, Johannes Weiner <hannes@...xchg.org>,
 Sweet Tea Dorminy <sweettea@...gle.com>,
 Lorenzo Stoakes <lorenzo.stoakes@...cle.com>,
 "Liam R . Howlett" <liam.howlett@...cle.com>, Mike Rapoport
 <rppt@...nel.org>, Suren Baghdasaryan <surenb@...gle.com>,
 Vlastimil Babka <vbabka@...e.cz>, Christian Brauner <brauner@...nel.org>,
 Wei Yang <richard.weiyang@...il.com>, David Hildenbrand <david@...hat.com>,
 Miaohe Lin <linmiaohe@...wei.com>, Al Viro <viro@...iv.linux.org.uk>,
 Yu Zhao <yuzhao@...gle.com>, Roman Gushchin <roman.gushchin@...ux.dev>,
 Mateusz Guzik <mjguzik@...il.com>, Matthew Wilcox <willy@...radead.org>,
 Baolin Wang <baolin.wang@...ux.alibaba.com>,
 Aboorva Devarajan <aboorvad@...ux.ibm.com>, linux-mm@...ck.org,
 linux-kernel@...r.kernel.org, linux-trace-kernel@...r.kernel.org,
 christian.koenig@....com
Subject: Re: [RFC PATCH v7 2/2] mm: Fix OOM killer inaccuracy on large
 many-core systems

On 2025-11-07 09:43, Mathieu Desnoyers wrote:
> On 2025-11-06 19:32, Shakeel Butt wrote:
> 
> [...]
> 
>>> [   14.858862][   T67] BUG: Bad rss-counter state mm:ffff8881000655c0 
>>> type:MM_ANONPAGES val:0 Comm:kworker/u9:0 Pid:67
>>> [   14.894890][   T69] BUG: Bad rss-counter state mm:ffff888100061cc0 
>>> type:MM_FILEPAGES val:0 Comm:kworker/u9:0 Pid:69
>>> [   14.896108][   T69] BUG: Bad rss-counter state mm:ffff888100061cc0 
>>> type:MM_ANONPAGES val:0 Comm:kworker/u9:0 Pid:69
>>
>> Hmm this shows that percpu_counter_tree_precise_sum() is returning 0 but
>> percpu_counter_tree_approximate_sum() is off more than
>> counter->inaccuracy. I have not dig deeper to find why but this needs to
>> be resolved before considering this series for upstream.
> 
> I notice that those BUG show up while loading modules at boot in kworker 
> context, e.g.:
> 
> [   14.858862][   T67] BUG: Bad rss-counter state mm:ffff8881000655c0 
> type:MM_ANONPAGES val:0 Comm:kworker/u9:0 Pid:67
> [   14.894890][   T69] BUG: Bad rss-counter state mm:ffff888100061cc0 
> type:MM_FILEPAGES val:0 Comm:kworker/u9:0 Pid:69
> [   14.896108][   T69] BUG: Bad rss-counter state mm:ffff888100061cc0 
> type:MM_ANONPAGES val:0 Comm:kworker/u9:0 Pid:69
> [   14.918858][   T71] module: module-autoload: duplicate request for 
> module crypto-aes
> [   14.919479][   T71] module: module-autoload: duplicate request for 
> module crypto-aes-all
> [   14.920801][    T1] krb5: Running aes128-cts-hmac-sha256-128 enc 
> plain<block
> [   14.921844][    T1] krb5: Running aes128-cts-hmac-sha256-128 enc 
> plain==block
> [   14.922852][    T1] krb5: Running aes128-cts-hmac-sha256-128 enc 
> plain>block
> [   14.923843][    T1] krb5: Running aes256-cts-hmac-sha384-192 enc no 
> plain
> [   14.939591][    T1] krb5: Running aes256-cts-hmac-sha384-192 enc 
> plain<block
> [   14.940614][    T1] krb5: Running aes256-cts-hmac-sha384-192 enc 
> plain==block
> [   14.941586][    T1] krb5: Running aes256-cts-hmac-sha384-192 enc 
> plain>block
> [   14.942547][    T1] krb5: Running camellia128-cts-cmac enc no plain
> [   15.018568][   T85] BUG: Bad rss-counter state mm:ffff888160f81340 
> type:MM_ANONPAGES val:0 Comm:kworker/u9:0 Pid:85b
> 
> I used "module_init" similarly to lib/percpu_counter.c, but I think it
> happens too late in the boot sequence:
> 
>    module_init(percpu_counter_startup);
> 
> module_init maps to __initcall within a built-in compile unit, which
> maps to device_initcall(), which happens quite late within the sequence
> called from do_initcalls(), called from do_basic_setup().
> 
> And even do_basic_setup is documented as:
> 
>   * Ok, the machine is now initialized. None of the devices
>   * have been touched yet, but the CPU subsystem is up and
>   * running, and memory and process management works.
> 
> which clearly requires that the mm subsystem is expected to
> be ready at that point.
> 
> It probably was not an issue for the non-hierarchical percpu
> counters because all it was initializing is handling of CPU hotplug,
> but the new hierarchical counters initialize the pre-calculated
> inaccuracy value which is used to figure out whether the approximate
> sum is sufficient to compare values or if the precise sum is needed.
> 
> I think this is why we are hitting this BUG.
> 
> Now I wonder where I should move this initialization. It requires
> "nr_cpu_ids" to be initialized, and pretty much need to be done
> before mms are created. I'm starting to suspect that the module init
> code can spawn kworkers that have a mm before the init process runs.

At least on x86, nr_cpu_ids appears to be set by set_nr_cpu_ids()
through early_param("possible_cpus", setup_possible_cpus), which is
AFAIU called from parse_early_param(), which happens very early in the
boot sequence.

It would make sense to call an explicit percpu counter tree init
function from start_kernel() between the call to mm_core_init() and the
call to maple_tree_init(). This way it would be initialized right after
mm, but given that the hierarchical counter tree is a lib that can be
used for other purposes than mm accounting, I think it makes sense
to call its init explicitly from start_kernel() rather than bury
it within mm_core_init().

Thoughts ?

Thanks,

Mathieu

-- 
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com

Powered by blists - more mailing lists