linux-kernel - Re: [PATCH v6 2/3] sched: Move task_mm_cid

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <35fe8e74229af24f45954dd27789363dd5c2f8b8.camel@redhat.com>
Date: Thu, 13 Feb 2025 15:37:47 +0100
From: Gabriele Monaco <gmonaco@...hat.com>
To: Mathieu Desnoyers <mathieu.desnoyers@...icios.com>, 
	linux-kernel@...r.kernel.org
Cc: linux-mm@...ck.org, aubrey.li@...ux.intel.com, yu.c.chen@...el.com,
 Andrew Morton <akpm@...ux-foundation.org>, Ingo Molnar <mingo@...hat.com>,
 Peter Zijlstra	 <peterz@...radead.org>, Ingo Molnar <mingo@...nel.org>,
 "Paul E. McKenney"	 <paulmck@...nel.org>, Shuah Khan <shuah@...nel.org>
Subject: Re: [PATCH v6 2/3] sched: Move task_mm_cid_work to mm delayed work



On Thu, 2025-02-13 at 08:55 -0500, Mathieu Desnoyers wrote:
> On 2025-02-13 08:25, Gabriele Monaco wrote:
> > On Thu, 2025-02-13 at 14:52 +0800, kernel test robot wrote:
> > > kernel test robot noticed
> > > "WARNING:at_kernel/workqueue.c:#__queue_delayed_work" on:
> > > 
> > > [    2.640924][    T0] ------------[ cut here ]------------
> > > [ 2.641646][ T0] WARNING: CPU: 0 PID: 0 at
> > > kernel/workqueue.c:2495
> > > __queue_delayed_work (kernel/workqueue.c:2495 (discriminator 9))
> > > [    2.642874][    T0] Modules linked in:
> > > [    2.643381][    T0] CPU: 0 UID: 0 PID: 0 Comm: swapper Not
> > > tainted
> > > 6.14.0-rc2-00002-g287adf9e9c1f #1
> > > [    2.644582][    T0] Hardware name: QEMU Standard PC (i440FX +
> > > PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
> > > [ 2.645943][ T0] RIP: 0010:__queue_delayed_work
> > > (kernel/workqueue.c:2495 (discriminator 9))
> > 
> > There seem to be major problems with this configuration, I'm trying
> > to
> > understand what's wrong but, for the time being, this patchset is
> > not
> > ready for inclusion.
> 
> I think there is an issue with the order of init functions at boot.
> 
> poking_init() calls mm_alloc(), which ends up calling mm_init().
> 
> The WARN_ON() is about a NULL wq pointer, which I suspect happens
> if poking_init() is called before workqueue_init_early(), which
> allocates system_wq.
> 
> Indeed, in start_kernel(), poking_init() is called before
> workqueue_init_early().
> 
> I'm not sure what are the init order dependencies across subsystems
> here.
> There is the following order in start_kernel():
> 
> [...]
>          mm_core_init();
>          poking_init();
>          ftrace_init();
> 
>          /* trace_printk can be enabled here */
>          early_trace_init();
> 
>          /*
>           * Set up the scheduler prior starting any interrupts (such
> as the
>           * timer interrupt). Full topology setup happens at
> smp_init()
>           * time - but meanwhile we still have a functioning
> scheduler.
>           */
>          sched_init();
> 
>          if (WARN(!irqs_disabled(),
>                   "Interrupts were enabled *very* early, fixing
> it\n"))
>                  local_irq_disable();
>          radix_tree_init();
>          maple_tree_init();
> 
>          /*
>           * Set up housekeeping before setting up workqueues to allow
> the unbound
>           * workqueue to take non-housekeeping into account.
>           */
>          housekeeping_init();
> 
>          /*
>           * Allow workqueue creation and work item
> queueing/cancelling
>           * early.  Work item execution depends on kthreads and
> starts after
>           * workqueue_init().
>           */
>          workqueue_init_early();
> [...]
> 
> So either we find a way to reorder this, or we make sure
> poking_init()
> does not require the workqueue.
> 
> Thanks,
> 
> Mathieu
> 

Nice suggestion! That seems the culprit..

>From the full dmesg of the failure I've seen there's also a problem
with disabling the delayed work synchronously, since mmdrop cannot
sleep if we are not in PREEMPT_RT.

I'm trying to come with some satisfactory solution for both, ideally:
1. the delayed work is not needed in early boot, we may have a better
place where to start it
2. we can cancel the work asynchronously on mmdrop and abort it if the
pcpu_cid is null, but it seems racy, perhaps there's a better place for
that too

Thanks,
Gabriele