linux-kernel - Re: [PATCH v3] sched/core: Tweak wait_task_inactive() to force dequeue sched

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <20250429153605.GA69823@pauld.westford.csb>
Date: Tue, 29 Apr 2025 11:36:05 -0400
From: Phil Auld <pauld@...hat.com>
To: John Stultz <jstultz@...gle.com>
Cc: LKML <linux-kernel@...r.kernel.org>, Ingo Molnar <mingo@...hat.com>,
	Peter Zijlstra <peterz@...radead.org>,
	Juri Lelli <juri.lelli@...hat.com>,
	Vincent Guittot <vincent.guittot@...aro.org>,
	Dietmar Eggemann <dietmar.eggemann@....com>,
	Steven Rostedt <rostedt@...dmis.org>,
	Ben Segall <bsegall@...gle.com>, Mel Gorman <mgorman@...e.de>,
	Valentin Schneider <vschneid@...hat.com>,
	K Prateek Nayak <kprateek.nayak@....com>, kernel-team@...roid.com,
	peter-yc.chang@...iatek.com
Subject: Re: [PATCH v3] sched/core: Tweak wait_task_inactive() to force
 dequeue sched_delayed tasks

Hi John,

On Tue, Apr 29, 2025 at 08:07:26AM -0700 John Stultz wrote:
> It was reported that in 6.12, smpboot_create_threads() was
> taking much longer then in 6.6.
> 
> I narrowed down the call path to:
>  smpboot_create_threads()
>  -> kthread_create_on_cpu()
>     -> kthread_bind()
>        -> __kthread_bind_mask()
>           ->wait_task_inactive()
> 
> Where in wait_task_inactive() we were regularly hitting the
> queued case, which sets a 1 tick timeout, which when called
> multiple times in a row, accumulates quickly into a long
> delay.
> 
> I noticed disabling the DELAY_DEQUEUE sched feature recovered
> the performance, and it seems the newly create tasks are usually
> sched_delayed and left on the runqueue.

This seems odd to me. Maybe I'm just misunderstanding something but
I don't see how newly created tasks should have accumulated enough
runtime to have negative lag that needs to be decayed. 

That said, I think it does make sense to dequeue in this case. 

Cheers,
Phil

> 
> So in wait_task_inactive() when we see the task
> p->se.sched_delayed, manually dequeue the sched_delayed task
> with DEQUEUE_DELAYED, so we don't have to constantly wait a
> tick.
> 
> This seems to work, but I've only lightly tested it, so I'd love
> close review and feedback in case I've missed something in
> wait_task_inactive(), or if there is a simpler alternative
> approach.
> 
> NOTE: Peter did highlight[1] his general distaste for the
> kthread_bind() through wait_task_inactive() functions, which
> suggests a deeper rework might be better, but I'm not familiar
> enough with all its users to have a sense of how that might be
> done, and this fix seems to address the problem and be more
> easily backported to 6.12-stable, so I wanted to submit it
> again, as a potentially more short-term solution.
> 
> [1]: https://lore.kernel.org/lkml/20250422085628.GA14170@noisy.programming.kicks-ass.net/
> 
> Cc: Ingo Molnar <mingo@...hat.com>
> Cc: Peter Zijlstra <peterz@...radead.org>
> Cc: Juri Lelli <juri.lelli@...hat.com>
> Cc: Vincent Guittot <vincent.guittot@...aro.org>
> Cc: Dietmar Eggemann <dietmar.eggemann@....com>
> Cc: Steven Rostedt <rostedt@...dmis.org>
> Cc: Ben Segall <bsegall@...gle.com>
> Cc: Mel Gorman <mgorman@...e.de>
> Cc: Valentin Schneider <vschneid@...hat.com>
> Cc: K Prateek Nayak <kprateek.nayak@....com>
> Cc: kernel-team@...roid.com
> Fixes: 152e11f6df29 ("sched/fair: Implement delayed dequeue")
> Reported-by: peter-yc.chang@...iatek.com
> Tested-by: K Prateek Nayak <kprateek.nayak@....com>
> Signed-off-by: John Stultz <jstultz@...gle.com>
> ---
> v2:
> * Rework & simplify the check as suggested by K Prateek Nayak
> * Added Reported-by tag for proper attribution
> v3:
> * Add Fixed-by: and Tested-by tag suggested by Prateek
> ---
>  kernel/sched/core.c | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> index c81cf642dba05..b986cd2fb19b7 100644
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -2283,6 +2283,12 @@ unsigned long wait_task_inactive(struct task_struct *p, unsigned int match_state
>  		 * just go back and repeat.
>  		 */
>  		rq = task_rq_lock(p, &rf);
> +		/*
> +		 * If task is sched_delayed, force dequeue it, to avoid always
> +		 * hitting the tick timeout in the queued case
> +		 */
> +		if (p->se.sched_delayed)
> +			dequeue_task(rq, p, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
>  		trace_sched_wait_task(p);
>  		running = task_on_cpu(rq, p);
>  		queued = task_on_rq_queued(p);
> -- 
> 2.49.0.901.g37484f566f-goog
> 
> 

--