linux-kernel - Re: [PATCH RT] fix IPI balancing for 4.14-rt

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20171121102436.3e25655f@gandalf.local.home>
Date:   Tue, 21 Nov 2017 10:24:36 -0500
From:   Steven Rostedt <rostedt@...dmis.org>
To:     Clark Williams <williams@...hat.com>
Cc:     Sebastian Andrzej Siewior <bigeasy@...utronix.de>,
        LKML <linux-kernel@...r.kernel.org>,
        Peter Zijlstra <peterz@...radead.org>,
        Thomas Gleixner <tglx@...utronix.de>,
        Ingo Molnar <mingo@...nel.org>,
        Daniel Bristot de Oliveira <bristot@...hat.com>,
        John Kacur <jkacur@...hat.com>, Scott Wood <swood@...hat.com>,
        Mike Galbraith <efault@....de>
Subject: Re: [PATCH RT] fix IPI balancing for 4.14-rt

On Tue, 21 Nov 2017 09:14:25 -0600
Clark Williams <williams@...hat.com> wrote:

> From 8ea8311b75a40bdea03e7f8228a0578b6367e9d1 Mon Sep 17 00:00:00 2001
> From: Clark Williams <williams@...hat.com>
> Date: Mon, 20 Nov 2017 14:26:12 -0600
> Subject: [PATCH] [rt] sched/rt: fix panic in double_lock_balance with
>  simplified IPI RT balancing
> 
> I was testing 4.14-rt1 on a large system (cores == 96) and saw that
> we were getting into an rt balancing storm, so I tried applying Steven's
> patch (not upstream yet):
> 
>     sched/rt: Simplify the IPI rt balancing logic
> 
> Booting the resulting kernel yielded a panic in
> double_lock_balance() due to irqs not being disabled.
> 
> This patch changes the calls to raw_spin_{lock,unlock} in the
> function rto_push_irq_work_function, to be raw_spin_{lock,unlock}_irq.
> Not sure if that's too heavy a hammer, but the resulting kernel boots
> and runs and survives 12h runs of rteval. Once Steven's patch goes in 
> upstream, we'll need something like this in RT.
> 
> Signed-off-by: Clark Williams <williams@...hat.com>
> ---
>  kernel/sched/rt.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c
> index 57fb251dd8ce..a5cd0cea2f0f 100644
> --- a/kernel/sched/rt.c
> +++ b/kernel/sched/rt.c
> @@ -2008,9 +2008,9 @@ void rto_push_irq_work_func(struct irq_work *work)
>  	 * When it gets updated, a check is made if a push is possible.
>  	 */
>  	if (has_pushable_tasks(rq)) {
> -		raw_spin_lock(&rq->lock);
> +		raw_spin_lock_irq(&rq->lock);
>  		push_rt_tasks(rq);
> -		raw_spin_unlock(&rq->lock);
> +		raw_spin_unlock_irq(&rq->lock);

This looks buggy to me. You know you just indiscriminately enabled
interrupts here.
 
>  	}
>  
>  	raw_spin_lock(&rq->rd->rto_lock);

Why is this patch necessary?

Is it because you have the irq_work running in non hard irq context? I
think you need something like this instead (if you haven't already
added it):

-- Steve

Index: linux-rt.git/kernel/sched/topology.c
===================================================================
--- linux-rt.git.orig/kernel/sched/topology.c
+++ linux-rt.git/kernel/sched/topology.c
@@ -257,6 +257,7 @@ static int init_rootdomain(struct root_d
 	rd->rto_cpu = -1;
 	raw_spin_lock_init(&rd->rto_lock);
 	init_irq_work(&rd->rto_push_work, rto_push_irq_work_func);
+	rd->rto_push_work.flags |= IRQ_WORK_HARD_IRQ;
 #endif
 
 	init_dl_bw(&rd->dl_bw);