linux-kernel - Re: single aio thread is migrated crazily by scheduler

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-Id: <20191118162633.GC32306@linux.vnet.ibm.com>
Date:   Mon, 18 Nov 2019 21:56:33 +0530
From:   Srikar Dronamraju <srikar@...ux.vnet.ibm.com>
To:     Dave Chinner <david@...morbit.com>
Cc:     Ming Lei <ming.lei@...hat.com>, linux-block@...r.kernel.org,
        linux-fsdevel@...r.kernel.org, linux-xfs@...r.kernel.org,
        linux-kernel@...r.kernel.org, Jeff Moyer <jmoyer@...hat.com>,
        Dave Chinner <dchinner@...hat.com>,
        Eric Sandeen <sandeen@...hat.com>,
        Christoph Hellwig <hch@....de>, Jens Axboe <axboe@...nel.dk>,
        Ingo Molnar <mingo@...hat.com>,
        Peter Zijlstra <peterz@...radead.org>,
        Tejun Heo <tj@...nel.org>
Subject: Re: single aio thread is migrated crazily by scheduler

* Dave Chinner <david@...morbit.com> [2019-11-16 10:40:05]:

> On Fri, Nov 15, 2019 at 03:08:43PM +0800, Ming Lei wrote:
> > On Fri, Nov 15, 2019 at 03:56:34PM +1100, Dave Chinner wrote:
> > > On Fri, Nov 15, 2019 at 09:08:24AM +0800, Ming Lei wrote:
> > I can reproduce the issue with 4k block size on another RH system, and
> > the login info of that system has been shared to you in RH BZ.
> > 
> > 1)
> 
> Almost all the fio task migrations are coming from migration/X
> kernel threads. i.e it's the scheduler active balancing that is
> causing the fio thread to bounce around.
> 

Can we try with the below patch.

-- 
Thanks and Regards
Srikar Dronamraju

--->8-----------------------------8<----------------------------------
>From 9687c1447532558aa564bd2e471b7987d6bda70f Mon Sep 17 00:00:00 2001
From: Srikar Dronamraju <srikar@...ux.vnet.ibm.com>
Date: Tue, 2 Jul 2019 16:38:29 -0500
Subject: [PATCH] sched/fair: Avoid active balance on small load imbalance

Skip active load balance when destination CPU is busy and the imbalance
is small and fix_small_imabalance is unable to calculate minor
imbalance. Its observed that active load balances can lead to ping-pong
of tasks between two CPUs.

Signed-off-by: Srikar Dronamraju <srikar@...ux.vnet.ibm.com>
---
 kernel/sched/fair.c | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 3599bdcab395..0db380c8eb6c 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -7094,6 +7094,7 @@ enum group_type {
 #define LBF_SOME_PINNED	0x08
 #define LBF_NOHZ_STATS	0x10
 #define LBF_NOHZ_AGAIN	0x20
+#define LBF_SMALL_IMBL	0x40
 
 struct lb_env {
 	struct sched_domain	*sd;
@@ -8386,6 +8387,8 @@ void fix_small_imbalance(struct lb_env *env, struct sd_lb_stats *sds)
 	/* Move if we gain throughput */
 	if (capa_move > capa_now)
 		env->imbalance = busiest->load_per_task;
+	else if (env->idle == CPU_NOT_IDLE)
+		env->flags |= LBF_SMALL_IMBL;
 }
 
 /**
@@ -8466,7 +8469,7 @@ static inline void calculate_imbalance(struct lb_env *env, struct sd_lb_stats *s
 	 * moved
 	 */
 	if (env->imbalance < busiest->load_per_task)
-		return fix_small_imbalance(env, sds);
+		fix_small_imbalance(env, sds);
 }
 
 /******* find_busiest_group() helpers end here *********************/
@@ -8732,6 +8735,13 @@ static int need_active_balance(struct lb_env *env)
 	if (voluntary_active_balance(env))
 		return 1;
 
+	/*
+	 * Destination CPU is not idle and fix_small_imbalance is unable
+	 * to calculate even minor imbalances, skip active balance.
+	 */
+	if (env->flags & LBF_SMALL_IMBL)
+		return 0;
+
 	return unlikely(sd->nr_balance_failed > sd->cache_nice_tries+2);
 }
 
-- 
2.18.1