linux-kernel - Re: [BUG] kernel freezes with latest tree

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <1326284711.2442.138.camel@twins>
Date:	Wed, 11 Jan 2012 13:25:11 +0100
From:	Peter Zijlstra <peterz@...radead.org>
To:	David Ahern <dsahern@...il.com>
Cc:	Linus Torvalds <torvalds@...ux-foundation.org>,
	Eric Dumazet <eric.dumazet@...il.com>,
	Ingo Molnar <mingo@...e.hu>,
	Thomas Gleixner <tglx@...utronix.de>,
	Martin Schwidefsky <schwidefsky@...ibm.com>,
	linux-kernel <linux-kernel@...r.kernel.org>,
	Frederic Weisbecker <fweisbec@...il.com>,
	Suresh Siddha <suresh.b.siddha@...el.com>
Subject: Re: [BUG] kernel freezes with latest tree

On Wed, 2012-01-11 at 10:04 +0100, Peter Zijlstra wrote:
> Maybe adding a few more NEED_BREAK bits
> and making it a counter and overflowing it into ABORT might be good.
> 
> 
I could reproduce and confirm something like the below makes the hang
go-away. I haven't managed to fully understand why we're stuck though
because we do release the runqueue locks and re-enable IRQs on this
lock-break.

My stuck machine had several CPUs stuck in a load-balance pass, so it
could be they're bouncing tasks back and forth without actually making
any progress what so ever.

I reproduced with hackbench 500, which results in 20000 tasks, spread
over 24 cpus that gives some 833 tasks per runqueue on average, easily
overflowing that lock-break scanning limit.

---
Subject: sched: Limit load-balance retries on lock-break
From: Peter Zijlstra <a.p.zijlstra@...llo.nl>
Date: Wed Jan 11 13:11:12 CET 2012

Eric and David reported dead machines and traced it to commit a195f004 ("sched:
Fix load-balance lock-breaking"), it turns out there's still a
scenario where we can end up re-trying forever.

Limit the number of retries and simply abort.

Reported-by: Eric Dumazet <eric.dumazet@...il.com>
Reported-by: David Ahern <dsahern@...il.com>
Signed-off-by: Peter Zijlstra <a.p.zijlstra@...llo.nl>
---
 kernel/sched/fair.c |    9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -3130,8 +3130,10 @@ task_hot(struct task_struct *p, u64 now,
 }
 
 #define LBF_ALL_PINNED	0x01
-#define LBF_NEED_BREAK	0x02
-#define LBF_ABORT	0x04
+#define LBF_NEED_BREAK	0x02	/* clears into HAD_BREAK */
+#define LBF_HAD_BREAK	0x04
+#define LBF_HAD_BREAKS	0x0C	/* count HAD_BREAKs overflows into ABORT */
+#define LBF_ABORT	0x10
 
 /*
  * can_migrate_task - may task p from runqueue rq be migrated to this_cpu?
@@ -4509,6 +4511,9 @@ static int load_balance(int this_cpu, st
 
 		if (lb_flags & LBF_NEED_BREAK) {
 			lb_flags &= ~LBF_NEED_BREAK;
+			lb_flags += LBF_HAD_BREAK;
+			if (lb_flags & LBF_ABORT)
+				goto out_balanced;
 			goto redo;
 		}
 

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/