| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Date: Wed, 11 Jan 2012 14:24:28 +0100
From: Eric Dumazet <eric.dumazet@...il.com>
To: Peter Zijlstra <peterz@...radead.org>
Cc: David Ahern <dsahern@...il.com>,
Linus Torvalds <torvalds@...ux-foundation.org>,
Ingo Molnar <mingo@...e.hu>,
Thomas Gleixner <tglx@...utronix.de>,
Martin Schwidefsky <schwidefsky@...ibm.com>,
linux-kernel <linux-kernel@...r.kernel.org>,
Frederic Weisbecker <fweisbec@...il.com>,
Suresh Siddha <suresh.b.siddha@...el.com>
Subject: Re: [BUG] kernel freezes with latest tree
Le mercredi 11 janvier 2012 à 13:25 +0100, Peter Zijlstra a écrit :
> On Wed, 2012-01-11 at 10:04 +0100, Peter Zijlstra wrote:
> > Maybe adding a few more NEED_BREAK bits
> > and making it a counter and overflowing it into ABORT might be good.
> >
> >
> I could reproduce and confirm something like the below makes the hang
> go-away. I haven't managed to fully understand why we're stuck though
> because we do release the runqueue locks and re-enable IRQs on this
> lock-break.
>
> My stuck machine had several CPUs stuck in a load-balance pass, so it
> could be they're bouncing tasks back and forth without actually making
> any progress what so ever.
>
> I reproduced with hackbench 500, which results in 20000 tasks, spread
> over 24 cpus that gives some 833 tasks per runqueue on average, easily
> overflowing that lock-break scanning limit.
>
> ---
> Subject: sched: Limit load-balance retries on lock-break
> From: Peter Zijlstra <a.p.zijlstra@...llo.nl>
> Date: Wed Jan 11 13:11:12 CET 2012
>
> Eric and David reported dead machines and traced it to commit a195f004 ("sched:
> Fix load-balance lock-breaking"), it turns out there's still a
> scenario where we can end up re-trying forever.
>
> Limit the number of retries and simply abort.
>
> Reported-by: Eric Dumazet <eric.dumazet@...il.com>
> Reported-by: David Ahern <dsahern@...il.com>
> Signed-off-by: Peter Zijlstra <a.p.zijlstra@...llo.nl>
> ---
> kernel/sched/fair.c | 9 +++++++--
> 1 file changed, 7 insertions(+), 2 deletions(-)
>
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -3130,8 +3130,10 @@ task_hot(struct task_struct *p, u64 now,
> }
>
> #define LBF_ALL_PINNED 0x01
> -#define LBF_NEED_BREAK 0x02
> -#define LBF_ABORT 0x04
> +#define LBF_NEED_BREAK 0x02 /* clears into HAD_BREAK */
> +#define LBF_HAD_BREAK 0x04
> +#define LBF_HAD_BREAKS 0x0C /* count HAD_BREAKs overflows into ABORT */
> +#define LBF_ABORT 0x10
>
> /*
> * can_migrate_task - may task p from runqueue rq be migrated to this_cpu?
> @@ -4509,6 +4511,9 @@ static int load_balance(int this_cpu, st
>
> if (lb_flags & LBF_NEED_BREAK) {
> lb_flags &= ~LBF_NEED_BREAK;
> + lb_flags += LBF_HAD_BREAK;
> + if (lb_flags & LBF_ABORT)
> + goto out_balanced;
> goto redo;
> }
>
>
The two lines :
lb_flags &= ~LBF_NEED_BREAK;
lb_flags += LBF_HAD_BREAK;
could be combined to :
lb_flags += LBF_HAD_BREAK - LBF_NEED_BREAK;
Anyway, your patch solved the problem on my machines, thanks a lot.
$ uptime
14:20:32 up 31 min, 2 users, load average: 170.18, 183.00, 156.53
Tested-by: Eric Dumazet <eric.dumazet@...il.com>
One perf top output, on the 32bit kernel, to check scheduler functions
dont use too many cycles ;)
12.31% hackbench [kernel.kallsyms] [k] __slab_free
8.98% hackbench [kernel.kallsyms] [k] _raw_spin_lock_irqsave
6.52% hackbench [kernel.kallsyms] [k] sock_alloc_send_pskb
6.25% hackbench [kernel.kallsyms] [k] _raw_spin_lock
5.33% hackbench [kernel.kallsyms] [k] fget_light
5.04% hackbench [kernel.kallsyms] [k] unix_stream_recvmsg
4.01% hackbench [kernel.kallsyms] [k] __copy_user_intel
3.14% hackbench [kernel.kallsyms] [k] __kmalloc_node_track_caller
2.61% hackbench [kernel.kallsyms] [k] __alloc_skb
2.55% hackbench [kernel.kallsyms] [k] skb_release_head_state
2.18% hackbench [kernel.kallsyms] [k] sock_wfree
2.00% hackbench [kernel.kallsyms] [k] cred_to_ucred
1.68% hackbench [kernel.kallsyms] [k] kfree
1.65% hackbench [kernel.kallsyms] [k] unix_stream_sendmsg
1.62% hackbench [kernel.kallsyms] [k] kmem_cache_alloc_node
1.55% hackbench [kernel.kallsyms] [k] unix_destruct_scm
1.43% hackbench [kernel.kallsyms] [k] __schedule
1.39% hackbench [kernel.kallsyms] [k] sched_clock_cpu
1.37% hackbench [kernel.kallsyms] [k] kmem_cache_free
1.30% hackbench [kernel.kallsyms] [k] sock_def_readable
1.25% hackbench [kernel.kallsyms] [k] __slab_alloc
1.19% hackbench [kernel.kallsyms] [k] sched_clock_local
1.16% hackbench [kernel.kallsyms] [k] fput
1.11% hackbench [kernel.kallsyms] [k] sysenter_past_esp
0.98% hackbench [kernel.kallsyms] [k] get_partial_node.isra.43
0.93% hackbench [kernel.kallsyms] [k] select_task_rq_fair
0.81% hackbench [unknown] [.] 0xffffe424
0.80% hackbench [kernel.kallsyms] [k] update_curr
0.75% hackbench [kernel.kallsyms] [k] skb_release_data
0.60% hackbench [kernel.kallsyms] [k] try_to_wake_up
0.56% hackbench [kernel.kallsyms] [k] vfs_write
0.54% hackbench libpthread-2.3.4.so [.] __pthread_disable_asynccancel
0.52% hackbench [kernel.kallsyms] [k] update_rq_clock
0.47% hackbench [kernel.kallsyms] [k] ktime_get
0.45% hackbench libpthread-2.3.4.so [.] __pthread_enable_asynccancel
0.45% hackbench [kernel.kallsyms] [k] skb_dequeue
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Powered by blists - more mailing lists