linux-kernel - Re: cpu stopper threads and load balancing leads to deadlock

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <1525354359.5576.1.camel@gmx.de>
Date:   Thu, 03 May 2018 15:32:39 +0200
From:   Mike Galbraith <efault@....de>
To:     Peter Zijlstra <peterz@...radead.org>
Cc:     Matt Fleming <matt@...eblueprint.co.uk>,
        Ingo Molnar <mingo@...nel.org>, linux-kernel@...r.kernel.org,
        Michal Hocko <mhocko@...e.com>,
        Paul McKenney <paulmck@...ux.vnet.ibm.com>
Subject: Re: cpu stopper threads and load balancing leads to deadlock

On Thu, 2018-05-03 at 14:49 +0200, Peter Zijlstra wrote:
> On Thu, May 03, 2018 at 02:40:21PM +0200, Mike Galbraith wrote:
> > On Thu, 2018-05-03 at 14:28 +0200, Peter Zijlstra wrote:
> > > 
> > > Hurm.. I don't see how this is 'new'. We moved the wakeup out from under
> > > stopper lock, but that should not affect the RCU state.
> > 
> > No, not new, just an additional woes from same spot.
> 
> Ah, ok. Does somsething like this make it go away?

Dang.  With $subject fix applied as well..

[  151.103732] smpboot: Booting Node 0 Processor 2 APIC 0x4
[  151.104908] =============================
[  151.104909] WARNING: suspicious RCU usage
[  151.104910] 4.17.0.g66d489e-tip-default #84 Tainted: G            E    
[  151.104911] -----------------------------
[  151.104912] kernel/sched/core.c:1625 suspicious rcu_dereference_check() usage!
[  151.104913] 
               other info that might help us debug this:

[  151.104914] 
               RCU used illegally from offline CPU!
               rcu_scheduler_active = 2, debug_locks = 0
[  151.104916] 3 locks held by swapper/2/0:
[  151.104916]  #0: 00000000560adb60 (stop_cpus_mutex){+.+.}, at: stop_machine_from_inactive_cpu+0x86/0x140
[  151.104923]  #1: 00000000e4fb0238 (&p->pi_lock){-.-.}, at: try_to_wake_up+0x2d/0x5f0
[  151.104929]  #2: 000000003341403b (rcu_read_lock){....}, at: rcu_read_lock+0x0/0x80
[  151.104934] 
               stack backtrace:
[  151.104937] CPU: 2 PID: 0 Comm: swapper/2 Kdump: loaded Tainted: G            E     4.17.0.g66d489e-tip-default #84
[  151.104938] Hardware name: MEDION MS-7848/MS-7848, BIOS M7848W08.20C 09/23/2013
[  151.104938] Call Trace:
[  151.104942]  dump_stack+0x78/0xb3
[  151.104945]  ttwu_stat+0x121/0x130
[  151.104949]  try_to_wake_up+0x2c2/0x5f0
[  151.104953]  ? cpu_stop_park+0x30/0x30
[  151.104956]  wake_up_q+0x4a/0x70
[  151.104959]  cpu_stop_queue_work+0x6b/0xa0
[  151.104963]  queue_stop_cpus_work+0x61/0xb0
[  151.104968]  stop_machine_from_inactive_cpu+0xd8/0x140
[  151.104970]  ? mtrr_restore+0x80/0x80
[  151.104976]  mtrr_ap_init+0x62/0x70
[  151.104979]  identify_secondary_cpu+0x18/0x80
[  151.104982]  smp_store_cpu_info+0x44/0x50
[  151.104985]  start_secondary+0x9a/0x1e0
[  151.104988]  secondary_startup_64+0xa5/0xb0

> diff --git a/kernel/stop_machine.c b/kernel/stop_machine.c
> index f89014a2c238..a32518c2ba4a 100644
> --- a/kernel/stop_machine.c
> +++ b/kernel/stop_machine.c
> @@ -650,8 +650,10 @@ int stop_machine_from_inactive_cpu(cpu_stop_fn_t fn, void *data,
>  	/* Schedule work on other CPUs and execute directly for local CPU */
>  	set_state(&msdata, MULTI_STOP_PREPARE);
>  	cpu_stop_init_done(&done, num_active_cpus());
> -	queue_stop_cpus_work(cpu_active_mask, multi_cpu_stop, &msdata,
> -			     &done);
> +
> +	RCU_NONIDLE(queue_stop_cpus_work(cpu_active_mask, multi_cpu_stop,
> +				         &msdata, &done));
> +
>  	ret = multi_cpu_stop(&msdata);
>  
>  	/* Busy wait for completion. */