linux-kernel - Re: Query regarding synchronize_sched_expedited and resched

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <20170919053749.GA12412@tardis>
Date:   Tue, 19 Sep 2017 13:37:49 +0800
From:   Boqun Feng <boqun.feng@...il.com>
To:     "Paul E. McKenney" <paulmck@...ux.vnet.ibm.com>
Cc:     Byungchul Park <byungchul.park@....com>,
        Steven Rostedt <rostedt@...dmis.org>,
        Neeraj Upadhyay <neeraju@...eaurora.org>,
        josh@...htriplett.org, mathieu.desnoyers@...icios.com,
        jiangshanlai@...il.com, linux-kernel@...r.kernel.org,
        sramana@...eaurora.org, prsood@...eaurora.org,
        pkondeti@...eaurora.org, markivx@...eaurora.org,
        peterz@...radead.org, kernel-team@....com
Subject: Re: Query regarding synchronize_sched_expedited and resched_cpu

On Mon, Sep 18, 2017 at 09:04:56PM -0700, Paul E. McKenney wrote:
> On Tue, Sep 19, 2017 at 11:48:22AM +0900, Byungchul Park wrote:
> > On Mon, Sep 18, 2017 at 07:33:29PM -0700, Paul E. McKenney wrote:
> > > > > Hello Paul and Steven,
> > > > > 

So I think this is another false positive, and the reason is we use
st->done for multiple purposes.

> > > > > This is saying:
> > > > > 
> > > > > Thread A
> > > > > --------
> > > > > takedown_cpu()
> > > > >    irq_lock_sparse()
> > > > >    wait_for_completion(&st->done) // Wait for completion of B

Thread A wait for the idle task on the outgoing to set the st->state to 
CPUHP_AP_IDLE_DEAD(i.e. the corresponding complete() is the one in
cpuhp_complete_idle_dead()), and it happens when we try to _offline_ a
cpu.

> > > > >    irq_unlock_sparse()
> > > > > 
> > > > > Thread B
> > > > > --------
> > > > > cpuhp_invoke_callback()
> > > > >    irq_lock_sparse() // Wait for A to irq_unlock_sparse()

irq_affinity_online_cpu() is called here, so it happens when we try to
_online_ a cpu.

> > > > >    (on the way going to complete(&st->done))

and we are going to complete(&st->done) in a hotplug thread context to
indicate the hotplug thread has finished its job(i.e. this complete() is
the one in cpuhp_thread_fun()).


So even though the &st->done are the same instance, the deadlock could
not happen, I think, as we could not up/down a same cpu at the same
time?

If I'm not missing something subtle. To fix this we can either

1)	have dedicated completion instances for different wait purposes
	in cpuhp_cpu_state.

or

2)	extend crossrelease to have the "subclass" concept, so that
	callsite of complete() and wait_for_completion() for the same
	completion instance but with different purposes could be
	differed by lockdep.

Thoughts?

Regards,
Boqun

> > > > > 
> > > > > So, lockdep consider this as a deadlock.
> > > > > Is it possible to happen?
> > > > 
> > > > In addition, if it's impossible, then we should fix lock class
> > > > assignments so that the locks actually have different classes.
> > > 
> > > Interesting, and thank you for the analysis!
> > > 
> > > The strange thing is that the way you describe it, this would be a
> > > deterministic deadlock.  Yet CPU hotplug operations complete just fine
> > > in my tests.  What am I missing here?
> > 
> > Hi, :)
> > 
> > Lockdep basically reports either (1) an actual deadlock happened at the
> > time or (2) a deadlock possibility, even w/o LOCKDEP_CROSSRELEASE.
> > 
> > Both are useful. But LOCKDEP_CROSSRELEASE can only do the latter. IOW,
> > the deadlock would actually happen _only_ when the two threads(A and B)
> > run simultaniously.
> > 
> > In your case, those two threads might run at different timings. So it's
> > not an actual deadlock, but still has a possibility for the problem to
> > happen later.
> 
> Fair enough, if the wakeup always happened first, deadlock might well
> be avoided.  If the sleep happened first, I suspect deadlock would
> be deterministic in this case.
> 
> 							Thanx, Paul
> 
> > > > > Thanks,
> > > > > Byungchul
> > > > > 
> > > > > > [   35.313943] 
> > > > > > [   35.313943] 3 locks held by torture_onoff/766:
> > > > > > [   35.313943]  #0:  (cpu_add_remove_lock){+.+.}, at: [<ffffffffb9060be2>] do_cpu_down+0x22/0x50
> > > > > > [   35.313943]  #1:  (cpu_hotplug_lock.rw_sem){++++}, at: [<ffffffffb90acc41>] percpu_down_write+0x21/0xf0
> > > > > > [   35.313943]  #2:  (sparse_irq_lock){+.+.}, at: [<ffffffffb90c5e42>] irq_lock_sparse+0x12/0x20
> > > > > > [   35.313943] 
> > > > > > [   35.313943] stack backtrace:
> > > > > > [   35.313943] CPU: 7 PID: 766 Comm: torture_onoff Not tainted 4.13.0-rc4+ #1
> > > > > > [   35.313943] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014
> > > > > > [   35.313943] Call Trace:
> > > > > > [   35.313943]  dump_stack+0x67/0x97
> > > > > > [   35.313943]  print_circular_bug+0x21d/0x330
> > > > > > [   35.313943]  ? add_lock_to_list.isra.31+0xc0/0xc0
> > > > > > [   35.313943]  check_prev_add+0x401/0x800
> > > > > > [   35.313943]  ? wake_up_q+0x70/0x70
> > > > > > [   35.313943]  __lock_acquire+0x1100/0x11a0
> > > > > > [   35.313943]  ? __lock_acquire+0x1100/0x11a0
> > > > > > [   35.313943]  ? add_lock_to_list.isra.31+0xc0/0xc0
> > > > > > [   35.313943]  lock_acquire+0x9e/0x1e0
> > > > > > [   35.313943]  ? takedown_cpu+0x86/0xf0
> > > > > > [   35.313943]  wait_for_completion+0x36/0x130
> > > > > > [   35.313943]  ? takedown_cpu+0x86/0xf0
> > > > > > [   35.313943]  ? stop_machine_cpuslocked+0xb9/0xd0
> > > > > > [   35.313943]  ? cpuhp_invoke_callback+0x8b0/0x8b0
> > > > > > [   35.313943]  ? cpuhp_complete_idle_dead+0x10/0x10
> > > > > > [   35.313943]  takedown_cpu+0x86/0xf0
> > > > > > [   35.313943]  cpuhp_invoke_callback+0xa7/0x8b0
> > > > > > [   35.313943]  cpuhp_down_callbacks+0x3d/0x80
> > > > > > [   35.313943]  _cpu_down+0xbb/0xf0
> > > > > > [   35.313943]  do_cpu_down+0x39/0x50
> > > > > > [   35.313943]  cpu_down+0xb/0x10
> > > > > > [   35.313943]  torture_offline+0x75/0x140
> > > > > > [   35.313943]  torture_onoff+0x102/0x1e0
> > > > > > [   35.313943]  kthread+0x142/0x180
> > > > > > [   35.313943]  ? torture_kthread_stopping+0x70/0x70
> > > > > > [   35.313943]  ? kthread_create_on_node+0x40/0x40
> > > > > > [   35.313943]  ret_from_fork+0x27/0x40
> > > > 
> > 
> 

Download attachment "signature.asc" of type "application/pgp-signature" (489 bytes)