linux-kernel - Re: [RFC] Deadlock via recursive wakeup via RCU with threadirqs

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20190628192407.GA89956@google.com>
Date:   Fri, 28 Jun 2019 15:24:07 -0400
From:   Joel Fernandes <joel@...lfernandes.org>
To:     "Paul E. McKenney" <paulmck@...ux.ibm.com>
Cc:     Sebastian Andrzej Siewior <bigeasy@...utronix.de>,
        Peter Zijlstra <peterz@...radead.org>,
        Steven Rostedt <rostedt@...dmis.org>,
        rcu <rcu@...r.kernel.org>, LKML <linux-kernel@...r.kernel.org>,
        Thomas Gleixner <tglx@...utronix.de>,
        Ingo Molnar <mingo@...hat.com>,
        Josh Triplett <josh@...htriplett.org>,
        Mathieu Desnoyers <mathieu.desnoyers@...icios.com>,
        Lai Jiangshan <jiangshanlai@...il.com>
Subject: Re: [RFC] Deadlock via recursive wakeup via RCU with threadirqs

On Fri, Jun 28, 2019 at 11:52:19AM -0700, Paul E. McKenney wrote:
> On Fri, Jun 28, 2019 at 08:40:26PM +0200, Sebastian Andrzej Siewior wrote:
> > On 2019-06-28 08:30:50 [-0700], Paul E. McKenney wrote:
> > > On Fri, Jun 28, 2019 at 03:54:33PM +0200, Peter Zijlstra wrote:
> > > > On Thu, Jun 27, 2019 at 11:41:07AM -0700, Paul E. McKenney wrote:
> > > > > Or just don't do the wakeup at all, if it comes to that.  I don't know
> > > > > of any way to determine whether rcu_read_unlock() is being called from
> > > > > the scheduler, but it has been some time since I asked Peter Zijlstra
> > > > > about that.
> > > > 
> > > > There (still) is no 'in-scheduler' state.
> > > 
> > > Well, my TREE03 + threadirqs rcutorture test ran for ten hours last
> > > night with no problems, so we just might be OK.
> > > 
> > > The apparent fix is below, though my approach would be to do backports
> > > for the full set of related changes.
> > > 
> > > Joel, Sebastian, how goes any testing from your end?  Any reason
> > > to believe that this does not represent a fix?  (Me, I am still
> > > concerned about doing raise_softirq() from within a threaded
> > > interrupt, but am not seeing failures.)

Are you concerned also about a regular process context executing in the
scheduler and using RCU, having this issue?
(not anything with threaded or not threaded IRQs, but just a path in the
scheduler that uses RCU).

I don't think Sebastian's lock up has to do with the fact that an interrupt
is threaded or not, except that ksoftirqd is awakened in the case where
threadirqs is passed.

> > For some reason it does not trigger as good as it did yesterday.
> 
> I swear that I wasn't watching!!!  ;-)
> 
> But I do know that feeling.

:-)

> > Commit
> > - 23634ebc1d946 ("rcu: Check for wakeup-safe conditions in
> >    rcu_read_unlock_special()") does not trigger the bug within 94
> >    attempts.
> > 
> > - 48d07c04b4cc1 ("rcu: Enable elimination of Tree-RCU softirq
> >   processing") needed 12 attempts to trigger the bug.
> 
> That matches my belief that 23634ebc1d946 ("rcu: Check for wakeup-safe
> conditions in rcu_read_unlock_special()") will at least greatly decrease
> the probability of this bug occurring.

I was just typing a reply that I can't reproduce it with:
  rcu: Check for wakeup-safe conditions in rcu_read_unlock_special()

I am trying to revert enough of this patch to see what would break things,
however I think a better exercise might be to understand more what the patch
does why it fixes things in the first place ;-) It is probably the
deferred_qs thing.

thanks!