netdev - Re: RE: RE: Re: [PATCH 09/11] drivers: infiniband: hw: Fix deadlock in irdma_cleanup_cm

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Message-ID: <7898f6c.4051b.180071605cc.Coremail.duoming@zju.edu.cn>
Date:   Fri, 8 Apr 2022 10:52:05 +0800 (GMT+08:00)
From:   duoming@....edu.cn
To:     "Saleem, Shiraz" <shiraz.saleem@...el.com>
Cc:     "Dan Carpenter" <dan.carpenter@...cle.com>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        "chris@...kel.net" <chris@...kel.net>,
        "jcmvbkbc@...il.com" <jcmvbkbc@...il.com>,
        "Ismail, Mustafa" <mustafa.ismail@...el.com>,
        "jgg@...pe.ca" <jgg@...pe.ca>,
        "wg@...ndegger.com" <wg@...ndegger.com>,
        "mkl@...gutronix.de" <mkl@...gutronix.de>,
        "davem@...emloft.net" <davem@...emloft.net>,
        "kuba@...nel.org" <kuba@...nel.org>,
        "pabeni@...hat.com" <pabeni@...hat.com>,
        "jes@...ined-monkey.org" <jes@...ined-monkey.org>,
        "gregkh@...uxfoundation.org" <gregkh@...uxfoundation.org>,
        "jirislaby@...nel.org" <jirislaby@...nel.org>,
        "alexander.deucher@....com" <alexander.deucher@....com>,
        "linux-xtensa@...ux-xtensa.org" <linux-xtensa@...ux-xtensa.org>,
        "linux-rdma@...r.kernel.org" <linux-rdma@...r.kernel.org>,
        "linux-can@...r.kernel.org" <linux-can@...r.kernel.org>,
        "netdev@...r.kernel.org" <netdev@...r.kernel.org>,
        "linux-hippi@...site.dk" <linux-hippi@...site.dk>,
        "linux-staging@...ts.linux.dev" <linux-staging@...ts.linux.dev>,
        "linux-serial@...r.kernel.org" <linux-serial@...r.kernel.org>,
        "linux-usb@...r.kernel.org" <linux-usb@...r.kernel.org>
Subject: Re: RE: RE: Re: [PATCH 09/11] drivers: infiniband: hw: Fix deadlock
 in irdma_cleanup_cm_core()

Hello,

On Fri, 8 Apr 2022 02:21:59 +0000 Saleem, Shiraz wrote:

> > > > > > > There is a deadlock in irdma_cleanup_cm_core(), which is shown
> > > > > > > below:
> > > > > > >
> > > > > > >    (Thread 1)              |      (Thread 2)
> > > > > > >                            | irdma_schedule_cm_timer()
> > > > > > > irdma_cleanup_cm_core()    |  add_timer()
> > > > > > >  spin_lock_irqsave() //(1) |  (wait a time)
> > > > > > >  ...                       | irdma_cm_timer_tick()
> > > > > > >  del_timer_sync()          |  spin_lock_irqsave() //(2)
> > > > > > >  (wait timer to stop)      |  ...
> > > > > > >
> > > > > > > We hold cm_core->ht_lock in position (1) of thread 1 and use
> > > > > > > del_timer_sync() to wait timer to stop, but timer handler also
> > > > > > > need cm_core->ht_lock in position (2) of thread 2.
> > > > > > > As a result, irdma_cleanup_cm_core() will block forever.
> > > > > > >
> > > > > > > This patch extracts del_timer_sync() from the protection of
> > > > > > > spin_lock_irqsave(), which could let timer handler to obtain
> > > > > > > the needed lock.
> > > > > > >
> > > > > > > Signed-off-by: Duoming Zhou <duoming@....edu.cn>
> > > > > > > ---
> > > > > > >  drivers/infiniband/hw/irdma/cm.c | 5 ++++-
> > > > > > >  1 file changed, 4 insertions(+), 1 deletion(-)
> > > > > > >
> > > > > > > diff --git a/drivers/infiniband/hw/irdma/cm.c
> > > > > > > b/drivers/infiniband/hw/irdma/cm.c
> > > > > > > index dedb3b7edd8..019dd8bfe08 100644
> > > > > > > --- a/drivers/infiniband/hw/irdma/cm.c
> > > > > > > +++ b/drivers/infiniband/hw/irdma/cm.c
> > > > > > > @@ -3252,8 +3252,11 @@ void irdma_cleanup_cm_core(struct
> > > > irdma_cm_core *cm_core)
> > > > > > >  		return;
> > > > > > >
> > > > > > >  	spin_lock_irqsave(&cm_core->ht_lock, flags);
> > > > > > > -	if (timer_pending(&cm_core->tcp_timer))
> > > > > > > +	if (timer_pending(&cm_core->tcp_timer)) {
> > > > > > > +		spin_unlock_irqrestore(&cm_core->ht_lock, flags);
> > > > > > >  		del_timer_sync(&cm_core->tcp_timer);
> > > > > > > +		spin_lock_irqsave(&cm_core->ht_lock, flags);
> > > > > > > +	}
> > > > > > >  	spin_unlock_irqrestore(&cm_core->ht_lock, flags);
> > > > > >
> > > > > > This lock doesn't seem to be protecting anything.  Also do we
> > > > > > need to check timer_pending()?  I think the del_timer_sync()
> > > > > > function will just return directly if there isn't a pending lock?
> > > > >
> > > > > Thanks a lot for your advice, I will remove the timer_pending()
> > > > > and the redundant lock.
> > > >
> > > > I didn't give any advice. :P I only ask questions when I don't know the answers.
> > > > Someone probably needs to look at &cm_core->ht_lock and figure out
> > > > what it's protecting.
> > > >
> > > Agreed on this fix.
> > >
> > > We should not lock around del_timer_sync or need to check on timer_pending.
> > >
> > > However, we do need serialize addition of a timer which can be called from
> > multiple paths, i.e. the timer handler and irdma_schedule_cm_timer.
> > 
> > I think we should replace the check "if (!timer_pending(&cm_core->tcp_timer))" to
> > "if (timer_pending(&cm_core->tcp_timer))" in irdma_cm_timer_tick(), and replace "if
> > (!was_timer_set)" to "if (was_timer_set)" in irdma_schedule_cm_timer() in order to
> > guarantee the timer could be executed. I will send the modified patch as soon as
> > possible.
> > 
> 
> No we don’t arm the timer if there's is one pending. Its also a bug to do so. 
> 
> https://elixir.bootlin.com/linux/v5.18-rc1/source/kernel/time/timer.c#L1143

You are right, I think we could add "mod_timer" in irdma_schedule_cm_timer and
irdma_cm_timer_tick() in order to start timer. 

I will send [PATCH V4 09/11] drivers: infiniband: hw: Fix deadlock in irdma_cleanup_cm_core().


Best regards,
Duoming Zhou