linux-kernel - RE: [PATCH 4.14 022/284] SUNRPC: avoid race between mod_timer() and del_timer

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <1c8cdc749fa34486abd435a15ec201fc@AcuMS.aculab.com>
Date:   Sun, 11 Sep 2022 12:43:12 +0000
From:   David Laight <David.Laight@...LAB.COM>
To:     'NeilBrown' <neilb@...e.de>, Eugeniu Rosca <erosca@...adit-jv.com>
CC:     Greg Kroah-Hartman <gregkh@...uxfoundation.org>,
        Trond Myklebust <trond.myklebust@...merspace.com>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        "stable@...r.kernel.org" <stable@...r.kernel.org>,
        Michael Rodin <mrodin@...adit-jv.com>,
        "Eugeniu Rosca" <erosca@...adit-jv.com>,
        Eugeniu Rosca <roscaeugeniu@...il.com>
Subject: RE: [PATCH 4.14 022/284] SUNRPC: avoid race between mod_timer() and
 del_timer_sync()

From: NeilBrown
> Sent: 08 September 2022 01:58
> 
> On Thu, 08 Sep 2022, Eugeniu Rosca wrote:
> > Hello all,
> >
> > On Mo, Apr 18, 2022 at 02:10:03 +0200, Greg Kroah-Hartman wrote:
> > > From: NeilBrown <neilb@...e.de>
> > >
> > > commit 3848e96edf4788f772d83990022fa7023a233d83 upstream.
> > >
> > > xprt_destory() claims XPRT_LOCKED and then calls del_timer_sync().
> > > Both xprt_unlock_connect() and xprt_release() call
> > >  ->release_xprt()
> > > which drops XPRT_LOCKED and *then* xprt_schedule_autodisconnect()
> > > which calls mod_timer().
> > >
> > > This may result in mod_timer() being called *after* del_timer_sync().
> > > When this happens, the timer may fire long after the xprt has been freed,
> > > and run_timer_softirq() will probably crash.
> > >
> > > The pairing of ->release_xprt() and xprt_schedule_autodisconnect() is
> > > always called under ->transport_lock.  So if we take ->transport_lock to
> > > call del_timer_sync(), we can be sure that mod_timer() will run first
> > > (if it runs at all).
> > >
> > > Cc: stable@...r.kernel.org
> > > Signed-off-by: NeilBrown <neilb@...e.de>
> > > Signed-off-by: Trond Myklebust <trond.myklebust@...merspace.com>
> > > Signed-off-by: Greg Kroah-Hartman <gregkh@...uxfoundation.org>
> > > ---
> > >  net/sunrpc/xprt.c |    7 +++++++
> > >  1 file changed, 7 insertions(+)
> > >
> > > --- a/net/sunrpc/xprt.c
> > > +++ b/net/sunrpc/xprt.c
> > > @@ -1520,7 +1520,14 @@ static void xprt_destroy(struct rpc_xprt
> > >  	 */
> > >  	wait_on_bit_lock(&xprt->state, XPRT_LOCKED, TASK_UNINTERRUPTIBLE);
> > >
> > > +	/*
> > > +	 * xprt_schedule_autodisconnect() can run after XPRT_LOCKED
> > > +	 * is cleared.  We use ->transport_lock to ensure the mod_timer()
> > > +	 * can only run *before* del_time_sync(), never after.
> > > +	 */
> > > +	spin_lock(&xprt->transport_lock);
> > >  	del_timer_sync(&xprt->timer);
> > > +	spin_unlock(&xprt->transport_lock);
> 
> I think it is sufficient to change the to spin_{,un}lock_bh()
> in older kernels.  The spinlock call need to match other uses of the
> same lock.

Every time I see this patch it looks wrong.
You need something to stop the code that is calling mod_timer()
running after the spin_unlock().
Now it might be that there is some other state that is
already set - in which case you only need to wait for the
spin_lock to be released - since it can't be obtained again
(to start the timer).

So I'd expect to see:
	spin_lock();
	if (nothing_set_earlier)
		xprt->destroying = 1;
	spin_unlock()
	del_timer_sync();

Looking at the code (for a change) is looks even worse.

del_timer_sync() isn't anywhere near enough.
All the timer callback function does is schedule some work.
So you also need to wait for the work to complete.

Changing it all to use delayed_work might reduce the problems.

Oh, any using proper mutex/locks instead of wait_on_bit_lock().

	David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)