Message-ID: <CAMArcTWFbDa5MAZ_iPHOr_jUh0=CurYod74x_2FxF=EAv28WiA@mail.gmail.com>
Date: Thu, 17 Apr 2025 00:01:57 +0900
From: Taehee Yoo <ap420073@...il.com>
To: Jakub Kicinski <kuba@...nel.org>
Cc: Stanislav Fomichev <stfomichev@...il.com>, Mina Almasry <almasrymina@...gle.com>, davem@...emloft.net, 
	pabeni@...hat.com, edumazet@...gle.com, andrew+netdev@...n.ch, 
	horms@...nel.org, asml.silence@...il.com, dw@...idwei.uk, sdf@...ichev.me, 
	skhawaja@...gle.com, simona.vetter@...ll.ch, kaiyuanz@...gle.com, 
	netdev@...r.kernel.org
Subject: Re: [PATCH net] net: devmem: fix kernel panic when socket close after
 module unload

On Wed, Apr 16, 2025 at 11:59 AM Jakub Kicinski <kuba@...nel.org> wrote:
>

Hi Mina, Stanislav and Jakub,
Thank you so much for the reviews!

> On Tue, 15 Apr 2025 11:59:40 -0700 Stanislav Fomichev wrote:
> > > commit 42f342387841 ("net: fix use-after-free in the
> > > netdev_nl_sock_priv_destroy()") and rolling back a few fixes, it's
> > > really introduced by commit 1d22d3060b9b ("net: drop rtnl_lock for
> > > queue_mgmt operations").

Yes, you're right.
The real fix would be commit 1d22d3060b9b
("net: drop rtnl_lock for queue_mgmt operations").

> > >
> > > My first question, does this issue still reproduce if you remove the
> > > per netdev locking and go back to relying on rtnl_locking? Or do we
> > > crash somewhere else in net_devmem_unbind_dmabuf? If so, where?
> > > Looking through the rest of the unbinding code, it's not clear to me
> > > any of it actually uses dev, so it may just be the locking...

Accessing binding->dev causes a crash after unreg.
I thought binding->dev is needed in the net_devmem_unbind_dmabuf(),
but it's not.
binding->dev is used only for locking, as you mentioned.

> >
> > A proper fix, most likely, will involve resetting binding->dev to NULL
> > when the device is going away.
>
> Right, tho a bit of work and tricky handling will be necessary to get
> that right. We're not holding a ref on binding->dev.
>
> I think we need to invert the socket mutex vs instance lock ordering.
> Make the priv mutex protect the binding->list and binding->dev.
> For that to work the binding needs to also store a pointer to its
> owning socket?
>
> Then in both uninstall paths (from socket and from netdev unreg) we can
> take the socket mutex, delete from list, clear the ->dev pointer,
> unlock, release the ref on the binding.
>
> The socket close path would probably need to lock the socket, look at
> the first entry, if entry has ->dev call netdev_hold(), release the
> socket, lock the netdev, lock the socket again, look at the ->dev, if
> NULL we raced - done. If not NULL release the socket, call unbind.
> netdev_put(). Restart this paragraph.
>
> I can't think of an easier way.

Thank you so much for a detailed guide :)
I tried what you suggested, then I tested cases A, B, and C.
I can't see any splats from lockdep, kasan, etc.
Also, I checked that bindings are released well by checking
/sys/kernel/debug/dma_buf/bufinfo.
I think this approach works well.
However, I tested this simply. So I'm not sure yet about race condition.
I need more tests targeting race condition.

I modified the locking order in the netdev_nl_bind_rx_doit().
And modified netdev_nl_sock_priv_destroy() code looks like:

void netdev_nl_sock_priv_destroy(struct netdev_nl_sock *priv)
{
        struct net_devmem_dmabuf_binding *binding;
        struct net_devmem_dmabuf_binding *temp;
        struct net_device *dev;

        mutex_lock(&priv->lock);
        list_for_each_entry_safe(binding, temp, &priv->bindings, list) {
                dev = binding->dev;
                if (dev) {
                        netdev_hold(dev, &priv->dev_tracker, GFP_KERNEL);
                        mutex_unlock(&priv->lock);
                        netdev_lock(dev);
                        mutex_lock(&priv->lock);
                        if (binding->dev)
                                net_devmem_unbind_dmabuf(binding);
                        mutex_unlock(&priv->lock);
                        netdev_unlock(dev);
                        netdev_put(dev, &priv->dev_tracker);
                        mutex_lock(&priv->lock);
                }
        }
        mutex_unlock(&priv->lock);
}

Also modified the uninstall code looks like:

static void mp_dmabuf_devmem_uninstall(void *mp_priv,
                                       struct netdev_rx_queue *rxq)
{
        struct net_devmem_dmabuf_binding *binding = mp_priv;
        struct netdev_rx_queue *bound_rxq;
        unsigned long xa_idx;

        mutex_lock(&binding->priv->lock);
        xa_for_each(&binding->bound_rxqs, xa_idx, bound_rxq) {
                if (bound_rxq == rxq) {
                        xa_erase(&binding->bound_rxqs, xa_idx);
                        if (xa_empty(&binding->bound_rxqs)) {
                                list_del(&binding->list);
                                binding->dev = NULL;
                                net_devmem_dmabuf_binding_put(binding);
                        }
                        break;
                }
        }
        mutex_unlock(&binding->priv->lock);
}

I think the uninstall code looks good to me, but
netdev_nl_sock_priv_destroy() is longer than I expected.
If this is okay with you, I would like to stabilize it with more tests.

>
> > Replacing rtnl with dev lock exposes the fact that we can't assume
> > that the binding->dev is still valid by the time we do unbind.
>
> Note that binding->dev is never accessed by net_devmem_unbind_dmabuf().
> So if the device was unregistered and its queues flushed, the only thing
> we touch the netdev pointer for is the instance lock :(
>

Powered by blists - more mailing lists