| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <6281cd92-10aa-4182-a456-81538cff822a@gmail.com>
Date: Thu, 11 Dec 2025 22:04:23 +0700
From: Bui Quang Minh <minhquangbui99@...il.com>
To: Jason Wang <jasowang@...hat.com>
Cc: netdev@...r.kernel.org, "Michael S. Tsirkin" <mst@...hat.com>,
Xuan Zhuo <xuanzhuo@...ux.alibaba.com>, Eugenio Pérez
<eperezma@...hat.com>, Andrew Lunn <andrew+netdev@...n.ch>,
"David S. Miller" <davem@...emloft.net>, Eric Dumazet <edumazet@...gle.com>,
Jakub Kicinski <kuba@...nel.org>, Paolo Abeni <pabeni@...hat.com>,
Alexei Starovoitov <ast@...nel.org>, Daniel Borkmann <daniel@...earbox.net>,
Jesper Dangaard Brouer <hawk@...nel.org>,
John Fastabend <john.fastabend@...il.com>,
Stanislav Fomichev <sdf@...ichev.me>, virtualization@...ts.linux.dev,
linux-kernel@...r.kernel.org, bpf@...r.kernel.org
Subject: Re: [PATCH net] virtio-net: enable all napis before scheduling refill
work
On 12/11/25 14:27, Jason Wang wrote:
> On Wed, Dec 10, 2025 at 11:33 PM Bui Quang Minh
> <minhquangbui99@...il.com> wrote:
>> On 12/10/25 12:45, Jason Wang wrote:
>>> On Tue, Dec 9, 2025 at 11:23 PM Bui Quang Minh <minhquangbui99@...il.com> wrote:
>>>> On 12/9/25 11:30, Jason Wang wrote:
>>>>> On Mon, Dec 8, 2025 at 11:35 PM Bui Quang Minh <minhquangbui99@...il.com> wrote:
>>>>>> Calling napi_disable() on an already disabled napi can cause the
>>>>>> deadlock. In commit 4bc12818b363 ("virtio-net: disable delayed refill
>>>>>> when pausing rx"), to avoid the deadlock, when pausing the RX in
>>>>>> virtnet_rx_pause[_all](), we disable and cancel the delayed refill work.
>>>>>> However, in the virtnet_rx_resume_all(), we enable the delayed refill
>>>>>> work too early before enabling all the receive queue napis.
>>>>>>
>>>>>> The deadlock can be reproduced by running
>>>>>> selftests/drivers/net/hw/xsk_reconfig.py with multiqueue virtio-net
>>>>>> device and inserting a cond_resched() inside the for loop in
>>>>>> virtnet_rx_resume_all() to increase the success rate. Because the worker
>>>>>> processing the delayed refilled work runs on the same CPU as
>>>>>> virtnet_rx_resume_all(), a reschedule is needed to cause the deadlock.
>>>>>> In real scenario, the contention on netdev_lock can cause the
>>>>>> reschedule.
>>>>>>
>>>>>> This fixes the deadlock by ensuring all receive queue's napis are
>>>>>> enabled before we enable the delayed refill work in
>>>>>> virtnet_rx_resume_all() and virtnet_open().
>>>>>>
>>>>>> Fixes: 4bc12818b363 ("virtio-net: disable delayed refill when pausing rx")
>>>>>> Reported-by: Paolo Abeni <pabeni@...hat.com>
>>>>>> Closes: https://netdev-ctrl.bots.linux.dev/logs/vmksft/drv-hw-dbg/results/400961/3-xdp-py/stderr
>>>>>> Signed-off-by: Bui Quang Minh <minhquangbui99@...il.com>
>>>>>> ---
>>>>>> drivers/net/virtio_net.c | 59 +++++++++++++++++++---------------------
>>>>>> 1 file changed, 28 insertions(+), 31 deletions(-)
>>>>>>
>>>>>> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
>>>>>> index 8e04adb57f52..f2b1ea65767d 100644
>>>>>> --- a/drivers/net/virtio_net.c
>>>>>> +++ b/drivers/net/virtio_net.c
>>>>>> @@ -2858,6 +2858,20 @@ static bool try_fill_recv(struct virtnet_info *vi, struct receive_queue *rq,
>>>>>> return err != -ENOMEM;
>>>>>> }
>>>>>>
>>>>>> +static void virtnet_rx_refill_all(struct virtnet_info *vi)
>>>>>> +{
>>>>>> + bool schedule_refill = false;
>>>>>> + int i;
>>>>>> +
>>>>>> + enable_delayed_refill(vi);
>>>>> This seems to be still racy?
>>>>>
>>>>> For example, in virtnet_open() we had:
>>>>>
>>>>> static int virtnet_open(struct net_device *dev)
>>>>> {
>>>>> struct virtnet_info *vi = netdev_priv(dev);
>>>>> int i, err;
>>>>>
>>>>> for (i = 0; i < vi->max_queue_pairs; i++) {
>>>>> err = virtnet_enable_queue_pair(vi, i);
>>>>> if (err < 0)
>>>>> goto err_enable_qp;
>>>>> }
>>>>>
>>>>> virtnet_rx_refill_all(vi);
>>>>>
>>>>> So NAPI and refill work is enabled in this case, so the refill work
>>>>> could be scheduled and run at the same time?
>>>> Yes, that's what we expect. We must ensure that refill work is scheduled
>>>> only when all NAPIs are enabled. The deadlock happens when refill work
>>>> is scheduled but there are still disabled RX NAPIs.
>>> Just to make sure we are on the same page, I meant, after refill work
>>> is enabled, rq0 is NAPI is enabled, in this case the refill work could
>>> be triggered by the rq0's NAPI so we may end up in the refill work
>>> that it tries to disable rq1's NAPI while holding the netdev lock.
>> I don't quite get your point. The current deadlock scenario is this
>>
>> virtnet_rx_resume_all
>> napi_enable(rq0) (the rq1 napi is still disabled)
>> enable_refill_work
>>
>> refill_work
>> napi_disable(rq0) -> still okay
>> napi_enable(rq0) -> still okay
>> napi_disable(rq1)
>> -> hold netdev_lock
>> -> stuck inside the while loop in napi_disable_locked
>> while (val & (NAPIF_STATE_SCHED | NAPIF_STATE_NPSVC)) {
>> usleep_range(20, 200);
>> val = READ_ONCE(n->state);
>> }
>>
>>
>> napi_enable(rq1)
>> -> stuck while trying to acquire the netdev_lock
>>
>> The problem is that we must not call napi_disable() on an already
>> disabled NAPI (rq1's NAPI in the example).
>>
>> In the new virtnet_open
>>
>> static int virtnet_open(struct net_device *dev)
>> {
>> struct virtnet_info *vi = netdev_priv(dev);
>> int i, err;
>>
>> // Note that at this point, refill work is still disabled, vi->refill_enabled == false,
>> // so even if virtnet_receive is called, the refill_work will not be scheduled.
>> for (i = 0; i < vi->max_queue_pairs; i++) {
>> err = virtnet_enable_queue_pair(vi, i);
>> if (err < 0)
>> goto err_enable_qp;
>> }
>>
>> // Here all RX NAPIs are enabled so it's safe to enable refill work again
>> virtnet_rx_refill_all(vi);
>>
> I meant this part:
>
> +static void virtnet_rx_refill_all(struct virtnet_info *vi)
> +{
> + bool schedule_refill = false;
> + int i;
> +
> + enable_delayed_refill(vi);
>
> refill_work could run here.
I don't see how this can trigger the current deadlock race. However, I
see that this code is racy, the try_fill_recv function is not safe to
concurrently executed on the same receive queue. So there is a
requirement that we need to call try_fill_recv before enabling napi. Is
it what you mean?
>
> + for (i = 0; i < vi->curr_queue_pairs; i++)
> + if (!try_fill_recv(vi, &vi->rq[i], GFP_KERNEL))
> + schedule_refill = true;
> +
>
> I think it can be fixed by moving enable_delayed_refill() here.
>
> + if (schedule_refill)
> + schedule_delayed_work(&vi->refill, 0);
> +}
Thanks,
Quang Minh.
Powered by blists - more mailing lists