linux-kernel - Re: [PATCH net v3] net: sched: fix packet stuck problem for lockless qdisc

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [day] [month] [year] [list]

Message-ID: <caadc833-ee47-8243-a238-57595e7fc446@huawei.com>
Date:   Mon, 12 Apr 2021 11:37:24 +0800
From:   Yunsheng Lin <linyunsheng@...wei.com>
To:     Hillf Danton <hdanton@...a.com>
CC:     Juergen Gross <jgross@...e.com>, <netdev@...r.kernel.org>,
        <linux-kernel@...r.kernel.org>, Jiri Kosina <JKosina@...e.com>
Subject: Re: [PATCH net v3] net: sched: fix packet stuck problem for lockless
 qdisc

On 2021/4/12 11:21, Hillf Danton wrote:
> On Mon, 12 Apr 2021 09:24:30  Yunsheng Lin wrote:
>> On 2021/4/9 17:09, Hillf Danton wrote:
>>> On Fri, 9 Apr 2021 07:31:03  Juergen Gross wrote:
>>>> On 25.03.21 04:13, Yunsheng Lin wrote:
>>>> I have a setup which is able to reproduce the issue quite reliably:
>>>>
>>>> In a Xen guest I'm mounting 8 NFS shares and run sysbench fileio on
>>>> each of them. The average latency reported by sysbench is well below
>>>> 1 msec, but at least once per hour I get latencies in the minute
>>>> range.
>>>>
>>>> With this patch I don't see these high latencies any longer (test
>>>> is running for more than 20 hours now).
>>>>
>>>> So you can add my:
>>>>
>>>> Tested-by: Juergen Gross <jgross@...e.com>
>>>>
>>>
>>> If retry is allowed in the dequeue method then a simple seqcount can do the
>>> work of serializing enqueuer and dequeuer. IIUC it was not attempted last year.
>>
>> At the first glance, I do not think the below patch fix the data race
> 
> Thanks for taking a look.
> 
>> described in the commit log, as it does not handle the time window
>> between dequeuing and q->seqlock releasing, as below:
>>
> Yes the time window does exist.
> 
>> The cpu1 may not see the qdisc->pad changed after pfifo_fast_dequeue(),
>> and cpu2 is not able to take the q->seqlock yet because cpu1 do not
>> release the q->seqlock.
>>
> It's now covered by extending the seqcount aperture a bit.
> 
> --- x/net/sched/sch_generic.c
> +++ y/net/sched/sch_generic.c
> @@ -380,14 +380,23 @@ void __qdisc_run(struct Qdisc *q)
>  {
>  	int quota = dev_tx_weight;
>  	int packets;
> +	int seq;
> +
> +again:
> +	seq = READ_ONCE(q->pad);
> +	smp_rmb();
>  
>  	while (qdisc_restart(q, &packets)) {
>  		quota -= packets;
>  		if (quota <= 0) {
>  			__netif_schedule(q);
> -			break;
> +			return;
>  		}
>  	}
> +
> +	smp_rmb();
> +	if (seq != READ_ONCE(q->pad))
> +		goto again;

As my understanding, there is still time window between q->pad checking
above and q->seqlock releasing in qdisc_run_end().

>  }
>  
>  unsigned long dev_trans_start(struct net_device *dev)
> @@ -632,6 +641,9 @@ static int pfifo_fast_enqueue(struct sk_
>  			return qdisc_drop(skb, qdisc, to_free);
>  	}
>  
> +	qdisc->pad++;
> +	smp_wmb();
> +
>  	qdisc_update_stats_at_enqueue(qdisc, pkt_len);
>  	return NET_XMIT_SUCCESS;
>  }
> 
> .
>