| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Date: Wed, 5 Jul 2023 06:20:26 +0000
From: Wei Fang <wei.fang@....com>
To: Andrew Lunn <andrew@...n.ch>
CC: "davem@...emloft.net" <davem@...emloft.net>,
"edumazet@...gle.com" <edumazet@...gle.com>,
"kuba@...nel.org" <kuba@...nel.org>,
"pabeni@...hat.com" <pabeni@...hat.com>,
"ast@...nel.org" <ast@...nel.org>,
"daniel@...earbox.net" <daniel@...earbox.net>,
"hawk@...nel.org" <hawk@...nel.org>,
"john.fastabend@...il.com" <john.fastabend@...il.com>,
Shenwei Wang <shenwei.wang@....com>,
Clark Wang <xiaoning.wang@....com>,
"netdev@...r.kernel.org" <netdev@...r.kernel.org>,
dl-linux-imx <linux-imx@....com>,
"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
"bpf@...r.kernel.org" <bpf@...r.kernel.org>
Subject: RE: [PATCH net 3/3] net: fec: increase the size of tx ring and update
thresholds of tx ring
> -----Original Message-----
> From: Andrew Lunn <andrew@...n.ch>
> Sent: 2023年7月5日 8:25
> To: Wei Fang <wei.fang@....com>
> Cc: davem@...emloft.net; edumazet@...gle.com; kuba@...nel.org;
> pabeni@...hat.com; ast@...nel.org; daniel@...earbox.net;
> hawk@...nel.org; john.fastabend@...il.com; Shenwei Wang
> <shenwei.wang@....com>; Clark Wang <xiaoning.wang@....com>;
> netdev@...r.kernel.org; dl-linux-imx <linux-imx@....com>;
> linux-kernel@...r.kernel.org; bpf@...r.kernel.org
> Subject: Re: [PATCH net 3/3] net: fec: increase the size of tx ring and update
> thresholds of tx ring
>
> > After trying various methods, I think that increase the size of tx BD
> > ring is simple and effective. Maybe the best resolution is that
> > allocate NAPI for each queue to improve the efficiency of the NAPI
> > callback, but this change is a bit big and I didn't try this method.
> > Perheps this method will be implemented in a future patch.
>
> How does this affect platforms like Vybrid with its fast Ethernet?
Sorry, I don't have the Vybrid platform, but I think I don't think it has much
impact, at most it just takes up some more memory.
> Does the burst latency go up?
No, for fec, when a packet is attached to the BDs, the software will immediately
trigger the hardware to send the packet. In addition, I think it may improve the
latency, because the size of the tx ring becomes larger, and more packets can be
attached to the BD ring for burst traffic.
>
> > In addtion, this patch also updates the tx_stop_threshold and the
> > tx_wake_threshold of the tx ring. In previous logic, the value of
> > tx_stop_threshold is 217, however, the value of tx_wake_threshold is
> > 147, it does not make sense that tx_wake_threshold is less than
> > tx_stop_threshold.
>
> What do these actually mean? I could imagine that as the ring fills you don't
> want to stop until it is 217/512 full. There is then some hysteresis, such that it
> has to drop below 147/512 before more can be added?
>
You must have misunderstood, let me explain more clearly, the queue will be
stopped when the available BDs are less than tx_stop_threshold (217 BDs). And
the queue will be waked when the available BDs are greater than tx_wake_threshold
(147 BDs). So in most cases, the available BDs are greater than tx_wake_threshold
when the queue is stopped, the only effect is to delay packet sending.
In my opinion, tx_wake_threshold should be greater than tx_stop_threshold, we
should stop queue when the available BDs are not enough for a skb to be attached.
And wake the queue when the available BDs are sufficient for a skb.
> > Besides, both XDP path and 'slow path' share the tx BD rings. So if
> > tx_stop_threshold is 217, in the case of heavy XDP traffic, the slow
> > path is easily to be stopped, this will have a serious impact on the
> > slow path.
>
> Please post your iperf results for various platforms, so we can see the effects of
> this. We generally don't accept tuning patches without benchmarks which
> prove the improvements, and also show there is no regression. And given the
> wide variety of SoCs using the FEC, i expect testing on a number of SoCs, but
> Fast and 1G.
>
Below are the results on i.MX6UL/8MM/8MP/8ULP/93 platforms, i.MX6UL and
8ULP only support Fast ethernet. Others support 1G.
1.1 i.MX6UL with tx ring size 512
root@...6ul7d:~# iperf3 -c 192.168.3.6
Connecting to host 192.168.3.6, port 5201
[ 5] local 192.168.3.9 port 47148 connected to 192.168.3.6 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 10.1 MBytes 84.9 Mbits/sec 0 103 KBytes
[ 5] 1.00-2.00 sec 9.88 MBytes 82.6 Mbits/sec 0 103 KBytes
[ 5] 2.00-3.00 sec 9.82 MBytes 82.7 Mbits/sec 0 103 KBytes
[ 5] 3.00-4.00 sec 9.82 MBytes 82.4 Mbits/sec 0 103 KBytes
[ 5] 4.00-5.00 sec 9.88 MBytes 82.9 Mbits/sec 0 103 KBytes
[ 5] 5.00-6.00 sec 9.94 MBytes 83.4 Mbits/sec 0 103 KBytes
[ 5] 6.00-7.00 sec 10.1 MBytes 84.3 Mbits/sec 0 103 KBytes
[ 5] 7.00-8.00 sec 9.82 MBytes 82.4 Mbits/sec 0 103 KBytes
[ 5] 8.00-9.00 sec 9.82 MBytes 82.4 Mbits/sec 0 103 KBytes
[ 5] 9.00-10.00 sec 9.88 MBytes 82.9 Mbits/sec 0 103 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 99.0 MBytes 83.1 Mbits/sec 0 sender
[ 5] 0.00-10.01 sec 98.8 MBytes 82.8 Mbits/sec receiver
1.2 i.MX6UL with tx ring size 1024
root@...6ul7d:~# iperf3 -c 192.168.3.6
Connecting to host 192.168.3.6, port 5201
[ 5] local 192.168.3.9 port 55670 connected to 192.168.3.6 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 10.2 MBytes 85.4 Mbits/sec 0 236 KBytes
[ 5] 1.00-2.00 sec 10.1 MBytes 84.6 Mbits/sec 0 236 KBytes
[ 5] 2.00-3.00 sec 10.2 MBytes 85.5 Mbits/sec 0 249 KBytes
[ 5] 3.00-4.00 sec 10.1 MBytes 85.1 Mbits/sec 0 249 KBytes
[ 5] 4.00-5.00 sec 10.1 MBytes 84.7 Mbits/sec 0 249 KBytes
[ 5] 5.00-6.00 sec 10.0 MBytes 84.1 Mbits/sec 0 249 KBytes
[ 5] 6.00-7.00 sec 10.1 MBytes 85.1 Mbits/sec 0 249 KBytes
[ 5] 7.00-8.00 sec 10.1 MBytes 84.9 Mbits/sec 0 249 KBytes
[ 5] 8.00-9.00 sec 10.3 MBytes 85.9 Mbits/sec 0 249 KBytes
[ 5] 9.00-10.00 sec 10.2 MBytes 85.6 Mbits/sec 0 249 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 101 MBytes 85.1 Mbits/sec 0 sender
[ 5] 0.00-10.01 sec 101 MBytes 84.5 Mbits/sec receiver
2.1 i.MX8ULP with tx ring size 512
root@...8ulpevk:~# iperf3 -c 192.168.3.6
Connecting to host 192.168.3.6, port 5201
[ 5] local 192.168.3.9 port 54342 connected to 192.168.3.6 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 10.8 MBytes 90.3 Mbits/sec 0 99.0 KBytes
[ 5] 1.00-2.00 sec 9.94 MBytes 83.4 Mbits/sec 0 99.0 KBytes
[ 5] 2.00-3.00 sec 10.2 MBytes 85.5 Mbits/sec 0 99.0 KBytes
[ 5] 3.00-4.00 sec 9.94 MBytes 83.4 Mbits/sec 0 99.0 KBytes
[ 5] 4.00-5.00 sec 10.2 MBytes 85.5 Mbits/sec 0 99.0 KBytes
[ 5] 5.00-6.00 sec 9.94 MBytes 83.4 Mbits/sec 0 99.0 KBytes
[ 5] 6.00-7.00 sec 9.69 MBytes 81.3 Mbits/sec 0 99.0 KBytes
[ 5] 7.00-8.00 sec 9.94 MBytes 83.4 Mbits/sec 0 99.0 KBytes
[ 5] 8.00-9.00 sec 9.69 MBytes 81.3 Mbits/sec 0 99.0 KBytes
[ 5] 9.00-10.00 sec 10.2 MBytes 85.5 Mbits/sec 0 99.0 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 100 MBytes 84.3 Mbits/sec 0 sender
[ 5] 0.00-9.90 sec 100 MBytes 84.7 Mbits/sec receiver
2.1 i.MX8ULP with tx ring size 1024
root@...8ulpevk:~# iperf3 -c 192.168.3.6
Connecting to host 192.168.3.6, port 5201
[ 5] local 192.168.3.9 port 44770 connected to 192.168.3.6 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 10.7 MBytes 90.1 Mbits/sec 0 93.3 KBytes
[ 5] 1.00-2.00 sec 9.94 MBytes 83.4 Mbits/sec 0 93.3 KBytes
[ 5] 2.00-3.00 sec 10.2 MBytes 85.5 Mbits/sec 0 93.3 KBytes
[ 5] 3.00-4.00 sec 10.1 MBytes 85.0 Mbits/sec 0 93.3 KBytes
[ 5] 4.00-5.00 sec 9.94 MBytes 83.4 Mbits/sec 0 100 KBytes
[ 5] 5.00-6.00 sec 10.2 MBytes 85.5 Mbits/sec 0 100 KBytes
[ 5] 6.00-7.00 sec 9.69 MBytes 81.3 Mbits/sec 0 100 KBytes
[ 5] 7.00-8.00 sec 9.94 MBytes 83.4 Mbits/sec 0 100 KBytes
[ 5] 8.00-9.00 sec 10.2 MBytes 85.5 Mbits/sec 0 100 KBytes
[ 5] 9.00-10.00 sec 9.69 MBytes 81.3 Mbits/sec 0 100 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 101 MBytes 84.4 Mbits/sec 0 sender
[ 5] 0.00-9.92 sec 100 MBytes 84.8 Mbits/sec receiver
3.1 i.MX8MM with tx ring size 512
root@...8mmevk:~# iperf3 -c 192.168.3.6
Connecting to host 192.168.3.6, port 5201
[ 5] local 192.168.3.9 port 55734 connected to 192.168.3.6 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 111 MBytes 934 Mbits/sec 0 577 KBytes
[ 5] 1.00-2.00 sec 112 MBytes 937 Mbits/sec 0 577 KBytes
[ 5] 2.00-3.00 sec 112 MBytes 942 Mbits/sec 0 609 KBytes
[ 5] 3.00-4.00 sec 113 MBytes 945 Mbits/sec 0 638 KBytes
[ 5] 4.00-5.00 sec 112 MBytes 941 Mbits/sec 0 638 KBytes
[ 5] 5.00-6.00 sec 112 MBytes 942 Mbits/sec 0 638 KBytes
[ 5] 6.00-7.00 sec 112 MBytes 942 Mbits/sec 0 638 KBytes
[ 5] 7.00-8.00 sec 112 MBytes 943 Mbits/sec 0 638 KBytes
[ 5] 8.00-9.00 sec 112 MBytes 943 Mbits/sec 0 638 KBytes
[ 5] 9.00-10.00 sec 112 MBytes 942 Mbits/sec 0 638 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.10 GBytes 941 Mbits/sec 0 sender
[ 5] 0.00-10.00 sec 1.09 GBytes 938 Mbits/sec receiver
3.2 i.MX8MM with tx ring size 1024
root@...8mmevk:~# iperf3 -c 192.168.3.6
Connecting to host 192.168.3.6, port 5201
[ 5] local 192.168.3.9 port 53350 connected to 192.168.3.6 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 114 MBytes 952 Mbits/sec 0 585 KBytes
[ 5] 1.00-2.00 sec 112 MBytes 942 Mbits/sec 0 585 KBytes
[ 5] 2.00-3.00 sec 113 MBytes 947 Mbits/sec 0 585 KBytes
[ 5] 3.00-4.00 sec 112 MBytes 940 Mbits/sec 0 648 KBytes
[ 5] 4.00-5.00 sec 112 MBytes 944 Mbits/sec 0 648 KBytes
[ 5] 5.00-6.00 sec 112 MBytes 944 Mbits/sec 0 648 KBytes
[ 5] 6.00-7.00 sec 111 MBytes 933 Mbits/sec 0 648 KBytes
[ 5] 7.00-8.00 sec 112 MBytes 944 Mbits/sec 0 648 KBytes
[ 5] 8.00-9.00 sec 112 MBytes 944 Mbits/sec 0 648 KBytes
[ 5] 9.00-10.00 sec 112 MBytes 944 Mbits/sec 0 648 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.10 GBytes 943 Mbits/sec 0 sender
[ 5] 0.00-10.00 sec 1.09 GBytes 940 Mbits/sec receiver
4.1 i.MX8MP with tx ring size 512
root@...8mpevk:~# iperf3 -c 192.168.3.6
Connecting to host 192.168.3.6, port 5201
[ 5] local 192.168.3.9 port 51892 connected to 192.168.3.6 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 114 MBytes 959 Mbits/sec 0 594 KBytes
[ 5] 1.00-2.00 sec 112 MBytes 940 Mbits/sec 0 626 KBytes
[ 5] 2.00-3.00 sec 113 MBytes 946 Mbits/sec 0 626 KBytes
[ 5] 3.00-4.00 sec 112 MBytes 937 Mbits/sec 0 626 KBytes
[ 5] 4.00-5.00 sec 112 MBytes 940 Mbits/sec 0 626 KBytes
[ 5] 5.00-6.00 sec 112 MBytes 940 Mbits/sec 0 626 KBytes
[ 5] 6.00-7.00 sec 113 MBytes 946 Mbits/sec 0 626 KBytes
[ 5] 7.00-8.00 sec 112 MBytes 939 Mbits/sec 0 626 KBytes
[ 5] 8.00-9.00 sec 111 MBytes 935 Mbits/sec 0 626 KBytes
[ 5] 9.00-10.00 sec 112 MBytes 943 Mbits/sec 0 626 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.10 GBytes 943 Mbits/sec 0 sender
[ 5] 0.00-10.00 sec 1.09 GBytes 940 Mbits/sec receiver
4.2 i.MX8MP with tx ring size 1024
root@...8mpevk:~# iperf3 -c 192.168.3.6
Connecting to host 192.168.3.6, port 5201
[ 5] local 192.168.3.9 port 37922 connected to 192.168.3.6 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 113 MBytes 951 Mbits/sec 0 608 KBytes
[ 5] 1.00-2.00 sec 112 MBytes 937 Mbits/sec 0 608 KBytes
[ 5] 2.00-3.00 sec 113 MBytes 947 Mbits/sec 0 608 KBytes
[ 5] 3.00-4.00 sec 111 MBytes 934 Mbits/sec 0 608 KBytes
[ 5] 4.00-5.00 sec 112 MBytes 942 Mbits/sec 0 608 KBytes
[ 5] 5.00-6.00 sec 112 MBytes 939 Mbits/sec 0 608 KBytes
[ 5] 6.00-7.00 sec 113 MBytes 949 Mbits/sec 0 608 KBytes
[ 5] 7.00-8.00 sec 112 MBytes 942 Mbits/sec 0 608 KBytes
[ 5] 8.00-9.00 sec 112 MBytes 936 Mbits/sec 0 608 KBytes
[ 5] 9.00-10.00 sec 112 MBytes 942 Mbits/sec 0 608 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.10 GBytes 942 Mbits/sec 0 sender
[ 5] 0.00-10.00 sec 1.09 GBytes 939 Mbits/sec receiver
5.1 i.MX93 with tx ring size 512
root@...93evk:~# iperf3 -c 192.168.3.6
Connecting to host 192.168.3.6, port 5201
[ 5] local 192.168.3.9 port 44216 connected to 192.168.3.6 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 115 MBytes 965 Mbits/sec 0 656 KBytes
[ 5] 1.00-2.00 sec 111 MBytes 934 Mbits/sec 0 656 KBytes
[ 5] 2.00-3.00 sec 112 MBytes 944 Mbits/sec 0 656 KBytes
[ 5] 3.00-4.00 sec 112 MBytes 944 Mbits/sec 0 656 KBytes
[ 5] 4.00-5.00 sec 112 MBytes 944 Mbits/sec 0 656 KBytes
[ 5] 5.00-6.00 sec 111 MBytes 933 Mbits/sec 0 656 KBytes
[ 5] 6.00-7.00 sec 112 MBytes 944 Mbits/sec 0 656 KBytes
[ 5] 7.00-8.00 sec 112 MBytes 944 Mbits/sec 0 656 KBytes
[ 5] 8.00-9.00 sec 112 MBytes 944 Mbits/sec 0 656 KBytes
[ 5] 9.00-10.00 sec 112 MBytes 944 Mbits/sec 0 656 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.10 GBytes 944 Mbits/sec 0 sender
[ 5] 0.00-10.00 sec 1.10 GBytes 941 Mbits/sec receiver
5.2 i.MX93 with tx ring size 1024
root@...93evk:~# iperf3 -c 192.168.3.6
Connecting to host 192.168.3.6, port 5201
[ 5] local 192.168.3.9 port 51058 connected to 192.168.3.6 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 114 MBytes 959 Mbits/sec 0 588 KBytes
[ 5] 1.00-2.00 sec 112 MBytes 935 Mbits/sec 0 649 KBytes
[ 5] 2.00-3.00 sec 112 MBytes 944 Mbits/sec 0 649 KBytes
[ 5] 3.00-4.00 sec 112 MBytes 944 Mbits/sec 0 649 KBytes
[ 5] 4.00-5.00 sec 112 MBytes 944 Mbits/sec 0 649 KBytes
[ 5] 5.00-6.00 sec 112 MBytes 944 Mbits/sec 0 649 KBytes
[ 5] 6.00-7.00 sec 111 MBytes 933 Mbits/sec 0 649 KBytes
[ 5] 7.00-8.00 sec 112 MBytes 944 Mbits/sec 0 649 KBytes
[ 5] 8.00-9.00 sec 112 MBytes 944 Mbits/sec 0 649 KBytes
[ 5] 9.00-10.00 sec 112 MBytes 944 Mbits/sec 0 649 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.10 GBytes 943 Mbits/sec 0 sender
[ 5] 0.00-10.00 sec 1.10 GBytes 940 Mbits/sec receiver
Powered by blists - more mailing lists