| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <a94c04be-56eb-0299-8cf3-0c2c34bf8d33@gmail.com>
Date: Fri, 5 Apr 2019 07:48:13 +0200
From: Rafał Miłecki <zajec5@...il.com>
To: Toshiaki Makita <makita.toshiaki@....ntt.co.jp>
Cc: Toshiaki Makita <toshiaki.makita1@...il.com>,
netdev@...r.kernel.org, "David S. Miller" <davem@...emloft.net>,
Stefano Brivio <sbrivio@...hat.com>,
Sabrina Dubroca <sd@...asysnail.net>,
David Ahern <dsahern@...il.com>, Felix Fietkau <nbd@....name>,
Jo-Philipp Wich <jo@...n.io>,
Koen Vandeputte <koen.vandeputte@...ntric.com>
Subject: Re: NAT performance regression caused by vlan GRO support
On 05.04.2019 06:26, Toshiaki Makita wrote:
> On 2019/04/05 5:22, Rafał Miłecki wrote:
>> On 04.04.2019 17:17, Toshiaki Makita wrote:
>>> On 19/04/04 (木) 21:57:15, Rafał Miłecki wrote:
>>>> I'd like to report a regression that goes back to the 2015. I know
>>>> it's damn
>>>> late, but the good thing is, the regression is still easy to
>>>> reproduce, verify &
>>>> revert.
>>>>
>>>> Long story short, starting with the commit 66e5133f19e9 ("vlan: Add
>>>> GRO support
>>>> for non hardware accelerated vlan") - which first hit kernel 4.2 - NAT
>>>> performance of my router dropped by 30% - 40%.
>>>>
>>>> My hardware is BCM47094 SoC (dual core ARM) with integrated network
>>>> controller
>>>> and external BCM53012 switch.
>>>>
>>>> Relevant setup:
>>>> * SoC network controller is wired to the hardware switch
>>>> * Switch passes 802.1q frames with VID 1 to four LAN ports
>>>> * Switch passes 802.1q frames with VID 2 to WAN port
>>>> * Linux does NAT for LAN (eth0.1) to WAN (eth0.2)
>>>> * Linux uses pfifo and "echo 2 > rps_cpus"
>>>> * Ryzen 5 PRO 2500U (x86_64) laptop connected to a LAN port
>>>> * Intel i7-2670QM laptop connected to a WAN port
>>>> * Speed of LAN to WAN measured using iperf & TCP over 10 minutes
>>>>
>>>> 1) 5.1.0-rc3
>>>> [ 6] 0.0-600.0 sec 39.9 GBytes 572 Mbits/sec
>>>>
>>>> 2) 5.1.0-rc3 + rtcache patch
>>>> [ 6] 0.0-600.0 sec 40.0 GBytes 572 Mbits/sec
>>>>
>>>> 3) 5.1.0-rc3 + disable GRO support
>>>> [ 6] 0.0-300.4 sec 27.5 GBytes 786 Mbits/sec
>>>>
>>>> 4) 5.1.0-rc3 + rtcache patch + disable GRO support
>>>> [ 6] 0.0-600.0 sec 65.6 GBytes 939 Mbits/sec
>>>
>>> Did you test it with disabling GRO by ethtool -K?
>>
>> Oh, I didn't know about such possibility! I just tested:
>> 1) Kernel with GRO support left in place (no local patch disabling it)
>> 2) ethtool -K eth0 gro off
>> and it bumped my NAT performance from 576 Mb/s to 939 Mb/s. I can reliably
>> break/fix NAT performance by just calling ethtool -K eth0 gro on/off.
>>
>>
>>> Is this the result with your reverting patch?
>>
>> Previous results were coming from kernel with patched
>> vlan_offload_init() - see
>> diff at the end of my first e-mail.
>>
>>
>>> It's late night in Japan so I think I will try to reproduce it tomorrow.
>
> My test results:
>
> Receiving packets from eth0.10, forwarding them to eth0.20 and applying
> MASQUERADE on eth0.20, using i40e 25G NIC on kernel 4.20.13.
> Disabled rxvlan by ethtool -K to exercise vlan_gro_receive().
> Measured TCP throughput by netperf.
>
> GRO on : 17 Gbps
> GRO off: 5 Gbps
>
> So I failed to reproduce your problem.
:( Thanks for trying & checking that!
> Would you check the CPU usage by "mpstat -P ALL" or similar (like "sar
> -u ALL -P ALL") to check if the traffic is able to consume 100% CPU on
> your machine?
1) ethtool -K eth0 gro on + iperf running (577 Mb/s)
root@...nWrt:/# mpstat -P ALL 10 3
Linux 5.1.0-rc3+ (OpenWrt) 03/27/19 _armv7l_ (2 CPU)
16:33:40 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
16:33:50 all 0.00 0.00 0.00 0.00 0.00 58.79 0.00 0.00 41.21
16:33:50 0 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00
16:33:50 1 0.00 0.00 0.00 0.00 0.00 17.58 0.00 0.00 82.42
16:33:50 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
16:34:00 all 0.00 0.00 0.05 0.00 0.00 59.44 0.00 0.00 40.51
16:34:00 0 0.00 0.00 0.10 0.00 0.00 99.90 0.00 0.00 0.00
16:34:00 1 0.00 0.00 0.00 0.00 0.00 18.98 0.00 0.00 81.02
16:34:00 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
16:34:10 all 0.00 0.00 0.00 0.00 0.00 59.59 0.00 0.00 40.41
16:34:10 0 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00
16:34:10 1 0.00 0.00 0.00 0.00 0.00 19.18 0.00 0.00 80.82
Average: CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
Average: all 0.00 0.00 0.02 0.00 0.00 59.27 0.00 0.00 40.71
Average: 0 0.00 0.00 0.03 0.00 0.00 99.97 0.00 0.00 0.00
Average: 1 0.00 0.00 0.00 0.00 0.00 18.58 0.00 0.00 81.42
2) ethtool -K eth0 gro off + iperf running (941 Mb/s)
root@...nWrt:/# mpstat -P ALL 10 3
Linux 5.1.0-rc3+ (OpenWrt) 03/27/19 _armv7l_ (2 CPU)
16:34:39 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
16:34:49 all 0.00 0.00 0.05 0.00 0.00 86.91 0.00 0.00 13.04
16:34:49 0 0.00 0.00 0.10 0.00 0.00 78.22 0.00 0.00 21.68
16:34:49 1 0.00 0.00 0.00 0.00 0.00 95.60 0.00 0.00 4.40
16:34:49 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
16:34:59 all 0.00 0.00 0.10 0.00 0.00 87.06 0.00 0.00 12.84
16:34:59 0 0.00 0.00 0.20 0.00 0.00 79.72 0.00 0.00 20.08
16:34:59 1 0.00 0.00 0.00 0.00 0.00 94.41 0.00 0.00 5.59
16:34:59 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
16:35:09 all 0.00 0.00 0.05 0.00 0.00 85.71 0.00 0.00 14.24
16:35:09 0 0.00 0.00 0.10 0.00 0.00 79.42 0.00 0.00 20.48
16:35:09 1 0.00 0.00 0.00 0.00 0.00 92.01 0.00 0.00 7.99
Average: CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
Average: all 0.00 0.00 0.07 0.00 0.00 86.56 0.00 0.00 13.37
Average: 0 0.00 0.00 0.13 0.00 0.00 79.12 0.00 0.00 20.75
Average: 1 0.00 0.00 0.00 0.00 0.00 94.01 0.00 0.00 5.99
3) System idle (no iperf)
root@...nWrt:/# mpstat -P ALL 10 1
Linux 5.1.0-rc3+ (OpenWrt) 03/27/19 _armv7l_ (2 CPU)
16:35:31 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
16:35:41 all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
16:35:41 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
16:35:41 1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
Average: CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
Average: all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
Average: 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
Average: 1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
> If CPU is 100%, perf may help us analyze your problem. If it's
> available, try running below while testing:
> # perf record -a -g -- sleep 5
>
> And then run this after testing:
> # perf report --no-child
I can see my CPU 0 is fully loaded when using "gro on". I'll try perf now.
Powered by blists - more mailing lists