| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <87mul421sg.fsf@toke.dk>
Date: Fri, 05 Apr 2019 12:18:23 +0200
From: Toke Høiland-Jørgensen <toke@...hat.com>
To: Toshiaki Makita <makita.toshiaki@....ntt.co.jp>,
Felix Fietkau <nbd@....name>,
Rafał Miłecki <zajec5@...il.com>
Cc: Toshiaki Makita <toshiaki.makita1@...il.com>,
netdev@...r.kernel.org, "David S. Miller" <davem@...emloft.net>,
Stefano Brivio <sbrivio@...hat.com>,
Sabrina Dubroca <sd@...asysnail.net>,
David Ahern <dsahern@...il.com>, Jo-Philipp Wich <jo@...n.io>,
Koen Vandeputte <koen.vandeputte@...ntric.com>
Subject: Re: NAT performance regression caused by vlan GRO support
Toshiaki Makita <makita.toshiaki@....ntt.co.jp> writes:
> On 2019/04/05 16:14, Felix Fietkau wrote:
>> On 2019-04-05 09:11, Rafał Miłecki wrote:
>>> On 05.04.2019 07:48, Rafał Miłecki wrote:
>>>> On 05.04.2019 06:26, Toshiaki Makita wrote:
>>>>> My test results:
>>>>>
>>>>> Receiving packets from eth0.10, forwarding them to eth0.20 and applying
>>>>> MASQUERADE on eth0.20, using i40e 25G NIC on kernel 4.20.13.
>>>>> Disabled rxvlan by ethtool -K to exercise vlan_gro_receive().
>>>>> Measured TCP throughput by netperf.
>>>>>
>>>>> GRO on : 17 Gbps
>>>>> GRO off: 5 Gbps
>>>>>
>>>>> So I failed to reproduce your problem.
>>>>
>>>> :( Thanks for trying & checking that!
>>>>
>>>>
>>>>> Would you check the CPU usage by "mpstat -P ALL" or similar (like "sar
>>>>> -u ALL -P ALL") to check if the traffic is able to consume 100% CPU on
>>>>> your machine?
>>>>
>>>> 1) ethtool -K eth0 gro on + iperf running (577 Mb/s)
>>>> root@...nWrt:/# mpstat -P ALL 10 3
>>>> Linux 5.1.0-rc3+ (OpenWrt) 03/27/19 _armv7l_ (2 CPU)
>>>>
>>>> 16:33:40 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
>>>> 16:33:50 all 0.00 0.00 0.00 0.00 0.00 58.79 0.00 0.00 41.21
>>>> 16:33:50 0 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00
>>>> 16:33:50 1 0.00 0.00 0.00 0.00 0.00 17.58 0.00 0.00 82.42
>>>>
>>>> 16:33:50 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
>>>> 16:34:00 all 0.00 0.00 0.05 0.00 0.00 59.44 0.00 0.00 40.51
>>>> 16:34:00 0 0.00 0.00 0.10 0.00 0.00 99.90 0.00 0.00 0.00
>>>> 16:34:00 1 0.00 0.00 0.00 0.00 0.00 18.98 0.00 0.00 81.02
>>>>
>>>> 16:34:00 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
>>>> 16:34:10 all 0.00 0.00 0.00 0.00 0.00 59.59 0.00 0.00 40.41
>>>> 16:34:10 0 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00
>>>> 16:34:10 1 0.00 0.00 0.00 0.00 0.00 19.18 0.00 0.00 80.82
>>>>
>>>> Average: CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
>>>> Average: all 0.00 0.00 0.02 0.00 0.00 59.27 0.00 0.00 40.71
>>>> Average: 0 0.00 0.00 0.03 0.00 0.00 99.97 0.00 0.00 0.00
>>>> Average: 1 0.00 0.00 0.00 0.00 0.00 18.58 0.00 0.00 81.42
>>>>
>>>>
>>>> 2) ethtool -K eth0 gro off + iperf running (941 Mb/s)
>>>> root@...nWrt:/# mpstat -P ALL 10 3
>>>> Linux 5.1.0-rc3+ (OpenWrt) 03/27/19 _armv7l_ (2 CPU)
>>>>
>>>> 16:34:39 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
>>>> 16:34:49 all 0.00 0.00 0.05 0.00 0.00 86.91 0.00 0.00 13.04
>>>> 16:34:49 0 0.00 0.00 0.10 0.00 0.00 78.22 0.00 0.00 21.68
>>>> 16:34:49 1 0.00 0.00 0.00 0.00 0.00 95.60 0.00 0.00 4.40
>>>>
>>>> 16:34:49 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
>>>> 16:34:59 all 0.00 0.00 0.10 0.00 0.00 87.06 0.00 0.00 12.84
>>>> 16:34:59 0 0.00 0.00 0.20 0.00 0.00 79.72 0.00 0.00 20.08
>>>> 16:34:59 1 0.00 0.00 0.00 0.00 0.00 94.41 0.00 0.00 5.59
>>>>
>>>> 16:34:59 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
>>>> 16:35:09 all 0.00 0.00 0.05 0.00 0.00 85.71 0.00 0.00 14.24
>>>> 16:35:09 0 0.00 0.00 0.10 0.00 0.00 79.42 0.00 0.00 20.48
>>>> 16:35:09 1 0.00 0.00 0.00 0.00 0.00 92.01 0.00 0.00 7.99
>>>>
>>>> Average: CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
>>>> Average: all 0.00 0.00 0.07 0.00 0.00 86.56 0.00 0.00 13.37
>>>> Average: 0 0.00 0.00 0.13 0.00 0.00 79.12 0.00 0.00 20.75
>>>> Average: 1 0.00 0.00 0.00 0.00 0.00 94.01 0.00 0.00 5.99
>>>>
>>>>
>>>> 3) System idle (no iperf)
>>>> root@...nWrt:/# mpstat -P ALL 10 1
>>>> Linux 5.1.0-rc3+ (OpenWrt) 03/27/19 _armv7l_ (2 CPU)
>>>>
>>>> 16:35:31 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
>>>> 16:35:41 all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
>>>> 16:35:41 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
>>>> 16:35:41 1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
>>>>
>>>> Average: CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
>>>> Average: all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
>>>> Average: 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
>>>> Average: 1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
>>>>
>>>>
>>>>> If CPU is 100%, perf may help us analyze your problem. If it's
>>>>> available, try running below while testing:
>>>>> # perf record -a -g -- sleep 5
>>>>>
>>>>> And then run this after testing:
>>>>> # perf report --no-child
>>>>
>>>> I can see my CPU 0 is fully loaded when using "gro on". I'll try perf now.
>>>
>>> I guess its GRO + csum_partial() to be blamed for this performance drop.
>>>
>>> Maybe csum_partial() is very fast on your powerful machine and few extra calls
>>> don't make a difference? I can imagine it affecting much slower home router with
>>> ARM cores.
>> Most high performance Ethernet devices implement hardware checksum
>> offload, which completely gets rid of this overhead.
>> Unfortunately, the BCM53xx/47xx Ethernet MAC doesn't have this, which is
>> why you're getting such crappy performance.
>
> Hmm... now I disabled rx checksum and tried the test again, and indeed I
> see csum_partial from GRO path. But I also see csum_partial even without
> GRO from nf_conntrack_in -> tcp_packet -> __skb_checksum_complete.
> Probably Rafał disabled nf_conntrack_checksum sysctl knob?
>
> But anyway even with disabling rx csum offload my machine has better
> performance with GRO.
But you're also running at way higher speeds, where the benefit of GRO
is higher.
> I'm sure in some cases GRO should be disabled, but I guess it's
> difficult to determine whether we should disable GRO or not
> automatically when csum offload is not available.
As a first approximation, maybe just:
if (!has_hardware_cksum_offload(netdev) && link_rate(netdev) <= 1Gbps)
disable_gro();
We used 1Gbps as the threshold for when to split GRO packets by default
in sck_cake as well...
-Toke
Powered by blists - more mailing lists