netdev - Re: NAT performance regression caused by vlan GRO support

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <f23dd3f7-691a-6f9b-fca0-7a78f44fcb97@gmail.com>
Date:   Fri, 5 Apr 2019 10:24:29 +0200
From:   Rafał Miłecki <zajec5@...il.com>
To:     Toshiaki Makita <makita.toshiaki@....ntt.co.jp>,
        Felix Fietkau <nbd@....name>
Cc:     Toshiaki Makita <toshiaki.makita1@...il.com>,
        netdev@...r.kernel.org, "David S. Miller" <davem@...emloft.net>,
        Stefano Brivio <sbrivio@...hat.com>,
        Sabrina Dubroca <sd@...asysnail.net>,
        David Ahern <dsahern@...il.com>, Jo-Philipp Wich <jo@...n.io>,
        Koen Vandeputte <koen.vandeputte@...ntric.com>
Subject: Re: NAT performance regression caused by vlan GRO support

On 05.04.2019 10:12, Rafał Miłecki wrote:
> On 05.04.2019 09:58, Toshiaki Makita wrote:
>> On 2019/04/05 16:14, Felix Fietkau wrote:
>>> On 2019-04-05 09:11, Rafał Miłecki wrote:
>>>> I guess its GRO + csum_partial() to be blamed for this performance drop.
>>>>
>>>> Maybe csum_partial() is very fast on your powerful machine and few extra calls
>>>> don't make a difference? I can imagine it affecting much slower home router with
>>>> ARM cores.
>>> Most high performance Ethernet devices implement hardware checksum
>>> offload, which completely gets rid of this overhead.
>>> Unfortunately, the BCM53xx/47xx Ethernet MAC doesn't have this, which is
>>> why you're getting such crappy performance.
>>
>> Hmm... now I disabled rx checksum and tried the test again, and indeed I
>> see csum_partial from GRO path. But I also see csum_partial even without
>> GRO from nf_conntrack_in -> tcp_packet -> __skb_checksum_complete.
>> Probably Rafał disabled nf_conntrack_checksum sysctl knob?
>>
>> But anyway even with disabling rx csum offload my machine has better
>> performance with GRO. I'm sure in some cases GRO should be disabled, but
>> I guess it's difficult to determine whether we should disable GRO or not
>> automatically when csum offload is not available.
> 
> Few testing results:
> 
> 1) ethtool -K eth0 gro off; echo 0 > /proc/sys/net/netfilter/nf_conntrack_checksum
> [  6]  0.0-60.0 sec  6.57 GBytes   940 Mbits/sec
> 
> 2) ethtool -K eth0 gro off; echo 1 > /proc/sys/net/netfilter/nf_conntrack_checksum
> [  6]  0.0-60.0 sec  4.65 GBytes   666 Mbits/sec

For this case (GRO off and nf_conntrack_checksum enabled) I can confirm I see
csum_partial() in the perf output. It's taking 13,14% instead of 25,46% (as when
using GRO) though.

Samples: 38K of event 'cycles', Event count (approx.): 12209908413
   Overhead  Command          Shared Object           Symbol
+   13,14%  ksoftirqd/1      [kernel.kallsyms]       [k] csum_partial
+   10,16%  swapper          [kernel.kallsyms]       [k] v7_dma_inv_range
+    6,36%  swapper          [kernel.kallsyms]       [k] l2c210_inv_range
+    4,89%  swapper          [kernel.kallsyms]       [k] __irqentry_text_end
+    4,12%  ksoftirqd/1      [kernel.kallsyms]       [k] v7_dma_clean_range
+    3,78%  swapper          [kernel.kallsyms]       [k] bcma_host_soc_read32
+    2,76%  swapper          [kernel.kallsyms]       [k] arch_cpu_idle
+    2,45%  ksoftirqd/1      [kernel.kallsyms]       [k] __netif_receive_skb_core
+    2,37%  ksoftirqd/1      [kernel.kallsyms]       [k] l2c210_clean_range
+    1,76%  ksoftirqd/1      [kernel.kallsyms]       [k] bgmac_start_xmit
+    1,66%  swapper          [kernel.kallsyms]       [k] bgmac_poll
+    1,55%  ksoftirqd/1      [kernel.kallsyms]       [k] __dev_queue_xmit
+    1,11%  ksoftirqd/1      [kernel.kallsyms]       [k] skb_vlan_untag


> 3) ethtool -K eth0 gro on; echo 0 > /proc/sys/net/netfilter/nf_conntrack_checksum
> [  6]  0.0-60.0 sec  4.02 GBytes   575 Mbits/sec
> 
> 4) ethtool -K eth0 gro on; echo 1 > /proc/sys/net/netfilter/nf_conntrack_checksum
> [  6]  0.0-60.0 sec  4.04 GBytes   579 Mbits/sec