lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  PHC 
Open Source and information security mailing list archives
Hash Suite for Android: free password hash cracker in your pocket
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date:   Sun, 7 Apr 2019 13:54:59 +0200
From:   Rafał Miłecki <>
To:, "David S. Miller" <>,
        Toshiaki Makita <>,
        Toke Høiland-Jørgensen <>,
        Florian Westphal <>,
        Eric Dumazet <>
Cc:     Stefano Brivio <>,
        Sabrina Dubroca <>,
        David Ahern <>, Felix Fietkau <>,
        Jo-Philipp Wich <>,
        Koen Vandeputte <>
Subject: Re: NAT performance regression caused by vlan GRO support

Now I have some questions regarding possible optimizations. Note I'm too
familiar with the net subsystem so maybe I got wrong ideas.

On 07.04.2019 13:53, Rafał Miłecki wrote:
> On 04.04.2019 14:57, Rafał Miłecki wrote:
>> Long story short, starting with the commit 66e5133f19e9 ("vlan: Add GRO support
>> for non hardware accelerated vlan") - which first hit kernel 4.2 - NAT
>> performance of my router dropped by 30% - 40%.
> I'll try to provide some summary for this issue. I'll focus on TCP traffic as
> that's what I happened to test.
> Basically all slowdowns are related to the csum_partial(). Calculating checksum
> has a significant impact on NAT performance on less CPU powerful devices.
> **********
> GRO disabled
> Without GRO a csum_partial() is used only when validating TCP packets in the
> nf_conntrack_tcp_packet() (known as tcp_packet() in kernels older than 5.1).
> Simplified forward trace for that case:
> nf_conntrack_in
>      nf_conntrack_tcp_packet
>          tcp_error
>              if (state->net->ct.sysctl_checksum)
>                  nf_checksum
>                      nf_ip_checksum
>                          __skb_checksum_complete
> That validation can be disabled using nf_conntrack_checksum sysfs and it bumps
> NAT speed for me from 666 Mb/s to 940 Mb/s (+41%).
> **********
> GRO enabled
> First of all GRO also includes TCP validation that requires calculating a
> checksum.
> Simplified forward trace for that case:
> vlan_gro_receive
>      call_gro_receive
>          inet_gro_receive
>              indirect_call_gro_receive
>                  tcp4_gro_receive
>                      skb_gro_checksum_validate
>                      tcp_gro_receive
> *If* we had a way to disable that validation it *would* result in bumping NAT
> speed for me from 577 Mb/s to 825 Mb/s (+43%).

Could we have tcp4_gro_receive() behave similarly to the tcp_error() and make it
respect the nf_conntrack_checksum sysfs value?

Could we simply add something like:
if (dev_net(skb->dev)->ct.sysctl_checksum)
to it (to additionally protect a skb_gro_checksum_validate() call)?

> Secondly using GRO means we need to calculate a checksum before transmitting
> packets (applies to devices without HW checksum offloading). I think it's
> related to packets merging in the skb_gro_receive() and then setting
> vlan_gro_complete
>      inet_gro_complete
>          tcp4_gro_complete
>              tcp_gro_complete
>                  skb->ip_summed = CHECKSUM_PARTIAL;
> That results in bgmac calculating a checksum from the scratch, take a look at
> the bgmac_dma_tx_add() which does:
> if (skb->ip_summed == CHECKSUM_PARTIAL)
>      skb_checksum_help(skb);
> Performing that whole checksum calculation will always result in GRO slowing
> down NAT for me when using BCM47094 SoC with that not-so-powerful ARM CPUs.

Is this possible to avoid CHECKSUM_PARTIAL & skb_checksum_help() which has to
calculate a whole checksum? It's definitely possible to *update* checksum after
simple packet changes (e.g. amending an IP or port). Would that be possible to
use similar method when dealing with packets with GRO enabled?

If not, maybe w really need to think about some good & clever condition for
disabling GRO by default on hw without checksum offloading.

Powered by blists - more mailing lists