netdev - Re: Network performance regression with linux 6.1.y. Issue bisected to "5eddb24901ee49eee23c0bfce6af2e83fd5679bd" (gro: add support of (hw)gro packets to gro stack)

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CANn89iKeNj4uUAVW2GJUiD5COqvUJjey-4-gpuUTp-er=2hAWg@mail.gmail.com>
Date:   Thu, 5 Jan 2023 19:49:33 +0100
From:   Eric Dumazet <edumazet@...gle.com>
To:     Jaroslav Pulchart <jaroslav.pulchart@...ddata.com>
Cc:     netdev@...r.kernel.org, lixiaoyan@...gle.com, pabeni@...hat.com,
        davem@...emloft.net, Igor Raits <igor.raits@...ddata.com>
Subject: Re: Network performance regression with linux 6.1.y. Issue bisected
 to "5eddb24901ee49eee23c0bfce6af2e83fd5679bd" (gro: add support of (hw)gro
 packets to gro stack)

On Thu, Jan 5, 2023 at 6:54 PM Jaroslav Pulchart
<jaroslav.pulchart@...ddata.com> wrote:
>
> It is at KVM based VMs "CentOS 9 Stream" and "CentOS 8 Stream" using
> upstream kernel 6.1.y. Hosted on Dell PowerEdge 7525 servers (2x AMD
> 74F3) and OS CentOS 9 Stream again with upstream kernel 6.1.y or
> 6.0.y.
>
> # ethtool -k eth0
> Features for eth0:
> rx-checksumming: on [fixed]
> tx-checksumming: on
> tx-checksum-ipv4: off [fixed]
> tx-checksum-ip-generic: on
> tx-checksum-ipv6: off [fixed]
> tx-checksum-fcoe-crc: off [fixed]
> tx-checksum-sctp: off [fixed]
> scatter-gather: on
> tx-scatter-gather: on
> tx-scatter-gather-fraglist: off [fixed]
> tcp-segmentation-offload: on
> tx-tcp-segmentation: on
> tx-tcp-ecn-segmentation: on
> tx-tcp-mangleid-segmentation: off
> tx-tcp6-segmentation: on
> generic-segmentation-offload: on
> generic-receive-offload: on
> large-receive-offload: off [fixed]
> rx-vlan-offload: off [fixed]
> tx-vlan-offload: off [fixed]
> ntuple-filters: off [fixed]
> receive-hashing: off [fixed]
> highdma: on [fixed]
> rx-vlan-filter: on [fixed]
> vlan-challenged: off [fixed]
> tx-lockless: off [fixed]
> netns-local: off [fixed]
> tx-gso-robust: on [fixed]
> tx-fcoe-segmentation: off [fixed]
> tx-gre-segmentation: off [fixed]
> tx-gre-csum-segmentation: off [fixed]
> tx-ipxip4-segmentation: off [fixed]
> tx-ipxip6-segmentation: off [fixed]
> tx-udp_tnl-segmentation: off [fixed]
> tx-udp_tnl-csum-segmentation: off [fixed]
> tx-gso-partial: off [fixed]
> tx-tunnel-remcsum-segmentation: off [fixed]
> tx-sctp-segmentation: off [fixed]
> tx-esp-segmentation: off [fixed]
> tx-udp-segmentation: off [fixed]
> tx-gso-list: off [fixed]
> fcoe-mtu: off [fixed]
> tx-nocache-copy: off
> loopback: off [fixed]
> rx-fcs: off [fixed]
> rx-all: off [fixed]
> tx-vlan-stag-hw-insert: off [fixed]
> rx-vlan-stag-hw-parse: off [fixed]
> rx-vlan-stag-filter: off [fixed]
> l2-fwd-offload: off [fixed]
> hw-tc-offload: off [fixed]
> esp-hw-offload: off [fixed]
> esp-tx-csum-hw-offload: off [fixed]
> rx-udp_tunnel-port-offload: off [fixed]
> tls-hw-tx-offload: off [fixed]
> tls-hw-rx-offload: off [fixed]
> rx-gro-hw: on
> tls-hw-record: off [fixed]
> rx-gro-list: off
> macsec-hw-offload: off [fixed]
> rx-udp-gro-forwarding: off
> hsr-tag-ins-offload: off [fixed]
> hsr-tag-rm-offload: off [fixed]
> hsr-fwd-offload: off [fixed]
> hsr-dup-offload: off [fixed]
>
> # ethtool -i eth0
> driver: virtio_net
> version: 1.0.0
> firmware-version:
> expansion-rom-version:
> bus-info: 0000:03:00.0
> supports-statistics: yes
> supports-test: no
> supports-eeprom-access: no
> supports-register-dump: no
> supports-priv-flags: no

Random guess, can you try:

diff --git a/net/core/gro.c b/net/core/gro.c
index fd8c6a7e8d3e2e6b439109d0089f44a547c7347e..f162674e7ae1bdf96bcbf7e1ed7326729d862f9a
100644
--- a/net/core/gro.c
+++ b/net/core/gro.c
@@ -500,7 +500,8 @@ static enum gro_result dev_gro_receive(struct
napi_struct *napi, struct sk_buff
        BUILD_BUG_ON(!IS_ALIGNED(offsetof(struct napi_gro_cb, zeroed),
                                        sizeof(u32))); /* Avoid slow
unaligned acc */
        *(u32 *)&NAPI_GRO_CB(skb)->zeroed = 0;
-       NAPI_GRO_CB(skb)->flush = skb_has_frag_list(skb);
+       NAPI_GRO_CB(skb)->flush = skb_has_frag_list(skb) ||
+                                 (skb_shinfo(skb)->gso_type & SKB_GSO_DODGY);
        NAPI_GRO_CB(skb)->is_atomic = 1;
        NAPI_GRO_CB(skb)->count = 1;
        if (unlikely(skb_is_gso(skb))) {



>
> čt 5. 1. 2023 v 18:43 odesílatel Eric Dumazet <edumazet@...gle.com> napsal:
> >
> > On Thu, Jan 5, 2023 at 6:37 PM Jaroslav Pulchart
> > <jaroslav.pulchart@...ddata.com> wrote:
> > >
> > > Hello,
> > >
> > > I would like to report a 6.1,y regression in a network performance
> > > observed when using "git clone".
> > >
> > > BAD: "git clone" speed with kernel 6.1,y:
> > >    # git clone git@...hub.com:..../.....git
> > >    ...
> > >    Receiving objects:   8% (47797/571306), 20.69 MiB | 3.27 MiB/s
> > >
> > > GOOD: "git clone" speed with kernel 6.0,y:
> > >    # git clone git@...hub.com:..../.....git
> > >    ...
> > >    Receiving objects:  72% (411341/571306), 181.05 MiB | 60.27 MiB/s
> > >
> > > I bisected the issue to a commit
> > > 5eddb24901ee49eee23c0bfce6af2e83fd5679bd "gro: add support of (hw)gro
> > > packets to gro stack". Reverting it from 6.1.y branch makes the git
> > > clone fast like with 6.0.y.
> > >
> >
> > Hmm, please provide more information.
> >
> > NIC used ? (ethtool -i eth0)
> >
> > ethtool -k eth0  # replace by your netdev name
> >
> > And packet captures would be nice (with and without the patch)
> >
> > Thanks.
>
>
>
> --
> Jaroslav Pulchart
> Sr. Principal SW Engineer
> GoodData