netdev - Re: Network performance regression with linux 6.1.y. Issue bisected to "5eddb24901ee49eee23c0bfce6af2e83fd5679bd" (gro: add support of (hw)gro packets to gro stack)

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <CAK8fFZ7nWOhutDREJce9+ywnRTH_z4iCkrM2=Hnx00wXqGYxZQ@mail.gmail.com>
Date:   Fri, 6 Jan 2023 16:04:44 +0100
From:   Jaroslav Pulchart <jaroslav.pulchart@...ddata.com>
To:     Eric Dumazet <edumazet@...gle.com>
Cc:     netdev@...r.kernel.org, lixiaoyan@...gle.com, pabeni@...hat.com,
        davem@...emloft.net, Igor Raits <igor.raits@...ddata.com>
Subject: Re: Network performance regression with linux 6.1.y. Issue bisected
 to "5eddb24901ee49eee23c0bfce6af2e83fd5679bd" (gro: add support of (hw)gro
 packets to gro stack)

OK, I re-tested it with a new version of patch and it still works.



pá 6. 1. 2023 v 15:04 odesílatel Eric Dumazet <edumazet@...gle.com> napsal:
>
> On Thu, Jan 5, 2023 at 8:56 PM Jaroslav Pulchart
> <jaroslav.pulchart@...ddata.com> wrote:
> >
> > I can and I did. Your "Random guess" patch fix the problem and speed
> > is like with 6.0.y:
> >
> >   # git clone ...
> >   ...
> >   Receiving objects: 100% (571306/571306), 350.58 MiB | 67.51 MiB/s, done.
> >
> > Thanks,
>
> Excellent :)
>
> I will move the new test out of the fast path though, in a section we
> are sure gso_size is != 0.
>
> Something like this.
>
> diff --git a/net/core/gro.c b/net/core/gro.c
> index fd8c6a7e8d3e2e6b439109d0089f44a547c7347e..506f83d715f873c9bc3727e28ace71e00fa79d2f
> 100644
> --- a/net/core/gro.c
> +++ b/net/core/gro.c
> @@ -505,8 +505,9 @@ static enum gro_result dev_gro_receive(struct
> napi_struct *napi, struct sk_buff
>         NAPI_GRO_CB(skb)->count = 1;
>         if (unlikely(skb_is_gso(skb))) {
>                 NAPI_GRO_CB(skb)->count = skb_shinfo(skb)->gso_segs;
> -               /* Only support TCP at the moment. */
> -               if (!skb_is_gso_tcp(skb))
> +               /* Only support TCP and non DODGY users. */
> +               if (!skb_is_gso_tcp(skb) ||
> +                   (skb_shinfo(skb)->gso_type & SKB_GSO_DODGY))
>                         NAPI_GRO_CB(skb)->flush = 1;
>         }
>
>
>
> > JP
> >
> > čt 5. 1. 2023 v 19:49 odesílatel Eric Dumazet <edumazet@...gle.com> napsal:
> > >
> > > On Thu, Jan 5, 2023 at 6:54 PM Jaroslav Pulchart
> > > <jaroslav.pulchart@...ddata.com> wrote:
> > > >
> > > > It is at KVM based VMs "CentOS 9 Stream" and "CentOS 8 Stream" using
> > > > upstream kernel 6.1.y. Hosted on Dell PowerEdge 7525 servers (2x AMD
> > > > 74F3) and OS CentOS 9 Stream again with upstream kernel 6.1.y or
> > > > 6.0.y.
> > > >
> > > > # ethtool -k eth0
> > > > Features for eth0:
> > > > rx-checksumming: on [fixed]
> > > > tx-checksumming: on
> > > > tx-checksum-ipv4: off [fixed]
> > > > tx-checksum-ip-generic: on
> > > > tx-checksum-ipv6: off [fixed]
> > > > tx-checksum-fcoe-crc: off [fixed]
> > > > tx-checksum-sctp: off [fixed]
> > > > scatter-gather: on
> > > > tx-scatter-gather: on
> > > > tx-scatter-gather-fraglist: off [fixed]
> > > > tcp-segmentation-offload: on
> > > > tx-tcp-segmentation: on
> > > > tx-tcp-ecn-segmentation: on
> > > > tx-tcp-mangleid-segmentation: off
> > > > tx-tcp6-segmentation: on
> > > > generic-segmentation-offload: on
> > > > generic-receive-offload: on
> > > > large-receive-offload: off [fixed]
> > > > rx-vlan-offload: off [fixed]
> > > > tx-vlan-offload: off [fixed]
> > > > ntuple-filters: off [fixed]
> > > > receive-hashing: off [fixed]
> > > > highdma: on [fixed]
> > > > rx-vlan-filter: on [fixed]
> > > > vlan-challenged: off [fixed]
> > > > tx-lockless: off [fixed]
> > > > netns-local: off [fixed]
> > > > tx-gso-robust: on [fixed]
> > > > tx-fcoe-segmentation: off [fixed]
> > > > tx-gre-segmentation: off [fixed]
> > > > tx-gre-csum-segmentation: off [fixed]
> > > > tx-ipxip4-segmentation: off [fixed]
> > > > tx-ipxip6-segmentation: off [fixed]
> > > > tx-udp_tnl-segmentation: off [fixed]
> > > > tx-udp_tnl-csum-segmentation: off [fixed]
> > > > tx-gso-partial: off [fixed]
> > > > tx-tunnel-remcsum-segmentation: off [fixed]
> > > > tx-sctp-segmentation: off [fixed]
> > > > tx-esp-segmentation: off [fixed]
> > > > tx-udp-segmentation: off [fixed]
> > > > tx-gso-list: off [fixed]
> > > > fcoe-mtu: off [fixed]
> > > > tx-nocache-copy: off
> > > > loopback: off [fixed]
> > > > rx-fcs: off [fixed]
> > > > rx-all: off [fixed]
> > > > tx-vlan-stag-hw-insert: off [fixed]
> > > > rx-vlan-stag-hw-parse: off [fixed]
> > > > rx-vlan-stag-filter: off [fixed]
> > > > l2-fwd-offload: off [fixed]
> > > > hw-tc-offload: off [fixed]
> > > > esp-hw-offload: off [fixed]
> > > > esp-tx-csum-hw-offload: off [fixed]
> > > > rx-udp_tunnel-port-offload: off [fixed]
> > > > tls-hw-tx-offload: off [fixed]
> > > > tls-hw-rx-offload: off [fixed]
> > > > rx-gro-hw: on
> > > > tls-hw-record: off [fixed]
> > > > rx-gro-list: off
> > > > macsec-hw-offload: off [fixed]
> > > > rx-udp-gro-forwarding: off
> > > > hsr-tag-ins-offload: off [fixed]
> > > > hsr-tag-rm-offload: off [fixed]
> > > > hsr-fwd-offload: off [fixed]
> > > > hsr-dup-offload: off [fixed]
> > > >
> > > > # ethtool -i eth0
> > > > driver: virtio_net
> > > > version: 1.0.0
> > > > firmware-version:
> > > > expansion-rom-version:
> > > > bus-info: 0000:03:00.0
> > > > supports-statistics: yes
> > > > supports-test: no
> > > > supports-eeprom-access: no
> > > > supports-register-dump: no
> > > > supports-priv-flags: no
> > >
> > > Random guess, can you try:
> > >
> > > diff --git a/net/core/gro.c b/net/core/gro.c
> > > index fd8c6a7e8d3e2e6b439109d0089f44a547c7347e..f162674e7ae1bdf96bcbf7e1ed7326729d862f9a
> > > 100644
> > > --- a/net/core/gro.c
> > > +++ b/net/core/gro.c
> > > @@ -500,7 +500,8 @@ static enum gro_result dev_gro_receive(struct
> > > napi_struct *napi, struct sk_buff
> > >         BUILD_BUG_ON(!IS_ALIGNED(offsetof(struct napi_gro_cb, zeroed),
> > >                                         sizeof(u32))); /* Avoid slow
> > > unaligned acc */
> > >         *(u32 *)&NAPI_GRO_CB(skb)->zeroed = 0;
> > > -       NAPI_GRO_CB(skb)->flush = skb_has_frag_list(skb);
> > > +       NAPI_GRO_CB(skb)->flush = skb_has_frag_list(skb) ||
> > > +                                 (skb_shinfo(skb)->gso_type & SKB_GSO_DODGY);
> > >         NAPI_GRO_CB(skb)->is_atomic = 1;
> > >         NAPI_GRO_CB(skb)->count = 1;
> > >         if (unlikely(skb_is_gso(skb))) {
> > >
> > >
> > >
> > > >
> > > > čt 5. 1. 2023 v 18:43 odesílatel Eric Dumazet <edumazet@...gle.com> napsal:
> > > > >
> > > > > On Thu, Jan 5, 2023 at 6:37 PM Jaroslav Pulchart
> > > > > <jaroslav.pulchart@...ddata.com> wrote:
> > > > > >
> > > > > > Hello,
> > > > > >
> > > > > > I would like to report a 6.1,y regression in a network performance
> > > > > > observed when using "git clone".
> > > > > >
> > > > > > BAD: "git clone" speed with kernel 6.1,y:
> > > > > >    # git clone git@...hub.com:..../.....git
> > > > > >    ...
> > > > > >    Receiving objects:   8% (47797/571306), 20.69 MiB | 3.27 MiB/s
> > > > > >
> > > > > > GOOD: "git clone" speed with kernel 6.0,y:
> > > > > >    # git clone git@...hub.com:..../.....git
> > > > > >    ...
> > > > > >    Receiving objects:  72% (411341/571306), 181.05 MiB | 60.27 MiB/s
> > > > > >
> > > > > > I bisected the issue to a commit
> > > > > > 5eddb24901ee49eee23c0bfce6af2e83fd5679bd "gro: add support of (hw)gro
> > > > > > packets to gro stack". Reverting it from 6.1.y branch makes the git
> > > > > > clone fast like with 6.0.y.
> > > > > >
> > > > >
> > > > > Hmm, please provide more information.
> > > > >
> > > > > NIC used ? (ethtool -i eth0)
> > > > >
> > > > > ethtool -k eth0  # replace by your netdev name
> > > > >
> > > > > And packet captures would be nice (with and without the patch)
> > > > >
> > > > > Thanks.
> > > >
> > > >
> > > >
> > > > --
> > > > Jaroslav Pulchart
> > > > Sr. Principal SW Engineer
> > > > GoodData
> >
> >
> >
> > --
> > Jaroslav Pulchart
> > Sr. Principal SW Engineer
> > GoodData



-- 
Jaroslav Pulchart
Sr. Principal SW Engineer
GoodData