| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <CAK8fFZ7cYRkGjUJD2D86G6Jh9YRmP_L+7Ke6CLFSyFmRkoe-Hg@mail.gmail.com>
Date: Thu, 5 Jan 2023 20:56:08 +0100
From: Jaroslav Pulchart <jaroslav.pulchart@...ddata.com>
To: Eric Dumazet <edumazet@...gle.com>
Cc: netdev@...r.kernel.org, lixiaoyan@...gle.com, pabeni@...hat.com,
davem@...emloft.net, Igor Raits <igor.raits@...ddata.com>
Subject: Re: Network performance regression with linux 6.1.y. Issue bisected
to "5eddb24901ee49eee23c0bfce6af2e83fd5679bd" (gro: add support of (hw)gro
packets to gro stack)
I can and I did. Your "Random guess" patch fix the problem and speed
is like with 6.0.y:
# git clone ...
...
Receiving objects: 100% (571306/571306), 350.58 MiB | 67.51 MiB/s, done.
Thanks,
JP
čt 5. 1. 2023 v 19:49 odesílatel Eric Dumazet <edumazet@...gle.com> napsal:
>
> On Thu, Jan 5, 2023 at 6:54 PM Jaroslav Pulchart
> <jaroslav.pulchart@...ddata.com> wrote:
> >
> > It is at KVM based VMs "CentOS 9 Stream" and "CentOS 8 Stream" using
> > upstream kernel 6.1.y. Hosted on Dell PowerEdge 7525 servers (2x AMD
> > 74F3) and OS CentOS 9 Stream again with upstream kernel 6.1.y or
> > 6.0.y.
> >
> > # ethtool -k eth0
> > Features for eth0:
> > rx-checksumming: on [fixed]
> > tx-checksumming: on
> > tx-checksum-ipv4: off [fixed]
> > tx-checksum-ip-generic: on
> > tx-checksum-ipv6: off [fixed]
> > tx-checksum-fcoe-crc: off [fixed]
> > tx-checksum-sctp: off [fixed]
> > scatter-gather: on
> > tx-scatter-gather: on
> > tx-scatter-gather-fraglist: off [fixed]
> > tcp-segmentation-offload: on
> > tx-tcp-segmentation: on
> > tx-tcp-ecn-segmentation: on
> > tx-tcp-mangleid-segmentation: off
> > tx-tcp6-segmentation: on
> > generic-segmentation-offload: on
> > generic-receive-offload: on
> > large-receive-offload: off [fixed]
> > rx-vlan-offload: off [fixed]
> > tx-vlan-offload: off [fixed]
> > ntuple-filters: off [fixed]
> > receive-hashing: off [fixed]
> > highdma: on [fixed]
> > rx-vlan-filter: on [fixed]
> > vlan-challenged: off [fixed]
> > tx-lockless: off [fixed]
> > netns-local: off [fixed]
> > tx-gso-robust: on [fixed]
> > tx-fcoe-segmentation: off [fixed]
> > tx-gre-segmentation: off [fixed]
> > tx-gre-csum-segmentation: off [fixed]
> > tx-ipxip4-segmentation: off [fixed]
> > tx-ipxip6-segmentation: off [fixed]
> > tx-udp_tnl-segmentation: off [fixed]
> > tx-udp_tnl-csum-segmentation: off [fixed]
> > tx-gso-partial: off [fixed]
> > tx-tunnel-remcsum-segmentation: off [fixed]
> > tx-sctp-segmentation: off [fixed]
> > tx-esp-segmentation: off [fixed]
> > tx-udp-segmentation: off [fixed]
> > tx-gso-list: off [fixed]
> > fcoe-mtu: off [fixed]
> > tx-nocache-copy: off
> > loopback: off [fixed]
> > rx-fcs: off [fixed]
> > rx-all: off [fixed]
> > tx-vlan-stag-hw-insert: off [fixed]
> > rx-vlan-stag-hw-parse: off [fixed]
> > rx-vlan-stag-filter: off [fixed]
> > l2-fwd-offload: off [fixed]
> > hw-tc-offload: off [fixed]
> > esp-hw-offload: off [fixed]
> > esp-tx-csum-hw-offload: off [fixed]
> > rx-udp_tunnel-port-offload: off [fixed]
> > tls-hw-tx-offload: off [fixed]
> > tls-hw-rx-offload: off [fixed]
> > rx-gro-hw: on
> > tls-hw-record: off [fixed]
> > rx-gro-list: off
> > macsec-hw-offload: off [fixed]
> > rx-udp-gro-forwarding: off
> > hsr-tag-ins-offload: off [fixed]
> > hsr-tag-rm-offload: off [fixed]
> > hsr-fwd-offload: off [fixed]
> > hsr-dup-offload: off [fixed]
> >
> > # ethtool -i eth0
> > driver: virtio_net
> > version: 1.0.0
> > firmware-version:
> > expansion-rom-version:
> > bus-info: 0000:03:00.0
> > supports-statistics: yes
> > supports-test: no
> > supports-eeprom-access: no
> > supports-register-dump: no
> > supports-priv-flags: no
>
> Random guess, can you try:
>
> diff --git a/net/core/gro.c b/net/core/gro.c
> index fd8c6a7e8d3e2e6b439109d0089f44a547c7347e..f162674e7ae1bdf96bcbf7e1ed7326729d862f9a
> 100644
> --- a/net/core/gro.c
> +++ b/net/core/gro.c
> @@ -500,7 +500,8 @@ static enum gro_result dev_gro_receive(struct
> napi_struct *napi, struct sk_buff
> BUILD_BUG_ON(!IS_ALIGNED(offsetof(struct napi_gro_cb, zeroed),
> sizeof(u32))); /* Avoid slow
> unaligned acc */
> *(u32 *)&NAPI_GRO_CB(skb)->zeroed = 0;
> - NAPI_GRO_CB(skb)->flush = skb_has_frag_list(skb);
> + NAPI_GRO_CB(skb)->flush = skb_has_frag_list(skb) ||
> + (skb_shinfo(skb)->gso_type & SKB_GSO_DODGY);
> NAPI_GRO_CB(skb)->is_atomic = 1;
> NAPI_GRO_CB(skb)->count = 1;
> if (unlikely(skb_is_gso(skb))) {
>
>
>
> >
> > čt 5. 1. 2023 v 18:43 odesílatel Eric Dumazet <edumazet@...gle.com> napsal:
> > >
> > > On Thu, Jan 5, 2023 at 6:37 PM Jaroslav Pulchart
> > > <jaroslav.pulchart@...ddata.com> wrote:
> > > >
> > > > Hello,
> > > >
> > > > I would like to report a 6.1,y regression in a network performance
> > > > observed when using "git clone".
> > > >
> > > > BAD: "git clone" speed with kernel 6.1,y:
> > > > # git clone git@...hub.com:..../.....git
> > > > ...
> > > > Receiving objects: 8% (47797/571306), 20.69 MiB | 3.27 MiB/s
> > > >
> > > > GOOD: "git clone" speed with kernel 6.0,y:
> > > > # git clone git@...hub.com:..../.....git
> > > > ...
> > > > Receiving objects: 72% (411341/571306), 181.05 MiB | 60.27 MiB/s
> > > >
> > > > I bisected the issue to a commit
> > > > 5eddb24901ee49eee23c0bfce6af2e83fd5679bd "gro: add support of (hw)gro
> > > > packets to gro stack". Reverting it from 6.1.y branch makes the git
> > > > clone fast like with 6.0.y.
> > > >
> > >
> > > Hmm, please provide more information.
> > >
> > > NIC used ? (ethtool -i eth0)
> > >
> > > ethtool -k eth0 # replace by your netdev name
> > >
> > > And packet captures would be nice (with and without the patch)
> > >
> > > Thanks.
> >
> >
> >
> > --
> > Jaroslav Pulchart
> > Sr. Principal SW Engineer
> > GoodData
--
Jaroslav Pulchart
Sr. Principal SW Engineer
GoodData
Powered by blists - more mailing lists