netdev - Re: Network performance regression with linux 6.1.y. Issue bisected to "5eddb24901ee49eee23c0bfce6af2e83fd5679bd" (gro: add support of (hw)gro packets to gro stack)

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CAK8fFZ7cyhnUsFiCE-mpQF9P_Q7M70RiDbXGNvjA+2Y_PyuQYQ@mail.gmail.com>
Date:   Thu, 5 Jan 2023 18:54:30 +0100
From:   Jaroslav Pulchart <jaroslav.pulchart@...ddata.com>
To:     Eric Dumazet <edumazet@...gle.com>
Cc:     netdev@...r.kernel.org, lixiaoyan@...gle.com, pabeni@...hat.com,
        davem@...emloft.net, Igor Raits <igor.raits@...ddata.com>
Subject: Re: Network performance regression with linux 6.1.y. Issue bisected
 to "5eddb24901ee49eee23c0bfce6af2e83fd5679bd" (gro: add support of (hw)gro
 packets to gro stack)

It is at KVM based VMs "CentOS 9 Stream" and "CentOS 8 Stream" using
upstream kernel 6.1.y. Hosted on Dell PowerEdge 7525 servers (2x AMD
74F3) and OS CentOS 9 Stream again with upstream kernel 6.1.y or
6.0.y.

# ethtool -k eth0
Features for eth0:
rx-checksumming: on [fixed]
tx-checksumming: on
tx-checksum-ipv4: off [fixed]
tx-checksum-ip-generic: on
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: on
tx-tcp-mangleid-segmentation: off
tx-tcp6-segmentation: on
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: off [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: on [fixed]
rx-vlan-filter: on [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: on [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-gre-csum-segmentation: off [fixed]
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: off [fixed]
tx-tunnel-remcsum-segmentation: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: off [fixed]
tx-gso-list: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: on
tls-hw-record: off [fixed]
rx-gro-list: off
macsec-hw-offload: off [fixed]
rx-udp-gro-forwarding: off
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: off [fixed]
hsr-dup-offload: off [fixed]

# ethtool -i eth0
driver: virtio_net
version: 1.0.0
firmware-version:
expansion-rom-version:
bus-info: 0000:03:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

čt 5. 1. 2023 v 18:43 odesílatel Eric Dumazet <edumazet@...gle.com> napsal:
>
> On Thu, Jan 5, 2023 at 6:37 PM Jaroslav Pulchart
> <jaroslav.pulchart@...ddata.com> wrote:
> >
> > Hello,
> >
> > I would like to report a 6.1,y regression in a network performance
> > observed when using "git clone".
> >
> > BAD: "git clone" speed with kernel 6.1,y:
> >    # git clone git@...hub.com:..../.....git
> >    ...
> >    Receiving objects:   8% (47797/571306), 20.69 MiB | 3.27 MiB/s
> >
> > GOOD: "git clone" speed with kernel 6.0,y:
> >    # git clone git@...hub.com:..../.....git
> >    ...
> >    Receiving objects:  72% (411341/571306), 181.05 MiB | 60.27 MiB/s
> >
> > I bisected the issue to a commit
> > 5eddb24901ee49eee23c0bfce6af2e83fd5679bd "gro: add support of (hw)gro
> > packets to gro stack". Reverting it from 6.1.y branch makes the git
> > clone fast like with 6.0.y.
> >
>
> Hmm, please provide more information.
>
> NIC used ? (ethtool -i eth0)
>
> ethtool -k eth0  # replace by your netdev name
>
> And packet captures would be nice (with and without the patch)
>
> Thanks.



-- 
Jaroslav Pulchart
Sr. Principal SW Engineer
GoodData