lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list] 
Message-ID: <c5a6fca9-fe20-4bd9-8a65-104197410264@de-wie-exch3b.green.sophos>
Date:	Wed, 4 Feb 2015 13:34:33 +0100
From:	Reiner Herrmann <reiner.herrmann@...hos.com>
To:	Tom Herbert <therbert@...gle.com>
CC:	Linux Netdev List <netdev@...r.kernel.org>, <svens@...ckframe.org>,
	Vittorio Curcio <Vittorio.Curcio@...hos.com>
Subject: Re: low vxlan throughput with tso enabled

On 02/03/2015 05:26 PM, Tom Herbert wrote:
> On Tue, Feb 3, 2015 at 7:50 AM, Reiner Herrmann
> <reiner.herrmann@...hos.com> wrote:
>> I have a vxlan tunnel established between two network interfaces, which
>> both have a MTU of 1500. The vxlan interfaces have the same MTU.
>> With TSO enabled, I observe low throughput with TCP connections (<100 kB/s).
>> Disabling TSO works around this issue and throughput is as expected.
>> Can someone please explain how TSO is influencing the tunnel to cause
>> such a difference?
> Please provide more information. What NIC? What is the link rate? What
> are you running to test. Also report 'ethtool -k' of both tunnel and
> Ethernet interfaces, and tcpdump on the interfaces (to see GSO
> packets).

Both are GBit NICs (it can be reproduced with different GBit NICs).
One has a link rate of 100 Mbps, the other 1000 Mbps.

To test the throughput, I'm using iperf (in TCP mode) between the two vxlan
interfaces.
I attached captures of the Ethernet interfaces from both devices, one with TSO
enabled on both vxlan interfaces (low throughput), and one with TSO disabled
on the vxlan interface of the iperf client (high throughput).

And below is the ethtool information you asked for.
Thank you for having a look.


Endpoint 1:
===========

Features for eth0:
rx-checksumming: on
tx-checksumming: on
     tx-checksum-ipv4: off [fixed]
     tx-checksum-ip-generic: on
     tx-checksum-ipv6: off [fixed]
     tx-checksum-fcoe-crc: off [fixed]
     tx-checksum-sctp: off [fixed]
scatter-gather: on
     tx-scatter-gather: on
     tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
     tx-tcp-segmentation: on
     tx-tcp-ecn-segmentation: off [fixed]
     tx-tcp6-segmentation: on
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: on [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-mpls-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off
rx-all: off
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]

Features for vxlan0:
rx-checksumming: on
tx-checksumming: on
     tx-checksum-ipv4: off [fixed]
     tx-checksum-ip-generic: on
     tx-checksum-ipv6: off [fixed]
     tx-checksum-fcoe-crc: off [fixed]
     tx-checksum-sctp: off [fixed]
scatter-gather: on
     tx-scatter-gather: on
     tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
     tx-tcp-segmentation: on
     tx-tcp-ecn-segmentation: on
     tx-tcp6-segmentation: on
udp-fragmentation-offload: on
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: off [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: on [fixed]
netns-local: on [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-mpls-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: on
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]


Endpoint 2:
===========

Features for eth0:
rx-checksumming: on
tx-checksumming: on
     tx-checksum-ipv4: off [fixed]
     tx-checksum-ip-generic: on
     tx-checksum-ipv6: off [fixed]
     tx-checksum-fcoe-crc: off [fixed]
     tx-checksum-sctp: off [fixed]
scatter-gather: on
     tx-scatter-gather: on
     tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
     tx-tcp-segmentation: on
     tx-tcp-ecn-segmentation: off [fixed]
     tx-tcp6-segmentation: on
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-ipip-segmentation: off [fixed]
tx-sit-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-mpls-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off
rx-all: off
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
busy-poll: off [fixed]

Features for vxlan0:
rx-checksumming: on
tx-checksumming: on
     tx-checksum-ipv4: off [fixed]
     tx-checksum-ip-generic: on
     tx-checksum-ipv6: off [fixed]
     tx-checksum-fcoe-crc: off [fixed]
     tx-checksum-sctp: off [fixed]
scatter-gather: on
     tx-scatter-gather: on
     tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
     tx-tcp-segmentation: on
     tx-tcp-ecn-segmentation: on
     tx-tcp6-segmentation: on
udp-fragmentation-offload: on
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: off [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: on [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-ipip-segmentation: off [fixed]
tx-sit-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-mpls-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: on
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
busy-poll: off [fixed]



Download attachment "device1_eth0_tso_off.pcap.xz" of type "application/x-xz" (338104 bytes)

Download attachment "device1_eth0_tso_on.pcap.xz" of type "application/x-xz" (31604 bytes)

Download attachment "device2_eth0_tso_off.pcap.xz" of type "application/x-xz" (334164 bytes)

Download attachment "device2_eth0_tso_on.pcap.xz" of type "application/x-xz" (35744 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ