[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Message-ID: <20201021113729.4d4eeffa@kicinski-fedora-pc1c0hjn.dhcp.thefacebook.com>
Date: Wed, 21 Oct 2020 11:37:29 -0700
From: Jakub Kicinski <kuba@...nel.org>
To: Stephen Hemminger <stephen@...workplumber.org>
Cc: netdev@...r.kernel.org, Jay Vosburgh <j.vosburgh@...il.com>,
Jiri Pirko <jiri@...nulli.us>, Jarod Wilson <jarod@...hat.com>
Subject: Re: [Bug 209767] New: Bonding 802.3ad layer2+3 transmits on both
slaves within single connection
CC some bonding folks
On Tue, 20 Oct 2020 07:54:29 -0700 Stephen Hemminger wrote:
> Begin forwarded message:
>
> Date: Tue, 20 Oct 2020 10:42:34 +0000
> From: bugzilla-daemon@...zilla.kernel.org
> To: stephen@...workplumber.org
> Subject: [Bug 209767] New: Bonding 802.3ad layer2+3 transmits on both slaves within single connection
>
>
> https://bugzilla.kernel.org/show_bug.cgi?id=209767
>
> Bug ID: 209767
> Summary: Bonding 802.3ad layer2+3 transmits on both slaves
> within single connection
> Product: Networking
> Version: 2.5
> Kernel Version: 5.8.11-1.el8.elrepo.x86_64 and
> 3.10.0-1127.19.1.el7.x86_64
> Hardware: All
> OS: Linux
> Tree: Mainline
> Status: NEW
> Severity: normal
> Priority: P1
> Component: Other
> Assignee: stephen@...workplumber.org
> Reporter: onno.zweers@...f.nl
> Regression: No
>
> Dear people,
>
> I'm seeing bonding behavior I don't understand and neither do several network
> experts I've consulted.
>
> We have two servers, both with two 25 Gbit interfaces in a bond (802.3ad with
> layer2+3 hashing). We tuned the systems according to
> https://fasterdata.es.net/host-tuning/linux/. I run `iperf3 --server` on server
> 1 and connect to it with `iperf3 --client server1` from server 2. We notice
> that sometimes the connection is good (24.7 Gbit/s, no retransmits) and
> sometimes there are many retransmits (sometimes as many as >30,000 in a 10
> second run) and then the bandwidth may drop to 15 Gbit/s or even lower. The
> servers are idle except for the iperf3 runs. When we bring down one slave on
> server 1, the result is always perfect; no retransmits and good throughput.
>
> We have captured traffic with tcpdump on server 1 at the slave level (I'll try
> to add the pcap files). To our surprise, we see that the data channel ACK
> packets are sometimes sent over one slave and sometimes over the other. We
> think this causes packet misordering in the network switches, and thus
> retransmits and loss of bandwidth.
>
> Our understanding of layer2+3 hashing is that for a single connection, all
> traffic should go over the same slave. Therefore, we don't understand why
> server 1 sends ACK packets out over both slaves.
>
> I've read the documentation at
> https://www.kernel.org/doc/Documentation/networking/bonding.txt but I couldn't
> find the observed behaviour explained there.
>
> We have tested several Centos 7 and Centos 8 kernels, including recent elrepo
> kernels, but all show this behaviour. We have tried teaming instead of bonding
> but it has the same behaviour. We have tried other hashing algorithms like
> layer3+4 but they seem to have the same issue. It occurs with both IPv4 and
> IPv6.
>
> Is this behaviour to be expected? If yes, is it documented anywhere? Will it
> degrade throughput in real life traffic (with multiple concurrent data
> streams)?
> If the behaviour is not expected, are we doing something wrong, or might it be
> a bug?
>
> Thanks,
> Onno
>
Powered by blists - more mailing lists