netdev - Fw: [Bug 209767] New: Bonding 802.3ad layer2+3 transmits on both slaves within single connection

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <20201020075429.291c34bb@hermes.local>
Date:   Tue, 20 Oct 2020 07:54:29 -0700
From:   Stephen Hemminger <stephen@...workplumber.org>
To:     netdev@...r.kernel.org
Subject: Fw: [Bug 209767] New: Bonding 802.3ad layer2+3 transmits on both
 slaves within single connection

Begin forwarded message:

Date: Tue, 20 Oct 2020 10:42:34 +0000
From: bugzilla-daemon@...zilla.kernel.org
To: stephen@...workplumber.org
Subject: [Bug 209767] New: Bonding 802.3ad layer2+3 transmits on both slaves within single connection

https://bugzilla.kernel.org/show_bug.cgi?id=209767

            Bug ID: 209767
           Summary: Bonding 802.3ad layer2+3 transmits on both slaves
                    within single connection
           Product: Networking
           Version: 2.5
    Kernel Version: 5.8.11-1.el8.elrepo.x86_64 and
                    3.10.0-1127.19.1.el7.x86_64
          Hardware: All
                OS: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: Other
          Assignee: stephen@...workplumber.org
          Reporter: onno.zweers@...f.nl
        Regression: No

Dear people,

I'm seeing bonding behavior I don't understand and neither do several network
experts I've consulted.

We have two servers, both with two 25 Gbit interfaces in a bond (802.3ad with
layer2+3 hashing). We tuned the systems according to
https://fasterdata.es.net/host-tuning/linux/. I run `iperf3 --server` on server
1 and connect to it with `iperf3 --client server1` from server 2. We notice
that sometimes the connection is good (24.7 Gbit/s, no retransmits) and
sometimes there are many retransmits (sometimes as many as >30,000 in a 10
second run) and then the bandwidth may drop to 15 Gbit/s or even lower. The
servers are idle except for the iperf3 runs. When we bring down one slave on
server 1, the result is always perfect; no retransmits and good throughput.

We have captured traffic with tcpdump on server 1 at the slave level (I'll try
to add the pcap files). To our surprise, we see that the data channel ACK
packets are sometimes sent over one slave and sometimes over the other. We
think this causes packet misordering in the network switches, and thus
retransmits and loss of bandwidth.

Our understanding of layer2+3 hashing is that for a single connection, all
traffic should go over the same slave. Therefore, we don't understand why
server 1 sends ACK packets out over both slaves.

I've read the documentation at
https://www.kernel.org/doc/Documentation/networking/bonding.txt but I couldn't
find the observed behaviour explained there.

We have tested several Centos 7 and Centos 8 kernels, including recent elrepo
kernels, but all show this behaviour. We have tried teaming instead of bonding
but it has the same behaviour. We have tried other hashing algorithms like
layer3+4 but they seem to have the same issue. It occurs with both IPv4 and
IPv6.

Is this behaviour to be expected? If yes, is it documented anywhere? Will it
degrade throughput in real life traffic (with multiple concurrent data
streams)?
If the behaviour is not expected, are we doing something wrong, or might it be
a bug?

Thanks,
Onno

-- 
You are receiving this mail because:
You are the assignee for the bug.