Message-ID: <ZG_MyiWSARvzKOeT@toxoplasmosis>
Date: Thu, 25 May 2023 23:02:02 +0200
From: Moviuro <moviuro@...il.com>
To: Jay Vosburgh <jay.vosburgh@...onical.com>
Cc: netdev@...r.kernel.org
Subject: Re: Secondary bond slave receiving packets when preferred is up

On 23-05-23 16:01:38, Jay Vosburgh wrote:
> Moviuro <moviuro@...il.com> wrote:
> 
> >Hi there,
> >
> >On 2 similar machines, some (random?) packets are received on a wireless
> >bond slave when the preferred eth interface is connected: this causes
> >local packet loss and at worst, disconnects (e.g. SSH and KDEConnect).
> >
> >My setup looks fine, inspired by the Arch wiki[0], see
> >/proc/net/bonding/bond0 below. The archlinux community has not been able
> >to help so far[1].
> >
> >Sure enough, there's some noise on the WiFi interface:
> >
> >root@149 # tcpdump -ttttnei wlp3s0 host 192.168.1.149 and not arp
> >2023-05-23 09:29:46.771535 11:11:11:11:11:74 > BB:BB:BB:BB:BB:33, ethertype IPv4 (0x0800), length 98: 192.168.1.1 > 192.168.1.149: ICMP echo request , id 64306, seq 53425, length 64
> >2023-05-23 09:36:04.710859 bb:bb:bb:bb:bb:32 > BB:BB:BB:BB:BB:33, ethertype IPv4 (0x0800), length 98: 192.168.1.111 > 192.168.1.149: ICMP echo reque st, id 1, seq 2390, length 64
> 
> 	Some amount of random traffic arriving on the inactive interface
> of an active-backup bond is expected; switches send traffic to such
> places for various reasons.  My initial guess would be that the switch's
> forwarding entry for whatever BB:BB:BB:BB:BB:33 is expired, and the
> switch flooded traffic for that destination to all ports.  As an aside,
> what is that MAC address?  The last octet (33) doesn't appear in any of
> the bond info dumps you list later for the .149 host.

Hi Jay,

BB:BB:BB:BB:BB:33 is the MAC address that was assigned to the bond:

root@149 # ip a show dev bond0 
2: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether BB:BB:BB:BB:BB:33 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.149/24 metric 1024 brd 192.168.1.255 scope global dynamic bond0
       valid_lft 30639sec preferred_lft 30639sec
    inet6 [...]

Similarly

* bb:bb:bb:bb:bb:32 is that of the bond on 111
* pp:pp:pp:pp:pp:af is that of my phone (173)
* 11:11:11:11:11:74 is that of my router (1)

You write about my switch possibly flooding its ports to find the host,
but here's a capture (173 is my phone which is trying to use KDEConnect[0]):
(wireshark's "summary" of packets, captured with `tcpdump -tttnei $iface
host 192.168.1.173 and not arp`)

if   num   time            src           dst
eth  6147  18:07:13,711496 192.168.1.111 192.168.1.173 TCP 66  [TCP Keep-Alive] 43768 → 1716 [ACK] Seq=1614043844 Ack=2714768918 Win=75776 Len=0 TSval=277775293 TSecr=3246552226
wlp  13    18:07:13,790981 192.168.1.173 192.168.1.111 TCP 66  [TCP Previous segment not captured] 1716 → 43768 [ACK] Seq=2714768918 Ack=1614043845 Win=1026 Len=0 TSval=3246562426 TSecr=277642263
eth  6148  18:07:18,831498 192.168.1.111 192.168.1.173 TCP 66  [TCP Keep-Alive] 43768 → 1716 [ACK] Seq=1614043844 Ack=2714768918 Win=75776 Len=0 TSval=277780413 TSecr=3246552226
wlp  14    18:07:18,837907 192.168.1.173 192.168.1.111 TCP 66  [TCP Dup ACK 13#1] 1716 → 43768 [ACK] Seq=2714768918 Ack=1614043845 Win=1026 Len=0 TSval=3246567474 TSecr=277642263
eth  6149  18:07:23,951497 192.168.1.111 192.168.1.173 TCP 66  [TCP Keep-Alive] 43768 → 1716 [ACK] Seq=1614043844 Ack=2714768918 Win=75776 Len=0 TSval=277785533 TSecr=3246552226
wlp  15    18:07:23,968739 192.168.1.173 192.168.1.111 TCP 66  [TCP Dup ACK 13#2] 1716 → 43768 [ACK] Seq=2714768918 Ack=1614043845 Win=1026 Len=0 TSval=3246572602 TSecr=277642263
eth  6150  18:11:02,898406 192.168.1.173 192.168.1.111 TLSv1.2 220 Application Data
eth  6151  18:11:02,902377 192.168.1.111 192.168.1.173 TCP 54  43768 → 1716 [RST] Seq=1614043845 Win=0 Len=0

There are a few packets (13..15) that only appear on the wifi capture,
there are no similar packets on the eth capture, ending with a RST
instead.

I can share the full captures with you if needed.

> 	In any event, an inactive bond interface will pass incoming
> traffic in two cases:
> 
> 	1) its destination MAC address is in the link local reserved
> range, 01:80:c2:00:00:0?, which is used for things like Spanning Tree or
> LACP; the complete list can be found at
> 
> https://standards.ieee.org/products-programs/regauth/grpmac/public/
> 
> 	These should not be ARP or IP, and this is unlikely to be your
> situation.
> 
> 	2) Something is bound directly to the bond interface itself via
> a raw socket or the like; an example of this is LLDP, which needs to
> exchange protocol frames at the interface level.

I had to look it up, but nothing running on my own machines as far as I
can tell.

root@111 # tcpdump -i wlp5s0 ether proto 0x88cc
19:01:01.309306 LLDP, length 129: U6Piano
19:01:31.277979 LLDP, length 129: U6Piano
[...]
root@111 # tcpdump -i enp4s0 ether proto 0x88cc
19:12:42.796503 LLDP, length 93: USW-24-PoE
19:13:12.798349 LLDP, length 93: USW-24-PoE
[...]
root@111 # tcpdump -i bond0 ether proto 0x88cc
19:13:12.798349 LLDP, length 93: USW-24-PoE
19:13:42.800450 LLDP, length 93: USW-24-PoE
[...]

# cat /proc/net/raw
  sl  local_address rem_address   st tx_queue rx_queue tr tm->when retrnsmt   uid  timeout inode ref pointer drops

> 	Even if the bond accepted some IP traffic on the inactive
> interface and sent it up the stack, any reply should go back out the
> active interface.  This is based on the lack of failovers in the bond
> status stuff, and presuming that the routing table on .111 and .149 is
> what I'd expect (basically, a default route and subnet route for
> 192.168.1.0/24 that go through the bond only).

Indeed, nothing weird:

root@149 # ip ro show
default via 192.168.1.1 dev bond0 proto dhcp src 192.168.1.149 metric 1024 
192.168.1.0/24 dev bond0 proto kernel scope link src 192.168.1.149 metric 1024 
192.168.1.1 dev bond0 proto dhcp scope link src 192.168.1.149 metric 1024

root@111 # ip ro show
default via 192.168.1.1 dev bond0 proto dhcp src 192.168.1.111 metric 1024 
10.5.0.0/24 dev wg0 proto kernel scope link src 10.5.0.35 
10.10.10.0/24 via 10.5.0.1 dev wg0 proto static onlink 
10.10.20.0/24 via 10.5.0.1 dev wg0 proto static onlink 
10.10.30.0/24 via 10.5.0.1 dev wg0 proto static onlink 
10.10.40.0/24 via 10.5.0.1 dev wg0 proto static onlink 
192.168.1.0/24 dev bond0 proto kernel scope link src 192.168.1.111 metric 1024 
192.168.1.1 dev bond0 proto dhcp scope link src 192.168.1.111 metric 1024

> 	Some suggestions that might help:
> 
> 	1) Check rp_filter; if it's not enabled, then turn it on in
> strict mode.  This means insuring that the sysctls for .all, the bond
> and its interfaces are all set to 1, e.g.,
> 
> net.ipv4.conf.all.rp_filter = 1
> net.ipv4.conf.bond0.rp_filter = 1
> net.ipv4.conf.wlp5s0.rp_filter = 1
> [... and so on ...]
> 
> 	Setting any of them to 2 will enable loose mode (the maximum
> value between .all and the interface is what counts).  Loose mode, or
> rp_filter being off entirely, might be your problem if your routing is
> not simple (e.g., you've got other IP networks that you didn't
> describe).  The docs for this can be found at
> 
> https://docs.kernel.org/networking/ip-sysctl.html

After a few changes:

root@111 # sysctl -a|grep '\.rp_filter'
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.bond0.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.enp4s0.rp_filter = 1
net.ipv4.conf.lo.rp_filter = 2
net.ipv4.conf.wg0.rp_filter = 1
net.ipv4.conf.wlp5s0.rp_filter = 1

But tcpdump(8) still shows some traffic on the wifi interface (see
above).

> 	2) Enable the bonding option fail_over_mac = follow, this will
> cause the MAC of the bond interfaces to not be all set to the same MAC.
> If somehow the switch is getting confused by seeing the same MAC from
> multiple ports, this may help.

This change caused the failover to not work.

root@149 # cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v6.3.2-zen1-1.1-zen

Bonding Mode: fault-tolerance (active-backup) (fail_over_mac follow)
[...]

# journalctl
[...]
May 24 12:00:35 houndsditch systemd-networkd[276]: enp0s31f6: Lost carrier
May 24 12:00:36 houndsditch kernel: bond0: (slave enp0s31f6): link status definitely down, disabling slave
May 24 12:00:36 houndsditch kernel: bond0: (slave wlp3s0): making interface the new active one
May 24 12:00:36 houndsditch kernel: bond0: (slave wlp3s0): Error 16 setting MAC of new active slave
[...]

After that, networking stays down. The router doesn't see any DHCP
requests, the machine is unable to reach the network.

Same if I tried to unplug replug the eth interface:

May 24 14:21:31 houndsditch systemd-networkd[271]: enp0s31f6: Lost carrier
May 24 14:21:31 houndsditch kernel: e1000e 0000:00:1f.6 enp0s31f6: NIC Link is Down
May 24 14:21:31 houndsditch kernel: bond0: (slave enp0s31f6): link status definitely down, disabling slave
May 24 14:21:31 houndsditch kernel: bond0: (slave wlp3s0): making interface the new active one
May 24 14:21:31 houndsditch kernel: bond0: (slave wlp3s0): Error 16 setting MAC of new active slave
May 24 14:21:40 houndsditch kernel: e1000e 0000:00:1f.6 enp0s31f6: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
May 24 14:21:40 houndsditch systemd-networkd[271]: enp0s31f6: Gained carrier
May 24 14:21:41 houndsditch kernel: bond0: (slave enp0s31f6): link status definitely up, 1000 Mbps full duplex
May 24 14:21:41 houndsditch kernel: bond0: (slave enp0s31f6): making interface the new active one
May 24 14:21:41 houndsditch kernel: bond0: (slave wlp3s0): Error 16 setting MAC of old active slave
May 24 14:21:41 houndsditch sshfs[1013]: Timeout, server 192.168.1.100 not responding.

However, that change (fail_over_mac follow) made it so that unifi (the
software config util for my switch and WAPs[1]) was no longer confused
about where in the topology the machine was. With fail_over_mac=follow,
149 was properly shown as connected to a port on the switch instead of
connected by WiFi to the WAP (which is also somewhat correct?).

I have a limited shell access to my WAP and switch, maybe I'd have to
look in there?

[0] https://kdeconnect.kde.org/ ; https://github.com/bboozzoo/mconnect
[1] https://community.ui.com/releases/UniFi-Network-Application-7-3-83/88f5ff3f-4c13-45e2-b57e-ad04b4140528;
also https://ui.com/consoles

Powered by blists - more mailing lists