Date:   Thu, 28 Jul 2022 10:45:02 -0400
From:   Brian Hutchinson <b.hutchman@...il.com>
To:     Vladimir Oltean <olteanv@...il.com>
Cc:     Florian Fainelli <f.fainelli@...il.com>, netdev@...r.kernel.org,
        andrew@...n.ch, woojung.huh@...rochip.com,
        UNGLinuxDriver@...rochip.com, j.vosburgh@...il.com,
        vfalico@...il.com, andy@...yhouse.net, davem@...emloft.net,
        kuba@...nel.org
Subject: Re: Bonded multicast traffic causing packet loss when using DSA with
 Microchip KSZ9567 switch

Hi Vladimir,

On Wed, Jul 27, 2022 at 7:32 PM Vladimir Oltean <olteanv@...il.com> wrote:

> > bond1: flags=5187<UP,BROADCAST,RUNNING,MASTER,MULTICAST>  mtu 1500  metric 1
> >        inet 192.168.1.6  netmask 255.255.255.0  broadcast 0.0.0.0
> >        inet6 fd1c:a799:6054:0:60e2:5ff:fe75:6716  prefixlen 64  scopeid 0x0<global>
> >        inet6 fe80::60e2:5ff:fe75:6716  prefixlen 64  scopeid 0x20<link>
> >        ether 62:e2:05:75:67:16  txqueuelen 1000  (Ethernet)
>
> I see bond1, lan1 and lan2 all have the same MAC address (62:e2:05:75:67:16).
> Does this happen even when they are all different?

So I have (when bond is setup using Systemd) assigned unique MAC
addresses for eth0, lan1 and lan2 ... but default action of bonding is
to assign the bond (bond1) and the slaves (lan1, lan2) a MAC that is
all the same among all the interfaces.  There are settings (controlled
by fail_over_mac) to specify which MAC is chosen to seed the MAC of
the other interfaces but bottom line is bonding makes both the bond
and active slave at a minimum the same MAC.

>
> >        RX packets 2557  bytes 3317974 (3.1 MiB)
> >        RX errors 0  dropped 2  overruns 0  frame 0
> >        TX packets 2370  bytes 3338160 (3.1 MiB)
> >        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
> >
> > eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1506  metric 1
> >        inet6 fe80::f21f:afff:fe6b:b218  prefixlen 64  scopeid 0x20<link>
> >        ether f0:1f:af:6b:b2:18  txqueuelen 1000  (Ethernet)
> >        RX packets 2557  bytes 3371671 (3.2 MiB)
> >        RX errors 0  dropped 0  overruns 0  frame 0
> >        TX packets 2394  bytes 3345891 (3.1 MiB)
> >        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
> >
> > lan1: flags=6211<UP,BROADCAST,RUNNING,SLAVE,MULTICAST>  mtu 1500  metric 1
> >        ether 62:e2:05:75:67:16  txqueuelen 1000  (Ethernet)
> >        RX packets 248  bytes 19384 (18.9 KiB)
> >        RX errors 0  dropped 0  overruns 0  frame 0
> >        TX packets 2370  bytes 3338160 (3.1 MiB)
> >        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
> >
> > lan2: flags=6211<UP,BROADCAST,RUNNING,SLAVE,MULTICAST>  mtu 1500  metric 1
> >        ether 62:e2:05:75:67:16  txqueuelen 1000  (Ethernet)
> >        RX packets 2309  bytes 3298590 (3.1 MiB)
> >        RX errors 0  dropped 1  overruns 0  frame 0
>
> I find this extremely strange. AFAIK, ifconfig reads stats from /proc/net/dev,
> which in turn takes them from the driver using dev_get_stats():
> https://elixir.bootlin.com/linux/v5.10.69/source/net/core/net-procfs.c#L78
>
> But DSA didn't even report the "dropped" count via ndo_get_stats64 in 5.10...
> https://elixir.bootlin.com/linux/v5.10.69/source/net/dsa/slave.c#L1257
>
> I have no idea why this shows 1. I'll have to ignore this information
> for now.
>
.
.
.
>
> Would you mind trying to test the exact same scenario again with the
> patch attached? (also pasted below in plain text) Still the same MAC
> address for all interfaces for now.

No problem at all.  I'm stumped too and welcome ideas to figure out
what is going on.

>
> From 033e3a8650a498de73cd202375b2e3f843e9a376 Mon Sep 17 00:00:00 2001
> From: Vladimir Oltean <vladimir.oltean@....com>
> Date: Thu, 28 Jul 2022 02:07:08 +0300
> Subject: [PATCH] ksz9477: force-disable address learning
>
> I suspect that what Brian Hutchinson experiences with the rx_discards
> counter incrementing is due to his setup, where 2 external switches
> connect together 2 bonded KSZ9567 switch ports, in such a way that one
> KSZ port is able to see packets sent by the other (this is probably
> aggravated by the multicast sent at a high data rate, which is treated
> as broadcast by the external switches and flooded).

So I mentioned in a recent PM that I was looking at other vendor DSA
drivers and I see code that smells like some of the concerns you have.

I did some grepping on /drivers/net/dsa and while I get hits for
things like 'flood', 'multicast', 'igmp' etc. in marvel and broadcom
drivers ... I get nothing on microchip.  Hardware documentation has
whole section on ingress and egress rate limiting and shaping but
doesn't look like drivers use any of it.

Example:

/drivers/net/dsa/mv88e6xxx$ grep -i multicast *.c
chip.c: { "in_multicasts",              4, 0x07, STATS_TYPE_BANK0, },
chip.c: { "out_multicasts",             4, 0x12, STATS_TYPE_BANK0, },
chip.c:                  is_multicast_ether_addr(addr))
chip.c: /* Upstream ports flood frames with unknown unicast or multicast DA */
chip.c:  * forwarding of unknown unicasts and multicasts.
chip.c:         dev_err(ds->dev, "p%d: failed to load multicast MAC address\n",
chip.c:                                  bool unicast, bool multicast)
chip.c:                                                       multicast);
global2.c:      /* Consider the frames with reserved multicast destination
global2.c:      /* Consider the frames with reserved multicast destination
port.c:                              bool unicast, bool multicast)
port.c: if (unicast && multicast)
port.c: else if (multicast)
port.c:                                       int port, bool multicast)
port.c: if (multicast)
port.c:                              bool unicast, bool multicast)
port.c: return mv88e6185_port_set_default_for
ward(chip, port, multicast);

Wondering if some needed support is missing.

Will try your patch and report back.

Regards,

Brian

Powered by blists - more mailing lists