Message-ID: <20250129145845.3988cf04@wsk>
Date: Wed, 29 Jan 2025 14:58:45 +0100
From: Lukasz Majewski <lukma@...x.de>
To: Frieder Schrempf <frieder.schrempf@...tron.de>
Cc: Andrew Lunn <andrew@...n.ch>, "netdev@...r.kernel.org"
 <netdev@...r.kernel.org>
Subject: Re: KSZ9477 HSR Offloading

Hi Frieder,

> Hi Lukasz,
> 
> On 29.01.25 12:17 PM, Lukasz Majewski wrote:
> > Hi Frieder,
> >   
> >> On 29.01.25 8:24 AM, Frieder Schrempf wrote:  
> >>> Hi Andrew,
> >>>
> >>> On 28.01.25 6:51 PM, Andrew Lunn wrote:    
> >>>> On Tue, Jan 28, 2025 at 05:14:46PM +0100, Frieder Schrempf
> >>>> wrote:    
> >>>>> Hi,
> >>>>>
> >>>>> I'm trying out HSR support on KSZ9477 with v6.12. My setup looks
> >>>>> like this:
> >>>>>
> >>>>> +-------------+         +-------------+
> >>>>> |             |         |             |
> >>>>> |   Node A    |         |   Node D    |
> >>>>> |             |         |             |
> >>>>> |             |         |             |
> >>>>> | LAN1   LAN2 |         | LAN1   LAN2 |
> >>>>> +--+-------+--+         +--+------+---+
> >>>>>    |       |               |      |
> >>>>>    |       +---------------+      |
> >>>>>    |                              |
> >>>>>    |       +---------------+      |
> >>>>>    |       |               |      |
> >>>>> +--+-------+--+         +--+------+---+
> >>>>> | LAN1   LAN2 |         | LAN1   LAN2 |
> >>>>> |             |         |             |
> >>>>> |             |         |             |
> >>>>> |   Node B    |         |   Node C    |
> >>>>> |             |         |             |
> >>>>> +-------------+         +-------------+
> >>>>>
> >>>>> On each device the LAN1 and LAN2 are added as HSR slaves. Then I
> >>>>> try to do ping tests between each of the HSR interfaces.
> >>>>>
> >>>>> The result is that I can reach the neighboring nodes just fine,
> >>>>> but I can't reach the remote node that needs packages to be
> >>>>> forwarded through the other nodes. For example I can't ping from
> >>>>> node A to C.
> >>>>>
> >>>>> I've tried to disable HW offloading in the driver and then
> >>>>> everything starts working.
> >>>>>
> >>>>> Is this a problem with HW offloading in the KSZ driver, or am I
> >>>>> missing something essential?    
> > 
> > Thanks for looking and testing such large scale setup.
> >   
> >>>>
> >>>> How are IP addresses configured? I assume you have a bridge, LAN1
> >>>> and LAN2 are members of the bridge, and the IP address is on the
> >>>> bridge interface?    
> >>>
> >>> I have a HSR interface on each node that covers LAN1 and LAN2 as
> >>> slaves and the IP addresses are on those HSR interfaces. For node
> >>> A:
> >>>
> >>> ip link add name hsr type hsr slave1 lan1 slave2 lan2 supervision
> >>> 45 version 1
> >>> ip addr add 172.20.1.1/24 dev hsr
> >>>
> >>> The other nodes have the addresses 172.20.1.2/24, 172.20.1.3/24
> >>> and 172.20.1.4/24 respectively.
> >>>
> >>> Then on node A, I'm doing:
> >>>
> >>> ping 172.20.1.2 # neighboring node B works
> >>> ping 172.20.1.4 # neighboring node D works
> >>> ping 172.20.1.3 # remote node C works only if I disable
> >>> offloading    
> >>
> >> BTW, it's enough to disable the offloading of the forwarding for
> >> HSR frames to make it work.
> >>
> >> --- a/drivers/net/dsa/microchip/ksz9477.c
> >> +++ b/drivers/net/dsa/microchip/ksz9477.c
> >> @@ -1267,7 +1267,7 @@ int ksz9477_tc_cbs_set_cinc(struct ksz_device
> >> *dev, int port, u32 val)
> >>   * Moreover, the NETIF_F_HW_HSR_FWD feature is also enabled, as
> >> HSR frames
> >>   * can be forwarded in the switch fabric between HSR ports.
> >>   */
> >> -#define KSZ9477_SUPPORTED_HSR_FEATURES (NETIF_F_HW_HSR_DUP |
> >> NETIF_F_HW_HSR_FWD)
> >> +#define KSZ9477_SUPPORTED_HSR_FEATURES (NETIF_F_HW_HSR_DUP)
> >>
> >>  void ksz9477_hsr_join(struct dsa_switch *ds, int port, struct
> >> net_device *hsr)
> >>  {
> >> @@ -1279,16 +1279,6 @@ void ksz9477_hsr_join(struct dsa_switch *ds,
> >> int port, struct net_device *hsr)
> >>         /* Program which port(s) shall support HSR */
> >>         ksz_rmw32(dev, REG_HSR_PORT_MAP__4, BIT(port), BIT(port));
> >>
> >> -       /* Forward frames between HSR ports (i.e. bridge together
> >> HSR ports) */
> >> -       if (dev->hsr_ports) {
> >> -               dsa_hsr_foreach_port(hsr_dp, ds, hsr)
> >> -                       hsr_ports |= BIT(hsr_dp->index);
> >> -
> >> -               hsr_ports |= BIT(dsa_upstream_port(ds, port));
> >> -               dsa_hsr_foreach_port(hsr_dp, ds, hsr)
> >> -                       ksz9477_cfg_port_member(dev, hsr_dp->index,
> >> hsr_ports);
> >> -       }
> >> -
> >>         if (!dev->hsr_ports) {
> >>                 /* Enable discarding of received HSR frames */
> >>                 ksz_read8(dev, REG_HSR_ALU_CTRL_0__1, &data);  
> > 
> > This means that KSZ9477 forwarding is dropping frames when HW
> > acceleration is used (for non "neighbour" nodes).
> > 
> > On my setup I only had 2 KSZ9477 devel boards.
> > 
> > And as you wrote - the SW based one works, so extending
> > https://elixir.bootlin.com/linux/v6.12-rc2/source/tools/testing/selftests/net/hsr
> > 
> > would not help in this case.  
> 
> I see. With two boards you can't test the accelerated forwarding. So
> how did you test the forwarding at all? Or are you telling me, that
> this was added to the driver without prior testing (which seems a bit
> bold and unusual)?

The packet forwarding is for generating two frames copies on two HSR
coupled ports on a single KSZ9477:

https://ww1.microchip.com/downloads/aemDocuments/documents/UNG/ApplicationNotes/ApplicationNotes/AN3474-KSZ9477-High-Availability-Seamless-Redundancy-Application-Note-00003474A.pdf

The KSZ9477 chip also supports RX packet duplication removal, but
cannot guarantee 100% success (so as a fallback it is done in SW).

The infrastructure from:
https://elixir.bootlin.com/linux/v6.13/source/tools/testing/selftests/net/hsr/hsr_redbox.sh#L50

is enough to test HW accelerated forwarding (of KSZ9477) from NS1 and
NS2.

> 
> Anyway, do you have any suggestions for debugging this? Unfortunately
> I know almost nothing about this topic. But I can offer to test on my
> setup, at least for now. I don't know how long I will still have
> access to the hardware.

For some reason only frames to neighbours are delivered.

So those are removed at some point (either in KSZ9477 HW or in HSR
driver itself).

Do you have some dumps from tshark/wireshark to share?

> 
> If we can't find a proper solution in the long run, I will probably
> send a patch to disable the accelerated forwarding to at least make
> HSR work by default.

As I've noted above - the HW accelerated forwarding is in the KSZ9477
chip.

The code which you uncomment, is following what I've understood from
the standard (and maybe the bug is somewhere there).

> 
> Thanks
> Frieder




Best regards,

Lukasz Majewski

--

DENX Software Engineering GmbH,      Managing Director: Erika Unter
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email: lukma@...x.de

Content of type "application/pgp-signature" skipped

Powered by blists - more mailing lists