Message-ID: <42a2f96f-34cd-4d95-99d5-3b4c4af226af@kontron.de>
Date: Wed, 29 Jan 2025 18:07:49 +0100
From: Frieder Schrempf <frieder.schrempf@...tron.de>
To: Lukasz Majewski <lukma@...x.de>
Cc: Andrew Lunn <andrew@...n.ch>,
 "netdev@...r.kernel.org" <netdev@...r.kernel.org>
Subject: Re: KSZ9477 HSR Offloading

On 29.01.25 2:58 PM, Lukasz Majewski wrote:
> Hi Frieder,
> 
>> Hi Lukasz,
>>
>> On 29.01.25 12:17 PM, Lukasz Majewski wrote:
>>> Hi Frieder,
>>>   
>>>> On 29.01.25 8:24 AM, Frieder Schrempf wrote:  
>>>>> Hi Andrew,
>>>>>
>>>>> On 28.01.25 6:51 PM, Andrew Lunn wrote:    
>>>>>> On Tue, Jan 28, 2025 at 05:14:46PM +0100, Frieder Schrempf
>>>>>> wrote:    
>>>>>>> Hi,
>>>>>>>
>>>>>>> I'm trying out HSR support on KSZ9477 with v6.12. My setup looks
>>>>>>> like this:
>>>>>>>
>>>>>>> +-------------+         +-------------+
>>>>>>> |             |         |             |
>>>>>>> |   Node A    |         |   Node D    |
>>>>>>> |             |         |             |
>>>>>>> |             |         |             |
>>>>>>> | LAN1   LAN2 |         | LAN1   LAN2 |
>>>>>>> +--+-------+--+         +--+------+---+
>>>>>>>    |       |               |      |
>>>>>>>    |       +---------------+      |
>>>>>>>    |                              |
>>>>>>>    |       +---------------+      |
>>>>>>>    |       |               |      |
>>>>>>> +--+-------+--+         +--+------+---+
>>>>>>> | LAN1   LAN2 |         | LAN1   LAN2 |
>>>>>>> |             |         |             |
>>>>>>> |             |         |             |
>>>>>>> |   Node B    |         |   Node C    |
>>>>>>> |             |         |             |
>>>>>>> +-------------+         +-------------+
>>>>>>>
>>>>>>> On each device the LAN1 and LAN2 are added as HSR slaves. Then I
>>>>>>> try to do ping tests between each of the HSR interfaces.
>>>>>>>
>>>>>>> The result is that I can reach the neighboring nodes just fine,
>>>>>>> but I can't reach the remote node that needs packages to be
>>>>>>> forwarded through the other nodes. For example I can't ping from
>>>>>>> node A to C.
>>>>>>>
>>>>>>> I've tried to disable HW offloading in the driver and then
>>>>>>> everything starts working.
>>>>>>>
>>>>>>> Is this a problem with HW offloading in the KSZ driver, or am I
>>>>>>> missing something essential?    
>>>
>>> Thanks for looking and testing such large scale setup.
>>>   
>>>>>>
>>>>>> How are IP addresses configured? I assume you have a bridge, LAN1
>>>>>> and LAN2 are members of the bridge, and the IP address is on the
>>>>>> bridge interface?    
>>>>>
>>>>> I have a HSR interface on each node that covers LAN1 and LAN2 as
>>>>> slaves and the IP addresses are on those HSR interfaces. For node
>>>>> A:
>>>>>
>>>>> ip link add name hsr type hsr slave1 lan1 slave2 lan2 supervision
>>>>> 45 version 1
>>>>> ip addr add 172.20.1.1/24 dev hsr
>>>>>
>>>>> The other nodes have the addresses 172.20.1.2/24, 172.20.1.3/24
>>>>> and 172.20.1.4/24 respectively.
>>>>>
>>>>> Then on node A, I'm doing:
>>>>>
>>>>> ping 172.20.1.2 # neighboring node B works
>>>>> ping 172.20.1.4 # neighboring node D works
>>>>> ping 172.20.1.3 # remote node C works only if I disable
>>>>> offloading    
>>>>
>>>> BTW, it's enough to disable the offloading of the forwarding for
>>>> HSR frames to make it work.
>>>>
>>>> --- a/drivers/net/dsa/microchip/ksz9477.c
>>>> +++ b/drivers/net/dsa/microchip/ksz9477.c
>>>> @@ -1267,7 +1267,7 @@ int ksz9477_tc_cbs_set_cinc(struct ksz_device
>>>> *dev, int port, u32 val)
>>>>   * Moreover, the NETIF_F_HW_HSR_FWD feature is also enabled, as
>>>> HSR frames
>>>>   * can be forwarded in the switch fabric between HSR ports.
>>>>   */
>>>> -#define KSZ9477_SUPPORTED_HSR_FEATURES (NETIF_F_HW_HSR_DUP |
>>>> NETIF_F_HW_HSR_FWD)
>>>> +#define KSZ9477_SUPPORTED_HSR_FEATURES (NETIF_F_HW_HSR_DUP)
>>>>
>>>>  void ksz9477_hsr_join(struct dsa_switch *ds, int port, struct
>>>> net_device *hsr)
>>>>  {
>>>> @@ -1279,16 +1279,6 @@ void ksz9477_hsr_join(struct dsa_switch *ds,
>>>> int port, struct net_device *hsr)
>>>>         /* Program which port(s) shall support HSR */
>>>>         ksz_rmw32(dev, REG_HSR_PORT_MAP__4, BIT(port), BIT(port));
>>>>
>>>> -       /* Forward frames between HSR ports (i.e. bridge together
>>>> HSR ports) */
>>>> -       if (dev->hsr_ports) {
>>>> -               dsa_hsr_foreach_port(hsr_dp, ds, hsr)
>>>> -                       hsr_ports |= BIT(hsr_dp->index);
>>>> -
>>>> -               hsr_ports |= BIT(dsa_upstream_port(ds, port));
>>>> -               dsa_hsr_foreach_port(hsr_dp, ds, hsr)
>>>> -                       ksz9477_cfg_port_member(dev, hsr_dp->index,
>>>> hsr_ports);
>>>> -       }
>>>> -
>>>>         if (!dev->hsr_ports) {
>>>>                 /* Enable discarding of received HSR frames */
>>>>                 ksz_read8(dev, REG_HSR_ALU_CTRL_0__1, &data);  
>>>
>>> This means that KSZ9477 forwarding is dropping frames when HW
>>> acceleration is used (for non "neighbour" nodes).
>>>
>>> On my setup I only had 2 KSZ9477 devel boards.
>>>
>>> And as you wrote - the SW based one works, so extending
>>> https://elixir.bootlin.com/linux/v6.12-rc2/source/tools/testing/selftests/net/hsr
>>>
>>> would not help in this case.  
>>
>> I see. With two boards you can't test the accelerated forwarding. So
>> how did you test the forwarding at all? Or are you telling me, that
>> this was added to the driver without prior testing (which seems a bit
>> bold and unusual)?
> 
> The packet forwarding is for generating two frames copies on two HSR
> coupled ports on a single KSZ9477:

Isn't that what duplication aka NETIF_F_HW_HSR_DUP is for?

> 
> https://ww1.microchip.com/downloads/aemDocuments/documents/UNG/ApplicationNotes/ApplicationNotes/AN3474-KSZ9477-High-Availability-Seamless-Redundancy-Application-Note-00003474A.pdf
> 
> The KSZ9477 chip also supports RX packet duplication removal, but
> cannot guarantee 100% success (so as a fallback it is done in SW).
> 
> The infrastructure from:
> https://elixir.bootlin.com/linux/v6.13/source/tools/testing/selftests/net/hsr/hsr_redbox.sh#L50
> 
> is enough to test HW accelerated forwarding (of KSZ9477) from NS1 and
> NS2.

I'm not really sure if I get it. In this setup NS1 and NS2 are connected
via HSR link (two physical links). On one side packets are sent
duplicated on both physical ports. On the receiving side the duplication
is removed and one packet is forwarded to the CPU.

Where is forwarding involved here? Isn't forwarding only for cases with
one intermediate node between the sending and receiving node?

> 
>>
>> Anyway, do you have any suggestions for debugging this? Unfortunately
>> I know almost nothing about this topic. But I can offer to test on my
>> setup, at least for now. I don't know how long I will still have
>> access to the hardware.
> 
> For some reason only frames to neighbours are delivered.
> 
> So those are removed at some point (either in KSZ9477 HW or in HSR
> driver itself).
> 
> Do you have some dumps from tshark/wireshark to share?
> 
>>
>> If we can't find a proper solution in the long run, I will probably
>> send a patch to disable the accelerated forwarding to at least make
>> HSR work by default.
> 
> As I've noted above - the HW accelerated forwarding is in the KSZ9477
> chip.

Yeah, but if the HW accelerated forwarding doesn't work it would be
better to use no acceleration and have it work in SW at least by
default, right?

> 
> The code which you uncomment, is following what I've understood from
> the standard (and maybe the bug is somewhere there).

Ok, thanks for explaining. I will see if I can find some time to gather
some more information on the problem.

Powered by blists - more mailing lists