Date:   Thu, 26 Apr 2018 15:37:33 +0200
From:   Michal Vokáč <vokac.m@...il.com>
To:     netdev@...r.kernel.org
Cc:     Andrew Lunn "<andrew@...n.ch>;" Vivien Didelot
         "<vivien.didelot@...oirfairelinux.com>;" Florian Fainelli 
        <f.fainelli@...il.com>
Subject: i.MX6S/DL and QCA8334 switch using DSA driver - CPU port not working

Ahoj,

Sorry for bothering you guys but I could not come up with a more appropriate
place to ask my question.

I would very much appreciate some adivce to the DSA/QCA8K driver usage.
I am not (yet) very educated in networking internals so my wording may not be
right and my expectations may be wrong.

I am working on custom boards based on i.MX6S and i.MX6DL with QCA8334 four
port switch chip. The switch is connected to CPU using MDIO and RGMII
interface.
   _____                 _____________
  |     |               |   QCA8334   |
  |     |  RGMII        |        port2| eth2 <---> LAN (DHCP server)
  | CPU | <------> eth0 |port0        |
  |     |   MDIO        |        port3| eth1 <---> Some device
  |_____| <------>      |_____________|

HW issue is out of question as we are using the HW for a long time with
older kernel and a custom PHY driver for the switch.

I am now in a process of upgrading the kernel from 3.14 to 4.9 and I would
like to get rid of our ugly driver and utilize the mainline code.

I went through a lot of documentation/slides/video regarding the DSA subsystem.
I am still having a hard time to make the switch work as I expect with DSA and
qca8k driver.

I will describe my problem in three steps:

  - 1. The old version - this is what we were using so far.
  - 2. Current state - this is what I was able to come up with.
  - 3. Questions

------------------
1. The old version
------------------

  - Linux 3.14 (Freescale 3.14-1.1.x-imx branch)
  - Custom PHY driver for the switch chip

Device tree snippet:

&fec {
	pinctrl-names = "default";
	pinctrl-0 = <&pinctrl_enet_4>;
	phy-mode = "rgmii";
	phy-reset-gpios = <&gpio1 25 0>;
	phy-reset-duration = <0x14>;
	phy-supply = <&sw2_reg>;
	phy-handle = <&ethphy0>;
	status = "okay";

	mdio {
		#address-cells = <1>;
		#size-cells = <0>;
		ethphy0: ethernet-phy@0 {
			compatible = "ethernet-phy-id004d.d036";
			reg = <0>;
			max-speed = <1000>;
		};
	};
};

In my old setup I see only one ethernet interface.
The driver leaves most internal registers of the switch in its default state.
The major change is that forwarding is enabled not just across all the user
ports (HW default) but to the CPU port as well.

	qca8k_write(priv, 0x0624, 0x007F7F7F);

Regardless of state of the eth0 interface, communication is possible through
the switch. Device connected to port3 can get IP address from local DHCP server.
This is what you get by default without touching a single bit of the switch
internal registers.

Then when I bring up the eth0 interface also the CPU itself can get IP address
from local DHCP server and communicate to LAN or with the device.

----------------
2. Current state
----------------

  - Linux 4.9.84 (Freescale 4.9-1.0.x-imx branch)
  - Mainline DSA drivers
  - Mainline qca8k driver

The Freescale branch does not introduce any changes to the DSA nor to the QCA8K
drivers from mainline.

The QCA8K driver currently supports only the seven-port variant QCA8337.
AFAIK the only difference between the QCA8337 and QCA8334 we use is the number
of ports - 7 vs 4 respectively. So I added "qca,qca8334" compatible string to
the qca8k_of_match table in qca8k.c driver.

These corresponding configs are enabled:

  CONFIG_HAVE_NET_DSA=y
  CONFIG_NET_DSA=y
  CONFIG_NET_DSA_TAG_QCA=y
  CONFIG_NET_DSA_QCA8K=y

Device tree snippet:

&fec {
	pinctrl-names = "default";
	pinctrl-0 = <&pinctrl_enet_4>;
	phy-mode = "rgmii";
	phy-reset-gpios = <&gpio1 25 0>;
	phy-reset-duration = <0x14>;
	phy-supply = <&sw2_reg>;
	phy-handle = <&ethphy0>;
	status = "okay";

	mdio {
		#address-cells = <1>;
		#size-cells = <0>;

		phy_port2: phy@1 {
		    reg = <1>;
		};

		phy_port3: phy@2 {
		    reg = <2>;
		};

		switch@0 {
			compatible = "qca,qca8334";
			#address-cells = <1>;
			#size-cells = <0>;
			reg = <0>;

			ports {
				#address-cells = <1>;
				#size-cells = <0>;
				ethphy0: port@0 {
					reg = <0>;
					label = "cpu";
					phy-mode = "rgmii";
					ethernet = <&fec>;
					fixed-link {
						speed = <1000>;
						full-duplex;
					};
				};
				ethphy2: port@2 {
					reg = <2>;
					label = "eth2";
					phy-handle = <&phy_port2>;
				};

				ethphy3: port@3 {
					reg = <3>;
					label = "eth1";
					phy-handle = <&phy_port3>;
				};
			};
		};
	};
};

  # dmesg
  [    1.954903] libphy: fec_enet_mii_bus: probed
  [    1.955457] mdio_bus 2188000.ethernet-1:00: mdio_device_register
  [    1.955763] dsa2: enabling port: 2
  [    1.955770] dsa2: enabling port: 3
  [    1.956493] fec 2188000.ethernet eth0: registered PHC device 0
  ...
  [    2.920810] dsa2: enabling port: 2
  [    2.920817] dsa2: enabling port: 3
  [    2.920878] DSA: switch 0 0 parsed
  [    2.920884] DSA: tree 0 parsed
  [    2.927628] libphy: dsa slave smi: probed
  [    2.928161] Generic PHY 2188000.ethernet-1:01: attached PHY driver [Generic PHY] (mii_bus:phy_addr=2188000.ethernet-1:01, irq=-1)
  [    2.929326] Generic PHY dsa-0.0:02: attached PHY driver [Generic PHY] (mii_bus:phy_addr=dsa-0.0:02, irq=-1)
  
  # ip a
  2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
       link/ether 68:7c:d5:04:01:5a brd ff:ff:ff:ff:ff:ff
  3: eth2@...0: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN group default qlen 1000
       link/ether 68:7c:d5:04:01:5a brd ff:ff:ff:ff:ff:ff
  4: eth1@...0: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN group default qlen 1000
       link/ether 68:7c:d5:04:01:5a brd ff:ff:ff:ff:ff:ff

At this point I am able to bring up all three interfaces, create a bridge, add
eth1 and eth2 to the bridge and bring up the bridge. But the bridge does not
work. No communication is possible from one port to the other.

To make the bridge work I need to enable forwarding across all the switch ports
at setup.

--- a/drivers/net/dsa/qca8k.c
+++ b/drivers/net/dsa/qca8k.c
@@ -578,12 +578,12 @@ qca8k_setup(struct dsa_switch *ds)
    		if (ds->enabled_port_mask & BIT(i))
    			qca8k_port_set_status(priv, i, 0);
    
-	/* Forward all unknown frames to CPU port for Linux processing */
+	/* Forward all unknown frames to all pors */
    	qca8k_write(priv, QCA8K_REG_GLOBAL_FW_CTRL1,
    		    BIT(0) << QCA8K_GLOBAL_FW_CTRL1_IGMP_DP_S |
-		    BIT(0) << QCA8K_GLOBAL_FW_CTRL1_BC_DP_S |
-		    BIT(0) << QCA8K_GLOBAL_FW_CTRL1_MC_DP_S |
-		    BIT(0) << QCA8K_GLOBAL_FW_CTRL1_UC_DP_S);
+		    0x7f << QCA8K_GLOBAL_FW_CTRL1_BC_DP_S |
+		    0x7f << QCA8K_GLOBAL_FW_CTRL1_MC_DP_S |
+		    0x7f << QCA8K_GLOBAL_FW_CTRL1_UC_DP_S);
    
    	/* Setup connection between CPU port & user ports */
    	for (i = 0; i < DSA_MAX_PORTS; i++) {
--

Then when I do:

  # ifconfig eth0 up
  # ifconfig eth1 up
  # ifconfig eth2 up
  # brctl addbr br0
  # brctl addif eth1
  # brctl addif eth2
  # ifconfig br0 up

the bridge works fine.
But I am still not able to make work the CPU port though.

  # udhcpc -i eth2
  Sending discover...
  [FOREVER]

The same for eth1, eth2 and br0.

I suspect the problem may be at different levels:

  - The RGMII interface is not properly configured
   -- at the CPU side, or
   -- at the switch chip side.
  - Some setup that I have not done needs to be done (in userspace).

------------
3. Questions
------------

Q: What is the correct interface that the CPU should use to talk to the outside
world? I am convinced any of the eth1, eth2, br0 interfaces can be used.

Q: Is there something else that needs to be done to get networking going on the
CPU interface?

Q: Is my device tree correct?

Q: Lets assume the CPU port is working. Is NFS boot of the CPU possible from at
least one port of the switch?

Any hints how to deal with that will be much appreciated.
BR, Michal

Powered by blists - more mailing lists