Message-ID: <YyB2L8dfnJfnrqWI@lore-desk>
Date:   Tue, 13 Sep 2022 14:23:11 +0200
From:   Lorenzo Bianconi <lorenzo@...nel.org>
To:     Arınç ÜNAL <arinc.unal@...nc9.com>
Cc:     Sergio Paracuellos <sergio.paracuellos@...il.com>,
        netdev <netdev@...r.kernel.org>, paulmck@...nel.org
Subject: Re: mtk_eth_soc for mt7621 won't work after 6.0-rc1

> On 13.09.2022 14:07, Sergio Paracuellos wrote:
> > Hi Lorenzo,
> > 
> > On Tue, Sep 13, 2022 at 12:58 PM Lorenzo Bianconi <lorenzo@...nel.org> wrote:
> > > 
> > > > On 13.09.2022 11:31, Sergio Paracuellos wrote:
> > > > > Hi Lorenzo,
> > > > > 
> > > > > On Tue, Sep 13, 2022 at 5:32 AM Sergio Paracuellos
> > > > > <sergio.paracuellos@...il.com> wrote:
> > > > > > 
> > > > > > Hi Lorenzo,
> > > > > > 
> > > > > > On Mon, Sep 12, 2022 at 10:30 PM Lorenzo Bianconi <lorenzo@...nel.org> wrote:
> > > > > > > 
> > > > > > > > Hi Lorenzo,
> > > > > > > > 
> > > > > > > > On 12.09.2022 21:01, Lorenzo Bianconi wrote:
> > > > > > > > > > > Ethernet for MT7621 SoCs no longer works after changes introduced to
> > > > > > > > > > > mtk_eth_soc with 6.0-rc1. Ethernet interfaces initialise fine. Packets are
> > > > > > > > > > > sent out from the interface fine but won't be received on the interface.
> > > > > > > > > > > 
> > > > > > > > > > > Tested with MT7530 DSA switch connected to gmac0 and ICPlus IP1001 PHY
> > > > > > > > > > > connected to gmac1 of the SoC.
> > > > > > > > > > > 
> > > > > > > > > > > Last working kernel is 5.19. The issue is present on 6.0-rc5.
> > > > > > > > > > > 
> > > > > > > > > > > Arınç
> > > > > > > > > > 
> > > > > > > > > > Hi Arınç,
> > > > > > > > > > 
> > > > > > > > > > thx for testing and reporting the issue. Can you please identify
> > > > > > > > > > the offending commit running git bisect?
> > > > > > > > > > 
> > > > > > > > > > Regards,
> > > > > > > > > > Lorenzo
> > > > > > > > > 
> > > > > > > > > Hi Arınç,
> > > > > > > > > 
> > > > > > > > > just a small update. I tested a mt7621 based board (Buffalo WSR-1166DHP) with
> > > > > > > > > OpenWrt master + my mtk_eth_soc series and it works fine. Can you please
> > > > > > > > > provide more details about your development board/environment?
> > > > > > > > 
> > > > > > > > I've got a GB-PC2, Sergio has got a GB-PC1. We both use Neil's gnubee-tools
> > > > > > > > which makes an image with filesystem and any Linux kernel of choice with
> > > > > > > > slight modifications (maybe not at all) on the kernel.
> > > > > > > > 
> > > > > > > > https://github.com/neilbrown/gnubee-tools
> > > > > > > > 
> > > > > > > > Sergio experiences the same problem on GB-PC1.
> > > > > > > 
> > > > > > > ack, can you please run git bisect in order to identify the offending commit?
> > > > > > > What is the latest kernel version that is working properly? 5.19.8?
> > > > > > 
> > > > > > I'll try to get time today to properly bisect and identify the
> > > > > > offending commit. I get a working platform with 5.19.8, yes but with
> > > > > > v6-rc-1 my network is totally broken.
> > > > > 
> > > > > + [cc: Paul E. McKenney <paulmck@...nel.org> as commit author]
> > > > > 
> > > > > Ok, so I have bisected the issue to:
> > > > > 1cf1144e8473e8c3180ac8b91309e29b6acfd95f] rcu-tasks: Be more patient
> > > > > for RCU Tasks boot-time testing
> > > > > 
> > > > > This is the complete bisect log:
> > > > > 
> > > > > $ git bisect log
> > > > > git bisect start
> > > > > # good: [70cb6afe0e2ff1b7854d840978b1849bffb3ed21] Linux 5.19.8
> > > > > git bisect good 70cb6afe0e2ff1b7854d840978b1849bffb3ed21
> > > > > # bad: [568035b01cfb107af8d2e4bd2fb9aea22cf5b868] Linux 6.0-rc1
> > > > > git bisect bad 568035b01cfb107af8d2e4bd2fb9aea22cf5b868
> > > > > # good: [3d7cb6b04c3f3115719235cc6866b10326de34cd] Linux 5.19
> > > > > git bisect good 3d7cb6b04c3f3115719235cc6866b10326de34cd
> > > > > # bad: [b44f2fd87919b5ae6e1756d4c7ba2cbba22238e1] Merge tag
> > > > > 'drm-next-2022-08-03' of git://anongit.freedesktop.org/drm/drm
> > > > > git bisect bad b44f2fd87919b5ae6e1756d4c7ba2cbba22238e1
> > > > > # bad: [526942b8134cc34d25d27f95dfff98b8ce2f6fcd] Merge tag
> > > > > 'ata-5.20-rc1' of
> > > > > git://git.kernel.org/pub/scm/linux/kernel/git/dlemoal/libata
> > > > > git bisect bad 526942b8134cc34d25d27f95dfff98b8ce2f6fcd
> > > > > # good: [2e7a95156d64667a8ded606829d57c6fc92e41df] Merge tag
> > > > > 'regmap-v5.20' of
> > > > > git://git.kernel.org/pub/scm/linux/kernel/git/broonie/regmap
> > > > > git bisect good 2e7a95156d64667a8ded606829d57c6fc92e41df
> > > > > # good: [c013d0af81f60cc7dbe357c4e2a925fb6738dbfe] Merge tag
> > > > > 'for-5.20/block-2022-07-29' of git://git.kernel.dk/linux-block
> > > > > git bisect good c013d0af81f60cc7dbe357c4e2a925fb6738dbfe
> > > > > # bad: [aad26f55f47a33d6de3df65f0b18e2886059ed6d] Merge tag 'docs-6.0'
> > > > > of git://git.lwn.net/linux
> > > > > git bisect bad aad26f55f47a33d6de3df65f0b18e2886059ed6d
> > > > > # good: [c2a24a7a036b3bd3a2e6c66730dfc777cae6540a] Merge tag
> > > > > 'v5.20-p1' of git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6
> > > > > git bisect good c2a24a7a036b3bd3a2e6c66730dfc777cae6540a
> > > > > # bad: [34bc7b454dc31f75a0be7ee8ab378135523d7c51] Merge branch
> > > > > 'ctxt.2022.07.05a' into HEAD
> > > > > git bisect bad 34bc7b454dc31f75a0be7ee8ab378135523d7c51
> > > > > # bad: [e72ee5e1a866b85cb6c3d4c80a1125976020a7e8] rcu-tasks: Use
> > > > > delayed_work to delay rcu_tasks_verify_self_tests()
> > > > > git bisect bad e72ee5e1a866b85cb6c3d4c80a1125976020a7e8
> > > > > # good: [f90f19da88bfe32dd1fdfd104de4c0526a3be701] rcu-tasks: Make RCU
> > > > > Tasks Trace stall warning handle idle offline tasks
> > > > > git bisect good f90f19da88bfe32dd1fdfd104de4c0526a3be701
> > > > > # good: [dc7d54b45170e1e3ced9f86718aa4274fd727790] rcu-tasks: Pull in
> > > > > tasks blocked within RCU Tasks Trace readers
> > > > > git bisect good dc7d54b45170e1e3ced9f86718aa4274fd727790
> > > > > # good: [e386b6725798eec07facedf4d4bb710c079fd25c] rcu-tasks:
> > > > > Eliminate RCU Tasks Trace IPIs to online CPUs
> > > > > git bisect good e386b6725798eec07facedf4d4bb710c079fd25c
> > > > > # good: [eea3423b162d5d5cdc08af23e8ee2c2d1134fd07] rcu-tasks: Update comments
> > > > > git bisect good eea3423b162d5d5cdc08af23e8ee2c2d1134fd07
> > > > > # bad: [1cf1144e8473e8c3180ac8b91309e29b6acfd95f] rcu-tasks: Be more
> > > > > patient for RCU Tasks boot-time testing
> > > > > git bisect bad 1cf1144e8473e8c3180ac8b91309e29b6acfd95f
> > > > > # first bad commit: [1cf1144e8473e8c3180ac8b91309e29b6acfd95f]
> > > > > rcu-tasks: Be more patient for RCU Tasks boot-time testing
> > > > > 
> > > > > I don't really understand the relationship with my broken network
> > > > > issue. I am using debian buster and the effect I see is that when the
> > > > > network interface becomes up it hangs waiting for a "task running to
> > > > > raise network interfaces". After about one minute the system boots,
> > > > > the login prompt is shown but I cannot configure at all network
> > > > > interfaces: dhclient does not respond and manually ifconfig does not
> > > > > help also:
> > > > > 
> > > > > root@...bee:~#
> > > > > root@...bee:~# dhclient ethblack
> > > > > ^C
> > > > > root@...bee:~# ifconfig ethblack 192.168.1.101
> > > > > root@...bee:~# ping 19^C
> > > > > root@...bee:~# ping 192.168.1.47
> > > > > PING 192.168.1.47 (192.168.1.47) 56(84) bytes of data.
> > > > > ^C
> > > > > --- 192.168.1.47 ping statistics ---
> > > > > 3 packets transmitted, 0 received, 100% packet loss, time 120ms
> > > > > 
> > > > > I have tried to revert the bad commit directly in v6.0-rc1 but
> > > > > conflicts appeared with the git revert command in
> > > > > 'kernel/rcu/tasks.h', so I am not sure what I can do now.
> > > > 
> > > > I've pinpointed the issue to 23233e577ef973c2c5d0dd757a0a4605e34ecb57 ("net:
> > > > ethernet: mtk_eth_soc: rely on page_pool for single page buffers"). Ethernet
> > > > works fine after reverting this and newer commits for mtk_eth_soc.
> > > 
> > > Hi Arınç,
> > > 
> > > yes, I run some bisect here as well and this seems the offending commit. Can
> > > you please try the patch below?
> > > 
> > > Regards,
> > > Lorenzo
> > > 
> > > diff --git a/drivers/net/ethernet/mediatek/mtk_eth_soc.c b/drivers/net/ethernet/mediatek/mtk_eth_soc.c
> > > index ec617966c953..67a64a2272b9 100644
> > > --- a/drivers/net/ethernet/mediatek/mtk_eth_soc.c
> > > +++ b/drivers/net/ethernet/mediatek/mtk_eth_soc.c
> > > @@ -1470,7 +1470,7 @@ static void mtk_update_rx_cpu_idx(struct mtk_eth *eth)
> > > 
> > >   static bool mtk_page_pool_enabled(struct mtk_eth *eth)
> > >   {
> > > -       return !eth->hwlro;
> > > +       return MTK_HAS_CAPS(eth->soc->caps, MTK_NETSYS_V2);
> > >   }
> > > 
> > >   static struct page_pool *mtk_create_page_pool(struct mtk_eth *eth,
> > 
> > I have applied this patch on the top of v6-0-rc5 and the network is
> > back, so this patch seems to fix the network issue for my GNUBee pC1:
> > 
> > Tested-by: Sergio Paracuellos <sergio.paracuellos@...il.com>
> 
> Can confirm the same behaviour on my GB-PC2.
> 
> Tested-by: Arınç ÜNAL <arinc.unal@...nc9.com>

I debugged a bit more the problem and the issue is due to a 2 bytes alignment
introduced by mt7621 on packet data.
Since mt7621 is a low-end SoC and I do not have other SoCs for testing, I will
enable xdp support just for MT7986 for the moment. Thanks a lot for reporting the
issue and for testing.

Regards,
Lorenzo

> 
> Arınç

Download attachment "signature.asc" of type "application/pgp-signature" (229 bytes)

Powered by blists - more mailing lists