Message-ID: <10e9ead9-5adc-5065-0c13-702aabd5dcb0@arinc9.com>
Date:   Tue, 13 Sep 2022 13:42:28 +0300
From:   Arınç ÜNAL <arinc.unal@...nc9.com>
To:     Sergio Paracuellos <sergio.paracuellos@...il.com>,
        Lorenzo Bianconi <lorenzo@...nel.org>
Cc:     netdev <netdev@...r.kernel.org>, paulmck@...nel.org
Subject: Re: mtk_eth_soc for mt7621 won't work after 6.0-rc1

On 13.09.2022 11:31, Sergio Paracuellos wrote:
> Hi Lorenzo,
> 
> On Tue, Sep 13, 2022 at 5:32 AM Sergio Paracuellos
> <sergio.paracuellos@...il.com> wrote:
>>
>> Hi Lorenzo,
>>
>> On Mon, Sep 12, 2022 at 10:30 PM Lorenzo Bianconi <lorenzo@...nel.org> wrote:
>>>
>>>> Hi Lorenzo,
>>>>
>>>> On 12.09.2022 21:01, Lorenzo Bianconi wrote:
>>>>>>> Ethernet for MT7621 SoCs no longer works after changes introduced to
>>>>>>> mtk_eth_soc with 6.0-rc1. Ethernet interfaces initialise fine. Packets are
>>>>>>> sent out from the interface fine but won't be received on the interface.
>>>>>>>
>>>>>>> Tested with MT7530 DSA switch connected to gmac0 and ICPlus IP1001 PHY
>>>>>>> connected to gmac1 of the SoC.
>>>>>>>
>>>>>>> Last working kernel is 5.19. The issue is present on 6.0-rc5.
>>>>>>>
>>>>>>> Arınç
>>>>>>
>>>>>> Hi Arınç,
>>>>>>
>>>>>> thx for testing and reporting the issue. Can you please identify
>>>>>> the offending commit running git bisect?
>>>>>>
>>>>>> Regards,
>>>>>> Lorenzo
>>>>>
>>>>> Hi Arınç,
>>>>>
>>>>> just a small update. I tested a mt7621 based board (Buffalo WSR-1166DHP) with
>>>>> OpenWrt master + my mtk_eth_soc series and it works fine. Can you please
>>>>> provide more details about your development board/environment?
>>>>
>>>> I've got a GB-PC2, Sergio has got a GB-PC1. We both use Neil's gnubee-tools
>>>> which makes an image with filesystem and any Linux kernel of choice with
>>>> slight modifications (maybe not at all) on the kernel.
>>>>
>>>> https://github.com/neilbrown/gnubee-tools
>>>>
>>>> Sergio experiences the same problem on GB-PC1.
>>>
>>> ack, can you please run git bisect in order to identify the offending commit?
>>> What is the latest kernel version that is working properly? 5.19.8?
>>
>> I'll try to get time today to properly bisect and identify the
>> offending commit. I get a working platform with 5.19.8, yes but with
>> v6-rc-1 my network is totally broken.
> 
> + [cc: Paul E. McKenney <paulmck@...nel.org> as commit author]
> 
> Ok, so I have bisected the issue to:
> 1cf1144e8473e8c3180ac8b91309e29b6acfd95f] rcu-tasks: Be more patient
> for RCU Tasks boot-time testing
> 
> This is the complete bisect log:
> 
> $ git bisect log
> git bisect start
> # good: [70cb6afe0e2ff1b7854d840978b1849bffb3ed21] Linux 5.19.8
> git bisect good 70cb6afe0e2ff1b7854d840978b1849bffb3ed21
> # bad: [568035b01cfb107af8d2e4bd2fb9aea22cf5b868] Linux 6.0-rc1
> git bisect bad 568035b01cfb107af8d2e4bd2fb9aea22cf5b868
> # good: [3d7cb6b04c3f3115719235cc6866b10326de34cd] Linux 5.19
> git bisect good 3d7cb6b04c3f3115719235cc6866b10326de34cd
> # bad: [b44f2fd87919b5ae6e1756d4c7ba2cbba22238e1] Merge tag
> 'drm-next-2022-08-03' of git://anongit.freedesktop.org/drm/drm
> git bisect bad b44f2fd87919b5ae6e1756d4c7ba2cbba22238e1
> # bad: [526942b8134cc34d25d27f95dfff98b8ce2f6fcd] Merge tag
> 'ata-5.20-rc1' of
> git://git.kernel.org/pub/scm/linux/kernel/git/dlemoal/libata
> git bisect bad 526942b8134cc34d25d27f95dfff98b8ce2f6fcd
> # good: [2e7a95156d64667a8ded606829d57c6fc92e41df] Merge tag
> 'regmap-v5.20' of
> git://git.kernel.org/pub/scm/linux/kernel/git/broonie/regmap
> git bisect good 2e7a95156d64667a8ded606829d57c6fc92e41df
> # good: [c013d0af81f60cc7dbe357c4e2a925fb6738dbfe] Merge tag
> 'for-5.20/block-2022-07-29' of git://git.kernel.dk/linux-block
> git bisect good c013d0af81f60cc7dbe357c4e2a925fb6738dbfe
> # bad: [aad26f55f47a33d6de3df65f0b18e2886059ed6d] Merge tag 'docs-6.0'
> of git://git.lwn.net/linux
> git bisect bad aad26f55f47a33d6de3df65f0b18e2886059ed6d
> # good: [c2a24a7a036b3bd3a2e6c66730dfc777cae6540a] Merge tag
> 'v5.20-p1' of git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6
> git bisect good c2a24a7a036b3bd3a2e6c66730dfc777cae6540a
> # bad: [34bc7b454dc31f75a0be7ee8ab378135523d7c51] Merge branch
> 'ctxt.2022.07.05a' into HEAD
> git bisect bad 34bc7b454dc31f75a0be7ee8ab378135523d7c51
> # bad: [e72ee5e1a866b85cb6c3d4c80a1125976020a7e8] rcu-tasks: Use
> delayed_work to delay rcu_tasks_verify_self_tests()
> git bisect bad e72ee5e1a866b85cb6c3d4c80a1125976020a7e8
> # good: [f90f19da88bfe32dd1fdfd104de4c0526a3be701] rcu-tasks: Make RCU
> Tasks Trace stall warning handle idle offline tasks
> git bisect good f90f19da88bfe32dd1fdfd104de4c0526a3be701
> # good: [dc7d54b45170e1e3ced9f86718aa4274fd727790] rcu-tasks: Pull in
> tasks blocked within RCU Tasks Trace readers
> git bisect good dc7d54b45170e1e3ced9f86718aa4274fd727790
> # good: [e386b6725798eec07facedf4d4bb710c079fd25c] rcu-tasks:
> Eliminate RCU Tasks Trace IPIs to online CPUs
> git bisect good e386b6725798eec07facedf4d4bb710c079fd25c
> # good: [eea3423b162d5d5cdc08af23e8ee2c2d1134fd07] rcu-tasks: Update comments
> git bisect good eea3423b162d5d5cdc08af23e8ee2c2d1134fd07
> # bad: [1cf1144e8473e8c3180ac8b91309e29b6acfd95f] rcu-tasks: Be more
> patient for RCU Tasks boot-time testing
> git bisect bad 1cf1144e8473e8c3180ac8b91309e29b6acfd95f
> # first bad commit: [1cf1144e8473e8c3180ac8b91309e29b6acfd95f]
> rcu-tasks: Be more patient for RCU Tasks boot-time testing
> 
> I don't really understand the relationship with my broken network
> issue. I am using debian buster and the effect I see is that when the
> network interface becomes up it hangs waiting for a "task running to
> raise network interfaces". After about one minute the system boots,
> the login prompt is shown but I cannot configure at all network
> interfaces: dhclient does not respond and manually ifconfig does not
> help also:
> 
> root@...bee:~#
> root@...bee:~# dhclient ethblack
> ^C
> root@...bee:~# ifconfig ethblack 192.168.1.101
> root@...bee:~# ping 19^C
> root@...bee:~# ping 192.168.1.47
> PING 192.168.1.47 (192.168.1.47) 56(84) bytes of data.
> ^C
> --- 192.168.1.47 ping statistics ---
> 3 packets transmitted, 0 received, 100% packet loss, time 120ms
> 
> I have tried to revert the bad commit directly in v6.0-rc1 but
> conflicts appeared with the git revert command in
> 'kernel/rcu/tasks.h', so I am not sure what I can do now.

I've pinpointed the issue to 23233e577ef973c2c5d0dd757a0a4605e34ecb57 
("net: ethernet: mtk_eth_soc: rely on page_pool for single page 
buffers"). Ethernet works fine after reverting this and newer commits 
for mtk_eth_soc.

Arınç

Powered by blists - more mailing lists