netdev - Re: [6.12.15][be2net?] Voluntary context switch within RCU read-side critical section!

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <4604b36b-4822-4755-a45c-c37d47a3adc2@blackwall.org>
Date: Wed, 26 Feb 2025 15:11:18 +0200
From: Nikolay Aleksandrov <razor@...ckwall.org>
To: Ian Kumlien <ian.kumlien@...il.com>
Cc: Jakub Kicinski <kuba@...nel.org>,
 Linux Kernel Network Developers <netdev@...r.kernel.org>,
 Sathya Perla <sathya.perla@...adcom.com>
Subject: Re: [6.12.15][be2net?] Voluntary context switch within RCU read-side
 critical section!

On 2/26/25 14:26, Ian Kumlien wrote:
> On Wed, Feb 26, 2025 at 1:00 PM Nikolay Aleksandrov <razor@...ckwall.org> wrote:
>>
>> On 2/26/25 13:52, Ian Kumlien wrote:
>>> On Wed, Feb 26, 2025 at 11:33 AM Nikolay Aleksandrov
>>> <razor@...ckwall.org> wrote:
>>>>
>>>> On 2/26/25 11:55, Ian Kumlien wrote:
>>>>> On Wed, Feb 26, 2025 at 10:24 AM Ian Kumlien <ian.kumlien@...il.com> wrote:
>>>>>>
>>>>>> On Wed, Feb 26, 2025 at 2:05 AM Jakub Kicinski <kuba@...nel.org> wrote:
>>>>>>>
>>>>>>> On Tue, 25 Feb 2025 11:13:47 +0100 Ian Kumlien wrote:
>>>>>>>> Same thing happens in 6.13.4, FYI
>>>>>>>
>>>>>>> Could you do a minor bisection? Does it not happen with 6.11?
>>>>>>> Nothing jumps out at quick look.
>>>>>>
>>>>>> I have to admint that i haven't been tracking it too closely until it
>>>>>> turned out to be an issue
>>>>>> (makes network traffic over wireguard, through that node very slow)
>>>>>>
>>>>>> But i'm pretty sure it was ok in early 6.12.x - I'll try to do a bisect though
>>>>>> (it's a gw to reach a internal server network in the basement, so not
>>>>>> the best setup for this)
>>>>>
>>>>> Since i'm at work i decided to check if i could find all the boot
>>>>> logs, which is actually done nicely by systemd
>>>>> first known bad: 6.11.7-300.fc41.x86_64
>>>>> last known ok: 6.11.6-200.fc40.x86_64
>>>>>
>>>>> Narrows the field for a bisect at least, =)
>>>>>
>>>>
>>>> Saw bridge, took a look. :)
>>>>
>>>> I think there are multiple issues with benet's be_ndo_bridge_getlink()
>>>> because it calls be_cmd_get_hsw_config() which can sleep in multiple
>>>> places, e.g. the most obvious is the mutex_lock() in the beginning of
>>>> be_cmd_get_hsw_config(), then we have the call trace here which is:
>>>> be_cmd_get_hsw_config -> be_mcc_notify_wait -> be_mcc_wait_compl -> usleep_range()
>>>>
>>>> Maybe you updated some tool that calls down that path along with the kernel and system
>>>> so you started seeing it in Fedora 41?
>>>
>>> Could be but it's pretty barebones
>>>
>>>> IMO this has been problematic for a very long time, but obviously it depends on the
>>>> chip type. Could you share your benet chip type to confirm the path?
>>>
>>> I don't know how to find the actual chip information but it's identified as:
>>> Emulex Corporation OneConnect NIC (Skyhawk) (rev 10)
>>>
>>
>> Good, that confirms it. The skyhawk chip falls in the "else" of the block in
>> be_ndo_bridge_getlink() which calls be_cmd_get_hsw_config().
>>
>>>> For the blamed commit I'd go with:
>>>>  commit b71724147e73
>>>>  Author: Sathya Perla <sathya.perla@...adcom.com>
>>>>  Date:   Wed Jul 27 05:26:18 2016 -0400
>>>>
>>>>      be2net: replace polling with sleeping in the FW completion path
>>>>
>>>> This one changed the udelay() (which is safe) to usleep_range() and the spinlock
>>>> to a mutex.
>>>
>>> So, first try will be to try without that patch then, =)
>>>
>>
>> That would be a good try, yes. It is not a straight-forward revert though since a lot
>> of changes have happened since that commit. Let me know if you need help with that,
>> I can prepare the revert to test.
> 
> Yeah, looked at the size of it and... well... I dunno if i'd have the time =)
> 

Can you try the attached patch?
It is on top of net-next (but also applies to Linus' tree):
 git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git

It partially reverts the mentioned commit above (only mutex -> spinlock and usleep -> udelay)
because the commit does many more things.

Also +CC original patch author which I forgot to do.

Thanks,
 Nik


View attachment "0001-benet-fix.patch" of type "text/x-patch" (26486 bytes)