Message-ID: <0dafa9d7-9c79-f367-a343-8ad38f7bde07@molgen.mpg.de>
Date:   Mon, 14 Mar 2022 15:36:19 +0100
From:   Donald Buczek <buczek@...gen.mpg.de>
To:     Manish Chopra <manishc@...vell.com>,
        Linus Torvalds <torvalds@...ux-foundation.org>,
        Jakub Kicinski <jakub.kicinski@...ronome.com>
Cc:     Paul Menzel <pmenzel@...gen.mpg.de>,
        "netdev@...r.kernel.org" <netdev@...r.kernel.org>,
        Ariel Elior <aelior@...vell.com>,
        Alok Prasad <palok@...vell.com>,
        Prabhakar Kushwaha <pkushwaha@...vell.com>,
        "David S. Miller" <davem@...emloft.net>,
        Greg KH <gregkh@...uxfoundation.org>,
        "stable@...r.kernel.org" <stable@...r.kernel.org>,
        "it+netdev@...gen.mpg.de" <it+netdev@...gen.mpg.de>,
        "regressions@...ts.linux.dev" <regressions@...ts.linux.dev>
Subject: Re: [EXT] Re: [PATCH v2 net-next 1/2] bnx2x: Utilize firmware
 7.13.21.0

Dear Manish,

On 3/11/22 1:11 PM, Manish Chopra wrote:
>> -----Original Message-----
>> From: Linus Torvalds <torvalds@...ux-foundation.org>
>> Sent: Thursday, March 10, 2022 3:48 AM
>> To: Manish Chopra <manishc@...vell.com>
>> Cc: Paul Menzel <pmenzel@...gen.mpg.de>; kuba@...nel.org;
>> netdev@...r.kernel.org; Ariel Elior <aelior@...vell.com>; Alok Prasad
>> <palok@...vell.com>; Prabhakar Kushwaha <pkushwaha@...vell.com>;
>> David S. Miller <davem@...emloft.net>; Greg KH
>> <gregkh@...uxfoundation.org>; stable@...r.kernel.org;
>> it+netdev@...gen.mpg.de; regressions@...ts.linux.dev
>> Subject: Re: [EXT] Re: [PATCH v2 net-next 1/2] bnx2x: Utilize firmware
>> 7.13.21.0
>>
>> On Wed, Mar 9, 2022 at 11:46 AM Manish Chopra <manishc@...vell.com>
>> wrote:
>>>
>>> This has not changed anything functionally from driver/device perspective,
>> FW is still being loaded only when device is opened.
>>> bnx2x_init_firmware() [I guess, perhaps the name is misleading] just
>> request_firmware() to prepare the metadata to be used when device will be
>> opened.
>>
>> So how do you explain the report by Paul Menzel that things used to work and
>> no longer work now?
>>
> 
> The issue which Paul mentioned had to do with "/lib/firmware/bnx2x/* file not found" when driver probes, which was introduced by the patch in subject,
> And the commit e13ad1443684 ("bnx2x: fix driver load from initrd") fixes this issue. So things should work as it is with the mentioned fixed commit.
> The only discussion led by this problem now is why the request_firmware() was moved early on [from open() to probe()] by the patch in subject.
> I explained the intention to do this in my earlier emails and let me add more details below -
> 
> Note that we have just moved request_firmware() logic, *not* something significant which has to do with actual FW loading or device initialization from the
> FW file data which could cause significant functional change for this device/driver, FW load/init part still stays in open flow.
> 
> Before the patch in subject, driver used to only work with fixed/specific FW version file whose version was statically known to the driver function at probe() time to take
> some decision to fail the function probe early in the system if the function is supposed to run with a FW version which is not the same version loaded on the device by another PF (different ENV).
> Now when we sent this new FW patch (in subject) then we got feedback from community to maintain backward compatibility with older FW versions as well and we did it in same v2 patch legitimately,
> just that now we can work with both older or newer FW file so we need this run time FW version information to cache (based on request_firmware() return success value for an old FW file or new FW file)
> which will be used in follow up probe() flows to decide the function probe failure early If there could be FW version mismatches against the loaded FW on the device by other PFs already

There might be something more wrong with the patch in the subject: The usability of the ports from a single card (with older firmware?) now depends on the order the ports are enabled (first port enabled is working, second port enabled is not working, driver complaining about a firmware mismatch).

In the following examples, the driver was not built-in to the kernel but loaded from the root filesystem instead, so there is no initramfs related problem here.

For the records:

root@ira:~# dmesg|grep bnx2x
[   18.749871] bnx2x 0000:45:00.0: msix capability found
[   18.766534] bnx2x 0000:45:00.0: part number 394D4342-31373735-31314131-473331
[   18.799198] bnx2x 0000:45:00.0: 32.000 Gb/s available PCIe bandwidth (5.0 GT/s PCIe x8 link)
[   18.807638] bnx2x 0000:45:00.1: msix capability found
[   18.824509] bnx2x 0000:45:00.1: part number 394D4342-31373735-31314131-473331
[   18.857171] bnx2x 0000:45:00.1: 32.000 Gb/s available PCIe bandwidth (5.0 GT/s PCIe x8 link)
[   18.865619] bnx2x 0000:46:00.0: msix capability found
[   18.882636] bnx2x 0000:46:00.0: part number 394D4342-31373735-31314131-473331
[   18.915196] bnx2x 0000:46:00.0: 32.000 Gb/s available PCIe bandwidth (5.0 GT/s PCIe x8 link)
[   18.923636] bnx2x 0000:46:00.1: msix capability found
[   18.940505] bnx2x 0000:46:00.1: part number 394D4342-31373735-31314131-473331
[   18.973167] bnx2x 0000:46:00.1: 32.000 Gb/s available PCIe bandwidth (5.0 GT/s PCIe x8 link)
[   46.480660] bnx2x 0000:45:00.0 net04: renamed from eth4
[   46.494677] bnx2x 0000:45:00.1 net05: renamed from eth5
[   46.508544] bnx2x 0000:46:00.0 net06: renamed from eth6
[   46.524641] bnx2x 0000:46:00.1 net07: renamed from eth7
root@ira:~# ls /lib/firmware/bnx2x/
bnx2x-e1-6.0.34.0.fw   bnx2x-e1-7.13.1.0.fw   bnx2x-e1-7.8.2.0.fw     bnx2x-e1h-7.12.30.0.fw  bnx2x-e1h-7.8.19.0.fw  bnx2x-e2-7.10.51.0.fw  bnx2x-e2-7.8.17.0.fw
bnx2x-e1-6.2.5.0.fw    bnx2x-e1-7.13.11.0.fw  bnx2x-e1h-6.0.34.0.fw   bnx2x-e1h-7.13.1.0.fw   bnx2x-e1h-7.8.2.0.fw   bnx2x-e2-7.12.30.0.fw  bnx2x-e2-7.8.19.0.fw
bnx2x-e1-6.2.9.0.fw    bnx2x-e1-7.13.15.0.fw  bnx2x-e1h-6.2.5.0.fw    bnx2x-e1h-7.13.11.0.fw  bnx2x-e2-6.0.34.0.fw   bnx2x-e2-7.13.1.0.fw   bnx2x-e2-7.8.2.0.fw
bnx2x-e1-7.0.20.0.fw   bnx2x-e1-7.13.21.0.fw  bnx2x-e1h-6.2.9.0.fw    bnx2x-e1h-7.13.15.0.fw  bnx2x-e2-6.2.5.0.fw    bnx2x-e2-7.13.11.0.fw
bnx2x-e1-7.0.23.0.fw   bnx2x-e1-7.2.16.0.fw   bnx2x-e1h-7.0.20.0.fw   bnx2x-e1h-7.13.21.0.fw  bnx2x-e2-6.2.9.0.fw    bnx2x-e2-7.13.15.0.fw
bnx2x-e1-7.0.29.0.fw   bnx2x-e1-7.2.51.0.fw   bnx2x-e1h-7.0.23.0.fw   bnx2x-e1h-7.2.16.0.fw   bnx2x-e2-7.0.20.0.fw   bnx2x-e2-7.13.21.0.fw
bnx2x-e1-7.10.51.0.fw  bnx2x-e1-7.8.17.0.fw   bnx2x-e1h-7.0.29.0.fw   bnx2x-e1h-7.2.51.0.fw   bnx2x-e2-7.0.23.0.fw   bnx2x-e2-7.2.16.0.fw
bnx2x-e1-7.12.30.0.fw  bnx2x-e1-7.8.19.0.fw   bnx2x-e1h-7.10.51.0.fw  bnx2x-e1h-7.8.17.0.fw   bnx2x-e2-7.0.29.0.fw   bnx2x-e2-7.2.51.0.fw

Now with v5.10.95, the first kernel of the series which includes fdcfabd0952d ("bnx2x: Utilize firmware 7.13.21.0") and later:

root@ira:~# dmesg -w &
[...]
root@ira:~# ip link set net04 up
[   88.504536] bnx2x 0000:45:00.0 net04: using MSI-X  IRQs: sp 47  fp[0] 49 ... fp[7] 56
root@ira:~# ip link set net05 up
[   90.825820] bnx2x: [bnx2x_compare_fw_ver:2380(net05)]bnx2x with FW 120d07 was already loaded which mismatches my 150d07 FW. Aborting
RTNETLINK answers: Device or resource busy
root@ira:~# ip link set net04 down
root@ira:~# ip link set net05 down
root@ira:~# ip link set net05 up
[  114.462448] bnx2x 0000:45:00.1 net05: using MSI-X  IRQs: sp 58  fp[0] 60 ... fp[7] 67
root@ira:~# ip link set net04 up
[  117.247763] bnx2x: [bnx2x_compare_fw_ver:2380(net04)]bnx2x with FW 120d07 was already loaded which mismatches my 150d07 FW. Aborting
RTNETLINK answers: Device or resource busy

With v5.10.94, both ports work fine:

root@ira:~# dmesg -w &
[...]
root@ira:~# ip link set net04 up
[  133.126647] bnx2x 0000:45:00.0 net04: using MSI-X  IRQs: sp 47  fp[0] 49 ... fp[7] 56
root@ira:~# ip link set net05 up
[  136.215169] bnx2x 0000:45:00.1 net05: using MSI-X  IRQs: sp 58  fp[0] 60 ... fp[7] 67

Best
   Donald


> So we need to understand why we should not call request_firmware() in probe or at least what's really harmful in doing that in probe() if some of the follow up probe flows needs
> some of the metadata info (like the run time FW versions info in this case which we get based on request_firmware() return value), we could avoid this but we don't want
> to add some ugly/unsuitable file APIs checks to know which FW version file is available on the file system if there is already an API request_firmware() available for this to be used.
> 
> Please let us know. Thanks.
> 
>> You can't do request_firmware() early. When you actually then push the
>> firmware to the device is immaterial - but request_firmware() has to be done
>> after the system is up and running.
>>
>>                   Linus
> 


-- 
Donald Buczek
buczek@...gen.mpg.de
Tel: +49 30 8413 1433

Powered by blists - more mailing lists