netdev - Re: Potential regression/bug in net/mlx5 driver

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:   Fri, 14 Apr 2023 10:37:07 -0400
From:   Paul Moore <paul@...l-moore.com>
To:     Jakub Kicinski <kuba@...nel.org>
Cc:     Saeed Mahameed <saeedm@...dia.com>,
        Leon Romanovsky <leon@...nel.org>,
        Linux regressions mailing list <regressions@...ts.linux.dev>,
        Saeed Mahameed <saeed@...nel.org>,
        Shay Drory <shayd@...dia.com>, netdev@...r.kernel.org,
        selinux@...r.kernel.org, Tariq Toukan <tariqt@...dia.com>
Subject: Re: Potential regression/bug in net/mlx5 driver

On Thu, Apr 13, 2023 at 11:26 PM Jakub Kicinski <kuba@...nel.org> wrote:
> On Thu, 13 Apr 2023 20:03:18 -0700 Saeed Mahameed wrote:
> > On 13 Apr 15:51, Jakub Kicinski wrote:
> > >On Thu, 13 Apr 2023 15:34:21 -0700 Saeed Mahameed wrote:

...

> > >The question is who's supposed to be paying the price of mlx5 being
> > >used for old and new parts? What is fair to expect from the user
> > >when the FW Paul has presumably works just fine for him?
> > >
> > Upgrade FW when possible, it is always easier than upgrading the kernel.
> > Anyways this was a very rare FW/Arch bug, We should've exposed an
> > explicit cap for this new type of PF when we had the chance, now it's too
> > late since a proper fix will require FW and Driver upgrades and breaking
> > the current solution we have over other OSes as well.
> >
> > Yes I can craft an if condition to explicitly check for chip id and FW
> > version for this corner case, which has no precedence in mlx5, but I prefer
> > to ask to upgrade FW first, and if that's an acceptable solution, I would
> > like to keep the mlx5 clean and device agnostic as much as possible.
>
> IMO you either need a fully fleshed out FW update story, with advanced
> warnings for a few releases, distributing the FW via linux-firmware or
> fwupdmgr or such.  Or deal with the corner cases in the driver :(
>
> We can get Paul to update, sure, but if he noticed so quickly the
> question remains how many people out in the wild will get affected
> and not know what the cause is?

I think it is that last bit which is the real issue, at least from a
regression standpoint.  I didn't see anything on the console or in the
logs to indicate that ancient/buggy FW was the issue, even once I
bisected the kernel (which your average user isn't going to do) it
wasn't clear that it was a FW problem.  Perhaps the mlx5 driver should
perform a simple FW version check on initialization and
pr_warn()/pr_err() if the loaded FW is below a support threshold?
Seeing a "mlx5: hey idiot, your FW is ancient, you need to upgrade!"
line on my console/dmesg would have sent me in the right direction and
likely avoided all of this ...

-- 
paul-moore.com