netdev - Re: [PATCH net 1/2] mlx5: fix possible ptp queue fifo overflow

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <fbffc35d-81d9-a4ab-6d0b-0ccb417914ea@meta.com>
Date:   Wed, 25 Jan 2023 21:24:18 +0000
From:   Vadim Fedorenko <vadfed@...a.com>
To:     Saeed Mahameed <saeed@...nel.org>
CC:     Gal Pressman <gal@...dia.com>,
        "netdev@...r.kernel.org" <netdev@...r.kernel.org>,
        Aya Levin <ayal@...dia.com>,
        Saeed Mahameed <saeedm@...dia.com>,
        Jakub Kicinski <kuba@...nel.org>
Subject: Re: [PATCH net 1/2] mlx5: fix possible ptp queue fifo overflow

On 25/01/2023 20:33, Saeed Mahameed wrote:
> On 25 Jan 14:42, Vadim Fedorenko wrote:
>> On 24/01/2023 14:39, Gal Pressman wrote:
>>> Anyway, I'd like to zoom out for a second, the whole fifo was designed
>>> under the assumption that completions are in-order (this is a guarantee
>>> for all SQs, not just ptp ones!), this fix seems more of a bandage that
>>> potentially hides a more severe issue.
>>>
>>>>
>>>> It really shows that CQE are coming OOO sometimes.
>>>
>>> Can we reproduce it somehow?
>>> Can you please try to update your firmware version? I'm quite confident
>>> that this issue is fixed already.
>>>
> 
> Hi Vadim,

Hi Saeed,
Thanks for taking a look at the issue.

> As Gal pointed out above,
> we shouldn't be seeing OOO on TX data path, otherwise, what's the point
> of the fifo ? Also you can't have a proper reseliency since it seems when
> this OOO happen the skb_cc, which is derived from the we_counter seems to
> fall out of range which makes me think it can be a completely random
> value, so we can't really be protected from all OOO scenarios.
>

Well, from my log I haven't seen any random values actually, just real 
1-2 cqe out of the order, and with my patches these CQEs are simply 
dropped and CQE timestamp is returned instead of port timestamp. Which 
is good enough.

> This is clearly a FW bug and we will get to the bottom of
> this internally, Can you please create a bug request ?
> 

Thanks for confirming FW bug, let's work on it.

> For the SKB leak, I will take the 2nd patch as is and improve it as
> necessary if that's ok with you.
> 
I have already prepared v3 with the changes that Jakub suggested. I'll 
reorder patches to have fix for has_room() and SKB leak fix in the first 
(or even separate) patch and OOO fixes in the second one and we can 
review them separately.

Thanks,
Vadim