netdev - Re: mlx5 and gre tunneling

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <f3065c55-5123-4cd7-8e93-85a74b150a27@akamai.com>
Date: Tue, 16 Apr 2024 09:55:47 -0400
From: Jason Baron <jbaron@...mai.com>
To: Gal Pressman <gal@...dia.com>, saeedm@...dia.com
Cc: netdev@...r.kernel.org
Subject: Re: mlx5 and gre tunneling

On 4/16/24 8:28 AM, Gal Pressman wrote:
> On 08/04/2024 16:41, Jason Baron wrote:
>> Hi,
>>
>> I recently found an issue where if I send udp traffic in a GRE tunnel
>> over a mellanox 5 NIC where tx-gre-segmentation is enbalbed on the NIC,
>> then packets on the receive side are corrupted to a point that they are
>> never passed up to the user receive socket. I took a look at the
>> received traffic and the inner ip headers appear corrupted as well as
>> the payloads. This reproduces every time for me on both AMD and Intel
>> based x86 systems.
>>
>> The reproducer is quite simple. For example something like this will work:
>>
>> https://urldefense.com/v3/__https://github.com/rom1v/udp-segmentation__;!!GjvTz_vk!TPSVKAaeP_0RAV6hCgRl1GVxyz54xSI1oNXyo8HgWbTXLQ8ZyPRZIlOhPq68YerjtMBMo4bm$
>>
>> It just needs to be modified to actually pass the traffic through the
>> NIC (ie not localhost). As long as the original UDP packet needs to be
>> segmented I see the corruption. That is if it all fits in one packet, I
>> don't see the corruption. Turning off tx-gre-segmentation on the
>> mellanox NIC makes the problem go away (as it gets segmented first in
>> software). Also, I've successfully run this test with other NICs. So
>> this appears to be something specific to the Mellanox NIC.
>>
>> Here's an example one that fails, with the latest upstream (6.8) kernel,
>> for example:
>>
>> driver: mlx5_core
>> version: 6.8.0+
>> firmware-version: 16.35.3502 (MT_0000000242)
>>
>> Let me know if I can fill in any more details.
>>
>> Thanks!
>>
>> -Jason
>>
> 
> Hi Jason, thanks for the report!
> 
> I have managed to reproduce the issue on our side, let me see what went
> wrong.
> 

Hi Gal,

Thanks for looking into this.

We've also found that vxlan encapsulation also fails using the same 
testcase as used for gre tunneling. For vxlan encapsulation if we turn 
off 'tx-udp_tnl-csum-segmentation' then things work again.

Thanks,

-Jason