linux-kernel - Re: [Intel-wired-lan] bug with rx-udp-gro-forwarding offloading?

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <e7e49ed5-09e2-da48-002d-c7eccc9f9451@intel.com>
Date:   Mon, 26 Jun 2023 16:17:33 +0200
From:   Alexander Lobakin <aleksander.lobakin@...el.com>
To:     Ian Kumlien <ian.kumlien@...il.com>
CC:     intel-wired-lan <intel-wired-lan@...ts.osuosl.org>,
        Jakub Kicinski <kuba@...nel.org>,
        Eric Dumazet <edumazet@...gle.com>,
        "netdev@...r.kernel.org" <netdev@...r.kernel.org>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>
Subject: Re: [Intel-wired-lan] bug with rx-udp-gro-forwarding offloading?

From: Ian Kumlien <ian.kumlien@...il.com>
Date: Sun, 25 Jun 2023 12:59:54 +0200

> It could actually be that it's related to: rx-gro-list but
> rx-udp-gro-forwarding makes it trigger quicker...  I have yet to
> trigger it on igb

Hi, the rx-udp-gro-forwarding author here.

(good thing this appeared on IWL, which I read time to time, but please
 Cc netdev next time)
(thus +Cc Jakub, Eric, and netdev)

> 
> On Sat, Jun 24, 2023 at 10:03 PM Ian Kumlien <ian.kumlien@...il.com> wrote:
>>
>> Hi again,
>>
>> I suspect that I have rounded this down to the rx-udp-gro-forwarding
>> option... I know it's not on by default but....
>>
>> So, I have a machine with four nics, all using ixgbe, they are all:
>> 06:00.0 Ethernet controller: Intel Corporation Ethernet Connection
>> X553 1GbE (rev 11)
>> 06:00.1 Ethernet controller: Intel Corporation Ethernet Connection
>> X553 1GbE (rev 11)
>> 07:00.0 Ethernet controller: Intel Corporation Ethernet Connection
>> X553 1GbE (rev 11)
>> 07:00.1 Ethernet controller: Intel Corporation Ethernet Connection
>> X553 1GbE (rev 11)
>>
>> But I have been playing with various... currently i do:
>> for interface in eno1 eno2 eno3 eno4 ; do
>>   for offload in ntuple hw-tc-offload rx-gro-list ; do
>>     ethtool -K $interface $offload on > /dev/null
>>   done
>>   ethtool -G $interface rx 8192 tx 8192 > /devYnull
>> done
>>
>> And it all seems to work just fine for my little firewall
>>
>> However, enabling rx-udp-gro-forwarding results in the attached oooops
>> (sorry, can't see more, been recreating by watching shows on HBO
>> max... )

Where's the mentioned oops? Where's the original message?

Can't this be related to [0]?

rx-udp-gro-forwarding is here for, uhm... 3 years? And UDP GRO in
general is much longer. Is this a non-mainline kernel?
So many questions :D

>>
>> The code seems to decode to:
>> Code: c3 08 66 89 5c 02 04 45 84 e4 0f 85 27 fd ff ff 49 8b 1e 49 8b
>> ae c8 00 00 00 41 0f b7 86 b8 00 00 00 45 0f b7 a6 b6 00 00 00 <48> 8b
>> b3 c8 00 00 00 0f b7 8b b6 00 00 00 49 01 ec 48 01 c5 48 8d
>> All code
>> ========
>>    0: c3                    ret
>>    1: 08 66 89              or     %ah,-0x77(%rsi)
>>    4: 5c                    pop    %rsp
>>    5: 02 04 45 84 e4 0f 85 add    -0x7af01b7c(,%rax,2),%al
>>    c: 27                    (bad)
>>    d: fd                    std
>>    e: ff                    (bad)
>>    f: ff 49 8b              decl   -0x75(%rcx)
>>   12: 1e                    (bad)
>>   13: 49 8b ae c8 00 00 00 mov    0xc8(%r14),%rbp
>>   1a: 41 0f b7 86 b8 00 00 movzwl 0xb8(%r14),%eax
>>   21: 00
>>   22: 45 0f b7 a6 b6 00 00 movzwl 0xb6(%r14),%r12d
>>   29: 00
>>   2a:* 48 8b b3 c8 00 00 00 mov    0xc8(%rbx),%rsi <-- trapping instruction
>>   31: 0f b7 8b b6 00 00 00 movzwl 0xb6(%rbx),%ecx
>>   38: 49 01 ec              add    %rbp,%r12
>>   3b: 48 01 c5              add    %rax,%rbp
>>   3e: 48                    rex.W
>>   3f: 8d                    .byte 0x8d
>>
>> Code starting with the faulting instruction
>> ===========================================
>>    0: 48 8b b3 c8 00 00 00 mov    0xc8(%rbx),%rsi
>>    7: 0f b7 8b b6 00 00 00 movzwl 0xb6(%rbx),%ecx
>>    e: 49 01 ec              add    %rbp,%r12
>>   11: 48 01 c5              add    %rax,%rbp
>>   14: 48                    rex.W
>>   15: 8d                    .byte 0x8d
>>
>> But correlating that with the source is beyond me, it could be generic
>> but i thought i'd send it you first since it's part of the redhat
>> guide to speeding up udp traffic
[0]
https://lore.kernel.org/netdev/f83d79d6-f8d7-a229-941a-7d7427975160@nvidia.com

Thanks,
Olek