linux-kernel - Re: [Intel-wired-lan] bug with rx-udp-gro-forwarding offloading?

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite for Android: free password hash cracker in your pocket
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <CAA85sZtyM+X_oHcpOBNSgF=kmB6k32bpB8FCJN5cVE14YCba+A@mail.gmail.com>
Date:   Mon, 26 Jun 2023 16:25:24 +0200
From:   Ian Kumlien <ian.kumlien@...il.com>
To:     Alexander Lobakin <aleksander.lobakin@...el.com>
Cc:     intel-wired-lan <intel-wired-lan@...ts.osuosl.org>,
        Jakub Kicinski <kuba@...nel.org>,
        Eric Dumazet <edumazet@...gle.com>,
        "netdev@...r.kernel.org" <netdev@...r.kernel.org>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>
Subject: Re: [Intel-wired-lan] bug with rx-udp-gro-forwarding offloading?

On Mon, Jun 26, 2023 at 4:18 PM Alexander Lobakin
<aleksander.lobakin@...el.com> wrote:
>
> From: Ian Kumlien <ian.kumlien@...il.com>
> Date: Sun, 25 Jun 2023 12:59:54 +0200
>
> > It could actually be that it's related to: rx-gro-list but
> > rx-udp-gro-forwarding makes it trigger quicker...  I have yet to
> > trigger it on igb
>
> Hi, the rx-udp-gro-forwarding author here.
>
> (good thing this appeared on IWL, which I read time to time, but please
>  Cc netdev next time)
> (thus +Cc Jakub, Eric, and netdev)

Well, two things, it seems like rx-udp-gro-forwarding accelerates it
but the issue is actually in: rx-gro-list

And since i've only been able to trigger it in ixgbe i thought it
might be a driver issue =)

> > On Sat, Jun 24, 2023 at 10:03 PM Ian Kumlien <ian.kumlien@...il.com> wrote:
> >>
> >> Hi again,
> >>
> >> I suspect that I have rounded this down to the rx-udp-gro-forwarding
> >> option... I know it's not on by default but....
> >>
> >> So, I have a machine with four nics, all using ixgbe, they are all:
> >> 06:00.0 Ethernet controller: Intel Corporation Ethernet Connection
> >> X553 1GbE (rev 11)
> >> 06:00.1 Ethernet controller: Intel Corporation Ethernet Connection
> >> X553 1GbE (rev 11)
> >> 07:00.0 Ethernet controller: Intel Corporation Ethernet Connection
> >> X553 1GbE (rev 11)
> >> 07:00.1 Ethernet controller: Intel Corporation Ethernet Connection
> >> X553 1GbE (rev 11)
> >>
> >> But I have been playing with various... currently i do:
> >> for interface in eno1 eno2 eno3 eno4 ; do
> >>   for offload in ntuple hw-tc-offload rx-gro-list ; do
> >>     ethtool -K $interface $offload on > /dev/null
> >>   done
> >>   ethtool -G $interface rx 8192 tx 8192 > /devYnull
> >> done
> >>
> >> And it all seems to work just fine for my little firewall
> >>
> >> However, enabling rx-udp-gro-forwarding results in the attached oooops
> >> (sorry, can't see more, been recreating by watching shows on HBO
> >> max... )
>
> Where's the mentioned oops? Where's the original message?

Held by the mailing list since i can only get a screenshot of it...
Will attach the latest one to this email
(I wish that i could easily get a larger backtrace but i haven't
looked in further atm)

> Can't this be related to [0]?

Don't know, my main test has been running video streams in the
background - eventually they cause a oops (within 40 minutes or so)
But i doubt it's counted as tunnel data ;)

> rx-udp-gro-forwarding is here for, uhm... 3 years? And UDP GRO in
> general is much longer. Is this a non-mainline kernel?

Mainline 6.3.9 and now  6.4.0

> So many questions :D
>
> >>
> >> The code seems to decode to:
> >> Code: c3 08 66 89 5c 02 04 45 84 e4 0f 85 27 fd ff ff 49 8b 1e 49 8b
> >> ae c8 00 00 00 41 0f b7 86 b8 00 00 00 45 0f b7 a6 b6 00 00 00 <48> 8b
> >> b3 c8 00 00 00 0f b7 8b b6 00 00 00 49 01 ec 48 01 c5 48 8d
> >> All code
> >> ========
> >>    0: c3                    ret
> >>    1: 08 66 89              or     %ah,-0x77(%rsi)
> >>    4: 5c                    pop    %rsp
> >>    5: 02 04 45 84 e4 0f 85 add    -0x7af01b7c(,%rax,2),%al
> >>    c: 27                    (bad)
> >>    d: fd                    std
> >>    e: ff                    (bad)
> >>    f: ff 49 8b              decl   -0x75(%rcx)
> >>   12: 1e                    (bad)
> >>   13: 49 8b ae c8 00 00 00 mov    0xc8(%r14),%rbp
> >>   1a: 41 0f b7 86 b8 00 00 movzwl 0xb8(%r14),%eax
> >>   21: 00
> >>   22: 45 0f b7 a6 b6 00 00 movzwl 0xb6(%r14),%r12d
> >>   29: 00
> >>   2a:* 48 8b b3 c8 00 00 00 mov    0xc8(%rbx),%rsi <-- trapping instruction
> >>   31: 0f b7 8b b6 00 00 00 movzwl 0xb6(%rbx),%ecx
> >>   38: 49 01 ec              add    %rbp,%r12
> >>   3b: 48 01 c5              add    %rax,%rbp
> >>   3e: 48                    rex.W
> >>   3f: 8d                    .byte 0x8d
> >>
> >> Code starting with the faulting instruction
> >> ===========================================
> >>    0: 48 8b b3 c8 00 00 00 mov    0xc8(%rbx),%rsi
> >>    7: 0f b7 8b b6 00 00 00 movzwl 0xb6(%rbx),%ecx
> >>    e: 49 01 ec              add    %rbp,%r12
> >>   11: 48 01 c5              add    %rax,%rbp
> >>   14: 48                    rex.W
> >>   15: 8d                    .byte 0x8d
> >>
> >> But correlating that with the source is beyond me, it could be generic
> >> but i thought i'd send it you first since it's part of the redhat
> >> guide to speeding up udp traffic
> [0]
> https://lore.kernel.org/netdev/f83d79d6-f8d7-a229-941a-7d7427975160@nvidia.com
>
> Thanks,
> Olek

Download attachment "iKVM_capture (1).jpg" of type "image/jpeg" (193781 bytes)