Message-ID: <87plr1hjyc.fsf@cloudflare.com>
Date: Thu, 25 Jul 2024 11:58:19 +0200
From: Jakub Sitnicki <jakub@...udflare.com>
To: Eric Dumazet <edumazet@...gle.com>
Cc: Paolo Abeni <pabeni@...hat.com>,  syzbot
 <syzbot+e15b7e15b8a751a91d9a@...kaller.appspotmail.com>,
  davem@...emloft.net,  dsahern@...nel.org,  kuba@...nel.org,
  linux-kernel@...r.kernel.org,  netdev@...r.kernel.org,
  soheil@...gle.com,  syzkaller-bugs@...glegroups.com,  willemb@...gle.com,
  kernel-team@...udflare.com
Subject: Re: [syzbot] [net?] WARNING in skb_warn_bad_offload (5)

On Tue, Jul 23, 2024 at 10:04 PM +02, Jakub Sitnicki wrote:
> On Fri, Jul 19, 2024 at 09:34 PM +02, Jakub Sitnicki wrote:
>> On Tue, Jul 16, 2024 at 07:54 AM -07, Eric Dumazet wrote:
>>> On Tue, Jul 16, 2024 at 3:17 AM Jakub Sitnicki <jakub@...udflare.com> wrote:
>>>>
>>>> On Tue, Jul 16, 2024 at 12:04 PM +02, Paolo Abeni wrote:
>>>> > On 7/16/24 03:23, syzbot wrote:
>>>> >> syzbot found the following issue on:
>>>> >> HEAD commit:    80ab5445da62 Merge tag 'wireless-next-2024-07-11' of git:/..
>>>> >> git tree:       net-next
>>>> >> console+strace: https://syzkaller.appspot.com/x/log.txt?x=175fb821980000
>>>> >> kernel config:  https://syzkaller.appspot.com/x/.config?x=2dbcdd8641c4638f
>>>> >> dashboard link: https://syzkaller.appspot.com/bug?extid=e15b7e15b8a751a91d9a
>>>> >> compiler:       Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
>>>> >> syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=172bf566980000
>>>> >> C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=12fff535980000
>>>> >> Downloadable assets:
>>>> >> disk image: https://storage.googleapis.com/syzbot-assets/184da3869c30/disk-80ab5445.raw.xz
>>>> >> vmlinux: https://storage.googleapis.com/syzbot-assets/85bfe9b60f21/vmlinux-80ab5445.xz
>>>> >> kernel image: https://storage.googleapis.com/syzbot-assets/06064623a948/bzImage-80ab5445.xz
>>>> >> The issue was bisected to:
>>>> >> commit 10154dbded6d6a2fecaebdfda206609de0f121a9
>>>> >> Author: Jakub Sitnicki <jakub@...udflare.com>
>>>> >> Date:   Wed Jun 26 17:51:26 2024 +0000
>>>> >>      udp: Allow GSO transmit from devices with no checksum offload
>>>> >> bisection log:  https://syzkaller.appspot.com/x/bisect.txt?x=142ccbed980000
>>>> >> final oops:     https://syzkaller.appspot.com/x/report.txt?x=162ccbed980000
>>>> >> console output: https://syzkaller.appspot.com/x/log.txt?x=122ccbed980000
>>>> >> IMPORTANT: if you fix the issue, please add the following tag to the commit:
>>>> >> Reported-by: syzbot+e15b7e15b8a751a91d9a@...kaller.appspotmail.com
>>>> >> Fixes: 10154dbded6d ("udp: Allow GSO transmit from devices with no checksum offload")
>>>> >> skb frag:     00000080: 62 3f 77 e4 0e 82 0d 2f 85 cc 44 ea 25 5a 99 76
>>>> >> skb frag:     00000090: f2 53
>>>> >> ------------[ cut here ]------------
>>>> >> ip6tnl0: caps=(0x00000006401d7869, 0x00000006401d7869)
>>>> >> WARNING: CPU: 0 PID: 5112 at net/core/dev.c:3293 skb_warn_bad_offload+0x166/0x1a0 net/core/dev.c:3291
>>>> >> Modules linked in:
>>>> >> CPU: 0 PID: 5112 Comm: syz-executor391 Not tainted 6.10.0-rc7-syzkaller-01603-g80ab5445da62 #0
>>>> >> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 06/07/2024
>>>> >> RIP: 0010:skb_warn_bad_offload+0x166/0x1a0 net/core/dev.c:3291
>>>> >> Code: e8 5f 94 a3 f8 49 8b 04 24 48 8d 88 a0 03 00 00 48 85 c0 48 0f 44 cd 48 c7 c7 00 cc c5 8c 4c 89 f6 48 89 da e8 fb 92 ff f7 90 <0f> 0b 90 90 5b 41 5c 41 5d 41 5e 41 5f 5d c3 cc cc cc cc 44 89 f9
>>>> >> RSP: 0018:ffffc900034bedc8 EFLAGS: 00010246
>>>> >> RAX: 7d287cad4185da00 RBX: ffff888040cdc0b8 RCX: ffff888023d1bc00
>>>> >> RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
>>>> >> RBP: ffffffff8cc5cbc0 R08: ffffffff815857b2 R09: fffffbfff1c39994
>>>> >> R10: dffffc0000000000 R11: fffffbfff1c39994 R12: ffff888022880518
>>>> >> R13: dffffc0000000000 R14: ffff888040cdc130 R15: ffff888040cdc130
>>>> >> FS:  000055556e9e9380(0000) GS:ffff8880b9400000(0000) knlGS:0000000000000000
>>>> >> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>>> >> CR2: 0000000020001180 CR3: 000000007c876000 CR4: 00000000003506f0
>>>> >> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>>>> >> DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
>>>> >> Call Trace:
>>>> >>   <TASK>
>>>> >>   __skb_gso_segment+0x3be/0x4c0 net/core/gso.c:127
>>>> >>   skb_gso_segment include/net/gso.h:83 [inline]
>>>> >>   validate_xmit_skb+0x585/0x1120 net/core/dev.c:3661
>>>> >>   __dev_queue_xmit+0x17a4/0x3e90 net/core/dev.c:4415
>>>> >>   neigh_output include/net/neighbour.h:542 [inline]
>>>> >>   ip6_finish_output2+0xffa/0x1680 net/ipv6/ip6_output.c:137
>>>> >>   ip6_finish_output+0x41e/0x810 net/ipv6/ip6_output.c:222
>>>> >>   ip6_send_skb+0x112/0x230 net/ipv6/ip6_output.c:1958
>>>> >>   udp_v6_send_skb+0xbf5/0x1870 net/ipv6/udp.c:1292
>>>> >>   udpv6_sendmsg+0x23b3/0x3270 net/ipv6/udp.c:1588
>>>> >>   sock_sendmsg_nosec net/socket.c:730 [inline]
>>>> >>   __sock_sendmsg+0xef/0x270 net/socket.c:745
>>>> >>   ____sys_sendmsg+0x525/0x7d0 net/socket.c:2585
>>>> >>   ___sys_sendmsg net/socket.c:2639 [inline]
>>>> >>   __sys_sendmmsg+0x3b2/0x740 net/socket.c:2725
>>>> >>   __do_sys_sendmmsg net/socket.c:2754 [inline]
>>>> >>   __se_sys_sendmmsg net/socket.c:2751 [inline]
>>>> >>   __x64_sys_sendmmsg+0xa0/0xb0 net/socket.c:2751
>>>> >>   do_syscall_x64 arch/x86/entry/common.c:52 [inline]
>>>> >>   do_syscall_64+0xf3/0x230 arch/x86/entry/common.c:83
>>>> >>   entry_SYSCALL_64_after_hwframe+0x77/0x7f
>>>> >> RIP: 0033:0x7f04f688fe89
>>>> >> Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 01 1a 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48
>>>> >> RSP: 002b:00007ffeebc526e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000133
>>>> >> RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 00007f04f688fe89
>>>> >> RDX: 0000000000000001 RSI: 0000000020003cc0 RDI: 0000000000000003
>>>> >> RBP: 00000000000f4240 R08: 0000000000000000 R09: 0000000000000001
>>>> >> R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffeebc52740
>>>> >> R13: 00007f04f68dd406 R14: 0000000000000003 R15: 00007ffeebc52720
>>>> >>   </TASK>
>>>> >
>>>> > Looking at the console log, the the relevant GSO packet is an UFO one with
>>>> > CSUM_NONE. commit 10154dbded6d6a2fecaebdfda206609de0f121a9 only adjust the skb
>>>> > csum for USO packets. @Jakub S. could you please have a look?
>>>>
>>>> Will do. Thanks for the hint.
>>>
>>> The trigger for the bug is the following :
>>>
>>> setsockopt(3, SOL_IPV6, IPV6_HOPOPTS,
>>> "\0\3\0\0\0\0\0\0\5\2\0\0\0\1\0\302\4\200\0\0\0\5\2\0\6\302\4\0\0\0\1\302"...,
>>> 40) = 0
>>>
>>> Some random IPV6_HOPTS, with multiple IPV6_TLV_JUMBO options
>>>
>>> Non GSO path sends a malformed packet just fine, but GSO complains loudly.
>>>
>>> (flowlabel 0x754ca, hlim 64, next-header Options (0) payload length:
>>> 186) localhost > localhost: HBH
>>> (pad1)(pad1)(pad1)(pad1)(pad1)(pad1)(rtalert: 0x0000)
>>> (pad1)(padn)(jumbo: 2147483648 - payload len != 0) (rtalert: 0x0006)
>>> (jumbo: 1 - already seen)  [|hbhopt]
>>
>> Thank you for the hint. Extracted a reproducer. Fix will follow.
>
> Thanks for the patience.
>
> I've got a fix to propose which pulls the gso_skb->ip_summed tweak added
> to __udp_gso_segment() in commit 10154dbded6d ("udp: Allow GSO transmit
> from devices with no checksum offload") from gso/udp code up to the udp
> layer (udp[_v6]_send_skb()).
>
> This warning can also be triggered for an ipv4 tunnel by turning off
> csum offload (ethtool -K $tnl tx-checksum-ip-generic off). Will extend
> udpgso.sh test to cover both v4 and v6.
>
> [...]

Proposed fix now posted:

https://lore.kernel.org/all/20240725-udp-gso-egress-from-tunnel-v1-0-5e5530ead524@cloudflare.com/

Powered by blists - more mailing lists