Message-ID: <e462a91e-8bea-8b72-481c-4a36699e4149@mojatatu.com>
Date: Tue, 23 May 2023 08:36:35 -0300
From: Pedro Tammela <pctammela@...atatu.com>
To: Peilin Ye <yepeilin.cs@...il.com>
Cc: "David S. Miller" <davem@...emloft.net>,
 Eric Dumazet <edumazet@...gle.com>, Jakub Kicinski <kuba@...nel.org>,
 Paolo Abeni <pabeni@...hat.com>, Jamal Hadi Salim <jhs@...atatu.com>,
 Cong Wang <xiyou.wangcong@...il.com>, Jiri Pirko <jiri@...nulli.us>,
 Peilin Ye <peilin.ye@...edance.com>, Daniel Borkmann <daniel@...earbox.net>,
 John Fastabend <john.fastabend@...il.com>, Hillf Danton <hdanton@...a.com>,
 netdev@...r.kernel.org, linux-kernel@...r.kernel.org,
 Cong Wang <cong.wang@...edance.com>, Vlad Buslov <vladbu@...dia.com>
Subject: Re: [PATCH v2 net 6/6] net/sched: qdisc_destroy() old ingress and
 clsact Qdiscs before grafting

On 23/05/2023 01:40, Peilin Ye wrote:
> Hi Pedro,
> 
> On Tue, May 23, 2023 at 12:51:44AM -0300, Pedro Tammela wrote:
>> With V2 patches 5 and 6 applied I was still able to trigger an oops.
>>
>> Branch is 'net' + patches 5 & 6:
>> 145f639b9403 (HEAD -> main) net/sched: qdisc_destroy() old ingress and
>> clsact Qdiscs before grafting
>> 1aac74ef9673 net/sched: Refactor qdisc_graft() for ingress and clsact Qdiscs
>> 18c40a1cc1d9 (origin/main, origin/HEAD) net/handshake: Fix sock->file
>> allocation
>>
>> Kernel config is the same as in the syzbot report.
>> Note that this was on a _single core_ VM.
>> I will double check if v1 is triggering this issue (basically run the repro
>> for a long time). For multi-core my VM is running OOM even on a 32Gb system.
>> I will check if we have a spare server to run the repro.
> 
> Thanks for testing this, but the syzbot reproducer creates ingress Qdiscs
> under TC_H_ROOT, which isn't covered by [6/6] i.e. it exercises the
> "!ingress" path in qdisc_graft().  I think that's why you are still seeing
> the oops.  Adding sch_{ingress,clsact} to TC_H_ROOT is no longer possible
> after [1,2/6], and I think we'll need a different reproducer for [5,6/6].
> 
> However I just noticed that for some reason my git-send-email in my new
> setup didn't auto-generate From: tags with my work email, so Author: will
> be my personal email (I have to send patches from personal email to avoid
> "[External] " subject prefixes) ... I will fix it in v3 soon.  Sorry in
> advance for spamming.
> 
> Thanks,
> Peilin Ye
> 

I see,
We need to make sure then, when the time comes, that all the required 
patches are back ported in the same bundle so we don't have a partial 
fix; given that they target different commit tags.

I was still able to trigger an oops with the full patchset:

[  104.944353][ T6588] ------------[ cut here ]------------
[  104.944896][ T6588] jump label: negative count!
[  104.945780][ T6588] WARNING: CPU: 0 PID: 6588 at 
kernel/jump_label.c:263 static_key_slow_try_dec+0xf2/0x110
[  104.946795][ T6588] Modules linked in:
[  104.947111][ T6588] CPU: 0 PID: 6588 Comm: repro Not tainted 
6.4.0-rc2-00191-g4a3f9100193d #3
[  104.947765][ T6588] Hardware name: QEMU Standard PC (i440FX + PIIX, 
1996), BIOS 1.16.0-debian-1.16.0-5 04/01/2014
[  104.948557][ T6588] RIP: 0010:static_key_slow_try_dec+0xf2/0x110
[  104.949064][ T6588] Code: d5 ff e8 c1 33 d5 ff 44 89 e8 5b 5d 41 5c 
41 5d c3 44 89 e5 e9 66 ff ff ff e8 aa 33 d5 ff 48 c7 c7 00 9c 56 8a e8 
4e ce 9c ff <0f> 0b eb ae 48 89 df e8 02 4b 28 00 e9 42 ff ff ff 66 66 
2e 0f 1f
[  104.951134][ T6588] RSP: 0018:ffffc900079cf2c0 EFLAGS: 00010286
[  104.951646][ T6588] RAX: 0000000000000000 RBX: ffffffff9213a160 RCX: 
0000000000000000
[  104.952269][ T6588] RDX: ffff888112f83b80 RSI: ffffffff814c7747 RDI: 
0000000000000001
[  104.952901][ T6588] RBP: 00000000ffffffff R08: 0000000000000001 R09: 
0000000000000000
[  104.953523][ T6588] R10: 0000000000000001 R11: 0000000000000001 R12: 
00000000ffffffff
[  104.954133][ T6588] R13: ffff88816a514001 R14: 0000000000000001 R15: 
ffffffff8e7b0680
[  104.954746][ T6588] FS:  00007f76c65d56c0(0000) 
GS:ffff8881f5a00000(0000) knlGS:0000000000000000
[  104.955430][ T6588] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  104.955941][ T6588] CR2: 00007f9a40357a50 CR3: 000000011461e000 CR4: 
0000000000350ef0
[  104.956559][ T6588] Call Trace:
[  104.956829][ T6588]  <TASK>
[  104.957062][ T6588]  ? clsact_egress_block_get+0x40/0x40
[  104.957507][ T6588]  static_key_slow_dec+0x60/0xc0
[  104.957906][ T6588]  qdisc_create+0xa45/0x1090
[  104.958274][ T6588]  ? tc_get_qdisc+0xb70/0xb70
[  104.958646][ T6588]  tc_modify_qdisc+0x491/0x1b70
[  104.959031][ T6588]  ? qdisc_create+0x1090/0x1090
[  104.959420][ T6588]  ? bpf_lsm_capable+0x9/0x10
[  104.959797][ T6588]  ? qdisc_create+0x1090/0x1090
[  104.960175][ T6588]  rtnetlink_rcv_msg+0x439/0xd30
[  104.960625][ T6588]  ? rtnl_getlink+0xb10/0xb10
[  104.960995][ T6588]  ? __x64_sys_sendmmsg+0x9c/0x100
[  104.961397][ T6588]  ? do_syscall_64+0x38/0xb0
[  104.961767][ T6588]  ? netdev_core_pick_tx+0x390/0x390
[  104.962195][ T6588]  netlink_rcv_skb+0x166/0x440
[  104.962584][ T6588]  ? rtnl_getlink+0xb10/0xb10
[  104.962954][ T6588]  ? netlink_ack+0x1370/0x1370
[  104.963336][ T6588]  ? kasan_set_track+0x25/0x30
[  104.963731][ T6588]  ? netlink_deliver_tap+0x1b1/0xd00
[  104.964156][ T6588]  netlink_unicast+0x530/0x800
[  104.964537][ T6588]  ? netlink_attachskb+0x880/0x880
[  104.964951][ T6588]  ? __sanitizer_cov_trace_switch+0x54/0x90
[  104.965405][ T6588]  ? __phys_addr_symbol+0x30/0x70
[  104.965793][ T6588]  ? __check_object_size+0x323/0x740
[  104.966205][ T6588]  netlink_sendmsg+0x90b/0xe10
[  104.966577][ T6588]  ? netlink_unicast+0x800/0x800
[  104.966965][ T6588]  ? bpf_lsm_socket_sendmsg+0x9/0x10
[  104.967374][ T6588]  ? netlink_unicast+0x800/0x800
[  104.967761][ T6588]  sock_sendmsg+0xd9/0x180
[  104.968109][ T6588]  ____sys_sendmsg+0x264/0x910
[  104.968490][ T6588]  ? kernel_sendmsg+0x50/0x50
[  104.968871][ T6588]  ? __copy_msghdr+0x460/0x460
[  104.969244][ T6588]  ? lockdep_hardirqs_on_prepare+0x410/0x410
[  104.969716][ T6588]  ? find_held_lock+0x2d/0x110
[  104.970088][ T6588]  ___sys_sendmsg+0x11d/0x1b0
[  104.970491][ T6588]  ? do_recvmmsg+0x700/0x700
[  104.970853][ T6588]  ? __fget_files+0x260/0x420
[  104.971221][ T6588]  ? reacquire_held_locks+0x4b0/0x4b0
[  104.971649][ T6588]  ? __fget_files+0x282/0x420
[  104.972018][ T6588]  ? __fget_light+0xe6/0x270
[  104.972384][ T6588]  __sys_sendmmsg+0x18e/0x430
[  104.972766][ T6588]  ? __ia32_sys_sendmsg+0xb0/0xb0
[  104.973167][ T6588]  ? reacquire_held_locks+0x4b0/0x4b0
[  104.973588][ T6588]  ? rcu_is_watching+0x12/0xb0
[  104.973958][ T6588]  ? xfd_validate_state+0x5d/0x180
[  104.974355][ T6588]  ? restore_fpregs_from_fpstate+0xc1/0x1d0
[  104.974796][ T6588]  ? unlock_page_memcg+0x2d0/0x2d0
[  104.975178][ T6588]  ? do_futex+0x350/0x350
[  104.975501][ T6588]  __x64_sys_sendmmsg+0x9c/0x100
[  104.975867][ T6588]  ? syscall_enter_from_user_mode+0x26/0x80
[  104.976307][ T6588]  do_syscall_64+0x38/0xb0
[  104.976649][ T6588]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  104.977088][ T6588] RIP: 0033:0x7f76c66ee89d
[  104.977417][ T6588] Code: 5d c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 
0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 
24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4b 05 0e 00 f7 d8 64 
89 01 48
[  104.978835][ T6588] RSP: 002b:00007f76c65d4c68 EFLAGS: 00000203 
ORIG_RAX: 0000000000000133
[  104.979468][ T6588] RAX: ffffffffffffffda RBX: 00007f76c65d5cdc RCX: 
00007f76c66ee89d
[  104.980073][ T6588] RDX: 040000000000009f RSI: 00000000200002c0 RDI: 
0000000000000007
[  104.980739][ T6588] RBP: 00007f76c65d4de0 R08: 0000000100000001 R09: 
0000000000000000
[  104.981352][ T6588] R10: 0000000000000000 R11: 0000000000000203 R12: 
fffffffffffffeb8
[  104.981968][ T6588] R13: 0000000000000002 R14: 00007ffce5a5fd90 R15: 
00007f76c65b5000
[  104.982587][ T6588]  </TASK>
[  104.982833][ T6588] Kernel panic - not syncing: kernel: panic_on_warn 
set ...
[  104.983391][ T6588] CPU: 0 PID: 6588 Comm: repro Not tainted 
6.4.0-rc2-00191-g4a3f9100193d #3
[  104.984054][ T6588] Hardware name: QEMU Standard PC (i440FX + PIIX, 
1996), BIOS 1.16.0-debian-1.16.0-5 04/01/2014
[  104.984854][ T6588] Call Trace:
[  104.985121][ T6588]  <TASK>
[  104.985359][ T6588]  dump_stack_lvl+0xd9/0x1b0
[  104.985733][ T6588]  panic+0x689/0x730
[  104.986053][ T6588]  ? panic_smp_self_stop+0xa0/0xa0
[  104.986467][ T6588]  ? show_trace_log_lvl+0x28e/0x400
[  104.986888][ T6588]  ? static_key_slow_try_dec+0xf2/0x110
[  104.987329][ T6588]  check_panic_on_warn+0xab/0xb0
[  104.987729][ T6588]  __warn+0xf2/0x380
[  104.988046][ T6588]  ? static_key_slow_try_dec+0xf2/0x110
[  104.988476][ T6588]  report_bug+0x3bc/0x580
[  104.988838][ T6588]  handle_bug+0x3c/0x70
[  104.989169][ T6588]  exc_invalid_op+0x17/0x40
[  104.989529][ T6588]  asm_exc_invalid_op+0x1a/0x20
[  104.989914][ T6588] RIP: 0010:static_key_slow_try_dec+0xf2/0x110
[  104.990402][ T6588] Code: d5 ff e8 c1 33 d5 ff 44 89 e8 5b 5d 41 5c 
41 5d c3 44 89 e5 e9 66 ff ff ff e8 aa 33 d5 ff 48 c7 c7 00 9c 56 8a e8 
4e ce 9c ff <0f> 0b eb ae 48 89 df e8 02 4b 28 00 e9 42 ff ff ff 66 66 
2e 0f 1f
[  104.990702][ T6588] RSP: 0018:ffffc900079cf2c0 EFLAGS: 00010286
[  104.990702][ T6588] RAX: 0000000000000000 RBX: ffffffff9213a160 RCX: 
0000000000000000
[  104.990702][ T6588] RDX: ffff888112f83b80 RSI: ffffffff814c7747 RDI: 
0000000000000001
[  104.990702][ T6588] RBP: 00000000ffffffff R08: 0000000000000001 R09: 
0000000000000000
[  104.990702][ T6588] R10: 0000000000000001 R11: 0000000000000001 R12: 
00000000ffffffff
[  104.990702][ T6588] R13: ffff88816a514001 R14: 0000000000000001 R15: 
ffffffff8e7b0680
[  104.990702][ T6588]  ? __warn_printk+0x187/0x310
[  104.990702][ T6588]  ? clsact_egress_block_get+0x40/0x40
[  104.990702][ T6588]  static_key_slow_dec+0x60/0xc0
[  104.990702][ T6588]  qdisc_create+0xa45/0x1090
[  104.990702][ T6588]  ? tc_get_qdisc+0xb70/0xb70
[  104.990702][ T6588]  tc_modify_qdisc+0x491/0x1b70
[  104.990702][ T6588]  ? qdisc_create+0x1090/0x1090
[  104.990702][ T6588]  ? bpf_lsm_capable+0x9/0x10
[  104.990702][ T6588]  ? qdisc_create+0x1090/0x1090
[  104.990702][ T6588]  rtnetlink_rcv_msg+0x439/0xd30
[  104.990702][ T6588]  ? rtnl_getlink+0xb10/0xb10
[  104.990702][ T6588]  ? __x64_sys_sendmmsg+0x9c/0x100
[  104.990702][ T6588]  ? do_syscall_64+0x38/0xb0
[  104.990702][ T6588]  ? netdev_core_pick_tx+0x390/0x390
[  104.990702][ T6588]  netlink_rcv_skb+0x166/0x440
[  104.990702][ T6588]  ? rtnl_getlink+0xb10/0xb10
[  104.990702][ T6588]  ? netlink_ack+0x1370/0x1370
[  104.990702][ T6588]  ? kasan_set_track+0x25/0x30
[  104.990702][ T6588]  ? netlink_deliver_tap+0x1b1/0xd00
[  104.990702][ T6588]  netlink_unicast+0x530/0x800
[  104.990702][ T6588]  ? netlink_attachskb+0x880/0x880
[  104.990702][ T6588]  ? __sanitizer_cov_trace_switch+0x54/0x90
[  104.990702][ T6588]  ? __phys_addr_symbol+0x30/0x70
[  104.990702][ T6588]  ? __check_object_size+0x323/0x740
[  104.990702][ T6588]  netlink_sendmsg+0x90b/0xe10
[  104.990702][ T6588]  ? netlink_unicast+0x800/0x800
[  104.990702][ T6588]  ? bpf_lsm_socket_sendmsg+0x9/0x10
[  104.990702][ T6588]  ? netlink_unicast+0x800/0x800
[  104.990702][ T6588]  sock_sendmsg+0xd9/0x180
[  104.990702][ T6588]  ____sys_sendmsg+0x264/0x910
[  104.990702][ T6588]  ? kernel_sendmsg+0x50/0x50
[  104.990702][ T6588]  ? __copy_msghdr+0x460/0x460
[  104.990702][ T6588]  ? lockdep_hardirqs_on_prepare+0x410/0x410
[  104.990702][ T6588]  ? find_held_lock+0x2d/0x110
[  104.990702][ T6588]  ___sys_sendmsg+0x11d/0x1b0
[  104.990702][ T6588]  ? do_recvmmsg+0x700/0x700
[  104.990702][ T6588]  ? __fget_files+0x260/0x420
[  104.990702][ T6588]  ? reacquire_held_locks+0x4b0/0x4b0
[  104.990702][ T6588]  ? __fget_files+0x282/0x420
[  104.990702][ T6588]  ? __fget_light+0xe6/0x270
[  104.990702][ T6588]  __sys_sendmmsg+0x18e/0x430
[  104.990702][ T6588]  ? __ia32_sys_sendmsg+0xb0/0xb0
[  104.990702][ T6588]  ? reacquire_held_locks+0x4b0/0x4b0
[  104.990702][ T6588]  ? rcu_is_watching+0x12/0xb0
[  104.990702][ T6588]  ? xfd_validate_state+0x5d/0x180
[  104.990702][ T6588]  ? restore_fpregs_from_fpstate+0xc1/0x1d0
[  104.990702][ T6588]  ? unlock_page_memcg+0x2d0/0x2d0
[  104.990702][ T6588]  ? do_futex+0x350/0x350
[  104.990702][ T6588]  __x64_sys_sendmmsg+0x9c/0x100
[  104.990702][ T6588]  ? syscall_enter_from_user_mode+0x26/0x80
[  104.990702][ T6588]  do_syscall_64+0x38/0xb0
[  104.990702][ T6588]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  104.990702][ T6588] RIP: 0033:0x7f76c66ee89d
[  104.990702][ T6588] Code: 5d c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 
0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 
24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4b 05 0e 00 f7 d8 64 
89 01 48
[  104.990702][ T6588] RSP: 002b:00007f76c65d4c68 EFLAGS: 00000203 
ORIG_RAX: 0000000000000133
[  104.990702][ T6588] RAX: ffffffffffffffda RBX: 00007f76c65d5cdc RCX: 
00007f76c66ee89d
[  104.990702][ T6588] RDX: 040000000000009f RSI: 00000000200002c0 RDI: 
0000000000000007
[  104.990702][ T6588] RBP: 00007f76c65d4de0 R08: 0000000100000001 R09: 
0000000000000000
[  104.990702][ T6588] R10: 0000000000000000 R11: 0000000000000203 R12: 
fffffffffffffeb8
[  104.990702][ T6588] R13: 0000000000000002 R14: 00007ffce5a5fd90 R15: 
00007f76c65b5000
[  104.990702][ T6588]  </TASK>
[  104.990702][ T6588] Kernel Offset: disabled
[  104.990702][ T6588] Rebooting in 86400 seconds..

Powered by blists - more mailing lists