Message-ID: <5b28cd6f-d921-b095-1190-474bcce89e53@mojatatu.com>
Date:   Tue, 23 May 2023 00:51:44 -0300
From:   Pedro Tammela <pctammela@...atatu.com>
To:     Peilin Ye <yepeilin.cs@...il.com>,
        "David S. Miller" <davem@...emloft.net>,
        Eric Dumazet <edumazet@...gle.com>,
        Jakub Kicinski <kuba@...nel.org>,
        Paolo Abeni <pabeni@...hat.com>,
        Jamal Hadi Salim <jhs@...atatu.com>,
        Cong Wang <xiyou.wangcong@...il.com>,
        Jiri Pirko <jiri@...nulli.us>
Cc:     Peilin Ye <peilin.ye@...edance.com>,
        Daniel Borkmann <daniel@...earbox.net>,
        John Fastabend <john.fastabend@...il.com>,
        Hillf Danton <hdanton@...a.com>, netdev@...r.kernel.org,
        linux-kernel@...r.kernel.org, Cong Wang <cong.wang@...edance.com>,
        Vlad Buslov <vladbu@...dia.com>
Subject: Re: [PATCH v2 net 6/6] net/sched: qdisc_destroy() old ingress and
 clsact Qdiscs before grafting

On 22/05/2023 20:55, Peilin Ye wrote:
> mini_Qdisc_pair::p_miniq is a double pointer to mini_Qdisc [...]


Hi Peilin!

With V2 patches 5 and 6 applied I was still able to trigger an oops.

Branch is 'net' + patches 5 & 6:
145f639b9403 (HEAD -> main) net/sched: qdisc_destroy() old ingress and 
clsact Qdiscs before grafting
1aac74ef9673 net/sched: Refactor qdisc_graft() for ingress and clsact Qdiscs
18c40a1cc1d9 (origin/main, origin/HEAD) net/handshake: Fix sock->file 
allocation

Kernel config is the same as in the syzbot report.
Note that this was on a _single core_ VM.
I will double check if v1 is triggering this issue (basically run the 
repro for a long time). For multi-core my VM is running OOM even on a 
32Gb system. I will check if we have a spare server to run the repro.

[  695.782780][T12033] 
==================================================================
[  695.783617][T12033] BUG: KASAN: slab-use-after-free in 
mini_qdisc_pair_swap+0x1c2/0x1f0
[  695.784323][T12033] Write of size 8 at addr ffff888060cafb08 by task 
repro/12033
[  695.784996][T12033]
[  695.785210][T12033] CPU: 0 PID: 12033 Comm: repro Not tainted 
6.4.0-rc2-00187-g145f639b9403 #1
[  695.785981][T12033] Hardware name: QEMU Standard PC (i440FX + PIIX, 
1996), BIOS 1.16.0-debian-1.16.0-5 04/01/2014
[  695.786883][T12033] Call Trace:
[  695.787178][T12033]  <TASK>
[  695.787444][T12033]  dump_stack_lvl+0xd9/0x1b0
[  695.787871][T12033]  print_report+0xc4/0x5f0
[  695.788283][T12033]  ? __virt_addr_valid+0x5e/0x2d0
[  695.788736][T12033]  ? __phys_addr+0xc6/0x140
[  695.789138][T12033]  ? mini_qdisc_pair_swap+0x1c2/0x1f0
[  695.789604][T12033]  kasan_report+0xc0/0xf0
[  695.789604][T12033]  ? mini_qdisc_pair_swap+0x1c2/0x1f0
[  695.789604][T12033]  mini_qdisc_pair_swap+0x1c2/0x1f0
[  695.789604][T12033]  ? ingress_init+0x1c0/0x1c0
[  695.789604][T12033]  tcf_chain0_head_change.isra.0+0xb9/0x120
[  695.789604][T12033]  tc_new_tfilter+0x1ebb/0x22b0
[  695.789604][T12033]  ? tc_del_tfilter+0x1570/0x1570
[  695.789604][T12033]  ? lockdep_hardirqs_on_prepare+0x410/0x410
[  695.789604][T12033]  ? kasan_quarantine_put+0x102/0x230
[  695.789604][T12033]  ? lockdep_hardirqs_on+0x7d/0x100
[  695.789604][T12033]  ? rtnetlink_rcv_msg+0x94a/0xd30
[  695.789604][T12033]  ? reacquire_held_locks+0x4b0/0x4b0
[  695.789604][T12033]  ? bpf_lsm_capable+0x9/0x10
[  695.789604][T12033]  ? tc_del_tfilter+0x1570/0x1570
[  695.789604][T12033]  rtnetlink_rcv_msg+0x98a/0xd30
[  695.789604][T12033]  ? rtnl_getlink+0xb10/0xb10
[  695.789604][T12033]  ? reacquire_held_locks+0x4b0/0x4b0
[  695.789604][T12033]  ? netdev_core_pick_tx+0x390/0x390
[  695.789604][T12033]  netlink_rcv_skb+0x166/0x440
[  695.789604][T12033]  ? rtnl_getlink+0xb10/0xb10
[  695.789604][T12033]  ? netlink_ack+0x1370/0x1370
[  695.789604][T12033]  ? kasan_set_track+0x25/0x30
[  695.789604][T12033]  ? netlink_deliver_tap+0x1b1/0xd00
[  695.789604][T12033]  netlink_unicast+0x530/0x800
[  695.789604][T12033]  ? netlink_attachskb+0x880/0x880
[  695.789604][T12033]  ? __sanitizer_cov_trace_switch+0x54/0x90
[  695.789604][T12033]  ? __phys_addr_symbol+0x30/0x70
[  695.789604][T12033]  ? __check_object_size+0x323/0x740
[  695.789604][T12033]  netlink_sendmsg+0x90b/0xe10
[  695.789604][T12033]  ? netlink_unicast+0x800/0x800
[  695.789604][T12033]  ? bpf_lsm_socket_sendmsg+0x9/0x10
[  695.789604][T12033]  ? netlink_unicast+0x800/0x800
[  695.789604][T12033]  sock_sendmsg+0xd9/0x180
[  695.789604][T12033]  ____sys_sendmsg+0x264/0x910
[  695.789604][T12033]  ? kernel_sendmsg+0x50/0x50
[  695.789604][T12033]  ? __copy_msghdr+0x460/0x460
[  695.789604][T12033]  ___sys_sendmsg+0x11d/0x1b0
[  695.789604][T12033]  ? do_recvmmsg+0x700/0x700
[  695.789604][T12033]  ? find_held_lock+0x2d/0x110
[  695.789604][T12033]  ? __might_fault+0xe5/0x190
[  695.789604][T12033]  ? reacquire_held_locks+0x4b0/0x4b0
[  695.789604][T12033]  __sys_sendmmsg+0x18e/0x430
[  695.789604][T12033]  ? __ia32_sys_sendmsg+0xb0/0xb0
[  695.789604][T12033]  ? reacquire_held_locks+0x4b0/0x4b0
[  695.789604][T12033]  ? rcu_is_watching+0x12/0xb0
[  695.789604][T12033]  ? xfd_validate_state+0x5d/0x180
[  695.789604][T12033]  ? restore_fpregs_from_fpstate+0xc1/0x1d0
[  695.789604][T12033]  ? unlock_page_memcg+0x2d0/0x2d0
[  695.789604][T12033]  ? do_futex+0x350/0x350
[  695.789604][T12033]  __x64_sys_sendmmsg+0x9c/0x100
[  695.789604][T12033]  ? syscall_enter_from_user_mode+0x26/0x80
[  695.789604][T12033]  do_syscall_64+0x38/0xb0
[  695.789604][T12033]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  695.789604][T12033] RIP: 0033:0x7f4aca44a89d
[  695.789604][T12033] Code: 5d c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 
0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 
24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4b 05 0e 00 f7 d8 64 
89 01 48
[  695.789604][T12033] RSP: 002b:00007f4aca2eec68 EFLAGS: 00000203 
ORIG_RAX: 0000000000000133
[  695.789604][T12033] RAX: ffffffffffffffda RBX: 00007f4aca2efcdc RCX: 
00007f4aca44a89d
[  695.789604][T12033] RDX: 040000000000009f RSI: 00000000200002c0 RDI: 
0000000000000007
[  695.789604][T12033] RBP: 00007f4aca2eede0 R08: 0000000000000000 R09: 
0000000000000000
[  695.789604][T12033] R10: 0000000000000000 R11: 0000000000000203 R12: 
fffffffffffffeb8
[  695.789604][T12033] R13: 000000000000006e R14: 00007ffd1a53f720 R15: 
00007f4aca2cf000
[  695.789604][T12033]  </TASK>
[  695.789604][T12033]
[  695.789604][T12033] Allocated by task 12031:
[  695.789604][T12033]  kasan_save_stack+0x20/0x40
[  695.789604][T12033]  kasan_set_track+0x25/0x30
[  695.789604][T12033]  __kasan_kmalloc+0xa2/0xb0
[  695.789604][T12033]  __kmalloc_node+0x60/0x100
[  695.789604][T12033]  qdisc_alloc+0xb3/0xa90
[  695.789604][T12033]  qdisc_create+0xcf/0x1020
[  695.789604][T12033]  tc_modify_qdisc+0x495/0x1ab0
[  695.789604][T12033]  rtnetlink_rcv_msg+0x439/0xd30
[  695.789604][T12033]  netlink_rcv_skb+0x166/0x440
[  695.789604][T12033]  netlink_unicast+0x530/0x800
[  695.789604][T12033]  netlink_sendmsg+0x90b/0xe10
[  695.789604][T12033]  sock_sendmsg+0xd9/0x180
[  695.789604][T12033]  ____sys_sendmsg+0x264/0x910
[  695.789604][T12033]  ___sys_sendmsg+0x11d/0x1b0
[  695.789604][T12033]  __sys_sendmmsg+0x18e/0x430
[  695.789604][T12033]  __x64_sys_sendmmsg+0x9c/0x100
[  695.789604][T12033]  do_syscall_64+0x38/0xb0
[  695.789604][T12033]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  695.789604][T12033]
[  695.789604][T12033] Freed by task 15:
[  695.789604][T12033]  kasan_save_stack+0x20/0x40
[  695.789604][T12033]  kasan_set_track+0x25/0x30
[  695.789604][T12033]  kasan_save_free_info+0x2e/0x40
[  695.789604][T12033]  ____kasan_slab_free+0x15e/0x1b0
[  695.789604][T12033]  slab_free_freelist_hook+0x10b/0x1e0
[  695.789604][T12033]  __kmem_cache_free+0xaf/0x2e0
[  695.789604][T12033]  rcu_core+0x7f7/0x1ac0
[  695.789604][T12033]  __do_softirq+0x1d8/0x8fd
[  695.789604][T12033]
[  695.789604][T12033] Last potentially related work creation:
[  695.789604][T12033]  kasan_save_stack+0x20/0x40
[  695.789604][T12033]  __kasan_record_aux_stack+0xbf/0xd0
[  695.789604][T12033]  __call_rcu_common.constprop.0+0x9a/0x790
[  695.789604][T12033]  qdisc_put_unlocked+0x74/0x90
[  695.789604][T12033]  tcf_block_release+0x90/0xa0
[  695.789604][T12033]  tc_new_tfilter+0xa5e/0x22b0
[  695.789604][T12033]  rtnetlink_rcv_msg+0x98a/0xd30
[  695.789604][T12033]  netlink_rcv_skb+0x166/0x440
[  695.789604][T12033]  netlink_unicast+0x530/0x800
[  695.789604][T12033]  netlink_sendmsg+0x90b/0xe10
[  695.789604][T12033]  sock_sendmsg+0xd9/0x180
[  695.789604][T12033]  ____sys_sendmsg+0x264/0x910
[  695.789604][T12033]  ___sys_sendmsg+0x11d/0x1b0
[  695.789604][T12033]  __sys_sendmmsg+0x18e/0x430
[  695.789604][T12033]  __x64_sys_sendmmsg+0x9c/0x100
[  695.789604][T12033]  do_syscall_64+0x38/0xb0
[  695.789604][T12033]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  695.789604][T12033]
[  695.789604][T12033] Second to last potentially related work creation:
[  695.789604][T12033]  kasan_save_stack+0x20/0x40
[  695.789604][T12033]  __kasan_record_aux_stack+0xbf/0xd0
[  695.789604][T12033]  __call_rcu_common.constprop.0+0x9a/0x790
[  695.789604][T12033]  rht_deferred_worker+0x10fd/0x2010
[  695.789604][T12033]  process_one_work+0x9f9/0x15f0
[  695.789604][T12033]  worker_thread+0x687/0x1110
[  695.789604][T12033]  kthread+0x334/0x430
[  695.789604][T12033]  ret_from_fork+0x1f/0x30
[  695.789604][T12033]
[  695.789604][T12033] The buggy address belongs to the object at 
ffff888060caf800
[  695.789604][T12033]  which belongs to the cache kmalloc-1k of size 1024
[  695.789604][T12033] The buggy address is located 776 bytes inside of
[  695.789604][T12033]  freed 1024-byte region [ffff888060caf800, 
ffff888060cafc00)
[  695.789604][T12033]
[  695.789604][T12033] The buggy address belongs to the physical page:
[  695.789604][T12033] page:ffffea0001832b00 refcount:1 mapcount:0 
mapping:0000000000000000 index:0x0 pfn:0x60cac
[  695.789604][T12033] head:ffffea0001832b00 order:2 entire_mapcount:0 
nr_pages_mapped:0 pincount:0
[  695.789604][T12033] flags: 
0xfff00000010200(slab|head|node=0|zone=1|lastcpupid=0x7ff)
[  695.789604][T12033] page_type: 0xffffffff()
[  695.789604][T12033] raw: 00fff00000010200 ffff888012441dc0 
ffffea0000bac600 dead000000000002
[  695.789604][T12033] raw: 0000000000000000 0000000000080008 
00000001ffffffff 0000000000000000
[  695.789604][T12033] page dumped because: kasan: bad access detected
[  695.789604][T12033] page_owner tracks the page as allocated
[  695.789604][T12033] page last allocated via order 2, migratetype 
Unmovable, gfp_mask 
0xd20c0(__GFP_IO|__GFP_FS|__GFP_NOWARN|__GFP_NORETRY|__GFP_COMP|__GFP_NOMEMALLOC), 
pid 13324, tgid 13318 (repro), ts 170262603079, free_ts 0
[  695.789604][T12033]  get_page_from_freelist+0xe71/0x2e80
[  695.789604][T12033]  __alloc_pages+0x1c8/0x4a0
[  695.789604][T12033]  alloc_pages+0x1a9/0x270
[  695.789604][T12033]  allocate_slab+0x24e/0x380
[  695.789604][T12033]  ___slab_alloc+0x89a/0x1400
[  695.789604][T12033]  __slab_alloc.constprop.0+0x56/0xa0
[  695.789604][T12033]  __kmem_cache_alloc_node+0x126/0x330
[  695.789604][T12033]  kmalloc_trace+0x25/0xe0
[  695.789604][T12033]  fl_change+0x1b3/0x51e0
[  695.789604][T12033]  tc_new_tfilter+0x992/0x22b0
[  695.789604][T12033]  rtnetlink_rcv_msg+0x98a/0xd30
[  695.789604][T12033]  netlink_rcv_skb+0x166/0x440
[  695.789604][T12033]  netlink_unicast+0x530/0x800
[  695.789604][T12033]  netlink_sendmsg+0x90b/0xe10
[  695.789604][T12033]  sock_sendmsg+0xd9/0x180
[  695.789604][T12033]  ____sys_sendmsg+0x264/0x910
[  695.789604][T12033] page_owner free stack trace missing
[  695.789604][T12033]
[  695.789604][T12033] Memory state around the buggy address:
[  695.789604][T12033]  ffff888060cafa00: fb fb fb fb fb fb fb fb fb fb 
fb fb fb fb fb fb
[  695.789604][T12033]  ffff888060cafa80: fb fb fb fb fb fb fb fb fb fb 
fb fb fb fb fb fb
[  695.789604][T12033] >ffff888060cafb00: fb fb fb fb fb fb fb fb fb fb 
fb fb fb fb fb fb
[  695.789604][T12033]                       ^
[  695.789604][T12033]  ffff888060cafb80: fb fb fb fb fb fb fb fb fb fb 
fb fb fb fb fb fb
[  695.789604][T12033]  ffff888060cafc00: fc fc fc fc fc fc fc fc fc fc 
fc fc fc fc fc fc
[  695.789604][T12033] 
==================================================================
[  695.996261][T12042] __nla_validate_parse: 32 callbacks suppressed
[  695.996271][T12042] netlink: 24 bytes leftover after parsing 
attributes in process `repro'.
[  696.473670][T12046] netlink: 24 bytes leftover after parsing 
attributes in process `repro'.
[  696.660496][T12033] Kernel panic - not syncing: KASAN: panic_on_warn 
set ...
[  696.661250][T12033] CPU: 0 PID: 12033 Comm: repro Not tainted 
6.4.0-rc2-00187-g145f639b9403 #1
[  696.661947][T12033] Hardware name: QEMU Standard PC (i440FX + PIIX, 
1996), BIOS 1.16.0-debian-1.16.0-5 04/01/2014
[  696.662768][T12033] Call Trace:
[  696.663031][T12033]  <TASK>
[  696.663268][T12033]  dump_stack_lvl+0xd9/0x1b0
[  696.663659][T12033]  panic+0x689/0x730
[  696.663977][T12033]  ? panic_smp_self_stop+0xa0/0xa0
[  696.664396][T12033]  ? preempt_schedule_thunk+0x1a/0x20
[  696.664829][T12033]  ? preempt_schedule_common+0x45/0xc0
[  696.665263][T12033]  ? mini_qdisc_pair_swap+0x1c2/0x1f0
[  696.665698][T12033]  check_panic_on_warn+0xab/0xb0
[  696.666087][T12033]  ? mini_qdisc_pair_swap+0x1c2/0x1f0
[  696.666519][T12033]  end_report+0xe9/0x120
[  696.666861][T12033]  kasan_report+0xcd/0xf0
[  696.667209][T12033]  ? mini_qdisc_pair_swap+0x1c2/0x1f0
[  696.667639][T12033]  mini_qdisc_pair_swap+0x1c2/0x1f0
[  696.668064][T12033]  ? ingress_init+0x1c0/0x1c0
[  696.668451][T12033]  tcf_chain0_head_change.isra.0+0xb9/0x120
[  696.668964][T12033]  tc_new_tfilter+0x1ebb/0x22b0
[  696.669391][T12033]  ? tc_del_tfilter+0x1570/0x1570
[  696.669608][T12033]  ? lockdep_hardirqs_on_prepare+0x410/0x410
[  696.669608][T12033]  ? kasan_quarantine_put+0x102/0x230
[  696.669608][T12033]  ? lockdep_hardirqs_on+0x7d/0x100
[  696.669608][T12033]  ? rtnetlink_rcv_msg+0x94a/0xd30
[  696.669608][T12033]  ? reacquire_held_locks+0x4b0/0x4b0
[  696.669608][T12033]  ? bpf_lsm_capable+0x9/0x10
[  696.669608][T12033]  ? tc_del_tfilter+0x1570/0x1570
[  696.669608][T12033]  rtnetlink_rcv_msg+0x98a/0xd30
[  696.669608][T12033]  ? rtnl_getlink+0xb10/0xb10
[  696.669608][T12033]  ? reacquire_held_locks+0x4b0/0x4b0
[  696.669608][T12033]  ? netdev_core_pick_tx+0x390/0x390
[  696.669608][T12033]  netlink_rcv_skb+0x166/0x440
[  696.669608][T12033]  ? rtnl_getlink+0xb10/0xb10
[  696.669608][T12033]  ? netlink_ack+0x1370/0x1370
[  696.669608][T12033]  ? kasan_set_track+0x25/0x30
[  696.669608][T12033]  ? netlink_deliver_tap+0x1b1/0xd00
[  696.669608][T12033]  netlink_unicast+0x530/0x800
[  696.669608][T12033]  ? netlink_attachskb+0x880/0x880
[  696.669608][T12033]  ? __sanitizer_cov_trace_switch+0x54/0x90
[  696.669608][T12033]  ? __phys_addr_symbol+0x30/0x70
[  696.669608][T12033]  ? __check_object_size+0x323/0x740
[  696.669608][T12033]  netlink_sendmsg+0x90b/0xe10
[  696.669608][T12033]  ? netlink_unicast+0x800/0x800
[  696.669608][T12033]  ? bpf_lsm_socket_sendmsg+0x9/0x10
[  696.669608][T12033]  ? netlink_unicast+0x800/0x800
[  696.669608][T12033]  sock_sendmsg+0xd9/0x180
[  696.669608][T12033]  ____sys_sendmsg+0x264/0x910
[  696.669608][T12033]  ? kernel_sendmsg+0x50/0x50
[  696.669608][T12033]  ? __copy_msghdr+0x460/0x460
[  696.669608][T12033]  ___sys_sendmsg+0x11d/0x1b0
[  696.669608][T12033]  ? do_recvmmsg+0x700/0x700
[  696.669608][T12033]  ? find_held_lock+0x2d/0x110
[  696.669608][T12033]  ? __might_fault+0xe5/0x190
[  696.669608][T12033]  ? reacquire_held_locks+0x4b0/0x4b0
[  696.669608][T12033]  __sys_sendmmsg+0x18e/0x430
[  696.669608][T12033]  ? __ia32_sys_sendmsg+0xb0/0xb0
[  696.669608][T12033]  ? reacquire_held_locks+0x4b0/0x4b0
[  696.669608][T12033]  ? rcu_is_watching+0x12/0xb0
[  696.669608][T12033]  ? xfd_validate_state+0x5d/0x180
[  696.669608][T12033]  ? restore_fpregs_from_fpstate+0xc1/0x1d0
[  696.669608][T12033]  ? unlock_page_memcg+0x2d0/0x2d0
[  696.669608][T12033]  ? do_futex+0x350/0x350
[  696.669608][T12033]  __x64_sys_sendmmsg+0x9c/0x100
[  696.669608][T12033]  ? syscall_enter_from_user_mode+0x26/0x80
[  696.669608][T12033]  do_syscall_64+0x38/0xb0
[  696.669608][T12033]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  696.669608][T12033] RIP: 0033:0x7f4aca44a89d
[  696.669608][T12033] Code: 5d c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 
0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 
24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4b 05 0e 00 f7 d8 64 
89 01 48
[  696.669608][T12033] RSP: 002b:00007f4aca2eec68 EFLAGS: 00000203 
ORIG_RAX: 0000000000000133
[  696.669608][T12033] RAX: ffffffffffffffda RBX: 00007f4aca2efcdc RCX: 
00007f4aca44a89d
[  696.669608][T12033] RDX: 040000000000009f RSI: 00000000200002c0 RDI: 
0000000000000007
[  696.669608][T12033] RBP: 00007f4aca2eede0 R08: 0000000000000000 R09: 
0000000000000000
[  696.669608][T12033] R10: 0000000000000000 R11: 0000000000000203 R12: 
fffffffffffffeb8
[  696.669608][T12033] R13: 000000000000006e R14: 00007ffd1a53f720 R15: 
00007f4aca2cf000
[  696.669608][T12033]  </TASK>
[  696.669608][T12033] Kernel Offset: disabled
[  696.669608][T12033] Rebooting in 86400 seconds..

Powered by blists - more mailing lists