netdev - Re: [syzbot] [net?] possible deadlock in team_del

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <ZoZ2RH9BcahEB9Sb@nanopsycho.orion>
Date: Thu, 4 Jul 2024 12:15:32 +0200
From: Jiri Pirko <jiri@...nulli.us>
To: syzbot <syzbot+705c61d60b091ef42c04@...kaller.appspotmail.com>
Cc: davem@...emloft.net, edumazet@...gle.com, kuba@...nel.org,
	linux-kernel@...r.kernel.org, netdev@...r.kernel.org,
	pabeni@...hat.com, syzkaller-bugs@...glegroups.com
Subject: Re: [syzbot] [net?] possible deadlock in team_del_slave (3)

Fri, Apr 26, 2024 at 01:59:32PM CEST, syzbot+705c61d60b091ef42c04@...kaller.appspotmail.com wrote:
>Hello,
>
>syzbot found the following issue on:
>
>HEAD commit:    480e035fc4c7 Merge tag 'drm-next-2024-03-13' of https://gi..
>git tree:       upstream
>console+strace: https://syzkaller.appspot.com/x/log.txt?x=1662179e180000
>kernel config:  https://syzkaller.appspot.com/x/.config?x=1e5b814e91787669
>dashboard link: https://syzkaller.appspot.com/bug?extid=705c61d60b091ef42c04
>compiler:       Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
>syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=1058e7b9180000
>C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=11919365180000
>
>Downloadable assets:
>disk image: https://storage.googleapis.com/syzbot-assets/5f73b6ef963d/disk-480e035f.raw.xz
>vmlinux: https://storage.googleapis.com/syzbot-assets/46c949396aad/vmlinux-480e035f.xz
>kernel image: https://storage.googleapis.com/syzbot-assets/e3b4d0f5a5f8/bzImage-480e035f.xz
>
>IMPORTANT: if you fix the issue, please add the following tag to the commit:
>Reported-by: syzbot+705c61d60b091ef42c04@...kaller.appspotmail.com
>
>======================================================
>WARNING: possible circular locking dependency detected
>6.8.0-syzkaller-08073-g480e035fc4c7 #0 Not tainted
>------------------------------------------------------
>syz-executor419/5074 is trying to acquire lock:
>ffff888023dc4d20 (team->team_lock_key){+.+.}-{3:3}, at: team_del_slave+0x32/0x1d0 drivers/net/team/team.c:1988
>
>but task is already holding lock:
>ffff88802a210768 (&rdev->wiphy.mtx){+.+.}-{3:3}, at: nl80211_del_interface+0x11a/0x140 net/wireless/nl80211.c:4389
>
>which lock already depends on the new lock.
>
>
>the existing dependency chain (in reverse order) is:
>
>-> #1 (&rdev->wiphy.mtx){+.+.}-{3:3}:
>       lock_acquire+0x1e4/0x530 kernel/locking/lockdep.c:5754
>       __mutex_lock_common kernel/locking/mutex.c:608 [inline]
>       __mutex_lock+0x136/0xd70 kernel/locking/mutex.c:752
>       wiphy_lock include/net/cfg80211.h:5951 [inline]
>       ieee80211_open+0xe7/0x200 net/mac80211/iface.c:449
>       __dev_open+0x2d3/0x450 net/core/dev.c:1430
>       dev_open+0xae/0x1b0 net/core/dev.c:1466
>       team_port_add drivers/net/team/team.c:1214 [inline]
>       team_add_slave+0x9b3/0x2750 drivers/net/team/team.c:1974
>       do_set_master net/core/rtnetlink.c:2685 [inline]
>       do_setlink+0xe70/0x41f0 net/core/rtnetlink.c:2891
>       rtnl_setlink+0x40d/0x5a0 net/core/rtnetlink.c:3185
>       rtnetlink_rcv_msg+0x89b/0x10d0 net/core/rtnetlink.c:6595
>       netlink_rcv_skb+0x1e3/0x430 net/netlink/af_netlink.c:2559
>       netlink_unicast_kernel net/netlink/af_netlink.c:1335 [inline]
>       netlink_unicast+0x7ea/0x980 net/netlink/af_netlink.c:1361
>       netlink_sendmsg+0x8e1/0xcb0 net/netlink/af_netlink.c:1905
>       sock_sendmsg_nosec net/socket.c:730 [inline]
>       __sock_sendmsg+0x221/0x270 net/socket.c:745
>       ____sys_sendmsg+0x525/0x7d0 net/socket.c:2584
>       ___sys_sendmsg net/socket.c:2638 [inline]
>       __sys_sendmsg+0x2b0/0x3a0 net/socket.c:2667
>       do_syscall_64+0xfb/0x240
>       entry_SYSCALL_64_after_hwframe+0x6d/0x75
>
>-> #0 (team->team_lock_key){+.+.}-{3:3}:
>       check_prev_add kernel/locking/lockdep.c:3134 [inline]
>       check_prevs_add kernel/locking/lockdep.c:3253 [inline]
>       validate_chain+0x18cb/0x58e0 kernel/locking/lockdep.c:3869
>       __lock_acquire+0x1346/0x1fd0 kernel/locking/lockdep.c:5137
>       lock_acquire+0x1e4/0x530 kernel/locking/lockdep.c:5754
>       __mutex_lock_common kernel/locking/mutex.c:608 [inline]
>       __mutex_lock+0x136/0xd70 kernel/locking/mutex.c:752
>       team_del_slave+0x32/0x1d0 drivers/net/team/team.c:1988
>       team_device_event+0x200/0x5b0 drivers/net/team/team.c:3029
>       notifier_call_chain+0x18f/0x3b0 kernel/notifier.c:93
>       call_netdevice_notifiers_extack net/core/dev.c:1988 [inline]
>       call_netdevice_notifiers net/core/dev.c:2002 [inline]
>       unregister_netdevice_many_notify+0xd96/0x16d0 net/core/dev.c:11096
>       unregister_netdevice_many net/core/dev.c:11154 [inline]
>       unregister_netdevice_queue+0x303/0x370 net/core/dev.c:11033
>       unregister_netdevice include/linux/netdevice.h:3115 [inline]
>       _cfg80211_unregister_wdev+0x162/0x560 net/wireless/core.c:1206
>       ieee80211_if_remove+0x25d/0x3a0 net/mac80211/iface.c:2242
>       ieee80211_del_iface+0x19/0x30 net/mac80211/cfg.c:202
>       rdev_del_virtual_intf net/wireless/rdev-ops.h:62 [inline]
>       cfg80211_remove_virtual_intf+0x230/0x3f0 net/wireless/util.c:2847
>       genl_family_rcv_msg_doit net/netlink/genetlink.c:1113 [inline]
>       genl_family_rcv_msg net/netlink/genetlink.c:1193 [inline]
>       genl_rcv_msg+0xb14/0xec0 net/netlink/genetlink.c:1208
>       netlink_rcv_skb+0x1e3/0x430 net/netlink/af_netlink.c:2559
>       genl_rcv+0x28/0x40 net/netlink/genetlink.c:1217
>       netlink_unicast_kernel net/netlink/af_netlink.c:1335 [inline]
>       netlink_unicast+0x7ea/0x980 net/netlink/af_netlink.c:1361
>       netlink_sendmsg+0x8e1/0xcb0 net/netlink/af_netlink.c:1905
>       sock_sendmsg_nosec net/socket.c:730 [inline]
>       __sock_sendmsg+0x221/0x270 net/socket.c:745
>       ____sys_sendmsg+0x525/0x7d0 net/socket.c:2584
>       ___sys_sendmsg net/socket.c:2638 [inline]
>       __sys_sendmsg+0x2b0/0x3a0 net/socket.c:2667
>       do_syscall_64+0xfb/0x240
>       entry_SYSCALL_64_after_hwframe+0x6d/0x75

I wonder, since we already rely on rtnl in lots of team code, perhaps we
can remove team->lock completely and convert the rest of the code to be
protected by rtnl lock as well.



>
>other info that might help us debug this:
>
> Possible unsafe locking scenario:
>
>       CPU0                    CPU1
>       ----                    ----
>  lock(&rdev->wiphy.mtx);
>                               lock(team->team_lock_key);
>                               lock(&rdev->wiphy.mtx);
>  lock(team->team_lock_key);
>
> *** DEADLOCK ***
>
>3 locks held by syz-executor419/5074:
> #0: ffffffff8f3f1a30 (cb_lock){++++}-{3:3}, at: genl_rcv+0x19/0x40 net/netlink/genetlink.c:1216
> #1: ffffffff8f38ce88 (rtnl_mutex){+.+.}-{3:3}, at: nl80211_pre_doit+0x5f/0x8b0 net/wireless/nl80211.c:16401
> #2: ffff88802a210768 (&rdev->wiphy.mtx){+.+.}-{3:3}, at: nl80211_del_interface+0x11a/0x140 net/wireless/nl80211.c:4389
>
>stack backtrace:
>CPU: 1 PID: 5074 Comm: syz-executor419 Not tainted 6.8.0-syzkaller-08073-g480e035fc4c7 #0
>Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 02/29/2024
>Call Trace:
> <TASK>
> __dump_stack lib/dump_stack.c:88 [inline]
> dump_stack_lvl+0x241/0x360 lib/dump_stack.c:114
> check_noncircular+0x36a/0x4a0 kernel/locking/lockdep.c:2187
> check_prev_add kernel/locking/lockdep.c:3134 [inline]
> check_prevs_add kernel/locking/lockdep.c:3253 [inline]
> validate_chain+0x18cb/0x58e0 kernel/locking/lockdep.c:3869
> __lock_acquire+0x1346/0x1fd0 kernel/locking/lockdep.c:5137
> lock_acquire+0x1e4/0x530 kernel/locking/lockdep.c:5754
> __mutex_lock_common kernel/locking/mutex.c:608 [inline]
> __mutex_lock+0x136/0xd70 kernel/locking/mutex.c:752
> team_del_slave+0x32/0x1d0 drivers/net/team/team.c:1988
> team_device_event+0x200/0x5b0 drivers/net/team/team.c:3029
> notifier_call_chain+0x18f/0x3b0 kernel/notifier.c:93
> call_netdevice_notifiers_extack net/core/dev.c:1988 [inline]
> call_netdevice_notifiers net/core/dev.c:2002 [inline]
> unregister_netdevice_many_notify+0xd96/0x16d0 net/core/dev.c:11096
> unregister_netdevice_many net/core/dev.c:11154 [inline]
> unregister_netdevice_queue+0x303/0x370 net/core/dev.c:11033
> unregister_netdevice include/linux/netdevice.h:3115 [inline]
> _cfg80211_unregister_wdev+0x162/0x560 net/wireless/core.c:1206
> ieee80211_if_remove+0x25d/0x3a0 net/mac80211/iface.c:2242
> ieee80211_del_iface+0x19/0x30 net/mac80211/cfg.c:202
> rdev_del_virtual_intf net/wireless/rdev-ops.h:62 [inline]
> cfg80211_remove_virtual_intf+0x230/0x3f0 net/wireless/util.c:2847
> genl_family_rcv_msg_doit net/netlink/genetlink.c:1113 [inline]
> genl_family_rcv_msg net/netlink/genetlink.c:1193 [inline]
> genl_rcv_msg+0xb14/0xec0 net/netlink/genetlink.c:1208
> netlink_rcv_skb+0x1e3/0x430 net/netlink/af_netlink.c:2559
> genl_rcv+0x28/0x40 net/netlink/genetlink.c:1217
> netlink_unicast_kernel net/netlink/af_netlink.c:1335 [inline]
> netlink_unicast+0x7ea/0x980 net/netlink/af_netlink.c:1361
> netlink_sendmsg+0x8e1/0xcb0 net/netlink/af_netlink.c:1905
> sock_sendmsg_nosec net/socket.c:730 [inline]
> __sock_sendmsg+0x221/0x270 net/socket.c:745
> ____sys_sendmsg+0x525/0x7d0 net/socket.c:2584
> ___sys_sendmsg net/socket.c:2638 [inline]
> __sys_sendmsg+0x2b0/0x3a0 net/socket.c:2667
> do_syscall_64+0xfb/0x240
> entry_SYSCALL_64_after_hwframe+0x6d/0x75
>RIP: 0033:0x7f963cb981a9
>Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 d1 19 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48
>RSP: 002b:00007ffdde1419a8 EFLAGS: 00000246 ORIG_RAX: 000000000000002e
>RAX: ffffffffffffffda RBX: 00007f963cbe53f6 RCX: 00007f963cb981a9
>RDX: 0000000000000000 RSI: 0000000020000400 RDI: 0000000000000004
>RBP: 00007f963cc17440 R08: 0000000000000000 R09: 0000000000000000
>R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000031
>R13: 0000000000000003 R14: 0000000000050012 R15: 00007ffdde141a02
> </TASK>
>team0: Port device wlan0 removed
>
>
>---
>This report is generated by a bot. It may contain errors.
>See https://goo.gl/tpsmEJ for more information about syzbot.
>syzbot engineers can be reached at syzkaller@...glegroups.com.
>
>syzbot will keep track of this issue. See:
>https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
>
>If the report is already addressed, let syzbot know by replying with:
>#syz fix: exact-commit-title
>
>If you want syzbot to run the reproducer, reply with:
>#syz test: git://repo/address.git branch-or-commit-hash
>If you attach or paste a git patch, syzbot will apply it before testing.
>
>If you want to overwrite report's subsystems, reply with:
>#syz set subsystems: new-subsystem
>(See the list of subsystem names on the web dashboard)
>
>If the report is a duplicate of another one, reply with:
>#syz dup: exact-subject-of-another-report
>
>If you want to undo deduplication, reply with:
>#syz undup