Date:   Mon, 20 Sep 2021 21:08:08 +0800
From:   Hao Sun <sunhao.th@...il.com>
To:     clm@...com, dsterba@...e.com, Josef Bacik <josef@...icpanda.com>,
        linux-btrfs@...r.kernel.org
Cc:     Linux Kernel Mailing List <linux-kernel@...r.kernel.org>
Subject: possible deadlock in __btrfs_release_delayed_node

Hello,

When using Healer to fuzz the latest Linux kernel, the following crash
was triggered.

HEAD commit: 4357f03d6611 Merge tag 'pm-5.15-rc2
git tree: upstream
console output:
https://drive.google.com/file/d/1Is2XjAwyco_VmymNbbsMH4xZysHsgfag/view?usp=sharing
kernel config: https://drive.google.com/file/d/1HKZtF_s3l6PL3OoQbNq_ei9CdBus-Tz0/view?usp=sharing

Sorry, I don't have a reproducer for this crash, hope the symbolized
report can help.
If you fix this issue, please add the following tag to the commit:
Reported-by: Hao Sun <sunhao.th@...il.com>

loop9: detected capacity change from 0 to 32768
BTRFS info (device loop9): disk space caching is enabled
BTRFS info (device loop9): has skinny extents
BTRFS info (device loop9): enabling ssd optimizations
FAULT_INJECTION: forcing a failure.
name failslab, interval 1, probability 0, space 0, times 0
CPU: 3 PID: 16884 Comm: syz-executor Not tainted 5.15.0-rc1+ #19
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
rel-1.12.0-59-gc9ba5276e321-prebuilt.qemu.org 04/01/2014
Call Trace:
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0x8d/0xcf lib/dump_stack.c:106
 fail_dump lib/fault-inject.c:52 [inline]
 should_fail+0x13c/0x160 lib/fault-inject.c:146
 should_failslab+0x5/0x10 mm/slab_common.c:1328
 slab_pre_alloc_hook.constprop.99+0x4e/0xc0 mm/slab.h:494
 slab_alloc_node mm/slub.c:3120 [inline]
 slab_alloc mm/slub.c:3214 [inline]
 kmem_cache_alloc+0x44/0x280 mm/slub.c:3219
 btrfs_add_delayed_tree_ref+0xc3/0x580 fs/btrfs/delayed-ref.c:917
 btrfs_alloc_tree_block+0x478/0x670 fs/btrfs/extent-tree.c:4853
 __btrfs_cow_block+0x16f/0x820 fs/btrfs/ctree.c:415
 btrfs_cow_block+0x12a/0x300 fs/btrfs/ctree.c:570
 btrfs_search_slot+0x6b0/0xee0 fs/btrfs/ctree.c:1768
 btrfs_lookup_inode+0x50/0x110 fs/btrfs/inode-item.c:408
 __btrfs_update_delayed_inode+0x81/0x3a0 fs/btrfs/delayed-inode.c:948
 btrfs_commit_inode_delayed_inode+0x14a/0x160 fs/btrfs/delayed-inode.c:1189
 btrfs_log_inode+0x438/0x1ab0 fs/btrfs/tree-log.c:5385
 btrfs_log_inode_parent+0x272/0x1110 fs/btrfs/tree-log.c:6168
 btrfs_log_dentry_safe+0x3a/0x50 fs/btrfs/tree-log.c:6269
 btrfs_sync_file+0x2cd/0x760 fs/btrfs/file.c:2264
 vfs_fsync_range+0x48/0xa0 fs/sync.c:200
 generic_write_sync include/linux/fs.h:2955 [inline]
 btrfs_file_write_iter+0x34f/0x510 fs/btrfs/file.c:2034
 call_write_iter include/linux/fs.h:2163 [inline]
 new_sync_write+0x18d/0x260 fs/read_write.c:507
 vfs_write+0x43b/0x4a0 fs/read_write.c:594
 ksys_write+0xd2/0x120 fs/read_write.c:647
 do_syscall_x64 arch/x86/entry/common.c:50 [inline]
 do_syscall_64+0x34/0xb0 arch/x86/entry/common.c:80
 entry_SYSCALL_64_after_hwframe+0x44/0xae
RIP: 0033:0x46ae99
Code: f7 d8 64 89 02 b8 ff ff ff ff c3 66 0f 1f 44 00 00 48 89 f8 48
89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d
01 f0 ff ff 73 01 c3 48 c7 c1 bc ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007f685738cc48 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
RAX: ffffffffffffffda RBX: 000000000078c0a0 RCX: 000000000046ae99
RDX: 000000000000000b RSI: 0000000020005940 RDI: 0000000000000003
RBP: 00007f685738cc80 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 000000000000001e
R13: 0000000000000000 R14: 000000000078c0a0 R15: 00007ffc71ddb640
BTRFS: error (device loop9) in __btrfs_update_delayed_inode:995:
errno=-12 Out of memory
BTRFS info (device loop9): forced readonly

======================================================
WARNING: possible circular locking dependency detected
5.15.0-rc1+ #19 Not tainted
------------------------------------------------------
syz-executor/16884 is trying to acquire lock:
ffff88810f946ab0 (&delayed_node->mutex){+.+.}-{3:3}, at:
__btrfs_release_delayed_node+0x4e/0x410 fs/btrfs/delayed-inode.c:262

but task is already holding lock:
ffff888016e696e8 (btrfs-tree-01/1){+.+.}-{3:3}, at:
__btrfs_tree_lock+0x2e/0x1a0 fs/btrfs/locking.c:112

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #1 (btrfs-tree-01/1){+.+.}-{3:3}:
       __lock_release kernel/locking/lockdep.c:5341 [inline]
       lock_release+0x135/0x2e0 kernel/locking/lockdep.c:5645
       up_write+0x12/0x130 kernel/locking/rwsem.c:1569
       __btrfs_cow_block+0x552/0x820 fs/btrfs/ctree.c:491
       btrfs_cow_block+0x12a/0x300 fs/btrfs/ctree.c:570
       btrfs_search_slot+0x6b0/0xee0 fs/btrfs/ctree.c:1768
       btrfs_insert_empty_items+0x80/0xf0 fs/btrfs/ctree.c:3905
       btrfs_insert_delayed_item fs/btrfs/delayed-inode.c:736 [inline]
       btrfs_insert_delayed_items fs/btrfs/delayed-inode.c:783 [inline]
       __btrfs_commit_inode_delayed_items fs/btrfs/delayed-inode.c:1041 [inline]
       __btrfs_run_delayed_items+0x307/0x6a0 fs/btrfs/delayed-inode.c:1083
       btrfs_commit_transaction+0x268/0x1450 fs/btrfs/transaction.c:2170
       btrfs_sync_fs+0x9a/0x430 fs/btrfs/super.c:1426
       __sync_filesystem fs/sync.c:39 [inline]
       sync_filesystem+0x9b/0xd0 fs/sync.c:67
       generic_shutdown_super+0x30/0x170 fs/super.c:448
       kill_anon_super+0x15/0x30 fs/super.c:1057
       btrfs_kill_super+0x19/0x30 fs/btrfs/super.c:2348
       deactivate_locked_super+0x43/0x80 fs/super.c:335
       deactivate_super+0x53/0x80 fs/super.c:366
       cleanup_mnt+0x138/0x1b0 fs/namespace.c:1137
       task_work_run+0x86/0xd0 kernel/task_work.c:164
       tracehook_notify_resume include/linux/tracehook.h:189 [inline]
       exit_to_user_mode_loop kernel/entry/common.c:175 [inline]
       exit_to_user_mode_prepare+0x271/0x280 kernel/entry/common.c:209
       __syscall_exit_to_user_mode_work kernel/entry/common.c:291 [inline]
       syscall_exit_to_user_mode+0x19/0x60 kernel/entry/common.c:302
       do_syscall_64+0x40/0xb0 arch/x86/entry/common.c:86
       entry_SYSCALL_64_after_hwframe+0x44/0xae

-> #0 (&delayed_node->mutex){+.+.}-{3:3}:
       check_prev_add kernel/locking/lockdep.c:3051 [inline]
       check_prevs_add kernel/locking/lockdep.c:3174 [inline]
       validate_chain kernel/locking/lockdep.c:3789 [inline]
       __lock_acquire+0x161f/0x1d60 kernel/locking/lockdep.c:5015
       lock_acquire+0x1f9/0x340 kernel/locking/lockdep.c:5625
       __mutex_lock_common kernel/locking/mutex.c:596 [inline]
       __mutex_lock+0x98/0xa50 kernel/locking/mutex.c:729
       __btrfs_release_delayed_node+0x4e/0x410 fs/btrfs/delayed-inode.c:262
       btrfs_release_delayed_node fs/btrfs/delayed-inode.c:287 [inline]
       btrfs_commit_inode_delayed_inode+0x104/0x160
fs/btrfs/delayed-inode.c:1201
       btrfs_log_inode+0x438/0x1ab0 fs/btrfs/tree-log.c:5385
       btrfs_log_inode_parent+0x272/0x1110 fs/btrfs/tree-log.c:6168
       btrfs_log_dentry_safe+0x3a/0x50 fs/btrfs/tree-log.c:6269
       btrfs_sync_file+0x2cd/0x760 fs/btrfs/file.c:2264
       vfs_fsync_range+0x48/0xa0 fs/sync.c:200
       generic_write_sync include/linux/fs.h:2955 [inline]
       btrfs_file_write_iter+0x34f/0x510 fs/btrfs/file.c:2034
       call_write_iter include/linux/fs.h:2163 [inline]
       new_sync_write+0x18d/0x260 fs/read_write.c:507
       vfs_write+0x43b/0x4a0 fs/read_write.c:594
       ksys_write+0xd2/0x120 fs/read_write.c:647
       do_syscall_x64 arch/x86/entry/common.c:50 [inline]
       do_syscall_64+0x34/0xb0 arch/x86/entry/common.c:80
       entry_SYSCALL_64_after_hwframe+0x44/0xae

other info that might help us debug this:

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(btrfs-tree-01/1);
                               lock(&delayed_node->mutex);
                               lock(btrfs-tree-01/1);
  lock(&delayed_node->mutex);

 *** DEADLOCK ***

6 locks held by syz-executor/16884:
 #0: ffff888104f336f0 (&f->f_pos_lock){+.+.}-{3:3}, at:
__fdget_pos+0x55/0x60 fs/file.c:990
 #1: ffff88811394b460 (sb_writers#18){.+.+}-{0:0}, at:
ksys_write+0xd2/0x120 fs/read_write.c:647
 #2: ffff888110cfab70 (&sb->s_type->i_mutex_key#26){++++}-{3:3}, at:
inode_lock include/linux/fs.h:786 [inline]
 #2: ffff888110cfab70 (&sb->s_type->i_mutex_key#26){++++}-{3:3}, at:
btrfs_inode_lock+0x65/0xb0 fs/btrfs/inode.c:126
 #3: ffff888110cfa9f8 (&ei->i_mmap_lock){+.+.}-{3:3}, at:
btrfs_inode_lock+0x80/0xb0 fs/btrfs/inode.c:129
 #4: ffff88811394b650 (sb_internal#2){.+.+}-{0:0}, at:
btrfs_sync_file+0x29e/0x760 fs/btrfs/file.c:2257
 #5: ffff888016e696e8 (btrfs-tree-01/1){+.+.}-{3:3}, at:
__btrfs_tree_lock+0x2e/0x1a0 fs/btrfs/locking.c:112

stack backtrace:
CPU: 1 PID: 16884 Comm: syz-executor Not tainted 5.15.0-rc1+ #19
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
rel-1.12.0-59-gc9ba5276e321-prebuilt.qemu.org 04/01/2014
Call Trace:
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0x8d/0xcf lib/dump_stack.c:106
 check_noncircular+0x105/0x120 kernel/locking/lockdep.c:2131
 check_prev_add kernel/locking/lockdep.c:3051 [inline]
 check_prevs_add kernel/locking/lockdep.c:3174 [inline]
 validate_chain kernel/locking/lockdep.c:3789 [inline]
 __lock_acquire+0x161f/0x1d60 kernel/locking/lockdep.c:5015
 lock_acquire+0x1f9/0x340 kernel/locking/lockdep.c:5625
 __mutex_lock_common kernel/locking/mutex.c:596 [inline]
 __mutex_lock+0x98/0xa50 kernel/locking/mutex.c:729
 __btrfs_release_delayed_node+0x4e/0x410 fs/btrfs/delayed-inode.c:262
 btrfs_release_delayed_node fs/btrfs/delayed-inode.c:287 [inline]
 btrfs_commit_inode_delayed_inode+0x104/0x160 fs/btrfs/delayed-inode.c:1201
 btrfs_log_inode+0x438/0x1ab0 fs/btrfs/tree-log.c:5385
 btrfs_log_inode_parent+0x272/0x1110 fs/btrfs/tree-log.c:6168
 btrfs_log_dentry_safe+0x3a/0x50 fs/btrfs/tree-log.c:6269
 btrfs_sync_file+0x2cd/0x760 fs/btrfs/file.c:2264
 vfs_fsync_range+0x48/0xa0 fs/sync.c:200
 generic_write_sync include/linux/fs.h:2955 [inline]
 btrfs_file_write_iter+0x34f/0x510 fs/btrfs/file.c:2034
 call_write_iter include/linux/fs.h:2163 [inline]
 new_sync_write+0x18d/0x260 fs/read_write.c:507
 vfs_write+0x43b/0x4a0 fs/read_write.c:594
 ksys_write+0xd2/0x120 fs/read_write.c:647
 do_syscall_x64 arch/x86/entry/common.c:50 [inline]
 do_syscall_64+0x34/0xb0 arch/x86/entry/common.c:80
 entry_SYSCALL_64_after_hwframe+0x44/0xae
RIP: 0033:0x46ae99
Code: f7 d8 64 89 02 b8 ff ff ff ff c3 66 0f 1f 44 00 00 48 89 f8 48
89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d
01 f0 ff ff 73 01 c3 48 c7 c1 bc ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007f685738cc48 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
RAX: ffffffffffffffda RBX: 000000000078c0a0 RCX: 000000000046ae99
RDX: 000000000000000b RSI: 0000000020005940 RDI: 0000000000000003
RBP: 00007f685738cc80 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 000000000000001e
R13: 0000000000000000 R14: 000000000078c0a0 R15: 00007ffc71ddb640

Powered by blists - more mailing lists