Message-ID: <CAJg=8jz4OwA9LdXtWJuPup+wGVJ8kKFXSboT3G8kjPXBSa-qHA@mail.gmail.com>
Date: Mon, 10 Jun 2024 14:52:36 -0700
From: Marius Fleischer <fleischermarius@...il.com>
To: Alexander Viro <viro@...iv.linux.org.uk>, linux-fsdevel@...r.kernel.org, 
	linux-kernel@...r.kernel.org
Cc: syzkaller@...glegroups.com, harrisonmichaelgreen@...il.com
Subject: possible deadlock in freeze_super

Hi,

We would like to report the following bug which has been found by our
modified version of syzkaller.

======================================================
description: possible deadlock in freeze_super
affected file: fs/super.c
kernel version: 5.15.159
kernel commit: a38297e3fb012ddfa7ce0321a7e5a8daeb1872b6
git tree: upstream
kernel config: attached
crash reproducer: attached
======================================================
Crash log:
WARNING: possible circular locking dependency detected
5.15.159 #1 Not tainted
------------------------------------------------------
kworker/1:1H/156 is trying to acquire lock:
ffff8881519ec0e0 (&type->s_umount_key#54){+.+.}-{3:3}, at:
freeze_super+0x41/0x3c0 fs/super.c:1682

but task is already holding lock:
ffffc90001aefdb0
((work_completion)(&(&gl->gl_work)->work)){+.+.}-{0:0}, at:
process_one_work+0x8d4/0x1550 kernel/workqueue.c:2285

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #2 ((work_completion)(&(&gl->gl_work)->work)){+.+.}-{0:0}:
       process_one_work+0x92b/0x1550 kernel/workqueue.c:2286
       worker_thread+0x65d/0x1130 kernel/workqueue.c:2457
       kthread+0x3e5/0x4d0 kernel/kthread.c:334
       ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:300

-> #1 ((wq_completion)glock_workqueue){+.+.}-{0:0}:
       flush_workqueue+0x11f/0x1300 kernel/workqueue.c:2830
       gfs2_gl_hash_clear+0xad/0x270 fs/gfs2/glock.c:2178
       gfs2_put_super+0x493/0x6a0 fs/gfs2/super.c:624
       generic_shutdown_super+0x154/0x380 fs/super.c:475
       kill_block_super+0x97/0xf0 fs/super.c:1414
       gfs2_kill_sb+0x104/0x160 fs/gfs2/ops_fstype.c:1740
       deactivate_locked_super+0x94/0x160 fs/super.c:335
       deactivate_super+0xad/0xd0 fs/super.c:366
       cleanup_mnt+0x3a2/0x540 fs/namespace.c:1143
       task_work_run+0xdd/0x1a0 kernel/task_work.c:164
       tracehook_notify_resume include/linux/tracehook.h:189 [inline]
       exit_to_user_mode_loop kernel/entry/common.c:181 [inline]
       exit_to_user_mode_prepare+0x253/0x280 kernel/entry/common.c:214
       __syscall_exit_to_user_mode_work kernel/entry/common.c:296 [inline]
       syscall_exit_to_user_mode+0x19/0x60 kernel/entry/common.c:307
       do_syscall_64+0x42/0xb0 arch/x86/entry/common.c:86
       entry_SYSCALL_64_after_hwframe+0x66/0xd0

-> #0 (&type->s_umount_key#54){+.+.}-{3:3}:
       check_prev_add kernel/locking/lockdep.c:3053 [inline]
       check_prevs_add kernel/locking/lockdep.c:3172 [inline]
       validate_chain kernel/locking/lockdep.c:3788 [inline]
       __lock_acquire+0x2a41/0x5340 kernel/locking/lockdep.c:5012
       lock_acquire kernel/locking/lockdep.c:5623 [inline]
       lock_acquire+0x1ab/0x4e0 kernel/locking/lockdep.c:5588
       down_write+0x92/0x220 kernel/locking/rwsem.c:1551
       freeze_super+0x41/0x3c0 fs/super.c:1682
       freeze_go_sync+0x1d6/0x320 fs/gfs2/glops.c:584
       do_xmote+0x2fd/0xc70 fs/gfs2/glock.c:742
       run_queue+0x31f/0x680 fs/gfs2/glock.c:870
       glock_work_func+0xff/0x390 fs/gfs2/glock.c:1037
       process_one_work+0x9bc/0x1550 kernel/workqueue.c:2310
       worker_thread+0x65d/0x1130 kernel/workqueue.c:2457
       kthread+0x3e5/0x4d0 kernel/kthread.c:334
       ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:300

other info that might help us debug this:

Chain exists of:
  &type->s_umount_key#54 --> (wq_completion)glock_workqueue -->
(work_completion)(&(&gl->gl_work)->work)

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock((work_completion)(&(&gl->gl_work)->work));
                               lock((wq_completion)glock_workqueue);
                               lock((work_completion)(&(&gl->gl_work)->work));
  lock(&type->s_umount_key#54);

 *** DEADLOCK ***

2 locks held by kworker/1:1H/156:
 #0: ffff888146f38138 ((wq_completion)glock_workqueue){+.+.}-{0:0},
at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline]
 #0: ffff888146f38138 ((wq_completion)glock_workqueue){+.+.}-{0:0},
at: arch_atomic_long_set include/linux/atomic/atomic-long.h:41
[inline]
 #0: ffff888146f38138 ((wq_completion)glock_workqueue){+.+.}-{0:0},
at: atomic_long_set include/linux/atomic/atomic-instrumented.h:1198
[inline]
 #0: ffff888146f38138 ((wq_completion)glock_workqueue){+.+.}-{0:0},
at: set_work_data kernel/workqueue.c:635 [inline]
 #0: ffff888146f38138 ((wq_completion)glock_workqueue){+.+.}-{0:0},
at: set_work_pool_and_clear_pending kernel/workqueue.c:662 [inline]
 #0: ffff888146f38138 ((wq_completion)glock_workqueue){+.+.}-{0:0},
at: process_one_work+0x8a0/0x1550 kernel/workqueue.c:2281
 #1: ffffc90001aefdb0
((work_completion)(&(&gl->gl_work)->work)){+.+.}-{0:0}, at:
process_one_work+0x8d4/0x1550 kernel/workqueue.c:2285

stack backtrace:
CPU: 1 PID: 156 Comm: kworker/1:1H Not tainted 5.15.159 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-1 04/01/2014
Workqueue: glock_workqueue glock_work_func
Call Trace:
 <TASK>
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0xcd/0x134 lib/dump_stack.c:106
 check_noncircular+0x268/0x310 kernel/locking/lockdep.c:2133
 check_prev_add kernel/locking/lockdep.c:3053 [inline]
 check_prevs_add kernel/locking/lockdep.c:3172 [inline]
 validate_chain kernel/locking/lockdep.c:3788 [inline]
 __lock_acquire+0x2a41/0x5340 kernel/locking/lockdep.c:5012
 lock_acquire kernel/locking/lockdep.c:5623 [inline]
 lock_acquire+0x1ab/0x4e0 kernel/locking/lockdep.c:5588
 down_write+0x92/0x220 kernel/locking/rwsem.c:1551
 freeze_super+0x41/0x3c0 fs/super.c:1682
 freeze_go_sync+0x1d6/0x320 fs/gfs2/glops.c:584
 do_xmote+0x2fd/0xc70 fs/gfs2/glock.c:742
 run_queue+0x31f/0x680 fs/gfs2/glock.c:870
 glock_work_func+0xff/0x390 fs/gfs2/glock.c:1037
 process_one_work+0x9bc/0x1550 kernel/workqueue.c:2310
 worker_thread+0x65d/0x1130 kernel/workqueue.c:2457
 kthread+0x3e5/0x4d0 kernel/kthread.c:334
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:300
 </TASK>
======================================================

Wishing you a nice day!

Best,
Marius

View attachment "repro.c" of type "text/x-csrc" (1107857 bytes)

Download attachment "repro.syz" of type "application/octet-stream" (323315 bytes)

Download attachment "config-5.15.159" of type "application/octet-stream" (227076 bytes)

Powered by blists - more mailing lists