Message-ID: <CAKFNMo=mGh9i7eqeg1WU__vYZZHV71fYj9A+6E-R8rvba5nbBA@mail.gmail.com>
Date: Sun, 19 May 2024 04:20:20 +0900
From: Ryusuke Konishi <konishi.ryusuke@...il.com>
To: syzbot <syzbot+c48f1971ba117125f94c@...kaller.appspotmail.com>
Cc: linux-kernel@...r.kernel.org, linux-nilfs@...r.kernel.org, 
	syzkaller-bugs@...glegroups.com
Subject: Re: [syzbot] [nilfs?] possible deadlock in nilfs_evict_inode (2)

On Sun, May 19, 2024 at 3:53 AM syzbot
<syzbot+c48f1971ba117125f94c@...kaller.appspotmail.com> wrote:
>
> Hello,
>
> syzbot found the following issue on:
>
> HEAD commit:    6bfd2d442af5 Merge tag 'irq-core-2024-05-12' of git://git..
> git tree:       upstream
> console output: https://syzkaller.appspot.com/x/log.txt?x=13aefc20980000
> kernel config:  https://syzkaller.appspot.com/x/.config?x=395546166dcfe360
> dashboard link: https://syzkaller.appspot.com/bug?extid=c48f1971ba117125f94c
> compiler:       gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40
> userspace arch: i386
>
> Unfortunately, I don't have any reproducer for this issue yet.
>
> Downloadable assets:
> disk image (non-bootable): https://storage.googleapis.com/syzbot-assets/7bc7510fe41f/non_bootable_disk-6bfd2d44.raw.xz
> vmlinux: https://storage.googleapis.com/syzbot-assets/7ad901fe99c6/vmlinux-6bfd2d44.xz
> kernel image: https://storage.googleapis.com/syzbot-assets/8d6ef2df621f/bzImage-6bfd2d44.xz
>
> IMPORTANT: if you fix the issue, please add the following tag to the commit:
> Reported-by: syzbot+c48f1971ba117125f94c@...kaller.appspotmail.com
>
> ======================================================
> WARNING: possible circular locking dependency detected
> 6.9.0-syzkaller-01893-g6bfd2d442af5 #0 Not tainted
> ------------------------------------------------------
> kswapd0/111 is trying to acquire lock:
> ffff888018e7e610 (sb_internal#4){.+.+}-{0:0}, at: nilfs_evict_inode+0x157/0x550 fs/nilfs2/inode.c:924
>
> but task is already holding lock:
> ffffffff8d9390c0 (fs_reclaim){+.+.}-{0:0}, at: balance_pgdat+0x166/0x1a10 mm/vmscan.c:6782
>
> which lock already depends on the new lock.
>
>
> the existing dependency chain (in reverse order) is:
>
> -> #2 (fs_reclaim){+.+.}-{0:0}:
>        __fs_reclaim_acquire mm/page_alloc.c:3698 [inline]
>        fs_reclaim_acquire+0x102/0x160 mm/page_alloc.c:3712
>        might_alloc include/linux/sched/mm.h:312 [inline]
>        prepare_alloc_pages.constprop.0+0x155/0x560 mm/page_alloc.c:4346
>        __alloc_pages+0x194/0x2460 mm/page_alloc.c:4564
>        alloc_pages_mpol+0x275/0x610 mm/mempolicy.c:2264
>        folio_alloc+0x1e/0x40 mm/mempolicy.c:2342
>        filemap_alloc_folio+0x3ba/0x490 mm/filemap.c:984
>        __filemap_get_folio+0x527/0xa90 mm/filemap.c:1926
>        pagecache_get_page+0x2c/0x260 mm/folio-compat.c:93
>        block_write_begin+0x38/0x4a0 fs/buffer.c:2209
>        nilfs_write_begin+0x9f/0x1a0 fs/nilfs2/inode.c:262
>        page_symlink+0x356/0x450 fs/namei.c:5236
>        nilfs_symlink+0x23c/0x3c0 fs/nilfs2/namei.c:153
>        vfs_symlink fs/namei.c:4489 [inline]
>        vfs_symlink+0x3e8/0x630 fs/namei.c:4473
>        do_symlinkat+0x263/0x310 fs/namei.c:4515
>        __do_sys_symlink fs/namei.c:4536 [inline]
>        __se_sys_symlink fs/namei.c:4534 [inline]
>        __ia32_sys_symlink+0x78/0xa0 fs/namei.c:4534
>        do_syscall_32_irqs_on arch/x86/entry/common.c:165 [inline]
>        __do_fast_syscall_32+0x75/0x120 arch/x86/entry/common.c:386
>        do_fast_syscall_32+0x32/0x80 arch/x86/entry/common.c:411
>        entry_SYSENTER_compat_after_hwframe+0x84/0x8e
>
> -> #1 (&nilfs->ns_segctor_sem){++++}-{3:3}:
>        down_read+0x9a/0x330 kernel/locking/rwsem.c:1526
>        nilfs_transaction_begin+0x326/0xa40 fs/nilfs2/segment.c:223
>        nilfs_symlink+0x114/0x3c0 fs/nilfs2/namei.c:140
>        vfs_symlink fs/namei.c:4489 [inline]
>        vfs_symlink+0x3e8/0x630 fs/namei.c:4473
>        do_symlinkat+0x263/0x310 fs/namei.c:4515
>        __do_sys_symlink fs/namei.c:4536 [inline]
>        __se_sys_symlink fs/namei.c:4534 [inline]
>        __ia32_sys_symlink+0x78/0xa0 fs/namei.c:4534
>        do_syscall_32_irqs_on arch/x86/entry/common.c:165 [inline]
>        __do_fast_syscall_32+0x75/0x120 arch/x86/entry/common.c:386
>        do_fast_syscall_32+0x32/0x80 arch/x86/entry/common.c:411
>        entry_SYSENTER_compat_after_hwframe+0x84/0x8e
>
> -> #0 (sb_internal#4){.+.+}-{0:0}:
>        check_prev_add kernel/locking/lockdep.c:3134 [inline]
>        check_prevs_add kernel/locking/lockdep.c:3253 [inline]
>        validate_chain kernel/locking/lockdep.c:3869 [inline]
>        __lock_acquire+0x2478/0x3b30 kernel/locking/lockdep.c:5137
>        lock_acquire kernel/locking/lockdep.c:5754 [inline]
>        lock_acquire+0x1b1/0x560 kernel/locking/lockdep.c:5719
>        percpu_down_read include/linux/percpu-rwsem.h:51 [inline]
>        __sb_start_write include/linux/fs.h:1661 [inline]
>        sb_start_intwrite include/linux/fs.h:1844 [inline]
>        nilfs_transaction_begin+0x21b/0xa40 fs/nilfs2/segment.c:220
>        nilfs_evict_inode+0x157/0x550 fs/nilfs2/inode.c:924
>        evict+0x2ed/0x6c0 fs/inode.c:667
>        iput_final fs/inode.c:1741 [inline]
>        iput.part.0+0x5a8/0x7f0 fs/inode.c:1767
>        iput+0x5c/0x80 fs/inode.c:1757
>        dentry_unlink_inode+0x295/0x440 fs/dcache.c:400
>        __dentry_kill+0x1d0/0x600 fs/dcache.c:603
>        shrink_kill fs/dcache.c:1048 [inline]
>        shrink_dentry_list+0x140/0x5d0 fs/dcache.c:1075
>        prune_dcache_sb+0xeb/0x150 fs/dcache.c:1156
>        super_cache_scan+0x32a/0x550 fs/super.c:221
>        do_shrink_slab+0x44f/0x11c0 mm/shrinker.c:435
>        shrink_slab_memcg mm/shrinker.c:548 [inline]
>        shrink_slab+0xa87/0x1310 mm/shrinker.c:626
>        shrink_one+0x493/0x7c0 mm/vmscan.c:4774
>        shrink_many mm/vmscan.c:4835 [inline]
>        lru_gen_shrink_node+0x89f/0x1750 mm/vmscan.c:4935
>        shrink_node mm/vmscan.c:5894 [inline]
>        kswapd_shrink_node mm/vmscan.c:6704 [inline]
>        balance_pgdat+0x10d1/0x1a10 mm/vmscan.c:6895
>        kswapd+0x5ea/0xbf0 mm/vmscan.c:7164
>        kthread+0x2c1/0x3a0 kernel/kthread.c:388
>        ret_from_fork+0x45/0x80 arch/x86/kernel/process.c:147
>        ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244
>
> other info that might help us debug this:
>
> Chain exists of:
>   sb_internal#4 --> &nilfs->ns_segctor_sem --> fs_reclaim
>
>  Possible unsafe locking scenario:
>
>        CPU0                    CPU1
>        ----                    ----
>   lock(fs_reclaim);
>                                lock(&nilfs->ns_segctor_sem);
>                                lock(fs_reclaim);
>   rlock(sb_internal#4);
>
>  *** DEADLOCK ***
>
> 2 locks held by kswapd0/111:
>  #0: ffffffff8d9390c0 (fs_reclaim){+.+.}-{0:0}, at: balance_pgdat+0x166/0x1a10 mm/vmscan.c:6782
>  #1: ffff888018e7e0e0 (&type->s_umount_key#74){++++}-{3:3}, at: super_trylock_shared fs/super.c:561 [inline]
>  #1: ffff888018e7e0e0 (&type->s_umount_key#74){++++}-{3:3}, at: super_cache_scan+0x96/0x550 fs/super.c:196
>
> stack backtrace:
> CPU: 2 PID: 111 Comm: kswapd0 Not tainted 6.9.0-syzkaller-01893-g6bfd2d442af5 #0
> Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
> Call Trace:
>  <TASK>
>  __dump_stack lib/dump_stack.c:88 [inline]
>  dump_stack_lvl+0x116/0x1f0 lib/dump_stack.c:114
>  check_noncircular+0x31a/0x400 kernel/locking/lockdep.c:2187
>  check_prev_add kernel/locking/lockdep.c:3134 [inline]
>  check_prevs_add kernel/locking/lockdep.c:3253 [inline]
>  validate_chain kernel/locking/lockdep.c:3869 [inline]
>  __lock_acquire+0x2478/0x3b30 kernel/locking/lockdep.c:5137
>  lock_acquire kernel/locking/lockdep.c:5754 [inline]
>  lock_acquire+0x1b1/0x560 kernel/locking/lockdep.c:5719
>  percpu_down_read include/linux/percpu-rwsem.h:51 [inline]
>  __sb_start_write include/linux/fs.h:1661 [inline]
>  sb_start_intwrite include/linux/fs.h:1844 [inline]
>  nilfs_transaction_begin+0x21b/0xa40 fs/nilfs2/segment.c:220
>  nilfs_evict_inode+0x157/0x550 fs/nilfs2/inode.c:924
>  evict+0x2ed/0x6c0 fs/inode.c:667
>  iput_final fs/inode.c:1741 [inline]
>  iput.part.0+0x5a8/0x7f0 fs/inode.c:1767
>  iput+0x5c/0x80 fs/inode.c:1757
>  dentry_unlink_inode+0x295/0x440 fs/dcache.c:400
>  __dentry_kill+0x1d0/0x600 fs/dcache.c:603
>  shrink_kill fs/dcache.c:1048 [inline]
>  shrink_dentry_list+0x140/0x5d0 fs/dcache.c:1075
>  prune_dcache_sb+0xeb/0x150 fs/dcache.c:1156
>  super_cache_scan+0x32a/0x550 fs/super.c:221
>  do_shrink_slab+0x44f/0x11c0 mm/shrinker.c:435
>  shrink_slab_memcg mm/shrinker.c:548 [inline]
>  shrink_slab+0xa87/0x1310 mm/shrinker.c:626
>  shrink_one+0x493/0x7c0 mm/vmscan.c:4774
>  shrink_many mm/vmscan.c:4835 [inline]
>  lru_gen_shrink_node+0x89f/0x1750 mm/vmscan.c:4935
>  shrink_node mm/vmscan.c:5894 [inline]
>  kswapd_shrink_node mm/vmscan.c:6704 [inline]
>  balance_pgdat+0x10d1/0x1a10 mm/vmscan.c:6895
>  kswapd+0x5ea/0xbf0 mm/vmscan.c:7164
>  kthread+0x2c1/0x3a0 kernel/kthread.c:388
>  ret_from_fork+0x45/0x80 arch/x86/kernel/process.c:147
>  ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244
>  </TASK>
>
>
> ---
> This report is generated by a bot. It may contain errors.
> See https://goo.gl/tpsmEJ for more information about syzbot.
> syzbot engineers can be reached at syzkaller@...glegroups.com.
>
> syzbot will keep track of this issue. See:
> https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
>
> If the report is already addressed, let syzbot know by replying with:
> #syz fix: exact-commit-title
>
> If you want to overwrite report's subsystems, reply with:
> #syz set subsystems: new-subsystem
> (See the list of subsystem names on the web dashboard)
>
> If the report is a duplicate of another one, reply with:
> #syz dup: exact-subject-of-another-report
>
> If you want to undo deduplication, reply with:
> #syz undup

Similarly, it seems to have the same root as the report below, but I
can't confirm it right now, so I'll leave it now.

https://syzkaller.appspot.com/bug?extid=ca73f5a22aec76875d85

Similarly, the GFP flags on the symlink's page cache allocation
appears to be causing this circular lock dependency.

Ryusuke Konishi

Powered by blists - more mailing lists