Message-ID: <CACT4Y+Y8CuDydXhKA2ZtdPYY=vQLkq_1L3Stv=TNGwgs=MPK=A@mail.gmail.com>
Date:   Tue, 7 Nov 2017 12:41:40 +0100
From:   Dmitry Vyukov <dvyukov@...gle.com>
To:     syzbot 
        <bot+c9695f0404c72c8f0b7abbac660d986af9008fc5@...kaller.appspotmail.com>
Cc:     LKML <linux-kernel@...r.kernel.org>,
        Ingo Molnar <mingo@...hat.com>,
        Steven Rostedt <rostedt@...dmis.org>,
        syzkaller-bugs@...glegroups.com,
        Peter Zijlstra <peterz@...radead.org>
Subject: Re: possible deadlock in perf_trace_destroy (2)

On Tue, Nov 7, 2017 at 12:39 PM, syzbot
<bot+c9695f0404c72c8f0b7abbac660d986af9008fc5@...kaller.appspotmail.com>
wrote:
> Hello,
>
> syzkaller hit the following crash on
> 5a3517e009e979f21977d362212b7729c5165d92
> git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/master
> compiler: gcc (GCC) 7.1.1 20170620
> .config is attached
> Raw console output is attached.
>
> syzkaller reproducer is attached. See https://goo.gl/kgGztJ
> for information about syzkaller reproducers
>
>
> ======================================================
> WARNING: possible circular locking dependency detected
> 4.14.0-rc7-next-20171103+ #10 Not tainted
> ------------------------------------------------------
> syz-executor1/3061 is trying to acquire lock:
>  (event_mutex){+.+.}, at: [<ffffffff81776de8>] perf_trace_destroy+0x28/0x100
> kernel/trace/trace_event_perf.c:234
>
> but task is already holding lock:
>  (&mm->mmap_sem){++++}, at: [<ffffffff819448b8>] vm_mmap_pgoff+0x198/0x280
> mm/util.c:331
>
> which lock already depends on the new lock.
>
>
> the existing dependency chain (in reverse order) is:
>
> -> #7 (&mm->mmap_sem){++++}:
>        lock_acquire+0x1d5/0x580 kernel/locking/lockdep.c:4004
>        __might_fault+0x13a/0x1d0 mm/memory.c:4530
>        _copy_to_user+0x2c/0xc0 lib/usercopy.c:25
>        copy_to_user include/linux/uaccess.h:155 [inline]
>        filldir+0x1a7/0x320 fs/readdir.c:196
>        dir_emit_dot include/linux/fs.h:3343 [inline]
>        dir_emit_dots include/linux/fs.h:3354 [inline]
>        dcache_readdir+0x12d/0x5e0 fs/libfs.c:192
>        iterate_dir+0x1ca/0x540 fs/readdir.c:51
>        SYSC_getdents fs/readdir.c:231 [inline]
>        SyS_getdents+0x225/0x450 fs/readdir.c:212
>        entry_SYSCALL_64_fastpath+0x1f/0xbe
>
> -> #6 (&sb->s_type->i_mutex_key#5){++++}:
>        rq_lock kernel/sched/sched.h:1747 [inline]
>        ttwu_queue kernel/sched/core.c:1860 [inline]
>        try_to_wake_up+0xa53/0x1850 kernel/sched/core.c:2075
>
> -> #5 ((completion)&req.done){+.+.}:
>        lock_acquire+0x1d5/0x580 kernel/locking/lockdep.c:4004
>        complete_acquire include/linux/completion.h:40 [inline]
>        __wait_for_common kernel/sched/completion.c:109 [inline]
>        wait_for_common kernel/sched/completion.c:123 [inline]
>        wait_for_completion+0xcb/0x7b0 kernel/sched/completion.c:144
>        devtmpfs_create_node+0x32b/0x4a0 drivers/base/devtmpfs.c:115
>        device_add+0x120f/0x1640 drivers/base/core.c:1824
>        device_create_groups_vargs+0x1f3/0x250 drivers/base/core.c:2430
>        device_create_vargs drivers/base/core.c:2470 [inline]
>        device_create+0xda/0x110 drivers/base/core.c:2506
>        msr_device_create+0x26/0x40 arch/x86/kernel/msr.c:188
>        cpuhp_invoke_callback+0x2ea/0x1d20 kernel/cpu.c:182
>        cpuhp_thread_fun+0x48b/0x7e0 kernel/cpu.c:571
>        smpboot_thread_fn+0x450/0x7c0 kernel/smpboot.c:164
>        kthread+0x3c9/0x4b0 kernel/kthread.c:242
>        ret_from_fork+0x2a/0x40 arch/x86/entry/entry_64.S:432
>
> -> #4 (cpuhp_state-up){+.+.}:
>        lock_acquire+0x1d5/0x580 kernel/locking/lockdep.c:4004
>        cpuhp_lock_acquire kernel/cpu.c:85 [inline]
>        cpuhp_invoke_ap_callback kernel/cpu.c:605 [inline]
>        cpuhp_issue_call+0x1e5/0x520 kernel/cpu.c:1495
>        __cpuhp_setup_state_cpuslocked+0x2c7/0x5f0 kernel/cpu.c:1642
>        __cpuhp_setup_state+0xb0/0x140 kernel/cpu.c:1671
>        cpuhp_setup_state include/linux/cpuhotplug.h:200 [inline]
>        page_writeback_init+0x4d/0x71 mm/page-writeback.c:2083
>        pagecache_init+0x48/0x4f mm/filemap.c:977
>        start_kernel+0x6bc/0x74f init/main.c:690
>        x86_64_start_reservations+0x2a/0x2c arch/x86/kernel/head64.c:378
>        x86_64_start_kernel+0x77/0x7a arch/x86/kernel/head64.c:359
>        secondary_startup_64+0xa5/0xb0 arch/x86/kernel/head_64.S:240
>
> -> #3 (cpuhp_state_mutex){+.+.}:
>        lock_acquire+0x1d5/0x580 kernel/locking/lockdep.c:4004
>        __mutex_lock_common kernel/locking/mutex.c:756 [inline]
>        __mutex_lock+0x16f/0x19d0 kernel/locking/mutex.c:893
>        mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
>        __cpuhp_setup_state_cpuslocked+0x5b/0x5f0 kernel/cpu.c:1617
>        __cpuhp_setup_state+0xb0/0x140 kernel/cpu.c:1671
>        cpuhp_setup_state_nocalls include/linux/cpuhotplug.h:228 [inline]
>        kvm_guest_init+0x1f3/0x20f arch/x86/kernel/kvm.c:495
>        setup_arch+0x17cb/0x19e5 arch/x86/kernel/setup.c:1264
>        start_kernel+0xa5/0x74f init/main.c:530
>        x86_64_start_reservations+0x2a/0x2c arch/x86/kernel/head64.c:378
>        x86_64_start_kernel+0x77/0x7a arch/x86/kernel/head64.c:359
>        secondary_startup_64+0xa5/0xb0 arch/x86/kernel/head_64.S:240
>
> -> #2 (cpu_hotplug_lock.rw_sem){++++}:
>        lock_acquire+0x1d5/0x580 kernel/locking/lockdep.c:4004
>        percpu_down_read_preempt_disable include/linux/percpu-rwsem.h:36
> [inline]
>        percpu_down_read include/linux/percpu-rwsem.h:59 [inline]
>        cpus_read_lock+0x42/0x90 kernel/cpu.c:293
>        static_key_slow_inc+0x9d/0x3c0 kernel/jump_label.c:123
>        tracepoint_add_func kernel/tracepoint.c:223 [inline]
>        tracepoint_probe_register_prio+0x80d/0x9a0 kernel/tracepoint.c:283
>        tracepoint_probe_register+0x2a/0x40 kernel/tracepoint.c:304
>        trace_event_reg+0x167/0x320 kernel/trace/trace_events.c:305
>        perf_trace_event_reg kernel/trace/trace_event_perf.c:122 [inline]
>        perf_trace_event_init kernel/trace/trace_event_perf.c:197 [inline]
>        perf_trace_init+0x4ef/0xab0 kernel/trace/trace_event_perf.c:221
>        perf_tp_event_init+0x7d/0xf0 kernel/events/core.c:7953
>        perf_try_init_event+0xc9/0x1f0 kernel/events/core.c:9179
>        perf_init_event kernel/events/core.c:9217 [inline]
>        perf_event_alloc+0x1c5b/0x2a00 kernel/events/core.c:9481
>        SYSC_perf_event_open+0x842/0x2f10 kernel/events/core.c:9936
>        SyS_perf_event_open+0x39/0x50 kernel/events/core.c:9822
>        entry_SYSCALL_64_fastpath+0x1f/0xbe
>
> -> #1 (tracepoints_mutex){+.+.}:
>        lock_acquire+0x1d5/0x580 kernel/locking/lockdep.c:4004
>        __mutex_lock_common kernel/locking/mutex.c:756 [inline]
>        __mutex_lock+0x16f/0x19d0 kernel/locking/mutex.c:893
>        mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
>        tracepoint_probe_register_prio+0xa0/0x9a0 kernel/tracepoint.c:279
>        tracepoint_probe_register+0x2a/0x40 kernel/tracepoint.c:304
>        trace_event_reg+0x167/0x320 kernel/trace/trace_events.c:305
>        perf_trace_event_reg kernel/trace/trace_event_perf.c:122 [inline]
>        perf_trace_event_init kernel/trace/trace_event_perf.c:197 [inline]
>        perf_trace_init+0x4ef/0xab0 kernel/trace/trace_event_perf.c:221
>        perf_tp_event_init+0x7d/0xf0 kernel/events/core.c:7953
>        perf_try_init_event+0xc9/0x1f0 kernel/events/core.c:9179
>        perf_init_event kernel/events/core.c:9217 [inline]
>        perf_event_alloc+0x1c5b/0x2a00 kernel/events/core.c:9481
>        SYSC_perf_event_open+0x842/0x2f10 kernel/events/core.c:9936
>        SyS_perf_event_open+0x39/0x50 kernel/events/core.c:9822
>        entry_SYSCALL_64_fastpath+0x1f/0xbe
>
> -> #0 (event_mutex){+.+.}:
>        check_prevs_add kernel/locking/lockdep.c:2031 [inline]
>        validate_chain kernel/locking/lockdep.c:2473 [inline]
>        __lock_acquire+0x3374/0x4770 kernel/locking/lockdep.c:3500
>        lock_acquire+0x1d5/0x580 kernel/locking/lockdep.c:4004
>        __mutex_lock_common kernel/locking/mutex.c:756 [inline]
>        __mutex_lock+0x16f/0x19d0 kernel/locking/mutex.c:893
>        mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
>        perf_trace_destroy+0x28/0x100 kernel/trace/trace_event_perf.c:234
>        tp_perf_event_destroy+0x15/0x20 kernel/events/core.c:7937
>        _free_event+0x3bd/0x10f0 kernel/events/core.c:4105
>        put_event+0x24/0x30 kernel/events/core.c:4188
>        perf_mmap_close+0x60d/0x1010 kernel/events/core.c:5147
>        remove_vma+0xb4/0x1b0 mm/mmap.c:172
>        remove_vma_list mm/mmap.c:2475 [inline]
>        do_munmap+0x82a/0xdf0 mm/mmap.c:2714
>        mmap_region+0x59e/0x15a0 mm/mmap.c:1631
>        do_mmap+0x6a1/0xd50 mm/mmap.c:1468
>        do_mmap_pgoff include/linux/mm.h:2200 [inline]
>        vm_mmap_pgoff+0x1de/0x280 mm/util.c:333
>        SYSC_mmap_pgoff mm/mmap.c:1518 [inline]
>        SyS_mmap_pgoff+0x23b/0x5f0 mm/mmap.c:1476
>        SYSC_mmap arch/x86/kernel/sys_x86_64.c:100 [inline]
>        SyS_mmap+0x16/0x20 arch/x86/kernel/sys_x86_64.c:91
>        entry_SYSCALL_64_fastpath+0x1f/0xbe
>
> other info that might help us debug this:
>
> Chain exists of:
>   event_mutex --> &sb->s_type->i_mutex_key#5 --> &mm->mmap_sem
>
>  Possible unsafe locking scenario:
>
>        CPU0                    CPU1
>        ----                    ----
>   lock(&mm->mmap_sem);
>                                lock(&sb->s_type->i_mutex_key#5);
>                                lock(&mm->mmap_sem);
>   lock(event_mutex);
>
>  *** DEADLOCK ***
>
> 1 lock held by syz-executor1/3061:
>  #0:  (&mm->mmap_sem){++++}, at: [<ffffffff819448b8>]
> vm_mmap_pgoff+0x198/0x280 mm/util.c:331
>
> stack backtrace:
> CPU: 0 PID: 3061 Comm: syz-executor1 Not tainted 4.14.0-rc7-next-20171103+
> #10
> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
> Call Trace:
>  __dump_stack lib/dump_stack.c:17 [inline]
>  dump_stack+0x194/0x257 lib/dump_stack.c:53
>  print_circular_bug+0x503/0x710 kernel/locking/lockdep.c:1271
>  check_prev_add+0x8b1/0x1580 kernel/locking/lockdep.c:1914
>  check_prevs_add kernel/locking/lockdep.c:2031 [inline]
>  validate_chain kernel/locking/lockdep.c:2473 [inline]
>  __lock_acquire+0x3374/0x4770 kernel/locking/lockdep.c:3500
>  lock_acquire+0x1d5/0x580 kernel/locking/lockdep.c:4004
>  __mutex_lock_common kernel/locking/mutex.c:756 [inline]
>  __mutex_lock+0x16f/0x19d0 kernel/locking/mutex.c:893
>  mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
>  perf_trace_destroy+0x28/0x100 kernel/trace/trace_event_perf.c:234
>  tp_perf_event_destroy+0x15/0x20 kernel/events/core.c:7937
>  _free_event+0x3bd/0x10f0 kernel/events/core.c:4105
>  put_event+0x24/0x30 kernel/events/core.c:4188
>  perf_mmap_close+0x60d/0x1010 kernel/events/core.c:5147
>  remove_vma+0xb4/0x1b0 mm/mmap.c:172
>  remove_vma_list mm/mmap.c:2475 [inline]
>  do_munmap+0x82a/0xdf0 mm/mmap.c:2714
>  mmap_region+0x59e/0x15a0 mm/mmap.c:1631
>  do_mmap+0x6a1/0xd50 mm/mmap.c:1468
>  do_mmap_pgoff include/linux/mm.h:2200 [inline]
>  vm_mmap_pgoff+0x1de/0x280 mm/util.c:333
>  SYSC_mmap_pgoff mm/mmap.c:1518 [inline]
>  SyS_mmap_pgoff+0x23b/0x5f0 mm/mmap.c:1476
>  SYSC_mmap arch/x86/kernel/sys_x86_64.c:100 [inline]
>  SyS_mmap+0x16/0x20 arch/x86/kernel/sys_x86_64.c:91
>  entry_SYSCALL_64_fastpath+0x1f/0xbe
> RIP: 0033:0x447c89
> RSP: 002b:00007f49b0affbd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000009
> RAX: ffffffffffffffda RBX: 00007f49b0b006cc RCX: 0000000000447c89
> RDX: 0000000000000000 RSI: 0000000000003000 RDI: 0000000020007000
> RBP: 0000000000000086 R08: 0000000000000000 R09: 0000000000000000
> R10: 0000000000000032 R11: 0000000000000246 R12: 0000000000000000
> R13: 0000000000000000 R14: 00007f49b0b009c0 R15: 00007f49b0b00700


Set of mutexes looks somewhat similar to "possible deadlock in
lru_add_drain_all":
https://groups.google.com/forum/#!topic/syzkaller-bugs/uZpWvxGXT68
But this one has perf/tracepoints involved.
+Peter


> ---
> This bug is generated by a dumb bot. It may contain errors.
> See https://goo.gl/tpsmEJ for details.
> Direct all questions to syzkaller@...glegroups.com.
> Please credit me with: Reported-by: syzbot <syzkaller@...glegroups.com>
>
> syzbot will keep track of this bug report.
> Once a fix for this bug is committed, please reply to this email with:
> #syz fix: exact-commit-title
> To mark this as a duplicate of another syzbot report, please reply with:
> #syz dup: exact-subject-of-another-report
> If it's a one-off invalid bug report, please reply with:
> #syz invalid
> Note: if the crash happens again, it will cause creation of a new bug
> report.
> Note: all commands must start from beginning of the line.
>
> --
> You received this message because you are subscribed to the Google Groups
> "syzkaller-bugs" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to syzkaller-bugs+unsubscribe@...glegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/syzkaller-bugs/001a114aa95c553065055d6307e2%40google.com.
> For more options, visit https://groups.google.com/d/optout.

Powered by blists - more mailing lists