linux-kernel - Re: [syzbot] WARNING in percpu_ref_switch_to_atomic

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Message-ID: <839f6bce-101d-a971-ad17-61e02f5007a2@gmail.com>
Date:   Thu, 29 Apr 2021 10:50:29 +0100
From:   Pavel Begunkov <asml.silence@...il.com>
To:     Hillf Danton <hdanton@...a.com>,
        syzbot <syzbot+842c825a3ade3fb904bf@...kaller.appspotmail.com>
Cc:     Jens Axboe <axboe@...nel.dk>, linux-kernel@...r.kernel.org,
        syzkaller-bugs@...glegroups.com
Subject: Re: [syzbot] WARNING in percpu_ref_switch_to_atomic_rcu (2)

On 4/29/21 10:38 AM, Hillf Danton wrote:
> On Wed, 28 Apr 2021 07:40:31 
>> Hello,
>>
>> syzbot found the following issue on:
>>
>> HEAD commit:    57fa2369 Merge tag 'cfi-v5.13-rc1' of git://git.kernel.org..
>> git tree:       upstream
>> console output: https://syzkaller.appspot.com/x/log.txt?x=173e8725d00000
>> kernel config:  https://syzkaller.appspot.com/x/.config?x=c37352b2923ef305
>> dashboard link: https://syzkaller.appspot.com/bug?extid=842c825a3ade3fb904bf
>> syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=135f0c7dd00000
>>
>> IMPORTANT: if you fix the issue, please add the following tag to the commit:
>> Reported-by: syzbot+842c825a3ade3fb904bf@...kaller.appspotmail.com
>>
>> ------------[ cut here ]------------
>> percpu ref (free_ioctx_reqs) <= 0 (0) after switching to atomic
> 
> Is it ctx->refs leak?
> 
>> WARNING: CPU: 0 PID: 9966 at lib/percpu-refcount.c:196 percpu_ref_switch_to_atomic_rcu+0x3e4/0x4a0 lib/percpu-refcount.c:196
>> Modules linked in:
>> CPU: 0 PID: 9966 Comm: kworker/u4:4 Not tainted 5.12.0-syzkaller #0
>> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
>> Workqueue: events_unbound call_usermodehelper_exec_work
>> RIP: 0010:percpu_ref_switch_to_atomic_rcu+0x3e4/0x4a0 lib/percpu-refcount.c:196
>> Code: df 48 89 fa 48 c1 ea 03 80 3c 02 00 0f 85 ba 00 00 00 48 8b 44 24 08 4c 89 e2 48 c7 c7 80 05 c2 89 48 8b 70 e8 e8 cb 7c fb 04 <0f> 0b e9 3b ff ff ff 48 c7 c7 a4 27 c6 8d e8 39 a9 ee fd e9 19 fd
>> RSP: 0018:ffffc90000007e28 EFLAGS: 00010282
>> RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
>> RDX: ffff88803c80b880 RSI: ffffffff815c3fc5 RDI: fffff52000000fb7
>> RBP: ffff888027181980 R08: 0000000000000000 R09: 0000000000000000
>> R10: ffffffff815bcd5e R11: 0000000000000000 R12: 0000000000000000
>> R13: 0000607f46081cc0 R14: 0000000000000007 R15: 0000000000000002
>> FS:  0000000000000000(0000) GS:ffff8880b9c00000(0000) knlGS:0000000000000000
>> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> CR2: 0000000020000300 CR3: 0000000012084000 CR4: 00000000001506f0
>> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
>> Call Trace:
>>  <IRQ>
>>  rcu_do_batch kernel/rcu/tree.c:2559 [inline]
>>  rcu_core+0x74a/0x12f0 kernel/rcu/tree.c:2794
>>  __do_softirq+0x29b/0x9f6 kernel/softirq.c:559
>>  invoke_softirq kernel/softirq.c:433 [inline]
>>  __irq_exit_rcu kernel/softirq.c:637 [inline]
>>  irq_exit_rcu+0x134/0x200 kernel/softirq.c:649
>>  sysvec_apic_timer_interrupt+0x93/0xc0 arch/x86/kernel/apic/apic.c:1100
>>  </IRQ>
>>  asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:632
>> RIP: 0010:preempt_count arch/x86/include/asm/preempt.h:27 [inline]
>> RIP: 0010:check_kcov_mode kernel/kcov.c:163 [inline]
>> RIP: 0010:__sanitizer_cov_trace_pc+0x0/0x60 kernel/kcov.c:197
>> Code: f0 4d 89 03 e9 f2 fc ff ff b9 ff ff ff ff ba 08 00 00 00 4d 8b 03 48 0f bd ca 49 8b 45 00 48 63 c9 e9 64 ff ff ff 0f 1f 40 00 <65> 8b 05 29 07 8e 7e 89 c1 48 8b 34 24 81 e1 00 01 00 00 65 48 8b
>> RSP: 0018:ffffc9000b2f78d0 EFLAGS: 00000293
>> RAX: 0000000000000000 RBX: 0000000000000200 RCX: 0000000000000000
>> RDX: ffff88803c80b880 RSI: ffffffff816465bf RDI: 0000000000000003
>> RBP: 0000000000044664 R08: 0000000000000000 R09: ffffffff901948b7
>> R10: ffffffff816465b5 R11: 0000000000000000 R12: 0000000000000000
>> R13: dffffc0000000000 R14: ffffffff8bf92b90 R15: 0000000000002192
>>  __seqprop_raw_spinlock_sequence include/linux/seqlock.h:276 [inline]
>>  timekeeping_get_delta kernel/time/timekeeping.c:252 [inline]
>>  timekeeping_get_ns kernel/time/timekeeping.c:386 [inline]
>>  ktime_get_with_offset+0x1f5/0x500 kernel/time/timekeeping.c:872
>>  ktime_get_boottime include/linux/timekeeping.h:94 [inline]
>>  ktime_get_boottime_ns include/linux/timekeeping.h:163 [inline]
>>  copy_process+0x350f/0x71a0 kernel/fork.c:2226
>>  kernel_clone+0xe7/0xab0 kernel/fork.c:2500
>>  kernel_thread+0xb5/0xf0 kernel/fork.c:2552
>>  call_usermodehelper_exec_work kernel/umh.c:172 [inline]
>>  call_usermodehelper_exec_work+0xcc/0x180 kernel/umh.c:158
>>  process_one_work+0x98d/0x1600 kernel/workqueue.c:2275
>>  worker_thread+0x64c/0x1120 kernel/workqueue.c:2421
>>  kthread+0x3b1/0x4a0 kernel/kthread.c:292
>>  ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:294
> 
> Before finding an answer to the question above, spot a chance to add a line
> in bid to restore the old behavior - reinit completion, otherwise refs cannot
> make much sense in terms of race that ctx goes home without waiting for its
> refs to drop down to ground.

I'd highly encourage you to add understandable descriptions
because this one doesn't make any sense. Why you add it?

Even more curious, is it io_uring related? Because the repro
does only aio, as well as the error happened in aio, see
free_ioctx_reqs()

> 
> +++ x/fs/io_uring.c
> @@ -1085,6 +1085,7 @@ static void io_refs_resurrect(struct per
>  	if (!got)
>  		wait_for_completion(compl);
>  	percpu_ref_resurrect(ref);
> +	reinit_completion(compl);
>  	if (got)
>  		percpu_ref_put(ref);
>  }
> 

-- 
Pavel Begunkov