linux-kernel - Re: [syzbot] WARNING in __percpu_ref

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Message-ID: <79ce5bcd-b314-dded-0b36-0a8fb66f5a7a@gmail.com>
Date:   Thu, 18 Mar 2021 14:28:11 +0000
From:   Pavel Begunkov <asml.silence@...il.com>
To:     Hillf Danton <hdanton@...a.com>, Ming Lei <ming.lei@...hat.com>
Cc:     syzbot <syzbot+d6218cb2fae0b2411e9d@...kaller.appspotmail.com>,
        axboe@...nel.dk, io-uring@...r.kernel.org,
        linux-kernel@...r.kernel.org, syzkaller-bugs@...glegroups.com
Subject: Re: [syzbot] WARNING in __percpu_ref_exit (2)

On 18/03/2021 08:33, Hillf Danton wrote:
> On Mon, 15 Mar 2021 12:18:20 +0000 Pavel Begunkov wrote:
>> On 15/03/2021 11:58, syzbot wrote:
>>> Hello,
>>>
>>> syzbot found the following issue on:
>>>
>>> HEAD commit:    75013c6c Merge tag 'perf_urgent_for_v5.12-rc3' of git://gi..
>>> git tree:       upstream
>>> console output: https://syzkaller.appspot.com/x/log.txt?x=174df32ad00000
>>> kernel config:  https://syzkaller.appspot.com/x/.config?x=844457676c06b88c
>>> dashboard link: https://syzkaller.appspot.com/bug?extid=d6218cb2fae0b2411e9d
>>> userspace arch: i386
>>>
>>> Unfortunately, I don't have any reproducer for this issue yet.
>>>
>>> IMPORTANT: if you fix the issue, please add the following tag to the commit:
>>> Reported-by: syzbot+d6218cb2fae0b2411e9d@...kaller.appspotmail.com
>>> ------------[ cut here ]------------
>>> WARNING: CPU: 1 PID: 53 at lib/percpu-refcount.c:113 __percpu_ref_exit+0x98/0x100 lib/percpu-refcount.c:113
>>
>> if (percpu_count) {
>> 	/* non-NULL confirm_switch indicates switching in progress */
>> 	WARN_ON_ONCE(ref->data && ref->data->confirm_switch);
>> 	...
>> }
>>
>> Points to this warning. Not sure, but not yet included
>> "io_uring: halt SQO submission on ctx exit" may fix it or at least is
>> related.
> 
> Seems it does not, nor related, see below.
>>
>>> Modules linked in:
>>> CPU: 1 PID: 53 Comm: kworker/u4:2 Not tainted 5.12.0-rc2-syzkaller #0
>>> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
>>> Workqueue: events_unbound io_ring_exit_work
>>> RIP: 0010:__percpu_ref_exit+0x98/0x100 lib/percpu-refcount.c:113
>>> Code: fd 49 8d 7c 24 10 48 b8 00 00 00 00 00 fc ff df 48 89 fa 48 c1 ea 03 80 3c 02 00 75 61 49 83 7c 24 10 00 74 07 e8 28 42 ac fd <0f> 0b e8 21 42 ac fd 48 89 ef e8 e9 fa da fd 48 89 da 48 b8 00 00
>>> RSP: 0018:ffffc90000f1fb78 EFLAGS: 00010293
>>> RAX: 0000000000000000 RBX: ffff88805c976000 RCX: 0000000000000000
>>> RDX: ffff888011839bc0 RSI: ffffffff83c76be8 RDI: ffff88802b2a9010
>>> RBP: 0000607f46077778 R08: 0000000000000000 R09: ffffffff8fab0967
>>> R10: ffffffff83c76b88 R11: 0000000000000009 R12: ffff88802b2a9000
>>> R13: 0000000000000001 R14: ffff88802b2a9000 R15: dffffc0000000000
>>> FS:  0000000000000000(0000) GS:ffff8880b9c00000(0000) knlGS:0000000000000000
>>> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>> CR2: 00000000085a0004 CR3: 000000001896a000 CR4: 00000000001506f0
>>> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>>> DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
>>> Call Trace:
>>>  percpu_ref_exit+0x3b/0x140 lib/percpu-refcount.c:134
>>>  io_ring_ctx_free fs/io_uring.c:8419 [inline]
>>>  io_ring_exit_work+0x599/0xcf0 fs/io_uring.c:8565
>>>  process_one_work+0x98d/0x1600 kernel/workqueue.c:2275
>>>  worker_thread+0x64c/0x1120 kernel/workqueue.c:2421
>>>  kthread+0x3b1/0x4a0 kernel/kthread.c:292
>>>  ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:294
>>>
>>>
>>> ---
>>> This report is generated by a bot. It may contain errors.
>>> See https://goo.gl/tpsmEJ for more information about syzbot.
>>> syzbot engineers can be reached at syzkaller@...glegroups.com.
>>>
>>> syzbot will keep track of this issue. See:
>>> https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
>>>
>>
>> -- 
>> Pavel Begunkov
> 
> Thoughts for sync RCU are appreciated if the chance for the race
> between rcu and workqueue is not zero on killing io ctx.

Would you elaborate? Because your case below doesn't make much
sense.

1) io_ring_ctx_wait_and_kill() indeed kills ctx->refs
2) io_ring_exit_work() waits for a completion signaled by
ctx->refs hitting 0
3) and only then calls io_ring_ctx_free().

And 2) won't complete until the switching is over

> 
> CPU0
> ----
> io_ring_ctx_wait_and_kill
> percpu_ref_kill(&ctx->refs);
> percpu_ref_kill_and_confirm(ref, NULL);
> spin_lock_irqsave(&percpu_ref_switch_lock, flags);
> __percpu_ref_switch_mode(ref, confirm_switch);
>   __percpu_ref_switch_to_atomic
>     ref->data->confirm_switch = confirm_switch ?:
> 		percpu_ref_noop_confirm_switch;
>     call_rcu(&ref->data->rcu, percpu_ref_switch_to_atomic_rcu);
> spin_unlock_irqrestore(&percpu_ref_switch_lock, flags);
> 
> INIT_WORK(&ctx->exit_work, io_ring_exit_work);
> queue_work(system_unbound_wq, &ctx->exit_work);
> 
> 						CPU1
> 						----
> 						io_ring_exit_work
> 						io_ring_ctx_free(ctx);
> 						percpu_ref_exit(&ctx->refs);
> 						__percpu_ref_exit(ref);
> 						WARN_ON_ONCE(ref->data &&
> 							ref->data->confirm_switch);
> 
> 
> percpu_ref_switch_to_atomic_rcu
>   percpu_ref_call_confirm_rcu(rcu);
>     data->confirm_switch(ref);
>     data->confirm_switch = NULL;
>     wake_up_all(&percpu_ref_switch_waitq);
> 

-- 
Pavel Begunkov