Message-Id: <20181114150859.GT4170@linux.ibm.com>
Date:   Wed, 14 Nov 2018 07:08:59 -0800
From:   "Paul E. McKenney" <paulmck@...ux.ibm.com>
To:     Kyungtae Kim <kt0755@...il.com>
Cc:     josh@...htriplett.org, rostedt@...dmis.org,
        mathieu.desnoyers@...icios.com, jiangshanlai@...il.com,
        Byoungyoung Lee <lifeasageek@...il.com>,
        DaeRyong Jeong <threeearcat@...il.com>,
        syzkaller@...glegroups.com, linux-kernel@...r.kernel.org
Subject: Re: KMSAN: uninit-value in rcu_accelerate_cbs / KMSAN: uninit-value
 in rcu_process_callbacks

On Wed, Nov 14, 2018 at 04:03:33AM -0500, Kyungtae Kim wrote:
> We report two crashes in v4.19-rc8 (4.20-rc1 as well, I guess):
> (Unfortunately, there is no repro for those.)
> 
> The two crashes seem to share the same issue.
> In both cases, (uninitialized) memory access violation occurs
> when "rdp->cblist" is about to be accessed (kernel/rcu/tree.c:2838,1728).
> I guess those are freed before the use, but I still haven't figured
> out the reason why.
> I'm looking forward to some help.

You lost me on this one.  In both cases, rdp references a per-CPU
variable that is implicitly initialized to all zeroes, due to being
(sort of) a C-language global.

If a callback is queued early, then the following lines in __call_rcu()
will make an honest list of that field because of the :

		if (rcu_segcblist_empty(&rdp->cblist))
			rcu_segcblist_init(&rdp->cblist);

Otherwise, when rcu_init() is invoked during early boot, we have this
in rcu_init_percpu_data(), which is called from rcutree_prepare_cpu()
which is called from rcu_init(), which is called from start_kernel():

	if (rcu_segcblist_empty(&rdp->cblist) && /* No early-boot CBs? */
	    !init_nocb_callback_list(rdp))
		rcu_segcblist_init(&rdp->cblist);  /* Re-enable callbacks. */

So either init_nocb_callback_list() initializes the alternative callback
lists for a no-CBs CPU or rcu_segcblist_init() again makes an honest
list of that field.

My guess is that your tool is missing the

	rdp = this_cpu_ptr(rsp->rda);

in the __call_rcu() case, and also missing the

	struct rcu_data *rdp = per_cpu_ptr(rsp->rda, cpu);

Note that the ->rda field is explicitly compile-time initialized to
the base address of the per-CPU variable, which is rcu_preempt_data,
rcu_bh_data, or rcu_sched_data, depending on which RCU flavor is at hand.
(In v4.20-rc1, these are all merged into a single flavor to rule them all.)

Alternatively, your tool might be missing the implicit initialization
of per-CPU variables.

Or maybe I am missing something.  If so, please let me know what it is.

							Thanx, Paul

> Crash log 1
> =========================================
> BUG: KMSAN: uninit-value in __rcu_process_callbacks
> kernel/rcu/tree.c:2838 [inline]
> BUG: KMSAN: uninit-value in rcu_process_callbacks+0x5ac/0x1cb0
> kernel/rcu/tree.c:2864
> CPU: 0 PID: 20 Comm: kauditd Not tainted 4.19.0-rc8+ #18
> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
> Call Trace:
>  <IRQ>
>  __dump_stack lib/dump_stack.c:77 [inline]
>  dump_stack+0x305/0x460 lib/dump_stack.c:113
>  kmsan_report+0x1a2/0x2e0 mm/kmsan/kmsan.c:917
>  __msan_warning+0x7d/0xe0 mm/kmsan/kmsan_instr.c:500
>  __rcu_process_callbacks kernel/rcu/tree.c:2838 [inline]
>  rcu_process_callbacks+0x5ac/0x1cb0 kernel/rcu/tree.c:2864
>  __do_softirq+0x5ff/0xa55 kernel/softirq.c:292
>  invoke_softirq kernel/softirq.c:373 [inline]
>  irq_exit+0x22d/0x270 kernel/softirq.c:414
>  exiting_irq+0xe/0x10 arch/x86/include/asm/apic.h:536
>  smp_apic_timer_interrupt+0x64/0x90 arch/x86/kernel/apic/apic.c:1059
>  apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:869
>  </IRQ>
> RIP: 0010:finish_lock_switch+0x2b/0x40 kernel/sched/core.c:2578
> Code: 48 89 e5 53 48 89 fb e8 e3 43 9a 00 8b b8 88 0c 00 00 48 8b 00
> 48 85 c0 75 12 48 89 df e8 7d 38 9a 00 c6 00 00 c6 03 00 fb 5b <5d> c3
> e8 de 42 9a 00 eb e7 66 66 66 2e 0f 1f 84 00 00 00 00 00 55
> RSP: 0018:ffff88010622fca0 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
> RAX: ffff8801105bcc40 RBX: ffff8801061554c0 RCX: ffff8801105bdc40
> RDX: ffff8801105bdc40 RSI: aaaaaaaaaaaab000 RDI: ffffea00077ec560
> RBP: ffff88010622fca0 R08: ffffffff7fffffff R09: 0000000000000002
> R10: 0000000000000000 R11: 0000000000000000 R12: ffff8800751cb880
> R13: 0000000000000000 R14: ffff880106155db8 R15: ffff88013fcb9c40
>  finish_task_switch+0xe3/0x270 kernel/sched/core.c:2679
>  context_switch kernel/sched/core.c:2832 [inline]
>  __schedule+0x78f/0x8f0 kernel/sched/core.c:3479
>  schedule+0x1cc/0x300 kernel/sched/core.c:3523
>  kauditd_thread+0xc64/0xee0 kernel/audit.c:889
>  kthread+0x5b1/0x5f0 kernel/kthread.c:247
>  ret_from_fork+0x35/0x40 arch/x86/entry/entry_64.S:416
> 
> Uninit was created at:
>  kmsan_save_stack_with_flags mm/kmsan/kmsan.c:255 [inline]
>  kmsan_internal_alloc_meta_for_pages+0x157/0x730 mm/kmsan/kmsan.c:693
>  kmsan_alloc_page+0x80/0xe0 mm/kmsan/kmsan_hooks.c:320
>  __alloc_pages_nodemask+0x128c/0x69b0 mm/page_alloc.c:4416
>  alloc_pages_current+0x51f/0x760 mm/mempolicy.c:2093
>  alloc_pages include/linux/gfp.h:511 [inline]
>  alloc_slab_page mm/slub.c:1459 [inline]
>  allocate_slab mm/slub.c:1604 [inline]
>  new_slab+0x552/0x1f30 mm/slub.c:1675
>  new_slab_objects mm/slub.c:2438 [inline]
>  ___slab_alloc+0x1414/0x1dd0 mm/slub.c:2590
>  __slab_alloc mm/slub.c:2630 [inline]
>  slab_alloc_node mm/slub.c:2693 [inline]
>  slab_alloc mm/slub.c:2735 [inline]
>  kmem_cache_alloc+0xc9b/0xda0 mm/slub.c:2740
>  kmem_cache_zalloc include/linux/slab.h:697 [inline]
>  avc_alloc_node+0x109/0xb90 security/selinux/avc.c:572
>  avc_update_node+0x172/0x1ee0 security/selinux/avc.c:859
>  avc_denied+0x312/0x360 security/selinux/avc.c:1024
>  avc_has_perm_noaudit+0x733/0x770 security/selinux/avc.c:1155
>  avc_has_perm+0x172/0x480 security/selinux/avc.c:1184
>  sock_has_perm security/selinux/hooks.c:4539 [inline]
>  selinux_socket_sendmsg+0x297/0x360 security/selinux/hooks.c:4875
>  security_socket_sendmsg+0x127/0x200 security/security.c:1410
>  sock_sendmsg net/socket.c:628 [inline]
>  ___sys_sendmsg+0xd5f/0x1290 net/socket.c:2116
>  __sys_sendmsg net/socket.c:2154 [inline]
>  __do_sys_sendmsg net/socket.c:2163 [inline]
>  __se_sys_sendmsg+0x307/0x460 net/socket.c:2161
>  __x64_sys_sendmsg+0x4a/0x70 net/socket.c:2161
>  do_syscall_64+0xb8/0x100 arch/x86/entry/common.c:291
>  entry_SYSCALL_64_after_hwframe+0x63/0xe7
> =========================================================
> 
> Crash log 2
> =========================================================
> BUG: KMSAN: uninit-value in rcu_accelerate_cbs+0x821/0x990
> kernel/rcu/tree.c:1728
> CPU: 0 PID: 10 Comm: rcu_sched Not tainted 4.19.0-rc8+ #18
> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
> Call Trace:
>  __dump_stack lib/dump_stack.c:77 [inline]
>  dump_stack+0x305/0x460 lib/dump_stack.c:113
>  kmsan_report+0x1a2/0x2e0 mm/kmsan/kmsan.c:917
>  __msan_warning+0x7d/0xe0 mm/kmsan/kmsan_instr.c:500
>  rcu_accelerate_cbs+0x821/0x990 kernel/rcu/tree.c:1728
>  __note_gp_changes+0x2ac/0x9e0 kernel/rcu/tree.c:1807
>  rcu_gp_cleanup kernel/rcu/tree.c:2109 [inline]
>  rcu_gp_kthread+0x3019/0x3990 kernel/rcu/tree.c:2236
>  kthread+0x5b1/0x5f0 kernel/kthread.c:247
>  ret_from_fork+0x35/0x40 arch/x86/entry/entry_64.S:416
> 
> Uninit was created at:
>  kmsan_save_stack_with_flags mm/kmsan/kmsan.c:255 [inline]
>  kmsan_internal_alloc_meta_for_pages+0x157/0x730 mm/kmsan/kmsan.c:693
>  kmsan_alloc_page+0x80/0xe0 mm/kmsan/kmsan_hooks.c:320
>  __alloc_pages_nodemask+0x128c/0x69b0 mm/page_alloc.c:4416
>  alloc_pages_current+0x51f/0x760 mm/mempolicy.c:2093
>  alloc_pages include/linux/gfp.h:511 [inline]
>  alloc_slab_page mm/slub.c:1459 [inline]
>  allocate_slab mm/slub.c:1604 [inline]
>  new_slab+0x552/0x1f30 mm/slub.c:1675
>  new_slab_objects mm/slub.c:2438 [inline]
>  ___slab_alloc+0x1414/0x1dd0 mm/slub.c:2590
>  __slab_alloc mm/slub.c:2630 [inline]
>  slab_alloc_node mm/slub.c:2693 [inline]
>  slab_alloc mm/slub.c:2735 [inline]
>  kmem_cache_alloc+0xc9b/0xda0 mm/slub.c:2740
>  kmem_cache_zalloc include/linux/slab.h:697 [inline]
>  avc_alloc_node+0x109/0xb90 security/selinux/avc.c:572
>  avc_insert security/selinux/avc.c:696 [inline]
>  avc_compute_av+0x31e/0x1050 security/selinux/avc.c:1008
>  avc_has_perm_noaudit+0x516/0x770 security/selinux/avc.c:1149
>  avc_has_perm+0x172/0x480 security/selinux/avc.c:1184
>  selinux_socket_create+0x248/0x3c0 security/selinux/hooks.c:4560
>  security_socket_create+0x146/0x210 security/security.c:1372
>  __sock_create+0x26b/0xf30 net/socket.c:1232
>  sock_create net/socket.c:1317 [inline]
>  __sys_socket+0x180/0x670 net/socket.c:1347
>  __do_sys_socket net/socket.c:1356 [inline]
>  __se_sys_socket+0x8d/0xb0 net/socket.c:1354
>  __x64_sys_socket+0x4a/0x70 net/socket.c:1354
>  do_syscall_64+0xb8/0x100 arch/x86/entry/common.c:291
>  entry_SYSCALL_64_after_hwframe+0x63/0xe7
> =========================================================
> 
> Thanks,
> Kyungtae Kim
> 

Powered by blists - more mailing lists