netdev - Re: net/ipv6: use-after-free in __call_rcu/in6_dev_finish_destroy

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <CAAeHK+yzuh7mAmLR9DgHsLYBykqD8YhFcv=Nx6Az_f+PV_YMzw@mail.gmail.com>
Date:   Wed, 26 Apr 2017 17:15:12 +0200
From:   Andrey Konovalov <andreyknvl@...gle.com>
To:     "Paul E. McKenney" <paulmck@...ux.vnet.ibm.com>,
        David Ahern <dsa@...ulusnetworks.com>
Cc:     "David S. Miller" <davem@...emloft.net>,
        Alexey Kuznetsov <kuznet@....inr.ac.ru>,
        James Morris <jmorris@...ei.org>,
        Hideaki YOSHIFUJI <yoshfuji@...ux-ipv6.org>,
        Patrick McHardy <kaber@...sh.net>,
        netdev <netdev@...r.kernel.org>,
        LKML <linux-kernel@...r.kernel.org>,
        Josh Triplett <josh@...htriplett.org>,
        Steven Rostedt <rostedt@...dmis.org>,
        Mathieu Desnoyers <mathieu.desnoyers@...icios.com>,
        Lai Jiangshan <jiangshanlai@...il.com>,
        Eric Dumazet <edumazet@...gle.com>,
        Cong Wang <xiyou.wangcong@...il.com>,
        Dmitry Vyukov <dvyukov@...gle.com>,
        Kostya Serebryany <kcc@...gle.com>,
        syzkaller <syzkaller@...glegroups.com>
Subject: Re: net/ipv6: use-after-free in __call_rcu/in6_dev_finish_destroy_rcu

On Wed, Apr 26, 2017 at 3:59 PM, Paul E. McKenney
<paulmck@...ux.vnet.ibm.com> wrote:
> On Wed, Apr 26, 2017 at 02:34:15PM +0200, Andrey Konovalov wrote:
>> Hi,
>>
>> I've got the following error report while fuzzing the kernel with syzkaller.
>>
>> On commit 5a7ad1146caa895ad718a534399e38bd2ba721b7 (4.11-rc8).
>>
>> Unfortunately it's not reproducible.
>>
>> I'm not sure whether is is an issue with rcu or ipv6.
>>
>> ==================================================================
>> BUG: KASAN: use-after-free in __call_rcu.constprop.77+0x13be/0x1640
>
> Does building with CONFIG_DEBUG_OBJECTS_RCU_HEAD=y show any splats?
> (Yeah, kind of a stupid question if it is not reproducible, but had
> to ask!)

+David

I've enabled CONFIG_DEBUG_OBJECTS_RCU_HEAD and this is what I get.

Apparently the rcu warning is related to the fib6_del_route bug I've
been trying to reproduce:
https://groups.google.com/forum/#!msg/syzkaller/3SS80JbVPKA/2tfIAcW7DwAJ

Adding David, who provided the fix:
https://patchwork.ozlabs.org/patch/754913/

I've managed to extract a reproducer, attached together with the
.config that I used.

On commit 5a7ad1146caa895ad718a534399e38bd2ba721b7 (4.11-rc8) with
David's patch applied.

------------[ cut here ]------------
WARNING: CPU: 1 PID: 5911 at lib/debugobjects.c:289
debug_print_object+0x175/0x210
ODEBUG: activate active (active state 1) object type: rcu_head hint:
        (null)
Modules linked in:
CPU: 1 PID: 5911 Comm: a.out Not tainted 4.11.0-rc8+ #271
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
Call Trace:
 __dump_stack lib/dump_stack.c:16
 dump_stack+0x192/0x22d lib/dump_stack.c:52
 __warn+0x19f/0x1e0 kernel/panic.c:549
 warn_slowpath_fmt+0xe0/0x120 kernel/panic.c:564
 debug_print_object+0x175/0x210 lib/debugobjects.c:286
 debug_object_activate+0x574/0x7e0 lib/debugobjects.c:442
 debug_rcu_head_queue kernel/rcu/rcu.h:75
 __call_rcu.constprop.76+0xff/0x9c0 kernel/rcu/tree.c:3229
 call_rcu_sched+0x12/0x20 kernel/rcu/tree.c:3288
 rt6_rcu_free net/ipv6/ip6_fib.c:158
 rt6_release+0x1ea/0x290 net/ipv6/ip6_fib.c:188
 fib6_del_route net/ipv6/ip6_fib.c:1461
 fib6_del+0xa42/0xdc0 net/ipv6/ip6_fib.c:1500
 __ip6_del_rt+0x100/0x160 net/ipv6/route.c:2174
 ip6_del_rt+0x140/0x1b0 net/ipv6/route.c:2187
 __ipv6_ifa_notify+0x269/0x780 net/ipv6/addrconf.c:5520
 addrconf_ifdown+0xe60/0x1a20 net/ipv6/addrconf.c:3672
 addrconf_notify+0x1be/0x2590 net/ipv6/addrconf.c:3477
 notifier_call_chain+0x145/0x2f0 kernel/notifier.c:93
 __raw_notifier_call_chain kernel/notifier.c:394
 raw_notifier_call_chain+0x2d/0x40 kernel/notifier.c:401
 call_netdevice_notifiers_info+0x51/0x90 net/core/dev.c:1647
 call_netdevice_notifiers net/core/dev.c:1663
 __dev_notify_flags+0x1fd/0x320 net/core/dev.c:6499
 dev_change_flags+0xf5/0x140 net/core/dev.c:6530
 dev_ifsioc+0x619/0x9e0 net/core/dev_ioctl.c:254
 dev_ioctl+0x238/0xfe0 net/core/dev_ioctl.c:532
 sock_do_ioctl+0x94/0xb0 net/socket.c:913
 sock_ioctl+0x27a/0x410 net/socket.c:1004
 vfs_ioctl fs/ioctl.c:45
 do_vfs_ioctl+0x1cd/0x15a0 fs/ioctl.c:685
 SYSC_ioctl fs/ioctl.c:700
 SyS_ioctl+0x8f/0xc0 fs/ioctl.c:691
 entry_SYSCALL_64_fastpath+0x1a/0xa9 arch/x86/entry/entry_64.S:204
RIP: 0033:0x7fd579b89b79
RSP: 002b:00007fd207bcae58 EFLAGS: 00000206 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 00007fd207bcb700 RCX: 00007fd579b89b79
RDX: 0000000020000000 RSI: 0000000000008914 RDI: 000000000000017a
RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000206 R12: 00007fd57a055220
R13: 00007fd207bcb9c0 R14: 00007fd57a480040 R15: 0000000000000003
---[ end trace 8997f4cb685b1a22 ]---
------------[ cut here ]------------
WARNING: CPU: 1 PID: 5911 at lib/debugobjects.c:289
debug_print_object+0x175/0x210
ODEBUG: active_state active (active state 1) object type: rcu_head
hint:           (null)
Modules linked in:
CPU: 1 PID: 5911 Comm: a.out Tainted: G        W       4.11.0-rc8+ #271
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
Call Trace:
 __dump_stack lib/dump_stack.c:16
 dump_stack+0x192/0x22d lib/dump_stack.c:52
 __warn+0x19f/0x1e0 kernel/panic.c:549
 warn_slowpath_fmt+0xe0/0x120 kernel/panic.c:564
 debug_print_object+0x175/0x210 lib/debugobjects.c:286
 debug_object_active_state+0x46e/0x630 lib/debugobjects.c:696
 debug_rcu_head_queue kernel/rcu/rcu.h:76
 __call_rcu.constprop.76+0x11b/0x9c0 kernel/rcu/tree.c:3229
 call_rcu_sched+0x12/0x20 kernel/rcu/tree.c:3288
 rt6_rcu_free net/ipv6/ip6_fib.c:158
 rt6_release+0x1ea/0x290 net/ipv6/ip6_fib.c:188
 fib6_del_route net/ipv6/ip6_fib.c:1461
 fib6_del+0xa42/0xdc0 net/ipv6/ip6_fib.c:1500
 __ip6_del_rt+0x100/0x160 net/ipv6/route.c:2174
 ip6_del_rt+0x140/0x1b0 net/ipv6/route.c:2187
 __ipv6_ifa_notify+0x269/0x780 net/ipv6/addrconf.c:5520
 addrconf_ifdown+0xe60/0x1a20 net/ipv6/addrconf.c:3672
 addrconf_notify+0x1be/0x2590 net/ipv6/addrconf.c:3477
 notifier_call_chain+0x145/0x2f0 kernel/notifier.c:93
 __raw_notifier_call_chain kernel/notifier.c:394
 raw_notifier_call_chain+0x2d/0x40 kernel/notifier.c:401
 call_netdevice_notifiers_info+0x51/0x90 net/core/dev.c:1647
 call_netdevice_notifiers net/core/dev.c:1663
 __dev_notify_flags+0x1fd/0x320 net/core/dev.c:6499
 dev_change_flags+0xf5/0x140 net/core/dev.c:6530
 dev_ifsioc+0x619/0x9e0 net/core/dev_ioctl.c:254
 dev_ioctl+0x238/0xfe0 net/core/dev_ioctl.c:532
 sock_do_ioctl+0x94/0xb0 net/socket.c:913
 sock_ioctl+0x27a/0x410 net/socket.c:1004
 vfs_ioctl fs/ioctl.c:45
 do_vfs_ioctl+0x1cd/0x15a0 fs/ioctl.c:685
 SYSC_ioctl fs/ioctl.c:700
 SyS_ioctl+0x8f/0xc0 fs/ioctl.c:691
 entry_SYSCALL_64_fastpath+0x1a/0xa9 arch/x86/entry/entry_64.S:204
RIP: 0033:0x7fd579b89b79
RSP: 002b:00007fd207bcae58 EFLAGS: 00000206 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 00007fd207bcb700 RCX: 00007fd579b89b79
RDX: 0000000020000000 RSI: 0000000000008914 RDI: 000000000000017a
RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000206 R12: 00007fd57a055220
R13: 00007fd207bcb9c0 R14: 00007fd57a480040 R15: 0000000000000003
---[ end trace 8997f4cb685b1a23 ]---
------------[ cut here ]------------
WARNING: CPU: 1 PID: 5911 at kernel/rcu/tree.c:3232
__call_rcu.constprop.76+0x446/0x9c0
__call_rcu(): Leaked duplicate callback
Modules linked in:
CPU: 1 PID: 5911 Comm: a.out Tainted: G        W       4.11.0-rc8+ #271
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
Call Trace:
 __dump_stack lib/dump_stack.c:16
 dump_stack+0x192/0x22d lib/dump_stack.c:52
 __warn+0x19f/0x1e0 kernel/panic.c:549
 warn_slowpath_fmt+0xe0/0x120 kernel/panic.c:564
 __call_rcu.constprop.76+0x446/0x9c0 kernel/rcu/tree.c:3232
 call_rcu_sched+0x12/0x20 kernel/rcu/tree.c:3288
 rt6_rcu_free net/ipv6/ip6_fib.c:158
 rt6_release+0x1ea/0x290 net/ipv6/ip6_fib.c:188
 fib6_del_route net/ipv6/ip6_fib.c:1461
 fib6_del+0xa42/0xdc0 net/ipv6/ip6_fib.c:1500
 __ip6_del_rt+0x100/0x160 net/ipv6/route.c:2174
 ip6_del_rt+0x140/0x1b0 net/ipv6/route.c:2187
 __ipv6_ifa_notify+0x269/0x780 net/ipv6/addrconf.c:5520
 addrconf_ifdown+0xe60/0x1a20 net/ipv6/addrconf.c:3672
 addrconf_notify+0x1be/0x2590 net/ipv6/addrconf.c:3477
 notifier_call_chain+0x145/0x2f0 kernel/notifier.c:93
 __raw_notifier_call_chain kernel/notifier.c:394
 raw_notifier_call_chain+0x2d/0x40 kernel/notifier.c:401
 call_netdevice_notifiers_info+0x51/0x90 net/core/dev.c:1647
 call_netdevice_notifiers net/core/dev.c:1663
 __dev_notify_flags+0x1fd/0x320 net/core/dev.c:6499
 dev_change_flags+0xf5/0x140 net/core/dev.c:6530
 dev_ifsioc+0x619/0x9e0 net/core/dev_ioctl.c:254
 dev_ioctl+0x238/0xfe0 net/core/dev_ioctl.c:532
 sock_do_ioctl+0x94/0xb0 net/socket.c:913
 sock_ioctl+0x27a/0x410 net/socket.c:1004
 vfs_ioctl fs/ioctl.c:45
 do_vfs_ioctl+0x1cd/0x15a0 fs/ioctl.c:685
 SYSC_ioctl fs/ioctl.c:700
 SyS_ioctl+0x8f/0xc0 fs/ioctl.c:691
 entry_SYSCALL_64_fastpath+0x1a/0xa9 arch/x86/entry/entry_64.S:204
RIP: 0033:0x7fd579b89b79
RSP: 002b:00007fd207bcae58 EFLAGS: 00000206 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 00007fd207bcb700 RCX: 00007fd579b89b79
RDX: 0000000020000000 RSI: 0000000000008914 RDI: 000000000000017a
RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000206 R12: 00007fd57a055220
R13: 00007fd207bcb9c0 R14: 00007fd57a480040 R15: 0000000000000003
---[ end trace 8997f4cb685b1a24 ]---


>
>                                                         Thanx, Paul
>
>> kernel/rcu/tree.c:3269 at addr ffff88003b842280
>> Write of size 8 by task kworker/u10:1/180
>> CPU: 2 PID: 180 Comm: kworker/u10:1 Not tainted 4.11.0-rc8+ #270
>> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
>> Workqueue: events_unbound call_usermodehelper_exec_work
>> Call Trace:
>>  __dump_stack lib/dump_stack.c:16 [inline]
>>  dump_stack+0x192/0x22d lib/dump_stack.c:52
>>  kasan_object_err+0x1c/0x70 mm/kasan/report.c:164
>>  print_address_description mm/kasan/report.c:202 [inline]
>>  kasan_report_error mm/kasan/report.c:291 [inline]
>>  kasan_report+0x252/0x510 mm/kasan/report.c:347
>>  __asan_report_store8_noabort+0x17/0x20 mm/kasan/report.c:373
>>  __call_rcu.constprop.77+0x13be/0x1640 kernel/rcu/tree.c:3269
>>  call_rcu_sched+0x12/0x20 kernel/rcu/tree.c:3288
>>  free_pid+0x446/0x5d0 kernel/pid.c:293
>>  __change_pid+0x2a1/0x3d0 kernel/pid.c:411
>>  detach_pid+0x1f/0x30 kernel/pid.c:416
>>  __unhash_process kernel/exit.c:74 [inline]
>>  __exit_signal kernel/exit.c:155 [inline]
>>  release_task+0xbb0/0x1d90 kernel/exit.c:199
>>  wait_task_zombie kernel/exit.c:1230 [inline]
>>  wait_consider_task+0x11fe/0x3410 kernel/exit.c:1458
>>  do_wait_thread kernel/exit.c:1521 [inline]
>>  do_wait+0x3ea/0x8e0 kernel/exit.c:1592
>>  SYSC_wait4 kernel/exit.c:1720 [inline]
>>  SyS_wait4+0x208/0x340 kernel/exit.c:1689
>>  call_usermodehelper_exec_sync kernel/kmod.c:292 [inline]
>>  call_usermodehelper_exec_work+0x1a7/0x2b0 kernel/kmod.c:329
>>  process_one_work+0x9f7/0x1580 kernel/workqueue.c:2097
>>  worker_thread+0x1df/0x14b0 kernel/workqueue.c:2231
>>  kthread+0x31f/0x3f0 kernel/kthread.c:231
>>  ret_from_fork+0x2c/0x40 arch/x86/entry/entry_64.S:430
>> Object at ffff88003b842018, in cache kmalloc-1024 size: 1024
>> Allocated:
>> PID = 1
>>  save_stack_trace+0x16/0x20 arch/x86/kernel/stacktrace.c:59
>>  save_stack+0x43/0xd0 mm/kasan/kasan.c:513
>>  set_track mm/kasan/kasan.c:525 [inline]
>>  kasan_kmalloc+0xad/0xe0 mm/kasan/kasan.c:616
>>  kmem_cache_alloc_trace+0x61/0x170 mm/slub.c:2745
>>  kmalloc include/linux/slab.h:490 [inline]
>>  kzalloc include/linux/slab.h:663 [inline]
>>  ipv6_add_dev+0x199/0x1380 net/ipv6/addrconf.c:380
>>  addrconf_init+0xd0/0x29a net/ipv6/addrconf.c:6405
>>  inet6_init+0x2f6/0x584 net/ipv6/af_inet6.c:962
>>  do_one_initcall+0xf3/0x380 init/main.c:792
>>  do_initcall_level init/main.c:858 [inline]
>>  do_initcalls init/main.c:866 [inline]
>>  do_basic_setup init/main.c:884 [inline]
>>  kernel_init_freeable+0x54d/0x622 init/main.c:1035
>>  kernel_init+0x13/0x180 init/main.c:959
>>  ret_from_fork+0x2c/0x40 arch/x86/entry/entry_64.S:430
>> Freed:
>> PID = 6479
>>  save_stack_trace+0x16/0x20 arch/x86/kernel/stacktrace.c:59
>>  save_stack+0x43/0xd0 mm/kasan/kasan.c:513
>>  set_track mm/kasan/kasan.c:525 [inline]
>>  kasan_slab_free+0x73/0xc0 mm/kasan/kasan.c:589
>>  slab_free_hook mm/slub.c:1357 [inline]
>>  slab_free_freelist_hook mm/slub.c:1379 [inline]
>>  slab_free mm/slub.c:2961 [inline]
>>  kfree+0x91/0x190 mm/slub.c:3882
>>  in6_dev_finish_destroy_rcu+0x97/0xc0 net/ipv6/addrconf_core.c:150
>>  __rcu_reclaim kernel/rcu/rcu.h:118 [inline]
>>  rcu_do_batch.isra.65+0x6de/0xbd0 kernel/rcu/tree.c:2879
>>  invoke_rcu_callbacks kernel/rcu/tree.c:3142 [inline]
>>  __rcu_process_callbacks kernel/rcu/tree.c:3109 [inline]
>>  rcu_process_callbacks+0x23f/0x810 kernel/rcu/tree.c:3126
>>  __do_softirq+0x253/0x78b kernel/softirq.c:284
>> Memory state around the buggy address:
>>  ffff88003b842180: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
>>  ffff88003b842200: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
>> >ffff88003b842280: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
>>                    ^
>>  ffff88003b842300: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
>>  ffff88003b842380: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
>> ==================================================================
>>
>

View attachment "rt6_rcu_free-warn-poc.c" of type "text/x-csrc" (11015 bytes)

Download attachment ".config" of type "application/octet-stream" (127560 bytes)