Date:   Tue, 28 Feb 2017 23:43:26 +0800
From:   Xin Long <lucien.xin@...il.com>
To:     Dmitry Vyukov <dvyukov@...gle.com>
Cc:     Andrey Konovalov <andreyknvl@...gle.com>,
        Vlad Yasevich <vyasevich@...il.com>,
        Neil Horman <nhorman@...driver.com>,
        "David S. Miller" <davem@...emloft.net>,
        linux-sctp@...r.kernel.org, netdev <netdev@...r.kernel.org>,
        LKML <linux-kernel@...r.kernel.org>,
        Kostya Serebryany <kcc@...gle.com>,
        Eric Dumazet <edumazet@...gle.com>,
        syzkaller <syzkaller@...glegroups.com>
Subject: Re: net/sctp: use-after-free in sctp_hash_transport

On Tue, Feb 28, 2017 at 11:35 PM, Dmitry Vyukov <dvyukov@...gle.com> wrote:
> On Mon, Feb 27, 2017 at 5:27 PM, Xin Long <lucien.xin@...il.com> wrote:
>> On Mon, Feb 27, 2017 at 11:45 PM, Andrey Konovalov
>> <andreyknvl@...gle.com> wrote:
>>> Hi,
>>>
>>> I've got the following error report while fuzzing the kernel with syzkaller.
>>>
>>> On commit e5d56efc97f8240d0b5d66c03949382b6d7e5570 (Feb 26).
>>>
>>> A reproducer and .config are attached.
>>>
>>> ===============================
>>> [ ERR: suspicious RCU usage.  ]
>>> 4.10.0+ #54 Not tainted
>>> -------------------------------
>>> ./include/linux/rhashtable.h:602 suspicious rcu_dereference_check() usage!
>>>
>>> other info that might help us debug this:
>>>
>>>
>>> rcu_scheduler_active = 2, debug_locks = 0
>>> 1 lock held by a.out/4189:
>>>  #0:  (sk_lock-AF_INET6){+.+.+.}, at: [<ffffffff84510c78>]
>>> sctp_setsockopt+0x318/0x5f10
>>>
>>> stack backtrace:
>>> CPU: 1 PID: 4189 Comm: a.out Not tainted 4.10.0+ #54
>>> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
>>> Call Trace:
>>>  __dump_stack lib/dump_stack.c:15
>>>  dump_stack+0x292/0x398 lib/dump_stack.c:51
>>>  lockdep_rcu_suspicious+0x139/0x180 kernel/locking/lockdep.c:4452
>>>  __rhashtable_lookup ./include/linux/rhashtable.h:602
>>>  rhltable_lookup ./include/linux/rhashtable.h:690
>>>  sctp_hash_transport+0x826/0xcc0 net/sctp/input.c:887
>>>  sctp_assoc_add_peer+0xd0b/0x1470 net/sctp/associola.c:716
>>>  __sctp_connect+0x26d/0xdb0 net/sctp/socket.c:1184
>>>  __sctp_setsockopt_connectx+0x197/0x200 net/sctp/socket.c:1338
>>>  sctp_setsockopt_connectx net/sctp/socket.c:1370
>>>  sctp_setsockopt+0x15fa/0x5f10 net/sctp/socket.c:3936
>>>  sock_common_setsockopt+0x95/0xd0 net/core/sock.c:2725
>>>  SYSC_setsockopt net/socket.c:1786
>>>  SyS_setsockopt+0x270/0x3a0 net/socket.c:1765
>>>  entry_SYSCALL_64_fastpath+0x1f/0xc2 arch/x86/entry/entry_64.S:204
>>> RIP: 0033:0x7f3e27a55b79
>>> RSP: 002b:00007f3e2296fd98 EFLAGS: 00000206 ORIG_RAX: 0000000000000036
>>> RAX: ffffffffffffffda RBX: 00007f3e229709c0 RCX: 00007f3e27a55b79
>>> RDX: 000000000000006e RSI: 0000000000000084 RDI: 0000000000000003
>>> RBP: 00007f3e27f21220 R08: 0000000000000010 R09: 0000000000000000
>>> R10: 0000000020004000 R11: 0000000000000206 R12: 0000000000000000
>>> R13: 00007f3e229709c0 R14: 00007f3e2834c040 R15: 0000000000000003
>>> ==================================================================
>>> BUG: KASAN: use-after-free in sctp_hash_transport+0x855/0xcc0 at addr
>>> ffff8800671e1f8c
>>> Read of size 4 by task a.out/4189
>>> CPU: 1 PID: 4189 Comm: a.out Not tainted 4.10.0+ #54
>>> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
>>> Call Trace:
>>>  __dump_stack lib/dump_stack.c:15
>>>  dump_stack+0x292/0x398 lib/dump_stack.c:51
>>>  kasan_object_err+0x1c/0x70 mm/kasan/report.c:162
>>>  print_address_description mm/kasan/report.c:200
>>>  kasan_report_error mm/kasan/report.c:289
>>>  kasan_report.part.1+0x20e/0x4e0 mm/kasan/report.c:311
>>>  kasan_report mm/kasan/report.c:331
>>>  __asan_report_load4_noabort+0x29/0x30 mm/kasan/report.c:331
>>>  rht_key_hashfn ./include/linux/rhashtable.h:254
>>>  __rhashtable_lookup ./include/linux/rhashtable.h:604
>>>  rhltable_lookup ./include/linux/rhashtable.h:690
>>>  sctp_hash_transport+0x855/0xcc0 net/sctp/input.c:887
>>>  sctp_assoc_add_peer+0xd0b/0x1470 net/sctp/associola.c:716
>>>  __sctp_connect+0x26d/0xdb0 net/sctp/socket.c:1184
>>>  __sctp_setsockopt_connectx+0x197/0x200 net/sctp/socket.c:1338
>>>  sctp_setsockopt_connectx net/sctp/socket.c:1370
>>>  sctp_setsockopt+0x15fa/0x5f10 net/sctp/socket.c:3936
>>>  sock_common_setsockopt+0x95/0xd0 net/core/sock.c:2725
>>>  SYSC_setsockopt net/socket.c:1786
>>>  SyS_setsockopt+0x270/0x3a0 net/socket.c:1765
>>>  entry_SYSCALL_64_fastpath+0x1f/0xc2 arch/x86/entry/entry_64.S:204
>>> RIP: 0033:0x7f3e27a55b79
>>> RSP: 002b:00007f3e2296fd98 EFLAGS: 00000206 ORIG_RAX: 0000000000000036
>>> RAX: ffffffffffffffda RBX: 00007f3e229709c0 RCX: 00007f3e27a55b79
>>> RDX: 000000000000006e RSI: 0000000000000084 RDI: 0000000000000003
>>> RBP: 00007f3e27f21220 R08: 0000000000000010 R09: 0000000000000000
>>> R10: 0000000020004000 R11: 0000000000000206 R12: 0000000000000000
>>> R13: 00007f3e229709c0 R14: 00007f3e2834c040 R15: 0000000000000003
>>> Object at ffff8800671e1f80, in cache kmalloc-1024 size: 1024
>>> Allocated:
>>> PID = 1
>>> save_stack_trace+0x16/0x20 arch/x86/kernel/stacktrace.c:57
>>>  save_stack+0x43/0xd0 mm/kasan/kasan.c:502
>>>  set_track mm/kasan/kasan.c:514
>>>  kasan_kmalloc+0xad/0xe0 mm/kasan/kasan.c:605
>>>  __kmalloc+0xa0/0x2d0 mm/slub.c:3745
>>>  kmalloc ./include/linux/slab.h:495
>>>  kzalloc ./include/linux/slab.h:663
>>>  bucket_table_alloc+0x618/0x930 lib/rhashtable.c:224
>>>  rhashtable_init+0x5f8/0xc60 lib/rhashtable.c:1006
>>>  rhltable_init+0x53/0xa0 lib/rhashtable.c:1037
>>>  sctp_transport_hashtable_init+0x1c/0x20 net/sctp/input.c:865
>>>  sctp_init+0x62c/0x88f net/sctp/protocol.c:1486
>>>  do_one_initcall+0xf3/0x390 init/main.c:788
>>>  do_initcall_level init/main.c:854
>>>  do_initcalls init/main.c:862
>>>  do_basic_setup init/main.c:880
>>>  kernel_init_freeable+0x5cc/0x6a6 init/main.c:1031
>>>  kernel_init+0x13/0x180 init/main.c:955
>>>  ret_from_fork+0x31/0x40 arch/x86/entry/entry_64.S:430
>>> Freed:
>>> PID = 0
>>>  save_stack_trace+0x16/0x20 arch/x86/kernel/stacktrace.c:57
>>>  save_stack+0x43/0xd0 mm/kasan/kasan.c:502
>>>  set_track mm/kasan/kasan.c:514
>>>  kasan_slab_free+0x73/0xc0 mm/kasan/kasan.c:578
>>>  slab_free_hook mm/slub.c:1357
>>>  slab_free_freelist_hook mm/slub.c:1379
>>>  slab_free mm/slub.c:2961
>>>  kfree+0xe8/0x2b0 mm/slub.c:3882
>>>  kvfree+0x36/0x60 mm/util.c:335
>>>  bucket_table_free+0xd7/0x260 lib/rhashtable.c:152
>>>  bucket_table_free_rcu+0x16/0x20 lib/rhashtable.c:157
>>>  __rcu_reclaim kernel/rcu/rcu.h:118
>>>  rcu_do_batch.isra.64+0x94c/0xcc0 kernel/rcu/tree.c:2877
>>>  invoke_rcu_callbacks kernel/rcu/tree.c:3140
>>>  __rcu_process_callbacks kernel/rcu/tree.c:3107
>>>  rcu_process_callbacks+0x2cc/0xb90 kernel/rcu/tree.c:3124
>>>  __do_softirq+0x2fb/0xb7d kernel/softirq.c:284
>>> Memory state around the buggy address:
>>>  ffff8800671e1e80: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
>>>  ffff8800671e1f00: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
>>>>ffff8800671e1f80: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
>>>                       ^
>>>  ffff8800671e2000: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
>>>  ffff8800671e2080: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
>>> ==================================================================
>>>
>>> ...
>>> ...
>>> ...
>>>
>>> kasan: CONFIG_KASAN_INLINE enabled
>>> kasan: GPF could be caused by NULL-ptr deref or user memory access
>>> general protection fault: 0000 [#1] SMP KASAN
>>> Modules linked in:
>>> CPU: 1 PID: 4189 Comm: a.out Tainted: G    B           4.10.0+ #54
>>> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
>>> task: ffff880069111600 task.stack: ffff880068638000
>>> RIP: 0010:rht_bucket_nested+0x18e/0x560 lib/rhashtable.c:1126
>>> RSP: 0018:ffff88006863eda8 EFLAGS: 00010246
>>> RAX: dffffc0000000000 RBX: 000000006be18a80 RCX: 1ffff1000d0c7dbf
>>> RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffffffff85a5ee80
>>> RBP: ffff88006863ee60 R08: 0000000000000001 R09: 0000000000000000
>>> R10: 0000000000000000 R11: dffffc0000000000 R12: 0000000000000001
>>> R13: ffff8800671e1f84 R14: ffff8800671e1f80 R15: dffffc0000000000
>>> FS:  00007f3e22970700(0000) GS:ffff88006cb00000(0000) knlGS:0000000000000000
>>> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>> CR2: 00007f3e2216ef38 CR3: 0000000065589000 CR4: 00000000000006e0
>>> Call Trace:
>>>  rht_bucket ./include/linux/rhashtable.h:404
>>>  __rhashtable_lookup ./include/linux/rhashtable.h:605
>>>  rhltable_lookup ./include/linux/rhashtable.h:690
>>>  sctp_hash_transport+0xaa9/0xcc0 net/sctp/input.c:887
>>>  sctp_assoc_add_peer+0xd0b/0x1470 net/sctp/associola.c:716
>>>  __sctp_connect+0x26d/0xdb0 net/sctp/socket.c:1184
>>>  __sctp_setsockopt_connectx+0x197/0x200 net/sctp/socket.c:1338
>>>  sctp_setsockopt_connectx net/sctp/socket.c:1370
>>>  sctp_setsockopt+0x15fa/0x5f10 net/sctp/socket.c:3936
>>>  sock_common_setsockopt+0x95/0xd0 net/core/sock.c:2725
>>>  SYSC_setsockopt net/socket.c:1786
>>>  SyS_setsockopt+0x270/0x3a0 net/socket.c:1765
>>>  entry_SYSCALL_64_fastpath+0x1f/0xc2 arch/x86/entry/entry_64.S:204
>>> RIP: 0033:0x7f3e27a55b79
>>> RSP: 002b:00007f3e2296fd98 EFLAGS: 00000206 ORIG_RAX: 0000000000000036
>>> RAX: ffffffffffffffda RBX: 00007f3e229709c0 RCX: 00007f3e27a55b79
>>> RDX: 000000000000006e RSI: 0000000000000084 RDI: 0000000000000003
>>> RBP: 00007f3e27f21220 R08: 0000000000000010 R09: 0000000000000000
>>> R10: 0000000020004000 R11: 0000000000000206 R12: 0000000000000000
>>> R13: 00007f3e229709c0 R14: 00007f3e2834c040 R15: 0000000000000003
>>> Code: 34 14 97 03 00 0f 84 a4 01 00 00 e8 9d e2 4b ff 8b 85 70 ff ff
>>> ff 4d 8d 24 c4 48 b8 00 00 00 00 00 fc ff df 4c 89 e2 48 c1 ea 03 <80>
>>> 3c 02 00 0f 85 b1 03 00 00 4c 89 ea 48 b8 00 00 00 00 00 fc
>>> RIP: rht_bucket_nested+0x18e/0x560 RSP: ffff88006863eda8
>>> ---[ end trace 9d36cf16fcdf072c ]--
>>
>> Just notice that rhltable_lookup doesn't call rcu_read_lock inside, unlike
>> rhltable_insert_key().
>>
>> will post a fix it soon.
>
>
> rhltable_lookup returns rcu-protected objects, so locking rcu inside
> of rhltable_lookup is pointless. And the comment says the same "This
> must only be called under the RCU read lock". The bug should be upper
> the stack, in sctp code.
Yep, the patch is here already:
https://patchwork.ozlabs.org/patch/733280/

Sorry for not saying it clearly.

Powered by blists - more mailing lists