netdev - Re: kernel panic: stack is corrupted in udp4_lib

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Date:   Mon, 7 Jan 2019 10:04:58 +0100
From:   Dmitry Vyukov <dvyukov@...gle.com>
To:     Stefano Brivio <sbrivio@...hat.com>
Cc:     Willem de Bruijn <willemdebruijn.kernel@...il.com>,
        Eric Dumazet <eric.dumazet@...il.com>,
        syzbot <syzbot+4ad25edc7a33e4ab91e0@...kaller.appspotmail.com>,
        David Miller <davem@...emloft.net>,
        Alexey Kuznetsov <kuznet@....inr.ac.ru>,
        linux-kernel <linux-kernel@...r.kernel.org>,
        Network Development <netdev@...r.kernel.org>,
        syzkaller-bugs <syzkaller-bugs@...glegroups.com>,
        Hideaki YOSHIFUJI <yoshfuji@...ux-ipv6.org>
Subject: Re: kernel panic: stack is corrupted in udp4_lib_lookup2

On Fri, Jan 4, 2019 at 7:05 PM Stefano Brivio <sbrivio@...hat.com> wrote:
>
> On Fri, 4 Jan 2019 18:26:16 +0100
> Dmitry Vyukov <dvyukov@...gle.com> wrote:
>
> > On Fri, Jan 4, 2019 at 6:14 PM Stefano Brivio <sbrivio@...hat.com> wrote:
> > >
> > > On Fri, 4 Jan 2019 12:05:04 +0100
> > > Dmitry Vyukov <dvyukov@...gle.com> wrote:
> > >
> > > > I've added these as tests:
> > > >
> > > > https://github.com/google/syzkaller/blob/master/pkg/report/testdata/linux/report/341
> > > > https://github.com/google/syzkaller/blob/master/pkg/report/testdata/linux/report/342
> > > > https://github.com/google/syzkaller/blob/master/pkg/report/testdata/linux/report/343
> > > > https://github.com/google/syzkaller/blob/master/pkg/report/testdata/linux/report/344
> > > >
> > > > Will try to figure out how to distinguish them from true corrupted
> > > > reports. Usually when Call Trace does not have any frames, it's a sign
> > > > of a corrupted report, and in other crashes we see the same report but
> > > > with a stack trace. But some stack-corruption-related reliably don't
> > > > have stack traces (not corrupted). But then some other
> > > > stack-corruption-related crashes do have stack traces, and for these
> > > > no stack trace again means a corrupted kernel output. Amusingly this
> > > > is one of the most complex parts of syzkaller.
> > >
> > > I'm not sure how complicated that would be, but what about some metric
> > > based on valid symbol names being reported?
> >
> > Please elaborate. What do you mean by "valid symbol names"?
>
> I mean a symbol name listed in /proc/kallsyms on the running system.
>
> This is usually my minimum threshold for "I can do something with this
> report" -- which doesn't mean it's necessarily valid, but well, if you
> have that, it means that at least something worked in the reporting,
> and you can at least start having a look at a specific function.
>
> > Note that corrupted output detection solves 2 problems:
> > 1. Do we think the output is truncated to the point of being not useful?
> > E.g. sometimes kernel produces just 1 line:
> >
> > general protection fault: 0000 [#1] PREEMPT SMP KASAN
> >
> > This is sure a crash, but it's not too useful to report.
>
> Sure. In those tests above you have:
> - 341: udp6_lib_lookup2+0x622, handle_irq+0x2cb
> - 342: __sanitizer_cov_trace_pc+0x8, handle_irq+0x2cb
> - 343: __udp6_lib_err, etc.
> - 344: __udp6_lib_lookup+0x1d, etc.
>
> and this makes all those reports at least minimally useful.
>
> > 2. Do we have any reasons to think we extracted bogus crash identity?
> > E.g. crash intermixed with output from another thread so that we say
> > "something-bad in function foo", when in fact function foo come from
> > output of the second non-crashing thread.
>
> Okay, this looks way more complicated :)

Yeah, unfortunately, it's quite complicated.
Just today this gen popped up. You won't find any ODEBUG checks at
that stack, it's completely unrelated and come from another task.

------------[ cut here ]------------
ODEBUG: free active (active state 0) object type: timer_list hint:
delayed_work_timer_fn+0x0/0x90 kernel/workqueue.c:4916
WARNING: CPU: 1 PID: 45 at lib/debugobjects.c:325
debug_print_object+0x16a/0x250 lib/debugobjects.c:325
CPU: 0 PID: 13619 Comm: syz-executor1 Not tainted 4.20.0+ #13
Kernel panic - not syncing: panic_on_warn set ...
Hardware name: Google Google Compute Engine/Google Compute Engine,
BIOS Google 01/01/2011
Call Trace:
 __dump_stack lib/dump_stack.c:77 [inline]
 dump_stack+0x1db/0x2d0 lib/dump_stack.c:113
 warn_alloc.cold+0xc2/0x1c8 mm/page_alloc.c:3570
 __vmalloc_node_range+0x57a/0x910 mm/vmalloc.c:1766
 __vmalloc_node mm/vmalloc.c:1795 [inline]
 __vmalloc_node_flags mm/vmalloc.c:1809 [inline]
 vmalloc+0x6b/0x90 mm/vmalloc.c:1831
 sel_write_load+0x1de/0x470 security/selinux/selinuxfs.c:557
 __vfs_write+0x116/0xb40 fs/read_write.c:485
 vfs_write+0x20c/0x580 fs/read_write.c:549
 ksys_write+0x105/0x260 fs/read_write.c:598
 __do_sys_write fs/read_write.c:610 [inline]
 __se_sys_write fs/read_write.c:607 [inline]
 __x64_sys_write+0x73/0xb0 fs/read_write.c:607
 do_syscall_64+0x1a3/0x800 arch/x86/entry/common.c:290
 entry_SYSCALL_64_after_hwframe+0x49/0xbe