linux-kernel - Re: WARNING in sk_stream_kill

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <X8kjPIrLJUd8uQIX@elver.google.com>
Date:   Thu, 3 Dec 2020 18:41:16 +0100
From:   Marco Elver <elver@...gle.com>
To:     Eric Dumazet <edumazet@...gle.com>
Cc:     netdev <netdev@...r.kernel.org>,
        Andrew Morton <akpm@...ux-foundation.org>,
        David Miller <davem@...emloft.net>,
        Dmitry Vyukov <dvyukov@...gle.com>,
        Alexander Potapenko <glider@...gle.com>,
        Jann Horn <jannh@...gle.com>, Jakub Kicinski <kuba@...nel.org>,
        LKML <linux-kernel@...r.kernel.org>,
        Stephen Rothwell <sfr@...b.auug.org.au>,
        syzkaller-bugs <syzkaller-bugs@...glegroups.com>,
        Willem de Bruijn <willemb@...gle.com>,
        syzbot <syzbot+7b99aafdcc2eedea6178@...kaller.appspotmail.com>
Subject: Re: WARNING in sk_stream_kill_queues (5)

On Thu, Dec 03, 2020 at 05:42PM +0100, Eric Dumazet wrote:
> On Thu, Dec 3, 2020 at 5:34 PM Marco Elver <elver@...gle.com> wrote:
> >
> > On Thu, 3 Dec 2020 at 17:27, Eric Dumazet <edumazet@...gle.com> wrote:
> > > On Thu, Dec 3, 2020 at 4:58 PM Marco Elver <elver@...gle.com> wrote:
> > > >
> > > > On Mon, Nov 30, 2020 at 12:40AM -0800, syzbot wrote:
> > > > > Hello,
> > > > >
> > > > > syzbot found the following issue on:
> > > > >
> > > > > HEAD commit:    6147c83f Add linux-next specific files for 20201126
> > > > > git tree:       linux-next
> > > > > console output: https://syzkaller.appspot.com/x/log.txt?x=117c9679500000
> > > > > kernel config:  https://syzkaller.appspot.com/x/.config?x=9b91566da897c24f
> > > > > dashboard link: https://syzkaller.appspot.com/bug?extid=7b99aafdcc2eedea6178
> > > > > compiler:       gcc (GCC) 10.1.0-syz 20200507
> > > > > syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=103bf743500000
> > > > > C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=167c60c9500000
> > > > >
> > > > > The issue was bisected to:
> > > > >
> > > > > commit 145cd60fb481328faafba76842aa0fd242e2b163
> > > > > Author: Alexander Potapenko <glider@...gle.com>
> > > > > Date:   Tue Nov 24 05:38:44 2020 +0000
> > > > >
> > > > >     mm, kfence: insert KFENCE hooks for SLUB
> > > > >
> > > > > bisection log:  https://syzkaller.appspot.com/x/bisect.txt?x=13abe5b3500000
> > > > > final oops:     https://syzkaller.appspot.com/x/report.txt?x=106be5b3500000
> > > > > console output: https://syzkaller.appspot.com/x/log.txt?x=17abe5b3500000
> > > > >
> > > > > IMPORTANT: if you fix the issue, please add the following tag to the commit:
> > > > > Reported-by: syzbot+7b99aafdcc2eedea6178@...kaller.appspotmail.com
> > > > > Fixes: 145cd60fb481 ("mm, kfence: insert KFENCE hooks for SLUB")
> > > > >
> > > > > ------------[ cut here ]------------
> > > > > WARNING: CPU: 0 PID: 11307 at net/core/stream.c:207 sk_stream_kill_queues+0x3c3/0x530 net/core/stream.c:207
> > > > [...]
> > > > > Call Trace:
> > > > >  inet_csk_destroy_sock+0x1a5/0x490 net/ipv4/inet_connection_sock.c:885
> > > > >  __tcp_close+0xd3e/0x1170 net/ipv4/tcp.c:2585
> > > > >  tcp_close+0x29/0xc0 net/ipv4/tcp.c:2597
> > > > >  inet_release+0x12e/0x280 net/ipv4/af_inet.c:431
> > > > >  __sock_release+0xcd/0x280 net/socket.c:596
> > > > >  sock_close+0x18/0x20 net/socket.c:1255
> > > > >  __fput+0x283/0x920 fs/file_table.c:280
> > > > >  task_work_run+0xdd/0x190 kernel/task_work.c:140
> > > > >  exit_task_work include/linux/task_work.h:30 [inline]
> > > > >  do_exit+0xb89/0x29e0 kernel/exit.c:823
> > > > >  do_group_exit+0x125/0x310 kernel/exit.c:920
> > > > >  get_signal+0x3ec/0x2010 kernel/signal.c:2770
> > > > >  arch_do_signal_or_restart+0x2a8/0x1eb0 arch/x86/kernel/signal.c:811
> > > > >  handle_signal_work kernel/entry/common.c:144 [inline]
> > > > >  exit_to_user_mode_loop kernel/entry/common.c:168 [inline]
> > > > >  exit_to_user_mode_prepare+0x124/0x200 kernel/entry/common.c:198
> > > > >  syscall_exit_to_user_mode+0x36/0x260 kernel/entry/common.c:275
> > > > >  entry_SYSCALL_64_after_hwframe+0x44/0xa9
> > > >
> > > > I've been debugging this and I think enabling KFENCE uncovered that some
> > > > code is assuming that the following is always true:
> > > >
> > > >         ksize(kmalloc(S)) == ksize(kmalloc(S))
> > > >
> > >
> > >
> > > I do not think we make this assumption.
> > >
> > > Each skb tracks the 'truesize' which is populated from __alloc_skb()
> > > using ksize(allocated head) .
> > >
> > > So if ksize() decides to give us random data, it should be still fine,
> > > because we use ksize(buff) only once at alloc skb time, and record the
> > > value in skb->truesize
> > >  (only the socket buffer accounting would be off)
> >
> > Good, thanks for clarifying. So something else must be off then.
> 
> Actually we might have the following assumption :
> 
> buff = kmalloc(size, GFP...)
> if (buff)
>    ASSERT(ksize(buff) >= size)
> 
> So obviously ksize() should not be completely random ;)

One more experiment -- simply adding

--- a/net/core/skbuff.c
+++ b/net/core/skbuff.c
@@ -207,7 +207,21 @@ struct sk_buff *__alloc_skb(unsigned int size, gfp_t gfp_mask,
 	 */
 	size = SKB_DATA_ALIGN(size);
 	size += SKB_DATA_ALIGN(sizeof(struct skb_shared_info));
+	size = 1 << kmalloc_index(size); /* HACK */
 	data = kmalloc_reserve(size, gfp_mask, node, &pfmemalloc);


also got rid of the warnings. Something must be off with some value that
is computed in terms of ksize(). If not, I don't have any explanation
for why the above hides the problem.

Thanks,
-- Marco