netdev - Re: [PATCH v6 net-next 0/6] tcp: Introduce optional per-netns ehash.

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <20221011215359.13173-1-kuniyu@amazon.com>
Date:   Tue, 11 Oct 2022 14:53:59 -0700
From:   Kuniyuki Iwashima <kuniyu@...zon.com>
To:     <edumazet@...gle.com>
CC:     <davem@...emloft.net>, <kuba@...nel.org>, <kuni1840@...il.com>,
        <kuniyu@...zon.com>, <netdev@...r.kernel.org>, <pabeni@...hat.com>,
        <patchwork-bot+netdevbpf@...nel.org>
Subject: Re: [PATCH v6 net-next 0/6] tcp: Introduce optional per-netns ehash.

From:   Eric Dumazet <edumazet@...gle.com>
Date:   Tue, 11 Oct 2022 14:46:03 -0700
> On Tue, Sep 20, 2022 at 12:00 PM <patchwork-bot+netdevbpf@...nel.org> wrote:
> >
> > Hello:
> >
> > This series was applied to netdev/net-next.git (master)
> > by Jakub Kicinski <kuba@...nel.org>:
> >
> > On Wed, 7 Sep 2022 18:10:16 -0700 you wrote:
> > > The more sockets we have in the hash table, the longer we spend looking
> > > up the socket.  While running a number of small workloads on the same
> > > host, they penalise each other and cause performance degradation.
> > >
> > > The root cause might be a single workload that consumes much more
> > > resources than the others.  It often happens on a cloud service where
> > > different workloads share the same computing resource.
> > >
> > > [...]
> >
> > Here is the summary with links:
> >   - [v6,net-next,1/6] tcp: Clean up some functions.
> >     https://git.kernel.org/netdev/net-next/c/08eaef904031
> >   - [v6,net-next,2/6] tcp: Don't allocate tcp_death_row outside of struct netns_ipv4.
> >     https://git.kernel.org/netdev/net-next/c/e9bd0cca09d1
> >   - [v6,net-next,3/6] tcp: Set NULL to sk->sk_prot->h.hashinfo.
> >     https://git.kernel.org/netdev/net-next/c/429e42c1c54e
> >   - [v6,net-next,4/6] tcp: Access &tcp_hashinfo via net.
> >     https://git.kernel.org/netdev/net-next/c/4461568aa4e5
> >   - [v6,net-next,5/6] tcp: Save unnecessary inet_twsk_purge() calls.
> >     https://git.kernel.org/netdev/net-next/c/edc12f032a5a
> >   - [v6,net-next,6/6] tcp: Introduce optional per-netns ehash.
> >     https://git.kernel.org/netdev/net-next/c/d1e5e6408b30
> >
> > You are awesome, thank you!
> > --
> > Deet-doot-dot, I am a bot.
> > https://korg.docs.kernel.org/patchwork/pwbot.html
> >
> >
> 
> Note that this series is causing issues.
> 
> BUG: KASAN: use-after-free in tcp_or_dccp_get_hashinfo
> include/net/inet_hashtables.h:181 [inline]
> BUG: KASAN: use-after-free in reqsk_queue_unlink+0x320/0x350
> net/ipv4/inet_connection_sock.c:913
> Read of size 8 at addr ffff88807545bd80 by task syz-executor.2/8301
> 
> CPU: 1 PID: 8301 Comm: syz-executor.2 Not tainted
> 6.0.0-syzkaller-02757-gaf7d23f9d96a #0
> Hardware name: Google Google Compute Engine/Google Compute Engine,
> BIOS Google 09/22/2022
> Call Trace:
> <IRQ>
> __dump_stack lib/dump_stack.c:88 [inline]
> dump_stack_lvl+0xcd/0x134 lib/dump_stack.c:106
> print_address_description mm/kasan/report.c:317 [inline]
> print_report.cold+0x2ba/0x719 mm/kasan/report.c:433
> kasan_report+0xb1/0x1e0 mm/kasan/report.c:495
> tcp_or_dccp_get_hashinfo include/net/inet_hashtables.h:181 [inline]
> reqsk_queue_unlink+0x320/0x350 net/ipv4/inet_connection_sock.c:913
> inet_csk_reqsk_queue_drop net/ipv4/inet_connection_sock.c:927 [inline]
> inet_csk_reqsk_queue_drop_and_put net/ipv4/inet_connection_sock.c:939 [inline]
> reqsk_timer_handler+0x724/0x1160 net/ipv4/inet_connection_sock.c:1053
> call_timer_fn+0x1a0/0x6b0 kernel/time/timer.c:1474
> expire_timers kernel/time/timer.c:1519 [inline]
> __run_timers.part.0+0x674/0xa80 kernel/time/timer.c:1790
> __run_timers kernel/time/timer.c:1768 [inline]
> run_timer_softirq+0xb3/0x1d0 kernel/time/timer.c:1803
> __do_softirq+0x1d0/0x9c8 kernel/softirq.c:571
> invoke_softirq kernel/softirq.c:445 [inline]
> __irq_exit_rcu+0x123/0x180 kernel/softirq.c:650
> irq_exit_rcu+0x5/0x20 kernel/softirq.c:662
> sysvec_apic_timer_interrupt+0x93/0xc0 arch/x86/kernel/apic/apic.c:1107
> </IRQ>
> 
> We forgot to make sure there were no TCP_NEW_SYN_RECV sockets in the
> hash tables before deleting the hash tables.
> 
> Probably inet_twsk_purge() (when called when
> et->ipv4.tcp_death_row.hashinfo->pernet is true)
> also needs to remove all TCP_NEW_SYN_RECV request sockets.

Exactly, I'll do a quick test and post a fix.
Thank you for reporting!