netdev - Re: [PATCH net] tcp: prevent concurrent execution of tcp_sk_exit

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CANn89i+aZHCc-uGyjqxsuio-6JsUJ3wk7gLrTZK=XgpgfMKoFg@mail.gmail.com>
Date: Mon, 19 Aug 2024 17:36:17 +0200
From: Eric Dumazet <edumazet@...gle.com>
To: Paolo Abeni <pabeni@...hat.com>
Cc: Florian Westphal <fw@...len.de>, netdev@...r.kernel.org, 
	"David S. Miller" <davem@...emloft.net>, Jakub Kicinski <kuba@...nel.org>, 
	Kuniyuki Iwashima <kuniyu@...zon.com>, Jason Xing <kerneljasonxing@...il.com>, 
	syzbot+8ea26396ff85d23a8929@...kaller.appspotmail.com
Subject: Re: [PATCH net] tcp: prevent concurrent execution of tcp_sk_exit_batch

On Thu, Aug 15, 2024 at 12:47 PM Paolo Abeni <pabeni@...hat.com> wrote:
>
> On 8/13/24 00:28, Florian Westphal wrote:
> > Its possible that two threads call tcp_sk_exit_batch() concurrently,
> > once from the cleanup_net workqueue, once from a task that failed to clone
> > a new netns.  In the latter case, error unwinding calls the exit handlers
> > in reverse order for the 'failed' netns.
> >
> > tcp_sk_exit_batch() calls tcp_twsk_purge().
> > Problem is that since commit b099ce2602d8 ("net: Batch inet_twsk_purge"),
> > this function picks up twsk in any dying netns, not just the one passed
> > in via exit_batch list.
> >
> > This means that the error unwind of setup_net() can "steal" and destroy
> > timewait sockets belonging to the exiting netns.
> >
> > This allows the netns exit worker to proceed to call
> >
> > WARN_ON_ONCE(!refcount_dec_and_test(&net->ipv4.tcp_death_row.tw_refcount));
> >
> > without the expected 1 -> 0 transition, which then splats.
> >
> > At same time, error unwind path that is also running inet_twsk_purge()
> > will splat as well:
> >
> > WARNING: .. at lib/refcount.c:31 refcount_warn_saturate+0x1ed/0x210
> > ...
> >   refcount_dec include/linux/refcount.h:351 [inline]
> >   inet_twsk_kill+0x758/0x9c0 net/ipv4/inet_timewait_sock.c:70
> >   inet_twsk_deschedule_put net/ipv4/inet_timewait_sock.c:221
> >   inet_twsk_purge+0x725/0x890 net/ipv4/inet_timewait_sock.c:304
> >   tcp_sk_exit_batch+0x1c/0x170 net/ipv4/tcp_ipv4.c:3522
> >   ops_exit_list+0x128/0x180 net/core/net_namespace.c:178
> >   setup_net+0x714/0xb40 net/core/net_namespace.c:375
> >   copy_net_ns+0x2f0/0x670 net/core/net_namespace.c:508
> >   create_new_namespaces+0x3ea/0xb10 kernel/nsproxy.c:110
> >
> > ... because refcount_dec() of tw_refcount unexpectedly dropped to 0.
> >
> > This doesn't seem like an actual bug (no tw sockets got lost and I don't
> > see a use-after-free) but as erroneous trigger of debug check.
> >
> > Add a mutex to force strict ordering: the task that calls tcp_twsk_purge()
> > blocks other task from doing final _dec_and_test before mutex-owner has
> > removed all tw sockets of dying netns.
> >
> > Fixes: e9bd0cca09d1 ("tcp: Don't allocate tcp_death_row outside of struct netns_ipv4.")
> > Cc: Kuniyuki Iwashima <kuniyu@...zon.com>
> > Cc: Jason Xing <kerneljasonxing@...il.com>
> > Reported-by: syzbot+8ea26396ff85d23a8929@...kaller.appspotmail.com
> > Closes: https://lore.kernel.org/netdev/0000000000003a5292061f5e4e19@google.com/
> > Link: https://lore.kernel.org/netdev/20240812140104.GA21559@breakpoint.cc/
> > Signed-off-by: Florian Westphal <fw@...len.de>
>
> The fixes LGTM, but I prefer to keep the patch in PW a little longer to
> allow Eric having a look here.

Excellent patch, thanks a lot Florian !

I had this syzbot report in my queue for a while, but I presumed this
was caused by RDS,
thus waiting for more signal.

Reviewed-by: Eric Dumazet <edumazet@...gle.com>