netdev - Re: TCP sender stuck in persist despite peer advertising non-zero window

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CAK6E8=fCwjP47DvSj4YQQ6xn25bVBN_1mFtrBwOJPYU6jXVcgQ@mail.gmail.com>
Date:   Thu, 15 Oct 2020 14:39:24 -0700
From:   Yuchung Cheng <ycheng@...gle.com>
To:     Neal Cardwell <ncardwell@...gle.com>
Cc:     Apollon Oikonomopoulos <apoikos@...sg.gr>,
        Netdev <netdev@...r.kernel.org>,
        Eric Dumazet <edumazet@...gle.com>,
        Soheil Hassas Yeganeh <soheil@...gle.com>
Subject: Re: TCP sender stuck in persist despite peer advertising non-zero window

On Thu, Oct 15, 2020 at 1:22 PM Neal Cardwell <ncardwell@...gle.com> wrote:
>
> On Thu, Oct 15, 2020 at 2:31 PM Apollon Oikonomopoulos <apoikos@...sg.gr> wrote:
> >
> > Hi,
> >
> > I'm trying to debug a (possible) TCP issue we have been encountering
> > sporadically during the past couple of years. Currently we're running
> > 4.9.144, but we've been observing this since at least 3.16.
> >
> > Tl;DR: I believe we are seeing a case where snd_wl1 fails to be properly
> > updated, leading to inability to recover from a TCP persist state and
> > would appreciate some help debugging this.
>
> Thanks for the detailed report and diagnosis. I think we may need a
> fix something like the following patch below.
>
> Eric/Yuchung/Soheil, what do you think?
wow hard to believe how old this bug can be. The patch looks good but
can Apollon verify this patch fix the issue?

>
> commit 42b37c72aa73aaabd0c01b8c05c2205236279021
> Author: Neal Cardwell <ncardwell@...gle.com>
> Date:   Thu Oct 15 16:06:11 2020 -0400
>
>     tcp: fix to update snd_wl1 in bulk receiver fast path
>
>     In the header prediction fast path for a bulk data receiver, if no
>     data is newly acknowledged then we do not call tcp_ack() and do not
>     call tcp_ack_update_window(). This means that a bulk receiver that
>     receives large amounts of data can have the incoming sequence numbers
>     wrap, so that the check in tcp_may_update_window fails:
>        after(ack_seq, tp->snd_wl1)
>
>     The fix is to update snd_wl1 in the header prediction fast path for a
>     bulk data receiver, so that it keeps up and does not see wrapping
>     problems.
>
>     Signed-off-by: Neal Cardwell <ncardwell@...gle.com>
>     Reported-By: Apollon Oikonomopoulos <apoikos@...sg.gr>
>
> diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
> index b1ce2054291d..75be97f6a7da 100644
> --- a/net/ipv4/tcp_input.c
> +++ b/net/ipv4/tcp_input.c
> @@ -5766,6 +5766,8 @@ void tcp_rcv_established(struct sock *sk, struct
> sk_buff *skb)
>                                 tcp_data_snd_check(sk);
>                                 if (!inet_csk_ack_scheduled(sk))
>                                         goto no_ack;
> +                       } else {
> +                               tcp_update_wl(tp, TCP_SKB_CB(skb)->seq);
>                         }
>
>                         __tcp_ack_snd_check(sk, 0);
>
> neal