netdev - Re: TCP sender stuck in persist despite peer advertising non-zero window

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CADVnQym6OPVRcJ6PdR3hjN5Krcn0pugshdLZsrnzNQe1c52HXA@mail.gmail.com>
Date:   Thu, 15 Oct 2020 16:22:21 -0400
From:   Neal Cardwell <ncardwell@...gle.com>
To:     Apollon Oikonomopoulos <apoikos@...sg.gr>
Cc:     Netdev <netdev@...r.kernel.org>,
        Eric Dumazet <edumazet@...gle.com>,
        Yuchung Cheng <ycheng@...gle.com>,
        Soheil Hassas Yeganeh <soheil@...gle.com>
Subject: Re: TCP sender stuck in persist despite peer advertising non-zero window

On Thu, Oct 15, 2020 at 2:31 PM Apollon Oikonomopoulos <apoikos@...sg.gr> wrote:
>
> Hi,
>
> I'm trying to debug a (possible) TCP issue we have been encountering
> sporadically during the past couple of years. Currently we're running
> 4.9.144, but we've been observing this since at least 3.16.
>
> Tl;DR: I believe we are seeing a case where snd_wl1 fails to be properly
> updated, leading to inability to recover from a TCP persist state and
> would appreciate some help debugging this.

Thanks for the detailed report and diagnosis. I think we may need a
fix something like the following patch below.

Eric/Yuchung/Soheil, what do you think?

commit 42b37c72aa73aaabd0c01b8c05c2205236279021
Author: Neal Cardwell <ncardwell@...gle.com>
Date:   Thu Oct 15 16:06:11 2020 -0400

    tcp: fix to update snd_wl1 in bulk receiver fast path

    In the header prediction fast path for a bulk data receiver, if no
    data is newly acknowledged then we do not call tcp_ack() and do not
    call tcp_ack_update_window(). This means that a bulk receiver that
    receives large amounts of data can have the incoming sequence numbers
    wrap, so that the check in tcp_may_update_window fails:
       after(ack_seq, tp->snd_wl1)

    The fix is to update snd_wl1 in the header prediction fast path for a
    bulk data receiver, so that it keeps up and does not see wrapping
    problems.

    Signed-off-by: Neal Cardwell <ncardwell@...gle.com>
    Reported-By: Apollon Oikonomopoulos <apoikos@...sg.gr>

diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
index b1ce2054291d..75be97f6a7da 100644
--- a/net/ipv4/tcp_input.c
+++ b/net/ipv4/tcp_input.c
@@ -5766,6 +5766,8 @@ void tcp_rcv_established(struct sock *sk, struct
sk_buff *skb)
                                tcp_data_snd_check(sk);
                                if (!inet_csk_ack_scheduled(sk))
                                        goto no_ack;
+                       } else {
+                               tcp_update_wl(tp, TCP_SKB_CB(skb)->seq);
                        }

                        __tcp_ack_snd_check(sk, 0);

neal