Date:   Tue, 17 Apr 2018 15:29:41 +0200
From:   Florian Westphal <fw@...len.de>
To:     Michal Kubecek <mkubecek@...e.cz>, netdev@...r.kernel.org,
        Florian Westphal <fw@...len.de>,
        Marcelo Ricardo Leitner <marcelo.leitner@...il.com>,
        Eric Dumazet <eric.dumazet@...il.com>
Cc:     Jozsef Kadlecsik <kadlec@...ckhole.kfki.hu>
Subject: Re: tcp hang when socket fills up ?

Dominique Martinet <asmadeus@...ewreck.org> wrote:

[ CC Jozsef ]

> Could it have something to do with the way I setup the connection?
> I don't think the "both remotes call connect() with carefully selected
> source/dest port" is a very common case..
> 
> If you look at the tcpdump outputs I attached the sequence usually is
> something like
>  server > client SYN
>  client > server SYN
>  server > client SYNACK
>  client > server ACK
> 
> ultimately it IS a connection, but with an extra SYN packet in front of
> it (that first SYN opens up the conntrack of the nat so that the
> client's syn can come in, the client's conntrack will be that of a
> normal connection since its first SYN goes in directly after the
> server's (it didn't see the server's SYN))
> 
> Looking at my logs again, I'm seeing the same as you:
> 
> This looks like the actual SYN/SYN/SYNACK/ACK:
>  - 14.364090 seq=505004283 likely SYN coming out of server
>  - 14.661731 seq=1913287797 on next line it says receiver
> end=505004284 so likely the matching SYN from client
> Which this time gets a proper SYNACK from server:
> 14.662020 seq=505004283 ack=1913287798
> And following final dataless ACK:
> 14.687570 seq=1913287798 ack=505004284
> 
> Then as you point out some data ACK, where the scale poofs:
> 14.688762 seq=1913287798 ack=505004284+(0) sack=505004284+(0) win=229 end=1913287819
> 14.688793 tcp_in_window: sender end=1913287798 maxend=1913316998 maxwin=29312 scale=7 receiver end=505004284 maxend=505033596 maxwin=29200 scale=7
> 14.688824 tcp_in_window: 
> 14.688852 seq=1913287798 ack=505004284+(0) sack=505004284+(0) win=229 end=1913287819
> 14.688882 tcp_in_window: sender end=1913287819 maxend=1913287819 maxwin=229 scale=0 receiver end=505004284 maxend=505033596 maxwin=29200 scale=7
>
> As you say, only tcp_options() will clear only on side of the scales.
> We don't have sender->td_maxwin == 0 (printed) so I see no other way
> than we are in the last else if:
>  - we have after(end, sender->td_end) (end=1913287819 > sender
> end=1913287798)
>  - I assume the tcp state machine must be confused because of the
> SYN/SYN/SYNACK/ACK pattern and we probably enter the next check, 
> but since this is a data packet it doesn't have the tcp option for scale
> thus scale resets.

Yes, this looks correct. Jozsef, can you please have a look?

Problem seems to be that conntrack believes that ACK packet
re-initializes the connection:

 595                 /*
 596                  * RFC 793: "if a TCP is reinitialized ... then it need
 597                  * not wait at all; it must only be sure to use sequence
 598                  * numbers larger than those recently used."
 599                  */
 600                 sender->td_end =
 601                 sender->td_maxend = end;
 602                 sender->td_maxwin = (win == 0 ? 1 : win);
 603 
 604                 tcp_options(skb, dataoff, tcph, sender);

and last line clears the scale value (no wscale option in data packet).


Transitions are:
 server > client SYN          sNO -> sSS
 client > server SYN          sSS -> sS2
 server > client SYNACK       sS2 -> sSR /* here */
 client > server ACK          sSR -> sES

SYN/ACK was observed in original direction so we hit
state->state == TCP_CONNTRACK_SYN_RECV && dir == IP_CT_DIR_REPLY test
when we see the ack packet and end up in the 'TCP is reinitialized' branch.

AFAICS, without this, connection would move to sES just fine,
as the data ack is in window.

Powered by blists - more mailing lists