netdev - Re: Bug in tcp timestamp option ? TSecr in SYN-ACK != TSval in SYN

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Wed, 27 May 2015 00:09:42 -0700
From:	Gopakumar Choorakkot Edakkunni <gopakumar.c.e@...il.com>
To:	Eric Dumazet <eric.dumazet@...il.com>
Cc:	netdev@...r.kernel.org
Subject: Re: Bug in tcp timestamp option ? TSecr in SYN-ACK != TSval in SYN

Hi Eric,

Thanks a lot for the response, and sorry about the 3-times-email, was
not sure whether majordomo accepted my subscription or not and hence
the retx :)

So if a sequence happens as below

1. Client sends the first SYN with some TSval X
2. Server responds SYN-ACK with TSecr X
3. The SYN-ACK just gets dropped on the way back to the client
4. Client sends a SYN retry after N seconds with a new TSval Y
5. Server responds SYN-ACK with TSecr X

And if there is some firewall in between in the amazon environment
where the firewall expects to see the SYN-ACK with TSecr Y, then I
guess it matches the problem I saw ? In my case clearly the SYN-ACKs
never reached the client no matter how many times they were
retransmitted. So this would mean that if there is such a wierd
firewall in between, then one missing SYN-ACK can cause the tcp
connection to eventually timeout ! This of course is just guesswork
based on what we saw as the behaviour from tcpdump on server and
client side when the timeouts were happening. Does this sound like a
possibility - has anyone come across "interesting" firewalls like this
?

And about your question: "Are you establishing many active sessions
per minute to this particular target ?" - in my particular case there
were not more than three linux client boxes sitting behind a NAT
(sharing the same public IP) and talking to the same server. And each
client box opens a tcp socket once in 30 seconds to the server. So the
number of active sessions per 30 seconds is not more than 3 sessions.
Now if the NAT device had some bug and ended up NAT-ing more than one
client SYN packet to the same source port, then of course thats
another theory for why this TSecr/TSval mismatch can happen (other
than the SYN-ACK drop theory above).

Rgds,
Gopa.

On Tue, May 26, 2015 at 11:23 PM, Eric Dumazet <eric.dumazet@...il.com> wrote:
> On Tue, 2015-05-26 at 23:12 -0700, Eric Dumazet wrote:
>> On Tue, 2015-05-26 at 22:47 -0700, Gopakumar Choorakkot Edakkunni wrote:
>> > All,
>> >
>> > The original query I had posted is here :
>> > http://stackoverflow.com/questions/30414350/tcp-syn-ack-tsecr-not-matching-tsval-in-syn
>> > .. The summary is that once in a while, the TSval in SYN is not what
>> > is getting echoed in TSecr, and looks like something on amazon aws
>> > side is very strict about that and drops those packets. Any clues on
>> > this - whether its a known issue/fixed elsewhere etc. would be of
>> > great help.
>>
>> I guess that if you send SYN packets 3 times as your email did on
>> netdev, that might cause some issues...
>>
>> More seriously, server has a SYN_RECV socket with same tuple, because of
>> a SYN sent earlier :
>>
>> 8:36:00.593136 IP XX.YY.ZZ.VV.24548 > AA.BB.CC.DD.443: Flags [S], seq
>> 1204544933, win 29200, options [mss 1320,sackOK,TS val 6032576 ecr
>> 0,nop,wscale 7], length 0
>>
>> 18:36:00.593171 IP AA.BB.CC.DD.443 > XX.YY.ZZ.VV.24548: Flags [S.], seq
>> 986069863, ack 1204544934, win 14480, options [mss 1460,sackOK,TS val
>> 180940028 ecr 6001497,nop,wscale 5], length 0
>>
>> 18:36:00.992699 IP AA.BB.CC.DD.443 > XX.YY.ZZ.VV.24548: Flags [S.], seq
>> 986069863, ack 1204544934, win 14480, options [mss 1460,sackOK,TS val
>> 180940128 ecr 6001497,nop,wscale 5], length 0
>>
>>
>> From these traces, we can guess a SYN packet was sent about 31 seconds
>> earlier.
>>
>> SYNACK rtx do not update the TSECR : Initial SYN TSval value (6001497)
>> is mirrored.
>>
>> Are you establishing many active sessions per minute to this particular
>> target ?
>>
>
> Here is a packetdrill test to demonstrate behavior :
>
> // Test that SYNACK rtx tsecr is not changed (original SYN tsval)
>
> `../common/defaults.sh
> `
>
> // Create a socket.
> 0.000 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
> 0.000 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
>
> 0.000 bind(3, ..., ...) = 0
> 0.000 listen(3, 1) = 0
>
> // Establish a connection.
> 0.100 < S 0:0(0) win 20000 <mss 1000,sackOK,TS val 100 ecr 0>
> +0    > S. 0:0(0) ack 1 <mss 1460,sackOK,TS val 100 ecr 100>
>
> +0.100 < S 0:0(0) win 20000 <mss 1000,sackOK,TS val 199 ecr 0>
> // check rtx tsecr is sill 100, not 199
> +0     > S. 0:0(0) ack 1 <mss 1460,sackOK,TS val 200 ecr 100>
>
> +0.100 < . 1:1(0) ack 1 win 20000 <nop,nop,TS val 300 ecr 200>
> +0    accept(3, ..., ...) = 4
>
>
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html