netdev - Re: TCP keepalives ignored by kernel when the contain timestamps

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date:	Fri, 10 Jun 2011 11:07:48 -0500
From:	Charles Bearden <Charles.F.Bearden@....tmc.edu>
To:	Eric Dumazet <eric.dumazet@...il.com>
CC:	<netdev@...r.kernel.org>
Subject: Re: TCP keepalives ignored by kernel when the contain timestamps

On 06/10/2011 10:10 AM, Charles Bearden wrote:
> On 06/10/2011 08:56 AM, Eric Dumazet wrote:
>> Le jeudi 09 juin 2011 à 10:26 -0500, Charles Bearden a écrit :
>>> I have come across a case that looks like it might be a kernel bug. It appears
>>> that tcp keepalives sent by a remote system are ignored when they contain tcp
>>> timestamps, but are ACKed when they don't. When they are ignored, the remote
>>> system resets the connection after a number of retries.
>>>
>>> I have replicated this problem on both Ubuntu 10.04 with a 2.6.32-32-server
>>> kernel (x86_64) and CentOS 5.6 with a 2.6.18-238.12.1.el5 kernel. I'm sorry that
>>> I haven't had a chance to try to replicate the bug with a newer kernel, though a
>>> co-worker has looked through changelogs for more recent kernels and didn't find
>>> anything that looked relevant.
>>>
>>> From either of these hosts I run an application that connects to a remote host
>>> for 2-3 minutes, and that for most of that time sends no application data back
>>> and forth. After 30 seconds of no data from the Linux host, the remote host
>>> sends a garden variety keepalive. When the remote host includes tcp timestamps
>>> in the keepalives, they are ignored by the Linux host, and the remote host
>>> resets the connection after 10 unACKed keepalives. When timestamps are absent
>>> from the keepalives, the Linux host ACKs each one, and all is copacetic.
>>>
>>> Text output of a tcpdump trace of a connection that fails:
>>> http://pastebin.com/v6CpteJ9
>>>
>>> Text output of a tcpdump trace of a connection that succeeds:
>>> http://pastebin.com/KVLb3Mzh
>>>
>>> More details, in case you think they are relevant:
>>>
>>> My application creates a JDBC connection to a remote MS SQL Server and
>>> executes a statement that does not return a result set, and so it doesn't
>>> need to pass application data back and forth while it executes. The
>>> statement takes 2 or 3 minutes to complete. I connect to two different
>>> remote hosts: a Win2003 machine, and a Win2008R2 machine. The Win2003
>>> machine doesn't put timestamps in its keep-alives, so the application
>>> completes successfully when connecting to that host. If tcp timestamps
>>> are enabled on the Linux host, the Win2008 host includes them in its
>>> keepalives, and they are unACKed, so the connection is reset; if they
>>> are disabled on the Linux host, the Win2008 host doesn't include them in
>>> the keepalives, and the application completes successfully. I use (as
>>> you might expect) sysctl to disable tcp timestamps on the Linux hosts.
>>>
>>> I have dumps for all permutations of CentOS/Ubuntu, Win200[38], and +/-
>>> timestamps on the Linux side, and I will post them if the developers think that
>>> they would be useful.
>>
>> Hi Charles
>>
>> I could not reproduce the problem here, even using a quite old kernel as
>> receiver (2.6.9)
>>
>> 15:54:33.566192 IP 192.168.20.108.55926> 192.168.20.124.777: SWE
>> 479814493:479814493(0) win 14600<mss 1460,sackOK,timestamp 151666
>> 0,nop,wscale 7>
>> 15:54:33.566265 IP 192.168.20.124.777> 192.168.20.108.55926: S
>> 3714869381:3714869381(0) ack 479814494 win 5792<mss
>> 1460,sackOK,timestamp 54553041 151666,nop,wscale 2>
>> 15:54:33.566274 IP 192.168.20.108.55926> 192.168.20.124.777: . ack 1
>> win 115<nop,nop,timestamp 151666 54553041>
>> 15:54:33.566281 IP 192.168.20.108.55926> 192.168.20.124.777: P 1:5(4)
>> ack 1 win 115<nop,nop,timestamp 151666 54553041>
>> 15:54:33.566351 IP 192.168.20.124.777> 192.168.20.108.55926: . ack 5
>> win 1448<nop,nop,timestamp 54553041 151666>
>> 15:54:33.566375 IP 192.168.20.124.777> 192.168.20.108.55926: P 1:5(4)
>> ack 5 win 1448<nop,nop,timestamp 54553041 151666>
>> 15:54:33.566380 IP 192.168.20.108.55926> 192.168.20.124.777: . ack 5
>> win 115<nop,nop,timestamp 151666 54553041>
>> 15:54:43.577945 IP 192.168.20.108.55926> 192.168.20.124.777: . 4:5(1)
>> ack 5 win 115<nop,nop,timestamp 152668 54553041>
>> 15:54:43.578012 IP 192.168.20.124.777> 192.168.20.108.55926: . ack 5
>> win 1448<nop,nop,timestamp 54563053 152668,nop,nop,sack sack 1 {4:5}>
>> 15:54:53.597946 IP 192.168.20.108.55926> 192.168.20.124.777: . 4:5(1)
>> ack 5 win 115<nop,nop,timestamp 153670 54563053>
>> 15:54:53.598012 IP 192.168.20.124.777> 192.168.20.108.55926: . ack 5
>> win 1448<nop,nop,timestamp 54573073 153670,nop,nop,sack sack 1 {4:5}>
>>
>>
>> Are you sure frame tcp checksums are OK when the 'faulty' linux receive
>> them ? (tcpdump -v)
>
> I will check when I get into the office and let you know.

You are correct: the checksums in the keepalives are broken, though they are 
correct in the other segments from the Win2008 server. I have updated the pastes 
linked to above with 'tcpdump -v' output. I apologize for missing that problem 
the first time around.
Chuck
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html