linux-kernel - TCP stack sometimes loses ACKs ... or something

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <17744.10620.389409.136737@cse.unsw.edu.au>
Date:	Tue, 7 Nov 2006 17:36:44 +1100
From:	Neil Brown <neilb@...e.de>
To:	netdev@...r.kernel.org, linux-kernel@...r.kernel.org
Subject: TCP stack sometimes loses ACKs ... or something

I upgraded my notebook from 2.6.16 to 2.6.18 recently and noticed that
I couldn't talk to my VOIP device (which has a WEB interface).
Watching traffic I see the three-way-handshake working perfectly, and
then the first data packet is sent (a partial HTTP request: 
GET / HTTP/1.1 ....) and an ACK comes back from the device.
Then the next data packet (remainder of the HTTP request) is sent, but
tcpdump never sees the ACK, nor does the TCP stack.  So the data gets
recent repeatedly.  No ack. Ever.

With 2.6.16, The ack comes back just fine and the connection proceeds
as you would expect.

As it was a very reproducible problem I decided to try "git bisect"
and found 

 bad: [7b4f4b5ebceab67ce440a61081a69f0265e17c2a] [TCP]: Set default max buffers from memory pool size

I double checked as this seemed a fairly unlikely patch to cause the
problem, but this definitely is it.
The net effect of this patch is to change the last of the three
numbers in 
    cat /proc/sys/net/ipv4/tcp_[rw]mem 
from well below 2^20 to well above. 2^20 seems to be a significant
number. I set tcp_wmem to that and the ACK was lost.  I set it to
one less and the first ACK (at least) was accepted.
I ended up setting both r and w to 100000 and everything is fine.

Exploring more deeply, and comparing:
  - a failing connection (to VIOP box, [rw]mem large)
  - a working connection to VOIP box ([rw]mem small)
  - a working connection to another machine ([rw]mem irrelevant).
I find:

  The VIOP returns MSS=1360 in the SYN/ACK packet.  Other machine
    returns MSS=1460

  The ack that is getting lost contains data as well as the
  ACK. i.e. the same packet that ACKs at the TCP level includes the
  HTTP level reply.
  The matching ACK from the other machine (some Linux 2.6.8 I think)
   is a data-less ACK followed very quickly by the HTTP reply in
   a separate packet.

  The 'Timestamps' option coming back from the VOIP box is a little
  odd.  The Timestamp in the SYN/ACK is the same as the timestamp in
  the next ACK (the ack for the first partial HTTP request).
  The Timestamp in the next packet which is the one that gets lost has
  exactly the same TSval as previous packets, and TSecr is one more
  than in the previous packet.

I assume that one (or more) of these differences combined with the
large tcp_[rw]mem value cause the packet loss, but I have no idea
which.

Help?

I can make the tcp traces available if needed, but these are really
the only non-trivial differences.

I'm willing to test patches.

NeilBrown
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/