netdev - Re: NAT stops forwarding ACKs after PMTU discovery

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <5211DAA6.1070302@fatooh.org>
Date:	Mon, 19 Aug 2013 01:43:18 -0700
From:	Corey Hickey <bugfood-ml@...ooh.org>
To:	Eric Dumazet <eric.dumazet@...il.com>
CC:	Jozsef Kadlecsik <kadlec@...ckhole.kfki.hu>,
	Linux Netdev List <netdev@...r.kernel.org>,
	netfilter-devel@...r.kernel.org
Subject: Re: NAT stops forwarding ACKs after PMTU discovery

On 2013-08-18 17:03, Eric Dumazet wrote:
> On Sun, 2013-08-18 at 17:00 -0700, Eric Dumazet wrote:
> 
>> Code like this seems very suspect to me :
>>
>> before(sack, receiver->td_end + 1)
>>
> 
> My suggestion would be to try :
> 
> diff --git a/net/netfilter/nf_conntrack_proto_tcp.c b/net/netfilter/nf_conntrack_proto_tcp.c

[...]

Thanks for all your suggestions--I really wasn't expecting so much on a
weekend. Here's all the data I have for tonight.

I tried the linux-next kernel, then linux-next with your patch applied.
Neither of them fix the problem, unfortunately. I have taken tcpdumps
for a working SSH and a failing SSH.

http://fatooh.org/files/tmp/linux-next-patch1.tar.bz2

[localhost]
sudo tcpdump -ni br0 -s 0 -w /tmp/local.pcap 'host 10.15.24.13 or icmp'

[router eth0]
tcpdump -ni eth0 -s 0 -w /tmp/eth0.pcap \
    'host 10.15.24.13 or (icmp and host not 69.78.33.132)'
* the exclusion here is just to remove some unrelated clutter

[router tun0]
tcpdump -ni tun0 -s 0 -w /tmp/tun0.pcap -s 0 'host 10.15.24.13 or icmp'

[remote]
tcpdump -ni eth0 -s 0 -w remote.pcap 'host 192.168.61.56'

Some notes:

1. I tested the new kernels only on the Linux router, assuming that is
where it was intended.

2. I take back what I wrote earlier about every connection that involves
PMTU discovery failed; I may have been observing this wrong. For now,
the situation is that some connections stop forwarding packets from the
remote host immediately after the retransmit, while other work fine.

3. From local.pcap, you can see that my localhost doesn't actually
transmit a large packet, yet the router's eth0 sees a large packet come
in. I think this is due to TSO, but I'm not completely sure.

4. For some reason, I cannot reproduce this when SSHing to a host at
work that is running Debian sid with 3.10-1-amd64, but I can reproduce
it when SSHing to hosts running Centos 6.4 with
2.6.32-358.6.1.el6.x86_64 (which surely has a ton of patches applied,
for whatever that's worth).

5. I have only a vague understanding of SACK; I will be reading up on
this soon. I will also look into packetdrill for reproducing the
problem, if the SSH results aren't good enough.

6. If I reduce the MTU on localhost to match the path MTU, the problem
does go away.

Thanks again for all the help,
Corey
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html