netdev - strange routing issue--packets stop getting forwarded for a live connection

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <4E506A46.6060407@fatooh.org>
Date:	Sat, 20 Aug 2011 19:15:34 -0700
From:	Corey Hickey <bugfood-ml@...ooh.org>
To:	Linux Netdev List <netdev@...r.kernel.org>
Subject: strange routing issue--packets stop getting forwarded for a live
 connection

Hi,

Please forgive me for asking a user question on a dev list; does the
linux-net list no longer exist? Majordomo wouldn't subscribe me and I
see no recent history in the archives. If there's a better place for
this question, please tell me. Anyway:

I have a strange issue where, reliably, certain conditions cause my
Linux router to stop forwarding packets for a connection.

----------------------------------------------------------------------

This is my setup:

client      --> linux router          --> vpn --> work desktop
198.18.0.3      198.18.0.1    (eth0)              192.168.10.88
                192.168.6.230 (tun0)

All hosts are running Debian Sid with the stock Debian 3.0.0-1-amd64
kernel. tun0 is set up by openconnect (open-source client for cisco
anyconnnect), which has been historically reliable for me.

I noticed this problem happening when I replaced the router with a new
host. The old host was 32-bit, running Linux 2.6.38, and configured
identically (I think) with respect to routing and iptables. I didn't
have a problem then.

----------------------------------------------------------------------

I have seen this problem happen with http, sometimes, but the easiest
way to reproduce the issue every time is to use SSH with X11 forwarding
(I have no idea why). I can SSH, through my router and VPN connection,
to my desktop at work. I can log in, poke around, do whatever; as soon
as I run some particular X11 programs, the connection hangs. xlogo and
xeyes are fine, but rxvt and jconsole are not.

So, my baseline test is to run rxvt directly. This command always hangs:

$ ssh -X chickey@....168.10.88 rxvt

I have run simultaneous tcpdumps on the router: one on eth0 and the
other on tun0. I see the tcp connection and ssh sessions get set up,
then many encrypted packets go back and forth. At a certain, reliably
reproducible point, a 1368 byte packet comes in on eth0 and does not
leave tun0; the retransmissions do not get forwarded either.

I have not been able to figure out the cause of this. Here's what I have
investigated:

1. Number of packets on the connection; doesn't seem to matter, because
I can use SSH for other purposes just fine.

2. Transmission rate; doesn't seem to matter, because I can do
$ ssh -X chickey@....168.10.88 cat /dev/zero > /dev/null

3. MTU size; 1500 on eth0 and 1406 on tun0. Bigger packets have been
transferred fine.

4. VPN client bug; maybe, but I don't think so yet. I can do the same
thing if I SSH directly from the router. This is fine:
ssh -X 198.18.0.1 "ssh -X chickey@....168.10.88 rxvt"

5. Connection tracking issue; conntrack shows no change in stage for the
connection when it hangs.

6. Some firewall rule. Stripping down my iptables setup to the minimum
does not help. I have also removed all qdiscs.

----------------------------------------------------------------------

Can anybody please suggest something else I should try here? This is
very confusing to me.

I am attaching a tarball of tcpdumps and other pertinent information.

Thank you,
Corey

Download attachment "problem.tar.bz2" of type "application/octet-stream" (23175 bytes)