netdev - linux-3.0.18+r8169+ipv4/tcp forwarding = tso/gso weirdness and performance degration

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <20120314190156.622c8cd5@vostro>
Date:	Wed, 14 Mar 2012 19:01:56 +0200
From:	Timo Teras <timo.teras@....fi>
To:	netdev@...r.kernel.org
Cc:	Francois Romieu <romieu@...zoreil.com>
Subject: linux-3.0.18+r8169+ipv4/tcp forwarding = tso/gso weirdness and
 performance degration

Hi,

I have a router box running linux-3.0.18 (with grsec patches).

with the NIC hardware:
r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
r8169 0000:00:09.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18
r8169 0000:00:09.0: (unregistered net_device): no PCI Express capability
r8169 0000:00:09.0: eth0: RTL8169sc/8110sc at 0xf82f8000, 00:30:18:ab:6b:54, XID 18000000 IRQ 18
r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
r8169 0000:00:0b.0: PCI INT A -> GSI 19 (level, low) -> IRQ 19
r8169 0000:00:0b.0: (unregistered net_device): no PCI Express capability
r8169 0000:00:0b.0: eth1: RTL8169sc/8110sc at 0xf82fa000, 00:30:18:ab:6b:55, XID 18000000 IRQ 19
r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
r8169 0000:00:0c.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
r8169 0000:00:0c.0: (unregistered net_device): no PCI Express capability
r8169 0000:00:0c.0: eth2: RTL8169sc/8110sc at 0xf82fc000, 00:30:18:ab:6b:56, XID 18000000 IRQ 16

This box is working just as a plain IPv4 router (internal RFC1918
address space) forwarding packets.

It routes basically from eth2 to multiple vlans over bond0 consisting of eth0 and eth1.

I have most hw accel stuff turned off, and "ethtool -k eth0" says:
Offload parameters for eth0:
rx-checksumming: on
tx-checksumming: on
scatter-gather: off
tcp segmentation offload: off
udp fragmentation offload: off
generic segmentation offload: off

The same applies for all interfaces (except lo).

However, tcpdump on this box indicates that I'm receiving very
long (tcp length more than mtu) incoming packets on eth2 implying that
gso/tso got turned on somehow. eth2 is connected with cross-over cable
to similar box running a bit older linux box; but gso/tso is turned off
there too. When dumping simultaneously on the other side, it indicates
that all packets sent are normal length, and no merging was performed
earlier (fits mtu 1500).

So it would appear that the router box somehow insists on doing gso/tso,
and sadly it will also mess up on the send path (the incoming merged
packet is forwarded, but sent out short) causing lost segments and
serious performance degration.

Any pointers how to next debug/fix/workaround this issue?

-Timo
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html