netdev - Strange packet drops with heavy firewalling

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <m339z50x1l.fsf@ursa.amorsen.dk>
Date:	Fri, 09 Apr 2010 11:56:06 +0200
From:	Benny Amorsen <benny+usenet@...rsen.dk>
To:	netdev@...r.kernel.org
Subject: Strange packet drops with heavy firewalling

I have a netfilter-box which is dropping packets. ethtool -S counts
10-20 rx_discards per second on the interface.

The switch does not have flow control enabled; with flow control enabled
the rx_discards turn into tx_on_sent which ultimately cause the same
problem (the load is pretty constant so the switch has to drop the
packets instead).

perf top shows something like:
             5201.00 -  6.7% : _spin_unlock_irqrestore
             4232.00 -  5.5% : finish_task_switch
             3597.00 -  4.6% : tg3_poll	[tg3]
             3257.00 -  4.2% : handle_IRQ_event
             2515.00 -  3.2% : tick_nohz_restart_sched_tick
             1947.00 -  2.5% : nf_ct_tuple_equal
             1927.00 -  2.5% : tg3_start_xmit	[tg3]
             1879.00 -  2.4% : kmem_cache_alloc_node
             1625.00 -  2.1% : tick_nohz_stop_sched_tick
             1619.00 -  2.1% : ipt_do_table
             1595.00 -  2.1% : ip_route_input
             1547.00 -  2.0% : kmem_cache_free
             1474.00 -  1.9% : __alloc_skb
             1424.00 -  1.8% : fget_light
             1391.00 -  1.8% : nf_iterate

The rule set is quite large (more than 4000 rules), but organized so
that each packet only has to traverse a few rules before getting
accepted or rejected.

When the problem started we were using a different server, an old
two-socket 32-bit Xeon with hyperthreading. CPU usage often hit 100% on
one CPU with that server. After replacing the server with a ProLiant
DL160 G5 with a quad-core Xeon (without hyperthreading) the CPU usage
rarely exceeds 10% on any CPU, but the packet loss persists.

We're using the built-in dual Broadcom Corporation NetXtreme BCM5722 Gigabit
Ethernet PCI Express nics, and the kernel is
kernel-2.6.32.9-70.fc12.x86_64 from Fedora. Next step is probably
installing a better ethernet card, perhaps an Intel 82576-based one, so
that we can get multiqueue support.

The traffic is about 300Mbps (twice that if you count both in and out,
like Cisco).

/Benny

--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html