netdev - Re: Possible regression: Packet drops during iptables calls

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <1292340702.5934.5.camel@edumazet-laptop>
Date:	Tue, 14 Dec 2010 16:31:42 +0100
From:	Eric Dumazet <eric.dumazet@...il.com>
To:	Jesper Dangaard Brouer <hawk@...x.dk>
Cc:	Stephen Hemminger <shemminger@...tta.com>,
	netfilter-devel <netfilter-devel@...r.kernel.org>,
	netdev <netdev@...r.kernel.org>
Subject: Re: Possible regression: Packet drops during iptables calls

Le mardi 14 décembre 2010 à 15:46 +0100, Jesper Dangaard Brouer a
écrit :
> I'm experiencing RX packet drops during call to iptables, on my
> production servers.
> 
> Further investigations showed, that its only the CPU executing the
> iptables command that experience packet drops!?  Thus, a quick fix was
> to force the iptables command to run on one of the idle CPUs (This can
> be achieved with the "taskset" command).
> 
> I have a 2x Xeon 5550 CPU system, thus 16 CPUs (with HT enabled).  We
> only use 8 CPUs due to a multiqueue limitation of 8 queues in the
> 1Gbit/s NICs (82576 chips).  CPUs 0 to 7 is assigned for packet
> processing via smp_affinity.
> 
> Can someone explain why the packet drops only occur on the CPU
> executing the iptables command?
> 
> 

It blocks BH

take a look at commits :

24b36f0193467fa727b85b4c004016a8dae999b9
netfilter: {ip,ip6,arp}_tables: dont block bottom half more than
necessary 

001389b9581c13fe5fc357a0f89234f85af4215d
netfilter: {ip,ip6,arp}_tables: avoid lockdep false positive

for attempts to let BH fly ...

Unfortunately, lockdep rules :(


> What can we do to solve this issue?
> 
> 
> I should note that I have a very large ruleset on this machine, and
> the production machine is routing around 800 Mbit/s, in each
> direction.  The issue occurs on a simple iptables rule listing.
> 
> 
> I think (untested) the problem is related to kernel git commit:
> 
>  commit 942e4a2bd680c606af0211e64eb216be2e19bf61
>  Author: Stephen Hemminger <shemminger@...tta.com>
>  Date: Tue Apr 28 22:36:33 2009 -0700
> 
>  netfilter: revised locking for x_tables
> 
>  The x_tables are organized with a table structure and a per-cpu copies
>  of the counters and rules. On older kernels there was a reader/writer
>  lock per table which was a performance bottleneck. In 2.6.30-rc, this
>  was converted to use RCU and the counters/rules which solved the performance
>  problems for do_table but made replacing rules much slower because of
>  the necessary RCU grace period.
> 
>  This version uses a per-cpu set of spinlocks and counters to allow to
>  table processing to proceed without the cache thrashing of a global
>  reader lock and keeps the same performance for table updates.
> 
>  Signed-off-by: Stephen Hemminger <shemminger@...tta.com>
>  Acked-by: Linus Torvalds <torvalds@...ux-foundation.org>
>  Signed-off-by: David S. Miller <davem@...emloft.net>
> 


--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html