| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <1271970893.7895.6507.camel@edumazet-laptop>
Date: Thu, 22 Apr 2010 23:14:53 +0200
From: Eric Dumazet <eric.dumazet@...il.com>
To: Jesper Dangaard Brouer <hawk@...u.dk>
Cc: paulmck@...ux.vnet.ibm.com, Patrick McHardy <kaber@...sh.net>,
Changli Gao <xiaosuo@...il.com>, hawk@...x.dk,
Linux Kernel Network Hackers <netdev@...r.kernel.org>,
Netfilter Developers <netfilter-devel@...r.kernel.org>
Subject: Re: DDoS attack causing bad effect on conntrack searches
Le jeudi 22 avril 2010 à 23:03 +0200, Eric Dumazet a écrit :
> Le jeudi 22 avril 2010 à 22:38 +0200, Jesper Dangaard Brouer a écrit :
> > On Thu, 22 Apr 2010, Eric Dumazet wrote:
> >
> > > Le jeudi 22 avril 2010 à 08:51 -0700, Paul E. McKenney a écrit :
> > >> On Thu, Apr 22, 2010 at 04:53:49PM +0200, Eric Dumazet wrote:
> > >>> Le jeudi 22 avril 2010 à 16:36 +0200, Eric Dumazet a écrit :
> > >>>
> > >>> If we can do the 'retry' a 10 times, it means the attacker was really
> > >>> clever enough to inject new packets (new conntracks) at the right
> > >>> moment, in the right hash chain, and this sounds so higly incredible
> > >>> that I cannot believe it at all :)
> > >>
> > >> Or maybe the DoS attack is injecting so many new conntracks that a large
> > >> fraction of the hash chains are being modified at any given time?
> > >>
> >
> > I think its plausable, there is a lot of modification going on.
> > Approx 40.000 deletes/sec and 40.000 inserts/sec.
> > The hash bucket size is 300032, and with 80000 modifications/sec, we are
> > (potentially) changing 26.6% of the hash chains each second.
> >
>
> OK but a lookup last a fraction of a micro second, unless interrupted by
> hard irq.
>
> Probability of a change during a lookup should be very very small.
>
> Note that the scenario for a restart is :
>
> The lookup go through the chain.
> While it is examining one object, this object is deleted.
> The object is re-allocated by another cpu and inserted to a new chain.
>
>
> What exact version of kernel are you running ?
>
> > As can be seen from the graphs:
> > http://people.netfilter.org/hawk/DDoS/2010-04-12__001/list.html
> >
> > Notice that primarily CPU2 is doing the 40k deletes/sec, while CPU1 is
> > caught searching...
> >
> >
> > > maybe hash table has one slot :)
> >
> > Guess I have to reproduce the DoS attack in a testlab (I will first have
> > time Tuesday). So we can determine if its bad hashing or restart of the
> > search loop.
> >
Or very long chains, if attacker managed to find a jhash flaw.
You could add a lookup_restart counter :
include/linux/netfilter/nf_conntrack_common.h | 1 +
net/ipv4/netfilter/nf_conntrack_l3proto_ipv4_compat.c | 7 ++++---
net/netfilter/nf_conntrack_core.c | 4 +++-
net/netfilter/nf_conntrack_standalone.c | 7 ++++---
4 files changed, 12 insertions(+), 7 deletions(-)
diff --git a/include/linux/netfilter/nf_conntrack_common.h b/include/linux/netfilter/nf_conntrack_common.h
index c608677..cf9a8df 100644
--- a/include/linux/netfilter/nf_conntrack_common.h
+++ b/include/linux/netfilter/nf_conntrack_common.h
@@ -113,6 +113,7 @@ struct ip_conntrack_stat {
unsigned int expect_new;
unsigned int expect_create;
unsigned int expect_delete;
+ unsigned int lookup_restart;
};
/* call to create an explicit dependency on nf_conntrack. */
diff --git a/net/ipv4/netfilter/nf_conntrack_l3proto_ipv4_compat.c b/net/ipv4/netfilter/nf_conntrack_l3proto_ipv4_compat.c
index 2fb7b76..95f2227 100644
--- a/net/ipv4/netfilter/nf_conntrack_l3proto_ipv4_compat.c
+++ b/net/ipv4/netfilter/nf_conntrack_l3proto_ipv4_compat.c
@@ -336,12 +336,12 @@ static int ct_cpu_seq_show(struct seq_file *seq, void *v)
const struct ip_conntrack_stat *st = v;
if (v == SEQ_START_TOKEN) {
- seq_printf(seq, "entries searched found new invalid ignore delete delete_list insert insert_failed drop early_drop icmp_error expect_new expect_create expect_delete\n");
+ seq_printf(seq, "entries searched found new invalid ignore delete delete_list insert insert_failed drop early_drop icmp_error expect_new expect_create expect_delete lookup_restart\n");
return 0;
}
seq_printf(seq, "%08x %08x %08x %08x %08x %08x %08x %08x "
- "%08x %08x %08x %08x %08x %08x %08x %08x \n",
+ "%08x %08x %08x %08x %08x %08x %08x %08x %08x\n",
nr_conntracks,
st->searched,
st->found,
@@ -358,7 +358,8 @@ static int ct_cpu_seq_show(struct seq_file *seq, void *v)
st->expect_new,
st->expect_create,
- st->expect_delete
+ st->expect_delete,
+ st->lookup_restart
);
return 0;
}
diff --git a/net/netfilter/nf_conntrack_core.c b/net/netfilter/nf_conntrack_core.c
index 0c9bbe9..68e53f1 100644
--- a/net/netfilter/nf_conntrack_core.c
+++ b/net/netfilter/nf_conntrack_core.c
@@ -319,8 +319,10 @@ begin:
* not the expected one, we must restart lookup.
* We probably met an item that was moved to another chain.
*/
- if (get_nulls_value(n) != hash)
+ if (unlikely(get_nulls_value(n) != hash)) {
+ NF_CT_STAT_INC(net, lookup_restart);
goto begin;
+ }
local_bh_enable();
return NULL;
diff --git a/net/netfilter/nf_conntrack_standalone.c b/net/netfilter/nf_conntrack_standalone.c
index faa8eb3..c8a286e 100644
--- a/net/netfilter/nf_conntrack_standalone.c
+++ b/net/netfilter/nf_conntrack_standalone.c
@@ -252,12 +252,12 @@ static int ct_cpu_seq_show(struct seq_file *seq, void *v)
const struct ip_conntrack_stat *st = v;
if (v == SEQ_START_TOKEN) {
- seq_printf(seq, "entries searched found new invalid ignore delete delete_list insert insert_failed drop early_drop icmp_error expect_new expect_create expect_delete\n");
+ seq_printf(seq, "entries searched found new invalid ignore delete delete_list insert insert_failed drop early_drop icmp_error expect_new expect_create expect_delete lookup_restart\n");
return 0;
}
seq_printf(seq, "%08x %08x %08x %08x %08x %08x %08x %08x "
- "%08x %08x %08x %08x %08x %08x %08x %08x \n",
+ "%08x %08x %08x %08x %08x %08x %08x %08x %08x\n",
nr_conntracks,
st->searched,
st->found,
@@ -274,7 +274,8 @@ static int ct_cpu_seq_show(struct seq_file *seq, void *v)
st->expect_new,
st->expect_create,
- st->expect_delete
+ st->expect_delete,
+ st->lookup_restart
);
return 0;
}
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Powered by blists - more mailing lists