netdev - Re: DDoS attack causing bad effect on conntrack searches

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <q2h412e6f7f1004220613m488c2ee4r6d24a8d1e65997d4@mail.gmail.com>
Date:	Thu, 22 Apr 2010 21:13:48 +0800
From:	Changli Gao <xiaosuo@...il.com>
To:	hawk@...x.dk
Cc:	Eric Dumazet <eric.dumazet@...il.com>,
	Patrick McHardy <kaber@...sh.net>,
	Linux Kernel Network Hackers <netdev@...r.kernel.org>,
	netfilter-devel@...r.kernel.org,
	Paul E McKenney <paulmck@...ux.vnet.ibm.com>
Subject: Re: DDoS attack causing bad effect on conntrack searches

On Thu, Apr 22, 2010 at 8:58 PM, Jesper Dangaard Brouer <hawk@...x.dk> wrote:
>
> At an unnamed ISP, we experienced a DDoS attack against one of our
> customers.  This attack also caused problems for one of our Linux
> based routers.
>
> The attack was "only" generating 300 kpps (packets per sec), which
> usually isn't a problem for this (fairly old) Linux Router.  But the
> conntracking system chocked and reduced pps processing power to
> 40kpps.
>
> I do extensive RRD/graph monitoring of the machines.  The IP conntrack
> searches in the period exploded, to a stunning 700.000 searches per
> sec.
>
> http://people.netfilter.org/hawk/DDoS/2010-04-12__001/conntrack_searches001.png
>
> First I though it might be caused by bad hashing, but after reading
> the kernel code (func: __nf_conntrack_find()), I think its caused by
> the loop restart (goto begin) of the conntrack search, running under
> local_bh_disable().  These RCU changes to conntrack were introduced in
> ea781f19 by Eric Dumazet.
>
> Code: net/netfilter/nf_conntrack_core.c
> Func: __nf_conntrack_find()
>
> struct nf_conntrack_tuple_hash *
> __nf_conntrack_find(struct net *net, const struct nf_conntrack_tuple *tuple)
> {
>        struct nf_conntrack_tuple_hash *h;
>        struct hlist_nulls_node *n;
>        unsigned int hash = hash_conntrack(tuple);
>
>        /* Disable BHs the entire time since we normally need to disable them
>         * at least once for the stats anyway.
>         */
>        local_bh_disable();
> begin:
>        hlist_nulls_for_each_entry_rcu(h, n, &net->ct.hash[hash], hnnode) {
>                if (nf_ct_tuple_equal(tuple, &h->tuple)) {
>                        NF_CT_STAT_INC(net, found);
>                        local_bh_enable();
>                        return h;
>                }
>                NF_CT_STAT_INC(net, searched);
>        }
>        /*
>         * if the nulls value we got at the end of this lookup is
>         * not the expected one, we must restart lookup.
>         * We probably met an item that was moved to another chain.
>         */
>        if (get_nulls_value(n) != hash)
>                goto begin;
>        local_bh_enable();
>

We should add a retry limit there.

-- 
Regards，
Changli Gao(xiaosuo@...il.com)
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html