netdev - Re: ksoftirqd 100% after disabling IPv4 route cache on high pps.

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [day] [month] [year] [list]

Message-ID: <CAJygYd2UYe+96ddV2=q+hB+q3RP+sO=V=wnQi0CJAr_mx-ycwg@mail.gmail.com>
Date:	Wed, 5 Dec 2012 16:34:01 -0800
From:	叶雨飞 <sunyucong@...il.com>
To:	unlisted-recipients:; (no To-header on input)
Cc:	netdev <netdev@...r.kernel.org>
Subject: Re: ksoftirqd 100% after disabling IPv4 route cache on high pps.

Thanks Eric!

Following is only to help someone else who may run into the problem
and searching for options:

the ksoftirqd load is generated by route lookup subrouting. since this
is DDOS traffic we are dealing (more than 10x of our normal PPS) and I
have a fixed number of destinations, I've tested that I can filter
them efficiently in mangle table PREROUTING chain, and use hashlimit
to limit pps to send to the routing subsystem, ksoftirq problem is
gone and no kernel upgraded is needed until the inevitable.

Cheers.

On Wed, Dec 5, 2012 at 4:32 PM, 叶雨飞 <sunyucong@...il.com> wrote:
> Thanks Eric!
>
> Following is only to help someone else who may run into the problem and
> searching for options:
>
> the ksoftirqd load is generated by route lookup subrouting. since this is
> DDOS traffic we are dealing (more than 10x of our normal PPS) and I have a
> fixed number of destinations, I've tested that I can filter them efficiently
> in mangle table PREROUTING chain, and use hashlimit to limit pps to send to
> the routing subsystem, ksoftirq problem is gone and no kernel upgraded is
> needed until the inevitable.
>
> Cheers.
>
>
> On Wed, Dec 5, 2012 at 3:48 PM, Eric Dumazet <eric.dumazet@...il.com> wrote:
>>
>> On Wed, 2012-12-05 at 13:22 -0800, 叶雨飞 wrote:
>> > Hi Eric,
>> >
>> > After disabling the route cache, I found a forwarding performance
>> > problem (inevitable ?). Basically the kernel couldn't keep up under
>> > about 100k pps and ksoftirqd is dominating the CPU. This problem went
>> > away right away if I do
>> >
>> > echo  1 > rt_cache_rebuild_count
>> >
>> > and comes back as soon as i do echo -1 > rt_cache_rebuild_count.
>> >
>> > I then tried to use RPS/RFS to share the load on to mulitple cpus
>> > (since ksoftirqd is only using 1 core, clearly). but that has little
>> > effect.
>> >
>> > Is there some tweaks/patches you recommend?
>>
>> Yes, upgrade your kernel to 3.6 or 3.7, this problem has been solved for
>> good. No more IP route cache and good performance.
>>
>>
>>
>
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html