netdev - Re: ipsec smp scalability and cpu use fairness (softirqs)

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <20130820093912.3dbb4377@vostro>
Date:	Tue, 20 Aug 2013 09:39:12 +0300
From:	Timo Teras <timo.teras@....fi>
To:	Steffen Klassert <steffen.klassert@...unet.com>
Cc:	Andrew Collins <bsderandrew@...il.com>, netdev@...r.kernel.org
Subject: Re: ipsec smp scalability and cpu use fairness (softirqs)

On Tue, 20 Aug 2013 08:19:14 +0200
Steffen Klassert <steffen.klassert@...unet.com> wrote:

> On Tue, Aug 13, 2013 at 03:41:02PM +0300, Timo Teras wrote:
> > On Tue, 13 Aug 2013 13:56:52 +0200
> > Steffen Klassert <steffen.klassert@...unet.com> wrote:
> > 
> > > 
> > > I'll do some tests with current net-next on my own tomorrow and
> > > let you know about the results.
> > 
> > Yes, I've got pcrypt there. Apparently I had some of the cpu
> > bindings not right, so now it's looking a lot better. But it seems
> > that ksoftirqd on one of the CPUs becomes first bottleneck. I'll
> > try to figure out why.
> > 
> > Thanks on all the info so far, will continue experimenting here too.
> 
> Here are the promised test results:
> 
> I used my test boxes with two nodes (Intel Xeon X5550  @ 2.67GHz) and
> all cores utilized (16 logical cores). I did iperf box to box IPsec
> tunnel tests with the crypto algorithm:
> 
> pcrypt(authenc(hmac(sha1-ssse3),cbc(aes-asm)))
> 
> Throughput is at 1.70 Gbits/sec.
> 
> Same test without pcrypt, i.e. crypto algorithm:
> 
> authenc(hmac(sha1-ssse3),cbc(aes-asm))
> 
> Throughput is at 560 Mbits/sec.
> 
> Unfortunately I can't do forwarding tests, I have only two 10 Gbit
> NICs. Would be nice if I could get forwarding test results from
> somewhere.

I got basically the same results. (Managed to get 2.5 Gbit/s after some
cpumask experimenting.)

At this point it seems that one core cpu peaks at 100% softirq. It
seems to be the nic rx softirq. I am curious why it takes so much cpu,
because plain tcp at 10Gbit/s does not take much cpu at all. So even
though pcrypt is used, it seems it adds considerable overhead in
softirq rx path still. I wonder if it's the pcrypt do parallel
overhead or some generic ipsec/gre overhead; perhaps some locking thing.
I should profile it.

Thanks.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html