netdev - Re: SO_REUSEPORT - can it be done in kernel?

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <1298977984.3284.15.camel@edumazet-laptop>
Date:	Tue, 01 Mar 2011 12:13:04 +0100
From:	Eric Dumazet <eric.dumazet@...il.com>
To:	Thomas Graf <tgraf@...radead.org>
Cc:	Herbert Xu <herbert@...dor.apana.org.au>,
	David Miller <davem@...emloft.net>, rick.jones2@...com,
	therbert@...gle.com, wsommerfeld@...gle.com,
	daniel.baluta@...il.com, netdev@...r.kernel.org
Subject: Re: SO_REUSEPORT - can it be done in kernel?

Le mardi 01 mars 2011 à 06:07 -0500, Thomas Graf a écrit :
> On Tue, Mar 01, 2011 at 11:33:22AM +0100, Eric Dumazet wrote:
> > > I retested with net-2.6 w/o Herbert's patch:
> > > 
> > > named -n 1: 36.9kqps
> > > named:      16.2kqps
> > 
> > Thats better ;)
> > 
> > You could do "cat /proc/net/udp" to check if drops occur on port 53
> > socket (last column)
> > 
> > But maybe your queryperf is limited to few queries in flight (default is
> > 20 per queryperf instance) 
> 
> I tried -q 10, 20, 30, 50, 100. Starting with 20 I see drops, at q=50
> queryperf reports 99% drops.
> 
> I also tested again on the Intel machine that did ~650kqps using SO_REUSEPORT.
> 
> net-2.6: 106.3kqps, 101.2kqps
> net-2.6 lockless udp: 251.7kqps, 250.4kqps
> 
> I see drops in both test cases occur so I believe the rate supplied by the
> clients is sufficient.
> 
> The difference is obvious when looking at top and mpstat:
> 
> UDP lockless (250kqps):
> 
> Cpu0  : 46.4%us, 28.8%sy,  0.0%ni, 24.8%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
> Cpu1  :  2.0%us,  1.3%sy,  0.0%ni,  3.0%id,  0.0%wa,  0.0%hi, 93.6%si,  0.0%st
> Cpu2  : 45.9%us, 28.2%sy,  0.0%ni, 25.9%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
> Cpu3  : 50.0%us, 21.6%sy,  0.0%ni, 28.4%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
> Cpu4  : 45.4%us, 27.8%sy,  0.0%ni, 26.5%id,  0.0%wa,  0.0%hi,  0.3%si,  0.0%st
> Cpu5  : 50.7%us, 23.2%sy,  0.0%ni, 26.1%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
> Cpu6  : 45.2%us, 28.9%sy,  0.0%ni, 25.9%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
> Cpu7  : 50.5%us, 22.0%sy,  0.0%ni, 27.5%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
> Cpu8  : 45.3%us, 29.3%sy,  0.0%ni, 25.4%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
> Cpu9  : 50.8%us, 20.8%sy,  0.0%ni, 28.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
> Cpu10 : 46.1%us, 27.8%sy,  0.0%ni, 26.1%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
> Cpu11 : 27.2%us, 11.3%sy,  0.0%ni,  3.3%id,  0.0%wa,  0.0%hi, 58.1%si,  0.0%st

Its a bit strange two cpus spend time in softirq, unless you have two
queryperf sources, and a multiqueue NIC, or maybe you use two NICS ?

Mind use "perf top -C 1" and "perf top -C 11" to check what these cpus
do ?



--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html