netdev - Re: SO_REUSEPORT?

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Thu, 7 Aug 2008 12:03:59 -0700
From:	Stephen Hemminger <stephen.hemminger@...tta.com>
To:	Rick Jones <rick.jones2@...com>
Cc:	Tom Herbert <therbert@...gle.com>, netdev@...r.kernel.org
Subject: Re: SO_REUSEPORT?

On Thu, 07 Aug 2008 11:17:55 -0700
Rick Jones <rick.jones2@...com> wrote:

> Tom Herbert wrote:
> >>>We are looking at ways to scale TCP listeners.  I think we like is the
> >>>ability to listen on a port from multiple threads (sockets bound to
> >>>same port,  INADDR_ANY, and no interface binding) , which is what
> >>>SO_REUSEPORT would seem to allow.  Has this ever been implemented for
> >>>Linux or is there a good reason not to have it?
> >>
> >>On Linux, SO_REUSEADDR provide most of what SO_REUSEPORT provides on BSD.
> >>
> >>In any case, there is absolutely no point in creating multiple TCP listeners.
> >>Multiple threads can accept() on the same listener - at the same time.
> >>
> > 
> > 
> > We've been doing that, but then on wakeup it would seem that we're at
> > the mercy of scheduling-- basically which ever threads wakes up first
> > will get to process accept queue first.  This seems to bias towards
> > threads running on the same CPU as the wakeup is called, and   so this
> > method doesn't give us an even distribution of new connections across
> > the threads that we'd like.
> 
> How would the presence of multiple TCP LISTEN endpoints change that? 
> You'd then be at the mercy of whatever "scheduling" there was inside the 
> stack.
> 
> If you want to balance the threads, perhaps a dispatch thread, or a 
> virtual one - each thread knows how many connections it is servicing, 
> let them know how many the other threads are servicing, and if a thread 
> has N more connections than the other threads have it not go into 
> accept() that time around.  Might need some tweaking to handle 
> pathological starvation cases like all the other threads are hung I 
> suppose but the basic idea is there.
> 
> rick jones

I suspect thread balancing would actually hurt performance!
You would be better off to have a couple of "hot" threads that are doing
all the work and stay in cache. If you push the work around to all the
threads, you have worst case cache behaviour.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html