netdev - Re: [PATCH 0/5]: soreuseport: Bind multiple sockets to the same port

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CA+mtBx-DRiSg4z5OUvp4=pJxZ=Ki4pt-vz716ubsJx1PHnD7Nw@mail.gmail.com>
Date:	Wed, 16 Jan 2013 10:22:39 -0800
From:	Tom Herbert <therbert@...gle.com>
To:	David Laight <David.Laight@...lab.com>
Cc:	netdev@...r.kernel.org, davem@...emloft.net,
	netdev@...kandruth.co.uk, eric.dumazet@...il.com
Subject: Re: [PATCH 0/5]: soreuseport: Bind multiple sockets to the same port

> Hmmm.... do you need that sort of fairness between the threads?
>
Yes :-)

> If one request takes longer than average to process, then you
> don't want other requests to be delayed when there are other
> idle worker processes.
>
On a heavily loaded server processing thousands of requests/second,
law of large numbers hopefully applies where each connection
represents approximately same unit of work.

> Also having the same thread normally collect a request would
> make it more likely that the required code/data be in the
> cache of the cpu (assuming that the main reason for multiple
> threads is to load balance over multiple cpus, and with the
> threads tied to a single cpu).
>
Right.  Multiple listener sockets also imply that the work on the
connected sockets will be in the same thread or at least dispatched to
thread which is close to the same CPU.  soreuseport moves the start of
siloed processing into kernel.

> If there are a lot of processes sleeping in accept() (on the same
> socket) it might be worth looking at which is actually woken
> when a new connection arrives. If they are sleeping in poll/select
> it is probably more difficult (but not impossible) to avoid waking
> all the processes for every incoming connection.

We had considered solving this within accept.  The problem is that
there's no way to indicate how much work a thread should do via
accept.  For instance, an event loop usually would look like:

while (1) {
    fd = accept();
    process(fd);
}

With multiple threads, the number of accepted sockets in a particular
thread is non-deterministic.  It is even possible that one thread
could end up accepting all the connections, and the others are starved
(wake up but no connection to process.).  Since connections are the
unit of work, this creates imbalance among threads.  There was an
attempt to fix this in user space by sleeping for a while instead of
calling accept on threads for one that have already have a
disproportionate number of connections.  This was unpleasant-- it
needed shared state in user space and provided no granularity.

Tom
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html