linux-kernel - Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <45E432D3.8060208@argo.co.il>
Date:	Tue, 27 Feb 2007 15:32:03 +0200
From:	Avi Kivity <avi@...o.co.il>
To:	Ingo Molnar <mingo@...e.hu>
CC:	Theodore Tso <tytso@....edu>,
	Evgeniy Polyakov <johnpol@....mipt.ru>,
	Linus Torvalds <torvalds@...ux-foundation.org>,
	Ulrich Drepper <drepper@...hat.com>,
	linux-kernel@...r.kernel.org,
	Arjan van de Ven <arjan@...radead.org>,
	Christoph Hellwig <hch@...radead.org>,
	Andrew Morton <akpm@....com.au>,
	Alan Cox <alan@...rguk.ukuu.org.uk>,
	Zach Brown <zach.brown@...cle.com>,
	"David S. Miller" <davem@...emloft.net>,
	Suparna Bhattacharya <suparna@...ibm.com>,
	Davide Libenzi <davidel@...ilserver.org>,
	Jens Axboe <jens.axboe@...cle.com>,
	Thomas Gleixner <tglx@...utronix.de>
Subject: Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3

Ingo Molnar wrote:
> * Theodore Tso <tytso@....edu> wrote:
>
>   
>> I think what you are not hearing, and what everyone else is saying 
>> (INCLUDING Linus), is that for most programmers, state machines are 
>> much, much harder to program, understand, and debug compared to 
>> multi-threaded code. [...]
>>     
>
> btw., another crutial thing that i think Evgeniy is missing is that 
> threadlets /enable/ event loops to be used in practice! Right now the 
> epoll/kevent programming model requires a total 100% avoidance of all 
> context-switching in the 'main' event handler context while handling a 
> request. If just 1% of all requests happen to block it might cause a 
> /complete/ breakdown of an event loop's performance - it can easily 
> cause a 10x drop in performance or worse!
>
> So context-switching has to be avoided in 100% of the code that runs 
> while handling requests, file descriptors have to be set to nonblocking 
> (causing extra system calls), and all the syscalls that might return 
> incomplete with either -EINVAL or with a short read/write have to be 
> converted into a state machine. (or in the alternative, user-space 
> threading has to be used, which opens up another hornet's nest)
>
> /That/ is the main inhibiting factor of the measured use of event loops 
> within Linux! It has zero integration capabilities with 'usual' coding 
> techniques - driving the costs of its application up in the sky, and 
> pushing event based servers into niches.
>
>   

Having written such a niche event based server, I can 100% confirm what 
Ingo is saying here.  We had a single process drive I/O to the kernel 
through an event model (based on kernel aio extended with IO_CMD_POLL), 
and user level threads managed by a custom scheduler that managed I/O, 
timeouts, and thread scheduling.

We once considered dropping from a user-level thread model to a state 
machine model, but the effort was astronomical and we wouldn't see the 
rewards until it was all done, so naturally we didn't do it.

> With threadlets the picture changes dramatically: all we have to 
> concentrate on to get the performance of "100% event based servers" is 
> to handle 'most' rescheduling events in the event loop. A 10-20% context 
> switching ratio does not hurt at all. (it causes ~1% of throughput 
> loss.)
>
> Furthermore, even if a particular configuration or module of the server 
> (say Apache) happens to trigger a high rate of scheduling, the 
> performance breakdown model of threadlets is /vastly/ superior to event 
> based servers. The measurements so far have shown that the absolute 
> worst-case threading server performance is at around 60% of that of 
> non-context-switching servers - and even that level is reached 
> gradually, leaving time for action for the server owner. While with 
> fully event based servers there are mostly only two modes of 
> performance: 100% performance and near-0% performance: total breakdown.
>   

Yes.  Threadlets as the default aio solution (easy to use, acceptable 
performance even in worst cases), with specialized solutions where 
applicable (epoll for networking, aio for O_DIRECT disk) look like a 
good mix of performance and sanity.



-- 
error compiling committee.c: too many arguments to function

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/