linux-kernel - Re: Syslets, Threadlets, generic AIO support, v6

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-Id: <20070531093523.a6a62833.dada1@cosmosbay.com>
Date:	Thu, 31 May 2007 09:35:23 +0200
From:	Eric Dumazet <dada1@...mosbay.com>
To:	Ingo Molnar <mingo@...e.hu>
Cc:	Linus Torvalds <torvalds@...ux-foundation.org>,
	Davide Libenzi <davidel@...ilserver.org>,
	Ulrich Drepper <drepper@...hat.com>,
	Jeff Garzik <jeff@...zik.org>,
	Zach Brown <zach.brown@...cle.com>,
	Linux Kernel Mailing List <linux-kernel@...r.kernel.org>,
	Arjan van de Ven <arjan@...radead.org>,
	Christoph Hellwig <hch@...radead.org>,
	Andrew Morton <akpm@....com.au>,
	Alan Cox <alan@...rguk.ukuu.org.uk>,
	Evgeniy Polyakov <johnpol@....mipt.ru>,
	"David S. Miller" <davem@...emloft.net>,
	Suparna Bhattacharya <suparna@...ibm.com>,
	Jens Axboe <jens.axboe@...cle.com>,
	Thomas Gleixner <tglx@...utronix.de>
Subject: Re: Syslets, Threadlets, generic AIO support, v6

On Thu, 31 May 2007 08:13:03 +0200
Ingo Molnar <mingo@...e.hu> wrote:

> 
> * Linus Torvalds <torvalds@...ux-foundation.org> wrote:
> 
> > > I agree. What would be a good interface to allocate fds in such 
> > > area? We don't want to replicate syscalls, so maybe a special new 
> > > dup function?
> > 
> > I'd do it with something like "newfd = dup2(fd, NONLINEAR_FD)" or 
> > similar, and just have NONLINEAR_FD be some magic value (for example, 
> > make it be 0x40000000 - the bit that says "private, nonlinear" in the 
> > first place).
> > 
> > But what's gotten lost in the current discussion is that we probably 
> > don't actually _need_ such a private space. I'm just saying that if 
> > the *choice* is between memory-mapped interfaces and a private 
> > fd-space, we should probably go for the latter. "Everything is a file" 
> > is the UNIX way, after all. But there's little reason to introduce 
> > private fd's otherwise.
> 
> it's both a flexibility and a speedup thing as well:
> 
> flexibility: for libraries to be able to open files and keep them open 
> comes up regularly. For example currently glibc is quite wasteful in a 
> number of common networking related functions (Ulrich, please correct me 
> if i'm wrong), which could be optimized if glibc could just keep a 
> netlink channel fd open and could poll() it for changes and cache the 
> results if there are no changes (or something like that).
> 
> speedup: i suggested O_ANY 6 years ago as a speedup to Apache - 
> non-linear fds are cheaper to allocate/map:
> 
>   http://www.mail-archive.com/linux-kernel@vger.kernel.org/msg23820.html
> 
> (i definitely remember having written code for that too, but i cannot 
> find that in the archives. hm.) In theory we could avoid _all_ fd-bitmap 
> overhead as well and use a per-process list/pool of struct file buffers 
> plus a maximum-fd field as the 'non-linear fd allocator' (at the price 
> of only deallocating them at process exit time).

Only very few apps need to open more than 100.000 files.

As these files are likely sockets, O_ANY is not a solution.

A trick is to try to keep first 64 handles freed, so that kernel wont consume
too much cpu time and cache in get_unused_fd()

http://lkml.org/lkml/2005/9/15/307

This trick is portable (not linux centric).

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/