Date:	Mon, 26 Feb 2007 00:54:31 -0800
From:	Sarah Bailey <saharabeara@...il.com>
To:	Kernel development list <linux-kernel@...r.kernel.org>,
	usb-hacking@...s.cs.pdx.edu,
	USB development list <linux-usb-devel@...ts.sourceforge.net>
Subject: Re: [linux-usb-devel] usbfs2: Why asynchronous I/O?

On Sun, Feb 25, 2007 at 08:53:03AM -0800, David Brownell wrote:
> On Sunday 25 February 2007 12:57 am, Sarah Bailey wrote:
> > I haven't seen any evidence that the kernel-side aio is substantially
> > more efficient than the GNU libc implementation,
> 
> Face it:  spawning a new thread is fundamentally not as lightweight
> as just submitting an aiocb plus an URB.  And spawning ten threads
> costs a *LOT* more than submitting ten aiocbs and URBs.  (Count just
> the 4KB stacks associated with each thread, vs memory consumed by
> using AIO ... and remember to count the scheduling overheads.)

Yes, spawning a new thread is costly.  However, if someone writes their
own thread-based program and allocates the threads from a pool, that
argument is irrelevant.  Even with fibrils, you have a stack and
scheduling overhead.  With kernel AIO, you have also have some memory
overhead, and you also have context switch overhead when you call
kick_iocb or aio_complete.

Can someone point me at hard evidence one way or another?

> > so it seems like it would be better to leave the complexity in
> > userspace. 
> 
> Thing is, the kernel *already* has URBs.  And the concept behind them
> maps one-to-one onto AIOCBs.  All the kernel needs to expose is:
> mechanisms to submit (and probably cancel) async requests, then collect
> the responses when they're done.

It seems to me that you're arguing that URBs and AIOCBs go together on
the basis that they are both asynchronous and both have some sort of
completion function.  Just because two things are alike doesn't mean
that it's better to use them together.

> You're right that associating a thread with an URB is complexity.

That's not what I said.  I don't think that the libc implementation is
more complex than the kernel AIO implementation.  The libc code is
smaller than the kernel AIO (the core code in aio_misc.c is 721 lines,
while the kernel AIO core code is 1769 lines).  The libc code is simpler
and easier to understand, and therefore may be less error prone.

> > I also doubt that 
> > most userspace application writers know they aren't getting kernel-side
> > aio when they use aio_read(3) and aio_write(3).  Why implement something
> > that isn't going to be used?
> 
> I can't much help application writers that don't bother to read the
> relevant documentation (after it's been pointed out to them).

Where is this documentation?  There's a man page on io_submit, etc., but
how would an application writer know to look for it?  I've grepped
through the Documentation directory, and I didn't find anything that
looked relevant. 

> The gap between POSIX AIO and kernel AIO has been an ongoing problem.  This
> syslet/fibril/yadda-yadda stuff is just the latest spin.

Do you think that fibrils will replace the kernel AIO?  It seems like a
logicial conclusion, but does the kernel AIO give you anything more?
For example, I think it currently allows drivers to guarantee that the
requests will be queued in the order they are submitted.  GNU libc and
fibrils can't make that guarantee because the operations may block
before the request enters the hardware queue.  

On Sun, Feb 25, 2007 at 09:33:02AM -0800, Eric Blossom wrote:
> On Sun, Feb 25, 2007 at 08:53:03AM -0800, David Brownell wrote:
> > Without it, how would you achieve I/O overlap, and thus maximize the
> > I/O throughput accessible when streaming data to/from userspace?  The
> > description you gave of glibc aio support sure sounded to me like it
> > was incapable of achieving much in the way of I/O overlap.
> 
> I heartily agree with this sentiment.  The only way we've been able to
> sustain 32MB/s read and write bulk transfers across the USB from user
> space is by using the usbfs USBDEVFS_SUBMITURB ioctl.  We typically
> run with 8 to 16 outstanding requests, each around 4kB.

GNU libc can overlap I/O.  It implements the userland equivalent of
fibrils.  No one is arguing that we don't need I/O overlap. :)

> FWIW, the peripheral we're using is the Universal Software Radio
> Peripheral (USRP).  It's basically an FX2, FPGA, ADC's and DAC's.
>
> The code we use for this is in the GNU Radio code base,
> http://gnuradio.org/trac/wiki
> 
> The relevant part is usrp/host/lib/fusb*.{h,cc}

Yes, a sane interface to the USRP was one of the main motivations for
the new USB filesystem.  It remains to be seen whether we need a
non-standard interface like io_submit, or whether threads with blocking
I/O is fast enough and has the right semantics.

Sarah Bailey

Download attachment "signature.asc" of type "application/pgp-signature" (190 bytes)

Powered by blists - more mailing lists