Message-ID: <461CF098.3090003@qumranet.com>
Date:	Wed, 11 Apr 2007 17:28:40 +0300
From:	Avi Kivity <avi@...ranet.com>
To:	Rusty Russell <rusty@...tcorp.com.au>
CC:	Ingo Molnar <mingo@...e.hu>, kvm-devel@...ts.sourceforge.net,
	netdev <netdev@...r.kernel.org>
Subject: Re: [kvm-devel] QEMU PIC indirection patch for in-kernel APIC work

Rusty Russell wrote:
> On Wed, 2007-04-11 at 07:26 +0300, Avi Kivity wrote:
>   
>> Nope.  Being async is critical for copyless networking:
>>
>> - in the transmit path, so need to stop the sender (guest) from touching
>> the memory until it's on the wire.  This means 100% of packets sent will
>> be blocked.
>>     
>
> Hi Avi,
>
> 	You keep saying stuff like this, and I keep ignoring it.  OK, I'll
> bite:
>
> 	Why would we try to prevent the sender from altering the packets?
>
>   

To avoid data corruption.

The guest wants to send a packet.  It calls write(), which causes an skb 
to be allocated, data to be copied into it, the entire networking stack 
gets into gear, and the guest-side driver instructs the "device" to send 
the packet.

With async operations, the saga continues like this: the host-side 
driver allocates an skb, get_page()s and attaches the data to the new 
skb, this skb crosses the bridge, trickles into the real ethernet 
device, gets queued there, sent, interrupts fire, triggering async 
completion.  On this completion, we send a virtual interrupt to the 
guest, which tells it to destroy the skb and reclaim the pages attached 
to it.

Without async operations, we don't have a hook to notify the guest when 
to reclaim the skb.  If we do it too soon, the skb can be reclaimed and 
the memory reused before the real device gets to see it, so we end up 
sending data that we did not intend.  The only way to avoid it is to 
copy the data somewhere safe, but that is exactly what we don't want to do.

>> - multiple packets per operation (for interrupt mitigation) (like
>> lio_listio)
>>     
>
> The benefits for interrupt mitigation are less clear to me in a virtual
> environment (scheduling tends to make it happen anyway); I'd want to
> benchmark it.
>
>   

Yes, the guest will probably submit multiple packets in one hypercall.  
It would be nice for the userspace driver to be able to submit them to 
the host kernel in one syscall.

> Some kind of batching to reduce syscall overhead, perhaps, but TSO would
> go a fair way towards that anyway (probably not enough).
>
>   

For some workloads, sure.


>> - scatter/gather packets (iovecs)
>>     
>
> Yes, and this is already present in the tap device.  Anthony suggested a
> slightly nasty hack for multiple sg packets in one writev()/readv, which
> could also give us batching.
>
>   

No need for hacks if we get list aio support one day.

>> - configurable wakeup (by packet count/timeout) for queue management
>>     
>
> I'm not convinced that this is a showstopper, though.
>   

It probably isn't.  It's free with aio though.

>   
>> - hacks (tso)
>>     
>
> I'd usually go for a batch interface over TSO, but if the card we're
> sending to actually does TSO then TSO will probably win.
>   

Sure, if tso helps a regular host then it should help one that happens 
to be running a virtual machine.

>   
>> Most of these can be provided by a combination of the pending aio work,
>> the pending aio/fd integration, and the not-so-pending tap aio work.  As
>> the first two are available as patches and the third is limited to the
>> tap device, it is not unreasonable to try it out.  Maybe it will turn
>> out not to be as difficult as I predicted just a few lines above.
>>     
>
> Indeed, I don't think we're asking for a revolution a-la VJ-style
> channels.  But I'm still itching to get back to that, and this might yet
> provide an excuse 8)
>   

I'll be happy if this can be made to work.  It will make the paravirt 
guest-side driver work in kvm-less setups, which are useful for testing, 
and of course reduction in kernel code is beneficial.  It will be slower 
that in-kernel, but if we get the batching right, perhaps not 
significantly slower.  I'm mostly concerned that this depends on code 
that has eluded merging for such a long time.


-- 
error compiling committee.c: too many arguments to function

-
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Powered by blists - more mailing lists