netdev - Re: Network performance with small packets

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Wed, 09 Mar 2011 08:21:44 -0800
From:	Shirley Ma <mashirle@...ibm.com>
To:	Tom Lendacky <tahm@...ux.vnet.ibm.com>
Cc:	"Michael S. Tsirkin" <mst@...hat.com>,
	Rusty Russell <rusty@...tcorp.com.au>,
	Krishna Kumar2 <krkumar2@...ibm.com>,
	David Miller <davem@...emloft.net>, kvm@...r.kernel.org,
	netdev@...r.kernel.org, steved@...ibm.com
Subject: Re: Network performance with small packets - continued

On Wed, 2011-03-09 at 10:09 -0600, Tom Lendacky wrote:
> > 
> > > This spread out the kick_notify but still resulted in alot of
> them.  I
> > > decided to build on the delayed Tx buffer freeing and code up an
> > > "ethtool" like coalescing patch in order to delay the kick_notify
> until
> > > there were at least 5 packets on the ring or 2000 usecs, whichever
> > > occurred first.  Here are the
> > > 
> > > results of delaying the kick_notify (average of six runs):
> > >   Txn Rate: 107,106.36 Txn/Sec, Pkt Rate: 212,796 Pkts/Sec
> > >   Exits: 102,587.28 Exits/Sec
> > >   TxCPU: 3.03%  RxCPU: 99.33%
> > > 
> > > About a 23% increase over baseline and about 52% of baremetal.
> > > 
> > > Running the perf command against the guest I noticed almost 19% of
> the
> > > time being spent in _raw_spin_lock.  Enabling lockstat in the
> guest
> > > showed alot of contention in the "irq_desc_lock_class". Pinning
> the
> > > virtio1-input interrupt to a single cpu in the guest and
> re-running the
> > > last test resulted in
> > > 
> > > tremendous gains (average of six runs):
> > >   Txn Rate: 153,696.59 Txn/Sec, Pkt Rate: 305,358 Pkgs/Sec
> > >   Exits: 62,603.37 Exits/Sec
> > >   TxCPU: 3.73%  RxCPU: 98.52%
> > > 
> > > About a 77% increase over baseline and about 74% of baremetal.
> > > 
> > > Vhost is receiving a lot of notifications for packets that are to
> be
> > > transmitted (over 60% of the packets generate a kick_notify).
> Also, it
> > > looks like vhost is sending a lot of notifications for packets it
> has
> > > received before the guest can get scheduled to disable
> notifications and
> > > begin processing the packets
> > 
> > Hmm, is this really what happens to you?  The effect would be that
> guest
> > gets an interrupt while notifications are disabled in guest, right?
> Could
> > you add a counter and check this please?
> 
> The disabling of the interrupt/notifications is done by the guest.  So
> the 
> guest has to get scheduled and handle the notification before it
> disables 
> them.  The vhost_signal routine will keep injecting an interrupt until
> this 
> happens causing the contention in the guest.  I'll try the patches you
> specify 
> below and post the results.  They look like they should take care of
> this 
> issue.

In guest TX path, the guest interrupt should be disabled in the start
since it free_old_xmit_skbs in start_xmit call, it's not necessary to
receive any send completion interrupts to handle free old skbs. Then the
interrupt is only enabled when the netif queue is full. For multiple
streams TCP_RR test, we never hit netif queue full situation, the
cat /proc/interrupts/ send completion interrupts rate is 0, right?

Shirley

--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html