Date:	Mon,  9 Feb 2015 03:39:19 -0500
From:	Jason Wang <jasowang@...hat.com>
To:	netdev@...r.kernel.org, linux-kernel@...r.kernel.org,
	virtualization@...ts.linux-foundation.org, mst@...hat.com
Cc:	pagupta@...hat.com, Jason Wang <jasowang@...hat.com>
Subject: [PATCH RFC v5 net-next 0/6] enable tx interrupts for virtio-net

Hi:

This is a new version of trying to enable tx interrupts for
virtio-net.

We used to try to avoid tx interrupts and orphan packets before
transmission for virtio-net. This breaks socket accounting and can
lead serveral other side effects e.g:

- Several other functions which depends on socket accounting can not
  work  correctly (e.g  TCP Small Queue)
- No tx completion which make BQL or packet generator can not work
  correctly.

This series tries to solve the issue by enabling tx interrupts. To
minize the performance impacts of this, several optimizations were
used:

- In guest side, try to use delayed callbacks as much as possible.
- In host side, try to use interrupt coalescing for reduce tx
  interrupts. About 10% - 15% performance were improved with this.

Perforamnce test shows:
- Very few regression (less than 5%) were noticed TCP_RR on 1 vcpu 1
  queue guest.
- CPU utilization is increased in some cases.
- All other cases, tx interrupts can perform equal or better than
  orphaning (much more obvious when testing small packet transmission
  as a byproduct since socket accounting works for TCP packet).

Changes from RFCv4:
- fix the virtqueue_enable_cb_delayed() return value when only 1
  buffer is pending.
- try to disable callbacks by publish event index in
  virtqueue_disable_cb(). Tests shows about 2% - 3% improvement on
  multiple sessions of TCP_RR.
- Revert some of Micahel's tweaks from RFC v1 (see patch 3 for
  details).
- use netif_wake_subqueue() instead of netif_start_subqueue() in
  free_old_xmit_skbs(), since it may be called in tx napi.
- in start_xmit(), try to enable the callback only when current skb is
  the last in the list or tx has already been stopped. This avoid the
  callbacks enabling in heavy load.
- return ns instead of us in vhost_net_check_coalesce_and_signal()
- measure the time interval of real interrupts instead of calls to
  vhost_signal()
- drop bql from the series since it does not affact performance from
  the test result.
Changes from RFC V3:
- Don't free tx packets in ndo_start_xmit()
- Add interrupt coalescing support for virtio-net
Changes from RFC v2:
- clean up code, address issues raised by Jason
Changes from RFC v1:
- address comments by Jason Wang, use delayed cb everywhere
- rebased Jason's patch on top of mine and include it (with some
tweaks)

Test result:

Environment:
Host: net-next.git
Guest: net-next.git
Qemu: qemu.git
CPU: Two Intel(R) Xeon(R) CPU E5620  @ 2.40GHz
Card: Two Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network
Connection (rev 01) connected back to back
Coalescing in host: rx-usecs: 1
Coalescing in guest:
- none if no tx interrupt
- tx-frames 8 tx-usecs 64
Zerocopy disabled for vhost_net

1 VCPU 1 queue:

Guest TX:
sessions/sizes/+-throughput%/+-cpu%/+-per_cpu_throughput%/
1/64/+41.2/-2.6/+45.1/
2/64/+40.8/-2.7/+44.7/
4/64/+38.5/-2.8/+42.5/
8/64/+36.1/-2.1/+38.9/
1/256/+158.5/-5.0/+172.3/
2/256/+155.3/-4.7/+167.9/
4/256/+156.4/-5.3/+170.6/
8/256/+155.4/-4.9/+168.6/
1/512/+276.5/-3.0/+288.3/
2/512/+280.1/-3.3/+293.1/
4/512/+276.1/-2.7/+286.7/
8/512/+267.1/-3.2/+279.1/
1/1024/+344.0/-6.9/+376.9/
2/1024/+362.3/-2.2/+372.8/
4/1024/+370.4/-1.1/+375.8/
8/1024/+374.0/+0.6/+371.1/
1/4096/+15.1/-30.4/+65.5/
2/4096/+6.3/-32.9/+58.4/
4/4096/+6.9/-32.4/+58.3/
8/4096/+6.7/-26.4/+45.0/
1/16384/-0.6/-4.6/+4.2/
2/16384/+0.0/-5.0/+5.4/
4/16384/+0.0/-2.5/+2.6/
8/16384/-0.0/-13.8/+13.2/
1/65535/-0.5/-0.5/0.0/
2/65535/-0.0/+4.0/-3.9/
4/65535/-0.0/+9.4/-8.6/
8/65535/+0.1/+8.6/-7.8/

TCP_RR:
sessions/sizes/+-throughput%/+-cpu%/+-per_cpu_throughput%/
1/1/-2.4/+16.3/-16.6/
50/1/-3.4/-0.2/-3.2/
1/64/+3.3/+0.7/+2.7/
50/64/-3.6/+0.2/-3.8/
1/256/-0.4/+16.6/-14.7/
50/256/-6.1/+0.2/-6.2/

Guest RX:
sessions/sizes/+-throughput%/+-cpu%/+-per_cpu_throughput%/
1/64/+17.3/+4.9/+10.0/
2/64/+13.8/+1.1/+12.3/
4/64/+3.8/+1.0/+2.7/
8/64/+1.4/+2.1/-0.7
1/256/+24.5/-1.3/+19.1/
2/256/-3.9/+3.8/-7.5/
4/256/-1.3/+1.5/-2.8/
8/256/+1.5/+1.7/-0.2/
1/512/+2.9/+12.7/-8.6/
2/512/-0.0/+3.5/-3.4/
4/512/-1.4/+2.6/-4.0/
8/512/-1.3/+3.1/-4.2/
1/1024/-0.2/+6.0/-5.9/
2/1024/+0.0/+3.1/-3.1/
4/1024/+0.1/+0.9/-0.7/
8/1024/-1.1/-4.1/+3.2/
1/4096/-0.6/+4.0/-4.4/
2/4096/-0.0/+3.6/-3.4/
4/4096/+0.7/+3.8/-3.1/
8/4096/-0.5/+2.4/-2.9/
1/16384/-1.1/+4.9/-5.7/
2/16384/-0.0/+2.4/-2.4/
4/16384/-0.1/+4.7/-4.6/
8/16384/+1.3/-2.0/+3.2/
1/65535/+1.9/+6.0/-3.8/
2/65535/+0.0/+3.3/-3.2/
4/65535/+0.1/+5.1/-4.7/
8/65535/-0.7/+8.0/-8.5/

4 VCPU 1 queue:

Guest TX:
sessions/sizes/+-throughput%/+-cpu%/+-per_cpu_throughput%/
1/64/+45.9/+2.7/+41.9/
2/64/+28.8/+0.2/+28.4/
4/64/+11.2/-0.6/+11.9/
8/64/+10.0/-0.5/+10.6/
1/256/+162.4/+1.0/+160.1/
2/256/+99.7/-0.2/+99.9/
4/256/+10.2/-11.3/+24.2/
8/256/-0.0/-13.8/+16.0/
1/512/+232.4/-0.2/+232.8/
2/512/+105.9/-9.9/+128.5/
4/512/+6.7/-18.8/+31.4/
8/512/+0.0/-21.0/+26.6/
1/1024/+285.1/-2.9/+296.9/
2/1024/+99.3/-22.4/+156.8/
4/1024/+3.7/-24.0/+36.4/
8/1024/+0.1/-23.9/+31.6/
1/4096/+35.9/-17.4/+64.6/
2/4096/+11.4/-27.4/+53.4/
4/4096/+0.1/-36.0/+56.3/
8/4096/+0.0/-34.9/+53.7/
1/16384/-1.8/-9.5/+8.5/
2/16384/+0.0/-2.3/+2.4/
4/16384/+0.0/-3.0/+3.1/
8/16384/+0.1/-0.3/+0.4/
1/65535/-1.1/+3.1/-4.0/
2/65535/+0.0/+5.4/-5.1/
4/65535/+0.0/+7.2/-6.7/
8/65535/-0.0/+7.1/-6.7/

TCP_RR:
sessions/sizes/+-throughput%/+-cpu%/+-per_cpu_throughput%/
1/1/-0.5/+11.7/-11.0/
50/1/-1.9/+2.2/-4.1/
1/64/-0.9/+15.5/-14.3/
50/64/-3.3/+1.3/-4.5/
1/256/+0.6/+5.7/-5.5/
50/256/-1.6/+2.3/-3.8/

Guest RX:
sessions/sizes/+-throughput%/+-cpu%/+-per_cpu_throughput%/
1/64/+12.2/+10.7/+1.3/
2/64/-3.0/+1.9/-4.9/
4/64/+0.9/+0.6/+0.2/
8/64/-2.8/+11.9/-13.6/
1/256/+3.6/-1.8/+10.7/
2/256/+1.9/-0.8/+2.8/
4/256/+0.1/+2.1/-1.9/
8/256/+1.1/+4.3/-2.1/
1/512/+1.5/+1.8/-0.9/
2/512/-0.1/+5.5/-5.3/
4/512/+0.5/+12.9/-11.7/
8/512/+5.5/+10.2/-4.0/
1/1024/+2.6/+6.8/-4.0/
2/1024/-0.0/+3.3/-3.3/
4/1024/+0.1/+5.1/-4.7/
8/1024/+3.6/-2.9/+6.5/
1/4096/+1.0/+7.0/-5.7/
2/4096/-0.0/+1.7/-1.5/
4/4096/+0.1/+3.6/-3.4/
8/4096/-1.0/+2.6/-4.0/
1/16384/+0.6/+3.3/-2.7/
2/16384/+0.0/-2.9/+3.3/
4/16384/+1.5/+1.3/+0.7/
8/16384/+0.5/+8.5/-6.9/
1/65535/+0.5/+2.4/-2.0/
2/65535/+0.0/-1.2/+1.5/
4/65535/+0.1/-4.5/+4.9/
8/65535/+1.4/+7.1/-5.7/

4 VCPU 4 queues:

Guest TX:
sessions/sizes/+-throughput%/+-cpu%/+-per_cpu_throughput%/
1/64/+43.6/-7.9/+56.5/
2/64/+41.7/-3.8/+47.3/
4/64/+43.7/-3.7/+49.3/
8/64/+40.2/-3.5/+45.3/
1/256/+155.9/-2.9/+164.2/
2/256/+160.6/-3.2/+169.1/
4/256/+77.5/-23.2/+131.4/
8/256/+82.5/-21.1/+132.4/
1/512/+214.7/-0.7/+217.9/
2/512/+125.2/-10.1/+151.8/
4/512/+30.4/-37.0/+107.6/
8/512/+0.8/-17.1/+22.9/
1/1024/+250.0/+3.1/+239.5/
2/1024/+83.6/-19.7/+129.0/
4/1024/+18.0/-39.4/+95.2/
8/1024/+1.9/-20.5/+30.3/
1/4096/+26.6/-10.9/+42.1/
2/4096/+0.9/-2.3/+3.9/
4/4096/+0.9/-12.6/+16.8/
8/4096/-0.4/-29.4/+42.1/
1/16384/+12.0/-3.1/+15.9/
2/16384/-2.8/-2.1/+0.5/
4/16384/-1.6/-15.3/+18.2/
8/16384/+0.5/-17.9/+34.5/
1/65535/+12.2/-2.9/+15.8/
2/65535/-4.3/+4.7/-8.6/
4/65535/+1.1/-9.8/+15.9/
8/65535/-1.3/-21.8/+34.1/

TCP_RR:
sessions/sizes/+-throughput%/+-cpu%/+-per_cpu_throughput%/
1/1/+5.4/-0.4/+6.6/
50/1/+6.9/+2.8/+3.9/
1/64/+7.8/+8.4/-1.5/
50/64/+11.1/+0.8/+10.4/
1/256/+3.9/-2.9/+10.3/
50/256/+17.4/+2.7/+14.1/

Guest RX:
sessions/sizes/+-throughput%/+-cpu%/+-per_cpu_throughput%/
1/64/+3.9/+3.8/+9.8/
2/64/-11.0/+11.2/-20.4/
4/64/-5.2/+14.4/-16.6/
8/64/-8.2/+17.2/-21.6/
1/256/+37.3/-32.0/+47.7/
2/256/+16.2/-9.7/+27.7/
4/256/+0.7/+1.5/-0.9/
8/256/+2.1/+8.6/-5.8/
1/512/+2.7/-2.2/+5.5/
2/512/+1.5/+9.4/-7.4/
4/512/-0.0/+10.7/-9.6/
8/512/+4.0/+11.6/-7.0/
1/1024/-3.1/-5.7/+3.2/
2/1024/+0.0/+5.5/-4.9/
4/1024/-0.2/+7.0/-6.9/
8/1024/-1.5/+1.3/-2.6/
1/4096/-1.0/-10.5/+10.6/
2/4096/+0.0/-0.3/+1.0/
4/4096/+0.1/+15.1/-13.0/
8/4096/-1.3/+15.5/-14.6/
1/16384/-0.5/-11.6/+12.5/
2/16384/+0.0/+6.1/-5.9/
4/16384/-1.7/+24.2/-20.5/
8/16384/-2.2/+2.8/-2.6/
1/65535/-0.9/-12.4/+13.0/
2/65535/+0.1/+4.7/-4.3/
4/65535/0.0/+10.6/-9.5/
8/65535/-1.8/+9.4/-10.1/

I've also done test on mlx4, almos the same trends. So not post here.

Thanks

Jason Wang (6):
  virtio_ring: fix virtqueue_enable_cb() when only 1 buffers were
    pending
  virtio_ring: try to disable event index callbacks in
    virtqueue_disable_cb()
  virtio_net: enable tx interrupt
  virtio-net: add basic interrupt coalescing support
  vhost: let vhost_signal() returns whether signalled
  vhost_net: interrupt coalescing support

 drivers/net/virtio_net.c        | 203 +++++++++++++++++++++++++++++++---------
 drivers/vhost/net.c             | 199 +++++++++++++++++++++++++++++++++++++--
 drivers/vhost/vhost.c           |   7 +-
 drivers/vhost/vhost.h           |   2 +-
 drivers/virtio/virtio_ring.c    |   7 +-
 include/uapi/linux/vhost.h      |  12 +++
 include/uapi/linux/virtio_net.h |  12 +++
 7 files changed, 383 insertions(+), 59 deletions(-)

-- 
1.8.3.1

--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Powered by blists - more mailing lists