| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-Id: <1417490120.4405.2@smtp.corp.redhat.com> Date: Tue, 02 Dec 2014 03:23:20 +0008 From: Jason Wang <jasowang@...hat.com> To: "Michael S. Tsirkin" <mst@...hat.com> Cc: virtualization@...ts.linux-foundation.org, netdev@...r.kernel.org, linux-kernel@...r.kernel.org, davem@...emloft.net, pagupta@...hat.com Subject: Re: [PATCH RFC v4 net-next 0/5] virtio_net: enabling tx interrupts On Mon, Dec 1, 2014 at 6:42 PM, Michael S. Tsirkin <mst@...hat.com> wrote: > On Mon, Dec 01, 2014 at 06:17:03PM +0800, Jason Wang wrote: >> Hello: >> >> We used to orphan packets before transmission for virtio-net. This >> breaks >> socket accounting and can lead serveral functions won't work, e.g: >> >> - Byte Queue Limit depends on tx completion nofication to work. >> - Packet Generator depends on tx completion nofication for the last >> transmitted packet to complete. >> - TCP Small Queue depends on proper accounting of sk_wmem_alloc to >> work. >> >> This series tries to solve the issue by enabling tx interrupts. To >> minize >> the performance impacts of this, several optimizations were used: >> >> - In guest side, virtqueue_enable_cb_delayed() was used to delay >> the tx >> interrupt untile 3/4 pending packets were sent. >> - In host side, interrupt coalescing were used to reduce tx >> interrupts. >> >> Performance test results[1] (tx-frames 16 tx-usecs 16) shows: >> >> - For guest receiving. No obvious regression on throughput were >> noticed. More cpu utilization were noticed in few cases. >> - For guest transmission. Very huge improvement on througput for >> small >> packet transmission were noticed. This is expected since TSQ and >> other >> optimization for small packet transmission work after tx >> interrupt. But >> will use more cpu for large packets. >> - For TCP_RR, regression (10% on transaction rate and cpu >> utilization) were >> found. Tx interrupt won't help but cause overhead in this case. >> Using >> more aggressive coalescing parameters may help to reduce the >> regression. > > OK, you do have posted coalescing patches - does it help any? Helps a lot. For RX, it saves about 5% - 10% cpu. (reduce 60%-90% tx intrs) For small packet TX, it increases 33% - 245% throughput. (reduce about 60% inters) For TCP_RR, it increase the 3%-10% trans.rate. (reduce 40%-80% tx intrs) > > I'm not sure the regression is due to interrupts. > It would make sense for CPU but why would it > hurt transaction rate? Anyway guest need to take some cycles to handle tx interrupts. And transaction rate does increase if we coalesces more tx interurpts. > > > It's possible that we are deferring kicks too much due to BQL. > > As an experiment: do we get any of it back if we do > - if (kick || netif_xmit_stopped(txq)) > - virtqueue_kick(sq->vq); > + virtqueue_kick(sq->vq); > ? I will try, but during TCP_RR, at most 1 packets were pending, I suspect if BQL can help in this case. > > > If yes, we can just kick e.g. periodically, e.g. after queueing each > X bytes. Okay, let me try to see if this help. > >> Changes from RFC V3: >> - Don't free tx packets in ndo_start_xmit() >> - Add interrupt coalescing support for virtio-net >> Changes from RFC v2: >> - clean up code, address issues raised by Jason >> Changes from RFC v1: >> - address comments by Jason Wang, use delayed cb everywhere >> - rebased Jason's patch on top of mine and include it (with some >> tweaks) >> >> Please reivew. Comments were more than welcomed. >> >> [1] Performance Test result: >> >> Environment: >> - Two Intel(R) Xeon(R) CPU E5620 @ 2.40GHz machines connected back >> to back >> with 82599ES cards. >> - Both host and guest were net-next.git plus the patch >> - Coalescing parameters for the card: >> Adaptive RX: off TX: off >> rx-usecs: 1 >> rx-frames: 0 >> tx-usecs: 0 >> tx-frames: 0 >> - Vhost_net was enabled and zerocopy was disabled >> - Tests was done by netperf-2.6 >> - Guest has 2 vcpus with single queue virtio-net >> >> Results: >> - Numbers of square brackets are whose significance is grater than >> 95% >> >> Guest RX: >> >> size/sessions/+throughput/+cpu/+per_cpu_throughput/ >> 64/1/+2.0326/[+6.2807%]/-3.9970%/ >> 64/2/-0.2104%/[+3.2012%]/[-3.3058%]/ >> 64/4/+1.5956%/+2.2451%/-0.6353%/ >> 64/8/+1.1732%/+3.5123%/-2.2598%/ >> 256/1/+3.7619%/[+5.8117%]/-1.9372%/ >> 256/2/-0.0661%/[+3.2511%]/-3.2127%/ >> 256/4/+1.1435%/[-8.1842%]/[+10.1591%]/ >> 256/8/[+2.2447%]/[+6.2044%]/[-3.7283%]/ >> 1024/1/+9.1479%/[+12.0997%]/[-2.6332%]/ >> 1024/2/[-17.3341%]/[+0.0000%]/[-17.3341%]/ >> 1024/4/[-0.6284%]/-1.0376%/+0.4135%/ >> 1024/8/+1.1444%/-1.6069%/+2.7961%/ >> 4096/1/+0.0401%/-0.5993%/+0.6433%/ >> 4096/2/[-0.5894%]/-2.2071%/+1.6542%/ >> 4096/4/[-0.5560%]/-1.4969%/+0.9553%/ >> 4096/8/-0.3362%/+2.7086%/-2.9645%/ >> 16384/1/-0.0285%/+0.7247%/-0.7478%/ >> 16384/2/-0.5286%/+0.3287%/-0.8545%/ >> 16384/4/-0.3297%/-2.0543%/+1.7608%/ >> 16384/8/+1.0932%/+4.0253%/-2.8187%/ >> 65535/1/+0.0003%/-0.1502%/+0.1508%/ >> 65535/2/[-0.6065%]/+0.2309%/-0.8355%/ >> 65535/4/[-0.6861%]/[+3.9451%]/[-4.4554%]/ >> 65535/8/+1.8359%/+3.1590%/-1.2825%/ >> >> Guest RX: >> size/sessions/+throughput/+cpu/+per_cpu_throughput/ >> 64/1/[+65.0961%]/[-8.6807%]/[+80.7900%]/ >> 64/2/[+6.0288%]/[-2.2823%]/[+8.5052%]/ >> 64/4/[+5.9038%]/[-2.1834%]/[+8.2677%]/ >> 64/8/[+5.4154%]/[-2.1804%]/[+7.7651%]/ >> 256/1/[+184.6462%]/[+4.8906%]/[+171.3742%]/ >> 256/2/[+46.0731%]/[-8.9626%]/[+60.4539%]/ >> 256/4/[+45.8547%]/[-8.3027%]/[+59.0612%]/ >> 256/8/[+45.3486%]/[-8.4024%]/[+58.6817%]/ >> 1024/1/[+432.5372%]/[+3.9566%]/[+412.2689%]/ >> 1024/2/[-1.4207%]/[-23.6426%]/[+29.1025%]/ >> 1024/4/-0.1003%/[-13.6416%]/[+15.6804%]/ >> 1024/8/[+0.2200%]/[+2.0634%]/[-1.8061%]/ >> 4096/1/[+18.4835%]/[-46.1508%]/[+120.0283%]/ >> 4096/2/+0.1770%/[-26.2780%]/[+35.8848%]/ >> 4096/4/-0.1012%/-0.7353%/+0.6388%/ >> 4096/8/-0.6091%/+1.4159%/-1.9968%/ >> 16384/1/-0.0424%/[+11.9373%]/[-10.7021%]/ >> 16384/2/+0.0482%/+2.4685%/-2.3620%/ >> 16384/4/+0.0840%/[+5.3587%]/[-5.0064%]/ >> 16384/8/+0.0048%/[+5.0176%]/[-4.7733%]/ >> 65535/1/-0.0095%/[+10.9408%]/[-9.8705%]/ >> 65535/2/+0.1515%/[+8.1709%]/[-7.4137%]/ >> 65535/4/+0.0203%/[+5.4316%]/[-5.1325%]/ >> 65535/8/+0.1427%/[+6.2753%]/[-5.7705%]/ >> >> size/sessions/+trans.rate/+cpu/+per_cpu_trans.rate/ >> 64/1/+0.2346%/[+11.5080%]/[-10.1099%]/ >> 64/25/[-10.7893%]/-0.5791%/[-10.2697%]/ >> 64/50/[-11.5997%]/-0.3429%/[-11.2956%]/ >> 256/1/+0.7219%/[+13.2374%]/[-11.0524%]/ >> 256/25/-6.9567%/+0.8887%/[-7.7763%]/ >> 256/50/[-4.8814%]/-0.0338%/[-4.8492%]/ >> 4096/1/-1.6061%/-0.7561%/-0.8565%/ >> 4096/25/[+2.2120%]/[+1.0839%]/+1.1161%/ >> 4096/50/[+5.6180%]/[+3.2116%]/[+2.3315%]/ >> >> Jason Wang (4): >> virtio_net: enable tx interrupt >> virtio-net: optimize free_old_xmit_skbs stats >> virtio-net: add basic interrupt coalescing support >> vhost_net: interrupt coalescing support >> >> Michael S. Tsirkin (1): >> virtio_net: bql >> >> drivers/net/virtio_net.c | 211 >> ++++++++++++++++++++++++++++++++-------- >> drivers/vhost/net.c | 200 >> +++++++++++++++++++++++++++++++++++-- >> include/uapi/linux/vhost.h | 12 +++ >> include/uapi/linux/virtio_net.h | 12 +++ >> 4 files changed, 383 insertions(+), 52 deletions(-) >> >> -- >> 1.8.3.1 -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@...r.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Powered by blists - more mailing lists