linux-kernel - Re: [RFC] virtio: use mandatory barriers for remote processor vdevs

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20111211122544.GC11504@redhat.com>
Date:	Sun, 11 Dec 2011 14:25:45 +0200
From:	"Michael S. Tsirkin" <mst@...hat.com>
To:	Rusty Russell <rusty@...tcorp.com.au>, akong@...hat.com
Cc:	Benjamin Herrenschmidt <benh@...nel.crashing.org>,
	Ohad Ben-Cohen <ohad@...ery.com>,
	virtualization@...ts.linux-foundation.org, kvm@...r.kernel.org,
	linux-kernel@...r.kernel.org, linux-arm-kernel@...ts.infradead.org
Subject: Re: [RFC] virtio: use mandatory barriers for remote processor vdevs

On Sat, Dec 03, 2011 at 03:44:36PM +1030, Rusty Russell wrote:
> On Sat, 03 Dec 2011 10:09:44 +1100, Benjamin Herrenschmidt <benh@...nel.crashing.org> wrote:
> > On Tue, 2011-11-29 at 14:31 +0200, Ohad Ben-Cohen wrote:
> > > A trivial, albeit sub-optimal, solution would be to simply revert
> > > commit d57ed95 "virtio: use smp_XX barriers on SMP". Obviously, though,
> > > that's going to have a negative impact on performance of SMP-based
> > > virtualization use cases.
> > 
> > Have you measured the impact of using normal barriers (non-SMP ones)
> > like we use on normal HW drivers unconditionally ?
> > 
> > IE. If the difference is small enough I'd say just go for it and avoid
> > the bloat.
> 
> Yep.  Plan is:
> 1) Measure the difference.
> 2) Difference unmeassurable?  Use normal barriers (ie. revert d57ed95).
> 3) Difference small?  Revert d57ed95 for 3.2, revisit for 3.3.
> 4) Difference large?  Runtime switch based on "if you're PCI" for 3.2,
>    revisit for 3.3.
> 
> Cheers,
> Rusty.

Forwarding some results by Amos, who run multiple netperf streams in
parallel, from an external box to the guest.  TCP_STREAM results were
noisy.  This could be due to buffering done by TCP, where packet size
varies even as message size is constant.

TCP_RR results were consistent. In this benchmark, after switching
to mandatory barriers, CPU utilization increased by up to 35% while
throughput went down by up to 14%. the normalized throughput/cpu
regressed consistently, between 7 and 35%

The "fix" applied was simply this:

diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c
index 3198f2e..fdccb77 100644
--- a/drivers/virtio/virtio_ring.c
+++ b/drivers/virtio/virtio_ring.c
@@ -23,7 +23,7 @@
 
 /* virtio guest is communicating with a virtual "device" that actually runs on
  * a host processor.  Memory barriers are used to control SMP effects. */
-#ifdef CONFIG_SMP
+#if 0
 /* Where possible, use SMP barriers which are more lightweight than mandatory
  * barriers, because mandatory barriers control MMIO effects on accesses
  * through relaxed memory I/O windows (which virtio does not use). */



-- 
MST

View attachment "compare-external_host-guest.txt" of type "text/plain" (18833 bytes)