linux-kernel - Re: VM performance issue in KVM guests.

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <4BC57949.2090003@redhat.com>
Date:	Wed, 14 Apr 2010 11:14:01 +0300
From:	Avi Kivity <avi@...hat.com>
To:	"Zhang, Xiantao" <xiantao.zhang@...el.com>
CC:	"kvm@...r.kernel.org" <kvm@...r.kernel.org>,
	Marcelo Tosatti <mtosatti@...hat.com>,
	"Yang, Xiaowei" <xiaowei.yang@...el.com>,
	"Dong, Eddie" <eddie.dong@...el.com>, "Li, Xin" <xin.li@...el.com>,
	Ingo Molnar <mingo@...e.hu>,
	Peter Zijlstra <peterz@...radead.org>,
	Mike Galbraith <efault@....de>,
	Linux Kernel Mailing List <linux-kernel@...r.kernel.org>
Subject: Re: VM performance issue in KVM guests.

On 04/14/2010 06:24 AM, Zhang, Xiantao wrote:
>
>>>> Spin loops need to be addressed first, they are known to kill
>>>> performance in overcommit situations.
>>>>
>>>>          
>>> Even in overcommit case, if vcpu threads of one qemu are not
>>> scheduled or pulled to the same logical processor, the performance
>>> drop is tolerant like Xen's case today. But for KVM, it has to
>>> suffer from additional performance loss, since host's scheduler
>>> actively pulls these vcpu threads together.
>>>
>>>
>>>        
>> Can you quantify this loss?  Give examples of what happens?
>>      
> For example, one machine is configured with 2 pCPUs and there are two Windows guests running on the machine, and each guest is cconfigured with 2 vcpus and one webbench server runs in it.
> If use host's default scheduler, webbench's performance is very bad, but if pin each geust's vCPU0 to pCPU0 and vCPU1 to pCPU1, we can see 5-10X performance improvement with same CPU utilization.
> In addition, we also see kvm's perf scalability is also impacted in large systems, for some performance experiments, kvm's perf begins to drop when vCPU is overcommitted and pCPU are saturated, but once the wake_up_affine feature is switched off in scheduler, kvm's perf can keep rising in this case.
>    

Ok.  This is probably due to spinlock contention.

When vcpus are pinned to pcpus, there is a 50% chance that a guest's 
vcpus will be co-scheduled and spinlocks will perform will.

When vcpus are not pinned, but affine wakeups are disabled, there is a 
33% chance that vcpus will be co-scheduled.

When vcpus are not pinned and affine wakeups are enabled there is a 0% 
chance that vcpus will be co-scheduled.

Keeping both vcpus on the same core actually makes sense since they can 
communicate through the local cache faster than across cores.  What we 
need is to make sure that they don't spin.

Windows 2008 can report spinlock spinning through a hypercall.  Can you 
hook to that interface and see if it happens regularly?  Altenatively 
use a PLE capable host and trace the kvm_vcpu_on_spin() function.

-- 
error compiling committee.c: too many arguments to function

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/