linux-kernel - RE: VM performance issue in KVM guests.

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Fri, 16 Apr 2010 10:27:14 +0800
From:	"Zhang, Xiantao" <xiantao.zhang@...el.com>
To:	Avi Kivity <avi@...hat.com>
CC:	"kvm@...r.kernel.org" <kvm@...r.kernel.org>,
	Marcelo Tosatti <mtosatti@...hat.com>,
	"Yang, Xiaowei" <xiaowei.yang@...el.com>,
	"Dong, Eddie" <eddie.dong@...el.com>, "Li, Xin" <xin.li@...el.com>,
	Ingo Molnar <mingo@...e.hu>,
	Peter Zijlstra <peterz@...radead.org>,
	Mike Galbraith <efault@....de>,
	Linux Kernel Mailing List <linux-kernel@...r.kernel.org>
Subject: RE: VM performance issue in KVM guests.

Avi Kivity wrote:
> On 04/14/2010 06:24 AM, Zhang, Xiantao wrote:
>> 
>>>>> Spin loops need to be addressed first, they are known to kill
>>>>> performance in overcommit situations.
>>>>> 
>>>>> 
>>>> Even in overcommit case, if vcpu threads of one qemu are not
>>>> scheduled or pulled to the same logical processor, the performance
>>>> drop is tolerant like Xen's case today. But for KVM, it has to
>>>> suffer from additional performance loss, since host's scheduler
>>>> actively pulls these vcpu threads together.
>>>> 
>>>> 
>>>> 
>>> Can you quantify this loss?  Give examples of what happens?
>>> 
>> For example, one machine is configured with 2 pCPUs and there are
>> two Windows guests running on the machine, and each guest is
>> cconfigured with 2 vcpus and one webbench server runs in it.  
>> If use host's default scheduler, webbench's performance is very bad,
>> but if pin each geust's vCPU0 to pCPU0 and vCPU1 to pCPU1, we can
>> see 5-10X performance improvement with same CPU utilization.  
>> In addition, we also see kvm's perf scalability is also impacted in
>> large systems, for some performance experiments, kvm's perf begins
>> to drop when vCPU is overcommitted and pCPU are saturated, but once
>> the wake_up_affine feature is switched off in scheduler, kvm's perf
>> can keep rising in this case.    
>> 
> 
> Ok.  This is probably due to spinlock contention.

Yes, exactly. 

> When vcpus are pinned to pcpus, there is a 50% chance that a guest's
> vcpus will be co-scheduled and spinlocks will perform will.
> 
> When vcpus are not pinned, but affine wakeups are disabled, there is a
> 33% chance that vcpus will be co-scheduled.
> 
> When vcpus are not pinned and affine wakeups are enabled there is a 0%
> chance that vcpus will be co-scheduled.
> 
> Keeping both vcpus on the same core actually makes sense since they
> can communicate through the local cache faster than across cores. 
> What we need is to make sure that they don't spin.
> 
> Windows 2008 can report spinlock spinning through a hypercall.  Can
> you hook to that interface and see if it happens regularly? 
> Altenatively use a PLE capable host and trace the kvm_vcpu_on_spin()
> function. 
We only tried windows 2003 for the experiments, and have no data related to windows 2008. 
But maybe we can have  a try later.  Anyway, the key point is we have to enhance the scheduler to let it 
Know which threads are vcpu threads to avoid perf loss in this case.
Xiantao
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/