linux-kernel - RE: VM performance issue in KVM guests.

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Tue, 13 Apr 2010 08:50:08 +0800
From:	"Zhang, Xiantao" <xiantao.zhang@...el.com>
To:	Avi Kivity <avi@...hat.com>
CC:	"kvm@...r.kernel.org" <kvm@...r.kernel.org>,
	Marcelo Tosatti <mtosatti@...hat.com>,
	"Yang, Xiaowei" <xiaowei.yang@...el.com>,
	"Dong, Eddie" <eddie.dong@...el.com>, "Li, Xin" <xin.li@...el.com>,
	Ingo Molnar <mingo@...e.hu>,
	Peter Zijlstra <peterz@...radead.org>,
	Mike Galbraith <efault@....de>,
	Linux Kernel Mailing List <linux-kernel@...r.kernel.org>
Subject: RE: VM performance issue in KVM guests.

Avi Kivity wrote:
> On 04/12/2010 05:04 AM, Zhang, Xiantao wrote:
>> 
>>> What was the performance hit?  What was your I/O setup (image
>>> format, using aio?) 
>>> 
>> The issue only happens when vcpu number is over-committed(e.g.
>> vcpu/pcpu>2) and physical cpus are saturated. For example,  when run
>> webbench in windows OS in this case, its performance drops by 80%. 
>> In our experiment, we are using image file through virtio, and I
>> think aio should be used by default also.    
>> 
> 
> Is this on a machine that does pause-loop exits?  The current handing
> of PLE is very suboptimal.  With proper directed yield we should be
> much better there.
> 
> Without PLE, we need paravirtualized spinlocks, no way around it.

PLE has the ability to eliminate the issue at some extent, and pv solution should be helpful also.  But for windows guests running on machines without PLE, we still needs to enhance host side to resolve the issue.

>>>> After analysis about Linux scheduler, we found it is indeed caused
>>>> by the known features of Linux schduler, such as AFFINE_WAKEUPS,
>>>> SYNC_WAKEUPS etc. With these features on, linux schduler often
>>>> tries to schedule the vcpu threads of one guests to one same
>>>> logical processor when vcpus are over-committed and logical
>>>> processors are saturated. Once the vcpu threads of one VM are
>>>> scheduled to the same LP, system performance drops dramatically
>>>> with some workloads(like webbench running in windows OS).
>>> 
>> Since the hit only happens when physical cpus are saturated, and
>> sheduling non-virtualized multiple threads of one process to same
>> processor can benefit the performance due to cache share or other
>> affinities, but you know it hurts performance a lot once schedule
>> two vcpu theads to a same processor due to mutual spin-lock in
>> guests.     
>> 
> Spin loops need to be addressed first, they are known to kill
> performance in overcommit situations.

Even in overcommit case, if vcpu threads of one qemu are not scheduled or pulled to the same logical processor, the performance drop is tolerant like Xen's case today. But for KVM, it has to suffer from additional performance loss, since host's scheduler actively pulls these vcpu threads together. 
Xiantao 

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/