lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date:	Sun, 13 Dec 2009 13:58:21 +0200
From:	Ozan Çağlayan <ozan@...dus.org.tr>
To:	Avi Kivity <avi@...hat.com>
CC:	"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
	KVM list <kvm@...r.kernel.org>
Subject: Re: Poor KVM guest performance on an HP rack server

Ozan Çağlayan wrote:
> Avi Kivity wrote:
>> First, are you sure that kvm is enabled?  'info kvm' in the monitor.
> (qemu) info kvm
> kvm support: enabled

Resending it as my mail client wrapped all the lines, sorry..


(qemu) info kvm
kvm support: enabled

(qemu) info cpus
* CPU #0: pc=0x00000000c011ceb3 (halted) thread_id=26023
  CPU #1: pc=0x00000000c011ceb3 (halted) thread_id=26024

(qemu) info block
virtio0: type=hd removable=0 file=/dev/cciss/c1d0p1 ro=0 drv=host_device encrypted=0
ide1-cd0: type=cdrom removable=1 locked=0 [not inserted]
floppy0: type=floppy removable=1 locked=0 [not inserted]
sd0: type=floppy removable=1 locked=0 [not inserted]

-------

KVM related messages on the host server:

~ # dmesg |grep kvm
[ 3649.058945] loaded kvm module (kvm-kmod-2.6.32)
[ 3683.736737] kvm: emulating exchange as write (It's interesting, what does it mean?)

On the guest side:

~ # dmesg | grep kvm
[    0.000000] kvm-clock: cpu 0, msr 0:64a3e1, boot clock
[    0.000000] kvm-clock: cpu 0, msr 0:20103e1, primary cpu clock
[    0.065075] kvm-clock: cpu 1, msr 0:20203e1, secondary cpu clock

First of all if there are some specific outputs that will reveal the
IO/CPU boundness of the workload, I'd like to give them too. I'm not
much familiar with diagnostic/analysis tools..

>From host's POV:
----------------

Guest is idle:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
26117 root      20   0 2153m 175m 1356 S    5  0.5   0:33.57 qemu-kvm

Guest is configuring CUPS:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
26117 root      20   0 2129m 180m 1356 S  143  0.6   0:44.34 qemu-kvm (peak at 140,150~ %CPU)

load average reached 0.91 at the end of the configuration.

SMP view of top command shows that only Cpu2 and Cpu9 are involved,
other ones are zeroed (maybe coherent with -smp 2, don't have much idea):

Cpu2  : 28.4%us, 31.4%sy,  0.0%ni, 40.2%id,  0.0%wa,  0.0%hi,  0.0%si, 0.0%st
Cpu9  : 27.7%us, 26.7%sy,  0.0%ni, 45.5%id,  0.0%wa,  0.0%hi,  0.0%si, 0.0%st


vmstat -n 1 output on host:

procs -----------memory---------- ---swap-- -----io---- -system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 .
 .
 .
 2  0      0 30161636 117300 116416    0    0     0    36 20366 3868  2 2 96  0  0 (./configure)
 1  0      0 30161468 117300 116416    0    0     0     4 21673 4153  2 3 95  0  0
 2  0      0 30160848 117300 116416    0    0     0     0 16549 4215  2 2 95  0  0
 2  0      0 30160396 117300 116416    0    0     0     0 21089 4196  2 3 95  0  0
 1  0      0 30160308 117300 116416    0    0     0     0 17031 4189  2 3 95  0  0
 1  0      0 30160088 117300 116420    0    0     0   128 21772 4184  2 3 95  0  0
 4  0      0 30159848 117304 116416    0    0     0    28 22378 4117  2 3 95  0  0
 2  0      0 30159704 117312 116420    0    0     0    20 23530 4082  2 3 95  0  0
 1  0      0 30159624 117312 116420    0    0     0     0 20997 4098  2 3 95  0  0
 1  0      0 30159608 117324 116416    0    0     0    40 18519 4290  2 3 95  0  0
 1  0      0 30159600 117324 116416    0    0     0    76 20959 4187  2 3 95  0  0
 1  0      0 30159764 117324 116416    0    0     0     0 19718 4205  2 3 95  0  0
 1  0      0 30159848 117324 116416    0    0     0     0 19934 4095  2 3 95  0  0
 1  0      0 30159724 117324 116416    0    0     0     0 21821 4094  2 3 95  0  0
 1  0      0 30159684 117328 116412    0    0     0    16 19621 4093  2 2 95  0  0
 1  0      0 30159848 117328 116416    0    0     0    76 19630 4162  2 3 95  0  0
 1  0      0 30159724 117328 116416    0    0     0     0 22810 4033  2 3 95  0  0
 1  0      0 30159848 117328 116416    0    0     0     0 20525 4105  2 3 95  0  0
 1  0      0 30159560 117328 116416    0    0     0     0 15715 4188  2 2 95  0  0
 1  0      0 30159516 117328 116416    0    0     0     0 17686 4223  2 2 95  0  0
 1  0      0 30159476 117328 116416    0    0     0    76 17308 4174  2 2 95  0  0
 1  0      0 30159888 117328 116416    0    0     0     0 17973 4186  2 3 95  0  0
 1  0      0 30159848 117328 116416    0    0     0     0 18795 4147  2 3 95  0  0
 1  0      0 30159808 117328 116416    0    0     0     0 18346 4181  2 2 95  0  0
 1  0      0 30160096 117328 116416    0    0     0     0 17054 4202  2 2 95  0  0
 1  0      0 30159892 117328 116416    0    0     0    76 16767 4256  2 2 95  0  0
 2  0      0 30159600 117328 116416    0    0     0     8 19817 4191  2 3 95  0  0
 2  0      0 30159600 117328 116416    0    0     0     0 19709 4125  2 3 95  0  0
 1  0      0 30159476 117328 116416    0    0     0     0 17322 4153  2 2 95  0  0
 1  0      0 30159600 117328 116416    0    0     0     0 20106 4189  2 3 95  0  0
 1  0      0 30159808 117328 116416    0    0     0    80 17810 4202  2 2 95  0  0
 1  0      0 30159848 117328 116416    0    0     0     0 16723 4217  2 2 95  0  0
 1  0      0 30160052 117328 116416    0    0     0     0 17799 4140  2 2 95  0  0
 1  0      0 30159560 117328 116416    0    0     0     0 15878 4244  2 2 95  0  0
 2  0      0 30159764 117328 116416    0    0     0     0 19632 4124  2 3 95  0  0
 1  0      0 30160144 117328 116416    0    0     0   188 18119 4255  2 2 95  0  0
 1  0      0 30159860 117328 116416    0    0     0     0 17664 4139  2 2 95  0  0
 2  0      0 30159856 117328 116416    0    0     0     0 23323 4099  2 3 95  0  0
 2  0      0 30159776 117328 116416    0    0     0    12 30120 3790  2 3 94  0  0
 1  0      0 30159484 117328 116416    0    0     0     0 25599 4172  2 3 95  0  0
 2  0      0 30159732 117328 116416    0    0     0   128 41030 3579  2 4 94  0  0
 2  0      0 30159528 117328 116416    0    0     0     0 39064 3644  2 4 94  0  0
 1  0      0 30159816 117328 116416    0    0     0     0 39337 3556  2 4 94  0  0
 1  0      0 30159940 117328 116416    0    0     0     0 41252 3617  2 4 94  0  0
 2  0      0 30160112 117328 116416    0    0     0     0 36845 3668  2 3 94  0  0
 2  0      0 30160484 117328 116416    0    0     0   116 39011 3737  2 4 94  0  0
 1  0      0 30160196 117328 116416    0    0     0     0 38782 3646  2 4 94  0  0
 1  0      0 30160072 117328 116416    0    0     0     0 38546 3684  2 4 94  0  0
 1  0      0 30160444 117328 116416    0    0     0     0 37738 3641  2 4 94  0  0
 0  0      0 30160336 117328 116416    0    0     0    20 26019 3596  2 3 95  0  0
 0  0      0 30160460 117328 116416    0    0     0   184 2373 2286  0 0 100  0  0 (finished)
 0  0      0 30160460 117328 116416    0    0     0     0 2398 2279  0 0 100  0  0
 0  0      0 30160460 117328 116416    0    0     0     0 2356 2251  0 0 100  0  0
 0  0      0 30160460 117328 116416    0    0     0     0 2395 2275  0 0 100  0  0
 0  0      0 30160460 117328 116416    0    0     0     0 2354 2249  0 0 100  0  0
 0  0      0 30160460 117328 116416    0    0     0     0 2395 2283  0 0 100  0  0
 0  0      0 30160460 117328 116416    0    0     0     0 2367 2230  0 0 100  0  0
 0  0      0 30160460 117328 116416    0    0     0     0 2396 2277  0 0 100  0  0
 0  0      0 30160460 117328 116416    0    0     0     0 2344 2223  0 0 100  0  0

I also checked iotop on host and qemu-kvm appears occasionally with disk
writes at max 200 K/s.

>From guest's POV:
-----------------

Configuring CUPS:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
19258 root      20   0  3396 1656 1192 R   18  0.1   0:05.20 configure (Saw maximum 22 %CPU)

The following is the highest load average instant during configuration:

top - 11:35:32 up 33 min,  3 users,  load average: 0.56, 0.25, 0.15
Tasks:  99 total,   4 running,  95 sleeping,   0 stopped,   0 zombie
Cpu0  :  4.7%us, 68.3%sy,  0.0%ni, 27.0%id,  0.0%wa,  0.0%hi,  0.0%si, 0.0%st
Cpu1  :  4.3%us, 66.3%sy,  0.0%ni, 29.3%id,  0.0%wa,  0.0%hi,  0.0%si, 0.0%st
Mem:   2061660k total,   137564k used,  1924096k free,    58672k buffers
Swap:   307192k total,        0k used,   307192k free,    49240k cached

vmstat -n 1:
------------

(context switches and interrupts are much more higher during
configuration than the idle state)

procs -----------memory---------- ---swap-- -----io---- -system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0      0 1926012  58552  49172    0    0    27     4  165   46  1  8 91  0  0
 0  0      0 1925956  58552  49176    0    0     0     0   33   18  0  0 100  0  0
 0  0      0 1925832  58560  49172    0    0     0    32   71   78  0  0 100  0  0
 0  0      0 1925832  58560  49176    0    0     0     0   42   21  0  0 100  0  0
 0  0      0 1925832  58560  49176    0    0     0     0   74   73  0  1 99  0  0
 0  0      0 1925832  58560  49176    0    0     0     0   26   15  0  0 100  0  0
 0  0      0 1925832  58560  49176    0    0     0     0   95  216  0  0 100  0  0
 0  0      0 1925832  58560  49176    0    0     0     0   33   35  0  0 100  0  0
 0  0      0 1925832  58568  49176    0    0     0    12  143  166  1  0 99  0  0
 0  0      0 1925840  58568  49176    0    0     0     0  181  224  0  1 99  0  0
 0  0      0 1925880  58568  49176    0    0     0     0   67   71  0  0 100  0  0
 0  0      0 1925880  58568  49176    0    0     0     0   22   12  0  0 100  0  0
 2  0      0 1925880  58568  49176    0    0     0     0  108  119  0  0 100  0  0
 0  0      0 1925624  58576  49176    0    0     0    12  222  212  1  4 95  0  0
 0  0      0 1925540  58576  49176    0    0     0     0  220  185  0  2 98  0  0
 0  0      0 1925500  58576  49176    0    0     0     0   34   26  0  0 100  0  0
 0  0      0 1925508  58576  49176    0    0     0    12  164  121  0  1 98  0  0
 0  0      0 1925508  58576  49176    0    0     0     0   54   59  0  0 100  0  0
 0  0      0 1925508  58584  49176    0    0     0    36   81   87  0  0 100  0  0
 0  0      0 1925508  58584  49176    0    0     0     0   47   30  0  1 100  0  0
 0  0      0 1925508  58584  49176    0    0     0     0   77   80  0  0 100  0  0
 0  0      0 1925508  58584  49176    0    0     0     0   71   78  0  0 100  0  0
 0  0      0 1925436  58584  49176    0    0     0     0  175  216  0  0 100  0  0
 0  0      0 1925428  58584  49176    0    0     0     0   80   66  0  1 99  0  0
 0  0      0 1925428  58592  49176    0    0     0    48   84   91  0  0 100  0  0
 1  0      0 1924788  58592  49176    0    0     0     0 1413  401  2 33 65  0  0  (./configure)
 1  0      0 1924092  58592  49072    0    0     0     4 2210  599  5 57 38  0  0
 1  0      0 1924544  58592  49080    0    0     0     0 2131  349  5 51 45  0  0
 1  0      0 1924580  58592  49088    0    0     0     0 2188  564  5 54 42  0  0
 2  0      0 1924796  58592  49092    0    0     0     0 2153  386  5 53 42  0  0
 2  0      0 1924892  58604  49092    0    0     0   112 2148  409  5 54 41  0  0
 1  0      0 1924732  58604  49096    0    0     0     0 2153  415  5 56 39  0  0
 1  0      0 1924916  58604  49100    0    0     0     0 2197  508  5 56 39  0  0
 1  0      0 1924788  58604  49100    0    0     0     0 2134  376  5 56 40  0  0
 1  0      0 1924612  58604  49100    0    0     0     4 2199  522  6 52 42  0  0
 1  0      0 1924772  58612  49100    0    0     0    84 2184  405  6 53 41  0  0
 3  0      0 1924960  58612  49116    0    0     0     0 2187  453  5 53 42  0  0
 1  0      0 1924600  58612  49112    0    0     0     0 2156  474  5 54 42  0  0
 1  0      0 1924164  58612  49128    0    0     0     0 2151  420  5 55 40  0  0
 2  0      0 1924944  58612  49116    0    0     0     0 2130  357  5 53 42  0  0
 2  0      0 1924536  58620  49120    0    0     0    80 2218  691  6 54 41  0  0
 1  0      0 1924252  58620  49144    0    0     0     0 2121  358  5 54 41  0  0
 1  0      0 1924888  58620  49128    0    0     0     0 2158  422  6 54 41  0  0
 1  0      0 1922456  58620  49132    0    0     0     0 2125  386  6 50 44  0  0
 1  0      0 1924696  58620  49144    0    0     0     0 2195  451  5 53 43  0  0
 2  0      0 1924780  58628  49140    0    0     0    80 2143  379  6 51 44  0  0
 2  0      0 1924660  58628  49144    0    0     0     0 2151  389  6 51 44  0  0
 1  0      0 1924380  58628  49144    0    0     0     0 2130  315  7 50 43  0  0
 1  0      0 1924504  58628  49152    0    0     0     0 2169  400  6 52 42  0  0
 2  0      0 1924548  58628  49156    0    0     0     0 2126  320  6 51 44  0  0
 1  0      0 1924660  58636  49156    0    0     0    80 2161  367  5 52 43  0  0
 1  0      0 1924412  58636  49160    0    0     0     8 2133  323  5 52 43  0  0
 1  0      0 1924392  58636  49176    0    0     0     0 2182  575  5 55 40  0  0
 1  0      0 1924516  58636  49180    0    0     0     0 2128  318  6 52 43  0  0
 1  0      0 1923572  58636  49180    0    0     0     0 2164  432  7 51 43  0  0
 1  0      0 1924192  58644  49184    0    0     0    80 2164  451  5 52 43  0  0
 1  0      0 1923980  58644  49192    0    0     0     0 2160  389  7 50 44  0  0
 2  0      0 1924528  58644  49192    0    0     0     0 2122  318  6 51 43  0  0
 2  0      0 1924448  58644  49212    0    0     0     0 2166  418  7 51 43  0  0
 1  0      0 1923872  58644  49200    0    0     0    72 2137  322  6 52 43  0  0
 1  0      0 1923276  58652  49208    0    0     0    84 2171  401  5 53 42  0  0
 1  0      0 1923052  58652  49220    0    0     0     0 2175  554  6 54 41  0  0
 1  0      0 1924736  58652  49220    0    0     0     0 2172  395  6 54 40  0  0
 2  0      0 1924560  58652  49100    0    0     0     4 2145 1591  6 59 36  0  0
 2  0      0 1924460  58652  49212    0    0     0    28 2212  573  5 60 36  0  0
 3  0      0 1924060  58660  49224    0    0     0    96 2201  583  5 63 33  0  0
 1  0      0 1924128  58660  49240    0    0     0     0 2189  611  5 67 28  0  0
 1  0      0 1923668  58660  49240    0    0     0     0 2169  652  6 67 28  0  0
 4  0      0 1924212  58660  49240    0    0     0     0 2225  705  4 68 28  0  0
 1  0      0 1924136  58660  49240    0    0     0     0 2158  571  5 66 29  0  0
 1  0      0 1924176  58672  49228    0    0     0   152 2211  693  4 69 27  0  0
 2  0      0 1924072  58672  49240    0    0     0     0 2169  569  5 67 29  0  0
 4  0      0 1924180  58672  49240    0    0     0     0 2183  620  5 68 28  0  0
 1  0      0 1924140  58672  49240    0    0     0     0 2208  811  4 68 28  0  0
 1  0      0 1924240  58672  49244    0    0     0    20 2190  606  4 64 32  0  0
 0  0      0 1924800  58672  49236    0    0     0     0  646 1352  3 18 78  0  0
 0  0      0 1924808  58684  49236    0    0     0   200  111  162  0  1 99  0  0
 0  0      0 1924808  58684  49236    0    0     0     0   74   84  0  0 100  0  0 (finished)
 0  0      0 1924808  58684  49236    0    0     0     0  139  170  0  0 100  0  0
 0  0      0 1924808  58684  49236    0    0     0     0   53   67  0  0 99  0  0
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Powered by blists - more mailing lists