| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <1344291825.19049.176.camel@haakon2.linux-iscsi.org>
Date: Mon, 06 Aug 2012 15:23:45 -0700
From: "Nicholas A. Bellinger" <nab@...ux-iscsi.org>
To: target-devel <target-devel@...r.kernel.org>
Cc: linux-scsi <linux-scsi@...r.kernel.org>,
LKML <linux-kernel@...r.kernel.org>,
Jens Axboe <axboe@...nel.dk>, Christoph Hellwig <hch@....de>,
Roland Dreier <roland@...nel.org>,
Hannes Reinecke <mail@...nes-reinecke.de>,
Andy Grover <agrover@...hat.com>,
"Michael S. Tsirkin" <mst@...hat.com>,
Stefan Hajnoczi <stefanha@...il.com>,
Paolo Bonzini <pbonzini@...hat.com>,
James Bottomley <James.Bottomley@...senPartnership.com>
Subject: SCSI small block random I/O performance on 3.6-rc0 using SCSI
loopback ports
Hello SCSI + target folks,
I've recently been running fio benchmarks using rd_mcp (TCM ramdisk) +
SCSI loopback fabric ports to get an idea of how baseline bare-metal
target performance is shaping up for SCSI core using 3.6-rc0 code with
mixed small block random I/O workloads, and wanted to share these
results.
The machine in question is a dual Xeon-E5-2687W (Romley-EP) 3.10 Ghz w/
32x threads + 32 GB of DDR3 1600Mhz memory. Each rd_mcp is 500MB is in
size, and is setting emulate_write_cache=1 to expose WCE=1 to SCSI core.
The loopback SCSI LLD is also running with SCSI host_lockless mode
enabled.
The full results are attached below, with the principle results:
workload: | 25 write / 75 read | 75 write / 25 read |
----------------|---------------------|---------------------|
32x rd_mcp LUNs | ~1185K IOPs | ~1175K IOPs |
10x rd_mcp LUNs | ~944K IOPs | ~928K IOPs |
This is the first time we've been able to push SCSI LUNs beyond the 1
million mixed random IOPs range with loopback + rd_mcp..
For the 32x rd_mcp LUNs case user CPU load is being reported as ~95%,
and sys CPU load is at ~5%.
Reducing the number of loopback LUNs to 10x produces ~935k IOPs for the
same numjobs=32 workload, that ends up being about ~93.5K IOPs per SCSI
LUN..
For reference, this is using Jen's previous recommendations for optimal
small block random I/O performance with 32x loopback LUNs + numjobs=32
here:
[randrw]
rw=randrw
rwmixwrite=25
rwmixread=75
ioengine=libaio
direct=1
size=100G
iodepth=64
iodepth_batch=4
iodepth_batch_complete=32
numjobs=32
blocksize=4k
filename=/dev/sdb
filename=/dev/sdc
filename=/dev/sdd
filename=/dev/sde
filename=/dev/sdf
filename=/dev/sdg
filename=/dev/sdh
filename=/dev/sdi
filename=/dev/sdj
filename=/dev/sdk
filename=/dev/sdl
filename=/dev/sdm
filename=/dev/sdn
filename=/dev/sdo
filename=/dev/sdp
filename=/dev/sdq
filename=/dev/sdr
filename=/dev/sds
filename=/dev/sdt
filename=/dev/sdu
filename=/dev/sdv
filename=/dev/sdw
filename=/dev/sdx
filename=/dev/sdy
filename=/dev/sdz
filename=/dev/sdaa
filename=/dev/sdab
filename=/dev/sdac
filename=/dev/sdad
filename=/dev/sdae
filename=/dev/sdaf
filename=/dev/sdag
Along with the top-level output from /backstores + /loopback objects in
rtsadmin/targetlci shell:
/> ls backstores/
o- backstores ............................................................ [...]
o- fileio ................................................. [1 Storage Object]
| o- fileio_vhost_test .............. [/usr/src/fileio_vhost_test deactivated]
o- iblock ................................................. [0 Storage Object]
o- pscsi .................................................. [0 Storage Object]
o- rd_dr .................................................. [0 Storage Object]
o- rd_mcp ............................................... [32 Storage Objects]
o- ramdisk0 ............................................ [ramdisk activated]
o- ramdisk1 ............................................ [ramdisk activated]
o- ramdisk10 ........................................... [ramdisk activated]
o- ramdisk11 ........................................... [ramdisk activated]
o- ramdisk12 ........................................... [ramdisk activated]
o- ramdisk13 ........................................... [ramdisk activated]
o- ramdisk14 ........................................... [ramdisk activated]
o- ramdisk15 ........................................... [ramdisk activated]
o- ramdisk16 ........................................... [ramdisk activated]
o- ramdisk17 ........................................... [ramdisk activated]
o- ramdisk18 ........................................... [ramdisk activated]
o- ramdisk19 ........................................... [ramdisk activated]
o- ramdisk2 ............................................ [ramdisk activated]
o- ramdisk20 ........................................... [ramdisk activated]
o- ramdisk21 ........................................... [ramdisk activated]
o- ramdisk22 ........................................... [ramdisk activated]
o- ramdisk23 ........................................... [ramdisk activated]
o- ramdisk24 ........................................... [ramdisk activated]
o- ramdisk25 ........................................... [ramdisk activated]
o- ramdisk26 ........................................... [ramdisk activated]
o- ramdisk27 ........................................... [ramdisk activated]
o- ramdisk28 ........................................... [ramdisk activated]
o- ramdisk29 ........................................... [ramdisk activated]
o- ramdisk3 ............................................ [ramdisk activated]
o- ramdisk30 ........................................... [ramdisk activated]
o- ramdisk31 ........................................... [ramdisk activated]
o- ramdisk4 ............................................ [ramdisk activated]
o- ramdisk5 ............................................ [ramdisk activated]
o- ramdisk6 ............................................ [ramdisk activated]
o- ramdisk7 ............................................ [ramdisk activated]
o- ramdisk8 ............................................ [ramdisk activated]
o- ramdisk9 ............................................ [ramdisk activated]
/> ls loopback/
o- loopback ....................................................... [32 Targets]
o- naa.600140508aa83c15 ............................... [naa.6001405c56c93ba0]
| o- luns ............................................................ [1 LUN]
| o- lun0 ...................................... [rd_mcp/ramdisk7 (ramdisk)]
o- naa.6001405169ce2b93 ............................... [naa.6001405d6f9a3285]
| o- luns ............................................................ [1 LUN]
| o- lun0 ...................................... [rd_mcp/ramdisk5 (ramdisk)]
o- naa.60014051e0d1dcbb ............................... [naa.6001405c1e3e07ff]
| o- luns ............................................................ [1 LUN]
| o- lun0 ...................................... [rd_mcp/ramdisk1 (ramdisk)]
o- naa.60014051e9293b89 ............................... [naa.600140566b195426]
| o- luns ............................................................ [1 LUN]
| o- lun0 ...................................... [rd_mcp/ramdisk8 (ramdisk)]
o- naa.60014052154d6616 ............................... [naa.60014054eca52e01]
| o- luns ............................................................ [1 LUN]
| o- lun0 ..................................... [rd_mcp/ramdisk19 (ramdisk)]
o- naa.6001405269f15722 ............................... [naa.6001405503d329f0]
| o- luns ............................................................ [1 LUN]
| o- lun0 ..................................... [rd_mcp/ramdisk18 (ramdisk)]
o- naa.60014053bd681c72 ............................... [naa.6001405bd40c5e63]
| o- luns ............................................................ [1 LUN]
| o- lun0 ...................................... [rd_mcp/ramdisk6 (ramdisk)]
o- naa.600140544577c8ee ............................... [naa.60014058975b162e]
| o- luns ............................................................ [1 LUN]
| o- lun0 ..................................... [rd_mcp/ramdisk14 (ramdisk)]
o- naa.60014055fd219128 ............................... [naa.6001405854d06025]
| o- luns ............................................................ [1 LUN]
| o- lun0 ...................................... [rd_mcp/ramdisk4 (ramdisk)]
o- naa.60014056803182ac ............................... [naa.600140573b6189a5]
| o- luns ............................................................ [1 LUN]
| o- lun0 ..................................... [rd_mcp/ramdisk10 (ramdisk)]
o- naa.60014056bd49556d ............................... [naa.6001405a2b76c6d0]
| o- luns ............................................................ [1 LUN]
| o- lun0 ..................................... [rd_mcp/ramdisk28 (ramdisk)]
o- naa.600140574b9f5ab5 ............................... [naa.60014052389330ac]
| o- luns ............................................................ [1 LUN]
| o- lun0 ..................................... [rd_mcp/ramdisk13 (ramdisk)]
o- naa.600140579b070a5b ............................... [naa.60014057a53f54f0]
| o- luns ............................................................ [1 LUN]
| o- lun0 ...................................... [rd_mcp/ramdisk9 (ramdisk)]
o- naa.60014057ba8fa3d4 ............................... [naa.6001405b5e063bd3]
| o- luns ............................................................ [1 LUN]
| o- lun0 ..................................... [rd_mcp/ramdisk17 (ramdisk)]
o- naa.60014057d1d2a3b7 ............................... [naa.60014056a43e3b6f]
| o- luns ............................................................ [1 LUN]
| o- lun0 ..................................... [rd_mcp/ramdisk16 (ramdisk)]
o- naa.60014057d22f6d95 ............................... [naa.60014051fa1eddf1]
| o- luns ............................................................ [1 LUN]
| o- lun0 ..................................... [rd_mcp/ramdisk23 (ramdisk)]
o- naa.60014058d7563f6d ............................... [naa.6001405461fe7980]
| o- luns ............................................................ [1 LUN]
| o- lun0 ..................................... [rd_mcp/ramdisk21 (ramdisk)]
o- naa.60014058e31ee45c ............................... [naa.6001405c076ebea4]
| o- luns ............................................................ [1 LUN]
| o- lun0 ..................................... [rd_mcp/ramdisk31 (ramdisk)]
o- naa.60014059310d31fd ............................... [naa.6001405dd72b4e16]
| o- luns ............................................................ [1 LUN]
| o- lun0 ..................................... [rd_mcp/ramdisk12 (ramdisk)]
o- naa.6001405969efea55 ............................... [naa.6001405b790df9d1]
| o- luns ............................................................ [1 LUN]
| o- lun0 ..................................... [rd_mcp/ramdisk22 (ramdisk)]
o- naa.60014059849dabee ............................... [naa.6001405cfaae685a]
| o- luns ............................................................ [1 LUN]
| o- lun0 ..................................... [rd_mcp/ramdisk11 (ramdisk)]
o- naa.600140599e07bd25 ............................... [naa.6001405c22796f44]
| o- luns ............................................................ [1 LUN]
| o- lun0 ..................................... [rd_mcp/ramdisk24 (ramdisk)]
o- naa.6001405a5ce5a7a8 ............................... [naa.6001405d009f9147]
| o- luns ............................................................ [1 LUN]
| o- lun0 ..................................... [rd_mcp/ramdisk27 (ramdisk)]
o- naa.6001405ace9d83f9 ............................... [naa.600140523fad7754]
| o- luns ............................................................ [1 LUN]
| o- lun0 ...................................... [rd_mcp/ramdisk3 (ramdisk)]
o- naa.6001405af0bbc150 ............................... [naa.60014059260b6690]
| o- luns ............................................................ [1 LUN]
| o- lun0 ..................................... [rd_mcp/ramdisk15 (ramdisk)]
o- naa.6001405c275006b3 ............................... [naa.60014053aabec3bb]
| o- luns ............................................................ [1 LUN]
| o- lun0 ...................................... [rd_mcp/ramdisk2 (ramdisk)]
o- naa.6001405c43e2074a ............................... [naa.60014052a5e8aaad]
| o- luns ............................................................ [1 LUN]
| o- lun0 ..................................... [rd_mcp/ramdisk25 (ramdisk)]
o- naa.6001405c98babdf4 ............................... [naa.6001405aeb67b36e]
| o- luns ............................................................ [1 LUN]
| o- lun0 ..................................... [rd_mcp/ramdisk20 (ramdisk)]
o- naa.6001405d5ca3a6f2 ............................... [naa.6001405089a04ffe]
| o- luns ............................................................ [1 LUN]
| o- lun0 ..................................... [rd_mcp/ramdisk26 (ramdisk)]
o- naa.6001405e82fdb9a3 ............................... [naa.60014057e8b683e9]
| o- luns ............................................................ [1 LUN]
| o- lun0 ..................................... [rd_mcp/ramdisk30 (ramdisk)]
o- naa.6001405fb50ec5e8 ............................... [naa.60014054588a0e02]
| o- luns ............................................................ [1 LUN]
| o- lun0 ..................................... [rd_mcp/ramdisk29 (ramdisk)]
o- naa.6001405fe73bfb5b ............................... [naa.6001405b9442d8f8]
o- luns ............................................................ [1 LUN]
o- lun0 ...................................... [rd_mcp/ramdisk0 (ramdisk)]
/>
View attachment "3.6-rc0-loopback-rd_mcp-30x-25-75.txt" of type "text/plain" (36985 bytes)
View attachment "3.6-rc0-loopback-rd_mcp-30x-75-25.txt" of type "text/plain" (36913 bytes)
View attachment "3.6-rc0-loopback-rd_mcp-10x-25-75.txt" of type "text/plain" (35424 bytes)
View attachment "3.6-rc0-loopback-rd_mcp-10x-75-25.txt" of type "text/plain" (35633 bytes)
Powered by blists - more mailing lists