lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
Open Source and information security mailing list archives
| ||
|
Date: Mon, 30 Jul 2012 14:27:00 +0800 From: Asias He <asias@...hat.com> To: Sasha Levin <levinsasha928@...il.com> CC: linux-kernel@...r.kernel.org, Rusty Russell <rusty@...tcorp.com.au>, "Michael S. Tsirkin" <mst@...hat.com>, virtualization@...ts.linux-foundation.org, kvm@...r.kernel.org, Christoph Hellwig <hch@....de>, Minchan Kim <minchan.kim@...il.com> Subject: Re: [PATCH V4 3/3] virtio-blk: Add bio-based IO path for virtio-blk Hello Sasha, On 07/28/2012 02:35 PM, Sasha Levin wrote: > On 07/28/2012 04:21 AM, Asias He wrote: >> This patch introduces bio-based IO path for virtio-blk. >> >> Compared to request-based IO path, bio-based IO path uses driver >> provided ->make_request_fn() method to bypasses the IO scheduler. It >> handles the bio to device directly without allocating a request in block >> layer. This reduces the IO path in guest kernel to achieve high IOPS >> and lower latency. The downside is that guest can not use the IO >> scheduler to merge and sort requests. However, this is not a big problem >> if the backend disk in host side uses faster disk device. >> >> When the bio-based IO path is not enabled, virtio-blk still uses the >> original request-based IO path, no performance difference is observed. >> >> Performance evaluation: >> ----------------------------- >> 1) Fio test is performed in a 8 vcpu guest with ramdisk based guest using >> kvm tool. >> >> Short version: >> With bio-based IO path, sequential read/write, random read/write >> IOPS boost : 28%, 24%, 21%, 16% >> Latency improvement: 32%, 17%, 21%, 16% >> >> Long version: >> With bio-based IO path: >> seq-read : io=2048.0MB, bw=116996KB/s, iops=233991 , runt= 17925msec >> seq-write : io=2048.0MB, bw=100829KB/s, iops=201658 , runt= 20799msec >> rand-read : io=3095.7MB, bw=112134KB/s, iops=224268 , runt= 28269msec >> rand-write: io=3095.7MB, bw=96198KB/s, iops=192396 , runt= 32952msec >> clat (usec): min=0 , max=2631.6K, avg=58716.99, stdev=191377.30 >> clat (usec): min=0 , max=1753.2K, avg=66423.25, stdev=81774.35 >> clat (usec): min=0 , max=2915.5K, avg=61685.70, stdev=120598.39 >> clat (usec): min=0 , max=1933.4K, avg=76935.12, stdev=96603.45 >> cpu : usr=74.08%, sys=703.84%, ctx=29661403, majf=21354, minf=22460954 >> cpu : usr=70.92%, sys=702.81%, ctx=77219828, majf=13980, minf=27713137 >> cpu : usr=72.23%, sys=695.37%, ctx=88081059, majf=18475, minf=28177648 >> cpu : usr=69.69%, sys=654.13%, ctx=145476035, majf=15867, minf=26176375 >> With request-based IO path: >> seq-read : io=2048.0MB, bw=91074KB/s, iops=182147 , runt= 23027msec >> seq-write : io=2048.0MB, bw=80725KB/s, iops=161449 , runt= 25979msec >> rand-read : io=3095.7MB, bw=92106KB/s, iops=184211 , runt= 34416msec >> rand-write: io=3095.7MB, bw=82815KB/s, iops=165630 , runt= 38277msec >> clat (usec): min=0 , max=1932.4K, avg=77824.17, stdev=170339.49 >> clat (usec): min=0 , max=2510.2K, avg=78023.96, stdev=146949.15 >> clat (usec): min=0 , max=3037.2K, avg=74746.53, stdev=128498.27 >> clat (usec): min=0 , max=1363.4K, avg=89830.75, stdev=114279.68 >> cpu : usr=53.28%, sys=724.19%, ctx=37988895, majf=17531, minf=23577622 >> cpu : usr=49.03%, sys=633.20%, ctx=205935380, majf=18197, minf=27288959 >> cpu : usr=55.78%, sys=722.40%, ctx=101525058, majf=19273, minf=28067082 >> cpu : usr=56.55%, sys=690.83%, ctx=228205022, majf=18039, minf=26551985 >> >> 2) Fio test is performed in a 8 vcpu guest with Fusion-IO based guest using >> kvm tool. >> >> Short version: >> With bio-based IO path, sequential read/write, random read/write >> IOPS boost : 11%, 11%, 13%, 10% >> Latency improvement: 10%, 10%, 12%, 10% >> Long Version: >> With bio-based IO path: >> read : io=2048.0MB, bw=58920KB/s, iops=117840 , runt= 35593msec >> write: io=2048.0MB, bw=64308KB/s, iops=128616 , runt= 32611msec >> read : io=3095.7MB, bw=59633KB/s, iops=119266 , runt= 53157msec >> write: io=3095.7MB, bw=62993KB/s, iops=125985 , runt= 50322msec >> clat (usec): min=0 , max=1284.3K, avg=128109.01, stdev=71513.29 >> clat (usec): min=94 , max=962339 , avg=116832.95, stdev=65836.80 >> clat (usec): min=0 , max=1846.6K, avg=128509.99, stdev=89575.07 >> clat (usec): min=0 , max=2256.4K, avg=121361.84, stdev=82747.25 >> cpu : usr=56.79%, sys=421.70%, ctx=147335118, majf=21080, minf=19852517 >> cpu : usr=61.81%, sys=455.53%, ctx=143269950, majf=16027, minf=24800604 >> cpu : usr=63.10%, sys=455.38%, ctx=178373538, majf=16958, minf=24822612 >> cpu : usr=62.04%, sys=453.58%, ctx=226902362, majf=16089, minf=23278105 >> With request-based IO path: >> read : io=2048.0MB, bw=52896KB/s, iops=105791 , runt= 39647msec >> write: io=2048.0MB, bw=57856KB/s, iops=115711 , runt= 36248msec >> read : io=3095.7MB, bw=52387KB/s, iops=104773 , runt= 60510msec >> write: io=3095.7MB, bw=57310KB/s, iops=114619 , runt= 55312msec >> clat (usec): min=0 , max=1532.6K, avg=142085.62, stdev=109196.84 >> clat (usec): min=0 , max=1487.4K, avg=129110.71, stdev=114973.64 >> clat (usec): min=0 , max=1388.6K, avg=145049.22, stdev=107232.55 >> clat (usec): min=0 , max=1465.9K, avg=133585.67, stdev=110322.95 >> cpu : usr=44.08%, sys=590.71%, ctx=451812322, majf=14841, minf=17648641 >> cpu : usr=48.73%, sys=610.78%, ctx=418953997, majf=22164, minf=26850689 >> cpu : usr=45.58%, sys=581.16%, ctx=714079216, majf=21497, minf=22558223 >> cpu : usr=48.40%, sys=599.65%, ctx=656089423, majf=16393, minf=23824409 > > What are the cases where we'll see a performance degradation with using the bio path? Could we measure performance for those as well? > >> How to use: >> ----------------------------- >> Add 'virtio_blk.use_bio=1' to kernel cmdline or 'modprobe virtio_blk >> use_bio=1' to enable ->make_request_fn() based I/O path. > > If there are, in fact, no cases where performance is degraded, can use_bio=1 be the default? Here are some results on a SATA based image file. In this case, the bio path is slower than req path doing sequential read/write. qemu use_bio=0 (req-based) ------------------------------------- read : io=120964KB, bw=26098KB/s, iops=6470 , runt= 4635msec write: io=190236KB, bw=35505KB/s, iops=8829 , runt= 5358msec read : io=257400KB, bw=1963.7KB/s, iops=488 , runt=131081msec write: io=258944KB, bw=1302.7KB/s, iops=324 , runt=198872msec clat (msec): min=1 , max=1527 , avg=30.73, stdev=144.73 clat (usec): min=811 , max=247072 , avg=28451.71, stdev=16107.22 clat (msec): min=6 , max=2519 , avg=513.91, stdev=231.07 clat (msec): min=33 , max=2621 , avg=772.33, stdev=348.39 cpu : usr=4.05%, sys=14.56%, ctx=38199, majf=0, minf=4 cpu : usr=4.02%, sys=15.48%, ctx=53724, majf=0, minf=0 cpu : usr=0.15%, sys=0.30%, ctx=20535, majf=0, minf=16 cpu : usr=0.32%, sys=0.96%, ctx=101465, majf=0, minf=0 qemu use_bio=1 (bio-based) ------------------------------------- read : io=202736KB, bw=25569KB/s, iops=6360 , runt= 7929msec write: io=217844KB, bw=20335KB/s, iops=5060 , runt= 10713msec read : io=256980KB, bw=1958.2KB/s, iops=487 , runt=131235msec write: io=258288KB, bw=1423.9KB/s, iops=354 , runt=181405msec clat (usec): min=922 , max=1578.2K, avg=38702.18, stdev=99248.33 clat (usec): min=460 , max=241314 , avg=49326.52, stdev=18705.68 clat (msec): min=19 , max=2370 , avg=515.30, stdev=200.84 clat (msec): min=11 , max=3751 , avg=702.60, stdev=286.93 cpu : usr=2.54%, sys=8.75%, ctx=68522, majf=0, minf=6 cpu : usr=1.96%, sys=7.70%, ctx=70003, majf=0, minf=0 cpu : usr=0.39%, sys=1.46%, ctx=259459, majf=0, minf=16 cpu : usr=0.28%, sys=1.21%, ctx=265148, majf=0, minf=0 lkvm use_bio=0 (req-based) ------------------------------------- read : io=150120KB, bw=40420KB/s, iops=10037 , runt= 3714msec write: io=194932KB, bw=27029KB/s, iops=6722 , runt= 7212msec read : io=257136KB, bw=2001.1KB/s, iops=498 , runt=128443msec write: io=258276KB, bw=1537.2KB/s, iops=382 , runt=168028msec clat (msec): min=1 , max=1542 , avg=24.84, stdev=32.45 clat (msec): min=3 , max=628 , avg=35.62, stdev=39.71 clat (msec): min=8 , max=2540 , avg=503.28, stdev=236.97 clat (msec): min=41 , max=4398 , avg=653.88, stdev=302.61 cpu : usr=3.91%, sys=15.75%, ctx=26968, majf=0, minf=23 cpu : usr=2.50%, sys=10.56%, ctx=19090, majf=0, minf=0 cpu : usr=0.16%, sys=0.43%, ctx=20159, majf=0, minf=16 cpu : usr=0.18%, sys=0.53%, ctx=81364, majf=0, minf=0 lkvm use_bio=1 (bio-based) ------------------------------------- read : io=124812KB, bw=36537KB/s, iops=9060 , runt= 3416msec write: io=169180KB, bw=24406KB/s, iops=6065 , runt= 6932msec read : io=256200KB, bw=2089.3KB/s, iops=520 , runt=122630msec write: io=257988KB, bw=1545.7KB/s, iops=384 , runt=166910msec clat (msec): min=1 , max=1527 , avg=28.06, stdev=89.54 clat (msec): min=2 , max=344 , avg=41.12, stdev=38.70 clat (msec): min=8 , max=1984 , avg=490.63, stdev=207.28 clat (msec): min=33 , max=4131 , avg=659.19, stdev=304.71 cpu : usr=4.85%, sys=17.15%, ctx=31593, majf=0, minf=7 cpu : usr=3.04%, sys=11.45%, ctx=39377, majf=0, minf=0 cpu : usr=0.47%, sys=1.59%, ctx=262986, majf=0, minf=16 cpu : usr=0.47%, sys=1.46%, ctx=337410, majf=0, minf=0 -- Asias -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@...r.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Powered by blists - more mailing lists