file1: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.0 Starting 8 processes Jobs: 8 (f=8): [W(8)][100.0%][r=0KiB/s,w=82.2MiB/s][r=0,w=21.0k IOPS][eta 00m:00s] file1: (groupid=0, jobs=1): err= 0: pid=1131: Wed Dec 13 16:14:21 2017 write: IOPS=2908, BW=11.4MiB/s (11.9MB/s)(6816MiB/600001msec) slat (usec): min=5, max=787176, avg=316.82, stdev=3687.41 clat (usec): min=74, max=1056.3k, avg=5171.76, stdev=14861.09 lat (usec): min=135, max=1056.7k, avg=5491.61, stdev=15383.15 clat percentiles (usec): | 1.00th=[ 1532], 5.00th=[ 2311], 10.00th=[ 2573], 20.00th=[ 2933], | 30.00th=[ 3294], 40.00th=[ 3556], 50.00th=[ 3818], 60.00th=[ 4047], | 70.00th=[ 4359], 80.00th=[ 4686], 90.00th=[ 5342], 95.00th=[ 5997], | 99.00th=[ 38536], 99.50th=[ 89654], 99.90th=[225444], 99.95th=[278922], | 99.99th=[505414] bw ( KiB/s): min= 16, max=37683, per=13.12%, avg=11655.03, stdev=4424.17, samples=1199 iops : min= 4, max= 9420, avg=2913.44, stdev=1106.04, samples=1199 lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.05%, 1000=0.23% lat (msec) : 2=1.64%, 4=56.03%, 10=39.78%, 20=0.79%, 50=0.63% lat (msec) : 100=0.40%, 250=0.37%, 500=0.07%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=0.79%, sys=74.80%, ctx=868689, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,1744871,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 file1: (groupid=0, jobs=1): err= 0: pid=1132: Wed Dec 13 16:14:21 2017 write: IOPS=2802, BW=10.9MiB/s (11.5MB/s)(6567MiB/600001msec) slat (usec): min=5, max=1163.8k, avg=329.51, stdev=4078.35 clat (usec): min=52, max=1271.4k, avg=5367.10, stdev=16476.69 lat (usec): min=140, max=1271.5k, avg=5699.81, stdev=17066.63 clat percentiles (usec): | 1.00th=[ 1336], 5.00th=[ 2278], 10.00th=[ 2573], 20.00th=[ 2900], | 30.00th=[ 3261], 40.00th=[ 3490], 50.00th=[ 3752], 60.00th=[ 4015], | 70.00th=[ 4293], 80.00th=[ 4621], 90.00th=[ 5276], 95.00th=[ 6063], | 99.00th=[ 47449], 99.50th=[106431], 99.90th=[254804], 99.95th=[316670], | 99.99th=[463471] bw ( KiB/s): min= 96, max=34464, per=12.64%, avg=11225.62, stdev=4880.80, samples=1198 iops : min= 24, max= 8616, avg=2806.20, stdev=1220.20, samples=1198 lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.06%, 1000=0.33% lat (msec) : 2=1.78%, 4=57.81%, 10=37.44%, 20=0.85%, 50=0.73% lat (msec) : 100=0.44%, 250=0.42%, 500=0.10%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=0.79%, sys=71.65%, ctx=1044041, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,1681268,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 file1: (groupid=0, jobs=1): err= 0: pid=1133: Wed Dec 13 16:14:21 2017 write: IOPS=2985, BW=11.7MiB/s (12.2MB/s)(6997MiB/600001msec) slat (usec): min=5, max=748162, avg=307.69, stdev=3307.29 clat (usec): min=60, max=775858, avg=5038.81, stdev=13095.28 lat (usec): min=165, max=776031, avg=5349.64, stdev=13543.69 clat percentiles (usec): | 1.00th=[ 1483], 5.00th=[ 2311], 10.00th=[ 2573], 20.00th=[ 2933], | 30.00th=[ 3294], 40.00th=[ 3523], 50.00th=[ 3785], 60.00th=[ 4047], | 70.00th=[ 4293], 80.00th=[ 4686], 90.00th=[ 5342], 95.00th=[ 5997], | 99.00th=[ 36439], 99.50th=[ 82314], 99.90th=[204473], 99.95th=[246416], | 99.99th=[404751] bw ( KiB/s): min= 24, max=35440, per=13.46%, avg=11951.36, stdev=3904.50, samples=1200 iops : min= 6, max= 8860, avg=2987.58, stdev=976.12, samples=1200 lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.04%, 1000=0.23% lat (msec) : 2=1.69%, 4=56.62%, 10=39.10%, 20=0.82%, 50=0.68% lat (msec) : 100=0.41%, 250=0.35%, 500=0.04%, 750=0.01%, 1000=0.01% cpu : usr=0.84%, sys=76.03%, ctx=786053, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,1791122,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 file1: (groupid=0, jobs=1): err= 0: pid=1134: Wed Dec 13 16:14:21 2017 write: IOPS=2774, BW=10.8MiB/s (11.4MB/s)(6502MiB/600001msec) slat (usec): min=6, max=1165.7k, avg=332.59, stdev=4092.38 clat (usec): min=62, max=1168.9k, avg=5421.32, stdev=16731.39 lat (usec): min=142, max=1169.2k, avg=5757.12, stdev=17348.61 clat percentiles (usec): | 1.00th=[ 1467], 5.00th=[ 2311], 10.00th=[ 2606], 20.00th=[ 2966], | 30.00th=[ 3294], 40.00th=[ 3556], 50.00th=[ 3818], 60.00th=[ 4047], | 70.00th=[ 4359], 80.00th=[ 4686], 90.00th=[ 5342], 95.00th=[ 6063], | 99.00th=[ 47449], 99.50th=[105382], 99.90th=[258999], 99.95th=[320865], | 99.99th=[476054] bw ( KiB/s): min= 8, max=40232, per=12.54%, avg=11136.08, stdev=4803.28, samples=1197 iops : min= 2, max=10058, avg=2783.74, stdev=1200.81, samples=1197 lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.07%, 1000=0.25% lat (msec) : 2=1.71%, 4=55.77%, 10=39.67%, 20=0.81%, 50=0.73% lat (msec) : 100=0.43%, 250=0.42%, 500=0.10%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=0.74%, sys=71.57%, ctx=949754, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,1664518,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 file1: (groupid=0, jobs=1): err= 0: pid=1135: Wed Dec 13 16:14:21 2017 write: IOPS=2692, BW=10.5MiB/s (11.0MB/s)(6311MiB/600002msec) slat (usec): min=6, max=951339, avg=344.14, stdev=4217.19 clat (usec): min=45, max=955624, avg=5584.63, stdev=17319.60 lat (usec): min=144, max=955868, avg=5931.93, stdev=17960.09 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 2311], 10.00th=[ 2573], 20.00th=[ 2933], | 30.00th=[ 3261], 40.00th=[ 3523], 50.00th=[ 3785], 60.00th=[ 4015], | 70.00th=[ 4293], 80.00th=[ 4621], 90.00th=[ 5276], 95.00th=[ 6063], | 99.00th=[ 60031], 99.50th=[121111], 99.90th=[256902], 99.95th=[312476], | 99.99th=[497026] bw ( KiB/s): min= 48, max=23743, per=12.17%, avg=10806.28, stdev=5068.49, samples=1197 iops : min= 12, max= 5935, avg=2701.33, stdev=1267.12, samples=1197 lat (usec) : 50=0.01%, 250=0.02%, 500=0.01%, 750=0.05%, 1000=0.23% lat (msec) : 2=1.45%, 4=57.54%, 10=37.96%, 20=0.85%, 50=0.77% lat (msec) : 100=0.49%, 250=0.53%, 500=0.10%, 750=0.01%, 1000=0.01% cpu : usr=0.73%, sys=69.40%, ctx=1092449, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,1615619,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 file1: (groupid=0, jobs=1): err= 0: pid=1136: Wed Dec 13 16:14:21 2017 write: IOPS=2808, BW=10.0MiB/s (11.5MB/s)(6582MiB/600001msec) slat (usec): min=5, max=1164.6k, avg=328.77, stdev=4029.61 clat (usec): min=69, max=1167.1k, avg=5355.39, stdev=16199.32 lat (usec): min=133, max=1167.3k, avg=5687.28, stdev=16774.26 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 2343], 10.00th=[ 2606], 20.00th=[ 2966], | 30.00th=[ 3326], 40.00th=[ 3556], 50.00th=[ 3818], 60.00th=[ 4047], | 70.00th=[ 4293], 80.00th=[ 4621], 90.00th=[ 5276], 95.00th=[ 5997], | 99.00th=[ 46400], 99.50th=[102237], 99.90th=[235930], 99.95th=[299893], | 99.99th=[497026] bw ( KiB/s): min= 32, max=21088, per=12.69%, avg=11272.85, stdev=4574.15, samples=1197 iops : min= 8, max= 5272, avg=2817.91, stdev=1143.53, samples=1197 lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.03%, 1000=0.23% lat (msec) : 2=1.43%, 4=56.67%, 10=39.15%, 20=0.79%, 50=0.71% lat (msec) : 100=0.45%, 250=0.42%, 500=0.08%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=0.77%, sys=72.24%, ctx=899175, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,1684919,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 file1: (groupid=0, jobs=1): err= 0: pid=1137: Wed Dec 13 16:14:21 2017 write: IOPS=2813, BW=10.0MiB/s (11.5MB/s)(6593MiB/600001msec) slat (usec): min=5, max=585328, avg=328.40, stdev=3937.41 clat (usec): min=134, max=626978, avg=5346.05, stdev=15758.18 lat (usec): min=141, max=627505, avg=5677.59, stdev=16313.91 clat percentiles (usec): | 1.00th=[ 1418], 5.00th=[ 2278], 10.00th=[ 2573], 20.00th=[ 2933], | 30.00th=[ 3294], 40.00th=[ 3556], 50.00th=[ 3818], 60.00th=[ 4047], | 70.00th=[ 4293], 80.00th=[ 4686], 90.00th=[ 5407], 95.00th=[ 6063], | 99.00th=[ 42730], 99.50th=[104334], 99.90th=[250610], 99.95th=[316670], | 99.99th=[459277] bw ( KiB/s): min= 104, max=22252, per=12.67%, avg=11258.18, stdev=4395.06, samples=1200 iops : min= 26, max= 5563, avg=2814.34, stdev=1098.77, samples=1200 lat (usec) : 250=0.03%, 500=0.02%, 750=0.08%, 1000=0.27% lat (msec) : 2=1.71%, 4=55.92%, 10=39.49%, 20=0.86%, 50=0.71% lat (msec) : 100=0.38%, 250=0.42%, 500=0.10%, 750=0.01% cpu : usr=0.74%, sys=72.58%, ctx=934703, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,1687883,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 file1: (groupid=0, jobs=1): err= 0: pid=1138: Wed Dec 13 16:14:21 2017 write: IOPS=2422, BW=9689KiB/s (9921kB/s)(5677MiB/600001msec) slat (usec): min=5, max=836561, avg=385.80, stdev=5028.24 clat (usec): min=83, max=911210, avg=6206.53, stdev=20455.78 lat (usec): min=143, max=911433, avg=6595.48, stdev=21201.82 clat percentiles (usec): | 1.00th=[ 1303], 5.00th=[ 2245], 10.00th=[ 2540], 20.00th=[ 2900], | 30.00th=[ 3294], 40.00th=[ 3556], 50.00th=[ 3818], 60.00th=[ 4047], | 70.00th=[ 4359], 80.00th=[ 4752], 90.00th=[ 5473], 95.00th=[ 6259], | 99.00th=[ 86508], 99.50th=[149947], 99.90th=[295699], 99.95th=[354419], | 99.99th=[549454] bw ( KiB/s): min= 48, max=28440, per=10.94%, avg=9712.94, stdev=5482.06, samples=1198 iops : min= 12, max= 7110, avg=2428.04, stdev=1370.51, samples=1198 lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.03%, 1000=0.36% lat (msec) : 2=2.00%, 4=55.19%, 10=39.22%, 20=0.84%, 50=0.84% lat (msec) : 100=0.61%, 250=0.73%, 500=0.14%, 750=0.01%, 1000=0.01% cpu : usr=0.66%, sys=63.84%, ctx=1410998, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,1453347,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=86.7MiB/s (90.0MB/s), 9689KiB/s-11.7MiB/s (9921kB/s-12.2MB/s), io=50.8GiB (54.6GB), run=600001-600002msec Disk stats (read/write): vdb: ios=0/39272, merge=0/258, ticks=0/3274077, in_queue=2287355, util=22.13%