file1: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.0 Starting 8 processes Jobs: 8 (f=8): [W(8)][100.0%][r=0KiB/s,w=123MiB/s][r=0,w=31.5k IOPS][eta 00m:00s] file1: (groupid=0, jobs=1): err= 0: pid=1218: Wed Dec 13 14:58:41 2017 write: IOPS=3839, BW=14.0MiB/s (15.7MB/s)(8999MiB/600001msec) slat (usec): min=4, max=369392, avg=227.73, stdev=1229.83 clat (usec): min=113, max=373025, avg=3923.34, stdev=4848.17 lat (usec): min=118, max=373433, avg=4154.83, stdev=5012.74 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 4], 60.00th=[ 4], | 70.00th=[ 4], 80.00th=[ 5], 90.00th=[ 5], 95.00th=[ 5], | 99.00th=[ 6], 99.50th=[ 17], 99.90th=[ 79], 99.95th=[ 111], | 99.99th=[ 194] bw ( KiB/s): min= 3732, max=22216, per=12.82%, avg=15393.36, stdev=2641.49, samples=1200 iops : min= 933, max= 5554, avg=3847.98, stdev=660.39, samples=1200 lat (usec) : 250=0.04%, 500=0.01%, 750=0.01%, 1000=0.03% lat (msec) : 2=0.28%, 4=70.76%, 10=28.16%, 20=0.26%, 50=0.25% lat (msec) : 100=0.13%, 250=0.06%, 500=0.01% cpu : usr=1.07%, sys=91.99%, ctx=265847, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,2303836,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 file1: (groupid=0, jobs=1): err= 0: pid=1219: Wed Dec 13 14:58:41 2017 write: IOPS=3640, BW=14.2MiB/s (14.9MB/s)(8533MiB/600001msec) slat (usec): min=4, max=367205, avg=242.26, stdev=1701.75 clat (usec): min=105, max=400086, avg=4137.17, stdev=6855.07 lat (usec): min=110, max=400553, avg=4383.15, stdev=7106.79 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 4], 60.00th=[ 4], | 70.00th=[ 4], 80.00th=[ 5], 90.00th=[ 5], 95.00th=[ 5], | 99.00th=[ 11], 99.50th=[ 35], 99.90th=[ 112], 99.95th=[ 148], | 99.99th=[ 241] bw ( KiB/s): min= 264, max=23358, per=12.15%, avg=14587.82, stdev=3928.56, samples=1200 iops : min= 66, max= 5839, avg=3646.64, stdev=982.14, samples=1200 lat (usec) : 250=0.03%, 500=0.01%, 750=0.01%, 1000=0.03% lat (msec) : 2=0.22%, 4=70.42%, 10=28.26%, 20=0.29%, 50=0.37% lat (msec) : 100=0.25%, 250=0.11%, 500=0.01% cpu : usr=0.98%, sys=88.04%, ctx=464380, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,2184434,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 file1: (groupid=0, jobs=1): err= 0: pid=1220: Wed Dec 13 14:58:41 2017 write: IOPS=3808, BW=14.9MiB/s (15.6MB/s)(8925MiB/600001msec) slat (usec): min=4, max=348744, avg=230.25, stdev=1217.66 clat (usec): min=110, max=353791, avg=3955.93, stdev=4810.70 lat (usec): min=115, max=354077, avg=4189.87, stdev=4976.17 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 4], 60.00th=[ 4], | 70.00th=[ 4], 80.00th=[ 5], 90.00th=[ 5], 95.00th=[ 5], | 99.00th=[ 8], 99.50th=[ 22], 99.90th=[ 79], 99.95th=[ 104], | 99.99th=[ 180] bw ( KiB/s): min= 1360, max=21672, per=12.71%, avg=15258.19, stdev=2808.41, samples=1200 iops : min= 340, max= 5418, avg=3814.23, stdev=702.12, samples=1200 lat (usec) : 250=0.04%, 500=0.01%, 750=0.01%, 1000=0.04% lat (msec) : 2=0.25%, 4=70.75%, 10=28.05%, 20=0.32%, 50=0.32% lat (msec) : 100=0.15%, 250=0.05%, 500=0.01% cpu : usr=1.03%, sys=91.51%, ctx=315742, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,2284898,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 file1: (groupid=0, jobs=1): err= 0: pid=1221: Wed Dec 13 14:58:41 2017 write: IOPS=3699, BW=14.4MiB/s (15.2MB/s)(8670MiB/600001msec) slat (usec): min=4, max=384286, avg=237.31, stdev=1556.89 clat (usec): min=69, max=456496, avg=4071.97, stdev=6252.37 lat (usec): min=109, max=456673, avg=4313.09, stdev=6477.02 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 4], 60.00th=[ 4], | 70.00th=[ 4], 80.00th=[ 5], 90.00th=[ 5], 95.00th=[ 5], | 99.00th=[ 10], 99.50th=[ 31], 99.90th=[ 104], 99.95th=[ 136], | 99.99th=[ 211] bw ( KiB/s): min= 793, max=24016, per=12.35%, avg=14826.33, stdev=3502.09, samples=1200 iops : min= 198, max= 6004, avg=3706.22, stdev=875.53, samples=1200 lat (usec) : 100=0.01%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.03% lat (msec) : 2=0.26%, 4=70.92%, 10=27.75%, 20=0.31%, 50=0.36% lat (msec) : 100=0.22%, 250=0.10%, 500=0.01% cpu : usr=1.03%, sys=89.25%, ctx=473168, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,2219487,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 file1: (groupid=0, jobs=1): err= 0: pid=1222: Wed Dec 13 14:58:41 2017 write: IOPS=3625, BW=14.2MiB/s (14.9MB/s)(8498MiB/600001msec) slat (usec): min=4, max=375714, avg=243.16, stdev=1737.37 clat (usec): min=59, max=387462, avg=4154.04, stdev=7013.84 lat (usec): min=109, max=387723, avg=4400.98, stdev=7271.32 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 4], 60.00th=[ 4], | 70.00th=[ 4], 80.00th=[ 5], 90.00th=[ 5], 95.00th=[ 5], | 99.00th=[ 13], 99.50th=[ 39], 99.90th=[ 116], 99.95th=[ 153], | 99.99th=[ 236] bw ( KiB/s): min= 729, max=21739, per=12.09%, avg=14519.35, stdev=3986.29, samples=1200 iops : min= 182, max= 5434, avg=3629.62, stdev=996.59, samples=1200 lat (usec) : 100=0.01%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.03% lat (msec) : 2=0.28%, 4=70.67%, 10=27.86%, 20=0.33%, 50=0.40% lat (msec) : 100=0.25%, 250=0.13%, 500=0.01% cpu : usr=1.03%, sys=87.52%, ctx=510004, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,2175576,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 file1: (groupid=0, jobs=1): err= 0: pid=1223: Wed Dec 13 14:58:41 2017 write: IOPS=3891, BW=15.2MiB/s (15.9MB/s)(9120MiB/600001msec) slat (usec): min=4, max=256927, avg=228.15, stdev=981.91 clat (usec): min=55, max=268950, avg=3869.74, stdev=3893.99 lat (usec): min=145, max=300244, avg=4100.95, stdev=4027.72 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 4], 60.00th=[ 4], | 70.00th=[ 4], 80.00th=[ 5], 90.00th=[ 5], 95.00th=[ 5], | 99.00th=[ 6], 99.50th=[ 15], 99.90th=[ 65], 99.95th=[ 90], | 99.99th=[ 148] bw ( KiB/s): min= 1638, max=22276, per=12.99%, avg=15599.33, stdev=2421.87, samples=1200 iops : min= 409, max= 5569, avg=3899.47, stdev=605.49, samples=1200 lat (usec) : 100=0.01%, 250=0.05%, 500=0.02%, 750=0.02%, 1000=0.05% lat (msec) : 2=0.34%, 4=70.12%, 10=28.76%, 20=0.26%, 50=0.24% lat (msec) : 100=0.11%, 250=0.03%, 500=0.01% cpu : usr=1.06%, sys=93.28%, ctx=212422, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,2334616,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 file1: (groupid=0, jobs=1): err= 0: pid=1224: Wed Dec 13 14:58:41 2017 write: IOPS=3707, BW=14.5MiB/s (15.2MB/s)(8689MiB/600001msec) slat (usec): min=4, max=356512, avg=237.42, stdev=1554.30 clat (usec): min=42, max=376204, avg=4063.23, stdev=6248.90 lat (usec): min=109, max=376508, avg=4304.36, stdev=6472.77 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 4], 60.00th=[ 4], | 70.00th=[ 4], 80.00th=[ 5], 90.00th=[ 5], 95.00th=[ 5], | 99.00th=[ 10], 99.50th=[ 31], 99.90th=[ 99], 99.95th=[ 132], | 99.99th=[ 239] bw ( KiB/s): min= 1124, max=22706, per=12.40%, avg=14881.48, stdev=3507.39, samples=1200 iops : min= 281, max= 5676, avg=3719.98, stdev=876.86, samples=1200 lat (usec) : 50=0.01%, 250=0.05%, 500=0.02%, 750=0.01%, 1000=0.03% lat (msec) : 2=0.34%, 4=70.57%, 10=28.00%, 20=0.28%, 50=0.37% lat (msec) : 100=0.22%, 250=0.09%, 500=0.01% cpu : usr=1.02%, sys=89.30%, ctx=437871, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,2224313,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 file1: (groupid=0, jobs=1): err= 0: pid=1225: Wed Dec 13 14:58:41 2017 write: IOPS=3799, BW=14.8MiB/s (15.6MB/s)(8905MiB/600001msec) slat (usec): min=4, max=361544, avg=230.35, stdev=1332.97 clat (usec): min=51, max=368069, avg=3965.33, stdev=5387.38 lat (usec): min=110, max=368366, avg=4199.48, stdev=5581.52 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 4], 60.00th=[ 4], | 70.00th=[ 4], 80.00th=[ 5], 90.00th=[ 5], 95.00th=[ 5], | 99.00th=[ 7], 99.50th=[ 21], 99.90th=[ 89], 99.95th=[ 123], | 99.99th=[ 197] bw ( KiB/s): min= 963, max=22292, per=12.69%, avg=15237.54, stdev=3142.05, samples=1200 iops : min= 240, max= 5573, avg=3809.02, stdev=785.52, samples=1200 lat (usec) : 100=0.01%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.02% lat (msec) : 2=0.23%, 4=71.18%, 10=27.77%, 20=0.26%, 50=0.27% lat (msec) : 100=0.16%, 250=0.08%, 500=0.01% cpu : usr=1.01%, sys=91.25%, ctx=341973, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,2279628,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=117MiB/s (123MB/s), 14.2MiB/s-15.2MiB/s (14.9MB/s-15.9MB/s), io=68.7GiB (73.8GB), run=600001-600001msec Disk stats (read/write): vdb: ios=1/50323, merge=0/451, ticks=0/1016470, in_queue=362769, util=5.81%