file1: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.0 Starting 8 processes file1: Laying out IO file (1 file / 10240MiB) Jobs: 8 (f=8): [W(8)][100.0%][r=0KiB/s,w=102MiB/s][r=0,w=26.1k IOPS][eta 00m:00s] file1: (groupid=0, jobs=1): err= 0: pid=1360: Wed Dec 13 16:02:00 2017 write: IOPS=4178, BW=16.3MiB/s (17.1MB/s)(9793MiB/600001msec) slat (usec): min=3, max=765706, avg=201.29, stdev=1529.42 clat (usec): min=60, max=769481, avg=3609.99, stdev=6000.14 lat (usec): min=109, max=769867, avg=3815.72, stdev=6202.11 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 3], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 4], 60.00th=[ 4], | 70.00th=[ 4], 80.00th=[ 4], 90.00th=[ 5], 95.00th=[ 5], | 99.00th=[ 6], 99.50th=[ 12], 99.90th=[ 51], 99.95th=[ 74], | 99.99th=[ 313] bw ( KiB/s): min= 609, max=22883, per=12.66%, avg=16780.35, stdev=3058.72, samples=1198 iops : min= 152, max= 5720, avg=4194.72, stdev=764.69, samples=1198 lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.09%, 4=87.62%, 10=11.70%, 20=0.25%, 50=0.22% lat (msec) : 100=0.07%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.01% cpu : usr=1.26%, sys=92.80%, ctx=216533, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,2507016,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 file1: (groupid=0, jobs=1): err= 0: pid=1361: Wed Dec 13 16:02:00 2017 write: IOPS=4101, BW=16.0MiB/s (16.8MB/s)(9612MiB/600001msec) slat (usec): min=3, max=832923, avg=205.39, stdev=1646.45 clat (usec): min=62, max=837254, avg=3677.76, stdev=6496.35 lat (usec): min=106, max=837548, avg=3887.67, stdev=6720.67 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 3], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 4], 60.00th=[ 4], | 70.00th=[ 4], 80.00th=[ 4], 90.00th=[ 5], 95.00th=[ 5], | 99.00th=[ 7], 99.50th=[ 19], 99.90th=[ 69], 99.95th=[ 95], | 99.99th=[ 313] bw ( KiB/s): min= 384, max=22156, per=12.41%, avg=16450.03, stdev=3574.82, samples=1198 iops : min= 96, max= 5539, avg=4112.29, stdev=893.71, samples=1198 lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.09%, 4=88.25%, 10=10.87%, 20=0.29%, 50=0.31% lat (msec) : 100=0.12%, 250=0.03%, 500=0.01%, 750=0.01%, 1000=0.01% cpu : usr=1.27%, sys=91.00%, ctx=322342, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,2460684,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 file1: (groupid=0, jobs=1): err= 0: pid=1362: Wed Dec 13 16:02:00 2017 write: IOPS=4158, BW=16.2MiB/s (17.0MB/s)(9747MiB/600001msec) slat (usec): min=3, max=829989, avg=201.98, stdev=1546.44 clat (usec): min=82, max=833896, avg=3627.01, stdev=6048.97 lat (usec): min=108, max=834076, avg=3833.49, stdev=6252.83 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 3], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 4], 60.00th=[ 4], | 70.00th=[ 4], 80.00th=[ 4], 90.00th=[ 5], 95.00th=[ 5], | 99.00th=[ 6], 99.50th=[ 14], 99.90th=[ 57], 99.95th=[ 83], | 99.99th=[ 300] bw ( KiB/s): min= 1611, max=43503, per=12.60%, avg=16705.43, stdev=3274.39, samples=1198 iops : min= 402, max=10875, avg=4176.00, stdev=818.59, samples=1198 lat (usec) : 100=0.01%, 250=0.11%, 500=0.03%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.11%, 4=87.72%, 10=11.40%, 20=0.25%, 50=0.25% lat (msec) : 100=0.09%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.01% cpu : usr=1.27%, sys=92.33%, ctx=253025, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,2495350,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 file1: (groupid=0, jobs=1): err= 0: pid=1363: Wed Dec 13 16:02:00 2017 write: IOPS=4254, BW=16.6MiB/s (17.4MB/s)(9972MiB/600001msec) slat (usec): min=5, max=766806, avg=203.17, stdev=1368.21 clat (usec): min=131, max=771002, avg=3541.83, stdev=5341.13 lat (usec): min=138, max=771300, avg=3748.35, stdev=5520.27 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 3], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 4], 60.00th=[ 4], | 70.00th=[ 4], 80.00th=[ 4], 90.00th=[ 5], 95.00th=[ 5], | 99.00th=[ 6], 99.50th=[ 8], 99.90th=[ 36], 99.95th=[ 54], | 99.99th=[ 300] bw ( KiB/s): min= 1731, max=22701, per=12.88%, avg=17083.49, stdev=2671.55, samples=1198 iops : min= 432, max= 5675, avg=4270.50, stdev=667.85, samples=1198 lat (usec) : 250=0.05%, 500=0.02%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.18%, 4=87.04%, 10=12.26%, 20=0.20%, 50=0.16% lat (msec) : 100=0.04%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% cpu : usr=1.21%, sys=94.41%, ctx=145712, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,2552889,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 file1: (groupid=0, jobs=1): err= 0: pid=1364: Wed Dec 13 16:02:00 2017 write: IOPS=4055, BW=15.8MiB/s (16.6MB/s)(9506MiB/600001msec) slat (usec): min=3, max=847054, avg=208.13, stdev=1715.47 clat (usec): min=55, max=850110, avg=3718.75, stdev=6733.35 lat (usec): min=105, max=850414, avg=3931.37, stdev=6968.33 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 3], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 4], 60.00th=[ 4], | 70.00th=[ 4], 80.00th=[ 4], 90.00th=[ 5], 95.00th=[ 5], | 99.00th=[ 8], 99.50th=[ 22], 99.90th=[ 75], 99.95th=[ 110], | 99.99th=[ 317] bw ( KiB/s): min= 907, max=22252, per=12.29%, avg=16300.78, stdev=3677.63, samples=1197 iops : min= 226, max= 5563, avg=4074.82, stdev=919.42, samples=1197 lat (usec) : 100=0.01%, 250=0.04%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.12%, 4=87.75%, 10=11.24%, 20=0.31%, 50=0.33% lat (msec) : 100=0.14%, 250=0.04%, 500=0.01%, 750=0.01%, 1000=0.01% cpu : usr=1.31%, sys=90.45%, ctx=396245, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,2433447,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 file1: (groupid=0, jobs=1): err= 0: pid=1365: Wed Dec 13 16:02:00 2017 write: IOPS=4125, BW=16.1MiB/s (16.9MB/s)(9668MiB/600001msec) slat (usec): min=3, max=766661, avg=204.36, stdev=1540.93 clat (usec): min=102, max=771014, avg=3656.38, stdev=6059.01 lat (usec): min=107, max=771384, avg=3865.18, stdev=6265.22 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 3], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 4], 60.00th=[ 4], | 70.00th=[ 4], 80.00th=[ 4], 90.00th=[ 5], 95.00th=[ 5], | 99.00th=[ 6], 99.50th=[ 17], 99.90th=[ 63], 99.95th=[ 86], | 99.99th=[ 300] bw ( KiB/s): min= 1691, max=22549, per=12.49%, avg=16557.28, stdev=3511.44, samples=1198 iops : min= 422, max= 5637, avg=4138.97, stdev=877.87, samples=1198 lat (usec) : 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.09%, 4=87.64%, 10=11.51%, 20=0.30%, 50=0.29% lat (msec) : 100=0.11%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.01% cpu : usr=1.23%, sys=91.80%, ctx=295015, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,2475045,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 file1: (groupid=0, jobs=1): err= 0: pid=1366: Wed Dec 13 16:02:00 2017 write: IOPS=4142, BW=16.2MiB/s (16.0MB/s)(9709MiB/600001msec) slat (usec): min=3, max=769630, avg=202.97, stdev=1605.67 clat (usec): min=74, max=773586, avg=3641.33, stdev=6337.34 lat (usec): min=106, max=773767, avg=3848.80, stdev=6552.80 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 3], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 4], 60.00th=[ 4], | 70.00th=[ 4], 80.00th=[ 4], 90.00th=[ 5], 95.00th=[ 5], | 99.00th=[ 6], 99.50th=[ 15], 99.90th=[ 58], 99.95th=[ 86], | 99.99th=[ 334] bw ( KiB/s): min= 224, max=22995, per=12.55%, avg=16640.38, stdev=3320.37, samples=1198 iops : min= 56, max= 5748, avg=4159.72, stdev=830.10, samples=1198 lat (usec) : 100=0.01%, 250=0.02%, 500=0.01%, 750=0.03%, 1000=0.01% lat (msec) : 2=0.12%, 4=87.95%, 10=11.21%, 20=0.26%, 50=0.27% lat (msec) : 100=0.09%, 250=0.03%, 500=0.01%, 750=0.01%, 1000=0.01% cpu : usr=1.33%, sys=91.99%, ctx=297864, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,2485419,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 file1: (groupid=0, jobs=1): err= 0: pid=1367: Wed Dec 13 16:02:00 2017 write: IOPS=4130, BW=16.1MiB/s (16.9MB/s)(9681MiB/600001msec) slat (usec): min=3, max=836059, avg=203.03, stdev=1560.64 clat (usec): min=60, max=839666, avg=3651.98, stdev=6149.70 lat (usec): min=110, max=839905, avg=3859.61, stdev=6358.90 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 3], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 4], 60.00th=[ 4], | 70.00th=[ 4], 80.00th=[ 4], 90.00th=[ 5], 95.00th=[ 5], | 99.00th=[ 6], 99.50th=[ 16], 99.90th=[ 62], 99.95th=[ 87], | 99.99th=[ 334] bw ( KiB/s): min= 937, max=22484, per=12.50%, avg=16579.11, stdev=3330.88, samples=1198 iops : min= 234, max= 5621, avg=4144.45, stdev=832.70, samples=1198 lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.09%, 4=87.88%, 10=11.33%, 20=0.26%, 50=0.27% lat (msec) : 100=0.10%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.01% cpu : usr=1.28%, sys=91.92%, ctx=290466, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,2478319,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=129MiB/s (136MB/s), 15.8MiB/s-16.6MiB/s (16.6MB/s-17.4MB/s), io=75.9GiB (81.5GB), run=600001-600001msec Disk stats (read/write): vdb: ios=1/27804, merge=0/1743, ticks=1/4511916, in_queue=3364804, util=25.86%