fio ./seq-write.fio file1: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.0 Starting 8 processes Jobs: 8 (f=8): [W(8)][100.0%][r=0KiB/s,w=38.0MiB/s][r=0,w=9733 IOPS][eta 00m:00s] file1: (groupid=0, jobs=1): err= 0: pid=1153: Wed Dec 13 16:45:33 2017 write: IOPS=1570, BW=6284KiB/s (6435kB/s)(3682MiB/600002msec) slat (usec): min=11, max=2617.8k, avg=617.26, stdev=17969.35 clat (usec): min=13, max=2676.9k, avg=9558.41, stdev=71784.85 lat (usec): min=235, max=2677.4k, avg=10178.00, stdev=74245.06 clat percentiles (usec): | 1.00th=[ 865], 5.00th=[ 1336], 10.00th=[ 2008], | 20.00th=[ 2343], 30.00th=[ 2704], 40.00th=[ 2999], | 50.00th=[ 3359], 60.00th=[ 3621], 70.00th=[ 4228], | 80.00th=[ 5014], 90.00th=[ 6128], 95.00th=[ 7111], | 99.00th=[ 137364], 99.50th=[ 383779], 99.90th=[1082131], | 99.95th=[1535116], 99.99th=[2365588] bw ( KiB/s): min= 8, max=44184, per=14.97%, avg=7707.74, stdev=6514.82, samples=980 iops : min= 2, max=11046, avg=1926.59, stdev=1628.66, samples=980 lat (usec) : 20=0.01%, 250=0.04%, 500=0.06%, 750=0.12%, 1000=2.60% lat (msec) : 2=7.04%, 4=56.36%, 10=31.73%, 20=0.24%, 50=0.36% lat (msec) : 100=0.30%, 250=0.41%, 500=0.35%, 750=0.16%, 1000=0.09% lat (msec) : 2000=0.11%, >=2000=0.02% cpu : usr=0.36%, sys=40.40%, ctx=916341, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,942585,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 file1: (groupid=0, jobs=1): err= 0: pid=1154: Wed Dec 13 16:45:33 2017 write: IOPS=1438, BW=5753KiB/s (5891kB/s)(3371MiB/600001msec) slat (usec): min=11, max=2902.8k, avg=676.21, stdev=20444.71 clat (usec): min=23, max=4845.3k, avg=10439.08, stdev=82915.40 lat (usec): min=229, max=4845.5k, avg=11117.62, stdev=85919.93 clat percentiles (usec): | 1.00th=[ 832], 5.00th=[ 1237], 10.00th=[ 1827], | 20.00th=[ 2311], 30.00th=[ 2704], 40.00th=[ 2999], | 50.00th=[ 3261], 60.00th=[ 3490], 70.00th=[ 4015], | 80.00th=[ 4686], 90.00th=[ 5866], 95.00th=[ 6980], | 99.00th=[ 183501], 99.50th=[ 425722], 99.90th=[1333789], | 99.95th=[1971323], 99.99th=[2701132] bw ( KiB/s): min= 7, max=42276, per=14.19%, avg=7308.17, stdev=6556.14, samples=947 iops : min= 1, max=10569, avg=1826.70, stdev=1638.99, samples=947 lat (usec) : 50=0.01%, 250=0.05%, 500=0.07%, 750=0.18%, 1000=2.94% lat (msec) : 2=7.98%, 4=58.60%, 10=27.93%, 20=0.26%, 50=0.43% lat (msec) : 100=0.28%, 250=0.46%, 500=0.40%, 750=0.16%, 1000=0.12% lat (msec) : 2000=0.10%, >=2000=0.05% cpu : usr=0.29%, sys=36.57%, ctx=1089177, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,862978,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 file1: (groupid=0, jobs=1): err= 0: pid=1155: Wed Dec 13 16:45:33 2017 write: IOPS=1485, BW=5942KiB/s (6084kB/s)(3481MiB/600001msec) slat (usec): min=11, max=3129.5k, avg=654.20, stdev=19347.52 clat (usec): min=38, max=4083.4k, avg=10108.44, stdev=78745.11 lat (usec): min=231, max=4083.7k, avg=10764.95, stdev=81539.21 clat percentiles (usec): | 1.00th=[ 857], 5.00th=[ 1303], 10.00th=[ 1876], | 20.00th=[ 2376], 30.00th=[ 2769], 40.00th=[ 3032], | 50.00th=[ 3326], 60.00th=[ 3621], 70.00th=[ 4293], | 80.00th=[ 5014], 90.00th=[ 6128], 95.00th=[ 7111], | 99.00th=[ 149947], 99.50th=[ 408945], 99.90th=[1266680], | 99.95th=[1769997], 99.99th=[2399142] bw ( KiB/s): min= 8, max=40801, per=14.48%, avg=7458.17, stdev=6168.79, samples=958 iops : min= 2, max=10200, avg=1864.21, stdev=1542.13, samples=958 lat (usec) : 50=0.01%, 250=0.02%, 500=0.06%, 750=0.27%, 1000=2.63% lat (msec) : 2=8.13%, 4=55.45%, 10=31.31%, 20=0.25%, 50=0.37% lat (msec) : 100=0.30%, 250=0.42%, 500=0.35%, 750=0.18%, 1000=0.09% lat (msec) : 2000=0.11%, >=2000=0.03% cpu : usr=0.33%, sys=39.00%, ctx=996024, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,891228,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 file1: (groupid=0, jobs=1): err= 0: pid=1156: Wed Dec 13 16:45:33 2017 write: IOPS=1785, BW=7142KiB/s (7314kB/s)(4185MiB/600001msec) slat (usec): min=11, max=3142.7k, avg=540.50, stdev=15841.26 clat (usec): min=45, max=3915.5k, avg=8403.58, stdev=63204.82 lat (usec): min=231, max=3915.7k, avg=8946.44, stdev=65380.05 clat percentiles (usec): | 1.00th=[ 881], 5.00th=[ 1434], 10.00th=[ 2180], | 20.00th=[ 2442], 30.00th=[ 2868], 40.00th=[ 3195], | 50.00th=[ 3425], 60.00th=[ 3687], 70.00th=[ 4293], | 80.00th=[ 4948], 90.00th=[ 6063], 95.00th=[ 6980], | 99.00th=[ 85459], 99.50th=[ 295699], 99.90th=[1019216], | 99.95th=[1384121], 99.99th=[2365588] bw ( KiB/s): min= 16, max=37184, per=16.54%, avg=8516.84, stdev=6147.10, samples=1007 iops : min= 4, max= 9296, avg=2129.02, stdev=1536.76, samples=1007 lat (usec) : 50=0.01%, 250=0.02%, 500=0.06%, 750=0.13%, 1000=1.83% lat (msec) : 2=5.42%, 4=58.27%, 10=32.49%, 20=0.27%, 50=0.32% lat (msec) : 100=0.24%, 250=0.37%, 500=0.29%, 750=0.12%, 1000=0.06% lat (msec) : 2000=0.09%, >=2000=0.02% cpu : usr=0.39%, sys=46.14%, ctx=811687, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,1071347,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 file1: (groupid=0, jobs=1): err= 0: pid=1157: Wed Dec 13 16:45:33 2017 write: IOPS=1457, BW=5831KiB/s (5971kB/s)(3417MiB/600001msec) slat (usec): min=11, max=2990.9k, avg=666.92, stdev=19764.26 clat (usec): min=40, max=3392.7k, avg=10300.16, stdev=78330.46 lat (usec): min=231, max=3392.0k, avg=10969.40, stdev=80982.57 clat percentiles (usec): | 1.00th=[ 873], 5.00th=[ 1319], 10.00th=[ 1975], | 20.00th=[ 2311], 30.00th=[ 2737], 40.00th=[ 2999], | 50.00th=[ 3261], 60.00th=[ 3458], 70.00th=[ 3949], | 80.00th=[ 4686], 90.00th=[ 5800], 95.00th=[ 6980], | 99.00th=[ 191890], 99.50th=[ 450888], 99.90th=[1182794], | 99.95th=[1719665], 99.99th=[2466251] bw ( KiB/s): min= 8, max=34228, per=14.39%, avg=7411.45, stdev=6196.08, samples=946 iops : min= 2, max= 8557, avg=1852.53, stdev=1548.99, samples=946 lat (usec) : 50=0.01%, 250=0.01%, 500=0.09%, 750=0.26%, 1000=2.29% lat (msec) : 2=7.98%, 4=60.29%, 10=26.89%, 20=0.26%, 50=0.38% lat (msec) : 100=0.27%, 250=0.42%, 500=0.42%, 750=0.21%, 1000=0.10% lat (msec) : 2000=0.12%, >=2000=0.03% cpu : usr=0.32%, sys=36.90%, ctx=1018693, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,874628,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 file1: (groupid=0, jobs=1): err= 0: pid=1158: Wed Dec 13 16:45:33 2017 write: IOPS=1649, BW=6597KiB/s (6755kB/s)(3865MiB/600001msec) slat (usec): min=11, max=3143.6k, avg=586.92, stdev=17474.34 clat (usec): min=39, max=4084.4k, avg=9104.61, stdev=69703.21 lat (usec): min=202, max=4084.6k, avg=9693.89, stdev=72110.31 clat percentiles (usec): | 1.00th=[ 865], 5.00th=[ 1287], 10.00th=[ 2024], | 20.00th=[ 2409], 30.00th=[ 2835], 40.00th=[ 3163], | 50.00th=[ 3392], 60.00th=[ 3654], 70.00th=[ 4228], | 80.00th=[ 5014], 90.00th=[ 6128], 95.00th=[ 7111], | 99.00th=[ 119014], 99.50th=[ 354419], 99.90th=[1061159], | 99.95th=[1635779], 99.99th=[2432697] bw ( KiB/s): min= 8, max=40905, per=15.54%, avg=8003.47, stdev=6226.76, samples=990 iops : min= 2, max=10226, avg=2000.69, stdev=1556.67, samples=990 lat (usec) : 50=0.01%, 250=0.03%, 500=0.06%, 750=0.13%, 1000=2.61% lat (msec) : 2=6.82%, 4=56.22%, 10=32.18%, 20=0.27%, 50=0.37% lat (msec) : 100=0.24%, 250=0.40%, 500=0.32%, 750=0.14%, 1000=0.09% lat (msec) : 2000=0.09%, >=2000=0.02% cpu : usr=0.37%, sys=42.71%, ctx=873316, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,989567,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 file1: (groupid=0, jobs=1): err= 0: pid=1159: Wed Dec 13 16:45:33 2017 write: IOPS=1672, BW=6689KiB/s (6850kB/s)(3920MiB/600001msec) slat (usec): min=11, max=2633.0k, avg=578.72, stdev=17269.43 clat (usec): min=11, max=3011.0k, avg=8979.73, stdev=68815.08 lat (usec): min=237, max=3012.2k, avg=9560.80, stdev=71137.44 clat percentiles (usec): | 1.00th=[ 881], 5.00th=[ 1336], 10.00th=[ 1975], | 20.00th=[ 2376], 30.00th=[ 2802], 40.00th=[ 3064], | 50.00th=[ 3359], 60.00th=[ 3621], 70.00th=[ 4228], | 80.00th=[ 4883], 90.00th=[ 6063], 95.00th=[ 7046], | 99.00th=[ 117965], 99.50th=[ 346031], 99.90th=[1082131], | 99.95th=[1568670], 99.99th=[2499806] bw ( KiB/s): min= 8, max=42064, per=15.78%, avg=8128.29, stdev=6379.01, samples=986 iops : min= 2, max=10516, avg=2032.02, stdev=1594.76, samples=986 lat (usec) : 20=0.01%, 250=0.04%, 500=0.07%, 750=0.12%, 1000=2.44% lat (msec) : 2=7.78%, 4=56.46%, 10=31.22%, 20=0.26%, 50=0.32% lat (msec) : 100=0.24%, 250=0.40%, 500=0.32%, 750=0.15%, 1000=0.06% lat (msec) : 2000=0.10%, >=2000=0.02% cpu : usr=0.35%, sys=43.06%, ctx=957715, majf=0, minf=7 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,1003392,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 file1: (groupid=0, jobs=1): err= 0: pid=1160: Wed Dec 13 16:45:33 2017 write: IOPS=1816, BW=7264KiB/s (7438kB/s)(4256MiB/600001msec) slat (usec): min=12, max=3142.9k, avg=531.15, stdev=16191.13 clat (usec): min=47, max=3146.2k, avg=8270.14, stdev=63500.94 lat (usec): min=240, max=3146.4k, avg=8803.63, stdev=65645.85 clat percentiles (usec): | 1.00th=[ 873], 5.00th=[ 1401], 10.00th=[ 2114], | 20.00th=[ 2409], 30.00th=[ 2802], 40.00th=[ 3064], | 50.00th=[ 3359], 60.00th=[ 3621], 70.00th=[ 4228], | 80.00th=[ 4883], 90.00th=[ 5997], 95.00th=[ 6980], | 99.00th=[ 74974], 99.50th=[ 287310], 99.90th=[1002439], | 99.95th=[1434452], 99.99th=[2365588] bw ( KiB/s): min= 8, max=26781, per=17.00%, avg=8756.18, stdev=6125.17, samples=998 iops : min= 2, max= 6695, avg=2188.69, stdev=1531.26, samples=998 lat (usec) : 50=0.01%, 250=0.01%, 500=0.02%, 750=0.12%, 1000=2.32% lat (msec) : 2=5.61%, 4=58.24%, 10=31.92%, 20=0.27%, 50=0.35% lat (msec) : 100=0.22%, 250=0.38%, 500=0.27%, 750=0.12%, 1000=0.06% lat (msec) : 2000=0.09%, >=2000=0.01% cpu : usr=0.39%, sys=46.33%, ctx=803875, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,1089611,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=50.3MiB/s (52.7MB/s), 5753KiB/s-7264KiB/s (5891kB/s-7438kB/s), io=29.5GiB (31.6GB), run=600001-600002msec