lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Date:   Thu, 27 Feb 2020 10:27:16 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     Vincent Guittot <vincent.guittot@...aro.org>
Cc:     0day robot <lkp@...el.com>, LKML <linux-kernel@...r.kernel.org>,
        lkp@...ts.01.org
Subject: [sched/numa] 6942795656: fio.read_bw_MBps -37.5% regression

Greeting,

FYI, we noticed a -37.5% regression of fio.read_bw_MBps due to commit:


commit: 6942795656e5c8906b059622a23352a2e3c38d06 ("sched/numa: Replace runnable_load_avg by load_avg")
https://github.com/0day-ci/linux/commits/UPDATE-20200221-072833/Vincent-Guittot/remove-runnable_load_avg-and-improve-group_classify/20200217-172341

in testcase: fio-basic
on test machine: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 256G memory
with following parameters:

	disk: 2pmem
	fs: ext4
	mount_option: dax
	runtime: 200s
	nr_task: 50%
	time_based: tb
	rw: read
	bs: 2M
	ioengine: libaio
	test_size: 200G
	cpufreq_governor: performance
	ucode: 0x500002c

test-description: Fio is a tool that will spawn a number of threads or processes doing a particular type of I/O action as specified by the user.
test-url: https://github.com/axboe/fio



If you fix the issue, kindly add following tag
Reported-by: kernel test robot <rong.a.chen@...el.com>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
bs/compiler/cpufreq_governor/disk/fs/ioengine/kconfig/mount_option/nr_task/rootfs/runtime/rw/tbox_group/test_size/testcase/time_based/ucode:
  2M/gcc-7/performance/2pmem/ext4/libaio/x86_64-rhel-7.6/dax/50%/debian-x86_64-20191114.cgz/200s/read/lkp-csl-2sp6/200G/fio-basic/tb/0x500002c

commit: 
  1450cfa2e9 ("sched/fair: Reorder enqueue/dequeue_task_fair path")
  6942795656 ("sched/numa: Replace runnable_load_avg by load_avg")

1450cfa2e91bb42b 6942795656e5c8906b059622a23 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :4           25%           1:4     dmesg.WARNING:stack_recursion
         %stddev     %change         %stddev
             \          |                \  
     98.24           -64.9       33.30 ±  7%  fio.latency_100ms%
      0.11 ± 23%      +9.8        9.92 ± 26%  fio.latency_20ms%
      0.87 ± 13%     +32.5       33.34 ± 17%  fio.latency_250ms%
      0.02 ± 72%      +7.9        7.95 ± 29%  fio.latency_500ms%
      0.75 ± 61%     +14.7       15.48 ± 26%  fio.latency_50ms%
     44572           -37.5%      27854 ± 11%  fio.read_bw_MBps
  67633152          +218.6%  2.155e+08 ± 17%  fio.read_clat_90%_us
  68419584          +331.8%  2.954e+08 ± 12%  fio.read_clat_95%_us
  90701824 ± 13%    +297.7%  3.607e+08 ±  7%  fio.read_clat_99%_us
  66747576           +61.9%  1.081e+08 ± 10%  fio.read_clat_mean_us
   8444209 ±  8%    +864.2%   81414965 ± 11%  fio.read_clat_stddev
     22286           -37.5%      13927 ± 11%  fio.read_iops
   2137474           +61.9%    3460053 ± 10%  fio.read_slat_mean_us
    281650 ±  7%    +834.8%    2632906 ± 11%  fio.read_slat_stddev
     12498            +9.4%      13672 ±  2%  fio.time.involuntary_context_switches
    173045          +127.7%     394079 ± 10%  fio.time.minor_page_faults
   4457312           -37.5%    2785480 ± 11%  fio.workload
   7639552 ± 11%     -17.4%    6307328        meminfo.DirectMap2M
      2125 ±  5%     -21.8%       1661 ± 22%  slabinfo.task_group.active_objs
      2125 ±  5%     -21.8%       1661 ± 22%  slabinfo.task_group.num_objs
  18608329 ±  4%     -24.2%   14108299 ± 25%  cpuidle.C1E.usage
    949981 ± 97%    +476.5%    5477084 ± 65%  cpuidle.C6.usage
     15944 ±  4%    +187.3%      45809 ± 29%  cpuidle.POLL.time
      3274 ±  3%    +359.7%      15051 ± 33%  cpuidle.POLL.usage
    286841 ± 21%     -33.4%     191119 ±  7%  numa-numastat.node0.local_node
     10892 ± 62%     +98.8%      21653 ± 24%  numa-numastat.node0.other_node
    341302 ± 17%     +24.7%     425656 ±  3%  numa-numastat.node1.local_node
    361678 ± 15%     +20.3%     435249 ±  3%  numa-numastat.node1.numa_hit
  18606376 ±  4%     -24.2%   14106593 ± 25%  turbostat.C1E
    930856 ± 99%    +485.9%    5454072 ± 65%  turbostat.C6
    284.83            -9.1%     258.93        turbostat.PkgWatt
    194.88           -10.1%     175.20 ±  2%  turbostat.RAMWatt
     23333 ±  5%     +27.5%      29740 ±  4%  sched_debug.cfs_rq:/.exec_clock.stddev
      1031            +8.4%       1117 ±  5%  sched_debug.cfs_rq:/.load_avg.max
     24291 ±  5%     +28.7%      31264 ±  5%  sched_debug.cfs_rq:/.min_vruntime.stddev
     24291 ±  5%     +28.7%      31265 ±  5%  sched_debug.cfs_rq:/.spread0.stddev
    228.36 ±  3%     -12.7%     199.47 ±  8%  sched_debug.cfs_rq:/.util_est_enqueued.avg
    880.25           -42.1%     509.31        sched_debug.cfs_rq:/.util_est_enqueued.max
    213.90 ±  3%     -23.1%     164.53 ±  3%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
    504436           +15.4%     581941 ± 13%  sched_debug.cpu.max_idle_balance_cost.max
    450.46 ±173%   +1967.4%       9312 ± 92%  sched_debug.cpu.max_idle_balance_cost.stddev
      0.00 ±  3%     +33.3%       0.00 ± 32%  sched_debug.cpu.next_balance.stddev
    711.24 ±  3%     +25.5%     892.49 ± 14%  sched_debug.cpu.ttwu_count.stddev
     45086            -2.1%      44132        proc-vmstat.nr_slab_unreclaimable
     45832 ± 15%    +473.9%     263038 ± 16%  proc-vmstat.numa_hint_faults
     26159 ± 20%    +872.2%     254309 ± 15%  proc-vmstat.numa_hint_faults_local
    684142            -1.6%     673291        proc-vmstat.numa_hit
     19359 ±  7%    +152.7%      48919 ±  8%  proc-vmstat.numa_huge_pte_updates
    652862            -1.7%     642040        proc-vmstat.numa_local
    393344 ±  5%     -81.8%      71659 ± 36%  proc-vmstat.numa_pages_migrated
   9945716 ±  7%    +154.0%   25265990 ±  9%  proc-vmstat.numa_pte_updates
     18088 ±  6%     -15.8%      15223 ± 12%  proc-vmstat.pgactivate
   1869788           -17.0%    1551006        proc-vmstat.pgalloc_normal
    827276           +26.7%    1048183 ±  4%  proc-vmstat.pgfault
    393344 ±  5%     -81.8%      71659 ± 36%  proc-vmstat.pgmigrate_success
    419472 ±  3%     +73.9%     729478 ±  4%  numa-vmstat.node0.nr_active_anon
    419357 ±  3%     +74.0%     729487 ±  4%  numa-vmstat.node0.nr_anon_pages
    774.50 ±  2%     +76.0%       1363 ±  4%  numa-vmstat.node0.nr_anon_transparent_hugepages
    419480 ±  3%     +73.9%     729478 ±  4%  numa-vmstat.node0.nr_zone_active_anon
     12236 ± 56%    +375.3%      58162 ±103%  numa-vmstat.node0.numa_other
    433520 ±  2%     -71.7%     122601 ± 28%  numa-vmstat.node1.nr_active_anon
    424390 ±  3%     -73.2%     113726 ± 30%  numa-vmstat.node1.nr_anon_pages
    778.00 ±  2%     -75.6%     189.75 ± 34%  numa-vmstat.node1.nr_anon_transparent_hugepages
     55331 ± 91%     -94.9%       2798 ± 31%  numa-vmstat.node1.nr_inactive_anon
      7223 ±  8%     -13.2%       6266 ±  3%  numa-vmstat.node1.nr_kernel_stack
     55498 ± 91%     -93.5%       3634 ± 19%  numa-vmstat.node1.nr_mapped
      1499 ± 89%     -74.4%     383.25 ± 29%  numa-vmstat.node1.nr_page_table_pages
     64513 ± 77%     -81.7%      11805 ± 13%  numa-vmstat.node1.nr_shmem
    433520 ±  2%     -71.7%     122601 ± 28%  numa-vmstat.node1.nr_zone_active_anon
     55331 ± 91%     -94.9%       2798 ± 31%  numa-vmstat.node1.nr_zone_inactive_anon
   1680276 ±  3%     +74.3%    2928273 ±  4%  numa-meminfo.node0.Active
   1673650 ±  3%     +74.6%    2921940 ±  4%  numa-meminfo.node0.Active(anon)
   1585126 ±  3%     +76.4%    2796516 ±  4%  numa-meminfo.node0.AnonHugePages
   1673220 ±  3%     +74.6%    2921967 ±  4%  numa-meminfo.node0.AnonPages
   4473331 ±  5%     +34.7%    6026726 ±  2%  numa-meminfo.node0.MemUsed
   1731411 ±  3%     -71.6%     491655 ± 28%  numa-meminfo.node1.Active
   1730540 ±  3%     -71.7%     490490 ± 28%  numa-meminfo.node1.Active(anon)
   1592852 ±  2%     -75.5%     389474 ± 34%  numa-meminfo.node1.AnonHugePages
   1693936 ±  3%     -73.1%     454931 ± 30%  numa-meminfo.node1.AnonPages
    220820 ± 91%     -94.9%      11305 ± 31%  numa-meminfo.node1.Inactive
    220288 ± 91%     -94.9%      11149 ± 30%  numa-meminfo.node1.Inactive(anon)
      7222 ±  8%     -13.3%       6264 ±  3%  numa-meminfo.node1.KernelStack
    220659 ± 91%     -93.6%      14223 ± 18%  numa-meminfo.node1.Mapped
   4720887 ±  5%     -32.9%    3168169 ±  3%  numa-meminfo.node1.MemUsed
      5998 ± 89%     -74.4%       1534 ± 28%  numa-meminfo.node1.PageTables
    257078 ± 77%     -81.6%      47231 ± 13%  numa-meminfo.node1.Shmem
     35.47 ± 17%      -7.8       27.63 ± 19%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     35.05 ± 17%      -7.6       27.49 ± 18%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
     35.06 ± 17%      -7.6       27.51 ± 18%  perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     36.16 ± 16%      -7.5       28.65 ± 18%  perf-profile.calltrace.cycles-pp.secondary_startup_64
     35.33 ± 16%      -7.4       27.90 ± 18%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     35.33 ± 16%      -7.4       27.90 ± 18%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
     35.33 ± 16%      -7.4       27.90 ± 18%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
     35.47 ± 17%      -7.7       27.73 ± 18%  perf-profile.children.cycles-pp.intel_idle
     35.90 ± 17%      -7.6       28.25 ± 18%  perf-profile.children.cycles-pp.cpuidle_enter
     35.89 ± 17%      -7.6       28.24 ± 18%  perf-profile.children.cycles-pp.cpuidle_enter_state
     36.16 ± 16%      -7.5       28.65 ± 18%  perf-profile.children.cycles-pp.do_idle
     36.16 ± 16%      -7.5       28.65 ± 18%  perf-profile.children.cycles-pp.secondary_startup_64
     36.16 ± 16%      -7.5       28.65 ± 18%  perf-profile.children.cycles-pp.cpu_startup_entry
     35.33 ± 16%      -7.4       27.90 ± 18%  perf-profile.children.cycles-pp.start_secondary
      0.06 ± 14%      +0.0        0.08 ± 10%  perf-profile.children.cycles-pp.get_io_u
      0.04 ± 58%      +0.1        0.14 ± 15%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.23 ± 20%      +0.1        0.35 ± 18%  perf-profile.children.cycles-pp.scheduler_tick
     35.47 ± 17%      -7.8       27.71 ± 18%  perf-profile.self.cycles-pp.intel_idle
      0.06 ± 14%      +0.0        0.08 ± 10%  perf-profile.self.cycles-pp.get_io_u
      0.04 ± 58%      +0.1        0.14 ± 15%  perf-profile.self.cycles-pp.native_irq_return_iret
     42.56            -1.6%      41.89        perf-stat.i.MPKI
 5.983e+09           -36.2%  3.815e+09 ± 10%  perf-stat.i.branch-instructions
      0.09            +0.0        0.13 ±  5%  perf-stat.i.branch-miss-rate%
   4931406            -6.1%    4630438        perf-stat.i.branch-misses
     94.72            -0.8       93.88        perf-stat.i.cache-miss-rate%
 1.454e+09           -38.1%  8.999e+08 ± 11%  perf-stat.i.cache-misses
 1.528e+09           -37.5%  9.546e+08 ± 11%  perf-stat.i.cache-references
      3.79           +62.6%       6.16 ± 10%  perf-stat.i.cpi
     99.92            +4.4%     104.35        perf-stat.i.cpu-migrations
    104.53           +57.2%     164.30 ±  9%  perf-stat.i.cycles-between-cache-misses
      0.00 ± 24%      +0.0        0.00 ± 19%  perf-stat.i.dTLB-load-miss-rate%
 6.054e+09           -36.1%  3.868e+09 ± 10%  perf-stat.i.dTLB-loads
      0.00 ±  8%      +0.0        0.00 ±  6%  perf-stat.i.dTLB-store-miss-rate%
     37244 ± 16%     +27.9%      47636 ±  7%  perf-stat.i.dTLB-store-misses
 5.943e+09           -36.6%  3.767e+09 ± 10%  perf-stat.i.dTLB-stores
     44.62            +4.0       48.59        perf-stat.i.iTLB-load-miss-rate%
   1751240 ±  2%     -14.5%    1496904 ±  4%  perf-stat.i.iTLB-load-misses
   2173785           -27.0%    1587288 ±  4%  perf-stat.i.iTLB-loads
  3.57e+10           -36.4%   2.27e+10 ± 10%  perf-stat.i.instructions
     20430 ±  2%     -25.4%      15243 ±  6%  perf-stat.i.instructions-per-iTLB-miss
      0.27           -36.0%       0.17 ± 10%  perf-stat.i.ipc
      3552           +32.2%       4697 ±  4%  perf-stat.i.minor-faults
     46.28 ± 19%     +10.3       56.62 ±  6%  perf-stat.i.node-load-miss-rate%
  67879041 ± 18%     -47.8%   35408124 ± 16%  perf-stat.i.node-loads
      0.65 ±  4%     +34.1       34.79 ± 12%  perf-stat.i.node-store-miss-rate%
   1231258 ± 17%   +6328.0%   79144992 ±  5%  perf-stat.i.node-store-misses
  3.66e+08           -55.9%  1.616e+08 ± 14%  perf-stat.i.node-stores
      3552           +32.2%       4697 ±  4%  perf-stat.i.page-faults
     42.80            -1.7%      42.06        perf-stat.overall.MPKI
      0.08            +0.0        0.12 ±  8%  perf-stat.overall.branch-miss-rate%
     95.15            -0.9       94.26        perf-stat.overall.cache-miss-rate%
      3.79           +59.5%       6.05 ± 10%  perf-stat.overall.cpi
     93.08           +63.9%     152.59 ± 10%  perf-stat.overall.cycles-between-cache-misses
      0.00 ± 26%      +0.0        0.00 ±  6%  perf-stat.overall.dTLB-load-miss-rate%
      0.00 ± 16%      +0.0        0.00 ± 16%  perf-stat.overall.dTLB-store-miss-rate%
     44.61            +3.9       48.51        perf-stat.overall.iTLB-load-miss-rate%
     20415 ±  2%     -26.1%      15095 ±  6%  perf-stat.overall.instructions-per-iTLB-miss
      0.26           -36.6%       0.17 ± 10%  perf-stat.overall.ipc
     46.12 ± 19%     +10.1       56.26 ±  6%  perf-stat.overall.node-load-miss-rate%
      0.34 ± 17%     +33.2       33.58 ± 12%  perf-stat.overall.node-store-miss-rate%
   1606253            +1.4%    1628237        perf-stat.overall.path-length
 5.954e+09           -36.4%  3.784e+09 ± 10%  perf-stat.ps.branch-instructions
   4924598            -7.7%    4547566        perf-stat.ps.branch-misses
 1.447e+09           -38.3%  8.931e+08 ± 11%  perf-stat.ps.cache-misses
  1.52e+09           -37.7%  9.472e+08 ± 11%  perf-stat.ps.cache-references
     98.88            +4.9%     103.73        perf-stat.ps.cpu-migrations
 6.024e+09           -36.3%  3.837e+09 ± 10%  perf-stat.ps.dTLB-loads
     36708 ± 16%     +26.7%      46499 ±  7%  perf-stat.ps.dTLB-store-misses
 5.914e+09           -36.8%  3.737e+09 ± 10%  perf-stat.ps.dTLB-stores
   1741292 ±  2%     -14.6%    1487851 ±  4%  perf-stat.ps.iTLB-load-misses
   2161209           -26.9%    1579771 ±  4%  perf-stat.ps.iTLB-loads
 3.552e+10           -36.6%  2.251e+10 ± 10%  perf-stat.ps.instructions
      3479           +31.1%       4562 ±  4%  perf-stat.ps.minor-faults
  67539545 ± 18%     -48.1%   35041905 ± 16%  perf-stat.ps.node-loads
   1241582 ± 17%   +6292.6%   79369525 ±  5%  perf-stat.ps.node-store-misses
 3.642e+08           -56.2%  1.594e+08 ± 14%  perf-stat.ps.node-stores
      3479           +31.1%       4562 ±  4%  perf-stat.ps.page-faults
  7.16e+12           -36.7%  4.533e+12 ± 10%  perf-stat.total.instructions
     87848 ±  2%     +13.1%      99392        softirqs.CPU0.RCU
     77399 ±  5%     +16.8%      90404 ±  3%  softirqs.CPU1.RCU
     20680 ± 10%     -38.0%      12829 ± 11%  softirqs.CPU12.SCHED
     63366 ±  9%     +34.6%      85282 ±  6%  softirqs.CPU2.RCU
     14411 ± 17%     +42.3%      20503 ± 14%  softirqs.CPU2.SCHED
     71375 ±  2%     -31.7%      48730 ±  8%  softirqs.CPU24.RCU
     18288 ±  6%     -70.8%       5340 ± 24%  softirqs.CPU24.SCHED
     13435 ± 25%     -48.2%       6954 ± 14%  softirqs.CPU25.SCHED
     75740 ±  4%     -24.8%      56985 ±  7%  softirqs.CPU26.RCU
     21066 ±  6%     -58.0%       8842 ± 12%  softirqs.CPU26.SCHED
     55879 ± 11%     +21.2%      67724 ±  7%  softirqs.CPU29.RCU
     72914 ±  5%      +7.4%      78320 ±  4%  softirqs.CPU30.TIMER
     73132 ±  5%     +12.4%      82220 ±  5%  softirqs.CPU32.TIMER
     57542 ± 19%     +29.8%      74664 ±  4%  softirqs.CPU33.RCU
     10736 ± 47%     +77.5%      19062 ±  5%  softirqs.CPU33.SCHED
     73232 ±  3%     +11.8%      81879 ±  6%  softirqs.CPU33.TIMER
     73807 ±  7%     +17.5%      86690 ±  7%  softirqs.CPU34.TIMER
     67653 ±  6%     +15.3%      78030 ±  3%  softirqs.CPU36.RCU
     15841 ± 17%     +25.8%      19930 ±  7%  softirqs.CPU36.SCHED
     72518 ±  6%     +17.9%      85512 ± 11%  softirqs.CPU36.TIMER
     52732 ± 22%     +43.5%      75670 ±  9%  softirqs.CPU37.RCU
      9735 ± 46%     +87.2%      18226 ± 18%  softirqs.CPU37.SCHED
     56946 ±  9%     +30.4%      74270 ±  8%  softirqs.CPU42.RCU
     11172 ± 20%     +65.0%      18438 ± 18%  softirqs.CPU42.SCHED
     74163 ±  5%     +17.4%      87089 ±  7%  softirqs.CPU42.TIMER
     67216 ± 13%     +15.6%      77708 ±  3%  softirqs.CPU43.RCU
     15988 ± 23%     +28.6%      20568 ±  7%  softirqs.CPU43.SCHED
     72190 ±  4%     +17.0%      84445 ±  7%  softirqs.CPU43.TIMER
     71178 ±  5%     +13.4%      80709 ± 10%  softirqs.CPU44.TIMER
     71448 ±  3%     +19.6%      85485 ±  8%  softirqs.CPU45.TIMER
     72100 ±  6%     +16.4%      83936 ±  8%  softirqs.CPU46.TIMER
     63991 ± 10%     +16.0%      74252 ±  5%  softirqs.CPU47.RCU
      9100 ± 26%     +54.8%      14086 ± 13%  softirqs.CPU47.SCHED
     74711 ±  4%     +15.1%      86028 ±  9%  softirqs.CPU47.TIMER
     42540 ±  4%     +15.2%      49011        softirqs.CPU48.RCU
     70569 ±  7%     -20.5%      56123 ±  4%  softirqs.CPU50.RCU
     16387 ± 16%     -56.3%       7154 ± 16%  softirqs.CPU50.SCHED
      9666 ± 15%     -29.0%       6867 ± 26%  softirqs.CPU51.SCHED
     71206 ±  3%     -25.2%      53245 ±  3%  softirqs.CPU54.RCU
     17130 ±  9%     -64.3%       6114 ± 12%  softirqs.CPU54.SCHED
      9188 ± 23%     -43.1%       5225 ± 17%  softirqs.CPU55.SCHED
     11532 ± 37%     -50.6%       5701 ± 23%  softirqs.CPU56.SCHED
     67349 ±  8%     -25.9%      49908 ±  3%  softirqs.CPU58.RCU
     15473 ± 20%     -65.2%       5382 ± 11%  softirqs.CPU58.SCHED
     60278 ±  5%     +25.4%      75582 ± 10%  softirqs.CPU6.RCU
     12769 ± 11%     +27.4%      16268 ± 23%  softirqs.CPU6.SCHED
      9234 ± 26%     -43.5%       5213 ±  6%  softirqs.CPU60.SCHED
     63277 ±  9%     -21.4%      49708        softirqs.CPU61.RCU
     13706 ± 22%     -63.6%       4989 ±  6%  softirqs.CPU61.SCHED
     61652 ± 15%     -20.6%      48935 ±  3%  softirqs.CPU64.RCU
     14778 ± 35%     -70.0%       4438 ±  5%  softirqs.CPU64.SCHED
     10062 ± 68%     -54.9%       4538 ± 13%  softirqs.CPU65.SCHED
     67805 ± 17%     -31.0%      46752        softirqs.CPU66.RCU
     18786 ± 30%     -79.0%       3947        softirqs.CPU66.SCHED
     67465 ± 14%     -28.6%      48149 ±  3%  softirqs.CPU67.RCU
     17232 ± 29%     -75.7%       4179 ± 15%  softirqs.CPU67.SCHED
     65598 ± 15%     -26.5%      48222 ±  4%  softirqs.CPU68.RCU
     16849 ± 36%     -75.7%       4092 ± 14%  softirqs.CPU68.SCHED
     57812 ± 12%     -19.6%      46482 ±  3%  softirqs.CPU70.RCU
     12805 ± 30%     -70.9%       3728 ± 14%  softirqs.CPU70.SCHED
     14209 ± 50%     -74.9%       3570 ±  5%  softirqs.CPU71.SCHED
     59819           +59.6%      95469        softirqs.CPU72.RCU
     12222 ±  7%    +103.0%      24811 ±  5%  softirqs.CPU72.SCHED
     67645 ± 10%     +37.2%      92826 ±  3%  softirqs.CPU73.RCU
     16869 ± 20%     +40.1%      23636 ±  6%  softirqs.CPU73.SCHED
     52613 ± 10%     +77.5%      93403 ±  5%  softirqs.CPU74.RCU
      9444 ± 19%    +156.0%      24173 ±  9%  softirqs.CPU74.SCHED
     62708 ± 18%     +52.1%      95389        softirqs.CPU75.RCU
     14321 ± 36%     +70.8%      24455 ±  6%  softirqs.CPU75.SCHED
     71060 ± 11%     +28.0%      90963        softirqs.CPU77.RCU
     71421 ±  3%      +9.2%      77978 ±  5%  softirqs.CPU77.TIMER
     65662 ±  5%     +40.3%      92151        softirqs.CPU78.RCU
     16183 ± 15%     +43.8%      23267 ±  4%  softirqs.CPU78.SCHED
     71822 ±  5%      +7.2%      77025 ±  4%  softirqs.CPU78.TIMER
     63433 ±  5%     +39.0%      88177 ±  9%  softirqs.CPU79.RCU
     15151 ± 11%     +43.9%      21807 ± 15%  softirqs.CPU79.SCHED
     67711 ±  8%     +29.6%      87778 ±  3%  softirqs.CPU80.RCU
     16505 ± 14%     +47.8%      24401 ±  3%  softirqs.CPU80.SCHED
     72148 ±  4%     +11.9%      80767 ±  3%  softirqs.CPU80.TIMER
     70572 ±  5%     +16.9%      82478 ±  5%  softirqs.CPU81.TIMER
     72939 ±  5%     +18.5%      86441 ±  8%  softirqs.CPU82.TIMER
     67040 ±  7%     +21.0%      81095 ±  5%  softirqs.CPU83.RCU
     16105 ± 17%     +45.3%      23396 ±  8%  softirqs.CPU83.SCHED
     72257 ±  2%     +14.8%      82978 ±  8%  softirqs.CPU83.TIMER
     62802 ± 12%     +34.0%      84185 ±  7%  softirqs.CPU84.RCU
     14216 ± 22%     +61.7%      22988 ± 11%  softirqs.CPU84.SCHED
     72891 ±  5%     +17.2%      85435 ± 10%  softirqs.CPU84.TIMER
     71482 ±  5%     +17.1%      83677 ±  9%  softirqs.CPU87.TIMER
     73030 ±  6%     +15.4%      84305 ± 10%  softirqs.CPU89.TIMER
     71813 ±  8%     +19.6%      85872 ±  3%  softirqs.CPU90.RCU
     19033 ± 12%     +27.5%      24258        softirqs.CPU90.SCHED
     71728 ±  5%     +21.6%      87204 ±  6%  softirqs.CPU90.TIMER
     62569 ± 12%     +23.2%      77080        softirqs.CPU91.RCU
     14100 ± 26%     +52.4%      21492 ±  4%  softirqs.CPU91.SCHED
     72287 ±  6%     +16.8%      84404 ±  6%  softirqs.CPU91.TIMER
     58336 ± 13%     +40.3%      81856 ± 12%  softirqs.CPU92.RCU
     12005 ± 27%     +84.9%      22195 ± 17%  softirqs.CPU92.SCHED
     62597 ±  5%     +31.8%      82526 ±  7%  softirqs.CPU93.RCU
     13640 ±  9%     +67.7%      22870 ± 10%  softirqs.CPU93.SCHED
     72926 ±  5%     +17.8%      85913 ±  6%  softirqs.CPU93.TIMER
     59097 ± 16%     +33.6%      78962 ±  5%  softirqs.CPU94.RCU
     12406 ± 40%     +57.1%      19486 ±  9%  softirqs.CPU94.SCHED
     72443 ±  3%     +15.9%      83989 ±  8%  softirqs.CPU94.TIMER
     64865 ±  9%     +26.9%      82333 ±  7%  softirqs.CPU95.RCU
     15528 ± 21%     +42.5%      22123 ± 10%  softirqs.CPU95.SCHED
      2.50 ±173%   +6150.0%     156.25 ± 61%  interrupts.62:PCI-MSI.31981595-edge.i40e-eth0-TxRx-26
      0.00       +2.6e+103%      25.75 ±166%  interrupts.93:PCI-MSI.31981626-edge.i40e-eth0-TxRx-57
      4613 ± 24%     -26.3%       3397 ± 33%  interrupts.CPU0.NMI:Non-maskable_interrupts
      4613 ± 24%     -26.3%       3397 ± 33%  interrupts.CPU0.PMI:Performance_monitoring_interrupts
      7153 ± 11%     -26.9%       5226 ± 21%  interrupts.CPU11.NMI:Non-maskable_interrupts
      7153 ± 11%     -26.9%       5226 ± 21%  interrupts.CPU11.PMI:Performance_monitoring_interrupts
      4589 ± 25%     -29.9%       3217 ± 35%  interrupts.CPU12.NMI:Non-maskable_interrupts
      4589 ± 25%     -29.9%       3217 ± 35%  interrupts.CPU12.PMI:Performance_monitoring_interrupts
     72.50 ± 12%     +85.2%     134.25 ±  6%  interrupts.CPU12.RES:Rescheduling_interrupts
     90.50 ± 23%     +40.6%     127.25 ± 12%  interrupts.CPU13.RES:Rescheduling_interrupts
     56.75 ± 14%     +81.5%     103.00 ± 45%  interrupts.CPU16.TLB:TLB_shootdowns
    115.50 ± 10%    +162.3%     303.00 ± 49%  interrupts.CPU2.RES:Rescheduling_interrupts
     94.75 ± 20%     +45.9%     138.25 ±  5%  interrupts.CPU22.RES:Rescheduling_interrupts
     45.25 ± 21%     +86.7%      84.50 ± 29%  interrupts.CPU22.TLB:TLB_shootdowns
    105.75 ±  6%     +79.0%     189.25 ±  8%  interrupts.CPU24.RES:Rescheduling_interrupts
    482.50 ±  4%      -6.2%     452.75        interrupts.CPU25.CAL:Function_call_interrupts
      7319 ±  8%     -25.0%       5487 ±  7%  interrupts.CPU25.NMI:Non-maskable_interrupts
      7319 ±  8%     -25.0%       5487 ±  7%  interrupts.CPU25.PMI:Performance_monitoring_interrupts
    135.25 ±  7%     +31.4%     177.75 ±  3%  interrupts.CPU25.RES:Rescheduling_interrupts
    392.00 ± 64%     -65.2%     136.50 ± 98%  interrupts.CPU25.TLB:TLB_shootdowns
     81.00 ±  8%    +100.3%     162.25 ± 12%  interrupts.CPU26.RES:Rescheduling_interrupts
    346.00 ± 60%     -81.8%      63.00 ± 94%  interrupts.CPU27.TLB:TLB_shootdowns
    490.00 ±  5%      -7.6%     453.00        interrupts.CPU29.CAL:Function_call_interrupts
      6654 ± 15%     -29.3%       4702 ± 37%  interrupts.CPU30.NMI:Non-maskable_interrupts
      6654 ± 15%     -29.3%       4702 ± 37%  interrupts.CPU30.PMI:Performance_monitoring_interrupts
    134.50 ± 23%     -40.1%      80.50 ±  6%  interrupts.CPU33.RES:Rescheduling_interrupts
    124.25 ± 21%     -51.3%      60.50 ±  9%  interrupts.CPU34.RES:Rescheduling_interrupts
    200.00 ±103%     -87.9%      24.25 ± 53%  interrupts.CPU35.TLB:TLB_shootdowns
     72.75 ± 44%     -77.3%      16.50 ± 87%  interrupts.CPU41.TLB:TLB_shootdowns
    103.25 ± 23%     -30.0%      72.25 ± 33%  interrupts.CPU43.RES:Rescheduling_interrupts
    207.25 ± 97%     -86.5%      28.00 ± 38%  interrupts.CPU43.TLB:TLB_shootdowns
     75.50 ± 52%     -67.2%      24.75 ± 73%  interrupts.CPU44.TLB:TLB_shootdowns
    217.50 ± 90%     -91.6%      18.25 ± 11%  interrupts.CPU46.TLB:TLB_shootdowns
      7612           -68.2%       2423 ± 65%  interrupts.CPU47.NMI:Non-maskable_interrupts
      7612           -68.2%       2423 ± 65%  interrupts.CPU47.PMI:Performance_monitoring_interrupts
     33.00 ± 67%    +126.5%      74.75 ± 23%  interrupts.CPU5.TLB:TLB_shootdowns
    105.25 ± 20%     +58.0%     166.25 ±  9%  interrupts.CPU50.RES:Rescheduling_interrupts
    108.50 ± 13%     +43.3%     155.50 ±  3%  interrupts.CPU54.RES:Rescheduling_interrupts
    141.25 ±  6%     +14.5%     161.75 ±  4%  interrupts.CPU55.RES:Rescheduling_interrupts
      6918 ± 12%     -27.5%       5014 ± 32%  interrupts.CPU56.NMI:Non-maskable_interrupts
      6918 ± 12%     -27.5%       5014 ± 32%  interrupts.CPU56.PMI:Performance_monitoring_interrupts
    109.75 ± 20%     +49.4%     164.00 ±  3%  interrupts.CPU58.RES:Rescheduling_interrupts
    138.00 ±  9%     +21.7%     168.00 ±  5%  interrupts.CPU60.RES:Rescheduling_interrupts
    113.00 ± 11%     +46.9%     166.00 ±  4%  interrupts.CPU61.RES:Rescheduling_interrupts
    100.00 ± 34%     +67.8%     167.75        interrupts.CPU64.RES:Rescheduling_interrupts
     93.75 ± 25%     +80.0%     168.75 ±  3%  interrupts.CPU66.RES:Rescheduling_interrupts
    111.00 ± 23%     +52.0%     168.75        interrupts.CPU67.RES:Rescheduling_interrupts
    111.00 ± 23%     +54.1%     171.00 ±  2%  interrupts.CPU68.RES:Rescheduling_interrupts
    124.75 ±  9%     +37.1%     171.00 ±  2%  interrupts.CPU70.RES:Rescheduling_interrupts
    114.50 ± 32%     +55.7%     178.25 ±  5%  interrupts.CPU71.RES:Rescheduling_interrupts
    131.75 ± 16%     -74.2%      34.00 ± 48%  interrupts.CPU72.RES:Rescheduling_interrupts
     81.25 ± 58%     -80.0%      16.25 ± 59%  interrupts.CPU72.TLB:TLB_shootdowns
     98.50 ± 19%     -58.4%      41.00 ± 38%  interrupts.CPU73.RES:Rescheduling_interrupts
    209.75 ± 96%     -88.3%      24.50 ± 65%  interrupts.CPU73.TLB:TLB_shootdowns
      6712 ± 24%     -41.9%       3897 ± 35%  interrupts.CPU74.NMI:Non-maskable_interrupts
      6712 ± 24%     -41.9%       3897 ± 35%  interrupts.CPU74.PMI:Performance_monitoring_interrupts
    152.00 ±  7%     -71.4%      43.50 ± 54%  interrupts.CPU74.RES:Rescheduling_interrupts
    211.25 ± 91%     -93.7%      13.25 ± 32%  interrupts.CPU74.TLB:TLB_shootdowns
      6104 ± 24%     -51.9%       2938 ± 20%  interrupts.CPU75.NMI:Non-maskable_interrupts
      6104 ± 24%     -51.9%       2938 ± 20%  interrupts.CPU75.PMI:Performance_monitoring_interrupts
    119.50 ± 22%     -74.7%      30.25 ± 55%  interrupts.CPU75.RES:Rescheduling_interrupts
    103.75 ±  5%     -44.3%      57.75 ± 52%  interrupts.CPU76.RES:Rescheduling_interrupts
     90.25 ± 54%     -59.0%      37.00 ± 33%  interrupts.CPU76.TLB:TLB_shootdowns
      5961 ± 27%     -54.1%       2733 ± 49%  interrupts.CPU77.NMI:Non-maskable_interrupts
      5961 ± 27%     -54.1%       2733 ± 49%  interrupts.CPU77.PMI:Performance_monitoring_interrupts
     96.75 ± 45%     -58.7%      40.00 ± 69%  interrupts.CPU77.TLB:TLB_shootdowns
      5637 ± 26%     -63.5%       2057 ± 27%  interrupts.CPU78.NMI:Non-maskable_interrupts
      5637 ± 26%     -63.5%       2057 ± 27%  interrupts.CPU78.PMI:Performance_monitoring_interrupts
    119.00 ± 23%     -59.2%      48.50 ± 13%  interrupts.CPU78.RES:Rescheduling_interrupts
      6907 ± 12%     -64.2%       2474 ± 43%  interrupts.CPU79.NMI:Non-maskable_interrupts
      6907 ± 12%     -64.2%       2474 ± 43%  interrupts.CPU79.PMI:Performance_monitoring_interrupts
    119.50 ± 12%     -53.8%      55.25 ± 33%  interrupts.CPU79.RES:Rescheduling_interrupts
     91.75 ± 40%     -67.0%      30.25 ± 28%  interrupts.CPU79.TLB:TLB_shootdowns
     86.00 ± 35%     +56.7%     134.75 ± 18%  interrupts.CPU8.RES:Rescheduling_interrupts
      6728 ± 25%     -48.5%       3462 ± 52%  interrupts.CPU80.NMI:Non-maskable_interrupts
      6728 ± 25%     -48.5%       3462 ± 52%  interrupts.CPU80.PMI:Performance_monitoring_interrupts
    100.50 ± 16%     -40.8%      59.50 ± 16%  interrupts.CPU80.RES:Rescheduling_interrupts
     84.25 ± 50%     -68.2%      26.75 ± 13%  interrupts.CPU82.TLB:TLB_shootdowns
    521.75 ±  4%     -16.1%     437.75 ± 18%  interrupts.CPU83.CAL:Function_call_interrupts
    127.50 ± 45%     -57.3%      54.50 ± 38%  interrupts.CPU83.RES:Rescheduling_interrupts
     82.00 ± 51%     -61.6%      31.50 ± 28%  interrupts.CPU83.TLB:TLB_shootdowns
      7620           -68.2%       2425 ± 40%  interrupts.CPU84.NMI:Non-maskable_interrupts
      7620           -68.2%       2425 ± 40%  interrupts.CPU84.PMI:Performance_monitoring_interrupts
    116.00 ± 17%     -52.6%      55.00 ± 16%  interrupts.CPU84.RES:Rescheduling_interrupts
    522.50 ±  5%      -6.4%     489.25        interrupts.CPU85.CAL:Function_call_interrupts
     84.25 ± 53%     -65.6%      29.00 ± 43%  interrupts.CPU85.TLB:TLB_shootdowns
    523.00 ±  5%      -7.2%     485.50        interrupts.CPU86.CAL:Function_call_interrupts
     74.50 ± 52%     -68.5%      23.50 ± 25%  interrupts.CPU86.TLB:TLB_shootdowns
     88.00 ± 53%     -71.6%      25.00 ± 33%  interrupts.CPU89.TLB:TLB_shootdowns
    107.75 ± 24%     -53.4%      50.25 ± 10%  interrupts.CPU90.RES:Rescheduling_interrupts
    218.25 ± 86%     -85.0%      32.75 ± 38%  interrupts.CPU90.TLB:TLB_shootdowns
    127.00 ± 12%     -33.1%      85.00 ± 25%  interrupts.CPU91.RES:Rescheduling_interrupts
     81.75 ± 50%     -65.4%      28.25 ± 18%  interrupts.CPU91.TLB:TLB_shootdowns
      6328 ± 12%     -52.9%       2983 ± 40%  interrupts.CPU92.NMI:Non-maskable_interrupts
      6328 ± 12%     -52.9%       2983 ± 40%  interrupts.CPU92.PMI:Performance_monitoring_interrupts
    125.25 ± 16%     -51.9%      60.25 ± 31%  interrupts.CPU92.RES:Rescheduling_interrupts
      6390 ± 21%     -50.9%       3134 ± 45%  interrupts.CPU93.NMI:Non-maskable_interrupts
      6390 ± 21%     -50.9%       3134 ± 45%  interrupts.CPU93.PMI:Performance_monitoring_interrupts
    137.00 ±  9%     -55.7%      60.75 ± 23%  interrupts.CPU93.RES:Rescheduling_interrupts
     85.50 ± 56%     -65.5%      29.50 ± 10%  interrupts.CPU93.TLB:TLB_shootdowns
    531.75 ±  6%      -7.9%     489.75        interrupts.CPU94.CAL:Function_call_interrupts
    127.25 ± 26%     -52.1%      61.00 ± 41%  interrupts.CPU94.RES:Rescheduling_interrupts
     89.00 ± 57%     -67.1%      29.25 ±  5%  interrupts.CPU94.TLB:TLB_shootdowns
    531.00 ±  7%      -8.2%     487.50 ±  2%  interrupts.CPU95.CAL:Function_call_interrupts
      4741 ± 24%     -52.3%       2260 ± 74%  interrupts.CPU95.NMI:Non-maskable_interrupts
      4741 ± 24%     -52.3%       2260 ± 74%  interrupts.CPU95.PMI:Performance_monitoring_interrupts
    342.00 ± 45%     -77.8%      76.00 ± 65%  interrupts.CPU95.TLB:TLB_shootdowns
     11178 ± 24%     -48.7%       5734 ± 26%  interrupts.TLB:TLB_shootdowns


                                                                                
                                  fio.read_bw_MBps                              
                                                                                
  45000 +-------------------------------------------------------------------+   
        |                                                                   |   
  40000 |-+                                                                 |   
  35000 |-+                                        O      O O      O        |   
        | O  O O   O  O   O             O O O        O  O     O  O     O    |   
  30000 |-+      O      O            O                               O      |   
  25000 |-+                    O O             O O                        O |   
        |                   O                                               |   
  20000 |-+                                                                 |   
  15000 |-+                                                                 |   
        |                                                                   |   
  10000 |-+                                                                 |   
   5000 |-+                                                                 |   
        |                                                                   |   
      0 +-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                    fio.read_iops                               
                                                                                
  25000 +-------------------------------------------------------------------+   
        |                              .+.+.    .+.      .+                 |   
        |.+..+.+.+.+..+.+.+.+..+.+.+.+.     +..+   +.+..+                   |   
  20000 |-+                                                                 |   
        |                                          O      O O      O        |   
        | O  O O   O  O   O             O O O        O  O     O  O     O    |   
  15000 |-+      O      O            O                               O      |   
        |                      O O             O O                        O |   
  10000 |-+                 O                                               |   
        |                                                                   |   
        |                                                                   |   
   5000 |-+                                                                 |   
        |                                                                   |   
        |                                                                   |   
      0 +-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                 fio.read_clat_mean_us                          
                                                                                
  1.4e+08 +-----------------------------------------------------------------+   
          |                   O                                             |   
  1.2e+08 |-+                                                             O |   
          |                     O O    O       O  O                         |   
    1e+08 |-+    O O     O               O O          O O      O O   O      |   
          | O O      O O    O                O                          O   |   
    8e+07 |-+                                       O     O  O     O        |   
          |.+.+..+.+.+.+.+..+.+.+.+.+..+.+.+.+.+..+.+.+.+.+                 |   
    6e+07 |-+                                                               |   
          |                                                                 |   
    4e+07 |-+                                                               |   
          |                                                                 |   
    2e+07 |-+                                                               |   
          |                                                                 |   
        0 +-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                fio.read_clat_stddev                            
                                                                                
  1e+08 +-------------------------------------------------------------------+   
  9e+07 |-+                 O  O O                                          |   
        |      O                               O                     O      |   
  8e+07 |-+      O O    O            O           O                        O |   
  7e+07 |-+                             O O O           O        O          |   
        |                                            O        O        O    |   
  6e+07 |-O  O        O   O                        O        O               |   
  5e+07 |-+                                               O        O        |   
  4e+07 |-+                                                                 |   
        |                                                                   |   
  3e+07 |-+                                                                 |   
  2e+07 |-+                                                                 |   
        |   .+.               .+. .+.                                       |   
  1e+07 |.+.   +.+.+..+.+.+.+.   +   +..+.+.+..+.+.+.+..+.+                 |   
      0 +-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                 fio.read_clat_90__us                           
                                                                                
    3e+08 +-----------------------------------------------------------------+   
          |                                    O                            |   
  2.5e+08 |-+                                                               |   
          |                     O O                                  O      |   
          |                                                                 |   
    2e+08 |-+    O O O   O             O          O                       O |   
          |                              O O O        O O      O O      O   |   
  1.5e+08 |-O O        O    O                       O     O  O     O        |   
          |                                                                 |   
    1e+08 |-+                                                               |   
          |                                                                 |   
          |.+.+..+.+.+.+.+..+.+.+.+.+..+.+.+.+.+..+.+.+.+.+                 |   
    5e+07 |-+                                                               |   
          |                                                                 |   
        0 +-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                 fio.read_clat_99__us                           
                                                                                
    4e+08 +-----------------------------------------------------------------+   
          |      O O     O        O    O       O  O                  O    O |   
  3.5e+08 |-+        O                   O O O          O        O          |   
    3e+08 |-+                                         O        O        O   |   
          |   O        O    O                       O        O              |   
  2.5e+08 |-O                                             O        O        |   
          |                                                                 |   
    2e+08 |-+                                                               |   
          |                                                                 |   
  1.5e+08 |-+                                                               |   
    1e+08 |.+.+..          .+. .+. .+.. .+.+.          .+.                  |   
          |      +.+.+.+.+.   +   +    +     +.+..+.+.+   +                 |   
    5e+07 |-+                                                               |   
          |                                                                 |   
        0 +-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                 fio.read_slat_mean_us                          
                                                                                
  4.5e+06 +-----------------------------------------------------------------+   
          |                   O                                             |   
    4e+06 |-+                                                             O |   
  3.5e+06 |-+                   O O    O       O  O                         |   
          |      O O     O               O O          O O      O O   O      |   
    3e+06 |-O O      O O    O                O                          O   |   
  2.5e+06 |-+                                       O     O  O     O        |   
          |.+.+..+.+.+.+.+..+.+.+.+.+..+.+.+.+.+..+.+.+.+.+                 |   
    2e+06 |-+                                                               |   
  1.5e+06 |-+                                                               |   
          |                                                                 |   
    1e+06 |-+                                                               |   
   500000 |-+                                                               |   
          |                                                                 |   
        0 +-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                 fio.read_slat_stddev                           
                                                                                
  3.5e+06 +-----------------------------------------------------------------+   
          |                                                                 |   
    3e+06 |-+                 O O O            O                            |   
          |      O       O             O          O                  O      |   
  2.5e+06 |-+      O O                                                    O |   
          |                              O O O          O        O      O   |   
    2e+06 |-+          O                              O        O            |   
          | O O             O                       O     O  O     O        |   
  1.5e+06 |-+                                                               |   
          |                                                                 |   
    1e+06 |-+                                                               |   
          |                                                                 |   
   500000 |-+                                                               |   
          |.+.+..+.+.+.+.+..+.+.+.+.+..+.+.+.+.+..+.+.+.+.+                 |   
        0 +-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                fio.latency_100ms_                              
                                                                                
  100 +---------------------------------------------------------------------+   
   90 |-+  +                 +.   +      +                                  |   
      |                                                                     |   
   80 |-+                                                                   |   
   70 |-+                                                                   |   
      |                                                                     |   
   60 |-+                                                                   |   
   50 |-+                                                                   |   
   40 |-O  O        O    O                        O  O      O O             |   
      |        O      O             O  O O O  O O      O O      O  O   O  O |   
   30 |-+    O    O        O O  O                                    O      |   
   20 |-+                                                                   |   
      |                                                                     |   
   10 |-+                                                                   |   
    0 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                               fio.latency_250ms_                               
                                                                                
  45 +----------------------------------------------------------------------+   
     |                                                                    O |   
  40 |-+                  O                                                 |   
  35 |-+                     O                  O                           |   
     | O  O   O                O    O O             O    O    O    O        |   
  30 |-+    O      O  O O                O O O    O    O   O    O      O    |   
  25 |-+         O                                                   O      |   
     |                                                                      |   
  20 |-+                                                                    |   
  15 |-+                                                                    |   
     |                                                                      |   
  10 |-+                                                                    |   
   5 |-+                                                                    |   
     |   .+.                                                                |   
   0 +----------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                     fio.workload                               
                                                                                
  4.5e+06 +-----------------------------------------------------------------+   
          |                                                                 |   
    4e+06 |-+                                                               |   
  3.5e+06 |-+                                       O     O  O     O        |   
          | O O  O   O O    O            O O O        O O      O O      O   |   
    3e+06 |-+      O     O             O                             O      |   
  2.5e+06 |-+                   O O            O  O                       O |   
          |                   O                                             |   
    2e+06 |-+                                                               |   
  1.5e+06 |-+                                                               |   
          |                                                                 |   
    1e+06 |-+                                                               |   
   500000 |-+                                                               |   
          |                                                                 |   
        0 +-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                             fio.time.minor_page_faults                         
                                                                                
  500000 +------------------------------------------------------------------+   
  450000 |-+                                     O                        O |   
         |                            O      O          O        O          |   
  400000 |-+    O O      O   O O  O                   O        O            |   
  350000 |-O O      O O    O            O  O   O   O        O        O  O   |   
         |                                                O        O        |   
  300000 |-+                                                                |   
  250000 |-+                                                                |   
  200000 |-+                                                                |   
         |.+.+..+.+.+.+..+.+.+.+..+.+.+.+..+.+.+.+.+..+.+.+                 |   
  150000 |-+                                                                |   
  100000 |-+                                                                |   
         |                                                                  |   
   50000 |-+                                                                |   
       0 +------------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.5.0-rc3-00082-g6942795656e5c" of type "text/plain" (202322 bytes)

View attachment "job-script" of type "text/plain" (8376 bytes)

View attachment "job.yaml" of type "text/plain" (5888 bytes)

View attachment "reproduce" of type "text/plain" (904 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ