lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Date:   Tue, 28 Jul 2020 17:17:19 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     Nikolay Borisov <nborisov@...e.com>
Cc:     David Sterba <dsterba@...e.com>,
        LKML <linux-kernel@...r.kernel.org>, lkp@...ts.01.org,
        linux-btrfs@...r.kernel.org
Subject: [btrfs] b5fa368b72: fio.write_iops 71.4% improvement

Greeting,

FYI, we noticed a 71.4% improvement of fio.write_iops due to commit:


commit: b5fa368b728c9a825977b1a0c624448241d69345 ("btrfs: don't balance btree inode pages from buffered write path")
https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git master


in testcase: fio-basic
on test machine: 192 threads Intel(R) Xeon(R) CPU @ 2.20GHz with 192G memory
with following parameters:

	runtime: 300s
	disk: 1SSD
	fs: btrfs
	nr_task: 100%
	test_size: 128G
	rw: write
	bs: 4k
	ioengine: sync
	cpufreq_governor: performance
	ucode: 0x4002f01
	fs2: nfsv4

test-description: Fio is a tool that will spawn a number of threads or processes doing a particular type of I/O action as specified by the user.
test-url: https://github.com/axboe/fio





Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
bs/compiler/cpufreq_governor/disk/fs2/fs/ioengine/kconfig/nr_task/rootfs/runtime/rw/tbox_group/test_size/testcase/ucode:
  4k/gcc-9/performance/1SSD/nfsv4/btrfs/sync/x86_64-rhel-8.3/100%/debian-10.4-x86_64-20200603.cgz/300s/write/lkp-csl-2ap1/128G/fio-basic/0x4002f01

commit: 
  v5.8-rc6
  b5fa368b72 ("btrfs: don't balance btree inode pages from buffered write path")

        v5.8-rc6 b5fa368b728c9a825977b1a0c62 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          1:5          -20%            :4     kmsg.ACPI_Error
         %stddev     %change         %stddev
             \          |                \  
     42.79 ±  4%     -42.7        0.07 ± 26%  fio.latency_1000us%
      0.10 ± 27%      +0.1        0.20 ± 29%  fio.latency_100us%
      0.02 ± 70%      +0.0        0.06 ± 67%  fio.latency_10ms%
      0.03 ± 60%      +0.0        0.06 ± 29%  fio.latency_20us%
      0.33 ±  9%      +0.5        0.79 ±  8%  fio.latency_4ms%
      0.02 ± 52%      +0.0        0.05 ± 26%  fio.latency_4us%
     53.39 ±  3%     +44.0       97.44        fio.latency_500us%
      0.08 ± 67%      +0.1        0.17 ± 31%  fio.latency_50us%
    108.43 ±  4%     -41.3%      63.59 ±  3%  fio.time.elapsed_time
    108.43 ±  4%     -41.3%      63.59 ±  3%  fio.time.elapsed_time.max
     18762            -1.6%      18470        fio.time.percent_of_cpu_this_job_got
     20278 ±  4%     -42.3%      11691 ±  3%  fio.time.system_time
     65.64 ±  6%     -14.3%      56.24 ±  3%  fio.time.user_time
     10506 ±  3%     -33.5%       6990        fio.time.voluntary_context_switches
      1221 ±  4%     +71.4%       2092 ±  3%  fio.write_bw_MBps
    959283 ±  2%     -63.2%     353280 ±  2%  fio.write_clat_90%_us
    972390 ±  3%     -63.1%     358400 ±  3%  fio.write_clat_95%_us
   1003520 ±  2%     +70.6%    1712128 ± 14%  fio.write_clat_99%_us
    611974 ±  4%     -41.8%     356220 ±  3%  fio.write_clat_mean_us
    377411 ±  3%     -14.7%     322068 ±  8%  fio.write_clat_stddev
    312599 ±  4%     +71.4%     535760 ±  3%  fio.write_iops
      1.00          -100.0%       0.00        nfsstat.Client.nfs.v4.renew.percent
      0.81 ±  8%      +0.5        1.32 ±  6%  mpstat.cpu.all.irq%
      0.32 ±  8%      +0.2        0.49 ±  4%  mpstat.cpu.all.usr%
   1110824 ±  5%     +49.5%    1660937        vmstat.io.bo
      7459 ± 13%     +73.9%      12968 ± 10%  vmstat.system.cs
    268752           +45.6%     391191        vmstat.system.in
      3748 ±  6%     -18.2%       3068 ±  3%  slabinfo.blkdev_ioc.active_objs
      3877 ±  6%     -20.0%       3101 ±  3%  slabinfo.blkdev_ioc.num_objs
      9818 ±  3%     -14.5%       8397 ±  6%  slabinfo.fsnotify_mark_connector.active_objs
      9852 ±  3%     -14.8%       8397 ±  6%  slabinfo.fsnotify_mark_connector.num_objs
      5558 ±  5%     -23.3%       4263 ±  3%  slabinfo.numa_policy.active_objs
      5558 ±  5%     -23.3%       4263 ±  3%  slabinfo.numa_policy.num_objs
      0.09 ± 34%    +107.7%       0.18 ± 35%  sched_debug.cfs_rq:/.nr_spread_over.avg
    756.43 ± 24%     +23.5%     934.00 ± 12%  sched_debug.cfs_rq:/.util_est_enqueued.max
     31679 ± 21%     +20.4%      38137 ±  6%  sched_debug.cpu.nr_switches.max
      3331 ± 14%     +39.3%       4642 ± 12%  sched_debug.cpu.nr_switches.stddev
      2723 ± 18%     +53.4%       4177 ± 13%  sched_debug.cpu.sched_count.stddev
     15678 ± 34%     +51.4%      23732 ± 10%  sched_debug.cpu.ttwu_count.max
      1600 ± 20%     +64.7%       2635 ± 14%  sched_debug.cpu.ttwu_count.stddev
      1189 ± 18%     +44.5%       1718 ± 14%  sched_debug.cpu.ttwu_local.stddev
     64475 ±  3%     -28.4%      46163 ±  3%  meminfo.Active(file)
    198056 ± 15%     -27.8%     142902 ±  2%  meminfo.AnonHugePages
  75171408 ±  2%      -9.9%   67764399        meminfo.Cached
  11912259 ±  4%     +24.8%   14866005 ±  2%  meminfo.Dirty
  74116280 ±  2%     -10.0%   66727295        meminfo.Inactive
  73710102 ±  2%     -10.0%   66319107        meminfo.Inactive(file)
  77696534 ±  2%      -9.5%   70314186        meminfo.Memused
     16715 ±  5%      +6.1%      17728 ±  2%  meminfo.PageTables
   2260444 ±  8%    +111.9%    4789065 ± 20%  meminfo.Writeback
   1234924 ±  5%     +71.6%    2119275 ±  2%  meminfo.max_used_kB
   3104848 ±  2%     +25.6%    3899177 ± 16%  numa-meminfo.node0.Dirty
  18715353 ±  3%     -12.1%   16453156 ±  3%  numa-meminfo.node0.Inactive
  18560017 ±  2%     -11.4%   16447470 ±  3%  numa-meminfo.node0.Inactive(file)
    531113 ± 12%    +120.7%    1172025 ± 16%  numa-meminfo.node0.Writeback
     70463 ± 19%     -64.6%      24967 ± 45%  numa-meminfo.node1.AnonHugePages
    576225 ± 13%    +132.5%    1339552 ± 20%  numa-meminfo.node1.Writeback
   2894752 ±  6%     +32.7%    3841292 ± 14%  numa-meminfo.node2.Dirty
    579127 ±  4%    +102.1%    1170647 ± 17%  numa-meminfo.node2.Writeback
   2884453 ±  4%     +18.0%    3404964 ±  8%  numa-meminfo.node3.Dirty
    588523 ± 15%     +95.8%    1152468 ± 29%  numa-meminfo.node3.Writeback
     94371 ±  2%      +5.4%      99494        proc-vmstat.nr_active_anon
     16125 ±  2%     -28.4%      11548 ±  3%  proc-vmstat.nr_active_file
     93686 ±  2%      +5.4%      98717        proc-vmstat.nr_anon_pages
     96.20 ± 15%     -28.0%      69.25 ±  2%  proc-vmstat.nr_anon_transparent_hugepages
   2974469 ±  4%     +24.9%    3715787 ±  2%  proc-vmstat.nr_dirty
  18799078 ±  2%      -9.8%   16956228        proc-vmstat.nr_file_pages
  30001206            +6.1%   31837922        proc-vmstat.nr_free_pages
  18433369 ±  2%     -10.0%   16595505        proc-vmstat.nr_inactive_file
      4180 ±  4%      +5.9%       4427 ±  2%  proc-vmstat.nr_page_table_pages
     75045            -5.7%      70733        proc-vmstat.nr_slab_reclaimable
    568487 ±  8%    +110.9%    1198815 ± 19%  proc-vmstat.nr_writeback
  31289564           -13.1%   27191693 ±  3%  proc-vmstat.nr_written
     94371 ±  2%      +5.4%      99494        proc-vmstat.nr_zone_active_anon
     16125 ±  2%     -28.4%      11548 ±  3%  proc-vmstat.nr_zone_active_file
  18433369 ±  2%     -10.0%   16595504        proc-vmstat.nr_zone_inactive_file
   3548202 ±  3%     +38.7%    4920518 ±  5%  proc-vmstat.nr_zone_write_pending
     13925 ± 59%     +69.5%      23598 ± 52%  proc-vmstat.numa_pte_updates
     11022 ±  3%     -12.6%       9633 ±  9%  proc-vmstat.pgactivate
    689936 ±  3%     -26.4%     507464        proc-vmstat.pgfault
 1.252e+08           -13.0%  1.089e+08 ±  3%  proc-vmstat.pgpgout
   4644388 ±  2%     -12.2%    4079375 ±  3%  numa-vmstat.node0.nr_dirtied
    776668 ±  2%     +25.9%     977726 ± 16%  numa-vmstat.node0.nr_dirty
   4743861 ±  3%     -12.6%    4145352 ±  3%  numa-vmstat.node0.nr_file_pages
   4638950 ±  2%     -12.1%    4077166 ±  3%  numa-vmstat.node0.nr_inactive_file
    132134 ± 11%    +117.6%     287552 ± 17%  numa-vmstat.node0.nr_writeback
   3734251 ±  3%     -24.7%    2812833 ±  9%  numa-vmstat.node0.nr_written
   4638933 ±  2%     -12.1%    4077129 ±  3%  numa-vmstat.node0.nr_zone_inactive_file
    910136           +39.2%    1266526 ± 12%  numa-vmstat.node0.nr_zone_write_pending
    145469 ± 10%    +130.5%     335241 ± 20%  numa-vmstat.node1.nr_writeback
   3710861 ±  3%     -20.6%    2944822 ±  5%  numa-vmstat.node1.nr_written
    902940 ± 10%     +41.1%    1274145 ±  9%  numa-vmstat.node1.nr_zone_write_pending
   4617623 ±  2%     -10.2%    4147732        numa-vmstat.node2.nr_dirtied
    722637 ±  6%     +32.4%     956854 ± 15%  numa-vmstat.node2.nr_dirty
   4610974 ±  2%     -10.2%    4142908        numa-vmstat.node2.nr_inactive_file
    145787 ±  5%    +100.6%     292486 ± 18%  numa-vmstat.node2.nr_writeback
   3747968 ±  2%     -22.7%    2897220 ±  4%  numa-vmstat.node2.nr_written
   4610965 ±  2%     -10.2%    4142892        numa-vmstat.node2.nr_zone_inactive_file
    869651 ±  5%     +43.8%    1250518 ± 11%  numa-vmstat.node2.nr_zone_write_pending
   5259825 ±  3%      -9.9%    4737169 ±  3%  numa-vmstat.node2.numa_hit
   5166146 ±  3%     -10.4%    4631030 ±  3%  numa-vmstat.node2.numa_local
    723467 ±  4%     +17.5%     850121 ±  9%  numa-vmstat.node3.nr_dirty
    143928 ± 15%     +99.1%     286631 ± 27%  numa-vmstat.node3.nr_writeback
   3748541 ±  2%     -19.1%    3034197 ±  4%  numa-vmstat.node3.nr_written
    868605           +31.0%    1138022 ± 13%  numa-vmstat.node3.nr_zone_write_pending
      9.43 ± 52%      -7.5        1.92 ±173%  perf-profile.calltrace.cycles-pp.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
      9.43 ± 52%      -7.5        1.92 ±173%  perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
      9.43 ± 52%      -7.5        1.92 ±173%  perf-profile.calltrace.cycles-pp.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
      6.43 ± 91%      -6.4        0.00        perf-profile.calltrace.cycles-pp.mmput.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      6.43 ± 91%      -6.4        0.00        perf-profile.calltrace.cycles-pp.exit_mmap.mmput.do_exit.do_group_exit.__x64_sys_exit_group
      7.15 ± 60%      -6.2        1.00 ±173%  perf-profile.calltrace.cycles-pp.do_filp_open.do_sys_openat2.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
      7.15 ± 60%      -6.2        1.00 ±173%  perf-profile.calltrace.cycles-pp.path_openat.do_filp_open.do_sys_openat2.do_sys_open.do_syscall_64
      6.43 ± 91%      -5.4        1.00 ±173%  perf-profile.calltrace.cycles-pp.tlb_finish_mmu.exit_mmap.mmput.do_exit.do_group_exit
      6.43 ± 91%      -5.4        1.00 ±173%  perf-profile.calltrace.cycles-pp.tlb_flush_mmu.tlb_finish_mmu.exit_mmap.mmput.do_exit
      6.43 ± 91%      -5.4        1.00 ±173%  perf-profile.calltrace.cycles-pp.release_pages.tlb_flush_mmu.tlb_finish_mmu.exit_mmap.mmput
      5.01 ± 99%      +9.1       14.06 ± 64%  perf-profile.calltrace.cycles-pp.proc_reg_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.01 ± 99%      +9.1       14.06 ± 64%  perf-profile.calltrace.cycles-pp.seq_read.proc_reg_read.vfs_read.ksys_read.do_syscall_64
     33.87 ± 28%     +14.9       48.79 ± 17%  perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     33.87 ± 28%     +14.9       48.79 ± 17%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
      9.43 ± 52%      -7.5        1.92 ±173%  perf-profile.children.cycles-pp.__x64_sys_exit_group
      7.15 ± 60%      -6.1        1.00 ±173%  perf-profile.children.cycles-pp.do_sys_open
      7.15 ± 60%      -6.1        1.00 ±173%  perf-profile.children.cycles-pp.do_sys_openat2
      7.15 ± 60%      -6.1        1.00 ±173%  perf-profile.children.cycles-pp.do_filp_open
      7.15 ± 60%      -6.1        1.00 ±173%  perf-profile.children.cycles-pp.path_openat
      6.43 ± 91%      -5.4        1.00 ±173%  perf-profile.children.cycles-pp.tlb_finish_mmu
      6.43 ± 91%      -5.4        1.00 ±173%  perf-profile.children.cycles-pp.tlb_flush_mmu
      6.43 ± 91%      -5.4        1.00 ±173%  perf-profile.children.cycles-pp.release_pages
      5.01 ± 99%      +9.1       14.06 ± 64%  perf-profile.children.cycles-pp.proc_reg_read
     33.87 ± 28%     +16.7       50.58 ± 18%  perf-profile.children.cycles-pp.cpuidle_enter
     33.87 ± 28%     +16.7       50.58 ± 18%  perf-profile.children.cycles-pp.cpuidle_enter_state
  1.19e+10 ±  2%      +3.8%  1.234e+10        perf-stat.i.branch-instructions
  13293401 ±  3%     +72.9%   22981735 ±  2%  perf-stat.i.branch-misses
     45.97           +14.3       60.29        perf-stat.i.cache-miss-rate%
  54604331 ±  3%     +89.0%  1.032e+08 ±  5%  perf-stat.i.cache-misses
 1.053e+08 ±  2%     +60.1%  1.686e+08 ±  5%  perf-stat.i.cache-references
      7337 ± 13%     +64.0%      12031 ±  7%  perf-stat.i.context-switches
     11.09            -3.7%      10.68        perf-stat.i.cpi
    257.90           +14.3%     294.72 ±  2%  perf-stat.i.cpu-migrations
     18136 ±  2%     -68.9%       5640 ±  5%  perf-stat.i.cycles-between-cache-misses
  1.25e+10 ±  2%      +6.5%  1.332e+10        perf-stat.i.dTLB-loads
      0.01 ± 47%      -0.0        0.01 ± 14%  perf-stat.i.dTLB-store-miss-rate%
 1.015e+09 ±  4%     +47.6%  1.498e+09 ±  2%  perf-stat.i.dTLB-stores
   5180972 ±  5%     +48.9%    7715573 ±  7%  perf-stat.i.iTLB-load-misses
 4.909e+10 ±  2%      +4.9%   5.15e+10        perf-stat.i.instructions
     10020 ±  4%     -33.4%       6671 ±  6%  perf-stat.i.instructions-per-iTLB-miss
    133.00 ±  2%      +7.2%     142.54        perf-stat.i.metric.M/sec
      5169           +16.6%       6029        perf-stat.i.minor-faults
     91.08            -1.6       89.46        perf-stat.i.node-load-miss-rate%
  11734717 ±  3%     +43.0%   16776934        perf-stat.i.node-load-misses
   1258928 ±  6%     +63.5%    2058186 ± 15%  perf-stat.i.node-loads
   3479408 ±  3%    +109.0%    7272943 ±  9%  perf-stat.i.node-store-misses
      5169           +16.6%       6029        perf-stat.i.page-faults
      2.12 ±  2%     +54.2%       3.27 ±  4%  perf-stat.overall.MPKI
      0.11 ±  4%      +0.1        0.19 ±  2%  perf-stat.overall.branch-miss-rate%
     51.75 ±  2%      +9.5       61.25        perf-stat.overall.cache-miss-rate%
     11.27            -4.1%      10.81        perf-stat.overall.cpi
     10278 ±  2%     -47.4%       5409 ±  4%  perf-stat.overall.cycles-between-cache-misses
      0.01 ± 22%      -0.0        0.00 ± 17%  perf-stat.overall.dTLB-store-miss-rate%
      9564 ±  4%     -29.8%       6710 ±  6%  perf-stat.overall.instructions-per-iTLB-miss
      0.09            +4.2%       0.09        perf-stat.overall.ipc
    161762 ±  3%     -39.7%      97476 ±  2%  perf-stat.overall.path-length
  1.18e+10 ±  2%      +3.0%  1.215e+10        perf-stat.ps.branch-instructions
  12984033 ±  3%     +74.8%   22695778 ±  2%  perf-stat.ps.branch-misses
  53397122 ±  3%     +90.1%  1.015e+08 ±  5%  perf-stat.ps.cache-misses
 1.032e+08           +60.7%  1.658e+08 ±  5%  perf-stat.ps.cache-references
      7202 ± 13%     +63.9%      11805 ±  7%  perf-stat.ps.context-switches
    254.62           +13.7%     289.43 ±  2%  perf-stat.ps.cpu-migrations
 1.239e+10 ±  2%      +5.8%  1.311e+10        perf-stat.ps.dTLB-loads
 9.982e+08 ±  4%     +47.6%  1.473e+09 ±  2%  perf-stat.ps.dTLB-stores
   5100160 ±  5%     +48.8%    7590915 ±  7%  perf-stat.ps.iTLB-load-misses
 4.867e+10 ±  2%      +4.1%  5.069e+10        perf-stat.ps.instructions
      5046           +16.8%       5894        perf-stat.ps.minor-faults
  11558890 ±  3%     +42.8%   16501902        perf-stat.ps.node-load-misses
   1239797 ±  6%     +63.3%    2024797 ± 15%  perf-stat.ps.node-loads
   3402509 ±  3%    +110.3%    7156413 ±  9%  perf-stat.ps.node-store-misses
      5046           +16.8%       5894        perf-stat.ps.page-faults
 5.428e+12 ±  3%     -39.7%  3.271e+12 ±  2%  perf-stat.total.instructions
     66486 ± 10%     -42.5%      38199 ±  8%  softirqs.CPU0.TIMER
     62563 ±  8%     -50.1%      31214 ±  2%  softirqs.CPU1.TIMER
     56530 ±  5%     -46.8%      30065        softirqs.CPU10.TIMER
     55670 ±  6%     -41.7%      32480 ± 18%  softirqs.CPU100.TIMER
     55908 ±  6%     -47.1%      29600 ±  3%  softirqs.CPU101.TIMER
     55661 ±  5%     -47.4%      29280        softirqs.CPU102.TIMER
     56198 ±  8%     -42.6%      32248 ± 18%  softirqs.CPU103.TIMER
     55671 ±  5%     -47.7%      29128        softirqs.CPU104.TIMER
     57617 ± 12%     -49.8%      28926        softirqs.CPU105.TIMER
     55532 ±  5%     -48.1%      28849        softirqs.CPU106.TIMER
     56143 ±  7%     -48.5%      28895        softirqs.CPU107.TIMER
     55432 ±  5%     -48.1%      28758        softirqs.CPU108.TIMER
     56085 ±  5%     -48.7%      28791        softirqs.CPU109.TIMER
     56769 ±  5%     -47.7%      29715        softirqs.CPU11.TIMER
     55520 ±  6%     -48.0%      28859        softirqs.CPU110.TIMER
     55700 ±  5%     -48.0%      28978        softirqs.CPU111.TIMER
     55551 ±  5%     -46.7%      29600 ±  2%  softirqs.CPU112.TIMER
     55924 ±  5%     -48.3%      28939        softirqs.CPU113.TIMER
     55646 ±  5%     -48.1%      28890        softirqs.CPU114.TIMER
     55387 ±  5%     -47.5%      29071        softirqs.CPU115.TIMER
     55795 ±  5%     -48.5%      28749        softirqs.CPU116.TIMER
     55595 ±  5%     -48.2%      28825 ±  2%  softirqs.CPU117.TIMER
     55522 ±  5%     -47.9%      28944        softirqs.CPU118.TIMER
     56616 ±  6%     -49.5%      28568        softirqs.CPU119.TIMER
     56723 ±  5%     -47.4%      29850        softirqs.CPU12.TIMER
     55351 ±  6%     -47.7%      28975 ±  2%  softirqs.CPU120.TIMER
     55707 ±  6%     -48.4%      28744        softirqs.CPU121.TIMER
     55355 ±  5%     -48.4%      28562 ±  3%  softirqs.CPU122.TIMER
     56655 ±  6%     -49.2%      28766        softirqs.CPU123.TIMER
     55440 ±  6%     -47.6%      29031 ±  3%  softirqs.CPU124.TIMER
     55482 ±  6%     -48.3%      28702        softirqs.CPU125.TIMER
     56248 ±  5%     -49.0%      28708        softirqs.CPU126.TIMER
     55504 ±  6%     -48.1%      28803 ±  2%  softirqs.CPU127.TIMER
     56303 ±  6%     -49.1%      28663        softirqs.CPU128.TIMER
     55396 ±  6%     -48.6%      28453        softirqs.CPU129.TIMER
     56388 ±  5%     -46.0%      30438        softirqs.CPU13.TIMER
     56272 ±  6%     -49.4%      28498        softirqs.CPU130.TIMER
     55659 ±  6%     -48.1%      28883 ±  3%  softirqs.CPU131.TIMER
     55505 ±  6%     -48.5%      28613 ±  2%  softirqs.CPU132.TIMER
     55580 ±  6%     -47.6%      29138        softirqs.CPU133.TIMER
     55831 ±  7%     -42.9%      31876 ± 15%  softirqs.CPU134.TIMER
     55329 ±  6%     -48.2%      28645 ±  2%  softirqs.CPU135.TIMER
     55397 ±  6%     -48.7%      28430 ±  2%  softirqs.CPU136.TIMER
     55337 ±  6%     -48.2%      28668 ±  2%  softirqs.CPU137.TIMER
     55805 ±  6%     -48.9%      28523 ±  2%  softirqs.CPU138.TIMER
     55325 ±  6%     -48.2%      28650 ±  2%  softirqs.CPU139.TIMER
     57100 ±  5%     -47.0%      30235        softirqs.CPU14.TIMER
     55304 ±  6%     -48.1%      28716 ±  2%  softirqs.CPU140.TIMER
     55225 ±  6%     -48.3%      28559 ±  2%  softirqs.CPU141.TIMER
     55379 ±  6%     -48.3%      28605        softirqs.CPU142.TIMER
     55211 ±  6%     -47.5%      29006 ±  3%  softirqs.CPU143.TIMER
     56842 ±  5%     -49.2%      28881        softirqs.CPU144.TIMER
     55891 ±  5%     -48.5%      28766 ±  2%  softirqs.CPU145.TIMER
     55597 ±  5%     -48.3%      28761        softirqs.CPU146.TIMER
     55825 ±  5%     -46.3%      29966 ±  7%  softirqs.CPU147.TIMER
     55567 ±  5%     -48.3%      28724        softirqs.CPU148.TIMER
     55493 ±  6%     -47.9%      28938 ±  2%  softirqs.CPU149.TIMER
     57019 ±  5%     -47.3%      30071        softirqs.CPU15.TIMER
     55844 ±  5%     -48.9%      28557 ±  2%  softirqs.CPU150.TIMER
     55650 ±  5%     -48.6%      28584        softirqs.CPU151.TIMER
     56278 ±  5%     -49.5%      28412        softirqs.CPU152.TIMER
     56195 ±  5%     -49.2%      28574 ±  2%  softirqs.CPU153.TIMER
     55596 ±  5%     -48.9%      28391        softirqs.CPU154.TIMER
     55563 ±  5%     -48.8%      28430        softirqs.CPU155.TIMER
     55429 ±  6%     -48.6%      28501        softirqs.CPU156.TIMER
     55485 ±  6%     -48.7%      28438        softirqs.CPU157.TIMER
     55961 ±  5%     -49.3%      28396        softirqs.CPU158.TIMER
     55544 ±  5%     -48.8%      28448        softirqs.CPU159.TIMER
     56330 ±  5%     -47.0%      29855        softirqs.CPU16.TIMER
     55372 ±  5%     -49.1%      28168        softirqs.CPU160.TIMER
     55571 ±  5%     -48.8%      28432        softirqs.CPU161.TIMER
     55599 ±  5%     -48.7%      28503        softirqs.CPU162.TIMER
     56372 ±  5%     -49.7%      28382        softirqs.CPU163.TIMER
     55547 ±  5%     -48.5%      28606        softirqs.CPU164.TIMER
     55422 ±  5%     -48.7%      28443        softirqs.CPU165.TIMER
     55522 ±  5%     -48.9%      28349 ±  3%  softirqs.CPU166.TIMER
     55505 ±  5%     -48.5%      28604        softirqs.CPU167.TIMER
     56203 ±  6%     -47.4%      29557        softirqs.CPU168.TIMER
     55962 ±  5%     -47.0%      29673        softirqs.CPU169.TIMER
     56723 ±  6%     -46.1%      30556 ±  2%  softirqs.CPU17.TIMER
     55736 ±  6%     -47.1%      29459        softirqs.CPU170.TIMER
     55808 ±  6%     -47.0%      29560        softirqs.CPU171.TIMER
     55797 ±  6%     -47.0%      29586        softirqs.CPU172.TIMER
     55770 ±  6%     -47.1%      29521        softirqs.CPU173.TIMER
     55914 ±  5%     -46.3%      30047 ±  4%  softirqs.CPU174.TIMER
     55997 ±  6%     -46.0%      30217 ±  4%  softirqs.CPU175.TIMER
     55972 ±  5%     -47.0%      29647        softirqs.CPU176.TIMER
     55820 ±  5%     -47.5%      29320        softirqs.CPU177.TIMER
     55745 ±  5%     -47.0%      29526        softirqs.CPU178.TIMER
     55674 ±  5%     -47.2%      29406        softirqs.CPU179.TIMER
     56374 ±  5%     -47.2%      29763        softirqs.CPU18.TIMER
     55644 ±  6%     -47.3%      29325        softirqs.CPU180.TIMER
     55711 ±  5%     -47.4%      29296        softirqs.CPU181.TIMER
     55684 ±  6%     -47.1%      29475        softirqs.CPU182.TIMER
     55733 ±  5%     -47.3%      29395        softirqs.CPU183.TIMER
     55699 ±  6%     -46.7%      29715        softirqs.CPU184.TIMER
     55625 ±  6%     -47.0%      29470        softirqs.CPU185.TIMER
     55620 ±  6%     -47.2%      29366        softirqs.CPU186.TIMER
     56019 ±  6%     -47.3%      29496        softirqs.CPU187.TIMER
     56653 ±  6%     -47.4%      29821 ±  2%  softirqs.CPU188.TIMER
     55764 ±  6%     -47.6%      29224        softirqs.CPU189.TIMER
     56271 ±  5%     -45.2%      30823 ±  5%  softirqs.CPU19.TIMER
     55711 ±  6%     -47.1%      29451        softirqs.CPU190.TIMER
     55898 ±  5%     -47.2%      29541 ±  2%  softirqs.CPU191.TIMER
     59222 ±  7%     -49.3%      30007        softirqs.CPU2.TIMER
     56668 ±  5%     -47.3%      29875        softirqs.CPU20.TIMER
     56713 ±  5%     -45.2%      31085 ±  4%  softirqs.CPU21.TIMER
     56176 ±  5%     -47.2%      29656        softirqs.CPU22.TIMER
     56224 ±  5%     -46.6%      30010 ±  2%  softirqs.CPU23.TIMER
     56920 ±  6%     -47.3%      29976 ±  2%  softirqs.CPU24.TIMER
     57046 ±  6%     -47.2%      30120 ±  3%  softirqs.CPU25.TIMER
     51701 ±  7%     -46.7%      27554 ±  9%  softirqs.CPU26.TIMER
     56687 ±  6%     -47.0%      30026        softirqs.CPU27.TIMER
     57806 ±  6%     -48.8%      29608 ±  2%  softirqs.CPU28.TIMER
     58552 ±  8%     -49.6%      29494 ±  2%  softirqs.CPU29.TIMER
     56888 ±  5%     -47.7%      29726        softirqs.CPU3.TIMER
     56346 ±  6%     -49.1%      28702 ±  5%  softirqs.CPU30.TIMER
     56221 ±  6%     -46.0%      30350 ±  5%  softirqs.CPU31.TIMER
     56339 ±  6%     -47.5%      29574 ±  2%  softirqs.CPU32.TIMER
     56321 ±  6%     -47.6%      29532 ±  2%  softirqs.CPU33.TIMER
     56290 ±  6%     -47.8%      29411 ±  2%  softirqs.CPU34.TIMER
     56405 ±  6%     -47.5%      29593        softirqs.CPU35.TIMER
     56307 ±  6%     -47.1%      29769        softirqs.CPU36.TIMER
     56346 ±  6%     -46.0%      30423 ±  3%  softirqs.CPU37.TIMER
     56329 ±  6%     -47.8%      29411 ±  4%  softirqs.CPU38.TIMER
     56220 ±  6%     -47.4%      29565 ±  2%  softirqs.CPU39.TIMER
     56942 ±  5%     -47.6%      29840        softirqs.CPU4.TIMER
     56517 ±  7%     -48.1%      29317 ±  2%  softirqs.CPU40.TIMER
     56765 ±  7%     -48.2%      29411 ±  2%  softirqs.CPU41.TIMER
     56502 ±  6%     -48.0%      29362 ±  2%  softirqs.CPU42.TIMER
     56283 ±  6%     -47.8%      29364 ±  2%  softirqs.CPU43.TIMER
     56294 ±  6%     -48.0%      29257 ±  2%  softirqs.CPU44.TIMER
     56162 ±  6%     -47.8%      29305 ±  2%  softirqs.CPU45.TIMER
     56287 ±  6%     -48.1%      29201 ±  2%  softirqs.CPU46.TIMER
     56185 ±  6%     -45.7%      30488 ±  5%  softirqs.CPU47.TIMER
     56636 ±  5%     -47.3%      29843 ±  2%  softirqs.CPU48.TIMER
     56432 ±  5%     -47.1%      29873 ±  2%  softirqs.CPU49.TIMER
     57031 ±  5%     -47.6%      29912        softirqs.CPU5.TIMER
     56683 ±  5%     -47.8%      29578        softirqs.CPU50.TIMER
     56619 ±  5%     -47.7%      29620        softirqs.CPU51.TIMER
     56326 ±  6%     -47.7%      29460        softirqs.CPU52.TIMER
     56373 ±  5%     -47.8%      29421        softirqs.CPU53.TIMER
     56404 ±  5%     -47.6%      29545 ±  2%  softirqs.CPU54.TIMER
     56405 ±  6%     -48.1%      29249        softirqs.CPU55.TIMER
     56381 ±  5%     -48.0%      29310        softirqs.CPU56.TIMER
     57093 ±  6%     -48.6%      29336        softirqs.CPU57.TIMER
     56648 ±  5%     -47.5%      29717 ±  3%  softirqs.CPU58.TIMER
     56368 ±  5%     -47.9%      29381        softirqs.CPU59.TIMER
     56587 ±  6%     -46.8%      30076 ±  2%  softirqs.CPU6.TIMER
     56347 ±  5%     -48.0%      29312        softirqs.CPU60.TIMER
     56279 ±  5%     -48.0%      29279        softirqs.CPU61.TIMER
     56279 ±  5%     -48.1%      29204        softirqs.CPU62.TIMER
     56367 ±  5%     -47.1%      29844        softirqs.CPU63.TIMER
     56424 ±  5%     -48.1%      29297        softirqs.CPU64.TIMER
     56245 ±  6%     -47.8%      29347        softirqs.CPU65.TIMER
     56367 ±  5%     -48.0%      29329        softirqs.CPU66.TIMER
     56162 ±  5%     -47.8%      29298        softirqs.CPU67.TIMER
     56379 ±  5%     -48.0%      29307        softirqs.CPU68.TIMER
     56380 ±  5%     -47.9%      29364        softirqs.CPU69.TIMER
     56334 ±  5%     -47.9%      29332 ±  2%  softirqs.CPU7.TIMER
     56294 ±  5%     -41.8%      32789 ± 18%  softirqs.CPU70.TIMER
     56511 ±  5%     -48.2%      29248        softirqs.CPU71.TIMER
     57192 ±  5%     -46.1%      30832 ±  2%  softirqs.CPU72.TIMER
     56644 ±  6%     -46.5%      30324        softirqs.CPU73.TIMER
     56675 ±  6%     -46.4%      30393        softirqs.CPU74.TIMER
     56820 ±  5%     -46.6%      30348        softirqs.CPU75.TIMER
     56697 ±  5%     -46.6%      30275        softirqs.CPU76.TIMER
     56613 ±  6%     -46.3%      30385        softirqs.CPU77.TIMER
     56633 ±  6%     -46.5%      30291        softirqs.CPU78.TIMER
     56768 ±  5%     -46.4%      30429        softirqs.CPU79.TIMER
     56312 ±  5%     -47.6%      29530 ± 10%  softirqs.CPU8.TIMER
     56912 ±  5%     -45.2%      31210 ±  5%  softirqs.CPU80.TIMER
     56685 ±  5%     -44.5%      31452 ±  5%  softirqs.CPU81.TIMER
     56815 ±  5%     -46.9%      30171        softirqs.CPU82.TIMER
     56671 ±  5%     -46.7%      30190        softirqs.CPU83.TIMER
     56506 ±  5%     -46.4%      30263        softirqs.CPU84.TIMER
     56690 ±  5%     -46.6%      30244        softirqs.CPU85.TIMER
     56521 ±  6%     -46.7%      30098        softirqs.CPU86.TIMER
     56464 ±  5%     -46.8%      30030        softirqs.CPU87.TIMER
     56474 ±  6%     -46.6%      30175        softirqs.CPU88.TIMER
     56514 ±  6%     -46.5%      30248        softirqs.CPU89.TIMER
     56692 ±  4%     -47.0%      30047        softirqs.CPU9.TIMER
     56606 ±  6%     -44.4%      31483 ±  6%  softirqs.CPU90.TIMER
     56540 ±  5%     -46.7%      30164        softirqs.CPU91.TIMER
     56452 ±  6%     -46.7%      30073        softirqs.CPU92.TIMER
     56388 ±  6%     -46.6%      30101        softirqs.CPU93.TIMER
     56517 ±  6%     -46.8%      30083        softirqs.CPU94.TIMER
     56418 ±  5%     -46.6%      30149        softirqs.CPU95.TIMER
     55677 ±  5%     -46.6%      29747 ±  4%  softirqs.CPU96.TIMER
     54914 ±  6%     -45.5%      29944 ±  4%  softirqs.CPU97.TIMER
     55449 ±  6%     -47.3%      29213        softirqs.CPU98.TIMER
     55571 ±  6%     -47.2%      29339        softirqs.CPU99.TIMER
  10794853 ±  5%     -47.5%    5670358        softirqs.TIMER
    635.60 ± 86%     -88.4%      73.50 ± 42%  interrupts.33:PCI-MSI.524291-edge.eth0-TxRx-2
    109.60 ±  5%     -42.5%      63.00 ±  2%  interrupts.35:PCI-MSI.327680-edge.xhci_hcd
    224.80 ±  6%     -41.5%     131.50 ±  2%  interrupts.9:IO-APIC.9-fasteoi.acpi
    156605 ±  6%     -16.7%     130485 ±  2%  interrupts.CPU0.LOC:Local_timer_interrupts
    224.80 ±  6%     -41.5%     131.50 ±  2%  interrupts.CPU1.9:IO-APIC.9-fasteoi.acpi
    155527 ±  5%     -16.1%     130475 ±  2%  interrupts.CPU1.LOC:Local_timer_interrupts
    155161 ±  5%     -16.0%     130406 ±  2%  interrupts.CPU10.LOC:Local_timer_interrupts
    155151 ±  5%     -16.0%     130397 ±  2%  interrupts.CPU100.LOC:Local_timer_interrupts
    104.60 ±  9%     -34.5%      68.50 ± 10%  interrupts.CPU100.RES:Rescheduling_interrupts
    155233 ±  5%     -16.1%     130287 ±  2%  interrupts.CPU101.LOC:Local_timer_interrupts
     99.40 ±  5%     -38.9%      60.75 ±  7%  interrupts.CPU101.RES:Rescheduling_interrupts
    155036 ±  5%     -15.9%     130320 ±  2%  interrupts.CPU102.LOC:Local_timer_interrupts
     98.80 ±  5%     -31.4%      67.75 ±  9%  interrupts.CPU102.RES:Rescheduling_interrupts
    155147 ±  5%     -16.0%     130247 ±  2%  interrupts.CPU103.LOC:Local_timer_interrupts
    155188 ±  5%     -16.0%     130336 ±  2%  interrupts.CPU104.LOC:Local_timer_interrupts
    155930 ±  4%     -16.5%     130210 ±  2%  interrupts.CPU105.LOC:Local_timer_interrupts
    155207 ±  5%     -16.2%     130109 ±  2%  interrupts.CPU106.LOC:Local_timer_interrupts
    155221 ±  5%     -16.1%     130281 ±  2%  interrupts.CPU107.LOC:Local_timer_interrupts
    124.80 ± 23%     -41.5%      73.00 ± 24%  interrupts.CPU107.RES:Rescheduling_interrupts
    155224 ±  5%     -16.0%     130334 ±  2%  interrupts.CPU108.LOC:Local_timer_interrupts
    155167 ±  5%     -16.0%     130280 ±  2%  interrupts.CPU109.LOC:Local_timer_interrupts
    155456 ±  5%     -16.2%     130347 ±  2%  interrupts.CPU11.LOC:Local_timer_interrupts
    155176 ±  5%     -16.1%     130231 ±  2%  interrupts.CPU110.LOC:Local_timer_interrupts
    154992 ±  5%     -15.9%     130336 ±  2%  interrupts.CPU111.LOC:Local_timer_interrupts
    137.80 ± 51%     -55.9%      60.75 ±  8%  interrupts.CPU111.RES:Rescheduling_interrupts
    155214 ±  5%     -16.0%     130335 ±  2%  interrupts.CPU112.LOC:Local_timer_interrupts
     98.40 ±  5%     -31.4%      67.50 ± 11%  interrupts.CPU112.RES:Rescheduling_interrupts
    155108 ±  5%     -16.0%     130321 ±  2%  interrupts.CPU113.LOC:Local_timer_interrupts
    142.00 ± 48%     -57.9%      59.75 ±  6%  interrupts.CPU113.RES:Rescheduling_interrupts
    155057 ±  5%     -15.9%     130340 ±  2%  interrupts.CPU114.LOC:Local_timer_interrupts
    112.40 ± 26%     -46.2%      60.50 ±  7%  interrupts.CPU114.RES:Rescheduling_interrupts
    155231 ±  5%     -16.0%     130435 ±  2%  interrupts.CPU115.LOC:Local_timer_interrupts
     97.60 ±  3%     -34.4%      64.00 ± 15%  interrupts.CPU115.RES:Rescheduling_interrupts
    155398 ±  5%     -16.1%     130335 ±  2%  interrupts.CPU116.LOC:Local_timer_interrupts
    156102 ±  4%     -16.5%     130318 ±  2%  interrupts.CPU117.LOC:Local_timer_interrupts
    155129 ±  5%     -16.0%     130322 ±  2%  interrupts.CPU118.LOC:Local_timer_interrupts
    155178 ±  5%     -16.0%     130274 ±  2%  interrupts.CPU119.LOC:Local_timer_interrupts
    635.60 ± 86%     -88.4%      73.50 ± 42%  interrupts.CPU12.33:PCI-MSI.524291-edge.eth0-TxRx-2
    155074 ±  5%     -15.9%     130365 ±  2%  interrupts.CPU12.LOC:Local_timer_interrupts
    154793 ±  5%     -15.7%     130489 ±  2%  interrupts.CPU120.LOC:Local_timer_interrupts
    180.00 ± 84%     -63.3%      66.00 ±  9%  interrupts.CPU120.RES:Rescheduling_interrupts
    154980 ±  5%     -15.9%     130320 ±  2%  interrupts.CPU121.LOC:Local_timer_interrupts
    118.40 ± 22%     -45.1%      65.00 ±  5%  interrupts.CPU121.RES:Rescheduling_interrupts
    154988 ±  5%     -15.9%     130284 ±  2%  interrupts.CPU122.LOC:Local_timer_interrupts
    133.20 ± 51%     -52.3%      63.50 ±  8%  interrupts.CPU122.RES:Rescheduling_interrupts
    154569 ±  5%     -15.8%     130213 ±  2%  interrupts.CPU123.LOC:Local_timer_interrupts
    109.20 ± 18%     -41.2%      64.25 ±  6%  interrupts.CPU123.RES:Rescheduling_interrupts
    154519 ±  5%     -15.6%     130384 ±  2%  interrupts.CPU124.LOC:Local_timer_interrupts
    154581 ±  5%     -15.7%     130290 ±  2%  interrupts.CPU125.LOC:Local_timer_interrupts
    227.00 ± 90%     -68.4%      71.75 ±  8%  interrupts.CPU125.RES:Rescheduling_interrupts
    154704 ±  5%     -16.0%     130023 ±  2%  interrupts.CPU126.LOC:Local_timer_interrupts
    155535 ±  5%     -16.3%     130140 ±  2%  interrupts.CPU127.LOC:Local_timer_interrupts
    154679 ±  5%     -15.7%     130321 ±  2%  interrupts.CPU128.LOC:Local_timer_interrupts
     96.20 ±  3%     -35.0%      62.50 ±  3%  interrupts.CPU128.RES:Rescheduling_interrupts
    154630 ±  5%     -15.8%     130236 ±  2%  interrupts.CPU129.LOC:Local_timer_interrupts
    115.60 ± 21%     -36.4%      73.50 ± 24%  interrupts.CPU129.RES:Rescheduling_interrupts
    155224 ±  5%     -16.2%     130146 ±  2%  interrupts.CPU13.LOC:Local_timer_interrupts
    154854 ±  5%     -15.8%     130330 ±  2%  interrupts.CPU130.LOC:Local_timer_interrupts
    139.40 ± 45%     -50.7%      68.75 ± 18%  interrupts.CPU130.RES:Rescheduling_interrupts
    154721 ±  5%     -15.9%     130082 ±  2%  interrupts.CPU131.LOC:Local_timer_interrupts
    154683 ±  5%     -15.7%     130332 ±  2%  interrupts.CPU132.LOC:Local_timer_interrupts
    100.20 ± 10%     -29.9%      70.25 ± 13%  interrupts.CPU132.RES:Rescheduling_interrupts
    154602 ±  5%     -15.9%     130072 ±  2%  interrupts.CPU133.LOC:Local_timer_interrupts
    154833 ±  5%     -15.7%     130483 ±  2%  interrupts.CPU134.LOC:Local_timer_interrupts
    154975 ±  5%     -16.1%     130078 ±  2%  interrupts.CPU135.LOC:Local_timer_interrupts
    108.80 ± 22%     -42.6%      62.50 ±  5%  interrupts.CPU135.RES:Rescheduling_interrupts
    155386 ±  5%     -16.2%     130249 ±  2%  interrupts.CPU136.LOC:Local_timer_interrupts
    155670 ±  5%     -16.3%     130323 ±  2%  interrupts.CPU137.LOC:Local_timer_interrupts
    154836 ±  5%     -15.8%     130341 ±  2%  interrupts.CPU138.LOC:Local_timer_interrupts
    155355 ±  5%     -16.1%     130337 ±  2%  interrupts.CPU139.LOC:Local_timer_interrupts
    141.40 ± 48%     -52.8%      66.75 ± 19%  interrupts.CPU139.RES:Rescheduling_interrupts
    155248 ±  5%     -16.0%     130376 ±  2%  interrupts.CPU14.LOC:Local_timer_interrupts
    154691 ±  5%     -15.7%     130343 ±  2%  interrupts.CPU140.LOC:Local_timer_interrupts
    107.40 ± 12%     -41.1%      63.25 ±  5%  interrupts.CPU140.RES:Rescheduling_interrupts
    154898 ±  5%     -15.9%     130317 ±  2%  interrupts.CPU141.LOC:Local_timer_interrupts
    102.20 ± 14%     -37.9%      63.50 ±  5%  interrupts.CPU141.RES:Rescheduling_interrupts
    154722 ±  5%     -15.8%     130308 ±  2%  interrupts.CPU142.LOC:Local_timer_interrupts
    154649 ±  5%     -15.7%     130344 ±  2%  interrupts.CPU143.LOC:Local_timer_interrupts
    154839 ±  5%     -15.8%     130395 ±  2%  interrupts.CPU144.LOC:Local_timer_interrupts
    154.40 ± 53%     -52.9%      72.75 ± 16%  interrupts.CPU144.RES:Rescheduling_interrupts
    155129 ±  5%     -16.0%     130377 ±  2%  interrupts.CPU145.LOC:Local_timer_interrupts
    123.00 ± 26%     -44.7%      68.00 ±  9%  interrupts.CPU145.RES:Rescheduling_interrupts
    154846 ±  5%     -16.0%     129993 ±  3%  interrupts.CPU146.LOC:Local_timer_interrupts
    154747 ±  5%     -15.8%     130280 ±  2%  interrupts.CPU147.LOC:Local_timer_interrupts
    110.20 ± 19%     -43.1%      62.75 ±  8%  interrupts.CPU147.RES:Rescheduling_interrupts
    155686 ±  5%     -16.3%     130368 ±  2%  interrupts.CPU148.LOC:Local_timer_interrupts
    154708 ±  5%     -15.7%     130352 ±  2%  interrupts.CPU149.LOC:Local_timer_interrupts
    155187 ±  5%     -16.0%     130404 ±  2%  interrupts.CPU15.LOC:Local_timer_interrupts
      1601 ±187%     -96.4%      58.00 ± 10%  interrupts.CPU15.RES:Rescheduling_interrupts
    154836 ±  5%     -15.8%     130350 ±  2%  interrupts.CPU150.LOC:Local_timer_interrupts
    154774 ±  5%     -15.8%     130359 ±  2%  interrupts.CPU151.LOC:Local_timer_interrupts
    154927 ±  5%     -15.9%     130350 ±  2%  interrupts.CPU152.LOC:Local_timer_interrupts
    155107 ±  5%     -16.0%     130360 ±  2%  interrupts.CPU153.LOC:Local_timer_interrupts
    148.80 ± 57%     -59.2%      60.75 ±  8%  interrupts.CPU153.RES:Rescheduling_interrupts
    154925 ±  5%     -16.0%     130138 ±  3%  interrupts.CPU154.LOC:Local_timer_interrupts
    154901 ±  5%     -15.9%     130199 ±  3%  interrupts.CPU155.LOC:Local_timer_interrupts
    154946 ±  5%     -16.1%     130063 ±  3%  interrupts.CPU156.LOC:Local_timer_interrupts
    154919 ±  5%     -15.9%     130254 ±  2%  interrupts.CPU157.LOC:Local_timer_interrupts
    155041 ±  5%     -15.9%     130344 ±  2%  interrupts.CPU158.LOC:Local_timer_interrupts
    138.40 ± 63%     -56.3%      60.50 ±  6%  interrupts.CPU158.RES:Rescheduling_interrupts
    154888 ±  5%     -15.9%     130257 ±  2%  interrupts.CPU159.LOC:Local_timer_interrupts
    155109 ±  5%     -15.9%     130371 ±  2%  interrupts.CPU16.LOC:Local_timer_interrupts
    106.00 ± 17%     -38.2%      65.50 ±  9%  interrupts.CPU16.RES:Rescheduling_interrupts
    155187 ±  5%     -16.0%     130351 ±  2%  interrupts.CPU160.LOC:Local_timer_interrupts
    147.40 ± 72%     -59.0%      60.50 ±  6%  interrupts.CPU160.RES:Rescheduling_interrupts
    154767 ±  5%     -15.8%     130271 ±  2%  interrupts.CPU161.LOC:Local_timer_interrupts
    154871 ±  5%     -15.8%     130334 ±  2%  interrupts.CPU162.LOC:Local_timer_interrupts
     95.60 ±  4%     -38.3%      59.00 ±  3%  interrupts.CPU162.RES:Rescheduling_interrupts
    154939 ±  5%     -15.6%     130832 ±  3%  interrupts.CPU163.LOC:Local_timer_interrupts
    169.40 ± 69%     -61.5%      65.25 ± 10%  interrupts.CPU163.RES:Rescheduling_interrupts
    154816 ±  5%     -15.8%     130348 ±  2%  interrupts.CPU164.LOC:Local_timer_interrupts
    154889 ±  5%     -15.8%     130359 ±  2%  interrupts.CPU165.LOC:Local_timer_interrupts
    154886 ±  5%     -15.8%     130432 ±  2%  interrupts.CPU166.LOC:Local_timer_interrupts
    159.40 ± 64%     -58.8%      65.75 ± 12%  interrupts.CPU166.RES:Rescheduling_interrupts
    154898 ±  5%     -16.0%     130174 ±  3%  interrupts.CPU167.LOC:Local_timer_interrupts
    155616 ±  5%     -16.2%     130435 ±  2%  interrupts.CPU168.LOC:Local_timer_interrupts
    135.20 ± 39%     -50.8%      66.50 ±  2%  interrupts.CPU168.RES:Rescheduling_interrupts
    592.00 ± 31%     +44.3%     854.25 ± 23%  interrupts.CPU169.CAL:Function_call_interrupts
    155524 ±  5%     -16.5%     129816 ±  2%  interrupts.CPU169.LOC:Local_timer_interrupts
    154919 ±  5%     -15.8%     130398 ±  2%  interrupts.CPU17.LOC:Local_timer_interrupts
    140.40 ± 37%     -58.5%      58.25 ±  6%  interrupts.CPU17.RES:Rescheduling_interrupts
    155122 ±  5%     -16.1%     130205 ±  2%  interrupts.CPU170.LOC:Local_timer_interrupts
    828.80 ± 14%     +15.1%     954.00 ±  5%  interrupts.CPU171.CAL:Function_call_interrupts
    155336 ±  5%     -16.4%     129882 ±  2%  interrupts.CPU171.LOC:Local_timer_interrupts
    154807 ±  5%     -15.9%     130231 ±  2%  interrupts.CPU172.LOC:Local_timer_interrupts
    155676 ±  5%     -16.3%     130325 ±  2%  interrupts.CPU173.LOC:Local_timer_interrupts
    103.40 ± 10%     -37.4%      64.75 ± 10%  interrupts.CPU173.RES:Rescheduling_interrupts
    154852 ±  5%     -16.0%     130116 ±  2%  interrupts.CPU174.LOC:Local_timer_interrupts
    155504 ±  5%     -16.2%     130303 ±  2%  interrupts.CPU175.LOC:Local_timer_interrupts
    113.80 ± 34%     -46.2%      61.25        interrupts.CPU175.RES:Rescheduling_interrupts
    154952 ±  5%     -16.0%     130227 ±  2%  interrupts.CPU176.LOC:Local_timer_interrupts
    155.00 ± 66%     -60.6%      61.00 ±  2%  interrupts.CPU176.RES:Rescheduling_interrupts
    154796 ±  5%     -15.9%     130150 ±  2%  interrupts.CPU177.LOC:Local_timer_interrupts
    154801 ±  5%     -15.8%     130322 ±  2%  interrupts.CPU178.LOC:Local_timer_interrupts
    117.60 ± 23%     -47.5%      61.75 ±  5%  interrupts.CPU178.RES:Rescheduling_interrupts
    154796 ±  5%     -16.0%     130100 ±  2%  interrupts.CPU179.LOC:Local_timer_interrupts
    155664 ±  5%     -16.2%     130380 ±  2%  interrupts.CPU18.LOC:Local_timer_interrupts
    111.80 ± 26%     -43.2%      63.50 ± 18%  interrupts.CPU18.RES:Rescheduling_interrupts
    154824 ±  5%     -15.6%     130594 ±  2%  interrupts.CPU180.LOC:Local_timer_interrupts
    154989 ±  5%     -15.9%     130345 ±  2%  interrupts.CPU181.LOC:Local_timer_interrupts
    154876 ±  5%     -15.9%     130277 ±  2%  interrupts.CPU182.LOC:Local_timer_interrupts
    154858 ±  5%     -16.1%     129967 ±  2%  interrupts.CPU183.LOC:Local_timer_interrupts
    154904 ±  5%     -15.9%     130318 ±  2%  interrupts.CPU184.LOC:Local_timer_interrupts
    155973 ±  5%     -16.5%     130309 ±  2%  interrupts.CPU185.LOC:Local_timer_interrupts
    154868 ±  5%     -15.9%     130293 ±  2%  interrupts.CPU186.LOC:Local_timer_interrupts
    154899 ±  5%     -15.9%     130260 ±  2%  interrupts.CPU187.LOC:Local_timer_interrupts
    154890 ±  5%     -15.9%     130206 ±  2%  interrupts.CPU188.LOC:Local_timer_interrupts
    154772 ±  5%     -15.8%     130344 ±  2%  interrupts.CPU189.LOC:Local_timer_interrupts
    155133 ±  5%     -16.0%     130363 ±  2%  interrupts.CPU19.LOC:Local_timer_interrupts
    100.80 ±  6%     -30.6%      70.00 ± 16%  interrupts.CPU19.RES:Rescheduling_interrupts
    154822 ±  5%     -15.9%     130226 ±  2%  interrupts.CPU190.LOC:Local_timer_interrupts
    155088 ±  5%     -16.0%     130320 ±  2%  interrupts.CPU191.LOC:Local_timer_interrupts
      1191 ± 29%     -26.4%     877.00 ±  4%  interrupts.CPU2.CAL:Function_call_interrupts
    155869 ±  5%     -16.3%     130453 ±  2%  interrupts.CPU2.LOC:Local_timer_interrupts
    155082 ±  5%     -15.9%     130362 ±  2%  interrupts.CPU20.LOC:Local_timer_interrupts
    214.40 ±107%     -64.4%      76.25 ± 15%  interrupts.CPU20.RES:Rescheduling_interrupts
    155900 ±  4%     -16.4%     130378 ±  2%  interrupts.CPU21.LOC:Local_timer_interrupts
    163.60 ± 47%     -51.3%      79.75 ± 23%  interrupts.CPU21.RES:Rescheduling_interrupts
    155168 ±  5%     -16.0%     130381 ±  2%  interrupts.CPU22.LOC:Local_timer_interrupts
    100.60 ± 10%     -38.1%      62.25 ±  5%  interrupts.CPU22.RES:Rescheduling_interrupts
    155752 ±  5%     -16.3%     130325 ±  2%  interrupts.CPU23.LOC:Local_timer_interrupts
    155675 ±  5%     -16.2%     130522 ±  2%  interrupts.CPU24.LOC:Local_timer_interrupts
      1091 ± 63%     -77.7%     243.25 ± 21%  interrupts.CPU24.RES:Rescheduling_interrupts
    155252 ±  5%     -16.1%     130285 ±  2%  interrupts.CPU25.LOC:Local_timer_interrupts
    154925 ±  5%     -15.8%     130380 ±  2%  interrupts.CPU26.LOC:Local_timer_interrupts
    154651 ±  5%     -15.7%     130393 ±  2%  interrupts.CPU27.LOC:Local_timer_interrupts
    154714 ±  5%     -15.7%     130438 ±  2%  interrupts.CPU28.LOC:Local_timer_interrupts
    154446 ±  5%     -15.6%     130352 ±  2%  interrupts.CPU29.LOC:Local_timer_interrupts
    155543 ±  5%     -16.2%     130422 ±  2%  interrupts.CPU3.LOC:Local_timer_interrupts
    155031 ±  5%     -15.9%     130375 ±  2%  interrupts.CPU30.LOC:Local_timer_interrupts
    155350 ±  5%     -16.1%     130363 ±  2%  interrupts.CPU31.LOC:Local_timer_interrupts
    154597 ±  5%     -15.7%     130349 ±  2%  interrupts.CPU32.LOC:Local_timer_interrupts
    202.20 ± 95%     -63.4%      74.00 ± 14%  interrupts.CPU32.RES:Rescheduling_interrupts
    154893 ±  5%     -15.8%     130372 ±  2%  interrupts.CPU33.LOC:Local_timer_interrupts
    155319 ±  5%     -16.1%     130349 ±  2%  interrupts.CPU34.LOC:Local_timer_interrupts
    154923 ±  5%     -15.9%     130343 ±  2%  interrupts.CPU35.LOC:Local_timer_interrupts
    154652 ±  5%     -15.7%     130342 ±  2%  interrupts.CPU36.LOC:Local_timer_interrupts
    154681 ±  5%     -15.7%     130340 ±  2%  interrupts.CPU37.LOC:Local_timer_interrupts
    154920 ±  5%     -15.8%     130449 ±  2%  interrupts.CPU38.LOC:Local_timer_interrupts
    154828 ±  5%     -15.8%     130359 ±  2%  interrupts.CPU39.LOC:Local_timer_interrupts
    155188 ±  5%     -16.0%     130398 ±  2%  interrupts.CPU4.LOC:Local_timer_interrupts
    155738 ±  4%     -16.4%     130253 ±  2%  interrupts.CPU40.LOC:Local_timer_interrupts
    155439 ±  5%     -16.2%     130324 ±  2%  interrupts.CPU41.LOC:Local_timer_interrupts
    154889 ±  5%     -15.8%     130340 ±  2%  interrupts.CPU42.LOC:Local_timer_interrupts
    145.80 ± 33%     -41.9%      84.75 ± 34%  interrupts.CPU42.RES:Rescheduling_interrupts
    155327 ±  5%     -16.1%     130357 ±  2%  interrupts.CPU43.LOC:Local_timer_interrupts
    155037 ±  5%     -15.9%     130394 ±  2%  interrupts.CPU44.LOC:Local_timer_interrupts
    117.80 ± 16%     -42.3%      68.00 ± 13%  interrupts.CPU44.RES:Rescheduling_interrupts
    154713 ±  5%     -15.7%     130369 ±  2%  interrupts.CPU45.LOC:Local_timer_interrupts
    131.60 ± 50%     -48.5%      67.75 ± 15%  interrupts.CPU45.RES:Rescheduling_interrupts
    155128 ±  5%     -16.0%     130317 ±  2%  interrupts.CPU46.LOC:Local_timer_interrupts
    154640 ±  5%     -15.7%     130351 ±  2%  interrupts.CPU47.LOC:Local_timer_interrupts
      1165 ± 10%     -21.4%     915.75 ±  5%  interrupts.CPU48.CAL:Function_call_interrupts
    155242 ±  5%     -16.0%     130446 ±  2%  interrupts.CPU48.LOC:Local_timer_interrupts
    155149 ±  5%     -15.9%     130473 ±  2%  interrupts.CPU49.LOC:Local_timer_interrupts
    155195 ±  5%     -16.0%     130409 ±  2%  interrupts.CPU5.LOC:Local_timer_interrupts
    154952 ±  5%     -15.9%     130385 ±  2%  interrupts.CPU50.LOC:Local_timer_interrupts
    154947 ±  5%     -15.9%     130317 ±  2%  interrupts.CPU51.LOC:Local_timer_interrupts
    155314 ±  5%     -16.0%     130442 ±  2%  interrupts.CPU52.LOC:Local_timer_interrupts
    154964 ±  5%     -15.9%     130397 ±  2%  interrupts.CPU53.LOC:Local_timer_interrupts
    154970 ±  5%     -15.9%     130407 ±  2%  interrupts.CPU54.LOC:Local_timer_interrupts
    154822 ±  5%     -15.8%     130394 ±  2%  interrupts.CPU55.LOC:Local_timer_interrupts
    154815 ±  5%     -15.7%     130457 ±  2%  interrupts.CPU56.LOC:Local_timer_interrupts
    155313 ±  5%     -16.0%     130389 ±  2%  interrupts.CPU57.LOC:Local_timer_interrupts
    949.80 ±154%     -93.1%      65.25 ± 10%  interrupts.CPU57.RES:Rescheduling_interrupts
    874.40 ±  7%     +14.9%       1005 ± 15%  interrupts.CPU58.CAL:Function_call_interrupts
    154815 ±  5%     -15.8%     130381 ±  2%  interrupts.CPU58.LOC:Local_timer_interrupts
    154910 ±  5%     -15.8%     130419 ±  2%  interrupts.CPU59.LOC:Local_timer_interrupts
    109.60 ±  5%     -42.5%      63.00 ±  2%  interrupts.CPU6.35:PCI-MSI.327680-edge.xhci_hcd
    155083 ±  5%     -15.9%     130355 ±  2%  interrupts.CPU6.LOC:Local_timer_interrupts
    154815 ±  5%     -15.8%     130402 ±  2%  interrupts.CPU60.LOC:Local_timer_interrupts
    154891 ±  5%     -15.8%     130374 ±  2%  interrupts.CPU61.LOC:Local_timer_interrupts
    154873 ±  5%     -15.8%     130377 ±  2%  interrupts.CPU62.LOC:Local_timer_interrupts
    220.20 ±105%     -69.8%      66.50 ± 12%  interrupts.CPU62.RES:Rescheduling_interrupts
    154838 ±  5%     -15.7%     130550 ±  2%  interrupts.CPU63.LOC:Local_timer_interrupts
    155494 ±  6%     -16.2%     130372 ±  2%  interrupts.CPU64.LOC:Local_timer_interrupts
    158.20 ± 62%     -56.2%      69.25 ± 11%  interrupts.CPU64.RES:Rescheduling_interrupts
    154968 ±  5%     -15.9%     130280 ±  2%  interrupts.CPU65.LOC:Local_timer_interrupts
    154807 ±  5%     -15.8%     130362 ±  2%  interrupts.CPU66.LOC:Local_timer_interrupts
    154713 ±  5%     -15.7%     130453 ±  2%  interrupts.CPU67.LOC:Local_timer_interrupts
    174.40 ± 57%     -57.3%      74.50 ± 18%  interrupts.CPU67.RES:Rescheduling_interrupts
    154793 ±  5%     -15.8%     130326 ±  2%  interrupts.CPU68.LOC:Local_timer_interrupts
    154802 ±  5%     -15.8%     130369 ±  2%  interrupts.CPU69.LOC:Local_timer_interrupts
     98.80 ±  5%     -29.7%      69.50 ± 12%  interrupts.CPU69.RES:Rescheduling_interrupts
    155295 ±  5%     -16.1%     130369 ±  2%  interrupts.CPU7.LOC:Local_timer_interrupts
    154707 ±  5%     -15.7%     130460 ±  2%  interrupts.CPU70.LOC:Local_timer_interrupts
    152.60 ± 34%     -49.7%      76.75 ± 25%  interrupts.CPU70.RES:Rescheduling_interrupts
    154773 ±  5%     -15.9%     130216 ±  2%  interrupts.CPU71.LOC:Local_timer_interrupts
    155930 ±  5%     -16.3%     130553 ±  2%  interrupts.CPU72.LOC:Local_timer_interrupts
    155727 ±  5%     -16.2%     130445 ±  2%  interrupts.CPU73.LOC:Local_timer_interrupts
    155144 ±  5%     -15.9%     130434 ±  2%  interrupts.CPU74.LOC:Local_timer_interrupts
    155140 ±  5%     -15.9%     130427 ±  2%  interrupts.CPU75.LOC:Local_timer_interrupts
    154868 ±  5%     -15.8%     130366 ±  2%  interrupts.CPU76.LOC:Local_timer_interrupts
    155515 ±  5%     -16.1%     130402 ±  2%  interrupts.CPU77.LOC:Local_timer_interrupts
    154868 ±  5%     -15.8%     130397 ±  2%  interrupts.CPU78.LOC:Local_timer_interrupts
    155450 ±  5%     -16.1%     130391 ±  2%  interrupts.CPU79.LOC:Local_timer_interrupts
     78.00 ±112%    +127.2%     177.25 ± 77%  interrupts.CPU79.TLB:TLB_shootdowns
    155495 ±  5%     -16.2%     130374 ±  2%  interrupts.CPU8.LOC:Local_timer_interrupts
    156138 ±  4%     -16.5%     130382 ±  2%  interrupts.CPU80.LOC:Local_timer_interrupts
    842.20 ±  5%     +10.2%     927.75 ±  8%  interrupts.CPU81.CAL:Function_call_interrupts
    154816 ±  5%     -15.9%     130252 ±  2%  interrupts.CPU81.LOC:Local_timer_interrupts
     77.20 ± 93%    +151.6%     194.25 ± 55%  interrupts.CPU81.TLB:TLB_shootdowns
    154938 ±  5%     -15.8%     130398 ±  2%  interrupts.CPU82.LOC:Local_timer_interrupts
    154926 ±  5%     -15.8%     130405 ±  2%  interrupts.CPU83.LOC:Local_timer_interrupts
    154946 ±  5%     -15.7%     130577 ±  2%  interrupts.CPU84.LOC:Local_timer_interrupts
    155023 ±  5%     -15.9%     130402 ±  2%  interrupts.CPU85.LOC:Local_timer_interrupts
    154910 ±  5%     -15.9%     130355 ±  2%  interrupts.CPU86.LOC:Local_timer_interrupts
    855.20 ±  4%     +10.4%     943.75 ±  8%  interrupts.CPU87.CAL:Function_call_interrupts
    154873 ±  5%     -15.8%     130378 ±  2%  interrupts.CPU87.LOC:Local_timer_interrupts
    844.00 ±  5%      +5.7%     892.25 ±  6%  interrupts.CPU88.CAL:Function_call_interrupts
    154978 ±  5%     -15.9%     130387 ±  2%  interrupts.CPU88.LOC:Local_timer_interrupts
    158671 ±  5%     -17.8%     130355 ±  2%  interrupts.CPU89.LOC:Local_timer_interrupts
    155673 ±  5%     -16.4%     130093 ±  2%  interrupts.CPU9.LOC:Local_timer_interrupts
    154865 ±  5%     -15.9%     130210 ±  2%  interrupts.CPU90.LOC:Local_timer_interrupts
    154871 ±  5%     -15.9%     130280 ±  2%  interrupts.CPU91.LOC:Local_timer_interrupts
    154952 ±  5%     -16.0%     130193 ±  2%  interrupts.CPU92.LOC:Local_timer_interrupts
    154839 ±  5%     -15.8%     130386 ±  2%  interrupts.CPU93.LOC:Local_timer_interrupts
    154880 ±  5%     -15.9%     130247 ±  2%  interrupts.CPU94.LOC:Local_timer_interrupts
    155991 ±  5%     -16.4%     130394 ±  2%  interrupts.CPU95.LOC:Local_timer_interrupts
    155893 ±  5%     -16.4%     130305 ±  2%  interrupts.CPU96.LOC:Local_timer_interrupts
    181.60 ± 67%     -65.3%      63.00 ±  6%  interrupts.CPU96.RES:Rescheduling_interrupts
    155043 ±  5%     -15.9%     130349 ±  2%  interrupts.CPU97.LOC:Local_timer_interrupts
    118.60 ± 31%     -46.2%      63.75 ±  8%  interrupts.CPU97.RES:Rescheduling_interrupts
    155135 ±  5%     -16.0%     130301 ±  2%  interrupts.CPU98.LOC:Local_timer_interrupts
    155247 ±  5%     -16.1%     130309 ±  2%  interrupts.CPU99.LOC:Local_timer_interrupts
    115.40 ± 21%     -43.2%      65.50 ± 13%  interrupts.CPU99.RES:Rescheduling_interrupts
  29778614 ±  5%     -16.0%   25023151 ±  2%  interrupts.LOC:Local_timer_interrupts
     61754 ± 13%     +43.5%      88618 ± 11%  interrupts.RES:Rescheduling_interrupts


                                                                                
                                 fio.write_bw_MBps                              
                                                                                
  2400 +--------------------------------------------------------------------+   
       |  O     O                          O   O                            |   
  2200 |-O  O      O O    O      O  O   O    O  O   O    O OO             O |   
       |      O  O     OO   O  O   O  O   O       O    O      O O OO O O    |   
  2000 |-+                   O                        O                  O  |   
       |                                                                    |   
  1800 |-+                                                                  |   
       |                                                                    |   
  1600 |-+                                                                  |   
       |                                                                    |   
  1400 |-+       +.+.                                                       |   
       |   .+   +    +.    .+       +       .+                              |   
  1200 |.++  + +       ++.+  :.+.+.+ + .+.++  +                             |   
       |      +              +        +        +                            |   
  1000 +--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                   fio.write_iops                               
                                                                                
  600000 +------------------------------------------------------------------+   
         |  O    O                          O O OO   O       O            O |   
  550000 |-O  O    O OO O OO O    O OO O O O            O OO   OO           |   
         |      O              OO                  O              O OO O O  |   
  500000 |-+                                          O                     |   
         |                                                                  |   
  450000 |-+                                                                |   
         |                                                                  |   
  400000 |-+                                                                |   
         |         +.                                                       |   
  350000 |-+      +  ++.                                                    |   
         |. +.+. +      +. +.+. +. .++. .+. +.+.                            |   
  300000 |-+    +         +    +  +    +   +    +                           |   
         |                                                                  |   
  250000 +------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                               fio.write_clat_mean_us                           
                                                                                
  700000 +------------------------------------------------------------------+   
         |                                                                  |   
  650000 |-+    +              +       +.       +                           |   
         |.++. +:       +.++. + +.+.+ +  +.++. +                            |   
  600000 |-+  +  :     +     +       +        +                             |   
  550000 |-+     +.   +                                                     |   
         |         +.+                                                      |   
  500000 |-+                                                                |   
         |                                                                  |   
  450000 |-+                                                                |   
  400000 |-+                                                                |   
         |                                            O                     |   
  350000 |-+    O  O    O O  O OO   OO O O O       O    O      OO O OO O O  |   
         | OO O  O   OO    O      O         O O OO   O    OO O            O |   
  300000 +------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                fio.write_clat_90__us                           
                                                                                
   1e+06 +------------------------------------------------------------------+   
         |  +. + +.+    +  +. + +   + +  +  +. +                            |   
  900000 |-+  +              +       +        +                             |   
         |                                                                  |   
  800000 |-+                                                                |   
         |                                                                  |   
  700000 |-+                                                                |   
         |                                                                  |   
  600000 |-+                                                                |   
         |                                                                  |   
  500000 |-+                                                                |   
         |                                                                  |   
  400000 |-+                                                                |   
         | OO O O  O OO O O  O O  O OO O O OO O OO O OO O OO O OO O OO O OO |   
  300000 +------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                 fio.write_clat_95__us                          
                                                                                
  1.1e+06 +-----------------------------------------------------------------+   
          |                                                                 |   
    1e+06 |.++. .++.+.++. .++. +. .++. .+ .+.+ .+                           |   
          |    +         +    +  +    +  +    +                             |   
   900000 |-+                                                               |   
   800000 |-+                                                               |   
          |                                                                 |   
   700000 |-+                                                               |   
          |                                                                 |   
   600000 |-+                                                               |   
   500000 |-+                                                               |   
          |                                                                 |   
   400000 |-+                                                               |   
          | OO O O  O OO O OO OO O OO O OO O OO O OO O OO O OO OO O OO O OO |   
   300000 +-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                fio.latency_500us_                              
                                                                                
  100 +---------------------------------------------------------------------+   
   95 |-O OO   O OO O O OO O O  O O O OO O O OO O O OO O O OO O O OO O O OO |   
      |                                                                     |   
   90 |-+    O                 O                                            |   
   85 |-+                                                                   |   
      |                                                                     |   
   80 |-+                                                                   |   
   75 |-+                                                                   |   
   70 |-+                                                                   |   
      |                                                                     |   
   65 |-+        ++.+                                                       |   
   60 |-+       +    :                                                      |   
      |        +     :                                                      |   
   55 |.+.++. +       +.++.+. .++.+.+. +.+.+.+                              |   
   50 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                               fio.latency_1000us_                              
                                                                                
  50 +----------------------------------------------------------------------+   
  45 |.+             +.       +.       +.                                   |   
     | +.++.         : +.++. +  ++.+. +  ++.+.                              |   
  40 |-+    +.+     :       +        +        +                             |   
  35 |-+       +    :                                                       |   
     |          +  :                                                        |   
  30 |-+         :.+                                                        |   
  25 |-+         +                                                          |   
  20 |-+                                                                    |   
     |                                                                      |   
  15 |-+                                                                    |   
  10 |-+                                                                    |   
     |                                                                      |   
   5 |-+                                                                    |   
   0 +----------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                fio.time.system_time                            
                                                                                
  22000 +-------------------------------------------------------------------+   
        |     : :            : +      : :     :                             |   
  20000 |.++. : :      +.+.+ :  +.++. : +.+.+ :                             |   
        |    +   +     :    +        +       +                              |   
  18000 |-+       :.+.+                                                     |   
        |         +                                                         |   
  16000 |-+                                                                 |   
        |                                                                   |   
  14000 |-+                                                                 |   
        |                                                                   |   
  12000 |-+       O    O      O    O   O  O        O  O           O OO O O  |   
        | OO O   O  O O  O OO     O  O  O   OO O O  O   OO O OO O         O |   
  10000 |-+    O                O                                           |   
        |                                                                   |   
   8000 +-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                        fio.time.percent_of_cpu_this_job_got                    
                                                                                
  19000 +-------------------------------------------------------------------+   
        |.++.+.+.++.+.++.+.++.+.+.++.+.++.+.++.+                            |   
  18500 |-OO O   OO O OO O OO O   OO O OO O OO O O OO O OO O OO O O OO O OO |   
  18000 |-+                                                                 |   
        |                                                                   |   
  17500 |-+                                                                 |   
        |                                                                   |   
  17000 |-+                                                                 |   
        |                                                                   |   
  16500 |-+                                                                 |   
  16000 |-+                                                                 |   
        |                                                                   |   
  15500 |-+    O                O                                           |   
        |                                                                   |   
  15000 +-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                               fio.time.elapsed_time                            
                                                                                
  120 +---------------------------------------------------------------------+   
      |      +               +        +       +                             |   
  110 |.+.+ : :        .++  : + +.+  : :.+.+  :                             |   
      |    :: :      .+   + :  +   + : +    +:                              |   
  100 |-+  +   +.  .+      +        +        +                              |   
      |          ++                                                         |   
   90 |-+                                                                   |   
      |                                                                     |   
   80 |-+                                                                   |   
      |                                                                     |   
   70 |-+                                                                   |   
      |      O   O    O      O O  O   O  O        O  O            OO O O O  |   
   60 |-O OO   O  O O   OO O    O   O  O   O OO O   O  O O OO O O         O |   
      |                                                                     |   
   50 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                             fio.time.elapsed_time.max                          
                                                                                
  120 +---------------------------------------------------------------------+   
      |      +               +        +       +                             |   
  110 |.+.+ : :        .++  : + +.+  : :.+.+  :                             |   
      |    :: :      .+   + :  +   + : +    +:                              |   
  100 |-+  +   +.  .+      +        +        +                              |   
      |          ++                                                         |   
   90 |-+                                                                   |   
      |                                                                     |   
   80 |-+                                                                   |   
      |                                                                     |   
   70 |-+                                                                   |   
      |      O   O    O      O O  O   O  O        O  O            OO O O O  |   
   60 |-O OO   O  O O   OO O    O   O  O   O OO O   O  O O OO O O         O |   
      |                                                                     |   
   50 +---------------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.8.0-rc6-00001-gb5fa368b728c9" of type "text/plain" (158415 bytes)

View attachment "job-script" of type "text/plain" (8120 bytes)

View attachment "job.yaml" of type "text/plain" (5745 bytes)

View attachment "reproduce" of type "text/plain" (1192 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ