lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Date:   Sun, 5 May 2019 13:41:13 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     Holger Hoffstätte 
        <holger.hoffstaette@...glemail.com>
Cc:     Jens Axboe <axboe@...nel.dk>,
        Gwendal Grignou <gwendal@...omium.org>,
        Benjamin Gordon <bmgordon@...omium.org>,
        Guenter Roeck <groeck@...omium.org>,
        LKML <linux-kernel@...r.kernel.org>, Jens Axboe <axboe@...com>,
        lkp@...org
Subject: [loop]  56a85fd837:  fxmark.hdd_btrfs_MWUM_1_bufferedio.works/sec
 224.0% improvement

Greeting,

FYI, we noticed a 224.0% improvement of fxmark.hdd_btrfs_MWUM_1_bufferedio.works/sec due to commit:


commit: 56a85fd8376ef32458efb6ea97a820754e12f6bb ("loop: properly observe rotational flag of underlying device")
https://git.kernel.org/cgit/linux/kernel/git/axboe/linux-block.git for-5.2/block

in testcase: fxmark
on test machine: 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 64G memory
with following parameters:

	disk: 1HDD
	media: hdd
	test: MWUM
	fstype: btrfs
	directio: bufferedio
	cpufreq_governor: performance
	ucode: 0x42d


In addition to that, the commit also has significant impact on the following tests:

+------------------+-----------------------------------------------------------------------+
| testcase: change | fxmark: fxmark.hdd_btrfs_MWUM_1_directio.works/sec 236.5% improvement |
| test machine     | 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 64G memory  |
| test parameters  | cpufreq_governor=performance                                          |
|                  | directio=directio                                                     |
|                  | disk=1HDD                                                             |
|                  | fstype=btrfs                                                          |
|                  | media=hdd                                                             |
|                  | test=MWUM                                                             |
|                  | ucode=0x42d                                                           |
+------------------+-----------------------------------------------------------------------+
| testcase: change | fxmark: fxmark.hdd_btrfs_MWUL_1_directio.works/sec -33.9% regression  |
| test machine     | 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 64G memory  |
| test parameters  | cpufreq_governor=performance                                          |
|                  | directio=directio                                                     |
|                  | disk=1HDD                                                             |
|                  | fstype=btrfs                                                          |
|                  | media=hdd                                                             |
|                  | test=MWUL                                                             |
|                  | ucode=0x42d                                                           |
+------------------+-----------------------------------------------------------------------+


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
compiler/cpufreq_governor/directio/disk/fstype/kconfig/media/rootfs/tbox_group/test/testcase/ucode:
  gcc-7/performance/bufferedio/1HDD/btrfs/x86_64-rhel-7.6/hdd/debian-x86_64-2018-04-03.cgz/ivb44/MWUM/fxmark/0x42d

commit: 
  4438cf50e7 ("doc, block, bfq: add information on bfq execution time")
  56a85fd837 ("loop: properly observe rotational flag of underlying device")

4438cf50e7b315ff 56a85fd8376ef32458efb6ea97a 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          0:4            2%           0:4     perf-profile.children.cycles-pp.error_entry
          0:4            3%           0:4     perf-profile.self.cycles-pp.error_entry
         %stddev     %change         %stddev
             \          |                \  
     17.58 ±  4%     -47.0%       9.31 ± 52%  fxmark.hdd_btrfs_MWUM_12_bufferedio.user_util
     10.58 ±  3%     -24.7%       7.97 ± 20%  fxmark.hdd_btrfs_MWUM_16_bufferedio.iowait_util
     20.54 ±  4%      -8.0%      18.89        fxmark.hdd_btrfs_MWUM_1_bufferedio.iowait_sec
     86.59           -17.3%      71.58        fxmark.hdd_btrfs_MWUM_1_bufferedio.iowait_util
     24.03 ±  3%     +10.9%      26.66        fxmark.hdd_btrfs_MWUM_1_bufferedio.real_sec
     23.97 ±  3%     +11.0%      26.60        fxmark.hdd_btrfs_MWUM_1_bufferedio.secs
      0.10 ±  4%     +39.0%       0.14 ±  9%  fxmark.hdd_btrfs_MWUM_1_bufferedio.softirq_sec
      0.43 ±  6%     +24.6%       0.54 ±  7%  fxmark.hdd_btrfs_MWUM_1_bufferedio.softirq_util
      2.83 ±  3%    +148.2%       7.01        fxmark.hdd_btrfs_MWUM_1_bufferedio.sys_sec
     11.92 ±  2%    +122.9%      26.57        fxmark.hdd_btrfs_MWUM_1_bufferedio.sys_util
      0.25 ± 12%     +36.6%       0.35 ±  3%  fxmark.hdd_btrfs_MWUM_1_bufferedio.user_sec
      1.06 ±  8%     +23.2%       1.31 ±  3%  fxmark.hdd_btrfs_MWUM_1_bufferedio.user_util
     47985          +260.0%     172754        fxmark.hdd_btrfs_MWUM_1_bufferedio.works
      2004 ±  4%    +224.0%       6495        fxmark.hdd_btrfs_MWUM_1_bufferedio.works/sec
      0.29 ±  9%     -23.8%       0.22 ±  6%  fxmark.hdd_btrfs_MWUM_2_bufferedio.softirq_util
      2.17 ± 30%     -41.1%       1.28 ±  5%  fxmark.hdd_btrfs_MWUM_2_bufferedio.sys_sec
      4.02 ± 41%     -46.2%       2.16 ±  5%  fxmark.hdd_btrfs_MWUM_2_bufferedio.sys_util
     15122 ±104%     -83.7%       2468 ± 15%  fxmark.hdd_btrfs_MWUM_2_bufferedio.works
    599.22 ±115%     -86.3%      82.18 ± 15%  fxmark.hdd_btrfs_MWUM_2_bufferedio.works/sec
      2747 ±166%     -97.2%      76.08 ± 25%  fxmark.hdd_btrfs_MWUM_4_bufferedio.works/sec
    107.14 ± 13%    +451.3%     590.70 ±  2%  fxmark.time.elapsed_time
    107.14 ± 13%    +451.3%     590.70 ±  2%  fxmark.time.elapsed_time.max
   1092168 ±  6%    +443.2%    5932256 ±  3%  fxmark.time.file_system_outputs
    116421 ±  2%    +314.6%     482732        fxmark.time.involuntary_context_switches
     10.50 ± 10%    +145.2%      25.75 ± 19%  fxmark.time.percent_of_cpu_this_job_got
      6.12 ±  2%    +178.6%      17.05        fxmark.time.system_time
      5.29 ± 21%   +2488.7%     137.01 ± 20%  fxmark.time.user_time
     61250 ± 39%    +161.7%     160299 ±  4%  fxmark.time.voluntary_context_switches
     29.38 ±  8%     -12.2       17.16 ±  4%  mpstat.cpu.all.idle%
      7.55 ± 13%      -2.2        5.32 ±  3%  mpstat.cpu.all.sys%
      6.79 ± 26%     +16.8       23.60 ±  5%  mpstat.cpu.all.usr%
    524443 ±  7%    +221.0%    1683330 ±  4%  numa-numastat.node0.local_node
    533969 ±  6%    +216.0%    1687419 ±  4%  numa-numastat.node0.numa_hit
     92268 ± 47%    +167.2%     246564 ± 23%  numa-numastat.node1.local_node
     98645 ± 44%    +161.9%     258346 ± 21%  numa-numastat.node1.numa_hit
     27.75 ±  9%     -40.5%      16.50 ±  6%  vmstat.cpu.id
      6.25 ± 28%    +272.0%      23.25 ±  5%  vmstat.cpu.us
    178.50 ± 27%     -78.3%      38.75 ±  4%  vmstat.io.bi
     36059 ±  6%     -14.2%      30956 ±  2%  vmstat.io.bo
   1328891           +24.6%    1655470        vmstat.memory.cache
     11112 ± 12%     -20.5%       8834 ±  3%  vmstat.system.cs
     28.80 ±  9%     -40.5%      17.14 ±  5%  iostat.cpu.idle
      7.14 ± 13%     -24.5%       5.39 ±  2%  iostat.cpu.system
      6.46 ± 27%    +263.6%      23.49 ±  5%  iostat.cpu.user
      1.37 ±  3%     -10.7%       1.22        iostat.sda.avgqu-sz
    346.97 ±  6%     -26.2%     255.90 ±  3%  iostat.sda.avgrq-sz
      9.84 ±  2%     +11.0%      10.93        iostat.sda.util
    121.14 ±  2%     +23.7%     149.85        iostat.sda.w/s
     19961 ±  5%     -11.8%      17610        iostat.sda.wkB/s
     88.86 ±  4%      +9.6%      97.40        iostat.sda.wrqm/s
  20279326 ± 35%    +650.7%  1.522e+08 ± 38%  cpuidle.C1.time
    299578 ± 16%    +525.9%    1875124 ± 22%  cpuidle.C1.usage
  96298070 ± 27%    +363.9%  4.467e+08 ± 32%  cpuidle.C1E.time
    324745 ± 29%    +279.6%    1232581 ± 35%  cpuidle.C1E.usage
 1.005e+08 ± 37%    +211.6%  3.131e+08 ±  8%  cpuidle.C3.time
    282616 ± 35%    +139.7%     677380 ± 52%  cpuidle.C3.usage
 2.981e+08 ± 13%    +210.5%  9.256e+08 ±  8%  cpuidle.C6.time
    363597 ±  7%    +191.1%    1058366 ± 11%  cpuidle.C6.usage
     48812 ± 35%   +1775.0%     915237 ±131%  cpuidle.POLL.time
     76720 ± 42%    +418.0%     397431 ± 28%  cpuidle.POLL.usage
    322430           +44.9%     467345        meminfo.Active
     72883 ±  5%    +194.7%     214811        meminfo.Active(file)
    101543 ± 10%     +95.2%     198169        meminfo.AnonHugePages
   1236230           +14.6%    1416469        meminfo.Cached
    778.50 ±  7%     +40.3%       1092 ±  5%  meminfo.Dirty
     55357 ±  2%     +77.4%      98223        meminfo.Inactive
     16246           +38.9%      22567 ±  4%  meminfo.Inactive(anon)
     39110 ±  3%     +93.4%      75655 ±  2%  meminfo.Inactive(file)
     92300          +158.6%     238718        meminfo.KReclaimable
   1942348           +16.1%    2254116        meminfo.Memused
    227.75 ±130%    +641.4%       1688 ± 24%  meminfo.Mlocked
     92300          +158.6%     238718        meminfo.SReclaimable
     89450           +45.9%     130499        meminfo.SUnreclaim
     16467           +43.6%      23644 ±  4%  meminfo.Shmem
    181752          +103.1%     369219        meminfo.Slab
      1233 ±  7%     -64.2%     441.75 ± 10%  meminfo.Writeback
     18996 ± 14%     -77.5%       4271 ±  2%  meminfo.max_used_kB
    191433 ± 26%     +73.9%     332877 ± 15%  numa-meminfo.node0.Active
     72301 ±  5%    +179.2%     201850 ±  3%  numa-meminfo.node0.Active(file)
    755.50 ± 10%     +35.3%       1022 ±  6%  numa-meminfo.node0.Dirty
    676145 ±  4%     +29.0%     872381        numa-meminfo.node0.FilePages
     47747 ±  9%     +89.1%      90308 ±  5%  numa-meminfo.node0.Inactive
     11387 ± 49%     +81.2%      20635 ±  5%  numa-meminfo.node0.Inactive(anon)
     36359 ±  7%     +91.6%      69672 ±  6%  numa-meminfo.node0.Inactive(file)
     68533          +200.0%     205610        numa-meminfo.node0.KReclaimable
   1072189 ±  4%     +33.0%    1425930 ±  4%  numa-meminfo.node0.MemUsed
    131.50 ±130%    +530.8%     829.50 ± 24%  numa-meminfo.node0.Mlocked
     68533          +200.0%     205610        numa-meminfo.node0.SReclaimable
     55745 ±  3%     +70.3%      94941 ±  3%  numa-meminfo.node0.SUnreclaim
     11600 ± 48%     +86.9%      21680 ±  5%  numa-meminfo.node0.Shmem
    124278 ±  2%    +141.8%     300552        numa-meminfo.node0.Slab
    867.50 ± 51%     -68.1%     276.50 ± 25%  numa-meminfo.node0.Writeback
    511.50 ± 47%   +2449.8%      13042 ± 43%  numa-meminfo.node1.Active(file)
     23701 ± 10%     +39.7%      33120 ±  9%  numa-meminfo.node1.KReclaimable
     13731 ± 24%     -21.5%      10779        numa-meminfo.node1.Mapped
     95.75 ±131%    +797.1%     859.00 ± 33%  numa-meminfo.node1.Mlocked
     23701 ± 10%     +39.7%      33120 ±  9%  numa-meminfo.node1.SReclaimable
     18083 ±  5%    +179.1%      50464 ±  3%  numa-vmstat.node0.nr_active_file
    307479 ± 11%    +519.9%    1906179 ± 14%  numa-vmstat.node0.nr_dirtied
    198.25 ±  7%     +29.0%     255.75 ±  7%  numa-vmstat.node0.nr_dirty
    169038 ±  4%     +29.0%     218101        numa-vmstat.node0.nr_file_pages
      2846 ± 49%     +81.2%       5159 ±  5%  numa-vmstat.node0.nr_inactive_anon
      9083 ±  7%     +91.8%      17421 ±  6%  numa-vmstat.node0.nr_inactive_file
     32.75 ±131%    +531.3%     206.75 ± 24%  numa-vmstat.node0.nr_mlock
      2900 ± 48%     +86.9%       5420 ±  5%  numa-vmstat.node0.nr_shmem
     17128          +200.1%      51402        numa-vmstat.node0.nr_slab_reclaimable
     13938 ±  3%     +70.3%      23736 ±  3%  numa-vmstat.node0.nr_slab_unreclaimable
    185.75 ± 47%     -65.0%      65.00 ± 27%  numa-vmstat.node0.nr_writeback
    306114 ± 11%    +521.7%    1903090 ± 14%  numa-vmstat.node0.nr_written
     18083 ±  5%    +179.1%      50464 ±  3%  numa-vmstat.node0.nr_zone_active_file
      2846 ± 49%     +81.2%       5159 ±  5%  numa-vmstat.node0.nr_zone_inactive_anon
      9083 ±  7%     +91.8%      17421 ±  6%  numa-vmstat.node0.nr_zone_inactive_file
    707100 ±  8%     +87.1%    1322828 ±  6%  numa-vmstat.node0.numa_hit
    697656 ±  8%     +89.0%    1318732 ±  6%  numa-vmstat.node0.numa_local
    128.00 ± 47%   +2446.5%       3259 ± 43%  numa-vmstat.node1.nr_active_file
     39945 ± 69%   +1064.1%     465000 ± 35%  numa-vmstat.node1.nr_dirtied
      3432 ± 24%     -21.5%       2694        numa-vmstat.node1.nr_mapped
     23.75 ±131%    +803.2%     214.50 ± 33%  numa-vmstat.node1.nr_mlock
      5924 ± 10%     +39.8%       8280 ±  9%  numa-vmstat.node1.nr_slab_reclaimable
     39837 ± 69%   +1065.8%     464426 ± 35%  numa-vmstat.node1.nr_written
    128.00 ± 47%   +2446.5%       3259 ± 43%  numa-vmstat.node1.nr_zone_active_file
     62408            +1.2%      63137        proc-vmstat.nr_active_anon
     18216 ±  4%    +194.8%      53704        proc-vmstat.nr_active_file
    714319 ± 10%    +500.0%    4285627 ±  3%  proc-vmstat.nr_dirtied
    205.75 ±  6%     +32.9%     273.50 ±  4%  proc-vmstat.nr_dirty
    308956           +14.6%     354129        proc-vmstat.nr_file_pages
      4062           +38.9%       5643 ±  4%  proc-vmstat.nr_inactive_anon
      9675 ±  4%     +95.6%      18921 ±  2%  proc-vmstat.nr_inactive_file
     11885            -1.1%      11758        proc-vmstat.nr_kernel_stack
     57.00 ±130%    +639.9%     421.75 ± 24%  proc-vmstat.nr_mlock
      1122            -3.6%       1082        proc-vmstat.nr_page_table_pages
      4119           +43.5%       5912 ±  4%  proc-vmstat.nr_shmem
     23069          +158.9%      59734        proc-vmstat.nr_slab_reclaimable
     22375           +45.9%      32639        proc-vmstat.nr_slab_unreclaimable
    252.75 ± 11%     -58.3%     105.50 ±  9%  proc-vmstat.nr_writeback
    707337 ±  9%    +502.7%    4262826 ±  3%  proc-vmstat.nr_written
     62408            +1.2%      63137        proc-vmstat.nr_zone_active_anon
     18216 ±  4%    +194.8%      53704        proc-vmstat.nr_zone_active_file
      4062           +38.9%       5643 ±  4%  proc-vmstat.nr_zone_inactive_anon
      9675 ±  4%     +95.6%      18921 ±  2%  proc-vmstat.nr_zone_inactive_file
    447.25 ±  8%     -16.8%     372.00 ±  4%  proc-vmstat.nr_zone_write_pending
    662642 ±  5%    +198.2%    1975999        proc-vmstat.numa_hit
    646735 ±  5%    +203.1%    1960116        proc-vmstat.numa_local
     80763 ±  6%    +217.7%     256554        proc-vmstat.pgactivate
    814052 ±  5%    +201.0%    2450399        proc-vmstat.pgalloc_normal
      1823 ±  7%    +121.4%       4038 ±  5%  proc-vmstat.pgdeactivate
    431214 ±  8%    +250.6%    1511771 ±  2%  proc-vmstat.pgfault
    773651 ±  6%    +207.9%    2382081 ±  2%  proc-vmstat.pgfree
     19211 ± 10%     +21.0%      23238 ±  4%  proc-vmstat.pgpgin
   3859882 ± 10%    +375.5%   18352553 ±  3%  proc-vmstat.pgpgout
     29797            +1.1%      30135        proc-vmstat.pgrotated
    471257          +272.1%    1753661        proc-vmstat.slabs_scanned
     78816          +138.1%     187665        slabinfo.Acpi-Parse.active_objs
      1100          +134.5%       2579        slabinfo.Acpi-Parse.active_slabs
     80343          +134.4%     188306        slabinfo.Acpi-Parse.num_objs
      1100          +134.5%       2579        slabinfo.Acpi-Parse.num_slabs
      1007 ±  6%    +193.5%       2957 ±  2%  slabinfo.Acpi-State.active_objs
      1325 ±  5%    +146.9%       3272 ±  2%  slabinfo.Acpi-State.num_objs
    495.50 ±  2%     +22.5%     607.00 ±  5%  slabinfo.avc_xperms_data.active_objs
    204.75 ±  4%    +161.1%     534.50 ±  6%  slabinfo.blkdev_ioc.active_objs
    496.50 ±  2%     +63.4%     811.50 ±  5%  slabinfo.blkdev_ioc.num_objs
     36217 ±  3%    +301.5%     145411        slabinfo.btrfs_delayed_node.active_objs
    703.50 ±  3%    +298.1%       2800        slabinfo.btrfs_delayed_node.active_slabs
     36612 ±  3%    +297.8%     145644        slabinfo.btrfs_delayed_node.num_objs
    703.50 ±  3%    +298.1%       2800        slabinfo.btrfs_delayed_node.num_slabs
      1382 ±  5%    +214.6%       4350        slabinfo.btrfs_extent_buffer.active_objs
      1722 ±  5%    +170.7%       4662        slabinfo.btrfs_extent_buffer.num_objs
      5553 ±  3%    +112.5%      11801        slabinfo.btrfs_extent_map.active_objs
    108.25 ±  2%    +166.5%     288.50        slabinfo.btrfs_extent_map.active_slabs
      6082 ±  2%    +166.3%      16199        slabinfo.btrfs_extent_map.num_objs
    108.25 ±  2%    +166.5%     288.50        slabinfo.btrfs_extent_map.num_slabs
     35872 ±  3%    +305.2%     145366        slabinfo.btrfs_inode.active_objs
      1291 ±  3%    +302.5%       5196        slabinfo.btrfs_inode.active_slabs
     36151 ±  3%    +302.5%     145512        slabinfo.btrfs_inode.num_objs
      1291 ±  3%    +302.5%       5196        slabinfo.btrfs_inode.num_slabs
    377.50 ± 16%    +135.5%     889.00 ± 14%  slabinfo.btrfs_ordered_extent.active_objs
    572.00 ± 19%     +90.6%       1090 ± 11%  slabinfo.btrfs_ordered_extent.num_objs
      1355 ± 11%    +144.6%       3316 ±  5%  slabinfo.btrfs_path.active_objs
      1537 ± 10%    +127.5%       3496 ±  5%  slabinfo.btrfs_path.num_objs
     81315          +133.1%     189530        slabinfo.dentry.active_objs
      1959          +130.7%       4520        slabinfo.dentry.active_slabs
     82321          +130.6%     189873        slabinfo.dentry.num_objs
      1959          +130.7%       4520        slabinfo.dentry.num_slabs
      1818           +80.5%       3282 ±  4%  slabinfo.kmalloc-128.active_objs
      4048           +16.7%       4723 ±  3%  slabinfo.kmalloc-128.num_objs
      1522 ±  8%     -20.2%       1215        slabinfo.kmalloc-192.active_objs
    289.00           -43.5%     163.25 ±  2%  slabinfo.kmalloc-8k.active_objs
    334.75           -39.2%     203.50 ±  2%  slabinfo.kmalloc-8k.num_objs
    630.75 ± 16%     +37.9%     869.50 ± 12%  slabinfo.proc_inode_cache.active_objs
     13320           +20.5%      16048        slabinfo.radix_tree_node.active_objs
     14608           +13.3%      16544        slabinfo.radix_tree_node.num_objs
     12170 ± 15%    +542.4%      78185 ±  6%  softirqs.BLOCK
      3690 ±  5%    +635.8%      27156 ±  6%  softirqs.CPU0.BLOCK
     27309 ± 10%    +322.2%     115303 ±  5%  softirqs.CPU0.RCU
     14409 ±  8%    +126.3%      32613 ± 11%  softirqs.CPU0.SCHED
      2616 ±  2%    +686.6%      20580 ±  2%  softirqs.CPU0.TASKLET
     56869 ± 11%    +426.4%     299346 ±  3%  softirqs.CPU0.TIMER
      6362 ± 19%    +421.6%      33187 ±  6%  softirqs.CPU1.BLOCK
     14732 ± 17%    +332.9%      63775 ±  3%  softirqs.CPU1.RCU
      8987 ± 10%    +204.7%      27380 ± 10%  softirqs.CPU1.SCHED
      7344 ± 20%    +485.8%      43021 ±  5%  softirqs.CPU1.TASKLET
     35918 ± 13%    +326.5%     153174 ±  9%  softirqs.CPU1.TIMER
      3058 ± 22%    +388.5%      14938 ± 12%  softirqs.CPU16.RCU
      9104 ± 11%    +309.3%      37263 ± 17%  softirqs.CPU16.TIMER
      3235 ± 11%    +425.9%      17011 ±  8%  softirqs.CPU17.RCU
      9182 ±  7%    +300.7%      36795 ± 15%  softirqs.CPU17.TIMER
      3656 ± 43%    +382.4%      17640 ±  7%  softirqs.CPU18.RCU
      9113 ±  9%    +300.5%      36505 ± 17%  softirqs.CPU18.TIMER
      3133 ±  9%    +437.0%      16828 ± 10%  softirqs.CPU19.RCU
      8885 ±  6%    +308.0%      36249 ± 16%  softirqs.CPU19.TIMER
      9057 ± 18%    +382.8%      43727 ±  7%  softirqs.CPU2.RCU
      3663 ± 24%    +212.0%      11428 ± 13%  softirqs.CPU2.SCHED
     25233 ± 35%    +294.5%      99549 ± 10%  softirqs.CPU2.TIMER
      2690 ± 43%    +257.3%       9612 ± 17%  softirqs.CPU20.RCU
      7410 ±  2%    +183.8%      21032 ± 19%  softirqs.CPU20.TIMER
      2654 ± 38%    +277.0%      10008 ± 10%  softirqs.CPU21.RCU
      7516 ±  5%    +233.3%      25051 ± 27%  softirqs.CPU21.TIMER
      2433 ± 40%    +334.3%      10568 ± 31%  softirqs.CPU22.RCU
      9428 ± 39%    +120.4%      20784 ± 20%  softirqs.CPU22.TIMER
      7385 ±  7%    +210.0%      22895 ± 21%  softirqs.CPU23.TIMER
     10963 ± 17%    +312.7%      45240 ±  6%  softirqs.CPU3.RCU
      4502 ± 19%    +125.6%      10157 ± 10%  softirqs.CPU3.SCHED
     25273 ± 17%    +303.7%     102035 ±  6%  softirqs.CPU3.TIMER
      3945 ±  7%    +562.4%      26134 ±  2%  softirqs.CPU4.RCU
     11596 ±  5%    +433.3%      61840 ± 10%  softirqs.CPU4.TIMER
      4012 ±  7%    +552.0%      26160 ±  7%  softirqs.CPU5.RCU
     11690 ±  4%    +424.2%      61275 ± 17%  softirqs.CPU5.TIMER
      4329 ± 18%    +557.8%      28479 ±  4%  softirqs.CPU6.RCU
     11685 ±  3%    +417.8%      60510 ± 14%  softirqs.CPU6.TIMER
      4890 ±  8%    +412.6%      25065 ± 12%  softirqs.CPU7.RCU
     12680 ± 13%    +400.3%      63441 ± 15%  softirqs.CPU7.TIMER
    153116 ± 10%    +247.1%     531466 ±  4%  softirqs.RCU
    110587 ±  4%     +76.4%     195064 ±  4%  softirqs.SCHED
     10691 ± 14%    +501.7%      64332 ±  2%  softirqs.TASKLET
    465137 ±  5%    +189.7%    1347412 ±  8%  softirqs.TIMER
      0.12 ± 22%     -80.1%       0.02 ± 60%  perf-stat.i.MPKI
  32264538 ± 34%     -84.8%    4908413 ± 33%  perf-stat.i.branch-instructions
      0.14 ± 20%      -0.1        0.02 ±  6%  perf-stat.i.branch-miss-rate%
   2347695 ± 38%     -84.9%     354908 ± 34%  perf-stat.i.branch-misses
      0.55 ± 20%      -0.4        0.10 ±  3%  perf-stat.i.cache-miss-rate%
    218451 ± 26%     -83.1%      36882 ±  6%  perf-stat.i.cache-misses
    765568 ± 23%     -84.2%     120694 ±  2%  perf-stat.i.cache-references
     11274 ± 10%     -21.7%       8829 ±  3%  perf-stat.i.context-switches
      0.03 ± 13%     -80.8%       0.01 ± 34%  perf-stat.i.cpi
 1.904e+08 ± 29%     -84.4%   29787431 ± 16%  perf-stat.i.cpu-cycles
    267.06 ±  2%     -17.0%     221.78 ±  4%  perf-stat.i.cpu-migrations
     16.67 ± 19%     -83.5%       2.74 ± 21%  perf-stat.i.cycles-between-cache-misses
      0.00 ± 17%      -0.0        0.00 ± 40%  perf-stat.i.dTLB-load-miss-rate%
     58282 ± 12%     -81.8%      10594 ± 12%  perf-stat.i.dTLB-load-misses
  30020546 ± 33%     -84.6%    4617083 ± 30%  perf-stat.i.dTLB-loads
      0.00 ±  3%      -0.0        0.00 ± 21%  perf-stat.i.dTLB-store-miss-rate%
      8607 ± 12%     -83.2%       1448 ±  4%  perf-stat.i.dTLB-store-misses
  15481321 ± 25%     -84.7%    2370348 ± 20%  perf-stat.i.dTLB-stores
      1.62 ± 16%      -1.3        0.28 ±  2%  perf-stat.i.iTLB-load-miss-rate%
     17137 ± 13%     -83.3%       2865 ±  4%  perf-stat.i.iTLB-load-misses
      3563 ±  6%     -82.2%     634.96 ±  5%  perf-stat.i.iTLB-loads
  1.53e+08 ± 34%     -84.8%   23283520 ± 33%  perf-stat.i.instructions
    164.81 ± 38%     -83.5%      27.27 ± 33%  perf-stat.i.instructions-per-iTLB-miss
      0.01 ± 22%     -82.9%       0.00 ± 23%  perf-stat.i.ipc
      3737 ±  5%     -34.5%       2449        perf-stat.i.minor-faults
      0.39 ± 10%      -0.3        0.07 ± 25%  perf-stat.i.node-load-miss-rate%
      7270 ±  9%     -80.5%       1414 ± 24%  perf-stat.i.node-load-misses
     43041 ± 28%     -83.3%       7194 ± 20%  perf-stat.i.node-loads
      0.28 ± 28%      -0.2        0.06 ± 40%  perf-stat.i.node-store-miss-rate%
     46796 ± 52%     -81.0%       8897 ± 30%  perf-stat.i.node-stores
      3738 ±  5%     -34.5%       2449        perf-stat.i.page-faults
  32033779 ± 34%     -84.6%    4918392 ± 33%  perf-stat.ps.branch-instructions
   2331066 ± 38%     -84.7%     355655 ± 34%  perf-stat.ps.branch-misses
    217143 ± 26%     -83.0%      36975 ±  6%  perf-stat.ps.cache-misses
    760933 ± 23%     -84.1%     120998 ±  2%  perf-stat.ps.cache-references
     11164 ± 10%     -21.0%       8816 ±  3%  perf-stat.ps.context-switches
 1.892e+08 ± 28%     -84.2%   29855538 ± 16%  perf-stat.ps.cpu-cycles
    264.50 ±  2%     -16.3%     221.45 ±  4%  perf-stat.ps.cpu-migrations
     57898 ± 12%     -81.7%      10617 ± 12%  perf-stat.ps.dTLB-load-misses
  29806077 ± 32%     -84.5%    4626290 ± 30%  perf-stat.ps.dTLB-loads
      8557 ± 11%     -83.0%       1452 ±  4%  perf-stat.ps.dTLB-store-misses
  15376824 ± 25%     -84.6%    2375476 ± 20%  perf-stat.ps.dTLB-stores
     17027 ± 13%     -83.1%       2871 ±  4%  perf-stat.ps.iTLB-load-misses
      3538 ±  6%     -82.0%     635.92 ±  5%  perf-stat.ps.iTLB-loads
 1.519e+08 ± 34%     -84.6%   23330678 ± 33%  perf-stat.ps.instructions
      3702 ±  5%     -34.0%       2445        perf-stat.ps.minor-faults
      7222 ±  9%     -80.4%       1417 ± 24%  perf-stat.ps.node-load-misses
     42707 ± 28%     -83.1%       7201 ± 20%  perf-stat.ps.node-loads
     46491 ± 52%     -80.8%       8914 ± 30%  perf-stat.ps.node-stores
      3703 ±  5%     -34.0%       2445        perf-stat.ps.page-faults
     90.25 ± 14%    +487.3%     530.00 ± 20%  interrupts.35:IR-PCI-MSI.2621441-edge.eth0-TxRx-0
    114.00 ± 46%    +168.2%     305.75 ±  2%  interrupts.36:IR-PCI-MSI.2621442-edge.eth0-TxRx-1
     58.00 ± 17%    +764.7%     501.50 ± 39%  interrupts.37:IR-PCI-MSI.2621443-edge.eth0-TxRx-2
    150.75 ± 80%    +267.7%     554.25 ± 40%  interrupts.40:IR-PCI-MSI.2621446-edge.eth0-TxRx-5
     79.75 ± 54%    +392.2%     392.50 ± 38%  interrupts.41:IR-PCI-MSI.2621447-edge.eth0-TxRx-6
     58.00 ± 16%    +563.4%     384.75 ± 21%  interrupts.42:IR-PCI-MSI.2621448-edge.eth0-TxRx-7
     14555 ± 60%    +774.3%     127259 ±  2%  interrupts.75:IR-PCI-MSI.2097152-edge.isci-msix
     83416 ± 14%     +46.7%     122365 ± 12%  interrupts.CAL:Function_call_interrupts
     33.75 ± 33%    +462.2%     189.75 ± 17%  interrupts.CPU0.35:IR-PCI-MSI.2621441-edge.eth0-TxRx-0
     83.00 ± 62%    +207.2%     255.00 ± 34%  interrupts.CPU0.36:IR-PCI-MSI.2621442-edge.eth0-TxRx-1
     16.75 ± 25%   +1120.9%     204.50 ± 52%  interrupts.CPU0.37:IR-PCI-MSI.2621443-edge.eth0-TxRx-2
     81.75 ± 81%    +244.3%     281.50 ± 39%  interrupts.CPU0.40:IR-PCI-MSI.2621446-edge.eth0-TxRx-5
     24.00 ± 55%    +508.3%     146.00 ± 53%  interrupts.CPU0.41:IR-PCI-MSI.2621447-edge.eth0-TxRx-6
     15.25 ±  5%    +713.1%     124.00 ± 21%  interrupts.CPU0.42:IR-PCI-MSI.2621448-edge.eth0-TxRx-7
      3862 ± 57%    +963.2%      41061 ±  2%  interrupts.CPU0.75:IR-PCI-MSI.2097152-edge.isci-msix
      3046 ±  6%    +185.7%       8703 ± 32%  interrupts.CPU0.CAL:Function_call_interrupts
    215872 ± 13%    +444.1%    1174553 ±  4%  interrupts.CPU0.LOC:Local_timer_interrupts
     13610 ± 36%    +177.6%      37781 ±  5%  interrupts.CPU0.RES:Rescheduling_interrupts
    133.50 ± 20%    +186.7%     382.75 ± 18%  interrupts.CPU0.TLB:TLB_shootdowns
     39.25 ± 18%    +581.5%     267.50 ± 24%  interrupts.CPU1.37:IR-PCI-MSI.2621443-edge.eth0-TxRx-2
     70.50 ± 20%     +74.5%     123.00 ±  4%  interrupts.CPU1.39:IR-PCI-MSI.2621445-edge.eth0-TxRx-4
     47.50 ± 40%    +394.2%     234.75 ± 23%  interrupts.CPU1.41:IR-PCI-MSI.2621447-edge.eth0-TxRx-6
     18.75 ±  6%    +348.0%      84.00 ± 18%  interrupts.CPU1.42:IR-PCI-MSI.2621448-edge.eth0-TxRx-7
     10570 ± 63%    +713.9%      86036 ±  5%  interrupts.CPU1.75:IR-PCI-MSI.2097152-edge.isci-msix
      2771 ± 12%    +151.5%       6971 ± 22%  interrupts.CPU1.CAL:Function_call_interrupts
    158997 ± 16%    +401.8%     797825 ±  6%  interrupts.CPU1.LOC:Local_timer_interrupts
      1.75 ±173%  +3.5e+05%       6043 ± 36%  interrupts.CPU1.NMI:Non-maskable_interrupts
      1.75 ±173%  +3.5e+05%       6043 ± 36%  interrupts.CPU1.PMI:Performance_monitoring_interrupts
     12388 ± 57%    +157.6%      31915 ±  7%  interrupts.CPU1.RES:Rescheduling_interrupts
     86.00 ± 18%    +162.2%     225.50 ± 20%  interrupts.CPU1.TLB:TLB_shootdowns
      2.25 ± 96%   +4522.2%     104.00 ±162%  interrupts.CPU15.RES:Rescheduling_interrupts
      2108 ± 17%    +176.0%       5817 ± 45%  interrupts.CPU16.CAL:Function_call_interrupts
     23021 ±  7%    +736.9%     192658 ± 13%  interrupts.CPU16.LOC:Local_timer_interrupts
    507.00 ± 78%    +390.3%       2486 ± 40%  interrupts.CPU16.RES:Rescheduling_interrupts
      2179 ± 15%     +70.4%       3714 ±  4%  interrupts.CPU17.CAL:Function_call_interrupts
     23398 ±  7%    +721.0%     192090 ± 13%  interrupts.CPU17.LOC:Local_timer_interrupts
    680.00 ± 89%    +136.8%       1610 ± 28%  interrupts.CPU17.RES:Rescheduling_interrupts
      2453 ± 15%     +85.4%       4548 ± 25%  interrupts.CPU18.CAL:Function_call_interrupts
     24008 ±  6%    +703.4%     192897 ± 13%  interrupts.CPU18.LOC:Local_timer_interrupts
      2090 ± 11%    +120.6%       4610 ± 46%  interrupts.CPU19.CAL:Function_call_interrupts
     23873 ±  7%    +706.9%     192644 ± 13%  interrupts.CPU19.LOC:Local_timer_interrupts
    577.75 ± 91%    +226.9%       1888 ± 30%  interrupts.CPU19.RES:Rescheduling_interrupts
     19.50 ± 53%    +306.4%      79.25 ± 21%  interrupts.CPU2.40:IR-PCI-MSI.2621446-edge.eth0-TxRx-5
     22.00 ± 34%    +678.4%     171.25 ± 22%  interrupts.CPU2.42:IR-PCI-MSI.2621448-edge.eth0-TxRx-7
      2630 ±  5%    +117.0%       5707 ± 22%  interrupts.CPU2.CAL:Function_call_interrupts
     88898 ± 29%    +479.2%     514888 ± 10%  interrupts.CPU2.LOC:Local_timer_interrupts
      0.00         +4e+105%       4025 ± 49%  interrupts.CPU2.NMI:Non-maskable_interrupts
      0.00         +4e+105%       4025 ± 49%  interrupts.CPU2.PMI:Performance_monitoring_interrupts
      5082 ± 75%    +146.1%      12508 ± 12%  interrupts.CPU2.RES:Rescheduling_interrupts
     37.50 ± 51%    +149.3%      93.50 ± 44%  interrupts.CPU2.TLB:TLB_shootdowns
      1983 ± 13%     +37.3%       2722 ± 13%  interrupts.CPU20.CAL:Function_call_interrupts
     15478 ±  5%    +548.7%     100401 ± 24%  interrupts.CPU20.LOC:Local_timer_interrupts
     15466 ±  4%    +550.7%     100638 ± 24%  interrupts.CPU21.LOC:Local_timer_interrupts
     30.75 ± 25%    +861.8%     295.75 ± 32%  interrupts.CPU21.RES:Rescheduling_interrupts
      1859 ± 16%     +45.2%       2700 ± 22%  interrupts.CPU22.CAL:Function_call_interrupts
     15198 ±  3%    +561.7%     100566 ± 24%  interrupts.CPU22.LOC:Local_timer_interrupts
      1906 ± 15%    +129.3%       4372 ± 48%  interrupts.CPU23.CAL:Function_call_interrupts
     15510 ±  4%    +547.1%     100373 ± 24%  interrupts.CPU23.LOC:Local_timer_interrupts
    227.00 ± 90%    +863.1%       2186 ± 78%  interrupts.CPU23.RES:Rescheduling_interrupts
     46.25 ± 97%    +224.9%     150.25 ± 10%  interrupts.CPU3.38:IR-PCI-MSI.2621444-edge.eth0-TxRx-3
      2824 ± 19%    +117.8%       6150 ± 26%  interrupts.CPU3.CAL:Function_call_interrupts
     88752 ± 29%    +479.3%     514160 ± 10%  interrupts.CPU3.LOC:Local_timer_interrupts
      4973 ± 30%    +107.9%      10339 ± 16%  interrupts.CPU3.RES:Rescheduling_interrupts
      2271 ± 10%    +115.2%       4887 ± 23%  interrupts.CPU4.CAL:Function_call_interrupts
     33054 ±  9%    +852.6%     314879 ± 10%  interrupts.CPU4.LOC:Local_timer_interrupts
      1762 ± 36%    +122.6%       3922 ± 36%  interrupts.CPU4.RES:Rescheduling_interrupts
      2366 ±  7%    +164.8%       6265 ± 36%  interrupts.CPU5.CAL:Function_call_interrupts
     33087 ± 10%    +851.6%     314849 ± 10%  interrupts.CPU5.LOC:Local_timer_interrupts
      1270 ± 41%    +181.5%       3575 ± 40%  interrupts.CPU5.RES:Rescheduling_interrupts
      2543 ± 10%    +115.9%       5493 ± 19%  interrupts.CPU6.CAL:Function_call_interrupts
     33202 ±  9%    +847.9%     314737 ± 10%  interrupts.CPU6.LOC:Local_timer_interrupts
      1275 ± 70%    +188.7%       3683 ± 31%  interrupts.CPU6.RES:Rescheduling_interrupts
     32.75 ±110%    +425.2%     172.00 ± 45%  interrupts.CPU7.40:IR-PCI-MSI.2621446-edge.eth0-TxRx-5
      2281 ± 13%    +148.9%       5679 ± 24%  interrupts.CPU7.CAL:Function_call_interrupts
     33380 ±  9%    +844.3%     315206 ± 10%  interrupts.CPU7.LOC:Local_timer_interrupts
    578.75 ± 34%    +526.6%       3626 ± 40%  interrupts.CPU7.RES:Rescheduling_interrupts
   1047530 ± 12%    +438.4%    5640297 ±  9%  interrupts.LOC:Local_timer_interrupts
    863.25 ± 59%   +4898.7%      43151 ± 24%  interrupts.NMI:Non-maskable_interrupts
    863.25 ± 59%   +4898.7%      43151 ± 24%  interrupts.PMI:Performance_monitoring_interrupts
     45861 ± 29%    +162.8%     120545 ±  2%  interrupts.RES:Rescheduling_interrupts
    343.25 ± 14%    +180.6%     963.25 ± 17%  interrupts.TLB:TLB_shootdowns
    959.64 ±100%     -79.6%     195.46 ±173%  sched_debug.cfs_rq:/.MIN_vruntime.stddev
     17007 ± 62%   +1860.2%     333391 ± 98%  sched_debug.cfs_rq:/.load.avg
      0.00       +1.1e+107%     111405 ± 24%  sched_debug.cfs_rq:/.load.min
    188.18 ± 17%    +990.0%       2051 ± 83%  sched_debug.cfs_rq:/.load_avg.avg
      1296 ± 27%    +175.7%       3574 ± 54%  sched_debug.cfs_rq:/.load_avg.max
      4.75 ± 47%  +31555.4%       1503 ± 96%  sched_debug.cfs_rq:/.load_avg.min
    959.64 ±100%     -79.6%     195.46 ±173%  sched_debug.cfs_rq:/.max_vruntime.stddev
     11846 ±  6%   +7399.1%     888367 ± 11%  sched_debug.cfs_rq:/.min_vruntime.avg
     33194 ± 13%   +2749.7%     945932 ± 10%  sched_debug.cfs_rq:/.min_vruntime.max
      6672 ±  8%  +11820.6%     795381 ± 14%  sched_debug.cfs_rq:/.min_vruntime.min
      4324 ±  8%   +1296.5%      60389 ± 27%  sched_debug.cfs_rq:/.min_vruntime.stddev
      0.29 ± 13%    +148.6%       0.73 ±  5%  sched_debug.cfs_rq:/.nr_running.avg
      0.00       +5.2e+101%       0.52 ± 13%  sched_debug.cfs_rq:/.nr_running.min
      0.45 ±  4%     -51.6%       0.22 ± 16%  sched_debug.cfs_rq:/.nr_running.stddev
    121.51 ± 22%     -93.8%       7.58 ± 34%  sched_debug.cfs_rq:/.removed.load_avg.avg
      1024           -88.0%     123.37 ± 38%  sched_debug.cfs_rq:/.removed.load_avg.max
    326.63 ± 10%     -91.3%      28.46 ± 31%  sched_debug.cfs_rq:/.removed.load_avg.stddev
      5605 ± 22%     -93.8%     349.00 ± 34%  sched_debug.cfs_rq:/.removed.runnable_sum.avg
     47413           -88.0%       5684 ± 37%  sched_debug.cfs_rq:/.removed.runnable_sum.max
     15063 ± 10%     -91.3%       1308 ± 31%  sched_debug.cfs_rq:/.removed.runnable_sum.stddev
     39.99 ± 20%     -92.6%       2.96 ± 44%  sched_debug.cfs_rq:/.removed.util_avg.avg
    431.75 ± 13%     -87.9%      52.25 ± 34%  sched_debug.cfs_rq:/.removed.util_avg.max
    111.67 ±  9%     -90.1%      11.03 ± 41%  sched_debug.cfs_rq:/.removed.util_avg.stddev
      9.48 ± 97%   +1659.9%     166.91 ± 57%  sched_debug.cfs_rq:/.runnable_load_avg.avg
      0.00       +8.7e+103%      86.54 ± 25%  sched_debug.cfs_rq:/.runnable_load_avg.min
     15572 ± 68%   +1898.0%     311140 ±103%  sched_debug.cfs_rq:/.runnable_weight.avg
      0.00       +9.5e+106%      94876 ± 22%  sched_debug.cfs_rq:/.runnable_weight.min
   -270.52        +25569.1%     -69440        sched_debug.cfs_rq:/.spread0.min
      4326 ±  8%   +1295.8%      60390 ± 27%  sched_debug.cfs_rq:/.spread0.stddev
      1641 ± 26%     -51.0%     803.39 ±  5%  sched_debug.cfs_rq:/.util_avg.max
    102.00 ± 92%    +202.1%     308.14 ± 23%  sched_debug.cfs_rq:/.util_avg.min
    289.84 ± 12%     -41.1%     170.78 ± 21%  sched_debug.cfs_rq:/.util_avg.stddev
    443.50 ±  7%     -72.5%     121.90 ± 64%  sched_debug.cfs_rq:/.util_est_enqueued.max
      0.00       +3.4e+103%      34.41 ±116%  sched_debug.cfs_rq:/.util_est_enqueued.min
    121.85 ± 13%     -78.4%      26.33 ± 53%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
    540117 ±  3%     +46.0%     788667 ±  3%  sched_debug.cpu.avg_idle.avg
     15537 ±127%   +3862.8%     615726 ±  7%  sched_debug.cpu.avg_idle.min
    246385 ±  5%     -54.1%     112968 ± 26%  sched_debug.cpu.avg_idle.stddev
     32423          +882.0%     318397 ±  5%  sched_debug.cpu.clock.avg
     32425          +881.9%     318397 ±  5%  sched_debug.cpu.clock.max
     32420          +882.1%     318396 ±  5%  sched_debug.cpu.clock.min
      1.27 ±  5%     -73.8%       0.33 ± 16%  sched_debug.cpu.clock.stddev
     32423          +882.0%     318397 ±  5%  sched_debug.cpu.clock_task.avg
     32425          +881.9%     318397 ±  5%  sched_debug.cpu.clock_task.max
     32420          +882.1%     318396 ±  5%  sched_debug.cpu.clock_task.min
      1.27 ±  5%     -73.8%       0.33 ± 16%  sched_debug.cpu.clock_task.stddev
     10.12 ± 94%   +1554.2%     167.40 ± 61%  sched_debug.cpu.cpu_load[0].avg
      0.00       +8.9e+103%      89.29 ± 33%  sched_debug.cpu.cpu_load[0].min
      8.52 ± 61%   +1877.0%     168.35 ± 44%  sched_debug.cpu.cpu_load[1].avg
      0.00       +9.2e+103%      91.70 ± 27%  sched_debug.cpu.cpu_load[1].min
     21.44 ±129%    +743.7%     180.86 ±113%  sched_debug.cpu.cpu_load[1].stddev
      8.85 ± 36%   +1590.0%     149.64 ± 44%  sched_debug.cpu.cpu_load[2].avg
     83.25 ±117%    +630.1%     607.79 ±107%  sched_debug.cpu.cpu_load[2].max
      0.00       +8.2e+103%      82.36 ± 40%  sched_debug.cpu.cpu_load[2].min
     13.58 ± 99%   +1053.0%     156.60 ±114%  sched_debug.cpu.cpu_load[2].stddev
      9.90 ± 23%   +1155.0%     124.26 ± 48%  sched_debug.cpu.cpu_load[3].avg
     76.50 ± 42%    +541.2%     490.54 ±111%  sched_debug.cpu.cpu_load[3].max
      0.00       +6.4e+103%      63.96 ± 42%  sched_debug.cpu.cpu_load[3].min
     12.92 ± 44%    +896.9%     128.83 ±116%  sched_debug.cpu.cpu_load[3].stddev
     11.53 ± 14%    +750.7%      98.10 ± 49%  sched_debug.cpu.cpu_load[4].avg
      0.00       +4.7e+103%      47.34 ± 37%  sched_debug.cpu.cpu_load[4].min
    420.46 ± 17%   +1601.5%       7153 ±  4%  sched_debug.cpu.curr->pid.avg
      1627          +467.6%       9236 ±  4%  sched_debug.cpu.curr->pid.max
      0.00       +5.7e+105%       5712 ±  8%  sched_debug.cpu.curr->pid.min
    668.78 ±  7%    +133.1%       1558 ±  7%  sched_debug.cpu.curr->pid.stddev
     16656 ± 66%   +1924.8%     337260 ± 98%  sched_debug.cpu.load.avg
      0.00       +1.2e+107%     120213 ± 27%  sched_debug.cpu.load.min
      7312 ±  2%   +3882.6%     291227 ±  5%  sched_debug.cpu.nr_load_updates.avg
     14652 ±  2%   +1900.1%     293066 ±  4%  sched_debug.cpu.nr_load_updates.max
      5756 ±  2%   +4946.5%     290501 ±  5%  sched_debug.cpu.nr_load_updates.min
      1428 ±  2%     -48.9%     730.36 ± 50%  sched_debug.cpu.nr_load_updates.stddev
      0.29 ± 16%    +182.4%       0.82 ±  9%  sched_debug.cpu.nr_running.avg
      0.00       +6.4e+101%       0.64 ± 14%  sched_debug.cpu.nr_running.min
      0.45 ±  5%     -53.5%       0.21 ±  9%  sched_debug.cpu.nr_running.stddev
      1375        +43827.9%     604207 ±  3%  sched_debug.cpu.nr_switches.avg
     13207 ± 12%   +4702.9%     634333 ±  3%  sched_debug.cpu.nr_switches.max
    310.50 ±  7%  +1.9e+05%     581402 ±  4%  sched_debug.cpu.nr_switches.min
      2126 ±  9%    +906.4%      21402 ± 24%  sched_debug.cpu.nr_switches.stddev
     10.25 ± 27%   +1350.4%     148.66 ± 40%  sched_debug.cpu.nr_uninterruptible.max
      3.94 ± 11%   +2183.1%      89.93 ± 57%  sched_debug.cpu.nr_uninterruptible.stddev
     32420          +882.1%     318396 ±  5%  sched_debug.cpu_clk
     29544          +968.0%     315520 ±  5%  sched_debug.ktime
     32903          +869.1%     318879 ±  5%  sched_debug.sched_clk
     24.00           -38.6%      14.73 ±  2%  sched_debug.sysctl_sched.sysctl_sched_latency
      3.00           -38.6%       1.84 ±  2%  sched_debug.sysctl_sched.sysctl_sched_min_granularity
      4.00           -38.6%       2.45 ±  2%  sched_debug.sysctl_sched.sysctl_sched_wakeup_granularity
     17.69 ± 73%     -17.7        0.00        perf-profile.calltrace.cycles-pp.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      9.33 ± 71%      -9.3        0.00        perf-profile.calltrace.cycles-pp.vfs_unlink.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      8.87 ± 71%      -8.9        0.00        perf-profile.calltrace.cycles-pp.btrfs_unlink.vfs_unlink.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      6.61 ± 69%      -6.6        0.00        perf-profile.calltrace.cycles-pp.btrfs_unlink_inode.btrfs_unlink.vfs_unlink.do_unlinkat.do_syscall_64
      6.61 ± 69%      -6.6        0.00        perf-profile.calltrace.cycles-pp.__btrfs_unlink_inode.btrfs_unlink_inode.btrfs_unlink.vfs_unlink.do_unlinkat
      6.50 ± 76%      -6.5        0.00        perf-profile.calltrace.cycles-pp.evict.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.97 ± 72%      -6.0        0.00        perf-profile.calltrace.cycles-pp.btrfs_evict_inode.evict.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      4.96 ± 66%      -5.0        0.00        perf-profile.calltrace.cycles-pp.btrfs_lookup_dir_item.__btrfs_unlink_inode.btrfs_unlink_inode.btrfs_unlink.vfs_unlink
      4.70 ± 67%      -4.7        0.00        perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_lookup_dir_item.__btrfs_unlink_inode.btrfs_unlink_inode.btrfs_unlink
      0.00            +0.7        0.71 ± 18%  perf-profile.calltrace.cycles-pp.task_tick_fair.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer
      0.00            +0.8        0.76 ±  8%  perf-profile.calltrace.cycles-pp.push_leaf_right.split_leaf.btrfs_search_slot.btrfs_insert_empty_items.insert_with_overflow
      0.00            +0.8        0.84 ± 24%  perf-profile.calltrace.cycles-pp.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry
      0.00            +1.0        1.01 ± 11%  perf-profile.calltrace.cycles-pp.split_leaf.btrfs_search_slot.btrfs_insert_empty_items.insert_with_overflow.btrfs_insert_dir_item
      0.00            +1.0        1.05 ± 10%  perf-profile.calltrace.cycles-pp.__sched_text_start.schedule_idle.do_idle.cpu_startup_entry.start_secondary
      0.00            +1.1        1.05 ± 10%  perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      0.00            +1.1        1.09 ± 20%  perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry.start_secondary
      0.00            +1.3        1.28 ± 40%  perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
      0.00            +1.5        1.49 ± 50%  perf-profile.calltrace.cycles-pp.btrfs_lookup_dentry.btrfs_lookup.path_openat.do_filp_open.do_sys_open
      0.00            +1.5        1.50 ± 50%  perf-profile.calltrace.cycles-pp.btrfs_lookup.path_openat.do_filp_open.do_sys_open.do_syscall_64
      0.00            +1.5        1.53 ± 30%  perf-profile.calltrace.cycles-pp.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
      0.00            +1.9        1.92 ± 67%  perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_insert_empty_items.btrfs_new_inode.btrfs_create.path_openat
      0.00            +2.0        1.96 ± 22%  perf-profile.calltrace.cycles-pp.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle
      0.20 ±173%      +2.0        2.17 ± 59%  perf-profile.calltrace.cycles-pp.__btrfs_update_delayed_inode.__btrfs_run_delayed_items.flush_space.btrfs_async_reclaim_metadata_space.process_one_work
      0.00            +2.1        2.12 ± 66%  perf-profile.calltrace.cycles-pp.btrfs_insert_empty_items.btrfs_new_inode.btrfs_create.path_openat.do_filp_open
      0.00            +2.1        2.13 ± 66%  perf-profile.calltrace.cycles-pp.btrfs_insert_empty_items.__btrfs_run_delayed_items.flush_space.btrfs_async_reclaim_metadata_space.process_one_work
      0.35 ±173%      +2.3        2.67 ± 24%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
      0.00            +2.6        2.55 ± 25%  perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      1.03 ±103%      +2.9        3.92 ± 32%  perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
      1.03 ±103%      +3.1        4.13 ± 33%  perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt
      0.78 ± 88%      +3.4        4.14 ± 27%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle
      1.25 ±100%      +3.5        4.73 ± 30%  perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      0.00            +3.7        3.70 ± 28%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      0.00            +4.3        4.30 ± 61%  perf-profile.calltrace.cycles-pp.btrfs_new_inode.btrfs_create.path_openat.do_filp_open.do_sys_open
      0.00            +4.4        4.39 ± 22%  perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_insert_empty_items.insert_with_overflow.btrfs_insert_dir_item.btrfs_add_link
      0.00            +4.6        4.60 ± 29%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      0.00            +5.3        5.28 ± 20%  perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt
      0.00            +5.4        5.42 ± 19%  perf-profile.calltrace.cycles-pp.apic_timer_interrupt
      0.00            +6.0        5.96 ± 15%  perf-profile.calltrace.cycles-pp.btrfs_insert_empty_items.insert_with_overflow.btrfs_insert_dir_item.btrfs_add_link.btrfs_create
      0.00            +6.0        6.02 ± 14%  perf-profile.calltrace.cycles-pp.insert_with_overflow.btrfs_insert_dir_item.btrfs_add_link.btrfs_create.path_openat
      1.15 ± 75%      +6.1        7.28 ± 22%  perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry
      0.00            +6.8        6.84 ± 17%  perf-profile.calltrace.cycles-pp.btrfs_insert_dir_item.btrfs_add_link.btrfs_create.path_openat.do_filp_open
      0.00            +7.2        7.19 ± 18%  perf-profile.calltrace.cycles-pp.btrfs_add_link.btrfs_create.path_openat.do_filp_open.do_sys_open
      0.00            +7.4        7.36 ± 22%  perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary
      0.00           +15.6       15.58 ± 37%  perf-profile.calltrace.cycles-pp.btrfs_create.path_openat.do_filp_open.do_sys_open.do_syscall_64
      0.00           +17.5       17.47 ± 23%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary
      0.00           +19.6       19.60 ± 40%  perf-profile.calltrace.cycles-pp.path_openat.do_filp_open.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00           +26.5       26.45 ± 23%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      6.59 ± 65%     +27.7       34.28 ± 18%  perf-profile.calltrace.cycles-pp.secondary_startup_64
      0.00           +31.6       31.64 ± 19%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      0.00           +31.7       31.70 ± 19%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
      0.00           +31.7       31.70 ± 19%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
     17.69 ± 73%     -17.6        0.10 ±173%  perf-profile.children.cycles-pp.do_unlinkat
      9.33 ± 71%      -9.3        0.04 ±173%  perf-profile.children.cycles-pp.vfs_unlink
      8.87 ± 71%      -8.8        0.04 ±173%  perf-profile.children.cycles-pp.btrfs_unlink
      6.61 ± 69%      -6.6        0.03 ±173%  perf-profile.children.cycles-pp.__btrfs_unlink_inode
      6.61 ± 69%      -6.6        0.03 ±173%  perf-profile.children.cycles-pp.btrfs_unlink_inode
      0.02 ±173%      +0.1        0.10 ± 31%  perf-profile.children.cycles-pp.__intel_pmu_enable_all
      0.00            +0.1        0.08 ± 26%  perf-profile.children.cycles-pp.__alloc_file
      0.00            +0.1        0.09 ± 32%  perf-profile.children.cycles-pp.resched_curr
      0.00            +0.1        0.10 ± 35%  perf-profile.children.cycles-pp.alloc_empty_file
      0.03 ±173%      +0.1        0.12 ± 45%  perf-profile.children.cycles-pp.reweight_entity
      0.00            +0.1        0.10 ± 35%  perf-profile.children.cycles-pp.tick_nohz_idle_enter
      0.00            +0.1        0.10 ± 40%  perf-profile.children.cycles-pp.hrtimer_forward
      0.00            +0.1        0.10 ± 36%  perf-profile.children.cycles-pp.find_inode
      0.04 ±100%      +0.1        0.14 ± 31%  perf-profile.children.cycles-pp.sched_clock_idle_wakeup_event
      0.00            +0.1        0.10 ± 43%  perf-profile.children.cycles-pp.cpuidle_not_available
      0.04 ±100%      +0.1        0.15 ± 32%  perf-profile.children.cycles-pp.cpu_load_update
      0.02 ±173%      +0.1        0.12 ± 29%  perf-profile.children.cycles-pp.rcu_irq_enter
      0.00            +0.1        0.11 ± 31%  perf-profile.children.cycles-pp.note_gp_changes
      0.00            +0.1        0.11 ± 51%  perf-profile.children.cycles-pp.btrfs_alloc_delayed_item
      0.02 ±173%      +0.1        0.14 ± 33%  perf-profile.children.cycles-pp.run_local_timers
      0.04 ±100%      +0.1        0.16 ± 30%  perf-profile.children.cycles-pp.rb_insert_color
      0.00            +0.1        0.12 ± 28%  perf-profile.children.cycles-pp.__hrtimer_get_next_event
      0.00            +0.1        0.12 ± 67%  perf-profile.children.cycles-pp.hrtimer_active
      0.00            +0.1        0.12 ± 53%  perf-profile.children.cycles-pp.mutex_spin_on_owner
      0.05 ±100%      +0.1        0.17 ± 18%  perf-profile.children.cycles-pp.rcu_idle_exit
      0.11 ± 62%      +0.1        0.23 ± 16%  perf-profile.children.cycles-pp.__switch_to_asm
      0.02 ±173%      +0.1        0.15 ± 18%  perf-profile.children.cycles-pp.switch_mm_irqs_off
      0.02 ±173%      +0.1        0.15 ± 50%  perf-profile.children.cycles-pp.perf_event_task_tick
      0.02 ±173%      +0.1        0.15 ± 35%  perf-profile.children.cycles-pp.nr_iowait_cpu
      0.02 ±173%      +0.1        0.16 ± 61%  perf-profile.children.cycles-pp.hrtimer_get_next_event
      0.02 ±173%      +0.1        0.16 ± 22%  perf-profile.children.cycles-pp.cpuacct_charge
      0.00            +0.1        0.14 ± 15%  perf-profile.children.cycles-pp.get_cpu_device
      0.02 ±173%      +0.1        0.16 ± 42%  perf-profile.children.cycles-pp.__x64_sys_close
      0.00            +0.1        0.14 ± 51%  perf-profile.children.cycles-pp.btrfs_find_free_objectid
      0.00            +0.1        0.14 ± 39%  perf-profile.children.cycles-pp.complete_walk
      0.00            +0.2        0.16 ± 30%  perf-profile.children.cycles-pp.update_blocked_averages
      0.10 ± 80%      +0.2        0.26 ± 36%  perf-profile.children.cycles-pp.__pagevec_release
      0.00            +0.2        0.16 ± 53%  perf-profile.children.cycles-pp.rcu_eqs_enter
      0.00            +0.2        0.16 ± 54%  perf-profile.children.cycles-pp.btrfs_find_free_ino
      0.04 ±173%      +0.2        0.20 ± 42%  perf-profile.children.cycles-pp.finish_task_switch
      0.00            +0.2        0.17 ± 61%  perf-profile.children.cycles-pp.__mutex_lock
      0.00            +0.2        0.18 ± 25%  perf-profile.children.cycles-pp.inode_insert5
      0.02 ±173%      +0.2        0.20 ±  9%  perf-profile.children.cycles-pp.btrfs_update_root_times
      0.02 ±173%      +0.2        0.21 ± 38%  perf-profile.children.cycles-pp.update_ts_time_stats
      0.00            +0.2        0.19 ± 25%  perf-profile.children.cycles-pp.pm_qos_request
      0.02 ±173%      +0.2        0.21 ± 15%  perf-profile.children.cycles-pp.timerqueue_add
      0.00            +0.2        0.19 ± 28%  perf-profile.children.cycles-pp.insert_inode_locked4
      0.02 ±173%      +0.2        0.22 ± 45%  perf-profile.children.cycles-pp.interrupt_entry
      0.09 ±112%      +0.2        0.29 ±  9%  perf-profile.children.cycles-pp.__hrtimer_next_event_base
      0.02 ±173%      +0.2        0.24 ± 68%  perf-profile.children.cycles-pp.__push_leaf_left
      0.03 ±173%      +0.2        0.24 ± 27%  perf-profile.children.cycles-pp.tick_nohz_irq_exit
      0.02 ±173%      +0.2        0.25 ± 31%  perf-profile.children.cycles-pp.enqueue_hrtimer
      0.00            +0.2        0.23 ± 10%  perf-profile.children.cycles-pp.timerqueue_del
      0.00            +0.2        0.23 ± 43%  perf-profile.children.cycles-pp.rebalance_domains
      0.00            +0.2        0.24 ± 52%  perf-profile.children.cycles-pp.d_instantiate_new
      0.00            +0.3        0.25 ± 63%  perf-profile.children.cycles-pp.update_sd_lb_stats
      0.03 ±173%      +0.3        0.29 ± 45%  perf-profile.children.cycles-pp.rcu_sched_clock_irq
      0.00            +0.3        0.26 ± 60%  perf-profile.children.cycles-pp.find_busiest_group
      0.07 ±113%      +0.3        0.33 ± 33%  perf-profile.children.cycles-pp.__next_timer_interrupt
      0.00            +0.3        0.27 ± 20%  perf-profile.children.cycles-pp.__remove_hrtimer
      0.06 ±173%      +0.3        0.33 ± 29%  perf-profile.children.cycles-pp.update_rq_clock
      0.07 ±113%      +0.3        0.34 ± 43%  perf-profile.children.cycles-pp.__fput
      0.11 ±173%      +0.3        0.40 ± 66%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.10 ± 80%      +0.3        0.39 ± 29%  perf-profile.children.cycles-pp.push_leaf_left
      0.09 ± 64%      +0.3        0.40 ± 54%  perf-profile.children.cycles-pp.run_timer_softirq
      0.19 ± 79%      +0.3        0.52 ± 61%  perf-profile.children.cycles-pp.btrfs_leaf_free_space
      0.00            +0.3        0.34 ± 56%  perf-profile.children.cycles-pp.load_balance
      0.11 ± 83%      +0.3        0.45 ± 31%  perf-profile.children.cycles-pp.update_curr
      0.19 ± 86%      +0.4        0.56 ± 33%  perf-profile.children.cycles-pp.dequeue_entity
      0.11 ± 83%      +0.4        0.48 ± 65%  perf-profile.children.cycles-pp.btrfs_get_or_create_delayed_node
      0.03 ±173%      +0.4        0.40 ± 44%  perf-profile.children.cycles-pp.do_dentry_open
      0.05 ±100%      +0.4        0.43 ± 35%  perf-profile.children.cycles-pp.native_apic_msr_eoi_write
      0.07 ±113%      +0.4        0.45 ± 35%  perf-profile.children.cycles-pp.task_work_run
      0.02 ±173%      +0.4        0.41 ± 84%  perf-profile.children.cycles-pp.btrfs_delayed_inode_release_metadata
      0.26 ± 73%      +0.4        0.68 ± 23%  perf-profile.children.cycles-pp.dequeue_task_fair
      0.00            +0.4        0.42 ± 87%  perf-profile.children.cycles-pp.queued_write_lock_slowpath
      0.00            +0.4        0.42 ± 14%  perf-profile.children.cycles-pp.cpuidle_governor_latency_req
      0.05 ±100%      +0.4        0.49 ± 30%  perf-profile.children.cycles-pp.tick_irq_enter
      0.02 ±173%      +0.4        0.46 ± 57%  perf-profile.children.cycles-pp.__push_leaf_right
      0.11 ± 77%      +0.5        0.56 ± 53%  perf-profile.children.cycles-pp.swapgs_restore_regs_and_return_to_usermode
      0.20 ± 71%      +0.5        0.67 ± 25%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.38 ± 65%      +0.5        0.87 ± 13%  perf-profile.children.cycles-pp.push_leaf_right
      0.08 ±100%      +0.5        0.58 ± 29%  perf-profile.children.cycles-pp.get_next_timer_interrupt
      0.00            +0.5        0.51 ± 50%  perf-profile.children.cycles-pp.inode_tree_add
      0.00            +0.5        0.52 ± 81%  perf-profile.children.cycles-pp.alloc_inode
      0.17 ± 72%      +0.5        0.70 ± 46%  perf-profile.children.cycles-pp.exit_to_usermode_loop
      0.27 ± 71%      +0.5        0.81 ± 27%  perf-profile.children.cycles-pp.native_write_msr
      0.00            +0.6        0.56 ± 50%  perf-profile.children.cycles-pp.btrfs_insert_delayed_dir_index
      0.00            +0.6        0.58 ± 82%  perf-profile.children.cycles-pp.new_inode_pseudo
      0.09 ± 69%      +0.6        0.68 ± 37%  perf-profile.children.cycles-pp.lapic_next_deadline
      0.07 ± 59%      +0.6        0.67 ± 21%  perf-profile.children.cycles-pp.irq_enter
      0.24 ± 95%      +0.6        0.88 ± 61%  perf-profile.children.cycles-pp.btrfs_delayed_update_inode
      0.02 ±173%      +0.6        0.67 ± 43%  perf-profile.children.cycles-pp.ktime_get_update_offsets_now
      0.00            +0.7        0.66 ± 70%  perf-profile.children.cycles-pp.new_inode
      0.20 ± 60%      +0.7        0.88 ± 40%  perf-profile.children.cycles-pp.unlock_up
      0.00            +0.7        0.70 ± 42%  perf-profile.children.cycles-pp.prepare_to_wait_event
      0.10 ±127%      +0.7        0.81 ± 46%  perf-profile.children.cycles-pp.read_tsc
      0.11 ± 62%      +0.7        0.83 ± 27%  perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
      0.02 ±173%      +0.8        0.78 ± 17%  perf-profile.children.cycles-pp.task_tick_fair
      0.29 ±135%      +0.8        1.08 ± 79%  perf-profile.children.cycles-pp.__btrfs_release_delayed_node
      0.26 ± 89%      +0.8        1.06 ± 56%  perf-profile.children.cycles-pp.btrfs_update_inode
      0.12 ± 97%      +0.8        0.93 ± 56%  perf-profile.children.cycles-pp.wait_reserve_ticket
      0.13 ± 67%      +0.8        0.96 ± 28%  perf-profile.children.cycles-pp.tick_nohz_next_event
      0.13 ± 70%      +0.9        1.05 ± 33%  perf-profile.children.cycles-pp.clockevents_program_event
      0.16 ± 59%      +1.0        1.16 ± 24%  perf-profile.children.cycles-pp.ktime_get
      0.21 ± 83%      +1.0        1.22 ± 24%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.43 ± 97%      +1.1        1.50 ± 27%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.05 ±102%      +1.1        1.16 ± 11%  perf-profile.children.cycles-pp.schedule_idle
      0.24 ± 81%      +1.3        1.55 ± 18%  perf-profile.children.cycles-pp.autoremove_wake_function
      0.32 ± 89%      +1.3        1.64 ± 17%  perf-profile.children.cycles-pp.__wake_up_common
      0.09 ± 64%      +1.4        1.46 ± 25%  perf-profile.children.cycles-pp.split_leaf
      0.15 ±100%      +1.4        1.58 ± 22%  perf-profile.children.cycles-pp.scheduler_tick
      0.00            +1.6        1.57 ± 53%  perf-profile.children.cycles-pp.btrfs_lookup_dentry
      0.00            +1.6        1.58 ± 54%  perf-profile.children.cycles-pp.btrfs_lookup
      0.23 ±109%      +1.7        1.89 ± 47%  perf-profile.children.cycles-pp.btrfs_tree_read_lock
      0.00            +1.7        1.73 ± 43%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      0.37 ± 74%      +1.8        2.15 ± 49%  perf-profile.children.cycles-pp.btrfs_read_lock_root_node
      2.21 ± 71%      +1.9        4.13 ± 23%  perf-profile.children.cycles-pp._raw_spin_lock
      1.77 ± 58%      +2.0        3.76 ± 29%  perf-profile.children.cycles-pp.setup_items_for_insert
      0.47 ± 74%      +2.4        2.88 ± 21%  perf-profile.children.cycles-pp.menu_select
      1.29 ± 93%      +2.8        4.12 ± 29%  perf-profile.children.cycles-pp.update_process_times
      1.31 ± 91%      +3.0        4.29 ± 30%  perf-profile.children.cycles-pp.tick_sched_handle
      1.58 ± 84%      +3.3        4.92 ± 26%  perf-profile.children.cycles-pp.tick_sched_timer
      0.00            +4.3        4.33 ± 61%  perf-profile.children.cycles-pp.btrfs_new_inode
      1.81 ± 81%      +4.8        6.62 ± 23%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.00            +6.0        6.04 ± 14%  perf-profile.children.cycles-pp.insert_with_overflow
      0.00            +6.9        6.86 ± 18%  perf-profile.children.cycles-pp.btrfs_insert_dir_item
      2.08 ± 78%      +7.0        9.07 ± 25%  perf-profile.children.cycles-pp.hrtimer_interrupt
      0.00            +7.2        7.21 ± 18%  perf-profile.children.cycles-pp.btrfs_add_link
      1.62 ± 61%      +9.9       11.49 ± 20%  perf-profile.children.cycles-pp.btrfs_insert_empty_items
      2.70 ± 73%     +10.9       13.55 ± 14%  perf-profile.children.cycles-pp.smp_apic_timer_interrupt
      3.32 ± 70%     +11.0       14.31 ± 13%  perf-profile.children.cycles-pp.apic_timer_interrupt
      0.00           +15.6       15.63 ± 37%  perf-profile.children.cycles-pp.btrfs_create
      1.85 ± 60%     +16.9       18.78 ± 21%  perf-profile.children.cycles-pp.intel_idle
      0.31 ± 87%     +19.6       19.91 ± 39%  perf-profile.children.cycles-pp.path_openat
      0.31 ± 87%     +19.7       19.97 ± 39%  perf-profile.children.cycles-pp.do_filp_open
      0.31 ±101%     +20.0       20.34 ± 40%  perf-profile.children.cycles-pp.do_sys_open
      5.93 ± 68%     +22.9       28.78 ± 20%  perf-profile.children.cycles-pp.cpuidle_enter_state
      6.59 ± 65%     +27.7       34.27 ± 18%  perf-profile.children.cycles-pp.do_idle
      6.59 ± 65%     +27.7       34.28 ± 18%  perf-profile.children.cycles-pp.secondary_startup_64
      6.59 ± 65%     +27.7       34.28 ± 18%  perf-profile.children.cycles-pp.cpu_startup_entry
      0.00           +31.7       31.70 ± 19%  perf-profile.children.cycles-pp.start_secondary
      0.04 ±100%      +0.1        0.12 ± 21%  perf-profile.self.cycles-pp.sched_clock_idle_wakeup_event
      0.00            +0.1        0.08 ± 26%  perf-profile.self.cycles-pp.rcu_irq_enter
      0.00            +0.1        0.09 ± 32%  perf-profile.self.cycles-pp.resched_curr
      0.00            +0.1        0.09 ± 46%  perf-profile.self.cycles-pp.cpuidle_not_available
      0.02 ±173%      +0.1        0.12 ± 24%  perf-profile.self.cycles-pp.switch_mm_irqs_off
      0.00            +0.1        0.10 ± 40%  perf-profile.self.cycles-pp.hrtimer_forward
      0.00            +0.1        0.10 ± 36%  perf-profile.self.cycles-pp.find_inode
      0.04 ±100%      +0.1        0.15 ± 32%  perf-profile.self.cycles-pp.cpu_load_update
      0.04 ±100%      +0.1        0.15 ± 26%  perf-profile.self.cycles-pp.rb_insert_color
      0.00            +0.1        0.11 ± 28%  perf-profile.self.cycles-pp.enqueue_entity
      0.09 ± 64%      +0.1        0.20 ± 18%  perf-profile.self.cycles-pp.__softirqentry_text_start
      0.02 ±173%      +0.1        0.14 ± 33%  perf-profile.self.cycles-pp.timerqueue_add
      0.00            +0.1        0.12 ± 53%  perf-profile.self.cycles-pp.mutex_spin_on_owner
      0.02 ±173%      +0.1        0.14 ± 16%  perf-profile.self.cycles-pp.__next_timer_interrupt
      0.11 ± 62%      +0.1        0.23 ± 16%  perf-profile.self.cycles-pp.__switch_to_asm
      0.02 ±173%      +0.1        0.14 ± 47%  perf-profile.self.cycles-pp.irq_exit
      0.00            +0.1        0.13 ± 23%  perf-profile.self.cycles-pp.task_tick_fair
      0.00            +0.1        0.13 ± 54%  perf-profile.self.cycles-pp.clockevents_program_event
      0.02 ±173%      +0.1        0.15 ± 50%  perf-profile.self.cycles-pp.perf_event_task_tick
      0.02 ±173%      +0.1        0.15 ± 27%  perf-profile.self.cycles-pp.smp_apic_timer_interrupt
      0.02 ±173%      +0.1        0.15 ± 35%  perf-profile.self.cycles-pp.nr_iowait_cpu
      0.02 ±173%      +0.1        0.16 ± 22%  perf-profile.self.cycles-pp.cpuacct_charge
      0.00            +0.1        0.14 ± 15%  perf-profile.self.cycles-pp.get_cpu_device
      0.00            +0.1        0.15 ± 41%  perf-profile.self.cycles-pp.lapic_next_deadline
      0.00            +0.2        0.16 ± 53%  perf-profile.self.cycles-pp.rcu_eqs_enter
      0.05 ±102%      +0.2        0.21 ± 27%  perf-profile.self.cycles-pp.tick_nohz_next_event
      0.05 ±102%      +0.2        0.21 ± 56%  perf-profile.self.cycles-pp.run_timer_softirq
      0.00            +0.2        0.17 ± 36%  perf-profile.self.cycles-pp.finish_task_switch
      0.03 ±173%      +0.2        0.20 ± 42%  perf-profile.self.cycles-pp.rcu_sched_clock_irq
      0.00            +0.2        0.18 ± 34%  perf-profile.self.cycles-pp.tick_sched_timer
      0.09 ±124%      +0.2        0.26 ± 62%  perf-profile.self.cycles-pp.btrfs_tree_read_lock
      0.04 ±100%      +0.2        0.22 ± 32%  perf-profile.self.cycles-pp.perf_mux_hrtimer_handler
      0.00            +0.2        0.19 ± 25%  perf-profile.self.cycles-pp.pm_qos_request
      0.02 ±173%      +0.2        0.21 ± 41%  perf-profile.self.cycles-pp.try_to_wake_up
      0.02 ±173%      +0.2        0.22 ± 45%  perf-profile.self.cycles-pp.interrupt_entry
      0.07 ±103%      +0.2        0.27 ± 20%  perf-profile.self.cycles-pp.__hrtimer_next_event_base
      0.00            +0.2        0.20 ± 54%  perf-profile.self.cycles-pp.__hrtimer_run_queues
      0.11 ± 77%      +0.2        0.31 ± 19%  perf-profile.self.cycles-pp.swapgs_restore_regs_and_return_to_usermode
      0.00            +0.3        0.27 ± 32%  perf-profile.self.cycles-pp.update_rq_clock
      0.04 ±100%      +0.3        0.34 ± 39%  perf-profile.self.cycles-pp.do_idle
      0.28 ± 67%      +0.3        0.58 ± 15%  perf-profile.self.cycles-pp.__sched_text_start
      0.08 ±173%      +0.3        0.40 ± 66%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.04 ±173%      +0.4        0.42 ± 61%  perf-profile.self.cycles-pp.__btrfs_release_delayed_node
      0.05 ±100%      +0.4        0.43 ± 35%  perf-profile.self.cycles-pp.native_apic_msr_eoi_write
      0.10 ± 80%      +0.4        0.49 ± 25%  perf-profile.self.cycles-pp.ktime_get
      0.00            +0.4        0.39 ± 59%  perf-profile.self.cycles-pp.inode_tree_add
      0.20 ± 71%      +0.5        0.67 ± 25%  perf-profile.self.cycles-pp.native_irq_return_iret
      0.02 ±173%      +0.5        0.56 ± 41%  perf-profile.self.cycles-pp.ktime_get_update_offsets_now
      0.27 ± 71%      +0.5        0.81 ± 27%  perf-profile.self.cycles-pp.native_write_msr
      0.24 ± 63%      +0.6        0.84 ± 27%  perf-profile.self.cycles-pp.cpuidle_enter_state
      0.10 ±127%      +0.7        0.76 ± 46%  perf-profile.self.cycles-pp.read_tsc
      0.24 ± 70%      +0.9        1.16 ± 27%  perf-profile.self.cycles-pp.menu_select
      0.00            +1.7        1.72 ± 44%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      1.85 ± 60%     +16.9       18.76 ± 21%  perf-profile.self.cycles-pp.intel_idle


                                                                                
                      fxmark.hdd_btrfs_MWUM_1_bufferedio.works                  
                                                                                
  180000 +-+----------------------------------------------------------------+   
         O     O O  O     O  O O     O  O  O O  O  O  O O  O  O  O  O O  O  O   
  160000 +-+                                                                |   
  140000 +-+                                                                |   
         |                                                                  |   
  120000 +-+                                                                |   
  100000 +-+                                                                |   
         |                                                                  |   
   80000 +-+                                                                |   
   60000 +-+                                                                |   
         |..+..+.+..+..+..+    +     +     +    +..+..+.+..+..+..+..+.+     |   
   40000 +-+               :   ::   : :   : :  :                            |   
   20000 +-+                : :  :  :  :  : :  :                            |   
         |                  : :  : :   : :   ::                             |   
       0 +-+O----------O----------O-----------------------------------------+   
                                                                                
                                                                                                                                                                
                   fxmark.hdd_btrfs_MWUM_1_bufferedio.works_sec                 
                                                                                
  7000 O-+----------------------------------------------O-----O-------------+   
       |     O  O  O    O  O  O     O     O    O  O  O     O     O O  O  O  O   
  6000 +-+                             O    O                               |   
       |                                                                    |   
  5000 +-+                                                                  |   
       |                                                                    |   
  4000 +-+                                                                  |   
       |                                                                    |   
  3000 +-+                                                                  |   
       |                                                                    |   
  2000 +-++..+..+..+.+..+     +     +     +    +..+..+..+..+..+..+.+..+     |   
       |                 :   : :   : :   : :  :                             |   
  1000 +-+                :  :  :  :  :  : :  :                             |   
       |                  : :   : :   : :   ::                              |   
     0 +-+O----------O-----------O------------------------------------------+   
                                                                                
                                                                                                                                                                
                   fxmark.hdd_btrfs_MWUM_1_bufferedio.sys_sec                   
                                                                                
  8 +-+------------------------------O-----O--------------------------------+   
    O     O  O  O     O  O  O     O     O     O                          O  |   
  7 +-+                                          O  O  O  O  O  O  O  O     O   
  6 +-+                                                                     |   
    |                                                                       |   
  5 +-+                                                                     |   
    |                                                                       |   
  4 +-+                                                                     |   
    |                                                                       |   
  3 +-++..+..+..+..+..+     +     +     +     +..+..+..+..+..+..+..+..+     |   
  2 +-+                :   : :   : :   : :   :                              |   
    |                  :   : :   : :   : :   :                              |   
  1 +-+                 : :   : :   : :   : :                               |   
    |                   : :   : :   : :   : :                               |   
  0 +-+O-----------O-----------O--------------------------------------------+   
                                                                                
                                                                                                                                                                
                   fxmark.hdd_btrfs_MWUM_1_bufferedio.sys_util                  
                                                                                
  30 +-+--------------------------------------------------------------------+   
     O     O  O  O     O  O  O     O     O    O     O  O  O  O              |   
  25 +-+                              O    O     O              O  O  O  O  O   
     |                                                                      |   
     |                                                                      |   
  20 +-+                                                                    |   
     |                                                                      |   
  15 +-+                                                                    |   
     |    .+..+..+..+..+     +           +    +..+..  .+..  .+..+..         |   
  10 +-++.              :    ::    +     :    :     +.    +.       +..+     |   
     |                  :   : :   : :   : :  :                              |   
     |                   :  :  :  : :   : :  :                              |   
   5 +-+                 : :   : :   : :  : :                               |   
     |                    ::    ::   : :   ::                               |   
   0 +-+O-----------O-----------O-------------------------------------------+   
                                                                                
                                                                                                                                                                
                 fxmark.hdd_btrfs_MWUM_1_bufferedio.iowait_util                 
                                                                                
  90 +-+--------------------------------------------------------------------+   
     |     +..+..+..+..+     +     +     +    +..+.    +.    +..+.    +     |   
  80 +-+               :     :     :     :    :                             |   
  70 O-+   O  O  O     O  O  O     O  O  O O  O  O  O  O  O  O  O  O  O  O  O   
     |                  :   : :   : :   ::   :                              |   
  60 +-+                :   : :   : :   : :  :                              |   
  50 +-+                :   : :   : :   : :  :                              |   
     |                  :   : :   : :   : :  :                              |   
  40 +-+                 : :   : :   : :  : :                               |   
  30 +-+                 : :   : :   : :  : :                               |   
     |                   : :   : :   : :  : :                               |   
  20 +-+                 : :   : :   : :   ::                               |   
  10 +-+                  :     :     :    :                                |   
     |                    :     :     :    :                                |   
   0 +-+O-----------O-----------O-------------------------------------------+   
                                                                                
                                                                                                                                                                
                             fxmark.time.system_time                            
                                                                                
  20 +-+---------O-----------------O----------------------------------------+   
  18 O-+   O  O        O  O  O           O    O  O  O  O  O  O  O           |   
     |                                O    O                       O  O  O  O   
  16 +-+                                                                    |   
  14 +-+                                                                    |   
     |                                                                      |   
  12 +-+                                                                    |   
  10 +-+                                                                    |   
   8 +-+                                                                    |   
     |  +..+..+..  .+..+     +     +                  .+..+..               |   
   6 +-+         +.     :   : :   : :    +    +..+..+.       +..+..+..+     |   
   4 +-+                :   : :   : :   : :  :                              |   
     |                   : :   : :   :  : :  :                              |   
   2 +-+                 : :   : :   : :   ::                               |   
   0 +-+O-----------O-----------O-------------------------------------------+   
                                                                                
                                                                                                                                                                
                             fxmark.time.elapsed_time                           
                                                                                
  800 +-+-------------------------------------------------------------------+   
      |                            O     O     O                            |   
  700 O-+   O  O  O       O                          O  O  O                |   
  600 +-+               O    O        O     O     O          O  O     O  O  |   
      |                                                            O        O   
  500 +-+                                                                   |   
      |                                                                     |   
  400 +-+                                                                   |   
      |                                                                     |   
  300 +-+                                                                   |   
  200 +-+                                                                   |   
      |                                                                     |   
  100 +-++..+..+..+..+..+    +..   +..   +..   +..+..+..+..+.  .+..+..+     |   
      |.                 + ..    ..    ..    ..              +.             |   
    0 +-+O-----------O----------O-------------------------------------------+   
                                                                                
                                                                                                                                                                
                           fxmark.time.elapsed_time.max                         
                                                                                
  800 +-+-------------------------------------------------------------------+   
      |                            O     O     O                            |   
  700 O-+   O  O  O       O                          O  O  O                |   
  600 +-+               O    O        O     O     O          O  O     O  O  |   
      |                                                            O        O   
  500 +-+                                                                   |   
      |                                                                     |   
  400 +-+                                                                   |   
      |                                                                     |   
  300 +-+                                                                   |   
  200 +-+                                                                   |   
      |                                                                     |   
  100 +-++..+..+..+..+..+    +..   +..   +..   +..+..+..+..+.  .+..+..+     |   
      |.                 + ..    ..    ..    ..              +.             |   
    0 +-+O-----------O----------O-------------------------------------------+   
                                                                                
                                                                                                                                                                
                      fxmark.time.involuntary_context_switches                  
                                                                                
  600000 +-+----------------------------------------------------------------+   
         |                                                                  |   
  500000 O-+     O        O          O     O    O                        O  |   
         |     O    O        O O        O    O     O  O O  O  O  O  O O     O   
         |                                                                  |   
  400000 +-+                                                                |   
         |                                                                  |   
  300000 +-+                                                                |   
         |                                                                  |   
  200000 +-+                                                                |   
         |                                                                  |   
         |..+..+.+..+..+..+    +     +     +    +..+..+.+..+..+..+..+.+     |   
  100000 +-+               +  : +   + +   + :  +                            |   
         |                  + :  + +   + +  : +                             |   
       0 +-+O----------O----------O-----------------------------------------+   
                                                                                
                                                                                                                                                                
                           fxmark.time.file_system_outputs                      
                                                                                
  8e+06 +-+-----------------------------------------------------------------+   
        |          O                                                        |   
  7e+06 O-+                         O     O     O                           |   
  6e+06 +-+   O  O       O  O  O                   O O  O  O     O    O  O  |   
        |                              O     O                O     O       O   
  5e+06 +-+                                                                 |   
        |                                                                   |   
  4e+06 +-+                                                                 |   
        |                                                                   |   
  3e+06 +-+                                                                 |   
  2e+06 +-+                                                                 |   
        |                                                                   |   
  1e+06 +-++..+..+.+..+..+..   +..  +..   +..   +..+.+..+..+..+..+..+.+     |   
        |                    ..    +    ..    ..                            |   
      0 +-+O----------O-----------O-----------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample

***************************************************************************************************
ivb44: 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 64G memory
=========================================================================================
compiler/cpufreq_governor/directio/disk/fstype/kconfig/media/rootfs/tbox_group/test/testcase/ucode:
  gcc-7/performance/directio/1HDD/btrfs/x86_64-rhel-7.6/hdd/debian-x86_64-2018-04-03.cgz/ivb44/MWUM/fxmark/0x42d

commit: 
  4438cf50e7 ("doc, block, bfq: add information on bfq execution time")
  56a85fd837 ("loop: properly observe rotational flag of underlying device")

4438cf50e7b315ff 56a85fd8376ef32458efb6ea97a 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
      3371 ±165%     -96.6%     113.96 ± 20%  fxmark.hdd_btrfs_MWUM_12_directio.works/sec
     20.68           -12.1%      18.18 ±  2%  fxmark.hdd_btrfs_MWUM_1_directio.iowait_sec
     87.19           -18.4%      71.15        fxmark.hdd_btrfs_MWUM_1_directio.iowait_util
      0.10 ± 14%     +33.3%       0.14 ±  5%  fxmark.hdd_btrfs_MWUM_1_directio.softirq_sec
      2.71 ±  3%    +154.4%       6.90        fxmark.hdd_btrfs_MWUM_1_directio.sys_sec
     11.43 ±  2%    +136.3%      27.02        fxmark.hdd_btrfs_MWUM_1_directio.sys_util
      0.22 ± 10%     +46.8%       0.33 ±  3%  fxmark.hdd_btrfs_MWUM_1_directio.user_sec
      0.94 ±  9%     +36.5%       1.28 ±  4%  fxmark.hdd_btrfs_MWUM_1_directio.user_util
     47985          +259.9%     172720        fxmark.hdd_btrfs_MWUM_1_directio.works
      1999          +236.5%       6728 ±  2%  fxmark.hdd_btrfs_MWUM_1_directio.works/sec
      1.94 ± 10%     -32.8%       1.30        fxmark.hdd_btrfs_MWUM_2_directio.sys_sec
      3.28 ±  9%     -32.8%       2.20        fxmark.hdd_btrfs_MWUM_2_directio.sys_util
      8087 ± 25%     -69.5%       2469 ±  6%  fxmark.hdd_btrfs_MWUM_2_directio.works
    269.28 ± 25%     -69.6%      81.99 ±  6%  fxmark.hdd_btrfs_MWUM_2_directio.works/sec
     22.08 ± 89%    +142.2%      53.47        fxmark.hdd_btrfs_MWUM_4_directio.iowait_sec
     12.83 ± 88%    +134.5%      30.10        fxmark.hdd_btrfs_MWUM_4_directio.real_sec
      3.21 ± 85%     -58.4%       1.34 ±  3%  fxmark.hdd_btrfs_MWUM_4_directio.sys_util
      8.24 ±166%     -97.5%       0.21        fxmark.hdd_btrfs_MWUM_4_directio.user_util
      3255 ±167%     -97.9%      69.79 ±  2%  fxmark.hdd_btrfs_MWUM_4_directio.works/sec
    103.91 ± 14%    +459.5%     581.40 ±  2%  fxmark.time.elapsed_time
    103.91 ± 14%    +459.5%     581.40 ±  2%  fxmark.time.elapsed_time.max
   1090720 ±  5%    +467.3%    6187885        fxmark.time.file_system_outputs
    115485 ±  5%    +314.7%     478935 ±  2%  fxmark.time.involuntary_context_switches
     13.50 ± 28%     +48.1%      20.00 ± 10%  fxmark.time.percent_of_cpu_this_job_got
      6.04          +184.6%      17.20        fxmark.time.system_time
      8.28 ± 54%   +1141.9%     102.86 ± 15%  fxmark.time.user_time
     47465 ±  9%    +246.3%     164378 ±  4%  fxmark.time.voluntary_context_switches
     27.53 ± 13%      -7.7       19.85 ±  5%  mpstat.cpu.all.idle%
      7.44 ± 11%      -2.1        5.37        mpstat.cpu.all.sys%
      9.91 ± 23%     +11.0       20.93 ±  4%  mpstat.cpu.all.usr%
    520961 ± 14%    +241.0%    1776251        numa-numastat.node0.local_node
    532808 ± 14%    +234.4%    1781585 ±  2%  numa-numastat.node0.numa_hit
     97321 ± 37%     +57.3%     153082 ± 17%  numa-numastat.node1.numa_hit
     23.75 ± 13%     -20.0%      19.00 ±  4%  vmstat.cpu.id
      9.75 ± 22%    +108.5%      20.33 ±  4%  vmstat.cpu.us
    171.25 ± 29%     -75.5%      42.00 ± 12%  vmstat.io.bi
     35370 ±  5%      -9.1%      32142 ±  2%  vmstat.io.bo
   1323519           +25.1%    1655202        vmstat.memory.cache
     10548 ±  9%     -12.0%       9282 ±  2%  vmstat.system.cs
     27.25 ± 13%     -27.2%      19.84 ±  5%  iostat.cpu.idle
      6.96 ± 11%     -21.3%       5.48        iostat.cpu.system
      9.53 ± 23%    +118.5%      20.83 ±  4%  iostat.cpu.user
      1.40 ±  6%     -10.4%       1.25        iostat.sda.avgqu-sz
    341.40 ±  6%     -22.7%     263.81        iostat.sda.avgrq-sz
      9.62 ±  2%     +14.3%      11.00        iostat.sda.util
    118.55 ±  3%     +27.3%     150.89        iostat.sda.w/s
     19735 ±  4%      -8.0%      18156        iostat.sda.wkB/s
     85.26 ±  2%     +16.7%      99.46        iostat.sda.wrqm/s
  21547184 ± 47%    +444.1%  1.172e+08 ± 67%  cpuidle.C1.time
    315822 ± 33%    +399.1%    1576235 ± 36%  cpuidle.C1.usage
 1.001e+08 ± 37%    +512.4%  6.129e+08 ±  2%  cpuidle.C1E.time
    265278 ± 39%    +476.8%    1530170        cpuidle.C1E.usage
  38041474 ± 54%    +344.6%  1.691e+08 ± 47%  cpuidle.C3.time
    108012 ± 48%    +367.2%     504612 ± 44%  cpuidle.C3.usage
   4.1e+08 ± 19%    +146.6%  1.011e+09 ±  8%  cpuidle.C6.time
    410345 ± 30%    +291.0%    1604416 ± 19%  cpuidle.C6.usage
     44723 ± 26%    +400.1%     223649 ±  3%  cpuidle.POLL.time
     58595 ± 18%    +525.8%     366715 ±  4%  cpuidle.POLL.usage
    320784           +44.9%     464888        meminfo.Active
     70618 ±  4%    +200.6%     212264        meminfo.Active(file)
    102470 ±  8%     +92.9%     197646        meminfo.AnonHugePages
   1232869           +14.8%    1415574        meminfo.Cached
    778.75 ±  9%     +41.3%       1100 ±  2%  meminfo.Dirty
     54344           +77.7%      96586        meminfo.Inactive
     16242           +34.6%      21862 ±  7%  meminfo.Inactive(anon)
     38101 ±  2%     +96.1%      74722        meminfo.Inactive(file)
     91380          +160.8%     238298        meminfo.KReclaimable
   1937696           +16.2%    2252355        meminfo.Memused
    234.00 ±173%    +457.7%       1305 ± 33%  meminfo.Mlocked
     91380          +160.8%     238298        meminfo.SReclaimable
     89314           +45.1%     129592        meminfo.SUnreclaim
     16473           +38.9%      22883 ±  6%  meminfo.Shmem
    180695          +103.6%     367892        meminfo.Slab
      1150 ± 17%     -56.9%     496.00 ±  6%  meminfo.Writeback
     19295 ± 14%     -77.6%       4328 ±  2%  meminfo.max_used_kB
    175441 ± 29%     +68.6%     295809 ± 10%  numa-meminfo.node0.Active
     69633 ±  5%    +195.6%     205857        numa-meminfo.node0.Active(file)
    819.25 ±  7%     +27.7%       1046 ±  4%  numa-meminfo.node0.Dirty
    679771 ±  3%     +26.1%     857241 ±  3%  numa-meminfo.node0.FilePages
     49955 ±  5%     +76.5%      88160 ±  4%  numa-meminfo.node0.Inactive
     35584 ±  7%    +106.7%      73557        numa-meminfo.node0.Inactive(file)
     67037 ±  2%    +207.5%     206156 ±  2%  numa-meminfo.node0.KReclaimable
   1059620 ±  4%     +29.2%    1368715 ±  4%  numa-meminfo.node0.MemUsed
    135.25 ±173%    +460.7%     758.33 ± 33%  numa-meminfo.node0.Mlocked
     67037 ±  2%    +207.5%     206156 ±  2%  numa-meminfo.node0.SReclaimable
     55564 ±  3%     +68.0%      93333 ±  2%  numa-meminfo.node0.SUnreclaim
    122602 ±  2%    +144.3%     299490 ±  2%  numa-meminfo.node0.Slab
    985.00 ± 84%    +588.7%       6783 ± 35%  numa-meminfo.node1.Active(file)
     52798 ± 63%    +158.8%     136624 ± 17%  numa-meminfo.node1.AnonHugePages
      1871 ± 15%    +288.0%       7261 ± 69%  numa-meminfo.node1.Inactive(anon)
     24434           +31.6%      32159 ±  9%  numa-meminfo.node1.KReclaimable
     24434           +31.6%      32159 ±  9%  numa-meminfo.node1.SReclaimable
      1885 ± 15%    +286.4%       7286 ± 69%  numa-meminfo.node1.Shmem
     58206 ±  3%     +17.6%      68427 ±  6%  numa-meminfo.node1.Slab
     17437 ±  5%    +195.0%      51435        numa-vmstat.node0.nr_active_file
    309678 ± 17%    +598.5%    2163146 ±  5%  numa-vmstat.node0.nr_dirtied
    203.75 ± 12%     +34.8%     274.67 ±  2%  numa-vmstat.node0.nr_dirty
    169986 ±  3%     +26.1%     214269 ±  3%  numa-vmstat.node0.nr_file_pages
      8910 ±  7%    +106.2%      18377        numa-vmstat.node0.nr_inactive_file
     33.25 ±173%    +469.4%     189.33 ± 33%  numa-vmstat.node0.nr_mlock
     16776 ±  2%    +207.1%      51520 ±  2%  numa-vmstat.node0.nr_slab_reclaimable
     13892 ±  3%     +67.9%      23328 ±  2%  numa-vmstat.node0.nr_slab_unreclaimable
    308469 ± 17%    +600.1%    2159645 ±  5%  numa-vmstat.node0.nr_written
     17437 ±  5%    +195.0%      51435        numa-vmstat.node0.nr_zone_active_file
      8910 ±  7%    +106.2%      18377        numa-vmstat.node0.nr_zone_inactive_file
    736094 ±  8%     +83.4%    1350345 ±  7%  numa-vmstat.node0.numa_hit
    724374 ±  8%     +85.7%    1345015 ±  7%  numa-vmstat.node0.numa_local
    245.25 ± 85%    +591.4%       1695 ± 35%  numa-vmstat.node1.nr_active_file
     31300 ± 69%    +728.5%     259314 ± 51%  numa-vmstat.node1.nr_dirtied
    467.75 ± 15%    +288.0%       1815 ± 69%  numa-vmstat.node1.nr_inactive_anon
     24.25 ±173%    +460.8%     136.00 ± 33%  numa-vmstat.node1.nr_mlock
    471.00 ± 15%    +286.6%       1821 ± 69%  numa-vmstat.node1.nr_shmem
      6109           +31.6%       8039 ±  9%  numa-vmstat.node1.nr_slab_reclaimable
     31188 ± 69%    +730.6%     259044 ± 51%  numa-vmstat.node1.nr_written
    245.25 ± 85%    +591.4%       1695 ± 35%  numa-vmstat.node1.nr_zone_active_file
    467.75 ± 15%    +288.0%       1815 ± 69%  numa-vmstat.node1.nr_zone_inactive_anon
     17678 ±  4%    +200.4%      53103        proc-vmstat.nr_active_file
    695682 ±  9%    +529.4%    4378932        proc-vmstat.nr_dirtied
    209.00 ±  8%     +30.9%     273.67 ±  5%  proc-vmstat.nr_dirty
    308255           +14.8%     353955        proc-vmstat.nr_file_pages
      4060           +34.6%       5466 ±  7%  proc-vmstat.nr_inactive_anon
      9528 ±  2%     +96.2%      18697        proc-vmstat.nr_inactive_file
     11934            -1.5%      11752        proc-vmstat.nr_kernel_stack
      7469            -1.1%       7386        proc-vmstat.nr_mapped
     58.75 ±173%    +454.3%     325.67 ± 33%  proc-vmstat.nr_mlock
      1110            -2.5%       1082        proc-vmstat.nr_page_table_pages
      4116           +39.0%       5721 ±  6%  proc-vmstat.nr_shmem
     22890          +160.3%      59580        proc-vmstat.nr_slab_reclaimable
     22321           +45.1%      32396        proc-vmstat.nr_slab_unreclaimable
    309.50 ± 10%     -64.1%     111.00 ±  8%  proc-vmstat.nr_writeback
    689573 ±  9%    +531.7%    4356002        proc-vmstat.nr_written
     17678 ±  4%    +200.4%      53103        proc-vmstat.nr_zone_active_file
      4060           +34.6%       5466 ±  7%  proc-vmstat.nr_zone_inactive_anon
      9528 ±  2%     +96.2%      18697        proc-vmstat.nr_zone_inactive_file
    510.50 ±  6%     -25.6%     380.00 ±  3%  proc-vmstat.nr_zone_write_pending
      1222 ± 53%     -97.5%      30.67 ±105%  proc-vmstat.numa_hint_faults_local
    658998 ±  6%    +197.8%    1962342        proc-vmstat.numa_hit
    643072 ±  6%    +202.7%    1946433        proc-vmstat.numa_local
     12977 ± 40%     -54.8%       5864 ± 69%  proc-vmstat.numa_pages_migrated
     81778 ±  8%    +210.7%     254110        proc-vmstat.pgactivate
    818853 ±  6%    +197.9%    2439647        proc-vmstat.pgalloc_normal
      1848 ± 12%    +115.4%       3983 ±  3%  proc-vmstat.pgdeactivate
    429974 ±  9%    +246.4%    1489231 ±  2%  proc-vmstat.pgfault
    798429 ±  7%    +204.6%    2431959        proc-vmstat.pgfree
     12977 ± 40%     -54.8%       5864 ± 69%  proc-vmstat.pgmigrate_success
     19082 ± 12%     +31.3%      25060 ±  8%  proc-vmstat.pgpgin
   3756945 ± 10%    +397.8%   18702952        proc-vmstat.pgpgout
    471743          +270.9%    1749602        proc-vmstat.slabs_scanned
     12230 ± 12%    +513.7%      75057 ±  8%  softirqs.BLOCK
      3697 ±  3%    +588.2%      25444 ±  2%  softirqs.CPU0.BLOCK
     27049 ±  6%    +322.0%     114156 ±  2%  softirqs.CPU0.RCU
     13868 ±  8%    +156.3%      35549 ±  3%  softirqs.CPU0.SCHED
      2637          +661.4%      20081        softirqs.CPU0.TASKLET
     57544 ± 13%    +406.7%     291604 ±  2%  softirqs.CPU0.TIMER
      6178 ± 22%    +491.3%      36529 ±  4%  softirqs.CPU1.BLOCK
     13555 ± 16%    +363.5%      62832 ±  2%  softirqs.CPU1.RCU
      8817 ±  9%    +208.4%      27190 ± 11%  softirqs.CPU1.SCHED
      6914 ± 22%    +518.8%      42781 ±  3%  softirqs.CPU1.TASKLET
     31444 ± 27%    +373.4%     148863 ± 14%  softirqs.CPU1.TIMER
      5257 ± 16%    +221.2%      16886 ±  8%  softirqs.CPU16.RCU
      8750 ± 14%    +290.3%      34157 ± 15%  softirqs.CPU16.TIMER
      3820 ± 24%    +326.3%      16287 ±  5%  softirqs.CPU17.RCU
      9592 ± 29%    +253.1%      33867 ± 16%  softirqs.CPU17.TIMER
      4403 ± 30%    +260.0%      15851 ±  5%  softirqs.CPU18.RCU
      9074 ± 13%    +275.4%      34066 ± 16%  softirqs.CPU18.TIMER
      4065 ± 11%    +286.3%      15702 ± 12%  softirqs.CPU19.RCU
      8643 ± 11%    +288.6%      33591 ± 17%  softirqs.CPU19.TIMER
      8226 ± 21%    +445.4%      44863 ±  6%  softirqs.CPU2.RCU
      3151 ± 25%    +237.0%      10622 ±  7%  softirqs.CPU2.SCHED
     17684 ± 41%    +455.2%      98184 ± 18%  softirqs.CPU2.TIMER
      7360 ±  8%    +156.3%      18864 ±  7%  softirqs.CPU20.TIMER
      2335 ± 18%    +283.0%       8943 ± 20%  softirqs.CPU21.RCU
      7855 ± 15%    +178.4%      21867 ± 18%  softirqs.CPU21.TIMER
      2587 ± 24%    +276.7%       9747 ± 20%  softirqs.CPU22.RCU
      7339 ± 14%    +236.0%      24664 ± 35%  softirqs.CPU22.TIMER
      2880 ±  8%    +245.6%       9954 ± 32%  softirqs.CPU23.RCU
      7477 ± 11%    +161.4%      19549 ±  6%  softirqs.CPU23.TIMER
      8954 ± 20%    +370.9%      42162 ±  5%  softirqs.CPU3.RCU
      3510 ± 24%    +250.9%      12316 ±  7%  softirqs.CPU3.SCHED
     18856 ± 40%    +432.9%     100493 ± 12%  softirqs.CPU3.TIMER
      5110 ±  6%    +416.5%      26393 ±  7%  softirqs.CPU4.RCU
     12345 ± 20%    +391.1%      60631 ± 14%  softirqs.CPU4.TIMER
      5957 ± 23%    +325.4%      25343 ±  5%  softirqs.CPU5.RCU
     11487 ± 13%    +443.1%      62393 ±  8%  softirqs.CPU5.TIMER
      4832 ± 15%    +455.2%      26831 ± 12%  softirqs.CPU6.RCU
     11332 ± 16%    +454.5%      62839 ± 18%  softirqs.CPU6.TIMER
      5257 ± 17%    +407.4%      26674 ±  3%  softirqs.CPU7.RCU
     12022 ± 17%    +438.2%      64705 ± 17%  softirqs.CPU7.TIMER
    165149          +217.1%     523772 ±  3%  softirqs.RCU
    101949 ± 14%     +98.9%     202828        softirqs.SCHED
     10281 ± 14%    +518.5%      63586 ±  2%  softirqs.TASKLET
    429927 ± 13%    +208.1%    1324678 ±  9%  softirqs.TIMER
     78066          +140.2%     187478        slabinfo.Acpi-Parse.active_objs
      1093          +135.6%       2576        slabinfo.Acpi-Parse.active_slabs
     79862          +135.5%     188106        slabinfo.Acpi-Parse.num_objs
      1093          +135.6%       2576        slabinfo.Acpi-Parse.num_slabs
      1126 ±  7%    +160.5%       2934        slabinfo.Acpi-State.active_objs
      1440 ±  5%    +128.4%       3290        slabinfo.Acpi-State.num_objs
    230.25 ± 10%    +118.0%     502.00 ±  6%  slabinfo.blkdev_ioc.active_objs
    535.50 ±  6%     +48.3%     794.33 ±  2%  slabinfo.blkdev_ioc.num_objs
     35428 ±  2%    +308.2%     144624        slabinfo.btrfs_delayed_node.active_objs
    688.00 ±  2%    +304.9%       2785        slabinfo.btrfs_delayed_node.active_slabs
     35813 ±  2%    +304.5%     144873        slabinfo.btrfs_delayed_node.num_objs
    688.00 ±  2%    +304.9%       2785        slabinfo.btrfs_delayed_node.num_slabs
      1354 ±  4%    +219.1%       4320        slabinfo.btrfs_extent_buffer.active_objs
      1659 ±  3%    +178.4%       4618        slabinfo.btrfs_extent_buffer.num_objs
      5328 ±  3%    +121.1%      11783 ±  2%  slabinfo.btrfs_extent_map.active_objs
    104.00 ±  3%    +174.4%     285.33        slabinfo.btrfs_extent_map.active_slabs
      5851 ±  3%    +173.6%      16010        slabinfo.btrfs_extent_map.num_objs
    104.00 ±  3%    +174.4%     285.33        slabinfo.btrfs_extent_map.num_slabs
     35092 ±  2%    +312.1%     144613        slabinfo.btrfs_inode.active_objs
      1261 ±  2%    +309.7%       5169        slabinfo.btrfs_inode.active_slabs
     35342 ±  2%    +309.6%     144765        slabinfo.btrfs_inode.num_objs
      1261 ±  2%    +309.7%       5169        slabinfo.btrfs_inode.num_slabs
    361.75 ± 17%    +144.9%     886.00 ±  5%  slabinfo.btrfs_ordered_extent.active_objs
    514.25 ±  7%    +107.9%       1069 ±  8%  slabinfo.btrfs_ordered_extent.num_objs
      1376 ±  7%    +142.0%       3331        slabinfo.btrfs_path.active_objs
      1577 ±  6%    +123.4%       3522        slabinfo.btrfs_path.num_objs
     80653          +134.7%     189273        slabinfo.dentry.active_objs
      1945          +131.9%       4513        slabinfo.dentry.active_slabs
     81731          +131.9%     189574        slabinfo.dentry.num_objs
      1945          +131.9%       4513        slabinfo.dentry.num_slabs
      1886           +88.2%       3550 ± 14%  slabinfo.kmalloc-128.active_objs
      4081           +21.9%       4975 ±  7%  slabinfo.kmalloc-128.num_objs
      1517 ±  5%     -17.8%       1247 ±  2%  slabinfo.kmalloc-192.active_objs
      8374 ±  3%      -9.6%       7568        slabinfo.kmalloc-8.active_objs
    287.25 ±  2%     -42.8%     164.33        slabinfo.kmalloc-8k.active_objs
    331.50 ±  2%     -39.5%     200.67        slabinfo.kmalloc-8k.num_objs
      3826           -11.5%       3385        slabinfo.lsm_file_cache.active_objs
    638.50 ±  4%     -19.8%     512.33 ±  2%  slabinfo.pde_opener.active_objs
      1000 ±  3%     -10.4%     896.33        slabinfo.pid.active_objs
      1720 ±  5%     +11.4%       1916 ±  6%  slabinfo.pool_workqueue.num_objs
     13389           +19.4%      15989        slabinfo.radix_tree_node.active_objs
     14706           +12.0%      16473        slabinfo.radix_tree_node.num_objs
      5262 ±  3%     -11.7%       4649        slabinfo.vm_area_struct.active_objs
      6173 ±  3%      -9.2%       5606 ±  2%  slabinfo.vm_area_struct.num_objs
    605.00 ±  6%     +67.7%       1014 ±  5%  slabinfo.xfrm_dst_cache.active_objs
    953.75 ±  4%     +41.4%       1348 ±  4%  slabinfo.xfrm_dst_cache.num_objs
      0.20 ± 81%     -88.1%       0.02 ± 17%  perf-stat.i.MPKI
  28930661 ± 28%     -83.6%    4746705 ± 13%  perf-stat.i.branch-instructions
      0.15 ± 15%      -0.1        0.03        perf-stat.i.branch-miss-rate%
   2105071 ± 27%     -83.3%     352162 ± 12%  perf-stat.i.branch-misses
      0.64 ± 16%      -0.5        0.11 ±  5%  perf-stat.i.cache-miss-rate%
    248420 ± 14%     -81.5%      45906 ±  8%  perf-stat.i.cache-misses
    746487 ±  9%     -80.3%     146828 ±  8%  perf-stat.i.cache-references
     10753 ±  9%     -13.8%       9266 ±  2%  perf-stat.i.context-switches
      0.04 ± 56%     -86.1%       0.00 ±  6%  perf-stat.i.cpi
 1.745e+08 ± 16%     -82.0%   31326038 ±  8%  perf-stat.i.cpu-cycles
    248.01 ±  3%     -10.1%     222.96 ±  6%  perf-stat.i.cpu-migrations
     14.01 ± 28%     -82.8%       2.41 ± 13%  perf-stat.i.cycles-between-cache-misses
      0.01 ± 67%      -0.0        0.00 ±  3%  perf-stat.i.dTLB-load-miss-rate%
     63485 ± 19%     -82.7%      11007 ± 10%  perf-stat.i.dTLB-load-misses
  27498834 ± 25%     -83.7%    4474433 ± 13%  perf-stat.i.dTLB-loads
      0.00 ± 31%      -0.0        0.00 ±  9%  perf-stat.i.dTLB-store-miss-rate%
      9473 ±  5%     -79.2%       1972 ±  9%  perf-stat.i.dTLB-store-misses
  14624912 ± 16%     -82.5%    2559744 ±  6%  perf-stat.i.dTLB-stores
      1.66 ± 15%      -1.4        0.29        perf-stat.i.iTLB-load-miss-rate%
     16736 ± 21%     -82.8%       2885 ± 10%  perf-stat.i.iTLB-load-misses
      3205 ± 16%     -81.6%     591.28 ± 16%  perf-stat.i.iTLB-loads
 1.378e+08 ± 27%     -83.6%   22584373 ± 13%  perf-stat.i.instructions
    157.98 ± 38%     -82.7%      27.37 ±  5%  perf-stat.i.instructions-per-iTLB-miss
      0.01 ± 21%     -82.3%       0.00 ± 10%  perf-stat.i.ipc
      3728 ±  5%     -34.3%       2450        perf-stat.i.minor-faults
     10359 ±  6%      -7.1%       9624 ±  3%  perf-stat.i.msec
      0.45 ± 45%      -0.4        0.08 ±  3%  perf-stat.i.node-load-miss-rate%
     10995 ± 44%     -84.7%       1679 ±  4%  perf-stat.i.node-load-misses
     48917 ± 17%     -84.8%       7451 ±  7%  perf-stat.i.node-loads
      0.31 ± 41%      -0.2        0.07 ±  8%  perf-stat.i.node-store-miss-rate%
      8224 ± 55%     -79.8%       1661 ± 49%  perf-stat.i.node-store-misses
     57973 ± 31%     -86.0%       8139 ± 33%  perf-stat.i.node-stores
      3728 ±  5%     -34.3%       2450        perf-stat.i.page-faults
  28746595 ± 28%     -83.5%    4756457 ± 13%  perf-stat.ps.branch-instructions
   2092001 ± 27%     -83.1%     352907 ± 12%  perf-stat.ps.branch-misses
    247060 ± 13%     -81.4%      45998 ±  8%  perf-stat.ps.cache-misses
    742450 ±  9%     -80.2%     147137 ±  8%  perf-stat.ps.cache-references
     10647 ±  9%     -13.1%       9251 ±  2%  perf-stat.ps.context-switches
 1.735e+08 ± 15%     -81.9%   31393197 ±  8%  perf-stat.ps.cpu-cycles
    245.60 ±  2%      -9.4%     222.60 ±  6%  perf-stat.ps.cpu-migrations
     63084 ± 18%     -82.5%      11028 ± 10%  perf-stat.ps.dTLB-load-misses
  27323655 ± 25%     -83.6%    4483293 ± 13%  perf-stat.ps.dTLB-loads
      9423 ±  5%     -79.0%       1976 ±  9%  perf-stat.ps.dTLB-store-misses
  14535653 ± 15%     -82.4%    2564990 ±  6%  perf-stat.ps.dTLB-stores
     16632 ± 21%     -82.6%       2891 ± 10%  perf-stat.ps.iTLB-load-misses
      3184 ± 15%     -81.4%     592.02 ± 16%  perf-stat.ps.iTLB-loads
 1.369e+08 ± 27%     -83.5%   22630330 ± 13%  perf-stat.ps.instructions
      3694 ±  5%     -33.8%       2446        perf-stat.ps.minor-faults
     10277 ±  6%      -6.5%       9611 ±  3%  perf-stat.ps.msec
     10916 ± 44%     -84.6%       1680 ±  4%  perf-stat.ps.node-load-misses
     48575 ± 17%     -84.7%       7453 ±  7%  perf-stat.ps.node-loads
      8173 ± 55%     -79.6%       1664 ± 49%  perf-stat.ps.node-store-misses
     57618 ± 31%     -85.9%       8147 ± 33%  perf-stat.ps.node-stores
      3694 ±  5%     -33.8%       2446        perf-stat.ps.page-faults
    172.25 ± 91%    +218.3%     548.33 ± 24%  interrupts.35:IR-PCI-MSI.2621441-edge.eth0-TxRx-0
    102.25 ± 38%    +351.5%     461.67 ± 29%  interrupts.36:IR-PCI-MSI.2621442-edge.eth0-TxRx-1
     57.50 ± 10%    +427.0%     303.00 ±  2%  interrupts.37:IR-PCI-MSI.2621443-edge.eth0-TxRx-2
    202.50 ±107%    +199.6%     606.67        interrupts.40:IR-PCI-MSI.2621446-edge.eth0-TxRx-5
     71.00 ± 42%    +355.9%     323.67 ±  7%  interrupts.41:IR-PCI-MSI.2621447-edge.eth0-TxRx-6
     75.50 ± 28%    +515.0%     464.33 ± 47%  interrupts.42:IR-PCI-MSI.2621448-edge.eth0-TxRx-7
     94.00 ± 35%    +355.3%     428.00 ± 24%  interrupts.CPU0.36:IR-PCI-MSI.2621442-edge.eth0-TxRx-1
    111.75 ±103%    +204.5%     340.33 ±  2%  interrupts.CPU0.40:IR-PCI-MSI.2621446-edge.eth0-TxRx-5
     20.75 ± 35%    +377.1%      99.00 ±  2%  interrupts.CPU0.41:IR-PCI-MSI.2621447-edge.eth0-TxRx-6
     21.50 ± 20%    +583.7%     147.00 ± 52%  interrupts.CPU0.42:IR-PCI-MSI.2621448-edge.eth0-TxRx-7
      2810 ±  9%     +83.1%       5146 ±  9%  interrupts.CPU0.CAL:Function_call_interrupts
    189007 ± 27%    +518.5%    1169077 ±  2%  interrupts.CPU0.LOC:Local_timer_interrupts
    680.75 ±128%    +545.9%       4397 ± 56%  interrupts.CPU0.NMI:Non-maskable_interrupts
    680.75 ±128%    +545.9%       4397 ± 56%  interrupts.CPU0.PMI:Performance_monitoring_interrupts
     10187 ± 10%    +260.7%      36743 ±  3%  interrupts.CPU0.RES:Rescheduling_interrupts
    123.25 ± 38%    +234.3%     412.00 ± 22%  interrupts.CPU0.TLB:TLB_shootdowns
     51.50 ± 35%    +348.5%     231.00 ± 26%  interrupts.CPU1.35:IR-PCI-MSI.2621441-edge.eth0-TxRx-0
     38.75 ± 15%    +434.2%     207.00 ±  4%  interrupts.CPU1.37:IR-PCI-MSI.2621443-edge.eth0-TxRx-2
     37.00 ± 16%    +247.7%     128.67 ±  6%  interrupts.CPU1.39:IR-PCI-MSI.2621445-edge.eth0-TxRx-4
     42.25 ± 31%    +427.8%     223.00 ±  9%  interrupts.CPU1.41:IR-PCI-MSI.2621447-edge.eth0-TxRx-6
      2948 ±  8%     +73.3%       5110 ±  5%  interrupts.CPU1.CAL:Function_call_interrupts
    136336 ± 27%    +490.1%     804511 ±  3%  interrupts.CPU1.LOC:Local_timer_interrupts
     10477 ±  8%    +201.5%      31595 ±  4%  interrupts.CPU1.RES:Rescheduling_interrupts
    131.25 ± 13%     +49.6%     196.33 ±  9%  interrupts.CPU1.TLB:TLB_shootdowns
      2226 ± 15%    +110.4%       4685 ± 27%  interrupts.CPU16.CAL:Function_call_interrupts
     27285 ± 20%    +580.5%     185667 ± 12%  interrupts.CPU16.LOC:Local_timer_interrupts
     27339 ± 19%    +581.6%     186357 ± 12%  interrupts.CPU17.LOC:Local_timer_interrupts
      2180 ± 14%    +146.0%       5365 ± 25%  interrupts.CPU18.CAL:Function_call_interrupts
     27681 ± 18%    +574.6%     186736 ± 12%  interrupts.CPU18.LOC:Local_timer_interrupts
    530.25 ± 39%    +474.1%       3044 ± 54%  interrupts.CPU18.RES:Rescheduling_interrupts
      2163 ± 19%    +157.8%       5576 ± 43%  interrupts.CPU19.CAL:Function_call_interrupts
     27638 ± 17%    +576.2%     186887 ± 12%  interrupts.CPU19.LOC:Local_timer_interrupts
     22.50 ± 37%    +891.1%     223.00 ± 53%  interrupts.CPU2.42:IR-PCI-MSI.2621448-edge.eth0-TxRx-7
      2509 ± 15%    +103.5%       5106 ± 24%  interrupts.CPU2.CAL:Function_call_interrupts
     68246 ± 44%    +663.7%     521166 ±  3%  interrupts.CPU2.LOC:Local_timer_interrupts
      3283 ± 16%    +202.1%       9920 ±  6%  interrupts.CPU2.RES:Rescheduling_interrupts
      1985 ± 27%     +77.2%       3517 ± 22%  interrupts.CPU20.CAL:Function_call_interrupts
     20176 ± 25%    +338.7%      88514 ±  4%  interrupts.CPU20.LOC:Local_timer_interrupts
    158.75 ± 85%    +423.5%     831.00 ± 81%  interrupts.CPU20.RES:Rescheduling_interrupts
     17714 ± 21%    +403.0%      89101 ±  4%  interrupts.CPU21.LOC:Local_timer_interrupts
     19989 ± 26%    +345.8%      89114 ±  4%  interrupts.CPU22.LOC:Local_timer_interrupts
      1917 ± 16%     +53.3%       2938 ± 14%  interrupts.CPU23.CAL:Function_call_interrupts
     20259 ± 25%    +337.8%      88694 ±  4%  interrupts.CPU23.LOC:Local_timer_interrupts
     65.50 ± 47%    +935.6%     678.33 ± 42%  interrupts.CPU23.RES:Rescheduling_interrupts
     42.25 ±106%    +213.2%     132.33 ±  3%  interrupts.CPU3.38:IR-PCI-MSI.2621444-edge.eth0-TxRx-3
      2459 ±  5%     +65.8%       4079 ± 11%  interrupts.CPU3.CAL:Function_call_interrupts
     67886 ± 44%    +668.3%     521548 ±  2%  interrupts.CPU3.LOC:Local_timer_interrupts
      3210 ± 11%    +258.6%      11513 ±  6%  interrupts.CPU3.RES:Rescheduling_interrupts
     30.75 ± 83%    +291.3%     120.33 ± 14%  interrupts.CPU3.TLB:TLB_shootdowns
      2426 ± 13%     +83.4%       4448 ± 36%  interrupts.CPU4.CAL:Function_call_interrupts
     37124 ± 15%    +761.3%     319758 ±  8%  interrupts.CPU4.LOC:Local_timer_interrupts
      2399 ± 13%     +49.5%       3586 ± 15%  interrupts.CPU5.CAL:Function_call_interrupts
     36954 ± 15%    +766.0%     320024 ±  8%  interrupts.CPU5.LOC:Local_timer_interrupts
      1160 ± 20%    +270.2%       4294 ± 35%  interrupts.CPU5.RES:Rescheduling_interrupts
      2312 ± 13%     +88.3%       4354 ± 36%  interrupts.CPU6.CAL:Function_call_interrupts
     36695 ± 15%    +771.4%     319752 ±  8%  interrupts.CPU6.LOC:Local_timer_interrupts
     26.75 ± 93%    +454.5%     148.33 ± 20%  interrupts.CPU7.35:IR-PCI-MSI.2621441-edge.eth0-TxRx-0
      2264 ± 14%     +44.9%       3280 ±  4%  interrupts.CPU7.CAL:Function_call_interrupts
     37045 ± 16%    +763.5%     319871 ±  8%  interrupts.CPU7.LOC:Local_timer_interrupts
    988469 ± 19%    +467.0%    5604547 ±  4%  interrupts.LOC:Local_timer_interrupts
    682.50 ±128%   +4928.1%      34316 ± 64%  interrupts.NMI:Non-maskable_interrupts
    682.50 ±128%   +4928.1%      34316 ± 64%  interrupts.PMI:Performance_monitoring_interrupts
     42962 ± 15%    +174.9%     118108 ±  3%  interrupts.RES:Rescheduling_interrupts
    363.75 ± 25%    +174.7%     999.33 ±  8%  interrupts.TLB:TLB_shootdowns
     25990 ±122%   +2646.0%     713704 ± 54%  sched_debug.cfs_rq:/.load.avg
     64235 ± 51%   +9796.7%    6357197 ± 67%  sched_debug.cfs_rq:/.load.max
     12139 ±173%   +1078.2%     143022 ± 13%  sched_debug.cfs_rq:/.load.min
     17932 ± 52%   +9596.0%    1738721 ± 67%  sched_debug.cfs_rq:/.load.stddev
     38895 ±121%   +1749.5%     719393 ± 29%  sched_debug.cfs_rq:/.min_vruntime.avg
     55160 ± 81%   +1328.0%     787707 ± 27%  sched_debug.cfs_rq:/.min_vruntime.max
     34031 ±132%   +1714.5%     617519 ± 30%  sched_debug.cfs_rq:/.min_vruntime.min
      4690 ± 35%   +1321.4%      66668 ± 12%  sched_debug.cfs_rq:/.min_vruntime.stddev
      0.38 ± 34%     +91.5%       0.72 ±  2%  sched_debug.cfs_rq:/.nr_running.avg
      0.08 ±173%    +540.0%       0.53 ± 17%  sched_debug.cfs_rq:/.nr_running.min
      0.42 ± 16%     -47.0%       0.22 ± 16%  sched_debug.cfs_rq:/.nr_running.stddev
     51.26 ± 71%     -76.5%      12.07 ±129%  sched_debug.cfs_rq:/.removed.load_avg.avg
    853.33 ± 34%     -88.0%     102.43 ± 81%  sched_debug.cfs_rq:/.removed.load_avg.max
    192.81 ± 48%     -84.9%      29.03 ±107%  sched_debug.cfs_rq:/.removed.load_avg.stddev
      2358 ± 71%     -76.5%     554.12 ±128%  sched_debug.cfs_rq:/.removed.runnable_sum.avg
     39372 ± 34%     -88.0%       4728 ± 81%  sched_debug.cfs_rq:/.removed.runnable_sum.max
      8865 ± 48%     -84.9%       1335 ±107%  sched_debug.cfs_rq:/.removed.runnable_sum.stddev
     19.30 ± 70%     -72.7%       5.27 ±133%  sched_debug.cfs_rq:/.removed.util_avg.avg
    334.25 ± 42%     -88.3%      39.03 ± 96%  sched_debug.cfs_rq:/.removed.util_avg.max
     73.16 ± 55%     -84.2%      11.55 ±118%  sched_debug.cfs_rq:/.removed.util_avg.stddev
     20.06 ±135%    +715.8%     163.62 ± 23%  sched_debug.cfs_rq:/.runnable_load_avg.avg
     36.92 ± 91%   +1230.2%     491.07 ± 42%  sched_debug.cfs_rq:/.runnable_load_avg.max
     11.00 ±173%    +729.4%      91.23 ±  7%  sched_debug.cfs_rq:/.runnable_load_avg.min
     10.98 ± 68%   +1078.5%     129.38 ± 48%  sched_debug.cfs_rq:/.runnable_load_avg.stddev
     20325 ±130%   +3215.0%     673797 ± 55%  sched_debug.cfs_rq:/.runnable_weight.avg
     36337 ± 89%  +17255.7%    6306603 ± 67%  sched_debug.cfs_rq:/.runnable_weight.max
     10942 ±173%    +920.9%     111713 ± 12%  sched_debug.cfs_rq:/.runnable_weight.min
     11513 ± 61%  +14930.8%    1730643 ± 67%  sched_debug.cfs_rq:/.runnable_weight.stddev
     -1228         +6393.3%     -79797        sched_debug.cfs_rq:/.spread0.min
      4690 ± 35%   +1321.3%      66669 ± 12%  sched_debug.cfs_rq:/.spread0.stddev
    613.69 ±  7%     -18.4%     501.06 ±  6%  sched_debug.cfs_rq:/.util_avg.avg
      1466 ± 22%     -47.2%     773.77 ±  8%  sched_debug.cfs_rq:/.util_avg.max
    390.67 ± 34%     -71.6%     110.77 ± 16%  sched_debug.cfs_rq:/.util_est_enqueued.max
      0.08 ±173%  +12340.0%      10.37 ±134%  sched_debug.cfs_rq:/.util_est_enqueued.min
    110.31 ± 36%     -73.9%      28.77 ± 15%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
    525263 ± 10%     +35.1%     709585 ±  9%  sched_debug.cpu.avg_idle.avg
    113812 ±169%    +390.1%     557751 ± 17%  sched_debug.cpu.avg_idle.min
     47864 ± 54%    +540.1%     306389        sched_debug.cpu.clock.avg
     47866 ± 54%    +540.1%     306389        sched_debug.cpu.clock.max
     47862 ± 54%    +540.1%     306388        sched_debug.cpu.clock.min
      1.05 ± 31%     -70.7%       0.31 ±  5%  sched_debug.cpu.clock.stddev
     47864 ± 54%    +540.1%     306389        sched_debug.cpu.clock_task.avg
     47866 ± 54%    +540.1%     306389        sched_debug.cpu.clock_task.max
     47862 ± 54%    +540.1%     306388        sched_debug.cpu.clock_task.min
      1.05 ± 31%     -70.7%       0.31 ±  5%  sched_debug.cpu.clock_task.stddev
     20.50 ±132%    +659.5%     155.68 ± 25%  sched_debug.cpu.cpu_load[0].avg
     44.08 ± 72%    +744.9%     372.47 ± 56%  sched_debug.cpu.cpu_load[0].max
     11.00 ±173%    +732.7%      91.60 ±  6%  sched_debug.cpu.cpu_load[0].min
     11.62 ± 64%    +747.7%      98.48 ± 63%  sched_debug.cpu.cpu_load[0].stddev
     21.23 ±118%    +457.1%     118.31 ± 26%  sched_debug.cpu.cpu_load[1].avg
     66.50 ± 70%    +300.2%     266.13 ± 47%  sched_debug.cpu.cpu_load[1].max
     11.00 ±173%    +533.9%      69.73 ± 13%  sched_debug.cpu.cpu_load[1].min
     12.94 ± 54%    +421.5%      67.47 ± 51%  sched_debug.cpu.cpu_load[1].stddev
    259.42 ± 13%     -50.6%     128.27 ± 31%  sched_debug.cpu.cpu_load[4].max
    957.37 ± 94%    +594.8%       6651        sched_debug.cpu.curr->pid.avg
      2129 ± 39%    +315.8%       8853        sched_debug.cpu.curr->pid.max
    446.83 ±173%   +1051.5%       5145 ±  2%  sched_debug.cpu.curr->pid.min
    708.90 ± 11%    +128.0%       1616 ±  4%  sched_debug.cpu.curr->pid.stddev
     25516 ±125%   +2655.5%     703092 ± 57%  sched_debug.cpu.load.avg
     64235 ± 51%   +9748.4%    6326187 ± 68%  sched_debug.cpu.load.max
     12139 ±173%   +1040.9%     138498 ± 18%  sched_debug.cpu.load.min
     17351 ± 56%   +9861.2%    1728442 ± 68%  sched_debug.cpu.load.stddev
     21460 ±120%   +1188.5%     276515        sched_debug.cpu.nr_load_updates.avg
     26877 ± 91%    +939.1%     279268        sched_debug.cpu.nr_load_updates.max
     20126 ±130%   +1268.8%     275499        sched_debug.cpu.nr_load_updates.min
      0.37 ± 36%     +92.6%       0.71 ±  4%  sched_debug.cpu.nr_running.avg
      0.08 ±173%    +500.0%       0.50 ± 16%  sched_debug.cpu.nr_running.min
      0.42 ± 16%     -42.6%       0.24 ± 15%  sched_debug.cpu.nr_running.stddev
     43921 ±167%   +1277.1%     604829 ±  5%  sched_debug.cpu.nr_switches.avg
     54096 ±130%   +1084.5%     640782 ±  5%  sched_debug.cpu.nr_switches.max
     42673 ±171%   +1252.3%     577085 ±  5%  sched_debug.cpu.nr_switches.min
      2063 ± 10%   +1163.0%      26068 ± 20%  sched_debug.cpu.nr_switches.stddev
     -1.94         -5537.1%     105.53 ± 47%  sched_debug.cpu.nr_uninterruptible.avg
     15.08 ± 45%   +1351.0%     218.87 ± 35%  sched_debug.cpu.nr_uninterruptible.max
     11.51 ±106%    +687.1%      90.62 ± 38%  sched_debug.cpu.nr_uninterruptible.stddev
     47862 ± 54%    +540.1%     306388        sched_debug.cpu_clk
     44986 ± 58%    +574.6%     303491        sched_debug.ktime
     48925 ± 52%    +527.3%     306925        sched_debug.sched_clk
     12.36 ±122%     -12.4        0.00        perf-profile.calltrace.cycles-pp.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      6.08 ±103%      -6.1        0.00        perf-profile.calltrace.cycles-pp.vfs_unlink.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.62 ±101%      -5.6        0.00        perf-profile.calltrace.cycles-pp.btrfs_unlink.vfs_unlink.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +0.7        0.71 ± 26%  perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      0.00            +1.1        1.14 ± 72%  perf-profile.calltrace.cycles-pp.btrfs_insert_empty_items.btrfs_new_inode.btrfs_create.path_openat.do_filp_open
      0.18 ±173%      +1.3        1.48 ± 60%  perf-profile.calltrace.cycles-pp.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry
      0.00            +1.4        1.36 ± 18%  perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
      0.15 ±173%      +1.5        1.66 ± 53%  perf-profile.calltrace.cycles-pp.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
      0.31 ±101%      +1.8        2.16 ± 51%  perf-profile.calltrace.cycles-pp.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle
      0.00            +1.9        1.91 ± 57%  perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry.start_secondary
      0.45 ±114%      +2.3        2.76 ± 54%  perf-profile.calltrace.cycles-pp.__btrfs_drop_extents.insert_reserved_file_extent.btrfs_finish_ordered_io.normal_work_helper.process_one_work
      0.00            +2.4        2.42 ± 58%  perf-profile.calltrace.cycles-pp.btrfs_new_inode.btrfs_create.path_openat.do_filp_open.do_sys_open
      0.54 ±122%      +2.5        3.02 ± 56%  perf-profile.calltrace.cycles-pp.insert_reserved_file_extent.btrfs_finish_ordered_io.normal_work_helper.process_one_work.worker_thread
      0.00            +2.8        2.77 ± 27%  perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_insert_empty_items.insert_with_overflow.btrfs_insert_dir_item.btrfs_add_link
      0.27 ±173%      +2.8        3.09 ± 45%  perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
      0.27 ±173%      +2.9        3.18 ± 42%  perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt
      0.58 ±100%      +3.0        3.54 ± 41%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
      0.93 ±118%      +3.1        4.05 ± 60%  perf-profile.calltrace.cycles-pp.normal_work_helper.process_one_work.worker_thread.kthread.ret_from_fork
      0.00            +3.4        3.39 ± 26%  perf-profile.calltrace.cycles-pp.btrfs_insert_empty_items.insert_with_overflow.btrfs_insert_dir_item.btrfs_add_link.btrfs_create
      0.00            +3.4        3.42 ± 26%  perf-profile.calltrace.cycles-pp.insert_with_overflow.btrfs_insert_dir_item.btrfs_add_link.btrfs_create.path_openat
      0.49 ±100%      +3.5        3.95 ± 38%  perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      0.00            +3.9        3.89 ± 54%  perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      0.00            +4.1        4.08 ± 31%  perf-profile.calltrace.cycles-pp.btrfs_insert_dir_item.btrfs_add_link.btrfs_create.path_openat.do_filp_open
      0.00            +4.2        4.23 ± 31%  perf-profile.calltrace.cycles-pp.btrfs_add_link.btrfs_create.path_openat.do_filp_open.do_sys_open
      0.79 ±101%      +4.3        5.06 ± 38%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle
      1.25 ±101%      +7.5        8.72 ± 39%  perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry
      0.00            +8.0        8.02 ± 34%  perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary
      0.00            +8.7        8.71 ± 42%  perf-profile.calltrace.cycles-pp.btrfs_create.path_openat.do_filp_open.do_sys_open.do_syscall_64
      0.00           +11.1       11.08 ± 45%  perf-profile.calltrace.cycles-pp.path_openat.do_filp_open.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00           +21.0       20.98 ± 46%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary
      0.00           +31.1       31.14 ± 46%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      0.00           +38.0       37.98 ± 45%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      0.00           +38.1       38.11 ± 45%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
      0.00           +38.1       38.11 ± 45%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
     12.36 ±122%     -12.4        0.00        perf-profile.children.cycles-pp.do_unlinkat
      6.08 ±103%      -6.1        0.00        perf-profile.children.cycles-pp.vfs_unlink
      5.62 ±101%      -5.6        0.00        perf-profile.children.cycles-pp.btrfs_unlink
      0.00            +0.1        0.07 ±  7%  perf-profile.children.cycles-pp.do_faccessat
      0.01 ±173%      +0.1        0.08 ± 31%  perf-profile.children.cycles-pp.block_rsv_use_bytes
      0.00            +0.1        0.09 ± 39%  perf-profile.children.cycles-pp.may_open
      0.01 ±173%      +0.1        0.10 ± 22%  perf-profile.children.cycles-pp.run_local_timers
      0.00            +0.1        0.09 ± 22%  perf-profile.children.cycles-pp.trigger_load_balance
      0.01 ±173%      +0.1        0.11 ± 68%  perf-profile.children.cycles-pp.__x64_sys_close
      0.00            +0.1        0.10 ± 31%  perf-profile.children.cycles-pp.resched_curr
      0.00            +0.1        0.10 ± 52%  perf-profile.children.cycles-pp.__d_alloc
      0.01 ±173%      +0.1        0.12 ± 29%  perf-profile.children.cycles-pp.tick_check_broadcast_expired
      0.01 ±173%      +0.1        0.12 ± 42%  perf-profile.children.cycles-pp.__intel_pmu_enable_all
      0.01 ±173%      +0.1        0.12 ± 23%  perf-profile.children.cycles-pp.security_inode_alloc
      0.00            +0.1        0.11 ± 26%  perf-profile.children.cycles-pp.call_cpuidle
      0.00            +0.1        0.11 ± 29%  perf-profile.children.cycles-pp.tick_nohz_tick_stopped
      0.01 ±173%      +0.1        0.14 ± 39%  perf-profile.children.cycles-pp.note_gp_changes
      0.00            +0.1        0.13 ± 68%  perf-profile.children.cycles-pp.__push_leaf_left
      0.01 ±173%      +0.1        0.15 ± 66%  perf-profile.children.cycles-pp.cpu_load_update
      0.01 ±173%      +0.1        0.15 ± 30%  perf-profile.children.cycles-pp.d_alloc
      0.01 ±173%      +0.1        0.15 ± 35%  perf-profile.children.cycles-pp.alloc_set_pte
      0.01 ±173%      +0.1        0.15 ± 55%  perf-profile.children.cycles-pp.get_cpu_device
      0.00            +0.1        0.14 ± 43%  perf-profile.children.cycles-pp.btrfs_set_token_64
      0.01 ±173%      +0.1        0.16 ± 44%  perf-profile.children.cycles-pp.inode_init_always
      0.00            +0.1        0.15 ± 33%  perf-profile.children.cycles-pp.ksoftirqd_running
      0.07 ±107%      +0.1        0.22 ± 26%  perf-profile.children.cycles-pp.intel_pmu_disable_all
      0.00            +0.2        0.15 ±  5%  perf-profile.children.cycles-pp.run_rebalance_domains
      0.01 ±173%      +0.2        0.16 ± 49%  perf-profile.children.cycles-pp.perf_event_task_tick
      0.06 ±128%      +0.2        0.21 ± 41%  perf-profile.children.cycles-pp.select_task_rq_fair
      0.01 ±173%      +0.2        0.17 ± 17%  perf-profile.children.cycles-pp.hrtimer_forward
      0.00            +0.2        0.16 ± 41%  perf-profile.children.cycles-pp.tick_program_event
      0.01 ±173%      +0.2        0.17 ± 38%  perf-profile.children.cycles-pp.wp_page_copy
      0.00            +0.2        0.17 ± 22%  perf-profile.children.cycles-pp.idle_cpu
      0.00            +0.2        0.17 ± 22%  perf-profile.children.cycles-pp.cpu_load_update_active
      0.00            +0.2        0.17 ± 45%  perf-profile.children.cycles-pp.update_cfs_group
      0.01 ±173%      +0.2        0.18 ± 13%  perf-profile.children.cycles-pp.update_blocked_averages
      0.01 ±173%      +0.2        0.18 ± 82%  perf-profile.children.cycles-pp.do_dentry_open
      0.03 ±173%      +0.2        0.20 ± 30%  perf-profile.children.cycles-pp.push_leaf_left
      0.01 ±173%      +0.2        0.19 ± 43%  perf-profile.children.cycles-pp.d_alloc_parallel
      0.05 ±173%      +0.2        0.24 ± 50%  perf-profile.children.cycles-pp.rb_next
      0.03 ±173%      +0.2        0.21 ± 62%  perf-profile.children.cycles-pp.rcu_idle_exit
      0.00            +0.2        0.19 ± 47%  perf-profile.children.cycles-pp.__hrtimer_get_next_event
      0.00            +0.2        0.20 ± 48%  perf-profile.children.cycles-pp.rcu_eqs_enter
      0.01 ±173%      +0.2        0.21 ± 72%  perf-profile.children.cycles-pp.tick_nohz_irq_exit
      0.00            +0.2        0.20 ± 52%  perf-profile.children.cycles-pp.pm_qos_read_value
      0.03 ±173%      +0.2        0.23 ± 50%  perf-profile.children.cycles-pp.do_wp_page
      0.01 ±173%      +0.2        0.24 ± 67%  perf-profile.children.cycles-pp.__push_leaf_right
      0.10 ±146%      +0.2        0.33 ± 58%  perf-profile.children.cycles-pp.native_apic_msr_eoi_write
      0.00            +0.2        0.24 ± 26%  perf-profile.children.cycles-pp.btrfs_alloc_inode
      0.00            +0.2        0.24 ± 54%  perf-profile.children.cycles-pp.inode_tree_add
      0.05 ±173%      +0.2        0.29 ± 57%  perf-profile.children.cycles-pp.clear_state_bit
      0.06 ±173%      +0.3        0.31 ± 47%  perf-profile.children.cycles-pp.timerqueue_del
      0.07 ±173%      +0.3        0.33 ± 49%  perf-profile.children.cycles-pp.__remove_hrtimer
      0.01 ±173%      +0.3        0.27 ± 52%  perf-profile.children.cycles-pp.poll_idle
      0.04 ±173%      +0.3        0.30 ± 51%  perf-profile.children.cycles-pp.find_busiest_group
      0.03 ±173%      +0.3        0.30 ± 31%  perf-profile.children.cycles-pp.interrupt_entry
      0.01 ±173%      +0.3        0.29 ± 73%  perf-profile.children.cycles-pp.hrtimer_next_event_without
      0.10 ±100%      +0.3        0.38 ± 19%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.01 ±173%      +0.3        0.30 ± 51%  perf-profile.children.cycles-pp.update_sd_lb_stats
      0.00            +0.3        0.29 ± 36%  perf-profile.children.cycles-pp.new_slab
      0.04 ±173%      +0.3        0.34 ± 64%  perf-profile.children.cycles-pp.__hrtimer_next_event_base
      0.07 ±173%      +0.3        0.38 ± 48%  perf-profile.children.cycles-pp.btrfs_get_or_create_delayed_node
      0.13 ±115%      +0.3        0.44 ± 33%  perf-profile.children.cycles-pp.update_rq_clock
      0.07 ±107%      +0.3        0.39 ± 25%  perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
      0.01 ±173%      +0.3        0.33 ± 13%  perf-profile.children.cycles-pp.rcu_sched_clock_irq
      0.00            +0.3        0.33 ± 13%  perf-profile.children.cycles-pp.menu_reflect
      0.01 ±173%      +0.3        0.36 ± 55%  perf-profile.children.cycles-pp.timerqueue_add
      0.21 ±127%      +0.4        0.57 ± 45%  perf-profile.children.cycles-pp.btrfs_tree_lock
      0.00            +0.4        0.37 ± 38%  perf-profile.children.cycles-pp.___slab_alloc
      0.00            +0.4        0.37 ± 38%  perf-profile.children.cycles-pp.__slab_alloc
      0.08 ±100%      +0.4        0.46 ± 16%  perf-profile.children.cycles-pp.run_timer_softirq
      0.18 ±122%      +0.4        0.56 ± 17%  perf-profile.children.cycles-pp.push_leaf_right
      0.01 ±173%      +0.4        0.40 ± 32%  perf-profile.children.cycles-pp.alloc_inode
      0.01 ±173%      +0.4        0.42 ± 36%  perf-profile.children.cycles-pp.new_inode_pseudo
      0.07 ±107%      +0.4        0.49 ± 28%  perf-profile.children.cycles-pp.native_sched_clock
      0.01 ±173%      +0.4        0.44 ± 67%  perf-profile.children.cycles-pp.enqueue_hrtimer
      0.03 ±173%      +0.4        0.45 ± 77%  perf-profile.children.cycles-pp.swapgs_restore_regs_and_return_to_usermode
      0.07 ±107%      +0.4        0.50 ± 26%  perf-profile.children.cycles-pp.sched_clock
      0.01 ±173%      +0.4        0.45 ± 41%  perf-profile.children.cycles-pp.new_inode
      0.00            +0.4        0.45 ± 54%  perf-profile.children.cycles-pp.ktime_get_update_offsets_now
      0.13 ±115%      +0.5        0.59 ± 29%  perf-profile.children.cycles-pp.lapic_next_deadline
      0.01 ±173%      +0.5        0.49 ± 72%  perf-profile.children.cycles-pp.wait_reserve_ticket
      0.00            +0.5        0.48 ± 66%  perf-profile.children.cycles-pp.prepare_to_wait_event
      0.00            +0.5        0.49 ± 56%  perf-profile.children.cycles-pp.btrfs_insert_delayed_dir_index
      0.09 ±173%      +0.5        0.58 ± 68%  perf-profile.children.cycles-pp.irq_enter
      0.00            +0.5        0.50 ± 38%  perf-profile.children.cycles-pp.find_next_bit
      0.17 ±119%      +0.5        0.67 ± 36%  perf-profile.children.cycles-pp.btrfs_delayed_update_inode
      0.07 ±107%      +0.5        0.57 ± 18%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.04 ±173%      +0.5        0.54 ± 64%  perf-profile.children.cycles-pp.rebalance_domains
      0.04 ±173%      +0.5        0.57 ± 61%  perf-profile.children.cycles-pp.load_balance
      0.24 ±133%      +0.5        0.77 ± 33%  perf-profile.children.cycles-pp.btrfs_update_inode
      0.11 ±128%      +0.6        0.67 ± 22%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.03 ±173%      +0.6        0.63 ± 66%  perf-profile.children.cycles-pp.cpuidle_governor_latency_req
      0.17 ±100%      +0.6        0.79 ± 12%  perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
      0.14 ±107%      +0.6        0.76 ± 17%  perf-profile.children.cycles-pp.native_write_msr
      0.00            +0.7        0.69 ± 71%  perf-profile.children.cycles-pp.__next_timer_interrupt
      0.10 ±173%      +0.7        0.83 ± 30%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      0.21 ±135%      +0.8        0.98 ± 17%  perf-profile.children.cycles-pp.clockevents_program_event
      0.03 ±173%      +0.8        0.80 ± 27%  perf-profile.children.cycles-pp.schedule_idle
      0.16 ±139%      +0.8        0.95 ±  7%  perf-profile.children.cycles-pp.ktime_get
      0.04 ±173%      +0.8        0.84 ± 20%  perf-profile.children.cycles-pp.split_leaf
      0.16 ±139%      +0.9        1.03 ± 30%  perf-profile.children.cycles-pp.__wake_up_common_lock
      0.00            +0.9        0.87 ± 56%  perf-profile.children.cycles-pp.btrfs_lookup
      0.00            +0.9        0.87 ± 56%  perf-profile.children.cycles-pp.btrfs_lookup_dentry
      0.14 ±173%      +0.9        1.07 ± 18%  perf-profile.children.cycles-pp.autoremove_wake_function
      0.06 ±128%      +0.9        1.01 ±115%  perf-profile.children.cycles-pp.btrfs_run_delalloc_range
      0.06 ±128%      +0.9        1.01 ±115%  perf-profile.children.cycles-pp.cow_file_range
      0.13 ±115%      +1.0        1.09 ± 61%  perf-profile.children.cycles-pp.get_next_timer_interrupt
      0.15 ±154%      +1.0        1.13 ± 15%  perf-profile.children.cycles-pp.__wake_up_common
      0.19 ±100%      +1.0        1.19 ± 13%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.26 ±109%      +1.3        1.53 ± 51%  perf-profile.children.cycles-pp.btrfs_read_lock_root_node
      0.12 ±173%      +1.3        1.40 ± 51%  perf-profile.children.cycles-pp.btrfs_tree_read_lock
      0.23 ±127%      +1.5        1.70 ± 61%  perf-profile.children.cycles-pp.tick_nohz_next_event
      0.04 ±173%      +1.6        1.63 ± 13%  perf-profile.children.cycles-pp.scheduler_tick
      0.82 ±134%      +1.9        2.72 ± 37%  perf-profile.children.cycles-pp.setup_items_for_insert
      0.24 ±122%      +2.0        2.20 ± 62%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.00            +2.5        2.45 ± 59%  perf-profile.children.cycles-pp.btrfs_new_inode
      0.54 ±122%      +2.5        3.02 ± 56%  perf-profile.children.cycles-pp.insert_reserved_file_extent
      0.41 ±149%      +3.1        3.48 ± 38%  perf-profile.children.cycles-pp.update_process_times
      1.00 ±102%      +3.1        4.12 ± 26%  perf-profile.children.cycles-pp.__softirqentry_text_start
      0.93 ±118%      +3.1        4.05 ± 60%  perf-profile.children.cycles-pp.normal_work_helper
      0.42 ±149%      +3.1        3.57 ± 35%  perf-profile.children.cycles-pp.tick_sched_handle
      0.00            +3.4        3.45 ± 27%  perf-profile.children.cycles-pp.insert_with_overflow
      0.69 ±111%      +3.5        4.15 ± 32%  perf-profile.children.cycles-pp.tick_sched_timer
      0.77 ±106%      +3.7        4.51 ± 18%  perf-profile.children.cycles-pp.irq_exit
      0.00            +4.1        4.11 ± 32%  perf-profile.children.cycles-pp.btrfs_insert_dir_item
      0.38 ±102%      +4.2        4.61 ± 54%  perf-profile.children.cycles-pp.menu_select
      0.00            +4.3        4.26 ± 32%  perf-profile.children.cycles-pp.btrfs_add_link
      0.97 ±112%      +5.2        6.13 ± 21%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.85 ±106%      +5.9        6.77 ± 33%  perf-profile.children.cycles-pp.btrfs_insert_empty_items
      1.24 ±104%      +7.3        8.55 ± 21%  perf-profile.children.cycles-pp.hrtimer_interrupt
      0.00            +8.8        8.78 ± 42%  perf-profile.children.cycles-pp.btrfs_create
      0.24 ±101%     +11.2       11.40 ± 46%  perf-profile.children.cycles-pp.path_openat
      0.24 ±101%     +11.2       11.42 ± 46%  perf-profile.children.cycles-pp.do_filp_open
      0.30 ±103%     +11.3       11.62 ± 46%  perf-profile.children.cycles-pp.do_sys_open
      1.78 ±100%     +12.0       13.76 ± 15%  perf-profile.children.cycles-pp.smp_apic_timer_interrupt
      2.22 ±100%     +12.4       14.62 ± 14%  perf-profile.children.cycles-pp.apic_timer_interrupt
      0.00           +38.1       38.11 ± 45%  perf-profile.children.cycles-pp.start_secondary
      0.00            +0.1        0.07 ± 28%  perf-profile.self.cycles-pp.sched_clock_idle_wakeup_event
      0.00            +0.1        0.08 ± 36%  perf-profile.self.cycles-pp.irq_enter
      0.00            +0.1        0.08 ± 48%  perf-profile.self.cycles-pp.___slab_alloc
      0.00            +0.1        0.09 ± 27%  perf-profile.self.cycles-pp.trigger_load_balance
      0.01 ±173%      +0.1        0.10 ± 22%  perf-profile.self.cycles-pp.tick_check_broadcast_expired
      0.00            +0.1        0.10 ± 31%  perf-profile.self.cycles-pp.resched_curr
      0.00            +0.1        0.11 ± 26%  perf-profile.self.cycles-pp.call_cpuidle
      0.00            +0.1        0.13 ± 33%  perf-profile.self.cycles-pp.perf_mux_hrtimer_handler
      0.00            +0.1        0.13 ± 31%  perf-profile.self.cycles-pp.tick_nohz_get_sleep_length
      0.01 ±173%      +0.1        0.14 ± 62%  perf-profile.self.cycles-pp.get_cpu_device
      0.00            +0.1        0.13 ± 43%  perf-profile.self.cycles-pp.clockevents_program_event
      0.00            +0.1        0.13 ± 36%  perf-profile.self.cycles-pp.__hrtimer_get_next_event
      0.07 ±107%      +0.1        0.20 ± 20%  perf-profile.self.cycles-pp._raw_read_lock
      0.01 ±173%      +0.1        0.15 ± 66%  perf-profile.self.cycles-pp.cpu_load_update
      0.00            +0.1        0.14 ± 37%  perf-profile.self.cycles-pp.filemap_map_pages
      0.00            +0.1        0.15 ± 33%  perf-profile.self.cycles-pp.ksoftirqd_running
      0.01 ±173%      +0.2        0.16 ± 49%  perf-profile.self.cycles-pp.perf_event_task_tick
      0.00            +0.2        0.15 ± 55%  perf-profile.self.cycles-pp.scheduler_tick
      0.00            +0.2        0.16 ± 24%  perf-profile.self.cycles-pp.new_slab
      0.01 ±173%      +0.2        0.17 ± 17%  perf-profile.self.cycles-pp.hrtimer_forward
      0.03 ±173%      +0.2        0.18 ± 44%  perf-profile.self.cycles-pp.try_to_wake_up
      0.00            +0.2        0.16 ± 41%  perf-profile.self.cycles-pp.tick_program_event
      0.00            +0.2        0.16 ± 23%  perf-profile.self.cycles-pp.cpu_load_update_active
      0.00            +0.2        0.16 ± 44%  perf-profile.self.cycles-pp.update_cfs_group
      0.00            +0.2        0.17 ± 22%  perf-profile.self.cycles-pp.idle_cpu
      0.03 ±173%      +0.2        0.19 ± 62%  perf-profile.self.cycles-pp.swapgs_restore_regs_and_return_to_usermode
      0.01 ±173%      +0.2        0.18 ± 37%  perf-profile.self.cycles-pp.update_sd_lb_stats
      0.07 ±107%      +0.2        0.25 ± 12%  perf-profile.self.cycles-pp.update_rq_clock
      0.05 ±173%      +0.2        0.24 ± 50%  perf-profile.self.cycles-pp.rb_next
      0.00            +0.2        0.19 ± 55%  perf-profile.self.cycles-pp.rcu_eqs_enter
      0.00            +0.2        0.20 ± 50%  perf-profile.self.cycles-pp.pm_qos_read_value
      0.00            +0.2        0.21 ± 46%  perf-profile.self.cycles-pp.smp_apic_timer_interrupt
      0.01 ±173%      +0.2        0.23 ± 41%  perf-profile.self.cycles-pp.__hrtimer_run_queues
      0.10 ±146%      +0.2        0.32 ± 59%  perf-profile.self.cycles-pp.native_apic_msr_eoi_write
      0.00            +0.2        0.23 ± 48%  perf-profile.self.cycles-pp.inode_tree_add
      0.05 ±173%      +0.2        0.29 ± 57%  perf-profile.self.cycles-pp.__btrfs_release_delayed_node
      0.07 ±107%      +0.2        0.31 ± 67%  perf-profile.self.cycles-pp.setup_items_for_insert
      0.01 ±173%      +0.3        0.26 ± 48%  perf-profile.self.cycles-pp.timerqueue_add
      0.01 ±173%      +0.3        0.27 ± 52%  perf-profile.self.cycles-pp.poll_idle
      0.03 ±173%      +0.3        0.30 ± 31%  perf-profile.self.cycles-pp.interrupt_entry
      0.10 ±100%      +0.3        0.37 ± 20%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.00            +0.3        0.28 ± 12%  perf-profile.self.cycles-pp.menu_reflect
      0.01 ±173%      +0.3        0.29 ± 51%  perf-profile.self.cycles-pp.hrtimer_interrupt
      0.03 ±173%      +0.3        0.31 ± 62%  perf-profile.self.cycles-pp.__hrtimer_next_event_base
      0.01 ±173%      +0.3        0.30 ± 16%  perf-profile.self.cycles-pp.rcu_sched_clock_irq
      0.07 ±107%      +0.3        0.37 ± 19%  perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
      0.03 ±173%      +0.3        0.33 ± 37%  perf-profile.self.cycles-pp.run_timer_softirq
      0.00            +0.3        0.31 ± 72%  perf-profile.self.cycles-pp.__next_timer_interrupt
      0.06 ±128%      +0.3        0.39 ± 50%  perf-profile.self.cycles-pp.tick_nohz_next_event
      0.00            +0.4        0.37 ± 63%  perf-profile.self.cycles-pp.ktime_get_update_offsets_now
      0.03 ±173%      +0.4        0.43 ± 33%  perf-profile.self.cycles-pp.native_sched_clock
      0.01 ±173%      +0.4        0.44 ± 38%  perf-profile.self.cycles-pp.ktime_get
      0.00            +0.5        0.46 ± 44%  perf-profile.self.cycles-pp.find_next_bit
      0.10 ±100%      +0.5        0.64 ± 19%  perf-profile.self.cycles-pp.__sched_text_start
      0.17 ±100%      +0.5        0.72 ± 27%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.11 ±128%      +0.5        0.66 ± 23%  perf-profile.self.cycles-pp.native_irq_return_iret
      0.14 ±107%      +0.6        0.76 ± 17%  perf-profile.self.cycles-pp.native_write_msr
      0.10 ±173%      +0.7        0.82 ± 31%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      0.12 ±105%      +1.6        1.68 ± 40%  perf-profile.self.cycles-pp.menu_select



***************************************************************************************************
ivb44: 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 64G memory
=========================================================================================
compiler/cpufreq_governor/directio/disk/fstype/kconfig/media/rootfs/tbox_group/test/testcase/ucode:
  gcc-7/performance/directio/1HDD/btrfs/x86_64-rhel-7.6/hdd/debian-x86_64-2018-04-03.cgz/ivb44/MWUL/fxmark/0x42d

commit: 
  4438cf50e7 ("doc, block, bfq: add information on bfq execution time")
  56a85fd837 ("loop: properly observe rotational flag of underlying device")

4438cf50e7b315ff 56a85fd8376ef32458efb6ea97a 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
    193.22 ±  5%     +52.0%     293.74        fxmark.hdd_btrfs_MWUL_12_directio.idle_sec
     33.12 ±  5%     +67.1%      55.34        fxmark.hdd_btrfs_MWUL_12_directio.iowait_sec
     14.29           +10.1%      15.73        fxmark.hdd_btrfs_MWUL_12_directio.iowait_util
     19.83 ±  4%     +52.2%      30.17        fxmark.hdd_btrfs_MWUL_12_directio.real_sec
     19.82 ±  4%     +52.0%      30.12        fxmark.hdd_btrfs_MWUL_12_directio.secs
      4.96           -53.2%       2.32 ± 10%  fxmark.hdd_btrfs_MWUL_12_directio.sys_sec
      2.15 ±  4%     -69.3%       0.66 ± 10%  fxmark.hdd_btrfs_MWUL_12_directio.sys_util
      0.13 ±  5%     -38.0%       0.08 ±  2%  fxmark.hdd_btrfs_MWUL_12_directio.user_util
     41694           -90.3%       4050 ± 12%  fxmark.hdd_btrfs_MWUL_12_directio.works
      2109 ±  4%     -93.6%     134.50 ± 12%  fxmark.hdd_btrfs_MWUL_12_directio.works/sec
    272.51 ± 10%     +50.6%     410.51        fxmark.hdd_btrfs_MWUL_16_directio.idle_sec
     33.40 ± 12%     +66.8%      55.72        fxmark.hdd_btrfs_MWUL_16_directio.iowait_sec
     10.70           +11.0%      11.88        fxmark.hdd_btrfs_MWUL_16_directio.iowait_util
     19.98 ± 10%     +51.0%      30.17        fxmark.hdd_btrfs_MWUL_16_directio.real_sec
     19.92 ± 10%     +51.6%      30.20        fxmark.hdd_btrfs_MWUL_16_directio.secs
      5.28 ±  4%     -55.1%       2.37 ±  6%  fxmark.hdd_btrfs_MWUL_16_directio.sys_sec
      1.71 ±  8%     -70.4%       0.51 ±  6%  fxmark.hdd_btrfs_MWUL_16_directio.sys_util
      0.11 ±  9%     -37.0%       0.07 ±  5%  fxmark.hdd_btrfs_MWUL_16_directio.user_util
     41487           -91.0%       3729 ± 12%  fxmark.hdd_btrfs_MWUL_16_directio.works
      2106 ± 10%     -94.1%     123.51 ± 12%  fxmark.hdd_btrfs_MWUL_16_directio.works/sec
      2.92 ±  3%    +562.3%      19.34 ±  2%  fxmark.hdd_btrfs_MWUL_1_directio.iowait_sec
      4.47 ±  2%    +507.0%      27.16 ±  2%  fxmark.hdd_btrfs_MWUL_1_directio.real_sec
      4.41 ±  2%    +514.1%      27.10 ±  2%  fxmark.hdd_btrfs_MWUL_1_directio.secs
      0.02 ± 19%    +488.9%       0.13 ± 16%  fxmark.hdd_btrfs_MWUL_1_directio.softirq_sec
      1.38 ±  2%    +411.2%       7.07 ±  2%  fxmark.hdd_btrfs_MWUL_1_directio.sys_sec
     31.18 ±  2%     -15.7%      26.29 ±  2%  fxmark.hdd_btrfs_MWUL_1_directio.sys_util
      0.11 ±  6%    +218.2%       0.35 ±  4%  fxmark.hdd_btrfs_MWUL_1_directio.user_sec
      2.48 ±  7%     -47.6%       1.30 ±  3%  fxmark.hdd_btrfs_MWUL_1_directio.user_util
     40914          +306.1%     166158        fxmark.hdd_btrfs_MWUL_1_directio.works
      9278 ±  2%     -33.9%       6136 ±  2%  fxmark.hdd_btrfs_MWUL_1_directio.works/sec
      4.26 ± 16%     +64.6%       7.02 ± 38%  fxmark.hdd_btrfs_MWUL_2_directio.idle_sec
     21.71 ± 15%    +132.2%      50.41 ±  5%  fxmark.hdd_btrfs_MWUL_2_directio.iowait_sec
     74.52 ±  4%     +14.4%      85.28 ±  5%  fxmark.hdd_btrfs_MWUL_2_directio.iowait_util
     14.81 ± 11%    +103.3%      30.11        fxmark.hdd_btrfs_MWUL_2_directio.real_sec
     14.74 ± 11%    +103.8%      30.05        fxmark.hdd_btrfs_MWUL_2_directio.secs
      0.33 ± 13%     -28.2%       0.24 ± 18%  fxmark.hdd_btrfs_MWUL_2_directio.softirq_util
      2.79 ±  3%     -51.2%       1.36 ±  2%  fxmark.hdd_btrfs_MWUL_2_directio.sys_sec
      9.69 ±  8%     -76.3%       2.30 ±  2%  fxmark.hdd_btrfs_MWUL_2_directio.sys_util
      0.65           -52.3%       0.31 ±  6%  fxmark.hdd_btrfs_MWUL_2_directio.user_util
     41050           -92.1%       3249 ±  6%  fxmark.hdd_btrfs_MWUL_2_directio.works
      2821 ± 10%     -96.2%     108.14 ±  6%  fxmark.hdd_btrfs_MWUL_2_directio.works/sec
     29.77 ±  5%    +114.6%      63.89        fxmark.hdd_btrfs_MWUL_4_directio.idle_sec
     22.13 ±  6%    +133.3%      51.63        fxmark.hdd_btrfs_MWUL_4_directio.iowait_sec
     14.37 ±  6%    +110.4%      30.22        fxmark.hdd_btrfs_MWUL_4_directio.real_sec
     14.30 ±  6%    +110.9%      30.16        fxmark.hdd_btrfs_MWUL_4_directio.secs
      0.10 ± 14%     +42.9%       0.15 ± 16%  fxmark.hdd_btrfs_MWUL_4_directio.softirq_sec
      0.19 ±  9%     -32.1%       0.13 ± 17%  fxmark.hdd_btrfs_MWUL_4_directio.softirq_util
      3.59 ±  3%     -38.4%       2.21 ± 10%  fxmark.hdd_btrfs_MWUL_4_directio.sys_sec
      6.45 ±  3%     -71.0%       1.87 ±  9%  fxmark.hdd_btrfs_MWUL_4_directio.sys_util
      0.21 ± 10%     +15.3%       0.24 ±  4%  fxmark.hdd_btrfs_MWUL_4_directio.user_sec
      0.38 ±  7%     -45.5%       0.21 ±  4%  fxmark.hdd_btrfs_MWUL_4_directio.user_util
     41386           -82.2%       7348 ± 15%  fxmark.hdd_btrfs_MWUL_4_directio.works
      2906 ±  6%     -91.6%     243.53 ± 14%  fxmark.hdd_btrfs_MWUL_4_directio.works/sec
    110.14 ± 10%     +61.5%     177.85        fxmark.hdd_btrfs_MWUL_8_directio.idle_sec
     31.07 ± 10%     +74.5%      54.21        fxmark.hdd_btrfs_MWUL_8_directio.iowait_sec
     18.81 ± 10%     +60.1%      30.10        fxmark.hdd_btrfs_MWUL_8_directio.real_sec
     18.78 ± 10%     +60.0%      30.05        fxmark.hdd_btrfs_MWUL_8_directio.secs
      0.10 ± 11%     -32.5%       0.07 ± 16%  fxmark.hdd_btrfs_MWUL_8_directio.softirq_util
      4.58 ±  5%     -51.4%       2.22 ±  8%  fxmark.hdd_btrfs_MWUL_8_directio.sys_sec
      3.15 ±  5%     -69.9%       0.95 ±  8%  fxmark.hdd_btrfs_MWUL_8_directio.sys_util
      0.34 ± 11%     -20.0%       0.27 ±  4%  fxmark.hdd_btrfs_MWUL_8_directio.user_sec
      0.23 ± 13%     -50.6%       0.12 ±  4%  fxmark.hdd_btrfs_MWUL_8_directio.user_util
     41677           -89.7%       4290 ±  9%  fxmark.hdd_btrfs_MWUL_8_directio.works
      2246 ± 11%     -93.6%     142.77 ±  9%  fxmark.hdd_btrfs_MWUL_8_directio.works/sec
    209.59 ±  2%    +407.7%       1064        fxmark.time.elapsed_time
    209.59 ±  2%    +407.7%       1064        fxmark.time.elapsed_time.max
     14186 ±  2%     -20.7%      11254 ±  4%  fxmark.time.file_system_inputs
   1402120 ±  2%    +363.3%    6495744        fxmark.time.file_system_outputs
    108821 ±  4%    +381.2%     523616 ±  5%  fxmark.time.involuntary_context_switches
      7.25 ± 17%     -69.0%       2.25 ± 19%  fxmark.time.percent_of_cpu_this_job_got
      7.89          +141.6%      19.06        fxmark.time.system_time
    167121 ±  2%     +15.3%     192617 ±  2%  fxmark.time.voluntary_context_switches
     35.66 ±  3%      +9.8       45.51        mpstat.cpu.all.iowait%
      5.05            -1.5        3.53 ±  2%  mpstat.cpu.all.sys%
      4.43 ± 15%      -3.5        0.98 ± 16%  mpstat.cpu.all.usr%
    694220 ±  4%    +282.7%    2656546        numa-numastat.node0.local_node
    703679 ±  4%    +279.0%    2666820        numa-numastat.node0.numa_hit
    159588 ± 11%    +155.5%     407780 ± 11%  numa-numastat.node1.local_node
    166029 ± 10%    +149.0%     413398 ± 12%  numa-numastat.node1.numa_hit
     52.50 ±  2%      -7.1%      48.75        vmstat.cpu.id
     35.25 ±  3%     +28.4%      45.25        vmstat.cpu.wa
     99.75 ± 13%     -75.9%      24.00 ± 12%  vmstat.io.bi
     44840           -43.0%      25581        vmstat.io.bo
   1338807           +26.4%    1692708        vmstat.memory.cache
     18901           -59.9%       7576        vmstat.system.cs
     17520 ±  2%     -22.9%      13503 ± 10%  vmstat.system.in
  62746964 ± 12%    +374.6%  2.978e+08 ± 46%  cpuidle.C1.time
   1425690 ±  2%    +173.1%    3893177 ± 39%  cpuidle.C1.usage
 2.554e+08 ± 21%    +484.7%  1.494e+09 ±  9%  cpuidle.C1E.time
    940670 ± 50%    +394.7%    4653803 ± 42%  cpuidle.C1E.usage
 1.083e+09 ± 15%    +332.0%  4.679e+09 ±  9%  cpuidle.C6.time
   1591508 ± 31%    +249.6%    5564109 ±  2%  cpuidle.C6.usage
    218610 ±  2%    +303.8%     882839 ± 73%  cpuidle.POLL.time
    314892 ±  2%    +135.6%     741934 ± 16%  cpuidle.POLL.usage
     54.36 ±  2%      -8.4%      49.79        iostat.cpu.idle
     35.27 ±  3%     +28.8%      45.42        iostat.cpu.iowait
      4.68           -23.5%       3.59        iostat.cpu.system
      4.29 ± 16%     -77.7%       0.96 ± 16%  iostat.cpu.user
      1.38           -18.3%       1.13        iostat.sda.avgqu-sz
    389.97           -47.9%     203.35        iostat.sda.avgrq-sz
     10.65            +6.2%      11.31        iostat.sda.util
    132.07           +19.4%     157.69        iostat.sda.w/s
     24369           -38.4%      15019        iostat.sda.wkB/s
     91.11           +17.1%     106.68        iostat.sda.wrqm/s
    196784 ± 20%     +94.6%     382883 ±  9%  numa-meminfo.node0.Active
     80877 ±  5%    +184.1%     229800 ±  3%  numa-meminfo.node0.Active(file)
    657695 ±  2%     +33.4%     877093 ±  3%  numa-meminfo.node0.FilePages
     43803 ± 11%    +142.4%     106184 ±  4%  numa-meminfo.node0.Inactive
     32692 ±  8%    +188.5%      94307 ±  5%  numa-meminfo.node0.Inactive(file)
     58506 ±  2%    +204.2%     177961        numa-meminfo.node0.KReclaimable
   1016173 ±  5%     +38.7%    1409076 ±  2%  numa-meminfo.node0.MemUsed
     58506 ±  2%    +204.2%     177961        numa-meminfo.node0.SReclaimable
     54423 ±  4%     +58.7%      86372 ±  5%  numa-meminfo.node0.SUnreclaim
    112930 ±  2%    +134.1%     264334 ±  2%  numa-meminfo.node0.Slab
      9361 ± 55%     +70.2%      15936 ± 25%  numa-meminfo.node1.Inactive
     32598 ±  4%     +40.9%      45932 ±  3%  numa-meminfo.node1.KReclaimable
     32598 ±  4%     +40.9%      45932 ±  3%  numa-meminfo.node1.SReclaimable
     70803 ±  3%     +23.3%      87282 ±  6%  numa-meminfo.node1.Slab
    338162           +47.6%     499163        meminfo.Active
     87276          +180.0%     244379        meminfo.Active(file)
    158421           +31.4%     208195        meminfo.AnonHugePages
   1247393           +17.8%    1469212        meminfo.Cached
     53141          +129.7%     122069        meminfo.Inactive
     17515           +34.6%      23581        meminfo.Inactive(anon)
     35625          +176.5%      98488        meminfo.Inactive(file)
     91080          +145.8%     223850        meminfo.KReclaimable
   1925145           +18.7%    2285482        meminfo.Memused
    697.25 ±100%    +140.0%       1673 ± 13%  meminfo.Mlocked
     91080          +145.8%     223850        meminfo.SReclaimable
     92595           +37.9%     127720        meminfo.SUnreclaim
     17846           +39.9%      24961        meminfo.Shmem
    183676           +91.4%     351571        meminfo.Slab
      9537 ±  3%     -74.7%       2414        meminfo.max_used_kB
     20184 ±  5%    +184.6%      57444 ±  3%  numa-vmstat.node0.nr_active_file
    754305 ± 13%    +254.8%    2676564 ±  6%  numa-vmstat.node0.nr_dirtied
    179.50 ±  8%     +26.0%     226.25 ±  3%  numa-vmstat.node0.nr_dirty
    164374 ±  2%     +33.4%     219262 ±  3%  numa-vmstat.node0.nr_file_pages
      8159 ±  8%    +188.9%      23570 ±  5%  numa-vmstat.node0.nr_inactive_file
     88.00 ±102%    +153.7%     223.25        numa-vmstat.node0.nr_mlock
     14626 ±  2%    +204.2%      44487        numa-vmstat.node0.nr_slab_reclaimable
     13605 ±  4%     +58.7%      21593 ±  5%  numa-vmstat.node0.nr_slab_unreclaimable
    146.25 ± 20%     -69.6%      44.50 ± 24%  numa-vmstat.node0.nr_writeback
    745730 ± 13%    +258.4%    2672434 ±  6%  numa-vmstat.node0.nr_written
     20184 ±  5%    +184.6%      57444 ±  3%  numa-vmstat.node0.nr_zone_active_file
      8159 ±  8%    +188.9%      23570 ±  5%  numa-vmstat.node0.nr_zone_inactive_file
    310.50 ±  8%     -23.5%     237.50 ±  6%  numa-vmstat.node0.nr_zone_write_pending
    658998 ±  4%    +145.9%    1620694 ±  3%  numa-vmstat.node0.numa_hit
    649527 ±  5%    +147.9%    1610414 ±  3%  numa-vmstat.node0.numa_local
    144908 ± 60%    +212.9%     453395 ± 42%  numa-vmstat.node1.nr_dirtied
      8149 ±  4%     +40.9%      11482 ±  3%  numa-vmstat.node1.nr_slab_reclaimable
    142126 ± 61%    +218.6%     452773 ± 43%  numa-vmstat.node1.nr_written
    574963 ±  5%     +30.1%     747952 ±  9%  numa-vmstat.node1.numa_hit
    413376 ±  8%     +42.2%     587779 ± 11%  numa-vmstat.node1.numa_local
     62731            +1.5%      63696        proc-vmstat.nr_active_anon
     21848          +179.7%      61105        proc-vmstat.nr_active_file
     62693            +1.2%      63433        proc-vmstat.nr_anon_pages
     76.75           +31.9%     101.25        proc-vmstat.nr_anon_transparent_hugepages
   1767315 ±  3%    +253.1%    6241266        proc-vmstat.nr_dirtied
    189.50 ±  2%     +18.5%     224.50 ±  2%  proc-vmstat.nr_dirty
    311881           +17.8%     367315        proc-vmstat.nr_file_pages
      4379           +34.6%       5895        proc-vmstat.nr_inactive_anon
      8891          +176.9%      24624        proc-vmstat.nr_inactive_file
     12107            -3.3%      11705        proc-vmstat.nr_kernel_stack
    174.75 ±100%    +139.3%     418.25 ± 13%  proc-vmstat.nr_mlock
      1122            -2.4%       1095        proc-vmstat.nr_page_table_pages
      4461           +39.9%       6241        proc-vmstat.nr_shmem
     22819          +145.3%      55981        proc-vmstat.nr_slab_reclaimable
     23153           +37.9%      31928        proc-vmstat.nr_slab_unreclaimable
    174.75 ± 14%     -67.1%      57.50 ±  8%  proc-vmstat.nr_writeback
   1739838 ±  3%    +257.2%    6215147        proc-vmstat.nr_written
     62731            +1.5%      63696        proc-vmstat.nr_zone_active_anon
     21848          +179.7%      61105        proc-vmstat.nr_zone_active_file
      4379           +34.6%       5895        proc-vmstat.nr_zone_inactive_anon
      8891          +176.9%      24624        proc-vmstat.nr_zone_inactive_file
    345.25 ±  7%     -27.7%     249.75 ±  2%  proc-vmstat.nr_zone_write_pending
      3707 ± 82%     -66.0%       1262 ±139%  proc-vmstat.numa_hint_faults_local
    897890          +246.4%    3110289        proc-vmstat.numa_hit
    881981          +250.8%    3094390        proc-vmstat.numa_local
    142288          +142.1%     344443 ±  2%  proc-vmstat.pgactivate
   1074500          +235.2%    3601375        proc-vmstat.pgalloc_normal
      3013 ±  2%    +100.8%       6052 ±  2%  proc-vmstat.pgdeactivate
    658693 ±  2%    +288.6%    2559777        proc-vmstat.pgfault
   1069950          +235.7%    3591932        proc-vmstat.pgfree
     21175 ± 11%     +25.8%      26646 ±  7%  proc-vmstat.pgpgin
   9508094 ±  3%    +187.4%   27321590        proc-vmstat.pgpgout
     30091            +1.9%      30666        proc-vmstat.pgrotated
    439624          +285.0%    1692566        proc-vmstat.slabs_scanned
    446.93 ± 24%     +55.0%     692.87 ± 10%  sched_debug.cfs_rq:/.load_avg.avg
    149.94 ± 70%    +139.6%     359.32 ± 21%  sched_debug.cfs_rq:/.load_avg.min
     58835 ± 13%     +27.5%      75024 ± 10%  sched_debug.cfs_rq:/.min_vruntime.avg
     41480 ± 26%     +56.1%      64749 ± 14%  sched_debug.cfs_rq:/.min_vruntime.min
      0.39 ± 16%     -22.9%       0.30 ±  3%  sched_debug.cfs_rq:/.nr_running.stddev
    869.56 ± 10%     -27.8%     627.99 ±  5%  sched_debug.cfs_rq:/.util_avg.max
     69.46 ±117%     -91.1%       6.17 ± 26%  sched_debug.cfs_rq:/.util_est_enqueued.avg
    255.56 ± 26%     -76.6%      59.80 ± 38%  sched_debug.cfs_rq:/.util_est_enqueued.max
     57.09 ± 20%     -73.4%      15.16 ± 31%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
    123588          +346.0%     551256 ±  2%  sched_debug.cpu.clock.avg
    123589          +346.0%     551257 ±  2%  sched_debug.cpu.clock.max
    123587          +346.0%     551256 ±  2%  sched_debug.cpu.clock.min
      0.61 ± 11%     -47.4%       0.32 ± 11%  sched_debug.cpu.clock.stddev
    123588          +346.0%     551256 ±  2%  sched_debug.cpu.clock_task.avg
    123589          +346.0%     551257 ±  2%  sched_debug.cpu.clock_task.max
    123587          +346.0%     551256 ±  2%  sched_debug.cpu.clock_task.min
      0.61 ± 11%     -47.4%       0.32 ± 11%  sched_debug.cpu.clock_task.stddev
    135.94 ± 19%     -45.5%      74.09 ± 35%  sched_debug.cpu.cpu_load[4].max
      1861 ± 22%    +331.6%       8035 ±  8%  sched_debug.cpu.curr->pid.avg
      4221          +251.0%      14814 ±  2%  sched_debug.cpu.curr->pid.max
    844.69 ±100%    +502.1%       5085 ± 12%  sched_debug.cpu.curr->pid.min
      1327 ± 30%    +195.7%       3924 ±  5%  sched_debug.cpu.curr->pid.stddev
     96381          +443.8%     524082 ±  2%  sched_debug.cpu.nr_load_updates.avg
     99314          +430.8%     527202 ±  2%  sched_debug.cpu.nr_load_updates.max
     95092          +449.7%     522696 ±  2%  sched_debug.cpu.nr_load_updates.min
      0.39 ± 16%     -22.2%       0.30 ±  3%  sched_debug.cpu.nr_running.stddev
    239136 ± 13%    +185.0%     681612 ±  3%  sched_debug.cpu.nr_switches.avg
    255369 ± 11%    +179.1%     712761 ±  3%  sched_debug.cpu.nr_switches.max
    225112 ± 15%    +189.1%     650706 ±  4%  sched_debug.cpu.nr_switches.min
     10213 ± 26%    +132.5%      23748 ± 29%  sched_debug.cpu.nr_switches.stddev
    -20.55          -959.0%     176.51 ± 27%  sched_debug.cpu.nr_uninterruptible.avg
     58.31 ± 23%    +751.7%     496.62 ±  8%  sched_debug.cpu.nr_uninterruptible.max
   -146.56          +106.4%    -302.54        sched_debug.cpu.nr_uninterruptible.min
     75.90 ± 44%    +341.5%     335.12 ± 10%  sched_debug.cpu.nr_uninterruptible.stddev
    123587          +346.0%     551256 ±  2%  sched_debug.cpu_clk
    120708          +354.3%     548379 ±  2%  sched_debug.ktime
    124070          +344.7%     551740 ±  2%  sched_debug.sched_clk
     78299          +124.6%     175827        slabinfo.Acpi-Parse.active_objs
      1085          +122.4%       2413        slabinfo.Acpi-Parse.active_slabs
     79233          +122.4%     176218        slabinfo.Acpi-Parse.num_objs
      1085          +122.4%       2413        slabinfo.Acpi-Parse.num_slabs
      3271 ±  4%     +32.7%       4339 ±  3%  slabinfo.Acpi-State.active_objs
      3610 ±  4%     +27.9%       4619 ±  3%  slabinfo.Acpi-State.num_objs
      9125           -11.5%       8073 ±  6%  slabinfo.anon_vma_chain.num_objs
      1000 ±  2%     -11.3%     886.75        slabinfo.avc_xperms_data.active_objs
    403.50           +81.2%     731.25 ±  4%  slabinfo.blkdev_ioc.active_objs
    656.50 ±  3%     +52.8%       1003 ±  4%  slabinfo.blkdev_ioc.num_objs
     35676          +276.2%     134224        slabinfo.btrfs_delayed_node.active_objs
    690.25          +274.4%       2584        slabinfo.btrfs_delayed_node.active_slabs
     35920          +274.2%     134423        slabinfo.btrfs_delayed_node.num_objs
    690.25          +274.4%       2584        slabinfo.btrfs_delayed_node.num_slabs
      1912 ±  4%    +127.6%       4352        slabinfo.btrfs_extent_buffer.active_objs
      2176 ±  3%    +116.0%       4700        slabinfo.btrfs_extent_buffer.num_objs
      7889 ±  2%     +39.5%      11008        slabinfo.btrfs_extent_map.active_objs
    155.00 ±  2%     +76.5%     273.50        slabinfo.btrfs_extent_map.active_slabs
      8701 ±  2%     +76.4%      15347        slabinfo.btrfs_extent_map.num_objs
    155.00 ±  2%     +76.5%     273.50        slabinfo.btrfs_extent_map.num_slabs
     34867          +284.7%     134127        slabinfo.btrfs_inode.active_objs
      1251          +283.1%       4794        slabinfo.btrfs_inode.active_slabs
     35056          +283.0%     134254        slabinfo.btrfs_inode.num_objs
      1251          +283.1%       4794        slabinfo.btrfs_inode.num_slabs
      2732           +23.4%       3371 ±  4%  slabinfo.btrfs_path.active_objs
      2893           +22.6%       3546 ±  4%  slabinfo.btrfs_path.num_objs
     79973          +121.5%     177127        slabinfo.dentry.active_objs
      1920          +119.9%       4223        slabinfo.dentry.active_slabs
     80675          +119.9%     177383        slabinfo.dentry.num_objs
      1920          +119.9%       4223        slabinfo.dentry.num_slabs
      1401 ±  5%     -11.0%       1247 ±  5%  slabinfo.eventpoll_pwq.num_objs
      2269 ±  3%     +74.7%       3965 ±  3%  slabinfo.kmalloc-128.active_objs
      4080           +24.4%       5074        slabinfo.kmalloc-128.num_objs
      2679 ±  7%     -44.5%       1486 ±  3%  slabinfo.kmalloc-192.active_objs
      3481 ±  5%     -21.3%       2739        slabinfo.kmalloc-192.num_objs
    898.50 ±  7%     -12.0%     790.75 ±  3%  slabinfo.kmalloc-rcl-64.active_objs
      4372           -11.3%       3880        slabinfo.lsm_file_cache.active_objs
     91.75 ± 29%    +115.8%     198.00 ±  7%  slabinfo.nfs_commit_data.active_objs
    187.00 ± 28%     +76.9%     330.75 ±  5%  slabinfo.nfs_commit_data.num_objs
     84.00 ±  9%    +103.6%     171.00 ± 11%  slabinfo.nfs_read_data.active_objs
    151.50 ±  8%    +121.3%     335.25 ± 13%  slabinfo.nfs_read_data.num_objs
    908.75 ±  2%     -19.8%     729.00        slabinfo.pde_opener.active_objs
      1145           -10.9%       1020        slabinfo.pid.active_objs
     13795           +17.7%      16232        slabinfo.radix_tree_node.active_objs
     14780           +14.7%      16948        slabinfo.radix_tree_node.num_objs
    892.00 ±  9%     -22.6%     690.75 ± 10%  slabinfo.skbuff_head_cache.active_objs
      1529 ±  7%     -11.8%       1348 ±  6%  slabinfo.skbuff_head_cache.num_objs
      6474 ±  4%     -12.3%       5678 ±  3%  slabinfo.vm_area_struct.active_objs
      7214 ±  5%     -12.2%       6335 ±  3%  slabinfo.vm_area_struct.num_objs
      1669 ±  7%     -13.3%       1448 ±  6%  slabinfo.xfrm_dst_cache.active_objs
     33612 ±  5%    +341.3%     148332        softirqs.BLOCK
      3556          +682.7%      27837        softirqs.CPU0.BLOCK
     39960 ±  5%    +318.1%     167062        softirqs.CPU0.RCU
     18883 ±  2%    +243.1%      64797 ±  2%  softirqs.CPU0.SCHED
      2563          +785.9%      22706        softirqs.CPU0.TASKLET
    104220 ±  9%    +403.9%     525161 ±  4%  softirqs.CPU0.TIMER
     14807 ±  5%    +455.0%      82179 ±  2%  softirqs.CPU1.BLOCK
     30579 ±  7%    +326.8%     130524 ± 10%  softirqs.CPU1.RCU
     14199 ±  3%    +330.5%      61134 ±  3%  softirqs.CPU1.SCHED
     17373 ±  4%    +462.2%      97667        softirqs.CPU1.TASKLET
     74093 ± 12%    +308.1%     302342 ± 17%  softirqs.CPU1.TIMER
     12767 ± 12%    +335.4%      55586 ± 22%  softirqs.CPU16.RCU
      6385 ±  3%    +250.6%      22390 ±  3%  softirqs.CPU16.SCHED
     35098 ± 12%    +236.1%     117949 ± 18%  softirqs.CPU16.TIMER
     13243 ±  8%    +326.8%      56520 ± 18%  softirqs.CPU17.RCU
      6331 ±  2%    +250.4%      22186 ±  2%  softirqs.CPU17.SCHED
     34984 ± 11%    +238.0%     118241 ± 18%  softirqs.CPU17.TIMER
     13151 ±  8%    +313.0%      54310 ± 12%  softirqs.CPU18.RCU
      6560 ±  3%    +239.5%      22271 ±  2%  softirqs.CPU18.SCHED
     34836 ± 10%    +235.0%     116695 ± 18%  softirqs.CPU18.TIMER
     13919 ± 14%    +286.7%      53831 ± 15%  softirqs.CPU19.RCU
      6518 ±  3%    +244.8%      22476 ±  2%  softirqs.CPU19.SCHED
     34164 ± 10%    +230.8%     113004 ± 14%  softirqs.CPU19.TIMER
     23237 ±  5%    +371.3%     109508 ± 13%  softirqs.CPU2.RCU
      8367 ±  3%    +313.9%      34630        softirqs.CPU2.SCHED
     60014 ± 15%    +312.2%     247368 ± 18%  softirqs.CPU2.TIMER
      8210 ± 22%    +263.8%      29871 ± 19%  softirqs.CPU20.RCU
     19520 ± 10%    +197.9%      58150 ± 17%  softirqs.CPU20.TIMER
      7227 ± 15%    +292.1%      28340 ± 17%  softirqs.CPU21.RCU
     19563 ± 13%    +176.8%      54148 ±  9%  softirqs.CPU21.TIMER
      6738 ±  8%    +312.1%      27771 ± 16%  softirqs.CPU22.RCU
     19856 ± 12%    +193.2%      58227 ± 17%  softirqs.CPU22.TIMER
      7068 ± 12%    +295.8%      27976 ± 21%  softirqs.CPU23.RCU
     20532 ± 13%    +206.9%      63019 ± 26%  softirqs.CPU23.TIMER
     24162 ± 12%    +376.9%     115228 ± 13%  softirqs.CPU3.RCU
      8685 ±  4%    +304.9%      35167        softirqs.CPU3.SCHED
     70700 ± 16%    +252.2%     249000 ± 17%  softirqs.CPU3.TIMER
     18076 ±  8%    +360.9%      83322 ± 12%  softirqs.CPU4.RCU
      6638 ±  3%    +259.3%      23854 ±  2%  softirqs.CPU4.SCHED
     52818 ± 16%    +254.0%     186954 ± 22%  softirqs.CPU4.TIMER
     19156 ±  6%    +349.5%      86102 ± 13%  softirqs.CPU5.RCU
      6655 ±  4%    +258.2%      23839 ±  2%  softirqs.CPU5.SCHED
     53180 ± 15%    +253.1%     187754 ± 22%  softirqs.CPU5.TIMER
     18758 ±  4%    +376.4%      89360 ± 15%  softirqs.CPU6.RCU
      6639 ±  5%    +258.1%      23772        softirqs.CPU6.SCHED
     52940 ± 16%    +256.6%     188770 ± 23%  softirqs.CPU6.TIMER
     18941 ±  2%    +327.8%      81024 ±  6%  softirqs.CPU7.RCU
      6768 ±  3%    +250.0%      23685 ±  2%  softirqs.CPU7.SCHED
     54451 ± 16%    +245.9%     188338 ± 23%  softirqs.CPU7.TIMER
    324173 ±  6%    +286.1%    1251558 ± 11%  softirqs.RCU
    171016          +174.8%     469998        softirqs.SCHED
     20655 ±  4%    +486.3%     121103        softirqs.TASKLET
    954209 ±  9%    +212.1%    2978155 ± 13%  softirqs.TIMER
      0.10 ± 44%     -87.6%       0.01 ± 26%  perf-stat.i.MPKI
  11917649 ± 28%     -79.1%    2486349 ± 19%  perf-stat.i.branch-instructions
      0.07 ±  8%      -0.1        0.01 ±  3%  perf-stat.i.branch-miss-rate%
    889209 ± 27%     -80.1%     176655 ± 21%  perf-stat.i.branch-misses
      0.30 ±  5%      -0.2        0.06 ±  3%  perf-stat.i.cache-miss-rate%
    124863 ±  8%     -83.0%      21240 ±  4%  perf-stat.i.cache-misses
    394540 ± 11%     -82.9%      67377 ±  3%  perf-stat.i.cache-references
     18966 ±  2%     -60.0%       7583        perf-stat.i.context-switches
      0.02 ± 31%     -84.8%       0.00 ± 16%  perf-stat.i.cpi
  81719357 ± 15%     -80.9%   15619893 ± 12%  perf-stat.i.cpu-cycles
    296.41           -56.4%     129.34 ±  2%  perf-stat.i.cpu-migrations
      6.07 ± 12%     -77.5%       1.37 ± 10%  perf-stat.i.cycles-between-cache-misses
      0.00 ± 32%      -0.0        0.00 ± 25%  perf-stat.i.dTLB-load-miss-rate%
     26515 ± 18%     -78.6%       5678 ± 19%  perf-stat.i.dTLB-load-misses
  11211406 ± 28%     -78.8%    2378574 ± 19%  perf-stat.i.dTLB-loads
      0.00 ± 17%      -0.0        0.00 ± 14%  perf-stat.i.dTLB-store-miss-rate%
      4617 ± 17%     -81.9%     836.25 ±  6%  perf-stat.i.dTLB-store-misses
   6514374 ± 21%     -80.4%    1276707 ± 13%  perf-stat.i.dTLB-stores
      0.81 ±  3%      -0.7        0.15 ±  2%  perf-stat.i.iTLB-load-miss-rate%
      7806 ± 14%     -78.8%       1657 ± 11%  perf-stat.i.iTLB-load-misses
      1566 ± 22%     -75.4%     384.63 ±  5%  perf-stat.i.iTLB-loads
  56576304 ± 28%     -79.1%   11809110 ± 19%  perf-stat.i.instructions
     63.87 ± 18%     -79.4%      13.18 ± 17%  perf-stat.i.instructions-per-iTLB-miss
      0.01 ± 17%     -77.3%       0.00 ±  8%  perf-stat.i.ipc
      2912           -20.4%       2317        perf-stat.i.minor-faults
     16709 ±  2%     -16.6%      13928        perf-stat.i.msec
      0.24 ±  7%      -0.2        0.04 ± 16%  perf-stat.i.node-load-miss-rate%
      6615 ± 76%     -85.3%     969.37 ± 38%  perf-stat.i.node-load-misses
     21880 ± 19%     -79.9%       4403 ± 16%  perf-stat.i.node-loads
      0.18 ± 26%      -0.2        0.03 ± 14%  perf-stat.i.node-store-miss-rate%
      6679 ± 17%     -85.1%     997.76 ± 12%  perf-stat.i.node-store-misses
     31549 ±  6%     -81.1%       5950 ± 14%  perf-stat.i.node-stores
      2912           -20.4%       2317        perf-stat.i.page-faults
      7.48 ±  4%      -0.4        7.09 ±  2%  perf-stat.overall.branch-miss-rate%
  11892777 ± 28%     -79.0%    2493017 ± 19%  perf-stat.ps.branch-instructions
    887326 ± 27%     -80.0%     177145 ± 21%  perf-stat.ps.branch-misses
    124722 ±  8%     -82.9%      21307 ±  4%  perf-stat.ps.cache-misses
    394052 ± 11%     -82.8%      67593 ±  3%  perf-stat.ps.cache-references
     18874 ±  2%     -59.9%       7576        perf-stat.ps.context-switches
  81586204 ± 15%     -80.8%   15666785 ± 12%  perf-stat.ps.cpu-cycles
    294.95           -56.2%     129.22 ±  2%  perf-stat.ps.cpu-migrations
     26472 ± 18%     -78.5%       5693 ± 19%  perf-stat.ps.dTLB-load-misses
  11187646 ± 28%     -78.7%    2384826 ± 19%  perf-stat.ps.dTLB-loads
      4613 ± 17%     -81.8%     838.99 ±  5%  perf-stat.ps.dTLB-store-misses
   6503546 ± 21%     -80.3%    1280306 ± 13%  perf-stat.ps.dTLB-stores
      7794 ± 14%     -78.7%       1661 ± 11%  perf-stat.ps.iTLB-load-misses
      1562 ± 22%     -75.3%     385.47 ±  5%  perf-stat.ps.iTLB-loads
  56457333 ± 28%     -79.0%   11840667 ± 19%  perf-stat.ps.instructions
      2899           -20.2%       2315        perf-stat.ps.minor-faults
     16645 ±  2%     -16.4%      13919        perf-stat.ps.msec
      6599 ± 76%     -85.3%     971.51 ± 38%  perf-stat.ps.node-load-misses
     21822 ± 19%     -79.8%       4410 ± 16%  perf-stat.ps.node-loads
      6669 ± 17%     -85.0%       1000 ± 12%  perf-stat.ps.node-store-misses
     31502 ±  6%     -81.1%       5966 ± 14%  perf-stat.ps.node-stores
      2899           -20.2%       2315        perf-stat.ps.page-faults
    244.50 ± 60%    +308.3%     998.25 ± 27%  interrupts.35:IR-PCI-MSI.2621441-edge.eth0-TxRx-0
    234.00 ± 68%    +242.7%     802.00 ± 32%  interrupts.36:IR-PCI-MSI.2621442-edge.eth0-TxRx-1
    219.25 ± 56%    +214.8%     690.25 ± 33%  interrupts.37:IR-PCI-MSI.2621443-edge.eth0-TxRx-2
    165.00 ± 31%    +233.3%     550.00 ±  2%  interrupts.38:IR-PCI-MSI.2621444-edge.eth0-TxRx-3
    133.00 ± 30%    +464.8%     751.25 ± 30%  interrupts.39:IR-PCI-MSI.2621445-edge.eth0-TxRx-4
    197.50 ± 73%    +181.9%     556.75 ±  2%  interrupts.40:IR-PCI-MSI.2621446-edge.eth0-TxRx-5
    119.25 ± 10%    +365.0%     554.50 ±  3%  interrupts.41:IR-PCI-MSI.2621447-edge.eth0-TxRx-6
    162.75 ± 29%    +430.0%     862.50 ± 36%  interrupts.42:IR-PCI-MSI.2621448-edge.eth0-TxRx-7
     29671 ± 57%    +711.6%     240799        interrupts.75:IR-PCI-MSI.2097152-edge.isci-msix
    108741 ±  2%     +69.5%     184290 ±  4%  interrupts.CAL:Function_call_interrupts
    183.00 ± 84%    +312.7%     755.25 ± 29%  interrupts.CPU0.36:IR-PCI-MSI.2621442-edge.eth0-TxRx-1
     27.25 ± 38%    +280.7%     103.75 ±  3%  interrupts.CPU0.38:IR-PCI-MSI.2621444-edge.eth0-TxRx-3
     19.00 ± 39%    +668.4%     146.00 ± 42%  interrupts.CPU0.39:IR-PCI-MSI.2621445-edge.eth0-TxRx-4
     75.50 ± 90%    +165.9%     200.75 ±  4%  interrupts.CPU0.40:IR-PCI-MSI.2621446-edge.eth0-TxRx-5
     18.00 ± 26%    +480.6%     104.50 ±  3%  interrupts.CPU0.41:IR-PCI-MSI.2621447-edge.eth0-TxRx-6
     29.50 ± 44%    +501.7%     177.50 ± 49%  interrupts.CPU0.42:IR-PCI-MSI.2621448-edge.eth0-TxRx-7
      3760 ± 57%   +1105.1%      45313        interrupts.CPU0.75:IR-PCI-MSI.2097152-edge.isci-msix
      3356 ± 10%    +178.2%       9336 ± 16%  interrupts.CPU0.CAL:Function_call_interrupts
    424176 ±  2%    +384.8%    2056303 ±  7%  interrupts.CPU0.LOC:Local_timer_interrupts
     32371 ± 12%     +34.8%      43643 ±  8%  interrupts.CPU0.RES:Rescheduling_interrupts
    176.25 ± 31%    +234.9%     590.25 ± 38%  interrupts.CPU0.TLB:TLB_shootdowns
     35.75 ± 58%    +772.0%     311.75 ±  8%  interrupts.CPU1.35:IR-PCI-MSI.2621441-edge.eth0-TxRx-0
    130.00 ± 26%    +306.0%     527.75 ± 26%  interrupts.CPU1.37:IR-PCI-MSI.2621443-edge.eth0-TxRx-2
     22.50 ± 26%    +325.6%      95.75 ±  4%  interrupts.CPU1.38:IR-PCI-MSI.2621444-edge.eth0-TxRx-3
     41.25 ± 26%    +437.6%     221.75 ± 18%  interrupts.CPU1.39:IR-PCI-MSI.2621445-edge.eth0-TxRx-4
     98.75 ±  8%    +353.4%     447.75 ±  3%  interrupts.CPU1.41:IR-PCI-MSI.2621447-edge.eth0-TxRx-6
     24.00 ± 26%    +427.1%     126.50 ± 16%  interrupts.CPU1.42:IR-PCI-MSI.2621448-edge.eth0-TxRx-7
     25787 ± 58%    +657.4%     195323        interrupts.CPU1.75:IR-PCI-MSI.2097152-edge.isci-msix
      3547 ± 15%    +141.9%       8581 ± 14%  interrupts.CPU1.CAL:Function_call_interrupts
    366025 ±  3%    +356.0%    1669062 ±  6%  interrupts.CPU1.LOC:Local_timer_interrupts
     28035 ± 18%     +58.6%      44474 ± 14%  interrupts.CPU1.RES:Rescheduling_interrupts
     92.25 ± 18%    +208.4%     284.50 ± 16%  interrupts.CPU1.TLB:TLB_shootdowns
      2.75 ± 30%   +3809.1%     107.50 ±161%  interrupts.CPU12.RES:Rescheduling_interrupts
      6458 ±  7%    +104.2%      13190 ±  8%  interrupts.CPU16.CAL:Function_call_interrupts
    161518 ±  5%    +270.6%     598601 ± 17%  interrupts.CPU16.LOC:Local_timer_interrupts
      6882 ±  9%     +91.8%      13202 ±  7%  interrupts.CPU17.CAL:Function_call_interrupts
    162111 ±  5%    +269.3%     598629 ± 17%  interrupts.CPU17.LOC:Local_timer_interrupts
      6071 ±  9%    +115.1%      13057 ± 12%  interrupts.CPU18.CAL:Function_call_interrupts
    162778 ±  5%    +267.4%     598039 ± 17%  interrupts.CPU18.LOC:Local_timer_interrupts
      8133 ±  7%     +61.6%      13146 ±  4%  interrupts.CPU19.CAL:Function_call_interrupts
    162807 ±  5%    +276.2%     612554 ± 13%  interrupts.CPU19.LOC:Local_timer_interrupts
     97.00 ± 22%    +454.1%     537.50 ± 35%  interrupts.CPU2.42:IR-PCI-MSI.2621448-edge.eth0-TxRx-7
      3447 ± 13%    +152.5%       8704 ± 18%  interrupts.CPU2.CAL:Function_call_interrupts
    299158 ±  4%    +329.7%    1285348 ±  9%  interrupts.CPU2.LOC:Local_timer_interrupts
      5497 ± 12%     +39.9%       7691 ±  5%  interrupts.CPU2.RES:Rescheduling_interrupts
     40.25 ± 19%    +106.8%      83.25 ± 39%  interrupts.CPU2.TLB:TLB_shootdowns
      4339 ± 16%     +44.2%       6256 ±  5%  interrupts.CPU20.CAL:Function_call_interrupts
     82165 ± 11%    +251.6%     288905 ± 16%  interrupts.CPU20.LOC:Local_timer_interrupts
      3425 ± 14%    +118.5%       7485 ±  5%  interrupts.CPU21.CAL:Function_call_interrupts
     82124 ± 12%    +266.4%     300890 ±  8%  interrupts.CPU21.LOC:Local_timer_interrupts
      3889 ± 15%     +80.2%       7009 ± 12%  interrupts.CPU22.CAL:Function_call_interrupts
     82060 ± 12%    +252.2%     289035 ± 16%  interrupts.CPU22.LOC:Local_timer_interrupts
      3156 ± 10%    +117.7%       6870 ± 14%  interrupts.CPU23.CAL:Function_call_interrupts
     82305 ± 12%    +264.2%     299736 ±  9%  interrupts.CPU23.LOC:Local_timer_interrupts
    102.00 ± 24%    +241.7%     348.50 ±  2%  interrupts.CPU3.38:IR-PCI-MSI.2621444-edge.eth0-TxRx-3
      3307 ±  2%    +138.3%       7882 ± 13%  interrupts.CPU3.CAL:Function_call_interrupts
    299207 ±  4%    +327.1%    1277903 ± 10%  interrupts.CPU3.LOC:Local_timer_interrupts
      4325 ±  6%     +41.9%       6138 ± 10%  interrupts.CPU3.RES:Rescheduling_interrupts
     67.00 ± 26%    +455.2%     372.00 ± 33%  interrupts.CPU4.39:IR-PCI-MSI.2621445-edge.eth0-TxRx-4
      2950 ±  4%    +145.3%       7239 ± 20%  interrupts.CPU4.CAL:Function_call_interrupts
    234082 ±  4%    +300.8%     938197 ± 13%  interrupts.CPU4.LOC:Local_timer_interrupts
      2753 ± 11%     +38.7%       3818 ± 14%  interrupts.CPU4.RES:Rescheduling_interrupts
      3.75 ± 54%   +1540.0%      61.50 ±143%  interrupts.CPU40.RES:Rescheduling_interrupts
      3093 ±  5%    +137.8%       7354 ± 17%  interrupts.CPU5.CAL:Function_call_interrupts
    233637 ±  4%    +300.4%     935496 ± 14%  interrupts.CPU5.LOC:Local_timer_interrupts
      2616 ±  9%     +39.5%       3649 ± 18%  interrupts.CPU5.RES:Rescheduling_interrupts
      3110 ±  5%    +122.7%       6927 ± 11%  interrupts.CPU6.CAL:Function_call_interrupts
    233360 ±  4%    +300.6%     934844 ± 14%  interrupts.CPU6.LOC:Local_timer_interrupts
    624.00 ± 17%     -53.3%     291.50 ± 35%  interrupts.CPU6.NMI:Non-maskable_interrupts
    624.00 ± 17%     -53.3%     291.50 ± 35%  interrupts.CPU6.PMI:Performance_monitoring_interrupts
     65.00 ± 57%    +596.9%     453.00 ± 26%  interrupts.CPU7.35:IR-PCI-MSI.2621441-edge.eth0-TxRx-0
     75.75 ± 38%    +249.8%     265.00 ±  2%  interrupts.CPU7.40:IR-PCI-MSI.2621446-edge.eth0-TxRx-5
      2918          +129.8%       6706 ± 18%  interrupts.CPU7.CAL:Function_call_interrupts
    233720 ±  4%    +305.4%     947415 ± 11%  interrupts.CPU7.LOC:Local_timer_interrupts
   3522671 ±  4%    +293.1%   13846285 ± 11%  interrupts.LOC:Local_timer_interrupts
     94.00           +18.1%     111.00        interrupts.MCP:Machine_check_polls
     89232 ±  4%     +39.8%     124725 ±  6%  interrupts.RES:Rescheduling_interrupts
    495.75 ± 13%    +164.4%       1310 ± 36%  interrupts.TLB:TLB_shootdowns
     15.33 ± 17%      -9.6        5.77 ± 14%  perf-profile.calltrace.cycles-pp.ret_from_fork
     15.33 ± 17%      -9.6        5.77 ± 14%  perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
      9.42 ± 68%      -9.4        0.00        perf-profile.calltrace.cycles-pp.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe
     10.50 ± 55%      -9.1        1.43 ±  6%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
     10.49 ± 55%      -9.1        1.42 ±  6%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
     10.36 ± 22%      -6.4        3.93 ± 16%  perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork
     10.12 ± 21%      -6.3        3.82 ± 17%  perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork
      4.03 ± 56%      -2.9        1.15 ± 11%  perf-profile.calltrace.cycles-pp.btrfs_async_reclaim_metadata_space.process_one_work.worker_thread.kthread.ret_from_fork
      4.35 ± 25%      -2.8        1.53 ± 15%  perf-profile.calltrace.cycles-pp.kthread_worker_fn.kthread.ret_from_fork
      4.30 ± 25%      -2.8        1.51 ± 15%  perf-profile.calltrace.cycles-pp.loop_queue_work.kthread_worker_fn.kthread.ret_from_fork
      3.93 ± 54%      -2.8        1.14 ± 11%  perf-profile.calltrace.cycles-pp.flush_space.btrfs_async_reclaim_metadata_space.process_one_work.worker_thread.kthread
      3.61 ± 28%      -2.6        1.03 ±  8%  perf-profile.calltrace.cycles-pp.btrfs_finish_ordered_io.normal_work_helper.process_one_work.worker_thread.kthread
      3.62 ± 28%      -2.6        1.03 ±  8%  perf-profile.calltrace.cycles-pp.normal_work_helper.process_one_work.worker_thread.kthread.ret_from_fork
      2.64 ± 35%      -2.1        0.50 ± 59%  perf-profile.calltrace.cycles-pp.__btrfs_drop_extents.insert_reserved_file_extent.btrfs_finish_ordered_io.normal_work_helper.process_one_work
      2.82 ± 34%      -2.1        0.70 ±  9%  perf-profile.calltrace.cycles-pp.insert_reserved_file_extent.btrfs_finish_ordered_io.normal_work_helper.process_one_work.worker_thread
      6.26 ± 12%      -2.0        4.21 ± 36%  perf-profile.calltrace.cycles-pp.start_kernel.secondary_startup_64
      6.25 ± 12%      -2.0        4.21 ± 36%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_kernel.secondary_startup_64
      6.22 ± 12%      -2.0        4.20 ± 36%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_kernel.secondary_startup_64
      2.28 ± 37%      -2.0        0.31 ±100%  perf-profile.calltrace.cycles-pp.lo_write_bvec.loop_queue_work.kthread_worker_fn.kthread.ret_from_fork
      2.24 ± 37%      -1.9        0.31 ±100%  perf-profile.calltrace.cycles-pp.do_iter_write.lo_write_bvec.loop_queue_work.kthread_worker_fn.kthread
      2.19 ± 39%      -1.9        0.30 ±100%  perf-profile.calltrace.cycles-pp.do_iter_readv_writev.do_iter_write.lo_write_bvec.loop_queue_work.kthread_worker_fn
      2.19 ± 39%      -1.9        0.30 ±100%  perf-profile.calltrace.cycles-pp.btrfs_file_write_iter.do_iter_readv_writev.do_iter_write.lo_write_bvec.loop_queue_work
      2.09 ± 38%      -1.8        0.28 ±100%  perf-profile.calltrace.cycles-pp.btrfs_buffered_write.btrfs_file_write_iter.do_iter_readv_writev.do_iter_write.lo_write_bvec
      1.98 ± 20%      -1.0        0.97 ± 11%  perf-profile.calltrace.cycles-pp.btrfs_sync_file.loop_queue_work.kthread_worker_fn.kthread.ret_from_fork
      1.01 ± 23%      -0.8        0.26 ±100%  perf-profile.calltrace.cycles-pp.btrfs_run_delayed_refs.flush_space.btrfs_async_reclaim_metadata_space.process_one_work.worker_thread
      1.01 ± 23%      -0.8        0.26 ±100%  perf-profile.calltrace.cycles-pp.__btrfs_run_delayed_refs.btrfs_run_delayed_refs.flush_space.btrfs_async_reclaim_metadata_space.process_one_work
      2.21 ± 23%      -0.7        1.48 ± 33%  perf-profile.calltrace.cycles-pp.fb_flashcursor.process_one_work.worker_thread.kthread.ret_from_fork
      2.21 ± 23%      -0.7        1.48 ± 33%  perf-profile.calltrace.cycles-pp.bit_cursor.fb_flashcursor.process_one_work.worker_thread.kthread
      2.21 ± 23%      -0.7        1.48 ± 33%  perf-profile.calltrace.cycles-pp.soft_cursor.bit_cursor.fb_flashcursor.process_one_work.worker_thread
      2.21 ± 23%      -0.7        1.48 ± 33%  perf-profile.calltrace.cycles-pp.mga_dirty_update.soft_cursor.bit_cursor.fb_flashcursor.process_one_work
      2.12 ± 22%      -0.7        1.42 ± 34%  perf-profile.calltrace.cycles-pp.memcpy_toio.mga_dirty_update.soft_cursor.bit_cursor.fb_flashcursor
      0.79 ± 37%      -0.5        0.27 ±100%  perf-profile.calltrace.cycles-pp.irq_exit.do_IRQ.ret_from_intr.cpuidle_enter_state.do_idle
      0.13 ±173%      +0.5        0.66 ±  7%  perf-profile.calltrace.cycles-pp.load_balance.rebalance_domains.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt
      0.14 ±173%      +0.5        0.69 ± 10%  perf-profile.calltrace.cycles-pp.lapic_next_deadline.clockevents_program_event.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      0.00            +0.6        0.61 ±  8%  perf-profile.calltrace.cycles-pp.run_rebalance_domains.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt
      0.53 ± 62%      +0.6        1.16 ± 17%  perf-profile.calltrace.cycles-pp.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      0.00            +0.7        0.67 ± 19%  perf-profile.calltrace.cycles-pp.run_timer_softirq.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt
      0.15 ±173%      +0.7        0.83 ±  8%  perf-profile.calltrace.cycles-pp.rebalance_domains.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt
      0.15 ±173%      +0.9        1.10 ±  7%  perf-profile.calltrace.cycles-pp.do_filp_open.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.16 ±173%      +1.0        1.12 ±  6%  perf-profile.calltrace.cycles-pp.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.92 ± 12%      +1.3        2.21 ± 22%  perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
      0.48 ±105%      +1.6        2.12 ± 23%  perf-profile.calltrace.cycles-pp.clockevents_program_event.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
      1.89 ± 34%      +2.1        3.99 ± 21%  perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
      2.10 ± 30%      +2.4        4.52 ± 22%  perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt
      2.53 ± 31%      +3.3        5.81 ± 26%  perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      4.15 ± 27%      +4.3        8.43 ± 16%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
      5.91 ± 29%      +6.4       12.27 ± 14%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle
     10.84 ± 28%      +9.0       19.87 ± 12%  perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry
     10.61 ± 32%     +10.2       20.81 ± 10%  perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary
     34.32 ± 28%     +15.4       49.73 ±  7%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary
     49.61 ± 29%     +25.3       74.93 ±  3%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     64.73 ± 24%     +25.5       90.24        perf-profile.calltrace.cycles-pp.secondary_startup_64
     58.35 ± 27%     +27.5       85.86 ±  2%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     58.47 ± 27%     +27.6       86.03 ±  2%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
     58.47 ± 27%     +27.6       86.03 ±  2%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
     16.37 ± 80%     -14.0        2.37 ± 10%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     16.34 ± 80%     -14.0        2.36 ±  9%  perf-profile.children.cycles-pp.do_syscall_64
     15.33 ± 17%      -9.6        5.77 ± 14%  perf-profile.children.cycles-pp.kthread
     15.34 ± 17%      -9.6        5.79 ± 14%  perf-profile.children.cycles-pp.ret_from_fork
      9.43 ± 68%      -9.4        0.00        perf-profile.children.cycles-pp.do_unlinkat
     10.36 ± 22%      -6.4        3.93 ± 16%  perf-profile.children.cycles-pp.worker_thread
     10.12 ± 21%      -6.3        3.82 ± 17%  perf-profile.children.cycles-pp.process_one_work
      7.51 ± 42%      -6.0        1.49 ±  8%  perf-profile.children.cycles-pp.btrfs_search_slot
      4.13 ± 53%      -3.0        1.14 ± 11%  perf-profile.children.cycles-pp.flush_space
      3.18 ±122%      -2.9        0.29 ± 34%  perf-profile.children.cycles-pp.down_write
      4.03 ± 56%      -2.9        1.15 ± 11%  perf-profile.children.cycles-pp.btrfs_async_reclaim_metadata_space
      3.13 ±122%      -2.9        0.28 ± 31%  perf-profile.children.cycles-pp.call_rwsem_down_write_failed
      3.13 ±122%      -2.9        0.28 ± 31%  perf-profile.children.cycles-pp.rwsem_down_write_failed
      4.35 ± 25%      -2.8        1.53 ± 15%  perf-profile.children.cycles-pp.kthread_worker_fn
      4.30 ± 25%      -2.8        1.51 ± 15%  perf-profile.children.cycles-pp.loop_queue_work
      3.62 ± 28%      -2.6        1.03 ±  8%  perf-profile.children.cycles-pp.normal_work_helper
      3.61 ± 28%      -2.6        1.03 ±  8%  perf-profile.children.cycles-pp.btrfs_finish_ordered_io
      2.92 ± 34%      -2.2        0.76 ± 13%  perf-profile.children.cycles-pp.__btrfs_drop_extents
      2.37 ±118%      -2.1        0.25 ± 31%  perf-profile.children.cycles-pp.rwsem_spin_on_owner
      2.82 ± 34%      -2.1        0.70 ±  9%  perf-profile.children.cycles-pp.insert_reserved_file_extent
      2.26 ±102%      -2.1        0.18 ± 14%  perf-profile.children.cycles-pp.__btrfs_run_delayed_items
      6.26 ± 12%      -2.0        4.21 ± 36%  perf-profile.children.cycles-pp.start_kernel
      2.28 ± 37%      -1.8        0.51 ± 22%  perf-profile.children.cycles-pp.lo_write_bvec
      2.24 ± 37%      -1.7        0.50 ± 24%  perf-profile.children.cycles-pp.do_iter_write
      2.19 ± 39%      -1.7        0.48 ± 26%  perf-profile.children.cycles-pp.do_iter_readv_writev
      2.19 ± 39%      -1.7        0.48 ± 26%  perf-profile.children.cycles-pp.btrfs_file_write_iter
      2.10 ± 38%      -1.6        0.46 ± 24%  perf-profile.children.cycles-pp.btrfs_buffered_write
      1.82 ± 28%      -1.6        0.20 ± 21%  perf-profile.children.cycles-pp.btrfs_read_lock_root_node
      1.97 ± 44%      -1.6        0.37 ± 18%  perf-profile.children.cycles-pp.btrfs_lookup_file_extent
      1.64 ± 45%      -1.6        0.06 ± 63%  perf-profile.children.cycles-pp.btrfs_lookup_dir_item
      2.22 ± 54%      -1.5        0.68 ± 16%  perf-profile.children.cycles-pp.__sched_text_start
      1.70 ± 21%      -1.5        0.17 ± 20%  perf-profile.children.cycles-pp.btrfs_tree_read_lock
      1.85 ± 52%      -1.5        0.38 ± 30%  perf-profile.children.cycles-pp.try_to_wake_up
      1.61 ± 54%      -1.4        0.21 ± 15%  perf-profile.children.cycles-pp.btrfs_del_items
      1.22 ± 85%      -1.1        0.09 ± 21%  perf-profile.children.cycles-pp.__btrfs_update_delayed_inode
      1.81 ± 50%      -1.1        0.72 ± 19%  perf-profile.children.cycles-pp._raw_spin_lock
      1.98 ± 20%      -1.0        0.97 ± 11%  perf-profile.children.cycles-pp.btrfs_sync_file
      1.09 ± 60%      -1.0        0.12 ± 54%  perf-profile.children.cycles-pp.__wake_up_common
      1.15 ± 38%      -0.9        0.20 ± 63%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      1.05 ± 60%      -0.9        0.12 ± 48%  perf-profile.children.cycles-pp.autoremove_wake_function
      1.48 ± 27%      -0.9        0.59 ±  4%  perf-profile.children.cycles-pp.do_writepages
      0.99 ± 67%      -0.9        0.10 ± 45%  perf-profile.children.cycles-pp.__wake_up_common_lock
      1.48 ± 28%      -0.9        0.61 ±  3%  perf-profile.children.cycles-pp.__filemap_fdatawrite_range
      1.23 ± 58%      -0.8        0.41 ± 16%  perf-profile.children.cycles-pp.schedule
      0.84 ± 58%      -0.8        0.03 ±105%  perf-profile.children.cycles-pp.reserve_metadata_bytes
      0.83 ± 88%      -0.8        0.06 ± 58%  perf-profile.children.cycles-pp.btrfs_release_path
      1.06 ± 42%      -0.8        0.28 ± 22%  perf-profile.children.cycles-pp.setup_items_for_insert
      1.01 ± 47%      -0.8        0.26 ± 21%  perf-profile.children.cycles-pp.schedule_idle
      1.47 ± 29%      -0.7        0.73 ± 10%  perf-profile.children.cycles-pp.__btrfs_cow_block
      0.87 ± 22%      -0.7        0.13 ± 25%  perf-profile.children.cycles-pp.prepare_to_wait_event
      1.47 ± 29%      -0.7        0.73 ± 10%  perf-profile.children.cycles-pp.btrfs_cow_block
      2.21 ± 23%      -0.7        1.48 ± 33%  perf-profile.children.cycles-pp.fb_flashcursor
      2.21 ± 23%      -0.7        1.48 ± 33%  perf-profile.children.cycles-pp.bit_cursor
      2.21 ± 23%      -0.7        1.48 ± 33%  perf-profile.children.cycles-pp.soft_cursor
      2.21 ± 23%      -0.7        1.48 ± 33%  perf-profile.children.cycles-pp.mga_dirty_update
      2.21 ± 23%      -0.7        1.48 ± 33%  perf-profile.children.cycles-pp.memcpy_toio
      0.91 ± 77%      -0.7        0.18 ± 20%  perf-profile.children.cycles-pp.read_block_for_search
      0.90 ± 50%      -0.7        0.18 ± 24%  perf-profile.children.cycles-pp.btrfs_get_token_32
      0.82 ± 45%      -0.7        0.11 ± 42%  perf-profile.children.cycles-pp.btrfs_tree_lock
      1.21 ±  7%      -0.7        0.51 ± 17%  perf-profile.children.cycles-pp.btrfs_run_delayed_refs
      1.21 ±  7%      -0.7        0.51 ± 17%  perf-profile.children.cycles-pp.__btrfs_run_delayed_refs
      0.77 ± 50%      -0.7        0.10 ± 37%  perf-profile.children.cycles-pp.btrfs_lock_root_node
      0.73 ± 74%      -0.7        0.07 ± 23%  perf-profile.children.cycles-pp.start_transaction
      0.94 ± 34%      -0.6        0.30 ± 21%  perf-profile.children.cycles-pp.start_ordered_ops
      0.91 ± 37%      -0.6        0.28 ± 25%  perf-profile.children.cycles-pp.extent_writepages
      0.77 ± 51%      -0.6        0.14 ± 53%  perf-profile.children.cycles-pp.generic_bin_search
      0.71 ± 74%      -0.6        0.08 ± 23%  perf-profile.children.cycles-pp.btrfs_lookup_inode
      0.91 ± 37%      -0.6        0.28 ± 23%  perf-profile.children.cycles-pp.btrfs_fdatawrite_range
      0.85 ± 33%      -0.6        0.27 ± 23%  perf-profile.children.cycles-pp.extent_write_cache_pages
      0.69 ± 44%      -0.6        0.12 ± 12%  perf-profile.children.cycles-pp.memmove
      0.74 ± 53%      -0.6        0.18 ± 30%  perf-profile.children.cycles-pp.ttwu_do_activate
      0.86 ± 38%      -0.5        0.34 ± 53%  perf-profile.children.cycles-pp.blk_done_softirq
      1.03 ± 30%      -0.5        0.52 ± 15%  perf-profile.children.cycles-pp.btrfs_commit_transaction
      0.75 ± 33%      -0.5        0.25 ± 29%  perf-profile.children.cycles-pp.__extent_writepage
      0.77 ± 40%      -0.5        0.28 ± 47%  perf-profile.children.cycles-pp.blk_update_request
      0.60 ± 84%      -0.5        0.12 ± 16%  perf-profile.children.cycles-pp.find_extent_buffer
      0.62 ± 41%      -0.5        0.14 ± 28%  perf-profile.children.cycles-pp.btrfs_copy_from_user
      0.70 ± 39%      -0.5        0.22 ± 51%  perf-profile.children.cycles-pp.btrfs_end_bio
      0.54 ± 39%      -0.5        0.06 ± 20%  perf-profile.children.cycles-pp.memmove_extent_buffer
      0.60 ± 44%      -0.5        0.13 ± 27%  perf-profile.children.cycles-pp.iov_iter_copy_from_user_atomic
      0.52 ±124%      -0.5        0.05 ± 64%  perf-profile.children.cycles-pp.write
      0.58 ± 43%      -0.5        0.12 ± 28%  perf-profile.children.cycles-pp.memcpy_from_page
      1.22 ± 21%      -0.5        0.77 ± 17%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.58 ± 61%      -0.4        0.14 ± 23%  perf-profile.children.cycles-pp.btrfs_set_token_32
      0.53 ± 38%      -0.4        0.11 ± 74%  perf-profile.children.cycles-pp.btrfs_dirty_pages
      0.65 ± 45%      -0.4        0.23 ± 64%  perf-profile.children.cycles-pp.scsi_io_completion
      0.54 ± 56%      -0.4        0.13 ± 42%  perf-profile.children.cycles-pp.kmem_cache_alloc
      0.64 ± 43%      -0.4        0.23 ± 64%  perf-profile.children.cycles-pp.scsi_end_request
      0.65 ± 52%      -0.4        0.26 ± 12%  perf-profile.children.cycles-pp.pick_next_task_fair
      0.45 ± 53%      -0.4        0.09 ±106%  perf-profile.children.cycles-pp.end_bio_extent_writepage
      0.39 ± 75%      -0.4        0.03 ±100%  perf-profile.children.cycles-pp.btrfs_block_rsv_add
      0.65 ± 30%      -0.3        0.35 ± 21%  perf-profile.children.cycles-pp.copy_page
      0.37 ± 28%      -0.3        0.08 ± 17%  perf-profile.children.cycles-pp.__clear_extent_bit
      0.33 ± 42%      -0.3        0.05 ±102%  perf-profile.children.cycles-pp.smpboot_thread_fn
      0.59 ± 19%      -0.3        0.32 ± 24%  perf-profile.children.cycles-pp.btrfs_write_and_wait_transaction
      0.32 ± 40%      -0.3        0.05 ± 62%  perf-profile.children.cycles-pp.map_private_extent_buffer
      0.37 ± 20%      -0.3        0.11 ± 28%  perf-profile.children.cycles-pp.__extent_writepage_io
      0.36 ± 31%      -0.3        0.10 ± 42%  perf-profile.children.cycles-pp.poll_idle
      0.34 ± 46%      -0.3        0.08 ± 45%  perf-profile.children.cycles-pp.btrfs_mark_buffer_dirty
      0.40 ± 18%      -0.3        0.14 ± 36%  perf-profile.children.cycles-pp.__libc_fork
      0.57 ± 22%      -0.3        0.31 ± 19%  perf-profile.children.cycles-pp.btree_write_cache_pages
      0.53 ± 16%      -0.2        0.28 ± 17%  perf-profile.children.cycles-pp.__btrfs_free_extent
      0.41 ± 30%      -0.2        0.16 ± 28%  perf-profile.children.cycles-pp.__set_extent_bit
      0.33 ± 32%      -0.2        0.09 ± 60%  perf-profile.children.cycles-pp.kmem_cache_free
      0.32 ± 53%      -0.2        0.08 ± 33%  perf-profile.children.cycles-pp.memcpy_extent_buffer
      0.32 ± 67%      -0.2        0.09 ± 16%  perf-profile.children.cycles-pp.link_path_walk
      0.33 ± 37%      -0.2        0.11 ± 28%  perf-profile.children.cycles-pp.submit_extent_page
      0.30 ± 28%      -0.2        0.08 ± 28%  perf-profile.children.cycles-pp.__alloc_pages_nodemask
      0.45 ± 26%      -0.2        0.24 ± 34%  perf-profile.children.cycles-pp.btrfs_csum_file_blocks
      0.50 ±  8%      -0.2        0.28 ± 24%  perf-profile.children.cycles-pp.btrfs_log_changed_extents
      0.28 ± 27%      -0.2        0.08 ± 24%  perf-profile.children.cycles-pp.get_page_from_freelist
      0.26 ± 43%      -0.2        0.06 ± 71%  perf-profile.children.cycles-pp.set_extent_buffer_dirty
      0.23 ± 46%      -0.2        0.03 ±100%  perf-profile.children.cycles-pp.___might_sleep
      0.23 ± 71%      -0.2        0.03 ±100%  perf-profile.children.cycles-pp.__radix_tree_lookup
      0.58 ± 11%      -0.2        0.38 ± 20%  perf-profile.children.cycles-pp.btrfs_log_dentry_safe
      0.58 ± 11%      -0.2        0.38 ± 20%  perf-profile.children.cycles-pp.btrfs_log_inode_parent
      0.58 ± 12%      -0.2        0.38 ± 19%  perf-profile.children.cycles-pp.btrfs_log_inode
      0.31 ± 46%      -0.2        0.12 ± 29%  perf-profile.children.cycles-pp.add_pending_csums
      0.36 ± 21%      -0.2        0.17 ± 21%  perf-profile.children.cycles-pp.write_one_eb
      0.21 ± 50%      -0.2        0.03 ±100%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.21 ± 20%      -0.2        0.03 ±100%  perf-profile.children.cycles-pp.clear_state_bit
      0.24 ± 38%      -0.2        0.06 ± 11%  perf-profile.children.cycles-pp.select_task_rq_fair
      0.31 ± 34%      -0.2        0.14 ± 18%  perf-profile.children.cycles-pp.pagecache_get_page
      0.25 ± 66%      -0.2        0.08 ± 70%  perf-profile.children.cycles-pp.set_next_entity
      0.25 ± 19%      -0.2        0.08 ± 76%  perf-profile.children.cycles-pp.__filemap_fdatawait_range
      0.25 ± 19%      -0.2        0.08 ± 76%  perf-profile.children.cycles-pp.filemap_fdatawait_range
      0.20 ± 36%      -0.2        0.03 ±105%  perf-profile.children.cycles-pp.btrfs_unlock_up_safe
      0.34 ± 24%      -0.2        0.18 ± 22%  perf-profile.children.cycles-pp.read
      0.20 ± 14%      -0.2        0.04 ±102%  perf-profile.children.cycles-pp.run_ksoftirqd
      0.19 ± 46%      -0.2        0.03 ±100%  perf-profile.children.cycles-pp.btrfs_get_token_64
      0.21 ± 35%      -0.2        0.06 ± 14%  perf-profile.children.cycles-pp.update_curr
      0.24 ± 27%      -0.1        0.10 ± 33%  perf-profile.children.cycles-pp.lock_extent_bits
      0.20 ± 43%      -0.1        0.06 ± 15%  perf-profile.children.cycles-pp.lock_and_cleanup_extent_if_need
      0.19 ± 32%      -0.1        0.05 ± 61%  perf-profile.children.cycles-pp.clear_extent_bit
      0.18 ± 23%      -0.1        0.05 ±114%  perf-profile.children.cycles-pp.blk_mq_do_dispatch_sched
      0.16 ± 31%      -0.1        0.04 ±107%  perf-profile.children.cycles-pp._cond_resched
      0.14 ± 69%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.__switch_to
      0.18 ± 32%      -0.1        0.07 ± 70%  perf-profile.children.cycles-pp.blk_flush_plug_list
      0.18 ± 34%      -0.1        0.07 ± 70%  perf-profile.children.cycles-pp.blk_mq_flush_plug_list
      0.15 ± 16%      -0.1        0.04 ± 58%  perf-profile.children.cycles-pp.__lookup_extent_mapping
      0.21 ± 14%      -0.1        0.10 ± 27%  perf-profile.children.cycles-pp.end_bio_extent_buffer_writepage
      0.14 ± 63%      -0.1        0.04 ± 58%  perf-profile.children.cycles-pp.do_anonymous_page
      0.14 ± 35%      -0.1        0.04 ±113%  perf-profile.children.cycles-pp.btrfs_writepage_endio_finish_ordered
      0.14 ± 35%      -0.1        0.04 ±113%  perf-profile.children.cycles-pp.end_extent_writepage
      0.18 ± 23%      -0.1        0.08 ± 59%  perf-profile.children.cycles-pp.seq_read
      0.22 ± 35%      -0.1        0.14 ± 24%  perf-profile.children.cycles-pp.blk_mq_run_hw_queue
      0.16 ± 34%      -0.1        0.08 ± 22%  perf-profile.children.cycles-pp.find_get_entry
      0.22 ± 25%      -0.1        0.14 ± 32%  perf-profile.children.cycles-pp.__blk_mq_run_hw_queue
      0.22 ± 25%      -0.1        0.14 ± 32%  perf-profile.children.cycles-pp.blk_mq_sched_dispatch_requests
      0.13 ± 31%      -0.1        0.05 ±114%  perf-profile.children.cycles-pp.btrfs_wait_extents
      0.17 ± 17%      -0.1        0.10 ± 30%  perf-profile.children.cycles-pp.proc_reg_read
      0.20 ± 28%      -0.1        0.13 ± 27%  perf-profile.children.cycles-pp.__blk_mq_delay_run_hw_queue
      0.10 ± 38%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.__etree_search
      0.03 ±100%      +0.1        0.10 ±  9%  perf-profile.children.cycles-pp.call_timer_fn
      0.01 ±173%      +0.1        0.09 ± 26%  perf-profile.children.cycles-pp.__intel_pmu_enable_all
      0.00            +0.1        0.08 ± 26%  perf-profile.children.cycles-pp.hrtimer_run_queues
      0.03 ±100%      +0.1        0.12 ± 33%  perf-profile.children.cycles-pp.tick_program_event
      0.04 ±173%      +0.1        0.13 ± 36%  perf-profile.children.cycles-pp.rcu_segcblist_ready_cbs
      0.04 ±115%      +0.1        0.14 ± 34%  perf-profile.children.cycles-pp.profile_tick
      0.15 ±  8%      +0.1        0.25 ± 34%  perf-profile.children.cycles-pp.rcu_eqs_exit
      0.00            +0.1        0.10 ± 27%  perf-profile.children.cycles-pp.call_function_single_interrupt
      0.08 ± 68%      +0.1        0.21 ± 32%  perf-profile.children.cycles-pp.calc_global_load_tick
      0.23 ±  5%      +0.1        0.37 ± 26%  perf-profile.children.cycles-pp.rcu_idle_exit
      0.09 ± 33%      +0.1        0.23 ± 12%  perf-profile.children.cycles-pp.run_local_timers
      0.35 ±  8%      +0.1        0.49 ± 10%  perf-profile.children.cycles-pp.native_sched_clock
      0.36 ±  7%      +0.1        0.51 ± 11%  perf-profile.children.cycles-pp.sched_clock
      0.14 ± 34%      +0.2        0.29 ± 13%  perf-profile.children.cycles-pp.intel_pmu_disable_all
      0.48 ± 10%      +0.2        0.64 ±  9%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.17 ± 19%      +0.2        0.33 ± 33%  perf-profile.children.cycles-pp.cpu_load_update
      0.18 ± 15%      +0.2        0.38 ± 29%  perf-profile.children.cycles-pp.perf_event_task_tick
      0.06 ±104%      +0.2        0.25 ± 22%  perf-profile.children.cycles-pp.ksoftirqd_running
      0.09 ± 50%      +0.2        0.28 ± 32%  perf-profile.children.cycles-pp.timekeeping_max_deferment
      0.11 ± 59%      +0.2        0.32 ± 36%  perf-profile.children.cycles-pp.account_process_tick
      0.23 ± 35%      +0.2        0.44 ± 19%  perf-profile.children.cycles-pp.ktime_get_update_offsets_now
      0.54 ± 32%      +0.2        0.77 ±  9%  perf-profile.children.cycles-pp.native_write_msr
      0.16 ± 24%      +0.2        0.40 ± 33%  perf-profile.children.cycles-pp.run_posix_cpu_timers
      0.47 ± 34%      +0.3        0.74 ± 12%  perf-profile.children.cycles-pp.lapic_next_deadline
      0.31 ± 43%      +0.3        0.59 ±  8%  perf-profile.children.cycles-pp.find_busiest_group
      0.27 ± 48%      +0.3        0.56 ± 10%  perf-profile.children.cycles-pp.update_sd_lb_stats
      0.13 ± 60%      +0.3        0.43 ± 51%  perf-profile.children.cycles-pp.tick_sched_do_timer
      0.22 ± 36%      +0.3        0.53 ± 22%  perf-profile.children.cycles-pp.tick_nohz_irq_exit
      0.29 ± 33%      +0.3        0.64 ±  9%  perf-profile.children.cycles-pp.run_rebalance_domains
      0.39 ± 22%      +0.4        0.76 ± 17%  perf-profile.children.cycles-pp.run_timer_softirq
      0.38 ± 32%      +0.4        0.80 ±  8%  perf-profile.children.cycles-pp.load_balance
      0.24 ± 54%      +0.5        0.73 ± 30%  perf-profile.children.cycles-pp.rcu_sched_clock_irq
      0.36 ± 44%      +0.5        0.90 ± 12%  perf-profile.children.cycles-pp.rebalance_domains
      0.67 ± 31%      +0.6        1.23 ± 17%  perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
      1.23 ± 18%      +1.1        2.38 ± 24%  perf-profile.children.cycles-pp.scheduler_tick
      0.85 ± 44%      +1.4        2.20 ± 22%  perf-profile.children.cycles-pp.clockevents_program_event
      0.93 ± 32%      +1.5        2.41 ± 18%  perf-profile.children.cycles-pp.ktime_get
      2.21 ± 28%      +2.1        4.27 ± 22%  perf-profile.children.cycles-pp.update_process_times
      2.42 ± 24%      +2.3        4.76 ± 23%  perf-profile.children.cycles-pp.tick_sched_handle
      2.95 ± 25%      +3.1        6.03 ± 24%  perf-profile.children.cycles-pp.tick_sched_timer
      5.00 ± 27%      +3.8        8.76 ± 15%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      6.79 ± 30%      +5.7       12.53 ± 13%  perf-profile.children.cycles-pp.hrtimer_interrupt
     11.63 ± 29%      +8.3       19.95 ± 12%  perf-profile.children.cycles-pp.smp_apic_timer_interrupt
     12.37 ± 27%      +8.8       21.16 ± 11%  perf-profile.children.cycles-pp.apic_timer_interrupt
     37.70 ± 26%     +14.5       52.19 ±  6%  perf-profile.children.cycles-pp.intel_idle
     54.85 ± 25%     +24.2       79.06 ±  2%  perf-profile.children.cycles-pp.cpuidle_enter_state
     64.73 ± 24%     +25.5       90.24        perf-profile.children.cycles-pp.secondary_startup_64
     64.73 ± 24%     +25.5       90.24        perf-profile.children.cycles-pp.cpu_startup_entry
     64.71 ± 24%     +25.6       90.27        perf-profile.children.cycles-pp.do_idle
     58.47 ± 27%     +27.6       86.03 ±  2%  perf-profile.children.cycles-pp.start_secondary
      2.37 ±118%      -2.1        0.25 ± 31%  perf-profile.self.cycles-pp.rwsem_spin_on_owner
      1.15 ± 38%      -0.9        0.20 ± 63%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      2.19 ± 23%      -0.7        1.48 ± 33%  perf-profile.self.cycles-pp.memcpy_toio
      0.78 ± 49%      -0.6        0.14 ± 25%  perf-profile.self.cycles-pp.btrfs_get_token_32
      0.67 ± 47%      -0.6        0.11 ±  9%  perf-profile.self.cycles-pp.memmove
      0.58 ± 43%      -0.5        0.12 ± 28%  perf-profile.self.cycles-pp.memcpy_from_page
      0.54 ± 46%      -0.4        0.12 ± 48%  perf-profile.self.cycles-pp.__sched_text_start
      0.55 ± 63%      -0.4        0.12 ± 16%  perf-profile.self.cycles-pp.btrfs_set_token_32
      0.37 ± 61%      -0.3        0.07 ± 62%  perf-profile.self.cycles-pp.generic_bin_search
      0.62 ± 34%      -0.3        0.35 ± 21%  perf-profile.self.cycles-pp.copy_page
      0.34 ± 28%      -0.3        0.07 ± 66%  perf-profile.self.cycles-pp.poll_idle
      0.30 ± 39%      -0.3        0.05 ± 62%  perf-profile.self.cycles-pp.map_private_extent_buffer
      0.23 ± 71%      -0.2        0.03 ±100%  perf-profile.self.cycles-pp.__radix_tree_lookup
      0.27 ± 64%      -0.2        0.07 ± 68%  perf-profile.self.cycles-pp.kmem_cache_alloc
      0.23 ± 46%      -0.2        0.03 ±100%  perf-profile.self.cycles-pp.___might_sleep
      0.20 ± 51%      -0.2        0.03 ±100%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.22 ± 43%      -0.2        0.07 ± 72%  perf-profile.self.cycles-pp.try_to_wake_up
      0.18 ± 40%      -0.1        0.05 ± 59%  perf-profile.self.cycles-pp.kmem_cache_free
      0.14 ± 67%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.__switch_to
      0.14 ± 20%      -0.1        0.04 ± 58%  perf-profile.self.cycles-pp.__lookup_extent_mapping
      0.09 ± 24%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.__etree_search
      0.08 ± 17%      -0.0        0.04 ± 58%  perf-profile.self.cycles-pp.find_get_entry
      0.00            +0.1        0.06 ± 20%  perf-profile.self.cycles-pp.__intel_pmu_disable_all
      0.02 ±173%      +0.1        0.09 ± 25%  perf-profile.self.cycles-pp.rebalance_domains
      0.01 ±173%      +0.1        0.09 ± 17%  perf-profile.self.cycles-pp.apic_timer_interrupt
      0.03 ±100%      +0.1        0.11 ± 36%  perf-profile.self.cycles-pp.tick_nohz_idle_got_tick
      0.00            +0.1        0.08 ± 26%  perf-profile.self.cycles-pp.hrtimer_run_queues
      0.03 ±100%      +0.1        0.11 ± 11%  perf-profile.self.cycles-pp.update_process_times
      0.01 ±173%      +0.1        0.10 ± 36%  perf-profile.self.cycles-pp.tick_irq_enter
      0.00            +0.1        0.09 ± 17%  perf-profile.self.cycles-pp.intel_pmu_disable_all
      0.03 ±100%      +0.1        0.12 ± 33%  perf-profile.self.cycles-pp.tick_program_event
      0.04 ±115%      +0.1        0.14 ± 34%  perf-profile.self.cycles-pp.profile_tick
      0.08 ± 59%      +0.1        0.17 ± 20%  perf-profile.self.cycles-pp.rcu_eqs_exit
      0.03 ±173%      +0.1        0.13 ± 38%  perf-profile.self.cycles-pp.load_balance
      0.04 ±106%      +0.1        0.15 ± 23%  perf-profile.self.cycles-pp.run_local_timers
      0.08 ± 68%      +0.1        0.20 ± 33%  perf-profile.self.cycles-pp.calc_global_load_tick
      0.01 ±173%      +0.1        0.14 ± 39%  perf-profile.self.cycles-pp.tick_sched_do_timer
      0.16 ± 41%      +0.1        0.29 ± 19%  perf-profile.self.cycles-pp.hrtimer_interrupt
      0.16 ± 42%      +0.2        0.32 ± 28%  perf-profile.self.cycles-pp.ktime_get_update_offsets_now
      0.30 ±  4%      +0.2        0.46 ± 11%  perf-profile.self.cycles-pp.native_sched_clock
      0.17 ± 19%      +0.2        0.33 ± 33%  perf-profile.self.cycles-pp.cpu_load_update
      0.21 ± 57%      +0.2        0.37 ± 20%  perf-profile.self.cycles-pp.update_sd_lb_stats
      0.03 ±100%      +0.2        0.20 ± 21%  perf-profile.self.cycles-pp.rcu_core
      0.18 ± 15%      +0.2        0.37 ± 29%  perf-profile.self.cycles-pp.perf_event_task_tick
      0.06 ±104%      +0.2        0.25 ± 22%  perf-profile.self.cycles-pp.ksoftirqd_running
      0.11 ± 59%      +0.2        0.32 ± 36%  perf-profile.self.cycles-pp.account_process_tick
      0.06 ± 73%      +0.2        0.28 ± 31%  perf-profile.self.cycles-pp.timekeeping_max_deferment
      0.54 ± 32%      +0.2        0.77 ±  9%  perf-profile.self.cycles-pp.native_write_msr
      0.16 ± 24%      +0.2        0.40 ± 33%  perf-profile.self.cycles-pp.run_posix_cpu_timers
      0.30 ± 23%      +0.3        0.58 ± 19%  perf-profile.self.cycles-pp.run_timer_softirq
      0.12 ± 21%      +0.4        0.48 ± 44%  perf-profile.self.cycles-pp.tick_sched_timer
      0.20 ± 35%      +0.4        0.60 ± 27%  perf-profile.self.cycles-pp.rcu_sched_clock_irq
      1.10 ± 25%      +0.9        2.00 ±  5%  perf-profile.self.cycles-pp.cpuidle_enter_state
      0.43 ± 40%      +1.2        1.61 ± 29%  perf-profile.self.cycles-pp.ktime_get
     37.57 ± 26%     +14.5       52.03 ±  6%  perf-profile.self.cycles-pp.intel_idle





Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.1.0-rc3-00011-g56a85fd" of type "text/plain" (188879 bytes)

View attachment "job-script" of type "text/plain" (7527 bytes)

View attachment "job.yaml" of type "text/plain" (5170 bytes)

View attachment "reproduce" of type "text/plain" (254 bytes)

Powered by blists - more mailing lists