lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Date:   Mon, 27 May 2019 17:17:19 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     Johannes Thumshirn <jthumshirn@...e.de>
Cc:     David Sterba <dsterba@...e.com>,
        Nikolay Borisov <nborisov@...e.com>, Qu Wenruo <wqu@...e.com>,
        LKML <linux-kernel@...r.kernel.org>,
        Linus Torvalds <torvalds@...ux-foundation.org>, lkp@...org
Subject: [btrfs] 2996e1f8bc: aim7.jobs-per-min -13.2% regression

Greeting,

FYI, we noticed a -13.2% regression of aim7.jobs-per-min due to commit:


commit: 2996e1f8bcadf0436cc67b63af01523f6cf5d43f ("btrfs: factor our read/write stage off csum_tree_block into its callers")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master

in testcase: aim7
on test machine: 72 threads Intel(R) Xeon(R) Gold 6139 CPU @ 2.30GHz with 128G memory
with following parameters:

	disk: 4BRD_12G
	md: RAID0
	fs: btrfs
	test: sync_disk_rw
	load: 20
	ucode: 0x200005e
	cpufreq_governor: performance

test-description: AIM7 is a traditional UNIX system level benchmark suite which is used to test and measure the performance of multiuser system.
test-url: https://sourceforge.net/projects/aimbench/files/aim-suite7/

In addition to that, the commit also has significant impact on the following tests:

+------------------+-----------------------------------------------------------------------+
| testcase: change | aim7: aim7.jobs-per-min -19.1% regression                             |
| test machine     | 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz with 384G memory |
| test parameters  | cpufreq_governor=performance                                          |
|                  | disk=4BRD_12G                                                         |
|                  | fs=btrfs                                                              |
|                  | load=20                                                               |
|                  | md=RAID0                                                              |
|                  | test=sync_disk_rw                                                     |
+------------------+-----------------------------------------------------------------------+


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/load/md/rootfs/tbox_group/test/testcase/ucode:
  gcc-7/performance/4BRD_12G/btrfs/x86_64-rhel-7.6/20/RAID0/debian-x86_64-2019-05-14.cgz/lkp-skl-2sp7/sync_disk_rw/aim7/0x200005e

commit: 
  v5.1-rc7
  2996e1f8bc ("btrfs: factor our read/write stage off csum_tree_block into its callers")

        v5.1-rc7 2996e1f8bcadf0436cc67b63af0 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          1:4          -25%            :4     kmsg.Firmware_Bug]:the_BIOS_has_corrupted_hw-PMU_resources(MSR#is#e)
         %stddev     %change         %stddev
             \          |                \  
      1207           -13.2%       1047        aim7.jobs-per-min
     99.41           +15.2%     114.56        aim7.time.elapsed_time
     99.41           +15.2%     114.56        aim7.time.elapsed_time.max
    884.50 ±  2%     +13.5%       1004 ±  3%  aim7.time.involuntary_context_switches
    704.41           +18.2%     832.33        aim7.time.system_time
  21484045           -10.5%   19231461        aim7.time.voluntary_context_switches
      0.02 ± 38%      -0.0        0.00 ± 76%  mpstat.cpu.all.soft%
     59.02 ±  6%     -58.9%      24.26        boot-time.boot
      3928 ±  8%     -62.1%       1489        boot-time.idle
     19798           -12.9%      17235        iostat.md0.w/s
    448478           -13.1%     389592        iostat.md0.wkB/s
    130905 ±  6%     -20.8%     103683 ±  2%  meminfo.AnonHugePages
    100591 ±  2%     -12.9%      87596        meminfo.max_used_kB
      5058 ±  4%     +17.1%       5925 ±  9%  numa-meminfo.node1.KernelStack
      1737 ± 33%     +66.4%       2889 ± 34%  numa-meminfo.node1.PageTables
   1174778 ± 23%     +37.5%    1615284 ±  2%  numa-numastat.node1.local_node
   1189955 ± 23%     +36.7%    1626977 ±  2%  numa-numastat.node1.numa_hit
    344576           -12.9%     300018        vmstat.io.bo
    558315           -21.3%     439168        vmstat.system.cs
    194.50 ± 16%     -29.7%     136.75 ± 15%  numa-vmstat.node1.nr_dirty
      5061 ±  4%     +17.1%       5925 ±  9%  numa-vmstat.node1.nr_kernel_stack
    433.75 ± 33%     +66.4%     721.75 ± 34%  numa-vmstat.node1.nr_page_table_pages
   1065659 ± 23%     +41.9%    1511723 ±  8%  numa-vmstat.node1.numa_hit
    897602 ± 28%     +50.0%    1346715 ± 10%  numa-vmstat.node1.numa_local
      3240 ±  5%     -23.0%       2496        slabinfo.skbuff_head_cache.active_objs
      3264 ±  4%     -19.6%       2624        slabinfo.skbuff_head_cache.num_objs
      1339 ±  2%     -24.6%       1010        slabinfo.task_struct.active_objs
    336.75 ±  2%    +202.4%       1018        slabinfo.task_struct.active_slabs
      1348 ±  2%     -24.5%       1018        slabinfo.task_struct.num_objs
    336.75 ±  2%    +202.4%       1018        slabinfo.task_struct.num_slabs
  23341140 ±  9%     -23.6%   17828343        cpuidle.C1.usage
 1.777e+09 ± 28%    +215.6%  5.608e+09 ±  8%  cpuidle.C1E.time
  12674253 ± 20%     +66.3%   21080014 ±  3%  cpuidle.C1E.usage
 3.182e+09 ± 12%     -87.4%   4.02e+08 ±107%  cpuidle.C6.time
   3770499 ±  3%     -83.8%     609866 ±127%  cpuidle.C6.usage
   6880882 ± 13%     -46.1%    3706529        cpuidle.POLL.time
   1441739 ± 10%     -40.2%     862558        cpuidle.POLL.usage
  23339885 ±  9%     -23.6%   17822261        turbostat.C1
     18.49 ± 15%      -3.6       14.86        turbostat.C1%
  12673355 ± 20%     +66.3%   21078440 ±  3%  turbostat.C1E
     24.32 ± 28%     +42.5       66.84 ±  7%  turbostat.C1E%
   3765573 ±  3%     -84.0%     603105 ±128%  turbostat.C6
     43.48 ± 11%     -38.7        4.77 ±108%  turbostat.C6%
      3.73 ± 42%     -90.1%       0.37 ± 12%  turbostat.CPU%c6
  14813103 ±  2%     +16.9%   17318955        turbostat.IRQ
      1.59 ± 51%    -100.0%       0.00        turbostat.Pkg%pc2
     40.05            +1.7%      40.72        turbostat.RAMWatt
     59758            -1.1%      59093        proc-vmstat.nr_anon_pages
     48741            +1.0%      49245        proc-vmstat.nr_free_cma
      4606            -4.6%       4393        proc-vmstat.nr_inactive_anon
      6367            -2.2%       6227        proc-vmstat.nr_mapped
     34775            +2.1%      35519        proc-vmstat.nr_slab_unreclaimable
      4606            -4.6%       4393        proc-vmstat.nr_zone_inactive_anon
    428.50 ± 91%    +310.7%       1759 ± 63%  proc-vmstat.numa_pages_migrated
    578.25 ± 37%   +2906.7%      17386 ± 85%  proc-vmstat.numa_pte_updates
    253947           +13.1%     287290        proc-vmstat.pgfault
    717071            +5.7%     758253        proc-vmstat.pgfree
    428.50 ± 91%    +310.7%       1759 ± 63%  proc-vmstat.pgmigrate_success
    294.76 ± 12%     -18.2%     241.15 ±  8%  sched_debug.cfs_rq:/.load_avg.stddev
     18953 ± 23%     -39.3%      11499 ± 22%  sched_debug.cfs_rq:/.min_vruntime.avg
     12508 ± 15%     -50.6%       6174 ± 36%  sched_debug.cfs_rq:/.min_vruntime.min
      3003 ± 15%     +22.6%       3681 ±  6%  sched_debug.cfs_rq:/.min_vruntime.stddev
    250.34 ± 11%     -19.7%     200.99 ± 12%  sched_debug.cfs_rq:/.removed.load_avg.stddev
      3092 ± 23%     -37.1%       1945 ± 24%  sched_debug.cfs_rq:/.removed.runnable_sum.avg
     11538 ± 11%     -19.9%       9247 ± 12%  sched_debug.cfs_rq:/.removed.runnable_sum.stddev
     98.26 ±  5%     -19.7%      78.89 ± 24%  sched_debug.cfs_rq:/.removed.util_avg.stddev
      1.86 ± 15%     +26.4%       2.36 ± 11%  sched_debug.cfs_rq:/.runnable_load_avg.avg
      4.90 ±  4%      +9.1%       5.34 ±  4%  sched_debug.cfs_rq:/.runnable_load_avg.stddev
      2166 ± 16%     +24.6%       2699 ± 13%  sched_debug.cfs_rq:/.runnable_weight.avg
     18058 ±  2%     +15.6%      20869 ±  5%  sched_debug.cfs_rq:/.runnable_weight.max
      5486 ±  6%      +9.5%       6009 ±  4%  sched_debug.cfs_rq:/.runnable_weight.stddev
      2997 ± 15%     +23.0%       3687 ±  7%  sched_debug.cfs_rq:/.spread0.stddev
    256.16 ± 10%     +16.0%     297.04 ±  5%  sched_debug.cfs_rq:/.util_avg.stddev
     65601 ±  8%     -53.8%      30291        sched_debug.cpu.clock.avg
     65604 ±  8%     -53.8%      30295        sched_debug.cpu.clock.max
     65598 ±  8%     -53.8%      30285        sched_debug.cpu.clock.min
      1.71 ±  3%     +21.8%       2.09 ± 15%  sched_debug.cpu.clock.stddev
     65601 ±  8%     -53.8%      30291        sched_debug.cpu.clock_task.avg
     65604 ±  8%     -53.8%      30295        sched_debug.cpu.clock_task.max
     65598 ±  8%     -53.8%      30285        sched_debug.cpu.clock_task.min
      1.71 ±  3%     +21.8%       2.09 ± 15%  sched_debug.cpu.clock_task.stddev
      8296 ±  5%     -12.4%       7266        sched_debug.cpu.nr_load_updates.avg
      6306 ±  9%     -21.8%       4931 ±  9%  sched_debug.cpu.nr_load_updates.min
    339.25 ± 12%     -29.8%     238.25 ±  8%  sched_debug.cpu.nr_switches.min
      1141 ±  7%      +8.3%       1236 ±  5%  sched_debug.cpu.nr_switches.stddev
      4.91 ± 20%     -30.4%       3.42 ± 14%  sched_debug.cpu.nr_uninterruptible.stddev
     65598 ±  8%     -53.8%      30288        sched_debug.cpu_clk
     62902 ±  8%     -56.1%      27590        sched_debug.ktime
     66171 ±  8%     -53.4%      30858        sched_debug.sched_clk
     23.30 ± 14%     -36.4%      14.81        perf-stat.i.MPKI
 1.299e+09           +29.9%  1.687e+09        perf-stat.i.branch-instructions
      2.54 ±  9%      -0.9        1.64 ±  2%  perf-stat.i.branch-miss-rate%
  24240691 ±  2%      +8.8%   26384355 ±  2%  perf-stat.i.branch-misses
  32469998            +4.7%   34002480        perf-stat.i.cache-misses
 1.184e+08 ±  2%      +4.3%  1.236e+08        perf-stat.i.cache-references
    574477           -21.6%     450402        perf-stat.i.context-switches
      4.71 ±  2%     -21.4%       3.70        perf-stat.i.cpi
 3.026e+10            -1.3%  2.987e+10        perf-stat.i.cpu-cycles
     14.78 ± 12%     -13.5%      12.78 ±  6%  perf-stat.i.cpu-migrations
      1091 ±  3%     +21.9%       1331 ±  8%  perf-stat.i.cycles-between-cache-misses
      0.10 ± 28%      -0.1        0.01 ± 21%  perf-stat.i.dTLB-load-miss-rate%
    265935 ±  9%     -55.1%     119534 ±  5%  perf-stat.i.dTLB-load-misses
 1.798e+09           +12.4%  2.021e+09        perf-stat.i.dTLB-loads
      0.02 ± 23%      -0.0        0.00 ± 13%  perf-stat.i.dTLB-store-miss-rate%
     33528 ±  5%     -50.3%      16673 ±  2%  perf-stat.i.dTLB-store-misses
      5.94 ± 24%      -4.5        1.47 ± 21%  perf-stat.i.iTLB-load-miss-rate%
   8419599           -13.3%    7296814        perf-stat.i.iTLB-loads
 6.595e+09           +20.3%  7.936e+09        perf-stat.i.instructions
      0.22 ±  3%     +25.4%       0.28        perf-stat.i.ipc
      2351            -0.9%       2329        perf-stat.i.minor-faults
  10288903 ±  2%      -5.6%    9715506        perf-stat.i.node-load-misses
    765100           +11.5%     852708 ±  3%  perf-stat.i.node-loads
   3887667           -10.6%    3474607        perf-stat.i.node-store-misses
    754448 ±  5%     -15.2%     639938 ±  2%  perf-stat.i.node-stores
      2351            -0.9%       2329        perf-stat.i.page-faults
     17.96 ±  3%     -13.3%      15.57        perf-stat.overall.MPKI
      1.87            -0.3        1.56 ±  2%  perf-stat.overall.branch-miss-rate%
      4.59           -18.0%       3.76        perf-stat.overall.cpi
    932.33            -5.7%     878.73        perf-stat.overall.cycles-between-cache-misses
      0.01 ± 10%      -0.0        0.01 ±  4%  perf-stat.overall.dTLB-load-miss-rate%
      0.00 ±  6%      -0.0        0.00 ±  2%  perf-stat.overall.dTLB-store-miss-rate%
     67735 ±  6%     +83.2%     124066 ± 24%  perf-stat.overall.instructions-per-iTLB-miss
      0.22           +21.9%       0.27        perf-stat.overall.ipc
     93.08            -1.1       91.93        perf-stat.overall.node-load-miss-rate%
 1.286e+09           +30.0%  1.672e+09        perf-stat.ps.branch-instructions
  23999328 ±  2%      +9.0%   26157341 ±  2%  perf-stat.ps.branch-misses
  32137766            +4.9%   33699623        perf-stat.ps.cache-misses
 1.172e+08 ±  2%      +4.5%  1.225e+08        perf-stat.ps.cache-references
    568582           -21.5%     446385        perf-stat.ps.context-switches
 2.996e+10            -1.2%   2.96e+10        perf-stat.ps.cpu-cycles
    263523 ±  9%     -55.0%     118523 ±  5%  perf-stat.ps.dTLB-load-misses
  1.78e+09           +12.5%  2.003e+09        perf-stat.ps.dTLB-loads
     33223 ±  5%     -50.2%      16536 ±  2%  perf-stat.ps.dTLB-store-misses
   8333338           -13.2%    7231920        perf-stat.ps.iTLB-loads
 6.528e+09           +20.5%  7.866e+09        perf-stat.ps.instructions
      2328            -0.8%       2310        perf-stat.ps.minor-faults
  10183467 ±  2%      -5.4%    9628920        perf-stat.ps.node-load-misses
    757265           +11.6%     845120 ±  3%  perf-stat.ps.node-loads
   3847829           -10.5%    3443639        perf-stat.ps.node-store-misses
    746711 ±  5%     -15.1%     634243 ±  2%  perf-stat.ps.node-stores
      2328            -0.8%       2310        perf-stat.ps.page-faults
  6.55e+11           +38.9%  9.097e+11        perf-stat.total.instructions
     26715 ±  6%     +15.3%      30804 ±  3%  softirqs.CPU0.RCU
     25813 ±  3%     +17.5%      30331 ±  2%  softirqs.CPU1.RCU
     26114 ±  2%     +13.8%      29715 ±  4%  softirqs.CPU10.RCU
     16814 ±  6%     +17.4%      19743 ±  5%  softirqs.CPU10.SCHED
     38584           +23.0%      47474 ± 19%  softirqs.CPU10.TIMER
     25226 ±  6%     +24.3%      31347 ±  2%  softirqs.CPU11.RCU
     15728 ±  4%     +20.0%      18878 ±  2%  softirqs.CPU11.SCHED
     37200 ±  2%     +27.9%      47567 ± 21%  softirqs.CPU11.TIMER
     26744 ±  8%     +17.8%      31517 ±  4%  softirqs.CPU12.RCU
     25992 ±  2%     +26.8%      32949 ±  4%  softirqs.CPU13.RCU
     16769 ±  4%     +13.3%      18999 ±  2%  softirqs.CPU13.SCHED
     25183           +23.4%      31086 ±  6%  softirqs.CPU14.RCU
     37036 ±  2%     +24.4%      46059 ± 23%  softirqs.CPU14.TIMER
     24448 ±  2%     +21.7%      29750 ±  2%  softirqs.CPU15.RCU
     26602 ±  6%     +12.5%      29919        softirqs.CPU16.RCU
     37821 ±  3%      +9.5%      41426 ±  2%  softirqs.CPU16.TIMER
     28288 ±  2%     +19.4%      33785 ±  4%  softirqs.CPU18.RCU
     38075 ±  2%     +23.4%      46999 ± 21%  softirqs.CPU19.TIMER
     26162 ±  2%     +20.0%      31395 ±  2%  softirqs.CPU2.RCU
     16553 ±  4%     +15.5%      19112 ±  5%  softirqs.CPU2.SCHED
     37904 ±  2%     +18.5%      44898 ±  6%  softirqs.CPU2.TIMER
     27787 ±  6%     +15.0%      31953 ±  6%  softirqs.CPU20.RCU
     16907 ±  6%      +8.9%      18409 ±  5%  softirqs.CPU20.SCHED
     38188 ±  2%     +11.0%      42371 ±  3%  softirqs.CPU20.TIMER
     26947           +14.9%      30957 ±  6%  softirqs.CPU21.RCU
     37668           +10.8%      41744 ±  2%  softirqs.CPU21.TIMER
     27984 ±  6%     +17.3%      32821 ±  5%  softirqs.CPU22.RCU
     37415 ±  3%     +10.9%      41475        softirqs.CPU22.TIMER
     26636 ±  9%     +18.8%      31634 ±  6%  softirqs.CPU23.RCU
     26525 ±  2%     +22.9%      32606 ±  8%  softirqs.CPU24.RCU
     37040 ±  3%     +13.0%      41870 ±  2%  softirqs.CPU24.TIMER
     26441           +22.3%      32342        softirqs.CPU25.RCU
     37168 ±  4%     +12.8%      41929 ±  3%  softirqs.CPU25.TIMER
     26599 ±  5%     +21.0%      32182 ±  4%  softirqs.CPU26.RCU
     26510 ±  4%     +20.6%      31982 ±  2%  softirqs.CPU27.RCU
     16559 ±  4%     +13.4%      18775 ±  2%  softirqs.CPU27.SCHED
     37518           +10.7%      41515 ±  2%  softirqs.CPU28.TIMER
     25728 ± 10%     +27.6%      32830 ±  8%  softirqs.CPU29.RCU
     36817 ±  2%     +11.6%      41102        softirqs.CPU29.TIMER
     26141 ±  2%     +13.2%      29590 ±  2%  softirqs.CPU3.RCU
     17273 ±  4%     +20.2%      20755        softirqs.CPU3.SCHED
     26476 ±  3%     +18.2%      31287 ±  3%  softirqs.CPU30.RCU
     15921 ±  4%     +19.6%      19042 ± 10%  softirqs.CPU30.SCHED
     37032           +11.8%      41391 ±  2%  softirqs.CPU30.TIMER
     27534 ±  6%     +14.7%      31581 ±  4%  softirqs.CPU31.RCU
     36721           +13.4%      41633 ±  2%  softirqs.CPU31.TIMER
     25118 ±  3%     +23.7%      31072 ±  3%  softirqs.CPU32.RCU
     15548 ±  2%     +16.9%      18182 ±  6%  softirqs.CPU33.SCHED
     36529 ±  3%     +12.9%      41256        softirqs.CPU33.TIMER
     26802 ±  7%     +15.8%      31044 ±  3%  softirqs.CPU34.RCU
     16424 ±  3%     +12.1%      18409 ±  6%  softirqs.CPU34.SCHED
     27024 ±  6%     +20.6%      32603 ±  2%  softirqs.CPU35.RCU
     37345           +11.2%      41546        softirqs.CPU35.TIMER
     28719 ±  4%     +12.8%      32400 ±  2%  softirqs.CPU36.RCU
     36406 ±  2%      +9.0%      39681        softirqs.CPU36.TIMER
     30989 ±  7%     +23.5%      38266        softirqs.CPU38.RCU
     15275 ±  3%      +8.6%      16592        softirqs.CPU38.SCHED
     28633           +18.3%      33883        softirqs.CPU39.RCU
     26313           +17.6%      30942        softirqs.CPU4.RCU
     37319 ±  3%     +14.1%      42577        softirqs.CPU4.TIMER
     28360 ±  3%     +18.3%      33549        softirqs.CPU40.RCU
     26803 ±  3%     +22.6%      32857 ±  3%  softirqs.CPU41.RCU
     27921 ±  2%     +21.5%      33914 ±  5%  softirqs.CPU42.RCU
     27840 ±  2%     +23.2%      34308        softirqs.CPU43.RCU
     36332            +9.6%      39810 ±  3%  softirqs.CPU43.TIMER
     27988 ±  2%     +23.6%      34585        softirqs.CPU44.RCU
     26964 ±  4%     +23.0%      33161 ±  2%  softirqs.CPU45.RCU
     26647 ±  4%     +19.6%      31866 ±  5%  softirqs.CPU46.RCU
     26733 ±  6%     +27.6%      34120 ±  2%  softirqs.CPU48.RCU
     26780 ±  5%     +26.4%      33860 ±  2%  softirqs.CPU49.RCU
     24665 ±  8%     +24.2%      30646        softirqs.CPU5.RCU
     15335 ±  4%     +20.8%      18532 ±  5%  softirqs.CPU5.SCHED
     36270 ±  2%     +12.6%      40855 ±  2%  softirqs.CPU5.TIMER
     26971 ±  2%     +14.9%      30992 ±  6%  softirqs.CPU50.RCU
     27424 ±  2%     +21.4%      33300 ±  2%  softirqs.CPU51.RCU
     15504 ±  4%      +9.0%      16902 ±  2%  softirqs.CPU51.SCHED
     27430           +19.0%      32642 ±  6%  softirqs.CPU52.RCU
     27558           +19.3%      32887 ±  2%  softirqs.CPU53.RCU
     24817 ±  6%     +24.4%      30881 ±  2%  softirqs.CPU54.RCU
     26529 ±  2%     +13.3%      30048 ±  4%  softirqs.CPU55.RCU
     15454 ±  4%     +16.1%      17936 ±  5%  softirqs.CPU55.SCHED
     25606 ±  8%     +25.1%      32040        softirqs.CPU56.RCU
     26139 ±  4%     +24.1%      32437 ±  8%  softirqs.CPU57.RCU
     14558 ±  3%     +19.2%      17360 ±  3%  softirqs.CPU57.SCHED
     26083           +21.6%      31717 ±  3%  softirqs.CPU58.RCU
     15101 ±  3%     +13.3%      17105 ±  4%  softirqs.CPU58.SCHED
     14837 ±  6%     +11.9%      16598 ±  6%  softirqs.CPU59.SCHED
     35970 ±  3%     +12.0%      40295        softirqs.CPU59.TIMER
     26271           +16.8%      30681 ±  2%  softirqs.CPU6.RCU
     15985 ±  6%     +18.3%      18914 ±  8%  softirqs.CPU6.SCHED
     37813 ±  2%     +10.9%      41946        softirqs.CPU6.TIMER
     24423 ±  3%     +24.7%      30453 ±  6%  softirqs.CPU60.RCU
     14485 ±  2%     +20.9%      17519 ±  4%  softirqs.CPU60.SCHED
     36185 ±  2%     +11.9%      40494        softirqs.CPU60.TIMER
     23959 ±  2%     +24.8%      29893        softirqs.CPU61.RCU
     14903 ±  6%     +12.5%      16767 ±  3%  softirqs.CPU61.SCHED
     24809 ±  2%     +18.0%      29264        softirqs.CPU62.RCU
     14321 ±  2%     +18.1%      16916 ±  2%  softirqs.CPU62.SCHED
     36576            +9.0%      39877        softirqs.CPU62.TIMER
     23487 ±  3%     +25.9%      29568 ±  2%  softirqs.CPU63.RCU
     13582 ±  2%     +25.3%      17014 ±  3%  softirqs.CPU63.SCHED
     35911           +12.2%      40280        softirqs.CPU63.TIMER
     25668 ±  4%     +18.4%      30390 ±  3%  softirqs.CPU64.RCU
     14666 ±  4%     +13.6%      16660 ±  5%  softirqs.CPU64.SCHED
     24078           +29.4%      31156 ±  5%  softirqs.CPU65.RCU
     14177 ±  8%     +20.5%      17090 ±  2%  softirqs.CPU65.SCHED
     35962 ±  3%     +13.1%      40661        softirqs.CPU65.TIMER
     23043 ±  7%     +31.2%      30227 ±  8%  softirqs.CPU66.RCU
     36661           +10.5%      40517 ±  2%  softirqs.CPU66.TIMER
     25069 ±  4%     +14.8%      28791 ±  2%  softirqs.CPU67.RCU
     13930 ±  3%     +24.3%      17313 ±  6%  softirqs.CPU67.SCHED
     35788           +13.1%      40494        softirqs.CPU67.TIMER
     23620 ±  3%     +20.5%      28463        softirqs.CPU68.RCU
     13693 ±  4%     +26.3%      17294 ±  5%  softirqs.CPU68.SCHED
     36756           +11.4%      40930        softirqs.CPU68.TIMER
     24380 ±  2%     +19.4%      29104 ±  6%  softirqs.CPU69.RCU
     14512 ±  7%     +23.9%      17978 ±  8%  softirqs.CPU69.SCHED
     25915           +18.1%      30605        softirqs.CPU7.RCU
     15855 ±  2%     +16.6%      18490        softirqs.CPU7.SCHED
     36892 ±  2%     +12.5%      41512 ±  2%  softirqs.CPU7.TIMER
     24291           +20.2%      29207        softirqs.CPU70.RCU
     14647 ±  7%     +20.2%      17609        softirqs.CPU70.SCHED
     36029 ±  2%     +12.8%      40633        softirqs.CPU70.TIMER
     24761 ±  6%     +19.6%      29619        softirqs.CPU71.RCU
     36389           +10.2%      40114        softirqs.CPU71.TIMER
     26860 ±  2%     +13.7%      30535 ±  4%  softirqs.CPU8.RCU
     16226 ±  2%     +20.1%      19492 ±  4%  softirqs.CPU8.SCHED
     37997           +11.7%      42458 ±  2%  softirqs.CPU8.TIMER
     25499 ±  2%     +17.6%      29981        softirqs.CPU9.RCU
     17382 ±  8%     +11.0%      19298 ±  6%  softirqs.CPU9.SCHED
     36979            +9.1%      40348        softirqs.CPU9.TIMER
   1901918           +19.3%    2268164        softirqs.RCU
   1148813           +10.9%    1273763        softirqs.SCHED
   2693395           +10.3%    2970253        softirqs.TIMER
     93123 ±  2%     +20.6%     112353        interrupts.CAL:Function_call_interrupts
      1307           +21.0%       1581        interrupts.CPU0.CAL:Function_call_interrupts
    196590 ±  3%     +18.0%     231965        interrupts.CPU0.LOC:Local_timer_interrupts
      1982 ± 18%     -47.7%       1037 ± 54%  interrupts.CPU0.NMI:Non-maskable_interrupts
      1982 ± 18%     -47.7%       1037 ± 54%  interrupts.CPU0.PMI:Performance_monitoring_interrupts
      1311 ±  2%     +20.8%       1584        interrupts.CPU1.CAL:Function_call_interrupts
    196507 ±  3%     +17.9%     231706        interrupts.CPU1.LOC:Local_timer_interrupts
      1315 ±  2%     +20.3%       1581        interrupts.CPU10.CAL:Function_call_interrupts
    196466 ±  2%     +17.7%     231175        interrupts.CPU10.LOC:Local_timer_interrupts
      1278 ±  6%     +25.0%       1598 ±  2%  interrupts.CPU11.CAL:Function_call_interrupts
    196467 ±  2%     +18.1%     232064        interrupts.CPU11.LOC:Local_timer_interrupts
      2906 ± 34%     +75.3%       5096 ±  7%  interrupts.CPU11.RES:Rescheduling_interrupts
      1312 ±  2%     +16.6%       1530 ±  4%  interrupts.CPU12.CAL:Function_call_interrupts
    196661 ±  3%     +17.9%     231954        interrupts.CPU12.LOC:Local_timer_interrupts
    196407 ±  3%     +17.7%     231133        interrupts.CPU13.LOC:Local_timer_interrupts
      3966 ± 15%     +32.6%       5258 ±  7%  interrupts.CPU13.RES:Rescheduling_interrupts
    196446 ±  3%     +18.0%     231766        interrupts.CPU14.LOC:Local_timer_interrupts
    196307 ±  3%     +18.0%     231649        interrupts.CPU15.LOC:Local_timer_interrupts
      1311 ±  2%     +18.0%       1547 ±  3%  interrupts.CPU16.CAL:Function_call_interrupts
    196278 ±  3%     +17.7%     230984        interrupts.CPU16.LOC:Local_timer_interrupts
      2535 ± 36%     -78.5%     545.25 ± 51%  interrupts.CPU16.NMI:Non-maskable_interrupts
      2535 ± 36%     -78.5%     545.25 ± 51%  interrupts.CPU16.PMI:Performance_monitoring_interrupts
      1220 ± 11%     +23.2%       1502 ±  6%  interrupts.CPU17.CAL:Function_call_interrupts
    196493 ±  3%     +17.5%     230974        interrupts.CPU17.LOC:Local_timer_interrupts
      1306           +21.1%       1582        interrupts.CPU18.CAL:Function_call_interrupts
    196318 ±  2%     +18.0%     231690        interrupts.CPU18.LOC:Local_timer_interrupts
      1329           +18.7%       1578        interrupts.CPU19.CAL:Function_call_interrupts
    197092 ±  2%     +17.7%     231886        interrupts.CPU19.LOC:Local_timer_interrupts
      1310 ±  2%     +20.4%       1577        interrupts.CPU2.CAL:Function_call_interrupts
    196499 ±  3%     +17.8%     231564        interrupts.CPU2.LOC:Local_timer_interrupts
      1328 ±  4%     +20.5%       1601 ±  3%  interrupts.CPU20.CAL:Function_call_interrupts
    197064 ±  2%     +17.6%     231838        interrupts.CPU20.LOC:Local_timer_interrupts
      1311 ±  2%     +20.4%       1578        interrupts.CPU21.CAL:Function_call_interrupts
    196901 ±  2%     +17.7%     231704        interrupts.CPU21.LOC:Local_timer_interrupts
      1313 ±  2%     +20.6%       1583        interrupts.CPU22.CAL:Function_call_interrupts
    196919 ±  2%     +17.5%     231471        interrupts.CPU22.LOC:Local_timer_interrupts
      1309 ±  2%     +17.7%       1540 ±  5%  interrupts.CPU23.CAL:Function_call_interrupts
    196812 ±  2%     +17.7%     231745        interrupts.CPU23.LOC:Local_timer_interrupts
      1311 ±  2%     +20.6%       1582        interrupts.CPU24.CAL:Function_call_interrupts
    196729 ±  2%     +17.9%     231969        interrupts.CPU24.LOC:Local_timer_interrupts
      1311 ±  2%     +20.4%       1579        interrupts.CPU25.CAL:Function_call_interrupts
    196696 ±  2%     +17.8%     231715        interrupts.CPU25.LOC:Local_timer_interrupts
      1152 ± 74%    +184.9%       3281 ± 11%  interrupts.CPU25.NMI:Non-maskable_interrupts
      1152 ± 74%    +184.9%       3281 ± 11%  interrupts.CPU25.PMI:Performance_monitoring_interrupts
      1311 ±  2%     +20.6%       1581        interrupts.CPU26.CAL:Function_call_interrupts
    197080 ±  2%     +17.6%     231696        interrupts.CPU26.LOC:Local_timer_interrupts
      1909 ± 42%     +67.9%       3207 ± 10%  interrupts.CPU26.NMI:Non-maskable_interrupts
      1909 ± 42%     +67.9%       3207 ± 10%  interrupts.CPU26.PMI:Performance_monitoring_interrupts
      1310 ±  2%     +20.6%       1580        interrupts.CPU27.CAL:Function_call_interrupts
    196958 ±  2%     +17.6%     231702        interrupts.CPU27.LOC:Local_timer_interrupts
      1309 ±  2%     +20.7%       1580        interrupts.CPU28.CAL:Function_call_interrupts
    196575 ±  2%     +17.9%     231714        interrupts.CPU28.LOC:Local_timer_interrupts
      1308 ±  2%     +20.9%       1582        interrupts.CPU29.CAL:Function_call_interrupts
    196943 ±  2%     +17.3%     230984        interrupts.CPU29.LOC:Local_timer_interrupts
      1310 ±  2%     +21.3%       1588        interrupts.CPU3.CAL:Function_call_interrupts
    196420 ±  2%     +16.9%     229682        interrupts.CPU3.LOC:Local_timer_interrupts
      2937 ± 21%     -39.9%       1766 ± 23%  interrupts.CPU3.NMI:Non-maskable_interrupts
      2937 ± 21%     -39.9%       1766 ± 23%  interrupts.CPU3.PMI:Performance_monitoring_interrupts
      4820 ± 15%     +88.0%       9063 ±  8%  interrupts.CPU3.RES:Rescheduling_interrupts
      1310 ±  2%     +20.8%       1582        interrupts.CPU30.CAL:Function_call_interrupts
    196831 ±  2%     +17.7%     231691        interrupts.CPU30.LOC:Local_timer_interrupts
      1209 ±113%    +149.6%       3018 ± 34%  interrupts.CPU30.NMI:Non-maskable_interrupts
      1209 ±113%    +149.6%       3018 ± 34%  interrupts.CPU30.PMI:Performance_monitoring_interrupts
      1304 ±  2%     +23.0%       1604        interrupts.CPU31.CAL:Function_call_interrupts
    196479 ±  2%     +18.0%     231820        interrupts.CPU31.LOC:Local_timer_interrupts
      1419 ± 73%     +81.5%       2575 ± 26%  interrupts.CPU31.NMI:Non-maskable_interrupts
      1419 ± 73%     +81.5%       2575 ± 26%  interrupts.CPU31.PMI:Performance_monitoring_interrupts
      1309 ±  2%     +20.9%       1583        interrupts.CPU32.CAL:Function_call_interrupts
    196861 ±  2%     +17.7%     231732        interrupts.CPU32.LOC:Local_timer_interrupts
      1159 ± 23%     +36.6%       1583        interrupts.CPU33.CAL:Function_call_interrupts
    196346 ±  2%     +18.0%     231694        interrupts.CPU33.LOC:Local_timer_interrupts
      1310 ±  2%     +20.8%       1583        interrupts.CPU34.CAL:Function_call_interrupts
    195339           +18.6%     231743        interrupts.CPU34.LOC:Local_timer_interrupts
      1284 ±  2%     +21.7%       1563        interrupts.CPU35.CAL:Function_call_interrupts
    195354           +18.6%     231781        interrupts.CPU35.LOC:Local_timer_interrupts
      1064 ± 25%     +45.6%       1549 ±  4%  interrupts.CPU36.CAL:Function_call_interrupts
    196631 ±  2%     +17.6%     231216        interrupts.CPU36.LOC:Local_timer_interrupts
      2113 ± 33%     -64.2%     756.50 ± 29%  interrupts.CPU36.NMI:Non-maskable_interrupts
      2113 ± 33%     -64.2%     756.50 ± 29%  interrupts.CPU36.PMI:Performance_monitoring_interrupts
      1203 ±  9%     +29.8%       1561 ±  2%  interrupts.CPU37.CAL:Function_call_interrupts
    196745 ±  3%     +17.6%     231316        interrupts.CPU37.LOC:Local_timer_interrupts
      1302 ±  2%     +21.4%       1580        interrupts.CPU38.CAL:Function_call_interrupts
    196339 ±  3%     +17.6%     230859        interrupts.CPU38.LOC:Local_timer_interrupts
      1290 ±  3%     +22.4%       1580        interrupts.CPU39.CAL:Function_call_interrupts
    196222 ±  3%     +16.9%     229415        interrupts.CPU39.LOC:Local_timer_interrupts
      1302           +19.1%       1551 ±  4%  interrupts.CPU4.CAL:Function_call_interrupts
    196114 ±  3%     +18.2%     231711        interrupts.CPU4.LOC:Local_timer_interrupts
      1282 ±  2%     +23.2%       1579        interrupts.CPU40.CAL:Function_call_interrupts
    196347 ±  3%     +17.2%     230205        interrupts.CPU40.LOC:Local_timer_interrupts
      1295           +21.9%       1579        interrupts.CPU41.CAL:Function_call_interrupts
    196386 ±  2%     +17.6%     230956        interrupts.CPU41.LOC:Local_timer_interrupts
      2528 ± 16%     -60.4%       1001 ± 26%  interrupts.CPU41.NMI:Non-maskable_interrupts
      2528 ± 16%     -60.4%       1001 ± 26%  interrupts.CPU41.PMI:Performance_monitoring_interrupts
      5041 ± 17%     -55.2%       2256 ± 35%  interrupts.CPU41.RES:Rescheduling_interrupts
    196437 ±  2%     +17.7%     231202        interrupts.CPU42.LOC:Local_timer_interrupts
      2533 ± 30%     -65.8%     867.50 ± 67%  interrupts.CPU42.NMI:Non-maskable_interrupts
      2533 ± 30%     -65.8%     867.50 ± 67%  interrupts.CPU42.PMI:Performance_monitoring_interrupts
      1287 ±  4%     +22.8%       1580        interrupts.CPU43.CAL:Function_call_interrupts
    196441 ±  2%     +17.5%     230898        interrupts.CPU43.LOC:Local_timer_interrupts
      1184 ± 15%     +33.3%       1578        interrupts.CPU44.CAL:Function_call_interrupts
    196438 ±  2%     +18.0%     231740        interrupts.CPU44.LOC:Local_timer_interrupts
      2587 ± 38%     -63.2%     953.00 ± 57%  interrupts.CPU44.NMI:Non-maskable_interrupts
      2587 ± 38%     -63.2%     953.00 ± 57%  interrupts.CPU44.PMI:Performance_monitoring_interrupts
      1307 ±  2%     +20.7%       1579        interrupts.CPU45.CAL:Function_call_interrupts
    196461 ±  2%     +17.2%     230176        interrupts.CPU45.LOC:Local_timer_interrupts
      1222 ± 10%     +29.1%       1578        interrupts.CPU46.CAL:Function_call_interrupts
    196498 ±  2%     +17.5%     230967        interrupts.CPU46.LOC:Local_timer_interrupts
      2619 ± 33%     -63.7%     951.50 ±106%  interrupts.CPU46.NMI:Non-maskable_interrupts
      2619 ± 33%     -63.7%     951.50 ±106%  interrupts.CPU46.PMI:Performance_monitoring_interrupts
      1305 ±  2%     +21.0%       1580        interrupts.CPU47.CAL:Function_call_interrupts
    196734 ±  3%     +17.4%     230968        interrupts.CPU47.LOC:Local_timer_interrupts
      1325 ±  2%     +19.0%       1577        interrupts.CPU48.CAL:Function_call_interrupts
    196228 ±  3%     +18.1%     231730        interrupts.CPU48.LOC:Local_timer_interrupts
      1331 ±  3%     +18.5%       1578        interrupts.CPU49.CAL:Function_call_interrupts
    196553 ±  3%     +17.5%     230948        interrupts.CPU49.LOC:Local_timer_interrupts
      1306 ±  2%     +21.5%       1586        interrupts.CPU5.CAL:Function_call_interrupts
    196436 ±  2%     +17.6%     230955        interrupts.CPU5.LOC:Local_timer_interrupts
      2656 ± 16%     -52.5%       1260 ± 56%  interrupts.CPU5.NMI:Non-maskable_interrupts
      2656 ± 16%     -52.5%       1260 ± 56%  interrupts.CPU5.PMI:Performance_monitoring_interrupts
      1307 ±  2%     +20.7%       1578        interrupts.CPU50.CAL:Function_call_interrupts
    196306 ±  3%     +17.7%     231081        interrupts.CPU50.LOC:Local_timer_interrupts
      1312           +20.3%       1578        interrupts.CPU51.CAL:Function_call_interrupts
    196454 ±  2%     +18.0%     231823        interrupts.CPU51.LOC:Local_timer_interrupts
      1316 ±  2%     +19.9%       1578        interrupts.CPU52.CAL:Function_call_interrupts
    196419 ±  2%     +17.7%     231185        interrupts.CPU52.LOC:Local_timer_interrupts
      2242 ± 23%     -74.4%     573.25 ± 57%  interrupts.CPU52.NMI:Non-maskable_interrupts
      2242 ± 23%     -74.4%     573.25 ± 57%  interrupts.CPU52.PMI:Performance_monitoring_interrupts
      1308           +17.3%       1534 ±  3%  interrupts.CPU53.CAL:Function_call_interrupts
    196425 ±  2%     +17.9%     231631        interrupts.CPU53.LOC:Local_timer_interrupts
      1227 ± 13%     +28.4%       1575        interrupts.CPU54.CAL:Function_call_interrupts
    196179 ±  2%     +18.0%     231486        interrupts.CPU54.LOC:Local_timer_interrupts
      1288 ±  5%     +19.5%       1539 ±  5%  interrupts.CPU55.CAL:Function_call_interrupts
    196633 ±  2%     +17.8%     231726        interrupts.CPU55.LOC:Local_timer_interrupts
      1308 ±  2%     +21.0%       1582        interrupts.CPU56.CAL:Function_call_interrupts
    196902 ±  2%     +17.7%     231729        interrupts.CPU56.LOC:Local_timer_interrupts
    196827 ±  2%     +17.8%     231949        interrupts.CPU57.LOC:Local_timer_interrupts
      2311 ± 41%     +73.9%       4018 ± 29%  interrupts.CPU57.RES:Rescheduling_interrupts
      1314 ±  2%     +18.3%       1554        interrupts.CPU58.CAL:Function_call_interrupts
    196927 ±  2%     +17.7%     231731        interrupts.CPU58.LOC:Local_timer_interrupts
      1310 ±  2%     +20.7%       1580        interrupts.CPU59.CAL:Function_call_interrupts
    196866 ±  2%     +17.7%     231802        interrupts.CPU59.LOC:Local_timer_interrupts
      1311 ±  2%     +20.9%       1585        interrupts.CPU6.CAL:Function_call_interrupts
    196416 ±  2%     +17.6%     230968        interrupts.CPU6.LOC:Local_timer_interrupts
      2423 ± 31%     -61.9%     922.25 ± 88%  interrupts.CPU6.NMI:Non-maskable_interrupts
      2423 ± 31%     -61.9%     922.25 ± 88%  interrupts.CPU6.PMI:Performance_monitoring_interrupts
      1314 ±  2%     +20.3%       1580        interrupts.CPU60.CAL:Function_call_interrupts
    196951 ±  2%     +17.7%     231714        interrupts.CPU60.LOC:Local_timer_interrupts
    196924 ±  2%     +17.7%     231716        interrupts.CPU61.LOC:Local_timer_interrupts
    953.50 ± 81%    +189.0%       2755 ± 31%  interrupts.CPU61.NMI:Non-maskable_interrupts
    953.50 ± 81%    +189.0%       2755 ± 31%  interrupts.CPU61.PMI:Performance_monitoring_interrupts
      1314 ±  2%     +19.9%       1576        interrupts.CPU62.CAL:Function_call_interrupts
    196678 ±  2%     +17.8%     231687        interrupts.CPU62.LOC:Local_timer_interrupts
      1277 ±  5%     +23.6%       1577        interrupts.CPU63.CAL:Function_call_interrupts
    196543 ±  2%     +17.9%     231706        interrupts.CPU63.LOC:Local_timer_interrupts
      1869 ± 43%    +108.0%       3888 ± 28%  interrupts.CPU63.RES:Rescheduling_interrupts
      1310 ±  2%     +20.6%       1580        interrupts.CPU64.CAL:Function_call_interrupts
    197006 ±  2%     +17.6%     231724        interrupts.CPU64.LOC:Local_timer_interrupts
      1312 ±  2%     +20.1%       1575        interrupts.CPU65.CAL:Function_call_interrupts
    196073 ±  2%     +18.2%     231756        interrupts.CPU65.LOC:Local_timer_interrupts
      1143 ± 95%    +147.4%       2828 ± 24%  interrupts.CPU65.NMI:Non-maskable_interrupts
      1143 ± 95%    +147.4%       2828 ± 24%  interrupts.CPU65.PMI:Performance_monitoring_interrupts
      1311 ±  2%     +17.4%       1540 ±  4%  interrupts.CPU66.CAL:Function_call_interrupts
    196845 ±  2%     +17.8%     231921        interrupts.CPU66.LOC:Local_timer_interrupts
      1311 ±  2%     +20.5%       1580        interrupts.CPU67.CAL:Function_call_interrupts
    195708 ±  2%     +18.4%     231743        interrupts.CPU67.LOC:Local_timer_interrupts
      1310 ±  2%     +20.7%       1581        interrupts.CPU68.CAL:Function_call_interrupts
    196837 ±  2%     +17.8%     231795        interrupts.CPU68.LOC:Local_timer_interrupts
      1516 ± 59%    +137.0%       3593 ± 26%  interrupts.CPU68.RES:Rescheduling_interrupts
      1311 ±  2%     +20.5%       1581        interrupts.CPU69.CAL:Function_call_interrupts
    196838 ±  2%     +17.8%     231870        interrupts.CPU69.LOC:Local_timer_interrupts
      1306 ±  2%     +21.5%       1586        interrupts.CPU7.CAL:Function_call_interrupts
    196395 ±  2%     +17.7%     231082        interrupts.CPU7.LOC:Local_timer_interrupts
      3299 ± 36%     +68.7%       5567 ±  8%  interrupts.CPU7.RES:Rescheduling_interrupts
      1310           +20.7%       1581        interrupts.CPU70.CAL:Function_call_interrupts
    195366           +18.6%     231713        interrupts.CPU70.LOC:Local_timer_interrupts
      1308           +18.1%       1545 ±  3%  interrupts.CPU71.CAL:Function_call_interrupts
    196019 ±  2%     +18.2%     231692        interrupts.CPU71.LOC:Local_timer_interrupts
      1280 ±  3%     +23.9%       1586        interrupts.CPU8.CAL:Function_call_interrupts
    196421 ±  2%     +18.0%     231850        interrupts.CPU8.LOC:Local_timer_interrupts
      2724 ± 33%     -58.3%       1137 ± 62%  interrupts.CPU8.NMI:Non-maskable_interrupts
      2724 ± 33%     -58.3%       1137 ± 62%  interrupts.CPU8.PMI:Performance_monitoring_interrupts
      3448 ± 10%     +81.5%       6257 ± 16%  interrupts.CPU8.RES:Rescheduling_interrupts
      1300           +18.6%       1541 ±  5%  interrupts.CPU9.CAL:Function_call_interrupts
    196415 ±  2%     +17.2%     230179        interrupts.CPU9.LOC:Local_timer_interrupts
  14149246 ±  2%     +17.8%   16663738        interrupts.LOC:Local_timer_interrupts
      5.72 ±  6%      -1.0        4.73 ±  6%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.finish_wait.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot
      5.62 ±  7%      -1.0        4.65 ±  6%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.finish_wait.btrfs_tree_read_lock.btrfs_read_lock_root_node
      1.32 ± 36%      -0.9        0.41 ± 57%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath.queued_write_lock_slowpath.btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot
     13.67            -0.8       12.82 ±  5%  perf-profile.calltrace.cycles-pp.btrfs_csum_file_blocks.btrfs_log_changed_extents.btrfs_log_inode.btrfs_log_inode_parent.btrfs_log_dentry_safe
      5.44            -0.6        4.88 ±  6%  perf-profile.calltrace.cycles-pp.btrfs_lookup_csum.btrfs_csum_file_blocks.btrfs_log_changed_extents.btrfs_log_inode.btrfs_log_inode_parent
      5.43            -0.6        4.87 ±  6%  perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_lookup_csum.btrfs_csum_file_blocks.btrfs_log_changed_extents.btrfs_log_inode
      4.80 ±  3%      -0.5        4.26 ±  5%  perf-profile.calltrace.cycles-pp.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items.btrfs_log_inode.btrfs_log_inode_parent
      4.78 ±  3%      -0.5        4.24 ±  6%  perf-profile.calltrace.cycles-pp.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items.btrfs_log_inode
      2.84 ±  9%      -0.5        2.33 ±  7%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.finish_wait.btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot
      2.78 ±  9%      -0.5        2.29 ±  7%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.finish_wait.btrfs_tree_lock.btrfs_lock_root_node
      4.64            -0.4        4.21 ±  5%  perf-profile.calltrace.cycles-pp.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_csum.btrfs_csum_file_blocks.btrfs_log_changed_extents
      4.62            -0.4        4.19 ±  5%  perf-profile.calltrace.cycles-pp.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_csum.btrfs_csum_file_blocks
      2.22 ±  3%      -0.4        1.80 ±  7%  perf-profile.calltrace.cycles-pp.finish_wait.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items
      1.51 ±  3%      -0.4        1.12 ± 26%  perf-profile.calltrace.cycles-pp.queued_write_lock_slowpath.btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items
      0.62 ±  4%      -0.4        0.26 ±100%  perf-profile.calltrace.cycles-pp.queued_read_lock_slowpath.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent
      0.69 ±  6%      -0.3        0.40 ± 57%  perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary
      2.47            -0.3        2.21 ±  4%  perf-profile.calltrace.cycles-pp.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items.btrfs_csum_file_blocks
      2.48            -0.3        2.22 ±  4%  perf-profile.calltrace.cycles-pp.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items.btrfs_csum_file_blocks.btrfs_log_changed_extents
      0.67 ±  4%      -0.2        0.44 ± 58%  perf-profile.calltrace.cycles-pp.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_log_all_xattrs.btrfs_log_inode.btrfs_log_inode_parent
      0.65 ±  4%      -0.2        0.42 ± 58%  perf-profile.calltrace.cycles-pp.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_log_all_xattrs.btrfs_log_inode
      0.66 ±  5%      -0.2        0.44 ± 57%  perf-profile.calltrace.cycles-pp.queued_write_lock_slowpath.btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent
      0.82 ±  7%      -0.2        0.62 ±  8%  perf-profile.calltrace.cycles-pp.add_pending_csums.btrfs_finish_ordered_io.normal_work_helper.process_one_work.worker_thread
      1.44            -0.2        1.25 ±  6%  perf-profile.calltrace.cycles-pp.finish_wait.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_csum
      0.82 ±  7%      -0.2        0.62 ±  8%  perf-profile.calltrace.cycles-pp.btrfs_csum_file_blocks.add_pending_csums.btrfs_finish_ordered_io.normal_work_helper.process_one_work
      1.93 ±  2%      -0.2        1.75 ±  7%  perf-profile.calltrace.cycles-pp.finish_wait.btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items
      0.71 ±  6%      -0.2        0.54 ±  3%  perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      1.53 ±  2%      -0.1        1.40 ±  4%  perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.btrfs_fdatawrite_range.start_ordered_ops.btrfs_sync_file
      1.53 ±  2%      -0.1        1.40 ±  4%  perf-profile.calltrace.cycles-pp.extent_writepages.do_writepages.__filemap_fdatawrite_range.btrfs_fdatawrite_range.start_ordered_ops
      1.55 ±  2%      -0.1        1.42 ±  4%  perf-profile.calltrace.cycles-pp.start_ordered_ops.btrfs_sync_file.btrfs_file_write_iter.new_sync_write.vfs_write
      1.54 ±  2%      -0.1        1.41 ±  4%  perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.btrfs_fdatawrite_range.start_ordered_ops.btrfs_sync_file.btrfs_file_write_iter
      1.54 ±  2%      -0.1        1.41 ±  4%  perf-profile.calltrace.cycles-pp.btrfs_fdatawrite_range.start_ordered_ops.btrfs_sync_file.btrfs_file_write_iter.new_sync_write
      0.86 ±  3%      -0.1        0.74 ±  4%  perf-profile.calltrace.cycles-pp.btrfs_cow_block.btrfs_search_slot.btrfs_insert_empty_items.btrfs_log_inode.btrfs_log_inode_parent
      0.67 ±  9%      -0.1        0.55 ±  7%  perf-profile.calltrace.cycles-pp.btrfs_run_delayed_refs.flush_space.btrfs_async_reclaim_metadata_space.process_one_work.worker_thread
      0.67 ±  9%      -0.1        0.55 ±  7%  perf-profile.calltrace.cycles-pp.__btrfs_run_delayed_refs.btrfs_run_delayed_refs.flush_space.btrfs_async_reclaim_metadata_space.process_one_work
      1.03 ±  5%      -0.1        0.91 ±  7%  perf-profile.calltrace.cycles-pp.extent_write_cache_pages.extent_writepages.do_writepages.__filemap_fdatawrite_range.btrfs_fdatawrite_range
      0.85 ±  3%      -0.1        0.73 ±  4%  perf-profile.calltrace.cycles-pp.__btrfs_cow_block.btrfs_cow_block.btrfs_search_slot.btrfs_insert_empty_items.btrfs_log_inode
      0.75 ±  2%      -0.1        0.64        perf-profile.calltrace.cycles-pp.btrfs_write_marked_extents.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter.new_sync_write
      0.74 ±  2%      -0.1        0.63 ±  2%  perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter
      0.74 ±  2%      -0.1        0.63 ±  2%  perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_sync_log.btrfs_sync_file
      0.74 ±  2%      -0.1        0.63 ±  2%  perf-profile.calltrace.cycles-pp.btree_write_cache_pages.do_writepages.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_sync_log
      1.24 ±  5%      +0.1        1.39 ±  2%  perf-profile.calltrace.cycles-pp.osq_lock.__mutex_lock.wait_log_commit.btrfs_sync_log.btrfs_sync_file
      1.38 ±  5%      +0.2        1.56 ±  4%  perf-profile.calltrace.cycles-pp.__mutex_lock.wait_log_commit.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter
      1.59 ±  6%      +0.2        1.80 ±  4%  perf-profile.calltrace.cycles-pp.wait_log_commit.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter.new_sync_write
      0.76 ±  7%      +0.2        0.99 ±  4%  perf-profile.calltrace.cycles-pp.setup_items_for_insert.__btrfs_drop_extents.insert_reserved_file_extent.btrfs_finish_ordered_io.normal_work_helper
      0.83 ±  3%      +0.2        1.08 ±  4%  perf-profile.calltrace.cycles-pp.setup_items_for_insert.__btrfs_drop_extents.btrfs_log_changed_extents.btrfs_log_inode.btrfs_log_inode_parent
      0.61 ±  8%      +0.3        0.88 ±  5%  perf-profile.calltrace.cycles-pp.btrfs_unlock_up_safe.setup_items_for_insert.__btrfs_drop_extents.insert_reserved_file_extent.btrfs_finish_ordered_io
      0.59 ±  8%      +0.3        0.85 ±  5%  perf-profile.calltrace.cycles-pp.__wake_up_common_lock.btrfs_unlock_up_safe.setup_items_for_insert.__btrfs_drop_extents.insert_reserved_file_extent
      1.78 ±  7%      +0.3        2.06 ±  2%  perf-profile.calltrace.cycles-pp.prepare_to_wait_event.btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent
      0.68 ±  3%      +0.3        0.96 ±  4%  perf-profile.calltrace.cycles-pp.btrfs_unlock_up_safe.setup_items_for_insert.__btrfs_drop_extents.btrfs_log_changed_extents.btrfs_log_inode
      0.66 ±  3%      +0.3        0.93 ±  4%  perf-profile.calltrace.cycles-pp.__wake_up_common_lock.btrfs_unlock_up_safe.setup_items_for_insert.__btrfs_drop_extents.btrfs_log_changed_extents
      0.55 ±  2%      +0.3        0.88 ±  3%  perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.unlock_up.btrfs_search_slot.btrfs_insert_empty_items
      2.89 ±  3%      +0.4        3.25 ±  2%  perf-profile.calltrace.cycles-pp.prepare_to_wait_event.btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items
      0.74            +0.4        1.13 ±  2%  perf-profile.calltrace.cycles-pp.unlock_up.btrfs_search_slot.btrfs_insert_empty_items.btrfs_log_inode.btrfs_log_inode_parent
      0.71 ±  2%      +0.4        1.12        perf-profile.calltrace.cycles-pp.__wake_up_common_lock.unlock_up.btrfs_search_slot.btrfs_insert_empty_items.btrfs_log_inode
      0.27 ±100%      +0.4        0.71 ± 14%  perf-profile.calltrace.cycles-pp.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle
      0.39 ± 57%      +0.5        0.85 ±  2%  perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.unlock_up.btrfs_search_slot
      0.94 ±  8%      +0.5        1.48 ± 14%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle
      0.00            +0.5        0.54 ±  4%  perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common
      0.00            +0.5        0.54 ±  4%  perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      0.13 ±173%      +0.6        0.73 ± 11%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
      0.00            +0.6        0.64 ± 11%  perf-profile.calltrace.cycles-pp.__save_stack_trace.save_stack_trace_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair
      4.37            +0.6        5.01        perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.prepare_to_wait_event.btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot
      0.00            +0.7        0.67 ± 11%  perf-profile.calltrace.cycles-pp.save_stack_trace_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.ttwu_do_activate
      0.00            +0.7        0.68 ± 35%  perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.btrfs_unlock_up_safe
      0.00            +0.7        0.72 ± 34%  perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.btrfs_unlock_up_safe.setup_items_for_insert
      1.56 ±  7%      +0.7        2.30 ± 13%  perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry
      1.69 ±  7%      +0.8        2.47 ± 13%  perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary
      4.09 ±  5%      +0.8        4.90        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.prepare_to_wait_event.btrfs_tree_lock.btrfs_lock_root_node
      0.00            +0.8        0.81 ±  3%  perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.unlock_up
      0.00            +1.0        0.97 ± 12%  perf-profile.calltrace.cycles-pp.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.ttwu_do_activate.sched_ttwu_pending
      0.00            +1.0        1.00 ± 21%  perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.btrfs_unlock_up_safe.setup_items_for_insert.__btrfs_drop_extents
      0.00            +1.1        1.09 ± 11%  perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.ttwu_do_activate.sched_ttwu_pending.do_idle
      0.00            +1.2        1.16 ± 11%  perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.sched_ttwu_pending.do_idle.cpu_startup_entry
      0.00            +1.2        1.16 ± 11%  perf-profile.calltrace.cycles-pp.ttwu_do_activate.sched_ttwu_pending.do_idle.cpu_startup_entry.start_secondary
      0.00            +1.3        1.33 ± 11%  perf-profile.calltrace.cycles-pp.sched_ttwu_pending.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     48.26            +3.1       51.38 ±  3%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
     48.26            +3.1       51.38 ±  3%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
     48.24            +3.1       51.36 ±  3%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     38.83            -2.2       36.66 ±  4%  perf-profile.children.cycles-pp.btrfs_search_slot
     22.19            -1.8       20.40 ±  4%  perf-profile.children.cycles-pp.btrfs_read_lock_root_node
     22.00            -1.8       20.24 ±  4%  perf-profile.children.cycles-pp.btrfs_tree_read_lock
     10.23 ±  3%      -1.2        8.99 ±  5%  perf-profile.children.cycles-pp.finish_wait
     14.49            -1.0       13.45 ±  5%  perf-profile.children.cycles-pp.btrfs_csum_file_blocks
     16.59            -0.9       15.74 ±  4%  perf-profile.children.cycles-pp.btrfs_insert_empty_items
      2.97 ±  5%      -0.7        2.29 ±  7%  perf-profile.children.cycles-pp.queued_read_lock_slowpath
      5.68            -0.6        5.06 ±  5%  perf-profile.children.cycles-pp.btrfs_lookup_csum
      2.98 ±  4%      -0.5        2.45 ±  6%  perf-profile.children.cycles-pp.queued_write_lock_slowpath
      1.79 ±  7%      -0.4        1.37 ±  3%  perf-profile.children.cycles-pp.__schedule
      1.14 ±  7%      -0.3        0.86 ±  2%  perf-profile.children.cycles-pp.schedule
      0.43 ± 14%      -0.2        0.18 ±  8%  perf-profile.children.cycles-pp.poll_idle
      2.35 ±  2%      -0.2        2.10 ±  3%  perf-profile.children.cycles-pp.__filemap_fdatawrite_range
      2.33 ±  2%      -0.2        2.09 ±  3%  perf-profile.children.cycles-pp.do_writepages
      2.16            -0.2        1.92 ±  5%  perf-profile.children.cycles-pp._raw_spin_lock
      0.82 ±  7%      -0.2        0.62 ±  8%  perf-profile.children.cycles-pp.add_pending_csums
      1.12 ±  3%      -0.2        0.95 ±  3%  perf-profile.children.cycles-pp.btrfs_cow_block
      1.11 ±  3%      -0.2        0.94 ±  3%  perf-profile.children.cycles-pp.__btrfs_cow_block
      0.71 ±  6%      -0.2        0.55 ±  3%  perf-profile.children.cycles-pp.schedule_idle
      1.59 ±  2%      -0.1        1.45 ±  4%  perf-profile.children.cycles-pp.extent_writepages
      1.55 ±  2%      -0.1        1.42 ±  5%  perf-profile.children.cycles-pp.btrfs_fdatawrite_range
      1.55 ±  2%      -0.1        1.42 ±  4%  perf-profile.children.cycles-pp.start_ordered_ops
      0.68 ±  9%      -0.1        0.55 ±  7%  perf-profile.children.cycles-pp.btrfs_run_delayed_refs
      0.68 ±  9%      -0.1        0.55 ±  7%  perf-profile.children.cycles-pp.__btrfs_run_delayed_refs
      1.05 ±  5%      -0.1        0.94 ±  7%  perf-profile.children.cycles-pp.extent_write_cache_pages
      0.53 ±  9%      -0.1        0.41        perf-profile.children.cycles-pp.dequeue_task_fair
      0.75 ±  2%      -0.1        0.64        perf-profile.children.cycles-pp.btrfs_write_marked_extents
      0.50 ±  4%      -0.1        0.39 ±  2%  perf-profile.children.cycles-pp.read_block_for_search
      0.74 ±  2%      -0.1        0.63 ±  2%  perf-profile.children.cycles-pp.btree_write_cache_pages
      0.54 ±  3%      -0.1        0.43 ±  5%  perf-profile.children.cycles-pp.btrfs_get_token_32
      0.47 ±  8%      -0.1        0.38        perf-profile.children.cycles-pp.dequeue_entity
      1.02 ±  2%      -0.1        0.93 ±  3%  perf-profile.children.cycles-pp.flush_write_bio
      1.02 ±  2%      -0.1        0.93 ±  3%  perf-profile.children.cycles-pp.submit_one_bio
      0.54 ±  5%      -0.1        0.45 ±  5%  perf-profile.children.cycles-pp.btrfs_del_items
      0.97 ±  2%      -0.1        0.89 ±  3%  perf-profile.children.cycles-pp.btrfs_map_bio
      0.93 ±  3%      -0.1        0.84 ±  2%  perf-profile.children.cycles-pp.submit_bio
      0.93 ±  3%      -0.1        0.84 ±  2%  perf-profile.children.cycles-pp.generic_make_request
      0.38 ±  3%      -0.1        0.29 ±  9%  perf-profile.children.cycles-pp.__list_add_valid
      0.56 ±  2%      -0.1        0.47 ±  4%  perf-profile.children.cycles-pp.brd_do_bvec
      0.40 ±  2%      -0.1        0.32 ±  3%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.34 ± 10%      -0.1        0.26 ±  7%  perf-profile.children.cycles-pp.update_load_avg
      0.49 ±  2%      -0.1        0.41 ±  4%  perf-profile.children.cycles-pp.btree_submit_bio_hook
      0.40 ±  6%      -0.1        0.32 ±  4%  perf-profile.children.cycles-pp._raw_read_lock
      0.33 ±  4%      -0.1        0.26 ± 10%  perf-profile.children.cycles-pp.btrfs_tree_read_unlock
      0.82 ±  3%      -0.1        0.75 ±  3%  perf-profile.children.cycles-pp.brd_make_request
      0.31 ±  8%      -0.1        0.24 ±  6%  perf-profile.children.cycles-pp.btrfs_set_token_32
      0.38 ±  7%      -0.1        0.31 ±  5%  perf-profile.children.cycles-pp.pick_next_task_fair
      0.33 ±  4%      -0.1        0.27 ±  4%  perf-profile.children.cycles-pp.find_extent_buffer
      0.35 ±  4%      -0.1        0.29 ±  5%  perf-profile.children.cycles-pp.select_task_rq_fair
      0.37 ±  3%      -0.1        0.31 ±  2%  perf-profile.children.cycles-pp.copy_page
      0.29 ±  4%      -0.1        0.23 ±  6%  perf-profile.children.cycles-pp.memmove
      0.37 ±  3%      -0.1        0.31 ±  2%  perf-profile.children.cycles-pp.copy_extent_buffer_full
      0.31 ±  3%      -0.1        0.25 ±  5%  perf-profile.children.cycles-pp.__set_extent_bit
      0.19 ± 10%      -0.1        0.13 ± 14%  perf-profile.children.cycles-pp.tick_nohz_idle_exit
      0.42 ±  5%      -0.1        0.37 ±  5%  perf-profile.children.cycles-pp.alloc_tree_block_no_bg_flush
      0.33 ±  4%      -0.1        0.28 ±  4%  perf-profile.children.cycles-pp.generic_bin_search
      0.42 ±  5%      -0.1        0.37 ±  5%  perf-profile.children.cycles-pp.btrfs_alloc_tree_block
      0.23 ±  4%      -0.1        0.18 ±  7%  perf-profile.children.cycles-pp.memmove_extent_buffer
      0.12 ± 11%      -0.0        0.07 ± 10%  perf-profile.children.cycles-pp.finish_task_switch
      0.20 ±  6%      -0.0        0.15 ±  7%  perf-profile.children.cycles-pp.btrfs_mark_buffer_dirty
      0.22 ±  5%      -0.0        0.18 ±  4%  perf-profile.children.cycles-pp.btrfs_lookup_csums_range
      0.17 ± 11%      -0.0        0.12 ±  8%  perf-profile.children.cycles-pp.update_rq_clock
      0.21 ±  4%      -0.0        0.17 ±  9%  perf-profile.children.cycles-pp.alloc_extent_buffer
      0.20 ±  5%      -0.0        0.16 ±  6%  perf-profile.children.cycles-pp.btrfs_tree_unlock
      0.12 ±  8%      -0.0        0.08 ±  5%  perf-profile.children.cycles-pp.brd_insert_page
      0.22 ±  3%      -0.0        0.18 ±  3%  perf-profile.children.cycles-pp.update_curr
      0.14 ± 10%      -0.0        0.11 ±  7%  perf-profile.children.cycles-pp.set_extent_buffer_dirty
      0.22 ±  5%      -0.0        0.18 ±  4%  perf-profile.children.cycles-pp.map_private_extent_buffer
      0.16 ±  4%      -0.0        0.12 ± 12%  perf-profile.children.cycles-pp.pagecache_get_page
      0.06            -0.0        0.03 ±100%  perf-profile.children.cycles-pp.___might_sleep
      0.18 ±  2%      -0.0        0.15 ±  5%  perf-profile.children.cycles-pp.__radix_tree_lookup
      0.17 ±  5%      -0.0        0.13 ±  9%  perf-profile.children.cycles-pp.btrfs_set_path_blocking
      0.16 ±  5%      -0.0        0.12 ±  6%  perf-profile.children.cycles-pp.sched_clock
      0.21 ±  8%      -0.0        0.17 ± 10%  perf-profile.children.cycles-pp.btrfs_wait_ordered_range
      0.14 ± 11%      -0.0        0.10 ± 10%  perf-profile.children.cycles-pp.update_ts_time_stats
      0.21 ±  4%      -0.0        0.18 ±  2%  perf-profile.children.cycles-pp.set_extent_bit
      0.13 ±  5%      -0.0        0.10 ± 10%  perf-profile.children.cycles-pp.btrfs_set_lock_blocking_write
      0.13 ± 10%      -0.0        0.10 ±  4%  perf-profile.children.cycles-pp.btrfs_truncate_item
      0.12 ± 11%      -0.0        0.10 ±  9%  perf-profile.children.cycles-pp.truncate_one_csum
      0.10 ± 10%      -0.0        0.07 ± 11%  perf-profile.children.cycles-pp.ttwu_do_wakeup
      0.13 ±  8%      -0.0        0.10 ± 10%  perf-profile.children.cycles-pp.nr_iowait_cpu
      0.15 ±  7%      -0.0        0.12 ±  3%  perf-profile.children.cycles-pp.kmem_cache_alloc
      0.13 ±  6%      -0.0        0.10 ±  7%  perf-profile.children.cycles-pp.memcpy_extent_buffer
      0.10 ±  7%      -0.0        0.07 ± 11%  perf-profile.children.cycles-pp.lock_extent_bits
      0.12 ±  9%      -0.0        0.09 ±  7%  perf-profile.children.cycles-pp.__set_page_dirty_nobuffers
      0.17 ±  4%      -0.0        0.14        perf-profile.children.cycles-pp.sched_clock_cpu
      0.15 ±  5%      -0.0        0.12 ±  6%  perf-profile.children.cycles-pp.native_sched_clock
      0.06 ±  6%      -0.0        0.04 ± 57%  perf-profile.children.cycles-pp.__alloc_pages_nodemask
      0.06 ±  6%      -0.0        0.04 ± 57%  perf-profile.children.cycles-pp.alloc_extent_state
      0.09 ±  4%      -0.0        0.07 ± 10%  perf-profile.children.cycles-pp.cpuacct_charge
      0.17 ±  4%      -0.0        0.15 ±  4%  perf-profile.children.cycles-pp.write_one_eb
      0.07 ± 11%      -0.0        0.05 ±  8%  perf-profile.children.cycles-pp.reweight_entity
      0.11 ±  7%      -0.0        0.10 ±  5%  perf-profile.children.cycles-pp.clear_state_bit
      0.08 ±  6%      -0.0        0.06 ±  7%  perf-profile.children.cycles-pp.read_extent_buffer
      0.08 ±  5%      -0.0        0.07 ±  7%  perf-profile.children.cycles-pp.__lookup_extent_mapping
      0.07 ± 10%      -0.0        0.05 ±  8%  perf-profile.children.cycles-pp.__update_load_avg_se
      0.16 ±  5%      -0.0        0.14 ±  5%  perf-profile.children.cycles-pp.add_delayed_ref_head
      0.10 ±  5%      -0.0        0.08 ±  8%  perf-profile.children.cycles-pp.__test_set_page_writeback
      0.10 ±  4%      -0.0        0.09        perf-profile.children.cycles-pp.__extent_writepage_io
      0.07 ±  7%      -0.0        0.05 ±  8%  perf-profile.children.cycles-pp.__etree_search
      0.06            -0.0        0.05        perf-profile.children.cycles-pp.btrfs_set_extent_delalloc
      0.06 ±  7%      +0.0        0.08 ± 10%  perf-profile.children.cycles-pp.load_balance
      0.10 ± 15%      +0.0        0.13 ±  6%  perf-profile.children.cycles-pp.btrfs_tree_read_unlock_blocking
      0.12 ±  8%      +0.0        0.15 ±  5%  perf-profile.children.cycles-pp.rebalance_domains
      0.03 ±100%      +0.0        0.06 ±  6%  perf-profile.children.cycles-pp.run_rebalance_domains
      0.33 ±  6%      +0.0        0.36 ±  5%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.12 ±  6%      +0.0        0.17 ± 18%  perf-profile.children.cycles-pp.rcu_core
      0.06 ± 14%      +0.0        0.10 ± 14%  perf-profile.children.cycles-pp.rcu_sched_clock_irq
      0.12 ± 10%      +0.1        0.17 ±  8%  perf-profile.children.cycles-pp.prepare_to_wait
      0.00            +0.1        0.05 ±  9%  perf-profile.children.cycles-pp.update_sd_lb_stats
      0.00            +0.1        0.06 ± 11%  perf-profile.children.cycles-pp.find_busiest_group
      0.01 ±173%      +0.1        0.08 ± 31%  perf-profile.children.cycles-pp.run_local_timers
      0.04 ± 60%      +0.1        0.12 ± 25%  perf-profile.children.cycles-pp.cpu_load_update_active
      0.14 ± 15%      +0.1        0.24 ± 14%  perf-profile.children.cycles-pp.scheduler_tick
      0.00            +0.1        0.10 ±  7%  perf-profile.children.cycles-pp.save_stack_address
      0.00            +0.1        0.11 ± 11%  perf-profile.children.cycles-pp.__unwind_start
      0.37 ±  7%      +0.1        0.49 ± 10%  perf-profile.children.cycles-pp.__softirqentry_text_start
      1.25 ±  5%      +0.1        1.40 ±  2%  perf-profile.children.cycles-pp.osq_lock
      0.00            +0.2        0.16 ±  9%  perf-profile.children.cycles-pp.__module_text_address
      1.42 ±  5%      +0.2        1.59 ±  3%  perf-profile.children.cycles-pp.__mutex_lock
      0.00            +0.2        0.17 ±  7%  perf-profile.children.cycles-pp.is_module_text_address
      0.32 ± 10%      +0.2        0.53 ± 13%  perf-profile.children.cycles-pp.tick_sched_handle
      1.59 ±  6%      +0.2        1.80 ±  4%  perf-profile.children.cycles-pp.wait_log_commit
      0.31 ± 10%      +0.2        0.52 ± 12%  perf-profile.children.cycles-pp.update_process_times
      0.56 ±  8%      +0.2        0.78 ± 13%  perf-profile.children.cycles-pp.irq_exit
      0.00            +0.2        0.23 ±  5%  perf-profile.children.cycles-pp.kernel_text_address
      0.41 ±  8%      +0.2        0.65 ± 11%  perf-profile.children.cycles-pp.tick_sched_timer
      0.00            +0.3        0.26 ±  5%  perf-profile.children.cycles-pp.__kernel_text_address
      0.59 ±  8%      +0.3        0.85 ±  8%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.00            +0.3        0.27 ±  6%  perf-profile.children.cycles-pp.unwind_get_return_address
      0.00            +0.3        0.28 ±  7%  perf-profile.children.cycles-pp.__module_address
      0.40 ± 10%      +0.3        0.68 ± 18%  perf-profile.children.cycles-pp.clockevents_program_event
      0.62 ±  8%      +0.3        0.94 ± 17%  perf-profile.children.cycles-pp.ktime_get
      0.89            +0.4        1.32        perf-profile.children.cycles-pp.unlock_up
      1.90            +0.4        2.34 ±  3%  perf-profile.children.cycles-pp.setup_items_for_insert
      1.14 ±  8%      +0.5        1.67 ± 12%  perf-profile.children.cycles-pp.hrtimer_interrupt
      0.00            +0.6        0.56 ±  5%  perf-profile.children.cycles-pp.__orc_find
      1.40 ±  2%      +0.6        1.97 ±  3%  perf-profile.children.cycles-pp.btrfs_unlock_up_safe
      0.00            +0.7        0.72 ±  5%  perf-profile.children.cycles-pp.orc_find
      1.82 ±  7%      +0.7        2.57 ± 12%  perf-profile.children.cycles-pp.smp_apic_timer_interrupt
      1.98 ±  7%      +0.8        2.76 ± 11%  perf-profile.children.cycles-pp.apic_timer_interrupt
      0.45 ±  8%      +0.9        1.38 ± 11%  perf-profile.children.cycles-pp.sched_ttwu_pending
      1.77 ±  4%      +1.1        2.92 ±  2%  perf-profile.children.cycles-pp.__wake_up_common
      1.66 ±  5%      +1.2        2.81 ±  2%  perf-profile.children.cycles-pp.autoremove_wake_function
      1.58 ±  4%      +1.3        2.83 ±  2%  perf-profile.children.cycles-pp.try_to_wake_up
      0.00            +1.3        1.32 ±  4%  perf-profile.children.cycles-pp.unwind_next_frame
      2.66            +1.4        4.01 ±  2%  perf-profile.children.cycles-pp.__wake_up_common_lock
      0.00            +1.8        1.75 ±  4%  perf-profile.children.cycles-pp.__save_stack_trace
      0.00            +1.8        1.85 ±  4%  perf-profile.children.cycles-pp.save_stack_trace_tsk
      0.61 ±  4%      +2.5        3.07 ±  5%  perf-profile.children.cycles-pp.ttwu_do_activate
      0.58 ±  3%      +2.5        3.04 ±  5%  perf-profile.children.cycles-pp.enqueue_task_fair
      0.44 ±  5%      +2.5        2.92 ±  5%  perf-profile.children.cycles-pp.enqueue_entity
      0.00            +2.6        2.55 ±  5%  perf-profile.children.cycles-pp.__account_scheduler_latency
     48.26            +3.1       51.38 ±  3%  perf-profile.children.cycles-pp.start_secondary
      0.41 ± 16%      -0.2        0.17 ± 10%  perf-profile.self.cycles-pp.poll_idle
      0.68 ±  8%      -0.2        0.51 ±  8%  perf-profile.self.cycles-pp.btrfs_tree_read_lock
      1.22            -0.2        1.06 ±  4%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.93 ±  3%      -0.2        0.77        perf-profile.self.cycles-pp._raw_spin_lock
      0.59 ±  3%      -0.2        0.44 ±  5%  perf-profile.self.cycles-pp.prepare_to_wait_event
      0.45 ±  7%      -0.1        0.32 ±  3%  perf-profile.self.cycles-pp.__schedule
      0.85 ±  3%      -0.1        0.73 ±  6%  perf-profile.self.cycles-pp.queued_write_lock_slowpath
      0.52 ±  5%      -0.1        0.42 ±  4%  perf-profile.self.cycles-pp.queued_read_lock_slowpath
      0.47 ±  4%      -0.1        0.38 ±  7%  perf-profile.self.cycles-pp.btrfs_get_token_32
      0.38 ±  3%      -0.1        0.29 ±  8%  perf-profile.self.cycles-pp.__list_add_valid
      0.40 ±  2%      -0.1        0.31 ±  3%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.40 ±  4%      -0.1        0.32 ±  3%  perf-profile.self.cycles-pp._raw_read_lock
      0.35            -0.1        0.27 ±  8%  perf-profile.self.cycles-pp.btrfs_tree_lock
      0.33 ±  5%      -0.1        0.26 ± 10%  perf-profile.self.cycles-pp.btrfs_tree_read_unlock
      0.30 ±  7%      -0.1        0.23 ± 11%  perf-profile.self.cycles-pp.try_to_wake_up
      0.29 ±  7%      -0.1        0.23 ±  6%  perf-profile.self.cycles-pp.btrfs_set_token_32
      0.29 ±  4%      -0.1        0.23 ±  6%  perf-profile.self.cycles-pp.memmove
      0.36 ±  4%      -0.1        0.31 ±  2%  perf-profile.self.cycles-pp.copy_page
      0.10 ± 14%      -0.0        0.06 ±  9%  perf-profile.self.cycles-pp.finish_task_switch
      0.38 ±  5%      -0.0        0.34 ±  3%  perf-profile.self.cycles-pp.brd_do_bvec
      0.15 ± 14%      -0.0        0.11 ± 17%  perf-profile.self.cycles-pp.update_load_avg
      0.20 ±  5%      -0.0        0.15 ±  7%  perf-profile.self.cycles-pp.btrfs_tree_unlock
      0.26 ±  5%      -0.0        0.22 ± 10%  perf-profile.self.cycles-pp.do_idle
      0.21 ±  5%      -0.0        0.18 ±  4%  perf-profile.self.cycles-pp.generic_bin_search
      0.21 ±  6%      -0.0        0.17 ±  4%  perf-profile.self.cycles-pp.select_task_rq_fair
      0.19 ±  9%      -0.0        0.15 ±  5%  perf-profile.self.cycles-pp.free_extent_buffer
      0.18 ±  2%      -0.0        0.15 ±  5%  perf-profile.self.cycles-pp.__radix_tree_lookup
      0.10 ±  8%      -0.0        0.07 ±  5%  perf-profile.self.cycles-pp.pick_next_task_fair
      0.21 ±  5%      -0.0        0.18 ±  6%  perf-profile.self.cycles-pp.map_private_extent_buffer
      0.21 ±  5%      -0.0        0.18 ±  6%  perf-profile.self.cycles-pp.btrfs_root_node
      0.15 ±  7%      -0.0        0.12 ±  3%  perf-profile.self.cycles-pp.native_sched_clock
      0.13 ±  8%      -0.0        0.10 ± 10%  perf-profile.self.cycles-pp.nr_iowait_cpu
      0.13 ±  3%      -0.0        0.10 ± 10%  perf-profile.self.cycles-pp.btrfs_set_lock_blocking_write
      0.19 ±  2%      -0.0        0.16 ±  6%  perf-profile.self.cycles-pp.btrfs_search_slot
      0.09 ±  4%      -0.0        0.07 ± 10%  perf-profile.self.cycles-pp.cpuacct_charge
      0.08 ±  6%      -0.0        0.06 ±  9%  perf-profile.self.cycles-pp.read_extent_buffer
      0.11 ±  7%      -0.0        0.09 ± 11%  perf-profile.self.cycles-pp.sched_ttwu_pending
      0.07 ±  6%      -0.0        0.05        perf-profile.self.cycles-pp.reweight_entity
      0.11 ±  4%      -0.0        0.10        perf-profile.self.cycles-pp.__wake_up_common
      0.05 ±  8%      +0.0        0.10 ± 15%  perf-profile.self.cycles-pp.rcu_sched_clock_irq
      0.00            +0.1        0.06 ±  7%  perf-profile.self.cycles-pp.orc_find
      0.00            +0.1        0.06 ± 11%  perf-profile.self.cycles-pp.save_stack_address
      0.00            +0.1        0.08 ± 31%  perf-profile.self.cycles-pp.run_local_timers
      0.00            +0.1        0.10 ±  5%  perf-profile.self.cycles-pp.save_stack_trace_tsk
      0.01 ±173%      +0.1        0.12 ± 23%  perf-profile.self.cycles-pp.cpu_load_update_active
      1.24 ±  4%      +0.2        1.39 ±  2%  perf-profile.self.cycles-pp.osq_lock
      0.00            +0.3        0.28 ±  7%  perf-profile.self.cycles-pp.__module_address
      0.45 ±  9%      +0.3        0.79 ± 19%  perf-profile.self.cycles-pp.ktime_get
      0.00            +0.4        0.45 ±  7%  perf-profile.self.cycles-pp.__account_scheduler_latency
      0.00            +0.5        0.51 ±  5%  perf-profile.self.cycles-pp.unwind_next_frame
      0.00            +0.6        0.56 ±  5%  perf-profile.self.cycles-pp.__orc_find


                                                                                
                                 aim7.jobs-per-min                              
                                                                                
  1250 +-+------------------------------------------------------------------+   
  1200 +-+.+.++.+.+.+.+.+.+.++.+.+.+.+                                      |   
       |                                                                    |   
  1150 +-+                                                                  |   
  1100 +-+                                    O O O O OO O                  |   
  1050 +-+                                                 O O O O O OO O O O   
  1000 +-+                                                                  |   
       |                                                                    |   
   950 +-+                                                                  |   
   900 +-+                                                                  |   
   850 O-+      O O O O O O  O   O     O  O                                 |   
   800 +-+                  O  O         O  O                               |   
       | O                         O                                        |   
   750 +-+ O OO                      O                                      |   
   700 +-+------------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample

***************************************************************************************************
lkp-ivb-ep01: 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz with 384G memory
=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/load/md/rootfs/tbox_group/test/testcase:
  gcc-7/performance/4BRD_12G/btrfs/x86_64-rhel-7.6/20/RAID0/debian-x86_64-2018-04-03.cgz/lkp-ivb-ep01/sync_disk_rw/aim7

commit: 
  v5.1-rc7
  2996e1f8bc ("btrfs: factor our read/write stage off csum_tree_block into its callers")

        v5.1-rc7 2996e1f8bcadf0436cc67b63af0 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
    474.83           -19.1%     384.18        aim7.jobs-per-min
    252.81           +23.6%     312.51        aim7.time.elapsed_time
    252.81           +23.6%     312.51        aim7.time.elapsed_time.max
      3369 ±  3%     +15.9%       3905 ±  3%  aim7.time.involuntary_context_switches
      2829            +3.5%       2927        aim7.time.maximum_resident_set_size
      8026 ±  4%     +18.3%       9498 ±  3%  aim7.time.minor_page_faults
      1763 ±  2%     +23.9%       2185 ±  2%  aim7.time.system_time
  27540068           -11.4%   24395342        aim7.time.voluntary_context_switches
     39442 ±  2%     -22.7%      30475 ±  2%  meminfo.max_used_kB
      0.11 ± 10%      -0.0        0.08 ±  2%  mpstat.cpu.all.usr%
      5334 ±  8%     -15.2%       4524 ± 10%  numa-meminfo.node0.KernelStack
      2058 ± 29%     +43.2%       2949 ± 16%  numa-meminfo.node1.PageTables
      5335 ±  8%     -15.2%       4524 ± 10%  numa-vmstat.node0.nr_kernel_stack
    514.25 ± 29%     +43.3%     736.75 ± 16%  numa-vmstat.node1.nr_page_table_pages
    137189           -19.6%     110330        vmstat.io.bo
    295976           -28.3%     212126        vmstat.system.cs
     78.76            +1.6%      80.02        iostat.cpu.idle
     21.12            -5.8%      19.89        iostat.cpu.system
      7896           -19.5%       6356        iostat.md0.w/s
    178487           -19.6%     143527        iostat.md0.wkB/s
     20790            +3.9%      21601        proc-vmstat.nr_slab_unreclaimable
      5524 ±  9%     +24.1%       6853 ±  4%  proc-vmstat.numa_hint_faults
      3580 ± 12%     +27.7%       4572 ±  6%  proc-vmstat.numa_hint_faults_local
    588396           +23.2%     724636 ±  2%  proc-vmstat.pgfault
    969401           +12.8%    1093038        proc-vmstat.pgfree
 3.216e+09 ± 13%     -19.2%  2.598e+09 ±  3%  cpuidle.C1.time
  34331442 ±  2%     -20.2%   27399001 ±  2%  cpuidle.C1.usage
  5.21e+08 ± 52%     +71.0%  8.911e+08 ± 26%  cpuidle.C1E.time
   5266261 ± 24%     +46.0%    7687200 ± 16%  cpuidle.C1E.usage
 1.353e+09 ± 51%     +98.2%  2.683e+09 ± 27%  cpuidle.C3.time
   6733067 ± 28%     +53.5%   10335866 ± 16%  cpuidle.C3.usage
   5202587 ± 21%     +45.4%    7564469 ±  7%  cpuidle.C6.usage
    170635 ± 77%     -56.8%      73760        cpuidle.POLL.usage
      4104 ±  3%     -16.5%       3428 ± 10%  slabinfo.btrfs_extent_buffer.active_objs
      4111 ±  3%     -16.5%       3434 ± 10%  slabinfo.btrfs_extent_buffer.num_objs
     11182 ±  2%     -10.7%       9988        slabinfo.filp.active_objs
     11535           -11.0%      10262        slabinfo.filp.num_objs
      5266           +11.2%       5856        slabinfo.kmalloc-96.active_objs
      5276           +11.1%       5863        slabinfo.kmalloc-96.num_objs
    946.75           -13.2%     822.25        slabinfo.task_struct.active_objs
    960.25 ±  2%     -12.5%     840.50        slabinfo.task_struct.num_objs
    309.00            -2.9%     300.00        turbostat.Avg_MHz
  34327518 ±  2%     -20.2%   27391739 ±  2%  turbostat.C1
     31.35 ± 13%     -11.0       20.34        turbostat.C1%
   5265915 ± 24%     +46.0%    7686822 ± 16%  turbostat.C1E
   6732546 ± 28%     +53.5%   10335310 ± 16%  turbostat.C3
   5199490 ± 21%     +45.4%    7561891 ±  7%  turbostat.C6
  19750059 ± 13%     +33.5%   26375491 ±  2%  turbostat.IRQ
     32.10            -1.7%      31.56        turbostat.RAMWatt
     20520           +24.5%      25540 ±  2%  turbostat.SMI
      2739 ± 11%    +115.5%       5902 ± 38%  sched_debug.cfs_rq:/.MIN_vruntime.avg
    109456 ± 12%     +38.8%     151967 ± 11%  sched_debug.cfs_rq:/.MIN_vruntime.max
     17097 ± 11%     +64.1%      28053 ± 17%  sched_debug.cfs_rq:/.MIN_vruntime.stddev
      2739 ± 11%    +115.5%       5902 ± 38%  sched_debug.cfs_rq:/.max_vruntime.avg
    109456 ± 12%     +38.8%     151967 ± 11%  sched_debug.cfs_rq:/.max_vruntime.max
     17097 ± 11%     +64.1%      28053 ± 17%  sched_debug.cfs_rq:/.max_vruntime.stddev
    188026           +23.2%     231708        sched_debug.cfs_rq:/.min_vruntime.avg
    224730 ±  2%     +24.3%     279273 ±  3%  sched_debug.cfs_rq:/.min_vruntime.max
    155086           +22.2%     189563        sched_debug.cfs_rq:/.min_vruntime.min
     15264 ±  9%     +27.7%      19493 ± 12%  sched_debug.cfs_rq:/.min_vruntime.stddev
     15264 ±  9%     +27.7%      19493 ± 12%  sched_debug.cfs_rq:/.spread0.stddev
    167631           +20.1%     201400        sched_debug.cpu.clock.avg
    167635           +20.1%     201404        sched_debug.cpu.clock.max
    167627           +20.1%     201395        sched_debug.cpu.clock.min
      2.34 ±  4%     +13.4%       2.65 ±  5%  sched_debug.cpu.clock.stddev
    167631           +20.1%     201400        sched_debug.cpu.clock_task.avg
    167635           +20.1%     201404        sched_debug.cpu.clock_task.max
    167627           +20.1%     201395        sched_debug.cpu.clock_task.min
      2.34 ±  4%     +13.4%       2.65 ±  5%  sched_debug.cpu.clock_task.stddev
     17.73 ± 10%     -22.0%      13.83 ±  7%  sched_debug.cpu.cpu_load[0].avg
     44.72 ± 22%     -29.1%      31.70 ± 21%  sched_debug.cpu.cpu_load[0].stddev
     15.81 ± 10%     -21.7%      12.38 ± 11%  sched_debug.cpu.cpu_load[4].avg
     34.15 ± 32%     -54.3%      15.61 ± 53%  sched_debug.cpu.cpu_load[4].stddev
      4754           +16.2%       5526        sched_debug.cpu.curr->pid.max
      1004 ±  3%     +12.1%       1126        sched_debug.cpu.curr->pid.stddev
    130427           +23.1%     160557        sched_debug.cpu.nr_load_updates.avg
    138214           +23.6%     170891        sched_debug.cpu.nr_load_updates.max
    127564           +23.5%     157516        sched_debug.cpu.nr_load_updates.min
    959187            -9.5%     867604        sched_debug.cpu.nr_switches.max
    771024            -9.2%     700458 ±  4%  sched_debug.cpu.nr_switches.min
    167627           +20.1%     201395        sched_debug.cpu_clk
    164058           +20.6%     197822        sched_debug.ktime
    168073           +20.1%     201837        sched_debug.sched_clk
 8.454e+08 ±  2%     +20.1%  1.015e+09        perf-stat.i.branch-instructions
  16630864 ±  7%     +19.8%   19922156 ± 11%  perf-stat.i.branch-misses
     20.26            -4.1       16.20 ±  2%  perf-stat.i.cache-miss-rate%
  14888468 ±  2%     -16.0%   12505705        perf-stat.i.cache-misses
  74346170 ±  2%      +5.5%   78449759 ±  2%  perf-stat.i.cache-references
    299258           -28.5%     214099        perf-stat.i.context-switches
      3.21           -11.4%       2.85        perf-stat.i.cpi
    218.00 ±  4%     -10.1%     196.01        perf-stat.i.cpu-migrations
    816.02           +17.2%     956.56        perf-stat.i.cycles-between-cache-misses
   1.1e+09 ±  2%      +7.0%  1.177e+09        perf-stat.i.dTLB-loads
      0.11 ± 10%      -0.0        0.09 ±  6%  perf-stat.i.dTLB-store-miss-rate%
    572872 ±  9%     -19.6%     460348 ±  7%  perf-stat.i.dTLB-store-misses
     46.23 ±  3%      +6.8       53.01 ±  2%  perf-stat.i.iTLB-load-miss-rate%
   2592240           -27.0%    1892449 ±  2%  perf-stat.i.iTLB-loads
 4.012e+09 ±  2%     +13.9%  4.569e+09        perf-stat.i.instructions
      1948 ±  6%     +14.6%       2232 ±  4%  perf-stat.i.instructions-per-iTLB-miss
      0.32           +15.0%       0.37        perf-stat.i.ipc
   6791873           -19.7%    5457038        perf-stat.i.node-load-misses
   7294618           -19.3%    5886107        perf-stat.i.node-loads
   4662051 ±  2%     -19.9%    3733205        perf-stat.i.node-store-misses
   6959162 ±  2%     -19.3%    5614550        perf-stat.i.node-stores
     18.53 ±  2%      -7.4%      17.17        perf-stat.overall.MPKI
     20.03            -4.1       15.95        perf-stat.overall.cache-miss-rate%
      3.05           -13.5%       2.64        perf-stat.overall.cpi
    822.35           +17.2%     964.05        perf-stat.overall.cycles-between-cache-misses
      0.11 ± 11%      -0.0        0.09 ±  5%  perf-stat.overall.dTLB-store-miss-rate%
     44.19 ±  3%      +7.3       51.51 ±  2%  perf-stat.overall.iTLB-load-miss-rate%
      1959 ±  6%     +16.2%       2275 ±  4%  perf-stat.overall.instructions-per-iTLB-miss
      0.33           +15.6%       0.38        perf-stat.overall.ipc
  8.42e+08 ±  2%     +20.2%  1.012e+09        perf-stat.ps.branch-instructions
  16565866 ±  7%     +19.9%   19859511 ± 11%  perf-stat.ps.branch-misses
  14829536 ±  2%     -15.9%   12466228        perf-stat.ps.cache-misses
  74051441 ±  2%      +5.6%   78201714 ±  2%  perf-stat.ps.cache-references
    298067           -28.4%     213421        perf-stat.ps.context-switches
    217.13 ±  4%     -10.0%     195.39        perf-stat.ps.cpu-migrations
 1.095e+09 ±  2%      +7.1%  1.173e+09        perf-stat.ps.dTLB-loads
    570616 ±  9%     -19.6%     458896 ±  7%  perf-stat.ps.dTLB-store-misses
   2581925           -26.9%    1886455 ±  2%  perf-stat.ps.iTLB-loads
 3.996e+09 ±  2%     +14.0%  4.555e+09        perf-stat.ps.instructions
   6764857           -19.6%    5439764        perf-stat.ps.node-load-misses
   7265612           -19.2%    5867478        perf-stat.ps.node-loads
   4643500 ±  2%     -19.9%    3721390        perf-stat.ps.node-store-misses
   6931476 ±  2%     -19.3%    5596795        perf-stat.ps.node-stores
  1.02e+12           +42.1%   1.45e+12        perf-stat.total.instructions
    151.25 ± 11%     +94.5%     294.25 ± 46%  interrupts.43:IR-PCI-MSI.524295-edge.eth0-TxRx-6
    125.25           +24.6%     156.00        interrupts.94:IR-PCI-MSI.512000-edge.ahci[0000:00:1f.2]
     99753 ±  7%     +18.7%     118441        interrupts.CAL:Function_call_interrupts
      2335 ±  9%     +24.9%       2917 ±  8%  interrupts.CPU0.CAL:Function_call_interrupts
    500200 ±  4%     +27.5%     638000 ±  2%  interrupts.CPU0.LOC:Local_timer_interrupts
      2518 ±  9%     +15.3%       2904 ±  4%  interrupts.CPU1.CAL:Function_call_interrupts
    501181 ±  3%     +27.2%     637472 ±  2%  interrupts.CPU1.LOC:Local_timer_interrupts
      2365 ±  8%     +28.1%       3031 ±  6%  interrupts.CPU10.CAL:Function_call_interrupts
    498584 ±  4%     +27.8%     637281 ±  2%  interrupts.CPU10.LOC:Local_timer_interrupts
     14324 ± 23%     +37.1%      19636 ± 14%  interrupts.CPU10.RES:Rescheduling_interrupts
    446594 ± 24%     +42.8%     637802 ±  2%  interrupts.CPU11.LOC:Local_timer_interrupts
      2446 ±  7%     +24.3%       3039 ±  4%  interrupts.CPU12.CAL:Function_call_interrupts
    499544 ±  4%     +27.6%     637595 ±  2%  interrupts.CPU12.LOC:Local_timer_interrupts
      2265 ± 10%     +37.6%       3117 ±  6%  interrupts.CPU13.CAL:Function_call_interrupts
    500218 ±  4%     +27.5%     637698 ±  2%  interrupts.CPU13.LOC:Local_timer_interrupts
      0.75 ±173%  +81766.7%     614.00 ± 57%  interrupts.CPU13.NMI:Non-maskable_interrupts
      0.75 ±173%  +81766.7%     614.00 ± 57%  interrupts.CPU13.PMI:Performance_monitoring_interrupts
      2413 ±  5%     +24.0%       2993 ±  5%  interrupts.CPU14.CAL:Function_call_interrupts
    447745 ± 24%     +42.4%     637743 ±  2%  interrupts.CPU14.LOC:Local_timer_interrupts
     14160 ±  8%     +30.9%      18541 ± 18%  interrupts.CPU14.RES:Rescheduling_interrupts
      2450 ±  8%     +24.6%       3053 ±  3%  interrupts.CPU15.CAL:Function_call_interrupts
    447422 ± 24%     +42.5%     637614 ±  2%  interrupts.CPU15.LOC:Local_timer_interrupts
    499537 ±  4%     +27.7%     637734 ±  2%  interrupts.CPU16.LOC:Local_timer_interrupts
      2511 ±  8%     +14.4%       2872 ±  4%  interrupts.CPU17.CAL:Function_call_interrupts
    501205 ±  3%     +27.3%     637896 ±  2%  interrupts.CPU17.LOC:Local_timer_interrupts
      2466 ±  5%     +17.7%       2904 ±  7%  interrupts.CPU18.CAL:Function_call_interrupts
    498429 ±  4%     +27.9%     637539 ±  2%  interrupts.CPU18.LOC:Local_timer_interrupts
      2421 ±  8%     +19.2%       2885 ±  2%  interrupts.CPU19.CAL:Function_call_interrupts
    447347 ± 24%     +42.5%     637690 ±  2%  interrupts.CPU19.LOC:Local_timer_interrupts
     14512 ± 10%     +19.9%      17400 ± 16%  interrupts.CPU19.RES:Rescheduling_interrupts
      2472 ±  8%     +23.8%       3061 ±  4%  interrupts.CPU2.CAL:Function_call_interrupts
    498740 ±  4%     +27.9%     637673 ±  2%  interrupts.CPU2.LOC:Local_timer_interrupts
    498931 ±  4%     +27.8%     637604 ±  2%  interrupts.CPU20.LOC:Local_timer_interrupts
      2457 ±  7%     +26.2%       3102 ±  9%  interrupts.CPU21.CAL:Function_call_interrupts
    447732 ± 24%     +42.4%     637663 ±  2%  interrupts.CPU21.LOC:Local_timer_interrupts
    448745 ± 23%     +42.1%     637616 ±  2%  interrupts.CPU22.LOC:Local_timer_interrupts
      1062 ± 65%     -61.6%     407.75 ±169%  interrupts.CPU22.NMI:Non-maskable_interrupts
      1062 ± 65%     -61.6%     407.75 ±169%  interrupts.CPU22.PMI:Performance_monitoring_interrupts
      2525 ±  3%     +15.0%       2905 ±  3%  interrupts.CPU23.CAL:Function_call_interrupts
    448647 ± 23%     +42.2%     637857 ±  2%  interrupts.CPU23.LOC:Local_timer_interrupts
    499423 ±  4%     +27.7%     637620 ±  2%  interrupts.CPU24.LOC:Local_timer_interrupts
    499966 ±  4%     +27.6%     637919 ±  2%  interrupts.CPU25.LOC:Local_timer_interrupts
      2545 ±  5%     +16.7%       2971 ±  7%  interrupts.CPU26.CAL:Function_call_interrupts
    448780 ± 23%     +42.1%     637593 ±  2%  interrupts.CPU26.LOC:Local_timer_interrupts
    500205 ±  3%     +27.5%     637903 ±  2%  interrupts.CPU27.LOC:Local_timer_interrupts
      2401 ±  9%     +21.7%       2923 ±  5%  interrupts.CPU28.CAL:Function_call_interrupts
    498973 ±  4%     +27.8%     637717 ±  2%  interrupts.CPU28.LOC:Local_timer_interrupts
    502124 ±  3%     +27.0%     637676 ±  2%  interrupts.CPU29.LOC:Local_timer_interrupts
      2474 ± 11%     +21.2%       2998 ±  8%  interrupts.CPU3.CAL:Function_call_interrupts
    500168 ±  4%     +27.5%     637840 ±  2%  interrupts.CPU3.LOC:Local_timer_interrupts
    448818 ± 23%     +42.0%     637276 ±  2%  interrupts.CPU30.LOC:Local_timer_interrupts
      2535 ±  8%     +19.8%       3037 ±  2%  interrupts.CPU31.CAL:Function_call_interrupts
    501670 ±  3%     +27.0%     637350 ±  2%  interrupts.CPU31.LOC:Local_timer_interrupts
      2444 ±  5%     +20.1%       2936 ±  5%  interrupts.CPU32.CAL:Function_call_interrupts
    448601 ± 24%     +42.1%     637465 ±  2%  interrupts.CPU32.LOC:Local_timer_interrupts
    448644 ± 23%     +42.1%     637433 ±  2%  interrupts.CPU33.LOC:Local_timer_interrupts
      1298 ± 32%     -68.5%     408.75 ±173%  interrupts.CPU33.NMI:Non-maskable_interrupts
      1298 ± 32%     -68.5%     408.75 ±173%  interrupts.CPU33.PMI:Performance_monitoring_interrupts
     18952 ±  8%     -18.8%      15386 ± 16%  interrupts.CPU33.RES:Rescheduling_interrupts
    499974 ±  4%     +27.5%     637544 ±  2%  interrupts.CPU34.LOC:Local_timer_interrupts
    446755 ± 24%     +42.7%     637668 ±  2%  interrupts.CPU35.LOC:Local_timer_interrupts
      1076 ± 36%     -78.6%     230.00 ±157%  interrupts.CPU35.NMI:Non-maskable_interrupts
      1076 ± 36%     -78.6%     230.00 ±157%  interrupts.CPU35.PMI:Performance_monitoring_interrupts
      2621 ±  4%     +11.3%       2918 ±  5%  interrupts.CPU36.CAL:Function_call_interrupts
    447762 ± 24%     +42.4%     637548 ±  2%  interrupts.CPU36.LOC:Local_timer_interrupts
      2542 ± 10%     +23.9%       3151 ±  3%  interrupts.CPU37.CAL:Function_call_interrupts
    447655 ± 24%     +42.2%     636691 ±  2%  interrupts.CPU37.LOC:Local_timer_interrupts
     16018 ± 16%     +34.2%      21489 ±  4%  interrupts.CPU37.RES:Rescheduling_interrupts
    151.25 ± 11%     +94.5%     294.25 ± 46%  interrupts.CPU38.43:IR-PCI-MSI.524295-edge.eth0-TxRx-6
    448676 ± 24%     +42.1%     637520 ±  2%  interrupts.CPU38.LOC:Local_timer_interrupts
    500913 ±  3%     +27.3%     637659 ±  2%  interrupts.CPU39.LOC:Local_timer_interrupts
    448474 ± 24%     +42.2%     637512 ±  2%  interrupts.CPU4.LOC:Local_timer_interrupts
      2474 ±  8%     +24.2%       3073 ±  3%  interrupts.CPU5.CAL:Function_call_interrupts
    448794 ± 23%     +42.1%     637675 ±  2%  interrupts.CPU5.LOC:Local_timer_interrupts
      2416 ±  7%     +18.7%       2867 ±  6%  interrupts.CPU6.CAL:Function_call_interrupts
    499019 ±  4%     +27.8%     637624 ±  2%  interrupts.CPU6.LOC:Local_timer_interrupts
    449105 ± 23%     +42.0%     637874 ±  2%  interrupts.CPU7.LOC:Local_timer_interrupts
      2545 ±  4%     +18.4%       3014 ±  4%  interrupts.CPU8.CAL:Function_call_interrupts
    448672 ± 24%     +42.1%     637658 ±  2%  interrupts.CPU8.LOC:Local_timer_interrupts
    447744 ± 24%     +42.4%     637699 ±  2%  interrupts.CPU9.LOC:Local_timer_interrupts
  18961728 ± 13%     +34.5%   25504957 ±  2%  interrupts.LOC:Local_timer_interrupts
     70856 ±  3%     +27.7%      90455 ±  5%  softirqs.CPU0.RCU
     46993 ±  2%     +19.6%      56188 ±  2%  softirqs.CPU0.SCHED
     90395 ±  4%     +27.6%     115332 ±  2%  softirqs.CPU0.TIMER
     71106 ±  2%     +27.3%      90543 ±  3%  softirqs.CPU1.RCU
     44000           +20.6%      53050 ±  2%  softirqs.CPU1.SCHED
     94521 ±  6%     +16.0%     109649        softirqs.CPU1.TIMER
     72447 ±  5%     +23.7%      89595 ±  4%  softirqs.CPU10.RCU
     42542           +25.0%      53191 ±  4%  softirqs.CPU10.SCHED
     86643 ±  5%     +30.5%     113030        softirqs.CPU10.TIMER
     70388 ±  3%     +27.8%      89934 ±  2%  softirqs.CPU11.RCU
     44629 ±  3%     +19.4%      53308        softirqs.CPU11.SCHED
     71989 ±  3%     +30.1%      93643 ±  5%  softirqs.CPU12.RCU
     44508 ±  3%     +19.9%      53353 ±  2%  softirqs.CPU12.SCHED
     90388 ±  3%     +26.8%     114579 ±  3%  softirqs.CPU12.TIMER
     73320           +25.3%      91879 ±  3%  softirqs.CPU13.RCU
     44242           +20.1%      53144        softirqs.CPU13.SCHED
     89009 ±  4%     +27.3%     113304 ±  3%  softirqs.CPU13.TIMER
     70951 ±  6%     +28.3%      91061 ±  6%  softirqs.CPU14.RCU
     44755 ±  3%     +19.8%      53625 ±  2%  softirqs.CPU14.SCHED
     68460 ±  6%     +33.5%      91393 ±  3%  softirqs.CPU15.RCU
     44863 ±  2%     +19.2%      53481        softirqs.CPU15.SCHED
     72151 ±  4%     +26.3%      91161 ±  3%  softirqs.CPU16.RCU
     44009 ±  2%     +20.9%      53186 ±  2%  softirqs.CPU16.SCHED
     90040 ±  5%     +28.2%     115465        softirqs.CPU16.TIMER
     73480 ±  2%     +23.7%      90925 ±  2%  softirqs.CPU17.RCU
     43579 ±  2%     +22.2%      53237        softirqs.CPU17.SCHED
     89846 ±  5%     +28.3%     115255 ±  2%  softirqs.CPU17.TIMER
     71778 ±  4%     +27.0%      91150 ±  3%  softirqs.CPU18.RCU
     43811 ±  2%     +22.3%      53560 ±  3%  softirqs.CPU18.SCHED
     89950 ±  5%     +47.8%     132927 ± 22%  softirqs.CPU18.TIMER
     70427 ±  5%     +28.7%      90642 ±  3%  softirqs.CPU19.RCU
     44467 ±  3%     +19.6%      53172        softirqs.CPU19.SCHED
     72448 ±  3%     +25.8%      91106 ±  4%  softirqs.CPU2.RCU
     43284 ±  2%     +22.8%      53170 ±  2%  softirqs.CPU2.SCHED
     88648 ±  5%     +27.8%     113278 ±  4%  softirqs.CPU2.TIMER
     71858 ±  5%     +28.6%      92375 ±  3%  softirqs.CPU20.RCU
     43172           +22.4%      52826 ±  2%  softirqs.CPU20.SCHED
     89040 ±  5%     +26.3%     112429 ±  3%  softirqs.CPU20.TIMER
     72855 ±  5%     +28.0%      93223 ±  2%  softirqs.CPU21.RCU
     44066 ±  3%     +19.0%      52436 ±  3%  softirqs.CPU21.SCHED
     70991 ±  7%     +32.5%      94074 ±  4%  softirqs.CPU22.RCU
     44490 ±  3%     +19.0%      52958 ±  2%  softirqs.CPU22.SCHED
     71264 ±  5%     +28.1%      91257 ±  3%  softirqs.CPU23.RCU
     44233 ±  2%     +18.8%      52566 ±  2%  softirqs.CPU23.SCHED
     73463 ±  4%     +25.3%      92068 ±  4%  softirqs.CPU24.RCU
     43552 ±  2%     +21.6%      52956        softirqs.CPU24.SCHED
     89944 ±  5%     +27.3%     114465        softirqs.CPU24.TIMER
     73945           +23.4%      91231 ±  2%  softirqs.CPU25.RCU
     43633           +21.9%      53174 ±  3%  softirqs.CPU25.SCHED
     89171 ±  4%     +39.8%     124628 ± 20%  softirqs.CPU25.TIMER
     70343 ±  5%     +29.4%      91058 ±  6%  softirqs.CPU26.RCU
     43709 ±  3%     +20.8%      52784 ±  2%  softirqs.CPU26.SCHED
     72666 ±  3%     +25.0%      90831 ±  6%  softirqs.CPU27.RCU
     43708           +20.6%      52709        softirqs.CPU27.SCHED
     86809 ±  5%     +24.5%     108113 ±  6%  softirqs.CPU27.TIMER
     71506 ±  3%     +28.6%      91921 ±  5%  softirqs.CPU28.RCU
     43804 ±  2%     +20.1%      52606        softirqs.CPU28.SCHED
     73394 ±  3%     +22.8%      90149 ±  5%  softirqs.CPU29.RCU
     43517           +20.8%      52588        softirqs.CPU29.SCHED
     89568 ±  5%     +22.1%     109326 ±  6%  softirqs.CPU29.TIMER
     73421 ±  2%     +23.5%      90681 ±  4%  softirqs.CPU3.RCU
     44393           +21.5%      53943 ±  2%  softirqs.CPU3.SCHED
     89862 ±  4%     +24.0%     111453 ±  2%  softirqs.CPU3.TIMER
     71663 ±  7%     +29.0%      92457 ±  6%  softirqs.CPU30.RCU
     43602 ±  2%     +20.8%      52683 ±  2%  softirqs.CPU30.SCHED
     73484 ±  2%     +24.0%      91142        softirqs.CPU31.RCU
     43471           +21.6%      52874 ±  2%  softirqs.CPU31.SCHED
     88937 ±  5%     +26.5%     112542 ±  2%  softirqs.CPU31.TIMER
     64012 ±  4%     +24.6%      79739 ±  3%  softirqs.CPU32.RCU
     44472 ±  5%     +18.8%      52827        softirqs.CPU32.SCHED
     61122 ±  2%     +28.2%      78387 ±  3%  softirqs.CPU33.RCU
     44140 ±  3%     +19.5%      52769 ±  2%  softirqs.CPU33.SCHED
     61441           +26.8%      77902 ±  5%  softirqs.CPU34.RCU
     43298           +21.4%      52570 ±  2%  softirqs.CPU34.SCHED
     90552 ±  6%     +25.7%     113800 ±  3%  softirqs.CPU34.TIMER
     61123 ±  2%     +30.8%      79962 ±  2%  softirqs.CPU35.RCU
     44252 ±  3%     +19.5%      52863 ±  2%  softirqs.CPU35.SCHED
     59961 ±  3%     +30.3%      78141 ±  4%  softirqs.CPU36.RCU
     43798 ±  2%     +20.6%      52819 ±  2%  softirqs.CPU36.SCHED
     59914           +28.9%      77253 ±  2%  softirqs.CPU37.RCU
     44359 ±  3%     +19.4%      52965        softirqs.CPU37.SCHED
     60200 ±  3%     +29.1%      77719        softirqs.CPU38.RCU
     44765 ±  3%     +18.8%      53159 ±  2%  softirqs.CPU38.SCHED
     60108 ±  2%     +28.8%      77435        softirqs.CPU39.RCU
     44041           +20.3%      52987        softirqs.CPU39.SCHED
     88646 ±  4%     +28.5%     113914 ±  3%  softirqs.CPU39.TIMER
     69245 ±  4%     +31.6%      91096 ±  4%  softirqs.CPU4.RCU
     44364 ±  3%     +19.7%      53100 ±  2%  softirqs.CPU4.SCHED
     69900 ±  3%     +29.9%      90798 ±  3%  softirqs.CPU5.RCU
     44569 ±  3%     +21.2%      54036        softirqs.CPU5.SCHED
     71523 ±  4%     +27.3%      91057 ±  5%  softirqs.CPU6.RCU
     43811           +21.1%      53061 ±  2%  softirqs.CPU6.SCHED
     90070 ±  4%     +26.8%     114228 ±  2%  softirqs.CPU6.TIMER
     71179 ±  4%     +28.2%      91267 ±  4%  softirqs.CPU7.RCU
     45398 ±  4%     +19.0%      54043        softirqs.CPU7.SCHED
     68555 ±  5%     +32.1%      90537 ±  3%  softirqs.CPU8.RCU
     44109 ±  2%     +19.9%      52878 ±  2%  softirqs.CPU8.SCHED
     69352 ±  3%     +28.8%      89311 ±  3%  softirqs.CPU9.RCU
     44440 ±  3%     +21.0%      53755        softirqs.CPU9.SCHED
   2778312 ±  2%     +27.7%    3546581 ±  3%  softirqs.RCU
   1764877 ±  2%     +20.5%    2126626 ±  2%  softirqs.SCHED
   3831020 ± 12%     +19.8%    4591281 ±  2%  softirqs.TIMER
     38.39 ±  4%      -2.6       35.76 ±  3%  perf-profile.calltrace.cycles-pp.write
     37.65 ±  4%      -2.5       35.17 ±  4%  perf-profile.calltrace.cycles-pp.btrfs_file_write_iter.new_sync_write.vfs_write.ksys_write.do_syscall_64
     37.67 ±  4%      -2.5       35.19 ±  4%  perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     37.77 ±  4%      -2.5       35.30 ±  3%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     37.77 ±  4%      -2.5       35.30 ±  3%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
     37.75 ±  4%      -2.5       35.28 ±  3%  perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     37.74 ±  4%      -2.5       35.27 ±  4%  perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     36.01 ±  4%      -2.4       33.62 ±  4%  perf-profile.calltrace.cycles-pp.btrfs_sync_file.btrfs_file_write_iter.new_sync_write.vfs_write.ksys_write
      4.23 ±  8%      -1.5        2.74 ±  2%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.finish_wait.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot
      4.17 ±  8%      -1.5        2.70 ±  2%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.finish_wait.btrfs_tree_read_lock.btrfs_read_lock_root_node
      6.75 ±  5%      -1.1        5.60 ±  7%  perf-profile.calltrace.cycles-pp.btrfs_lookup_file_extent.__btrfs_drop_extents.insert_reserved_file_extent.btrfs_finish_ordered_io.normal_work_helper
      6.75 ±  5%      -1.1        5.60 ±  7%  perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_lookup_file_extent.__btrfs_drop_extents.insert_reserved_file_extent.btrfs_finish_ordered_io
      2.65 ± 11%      -1.0        1.60 ± 16%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.finish_wait.btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot
      6.82 ±  5%      -1.0        5.82 ±  4%  perf-profile.calltrace.cycles-pp.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent.__btrfs_drop_extents
      8.42 ±  5%      -0.9        7.53 ±  6%  perf-profile.calltrace.cycles-pp.insert_reserved_file_extent.btrfs_finish_ordered_io.normal_work_helper.process_one_work.worker_thread
      2.46 ± 15%      -0.9        1.58 ± 16%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.finish_wait.btrfs_tree_lock.btrfs_lock_root_node
      8.12 ±  5%      -0.9        7.27 ±  6%  perf-profile.calltrace.cycles-pp.__btrfs_drop_extents.insert_reserved_file_extent.btrfs_finish_ordered_io.normal_work_helper.process_one_work
      4.06 ±  9%      -0.9        3.21 ±  7%  perf-profile.calltrace.cycles-pp.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent.__btrfs_drop_extents.insert_reserved_file_extent
      9.60 ±  4%      -0.8        8.75 ±  4%  perf-profile.calltrace.cycles-pp.normal_work_helper.process_one_work.worker_thread.kthread.ret_from_fork
      9.58 ±  4%      -0.8        8.74 ±  4%  perf-profile.calltrace.cycles-pp.btrfs_finish_ordered_io.normal_work_helper.process_one_work.worker_thread.kthread
      1.25 ±  4%      -0.8        0.43 ± 59%  perf-profile.calltrace.cycles-pp.finish_wait.btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent
      1.32            -0.8        0.56 ±  9%  perf-profile.calltrace.cycles-pp.finish_wait.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items
      1.64 ±  6%      -0.6        1.07 ± 28%  perf-profile.calltrace.cycles-pp.queued_write_lock_slowpath.btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items
      2.04 ±  6%      -0.5        1.50 ±  6%  perf-profile.calltrace.cycles-pp.finish_wait.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent
      5.54 ±  5%      -0.5        5.08 ±  4%  perf-profile.calltrace.cycles-pp.btrfs_lookup_file_extent.__btrfs_drop_extents.btrfs_log_changed_extents.btrfs_log_inode.btrfs_log_inode_parent
      5.53 ±  5%      -0.5        5.08 ±  4%  perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_lookup_file_extent.__btrfs_drop_extents.btrfs_log_changed_extents.btrfs_log_inode
      2.18 ±  4%      -0.4        1.77 ±  6%  perf-profile.calltrace.cycles-pp.start_ordered_ops.btrfs_sync_file.btrfs_file_write_iter.new_sync_write.vfs_write
      2.15 ±  4%      -0.4        1.76 ±  6%  perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.btrfs_fdatawrite_range.start_ordered_ops.btrfs_sync_file.btrfs_file_write_iter
      0.70 ±  7%      -0.4        0.31 ±100%  perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter
      0.70 ±  7%      -0.4        0.31 ±100%  perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_sync_log.btrfs_sync_file
      2.16 ±  4%      -0.4        1.77 ±  6%  perf-profile.calltrace.cycles-pp.btrfs_fdatawrite_range.start_ordered_ops.btrfs_sync_file.btrfs_file_write_iter.new_sync_write
      0.69 ±  7%      -0.4        0.31 ±100%  perf-profile.calltrace.cycles-pp.btree_write_cache_pages.do_writepages.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_sync_log
      2.12 ±  4%      -0.4        1.74 ±  6%  perf-profile.calltrace.cycles-pp.extent_writepages.do_writepages.__filemap_fdatawrite_range.btrfs_fdatawrite_range.start_ordered_ops
      2.13 ±  4%      -0.4        1.75 ±  6%  perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.btrfs_fdatawrite_range.start_ordered_ops.btrfs_sync_file
      2.41 ±  7%      -0.3        2.12 ±  6%  perf-profile.calltrace.cycles-pp.btrfs_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent.__btrfs_drop_extents.btrfs_log_changed_extents
      1.61 ±  4%      -0.3        1.32 ± 11%  perf-profile.calltrace.cycles-pp.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_inode.__btrfs_update_delayed_inode.btrfs_commit_inode_delayed_items
      1.59 ±  4%      -0.3        1.31 ± 11%  perf-profile.calltrace.cycles-pp.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_inode.__btrfs_update_delayed_inode
      1.29 ±  4%      -0.3        1.02 ±  8%  perf-profile.calltrace.cycles-pp.extent_write_cache_pages.extent_writepages.do_writepages.__filemap_fdatawrite_range.btrfs_fdatawrite_range
      0.70 ±  6%      -0.3        0.43 ± 59%  perf-profile.calltrace.cycles-pp.btrfs_write_marked_extents.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter.new_sync_write
      1.59 ±  8%      -0.2        1.35 ±  5%  perf-profile.calltrace.cycles-pp.finish_wait.btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items
      2.14 ±  2%      -0.2        1.90 ±  7%  perf-profile.calltrace.cycles-pp.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items.btrfs_csum_file_blocks
      2.14 ±  2%      -0.2        1.91 ±  7%  perf-profile.calltrace.cycles-pp.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items.btrfs_csum_file_blocks.btrfs_log_changed_extents
      1.05 ±  7%      -0.2        0.82 ±  8%  perf-profile.calltrace.cycles-pp.__extent_writepage.extent_write_cache_pages.extent_writepages.do_writepages.__filemap_fdatawrite_range
      0.61 ±  8%      -0.2        0.40 ± 57%  perf-profile.calltrace.cycles-pp.generic_make_request.submit_bio.btrfs_map_bio.btrfs_submit_bio_hook.submit_one_bio
      1.12 ±  9%      -0.2        0.94        perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      0.80 ±  8%      -0.2        0.62 ± 10%  perf-profile.calltrace.cycles-pp.btrfs_run_delalloc_range.writepage_delalloc.__extent_writepage.extent_write_cache_pages.extent_writepages
      1.08 ± 10%      -0.2        0.91        perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary
      0.84 ±  8%      -0.2        0.68 ±  8%  perf-profile.calltrace.cycles-pp.writepage_delalloc.__extent_writepage.extent_write_cache_pages.extent_writepages.do_writepages
      0.77 ±  9%      -0.2        0.61 ± 10%  perf-profile.calltrace.cycles-pp.cow_file_range.btrfs_run_delalloc_range.writepage_delalloc.__extent_writepage.extent_write_cache_pages
      0.88 ±  4%      -0.1        0.73 ±  4%  perf-profile.calltrace.cycles-pp.finish_wait.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_csum
      0.81 ±  4%      -0.1        0.71 ±  7%  perf-profile.calltrace.cycles-pp.btrfs_submit_bio_hook.submit_one_bio.flush_write_bio.extent_writepages.do_writepages
      0.82 ±  5%      -0.1        0.72 ±  6%  perf-profile.calltrace.cycles-pp.flush_write_bio.extent_writepages.do_writepages.__filemap_fdatawrite_range.btrfs_fdatawrite_range
      0.81 ±  4%      -0.1        0.72 ±  7%  perf-profile.calltrace.cycles-pp.submit_one_bio.flush_write_bio.extent_writepages.do_writepages.__filemap_fdatawrite_range
      2.63 ±  2%      +0.2        2.85 ±  5%  perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry
      0.73 ±  6%      +0.3        0.99 ±  6%  perf-profile.calltrace.cycles-pp.setup_items_for_insert.__btrfs_drop_extents.btrfs_log_changed_extents.btrfs_log_inode.btrfs_log_inode_parent
      2.79 ±  2%      +0.3        3.07 ±  5%  perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary
      0.63 ±  2%      +0.3        0.95 ±  8%  perf-profile.calltrace.cycles-pp.btrfs_unlock_up_safe.setup_items_for_insert.__btrfs_drop_extents.insert_reserved_file_extent.btrfs_finish_ordered_io
      0.62 ±  2%      +0.3        0.94 ±  8%  perf-profile.calltrace.cycles-pp.__wake_up_common_lock.btrfs_unlock_up_safe.setup_items_for_insert.__btrfs_drop_extents.insert_reserved_file_extent
      0.79 ±  3%      +0.4        1.15 ±  9%  perf-profile.calltrace.cycles-pp.setup_items_for_insert.__btrfs_drop_extents.insert_reserved_file_extent.btrfs_finish_ordered_io.normal_work_helper
      2.34 ±  4%      +0.4        2.71 ±  6%  perf-profile.calltrace.cycles-pp.prepare_to_wait_event.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items
      0.42 ± 58%      +0.4        0.84 ±  6%  perf-profile.calltrace.cycles-pp.btrfs_unlock_up_safe.setup_items_for_insert.__btrfs_drop_extents.btrfs_log_changed_extents.btrfs_log_inode
      0.63 ±  8%      +0.4        1.06 ±  6%  perf-profile.calltrace.cycles-pp.unlock_up.btrfs_search_slot.btrfs_insert_empty_items.btrfs_log_inode.btrfs_log_inode_parent
      0.61 ±  8%      +0.4        1.05 ±  6%  perf-profile.calltrace.cycles-pp.__wake_up_common_lock.unlock_up.btrfs_search_slot.btrfs_insert_empty_items.btrfs_log_inode
      0.40 ± 58%      +0.4        0.84 ±  6%  perf-profile.calltrace.cycles-pp.__wake_up_common_lock.btrfs_unlock_up_safe.setup_items_for_insert.__btrfs_drop_extents.btrfs_log_changed_extents
      2.74 ±  5%      +0.5        3.28 ±  5%  perf-profile.calltrace.cycles-pp.prepare_to_wait_event.btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items
      0.28 ±100%      +0.6        0.87 ±  6%  perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.unlock_up
      0.30 ±100%      +0.6        0.90 ±  7%  perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.unlock_up.btrfs_search_slot
      0.00            +0.6        0.59 ±  5%  perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function
      0.31 ±100%      +0.6        0.91 ±  6%  perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.unlock_up.btrfs_search_slot.btrfs_insert_empty_items
      0.00            +0.6        0.61 ±  6%  perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common
      0.00            +0.6        0.62 ±  7%  perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      7.53 ±  3%      +0.8        8.32 ±  4%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.prepare_to_wait_event.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot
      0.00            +0.8        0.82 ± 10%  perf-profile.calltrace.cycles-pp.unwind_next_frame.__save_stack_trace.save_stack_trace_tsk.__account_scheduler_latency.enqueue_entity
      7.34 ±  3%      +0.8        8.17 ±  4%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.prepare_to_wait_event.btrfs_tree_read_lock.btrfs_read_lock_root_node
      4.47 ±  3%      +0.9        5.40 ±  9%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.prepare_to_wait_event.btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot
      4.33 ±  3%      +0.9        5.28 ±  9%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.prepare_to_wait_event.btrfs_tree_lock.btrfs_lock_root_node
      0.00            +1.2        1.24 ±  6%  perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.btrfs_unlock_up_safe
      0.00            +1.3        1.31 ±  6%  perf-profile.calltrace.cycles-pp.__save_stack_trace.save_stack_trace_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair
      0.00            +1.3        1.31 ±  6%  perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.btrfs_unlock_up_safe.setup_items_for_insert
      0.00            +1.3        1.34 ±  6%  perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.btrfs_unlock_up_safe.setup_items_for_insert.__btrfs_drop_extents
      0.00            +1.4        1.37 ±  6%  perf-profile.calltrace.cycles-pp.save_stack_trace_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.ttwu_do_activate
      0.80 ±  6%      +1.7        2.55 ±  4%  perf-profile.calltrace.cycles-pp.sched_ttwu_pending.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      0.00            +1.8        1.83 ±  4%  perf-profile.calltrace.cycles-pp.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.ttwu_do_activate.sched_ttwu_pending
      0.27 ±100%      +2.0        2.26 ±  5%  perf-profile.calltrace.cycles-pp.ttwu_do_activate.sched_ttwu_pending.do_idle.cpu_startup_entry.start_secondary
      0.26 ±100%      +2.0        2.25 ±  5%  perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.sched_ttwu_pending.do_idle.cpu_startup_entry
      0.00            +2.2        2.16 ±  6%  perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.ttwu_do_activate.sched_ttwu_pending.do_idle
     49.01 ±  3%      +2.3       51.35 ±  2%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
     49.01 ±  3%      +2.3       51.35 ±  2%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
     48.97 ±  3%      +2.4       51.33 ±  2%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     49.91 ±  2%      +2.5       52.45        perf-profile.calltrace.cycles-pp.secondary_startup_64
     38.39 ±  4%      -2.6       35.77 ±  3%  perf-profile.children.cycles-pp.write
     37.67 ±  4%      -2.5       35.19 ±  4%  perf-profile.children.cycles-pp.new_sync_write
     37.65 ±  4%      -2.5       35.17 ±  4%  perf-profile.children.cycles-pp.btrfs_file_write_iter
     37.76 ±  4%      -2.5       35.28 ±  3%  perf-profile.children.cycles-pp.ksys_write
     37.74 ±  4%      -2.5       35.27 ±  4%  perf-profile.children.cycles-pp.vfs_write
     36.01 ±  4%      -2.4       33.62 ±  4%  perf-profile.children.cycles-pp.btrfs_sync_file
     36.52 ±  3%      -2.3       34.18 ±  2%  perf-profile.children.cycles-pp.btrfs_search_slot
     19.62 ±  2%      -2.1       17.56 ±  2%  perf-profile.children.cycles-pp.btrfs_read_lock_root_node
     19.50 ±  2%      -2.0       17.47 ±  2%  perf-profile.children.cycles-pp.btrfs_tree_read_lock
      8.71 ±  2%      -1.9        6.79 ±  3%  perf-profile.children.cycles-pp.finish_wait
     38.42 ±  4%      -1.9       36.52 ±  2%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     38.41 ±  4%      -1.9       36.52 ±  2%  perf-profile.children.cycles-pp.do_syscall_64
     12.40 ±  4%      -1.5       10.86 ±  4%  perf-profile.children.cycles-pp.btrfs_lookup_file_extent
     14.73 ±  3%      -1.1       13.60 ±  3%  perf-profile.children.cycles-pp.__btrfs_drop_extents
      8.43 ±  5%      -0.9        7.53 ±  6%  perf-profile.children.cycles-pp.insert_reserved_file_extent
      9.60 ±  4%      -0.8        8.75 ±  4%  perf-profile.children.cycles-pp.normal_work_helper
      9.58 ±  4%      -0.8        8.74 ±  4%  perf-profile.children.cycles-pp.btrfs_finish_ordered_io
      2.97 ±  5%      -0.8        2.14 ±  4%  perf-profile.children.cycles-pp.queued_read_lock_slowpath
      3.06 ±  4%      -0.7        2.34 ±  3%  perf-profile.children.cycles-pp.queued_write_lock_slowpath
      2.92 ±  4%      -0.5        2.38 ±  6%  perf-profile.children.cycles-pp.__filemap_fdatawrite_range
      2.89 ±  5%      -0.5        2.37 ±  6%  perf-profile.children.cycles-pp.do_writepages
      2.08 ±  8%      -0.4        1.64 ± 10%  perf-profile.children.cycles-pp.btrfs_lookup_inode
      2.18 ±  8%      -0.4        1.75 ±  8%  perf-profile.children.cycles-pp.__btrfs_update_delayed_inode
      2.84 ±  4%      -0.4        2.42 ±  2%  perf-profile.children.cycles-pp.__schedule
      2.18 ±  4%      -0.4        1.77 ±  6%  perf-profile.children.cycles-pp.start_ordered_ops
      2.18 ±  4%      -0.4        1.78 ±  6%  perf-profile.children.cycles-pp.btrfs_fdatawrite_range
      2.18 ±  5%      -0.4        1.80 ±  5%  perf-profile.children.cycles-pp.extent_writepages
      1.33 ±  5%      -0.3        1.06 ±  7%  perf-profile.children.cycles-pp.extent_write_cache_pages
      1.75            -0.3        1.49 ±  4%  perf-profile.children.cycles-pp.schedule
      1.06 ±  8%      -0.2        0.83 ±  8%  perf-profile.children.cycles-pp.__extent_writepage
      0.66 ± 14%      -0.2        0.43 ± 13%  perf-profile.children.cycles-pp.btrfs_next_old_leaf
      0.83 ±  2%      -0.2        0.62 ±  7%  perf-profile.children.cycles-pp.update_load_avg
      1.14 ±  8%      -0.2        0.95        perf-profile.children.cycles-pp.schedule_idle
      0.80 ±  8%      -0.2        0.62 ± 10%  perf-profile.children.cycles-pp.btrfs_run_delalloc_range
      0.30 ± 24%      -0.2        0.12 ±  4%  perf-profile.children.cycles-pp.btrfs_commit_inode_delayed_inode
      0.84 ±  8%      -0.2        0.68 ±  8%  perf-profile.children.cycles-pp.writepage_delalloc
      1.24 ±  7%      -0.2        1.08 ±  8%  perf-profile.children.cycles-pp.flush_write_bio
      0.77 ±  9%      -0.2        0.61 ± 10%  perf-profile.children.cycles-pp.cow_file_range
      0.49 ±  6%      -0.2        0.33 ± 10%  perf-profile.children.cycles-pp.__list_del_entry_valid
      1.04 ±  9%      -0.1        0.89 ±  8%  perf-profile.children.cycles-pp.generic_make_request
      0.70 ±  7%      -0.1        0.55 ± 12%  perf-profile.children.cycles-pp.btree_write_cache_pages
      0.70 ±  6%      -0.1        0.56 ± 11%  perf-profile.children.cycles-pp.btrfs_write_marked_extents
      1.04 ±  9%      -0.1        0.90 ±  8%  perf-profile.children.cycles-pp.submit_bio
      0.62 ±  4%      -0.1        0.48 ±  6%  perf-profile.children.cycles-pp.read_block_for_search
      0.86 ± 11%      -0.1        0.72 ±  8%  perf-profile.children.cycles-pp.brd_make_request
      0.51 ±  7%      -0.1        0.40 ±  9%  perf-profile.children.cycles-pp.select_task_rq_fair
      0.84 ±  4%      -0.1        0.74 ±  6%  perf-profile.children.cycles-pp.btrfs_submit_bio_hook
      0.49 ± 10%      -0.1        0.39 ±  6%  perf-profile.children.cycles-pp.brd_do_bvec
      0.77 ±  6%      -0.1        0.67 ±  4%  perf-profile.children.cycles-pp.dequeue_entity
      0.61 ±  8%      -0.1        0.54 ±  4%  perf-profile.children.cycles-pp.pick_next_task_fair
      0.27 ± 10%      -0.1        0.20 ± 16%  perf-profile.children.cycles-pp.map_private_extent_buffer
      0.40 ±  8%      -0.1        0.33 ±  5%  perf-profile.children.cycles-pp.find_extent_buffer
      0.25 ±  4%      -0.1        0.18 ± 13%  perf-profile.children.cycles-pp.__update_load_avg_se
      0.21 ±  2%      -0.1        0.14 ± 16%  perf-profile.children.cycles-pp.write_one_eb
      0.09 ± 13%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.create_io_em
      0.21 ± 13%      -0.1        0.15 ± 16%  perf-profile.children.cycles-pp.btrfs_tree_read_unlock
      0.35 ± 14%      -0.1        0.30 ±  9%  perf-profile.children.cycles-pp.__list_add_valid
      0.21 ± 10%      -0.1        0.15 ±  7%  perf-profile.children.cycles-pp.tick_nohz_idle_exit
      0.10 ± 22%      -0.0        0.06 ± 14%  perf-profile.children.cycles-pp.xas_load
      0.12 ± 19%      -0.0        0.08 ± 16%  perf-profile.children.cycles-pp.__test_set_page_writeback
      0.16 ± 15%      -0.0        0.12 ± 17%  perf-profile.children.cycles-pp.block_group_cache_tree_search
      0.29 ±  5%      -0.0        0.25 ±  7%  perf-profile.children.cycles-pp.memmove
      0.29 ± 10%      -0.0        0.25 ±  7%  perf-profile.children.cycles-pp.kmem_cache_alloc
      0.26 ±  7%      -0.0        0.22 ±  8%  perf-profile.children.cycles-pp._raw_read_lock
      0.18 ±  8%      -0.0        0.15 ±  7%  perf-profile.children.cycles-pp.__switch_to
      0.13 ± 12%      -0.0        0.10 ±  7%  perf-profile.children.cycles-pp.update_cfs_rq_h_load
      0.16 ±  6%      -0.0        0.13 ± 10%  perf-profile.children.cycles-pp.___perf_sw_event
      0.11 ± 17%      -0.0        0.09 ± 10%  perf-profile.children.cycles-pp.btrfs_get_token_64
      0.11 ±  7%      -0.0        0.09 ±  7%  perf-profile.children.cycles-pp.tick_nohz_idle_enter
      0.07 ±  7%      -0.0        0.05 ±  8%  perf-profile.children.cycles-pp.btrfs_set_extent_delalloc
      0.25 ±  7%      +0.0        0.28 ±  2%  perf-profile.children.cycles-pp.copy_extent_buffer_full
      0.24 ±  9%      +0.0        0.28 ±  2%  perf-profile.children.cycles-pp.copy_page
      0.13 ±  9%      +0.1        0.20 ±  9%  perf-profile.children.cycles-pp.queue_work_on
      0.13 ± 12%      +0.1        0.20 ±  9%  perf-profile.children.cycles-pp.__queue_work
      0.07 ± 58%      +0.1        0.14 ± 20%  perf-profile.children.cycles-pp.btrfs_tree_read_unlock_blocking
      0.00            +0.1        0.08 ± 34%  perf-profile.children.cycles-pp.in_sched_functions
      0.04 ± 57%      +0.1        0.12 ± 10%  perf-profile.children.cycles-pp.reschedule_interrupt
      0.00            +0.1        0.08 ±  5%  perf-profile.children.cycles-pp.stack_access_ok
      0.09 ± 37%      +0.1        0.19 ± 17%  perf-profile.children.cycles-pp.tick_irq_enter
      0.00            +0.1        0.10 ±  8%  perf-profile.children.cycles-pp.scheduler_ipi
      0.15 ± 22%      +0.1        0.26 ± 15%  perf-profile.children.cycles-pp.irq_enter
      0.58 ±  3%      +0.1        0.71 ± 15%  perf-profile.children.cycles-pp.update_process_times
      1.26 ±  2%      +0.2        1.43 ±  8%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.43 ±  8%      +0.2        0.61 ± 12%  perf-profile.children.cycles-pp.btrfs_release_path
      0.00            +0.2        0.22 ± 12%  perf-profile.children.cycles-pp.save_stack_address
      1.87 ±  2%      +0.2        2.11 ±  7%  perf-profile.children.cycles-pp.hrtimer_interrupt
      0.00            +0.3        0.26 ± 16%  perf-profile.children.cycles-pp.__module_text_address
      0.00            +0.3        0.29 ± 13%  perf-profile.children.cycles-pp.is_module_text_address
      0.00            +0.3        0.29 ±  4%  perf-profile.children.cycles-pp.__unwind_start
      3.26 ±  2%      +0.3        3.61 ±  4%  perf-profile.children.cycles-pp.smp_apic_timer_interrupt
      0.00            +0.4        0.40 ± 10%  perf-profile.children.cycles-pp.kernel_text_address
      3.47 ±  3%      +0.4        3.90 ±  3%  perf-profile.children.cycles-pp.apic_timer_interrupt
      0.00            +0.4        0.43 ±  9%  perf-profile.children.cycles-pp.__kernel_text_address
      0.00            +0.5        0.46 ±  9%  perf-profile.children.cycles-pp.unwind_get_return_address
      0.00            +0.5        0.47 ±  7%  perf-profile.children.cycles-pp.__module_address
      0.78 ±  5%      +0.5        1.25 ±  5%  perf-profile.children.cycles-pp.unlock_up
      1.80 ±  3%      +0.6        2.42 ±  5%  perf-profile.children.cycles-pp.setup_items_for_insert
      1.26            +0.7        1.94 ±  4%  perf-profile.children.cycles-pp.btrfs_unlock_up_safe
      0.00            +0.8        0.85 ±  5%  perf-profile.children.cycles-pp.__orc_find
      0.00            +1.1        1.12 ±  4%  perf-profile.children.cycles-pp.orc_find
      1.87 ±  4%      +1.4        3.30 ±  2%  perf-profile.children.cycles-pp.__wake_up_common
      1.80 ±  4%      +1.4        3.23 ±  3%  perf-profile.children.cycles-pp.autoremove_wake_function
      1.83 ±  4%      +1.6        3.41 ±  3%  perf-profile.children.cycles-pp.try_to_wake_up
      2.43 ±  4%      +1.6        4.06        perf-profile.children.cycles-pp.__wake_up_common_lock
     14.82 ±  3%      +1.7       16.56 ±  2%  perf-profile.children.cycles-pp.prepare_to_wait_event
      0.85 ±  4%      +1.9        2.71 ±  2%  perf-profile.children.cycles-pp.sched_ttwu_pending
      0.00            +2.0        1.96 ±  5%  perf-profile.children.cycles-pp.unwind_next_frame
     49.01 ±  3%      +2.3       51.35 ±  2%  perf-profile.children.cycles-pp.start_secondary
     49.91 ±  2%      +2.5       52.45        perf-profile.children.cycles-pp.secondary_startup_64
     49.91 ±  2%      +2.5       52.45        perf-profile.children.cycles-pp.cpu_startup_entry
     49.90 ±  2%      +2.5       52.45        perf-profile.children.cycles-pp.do_idle
      0.00            +2.8        2.78 ±  4%  perf-profile.children.cycles-pp.__save_stack_trace
      0.00            +2.9        2.90 ±  4%  perf-profile.children.cycles-pp.save_stack_trace_tsk
      1.02 ± 10%      +3.7        4.73 ±  2%  perf-profile.children.cycles-pp.ttwu_do_activate
      0.82 ±  9%      +3.7        4.54 ±  2%  perf-profile.children.cycles-pp.enqueue_entity
      0.97 ±  9%      +3.7        4.69 ±  2%  perf-profile.children.cycles-pp.enqueue_task_fair
      0.00            +3.8        3.84 ±  2%  perf-profile.children.cycles-pp.__account_scheduler_latency
      0.54 ± 15%      -0.2        0.35 ±  5%  perf-profile.self.cycles-pp.btrfs_tree_read_lock
      0.48 ±  9%      -0.2        0.30 ±  7%  perf-profile.self.cycles-pp.prepare_to_wait_event
      0.49 ±  5%      -0.2        0.32 ± 11%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.74 ±  7%      -0.2        0.58        perf-profile.self.cycles-pp.queued_write_lock_slowpath
      0.35 ±  8%      -0.2        0.20 ±  7%  perf-profile.self.cycles-pp.btrfs_tree_lock
      0.48 ±  4%      -0.1        0.37 ±  6%  perf-profile.self.cycles-pp.__schedule
      0.69 ±  7%      -0.1        0.58 ±  4%  perf-profile.self.cycles-pp._raw_spin_lock
      0.44 ±  6%      -0.1        0.33 ±  6%  perf-profile.self.cycles-pp.queued_read_lock_slowpath
      0.27 ±  6%      -0.1        0.19 ± 16%  perf-profile.self.cycles-pp.update_load_avg
      0.23 ± 13%      -0.1        0.17 ± 18%  perf-profile.self.cycles-pp.brd_do_bvec
      0.24 ±  6%      -0.1        0.17 ± 11%  perf-profile.self.cycles-pp.__update_load_avg_se
      0.32 ±  2%      -0.1        0.26 ±  7%  perf-profile.self.cycles-pp.try_to_wake_up
      0.29 ±  9%      -0.1        0.23 ± 10%  perf-profile.self.cycles-pp.select_task_rq_fair
      0.24 ±  9%      -0.1        0.19 ± 15%  perf-profile.self.cycles-pp.map_private_extent_buffer
      0.21 ± 14%      -0.1        0.15 ± 16%  perf-profile.self.cycles-pp.btrfs_tree_read_unlock
      0.35 ± 14%      -0.1        0.30 ±  9%  perf-profile.self.cycles-pp.__list_add_valid
      0.26 ±  7%      -0.0        0.21 ±  8%  perf-profile.self.cycles-pp._raw_read_lock
      0.17 ±  9%      -0.0        0.14 ±  8%  perf-profile.self.cycles-pp.__switch_to
      0.13 ± 12%      -0.0        0.10 ±  7%  perf-profile.self.cycles-pp.update_cfs_rq_h_load
      0.28 ±  6%      -0.0        0.25 ±  7%  perf-profile.self.cycles-pp.memmove
      0.14 ±  8%      -0.0        0.11 ±  9%  perf-profile.self.cycles-pp.___perf_sw_event
      0.19 ±  9%      +0.0        0.22 ± 11%  perf-profile.self.cycles-pp.enqueue_entity
      0.00            +0.1        0.06 ± 16%  perf-profile.self.cycles-pp.__unwind_start
      0.00            +0.1        0.07 ± 17%  perf-profile.self.cycles-pp.kernel_text_address
      0.00            +0.1        0.08 ± 14%  perf-profile.self.cycles-pp.__save_stack_trace
      0.00            +0.1        0.08 ±  5%  perf-profile.self.cycles-pp.stack_access_ok
      0.00            +0.1        0.12 ± 13%  perf-profile.self.cycles-pp.save_stack_trace_tsk
      0.00            +0.1        0.14 ± 21%  perf-profile.self.cycles-pp.orc_find
      0.00            +0.1        0.14 ± 18%  perf-profile.self.cycles-pp.save_stack_address
      0.00            +0.5        0.46 ±  7%  perf-profile.self.cycles-pp.__module_address
      0.00            +0.6        0.61 ±  7%  perf-profile.self.cycles-pp.__account_scheduler_latency
      0.00            +0.6        0.62 ±  7%  perf-profile.self.cycles-pp.unwind_next_frame
      0.00            +0.8        0.84 ±  5%  perf-profile.self.cycles-pp.__orc_find





Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.1.0-rc7-00001-g2996e1f" of type "text/plain" (188885 bytes)

View attachment "job-script" of type "text/plain" (7785 bytes)

View attachment "job.yaml" of type "text/plain" (5247 bytes)

View attachment "reproduce" of type "text/plain" (1007 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ