lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Date:   Wed, 15 Apr 2020 13:00:25 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     "Darrick J. Wong" <darrick.wong@...cle.com>
Cc:     LKML <linux-kernel@...r.kernel.org>,
        "Darrick J. Wong" <darrick.wong@...cle.com>, lkp@...ts.01.org
Subject: [xfs] ef6d7398f5: filebench.sum_bytes_mb/s -47.5% regression

Greeting,

FYI, we noticed a -47.5% regression of filebench.sum_bytes_mb/s due to commit:


commit: ef6d7398f5c72c4fa030740a7552d00ecbf4abd7 ("xfs: deferred inode inactivation")
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git repair-hard-problems

in testcase: filebench
on test machine: 8 threads Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz with 8G memory
with following parameters:

	disk: 1HDD
	fs: xfs
	test: fileserver.f
	cpufreq_governor: performance
	ucode: 0x27


In addition to that, the commit also has significant impact on the following tests:

+------------------+----------------------------------------------------------------------+
| testcase: change | aim7: aim7.jobs-per-min -23.6% regression                            |
| test machine     | 72 threads Intel(R) Xeon(R) Gold 6139 CPU @ 2.30GHz with 128G memory |
| test parameters  | cpufreq_governor=performance                                         |
|                  | disk=1BRD_48G                                                        |
|                  | fs=xfs                                                               |
|                  | load=3000                                                            |
|                  | test=disk_rr                                                         |
|                  | ucode=0x2000065                                                      |
+------------------+----------------------------------------------------------------------+
| testcase: change | aim7: aim7.jobs-per-min -35.7% regression                            |
| test machine     | 72 threads Intel(R) Xeon(R) Gold 6139 CPU @ 2.30GHz with 128G memory |
| test parameters  | cpufreq_governor=performance                                         |
|                  | disk=1BRD_48G                                                        |
|                  | fs=xfs                                                               |
|                  | load=3000                                                            |
|                  | test=disk_wrt                                                        |
|                  | ucode=0x2000065                                                      |
+------------------+----------------------------------------------------------------------+


If you fix the issue, kindly add following tag
Reported-by: kernel test robot <rong.a.chen@...el.com>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/rootfs/tbox_group/test/testcase/ucode:
  gcc-7/performance/1HDD/xfs/x86_64-rhel-7.6/debian-x86_64-20191114.cgz/lkp-hsw-d01/fileserver.f/filebench/0x27

commit: 
  ad3a8d2d61 ("xfs: pass around xfs_inode_ag_walk iget/irele helper functions")
  ef6d7398f5 ("xfs: deferred inode inactivation")

ad3a8d2d61135b44 ef6d7398f5c72c4fa030740a755 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
    161.43           -47.5%      84.80        filebench.sum_bytes_mb/s
    406381           -46.9%     215630        filebench.sum_operations
      6772           -46.9%       3593        filebench.sum_operations/s
    615.67           -46.9%     326.67        filebench.sum_reads/s
      7.36           +88.0%      13.85        filebench.sum_time_ms/op
      1231           -46.9%     653.67        filebench.sum_writes/s
  12197290           -39.0%    7434264        filebench.time.file_system_outputs
      3339 ± 13%     -30.7%       2314 ±  5%  filebench.time.involuntary_context_switches
     18.00           -44.4%      10.00        filebench.time.percent_of_cpu_this_job_got
     10.45           -46.4%       5.61        filebench.time.system_time
    128533           -61.8%      49072        filebench.time.voluntary_context_switches
    250432           -26.0%     185230        meminfo.Active(file)
      2.38            -0.7        1.68 ± 10%  mpstat.cpu.all.sys%
     35079 ±  7%     -10.4%      31430 ±  2%  softirqs.CPU5.TIMER
     10155           -11.5%       8991        vmstat.io.bo
     33748            -6.9%      31421        vmstat.system.cs
     22.13            +3.6%      22.93        iostat.cpu.iowait
      2.49 ±  3%     -28.0%       1.79 ± 13%  iostat.cpu.system
      1.85            +1.7%       1.88        iostat.sda.avgqu-sz
     97.43            -3.4%      94.11        iostat.sda.w/s
     10254           -11.4%       9089        iostat.sda.wkB/s
     11.45           +18.7%      13.60 ±  9%  iostat.sda.wrqm/s
    629.00 ± 27%     +57.6%     991.00 ± 13%  interrupts.CPU3.RES:Rescheduling_interrupts
     99.67           -20.4%      79.33 ± 22%  interrupts.CPU5.CAL:Function_call_interrupts
    224441           -11.9%     197754 ±  4%  interrupts.CPU5.LOC:Local_timer_interrupts
    632.00 ±  8%     -45.7%     343.33 ± 29%  interrupts.CPU5.RES:Rescheduling_interrupts
     10421          -100.0%       0.00        interrupts.CPU6.19:IR-IO-APIC.19-fasteoi.ata_piix
    203183           +34.5%     273198 ± 10%  interrupts.CPU6.LOC:Local_timer_interrupts
    506.67 ± 14%     -23.1%     389.67 ± 14%  interrupts.CPU6.RES:Rescheduling_interrupts
    612.67 ± 17%     -32.3%     415.00 ± 18%  interrupts.CPU7.RES:Rescheduling_interrupts
    184694            -1.0%     182776        proc-vmstat.nr_active_anon
     62958           -26.4%      46344        proc-vmstat.nr_active_file
    184427            -1.0%     182501        proc-vmstat.nr_anon_pages
   1524685           -39.0%     929317        proc-vmstat.nr_dirtied
    237179            -7.3%     219853        proc-vmstat.nr_dirty
    549481            -1.4%     541699        proc-vmstat.nr_file_pages
     12694            -1.5%      12509        proc-vmstat.nr_inactive_anon
    200961            +4.5%     209929        proc-vmstat.nr_inactive_file
     12998            -1.3%      12827        proc-vmstat.nr_shmem
     15797            +5.0%      16586        proc-vmstat.nr_slab_reclaimable
      9867            +7.5%      10606        proc-vmstat.nr_slab_unreclaimable
    146324           -12.5%     128075        proc-vmstat.nr_written
    184693            -1.0%     182777        proc-vmstat.nr_zone_active_anon
     62960           -26.4%      46331        proc-vmstat.nr_zone_active_file
     12694            -1.5%      12509        proc-vmstat.nr_zone_inactive_anon
    200961            +4.5%     209932        proc-vmstat.nr_zone_inactive_file
    237240            -7.3%     219914        proc-vmstat.nr_zone_write_pending
   1646865           -35.9%    1055668        proc-vmstat.numa_hit
   1646865           -35.9%    1055668        proc-vmstat.numa_local
    312689           -47.8%     163203        proc-vmstat.pgactivate
   1780491           -33.1%    1191519        proc-vmstat.pgalloc_normal
   1458066 ± 11%     -20.1%    1164659        proc-vmstat.pgfree
    655093            -9.2%     594862        proc-vmstat.pgpgout
      1504           -21.6%       1180        sched_debug.cfs_rq:/.exec_clock.avg
      2573           -10.6%       2300        sched_debug.cfs_rq:/.exec_clock.max
    988.77 ±  5%     -28.5%     707.31 ±  9%  sched_debug.cfs_rq:/.exec_clock.min
    557.42 ±  3%      -9.0%     507.07 ±  5%  sched_debug.cfs_rq:/.exec_clock.stddev
      1445 ±135%    -538.7%      -6338        sched_debug.cfs_rq:/.spread0.avg
     12463 ± 21%     -62.6%       4658 ± 65%  sched_debug.cfs_rq:/.spread0.max
     -4954          +170.5%     -13399        sched_debug.cfs_rq:/.spread0.min
    292.58 ± 10%     -21.7%     229.12 ±  3%  sched_debug.cfs_rq:/.util_avg.stddev
    445084 ± 15%     +22.5%     545245 ±  7%  sched_debug.cpu.avg_idle.avg
     29625 ± 24%     +94.4%      57586 ± 16%  sched_debug.cpu.avg_idle.min
    324202 ±  2%     +10.5%     358398 ±  2%  sched_debug.cpu.avg_idle.stddev
      0.00 ± 16%     -55.8%       0.00 ± 59%  sched_debug.cpu.next_balance.stddev
    113960           -28.8%      81153 ±  2%  sched_debug.cpu.nr_switches.min
     22984 ±  5%     +60.3%      36841 ± 13%  sched_debug.cpu.nr_switches.stddev
    191.33           -10.2%     171.83        sched_debug.cpu.nr_uninterruptible.max
    110503           -32.3%      74797 ±  3%  sched_debug.cpu.sched_count.min
     20460 ±  4%     +75.8%      35970 ± 13%  sched_debug.cpu.sched_count.stddev
     53089 ±  2%     -31.4%      36440 ±  4%  sched_debug.cpu.sched_goidle.min
     10362 ±  4%     +75.0%      18138 ± 13%  sched_debug.cpu.sched_goidle.stddev
     56745 ±  2%     -34.0%      37460 ±  4%  sched_debug.cpu.ttwu_count.min
      9660 ±  8%     +83.2%      17701 ± 13%  sched_debug.cpu.ttwu_count.stddev
     79144 ±  2%     +14.2%      90399        sched_debug.cpu.ttwu_local.max
     47940           -31.0%      33087 ±  4%  sched_debug.cpu.ttwu_local.min
      9991 ±  5%     +80.1%      17992 ± 13%  sched_debug.cpu.ttwu_local.stddev
      5547 ±  9%     +30.1%       7214 ±  6%  slabinfo.anon_vma_chain.active_objs
      6073 ±  9%     +21.9%       7401 ±  7%  slabinfo.anon_vma_chain.num_objs
      3807           +12.0%       4265 ±  4%  slabinfo.filp.num_objs
    725.33 ±  8%     -23.5%     554.67 ± 10%  slabinfo.fsnotify_mark_connector.num_objs
      1082 ±  4%     +24.7%       1350 ±  2%  slabinfo.kmalloc-128.active_objs
      1082 ±  4%     +29.8%       1405        slabinfo.kmalloc-128.num_objs
     16221 ±  2%     +13.8%      18455        slabinfo.kmalloc-16.num_objs
      1680           +12.3%       1887        slabinfo.kmalloc-1k.active_objs
      2141           +20.7%       2584        slabinfo.kmalloc-1k.num_objs
      1167           +10.5%       1289        slabinfo.kmalloc-2k.active_objs
      1194           +14.8%       1371        slabinfo.kmalloc-2k.num_objs
    264.00           +60.7%     424.33        slabinfo.kmalloc-4k.active_objs
    280.67           +81.6%     509.67        slabinfo.kmalloc-4k.num_objs
      4536           +32.8%       6024 ±  2%  slabinfo.kmalloc-512.active_objs
      2765 ±  2%     -12.9%       2408        slabinfo.lsm_file_cache.active_objs
      2765 ±  2%     -12.9%       2408        slabinfo.lsm_file_cache.num_objs
    299.67 ±  5%     +94.9%     584.00 ±  4%  slabinfo.xfs_efd_item.active_objs
    299.67 ±  5%    +106.5%     618.67 ±  3%  slabinfo.xfs_efd_item.num_objs
      8578           +30.2%      11172        slabinfo.xfs_ili.active_objs
    409.00           +31.1%     536.33        slabinfo.xfs_ili.active_slabs
      8594           +31.2%      11271        slabinfo.xfs_ili.num_objs
    409.00           +31.1%     536.33        slabinfo.xfs_ili.num_slabs
      8565           +30.7%      11193        slabinfo.xfs_inode.active_objs
      8601           +31.4%      11304        slabinfo.xfs_inode.num_objs


                                                                                
                   filebench.time.percent_of_cpu_this_job_got                   
                                                                                
  20 +----------------------------------------------------------------------+   
  18 |-++.+  +.+   ++   ++.+  +   ++.+.+.++   +  +.++.+.++.+  +.+.+.+  +.+  |   
     |  : :  : :   ::   :  :  :   :       :   :  :         :  :     :  : :  |   
  16 |-+: :  : :   ::   :  :  :   :       :   :  :         :  :     :  : :  |   
  14 |-+: :  : :   ::   :  :  :   :       :   :  :         :  :     :  : :  |   
     |  :  : :  : :  : :    : :: :         : :: :           : :     : :  : :|   
  12 |-+:  : :  : :  : :    : :: :         : :: :           : :     : :  : :|   
  10 |-+:  : :  : :  : :O   : ::O: O      O:O::O:O          : :     : :  : :|   
   8 |-:   ::O O:O: O: :   O:: : :   O O O : : ::           ::       ::   ::|   
     | :   ::   : :  : :    :: : :         : : ::           ::       ::   ::|   
   6 |-:   ::   : :  : :    :: : :         : : ::           ::       ::   ::|   
   4 |-:    :    :    :      :  :           :  :             :       :    : |   
     | :    :    :    :      :  :           :  :             :       :    : |   
   2 |-:    :    :    :      :  :           :  :             :       :    : |   
   0 +----------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                      filebench.time.voluntary_context_switches                 
                                                                                
  140000 +------------------------------------------------------------------+   
         |  +.+  +.+  ++   ++.+  +   .+ .+.++  +   .+. +.++.+  +.++.   +.+  |   
  120000 |-+: :  : :  ::   :  :  :  +  +    :  :  +   +     :  :    +  : :  |   
         |  : :  : :  ::   :  :  :  :       :  :  :         :  :    :  : :  |   
  100000 |-+: :  : :  ::   :  :  :  :       :  :  :         :  :    :  : :  |   
         |  : : :  : :  : :   : : : :        : :: :          : :     : : : :|   
   80000 |-+: : :  : :  : :   : : : :        : :: :          : :     : : : :|   
         |  : : :  : :  : :   : : : :        : :: :          : :     : : : :|   
   60000 |-:   ::   ::  : :    :: : :        :: : :          ::      ::   ::|   
         | :O  ::O O:: O: :O  O:: ::  OO O OO:: ::O          ::      ::   ::|   
   40000 |-:   ::   ::  : :    :: ::         :: ::           ::      ::   ::|   
         | :   :    :    :     :   :          :  :            :       :   : |   
   20000 |-:   :    :    :     :   :          :  :            :       :   : |   
         | :   :    :    :     :   :          :  :            :       :   : |   
       0 +------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                          filebench.time.file_system_outputs                    
                                                                                
  1.4e+07 +-----------------------------------------------------------------+   
          |                                                                 |   
  1.2e+07 |-++.+  ++   ++  +.++   +  ++.++.+.+  +  ++.+.++.++  +.+.++  +.+  |   
          |  : :  ::   ::  :  :   :  :       :  :  :        :  :    :  : :  |   
    1e+07 |-+: :  ::   ::  :  :   :  :       :  :  :        :  :    :  : :  |   
          |  : :  : :  : : :   :  :  :       :  :  :         : :     : : :  |   
    8e+06 |-+: : :  : :  : :   : :: :      O : :: :          : :     : : : :|   
          | O: :O:OO:O: O: :  O:O::O: O OO   :O::O:O         : :     : : : :|   
    6e+06 |-+:  ::  : :  : :   : : ::         :: ::          : :     : :  ::|   
          | :   ::  : :  ::    : : ::         :: ::          ::      ::   ::|   
    4e+06 |-:   ::  : :  ::    : : ::         :: ::          ::      ::   ::|   
          | :   ::   ::   :     :: ::         :: ::           :       :   ::|   
    2e+06 |-:   :    :    :     :  :          :  :            :       :   : |   
          | :   :    :    :     :  :          :  :            :       :   : |   
        0 +-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                              filebench.sum_operations                          
                                                                                
  450000 +------------------------------------------------------------------+   
         |        .+       ++.+      .+ .+. +      .+. +. +.+  +.++.   +.+  |   
  400000 |-++.+  + :  ++   :  :  +  +  +   +:  +  +   +  +  :  :    +  : :  |   
  350000 |-+: :  : :  ::   :  :  :  :       :  :  :         :  :    :  : :  |   
         |  : :  : :  ::   :  :  :  :       :  :  :         :  :    :  : :  |   
  300000 |-+: :  : :  : : :   :  :: :        : :: :          : :     : : : :|   
  250000 |-+: : :  : :  : :   : : : :        : :: :          : :     : : : :|   
         | O: : :  :O: O: :O  :O: :O: O    OO:O::O:          : :     : : : :|   
  200000 |-+:  ::O  ::  : :   O:: : :  O O   : :: :          ::      ::   ::|   
  150000 |-:   ::   ::  : :    :: ::         :: ::           ::      ::   ::|   
         | :   ::   ::  : :    :: ::         :: ::           ::      ::   ::|   
  100000 |-:   ::   ::   :     ::  :          :  :            :       :   : |   
   50000 |-:   :    :    :     :   :          :  :            :       :   : |   
         | :   :    :    :     :   :          :  :            :       :   : |   
       0 +------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                            filebench.sum_operations_s                          
                                                                                
  7000 +--------------------------------------------------------------------+   
       |  +.+  + :  +.+  :  :   +  +.++.+.+:  +   ++.+.++.+ :  :    +  : :  |   
  6000 |-+: :  : :  : :  :  :   :  :       :  :   :         :  :    :  : :  |   
       |  : :  : :  : :  :  :   :  :       :  :   :         :  :    :  : :  |   
  5000 |-+: :  :  : : : :    :  :  :        : :   :         : :     : :  :  |   
       |  :  : :  : : : :    : :: :         : :: :          : :     : :  : :|   
  4000 |-+:  : :  : : : :    : :: :         : :: :          : :     : :  : :|   
       | O:  :O: O:O: : :O  O:O::O:  OO O OO:O::O:O          ::     : :   ::|   
  3000 |-:   ::   ::   ::    : : ::         :: : :           ::      ::   ::|   
       | :   ::   ::   ::    : : ::         :: : :           ::      ::   ::|   
  2000 |-:   ::   ::   ::    : : ::         :: : :           ::      ::   ::|   
       | :    :    :   :      :  :           :  :            :       :    : |   
  1000 |-:    :    :   :      :  :           :  :            :       :    : |   
       | :    :    :   :      :  :           :  :            :       :    : |   
     0 +--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                               filebench.sum_reads_s                            
                                                                                
  700 +---------------------------------------------------------------------+   
      |        .+                                          +     .+.        |   
  600 |-++.+  + :  +.+  +.+.+  +  +.+.++.+.+  +  +.+.++.+.+:   ++   +  +.+  |   
      |  : :  : :  : :  :   :  :  :        :  :  :         :   :    :  : :  |   
  500 |-+: :  : :  : :  :   :  :  :        :  :  :         :   :    :  : :  |   
      |  :  : :  : :  : :   :  :: :        :  :: :          :  :    :  : :  |   
  400 |-+:  : :  : :  : :   : : : :        : : : :          : :     : :  : :|   
      |  :  : :  : :  : :   : : : :      O :O: :O:          : :     : :  : :|   
  300 |-O:  :O: O:O: O: :   O:: :O: O OO    :: : :          : :      ::   ::|   
      | :   ::   ::   ::     :: ::          :: ::           : :      ::   ::|   
  200 |-:   ::   ::   ::     :: ::          :: ::           : :      ::   ::|   
      | :    :    :    :     ::  :          ::  :            ::      ::   ::|   
  100 |-:    :    :    :     :   :          :   :            :       :    : |   
      | :    :    :    :     :   :          :   :            :       :    : |   
    0 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                              filebench.sum_writes_s                            
                                                                                
  1400 +--------------------------------------------------------------------+   
       |        .+                                         .+     +.        |   
  1200 |-++.+  + :  +.+  +.++   +  +.++.+.++  +   ++.+.++.+ :  +.+  +  +.+  |   
       |  : :  : :  : :  :  :   :  :       :  :   :         :  :    :  : :  |   
  1000 |-+: :  : :  : :  :  :   :  :       :  :   :         :  :    :  : :  |   
       |  :  : :  : : :  :   :  :  :        : ::  :         :  :    :  : :  |   
   800 |-+:  : :  : : : :    : :: :         : :: :          : :     : :  : :|   
       |  :  : :  : : : :    : :: :       OO:O::O:O         : :     : :  : :|   
   600 |-O:  :O: O:O: O::O  O:O: ::  OO O   : :: :           ::      ::   ::|   
       | :   ::   ::   ::    : : ::         :: : :           ::      ::   ::|   
   400 |-:   ::   ::   ::    : : ::         :: : :           ::      ::   ::|   
       | :    :    :   ::     :: ::          :  ::           ::      ::   ::|   
   200 |-:    :    :   :      :  :           :  :            :       :    : |   
       | :    :    :   :      :  :           :  :            :       :    : |   
     0 +--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                             filebench.sum_bytes_mb_s                           
                                                                                
  180 +---------------------------------------------------------------------+   
      |        .+        .+.+             .+               +   ++.+.   +.+  |   
  160 |-++.+  + :  +.+  +   :  +  +.+.++.+ :  +  +.+.++.+.+:   :    +  : :  |   
  140 |-+: :  : :  : :  :   :  :  :        :  :  :         :   :    :  : :  |   
      |  : :  : :  : :  :   :  :  :        :  :  :         :   :    :  : :  |   
  120 |-+:  : :  : :  : :   :  :: :        :  :: :          : :     : :  :  |   
  100 |-+:  : :  : :  : :   : : : :        : : : :          : :     : :  : :|   
      |  :  : : O: :  : :   : : :O: O    O :O: :O:          : :     : :  : :|   
   80 |-O:  :O:  :O: O: :   O:: : :   OO    :: : :          : :      ::   ::|   
   60 |-:   ::   ::   ::     :: ::          :: ::           : :      ::   ::|   
      | :   ::   ::   ::     :: ::          :: ::           : :      ::   ::|   
   40 |-:    :    :    :     ::  :          ::  :            :       :    ::|   
   20 |-:    :    :    :     :   :          :   :            :       :    : |   
      | :    :    :    :     :   :          :   :            :       :    : |   
    0 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                            filebench.sum_time_ms_op                            
                                                                                
  16 +----------------------------------------------------------------------+   
     |      OO             O         O O                                    |   
  14 |-OO      O O  O   O    O  O  O     OO O  O O                          |   
  12 |-+                                                                    |   
     |                                                                      |   
  10 |-+                                                                    |   
     |                                                                      |   
   8 |-+ .+  +.    ++         +   + .+.+.+    +  +. +.+.+                   |   
     |  + :  : +   ::   ++.+  :   :+      +   :  : +     +.+  +.+.+.+  +.+  |   
   6 |-+:  : : :  :  :  :  :  :: :        :  :: :          :  :     :  : : :|   
   4 |-+:  : :  : :  : :    : :: :         : :: :           : :     : :  : :|   
     |  :  ::   : :  : :    :: : :         : : ::           : :      ::   ::|   
   2 |-:   ::   : :  : :    :: : :         : : ::           ::       ::   ::|   
     | :    :    :    :      :  :           :  :             :       :    : |   
   0 +----------------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample

***************************************************************************************************
lkp-skl-2sp7: 72 threads Intel(R) Xeon(R) Gold 6139 CPU @ 2.30GHz with 128G memory
=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/load/rootfs/tbox_group/test/testcase/ucode:
  gcc-7/performance/1BRD_48G/xfs/x86_64-rhel-7.6/3000/debian-x86_64-20191114.cgz/lkp-skl-2sp7/disk_rr/aim7/0x2000065

commit: 
  ad3a8d2d61 ("xfs: pass around xfs_inode_ag_walk iget/irele helper functions")
  ef6d7398f5 ("xfs: deferred inode inactivation")

ad3a8d2d61135b44 ef6d7398f5c72c4fa030740a755 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :4          100%           4:4     kmsg.XFS(ram#):xlog_verify_grant_tail:space>BBTOB(tail_blocks)
           :4           25%           1:4     dmesg.WARNING:at#for_ip_interrupt_entry/0x
           :4           25%           1:4     dmesg.WARNING:at_ip__fsnotify_parent/0x
         %stddev     %change         %stddev
             \          |                \  
    432696           -23.6%     330548 ±  3%  aim7.jobs-per-min
     41.78           +30.9%      54.70 ±  3%  aim7.time.elapsed_time
     41.78           +30.9%      54.70 ±  3%  aim7.time.elapsed_time.max
      2588          +490.9%      15297 ±  6%  aim7.time.involuntary_context_switches
    152861 ±  2%      -4.6%     145889 ±  3%  aim7.time.minor_page_faults
    776.39          +126.1%       1755 ±  5%  aim7.time.system_time
    273.84           +10.1%     301.44        aim7.time.user_time
    917867           -70.4%     271957 ± 28%  aim7.time.voluntary_context_switches
     25.15 ±  2%      -3.9%      24.16        boot-time.boot
      1493 ±  2%      -3.6%       1440        boot-time.idle
      0.68 ±  8%     -11.8%       0.60        boot-time.smp_boot
     66.67           -25.2%      49.87 ±  2%  iostat.cpu.idle
     24.44           +74.2%      42.59 ±  2%  iostat.cpu.system
      8.88           -15.1%       7.54 ±  2%  iostat.cpu.user
     65.45           -17.2       48.28        mpstat.cpu.all.idle%
     25.34           +18.6       43.95 ±  2%  mpstat.cpu.all.sys%
      9.21            -1.4        7.76 ±  2%  mpstat.cpu.all.usr%
 1.067e+08 ± 76%     -98.4%    1710158 ± 68%  cpuidle.C1.time
   1176245 ± 66%     -96.3%      43552 ± 56%  cpuidle.C1.usage
    204678 ± 78%     -73.0%      55335 ± 31%  cpuidle.POLL.time
     15998 ± 48%     -51.3%       7784 ± 27%  cpuidle.POLL.usage
     66.00           -25.0%      49.50 ±  2%  vmstat.cpu.id
      5778 ± 29%   +2045.0%     123936 ± 18%  vmstat.io.bo
     23.33 ±  4%     +60.7%      37.50 ±  4%  vmstat.procs.r
     43070           -70.1%      12880 ± 29%  vmstat.system.cs
     56.06 ± 16%     +45.6%      81.61 ± 13%  sched_debug.cfs_rq:/.load_avg.avg
    196.38 ± 10%     +25.3%     246.11 ±  8%  sched_debug.cfs_rq:/.load_avg.stddev
     14.30 ± 81%    +196.6%      42.41 ± 23%  sched_debug.cfs_rq:/.removed.load_avg.avg
     95.84 ± 73%    +109.6%     200.92 ± 12%  sched_debug.cfs_rq:/.removed.load_avg.stddev
    661.20 ± 81%    +194.5%       1947 ± 24%  sched_debug.cfs_rq:/.removed.runnable_sum.avg
      4431 ± 73%    +108.1%       9223 ± 12%  sched_debug.cfs_rq:/.removed.runnable_sum.stddev
      5.26 ± 71%    +225.1%      17.10 ± 35%  sched_debug.cfs_rq:/.removed.util_avg.avg
     36.94 ± 71%    +122.5%      82.19 ± 23%  sched_debug.cfs_rq:/.removed.util_avg.stddev
     -4300           -64.1%      -1543        sched_debug.cfs_rq:/.spread0.min
    -68.33           -36.7%     -43.25        sched_debug.cpu.nr_uninterruptible.min
      6602 ±  9%     +25.8%       8308 ±  9%  meminfo.Active(file)
   1184206           -38.3%     730422 ± 10%  meminfo.Dirty
   1202985           -11.3%    1067078        meminfo.Inactive
   1182103           -11.4%    1046786        meminfo.Inactive(file)
    107360           +47.4%     158228        meminfo.KReclaimable
   3867414           +45.8%    5636973 ±  2%  meminfo.Memused
    107360           +47.4%     158228        meminfo.SReclaimable
    233615           +10.9%     259004        meminfo.SUnreclaim
     40672           +15.3%      46904 ±  4%  meminfo.Shmem
    340976           +22.4%     417232        meminfo.Slab
    106984           +24.8%     133480 ±  2%  meminfo.max_used_kB
      3416 ±  7%     +18.2%       4038 ±  5%  numa-meminfo.node0.Active(file)
    594307           -38.7%     364272 ±  7%  numa-meminfo.node0.Dirty
    601477           -12.0%     529314        numa-meminfo.node0.Inactive
    593396           -12.3%     520420        numa-meminfo.node0.Inactive(file)
     52763 ± 22%     +47.8%      77977 ± 25%  numa-meminfo.node0.KReclaimable
     52763 ± 22%     +47.8%      77977 ± 25%  numa-meminfo.node0.SReclaimable
      3283 ± 10%     +29.4%       4250 ±  5%  numa-meminfo.node1.Active(file)
    596942           -38.4%     367701 ± 11%  numa-meminfo.node1.Dirty
    608469           -11.3%     540006 ±  2%  numa-meminfo.node1.Inactive
    595628           -11.3%     528594 ±  2%  numa-meminfo.node1.Inactive(file)
     97217 ± 41%     +33.3%     129616 ± 36%  numa-meminfo.node1.SUnreclaim
    151726 ± 30%     +38.3%     209861 ± 32%  numa-meminfo.node1.Slab
    906.67 ±  7%     +18.9%       1078 ±  5%  numa-vmstat.node0.nr_active_file
  14110740           +10.0%   15526999 ±  3%  numa-vmstat.node0.nr_dirtied
    147732           -38.5%      90864 ±  7%  numa-vmstat.node0.nr_dirty
    147430           -12.0%     129743        numa-vmstat.node0.nr_inactive_file
     13197 ± 22%     +47.8%      19502 ± 25%  numa-vmstat.node0.nr_slab_reclaimable
      8881 ± 69%   +3983.4%     362650 ± 22%  numa-vmstat.node0.nr_written
    889.00 ±  8%     +20.5%       1071 ±  4%  numa-vmstat.node0.nr_zone_active_file
    147421           -12.0%     129743        numa-vmstat.node0.nr_zone_inactive_file
    147593           -38.3%      91058 ±  7%  numa-vmstat.node0.nr_zone_write_pending
  14674076           +11.4%   16352027 ±  4%  numa-vmstat.node0.numa_hit
  14515762           +11.8%   16233066 ±  4%  numa-vmstat.node0.numa_local
    876.67 ± 11%     +22.4%       1073 ±  7%  numa-vmstat.node1.nr_active_file
  14106889           +11.4%   15716199        numa-vmstat.node1.nr_dirtied
    148917           -38.5%      91636 ± 11%  numa-vmstat.node1.nr_dirty
    148532           -11.3%     131813 ±  2%  numa-vmstat.node1.nr_inactive_file
     24315 ± 41%     +33.3%      32401 ± 36%  numa-vmstat.node1.nr_slab_unreclaimable
      7671 ± 70%   +4740.4%     371338 ± 20%  numa-vmstat.node1.nr_written
    877.67 ± 11%     +23.0%       1079 ±  6%  numa-vmstat.node1.nr_zone_active_file
    148518           -11.2%     131816 ±  2%  numa-vmstat.node1.nr_zone_inactive_file
    148746           -38.3%      91772 ± 11%  numa-vmstat.node1.nr_zone_write_pending
  14713499           +12.6%   16568708 ±  2%  numa-vmstat.node1.numa_hit
  14690883           +12.4%   16506391        numa-vmstat.node1.numa_local
      1685 ±  3%     +24.0%       2090 ±  6%  proc-vmstat.nr_active_file
     91708            -1.8%      90082        proc-vmstat.nr_anon_pages
    297606           -38.4%     183277 ±  9%  proc-vmstat.nr_dirty
   3205078            -1.5%    3157711        proc-vmstat.nr_dirty_background_threshold
   6417993            -1.5%    6323145        proc-vmstat.nr_dirty_threshold
    582155            -5.7%     548837        proc-vmstat.nr_file_pages
  31953714            -1.4%   31513698        proc-vmstat.nr_free_pages
      5247            -3.3%       5076        proc-vmstat.nr_inactive_anon
    297058           -11.7%     262306        proc-vmstat.nr_inactive_file
     57835            -5.1%      54883        proc-vmstat.nr_kernel_stack
      8427 ±  2%      -5.6%       7958 ±  2%  proc-vmstat.nr_mapped
     10139           +15.5%      11710 ±  4%  proc-vmstat.nr_shmem
     26818           +47.5%      39554        proc-vmstat.nr_slab_reclaimable
     58483           +10.8%      64782        proc-vmstat.nr_slab_unreclaimable
     27205 ± 69%   +6113.5%    1690420 ± 21%  proc-vmstat.nr_written
      1685 ±  3%     +24.0%       2090 ±  6%  proc-vmstat.nr_zone_active_file
      5247            -3.3%       5076        proc-vmstat.nr_zone_inactive_anon
    297058           -11.7%     262306        proc-vmstat.nr_zone_inactive_file
    297220           -38.2%     183675 ±  9%  proc-vmstat.nr_zone_write_pending
  74823426            +2.3%   76515419        proc-vmstat.numa_hit
  74800073            +2.3%   76492064        proc-vmstat.numa_local
  75988469            +2.0%   77504714        proc-vmstat.pgalloc_normal
    297303 ±  3%     +11.0%     330078 ±  2%  proc-vmstat.pgfault
  75678037            +1.2%   76591927        proc-vmstat.pgfree
    256162 ± 29%   +2705.7%    7187122 ± 20%  proc-vmstat.pgpgout
      1800           +16.9%       2104 ±  3%  slabinfo.blkdev_ioc.active_objs
      1800           +16.9%       2104 ±  3%  slabinfo.blkdev_ioc.num_objs
      3729           +71.0%       6375 ± 10%  slabinfo.kmalloc-128.active_objs
      3729           +73.4%       6464 ± 11%  slabinfo.kmalloc-128.num_objs
     27477          +215.6%      86725 ± 23%  slabinfo.kmalloc-16.active_objs
    107.33          +255.9%     382.00 ± 15%  slabinfo.kmalloc-16.active_slabs
     27477          +256.3%      97902 ± 15%  slabinfo.kmalloc-16.num_objs
    107.33          +255.9%     382.00 ± 15%  slabinfo.kmalloc-16.num_slabs
      4809 ±  5%     +19.6%       5751 ±  4%  slabinfo.kmalloc-1k.active_objs
      4920 ±  3%     +23.2%       6060 ±  5%  slabinfo.kmalloc-1k.num_objs
      1206           +17.5%       1417        slabinfo.kmalloc-4k.active_objs
      1221           +17.8%       1437        slabinfo.kmalloc-4k.num_objs
     10022 ±  3%    +332.0%      43300        slabinfo.kmalloc-512.active_objs
    325.67 ±  3%    +384.6%       1578        slabinfo.kmalloc-512.active_slabs
     10430 ±  3%    +384.4%      50523        slabinfo.kmalloc-512.num_objs
    325.67 ±  3%    +384.6%       1578        slabinfo.kmalloc-512.num_slabs
      1397           +18.4%       1654        slabinfo.kmalloc-8k.active_objs
      1407           +22.7%       1727        slabinfo.kmalloc-8k.num_objs
     33546           +25.7%      42184        slabinfo.radix_tree_node.active_objs
    601.33           +28.5%     772.75 ±  2%  slabinfo.radix_tree_node.active_slabs
     33709           +28.4%      43292 ±  2%  slabinfo.radix_tree_node.num_objs
    601.33           +28.5%     772.75 ±  2%  slabinfo.radix_tree_node.num_slabs
      2049           +18.9%       2436        slabinfo.xfs_buf.active_objs
      2049           +18.9%       2436        slabinfo.xfs_buf.num_objs
      3137           +49.0%       4673 ± 11%  slabinfo.xfs_buf_item.active_objs
      3137           +49.2%       4679 ± 11%  slabinfo.xfs_buf_item.num_objs
      1720          +102.3%       3479 ± 13%  slabinfo.xfs_efd_item.active_objs
      1720          +107.5%       3569 ± 14%  slabinfo.xfs_efd_item.num_objs
      3321 ±  5%   +1005.6%      36725 ±  2%  slabinfo.xfs_ili.active_objs
     87.33 ±  5%   +1025.6%     983.00 ±  6%  slabinfo.xfs_ili.active_slabs
      3695 ±  5%   +1017.9%      41307 ±  6%  slabinfo.xfs_ili.num_objs
     87.33 ±  5%   +1025.6%     983.00 ±  6%  slabinfo.xfs_ili.num_slabs
      2779         +1208.7%      36369 ±  2%  slabinfo.xfs_inode.active_objs
    104.67         +1109.1%       1265 ±  5%  slabinfo.xfs_inode.active_slabs
      3362         +1104.6%      40507 ±  5%  slabinfo.xfs_inode.num_objs
    104.67         +1109.1%       1265 ±  5%  slabinfo.xfs_inode.num_slabs
     14255 ±  4%     -19.0%      11547 ±  9%  softirqs.CPU0.SCHED
     21629           +14.9%      24848 ±  5%  softirqs.CPU0.TIMER
      9925 ± 15%     -31.6%       6788 ± 11%  softirqs.CPU1.SCHED
     21647 ±  2%     +17.1%      25348 ± 12%  softirqs.CPU1.TIMER
     20440           +15.8%      23675 ±  4%  softirqs.CPU10.TIMER
     20616           +15.6%      23832 ±  3%  softirqs.CPU11.TIMER
     20573           +17.9%      24249 ±  6%  softirqs.CPU12.TIMER
     21026           +13.0%      23765 ±  4%  softirqs.CPU13.TIMER
     20696           +13.1%      23400 ±  4%  softirqs.CPU14.TIMER
     20772 ±  4%     +26.9%      26361 ± 14%  softirqs.CPU15.TIMER
     20871           +15.5%      24104 ±  4%  softirqs.CPU16.TIMER
     21176 ±  2%     +12.5%      23833 ±  4%  softirqs.CPU17.TIMER
     20950 ±  4%     +17.6%      24628 ±  2%  softirqs.CPU18.TIMER
     21617 ±  9%     +14.9%      24841 ±  2%  softirqs.CPU19.TIMER
     28314 ±  5%     -22.9%      21823 ±  4%  softirqs.CPU20.RCU
     21264           +30.4%      27718 ±  5%  softirqs.CPU21.TIMER
     20974 ±  2%     +17.7%      24694        softirqs.CPU22.TIMER
     20310 ±  4%     +19.3%      24227 ±  5%  softirqs.CPU23.TIMER
     21065 ±  4%     +14.6%      24135 ±  4%  softirqs.CPU24.TIMER
     23495 ±  6%      -9.2%      21331        softirqs.CPU25.RCU
     21487 ±  7%     +11.6%      23988 ±  3%  softirqs.CPU25.TIMER
     20931           +17.4%      24581 ±  5%  softirqs.CPU26.TIMER
     20194           +18.5%      23929 ±  4%  softirqs.CPU27.TIMER
     20797           +30.4%      27129 ±  7%  softirqs.CPU28.TIMER
     20860 ±  3%     +14.3%      23842 ±  4%  softirqs.CPU29.TIMER
     25319 ±  8%     -15.5%      21398 ±  2%  softirqs.CPU3.RCU
      9554 ±  4%     -42.6%       5485 ±  7%  softirqs.CPU3.SCHED
     20236           +22.5%      24790 ±  6%  softirqs.CPU30.TIMER
     20868 ±  2%     +16.6%      24331 ±  4%  softirqs.CPU31.TIMER
     19891           +20.3%      23923 ±  3%  softirqs.CPU32.TIMER
     20094           +19.4%      23996 ±  2%  softirqs.CPU33.TIMER
     20475 ±  2%     +22.4%      25054 ±  8%  softirqs.CPU34.TIMER
     20382           +19.3%      24315 ±  2%  softirqs.CPU35.TIMER
     19879           +21.3%      24116 ±  3%  softirqs.CPU36.TIMER
     19874           +35.7%      26969 ± 22%  softirqs.CPU37.TIMER
     19876 ±  2%     +31.7%      26180 ± 18%  softirqs.CPU38.TIMER
     20143 ±  2%     +17.5%      23659 ±  3%  softirqs.CPU39.TIMER
     21161 ±  3%     +15.8%      24500 ±  7%  softirqs.CPU4.TIMER
      9108 ± 12%     -39.8%       5480 ±  3%  softirqs.CPU40.SCHED
     20063           +18.3%      23727 ±  3%  softirqs.CPU41.TIMER
     20121 ±  2%     +16.5%      23436 ±  4%  softirqs.CPU42.TIMER
     20264           +17.0%      23717 ±  4%  softirqs.CPU44.TIMER
     19914           +17.3%      23363 ±  4%  softirqs.CPU45.TIMER
     24108 ±  8%     -12.2%      21158        softirqs.CPU46.RCU
     19959 ±  2%     +17.3%      23410 ±  4%  softirqs.CPU47.TIMER
     19958           +18.7%      23681 ±  4%  softirqs.CPU48.TIMER
     20171           +15.4%      23281 ±  4%  softirqs.CPU49.TIMER
     20857 ±  2%     +14.9%      23966 ±  5%  softirqs.CPU5.TIMER
     20106           +15.8%      23274 ±  4%  softirqs.CPU51.TIMER
     20253           +13.8%      23049 ±  4%  softirqs.CPU52.TIMER
     20067           +15.8%      23246 ±  4%  softirqs.CPU53.TIMER
     20115           +20.1%      24156 ±  4%  softirqs.CPU54.TIMER
     19698           +22.7%      24167 ±  4%  softirqs.CPU55.TIMER
     19988           +11.5%      22294 ±  7%  softirqs.CPU56.TIMER
     20085           +18.4%      23780 ±  2%  softirqs.CPU57.TIMER
     20216           +16.5%      23550 ±  3%  softirqs.CPU58.TIMER
     20284 ±  3%     +16.5%      23639 ±  3%  softirqs.CPU59.TIMER
     21191           +14.0%      24159 ±  5%  softirqs.CPU6.TIMER
     20451 ±  4%     +14.8%      23476 ±  3%  softirqs.CPU61.TIMER
     19623           +25.0%      24525 ±  4%  softirqs.CPU63.TIMER
     19637 ±  2%     +36.7%      26838 ± 19%  softirqs.CPU64.TIMER
     20372 ±  4%     +14.4%      23310 ±  3%  softirqs.CPU65.TIMER
     19509           +23.2%      24043 ±  4%  softirqs.CPU66.TIMER
     19708           +19.1%      23477 ±  3%  softirqs.CPU67.TIMER
     19343           +22.1%      23616 ±  3%  softirqs.CPU68.TIMER
     19604 ±  2%     +20.8%      23682 ±  3%  softirqs.CPU69.TIMER
     20579           +18.2%      24325 ±  4%  softirqs.CPU7.TIMER
     19444           +26.9%      24667 ±  6%  softirqs.CPU70.TIMER
     19559 ±  2%     +22.0%      23863 ±  4%  softirqs.CPU71.TIMER
     20559           +16.8%      24017 ±  5%  softirqs.CPU8.TIMER
     20290           +15.5%      23444 ±  4%  softirqs.CPU9.TIMER
      4794 ± 81%     -90.5%     456.50 ± 30%  softirqs.NET_RX
    608899           -32.5%     410938        softirqs.SCHED
   1489843           +16.7%    1739375 ±  3%  softirqs.TIMER
     84524 ±  2%     +32.4%     111915 ±  3%  interrupts.CPU0.LOC:Local_timer_interrupts
     84530 ±  2%     +32.3%     111817 ±  3%  interrupts.CPU1.LOC:Local_timer_interrupts
     84479 ±  2%     +32.5%     111893 ±  3%  interrupts.CPU10.LOC:Local_timer_interrupts
    139.67 ± 11%    +121.4%     309.25 ± 40%  interrupts.CPU10.RES:Rescheduling_interrupts
    120.33 ± 95%     -97.7%       2.75 ± 64%  interrupts.CPU10.TLB:TLB_shootdowns
     84496 ±  2%     +32.4%     111898 ±  3%  interrupts.CPU11.LOC:Local_timer_interrupts
    129.67 ±  2%    +133.1%     302.25 ± 37%  interrupts.CPU11.RES:Rescheduling_interrupts
     84501 ±  2%     +32.4%     111890 ±  3%  interrupts.CPU12.LOC:Local_timer_interrupts
      5020 ±  2%     -14.1%       4313 ±  9%  interrupts.CPU12.NMI:Non-maskable_interrupts
      5020 ±  2%     -14.1%       4313 ±  9%  interrupts.CPU12.PMI:Performance_monitoring_interrupts
    135.00 ±  6%     +91.7%     258.75 ± 16%  interrupts.CPU12.RES:Rescheduling_interrupts
     84295 ±  2%     +32.8%     111914 ±  3%  interrupts.CPU13.LOC:Local_timer_interrupts
      4943           -12.4%       4331 ±  7%  interrupts.CPU13.NMI:Non-maskable_interrupts
      4943           -12.4%       4331 ±  7%  interrupts.CPU13.PMI:Performance_monitoring_interrupts
    127.00 ± 10%    +108.3%     264.50 ± 16%  interrupts.CPU13.RES:Rescheduling_interrupts
     84507 ±  2%     +32.4%     111890 ±  3%  interrupts.CPU14.LOC:Local_timer_interrupts
    127.33 ± 18%    +108.1%     265.00 ±  8%  interrupts.CPU14.RES:Rescheduling_interrupts
     84483 ±  2%     +32.4%     111886 ±  3%  interrupts.CPU15.LOC:Local_timer_interrupts
    136.00 ± 12%     +84.9%     251.50 ± 11%  interrupts.CPU15.RES:Rescheduling_interrupts
     84440 ±  2%     +32.5%     111890 ±  3%  interrupts.CPU16.LOC:Local_timer_interrupts
    135.33 ± 16%    +115.0%     291.00 ± 12%  interrupts.CPU16.RES:Rescheduling_interrupts
     84362 ±  2%     +32.6%     111845 ±  3%  interrupts.CPU17.LOC:Local_timer_interrupts
    143.67 ± 13%     +66.7%     239.50 ±  9%  interrupts.CPU17.RES:Rescheduling_interrupts
     83614 ±  3%     +33.9%     111921 ±  3%  interrupts.CPU18.LOC:Local_timer_interrupts
    164.00 ±  9%     +61.3%     264.50 ± 14%  interrupts.CPU18.RES:Rescheduling_interrupts
     83682 ±  3%     +33.8%     111929 ±  3%  interrupts.CPU19.LOC:Local_timer_interrupts
    138.67 ±  8%     +86.4%     258.50 ± 11%  interrupts.CPU19.RES:Rescheduling_interrupts
     84524 ±  2%     +32.3%     111834 ±  3%  interrupts.CPU2.LOC:Local_timer_interrupts
     83761 ±  3%     +33.6%     111910 ±  3%  interrupts.CPU20.LOC:Local_timer_interrupts
    140.33 ± 14%     +89.0%     265.25 ± 16%  interrupts.CPU20.RES:Rescheduling_interrupts
     83713 ±  3%     +33.7%     111909 ±  3%  interrupts.CPU21.LOC:Local_timer_interrupts
    175.67 ±  2%     +77.8%     312.25 ± 29%  interrupts.CPU21.RES:Rescheduling_interrupts
     83765 ±  3%     +33.6%     111918 ±  3%  interrupts.CPU22.LOC:Local_timer_interrupts
      4998 ±  2%     -11.1%       4442 ±  6%  interrupts.CPU22.NMI:Non-maskable_interrupts
      4998 ±  2%     -11.1%       4442 ±  6%  interrupts.CPU22.PMI:Performance_monitoring_interrupts
    156.00 ± 14%     +68.9%     263.50 ± 16%  interrupts.CPU22.RES:Rescheduling_interrupts
     83752 ±  3%     +33.7%     111973 ±  3%  interrupts.CPU23.LOC:Local_timer_interrupts
    143.00 ±  8%     +69.2%     242.00 ±  7%  interrupts.CPU23.RES:Rescheduling_interrupts
     83718 ±  3%     +33.8%     112007 ±  3%  interrupts.CPU24.LOC:Local_timer_interrupts
    183.67 ± 14%     +36.4%     250.50 ±  9%  interrupts.CPU24.RES:Rescheduling_interrupts
     83740 ±  3%     +33.7%     111946 ±  3%  interrupts.CPU25.LOC:Local_timer_interrupts
    150.33 ±  6%     +75.8%     264.25 ±  7%  interrupts.CPU25.RES:Rescheduling_interrupts
     83755 ±  3%     +33.6%     111935 ±  3%  interrupts.CPU26.LOC:Local_timer_interrupts
    146.00 ±  7%     +95.0%     284.75 ± 20%  interrupts.CPU26.RES:Rescheduling_interrupts
     83759 ±  3%     +33.6%     111927 ±  3%  interrupts.CPU27.LOC:Local_timer_interrupts
    137.33 ±  7%     +76.9%     243.00 ± 12%  interrupts.CPU27.RES:Rescheduling_interrupts
     83754 ±  3%     +33.6%     111902 ±  3%  interrupts.CPU28.LOC:Local_timer_interrupts
    154.33 ± 13%     +74.6%     269.50 ± 10%  interrupts.CPU28.RES:Rescheduling_interrupts
     83901 ±  3%     +33.4%     111891 ±  3%  interrupts.CPU29.LOC:Local_timer_interrupts
    142.33 ±  3%     +82.0%     259.00 ± 11%  interrupts.CPU29.RES:Rescheduling_interrupts
     84538 ±  2%     +32.2%     111781 ±  3%  interrupts.CPU3.LOC:Local_timer_interrupts
     83743 ±  3%     +33.5%     111770 ±  3%  interrupts.CPU30.LOC:Local_timer_interrupts
    148.67 ±  7%     +83.3%     272.50 ± 12%  interrupts.CPU30.RES:Rescheduling_interrupts
     83726 ±  3%     +33.7%     111931 ±  3%  interrupts.CPU31.LOC:Local_timer_interrupts
    147.33 ± 11%     +80.7%     266.25 ±  8%  interrupts.CPU31.RES:Rescheduling_interrupts
     83770 ±  3%     +33.6%     111927 ±  3%  interrupts.CPU32.LOC:Local_timer_interrupts
    147.00 ±  9%     +74.0%     255.75 ± 12%  interrupts.CPU32.RES:Rescheduling_interrupts
     84058 ±  3%     +33.1%     111906 ±  3%  interrupts.CPU33.LOC:Local_timer_interrupts
    141.00 ±  7%     +75.5%     247.50 ±  4%  interrupts.CPU33.RES:Rescheduling_interrupts
     83755 ±  3%     +33.6%     111902 ±  3%  interrupts.CPU34.LOC:Local_timer_interrupts
    138.33 ±  5%     +83.3%     253.50 ±  7%  interrupts.CPU34.RES:Rescheduling_interrupts
     83764 ±  3%     +33.6%     111917 ±  3%  interrupts.CPU35.LOC:Local_timer_interrupts
    146.00 ±  9%     +63.9%     239.25 ± 14%  interrupts.CPU35.RES:Rescheduling_interrupts
     84498 ±  2%     +32.4%     111913 ±  3%  interrupts.CPU36.LOC:Local_timer_interrupts
    138.33 ± 13%     +93.4%     267.50 ± 13%  interrupts.CPU36.RES:Rescheduling_interrupts
     84495 ±  2%     +32.4%     111885 ±  3%  interrupts.CPU37.LOC:Local_timer_interrupts
    123.67 ± 12%     +86.0%     230.00 ± 10%  interrupts.CPU37.RES:Rescheduling_interrupts
     84481 ±  2%     +32.3%     111757 ±  3%  interrupts.CPU38.LOC:Local_timer_interrupts
    135.33 ± 28%     +92.5%     260.50 ± 14%  interrupts.CPU38.RES:Rescheduling_interrupts
     84544 ±  2%     +32.3%     111869 ±  3%  interrupts.CPU39.LOC:Local_timer_interrupts
    123.67 ± 18%     +99.5%     246.75 ±  9%  interrupts.CPU39.RES:Rescheduling_interrupts
     84505 ±  2%     +32.4%     111859 ±  3%  interrupts.CPU4.LOC:Local_timer_interrupts
    146.67 ± 11%    +216.5%     464.25 ± 76%  interrupts.CPU4.RES:Rescheduling_interrupts
     84529 ±  2%     +32.4%     111894 ±  3%  interrupts.CPU40.LOC:Local_timer_interrupts
    156.00 ± 22%     +55.3%     242.25 ± 15%  interrupts.CPU40.RES:Rescheduling_interrupts
     84588 ±  2%     +32.3%     111895 ±  3%  interrupts.CPU41.LOC:Local_timer_interrupts
    132.67 ± 12%     +82.0%     241.50 ± 10%  interrupts.CPU41.RES:Rescheduling_interrupts
     84577 ±  2%     +32.3%     111881 ±  3%  interrupts.CPU42.LOC:Local_timer_interrupts
    124.00 ± 10%    +117.5%     269.75 ± 16%  interrupts.CPU42.RES:Rescheduling_interrupts
     84294 ±  2%     +32.8%     111930 ±  3%  interrupts.CPU43.LOC:Local_timer_interrupts
    128.67 ±  7%    +104.0%     262.50 ± 11%  interrupts.CPU43.RES:Rescheduling_interrupts
     84475 ±  2%     +32.5%     111898 ±  3%  interrupts.CPU44.LOC:Local_timer_interrupts
    119.33 ±  5%    +116.6%     258.50 ± 12%  interrupts.CPU44.RES:Rescheduling_interrupts
     84488 ±  2%     +32.4%     111896 ±  3%  interrupts.CPU45.LOC:Local_timer_interrupts
      5041           -23.5%       3856 ± 29%  interrupts.CPU45.NMI:Non-maskable_interrupts
      5041           -23.5%       3856 ± 29%  interrupts.CPU45.PMI:Performance_monitoring_interrupts
    129.00 ±  6%     +88.2%     242.75 ± 14%  interrupts.CPU45.RES:Rescheduling_interrupts
     84509 ±  2%     +32.6%     112060 ±  3%  interrupts.CPU46.LOC:Local_timer_interrupts
    148.33 ± 17%     +69.9%     252.00 ± 16%  interrupts.CPU46.RES:Rescheduling_interrupts
     84501 ±  2%     +32.5%     111927 ±  3%  interrupts.CPU47.LOC:Local_timer_interrupts
    125.33 ± 14%    +102.9%     254.25 ± 14%  interrupts.CPU47.RES:Rescheduling_interrupts
     84498 ±  2%     +32.5%     111931 ±  3%  interrupts.CPU48.LOC:Local_timer_interrupts
      4902           -11.0%       4360 ±  8%  interrupts.CPU48.NMI:Non-maskable_interrupts
      4902           -11.0%       4360 ±  8%  interrupts.CPU48.PMI:Performance_monitoring_interrupts
    122.67 ± 13%     +95.9%     240.25 ±  9%  interrupts.CPU48.RES:Rescheduling_interrupts
     84484 ±  2%     +32.2%     111664 ±  3%  interrupts.CPU49.LOC:Local_timer_interrupts
    128.67 ± 14%     +89.6%     244.00 ± 17%  interrupts.CPU49.RES:Rescheduling_interrupts
     84461 ±  2%     +32.5%     111899 ±  3%  interrupts.CPU5.LOC:Local_timer_interrupts
    151.67 ±  8%     +66.3%     252.25 ± 11%  interrupts.CPU5.RES:Rescheduling_interrupts
     84433 ±  2%     +32.6%     111917 ±  3%  interrupts.CPU50.LOC:Local_timer_interrupts
    129.00 ± 18%     +91.3%     246.75 ± 13%  interrupts.CPU50.RES:Rescheduling_interrupts
     84509 ±  2%     +32.4%     111888 ±  3%  interrupts.CPU51.LOC:Local_timer_interrupts
    133.67 ± 28%     +84.8%     247.00 ± 12%  interrupts.CPU51.RES:Rescheduling_interrupts
     84432 ±  2%     +32.7%     112070 ±  3%  interrupts.CPU52.LOC:Local_timer_interrupts
    117.33 ± 16%    +121.6%     260.00 ± 12%  interrupts.CPU52.RES:Rescheduling_interrupts
     84480 ±  2%     +32.4%     111884 ±  3%  interrupts.CPU53.LOC:Local_timer_interrupts
    124.00 ± 13%     +89.5%     235.00 ± 12%  interrupts.CPU53.RES:Rescheduling_interrupts
     83624 ±  3%     +33.8%     111898 ±  3%  interrupts.CPU54.LOC:Local_timer_interrupts
    122.67 ± 10%    +103.8%     250.00 ±  7%  interrupts.CPU54.RES:Rescheduling_interrupts
     83674 ±  3%     +33.8%     111922 ±  3%  interrupts.CPU55.LOC:Local_timer_interrupts
    130.33 ±  5%     +77.6%     231.50 ±  7%  interrupts.CPU55.RES:Rescheduling_interrupts
     83792 ±  3%     +33.6%     111913 ±  3%  interrupts.CPU56.LOC:Local_timer_interrupts
    122.00 ±  9%     +93.9%     236.50 ± 13%  interrupts.CPU56.RES:Rescheduling_interrupts
     83743 ±  3%     +33.7%     111929 ±  3%  interrupts.CPU57.LOC:Local_timer_interrupts
    141.33 ± 14%     +74.4%     246.50 ± 16%  interrupts.CPU57.RES:Rescheduling_interrupts
     83410 ±  4%     +34.2%     111913 ±  3%  interrupts.CPU58.LOC:Local_timer_interrupts
    128.33 ± 11%     +95.2%     250.50 ± 12%  interrupts.CPU58.RES:Rescheduling_interrupts
     83784 ±  3%     +33.6%     111901 ±  3%  interrupts.CPU59.LOC:Local_timer_interrupts
    123.00 ± 13%    +103.3%     250.00 ± 10%  interrupts.CPU59.RES:Rescheduling_interrupts
     84532 ±  2%     +32.4%     111904 ±  3%  interrupts.CPU6.LOC:Local_timer_interrupts
     83703 ±  3%     +33.7%     111905 ±  3%  interrupts.CPU60.LOC:Local_timer_interrupts
    107.00 ± 12%    +124.1%     239.75 ±  4%  interrupts.CPU60.RES:Rescheduling_interrupts
    111.67 ±128%     -97.5%       2.75 ± 53%  interrupts.CPU60.TLB:TLB_shootdowns
     83745 ±  3%     +33.5%     111824 ±  3%  interrupts.CPU61.LOC:Local_timer_interrupts
      4893 ±  3%     -10.4%       4382 ±  8%  interrupts.CPU61.NMI:Non-maskable_interrupts
      4893 ±  3%     -10.4%       4382 ±  8%  interrupts.CPU61.PMI:Performance_monitoring_interrupts
    117.00 ± 10%    +105.6%     240.50 ±  8%  interrupts.CPU61.RES:Rescheduling_interrupts
     83766 ±  3%     +33.6%     111899 ±  3%  interrupts.CPU62.LOC:Local_timer_interrupts
    118.00 ± 11%    +102.3%     238.75 ±  9%  interrupts.CPU62.RES:Rescheduling_interrupts
     83593 ±  3%     +33.9%     111895 ±  3%  interrupts.CPU63.LOC:Local_timer_interrupts
    108.33 ± 20%    +137.9%     257.75 ±  2%  interrupts.CPU63.RES:Rescheduling_interrupts
     83746 ±  3%     +33.6%     111905 ±  3%  interrupts.CPU64.LOC:Local_timer_interrupts
    112.00 ±  7%    +135.7%     264.00 ± 13%  interrupts.CPU64.RES:Rescheduling_interrupts
     83730 ±  3%     +33.6%     111890 ±  3%  interrupts.CPU65.LOC:Local_timer_interrupts
    109.33 ±  5%    +125.7%     246.75 ±  7%  interrupts.CPU65.RES:Rescheduling_interrupts
     83698 ±  3%     +33.7%     111879 ±  3%  interrupts.CPU66.LOC:Local_timer_interrupts
    135.33 ±  7%     +76.8%     239.25 ±  6%  interrupts.CPU66.RES:Rescheduling_interrupts
     83748 ±  3%     +33.6%     111889 ±  3%  interrupts.CPU67.LOC:Local_timer_interrupts
    125.00 ± 13%    +117.8%     272.25 ±  8%  interrupts.CPU67.RES:Rescheduling_interrupts
     83754 ±  3%     +33.6%     111902 ±  3%  interrupts.CPU68.LOC:Local_timer_interrupts
    114.00 ± 13%    +134.2%     267.00 ± 10%  interrupts.CPU68.RES:Rescheduling_interrupts
     83761 ±  3%     +33.6%     111881 ±  3%  interrupts.CPU69.LOC:Local_timer_interrupts
    120.00 ± 13%    +127.5%     273.00 ± 23%  interrupts.CPU69.RES:Rescheduling_interrupts
     84486 ±  2%     +32.4%     111895 ±  3%  interrupts.CPU7.LOC:Local_timer_interrupts
    138.67 ±  8%     +74.5%     242.00 ± 11%  interrupts.CPU7.RES:Rescheduling_interrupts
     83748 ±  3%     +33.6%     111884 ±  3%  interrupts.CPU70.LOC:Local_timer_interrupts
    129.67 ± 37%    +102.6%     262.75 ± 10%  interrupts.CPU70.RES:Rescheduling_interrupts
     83818 ±  3%     +33.6%     111984 ±  3%  interrupts.CPU71.LOC:Local_timer_interrupts
    151.67 ± 12%     +74.9%     265.25 ± 12%  interrupts.CPU71.RES:Rescheduling_interrupts
     84500 ±  2%     +32.0%     111523 ±  3%  interrupts.CPU8.LOC:Local_timer_interrupts
    135.00 ± 11%    +459.4%     755.25 ±119%  interrupts.CPU8.RES:Rescheduling_interrupts
     84434 ±  2%     +32.4%     111774 ±  3%  interrupts.CPU9.LOC:Local_timer_interrupts
    136.33 ± 12%     +84.3%     251.25 ± 19%  interrupts.CPU9.RES:Rescheduling_interrupts
   6056002 ±  3%     +33.0%    8056230 ±  3%  interrupts.LOC:Local_timer_interrupts
     14039 ±  4%     +62.0%      22738 ±  7%  interrupts.RES:Rescheduling_interrupts
     22.36 ±  4%     -17.3        5.09 ± 14%  perf-profile.calltrace.cycles-pp.secondary_startup_64
     22.09 ±  4%     -17.1        4.99 ± 14%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
     22.09 ±  4%     -17.1        4.99 ± 14%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
     22.08 ±  4%     -17.1        4.99 ± 14%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     21.79 ±  4%     -17.0        4.79 ± 15%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
     21.80 ±  4%     -17.0        4.82 ± 15%  perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     21.42 ±  4%     -16.9        4.47 ± 17%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     34.65           -11.8       22.84 ±  8%  perf-profile.calltrace.cycles-pp.write
     29.53            -9.9       19.66 ±  8%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
     29.34            -9.8       19.55 ±  8%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     25.25            -8.4       16.81 ±  8%  perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     23.41            -8.4       15.01 ±  7%  perf-profile.calltrace.cycles-pp.read
     24.55            -8.2       16.38 ±  8%  perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     22.42            -7.5       14.96 ±  8%  perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     21.52            -7.2       14.35 ±  8%  perf-profile.calltrace.cycles-pp.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write.do_syscall_64
     18.70            -6.8       11.89 ±  7%  perf-profile.calltrace.cycles-pp.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write
     18.58            -6.8       11.81 ±  7%  perf-profile.calltrace.cycles-pp.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write.vfs_write
     18.01            -6.4       11.65 ±  7%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.read
     17.81            -6.3       11.52 ±  7%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
     14.71            -5.6        9.16 ±  7%  perf-profile.calltrace.cycles-pp.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write
     13.74 ±  2%      -5.0        8.78 ±  7%  perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
     13.05 ±  2%      -4.7        8.34 ±  7%  perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
     11.14 ±  2%      -4.1        7.03 ±  7%  perf-profile.calltrace.cycles-pp.new_sync_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
     10.91 ±  2%      -4.0        6.87 ±  7%  perf-profile.calltrace.cycles-pp.xfs_file_read_iter.new_sync_read.vfs_read.ksys_read.do_syscall_64
     10.75            -3.9        6.84 ±  6%  perf-profile.calltrace.cycles-pp.lseek64
     10.17            -3.8        6.38 ±  7%  perf-profile.calltrace.cycles-pp.xfs_file_buffered_aio_read.xfs_file_read_iter.new_sync_read.vfs_read.ksys_read
      9.30            -3.5        5.79 ±  7%  perf-profile.calltrace.cycles-pp.generic_file_read_iter.xfs_file_buffered_aio_read.xfs_file_read_iter.new_sync_read.vfs_read
      8.02            -3.0        4.99 ±  7%  perf-profile.calltrace.cycles-pp.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
      4.33 ±  2%      -2.1        2.19 ±  8%  perf-profile.calltrace.cycles-pp.close
      4.27 ±  2%      -2.1        2.16 ±  9%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.close
      4.27 ±  2%      -2.1        2.16 ±  9%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.close
      4.25 ±  2%      -2.1        2.14 ±  9%  perf-profile.calltrace.cycles-pp.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.close
      4.24 ±  2%      -2.1        2.14 ±  9%  perf-profile.calltrace.cycles-pp.task_work_run.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.close
      4.24 ±  2%      -2.1        2.13 ±  9%  perf-profile.calltrace.cycles-pp.__fput.task_work_run.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
      4.11 ±  2%      -2.1        2.05 ±  9%  perf-profile.calltrace.cycles-pp.dput.__fput.task_work_run.exit_to_usermode_loop.do_syscall_64
      4.09 ±  2%      -2.1        2.04 ±  9%  perf-profile.calltrace.cycles-pp.__dentry_kill.dput.__fput.task_work_run.exit_to_usermode_loop
      5.46            -1.9        3.57 ±  7%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.lseek64
      5.25            -1.8        3.44 ±  7%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.lseek64
      4.74            -1.6        3.18 ±  8%  perf-profile.calltrace.cycles-pp.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      4.58            -1.5        3.07 ±  8%  perf-profile.calltrace.cycles-pp.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor.iomap_apply
      3.78            -1.4        2.35 ±  6%  perf-profile.calltrace.cycles-pp.copy_page_to_iter.generic_file_read_iter.xfs_file_buffered_aio_read.xfs_file_read_iter.new_sync_read
      3.36 ±  2%      -1.4        1.97 ±  6%  perf-profile.calltrace.cycles-pp.iomap_write_end.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
      3.20            -1.3        1.91 ±  6%  perf-profile.calltrace.cycles-pp.copyout.copy_page_to_iter.generic_file_read_iter.xfs_file_buffered_aio_read.xfs_file_read_iter
      2.17            -1.3        0.90 ±  8%  perf-profile.calltrace.cycles-pp.iomap_set_range_uptodate.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      3.27 ±  2%      -1.3        2.01 ±  9%  perf-profile.calltrace.cycles-pp.evict.__dentry_kill.dput.__fput.task_work_run
      3.10            -1.3        1.85 ±  6%  perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyout.copy_page_to_iter.generic_file_read_iter.xfs_file_buffered_aio_read
      3.24 ±  2%      -1.2        2.00 ±  9%  perf-profile.calltrace.cycles-pp.truncate_inode_pages_range.evict.__dentry_kill.dput.__fput
      2.62            -1.0        1.59 ±  6%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.write
      2.64            -1.0        1.64 ±  5%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.lseek64
      2.65 ±  2%      -1.0        1.66 ±  6%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.read
      2.53            -1.0        1.58 ± 10%  perf-profile.calltrace.cycles-pp.mark_page_accessed.generic_file_read_iter.xfs_file_buffered_aio_read.xfs_file_read_iter.new_sync_read
      2.25            -0.9        1.38 ±  7%  perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.lseek64
      2.26            -0.9        1.40 ±  6%  perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.read
      2.23            -0.8        1.40 ±  8%  perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.write
      1.44            -0.8        0.69 ±  6%  perf-profile.calltrace.cycles-pp.iomap_set_range_uptodate.iomap_write_end.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      1.78 ±  3%      -0.7        1.07 ± 11%  perf-profile.calltrace.cycles-pp.__pagevec_release.truncate_inode_pages_range.evict.__dentry_kill.dput
      2.46            -0.7        1.76 ±  9%  perf-profile.calltrace.cycles-pp.xfs_buffered_write_iomap_begin.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write
      1.67 ±  3%      -0.7        0.98 ± 11%  perf-profile.calltrace.cycles-pp.release_pages.__pagevec_release.truncate_inode_pages_range.evict.__dentry_kill
      0.96            -0.7        0.28 ±100%  perf-profile.calltrace.cycles-pp.__alloc_pages_nodemask.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor
      1.48            -0.7        0.81 ±  5%  perf-profile.calltrace.cycles-pp.pagecache_get_page.generic_file_read_iter.xfs_file_buffered_aio_read.xfs_file_read_iter.new_sync_read
      1.39            -0.6        0.75 ±  5%  perf-profile.calltrace.cycles-pp.find_get_entry.pagecache_get_page.generic_file_read_iter.xfs_file_buffered_aio_read.xfs_file_read_iter
      0.88 ±  2%      -0.6        0.28 ±100%  perf-profile.calltrace.cycles-pp.__activate_page.pagevec_lru_move_fn.activate_page.mark_page_accessed.generic_file_read_iter
      1.66            -0.6        1.06 ± 12%  perf-profile.calltrace.cycles-pp.activate_page.mark_page_accessed.generic_file_read_iter.xfs_file_buffered_aio_read.xfs_file_read_iter
      0.85            -0.6        0.27 ±100%  perf-profile.calltrace.cycles-pp.find_get_entry.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor
      1.58            -0.6        1.01 ± 12%  perf-profile.calltrace.cycles-pp.pagevec_lru_move_fn.activate_page.mark_page_accessed.generic_file_read_iter.xfs_file_buffered_aio_read
      2.24            -0.5        1.70 ±  9%  perf-profile.calltrace.cycles-pp.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor
      1.73 ±  6%      -0.5        1.20 ± 10%  perf-profile.calltrace.cycles-pp.xfs_file_aio_write_checks.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write
      1.49            -0.5        1.02 ±  7%  perf-profile.calltrace.cycles-pp.iov_iter_copy_from_user_atomic.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
      1.26            -0.5        0.80 ±  8%  perf-profile.calltrace.cycles-pp.ksys_lseek.do_syscall_64.entry_SYSCALL_64_after_hwframe.lseek64
      1.40 ±  3%      -0.4        0.97 ±  7%  perf-profile.calltrace.cycles-pp.iomap_set_page_dirty.iomap_write_end.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      0.82            -0.4        0.42 ± 58%  perf-profile.calltrace.cycles-pp.security_file_permission.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.68 ± 13%      -0.4        0.29 ±100%  perf-profile.calltrace.cycles-pp.file_update_time.xfs_file_aio_write_checks.xfs_file_buffered_aio_write.new_sync_write.vfs_write
      1.31            -0.4        0.93 ± 10%  perf-profile.calltrace.cycles-pp.__lru_cache_add.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin
      0.80 ±  4%      -0.4        0.42 ± 58%  perf-profile.calltrace.cycles-pp.__set_page_dirty.iomap_set_page_dirty.iomap_write_end.iomap_write_actor.iomap_apply
      1.26            -0.4        0.88 ± 10%  perf-profile.calltrace.cycles-pp.pagevec_lru_move_fn.__lru_cache_add.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin
      1.20            -0.4        0.83 ±  6%  perf-profile.calltrace.cycles-pp.copyin.iov_iter_copy_from_user_atomic.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      1.02            -0.4        0.67 ±  7%  perf-profile.calltrace.cycles-pp.security_file_permission.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.10            -0.3        0.76 ±  6%  perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyin.iov_iter_copy_from_user_atomic.iomap_write_actor.iomap_apply
      0.87            -0.1        0.73 ±  8%  perf-profile.calltrace.cycles-pp.__add_to_page_cache_locked.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin
      0.69 ±  4%      -0.1        0.61 ±  6%  perf-profile.calltrace.cycles-pp.memset_erms.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      0.65            +0.1        0.77 ±  3%  perf-profile.calltrace.cycles-pp.xfs_generic_create.path_openat.do_filp_open.do_sys_openat2.do_sys_open
      0.64            +0.1        0.76 ±  2%  perf-profile.calltrace.cycles-pp.xfs_create.xfs_generic_create.path_openat.do_filp_open.do_sys_openat2
      0.55            +0.1        0.68 ±  2%  perf-profile.calltrace.cycles-pp.rwsem_spin_on_owner.rwsem_optimistic_spin.rwsem_down_write_slowpath.do_unlinkat.do_syscall_64
      0.00            +0.7        0.69 ±  2%  perf-profile.calltrace.cycles-pp.rwsem_spin_on_owner.rwsem_optimistic_spin.rwsem_down_write_slowpath.path_openat.do_filp_open
      1.82           +21.7       23.51 ±  7%  perf-profile.calltrace.cycles-pp.unlink
      1.80           +21.7       23.50 ±  7%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.unlink
      1.80           +21.7       23.50 ±  7%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink
      1.78           +21.7       23.49 ±  7%  perf-profile.calltrace.cycles-pp.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink
      1.18 ±  2%     +21.8       22.97 ±  7%  perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink
      0.77 ±  2%     +21.8       22.56 ±  8%  perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.path_openat.do_filp_open.do_sys_openat2.do_sys_open
      0.00           +21.8       21.80 ±  8%  perf-profile.calltrace.cycles-pp.osq_lock.rwsem_optimistic_spin.rwsem_down_write_slowpath.path_openat.do_filp_open
      1.71           +21.8       23.52 ±  8%  perf-profile.calltrace.cycles-pp.creat
      1.69           +21.8       23.50 ±  8%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.creat
      1.69           +21.8       23.50 ±  8%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.creat
      1.69           +21.8       23.50 ±  8%  perf-profile.calltrace.cycles-pp.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe.creat
      1.69           +21.8       23.50 ±  8%  perf-profile.calltrace.cycles-pp.do_sys_openat2.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe.creat
      1.67           +21.8       23.48 ±  8%  perf-profile.calltrace.cycles-pp.do_filp_open.do_sys_openat2.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.66           +21.8       23.48 ±  8%  perf-profile.calltrace.cycles-pp.path_openat.do_filp_open.do_sys_openat2.do_sys_open.do_syscall_64
      0.64 ±  2%     +21.9       22.51 ±  8%  perf-profile.calltrace.cycles-pp.rwsem_optimistic_spin.rwsem_down_write_slowpath.path_openat.do_filp_open.do_sys_openat2
      1.02           +21.9       22.91 ±  7%  perf-profile.calltrace.cycles-pp.rwsem_optimistic_spin.rwsem_down_write_slowpath.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00           +22.2       22.21 ±  8%  perf-profile.calltrace.cycles-pp.osq_lock.rwsem_optimistic_spin.rwsem_down_write_slowpath.do_unlinkat.do_syscall_64
     22.36 ±  4%     -17.3        5.09 ± 14%  perf-profile.children.cycles-pp.do_idle
     22.36 ±  4%     -17.3        5.09 ± 14%  perf-profile.children.cycles-pp.secondary_startup_64
     22.36 ±  4%     -17.3        5.09 ± 14%  perf-profile.children.cycles-pp.cpu_startup_entry
     22.08 ±  4%     -17.2        4.92 ± 15%  perf-profile.children.cycles-pp.cpuidle_enter
     22.08 ±  4%     -17.2        4.92 ± 15%  perf-profile.children.cycles-pp.cpuidle_enter_state
     21.68 ±  4%     -17.1        4.57 ± 17%  perf-profile.children.cycles-pp.intel_idle
     22.09 ±  4%     -17.1        4.99 ± 14%  perf-profile.children.cycles-pp.start_secondary
     35.07           -12.0       23.10 ±  8%  perf-profile.children.cycles-pp.write
     23.80            -8.5       15.26 ±  7%  perf-profile.children.cycles-pp.read
     25.27            -8.4       16.83 ±  8%  perf-profile.children.cycles-pp.ksys_write
     24.59            -8.2       16.41 ±  8%  perf-profile.children.cycles-pp.vfs_write
     22.44            -7.5       14.99 ±  8%  perf-profile.children.cycles-pp.new_sync_write
     21.55            -7.2       14.37 ±  8%  perf-profile.children.cycles-pp.xfs_file_buffered_aio_write
     18.71            -6.8       11.90 ±  7%  perf-profile.children.cycles-pp.iomap_file_buffered_write
     18.60            -6.8       11.83 ±  7%  perf-profile.children.cycles-pp.iomap_apply
     14.77            -5.6        9.19 ±  7%  perf-profile.children.cycles-pp.iomap_write_actor
     13.75 ±  2%      -5.0        8.79 ±  7%  perf-profile.children.cycles-pp.ksys_read
     13.07 ±  2%      -4.7        8.36 ±  7%  perf-profile.children.cycles-pp.vfs_read
     11.15 ±  2%      -4.1        7.04 ±  7%  perf-profile.children.cycles-pp.new_sync_read
     11.13            -4.1        7.06 ±  6%  perf-profile.children.cycles-pp.lseek64
     10.92 ±  2%      -4.1        6.87 ±  7%  perf-profile.children.cycles-pp.xfs_file_read_iter
     10.20 ±  2%      -3.8        6.39 ±  7%  perf-profile.children.cycles-pp.xfs_file_buffered_aio_read
      9.33            -3.5        5.80 ±  7%  perf-profile.children.cycles-pp.generic_file_read_iter
      8.05            -3.0        5.01 ±  7%  perf-profile.children.cycles-pp.iomap_write_begin
      7.94            -3.0        4.91 ±  6%  perf-profile.children.cycles-pp.entry_SYSCALL_64
      7.67            -2.9        4.73 ±  7%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      6.10            -2.2        3.90 ±  7%  perf-profile.children.cycles-pp.pagecache_get_page
      4.34 ±  2%      -2.2        2.19 ±  8%  perf-profile.children.cycles-pp.close
      4.24 ±  2%      -2.1        2.14 ±  9%  perf-profile.children.cycles-pp.task_work_run
      4.24 ±  2%      -2.1        2.13 ±  9%  perf-profile.children.cycles-pp.__fput
      4.25 ±  2%      -2.1        2.15 ±  9%  perf-profile.children.cycles-pp.exit_to_usermode_loop
      4.12 ±  2%      -2.1        2.06 ±  9%  perf-profile.children.cycles-pp.dput
      4.09 ±  2%      -2.1        2.04 ±  9%  perf-profile.children.cycles-pp.__dentry_kill
      3.62            -2.0        1.59 ±  7%  perf-profile.children.cycles-pp.iomap_set_range_uptodate
      4.33            -1.6        2.68 ±  6%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      4.76            -1.6        3.20 ±  8%  perf-profile.children.cycles-pp.grab_cache_page_write_begin
      3.81            -1.4        2.36 ±  6%  perf-profile.children.cycles-pp.copy_page_to_iter
      3.37 ±  2%      -1.4        1.98 ±  6%  perf-profile.children.cycles-pp.iomap_write_end
      3.21            -1.3        1.91 ±  6%  perf-profile.children.cycles-pp.copyout
      3.27 ±  2%      -1.3        2.01 ±  9%  perf-profile.children.cycles-pp.evict
      3.25 ±  2%      -1.2        2.00 ±  9%  perf-profile.children.cycles-pp.truncate_inode_pages_range
      2.27            -1.0        1.29 ±  5%  perf-profile.children.cycles-pp.find_get_entry
      2.94            -1.0        1.98 ± 11%  perf-profile.children.cycles-pp.pagevec_lru_move_fn
      2.54            -1.0        1.58 ± 10%  perf-profile.children.cycles-pp.mark_page_accessed
      1.82 ±  3%      -0.7        1.07 ± 10%  perf-profile.children.cycles-pp.release_pages
      2.50            -0.7        1.78 ±  9%  perf-profile.children.cycles-pp.xfs_buffered_write_iomap_begin
      1.78 ±  3%      -0.7        1.07 ± 11%  perf-profile.children.cycles-pp.__pagevec_release
      1.87            -0.6        1.22 ±  8%  perf-profile.children.cycles-pp.security_file_permission
      1.66            -0.6        1.06 ± 12%  perf-profile.children.cycles-pp.activate_page
      1.46 ±  3%      -0.6        0.90 ± 11%  perf-profile.children.cycles-pp.__fdget_pos
      2.25            -0.5        1.70 ±  9%  perf-profile.children.cycles-pp.add_to_page_cache_lru
      1.76 ±  6%      -0.5        1.22 ±  9%  perf-profile.children.cycles-pp.xfs_file_aio_write_checks
      1.57            -0.5        1.04 ±  8%  perf-profile.children.cycles-pp.xfs_ilock
      0.75 ±  4%      -0.5        0.25 ±  7%  perf-profile.children.cycles-pp.__list_del_entry_valid
      1.50            -0.5        1.03 ±  7%  perf-profile.children.cycles-pp.iov_iter_copy_from_user_atomic
      1.28            -0.5        0.81 ±  7%  perf-profile.children.cycles-pp.ksys_lseek
      1.25 ±  2%      -0.5        0.79 ± 11%  perf-profile.children.cycles-pp.common_file_perm
      2.10 ±  2%      -0.5        1.65 ± 14%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      1.17 ±  3%      -0.4        0.72 ± 12%  perf-profile.children.cycles-pp.__fget_light
      1.42 ±  2%      -0.4        0.98 ±  7%  perf-profile.children.cycles-pp.iomap_set_page_dirty
      2.07 ±  6%      -0.4        1.63 ± 14%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      0.97 ±  2%      -0.4        0.54 ±  8%  perf-profile.children.cycles-pp.__alloc_pages_nodemask
      1.06            -0.4        0.65 ±  2%  perf-profile.children.cycles-pp.xas_load
      1.32            -0.4        0.93 ± 10%  perf-profile.children.cycles-pp.__lru_cache_add
      1.22            -0.4        0.84 ±  7%  perf-profile.children.cycles-pp.copyin
      0.80 ±  2%      -0.4        0.41 ±  8%  perf-profile.children.cycles-pp.get_page_from_freelist
      0.92 ±  2%      -0.4        0.54 ±  9%  perf-profile.children.cycles-pp.__activate_page
      0.92            -0.3        0.59 ±  9%  perf-profile.children.cycles-pp.down_write
      0.47            -0.3        0.14 ± 13%  perf-profile.children.cycles-pp.free_unref_page_list
      0.91 ±  2%      -0.3        0.60 ±  8%  perf-profile.children.cycles-pp.xfs_iunlock
      0.94            -0.3        0.66 ±  8%  perf-profile.children.cycles-pp.___might_sleep
      0.81 ±  4%      -0.3        0.54 ±  8%  perf-profile.children.cycles-pp.__set_page_dirty
      0.65            -0.3        0.39 ±  5%  perf-profile.children.cycles-pp.iov_iter_fault_in_readable
      0.53            -0.3        0.28 ± 11%  perf-profile.children.cycles-pp.workingset_activation
      0.53            -0.3        0.28 ±  9%  perf-profile.children.cycles-pp.advance_inactive_age
      0.70 ±  4%      -0.2        0.46 ±  7%  perf-profile.children.cycles-pp.__mod_lruvec_state
      0.59 ±  4%      -0.2        0.35 ±  8%  perf-profile.children.cycles-pp.truncate_cleanup_page
      0.63 ±  4%      -0.2        0.39 ±  8%  perf-profile.children.cycles-pp.xfs_buffered_write_iomap_end
      0.68 ±  2%      -0.2        0.47 ±  7%  perf-profile.children.cycles-pp.fsnotify
      1.11 ±  8%      -0.2        0.90 ± 13%  perf-profile.children.cycles-pp.__xfs_trans_commit
      0.61 ±  7%      -0.2        0.40 ± 11%  perf-profile.children.cycles-pp.account_page_dirtied
      0.59 ±  2%      -0.2        0.38 ±  7%  perf-profile.children.cycles-pp.__pagevec_lru_add_fn
      0.51 ±  5%      -0.2        0.30 ±  8%  perf-profile.children.cycles-pp.__cancel_dirty_page
      1.07 ±  9%      -0.2        0.87 ± 13%  perf-profile.children.cycles-pp.xfs_log_commit_cil
      0.67 ±  8%      -0.2        0.48 ±  9%  perf-profile.children.cycles-pp.touch_atime
      0.42 ±  4%      -0.2        0.24 ±  7%  perf-profile.children.cycles-pp.account_page_cleaned
      0.58            -0.2        0.40 ±  5%  perf-profile.children.cycles-pp.delete_from_page_cache_batch
      0.42 ±  5%      -0.2        0.25 ± 10%  perf-profile.children.cycles-pp.xfs_break_layouts
      0.53 ±  2%      -0.2        0.35 ±  7%  perf-profile.children.cycles-pp.xfs_file_llseek
      0.70 ± 13%      -0.2        0.53 ± 11%  perf-profile.children.cycles-pp.file_update_time
      0.27 ± 18%      -0.2        0.10 ± 38%  perf-profile.children.cycles-pp.start_kernel
      0.22 ±  6%      -0.2        0.05 ± 62%  perf-profile.children.cycles-pp.enqueue_task_fair
      0.22 ±  6%      -0.2        0.05 ± 62%  perf-profile.children.cycles-pp.ttwu_do_activate
      0.22 ±  6%      -0.2        0.05 ± 62%  perf-profile.children.cycles-pp.activate_task
      0.60            -0.2        0.43 ± 10%  perf-profile.children.cycles-pp.xfs_file_write_iter
      0.49            -0.2        0.33 ± 10%  perf-profile.children.cycles-pp.__might_sleep
      0.21 ±  3%      -0.2        0.05 ± 62%  perf-profile.children.cycles-pp.enqueue_entity
      0.89            -0.1        0.74 ±  8%  perf-profile.children.cycles-pp.__add_to_page_cache_locked
      0.22 ±  3%      -0.1        0.08 ± 22%  perf-profile.children.cycles-pp.try_to_wake_up
      0.17 ±  2%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.__account_scheduler_latency
      0.45 ±  2%      -0.1        0.30 ±  6%  perf-profile.children.cycles-pp._cond_resched
      0.42 ±  3%      -0.1        0.28 ±  9%  perf-profile.children.cycles-pp.up_write
      0.61 ± 13%      -0.1        0.48 ± 13%  perf-profile.children.cycles-pp._raw_spin_lock
      0.46 ±  4%      -0.1        0.34 ±  8%  perf-profile.children.cycles-pp.__sb_start_write
      0.43 ±  3%      -0.1        0.30 ±  7%  perf-profile.children.cycles-pp.__fsnotify_parent
      0.52            -0.1        0.40 ± 10%  perf-profile.children.cycles-pp.xas_store
      0.42 ±  2%      -0.1        0.30 ±  9%  perf-profile.children.cycles-pp.current_time
      0.42 ±  2%      -0.1        0.30 ±  7%  perf-profile.children.cycles-pp.down_read
      0.15 ±  6%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.stack_trace_save_tsk
      0.31 ±  4%      -0.1        0.19 ±  4%  perf-profile.children.cycles-pp.xas_start
      0.14 ±  6%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.arch_stack_walk
      0.26 ±  6%      -0.1        0.15 ± 10%  perf-profile.children.cycles-pp.xfs_break_leased_layouts
      0.32 ±  2%      -0.1        0.21 ± 20%  perf-profile.children.cycles-pp.xfs_trans_read_buf_map
      0.18 ±  2%      -0.1        0.08 ± 16%  perf-profile.children.cycles-pp.__sched_text_start
      0.15 ±  3%      -0.1        0.05 ± 62%  perf-profile.children.cycles-pp.xfs_read_agi
      0.25 ±  3%      -0.1        0.15 ± 10%  perf-profile.children.cycles-pp.__mod_node_page_state
      0.36            -0.1        0.26 ±  7%  perf-profile.children.cycles-pp.atime_needs_update
      0.43            -0.1        0.34 ±  7%  perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
      0.15 ±  3%      -0.1        0.06 ± 60%  perf-profile.children.cycles-pp.xfs_buf_item_release
      0.23 ±  2%      -0.1        0.14 ± 20%  perf-profile.children.cycles-pp.xfs_buf_read_map
      0.14            -0.1        0.05 ± 60%  perf-profile.children.cycles-pp.up
      0.23 ±  4%      -0.1        0.14 ± 18%  perf-profile.children.cycles-pp.xfs_buf_get_map
      0.20 ±  2%      -0.1        0.12 ± 21%  perf-profile.children.cycles-pp.xfs_buf_find
      0.14            -0.1        0.05 ± 62%  perf-profile.children.cycles-pp.xfs_buf_unlock
      0.11 ±  4%      -0.1        0.03 ±102%  perf-profile.children.cycles-pp.xfs_inactive_truncate
      0.72 ±  4%      -0.1        0.64 ±  6%  perf-profile.children.cycles-pp.memset_erms
      0.22 ±  3%      -0.1        0.14 ±  5%  perf-profile.children.cycles-pp.unlock_page
      0.22 ±  3%      -0.1        0.14 ±  5%  perf-profile.children.cycles-pp.__x64_sys_write
      0.15 ±  3%      -0.1        0.07 ± 64%  perf-profile.children.cycles-pp.xfs_iunlink_remove
      0.31 ±  8%      -0.1        0.23 ±  9%  perf-profile.children.cycles-pp.percpu_counter_add_batch
      0.14 ±  3%      -0.1        0.06 ± 16%  perf-profile.children.cycles-pp.schedule
      0.23 ±  2%      -0.1        0.16 ±  7%  perf-profile.children.cycles-pp.xfs_bmbt_to_iomap
      0.22            -0.1        0.15 ±  5%  perf-profile.children.cycles-pp.__x64_sys_read
      0.31 ±  6%      -0.1        0.23 ±  9%  perf-profile.children.cycles-pp.__mod_memcg_state
      0.21 ±  2%      -0.1        0.14 ±  8%  perf-profile.children.cycles-pp.__x64_sys_lseek
      0.26            -0.1        0.19 ±  4%  perf-profile.children.cycles-pp.xfs_iext_lookup_extent
      0.21            -0.1        0.14 ± 13%  perf-profile.children.cycles-pp.node_dirty_ok
      0.27 ±  4%      -0.1        0.20 ±  9%  perf-profile.children.cycles-pp.apparmor_file_permission
      0.23            -0.1        0.16 ± 10%  perf-profile.children.cycles-pp.mem_cgroup_commit_charge
      0.23 ±  2%      -0.1        0.16 ±  7%  perf-profile.children.cycles-pp.rcu_all_qs
      0.16            -0.1        0.10 ±  5%  perf-profile.children.cycles-pp.aa_file_perm
      0.20 ±  4%      -0.1        0.14 ±  8%  perf-profile.children.cycles-pp.up_read
      0.19 ±  2%      -0.1        0.12 ± 13%  perf-profile.children.cycles-pp.page_mapping
      0.15 ±  3%      -0.1        0.10 ± 11%  perf-profile.children.cycles-pp.iov_iter_advance
      0.18 ±  2%      -0.1        0.13 ± 11%  perf-profile.children.cycles-pp.timestamp_truncate
      0.08 ± 10%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.dec_zone_page_state
      0.15 ±  5%      -0.1        0.10 ±  8%  perf-profile.children.cycles-pp.pagevec_lookup_entries
      0.15 ±  5%      -0.1        0.10 ±  8%  perf-profile.children.cycles-pp.find_get_entries
      0.08 ±  6%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.xfs_get_extsz_hint
      0.08 ±  6%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.open64
      0.22 ±  3%      -0.0        0.17 ± 11%  perf-profile.children.cycles-pp.generic_write_checks
      0.17            -0.0        0.12 ± 10%  perf-profile.children.cycles-pp.__might_fault
      0.07 ± 17%      -0.0        0.03 ±100%  perf-profile.children.cycles-pp.xfs_log_ticket_ungrant
      0.07            -0.0        0.03 ±100%  perf-profile.children.cycles-pp.__mark_inode_dirty
      0.07            -0.0        0.03 ±100%  perf-profile.children.cycles-pp.kmem_cache_alloc
      0.12 ±  4%      -0.0        0.08 ±  6%  perf-profile.children.cycles-pp.__sb_end_write
      0.17 ±  2%      -0.0        0.13 ± 13%  perf-profile.children.cycles-pp.fpregs_assert_state_consistent
      0.11 ±  4%      -0.0        0.07 ± 11%  perf-profile.children.cycles-pp.__list_add_valid
      0.24 ±  3%      -0.0        0.20 ±  7%  perf-profile.children.cycles-pp.xfs_da3_node_lookup_int
      0.08 ± 10%      -0.0        0.04 ± 58%  perf-profile.children.cycles-pp.times
      0.08 ± 14%      -0.0        0.04 ± 59%  perf-profile.children.cycles-pp.xlog_grant_add_space
      0.10 ±  8%      -0.0        0.06 ±  6%  perf-profile.children.cycles-pp.mem_cgroup_update_lru_size
      0.08 ±  6%      -0.0        0.04 ± 58%  perf-profile.children.cycles-pp.node_page_state
      0.12 ±  4%      -0.0        0.08        perf-profile.children.cycles-pp.rw_verify_area
      0.15            -0.0        0.12 ± 13%  perf-profile.children.cycles-pp.mem_cgroup_charge_statistics
      0.08 ±  6%      -0.0        0.04 ± 58%  perf-profile.children.cycles-pp.alloc_pages_current
      0.09 ±  5%      -0.0        0.06 ± 11%  perf-profile.children.cycles-pp.xfs_free_eofblocks
      0.09 ± 13%      -0.0        0.06 ± 11%  perf-profile.children.cycles-pp.update_load_avg
      0.07 ± 11%      -0.0        0.04 ± 57%  perf-profile.children.cycles-pp.free_unref_page_commit
      0.10 ±  8%      -0.0        0.07 ±  6%  perf-profile.children.cycles-pp.mem_cgroup_uncharge_list
      0.14 ±  3%      -0.0        0.11 ±  8%  perf-profile.children.cycles-pp.workingset_update_node
      0.09 ±  5%      -0.0        0.05 ±  9%  perf-profile.children.cycles-pp.file_modified
      0.11 ±  4%      -0.0        0.08 ±  6%  perf-profile.children.cycles-pp.__xa_set_mark
      0.11 ±  4%      -0.0        0.08 ±  6%  perf-profile.children.cycles-pp.unaccount_page_cache_page
      0.11 ±  7%      -0.0        0.08 ±  8%  perf-profile.children.cycles-pp.xas_init_marks
      0.26 ±  4%      -0.0        0.23 ±  3%  perf-profile.children.cycles-pp.xfs_errortag_test
      0.11 ± 11%      -0.0        0.09 ±  5%  perf-profile.children.cycles-pp.wait_for_stable_page
      0.09 ±  5%      -0.0        0.07 ± 17%  perf-profile.children.cycles-pp.__mod_zone_page_state
      0.10 ±  4%      -0.0        0.07 ± 10%  perf-profile.children.cycles-pp.xfs_release
      0.07 ±  7%      -0.0        0.04 ± 58%  perf-profile.children.cycles-pp.xfs_dir3_leaf_check_int
      0.08 ±  5%      -0.0        0.06 ±  7%  perf-profile.children.cycles-pp.xas_clear_mark
      0.06 ±  7%      -0.0        0.04 ± 57%  perf-profile.children.cycles-pp.xfs_trans_unreserve_and_mod_sb
      0.08            -0.0        0.06 ±  9%  perf-profile.children.cycles-pp.iov_iter_init
      0.13 ±  3%      -0.0        0.11 ±  6%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.12 ±  3%      -0.0        0.10 ±  7%  perf-profile.children.cycles-pp.xfs_da_read_buf
      0.09 ±  9%      -0.0        0.07 ± 10%  perf-profile.children.cycles-pp.xfs_vn_lookup
      0.09 ±  5%      -0.0        0.08 ±  6%  perf-profile.children.cycles-pp.ktime_get_coarse_real_ts64
      0.08 ±  6%      -0.0        0.06 ±  6%  perf-profile.children.cycles-pp.xfs_da3_node_read
      0.02 ±141%      +0.0        0.05 ±  9%  perf-profile.children.cycles-pp.xfs_verify_agino
      0.02 ±141%      +0.0        0.06 ±  7%  perf-profile.children.cycles-pp.xfs_iunlink
      0.12 ±  3%      +0.0        0.17 ± 14%  perf-profile.children.cycles-pp.xfs_buf_item_format
      0.08            +0.0        0.12 ± 17%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.05            +0.0        0.10 ± 15%  perf-profile.children.cycles-pp.xfs_inobt_get_maxrecs
      0.11 ±  8%      +0.1        0.20 ± 11%  perf-profile.children.cycles-pp.__xfs_btree_check_sblock
      0.11            +0.1        0.21 ± 13%  perf-profile.children.cycles-pp.xfs_btree_increment
      0.12 ±  4%      +0.1        0.22 ± 11%  perf-profile.children.cycles-pp.xfs_btree_get_rec
      0.06 ±  8%      +0.1        0.17 ± 52%  perf-profile.children.cycles-pp.xfs_difree_finobt
      0.26            +0.1        0.38 ± 17%  perf-profile.children.cycles-pp.xfs_dialloc
      0.65            +0.1        0.77 ±  3%  perf-profile.children.cycles-pp.xfs_generic_create
      0.64            +0.1        0.76 ±  2%  perf-profile.children.cycles-pp.xfs_create
      0.00            +0.1        0.13 ± 39%  perf-profile.children.cycles-pp.brd_do_bvec
      0.00            +0.1        0.13 ± 35%  perf-profile.children.cycles-pp.xfs_inactive_worker
      0.29            +0.1        0.42 ± 15%  perf-profile.children.cycles-pp.xfs_dir_ialloc
      0.28            +0.1        0.42 ± 16%  perf-profile.children.cycles-pp.xfs_ialloc
      0.00            +0.1        0.13 ± 40%  perf-profile.children.cycles-pp.submit_bio
      0.00            +0.1        0.13 ± 40%  perf-profile.children.cycles-pp.generic_make_request
      0.00            +0.1        0.13 ± 40%  perf-profile.children.cycles-pp.brd_make_request
      0.22            +0.1        0.35 ± 18%  perf-profile.children.cycles-pp.xfs_dialloc_ag
      0.20 ±  4%      +0.1        0.34 ±  9%  perf-profile.children.cycles-pp.xfs_inobt_get_rec
      0.16            +0.1        0.31 ± 10%  perf-profile.children.cycles-pp.xfs_btree_check_sblock
      0.00            +0.2        0.16 ± 36%  perf-profile.children.cycles-pp.__writeback_single_inode
      0.00            +0.2        0.16 ± 36%  perf-profile.children.cycles-pp.do_writepages
      0.00            +0.2        0.16 ± 36%  perf-profile.children.cycles-pp.xfs_vm_writepages
      0.00            +0.2        0.16 ± 36%  perf-profile.children.cycles-pp.wb_workfn
      0.00            +0.2        0.16 ± 36%  perf-profile.children.cycles-pp.wb_writeback
      0.00            +0.2        0.16 ± 36%  perf-profile.children.cycles-pp.writeback_sb_inodes
      1.26            +0.2        1.45 ±  2%  perf-profile.children.cycles-pp.rwsem_spin_on_owner
      0.38 ±  2%      +0.2        0.58 ± 10%  perf-profile.children.cycles-pp.xfs_check_agi_freecount
      0.00            +0.4        0.39 ± 25%  perf-profile.children.cycles-pp.process_one_work
      0.00            +0.4        0.39 ± 24%  perf-profile.children.cycles-pp.worker_thread
      0.02 ±141%      +0.4        0.43 ± 23%  perf-profile.children.cycles-pp.ret_from_fork
      0.02 ±141%      +0.4        0.43 ± 23%  perf-profile.children.cycles-pp.kthread
      0.00            +0.6        0.56 ± 32%  perf-profile.children.cycles-pp.xfs_inode_free_blocks
      0.00            +0.7        0.68 ± 21%  perf-profile.children.cycles-pp.xfs_inactive_inode
      0.00            +0.7        0.69 ± 21%  perf-profile.children.cycles-pp.xfs_ici_walk_fns
      0.00            +0.7        0.69 ± 21%  perf-profile.children.cycles-pp.xfs_ici_walk_ag
      1.82           +21.7       23.52 ±  7%  perf-profile.children.cycles-pp.unlink
      1.78           +21.7       23.49 ±  7%  perf-profile.children.cycles-pp.do_unlinkat
      1.75           +21.8       23.54 ±  7%  perf-profile.children.cycles-pp.do_sys_open
      1.75           +21.8       23.54 ±  7%  perf-profile.children.cycles-pp.do_sys_openat2
      1.72           +21.8       23.52 ±  8%  perf-profile.children.cycles-pp.creat
      1.71           +21.8       23.51 ±  7%  perf-profile.children.cycles-pp.path_openat
      1.71           +21.8       23.52 ±  7%  perf-profile.children.cycles-pp.do_filp_open
     60.95           +23.2       84.17        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     60.40           +23.4       83.83        perf-profile.children.cycles-pp.do_syscall_64
      0.57 ±  3%     +43.5       44.04 ±  8%  perf-profile.children.cycles-pp.osq_lock
      1.96           +43.6       45.53 ±  8%  perf-profile.children.cycles-pp.rwsem_down_write_slowpath
      1.67           +43.8       45.43 ±  8%  perf-profile.children.cycles-pp.rwsem_optimistic_spin
     21.68 ±  4%     -17.1        4.57 ± 17%  perf-profile.self.cycles-pp.intel_idle
     11.35            -3.8        7.58 ±  6%  perf-profile.self.cycles-pp.do_syscall_64
      7.65            -2.9        4.72 ±  7%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      7.02            -2.7        4.36 ±  6%  perf-profile.self.cycles-pp.entry_SYSCALL_64
      3.57            -2.0        1.57 ±  7%  perf-profile.self.cycles-pp.iomap_set_range_uptodate
      4.28            -1.6        2.66 ±  6%  perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      1.30            -0.6        0.71 ±  7%  perf-profile.self.cycles-pp.find_get_entry
      0.74 ±  3%      -0.5        0.25 ±  7%  perf-profile.self.cycles-pp.__list_del_entry_valid
      2.06 ±  5%      -0.4        1.63 ± 14%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      1.13 ±  3%      -0.4        0.70 ± 12%  perf-profile.self.cycles-pp.__fget_light
      1.07 ±  3%      -0.4        0.68 ± 12%  perf-profile.self.cycles-pp.common_file_perm
      0.76 ±  3%      -0.3        0.45 ±  4%  perf-profile.self.cycles-pp.xas_load
      0.91            -0.3        0.64 ±  8%  perf-profile.self.cycles-pp.___might_sleep
      0.64            -0.3        0.38 ±  5%  perf-profile.self.cycles-pp.iov_iter_fault_in_readable
      0.52            -0.3        0.27 ± 11%  perf-profile.self.cycles-pp.advance_inactive_age
      0.71 ±  5%      -0.2        0.46 ±  6%  perf-profile.self.cycles-pp.xfs_file_read_iter
      0.69 ±  2%      -0.2        0.46 ± 10%  perf-profile.self.cycles-pp.iomap_apply
      0.56            -0.2        0.34 ±  5%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.70 ±  4%      -0.2        0.49 ±  8%  perf-profile.self.cycles-pp.generic_file_read_iter
      0.67 ±  2%      -0.2        0.46 ±  6%  perf-profile.self.cycles-pp.fsnotify
      0.48 ±  3%      -0.2        0.29 ± 10%  perf-profile.self.cycles-pp.xfs_buffered_write_iomap_end
      0.80            -0.2        0.61 ±  9%  perf-profile.self.cycles-pp.xfs_buffered_write_iomap_begin
      0.49 ±  6%      -0.2        0.30        perf-profile.self.cycles-pp.write
      0.41 ±  2%      -0.2        0.24 ±  9%  perf-profile.self.cycles-pp.__activate_page
      0.52 ±  3%      -0.2        0.35 ± 11%  perf-profile.self.cycles-pp.iomap_write_actor
      0.51 ±  2%      -0.2        0.34 ±  7%  perf-profile.self.cycles-pp.xfs_file_llseek
      0.45 ±  6%      -0.2        0.29 ±  2%  perf-profile.self.cycles-pp.read
      0.41            -0.2        0.25 ±  8%  perf-profile.self.cycles-pp.lseek64
      0.58 ±  2%      -0.2        0.42 ±  9%  perf-profile.self.cycles-pp.xfs_file_write_iter
      0.40            -0.1        0.26 ±  8%  perf-profile.self.cycles-pp.pagecache_get_page
      0.35 ±  2%      -0.1        0.21 ±  3%  perf-profile.self.cycles-pp.release_pages
      0.44            -0.1        0.30 ±  9%  perf-profile.self.cycles-pp.__might_sleep
      0.40            -0.1        0.26 ±  7%  perf-profile.self.cycles-pp.down_write
      0.38 ±  3%      -0.1        0.24 ±  8%  perf-profile.self.cycles-pp.__pagevec_lru_add_fn
      0.34 ±  2%      -0.1        0.20 ±  5%  perf-profile.self.cycles-pp.iomap_write_end
      0.40 ±  2%      -0.1        0.26 ±  9%  perf-profile.self.cycles-pp.up_write
      0.31            -0.1        0.18 ±  7%  perf-profile.self.cycles-pp.__fdget_pos
      0.37 ±  3%      -0.1        0.25 ±  9%  perf-profile.self.cycles-pp.xfs_file_buffered_aio_write
      0.40 ±  4%      -0.1        0.29 ±  7%  perf-profile.self.cycles-pp.__fsnotify_parent
      0.31 ±  3%      -0.1        0.20 ±  7%  perf-profile.self.cycles-pp.xfs_iunlock
      0.32            -0.1        0.21 ±  9%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.34 ±  4%      -0.1        0.23 ±  5%  perf-profile.self.cycles-pp.mark_page_accessed
      0.35 ±  2%      -0.1        0.25 ±  7%  perf-profile.self.cycles-pp.iomap_write_begin
      0.28 ±  4%      -0.1        0.18 ±  6%  perf-profile.self.cycles-pp.__mod_lruvec_state
      0.28 ±  4%      -0.1        0.18 ±  6%  perf-profile.self.cycles-pp.xas_start
      0.22 ±  4%      -0.1        0.12 ±  7%  perf-profile.self.cycles-pp.ksys_lseek
      0.26 ±  3%      -0.1        0.17 ±  5%  perf-profile.self.cycles-pp.vfs_write
      0.26            -0.1        0.17 ±  8%  perf-profile.self.cycles-pp.xfs_ilock
      0.24 ±  6%      -0.1        0.15 ±  8%  perf-profile.self.cycles-pp.xfs_break_leased_layouts
      0.24 ±  3%      -0.1        0.15 ± 11%  perf-profile.self.cycles-pp.__mod_node_page_state
      0.25            -0.1        0.15 ±  8%  perf-profile.self.cycles-pp.get_page_from_freelist
      0.28            -0.1        0.19 ±  9%  perf-profile.self.cycles-pp.iov_iter_copy_from_user_atomic
      0.21 ± 12%      -0.1        0.13 ± 16%  perf-profile.self.cycles-pp.account_page_dirtied
      0.24            -0.1        0.16 ±  7%  perf-profile.self.cycles-pp.security_file_permission
      0.71 ±  4%      -0.1        0.63 ±  5%  perf-profile.self.cycles-pp.memset_erms
      0.23 ±  4%      -0.1        0.15 ±  5%  perf-profile.self.cycles-pp.down_read
      0.36 ±  2%      -0.1        0.28 ±  5%  perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
      0.25 ±  3%      -0.1        0.17 ±  8%  perf-profile.self.cycles-pp.__sb_start_write
      0.19 ±  2%      -0.1        0.12 ±  7%  perf-profile.self.cycles-pp.iomap_set_page_dirty
      0.20 ±  2%      -0.1        0.13 ±  6%  perf-profile.self.cycles-pp.__x64_sys_write
      0.21 ±  3%      -0.1        0.14 ±  8%  perf-profile.self.cycles-pp.unlock_page
      0.20 ±  4%      -0.1        0.12 ±  8%  perf-profile.self.cycles-pp.new_sync_write
      0.22 ±  4%      -0.1        0.14 ±  7%  perf-profile.self.cycles-pp._cond_resched
      0.23 ±  2%      -0.1        0.16 ±  7%  perf-profile.self.cycles-pp.xfs_bmbt_to_iomap
      0.19 ±  4%      -0.1        0.12 ±  7%  perf-profile.self.cycles-pp.ksys_write
      0.16 ± 10%      -0.1        0.09 ±  9%  perf-profile.self.cycles-pp.account_page_cleaned
      0.26            -0.1        0.19 ±  4%  perf-profile.self.cycles-pp.xfs_iext_lookup_extent
      0.22            -0.1        0.15 ±  8%  perf-profile.self.cycles-pp.vfs_read
      0.20 ±  6%      -0.1        0.14 ±  6%  perf-profile.self.cycles-pp.ksys_read
      0.19            -0.1        0.12 ±  8%  perf-profile.self.cycles-pp.__x64_sys_lseek
      0.16            -0.1        0.10 ±  5%  perf-profile.self.cycles-pp.aa_file_perm
      0.30 ±  6%      -0.1        0.23 ± 11%  perf-profile.self.cycles-pp.__mod_memcg_state
      0.26 ±  4%      -0.1        0.19 ±  6%  perf-profile.self.cycles-pp.copy_page_to_iter
      0.24 ±  6%      -0.1        0.18 ±  6%  perf-profile.self.cycles-pp.apparmor_file_permission
      0.20            -0.1        0.14 ±  6%  perf-profile.self.cycles-pp.__x64_sys_read
      0.19 ±  5%      -0.1        0.12 ±  8%  perf-profile.self.cycles-pp.up_read
      0.18 ±  2%      -0.1        0.12 ± 10%  perf-profile.self.cycles-pp.page_mapping
      0.16 ±  2%      -0.1        0.11 ±  4%  perf-profile.self.cycles-pp.xfs_file_aio_write_checks
      0.16 ±  5%      -0.1        0.10 ±  4%  perf-profile.self.cycles-pp.__add_to_page_cache_locked
      0.19 ±  2%      -0.1        0.14 ± 10%  perf-profile.self.cycles-pp.new_sync_read
      0.14 ±  3%      -0.1        0.09 ±  9%  perf-profile.self.cycles-pp.iov_iter_advance
      0.12 ±  3%      -0.1        0.07 ± 10%  perf-profile.self.cycles-pp.xfs_break_layouts
      0.24 ±  9%      -0.0        0.19 ± 11%  perf-profile.self.cycles-pp.percpu_counter_add_batch
      0.08 ±  6%      -0.0        0.03 ±100%  perf-profile.self.cycles-pp.node_page_state
      0.15 ±  6%      -0.0        0.10 ± 14%  perf-profile.self.cycles-pp.current_time
      0.16 ±  2%      -0.0        0.12 ± 12%  perf-profile.self.cycles-pp.timestamp_truncate
      0.16 ±  2%      -0.0        0.12 ± 10%  perf-profile.self.cycles-pp.rcu_all_qs
      0.11 ±  4%      -0.0        0.07 ± 10%  perf-profile.self.cycles-pp.__sb_end_write
      0.20            -0.0        0.16 ± 12%  perf-profile.self.cycles-pp.xas_store
      0.17 ±  5%      -0.0        0.12 ± 11%  perf-profile.self.cycles-pp.fpregs_assert_state_consistent
      0.14            -0.0        0.10 ± 12%  perf-profile.self.cycles-pp.file_update_time
      0.08            -0.0        0.04 ± 58%  perf-profile.self.cycles-pp.delete_from_page_cache_batch
      0.12 ±  6%      -0.0        0.08 ±  8%  perf-profile.self.cycles-pp.find_get_entries
      0.08 ± 12%      -0.0        0.04 ± 57%  perf-profile.self.cycles-pp.__cancel_dirty_page
      0.16 ±  3%      -0.0        0.12 ± 11%  perf-profile.self.cycles-pp.atime_needs_update
      0.10 ±  4%      -0.0        0.07 ±  7%  perf-profile.self.cycles-pp.__list_add_valid
      0.08 ± 14%      -0.0        0.04 ± 59%  perf-profile.self.cycles-pp.xlog_grant_add_space
      0.13            -0.0        0.09 ± 17%  perf-profile.self.cycles-pp.node_dirty_ok
      0.09 ±  9%      -0.0        0.05 ±  8%  perf-profile.self.cycles-pp.xfs_file_buffered_aio_read
      0.13 ±  3%      -0.0        0.09 ±  7%  perf-profile.self.cycles-pp.workingset_update_node
      0.12 ±  4%      -0.0        0.08        perf-profile.self.cycles-pp.rw_verify_area
      0.08 ±  6%      -0.0        0.04 ± 58%  perf-profile.self.cycles-pp.iov_iter_init
      0.09 ±  5%      -0.0        0.06 ± 11%  perf-profile.self.cycles-pp.generic_write_checks
      0.10 ±  4%      -0.0        0.07 ± 14%  perf-profile.self.cycles-pp.iomap_file_buffered_write
      0.09            -0.0        0.06 ±  7%  perf-profile.self.cycles-pp.truncate_inode_pages_range
      0.09 ±  9%      -0.0        0.06 ±  7%  perf-profile.self.cycles-pp.mem_cgroup_update_lru_size
      0.11 ±  7%      -0.0        0.08 ± 10%  perf-profile.self.cycles-pp.wait_for_stable_page
      0.11 ±  7%      -0.0        0.08 ±  5%  perf-profile.self.cycles-pp.__alloc_pages_nodemask
      0.08            -0.0        0.05 ±  8%  perf-profile.self.cycles-pp.activate_page
      0.07 ±  7%      -0.0        0.04 ± 58%  perf-profile.self.cycles-pp.xfs_dir3_leaf_check_int
      0.08 ±  6%      -0.0        0.05 ±  8%  perf-profile.self.cycles-pp.touch_atime
      0.10            -0.0        0.08 ± 10%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.11            -0.0        0.09 ±  9%  perf-profile.self.cycles-pp.xfs_log_commit_cil
      0.08 ±  6%      -0.0        0.06 ±  9%  perf-profile.self.cycles-pp.xas_clear_mark
      0.07 ± 11%      -0.0        0.05 ±  8%  perf-profile.self.cycles-pp.pagevec_lru_move_fn
      0.08            -0.0        0.07 ±  7%  perf-profile.self.cycles-pp.ktime_get_coarse_real_ts64
      0.06            +0.0        0.10 ± 15%  perf-profile.self.cycles-pp.__xfs_btree_check_sblock
      0.03 ± 70%      +0.1        0.09 ± 16%  perf-profile.self.cycles-pp.xfs_inobt_get_maxrecs
      1.25            +0.2        1.43 ±  2%  perf-profile.self.cycles-pp.rwsem_spin_on_owner
      0.57 ±  3%     +43.2       43.75 ±  8%  perf-profile.self.cycles-pp.osq_lock



***************************************************************************************************
lkp-skl-2sp7: 72 threads Intel(R) Xeon(R) Gold 6139 CPU @ 2.30GHz with 128G memory
=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/load/rootfs/tbox_group/test/testcase/ucode:
  gcc-7/performance/1BRD_48G/xfs/x86_64-rhel-7.6/3000/debian-x86_64-20191114.cgz/lkp-skl-2sp7/disk_wrt/aim7/0x2000065

commit: 
  ad3a8d2d61 ("xfs: pass around xfs_inode_ag_walk iget/irele helper functions")
  ef6d7398f5 ("xfs: deferred inode inactivation")

ad3a8d2d61135b44 ef6d7398f5c72c4fa030740a755 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :4           75%           3:4     kmsg.XFS(ram#):xlog_verify_grant_tail:space>BBTOB(tail_blocks)
         %stddev     %change         %stddev
             \          |                \  
    491983           -35.7%     316419        aim7.jobs-per-min
     36.75           +55.2%      57.06        aim7.time.elapsed_time
     36.75           +55.2%      57.06        aim7.time.elapsed_time.max
      1331 ±  3%    +355.2%       6060 ±  2%  aim7.time.involuntary_context_switches
    414.89          +285.4%       1598        aim7.time.system_time
     87.74            +5.9%      92.95        aim7.time.user_time
    911402           -49.8%     457462        aim7.time.voluntary_context_switches
      7764 ±140%    +100.1%      15534 ± 70%  numa-numastat.node0.other_node
  61754017 ±117%     -96.2%    2329702 ±  6%  cpuidle.C1.time
    740827 ±122%     -90.4%      71356 ±  2%  cpuidle.C1.usage
     81.81           -26.5%      60.16        iostat.cpu.idle
     14.75          +153.7%      37.43        iostat.cpu.system
      3.44           -30.1%       2.40        iostat.cpu.user
     81.13           -22.2       58.89        mpstat.cpu.all.idle%
     15.31           +23.3       38.64        mpstat.cpu.all.sys%
      3.56            -1.1        2.46        mpstat.cpu.all.usr%
     13301 ± 19%     -38.7%       8151 ± 12%  sched_debug.cfs_rq:/.min_vruntime.avg
      6502 ± 20%     -37.1%       4087 ± 28%  sched_debug.cfs_rq:/.min_vruntime.min
      1503 ±  9%     -22.3%       1168 ±  9%  sched_debug.cfs_rq:/.util_avg.max
     81.33           -26.2%      60.00        vmstat.cpu.id
      3359 ±  4%    +138.4%       8008 ±  5%  vmstat.io.bo
     12.67 ±  3%    +110.5%      26.67 ±  4%  vmstat.procs.r
     47928           -56.9%      20677        vmstat.system.cs
     40549 ±  5%     +36.7%      55433 ±  9%  meminfo.AnonHugePages
     25798 ±  3%     +12.5%      29012 ±  2%  meminfo.Inactive
      7703 ± 13%     +23.0%       9471 ±  4%  meminfo.Inactive(file)
    102122           +28.0%     130762        meminfo.KReclaimable
    102122           +28.0%     130762        meminfo.SReclaimable
     30222 ±  2%     +49.5%      45191 ±  2%  meminfo.Shmem
    334337           +12.7%     376959        meminfo.Slab
     71061           -31.4%      48748        meminfo.max_used_kB
    143793 ± 20%     +43.1%     205809 ± 16%  numa-meminfo.node0.Active
    143598 ± 20%     +43.3%     205739 ± 16%  numa-meminfo.node0.Active(anon)
    141249 ± 20%     +44.5%     204098 ± 17%  numa-meminfo.node0.AnonPages
      8634 ± 24%     +41.3%      12197 ±  9%  numa-meminfo.node0.Inactive
    232652 ± 12%     -22.8%     179603 ± 18%  numa-meminfo.node1.Active
    232631 ± 12%     -22.9%     179458 ± 18%  numa-meminfo.node1.Active(anon)
    223072 ± 12%     -30.2%     155683 ± 22%  numa-meminfo.node1.AnonPages
     49548 ± 14%     +45.5%      72071 ± 14%  numa-meminfo.node1.KReclaimable
     49548 ± 14%     +45.5%      72071 ± 14%  numa-meminfo.node1.SReclaimable
     22838 ± 14%     +56.9%      35826 ±  8%  numa-meminfo.node1.Shmem
     35894 ± 20%     +43.4%      51458 ± 16%  numa-vmstat.node0.nr_active_anon
     35309 ± 20%     +44.6%      51047 ± 17%  numa-vmstat.node0.nr_anon_pages
  13619965           +15.4%   15716693 ±  2%  numa-vmstat.node0.nr_dirtied
    924.00 ± 18%     +27.5%       1178 ±  8%  numa-vmstat.node0.nr_inactive_file
      1173 ± 50%    +312.3%       4837 ±  7%  numa-vmstat.node0.nr_written
     35894 ± 20%     +43.4%      51458 ± 16%  numa-vmstat.node0.nr_zone_active_anon
    910.00 ± 18%     +29.4%       1177 ±  7%  numa-vmstat.node0.nr_zone_inactive_file
  14144986           +14.9%   16253886 ±  2%  numa-vmstat.node0.numa_hit
  13993881           +15.0%   16095327 ±  2%  numa-vmstat.node0.numa_local
     58134 ± 12%     -22.7%      44948 ± 18%  numa-vmstat.node1.nr_active_anon
     55744 ± 12%     -30.0%      39032 ± 22%  numa-vmstat.node1.nr_anon_pages
  13700941 ±  3%     +14.5%   15682059 ±  2%  numa-vmstat.node1.nr_dirtied
      5709 ± 14%     +56.5%       8933 ±  8%  numa-vmstat.node1.nr_shmem
     12389 ± 14%     +45.3%      18002 ± 14%  numa-vmstat.node1.nr_slab_reclaimable
    944.00 ± 19%    +322.4%       3987 ± 19%  numa-vmstat.node1.nr_written
     58134 ± 12%     -22.7%      44948 ± 18%  numa-vmstat.node1.nr_zone_active_anon
    788.00 ±  5%     -20.6%     625.33 ±  6%  numa-vmstat.node1.nr_zone_write_pending
  14331122 ±  2%     +14.1%   16354286        numa-vmstat.node1.numa_hit
  14301275 ±  2%     +14.2%   16331660        numa-vmstat.node1.numa_local
     94034            +2.5%      96378        proc-vmstat.nr_active_anon
     91054            -1.1%      90051        proc-vmstat.nr_anon_pages
      1671 ± 10%     -10.1%       1502 ±  7%  proc-vmstat.nr_dirty
    284824            +1.5%     288962        proc-vmstat.nr_file_pages
      4523            +8.1%       4888        proc-vmstat.nr_inactive_anon
      1920 ±  8%     +22.9%       2360 ±  2%  proc-vmstat.nr_inactive_file
     56868            -3.6%      54815        proc-vmstat.nr_kernel_stack
      7478            +3.3%       7723        proc-vmstat.nr_mapped
      7555 ±  2%     +49.2%      11273 ±  2%  proc-vmstat.nr_shmem
     25538           +28.0%      32690        proc-vmstat.nr_slab_reclaimable
     58047            +6.1%      61588        proc-vmstat.nr_slab_unreclaimable
      6231 ± 35%    +221.3%      20018 ± 12%  proc-vmstat.nr_written
     94034            +2.5%      96378        proc-vmstat.nr_zone_active_anon
      4523            +8.1%       4888        proc-vmstat.nr_zone_inactive_anon
      1920 ±  8%     +22.9%       2360 ±  2%  proc-vmstat.nr_zone_inactive_file
      1514 ±  4%     -18.6%       1232        proc-vmstat.nr_zone_write_pending
      7133 ±  6%     +27.4%       9085 ±  4%  proc-vmstat.pgactivate
    290989 ±  7%     +16.4%     338831        proc-vmstat.pgfault
    132844 ±  5%    +257.2%     474470 ±  4%  proc-vmstat.pgpgout
      1729 ±  2%     +31.3%       2270        slabinfo.blkdev_ioc.active_objs
      1729 ±  2%     +31.3%       2270        slabinfo.blkdev_ioc.num_objs
      2232           +14.6%       2557        slabinfo.ip6-frags.active_objs
      2232           +14.6%       2557        slabinfo.ip6-frags.num_objs
      3747           +12.9%       4229        slabinfo.kmalloc-128.active_objs
      3747           +12.9%       4229        slabinfo.kmalloc-128.num_objs
     27818          +221.1%      89333 ±  3%  slabinfo.kmalloc-16.active_objs
    108.67          +226.4%     354.67 ±  3%  slabinfo.kmalloc-16.active_slabs
     27818          +226.6%      90860 ±  3%  slabinfo.kmalloc-16.num_objs
    108.67          +226.4%     354.67 ±  3%  slabinfo.kmalloc-16.num_slabs
      1210           +22.5%       1481        slabinfo.kmalloc-4k.active_objs
      1223           +22.3%       1495        slabinfo.kmalloc-4k.num_objs
     10797          +155.1%      27539 ±  2%  slabinfo.kmalloc-512.active_objs
    346.67          +186.7%     994.00 ±  2%  slabinfo.kmalloc-512.active_slabs
     11112          +186.4%      31820 ±  2%  slabinfo.kmalloc-512.num_objs
    346.67          +186.7%     994.00 ±  2%  slabinfo.kmalloc-512.num_slabs
      1376           +18.2%       1627        slabinfo.kmalloc-8k.active_objs
      1387           +20.9%       1677        slabinfo.kmalloc-8k.num_objs
      1781 ±  4%     +11.7%       1989 ±  4%  slabinfo.skbuff_ext_cache.active_objs
      1781 ±  4%     +11.7%       1989 ±  4%  slabinfo.skbuff_ext_cache.num_objs
      1104 ±  6%     -22.7%     853.33 ± 16%  slabinfo.task_group.active_objs
      1104 ±  6%     -22.7%     853.33 ± 16%  slabinfo.task_group.num_objs
      1837           +14.0%       2094        slabinfo.xfs_btree_cur.active_objs
      1837           +14.0%       2094        slabinfo.xfs_btree_cur.num_objs
      1969           +25.1%       2465        slabinfo.xfs_buf.active_objs
      1969           +25.1%       2465        slabinfo.xfs_buf.num_objs
      3002           +24.3%       3731 ±  2%  slabinfo.xfs_buf_item.active_objs
      3002           +24.3%       3731 ±  2%  slabinfo.xfs_buf_item.num_objs
      1687           +15.5%       1949        slabinfo.xfs_da_state.active_objs
      1687           +15.5%       1949        slabinfo.xfs_da_state.num_objs
      1635           +44.6%       2364        slabinfo.xfs_efd_item.active_objs
      1635           +44.6%       2364        slabinfo.xfs_efd_item.num_objs
      3235 ±  3%    +550.4%      21040 ±  2%  slabinfo.xfs_ili.active_objs
     84.33 ±  3%    +591.7%     583.33        slabinfo.xfs_ili.active_slabs
      3560 ±  3%    +589.1%      24535        slabinfo.xfs_ili.num_objs
     84.33 ±  3%    +591.7%     583.33        slabinfo.xfs_ili.num_slabs
      2657          +672.6%      20529 ±  2%  slabinfo.xfs_inode.active_objs
     99.67          +650.8%     748.33        slabinfo.xfs_inode.active_slabs
      3200          +648.6%      23958        slabinfo.xfs_inode.num_objs
     99.67          +650.8%     748.33        slabinfo.xfs_inode.num_slabs
     18576 ± 11%     +19.4%      22188 ±  3%  softirqs.CPU0.RCU
     19982 ±  2%     +32.9%      26561 ±  3%  softirqs.CPU0.TIMER
     18554 ±  8%     +20.7%      22387        softirqs.CPU1.RCU
     19159 ±  4%     +37.4%      26330 ±  2%  softirqs.CPU1.TIMER
     18460 ±  9%     +15.0%      21229 ±  2%  softirqs.CPU10.RCU
     18790           +35.2%      25395        softirqs.CPU10.TIMER
     18773 ±  4%     +11.8%      20981 ±  2%  softirqs.CPU11.RCU
     19915 ±  5%     +27.4%      25376 ±  2%  softirqs.CPU11.TIMER
     18147 ±  8%     +15.7%      20989        softirqs.CPU12.RCU
     19348 ±  2%     +31.3%      25399 ±  2%  softirqs.CPU12.TIMER
     17829 ±  8%     +15.8%      20653 ±  2%  softirqs.CPU13.RCU
     18343           +38.6%      25426 ±  2%  softirqs.CPU13.TIMER
     18589 ±  8%     +14.4%      21262 ±  2%  softirqs.CPU14.RCU
     18532 ±  3%     +36.6%      25307 ±  2%  softirqs.CPU14.TIMER
     18837 ±  6%     +19.8%      22560 ±  6%  softirqs.CPU15.RCU
     19443 ±  9%     +31.8%      25631 ±  2%  softirqs.CPU15.TIMER
     18559 ±  6%     +15.9%      21511        softirqs.CPU16.RCU
     18781 ±  3%     +35.5%      25442 ±  2%  softirqs.CPU16.TIMER
     18244 ±  6%     +20.4%      21963 ±  3%  softirqs.CPU17.RCU
     18726           +34.4%      25175 ±  3%  softirqs.CPU17.TIMER
     18709 ±  8%     +18.2%      22112 ±  2%  softirqs.CPU18.RCU
     19522 ±  3%     +34.8%      26319 ±  3%  softirqs.CPU18.TIMER
     19313 ±  9%     +46.2%      28231 ± 15%  softirqs.CPU19.TIMER
     18352 ±  7%     +21.3%      22266 ±  3%  softirqs.CPU2.RCU
     18736 ±  2%     +44.3%      27030 ±  8%  softirqs.CPU2.TIMER
     18642 ±  6%     +32.1%      24621 ±  5%  softirqs.CPU20.TIMER
     23362 ± 15%     +20.7%      28196 ±  6%  softirqs.CPU21.TIMER
     19243 ± 11%     +17.6%      22627 ±  5%  softirqs.CPU22.RCU
     19539 ±  3%     +40.2%      27389 ±  4%  softirqs.CPU22.TIMER
     18538 ± 11%     +17.2%      21724 ±  3%  softirqs.CPU23.RCU
     18386 ±  4%     +39.6%      25667 ±  2%  softirqs.CPU23.TIMER
     18438 ±  4%     +39.5%      25713        softirqs.CPU24.TIMER
     18171 ±  7%     +15.8%      21038 ±  3%  softirqs.CPU25.RCU
     18493 ±  2%     +37.3%      25394 ±  2%  softirqs.CPU25.TIMER
     18385 ±  8%     +16.7%      21461 ±  3%  softirqs.CPU26.RCU
     18820 ±  2%     +39.1%      26186        softirqs.CPU26.TIMER
     18325 ±  2%     +37.2%      25142        softirqs.CPU27.TIMER
     18303 ± 10%     +14.0%      20868 ±  3%  softirqs.CPU28.RCU
     18206 ±  3%     +39.1%      25329 ±  3%  softirqs.CPU28.TIMER
     18763 ±  8%     +10.5%      20736 ±  3%  softirqs.CPU29.RCU
     18586 ±  2%     +34.3%      24958 ±  2%  softirqs.CPU29.TIMER
     18652 ±  6%     +11.8%      20846        softirqs.CPU3.RCU
     19151 ±  5%     +32.0%      25283 ±  2%  softirqs.CPU3.TIMER
     18503 ±  2%     +36.3%      25223        softirqs.CPU30.TIMER
     18767 ±  9%     +14.4%      21477        softirqs.CPU31.RCU
     18942           +35.7%      25697        softirqs.CPU31.TIMER
     18365 ± 10%     +15.2%      21156        softirqs.CPU32.RCU
     18418 ±  3%     +36.8%      25194        softirqs.CPU32.TIMER
     18489 ±  2%     +37.5%      25420        softirqs.CPU34.TIMER
     17988 ±  9%     +21.8%      21911        softirqs.CPU35.RCU
     18160           +42.4%      25855        softirqs.CPU35.TIMER
     18436 ±  9%     +13.4%      20911        softirqs.CPU36.RCU
     17982           +35.6%      24379 ±  3%  softirqs.CPU36.TIMER
     18270 ± 12%     +21.5%      22198        softirqs.CPU37.RCU
     18591 ±  2%     +60.1%      29769 ± 17%  softirqs.CPU37.TIMER
     19088 ± 13%     +20.2%      22952 ±  3%  softirqs.CPU38.RCU
     18745 ±  5%     +38.8%      26022 ±  6%  softirqs.CPU38.TIMER
     18245 ± 11%     +17.5%      21442 ±  3%  softirqs.CPU39.RCU
     18583 ±  2%     +34.9%      25069 ±  3%  softirqs.CPU39.TIMER
     18404 ±  8%     +14.5%      21064        softirqs.CPU4.RCU
     18751 ±  2%     +34.8%      25285 ±  2%  softirqs.CPU4.TIMER
     18364 ±  9%     +18.2%      21706 ±  2%  softirqs.CPU40.RCU
     18521           +36.9%      25353 ±  5%  softirqs.CPU40.TIMER
     18033 ±  9%     +17.5%      21184        softirqs.CPU41.RCU
     18345           +35.6%      24884 ±  3%  softirqs.CPU41.TIMER
     18680 ±  5%     +12.9%      21087        softirqs.CPU42.RCU
     18811 ±  3%     +32.8%      24974 ±  3%  softirqs.CPU42.TIMER
     18482 ±  9%     +15.9%      21427        softirqs.CPU43.RCU
     17937 ±  2%     +39.4%      25007 ±  4%  softirqs.CPU43.TIMER
     18356 ±  6%     +17.9%      21649 ±  3%  softirqs.CPU44.RCU
     18768           +34.0%      25157 ±  4%  softirqs.CPU44.TIMER
     18256 ±  8%     +15.2%      21026 ±  3%  softirqs.CPU45.RCU
     18248           +35.9%      24802 ±  3%  softirqs.CPU45.TIMER
     17971 ± 10%     +18.4%      21280        softirqs.CPU46.RCU
     18335           +35.6%      24865 ±  3%  softirqs.CPU46.TIMER
     17962 ±  8%     +14.4%      20557 ±  2%  softirqs.CPU47.RCU
     18230 ±  2%     +36.5%      24889 ±  2%  softirqs.CPU47.TIMER
     17621 ±  9%     +20.3%      21191 ±  3%  softirqs.CPU48.RCU
     18469           +35.1%      24954 ±  2%  softirqs.CPU48.TIMER
     17893 ±  9%     +20.8%      21620        softirqs.CPU49.RCU
     18258           +44.9%      26457 ±  5%  softirqs.CPU49.TIMER
     18226 ±  8%     +17.1%      21341 ±  2%  softirqs.CPU5.RCU
     19054 ±  5%     +32.1%      25164 ±  2%  softirqs.CPU5.TIMER
     18341 ±  9%     +15.2%      21129 ±  2%  softirqs.CPU50.RCU
     18540 ±  4%     +33.8%      24813 ±  3%  softirqs.CPU50.TIMER
     17898 ± 11%     +21.6%      21765 ±  5%  softirqs.CPU51.RCU
     17940 ±  2%     +43.5%      25743 ±  2%  softirqs.CPU51.TIMER
     18426 ±  3%     +36.8%      25199 ±  3%  softirqs.CPU52.TIMER
     19024 ±  7%     +35.4%      25760        softirqs.CPU53.TIMER
     18871 ±  2%     +34.3%      25339 ±  2%  softirqs.CPU54.TIMER
     18661 ±  9%     +18.7%      22155 ±  7%  softirqs.CPU55.RCU
     18144           +43.0%      25954 ±  7%  softirqs.CPU55.TIMER
     17744 ±  8%     +20.7%      21418 ±  2%  softirqs.CPU56.RCU
     16517 ±  8%     +50.2%      24801 ±  4%  softirqs.CPU56.TIMER
     18737 ±  8%     +16.9%      21904 ±  5%  softirqs.CPU57.RCU
     18722 ±  5%     +32.8%      24869 ±  6%  softirqs.CPU57.TIMER
     18879 ±  7%     +33.7%      25247        softirqs.CPU58.TIMER
     18047 ± 11%     +20.3%      21719        softirqs.CPU59.RCU
     17831           +41.6%      25255        softirqs.CPU59.TIMER
     17642 ±  9%     +18.4%      20891        softirqs.CPU6.RCU
     18419           +37.4%      25316 ±  3%  softirqs.CPU6.TIMER
     19443 ±  6%     +27.9%      24864        softirqs.CPU60.TIMER
     18129 ±  2%     +37.4%      24910 ±  3%  softirqs.CPU61.TIMER
     17870 ±  8%     +17.2%      20945 ±  3%  softirqs.CPU62.RCU
     17983           +39.8%      25137 ±  2%  softirqs.CPU62.TIMER
     18454 ±  3%     +34.7%      24851 ±  3%  softirqs.CPU63.TIMER
     18627 ±  8%     +11.6%      20789 ±  2%  softirqs.CPU64.RCU
     18557 ±  8%     +12.3%      20831 ±  2%  softirqs.CPU65.RCU
     18139 ±  3%     +56.2%      28335 ± 15%  softirqs.CPU65.TIMER
     17746 ±  9%     +17.6%      20867 ±  3%  softirqs.CPU66.RCU
     17823 ±  2%     +39.5%      24872 ±  2%  softirqs.CPU66.TIMER
     18511 ±  5%     +13.2%      20957        softirqs.CPU67.RCU
     18338 ±  5%     +35.7%      24885 ±  2%  softirqs.CPU67.TIMER
     18296 ±  5%     +16.8%      21376 ±  2%  softirqs.CPU68.RCU
     18003 ±  4%     +39.8%      25166 ±  3%  softirqs.CPU68.TIMER
     17867 ± 11%     +20.8%      21586 ±  4%  softirqs.CPU69.RCU
     17333           +45.9%      25281 ±  4%  softirqs.CPU69.TIMER
     18222 ±  9%     +17.2%      21358        softirqs.CPU7.RCU
     17754 ± 10%     +19.6%      21232 ±  3%  softirqs.CPU70.RCU
     17639           +41.4%      24938 ±  2%  softirqs.CPU70.TIMER
     17966 ±  9%     +17.2%      21053        softirqs.CPU71.RCU
     17626 ±  2%     +40.0%      24682 ±  2%  softirqs.CPU71.TIMER
     18881 ±  2%     +32.6%      25040 ±  2%  softirqs.CPU9.TIMER
   1337363 ±  9%     +15.6%    1545379 ±  2%  softirqs.RCU
   1351267           +36.4%    1842778        softirqs.TIMER
     74781 ±  4%     +56.8%     117224        interrupts.CPU0.LOC:Local_timer_interrupts
      3055 ±  7%     +34.1%       4096 ±  5%  interrupts.CPU0.NMI:Non-maskable_interrupts
      3055 ±  7%     +34.1%       4096 ±  5%  interrupts.CPU0.PMI:Performance_monitoring_interrupts
     74841 ±  4%     +56.6%     117205        interrupts.CPU1.LOC:Local_timer_interrupts
      3048 ±  5%     +35.7%       4135 ±  5%  interrupts.CPU1.NMI:Non-maskable_interrupts
      3048 ±  5%     +35.7%       4135 ±  5%  interrupts.CPU1.PMI:Performance_monitoring_interrupts
     74836 ±  4%     +56.6%     117168        interrupts.CPU10.LOC:Local_timer_interrupts
      2920 ±  6%     +47.2%       4298 ± 10%  interrupts.CPU10.NMI:Non-maskable_interrupts
      2920 ±  6%     +47.2%       4298 ± 10%  interrupts.CPU10.PMI:Performance_monitoring_interrupts
    120.33 ± 12%    +154.3%     306.00 ± 52%  interrupts.CPU10.RES:Rescheduling_interrupts
     74779 ±  4%     +56.8%     117235        interrupts.CPU11.LOC:Local_timer_interrupts
      3126 ± 12%     +32.5%       4143 ±  5%  interrupts.CPU11.NMI:Non-maskable_interrupts
      3126 ± 12%     +32.5%       4143 ±  5%  interrupts.CPU11.PMI:Performance_monitoring_interrupts
    128.00 ± 15%     +38.0%     176.67 ±  7%  interrupts.CPU11.RES:Rescheduling_interrupts
     74772 ±  4%     +56.8%     117208        interrupts.CPU12.LOC:Local_timer_interrupts
    119.33 ± 13%     +49.7%     178.67 ±  5%  interrupts.CPU12.RES:Rescheduling_interrupts
     74603 ±  5%     +57.1%     117210        interrupts.CPU13.LOC:Local_timer_interrupts
      3167 ±  8%     +27.8%       4046 ±  5%  interrupts.CPU13.NMI:Non-maskable_interrupts
      3167 ±  8%     +27.8%       4046 ±  5%  interrupts.CPU13.PMI:Performance_monitoring_interrupts
    103.00 ±  4%     +68.3%     173.33 ±  6%  interrupts.CPU13.RES:Rescheduling_interrupts
     74832 ±  4%     +56.7%     117238        interrupts.CPU14.LOC:Local_timer_interrupts
      3106 ±  9%     +31.2%       4074 ±  6%  interrupts.CPU14.NMI:Non-maskable_interrupts
      3106 ±  9%     +31.2%       4074 ±  6%  interrupts.CPU14.PMI:Performance_monitoring_interrupts
    106.67           +96.6%     209.67 ± 21%  interrupts.CPU14.RES:Rescheduling_interrupts
      1392 ±139%     -99.8%       3.33 ± 56%  interrupts.CPU14.TLB:TLB_shootdowns
     74815 ±  4%     +56.6%     117131        interrupts.CPU15.LOC:Local_timer_interrupts
      2967 ±  5%     +36.3%       4042 ±  6%  interrupts.CPU15.NMI:Non-maskable_interrupts
      2967 ±  5%     +36.3%       4042 ±  6%  interrupts.CPU15.PMI:Performance_monitoring_interrupts
     74720 ±  4%     +56.6%     117025        interrupts.CPU16.LOC:Local_timer_interrupts
      3191 ±  2%     +28.4%       4097 ±  5%  interrupts.CPU16.NMI:Non-maskable_interrupts
      3191 ±  2%     +28.4%       4097 ±  5%  interrupts.CPU16.PMI:Performance_monitoring_interrupts
    136.00 ±  8%     +34.6%     183.00 ± 10%  interrupts.CPU16.RES:Rescheduling_interrupts
     74797 ±  4%     +56.7%     117202        interrupts.CPU17.LOC:Local_timer_interrupts
      2964 ±  6%     +39.1%       4123 ±  5%  interrupts.CPU17.NMI:Non-maskable_interrupts
      2964 ±  6%     +39.1%       4123 ±  5%  interrupts.CPU17.PMI:Performance_monitoring_interrupts
    109.00 ± 26%     +56.9%     171.00 ± 11%  interrupts.CPU17.RES:Rescheduling_interrupts
     74796 ±  5%     +56.7%     117236        interrupts.CPU18.LOC:Local_timer_interrupts
      2440 ± 23%     +69.3%       4132 ±  4%  interrupts.CPU18.NMI:Non-maskable_interrupts
      2440 ± 23%     +69.3%       4132 ±  4%  interrupts.CPU18.PMI:Performance_monitoring_interrupts
    135.33 ± 19%     +34.7%     182.33 ± 14%  interrupts.CPU18.RES:Rescheduling_interrupts
     74597 ±  5%     +57.2%     117242        interrupts.CPU19.LOC:Local_timer_interrupts
      3213 ± 10%     +27.9%       4108 ±  5%  interrupts.CPU19.NMI:Non-maskable_interrupts
      3213 ± 10%     +27.9%       4108 ±  5%  interrupts.CPU19.PMI:Performance_monitoring_interrupts
    130.00 ± 13%     +26.4%     164.33 ± 12%  interrupts.CPU19.RES:Rescheduling_interrupts
     74791 ±  4%     +56.7%     117235        interrupts.CPU2.LOC:Local_timer_interrupts
    119.67 ± 14%    +541.2%     767.33 ± 79%  interrupts.CPU2.RES:Rescheduling_interrupts
     74648 ±  5%     +57.1%     117244        interrupts.CPU20.LOC:Local_timer_interrupts
      3221 ±  6%     +28.0%       4122 ±  5%  interrupts.CPU20.NMI:Non-maskable_interrupts
      3221 ±  6%     +28.0%       4122 ±  5%  interrupts.CPU20.PMI:Performance_monitoring_interrupts
    102.00 ±  5%     +81.4%     185.00 ±  6%  interrupts.CPU20.RES:Rescheduling_interrupts
     74596 ±  5%     +57.2%     117280        interrupts.CPU21.LOC:Local_timer_interrupts
      3207 ±  5%     +29.5%       4154 ±  5%  interrupts.CPU21.NMI:Non-maskable_interrupts
      3207 ±  5%     +29.5%       4154 ±  5%  interrupts.CPU21.PMI:Performance_monitoring_interrupts
    111.00 ± 11%     +65.2%     183.33 ± 13%  interrupts.CPU21.RES:Rescheduling_interrupts
     74663 ±  5%     +57.1%     117277        interrupts.CPU22.LOC:Local_timer_interrupts
      3309 ±  6%     +27.7%       4225 ±  5%  interrupts.CPU22.NMI:Non-maskable_interrupts
      3309 ±  6%     +27.7%       4225 ±  5%  interrupts.CPU22.PMI:Performance_monitoring_interrupts
     74656 ±  5%     +57.1%     117250        interrupts.CPU23.LOC:Local_timer_interrupts
      3138 ±  6%     +31.1%       4116 ±  4%  interrupts.CPU23.NMI:Non-maskable_interrupts
      3138 ±  6%     +31.1%       4116 ±  4%  interrupts.CPU23.PMI:Performance_monitoring_interrupts
    107.67 ±  9%     +62.5%     175.00 ± 13%  interrupts.CPU23.RES:Rescheduling_interrupts
     74646 ±  5%     +57.1%     117264        interrupts.CPU24.LOC:Local_timer_interrupts
      3034 ±  6%     +37.9%       4185        interrupts.CPU24.NMI:Non-maskable_interrupts
      3034 ±  6%     +37.9%       4185        interrupts.CPU24.PMI:Performance_monitoring_interrupts
     96.67 ± 17%     +78.3%     172.33 ±  7%  interrupts.CPU24.RES:Rescheduling_interrupts
     74655 ±  5%     +57.1%     117252        interrupts.CPU25.LOC:Local_timer_interrupts
    123.33 ± 16%     +45.4%     179.33 ± 10%  interrupts.CPU25.RES:Rescheduling_interrupts
     74851 ±  5%     +56.6%     117248        interrupts.CPU26.LOC:Local_timer_interrupts
      3147 ±  6%     +30.0%       4092 ±  7%  interrupts.CPU26.NMI:Non-maskable_interrupts
      3147 ±  6%     +30.0%       4092 ±  7%  interrupts.CPU26.PMI:Performance_monitoring_interrupts
    118.33 ±  7%     +50.7%     178.33 ±  6%  interrupts.CPU26.RES:Rescheduling_interrupts
     74515 ±  4%     +57.4%     117254        interrupts.CPU27.LOC:Local_timer_interrupts
    100.00 ±  4%     +58.3%     158.33 ±  7%  interrupts.CPU27.RES:Rescheduling_interrupts
     74483 ±  4%     +56.9%     116883        interrupts.CPU28.LOC:Local_timer_interrupts
    112.33 ±  5%     +58.2%     177.67 ±  4%  interrupts.CPU28.RES:Rescheduling_interrupts
     74694 ±  5%     +56.9%     117214        interrupts.CPU29.LOC:Local_timer_interrupts
    106.00 ±  5%     +47.5%     156.33 ±  4%  interrupts.CPU29.RES:Rescheduling_interrupts
     74838 ±  4%     +56.6%     117194        interrupts.CPU3.LOC:Local_timer_interrupts
     74661 ±  5%     +57.0%     117232        interrupts.CPU30.LOC:Local_timer_interrupts
    107.67 ±  7%     +65.9%     178.67 ± 14%  interrupts.CPU30.RES:Rescheduling_interrupts
     74643 ±  5%     +57.1%     117237        interrupts.CPU31.LOC:Local_timer_interrupts
    114.33 ±  5%     +52.8%     174.67 ±  7%  interrupts.CPU31.RES:Rescheduling_interrupts
     74660 ±  5%     +57.1%     117255        interrupts.CPU32.LOC:Local_timer_interrupts
     99.33 ± 19%     +68.5%     167.33 ±  7%  interrupts.CPU32.RES:Rescheduling_interrupts
     74659 ±  5%     +57.1%     117267        interrupts.CPU33.LOC:Local_timer_interrupts
    115.33 ±  5%     +30.9%     151.00 ±  3%  interrupts.CPU33.RES:Rescheduling_interrupts
     74546 ±  4%     +57.3%     117253        interrupts.CPU34.LOC:Local_timer_interrupts
     74637 ±  5%     +57.1%     117246        interrupts.CPU35.LOC:Local_timer_interrupts
      2928 ±  4%     +40.0%       4100 ±  7%  interrupts.CPU35.NMI:Non-maskable_interrupts
      2928 ±  4%     +40.0%       4100 ±  7%  interrupts.CPU35.PMI:Performance_monitoring_interrupts
    110.00 ±  3%     +36.7%     150.33 ± 14%  interrupts.CPU35.RES:Rescheduling_interrupts
     74560 ±  4%     +57.2%     117243        interrupts.CPU36.LOC:Local_timer_interrupts
    104.33 ± 19%     +83.4%     191.33 ± 10%  interrupts.CPU36.RES:Rescheduling_interrupts
     74769 ±  4%     +56.8%     117256        interrupts.CPU37.LOC:Local_timer_interrupts
      3171 ±  4%     +29.8%       4116 ±  6%  interrupts.CPU37.NMI:Non-maskable_interrupts
      3171 ±  4%     +29.8%       4116 ±  6%  interrupts.CPU37.PMI:Performance_monitoring_interrupts
    117.67 ± 20%     +46.5%     172.33 ±  9%  interrupts.CPU37.RES:Rescheduling_interrupts
     74761 ±  4%     +56.8%     117242        interrupts.CPU38.LOC:Local_timer_interrupts
      3064 ±  5%     +32.9%       4071 ±  5%  interrupts.CPU38.NMI:Non-maskable_interrupts
      3064 ±  5%     +32.9%       4071 ±  5%  interrupts.CPU38.PMI:Performance_monitoring_interrupts
    105.33 ±  7%     +57.3%     165.67 ±  4%  interrupts.CPU38.RES:Rescheduling_interrupts
     74783 ±  4%     +56.6%     117133        interrupts.CPU39.LOC:Local_timer_interrupts
      3093 ±  2%     +32.1%       4086 ±  6%  interrupts.CPU39.NMI:Non-maskable_interrupts
      3093 ±  2%     +32.1%       4086 ±  6%  interrupts.CPU39.PMI:Performance_monitoring_interrupts
     96.00 ± 10%     +63.2%     156.67 ±  2%  interrupts.CPU39.RES:Rescheduling_interrupts
     74834 ±  4%     +56.6%     117198        interrupts.CPU4.LOC:Local_timer_interrupts
      3063 ±  5%     +33.0%       4074 ±  5%  interrupts.CPU4.NMI:Non-maskable_interrupts
      3063 ±  5%     +33.0%       4074 ±  5%  interrupts.CPU4.PMI:Performance_monitoring_interrupts
    109.00 ± 12%    +374.9%     517.67 ± 91%  interrupts.CPU4.RES:Rescheduling_interrupts
     74794 ±  4%     +56.7%     117210        interrupts.CPU40.LOC:Local_timer_interrupts
      3031 ±  2%     +36.5%       4136 ±  5%  interrupts.CPU40.NMI:Non-maskable_interrupts
      3031 ±  2%     +36.5%       4136 ±  5%  interrupts.CPU40.PMI:Performance_monitoring_interrupts
    100.33 ±  9%     +86.0%     186.67 ±  6%  interrupts.CPU40.RES:Rescheduling_interrupts
     74775 ±  4%     +56.6%     117093        interrupts.CPU41.LOC:Local_timer_interrupts
      3110 ±  6%     +33.3%       4147 ±  6%  interrupts.CPU41.NMI:Non-maskable_interrupts
      3110 ±  6%     +33.3%       4147 ±  6%  interrupts.CPU41.PMI:Performance_monitoring_interrupts
    102.00 ±  7%     +75.2%     178.67 ± 12%  interrupts.CPU41.RES:Rescheduling_interrupts
     74807 ±  4%     +56.7%     117211        interrupts.CPU42.LOC:Local_timer_interrupts
      3187           +30.2%       4150 ±  5%  interrupts.CPU42.NMI:Non-maskable_interrupts
      3187           +30.2%       4150 ±  5%  interrupts.CPU42.PMI:Performance_monitoring_interrupts
    100.00 ± 13%     +75.7%     175.67 ± 14%  interrupts.CPU42.RES:Rescheduling_interrupts
     74794 ±  4%     +56.8%     117244        interrupts.CPU43.LOC:Local_timer_interrupts
     98.67 ± 12%     +87.5%     185.00 ± 11%  interrupts.CPU43.RES:Rescheduling_interrupts
     74795 ±  4%     +56.7%     117236        interrupts.CPU44.LOC:Local_timer_interrupts
      3042 ±  4%     +35.8%       4130 ±  5%  interrupts.CPU44.NMI:Non-maskable_interrupts
      3042 ±  4%     +35.8%       4130 ±  5%  interrupts.CPU44.PMI:Performance_monitoring_interrupts
     99.00 ± 11%     +59.6%     158.00 ±  3%  interrupts.CPU44.RES:Rescheduling_interrupts
     74827 ±  4%     +56.7%     117248        interrupts.CPU45.LOC:Local_timer_interrupts
      3161 ±  3%     +31.8%       4167 ±  3%  interrupts.CPU45.NMI:Non-maskable_interrupts
      3161 ±  3%     +31.8%       4167 ±  3%  interrupts.CPU45.PMI:Performance_monitoring_interrupts
     90.00 ±  5%    +111.5%     190.33        interrupts.CPU45.RES:Rescheduling_interrupts
     74794 ±  4%     +56.8%     117242        interrupts.CPU46.LOC:Local_timer_interrupts
      3004 ±  6%     +40.6%       4226 ±  8%  interrupts.CPU46.NMI:Non-maskable_interrupts
      3004 ±  6%     +40.6%       4226 ±  8%  interrupts.CPU46.PMI:Performance_monitoring_interrupts
    112.33 ±  9%     +50.4%     169.00 ±  6%  interrupts.CPU46.RES:Rescheduling_interrupts
     74808 ±  4%     +56.7%     117230        interrupts.CPU47.LOC:Local_timer_interrupts
      3294 ± 10%     +28.1%       4219 ±  6%  interrupts.CPU47.NMI:Non-maskable_interrupts
      3294 ± 10%     +28.1%       4219 ±  6%  interrupts.CPU47.PMI:Performance_monitoring_interrupts
    107.67 ± 12%     +74.0%     187.33 ± 11%  interrupts.CPU47.RES:Rescheduling_interrupts
     74807 ±  4%     +56.7%     117236        interrupts.CPU48.LOC:Local_timer_interrupts
      3141 ± 12%     +30.8%       4108 ±  4%  interrupts.CPU48.NMI:Non-maskable_interrupts
      3141 ± 12%     +30.8%       4108 ±  4%  interrupts.CPU48.PMI:Performance_monitoring_interrupts
     88.67 ±  8%     +81.2%     160.67 ±  9%  interrupts.CPU48.RES:Rescheduling_interrupts
     74738 ±  4%     +56.8%     117164        interrupts.CPU49.LOC:Local_timer_interrupts
      2649 ± 33%     +55.7%       4126 ±  5%  interrupts.CPU49.NMI:Non-maskable_interrupts
      2649 ± 33%     +55.7%       4126 ±  5%  interrupts.CPU49.PMI:Performance_monitoring_interrupts
    106.00 ±  9%     +53.1%     162.33 ±  2%  interrupts.CPU49.RES:Rescheduling_interrupts
     74811 ±  4%     +56.7%     117229        interrupts.CPU5.LOC:Local_timer_interrupts
      3057 ±  6%     +33.7%       4087 ±  5%  interrupts.CPU5.NMI:Non-maskable_interrupts
      3057 ±  6%     +33.7%       4087 ±  5%  interrupts.CPU5.PMI:Performance_monitoring_interrupts
     74776 ±  4%     +56.8%     117223        interrupts.CPU50.LOC:Local_timer_interrupts
      2609 ± 31%     +54.8%       4040 ±  5%  interrupts.CPU50.NMI:Non-maskable_interrupts
      2609 ± 31%     +54.8%       4040 ±  5%  interrupts.CPU50.PMI:Performance_monitoring_interrupts
     95.00 ± 12%     +69.5%     161.00 ± 13%  interrupts.CPU50.RES:Rescheduling_interrupts
     74791 ±  4%     +56.7%     117230        interrupts.CPU51.LOC:Local_timer_interrupts
      2445 ± 29%     +64.7%       4028 ±  4%  interrupts.CPU51.NMI:Non-maskable_interrupts
      2445 ± 29%     +64.7%       4028 ±  4%  interrupts.CPU51.PMI:Performance_monitoring_interrupts
     95.33 ±  4%     +64.0%     156.33 ±  8%  interrupts.CPU51.RES:Rescheduling_interrupts
     74768 ±  4%     +56.8%     117243        interrupts.CPU52.LOC:Local_timer_interrupts
      3027 ±  7%     +36.1%       4121 ±  5%  interrupts.CPU52.NMI:Non-maskable_interrupts
      3027 ±  7%     +36.1%       4121 ±  5%  interrupts.CPU52.PMI:Performance_monitoring_interrupts
     91.67 ± 12%     +90.2%     174.33 ±  9%  interrupts.CPU52.RES:Rescheduling_interrupts
     74482 ±  4%     +57.1%     117009        interrupts.CPU53.LOC:Local_timer_interrupts
      2511 ± 32%     +63.2%       4100 ±  5%  interrupts.CPU53.NMI:Non-maskable_interrupts
      2511 ± 32%     +63.2%       4100 ±  5%  interrupts.CPU53.PMI:Performance_monitoring_interrupts
    107.33 ± 16%     +51.6%     162.67        interrupts.CPU53.RES:Rescheduling_interrupts
     73.33 ±104%     -88.6%       8.33 ± 29%  interrupts.CPU53.TLB:TLB_shootdowns
     74636 ±  5%     +57.1%     117229        interrupts.CPU54.LOC:Local_timer_interrupts
      2515 ± 31%     +62.8%       4096 ±  4%  interrupts.CPU54.NMI:Non-maskable_interrupts
      2515 ± 31%     +62.8%       4096 ±  4%  interrupts.CPU54.PMI:Performance_monitoring_interrupts
    103.67 ± 12%     +77.2%     183.67 ±  8%  interrupts.CPU54.RES:Rescheduling_interrupts
     74637 ±  5%     +57.3%     117413        interrupts.CPU55.LOC:Local_timer_interrupts
     95.67 ± 11%     +57.8%     151.00 ± 16%  interrupts.CPU55.RES:Rescheduling_interrupts
     74637 ±  5%     +57.1%     117241        interrupts.CPU56.LOC:Local_timer_interrupts
     93.33 ± 10%     +78.9%     167.00 ± 10%  interrupts.CPU56.RES:Rescheduling_interrupts
     74474 ±  5%     +57.2%     117074        interrupts.CPU57.LOC:Local_timer_interrupts
      2938 ±  6%     +39.2%       4091 ±  5%  interrupts.CPU57.NMI:Non-maskable_interrupts
      2938 ±  6%     +39.2%       4091 ±  5%  interrupts.CPU57.PMI:Performance_monitoring_interrupts
    100.67 ±  3%     +76.5%     177.67        interrupts.CPU57.RES:Rescheduling_interrupts
     74631 ±  5%     +57.1%     117209        interrupts.CPU58.LOC:Local_timer_interrupts
    115.00 ± 14%    +156.8%     295.33 ± 50%  interrupts.CPU58.RES:Rescheduling_interrupts
     74639 ±  5%     +57.1%     117222        interrupts.CPU59.LOC:Local_timer_interrupts
    104.00 ± 14%     +76.3%     183.33 ± 12%  interrupts.CPU59.RES:Rescheduling_interrupts
     74818 ±  4%     +56.7%     117230        interrupts.CPU6.LOC:Local_timer_interrupts
      3200           +30.0%       4161 ±  4%  interrupts.CPU6.NMI:Non-maskable_interrupts
      3200           +30.0%       4161 ±  4%  interrupts.CPU6.PMI:Performance_monitoring_interrupts
    169.00 ± 52%    +519.3%       1046 ±103%  interrupts.CPU6.RES:Rescheduling_interrupts
     74628 ±  5%     +57.1%     117229        interrupts.CPU60.LOC:Local_timer_interrupts
     78.67 ± 15%    +261.9%     284.67 ± 63%  interrupts.CPU60.RES:Rescheduling_interrupts
     74611 ±  5%     +57.1%     117226        interrupts.CPU61.LOC:Local_timer_interrupts
     89.00 ± 16%     +94.0%     172.67 ±  5%  interrupts.CPU61.RES:Rescheduling_interrupts
     74643 ±  5%     +57.0%     117211        interrupts.CPU62.LOC:Local_timer_interrupts
     87.67 ±  4%     +83.7%     161.00 ±  8%  interrupts.CPU62.RES:Rescheduling_interrupts
     74640 ±  5%     +57.1%     117245        interrupts.CPU63.LOC:Local_timer_interrupts
     80.33 ±  6%    +123.2%     179.33 ±  5%  interrupts.CPU63.RES:Rescheduling_interrupts
     74661 ±  5%     +57.0%     117227        interrupts.CPU64.LOC:Local_timer_interrupts
     90.33 ±  8%     +86.0%     168.00 ±  5%  interrupts.CPU64.RES:Rescheduling_interrupts
     74639 ±  5%     +57.3%     117395        interrupts.CPU65.LOC:Local_timer_interrupts
     99.33 ± 15%     +94.6%     193.33 ± 30%  interrupts.CPU65.RES:Rescheduling_interrupts
     74611 ±  5%     +57.1%     117206        interrupts.CPU66.LOC:Local_timer_interrupts
     90.33 ±  9%    +108.1%     188.00 ±  6%  interrupts.CPU66.RES:Rescheduling_interrupts
     74631 ±  5%     +57.2%     117290        interrupts.CPU67.LOC:Local_timer_interrupts
      1995 ± 34%    +105.0%       4090 ±  5%  interrupts.CPU67.NMI:Non-maskable_interrupts
      1995 ± 34%    +105.0%       4090 ±  5%  interrupts.CPU67.PMI:Performance_monitoring_interrupts
     96.33 ±  3%     +70.9%     164.67 ±  8%  interrupts.CPU67.RES:Rescheduling_interrupts
     74849 ±  4%     +56.8%     117345        interrupts.CPU68.LOC:Local_timer_interrupts
      2031 ± 32%    +101.7%       4098 ±  4%  interrupts.CPU68.NMI:Non-maskable_interrupts
      2031 ± 32%    +101.7%       4098 ±  4%  interrupts.CPU68.PMI:Performance_monitoring_interrupts
     97.33 ± 18%     +79.8%     175.00 ± 16%  interrupts.CPU68.RES:Rescheduling_interrupts
     74599 ±  5%     +57.2%     117251        interrupts.CPU69.LOC:Local_timer_interrupts
      2038 ± 33%     +99.0%       4056 ±  5%  interrupts.CPU69.NMI:Non-maskable_interrupts
      2038 ± 33%     +99.0%       4056 ±  5%  interrupts.CPU69.PMI:Performance_monitoring_interrupts
     87.00 ± 18%    +134.9%     204.33 ±  2%  interrupts.CPU69.RES:Rescheduling_interrupts
     74784 ±  4%     +56.8%     117225        interrupts.CPU7.LOC:Local_timer_interrupts
      3044 ±  5%     +35.8%       4133 ±  4%  interrupts.CPU7.NMI:Non-maskable_interrupts
      3044 ±  5%     +35.8%       4133 ±  4%  interrupts.CPU7.PMI:Performance_monitoring_interrupts
    103.67 ±  5%     +55.0%     160.67 ±  8%  interrupts.CPU7.RES:Rescheduling_interrupts
     74603 ±  5%     +57.1%     117233        interrupts.CPU70.LOC:Local_timer_interrupts
      2560 ± 36%     +58.1%       4048 ±  3%  interrupts.CPU70.NMI:Non-maskable_interrupts
      2560 ± 36%     +58.1%       4048 ±  3%  interrupts.CPU70.PMI:Performance_monitoring_interrupts
     90.67 ± 19%     +80.9%     164.00        interrupts.CPU70.RES:Rescheduling_interrupts
     74688 ±  5%     +57.0%     117281        interrupts.CPU71.LOC:Local_timer_interrupts
      2005 ± 34%    +101.0%       4030 ±  5%  interrupts.CPU71.NMI:Non-maskable_interrupts
      2005 ± 34%    +101.0%       4030 ±  5%  interrupts.CPU71.PMI:Performance_monitoring_interrupts
     93.33 ± 18%     +99.3%     186.00        interrupts.CPU71.RES:Rescheduling_interrupts
     74829 ±  4%     +56.6%     117177        interrupts.CPU8.LOC:Local_timer_interrupts
      3229 ±  9%     +25.6%       4057 ±  4%  interrupts.CPU8.NMI:Non-maskable_interrupts
      3229 ±  9%     +25.6%       4057 ±  4%  interrupts.CPU8.PMI:Performance_monitoring_interrupts
    136.33 ± 33%     +36.9%     186.67 ± 11%  interrupts.CPU8.RES:Rescheduling_interrupts
     74828 ±  4%     +56.7%     117227        interrupts.CPU9.LOC:Local_timer_interrupts
      3081 ±  5%     +33.7%       4120 ±  6%  interrupts.CPU9.NMI:Non-maskable_interrupts
      3081 ±  5%     +33.7%       4120 ±  6%  interrupts.CPU9.PMI:Performance_monitoring_interrupts
    106.00 ± 17%    +101.6%     213.67 ± 20%  interrupts.CPU9.RES:Rescheduling_interrupts
   5379025 ±  4%     +56.9%    8439938        interrupts.LOC:Local_timer_interrupts
    211613 ±  7%     +31.5%     278168 ±  5%  interrupts.NMI:Non-maskable_interrupts
    211613 ±  7%     +31.5%     278168 ±  5%  interrupts.PMI:Performance_monitoring_interrupts
     11773           +44.7%      17040 ±  2%  interrupts.RES:Rescheduling_interrupts
      3281 ± 62%     -86.7%     436.67 ± 34%  interrupts.TLB:TLB_shootdowns
     55.62 ±  5%     -29.6       26.03        perf-profile.calltrace.cycles-pp.write
     33.31 ± 10%     -26.1        7.20 ±  6%  perf-profile.calltrace.cycles-pp.secondary_startup_64
     32.80 ± 10%     -25.7        7.08 ±  6%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
     32.80 ± 10%     -25.7        7.08 ±  6%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
     32.79 ± 10%     -25.7        7.08 ±  6%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     32.29 ± 11%     -25.4        6.88 ±  6%  perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     32.24 ± 11%     -25.4        6.85 ±  6%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
     31.48 ± 11%     -25.0        6.44 ±  7%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     46.23 ±  5%     -24.0       22.25        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
     45.90 ±  5%     -23.8       22.12        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     38.94 ±  5%     -20.1       18.82        perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     37.76 ±  5%     -19.5       18.30        perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     34.21 ±  5%     -17.5       16.67        perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     32.68 ±  5%     -16.7       15.99        perf-profile.calltrace.cycles-pp.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write.do_syscall_64
     28.25 ±  6%     -14.8       13.48 ±  2%  perf-profile.calltrace.cycles-pp.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write
     28.04 ±  6%     -14.7       13.39 ±  2%  perf-profile.calltrace.cycles-pp.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write.vfs_write
     21.53 ±  6%     -11.0       10.51 ±  2%  perf-profile.calltrace.cycles-pp.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write
     11.06 ±  6%      -5.4        5.65 ±  2%  perf-profile.calltrace.cycles-pp.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
      6.47 ±  6%      -3.3        3.15 ±  3%  perf-profile.calltrace.cycles-pp.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      4.94 ±  5%      -3.2        1.75 ±  3%  perf-profile.calltrace.cycles-pp.close
      6.23 ±  6%      -3.2        3.05 ±  2%  perf-profile.calltrace.cycles-pp.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor.iomap_apply
      5.11 ±  5%      -3.2        1.93        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.write
      4.86 ±  5%      -3.1        1.72 ±  3%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.close
      4.86 ±  5%      -3.1        1.72 ±  3%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.close
      4.83 ±  5%      -3.1        1.71 ±  3%  perf-profile.calltrace.cycles-pp.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.close
      4.82 ±  5%      -3.1        1.70 ±  3%  perf-profile.calltrace.cycles-pp.__fput.task_work_run.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
      4.83 ±  5%      -3.1        1.71 ±  3%  perf-profile.calltrace.cycles-pp.task_work_run.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.close
      4.80 ±  5%      -3.1        1.70 ±  3%  perf-profile.calltrace.cycles-pp.dput.__fput.task_work_run.exit_to_usermode_loop.do_syscall_64
      4.78 ±  4%      -3.1        1.69 ±  3%  perf-profile.calltrace.cycles-pp.__dentry_kill.dput.__fput.task_work_run.exit_to_usermode_loop
      5.18 ±  6%      -2.8        2.37 ±  2%  perf-profile.calltrace.cycles-pp.iomap_write_end.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
      4.14 ±  5%      -2.3        1.88 ±  3%  perf-profile.calltrace.cycles-pp.xfs_buffered_write_iomap_begin.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write
      3.90 ±  5%      -2.2        1.68 ±  2%  perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.write
      3.61 ±  4%      -1.9        1.66 ±  3%  perf-profile.calltrace.cycles-pp.evict.__dentry_kill.dput.__fput.task_work_run
      3.58 ±  4%      -1.9        1.65 ±  3%  perf-profile.calltrace.cycles-pp.truncate_inode_pages_range.evict.__dentry_kill.dput.__fput
      3.22 ±  6%      -1.8        1.41 ±  2%  perf-profile.calltrace.cycles-pp.iomap_set_range_uptodate.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      2.63 ±  2%      -1.4        1.19 ±  4%  perf-profile.calltrace.cycles-pp.xfs_file_aio_write_checks.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write
      2.83 ±  6%      -1.3        1.53        perf-profile.calltrace.cycles-pp.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor
      2.23 ±  4%      -1.2        1.03 ±  4%  perf-profile.calltrace.cycles-pp.iomap_set_page_dirty.iomap_write_end.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      2.23 ±  5%      -1.1        1.10        perf-profile.calltrace.cycles-pp.iov_iter_copy_from_user_atomic.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
      2.11 ±  7%      -1.1        0.98        perf-profile.calltrace.cycles-pp.iomap_set_range_uptodate.iomap_write_end.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      1.79 ±  4%      -0.9        0.90        perf-profile.calltrace.cycles-pp.copyin.iov_iter_copy_from_user_atomic.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      1.62 ±  5%      -0.8        0.83        perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyin.iov_iter_copy_from_user_atomic.iomap_write_actor.iomap_apply
      1.37 ±  7%      -0.8        0.60 ±  4%  perf-profile.calltrace.cycles-pp.find_get_entry.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor
      1.35 ±  6%      -0.7        0.61 ±  2%  perf-profile.calltrace.cycles-pp.security_file_permission.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.33 ±  6%      -0.7        0.60 ±  2%  perf-profile.calltrace.cycles-pp.__lru_cache_add.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin
      1.22 ±  7%      -0.7        0.54 ±  3%  perf-profile.calltrace.cycles-pp.pagevec_lru_move_fn.__lru_cache_add.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin
      1.19 ±  4%      -0.6        0.56 ±  2%  perf-profile.calltrace.cycles-pp.__pagevec_release.truncate_inode_pages_range.evict.__dentry_kill.dput
      1.16 ±  8%      -0.6        0.54 ±  4%  perf-profile.calltrace.cycles-pp.__alloc_pages_nodemask.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor
      1.18 ±  4%      -0.6        0.57 ±  6%  perf-profile.calltrace.cycles-pp.__set_page_dirty.iomap_set_page_dirty.iomap_write_end.iomap_write_actor.iomap_apply
      1.15 ±  4%      -0.6        0.54 ±  2%  perf-profile.calltrace.cycles-pp.release_pages.__pagevec_release.truncate_inode_pages_range.evict.__dentry_kill
      1.43 ±  5%      -0.5        0.90        perf-profile.calltrace.cycles-pp.__add_to_page_cache_locked.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin
      0.80            -0.5        0.34 ± 70%  perf-profile.calltrace.cycles-pp.vfs_unlink.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink
      1.11 ±  2%      -0.2        0.92        perf-profile.calltrace.cycles-pp.xfs_generic_create.path_openat.do_filp_open.do_sys_openat2.do_sys_open
      1.10            -0.2        0.91        perf-profile.calltrace.cycles-pp.xfs_create.xfs_generic_create.path_openat.do_filp_open.do_sys_openat2
      0.85 ±  4%      -0.1        0.74        perf-profile.calltrace.cycles-pp.rwsem_spin_on_owner.rwsem_optimistic_spin.rwsem_down_write_slowpath.do_unlinkat.do_syscall_64
      0.75 ±  5%      -0.1        0.67        perf-profile.calltrace.cycles-pp.rwsem_spin_on_owner.rwsem_optimistic_spin.rwsem_down_write_slowpath.path_openat.do_filp_open
      0.17 ±141%      +0.4        0.55 ±  2%  perf-profile.calltrace.cycles-pp.xfs_dir_ialloc.xfs_create.xfs_generic_create.path_openat.do_filp_open
      0.17 ±141%      +0.4        0.55 ±  2%  perf-profile.calltrace.cycles-pp.xfs_ialloc.xfs_dir_ialloc.xfs_create.xfs_generic_create.path_openat
      0.00            +0.5        0.51        perf-profile.calltrace.cycles-pp.xfs_dialloc.xfs_ialloc.xfs_dir_ialloc.xfs_create.xfs_generic_create
      2.75 ±  3%     +29.1       31.89        perf-profile.calltrace.cycles-pp.unlink
      2.72 ±  4%     +29.2       31.87        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.unlink
      2.72 ±  4%     +29.2       31.87        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink
      2.70 ±  4%     +29.2       31.85        perf-profile.calltrace.cycles-pp.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink
      2.77 ±  5%     +29.5       32.23        perf-profile.calltrace.cycles-pp.creat
      2.74 ±  5%     +29.5       32.21        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.creat
      2.74 ±  5%     +29.5       32.21        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.creat
      2.72 ±  6%     +29.5       32.20        perf-profile.calltrace.cycles-pp.do_sys_openat2.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe.creat
      2.72 ±  6%     +29.5       32.20        perf-profile.calltrace.cycles-pp.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe.creat
      2.69 ±  5%     +29.5       32.19        perf-profile.calltrace.cycles-pp.do_filp_open.do_sys_openat2.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.69 ±  5%     +29.5       32.19        perf-profile.calltrace.cycles-pp.path_openat.do_filp_open.do_sys_openat2.do_sys_open.do_syscall_64
      1.75 ±  6%     +29.6       31.31        perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink
      1.45 ±  7%     +29.8       31.24        perf-profile.calltrace.cycles-pp.rwsem_optimistic_spin.rwsem_down_write_slowpath.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.58 ± 11%     +29.9       30.48        perf-profile.calltrace.cycles-pp.osq_lock.rwsem_optimistic_spin.rwsem_down_write_slowpath.do_unlinkat.do_syscall_64
      1.21 ± 11%     +29.9       31.12        perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.path_openat.do_filp_open.do_sys_openat2.do_sys_open
      0.99 ± 14%     +30.1       31.05        perf-profile.calltrace.cycles-pp.rwsem_optimistic_spin.rwsem_down_write_slowpath.path_openat.do_filp_open.do_sys_openat2
      0.00           +30.4       30.37        perf-profile.calltrace.cycles-pp.osq_lock.rwsem_optimistic_spin.rwsem_down_write_slowpath.path_openat.do_filp_open
     55.78 ±  5%     -29.5       26.33        perf-profile.children.cycles-pp.write
     33.31 ± 10%     -26.1        7.20 ±  6%  perf-profile.children.cycles-pp.do_idle
     33.31 ± 10%     -26.1        7.20 ±  6%  perf-profile.children.cycles-pp.secondary_startup_64
     33.31 ± 10%     -26.1        7.20 ±  6%  perf-profile.children.cycles-pp.cpu_startup_entry
     32.79 ± 11%     -25.8        7.00 ±  6%  perf-profile.children.cycles-pp.cpuidle_enter
     32.79 ± 11%     -25.8        7.00 ±  6%  perf-profile.children.cycles-pp.cpuidle_enter_state
     32.80 ± 10%     -25.7        7.08 ±  6%  perf-profile.children.cycles-pp.start_secondary
     31.79 ± 11%     -25.2        6.56 ±  7%  perf-profile.children.cycles-pp.intel_idle
     38.95 ±  5%     -20.1       18.84        perf-profile.children.cycles-pp.ksys_write
     37.79 ±  5%     -19.5       18.33        perf-profile.children.cycles-pp.vfs_write
     34.23 ±  5%     -17.5       16.69        perf-profile.children.cycles-pp.new_sync_write
     32.72 ±  5%     -16.7       16.01        perf-profile.children.cycles-pp.xfs_file_buffered_aio_write
     28.26 ±  6%     -14.8       13.49 ±  2%  perf-profile.children.cycles-pp.iomap_file_buffered_write
     28.07 ±  6%     -14.7       13.41 ±  2%  perf-profile.children.cycles-pp.iomap_apply
     21.64 ±  6%     -11.1       10.56 ±  2%  perf-profile.children.cycles-pp.iomap_write_actor
     11.09 ±  6%      -5.4        5.67 ±  2%  perf-profile.children.cycles-pp.iomap_write_begin
      6.51 ±  6%      -3.3        3.17 ±  3%  perf-profile.children.cycles-pp.grab_cache_page_write_begin
      6.27 ±  6%      -3.2        3.07 ±  2%  perf-profile.children.cycles-pp.pagecache_get_page
      4.94 ±  5%      -3.2        1.75 ±  3%  perf-profile.children.cycles-pp.close
      4.84 ±  5%      -3.1        1.71 ±  3%  perf-profile.children.cycles-pp.exit_to_usermode_loop
      4.82 ±  5%      -3.1        1.71 ±  3%  perf-profile.children.cycles-pp.__fput
      4.82 ±  4%      -3.1        1.70 ±  3%  perf-profile.children.cycles-pp.dput
      4.83 ±  5%      -3.1        1.71 ±  3%  perf-profile.children.cycles-pp.task_work_run
      4.78 ±  4%      -3.1        1.69 ±  3%  perf-profile.children.cycles-pp.__dentry_kill
      5.33 ±  6%      -2.9        2.39        perf-profile.children.cycles-pp.iomap_set_range_uptodate
      5.21 ±  6%      -2.8        2.39 ±  2%  perf-profile.children.cycles-pp.iomap_write_end
      4.61 ±  5%      -2.7        1.95        perf-profile.children.cycles-pp.entry_SYSCALL_64
      4.46 ±  5%      -2.5        1.92        perf-profile.children.cycles-pp.syscall_return_via_sysret
      4.20 ±  5%      -2.3        1.90 ±  3%  perf-profile.children.cycles-pp.xfs_buffered_write_iomap_begin
      3.61 ±  4%      -1.9        1.66 ±  3%  perf-profile.children.cycles-pp.evict
      3.58 ±  4%      -1.9        1.65 ±  3%  perf-profile.children.cycles-pp.truncate_inode_pages_range
      2.69 ±  2%      -1.5        1.21 ±  4%  perf-profile.children.cycles-pp.xfs_file_aio_write_checks
      2.84 ±  6%      -1.3        1.53        perf-profile.children.cycles-pp.add_to_page_cache_lru
      2.26 ±  4%      -1.2        1.04 ±  4%  perf-profile.children.cycles-pp.iomap_set_page_dirty
      2.25 ±  5%      -1.1        1.11        perf-profile.children.cycles-pp.iov_iter_copy_from_user_atomic
      1.78 ±  4%      -1.0        0.79        perf-profile.children.cycles-pp.xfs_ilock
      1.81 ±  4%      -0.9        0.91        perf-profile.children.cycles-pp.copyin
      1.72 ±  4%      -0.8        0.87        perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      1.52 ±  4%      -0.8        0.68        perf-profile.children.cycles-pp.down_write
      1.40 ±  7%      -0.8        0.61 ±  4%  perf-profile.children.cycles-pp.find_get_entry
      1.37 ±  6%      -0.7        0.62 ±  2%  perf-profile.children.cycles-pp.security_file_permission
      1.33 ±  6%      -0.7        0.60 ±  2%  perf-profile.children.cycles-pp.__lru_cache_add
      1.14 ±  3%      -0.7        0.43 ±  6%  perf-profile.children.cycles-pp.xfs_buffered_write_iomap_end
      1.26 ±  6%      -0.7        0.56 ±  3%  perf-profile.children.cycles-pp.pagevec_lru_move_fn
      1.27 ±  4%      -0.7        0.60 ±  2%  perf-profile.children.cycles-pp.release_pages
      1.13 ±  5%      -0.7        0.48        perf-profile.children.cycles-pp.iov_iter_fault_in_readable
      1.19 ±  4%      -0.6        0.56 ±  2%  perf-profile.children.cycles-pp.__pagevec_release
      1.17 ±  7%      -0.6        0.55 ±  3%  perf-profile.children.cycles-pp.__alloc_pages_nodemask
      1.18 ±  4%      -0.6        0.57 ±  7%  perf-profile.children.cycles-pp.__set_page_dirty
      1.05 ±  6%      -0.6        0.45 ±  5%  perf-profile.children.cycles-pp.common_file_perm
      1.08 ±  6%      -0.6        0.48 ±  4%  perf-profile.children.cycles-pp.xfs_file_write_iter
      1.28 ±  2%      -0.6        0.71 ±  2%  perf-profile.children.cycles-pp.__xfs_trans_commit
      1.46 ±  5%      -0.5        0.91        perf-profile.children.cycles-pp.__add_to_page_cache_locked
      1.24 ±  2%      -0.5        0.69 ±  2%  perf-profile.children.cycles-pp.xfs_log_commit_cil
      1.00 ±  5%      -0.5        0.46 ±  3%  perf-profile.children.cycles-pp.xfs_iunlock
      2.02 ±  3%      -0.5        1.48        perf-profile.children.cycles-pp.rwsem_spin_on_owner
      0.94 ±  5%      -0.5        0.42 ±  5%  perf-profile.children.cycles-pp.truncate_cleanup_page
      0.99 ±  5%      -0.5        0.49 ±  6%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.91 ±  7%      -0.5        0.41 ±  2%  perf-profile.children.cycles-pp.get_page_from_freelist
      0.94 ±  4%      -0.5        0.44 ±  3%  perf-profile.children.cycles-pp.delete_from_page_cache_batch
      0.98 ±  3%      -0.5        0.50        perf-profile.children.cycles-pp.___might_sleep
      0.94 ±  3%      -0.5        0.46 ±  8%  perf-profile.children.cycles-pp.file_update_time
      0.84 ±  5%      -0.5        0.37 ±  2%  perf-profile.children.cycles-pp.__pagevec_lru_add_fn
      0.84 ±  8%      -0.5        0.38 ±  9%  perf-profile.children.cycles-pp.__fdget_pos
      0.84 ±  7%      -0.5        0.38 ±  4%  perf-profile.children.cycles-pp.xas_load
      0.81 ±  5%      -0.5        0.36 ±  4%  perf-profile.children.cycles-pp.__cancel_dirty_page
      0.86 ±  3%      -0.4        0.42 ± 11%  perf-profile.children.cycles-pp.account_page_dirtied
      1.43 ±  6%      -0.4        1.00 ±  5%  perf-profile.children.cycles-pp.apic_timer_interrupt
      0.70 ±  4%      -0.4        0.28 ±  2%  perf-profile.children.cycles-pp.xfs_break_layouts
      1.34 ±  6%      -0.4        0.93 ±  6%  perf-profile.children.cycles-pp.smp_apic_timer_interrupt
      0.72 ±  5%      -0.4        0.33 ±  3%  perf-profile.children.cycles-pp.up_write
      0.67 ±  3%      -0.4        0.28 ±  5%  perf-profile.children.cycles-pp.account_page_cleaned
      0.51 ±  8%      -0.4        0.12 ± 13%  perf-profile.children.cycles-pp.start_kernel
      0.69 ±  4%      -0.4        0.32 ±  6%  perf-profile.children.cycles-pp.__mod_lruvec_state
      0.67 ±  8%      -0.4        0.30 ±  9%  perf-profile.children.cycles-pp.__fget_light
      0.75 ±  4%      -0.4        0.38 ±  4%  perf-profile.children.cycles-pp.__sb_start_write
      1.13 ±  7%      -0.4        0.76 ±  5%  perf-profile.children.cycles-pp.xfs_inactive
      0.68            -0.4        0.32 ±  6%  perf-profile.children.cycles-pp._raw_spin_lock
      0.68 ±  6%      -0.4        0.33 ± 12%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      0.78 ±  4%      -0.3        0.44 ±  4%  perf-profile.children.cycles-pp.xas_store
      0.80            -0.3        0.50        perf-profile.children.cycles-pp.vfs_unlink
      0.79            -0.3        0.50        perf-profile.children.cycles-pp.xfs_vn_unlink
      0.79            -0.3        0.50        perf-profile.children.cycles-pp.xfs_remove
      0.56 ±  4%      -0.3        0.27 ± 10%  perf-profile.children.cycles-pp.percpu_counter_add_batch
      0.48 ±  4%      -0.3        0.20 ±  4%  perf-profile.children.cycles-pp._cond_resched
      0.52 ±  5%      -0.3        0.24        perf-profile.children.cycles-pp.__might_sleep
      0.58 ±  4%      -0.3        0.30 ± 10%  perf-profile.children.cycles-pp.irq_exit
      0.47 ±  8%      -0.3        0.21 ±  4%  perf-profile.children.cycles-pp.fsnotify
      0.48            -0.3        0.21 ±  7%  perf-profile.children.cycles-pp.xfs_trans_read_buf_map
      0.34 ±  4%      -0.3        0.08        perf-profile.children.cycles-pp.ttwu_do_activate
      0.34 ±  4%      -0.3        0.08        perf-profile.children.cycles-pp.activate_task
      0.52 ± 10%      -0.3        0.27 ± 30%  perf-profile.children.cycles-pp.balance_dirty_pages_ratelimited
      0.34 ±  3%      -0.3        0.08        perf-profile.children.cycles-pp.enqueue_task_fair
      0.98 ±  6%      -0.3        0.72 ±  5%  perf-profile.children.cycles-pp.xfs_inactive_ifree
      0.32 ±  3%      -0.2        0.08 ±  6%  perf-profile.children.cycles-pp.enqueue_entity
      0.42 ±  5%      -0.2        0.18 ±  2%  perf-profile.children.cycles-pp.xfs_break_leased_layouts
      0.52 ±  5%      -0.2        0.28 ± 13%  perf-profile.children.cycles-pp.__softirqentry_text_start
      0.31 ±  5%      -0.2        0.09 ±  5%  perf-profile.children.cycles-pp.try_to_wake_up
      0.39 ±  3%      -0.2        0.17 ±  5%  perf-profile.children.cycles-pp.__mod_memcg_state
      0.38 ±  4%      -0.2        0.16 ± 10%  perf-profile.children.cycles-pp.xfs_bmbt_to_iomap
      0.38 ±  7%      -0.2        0.16 ±  7%  perf-profile.children.cycles-pp.unlock_page
      0.38 ±  6%      -0.2        0.16 ±  2%  perf-profile.children.cycles-pp.__x64_sys_write
      0.35 ±  2%      -0.2        0.14 ± 11%  perf-profile.children.cycles-pp.xfs_buf_read_map
      0.39 ±  2%      -0.2        0.19 ± 18%  perf-profile.children.cycles-pp.xfs_vn_update_time
      0.34 ±  2%      -0.2        0.14 ± 11%  perf-profile.children.cycles-pp.xfs_buf_get_map
      0.37 ±  4%      -0.2        0.17 ±  2%  perf-profile.children.cycles-pp.mem_cgroup_commit_charge
      0.26 ±  3%      -0.2        0.07 ±  7%  perf-profile.children.cycles-pp.__account_scheduler_latency
      1.12            -0.2        0.92        perf-profile.children.cycles-pp.xfs_generic_create
      0.39 ±  5%      -0.2        0.21 ±  2%  perf-profile.children.cycles-pp.xfs_iext_lookup_extent
      1.10            -0.2        0.91        perf-profile.children.cycles-pp.xfs_create
      0.32 ±  6%      -0.2        0.14        perf-profile.children.cycles-pp.page_mapping
      0.32 ±  4%      -0.2        0.14        perf-profile.children.cycles-pp.__list_del_entry_valid
      0.43 ±  5%      -0.2        0.25        perf-profile.children.cycles-pp.xfs_errortag_test
      0.35            -0.2        0.17 ±  4%  perf-profile.children.cycles-pp.generic_write_checks
      0.30 ±  2%      -0.2        0.12 ± 13%  perf-profile.children.cycles-pp.xfs_buf_find
      0.32 ±  2%      -0.2        0.14 ±  3%  perf-profile.children.cycles-pp.__mod_node_page_state
      0.22 ±  3%      -0.2        0.05 ±  8%  perf-profile.children.cycles-pp.stack_trace_save_tsk
      0.38 ±  4%      -0.2        0.22 ±  5%  perf-profile.children.cycles-pp.xfs_da3_node_lookup_int
      0.41            -0.2        0.24 ±  5%  perf-profile.children.cycles-pp.xfs_dir_removename
      0.40 ±  2%      -0.2        0.24 ±  3%  perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
      0.33 ±  6%      -0.2        0.16 ±  2%  perf-profile.children.cycles-pp.current_time
      0.26 ±  3%      -0.2        0.10 ±  4%  perf-profile.children.cycles-pp.iov_iter_advance
      0.37 ±  2%      -0.2        0.21 ± 18%  perf-profile.children.cycles-pp.rcu_core
      0.40 ±  2%      -0.2        0.24 ±  5%  perf-profile.children.cycles-pp.xfs_dir2_node_removename
      0.26 ±  3%      -0.2        0.10 ±  4%  perf-profile.children.cycles-pp.__sched_text_start
      0.23 ±  2%      -0.2        0.07 ± 12%  perf-profile.children.cycles-pp.xfs_read_agi
      0.21 ±  2%      -0.2        0.05        perf-profile.children.cycles-pp.arch_stack_walk
      0.29 ±  2%      -0.2        0.14 ±  3%  perf-profile.children.cycles-pp.node_dirty_ok
      0.28 ±  7%      -0.2        0.13 ±  3%  perf-profile.children.cycles-pp.find_get_entries
      0.26 ±  3%      -0.2        0.11 ±  4%  perf-profile.children.cycles-pp.__fsnotify_parent
      0.28 ±  7%      -0.2        0.13        perf-profile.children.cycles-pp.pagevec_lookup_entries
      0.23 ±  3%      -0.1        0.08 ± 14%  perf-profile.children.cycles-pp.xfs_buf_item_release
      0.26 ±  8%      -0.1        0.12 ±  8%  perf-profile.children.cycles-pp.xas_start
      0.28 ±  6%      -0.1        0.15 ±  3%  perf-profile.children.cycles-pp.mem_cgroup_try_charge
      0.25 ±  3%      -0.1        0.11 ±  4%  perf-profile.children.cycles-pp.mem_cgroup_charge_statistics
      0.23 ±  7%      -0.1        0.09 ±  5%  perf-profile.children.cycles-pp.rcu_all_qs
      0.21 ±  3%      -0.1        0.07 ± 17%  perf-profile.children.cycles-pp.xfs_buf_unlock
      0.20 ±  4%      -0.1        0.06 ±  7%  perf-profile.children.cycles-pp.up
      0.25 ±  5%      -0.1        0.12 ±  4%  perf-profile.children.cycles-pp.free_unref_page_list
      0.22 ±  7%      -0.1        0.10 ± 17%  perf-profile.children.cycles-pp.__sb_end_write
      0.20 ±  4%      -0.1        0.09 ± 14%  perf-profile.children.cycles-pp.xfs_iunlink_remove
      0.64 ±  5%      -0.1        0.53 ±  7%  perf-profile.children.cycles-pp.xfs_ifree
      0.20 ±  4%      -0.1        0.09 ± 13%  perf-profile.children.cycles-pp.lock_page_memcg
      0.69 ±  8%      -0.1        0.58 ±  4%  perf-profile.children.cycles-pp.hrtimer_interrupt
      0.21 ±  5%      -0.1        0.11        perf-profile.children.cycles-pp.xfs_dir_createname
      0.23 ± 14%      -0.1        0.13 ± 16%  perf-profile.children.cycles-pp.ktime_get
      0.13 ± 15%      -0.1        0.03 ± 70%  perf-profile.children.cycles-pp.xfs_get_extsz_hint
      0.20 ±  6%      -0.1        0.10 ±  4%  perf-profile.children.cycles-pp.xfs_dir2_node_addname
      0.17 ±  7%      -0.1        0.07 ±  7%  perf-profile.children.cycles-pp.mem_cgroup_uncharge_list
      0.18 ±  9%      -0.1        0.08 ± 24%  perf-profile.children.cycles-pp.wait_for_stable_page
      0.18 ±  2%      -0.1        0.09 ± 10%  perf-profile.children.cycles-pp.xfs_trans_alloc
      0.18 ±  5%      -0.1        0.08        perf-profile.children.cycles-pp.get_mem_cgroup_from_mm
      0.20 ±  8%      -0.1        0.11 ±  7%  perf-profile.children.cycles-pp.workingset_update_node
      0.19 ±  6%      -0.1        0.09 ±  5%  perf-profile.children.cycles-pp.xfs_btree_lookup
      0.18 ±  5%      -0.1        0.08 ±  5%  perf-profile.children.cycles-pp.__xa_set_mark
      0.20 ±  4%      -0.1        0.10 ±  9%  perf-profile.children.cycles-pp.xfs_da_read_buf
      0.20 ±  2%      -0.1        0.11 ±  7%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.22 ±  2%      -0.1        0.14 ±  3%  perf-profile.children.cycles-pp.__xfs_dir3_data_check
      0.21 ±  2%      -0.1        0.12        perf-profile.children.cycles-pp.xfs_dir2_leafn_lookup_for_entry
      0.12 ±  6%      -0.1        0.03 ± 70%  perf-profile.children.cycles-pp.__list_add_valid
      0.16 ±  2%      -0.1        0.08 ±  6%  perf-profile.children.cycles-pp.unaccount_page_cache_page
      0.16 ±  2%      -0.1        0.08 ±  6%  perf-profile.children.cycles-pp.schedule
      0.22 ±  2%      -0.1        0.14 ±  5%  perf-profile.children.cycles-pp.xfs_dir3_data_check
      0.17 ±  4%      -0.1        0.09 ±  5%  perf-profile.children.cycles-pp.xas_init_marks
      0.14 ±  3%      -0.1        0.06        perf-profile.children.cycles-pp.file_modified
      0.20 ± 13%      -0.1        0.11 ±  4%  perf-profile.children.cycles-pp.apparmor_file_permission
      0.18 ±  2%      -0.1        0.10 ±  8%  perf-profile.children.cycles-pp.generic_write_check_limits
      0.15 ±  6%      -0.1        0.07 ±  7%  perf-profile.children.cycles-pp.aa_file_perm
      0.13 ±  9%      -0.1        0.05 ±  8%  perf-profile.children.cycles-pp.__mark_inode_dirty
      0.14 ±  6%      -0.1        0.06 ± 13%  perf-profile.children.cycles-pp.dec_zone_page_state
      0.14 ±  3%      -0.1        0.07 ±  7%  perf-profile.children.cycles-pp.xfs_vn_lookup
      0.20 ± 19%      -0.1        0.12 ± 11%  perf-profile.children.cycles-pp.clockevents_program_event
      0.14 ±  3%      -0.1        0.07 ±  7%  perf-profile.children.cycles-pp.xfs_lookup
      0.13 ± 12%      -0.1        0.06        perf-profile.children.cycles-pp.times
      0.13 ±  3%      -0.1        0.06 ±  7%  perf-profile.children.cycles-pp.xfs_dir_lookup
      0.13 ±  3%      -0.1        0.06 ±  7%  perf-profile.children.cycles-pp.alloc_pages_current
      0.13            -0.1        0.06 ±  7%  perf-profile.children.cycles-pp.xfs_trans_reserve
      0.13 ±  6%      -0.1        0.07 ±  7%  perf-profile.children.cycles-pp.xas_clear_mark
      0.12            -0.1        0.06 ±  8%  perf-profile.children.cycles-pp.xfs_log_reserve
      0.12 ±  4%      -0.1        0.05 ±  8%  perf-profile.children.cycles-pp.xfs_btree_lookup_get_block
      0.12 ± 16%      -0.1        0.06 ±  7%  perf-profile.children.cycles-pp.timestamp_truncate
      0.17 ± 14%      -0.1        0.11 ±  8%  perf-profile.children.cycles-pp.xfs_bmapi_reserve_delalloc
      0.12 ±  3%      -0.1        0.07 ±  7%  perf-profile.children.cycles-pp.xfs_da3_node_read
      0.11 ±  4%      -0.1        0.06 ±  8%  perf-profile.children.cycles-pp.xfs_dir2_node_lookup
      0.11 ±  4%      -0.1        0.05 ±  8%  perf-profile.children.cycles-pp.free_unref_page_commit
      0.11 ±  4%      -0.1        0.06 ±  8%  perf-profile.children.cycles-pp._xfs_trans_bjoin
      0.11 ±  7%      -0.0        0.06 ±  7%  perf-profile.children.cycles-pp.xfs_dir3_leaf_check_int
      0.10 ±  4%      -0.0        0.05 ±  8%  perf-profile.children.cycles-pp.__inc_node_state
      0.12 ±  3%      -0.0        0.08 ± 10%  perf-profile.children.cycles-pp.xfs_dir_ino_validate
      0.11 ±  7%      -0.0        0.07 ±  7%  perf-profile.children.cycles-pp.xfs_inode_item_format
      0.10 ±  8%      -0.0        0.06 ± 13%  perf-profile.children.cycles-pp.update_load_avg
      0.19 ±  4%      -0.0        0.16 ±  5%  perf-profile.children.cycles-pp.xfs_buf_item_format
      0.09 ± 15%      -0.0        0.06 ± 13%  perf-profile.children.cycles-pp.__percpu_counter_sum
      0.09 ±  5%      -0.0        0.06 ±  8%  perf-profile.children.cycles-pp.xfs_verify_ino
      0.06 ±  7%      -0.0        0.03 ± 70%  perf-profile.children.cycles-pp.pick_next_task_fair
      0.11 ±  4%      -0.0        0.08 ±  5%  perf-profile.children.cycles-pp.xfs_agino_range
      0.10 ± 14%      -0.0        0.07 ± 11%  perf-profile.children.cycles-pp.__percpu_counter_compare
      0.08 ±  5%      -0.0        0.06 ±  8%  perf-profile.children.cycles-pp.ktime_get_coarse_real_ts64
      0.13 ± 13%      -0.0        0.10 ±  8%  perf-profile.children.cycles-pp.menu_select
      0.09 ± 10%      +0.0        0.13 ±  9%  perf-profile.children.cycles-pp.xfs_inobt_get_maxrecs
      0.09 ± 14%      +0.1        0.14 ± 18%  perf-profile.children.cycles-pp.task_tick_fair
      0.49 ±  4%      +0.1        0.55 ±  2%  perf-profile.children.cycles-pp.xfs_dir_ialloc
      0.49 ±  4%      +0.1        0.55 ±  2%  perf-profile.children.cycles-pp.xfs_ialloc
      0.15 ± 11%      +0.1        0.21 ±  2%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.44 ±  5%      +0.1        0.51        perf-profile.children.cycles-pp.xfs_dialloc
      0.19 ±  6%      +0.1        0.29 ±  9%  perf-profile.children.cycles-pp.__xfs_btree_check_sblock
      0.20 ±  4%      +0.1        0.29 ±  5%  perf-profile.children.cycles-pp.xfs_btree_get_rec
      0.37 ±  5%      +0.1        0.49        perf-profile.children.cycles-pp.xfs_dialloc_ag
      0.33 ±  2%      +0.1        0.46 ±  4%  perf-profile.children.cycles-pp.xfs_inobt_get_rec
      0.17 ±  5%      +0.1        0.32 ±  9%  perf-profile.children.cycles-pp.xfs_btree_increment
      0.26 ±  6%      +0.2        0.44 ±  8%  perf-profile.children.cycles-pp.xfs_btree_check_sblock
      0.63 ±  3%      +0.2        0.83 ±  5%  perf-profile.children.cycles-pp.xfs_check_agi_freecount
      0.00            +0.3        0.28 ± 11%  perf-profile.children.cycles-pp.xfs_inactive_worker
      0.05            +0.3        0.37 ±  8%  perf-profile.children.cycles-pp.process_one_work
      0.05            +0.3        0.37 ±  7%  perf-profile.children.cycles-pp.worker_thread
      0.07 ±  7%      +0.3        0.40 ±  6%  perf-profile.children.cycles-pp.ret_from_fork
      0.07 ±  7%      +0.3        0.40 ±  6%  perf-profile.children.cycles-pp.kthread
      0.00            +0.5        0.51 ± 13%  perf-profile.children.cycles-pp.xfs_inode_free_blocks
      0.00            +0.8        0.77 ±  5%  perf-profile.children.cycles-pp.xfs_inactive_inode
      0.00            +0.8        0.79 ±  6%  perf-profile.children.cycles-pp.xfs_ici_walk_fns
      0.00            +0.8        0.79 ±  6%  perf-profile.children.cycles-pp.xfs_ici_walk_ag
      2.76 ±  3%     +29.1       31.89        perf-profile.children.cycles-pp.unlink
      2.70 ±  4%     +29.2       31.85        perf-profile.children.cycles-pp.do_unlinkat
      2.78 ±  5%     +29.5       32.23        perf-profile.children.cycles-pp.creat
      2.73 ±  5%     +29.5       32.20        perf-profile.children.cycles-pp.do_sys_openat2
      2.73 ±  5%     +29.5       32.21        perf-profile.children.cycles-pp.do_sys_open
      2.69 ±  5%     +29.5       32.19        perf-profile.children.cycles-pp.do_filp_open
      2.69 ±  5%     +29.5       32.19        perf-profile.children.cycles-pp.path_openat
     56.69 ±  5%     +31.5       88.15        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     56.38 ±  5%     +31.6       88.02        perf-profile.children.cycles-pp.do_syscall_64
      2.96 ±  8%     +59.5       62.42        perf-profile.children.cycles-pp.rwsem_down_write_slowpath
      2.44 ± 10%     +59.9       62.29        perf-profile.children.cycles-pp.rwsem_optimistic_spin
      0.74 ± 22%     +60.1       60.88        perf-profile.children.cycles-pp.osq_lock
     31.78 ± 11%     -25.2        6.55 ±  7%  perf-profile.self.cycles-pp.intel_idle
      6.53 ±  6%      -3.4        3.11        perf-profile.self.cycles-pp.do_syscall_64
      5.29 ±  7%      -2.9        2.37        perf-profile.self.cycles-pp.iomap_set_range_uptodate
      4.45 ±  5%      -2.5        1.91        perf-profile.self.cycles-pp.syscall_return_via_sysret
      4.06 ±  5%      -2.3        1.71        perf-profile.self.cycles-pp.entry_SYSCALL_64
      1.70 ±  4%      -0.8        0.86        perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      1.45 ±  4%      -0.8        0.62 ±  3%  perf-profile.self.cycles-pp.xfs_buffered_write_iomap_begin
      1.10 ±  4%      -0.6        0.46 ±  2%  perf-profile.self.cycles-pp.iov_iter_fault_in_readable
      1.05 ±  6%      -0.6        0.46 ±  5%  perf-profile.self.cycles-pp.xfs_file_write_iter
      1.09 ±  6%      -0.6        0.51        perf-profile.self.cycles-pp.iomap_apply
      0.90 ±  5%      -0.6        0.33 ±  8%  perf-profile.self.cycles-pp.xfs_buffered_write_iomap_end
      2.00 ±  3%      -0.5        1.47        perf-profile.self.cycles-pp.rwsem_spin_on_owner
      0.89 ±  7%      -0.5        0.38 ±  6%  perf-profile.self.cycles-pp.common_file_perm
      0.86 ±  5%      -0.5        0.41        perf-profile.self.cycles-pp.iomap_write_actor
      0.78 ±  4%      -0.5        0.33 ±  3%  perf-profile.self.cycles-pp.write
      0.94 ±  3%      -0.5        0.49 ±  2%  perf-profile.self.cycles-pp.___might_sleep
      0.75 ±  7%      -0.4        0.33 ±  3%  perf-profile.self.cycles-pp.find_get_entry
      0.69 ±  4%      -0.4        0.30        perf-profile.self.cycles-pp.down_write
      0.68 ±  6%      -0.4        0.31 ±  2%  perf-profile.self.cycles-pp.up_write
      0.66 ±  8%      -0.4        0.29 ± 10%  perf-profile.self.cycles-pp.__fget_light
      0.68 ±  6%      -0.4        0.33 ± 12%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      0.59 ±  6%      -0.3        0.26        perf-profile.self.cycles-pp.xas_load
      0.56 ±  7%      -0.3        0.24 ±  3%  perf-profile.self.cycles-pp.iomap_write_end
      0.60 ±  5%      -0.3        0.28 ±  7%  perf-profile.self.cycles-pp.xfs_file_buffered_aio_write
      0.54 ±  7%      -0.3        0.22 ±  4%  perf-profile.self.cycles-pp.__pagevec_lru_add_fn
      0.51 ±  6%      -0.3        0.21 ±  5%  perf-profile.self.cycles-pp.pagecache_get_page
      0.52            -0.3        0.25        perf-profile.self.cycles-pp._raw_spin_lock
      0.47 ±  8%      -0.3        0.21 ±  4%  perf-profile.self.cycles-pp.fsnotify
      0.47 ±  6%      -0.2        0.22        perf-profile.self.cycles-pp.__might_sleep
      0.43 ±  9%      -0.2        0.20 ±  2%  perf-profile.self.cycles-pp.iov_iter_copy_from_user_atomic
      0.45 ±  4%      -0.2        0.22 ±  2%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.47 ± 12%      -0.2        0.24 ± 36%  perf-profile.self.cycles-pp.balance_dirty_pages_ratelimited
      0.45 ±  3%      -0.2        0.22 ± 13%  perf-profile.self.cycles-pp.percpu_counter_add_batch
      0.40 ±  5%      -0.2        0.17 ±  4%  perf-profile.self.cycles-pp.xfs_break_leased_layouts
      0.41 ±  9%      -0.2        0.19 ±  2%  perf-profile.self.cycles-pp.vfs_write
      0.38 ±  4%      -0.2        0.16 ±  7%  perf-profile.self.cycles-pp.xfs_bmbt_to_iomap
      0.51 ±  6%      -0.2        0.30 ±  7%  perf-profile.self.cycles-pp.iomap_write_begin
      0.41 ±  5%      -0.2        0.20 ±  4%  perf-profile.self.cycles-pp.__sb_start_write
      0.36 ±  7%      -0.2        0.16 ±  6%  perf-profile.self.cycles-pp.unlock_page
      0.38 ±  2%      -0.2        0.17 ±  5%  perf-profile.self.cycles-pp.__mod_memcg_state
      0.35 ±  6%      -0.2        0.15 ±  5%  perf-profile.self.cycles-pp.__x64_sys_write
      0.33 ±  6%      -0.2        0.14 ±  3%  perf-profile.self.cycles-pp.ksys_write
      0.35 ± 11%      -0.2        0.16 ±  2%  perf-profile.self.cycles-pp.get_page_from_freelist
      0.33 ±  8%      -0.2        0.15 ±  3%  perf-profile.self.cycles-pp.iomap_set_page_dirty
      0.32 ±  6%      -0.2        0.14        perf-profile.self.cycles-pp.page_mapping
      0.31 ±  7%      -0.2        0.13 ±  3%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.31 ±  4%      -0.2        0.13 ±  3%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.32 ±  9%      -0.2        0.14 ±  3%  perf-profile.self.cycles-pp.new_sync_write
      0.34 ±  3%      -0.2        0.18 ±  2%  perf-profile.self.cycles-pp.release_pages
      0.30 ±  4%      -0.2        0.14 ±  6%  perf-profile.self.cycles-pp.__mod_node_page_state
      0.37 ±  5%      -0.2        0.20 ±  2%  perf-profile.self.cycles-pp.xfs_iext_lookup_extent
      0.28 ±  5%      -0.2        0.12 ±  4%  perf-profile.self.cycles-pp.xfs_ilock
      0.38 ±  3%      -0.2        0.22 ±  2%  perf-profile.self.cycles-pp.xfs_errortag_test
      0.29 ±  7%      -0.2        0.14 ±  3%  perf-profile.self.cycles-pp.xfs_iunlock
      0.26 ±  4%      -0.2        0.11 ±  4%  perf-profile.self.cycles-pp.xfs_file_aio_write_checks
      0.24 ±  3%      -0.1        0.09        perf-profile.self.cycles-pp.iov_iter_advance
      0.35 ±  2%      -0.1        0.21 ±  3%  perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
      0.24 ±  5%      -0.1        0.10        perf-profile.self.cycles-pp._cond_resched
      0.24            -0.1        0.10        perf-profile.self.cycles-pp.__fsnotify_parent
      0.24            -0.1        0.10 ±  8%  perf-profile.self.cycles-pp.account_page_cleaned
      0.32 ±  6%      -0.1        0.18 ±  4%  perf-profile.self.cycles-pp.xas_store
      0.24 ±  8%      -0.1        0.10 ±  4%  perf-profile.self.cycles-pp.find_get_entries
      0.26 ±  8%      -0.1        0.12 ±  3%  perf-profile.self.cycles-pp.__add_to_page_cache_locked
      0.22 ±  5%      -0.1        0.08        perf-profile.self.cycles-pp.xfs_break_layouts
      0.24 ±  7%      -0.1        0.11 ±  8%  perf-profile.self.cycles-pp.xas_start
      0.22 ±  6%      -0.1        0.10 ± 17%  perf-profile.self.cycles-pp.__sb_end_write
      0.22 ±  5%      -0.1        0.11 ±  4%  perf-profile.self.cycles-pp.file_update_time
      0.22 ±  4%      -0.1        0.11 ± 16%  perf-profile.self.cycles-pp.account_page_dirtied
      0.20 ±  4%      -0.1        0.09 ±  5%  perf-profile.self.cycles-pp.node_dirty_ok
      0.20 ±  4%      -0.1        0.09 ± 13%  perf-profile.self.cycles-pp.lock_page_memcg
      0.20 ±  6%      -0.1        0.10 ±  4%  perf-profile.self.cycles-pp.__mod_lruvec_state
      0.17 ±  7%      -0.1        0.07 ± 14%  perf-profile.self.cycles-pp.rcu_all_qs
      0.18 ±  5%      -0.1        0.08        perf-profile.self.cycles-pp.get_mem_cgroup_from_mm
      0.17 ±  4%      -0.1        0.07 ±  6%  perf-profile.self.cycles-pp.iomap_file_buffered_write
      0.20 ± 17%      -0.1        0.10 ± 22%  perf-profile.self.cycles-pp.ktime_get
      0.13 ± 16%      -0.1        0.03 ± 70%  perf-profile.self.cycles-pp.xfs_get_extsz_hint
      0.18 ±  5%      -0.1        0.08 ±  5%  perf-profile.self.cycles-pp.__fdget_pos
      0.16 ± 10%      -0.1        0.07 ± 20%  perf-profile.self.cycles-pp.wait_for_stable_page
      0.12 ± 11%      -0.1        0.03 ± 70%  perf-profile.self.cycles-pp.__mark_inode_dirty
      0.16 ±  2%      -0.1        0.08 ±  6%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.18 ±  2%      -0.1        0.10 ±  8%  perf-profile.self.cycles-pp.generic_write_check_limits
      0.18 ± 12%      -0.1        0.10        perf-profile.self.cycles-pp.apparmor_file_permission
      0.14 ±  3%      -0.1        0.06 ±  7%  perf-profile.self.cycles-pp.aa_file_perm
      0.15            -0.1        0.07        perf-profile.self.cycles-pp.generic_write_checks
      0.18 ±  7%      -0.1        0.10 ±  8%  perf-profile.self.cycles-pp.workingset_update_node
      0.15 ±  3%      -0.1        0.07        perf-profile.self.cycles-pp.truncate_inode_pages_range
      0.11 ±  7%      -0.1        0.03 ± 70%  perf-profile.self.cycles-pp.__list_add_valid
      0.13 ±  6%      -0.1        0.06 ±  8%  perf-profile.self.cycles-pp.current_time
      0.15 ±  5%      -0.1        0.08 ±  5%  perf-profile.self.cycles-pp.xfs_log_commit_cil
      0.13 ± 13%      -0.1        0.06        perf-profile.self.cycles-pp.__cancel_dirty_page
      0.16 ±  8%      -0.1        0.09 ±  5%  perf-profile.self.cycles-pp.__alloc_pages_nodemask
      0.13 ±  6%      -0.1        0.06 ±  7%  perf-profile.self.cycles-pp.security_file_permission
      0.12 ±  8%      -0.1        0.05 ±  8%  perf-profile.self.cycles-pp.dec_zone_page_state
      0.12 ± 11%      -0.1        0.06        perf-profile.self.cycles-pp.delete_from_page_cache_batch
      0.12 ±  6%      -0.1        0.06 ± 13%  perf-profile.self.cycles-pp.xas_clear_mark
      0.09 ± 15%      -0.1        0.03 ± 70%  perf-profile.self.cycles-pp.menu_select
      0.11 ± 19%      -0.1        0.06 ±  8%  perf-profile.self.cycles-pp.timestamp_truncate
      0.10 ±  4%      -0.1        0.05        perf-profile.self.cycles-pp.__lru_cache_add
      0.11 ±  7%      -0.0        0.06 ±  7%  perf-profile.self.cycles-pp.xfs_dir3_leaf_check_int
      0.10 ±  4%      -0.0        0.05 ±  8%  perf-profile.self.cycles-pp.__inc_node_state
      0.07 ±  6%      -0.0        0.03 ± 70%  perf-profile.self.cycles-pp.ktime_get_coarse_real_ts64
      0.08 ±  5%      -0.0        0.05 ±  8%  perf-profile.self.cycles-pp.__xfs_dir3_data_check
      0.15 ±  8%      -0.0        0.12 ±  6%  perf-profile.self.cycles-pp.memcpy_erms
      0.09            -0.0        0.06 ±  7%  perf-profile.self.cycles-pp.xfs_agino_range
      0.08 ± 12%      +0.0        0.12 ± 10%  perf-profile.self.cycles-pp.xfs_inobt_get_maxrecs
      0.10 ±  8%      +0.0        0.15 ±  8%  perf-profile.self.cycles-pp.__xfs_btree_check_sblock
      0.00            +0.1        0.06 ±  7%  perf-profile.self.cycles-pp.xfs_btree_increment
      0.14 ± 12%      +0.1        0.21 ±  2%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.00            +0.1        0.08 ± 12%  perf-profile.self.cycles-pp.xfs_btree_check_sblock
      0.74 ± 21%     +59.8       60.51        perf-profile.self.cycles-pp.osq_lock





Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.6.0-00207-gef6d7398f5c72" of type "text/plain" (204855 bytes)

View attachment "job-script" of type "text/plain" (7697 bytes)

View attachment "job.yaml" of type "text/plain" (5324 bytes)

View attachment "reproduce" of type "text/plain" (478 bytes)

Powered by blists - more mailing lists