lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Message-ID: <20210511083119.GH8539@xsang-OptiPlex-9020>
Date:   Tue, 11 May 2021 16:31:20 +0800
From:   kernel test robot <oliver.sang@...el.com>
To:     Johannes Weiner <hannes@...xchg.org>
Cc:     Linus Torvalds <torvalds@...ux-foundation.org>,
        Roman Gushchin <guro@...com>, Michal Hocko <mhocko@...e.com>,
        Shakeel Butt <shakeelb@...gle.com>,
        Michal Koutný <mkoutny@...e.com>,
        Balbir Singh <bsingharora@...il.com>,
        Tejun Heo <tj@...nel.org>,
        Andrew Morton <akpm@...ux-foundation.org>,
        LKML <linux-kernel@...r.kernel.org>, lkp@...ts.01.org,
        lkp@...el.com, ying.huang@...el.com, feng.tang@...el.com,
        zhengjun.xing@...el.com
Subject: [mm]  2d146aa3aa:  vm-scalability.throughput -2.8% regression



Greeting,

FYI, we noticed a -2.8% regression of vm-scalability.throughput due to commit:


commit: 2d146aa3aa842d7f5065802556b4f9a2c6e8ef12 ("mm: memcontrol: switch to rstat")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master


in testcase: vm-scalability
on test machine: 192 threads 4 sockets Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory
with following parameters:

	runtime: 300s
	test: lru-file-mmap-read-rand
	cpufreq_governor: performance
	ucode: 0x5003006

test-description: The motivation behind this suite is to exercise functions and regions of the mm/ of the Linux kernel which are of interest to us.
test-url: https://git.kernel.org/cgit/linux/kernel/git/wfg/vm-scalability.git/

In addition to that, the commit also has significant impact on the following tests:

+------------------+-------------------------------------------------------------------------------------+
| testcase: change | reaim: reaim.jobs_per_min -2.7% regression                                          |
| test machine     | 192 threads 4 sockets Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory |
| test parameters  | cpufreq_governor=performance                                                        |
|                  | nr_task=100%                                                                        |
|                  | runtime=300s                                                                        |
|                  | test=aim9                                                                           |
|                  | ucode=0x5003006                                                                     |
+------------------+-------------------------------------------------------------------------------------+


If you fix the issue, kindly add following tag
Reported-by: kernel test robot <oliver.sang@...el.com>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install                job.yaml  # job file is attached in this email
        bin/lkp split-job --compatible job.yaml  # generate the yaml file for lkp run
        bin/lkp run                    generated-yaml-file

=========================================================================================
compiler/cpufreq_governor/kconfig/rootfs/runtime/tbox_group/test/testcase/ucode:
  gcc-9/performance/x86_64-rhel-8.3/debian-10.4-x86_64-20200603.cgz/300s/lkp-csl-2ap4/lru-file-mmap-read-rand/vm-scalability/0x5003006

commit: 
  dc26532aed ("cgroup: rstat: punt root-level optimization to individual controllers")
  2d146aa3aa ("mm: memcontrol: switch to rstat")

dc26532aed0ab25c 2d146aa3aa842d7f5065802556b 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
      8798            -3.0%       8533        vm-scalability.median
   1696442            -2.8%    1648941        vm-scalability.throughput
    378.62            +2.8%     389.16        vm-scalability.time.elapsed_time
    378.62            +2.8%     389.16        vm-scalability.time.elapsed_time.max
  48931042            -1.2%   48355398        vm-scalability.time.major_page_faults
   3116506 ±  3%     -19.3%    2514494 ± 10%  vm-scalability.time.maximum_resident_set_size
    854811            -2.2%     835712        vm-scalability.time.minor_page_faults
     58258            +3.4%      60219        vm-scalability.time.system_time
     36.55 ±  5%     -25.4%      27.27 ±  5%  vm-scalability.time.user_time
    929.17 ± 53%    +874.8%       9057 ± 99%  softirqs.CPU12.NET_RX
      6700            -1.9%       6573        vmstat.system.cs
    362.61 ±  5%     -10.3%     325.09 ±  5%  perf-sched.wait_and_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
    359.35 ±  3%      -9.6%     324.98 ±  5%  perf-sched.wait_time.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      1670 ± 48%    +956.2%      17638 ±103%  interrupts.40:PCI-MSI.524291-edge.eth0-TxRx-2
    185.83 ± 25%     -53.5%      86.33 ± 28%  interrupts.CPU116.RES:Rescheduling_interrupts
      1670 ± 48%    +956.2%      17638 ±103%  interrupts.CPU12.40:PCI-MSI.524291-edge.eth0-TxRx-2
     97.00 ± 13%     -27.5%      70.33 ± 20%  interrupts.CPU176.RES:Rescheduling_interrupts
    107.17 ± 21%     -33.9%      70.83 ± 18%  interrupts.CPU189.RES:Rescheduling_interrupts
    361.67 ±  6%     +11.1%     401.67 ±  3%  interrupts.CPU2.RES:Rescheduling_interrupts
    284.67 ±  5%     +43.1%     407.50 ± 23%  interrupts.CPU41.RES:Rescheduling_interrupts
    268.33 ±  9%     +15.7%     310.33 ± 12%  interrupts.CPU46.RES:Rescheduling_interrupts
    226.00 ±  7%     +18.4%     267.67 ±  6%  interrupts.CPU52.RES:Rescheduling_interrupts
    138.83 ± 10%     -17.5%     114.50 ± 14%  interrupts.CPU71.RES:Rescheduling_interrupts
    562317 ±  3%      -8.4%     515046 ±  6%  interrupts.CPU72.TLB:TLB_shootdowns
      3617 ± 11%     -29.9%       2535 ± 15%  proc-vmstat.allocstall_normal
      1016 ±  4%     -13.4%     879.83 ±  8%  proc-vmstat.kswapd_low_wmark_hit_quickly
      1131            -6.2%       1060 ±  2%  proc-vmstat.nr_isolated_file
   1684103            -1.3%    1662379        proc-vmstat.nr_page_table_pages
   1189118            -1.6%    1169561        proc-vmstat.nr_slab_reclaimable
  1.54e+09            -1.2%  1.522e+09        proc-vmstat.pgalloc_normal
 1.004e+08            -1.1%   99249683        proc-vmstat.pgfault
 1.557e+09            -1.1%  1.539e+09        proc-vmstat.pgfree
  48883247            -1.2%   48308956        proc-vmstat.pgmajfault
     99332            +2.6%     101937        proc-vmstat.pgreuse
 2.229e+09            -1.2%  2.201e+09        proc-vmstat.pgscan_file
 1.496e+09            -1.2%  1.478e+09        proc-vmstat.pgsteal_file
  17500257            -4.3%   16741696        proc-vmstat.workingset_nodereclaim
   4747501            -2.4%    4631656        proc-vmstat.workingset_nodes
 1.486e+08            -3.6%  1.433e+08        proc-vmstat.workingset_refault_file
     80.71 ±  5%     -25.4       55.30 ±  6%  perf-profile.calltrace.cycles-pp.do_access
     80.69 ±  5%     -25.4       55.28 ±  6%  perf-profile.calltrace.cycles-pp.asm_exc_page_fault.do_access
     80.66 ±  5%     -25.4       55.26 ±  6%  perf-profile.calltrace.cycles-pp.exc_page_fault.asm_exc_page_fault.do_access
     80.66 ±  5%     -25.4       55.26 ±  6%  perf-profile.calltrace.cycles-pp.do_user_addr_fault.exc_page_fault.asm_exc_page_fault.do_access
     80.65 ±  5%     -25.4       55.25 ±  6%  perf-profile.calltrace.cycles-pp.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault.do_access
      0.67 ± 13%      -0.4        0.26 ±100%  perf-profile.calltrace.cycles-pp.free_pcppages_bulk.free_unref_page_list.shrink_page_list.shrink_inactive_list.shrink_lruvec
     17.05 ± 28%     +25.2       42.25 ±  8%  perf-profile.calltrace.cycles-pp.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
     17.06 ± 28%     +25.2       42.26 ±  8%  perf-profile.calltrace.cycles-pp.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
     17.06 ± 28%     +25.2       42.26 ±  8%  perf-profile.calltrace.cycles-pp.exc_page_fault.asm_exc_page_fault
     17.06 ± 28%     +25.2       42.26 ±  8%  perf-profile.calltrace.cycles-pp.asm_exc_page_fault
     80.74 ±  5%     -25.4       55.31 ±  6%  perf-profile.children.cycles-pp.do_access
      0.14 ±  6%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.__count_memcg_events
      0.27 ± 16%      -0.1        0.18 ±  6%  perf-profile.children.cycles-pp.try_charge
      0.09 ±  4%      +0.0        0.11 ±  8%  perf-profile.children.cycles-pp.workingset_refault
      0.29 ±  3%      +0.1        0.34 ± 10%  perf-profile.children.cycles-pp.xfs_filemap_map_pages
      0.28 ±  3%      +0.1        0.34 ± 11%  perf-profile.children.cycles-pp.filemap_map_pages
      0.11 ±  6%      +0.1        0.18 ± 22%  perf-profile.children.cycles-pp.page_add_file_rmap
      0.12 ±  6%      +0.1        0.19 ± 19%  perf-profile.children.cycles-pp.do_set_pte
      0.54 ±  8%      +0.2        0.76 ± 18%  perf-profile.children.cycles-pp.__mod_memcg_lruvec_state
      0.14 ±  6%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.__count_memcg_events
      0.40 ± 38%      -0.1        0.28        perf-stat.i.branch-miss-rate%
  29502377 ±  5%      -8.9%   26877024        perf-stat.i.branch-misses
 1.102e+08            -8.6%  1.008e+08        perf-stat.i.cache-misses
 2.578e+08 ±  4%      -7.1%  2.395e+08 ±  2%  perf-stat.i.cache-references
      6587            -2.0%       6459        perf-stat.i.context-switches
      3923            +9.0%       4277        perf-stat.i.cycles-between-cache-misses
      0.12 ± 11%      -0.0        0.10 ± 11%  perf-stat.i.dTLB-load-miss-rate%
 1.494e+10            +1.3%  1.515e+10        perf-stat.i.dTLB-loads
      0.02 ± 14%      -0.0        0.02 ±  8%  perf-stat.i.dTLB-store-miss-rate%
   3782013 ±  2%      -8.2%    3470625 ±  2%  perf-stat.i.iTLB-load-misses
     14984            +7.6%      16129 ±  2%  perf-stat.i.instructions-per-iTLB-miss
    125639            -3.7%     120964        perf-stat.i.major-faults
    385.69 ±  9%     -12.7%     336.62 ±  2%  perf-stat.i.metric.K/sec
  24655193            -3.6%   23766292        perf-stat.i.node-load-misses
     62.43            -4.3       58.12        perf-stat.i.node-store-miss-rate%
  16903554           -18.7%   13745638        perf-stat.i.node-store-misses
    132189            -3.6%     127372        perf-stat.i.page-faults
      4.30 ±  3%      -6.7%       4.01 ±  2%  perf-stat.overall.MPKI
      0.22 ±  4%      -0.0        0.20        perf-stat.overall.branch-miss-rate%
      4354           +10.2%       4797        perf-stat.overall.cycles-between-cache-misses
     84.81            -2.2       82.63        perf-stat.overall.iTLB-load-miss-rate%
     15931            +8.8%      17329 ±  2%  perf-stat.overall.instructions-per-iTLB-miss
     60.20            -4.1       56.14        perf-stat.overall.node-store-miss-rate%
     43107            +2.6%      44240        perf-stat.overall.path-length
  29668531 ±  4%      -8.2%   27224284        perf-stat.ps.branch-misses
 1.121e+08            -8.7%  1.024e+08        perf-stat.ps.cache-misses
 2.623e+08 ±  3%      -6.7%  2.446e+08 ±  2%  perf-stat.ps.cache-references
      6672            -2.0%       6538        perf-stat.ps.context-switches
 1.536e+10            +1.2%  1.555e+10        perf-stat.ps.dTLB-loads
   3828017            -8.0%    3520592 ±  2%  perf-stat.ps.iTLB-load-misses
    128812            -3.7%     124041        perf-stat.ps.major-faults
  25194762            -3.7%   24262604        perf-stat.ps.node-load-misses
  17325664           -18.8%   14075191        perf-stat.ps.node-store-misses
    135306            -3.6%     130410        perf-stat.ps.page-faults
 2.316e+13            +2.6%  2.377e+13        perf-stat.total.instructions


                                                                                
                               vm-scalability.median                            
                                                                                
  9000 +--------------------------------------------------------------------+   
       | +.   +    +     +      +.    +.  +   +     +  +    +    + .+      +|   
  8800 |++ +++ +. + +.+ + +.++  : + ++  ++ :.+ ++ .+ ++ +   :+.+ :+  +++ .+ |   
       |         +     +      ++   +       +     +       +++    +    O  +   |   
  8600 |-+     O    O     O   O   O      O                      O O         |   
       |O    OO  OOO  O OO          O O        O          OO O   O    O     |   
  8400 |-+                  OO OO       O    OO  O OOOOO O  O  O    O       |   
       |               O                  OO                                |   
  8200 |-+                                      O                           |   
       |                                                                    |   
  8000 |-+                                                                  |   
       | O O                       O O                                      |   
  7800 |-+  O                                                               |   
       |                                                                    |   
  7600 +--------------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample

***************************************************************************************************
lkp-csl-2ap3: 192 threads 4 sockets Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/nr_task/rootfs/runtime/tbox_group/test/testcase/ucode:
  gcc-9/performance/x86_64-rhel-8.3/100%/debian-10.4-x86_64-20200603.cgz/300s/lkp-csl-2ap3/aim9/reaim/0x5003006

commit: 
  dc26532aed ("cgroup: rstat: punt root-level optimization to individual controllers")
  2d146aa3aa ("mm: memcontrol: switch to rstat")

dc26532aed0ab25c 2d146aa3aa842d7f5065802556b 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
     84.10 ±  2%     +54.3%     129.73 ±  2%  reaim.child_systime
     98.64            -2.7%      96.02        reaim.child_utime
    123355            -2.7%     120037        reaim.jobs_per_min
    642.47            -2.7%     625.20        reaim.jobs_per_min_child
    124335            -2.5%     121192        reaim.max_jobs_per_min
     11.21            +2.8%      11.52        reaim.parent_time
      1.01 ±  4%     +87.3%       1.90 ±  2%  reaim.std_dev_percent
      0.11 ±  4%     +91.0%       0.21 ±  2%  reaim.std_dev_time
    304.73            +2.3%     311.86        reaim.time.elapsed_time
    304.73            +2.3%     311.86        reaim.time.elapsed_time.max
      1380           +20.7%       1666        reaim.time.percent_of_cpu_this_job_got
      1938 ±  2%     +54.2%       2988 ±  2%  reaim.time.system_time
      2270            -2.6%       2210        reaim.time.user_time
   3236827            -1.1%    3200863        reaim.time.voluntary_context_switches
      0.33 ±  4%      +0.1        0.39 ±  4%  mpstat.cpu.all.soft%
      3.55            +1.6        5.11        mpstat.cpu.all.sys%
    122697 ±  2%      -9.0%     111603 ±  3%  meminfo.Active
    122697 ±  2%      -9.0%     111603 ±  3%  meminfo.Active(anon)
    141299 ±  2%      -8.5%     129326 ±  3%  meminfo.Shmem
    139.33 ±  3%     +17.6%     163.83 ±  3%  turbostat.Avg_MHz
     10.90 ±  9%      +1.9       12.76 ±  8%  turbostat.Busy%
     91.46            +4.1%      95.19 ±  2%  turbostat.PkgWatt
     90.00            -1.7%      88.50        vmstat.cpu.id
     13.17 ±  2%     +25.3%      16.50 ±  4%  vmstat.procs.r
     30506            -1.8%      29962        vmstat.system.cs
     30622 ±  2%      -9.0%      27871 ±  3%  proc-vmstat.nr_active_anon
    279241            -1.1%     276304        proc-vmstat.nr_file_pages
     11132 ±  2%      -5.5%      10519        proc-vmstat.nr_mapped
     35312 ±  2%      -8.3%      32376 ±  2%  proc-vmstat.nr_shmem
     40363            -5.7%      38052        proc-vmstat.nr_slab_reclaimable
    110643            -3.7%     106541        proc-vmstat.nr_slab_unreclaimable
     30622 ±  2%      -9.0%      27871 ±  3%  proc-vmstat.nr_zone_active_anon
     27867 ±  4%     +13.4%      31609 ±  8%  softirqs.CPU102.RCU
     27291 ±  2%     +13.1%      30868 ±  6%  softirqs.CPU115.RCU
     27063 ±  4%     +10.6%      29942 ±  5%  softirqs.CPU168.RCU
     27214 ±  4%     +12.6%      30637 ±  8%  softirqs.CPU19.RCU
     26223 ±  5%     +12.4%      29479 ±  7%  softirqs.CPU20.RCU
     26918 ±  3%     +11.2%      29939 ±  5%  softirqs.CPU21.RCU
     27358 ±  3%     +12.8%      30859 ±  6%  softirqs.CPU72.RCU
     72679 ±  5%     -14.8%      61952        slabinfo.cred_jar.active_objs
      1732 ±  5%     -14.8%       1475        slabinfo.cred_jar.active_slabs
     72758 ±  5%     -14.8%      61977        slabinfo.cred_jar.num_objs
      1732 ±  5%     -14.8%       1475        slabinfo.cred_jar.num_slabs
    163390 ±  2%      -9.4%     148066        slabinfo.kmalloc-32.active_objs
      1276 ±  2%      -9.3%       1157        slabinfo.kmalloc-32.active_slabs
    163419 ±  2%      -9.3%     148174        slabinfo.kmalloc-32.num_objs
      1276 ±  2%      -9.3%       1157        slabinfo.kmalloc-32.num_slabs
     17365 ±  4%     -12.4%      15219        slabinfo.kmalloc-rcl-512.active_objs
     17851 ±  5%     -13.8%      15380        slabinfo.kmalloc-rcl-512.num_objs
     46066 ±  8%     -32.6%      31064        slabinfo.pid.active_objs
    719.33 ±  8%     -32.6%     485.17        slabinfo.pid.active_slabs
     46066 ±  8%     -32.6%      31064        slabinfo.pid.num_objs
    719.33 ±  8%     -32.6%     485.17        slabinfo.pid.num_slabs
     23819 ±  2%      -8.2%      21874 ±  3%  slabinfo.proc_inode_cache.num_objs
     50922 ±  4%     -18.8%      41342        slabinfo.radix_tree_node.active_objs
    908.83 ±  4%     -18.8%     737.83        slabinfo.radix_tree_node.active_slabs
     50932 ±  4%     -18.8%      41342        slabinfo.radix_tree_node.num_objs
    908.83 ±  4%     -18.8%     737.83        slabinfo.radix_tree_node.num_slabs
     26831 ±  2%     -13.0%      23354        slabinfo.shmem_inode_cache.active_objs
     26831 ±  2%     -13.0%      23354        slabinfo.shmem_inode_cache.num_objs
      9939 ±  6%     -14.2%       8531        slabinfo.sock_inode_cache.active_objs
      9939 ±  6%     -14.2%       8531        slabinfo.sock_inode_cache.num_objs
     58065 ±  6%     -21.1%      45815        slabinfo.task_delay_info.active_objs
      1138 ±  6%     -21.1%     898.00        slabinfo.task_delay_info.active_slabs
     58067 ±  6%     -21.1%      45815        slabinfo.task_delay_info.num_objs
      1138 ±  6%     -21.1%     898.00        slabinfo.task_delay_info.num_slabs
     31451 ±  2%     +12.7%      35438 ±  3%  slabinfo.vmap_area.active_objs
 4.963e+09            -4.1%  4.761e+09        perf-stat.i.branch-instructions
  41054171 ±  2%      +5.6%   43339523 ±  2%  perf-stat.i.cache-misses
     30475            -2.6%      29684        perf-stat.i.context-switches
 5.083e+10 ±  3%     +18.7%  6.036e+10 ±  3%  perf-stat.i.cpu-cycles
 8.482e+09            -3.5%  8.187e+09        perf-stat.i.dTLB-loads
 5.232e+09            -3.1%  5.072e+09        perf-stat.i.dTLB-stores
   8995832            +7.5%    9666624        perf-stat.i.iTLB-load-misses
 3.179e+10            -3.9%  3.056e+10        perf-stat.i.instructions
      0.26 ±  3%     +19.0%       0.32 ±  3%  perf-stat.i.metric.GHz
     98.17            -3.3%      94.94        perf-stat.i.metric.M/sec
    291539            -3.1%     282571        perf-stat.i.minor-faults
   4098219 ±  2%     -16.3%    3428560        perf-stat.i.node-store-misses
    291652            -3.1%     282684        perf-stat.i.page-faults
      6.08 ± 21%     +24.5%       7.56 ± 21%  perf-stat.overall.MPKI
      1.60 ±  3%     +23.8%       1.98 ±  3%  perf-stat.overall.cpi
      1236 ±  2%     +12.4%       1389        perf-stat.overall.cycles-between-cache-misses
     66.75 ±  4%      +6.5       73.24 ±  8%  perf-stat.overall.iTLB-load-miss-rate%
      3537           -10.5%       3164        perf-stat.overall.instructions-per-iTLB-miss
      0.63 ±  3%     -19.3%       0.51 ±  3%  perf-stat.overall.ipc
     68.80            -3.4       65.35        perf-stat.overall.node-store-miss-rate%
 4.991e+09            -3.3%  4.827e+09        perf-stat.ps.branch-instructions
  41348616 ±  2%      +6.8%   44155400 ±  2%  perf-stat.ps.cache-misses
     30650            -1.8%      30099        perf-stat.ps.context-switches
 5.113e+10 ±  3%     +20.0%  6.135e+10 ±  3%  perf-stat.ps.cpu-cycles
      2894            +2.2%       2958        perf-stat.ps.cpu-migrations
 8.531e+09            -2.7%    8.3e+09        perf-stat.ps.dTLB-loads
 5.261e+09            -2.3%  5.141e+09        perf-stat.ps.dTLB-stores
   9039686            +8.3%    9792543        perf-stat.ps.iTLB-load-misses
 3.198e+10            -3.1%  3.098e+10        perf-stat.ps.instructions
    293396            -2.2%     286854        perf-stat.ps.minor-faults
   4129671           -15.4%    3492596        perf-stat.ps.node-store-misses
    293509            -2.2%     286969        perf-stat.ps.page-faults
      0.15 ± 39%     -93.5%       0.01 ± 69%  perf-sched.sch_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      0.01 ±  6%     -33.3%       0.01        perf-sched.sch_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      0.02 ± 11%     -69.6%       0.00 ± 10%  perf-sched.sch_delay.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      0.02 ± 69%     -80.0%       0.00 ± 20%  perf-sched.sch_delay.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
      0.32 ± 62%     -98.8%       0.00 ± 27%  perf-sched.sch_delay.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown]
      0.01 ± 17%     -61.7%       0.01 ± 20%  perf-sched.sch_delay.avg.ms.io_schedule.__lock_page_killable.filemap_fault.__do_fault
      0.00 ± 20%     -54.5%       0.00 ± 74%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.down_write.__vma_adjust.shift_arg_pages
      0.06 ±201%     -99.2%       0.00 ±223%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.dput.do_unlinkat.do_syscall_64
      0.05 ± 94%     -88.8%       0.01 ± 34%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.generic_perform_write.__generic_file_write_iter.generic_file_write_iter
      0.02 ± 44%     -81.6%       0.00 ± 27%  perf-sched.sch_delay.avg.ms.rwsem_down_read_slowpath.path_openat.do_filp_open.do_sys_openat2
      0.30 ± 13%     -99.0%       0.00        perf-sched.sch_delay.avg.ms.rwsem_down_read_slowpath.walk_component.path_lookupat.isra.0
      0.10 ± 56%     -95.8%       0.00 ± 46%  perf-sched.sch_delay.avg.ms.rwsem_down_write_slowpath.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.13 ± 84%     -97.6%       0.00 ± 66%  perf-sched.sch_delay.avg.ms.rwsem_down_write_slowpath.filename_create.do_linkat.__x64_sys_link
      0.19 ± 69%     -98.1%       0.00 ± 21%  perf-sched.sch_delay.avg.ms.rwsem_down_write_slowpath.filename_create.do_mknodat.do_syscall_64
      0.09 ± 50%     -96.1%       0.00 ± 64%  perf-sched.sch_delay.avg.ms.rwsem_down_write_slowpath.path_openat.do_filp_open.do_sys_openat2
      0.03 ± 37%     -48.4%       0.01 ± 30%  perf-sched.sch_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
      7.03 ± 24%     -99.6%       0.03 ± 40%  perf-sched.sch_delay.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      5.94 ± 20%     -46.3%       3.19 ± 32%  perf-sched.sch_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      9.44 ± 25%     -70.2%       2.81 ± 22%  perf-sched.sch_delay.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      0.03 ± 64%    +678.2%       0.23 ± 46%  perf-sched.sch_delay.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault.[unknown]
      5.41 ± 53%     -94.2%       0.31 ±109%  perf-sched.sch_delay.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
      4.67 ± 65%     -99.7%       0.01 ±104%  perf-sched.sch_delay.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown]
      2.73 ± 59%     -94.8%       0.14 ±208%  perf-sched.sch_delay.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
      1.39 ±124%     -91.3%       0.12 ±113%  perf-sched.sch_delay.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.27 ±215%     -98.1%       0.01 ± 36%  perf-sched.sch_delay.max.ms.io_schedule.__lock_page.pagecache_get_page.shmem_getpage_gfp
      1.25 ± 35%     -72.5%       0.34 ± 70%  perf-sched.sch_delay.max.ms.io_schedule.__lock_page_killable.filemap_fault.__do_fault
      3.42 ± 22%     -45.5%       1.86 ± 44%  perf-sched.sch_delay.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      0.12 ±208%     -95.8%       0.01 ± 37%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.__alloc_pages_nodemask.__pmd_alloc.copy_page_range
      3.79 ± 36%     -59.5%       1.53 ± 20%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
      0.01 ± 81%     -67.2%       0.00 ± 11%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.down_read.walk_component.link_path_walk
      0.57 ±185%     -97.3%       0.02 ±107%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.down_write.dup_mmap.dup_mm
      0.07 ±184%     -99.3%       0.00 ±223%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.dput.do_unlinkat.do_syscall_64
      4.87 ± 57%     -99.9%       0.01 ± 39%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.dput.scan_positives.dcache_readdir
      2.55 ± 98%     -93.8%       0.16 ±165%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.generic_perform_write.__generic_file_write_iter.generic_file_write_iter
      0.24 ±200%     -96.6%       0.01 ± 16%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.pagecache_get_page.shmem_getpage_gfp.shmem_write_begin
      0.04 ±105%     -85.9%       0.01 ± 73%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.task_work_run.exit_to_user_mode_prepare.syscall_exit_to_user_mode
      0.20 ±120%     -94.9%       0.01 ±163%  perf-sched.sch_delay.max.ms.rwsem_down_read_slowpath.down_read_killable.iterate_dir.__x64_sys_getdents64
      0.58 ± 62%     -97.7%       0.01 ± 61%  perf-sched.sch_delay.max.ms.rwsem_down_read_slowpath.path_openat.do_filp_open.do_sys_openat2
      7.71 ± 16%     -99.8%       0.01 ± 25%  perf-sched.sch_delay.max.ms.rwsem_down_read_slowpath.walk_component.path_lookupat.isra.0
      6.08 ± 30%     -99.4%       0.04 ±170%  perf-sched.sch_delay.max.ms.rwsem_down_write_slowpath.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.84 ±102%     -96.2%       0.03 ±205%  perf-sched.sch_delay.max.ms.rwsem_down_write_slowpath.dup_mmap.dup_mm.copy_process
      4.56 ± 87%     -99.8%       0.01 ± 49%  perf-sched.sch_delay.max.ms.rwsem_down_write_slowpath.filename_create.do_linkat.__x64_sys_link
      6.50 ± 49%     -99.8%       0.01 ±  5%  perf-sched.sch_delay.max.ms.rwsem_down_write_slowpath.filename_create.do_mknodat.do_syscall_64
      8.19 ± 20%     -99.2%       0.07 ±139%  perf-sched.sch_delay.max.ms.rwsem_down_write_slowpath.path_openat.do_filp_open.do_sys_openat2
      0.75 ± 61%     -94.5%       0.04 ±192%  perf-sched.sch_delay.max.ms.rwsem_down_write_slowpath.unlink_file_vma.free_pgtables.exit_mmap
      0.24 ±152%     -99.4%       0.00 ±152%  perf-sched.sch_delay.max.ms.rwsem_down_write_slowpath.vma_link.mmap_region.do_mmap
      0.01 ± 79%    +359.3%       0.04 ±105%  perf-sched.sch_delay.max.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
      0.01 ±  3%     -52.6%       0.01 ±  9%  perf-sched.total_sch_delay.average.ms
     12.56 ± 26%     -35.1%       8.15 ± 35%  perf-sched.total_sch_delay.max.ms
     30.21          +109.9%      63.41 ±  3%  perf-sched.total_wait_and_delay.average.ms
    157346           -10.3%     141167 ±  2%  perf-sched.total_wait_and_delay.count.ms
     30.20          +109.9%      63.41 ±  3%  perf-sched.total_wait_time.average.ms
      1.83 ±  3%   +4426.0%      82.86 ±  4%  perf-sched.wait_and_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      3.48 ± 10%    +387.5%      16.97 ±  4%  perf-sched.wait_and_delay.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      8.49 ± 75%   +2944.8%     258.38 ± 11%  perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
    132.38 ± 26%    +371.3%     623.98 ± 45%  perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown]
     94.13 ± 18%    +122.5%     209.45 ± 29%  perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
      0.04 ± 14%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.sched_exec.bprm_execve
    431.12 ±  5%     +37.1%     590.85 ±  3%  perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
    208.00           -90.7%      19.33 ±155%  perf-sched.wait_and_delay.count.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
     41990           -10.8%      37469 ±  2%  perf-sched.wait_and_delay.count.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
     42011           -12.7%      36693 ±  2%  perf-sched.wait_and_delay.count.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
     29.67 ± 19%     -45.5%      16.17 ± 22%  perf-sched.wait_and_delay.count.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown]
    889.50 ±  7%    -100.0%       0.00        perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.stop_one_cpu.sched_exec.bprm_execve
    155.33 ± 33%     -65.6%      53.50 ± 21%  perf-sched.wait_and_delay.count.rwsem_down_read_slowpath.walk_component.path_lookupat.isra.0
    982.50 ±  6%     +44.7%       1422 ± 34%  perf-sched.wait_and_delay.count.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
      5738 ±  5%     -15.3%       4862 ±  4%  perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork
      1514 ± 41%    +230.5%       5006        perf-sched.wait_and_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      1032          +384.9%       5007        perf-sched.wait_and_delay.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
    685.22 ± 70%    +630.6%       5006        perf-sched.wait_and_delay.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
      1023          +389.0%       5004        perf-sched.wait_and_delay.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown]
      1014          +393.3%       5004        perf-sched.wait_and_delay.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
      3.70 ± 59%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.sched_exec.bprm_execve
      1027          +387.0%       5004        perf-sched.wait_and_delay.max.ms.rwsem_down_read_slowpath.walk_component.path_lookupat.isra.0
    362.48 ± 10%     -37.3%     227.31 ± 30%  perf-sched.wait_and_delay.max.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
      3081 ±  9%    +100.1%       6166 ± 14%  perf-sched.wait_and_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork
      0.02 ± 55%     -79.6%       0.01 ± 98%  perf-sched.wait_time.avg.ms.d_alloc_parallel.__lookup_slow.walk_component.link_path_walk.part
      1.82 ±  3%   +4445.6%      82.85 ±  4%  perf-sched.wait_time.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      3.47 ± 10%    +389.5%      16.96 ±  4%  perf-sched.wait_time.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      5.18 ± 92%   +1930.5%     105.14 ± 64%  perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault.[unknown]
     11.39 ± 43%   +2168.2%     258.38 ± 11%  perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
    132.06 ± 26%    +372.5%     623.97 ± 45%  perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown]
     94.06 ± 18%    +122.7%     209.45 ± 29%  perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
      0.02 ± 57%     -90.4%       0.00 ±223%  perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.ret_from_fork.[unknown]
      0.47 ± 61%    +118.1%       1.03 ± 39%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.__kmalloc_node.__vmalloc_node_range.dup_task_struct
      1.51 ±100%  +20748.9%     315.17 ±106%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.down_write.anon_vma_clone.__split_vma
      0.05 ± 50%     -98.7%       0.00 ±223%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.dput.d_alloc_parallel.__lookup_slow
     36.86 ± 85%    +717.8%     301.47 ± 43%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.generic_perform_write.__generic_file_write_iter.generic_file_write_iter
      3.18 ± 14%   +3169.3%     104.10 ± 82%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.shmem_undo_range.shmem_truncate_range.shmem_evict_inode
      0.10 ± 43%     -88.9%       0.01 ± 73%  perf-sched.wait_time.avg.ms.rwsem_down_read_slowpath.path_openat.do_filp_open.do_sys_openat2
      0.11 ± 37%     -90.7%       0.01 ± 34%  perf-sched.wait_time.avg.ms.rwsem_down_read_slowpath.walk_component.link_path_walk.part.0
    431.09 ±  5%     +37.1%     590.84 ±  3%  perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
      0.95 ± 28%     -81.9%       0.17 ±132%  perf-sched.wait_time.max.ms.d_alloc_parallel.__lookup_slow.walk_component.link_path_walk.part
      1514 ± 41%    +230.5%       5006        perf-sched.wait_time.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      1024          +388.6%       5007        perf-sched.wait_time.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
    506.35 ± 97%    +888.4%       5004        perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault.[unknown]
    852.80 ± 44%    +487.0%       5006        perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
      1019          +390.9%       5004        perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown]
      1011          +394.6%       5004        perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
      0.59 ± 42%     -92.5%       0.04 ±223%  perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.ret_from_fork.[unknown]
      2.98 ± 85%  +83912.9%       2505 ± 99%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.down_write.anon_vma_clone.__split_vma
      0.25 ± 64%     -99.5%       0.00 ±223%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.dput.d_alloc_parallel.__lookup_slow
      4.39 ± 35%     -64.9%       1.54 ± 85%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.prepare_creds.copy_creds
      5.41 ± 48%  +61566.5%       3337 ± 70%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.remove_vma.exit_mmap.mmput
      0.78 ±138%  +1.1e+05%     836.18 ±222%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.wp_page_copy.__handle_mm_fault.handle_mm_fault
      1.36 ± 78%     -89.9%       0.14 ± 96%  perf-sched.wait_time.max.ms.rwsem_down_read_slowpath.path_openat.do_filp_open.do_sys_openat2
      3.77 ± 15%     -84.0%       0.60 ± 34%  perf-sched.wait_time.max.ms.rwsem_down_read_slowpath.walk_component.link_path_walk.part.0
      1022          +389.7%       5004        perf-sched.wait_time.max.ms.rwsem_down_read_slowpath.walk_component.path_lookupat.isra.0
    178.25 ±210%     -99.4%       1.10 ±124%  perf-sched.wait_time.max.ms.rwsem_down_write_slowpath.dup_mmap.dup_mm.copy_process
      1.92 ± 85%     -88.7%       0.22 ±223%  perf-sched.wait_time.max.ms.rwsem_down_write_slowpath.vma_link.mmap_region.do_mmap
    362.45 ± 10%     -37.3%     227.27 ± 30%  perf-sched.wait_time.max.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
      3081 ±  9%    +100.1%       6166 ± 14%  perf-sched.wait_time.max.ms.smpboot_thread_fn.kthread.ret_from_fork
      7.00 ± 64%      -7.0        0.00        perf-profile.calltrace.cycles-pp.scan_positives.dcache_readdir.iterate_dir.__x64_sys_getdents64.do_syscall_64
      7.19 ± 63%      -6.9        0.29 ±100%  perf-profile.calltrace.cycles-pp.iterate_dir.__x64_sys_getdents64.do_syscall_64.entry_SYSCALL_64_after_hwframe
      7.18 ± 63%      -6.9        0.28 ±100%  perf-profile.calltrace.cycles-pp.dcache_readdir.iterate_dir.__x64_sys_getdents64.do_syscall_64.entry_SYSCALL_64_after_hwframe
      6.83 ± 65%      -6.8        0.00        perf-profile.calltrace.cycles-pp._raw_spin_lock.scan_positives.dcache_readdir.iterate_dir.__x64_sys_getdents64
      7.19 ± 63%      -6.8        0.37 ± 71%  perf-profile.calltrace.cycles-pp.__x64_sys_getdents64.do_syscall_64.entry_SYSCALL_64_after_hwframe
      6.56 ± 67%      -6.6        0.00        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.scan_positives.dcache_readdir.iterate_dir
      2.90 ± 38%      -2.5        0.38 ± 70%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_fork
      2.90 ± 38%      -2.5        0.38 ± 70%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_fork
      2.90 ± 38%      -2.5        0.38 ± 70%  perf-profile.calltrace.cycles-pp.__do_sys_clone.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_fork
      2.90 ± 38%      -2.5        0.38 ± 70%  perf-profile.calltrace.cycles-pp.kernel_clone.__do_sys_clone.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_fork
      3.11 ± 36%      -2.5        0.62 ± 13%  perf-profile.calltrace.cycles-pp.__libc_fork
      1.23 ± 63%      -1.0        0.27 ±100%  perf-profile.calltrace.cycles-pp.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.08 ± 18%      -0.5        0.58 ± 52%  perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
      1.53 ± 20%      -0.5        1.04 ±  9%  perf-profile.calltrace.cycles-pp.execve
      1.53 ± 20%      -0.5        1.04 ±  9%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.execve
      1.53 ± 20%      -0.5        1.04 ±  9%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve
      1.53 ± 20%      -0.5        1.04 ±  9%  perf-profile.calltrace.cycles-pp.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve
      1.52 ± 20%      -0.5        1.04 ±  9%  perf-profile.calltrace.cycles-pp.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve
      0.89 ± 15%      -0.3        0.62 ±  8%  perf-profile.calltrace.cycles-pp.walk_component.link_path_walk.path_openat.do_filp_open.do_sys_openat2
      0.93 ± 15%      -0.3        0.67 ±  7%  perf-profile.calltrace.cycles-pp.link_path_walk.path_openat.do_filp_open.do_sys_openat2.do_sys_open
      1.17 ± 10%      +0.3        1.51 ± 10%  perf-profile.calltrace.cycles-pp.zap_pte_range.unmap_page_range.unmap_vmas.exit_mmap.mmput
      1.21 ± 10%      +0.3        1.56 ± 10%  perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.exit_mmap.mmput.do_exit
      1.26 ± 10%      +0.4        1.61 ±  9%  perf-profile.calltrace.cycles-pp.unmap_vmas.exit_mmap.mmput.do_exit.do_group_exit
      0.35 ± 70%      +0.4        0.71 ±  9%  perf-profile.calltrace.cycles-pp.page_remove_rmap.zap_pte_range.unmap_page_range.unmap_vmas.exit_mmap
      0.48 ± 45%      +0.4        0.89 ± 11%  perf-profile.calltrace.cycles-pp.shmem_file_read_iter.new_sync_read.vfs_read.ksys_read.do_syscall_64
      0.98 ±  9%      +0.4        1.41 ± 10%  perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.66 ± 14%      +0.4        1.08 ± 10%  perf-profile.calltrace.cycles-pp.copy_page_range.dup_mmap.dup_mm.copy_process.kernel_clone
      1.03 ±  8%      +0.5        1.49 ± 10%  perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.61 ±  8%      +0.5        1.11 ± 18%  perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.64 ± 14%      +0.5        1.15 ±  6%  perf-profile.calltrace.cycles-pp.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.73 ± 13%      +0.5        2.24 ±  7%  perf-profile.calltrace.cycles-pp.__softirqentry_text_start.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      0.65 ±  8%      +0.5        1.17 ± 17%  perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.55 ± 46%      +0.6        1.13 ±  6%  perf-profile.calltrace.cycles-pp.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.41 ± 71%      +0.6        1.00 ± 16%  perf-profile.calltrace.cycles-pp.anon_vma_clone.anon_vma_fork.dup_mmap.dup_mm.copy_process
      0.20 ±141%      +0.6        0.82 ± 10%  perf-profile.calltrace.cycles-pp.copy_pte_range.copy_page_range.dup_mmap.dup_mm.copy_process
      1.85 ± 13%      +0.7        2.54 ±  8%  perf-profile.calltrace.cycles-pp.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      0.00            +0.7        0.72 ± 22%  perf-profile.calltrace.cycles-pp.__do_munmap.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.18 ±141%      +0.7        0.92 ± 23%  perf-profile.calltrace.cycles-pp.task_work_run.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +0.8        0.78 ± 27%  perf-profile.calltrace.cycles-pp.do_anonymous_page.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault
      0.58 ± 46%      +0.8        1.36 ± 36%  perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry.start_secondary
      0.00            +0.9        0.85 ± 34%  perf-profile.calltrace.cycles-pp.shmem_undo_range.shmem_truncate_range.shmem_evict_inode.evict.do_unlinkat
      0.75 ± 11%      +0.9        1.60 ±  7%  perf-profile.calltrace.cycles-pp.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +0.9        0.86 ± 34%  perf-profile.calltrace.cycles-pp.shmem_truncate_range.shmem_evict_inode.evict.do_unlinkat.do_syscall_64
      0.00            +0.9        0.86 ± 34%  perf-profile.calltrace.cycles-pp.evict.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +0.9        0.86 ± 34%  perf-profile.calltrace.cycles-pp.shmem_evict_inode.evict.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +0.9        0.87 ± 25%  perf-profile.calltrace.cycles-pp.__fput.task_work_run.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
      0.00            +0.9        0.90 ± 32%  perf-profile.calltrace.cycles-pp.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +0.9        0.90 ± 33%  perf-profile.calltrace.cycles-pp.wp_page_copy.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault
      0.00            +1.0        0.96 ± 11%  perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
      0.10 ±223%      +1.0        1.06 ± 37%  perf-profile.calltrace.cycles-pp.ktime_get.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      0.43 ± 72%      +1.0        1.42 ± 34%  perf-profile.calltrace.cycles-pp.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.73 ± 10%      +1.0        1.76 ± 11%  perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
      2.22 ± 11%      +1.1        3.34 ± 11%  perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
      0.75 ± 11%      +1.1        1.87 ±  8%  perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
      1.05 ± 11%      +1.2        2.22 ± 25%  perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      0.80 ± 11%      +1.2        2.01 ±  8%  perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      2.32 ± 11%      +1.4        3.70 ± 13%  perf-profile.calltrace.cycles-pp.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
      2.55 ± 11%      +1.4        3.94 ± 24%  perf-profile.calltrace.cycles-pp.shmem_getpage_gfp.shmem_write_begin.generic_perform_write.__generic_file_write_iter.generic_file_write_iter
      2.60 ± 11%      +1.4        3.99 ± 24%  perf-profile.calltrace.cycles-pp.shmem_write_begin.generic_perform_write.__generic_file_write_iter.generic_file_write_iter.new_sync_write
      2.56 ± 11%      +1.4        3.98 ± 12%  perf-profile.calltrace.cycles-pp.exc_page_fault.asm_exc_page_fault
      2.55 ± 11%      +1.4        3.97 ± 12%  perf-profile.calltrace.cycles-pp.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
      2.60 ± 11%      +1.4        4.03 ± 11%  perf-profile.calltrace.cycles-pp.asm_exc_page_fault
      3.33 ± 27%      +1.6        4.96 ± 10%  perf-profile.calltrace.cycles-pp.__do_sys_clone.do_syscall_64.entry_SYSCALL_64_after_hwframe
      3.33 ± 27%      +1.6        4.96 ± 10%  perf-profile.calltrace.cycles-pp.kernel_clone.__do_sys_clone.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.09 ±223%      +1.9        1.94 ± 41%  perf-profile.calltrace.cycles-pp.shmem_add_to_page_cache.shmem_getpage_gfp.shmem_write_begin.generic_perform_write.__generic_file_write_iter
      1.09 ± 17%      +1.9        3.03 ± 16%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      1.96 ± 12%      +3.1        5.10 ± 10%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      1.98 ± 13%      +3.2        5.19 ± 10%  perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      3.84 ± 10%      +3.7        7.54 ± 10%  perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      3.94 ± 10%      +3.8        7.72 ± 10%  perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      4.29 ± 13%      +4.6        8.93 ±  8%  perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle
      4.59 ± 12%      +5.5       10.06 ±  9%  perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     32.38 ± 10%     +14.5       46.86 ±  8%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     37.20 ±  8%     +20.0       57.16 ±  6%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
     37.47 ±  8%     +20.3       57.80 ±  6%  perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     39.04 ±  8%     +21.9       60.91 ±  5%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     39.04 ±  8%     +21.9       60.94 ±  5%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     39.04 ±  8%     +21.9       60.94 ±  5%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
     39.29 ±  8%     +22.0       61.29 ±  5%  perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
     38.93 ±  6%      -7.9       31.04 ±  8%  perf-profile.children.cycles-pp.do_syscall_64
     39.07 ±  6%      -7.9       31.19 ±  8%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
      8.37 ± 53%      -7.2        1.22 ± 18%  perf-profile.children.cycles-pp._raw_spin_lock
      7.19 ± 63%      -6.7        0.48 ± 24%  perf-profile.children.cycles-pp.iterate_dir
      7.19 ± 63%      -6.7        0.48 ± 23%  perf-profile.children.cycles-pp.__x64_sys_getdents64
      7.18 ± 63%      -6.7        0.48 ± 23%  perf-profile.children.cycles-pp.dcache_readdir
      7.00 ± 64%      -6.6        0.40 ± 25%  perf-profile.children.cycles-pp.scan_positives
      7.90 ± 56%      -6.6        1.33 ± 43%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      4.42 ± 79%      -3.8        0.63 ± 24%  perf-profile.children.cycles-pp.rwsem_down_write_slowpath
      3.79 ± 90%      -3.5        0.25 ± 29%  perf-profile.children.cycles-pp.osq_lock
      3.12 ± 36%      -2.5        0.62 ± 12%  perf-profile.children.cycles-pp.__libc_fork
      1.92 ± 80%      -1.6        0.36 ±  5%  perf-profile.children.cycles-pp.unlink_file_vma
      0.84 ± 13%      -0.8        0.05 ± 46%  perf-profile.children.cycles-pp.read
      2.80 ± 13%      -0.7        2.06 ±  7%  perf-profile.children.cycles-pp.path_openat
      2.81 ± 13%      -0.7        2.06 ±  7%  perf-profile.children.cycles-pp.do_filp_open
      1.36 ± 57%      -0.7        0.62 ±  8%  perf-profile.children.cycles-pp.ksys_mmap_pgoff
      2.87 ± 13%      -0.7        2.15 ±  6%  perf-profile.children.cycles-pp.do_sys_openat2
      2.87 ± 12%      -0.7        2.15 ±  6%  perf-profile.children.cycles-pp.do_sys_open
      1.31 ± 13%      -0.7        0.60 ± 12%  perf-profile.children.cycles-pp.common_file_perm
      1.33 ± 13%      -0.7        0.63 ± 12%  perf-profile.children.cycles-pp.security_file_permission
      2.61 ± 15%      -0.7        1.95 ± 15%  perf-profile.children.cycles-pp.rcu_do_batch
      2.66 ± 15%      -0.6        2.05 ± 14%  perf-profile.children.cycles-pp.rcu_core
      1.55 ± 19%      -0.5        1.04 ±  9%  perf-profile.children.cycles-pp.execve
      1.12 ± 15%      -0.5        0.61 ± 11%  perf-profile.children.cycles-pp.refill_obj_stock
      0.84 ± 22%      -0.4        0.40 ± 49%  perf-profile.children.cycles-pp.run_ksoftirqd
      0.87 ± 21%      -0.4        0.43 ± 46%  perf-profile.children.cycles-pp.smpboot_thread_fn
      1.08 ± 18%      -0.4        0.66 ± 28%  perf-profile.children.cycles-pp.kthread
      1.10 ± 17%      -0.4        0.68 ± 12%  perf-profile.children.cycles-pp.page_counter_try_charge
      0.89 ± 48%      -0.4        0.48 ±  9%  perf-profile.children.cycles-pp.__vma_adjust
      1.23 ± 17%      -0.4        0.84 ± 23%  perf-profile.children.cycles-pp.ret_from_fork
      0.91 ± 16%      -0.4        0.53 ±  7%  perf-profile.children.cycles-pp.page_counter_uncharge
      0.90 ± 16%      -0.4        0.53 ±  7%  perf-profile.children.cycles-pp.page_counter_cancel
      1.00 ± 17%      -0.4        0.63 ± 14%  perf-profile.children.cycles-pp.__memcg_kmem_charge
      0.74 ± 15%      -0.3        0.41 ± 11%  perf-profile.children.cycles-pp.drain_obj_stock
      0.54 ± 52%      -0.3        0.23 ±  9%  perf-profile.children.cycles-pp.vma_link
      0.57 ± 21%      -0.3        0.27 ± 12%  perf-profile.children.cycles-pp.propagate_protected_usage
      0.59 ± 13%      -0.3        0.32 ± 16%  perf-profile.children.cycles-pp.do_open
      0.39 ± 18%      -0.2        0.14 ±  8%  perf-profile.children.cycles-pp.lockref_put_or_lock
      0.47 ± 17%      -0.2        0.26 ±  7%  perf-profile.children.cycles-pp.__memcg_kmem_uncharge
      0.22 ± 18%      -0.2        0.06 ± 52%  perf-profile.children.cycles-pp.apparmor_task_getsecid
      0.22 ± 18%      -0.2        0.06 ± 52%  perf-profile.children.cycles-pp.security_task_getsecid_subj
      0.31 ± 14%      -0.2        0.16 ± 17%  perf-profile.children.cycles-pp.apparmor_file_alloc_security
      0.34 ± 14%      -0.1        0.19 ± 16%  perf-profile.children.cycles-pp.security_file_alloc
      0.20 ± 18%      -0.1        0.05 ± 51%  perf-profile.children.cycles-pp.ima_file_check
      0.43 ± 13%      -0.1        0.29 ±  9%  perf-profile.children.cycles-pp.workingset_age_nonresident
      0.25 ± 15%      -0.1        0.11 ±  8%  perf-profile.children.cycles-pp.lockref_get_not_dead
      0.26 ± 15%      -0.1        0.13 ±  8%  perf-profile.children.cycles-pp.__legitimize_path
      0.48 ± 15%      -0.1        0.35 ± 14%  perf-profile.children.cycles-pp.uncharge_batch
      0.27 ± 15%      -0.1        0.14 ±  5%  perf-profile.children.cycles-pp.try_to_unlazy
      0.35 ± 13%      -0.1        0.22 ± 12%  perf-profile.children.cycles-pp.step_into
      0.30 ± 10%      -0.1        0.18 ± 12%  perf-profile.children.cycles-pp.task_tick_fair
      0.18 ± 18%      -0.1        0.06 ± 47%  perf-profile.children.cycles-pp.aa_get_task_label
      0.26 ± 13%      -0.1        0.14 ± 14%  perf-profile.children.cycles-pp.do_dentry_open
      0.20 ± 13%      -0.1        0.09 ± 17%  perf-profile.children.cycles-pp.security_file_open
      0.20 ± 14%      -0.1        0.08 ± 16%  perf-profile.children.cycles-pp.apparmor_file_open
      0.22 ± 13%      -0.1        0.11 ± 16%  perf-profile.children.cycles-pp.security_file_free
      0.22 ± 13%      -0.1        0.11 ± 17%  perf-profile.children.cycles-pp.apparmor_file_free_security
      0.22 ± 13%      -0.1        0.11 ±  9%  perf-profile.children.cycles-pp.terminate_walk
      0.31 ± 13%      -0.1        0.21 ±  5%  perf-profile.children.cycles-pp.lookup_fast
      0.18 ±  7%      -0.1        0.09 ± 12%  perf-profile.children.cycles-pp.update_cfs_group
      0.18 ± 15%      -0.1        0.10 ± 21%  perf-profile.children.cycles-pp.__x64_sys_kill
      0.17 ± 14%      -0.1        0.09 ± 23%  perf-profile.children.cycles-pp.kill_something_info
      0.27 ±  7%      -0.1        0.19 ± 10%  perf-profile.children.cycles-pp.__entry_text_start
      0.16 ± 17%      -0.1        0.09 ± 19%  perf-profile.children.cycles-pp.kill_pid_info
      0.13 ± 16%      -0.1        0.05 ± 48%  perf-profile.children.cycles-pp.apparmor_task_kill
      0.13 ± 16%      -0.1        0.06 ± 18%  perf-profile.children.cycles-pp.security_task_kill
      0.13 ± 18%      -0.1        0.06 ± 17%  perf-profile.children.cycles-pp.group_send_sig_info
      0.08 ±  8%      -0.0        0.06 ± 13%  perf-profile.children.cycles-pp.rcu_cblist_dequeue
      0.08 ± 13%      -0.0        0.06 ± 11%  perf-profile.children.cycles-pp.lockref_put_return
      0.06 ± 11%      +0.0        0.08 ±  6%  perf-profile.children.cycles-pp.create_elf_tables
      0.08 ±  8%      +0.0        0.10 ±  8%  perf-profile.children.cycles-pp.flush_tlb_func
      0.11 ±  8%      +0.0        0.13 ±  8%  perf-profile.children.cycles-pp.flush_tlb_mm_range
      0.00            +0.1        0.06 ± 16%  perf-profile.children.cycles-pp.hrtimer_get_next_event
      0.00            +0.1        0.06 ± 16%  perf-profile.children.cycles-pp.cpumask_next_and
      0.00            +0.1        0.07 ± 19%  perf-profile.children.cycles-pp.call_cpuidle
      0.05 ± 47%      +0.1        0.11 ± 41%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.01 ±223%      +0.1        0.08 ± 30%  perf-profile.children.cycles-pp.pagevec_lru_move_fn
      0.01 ±223%      +0.1        0.08 ± 19%  perf-profile.children.cycles-pp.trigger_load_balance
      0.24 ± 14%      +0.1        0.31 ± 11%  perf-profile.children.cycles-pp.update_sd_lb_stats
      0.01 ±223%      +0.1        0.09 ± 32%  perf-profile.children.cycles-pp.__hrtimer_get_next_event
      0.00            +0.1        0.08 ± 13%  perf-profile.children.cycles-pp.cgroup_rstat_updated
      0.07 ± 14%      +0.1        0.15 ± 20%  perf-profile.children.cycles-pp.__intel_pmu_enable_all
      0.04 ± 73%      +0.1        0.12 ± 22%  perf-profile.children.cycles-pp.update_irq_load_avg
      0.24 ± 14%      +0.1        0.33 ± 11%  perf-profile.children.cycles-pp.find_busiest_group
      0.02 ±141%      +0.1        0.11 ± 29%  perf-profile.children.cycles-pp.free_pgd_range
      0.04 ± 73%      +0.1        0.13 ± 20%  perf-profile.children.cycles-pp.hrtimer_next_event_without
      0.04 ± 82%      +0.1        0.14 ± 32%  perf-profile.children.cycles-pp.__hrtimer_next_event_base
      0.06 ± 11%      +0.1        0.16 ± 10%  perf-profile.children.cycles-pp.arch_scale_freq_tick
      0.09 ± 28%      +0.1        0.20 ± 29%  perf-profile.children.cycles-pp.tsc_verify_tsc_adjust
      0.10 ± 23%      +0.1        0.20 ± 27%  perf-profile.children.cycles-pp.arch_cpu_idle_enter
      0.01 ±223%      +0.1        0.12 ± 43%  perf-profile.children.cycles-pp.cpuidle_governor_latency_req
      0.05 ± 52%      +0.1        0.16 ± 12%  perf-profile.children.cycles-pp.get_next_timer_interrupt
      0.01 ±223%      +0.1        0.12 ± 49%  perf-profile.children.cycles-pp.timerqueue_add
      0.11 ± 10%      +0.1        0.23 ± 11%  perf-profile.children.cycles-pp.read_tsc
      0.01 ±223%      +0.1        0.13 ± 44%  perf-profile.children.cycles-pp.enqueue_hrtimer
      0.06 ± 53%      +0.1        0.18 ± 28%  perf-profile.children.cycles-pp.hrtimer_update_next_event
      0.02 ±223%      +0.1        0.15 ± 51%  perf-profile.children.cycles-pp.timerqueue_del
      0.01 ±223%      +0.1        0.14 ± 37%  perf-profile.children.cycles-pp.rcu_eqs_exit
      0.04 ±109%      +0.1        0.17 ± 36%  perf-profile.children.cycles-pp.asm_sysvec_irq_work
      0.04 ±109%      +0.1        0.17 ± 36%  perf-profile.children.cycles-pp.sysvec_irq_work
      0.04 ±109%      +0.1        0.17 ± 36%  perf-profile.children.cycles-pp.__sysvec_irq_work
      0.04 ±109%      +0.1        0.17 ± 36%  perf-profile.children.cycles-pp.irq_work_run
      0.04 ±109%      +0.1        0.17 ± 36%  perf-profile.children.cycles-pp.irq_work_single
      0.04 ±109%      +0.1        0.17 ± 36%  perf-profile.children.cycles-pp.printk
      0.13 ± 28%      +0.1        0.27 ± 21%  perf-profile.children.cycles-pp.rcu_sched_clock_irq
      0.11 ± 16%      +0.1        0.25 ± 17%  perf-profile.children.cycles-pp.update_rq_clock
      0.10 ± 20%      +0.1        0.24 ± 18%  perf-profile.children.cycles-pp.update_blocked_averages
      0.04 ±112%      +0.1        0.18 ± 36%  perf-profile.children.cycles-pp.irq_work_run_list
      0.28 ± 15%      +0.1        0.43 ± 14%  perf-profile.children.cycles-pp.load_balance
      0.12 ± 10%      +0.1        0.26 ±  8%  perf-profile.children.cycles-pp.native_sched_clock
      0.04 ±130%      +0.2        0.20 ± 52%  perf-profile.children.cycles-pp.__remove_hrtimer
      0.10 ± 16%      +0.2        0.26 ± 54%  perf-profile.children.cycles-pp.lru_add_drain
      0.10 ± 12%      +0.2        0.26 ± 55%  perf-profile.children.cycles-pp.lru_add_drain_cpu
      0.10 ± 24%      +0.2        0.26 ± 19%  perf-profile.children.cycles-pp.run_rebalance_domains
      0.09 ± 19%      +0.2        0.25 ± 44%  perf-profile.children.cycles-pp.mem_cgroup_charge_statistics
      0.04 ±115%      +0.2        0.20 ± 39%  perf-profile.children.cycles-pp.rcu_idle_exit
      0.18 ± 15%      +0.2        0.35 ± 27%  perf-profile.children.cycles-pp.lapic_next_deadline
      0.14 ± 14%      +0.2        0.31 ±  8%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.05 ± 45%      +0.2        0.25 ± 41%  perf-profile.children.cycles-pp.page_add_new_anon_rmap
      0.04 ± 72%      +0.2        0.27 ± 53%  perf-profile.children.cycles-pp.uncharge_page
      0.22 ± 29%      +0.2        0.46 ± 43%  perf-profile.children.cycles-pp.tick_nohz_irq_exit
      0.22 ± 20%      +0.2        0.46 ± 35%  perf-profile.children.cycles-pp.irqtime_account_irq
      0.12 ± 53%      +0.3        0.37 ± 54%  perf-profile.children.cycles-pp.tick_irq_enter
      0.12 ± 54%      +0.3        0.39 ± 51%  perf-profile.children.cycles-pp.irq_enter_rcu
      0.44 ± 11%      +0.3        0.71 ± 17%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.60 ± 10%      +0.3        0.92 ±  9%  perf-profile.children.cycles-pp.page_remove_rmap
      0.28 ± 10%      +0.4        0.63 ± 11%  perf-profile.children.cycles-pp.rebalance_domains
      1.26 ± 10%      +0.4        1.62 ±  7%  perf-profile.children.cycles-pp.__x64_sys_brk
      0.16 ± 15%      +0.4        0.54 ± 51%  perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
      0.35 ±  9%      +0.4        0.73 ± 42%  perf-profile.children.cycles-pp.lru_cache_add
      0.46 ±  9%      +0.4        0.84 ± 26%  perf-profile.children.cycles-pp.do_anonymous_page
      0.18 ± 18%      +0.4        0.56 ± 50%  perf-profile.children.cycles-pp.__count_memcg_events
      0.14 ± 10%      +0.4        0.54 ± 38%  perf-profile.children.cycles-pp.__mod_lruvec_page_state
      0.22 ±  8%      +0.4        0.65 ± 41%  perf-profile.children.cycles-pp.__delete_from_page_cache
      0.14 ± 11%      +0.4        0.57 ± 47%  perf-profile.children.cycles-pp.unaccount_page_cache_page
      0.31 ±  8%      +0.4        0.75 ± 36%  perf-profile.children.cycles-pp.truncate_inode_page
      0.26 ±  9%      +0.4        0.70 ± 38%  perf-profile.children.cycles-pp.delete_from_page_cache
      1.40 ± 10%      +0.5        1.85 ±  9%  perf-profile.children.cycles-pp.zap_pte_range
      0.75 ±  7%      +0.5        1.20 ±  9%  perf-profile.children.cycles-pp.scheduler_tick
      1.46 ± 10%      +0.5        1.91 ±  8%  perf-profile.children.cycles-pp.unmap_page_range
      1.52 ± 10%      +0.5        1.99 ±  8%  perf-profile.children.cycles-pp.unmap_vmas
      0.47 ± 11%      +0.5        1.00 ± 32%  perf-profile.children.cycles-pp.wp_page_copy
      0.42 ± 10%      +0.5        0.94 ± 47%  perf-profile.children.cycles-pp.__pagevec_lru_add
      0.28 ± 17%      +0.5        0.82 ± 44%  perf-profile.children.cycles-pp.get_mem_cgroup_from_mm
      2.45 ± 12%      +0.6        3.07 ±  6%  perf-profile.children.cycles-pp.irq_exit_rcu
      0.66 ± 17%      +0.7        1.37 ± 36%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.69 ± 15%      +0.9        1.55 ± 32%  perf-profile.children.cycles-pp.clockevents_program_event
      1.16 ±  7%      +0.9        2.06 ± 10%  perf-profile.children.cycles-pp.update_process_times
      3.16 ± 11%      +0.9        4.06 ± 10%  perf-profile.children.cycles-pp.__handle_mm_fault
      1.18 ±  7%      +1.0        2.17 ±  8%  perf-profile.children.cycles-pp.tick_sched_handle
      1.24 ±  7%      +1.1        2.32 ±  7%  perf-profile.children.cycles-pp.tick_sched_timer
      3.70 ± 11%      +1.1        4.85 ± 11%  perf-profile.children.cycles-pp.asm_exc_page_fault
      3.31 ± 11%      +1.2        4.48 ± 12%  perf-profile.children.cycles-pp.handle_mm_fault
      3.56 ± 11%      +1.2        4.74 ± 11%  perf-profile.children.cycles-pp.do_user_addr_fault
      3.57 ± 11%      +1.2        4.75 ± 11%  perf-profile.children.cycles-pp.exc_page_fault
      1.06 ± 11%      +1.2        2.24 ± 25%  perf-profile.children.cycles-pp.menu_select
      0.83 ± 19%      +1.2        2.01 ± 25%  perf-profile.children.cycles-pp.ktime_get
      0.73 ± 15%      +1.2        1.95 ± 41%  perf-profile.children.cycles-pp.shmem_add_to_page_cache
      0.71 ± 16%      +1.3        2.02 ± 42%  perf-profile.children.cycles-pp.mem_cgroup_charge
      2.87 ± 11%      +1.4        4.25 ± 22%  perf-profile.children.cycles-pp.shmem_getpage_gfp
      2.60 ± 11%      +1.4        4.00 ± 24%  perf-profile.children.cycles-pp.shmem_write_begin
      1.58 ± 11%      +1.8        3.39 ± 15%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      2.57 ±  9%      +3.0        5.61 ± 10%  perf-profile.children.cycles-pp.hrtimer_interrupt
      2.60 ± 10%      +3.1        5.70 ±  9%  perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
      0.39 ± 15%      +3.6        3.95 ± 51%  perf-profile.children.cycles-pp.__mod_memcg_state
      0.59 ± 12%      +3.6        4.20 ± 47%  perf-profile.children.cycles-pp.__mod_memcg_lruvec_state
      5.52 ± 11%      +4.5        9.99 ±  7%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
      5.86 ± 11%      +4.9       10.72 ±  7%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
     32.58 ± 10%     +14.5       47.10 ±  8%  perf-profile.children.cycles-pp.intel_idle
     37.70 ±  8%     +20.4       58.12 ±  6%  perf-profile.children.cycles-pp.cpuidle_enter_state
     37.71 ±  8%     +20.4       58.13 ±  6%  perf-profile.children.cycles-pp.cpuidle_enter
     39.04 ±  8%     +21.9       60.94 ±  5%  perf-profile.children.cycles-pp.start_secondary
     39.29 ±  8%     +22.0       61.29 ±  5%  perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     39.29 ±  8%     +22.0       61.29 ±  5%  perf-profile.children.cycles-pp.cpu_startup_entry
     39.29 ±  8%     +22.0       61.29 ±  5%  perf-profile.children.cycles-pp.do_idle
      7.81 ± 56%      -6.5        1.32 ± 43%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      3.73 ± 90%      -3.5        0.25 ± 30%  perf-profile.self.cycles-pp.osq_lock
      1.23 ± 13%      -0.7        0.54 ± 12%  perf-profile.self.cycles-pp.common_file_perm
      2.15 ± 15%      -0.6        1.56 ± 17%  perf-profile.self.cycles-pp.kmem_cache_free
      0.56 ± 20%      -0.3        0.26 ± 12%  perf-profile.self.cycles-pp.propagate_protected_usage
      0.83 ± 15%      -0.3        0.56 ± 11%  perf-profile.self.cycles-pp.page_counter_try_charge
      0.59 ± 14%      -0.2        0.38 ±  7%  perf-profile.self.cycles-pp.page_counter_cancel
      0.41 ± 15%      -0.2        0.20 ± 15%  perf-profile.self.cycles-pp.drain_obj_stock
      0.37 ± 14%      -0.2        0.19 ± 14%  perf-profile.self.cycles-pp.refill_obj_stock
      0.30 ± 13%      -0.2        0.15 ± 17%  perf-profile.self.cycles-pp.apparmor_file_alloc_security
      0.42 ± 13%      -0.1        0.29 ±  9%  perf-profile.self.cycles-pp.workingset_age_nonresident
      0.18 ± 18%      -0.1        0.06 ± 47%  perf-profile.self.cycles-pp.aa_get_task_label
      0.22 ± 13%      -0.1        0.11 ± 17%  perf-profile.self.cycles-pp.apparmor_file_free_security
      0.20 ± 14%      -0.1        0.08 ± 16%  perf-profile.self.cycles-pp.apparmor_file_open
      0.14 ± 33%      -0.1        0.05 ± 45%  perf-profile.self.cycles-pp.rwsem_down_write_slowpath
      0.18 ±  7%      -0.1        0.09 ± 12%  perf-profile.self.cycles-pp.update_cfs_group
      0.16 ± 11%      -0.1        0.09 ± 11%  perf-profile.self.cycles-pp.lockref_put_or_lock
      0.13 ± 12%      -0.0        0.09 ±  7%  perf-profile.self.cycles-pp.lockref_get_not_dead
      0.09 ± 14%      -0.0        0.04 ± 75%  perf-profile.self.cycles-pp.update_load_avg
      0.08 ±  8%      -0.0        0.05 ±  9%  perf-profile.self.cycles-pp.rcu_cblist_dequeue
      0.08 ± 13%      -0.0        0.06 ± 11%  perf-profile.self.cycles-pp.lockref_put_return
      0.04 ± 72%      +0.0        0.08 ± 17%  perf-profile.self.cycles-pp.__softirqentry_text_start
      0.17 ± 14%      +0.1        0.23 ±  9%  perf-profile.self.cycles-pp.update_sd_lb_stats
      0.04 ± 73%      +0.1        0.09 ± 18%  perf-profile.self.cycles-pp.update_rq_clock
      0.04 ± 73%      +0.1        0.11 ± 19%  perf-profile.self.cycles-pp.update_blocked_averages
      0.00            +0.1        0.07 ± 21%  perf-profile.self.cycles-pp.trigger_load_balance
      0.00            +0.1        0.08 ± 13%  perf-profile.self.cycles-pp.cgroup_rstat_updated
      0.07 ± 14%      +0.1        0.15 ± 20%  perf-profile.self.cycles-pp.__intel_pmu_enable_all
      0.04 ± 73%      +0.1        0.12 ± 22%  perf-profile.self.cycles-pp.update_irq_load_avg
      0.02 ±142%      +0.1        0.11 ± 18%  perf-profile.self.cycles-pp.hrtimer_interrupt
      0.03 ±108%      +0.1        0.13 ± 31%  perf-profile.self.cycles-pp.__hrtimer_next_event_base
      0.06 ± 11%      +0.1        0.16 ± 10%  perf-profile.self.cycles-pp.arch_scale_freq_tick
      0.18 ± 10%      +0.1        0.28 ±  9%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.03 ± 99%      +0.1        0.13 ± 41%  perf-profile.self.cycles-pp.workingset_activation
      0.01 ±223%      +0.1        0.12 ± 46%  perf-profile.self.cycles-pp.__hrtimer_run_queues
      0.13 ± 28%      +0.1        0.24 ± 17%  perf-profile.self.cycles-pp.rcu_sched_clock_irq
      0.11 ± 14%      +0.1        0.23 ± 11%  perf-profile.self.cycles-pp.read_tsc
      0.06 ± 60%      +0.1        0.19 ± 24%  perf-profile.self.cycles-pp.do_idle
      0.11 ±  9%      +0.1        0.24 ±  8%  perf-profile.self.cycles-pp.native_sched_clock
      0.06 ± 46%      +0.1        0.19 ± 38%  perf-profile.self.cycles-pp.obj_cgroup_charge
      0.18 ± 15%      +0.2        0.35 ± 27%  perf-profile.self.cycles-pp.lapic_next_deadline
      0.17 ± 25%      +0.2        0.36 ± 44%  perf-profile.self.cycles-pp.update_process_times
      0.04 ± 72%      +0.2        0.26 ± 53%  perf-profile.self.cycles-pp.uncharge_page
      0.44 ± 11%      +0.3        0.71 ± 17%  perf-profile.self.cycles-pp.native_irq_return_iret
      0.18 ± 17%      +0.4        0.56 ± 50%  perf-profile.self.cycles-pp.__count_memcg_events
      0.13 ± 11%      +0.4        0.54 ± 39%  perf-profile.self.cycles-pp.__mod_lruvec_page_state
      0.27 ± 18%      +0.5        0.81 ± 44%  perf-profile.self.cycles-pp.get_mem_cgroup_from_mm
      0.38 ± 16%      +0.6        0.96 ±  8%  perf-profile.self.cycles-pp.cpuidle_enter_state
      0.16 ± 23%      +0.8        0.93 ± 47%  perf-profile.self.cycles-pp.mem_cgroup_charge
      0.74 ± 20%      +1.1        1.82 ± 28%  perf-profile.self.cycles-pp.ktime_get
      0.39 ± 15%      +3.5        3.94 ± 51%  perf-profile.self.cycles-pp.__mod_memcg_state
     32.58 ± 10%     +14.5       47.10 ±  8%  perf-profile.self.cycles-pp.intel_idle
      1391 ± 20%     -58.2%     581.50 ± 27%  interrupts.CPU0.NMI:Non-maskable_interrupts
      1391 ± 20%     -58.2%     581.50 ± 27%  interrupts.CPU0.PMI:Performance_monitoring_interrupts
    336.00 ±  6%     +27.5%     428.33 ±  7%  interrupts.CPU0.RES:Rescheduling_interrupts
      1149 ± 15%     -43.6%     647.83 ± 36%  interrupts.CPU1.NMI:Non-maskable_interrupts
      1149 ± 15%     -43.6%     647.83 ± 36%  interrupts.CPU1.PMI:Performance_monitoring_interrupts
      1196 ± 16%     -58.3%     498.67 ± 30%  interrupts.CPU10.NMI:Non-maskable_interrupts
      1196 ± 16%     -58.3%     498.67 ± 30%  interrupts.CPU10.PMI:Performance_monitoring_interrupts
      1127 ± 15%     -64.2%     403.83 ± 51%  interrupts.CPU100.NMI:Non-maskable_interrupts
      1127 ± 15%     -64.2%     403.83 ± 51%  interrupts.CPU100.PMI:Performance_monitoring_interrupts
    303.00 ±  6%     +19.1%     360.83 ±  6%  interrupts.CPU100.RES:Rescheduling_interrupts
      1206 ± 23%     -66.0%     409.83 ± 35%  interrupts.CPU101.NMI:Non-maskable_interrupts
      1206 ± 23%     -66.0%     409.83 ± 35%  interrupts.CPU101.PMI:Performance_monitoring_interrupts
    292.17 ±  7%     +28.8%     376.17 ±  8%  interrupts.CPU101.RES:Rescheduling_interrupts
      1060 ± 17%     -55.8%     468.50 ± 43%  interrupts.CPU102.NMI:Non-maskable_interrupts
      1060 ± 17%     -55.8%     468.50 ± 43%  interrupts.CPU102.PMI:Performance_monitoring_interrupts
    301.17 ±  5%     +26.2%     380.00 ±  7%  interrupts.CPU102.RES:Rescheduling_interrupts
      1163 ± 20%     -58.8%     478.67 ± 23%  interrupts.CPU103.NMI:Non-maskable_interrupts
      1163 ± 20%     -58.8%     478.67 ± 23%  interrupts.CPU103.PMI:Performance_monitoring_interrupts
    313.67 ±  5%     +23.5%     387.50 ±  4%  interrupts.CPU103.RES:Rescheduling_interrupts
      1228 ± 26%     -50.6%     606.67 ± 30%  interrupts.CPU104.NMI:Non-maskable_interrupts
      1228 ± 26%     -50.6%     606.67 ± 30%  interrupts.CPU104.PMI:Performance_monitoring_interrupts
      1209 ± 25%     -59.7%     487.67 ± 39%  interrupts.CPU105.NMI:Non-maskable_interrupts
      1209 ± 25%     -59.7%     487.67 ± 39%  interrupts.CPU105.PMI:Performance_monitoring_interrupts
    293.00 ±  6%     +28.5%     376.50 ±  6%  interrupts.CPU105.RES:Rescheduling_interrupts
      1030 ± 30%     -56.0%     453.00 ± 32%  interrupts.CPU106.NMI:Non-maskable_interrupts
      1030 ± 30%     -56.0%     453.00 ± 32%  interrupts.CPU106.PMI:Performance_monitoring_interrupts
    310.50 ±  5%     +21.9%     378.50 ±  4%  interrupts.CPU106.RES:Rescheduling_interrupts
    325.67 ±  6%     +19.1%     388.00 ±  6%  interrupts.CPU108.RES:Rescheduling_interrupts
      1093 ± 37%     -59.4%     443.83 ± 31%  interrupts.CPU109.NMI:Non-maskable_interrupts
      1093 ± 37%     -59.4%     443.83 ± 31%  interrupts.CPU109.PMI:Performance_monitoring_interrupts
    321.17 ±  3%     +22.3%     392.67 ±  7%  interrupts.CPU109.RES:Rescheduling_interrupts
    248.17 ±  5%     +27.7%     317.00 ± 11%  interrupts.CPU11.RES:Rescheduling_interrupts
    315.17 ±  3%     +19.1%     375.50 ±  5%  interrupts.CPU110.RES:Rescheduling_interrupts
    320.67 ±  4%     +18.7%     380.50 ±  6%  interrupts.CPU111.RES:Rescheduling_interrupts
    312.33 ±  4%     +23.9%     387.00 ±  8%  interrupts.CPU112.RES:Rescheduling_interrupts
      1448 ±  6%     -14.9%       1232 ± 12%  interrupts.CPU113.CAL:Function_call_interrupts
    304.33 ±  7%     +20.1%     365.50 ±  7%  interrupts.CPU113.RES:Rescheduling_interrupts
      1192 ± 22%     -61.2%     463.17 ± 20%  interrupts.CPU114.NMI:Non-maskable_interrupts
      1192 ± 22%     -61.2%     463.17 ± 20%  interrupts.CPU114.PMI:Performance_monitoring_interrupts
      1207 ± 25%     -52.5%     573.17 ± 19%  interrupts.CPU116.NMI:Non-maskable_interrupts
      1207 ± 25%     -52.5%     573.17 ± 19%  interrupts.CPU116.PMI:Performance_monitoring_interrupts
    329.67 ±  6%     +20.8%     398.33 ±  5%  interrupts.CPU116.RES:Rescheduling_interrupts
    304.17 ±  4%     +26.0%     383.33 ±  8%  interrupts.CPU117.RES:Rescheduling_interrupts
      1146 ± 23%     -45.6%     624.00 ± 27%  interrupts.CPU118.NMI:Non-maskable_interrupts
      1146 ± 23%     -45.6%     624.00 ± 27%  interrupts.CPU118.PMI:Performance_monitoring_interrupts
      1145 ± 21%     -51.5%     555.67 ± 19%  interrupts.CPU119.NMI:Non-maskable_interrupts
      1145 ± 21%     -51.5%     555.67 ± 19%  interrupts.CPU119.PMI:Performance_monitoring_interrupts
    282.67 ±  4%     +30.3%     368.33 ±  9%  interrupts.CPU119.RES:Rescheduling_interrupts
      1313 ± 17%     -54.6%     596.17 ± 32%  interrupts.CPU12.NMI:Non-maskable_interrupts
      1313 ± 17%     -54.6%     596.17 ± 32%  interrupts.CPU12.PMI:Performance_monitoring_interrupts
    276.33 ±  6%     +21.8%     336.50 ±  9%  interrupts.CPU12.RES:Rescheduling_interrupts
    310.33 ±  3%     +26.9%     393.83 ±  9%  interrupts.CPU120.RES:Rescheduling_interrupts
    315.17 ±  6%     +23.1%     388.00 ±  4%  interrupts.CPU121.RES:Rescheduling_interrupts
    314.33 ±  4%     +22.9%     386.33 ±  7%  interrupts.CPU122.RES:Rescheduling_interrupts
    299.00 ±  6%     +25.4%     375.00 ±  6%  interrupts.CPU123.RES:Rescheduling_interrupts
    313.00 ±  6%     +18.8%     372.00 ±  2%  interrupts.CPU124.RES:Rescheduling_interrupts
    290.83 ±  6%     +26.6%     368.33 ±  8%  interrupts.CPU125.RES:Rescheduling_interrupts
    305.17 ±  8%     +21.4%     370.33 ±  6%  interrupts.CPU126.RES:Rescheduling_interrupts
    290.67 ±  7%     +31.1%     381.17 ±  5%  interrupts.CPU127.RES:Rescheduling_interrupts
    293.17 ±  8%     +28.9%     378.00 ±  8%  interrupts.CPU128.RES:Rescheduling_interrupts
    305.50 ±  6%     +18.5%     362.17 ±  7%  interrupts.CPU129.RES:Rescheduling_interrupts
      1268 ± 20%     -62.3%     478.33 ± 16%  interrupts.CPU13.NMI:Non-maskable_interrupts
      1268 ± 20%     -62.3%     478.33 ± 16%  interrupts.CPU13.PMI:Performance_monitoring_interrupts
    289.00 ±  6%     +10.5%     319.33 ±  8%  interrupts.CPU13.RES:Rescheduling_interrupts
    298.83 ±  8%     +23.8%     370.00 ±  7%  interrupts.CPU130.RES:Rescheduling_interrupts
    295.17 ±  6%     +24.5%     367.50 ±  7%  interrupts.CPU131.RES:Rescheduling_interrupts
    328.17 ±  5%     +19.1%     391.00 ±  4%  interrupts.CPU132.RES:Rescheduling_interrupts
    307.83 ±  8%     +26.5%     389.50 ±  6%  interrupts.CPU133.RES:Rescheduling_interrupts
    312.67 ±  9%     +22.2%     382.17 ±  7%  interrupts.CPU134.RES:Rescheduling_interrupts
    304.83 ±  4%     +21.2%     369.50 ±  5%  interrupts.CPU135.RES:Rescheduling_interrupts
    299.83 ±  6%     +24.9%     374.50 ±  7%  interrupts.CPU136.RES:Rescheduling_interrupts
    290.50 ±  4%     +26.5%     367.50 ±  4%  interrupts.CPU137.RES:Rescheduling_interrupts
    313.17 ±  6%     +25.0%     391.50 ±  8%  interrupts.CPU139.RES:Rescheduling_interrupts
      1338 ±  9%     -62.2%     505.50 ± 22%  interrupts.CPU14.NMI:Non-maskable_interrupts
      1338 ±  9%     -62.2%     505.50 ± 22%  interrupts.CPU14.PMI:Performance_monitoring_interrupts
    975.67 ± 32%     -45.9%     527.50 ± 28%  interrupts.CPU140.NMI:Non-maskable_interrupts
    975.67 ± 32%     -45.9%     527.50 ± 28%  interrupts.CPU140.PMI:Performance_monitoring_interrupts
    296.50 ±  9%     +27.7%     378.67 ±  7%  interrupts.CPU140.RES:Rescheduling_interrupts
    301.17 ±  7%     +26.1%     379.67 ±  4%  interrupts.CPU141.RES:Rescheduling_interrupts
    295.83 ±  4%     +22.8%     363.17 ± 10%  interrupts.CPU142.RES:Rescheduling_interrupts
    303.00 ± 11%     +29.2%     391.50 ±  6%  interrupts.CPU143.RES:Rescheduling_interrupts
    314.33 ±  5%     +20.1%     377.50 ±  6%  interrupts.CPU144.RES:Rescheduling_interrupts
    311.83 ±  8%     +20.4%     375.50 ±  3%  interrupts.CPU145.RES:Rescheduling_interrupts
    310.00 ±  4%     +21.8%     377.67 ±  7%  interrupts.CPU146.RES:Rescheduling_interrupts
    292.33 ±  8%     +30.1%     380.33 ±  4%  interrupts.CPU147.RES:Rescheduling_interrupts
    294.50 ± 12%     +22.8%     361.67 ±  7%  interrupts.CPU148.RES:Rescheduling_interrupts
    289.50 ±  8%     +28.2%     371.00 ±  5%  interrupts.CPU149.RES:Rescheduling_interrupts
      1110 ± 21%     -56.0%     488.50 ± 45%  interrupts.CPU15.NMI:Non-maskable_interrupts
      1110 ± 21%     -56.0%     488.50 ± 45%  interrupts.CPU15.PMI:Performance_monitoring_interrupts
    306.17 ±  6%     +23.1%     376.83 ±  6%  interrupts.CPU150.RES:Rescheduling_interrupts
    310.50 ±  5%     +24.5%     386.50 ±  6%  interrupts.CPU151.RES:Rescheduling_interrupts
      1097 ± 35%     -46.5%     586.83 ± 16%  interrupts.CPU152.NMI:Non-maskable_interrupts
      1097 ± 35%     -46.5%     586.83 ± 16%  interrupts.CPU152.PMI:Performance_monitoring_interrupts
    312.50 ±  5%     +26.6%     395.50 ±  3%  interrupts.CPU152.RES:Rescheduling_interrupts
    301.17 ± 10%     +21.0%     364.50 ±  6%  interrupts.CPU153.RES:Rescheduling_interrupts
    309.67 ±  4%     +25.9%     390.00 ±  5%  interrupts.CPU154.RES:Rescheduling_interrupts
    298.67 ±  9%     +25.1%     373.50 ±  7%  interrupts.CPU155.RES:Rescheduling_interrupts
    285.00 ±  9%     +35.5%     386.17 ±  2%  interrupts.CPU156.RES:Rescheduling_interrupts
    294.33 ±  7%     +34.5%     396.00 ± 10%  interrupts.CPU157.RES:Rescheduling_interrupts
    315.83 ±  4%     +21.8%     384.83        interrupts.CPU158.RES:Rescheduling_interrupts
    292.50 ±  7%     +21.6%     355.67 ±  6%  interrupts.CPU160.RES:Rescheduling_interrupts
    304.67 ±  9%     +24.7%     379.83 ±  3%  interrupts.CPU161.RES:Rescheduling_interrupts
    278.67 ± 10%     +31.4%     366.17 ±  7%  interrupts.CPU162.RES:Rescheduling_interrupts
    312.83 ±  9%     +19.2%     373.00 ±  5%  interrupts.CPU163.RES:Rescheduling_interrupts
    297.33 ±  3%     +24.3%     369.67 ±  7%  interrupts.CPU164.RES:Rescheduling_interrupts
      1022 ± 40%     -48.1%     530.83 ± 23%  interrupts.CPU166.NMI:Non-maskable_interrupts
      1022 ± 40%     -48.1%     530.83 ± 23%  interrupts.CPU166.PMI:Performance_monitoring_interrupts
    295.67 ±  9%     +23.2%     364.33 ±  3%  interrupts.CPU166.RES:Rescheduling_interrupts
    303.67 ±  8%     +24.1%     377.00 ± 10%  interrupts.CPU167.RES:Rescheduling_interrupts
    319.00 ±  7%     +20.4%     384.17 ±  5%  interrupts.CPU168.RES:Rescheduling_interrupts
    319.00 ± 10%     +18.1%     376.83 ±  6%  interrupts.CPU169.RES:Rescheduling_interrupts
      1067 ± 19%     -58.6%     442.17 ± 29%  interrupts.CPU17.NMI:Non-maskable_interrupts
      1067 ± 19%     -58.6%     442.17 ± 29%  interrupts.CPU17.PMI:Performance_monitoring_interrupts
    293.17 ±  5%     +21.5%     356.33 ±  8%  interrupts.CPU170.RES:Rescheduling_interrupts
    301.33 ±  7%     +21.3%     365.50 ±  6%  interrupts.CPU171.RES:Rescheduling_interrupts
    962.17 ± 35%     -39.3%     584.50 ± 22%  interrupts.CPU172.NMI:Non-maskable_interrupts
    962.17 ± 35%     -39.3%     584.50 ± 22%  interrupts.CPU172.PMI:Performance_monitoring_interrupts
    292.33 ±  8%     +29.2%     377.83 ±  6%  interrupts.CPU172.RES:Rescheduling_interrupts
    303.50 ±  9%     +19.7%     363.33 ±  4%  interrupts.CPU173.RES:Rescheduling_interrupts
    286.67 ±  7%     +22.5%     351.17 ±  4%  interrupts.CPU174.RES:Rescheduling_interrupts
      1050 ± 27%     -41.7%     612.00 ± 32%  interrupts.CPU175.NMI:Non-maskable_interrupts
      1050 ± 27%     -41.7%     612.00 ± 32%  interrupts.CPU175.PMI:Performance_monitoring_interrupts
    307.17 ±  5%     +19.4%     366.67 ±  5%  interrupts.CPU175.RES:Rescheduling_interrupts
    316.00 ±  7%     +13.6%     359.00 ±  3%  interrupts.CPU176.RES:Rescheduling_interrupts
      1088 ± 27%     -40.4%     648.33 ± 22%  interrupts.CPU178.NMI:Non-maskable_interrupts
      1088 ± 27%     -40.4%     648.33 ± 22%  interrupts.CPU178.PMI:Performance_monitoring_interrupts
    280.17 ±  3%     +25.8%     352.33 ±  8%  interrupts.CPU178.RES:Rescheduling_interrupts
    303.83 ±  5%     +18.0%     358.50 ±  5%  interrupts.CPU179.RES:Rescheduling_interrupts
      1153 ± 21%     -55.6%     512.00 ± 37%  interrupts.CPU18.NMI:Non-maskable_interrupts
      1153 ± 21%     -55.6%     512.00 ± 37%  interrupts.CPU18.PMI:Performance_monitoring_interrupts
    294.33 ±  3%     +25.7%     369.83 ±  7%  interrupts.CPU183.RES:Rescheduling_interrupts
      1011 ± 42%     -46.9%     536.50 ± 17%  interrupts.CPU184.NMI:Non-maskable_interrupts
      1011 ± 42%     -46.9%     536.50 ± 17%  interrupts.CPU184.PMI:Performance_monitoring_interrupts
    293.50 ±  8%     +19.3%     350.17 ±  7%  interrupts.CPU184.RES:Rescheduling_interrupts
    303.67 ±  6%     +14.5%     347.67 ±  6%  interrupts.CPU185.RES:Rescheduling_interrupts
    322.33 ±  5%     +18.5%     382.00 ±  4%  interrupts.CPU186.RES:Rescheduling_interrupts
    304.00 ± 10%     +26.8%     385.50 ±  4%  interrupts.CPU187.RES:Rescheduling_interrupts
      1010 ± 37%     -44.2%     563.33 ± 15%  interrupts.CPU188.NMI:Non-maskable_interrupts
      1010 ± 37%     -44.2%     563.33 ± 15%  interrupts.CPU188.PMI:Performance_monitoring_interrupts
    291.00 ±  7%     +20.2%     349.83 ±  8%  interrupts.CPU188.RES:Rescheduling_interrupts
    273.50 ±  4%     +25.9%     344.33 ±  7%  interrupts.CPU189.RES:Rescheduling_interrupts
    293.83 ±  7%     +25.1%     367.50 ± 13%  interrupts.CPU190.RES:Rescheduling_interrupts
    283.17 ±  9%     +26.5%     358.33 ±  8%  interrupts.CPU191.RES:Rescheduling_interrupts
      1145 ± 35%     -53.1%     537.33 ± 47%  interrupts.CPU2.NMI:Non-maskable_interrupts
      1145 ± 35%     -53.1%     537.33 ± 47%  interrupts.CPU2.PMI:Performance_monitoring_interrupts
      1146 ± 23%     -55.1%     514.33 ± 24%  interrupts.CPU20.NMI:Non-maskable_interrupts
      1146 ± 23%     -55.1%     514.33 ± 24%  interrupts.CPU20.PMI:Performance_monitoring_interrupts
      1142 ± 17%     -54.0%     525.50 ± 35%  interrupts.CPU21.NMI:Non-maskable_interrupts
      1142 ± 17%     -54.0%     525.50 ± 35%  interrupts.CPU21.PMI:Performance_monitoring_interrupts
      1166 ± 25%     -49.0%     594.67 ± 27%  interrupts.CPU22.NMI:Non-maskable_interrupts
      1166 ± 25%     -49.0%     594.67 ± 27%  interrupts.CPU22.PMI:Performance_monitoring_interrupts
    256.17 ±  5%     +20.0%     307.33 ±  4%  interrupts.CPU22.RES:Rescheduling_interrupts
      1210 ± 24%     -55.7%     536.00 ± 45%  interrupts.CPU23.NMI:Non-maskable_interrupts
      1210 ± 24%     -55.7%     536.00 ± 45%  interrupts.CPU23.PMI:Performance_monitoring_interrupts
    267.17 ±  6%     +24.8%     333.50 ±  7%  interrupts.CPU24.RES:Rescheduling_interrupts
    277.17 ±  4%     +12.4%     311.67 ±  4%  interrupts.CPU25.RES:Rescheduling_interrupts
    255.17 ± 10%     +26.6%     323.00 ±  4%  interrupts.CPU28.RES:Rescheduling_interrupts
    244.50 ±  8%     +21.1%     296.00 ±  8%  interrupts.CPU29.RES:Rescheduling_interrupts
      1088 ± 25%     -42.7%     624.00 ± 33%  interrupts.CPU3.NMI:Non-maskable_interrupts
      1088 ± 25%     -42.7%     624.00 ± 33%  interrupts.CPU3.PMI:Performance_monitoring_interrupts
    990.17 ± 20%     -45.7%     537.50 ± 48%  interrupts.CPU30.NMI:Non-maskable_interrupts
    990.17 ± 20%     -45.7%     537.50 ± 48%  interrupts.CPU30.PMI:Performance_monitoring_interrupts
    971.67 ± 28%     -48.2%     503.67 ± 43%  interrupts.CPU31.NMI:Non-maskable_interrupts
    971.67 ± 28%     -48.2%     503.67 ± 43%  interrupts.CPU31.PMI:Performance_monitoring_interrupts
      1109 ± 24%     -61.0%     433.00 ± 38%  interrupts.CPU32.NMI:Non-maskable_interrupts
      1109 ± 24%     -61.0%     433.00 ± 38%  interrupts.CPU32.PMI:Performance_monitoring_interrupts
      1217 ± 10%     -58.3%     507.83 ± 47%  interrupts.CPU33.NMI:Non-maskable_interrupts
      1217 ± 10%     -58.3%     507.83 ± 47%  interrupts.CPU33.PMI:Performance_monitoring_interrupts
    252.83 ±  7%     +22.1%     308.83 ±  9%  interrupts.CPU33.RES:Rescheduling_interrupts
      1046 ± 15%     -42.7%     599.50 ± 32%  interrupts.CPU34.NMI:Non-maskable_interrupts
      1046 ± 15%     -42.7%     599.50 ± 32%  interrupts.CPU34.PMI:Performance_monitoring_interrupts
      1023 ± 18%     -52.4%     486.83 ± 43%  interrupts.CPU35.NMI:Non-maskable_interrupts
      1023 ± 18%     -52.4%     486.83 ± 43%  interrupts.CPU35.PMI:Performance_monitoring_interrupts
    258.50 ±  4%     +17.3%     303.17 ±  5%  interrupts.CPU35.RES:Rescheduling_interrupts
    251.33 ± 11%     +23.7%     310.83 ±  5%  interrupts.CPU36.RES:Rescheduling_interrupts
    953.17 ± 18%     -49.3%     483.67 ± 27%  interrupts.CPU39.NMI:Non-maskable_interrupts
    953.17 ± 18%     -49.3%     483.67 ± 27%  interrupts.CPU39.PMI:Performance_monitoring_interrupts
      1202 ±  8%     -56.8%     519.00 ± 25%  interrupts.CPU4.NMI:Non-maskable_interrupts
      1202 ±  8%     -56.8%     519.00 ± 25%  interrupts.CPU4.PMI:Performance_monitoring_interrupts
    860.33 ± 35%     -46.9%     456.50 ± 37%  interrupts.CPU41.NMI:Non-maskable_interrupts
    860.33 ± 35%     -46.9%     456.50 ± 37%  interrupts.CPU41.PMI:Performance_monitoring_interrupts
    268.67 ±  9%     +19.3%     320.50 ±  9%  interrupts.CPU42.RES:Rescheduling_interrupts
    978.67 ± 23%     -47.3%     515.67 ± 40%  interrupts.CPU43.NMI:Non-maskable_interrupts
    978.67 ± 23%     -47.3%     515.67 ± 40%  interrupts.CPU43.PMI:Performance_monitoring_interrupts
    248.67 ±  7%     +20.4%     299.33 ±  6%  interrupts.CPU43.RES:Rescheduling_interrupts
    921.17 ± 29%     -50.5%     456.00 ± 35%  interrupts.CPU44.NMI:Non-maskable_interrupts
    921.17 ± 29%     -50.5%     456.00 ± 35%  interrupts.CPU44.PMI:Performance_monitoring_interrupts
    937.33 ± 28%     -44.4%     521.17 ± 37%  interrupts.CPU45.NMI:Non-maskable_interrupts
    937.33 ± 28%     -44.4%     521.17 ± 37%  interrupts.CPU45.PMI:Performance_monitoring_interrupts
    260.50 ±  4%     +15.8%     301.67 ±  7%  interrupts.CPU45.RES:Rescheduling_interrupts
    253.17 ±  9%     +17.9%     298.50 ±  6%  interrupts.CPU46.RES:Rescheduling_interrupts
    993.00 ± 27%     -47.8%     518.67 ± 19%  interrupts.CPU48.NMI:Non-maskable_interrupts
    993.00 ± 27%     -47.8%     518.67 ± 19%  interrupts.CPU48.PMI:Performance_monitoring_interrupts
    300.17 ±  6%     +12.3%     337.17 ±  3%  interrupts.CPU48.RES:Rescheduling_interrupts
    272.67 ±  5%     +21.5%     331.33 ±  6%  interrupts.CPU49.RES:Rescheduling_interrupts
      1302 ± 15%     -57.8%     550.17 ± 54%  interrupts.CPU5.NMI:Non-maskable_interrupts
      1302 ± 15%     -57.8%     550.17 ± 54%  interrupts.CPU5.PMI:Performance_monitoring_interrupts
      1154 ± 11%     -45.1%     634.00 ± 38%  interrupts.CPU50.NMI:Non-maskable_interrupts
      1154 ± 11%     -45.1%     634.00 ± 38%  interrupts.CPU50.PMI:Performance_monitoring_interrupts
      1280 ± 17%     -53.0%     602.33 ± 32%  interrupts.CPU51.NMI:Non-maskable_interrupts
      1280 ± 17%     -53.0%     602.33 ± 32%  interrupts.CPU51.PMI:Performance_monitoring_interrupts
      1248 ± 12%     -51.6%     603.67 ± 33%  interrupts.CPU52.NMI:Non-maskable_interrupts
      1248 ± 12%     -51.6%     603.67 ± 33%  interrupts.CPU52.PMI:Performance_monitoring_interrupts
    254.00 ±  5%     +18.0%     299.83 ±  9%  interrupts.CPU52.RES:Rescheduling_interrupts
      1288 ± 19%     -60.9%     503.17 ± 31%  interrupts.CPU53.NMI:Non-maskable_interrupts
      1288 ± 19%     -60.9%     503.17 ± 31%  interrupts.CPU53.PMI:Performance_monitoring_interrupts
    247.17 ±  6%     +16.5%     288.00 ±  5%  interrupts.CPU53.RES:Rescheduling_interrupts
      1201 ±  6%     -44.0%     672.83 ± 25%  interrupts.CPU54.NMI:Non-maskable_interrupts
      1201 ±  6%     -44.0%     672.83 ± 25%  interrupts.CPU54.PMI:Performance_monitoring_interrupts
      1200 ± 20%     -53.3%     560.50 ± 49%  interrupts.CPU55.NMI:Non-maskable_interrupts
      1200 ± 20%     -53.3%     560.50 ± 49%  interrupts.CPU55.PMI:Performance_monitoring_interrupts
    254.50 ±  8%     +16.0%     295.17 ± 11%  interrupts.CPU55.RES:Rescheduling_interrupts
      1116 ± 26%     -61.0%     435.67 ± 30%  interrupts.CPU56.NMI:Non-maskable_interrupts
      1116 ± 26%     -61.0%     435.67 ± 30%  interrupts.CPU56.PMI:Performance_monitoring_interrupts
    264.33 ±  6%     +15.1%     304.17 ±  3%  interrupts.CPU56.RES:Rescheduling_interrupts
    249.67 ±  9%     +17.8%     294.17 ±  6%  interrupts.CPU57.RES:Rescheduling_interrupts
     89.00 ± 14%     -19.5%      71.67 ± 13%  interrupts.CPU58.TLB:TLB_shootdowns
      1059 ±  5%     -56.8%     457.67 ± 43%  interrupts.CPU59.NMI:Non-maskable_interrupts
      1059 ±  5%     -56.8%     457.67 ± 43%  interrupts.CPU59.PMI:Performance_monitoring_interrupts
      1659 ± 18%     -25.2%       1242 ± 12%  interrupts.CPU6.CAL:Function_call_interrupts
      1172 ±  3%     -53.4%     546.33 ± 27%  interrupts.CPU6.NMI:Non-maskable_interrupts
      1172 ±  3%     -53.4%     546.33 ± 27%  interrupts.CPU6.PMI:Performance_monitoring_interrupts
      1107 ± 17%     -46.9%     588.00 ± 31%  interrupts.CPU60.NMI:Non-maskable_interrupts
      1107 ± 17%     -46.9%     588.00 ± 31%  interrupts.CPU60.PMI:Performance_monitoring_interrupts
    238.33 ±  6%     +22.7%     292.50 ±  9%  interrupts.CPU60.RES:Rescheduling_interrupts
      1075 ± 19%     -49.3%     545.00 ± 10%  interrupts.CPU61.NMI:Non-maskable_interrupts
      1075 ± 19%     -49.3%     545.00 ± 10%  interrupts.CPU61.PMI:Performance_monitoring_interrupts
    257.83 ±  7%     +21.4%     313.00 ± 10%  interrupts.CPU61.RES:Rescheduling_interrupts
      1148 ± 14%     -58.6%     475.33 ± 47%  interrupts.CPU62.NMI:Non-maskable_interrupts
      1148 ± 14%     -58.6%     475.33 ± 47%  interrupts.CPU62.PMI:Performance_monitoring_interrupts
      1113 ± 16%     -51.4%     541.00 ± 62%  interrupts.CPU63.NMI:Non-maskable_interrupts
      1113 ± 16%     -51.4%     541.00 ± 62%  interrupts.CPU63.PMI:Performance_monitoring_interrupts
      1061 ± 15%     -48.1%     551.00 ± 35%  interrupts.CPU64.NMI:Non-maskable_interrupts
      1061 ± 15%     -48.1%     551.00 ± 35%  interrupts.CPU64.PMI:Performance_monitoring_interrupts
    252.50 ±  5%     +19.5%     301.83 ±  5%  interrupts.CPU64.RES:Rescheduling_interrupts
    257.83 ±  9%     +14.5%     295.17 ±  4%  interrupts.CPU65.RES:Rescheduling_interrupts
    250.00 ±  5%     +19.5%     298.83 ±  5%  interrupts.CPU66.RES:Rescheduling_interrupts
      1171 ± 24%     -51.7%     565.67 ± 35%  interrupts.CPU67.NMI:Non-maskable_interrupts
      1171 ± 24%     -51.7%     565.67 ± 35%  interrupts.CPU67.PMI:Performance_monitoring_interrupts
    238.83 ±  7%     +25.5%     299.83 ±  4%  interrupts.CPU67.RES:Rescheduling_interrupts
      1134 ± 19%     -52.2%     542.50 ± 33%  interrupts.CPU68.NMI:Non-maskable_interrupts
      1134 ± 19%     -52.2%     542.50 ± 33%  interrupts.CPU68.PMI:Performance_monitoring_interrupts
      1180 ± 11%     -56.5%     513.67 ± 26%  interrupts.CPU69.NMI:Non-maskable_interrupts
      1180 ± 11%     -56.5%     513.67 ± 26%  interrupts.CPU69.PMI:Performance_monitoring_interrupts
    235.67 ±  9%     +29.0%     304.00 ±  8%  interrupts.CPU69.RES:Rescheduling_interrupts
      1224 ± 21%     -63.3%     449.17 ± 23%  interrupts.CPU7.NMI:Non-maskable_interrupts
      1224 ± 21%     -63.3%     449.17 ± 23%  interrupts.CPU7.PMI:Performance_monitoring_interrupts
    258.17 ±  6%     +29.7%     334.83 ± 12%  interrupts.CPU7.RES:Rescheduling_interrupts
      1201 ± 14%     -52.6%     569.67 ± 25%  interrupts.CPU70.NMI:Non-maskable_interrupts
      1201 ± 14%     -52.6%     569.67 ± 25%  interrupts.CPU70.PMI:Performance_monitoring_interrupts
    252.83 ±  5%     +19.4%     302.00 ±  7%  interrupts.CPU70.RES:Rescheduling_interrupts
      1273 ± 20%     -60.5%     503.17 ± 35%  interrupts.CPU71.NMI:Non-maskable_interrupts
      1273 ± 20%     -60.5%     503.17 ± 35%  interrupts.CPU71.PMI:Performance_monitoring_interrupts
      1347 ± 13%     -57.2%     577.00 ± 33%  interrupts.CPU72.NMI:Non-maskable_interrupts
      1347 ± 13%     -57.2%     577.00 ± 33%  interrupts.CPU72.PMI:Performance_monitoring_interrupts
    283.00 ±  6%     +20.7%     341.67 ± 12%  interrupts.CPU72.RES:Rescheduling_interrupts
    276.83 ±  3%     +14.2%     316.17 ±  5%  interrupts.CPU73.RES:Rescheduling_interrupts
      1197 ± 18%     -55.4%     534.33 ± 25%  interrupts.CPU74.NMI:Non-maskable_interrupts
      1197 ± 18%     -55.4%     534.33 ± 25%  interrupts.CPU74.PMI:Performance_monitoring_interrupts
    279.33 ±  5%     +13.2%     316.17 ±  5%  interrupts.CPU74.RES:Rescheduling_interrupts
      1075 ± 26%     -48.5%     553.67 ± 53%  interrupts.CPU76.NMI:Non-maskable_interrupts
      1075 ± 26%     -48.5%     553.67 ± 53%  interrupts.CPU76.PMI:Performance_monitoring_interrupts
    997.17 ± 23%     -48.8%     510.67 ± 42%  interrupts.CPU77.NMI:Non-maskable_interrupts
    997.17 ± 23%     -48.8%     510.67 ± 42%  interrupts.CPU77.PMI:Performance_monitoring_interrupts
      1259 ± 10%     -50.8%     620.00 ± 36%  interrupts.CPU78.NMI:Non-maskable_interrupts
      1259 ± 10%     -50.8%     620.00 ± 36%  interrupts.CPU78.PMI:Performance_monitoring_interrupts
    243.50 ±  6%     +19.9%     292.00 ±  8%  interrupts.CPU78.RES:Rescheduling_interrupts
      1204 ± 13%     -59.0%     494.50 ± 33%  interrupts.CPU79.NMI:Non-maskable_interrupts
      1204 ± 13%     -59.0%     494.50 ± 33%  interrupts.CPU79.PMI:Performance_monitoring_interrupts
    253.33 ±  6%     +17.1%     296.67 ±  6%  interrupts.CPU79.RES:Rescheduling_interrupts
      1211 ± 19%     -50.9%     595.33 ± 34%  interrupts.CPU8.NMI:Non-maskable_interrupts
      1211 ± 19%     -50.9%     595.33 ± 34%  interrupts.CPU8.PMI:Performance_monitoring_interrupts
    273.17 ±  4%     +17.6%     321.33 ±  5%  interrupts.CPU8.RES:Rescheduling_interrupts
      1152 ± 19%     -56.4%     501.83 ± 39%  interrupts.CPU80.NMI:Non-maskable_interrupts
      1152 ± 19%     -56.4%     501.83 ± 39%  interrupts.CPU80.PMI:Performance_monitoring_interrupts
    261.67 ±  6%     +12.1%     293.33 ±  5%  interrupts.CPU80.RES:Rescheduling_interrupts
      1254 ±  7%     -58.2%     524.33 ± 37%  interrupts.CPU81.NMI:Non-maskable_interrupts
      1254 ±  7%     -58.2%     524.33 ± 37%  interrupts.CPU81.PMI:Performance_monitoring_interrupts
      1302 ±  9%     -64.4%     463.83 ± 24%  interrupts.CPU82.NMI:Non-maskable_interrupts
      1302 ±  9%     -64.4%     463.83 ± 24%  interrupts.CPU82.PMI:Performance_monitoring_interrupts
    246.00 ±  5%     +16.5%     286.50 ±  7%  interrupts.CPU82.RES:Rescheduling_interrupts
      1233 ± 15%     -53.8%     569.83 ± 34%  interrupts.CPU83.NMI:Non-maskable_interrupts
      1233 ± 15%     -53.8%     569.83 ± 34%  interrupts.CPU83.PMI:Performance_monitoring_interrupts
      1246 ± 22%     -58.7%     514.83 ± 61%  interrupts.CPU84.NMI:Non-maskable_interrupts
      1246 ± 22%     -58.7%     514.83 ± 61%  interrupts.CPU84.PMI:Performance_monitoring_interrupts
      1263 ± 21%     -53.2%     591.00 ± 27%  interrupts.CPU85.NMI:Non-maskable_interrupts
      1263 ± 21%     -53.2%     591.00 ± 27%  interrupts.CPU85.PMI:Performance_monitoring_interrupts
      1297 ± 15%     -63.6%     471.83 ± 37%  interrupts.CPU86.NMI:Non-maskable_interrupts
      1297 ± 15%     -63.6%     471.83 ± 37%  interrupts.CPU86.PMI:Performance_monitoring_interrupts
      1262 ± 12%     -63.1%     465.83 ± 40%  interrupts.CPU87.NMI:Non-maskable_interrupts
      1262 ± 12%     -63.1%     465.83 ± 40%  interrupts.CPU87.PMI:Performance_monitoring_interrupts
      1251 ± 14%     -57.0%     537.33 ± 35%  interrupts.CPU88.NMI:Non-maskable_interrupts
      1251 ± 14%     -57.0%     537.33 ± 35%  interrupts.CPU88.PMI:Performance_monitoring_interrupts
      1152 ± 17%     -54.5%     524.67 ± 44%  interrupts.CPU89.NMI:Non-maskable_interrupts
      1152 ± 17%     -54.5%     524.67 ± 44%  interrupts.CPU89.PMI:Performance_monitoring_interrupts
      1335 ± 15%     -57.7%     564.83 ± 30%  interrupts.CPU9.NMI:Non-maskable_interrupts
      1335 ± 15%     -57.7%     564.83 ± 30%  interrupts.CPU9.PMI:Performance_monitoring_interrupts
      1241 ± 19%     -51.2%     605.17 ± 46%  interrupts.CPU90.NMI:Non-maskable_interrupts
      1241 ± 19%     -51.2%     605.17 ± 46%  interrupts.CPU90.PMI:Performance_monitoring_interrupts
      1226 ± 21%     -56.2%     537.33 ± 39%  interrupts.CPU91.NMI:Non-maskable_interrupts
      1226 ± 21%     -56.2%     537.33 ± 39%  interrupts.CPU91.PMI:Performance_monitoring_interrupts
    245.33 ±  7%     +20.9%     296.50 ±  7%  interrupts.CPU91.RES:Rescheduling_interrupts
      1188 ± 11%     -50.0%     594.33 ± 37%  interrupts.CPU93.NMI:Non-maskable_interrupts
      1188 ± 11%     -50.0%     594.33 ± 37%  interrupts.CPU93.PMI:Performance_monitoring_interrupts
      1177 ± 17%     -45.6%     640.17 ± 33%  interrupts.CPU94.NMI:Non-maskable_interrupts
      1177 ± 17%     -45.6%     640.17 ± 33%  interrupts.CPU94.PMI:Performance_monitoring_interrupts
    236.50 ± 11%     +20.7%     285.50 ±  7%  interrupts.CPU94.RES:Rescheduling_interrupts
    241.67 ±  3%     +37.2%     331.67 ± 18%  interrupts.CPU95.RES:Rescheduling_interrupts
      1223 ± 25%     -60.3%     486.17 ± 26%  interrupts.CPU96.NMI:Non-maskable_interrupts
      1223 ± 25%     -60.3%     486.17 ± 26%  interrupts.CPU96.PMI:Performance_monitoring_interrupts
    313.00 ±  6%     +28.7%     402.83 ±  7%  interrupts.CPU96.RES:Rescheduling_interrupts
      1155 ± 20%     -53.7%     535.17 ± 53%  interrupts.CPU97.NMI:Non-maskable_interrupts
      1155 ± 20%     -53.7%     535.17 ± 53%  interrupts.CPU97.PMI:Performance_monitoring_interrupts
    313.50 ± 11%     +22.0%     382.50 ±  7%  interrupts.CPU97.RES:Rescheduling_interrupts
    299.17 ±  4%     +27.4%     381.00 ±  9%  interrupts.CPU98.RES:Rescheduling_interrupts
      1190 ± 19%     -61.6%     457.17 ± 32%  interrupts.CPU99.NMI:Non-maskable_interrupts
      1190 ± 19%     -61.6%     457.17 ± 32%  interrupts.CPU99.PMI:Performance_monitoring_interrupts
    303.50 ±  8%     +28.9%     391.33 ±  9%  interrupts.CPU99.RES:Rescheduling_interrupts
    204424 ±  8%     -45.2%     112016 ±  9%  interrupts.NMI:Non-maskable_interrupts
    204424 ±  8%     -45.2%     112016 ±  9%  interrupts.PMI:Performance_monitoring_interrupts
     54352           +20.3%      65372 ±  3%  interrupts.RES:Rescheduling_interrupts





Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


---
0DAY/LKP+ Test Infrastructure                   Open Source Technology Center
https://lists.01.org/hyperkitty/list/lkp@lists.01.org       Intel Corporation

Thanks,
Oliver Sang


View attachment "config-5.12.0-11208-g2d146aa3aa84" of type "text/plain" (173849 bytes)

View attachment "job-script" of type "text/plain" (8021 bytes)

View attachment "job.yaml" of type "text/plain" (5491 bytes)

View attachment "reproduce" of type "text/plain" (45068 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ