lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite for Android: free password hash cracker in your pocket
[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Date:   Thu, 17 Sep 2020 09:29:23 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     Alex Kogan <alex.kogan@...cle.com>
Cc:     linux@...linux.org.uk, peterz@...radead.org, mingo@...hat.com,
        will.deacon@....com, arnd@...db.de, longman@...hat.com,
        linux-arch@...r.kernel.org, linux-arm-kernel@...ts.infradead.org,
        linux-kernel@...r.kernel.org, tglx@...utronix.de, bp@...en8.de,
        hpa@...or.com, x86@...nel.org, guohanjun@...wei.com,
        jglauber@...vell.com, steven.sistare@...cle.com,
        daniel.m.jordan@...cle.com, alex.kogan@...cle.com,
        dave.dice@...cle.com, 0day robot <lkp@...el.com>, lkp@...ts.01.org,
        ying.huang@...el.com, feng.tang@...el.com, zhengjun.xing@...el.com
Subject: [locking/qspinlock] 5df52a74ab: reaim.jobs_per_min 45.7% improvement

Greeting,

FYI, we noticed a 45.7% improvement of reaim.jobs_per_min due to commit:


commit: 5df52a74ab00917ec2522ab3ec2e91ce33fbf22a ("[PATCH v11 3/5] locking/qspinlock: Introduce CNA into the slow path of qspinlock")
url: https://github.com/0day-ci/linux/commits/Alex-Kogan/Add-NUMA-awareness-to-qspinlock/20200916-023855
base: https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git 1909760f5fc3f123e47b4e24e0ccdc0fc8f3f106

in testcase: reaim
on test machine: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 192G memory
with following parameters:

	runtime: 300s
	nr_task: 100%
	test: fserver
	cpufreq_governor: performance
	ucode: 0x5002f01

test-description: REAIM is an updated and improved version of AIM 7 benchmark.
test-url: https://sourceforge.net/projects/re-aim-7/





Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
compiler/cpufreq_governor/kconfig/nr_task/rootfs/runtime/tbox_group/test/testcase/ucode:
  gcc-9/performance/x86_64-rhel-8.3/100%/debian-10.4-x86_64-20200603.cgz/300s/lkp-csl-2sp5/fserver/reaim/0x5002f01

commit: 
  d0615c8da7 ("locking/qspinlock: Refactor the qspinlock slow path")
  5df52a74ab ("locking/qspinlock: Introduce CNA into the slow path of qspinlock")

d0615c8da74dd1dc 5df52a74ab00917ec2522ab3ec2 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          0:4            8%           0:4     perf-profile.children.cycles-pp.error_entry
         %stddev     %change         %stddev
             \          |                \  
     91.05 ±  2%     -56.1%      39.99        reaim.child_systime
     28.81            +3.3%      29.76        reaim.child_utime
    415207           +45.7%     604838        reaim.jobs_per_min
      4325           +45.7%       6300        reaim.jobs_per_min_child
     94.33            -5.5%      89.14        reaim.jti
    429501           +50.1%     644664        reaim.max_jobs_per_min
      1.40           -31.3%       0.96        reaim.parent_time
      5.17          +100.0%      10.34        reaim.std_dev_percent
      0.07           +23.8%       0.08        reaim.std_dev_time
    372486            -6.7%     347381        reaim.time.involuntary_context_switches
     51745 ±  7%    +258.5%     185528        reaim.time.major_page_faults
  84560705           +14.4%   96728177        reaim.time.minor_page_faults
      3497           -33.3%       2332        reaim.time.percent_of_cpu_this_job_got
      8036           -49.7%       4041        reaim.time.system_time
      2543           +18.2%       3006        reaim.time.user_time
   1855222           +27.1%    2357156        reaim.time.voluntary_context_switches
    847200           +14.4%     969600        reaim.workload
      6579           +18.2%       7775        meminfo.PageTables
      1759 ± 15%     +36.2%       2396 ± 10%  numa-meminfo.node0.Active
      1758 ± 15%     +36.2%       2394 ± 10%  numa-meminfo.node0.Active(anon)
   8809518 ±  3%    +268.2%   32435557 ± 13%  cpuidle.C1.time
    179248 ±  2%    +166.5%     477615 ± 20%  cpuidle.C1.usage
    323189 ±  8%    +447.3%    1768876 ± 33%  cpuidle.POLL.time
     67710 ±  2%    +437.6%     364029 ± 38%  cpuidle.POLL.usage
      1617 ±  5%     -10.7%       1444 ±  4%  slabinfo.kmalloc-rcl-512.active_objs
      1617 ±  5%     -10.7%       1444 ±  4%  slabinfo.kmalloc-rcl-512.num_objs
    778.00           +36.1%       1058        slabinfo.names_cache.active_objs
    778.00           +36.2%       1059        slabinfo.names_cache.num_objs
     61.50           +17.9%      72.50        vmstat.cpu.id
      8.00           +25.0%      10.00        vmstat.cpu.us
     36.25           -20.7%      28.75 ±  8%  vmstat.procs.r
     16803           +24.3%      20878        vmstat.system.cs
  30562045           +15.0%   35160790 ±  3%  numa-numastat.node0.local_node
  30588448           +15.0%   35168622 ±  3%  numa-numastat.node0.numa_hit
     26405 ± 30%     -70.3%       7850 ±170%  numa-numastat.node0.other_node
  30667122           +14.3%   35063923 ±  3%  numa-numastat.node1.local_node
  30671972           +14.4%   35087297 ±  3%  numa-numastat.node1.numa_hit
     61.77           +11.0       72.81        mpstat.cpu.all.idle%
      0.02 ± 27%      +0.2        0.21 ±  2%  mpstat.cpu.all.iowait%
      1.34            +0.5        1.82 ± 20%  mpstat.cpu.all.irq%
      0.39 ± 22%      +0.3        0.74 ± 15%  mpstat.cpu.all.soft%
     28.00           -13.7       14.33        mpstat.cpu.all.sys%
      8.49            +1.6       10.10        mpstat.cpu.all.usr%
    436.25 ± 15%     +36.8%     596.75 ±  9%  numa-vmstat.node0.nr_active_anon
    436.25 ± 15%     +36.8%     596.75 ±  9%  numa-vmstat.node0.nr_zone_active_anon
  16036199           +12.9%   18112792 ±  3%  numa-vmstat.node0.numa_hit
  15949972           +13.3%   18074885 ±  3%  numa-vmstat.node0.numa_local
      3770 ± 16%     +20.6%       4545 ± 12%  numa-vmstat.node1.nr_mapped
  15790592           +15.3%   18209998 ±  3%  numa-vmstat.node1.numa_hit
  15696794           +15.1%   18067666 ±  3%  numa-vmstat.node1.numa_local
      1702           +15.8%       1971        proc-vmstat.nr_page_table_pages
     68627            +2.7%      70457        proc-vmstat.nr_slab_unreclaimable
  61208825           +14.8%   70253284        proc-vmstat.numa_hit
  61177558           +14.8%   70222047        proc-vmstat.numa_local
    181604           +12.9%     205093        proc-vmstat.pgactivate
  67358848           +14.4%   77055734        proc-vmstat.pgalloc_normal
  85598468           +14.5%   98050870        proc-vmstat.pgfault
  67307484           +14.4%   77004440        proc-vmstat.pgfree
     20606 ±105%     -99.9%      23.66 ±112%  sched_debug.cfs_rq:/.MIN_vruntime.avg
     52144 ±  8%     -32.2%      35333 ±  8%  sched_debug.cfs_rq:/.exec_clock.avg
     54107 ±  8%     -31.0%      37345 ±  6%  sched_debug.cfs_rq:/.exec_clock.max
     51513 ±  8%     -32.7%      34665 ±  8%  sched_debug.cfs_rq:/.exec_clock.min
      1.52 ± 16%    -100.0%       0.00        sched_debug.cfs_rq:/.load_avg.min
     20606 ±105%     -99.9%      23.66 ±112%  sched_debug.cfs_rq:/.max_vruntime.avg
   4723149 ±  8%     -39.3%    2868680 ±  8%  sched_debug.cfs_rq:/.min_vruntime.avg
   4788797 ±  8%     -38.7%    2937893 ±  8%  sched_debug.cfs_rq:/.min_vruntime.max
   4640255 ±  8%     -39.7%    2799231 ±  9%  sched_debug.cfs_rq:/.min_vruntime.min
      0.38 ± 13%     -83.4%       0.06 ± 21%  sched_debug.cfs_rq:/.nr_running.avg
     51.49 ± 23%     +38.2%      71.18 ± 14%  sched_debug.cfs_rq:/.nr_spread_over.avg
     71.58 ± 20%    +106.7%     147.99 ±  9%  sched_debug.cfs_rq:/.nr_spread_over.max
     33.95 ± 31%     +42.0%      48.21 ± 15%  sched_debug.cfs_rq:/.nr_spread_over.min
      7.77 ± 10%     +92.9%      14.99 ±  8%  sched_debug.cfs_rq:/.nr_spread_over.stddev
    380.26 ± 12%     -80.8%      72.98 ± 13%  sched_debug.cfs_rq:/.runnable_avg.avg
      1019 ± 10%     -32.6%     687.18 ±  3%  sched_debug.cfs_rq:/.runnable_avg.max
    112.78 ± 30%    -100.0%       0.00        sched_debug.cfs_rq:/.runnable_avg.min
    178.28 ± 13%     -30.8%     123.46 ±  6%  sched_debug.cfs_rq:/.runnable_avg.stddev
    375.51 ± 12%     -80.6%      72.92 ± 13%  sched_debug.cfs_rq:/.util_avg.avg
    981.43 ± 11%     -30.0%     687.05 ±  3%  sched_debug.cfs_rq:/.util_avg.max
    108.93 ± 28%    -100.0%       0.00        sched_debug.cfs_rq:/.util_avg.min
    176.90 ± 13%     -30.2%     123.46 ±  6%  sched_debug.cfs_rq:/.util_avg.stddev
    195.49 ± 21%     -96.0%       7.92 ± 20%  sched_debug.cfs_rq:/.util_est_enqueued.avg
    624.00 ± 12%     -69.0%     193.50 ±  7%  sched_debug.cfs_rq:/.util_est_enqueued.max
    131.16 ± 10%     -76.3%      31.03 ±  9%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
    728287 ±  3%     +19.3%     868646        sched_debug.cpu.avg_idle.avg
    132525 ± 37%     -39.3%      80463 ± 23%  sched_debug.cpu.avg_idle.min
     23454 ± 16%     -98.0%     458.25 ±  6%  sched_debug.cpu.curr->pid.avg
     64630 ±  4%     -40.8%      38276 ±  6%  sched_debug.cpu.curr->pid.max
      8848 ± 13%     -55.1%       3975 ±  6%  sched_debug.cpu.curr->pid.stddev
      0.00 ±  3%     -42.4%       0.00 ±  3%  sched_debug.cpu.next_balance.stddev
      0.36 ± 14%     -89.0%       0.04 ±  6%  sched_debug.cpu.nr_running.avg
      1.52 ±  2%     -28.4%       1.09 ±  8%  sched_debug.cpu.nr_running.max
      0.23 ±  7%     -22.4%       0.18 ±  2%  sched_debug.cpu.nr_running.stddev
     22675 ±  8%     +28.8%      29201 ±  8%  sched_debug.cpu.nr_switches.min
     51.03 ± 18%    +180.8%     143.32 ± 12%  sched_debug.cpu.nr_uninterruptible.max
    -37.99          +247.4%    -131.97        sched_debug.cpu.nr_uninterruptible.min
     15.56 ±  6%    +224.0%      50.43 ±  4%  sched_debug.cpu.nr_uninterruptible.stddev
     22023 ±  8%     +30.4%      28725 ±  9%  sched_debug.cpu.sched_count.min
     10166 ±  9%     +30.7%      13282 ±  9%  sched_debug.cpu.sched_goidle.avg
      8656 ±  8%     +38.7%      12009 ±  9%  sched_debug.cpu.sched_goidle.min
      8404 ±  9%     +30.9%      11002 ±  9%  sched_debug.cpu.ttwu_count.avg
      7210 ±  9%     +36.7%       9855 ±  9%  sched_debug.cpu.ttwu_count.min
      7.16          +213.6%      22.44 ± 64%  perf-stat.i.MPKI
  7.47e+09            +2.2%  7.636e+09        perf-stat.i.branch-instructions
      1.13            +1.6        2.70 ± 52%  perf-stat.i.branch-miss-rate%
  63150440           +21.4%   76644167 ±  5%  perf-stat.i.branch-misses
     22.68            -9.8       12.92 ± 39%  perf-stat.i.cache-miss-rate%
  36603035           -35.6%   23563190 ±  4%  perf-stat.i.cache-misses
 1.518e+08           +32.6%  2.013e+08 ± 10%  perf-stat.i.cache-references
     16774           +23.8%      20758        perf-stat.i.context-switches
 1.013e+11           -29.8%  7.113e+10        perf-stat.i.cpu-cycles
      2475           +39.6%       3455        perf-stat.i.cpu-migrations
      0.02 ±  2%      +0.2        0.24 ± 60%  perf-stat.i.dTLB-load-miss-rate%
   1529728 ±  2%     +87.5%    2868746 ± 21%  perf-stat.i.dTLB-load-misses
    742222           +32.9%     986190 ±  8%  perf-stat.i.dTLB-store-misses
 1.981e+09           +13.9%  2.256e+09        perf-stat.i.dTLB-stores
   5778695            +8.2%    6251785 ±  6%  perf-stat.i.iTLB-load-misses
 3.807e+10            +4.9%  3.993e+10        perf-stat.i.instructions
      4905           -19.6%       3946 ±  7%  perf-stat.i.instructions-per-iTLB-miss
      0.40           +11.2%       0.44 ±  5%  perf-stat.i.ipc
    169.87 ±  6%    +256.9%     606.26        perf-stat.i.major-faults
      1.05           -30.1%       0.74        perf-stat.i.metric.GHz
      0.85 ±  5%     +31.9%       1.13 ±  2%  perf-stat.i.metric.K/sec
    164.77            +2.9%     169.61        perf-stat.i.metric.M/sec
    273923           +14.1%     312525        perf-stat.i.minor-faults
     85.23            -7.3       77.88        perf-stat.i.node-load-miss-rate%
   8000980           -35.9%    5131408 ±  2%  perf-stat.i.node-load-misses
   1288912           +11.1%    1432395        perf-stat.i.node-loads
     78.84           -12.5       66.32 ±  5%  perf-stat.i.node-store-miss-rate%
   4499737           -40.8%    2664950        perf-stat.i.node-store-misses
    916780           +30.9%    1200392        perf-stat.i.node-stores
    274093           +14.2%     313131        perf-stat.i.page-faults
      3.99           +26.3%       5.04 ± 10%  perf-stat.overall.MPKI
      0.85            +0.2        1.00 ±  5%  perf-stat.overall.branch-miss-rate%
     24.10           -12.2       11.90 ± 12%  perf-stat.overall.cache-miss-rate%
      2.66           -33.2%       1.77        perf-stat.overall.cpi
      2764 ±  2%      +8.4%       2997 ±  3%  perf-stat.overall.cycles-between-cache-misses
      0.02 ±  2%      +0.0        0.05 ± 21%  perf-stat.overall.dTLB-load-miss-rate%
      0.04            +0.0        0.04 ±  8%  perf-stat.overall.dTLB-store-miss-rate%
      0.38           +49.8%       0.56        perf-stat.overall.ipc
     86.09            -7.9       78.18        perf-stat.overall.node-load-miss-rate%
     83.07           -14.0       69.06        perf-stat.overall.node-store-miss-rate%
  13652177            -8.4%   12500244        perf-stat.overall.path-length
 7.492e+09            +2.3%  7.661e+09        perf-stat.ps.branch-instructions
  63388085           +21.5%   76989977 ±  5%  perf-stat.ps.branch-misses
  36699307           -35.3%   23728211 ±  4%  perf-stat.ps.cache-misses
 1.523e+08           +32.6%  2.018e+08 ± 10%  perf-stat.ps.cache-references
     16842           +24.0%      20893        perf-stat.ps.context-switches
 1.014e+11           -30.0%  7.102e+10        perf-stat.ps.cpu-cycles
      2486           +39.9%       3477        perf-stat.ps.cpu-migrations
   1536862 ±  2%     +86.8%    2871096 ± 21%  perf-stat.ps.dTLB-load-misses
    745080           +32.8%     989404 ±  8%  perf-stat.ps.dTLB-store-misses
 1.988e+09           +14.0%  2.267e+09        perf-stat.ps.dTLB-stores
   5782684            +8.2%    6259566 ±  5%  perf-stat.ps.iTLB-load-misses
 3.816e+10            +4.9%  4.003e+10        perf-stat.ps.instructions
    170.83 ±  7%    +258.5%     612.48        perf-stat.ps.major-faults
    275009           +14.3%     314248        perf-stat.ps.minor-faults
   8024413           -35.5%    5176284 ±  2%  perf-stat.ps.node-load-misses
   1296055           +11.5%    1444510        perf-stat.ps.node-loads
   4513106           -40.4%    2688851        perf-stat.ps.node-store-misses
    919879           +30.9%    1204492        perf-stat.ps.node-stores
    275180           +14.4%     314861        perf-stat.ps.page-faults
 1.157e+13            +4.8%  1.212e+13        perf-stat.total.instructions
     31336 ±  2%     +17.8%      36904 ±  3%  softirqs.CPU0.SCHED
     29353 ±  2%     +23.5%      36240 ±  3%  softirqs.CPU1.SCHED
     43857 ±  3%     +17.2%      51394 ±  5%  softirqs.CPU10.NET_RX
     28679           +21.6%      34866 ±  2%  softirqs.CPU10.SCHED
     44878 ±  4%     +26.2%      56619 ± 16%  softirqs.CPU11.NET_RX
     28839           +24.3%      35853 ±  3%  softirqs.CPU11.SCHED
     43229 ±  4%     +18.9%      51387 ±  6%  softirqs.CPU12.NET_RX
     28677           +22.7%      35178 ±  3%  softirqs.CPU12.SCHED
     45288           +10.5%      50052 ±  2%  softirqs.CPU13.NET_RX
     28682           +21.6%      34879 ±  2%  softirqs.CPU13.SCHED
     42141 ±  3%     +26.9%      53476 ±  4%  softirqs.CPU14.NET_RX
     28729           +22.1%      35080 ±  2%  softirqs.CPU14.SCHED
     44355 ±  3%     +10.8%      49131 ±  5%  softirqs.CPU15.NET_RX
     28634           +22.0%      34933 ±  3%  softirqs.CPU15.SCHED
     44104 ±  5%     +16.3%      51298 ±  6%  softirqs.CPU16.NET_RX
     28485           +22.3%      34825 ±  2%  softirqs.CPU16.SCHED
     28818           +21.2%      34940 ±  3%  softirqs.CPU17.SCHED
     28815           +22.9%      35414 ±  2%  softirqs.CPU18.SCHED
     43178           +20.4%      51986 ±  3%  softirqs.CPU19.NET_RX
     29087 ±  3%     +19.6%      34778 ±  3%  softirqs.CPU19.SCHED
     44717 ±  3%     +14.7%      51297 ±  6%  softirqs.CPU2.NET_RX
     28984           +21.8%      35294 ±  2%  softirqs.CPU2.SCHED
     44356 ±  2%     +15.0%      51025 ±  6%  softirqs.CPU20.NET_RX
     28834           +21.0%      34891 ±  3%  softirqs.CPU20.SCHED
     42907 ±  4%     +20.4%      51662 ±  7%  softirqs.CPU21.NET_RX
     28672           +22.2%      35025 ±  2%  softirqs.CPU21.SCHED
     28683           +21.8%      34945 ±  2%  softirqs.CPU22.SCHED
     29139 ±  2%     +20.3%      35064 ±  3%  softirqs.CPU23.SCHED
     29287 ±  2%     +19.6%      35029 ±  2%  softirqs.CPU24.SCHED
     44726 ±  3%     +17.3%      52482 ±  6%  softirqs.CPU25.NET_RX
     28571           +22.2%      34904 ±  2%  softirqs.CPU25.SCHED
     44972 ±  2%     +10.0%      49481 ±  7%  softirqs.CPU26.NET_RX
     28617           +22.2%      34967 ±  2%  softirqs.CPU26.SCHED
     28721           +22.5%      35183 ±  2%  softirqs.CPU27.SCHED
     45739 ±  2%      +9.9%      50261 ±  6%  softirqs.CPU28.NET_RX
     28797           +22.2%      35189 ±  3%  softirqs.CPU28.SCHED
     45037 ±  4%     +13.5%      51103 ±  7%  softirqs.CPU29.NET_RX
     28942           +21.1%      35049 ±  3%  softirqs.CPU29.SCHED
     43726 ±  5%     +17.9%      51536 ±  6%  softirqs.CPU3.NET_RX
     28660           +22.8%      35208 ±  3%  softirqs.CPU3.SCHED
     43924 ±  4%     +17.9%      51788 ±  6%  softirqs.CPU30.NET_RX
     28605           +22.9%      35160 ±  2%  softirqs.CPU30.SCHED
     44738 ±  2%     +14.0%      51018 ±  8%  softirqs.CPU31.NET_RX
     28525           +23.0%      35080 ±  3%  softirqs.CPU31.SCHED
     44269 ±  3%     +13.8%      50371 ±  6%  softirqs.CPU32.NET_RX
     50617           +11.0%      56186 ±  2%  softirqs.CPU32.RCU
     28729           +21.9%      35015 ±  2%  softirqs.CPU32.SCHED
     28775           +20.9%      34793 ±  2%  softirqs.CPU33.SCHED
     43305 ±  4%     +12.9%      48879 ±  7%  softirqs.CPU34.NET_RX
     28638           +22.2%      34998 ±  2%  softirqs.CPU34.SCHED
     50669 ±  2%     +12.5%      57026 ±  3%  softirqs.CPU35.RCU
     28492           +23.1%      35066 ±  2%  softirqs.CPU35.SCHED
     29276 ±  3%     +19.9%      35096 ±  2%  softirqs.CPU36.SCHED
     28654 ±  2%     +21.7%      34861 ±  3%  softirqs.CPU37.SCHED
     50752 ±  2%      +9.6%      55616 ±  2%  softirqs.CPU38.RCU
     28454           +22.8%      34936 ±  3%  softirqs.CPU38.SCHED
     28487           +22.2%      34801 ±  2%  softirqs.CPU39.SCHED
     28743           +22.8%      35304 ±  2%  softirqs.CPU4.SCHED
     44544 ±  5%     +11.7%      49768 ±  9%  softirqs.CPU40.NET_RX
     28550           +22.1%      34863 ±  3%  softirqs.CPU40.SCHED
     50810 ±  2%     +11.3%      56563 ±  3%  softirqs.CPU41.RCU
     28655           +22.3%      35032 ±  2%  softirqs.CPU41.SCHED
     45170 ±  6%     +13.7%      51352 ±  6%  softirqs.CPU42.NET_RX
     28518           +21.7%      34695 ±  2%  softirqs.CPU42.SCHED
     50807 ±  2%      +8.1%      54934 ±  3%  softirqs.CPU43.RCU
     28578           +22.6%      35036 ±  3%  softirqs.CPU43.SCHED
     44754 ±  3%     +13.5%      50778 ±  4%  softirqs.CPU44.NET_RX
     28636           +23.3%      35295 ±  3%  softirqs.CPU44.SCHED
     28345           +24.2%      35198 ±  2%  softirqs.CPU45.SCHED
     46288 ±  2%      +9.9%      50879 ±  6%  softirqs.CPU46.NET_RX
     52225 ±  6%      +7.1%      55921 ±  3%  softirqs.CPU46.RCU
     28752 ±  2%     +21.8%      35031 ±  3%  softirqs.CPU46.SCHED
     43887 ±  2%     +12.8%      49512 ±  7%  softirqs.CPU47.NET_RX
     50519 ±  2%     +10.7%      55911 ±  2%  softirqs.CPU47.RCU
     28516           +22.6%      34970 ±  3%  softirqs.CPU47.SCHED
     30033 ±  2%     +16.4%      34963 ±  2%  softirqs.CPU48.SCHED
     29484           +19.1%      35124 ±  2%  softirqs.CPU49.SCHED
     28753           +22.0%      35084 ±  2%  softirqs.CPU5.SCHED
     44597 ±  3%     +17.2%      52279 ±  2%  softirqs.CPU50.NET_RX
     29526           +19.1%      35171 ±  2%  softirqs.CPU50.SCHED
     44137           +16.1%      51253 ±  7%  softirqs.CPU51.NET_RX
     29663           +20.3%      35695 ±  2%  softirqs.CPU51.SCHED
     43768 ±  6%     +20.8%      52885 ±  6%  softirqs.CPU52.NET_RX
     29619           +19.9%      35507 ±  2%  softirqs.CPU52.SCHED
     42475 ±  4%     +22.1%      51878 ±  4%  softirqs.CPU53.NET_RX
     30051           +17.3%      35260 ±  2%  softirqs.CPU53.SCHED
     44059 ±  6%     +19.4%      52625 ±  7%  softirqs.CPU54.NET_RX
     29875           +17.4%      35066        softirqs.CPU54.SCHED
     29722           +18.3%      35154 ±  2%  softirqs.CPU55.SCHED
     43623 ±  3%     +15.5%      50389 ±  7%  softirqs.CPU56.NET_RX
     29500           +19.6%      35281 ±  2%  softirqs.CPU56.SCHED
     44928 ±  4%     +18.0%      53029 ±  5%  softirqs.CPU57.NET_RX
     29901 ±  2%     +18.3%      35365 ±  2%  softirqs.CPU57.SCHED
     43523 ±  3%     +20.4%      52386 ±  7%  softirqs.CPU58.NET_RX
     29630           +18.7%      35183 ±  2%  softirqs.CPU58.SCHED
     45144 ±  7%     +16.1%      52427 ± 10%  softirqs.CPU59.NET_RX
     29710 ±  2%     +19.2%      35426 ±  2%  softirqs.CPU59.SCHED
     45594 ±  3%     +14.7%      52301 ±  5%  softirqs.CPU6.NET_RX
     28614           +22.8%      35133 ±  3%  softirqs.CPU6.SCHED
     42332 ±  3%     +16.6%      49358 ±  6%  softirqs.CPU60.NET_RX
     29703           +19.6%      35537 ±  2%  softirqs.CPU60.SCHED
     42869 ±  5%     +22.3%      52443 ±  5%  softirqs.CPU61.NET_RX
     29194           +20.0%      35038 ±  2%  softirqs.CPU61.SCHED
     44402 ±  2%     +15.7%      51393 ±  5%  softirqs.CPU62.NET_RX
     29477           +19.5%      35239 ±  2%  softirqs.CPU62.SCHED
     44743 ±  3%     +15.4%      51633        softirqs.CPU63.NET_RX
     29756           +19.3%      35502 ±  2%  softirqs.CPU63.SCHED
     42981 ±  4%     +31.1%      56370 ± 20%  softirqs.CPU64.NET_RX
     29430           +21.3%      35709 ±  4%  softirqs.CPU64.SCHED
     43089 ±  2%     +17.5%      50634 ±  4%  softirqs.CPU65.NET_RX
     29470           +20.1%      35383 ±  2%  softirqs.CPU65.SCHED
     43585 ±  3%     +22.2%      53263 ±  5%  softirqs.CPU66.NET_RX
     29484           +20.4%      35502 ±  2%  softirqs.CPU66.SCHED
     29669           +18.3%      35097 ±  2%  softirqs.CPU67.SCHED
     43010           +15.8%      49814 ±  7%  softirqs.CPU68.NET_RX
     29500           +19.8%      35330 ±  2%  softirqs.CPU68.SCHED
     45182 ±  3%     +16.9%      52809 ±  7%  softirqs.CPU69.NET_RX
     29786           +18.2%      35212 ±  2%  softirqs.CPU69.SCHED
     45265 ±  2%     +14.1%      51629 ±  6%  softirqs.CPU7.NET_RX
     28447           +22.2%      34751 ±  4%  softirqs.CPU7.SCHED
     29553           +20.4%      35570 ±  2%  softirqs.CPU70.SCHED
     46032 ±  3%     +19.2%      54881 ± 11%  softirqs.CPU71.NET_RX
     29436           +20.0%      35316 ±  2%  softirqs.CPU71.SCHED
     44150 ±  2%     +17.2%      51757 ±  4%  softirqs.CPU72.NET_RX
     29687           +19.2%      35376 ±  2%  softirqs.CPU72.SCHED
     29410 ±  2%     +19.4%      35124        softirqs.CPU73.SCHED
     29595           +19.5%      35368        softirqs.CPU74.SCHED
     29561           +19.4%      35305 ±  2%  softirqs.CPU75.SCHED
     29573           +19.3%      35272 ±  2%  softirqs.CPU76.SCHED
     44691           +14.7%      51277 ±  3%  softirqs.CPU77.NET_RX
     29559           +18.6%      35061 ±  2%  softirqs.CPU77.SCHED
     29539           +19.2%      35197 ±  2%  softirqs.CPU78.SCHED
     43920 ±  4%     +15.7%      50809 ±  2%  softirqs.CPU79.NET_RX
     29344           +20.4%      35319 ±  2%  softirqs.CPU79.SCHED
     43484 ±  4%     +17.9%      51279 ±  7%  softirqs.CPU8.NET_RX
     28525           +21.6%      34681 ±  3%  softirqs.CPU8.SCHED
     29709           +18.7%      35251 ±  2%  softirqs.CPU80.SCHED
     42629 ±  4%     +19.4%      50914 ±  8%  softirqs.CPU81.NET_RX
     29446           +19.5%      35191 ±  2%  softirqs.CPU81.SCHED
     44074 ±  5%     +15.7%      51006 ±  2%  softirqs.CPU82.NET_RX
     49412           +12.1%      55405 ±  2%  softirqs.CPU82.RCU
     29419           +19.5%      35144 ±  2%  softirqs.CPU82.SCHED
     43015 ±  2%     +12.9%      48558 ±  5%  softirqs.CPU83.NET_RX
     29603           +19.6%      35395 ±  2%  softirqs.CPU83.SCHED
     44188 ±  5%     +15.8%      51184 ±  7%  softirqs.CPU84.NET_RX
     29510           +19.5%      35256 ±  2%  softirqs.CPU84.SCHED
     52136 ±  3%      +6.8%      55670 ±  3%  softirqs.CPU85.RCU
     29280           +20.3%      35212 ±  2%  softirqs.CPU85.SCHED
     44289 ±  4%     +14.7%      50805 ±  5%  softirqs.CPU86.NET_RX
     51516 ±  2%     +10.0%      56671 ±  4%  softirqs.CPU86.RCU
     29448           +19.7%      35241 ±  2%  softirqs.CPU86.SCHED
     29478           +18.8%      35030 ±  2%  softirqs.CPU87.SCHED
     51858            +8.5%      56267 ±  4%  softirqs.CPU88.RCU
     29288           +20.6%      35318        softirqs.CPU88.SCHED
     43345 ±  3%     +19.7%      51902 ±  5%  softirqs.CPU89.NET_RX
     29484           +19.6%      35266 ±  2%  softirqs.CPU89.SCHED
     28764           +21.4%      34906 ±  3%  softirqs.CPU9.SCHED
     44081 ±  7%     +24.6%      54920 ±  8%  softirqs.CPU90.NET_RX
     50851 ±  3%      +7.8%      54798        softirqs.CPU90.RCU
     29205           +19.8%      34988 ±  2%  softirqs.CPU90.SCHED
     44736 ±  2%     +16.0%      51898 ±  7%  softirqs.CPU91.NET_RX
     29462           +19.5%      35206 ±  3%  softirqs.CPU91.SCHED
     50818           +10.1%      55938 ±  5%  softirqs.CPU92.RCU
     29824           +17.7%      35115 ±  2%  softirqs.CPU92.SCHED
     50676 ±  2%     +10.4%      55933 ±  4%  softirqs.CPU93.RCU
     29438           +19.4%      35152 ±  2%  softirqs.CPU93.SCHED
     51367            +8.8%      55882 ±  4%  softirqs.CPU94.RCU
     29448           +20.5%      35498 ±  2%  softirqs.CPU94.SCHED
     44130 ±  4%     +15.5%      50963 ±  3%  softirqs.CPU95.NET_RX
     29554           +18.7%      35083        softirqs.CPU95.SCHED
   4295349           +14.4%    4915886        softirqs.NET_RX
   2800171           +20.6%    3377483 ±  2%  softirqs.SCHED
    574.75 ± 13%     +70.4%     979.50 ± 13%  interrupts.CPU0.TLB:TLB_shootdowns
      3102 ±  3%     -20.7%       2460 ±  6%  interrupts.CPU1.NMI:Non-maskable_interrupts
      3102 ±  3%     -20.7%       2460 ±  6%  interrupts.CPU1.PMI:Performance_monitoring_interrupts
    510.75 ± 16%     +66.2%     848.75 ± 13%  interrupts.CPU1.TLB:TLB_shootdowns
    442.50 ±  3%     +87.5%     829.75 ± 13%  interrupts.CPU10.TLB:TLB_shootdowns
      3108 ±  4%     -29.9%       2180 ± 25%  interrupts.CPU11.NMI:Non-maskable_interrupts
      3108 ±  4%     -29.9%       2180 ± 25%  interrupts.CPU11.PMI:Performance_monitoring_interrupts
    462.75 ± 10%     +69.5%     784.25 ± 19%  interrupts.CPU11.TLB:TLB_shootdowns
    463.75 ±  7%     +90.2%     882.25 ± 16%  interrupts.CPU12.TLB:TLB_shootdowns
      3102 ±  3%     -20.6%       2464 ±  7%  interrupts.CPU13.NMI:Non-maskable_interrupts
      3102 ±  3%     -20.6%       2464 ±  7%  interrupts.CPU13.PMI:Performance_monitoring_interrupts
    496.00 ±  7%     +79.5%     890.25 ±  4%  interrupts.CPU13.TLB:TLB_shootdowns
      3142 ±  5%     -30.2%       2194 ± 25%  interrupts.CPU14.NMI:Non-maskable_interrupts
      3142 ±  5%     -30.2%       2194 ± 25%  interrupts.CPU14.PMI:Performance_monitoring_interrupts
    475.00 ±  6%     +73.5%     824.00 ±  9%  interrupts.CPU14.TLB:TLB_shootdowns
      3118 ±  4%     -30.4%       2169 ± 25%  interrupts.CPU15.NMI:Non-maskable_interrupts
      3118 ±  4%     -30.4%       2169 ± 25%  interrupts.CPU15.PMI:Performance_monitoring_interrupts
    466.75 ± 10%     +84.0%     859.00 ± 10%  interrupts.CPU15.TLB:TLB_shootdowns
    446.75 ± 11%     +86.4%     832.75 ± 11%  interrupts.CPU16.TLB:TLB_shootdowns
      3103 ±  3%     -17.9%       2547 ±  7%  interrupts.CPU17.NMI:Non-maskable_interrupts
      3103 ±  3%     -17.9%       2547 ±  7%  interrupts.CPU17.PMI:Performance_monitoring_interrupts
    493.00 ±  8%     +70.1%     838.50 ± 13%  interrupts.CPU17.TLB:TLB_shootdowns
      3097 ±  4%     -29.3%       2191 ± 25%  interrupts.CPU18.NMI:Non-maskable_interrupts
      3097 ±  4%     -29.3%       2191 ± 25%  interrupts.CPU18.PMI:Performance_monitoring_interrupts
    491.75 ± 10%     +78.0%     875.50 ± 13%  interrupts.CPU18.TLB:TLB_shootdowns
      3087 ±  4%     -29.1%       2188 ± 25%  interrupts.CPU19.NMI:Non-maskable_interrupts
      3087 ±  4%     -29.1%       2188 ± 25%  interrupts.CPU19.PMI:Performance_monitoring_interrupts
    468.75 ±  7%     +73.3%     812.50 ± 16%  interrupts.CPU19.TLB:TLB_shootdowns
    496.00 ± 15%     +92.6%     955.25 ± 13%  interrupts.CPU2.TLB:TLB_shootdowns
      3109 ±  3%     -19.9%       2490 ±  5%  interrupts.CPU20.NMI:Non-maskable_interrupts
      3109 ±  3%     -19.9%       2490 ±  5%  interrupts.CPU20.PMI:Performance_monitoring_interrupts
    502.75 ± 11%     +71.4%     861.50 ± 10%  interrupts.CPU20.TLB:TLB_shootdowns
      3091 ±  4%     -19.9%       2477 ±  8%  interrupts.CPU21.NMI:Non-maskable_interrupts
      3091 ±  4%     -19.9%       2477 ±  8%  interrupts.CPU21.PMI:Performance_monitoring_interrupts
    474.25 ±  8%     +82.4%     865.00 ± 12%  interrupts.CPU21.TLB:TLB_shootdowns
      3103 ±  3%     -20.9%       2454 ±  7%  interrupts.CPU22.NMI:Non-maskable_interrupts
      3103 ±  3%     -20.9%       2454 ±  7%  interrupts.CPU22.PMI:Performance_monitoring_interrupts
    455.50 ± 20%     +89.1%     861.50 ± 17%  interrupts.CPU22.TLB:TLB_shootdowns
      3133 ±  3%     -30.9%       2163 ± 29%  interrupts.CPU23.NMI:Non-maskable_interrupts
      3133 ±  3%     -30.9%       2163 ± 29%  interrupts.CPU23.PMI:Performance_monitoring_interrupts
    479.25 ± 12%     +72.4%     826.25 ±  9%  interrupts.CPU23.TLB:TLB_shootdowns
    459.25 ± 12%    +100.9%     922.75 ± 13%  interrupts.CPU24.TLB:TLB_shootdowns
    505.50 ±  5%     +78.2%     900.75 ±  8%  interrupts.CPU25.TLB:TLB_shootdowns
    481.50 ± 14%     +96.3%     945.25 ± 14%  interrupts.CPU26.TLB:TLB_shootdowns
    498.00 ± 13%     +86.7%     930.00 ±  9%  interrupts.CPU27.TLB:TLB_shootdowns
    491.50 ± 10%     +82.6%     897.50 ± 19%  interrupts.CPU28.TLB:TLB_shootdowns
      3140 ±  3%     -39.7%       1894 ± 41%  interrupts.CPU29.NMI:Non-maskable_interrupts
      3140 ±  3%     -39.7%       1894 ± 41%  interrupts.CPU29.PMI:Performance_monitoring_interrupts
    459.00 ± 11%     +98.2%     909.75 ± 14%  interrupts.CPU29.TLB:TLB_shootdowns
      3136 ±  4%     -21.4%       2464 ±  6%  interrupts.CPU3.NMI:Non-maskable_interrupts
      3136 ±  4%     -21.4%       2464 ±  6%  interrupts.CPU3.PMI:Performance_monitoring_interrupts
    537.75 ± 13%     +68.6%     906.50 ± 11%  interrupts.CPU3.TLB:TLB_shootdowns
      3106 ±  3%     -30.3%       2165 ± 27%  interrupts.CPU30.NMI:Non-maskable_interrupts
      3106 ±  3%     -30.3%       2165 ± 27%  interrupts.CPU30.PMI:Performance_monitoring_interrupts
    465.50 ± 15%     +88.4%     877.00 ± 11%  interrupts.CPU30.TLB:TLB_shootdowns
      3096 ±  4%     -39.0%       1887 ± 41%  interrupts.CPU31.NMI:Non-maskable_interrupts
      3096 ±  4%     -39.0%       1887 ± 41%  interrupts.CPU31.PMI:Performance_monitoring_interrupts
    472.00 ± 12%     +83.0%     863.75 ± 10%  interrupts.CPU31.TLB:TLB_shootdowns
      3099 ±  4%     -50.4%       1538 ± 37%  interrupts.CPU32.NMI:Non-maskable_interrupts
      3099 ±  4%     -50.4%       1538 ± 37%  interrupts.CPU32.PMI:Performance_monitoring_interrupts
    470.50 ±  7%     +93.8%     911.75 ±  6%  interrupts.CPU32.TLB:TLB_shootdowns
    537.75 ± 14%     +67.8%     902.25 ± 13%  interrupts.CPU33.TLB:TLB_shootdowns
    444.00 ± 11%    +108.8%     927.25 ± 13%  interrupts.CPU34.TLB:TLB_shootdowns
    499.25 ± 12%     +81.0%     903.75 ± 20%  interrupts.CPU35.TLB:TLB_shootdowns
    487.50 ±  5%     +85.8%     906.00 ± 13%  interrupts.CPU36.TLB:TLB_shootdowns
    458.50 ± 17%     +94.5%     891.75 ± 12%  interrupts.CPU37.TLB:TLB_shootdowns
      1442 ±  4%      -5.9%       1357 ±  5%  interrupts.CPU38.CAL:Function_call_interrupts
    466.00 ± 11%     +98.8%     926.25 ± 13%  interrupts.CPU38.TLB:TLB_shootdowns
    480.75 ± 10%     +75.5%     843.50 ± 13%  interrupts.CPU39.TLB:TLB_shootdowns
    414.75 ±  6%    +115.9%     895.50 ±  5%  interrupts.CPU4.TLB:TLB_shootdowns
      3137 ±  2%     -21.6%       2459 ± 11%  interrupts.CPU40.NMI:Non-maskable_interrupts
      3137 ±  2%     -21.6%       2459 ± 11%  interrupts.CPU40.PMI:Performance_monitoring_interrupts
    486.50 ± 18%     +77.2%     862.00 ± 16%  interrupts.CPU40.TLB:TLB_shootdowns
    484.00 ± 17%     +95.8%     947.75 ± 13%  interrupts.CPU41.TLB:TLB_shootdowns
    453.75 ± 16%     +95.2%     885.50 ± 11%  interrupts.CPU42.TLB:TLB_shootdowns
      3116 ±  3%     -22.1%       2429 ±  7%  interrupts.CPU43.NMI:Non-maskable_interrupts
      3116 ±  3%     -22.1%       2429 ±  7%  interrupts.CPU43.PMI:Performance_monitoring_interrupts
    437.25 ±  9%     +99.8%     873.75 ± 11%  interrupts.CPU43.TLB:TLB_shootdowns
    452.75 ± 16%    +101.3%     911.25 ±  7%  interrupts.CPU44.TLB:TLB_shootdowns
      3180 ±  4%     -24.5%       2402 ±  9%  interrupts.CPU45.NMI:Non-maskable_interrupts
      3180 ±  4%     -24.5%       2402 ±  9%  interrupts.CPU45.PMI:Performance_monitoring_interrupts
    477.50 ± 14%     +88.8%     901.75 ± 11%  interrupts.CPU45.TLB:TLB_shootdowns
      3117 ±  3%     -20.9%       2465 ±  8%  interrupts.CPU46.NMI:Non-maskable_interrupts
      3117 ±  3%     -20.9%       2465 ±  8%  interrupts.CPU46.PMI:Performance_monitoring_interrupts
    493.50 ± 10%     +69.8%     837.75 ±  8%  interrupts.CPU46.TLB:TLB_shootdowns
    478.50 ± 16%     +76.7%     845.50 ±  4%  interrupts.CPU47.TLB:TLB_shootdowns
      3119 ±  3%     -31.3%       2142 ± 26%  interrupts.CPU48.NMI:Non-maskable_interrupts
      3119 ±  3%     -31.3%       2142 ± 26%  interrupts.CPU48.PMI:Performance_monitoring_interrupts
    464.50 ±  7%     +88.8%     876.75 ±  6%  interrupts.CPU48.TLB:TLB_shootdowns
    488.00 ± 11%     +82.2%     889.00 ± 15%  interrupts.CPU49.TLB:TLB_shootdowns
      3121 ±  4%     -19.0%       2527 ±  6%  interrupts.CPU5.NMI:Non-maskable_interrupts
      3121 ±  4%     -19.0%       2527 ±  6%  interrupts.CPU5.PMI:Performance_monitoring_interrupts
    458.00 ± 10%     +98.3%     908.00 ± 13%  interrupts.CPU5.TLB:TLB_shootdowns
      1382 ±  3%      +9.2%       1508 ±  4%  interrupts.CPU50.CAL:Function_call_interrupts
      3114 ±  4%     -21.5%       2445 ±  8%  interrupts.CPU50.NMI:Non-maskable_interrupts
      3114 ±  4%     -21.5%       2445 ±  8%  interrupts.CPU50.PMI:Performance_monitoring_interrupts
    492.00 ± 13%     +74.7%     859.75 ± 11%  interrupts.CPU50.TLB:TLB_shootdowns
      3123 ±  4%     -22.5%       2420 ±  6%  interrupts.CPU51.NMI:Non-maskable_interrupts
      3123 ±  4%     -22.5%       2420 ±  6%  interrupts.CPU51.PMI:Performance_monitoring_interrupts
    479.00 ± 17%     +74.2%     834.50 ±  8%  interrupts.CPU51.TLB:TLB_shootdowns
    458.25 ± 16%     +81.5%     831.50 ±  7%  interrupts.CPU52.TLB:TLB_shootdowns
      3144 ±  5%     -21.7%       2460 ±  8%  interrupts.CPU53.NMI:Non-maskable_interrupts
      3144 ±  5%     -21.7%       2460 ±  8%  interrupts.CPU53.PMI:Performance_monitoring_interrupts
    448.25 ±  8%    +102.2%     906.50 ± 15%  interrupts.CPU53.TLB:TLB_shootdowns
      3120 ±  4%     -30.2%       2177 ± 28%  interrupts.CPU54.NMI:Non-maskable_interrupts
      3120 ±  4%     -30.2%       2177 ± 28%  interrupts.CPU54.PMI:Performance_monitoring_interrupts
    464.00 ± 11%     +75.8%     815.50 ± 15%  interrupts.CPU54.TLB:TLB_shootdowns
      3107 ±  3%     -21.5%       2440 ± 10%  interrupts.CPU55.NMI:Non-maskable_interrupts
      3107 ±  3%     -21.5%       2440 ± 10%  interrupts.CPU55.PMI:Performance_monitoring_interrupts
    475.75 ± 15%     +80.9%     860.50 ±  6%  interrupts.CPU55.TLB:TLB_shootdowns
      3126 ±  4%     -32.1%       2121 ± 28%  interrupts.CPU56.NMI:Non-maskable_interrupts
      3126 ±  4%     -32.1%       2121 ± 28%  interrupts.CPU56.PMI:Performance_monitoring_interrupts
    504.75 ±  3%     +62.7%     821.00 ± 16%  interrupts.CPU56.TLB:TLB_shootdowns
    456.00 ±  6%     +79.9%     820.25 ± 11%  interrupts.CPU57.TLB:TLB_shootdowns
    436.50 ±  5%    +107.5%     905.75 ± 13%  interrupts.CPU58.TLB:TLB_shootdowns
    454.75 ±  9%     +74.9%     795.50 ± 12%  interrupts.CPU59.TLB:TLB_shootdowns
      1314 ±  3%      +8.6%       1426 ±  2%  interrupts.CPU6.CAL:Function_call_interrupts
      3102 ±  4%     -18.4%       2530 ±  8%  interrupts.CPU6.NMI:Non-maskable_interrupts
      3102 ±  4%     -18.4%       2530 ±  8%  interrupts.CPU6.PMI:Performance_monitoring_interrupts
    499.25 ± 13%     +61.0%     804.00 ± 10%  interrupts.CPU6.TLB:TLB_shootdowns
    478.25 ±  6%     +90.4%     910.75 ± 10%  interrupts.CPU60.TLB:TLB_shootdowns
    424.75 ±  7%    +111.9%     900.00 ±  9%  interrupts.CPU61.TLB:TLB_shootdowns
      1691 ±  4%      -8.8%       1542 ±  2%  interrupts.CPU62.RES:Rescheduling_interrupts
    460.75 ± 14%     +86.2%     858.00 ± 15%  interrupts.CPU62.TLB:TLB_shootdowns
    497.25 ± 15%     +69.8%     844.50 ± 16%  interrupts.CPU63.TLB:TLB_shootdowns
    472.75 ± 18%     +66.2%     785.50 ± 11%  interrupts.CPU64.TLB:TLB_shootdowns
    470.25 ±  2%     +69.9%     798.75 ± 11%  interrupts.CPU65.TLB:TLB_shootdowns
    457.00 ±  7%     +85.1%     845.75 ± 12%  interrupts.CPU66.TLB:TLB_shootdowns
      3087 ±  4%     -51.6%       1495 ± 30%  interrupts.CPU67.NMI:Non-maskable_interrupts
      3087 ±  4%     -51.6%       1495 ± 30%  interrupts.CPU67.PMI:Performance_monitoring_interrupts
      1686 ±  5%     -10.2%       1514 ±  2%  interrupts.CPU67.RES:Rescheduling_interrupts
    468.75 ±  7%     +84.0%     862.50 ± 13%  interrupts.CPU67.TLB:TLB_shootdowns
    436.75 ±  8%     +98.7%     868.00 ± 14%  interrupts.CPU68.TLB:TLB_shootdowns
      3092 ±  4%     -29.5%       2181 ± 28%  interrupts.CPU69.NMI:Non-maskable_interrupts
      3092 ±  4%     -29.5%       2181 ± 28%  interrupts.CPU69.PMI:Performance_monitoring_interrupts
    475.00 ± 16%     +76.9%     840.25 ±  9%  interrupts.CPU69.TLB:TLB_shootdowns
      3110 ±  4%     -20.6%       2470 ±  8%  interrupts.CPU7.NMI:Non-maskable_interrupts
      3110 ±  4%     -20.6%       2470 ±  8%  interrupts.CPU7.PMI:Performance_monitoring_interrupts
    480.50 ±  7%     +74.7%     839.25 ±  8%  interrupts.CPU7.TLB:TLB_shootdowns
      3102 ±  3%     -30.9%       2144 ± 28%  interrupts.CPU70.NMI:Non-maskable_interrupts
      3102 ±  3%     -30.9%       2144 ± 28%  interrupts.CPU70.PMI:Performance_monitoring_interrupts
    450.50 ±  7%     +92.7%     868.25 ± 12%  interrupts.CPU70.TLB:TLB_shootdowns
      3140 ±  3%     -40.5%       1868 ± 37%  interrupts.CPU71.NMI:Non-maskable_interrupts
      3140 ±  3%     -40.5%       1868 ± 37%  interrupts.CPU71.PMI:Performance_monitoring_interrupts
    429.50 ±  5%    +100.0%     859.00 ± 13%  interrupts.CPU71.TLB:TLB_shootdowns
    475.25 ± 10%     +91.2%     908.75 ±  4%  interrupts.CPU72.TLB:TLB_shootdowns
      3097 ±  3%     -30.6%       2149 ± 27%  interrupts.CPU73.NMI:Non-maskable_interrupts
      3097 ±  3%     -30.6%       2149 ± 27%  interrupts.CPU73.PMI:Performance_monitoring_interrupts
    480.50 ±  6%     +73.1%     831.75 ±  7%  interrupts.CPU73.TLB:TLB_shootdowns
      1498 ±  6%      -7.7%       1383 ±  2%  interrupts.CPU74.CAL:Function_call_interrupts
      3120 ±  3%     -31.0%       2153 ± 28%  interrupts.CPU74.NMI:Non-maskable_interrupts
      3120 ±  3%     -31.0%       2153 ± 28%  interrupts.CPU74.PMI:Performance_monitoring_interrupts
    483.75 ±  8%     +86.3%     901.00 ± 18%  interrupts.CPU74.TLB:TLB_shootdowns
      3129 ±  3%     -31.8%       2135 ± 27%  interrupts.CPU75.NMI:Non-maskable_interrupts
      3129 ±  3%     -31.8%       2135 ± 27%  interrupts.CPU75.PMI:Performance_monitoring_interrupts
      1619 ±  5%      -8.1%       1487 ±  2%  interrupts.CPU75.RES:Rescheduling_interrupts
    497.00 ± 17%     +70.7%     848.50 ± 10%  interrupts.CPU75.TLB:TLB_shootdowns
      3123 ±  4%     -33.0%       2091 ± 22%  interrupts.CPU76.NMI:Non-maskable_interrupts
      3123 ±  4%     -33.0%       2091 ± 22%  interrupts.CPU76.PMI:Performance_monitoring_interrupts
    486.25 ± 12%     +76.6%     858.50 ±  9%  interrupts.CPU76.TLB:TLB_shootdowns
      3148 ±  2%     -32.7%       2120 ± 20%  interrupts.CPU77.NMI:Non-maskable_interrupts
      3148 ±  2%     -32.7%       2120 ± 20%  interrupts.CPU77.PMI:Performance_monitoring_interrupts
    473.75 ± 13%     +99.6%     945.50 ±  5%  interrupts.CPU77.TLB:TLB_shootdowns
    458.00 ± 13%     +91.4%     876.75 ± 12%  interrupts.CPU78.TLB:TLB_shootdowns
      3096 ±  4%     -32.0%       2105 ± 20%  interrupts.CPU79.NMI:Non-maskable_interrupts
      3096 ±  4%     -32.0%       2105 ± 20%  interrupts.CPU79.PMI:Performance_monitoring_interrupts
    485.75 ±  4%     +91.1%     928.50 ± 12%  interrupts.CPU79.TLB:TLB_shootdowns
      3128 ±  4%     -20.9%       2475 ±  6%  interrupts.CPU8.NMI:Non-maskable_interrupts
      3128 ±  4%     -20.9%       2475 ±  6%  interrupts.CPU8.PMI:Performance_monitoring_interrupts
    471.50 ±  9%     +90.1%     896.25 ±  8%  interrupts.CPU8.TLB:TLB_shootdowns
      3105 ±  3%     -32.9%       2084 ± 21%  interrupts.CPU80.NMI:Non-maskable_interrupts
      3105 ±  3%     -32.9%       2084 ± 21%  interrupts.CPU80.PMI:Performance_monitoring_interrupts
      1640 ±  2%     -10.3%       1472 ±  2%  interrupts.CPU80.RES:Rescheduling_interrupts
    462.00 ±  8%     +87.1%     864.25 ± 11%  interrupts.CPU80.TLB:TLB_shootdowns
      3101 ±  4%     -22.1%       2417 ± 11%  interrupts.CPU81.NMI:Non-maskable_interrupts
      3101 ±  4%     -22.1%       2417 ± 11%  interrupts.CPU81.PMI:Performance_monitoring_interrupts
    460.50 ±  9%     +98.7%     915.00 ± 11%  interrupts.CPU81.TLB:TLB_shootdowns
      3100 ±  3%     -30.0%       2170 ± 29%  interrupts.CPU82.NMI:Non-maskable_interrupts
      3100 ±  3%     -30.0%       2170 ± 29%  interrupts.CPU82.PMI:Performance_monitoring_interrupts
    458.25 ± 14%     +86.3%     853.75 ± 12%  interrupts.CPU82.TLB:TLB_shootdowns
    485.25 ± 13%     +88.5%     914.75 ± 11%  interrupts.CPU83.TLB:TLB_shootdowns
      3096 ±  4%     -31.7%       2114 ± 21%  interrupts.CPU84.NMI:Non-maskable_interrupts
      3096 ±  4%     -31.7%       2114 ± 21%  interrupts.CPU84.PMI:Performance_monitoring_interrupts
    505.25 ± 22%     +78.7%     902.75 ± 18%  interrupts.CPU84.TLB:TLB_shootdowns
      3116 ±  4%     -20.9%       2465 ±  7%  interrupts.CPU85.NMI:Non-maskable_interrupts
      3116 ±  4%     -20.9%       2465 ±  7%  interrupts.CPU85.PMI:Performance_monitoring_interrupts
    480.75 ± 12%     +82.3%     876.25 ±  7%  interrupts.CPU85.TLB:TLB_shootdowns
    496.75 ± 13%     +76.7%     877.75 ± 11%  interrupts.CPU86.TLB:TLB_shootdowns
    459.00 ± 13%     +71.6%     787.50 ± 16%  interrupts.CPU87.TLB:TLB_shootdowns
    446.50 ± 14%    +111.5%     944.50 ± 11%  interrupts.CPU88.TLB:TLB_shootdowns
      3120 ±  3%     -22.3%       2424 ±  9%  interrupts.CPU89.NMI:Non-maskable_interrupts
      3120 ±  3%     -22.3%       2424 ±  9%  interrupts.CPU89.PMI:Performance_monitoring_interrupts
    454.00 ± 12%     +83.8%     834.50 ±  9%  interrupts.CPU89.TLB:TLB_shootdowns
      3162 ±  5%     -19.6%       2541 ±  8%  interrupts.CPU9.NMI:Non-maskable_interrupts
      3162 ±  5%     -19.6%       2541 ±  8%  interrupts.CPU9.PMI:Performance_monitoring_interrupts
    461.75 ± 16%     +85.9%     858.50 ± 15%  interrupts.CPU9.TLB:TLB_shootdowns
    447.25 ±  5%     +90.0%     849.75 ± 13%  interrupts.CPU90.TLB:TLB_shootdowns
    440.75 ±  5%     +94.7%     858.00 ± 17%  interrupts.CPU91.TLB:TLB_shootdowns
      1651 ±  2%     -10.6%       1476 ±  3%  interrupts.CPU92.RES:Rescheduling_interrupts
    471.25 ±  9%     +76.3%     831.00 ±  7%  interrupts.CPU92.TLB:TLB_shootdowns
      3104 ±  3%     -40.5%       1846 ± 41%  interrupts.CPU93.NMI:Non-maskable_interrupts
      3104 ±  3%     -40.5%       1846 ± 41%  interrupts.CPU93.PMI:Performance_monitoring_interrupts
      1647 ±  4%      -8.4%       1508 ±  3%  interrupts.CPU93.RES:Rescheduling_interrupts
    556.25 ± 14%     +60.3%     891.75 ±  2%  interrupts.CPU93.TLB:TLB_shootdowns
      3093 ±  3%     -30.6%       2147 ± 29%  interrupts.CPU94.NMI:Non-maskable_interrupts
      3093 ±  3%     -30.6%       2147 ± 29%  interrupts.CPU94.PMI:Performance_monitoring_interrupts
    438.00 ± 12%    +103.0%     889.25 ± 21%  interrupts.CPU94.TLB:TLB_shootdowns
    276499 ±  2%     -22.2%     215100 ± 11%  interrupts.NMI:Non-maskable_interrupts
    276499 ±  2%     -22.2%     215100 ± 11%  interrupts.PMI:Performance_monitoring_interrupts
     45786 ±  8%    +103.8%      93329 ± 23%  interrupts.TLB:TLB_shootdowns
     64.68 ±  2%     -64.7        0.00        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.scan_positives.dcache_readdir.iterate_dir
     66.88 ±  2%     -52.7       14.20 ±  5%  perf-profile.calltrace.cycles-pp.dcache_readdir.iterate_dir.__x64_sys_getdents64.do_syscall_64.entry_SYSCALL_64_after_hwframe
     66.90 ±  2%     -52.7       14.22 ±  5%  perf-profile.calltrace.cycles-pp.__x64_sys_getdents64.do_syscall_64.entry_SYSCALL_64_after_hwframe
     66.90 ±  2%     -52.7       14.22 ±  5%  perf-profile.calltrace.cycles-pp.iterate_dir.__x64_sys_getdents64.do_syscall_64.entry_SYSCALL_64_after_hwframe
     65.76 ±  2%     -52.1       13.63 ±  5%  perf-profile.calltrace.cycles-pp.scan_positives.dcache_readdir.iterate_dir.__x64_sys_getdents64.do_syscall_64
     65.27 ±  2%     -52.0       13.30 ±  5%  perf-profile.calltrace.cycles-pp._raw_spin_lock.scan_positives.dcache_readdir.iterate_dir.__x64_sys_getdents64
     70.98 ±  2%     -46.2       24.74 ± 13%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
     71.11 ±  2%     -46.1       24.96 ± 13%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.70 ± 14%      +0.4        1.15 ±  6%  perf-profile.calltrace.cycles-pp.div_short
      0.61 ± 13%      +0.5        1.07 ±  6%  perf-profile.calltrace.cycles-pp.div_int
      0.64 ±  5%      +0.5        1.13 ±  2%  perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
      0.58 ±  7%      +0.5        1.10 ±  3%  perf-profile.calltrace.cycles-pp.begin_new_exec.load_elf_binary.exec_binprm.bprm_execve.do_execveat_common
      0.68 ±  5%      +0.5        1.20 ±  2%  perf-profile.calltrace.cycles-pp.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
      0.00            +0.5        0.54 ±  4%  perf-profile.calltrace.cycles-pp.setlocale
      0.68 ± 12%      +0.6        1.23 ±  8%  perf-profile.calltrace.cycles-pp.add_short.add_short
      0.00            +0.6        0.58        perf-profile.calltrace.cycles-pp.filemap_map_pages.do_fault.__handle_mm_fault.handle_mm_fault.do_user_addr_fault
      0.76 ±  5%      +0.6        1.36 ±  2%  perf-profile.calltrace.cycles-pp.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
      0.77 ±  5%      +0.6        1.37 ±  2%  perf-profile.calltrace.cycles-pp.exc_page_fault.asm_exc_page_fault
      0.43 ± 57%      +0.6        1.03 ±  4%  perf-profile.calltrace.cycles-pp.mmput.begin_new_exec.load_elf_binary.exec_binprm.bprm_execve
      0.43 ± 57%      +0.6        1.03 ±  4%  perf-profile.calltrace.cycles-pp.exit_mmap.mmput.begin_new_exec.load_elf_binary.exec_binprm
      0.73 ±  9%      +0.6        1.34 ±  5%  perf-profile.calltrace.cycles-pp.exit_mmap.mmput.do_exit.do_group_exit.__x64_sys_exit_group
      0.43 ± 58%      +0.6        1.05 ±  6%  perf-profile.calltrace.cycles-pp.__do_sys_clone.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.43 ± 58%      +0.6        1.05 ±  6%  perf-profile.calltrace.cycles-pp._do_fork.__do_sys_clone.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.73 ±  9%      +0.6        1.35 ±  6%  perf-profile.calltrace.cycles-pp.mmput.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      0.41 ± 58%      +0.6        1.04 ±  6%  perf-profile.calltrace.cycles-pp.link_path_walk.path_openat.do_filp_open.do_sys_openat2.do_sys_open
      0.79 ±  5%      +0.6        1.42 ±  2%  perf-profile.calltrace.cycles-pp.asm_exc_page_fault
      0.00            +0.6        0.65 ±  4%  perf-profile.calltrace.cycles-pp.path_lookupat.filename_lookup.vfs_statx.__do_sys_newstat.do_syscall_64
      0.27 ±100%      +0.7        0.93 ±  9%  perf-profile.calltrace.cycles-pp.mem_rtns_1
      0.00            +0.7        0.66 ±  3%  perf-profile.calltrace.cycles-pp.filename_lookup.vfs_statx.__do_sys_newstat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +0.7        0.67        perf-profile.calltrace.cycles-pp.do_fault.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault
      0.46 ± 59%      +0.7        1.15 ±  4%  perf-profile.calltrace.cycles-pp.do_sys_openat2.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe.creat64
      0.46 ± 59%      +0.7        1.15 ±  4%  perf-profile.calltrace.cycles-pp.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe.creat64
      0.00            +0.7        0.69 ±  8%  perf-profile.calltrace.cycles-pp.refill_obj_stock.kmem_cache_free.rcu_do_batch.rcu_core.__softirqentry_text_start
      0.00            +0.7        0.69 ±  4%  perf-profile.calltrace.cycles-pp.vfs_statx.__do_sys_newstat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.46 ± 59%      +0.7        1.16 ±  4%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.creat64
      0.46 ± 59%      +0.7        1.16 ±  4%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.creat64
      0.00            +0.7        0.70 ±  4%  perf-profile.calltrace.cycles-pp.__do_sys_newstat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.48 ± 59%      +0.7        1.19 ±  5%  perf-profile.calltrace.cycles-pp.creat64
      0.27 ±100%      +0.7        0.99 ±  6%  perf-profile.calltrace.cycles-pp.walk_component.link_path_walk.path_openat.do_filp_open.do_sys_openat2
      0.82 ± 10%      +0.7        1.55 ± 11%  perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      0.14 ±173%      +0.7        0.88 ±  8%  perf-profile.calltrace.cycles-pp.__libc_fork
      0.62 ± 12%      +0.8        1.38 ± 14%  perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle
      0.90 ±  8%      +0.8        1.65 ±  5%  perf-profile.calltrace.cycles-pp.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.90 ±  8%      +0.8        1.65 ±  5%  perf-profile.calltrace.cycles-pp.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.90 ±  8%      +0.8        1.65 ±  5%  perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +0.8        0.77 ±  9%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_fork
      0.00            +0.8        0.77 ±  9%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_fork
      0.00            +0.8        0.77 ±  9%  perf-profile.calltrace.cycles-pp.__do_sys_clone.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_fork
      0.00            +0.8        0.77 ±  9%  perf-profile.calltrace.cycles-pp._do_fork.__do_sys_clone.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_fork
      0.90 ±  8%      +0.8        1.68 ±  7%  perf-profile.calltrace.cycles-pp.__strcat_chk
      0.15 ±173%      +0.8        0.93 ± 12%  perf-profile.calltrace.cycles-pp.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64
      0.16 ±173%      +0.8        0.97 ± 11%  perf-profile.calltrace.cycles-pp.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.17 ±173%      +0.8        0.99 ± 12%  perf-profile.calltrace.cycles-pp.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.16 ±173%      +0.8        0.99 ± 12%  perf-profile.calltrace.cycles-pp.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +0.9        0.90 ± 29%  perf-profile.calltrace.cycles-pp.dup_mmap.dup_mm.copy_process._do_fork.__do_sys_clone
      1.04 ±  5%      +0.9        1.97 ±  3%  perf-profile.calltrace.cycles-pp.load_elf_binary.exec_binprm.bprm_execve.do_execveat_common.__x64_sys_execve
      1.04 ±  5%      +0.9        1.99 ±  3%  perf-profile.calltrace.cycles-pp.exec_binprm.bprm_execve.do_execveat_common.__x64_sys_execve.do_syscall_64
      1.41 ± 13%      +1.0        2.42 ±  5%  perf-profile.calltrace.cycles-pp.div_long
      0.00            +1.1        1.05 ± 30%  perf-profile.calltrace.cycles-pp.dup_mm.copy_process._do_fork.__do_sys_clone.do_syscall_64
      1.52 ± 13%      +1.1        2.60 ±  6%  perf-profile.calltrace.cycles-pp.add_long.add_long
      1.52 ± 13%      +1.1        2.63 ±  7%  perf-profile.calltrace.cycles-pp.add_int.add_int
      0.48 ± 59%      +1.3        1.74 ± 24%  perf-profile.calltrace.cycles-pp.kmem_cache_free.rcu_do_batch.rcu_core.__softirqentry_text_start.run_ksoftirqd
      1.65 ±  8%      +1.3        2.99 ±  6%  perf-profile.calltrace.cycles-pp.__strncat_chk
      0.55 ± 58%      +1.4        1.95 ± 22%  perf-profile.calltrace.cycles-pp.rcu_do_batch.rcu_core.__softirqentry_text_start.run_ksoftirqd.smpboot_thread_fn
      0.56 ± 58%      +1.4        1.96 ± 22%  perf-profile.calltrace.cycles-pp.__softirqentry_text_start.run_ksoftirqd.smpboot_thread_fn.kthread.ret_from_fork
      0.56 ± 58%      +1.4        1.96 ± 22%  perf-profile.calltrace.cycles-pp.rcu_core.__softirqentry_text_start.run_ksoftirqd.smpboot_thread_fn.kthread
      0.56 ± 58%      +1.4        1.96 ± 22%  perf-profile.calltrace.cycles-pp.run_ksoftirqd.smpboot_thread_fn.kthread.ret_from_fork
      0.57 ± 58%      +1.4        1.99 ± 21%  perf-profile.calltrace.cycles-pp.smpboot_thread_fn.kthread.ret_from_fork
      0.14 ±173%      +1.5        1.64 ±  7%  perf-profile.calltrace.cycles-pp.copy_process._do_fork.__do_sys_clone.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.63 ± 58%      +1.5        2.13 ± 20%  perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
      0.64 ± 58%      +1.5        2.15 ± 20%  perf-profile.calltrace.cycles-pp.ret_from_fork
      1.00 ± 17%      +2.1        3.09 ± 43%  perf-profile.calltrace.cycles-pp.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.00 ± 17%      +2.1        3.08 ± 43%  perf-profile.calltrace.cycles-pp.do_sys_openat2.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.43 ±  9%      +4.4        9.83 ±  6%  perf-profile.calltrace.cycles-pp.string_rtns_1
      0.00            +5.8        5.81 ±  7%  perf-profile.calltrace.cycles-pp.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe.__open64_nocancel
      0.00            +5.8        5.81 ±  7%  perf-profile.calltrace.cycles-pp.do_sys_openat2.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe.__open64_nocancel
      0.00            +5.8        5.81 ±  7%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__open64_nocancel
      0.00            +5.8        5.81 ±  7%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__open64_nocancel
      0.00            +5.8        5.82 ±  7%  perf-profile.calltrace.cycles-pp.__open64_nocancel
      4.44 ± 12%      +5.9       10.34 ±  4%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      5.53 ± 11%      +6.7       12.21 ±  5%  perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      0.00            +6.7        6.72 ± 13%  perf-profile.calltrace.cycles-pp._raw_spin_lock.lockref_get_not_dead.__legitimize_path.unlazy_walk.complete_walk
      5.33 ± 11%      +6.7       12.05 ±  5%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
      0.00            +6.7        6.72 ± 13%  perf-profile.calltrace.cycles-pp.unlazy_walk.complete_walk.path_openat.do_filp_open.do_sys_openat2
      0.00            +6.7        6.72 ± 13%  perf-profile.calltrace.cycles-pp.__legitimize_path.unlazy_walk.complete_walk.path_openat.do_filp_open
      0.00            +6.7        6.72 ± 13%  perf-profile.calltrace.cycles-pp.lockref_get_not_dead.__legitimize_path.unlazy_walk.complete_walk.path_openat
      0.00            +6.7        6.72 ± 13%  perf-profile.calltrace.cycles-pp.complete_walk.path_openat.do_filp_open.do_sys_openat2.do_sys_open
      6.10 ± 10%      +6.8       12.93 ±  5%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      6.10 ± 10%      +6.8       12.93 ±  5%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
      6.10 ± 10%      +6.8       12.93 ±  5%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
      6.17 ± 10%      +6.9       13.07 ±  4%  perf-profile.calltrace.cycles-pp.secondary_startup_64
      1.22 ± 13%      +8.6        9.79 ±  8%  perf-profile.calltrace.cycles-pp.path_openat.do_filp_open.do_sys_openat2.do_sys_open.do_syscall_64
      1.22 ± 13%      +8.6        9.80 ±  8%  perf-profile.calltrace.cycles-pp.do_filp_open.do_sys_openat2.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00           +12.6       12.57 ±  5%  perf-profile.calltrace.cycles-pp.__cna_queued_spin_lock_slowpath._raw_spin_lock.scan_positives.dcache_readdir.iterate_dir
      1.48 ±  5%     +16.3       17.75 ±  6%  perf-profile.calltrace.cycles-pp.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve
      1.50 ±  5%     +16.3       17.78 ±  6%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.execve
      1.49 ±  5%     +16.3       17.78 ±  6%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve
      1.49 ±  5%     +16.3       17.78 ±  6%  perf-profile.calltrace.cycles-pp.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve
      1.51 ±  5%     +16.3       17.80 ±  6%  perf-profile.calltrace.cycles-pp.execve
      0.00           +16.5       16.49 ±  5%  perf-profile.calltrace.cycles-pp._raw_spin_lock.lockref_get_not_dead.__legitimize_path.unlazy_walk.pick_link
      0.00           +16.5       16.50 ±  5%  perf-profile.calltrace.cycles-pp.unlazy_walk.pick_link.step_into.path_openat.do_filp_open
      0.00           +16.5       16.50 ±  5%  perf-profile.calltrace.cycles-pp.__legitimize_path.unlazy_walk.pick_link.step_into.path_openat
      0.00           +16.5       16.50 ±  5%  perf-profile.calltrace.cycles-pp.lockref_get_not_dead.__legitimize_path.unlazy_walk.pick_link.step_into
      0.00           +16.5       16.50 ±  5%  perf-profile.calltrace.cycles-pp.pick_link.step_into.path_openat.do_filp_open.do_open_execat
      0.00           +16.9       16.87 ±  5%  perf-profile.calltrace.cycles-pp.step_into.path_openat.do_filp_open.do_open_execat.bprm_execve
      0.00           +16.9       16.91 ±  5%  perf-profile.calltrace.cycles-pp.do_filp_open.do_open_execat.bprm_execve.do_execveat_common.__x64_sys_execve
      0.00           +16.9       16.91 ±  5%  perf-profile.calltrace.cycles-pp.path_openat.do_filp_open.do_open_execat.bprm_execve.do_execveat_common
      0.00           +16.9       16.91 ±  5%  perf-profile.calltrace.cycles-pp.do_open_execat.bprm_execve.do_execveat_common.__x64_sys_execve.do_syscall_64
      1.24 ±  6%     +18.0       19.29 ±  4%  perf-profile.calltrace.cycles-pp.bprm_execve.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00           +23.2       23.20 ±  6%  perf-profile.calltrace.cycles-pp.__cna_queued_spin_lock_slowpath._raw_spin_lock.lockref_get_not_dead.__legitimize_path.unlazy_walk
     66.69 ±  2%     -66.7        0.00        perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
     66.88 ±  2%     -52.7       14.20 ±  5%  perf-profile.children.cycles-pp.dcache_readdir
     66.90 ±  2%     -52.7       14.22 ±  5%  perf-profile.children.cycles-pp.__x64_sys_getdents64
     66.90 ±  2%     -52.7       14.22 ±  5%  perf-profile.children.cycles-pp.iterate_dir
     65.76 ±  2%     -52.1       13.63 ±  5%  perf-profile.children.cycles-pp.scan_positives
     67.52 ±  2%     -28.2       39.28 ±  5%  perf-profile.children.cycles-pp._raw_spin_lock
     75.25 ±  2%     -22.2       53.04 ±  3%  perf-profile.children.cycles-pp.do_syscall_64
     75.60 ±  2%     -22.0       53.65 ±  3%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.08 ±  5%      +0.0        0.10 ±  7%  perf-profile.children.cycles-pp.update_cfs_group
      0.05            +0.0        0.08 ±  6%  perf-profile.children.cycles-pp.__send_signal
      0.05            +0.0        0.08 ±  5%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.08 ± 10%      +0.0        0.11 ±  9%  perf-profile.children.cycles-pp.vma_interval_tree_remove
      0.05            +0.0        0.08 ±  5%  perf-profile.children.cycles-pp.__setup_rt_frame
      0.04 ± 57%      +0.0        0.07 ± 15%  perf-profile.children.cycles-pp.restore_sigcontext
      0.04 ± 58%      +0.0        0.08 ± 10%  perf-profile.children.cycles-pp.generic_file_write_iter
      0.06 ± 11%      +0.0        0.10 ± 15%  perf-profile.children.cycles-pp.__x64_sys_rt_sigreturn
      0.04 ± 58%      +0.0        0.08        perf-profile.children.cycles-pp.try_charge
      0.04 ± 57%      +0.0        0.08 ±  6%  perf-profile.children.cycles-pp.rmqueue
      0.07 ± 26%      +0.0        0.11 ±  4%  perf-profile.children.cycles-pp.drm_fb_helper_dirty_work
      0.05 ±  8%      +0.0        0.09 ±  4%  perf-profile.children.cycles-pp._copy_from_user
      0.04 ± 57%      +0.0        0.08 ± 10%  perf-profile.children.cycles-pp.group_send_sig_info
      0.04 ± 57%      +0.0        0.08 ±  5%  perf-profile.children.cycles-pp.__x64_sys_link
      0.04 ± 57%      +0.0        0.08 ±  5%  perf-profile.children.cycles-pp.do_linkat
      0.06 ±  9%      +0.0        0.10 ± 11%  perf-profile.children.cycles-pp.sort_rtns_1
      0.03 ±100%      +0.0        0.07 ±  7%  perf-profile.children.cycles-pp.compar2
      0.03 ±100%      +0.0        0.07 ± 12%  perf-profile.children.cycles-pp.sighandler
      0.07 ±  6%      +0.0        0.11 ±  6%  perf-profile.children.cycles-pp.__check_object_size
      0.05            +0.0        0.09 ±  8%  perf-profile.children.cycles-pp.__local_bh_enable_ip
      0.06 ±  6%      +0.0        0.11 ±  4%  perf-profile.children.cycles-pp.inode_permission
      0.06            +0.0        0.10 ±  8%  perf-profile.children.cycles-pp.entry_SYSCALL_64
      0.16 ±  4%      +0.0        0.20 ±  2%  perf-profile.children.cycles-pp.task_tick_fair
      0.07 ± 10%      +0.0        0.11 ±  4%  perf-profile.children.cycles-pp.alloc_pages_vma
      0.04 ± 57%      +0.0        0.08 ± 27%  perf-profile.children.cycles-pp.rebalance_domains
      0.08 ±  5%      +0.0        0.12 ±  6%  perf-profile.children.cycles-pp.kill_pid_info
      0.07 ±  7%      +0.0        0.11 ±  6%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.05            +0.0        0.10 ±  5%  perf-profile.children.cycles-pp.free_pages_and_swap_cache
      0.04 ± 58%      +0.0        0.09 ± 10%  perf-profile.children.cycles-pp.do_notify_parent
      0.05 ±  8%      +0.0        0.10 ± 11%  perf-profile.children.cycles-pp.ima_file_check
      0.05 ±  9%      +0.0        0.10 ±  7%  perf-profile.children.cycles-pp.mem_cgroup_charge
      0.04 ± 57%      +0.0        0.08 ±  5%  perf-profile.children.cycles-pp.copy_page
      0.06 ±  6%      +0.0        0.11 ±  6%  perf-profile.children.cycles-pp.update_load_avg
      0.05 ±  8%      +0.0        0.10 ±  7%  perf-profile.children.cycles-pp.unlock_page
      0.04 ± 57%      +0.0        0.09 ± 17%  perf-profile.children.cycles-pp.mm_init
      0.03 ±100%      +0.0        0.07 ± 15%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      0.03 ±100%      +0.0        0.07 ± 11%  perf-profile.children.cycles-pp.security_task_kill
      0.03 ±100%      +0.0        0.07 ± 11%  perf-profile.children.cycles-pp.apparmor_task_kill
      0.03 ±100%      +0.0        0.07 ±  5%  perf-profile.children.cycles-pp.link
      0.06 ±  7%      +0.0        0.11 ±  8%  perf-profile.children.cycles-pp.security_task_getsecid
      0.07 ±  7%      +0.0        0.11 ±  7%  perf-profile.children.cycles-pp.ip_finish_output2
      0.07 ± 22%      +0.0        0.12 ±  8%  perf-profile.children.cycles-pp.devkmsg_write.cold
      0.07 ± 22%      +0.0        0.12 ±  8%  perf-profile.children.cycles-pp.devkmsg_emit
      0.06 ±  6%      +0.0        0.11 ±  3%  perf-profile.children.cycles-pp.get_arg_page
      0.08 ±  8%      +0.1        0.13 ±  5%  perf-profile.children.cycles-pp.udp_sendmsg
      0.08 ±  5%      +0.1        0.13 ±  6%  perf-profile.children.cycles-pp.kill_something_info
      0.06 ± 11%      +0.1        0.11        perf-profile.children.cycles-pp.finish_task_switch
      0.06 ± 11%      +0.1        0.11 ±  6%  perf-profile.children.cycles-pp.__get_user_pages
      0.07 ± 14%      +0.1        0.12 ± 12%  perf-profile.children.cycles-pp.unlink
      0.04 ± 57%      +0.1        0.09 ±  4%  perf-profile.children.cycles-pp.aa_get_task_label
      0.03 ±100%      +0.1        0.08 ± 10%  perf-profile.children.cycles-pp.__generic_file_write_iter
      0.01 ±173%      +0.1        0.06 ± 13%  perf-profile.children.cycles-pp.wait_task_zombie
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.memmove
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.generic_permission
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.ip_local_deliver
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.arch_get_unmapped_area_topdown
      0.03 ±100%      +0.1        0.08 ±  6%  perf-profile.children.cycles-pp.udp_send_skb
      0.03 ±100%      +0.1        0.08 ±  6%  perf-profile.children.cycles-pp.ip_send_skb
      0.03 ±100%      +0.1        0.08 ±  6%  perf-profile.children.cycles-pp.__slab_alloc
      0.09 ± 28%      +0.1        0.14 ±  8%  perf-profile.children.cycles-pp.propagate_protected_usage
      0.05 ±  9%      +0.1        0.11 ±  8%  perf-profile.children.cycles-pp.apparmor_task_getsecid
      0.07 ±  5%      +0.1        0.12 ±  8%  perf-profile.children.cycles-pp.ip_output
      0.07 ± 10%      +0.1        0.12 ±  6%  perf-profile.children.cycles-pp.simple_lookup
      0.07 ± 12%      +0.1        0.12 ± 10%  perf-profile.children.cycles-pp.sync_regs
      0.06 ± 13%      +0.1        0.11 ±  4%  perf-profile.children.cycles-pp.read
      0.06 ± 11%      +0.1        0.11 ±  3%  perf-profile.children.cycles-pp.__get_user_pages_remote
      0.05 ±  9%      +0.1        0.11 ± 12%  perf-profile.children.cycles-pp.call_rcu
      0.03 ±100%      +0.1        0.08 ±  5%  perf-profile.children.cycles-pp.enqueue_task_fair
      0.09 ±  5%      +0.1        0.14 ±  6%  perf-profile.children.cycles-pp.__x64_sys_kill
      0.07 ± 22%      +0.1        0.13 ±  8%  perf-profile.children.cycles-pp.process_one_work
      0.04 ± 58%      +0.1        0.10 ± 11%  perf-profile.children.cycles-pp.__might_fault
      0.06 ± 13%      +0.1        0.12 ± 13%  perf-profile.children.cycles-pp.rcu_cblist_dequeue
      0.01 ±173%      +0.1        0.07 ±  7%  perf-profile.children.cycles-pp.user_path_at_empty
      0.01 ±173%      +0.1        0.07 ± 13%  perf-profile.children.cycles-pp.get_user_arg_ptr
      0.01 ±173%      +0.1        0.07 ± 17%  perf-profile.children.cycles-pp.perf_iterate_sb
      0.00            +0.1        0.05 ±  8%  perf-profile.children.cycles-pp.tcp_sendmsg
      0.00            +0.1        0.05 ±  8%  perf-profile.children.cycles-pp.pagecache_get_page
      0.00            +0.1        0.05 ±  8%  perf-profile.children.cycles-pp.ip_rcv
      0.00            +0.1        0.05 ±  8%  perf-profile.children.cycles-pp.__wake_up_common
      0.00            +0.1        0.05 ±  8%  perf-profile.children.cycles-pp.do_task_dead
      0.00            +0.1        0.05 ±  8%  perf-profile.children.cycles-pp.mark_page_accessed
      0.07 ± 10%      +0.1        0.12 ±  4%  perf-profile.children.cycles-pp.sched_exec
      0.07 ± 12%      +0.1        0.12 ±  6%  perf-profile.children.cycles-pp.__might_sleep
      0.06 ± 13%      +0.1        0.12 ±  3%  perf-profile.children.cycles-pp.d_add
      0.04 ± 57%      +0.1        0.09 ±  4%  perf-profile.children.cycles-pp.schedule_idle
      0.00            +0.1        0.05 ±  9%  perf-profile.children.cycles-pp.find_get_entry
      0.00            +0.1        0.05 ±  9%  perf-profile.children.cycles-pp.put_cred_rcu
      0.00            +0.1        0.05 ±  9%  perf-profile.children.cycles-pp.__list_add_valid
      0.03 ±100%      +0.1        0.08 ±  8%  perf-profile.children.cycles-pp.__pud_alloc
      0.00            +0.1        0.06 ±  9%  perf-profile.children.cycles-pp.__pagevec_lru_add_fn
      0.00            +0.1        0.06 ±  9%  perf-profile.children.cycles-pp.__vma_link_rb
      0.00            +0.1        0.06 ±  9%  perf-profile.children.cycles-pp.memset_erms
      0.00            +0.1        0.06 ±  9%  perf-profile.children.cycles-pp.read_tsc
      0.08 ± 17%      +0.1        0.14 ±  9%  perf-profile.children.cycles-pp.lockref_put_return
      0.07 ±  6%      +0.1        0.12 ±  4%  perf-profile.children.cycles-pp.wait4
      0.08 ±  6%      +0.1        0.13 ±  6%  perf-profile.children.cycles-pp.do_unlinkat
      0.00            +0.1        0.06 ± 14%  perf-profile.children.cycles-pp.up_read
      0.00            +0.1        0.06 ±  7%  perf-profile.children.cycles-pp.__ip_queue_xmit
      0.00            +0.1        0.06 ±  7%  perf-profile.children.cycles-pp.__netif_receive_skb_one_core
      0.00            +0.1        0.06 ±  7%  perf-profile.children.cycles-pp.put_task_stack
      0.00            +0.1        0.06 ±  7%  perf-profile.children.cycles-pp.common_file_perm
      0.00            +0.1        0.06 ±  7%  perf-profile.children.cycles-pp.alloc_bprm
      0.00            +0.1        0.06 ±  7%  perf-profile.children.cycles-pp.___perf_sw_event
      0.00            +0.1        0.06 ±  7%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.03 ±100%      +0.1        0.08 ± 13%  perf-profile.children.cycles-pp.__waitpid
      0.00            +0.1        0.06 ± 14%  perf-profile.children.cycles-pp.native_sched_clock
      0.08 ±  5%      +0.1        0.14 ±  8%  perf-profile.children.cycles-pp.new_sync_read
      0.08 ±  8%      +0.1        0.14 ±  7%  perf-profile.children.cycles-pp.security_file_free
      0.08 ± 14%      +0.1        0.14 ±  6%  perf-profile.children.cycles-pp.apparmor_file_free_security
      0.08 ± 15%      +0.1        0.14 ±  5%  perf-profile.children.cycles-pp.worker_thread
      0.01 ±173%      +0.1        0.07 ± 10%  perf-profile.children.cycles-pp.generic_perform_write
      0.01 ±173%      +0.1        0.07 ± 10%  perf-profile.children.cycles-pp.anon_vma_interval_tree_insert
      0.01 ±173%      +0.1        0.07        perf-profile.children.cycles-pp.setup_arg_pages
      0.01 ±173%      +0.1        0.07 ± 17%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.08 ± 11%      +0.1        0.14 ±  3%  perf-profile.children.cycles-pp.cfree
      0.06 ± 13%      +0.1        0.12 ±  3%  perf-profile.children.cycles-pp.do_truncate
      0.00            +0.1        0.06        perf-profile.children.cycles-pp.process_backlog
      0.00            +0.1        0.06        perf-profile.children.cycles-pp.__x64_sys_munmap
      0.00            +0.1        0.06 ± 11%  perf-profile.children.cycles-pp.__wake_up_common_lock
      0.00            +0.1        0.06        perf-profile.children.cycles-pp.shift_arg_pages
      0.00            +0.1        0.06 ± 11%  perf-profile.children.cycles-pp.sched_clock
      0.04 ± 60%      +0.1        0.10 ±  8%  perf-profile.children.cycles-pp.__pmd_alloc
      0.04 ± 58%      +0.1        0.10 ±  7%  perf-profile.children.cycles-pp.__rb_insert_augmented
      0.03 ±100%      +0.1        0.09 ±  5%  perf-profile.children.cycles-pp.do_softirq
      0.09 ±  5%      +0.1        0.15 ±  7%  perf-profile.children.cycles-pp.ksys_read
      0.08 ±  5%      +0.1        0.14 ±  7%  perf-profile.children.cycles-pp.vfs_read
      0.09 ±  5%      +0.1        0.15 ±  7%  perf-profile.children.cycles-pp.find_vma
      0.08 ± 13%      +0.1        0.14 ± 16%  perf-profile.children.cycles-pp.compar1
      0.01 ±173%      +0.1        0.07 ±  5%  perf-profile.children.cycles-pp.lock_page_memcg
      0.04 ± 59%      +0.1        0.11 ± 21%  perf-profile.children.cycles-pp.get_obj_cgroup_from_current
      0.01 ±173%      +0.1        0.07 ± 11%  perf-profile.children.cycles-pp.vfprintf
      0.04 ± 60%      +0.1        0.10 ±  4%  perf-profile.children.cycles-pp.try_to_wake_up
      0.10 ±  8%      +0.1        0.17 ±  3%  perf-profile.children.cycles-pp.filldir64
      0.10 ±  4%      +0.1        0.16 ±  9%  perf-profile.children.cycles-pp.arch_do_signal
      0.00            +0.1        0.06 ±  6%  perf-profile.children.cycles-pp.inet_recvmsg
      0.00            +0.1        0.06 ±  6%  perf-profile.children.cycles-pp.__memcg_kmem_uncharge_page
      0.00            +0.1        0.06 ±  6%  perf-profile.children.cycles-pp.ttwu_do_activate
      0.00            +0.1        0.06 ±  6%  perf-profile.children.cycles-pp.__legitimize_mnt
      0.00            +0.1        0.06 ± 13%  perf-profile.children.cycles-pp.__put_task_struct
      0.00            +0.1        0.06 ±  6%  perf-profile.children.cycles-pp.mem_cgroup_uncharge_list
      0.08 ± 10%      +0.1        0.15 ±  3%  perf-profile.children.cycles-pp.__get_free_pages
      0.07 ± 11%      +0.1        0.14 ±  6%  perf-profile.children.cycles-pp.up_write
      0.01 ±173%      +0.1        0.08 ± 20%  perf-profile.children.cycles-pp.pgd_alloc
      0.10 ± 14%      +0.1        0.17 ±  5%  perf-profile.children.cycles-pp.memcpy_erms
      0.10 ±  9%      +0.1        0.16 ±  9%  perf-profile.children.cycles-pp.perf_event_mmap
      0.06 ±  7%      +0.1        0.12 ±  6%  perf-profile.children.cycles-pp.security_file_open
      0.06 ±  7%      +0.1        0.12 ±  6%  perf-profile.children.cycles-pp.apparmor_file_open
      0.07 ±  7%      +0.1        0.13 ±  7%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.05            +0.1        0.12 ± 13%  perf-profile.children.cycles-pp.update_sd_lb_stats
      0.00            +0.1        0.07 ±  7%  perf-profile.children.cycles-pp.__tcp_transmit_skb
      0.00            +0.1        0.07 ±  7%  perf-profile.children.cycles-pp.net_rx_action
      0.00            +0.1        0.07 ±  7%  perf-profile.children.cycles-pp.flush_tlb_mm_range
      0.00            +0.1        0.07 ±  7%  perf-profile.children.cycles-pp.get_unmapped_area
      0.00            +0.1        0.07 ±  7%  perf-profile.children.cycles-pp._raw_spin_trylock
      0.00            +0.1        0.07 ±  7%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.08 ±  5%      +0.1        0.14 ±  5%  perf-profile.children.cycles-pp.__pte_alloc
      0.04 ± 58%      +0.1        0.11 ±  7%  perf-profile.children.cycles-pp.pagevec_lru_move_fn
      0.05            +0.1        0.12 ± 11%  perf-profile.children.cycles-pp.find_busiest_group
      0.04 ± 57%      +0.1        0.11 ± 13%  perf-profile.children.cycles-pp.vm_area_alloc
      0.07 ± 13%      +0.1        0.13 ±  3%  perf-profile.children.cycles-pp.page_add_file_rmap
      0.01 ±173%      +0.1        0.08 ±  8%  perf-profile.children.cycles-pp.copy_string_kernel
      0.00            +0.1        0.07 ±  6%  perf-profile.children.cycles-pp.copy_user_generic_unrolled
      0.00            +0.1        0.07 ±  6%  perf-profile.children.cycles-pp.notify_change
      0.00            +0.1        0.07 ± 12%  perf-profile.children.cycles-pp.fput_many
      0.10 ± 11%      +0.1        0.17 ±  3%  perf-profile.children.cycles-pp.wake_up_new_task
      0.07 ± 12%      +0.1        0.14 ±  9%  perf-profile.children.cycles-pp.__mod_memcg_state
      0.09 ±  9%      +0.1        0.16 ±  9%  perf-profile.children.cycles-pp.__dentry_kill
      0.03 ±100%      +0.1        0.10 ±  8%  perf-profile.children.cycles-pp.lru_add_drain_cpu
      0.01 ±173%      +0.1        0.08 ± 10%  perf-profile.children.cycles-pp.__libc_calloc
      0.00            +0.1        0.07        perf-profile.children.cycles-pp.enqueue_entity
      0.00            +0.1        0.07 ± 10%  perf-profile.children.cycles-pp.copy_fpstate_to_sigframe
      0.00            +0.1        0.07 ± 14%  perf-profile.children.cycles-pp.update_rq_clock
      0.08 ± 10%      +0.1        0.15 ±  7%  perf-profile.children.cycles-pp.down_read
      0.11 ±  7%      +0.1        0.18 ±  9%  perf-profile.children.cycles-pp.apparmor_file_alloc_security
      0.07 ± 12%      +0.1        0.14 ±  7%  perf-profile.children.cycles-pp.create_elf_tables
      0.03 ±100%      +0.1        0.10 ±  7%  perf-profile.children.cycles-pp.lru_add_drain
      0.00            +0.1        0.07 ±  5%  perf-profile.children.cycles-pp.sock_read_iter
      0.00            +0.1        0.07 ±  5%  perf-profile.children.cycles-pp.__perf_sw_event
      0.00            +0.1        0.07 ± 15%  perf-profile.children.cycles-pp.do_faccessat
      0.07 ± 26%      +0.1        0.15 ±  8%  perf-profile.children.cycles-pp.serial8250_console_write
      0.01 ±173%      +0.1        0.09 ±  9%  perf-profile.children.cycles-pp.time
      0.00            +0.1        0.08 ±  6%  perf-profile.children.cycles-pp.___slab_alloc
      0.05            +0.1        0.13 ± 11%  perf-profile.children.cycles-pp.pick_next_task_fair
      0.00            +0.1        0.08 ± 19%  perf-profile.children.cycles-pp.__anon_vma_prepare
      0.07 ±  7%      +0.1        0.14 ±  7%  perf-profile.children.cycles-pp.__slab_free
      0.09 ±  7%      +0.1        0.17 ±  4%  perf-profile.children.cycles-pp.strnlen_user
      0.07 ± 25%      +0.1        0.14 ± 11%  perf-profile.children.cycles-pp.uart_console_write
      0.11 ±  7%      +0.1        0.19 ± 11%  perf-profile.children.cycles-pp.strncpy_from_user
      0.05 ± 63%      +0.1        0.13 ± 12%  perf-profile.children.cycles-pp.serial8250_console_putchar
      0.04 ±102%      +0.1        0.12 ±  7%  perf-profile.children.cycles-pp.io_serial_in
      0.00            +0.1        0.08 ±  8%  perf-profile.children.cycles-pp.__put_anon_vma
      0.11 ±  6%      +0.1        0.19 ±  5%  perf-profile.children.cycles-pp.sock_sendmsg
      0.00            +0.1        0.08 ±  8%  perf-profile.children.cycles-pp.file_free_rcu
      0.18 ±  4%      +0.1        0.26 ±  7%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.13 ±  8%      +0.1        0.21 ±  3%  perf-profile.children.cycles-pp.wp_page_copy
      0.10 ± 21%      +0.1        0.18 ±  8%  perf-profile.children.cycles-pp.vprintk_emit
      0.08 ± 26%      +0.1        0.17 ±  7%  perf-profile.children.cycles-pp.console_unlock
      0.11 ±  6%      +0.1        0.19 ±  4%  perf-profile.children.cycles-pp.sock_write_iter
      0.11 ±  7%      +0.1        0.20 ±  5%  perf-profile.children.cycles-pp.__mmap
      0.01 ±173%      +0.1        0.10 ± 23%  perf-profile.children.cycles-pp.path_init
      0.10 ± 13%      +0.1        0.18 ±  8%  perf-profile.children.cycles-pp.__vm_munmap
      0.12 ± 10%      +0.1        0.21 ±  7%  perf-profile.children.cycles-pp.__close
      0.10 ±  5%      +0.1        0.18 ± 10%  perf-profile.children.cycles-pp.do_wait
      0.00            +0.1        0.09 ±  9%  perf-profile.children.cycles-pp.__do_fault
      0.00            +0.1        0.09 ± 31%  perf-profile.children.cycles-pp.rcu_sched_clock_irq
      0.05 ± 63%      +0.1        0.14 ± 12%  perf-profile.children.cycles-pp.wait_for_xmitr
      0.10 ± 10%      +0.1        0.19 ±  4%  perf-profile.children.cycles-pp.pte_alloc_one
      0.07 ±  7%      +0.1        0.15 ± 10%  perf-profile.children.cycles-pp.schedule
      0.15 ± 12%      +0.1        0.23 ±  4%  perf-profile.children.cycles-pp.__memcg_kmem_charge_page
      0.14 ±  8%      +0.1        0.23 ±  8%  perf-profile.children.cycles-pp.security_file_alloc
      0.07 ± 11%      +0.1        0.16 ± 26%  perf-profile.children.cycles-pp.__lock_parent
      0.00            +0.1        0.09 ± 26%  perf-profile.children.cycles-pp.lockref_get
      0.10 ±  4%      +0.1        0.19 ±  9%  perf-profile.children.cycles-pp.__do_sys_wait4
      0.10 ±  4%      +0.1        0.19 ±  9%  perf-profile.children.cycles-pp.kernel_wait4
      0.12 ±  3%      +0.1        0.22 ±  3%  perf-profile.children.cycles-pp.copy_pte_range
      0.10 ± 12%      +0.1        0.20 ±  5%  perf-profile.children.cycles-pp.__mod_memcg_lruvec_state
      0.12 ± 13%      +0.1        0.21 ±  3%  perf-profile.children.cycles-pp.__d_lookup_rcu
      0.10 ±  9%      +0.1        0.19 ±  7%  perf-profile.children.cycles-pp.clear_page_erms
      0.14 ± 10%      +0.1        0.24 ±  7%  perf-profile.children.cycles-pp.obj_cgroup_charge
      0.14 ±  7%      +0.1        0.23 ±  2%  perf-profile.children.cycles-pp.find_idlest_group
      0.15 ±  8%      +0.1        0.25 ±  9%  perf-profile.children.cycles-pp.vma_interval_tree_insert
      0.10 ± 14%      +0.1        0.20 ± 11%  perf-profile.children.cycles-pp.__d_alloc
      0.06 ±  6%      +0.1        0.16 ± 15%  perf-profile.children.cycles-pp.load_balance
      0.26 ±  4%      +0.1        0.35 ±  6%  perf-profile.children.cycles-pp.scheduler_tick
      0.11 ±  8%      +0.1        0.21 ±  5%  perf-profile.children.cycles-pp.prep_new_page
      0.11 ± 14%      +0.1        0.21 ±  5%  perf-profile.children.cycles-pp.down_write
      0.01 ±173%      +0.1        0.11 ± 41%  perf-profile.children.cycles-pp.irq_enter_rcu
      0.11 ± 13%      +0.1        0.21 ±  9%  perf-profile.children.cycles-pp.anon_vma_clone
      0.09 ±  5%      +0.1        0.19 ±  4%  perf-profile.children.cycles-pp.__clear_user
      0.15 ±  5%      +0.1        0.25 ±  9%  perf-profile.children.cycles-pp.getname_flags
      0.15 ±  7%      +0.1        0.25        perf-profile.children.cycles-pp.do_wp_page
      0.11 ±  7%      +0.1        0.21 ±  6%  perf-profile.children.cycles-pp.rwsem_spin_on_owner
      0.14 ± 11%      +0.1        0.24 ±  3%  perf-profile.children.cycles-pp.___might_sleep
      0.17 ± 14%      +0.1        0.28 ± 11%  perf-profile.children.cycles-pp.kill
      0.16 ±  7%      +0.1        0.27 ±  2%  perf-profile.children.cycles-pp.select_task_rq_fair
      0.10 ± 12%      +0.1        0.21 ± 10%  perf-profile.children.cycles-pp.vm_area_dup
      0.13 ±  8%      +0.1        0.25 ±  5%  perf-profile.children.cycles-pp.malloc
      0.00            +0.1        0.11 ± 13%  perf-profile.children.cycles-pp.newidle_balance
      0.19 ± 12%      +0.1        0.30 ±  3%  perf-profile.children.cycles-pp.page_counter_try_charge
      0.12 ±  6%      +0.1        0.24 ±  6%  perf-profile.children.cycles-pp.alloc_set_pte
      0.14 ±  8%      +0.1        0.26 ±  4%  perf-profile.children.cycles-pp.do_anonymous_page
      0.16 ±  6%      +0.1        0.28 ±  4%  perf-profile.children.cycles-pp.copy_strings
      0.17 ±  6%      +0.1        0.30 ± 16%  perf-profile.children.cycles-pp.__fput
      0.21 ± 10%      +0.1        0.34 ±  3%  perf-profile.children.cycles-pp.__memcg_kmem_charge
      0.16 ±  7%      +0.1        0.30 ±  2%  perf-profile.children.cycles-pp.get_page_from_freelist
      0.12 ± 13%      +0.1        0.27 ± 10%  perf-profile.children.cycles-pp.remove_vma
      0.12 ±  5%      +0.1        0.26 ± 11%  perf-profile.children.cycles-pp.do_dentry_open
      0.16 ±  9%      +0.1        0.30 ±  4%  perf-profile.children.cycles-pp.page_remove_rmap
      0.19 ±  5%      +0.1        0.34 ±  4%  perf-profile.children.cycles-pp.copy_page_range
      0.15 ± 14%      +0.1        0.30 ±  8%  perf-profile.children.cycles-pp.anon_vma_fork
      0.17 ± 11%      +0.2        0.33 ±  6%  perf-profile.children.cycles-pp.terminate_walk
      0.20 ±  6%      +0.2        0.35 ±  3%  perf-profile.children.cycles-pp.release_pages
      0.22 ± 15%      +0.2        0.38 ±  5%  perf-profile.children.cycles-pp.write
      0.15 ±  7%      +0.2        0.31 ±  8%  perf-profile.children.cycles-pp.__sched_text_start
      0.17 ±  9%      +0.2        0.34 ±  6%  perf-profile.children.cycles-pp.__xstat64
      0.23 ±  3%      +0.2        0.41 ±  2%  perf-profile.children.cycles-pp.new_sync_write
      0.23 ±  6%      +0.2        0.40 ± 14%  perf-profile.children.cycles-pp.task_work_run
      0.18 ±  8%      +0.2        0.36 ±  5%  perf-profile.children.cycles-pp.load_elf_interp
      0.26 ±  3%      +0.2        0.43 ±  3%  perf-profile.children.cycles-pp.ksys_write
      0.25 ±  2%      +0.2        0.43 ±  2%  perf-profile.children.cycles-pp.vfs_write
      0.16 ± 18%      +0.2        0.34 ± 10%  perf-profile.children.cycles-pp.unlink_anon_vmas
      0.21 ±  6%      +0.2        0.40 ±  4%  perf-profile.children.cycles-pp._dl_addr
      0.26 ± 18%      +0.2        0.44 ±  3%  perf-profile.children.cycles-pp.__memcg_kmem_uncharge
      0.17 ± 18%      +0.2        0.36 ± 10%  perf-profile.children.cycles-pp.unmap_region
      0.23 ±  8%      +0.2        0.42 ±  9%  perf-profile.children.cycles-pp.mprotect_fixup
      0.24 ±  6%      +0.2        0.45 ±  2%  perf-profile.children.cycles-pp.tlb_flush_mmu
      0.24 ±  7%      +0.2        0.44 ±  8%  perf-profile.children.cycles-pp.do_mprotect_pkey
      0.23 ± 10%      +0.2        0.43 ±  8%  perf-profile.children.cycles-pp.d_alloc
      0.24 ±  7%      +0.2        0.44 ±  9%  perf-profile.children.cycles-pp.__x64_sys_mprotect
      0.25 ±  5%      +0.2        0.45 ±  2%  perf-profile.children.cycles-pp.tlb_finish_mmu
      0.22 ± 11%      +0.2        0.43 ±  5%  perf-profile.children.cycles-pp.elf_map
      0.36 ±  2%      +0.2        0.57 ± 13%  perf-profile.children.cycles-pp.tick_sched_handle
      0.36 ±  2%      +0.2        0.57 ± 13%  perf-profile.children.cycles-pp.update_process_times
      0.42            +0.2        0.64 ± 11%  perf-profile.children.cycles-pp.tick_sched_timer
      0.00            +0.2        0.23 ±  4%  perf-profile.children.cycles-pp.cna_order_queue
      0.19 ± 19%      +0.2        0.42 ±  6%  perf-profile.children.cycles-pp.__x64_sys_sync
      0.19 ± 19%      +0.2        0.42 ±  6%  perf-profile.children.cycles-pp.ksys_sync
      0.19 ± 18%      +0.2        0.42 ±  6%  perf-profile.children.cycles-pp.sync
      0.18 ± 20%      +0.2        0.41 ±  7%  perf-profile.children.cycles-pp.iterate_supers
      0.34 ±  5%      +0.2        0.57 ±  9%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
      0.33 ±  4%      +0.2        0.57 ± 10%  perf-profile.children.cycles-pp.exit_to_user_mode_prepare
      0.31 ±  8%      +0.2        0.55        perf-profile.children.cycles-pp.__alloc_pages_nodemask
      0.26 ± 16%      +0.3        0.51 ±  8%  perf-profile.children.cycles-pp.vma_link
      0.29 ±  6%      +0.3        0.54 ±  4%  perf-profile.children.cycles-pp.setlocale
      0.23 ± 11%      +0.3        0.50 ±  5%  perf-profile.children.cycles-pp.lookup_fast
      0.31 ± 15%      +0.3        0.60 ± 13%  perf-profile.children.cycles-pp.__vma_adjust
      0.34 ± 10%      +0.3        0.64 ±  8%  perf-profile.children.cycles-pp.__alloc_file
      0.54 ±  2%      +0.3        0.84 ± 16%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.35 ±  9%      +0.3        0.65 ±  8%  perf-profile.children.cycles-pp.alloc_empty_file
      0.35 ± 14%      +0.3        0.67 ± 12%  perf-profile.children.cycles-pp.__split_vma
      0.47 ±  6%      +0.3        0.80 ±  6%  perf-profile.children.cycles-pp.zap_pte_range
      0.40 ±  7%      +0.3        0.74 ±  6%  perf-profile.children.cycles-pp.d_alloc_parallel
      0.50 ±  7%      +0.4        0.85 ±  6%  perf-profile.children.cycles-pp.unmap_page_range
      0.37 ± 17%      +0.4        0.73 ± 10%  perf-profile.children.cycles-pp.__do_munmap
      0.85 ±  4%      +0.4        1.22 ± 12%  perf-profile.children.cycles-pp.hrtimer_interrupt
      0.53 ±  7%      +0.4        0.90 ±  6%  perf-profile.children.cycles-pp.unmap_vmas
      0.43 ± 18%      +0.4        0.80        perf-profile.children.cycles-pp.page_counter_cancel
      0.45 ±  7%      +0.4        0.82 ±  4%  perf-profile.children.cycles-pp.__lookup_slow
      0.86 ±  4%      +0.4        1.24 ± 12%  perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
      0.44 ± 18%      +0.4        0.82        perf-profile.children.cycles-pp.page_counter_uncharge
      0.37 ± 27%      +0.4        0.77 ± 13%  perf-profile.children.cycles-pp.unlink_file_vma
      0.48 ± 11%      +0.4        0.89 ±  8%  perf-profile.children.cycles-pp.__libc_fork
      0.51 ±  9%      +0.4        0.94 ±  9%  perf-profile.children.cycles-pp.mem_rtns_1
      0.49 ±  9%      +0.4        0.92 ±  3%  perf-profile.children.cycles-pp.path_lookupat
      0.50 ±  8%      +0.4        0.94 ±  3%  perf-profile.children.cycles-pp.filename_lookup
      0.70 ± 14%      +0.4        1.15 ±  6%  perf-profile.children.cycles-pp.div_short
      0.53 ±  8%      +0.5        0.99 ±  3%  perf-profile.children.cycles-pp.__do_sys_newstat
      0.53 ±  8%      +0.5        0.98 ±  4%  perf-profile.children.cycles-pp.vfs_statx
      0.61 ± 13%      +0.5        1.07 ±  6%  perf-profile.children.cycles-pp.div_int
      0.42 ±  2%      +0.5        0.88 ± 10%  perf-profile.children.cycles-pp.lockref_put_or_lock
      0.60 ±  6%      +0.5        1.08 ±  3%  perf-profile.children.cycles-pp.filemap_map_pages
      0.44 ± 20%      +0.5        0.98 ±  5%  perf-profile.children.cycles-pp.drain_obj_stock
      0.68 ± 12%      +0.6        1.23 ±  7%  perf-profile.children.cycles-pp.add_short
      0.61 ± 15%      +0.6        1.18 ± 11%  perf-profile.children.cycles-pp.ksys_mmap_pgoff
      0.55 ± 13%      +0.6        1.13 ± 13%  perf-profile.children.cycles-pp.kmem_cache_alloc
      0.69 ±  6%      +0.6        1.28 ±  3%  perf-profile.children.cycles-pp.do_fault
      0.61 ± 14%      +0.6        1.20 ±  5%  perf-profile.children.cycles-pp.creat64
      0.65 ±  7%      +0.6        1.25 ±  6%  perf-profile.children.cycles-pp.begin_new_exec
      0.63 ± 12%      +0.6        1.23 ±  7%  perf-profile.children.cycles-pp.dup_mmap
      0.57 ± 18%      +0.6        1.19 ±  9%  perf-profile.children.cycles-pp.free_pgtables
      0.66 ± 12%      +0.6        1.29 ±  7%  perf-profile.children.cycles-pp.dup_mm
      0.68 ± 14%      +0.7        1.34 ± 10%  perf-profile.children.cycles-pp.mmap_region
      0.73 ± 13%      +0.7        1.42 ±  9%  perf-profile.children.cycles-pp.do_mmap
      0.77 ± 12%      +0.7        1.48 ±  9%  perf-profile.children.cycles-pp.vm_mmap_pgoff
      0.89 ±  3%      +0.7        1.60 ±  8%  perf-profile.children.cycles-pp.dput
      0.53 ± 20%      +0.7        1.26 ±  6%  perf-profile.children.cycles-pp.refill_obj_stock
      0.55 ± 36%      +0.7        1.29 ± 16%  perf-profile.children.cycles-pp.osq_lock
      0.91 ±  8%      +0.8        1.68 ±  5%  perf-profile.children.cycles-pp.do_exit
      0.91 ±  8%      +0.8        1.69 ±  5%  perf-profile.children.cycles-pp.__x64_sys_exit_group
      0.91 ±  8%      +0.8        1.69 ±  5%  perf-profile.children.cycles-pp.do_group_exit
      0.90 ±  8%      +0.8        1.68 ±  7%  perf-profile.children.cycles-pp.__strcat_chk
      0.43 ± 19%      +0.8        1.22 ± 13%  perf-profile.children.cycles-pp.irq_exit_rcu
      0.85 ± 10%      +0.8        1.64 ±  7%  perf-profile.children.cycles-pp.copy_process
      0.46 ± 18%      +0.8        1.26 ± 13%  perf-profile.children.cycles-pp.do_softirq_own_stack
      0.95 ±  7%      +0.8        1.79 ±  5%  perf-profile.children.cycles-pp.link_path_walk
      0.94 ±  7%      +0.8        1.78 ±  5%  perf-profile.children.cycles-pp.walk_component
      0.96 ± 10%      +0.9        1.82 ±  7%  perf-profile.children.cycles-pp.__do_sys_clone
      0.96 ± 10%      +0.9        1.82 ±  7%  perf-profile.children.cycles-pp._do_fork
      0.69 ± 30%      +0.9        1.56 ± 13%  perf-profile.children.cycles-pp.rwsem_optimistic_spin
      1.08 ±  5%      +0.9        1.97 ±  2%  perf-profile.children.cycles-pp.__handle_mm_fault
      0.71 ± 29%      +0.9        1.59 ± 14%  perf-profile.children.cycles-pp.rwsem_down_write_slowpath
      1.14 ±  5%      +0.9        2.08 ±  2%  perf-profile.children.cycles-pp.handle_mm_fault
      1.24 ±  5%      +1.0        2.25 ±  2%  perf-profile.children.cycles-pp.do_user_addr_fault
      1.41 ± 13%      +1.0        2.42 ±  5%  perf-profile.children.cycles-pp.div_long
      1.25 ±  5%      +1.0        2.26 ±  2%  perf-profile.children.cycles-pp.exc_page_fault
      1.15 ±  8%      +1.1        2.21 ±  5%  perf-profile.children.cycles-pp.load_elf_binary
      1.32 ±  5%      +1.1        2.39 ±  2%  perf-profile.children.cycles-pp.asm_exc_page_fault
      1.16 ±  8%      +1.1        2.24 ±  5%  perf-profile.children.cycles-pp.exec_binprm
      1.52 ± 13%      +1.1        2.60 ±  6%  perf-profile.children.cycles-pp.add_long
      1.52 ± 13%      +1.1        2.63 ±  7%  perf-profile.children.cycles-pp.add_int
      1.22 ±  9%      +1.1        2.33 ±  6%  perf-profile.children.cycles-pp.asm_call_on_stack
      1.34 ±  8%      +1.2        2.52 ±  6%  perf-profile.children.cycles-pp.exit_mmap
      1.34 ±  8%      +1.2        2.53 ±  6%  perf-profile.children.cycles-pp.mmput
      1.65 ±  8%      +1.2        2.89 ±  5%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
      1.40 ±  8%      +1.3        2.66 ±  7%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
      0.65 ± 26%      +1.3        1.96 ± 22%  perf-profile.children.cycles-pp.run_ksoftirqd
      0.66 ± 26%      +1.3        1.99 ± 21%  perf-profile.children.cycles-pp.smpboot_thread_fn
      1.65 ±  8%      +1.3        3.00 ±  6%  perf-profile.children.cycles-pp.__strncat_chk
      0.74 ± 24%      +1.4        2.13 ± 20%  perf-profile.children.cycles-pp.kthread
      0.78 ± 23%      +1.4        2.18 ± 20%  perf-profile.children.cycles-pp.ret_from_fork
      0.97 ± 24%      +2.0        2.97 ± 20%  perf-profile.children.cycles-pp.rcu_do_batch
      0.98 ± 25%      +2.0        2.99 ± 20%  perf-profile.children.cycles-pp.rcu_core
      1.10 ± 23%      +2.1        3.22 ± 18%  perf-profile.children.cycles-pp.__softirqentry_text_start
      1.07 ± 25%      +2.1        3.19 ± 21%  perf-profile.children.cycles-pp.kmem_cache_free
      5.45 ±  9%      +4.4        9.88 ±  6%  perf-profile.children.cycles-pp.string_rtns_1
      0.22 ±  5%      +5.9        6.09 ±  7%  perf-profile.children.cycles-pp.__open64_nocancel
      4.49 ± 12%      +6.0       10.45 ±  4%  perf-profile.children.cycles-pp.intel_idle
      5.59 ± 11%      +6.8       12.35 ±  4%  perf-profile.children.cycles-pp.cpuidle_enter_state
      5.59 ± 11%      +6.8       12.35 ±  4%  perf-profile.children.cycles-pp.cpuidle_enter
      6.10 ± 10%      +6.8       12.93 ±  5%  perf-profile.children.cycles-pp.start_secondary
      0.00            +6.9        6.88 ± 12%  perf-profile.children.cycles-pp.complete_walk
      6.17 ± 10%      +6.9       13.07 ±  4%  perf-profile.children.cycles-pp.secondary_startup_64
      6.17 ± 10%      +6.9       13.07 ±  4%  perf-profile.children.cycles-pp.cpu_startup_entry
      6.17 ± 10%      +6.9       13.07 ±  4%  perf-profile.children.cycles-pp.do_idle
      1.79 ±  7%      +8.5       10.31 ±  8%  perf-profile.children.cycles-pp.do_sys_openat2
      1.80 ±  7%      +8.5       10.32 ±  8%  perf-profile.children.cycles-pp.do_sys_open
      1.51 ±  5%     +16.3       17.80 ±  6%  perf-profile.children.cycles-pp.execve
      0.12 ±  6%     +16.4       16.57 ±  5%  perf-profile.children.cycles-pp.pick_link
      0.16 ±  5%     +16.8       16.96 ±  5%  perf-profile.children.cycles-pp.do_open_execat
      0.37 ±  5%     +17.0       17.34 ±  5%  perf-profile.children.cycles-pp.step_into
      1.40 ±  7%     +17.9       19.33 ±  4%  perf-profile.children.cycles-pp.bprm_execve
      1.66 ±  7%     +18.1       19.80 ±  3%  perf-profile.children.cycles-pp.do_execveat_common
      1.68 ±  7%     +18.2       19.83 ±  3%  perf-profile.children.cycles-pp.__x64_sys_execve
      0.26 ±  5%     +23.5       23.73 ±  6%  perf-profile.children.cycles-pp.lockref_get_not_dead
      0.28 ±  7%     +23.5       23.77 ±  6%  perf-profile.children.cycles-pp.__legitimize_path
      0.29 ±  8%     +23.5       23.79 ±  6%  perf-profile.children.cycles-pp.unlazy_walk
      1.80 ±  6%     +25.2       26.99 ±  5%  perf-profile.children.cycles-pp.path_openat
      1.81 ±  6%     +25.2       27.00 ±  5%  perf-profile.children.cycles-pp.do_filp_open
      0.00           +38.1       38.12 ±  5%  perf-profile.children.cycles-pp.__cna_queued_spin_lock_slowpath
     66.22 ±  2%     -66.2        0.00        perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      0.13 ±  3%      -0.1        0.04 ± 58%  perf-profile.self.cycles-pp.scan_positives
      0.08 ±  5%      +0.0        0.10 ±  7%  perf-profile.self.cycles-pp.update_cfs_group
      0.07 ± 10%      +0.0        0.10 ± 11%  perf-profile.self.cycles-pp.ktime_get_update_offsets_now
      0.08 ± 10%      +0.0        0.11 ±  9%  perf-profile.self.cycles-pp.vma_interval_tree_remove
      0.06 ±  9%      +0.0        0.10 ± 11%  perf-profile.self.cycles-pp.sort_rtns_1
      0.06            +0.0        0.10        perf-profile.self.cycles-pp.copy_pte_range
      0.04 ± 57%      +0.0        0.08 ±  5%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.05 ±  8%      +0.0        0.10 ±  5%  perf-profile.self.cycles-pp.unlock_page
      0.06 ±  6%      +0.0        0.11 ± 10%  perf-profile.self.cycles-pp.__handle_mm_fault
      0.05 ±  8%      +0.0        0.10 ±  4%  perf-profile.self.cycles-pp.d_alloc_parallel
      0.04 ± 57%      +0.0        0.08 ± 10%  perf-profile.self.cycles-pp.find_vma
      0.05            +0.0        0.10 ± 11%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.06 ± 11%      +0.0        0.11 ±  8%  perf-profile.self.cycles-pp.link_path_walk
      0.04 ± 57%      +0.0        0.08 ±  5%  perf-profile.self.cycles-pp.copy_page
      0.08 ±  5%      +0.0        0.12 ±  4%  perf-profile.self.cycles-pp.filldir64
      0.06 ±  6%      +0.0        0.11 ±  6%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.09 ± 26%      +0.0        0.14 ±  8%  perf-profile.self.cycles-pp.propagate_protected_usage
      0.04 ± 58%      +0.0        0.09 ±  7%  perf-profile.self.cycles-pp.__rb_insert_augmented
      0.04 ± 57%      +0.1        0.09 ±  4%  perf-profile.self.cycles-pp.aa_get_task_label
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.handle_mm_fault
      0.04 ± 58%      +0.1        0.09 ± 14%  perf-profile.self.cycles-pp.obj_cgroup_charge
      0.06 ±  7%      +0.1        0.11 ±  7%  perf-profile.self.cycles-pp.__might_sleep
      0.07 ±  7%      +0.1        0.12 ±  7%  perf-profile.self.cycles-pp.sync_regs
      0.06 ± 13%      +0.1        0.12 ± 13%  perf-profile.self.cycles-pp.rcu_cblist_dequeue
      0.01 ±173%      +0.1        0.07 ± 17%  perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      0.01 ±173%      +0.1        0.07 ±  7%  perf-profile.self.cycles-pp.strncpy_from_user
      0.01 ±173%      +0.1        0.07 ±  7%  perf-profile.self.cycles-pp.compar2
      0.00            +0.1        0.05 ±  8%  perf-profile.self.cycles-pp.mmap_region
      0.00            +0.1        0.05 ±  8%  perf-profile.self.cycles-pp.__mod_memcg_lruvec_state
      0.00            +0.1        0.05 ±  8%  perf-profile.self.cycles-pp.copy_user_generic_unrolled
      0.00            +0.1        0.05 ±  8%  perf-profile.self.cycles-pp.common_file_perm
      0.00            +0.1        0.05 ±  8%  perf-profile.self.cycles-pp.__vma_link_rb
      0.00            +0.1        0.05 ±  8%  perf-profile.self.cycles-pp.memset_erms
      0.00            +0.1        0.05 ±  8%  perf-profile.self.cycles-pp.native_sched_clock
      0.00            +0.1        0.05 ±  8%  perf-profile.self.cycles-pp.__list_add_valid
      0.06 ±  7%      +0.1        0.11 ±  3%  perf-profile.self.cycles-pp.apparmor_file_open
      0.00            +0.1        0.05 ±  9%  perf-profile.self.cycles-pp.up_read
      0.07 ± 14%      +0.1        0.13 ±  5%  perf-profile.self.cycles-pp.apparmor_file_free_security
      0.08 ± 17%      +0.1        0.14 ±  9%  perf-profile.self.cycles-pp.lockref_put_return
      0.04 ± 57%      +0.1        0.09 ±  8%  perf-profile.self.cycles-pp.free_pages_and_swap_cache
      0.04 ± 57%      +0.1        0.10 ± 10%  perf-profile.self.cycles-pp.cfree
      0.07 ± 11%      +0.1        0.13 ±  5%  perf-profile.self.cycles-pp.down_read
      0.00            +0.1        0.06 ±  7%  perf-profile.self.cycles-pp.update_load_avg
      0.00            +0.1        0.06 ±  7%  perf-profile.self.cycles-pp.try_charge
      0.00            +0.1        0.06 ±  7%  perf-profile.self.cycles-pp.inode_permission
      0.08 ± 13%      +0.1        0.14 ± 19%  perf-profile.self.cycles-pp.compar1
      0.01 ±173%      +0.1        0.07 ± 10%  perf-profile.self.cycles-pp.anon_vma_interval_tree_insert
      0.01 ±173%      +0.1        0.07        perf-profile.self.cycles-pp.lock_page_memcg
      0.00            +0.1        0.06        perf-profile.self.cycles-pp.__legitimize_mnt
      0.00            +0.1        0.06 ± 16%  perf-profile.self.cycles-pp.vm_area_dup
      0.07 ± 11%      +0.1        0.14 ±  3%  perf-profile.self.cycles-pp.up_write
      0.08 ± 10%      +0.1        0.14 ±  5%  perf-profile.self.cycles-pp.strnlen_user
      0.00            +0.1        0.07 ±  7%  perf-profile.self.cycles-pp.entry_SYSCALL_64
      0.00            +0.1        0.07 ±  7%  perf-profile.self.cycles-pp._raw_spin_trylock
      0.10 ± 15%      +0.1        0.17 ±  5%  perf-profile.self.cycles-pp.memcpy_erms
      0.03 ±100%      +0.1        0.09 ±  4%  perf-profile.self.cycles-pp.page_add_file_rmap
      0.07 ±  7%      +0.1        0.14 ± 11%  perf-profile.self.cycles-pp.__slab_free
      0.07 ± 12%      +0.1        0.14 ±  9%  perf-profile.self.cycles-pp.__mod_memcg_state
      0.10 ± 10%      +0.1        0.18 ±  8%  perf-profile.self.cycles-pp.apparmor_file_alloc_security
      0.00            +0.1        0.07 ± 15%  perf-profile.self.cycles-pp.file_free_rcu
      0.01 ±173%      +0.1        0.09 ±  9%  perf-profile.self.cycles-pp.__alloc_file
      0.00            +0.1        0.08 ± 20%  perf-profile.self.cycles-pp.alloc_set_pte
      0.09 ± 14%      +0.1        0.17 ±  8%  perf-profile.self.cycles-pp.down_write
      0.00            +0.1        0.08 ± 10%  perf-profile.self.cycles-pp.__libc_calloc
      0.04 ±102%      +0.1        0.12 ±  7%  perf-profile.self.cycles-pp.io_serial_in
      0.12 ±  6%      +0.1        0.20 ±  2%  perf-profile.self.cycles-pp.find_idlest_group
      0.18 ±  4%      +0.1        0.26 ±  7%  perf-profile.self.cycles-pp.native_irq_return_iret
      0.00            +0.1        0.08 ± 27%  perf-profile.self.cycles-pp.rcu_sched_clock_irq
      0.15 ±  8%      +0.1        0.24 ± 10%  perf-profile.self.cycles-pp.vma_interval_tree_insert
      0.00            +0.1        0.09 ±  7%  perf-profile.self.cycles-pp.update_sd_lb_stats
      0.11 ± 14%      +0.1        0.21 ±  5%  perf-profile.self.cycles-pp.__d_lookup_rcu
      0.10 ±  9%      +0.1        0.19 ±  5%  perf-profile.self.cycles-pp.clear_page_erms
      0.11 ±  6%      +0.1        0.20 ±  4%  perf-profile.self.cycles-pp.release_pages
      0.11 ±  7%      +0.1        0.21 ±  6%  perf-profile.self.cycles-pp.rwsem_spin_on_owner
      0.13 ±  9%      +0.1        0.23 ±  4%  perf-profile.self.cycles-pp.___might_sleep
      0.18 ± 10%      +0.1        0.28 ±  4%  perf-profile.self.cycles-pp.page_counter_try_charge
      0.12 ±  5%      +0.1        0.23 ±  6%  perf-profile.self.cycles-pp.malloc
      0.11 ±  9%      +0.1        0.23 ±  4%  perf-profile.self.cycles-pp.lockref_get_not_dead
      0.12 ±  8%      +0.1        0.24 ±  4%  perf-profile.self.cycles-pp.page_remove_rmap
      0.24 ±  6%      +0.1        0.39 ±  5%  perf-profile.self.cycles-pp.zap_pte_range
      0.21 ±  7%      +0.2        0.39 ±  4%  perf-profile.self.cycles-pp._dl_addr
      0.09 ± 26%      +0.2        0.27 ± 12%  perf-profile.self.cycles-pp.refill_obj_stock
      0.06 ± 62%      +0.2        0.26 ± 24%  perf-profile.self.cycles-pp.drain_obj_stock
      0.00            +0.2        0.22 ±  5%  perf-profile.self.cycles-pp.cna_order_queue
      0.41 ±  7%      +0.3        0.70 ±  3%  perf-profile.self.cycles-pp.filemap_map_pages
      0.22 ± 22%      +0.3        0.54 ± 20%  perf-profile.self.cycles-pp.kmem_cache_alloc
      0.35 ± 18%      +0.3        0.68        perf-profile.self.cycles-pp.page_counter_cancel
      0.86 ±  3%      +0.3        1.20 ±  2%  perf-profile.self.cycles-pp._raw_spin_lock
      0.50 ±  9%      +0.4        0.92 ±  8%  perf-profile.self.cycles-pp.mem_rtns_1
      0.69 ± 14%      +0.4        1.14 ±  6%  perf-profile.self.cycles-pp.div_short
      0.61 ± 13%      +0.5        1.06 ±  6%  perf-profile.self.cycles-pp.div_int
      0.67 ± 13%      +0.5        1.22 ±  7%  perf-profile.self.cycles-pp.add_short
      0.54 ± 35%      +0.7        1.23 ± 15%  perf-profile.self.cycles-pp.osq_lock
      0.89 ±  8%      +0.8        1.65 ±  7%  perf-profile.self.cycles-pp.__strcat_chk
      1.40 ± 13%      +1.0        2.38 ±  6%  perf-profile.self.cycles-pp.div_long
      1.51 ± 12%      +1.1        2.56 ±  5%  perf-profile.self.cycles-pp.add_long
      1.50 ± 13%      +1.1        2.60 ±  7%  perf-profile.self.cycles-pp.add_int
      0.43 ± 51%      +1.3        1.69 ± 34%  perf-profile.self.cycles-pp.kmem_cache_free
      1.64 ±  8%      +1.3        2.95 ±  6%  perf-profile.self.cycles-pp.__strncat_chk
      5.38 ±  9%      +4.3        9.70 ±  6%  perf-profile.self.cycles-pp.string_rtns_1
      4.49 ± 12%      +6.0       10.45 ±  4%  perf-profile.self.cycles-pp.intel_idle
      0.00           +37.5       37.52 ±  5%  perf-profile.self.cycles-pp.__cna_queued_spin_lock_slowpath


                                                                                
                               reaim.time.user_time                             
                                                                                
  3100 +--------------------------------------------------------------------+   
       |                                                                    |   
  3000 |-O              O                   O                        O O  O |   
       |              O           O                                         |   
  2900 |-+  O O  O O       O O  O    O O  O   O  O O  O O  O O  O O         |   
       |                                                                    |   
  2800 |-+                                                                  |   
       |                                                                    |   
  2700 |-+                                                                  |   
       |                                                                    |   
  2600 |-+                                             .+..                 |   
       |.+..+.+..+.+..+.+..+.+..+.+.. .+.. .+        .+    +    +.+..+.+    |   
  2500 |-+                           +    +  +  .+.+.       + ..            |   
       |                                      +.             +              |   
  2400 +--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                              reaim.time.system_time                            
                                                                                
  9000 +--------------------------------------------------------------------+   
  8500 |-+                                                   +..            |   
       |                             +    +. .+..+.         +       .+.+    |   
  8000 |.+..+.+..+.+.. .+..+.+..+. .. + ..  +      +..+.  .+    +.+.        |   
  7500 |-+            +           +    +                +.                  |   
       |                                                                    |   
  7000 |-+                                                                  |   
  6500 |-+                                                                  |   
  6000 |-+                                                                  |   
       |                                                                    |   
  5500 |-+  O                                    O         O                |   
  5000 |-O    O  O O  O O  O O  O O  O O  O O O    O  O O    O  O O         |   
       |                                                                    |   
  4500 |-+                                                                  |   
  4000 +--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                      reaim.time.percent_of_cpu_this_job_got                    
                                                                                
  3800 +--------------------------------------------------------------------+   
       |                                                     +..            |   
  3600 |-+                          .+.  .+. .+..+.         +       .+.+    |   
  3400 |.+..+.+..+.+..+.+..+.+..+.+.   +.   +      +..+.  .+    +.+.        |   
       |                                                +.                  |   
  3200 |-+                                                                  |   
       |                                                                    |   
  3000 |-+                                                                  |   
       |                                                                    |   
  2800 |-O                                                                  |   
  2600 |-+  O O  O O  O O  O O  O O  O O  O O O  O O  O O  O O  O O         |   
       |                                                                    |   
  2400 |-+                                                                  |   
       |                                                             O O  O |   
  2200 +--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                            reaim.time.major_page_faults                        
                                                                                
  200000 +------------------------------------------------------------------+   
         |                                                           O O  O |   
  180000 |-+                                                                |   
  160000 |-+                                                                |   
         |                                                                  |   
  140000 |-+                     O                                          |   
         |                     O      O O    O           O        O         |   
  120000 |-+                       O       O   O      O         O           |   
         |             O                          O O      O  O             |   
  100000 |-O    O  O O    O O                                               |   
   80000 |-+  O                                                             |   
         |                                                                  |   
   60000 |.+..      .+.         .+.                                         |   
         |    +.+..+   +..+.+..+   +..+.+..+.+.  .+.+.+..+.+..+.+.+..+.+    |   
   40000 +------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                             reaim.time.minor_page_faults                       
                                                                                
  9.8e+07 +-----------------------------------------------------------------+   
          |             O           O                                O O  O |   
  9.6e+07 |-O  O O O  O    O O O  O   O  O O O  O O O  O O  O O O  O        |   
  9.4e+07 |-+                                                               |   
          |                                                                 |   
  9.2e+07 |-+                                                               |   
    9e+07 |-+                                                               |   
          |                                                                 |   
  8.8e+07 |-+                                                               |   
  8.6e+07 |-+                                            +..                |   
          |.+.. .+.  .+.+..+.+.  .+.+.  .+. .+..        +       +..         |   
  8.4e+07 |-+  +   +.          +.     +.   +     .+.+..+    +  :   +.+.+    |   
  8.2e+07 |-+                                   +            + :            |   
          |                                                   +             |   
    8e+07 +-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                 reaim.parent_time                              
                                                                                
  1.6 +---------------------------------------------------------------------+   
      |                                                                     |   
  1.5 |-+                                                    +              |   
      |                             +..  +..  +..+.         + +             |   
  1.4 |..+.+..+.+..+.       .+.    +    +    +     +..+.  .+   + .+..+.+    |   
      |              +..+.+.   +..+    +    +           +.      +           |   
  1.3 |-+                                                                   |   
      |                                                                     |   
  1.2 |-+                                                                   |   
      |                                                                     |   
  1.1 |-+                                                                   |   
      |                                                                     |   
    1 |-+O O  O O  O    O O  O O    O  O O  O O  O O  O O  O O  O O         |   
      |              O            O                                  O O  O |   
  0.9 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                reaim.child_systime                             
                                                                                
  110 +---------------------------------------------------------------------+   
      |                                                                     |   
  100 |-+                                     +..           .+..            |   
   90 |-+ .+.. .+..         .+.    .+.. .+.. +   +.  .+    +     .+..+.+    |   
      |..+    +    +.+..+.+.   +..+    +    +      +.  + ..     +           |   
   80 |-+                                               +                   |   
      |                                                                     |   
   70 |-+                                                                   |   
      |                                                                     |   
   60 |-+                                                                   |   
   50 |-+O O  O O  O O  O O  O O    O  O O  O O  O O  O    O O  O           |   
      |                           O                     O         O         |   
   40 |-+                                                            O O  O |   
      |                                                                     |   
   30 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                 reaim.jobs_per_min                             
                                                                                
  650000 +------------------------------------------------------------------+   
         |                                                                  |   
  600000 |-+           O           O                                 O O  O |   
         |      O  O      O O  O O    O O    O O  O O O  O O  O O O         |   
         | O  O      O                     O                                |   
  550000 |-+                                                                |   
         |                                                                  |   
  500000 |-+                                                                |   
         |                                                                  |   
  450000 |-+                                                                |   
         |            .+..+.      .+.. .+.. .+          .+.                 |   
         |.+..+.+..+.+      +..+.+    +    +  +    .+.+.   +.. .+.+..+.+    |   
  400000 |-+                                   +..+           +             |   
         |                                                                  |   
  350000 +------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                             reaim.jobs_per_min_child                           
                                                                                
  6500 +--------------------------------------------------------------------+   
       |                          O                                  O O  O |   
       |         O    O      O  O             O    O    O    O    O         |   
  6000 |-O  O O    O    O  O         O O  O O    O    O    O    O           |   
       |                                                                    |   
       |                                                                    |   
  5500 |-+                                                                  |   
       |                                                                    |   
  5000 |-+                                                                  |   
       |                                                                    |   
       |                                                                    |   
  4500 |-+           .+.+..      .+..  +..             .+..                 |   
       |.+..+.+..+.+.      +.+..+     +    .+.    .+..+    +    +.+.. .+    |   
       |                             +    +   +..+          + ..     +      |   
  4000 +--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                               reaim.max_jobs_per_min                           
                                                                                
  700000 +------------------------------------------------------------------+   
         |                                                                  |   
  650000 |-+       O   O           O                O         O      O O  O |   
         |                O    O O      O      O  O      O O      O         |   
  600000 |-O  O O    O      O         O    O O        O         O           |   
         |                                                                  |   
  550000 |-+                                                                |   
         |                                                                  |   
  500000 |-+                                                                |   
         |.                                                                 |   
  450000 |-+..        .+..+.+..   .+.. .+.. .+     .+.  .+.+..  +.          |   
         |    +.+..+.+         +.+    +    +  +  .+   +.       +  +..+.+    |   
  400000 |-+                                   +.             +             |   
         |                                                                  |   
  350000 +------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                   reaim.workload                               
                                                                                
  980000 +------------------------------------------------------------------+   
         |             O           O                                 O O  O |   
  960000 |-O  O O  O O    O O  O O    O O  O O O  O O O  O O  O O O         |   
  940000 |-+                                                                |   
         |                                                                  |   
  920000 |-+                                                                |   
  900000 |-+                                                                |   
         |                                                                  |   
  880000 |-+                                                                |   
  860000 |-+                                             +                  |   
         |.+.. .+.. .+.+..+.+.. .+.+.. .+.. .+         .. +     +.          |   
  840000 |-+  +    +           +      +    +  +  .+.+.+    +.. :  +..+.+    |   
  820000 |-+                                   +.              :            |   
         |                                                    +             |   
  800000 +------------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.9.0-rc2-00051-g5df52a74ab0091" of type "text/plain" (170245 bytes)

View attachment "job-script" of type "text/plain" (7751 bytes)

View attachment "job.yaml" of type "text/plain" (5302 bytes)

View attachment "reproduce" of type "text/plain" (10445 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ