lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date:   Mon, 22 May 2023 16:28:31 +0800
From:   kernel test robot <oliver.sang@...el.com>
To:     Wyes Karny <wyes.karny@....com>
CC:     <oe-lkp@...ts.linux.dev>, <lkp@...el.com>,
        <linux-pm@...r.kernel.org>, <ying.huang@...el.com>,
        <feng.tang@...el.com>, <fengwei.yin@...el.com>,
        <ray.huang@....com>, <rafael@...nel.org>,
        <viresh.kumar@...aro.org>, <linux-kernel@...r.kernel.org>,
        <gautham.shenoy@....com>, Wyes Karny <wyes.karny@....com>,
        <oliver.sang@...el.com>
Subject: Re: [PATCH v4 3/3] cpufreq: Return failure if fast_switch is not set
 and fast_switch_possible is set



Hello,

kernel test robot noticed a 302.3% regression of phoronix-test-suite.glibc-bench.sin.nanoseconds on:


commit: 278dfbd17c5d0767834471d7fea71d3127c3ecfb ("[PATCH v4 3/3] cpufreq: Return failure if fast_switch is not set and fast_switch_possible is set")
url: https://github.com/intel-lab-lkp/linux/commits/Wyes-Karny/amd_pstate-Add-fast_switch-callback/20230518-010448
base: https://git.kernel.org/cgit/linux/kernel/git/rafael/linux-pm.git linux-next
patch link: https://lore.kernel.org/all/20230517162817.8538-4-wyes.karny@amd.com/
patch subject: [PATCH v4 3/3] cpufreq: Return failure if fast_switch is not set and fast_switch_possible is set

testcase: phoronix-test-suite
test machine: 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz (Cascade Lake) with 512G memory
parameters:

	test: glibc-bench-1.6.1
	option_a: sin
	cpufreq_governor: performance

test-description: The Phoronix Test Suite is the most comprehensive testing and benchmarking platform available that provides an extensible framework for which new tests can be easily added.
test-url: http://www.phoronix-test-suite.com/

In addition to that, the commit also has significant impact on the following tests:

+------------------+-----------------------------------------------------------------------------------------------------------------------------------------+
| testcase: change | phoronix-test-suite: phoronix-test-suite.n-queens.0.seconds 313.6% regression                                                           |
| test machine     | 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz (Cascade Lake) with 512G memory                                           |
| test parameters  | cpufreq_governor=performance                                                                                                            |
|                  | test=n-queens-1.2.1                                                                                                                     |
+------------------+-----------------------------------------------------------------------------------------------------------------------------------------+
| testcase: change | filebench: filebench.sum_operations/s -72.1% regression                                                                                 |
| test machine     | 96 threads 2 sockets (Ice Lake) with 256G memory                                                                                        |
| test parameters  | cpufreq_governor=performance                                                                                                            |
|                  | disk=1HDD                                                                                                                               |
|                  | fs=xfs                                                                                                                                  |
|                  | test=cvar_example.f                                                                                                                     |
+------------------+-----------------------------------------------------------------------------------------------------------------------------------------+
| testcase: change | phoronix-test-suite: phoronix-test-suite.glibc-bench.ffs.nanoseconds 301.2% regression                                                  |
| test machine     | 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz (Cascade Lake) with 512G memory                                           |
| test parameters  | cpufreq_governor=performance                                                                                                            |
|                  | option_a=ffs                                                                                                                            |
|                  | test=glibc-bench-1.6.1                                                                                                                  |
+------------------+-----------------------------------------------------------------------------------------------------------------------------------------+
| testcase: change | phoronix-test-suite: phoronix-test-suite.glibc-bench.pthread_once.nanoseconds 300.8% regression                                                                                   |
| test machine     | 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz (Cascade Lake) with 512G memory                                           |
| test parameters  | cpufreq_governor=performance                                                                                                            |
|                  | option_a=pthread_once                                                                                                                   |
|                  | test=glibc-bench-1.6.1                                                                                                                  |
+------------------+-----------------------------------------------------------------------------------------------------------------------------------------+
| testcase: change | phoronix-test-suite: phoronix-test-suite.cyclictest.clock_nanosleepTIME_ABSTIME,Interval500ms,100000Loops.ms_average 5550.0% regression |
| test machine     | 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz (Cascade Lake) with 512G memory                                           |
| test parameters  | cpufreq_governor=performance                                                                                                            |
|                  | option_a=3                                                                                                                              |
|                  | test=cyclictest-1.0.0                                                                                                                   |
+------------------+-----------------------------------------------------------------------------------------------------------------------------------------+
| testcase: change | phoronix-test-suite: phoronix-test-suite.astcenc.Medium.seconds 106.6% regression                                                       |
| test machine     | 16 threads 1 sockets Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz (Coffee Lake) with 32G memory                                               |
| test parameters  | cpufreq_governor=performance                                                                                                            |
|                  | option_a=Medium                                                                                                                         |
|                  | test=astcenc-1.1.0                                                                                                                      |
+------------------+-----------------------------------------------------------------------------------------------------------------------------------------+
| testcase: change | phoronix-test-suite: phoronix-test-suite.pyperformance.pickle_pure_python.milliseconds -99.4% regression                                |
| test machine     | 16 threads 1 sockets Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz (Coffee Lake) with 32G memory                                               |
| test parameters  | cpufreq_governor=performance                                                                                                            |
|                  | option_a=pickle_pure_python                                                                                                             |
|                  | test=pyperformance-1.0.2                                                                                                                |
+------------------+-----------------------------------------------------------------------------------------------------------------------------------------+
| testcase: change | phoronix-test-suite: phoronix-test-suite.x11perf.500pxCopyFromWindowToWindow.operations___second -7.3% regression                       |
| test machine     | 12 threads 1 sockets Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz (Coffee Lake) with 32G memory                                              |
| test parameters  | cpufreq_governor=performance                                                                                                            |
|                  | need_x=true                                                                                                                             |
|                  | option_a=500px Copy From Window To Window                                                                                               |
|                  | test=x11perf-1.1.1                                                                                                                      |
+------------------+-----------------------------------------------------------------------------------------------------------------------------------------+


If you fix the issue, kindly add following tag
| Reported-by: kernel test robot <oliver.sang@...el.com>
| Closes: https://lore.kernel.org/oe-lkp/202305221612.200d8fb1-oliver.sang@intel.com


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        sudo bin/lkp install job.yaml           # job file is attached in this email
        bin/lkp split-job --compatible job.yaml # generate the yaml file for lkp run
        sudo bin/lkp run generated-yaml-file

        # if come across any failure that blocks the test,
        # please remove ~/.lkp and /lkp dir to run from a clean state.

=========================================================================================
compiler/cpufreq_governor/kconfig/option_a/rootfs/tbox_group/test/testcase:
  gcc-11/performance/x86_64-rhel-8.3/sin/debian-x86_64-phoronix/lkp-csl-2sp7/glibc-bench-1.6.1/phoronix-test-suite

commit: 
  a525b95417 ("cpufreq/amd-pstate: Remove fast_switch_possible flag from active driver")
  278dfbd17c ("cpufreq: Return failure if fast_switch is not set and fast_switch_possible is set")

a525b954173e74cd 278dfbd17c5d0767834471d7fea 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :6          100%           6:6     kmsg.cpufreq:fast_switch_possible_is_enabled_but_fast_switch_callback_is_not_set
         %stddev     %change         %stddev
             \          |                \  
     72.00          +302.3%     289.69        phoronix-test-suite.glibc-bench.sin.nanoseconds
     86.29           +14.5%      98.84        phoronix-test-suite.time.elapsed_time
     86.29           +14.5%      98.84        phoronix-test-suite.time.elapsed_time.max
    307.00 ± 15%    +397.0%       1525 ±  3%  phoronix-test-suite.time.involuntary_context_switches
     74.33            +5.6%      78.50        phoronix-test-suite.time.percent_of_cpu_this_job_got
     61.97           +12.2%      69.52        phoronix-test-suite.time.user_time
      8020 ±  3%     +32.1%      10593        meminfo.Active(anon)
 8.199e+09            +9.8%  9.003e+09        cpuidle..time
   8533649           +12.1%    9564836        cpuidle..usage
     80285 ±  5%     -39.2%      48852 ± 28%  numa-meminfo.node0.AnonHugePages
     14184 ± 93%    +360.9%      65375 ± 24%  numa-meminfo.node1.AnonHugePages
    144.48 ±  5%     +13.8%     164.46        uptime.boot
     12935 ±  5%     +11.6%      14439        uptime.idle
      1.00          +100.0%       2.00        vmstat.procs.r
      2346          +108.3%       4888        vmstat.system.cs
     95898            -2.3%      93730        vmstat.system.in
      0.04 ±  8%      -0.0        0.03 ±  4%  mpstat.cpu.all.iowait%
      1.64            +2.0        3.59        mpstat.cpu.all.irq%
      0.09 ±  3%      +0.1        0.21 ±  2%  mpstat.cpu.all.soft%
      0.09            +0.2        0.28        mpstat.cpu.all.sys%
      0.90            +0.8        1.68        mpstat.cpu.all.usr%
      2004 ±  3%     +32.1%       2648        proc-vmstat.nr_active_anon
    110098            -4.0%     105659        proc-vmstat.nr_anon_pages
    286491            -1.6%     281968        proc-vmstat.nr_inactive_anon
     52653            -1.1%      52049        proc-vmstat.nr_slab_unreclaimable
      2004 ±  3%     +32.1%       2648        proc-vmstat.nr_zone_active_anon
    286491            -1.6%     281968        proc-vmstat.nr_zone_inactive_anon
    513152            +8.7%     557738        proc-vmstat.numa_hit
    494207            +9.0%     538756        proc-vmstat.numa_local
    592220            +9.6%     649301        proc-vmstat.pgalloc_normal
    681984 ±  2%     +10.6%     754452        proc-vmstat.pgfault
    492147 ±  2%     +10.6%     544510        proc-vmstat.pgfree
     89356 ±  2%     +16.7%     104272        proc-vmstat.pgreuse
    835328           +13.6%     948992        proc-vmstat.unevictable_pgs_scanned
      4.17            +4.7        8.90        turbostat.Busy%
      1607           -56.4%     700.00        turbostat.Bzy_MHz
     59849 ± 12%    +264.1%     217899 ±  7%  turbostat.C1E
      0.23 ±  4%      +0.1        0.30 ±  2%  turbostat.C1E%
   8348392           +10.9%    9254503        turbostat.C6
     71.41           -21.0%      56.44        turbostat.CPU%c1
     24.42 ±  2%     +41.9%      34.65        turbostat.CPU%c6
     51.50 ±  3%     -25.9%      38.17        turbostat.CoreTmp
   8545229           +11.7%    9542613        turbostat.IRQ
     35868 ±  6%     -96.2%       1375 ± 38%  turbostat.POLL
     18.83 ±  5%     -57.2%       8.06 ±  3%  turbostat.Pkg%pc2
      2.57 ±  7%     -74.2%       0.66 ±  4%  turbostat.Pkg%pc6
     51.67 ±  2%     -25.5%      38.50        turbostat.PkgTmp
    121.11           -42.2%      70.00        turbostat.PkgWatt
     53.52 ± 18%     -16.0       37.52 ± 22%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
     53.52 ± 18%     -16.0       37.52 ± 22%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
     10.74 ±103%      -8.3        2.42 ±155%  perf-profile.calltrace.cycles-pp.unmap_vmas.exit_mmap.__mmput.exit_mm.do_exit
     10.74 ±103%      -8.3        2.42 ±155%  perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.exit_mmap.__mmput.exit_mm
      8.15 ±116%      -6.9        1.26 ±145%  perf-profile.calltrace.cycles-pp.asm_exc_page_fault.fault_in_readable.fault_in_iov_iter_readable.generic_perform_write.__generic_file_write_iter
     10.74 ±103%      -6.6        4.09 ±178%  perf-profile.calltrace.cycles-pp.zap_pmd_range.unmap_page_range.unmap_vmas.exit_mmap.__mmput
     10.74 ±103%      -6.6        4.09 ±178%  perf-profile.calltrace.cycles-pp.zap_pte_range.zap_pmd_range.unmap_page_range.unmap_vmas.exit_mmap
      4.26 ±105%      -4.3        0.00        perf-profile.calltrace.cycles-pp.page_remove_rmap.zap_pte_range.zap_pmd_range.unmap_page_range.unmap_vmas
      4.26 ± 84%      -3.8        0.50 ±223%  perf-profile.calltrace.cycles-pp.__do_fault.do_read_fault.do_fault.__handle_mm_fault.handle_mm_fault
      5.19 ± 85%      -3.0        2.17 ±169%  perf-profile.calltrace.cycles-pp.do_fault.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault
      5.19 ± 85%      -3.0        2.17 ±169%  perf-profile.calltrace.cycles-pp.do_read_fault.do_fault.__handle_mm_fault.handle_mm_fault.do_user_addr_fault
     53.52 ± 18%     -16.0       37.52 ± 22%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     53.52 ± 18%     -16.0       37.52 ± 22%  perf-profile.children.cycles-pp.do_syscall_64
     10.74 ±103%      -6.6        4.09 ±178%  perf-profile.children.cycles-pp.unmap_vmas
     10.74 ±103%      -6.6        4.09 ±178%  perf-profile.children.cycles-pp.unmap_page_range
     10.74 ±103%      -6.6        4.09 ±178%  perf-profile.children.cycles-pp.zap_pmd_range
     10.74 ±103%      -6.6        4.09 ±178%  perf-profile.children.cycles-pp.zap_pte_range
      4.26 ±105%      -4.3        0.00        perf-profile.children.cycles-pp.page_remove_rmap
      4.26 ± 84%      -3.8        0.50 ±223%  perf-profile.children.cycles-pp.__do_fault
      5.18 ± 85%      -3.0        2.17 ±169%  perf-profile.children.cycles-pp.do_fault
      5.18 ± 85%      -3.0        2.17 ±169%  perf-profile.children.cycles-pp.do_read_fault
      2862 ± 20%   +5653.9%     164728 ±213%  sched_debug.cfs_rq:/.load.avg
     47107 ± 16%  +32534.6%   15373401 ±219%  sched_debug.cfs_rq:/.load.max
      9468 ± 12%  +16445.2%    1566546 ±218%  sched_debug.cfs_rq:/.load.stddev
     58.57 ± 33%     -58.9%      24.09 ± 46%  sched_debug.cfs_rq:/.load_avg.avg
    203.27 ± 18%     -53.0%      95.47 ± 50%  sched_debug.cfs_rq:/.load_avg.stddev
     37231 ± 14%     +51.4%      56368 ±  5%  sched_debug.cfs_rq:/.min_vruntime.max
     39.58 ± 51%     -77.9%       8.76 ±126%  sched_debug.cfs_rq:/.removed.load_avg.avg
    187.92 ± 25%     -66.1%      63.64 ±106%  sched_debug.cfs_rq:/.removed.load_avg.stddev
     18.16 ± 57%     -77.1%       4.16 ±123%  sched_debug.cfs_rq:/.removed.runnable_avg.avg
     18.16 ± 57%     -77.1%       4.16 ±123%  sched_debug.cfs_rq:/.removed.util_avg.avg
    214.61 ± 14%     -37.7%     133.71 ±  6%  sched_debug.cfs_rq:/.runnable_avg.avg
    248.38 ±  6%     -17.4%     205.07 ±  4%  sched_debug.cfs_rq:/.runnable_avg.stddev
    214.53 ± 14%     -37.7%     133.59 ±  6%  sched_debug.cfs_rq:/.util_avg.avg
    248.36 ±  6%     -17.4%     205.03 ±  4%  sched_debug.cfs_rq:/.util_avg.stddev
    715.50 ±  6%     -41.8%     416.50 ± 25%  sched_debug.cfs_rq:/.util_est_enqueued.max
     99.21 ±  6%     -31.7%      67.78 ± 24%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
   2382316 ± 46%     -49.6%    1201480 ± 12%  sched_debug.cpu.avg_idle.max
     36009           +21.1%      43615 ±  2%  sched_debug.cpu.clock_task.min
    117475 ± 75%     -88.0%      14136 ± 82%  sched_debug.cpu.max_idle_balance_cost.stddev
      0.00 ± 20%    +130.0%       0.00 ± 63%  sched_debug.cpu.next_balance.stddev
     52309           +15.6%      60475        sched_debug.sched_clk
     25.76 ±  4%     +48.2%      38.19        perf-stat.i.MPKI
 7.905e+08           -45.2%   4.33e+08        perf-stat.i.branch-instructions
      3.18 ±  2%      +2.8        5.96        perf-stat.i.branch-miss-rate%
  15210221           +64.8%   25060061        perf-stat.i.branch-misses
   2046023           +68.1%    3438514        perf-stat.i.cache-misses
  49258879           +69.3%   83372455        perf-stat.i.cache-references
      1997          +130.4%       4601        perf-stat.i.context-switches
      2.07 ±  2%     +12.5%       2.32        perf-stat.i.cpi
 5.568e+09            -6.3%  5.218e+09        perf-stat.i.cpu-cycles
      3074           -48.9%       1571        perf-stat.i.cycles-between-cache-misses
      0.31 ±  5%      +0.2        0.52        perf-stat.i.dTLB-load-miss-rate%
   1469231           +70.0%    2497301        perf-stat.i.dTLB-load-misses
 1.117e+09           -52.2%  5.343e+08        perf-stat.i.dTLB-loads
      0.06 ±  5%      +0.0        0.11        perf-stat.i.dTLB-store-miss-rate%
    140944 ±  3%     +63.8%     230910        perf-stat.i.dTLB-store-misses
 4.405e+08           -46.6%  2.354e+08        perf-stat.i.dTLB-stores
     61.61           +12.2       73.84        perf-stat.i.iTLB-load-miss-rate%
    799474            -3.9%     768539        perf-stat.i.iTLB-load-misses
    499584 ±  2%     -45.3%     273344 ±  3%  perf-stat.i.iTLB-loads
 5.589e+09           -55.2%  2.504e+09        perf-stat.i.instructions
      7134           -54.0%       3281        perf-stat.i.instructions-per-iTLB-miss
      1.00           -52.6%       0.47        perf-stat.i.ipc
      2.07 ±  4%     -58.4%       0.86 ±  6%  perf-stat.i.major-faults
      0.06            -6.3%       0.05        perf-stat.i.metric.GHz
    618.20           +40.6%     869.20        perf-stat.i.metric.K/sec
     24.35           -48.5%      12.53        perf-stat.i.metric.M/sec
     83.55            -3.9       79.68        perf-stat.i.node-load-miss-rate%
     92330 ±  2%     +36.8%     126293 ±  3%  perf-stat.i.node-load-misses
     38082 ±  2%     +17.1%      44601 ±  3%  perf-stat.i.node-loads
     42624 ± 14%     +41.2%      60176 ±  9%  perf-stat.i.node-store-misses
      8.82          +277.4%      33.29        perf-stat.overall.MPKI
      1.92            +3.9        5.79        perf-stat.overall.branch-miss-rate%
      1.00          +109.0%       2.08        perf-stat.overall.cpi
      2722 ±  2%     -44.3%       1517        perf-stat.overall.cycles-between-cache-misses
      0.13            +0.3        0.47        perf-stat.overall.dTLB-load-miss-rate%
      0.03 ±  3%      +0.1        0.10        perf-stat.overall.dTLB-store-miss-rate%
     61.55           +12.2       73.77        perf-stat.overall.iTLB-load-miss-rate%
      6987           -53.4%       3258        perf-stat.overall.instructions-per-iTLB-miss
      1.00           -52.2%       0.48        perf-stat.overall.ipc
     70.79            +3.1       73.87        perf-stat.overall.node-load-miss-rate%
     44.13 ± 10%      +7.7       51.80 ±  8%  perf-stat.overall.node-store-miss-rate%
  7.81e+08           -45.1%  4.286e+08        perf-stat.ps.branch-instructions
  15031718           +65.0%   24799436        perf-stat.ps.branch-misses
   2022310           +68.3%    3402668        perf-stat.ps.cache-misses
  48693838           +69.4%   82502207        perf-stat.ps.cache-references
      1974          +130.7%       4554        perf-stat.ps.context-switches
 5.503e+09            -6.2%  5.164e+09        perf-stat.ps.cpu-cycles
   1452373           +70.1%    2471118        perf-stat.ps.dTLB-load-misses
 1.103e+09           -52.1%  5.288e+08        perf-stat.ps.dTLB-loads
    139332 ±  3%     +64.0%     228525        perf-stat.ps.dTLB-store-misses
 4.353e+08           -46.5%   2.33e+08        perf-stat.ps.dTLB-stores
    790174            -3.7%     760575        perf-stat.ps.iTLB-load-misses
    493698 ±  2%     -45.2%     270481 ±  3%  perf-stat.ps.iTLB-loads
 5.521e+09           -55.1%  2.479e+09        perf-stat.ps.instructions
      2.04 ±  4%     -58.2%       0.85 ±  6%  perf-stat.ps.major-faults
     91256 ±  2%     +37.0%     125023 ±  3%  perf-stat.ps.node-load-misses
     37653 ±  2%     +17.3%      44171 ±  3%  perf-stat.ps.node-loads
     42127 ± 14%     +41.4%      59560 ±  9%  perf-stat.ps.node-store-misses
 4.781e+11           -49.8%  2.401e+11        perf-stat.total.instructions


***************************************************************************************************
lkp-csl-2sp7: 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz (Cascade Lake) with 512G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/rootfs/tbox_group/test/testcase:
  gcc-11/performance/x86_64-rhel-8.3/debian-x86_64-phoronix/lkp-csl-2sp7/n-queens-1.2.1/phoronix-test-suite

commit: 
  a525b95417 ("cpufreq/amd-pstate: Remove fast_switch_possible flag from active driver")
  278dfbd17c ("cpufreq: Return failure if fast_switch is not set and fast_switch_possible is set")

a525b954173e74cd 278dfbd17c5d0767834471d7fea 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :6          100%           6:6     kmsg.cpufreq:fast_switch_possible_is_enabled_but_fast_switch_callback_is_not_set
         %stddev     %change         %stddev
             \          |                \  
      4.14          +313.6%      17.10        phoronix-test-suite.n-queens.0.seconds
     35.10 ±  2%    +128.7%      80.28        phoronix-test-suite.time.elapsed_time
     35.10 ±  2%    +128.7%      80.28        phoronix-test-suite.time.elapsed_time.max
     28850            -3.0%      27994 ±  2%  phoronix-test-suite.time.maximum_resident_set_size
  2.27e+09 ±  4%     +22.0%  2.769e+09        cpuidle..time
   2439080 ±  3%     +16.5%    2841419        cpuidle..usage
      7803 ±  2%     +30.3%      10166        meminfo.Active(anon)
   2128118           +13.6%    2416662        meminfo.Committed_AS
      3883 ±  8%     -16.6%       3240 ±  9%  numa-meminfo.node0.PageTables
      9073 ±  5%     +15.6%      10489 ±  2%  numa-meminfo.node1.KernelStack
      1929 ± 14%     +49.6%       2885 ±  9%  numa-meminfo.node1.PageTables
    971.33 ±  8%     -16.6%     810.00 ±  9%  numa-vmstat.node0.nr_page_table_pages
      9074 ±  5%     +15.7%      10494 ±  3%  numa-vmstat.node1.nr_kernel_stack
    482.00 ± 14%     +49.8%     721.83 ±  9%  numa-vmstat.node1.nr_page_table_pages
     63.98           -28.9       35.05        mpstat.cpu.all.idle%
      0.09 ±  5%      -0.1        0.04        mpstat.cpu.all.iowait%
      1.64            +1.2        2.84        mpstat.cpu.all.irq%
      0.09 ±  4%      +0.0        0.11        mpstat.cpu.all.soft%
      0.13 ±  2%      +0.2        0.29        mpstat.cpu.all.sys%
     34.08 ±  2%     +27.6       61.67        mpstat.cpu.all.usr%
     65.33           -44.9%      36.00        vmstat.cpu.id
     31.67 ±  2%     +86.8%      59.17        vmstat.cpu.us
      2345 ±  3%     -54.4%       1069 ±  3%  vmstat.io.bi
     31.00           +94.1%      60.17        vmstat.procs.r
      2937           +67.8%       4927        vmstat.system.cs
     96735            -2.6%      94264        vmstat.system.in
    980.50 ±  2%     -53.7%     454.00        turbostat.Avg_MHz
     37.06 ±  2%     +28.7       65.74        turbostat.Busy%
      2671           -73.8%     700.00        turbostat.Bzy_MHz
     64543 ± 21%     -70.0%      19387 ±  7%  turbostat.C1
      0.05 ± 22%      -0.0        0.01        turbostat.C1%
     36674 ± 10%    +197.1%     108951 ± 11%  turbostat.C1E
      0.30 ±  6%      -0.1        0.17 ±  9%  turbostat.C1E%
   2305719 ±  4%     +16.6%    2687774        turbostat.C6
     63.05           -28.7       34.39        turbostat.C6%
     43.38           -53.5%      20.16        turbostat.CPU%c1
     19.56           -27.9%      14.11        turbostat.CPU%c6
     51.00 ±  3%     -24.2%      38.67        turbostat.CoreTmp
   3714942 ±  2%    +113.2%    7921708        turbostat.IRQ
     13286 ±  2%     -96.2%     509.50 ± 24%  turbostat.POLL
     10.25           -67.9%       3.30 ± 11%  turbostat.Pkg%pc2
      5.70 ±  5%     -97.2%       0.16 ±  5%  turbostat.Pkg%pc6
     50.83 ±  4%     -23.0%      39.17        turbostat.PkgTmp
    165.02           -49.5%      83.38        turbostat.PkgWatt
    139.21            +3.3%     143.74        turbostat.RAMWatt
      1950 ±  2%     +30.2%       2539        proc-vmstat.nr_active_anon
     26828            +3.9%      27867        proc-vmstat.nr_active_file
    107483            -4.1%     103038        proc-vmstat.nr_anon_pages
    283850            -1.6%     279280        proc-vmstat.nr_inactive_anon
     20564 ±  2%      +3.8%      21344        proc-vmstat.nr_kernel_stack
      1452            +5.1%       1526        proc-vmstat.nr_page_table_pages
     25654            +1.0%      25919        proc-vmstat.nr_slab_reclaimable
      1950 ±  2%     +30.2%       2539        proc-vmstat.nr_zone_active_anon
     26828            +3.9%      27867        proc-vmstat.nr_zone_active_file
    283850            -1.6%     279280        proc-vmstat.nr_zone_inactive_anon
    390716           +25.2%     489291        proc-vmstat.numa_hit
    371740           +26.5%     470277        proc-vmstat.numa_local
      6451 ± 56%     -90.3%     628.33 ± 59%  proc-vmstat.numa_pages_migrated
     21981 ± 24%     -75.9%       5294 ± 67%  proc-vmstat.numa_pte_updates
    461444           +24.3%     573395        proc-vmstat.pgalloc_normal
    484259           +33.7%     647486        proc-vmstat.pgfault
    361816           +29.2%     467634        proc-vmstat.pgfree
      6451 ± 56%     -90.3%     628.33 ± 59%  proc-vmstat.pgmigrate_success
     58553           +50.8%      88315 ±  2%  proc-vmstat.pgreuse
    335104          +102.6%     678912        proc-vmstat.unevictable_pgs_scanned
      2379 ± 34%    +159.9%       6184 ± 32%  sched_debug.cfs_rq:/.load.avg
      8773 ± 34%    +156.3%      22488 ± 53%  sched_debug.cfs_rq:/.load.stddev
     59.18 ± 31%     -59.2%      24.17 ± 32%  sched_debug.cfs_rq:/.load_avg.avg
      1020 ±  3%     -19.2%     824.58 ± 25%  sched_debug.cfs_rq:/.load_avg.max
    201.36 ± 21%     -49.2%     102.31 ± 30%  sched_debug.cfs_rq:/.load_avg.stddev
      4306 ± 17%   +1283.4%      59568 ±128%  sched_debug.cfs_rq:/.min_vruntime.stddev
     39.17 ± 50%     -80.4%       7.68 ± 95%  sched_debug.cfs_rq:/.removed.load_avg.avg
    185.53 ± 28%     -65.2%      64.59 ± 83%  sched_debug.cfs_rq:/.removed.load_avg.stddev
     17.61 ± 45%     -78.3%       3.82 ± 96%  sched_debug.cfs_rq:/.removed.runnable_avg.avg
     86.06 ± 23%     -62.7%      32.11 ± 84%  sched_debug.cfs_rq:/.removed.runnable_avg.stddev
     17.61 ± 45%     -78.3%       3.82 ± 96%  sched_debug.cfs_rq:/.removed.util_avg.avg
     86.06 ± 23%     -62.7%      32.11 ± 84%  sched_debug.cfs_rq:/.removed.util_avg.stddev
    225.01 ± 11%     -44.8%     124.22 ± 24%  sched_debug.cfs_rq:/.runnable_avg.avg
    253.39 ±  5%     -21.5%     198.91 ± 13%  sched_debug.cfs_rq:/.runnable_avg.stddev
      4306 ± 17%   +1283.1%      59563 ±128%  sched_debug.cfs_rq:/.spread0.stddev
    224.86 ± 11%     -44.8%     124.11 ± 24%  sched_debug.cfs_rq:/.util_avg.avg
    253.32 ±  5%     -21.5%     198.83 ± 13%  sched_debug.cfs_rq:/.util_avg.stddev
    783.83 ± 14%     -42.9%     447.67 ± 49%  sched_debug.cfs_rq:/.util_est_enqueued.max
     96.58 ± 11%     -32.0%      65.67 ± 33%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
      3.45 ± 29%    +151.3%       8.67 ±  5%  sched_debug.cpu.clock.stddev
     36256           +46.5%      53120 ± 26%  sched_debug.cpu.clock_task.min
    140.27 ± 23%     +57.5%     220.89 ±  8%  sched_debug.cpu.curr->pid.avg
    525.18 ± 13%     +27.1%     667.27 ±  3%  sched_debug.cpu.curr->pid.stddev
      0.00 ±  9%     +85.4%       0.00 ± 17%  sched_debug.cpu.next_balance.stddev
      0.06 ± 21%     +59.5%       0.10 ± 13%  sched_debug.cpu.nr_running.avg
     52700           +32.7%      69957 ± 20%  sched_debug.sched_clk
     14.93 ± 37%     -14.9        0.00        perf-profile.calltrace.cycles-pp.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe
     14.93 ± 37%     -14.9        0.00        perf-profile.calltrace.cycles-pp.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe
      7.40 ± 56%      -7.4        0.00        perf-profile.calltrace.cycles-pp.bprm_execve.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe
      7.40 ± 56%      -7.4        0.00        perf-profile.calltrace.cycles-pp.exec_binprm.bprm_execve.do_execveat_common.__x64_sys_execve.do_syscall_64
      7.40 ± 56%      -7.4        0.00        perf-profile.calltrace.cycles-pp.search_binary_handler.exec_binprm.bprm_execve.do_execveat_common.__x64_sys_execve
      7.40 ± 56%      -7.4        0.00        perf-profile.calltrace.cycles-pp.load_elf_binary.search_binary_handler.exec_binprm.bprm_execve.do_execveat_common
      5.05 ±105%      -5.0        0.00        perf-profile.calltrace.cycles-pp.__mmput.exec_mmap.begin_new_exec.load_elf_binary.search_binary_handler
      5.05 ±105%      -5.0        0.00        perf-profile.calltrace.cycles-pp.exit_mmap.__mmput.exec_mmap.begin_new_exec.load_elf_binary
      5.05 ±105%      -5.0        0.00        perf-profile.calltrace.cycles-pp.begin_new_exec.load_elf_binary.search_binary_handler.exec_binprm.bprm_execve
      5.05 ±105%      -5.0        0.00        perf-profile.calltrace.cycles-pp.exec_mmap.begin_new_exec.load_elf_binary.search_binary_handler.exec_binprm
      5.05 ±105%      -5.0        0.00        perf-profile.calltrace.cycles-pp.free_pgtables.exit_mmap.__mmput.exec_mmap.begin_new_exec
      0.00            +6.2        6.21 ± 52%  perf-profile.calltrace.cycles-pp.perf_release.__fput.task_work_run.do_exit.do_group_exit
      0.00            +6.2        6.21 ± 52%  perf-profile.calltrace.cycles-pp.perf_event_release_kernel.perf_release.__fput.task_work_run.do_exit
      0.00            +6.9        6.90 ± 46%  perf-profile.calltrace.cycles-pp.task_work_run.do_exit.do_group_exit.get_signal.arch_do_signal_or_restart
      0.00            +6.9        6.90 ± 46%  perf-profile.calltrace.cycles-pp.__fput.task_work_run.do_exit.do_group_exit.get_signal
     14.93 ± 37%     -14.9        0.00        perf-profile.children.cycles-pp.__x64_sys_execve
     14.93 ± 37%     -14.9        0.00        perf-profile.children.cycles-pp.do_execveat_common
      7.40 ± 56%      -7.4        0.00        perf-profile.children.cycles-pp.bprm_execve
      7.40 ± 56%      -7.4        0.00        perf-profile.children.cycles-pp.exec_binprm
      7.40 ± 56%      -7.4        0.00        perf-profile.children.cycles-pp.search_binary_handler
      7.40 ± 56%      -7.4        0.00        perf-profile.children.cycles-pp.load_elf_binary
      5.05 ±105%      -5.0        0.00        perf-profile.children.cycles-pp.begin_new_exec
      5.05 ±105%      -5.0        0.00        perf-profile.children.cycles-pp.exec_mmap
      5.88 ± 82%      -4.8        1.11 ±223%  perf-profile.children.cycles-pp.free_pgtables
      0.00            +6.2        6.21 ± 52%  perf-profile.children.cycles-pp.perf_release
      0.00            +6.2        6.21 ± 52%  perf-profile.children.cycles-pp.perf_event_release_kernel
      0.83 ±223%      +6.9        7.74 ± 41%  perf-profile.children.cycles-pp.task_work_run
      0.83 ±223%      +6.9        7.74 ± 41%  perf-profile.children.cycles-pp.__fput
      6.71 ± 75%      -6.0        0.70 ±223%  perf-profile.self.cycles-pp.zap_pte_range
     43.58 ±  5%     -71.0%      12.62 ±  5%  perf-stat.i.MPKI
 8.111e+09 ±  2%     -54.1%  3.721e+09        perf-stat.i.branch-instructions
     14.40            +1.9       16.33        perf-stat.i.branch-miss-rate%
 1.658e+09 ±  2%     -55.4%  7.401e+08        perf-stat.i.branch-misses
      7.61            +2.2        9.79 ±  4%  perf-stat.i.cache-miss-rate%
   2374185           -18.4%    1936172 ±  2%  perf-stat.i.cache-misses
  45065640           -25.1%   33764433        perf-stat.i.cache-references
      2358           +97.5%       4657        perf-stat.i.context-switches
      3.44 ±  3%     -58.6%       1.42 ±  2%  perf-stat.i.cpi
 9.928e+10           -54.8%  4.488e+10        perf-stat.i.cpu-cycles
    125.68 ±  2%     +12.2%     141.01        perf-stat.i.cpu-migrations
    157340 ±  3%     -56.5%      68500 ±  5%  perf-stat.i.cycles-between-cache-misses
      0.47 ±  5%      -0.3        0.17 ±  4%  perf-stat.i.dTLB-load-miss-rate%
   1300850           -25.3%     972044        perf-stat.i.dTLB-load-misses
 1.072e+10 ±  2%     -54.6%  4.867e+09        perf-stat.i.dTLB-loads
      0.08 ±  5%      -0.0        0.04 ±  2%  perf-stat.i.dTLB-store-miss-rate%
    200676            +9.5%     219785        perf-stat.i.dTLB-store-misses
 1.048e+10 ±  2%     -55.3%  4.684e+09        perf-stat.i.dTLB-stores
     77.21            -6.7       70.51        perf-stat.i.iTLB-load-miss-rate%
    835438 ±  7%     -34.9%     544154 ±  3%  perf-stat.i.iTLB-load-misses
    281802           -22.1%     219530        perf-stat.i.iTLB-loads
 1.023e+11 ±  2%     -55.3%  4.576e+10        perf-stat.i.instructions
      0.70 ±  2%     +21.8%       0.85        perf-stat.i.ipc
      5.05 ±  3%     -78.3%       1.10 ±  5%  perf-stat.i.major-faults
      1.03           -54.9%       0.47        perf-stat.i.metric.GHz
    667.47 ±  3%     -46.4%     357.69        perf-stat.i.metric.K/sec
    305.23 ±  2%     -54.7%     138.14        perf-stat.i.metric.M/sec
      6910           -28.6%       4935        perf-stat.i.minor-faults
    143333 ±  4%     -11.2%     127226 ±  3%  perf-stat.i.node-load-misses
     70783 ±  6%     -34.1%      46661 ±  4%  perf-stat.i.node-loads
     59.90 ±  4%      +7.5       67.39        perf-stat.i.node-store-miss-rate%
     78522 ± 14%     -46.0%      42369 ±  3%  perf-stat.i.node-stores
      6915           -28.6%       4936        perf-stat.i.page-faults
      0.44 ±  3%     +60.7%       0.71        perf-stat.overall.MPKI
     20.44            -0.5       19.92        perf-stat.overall.branch-miss-rate%
      5.27 ±  2%      +0.5        5.76 ±  2%  perf-stat.overall.cache-miss-rate%
      0.97            +1.0%       0.98        perf-stat.overall.cpi
     41833 ±  2%     -42.6%      24012 ±  3%  perf-stat.overall.cycles-between-cache-misses
      0.01 ±  3%      +0.0        0.02        perf-stat.overall.dTLB-load-miss-rate%
      0.00 ±  2%      +0.0        0.00        perf-stat.overall.dTLB-store-miss-rate%
     74.71            -3.5       71.17        perf-stat.overall.iTLB-load-miss-rate%
    123027 ±  5%     -29.9%      86229 ±  3%  perf-stat.overall.instructions-per-iTLB-miss
      1.03            -1.0%       1.02        perf-stat.overall.ipc
     66.94 ±  2%      +6.4       73.36        perf-stat.overall.node-load-miss-rate%
     51.71 ±  8%     +11.2       62.86        perf-stat.overall.node-store-miss-rate%
 7.879e+09           -52.8%  3.722e+09        perf-stat.ps.branch-instructions
  1.61e+09           -54.0%  7.415e+08        perf-stat.ps.branch-misses
   2305884           -18.8%    1872089 ±  2%  perf-stat.ps.cache-misses
  43800806           -25.8%   32479193        perf-stat.ps.cache-references
      2291          +100.8%       4600        perf-stat.ps.context-switches
     93316            +1.6%      94785        perf-stat.ps.cpu-clock
 9.645e+10           -53.4%  4.491e+10        perf-stat.ps.cpu-cycles
    122.10           +14.4%     139.72        perf-stat.ps.cpu-migrations
   1264505           -26.1%     934112        perf-stat.ps.dTLB-load-misses
 1.041e+10           -53.2%  4.871e+09        perf-stat.ps.dTLB-loads
    194993           +11.1%     216552        perf-stat.ps.dTLB-store-misses
 1.018e+10           -53.9%  4.692e+09        perf-stat.ps.dTLB-stores
    811464 ±  7%     -34.4%     532304 ±  3%  perf-stat.ps.iTLB-load-misses
    273639           -21.3%     215398        perf-stat.ps.iTLB-loads
  9.94e+10           -53.9%  4.583e+10        perf-stat.ps.instructions
      4.88 ±  3%     -78.6%       1.05 ±  4%  perf-stat.ps.major-faults
      6706           -28.0%       4827        perf-stat.ps.minor-faults
    139125 ±  4%     -10.6%     124398 ±  3%  perf-stat.ps.node-load-misses
     68666 ±  6%     -34.2%      45167 ±  4%  perf-stat.ps.node-loads
     76159 ± 14%     -46.0%      41093 ±  3%  perf-stat.ps.node-stores
      6711           -28.1%       4828        perf-stat.ps.page-faults
     93316            +1.6%      94785        perf-stat.ps.task-clock
 3.538e+12            +1.6%  3.593e+12        perf-stat.total.instructions



***************************************************************************************************
lkp-icl-2sp1: 96 threads 2 sockets (Ice Lake) with 256G memory
=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/rootfs/tbox_group/test/testcase:
  gcc-11/performance/1HDD/xfs/x86_64-rhel-8.3/debian-11.1-x86_64-20220510.cgz/lkp-icl-2sp1/cvar_example.f/filebench

commit: 
  a525b95417 ("cpufreq/amd-pstate: Remove fast_switch_possible flag from active driver")
  278dfbd17c ("cpufreq: Return failure if fast_switch is not set and fast_switch_possible is set")

a525b954173e74cd 278dfbd17c5d0767834471d7fea 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :6          100%           6:6     kmsg.cpufreq:fast_switch_possible_is_enabled_but_fast_switch_callback_is_not_set
         %stddev     %change         %stddev
             \          |                \  
      2831           -72.1%     788.78        filebench.sum_bytes_mb/s
  87006971           -72.1%   24234692        filebench.sum_operations
   1450022           -72.1%     403880        filebench.sum_operations/s
    483340           -72.1%     134626        filebench.sum_reads/s
      0.00          +100.0%       0.00        filebench.sum_time_ms/op
    283.33 ± 97%   +2342.1%       6919 ± 13%  filebench.time.involuntary_context_switches
     14383            +2.1%      14680        filebench.time.minor_page_faults
     93.00            -1.1%      92.00        filebench.time.percent_of_cpu_this_job_got
     72.15 ± 13%    +120.2%     158.88 ± 40%  boot-time.boot
      6397 ± 13%    +119.9%      14065 ± 39%  boot-time.idle
     96.77            -1.7%      95.12        iostat.cpu.idle
      2.79           +49.2%       4.17        iostat.cpu.system
     31818 ± 69%    +201.5%      95941 ±  8%  numa-meminfo.node1.AnonHugePages
    196985 ± 43%     +62.8%     320714 ± 20%  numa-meminfo.node1.Inactive
    141.31 ±  6%     +61.8%     228.60 ± 28%  uptime.boot
     12823 ±  6%     +59.4%      20437 ± 27%  uptime.idle
      2294 ±  2%    +142.4%       5562        vmstat.system.cs
     94416            -1.8%      92687        vmstat.system.in
      1697           +88.1%       3193        meminfo.Active
      1688           +88.7%       3185        meminfo.Active(anon)
     74749 ± 10%     +45.0%     108373 ±  4%  meminfo.AnonHugePages
   8341845 ± 11%     +28.7%   10738346 ± 19%  meminfo.DirectMap2M
      2.05            +1.2        3.27        mpstat.cpu.all.irq%
      0.12 ±  3%      +0.0        0.17 ±  6%  mpstat.cpu.all.soft%
      0.63            +0.1        0.76        mpstat.cpu.all.sys%
      0.42            +0.3        0.70        mpstat.cpu.all.usr%
      2.52 ± 51%      +2.9        5.39 ± 26%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      1.86 ± 76%      +3.5        5.39 ± 26%  perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      1.62 ± 74%      +3.8        5.39 ± 26%  perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
      1.62 ± 74%      +3.8        5.39 ± 26%  perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
      0.14 ±223%      +6.1        6.24 ± 46%  perf-profile.calltrace.cycles-pp.show_interrupts.seq_read_iter.proc_reg_read_iter.vfs_read.ksys_read
      0.38 ±164%      +6.8        7.16 ± 60%  perf-profile.calltrace.cycles-pp.proc_reg_read_iter.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.24 ±144%      +6.9        7.16 ± 60%  perf-profile.calltrace.cycles-pp.seq_read_iter.proc_reg_read_iter.vfs_read.ksys_read.do_syscall_64
      2.52 ± 51%      +2.9        5.39 ± 26%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      1.86 ± 76%      +3.5        5.39 ± 26%  perf-profile.children.cycles-pp.tick_sched_timer
      1.62 ± 74%      +3.8        5.39 ± 26%  perf-profile.children.cycles-pp.tick_sched_handle
      1.62 ± 74%      +3.8        5.39 ± 26%  perf-profile.children.cycles-pp.update_process_times
      0.14 ±223%      +5.7        5.83 ± 54%  perf-profile.children.cycles-pp.show_interrupts
      0.38 ±164%      +6.8        7.16 ± 60%  perf-profile.children.cycles-pp.proc_reg_read_iter
    421.67           +88.9%     796.33        proc-vmstat.nr_active_anon
     18437            +2.3%      18864        proc-vmstat.nr_kernel_stack
     11616            +3.4%      12017        proc-vmstat.nr_shmem
     55935            -2.0%      54811        proc-vmstat.nr_slab_unreclaimable
    421.67           +88.9%     796.33        proc-vmstat.nr_zone_active_anon
      2418           +12.1%       2710 ±  9%  proc-vmstat.numa_hint_faults_local
   1116500 ±  4%     -54.4%     509544 ±  2%  proc-vmstat.numa_hit
   1029620 ±  4%     -59.0%     422532 ±  2%  proc-vmstat.numa_local
   1837630 ±  5%     -66.1%     623575 ±  3%  proc-vmstat.pgalloc_normal
    308419            +4.3%     321665        proc-vmstat.pgfault
   1648136 ±  6%     -74.1%     427598 ±  5%  proc-vmstat.pgfree
     13210           +10.0%      14531        proc-vmstat.pgreuse
    510506           +11.8%     570880        proc-vmstat.unevictable_pgs_scanned
     85.17           -26.8%      62.33        turbostat.Avg_MHz
      4.09            +2.2        6.29        turbostat.Busy%
      2077           -51.9%       1000        turbostat.Bzy_MHz
      7434 ±  5%    +986.2%      80757 ±  2%  turbostat.C1
     29350 ±  3%    +127.1%      66640 ±  2%  turbostat.C1E
      3.51 ±  7%     +51.0%       5.30 ±  5%  turbostat.CPU%c1
     53.50           -15.3%      45.33 ±  5%  turbostat.CoreTmp
     16996 ±  5%     -96.8%     545.33 ± 30%  turbostat.POLL
     36.83           -16.8%      30.63 ±  3%  turbostat.Pkg%pc2
      1.58 ±  8%     -25.8%       1.17 ±  2%  turbostat.Pkg%pc6
     54.17           -13.5%      46.83 ±  4%  turbostat.PkgTmp
    209.90           -41.0%     123.86        turbostat.PkgWatt
    162.53           -10.2%     145.96        turbostat.RAMWatt
    100.43 ± 20%     -71.1%      28.98 ± 21%  sched_debug.cfs_rq:/.load_avg.avg
    272.22 ± 10%     -55.7%     120.65 ± 19%  sched_debug.cfs_rq:/.load_avg.stddev
     21531 ± 21%     -40.9%      12715 ± 19%  sched_debug.cfs_rq:/.min_vruntime.avg
     61.41 ± 33%     -83.9%       9.91 ± 54%  sched_debug.cfs_rq:/.removed.load_avg.avg
    237.67 ± 15%     -63.1%      87.63 ± 46%  sched_debug.cfs_rq:/.removed.load_avg.stddev
     24.37 ± 40%     -83.6%       3.99 ± 73%  sched_debug.cfs_rq:/.removed.runnable_avg.avg
    528.83           -41.3%     310.17 ± 56%  sched_debug.cfs_rq:/.removed.runnable_avg.max
     98.29 ± 21%     -65.0%      34.42 ± 61%  sched_debug.cfs_rq:/.removed.runnable_avg.stddev
     24.37 ± 40%     -83.6%       3.99 ± 73%  sched_debug.cfs_rq:/.removed.util_avg.avg
    528.83           -41.3%     310.17 ± 56%  sched_debug.cfs_rq:/.removed.util_avg.max
     98.29 ± 21%     -65.0%      34.42 ± 61%  sched_debug.cfs_rq:/.removed.util_avg.stddev
    311.92 ±  4%     -29.3%     220.52 ±  4%  sched_debug.cfs_rq:/.runnable_avg.avg
      1445 ± 11%     -29.1%       1024        sched_debug.cfs_rq:/.runnable_avg.max
    337.24 ±  7%     -26.1%     249.36 ±  3%  sched_debug.cfs_rq:/.runnable_avg.stddev
    311.31 ±  4%     -29.2%     220.40 ±  4%  sched_debug.cfs_rq:/.util_avg.avg
      1444 ± 11%     -29.1%       1024        sched_debug.cfs_rq:/.util_avg.max
    336.63 ±  7%     -25.9%     249.28 ±  3%  sched_debug.cfs_rq:/.util_avg.stddev
    502.50 ±  4%     +57.5%     791.33        sched_debug.cfs_rq:/.util_est_enqueued.max
      2321 ± 18%    +196.4%       6880 ±  6%  sched_debug.cpu.avg_idle.min
     78576 ± 12%    +111.1%     165880 ± 38%  sched_debug.cpu.clock.avg
     78581 ± 12%    +111.1%     165890 ± 38%  sched_debug.cpu.clock.max
     78570 ± 12%    +111.1%     165870 ± 38%  sched_debug.cpu.clock.min
      2.89 ± 26%     +96.7%       5.68 ±  5%  sched_debug.cpu.clock.stddev
     77698 ± 11%    +109.1%     162504 ± 38%  sched_debug.cpu.clock_task.avg
     78440 ± 12%    +110.4%     165076 ± 38%  sched_debug.cpu.clock_task.max
     48018           +54.5%      74192 ± 10%  sched_debug.cpu.clock_task.min
     78574 ± 12%    +111.1%     165870 ± 38%  sched_debug.cpu_clk
     77364 ± 12%    +112.8%     164658 ± 39%  sched_debug.ktime
     64011           +42.5%      91223 ±  9%  sched_debug.sched_clk
     12.22 ±  2%    +157.3%      31.43        perf-stat.i.MPKI
 1.623e+09           -60.9%   6.35e+08        perf-stat.i.branch-instructions
      1.03 ±  4%      +1.4        2.47 ±  2%  perf-stat.i.branch-miss-rate%
      0.58 ±  2%      +0.1        0.68 ±  5%  perf-stat.i.cache-miss-rate%
    562035 ±  2%     +11.1%     624178 ±  4%  perf-stat.i.cache-misses
  94757587 ±  2%      -5.8%   89223187        perf-stat.i.cache-references
      1864 ±  2%    +184.1%       5298        perf-stat.i.context-switches
      0.87           +79.6%       1.57        perf-stat.i.cpi
 6.768e+09           -33.0%  4.536e+09        perf-stat.i.cpu-cycles
    105.39            -2.3%     102.97        perf-stat.i.cpu-migrations
     17826 ±  4%     -45.7%       9673 ±  6%  perf-stat.i.cycles-between-cache-misses
      0.13            +0.2        0.35        perf-stat.i.dTLB-load-miss-rate%
   2476399            +3.2%    2556155        perf-stat.i.dTLB-load-misses
 1.953e+09           -60.6%  7.691e+08        perf-stat.i.dTLB-loads
      0.02 ±  2%      +0.0        0.06 ±  5%  perf-stat.i.dTLB-store-miss-rate%
  1.16e+09           -63.3%  4.259e+08        perf-stat.i.dTLB-stores
  7.77e+09           -60.4%   3.08e+09        perf-stat.i.instructions
      1.15           -42.5%       0.66        perf-stat.i.ipc
      0.09 ± 68%     -93.8%       0.01 ±141%  perf-stat.i.major-faults
      0.07           -32.9%       0.05        perf-stat.i.metric.GHz
     49.68           -61.6%      19.08        perf-stat.i.metric.M/sec
    107291 ±  7%     +31.6%     141162 ±  5%  perf-stat.i.node-load-misses
     12.20 ±  2%    +137.6%      28.98        perf-stat.overall.MPKI
      1.02 ±  4%      +1.4        2.47 ±  2%  perf-stat.overall.branch-miss-rate%
      0.59 ±  2%      +0.1        0.70 ±  4%  perf-stat.overall.cache-miss-rate%
      0.87           +69.1%       1.47        perf-stat.overall.cpi
     12075 ±  2%     -39.7%       7285 ±  4%  perf-stat.overall.cycles-between-cache-misses
      0.13            +0.2        0.33 ±  2%  perf-stat.overall.dTLB-load-miss-rate%
      0.02 ±  2%      +0.0        0.06 ±  5%  perf-stat.overall.dTLB-store-miss-rate%
      1.15           -40.9%       0.68        perf-stat.overall.ipc
 1.597e+09           -60.9%  6.242e+08        perf-stat.ps.branch-instructions
    551866 ±  2%     +11.2%     613416 ±  4%  perf-stat.ps.cache-misses
  93226966 ±  2%      -5.9%   87736015        perf-stat.ps.cache-references
      1834 ±  2%    +184.1%       5212        perf-stat.ps.context-switches
 6.659e+09           -33.0%   4.46e+09        perf-stat.ps.cpu-cycles
    103.65            -2.4%     101.20        perf-stat.ps.cpu-migrations
   2436431            +3.2%    2513516        perf-stat.ps.dTLB-load-misses
 1.922e+09           -60.7%   7.56e+08        perf-stat.ps.dTLB-loads
 1.141e+09           -63.3%  4.188e+08        perf-stat.ps.dTLB-stores
 7.644e+09           -60.4%  3.028e+09        perf-stat.ps.instructions
      0.09 ± 68%     -93.8%       0.01 ±141%  perf-stat.ps.major-faults
    105526 ±  7%     +31.5%     138766 ±  5%  perf-stat.ps.node-load-misses
 4.798e+11           -61.5%   1.85e+11        perf-stat.total.instructions



***************************************************************************************************
lkp-csl-2sp7: 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz (Cascade Lake) with 512G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/option_a/rootfs/tbox_group/test/testcase:
  gcc-11/performance/x86_64-rhel-8.3/ffs/debian-x86_64-phoronix/lkp-csl-2sp7/glibc-bench-1.6.1/phoronix-test-suite

commit: 
  a525b95417 ("cpufreq/amd-pstate: Remove fast_switch_possible flag from active driver")
  278dfbd17c ("cpufreq: Return failure if fast_switch is not set and fast_switch_possible is set")

a525b954173e74cd 278dfbd17c5d0767834471d7fea 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :6          100%           6:6     kmsg.cpufreq:fast_switch_possible_is_enabled_but_fast_switch_callback_is_not_set
         %stddev     %change         %stddev
             \          |                \  
      2.57          +301.2%      10.31        phoronix-test-suite.glibc-bench.ffs.nanoseconds
     55.10           +23.5%      68.06        phoronix-test-suite.time.elapsed_time
     55.10           +23.5%      68.06        phoronix-test-suite.time.elapsed_time.max
    300.00 ± 14%    +398.7%       1496 ±  2%  phoronix-test-suite.time.involuntary_context_switches
     62.33           +12.0%      69.83        phoronix-test-suite.time.percent_of_cpu_this_job_got
      5.40          +110.3%      11.36        phoronix-test-suite.time.system_time
     29.22           +24.6%      36.42        phoronix-test-suite.time.user_time
 5.284e+09           +17.9%  6.228e+09        cpuidle..time
   5539887           +18.5%    6564240        cpuidle..usage
      7874           +26.6%       9970        meminfo.Active(anon)
    525422 ±  7%     -16.3%     439748 ± 11%  meminfo.DirectMap4k
    114.59 ±  4%     +19.2%     136.55 ±  7%  uptime.boot
     10125 ±  4%     +16.8%      11822 ±  6%  uptime.idle
      1414 ±  5%     -16.7%       1177 ±  6%  vmstat.io.bi
      1.00          +100.0%       2.00        vmstat.procs.r
      2577           +93.3%       4983        vmstat.system.cs
     95670            -3.3%      92477        vmstat.system.in
      0.06 ± 13%      -0.0        0.05 ± 18%  mpstat.cpu.all.iowait%
      1.73            +1.8        3.56        mpstat.cpu.all.irq%
      0.10 ±  5%      +0.1        0.21        mpstat.cpu.all.soft%
      0.18            +0.2        0.40        mpstat.cpu.all.sys%
      0.79            +1.1        1.87        mpstat.cpu.all.usr%
    155.96 ± 32%     -28.9%     110.84 ± 34%  sched_debug.cfs_rq:/.load_avg.stddev
     32174 ± 20%     -24.5%      24289 ±  9%  sched_debug.cfs_rq:/.min_vruntime.avg
     45718 ± 11%     +25.8%      57491 ±  4%  sched_debug.cfs_rq:/.min_vruntime.max
    244.32 ± 11%     -18.3%     199.54 ±  8%  sched_debug.cfs_rq:/.runnable_avg.stddev
     18560 ± 27%     +88.6%      34998 ±  7%  sched_debug.cfs_rq:/.spread0.max
    244.33 ± 11%     -18.3%     199.53 ±  8%  sched_debug.cfs_rq:/.util_avg.stddev
      3.37 ± 26%    +161.7%       8.82 ±  4%  sched_debug.cpu.clock.stddev
     35302           +20.5%      42534        sched_debug.cpu.clock_task.min
    149.18 ± 36%     +74.1%     259.73 ± 31%  sched_debug.cpu.curr->pid.avg
    530.50 ± 18%     +29.1%     684.71 ± 13%  sched_debug.cpu.curr->pid.stddev
     51821           +14.8%      59506        sched_debug.sched_clk
      1968           +26.6%       2492        proc-vmstat.nr_active_anon
    108779            -6.0%     102231        proc-vmstat.nr_anon_pages
    285189            -2.4%     278339        proc-vmstat.nr_inactive_anon
     52438            -1.6%      51613        proc-vmstat.nr_slab_unreclaimable
      1968           +26.6%       2492        proc-vmstat.nr_zone_active_anon
    285189            -2.4%     278339        proc-vmstat.nr_zone_inactive_anon
    445853            +9.3%     487223        proc-vmstat.numa_hit
    426867            +9.7%     468279        proc-vmstat.numa_local
    513203            +9.1%     560135        proc-vmstat.pgalloc_normal
    570191           +12.7%     642729        proc-vmstat.pgfault
    419473            +9.4%     458954        proc-vmstat.pgfree
     69487           +20.8%      83936        proc-vmstat.pgreuse
    595200           +21.5%     722944        proc-vmstat.unevictable_pgs_scanned
      4.38            +4.7        9.07        turbostat.Busy%
      1570           -55.4%     700.00        turbostat.Bzy_MHz
     75921 ± 10%     -37.6%      47362 ±  2%  turbostat.C1
      0.04 ± 21%      -0.0        0.02        turbostat.C1%
     56181 ± 20%    +172.3%     153009 ±  9%  turbostat.C1E
   5377888           +18.0%    6343412        turbostat.C6
     70.60           -20.2%      56.35        turbostat.CPU%c1
     25.02 ±  2%     +38.2%      34.58        turbostat.CPU%c6
     47.67           -21.3%      37.50        turbostat.CoreTmp
   5554932           +18.2%    6565797        turbostat.IRQ
     25793           -96.1%       1008 ± 14%  turbostat.POLL
     17.91 ±  4%     -70.0%       5.38 ±  4%  turbostat.Pkg%pc2
      3.66 ±  4%     -94.8%       0.19 ±  3%  turbostat.Pkg%pc6
     47.83 ±  2%     -20.6%      38.00        turbostat.PkgTmp
    116.41           -40.0%      69.85        turbostat.PkgWatt
    141.47            +1.4%     143.42        turbostat.RAMWatt
     18.20 ± 15%     -11.1        7.12 ± 56%  perf-profile.calltrace.cycles-pp.generic_file_write_iter.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     18.20 ± 15%     -11.1        7.12 ± 56%  perf-profile.calltrace.cycles-pp.__generic_file_write_iter.generic_file_write_iter.vfs_write.ksys_write.do_syscall_64
     18.20 ± 15%     -11.1        7.12 ± 56%  perf-profile.calltrace.cycles-pp.generic_perform_write.__generic_file_write_iter.generic_file_write_iter.vfs_write.ksys_write
     18.20 ± 15%     -10.3        7.91 ± 60%  perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     18.20 ± 15%     -10.3        7.91 ± 60%  perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.24 ±107%      -4.4        0.79 ±223%  perf-profile.calltrace.cycles-pp.do_filp_open.do_sys_openat2.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.24 ±107%      -4.4        0.79 ±223%  perf-profile.calltrace.cycles-pp.path_openat.do_filp_open.do_sys_openat2.__x64_sys_openat.do_syscall_64
      3.86 ±100%      -3.9        0.00        perf-profile.calltrace.cycles-pp.__do_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.24 ±107%      -3.8        1.39 ±143%  perf-profile.calltrace.cycles-pp.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.24 ±107%      -3.8        1.39 ±143%  perf-profile.calltrace.cycles-pp.do_sys_openat2.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.53 ± 72%      -2.8        2.75 ±159%  perf-profile.calltrace.cycles-pp.shmem_write_begin.generic_perform_write.__generic_file_write_iter.generic_file_write_iter.vfs_write
      4.14 ±102%      -2.2        1.96 ±223%  perf-profile.calltrace.cycles-pp.shmem_alloc_and_acct_folio.shmem_get_folio_gfp.shmem_write_begin.generic_perform_write.__generic_file_write_iter
      4.14 ±102%      -1.4        2.75 ±159%  perf-profile.calltrace.cycles-pp.shmem_get_folio_gfp.shmem_write_begin.generic_perform_write.__generic_file_write_iter.generic_file_write_iter
     18.20 ± 15%     -11.1        7.12 ± 56%  perf-profile.children.cycles-pp.generic_file_write_iter
     18.20 ± 15%     -11.1        7.12 ± 56%  perf-profile.children.cycles-pp.__generic_file_write_iter
     18.20 ± 15%     -11.1        7.12 ± 56%  perf-profile.children.cycles-pp.generic_perform_write
     18.20 ± 15%     -10.3        7.91 ± 60%  perf-profile.children.cycles-pp.ksys_write
     18.20 ± 15%     -10.3        7.91 ± 60%  perf-profile.children.cycles-pp.vfs_write
      5.24 ±107%      -4.4        0.79 ±223%  perf-profile.children.cycles-pp.do_filp_open
      5.24 ±107%      -4.4        0.79 ±223%  perf-profile.children.cycles-pp.path_openat
      4.34 ±101%      -4.3        0.00        perf-profile.children.cycles-pp.do_vmi_align_munmap
      3.86 ±100%      -3.9        0.00        perf-profile.children.cycles-pp.__do_sys_brk
      5.24 ±107%      -3.8        1.39 ±143%  perf-profile.children.cycles-pp.__x64_sys_openat
      5.24 ±107%      -3.8        1.39 ±143%  perf-profile.children.cycles-pp.do_sys_openat2
      4.14 ±102%      -3.3        0.79 ±223%  perf-profile.children.cycles-pp.__alloc_pages
      5.53 ± 72%      -2.8        2.75 ±159%  perf-profile.children.cycles-pp.shmem_write_begin
      4.14 ±102%      -2.2        1.96 ±223%  perf-profile.children.cycles-pp.shmem_alloc_and_acct_folio
      4.14 ±102%      -1.4        2.75 ±159%  perf-profile.children.cycles-pp.shmem_get_folio_gfp
      4.76 ±111%      -4.0        0.79 ±223%  perf-profile.self.cycles-pp.fault_in_readable
      3.77 ±100%      -3.2        0.60 ±223%  perf-profile.self.cycles-pp.zap_pte_range
     35.29            +6.2%      37.47        perf-stat.i.MPKI
 1.103e+09           -49.5%  5.566e+08        perf-stat.i.branch-instructions
      3.99            +1.8        5.80        perf-stat.i.branch-miss-rate%
  19937132           +51.2%   30154000        perf-stat.i.branch-misses
      4.48            -0.2        4.25        perf-stat.i.cache-miss-rate%
   2311967 ±  2%     +57.9%    3651013        perf-stat.i.cache-misses
  52356349           +63.0%   85336312        perf-stat.i.cache-references
      2132 ±  3%    +120.5%       4703        perf-stat.i.context-switches
      2.80           -17.2%       2.31        perf-stat.i.cpi
 5.652e+09            -5.3%  5.353e+09        perf-stat.i.cpu-cycles
      2808           -45.3%       1536        perf-stat.i.cycles-between-cache-misses
      0.39 ±  2%      +0.1        0.48        perf-stat.i.dTLB-load-miss-rate%
   1543223 ±  3%     +63.8%    2527374        perf-stat.i.dTLB-load-misses
 1.017e+09           -44.3%  5.669e+08        perf-stat.i.dTLB-loads
      0.07 ±  3%      +0.0        0.09        perf-stat.i.dTLB-store-miss-rate%
    152204 ±  4%     +54.3%     234914        perf-stat.i.dTLB-store-misses
 5.998e+08           -51.9%  2.885e+08        perf-stat.i.dTLB-stores
     72.30            +4.0       76.32        perf-stat.i.iTLB-load-miss-rate%
   1341052 ±  2%     -32.8%     901087        perf-stat.i.iTLB-load-misses
    487115 ±  2%     -42.8%     278743 ±  2%  perf-stat.i.iTLB-loads
 4.089e+09           -39.9%  2.455e+09        perf-stat.i.instructions
      3035           -10.0%       2730        perf-stat.i.instructions-per-iTLB-miss
      0.71           -36.1%       0.45        perf-stat.i.ipc
      3.27 ±  4%     -62.7%       1.22        perf-stat.i.major-faults
      0.06            -5.3%       0.06        perf-stat.i.metric.GHz
    690.67 ±  2%     +27.4%     879.69        perf-stat.i.metric.K/sec
     28.18           -47.8%      14.72        perf-stat.i.metric.M/sec
      6125            -6.1%       5749        perf-stat.i.minor-faults
     82.46            -6.4       76.09        perf-stat.i.node-load-miss-rate%
    116280 ±  5%     +25.1%     145514 ±  4%  perf-stat.i.node-load-misses
      6128            -6.2%       5751        perf-stat.i.page-faults
     12.82          +171.2%      34.76        perf-stat.overall.MPKI
      1.81            +3.6        5.42        perf-stat.overall.branch-miss-rate%
      4.41            -0.1        4.28        perf-stat.overall.cache-miss-rate%
      1.38           +57.6%       2.18        perf-stat.overall.cpi
      2445 ±  2%     -40.0%       1466        perf-stat.overall.cycles-between-cache-misses
      0.15 ±  3%      +0.3        0.44        perf-stat.overall.dTLB-load-miss-rate%
      0.03 ±  4%      +0.1        0.08        perf-stat.overall.dTLB-store-miss-rate%
     73.35            +3.0       76.38        perf-stat.overall.iTLB-load-miss-rate%
      3049 ±  2%     -10.6%       2725        perf-stat.overall.instructions-per-iTLB-miss
      0.72           -36.6%       0.46        perf-stat.overall.ipc
 1.082e+09           -49.4%   5.48e+08        perf-stat.ps.branch-instructions
  19556338           +51.8%   29688966        perf-stat.ps.branch-misses
   2269777 ±  2%     +58.4%    3594809        perf-stat.ps.cache-misses
  51414505           +63.4%   84023854        perf-stat.ps.cache-references
      2094 ±  3%    +121.1%       4630        perf-stat.ps.context-switches
 5.549e+09            -5.0%  5.271e+09        perf-stat.ps.cpu-cycles
   1515443 ±  3%     +64.2%    2488466        perf-stat.ps.dTLB-load-misses
 9.979e+08           -44.1%  5.582e+08        perf-stat.ps.dTLB-loads
    149490 ±  4%     +54.7%     231314        perf-stat.ps.dTLB-store-misses
 5.886e+08           -51.7%  2.841e+08        perf-stat.ps.dTLB-stores
   1316324 ±  2%     -32.6%     887291        perf-stat.ps.iTLB-load-misses
    478150 ±  2%     -42.6%     274431 ±  2%  perf-stat.ps.iTLB-loads
 4.012e+09           -39.7%  2.418e+09        perf-stat.ps.instructions
      3.20 ±  4%     -62.6%       1.20        perf-stat.ps.major-faults
      6015            -5.9%       5661        perf-stat.ps.minor-faults
    114133 ±  5%     +25.6%     143299 ±  4%  perf-stat.ps.node-load-misses
      6018            -5.9%       5662        perf-stat.ps.page-faults
 2.216e+11           -28.0%  1.596e+11        perf-stat.total.instructions



***************************************************************************************************
lkp-csl-2sp7: 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz (Cascade Lake) with 512G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/option_a/rootfs/tbox_group/test/testcase:
  gcc-11/performance/x86_64-rhel-8.3/pthread_once/debian-x86_64-phoronix/lkp-csl-2sp7/glibc-bench-1.6.1/phoronix-test-suite

commit: 
  a525b95417 ("cpufreq/amd-pstate: Remove fast_switch_possible flag from active driver")
  278dfbd17c ("cpufreq: Return failure if fast_switch is not set and fast_switch_possible is set")

a525b954173e74cd 278dfbd17c5d0767834471d7fea 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :6          100%           6:6     kmsg.cpufreq:fast_switch_possible_is_enabled_but_fast_switch_callback_is_not_set
         %stddev     %change         %stddev
             \          |                \  
      2.59          +300.8%      10.40        phoronix-test-suite.glibc-bench.pthread_once.nanoseconds
     55.71           +22.1%      68.01        phoronix-test-suite.time.elapsed_time
     55.71           +22.1%      68.01        phoronix-test-suite.time.elapsed_time.max
    317.17 ± 10%    +366.7%       1480 ±  2%  phoronix-test-suite.time.involuntary_context_switches
     61.67           +13.0%      69.67        phoronix-test-suite.time.percent_of_cpu_this_job_got
      5.29          +112.4%      11.24        phoronix-test-suite.time.system_time
     29.32           +24.3%      36.46        phoronix-test-suite.time.user_time
      8014 ±  3%     +24.5%       9980        meminfo.Active(anon)
     14023 ± 28%     -33.3%       9356 ± 44%  numa-vmstat.node1.nr_slab_reclaimable
 5.345e+09           +16.4%  6.223e+09        cpuidle..time
   5596240           +17.4%    6571074        cpuidle..usage
     56094 ± 28%     -33.3%      37426 ± 44%  numa-meminfo.node1.KReclaimable
     56094 ± 28%     -33.3%      37426 ± 44%  numa-meminfo.node1.SReclaimable
    125.08 ± 12%     +91.5%     239.55 ± 50%  uptime.boot
     11056 ± 11%     +86.0%      20560 ± 49%  uptime.idle
     66.42 ± 22%    +152.7%     167.84 ± 72%  boot-time.boot
     46.41 ± 31%    +206.2%     142.14 ± 85%  boot-time.dhcp
      5588 ± 23%    +152.3%      14100 ± 72%  boot-time.idle
      2.58 ± 45%    +307.1%      10.52 ± 98%  boot-time.smp_boot
      1511           -20.4%       1202 ±  5%  vmstat.io.bi
      1.00          +100.0%       2.00        vmstat.procs.r
      2580 ±  2%     +94.1%       5008        vmstat.system.cs
     95728            -3.3%      92603        vmstat.system.in
      0.06 ±  3%      -0.0        0.04 ±  3%  mpstat.cpu.all.iowait%
      1.70            +1.8        3.55        mpstat.cpu.all.irq%
      0.10 ±  4%      +0.1        0.21        mpstat.cpu.all.soft%
      0.18            +0.2        0.40        mpstat.cpu.all.sys%
      0.79            +1.1        1.89        mpstat.cpu.all.usr%
      2003 ±  3%     +24.6%       2495        proc-vmstat.nr_active_anon
    108638            -5.6%     102534        proc-vmstat.nr_anon_pages
    285043            -2.2%     278915        proc-vmstat.nr_inactive_anon
     52421            -1.5%      51635        proc-vmstat.nr_slab_unreclaimable
      2003 ±  3%     +24.6%       2495        proc-vmstat.nr_zone_active_anon
    285043            -2.2%     278915        proc-vmstat.nr_zone_inactive_anon
    459003            +6.8%     490091        proc-vmstat.numa_hit
    440043            +7.3%     472049        proc-vmstat.numa_local
    529679            +6.8%     565786        proc-vmstat.pgalloc_normal
    592060 ±  2%      +9.0%     645487        proc-vmstat.pgfault
    431573            +7.1%     462298        proc-vmstat.pgfree
     72771 ±  3%     +16.0%      84416        proc-vmstat.pgreuse
    609792 ±  2%     +18.5%     722688        proc-vmstat.unevictable_pgs_scanned
      4.31            +4.8        9.10        turbostat.Busy%
      1576           -55.6%     700.00        turbostat.Bzy_MHz
     61354 ± 14%     -22.8%      47384 ±  2%  turbostat.C1
     49691 ± 12%    +226.7%     162332 ±  3%  turbostat.C1E
      0.28 ±  2%      +0.0        0.32        turbostat.C1E%
   5437863           +16.4%    6329781        turbostat.C6
     70.92           -20.8%      56.16        turbostat.CPU%c1
     24.77           +40.3%      34.74        turbostat.CPU%c6
     47.00           -19.5%      37.83        turbostat.CoreTmp
   5596785           +17.2%    6561866        turbostat.IRQ
     25649           -96.9%     792.83 ±  7%  turbostat.POLL
     17.44 ±  4%     -64.5%       6.19 ± 27%  turbostat.Pkg%pc2
      3.73 ±  5%     -94.8%       0.20 ±  2%  turbostat.Pkg%pc6
     47.00           -18.1%      38.50        turbostat.PkgTmp
    116.45           -40.0%      69.92        turbostat.PkgWatt
    141.45            +1.4%     143.45        turbostat.RAMWatt
      0.07 ± 32%     +74.4%       0.13 ±  7%  sched_debug.cfs_rq:/.h_nr_running.avg
      0.26 ± 13%     +29.7%       0.34 ±  3%  sched_debug.cfs_rq:/.h_nr_running.stddev
     34989 ± 16%     -29.9%      24543 ± 12%  sched_debug.cfs_rq:/.min_vruntime.avg
      0.07 ± 34%     +76.2%       0.13 ± 10%  sched_debug.cfs_rq:/.nr_running.avg
      0.26 ± 14%     +30.5%       0.33 ±  4%  sched_debug.cfs_rq:/.nr_running.stddev
     38.84 ± 25%     -83.8%       6.29 ±160%  sched_debug.cfs_rq:/.removed.load_avg.avg
    193.00 ± 13%     -78.0%      42.53 ±145%  sched_debug.cfs_rq:/.removed.load_avg.stddev
     16.82 ± 21%     -84.4%       2.62 ±190%  sched_debug.cfs_rq:/.removed.runnable_avg.avg
    514.17           -78.2%     112.33 ±157%  sched_debug.cfs_rq:/.removed.runnable_avg.max
     85.74 ± 10%     -81.2%      16.12 ±173%  sched_debug.cfs_rq:/.removed.runnable_avg.stddev
     16.82 ± 21%     -84.4%       2.62 ±190%  sched_debug.cfs_rq:/.removed.util_avg.avg
    514.00           -78.2%     112.17 ±157%  sched_debug.cfs_rq:/.removed.util_avg.max
     85.73 ± 10%     -81.2%      16.11 ±173%  sched_debug.cfs_rq:/.removed.util_avg.stddev
    205.16 ±  6%     -32.6%     138.25 ±  7%  sched_debug.cfs_rq:/.runnable_avg.avg
    250.68 ±  5%     -15.7%     211.41 ±  5%  sched_debug.cfs_rq:/.runnable_avg.stddev
     12358 ± 66%     -78.7%       2634 ±154%  sched_debug.cfs_rq:/.spread0.avg
    205.08 ±  6%     -32.7%     138.10 ±  7%  sched_debug.cfs_rq:/.util_avg.avg
    250.67 ±  5%     -15.7%     211.37 ±  5%  sched_debug.cfs_rq:/.util_avg.stddev
    760.17 ±  5%     -38.4%     468.33 ± 11%  sched_debug.cfs_rq:/.util_est_enqueued.max
    899715 ±  3%     +17.3%    1055048 ± 12%  sched_debug.cpu.avg_idle.avg
     68244 ± 21%    +149.7%     170398 ± 71%  sched_debug.cpu.clock.avg
     68250 ± 21%    +149.7%     170415 ± 71%  sched_debug.cpu.clock.max
     68235 ± 21%    +149.7%     170385 ± 71%  sched_debug.cpu.clock.min
      3.51 ± 13%    +140.4%       8.43 ±  9%  sched_debug.cpu.clock.stddev
     67132 ± 21%    +144.5%     164149 ± 70%  sched_debug.cpu.clock_task.avg
     67835 ± 21%    +147.7%     168044 ± 71%  sched_debug.cpu.clock_task.max
     35817           +19.1%      42655 ±  2%  sched_debug.cpu.clock_task.min
      3228 ± 46%    +287.0%      12494 ± 95%  sched_debug.cpu.clock_task.stddev
    133.56 ± 44%    +108.6%     278.63 ± 10%  sched_debug.cpu.curr->pid.avg
    502.90 ± 20%     +42.3%     715.86 ±  4%  sched_debug.cpu.curr->pid.stddev
      0.00 ± 13%    +135.2%       0.00 ± 29%  sched_debug.cpu.next_balance.stddev
      0.06 ± 41%    +111.1%       0.13 ± 11%  sched_debug.cpu.nr_running.avg
      0.24 ± 18%     +42.8%       0.34 ±  4%  sched_debug.cpu.nr_running.stddev
     68240 ± 21%    +149.7%     170386 ± 71%  sched_debug.cpu_clk
     67525 ± 21%    +151.3%     169670 ± 71%  sched_debug.ktime
     52331           +14.7%      60008        sched_debug.sched_clk
 1.263e+09           -52.8%  5.957e+08        perf-stat.i.branch-instructions
      3.93 ±  2%      +1.6        5.57        perf-stat.i.branch-miss-rate%
  19844846           +52.5%   30265786        perf-stat.i.branch-misses
      4.49            -0.2        4.27        perf-stat.i.cache-miss-rate%
   2283572           +59.1%    3633877        perf-stat.i.cache-misses
  52070252           +62.4%   84585057        perf-stat.i.cache-references
      2159 ±  2%    +117.9%       4705        perf-stat.i.context-switches
      2.70 ±  2%     -16.7%       2.25        perf-stat.i.cpi
 5.606e+09            -4.1%  5.374e+09        perf-stat.i.cpu-cycles
    101.91            -2.3%      99.54        perf-stat.i.cpu-migrations
      2804           -44.9%       1545        perf-stat.i.cycles-between-cache-misses
      0.37 ±  5%      +0.1        0.43        perf-stat.i.dTLB-load-miss-rate%
   1542705           +65.2%    2549066 ±  2%  perf-stat.i.dTLB-load-misses
 1.535e+09           -55.7%  6.799e+08        perf-stat.i.dTLB-loads
      0.07 ±  5%      +0.0        0.10 ±  3%  perf-stat.i.dTLB-store-miss-rate%
    152237 ±  3%     +59.7%     243190 ±  3%  perf-stat.i.dTLB-store-misses
 4.136e+08           -39.1%  2.517e+08        perf-stat.i.dTLB-stores
     71.73            +4.6       76.38        perf-stat.i.iTLB-load-miss-rate%
   1314924 ±  6%     -32.7%     884328        perf-stat.i.iTLB-load-misses
    493184           -44.6%     273276 ±  2%  perf-stat.i.iTLB-loads
 4.402e+09           -42.3%  2.542e+09        perf-stat.i.instructions
      3289 ±  4%     -12.6%       2873        perf-stat.i.instructions-per-iTLB-miss
      0.77           -39.2%       0.47        perf-stat.i.ipc
      3.16 ±  2%     -61.4%       1.22        perf-stat.i.major-faults
      0.06            -4.2%       0.06        perf-stat.i.metric.GHz
    697.29 ±  2%     +26.2%     880.12        perf-stat.i.metric.K/sec
     33.29           -52.2%      15.91        perf-stat.i.metric.M/sec
      6215 ±  3%      -7.1%       5771        perf-stat.i.minor-faults
     82.62            -6.8       75.78        perf-stat.i.node-load-miss-rate%
    117468 ±  6%     +24.2%     145903 ±  2%  perf-stat.i.node-load-misses
     49000 ±  6%     +22.0%      59761 ±  6%  perf-stat.i.node-loads
      6218 ±  3%      -7.2%       5772        perf-stat.i.page-faults
     11.84 ±  2%    +181.1%      33.29        perf-stat.overall.MPKI
      1.57            +3.5        5.08        perf-stat.overall.branch-miss-rate%
      1.27           +65.9%       2.11        perf-stat.overall.cpi
      2455           -39.8%       1479        perf-stat.overall.cycles-between-cache-misses
      0.10 ±  3%      +0.3        0.37        perf-stat.overall.dTLB-load-miss-rate%
      0.04 ±  4%      +0.1        0.10 ±  3%  perf-stat.overall.dTLB-store-miss-rate%
     72.66            +3.7       76.40        perf-stat.overall.iTLB-load-miss-rate%
      3360 ±  6%     -14.5%       2874        perf-stat.overall.instructions-per-iTLB-miss
      0.78           -39.7%       0.47        perf-stat.overall.ipc
  1.24e+09           -52.7%  5.865e+08        perf-stat.ps.branch-instructions
  19487700           +52.9%   29789495        perf-stat.ps.branch-misses
   2242413           +59.5%    3577319        perf-stat.ps.cache-misses
  51143893           +62.8%   83278509        perf-stat.ps.cache-references
      2120 ±  2%    +118.4%       4632        perf-stat.ps.context-switches
 5.505e+09            -3.9%  5.291e+09        perf-stat.ps.cpu-cycles
    100.07            -2.1%      98.01        perf-stat.ps.cpu-migrations
   1515252           +65.6%    2509672 ±  2%  perf-stat.ps.dTLB-load-misses
 1.506e+09           -55.6%  6.693e+08        perf-stat.ps.dTLB-loads
    149535 ±  3%     +60.1%     239445 ±  3%  perf-stat.ps.dTLB-store-misses
  4.06e+08           -39.0%  2.478e+08        perf-stat.ps.dTLB-stores
   1290945 ±  6%     -32.5%     870773        perf-stat.ps.iTLB-load-misses
    484246           -44.4%     269019 ±  2%  perf-stat.ps.iTLB-loads
  4.32e+09           -42.1%  2.502e+09        perf-stat.ps.instructions
      3.10 ±  2%     -61.2%       1.20        perf-stat.ps.major-faults
      6104 ±  3%      -6.9%       5681        perf-stat.ps.minor-faults
    115360 ±  6%     +24.5%     143643 ±  2%  perf-stat.ps.node-load-misses
     48119 ±  7%     +22.3%      58833 ±  6%  perf-stat.ps.node-loads
      6107 ±  3%      -6.9%       5683        perf-stat.ps.page-faults
 2.422e+11           -32.0%  1.646e+11        perf-stat.total.instructions
     12.90 ± 35%      -9.8        3.08 ± 72%  perf-profile.calltrace.cycles-pp.exit_mmap.__mmput.exit_mm.do_exit.do_group_exit
     11.79 ± 43%      -9.6        2.16 ±100%  perf-profile.calltrace.cycles-pp.unmap_vmas.exit_mmap.__mmput.exit_mm.do_exit
     11.79 ± 43%      -9.6        2.16 ±100%  perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.exit_mmap.__mmput.exit_mm
      8.19 ± 58%      -6.6        1.59 ±144%  perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
      6.66 ±133%      -6.0        0.70 ±223%  perf-profile.calltrace.cycles-pp.__do_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.62 ±121%      -4.9        0.70 ±223%  perf-profile.calltrace.cycles-pp.do_vmi_align_munmap.__do_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.62 ±121%      -4.9        0.70 ±223%  perf-profile.calltrace.cycles-pp.unmap_region.do_vmi_align_munmap.__do_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.38 ±109%      -4.7        0.67 ±223%  perf-profile.calltrace.cycles-pp.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.38 ±109%      -4.7        0.67 ±223%  perf-profile.calltrace.cycles-pp.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.38 ±109%      -4.7        0.67 ±223%  perf-profile.calltrace.cycles-pp.bprm_execve.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe
      4.58 ±108%      -4.6        0.00        perf-profile.calltrace.cycles-pp.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault.fault_in_readable
      5.84 ± 88%      -4.3        1.59 ±144%  perf-profile.calltrace.cycles-pp.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
      4.24 ±115%      -4.2        0.00        perf-profile.calltrace.cycles-pp.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
      4.24 ±115%      -4.2        0.00        perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
      4.24 ±115%      -4.2        0.00        perf-profile.calltrace.cycles-pp.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
      4.11 ±108%      -4.1        0.00        perf-profile.calltrace.cycles-pp.do_fault.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault
      4.11 ±108%      -4.1        0.00        perf-profile.calltrace.cycles-pp.do_read_fault.do_fault.__handle_mm_fault.handle_mm_fault.do_user_addr_fault
      4.11 ±108%      -4.1        0.00        perf-profile.calltrace.cycles-pp.__do_fault.do_read_fault.do_fault.__handle_mm_fault.handle_mm_fault
      4.58 ±108%      -3.8        0.79 ±223%  perf-profile.calltrace.cycles-pp.exc_page_fault.asm_exc_page_fault.fault_in_readable.fault_in_iov_iter_readable.generic_perform_write
      4.58 ±108%      -3.8        0.79 ±223%  perf-profile.calltrace.cycles-pp.do_user_addr_fault.exc_page_fault.asm_exc_page_fault.fault_in_readable.fault_in_iov_iter_readable
      3.13 ±100%      -3.1        0.00        perf-profile.calltrace.cycles-pp.exit_mm.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      3.13 ±100%      -3.1        0.00        perf-profile.calltrace.cycles-pp.__mmput.exit_mm.do_exit.do_group_exit.__x64_sys_exit_group
      3.13 ±100%      -3.1        0.00        perf-profile.calltrace.cycles-pp.page_remove_rmap.zap_pte_range.zap_pmd_range.unmap_page_range.unmap_vmas
      3.21 ±100%      -2.5        0.67 ±223%  perf-profile.calltrace.cycles-pp.exec_binprm.bprm_execve.do_execveat_common.__x64_sys_execve.do_syscall_64
      3.21 ±100%      -2.5        0.67 ±223%  perf-profile.calltrace.cycles-pp.search_binary_handler.exec_binprm.bprm_execve.do_execveat_common.__x64_sys_execve
      3.82 ±101%      -2.4        1.46 ±142%  perf-profile.calltrace.cycles-pp.rep_movs_alternative.copyin.copy_page_from_iter_atomic.generic_perform_write.__generic_file_write_iter
      3.82 ±101%      -2.4        1.46 ±142%  perf-profile.calltrace.cycles-pp.copyin.copy_page_from_iter_atomic.generic_perform_write.__generic_file_write_iter.generic_file_write_iter
      0.00            +8.4        8.42 ± 42%  perf-profile.calltrace.cycles-pp.proc_reg_read_iter.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +8.4        8.42 ± 42%  perf-profile.calltrace.cycles-pp.seq_read_iter.proc_reg_read_iter.vfs_read.ksys_read.do_syscall_64
      0.00           +10.3       10.30 ± 57%  perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00           +10.3       10.30 ± 57%  perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
     14.85 ± 40%     -12.7        2.16 ±100%  perf-profile.children.cycles-pp.unmap_vmas
     13.94 ± 25%     -10.9        3.08 ± 72%  perf-profile.children.cycles-pp.__mmput
     13.94 ± 25%     -10.9        3.08 ± 72%  perf-profile.children.cycles-pp.exit_mmap
     12.83 ± 34%     -10.7        2.16 ±100%  perf-profile.children.cycles-pp.unmap_page_range
     12.90 ± 35%      -9.8        3.08 ± 72%  perf-profile.children.cycles-pp.exit_mm
     10.42 ± 29%      -8.8        1.59 ±144%  perf-profile.children.cycles-pp.handle_mm_fault
     11.71 ± 47%      -8.4        3.31 ± 72%  perf-profile.children.cycles-pp.exc_page_fault
     11.71 ± 47%      -8.4        3.31 ± 72%  perf-profile.children.cycles-pp.do_user_addr_fault
      8.19 ± 58%      -6.6        1.59 ±144%  perf-profile.children.cycles-pp.__handle_mm_fault
      6.66 ±133%      -6.0        0.70 ±223%  perf-profile.children.cycles-pp.__do_sys_brk
      6.66 ±133%      -6.0        0.70 ±223%  perf-profile.children.cycles-pp.do_vmi_align_munmap
      6.66 ±133%      -6.0        0.70 ±223%  perf-profile.children.cycles-pp.unmap_region
      5.38 ±109%      -4.7        0.67 ±223%  perf-profile.children.cycles-pp.__x64_sys_execve
      5.38 ±109%      -4.7        0.67 ±223%  perf-profile.children.cycles-pp.do_execveat_common
      5.38 ±109%      -4.7        0.67 ±223%  perf-profile.children.cycles-pp.bprm_execve
      4.24 ±115%      -4.2        0.00        perf-profile.children.cycles-pp.__x64_sys_exit_group
      4.11 ±108%      -4.1        0.00        perf-profile.children.cycles-pp.do_fault
      4.11 ±108%      -4.1        0.00        perf-profile.children.cycles-pp.do_read_fault
      4.11 ±108%      -4.1        0.00        perf-profile.children.cycles-pp.__do_fault
      3.13 ±100%      -3.1        0.00        perf-profile.children.cycles-pp.page_remove_rmap
      4.89 ± 73%      -2.7        2.18 ±149%  perf-profile.children.cycles-pp.__alloc_pages
      3.21 ±100%      -2.5        0.67 ±223%  perf-profile.children.cycles-pp.exec_binprm
      3.21 ±100%      -2.5        0.67 ±223%  perf-profile.children.cycles-pp.search_binary_handler
      3.82 ±101%      -2.4        1.46 ±142%  perf-profile.children.cycles-pp.rep_movs_alternative
      3.82 ±101%      -2.4        1.46 ±142%  perf-profile.children.cycles-pp.copyin
      0.00            +8.4        8.42 ± 42%  perf-profile.children.cycles-pp.proc_reg_read_iter
      0.00            +8.4        8.42 ± 42%  perf-profile.children.cycles-pp.seq_read_iter
      0.00           +10.3       10.30 ± 57%  perf-profile.children.cycles-pp.ksys_read
      0.00           +10.3       10.30 ± 57%  perf-profile.children.cycles-pp.vfs_read
      3.54 ±104%      -2.8        0.70 ±223%  perf-profile.self.cycles-pp.zap_pte_range
      3.82 ±101%      -2.4        1.46 ±142%  perf-profile.self.cycles-pp.rep_movs_alternative



***************************************************************************************************
lkp-csl-2sp7: 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz (Cascade Lake) with 512G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/option_a/rootfs/tbox_group/test/testcase:
  gcc-11/performance/x86_64-rhel-8.3/3/debian-x86_64-phoronix/lkp-csl-2sp7/cyclictest-1.0.0/phoronix-test-suite

commit: 
  a525b95417 ("cpufreq/amd-pstate: Remove fast_switch_possible flag from active driver")
  278dfbd17c ("cpufreq: Return failure if fast_switch is not set and fast_switch_possible is set")

a525b954173e74cd 278dfbd17c5d0767834471d7fea 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :6          100%           6:6     kmsg.cpufreq:fast_switch_possible_is_enabled_but_fast_switch_callback_is_not_set
         %stddev     %change         %stddev
             \          |                \  
      1.00         +5550.0%      56.50 ± 90%  phoronix-test-suite.cyclictest.clock_nanosleepTIME_ABSTIME,Interval500ms,100000Loops.ms_average
     67.47           +12.4%      75.83        phoronix-test-suite.time.elapsed_time
     67.47           +12.4%      75.83        phoronix-test-suite.time.elapsed_time.max
    228.83 ± 13%    +555.6%       1500 ±  3%  phoronix-test-suite.time.involuntary_context_switches
      5.00          +270.0%      18.50 ±  2%  phoronix-test-suite.time.percent_of_cpu_this_job_got
 2.229e+08           -28.7%  1.588e+08        cpuidle..usage
      8121 ±  3%     +24.7%      10129        meminfo.Active(anon)
      1244           -14.1%       1069 ±  3%  vmstat.io.bi
      0.00       +1.3e+102%       1.33 ± 35%  vmstat.procs.r
      5430           +39.5%       7576        vmstat.system.cs
     97212            -3.0%      94332        vmstat.system.in
      0.05 ±  2%      -0.0        0.04 ±  5%  mpstat.cpu.all.iowait%
      0.90 ±  2%      +1.6        2.47        mpstat.cpu.all.irq%
      0.05 ±  3%      +0.1        0.16 ±  2%  mpstat.cpu.all.soft%
      0.10            +0.2        0.34 ±  2%  mpstat.cpu.all.sys%
      0.23            +1.1        1.36        mpstat.cpu.all.usr%
      2030 ±  3%     +24.7%       2531        proc-vmstat.nr_active_anon
    108456            -4.6%     103460        proc-vmstat.nr_anon_pages
    284847            -1.8%     279599        proc-vmstat.nr_inactive_anon
     52453            -1.2%      51832        proc-vmstat.nr_slab_unreclaimable
      2030 ±  3%     +24.7%       2531        proc-vmstat.nr_zone_active_anon
    284847            -1.8%     279599        proc-vmstat.nr_zone_inactive_anon
    474897            +6.5%     505695        proc-vmstat.numa_hit
    457187            +6.5%     486800        proc-vmstat.numa_local
    551902 ±  2%      +4.8%     578226        proc-vmstat.pgalloc_normal
    615780            +9.0%     671086        proc-vmstat.pgfault
    456513 ±  2%      +4.3%     476256        proc-vmstat.pgfree
     77912           +13.6%      88541 ±  2%  proc-vmstat.pgreuse
    667136           +12.5%     750208        proc-vmstat.unevictable_pgs_scanned
      2033           -77.0%     467.83        turbostat.Avg_MHz
      2778           -74.8%     700.00        turbostat.Bzy_MHz
     65709 ± 14%     -61.7%      25182 ±  8%  turbostat.C1
      0.02 ± 20%      -0.0        0.01        turbostat.C1%
     31391 ± 20%    +190.6%      91233 ±  7%  turbostat.C1E
   1797823 ±  4%     +31.5%    2364347        turbostat.C6
     26.41 ±  3%      +5.9       32.32        turbostat.C6%
      7.08 ±  6%     +87.7%      13.30        turbostat.CPU%c6
     58.00 ±  2%     -31.3%      39.83        turbostat.CoreTmp
  2.21e+08           -29.3%  1.563e+08        turbostat.POLL
     71.21           -10.2       60.97        turbostat.POLL%
      3.92 ±  4%     -45.7%       2.12 ±  7%  turbostat.Pkg%pc2
      1.58 ± 10%     -88.7%       0.18 ±  6%  turbostat.Pkg%pc6
     57.83 ±  3%     -30.5%      40.17        turbostat.PkgTmp
    223.68           -63.4%      81.80        turbostat.PkgWatt
      3459 ± 34%     +45.1%       5020 ± 11%  sched_debug.cfs_rq:/.load.avg
     10426 ± 20%     +36.6%      14243 ±  5%  sched_debug.cfs_rq:/.load.stddev
     62.38 ± 31%     -59.0%      25.56 ± 30%  sched_debug.cfs_rq:/.load_avg.avg
    212.19 ± 22%     -50.5%     104.97 ± 34%  sched_debug.cfs_rq:/.load_avg.stddev
     42608 ± 18%     +36.6%      58197 ±  2%  sched_debug.cfs_rq:/.min_vruntime.max
      4070 ± 12%     +28.5%       5231 ±  3%  sched_debug.cfs_rq:/.min_vruntime.stddev
     45.18 ± 45%     -77.5%      10.16 ± 58%  sched_debug.cfs_rq:/.removed.load_avg.avg
    200.73 ± 25%     -55.6%      89.17 ± 48%  sched_debug.cfs_rq:/.removed.load_avg.stddev
     19.50 ± 44%     -74.2%       5.03 ± 58%  sched_debug.cfs_rq:/.removed.runnable_avg.avg
     88.18 ± 24%     -50.0%      44.09 ± 48%  sched_debug.cfs_rq:/.removed.runnable_avg.stddev
     19.50 ± 44%     -74.2%       5.02 ± 58%  sched_debug.cfs_rq:/.removed.util_avg.avg
     88.18 ± 24%     -50.0%      44.07 ± 48%  sched_debug.cfs_rq:/.removed.util_avg.stddev
    199.68 ±  9%     -32.3%     135.14 ± 10%  sched_debug.cfs_rq:/.runnable_avg.avg
    246.87 ±  5%     -16.2%     206.81 ±  7%  sched_debug.cfs_rq:/.runnable_avg.stddev
      4075 ± 13%     +28.6%       5239 ±  3%  sched_debug.cfs_rq:/.spread0.stddev
    199.62 ±  9%     -32.4%     135.03 ± 10%  sched_debug.cfs_rq:/.util_avg.avg
    246.86 ±  5%     -16.2%     206.77 ±  7%  sched_debug.cfs_rq:/.util_avg.stddev
    812.67 ± 12%     -45.7%     441.00 ± 17%  sched_debug.cfs_rq:/.util_est_enqueued.max
    101.76 ± 19%     -31.7%      69.51 ± 16%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
      4205 ± 45%    +167.9%      11264 ± 12%  sched_debug.cpu.avg_idle.min
      3.54 ± 29%    +152.9%       8.96 ± 17%  sched_debug.cpu.clock.stddev
     35726           +19.2%      42576 ±  2%  sched_debug.cpu.clock_task.min
     52210           +14.9%      60013        sched_debug.sched_clk
      5.54 ± 76%      -5.5        0.00        perf-profile.calltrace.cycles-pp.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.54 ± 76%      -5.5        0.00        perf-profile.calltrace.cycles-pp.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.54 ± 76%      -5.5        0.00        perf-profile.calltrace.cycles-pp.bprm_execve.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.54 ± 76%      -5.5        0.00        perf-profile.calltrace.cycles-pp.exec_binprm.bprm_execve.do_execveat_common.__x64_sys_execve.do_syscall_64
      5.54 ± 76%      -5.5        0.00        perf-profile.calltrace.cycles-pp.search_binary_handler.exec_binprm.bprm_execve.do_execveat_common.__x64_sys_execve
      4.43 ±105%      -4.4        0.00        perf-profile.calltrace.cycles-pp.load_elf_binary.search_binary_handler.exec_binprm.bprm_execve.do_execveat_common
      4.80 ± 73%      -3.2        1.55 ±141%  perf-profile.calltrace.cycles-pp.page_remove_rmap.zap_pte_range.zap_pmd_range.unmap_page_range.unmap_vmas
      0.00            +5.6        5.63 ± 28%  perf-profile.calltrace.cycles-pp.__fput.task_work_run.do_exit.do_group_exit.get_signal
      0.00            +5.6        5.63 ± 28%  perf-profile.calltrace.cycles-pp.task_work_run.do_exit.do_group_exit.get_signal.arch_do_signal_or_restart
      0.00            +5.6        5.63 ± 28%  perf-profile.calltrace.cycles-pp.perf_release.__fput.task_work_run.do_exit.do_group_exit
      0.00            +5.6        5.63 ± 28%  perf-profile.calltrace.cycles-pp.perf_event_release_kernel.perf_release.__fput.task_work_run.do_exit
      4.80 ± 73%      +5.7       10.50 ± 15%  perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
      4.80 ± 73%      +5.7       10.50 ± 15%  perf-profile.calltrace.cycles-pp.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
      4.80 ± 73%      +5.7       10.50 ± 15%  perf-profile.calltrace.cycles-pp.arch_do_signal_or_restart.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
      4.80 ± 73%      +5.7       10.50 ± 15%  perf-profile.calltrace.cycles-pp.get_signal.arch_do_signal_or_restart.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode
      4.80 ± 73%      +5.7       10.50 ± 15%  perf-profile.calltrace.cycles-pp.do_group_exit.get_signal.arch_do_signal_or_restart.exit_to_user_mode_loop.exit_to_user_mode_prepare
      4.80 ± 73%      +5.7       10.50 ± 15%  perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.get_signal.arch_do_signal_or_restart.exit_to_user_mode_loop
      6.08 ± 47%      +7.5       13.60 ± 27%  perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
     33.77 ± 15%     +11.1       44.92 ±  8%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
     32.84 ± 17%     +12.1       44.92 ±  8%  perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
     32.84 ± 17%     +13.0       45.84 ±  5%  perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     33.77 ± 15%     +13.9       47.65 ±  3%  perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
     32.84 ± 17%     +14.8       47.65 ±  3%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
     32.84 ± 17%     +14.8       47.65 ±  3%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     32.84 ± 17%     +14.8       47.65 ±  3%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      5.98 ±108%      -6.0        0.00        perf-profile.children.cycles-pp.inode_permission
      5.54 ± 76%      -5.5        0.00        perf-profile.children.cycles-pp.__x64_sys_execve
      5.54 ± 76%      -5.5        0.00        perf-profile.children.cycles-pp.do_execveat_common
      5.54 ± 76%      -5.5        0.00        perf-profile.children.cycles-pp.bprm_execve
      5.54 ± 76%      -5.5        0.00        perf-profile.children.cycles-pp.exec_binprm
      5.54 ± 76%      -5.5        0.00        perf-profile.children.cycles-pp.search_binary_handler
      4.86 ±115%      -4.9        0.00        perf-profile.children.cycles-pp.override_creds
      4.43 ±105%      -4.4        0.00        perf-profile.children.cycles-pp.load_elf_binary
      4.80 ± 73%      -3.2        1.55 ±141%  perf-profile.children.cycles-pp.page_remove_rmap
      0.00            +5.6        5.63 ± 28%  perf-profile.children.cycles-pp.perf_release
      0.00            +5.6        5.63 ± 28%  perf-profile.children.cycles-pp.perf_event_release_kernel
      4.80 ± 73%      +5.7       10.50 ± 15%  perf-profile.children.cycles-pp.arch_do_signal_or_restart
      4.80 ± 73%      +5.7       10.50 ± 15%  perf-profile.children.cycles-pp.get_signal
      4.80 ± 73%      +6.6       11.42 ± 24%  perf-profile.children.cycles-pp.exit_to_user_mode_prepare
      4.80 ± 73%      +6.6       11.42 ± 24%  perf-profile.children.cycles-pp.exit_to_user_mode_loop
      6.08 ± 47%      +7.5       13.60 ± 27%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
     33.77 ± 15%     +11.2       44.92 ±  8%  perf-profile.children.cycles-pp.cpuidle_enter
     33.77 ± 15%     +11.2       44.92 ±  8%  perf-profile.children.cycles-pp.cpuidle_enter_state
     33.77 ± 15%     +12.1       45.84 ±  5%  perf-profile.children.cycles-pp.cpuidle_idle_call
     33.77 ± 15%     +13.9       47.65 ±  3%  perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     33.77 ± 15%     +13.9       47.65 ±  3%  perf-profile.children.cycles-pp.cpu_startup_entry
     33.77 ± 15%     +13.9       47.65 ±  3%  perf-profile.children.cycles-pp.do_idle
     32.84 ± 17%     +14.8       47.65 ±  3%  perf-profile.children.cycles-pp.start_secondary
      7.64 ±120%      -6.9        0.76 ±223%  perf-profile.self.cycles-pp.fault_in_readable
      4.86 ±115%      -4.9        0.00        perf-profile.self.cycles-pp.override_creds
      4.94 ±130%      -3.4        1.52 ±223%  perf-profile.self.cycles-pp.show_interrupts
 9.155e+09           -71.5%   2.61e+09        perf-stat.i.branch-instructions
      1.97 ±  7%      +0.5        2.48        perf-stat.i.branch-miss-rate%
  37281940           -28.1%   26816122        perf-stat.i.branch-misses
      9.37 ±  2%      -1.2        8.20 ±  3%  perf-stat.i.cache-miss-rate%
   1168532 ±  3%     +56.3%    1826809        perf-stat.i.cache-misses
  18204626 ±  6%     +74.1%   31696003        perf-stat.i.cache-references
      5294           +41.2%       7477        perf-stat.i.context-switches
      6.21           -29.6%       4.37        perf-stat.i.cpi
 2.021e+11           -76.6%  4.725e+10        perf-stat.i.cpu-cycles
    576582 ±  3%     -84.5%      89226 ±  3%  perf-stat.i.cycles-between-cache-misses
    462899 ±  6%     +85.2%     857410 ±  2%  perf-stat.i.dTLB-load-misses
 5.539e+09           -64.5%  1.966e+09        perf-stat.i.dTLB-loads
     58476 ±  5%     +58.1%      92464 ±  3%  perf-stat.i.dTLB-store-misses
 8.442e+08           -29.3%  5.971e+08        perf-stat.i.dTLB-stores
     88.97            -6.7       82.29        perf-stat.i.iTLB-load-miss-rate%
    151482 ±  2%     +37.7%     208592 ±  2%  perf-stat.i.iTLB-loads
 2.974e+10           -68.2%  9.452e+09        perf-stat.i.instructions
     30509 ±  2%     -67.6%       9882 ±  2%  perf-stat.i.instructions-per-iTLB-miss
      0.23           +17.4%       0.28        perf-stat.i.ipc
      2.72 ±  5%     -55.9%       1.20 ±  4%  perf-stat.i.major-faults
      2.11           -76.6%       0.49        perf-stat.i.metric.GHz
    243.40 ± 10%     +37.5%     334.64        perf-stat.i.metric.K/sec
    161.80           -66.7%      53.88        perf-stat.i.metric.M/sec
     83.31            -3.2       80.11        perf-stat.i.node-load-miss-rate%
    106552 ±  5%     +32.3%     140953 ±  7%  perf-stat.i.node-load-misses
     43996 ± 11%     +16.0%      51045 ±  2%  perf-stat.i.node-loads
     59564 ±  5%     +16.5%      69365 ±  8%  perf-stat.i.node-store-misses
      0.62 ±  7%    +447.4%       3.37 ±  2%  perf-stat.overall.MPKI
      0.41            +0.6        1.03        perf-stat.overall.branch-miss-rate%
      6.43 ±  4%      -0.7        5.76 ±  2%  perf-stat.overall.cache-miss-rate%
      6.80           -26.5%       5.00        perf-stat.overall.cpi
    172717 ±  4%     -85.1%      25771        perf-stat.overall.cycles-between-cache-misses
      0.01 ±  7%      +0.0        0.04        perf-stat.overall.dTLB-load-miss-rate%
      0.01 ±  6%      +0.0        0.02 ±  2%  perf-stat.overall.dTLB-store-miss-rate%
     86.59            -4.6       82.02        perf-stat.overall.iTLB-load-miss-rate%
     30298           -67.3%       9918        perf-stat.overall.instructions-per-iTLB-miss
      0.15           +36.0%       0.20        perf-stat.overall.ipc
     56.74 ±  5%      +6.0       62.77 ±  4%  perf-stat.overall.node-store-miss-rate%
 9.009e+09           -71.5%  2.571e+09        perf-stat.ps.branch-instructions
  36711771           -27.9%   26457021        perf-stat.ps.branch-misses
   1153423 ±  3%     +56.6%    1806281        perf-stat.ps.cache-misses
  17988261 ±  6%     +74.3%   31357377        perf-stat.ps.cache-references
      5212           +41.4%       7371        perf-stat.ps.context-switches
 1.989e+11           -76.6%  4.653e+10        perf-stat.ps.cpu-cycles
    457693 ±  6%     +85.4%     848579 ±  2%  perf-stat.ps.dTLB-load-misses
 5.451e+09           -64.5%  1.937e+09        perf-stat.ps.dTLB-loads
     57782 ±  5%     +58.3%      91486 ±  3%  perf-stat.ps.dTLB-store-misses
  8.31e+08           -29.2%  5.884e+08        perf-stat.ps.dTLB-stores
    149554 ±  2%     +37.7%     205884 ±  2%  perf-stat.ps.iTLB-loads
 2.926e+10           -68.2%   9.31e+09        perf-stat.ps.instructions
      2.68 ±  5%     -55.8%       1.19 ±  4%  perf-stat.ps.major-faults
    105070 ±  5%     +32.5%     139219 ±  7%  perf-stat.ps.node-load-misses
     43441 ± 11%     +16.2%      50497 ±  2%  perf-stat.ps.node-loads
     58710 ±  5%     +16.6%      68472 ±  8%  perf-stat.ps.node-store-misses
 1.984e+12           -65.3%  6.875e+11        perf-stat.total.instructions



***************************************************************************************************
lkp-cfl-e1: 16 threads 1 sockets Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz (Coffee Lake) with 32G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/option_a/rootfs/tbox_group/test/testcase:
  gcc-11/performance/x86_64-rhel-8.3/Medium/debian-x86_64-phoronix/lkp-cfl-e1/astcenc-1.1.0/phoronix-test-suite

commit: 
  a525b95417 ("cpufreq/amd-pstate: Remove fast_switch_possible flag from active driver")
  278dfbd17c ("cpufreq: Return failure if fast_switch is not set and fast_switch_possible is set")

a525b954173e74cd 278dfbd17c5d0767834471d7fea 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :6          100%           6:6     kmsg.cpufreq:fast_switch_possible_is_enabled_but_fast_switch_callback_is_not_set
         %stddev     %change         %stddev
             \          |                \  
      6.51          +106.6%      13.45        phoronix-test-suite.astcenc.Medium.seconds
     41.10          +223.2%     132.82        phoronix-test-suite.time.elapsed_time
     41.10          +223.2%     132.82        phoronix-test-suite.time.elapsed_time.max
      1674         +1056.2%      19360 ±  6%  phoronix-test-suite.time.involuntary_context_switches
    236.67           +56.9%     371.33        phoronix-test-suite.time.percent_of_cpu_this_job_got
     96.53          +406.6%     489.06        phoronix-test-suite.time.user_time
 5.719e+08          +161.8%  1.497e+09        cpuidle..time
    609347          +172.7%    1661621        cpuidle..usage
      6795           +16.3%       7902        meminfo.Active(anon)
    197921 ±  7%     +73.9%     344256 ±  2%  meminfo.AnonHugePages
    469519           +19.9%     562889        meminfo.AnonPages
     82.86           -12.8       70.05        mpstat.cpu.all.idle%
      0.72 ± 76%      -0.6        0.14 ±  4%  mpstat.cpu.all.iowait%
      0.63            +2.3        2.94        mpstat.cpu.all.irq%
      0.04 ± 11%      +0.1        0.16 ±  3%  mpstat.cpu.all.soft%
      0.43 ±  2%      +0.3        0.70        mpstat.cpu.all.sys%
     15.32           +10.7       26.00        mpstat.cpu.all.usr%
     82.83           -15.5%      70.00        vmstat.cpu.id
     14.00           +78.6%      25.00        vmstat.cpu.us
      2578 ± 12%     -70.2%     769.17 ±  2%  vmstat.io.bi
      2.00          +100.0%       4.00        vmstat.procs.r
      1565           +26.7%       1984        vmstat.system.cs
     15803            +4.2%      16461        vmstat.system.in
      1698           +16.3%       1976        proc-vmstat.nr_active_anon
     28204            +6.8%      30122        proc-vmstat.nr_active_file
    117478           +19.9%     140853        proc-vmstat.nr_anon_pages
     96.33 ±  7%     +74.2%     167.83 ±  2%  proc-vmstat.nr_anon_transparent_hugepages
    296875            +8.1%     320871        proc-vmstat.nr_inactive_anon
      6598            +2.5%       6764        proc-vmstat.nr_kernel_stack
     18110            +1.2%      18332        proc-vmstat.nr_mapped
      1365            +7.0%       1460        proc-vmstat.nr_page_table_pages
     15776            -1.9%      15472        proc-vmstat.nr_slab_unreclaimable
      1698           +16.3%       1976        proc-vmstat.nr_zone_active_anon
     28204            +6.8%      30122        proc-vmstat.nr_zone_active_file
    296875            +8.1%     320871        proc-vmstat.nr_zone_inactive_anon
    290802 ±  2%     +40.6%     408922        proc-vmstat.numa_hit
    290802 ±  2%     +40.7%     409089        proc-vmstat.numa_local
     18530            +4.7%      19409        proc-vmstat.pgactivate
    828771           +17.9%     976790        proc-vmstat.pgalloc_normal
    338161 ±  3%     +56.3%     528417        proc-vmstat.pgfault
    770641           +19.0%     917408        proc-vmstat.pgfree
     31372 ±  3%    +108.5%      65408        proc-vmstat.pgreuse
    465920          +150.6%    1167488        proc-vmstat.unevictable_pgs_scanned
     60.29           +56.9%      94.59        turbostat.Any%C0
    679.67           -63.6%     247.50        turbostat.Avg_MHz
     16.46           +14.5       30.92        turbostat.Busy%
      4144           -80.7%     800.00        turbostat.Bzy_MHz
      1700 ± 20%   +1986.3%      35467 ±  4%  turbostat.C1
      0.01            +0.0        0.02 ± 20%  turbostat.C1%
      0.06 ± 11%      -0.0        0.04 ± 23%  turbostat.C10%
      5370 ± 23%    +492.7%      31828 ±  2%  turbostat.C1E
      0.24 ± 52%      -0.1        0.13 ±  3%  turbostat.C1E%
      0.55 ± 72%      -0.4        0.13 ±  2%  turbostat.C3%
    327658          +127.0%     743779        turbostat.C6
     45.83           -13.8       32.04        turbostat.C6%
    253362          +229.7%     835400        turbostat.C8
     15.34           -15.9%      12.89        turbostat.CPU%c1
      0.83 ± 66%     -77.9%       0.18 ±  4%  turbostat.CPU%c3
     26.89           -37.2%      16.88        turbostat.CPU%c7
     28.37           -90.5%       2.69        turbostat.CorWatt
     55.00 ±  2%     -44.8%      30.33        turbostat.CoreTmp
    697284          +220.7%    2236189        turbostat.IRQ
     11573 ±  3%     -88.6%       1320 ± 14%  turbostat.POLL
      6.93           -92.2%       0.54 ± 13%  turbostat.Pkg%pc2
     21.26 ±  2%     -78.8%       4.51        turbostat.Pkg%pc3
     54.50 ±  2%     -42.8%      31.17        turbostat.PkgTmp
     28.82           -89.0%       3.17        turbostat.PkgWatt
    166.20           +92.8%     320.39        turbostat.Totl%C0
     33915 ± 16%    +104.7%      69438 ± 21%  sched_debug.cfs_rq:/.load.avg
    127082 ±  5%    +189.1%     367447 ± 47%  sched_debug.cfs_rq:/.load.max
     43893 ±  8%    +167.8%     117549 ± 33%  sched_debug.cfs_rq:/.load.stddev
    210.50 ± 31%     -53.3%      98.31 ± 17%  sched_debug.cfs_rq:/.load_avg.avg
      1208 ± 20%     -52.4%     575.11 ± 40%  sched_debug.cfs_rq:/.load_avg.max
     19.17 ± 50%     -77.8%       4.25 ± 67%  sched_debug.cfs_rq:/.load_avg.min
    321.23 ± 29%     -48.4%     165.60 ± 34%  sched_debug.cfs_rq:/.load_avg.stddev
      9895 ± 32%   +1532.2%     161504 ± 38%  sched_debug.cfs_rq:/.min_vruntime.avg
     16191 ± 22%   +1115.3%     196783 ± 35%  sched_debug.cfs_rq:/.min_vruntime.max
      5611 ± 19%   +2491.5%     145426 ± 40%  sched_debug.cfs_rq:/.min_vruntime.min
      2882 ± 24%    +363.4%      13357 ± 29%  sched_debug.cfs_rq:/.min_vruntime.stddev
      0.49           -14.9%       0.42 ±  8%  sched_debug.cfs_rq:/.nr_running.stddev
    735.49 ±  9%     -53.0%     345.89 ± 11%  sched_debug.cfs_rq:/.runnable_avg.avg
    214.50 ± 66%     -71.7%      60.78 ± 31%  sched_debug.cfs_rq:/.runnable_avg.min
      3969 ± 83%    +985.0%      43070 ± 53%  sched_debug.cfs_rq:/.spread0.max
      2884 ± 24%    +363.2%      13358 ± 29%  sched_debug.cfs_rq:/.spread0.stddev
    705.35 ±  9%     -52.8%     333.26 ± 11%  sched_debug.cfs_rq:/.util_avg.avg
     70.43 ± 11%     -27.1%      51.31 ± 29%  sched_debug.cfs_rq:/.util_est_enqueued.avg
    790.50           -57.0%     340.19 ± 44%  sched_debug.cfs_rq:/.util_est_enqueued.max
    195.01           -50.4%      96.73 ± 35%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
    550906 ± 19%     +47.6%     813196 ±  9%  sched_debug.cpu.avg_idle.avg
     35231          +137.2%      83557 ± 17%  sched_debug.cpu.clock_task.min
    485.08 ± 14%     +27.4%     617.87 ±  9%  sched_debug.cpu.curr->pid.avg
      1171           +97.3%       2311 ± 12%  sched_debug.cpu.curr->pid.max
    526.49 ±  2%     +54.7%     814.24 ± 12%  sched_debug.cpu.curr->pid.stddev
      4635 ±  2%    +128.9%      10611 ± 17%  sched_debug.cpu.nr_switches.avg
     10915 ± 18%     +90.5%      20797 ± 13%  sched_debug.cpu.nr_switches.max
      1339 ± 19%    +195.5%       3959 ± 27%  sched_debug.cpu.nr_switches.min
      2524 ± 10%     +89.8%       4792 ± 22%  sched_debug.cpu.nr_switches.stddev
     44326          +112.3%      94104 ± 16%  sched_debug.sched_clk
     87.63 ±  5%     -56.1%      38.49        perf-stat.i.MPKI
 1.616e+09           -64.6%  5.725e+08        perf-stat.i.branch-instructions
      5.71 ±  3%      -1.1        4.59        perf-stat.i.branch-miss-rate%
  64667069           -61.2%   25072928        perf-stat.i.branch-misses
      7.54            -1.1        6.47        perf-stat.i.cache-miss-rate%
   6852594           -46.7%    3654271        perf-stat.i.cache-misses
 4.577e+08           -63.0%  1.691e+08        perf-stat.i.cache-references
      1350           +40.7%       1899        perf-stat.i.context-switches
      2.07 ±  4%     -53.8%       0.96        perf-stat.i.cpi
 1.103e+10           -65.4%   3.82e+09        perf-stat.i.cpu-cycles
     29.77 ±  2%     +10.2%      32.81        perf-stat.i.cpu-migrations
      2461 ±  3%     -31.9%       1675        perf-stat.i.cycles-between-cache-misses
      0.39 ±  8%      -0.2        0.15 ±  8%  perf-stat.i.dTLB-load-miss-rate%
    804093 ± 10%     -37.4%     503607 ±  8%  perf-stat.i.dTLB-load-misses
 3.753e+09           -67.1%  1.236e+09        perf-stat.i.dTLB-loads
      0.07 ±  4%      -0.0        0.05 ± 13%  perf-stat.i.dTLB-store-miss-rate%
     74383 ±  2%     -25.9%      55137 ±  9%  perf-stat.i.dTLB-store-misses
 1.026e+09           -65.9%  3.495e+08        perf-stat.i.dTLB-stores
     73.52            -3.8       69.74        perf-stat.i.iTLB-load-miss-rate%
    190642 ±  3%     -15.4%     161312        perf-stat.i.iTLB-load-misses
  1.33e+10           -66.3%  4.476e+09        perf-stat.i.instructions
    107118 ± 17%     -67.2%      35109 ±  5%  perf-stat.i.instructions-per-iTLB-miss
      1.01           +18.4%       1.19        perf-stat.i.ipc
      4.27 ±  2%     -70.3%       1.27        perf-stat.i.major-faults
      0.14 ±  8%    -100.0%       0.00        perf-stat.i.metric.G/sec
      0.69           -65.3%       0.24        perf-stat.i.metric.GHz
    151.41 ± 21%     -73.9%      39.59 ±  2%  perf-stat.i.metric.K/sec
    284.19 ±  4%     -48.8%     145.40        perf-stat.i.metric.M/sec
      6118 ±  4%     -42.6%       3513        perf-stat.i.minor-faults
      1.09 ± 21%     -58.2%       0.46 ± 33%  perf-stat.i.node-load-misses
    243872 ±  2%     -60.2%      97050        perf-stat.i.node-loads
      1.36 ± 20%     -70.9%       0.39 ± 38%  perf-stat.i.node-store-misses
   1154861 ±  5%     -64.6%     409216 ±  3%  perf-stat.i.node-stores
      6123 ±  4%     -42.6%       3514        perf-stat.i.page-faults
     34.42            +9.6%      37.74        perf-stat.overall.MPKI
      4.00            +0.4        4.37        perf-stat.overall.branch-miss-rate%
      1.50            +0.7        2.17        perf-stat.overall.cache-miss-rate%
      0.83            +2.9%       0.85        perf-stat.overall.cpi
      1608           -35.1%       1043        perf-stat.overall.cycles-between-cache-misses
      0.02 ± 10%      +0.0        0.04 ±  9%  perf-stat.overall.dTLB-load-miss-rate%
      0.01 ±  2%      +0.0        0.02 ±  9%  perf-stat.overall.dTLB-store-miss-rate%
     74.02            -4.4       69.67        perf-stat.overall.iTLB-load-miss-rate%
     69782 ±  3%     -60.3%      27693        perf-stat.overall.instructions-per-iTLB-miss
      1.21            -2.8%       1.17        perf-stat.overall.ipc
 1.577e+09           -64.1%  5.665e+08        perf-stat.ps.branch-instructions
  63097630           -60.7%   24776524        perf-stat.ps.branch-misses
   6689008           -45.9%    3620492        perf-stat.ps.cache-misses
 4.465e+08           -62.6%  1.671e+08        perf-stat.ps.cache-references
      1318           +42.7%       1881        perf-stat.ps.context-switches
     15621            +1.7%      15880        perf-stat.ps.cpu-clock
 1.076e+10           -64.9%  3.778e+09        perf-stat.ps.cpu-cycles
     29.06 ±  2%     +11.9%      32.52        perf-stat.ps.cpu-migrations
    784798 ± 10%     -36.4%     499506 ±  8%  perf-stat.ps.dTLB-load-misses
 3.661e+09           -66.6%  1.223e+09        perf-stat.ps.dTLB-loads
     72623 ±  2%     -24.7%      54670 ±  9%  perf-stat.ps.dTLB-store-misses
 1.001e+09           -65.4%  3.463e+08        perf-stat.ps.dTLB-stores
    186142 ±  3%     -14.1%     159974        perf-stat.ps.iTLB-load-misses
 1.297e+10           -65.9%  4.429e+09        perf-stat.ps.instructions
      4.17 ±  2%     -70.0%       1.25        perf-stat.ps.major-faults
      5975 ±  4%     -41.8%       3478        perf-stat.ps.minor-faults
      1.07 ± 21%     -57.7%       0.45 ± 33%  perf-stat.ps.node-load-misses
    238147 ±  2%     -59.6%      96125        perf-stat.ps.node-loads
      1.32 ± 20%     -70.6%       0.39 ± 38%  perf-stat.ps.node-store-misses
   1127035 ±  5%     -64.0%     405385 ±  4%  perf-stat.ps.node-stores
      5979 ±  4%     -41.8%       3479        perf-stat.ps.page-faults
     15621            +1.7%      15880        perf-stat.ps.task-clock
  5.48e+11            +8.0%  5.918e+11        perf-stat.total.instructions
     65.13 ± 17%     -58.5        6.60 ± 24%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
     65.13 ± 17%     -58.4        6.70 ± 25%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
     16.88 ± 61%     -16.7        0.23 ±141%  perf-profile.calltrace.cycles-pp.exit_mmap.__mmput.exit_mm.do_exit.do_group_exit
     16.60 ± 46%     -16.2        0.45 ± 72%  perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
     13.65 ± 75%     -13.6        0.00        perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
     13.65 ± 75%     -13.6        0.00        perf-profile.calltrace.cycles-pp.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
     11.62 ± 77%     -11.4        0.24 ±142%  perf-profile.calltrace.cycles-pp.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
     11.62 ± 77%     -11.4        0.24 ±142%  perf-profile.calltrace.cycles-pp.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
     11.62 ± 77%     -11.4        0.24 ±142%  perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
     11.33 ± 78%     -11.3        0.00        perf-profile.calltrace.cycles-pp.zap_pmd_range.unmap_page_range.unmap_vmas.exit_mmap.__mmput
     11.33 ± 78%     -11.3        0.00        perf-profile.calltrace.cycles-pp.zap_pte_range.zap_pmd_range.unmap_page_range.unmap_vmas.exit_mmap
     10.87 ± 53%     -10.9        0.00        perf-profile.calltrace.cycles-pp.do_group_exit.get_signal.arch_do_signal_or_restart.exit_to_user_mode_loop.exit_to_user_mode_prepare
     10.87 ± 53%     -10.9        0.00        perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.get_signal.arch_do_signal_or_restart.exit_to_user_mode_loop
     10.87 ± 53%     -10.9        0.00        perf-profile.calltrace.cycles-pp.arch_do_signal_or_restart.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
     10.87 ± 53%     -10.9        0.00        perf-profile.calltrace.cycles-pp.get_signal.arch_do_signal_or_restart.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode
     10.30 ± 61%     -10.3        0.00        perf-profile.calltrace.cycles-pp.exit_mm.do_exit.do_group_exit.get_signal.arch_do_signal_or_restart
     10.30 ± 61%     -10.3        0.00        perf-profile.calltrace.cycles-pp.__mmput.exit_mm.do_exit.do_group_exit.get_signal
      9.48 ±128%      -9.5        0.00        perf-profile.calltrace.cycles-pp.__generic_file_write_iter.generic_file_write_iter.vfs_write.ksys_write.do_syscall_64
      9.48 ±128%      -9.5        0.00        perf-profile.calltrace.cycles-pp.generic_file_write_iter.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      9.48 ±128%      -9.5        0.00        perf-profile.calltrace.cycles-pp.generic_perform_write.__generic_file_write_iter.generic_file_write_iter.vfs_write.ksys_write
      8.10 ± 78%      -8.1        0.00        perf-profile.calltrace.cycles-pp.unmap_vmas.exit_mmap.__mmput.exit_mm.do_exit
      8.10 ± 78%      -8.1        0.00        perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.exit_mmap.__mmput.exit_mm
      6.58 ±103%      -6.4        0.23 ±141%  perf-profile.calltrace.cycles-pp.__mmput.exit_mm.do_exit.do_group_exit.__x64_sys_exit_group
      6.58 ±103%      -6.4        0.23 ±141%  perf-profile.calltrace.cycles-pp.exit_mm.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      6.01 ±110%      -6.0        0.00        perf-profile.calltrace.cycles-pp.page_remove_rmap.zap_pte_range.zap_pmd_range.unmap_page_range.unmap_vmas
      5.32 ±105%      -5.3        0.00        perf-profile.calltrace.cycles-pp.unmap_vmas.exit_mmap.__mmput.exec_mmap.begin_new_exec
      5.32 ±105%      -5.1        0.20 ±142%  perf-profile.calltrace.cycles-pp.__mmput.exec_mmap.begin_new_exec.load_elf_binary.search_binary_handler
      5.32 ±105%      -5.1        0.20 ±142%  perf-profile.calltrace.cycles-pp.exit_mmap.__mmput.exec_mmap.begin_new_exec.load_elf_binary
      0.00            +0.8        0.83 ± 34%  perf-profile.calltrace.cycles-pp.lapic_next_deadline.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      0.00            +1.0        0.97 ± 30%  perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry
      0.00            +1.0        0.99 ± 20%  perf-profile.calltrace.cycles-pp.update_rq_clock_task.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer
      0.00            +1.1        1.10 ± 26%  perf-profile.calltrace.cycles-pp.account_user_time.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
      0.00            +1.2        1.17 ± 34%  perf-profile.calltrace.cycles-pp.update_curr.task_tick_fair.scheduler_tick.update_process_times.tick_sched_handle
      0.00            +1.3        1.26 ± 35%  perf-profile.calltrace.cycles-pp.__do_softirq.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      0.00            +1.3        1.28 ± 36%  perf-profile.calltrace.cycles-pp.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.00            +1.3        1.34 ± 35%  perf-profile.calltrace.cycles-pp.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      0.00            +1.4        1.41 ± 32%  perf-profile.calltrace.cycles-pp.perf_rotate_context.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
      0.00            +1.4        1.45 ± 38%  perf-profile.calltrace.cycles-pp.sync_regs.asm_sysvec_apic_timer_interrupt
      0.00            +1.5        1.52 ± 48%  perf-profile.calltrace.cycles-pp.drm_fb_helper_damage_work.process_one_work.worker_thread.kthread.ret_from_fork
      0.00            +1.5        1.52 ± 48%  perf-profile.calltrace.cycles-pp.drm_fbdev_generic_helper_fb_dirty.drm_fb_helper_damage_work.process_one_work.worker_thread.kthread
      0.00            +1.6        1.60 ± 43%  perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork
      0.00            +1.6        1.63 ± 25%  perf-profile.calltrace.cycles-pp.update_load_avg.task_tick_fair.scheduler_tick.update_process_times.tick_sched_handle
      0.00            +1.7        1.65 ± 41%  perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork
      0.00            +1.9        1.86 ± 26%  perf-profile.calltrace.cycles-pp.asm_sysvec_irq_work
      0.00            +1.9        1.86 ± 26%  perf-profile.calltrace.cycles-pp.sysvec_irq_work.asm_sysvec_irq_work
      0.00            +1.9        1.86 ± 26%  perf-profile.calltrace.cycles-pp.__sysvec_irq_work.sysvec_irq_work.asm_sysvec_irq_work
      0.00            +1.9        1.86 ± 26%  perf-profile.calltrace.cycles-pp.irq_work_run.__sysvec_irq_work.sysvec_irq_work.asm_sysvec_irq_work
      0.00            +2.2        2.21 ± 16%  perf-profile.calltrace.cycles-pp.asm_sysvec_irq_work.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_irq_work
      0.00            +2.2        2.21 ± 16%  perf-profile.calltrace.cycles-pp.sysvec_irq_work.asm_sysvec_irq_work.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode
      0.00            +2.2        2.21 ± 16%  perf-profile.calltrace.cycles-pp.__sysvec_irq_work.sysvec_irq_work.asm_sysvec_irq_work.exit_to_user_mode_loop.exit_to_user_mode_prepare
      0.00            +2.2        2.21 ± 16%  perf-profile.calltrace.cycles-pp.irq_work_run.__sysvec_irq_work.sysvec_irq_work.asm_sysvec_irq_work.exit_to_user_mode_loop
      0.00            +2.2        2.21 ± 16%  perf-profile.calltrace.cycles-pp.irqentry_exit_to_user_mode.asm_sysvec_irq_work
      0.00            +2.2        2.21 ± 16%  perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_irq_work
      0.00            +2.2        2.21 ± 16%  perf-profile.calltrace.cycles-pp.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_irq_work
      0.00            +2.6        2.60 ± 17%  perf-profile.calltrace.cycles-pp.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      0.00            +3.0        2.96 ± 19%  perf-profile.calltrace.cycles-pp.io_serial_in.wait_for_lsr.serial8250_console_write.console_flush_all.console_unlock
      0.00            +3.6        3.62 ± 19%  perf-profile.calltrace.cycles-pp.wait_for_lsr.serial8250_console_write.console_flush_all.console_unlock.vprintk_emit
      0.00            +4.2        4.19 ± 20%  perf-profile.calltrace.cycles-pp.serial8250_console_write.console_flush_all.console_unlock.vprintk_emit._printk
      0.00            +4.8        4.76 ± 20%  perf-profile.calltrace.cycles-pp.irq_work_run_list.irq_work_run.__sysvec_irq_work.sysvec_irq_work.asm_sysvec_irq_work
      0.00            +4.8        4.76 ± 20%  perf-profile.calltrace.cycles-pp.irq_work_single.irq_work_run_list.irq_work_run.__sysvec_irq_work.sysvec_irq_work
      0.00            +4.8        4.76 ± 20%  perf-profile.calltrace.cycles-pp._printk.irq_work_single.irq_work_run_list.irq_work_run.__sysvec_irq_work
      0.00            +4.8        4.76 ± 20%  perf-profile.calltrace.cycles-pp.vprintk_emit._printk.irq_work_single.irq_work_run_list.irq_work_run
      0.00            +4.8        4.76 ± 20%  perf-profile.calltrace.cycles-pp.console_unlock.vprintk_emit._printk.irq_work_single.irq_work_run_list
      0.00            +4.8        4.76 ± 20%  perf-profile.calltrace.cycles-pp.console_flush_all.console_unlock.vprintk_emit._printk.irq_work_single
      0.00            +6.4        6.43 ±  9%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      0.00            +6.7        6.70 ±  7%  perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      0.00            +9.6        9.56 ± 11%  perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
      4.86 ±143%     +10.7       15.57 ± 23%  perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      0.00           +10.9       10.91 ± 13%  perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
      4.86 ±143%     +17.1       21.96 ± 21%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.00           +20.0       20.04 ± 25%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.00           +20.6       20.64 ± 25%  perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
     19.32 ± 61%     +22.1       41.42 ± 23%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
      0.58 ±223%     +22.2       22.76 ± 26%  perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.58 ±223%     +25.6       26.14 ± 27%  perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt
     16.94 ± 66%     +31.3       48.20 ± 17%  perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
     17.52 ± 65%     +33.4       50.92 ± 17%  perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     19.32 ± 61%     +33.5       52.83 ± 18%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
     17.52 ± 65%     +34.0       51.54 ± 16%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     17.52 ± 65%     +34.0       51.55 ± 16%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     17.52 ± 65%     +34.0       51.55 ± 16%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
     19.90 ± 60%     +37.5       57.38 ± 16%  perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
     65.13 ± 17%     -58.5        6.60 ± 24%  perf-profile.children.cycles-pp.do_syscall_64
     65.13 ± 17%     -58.4        6.70 ± 25%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     22.50 ± 42%     -22.0        0.50 ± 33%  perf-profile.children.cycles-pp.do_group_exit
     22.50 ± 42%     -22.0        0.50 ± 33%  perf-profile.children.cycles-pp.do_exit
     16.60 ± 46%     -16.1        0.50 ± 49%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
     10.87 ± 53%     -10.9        0.00        perf-profile.children.cycles-pp.arch_do_signal_or_restart
     10.87 ± 53%     -10.9        0.00        perf-profile.children.cycles-pp.get_signal
      0.00            +0.1        0.08 ± 11%  perf-profile.children.cycles-pp.perf_event_mmap
      0.00            +0.1        0.10 ± 38%  perf-profile.children.cycles-pp.do_anonymous_page
      0.00            +0.1        0.10 ± 41%  perf-profile.children.cycles-pp.run_timer_softirq
      0.00            +0.1        0.12 ± 61%  perf-profile.children.cycles-pp.__run_timers
      0.00            +0.1        0.12 ± 39%  perf-profile.children.cycles-pp.mas_alloc_nodes
      0.00            +0.1        0.13 ± 46%  perf-profile.children.cycles-pp.rcu_gp_fqs_loop
      0.00            +0.1        0.13 ± 69%  perf-profile.children.cycles-pp.flush_smp_call_function_queue
      0.00            +0.1        0.14 ± 28%  perf-profile.children.cycles-pp.irq_work_needs_cpu
      0.00            +0.1        0.14 ± 49%  perf-profile.children.cycles-pp.schedule_timeout
      0.00            +0.1        0.14 ± 46%  perf-profile.children.cycles-pp.perf_adjust_freq_unthr_context
      0.00            +0.1        0.14 ± 68%  perf-profile.children.cycles-pp.select_task_rq
      0.00            +0.1        0.14 ± 61%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.00            +0.2        0.15 ± 39%  perf-profile.children.cycles-pp.profile_tick
      0.00            +0.2        0.16 ± 94%  perf-profile.children.cycles-pp.enqueue_entity
      0.00            +0.2        0.16 ± 15%  perf-profile.children.cycles-pp.io_serial_out
      0.00            +0.2        0.16 ± 45%  perf-profile.children.cycles-pp.rcu_gp_kthread
      0.00            +0.2        0.16 ± 73%  perf-profile.children.cycles-pp.tick_nohz_irq_exit
      0.00            +0.2        0.16 ± 73%  perf-profile.children.cycles-pp.do_vmi_munmap
      0.00            +0.2        0.16 ± 73%  perf-profile.children.cycles-pp.do_vmi_align_munmap
      0.00            +0.2        0.16 ± 43%  perf-profile.children.cycles-pp.futex_wait
      0.00            +0.2        0.17 ± 50%  perf-profile.children.cycles-pp.hrtimer_forward
      0.00            +0.2        0.17 ± 52%  perf-profile.children.cycles-pp.drm_gem_vunmap_unlocked
      0.00            +0.2        0.17 ± 52%  perf-profile.children.cycles-pp.drm_gem_vunmap
      0.00            +0.2        0.18 ± 38%  perf-profile.children.cycles-pp.hrtimer_wakeup
      0.00            +0.2        0.18 ± 50%  perf-profile.children.cycles-pp.__x64_sys_futex
      0.00            +0.2        0.18 ± 50%  perf-profile.children.cycles-pp.do_futex
      0.00            +0.2        0.20 ± 47%  perf-profile.children.cycles-pp.kmem_cache_alloc
      0.00            +0.2        0.20 ± 18%  perf-profile.children.cycles-pp.cgroup_rstat_updated
      0.00            +0.2        0.21 ± 53%  perf-profile.children.cycles-pp.drm_client_buffer_vmap
      0.00            +0.2        0.21 ± 41%  perf-profile.children.cycles-pp.number
      0.00            +0.2        0.22 ± 49%  perf-profile.children.cycles-pp.hrtimer_next_event_without
      0.00            +0.2        0.22 ± 41%  perf-profile.children.cycles-pp.tick_check_oneshot_broadcast_this_cpu
      0.00            +0.2        0.22 ± 68%  perf-profile.children.cycles-pp.error_entry
      0.00            +0.2        0.22 ± 50%  perf-profile.children.cycles-pp.autoremove_wake_function
      0.00            +0.2        0.23 ± 36%  perf-profile.children.cycles-pp.__irqentry_text_end
      0.00            +0.2        0.24 ± 41%  perf-profile.children.cycles-pp.native_apic_mem_write
      0.00            +0.2        0.24 ± 47%  perf-profile.children.cycles-pp.nohz_balancer_kick
      0.00            +0.2        0.25 ± 40%  perf-profile.children.cycles-pp.irqentry_enter
      0.00            +0.3        0.25 ± 46%  perf-profile.children.cycles-pp.__wake_up_common
      0.00            +0.3        0.26 ± 49%  perf-profile.children.cycles-pp.hrtimer_active
      0.00            +0.3        0.26 ± 28%  perf-profile.children.cycles-pp.cpuidle_governor_latency_req
      0.00            +0.3        0.26 ± 38%  perf-profile.children.cycles-pp.task_tick_mm_cid
      0.00            +0.3        0.27 ± 37%  perf-profile.children.cycles-pp.acct_account_cputime
      0.00            +0.3        0.27 ± 64%  perf-profile.children.cycles-pp.enqueue_task_fair
      0.00            +0.3        0.28 ± 48%  perf-profile.children.cycles-pp.rb_insert_color
      0.00            +0.3        0.28 ± 55%  perf-profile.children.cycles-pp.activate_task
      0.00            +0.3        0.28 ± 39%  perf-profile.children.cycles-pp.__wake_up_common_lock
      0.00            +0.3        0.28 ± 64%  perf-profile.children.cycles-pp.trigger_load_balance
      0.00            +0.3        0.28 ± 51%  perf-profile.children.cycles-pp.timekeeping_advance
      0.00            +0.3        0.28 ± 51%  perf-profile.children.cycles-pp.update_wall_time
      0.00            +0.3        0.29 ± 53%  perf-profile.children.cycles-pp.irqtime_account_process_tick
      0.00            +0.3        0.29 ± 97%  perf-profile.children.cycles-pp.drm_atomic_helper_prepare_planes
      0.00            +0.3        0.29 ± 97%  perf-profile.children.cycles-pp.drm_gem_fb_vmap
      0.00            +0.3        0.30 ± 52%  perf-profile.children.cycles-pp.__do_huge_pmd_anonymous_page
      0.00            +0.3        0.30 ± 52%  perf-profile.children.cycles-pp.clear_huge_page
      0.00            +0.3        0.30 ± 48%  perf-profile.children.cycles-pp.account_process_tick
      0.00            +0.3        0.30 ± 50%  perf-profile.children.cycles-pp.update_rq_clock
      0.00            +0.3        0.31 ± 29%  perf-profile.children.cycles-pp.__calc_delta
      0.00            +0.3        0.31 ± 49%  perf-profile.children.cycles-pp.ttwu_do_activate
      0.00            +0.3        0.32 ± 66%  perf-profile.children.cycles-pp.ct_idle_exit
      0.00            +0.3        0.33 ± 62%  perf-profile.children.cycles-pp.get_next_timer_interrupt
      0.00            +0.3        0.33 ± 72%  perf-profile.children.cycles-pp.sprintf
      0.00            +0.3        0.34 ± 50%  perf-profile.children.cycles-pp.clear_page_erms
      0.00            +0.4        0.37 ± 28%  perf-profile.children.cycles-pp.pipe_write
      0.00            +0.4        0.37 ± 34%  perf-profile.children.cycles-pp.irq_work_tick
      0.00            +0.4        0.37 ± 52%  perf-profile.children.cycles-pp.check_cpu_stall
      0.00            +0.4        0.38 ± 50%  perf-profile.children.cycles-pp.rb_erase
      0.00            +0.4        0.38 ± 62%  perf-profile.children.cycles-pp.show_time_in_state
      0.00            +0.4        0.38 ± 62%  perf-profile.children.cycles-pp.show
      0.00            +0.4        0.38 ± 23%  perf-profile.children.cycles-pp.__cgroup_account_cputime_field
      0.00            +0.4        0.39 ± 60%  perf-profile.children.cycles-pp.vsnprintf
      0.00            +0.4        0.40 ± 42%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.00            +0.4        0.41 ± 59%  perf-profile.children.cycles-pp.perf_event_task_tick
      0.00            +0.4        0.41 ± 42%  perf-profile.children.cycles-pp.hrtimer_update_next_event
      0.00            +0.4        0.42 ± 27%  perf-profile.children.cycles-pp.wait_for_xmitr
      0.00            +0.4        0.42 ± 62%  perf-profile.children.cycles-pp.sysfs_kf_seq_show
      0.00            +0.4        0.42 ± 75%  perf-profile.children.cycles-pp.drm_gem_get_pages
      0.00            +0.4        0.42 ± 75%  perf-profile.children.cycles-pp.shmem_read_mapping_page_gfp
      0.00            +0.4        0.43 ± 25%  perf-profile.children.cycles-pp.dup_mmap
      0.00            +0.5        0.45 ± 23%  perf-profile.children.cycles-pp.dup_mm
      0.00            +0.5        0.46 ± 40%  perf-profile.children.cycles-pp.rb_next
      0.00            +0.5        0.47 ± 27%  perf-profile.children.cycles-pp.try_to_wake_up
      0.00            +0.5        0.47 ± 57%  perf-profile.children.cycles-pp.traverse
      0.00            +0.5        0.48 ± 11%  perf-profile.children.cycles-pp.__hrtimer_next_event_base
      0.00            +0.5        0.49 ± 63%  perf-profile.children.cycles-pp.run_posix_cpu_timers
      0.00            +0.5        0.49 ± 80%  perf-profile.children.cycles-pp.delay_tsc
      0.00            +0.5        0.50 ± 62%  perf-profile.children.cycles-pp.drm_gem_vmap_unlocked
      0.00            +0.5        0.50 ± 62%  perf-profile.children.cycles-pp.drm_gem_vmap
      0.00            +0.5        0.52 ± 31%  perf-profile.children.cycles-pp.lf
      0.00            +0.5        0.52 ± 31%  perf-profile.children.cycles-pp.con_scroll
      0.00            +0.5        0.52 ± 31%  perf-profile.children.cycles-pp.fbcon_scroll
      0.00            +0.5        0.52 ± 31%  perf-profile.children.cycles-pp.fbcon_redraw
      0.00            +0.5        0.52 ± 40%  perf-profile.children.cycles-pp.fbcon_putcs
      0.00            +0.5        0.54 ± 50%  perf-profile.children.cycles-pp.rcu_core
      0.00            +0.6        0.57 ± 30%  perf-profile.children.cycles-pp.vt_console_print
      0.00            +0.6        0.58 ± 38%  perf-profile.children.cycles-pp.__update_load_avg_se
      0.00            +0.6        0.60 ± 45%  perf-profile.children.cycles-pp.update_cfs_group
      0.00            +0.6        0.62 ± 28%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.00            +0.6        0.63 ± 31%  perf-profile.children.cycles-pp.copy_process
      0.00            +0.6        0.64 ± 29%  perf-profile.children.cycles-pp.__do_sys_clone
      0.00            +0.6        0.64 ± 29%  perf-profile.children.cycles-pp.kernel_clone
      0.00            +0.7        0.66 ± 47%  perf-profile.children.cycles-pp.tick_sched_do_timer
      0.00            +0.7        0.67 ± 47%  perf-profile.children.cycles-pp.seq_read_iter
      0.00            +0.7        0.73 ± 32%  perf-profile.children.cycles-pp.tick_nohz_next_event
      0.00            +0.7        0.73 ± 37%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.00            +0.7        0.75 ± 32%  perf-profile.children.cycles-pp.native_sched_clock
      0.00            +0.8        0.76 ± 67%  perf-profile.children.cycles-pp.drm_atomic_helper_commit_tail_rpm
      0.00            +0.8        0.76 ± 67%  perf-profile.children.cycles-pp.drm_atomic_helper_commit_planes
      0.00            +0.8        0.76 ± 67%  perf-profile.children.cycles-pp.ast_primary_plane_helper_atomic_update
      0.00            +0.8        0.76 ± 67%  perf-profile.children.cycles-pp.drm_fb_memcpy
      0.00            +0.8        0.76 ± 67%  perf-profile.children.cycles-pp.memcpy_toio
      0.00            +0.8        0.77 ± 27%  perf-profile.children.cycles-pp.rebalance_domains
      0.00            +0.8        0.82 ± 36%  perf-profile.children.cycles-pp._raw_spin_lock
      0.00            +0.9        0.85 ± 61%  perf-profile.children.cycles-pp.commit_tail
      0.00            +0.9        0.88 ± 23%  perf-profile.children.cycles-pp.timerqueue_add
      0.00            +0.9        0.88 ± 53%  perf-profile.children.cycles-pp.rcu_pending
      0.00            +0.9        0.94 ± 39%  perf-profile.children.cycles-pp.tick_irq_enter
      0.00            +1.0        0.96 ± 35%  perf-profile.children.cycles-pp.ktime_get_update_offsets_now
      0.00            +1.0        0.96 ± 22%  perf-profile.children.cycles-pp.read_tsc
      0.00            +1.0        0.98 ± 30%  perf-profile.children.cycles-pp.irqtime_account_irq
      0.00            +1.0        1.00 ± 36%  perf-profile.children.cycles-pp.timerqueue_del
      0.00            +1.0        1.00 ± 27%  perf-profile.children.cycles-pp.enqueue_hrtimer
      0.00            +1.0        1.02 ± 15%  perf-profile.children.cycles-pp.update_irq_load_avg
      0.00            +1.0        1.03 ± 31%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.00            +1.1        1.06 ± 29%  perf-profile.children.cycles-pp.arch_scale_freq_tick
      0.00            +1.1        1.10 ± 26%  perf-profile.children.cycles-pp.account_user_time
      0.00            +1.1        1.14 ± 64%  perf-profile.children.cycles-pp.drm_atomic_helper_commit
      0.00            +1.2        1.15 ± 62%  perf-profile.children.cycles-pp.drm_atomic_helper_dirtyfb
      0.00            +1.2        1.16 ± 28%  perf-profile.children.cycles-pp.irq_enter_rcu
      0.00            +1.2        1.16 ± 23%  perf-profile.children.cycles-pp.lapic_next_deadline
      0.00            +1.3        1.33 ± 17%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.00            +1.4        1.36 ± 45%  perf-profile.children.cycles-pp.rcu_sched_clock_irq
      0.00            +1.5        1.52 ± 48%  perf-profile.children.cycles-pp.drm_fb_helper_damage_work
      0.00            +1.5        1.52 ± 48%  perf-profile.children.cycles-pp.drm_fbdev_generic_helper_fb_dirty
      0.00            +1.6        1.57 ± 22%  perf-profile.children.cycles-pp.perf_rotate_context
      0.00            +1.6        1.58 ± 25%  perf-profile.children.cycles-pp.clockevents_program_event
      0.00            +1.6        1.60 ± 43%  perf-profile.children.cycles-pp.process_one_work
      0.00            +1.7        1.65 ± 41%  perf-profile.children.cycles-pp.worker_thread
      0.00            +1.7        1.66 ± 39%  perf-profile.children.cycles-pp.sync_regs
      0.00            +1.9        1.88 ± 23%  perf-profile.children.cycles-pp.update_load_avg
      0.00            +2.7        2.74 ± 20%  perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
      0.00            +3.6        3.56 ± 16%  perf-profile.children.cycles-pp.io_serial_in
      0.00            +3.9        3.90 ± 32%  perf-profile.children.cycles-pp.irqentry_exit_to_user_mode
      0.00            +4.0        4.04 ± 20%  perf-profile.children.cycles-pp.wait_for_lsr
      0.00            +4.2        4.19 ± 20%  perf-profile.children.cycles-pp.serial8250_console_write
      0.00            +4.8        4.76 ± 20%  perf-profile.children.cycles-pp.asm_sysvec_irq_work
      0.00            +4.8        4.76 ± 20%  perf-profile.children.cycles-pp.sysvec_irq_work
      0.00            +4.8        4.76 ± 20%  perf-profile.children.cycles-pp.__sysvec_irq_work
      0.00            +4.8        4.76 ± 20%  perf-profile.children.cycles-pp.irq_work_run
      0.00            +4.8        4.76 ± 20%  perf-profile.children.cycles-pp.irq_work_single
      0.00            +4.8        4.76 ± 20%  perf-profile.children.cycles-pp._printk
      0.00            +4.8        4.76 ± 20%  perf-profile.children.cycles-pp.vprintk_emit
      0.00            +4.8        4.78 ± 20%  perf-profile.children.cycles-pp.console_unlock
      0.00            +4.8        4.78 ± 20%  perf-profile.children.cycles-pp.console_flush_all
      0.00            +4.9        4.89 ± 20%  perf-profile.children.cycles-pp.irq_work_run_list
      4.86 ±143%     +10.9       15.80 ± 23%  perf-profile.children.cycles-pp.tick_sched_timer
      4.86 ±143%     +17.5       22.32 ± 21%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      4.86 ±143%     +21.9       26.72 ± 20%  perf-profile.children.cycles-pp.hrtimer_interrupt
     19.32 ± 61%     +22.1       41.41 ± 23%  perf-profile.children.cycles-pp.intel_idle
      4.86 ±143%     +22.7       27.60 ± 19%  perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
      5.44 ±123%     +27.2       32.68 ± 17%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
      5.44 ±123%     +31.4       36.88 ± 18%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
     17.52 ± 65%     +34.0       51.55 ± 16%  perf-profile.children.cycles-pp.start_secondary
     19.32 ± 61%     +34.4       53.68 ± 18%  perf-profile.children.cycles-pp.cpuidle_enter_state
     19.32 ± 61%     +34.5       53.82 ± 18%  perf-profile.children.cycles-pp.cpuidle_enter
     19.90 ± 60%     +36.8       56.72 ± 17%  perf-profile.children.cycles-pp.cpuidle_idle_call
     19.90 ± 60%     +37.5       57.37 ± 16%  perf-profile.children.cycles-pp.do_idle
     19.90 ± 60%     +37.5       57.38 ± 16%  perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     19.90 ± 60%     +37.5       57.38 ± 16%  perf-profile.children.cycles-pp.cpu_startup_entry
      6.01 ±110%      -6.0        0.02 ±223%  perf-profile.self.cycles-pp.page_remove_rmap
      0.00            +0.1        0.11 ± 69%  perf-profile.self.cycles-pp.timekeeping_advance
      0.00            +0.1        0.11 ± 40%  perf-profile.self.cycles-pp.irq_work_needs_cpu
      0.00            +0.1        0.13 ± 40%  perf-profile.self.cycles-pp.irq_work_run_list
      0.00            +0.1        0.14 ± 68%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.00            +0.1        0.14 ± 46%  perf-profile.self.cycles-pp.perf_adjust_freq_unthr_context
      0.00            +0.1        0.14 ± 20%  perf-profile.self.cycles-pp.tick_irq_enter
      0.00            +0.1        0.14 ± 71%  perf-profile.self.cycles-pp.nohz_balancer_kick
      0.00            +0.1        0.15 ± 56%  perf-profile.self.cycles-pp.number
      0.00            +0.2        0.15 ± 39%  perf-profile.self.cycles-pp.profile_tick
      0.00            +0.2        0.16 ± 15%  perf-profile.self.cycles-pp.io_serial_out
      0.00            +0.2        0.17 ± 50%  perf-profile.self.cycles-pp.hrtimer_forward
      0.00            +0.2        0.20 ± 51%  perf-profile.self.cycles-pp.tick_nohz_next_event
      0.00            +0.2        0.20 ± 18%  perf-profile.self.cycles-pp.cgroup_rstat_updated
      0.00            +0.2        0.21 ± 38%  perf-profile.self.cycles-pp.timerqueue_del
      0.00            +0.2        0.22 ± 41%  perf-profile.self.cycles-pp.tick_check_oneshot_broadcast_this_cpu
      0.00            +0.2        0.22 ± 68%  perf-profile.self.cycles-pp.error_entry
      0.00            +0.2        0.23 ± 45%  perf-profile.self.cycles-pp.reweight_entity
      0.00            +0.2        0.23 ± 36%  perf-profile.self.cycles-pp.__irqentry_text_end
      0.00            +0.2        0.24 ± 41%  perf-profile.self.cycles-pp.native_apic_mem_write
      0.00            +0.2        0.25 ± 47%  perf-profile.self.cycles-pp.hrtimer_active
      0.00            +0.2        0.25 ± 62%  perf-profile.self.cycles-pp.rb_insert_color
      0.00            +0.3        0.25 ± 38%  perf-profile.self.cycles-pp.task_tick_mm_cid
      0.00            +0.3        0.27 ± 37%  perf-profile.self.cycles-pp.acct_account_cputime
      0.00            +0.3        0.27 ± 69%  perf-profile.self.cycles-pp.trigger_load_balance
      0.00            +0.3        0.28 ± 28%  perf-profile.self.cycles-pp.perf_mux_hrtimer_handler
      0.00            +0.3        0.29 ± 53%  perf-profile.self.cycles-pp.irqtime_account_process_tick
      0.00            +0.3        0.30 ± 48%  perf-profile.self.cycles-pp.account_process_tick
      0.00            +0.3        0.31 ± 29%  perf-profile.self.cycles-pp.__calc_delta
      0.00            +0.3        0.31 ± 37%  perf-profile.self.cycles-pp.asm_sysvec_apic_timer_interrupt
      0.00            +0.3        0.32 ± 62%  perf-profile.self.cycles-pp.cpuidle_idle_call
      0.00            +0.3        0.32 ± 39%  perf-profile.self.cycles-pp.irq_work_tick
      0.00            +0.3        0.34 ± 50%  perf-profile.self.cycles-pp.clear_page_erms
      0.00            +0.4        0.36 ± 52%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.00            +0.4        0.37 ± 52%  perf-profile.self.cycles-pp.check_cpu_stall
      0.00            +0.4        0.38 ± 50%  perf-profile.self.cycles-pp.rb_erase
      0.00            +0.4        0.39 ± 54%  perf-profile.self.cycles-pp.rcu_sched_clock_irq
      0.00            +0.4        0.41 ± 59%  perf-profile.self.cycles-pp.rcu_pending
      0.00            +0.4        0.42 ± 13%  perf-profile.self.cycles-pp.__hrtimer_next_event_base
      0.00            +0.4        0.44 ± 70%  perf-profile.self.cycles-pp.task_tick_fair
      0.00            +0.4        0.44 ± 33%  perf-profile.self.cycles-pp.memcpy_toio
      0.00            +0.5        0.46 ± 40%  perf-profile.self.cycles-pp.rb_next
      0.00            +0.5        0.46 ± 26%  perf-profile.self.cycles-pp.irqtime_account_irq
      0.00            +0.5        0.49 ± 63%  perf-profile.self.cycles-pp.run_posix_cpu_timers
      0.00            +0.5        0.49 ± 80%  perf-profile.self.cycles-pp.delay_tsc
      0.00            +0.5        0.52 ± 56%  perf-profile.self.cycles-pp.update_process_times
      0.00            +0.5        0.55 ± 41%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.00            +0.6        0.57 ± 40%  perf-profile.self.cycles-pp.__update_load_avg_se
      0.00            +0.6        0.60 ± 24%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.00            +0.6        0.60 ± 45%  perf-profile.self.cycles-pp.update_cfs_group
      0.00            +0.6        0.63 ± 26%  perf-profile.self.cycles-pp.timerqueue_add
      0.00            +0.6        0.63 ± 26%  perf-profile.self.cycles-pp.account_user_time
      0.00            +0.6        0.65 ± 21%  perf-profile.self.cycles-pp.__sysvec_apic_timer_interrupt
      0.00            +0.7        0.67 ± 17%  perf-profile.self.cycles-pp.perf_rotate_context
      0.00            +0.7        0.73 ± 33%  perf-profile.self.cycles-pp.ktime_get_update_offsets_now
      0.00            +0.7        0.73 ± 30%  perf-profile.self.cycles-pp.native_sched_clock
      0.00            +0.8        0.75 ± 35%  perf-profile.self.cycles-pp.update_load_avg
      0.00            +0.8        0.81 ± 35%  perf-profile.self.cycles-pp._raw_spin_lock
      0.00            +0.8        0.84 ± 40%  perf-profile.self.cycles-pp.scheduler_tick
      0.00            +0.9        0.89 ± 28%  perf-profile.self.cycles-pp.hrtimer_interrupt
      0.00            +0.9        0.94 ± 25%  perf-profile.self.cycles-pp.read_tsc
      0.00            +1.0        1.02 ± 15%  perf-profile.self.cycles-pp.update_irq_load_avg
      0.00            +1.1        1.06 ± 29%  perf-profile.self.cycles-pp.arch_scale_freq_tick
      0.00            +1.2        1.16 ± 23%  perf-profile.self.cycles-pp.lapic_next_deadline
      0.00            +1.2        1.25 ± 25%  perf-profile.self.cycles-pp.__hrtimer_run_queues
      0.00            +1.3        1.27 ± 23%  perf-profile.self.cycles-pp.cpuidle_enter_state
      0.00            +1.3        1.33 ± 17%  perf-profile.self.cycles-pp.native_irq_return_iret
      0.00            +1.7        1.66 ± 39%  perf-profile.self.cycles-pp.sync_regs
      0.00            +3.6        3.56 ± 16%  perf-profile.self.cycles-pp.io_serial_in
     19.32 ± 61%     +22.1       41.41 ± 23%  perf-profile.self.cycles-pp.intel_idle



***************************************************************************************************
lkp-cfl-e1: 16 threads 1 sockets Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz (Coffee Lake) with 32G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/option_a/rootfs/tbox_group/test/testcase:
  gcc-11/performance/x86_64-rhel-8.3/pickle_pure_python/debian-x86_64-phoronix/lkp-cfl-e1/pyperformance-1.0.2/phoronix-test-suite

commit: 
  a525b95417 ("cpufreq/amd-pstate: Remove fast_switch_possible flag from active driver")
  278dfbd17c ("cpufreq: Return failure if fast_switch is not set and fast_switch_possible is set")

a525b954173e74cd 278dfbd17c5d0767834471d7fea 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :6          100%           6:6     kmsg.cpufreq:fast_switch_possible_is_enabled_but_fast_switch_callback_is_not_set
         %stddev     %change         %stddev
             \          |                \  
    391.00           -99.4%       2.22        phoronix-test-suite.pyperformance.pickle_pure_python.milliseconds
     91.97           +70.0%     156.34        phoronix-test-suite.time.elapsed_time
     91.97           +70.0%     156.34        phoronix-test-suite.time.elapsed_time.max
     75.83           +10.8%      84.00        phoronix-test-suite.time.percent_of_cpu_this_job_got
     68.51           +78.7%     122.45        phoronix-test-suite.time.user_time
     44.67 ±  3%      +9.1%      48.74        boot-time.boot
    578.11 ±  2%      +6.0%     612.59        boot-time.idle
 1.408e+09           +57.1%  2.211e+09        cpuidle..time
   1492421           +64.8%    2460130        cpuidle..usage
      6896           +18.6%       8177        meminfo.Active(anon)
    121274 ±  9%     +14.4%     138735 ±  2%  meminfo.AnonHugePages
    139.63           +49.3%     208.46        uptime.boot
      1999           +43.0%       2859        uptime.idle
     93.00            -5.7%      87.67        vmstat.cpu.id
      1112 ±  4%     -40.6%     660.50 ±  2%  vmstat.io.bi
      1468           +38.0%       2026        vmstat.system.cs
      0.30 ±  5%      -0.1        0.17 ±  2%  mpstat.cpu.all.iowait%
      0.59            +2.5        3.06        mpstat.cpu.all.irq%
      0.04 ±  7%      +0.2        0.20        mpstat.cpu.all.soft%
      0.29            +0.5        0.83        mpstat.cpu.all.sys%
      5.09            +2.5        7.59        mpstat.cpu.all.usr%
      1723           +18.6%       2044        proc-vmstat.nr_active_anon
     92126            -1.7%      90534        proc-vmstat.nr_anon_pages
      6684            +1.6%       6789        proc-vmstat.nr_kernel_stack
     15983            -2.6%      15569        proc-vmstat.nr_slab_unreclaimable
      1723           +18.6%       2044        proc-vmstat.nr_zone_active_anon
    691105           +15.3%     796970        proc-vmstat.numa_hit
    691096           +15.3%     796942        proc-vmstat.numa_local
     16545            +4.3%      17256        proc-vmstat.pgactivate
    748098           +17.2%     876972        proc-vmstat.pgalloc_normal
   1113450           +14.9%    1279681        proc-vmstat.pgfault
    696774           +18.2%     823764        proc-vmstat.pgfree
    347697            +8.8%     378312        proc-vmstat.pgreuse
    869760           +57.1%    1366784        proc-vmstat.unevictable_pgs_scanned
     78.73           +21.3%      95.53        turbostat.Any%C0
    279.33           -62.2%     105.50        turbostat.Avg_MHz
      6.12            +7.1       13.20        turbostat.Busy%
      4569           -82.5%     800.00        turbostat.Bzy_MHz
      2894 ±  6%   +1657.0%      50860        turbostat.C1
      0.01            +0.0        0.03        turbostat.C1%
     11256 ±  3%    +332.0%      48630        turbostat.C1E
     14141 ±  2%     +61.5%      22838 ±  3%  turbostat.C3
      0.22 ±  3%      -0.1        0.15        turbostat.C3%
    499577          +124.0%    1118904        turbostat.C6
     31.51            +9.4       40.94        turbostat.C6%
    937249           +30.0%    1218418        turbostat.C8
     61.61           -15.6       46.02        turbostat.C8%
      0.28 ± 12%     -28.7%       0.20 ±  2%  turbostat.CPU%c3
     31.36           +58.4%      49.69        turbostat.CPU%c6
     46.27           -54.4%      21.11        turbostat.CPU%c7
     24.55           -92.1%       1.95        turbostat.CorWatt
     52.50 ±  7%     -42.9%      30.00        turbostat.CoreTmp
   1519070           +68.8%    2564565        turbostat.IRQ
     26254           -93.4%       1721 ± 24%  turbostat.POLL
      3.21 ±  4%     -86.1%       0.44 ±  4%  turbostat.Pkg%pc2
     10.04 ±  2%     -62.6%       3.76 ±  2%  turbostat.Pkg%pc3
     51.83 ±  9%     -40.8%      30.67        turbostat.PkgTmp
     25.03           -90.3%       2.43        turbostat.PkgWatt
     97.03           +99.1%     193.14        turbostat.Totl%C0
      0.52 ± 23%     -18.7%       0.42 ±  8%  sched_debug.cfs_rq:/.h_nr_running.stddev
      9607 ± 15%    +152.8%      24290 ±  8%  sched_debug.cfs_rq:/.min_vruntime.avg
     16594 ± 20%    +249.3%      57957 ± 15%  sched_debug.cfs_rq:/.min_vruntime.max
      6011 ± 18%     +79.8%      10806 ±  6%  sched_debug.cfs_rq:/.min_vruntime.min
      2916 ± 29%    +387.4%      14213 ± 19%  sched_debug.cfs_rq:/.min_vruntime.stddev
    666.91 ± 29%     -47.1%     352.68 ± 21%  sched_debug.cfs_rq:/.runnable_avg.avg
    229.00 ± 93%     -78.1%      50.08 ± 41%  sched_debug.cfs_rq:/.runnable_avg.min
      3691 ± 73%    +741.5%      31063 ± 41%  sched_debug.cfs_rq:/.spread0.max
     -6899          +133.6%     -16116        sched_debug.cfs_rq:/.spread0.min
      2917 ± 29%    +387.4%      14220 ± 19%  sched_debug.cfs_rq:/.spread0.stddev
    637.35 ± 27%     -46.9%     338.25 ± 21%  sched_debug.cfs_rq:/.util_avg.avg
    212.83 ± 90%     -79.1%      44.39 ± 50%  sched_debug.cfs_rq:/.util_avg.min
    730.42 ± 16%     -73.8%     191.67 ± 23%  sched_debug.cfs_rq:/.util_est_enqueued.max
    186.21 ± 19%     -66.5%      62.29 ± 21%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
    264982 ± 37%     +89.7%     502726 ± 12%  sched_debug.cpu.avg_idle.min
     51644 ± 24%     +76.5%      91142 ± 16%  sched_debug.cpu.clock.avg
     51645 ± 24%     +76.5%      91144 ± 16%  sched_debug.cpu.clock.max
     51643 ± 24%     +76.5%      91140 ± 16%  sched_debug.cpu.clock.min
     50245 ± 24%     +76.1%      88500 ± 15%  sched_debug.cpu.clock_task.avg
     51446 ± 23%     +74.5%      89762 ± 15%  sched_debug.cpu.clock_task.max
     40424 ± 26%     +95.8%      79135 ± 17%  sched_debug.cpu.clock_task.min
      1323 ± 25%     +73.1%       2290 ± 12%  sched_debug.cpu.curr->pid.max
    515.04 ± 26%     +53.7%     791.50 ± 13%  sched_debug.cpu.curr->pid.stddev
      4983 ± 22%    +104.8%      10204 ± 16%  sched_debug.cpu.nr_switches.avg
     11553 ± 42%    +110.0%      24267 ± 14%  sched_debug.cpu.nr_switches.max
      1502 ± 29%    +147.1%       3713 ± 17%  sched_debug.cpu.nr_switches.min
      2809 ± 35%     +93.1%       5425 ± 15%  sched_debug.cpu.nr_switches.stddev
     51643 ± 24%     +76.5%      91140 ± 16%  sched_debug.cpu_clk
     50769 ± 24%     +77.8%      90265 ± 16%  sched_debug.ktime
     49225 ± 22%     +81.8%      89499 ± 15%  sched_debug.sched_clk
     44.71 ±  4%     -14.6%      38.20        perf-stat.i.MPKI
 1.929e+09           -76.9%  4.446e+08        perf-stat.i.branch-instructions
      2.33            +0.4        2.78        perf-stat.i.branch-miss-rate%
  17917862           -33.9%   11845681        perf-stat.i.branch-misses
      3.39 ±  3%      +1.6        5.04        perf-stat.i.cache-miss-rate%
   3279255 ±  3%      -6.9%    3053819        perf-stat.i.cache-misses
 1.045e+08           -41.7%   60884324        perf-stat.i.cache-references
      1373           +43.8%       1974        perf-stat.i.context-switches
      1.15 ±  2%     -27.1%       0.84        perf-stat.i.cpi
  4.39e+09           -65.5%  1.515e+09        perf-stat.i.cpu-cycles
     24.91            -5.4%      23.57 ±  2%  perf-stat.i.cpu-migrations
      1930 ±  5%     -66.8%     640.60        perf-stat.i.cycles-between-cache-misses
      0.19 ±  4%      -0.1        0.13 ±  2%  perf-stat.i.dTLB-load-miss-rate%
    567898           -11.4%     503424        perf-stat.i.dTLB-load-misses
 2.756e+09           -79.0%  5.784e+08        perf-stat.i.dTLB-loads
      0.04 ±  5%      -0.0        0.03        perf-stat.i.dTLB-store-miss-rate%
     81667           -21.4%      64183        perf-stat.i.dTLB-store-misses
 1.267e+09           -79.2%  2.639e+08        perf-stat.i.dTLB-stores
     44.98            +1.2       46.14        perf-stat.i.iTLB-load-miss-rate%
    548983 ±  2%     -52.0%     263472        perf-stat.i.iTLB-load-misses
    900745           -62.6%     336446        perf-stat.i.iTLB-loads
 8.909e+09           -76.8%   2.07e+09        perf-stat.i.instructions
     15290           -48.3%       7907        perf-stat.i.instructions-per-iTLB-miss
      1.81           -25.5%       1.35        perf-stat.i.ipc
      2.37           -46.6%       1.26 ±  2%  perf-stat.i.major-faults
      0.27           -65.5%       0.09        perf-stat.i.metric.GHz
    117.79 ±  5%     -59.4%      47.79        perf-stat.i.metric.K/sec
    378.49           -77.7%      84.21        perf-stat.i.metric.M/sec
     11121           -30.1%       7775        perf-stat.i.minor-faults
      7.42 ± 10%     -36.3%       4.73 ±  4%  perf-stat.i.node-load-misses
    177094 ±  3%     -34.2%     116568        perf-stat.i.node-loads
      8.67 ± 19%     -40.8%       5.13 ± 12%  perf-stat.i.node-store-misses
    364565 ±  4%     -32.5%     246258        perf-stat.i.node-stores
     11123           -30.1%       7776        perf-stat.i.page-faults
     11.73          +151.2%      29.48        perf-stat.overall.MPKI
      0.93            +1.7        2.66        perf-stat.overall.branch-miss-rate%
      3.14 ±  2%      +1.9        5.01        perf-stat.overall.cache-miss-rate%
      0.49           +48.6%       0.73        perf-stat.overall.cpi
      1339 ±  3%     -63.0%     496.35        perf-stat.overall.cycles-between-cache-misses
      0.02            +0.1        0.09        perf-stat.overall.dTLB-load-miss-rate%
      0.01            +0.0        0.02        perf-stat.overall.dTLB-store-miss-rate%
     37.87            +6.0       43.87        perf-stat.overall.iTLB-load-miss-rate%
     16234 ±  2%     -51.7%       7839        perf-stat.overall.instructions-per-iTLB-miss
      2.03           -32.7%       1.37        perf-stat.overall.ipc
 1.907e+09           -76.9%  4.412e+08        perf-stat.ps.branch-instructions
  17720496           -33.7%   11756414        perf-stat.ps.branch-misses
   3243482 ±  3%      -6.5%    3031172        perf-stat.ps.cache-misses
 1.034e+08           -41.4%   60556271        perf-stat.ps.cache-references
      1358           +44.3%       1959        perf-stat.ps.context-switches
 4.341e+09           -65.3%  1.504e+09        perf-stat.ps.cpu-cycles
     24.65            -4.8%      23.46 ±  2%  perf-stat.ps.cpu-migrations
    561792           -11.0%     500237        perf-stat.ps.dTLB-load-misses
 2.726e+09           -78.9%   5.74e+08        perf-stat.ps.dTLB-loads
     80786           -21.0%      63822        perf-stat.ps.dTLB-store-misses
 1.253e+09           -79.1%   2.62e+08        perf-stat.ps.dTLB-stores
    542990 ±  2%     -51.7%     262035        perf-stat.ps.iTLB-load-misses
    890835           -62.4%     335305        perf-stat.ps.iTLB-loads
 8.811e+09           -76.7%  2.054e+09        perf-stat.ps.instructions
      2.34           -46.6%       1.25 ±  2%  perf-stat.ps.major-faults
     11000           -29.7%       7730        perf-stat.ps.minor-faults
      7.34 ± 10%     -36.0%       4.70 ±  5%  perf-stat.ps.node-load-misses
    175176 ±  3%     -33.9%     115833        perf-stat.ps.node-loads
      8.57 ± 19%     -40.4%       5.11 ± 13%  perf-stat.ps.node-store-misses
    360620 ±  4%     -31.9%     245454        perf-stat.ps.node-stores
     11002           -29.7%       7731        perf-stat.ps.page-faults
 8.234e+11           -60.7%  3.233e+11        perf-stat.total.instructions
     51.62 ± 14%     -47.4        4.20 ± 10%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
     51.62 ± 14%     -47.4        4.22 ± 10%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
     12.88 ± 77%     -12.9        0.00        perf-profile.calltrace.cycles-pp.exit_mmap.__mmput.exit_mm.do_exit.do_group_exit
     11.55 ± 64%     -11.6        0.00        perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
     12.09 ± 63%     -10.5        1.56 ±  8%  perf-profile.calltrace.cycles-pp.asm_exc_page_fault
     10.37 ± 94%     -10.4        0.00        perf-profile.calltrace.cycles-pp.zap_pte_range.zap_pmd_range.unmap_page_range.unmap_vmas.exit_mmap
     10.36 ± 92%     -10.4        0.00        perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     10.36 ± 92%     -10.4        0.00        perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     10.04 ± 66%     -10.0        0.00        perf-profile.calltrace.cycles-pp.do_group_exit.get_signal.arch_do_signal_or_restart.exit_to_user_mode_loop.exit_to_user_mode_prepare
     10.04 ± 66%     -10.0        0.00        perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.get_signal.arch_do_signal_or_restart.exit_to_user_mode_loop
      8.93 ± 82%      -8.9        0.00        perf-profile.calltrace.cycles-pp.zap_pmd_range.unmap_page_range.unmap_vmas.exit_mmap.__mmput
      9.15 ± 58%      -8.7        0.47 ± 71%  perf-profile.calltrace.cycles-pp.bprm_execve.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe
      8.53 ± 91%      -8.5        0.00        perf-profile.calltrace.cycles-pp.exit_mm.do_exit.do_group_exit.get_signal.arch_do_signal_or_restart
      8.53 ± 91%      -8.5        0.00        perf-profile.calltrace.cycles-pp.__mmput.exit_mm.do_exit.do_group_exit.get_signal
      9.15 ± 58%      -8.4        0.73 ± 18%  perf-profile.calltrace.cycles-pp.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe
      9.15 ± 58%      -8.4        0.73 ± 18%  perf-profile.calltrace.cycles-pp.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe
      7.70 ± 84%      -7.7        0.00        perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
      7.70 ± 84%      -7.7        0.00        perf-profile.calltrace.cycles-pp.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
      7.18 ± 84%      -7.2        0.00        perf-profile.calltrace.cycles-pp.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
      7.18 ± 84%      -7.2        0.00        perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
      7.18 ± 84%      -7.2        0.00        perf-profile.calltrace.cycles-pp.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
      7.01 ± 90%      -7.0        0.00        perf-profile.calltrace.cycles-pp.arch_do_signal_or_restart.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
      7.01 ± 90%      -7.0        0.00        perf-profile.calltrace.cycles-pp.get_signal.arch_do_signal_or_restart.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode
      5.99 ± 92%      -6.0        0.00        perf-profile.calltrace.cycles-pp.generic_file_write_iter.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.99 ± 92%      -6.0        0.00        perf-profile.calltrace.cycles-pp.__generic_file_write_iter.generic_file_write_iter.vfs_write.ksys_write.do_syscall_64
      5.99 ± 92%      -6.0        0.00        perf-profile.calltrace.cycles-pp.generic_perform_write.__generic_file_write_iter.generic_file_write_iter.vfs_write.ksys_write
      5.90 ± 68%      -5.9        0.00        perf-profile.calltrace.cycles-pp.unmap_vmas.exit_mmap.__mmput.exit_mm.do_exit
      5.90 ± 68%      -5.9        0.00        perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.exit_mmap.__mmput.exit_mm
      0.00            +0.7        0.70 ±  6%  perf-profile.calltrace.cycles-pp.timerqueue_del.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      0.00            +0.7        0.71 ± 20%  perf-profile.calltrace.cycles-pp.rcu_sched_clock_irq.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
      0.00            +0.7        0.71 ± 19%  perf-profile.calltrace.cycles-pp.enqueue_hrtimer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      0.00            +0.8        0.77 ± 16%  perf-profile.calltrace.cycles-pp.load_balance.rebalance_domains.__do_softirq.__irq_exit_rcu.sysvec_apic_timer_interrupt
      0.00            +0.8        0.79 ± 24%  perf-profile.calltrace.cycles-pp.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.cpuidle_idle_call.do_idle
      0.00            +0.8        0.80 ± 16%  perf-profile.calltrace.cycles-pp.update_blocked_averages.run_rebalance_domains.__do_softirq.__irq_exit_rcu.sysvec_apic_timer_interrupt
      0.00            +0.8        0.80 ± 23%  perf-profile.calltrace.cycles-pp.perf_rotate_context.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
      0.00            +0.8        0.85 ± 16%  perf-profile.calltrace.cycles-pp.run_rebalance_domains.__do_softirq.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.00            +1.0        0.97 ± 16%  perf-profile.calltrace.cycles-pp.rebalance_domains.__do_softirq.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.00            +1.2        1.20 ± 25%  perf-profile.calltrace.cycles-pp.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      0.00            +1.4        1.41 ± 36%  perf-profile.calltrace.cycles-pp.serial8250_console_write.console_flush_all.console_unlock.vprintk_emit._printk
      0.00            +1.4        1.42 ± 26%  perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry
      0.00            +1.5        1.46 ± 20%  perf-profile.calltrace.cycles-pp.tick_irq_enter.irq_enter_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      0.00            +1.5        1.50 ± 18%  perf-profile.calltrace.cycles-pp.irq_enter_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      0.00            +1.7        1.71 ± 23%  perf-profile.calltrace.cycles-pp.vprintk_emit._printk.irq_work_single.irq_work_run_list.irq_work_run
      0.00            +1.7        1.71 ± 23%  perf-profile.calltrace.cycles-pp.console_unlock.vprintk_emit._printk.irq_work_single.irq_work_run_list
      0.00            +1.7        1.71 ± 23%  perf-profile.calltrace.cycles-pp.console_flush_all.console_unlock.vprintk_emit._printk.irq_work_single
      0.00            +1.7        1.71 ± 23%  perf-profile.calltrace.cycles-pp.irq_work_run_list.irq_work_run.__sysvec_irq_work.sysvec_irq_work.asm_sysvec_irq_work
      0.00            +1.7        1.71 ± 23%  perf-profile.calltrace.cycles-pp.irq_work_single.irq_work_run_list.irq_work_run.__sysvec_irq_work.sysvec_irq_work
      0.00            +1.7        1.71 ± 23%  perf-profile.calltrace.cycles-pp._printk.irq_work_single.irq_work_run_list.irq_work_run.__sysvec_irq_work
      0.00            +2.3        2.32 ± 10%  perf-profile.calltrace.cycles-pp.__do_softirq.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      0.00            +2.4        2.37 ± 19%  perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
      0.00            +2.8        2.85 ± 10%  perf-profile.calltrace.cycles-pp.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      0.00            +3.5        3.52 ± 20%  perf-profile.calltrace.cycles-pp.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
      0.00            +4.3        4.27 ±  9%  perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
      0.00            +4.8        4.76 ±  8%  perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
      0.00            +5.4        5.42 ±  6%  perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      0.00            +5.6        5.62 ± 82%  perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.rest_init
      0.00            +5.8        5.83 ± 79%  perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.rest_init.arch_call_rest_init
      0.00            +5.9        5.90 ± 79%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.rest_init.arch_call_rest_init.start_kernel.x86_64_start_reservations
      0.00            +5.9        5.90 ± 79%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.rest_init.arch_call_rest_init.start_kernel
      0.00            +5.9        5.90 ± 79%  perf-profile.calltrace.cycles-pp.x86_64_start_kernel.secondary_startup_64_no_verify
      0.00            +5.9        5.90 ± 79%  perf-profile.calltrace.cycles-pp.x86_64_start_reservations.x86_64_start_kernel.secondary_startup_64_no_verify
      0.00            +5.9        5.90 ± 79%  perf-profile.calltrace.cycles-pp.start_kernel.x86_64_start_reservations.x86_64_start_kernel.secondary_startup_64_no_verify
      0.00            +5.9        5.90 ± 79%  perf-profile.calltrace.cycles-pp.arch_call_rest_init.start_kernel.x86_64_start_reservations.x86_64_start_kernel.secondary_startup_64_no_verify
      0.00            +5.9        5.90 ± 79%  perf-profile.calltrace.cycles-pp.rest_init.arch_call_rest_init.start_kernel.x86_64_start_reservations.x86_64_start_kernel
      0.00            +9.2        9.21 ±  2%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.00           +11.2       11.19 ±  3%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      0.00           +11.7       11.74 ±  3%  perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      0.00           +18.1       18.08        perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
      0.00           +20.5       20.55 ±  3%  perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
     21.44 ± 35%     +40.0       61.39 ±  2%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
     25.50 ± 46%     +53.7       79.24 ±  5%  perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
     25.50 ± 46%     +57.7       83.24        perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
     25.50 ± 46%     +58.2       83.74 ±  5%  perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     25.58 ± 45%     +59.2       84.78 ±  4%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     25.58 ± 45%     +59.2       84.82 ±  4%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     25.58 ± 45%     +59.2       84.82 ±  4%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
     25.66 ± 45%     +65.0       90.72        perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
     51.62 ± 14%     -47.4        4.21 ± 10%  perf-profile.children.cycles-pp.do_syscall_64
     51.62 ± 14%     -47.4        4.22 ± 10%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     17.21 ± 61%     -16.9        0.28 ± 20%  perf-profile.children.cycles-pp.do_group_exit
     17.21 ± 61%     -16.9        0.28 ± 20%  perf-profile.children.cycles-pp.do_exit
     17.42 ± 68%     -16.9        0.51 ± 10%  perf-profile.children.cycles-pp.__mmput
     17.34 ± 69%     -16.8        0.51 ± 10%  perf-profile.children.cycles-pp.exit_mmap
     13.04 ± 75%     -12.8        0.22 ± 30%  perf-profile.children.cycles-pp.exit_mm
     11.55 ± 64%     -11.2        0.39 ± 30%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
     12.90 ± 53%     -11.1        1.78 ± 12%  perf-profile.children.cycles-pp.asm_exc_page_fault
     10.04 ± 66%     -10.0        0.00        perf-profile.children.cycles-pp.get_signal
     10.04 ± 66%     -10.0        0.01 ±223%  perf-profile.children.cycles-pp.arch_do_signal_or_restart
      9.15 ± 58%      -8.5        0.62 ± 21%  perf-profile.children.cycles-pp.bprm_execve
      9.15 ± 58%      -8.4        0.73 ± 18%  perf-profile.children.cycles-pp.__x64_sys_execve
      9.15 ± 58%      -8.4        0.73 ± 18%  perf-profile.children.cycles-pp.do_execveat_common
      5.99 ± 92%      -6.0        0.00        perf-profile.children.cycles-pp.generic_perform_write
      5.99 ± 92%      -6.0        0.01 ±223%  perf-profile.children.cycles-pp.__generic_file_write_iter
      5.99 ± 92%      -6.0        0.01 ±223%  perf-profile.children.cycles-pp.generic_file_write_iter
      0.00            +0.1        0.09 ± 24%  perf-profile.children.cycles-pp.mas_store_prealloc
      0.00            +0.1        0.10 ± 38%  perf-profile.children.cycles-pp.hrtimer_get_next_event
      0.00            +0.1        0.10 ± 19%  perf-profile.children.cycles-pp.sprintf
      0.00            +0.1        0.11 ± 23%  perf-profile.children.cycles-pp.show
      0.00            +0.1        0.11 ± 23%  perf-profile.children.cycles-pp.show_time_in_state
      0.00            +0.1        0.12 ± 24%  perf-profile.children.cycles-pp.traverse
      0.00            +0.1        0.12 ± 28%  perf-profile.children.cycles-pp.hrtimer_wakeup
      0.00            +0.1        0.13 ± 17%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.00            +0.1        0.13 ± 19%  perf-profile.children.cycles-pp.irq_work_needs_cpu
      0.00            +0.1        0.13 ± 28%  perf-profile.children.cycles-pp.calc_global_load_tick
      0.00            +0.1        0.14 ± 36%  perf-profile.children.cycles-pp.pm_qos_read_value
      0.00            +0.1        0.14 ± 48%  perf-profile.children.cycles-pp.cpuidle_not_available
      0.00            +0.1        0.14 ± 33%  perf-profile.children.cycles-pp.schedule_idle
      0.00            +0.1        0.14 ± 34%  perf-profile.children.cycles-pp.sysfs_kf_seq_show
      0.00            +0.1        0.14 ± 45%  perf-profile.children.cycles-pp.rcu_nocb_flush_deferred_wakeup
      0.00            +0.1        0.15 ± 45%  perf-profile.children.cycles-pp.trigger_load_balance
      0.00            +0.2        0.16 ± 43%  perf-profile.children.cycles-pp.enqueue_entity
      0.00            +0.2        0.16 ± 38%  perf-profile.children.cycles-pp.call_cpuidle
      0.00            +0.2        0.17 ± 53%  perf-profile.children.cycles-pp.ct_irq_enter
      0.00            +0.2        0.17 ± 39%  perf-profile.children.cycles-pp.x86_pmu_disable
      0.00            +0.2        0.18 ± 48%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.00            +0.2        0.18 ± 43%  perf-profile.children.cycles-pp.update_load_avg
      0.00            +0.2        0.19 ± 33%  perf-profile.children.cycles-pp.tick_nohz_tick_stopped
      0.00            +0.2        0.19 ± 22%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.00            +0.2        0.20 ± 47%  perf-profile.children.cycles-pp.task_tick_mm_cid
      0.00            +0.2        0.20 ± 39%  perf-profile.children.cycles-pp.enqueue_task_fair
      0.00            +0.2        0.21 ± 36%  perf-profile.children.cycles-pp.activate_task
      0.00            +0.2        0.22 ± 54%  perf-profile.children.cycles-pp.update_wall_time
      0.00            +0.2        0.22 ± 54%  perf-profile.children.cycles-pp.timekeeping_advance
      0.00            +0.2        0.22 ± 27%  perf-profile.children.cycles-pp.rb_insert_color
      0.00            +0.2        0.22 ± 30%  perf-profile.children.cycles-pp.ct_kernel_exit
      0.00            +0.2        0.22 ± 27%  perf-profile.children.cycles-pp.ct_nmi_enter
      0.00            +0.2        0.23 ± 24%  perf-profile.children.cycles-pp.rb_next
      0.00            +0.2        0.23 ± 27%  perf-profile.children.cycles-pp.ttwu_do_activate
      0.00            +0.2        0.24 ± 76%  perf-profile.children.cycles-pp.copy_page_to_iter
      0.00            +0.2        0.24 ± 76%  perf-profile.children.cycles-pp._copy_to_iter
      0.00            +0.2        0.24 ± 76%  perf-profile.children.cycles-pp.copyout
      0.00            +0.2        0.25 ± 32%  perf-profile.children.cycles-pp.tick_nohz_irq_exit
      0.00            +0.2        0.25 ± 32%  perf-profile.children.cycles-pp.rcu_core
      0.00            +0.3        0.26 ± 30%  perf-profile.children.cycles-pp.irq_work_tick
      0.00            +0.3        0.27 ± 29%  perf-profile.children.cycles-pp.try_to_wake_up
      0.00            +0.3        0.27 ± 47%  perf-profile.children.cycles-pp.get_cpu_device
      0.00            +0.3        0.28 ± 27%  perf-profile.children.cycles-pp.check_cpu_stall
      0.00            +0.3        0.30 ± 50%  perf-profile.children.cycles-pp.irqtime_account_process_tick
      0.00            +0.3        0.31 ± 73%  perf-profile.children.cycles-pp.perf_event_task_tick
      0.00            +0.3        0.31 ± 27%  perf-profile.children.cycles-pp.__intel_pmu_enable_all
      0.00            +0.3        0.31 ± 42%  perf-profile.children.cycles-pp.tsc_verify_tsc_adjust
      0.00            +0.3        0.32 ± 33%  perf-profile.children.cycles-pp.ktime_get_update_offsets_now
      0.00            +0.3        0.32 ± 38%  perf-profile.children.cycles-pp.arch_cpu_idle_enter
      0.00            +0.3        0.34 ± 36%  perf-profile.children.cycles-pp.tick_sched_do_timer
      0.00            +0.4        0.35 ±  9%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.00            +0.4        0.36 ± 42%  perf-profile.children.cycles-pp.ct_kernel_enter
      0.00            +0.4        0.37 ± 43%  perf-profile.children.cycles-pp.native_apic_mem_write
      0.00            +0.4        0.39 ± 23%  perf-profile.children.cycles-pp.get_next_timer_interrupt
      0.00            +0.4        0.39 ± 17%  perf-profile.children.cycles-pp.nr_iowait_cpu
      0.00            +0.4        0.40 ± 23%  perf-profile.children.cycles-pp.rb_erase
      0.00            +0.4        0.40 ± 31%  perf-profile.children.cycles-pp.local_clock
      0.00            +0.4        0.41 ± 30%  perf-profile.children.cycles-pp.tick_check_oneshot_broadcast_this_cpu
      0.08 ±223%      +0.4        0.50 ± 14%  perf-profile.children.cycles-pp.update_sd_lb_stats
      0.00            +0.5        0.46 ± 21%  perf-profile.children.cycles-pp.run_posix_cpu_timers
      0.08 ±223%      +0.5        0.55 ± 15%  perf-profile.children.cycles-pp.find_busiest_group
      0.00            +0.5        0.47 ± 20%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.00            +0.5        0.48 ± 26%  perf-profile.children.cycles-pp.__update_blocked_fair
      0.00            +0.5        0.49 ± 36%  perf-profile.children.cycles-pp.hrtimer_update_next_event
      0.00            +0.5        0.50 ± 25%  perf-profile.children.cycles-pp.irqentry_enter
      0.00            +0.5        0.51 ± 22%  perf-profile.children.cycles-pp.lapic_next_deadline
      0.00            +0.5        0.52 ± 33%  perf-profile.children.cycles-pp.hrtimer_next_event_without
      0.00            +0.5        0.52 ± 29%  perf-profile.children.cycles-pp.update_irq_load_avg
      0.00            +0.5        0.54 ± 26%  perf-profile.children.cycles-pp.cpuidle_governor_latency_req
      0.00            +0.6        0.55 ± 11%  perf-profile.children.cycles-pp.arch_scale_freq_tick
      0.00            +0.6        0.56 ± 29%  perf-profile.children.cycles-pp.ct_idle_exit
      0.00            +0.6        0.57 ± 18%  perf-profile.children.cycles-pp.tick_nohz_stop_idle
      0.00            +0.6        0.57 ± 22%  perf-profile.children.cycles-pp.read_tsc
      0.00            +0.6        0.58 ± 28%  perf-profile.children.cycles-pp.rcu_pending
      0.00            +0.6        0.59 ± 20%  perf-profile.children.cycles-pp.native_sched_clock
      0.00            +0.6        0.61 ±  6%  perf-profile.children.cycles-pp.__hrtimer_next_event_base
      0.00            +0.7        0.68 ± 25%  perf-profile.children.cycles-pp.timerqueue_add
      0.00            +0.7        0.74 ±  5%  perf-profile.children.cycles-pp.timerqueue_del
      0.00            +0.8        0.75 ± 13%  perf-profile.children.cycles-pp.irqtime_account_irq
      0.00            +0.8        0.76 ± 25%  perf-profile.children.cycles-pp.update_rq_clock_task
      0.08 ±223%      +0.8        0.84 ± 15%  perf-profile.children.cycles-pp.load_balance
      0.00            +0.8        0.76 ± 25%  perf-profile.children.cycles-pp.clockevents_program_event
      0.00            +0.8        0.78 ± 17%  perf-profile.children.cycles-pp.enqueue_hrtimer
      0.00            +0.8        0.80 ± 22%  perf-profile.children.cycles-pp.rcu_sched_clock_irq
      0.00            +0.9        0.89 ± 17%  perf-profile.children.cycles-pp.update_blocked_averages
      0.00            +0.9        0.89 ± 18%  perf-profile.children.cycles-pp.perf_rotate_context
      0.00            +0.9        0.92 ± 24%  perf-profile.children.cycles-pp.tick_nohz_next_event
      0.00            +0.9        0.95 ± 19%  perf-profile.children.cycles-pp.run_rebalance_domains
      0.08 ±223%      +1.0        1.04 ± 14%  perf-profile.children.cycles-pp.rebalance_domains
      0.00            +1.1        1.05 ± 17%  perf-profile.children.cycles-pp.ktime_get
      0.00            +1.3        1.34 ± 18%  perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
      0.00            +1.6        1.55 ± 22%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.00            +1.6        1.60 ± 18%  perf-profile.children.cycles-pp.tick_irq_enter
      0.00            +1.6        1.64 ± 16%  perf-profile.children.cycles-pp.irq_enter_rcu
      0.08 ±223%      +2.5        2.54 ± 10%  perf-profile.children.cycles-pp.__do_softirq
      0.00            +2.5        2.46 ±  7%  perf-profile.children.cycles-pp.asm_sysvec_irq_work
      0.00            +2.5        2.46 ±  7%  perf-profile.children.cycles-pp.sysvec_irq_work
      0.00            +2.5        2.46 ±  7%  perf-profile.children.cycles-pp.__sysvec_irq_work
      0.00            +2.5        2.46 ±  7%  perf-profile.children.cycles-pp.irq_work_run
      0.00            +2.5        2.46 ±  7%  perf-profile.children.cycles-pp.irq_work_single
      0.00            +2.5        2.46 ±  7%  perf-profile.children.cycles-pp._printk
      0.00            +2.5        2.52 ±  6%  perf-profile.children.cycles-pp.irq_work_run_list
      0.00            +2.8        2.76 ± 20%  perf-profile.children.cycles-pp.scheduler_tick
      0.08 ±223%      +3.0        3.13 ± 11%  perf-profile.children.cycles-pp.__irq_exit_rcu
      0.00            +3.8        3.76 ± 20%  perf-profile.children.cycles-pp.menu_select
      0.08 ±223%      +4.8        4.92 ±  8%  perf-profile.children.cycles-pp.update_process_times
      0.08 ±223%      +5.3        5.41 ±  7%  perf-profile.children.cycles-pp.tick_sched_handle
      0.08 ±223%      +5.8        5.90 ± 79%  perf-profile.children.cycles-pp.x86_64_start_kernel
      0.08 ±223%      +5.8        5.90 ± 79%  perf-profile.children.cycles-pp.x86_64_start_reservations
      0.08 ±223%      +5.8        5.90 ± 79%  perf-profile.children.cycles-pp.start_kernel
      0.08 ±223%      +5.8        5.90 ± 79%  perf-profile.children.cycles-pp.arch_call_rest_init
      0.08 ±223%      +5.8        5.90 ± 79%  perf-profile.children.cycles-pp.rest_init
      0.08 ±223%      +6.0        6.13 ±  4%  perf-profile.children.cycles-pp.tick_sched_timer
      0.08 ±223%      +9.8        9.85 ±  3%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.08 ±223%     +11.8       11.92 ±  3%  perf-profile.children.cycles-pp.hrtimer_interrupt
      0.08 ±223%     +12.4       12.48 ±  2%  perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
      0.16 ±223%     +18.6       18.80        perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
      0.16 ±223%     +20.1       20.24 ±  2%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
     21.44 ± 35%     +40.0       61.40 ±  2%  perf-profile.children.cycles-pp.intel_idle
     25.58 ± 45%     +59.1       84.64        perf-profile.children.cycles-pp.cpuidle_enter_state
     25.58 ± 45%     +59.2       84.82 ±  4%  perf-profile.children.cycles-pp.start_secondary
     25.58 ± 45%     +59.3       84.86        perf-profile.children.cycles-pp.cpuidle_enter
     25.58 ± 45%     +64.1       89.64        perf-profile.children.cycles-pp.cpuidle_idle_call
     25.66 ± 45%     +65.0       90.72        perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     25.66 ± 45%     +65.0       90.72        perf-profile.children.cycles-pp.cpu_startup_entry
     25.66 ± 45%     +65.0       90.72        perf-profile.children.cycles-pp.do_idle
      0.00            +0.1        0.13 ± 36%  perf-profile.self.cycles-pp.pm_qos_read_value
      0.00            +0.1        0.14 ± 48%  perf-profile.self.cycles-pp.cpuidle_not_available
      0.00            +0.1        0.14 ± 37%  perf-profile.self.cycles-pp.trigger_load_balance
      0.00            +0.1        0.14 ± 33%  perf-profile.self.cycles-pp.cpuidle_governor_latency_req
      0.00            +0.1        0.14 ± 45%  perf-profile.self.cycles-pp.rcu_nocb_flush_deferred_wakeup
      0.00            +0.1        0.15 ± 27%  perf-profile.self.cycles-pp.perf_mux_hrtimer_handler
      0.00            +0.2        0.15 ± 42%  perf-profile.self.cycles-pp.call_cpuidle
      0.00            +0.2        0.15 ± 32%  perf-profile.self.cycles-pp.timerqueue_del
      0.00            +0.2        0.16 ± 33%  perf-profile.self.cycles-pp.irqentry_enter
      0.00            +0.2        0.16 ± 40%  perf-profile.self.cycles-pp.x86_pmu_disable
      0.00            +0.2        0.18 ± 48%  perf-profile.self.cycles-pp.task_tick_mm_cid
      0.00            +0.2        0.18 ± 48%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.00            +0.2        0.19 ± 22%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.00            +0.2        0.20 ± 50%  perf-profile.self.cycles-pp.rcu_sched_clock_irq
      0.00            +0.2        0.20 ± 52%  perf-profile.self.cycles-pp.tsc_verify_tsc_adjust
      0.00            +0.2        0.20 ± 42%  perf-profile.self.cycles-pp.do_idle
      0.00            +0.2        0.20 ± 46%  perf-profile.self.cycles-pp.tick_nohz_stop_idle
      0.00            +0.2        0.21 ± 47%  perf-profile.self.cycles-pp.ct_kernel_enter
      0.00            +0.2        0.21 ± 32%  perf-profile.self.cycles-pp.ct_nmi_enter
      0.00            +0.2        0.22 ± 27%  perf-profile.self.cycles-pp.rb_insert_color
      0.00            +0.2        0.22 ± 20%  perf-profile.self.cycles-pp.cpuidle_enter
      0.00            +0.2        0.22 ± 34%  perf-profile.self.cycles-pp.tick_irq_enter
      0.00            +0.2        0.22 ± 29%  perf-profile.self.cycles-pp.rb_next
      0.00            +0.2        0.22 ± 37%  perf-profile.self.cycles-pp.load_balance
      0.00            +0.2        0.23 ± 25%  perf-profile.self.cycles-pp.irq_work_tick
      0.00            +0.2        0.24 ± 30%  perf-profile.self.cycles-pp.__irq_exit_rcu
      0.00            +0.2        0.24 ±  9%  perf-profile.self.cycles-pp.rebalance_domains
      0.00            +0.2        0.24 ± 25%  perf-profile.self.cycles-pp.update_rq_clock_task
      0.00            +0.2        0.24 ± 28%  perf-profile.self.cycles-pp.get_next_timer_interrupt
      0.00            +0.3        0.26 ± 23%  perf-profile.self.cycles-pp.hrtimer_interrupt
      0.00            +0.3        0.27 ± 52%  perf-profile.self.cycles-pp.get_cpu_device
      0.00            +0.3        0.28 ± 27%  perf-profile.self.cycles-pp.check_cpu_stall
      0.00            +0.3        0.28 ± 40%  perf-profile.self.cycles-pp.perf_rotate_context
      0.00            +0.3        0.28 ± 64%  perf-profile.self.cycles-pp.rcu_pending
      0.00            +0.3        0.30 ± 50%  perf-profile.self.cycles-pp.irqtime_account_process_tick
      0.00            +0.3        0.31 ± 27%  perf-profile.self.cycles-pp.__intel_pmu_enable_all
      0.00            +0.3        0.31 ± 37%  perf-profile.self.cycles-pp.__update_blocked_fair
      0.00            +0.3        0.34 ± 26%  perf-profile.self.cycles-pp.sysvec_apic_timer_interrupt
      0.00            +0.3        0.34 ±  6%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.00            +0.3        0.35 ± 39%  perf-profile.self.cycles-pp.scheduler_tick
      0.00            +0.4        0.35 ± 37%  perf-profile.self.cycles-pp.update_process_times
      0.00            +0.4        0.36 ± 44%  perf-profile.self.cycles-pp.native_apic_mem_write
      0.00            +0.4        0.38 ± 20%  perf-profile.self.cycles-pp.nr_iowait_cpu
      0.00            +0.4        0.39 ± 27%  perf-profile.self.cycles-pp.cpuidle_idle_call
      0.00            +0.4        0.40 ± 23%  perf-profile.self.cycles-pp.rb_erase
      0.00            +0.4        0.41 ± 30%  perf-profile.self.cycles-pp.tick_check_oneshot_broadcast_this_cpu
      0.00            +0.4        0.44 ± 12%  perf-profile.self.cycles-pp.irqtime_account_irq
      0.00            +0.5        0.46 ± 31%  perf-profile.self.cycles-pp.timerqueue_add
      0.00            +0.5        0.46 ± 21%  perf-profile.self.cycles-pp.run_posix_cpu_timers
      0.00            +0.5        0.51 ± 22%  perf-profile.self.cycles-pp.lapic_next_deadline
      0.00            +0.5        0.52 ± 29%  perf-profile.self.cycles-pp.update_irq_load_avg
      0.00            +0.5        0.54 ± 44%  perf-profile.self.cycles-pp.__hrtimer_run_queues
      0.00            +0.6        0.55 ± 11%  perf-profile.self.cycles-pp.arch_scale_freq_tick
      0.00            +0.6        0.55 ± 21%  perf-profile.self.cycles-pp.read_tsc
      0.00            +0.6        0.56 ± 12%  perf-profile.self.cycles-pp.__hrtimer_next_event_base
      0.00            +0.6        0.58 ± 19%  perf-profile.self.cycles-pp.native_sched_clock
      0.00            +0.6        0.60 ± 25%  perf-profile.self.cycles-pp.ktime_get
      0.00            +1.6        1.57 ± 29%  perf-profile.self.cycles-pp.menu_select
      0.00            +2.0        2.02 ± 18%  perf-profile.self.cycles-pp.cpuidle_enter_state
     21.44 ± 35%     +40.0       61.40 ±  2%  perf-profile.self.cycles-pp.intel_idle



***************************************************************************************************
lkp-cfl-d2: 12 threads 1 sockets Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz (Coffee Lake) with 32G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/need_x/option_a/rootfs/tbox_group/test/testcase:
  gcc-11/performance/x86_64-rhel-8.3/true/500px Copy From Window To Window/debian-x86_64-phoronix/lkp-cfl-d2/x11perf-1.1.1/phoronix-test-suite

commit: 
  a525b95417 ("cpufreq/amd-pstate: Remove fast_switch_possible flag from active driver")
  278dfbd17c ("cpufreq: Return failure if fast_switch is not set and fast_switch_possible is set")

a525b954173e74cd 278dfbd17c5d0767834471d7fea 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :6          100%           6:6     kmsg.cpufreq:fast_switch_possible_is_enabled_but_fast_switch_callback_is_not_set
         %stddev     %change         %stddev
             \          |                \  
    711.17 ± 13%    +756.8%       6093 ± 45%  phoronix-test-suite.time.involuntary_context_switches
    125556            -4.0%     120523        phoronix-test-suite.time.maximum_resident_set_size
    193366           -23.6%     147783        phoronix-test-suite.time.minor_page_faults
     35.17          +136.0%      83.00        phoronix-test-suite.time.percent_of_cpu_this_job_got
     28.44          +251.3%      99.90        phoronix-test-suite.time.user_time
   1411873 ±  7%     -79.4%     290969 ± 10%  phoronix-test-suite.time.voluntary_context_switches
     11105            -7.3%      10294        phoronix-test-suite.x11perf.500pxCopyFromWindowToWindow.operations___second
 1.596e+09           -10.0%  1.436e+09        cpuidle..time
   3803740 ±  2%     -31.3%    2612320        cpuidle..usage
      4.28            -4.0        0.32        mpstat.cpu.all.iowait%
      2.26 ±  2%      +1.4        3.70        mpstat.cpu.all.irq%
      0.20 ±  3%      +0.1        0.26        mpstat.cpu.all.soft%
      1.61 ±  2%      +0.6        2.23        mpstat.cpu.all.sys%
      2.87            +9.7       12.59        mpstat.cpu.all.usr%
     88.00            -9.1%      80.00        vmstat.cpu.id
      2.50 ± 20%    +380.0%      12.00        vmstat.cpu.us
      2093            -2.3%       2045        vmstat.io.bi
  10868294           -76.8%    2519224        vmstat.memory.cache
  21085319           +40.1%   29541029        vmstat.memory.free
      0.00         +1e+102%       1.00        vmstat.procs.r
     32259 ±  3%     -52.1%      15452 ±  2%  vmstat.system.cs
     18239            -7.5%      16870        vmstat.system.in
    459850            -9.1%     417876        meminfo.AnonPages
  10854060           -77.5%    2446909        meminfo.Cached
  10787077           -78.3%    2346035        meminfo.Committed_AS
   1211148 ±  2%      -7.4%    1121115        meminfo.Inactive(anon)
     99909           -24.9%      75030        meminfo.KReclaimable
    150648           -23.1%     115779        meminfo.Mapped
  22114323           +38.6%   30641210        meminfo.MemAvailable
  20999037           +40.7%   29538155        meminfo.MemFree
  11664521           -73.2%    3125406        meminfo.Memused
     22823           -71.3%       6547        meminfo.PageTables
     99909           -24.9%      75030        meminfo.SReclaimable
     96043           -35.6%      61884        meminfo.SUnreclaim
   9422747           -89.2%    1015386        meminfo.Shmem
    195953           -30.1%     136915        meminfo.Slab
   8667525           -96.4%     308501        meminfo.Unevictable
  12972755           -75.7%    3154603        meminfo.max_used_kB
      0.42 ± 21%     +77.8%       0.75 ± 21%  sched_debug.cfs_rq:/.h_nr_running.avg
     13959 ±  5%    +127.0%      31685 ± 12%  sched_debug.cfs_rq:/.min_vruntime.avg
     30077 ±  8%    +127.7%      68497 ± 16%  sched_debug.cfs_rq:/.min_vruntime.max
      7885 ±  9%    +118.6%      17233 ± 12%  sched_debug.cfs_rq:/.min_vruntime.min
      6353 ± 13%    +143.0%      15442 ± 18%  sched_debug.cfs_rq:/.min_vruntime.stddev
      0.41 ± 21%     +45.6%       0.60 ± 11%  sched_debug.cfs_rq:/.nr_running.avg
    370.63 ± 14%     +45.2%     537.98 ± 11%  sched_debug.cfs_rq:/.runnable_avg.avg
    961.25 ±  8%     +45.3%       1397 ±  6%  sched_debug.cfs_rq:/.runnable_avg.max
    259.86 ±  9%     +43.6%     373.22 ± 12%  sched_debug.cfs_rq:/.runnable_avg.stddev
      6354 ± 13%    +143.1%      15448 ± 18%  sched_debug.cfs_rq:/.spread0.stddev
    352.22 ± 13%     +32.7%     467.44 ± 11%  sched_debug.cfs_rq:/.util_avg.avg
    935.44 ±  7%     +32.4%       1238 ±  6%  sched_debug.cfs_rq:/.util_avg.max
    256.68 ±  9%     +34.4%     345.04 ± 10%  sched_debug.cfs_rq:/.util_avg.stddev
     76.34 ± 22%     +78.9%     136.54 ± 21%  sched_debug.cfs_rq:/.util_est_enqueued.avg
    753033 ±  6%     +18.8%     894877 ± 10%  sched_debug.cpu.avg_idle.avg
      0.54 ± 42%    +243.8%       1.85 ± 13%  sched_debug.cpu.clock.stddev
      0.00 ± 12%     +78.0%       0.00 ± 13%  sched_debug.cpu.next_balance.stddev
      0.43 ± 23%     +74.1%       0.75 ± 18%  sched_debug.cpu.nr_running.avg
    199704 ± 17%     -53.6%      92716 ± 19%  sched_debug.cpu.nr_switches.avg
    449460 ± 20%     -53.4%     209625 ± 18%  sched_debug.cpu.nr_switches.max
     58141 ± 39%     -71.6%      16484 ± 29%  sched_debug.cpu.nr_switches.min
    117521 ± 17%     -51.4%      57142 ± 25%  sched_debug.cpu.nr_switches.stddev
    114963            -9.1%     104454        proc-vmstat.nr_anon_pages
    553299           +38.5%     766481        proc-vmstat.nr_dirty_background_threshold
   1107953           +38.5%    1534837        proc-vmstat.nr_dirty_threshold
   2714060           -77.4%     612214        proc-vmstat.nr_file_pages
   5249754           +40.7%    7384644        proc-vmstat.nr_free_pages
    302779 ±  2%      -7.4%     280420        proc-vmstat.nr_inactive_anon
      6151            +1.4%       6236        proc-vmstat.nr_kernel_stack
     37671           -23.2%      28939        proc-vmstat.nr_mapped
      5705           -71.3%       1638        proc-vmstat.nr_page_table_pages
   2355704           -89.2%     253805        proc-vmstat.nr_shmem
     24977           -24.9%      18757        proc-vmstat.nr_slab_reclaimable
     24010           -35.6%      15470        proc-vmstat.nr_slab_unreclaimable
   2166906           -96.4%      76929        proc-vmstat.nr_unevictable
    302779 ±  2%      -7.4%     280420        proc-vmstat.nr_zone_inactive_anon
   2166906           -96.4%      76929        proc-vmstat.nr_zone_unevictable
  10747274 ±  3%     -92.9%     765514        proc-vmstat.numa_hit
  10756248 ±  3%     -92.9%     765422        proc-vmstat.numa_local
  10997298 ±  3%     -91.8%     902283        proc-vmstat.pgalloc_normal
    501839            +6.5%     534236 ±  2%  proc-vmstat.pgfault
  10872643 ±  3%     -92.9%     775875        proc-vmstat.pgfree
     55315           +28.2%      70928 ±  2%  proc-vmstat.pgreuse
  10027217 ±  3%     -97.0%     305188        proc-vmstat.unevictable_pgs_culled
  10027217 ±  3%     -97.0%     305188        proc-vmstat.unevictable_pgs_rescued
  10027229 ±  3%     -97.0%     305217        proc-vmstat.unevictable_pgs_scanned
     51.17           +88.2%      96.32        turbostat.Any%C0
    340.83           -51.4%     165.67        turbostat.Avg_MHz
      8.10           +12.6       20.71        turbostat.Busy%
      4202           -81.0%     800.00        turbostat.Bzy_MHz
    849019           -12.5%     743127        turbostat.C1E
      3.53            -1.0        2.48        turbostat.C1E%
     72327 ±  2%     -66.7%      24104 ±  2%  turbostat.C3
      2.99 ±  3%      -2.7        0.31 ±  2%  turbostat.C3%
    723419           +14.3%     827187        turbostat.C6
    848543           -20.8%     671679        turbostat.C8
     47.43           -12.0       35.46        turbostat.C8%
     19.64           +24.4%      24.45        turbostat.CPU%c1
      5.18 ±  3%     -92.3%       0.40 ±  4%  turbostat.CPU%c3
     27.31           -50.3%      13.58        turbostat.CPU%c7
     32.05           +32.8       64.90        turbostat.CPUGFX%
     18.82           -77.2%       4.28        turbostat.CorWatt
     69.50           -44.1%      38.83        turbostat.CoreTmp
     73.77           -12.0%      64.94        turbostat.GFX%C0
     26.07 ±  2%     +32.0%      34.42        turbostat.GFX%rc6
     11.01           -13.8%       9.48        turbostat.GFXWatt
   1003094 ± 11%     -99.4%       5641 ± 53%  turbostat.POLL
      0.10 ± 10%      -0.1        0.01 ± 35%  turbostat.POLL%
     68.83           -43.1%      39.17        turbostat.PkgTmp
     30.81           -52.5%      14.65        turbostat.PkgWatt
      2.72            -4.7%       2.59        turbostat.RAMWatt
     84.82          +168.2%     227.52        turbostat.Totl%C0
    100.09           -21.4%      78.69        perf-stat.i.MPKI
 4.224e+08           -21.1%  3.333e+08        perf-stat.i.branch-instructions
      2.16 ±  3%      +1.8        3.93        perf-stat.i.branch-miss-rate%
  12591395           +10.0%   13851976        perf-stat.i.branch-misses
  21154943           -12.8%   18457066 ±  2%  perf-stat.i.cache-misses
 1.278e+08           -21.3%  1.006e+08        perf-stat.i.cache-references
     32972 ±  3%     -52.4%      15699 ±  2%  perf-stat.i.context-switches
      2.39           -46.7%       1.27        perf-stat.i.cpi
 3.498e+09           -47.9%  1.823e+09        perf-stat.i.cpu-cycles
     26.99 ±  2%      +7.8%      29.10 ±  2%  perf-stat.i.cpu-migrations
    647.57 ± 10%     -67.8%     208.52 ±  6%  perf-stat.i.cycles-between-cache-misses
      0.21 ±  4%      -0.0        0.17 ±  3%  perf-stat.i.dTLB-load-miss-rate%
    616008 ±  3%     -22.4%     477839 ±  4%  perf-stat.i.dTLB-load-misses
 4.915e+08           -26.4%  3.616e+08        perf-stat.i.dTLB-loads
 2.558e+08           -33.0%  1.714e+08        perf-stat.i.dTLB-stores
    458573 ± 27%     -40.5%     272629 ± 14%  perf-stat.i.iTLB-load-misses
   4257436 ±  2%     -16.5%    3554676 ±  2%  perf-stat.i.iTLB-loads
 2.038e+09           -22.3%  1.584e+09        perf-stat.i.instructions
      0.50           +67.1%       0.83        perf-stat.i.ipc
      5.03 ±  7%     -58.7%       2.08 ± 10%  perf-stat.i.major-faults
      0.29           -47.9%       0.15        perf-stat.i.metric.GHz
    659.63 ±  2%     -21.7%     516.58        perf-stat.i.metric.K/sec
    108.42           -25.7%      80.56        perf-stat.i.metric.M/sec
      3039            +5.6%       3211 ±  2%  perf-stat.i.minor-faults
      0.00 ± 17%      +0.0        0.00 ± 12%  perf-stat.i.node-load-miss-rate%
   1450700            -6.0%    1363538        perf-stat.i.node-loads
      0.00 ± 28%      +0.0        0.00 ± 19%  perf-stat.i.node-store-miss-rate%
   6054128 ±  3%     -79.8%    1225203 ±  2%  perf-stat.i.node-stores
      3044            +5.5%       3213 ±  2%  perf-stat.i.page-faults
      2.98            +1.2        4.15        perf-stat.overall.branch-miss-rate%
     16.55            +1.8       18.35 ±  2%  perf-stat.overall.cache-miss-rate%
      1.72           -32.9%       1.15        perf-stat.overall.cpi
    165.40           -40.3%      98.75        perf-stat.overall.cycles-between-cache-misses
      0.58           +49.1%       0.87        perf-stat.overall.ipc
      0.00 ± 40%      +0.0        0.00 ±  9%  perf-stat.overall.node-store-miss-rate%
 4.195e+08           -21.1%   3.31e+08        perf-stat.ps.branch-instructions
  12510468            +9.9%   13751528        perf-stat.ps.branch-misses
  21007097           -12.7%   18338973 ±  2%  perf-stat.ps.cache-misses
 1.269e+08           -21.2%   99965770        perf-stat.ps.cache-references
     32738 ±  3%     -52.4%      15579 ±  2%  perf-stat.ps.context-switches
 3.474e+09           -47.9%   1.81e+09        perf-stat.ps.cpu-cycles
     26.81 ±  2%      +7.8%      28.90 ±  2%  perf-stat.ps.cpu-migrations
    611716 ±  3%     -22.4%     474612 ±  4%  perf-stat.ps.dTLB-load-misses
 4.882e+08           -26.4%  3.591e+08        perf-stat.ps.dTLB-loads
  2.54e+08           -33.0%  1.702e+08        perf-stat.ps.dTLB-stores
    455336 ± 27%     -40.5%     270782 ± 14%  perf-stat.ps.iTLB-load-misses
   4227298 ±  2%     -16.5%    3531717 ±  2%  perf-stat.ps.iTLB-loads
 2.024e+09           -22.3%  1.573e+09        perf-stat.ps.instructions
      5.00 ±  7%     -58.8%       2.06 ± 10%  perf-stat.ps.major-faults
      3019            +5.6%       3187 ±  2%  perf-stat.ps.minor-faults
   1440523            -5.9%    1354982        perf-stat.ps.node-loads
   6011497 ±  3%     -79.8%    1217164 ±  2%  perf-stat.ps.node-stores
      3024            +5.5%       3189 ±  2%  perf-stat.ps.page-faults
 2.908e+11           -19.6%  2.338e+11        perf-stat.total.instructions
     42.60 ±  8%     -23.2       19.38 ±  4%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
     40.68 ±  8%     -22.8       17.90 ±  5%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
     20.00 ± 15%     -16.6        3.44 ± 12%  perf-profile.calltrace.cycles-pp.drm_ioctl_kernel.drm_ioctl.__x64_sys_ioctl.do_syscall_64.entry_SYSCALL_64_after_hwframe
     20.45 ± 15%     -16.4        4.07 ±  9%  perf-profile.calltrace.cycles-pp.drm_ioctl.__x64_sys_ioctl.do_syscall_64.entry_SYSCALL_64_after_hwframe
     20.70 ± 14%     -16.3        4.44 ±  8%  perf-profile.calltrace.cycles-pp.__x64_sys_ioctl.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.64 ± 17%      -5.3        0.30 ±101%  perf-profile.calltrace.cycles-pp.evict.__dentry_kill.dentry_kill.dput.__fput
      5.64 ± 17%      -5.3        0.30 ±101%  perf-profile.calltrace.cycles-pp.shmem_evict_inode.evict.__dentry_kill.dentry_kill.dput
      5.62 ± 17%      -5.3        0.30 ±101%  perf-profile.calltrace.cycles-pp.shmem_undo_range.shmem_evict_inode.evict.__dentry_kill.dentry_kill
      7.24 ± 39%      -4.2        3.05 ± 15%  perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
      4.08 ± 18%      -1.5        2.60 ±  9%  perf-profile.calltrace.cycles-pp.do_writev.do_syscall_64.entry_SYSCALL_64_after_hwframe
      3.89 ± 18%      -1.4        2.47 ± 10%  perf-profile.calltrace.cycles-pp.vfs_writev.do_writev.do_syscall_64.entry_SYSCALL_64_after_hwframe
      3.71 ± 18%      -1.3        2.37 ± 10%  perf-profile.calltrace.cycles-pp.do_iter_write.vfs_writev.do_writev.do_syscall_64.entry_SYSCALL_64_after_hwframe
      3.55 ± 18%      -1.3        2.30 ± 10%  perf-profile.calltrace.cycles-pp.do_iter_readv_writev.do_iter_write.vfs_writev.do_writev.do_syscall_64
      3.49 ± 17%      -1.2        2.27 ± 11%  perf-profile.calltrace.cycles-pp.sock_write_iter.do_iter_readv_writev.do_iter_write.vfs_writev.do_writev
      3.29 ± 18%      -1.1        2.14 ± 10%  perf-profile.calltrace.cycles-pp.unix_stream_sendmsg.sock_write_iter.do_iter_readv_writev.do_iter_write.vfs_writev
      2.48 ± 23%      -1.0        1.48 ± 13%  perf-profile.calltrace.cycles-pp.__sys_recvmsg.do_syscall_64.entry_SYSCALL_64_after_hwframe
      3.70 ± 18%      -1.0        2.72 ±  9%  perf-profile.calltrace.cycles-pp.syscall_return_via_sysret
      2.26 ± 23%      -0.9        1.37 ± 12%  perf-profile.calltrace.cycles-pp.___sys_recvmsg.__sys_recvmsg.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.67 ± 39%      -0.8        0.90 ± 26%  perf-profile.calltrace.cycles-pp.__x64_sys_epoll_wait.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.00 ± 23%      -0.8        1.24 ± 11%  perf-profile.calltrace.cycles-pp.____sys_recvmsg.___sys_recvmsg.__sys_recvmsg.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.06 ± 41%      -0.7        0.33 ±101%  perf-profile.calltrace.cycles-pp.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait.do_syscall_64
      1.57 ± 39%      -0.7        0.86 ± 25%  perf-profile.calltrace.cycles-pp.do_epoll_wait.__x64_sys_epoll_wait.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.89 ± 24%      -0.7        1.19 ± 14%  perf-profile.calltrace.cycles-pp.sock_recvmsg.____sys_recvmsg.___sys_recvmsg.__sys_recvmsg.do_syscall_64
      1.48 ± 40%      -0.7        0.80 ± 27%  perf-profile.calltrace.cycles-pp.ep_poll.do_epoll_wait.__x64_sys_epoll_wait.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.75 ± 23%      -0.7        1.09 ± 17%  perf-profile.calltrace.cycles-pp.unix_stream_recvmsg.sock_recvmsg.____sys_recvmsg.___sys_recvmsg.__sys_recvmsg
      1.70 ± 23%      -0.7        1.05 ± 16%  perf-profile.calltrace.cycles-pp.unix_stream_read_generic.unix_stream_recvmsg.sock_recvmsg.____sys_recvmsg.___sys_recvmsg
      1.09 ± 18%      -0.6        0.44 ± 72%  perf-profile.calltrace.cycles-pp.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb.unix_stream_sendmsg.sock_write_iter
      1.12 ± 19%      -0.6        0.53 ± 47%  perf-profile.calltrace.cycles-pp.alloc_skb_with_frags.sock_alloc_send_pskb.unix_stream_sendmsg.sock_write_iter.do_iter_readv_writev
      1.26 ± 18%      -0.5        0.72 ± 16%  perf-profile.calltrace.cycles-pp.sock_alloc_send_pskb.unix_stream_sendmsg.sock_write_iter.do_iter_readv_writev.do_iter_write
      0.75 ± 16%      -0.5        0.28 ±100%  perf-profile.calltrace.cycles-pp.consume_skb.unix_stream_read_generic.unix_stream_recvmsg.sock_recvmsg.____sys_recvmsg
      1.14 ± 16%      -0.4        0.69 ± 49%  perf-profile.calltrace.cycles-pp.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.96 ± 17%      -0.4        0.53 ± 48%  perf-profile.calltrace.cycles-pp.unix_stream_recvmsg.sock_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64
      1.01 ± 16%      -0.4        0.59 ± 48%  perf-profile.calltrace.cycles-pp.sock_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.92 ± 16%      -0.4        0.52 ± 47%  perf-profile.calltrace.cycles-pp.unix_stream_read_generic.unix_stream_recvmsg.sock_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
      1.34 ± 16%      -0.4        0.95 ± 20%  perf-profile.calltrace.cycles-pp.__entry_text_start
      1.05 ± 14%      +0.4        1.44 ± 15%  perf-profile.calltrace.cycles-pp.__do_softirq.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      0.76 ± 18%      +0.5        1.22 ± 12%  perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
      1.14 ± 15%      +0.5        1.64 ± 12%  perf-profile.calltrace.cycles-pp.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      0.10 ±223%      +0.5        0.65 ± 11%  perf-profile.calltrace.cycles-pp.asm_exc_page_fault
      0.09 ±223%      +0.6        0.64 ± 18%  perf-profile.calltrace.cycles-pp.tick_irq_enter.irq_enter_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      0.09 ±223%      +0.6        0.66 ± 19%  perf-profile.calltrace.cycles-pp.irq_enter_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      0.08 ±223%      +0.6        0.73 ± 16%  perf-profile.calltrace.cycles-pp.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      1.40 ± 20%      +0.8        2.22 ±  9%  perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
      0.09 ±223%      +0.8        0.94 ± 23%  perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry
      1.68 ± 20%      +0.9        2.58 ±  8%  perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
      1.42 ± 10%      +1.0        2.40 ± 14%  perf-profile.calltrace.cycles-pp.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
      1.14 ± 20%      +1.0        2.16 ± 10%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
      1.89 ± 22%      +1.0        2.92 ± 10%  perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      3.14 ± 26%      +1.7        4.86 ±  9%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      3.81 ± 26%      +2.1        5.95 ±  7%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      3.93 ± 26%      +2.3        6.25 ±  7%  perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      6.30 ± 18%      +3.2        9.46 ±  9%  perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
      6.93 ± 20%      +3.5       10.46 ± 11%  perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
     37.49 ±  5%     +21.5       59.02 ±  8%  perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
     42.24 ±  4%     +22.4       64.61 ±  8%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     42.29 ±  4%     +22.4       64.67 ±  8%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
     42.28 ±  4%     +22.4       64.66 ±  8%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     39.44 ±  4%     +22.7       62.12 ±  8%  perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     27.18 ±  6%     +23.8       50.99 ±  2%  perf-profile.calltrace.cycles-pp.intel_idle_ibrs.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
     40.48 ±  6%     +25.5       65.93 ±  2%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
     45.98 ±  4%     +26.8       72.75 ±  2%  perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
     42.67 ±  8%     -23.3       19.40 ±  4%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     40.70 ±  8%     -22.8       17.93 ±  5%  perf-profile.children.cycles-pp.do_syscall_64
     20.00 ± 15%     -16.6        3.45 ± 12%  perf-profile.children.cycles-pp.drm_ioctl_kernel
     20.45 ± 15%     -16.4        4.08 ± 10%  perf-profile.children.cycles-pp.drm_ioctl
     20.71 ± 14%     -16.3        4.44 ±  8%  perf-profile.children.cycles-pp.__x64_sys_ioctl
     15.06 ± 17%     -13.4        1.71 ± 15%  perf-profile.children.cycles-pp.shmem_read_mapping_page_gfp
     14.79 ± 18%     -13.1        1.68 ± 17%  perf-profile.children.cycles-pp.shmem_get_folio_gfp
      5.78 ± 16%      -5.0        0.78 ± 14%  perf-profile.children.cycles-pp.dentry_kill
      5.80 ± 17%      -5.0        0.80 ± 13%  perf-profile.children.cycles-pp.__fput
      5.78 ± 16%      -5.0        0.78 ± 14%  perf-profile.children.cycles-pp.__dentry_kill
      5.77 ± 16%      -5.0        0.77 ± 14%  perf-profile.children.cycles-pp.evict
      5.77 ± 16%      -5.0        0.77 ± 14%  perf-profile.children.cycles-pp.shmem_evict_inode
      5.76 ± 16%      -5.0        0.77 ± 14%  perf-profile.children.cycles-pp.shmem_undo_range
      5.79 ± 16%      -5.0        0.80 ± 12%  perf-profile.children.cycles-pp.dput
      7.28 ± 39%      -4.2        3.08 ± 14%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
      3.96 ± 23%      -3.9        0.09 ± 43%  perf-profile.children.cycles-pp.poll_idle
      3.36 ± 18%      -2.9        0.45 ± 15%  perf-profile.children.cycles-pp.shmem_add_to_page_cache
      3.07 ± 17%      -2.7        0.32 ± 50%  perf-profile.children.cycles-pp.clear_page_erms
      3.00 ± 23%      -2.5        0.47 ± 30%  perf-profile.children.cycles-pp.shmem_alloc_and_acct_folio
      2.90 ± 16%      -2.5        0.41 ±  9%  perf-profile.children.cycles-pp.release_pages
      2.76 ± 16%      -2.4        0.34 ± 12%  perf-profile.children.cycles-pp.__pagevec_release
      2.76 ± 48%      -2.3        0.42 ± 35%  perf-profile.children.cycles-pp.check_move_unevictable_pages
      2.67 ± 48%      -2.3        0.41 ± 39%  perf-profile.children.cycles-pp.check_move_unevictable_folios
      2.52 ± 17%      -2.2        0.36 ± 15%  perf-profile.children.cycles-pp.truncate_inode_folio
      2.55 ± 24%      -2.1        0.42 ± 35%  perf-profile.children.cycles-pp.shmem_alloc_folio
      2.32 ± 17%      -2.0        0.33 ± 16%  perf-profile.children.cycles-pp.filemap_remove_folio
      2.17 ± 22%      -1.8        0.38 ± 33%  perf-profile.children.cycles-pp.vma_alloc_folio
      1.95 ± 17%      -1.7        0.30 ± 11%  perf-profile.children.cycles-pp.__filemap_remove_folio
      4.09 ± 18%      -1.5        2.60 ±  9%  perf-profile.children.cycles-pp.do_writev
      3.89 ± 18%      -1.4        2.47 ± 10%  perf-profile.children.cycles-pp.vfs_writev
      1.72 ± 19%      -1.4        0.32 ± 33%  perf-profile.children.cycles-pp.__folio_alloc
      1.75 ± 20%      -1.4        0.36 ± 31%  perf-profile.children.cycles-pp.__alloc_pages
      3.71 ± 18%      -1.3        2.37 ± 10%  perf-profile.children.cycles-pp.do_iter_write
      3.56 ± 18%      -1.2        2.32 ±  9%  perf-profile.children.cycles-pp.do_iter_readv_writev
      3.50 ± 17%      -1.2        2.27 ± 11%  perf-profile.children.cycles-pp.sock_write_iter
      1.32 ± 15%      -1.2        0.15 ± 19%  perf-profile.children.cycles-pp.free_unref_page_list
      3.29 ± 18%      -1.1        2.14 ± 10%  perf-profile.children.cycles-pp.unix_stream_sendmsg
      1.40 ± 23%      -1.1        0.29 ± 36%  perf-profile.children.cycles-pp.get_page_from_freelist
      1.26 ±  8%      -1.1        0.16 ± 30%  perf-profile.children.cycles-pp.xas_store
      2.90 ± 20%      -1.0        1.85 ± 14%  perf-profile.children.cycles-pp.sock_recvmsg
      1.24 ± 14%      -1.0        0.22 ± 21%  perf-profile.children.cycles-pp.__mod_lruvec_page_state
      2.70 ± 20%      -1.0        1.68 ± 17%  perf-profile.children.cycles-pp.unix_stream_recvmsg
      2.64 ± 20%      -1.0        1.64 ± 17%  perf-profile.children.cycles-pp.unix_stream_read_generic
      2.48 ± 23%      -1.0        1.49 ± 13%  perf-profile.children.cycles-pp.__sys_recvmsg
      1.24 ± 15%      -1.0        0.26 ± 49%  perf-profile.children.cycles-pp.folio_add_lru
      3.74 ± 17%      -1.0        2.76 ±  8%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      1.18 ± 25%      -1.0        0.23 ± 41%  perf-profile.children.cycles-pp.rmqueue
      2.26 ± 23%      -0.9        1.37 ± 12%  perf-profile.children.cycles-pp.___sys_recvmsg
      1.08 ± 16%      -0.8        0.24 ± 50%  perf-profile.children.cycles-pp.folio_batch_move_lru
      1.17 ± 66%      -0.8        0.33 ± 25%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      1.67 ± 39%      -0.8        0.90 ± 26%  perf-profile.children.cycles-pp.__x64_sys_epoll_wait
      2.00 ± 23%      -0.8        1.24 ± 11%  perf-profile.children.cycles-pp.____sys_recvmsg
      0.85 ± 28%      -0.7        0.12 ± 50%  perf-profile.children.cycles-pp.__mem_cgroup_charge
      1.57 ± 39%      -0.7        0.86 ± 25%  perf-profile.children.cycles-pp.do_epoll_wait
      0.87 ± 29%      -0.7        0.17 ± 31%  perf-profile.children.cycles-pp.__rmqueue_pcplist
      1.48 ± 40%      -0.7        0.80 ± 27%  perf-profile.children.cycles-pp.ep_poll
      0.76 ±  9%      -0.6        0.12 ± 21%  perf-profile.children.cycles-pp.__mod_lruvec_state
      0.74 ± 13%      -0.6        0.10 ± 22%  perf-profile.children.cycles-pp.free_pcppages_bulk
      0.73 ± 15%      -0.6        0.10 ± 50%  perf-profile.children.cycles-pp.lru_add_fn
      0.73 ± 15%      -0.6        0.12 ± 60%  perf-profile.children.cycles-pp.__mod_memcg_lruvec_state
      0.72 ± 29%      -0.6        0.10 ± 58%  perf-profile.children.cycles-pp.rmqueue_bulk
      1.26 ± 19%      -0.5        0.72 ± 16%  perf-profile.children.cycles-pp.sock_alloc_send_pskb
      1.12 ± 19%      -0.5        0.61 ± 18%  perf-profile.children.cycles-pp.alloc_skb_with_frags
      0.57 ± 11%      -0.5        0.08 ± 21%  perf-profile.children.cycles-pp.__mod_node_page_state
      1.09 ± 19%      -0.5        0.60 ± 19%  perf-profile.children.cycles-pp.__alloc_skb
      0.57 ± 37%      -0.5        0.08 ± 81%  perf-profile.children.cycles-pp.charge_memcg
      1.18 ± 14%      -0.5        0.69 ± 13%  perf-profile.children.cycles-pp.consume_skb
      0.60 ± 19%      -0.5        0.12 ± 21%  perf-profile.children.cycles-pp.filemap_unaccount_folio
      1.16 ± 20%      -0.5        0.70 ± 14%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.54 ± 15%      -0.5        0.08 ± 51%  perf-profile.children.cycles-pp.__free_one_page
      0.43 ± 26%      -0.4        0.04 ±101%  perf-profile.children.cycles-pp.find_lock_entries
      0.43 ± 17%      -0.4        0.05 ± 81%  perf-profile.children.cycles-pp.xas_load
      1.14 ± 16%      -0.4        0.76 ± 26%  perf-profile.children.cycles-pp.__sys_recvfrom
      0.54 ± 13%      -0.3        0.22 ± 30%  perf-profile.children.cycles-pp.vm_mmap_pgoff
      0.42 ± 18%      -0.3        0.10 ± 53%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.52 ± 15%      -0.3        0.20 ± 25%  perf-profile.children.cycles-pp.do_mmap
      0.60 ± 16%      -0.3        0.29 ± 22%  perf-profile.children.cycles-pp.__kmem_cache_alloc_node
      0.75 ± 19%      -0.3        0.45 ± 26%  perf-profile.children.cycles-pp.ep_poll_callback
      0.33 ± 21%      -0.3        0.04 ±104%  perf-profile.children.cycles-pp.xas_clear_mark
      0.62 ± 11%      -0.3        0.34 ± 19%  perf-profile.children.cycles-pp.unix_destruct_scm
      0.60 ± 12%      -0.3        0.34 ± 18%  perf-profile.children.cycles-pp.sock_wfree
      0.63 ± 11%      -0.3        0.37 ± 27%  perf-profile.children.cycles-pp.skb_release_head_state
      0.53 ± 20%      -0.3        0.27 ± 33%  perf-profile.children.cycles-pp.kmalloc_reserve
      0.30 ± 20%      -0.2        0.05 ± 72%  perf-profile.children.cycles-pp.xas_find_conflict
      0.30 ±  9%      -0.2        0.06 ± 80%  perf-profile.children.cycles-pp.rcu_do_batch
      0.28 ± 34%      -0.2        0.05 ±107%  perf-profile.children.cycles-pp.cgroup_rstat_updated
      0.52 ± 23%      -0.2        0.29 ± 17%  perf-profile.children.cycles-pp.skb_release_data
      0.41 ± 19%      -0.2        0.18 ± 34%  perf-profile.children.cycles-pp.mmap_region
      0.48 ± 20%      -0.2        0.25 ± 33%  perf-profile.children.cycles-pp.__kmalloc_node_track_caller
      0.78 ± 16%      -0.2        0.57 ± 18%  perf-profile.children.cycles-pp._raw_spin_lock
      0.24 ± 26%      -0.2        0.03 ±101%  perf-profile.children.cycles-pp.folio_unlock
      0.35 ±  9%      -0.2        0.14 ± 22%  perf-profile.children.cycles-pp.rcu_core
      0.48 ± 20%      -0.2        0.30 ± 21%  perf-profile.children.cycles-pp.kmem_cache_alloc_node
      0.42 ± 10%      -0.2        0.24 ± 21%  perf-profile.children.cycles-pp.unix_write_space
      0.39 ± 25%      -0.2        0.21 ± 34%  perf-profile.children.cycles-pp.ep_autoremove_wake_function
      0.49 ± 19%      -0.2        0.31 ± 28%  perf-profile.children.cycles-pp.memcg_slab_post_alloc_hook
      0.22 ± 21%      -0.2        0.05 ± 72%  perf-profile.children.cycles-pp.__list_add_valid
      0.21 ± 19%      -0.2        0.04 ±105%  perf-profile.children.cycles-pp.try_charge_memcg
      0.23 ± 24%      -0.2        0.06 ± 58%  perf-profile.children.cycles-pp.__folio_throttle_swaprate
      0.44 ± 23%      -0.2        0.27 ± 22%  perf-profile.children.cycles-pp.sock_poll
      0.22 ± 24%      -0.2        0.06 ±110%  perf-profile.children.cycles-pp.do_vmi_munmap
      0.32 ± 26%      -0.2        0.17 ± 33%  perf-profile.children.cycles-pp.__kmem_cache_free
      0.19 ± 25%      -0.2        0.04 ±115%  perf-profile.children.cycles-pp.blk_cgroup_congested
      0.28 ± 28%      -0.2        0.13 ± 17%  perf-profile.children.cycles-pp.unix_poll
      0.35 ± 29%      -0.1        0.21 ± 30%  perf-profile.children.cycles-pp.skb_copy_datagram_from_iter
      0.18 ± 20%      -0.1        0.04 ±110%  perf-profile.children.cycles-pp.memset_orig
      0.35 ± 27%      -0.1        0.22 ± 39%  perf-profile.children.cycles-pp.rep_movs_alternative
      0.30 ± 15%      -0.1        0.18 ± 27%  perf-profile.children.cycles-pp.__slab_free
      0.23 ± 24%      -0.1        0.11 ± 36%  perf-profile.children.cycles-pp.copy_msghdr_from_user
      0.16 ± 37%      -0.1        0.05 ± 77%  perf-profile.children.cycles-pp._copy_from_iter
      0.20 ± 28%      -0.1        0.10 ± 67%  perf-profile.children.cycles-pp.mod_objcg_state
      0.19 ± 25%      -0.1        0.08 ± 65%  perf-profile.children.cycles-pp.import_iovec
      0.18 ± 23%      -0.1        0.07 ± 80%  perf-profile.children.cycles-pp.__import_iovec
      0.30 ±  9%      -0.1        0.19 ± 33%  perf-profile.children.cycles-pp.kmem_cache_free
      0.15 ± 29%      -0.1        0.06 ± 64%  perf-profile.children.cycles-pp.mas_store_prealloc
      0.15 ± 20%      -0.1        0.08 ± 30%  perf-profile.children.cycles-pp.dma_resv_iter_walk_unlocked
      0.15 ± 19%      -0.1        0.09 ± 37%  perf-profile.children.cycles-pp.obj_cgroup_charge
      0.06 ± 50%      +0.1        0.12 ± 16%  perf-profile.children.cycles-pp.cpuidle_not_available
      0.02 ±141%      +0.1        0.08 ± 27%  perf-profile.children.cycles-pp.account_process_tick
      0.02 ±141%      +0.1        0.10 ± 15%  perf-profile.children.cycles-pp.link_path_walk
      0.06 ± 59%      +0.1        0.14 ± 20%  perf-profile.children.cycles-pp.rb_insert_color
      0.02 ±141%      +0.1        0.10 ± 46%  perf-profile.children.cycles-pp.tick_nohz_tick_stopped
      0.02 ±146%      +0.1        0.11 ± 30%  perf-profile.children.cycles-pp.cpuidle_reflect
      0.00            +0.1        0.10 ± 36%  perf-profile.children.cycles-pp.pipe_write
      0.01 ±223%      +0.1        0.11 ± 28%  perf-profile.children.cycles-pp.task_tick_idle
      0.05 ± 76%      +0.1        0.15 ± 24%  perf-profile.children.cycles-pp.dup_mmap
      0.20 ± 21%      +0.1        0.30 ± 24%  perf-profile.children.cycles-pp.ct_idle_exit
      0.00            +0.1        0.10 ± 36%  perf-profile.children.cycles-pp.show
      0.00            +0.1        0.10 ± 36%  perf-profile.children.cycles-pp.show_time_in_state
      0.09 ± 49%      +0.1        0.19 ± 18%  perf-profile.children.cycles-pp.copy_process
      0.06 ± 75%      +0.1        0.17 ± 21%  perf-profile.children.cycles-pp.dup_mm
      0.00            +0.1        0.11 ± 34%  perf-profile.children.cycles-pp.sysfs_kf_seq_show
      0.02 ±144%      +0.1        0.13 ± 28%  perf-profile.children.cycles-pp.do_futex
      0.01 ±223%      +0.1        0.12 ± 30%  perf-profile.children.cycles-pp.futex_wait
      0.05 ± 49%      +0.1        0.16 ± 44%  perf-profile.children.cycles-pp.ct_kernel_exit_state
      0.02 ±149%      +0.1        0.14 ± 35%  perf-profile.children.cycles-pp.ksys_mmap_pgoff
      0.05 ± 47%      +0.1        0.16 ± 20%  perf-profile.children.cycles-pp.error_entry
      0.02 ±141%      +0.1        0.13 ± 34%  perf-profile.children.cycles-pp.ct_kernel_exit
      0.19 ± 17%      +0.1        0.31 ± 24%  perf-profile.children.cycles-pp.irqtime_account_irq
      0.02 ±146%      +0.1        0.14 ± 31%  perf-profile.children.cycles-pp.__x64_sys_futex
      0.09 ± 49%      +0.1        0.21 ± 22%  perf-profile.children.cycles-pp.__do_sys_clone
      0.09 ± 49%      +0.1        0.21 ± 22%  perf-profile.children.cycles-pp.kernel_clone
      0.01 ±223%      +0.1        0.13 ± 58%  perf-profile.children.cycles-pp.__irqentry_text_end
      0.04 ±101%      +0.1        0.16 ± 41%  perf-profile.children.cycles-pp.irqentry_exit_to_user_mode
      0.00            +0.1        0.12 ± 45%  perf-profile.children.cycles-pp.traverse
      0.03 ±102%      +0.1        0.15 ± 45%  perf-profile.children.cycles-pp.vfs_write
      0.11 ± 15%      +0.1        0.24 ± 17%  perf-profile.children.cycles-pp.__intel_pmu_enable_all
      0.18 ± 22%      +0.1        0.31 ± 26%  perf-profile.children.cycles-pp.find_busiest_group
      0.04 ± 73%      +0.1        0.17 ± 45%  perf-profile.children.cycles-pp.ksys_write
      0.00            +0.1        0.13 ± 41%  perf-profile.children.cycles-pp.vsnprintf
      0.00            +0.1        0.14 ± 27%  perf-profile.children.cycles-pp.pipe_read
      0.05 ± 45%      +0.1        0.19 ± 18%  perf-profile.children.cycles-pp.hrtimer_update_next_event
      0.42 ± 19%      +0.2        0.58 ± 13%  perf-profile.children.cycles-pp.perf_rotate_context
      0.18 ± 29%      +0.2        0.34 ± 18%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.04 ± 71%      +0.2        0.20 ± 27%  perf-profile.children.cycles-pp.do_filp_open
      0.04 ± 71%      +0.2        0.20 ± 27%  perf-profile.children.cycles-pp.path_openat
      0.04 ± 75%      +0.2        0.22 ± 50%  perf-profile.children.cycles-pp.do_read_fault
      0.10 ± 32%      +0.2        0.28 ± 45%  perf-profile.children.cycles-pp.lapic_next_deadline
      0.05 ± 45%      +0.2        0.24 ± 22%  perf-profile.children.cycles-pp.native_apic_msr_eoi_write
      0.02 ±142%      +0.2        0.21 ± 56%  perf-profile.children.cycles-pp.filemap_map_pages
      0.14 ± 31%      +0.2        0.33 ± 11%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.04 ± 72%      +0.2        0.23 ± 18%  perf-profile.children.cycles-pp.__x64_sys_openat
      0.12 ± 25%      +0.2        0.31 ± 33%  perf-profile.children.cycles-pp.__hrtimer_next_event_base
      0.04 ± 71%      +0.2        0.23 ± 18%  perf-profile.children.cycles-pp.do_sys_openat2
      0.07 ± 58%      +0.2        0.26 ± 26%  perf-profile.children.cycles-pp.__mmput
      0.07 ± 58%      +0.2        0.26 ± 26%  perf-profile.children.cycles-pp.exit_mmap
      0.27 ± 27%      +0.2        0.46 ± 23%  perf-profile.children.cycles-pp.__handle_mm_fault
      0.14 ± 31%      +0.2        0.34 ± 29%  perf-profile.children.cycles-pp.hrtimer_next_event_without
      0.06 ± 25%      +0.2        0.25 ± 15%  perf-profile.children.cycles-pp.wait_for_xmitr
      0.29 ± 23%      +0.2        0.48 ± 25%  perf-profile.children.cycles-pp.handle_mm_fault
      0.10 ± 17%      +0.2        0.31 ± 39%  perf-profile.children.cycles-pp.arch_scale_freq_tick
      0.09 ± 23%      +0.2        0.31 ± 31%  perf-profile.children.cycles-pp.load_elf_binary
      0.35 ± 24%      +0.2        0.57 ± 20%  perf-profile.children.cycles-pp.exc_page_fault
      0.34 ± 24%      +0.2        0.57 ± 19%  perf-profile.children.cycles-pp.do_user_addr_fault
      0.39 ± 11%      +0.2        0.62 ± 34%  perf-profile.children.cycles-pp.rebalance_domains
      0.09 ± 21%      +0.2        0.32 ± 33%  perf-profile.children.cycles-pp.search_binary_handler
      0.10 ± 13%      +0.2        0.33 ± 31%  perf-profile.children.cycles-pp.bprm_execve
      0.09 ± 21%      +0.2        0.33 ± 32%  perf-profile.children.cycles-pp.exec_binprm
      0.01 ±223%      +0.3        0.26 ± 49%  perf-profile.children.cycles-pp.seq_read_iter
      0.47 ± 14%      +0.3        0.72 ± 14%  perf-profile.children.cycles-pp.irq_enter_rcu
      0.46 ± 14%      +0.3        0.71 ± 14%  perf-profile.children.cycles-pp.tick_irq_enter
      0.24 ± 28%      +0.3        0.51 ± 15%  perf-profile.children.cycles-pp.read_tsc
      0.12 ± 15%      +0.3        0.41 ± 27%  perf-profile.children.cycles-pp.__x64_sys_execve
      0.12 ± 17%      +0.3        0.41 ± 27%  perf-profile.children.cycles-pp.do_execveat_common
      0.41 ± 25%      +0.3        0.70 ± 11%  perf-profile.children.cycles-pp.asm_exc_page_fault
      0.14 ± 23%      +0.3        0.44 ± 30%  perf-profile.children.cycles-pp.clockevents_program_event
      0.50 ± 21%      +0.3        0.83 ± 18%  perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
      0.09 ± 33%      +0.4        0.48 ± 35%  perf-profile.children.cycles-pp.vfs_read
      0.09 ± 31%      +0.4        0.51 ± 32%  perf-profile.children.cycles-pp.ksys_read
      0.40 ± 13%      +0.4        0.82 ± 15%  perf-profile.children.cycles-pp.ktime_get
      1.54 ± 11%      +0.5        2.00 ± 10%  perf-profile.children.cycles-pp.__irq_exit_rcu
      0.98 ± 17%      +0.5        1.47 ± 11%  perf-profile.children.cycles-pp.scheduler_tick
      0.47 ± 19%      +0.6        1.03 ± 27%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      1.78 ± 19%      +0.9        2.69 ± 10%  perf-profile.children.cycles-pp.update_process_times
      2.09 ± 19%      +0.9        3.03 ±  9%  perf-profile.children.cycles-pp.tick_sched_handle
      1.22 ± 15%      +1.0        2.22 ± 10%  perf-profile.children.cycles-pp.intel_idle
      1.50 ±  9%      +1.1        2.56 ± 17%  perf-profile.children.cycles-pp.menu_select
      2.34 ± 20%      +1.1        3.43 ±  7%  perf-profile.children.cycles-pp.tick_sched_timer
      3.76 ± 21%      +1.8        5.56 ±  7%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      4.39 ± 21%      +2.2        6.63 ±  6%  perf-profile.children.cycles-pp.hrtimer_interrupt
      4.53 ± 21%      +2.5        7.00 ±  6%  perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
      6.93 ± 17%      +3.5       10.44 ±  7%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
      7.50 ± 17%      +3.8       11.31 ±  7%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
     42.29 ±  4%     +22.4       64.67 ±  8%  perf-profile.children.cycles-pp.start_secondary
     27.19 ±  6%     +23.8       51.00 ±  2%  perf-profile.children.cycles-pp.intel_idle_ibrs
     40.99 ±  6%     +25.6       66.55 ±  2%  perf-profile.children.cycles-pp.cpuidle_enter_state
     41.02 ±  6%     +25.6       66.63 ±  2%  perf-profile.children.cycles-pp.cpuidle_enter
     45.98 ±  4%     +26.8       72.75 ±  2%  perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     45.98 ±  4%     +26.8       72.75 ±  2%  perf-profile.children.cycles-pp.cpu_startup_entry
     45.95 ±  4%     +26.8       72.73 ±  2%  perf-profile.children.cycles-pp.do_idle
     43.10 ±  5%     +26.9       70.00 ±  2%  perf-profile.children.cycles-pp.cpuidle_idle_call
      3.86 ± 23%      -3.8        0.08 ± 59%  perf-profile.self.cycles-pp.poll_idle
      3.56 ± 19%      -3.4        0.15 ± 37%  perf-profile.self.cycles-pp.shmem_get_folio_gfp
      3.03 ± 17%      -2.7        0.28 ± 40%  perf-profile.self.cycles-pp.clear_page_erms
      3.81 ± 18%      -1.3        2.49 ± 15%  perf-profile.self.cycles-pp.syscall_exit_to_user_mode
      3.73 ± 17%      -1.0        2.76 ±  8%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      1.08 ± 26%      -0.9        0.15 ± 38%  perf-profile.self.cycles-pp.check_move_unevictable_folios
      0.64 ±  9%      -0.6        0.07 ± 71%  perf-profile.self.cycles-pp.xas_store
      0.66 ± 14%      -0.6        0.09 ± 62%  perf-profile.self.cycles-pp.release_pages
      2.03 ± 18%      -0.5        1.50 ±  3%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.61 ± 11%      -0.5        0.09 ± 83%  perf-profile.self.cycles-pp.shmem_add_to_page_cache
      0.54 ± 13%      -0.5        0.08 ± 21%  perf-profile.self.cycles-pp.__mod_node_page_state
      0.54 ± 15%      -0.5        0.09 ± 44%  perf-profile.self.cycles-pp.lru_add_fn
      0.52 ± 16%      -0.4        0.08 ± 82%  perf-profile.self.cycles-pp.rmqueue_bulk
      0.52 ± 11%      -0.4        0.09 ± 66%  perf-profile.self.cycles-pp.__mod_memcg_lruvec_state
      0.40 ± 15%      -0.3        0.06 ± 83%  perf-profile.self.cycles-pp.__mod_lruvec_page_state
      0.40 ± 17%      -0.3        0.06 ± 48%  perf-profile.self.cycles-pp.__free_one_page
      0.41 ± 18%      -0.3        0.10 ± 50%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.33 ± 29%      -0.3        0.04 ±105%  perf-profile.self.cycles-pp.shmem_alloc_folio
      0.30 ± 26%      -0.3        0.03 ±100%  perf-profile.self.cycles-pp.__filemap_remove_folio
      0.28 ± 21%      -0.2        0.03 ±101%  perf-profile.self.cycles-pp.vma_alloc_folio
      0.28 ± 16%      -0.2        0.04 ±112%  perf-profile.self.cycles-pp.__alloc_pages
      0.40 ±  3%      -0.2        0.16 ± 35%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.88 ± 10%      -0.2        0.64 ± 13%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.28 ± 35%      -0.2        0.05 ±107%  perf-profile.self.cycles-pp.cgroup_rstat_updated
      0.77 ± 16%      -0.2        0.56 ± 19%  perf-profile.self.cycles-pp._raw_spin_lock
      0.20 ± 23%      -0.2        0.05 ± 72%  perf-profile.self.cycles-pp.__list_add_valid
      0.39 ± 17%      -0.1        0.26 ± 29%  perf-profile.self.cycles-pp.memcg_slab_post_alloc_hook
      0.17 ± 21%      -0.1        0.04 ±110%  perf-profile.self.cycles-pp.memset_orig
      0.29 ± 17%      -0.1        0.16 ± 20%  perf-profile.self.cycles-pp.__slab_free
      0.15 ± 15%      -0.1        0.03 ±101%  perf-profile.self.cycles-pp.try_charge_memcg
      0.20 ±  9%      -0.1        0.10 ± 50%  perf-profile.self.cycles-pp.kmem_cache_free
      0.14 ± 23%      -0.1        0.04 ±107%  perf-profile.self.cycles-pp.__import_iovec
      0.18 ± 24%      -0.1        0.10 ± 30%  perf-profile.self.cycles-pp.sock_wfree
      0.12 ± 12%      -0.1        0.04 ±112%  perf-profile.self.cycles-pp.tick_nohz_stop_idle
      0.11 ± 27%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.filemap_unaccount_folio
      0.02 ±141%      +0.1        0.08 ± 27%  perf-profile.self.cycles-pp.account_process_tick
      0.06 ± 50%      +0.1        0.12 ± 16%  perf-profile.self.cycles-pp.cpuidle_not_available
      0.14 ± 29%      +0.1        0.22 ± 15%  perf-profile.self.cycles-pp.sysvec_apic_timer_interrupt
      0.01 ±223%      +0.1        0.09 ± 48%  perf-profile.self.cycles-pp.tick_nohz_tick_stopped
      0.02 ±144%      +0.1        0.10 ± 36%  perf-profile.self.cycles-pp.cpuidle_reflect
      0.06 ± 73%      +0.1        0.15 ± 29%  perf-profile.self.cycles-pp.update_sg_lb_stats
      0.05 ± 52%      +0.1        0.14 ± 20%  perf-profile.self.cycles-pp.rb_insert_color
      0.01 ±223%      +0.1        0.11 ± 28%  perf-profile.self.cycles-pp.task_tick_idle
      0.00            +0.1        0.10 ± 20%  perf-profile.self.cycles-pp.tick_sched_timer
      0.04 ± 47%      +0.1        0.16 ± 45%  perf-profile.self.cycles-pp.ct_kernel_exit_state
      0.05 ± 46%      +0.1        0.16 ± 20%  perf-profile.self.cycles-pp.error_entry
      0.01 ±223%      +0.1        0.13 ± 58%  perf-profile.self.cycles-pp.__irqentry_text_end
      0.11 ± 15%      +0.1        0.24 ± 17%  perf-profile.self.cycles-pp.__intel_pmu_enable_all
      0.05 ± 72%      +0.1        0.18 ± 47%  perf-profile.self.cycles-pp.tick_nohz_next_event
      0.10 ± 25%      +0.2        0.28 ± 38%  perf-profile.self.cycles-pp.__hrtimer_next_event_base
      0.05 ± 45%      +0.2        0.23 ± 21%  perf-profile.self.cycles-pp.native_apic_msr_eoi_write
      0.10 ± 32%      +0.2        0.28 ± 45%  perf-profile.self.cycles-pp.lapic_next_deadline
      0.21 ± 13%      +0.2        0.40 ± 33%  perf-profile.self.cycles-pp.ktime_get
      0.14 ± 29%      +0.2        0.33 ± 11%  perf-profile.self.cycles-pp.native_irq_return_iret
      0.10 ± 17%      +0.2        0.31 ± 39%  perf-profile.self.cycles-pp.arch_scale_freq_tick
      0.24 ± 27%      +0.3        0.50 ± 16%  perf-profile.self.cycles-pp.read_tsc
      0.72 ± 19%      +0.3        1.03 ± 14%  perf-profile.self.cycles-pp.cpuidle_enter_state
      1.22 ± 15%      +1.0        2.22 ± 10%  perf-profile.self.cycles-pp.intel_idle
     27.18 ±  6%     +23.8       50.98 ±  2%  perf-profile.self.cycles-pp.intel_idle_ibrs





Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki



View attachment "config-6.4.0-rc2-00007-g278dfbd17c5d" of type "text/plain" (158487 bytes)

View attachment "job-script" of type "text/plain" (7965 bytes)

View attachment "job.yaml" of type "text/plain" (5317 bytes)

View attachment "reproduce" of type "text/plain" (303 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ