lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Date:   Fri, 12 May 2023 16:29:25 +0800
From:   kernel test robot <oliver.sang@...el.com>
To:     Chen Yu <yu.c.chen@...el.com>
CC:     <oe-lkp@...ts.linux.dev>, <lkp@...el.com>,
        Tim Chen <tim.c.chen@...el.com>,
        K Prateek Nayak <kprateek.nayak@....com>,
        "kernel test robot" <yujie.liu@...el.com>,
        <linux-kernel@...r.kernel.org>, <ying.huang@...el.com>,
        <feng.tang@...el.com>, <fengwei.yin@...el.com>,
        <aubrey.li@...ux.intel.com>, <yu.c.chen@...el.com>,
        Peter Zijlstra <peterz@...radead.org>,
        Vincent Guittot <vincent.guittot@...aro.org>,
        "Ingo Molnar" <mingo@...hat.com>,
        Juri Lelli <juri.lelli@...hat.com>,
        Mel Gorman <mgorman@...hsingularity.net>,
        Dietmar Eggemann <dietmar.eggemann@....com>,
        Steven Rostedt <rostedt@...dmis.org>,
        Ben Segall <bsegall@...gle.com>,
        "Abel Wu" <wuyun.abel@...edance.com>,
        Yicong Yang <yangyicong@...ilicon.com>,
        "Gautham R . Shenoy" <gautham.shenoy@....com>,
        Honglei Wang <wanghonglei@...ichuxing.com>,
        Len Brown <len.brown@...el.com>,
        Chen Yu <yu.chen.surf@...il.com>,
        Tianchen Ding <dtcccc@...ux.alibaba.com>,
        "Joel Fernandes" <joel@...lfernandes.org>,
        Josh Don <joshdon@...gle.com>,
        "Arjan Van De Ven" <arjan.van.de.ven@...el.com>,
        Aaron Lu <aaron.lu@...el.com>, <oliver.sang@...el.com>
Subject: Re: [PATCH v8 2/2] sched/fair: Introduce SIS_CURRENT to wake up
 short task on current CPU



Hello,

kernel test robot noticed a 57.6% improvement of phoronix-test-suite.stress-ng.ContextSwitching.bogo_ops_s on:


commit: 485a13bb0c94668849d554f8f1f935a81f8dd2cd ("[PATCH v8 2/2] sched/fair: Introduce SIS_CURRENT to wake up short task on current CPU")
url: https://github.com/intel-lab-lkp/linux/commits/Chen-Yu/sched-fair-Record-the-average-duration-of-a-task/20230428-232326
base: https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git f31dcb152a3d0816e2f1deab4e64572336da197d
patch link: https://lore.kernel.org/all/4081178486e025c89dbb7cc0e62bbfab95fc794a.1682661027.git.yu.c.chen@intel.com/
patch subject: [PATCH v8 2/2] sched/fair: Introduce SIS_CURRENT to wake up short task on current CPU

testcase: phoronix-test-suite
test machine: 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz (Cascade Lake) with 512G memory
parameters:

	test: stress-ng-1.3.1
	option_a: Context Switching
	cpufreq_governor: performance

test-description: The Phoronix Test Suite is the most comprehensive testing and benchmarking platform available that provides an extensible framework for which new tests can be easily added.
test-url: http://www.phoronix-test-suite.com/





Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        sudo bin/lkp install job.yaml           # job file is attached in this email
        bin/lkp split-job --compatible job.yaml # generate the yaml file for lkp run
        sudo bin/lkp run generated-yaml-file

        # if come across any failure that blocks the test,
        # please remove ~/.lkp and /lkp dir to run from a clean state.

=========================================================================================
compiler/cpufreq_governor/kconfig/option_a/rootfs/tbox_group/test/testcase:
  gcc-11/performance/x86_64-rhel-8.3/Context Switching/debian-x86_64-phoronix/lkp-csl-2sp7/stress-ng-1.3.1/phoronix-test-suite

commit: 
  26bcaaa5c6 ("sched/fair: Record the average duration of a task")
  485a13bb0c ("sched/fair: Introduce SIS_CURRENT to wake up short task on current CPU")

26bcaaa5c63c4ad3 485a13bb0c94668849d554f8f1f 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
   3465032           +57.6%    5461531 ± 14%  phoronix-test-suite.stress-ng.ContextSwitching.bogo_ops_s
    114.80          +327.0%     490.19 ± 10%  phoronix-test-suite.time.elapsed_time
    114.80          +327.0%     490.19 ± 10%  phoronix-test-suite.time.elapsed_time.max
    147934 ±  7%  +14359.4%   21390420 ± 70%  phoronix-test-suite.time.involuntary_context_switches
    182173           +73.4%     315930 ±  5%  phoronix-test-suite.time.minor_page_faults
      4590           +26.9%       5826 ±  3%  phoronix-test-suite.time.percent_of_cpu_this_job_got
      4399          +448.9%      24148 ± 10%  phoronix-test-suite.time.system_time
    870.48          +408.4%       4425 ± 11%  phoronix-test-suite.time.user_time
 5.651e+08          +668.6%  4.343e+09 ± 19%  phoronix-test-suite.time.voluntary_context_switches
 3.528e+09 ±  3%    +190.8%  1.026e+10 ±  9%  cpuidle..time
 3.046e+08          +260.4%  1.098e+09 ± 11%  cpuidle..usage
    166.16          +226.0%     541.68 ±  9%  uptime.boot
      7741           +89.3%      14658 ±  6%  uptime.idle
     80501 ± 54%     +94.4%     156530 ± 24%  numa-meminfo.node0.AnonHugePages
     76.50 ±  8%    +788.0%     679.33 ± 50%  numa-meminfo.node0.Unevictable
     59794 ± 36%    +123.0%     133361 ± 15%  numa-meminfo.node1.Active
      7692 ± 12%    +967.3%      82097 ±  6%  numa-meminfo.node1.Active(anon)
    281420 ± 23%    +177.0%     779557 ± 21%  numa-numastat.node0.local_node
    292813 ± 23%    +167.7%     783881 ± 21%  numa-numastat.node0.numa_hit
    361211 ± 17%    +121.8%     801062 ± 22%  numa-numastat.node1.local_node
    369650 ± 17%    +120.7%     815910 ± 21%  numa-numastat.node1.numa_hit
    127614           +86.2%     237590 ± 31%  meminfo.Active
     11364 ±  6%    +664.0%      86826 ±  7%  meminfo.Active(anon)
    140254 ± 11%     +67.9%     235428 ±  6%  meminfo.AnonHugePages
    721246           +11.2%     802088 ±  2%  meminfo.Shmem
     83.00          +969.1%     887.33 ± 18%  meminfo.Unevictable
     38.83           -33.0%      26.00 ±  4%  vmstat.cpu.id
    806.50 ±  4%     -65.6%     277.83 ± 62%  vmstat.io.bi
     68.83           +28.3%      88.33 ±  3%  vmstat.procs.r
   8102056           +44.8%   11728871 ± 10%  vmstat.system.cs
    965452            -6.7%     900326 ±  5%  vmstat.system.in
     38.37           -12.3       26.09 ±  3%  mpstat.cpu.all.idle%
      0.03 ±  7%      -0.0        0.01 ± 14%  mpstat.cpu.all.iowait%
      7.20            -1.5        5.73 ±  7%  mpstat.cpu.all.irq%
      0.22 ±  2%      -0.1        0.16 ±  7%  mpstat.cpu.all.soft%
     45.11           +12.4       57.48        mpstat.cpu.all.sys%
      9.07            +1.5       10.52 ±  3%  mpstat.cpu.all.usr%
      1.67 ± 82%   +9040.0%     152.33 ± 56%  numa-vmstat.node0.nr_mlock
     18.67 ±  7%    +807.1%     169.33 ± 51%  numa-vmstat.node0.nr_unevictable
     18.67 ±  7%    +807.1%     169.33 ± 51%  numa-vmstat.node0.nr_zone_unevictable
    292934 ± 23%    +167.7%     784191 ± 21%  numa-vmstat.node0.numa_hit
    281540 ± 23%    +177.0%     779868 ± 21%  numa-vmstat.node0.numa_local
      1888 ± 12%    +986.6%      20520 ±  6%  numa-vmstat.node1.nr_active_anon
      1888 ± 12%    +986.6%      20520 ±  6%  numa-vmstat.node1.nr_zone_active_anon
    369806 ± 17%    +120.7%     816096 ± 21%  numa-vmstat.node1.numa_hit
    361367 ± 17%    +121.7%     801248 ± 22%  numa-vmstat.node1.numa_local
      2095           +10.6%       2317        turbostat.Avg_MHz
     75.48            +8.5       83.97        turbostat.Busy%
 1.658e+08          +224.5%  5.382e+08 ± 13%  turbostat.C1
      5.96 ±  4%      -1.5        4.48 ± 11%  turbostat.C1%
   2506058 ± 56%    +195.8%    7412626 ± 57%  turbostat.C1E
     20.85 ± 13%     -35.0%      13.55 ± 13%  turbostat.CPU%c1
  1.14e+08          +288.9%  4.435e+08 ± 10%  turbostat.IRQ
 1.347e+08          +306.7%  5.477e+08 ± 10%  turbostat.POLL
      3.86            -0.4        3.44 ±  6%  turbostat.POLL%
    250.18            +6.5%     266.34        turbostat.PkgWatt
      2872 ±  6%    +655.6%      21702 ±  7%  proc-vmstat.nr_active_anon
     29062           +29.7%      37689 ± 46%  proc-vmstat.nr_active_file
    488299            +6.5%     519898 ±  5%  proc-vmstat.nr_file_pages
     22777            -2.2%      22281        proc-vmstat.nr_kernel_stack
     21414            -5.0%      20337        proc-vmstat.nr_mapped
      3.00         +6705.6%     204.17 ± 20%  proc-vmstat.nr_mlock
      3621 ±  2%      +4.7%       3791        proc-vmstat.nr_page_table_pages
    180334           +11.2%     200518 ±  2%  proc-vmstat.nr_shmem
     26676           +22.7%      32721 ± 35%  proc-vmstat.nr_slab_reclaimable
     55226            +3.4%      57117 ±  2%  proc-vmstat.nr_slab_unreclaimable
     20.00         +1005.8%     221.17 ± 18%  proc-vmstat.nr_unevictable
      2872 ±  6%    +655.6%      21702 ±  7%  proc-vmstat.nr_zone_active_anon
     29062           +29.7%      37689 ± 46%  proc-vmstat.nr_zone_active_file
     20.00         +1005.8%     221.17 ± 18%  proc-vmstat.nr_zone_unevictable
    669039          +140.9%    1611469 ±  8%  proc-vmstat.numa_hit
    649181          +144.5%    1587192 ±  8%  proc-vmstat.numa_local
    784187          +125.5%    1768689 ±  8%  proc-vmstat.pgalloc_normal
    896335          +162.1%    2349602 ±  8%  proc-vmstat.pgfault
    686654 ±  2%    +139.5%    1644338 ±  8%  proc-vmstat.pgfree
    110663          +207.5%     340238 ±  8%  proc-vmstat.pgreuse
   1068544          +263.0%    3878912 ±  9%  proc-vmstat.unevictable_pgs_scanned
 1.054e+10           +40.3%  1.479e+10 ±  9%  perf-stat.i.branch-instructions
  2.04e+08           +11.5%  2.274e+08 ±  4%  perf-stat.i.branch-misses
   4915549 ±  4%     +11.9%    5499505 ±  8%  perf-stat.i.cache-misses
  1.07e+09           -11.9%  9.427e+08 ±  5%  perf-stat.i.cache-references
   8379617           +41.0%   11817356 ±  9%  perf-stat.i.context-switches
      4.01 ±  5%     -11.7%       3.54 ±  7%  perf-stat.i.cpi
 2.062e+11            +8.7%  2.243e+11        perf-stat.i.cpu-cycles
   3206852           -15.2%    2720851 ±  7%  perf-stat.i.cpu-migrations
 1.425e+10           +43.7%  2.047e+10 ± 10%  perf-stat.i.dTLB-loads
      0.07 ± 12%      -0.0        0.05 ± 14%  perf-stat.i.dTLB-store-miss-rate%
   5278267 ±  3%     -19.0%    4276247 ±  7%  perf-stat.i.dTLB-store-misses
  8.04e+09           +46.2%  1.176e+10 ± 10%  perf-stat.i.dTLB-stores
  29270062           +22.1%   35725942 ±  4%  perf-stat.i.iTLB-load-misses
  52729822           +27.5%   67248151 ±  6%  perf-stat.i.iTLB-loads
 5.149e+10           +41.5%  7.284e+10 ±  9%  perf-stat.i.instructions
      1.64 ±  6%     -74.5%       0.42 ± 10%  perf-stat.i.major-faults
      2.15            +8.8%       2.34        perf-stat.i.metric.GHz
    352.95           +41.6%     499.62 ±  9%  perf-stat.i.metric.M/sec
      5633           -24.3%       4266        perf-stat.i.minor-faults
     90.31            +3.0       93.28        perf-stat.i.node-load-miss-rate%
   2212505 ±  2%     +12.6%    2490436 ±  3%  perf-stat.i.node-load-misses
      5635           -24.3%       4266        perf-stat.i.page-faults
     20.77           -36.9%      13.11 ± 13%  perf-stat.overall.MPKI
      1.93            -0.4        1.54 ±  4%  perf-stat.overall.branch-miss-rate%
      0.46 ±  3%      +0.1        0.58 ±  7%  perf-stat.overall.cache-miss-rate%
      4.00           -22.4%       3.11 ±  9%  perf-stat.overall.cpi
      0.38 ±  8%      -0.1        0.24 ±  8%  perf-stat.overall.dTLB-load-miss-rate%
      0.07 ±  2%      -0.0        0.04 ± 16%  perf-stat.overall.dTLB-store-miss-rate%
     35.70            -1.0       34.72        perf-stat.overall.iTLB-load-miss-rate%
      1759           +15.6%       2034 ±  5%  perf-stat.overall.instructions-per-iTLB-miss
      0.25           +30.2%       0.33 ± 10%  perf-stat.overall.ipc
     93.90            +1.0       94.94        perf-stat.overall.node-load-miss-rate%
 1.044e+10           +41.3%  1.476e+10 ±  9%  perf-stat.ps.branch-instructions
  2.02e+08           +12.3%  2.269e+08 ±  4%  perf-stat.ps.branch-misses
   4888066 ±  4%     +12.3%    5487899 ±  8%  perf-stat.ps.cache-misses
 1.059e+09           -11.3%  9.402e+08 ±  5%  perf-stat.ps.cache-references
   8300663           +42.1%   11794849 ± 10%  perf-stat.ps.context-switches
 2.043e+11            +9.5%  2.237e+11        perf-stat.ps.cpu-cycles
   3175301           -14.6%    2712811 ±  7%  perf-stat.ps.cpu-migrations
 1.411e+10           +44.8%  2.043e+10 ± 10%  perf-stat.ps.dTLB-loads
   5227681 ±  3%     -18.4%    4263828 ±  7%  perf-stat.ps.dTLB-store-misses
 7.965e+09           +47.3%  1.174e+10 ± 10%  perf-stat.ps.dTLB-stores
  28995942           +22.9%   35648237 ±  4%  perf-stat.ps.iTLB-load-misses
  52224757           +28.5%   67109656 ±  6%  perf-stat.ps.iTLB-loads
 5.101e+10           +42.5%   7.27e+10 ± 10%  perf-stat.ps.instructions
      1.62 ±  6%     -74.3%       0.42 ± 10%  perf-stat.ps.major-faults
      5627           -24.4%       4256        perf-stat.ps.minor-faults
   2193451 ±  2%     +13.2%    2483610 ±  3%  perf-stat.ps.node-load-misses
      5629           -24.4%       4257        perf-stat.ps.page-faults
 5.883e+12          +506.6%  3.569e+13 ± 14%  perf-stat.total.instructions
    196075 ± 20%    +675.2%    1519913 ± 22%  sched_debug.cfs_rq:/.MIN_vruntime.avg
    880934 ±  2%    +959.5%    9333279 ± 22%  sched_debug.cfs_rq:/.MIN_vruntime.max
    361929 ±  7%    +821.7%    3335971 ± 21%  sched_debug.cfs_rq:/.MIN_vruntime.stddev
      0.57 ±  7%     +78.3%       1.01 ±  4%  sched_debug.cfs_rq:/.h_nr_running.avg
      0.60 ± 12%     +30.6%       0.78 ±  2%  sched_debug.cfs_rq:/.h_nr_running.stddev
      7613 ±  5%     +45.7%      11092 ±  2%  sched_debug.cfs_rq:/.load.avg
     32.92 ± 40%     -46.2%      17.70 ±  6%  sched_debug.cfs_rq:/.load_avg.avg
    556.17 ± 21%     -67.0%     183.28 ± 28%  sched_debug.cfs_rq:/.load_avg.max
     96.85 ± 32%     -67.7%      31.28 ± 22%  sched_debug.cfs_rq:/.load_avg.stddev
    196075 ± 20%    +675.2%    1519913 ± 22%  sched_debug.cfs_rq:/.max_vruntime.avg
    880934 ±  2%    +959.5%    9333279 ± 22%  sched_debug.cfs_rq:/.max_vruntime.max
    361929 ±  7%    +821.7%    3335971 ± 21%  sched_debug.cfs_rq:/.max_vruntime.stddev
    889655 ±  2%   +1037.3%   10118475 ± 12%  sched_debug.cfs_rq:/.min_vruntime.avg
    903843 ±  2%   +1051.9%   10411690 ± 12%  sched_debug.cfs_rq:/.min_vruntime.max
    876105 ±  2%   +1017.0%    9785960 ± 13%  sched_debug.cfs_rq:/.min_vruntime.min
      4900 ±  5%   +4241.8%     212778 ± 40%  sched_debug.cfs_rq:/.min_vruntime.stddev
      0.38 ±  5%     +72.0%       0.66 ±  3%  sched_debug.cfs_rq:/.nr_running.avg
    564.00           +68.6%     951.12 ±  4%  sched_debug.cfs_rq:/.runnable_avg.avg
      1263 ± 10%     +41.2%       1784 ±  5%  sched_debug.cfs_rq:/.runnable_avg.max
    203.26 ±  7%     +52.2%     309.32 ±  5%  sched_debug.cfs_rq:/.runnable_avg.stddev
      9113 ± 67%   +1960.8%     187805 ± 61%  sched_debug.cfs_rq:/.spread0.avg
     23301 ± 26%   +1963.7%     480870 ± 42%  sched_debug.cfs_rq:/.spread0.max
     -4421         +3175.2%    -144798        sched_debug.cfs_rq:/.spread0.min
      4878 ±  5%   +4258.8%     212641 ± 40%  sched_debug.cfs_rq:/.spread0.stddev
    378.07 ±  3%     +59.1%     601.51 ±  5%  sched_debug.cfs_rq:/.util_avg.avg
    939.33 ±  3%     +23.3%       1158 ±  5%  sched_debug.cfs_rq:/.util_avg.max
    146.76 ±  4%     +42.2%     208.76 ±  6%  sched_debug.cfs_rq:/.util_avg.stddev
     13.05 ±  7%   +1272.9%     179.21 ± 22%  sched_debug.cfs_rq:/.util_est_enqueued.avg
    475.17 ± 12%    +102.1%     960.48 ±  5%  sched_debug.cfs_rq:/.util_est_enqueued.max
     56.28 ±  8%    +334.6%     244.56 ±  3%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
    447090           -71.0%     129667 ± 38%  sched_debug.cpu.avg_idle.avg
    690047 ± 12%     -66.2%     233542 ± 44%  sched_debug.cpu.avg_idle.max
    105039 ± 10%     -67.7%      33934 ± 36%  sched_debug.cpu.avg_idle.stddev
     80235          +243.3%     275481 ±  8%  sched_debug.cpu.clock.avg
     80242          +243.3%     275492 ±  8%  sched_debug.cpu.clock.max
     80227          +243.4%     275469 ±  8%  sched_debug.cpu.clock.min
      4.22 ± 12%     +61.6%       6.81 ± 17%  sched_debug.cpu.clock.stddev
     78554          +238.5%     265921 ±  8%  sched_debug.cpu.clock_task.avg
     78911          +239.8%     268136 ±  8%  sched_debug.cpu.clock_task.max
     63258          +293.4%     248843 ±  8%  sched_debug.cpu.clock_task.min
      1588 ±  2%     +51.4%       2404 ± 15%  sched_debug.cpu.clock_task.stddev
      1756 ±  4%    +284.1%       6747 ± 12%  sched_debug.cpu.curr->pid.avg
      4095          +160.2%      10654 ±  6%  sched_debug.cpu.curr->pid.max
      1614          +170.2%       4363 ± 12%  sched_debug.cpu.curr->pid.stddev
    669155 ± 19%     -22.9%     515937        sched_debug.cpu.max_idle_balance_cost.max
     20831 ± 59%     -90.5%       1983 ± 28%  sched_debug.cpu.max_idle_balance_cost.stddev
      0.00 ± 13%     +87.8%       0.00 ± 12%  sched_debug.cpu.next_balance.stddev
      0.53 ±  5%     +83.7%       0.98 ±  5%  sched_debug.cpu.nr_running.avg
      0.59 ±  8%     +30.4%       0.77 ±  7%  sched_debug.cpu.nr_running.stddev
   2227313 ±  2%   +1137.4%   27560829 ± 16%  sched_debug.cpu.nr_switches.avg
   2457119 ±  3%   +1230.5%   32693099 ± 15%  sched_debug.cpu.nr_switches.max
   2000173 ±  3%   +1032.1%   22644429 ± 20%  sched_debug.cpu.nr_switches.min
    204086 ± 31%   +2005.5%    4297102 ± 37%  sched_debug.cpu.nr_switches.stddev
     80228          +243.4%     275469 ±  8%  sched_debug.cpu_clk
     79513          +245.5%     274754 ±  8%  sched_debug.ktime
     81061          +240.8%     276272 ±  8%  sched_debug.sched_clk
  58611259          +100.0%  1.172e+08        sched_debug.sysctl_sched.sysctl_sched_features
     43.00           -20.3       22.74 ± 68%  perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
     42.55           -20.0       22.50 ± 68%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
     42.53           -20.0       22.50 ± 68%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     42.46           -20.0       22.46 ± 68%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     20.34            -9.7       10.63 ± 70%  perf-profile.calltrace.cycles-pp.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     17.07            -8.2        8.89 ± 70%  perf-profile.calltrace.cycles-pp.sched_ttwu_pending.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary
     16.28            -7.8        8.47 ± 70%  perf-profile.calltrace.cycles-pp.ttwu_do_activate.sched_ttwu_pending.flush_smp_call_function_queue.do_idle.cpu_startup_entry
     16.15            -7.8        8.40 ± 70%  perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.sched_ttwu_pending.flush_smp_call_function_queue.do_idle
     15.06            -7.2        7.85 ± 70%  perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.sched_ttwu_pending.flush_smp_call_function_queue
     14.45            -7.1        7.33 ± 70%  perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.sched_ttwu_pending
     14.67            -7.1        7.60 ± 70%  perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     12.54            -6.1        6.48 ± 70%  perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
     12.50            -6.0        6.46 ± 70%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
      7.14            -3.8        3.38 ± 70%  perf-profile.calltrace.cycles-pp.intel_idle_irq.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
      7.19            -3.4        3.77 ± 70%  perf-profile.calltrace.cycles-pp.update_cfs_group.dequeue_entity.dequeue_task_fair.__schedule.schedule
      5.84            -2.9        2.99 ± 70%  perf-profile.calltrace.cycles-pp.update_cfs_group.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate
      5.73            -2.7        3.05 ± 70%  perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      5.60            -2.6        2.98 ± 70%  perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary
      7.18            -2.5        4.66 ± 47%  perf-profile.calltrace.cycles-pp.update_load_avg.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate
      4.80            -2.1        2.66 ± 70%  perf-profile.calltrace.cycles-pp.select_idle_cpu.select_idle_sibling.select_task_rq_fair.select_task_rq.try_to_wake_up
      6.55            -2.0        4.52 ± 31%  perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.pipe_write
      4.77            -2.0        2.76 ± 70%  perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
      6.54            -2.0        4.54 ± 29%  perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.pipe_read
      5.92            -1.7        4.26 ± 36%  perf-profile.calltrace.cycles-pp.select_idle_sibling.select_task_rq_fair.select_task_rq.try_to_wake_up.autoremove_wake_function
      3.32            -1.6        1.69 ± 70%  perf-profile.calltrace.cycles-pp.select_idle_core.select_idle_cpu.select_idle_sibling.select_task_rq_fair.select_task_rq
      6.75            -1.5        5.22 ± 26%  perf-profile.calltrace.cycles-pp.select_task_rq_fair.select_task_rq.try_to_wake_up.autoremove_wake_function.__wake_up_common
      3.12            -1.5        1.64 ± 71%  perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.sched_ttwu_pending.__sysvec_call_function_single.sysvec_call_function_single
      3.20            -1.4        1.76 ± 70%  perf-profile.calltrace.cycles-pp.ttwu_do_activate.sched_ttwu_pending.__sysvec_call_function_single.sysvec_call_function_single.asm_sysvec_call_function_single
      6.84            -1.4        5.42 ± 22%  perf-profile.calltrace.cycles-pp.select_task_rq.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      2.90            -1.4        1.52 ± 71%  perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.sched_ttwu_pending.__sysvec_call_function_single
      4.70            -1.3        3.42 ± 29%  perf-profile.calltrace.cycles-pp.update_load_avg.dequeue_entity.dequeue_task_fair.__schedule.schedule
      2.32            -1.1        1.17 ± 70%  perf-profile.calltrace.cycles-pp.update_cfs_group.enqueue_task_fair.activate_task.ttwu_do_activate.sched_ttwu_pending
      1.38            -1.1        0.26 ±100%  perf-profile.calltrace.cycles-pp.migrate_task_rq_fair.set_task_cpu.try_to_wake_up.autoremove_wake_function.__wake_up_common
      2.25            -1.0        1.22 ± 70%  perf-profile.calltrace.cycles-pp.asm_sysvec_call_function_single.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary
      1.25            -1.0        0.25 ±100%  perf-profile.calltrace.cycles-pp.__smp_call_single_queue.ttwu_queue_wakelist.try_to_wake_up.autoremove_wake_function.__wake_up_common
      2.14            -1.0        1.16 ± 70%  perf-profile.calltrace.cycles-pp.sysvec_call_function_single.asm_sysvec_call_function_single.flush_smp_call_function_queue.do_idle.cpu_startup_entry
      2.10            -1.0        1.13 ± 70%  perf-profile.calltrace.cycles-pp.__sysvec_call_function_single.sysvec_call_function_single.asm_sysvec_call_function_single.flush_smp_call_function_queue.do_idle
      2.09            -0.9        1.15 ± 70%  perf-profile.calltrace.cycles-pp.ttwu_queue_wakelist.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      1.99            -0.9        1.06 ± 70%  perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule_idle.do_idle.cpu_startup_entry
      1.82            -0.9        0.90 ± 70%  perf-profile.calltrace.cycles-pp.available_idle_cpu.select_idle_core.select_idle_cpu.select_idle_sibling.select_task_rq_fair
      1.97            -0.9        1.06 ± 70%  perf-profile.calltrace.cycles-pp.sched_ttwu_pending.__sysvec_call_function_single.sysvec_call_function_single.asm_sysvec_call_function_single.flush_smp_call_function_queue
      1.82            -0.9        0.96 ± 70%  perf-profile.calltrace.cycles-pp.set_next_entity.pick_next_task_fair.__schedule.schedule_idle.do_idle
      1.58            -0.8        0.79 ± 70%  perf-profile.calltrace.cycles-pp.set_task_cpu.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      1.62            -0.8        0.84 ± 70%  perf-profile.calltrace.cycles-pp.update_load_avg.set_next_entity.pick_next_task_fair.__schedule.schedule_idle
      1.46 ±  2%      -0.7        0.75 ± 70%  perf-profile.calltrace.cycles-pp.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
      1.04            -0.5        0.52 ± 70%  perf-profile.calltrace.cycles-pp.sched_mm_cid_migrate_to.activate_task.ttwu_do_activate.sched_ttwu_pending.flush_smp_call_function_queue
      0.87 ±  2%      -0.4        0.46 ± 70%  perf-profile.calltrace.cycles-pp.asm_sysvec_call_function_single.intel_idle_irq.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
      0.82            -0.4        0.43 ± 70%  perf-profile.calltrace.cycles-pp.sysvec_call_function_single.asm_sysvec_call_function_single.intel_idle_irq.cpuidle_enter_state.cpuidle_enter
      0.75 ±  2%      -0.4        0.36 ± 70%  perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry
      0.79            -0.4        0.41 ± 70%  perf-profile.calltrace.cycles-pp.__sysvec_call_function_single.sysvec_call_function_single.asm_sysvec_call_function_single.intel_idle_irq.cpuidle_enter_state
      0.74 ±  2%      -0.4        0.39 ± 70%  perf-profile.calltrace.cycles-pp.sched_ttwu_pending.__sysvec_call_function_single.sysvec_call_function_single.asm_sysvec_call_function_single.intel_idle_irq
      0.78            -0.4        0.42 ± 70%  perf-profile.calltrace.cycles-pp.switch_mm_irqs_off.__schedule.schedule_idle.do_idle.cpu_startup_entry
      0.69 ±  2%      -0.3        0.36 ± 70%  perf-profile.calltrace.cycles-pp.__flush_smp_call_function_queue.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary
      0.69            -0.3        0.38 ± 70%  perf-profile.calltrace.cycles-pp.finish_task_switch.__schedule.schedule_idle.do_idle.cpu_startup_entry
      0.78 ±  2%      -0.3        0.48 ± 70%  perf-profile.calltrace.cycles-pp.asm_sysvec_call_function_single.poll_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
      0.65 ±  2%      -0.3        0.37 ± 70%  perf-profile.calltrace.cycles-pp._raw_spin_lock.__schedule.schedule_idle.do_idle.cpu_startup_entry
      0.72 ±  2%      -0.3        0.45 ± 70%  perf-profile.calltrace.cycles-pp.sysvec_call_function_single.asm_sysvec_call_function_single.poll_idle.cpuidle_enter_state.cpuidle_enter
      0.70 ±  2%      -0.3        0.42 ± 70%  perf-profile.calltrace.cycles-pp.__sysvec_call_function_single.sysvec_call_function_single.asm_sysvec_call_function_single.poll_idle.cpuidle_enter_state
      0.65 ±  2%      -0.2        0.40 ± 70%  perf-profile.calltrace.cycles-pp.sched_ttwu_pending.__sysvec_call_function_single.sysvec_call_function_single.asm_sysvec_call_function_single.poll_idle
      0.56            -0.0        0.53 ±  2%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.prepare_to_wait_event.pipe_write.vfs_write.ksys_write
      0.51            +0.4        0.87 ± 49%  perf-profile.calltrace.cycles-pp.copy_page_from_iter.pipe_write.vfs_write.ksys_write.do_syscall_64
      0.92            +0.6        1.51 ± 44%  perf-profile.calltrace.cycles-pp.restore_fpregs_from_fpstate.switch_fpu_return.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
      0.00            +0.8        0.79 ± 47%  perf-profile.calltrace.cycles-pp._copy_from_iter.copy_page_from_iter.pipe_write.vfs_write.ksys_write
      1.46            +0.9        2.33 ± 30%  perf-profile.calltrace.cycles-pp.switch_fpu_return.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.51            +1.1        2.57 ± 34%  perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
     11.60            +2.0       13.62 ±  6%  perf-profile.calltrace.cycles-pp.__schedule.schedule.pipe_write.vfs_write.ksys_write
     11.63            +2.1       13.74 ±  7%  perf-profile.calltrace.cycles-pp.__schedule.schedule.pipe_read.vfs_read.ksys_read
     11.65            +2.2       13.83 ±  7%  perf-profile.calltrace.cycles-pp.schedule.pipe_write.vfs_write.ksys_write.do_syscall_64
     11.70            +2.3       13.97 ±  8%  perf-profile.calltrace.cycles-pp.schedule.pipe_read.vfs_read.ksys_read.do_syscall_64
      7.84            +2.6       10.46 ±  5%  perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_read
      8.12            +2.6       10.76 ±  5%  perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.pipe_read.vfs_read.ksys_read
      7.90            +2.7       10.58 ±  6%  perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_read.vfs_read
      7.78            +2.7       10.53 ±  7%  perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write
      6.05            +2.8        8.80 ± 35%  perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +2.8        2.75 ± 68%  perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up
      8.07            +2.8       10.84 ±  7%  perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.pipe_write.vfs_write.ksys_write
      8.31            +2.8       11.09 ±  6%  perf-profile.calltrace.cycles-pp.__wake_up_common_lock.pipe_read.vfs_read.ksys_read.do_syscall_64
      7.85            +2.8       10.66 ±  8%  perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write.vfs_write
      8.26            +2.9       11.18 ±  9%  perf-profile.calltrace.cycles-pp.__wake_up_common_lock.pipe_write.vfs_write.ksys_write.do_syscall_64
      0.00            +3.7        3.72 ± 43%  perf-profile.calltrace.cycles-pp.update_cfs_group.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up
     23.31            +6.5       29.83 ± 12%  perf-profile.calltrace.cycles-pp.pipe_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
     22.83            +6.6       29.45 ± 13%  perf-profile.calltrace.cycles-pp.pipe_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     23.81            +7.0       30.77 ± 14%  perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
     23.32            +7.1       30.44 ± 14%  perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     24.04            +7.2       31.19 ± 14%  perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
     23.57            +7.3       30.90 ± 15%  perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.11 ±  3%      +7.9        9.01 ± 30%  perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common
      0.88 ± 44%      +7.9        8.81 ± 30%  perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function
      1.13 ±  3%      +8.7        9.85 ± 36%  perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
     53.92           +17.5       71.42 ± 17%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
     54.24           +18.3       72.56 ± 18%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
     43.00           -20.3       22.74 ± 68%  perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     43.00           -20.3       22.74 ± 68%  perf-profile.children.cycles-pp.cpu_startup_entry
     42.94           -20.2       22.70 ± 68%  perf-profile.children.cycles-pp.do_idle
     42.55           -20.0       22.50 ± 68%  perf-profile.children.cycles-pp.start_secondary
     21.77           -10.2       11.58 ± 69%  perf-profile.children.cycles-pp.sched_ttwu_pending
     20.67            -9.8       10.87 ± 69%  perf-profile.children.cycles-pp.flush_smp_call_function_queue
     14.82            -7.1        7.75 ± 68%  perf-profile.children.cycles-pp.cpuidle_idle_call
     12.67            -6.1        6.60 ± 68%  perf-profile.children.cycles-pp.cpuidle_enter
     12.65            -6.1        6.59 ± 68%  perf-profile.children.cycles-pp.cpuidle_enter_state
     16.44            -5.1       11.38 ± 34%  perf-profile.children.cycles-pp.enqueue_entity
     13.12            -4.0        9.16 ± 29%  perf-profile.children.cycles-pp.dequeue_entity
      7.29            -3.8        3.46 ± 70%  perf-profile.children.cycles-pp.intel_idle_irq
     15.52            -3.4       12.13 ± 22%  perf-profile.children.cycles-pp.update_load_avg
      5.80            -2.7        3.11 ± 68%  perf-profile.children.cycles-pp.schedule_idle
      5.23            -2.3        2.96 ± 68%  perf-profile.children.cycles-pp.asm_sysvec_call_function_single
      4.93            -2.1        2.78 ± 69%  perf-profile.children.cycles-pp.sysvec_call_function_single
      4.79            -2.1        2.69 ± 69%  perf-profile.children.cycles-pp.__sysvec_call_function_single
      4.86            -2.0        2.85 ± 68%  perf-profile.children.cycles-pp.poll_idle
      4.85            -1.9        2.97 ± 51%  perf-profile.children.cycles-pp.select_idle_cpu
      5.97            -1.6        4.33 ± 35%  perf-profile.children.cycles-pp.select_idle_sibling
      3.35            -1.6        1.72 ± 69%  perf-profile.children.cycles-pp.select_idle_core
      6.76            -1.5        5.24 ± 25%  perf-profile.children.cycles-pp.select_task_rq_fair
      3.39            -1.5        1.88 ± 65%  perf-profile.children.cycles-pp.available_idle_cpu
      6.85            -1.4        5.42 ± 22%  perf-profile.children.cycles-pp.select_task_rq
      2.10            -0.8        1.30 ± 47%  perf-profile.children.cycles-pp.ttwu_queue_wakelist
      1.58            -0.8        0.79 ± 70%  perf-profile.children.cycles-pp.set_task_cpu
      1.38            -0.7        0.67 ± 70%  perf-profile.children.cycles-pp.migrate_task_rq_fair
      1.48 ±  2%      -0.7        0.78 ± 66%  perf-profile.children.cycles-pp.menu_select
      1.36            -0.6        0.74 ± 70%  perf-profile.children.cycles-pp.sched_mm_cid_migrate_to
      1.27            -0.6        0.67 ± 70%  perf-profile.children.cycles-pp.__smp_call_single_queue
      1.22            -0.6        0.64 ± 70%  perf-profile.children.cycles-pp.llist_add_batch
      1.59            -0.5        1.12 ± 29%  perf-profile.children.cycles-pp.finish_task_switch
      2.62            -0.5        2.17 ± 15%  perf-profile.children.cycles-pp._raw_spin_lock
      0.96            -0.4        0.51 ± 70%  perf-profile.children.cycles-pp.__flush_smp_call_function_queue
      0.76 ±  2%      -0.4        0.37 ± 70%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.53 ±  2%      -0.3        0.24 ± 70%  perf-profile.children.cycles-pp.remove_entity_load_avg
      0.49            -0.3        0.22 ± 70%  perf-profile.children.cycles-pp.tick_nohz_next_event
      0.56 ±  2%      -0.2        0.31 ± 70%  perf-profile.children.cycles-pp.wake_affine
      0.54            -0.2        0.30 ± 70%  perf-profile.children.cycles-pp._find_next_bit
      0.54 ±  2%      -0.2        0.31 ± 70%  perf-profile.children.cycles-pp.nohz_run_idle_balance
      0.46 ±  2%      -0.2        0.24 ± 70%  perf-profile.children.cycles-pp.llist_reverse_order
      0.41            -0.2        0.21 ± 70%  perf-profile.children.cycles-pp.send_call_function_single_ipi
      0.37 ±  2%      -0.2        0.18 ± 70%  perf-profile.children.cycles-pp.get_next_timer_interrupt
      0.35            -0.2        0.19 ± 70%  perf-profile.children.cycles-pp.pick_next_task_idle
      0.30            -0.2        0.15 ± 70%  perf-profile.children.cycles-pp.attach_entity_load_avg
      0.29            -0.1        0.15 ± 70%  perf-profile.children.cycles-pp.task_h_load
      0.30 ±  2%      -0.1        0.17 ± 70%  perf-profile.children.cycles-pp.__update_idle_core
      0.28 ±  4%      -0.1        0.15 ± 59%  perf-profile.children.cycles-pp.__irq_exit_rcu
      0.23 ±  3%      -0.1        0.12 ± 70%  perf-profile.children.cycles-pp.call_cpuidle
      0.21 ±  3%      -0.1        0.10 ± 70%  perf-profile.children.cycles-pp.__do_softirq
      0.21            -0.1        0.12 ± 70%  perf-profile.children.cycles-pp.ct_kernel_exit_state
      0.20 ±  7%      -0.1        0.11 ± 71%  perf-profile.children.cycles-pp.hrtimer_next_event_without
      0.20 ±  4%      -0.1        0.10 ± 70%  perf-profile.children.cycles-pp.hrtimer_get_next_event
      0.72 ±  5%      -0.1        0.63 ±  4%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
      0.18 ±  4%      -0.1        0.10 ± 70%  perf-profile.children.cycles-pp.__bitmap_andnot
      0.63 ±  5%      -0.1        0.56 ±  4%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
      0.14 ±  3%      -0.1        0.07 ± 70%  perf-profile.children.cycles-pp.put_prev_task_fair
      0.13 ±  5%      -0.1        0.06 ± 70%  perf-profile.children.cycles-pp.rebalance_domains
      0.16 ±  3%      -0.1        0.09 ± 70%  perf-profile.children.cycles-pp.tick_nohz_idle_exit
      0.14 ±  4%      -0.1        0.08 ± 70%  perf-profile.children.cycles-pp.cpuidle_governor_latency_req
      0.14 ±  4%      -0.1        0.08 ± 70%  perf-profile.children.cycles-pp.local_clock
      0.13 ±  3%      -0.1        0.08 ± 70%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.12 ±  3%      -0.1        0.07 ± 70%  perf-profile.children.cycles-pp.resched_curr
      0.11 ±  6%      -0.1        0.06 ± 70%  perf-profile.children.cycles-pp.newidle_balance
      0.11 ±  4%      -0.0        0.06 ± 70%  perf-profile.children.cycles-pp.ct_idle_exit
      0.10 ±  3%      -0.0        0.06 ± 71%  perf-profile.children.cycles-pp.__task_rq_lock
      0.07 ±  6%      -0.0        0.04 ± 71%  perf-profile.children.cycles-pp.load_balance
      0.09 ±  4%      -0.0        0.05 ± 71%  perf-profile.children.cycles-pp.ct_kernel_enter
      0.08 ±  5%      -0.0        0.05 ± 70%  perf-profile.children.cycles-pp.__mutex_lock
      0.06            +0.1        0.13 ± 63%  perf-profile.children.cycles-pp.__get_task_ioprio
      0.07            +0.1        0.16 ± 62%  perf-profile.children.cycles-pp.clear_buddies
      0.11 ±  6%      +0.1        0.20 ± 45%  perf-profile.children.cycles-pp.put_prev_entity
      0.10 ±  5%      +0.1        0.19 ± 54%  perf-profile.children.cycles-pp.__rdgsbase_inactive
      0.00            +0.1        0.11 ± 77%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode_prepare
      0.06 ±  7%      +0.1        0.20 ± 67%  perf-profile.children.cycles-pp.__cond_resched
      0.11 ±  3%      +0.1        0.25 ± 63%  perf-profile.children.cycles-pp.__list_add_valid
      0.18 ±  2%      +0.2        0.32 ± 52%  perf-profile.children.cycles-pp.__wrgsbase_inactive
      0.07 ±  6%      +0.2        0.24 ± 70%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.10 ±  5%      +0.2        0.26 ± 68%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.10 ±  5%      +0.2        0.27 ± 71%  perf-profile.children.cycles-pp.__entry_text_start
      0.08 ± 11%      +0.2        0.26 ± 72%  perf-profile.children.cycles-pp.__cgroup_account_cputime
      0.11 ±  4%      +0.2        0.29 ± 72%  perf-profile.children.cycles-pp.__might_sleep
      0.23 ±  2%      +0.2        0.42 ± 50%  perf-profile.children.cycles-pp.finish_wait
      0.28 ±  3%      +0.2        0.47 ± 50%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.13 ±  2%      +0.2        0.34 ± 68%  perf-profile.children.cycles-pp.entry_SYSRETQ_unsafe_stack
      0.10 ±  7%      +0.2        0.31 ± 71%  perf-profile.children.cycles-pp.pick_next_entity
      0.24 ±  3%      +0.2        0.45 ± 55%  perf-profile.children.cycles-pp.atime_needs_update
      0.14 ±  3%      +0.2        0.35 ± 76%  perf-profile.children.cycles-pp.current_time
      0.05 ±  8%      +0.2        0.27 ±104%  perf-profile.children.cycles-pp.set_next_buddy
      0.15 ±  7%      +0.2        0.38 ± 66%  perf-profile.children.cycles-pp.cpuacct_charge
      0.46            +0.2        0.68 ± 41%  perf-profile.children.cycles-pp.native_sched_clock
      0.38            +0.2        0.61 ± 47%  perf-profile.children.cycles-pp._copy_to_iter
      0.32 ±  2%      +0.3        0.57 ± 53%  perf-profile.children.cycles-pp.touch_atime
      0.00            +0.3        0.26 ± 61%  perf-profile.children.cycles-pp.wake_on_current
      0.40 ±  2%      +0.3        0.67 ± 49%  perf-profile.children.cycles-pp.copy_page_to_iter
      0.32 ±  2%      +0.3        0.59 ± 59%  perf-profile.children.cycles-pp.apparmor_file_permission
      0.13            +0.3        0.42 ± 78%  perf-profile.children.cycles-pp.__might_fault
      0.39            +0.3        0.71 ± 51%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.48            +0.3        0.81 ± 48%  perf-profile.children.cycles-pp._copy_from_iter
      0.36 ±  3%      +0.3        0.70 ± 61%  perf-profile.children.cycles-pp.security_file_permission
      0.18 ±  5%      +0.4        0.54 ± 75%  perf-profile.children.cycles-pp.update_min_vruntime
      0.52            +0.4        0.88 ± 49%  perf-profile.children.cycles-pp.copy_page_from_iter
      0.30 ±  2%      +0.4        0.67 ± 62%  perf-profile.children.cycles-pp.__might_resched
      0.34            +0.5        0.86 ± 65%  perf-profile.children.cycles-pp.os_xsave
      0.50 ±  2%      +0.5        1.03 ± 57%  perf-profile.children.cycles-pp.mutex_lock
      0.00            +0.6        0.57 ±107%  perf-profile.children.cycles-pp.check_preempt_wakeup
      0.92            +0.6        1.51 ± 44%  perf-profile.children.cycles-pp.restore_fpregs_from_fpstate
      0.46 ±  2%      +0.6        1.07 ± 63%  perf-profile.children.cycles-pp.update_rq_clock
      0.19 ±  3%      +0.6        0.84 ± 87%  perf-profile.children.cycles-pp.check_preempt_curr
      0.17 ±  2%      +0.7        0.84 ± 89%  perf-profile.children.cycles-pp.__calc_delta
      0.63 ±  2%      +0.8        1.42 ± 66%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.38 ±  3%      +0.8        1.19 ± 75%  perf-profile.children.cycles-pp.__update_load_avg_se
      1.46            +0.9        2.34 ± 30%  perf-profile.children.cycles-pp.switch_fpu_return
      1.52            +1.1        2.58 ± 35%  perf-profile.children.cycles-pp.exit_to_user_mode_prepare
     28.96            +1.6       30.56        perf-profile.children.cycles-pp.__schedule
      1.26            +1.6        2.88 ± 63%  perf-profile.children.cycles-pp.switch_mm_irqs_off
      0.46 ±  3%      +1.8        2.22 ± 87%  perf-profile.children.cycles-pp.reweight_entity
      1.01 ±  3%      +2.1        3.15 ± 79%  perf-profile.children.cycles-pp.update_curr
      6.06            +2.8        8.82 ± 35%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
     23.36            +4.5       27.87 ±  8%  perf-profile.children.cycles-pp.schedule
     15.63            +5.4       21.03 ±  6%  perf-profile.children.cycles-pp.try_to_wake_up
     16.19            +5.4       21.61 ±  6%  perf-profile.children.cycles-pp.__wake_up_common
     15.75            +5.5       21.26 ±  7%  perf-profile.children.cycles-pp.autoremove_wake_function
     16.58            +5.7       22.29 ±  8%  perf-profile.children.cycles-pp.__wake_up_common_lock
     23.33            +6.6       29.91 ± 12%  perf-profile.children.cycles-pp.pipe_read
     22.85            +6.7       29.51 ± 13%  perf-profile.children.cycles-pp.pipe_write
     23.82            +7.0       30.78 ± 14%  perf-profile.children.cycles-pp.vfs_read
     23.33            +7.1       30.45 ± 14%  perf-profile.children.cycles-pp.vfs_write
     24.04            +7.2       31.19 ± 14%  perf-profile.children.cycles-pp.ksys_read
     23.57            +7.3       30.90 ± 15%  perf-profile.children.cycles-pp.ksys_write
     53.93           +17.5       71.46 ± 17%  perf-profile.children.cycles-pp.do_syscall_64
     54.24           +18.3       72.56 ± 18%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     13.80            -4.6        9.18 ± 46%  perf-profile.self.cycles-pp.update_load_avg
      6.40            -3.4        3.00 ± 69%  perf-profile.self.cycles-pp.intel_idle_irq
      3.97            -1.7        2.29 ± 68%  perf-profile.self.cycles-pp.poll_idle
      3.32            -1.5        1.84 ± 65%  perf-profile.self.cycles-pp.available_idle_cpu
      3.10            -1.1        1.96 ± 37%  perf-profile.self.cycles-pp.try_to_wake_up
      1.36            -0.6        0.74 ± 70%  perf-profile.self.cycles-pp.sched_mm_cid_migrate_to
      1.20            -0.6        0.63 ± 70%  perf-profile.self.cycles-pp.llist_add_batch
      1.02            -0.5        0.54 ± 70%  perf-profile.self.cycles-pp.select_idle_core
      0.82            -0.4        0.42 ± 70%  perf-profile.self.cycles-pp.migrate_task_rq_fair
      2.06            -0.3        1.78 ±  4%  perf-profile.self.cycles-pp._raw_spin_lock
      0.54 ±  2%      -0.3        0.28 ± 70%  perf-profile.self.cycles-pp.menu_select
      0.51 ±  2%      -0.2        0.27 ± 70%  perf-profile.self.cycles-pp.__flush_smp_call_function_queue
      0.50            -0.2        0.28 ± 70%  perf-profile.self.cycles-pp._find_next_bit
      0.44 ±  3%      -0.2        0.23 ± 70%  perf-profile.self.cycles-pp.llist_reverse_order
      0.40            -0.2        0.21 ± 70%  perf-profile.self.cycles-pp.send_call_function_single_ipi
      0.40            -0.2        0.22 ± 70%  perf-profile.self.cycles-pp.sched_ttwu_pending
      0.36            -0.2        0.19 ± 70%  perf-profile.self.cycles-pp.flush_smp_call_function_queue
      0.40            -0.2        0.23 ± 70%  perf-profile.self.cycles-pp.do_idle
      0.30            -0.1        0.15 ± 70%  perf-profile.self.cycles-pp.attach_entity_load_avg
      0.29            -0.1        0.15 ± 70%  perf-profile.self.cycles-pp.task_h_load
      0.25 ±  2%      -0.1        0.14 ± 71%  perf-profile.self.cycles-pp.nohz_run_idle_balance
      0.58            -0.1        0.47 ±  8%  perf-profile.self.cycles-pp.finish_task_switch
      0.22 ±  3%      -0.1        0.12 ± 70%  perf-profile.self.cycles-pp.call_cpuidle
      0.21            -0.1        0.12 ± 70%  perf-profile.self.cycles-pp.ct_kernel_exit_state
      0.22 ±  2%      -0.1        0.12 ± 70%  perf-profile.self.cycles-pp.__update_idle_core
      0.44 ±  2%      -0.1        0.35 ± 12%  perf-profile.self.cycles-pp.__wake_up_common
      0.20            -0.1        0.11 ± 70%  perf-profile.self.cycles-pp.cpuidle_idle_call
      0.20 ±  6%      -0.1        0.12 ± 71%  perf-profile.self.cycles-pp.set_task_cpu
      0.17 ±  4%      -0.1        0.09 ± 70%  perf-profile.self.cycles-pp.__bitmap_andnot
      0.13 ±  3%      -0.1        0.08 ± 70%  perf-profile.self.cycles-pp.native_irq_return_iret
      0.11 ±  4%      -0.1        0.06 ± 70%  perf-profile.self.cycles-pp.cpuidle_enter_state
      0.12 ±  3%      -0.1        0.07 ± 70%  perf-profile.self.cycles-pp.resched_curr
      0.13 ±  3%      -0.1        0.08 ± 70%  perf-profile.self.cycles-pp.wake_affine
      0.10 ±  4%      -0.0        0.06 ± 71%  perf-profile.self.cycles-pp.newidle_balance
      0.08 ±  6%      -0.0        0.03 ± 70%  perf-profile.self.cycles-pp.remove_entity_load_avg
      0.12 ±  4%      -0.0        0.08 ± 25%  perf-profile.self.cycles-pp.copyin
      0.07 ±  6%      -0.0        0.04 ± 71%  perf-profile.self.cycles-pp.__hrtimer_next_event_base
      0.08 ±  6%      -0.0        0.04 ± 71%  perf-profile.self.cycles-pp.schedule_idle
      0.09 ±  7%      +0.1        0.14 ± 35%  perf-profile.self.cycles-pp.put_prev_entity
      0.06 ±  6%      +0.1        0.13 ± 56%  perf-profile.self.cycles-pp.activate_task
      0.13 ±  3%      +0.1        0.20 ± 41%  perf-profile.self.cycles-pp.atime_needs_update
      0.06 ±  6%      +0.1        0.14 ± 61%  perf-profile.self.cycles-pp.clear_buddies
      0.00            +0.1        0.08 ± 49%  perf-profile.self.cycles-pp.sched_clock_cpu
      0.05 ±  8%      +0.1        0.14 ± 68%  perf-profile.self.cycles-pp.__wake_up_common_lock
      0.10 ±  5%      +0.1        0.18 ± 52%  perf-profile.self.cycles-pp.__rdgsbase_inactive
      0.09            +0.1        0.18 ± 59%  perf-profile.self.cycles-pp.select_task_rq
      0.00            +0.1        0.10 ± 68%  perf-profile.self.cycles-pp._copy_to_iter
      0.00            +0.1        0.11 ± 71%  perf-profile.self.cycles-pp.__cond_resched
      0.00            +0.1        0.11 ± 72%  perf-profile.self.cycles-pp.security_file_permission
      0.06 ±  6%      +0.1        0.18 ± 75%  perf-profile.self.cycles-pp.ksys_read
      0.08 ±  4%      +0.1        0.20 ± 63%  perf-profile.self.cycles-pp._copy_from_iter
      0.06 ±  7%      +0.1        0.20 ± 73%  perf-profile.self.cycles-pp.__cgroup_account_cputime
      0.10 ±  4%      +0.1        0.25 ± 62%  perf-profile.self.cycles-pp.__list_add_valid
      0.07 ±  5%      +0.1        0.21 ± 70%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.06 ±  6%      +0.1        0.20 ± 84%  perf-profile.self.cycles-pp.check_preempt_curr
      0.00            +0.1        0.15 ± 64%  perf-profile.self.cycles-pp.exit_to_user_mode_prepare
      0.18 ±  2%      +0.2        0.32 ± 52%  perf-profile.self.cycles-pp.__wrgsbase_inactive
      0.06 ±  6%      +0.2        0.21 ± 77%  perf-profile.self.cycles-pp.ksys_write
      0.09 ±  5%      +0.2        0.25 ± 70%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.09 ±  4%      +0.2        0.25 ± 70%  perf-profile.self.cycles-pp.__might_sleep
      0.12 ±  4%      +0.2        0.29 ± 61%  perf-profile.self.cycles-pp.set_next_entity
      0.10 ±  5%      +0.2        0.27 ± 71%  perf-profile.self.cycles-pp.__entry_text_start
      0.25 ±  3%      +0.2        0.43 ± 54%  perf-profile.self.cycles-pp.vfs_read
      0.10 ±  5%      +0.2        0.28 ± 70%  perf-profile.self.cycles-pp.pick_next_entity
      0.12 ±  4%      +0.2        0.30 ± 72%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.48            +0.2        0.67 ±  9%  perf-profile.self.cycles-pp.select_idle_sibling
      0.13 ±  3%      +0.2        0.32 ± 68%  perf-profile.self.cycles-pp.entry_SYSRETQ_unsafe_stack
      0.23 ±  2%      +0.2        0.43 ± 52%  perf-profile.self.cycles-pp.mutex_lock
      0.05 ±  7%      +0.2        0.26 ±104%  perf-profile.self.cycles-pp.set_next_buddy
      0.09 ±  4%      +0.2        0.31 ± 79%  perf-profile.self.cycles-pp.do_syscall_64
      0.15 ±  7%      +0.2        0.37 ± 65%  perf-profile.self.cycles-pp.cpuacct_charge
      0.22 ±  2%      +0.2        0.45 ± 55%  perf-profile.self.cycles-pp.update_rq_clock
      0.44            +0.2        0.67 ± 41%  perf-profile.self.cycles-pp.native_sched_clock
      0.27            +0.2        0.50 ± 52%  perf-profile.self.cycles-pp.vfs_write
      0.00            +0.3        0.26 ± 61%  perf-profile.self.cycles-pp.wake_on_current
      0.10 ±  3%      +0.3        0.37 ± 80%  perf-profile.self.cycles-pp.schedule
      0.54            +0.3        0.82 ±  5%  perf-profile.self.cycles-pp.switch_fpu_return
      0.20 ±  3%      +0.3        0.50 ± 71%  perf-profile.self.cycles-pp.pick_next_task_fair
      0.17 ±  6%      +0.3        0.50 ± 76%  perf-profile.self.cycles-pp.update_min_vruntime
      0.24 ±  2%      +0.3        0.58 ± 63%  perf-profile.self.cycles-pp.dequeue_entity
      0.30 ±  3%      +0.4        0.66 ± 62%  perf-profile.self.cycles-pp.__might_resched
      0.23            +0.4        0.60 ± 70%  perf-profile.self.cycles-pp.select_task_rq_fair
      0.41            +0.4        0.83 ± 64%  perf-profile.self.cycles-pp.enqueue_task_fair
      0.36 ±  2%      +0.4        0.78 ± 60%  perf-profile.self.cycles-pp.pipe_write
      0.00            +0.5        0.46 ±107%  perf-profile.self.cycles-pp.check_preempt_wakeup
      0.60 ±  2%      +0.5        1.06 ± 56%  perf-profile.self.cycles-pp.enqueue_entity
      0.34            +0.5        0.86 ± 65%  perf-profile.self.cycles-pp.os_xsave
      0.92            +0.6        1.51 ± 44%  perf-profile.self.cycles-pp.restore_fpregs_from_fpstate
      0.36            +0.6        0.97 ± 64%  perf-profile.self.cycles-pp.reweight_entity
      0.17 ±  3%      +0.7        0.83 ± 89%  perf-profile.self.cycles-pp.__calc_delta
      0.61 ±  2%      +0.8        1.37 ± 66%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.46 ±  6%      +0.8        1.24 ± 78%  perf-profile.self.cycles-pp.update_curr
      0.37 ±  3%      +0.8        1.16 ± 76%  perf-profile.self.cycles-pp.__update_load_avg_se
      0.30            +0.8        1.10 ± 56%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      1.24            +1.6        2.85 ± 63%  perf-profile.self.cycles-pp.switch_mm_irqs_off
      4.50            +1.6        6.14 ± 34%  perf-profile.self.cycles-pp.syscall_exit_to_user_mode




Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests



View attachment "config-6.3.0-rc7-00020-g485a13bb0c94" of type "text/plain" (157070 bytes)

View attachment "job-script" of type "text/plain" (7504 bytes)

View attachment "job.yaml" of type "text/plain" (5136 bytes)

View attachment "reproduce" of type "text/plain" (301 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ