lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <YsmT6VAqPybIe9Pu@xsang-OptiPlex-9020>
Date:   Sat, 9 Jul 2022 22:42:49 +0800
From:   kernel test robot <oliver.sang@...el.com>
To:     Abel Wu <wuyun.abel@...edance.com>
Cc:     0day robot <lkp@...el.com>, LKML <linux-kernel@...r.kernel.org>,
        lkp@...ts.01.org, ying.huang@...el.com, feng.tang@...el.com,
        zhengjun.xing@...ux.intel.com, fengwei.yin@...el.com,
        guobing.chen@...el.com, ming.a.chen@...el.com, frank.du@...el.com,
        Shuhua.Fan@...el.com, wangyang.guo@...el.com,
        Wenhuan.Huang@...el.com, jessica.ji@...el.com, shan.kang@...el.com,
        guangli.li@...el.com, tiejun.li@...el.com, yu.ma@...el.com,
        dapeng1.mi@...el.com, jiebin.sun@...el.com, gengxin.xie@...el.com,
        fan.zhao@...el.com, Peter Zijlstra <peterz@...radead.org>,
        Mel Gorman <mgorman@...e.de>,
        Vincent Guittot <vincent.guittot@...aro.org>,
        Josh Don <joshdon@...gle.com>, Chen Yu <yu.c.chen@...el.com>,
        Tim Chen <tim.c.chen@...ux.intel.com>,
        K Prateek Nayak <kprateek.nayak@....com>,
        "Gautham R . Shenoy" <gautham.shenoy@....com>,
        Abel Wu <wuyun.abel@...edance.com>, aubrey.li@...ux.intel.com
Subject: [sched/fair]  32fe13cd7a:
 phoronix-test-suite.fio.SequentialWrite.IO_uring.Yes.No.4KB.DefaultTestDirectory.mb_s
 -11.7% regression



Greeting,

FYI, we noticed a -11.7% regression of phoronix-test-suite.fio.SequentialWrite.IO_uring.Yes.No.4KB.DefaultTestDirectory.mb_s due to commit:


commit: 32fe13cd7aa184ed349d698ebf6f420fa426dd73 ("[PATCH v4 7/7] sched/fair: de-entropy for SIS filter")
url: https://github.com/intel-lab-lkp/linux/commits/Abel-Wu/sched-fair-improve-scan-efficiency-of-SIS/20220619-200743
base: https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git f3dd3f674555bd9455c5ae7fafce0696bd9931b3
patch link: https://lore.kernel.org/lkml/20220619120451.95251-8-wuyun.abel@bytedance.com

in testcase: phoronix-test-suite
on test machine: 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 512G memory
with following parameters:

	test: fio-1.14.1
	option_a: Sequential Write
	option_b: IO_uring
	option_c: Yes
	option_d: No
	option_e: 4KB
	option_f: Default Test Directory
	cpufreq_governor: performance
	ucode: 0x500320a

test-description: The Phoronix Test Suite is the most comprehensive testing and benchmarking platform available that provides an extensible framework for which new tests can be easily added.
test-url: http://www.phoronix-test-suite.com/

In addition to that, the commit also has significant impact on the following tests:

+------------------+-------------------------------------------------------------------------------------+
| testcase: change | stress-ng: stress-ng.vm-rw.ops_per_sec 113.5% improvement                           |
| test machine     | 128 threads 2 sockets Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz with 128G memory |
| test parameters  | class=memory                                                                        |
|                  | cpufreq_governor=performance                                                        |
|                  | nr_threads=100%                                                                     |
|                  | test=vm-rw                                                                          |
|                  | testtime=60s                                                                        |
|                  | ucode=0xd000331                                                                     |
+------------------+-------------------------------------------------------------------------------------+


If you fix the issue, kindly add following tag
Reported-by: kernel test robot <oliver.sang@...el.com>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        sudo bin/lkp install job.yaml           # job file is attached in this email
        bin/lkp split-job --compatible job.yaml # generate the yaml file for lkp run
        sudo bin/lkp run generated-yaml-file

        # if come across any failure that blocks the test,
        # please remove ~/.lkp and /lkp dir to run from a clean state.

=========================================================================================
compiler/cpufreq_governor/kconfig/option_a/option_b/option_c/option_d/option_e/option_f/rootfs/tbox_group/test/testcase/ucode:
  gcc-11/performance/x86_64-rhel-8.3/Sequential Write/IO_uring/Yes/No/4KB/Default Test Directory/debian-x86_64-phoronix/lkp-csl-2sp7/fio-1.14.1/phoronix-test-suite/0x500320a

commit: 
  fcc108377a ("sched/fair: skip busy cores in SIS search")
  32fe13cd7a ("sched/fair: de-entropy for SIS filter")

fcc108377a7cf79c 32fe13cd7aa184ed349d698ebf6 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
    166666           -11.6%     147277 ±  3%  phoronix-test-suite.fio.SequentialWrite.IO_uring.Yes.No.4KB.DefaultTestDirectory.iops
    651.00           -11.7%     574.83 ±  3%  phoronix-test-suite.fio.SequentialWrite.IO_uring.Yes.No.4KB.DefaultTestDirectory.mb_s
      3145 ±  5%     -18.4%       2565 ± 12%  meminfo.Writeback
      0.19 ±  4%      -0.0        0.17 ±  2%  mpstat.cpu.all.iowait%
      2228 ± 33%     -37.5%       1392 ± 21%  numa-meminfo.node0.Writeback
    553.33 ± 37%     -35.9%     354.83 ± 18%  numa-vmstat.node0.nr_writeback
    445604 ±  4%     -12.5%     390116 ±  4%  vmstat.io.bo
  14697101 ±  3%     -11.0%   13074497 ±  4%  perf-stat.i.cache-misses
      9447 ±  8%     -37.6%       5890 ±  5%  perf-stat.i.cpu-migrations
      5125 ±  6%     +12.9%       5786 ±  6%  perf-stat.i.instructions-per-iTLB-miss
   2330431 ±  4%     -11.4%    2064845 ±  4%  perf-stat.i.node-loads
      2.55 ±104%      -1.6        0.96 ± 14%  perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
      2.62 ±102%      -1.6        0.99 ± 14%  perf-profile.children.cycles-pp.poll_idle
      0.82 ± 23%      -0.3        0.53 ± 23%  perf-profile.children.cycles-pp.asm_sysvec_call_function_single
      0.74 ± 23%      -0.3        0.46 ± 23%  perf-profile.children.cycles-pp.sysvec_call_function_single
      0.69 ± 24%      -0.3        0.44 ± 24%  perf-profile.children.cycles-pp.__sysvec_call_function_single
      0.38 ± 10%      -0.1        0.28 ± 18%  perf-profile.children.cycles-pp.__perf_event_header__init_id
      0.16 ± 13%      -0.0        0.11 ± 22%  perf-profile.children.cycles-pp.__task_pid_nr_ns
      2.10 ±108%      -1.3        0.79 ± 11%  perf-profile.self.cycles-pp.poll_idle
      0.16 ± 13%      -0.0        0.11 ± 22%  perf-profile.self.cycles-pp.__task_pid_nr_ns


***************************************************************************************************
lkp-icl-2sp6: 128 threads 2 sockets Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz with 128G memory
=========================================================================================
class/compiler/cpufreq_governor/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime/ucode:
  memory/gcc-11/performance/x86_64-rhel-8.3/100%/debian-11.1-x86_64-20220510.cgz/lkp-icl-2sp6/vm-rw/stress-ng/60s/0xd000331

commit: 
  fcc108377a ("sched/fair: skip busy cores in SIS search")
  32fe13cd7a ("sched/fair: de-entropy for SIS filter")

fcc108377a7cf79c 32fe13cd7aa184ed349d698ebf6 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
   7328835 ± 17%   +3441.0%  2.595e+08 ± 12%  stress-ng.time.involuntary_context_switches
    123165 ±  3%     -14.1%     105742 ±  2%  stress-ng.time.minor_page_faults
      8940           +32.8%      11872 ±  2%  stress-ng.time.percent_of_cpu_this_job_got
      5268           +33.4%       7027 ±  2%  stress-ng.time.system_time
    278.70           +21.5%     338.70 ±  2%  stress-ng.time.user_time
 2.554e+08           +13.3%  2.894e+08        stress-ng.time.voluntary_context_switches
 1.283e+08          +113.5%   2.74e+08 ±  6%  stress-ng.vm-rw.ops
   2139049          +113.5%    4567054 ±  6%  stress-ng.vm-rw.ops_per_sec
     39411 ± 34%     +56.3%      61612 ± 24%  numa-meminfo.node1.Mapped
      5013           -22.5%       3883 ±  4%  uptime.idle
 1.798e+09           -60.3%  7.135e+08 ± 21%  cpuidle..time
 1.701e+08           -87.3%   21598951 ± 90%  cpuidle..usage
     75821 ±  2%     -11.6%      67063 ±  5%  meminfo.Active
     75821 ±  2%     -11.6%      67063 ±  5%  meminfo.Active(anon)
     81710 ±  2%     +20.1%      98158 ±  3%  meminfo.Mapped
     26.00           -59.6%      10.50 ± 18%  vmstat.cpu.id
    112.00           +10.9%     124.17        vmstat.procs.r
   6561639           +31.6%    8634043 ±  2%  vmstat.system.cs
    990604           -62.4%     372118 ± 18%  vmstat.system.in
     24.13           -16.1        8.03 ± 23%  mpstat.cpu.all.idle%
      2.71            -1.6        1.11 ± 10%  mpstat.cpu.all.irq%
      0.17 ±  6%      -0.1        0.06 ± 30%  mpstat.cpu.all.soft%
     69.33           +17.4       86.71 ±  2%  mpstat.cpu.all.sys%
      3.66            +0.4        4.09        mpstat.cpu.all.usr%
 2.024e+09           +93.3%  3.912e+09 ± 16%  numa-vmstat.node0.nr_foll_pin_acquired
 2.024e+09           +93.3%  3.912e+09 ± 16%  numa-vmstat.node0.nr_foll_pin_released
 2.043e+09 ±  2%    +119.0%  4.473e+09        numa-vmstat.node1.nr_foll_pin_acquired
 2.043e+09 ±  2%    +119.0%  4.473e+09        numa-vmstat.node1.nr_foll_pin_released
      9865 ± 34%     +54.1%      15201 ± 23%  numa-vmstat.node1.nr_mapped
     18954 ±  2%     -11.5%      16767 ±  5%  proc-vmstat.nr_active_anon
 4.062e+09          +107.3%  8.419e+09 ±  7%  proc-vmstat.nr_foll_pin_acquired
 4.062e+09          +107.3%  8.419e+09 ±  7%  proc-vmstat.nr_foll_pin_released
     87380            +5.3%      92039        proc-vmstat.nr_inactive_anon
     24453            -3.2%      23658        proc-vmstat.nr_kernel_stack
     20437 ±  2%     +19.6%      24443 ±  3%  proc-vmstat.nr_mapped
     18954 ±  2%     -11.5%      16767 ±  5%  proc-vmstat.nr_zone_active_anon
     87380            +5.3%      92039        proc-vmstat.nr_zone_inactive_anon
    108777 ±  4%     -17.2%      90014        proc-vmstat.numa_hint_faults
     96756 ±  6%     -17.6%      79691 ±  2%  proc-vmstat.numa_hint_faults_local
    490607            -4.4%     469155        proc-vmstat.pgfault
     80.85           +10.9       91.75        turbostat.Busy%
      3221            -5.0%       3060        turbostat.Bzy_MHz
  77259218 ±  3%     -87.0%   10057388 ± 92%  turbostat.C1
      6.74 ±  2%      -5.9        0.85 ± 90%  turbostat.C1%
  92212921           -87.8%   11243535 ± 91%  turbostat.C1E
     12.00 ± 22%      -6.6        5.42 ± 57%  turbostat.C1E%
     16.39 ± 16%     -62.0%       6.24 ± 55%  turbostat.CPU%c1
      0.16 ±  3%     +74.7%       0.29 ±  6%  turbostat.IPC
  65322725           -62.5%   24502370 ± 18%  turbostat.IRQ
    339708           -86.5%      45941 ± 88%  turbostat.POLL
      0.05            -0.0        0.01 ± 82%  turbostat.POLL%
    165121 ± 23%    -100.0%      39.19 ±101%  sched_debug.cfs_rq:/.MIN_vruntime.avg
   2462709           -99.9%       3407 ±102%  sched_debug.cfs_rq:/.MIN_vruntime.max
    607348 ± 11%     -99.9%     348.57 ±100%  sched_debug.cfs_rq:/.MIN_vruntime.stddev
      0.56 ±  4%     +11.8%       0.62 ±  3%  sched_debug.cfs_rq:/.h_nr_running.avg
      2.58 ± 13%     -38.7%       1.58 ± 11%  sched_debug.cfs_rq:/.h_nr_running.max
      0.54 ±  9%     -39.7%       0.33 ±  6%  sched_debug.cfs_rq:/.h_nr_running.stddev
    165121 ± 23%    -100.0%      39.19 ±101%  sched_debug.cfs_rq:/.max_vruntime.avg
   2462709           -99.9%       3407 ±102%  sched_debug.cfs_rq:/.max_vruntime.max
    607348 ± 11%     -99.9%     348.57 ±100%  sched_debug.cfs_rq:/.max_vruntime.stddev
   2439879           +43.2%    3493834 ±  4%  sched_debug.cfs_rq:/.min_vruntime.avg
   2485561           +49.1%    3705888        sched_debug.cfs_rq:/.min_vruntime.max
   2129935           +34.5%    2865147 ±  2%  sched_debug.cfs_rq:/.min_vruntime.min
     35480 ± 17%    +324.2%     150497 ± 59%  sched_debug.cfs_rq:/.min_vruntime.stddev
      0.43 ±  3%     +27.9%       0.55        sched_debug.cfs_rq:/.nr_running.avg
      0.35 ±  5%     -57.2%       0.15 ±  4%  sched_debug.cfs_rq:/.nr_running.stddev
      2186 ± 15%     -27.9%       1575 ± 11%  sched_debug.cfs_rq:/.runnable_avg.max
    152.08 ±  6%    +134.5%     356.58 ± 31%  sched_debug.cfs_rq:/.runnable_avg.min
    399.32 ±  4%     -50.5%     197.69 ±  8%  sched_debug.cfs_rq:/.runnable_avg.stddev
     25106 ± 50%   +1121.1%     306577 ± 66%  sched_debug.cfs_rq:/.spread0.max
     35510 ± 17%    +323.3%     150305 ± 59%  sched_debug.cfs_rq:/.spread0.stddev
    545.95 ±  3%     +16.4%     635.59        sched_debug.cfs_rq:/.util_avg.avg
      1726 ± 15%     -26.7%       1266 ± 14%  sched_debug.cfs_rq:/.util_avg.max
    154.67 ±  2%    +112.9%     329.33 ± 30%  sched_debug.cfs_rq:/.util_avg.min
    317.35 ±  4%     -43.1%     180.53 ± 10%  sched_debug.cfs_rq:/.util_avg.stddev
    192.70 ±  6%    +104.5%     393.98 ±  7%  sched_debug.cfs_rq:/.util_est_enqueued.avg
      5359 ±  4%     -26.1%       3958 ±  8%  sched_debug.cpu.avg_idle.min
      4.69 ±  7%    +136.0%      11.07 ±  5%  sched_debug.cpu.clock.stddev
      2380 ±  4%     +31.0%       3117        sched_debug.cpu.curr->pid.avg
      1818 ±  3%     -65.9%     620.26 ±  8%  sched_debug.cpu.curr->pid.stddev
      0.00 ±  8%     +59.7%       0.00 ± 10%  sched_debug.cpu.next_balance.stddev
      2.58 ± 17%     -41.9%       1.50        sched_debug.cpu.nr_running.max
      0.52 ±  9%     -43.2%       0.29 ±  5%  sched_debug.cpu.nr_running.stddev
   1610935           +31.3%    2115112 ±  2%  sched_debug.cpu.nr_switches.avg
   1661619           +34.5%    2234069        sched_debug.cpu.nr_switches.max
   1415677 ±  3%     +20.3%    1702445        sched_debug.cpu.nr_switches.min
     30576 ± 26%    +151.6%      76923 ± 37%  sched_debug.cpu.nr_switches.stddev
     25.47           -91.3%       2.21 ± 69%  perf-stat.i.MPKI
 3.342e+10           +84.7%  6.172e+10 ±  5%  perf-stat.i.branch-instructions
      0.58            -0.3        0.33 ±  5%  perf-stat.i.branch-miss-rate%
 1.667e+08           -13.2%  1.448e+08 ±  2%  perf-stat.i.branch-misses
      0.63 ± 17%      +4.8        5.42 ± 39%  perf-stat.i.cache-miss-rate%
  18939524 ±  4%     -46.6%   10109353 ± 18%  perf-stat.i.cache-misses
 4.422e+09           -87.1%  5.724e+08 ± 77%  perf-stat.i.cache-references
   6897069           +30.8%    9023752 ±  2%  perf-stat.i.context-switches
      2.04           -43.3%       1.16 ±  5%  perf-stat.i.cpi
 3.523e+11            +3.8%  3.656e+11        perf-stat.i.cpu-cycles
   2322589           -86.6%     310934 ± 93%  perf-stat.i.cpu-migrations
     18560 ±  4%    +113.2%      39578 ± 15%  perf-stat.i.cycles-between-cache-misses
      0.20            -0.2        0.02 ± 70%  perf-stat.i.dTLB-load-miss-rate%
  85472762           -87.2%   10962661 ± 82%  perf-stat.i.dTLB-load-misses
 4.266e+10           +83.8%  7.841e+10 ±  5%  perf-stat.i.dTLB-loads
      0.10 ±  4%      -0.1        0.01 ± 72%  perf-stat.i.dTLB-store-miss-rate%
  25396322 ±  4%     -86.5%    3437369 ± 90%  perf-stat.i.dTLB-store-misses
 2.483e+10           +85.2%  4.598e+10 ±  5%  perf-stat.i.dTLB-stores
 1.699e+11           +85.8%  3.157e+11 ±  5%  perf-stat.i.instructions
      0.50           +73.4%       0.87 ±  4%  perf-stat.i.ipc
      2.75            +3.8%       2.86        perf-stat.i.metric.GHz
    822.90           +77.2%       1458 ±  5%  perf-stat.i.metric.M/sec
      5691            -3.4%       5500        perf-stat.i.minor-faults
     91.09            +4.6       95.71        perf-stat.i.node-load-miss-rate%
    334087 ± 17%     -67.4%     109033 ± 18%  perf-stat.i.node-loads
     70.09           +17.6       87.68 ±  6%  perf-stat.i.node-store-miss-rate%
   1559730 ±  5%     -64.9%     548115 ± 56%  perf-stat.i.node-stores
      5704            -3.3%       5513        perf-stat.i.page-faults
     26.03           -92.7%       1.89 ± 83%  perf-stat.overall.MPKI
      0.50            -0.3        0.24 ±  8%  perf-stat.overall.branch-miss-rate%
      0.43 ±  3%      +2.5        2.91 ± 60%  perf-stat.overall.cache-miss-rate%
      2.08           -44.0%       1.16 ±  5%  perf-stat.overall.cpi
     18664 ±  4%    +100.4%      37402 ± 16%  perf-stat.overall.cycles-between-cache-misses
      0.20            -0.2        0.01 ± 87%  perf-stat.overall.dTLB-load-miss-rate%
      0.10 ±  4%      -0.1        0.01 ± 96%  perf-stat.overall.dTLB-store-miss-rate%
      0.48           +79.1%       0.86 ±  4%  perf-stat.overall.ipc
     91.02            +5.1       96.07        perf-stat.overall.node-load-miss-rate%
     70.91           +17.6       88.54 ±  6%  perf-stat.overall.node-store-miss-rate%
 3.289e+10           +85.0%  6.085e+10 ±  5%  perf-stat.ps.branch-instructions
 1.641e+08           -13.1%  1.425e+08 ±  2%  perf-stat.ps.branch-misses
  18633656 ±  4%     -46.7%    9931368 ± 18%  perf-stat.ps.cache-misses
 4.354e+09           -87.1%  5.613e+08 ± 77%  perf-stat.ps.cache-references
   6788892           +31.0%    8894592 ±  2%  perf-stat.ps.context-switches
  3.47e+11            +3.9%  3.604e+11        perf-stat.ps.cpu-cycles
   2286778           -86.7%     304327 ± 94%  perf-stat.ps.cpu-migrations
  84173329           -87.2%   10770448 ± 82%  perf-stat.ps.dTLB-load-misses
 4.198e+10           +84.1%   7.73e+10 ±  5%  perf-stat.ps.dTLB-loads
  25001705 ±  4%     -86.5%    3364501 ± 91%  perf-stat.ps.dTLB-store-misses
 2.444e+10           +85.5%  4.533e+10 ±  5%  perf-stat.ps.dTLB-stores
 1.673e+11           +86.1%  3.112e+11 ±  5%  perf-stat.ps.instructions
     12.40            -1.5%      12.22        perf-stat.ps.major-faults
      5543            -3.9%       5329        perf-stat.ps.minor-faults
    332272 ± 17%     -66.0%     112911 ± 16%  perf-stat.ps.node-loads
   1533930 ±  5%     -65.2%     534337 ± 57%  perf-stat.ps.node-stores
      5556            -3.9%       5341        perf-stat.ps.page-faults
 1.065e+13           +86.7%  1.988e+13 ±  5%  perf-stat.total.instructions
     18.10           -16.2        1.91 ±142%  perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
     17.94           -16.1        1.88 ±142%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
     17.93           -16.1        1.88 ±142%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     17.90           -16.0        1.88 ±142%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     15.83            -8.0        7.86 ± 20%  perf-profile.calltrace.cycles-pp.read
     13.06            -8.0        5.11 ± 30%  perf-profile.calltrace.cycles-pp.pipe_read.new_sync_read.vfs_read.ksys_read.do_syscall_64
     13.21            -7.9        5.30 ± 29%  perf-profile.calltrace.cycles-pp.new_sync_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
     13.58            -7.9        5.68 ± 27%  perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
     14.72            -7.9        6.86 ± 22%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.read
     14.51            -7.8        6.73 ± 22%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
     13.77            -7.5        6.23 ± 23%  perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
      9.90            -7.1        2.83 ± 47%  perf-profile.calltrace.cycles-pp.__schedule.schedule.pipe_read.new_sync_read.vfs_read
      9.96            -7.0        2.92 ± 45%  perf-profile.calltrace.cycles-pp.schedule.pipe_read.new_sync_read.vfs_read.ksys_read
      7.84            -6.9        0.94 ±142%  perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      7.44            -6.8        0.63 ±142%  perf-profile.calltrace.cycles-pp.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      7.07            -6.2        0.85 ±142%  perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
      7.03            -6.2        0.84 ±142%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
      8.60            -5.3        3.30 ± 44%  perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write
      8.75            -5.3        3.49 ± 41%  perf-profile.calltrace.cycles-pp.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write.ksys_write
     10.84            -5.3        5.58 ± 27%  perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     10.53            -5.2        5.29 ± 28%  perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
      6.70            -5.2        1.49 ± 62%  perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.pipe_read.new_sync_read
     10.02            -5.2        4.83 ± 30%  perf-profile.calltrace.cycles-pp.pipe_write.new_sync_write.vfs_write.ksys_write.do_syscall_64
      5.64            -5.2        0.48 ±142%  perf-profile.calltrace.cycles-pp.sched_ttwu_pending.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary
     10.08            -5.1        4.98 ± 29%  perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      8.21            -5.1        3.14 ± 44%  perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write
      8.25            -5.1        3.20 ± 44%  perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write.new_sync_write
      5.11            -4.7        0.40 ±141%  perf-profile.calltrace.cycles-pp.ttwu_do_activate.sched_ttwu_pending.flush_smp_call_function_queue.do_idle.cpu_startup_entry
      5.07            -4.7        0.40 ±141%  perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.sched_ttwu_pending.flush_smp_call_function_queue.do_idle
      5.19            -4.6        0.55 ±141%  perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.pipe_read
     20.28            -4.6       15.69 ±  5%  perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyout.copy_page_to_iter.process_vm_rw_single_vec.process_vm_rw_core
     20.55            -4.3       16.22 ±  5%  perf-profile.calltrace.cycles-pp.copyout.copy_page_to_iter.process_vm_rw_single_vec.process_vm_rw_core.process_vm_rw
     10.91            -3.7        7.17 ± 17%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     24.78            -3.7       21.12 ±  2%  perf-profile.calltrace.cycles-pp.copy_page_to_iter.process_vm_rw_single_vec.process_vm_rw_core.process_vm_rw.__x64_sys_process_vm_readv
     10.95            -3.6        7.30 ± 16%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
     11.15            -3.0        8.18 ± 13%  perf-profile.calltrace.cycles-pp.write
      1.03 ±  4%      -0.4        0.62 ± 14%  perf-profile.calltrace.cycles-pp.stress_vm_child
      0.76 ±  4%      +0.5        1.22 ± 10%  perf-profile.calltrace.cycles-pp.stress_vm_rw
      0.63            +0.5        1.15 ± 23%  perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common
      0.53 ±  2%      +0.5        1.06 ±  9%  perf-profile.calltrace.cycles-pp.__might_fault.copy_page_to_iter.process_vm_rw_single_vec.process_vm_rw_core.process_vm_rw
      0.64            +0.5        1.18 ± 22%  perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      0.00            +0.6        0.60 ±  7%  perf-profile.calltrace.cycles-pp.__might_resched.copy_page_to_iter.process_vm_rw_single_vec.process_vm_rw_core.process_vm_rw
      0.00            +1.0        0.96 ± 15%  perf-profile.calltrace.cycles-pp.__schedule.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode
      0.00            +1.0        1.02 ± 17%  perf-profile.calltrace.cycles-pp.mod_node_page_state.gup_put_folio.unpin_user_pages.process_vm_rw_single_vec.process_vm_rw_core
      0.00            +1.0        1.02 ± 10%  perf-profile.calltrace.cycles-pp.__might_fault.copy_page_from_iter.process_vm_rw_single_vec.process_vm_rw_core.process_vm_rw
      0.00            +1.0        1.03 ± 15%  perf-profile.calltrace.cycles-pp.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
      0.00            +1.0        1.04 ± 17%  perf-profile.calltrace.cycles-pp.mod_node_page_state.gup_put_folio.unpin_user_pages_dirty_lock.process_vm_rw_single_vec.process_vm_rw_core
      0.00            +1.1        1.10 ± 16%  perf-profile.calltrace.cycles-pp.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.24 ±  2%      +1.4        2.60 ± 11%  perf-profile.calltrace.cycles-pp._raw_spin_lock.follow_page_pte.__get_user_pages.__get_user_pages_remote.process_vm_rw_single_vec
      0.00            +1.4        1.40 ± 16%  perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
      0.00            +1.5        1.46 ± 16%  perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
      1.04            +1.5        2.54 ± 14%  perf-profile.calltrace.cycles-pp.gup_put_folio.unpin_user_pages.process_vm_rw_single_vec.process_vm_rw_core.process_vm_rw
      0.99 ±  2%      +1.6        2.57 ± 14%  perf-profile.calltrace.cycles-pp.gup_put_folio.unpin_user_pages_dirty_lock.process_vm_rw_single_vec.process_vm_rw_core.process_vm_rw
      0.00            +1.7        1.69 ± 11%  perf-profile.calltrace.cycles-pp.follow_pud_mask.__get_user_pages.__get_user_pages_remote.process_vm_rw_single_vec.process_vm_rw_core
      1.35            +1.8        3.20 ± 14%  perf-profile.calltrace.cycles-pp.unpin_user_pages.process_vm_rw_single_vec.process_vm_rw_core.process_vm_rw.__x64_sys_process_vm_readv
      0.00            +2.0        1.97 ± 10%  perf-profile.calltrace.cycles-pp.follow_page_mask.__get_user_pages.__get_user_pages_remote.process_vm_rw_single_vec.process_vm_rw_core
      1.27 ±  2%      +2.0        3.30 ± 14%  perf-profile.calltrace.cycles-pp.unpin_user_pages_dirty_lock.process_vm_rw_single_vec.process_vm_rw_core.process_vm_rw.__x64_sys_process_vm_writev
      0.00            +2.1        2.12 ± 18%  perf-profile.calltrace.cycles-pp.mod_node_page_state.try_grab_page.follow_page_pte.__get_user_pages.__get_user_pages_remote
      0.00            +2.3        2.30 ± 11%  perf-profile.calltrace.cycles-pp.follow_pmd_mask.__get_user_pages.__get_user_pages_remote.process_vm_rw_single_vec.process_vm_rw_core
      3.16 ±  2%      +2.4        5.51 ± 11%  perf-profile.calltrace.cycles-pp.try_grab_page.follow_page_pte.__get_user_pages.__get_user_pages_remote.process_vm_rw_single_vec
     32.57            +5.2       37.78 ±  3%  perf-profile.calltrace.cycles-pp.process_vm_rw_single_vec.process_vm_rw_core.process_vm_rw.__x64_sys_process_vm_readv.do_syscall_64
     33.67            +5.6       39.24 ±  3%  perf-profile.calltrace.cycles-pp.process_vm_rw_core.process_vm_rw.__x64_sys_process_vm_readv.do_syscall_64.entry_SYSCALL_64_after_hwframe
      6.24            +6.1       12.34 ± 10%  perf-profile.calltrace.cycles-pp.__get_user_pages_remote.process_vm_rw_single_vec.process_vm_rw_core.process_vm_rw.__x64_sys_process_vm_readv
     34.23            +6.2       40.39 ±  3%  perf-profile.calltrace.cycles-pp.process_vm_rw.__x64_sys_process_vm_readv.do_syscall_64.entry_SYSCALL_64_after_hwframe.process_vm_readv
     34.22            +6.2       40.42 ±  3%  perf-profile.calltrace.cycles-pp.__x64_sys_process_vm_readv.do_syscall_64.entry_SYSCALL_64_after_hwframe.process_vm_readv
     34.39            +6.3       40.68 ±  3%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.process_vm_readv
     34.49            +6.4       40.88 ±  3%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.process_vm_readv
     34.87            +6.6       41.43 ±  3%  perf-profile.calltrace.cycles-pp.process_vm_readv
      6.26            +6.6       12.83 ± 11%  perf-profile.calltrace.cycles-pp.follow_page_pte.__get_user_pages.__get_user_pages_remote.process_vm_rw_single_vec.process_vm_rw_core
      7.43            +6.6       14.02 ±  8%  perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyin.copy_page_from_iter.process_vm_rw_single_vec.process_vm_rw_core
      7.70            +6.9       14.64 ±  8%  perf-profile.calltrace.cycles-pp.copyin.copy_page_from_iter.process_vm_rw_single_vec.process_vm_rw_core.process_vm_rw
      4.80 ±  2%      +7.2       11.95 ± 12%  perf-profile.calltrace.cycles-pp.__get_user_pages_remote.process_vm_rw_single_vec.process_vm_rw_core.process_vm_rw.__x64_sys_process_vm_writev
      9.52            +9.3       18.86 ±  8%  perf-profile.calltrace.cycles-pp.copy_page_from_iter.process_vm_rw_single_vec.process_vm_rw_core.process_vm_rw.__x64_sys_process_vm_writev
     10.80           +12.9       23.71 ± 11%  perf-profile.calltrace.cycles-pp.__get_user_pages.__get_user_pages_remote.process_vm_rw_single_vec.process_vm_rw_core.process_vm_rw
     16.02 ±  2%     +19.0       35.05 ± 10%  perf-profile.calltrace.cycles-pp.process_vm_rw_single_vec.process_vm_rw_core.process_vm_rw.__x64_sys_process_vm_writev.do_syscall_64
     16.70 ±  2%     +19.7       36.44 ± 10%  perf-profile.calltrace.cycles-pp.process_vm_rw_core.process_vm_rw.__x64_sys_process_vm_writev.do_syscall_64.entry_SYSCALL_64_after_hwframe
     17.15 ±  2%     +20.4       37.51 ± 10%  perf-profile.calltrace.cycles-pp.process_vm_rw.__x64_sys_process_vm_writev.do_syscall_64.entry_SYSCALL_64_after_hwframe.process_vm_writev
     17.17 ±  2%     +20.4       37.55 ± 10%  perf-profile.calltrace.cycles-pp.__x64_sys_process_vm_writev.do_syscall_64.entry_SYSCALL_64_after_hwframe.process_vm_writev
     17.26 ±  2%     +20.5       37.72 ± 10%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.process_vm_writev
     17.31 ±  2%     +20.5       37.83 ± 10%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.process_vm_writev
     17.60 ±  2%     +20.8       38.42 ± 10%  perf-profile.calltrace.cycles-pp.process_vm_writev
     18.10           -16.2        1.92 ±141%  perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     18.10           -16.2        1.92 ±141%  perf-profile.children.cycles-pp.cpu_startup_entry
     18.08           -16.2        1.92 ±141%  perf-profile.children.cycles-pp.do_idle
     17.94           -16.0        1.89 ±141%  perf-profile.children.cycles-pp.start_secondary
     16.00            -8.0        8.02 ± 19%  perf-profile.children.cycles-pp.read
     12.02            -7.9        4.10 ± 37%  perf-profile.children.cycles-pp.__schedule
     13.10            -7.9        5.18 ± 30%  perf-profile.children.cycles-pp.pipe_read
     13.22            -7.9        5.31 ± 29%  perf-profile.children.cycles-pp.new_sync_read
     13.60            -7.9        5.70 ± 27%  perf-profile.children.cycles-pp.vfs_read
     13.78            -7.5        6.25 ± 23%  perf-profile.children.cycles-pp.ksys_read
      7.62            -7.0        0.65 ±142%  perf-profile.children.cycles-pp.flush_smp_call_function_queue
      7.92            -7.0        0.96 ±142%  perf-profile.children.cycles-pp.cpuidle_idle_call
      7.18            -6.6        0.59 ±142%  perf-profile.children.cycles-pp.sched_ttwu_pending
      7.14            -6.3        0.86 ±142%  perf-profile.children.cycles-pp.cpuidle_enter
      7.24            -6.3        0.98 ±124%  perf-profile.children.cycles-pp.update_cfs_group
      7.12            -6.3        0.86 ±142%  perf-profile.children.cycles-pp.cpuidle_enter_state
      9.97            -6.0        3.96 ± 29%  perf-profile.children.cycles-pp.schedule
      6.52            -5.7        0.79 ±142%  perf-profile.children.cycles-pp.mwait_idle_with_hints
      7.11            -5.4        1.69 ± 57%  perf-profile.children.cycles-pp.ttwu_do_activate
      7.07            -5.4        1.66 ± 58%  perf-profile.children.cycles-pp.enqueue_task_fair
      8.61            -5.3        3.30 ± 44%  perf-profile.children.cycles-pp.__wake_up_common
     10.86            -5.2        5.61 ± 26%  perf-profile.children.cycles-pp.ksys_write
      8.76            -5.2        3.51 ± 41%  perf-profile.children.cycles-pp.__wake_up_common_lock
     10.55            -5.2        5.32 ± 28%  perf-profile.children.cycles-pp.vfs_write
      6.72            -5.2        1.50 ± 61%  perf-profile.children.cycles-pp.dequeue_task_fair
     10.04            -5.2        4.88 ± 30%  perf-profile.children.cycles-pp.pipe_write
     10.10            -5.1        5.00 ± 29%  perf-profile.children.cycles-pp.new_sync_write
      8.26            -5.1        3.20 ± 44%  perf-profile.children.cycles-pp.autoremove_wake_function
      8.22            -5.0        3.17 ± 44%  perf-profile.children.cycles-pp.try_to_wake_up
      5.62            -4.7        0.94 ± 62%  perf-profile.children.cycles-pp.enqueue_entity
     21.11            -4.4       16.69 ±  5%  perf-profile.children.cycles-pp.copyout
      5.21            -4.4        0.84 ± 70%  perf-profile.children.cycles-pp.dequeue_entity
      5.50            -4.3        1.25 ± 48%  perf-profile.children.cycles-pp.update_load_avg
     25.20            -3.0       22.21 ±  2%  perf-profile.children.cycles-pp.copy_page_to_iter
     11.21            -2.9        8.33 ± 13%  perf-profile.children.cycles-pp.write
      3.29            -2.5        0.83 ± 78%  perf-profile.children.cycles-pp.select_task_rq
      3.22            -2.4        0.78 ± 83%  perf-profile.children.cycles-pp.select_task_rq_fair
      2.78            -2.2        0.62 ± 93%  perf-profile.children.cycles-pp.select_idle_sibling
      1.80 ±  2%      -1.5        0.29 ±137%  perf-profile.children.cycles-pp.available_idle_cpu
      0.87            -0.6        0.22 ± 57%  perf-profile.children.cycles-pp.finish_task_switch
      0.98            -0.5        0.45 ± 23%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.94            -0.5        0.41 ± 27%  perf-profile.children.cycles-pp.prepare_to_wait_event
      0.60 ±  2%      -0.5        0.13 ± 76%  perf-profile.children.cycles-pp.switch_mm_irqs_off
      1.04 ±  4%      -0.4        0.63 ± 14%  perf-profile.children.cycles-pp.stress_vm_child
      0.58            -0.3        0.29 ± 28%  perf-profile.children.cycles-pp.update_rq_clock
      0.69            -0.3        0.40 ± 19%  perf-profile.children.cycles-pp.prepare_task_switch
      0.89 ±  3%      -0.3        0.62 ± 13%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
      0.58            -0.3        0.31 ± 21%  perf-profile.children.cycles-pp.__switch_to_asm
      0.52            -0.2        0.28 ± 21%  perf-profile.children.cycles-pp.___perf_sw_event
      0.76 ±  4%      -0.2        0.56 ± 12%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
      0.28 ±  3%      -0.2        0.09 ± 52%  perf-profile.children.cycles-pp._find_next_bit
      0.50            -0.2        0.33 ± 13%  perf-profile.children.cycles-pp.security_file_permission
      0.26 ±  2%      -0.2        0.09 ± 41%  perf-profile.children.cycles-pp.task_tick_fair
      0.24 ±  6%      -0.2        0.08 ± 57%  perf-profile.children.cycles-pp.__irq_exit_rcu
      0.48            -0.2        0.32 ±  5%  perf-profile.children.cycles-pp.set_next_entity
      0.42 ±  4%      -0.1        0.27 ± 13%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.30 ±  3%      -0.1        0.16 ± 18%  perf-profile.children.cycles-pp.scheduler_tick
      0.37 ±  4%      -0.1        0.24 ± 13%  perf-profile.children.cycles-pp.tick_sched_timer
      0.35 ±  4%      -0.1        0.22 ± 14%  perf-profile.children.cycles-pp.tick_sched_handle
      0.34 ±  4%      -0.1        0.22 ± 15%  perf-profile.children.cycles-pp.update_process_times
      0.25 ±  3%      -0.1        0.12 ± 40%  perf-profile.children.cycles-pp.find_vma
      0.19 ±  7%      -0.1        0.07 ± 56%  perf-profile.children.cycles-pp.__softirqentry_text_start
      0.68            -0.1        0.56 ±  5%  perf-profile.children.cycles-pp.mutex_lock
      0.26 ±  3%      -0.1        0.14 ± 34%  perf-profile.children.cycles-pp.find_extend_vma
      0.38 ±  2%      -0.1        0.26 ± 10%  perf-profile.children.cycles-pp.apparmor_file_permission
      0.37            -0.1        0.27 ±  9%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.34            -0.1        0.25 ± 10%  perf-profile.children.cycles-pp.perf_trace_sched_wakeup_template
      0.24 ±  3%      -0.1        0.16 ±  9%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.17 ±  4%      -0.1        0.09 ± 33%  perf-profile.children.cycles-pp.vmacache_find
      0.64            -0.1        0.56 ±  6%  perf-profile.children.cycles-pp.switch_fpu_return
      0.53 ±  4%      -0.1        0.46 ±  8%  perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
      0.53 ±  4%      -0.1        0.46 ±  8%  perf-profile.children.cycles-pp.hrtimer_interrupt
      0.19 ±  2%      -0.1        0.13 ±  9%  perf-profile.children.cycles-pp.native_sched_clock
      0.57            -0.1        0.50 ±  5%  perf-profile.children.cycles-pp.__switch_to
      0.27            -0.0        0.22 ±  2%  perf-profile.children.cycles-pp.mutex_unlock
      0.10 ±  5%      -0.0        0.06 ± 17%  perf-profile.children.cycles-pp.anon_pipe_buf_release
      0.13 ±  2%      -0.0        0.10 ±  9%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.17 ±  2%      -0.0        0.14 ±  8%  perf-profile.children.cycles-pp.perf_tp_event
      0.08 ±  4%      -0.0        0.05 ±  8%  perf-profile.children.cycles-pp.__list_add_valid
      0.10            -0.0        0.09        perf-profile.children.cycles-pp.finish_wait
      0.14            +0.0        0.16 ±  3%  perf-profile.children.cycles-pp.atime_needs_update
      0.12 ±  4%      +0.0        0.15 ±  2%  perf-profile.children.cycles-pp.file_update_time
      0.06 ±  6%      +0.0        0.09 ±  7%  perf-profile.children.cycles-pp.__rdgsbase_inactive
      0.10 ±  6%      +0.0        0.13 ±  5%  perf-profile.children.cycles-pp.__wrgsbase_inactive
      0.75            +0.0        0.79        perf-profile.children.cycles-pp.pick_next_task_fair
      0.05            +0.0        0.09 ±  5%  perf-profile.children.cycles-pp.pick_next_entity
      0.19 ±  2%      +0.0        0.23 ±  5%  perf-profile.children.cycles-pp.down_read_killable
      0.02 ±141%      +0.0        0.06 ± 11%  perf-profile.children.cycles-pp.perf_trace_sched_switch
      0.05            +0.0        0.10 ±  6%  perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
      0.02 ± 99%      +0.1        0.08 ±  6%  perf-profile.children.cycles-pp.kmalloc_slab
      0.14 ±  2%      +0.1        0.20        perf-profile.children.cycles-pp.down_read
      0.00            +0.1        0.06 ±  8%  perf-profile.children.cycles-pp.resched_curr
      0.02 ± 99%      +0.1        0.08 ±  8%  perf-profile.children.cycles-pp.perf_trace_sched_stat_runtime
      0.16 ±  3%      +0.1        0.21 ±  2%  perf-profile.children.cycles-pp.mmput
      0.00            +0.1        0.06 ±  9%  perf-profile.children.cycles-pp.ktime_get_coarse_real_ts64
      0.13 ±  2%      +0.1        0.19 ±  6%  perf-profile.children.cycles-pp.get_task_mm
      0.00            +0.1        0.07 ±  7%  perf-profile.children.cycles-pp.idr_find
      0.24 ±  2%      +0.1        0.31 ±  4%  perf-profile.children.cycles-pp.__update_load_avg_se
      0.02 ±141%      +0.1        0.09 ± 10%  perf-profile.children.cycles-pp.__calc_delta
      0.66            +0.1        0.73 ±  2%  perf-profile.children.cycles-pp.update_curr
      0.04 ± 44%      +0.1        0.12 ±  9%  perf-profile.children.cycles-pp.memcg_slab_free_hook
      0.00            +0.1        0.07 ± 18%  perf-profile.children.cycles-pp.cpumask_next_and
      0.00            +0.1        0.08 ± 10%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode_prepare
      0.08 ±  4%      +0.1        0.16 ± 10%  perf-profile.children.cycles-pp.up_read
      0.07 ± 11%      +0.1        0.15 ±  8%  perf-profile.children.cycles-pp.clockevents_program_event
      0.13            +0.1        0.22 ±  3%  perf-profile.children.cycles-pp.ttwu_do_wakeup
      0.06 ±  6%      +0.1        0.15 ±  9%  perf-profile.children.cycles-pp.current_time
      0.00            +0.1        0.09 ± 10%  perf-profile.children.cycles-pp.check_stack_object
      0.11 ±  4%      +0.1        0.20 ±  4%  perf-profile.children.cycles-pp.check_preempt_curr
      0.15 ±  4%      +0.1        0.26 ±  6%  perf-profile.children.cycles-pp.os_xsave
      0.12 ±  4%      +0.1        0.23 ±  9%  perf-profile.children.cycles-pp.syscall_enter_from_user_mode
      0.16 ±  3%      +0.1        0.27 ±  9%  perf-profile.children.cycles-pp.reweight_entity
      0.61 ±  2%      +0.1        0.75 ±  2%  perf-profile.children.cycles-pp.find_get_task_by_vpid
      0.19 ±  5%      +0.1        0.33 ±  6%  perf-profile.children.cycles-pp.__radix_tree_lookup
      0.10 ±  3%      +0.2        0.25 ± 10%  perf-profile.children.cycles-pp.__check_object_size
      0.13 ±  3%      +0.2        0.29 ±  9%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.11 ±  6%      +0.2        0.26 ± 11%  perf-profile.children.cycles-pp.follow_huge_addr
      0.37 ±  2%      +0.2        0.53 ±  6%  perf-profile.children.cycles-pp.mm_access
      0.00            +0.2        0.17 ±  9%  perf-profile.children.cycles-pp.check_preempt_wakeup
      0.00            +0.2        0.17 ± 14%  perf-profile.children.cycles-pp.put_prev_entity
      0.15 ±  3%      +0.2        0.39 ± 12%  perf-profile.children.cycles-pp.pud_huge
      0.14 ±  3%      +0.2        0.39 ± 11%  perf-profile.children.cycles-pp.mark_page_accessed
      0.19 ±  3%      +0.3        0.44 ±  9%  perf-profile.children.cycles-pp.kfree
      0.19 ±  3%      +0.3        0.50 ± 12%  perf-profile.children.cycles-pp.pmd_huge
      0.31 ±  3%      +0.3        0.65 ±  9%  perf-profile.children.cycles-pp.__kmalloc
      0.28 ±  3%      +0.4        0.67 ± 10%  perf-profile.children.cycles-pp.entry_SYSRETQ_unsafe_stack
      0.26 ±  3%      +0.4        0.70 ± 12%  perf-profile.children.cycles-pp.vm_normal_page
      0.28 ±  2%      +0.5        0.73 ± 11%  perf-profile.children.cycles-pp.folio_mark_accessed
      0.40 ±  2%      +0.5        0.86 ± 10%  perf-profile.children.cycles-pp.__entry_text_start
      0.77 ±  4%      +0.5        1.24 ± 10%  perf-profile.children.cycles-pp.stress_vm_rw
      2.54            +0.6        3.10 ±  4%  perf-profile.children.cycles-pp._raw_spin_lock
      0.44 ±  3%      +0.6        1.02 ± 10%  perf-profile.children.cycles-pp.__import_iovec
      0.46 ±  3%      +0.6        1.05 ± 10%  perf-profile.children.cycles-pp.import_iovec
      0.57 ±  2%      +0.7        1.26 ±  9%  perf-profile.children.cycles-pp._copy_from_user
      0.47 ±  3%      +0.7        1.18 ± 11%  perf-profile.children.cycles-pp.rcu_all_qs
      1.72            +0.9        2.58 ±  6%  perf-profile.children.cycles-pp.__cond_resched
      0.75 ±  2%      +1.0        1.71 ±  9%  perf-profile.children.cycles-pp.iovec_from_user
      0.00            +1.1        1.11 ± 16%  perf-profile.children.cycles-pp.exit_to_user_mode_loop
      0.98 ±  2%      +1.1        2.10 ± 10%  perf-profile.children.cycles-pp.__might_sleep
      0.70            +1.1        1.84 ±  9%  perf-profile.children.cycles-pp.exit_to_user_mode_prepare
      0.76 ±  3%      +1.2        1.95 ± 11%  perf-profile.children.cycles-pp.follow_pud_mask
      0.79            +1.3        2.06 ±  9%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
      0.93 ±  2%      +1.3        2.22 ± 10%  perf-profile.children.cycles-pp.follow_page_mask
      1.30 ±  2%      +1.5        2.80 ±  9%  perf-profile.children.cycles-pp.__might_fault
      1.03 ±  2%      +1.6        2.67 ± 11%  perf-profile.children.cycles-pp.follow_pmd_mask
      2.83            +1.7        4.55 ±  7%  perf-profile.children.cycles-pp.__might_resched
      1.37            +1.9        3.26 ± 14%  perf-profile.children.cycles-pp.unpin_user_pages
      1.30 ±  2%      +2.1        3.38 ± 14%  perf-profile.children.cycles-pp.unpin_user_pages_dirty_lock
      3.29 ±  2%      +2.6        5.87 ± 11%  perf-profile.children.cycles-pp.try_grab_page
     29.19            +2.6       31.78        perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      1.40            +3.0        4.42 ± 17%  perf-profile.children.cycles-pp.mod_node_page_state
      2.13 ±  2%      +3.2        5.35 ± 14%  perf-profile.children.cycles-pp.gup_put_folio
     34.26            +6.2       40.45 ±  3%  perf-profile.children.cycles-pp.__x64_sys_process_vm_readv
     34.96            +6.7       41.64 ±  3%  perf-profile.children.cycles-pp.process_vm_readv
      8.05            +7.0       15.10 ±  8%  perf-profile.children.cycles-pp.copyin
      6.54            +7.1       13.60 ± 11%  perf-profile.children.cycles-pp.follow_page_pte
     10.06            +9.7       19.72 ±  8%  perf-profile.children.cycles-pp.copy_page_from_iter
     10.99           +13.2       24.22 ± 11%  perf-profile.children.cycles-pp.__get_user_pages
     11.04           +13.3       24.31 ± 11%  perf-profile.children.cycles-pp.__get_user_pages_remote
     77.15           +15.3       92.46 ±  2%  perf-profile.children.cycles-pp.do_syscall_64
     77.51           +15.4       92.94 ±  2%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     17.18 ±  2%     +20.4       37.56 ± 10%  perf-profile.children.cycles-pp.__x64_sys_process_vm_writev
     17.69 ±  2%     +20.9       38.64 ± 10%  perf-profile.children.cycles-pp.process_vm_writev
     48.65           +24.3       72.98 ±  6%  perf-profile.children.cycles-pp.process_vm_rw_single_vec
     50.39           +25.4       75.74 ±  6%  perf-profile.children.cycles-pp.process_vm_rw_core
     51.40           +26.5       77.94 ±  6%  perf-profile.children.cycles-pp.process_vm_rw
      7.23            -6.3        0.97 ±125%  perf-profile.self.cycles-pp.update_cfs_group
      6.42            -5.6        0.78 ±142%  perf-profile.self.cycles-pp.mwait_idle_with_hints
      4.82            -4.2        0.67 ± 85%  perf-profile.self.cycles-pp.update_load_avg
      1.78 ±  2%      -1.5        0.29 ±137%  perf-profile.self.cycles-pp.available_idle_cpu
      0.96            -0.5        0.42 ± 26%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.98 ±  2%      -0.5        0.45 ± 28%  perf-profile.self.cycles-pp.__schedule
      0.59 ±  2%      -0.5        0.13 ± 78%  perf-profile.self.cycles-pp.switch_mm_irqs_off
      0.97            -0.4        0.61 ± 15%  perf-profile.self.cycles-pp.stress_vm_child
      0.45 ±  2%      -0.3        0.14 ± 58%  perf-profile.self.cycles-pp.update_rq_clock
      0.58            -0.3        0.30 ± 22%  perf-profile.self.cycles-pp.__switch_to_asm
      0.58            -0.3        0.33 ± 17%  perf-profile.self.cycles-pp.pipe_read
      0.35 ±  3%      -0.2        0.10 ± 59%  perf-profile.self.cycles-pp.__wake_up_common
      0.47            -0.2        0.24 ± 23%  perf-profile.self.cycles-pp.___perf_sw_event
      0.32 ±  2%      -0.2        0.13 ± 39%  perf-profile.self.cycles-pp.finish_task_switch
      0.38 ±  2%      -0.2        0.21 ± 20%  perf-profile.self.cycles-pp.prepare_to_wait_event
      0.25 ±  2%      -0.2        0.08 ± 53%  perf-profile.self.cycles-pp._find_next_bit
      0.31 ±  3%      -0.2        0.14 ± 27%  perf-profile.self.cycles-pp.enqueue_entity
      0.34 ± 15%      -0.2        0.18 ± 12%  perf-profile.self.cycles-pp.read
      0.26 ±  2%      -0.2        0.11 ± 20%  perf-profile.self.cycles-pp.try_to_wake_up
      0.29 ±  3%      -0.2        0.13 ± 30%  perf-profile.self.cycles-pp.prepare_task_switch
      0.45 ±  2%      -0.1        0.31 ±  9%  perf-profile.self.cycles-pp.mutex_lock
      0.26            -0.1        0.16 ± 14%  perf-profile.self.cycles-pp.apparmor_file_permission
      0.13 ±  2%      -0.1        0.03 ±103%  perf-profile.self.cycles-pp.perf_trace_sched_wakeup_template
      0.35            -0.1        0.26 ± 11%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.16 ±  3%      -0.1        0.08 ± 22%  perf-profile.self.cycles-pp.dequeue_entity
      0.56            -0.1        0.48 ±  5%  perf-profile.self.cycles-pp.__switch_to
      0.15 ±  2%      -0.1        0.08 ± 37%  perf-profile.self.cycles-pp.vmacache_find
      0.18 ±  2%      -0.1        0.11 ± 25%  perf-profile.self.cycles-pp.select_idle_sibling
      0.19 ±  3%      -0.1        0.13 ± 10%  perf-profile.self.cycles-pp.native_sched_clock
      0.13 ±  3%      -0.1        0.07 ± 17%  perf-profile.self.cycles-pp.security_file_permission
      0.19            -0.1        0.13 ± 18%  perf-profile.self.cycles-pp.enqueue_task_fair
      0.35 ±  2%      -0.1        0.30 ±  8%  perf-profile.self.cycles-pp.update_curr
      0.15 ±  2%      -0.1        0.10 ± 12%  perf-profile.self.cycles-pp.dequeue_task_fair
      0.26            -0.0        0.21 ±  2%  perf-profile.self.cycles-pp.mutex_unlock
      0.38            -0.0        0.34 ±  3%  perf-profile.self.cycles-pp.find_get_task_by_vpid
      0.09 ±  4%      -0.0        0.06 ± 13%  perf-profile.self.cycles-pp.anon_pipe_buf_release
      0.11 ±  4%      -0.0        0.08 ± 10%  perf-profile.self.cycles-pp.atime_needs_update
      0.21 ±  2%      -0.0        0.19 ±  3%  perf-profile.self.cycles-pp.vfs_read
      0.17 ±  2%      -0.0        0.14 ±  3%  perf-profile.self.cycles-pp.switch_fpu_return
      0.11 ±  4%      -0.0        0.09 ± 10%  perf-profile.self.cycles-pp.aa_file_perm
      0.08 ±  6%      -0.0        0.06 ± 13%  perf-profile.self.cycles-pp.select_task_rq
      0.05            +0.0        0.06        perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.06            +0.0        0.07        perf-profile.self.cycles-pp.set_next_entity
      0.07 ±  5%      +0.0        0.09        perf-profile.self.cycles-pp.get_task_mm
      0.06 ±  9%      +0.0        0.08        perf-profile.self.cycles-pp.__get_user_pages_remote
      0.06 ±  6%      +0.0        0.09 ±  5%  perf-profile.self.cycles-pp.__rdgsbase_inactive
      0.10 ±  5%      +0.0        0.13 ±  4%  perf-profile.self.cycles-pp.__wrgsbase_inactive
      0.11            +0.0        0.15 ±  2%  perf-profile.self.cycles-pp.pick_next_task_fair
      0.09 ±  9%      +0.0        0.13 ±  8%  perf-profile.self.cycles-pp.ktime_get
      0.03 ± 70%      +0.0        0.08 ±  6%  perf-profile.self.cycles-pp.pick_next_entity
      0.08 ±  6%      +0.1        0.13 ±  7%  perf-profile.self.cycles-pp.vfs_write
      0.00            +0.1        0.06 ±  9%  perf-profile.self.cycles-pp.resched_curr
      0.01 ±223%      +0.1        0.06 ± 11%  perf-profile.self.cycles-pp.perf_trace_sched_switch
      0.00            +0.1        0.06 ±  8%  perf-profile.self.cycles-pp.put_prev_entity
      0.00            +0.1        0.06 ±  8%  perf-profile.self.cycles-pp.syscall_exit_to_user_mode_prepare
      0.00            +0.1        0.06 ±  8%  perf-profile.self.cycles-pp.idr_find
      0.22 ±  2%      +0.1        0.28 ±  4%  perf-profile.self.cycles-pp.__update_load_avg_se
      0.00            +0.1        0.06 ± 11%  perf-profile.self.cycles-pp.ksys_write
      0.00            +0.1        0.06 ± 14%  perf-profile.self.cycles-pp.__wake_up_common_lock
      0.00            +0.1        0.06 ±  7%  perf-profile.self.cycles-pp.kmalloc_slab
      0.10 ±  6%      +0.1        0.16 ±  6%  perf-profile.self.cycles-pp.write
      0.00            +0.1        0.07 ± 15%  perf-profile.self.cycles-pp.check_preempt_wakeup
      0.08 ±  6%      +0.1        0.15 ±  9%  perf-profile.self.cycles-pp.up_read
      0.01 ±223%      +0.1        0.08 ± 12%  perf-profile.self.cycles-pp.perf_trace_sched_stat_runtime
      0.01 ±223%      +0.1        0.08 ± 10%  perf-profile.self.cycles-pp.__calc_delta
      0.00            +0.1        0.08 ± 10%  perf-profile.self.cycles-pp.check_stack_object
      0.00            +0.1        0.08 ± 20%  perf-profile.self.cycles-pp.exit_to_user_mode_loop
      0.02 ±141%      +0.1        0.09 ± 10%  perf-profile.self.cycles-pp.new_sync_write
      0.02 ±141%      +0.1        0.09 ± 10%  perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack
      0.06 ±  8%      +0.1        0.14 ± 11%  perf-profile.self.cycles-pp.follow_huge_addr
      0.10 ±  4%      +0.1        0.19 ± 10%  perf-profile.self.cycles-pp.syscall_enter_from_user_mode
      0.00            +0.1        0.09 ± 12%  perf-profile.self.cycles-pp.current_time
      0.06            +0.1        0.15 ± 11%  perf-profile.self.cycles-pp.__import_iovec
      0.05 ±  8%      +0.1        0.14 ± 13%  perf-profile.self.cycles-pp.__check_object_size
      0.03 ± 70%      +0.1        0.14 ± 15%  perf-profile.self.cycles-pp.syscall_exit_to_user_mode
      0.19 ±  3%      +0.1        0.30 ±  5%  perf-profile.self.cycles-pp.process_vm_readv
      0.15 ±  3%      +0.1        0.25 ±  5%  perf-profile.self.cycles-pp.os_xsave
      0.00            +0.1        0.11 ±  9%  perf-profile.self.cycles-pp.memcg_slab_free_hook
      0.07 ±  5%      +0.1        0.18 ± 10%  perf-profile.self.cycles-pp._copy_from_user
      0.15 ±  2%      +0.1        0.26 ±  6%  perf-profile.self.cycles-pp.pipe_write
      0.05            +0.1        0.16 ± 12%  perf-profile.self.cycles-pp.exit_to_user_mode_prepare
      0.12 ±  3%      +0.1        0.24 ±  8%  perf-profile.self.cycles-pp.__entry_text_start
      0.00            +0.1        0.12 ± 14%  perf-profile.self.cycles-pp.schedule
      0.14 ± 12%      +0.1        0.26 ±  5%  perf-profile.self.cycles-pp.process_vm_rw
      0.38            +0.1        0.51 ±  6%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.19 ±  7%      +0.1        0.32 ±  6%  perf-profile.self.cycles-pp.__radix_tree_lookup
      0.09 ±  4%      +0.1        0.23 ± 10%  perf-profile.self.cycles-pp.iovec_from_user
      0.09            +0.1        0.24 ± 11%  perf-profile.self.cycles-pp.mark_page_accessed
      0.14 ±  3%      +0.2        0.29 ±  9%  perf-profile.self.cycles-pp.process_vm_rw_core
      0.13 ±  2%      +0.2        0.28 ±  9%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.13 ±  2%      +0.2        0.29 ± 10%  perf-profile.self.cycles-pp.process_vm_writev
      0.10 ±  3%      +0.2        0.26 ± 13%  perf-profile.self.cycles-pp.pud_huge
      0.09 ±  5%      +0.2        0.26 ± 13%  perf-profile.self.cycles-pp.pmd_huge
      0.14 ±  3%      +0.2        0.31 ± 11%  perf-profile.self.cycles-pp.copyout
      0.14 ±  4%      +0.2        0.31 ± 10%  perf-profile.self.cycles-pp.kfree
      0.17 ±  3%      +0.2        0.35 ±  9%  perf-profile.self.cycles-pp.do_syscall_64
      0.18 ±  3%      +0.2        0.36 ±  9%  perf-profile.self.cycles-pp.__kmalloc
      0.18 ±  2%      +0.3        0.45 ± 10%  perf-profile.self.cycles-pp.copyin
      0.24 ±  3%      +0.3        0.50 ±  9%  perf-profile.self.cycles-pp.__might_fault
      0.11 ±  5%      +0.3        0.41 ± 22%  perf-profile.self.cycles-pp.ksys_read
      0.30 ±  3%      +0.3        0.65 ± 12%  perf-profile.self.cycles-pp.unpin_user_pages
      0.22 ±  3%      +0.4        0.58 ± 12%  perf-profile.self.cycles-pp.vm_normal_page
      0.26 ±  4%      +0.4        0.62 ± 11%  perf-profile.self.cycles-pp.rcu_all_qs
      0.28 ±  2%      +0.4        0.65 ± 10%  perf-profile.self.cycles-pp.entry_SYSRETQ_unsafe_stack
      0.23 ±  3%      +0.4        0.61 ± 12%  perf-profile.self.cycles-pp.folio_mark_accessed
      0.28 ±  4%      +0.5        0.73 ± 13%  perf-profile.self.cycles-pp.unpin_user_pages_dirty_lock
      0.67 ±  4%      +0.5        1.15 ± 12%  perf-profile.self.cycles-pp.stress_vm_rw
      0.56 ±  2%      +0.5        1.06 ±  8%  perf-profile.self.cycles-pp.process_vm_rw_single_vec
      0.91            +0.6        1.55 ±  8%  perf-profile.self.cycles-pp.__cond_resched
      0.81 ±  2%      +0.9        1.72 ± 10%  perf-profile.self.cycles-pp.__might_sleep
      0.61 ±  3%      +0.9        1.55 ± 11%  perf-profile.self.cycles-pp.follow_pud_mask
      1.94            +0.9        2.89 ±  7%  perf-profile.self.cycles-pp._raw_spin_lock
      1.60            +1.0        2.57 ±  7%  perf-profile.self.cycles-pp.copy_page_to_iter
      2.56 ±  2%      +1.0        3.61 ±  7%  perf-profile.self.cycles-pp.try_grab_page
      0.88 ±  2%      +1.1        1.98 ± 10%  perf-profile.self.cycles-pp.copy_page_from_iter
      0.82 ±  3%      +1.1        1.95 ± 11%  perf-profile.self.cycles-pp.follow_page_mask
      2.60            +1.4        3.98 ±  6%  perf-profile.self.cycles-pp.__might_resched
      0.88 ±  2%      +1.4        2.27 ± 11%  perf-profile.self.cycles-pp.follow_pmd_mask
      0.88 ±  2%      +1.4        2.28 ± 11%  perf-profile.self.cycles-pp.__get_user_pages
      1.43 ±  2%      +1.7        3.15 ± 12%  perf-profile.self.cycles-pp.gup_put_folio
      1.50 ±  3%      +2.3        3.79 ± 11%  perf-profile.self.cycles-pp.follow_page_pte
     28.89            +2.5       31.42        perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      1.26            +2.8        4.04 ± 18%  perf-profile.self.cycles-pp.mod_node_page_state





Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


-- 
0-DAY CI Kernel Test Service
https://01.org/lkp



View attachment "config-5.19.0-rc2-00017-g32fe13cd7aa1" of type "text/plain" (163951 bytes)

View attachment "job-script" of type "text/plain" (8089 bytes)

View attachment "job.yaml" of type "text/plain" (5202 bytes)

View attachment "reproduce" of type "text/plain" (296 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ