lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Date:   Tue, 11 Apr 2023 09:44:28 +0800
From:   kernel test robot <oliver.sang@...el.com>
To:     Aaron Thompson <dev@...ont.org>
CC:     <oe-lkp@...ts.linux.dev>, <lkp@...el.com>,
        <linux-kernel@...r.kernel.org>, <ying.huang@...el.com>,
        <feng.tang@...el.com>, <fengwei.yin@...el.com>,
        "Ingo Molnar" <mingo@...hat.com>,
        Peter Zijlstra <peterz@...radead.org>,
        Juri Lelli <juri.lelli@...hat.com>,
        Vincent Guittot <vincent.guittot@...aro.org>,
        Dietmar Eggemann <dietmar.eggemann@....com>,
        Steven Rostedt <rostedt@...dmis.org>,
        Ben Segall <bsegall@...gle.com>, Mel Gorman <mgorman@...e.de>,
        Daniel Bristot de Oliveira <bristot@...hat.com>,
        "Valentin Schneider" <vschneid@...hat.com>,
        Aaron Thompson <dev@...ont.org>
Subject: Re: [PATCH] sched/clock: Fix local_clock() before sched_clock_init()


Hello,

kernel test robot noticed a -7.2% regression of fio.write_iops on:


commit: 776f07f465f7ad182016f9653adebaf78b51bd5d ("[PATCH] sched/clock: Fix local_clock() before sched_clock_init()")
url: https://github.com/intel-lab-lkp/linux/commits/Aaron-Thompson/sched-clock-Fix-local_clock-before-sched_clock_init/20230406-121911
base: https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git 05bfb338fa8dd40b008ce443e397fc374f6bd107
patch link: https://lore.kernel.org/all/20230406040847.569970-1-dev@aaront.org/
patch subject: [PATCH] sched/clock: Fix local_clock() before sched_clock_init()

testcase: fio-basic
test machine: 96 threads 2 sockets (Ice Lake) with 256G memory
parameters:

	runtime: 300s
	disk: 1HDD
	fs: btrfs
	nr_task: 1
	test_size: 128G
	rw: randwrite
	bs: 4k
	ioengine: filestat
	cpufreq_governor: performance

test-description: Fio is a tool that will spawn a number of threads or processes doing a particular type of I/O action as specified by the user.
test-url: https://github.com/axboe/fio

In addition to that, the commit also has significant impact on the following tests:

+------------------+------------------------------------------------------------+
| testcase: change | fio-basic: fio.write_iops -8.4% regression                 |
| test machine     | 96 threads 2 sockets (Ice Lake) with 256G memory           |
| test parameters  | bs=4k                                                      |
|                  | cpufreq_governor=performance                               |
|                  | disk=1HDD                                                  |
|                  | fs=btrfs                                                   |
|                  | ioengine=filecreate                                        |
|                  | nr_task=1                                                  |
|                  | runtime=300s                                               |
|                  | rw=write                                                   |
|                  | test_size=128G                                             |
+------------------+------------------------------------------------------------+
| testcase: change | fio-basic: fio.write_iops -9.5% regression                 |
| test machine     | 96 threads 2 sockets (Ice Lake) with 256G memory           |
| test parameters  | bs=4k                                                      |
|                  | cpufreq_governor=performance                               |
|                  | disk=1HDD                                                  |
|                  | donorname=e4defrag.def                                     |
|                  | filesize=64G                                               |
|                  | fs=ext4                                                    |
|                  | ioengine=e4defrag                                          |
|                  | nr_task=1                                                  |
|                  | runtime=300s                                               |
|                  | rw=write                                                   |
|                  | test_size=128G                                             |
+------------------+------------------------------------------------------------+


If you fix the issue, kindly add following tag
| Reported-by: kernel test robot <oliver.sang@...el.com>
| Link: https://lore.kernel.org/oe-lkp/202304110934.8d6758ce-oliver.sang@intel.com


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        sudo bin/lkp install job.yaml           # job file is attached in this email
        bin/lkp split-job --compatible job.yaml # generate the yaml file for lkp run
        sudo bin/lkp run generated-yaml-file

        # if come across any failure that blocks the test,
        # please remove ~/.lkp and /lkp dir to run from a clean state.

=========================================================================================
bs/compiler/cpufreq_governor/disk/fs/ioengine/kconfig/nr_task/rootfs/runtime/rw/tbox_group/test_size/testcase:
  4k/gcc-11/performance/1HDD/btrfs/filestat/x86_64-rhel-8.3/1/debian-11.1-x86_64-20220510.cgz/300s/randwrite/lkp-icl-2sp1/128G/fio-basic

commit: 
  05bfb338fa ("vhost: Fix livepatch timeouts in vhost_worker()")
  776f07f465 ("sched/clock: Fix local_clock() before sched_clock_init()")

05bfb338fa8dd40b 776f07f465f7ad182016f9653ad 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
     19544            -7.2%      18128        fio.write_bw_MBps
   5003373            -7.2%    4640819        fio.write_iops
    823470 ±  3%     +89.5%    1560637 ± 13%  cpuidle..usage
      2.19 ±  2%     -17.0%       1.82 ± 13%  iostat.cpu.system
      4276 ±  3%      +7.0%       4575        vmstat.system.cs
     84945           +81.7%     154305 ±  8%  vmstat.system.in
      9583 ± 11%   +9786.0%     947386 ± 59%  turbostat.C1E
      0.40 ±  9%     +38.1       38.51 ± 92%  turbostat.C1E%
     94.71           -41.8       52.88 ± 61%  turbostat.C6%
      4.52 ±  5%    +977.5%      48.73 ± 64%  turbostat.CPU%c1
     90.43           -48.6%      46.48 ± 66%  turbostat.CPU%c6
    826477 ±  2%     +96.3%    1622496 ±  8%  turbostat.IRQ
      5.22 ± 25%     -70.2%       1.56 ± 93%  turbostat.Pkg%pc6
    205.55           +10.4%     226.91 ±  4%  turbostat.PkgWatt
      0.08 ±  3%      -0.0        0.05 ± 59%  perf-stat.i.dTLB-load-miss-rate%
   2625849 ±  3%     -37.4%    1644947 ± 59%  perf-stat.i.dTLB-load-misses
    275477 ±  3%     -28.7%     196339 ± 39%  perf-stat.i.dTLB-store-misses
 1.653e+09            -4.0%  1.588e+09 ±  2%  perf-stat.i.dTLB-stores
      0.08 ±  4%      -0.0        0.05 ± 58%  perf-stat.overall.dTLB-load-miss-rate%
     33.00 ± 20%     +10.4       43.41 ±  8%  perf-stat.overall.node-store-miss-rate%
   2285893 ±  3%     -37.0%    1439577 ± 59%  perf-stat.ps.dTLB-load-misses
    239679 ±  4%     -28.3%     171750 ± 39%  perf-stat.ps.dTLB-store-misses
 1.439e+09            -3.5%  1.389e+09 ±  2%  perf-stat.ps.dTLB-stores
     16.02 ± 24%     -11.4        4.66 ±115%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     16.02 ± 24%     -11.4        4.66 ±115%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
     15.90 ± 24%     -11.2        4.66 ±115%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     16.17 ± 24%     -10.9        5.22 ±108%  perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
     12.67 ± 37%      -9.0        3.71 ±105%  perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     12.36 ± 37%      -8.8        3.54 ±106%  perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
     12.42 ± 39%      -8.3        4.10 ± 99%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
      5.87 ± 39%      -4.2        1.70 ±124%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
     16.02 ± 24%     -11.4        4.66 ±115%  perf-profile.children.cycles-pp.start_secondary
     16.17 ± 24%     -10.9        5.22 ±108%  perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     16.17 ± 24%     -10.9        5.22 ±108%  perf-profile.children.cycles-pp.cpu_startup_entry
     16.05 ± 24%     -10.8        5.22 ±108%  perf-profile.children.cycles-pp.do_idle
     12.97 ± 36%      -8.7        4.28 ± 97%  perf-profile.children.cycles-pp.cpuidle_idle_call
     12.51 ± 38%      -8.4        4.10 ± 99%  perf-profile.children.cycles-pp.cpuidle_enter
     12.51 ± 38%      -8.4        4.10 ± 99%  perf-profile.children.cycles-pp.cpuidle_enter_state
      6.50 ± 37%      -5.3        1.19 ±177%  perf-profile.children.cycles-pp.sched_setaffinity
      5.87 ± 39%      -4.2        1.70 ±124%  perf-profile.children.cycles-pp.intel_idle
      5.87 ± 39%      -4.2        1.70 ±124%  perf-profile.self.cycles-pp.intel_idle


***************************************************************************************************
lkp-icl-2sp1: 96 threads 2 sockets (Ice Lake) with 256G memory
=========================================================================================
bs/compiler/cpufreq_governor/disk/fs/ioengine/kconfig/nr_task/rootfs/runtime/rw/tbox_group/test_size/testcase:
  4k/gcc-11/performance/1HDD/btrfs/filecreate/x86_64-rhel-8.3/1/debian-11.1-x86_64-20220510.cgz/300s/write/lkp-icl-2sp1/128G/fio-basic

commit: 
  05bfb338fa ("vhost: Fix livepatch timeouts in vhost_worker()")
  776f07f465 ("sched/clock: Fix local_clock() before sched_clock_init()")

05bfb338fa8dd40b 776f07f465f7ad182016f9653ad 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
     25979            -8.4%      23809 ±  2%  fio.write_bw_MBps
   6650855            -8.4%    6095205 ±  2%  fio.write_iops
    691384 ± 10%     +96.5%    1358339 ±  2%  cpuidle..usage
      2.21 ±  7%     -24.2%       1.68 ± 15%  iostat.cpu.system
      1.80 ± 11%      -0.7        1.11 ± 25%  mpstat.cpu.all.irq%
     82803           +92.1%     159083        vmstat.system.in
   2590257           -56.7%    1122631 ± 91%  perf-stat.i.dTLB-load-misses
    612377 ± 19%     -29.4%     432619 ± 24%  perf-stat.i.node-store-misses
     43.00 ±  9%      -7.9       35.09 ± 25%  perf-stat.overall.node-store-miss-rate%
   2176582 ±  2%     -56.9%     937477 ± 91%  perf-stat.ps.dTLB-load-misses
      1521 ±  7%     +41.9%       2158 ± 32%  turbostat.C1
     11064 ± 13%   +9543.4%    1066974 ± 27%  turbostat.C1E
      0.53 ±  9%     +60.6       61.08 ± 48%  turbostat.C1E%
    668232 ± 11%     -59.4%     271350 ± 96%  turbostat.C6
     93.80           -60.8       33.05 ± 90%  turbostat.C6%
      4.70 ±  5%   +1331.4%      67.35 ± 42%  turbostat.CPU%c1
     89.99           -68.9%      27.95 ±100%  turbostat.CPU%c6
    696486 ± 10%     +96.7%    1369738        turbostat.IRQ
     22.98 ± 11%     -75.9%       5.54 ±144%  turbostat.Pkg%pc2
      7.50 ± 55%     -89.4%       0.79 ±147%  turbostat.Pkg%pc6
    201.44 ±  4%     +15.7%     233.16 ±  4%  turbostat.PkgWatt
    161.53            +2.4%     165.33        turbostat.RAMWatt
     13.06 ± 24%     -10.6        2.44 ±223%  perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
     12.88 ± 22%     -10.4        2.44 ±223%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
     12.88 ± 22%     -10.4        2.44 ±223%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     12.88 ± 22%     -10.4        2.44 ±223%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     14.79 ± 30%      -8.7        6.13 ±110%  perf-profile.calltrace.cycles-pp.smp_call_function_single.event_function_call.perf_event_release_kernel.perf_release.__fput
     10.57 ± 22%      -8.3        2.22 ±223%  perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      8.85 ± 26%      -6.8        2.00 ±223%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
      8.84 ± 23%      -6.8        2.00 ±223%  perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
      6.35 ± 37%      -5.2        1.11 ±223%  perf-profile.calltrace.cycles-pp.open64
      5.82 ± 31%      -4.7        1.11 ±223%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.open64
      5.82 ± 31%      -4.7        1.11 ±223%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.open64
      5.82 ± 31%      -4.7        1.11 ±223%  perf-profile.calltrace.cycles-pp.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe.open64
      5.82 ± 31%      -4.7        1.11 ±223%  perf-profile.calltrace.cycles-pp.do_sys_openat2.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe.open64
      4.78 ± 41%      -3.7        1.11 ±223%  perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
      4.18 ± 54%      -3.1        1.11 ±223%  perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
      5.82 ± 31%      -2.5        3.29 ±145%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.execve
      5.82 ± 31%      -2.5        3.29 ±145%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve
      5.82 ± 31%      -2.5        3.29 ±145%  perf-profile.calltrace.cycles-pp.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve
      5.82 ± 31%      -2.5        3.29 ±145%  perf-profile.calltrace.cycles-pp.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve
      5.82 ± 31%      -2.5        3.29 ±145%  perf-profile.calltrace.cycles-pp.execve
      5.07 ± 30%      -1.8        3.29 ±145%  perf-profile.calltrace.cycles-pp.bprm_execve.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.07 ± 30%      -1.8        3.29 ±145%  perf-profile.calltrace.cycles-pp.exec_binprm.bprm_execve.do_execveat_common.__x64_sys_execve.do_syscall_64
      5.07 ± 30%      -1.8        3.29 ±145%  perf-profile.calltrace.cycles-pp.search_binary_handler.exec_binprm.bprm_execve.do_execveat_common.__x64_sys_execve
      5.07 ± 30%      -1.8        3.29 ±145%  perf-profile.calltrace.cycles-pp.load_elf_binary.search_binary_handler.exec_binprm.bprm_execve.do_execveat_common
      2.33 ± 56%      +8.2       10.51 ± 53%  perf-profile.calltrace.cycles-pp.cmd_record.run_builtin.main.__libc_start_main
      2.33 ± 56%      +8.2       10.51 ± 53%  perf-profile.calltrace.cycles-pp.__cmd_record.cmd_record.run_builtin.main.__libc_start_main
     13.06 ± 24%     -10.6        2.44 ±223%  perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     13.06 ± 24%     -10.6        2.44 ±223%  perf-profile.children.cycles-pp.cpu_startup_entry
     13.06 ± 24%     -10.6        2.44 ±223%  perf-profile.children.cycles-pp.do_idle
     12.88 ± 22%     -10.4        2.44 ±223%  perf-profile.children.cycles-pp.start_secondary
     15.62 ± 24%      -9.5        6.13 ±110%  perf-profile.children.cycles-pp.smp_call_function_single
     15.79 ± 25%      -9.4        6.35 ±107%  perf-profile.children.cycles-pp.event_function_call
     10.75 ± 24%      -8.5        2.22 ±223%  perf-profile.children.cycles-pp.cpuidle_idle_call
      9.02 ± 25%      -7.0        2.00 ±223%  perf-profile.children.cycles-pp.cpuidle_enter
      9.02 ± 25%      -7.0        2.00 ±223%  perf-profile.children.cycles-pp.cpuidle_enter_state
      6.35 ± 37%      -5.2        1.11 ±223%  perf-profile.children.cycles-pp.open64
      6.12 ± 36%      -4.6        1.56 ±223%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
      5.68 ± 46%      -4.4        1.33 ±223%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
      5.02 ± 45%      -3.5        1.56 ±223%  perf-profile.children.cycles-pp.sched_setaffinity
      5.82 ± 31%      -2.5        3.29 ±145%  perf-profile.children.cycles-pp.__x64_sys_execve
      5.82 ± 31%      -2.5        3.29 ±145%  perf-profile.children.cycles-pp.do_execveat_common
      5.82 ± 31%      -2.5        3.29 ±145%  perf-profile.children.cycles-pp.execve
      5.07 ± 30%      -1.8        3.29 ±145%  perf-profile.children.cycles-pp.bprm_execve
      5.07 ± 30%      -1.8        3.29 ±145%  perf-profile.children.cycles-pp.exec_binprm
      5.07 ± 30%      -1.8        3.29 ±145%  perf-profile.children.cycles-pp.search_binary_handler
      5.07 ± 30%      -1.8        3.29 ±145%  perf-profile.children.cycles-pp.load_elf_binary
      5.78 ± 31%      -0.4        5.43 ±192%  perf-profile.children.cycles-pp.vm_mmap_pgoff



***************************************************************************************************
lkp-icl-2sp1: 96 threads 2 sockets (Ice Lake) with 256G memory
=========================================================================================
bs/compiler/cpufreq_governor/disk/donorname/filesize/fs/ioengine/kconfig/nr_task/rootfs/runtime/rw/tbox_group/test_size/testcase:
  4k/gcc-11/performance/1HDD/e4defrag.def/64G/ext4/e4defrag/x86_64-rhel-8.3/1/debian-11.1-x86_64-20220510.cgz/300s/write/lkp-icl-2sp1/128G/fio-basic

commit: 
  05bfb338fa ("vhost: Fix livepatch timeouts in vhost_worker()")
  776f07f465 ("sched/clock: Fix local_clock() before sched_clock_init()")

05bfb338fa8dd40b 776f07f465f7ad182016f9653ad 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
     63.78           +17.6       81.39 ±  3%  fio.latency_10us%
     35.79           -17.6       18.19 ± 15%  fio.latency_4us%
    158.42           +10.5%     175.01        fio.time.elapsed_time
    158.42           +10.5%     175.01        fio.time.elapsed_time.max
    145.10           +10.3%     160.10        fio.time.system_time
     16835           +10.5%      18598        fio.time.voluntary_context_switches
    829.16            -9.5%     750.56        fio.write_bw_MBps
      5418           +11.2%       6026        fio.write_clat_90%_us
      5706           +10.3%       6293        fio.write_clat_95%_us
      6720           +12.5%       7562        fio.write_clat_99%_us
      4404           +10.7%       4876        fio.write_clat_mean_us
    212264            -9.5%     192144        fio.write_iops
     54.58 ±  4%      +7.1%      58.44 ±  4%  boot-time.boot
      2927 ±  2%     +43.5%       4199 ±  7%  meminfo.Active(anon)
      2.03            -0.8        1.23 ± 27%  mpstat.cpu.all.irq%
 1.481e+10           +11.4%  1.649e+10 ±  2%  cpuidle..time
  15316664          +119.6%   33632647        cpuidle..usage
     96.74            +0.9%      97.58        iostat.cpu.idle
      3.10           -26.9%       2.26 ± 15%  iostat.cpu.system
    254188 ± 11%     -21.9%     198559 ± 24%  numa-meminfo.node0.AnonPages.max
      1619 ± 17%     +53.0%       2477 ± 23%  numa-meminfo.node1.Active(anon)
    573.83 ± 12%     -12.2%     504.00 ±  3%  vmstat.io.bo
     95789           +99.2%     190775        vmstat.system.in
    404.50 ± 17%     +53.0%     619.00 ± 23%  numa-vmstat.node1.nr_active_anon
     20.33 ± 91%   +2578.7%     544.67 ± 92%  numa-vmstat.node1.nr_dirtied
     19.83 ± 92%   +2644.5%     544.33 ± 92%  numa-vmstat.node1.nr_written
    404.50 ± 17%     +53.0%     619.00 ± 23%  numa-vmstat.node1.nr_zone_active_anon
    731.33 ±  2%     +43.3%       1047 ±  7%  proc-vmstat.nr_active_anon
      2614            +7.7%       2814        proc-vmstat.nr_shmem
    701880            -1.7%     689729        proc-vmstat.nr_unevictable
    731.33 ±  2%     +43.3%       1047 ±  7%  proc-vmstat.nr_zone_active_anon
    701880            -1.7%     689729        proc-vmstat.nr_zone_unevictable
    189.67 ± 22%    +733.2%       1580 ±  7%  proc-vmstat.pgactivate
    504506            +7.9%     544284        proc-vmstat.pgfault
     23544            +5.6%      24871        proc-vmstat.pgreuse
   1221888            +9.8%    1341440        proc-vmstat.unevictable_pgs_scanned
     19613 ±  3%     +32.5%      25983 ±  5%  turbostat.C1
     77926        +39135.6%   30574974 ± 19%  turbostat.C1E
      0.20 ±  2%     +81.3       81.54 ± 36%  turbostat.C1E%
     95.88           -80.3       15.63 ±188%  turbostat.C6%
      3.73 ±  6%   +2148.9%      83.88 ± 33%  turbostat.CPU%c1
     92.15           -86.0%      12.88 ±215%  turbostat.CPU%c6
  15450088          +119.8%   33955567        turbostat.IRQ
     36.53 ±  2%     -86.0%       5.12 ±223%  turbostat.Pkg%pc2
      0.25 ± 17%     -91.4%       0.02 ±223%  turbostat.Pkg%pc6
    205.53           +15.4%     237.23 ±  4%  turbostat.PkgWatt
    158.98            +5.3%     167.43        turbostat.RAMWatt
 1.093e+09            -2.7%  1.063e+09        perf-stat.i.branch-instructions
 2.145e+09            -5.0%  2.038e+09        perf-stat.i.dTLB-loads
      0.03 ±  7%      -0.0        0.01 ±130%  perf-stat.i.dTLB-store-miss-rate%
    230599 ±  6%     -77.6%      51646 ±130%  perf-stat.i.dTLB-store-misses
 9.251e+08            -4.0%  8.883e+08        perf-stat.i.dTLB-stores
 6.636e+09            -3.8%  6.382e+09        perf-stat.i.instructions
     43.41            -4.2%      41.57        perf-stat.i.metric.M/sec
      0.02 ±  7%      -0.0        0.01 ±128%  perf-stat.overall.dTLB-store-miss-rate%
     31245            +6.3%      33223        perf-stat.overall.path-length
 1.086e+09            -2.7%  1.057e+09        perf-stat.ps.branch-instructions
 2.131e+09            -4.9%  2.026e+09        perf-stat.ps.dTLB-loads
    229121 ±  6%     -77.6%      51342 ±130%  perf-stat.ps.dTLB-store-misses
 9.193e+08            -3.9%  8.832e+08        perf-stat.ps.dTLB-stores
 6.594e+09            -3.8%  6.346e+09        perf-stat.ps.instructions
 1.048e+12            +6.3%  1.115e+12        perf-stat.total.instructions
     16.05 ±  4%      -5.5       10.54 ± 23%  perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
      9.09 ±  4%      -5.1        3.94 ± 43%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
     10.76 ±  4%      -4.4        6.36 ± 27%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
     11.20 ±  4%      -4.2        7.00 ± 25%  perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      5.70 ±  2%      -3.2        2.53 ± 45%  perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      5.20 ±  3%      -3.1        2.09 ± 52%  perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
      4.51 ±  2%      -2.6        1.94 ± 45%  perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
      5.35 ±  2%      -1.5        3.89 ±  7%  perf-profile.calltrace.cycles-pp.__filemap_get_folio.move_extent_per_page.ext4_move_extents.__ext4_ioctl.__x64_sys_ioctl
      5.48 ±  6%      -1.1        4.43 ±  9%  perf-profile.calltrace.cycles-pp.ext4_find_extent.ext4_swap_extents.move_extent_per_page.ext4_move_extents.__ext4_ioctl
      2.15 ±  4%      -1.0        1.11 ± 30%  perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
      5.55 ±  7%      -1.0        4.51 ±  9%  perf-profile.calltrace.cycles-pp.__ext4_ext_dirty.ext4_split_extent_at.ext4_swap_extents.move_extent_per_page.ext4_move_extents
      4.99 ±  7%      -0.9        4.09 ± 10%  perf-profile.calltrace.cycles-pp.ext4_extent_block_csum_set.__ext4_ext_dirty.ext4_split_extent_at.ext4_swap_extents.move_extent_per_page
      1.83 ±  7%      -0.9        0.94 ±  9%  perf-profile.calltrace.cycles-pp.folio_alloc.__filemap_get_folio.move_extent_per_page.ext4_move_extents.__ext4_ioctl
      4.88 ±  6%      -0.9        3.99 ±  9%  perf-profile.calltrace.cycles-pp.crc32c_pcl_intel_update.ext4_extent_block_csum_set.__ext4_ext_dirty.ext4_split_extent_at.ext4_swap_extents
      1.68 ±  7%      -0.9        0.81 ±  8%  perf-profile.calltrace.cycles-pp.__alloc_pages.folio_alloc.__filemap_get_folio.move_extent_per_page.ext4_move_extents
      4.92 ±  3%      -0.8        4.09 ±  8%  perf-profile.calltrace.cycles-pp.__ext4_ext_dirty.ext4_swap_extents.move_extent_per_page.ext4_move_extents.__ext4_ioctl
      1.46 ±  9%      -0.8        0.65 ± 11%  perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages.folio_alloc.__filemap_get_folio.move_extent_per_page
      5.14 ±  3%      -0.8        4.35 ±  9%  perf-profile.calltrace.cycles-pp.__ext4_ext_dirty.ext4_ext_insert_extent.ext4_split_extent_at.ext4_swap_extents.move_extent_per_page
      1.22 ±  8%      -0.8        0.45 ±105%  perf-profile.calltrace.cycles-pp.irq_enter_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      1.17 ±  7%      -0.8        0.41 ±105%  perf-profile.calltrace.cycles-pp.tick_irq_enter.irq_enter_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      4.45 ±  3%      -0.8        3.70 ±  9%  perf-profile.calltrace.cycles-pp.crc32c_pcl_intel_update.ext4_extent_block_csum_set.__ext4_ext_dirty.ext4_ext_insert_extent.ext4_split_extent_at
      4.38 ±  3%      -0.7        3.63 ±  8%  perf-profile.calltrace.cycles-pp.ext4_extent_block_csum_set.__ext4_ext_dirty.ext4_swap_extents.move_extent_per_page.ext4_move_extents
      4.21 ±  3%      -0.7        3.50 ±  8%  perf-profile.calltrace.cycles-pp.crc32c_pcl_intel_update.ext4_extent_block_csum_set.__ext4_ext_dirty.ext4_swap_extents.move_extent_per_page
      3.61 ±  4%      -0.7        2.92 ±  7%  perf-profile.calltrace.cycles-pp.mext_check_coverage.move_extent_per_page.ext4_move_extents.__ext4_ioctl.__x64_sys_ioctl
      4.58 ±  4%      -0.7        3.90 ±  9%  perf-profile.calltrace.cycles-pp.ext4_extent_block_csum_set.__ext4_ext_dirty.ext4_ext_insert_extent.ext4_split_extent_at.ext4_swap_extents
      1.41 ±  7%      -0.6        0.79 ± 21%  perf-profile.calltrace.cycles-pp.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      2.88 ±  3%      -0.5        2.35 ±  7%  perf-profile.calltrace.cycles-pp.ext4_find_extent.mext_check_coverage.move_extent_per_page.ext4_move_extents.__ext4_ioctl
      0.68 ±  7%      -0.4        0.29 ±101%  perf-profile.calltrace.cycles-pp.__kmalloc.ext4_find_extent.mext_check_coverage.move_extent_per_page.ext4_move_extents
      1.95 ±  7%      -0.4        1.58 ± 12%  perf-profile.calltrace.cycles-pp.__read_extent_tree_block.ext4_find_extent.ext4_swap_extents.move_extent_per_page.ext4_move_extents
      0.65 ±  7%      -0.4        0.28 ±100%  perf-profile.calltrace.cycles-pp.restore_fpregs_from_fpstate.switch_fpu_return.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
      1.44 ±  4%      -0.4        1.09 ±  8%  perf-profile.calltrace.cycles-pp.__kmalloc.ext4_find_extent.ext4_swap_extents.move_extent_per_page.ext4_move_extents
      1.56 ±  6%      -0.3        1.28 ± 13%  perf-profile.calltrace.cycles-pp.__getblk_gfp.__read_extent_tree_block.ext4_find_extent.ext4_swap_extents.move_extent_per_page
      1.22 ±  4%      -0.3        0.94 ±  8%  perf-profile.calltrace.cycles-pp.__kmem_cache_alloc_node.__kmalloc.ext4_find_extent.ext4_swap_extents.move_extent_per_page
      0.77 ±  4%      -0.3        0.50 ± 45%  perf-profile.calltrace.cycles-pp.switch_fpu_return.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.89 ±  4%      -0.2        0.68 ± 11%  perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.ioctl
      1.28 ±  7%      -0.2        1.07 ± 10%  perf-profile.calltrace.cycles-pp.ext4_find_extent.ext4_move_extents.__ext4_ioctl.__x64_sys_ioctl.do_syscall_64
      0.80 ±  4%      -0.2        0.61 ± 11%  perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.ioctl
      1.04 ±  4%      -0.2        0.88 ±  8%  perf-profile.calltrace.cycles-pp.__read_extent_tree_block.ext4_find_extent.mext_check_coverage.move_extent_per_page.ext4_move_extents
      0.91 ±  6%      -0.2        0.74 ± 12%  perf-profile.calltrace.cycles-pp.down_write.ext4_move_extents.__ext4_ioctl.__x64_sys_ioctl.do_syscall_64
      0.72 ±  7%      -0.2        0.56 ±  9%  perf-profile.calltrace.cycles-pp.folio_add_lru.filemap_add_folio.__filemap_get_folio.move_extent_per_page.ext4_move_extents
      0.89 ±  7%      -0.2        0.74 ±  5%  perf-profile.calltrace.cycles-pp.jbd2__journal_start.move_extent_per_page.ext4_move_extents.__ext4_ioctl.__x64_sys_ioctl
      0.88 ±  4%      -0.1        0.75 ± 14%  perf-profile.calltrace.cycles-pp.__find_get_block.__getblk_gfp.__read_extent_tree_block.ext4_find_extent.ext4_swap_extents
      0.85 ±  5%      -0.1        0.72 ±  9%  perf-profile.calltrace.cycles-pp.__getblk_gfp.__read_extent_tree_block.ext4_find_extent.mext_check_coverage.move_extent_per_page
      1.48 ±  4%      +0.3        1.75 ±  8%  perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry
      0.90 ±  6%      +0.5        1.35 ± 15%  perf-profile.calltrace.cycles-pp.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.cpuidle_idle_call.do_idle
      0.00            +0.7        0.68 ± 11%  perf-profile.calltrace.cycles-pp.lapic_next_deadline.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      0.00            +1.0        1.05 ± 16%  perf-profile.calltrace.cycles-pp.__memmove.ext4_ext_insert_extent.ext4_split_extent_at.ext4_swap_extents.move_extent_per_page
      0.28 ±100%      +1.2        1.52 ± 16%  perf-profile.calltrace.cycles-pp.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
     19.55 ±  4%      +7.2       26.73 ±  6%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
      2.42 ±  6%      +7.8       10.27 ± 18%  perf-profile.calltrace.cycles-pp.ext4_ext_try_to_merge.ext4_swap_extents.move_extent_per_page.ext4_move_extents.__ext4_ioctl
      2.32 ±  6%      +7.9       10.18 ± 18%  perf-profile.calltrace.cycles-pp.ext4_ext_try_to_merge_right.ext4_ext_try_to_merge.ext4_swap_extents.move_extent_per_page.ext4_move_extents
      1.90 ±  7%      +7.9        9.80 ± 19%  perf-profile.calltrace.cycles-pp.__memmove.ext4_ext_try_to_merge_right.ext4_ext_try_to_merge.ext4_swap_extents.move_extent_per_page
     16.40 ±  4%      -5.6       10.78 ± 23%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
      9.29 ±  4%      -5.2        4.06 ± 42%  perf-profile.children.cycles-pp.__hrtimer_run_queues
     10.98 ±  4%      -4.4        6.54 ± 27%  perf-profile.children.cycles-pp.hrtimer_interrupt
     11.41 ±  4%      -4.3        7.15 ± 25%  perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
      5.84 ±  2%      -3.2        2.60 ± 44%  perf-profile.children.cycles-pp.tick_sched_timer
      5.32 ±  3%      -3.2        2.15 ± 51%  perf-profile.children.cycles-pp.tick_sched_handle
     15.68 ±  4%      -2.7       13.00 ±  9%  perf-profile.children.cycles-pp.__ext4_ext_dirty
      4.68 ±  2%      -2.7        2.01 ± 45%  perf-profile.children.cycles-pp.update_process_times
     13.67 ±  4%      -2.4       11.32 ±  9%  perf-profile.children.cycles-pp.crc32c_pcl_intel_update
     13.99 ±  4%      -2.3       11.66 ±  9%  perf-profile.children.cycles-pp.ext4_extent_block_csum_set
      9.73 ±  4%      -1.8        7.93 ±  8%  perf-profile.children.cycles-pp.ext4_find_extent
      5.40 ±  2%      -1.5        3.93 ±  7%  perf-profile.children.cycles-pp.__filemap_get_folio
      2.24 ±  4%      -1.1        1.16 ± 30%  perf-profile.children.cycles-pp.scheduler_tick
      1.84 ±  7%      -0.9        0.95 ±  9%  perf-profile.children.cycles-pp.folio_alloc
      1.72 ±  7%      -0.9        0.85 ±  9%  perf-profile.children.cycles-pp.__alloc_pages
      1.51 ±  8%      -0.8        0.67 ± 10%  perf-profile.children.cycles-pp.get_page_from_freelist
      3.63 ±  3%      -0.7        2.93 ±  7%  perf-profile.children.cycles-pp.mext_check_coverage
      3.51 ±  5%      -0.6        2.86 ± 10%  perf-profile.children.cycles-pp.__read_extent_tree_block
      1.45 ±  7%      -0.6        0.82 ± 19%  perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
      1.02 ±  6%      -0.6        0.40 ± 63%  perf-profile.children.cycles-pp.rcu_sched_clock_irq
      1.20 ±  7%      -0.6        0.62 ± 45%  perf-profile.children.cycles-pp.tick_irq_enter
      1.10 ±  8%      -0.6        0.53 ± 24%  perf-profile.children.cycles-pp.perf_rotate_context
      1.24 ±  7%      -0.6        0.67 ± 42%  perf-profile.children.cycles-pp.irq_enter_rcu
      2.56 ±  4%      -0.6        2.00 ±  8%  perf-profile.children.cycles-pp.__kmalloc
      0.74 ± 12%      -0.5        0.24 ± 67%  perf-profile.children.cycles-pp.vprintk_emit
      0.74 ± 12%      -0.5        0.24 ± 67%  perf-profile.children.cycles-pp.console_unlock
      0.74 ± 12%      -0.5        0.24 ± 67%  perf-profile.children.cycles-pp.console_flush_all
      0.74 ± 12%      -0.5        0.23 ± 67%  perf-profile.children.cycles-pp.serial8250_console_write
      2.90 ±  5%      -0.5        2.39 ± 11%  perf-profile.children.cycles-pp.__getblk_gfp
      0.95 ± 18%      -0.5        0.45 ± 12%  perf-profile.children.cycles-pp.rmqueue
      0.83 ±  4%      -0.5        0.33 ± 58%  perf-profile.children.cycles-pp.rcu_pending
      0.64 ± 14%      -0.5        0.14 ± 94%  perf-profile.children.cycles-pp.timerqueue_del
      0.69 ± 10%      -0.5        0.20 ± 89%  perf-profile.children.cycles-pp.enqueue_hrtimer
      0.72 ± 12%      -0.5        0.23 ± 66%  perf-profile.children.cycles-pp.wait_for_lsr
      0.73 ±  8%      -0.5        0.25 ± 65%  perf-profile.children.cycles-pp.irq_work_run_list
      2.22 ±  4%      -0.5        1.75 ±  7%  perf-profile.children.cycles-pp.__kmem_cache_alloc_node
      0.62 ± 10%      -0.5        0.16 ± 99%  perf-profile.children.cycles-pp.timerqueue_add
      0.56 ± 10%      -0.5        0.11 ±182%  perf-profile.children.cycles-pp.run_posix_cpu_timers
      2.07 ±  6%      -0.4        1.62 ± 15%  perf-profile.children.cycles-pp.__do_softirq
      0.66 ±  8%      -0.4        0.24 ± 67%  perf-profile.children.cycles-pp.asm_sysvec_irq_work
      0.66 ±  8%      -0.4        0.24 ± 67%  perf-profile.children.cycles-pp.sysvec_irq_work
      0.66 ±  8%      -0.4        0.24 ± 67%  perf-profile.children.cycles-pp.__sysvec_irq_work
      0.66 ±  8%      -0.4        0.24 ± 67%  perf-profile.children.cycles-pp.irq_work_run
      0.66 ±  8%      -0.4        0.24 ± 67%  perf-profile.children.cycles-pp.irq_work_single
      0.66 ±  8%      -0.4        0.24 ± 67%  perf-profile.children.cycles-pp._printk
      0.65 ± 23%      -0.4        0.24 ± 19%  perf-profile.children.cycles-pp.rmqueue_bulk
      0.53 ±  4%      -0.4        0.15 ± 95%  perf-profile.children.cycles-pp.check_cpu_stall
      0.52 ± 12%      -0.4        0.15 ± 66%  perf-profile.children.cycles-pp.io_serial_in
      0.66 ±  6%      -0.3        0.33 ± 34%  perf-profile.children.cycles-pp.update_rq_clock
      0.60 ±  8%      -0.3        0.27 ± 30%  perf-profile.children.cycles-pp.ret_from_fork
      0.59 ±  8%      -0.3        0.27 ± 30%  perf-profile.children.cycles-pp.kthread
      1.28 ±  9%      -0.3        0.97 ±  6%  perf-profile.children.cycles-pp.__cond_resched
      1.59 ±  7%      -0.3        1.29 ±  7%  perf-profile.children.cycles-pp.down_write
      0.36 ± 12%      -0.3        0.06 ±152%  perf-profile.children.cycles-pp.tick_check_oneshot_broadcast_this_cpu
      0.53 ±  8%      -0.3        0.24 ± 39%  perf-profile.children.cycles-pp.run_rebalance_domains
      0.36 ± 29%      -0.3        0.07 ±115%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.45 ±  8%      -0.3        0.17 ± 46%  perf-profile.children.cycles-pp.process_one_work
      0.43 ±  7%      -0.3        0.15 ± 51%  perf-profile.children.cycles-pp.drm_fb_helper_damage_work
      0.43 ±  7%      -0.3        0.15 ± 51%  perf-profile.children.cycles-pp.drm_fbdev_fb_dirty
      0.50 ±  8%      -0.3        0.22 ± 40%  perf-profile.children.cycles-pp.update_blocked_averages
      1.11 ±  8%      -0.3        0.84 ±  9%  perf-profile.children.cycles-pp.__ext4_handle_dirty_metadata
      0.76 ± 10%      -0.3        0.49 ± 22%  perf-profile.children.cycles-pp.load_balance
      0.38 ±  8%      -0.3        0.11 ± 77%  perf-profile.children.cycles-pp.commit_tail
      0.38 ±  8%      -0.3        0.11 ± 77%  perf-profile.children.cycles-pp.ast_mode_config_helper_atomic_commit_tail
      0.46 ±  7%      -0.3        0.18 ± 42%  perf-profile.children.cycles-pp.worker_thread
      0.37 ±  8%      -0.3        0.10 ± 84%  perf-profile.children.cycles-pp.memcpy_toio
      0.37 ±  8%      -0.3        0.10 ± 84%  perf-profile.children.cycles-pp.drm_atomic_helper_commit_tail_rpm
      0.37 ±  8%      -0.3        0.10 ± 84%  perf-profile.children.cycles-pp.drm_atomic_helper_commit_planes
      0.37 ±  8%      -0.3        0.10 ± 84%  perf-profile.children.cycles-pp.ast_primary_plane_helper_atomic_update
      0.37 ±  8%      -0.3        0.10 ± 84%  perf-profile.children.cycles-pp.drm_fb_memcpy
      0.39 ±  8%      -0.3        0.13 ± 59%  perf-profile.children.cycles-pp.drm_atomic_helper_dirtyfb
      0.39 ±  8%      -0.3        0.13 ± 59%  perf-profile.children.cycles-pp.drm_atomic_commit
      0.39 ±  8%      -0.3        0.13 ± 59%  perf-profile.children.cycles-pp.drm_atomic_helper_commit
      1.76 ±  5%      -0.3        1.50 ±  8%  perf-profile.children.cycles-pp.__might_resched
      1.07 ±  9%      -0.2        0.82 ± 13%  perf-profile.children.cycles-pp.crc_1
      0.51 ±  7%      -0.2        0.26 ± 33%  perf-profile.children.cycles-pp.ct_idle_exit
      0.42 ±  8%      -0.2        0.18 ± 49%  perf-profile.children.cycles-pp.update_irq_load_avg
      0.29 ± 15%      -0.2        0.06 ±112%  perf-profile.children.cycles-pp.rb_next
      0.36 ±  8%      -0.2        0.13 ± 53%  perf-profile.children.cycles-pp.irq_work_tick
      0.93 ±  4%      -0.2        0.70 ± 11%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
      1.10 ±  4%      -0.2        0.88 ±  3%  perf-profile.children.cycles-pp.__might_sleep
      1.60 ±  5%      -0.2        1.38 ± 11%  perf-profile.children.cycles-pp.__find_get_block
      0.32 ±  7%      -0.2        0.11 ± 74%  perf-profile.children.cycles-pp.irqentry_enter
      0.83 ±  4%      -0.2        0.62 ± 10%  perf-profile.children.cycles-pp.exit_to_user_mode_prepare
      0.25 ± 12%      -0.2        0.05 ±141%  perf-profile.children.cycles-pp.rb_insert_color
      0.72 ±  6%      -0.2        0.52 ± 15%  perf-profile.children.cycles-pp.__brelse
      0.78 ±  4%      -0.2        0.59 ± 11%  perf-profile.children.cycles-pp.switch_fpu_return
      0.52 ± 10%      -0.2        0.35 ± 22%  perf-profile.children.cycles-pp.find_busiest_group
      0.49 ± 12%      -0.2        0.33 ± 22%  perf-profile.children.cycles-pp.update_sd_lb_stats
      0.29 ± 13%      -0.2        0.13 ± 18%  perf-profile.children.cycles-pp.node_dirty_ok
      0.66 ±  7%      -0.2        0.50 ± 10%  perf-profile.children.cycles-pp.restore_fpregs_from_fpstate
      0.90 ±  7%      -0.2        0.75 ±  6%  perf-profile.children.cycles-pp.jbd2__journal_start
      0.77 ±  6%      -0.1        0.63 ± 11%  perf-profile.children.cycles-pp.memset_erms
      0.72 ±  7%      -0.1        0.57 ±  9%  perf-profile.children.cycles-pp.folio_add_lru
      0.21 ± 21%      -0.1        0.07 ± 60%  perf-profile.children.cycles-pp.calc_global_load_tick
      0.85 ±  7%      -0.1        0.71 ±  8%  perf-profile.children.cycles-pp.kernel_fpu_begin_mask
      0.60 ±  5%      -0.1        0.47 ±  8%  perf-profile.children.cycles-pp.folio_batch_move_lru
      0.16 ± 13%      -0.1        0.04 ±118%  perf-profile.children.cycles-pp.__update_blocked_fair
      0.32 ±  9%      -0.1        0.20 ± 27%  perf-profile.children.cycles-pp.ct_kernel_enter
      0.59 ±  5%      -0.1        0.47 ±  9%  perf-profile.children.cycles-pp.arch_scale_freq_tick
      0.44 ± 10%      -0.1        0.33 ± 11%  perf-profile.children.cycles-pp.ext4_free_ext_path
      0.50 ± 11%      -0.1        0.39 ±  7%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      0.13 ± 19%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.do_group_exit
      0.13 ± 19%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.do_exit
      0.40 ±  5%      -0.1        0.30 ±  4%  perf-profile.children.cycles-pp.lru_add_fn
      0.68 ±  6%      -0.1        0.58 ±  7%  perf-profile.children.cycles-pp.up_write
      0.63 ±  7%      -0.1        0.53 ±  8%  perf-profile.children.cycles-pp.__ext4_journal_stop
      0.46 ±  9%      -0.1        0.37 ± 11%  perf-profile.children.cycles-pp.xas_load
      0.12 ± 19%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.__x64_sys_exit_group
      0.14 ± 23%      -0.1        0.05 ± 82%  perf-profile.children.cycles-pp.irq_work_needs_cpu
      0.46 ± 11%      -0.1        0.37 ± 11%  perf-profile.children.cycles-pp.rcu_all_qs
      0.38 ±  8%      -0.1        0.28 ± 17%  perf-profile.children.cycles-pp.fio_e4defrag_queue
      0.17 ± 30%      -0.1        0.08 ± 20%  perf-profile.children.cycles-pp.__schedule
      0.25 ± 11%      -0.1        0.16 ± 25%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.11 ± 22%      -0.1        0.02 ± 99%  perf-profile.children.cycles-pp.__mmput
      0.11 ± 22%      -0.1        0.02 ± 99%  perf-profile.children.cycles-pp.exit_mmap
      0.41 ±  6%      -0.1        0.33 ± 15%  perf-profile.children.cycles-pp._copy_from_user
      0.34 ± 12%      -0.1        0.26 ±  9%  perf-profile.children.cycles-pp._copy_to_user
      0.30 ±  9%      -0.1        0.23 ± 20%  perf-profile.children.cycles-pp.kmem_cache_alloc
      0.13 ±  9%      -0.1        0.06 ± 73%  perf-profile.children.cycles-pp.trigger_load_balance
      0.28 ± 10%      -0.1        0.21 ± 15%  perf-profile.children.cycles-pp.td_io_queue
      0.12 ± 23%      -0.1        0.05 ± 46%  perf-profile.children.cycles-pp.node_page_state
      0.09 ± 28%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.schedule
      0.33 ±  9%      -0.1        0.27 ± 13%  perf-profile.children.cycles-pp.os_xsave
      0.25 ±  8%      -0.1        0.20 ± 11%  perf-profile.children.cycles-pp.thread_main
      0.09 ± 12%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.put_io_u
      0.31 ±  5%      -0.1        0.26 ± 12%  perf-profile.children.cycles-pp.ext4_discard_preallocations
      0.10 ± 21%      -0.1        0.04 ± 74%  perf-profile.children.cycles-pp.crc_93
      0.08 ± 17%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.crc_97
      0.09 ± 22%      -0.0        0.04 ± 45%  perf-profile.children.cycles-pp.__x64_sys_openat
      0.09 ± 22%      -0.0        0.04 ± 45%  perf-profile.children.cycles-pp.do_sys_openat2
      0.16 ± 20%      -0.0        0.12 ± 17%  perf-profile.children.cycles-pp.bh_uptodate_or_lock
      0.20 ± 11%      -0.0        0.15 ± 19%  perf-profile.children.cycles-pp.ext4_can_extents_be_merged
      0.14 ± 18%      -0.0        0.09 ± 19%  perf-profile.children.cycles-pp.kmalloc_slab
      0.09 ± 19%      -0.0        0.05 ± 48%  perf-profile.children.cycles-pp.crc_53
      0.16 ± 13%      -0.0        0.12 ± 13%  perf-profile.children.cycles-pp.crc_19
      0.14 ± 19%      -0.0        0.10 ± 15%  perf-profile.children.cycles-pp.ksys_read
      0.13 ± 11%      -0.0        0.09 ± 23%  perf-profile.children.cycles-pp.__es_remove_extent
      0.08 ± 19%      -0.0        0.04 ± 72%  perf-profile.children.cycles-pp.crc_86
      0.14 ± 19%      -0.0        0.10 ± 17%  perf-profile.children.cycles-pp.vfs_read
      0.16 ±  6%      -0.0        0.13 ± 18%  perf-profile.children.cycles-pp.crc_11
      0.10 ± 10%      -0.0        0.07 ± 16%  perf-profile.children.cycles-pp.__mod_memcg_lruvec_state
      0.09 ± 21%      -0.0        0.06 ± 47%  perf-profile.children.cycles-pp.crc_80
      0.14 ±  9%      -0.0        0.11 ± 12%  perf-profile.children.cycles-pp.crc_42
      0.16 ±  8%      -0.0        0.13 ± 11%  perf-profile.children.cycles-pp.crc_13
      0.12 ± 17%      -0.0        0.09 ± 20%  perf-profile.children.cycles-pp.read
      0.16 ±  8%      -0.0        0.13 ±  8%  perf-profile.children.cycles-pp.crc_31
      0.09 ±  8%      -0.0        0.07 ± 11%  perf-profile.children.cycles-pp.crc_78
      0.06 ± 23%      +0.0        0.10 ± 12%  perf-profile.children.cycles-pp.arch_cpu_idle_exit
      0.19 ± 21%      +0.1        0.27 ± 10%  perf-profile.children.cycles-pp.xas_store
      0.46 ±  6%      +0.1        0.55 ±  8%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.41 ± 12%      +0.1        0.51 ±  5%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.00            +0.1        0.10 ± 17%  perf-profile.children.cycles-pp.workingset_update_node
      0.34 ±  7%      +0.2        0.52 ± 12%  perf-profile.children.cycles-pp.native_apic_msr_eoi_write
      1.53 ±  5%      +0.3        1.80 ±  7%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.16 ±  7%      +0.3        0.44 ± 23%  perf-profile.children.cycles-pp.tick_nohz_irq_exit
      0.26 ± 12%      +0.3        0.56 ± 36%  perf-profile.children.cycles-pp._raw_spin_trylock
      0.38 ± 14%      +0.3        0.72 ± 11%  perf-profile.children.cycles-pp.lapic_next_deadline
      0.97 ±  6%      +0.4        1.40 ± 14%  perf-profile.children.cycles-pp.tick_nohz_next_event
      0.47 ± 10%      +0.5        0.99 ± 10%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.97 ±  6%      +0.9        1.87 ± 25%  perf-profile.children.cycles-pp.ktime_get
      0.54 ± 10%      +1.0        1.56 ± 15%  perf-profile.children.cycles-pp.clockevents_program_event
     19.81 ±  3%      +7.3       27.10 ±  6%  perf-profile.children.cycles-pp.intel_idle
      2.44 ±  6%      +7.8       10.28 ± 18%  perf-profile.children.cycles-pp.ext4_ext_try_to_merge
      2.34 ±  6%      +7.9       10.20 ± 18%  perf-profile.children.cycles-pp.ext4_ext_try_to_merge_right
      2.26 ±  6%      +8.6       10.90 ± 18%  perf-profile.children.cycles-pp.__memmove
      3.70 ±  5%      -0.6        3.09 ±  8%  perf-profile.self.cycles-pp.ext4_find_extent
      0.53 ±  4%      -0.4        0.15 ± 95%  perf-profile.self.cycles-pp.check_cpu_stall
      0.52 ± 12%      -0.4        0.15 ± 66%  perf-profile.self.cycles-pp.io_serial_in
      0.45 ± 17%      -0.3        0.11 ± 86%  perf-profile.self.cycles-pp.perf_rotate_context
      0.35 ± 31%      -0.3        0.07 ±115%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.37 ± 12%      -0.3        0.09 ±120%  perf-profile.self.cycles-pp.timerqueue_add
      0.37 ±  8%      -0.3        0.10 ± 86%  perf-profile.self.cycles-pp.memcpy_toio
      0.36 ±  8%      -0.3        0.09 ±104%  perf-profile.self.cycles-pp.update_process_times
      0.44 ± 11%      -0.3        0.18 ± 57%  perf-profile.self.cycles-pp.__hrtimer_run_queues
      0.37 ±  8%      -0.3        0.11 ± 93%  perf-profile.self.cycles-pp.sysvec_apic_timer_interrupt
      1.44 ±  4%      -0.2        1.19 ± 11%  perf-profile.self.cycles-pp.__find_get_block
      1.04 ±  9%      -0.2        0.80 ± 13%  perf-profile.self.cycles-pp.crc_1
      0.41 ±  9%      -0.2        0.18 ± 50%  perf-profile.self.cycles-pp.update_irq_load_avg
      1.60 ±  5%      -0.2        1.38 ±  9%  perf-profile.self.cycles-pp.__might_resched
      0.33 ± 11%      -0.2        0.12 ± 50%  perf-profile.self.cycles-pp.irq_work_tick
      0.78 ± 10%      -0.2        0.58 ±  9%  perf-profile.self.cycles-pp.__cond_resched
      0.86 ±  5%      -0.2        0.67 ±  9%  perf-profile.self.cycles-pp.__kmem_cache_alloc_node
      0.24 ± 12%      -0.2        0.05 ±136%  perf-profile.self.cycles-pp.rb_insert_color
      0.69 ±  6%      -0.2        0.50 ± 17%  perf-profile.self.cycles-pp.__brelse
      0.55 ± 20%      -0.2        0.36 ± 17%  perf-profile.self.cycles-pp.__filemap_get_folio
      0.66 ±  7%      -0.2        0.50 ± 10%  perf-profile.self.cycles-pp.restore_fpregs_from_fpstate
      0.85 ±  6%      -0.2        0.70 ±  3%  perf-profile.self.cycles-pp.__might_sleep
      0.75 ±  7%      -0.1        0.61 ±  8%  perf-profile.self.cycles-pp.down_write
      0.25 ± 35%      -0.1        0.10 ± 21%  perf-profile.self.cycles-pp.get_page_from_freelist
      0.32 ± 14%      -0.1        0.18 ± 36%  perf-profile.self.cycles-pp.irqtime_account_irq
      0.20 ± 10%      -0.1        0.07 ± 71%  perf-profile.self.cycles-pp.update_rq_clock
      0.74 ±  7%      -0.1        0.61 ± 11%  perf-profile.self.cycles-pp.memset_erms
      0.19 ± 20%      -0.1        0.06 ± 57%  perf-profile.self.cycles-pp.calc_global_load_tick
      0.39 ± 10%      -0.1        0.27 ± 15%  perf-profile.self.cycles-pp.__ext4_handle_dirty_metadata
      0.22 ± 14%      -0.1        0.10 ± 42%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.59 ±  5%      -0.1        0.47 ±  9%  perf-profile.self.cycles-pp.arch_scale_freq_tick
      0.20 ± 12%      -0.1        0.09 ± 23%  perf-profile.self.cycles-pp.node_dirty_ok
      0.40 ±  8%      -0.1        0.29 ± 18%  perf-profile.self.cycles-pp.move_extent_per_page
      0.22 ± 13%      -0.1        0.11 ± 42%  perf-profile.self.cycles-pp.ct_kernel_enter
      0.21 ± 15%      -0.1        0.10 ± 33%  perf-profile.self.cycles-pp.rebalance_domains
      0.44 ± 10%      -0.1        0.34 ±  6%  perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      0.39 ± 10%      -0.1        0.30 ± 12%  perf-profile.self.cycles-pp.xas_load
      0.35 ± 10%      -0.1        0.27 ± 18%  perf-profile.self.cycles-pp.fio_e4defrag_queue
      0.24 ± 12%      -0.1        0.17 ± 17%  perf-profile.self.cycles-pp.rcu_pending
      0.16 ± 15%      -0.1        0.08 ± 30%  perf-profile.self.cycles-pp.load_balance
      0.26 ± 11%      -0.1        0.18 ± 19%  perf-profile.self.cycles-pp.__kmalloc
      0.61 ±  5%      -0.1        0.53 ±  8%  perf-profile.self.cycles-pp.__ext4_ext_dirty
      0.26 ±  5%      -0.1        0.19 ±  2%  perf-profile.self.cycles-pp.lru_add_fn
      0.25 ± 12%      -0.1        0.18 ± 18%  perf-profile.self.cycles-pp.td_io_queue
      0.35 ± 10%      -0.1        0.28 ± 13%  perf-profile.self.cycles-pp.__getblk_gfp
      0.39 ±  6%      -0.1        0.32 ± 10%  perf-profile.self.cycles-pp.kernel_fpu_begin_mask
      0.32 ± 10%      -0.1        0.25 ±  9%  perf-profile.self.cycles-pp.rcu_all_qs
      0.11 ± 10%      -0.1        0.05 ± 73%  perf-profile.self.cycles-pp.trigger_load_balance
      0.16 ± 20%      -0.1        0.10 ±  6%  perf-profile.self.cycles-pp.rmqueue
      0.22 ± 11%      -0.1        0.16 ± 13%  perf-profile.self.cycles-pp.irq_fpu_usable
      0.10 ± 28%      -0.1        0.04 ± 72%  perf-profile.self.cycles-pp.ct_idle_exit
      0.13 ±  8%      -0.1        0.07 ± 27%  perf-profile.self.cycles-pp.sched_clock_cpu
      0.08 ± 20%      -0.1        0.03 ±102%  perf-profile.self.cycles-pp.crc_84
      0.08 ± 14%      -0.1        0.03 ±101%  perf-profile.self.cycles-pp.put_io_u
      0.08 ± 15%      -0.0        0.03 ±101%  perf-profile.self.cycles-pp.jbd2_journal_stop
      0.08 ± 17%      -0.0        0.03 ±101%  perf-profile.self.cycles-pp.crc_93
      0.15 ± 12%      -0.0        0.10 ±  9%  perf-profile.self.cycles-pp.crc_19
      0.22 ±  9%      -0.0        0.18 ± 11%  perf-profile.self.cycles-pp.jbd2_journal_dirty_metadata
      0.16 ± 11%      -0.0        0.11 ±  6%  perf-profile.self.cycles-pp.memcg_slab_post_alloc_hook
      0.19 ±  5%      -0.0        0.15 ± 10%  perf-profile.self.cycles-pp.thread_main
      0.08 ± 12%      -0.0        0.04 ± 73%  perf-profile.self.cycles-pp.crc_60
      0.15 ±  9%      -0.0        0.11 ± 10%  perf-profile.self.cycles-pp.crc_31
      0.14 ± 15%      -0.0        0.11 ± 16%  perf-profile.self.cycles-pp.crc_28
      0.08 ± 14%      -0.0        0.05 ± 45%  perf-profile.self.cycles-pp.crc_80
      0.08 ± 19%      -0.0        0.05 ± 47%  perf-profile.self.cycles-pp.crc_50
      0.18 ±  5%      -0.0        0.15 ± 15%  perf-profile.self.cycles-pp.stop_this_handle
      0.14 ±  6%      -0.0        0.12 ±  9%  perf-profile.self.cycles-pp.crc_13
      0.07 ± 16%      -0.0        0.05 ±  8%  perf-profile.self.cycles-pp.crc_70
      0.14 ± 19%      +0.1        0.21 ± 12%  perf-profile.self.cycles-pp.do_idle
      0.38 ± 11%      +0.1        0.47 ±  6%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.15 ±  9%      +0.1        0.24 ±  5%  perf-profile.self.cycles-pp.asm_sysvec_apic_timer_interrupt
      0.00            +0.1        0.09 ± 18%  perf-profile.self.cycles-pp.workingset_update_node
      0.18 ±  8%      +0.1        0.27 ± 13%  perf-profile.self.cycles-pp.cpuidle_enter
      0.33 ±  7%      +0.2        0.52 ± 12%  perf-profile.self.cycles-pp.native_apic_msr_eoi_write
      0.26 ± 11%      +0.3        0.56 ± 37%  perf-profile.self.cycles-pp._raw_spin_trylock
      2.25 ±  3%      +0.3        2.56 ±  8%  perf-profile.self.cycles-pp.cpuidle_enter_state
      0.38 ± 13%      +0.3        0.72 ± 11%  perf-profile.self.cycles-pp.lapic_next_deadline
      0.46 ± 10%      +0.5        0.99 ± 10%  perf-profile.self.cycles-pp.native_irq_return_iret
      0.51 ±  9%      +0.8        1.33 ± 35%  perf-profile.self.cycles-pp.ktime_get
     19.81 ±  3%      +7.3       27.09 ±  6%  perf-profile.self.cycles-pp.intel_idle
      2.25 ±  7%      +8.6       10.86 ± 18%  perf-profile.self.cycles-pp.__memmove



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests



View attachment "config-6.3.0-rc3-00009-g776f07f465f7" of type "text/plain" (157147 bytes)

View attachment "job-script" of type "text/plain" (8405 bytes)

View attachment "job.yaml" of type "text/plain" (6032 bytes)

View attachment "reproduce" of type "text/plain" (714 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ