lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Date:   Fri, 30 Apr 2021 16:02:01 +0800
From:   kernel test robot <oliver.sang@...el.com>
To:     "Paul E. McKenney" <paulmck@...nel.org>
Cc:     John Stultz <john.stultz@...aro.org>,
        Thomas Gleixner <tglx@...utronix.de>,
        Stephen Boyd <sboyd@...nel.org>,
        Jonathan Corbet <corbet@....net>,
        Mark Rutland <Mark.Rutland@....com>,
        Marc Zyngier <maz@...nel.org>, Andi Kleen <ak@...ux.intel.com>,
        Chris Mason <clm@...com>, LKML <linux-kernel@...r.kernel.org>,
        Linux Memory Management List <linux-mm@...ck.org>,
        lkp@...ts.01.org, lkp@...el.com, ying.huang@...el.com,
        feng.tang@...el.com, zhengjun.xing@...el.com
Subject: [clocksource]  8c4b798316:  stress-ng.lockbus.ops_per_sec -8.5%
 regression



Greeting,

FYI, we noticed a -8.5% regression of stress-ng.lockbus.ops_per_sec due to commit:


commit: 8c4b798316c77d18b351a36e3660e412de762d79 ("clocksource: Retry clock read if long delays detected")
https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git master


in testcase: stress-ng
on test machine: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 512G memory
with following parameters:

	nr_threads: 100%
	disk: 1HDD
	testtime: 60s
	class: cpu-cache
	test: lockbus
	cpufreq_governor: performance
	ucode: 0x5003006




If you fix the issue, kindly add following tag
Reported-by: kernel test robot <oliver.sang@...el.com>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install                job.yaml  # job file is attached in this email
        bin/lkp split-job --compatible job.yaml  # generate the yaml file for lkp run
        bin/lkp run                    generated-yaml-file

=========================================================================================
class/compiler/cpufreq_governor/disk/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime/ucode:
  cpu-cache/gcc-9/performance/1HDD/x86_64-rhel-8.3/100%/debian-10.4-x86_64-20200603.cgz/lkp-csl-2sp7/lockbus/stress-ng/60s/0x5003006

commit: 
  5005eca245 ("clocksource: Provide module parameters to inject delays in watchdog")
  8c4b798316 ("clocksource: Retry clock read if long delays detected")

5005eca245d49ac4 8c4b798316c77d18b351a36e366 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
      4115            -8.5%       3766        stress-ng.lockbus.ops_per_sec
      6784 ±  2%     +23.0%       8344        stress-ng.time.percent_of_cpu_this_job_got
    208.45 ±  5%     +67.4%     348.85 ± 11%  stress-ng.time.system_time
      4019 ±  2%     +22.2%       4910        stress-ng.time.user_time
     42341 ± 48%    +330.6%     182325 ± 37%  cpuidle.POLL.time
    618938 ±  3%     +10.7%     685156 ±  2%  numa-meminfo.node1.Mapped
     64.00 ±  2%     +20.6%      77.17        vmstat.cpu.us
     26.34 ±  7%     -54.6%      11.96 ±  7%  iostat.cpu.system
     65.83 ±  3%     +21.4%      79.90        iostat.cpu.user
     13.77 ± 12%     -12.9        0.82 ± 38%  mpstat.cpu.all.irq%
     59.54 ±  4%     +13.9       73.47        mpstat.cpu.all.usr%
    142187 ±  5%     +13.1%     160881 ±  4%  numa-vmstat.node1.nr_mapped
    141628 ±  4%     +14.7%     162466 ±  4%  numa-vmstat.node1.nr_shmem
      8265 ± 58%     -86.7%       1097 ± 37%  softirqs.NET_RX
    175377 ±  8%     +15.2%     201967 ±  5%  softirqs.RCU
     51992            +2.4%      53264        interrupts.CAL:Function_call_interrupts
     56247 ±  6%     +32.5%      74499 ± 10%  interrupts.CPU0.LOC:Local_timer_interrupts
     92.33 ± 20%     +50.4%     138.83 ±  7%  interrupts.CPU0.RES:Rescheduling_interrupts
      5377 ± 28%     -42.3%       3100 ± 30%  interrupts.CPU10.NMI:Non-maskable_interrupts
      5377 ± 28%     -42.3%       3100 ± 30%  interrupts.CPU10.PMI:Performance_monitoring_interrupts
    291.50 ± 10%     +29.4%     377.33 ±  5%  interrupts.CPU47.RES:Rescheduling_interrupts
      1967 ±  3%     +20.3%       2366        interrupts.RES:Rescheduling_interrupts
    842.17 ± 66%    +446.0%       4597 ± 65%  proc-vmstat.nr_active_anon
      2978            +9.4%       3257        proc-vmstat.nr_page_table_pages
     20810            +1.7%      21165        proc-vmstat.nr_slab_reclaimable
    842.17 ± 66%    +446.0%       4597 ± 65%  proc-vmstat.nr_zone_active_anon
    131948 ±140%    +136.1%     311596 ± 38%  proc-vmstat.pgactivate
    518116 ±  2%      +4.6%     541978        proc-vmstat.pgfault
      4733 ±  6%     +50.7%       7131 ±  5%  proc-vmstat.pgreuse
      0.00 ± 55%    +587.5%       0.01 ± 25%  perf-sched.sch_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      0.00 ± 47%    +295.8%       0.02 ±  9%  perf-sched.sch_delay.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      0.01 ± 19%    +310.8%       0.03 ± 56%  perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
      0.00 ± 46%    +540.0%       0.03 ± 36%  perf-sched.sch_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      0.01 ± 28%    +694.1%       0.05 ± 75%  perf-sched.sch_delay.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
      0.01 ± 11%    +541.0%       0.04 ± 70%  perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
      2.80 ± 14%     +65.7%       4.63 ± 12%  perf-sched.sch_delay.max.ms.worker_thread.kthread.ret_from_fork
      2.72 ± 48%     -90.9%       0.25 ±223%  perf-sched.wait_and_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      4.33 ± 68%     -92.3%       0.33 ±223%  perf-sched.wait_and_delay.count.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      6.73 ± 64%     -93.1%       0.46 ±223%  perf-sched.wait_and_delay.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
   2516450 ±  2%     +19.5%    3007360        sched_debug.cfs_rq:/.min_vruntime.avg
   2786568 ±  3%     +14.7%    3195433        sched_debug.cfs_rq:/.min_vruntime.max
   1341933 ±  6%     +25.8%    1687931 ±  5%  sched_debug.cfs_rq:/.min_vruntime.min
      1.13 ± 97%     -99.1%       0.01 ±223%  sched_debug.cfs_rq:/.spread.avg
    108.61 ± 97%     -99.1%       1.00 ±223%  sched_debug.cfs_rq:/.spread.max
     11.03 ± 97%     -99.1%       0.10 ±223%  sched_debug.cfs_rq:/.spread.stddev
   1108101 ± 13%     -84.4%     172708 ± 41%  sched_debug.cfs_rq:/.spread0.avg
   1377563 ± 11%     -74.7%     348242 ± 17%  sched_debug.cfs_rq:/.spread0.max
    -65171         +1722.9%   -1188034        sched_debug.cfs_rq:/.spread0.min
   3820707 ±  9%     -36.2%    2436693 ± 14%  sched_debug.cpu.avg_idle.avg
  13038027 ± 12%     -40.3%    7789745 ± 14%  sched_debug.cpu.avg_idle.max
   2506387 ± 12%     -35.0%    1628581 ± 21%  sched_debug.cpu.avg_idle.stddev
     20.52 ± 36%    +998.3%     225.34 ± 13%  sched_debug.cpu.clock.stddev
     42166 ±  3%     +29.9%      54759        sched_debug.cpu.clock_task.min
      1867 ± 11%     -60.3%     741.00 ±  3%  sched_debug.cpu.clock_task.stddev
   3607599 ±  3%     -57.6%    1531356 ± 15%  sched_debug.cpu.max_idle_balance_cost.avg
   9226194 ± 11%     -47.4%    4857518 ± 11%  sched_debug.cpu.max_idle_balance_cost.max
   1892220 ±  8%     -56.3%     826153 ± 23%  sched_debug.cpu.max_idle_balance_cost.stddev
      1.00          -100.0%       0.00        sched_debug.sched_clock_stable()
      4.56 ±  4%     -64.0%       1.64 ± 11%  perf-stat.i.MPKI
 1.042e+09 ±  7%     +19.1%  1.242e+09 ± 10%  perf-stat.i.branch-instructions
      2.63 ±  4%      -1.6        1.05 ±  6%  perf-stat.i.branch-miss-rate%
  41155786 ±  8%     -50.2%   20492728 ±  8%  perf-stat.i.branch-misses
   3790635 ± 13%     -39.3%    2299993 ±  6%  perf-stat.i.cache-misses
  16736018 ±  6%     -34.3%   10987918 ±  5%  perf-stat.i.cache-references
      3040 ±  4%      -9.5%       2751 ±  3%  perf-stat.i.context-switches
    298.11 ±  8%     -80.7%      57.40 ± 22%  perf-stat.i.cpi
    125682 ±  2%     -14.8%     107100 ±  3%  perf-stat.i.cpu-clock
    191.72 ±  3%     -16.2%     160.71 ±  2%  perf-stat.i.cpu-migrations
    294407 ±  9%     -27.7%     212990 ±  5%  perf-stat.i.cycles-between-cache-misses
      0.02 ± 10%      -0.0        0.01 ± 12%  perf-stat.i.dTLB-load-miss-rate%
    121512 ±  5%     -34.4%      79730 ± 11%  perf-stat.i.dTLB-load-misses
      0.20 ±  8%      +0.0        0.24 ± 10%  perf-stat.i.dTLB-store-miss-rate%
 4.246e+08 ±  7%     -44.8%  2.341e+08 ±  8%  perf-stat.i.dTLB-stores
     77.32 ±  2%      +8.8       86.11        perf-stat.i.iTLB-load-miss-rate%
   1414999 ±  3%     -17.9%    1162074 ±  6%  perf-stat.i.iTLB-load-misses
    851842 ±  8%     -50.2%     423983 ±  8%  perf-stat.i.iTLB-loads
      2808 ±  7%     +83.6%       5156 ± 10%  perf-stat.i.instructions-per-iTLB-miss
      0.31 ±  9%     -54.3%       0.14 ± 11%  perf-stat.i.ipc
     93.97 ±  7%     -54.3%      42.90 ± 10%  perf-stat.i.major-faults
      1.78 ±  5%     +31.6%       2.34        perf-stat.i.metric.GHz
      1.26 ± 14%     -30.8%       0.87 ±  4%  perf-stat.i.metric.K/sec
      5445 ± 10%     -37.7%       3394 ±  7%  perf-stat.i.minor-faults
     74.27            +9.3       83.52        perf-stat.i.node-load-miss-rate%
    146539 ± 14%     -38.8%      89694 ±  7%  perf-stat.i.node-loads
     49.40 ±  4%     +28.1       77.46        perf-stat.i.node-store-miss-rate%
    390280 ±  6%     -56.4%     170291 ±  7%  perf-stat.i.node-stores
      5539 ± 10%     -38.0%       3437 ±  7%  perf-stat.i.page-faults
    125706 ±  2%     -14.8%     107108 ±  3%  perf-stat.i.task-clock
      3.86 ±  3%     -63.2%       1.42 ± 13%  perf-stat.overall.MPKI
      3.34 ±  2%      -2.6        0.77 ± 10%  perf-stat.overall.branch-miss-rate%
    162.35 ± 10%     -66.2%      54.82 ± 16%  perf-stat.overall.cpi
      0.01 ±  8%      -0.0        0.01 ± 10%  perf-stat.overall.dTLB-load-miss-rate%
     80.16 ±  2%      +5.1       85.23        perf-stat.overall.iTLB-load-miss-rate%
      1771 ± 10%    +180.8%       4973 ± 12%  perf-stat.overall.instructions-per-iTLB-miss
      0.01 ± 11%    +199.8%       0.02 ± 15%  perf-stat.overall.ipc
     73.89            +9.2       83.10        perf-stat.overall.node-load-miss-rate%
     34.75 ±  5%     +32.7       67.46 ±  2%  perf-stat.overall.node-store-miss-rate%
 2.203e+08 ± 18%    +348.7%  9.885e+08 ± 15%  perf-stat.ps.branch-instructions
    872384 ± 13%     +47.2%    1284050 ±  4%  perf-stat.ps.cache-misses
   4106664 ± 14%     +51.6%    6227454 ±  6%  perf-stat.ps.cache-references
      1871 ± 10%     +31.4%       2459        perf-stat.ps.context-switches
     71293 ±  9%     +28.9%      91877        perf-stat.ps.cpu-clock
 1.704e+11 ±  7%     +40.1%  2.386e+11        perf-stat.ps.cpu-cycles
    114.07 ±  7%     +21.1%     138.18 ±  3%  perf-stat.ps.cpu-migrations
     32365 ± 10%     +42.9%      46255 ±  8%  perf-stat.ps.dTLB-load-misses
 2.409e+08 ± 17%    +165.6%  6.397e+08 ± 13%  perf-stat.ps.dTLB-loads
    602712 ± 12%     +48.6%     895483 ±  5%  perf-stat.ps.iTLB-load-misses
  1.07e+09 ± 18%    +317.2%  4.464e+09 ± 15%  perf-stat.ps.instructions
     16.30 ± 16%     -24.1%      12.37 ±  7%  perf-stat.ps.major-faults
      1353 ±  9%     +38.3%       1871 ±  6%  perf-stat.ps.minor-faults
    101821 ± 18%    +138.8%     243173 ±  4%  perf-stat.ps.node-load-misses
     35793 ± 15%     +38.0%      49397 ±  4%  perf-stat.ps.node-loads
     43679 ± 17%    +201.9%     131871 ±  7%  perf-stat.ps.node-store-misses
     82413 ± 20%     -23.0%      63446 ±  4%  perf-stat.ps.node-stores
      1369 ±  9%     +37.5%       1883 ±  6%  perf-stat.ps.page-faults
     71298 ±  9%     +28.9%      91878        perf-stat.ps.task-clock
 4.783e+10 ±  3%    +472.2%  2.737e+11 ± 13%  perf-stat.total.instructions
     36.27 ±  5%     -14.1       22.21 ± 27%  perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
     35.94 ±  5%     -13.9       22.05 ± 27%  perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
     37.43 ±  5%     -10.6       26.84 ± 26%  perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
     42.76 ±  4%      -9.8       32.92 ± 14%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
     29.39 ±  7%      -8.9       20.48 ± 28%  perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
     19.99 ±  7%      -8.5       11.52 ± 32%  perf-profile.calltrace.cycles-pp.task_tick_fair.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer
      6.31 ± 32%      -5.6        0.71 ± 19%  perf-profile.calltrace.cycles-pp.perf_event_task_tick.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer
      5.16 ± 13%      -2.5        2.64 ± 31%  perf-profile.calltrace.cycles-pp.update_curr.task_tick_fair.scheduler_tick.update_process_times.tick_sched_handle
      5.78 ± 11%      -2.1        3.71 ± 36%  perf-profile.calltrace.cycles-pp.update_load_avg.task_tick_fair.scheduler_tick.update_process_times.tick_sched_handle
      4.04 ± 13%      -1.7        2.34 ± 29%  perf-profile.calltrace.cycles-pp.hrtimer_active.task_tick_fair.scheduler_tick.update_process_times.tick_sched_handle
      2.22 ± 16%      -1.7        0.53 ± 45%  perf-profile.calltrace.cycles-pp.rcu_sched_clock_irq.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
      2.72 ±  8%      -1.1        1.62 ± 13%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
      2.62 ±  7%      -1.1        1.56 ± 13%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.83 ± 11%      -0.8        1.03 ± 47%  perf-profile.calltrace.cycles-pp.update_cfs_group.task_tick_fair.scheduler_tick.update_process_times.tick_sched_handle
      1.23 ±  2%      -0.7        0.53 ± 48%  perf-profile.calltrace.cycles-pp.hrtimer_update_next_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      3.04 ±  6%      -0.7        2.37 ±  3%  perf-profile.calltrace.cycles-pp.load_balance.rebalance_domains.__softirqentry_text_start.irq_exit_rcu.sysvec_apic_timer_interrupt
      3.74 ±  6%      -0.6        3.14 ±  3%  perf-profile.calltrace.cycles-pp.rebalance_domains.__softirqentry_text_start.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      1.54 ±  8%      -0.6        0.98 ± 50%  perf-profile.calltrace.cycles-pp.__update_load_avg_cfs_rq.update_load_avg.task_tick_fair.scheduler_tick.update_process_times
      1.68 ± 10%      -0.6        1.13 ± 10%  perf-profile.calltrace.cycles-pp.execve
      1.68 ± 10%      -0.6        1.13 ± 10%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.execve
      1.68 ± 10%      -0.6        1.13 ± 10%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve
      1.68 ± 10%      -0.6        1.13 ± 10%  perf-profile.calltrace.cycles-pp.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve
      1.67 ± 10%      -0.5        1.12 ± 10%  perf-profile.calltrace.cycles-pp.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve
      2.05 ±  6%      -0.5        1.51 ±  4%  perf-profile.calltrace.cycles-pp.find_busiest_group.load_balance.rebalance_domains.__softirqentry_text_start.irq_exit_rcu
      1.39 ±  9%      -0.5        0.92 ± 11%  perf-profile.calltrace.cycles-pp.bprm_execve.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.66 ±  8%      -0.4        1.22 ±  9%  perf-profile.calltrace.cycles-pp.update_sd_lb_stats.find_busiest_group.load_balance.rebalance_domains.__softirqentry_text_start
      1.26 ±  9%      -0.4        0.83 ± 12%  perf-profile.calltrace.cycles-pp.exec_binprm.bprm_execve.do_execveat_common.__x64_sys_execve.do_syscall_64
      0.71 ±  8%      -0.4        0.28 ±100%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_fork
      0.71 ±  8%      -0.4        0.28 ±100%  perf-profile.calltrace.cycles-pp.__do_sys_clone.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_fork
      0.71 ±  8%      -0.4        0.28 ±100%  perf-profile.calltrace.cycles-pp.kernel_clone.__do_sys_clone.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_fork
      1.22 ±  9%      -0.4        0.80 ± 12%  perf-profile.calltrace.cycles-pp.load_elf_binary.exec_binprm.bprm_execve.do_execveat_common.__x64_sys_execve
      1.44 ± 22%      -0.4        1.07 ± 17%  perf-profile.calltrace.cycles-pp.smpboot_thread_fn.kthread.ret_from_fork
      0.71 ±  8%      -0.3        0.37 ± 71%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_fork
      0.70 ± 12%      -0.3        0.38 ± 72%  perf-profile.calltrace.cycles-pp.exit_mmap.mmput.do_exit.do_group_exit.__x64_sys_exit_group
      0.70 ± 12%      -0.3        0.38 ± 72%  perf-profile.calltrace.cycles-pp.mmput.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      0.87 ± 20%      -0.3        0.56 ± 49%  perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.update_sd_lb_stats.find_busiest_group.load_balance.rebalance_domains
      0.86 ± 11%      -0.3        0.55 ± 47%  perf-profile.calltrace.cycles-pp.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.86 ± 11%      -0.3        0.55 ± 47%  perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.86 ± 11%      -0.3        0.55 ± 47%  perf-profile.calltrace.cycles-pp.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.36 ±  8%      -0.3        1.08 ± 13%  perf-profile.calltrace.cycles-pp.asm_exc_page_fault
      1.30 ±  8%      -0.3        1.03 ± 13%  perf-profile.calltrace.cycles-pp.exc_page_fault.asm_exc_page_fault
      1.30 ±  8%      -0.3        1.03 ± 13%  perf-profile.calltrace.cycles-pp.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
      1.12 ±  9%      -0.2        0.88 ± 14%  perf-profile.calltrace.cycles-pp.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
      1.05 ±  9%      -0.2        0.82 ± 15%  perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
      0.92 ±  8%      -0.2        0.72 ±  9%  perf-profile.calltrace.cycles-pp.__libc_fork
      1.33 ±  8%      +0.4        1.72 ±  8%  perf-profile.calltrace.cycles-pp.new_sync_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.82 ± 11%      +0.7        1.51 ±  8%  perf-profile.calltrace.cycles-pp.seq_read_iter.new_sync_read.vfs_read.ksys_read.do_syscall_64
      0.70 ± 11%      +0.7        1.42 ±  9%  perf-profile.calltrace.cycles-pp.show_stat.seq_read_iter.new_sync_read.vfs_read.ksys_read
      0.31 ±104%      +0.8        1.09 ± 44%  perf-profile.calltrace.cycles-pp.rcu_core.__softirqentry_text_start.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      3.00 ± 11%      +2.2        5.17 ± 45%  perf-profile.calltrace.cycles-pp.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      0.00            +3.8        3.82 ± 19%  perf-profile.calltrace.cycles-pp.read_hpet.ktime_get.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
      0.85 ±  7%      +3.9        4.79 ± 44%  perf-profile.calltrace.cycles-pp.ktime_get.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
      0.00            +4.2        4.23 ± 20%  perf-profile.calltrace.cycles-pp.ktime_get.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
      0.00            +4.5        4.46 ± 44%  perf-profile.calltrace.cycles-pp.read_hpet.ktime_get.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt
      0.00            +5.7        5.73 ± 21%  perf-profile.calltrace.cycles-pp.read_hpet.ktime_get.sched_clock_tick.scheduler_tick.update_process_times
      0.00            +6.6        6.55 ± 22%  perf-profile.calltrace.cycles-pp.ktime_get.sched_clock_tick.scheduler_tick.update_process_times.tick_sched_handle
      0.00            +6.7        6.72 ± 22%  perf-profile.calltrace.cycles-pp.sched_clock_tick.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer
      3.02 ± 13%     +10.9       13.94 ± 22%  perf-profile.calltrace.cycles-pp.ktime_get_update_offsets_now.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
     64.95 ±  2%     +11.4       76.36 ±  3%  perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.00           +12.8       12.83 ± 23%  perf-profile.calltrace.cycles-pp.read_hpet.ktime_get_update_offsets_now.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
     55.01 ±  2%     +14.8       69.85 ±  4%  perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
     54.57 ±  2%     +15.2       69.76 ±  4%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      6.40 ±  6%     +15.2       21.64 ± 13%  perf-profile.calltrace.cycles-pp.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.00           +17.3       17.27 ± 13%  perf-profile.calltrace.cycles-pp.read_hpet.ktime_get.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt
      1.54 ±  2%     +17.6       19.16 ± 12%  perf-profile.calltrace.cycles-pp.ktime_get.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
     41.14 ±  5%     -16.3       24.84 ± 26%  perf-profile.children.cycles-pp.tick_sched_handle
     40.76 ±  5%     -16.1       24.67 ± 26%  perf-profile.children.cycles-pp.update_process_times
     42.58 ±  5%     -12.5       30.12 ± 25%  perf-profile.children.cycles-pp.tick_sched_timer
     48.92 ±  4%     -11.9       36.99 ± 13%  perf-profile.children.cycles-pp.__hrtimer_run_queues
     33.34 ±  7%     -10.5       22.86 ± 27%  perf-profile.children.cycles-pp.scheduler_tick
     22.36 ±  6%      -9.7       12.68 ± 31%  perf-profile.children.cycles-pp.task_tick_fair
      6.65 ± 26%      -5.8        0.83 ± 18%  perf-profile.children.cycles-pp.perf_event_task_tick
      4.88 ± 11%      -3.9        1.02 ± 35%  perf-profile.children.cycles-pp.ghes_notify_nmi
      5.18 ± 10%      -3.6        1.53 ± 24%  perf-profile.children.cycles-pp.irqtime_account_irq
      5.79 ± 10%      -2.6        3.21 ± 28%  perf-profile.children.cycles-pp.update_curr
      2.78 ±  7%      -2.5        0.26 ± 25%  perf-profile.children.cycles-pp.__intel_pmu_enable_all
      6.78 ± 10%      -2.4        4.41 ± 33%  perf-profile.children.cycles-pp.update_load_avg
     10.38 ±  7%      -2.1        8.23 ± 10%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
      4.48 ± 12%      -1.9        2.56 ± 29%  perf-profile.children.cycles-pp.hrtimer_active
      9.92 ±  7%      -1.9        8.00 ± 10%  perf-profile.children.cycles-pp.do_syscall_64
      2.54 ± 14%      -1.8        0.69 ± 20%  perf-profile.children.cycles-pp.rcu_sched_clock_irq
      2.66 ± 14%      -1.6        1.08 ± 35%  perf-profile.children.cycles-pp.native_sched_clock
      1.70 ±  2%      -1.1        0.57 ± 13%  perf-profile.children.cycles-pp._raw_spin_lock
      1.69 ±  2%      -1.1        0.58 ± 24%  perf-profile.children.cycles-pp.update_rq_clock
      1.74 ±  7%      -1.1        0.67 ± 38%  perf-profile.children.cycles-pp.native_flush_tlb_one_user
      1.36 ±  7%      -1.0        0.35 ± 17%  perf-profile.children.cycles-pp.trigger_load_balance
      1.15 ± 30%      -1.0        0.17 ± 47%  perf-profile.children.cycles-pp.nmi_handle
      1.26 ±  5%      -0.9        0.35 ± 29%  perf-profile.children.cycles-pp.arch_scale_freq_tick
      2.07 ± 10%      -0.9        1.20 ± 31%  perf-profile.children.cycles-pp.update_cfs_group
      3.57 ±  6%      -0.8        2.72 ±  4%  perf-profile.children.cycles-pp.load_balance
      0.99 ± 24%      -0.8        0.16 ± 24%  perf-profile.children.cycles-pp.sync_regs
      1.22 ±  4%      -0.8        0.39 ± 21%  perf-profile.children.cycles-pp.sched_clock_cpu
      4.41 ±  6%      -0.8        3.58 ±  4%  perf-profile.children.cycles-pp.rebalance_domains
      1.45 ±  2%      -0.8        0.67 ± 22%  perf-profile.children.cycles-pp.hrtimer_update_next_event
      2.38 ±  6%      -0.7        1.72 ±  5%  perf-profile.children.cycles-pp.find_busiest_group
      1.78 ± 10%      -0.6        1.15 ± 10%  perf-profile.children.cycles-pp.do_execveat_common
      1.79 ± 10%      -0.6        1.16 ± 10%  perf-profile.children.cycles-pp.__x64_sys_execve
      0.85 ±  8%      -0.6        0.22 ± 49%  perf-profile.children.cycles-pp.exc_nmi
      2.25 ±  6%      -0.6        1.62 ±  5%  perf-profile.children.cycles-pp.update_sd_lb_stats
      1.01 ±  6%      -0.6        0.39 ± 26%  perf-profile.children.cycles-pp.sched_clock
      1.68 ± 10%      -0.6        1.13 ± 10%  perf-profile.children.cycles-pp.execve
      1.48 ±  9%      -0.5        0.94 ± 10%  perf-profile.children.cycles-pp.bprm_execve
      1.34 ±  9%      -0.5        0.85 ± 11%  perf-profile.children.cycles-pp.exec_binprm
      1.30 ±  9%      -0.5        0.82 ± 12%  perf-profile.children.cycles-pp.load_elf_binary
      0.74 ±  3%      -0.5        0.27 ± 20%  perf-profile.children.cycles-pp.update_irq_load_avg
      2.50 ±  8%      -0.4        2.05 ±  9%  perf-profile.children.cycles-pp.asm_exc_page_fault
      1.33 ± 10%      -0.4        0.88 ± 16%  perf-profile.children.cycles-pp.mmput
      0.56 ±  7%      -0.4        0.11 ± 24%  perf-profile.children.cycles-pp.account_process_tick
      1.32 ± 10%      -0.4        0.88 ± 16%  perf-profile.children.cycles-pp.exit_mmap
      0.68 ±  6%      -0.4        0.24 ±  9%  perf-profile.children.cycles-pp.enqueue_hrtimer
      2.04 ±  8%      -0.4        1.60 ±  9%  perf-profile.children.cycles-pp.handle_mm_fault
      2.35 ±  8%      -0.4        1.92 ±  9%  perf-profile.children.cycles-pp.exc_page_fault
      2.35 ±  8%      -0.4        1.92 ±  8%  perf-profile.children.cycles-pp.do_user_addr_fault
      1.90 ±  8%      -0.4        1.47 ±  9%  perf-profile.children.cycles-pp.__handle_mm_fault
      0.99 ±  8%      -0.4        0.58 ±  8%  perf-profile.children.cycles-pp.do_sys_openat2
      0.99 ±  9%      -0.4        0.58 ±  8%  perf-profile.children.cycles-pp.do_sys_open
      0.96 ±  9%      -0.4        0.56 ±  6%  perf-profile.children.cycles-pp.do_filp_open
      0.95 ±  9%      -0.4        0.55 ±  6%  perf-profile.children.cycles-pp.path_openat
      0.60 ±  6%      -0.4        0.20 ± 12%  perf-profile.children.cycles-pp.timerqueue_add
      0.47 ± 18%      -0.4        0.09 ± 17%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.59 ± 41%      -0.4        0.21 ± 11%  perf-profile.children.cycles-pp.write
      0.76 ±  7%      -0.4        0.39 ±  9%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      1.45 ± 22%      -0.4        1.08 ± 17%  perf-profile.children.cycles-pp.smpboot_thread_fn
      0.69 ±  9%      -0.4        0.33 ± 28%  perf-profile.children.cycles-pp.acpi_os_read_memory
      0.58 ±  4%      -0.4        0.22 ± 29%  perf-profile.children.cycles-pp.__remove_hrtimer
      0.63 ±  4%      -0.4        0.28 ± 20%  perf-profile.children.cycles-pp.__hrtimer_next_event_base
      0.69 ± 11%      -0.3        0.35 ± 25%  perf-profile.children.cycles-pp.update_min_vruntime
      0.68 ±  2%      -0.3        0.38 ± 24%  perf-profile.children.cycles-pp.__hrtimer_get_next_event
      0.46 ±  4%      -0.3        0.16 ± 28%  perf-profile.children.cycles-pp.timerqueue_del
      0.70 ± 13%      -0.3        0.41 ± 13%  perf-profile.children.cycles-pp.begin_new_exec
      0.49 ± 41%      -0.3        0.21 ± 58%  perf-profile.children.cycles-pp.intel_pmu_handle_irq
      0.46 ±  2%      -0.3        0.19 ± 28%  perf-profile.children.cycles-pp.account_user_time
      1.04 ±  8%      -0.3        0.76 ±  8%  perf-profile.children.cycles-pp.do_fault
      0.36 ± 10%      -0.3        0.10 ± 29%  perf-profile.children.cycles-pp.ghes_copy_tofrom_phys
      0.39 ±  5%      -0.3        0.14 ± 37%  perf-profile.children.cycles-pp.__native_set_fixmap
      0.34 ±  6%      -0.3        0.09 ± 59%  perf-profile.children.cycles-pp.default_do_nmi
      0.66 ± 10%      -0.3        0.41 ± 16%  perf-profile.children.cycles-pp.vm_mmap_pgoff
      0.62 ± 10%      -0.2        0.38 ± 17%  perf-profile.children.cycles-pp.do_mmap
      0.67 ±  6%      -0.2        0.43 ± 14%  perf-profile.children.cycles-pp.try_to_wake_up
      0.88 ± 11%      -0.2        0.64 ± 17%  perf-profile.children.cycles-pp.do_exit
      0.65 ±  6%      -0.2        0.42 ± 14%  perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
      0.88 ± 11%      -0.2        0.64 ± 17%  perf-profile.children.cycles-pp.__x64_sys_exit_group
      0.88 ± 11%      -0.2        0.64 ± 17%  perf-profile.children.cycles-pp.do_group_exit
      0.35 ± 34%      -0.2        0.12 ± 22%  perf-profile.children.cycles-pp.pipe_write
      0.45 ± 25%      -0.2        0.22 ± 23%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
      0.83 ±  5%      -0.2        0.61 ± 17%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.51 ±  8%      -0.2        0.29 ± 12%  perf-profile.children.cycles-pp.kmem_cache_alloc
      0.56 ± 10%      -0.2        0.34 ± 16%  perf-profile.children.cycles-pp.mmap_region
      0.49 ±  8%      -0.2        0.29 ±  7%  perf-profile.children.cycles-pp.link_path_walk
      0.67 ± 11%      -0.2        0.46 ± 16%  perf-profile.children.cycles-pp.unmap_vmas
      0.47 ±  8%      -0.2        0.27 ±  8%  perf-profile.children.cycles-pp.walk_component
      0.53 ± 12%      -0.2        0.32 ± 17%  perf-profile.children.cycles-pp.ksys_mmap_pgoff
      0.72 ± 10%      -0.2        0.52 ± 10%  perf-profile.children.cycles-pp.filemap_map_pages
      0.92 ±  9%      -0.2        0.72 ±  9%  perf-profile.children.cycles-pp.__libc_fork
      0.30 ± 31%      -0.2        0.11 ± 18%  perf-profile.children.cycles-pp.pipe_read
      0.28 ± 31%      -0.2        0.09 ± 23%  perf-profile.children.cycles-pp.__wake_up_common_lock
      0.71 ±  8%      -0.2        0.53 ± 10%  perf-profile.children.cycles-pp.__do_sys_clone
      0.71 ±  7%      -0.2        0.53 ± 10%  perf-profile.children.cycles-pp.kernel_clone
      0.68 ±  8%      -0.2        0.50 ± 10%  perf-profile.children.cycles-pp.copy_process
      0.60 ± 12%      -0.2        0.42 ± 16%  perf-profile.children.cycles-pp.unmap_page_range
      0.55 ± 13%      -0.2        0.39 ± 18%  perf-profile.children.cycles-pp.zap_pte_range
      0.25 ±  5%      -0.2        0.09 ±  7%  perf-profile.children.cycles-pp.hrtimer_forward
      0.46 ± 10%      -0.2        0.30 ± 11%  perf-profile.children.cycles-pp.next_uptodate_page
      0.47 ±  8%      -0.2        0.32 ± 11%  perf-profile.children.cycles-pp.dup_mm
      0.46 ± 12%      -0.2        0.31 ± 14%  perf-profile.children.cycles-pp.setlocale
      0.45 ±  9%      -0.1        0.30 ± 12%  perf-profile.children.cycles-pp.dup_mmap
      0.20 ± 18%      -0.1        0.06 ± 72%  perf-profile.children.cycles-pp.native_set_fixmap
      0.23 ± 19%      -0.1        0.09 ±  5%  perf-profile.children.cycles-pp.proc_reg_read_iter
      0.24 ± 15%      -0.1        0.11 ± 36%  perf-profile.children.cycles-pp.sched_slice
      0.36 ±  9%      -0.1        0.24 ± 20%  perf-profile.children.cycles-pp.free_pgtables
      0.26 ±  3%      -0.1        0.14 ± 28%  perf-profile.children.cycles-pp.tick_program_event
      0.27 ± 10%      -0.1        0.15 ± 29%  perf-profile.children.cycles-pp.run_posix_cpu_timers
      0.31 ± 12%      -0.1        0.20 ± 18%  perf-profile.children.cycles-pp.__open64_nocancel
      0.21 ±  8%      -0.1        0.11 ±  6%  perf-profile.children.cycles-pp.alloc_empty_file
      0.25 ±  7%      -0.1        0.14 ± 45%  perf-profile.children.cycles-pp.calc_global_load_tick
      0.24 ±  6%      -0.1        0.14 ± 17%  perf-profile.children.cycles-pp.__split_vma
      0.18 ± 10%      -0.1        0.07 ± 16%  perf-profile.children.cycles-pp.rb_insert_color
      0.21 ±  9%      -0.1        0.10 ±  7%  perf-profile.children.cycles-pp.__alloc_file
      0.22 ±  9%      -0.1        0.12 ± 15%  perf-profile.children.cycles-pp.filename_lookup
      0.16 ± 55%      -0.1        0.06 ± 48%  perf-profile.children.cycles-pp.__poll
      0.22 ± 10%      -0.1        0.12 ± 17%  perf-profile.children.cycles-pp.path_lookupat
      0.27 ± 11%      -0.1        0.17 ± 21%  perf-profile.children.cycles-pp.__do_munmap
      0.47 ±  4%      -0.1        0.38 ±  9%  perf-profile.children.cycles-pp.__alloc_pages_nodemask
      0.20 ±  6%      -0.1        0.11 ± 18%  perf-profile.children.cycles-pp.__vma_adjust
      0.19 ± 11%      -0.1        0.11 ± 15%  perf-profile.children.cycles-pp.__lookup_slow
      0.17 ± 11%      -0.1        0.09 ± 15%  perf-profile.children.cycles-pp.vma_interval_tree_insert
      0.18 ± 15%      -0.1        0.10 ± 20%  perf-profile.children.cycles-pp.enqueue_task_fair
      0.17 ± 13%      -0.1        0.09 ± 16%  perf-profile.children.cycles-pp.__do_sys_newstat
      0.18 ± 15%      -0.1        0.10 ± 18%  perf-profile.children.cycles-pp.ttwu_do_activate
      0.34 ±  5%      -0.1        0.27 ± 14%  perf-profile.children.cycles-pp.kmem_cache_free
      0.17 ± 14%      -0.1        0.09 ± 16%  perf-profile.children.cycles-pp.vfs_statx
      0.28 ± 12%      -0.1        0.20 ±  7%  perf-profile.children.cycles-pp.call_timer_fn
      0.20 ±  5%      -0.1        0.13 ± 17%  perf-profile.children.cycles-pp.elf_map
      0.14 ± 11%      -0.1        0.07 ± 18%  perf-profile.children.cycles-pp.open64
      0.23 ±  9%      -0.1        0.15 ± 19%  perf-profile.children.cycles-pp.tlb_finish_mmu
      0.18 ±  9%      -0.1        0.10 ± 10%  perf-profile.children.cycles-pp.lookup_fast
      0.17 ±  9%      -0.1        0.10 ±  8%  perf-profile.children.cycles-pp.dput
      0.22 ±  9%      -0.1        0.15 ± 20%  perf-profile.children.cycles-pp.tlb_flush_mmu
      0.17 ±  9%      -0.1        0.10 ± 15%  perf-profile.children.cycles-pp.step_into
      0.19 ±  5%      -0.1        0.12 ± 10%  perf-profile.children.cycles-pp.__x64_sys_mprotect
      0.19 ±  5%      -0.1        0.12 ± 10%  perf-profile.children.cycles-pp.do_mprotect_pkey
      0.14 ±  9%      -0.1        0.07 ± 49%  perf-profile.children.cycles-pp.perf_trace_sched_stat_runtime
      0.17 ± 19%      -0.1        0.10 ± 13%  perf-profile.children.cycles-pp.select_task_rq_fair
      0.33 ±  4%      -0.1        0.27 ±  9%  perf-profile.children.cycles-pp.get_page_from_freelist
      0.20 ±  8%      -0.1        0.13 ± 22%  perf-profile.children.cycles-pp.cpuacct_charge
      0.13 ± 17%      -0.1        0.06 ± 17%  perf-profile.children.cycles-pp.kstat_irqs_usr
      0.18 ±  5%      -0.1        0.11 ± 10%  perf-profile.children.cycles-pp.mprotect_fixup
      0.17 ± 11%      -0.1        0.10 ± 23%  perf-profile.children.cycles-pp.vma_link
      0.12 ± 22%      -0.1        0.06 ± 15%  perf-profile.children.cycles-pp.touch_atime
      0.15 ± 15%      -0.1        0.09 ± 13%  perf-profile.children.cycles-pp.d_alloc_parallel
      0.09 ± 16%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.__xstat64
      0.14 ±  7%      -0.1        0.08 ± 32%  perf-profile.children.cycles-pp.__fput
      0.31 ±  9%      -0.1        0.26 ±  9%  perf-profile.children.cycles-pp.wp_page_copy
      0.15 ± 12%      -0.1        0.09 ± 14%  perf-profile.children.cycles-pp.do_open
      0.18 ± 16%      -0.1        0.12 ± 20%  perf-profile.children.cycles-pp.__mmap
      0.38 ±  6%      -0.1        0.32 ± 13%  perf-profile.children.cycles-pp.__calc_delta
      0.16 ±  8%      -0.1        0.11 ± 22%  perf-profile.children.cycles-pp.unlink_anon_vmas
      0.13 ± 17%      -0.1        0.07 ± 18%  perf-profile.children.cycles-pp.d_alloc
      0.11 ± 11%      -0.1        0.06 ± 46%  perf-profile.children.cycles-pp.remove_vma
      0.15 ± 17%      -0.1        0.10 ± 20%  perf-profile.children.cycles-pp.unlink_file_vma
      0.15 ±  8%      -0.1        0.09 ± 15%  perf-profile.children.cycles-pp.dequeue_task_fair
      0.12 ± 10%      -0.1        0.07 ± 21%  perf-profile.children.cycles-pp.vma_interval_tree_remove
      0.17 ±  8%      -0.1        0.11 ± 14%  perf-profile.children.cycles-pp.vmstat_update
      0.16 ±  8%      -0.1        0.11 ± 26%  perf-profile.children.cycles-pp.__cgroup_account_cputime_field
      0.09 ± 10%      -0.0        0.04 ± 73%  perf-profile.children.cycles-pp.__d_alloc
      0.14 ±  2%      -0.0        0.09 ± 15%  perf-profile.children.cycles-pp.load_elf_interp
      0.14 ± 13%      -0.0        0.10 ± 11%  perf-profile.children.cycles-pp._dl_addr
      0.16 ±  8%      -0.0        0.11 ± 14%  perf-profile.children.cycles-pp.refresh_cpu_vm_stats
      0.09 ±  5%      -0.0        0.04 ± 72%  perf-profile.children.cycles-pp.irq_work_run_list
      0.11 ±  9%      -0.0        0.06 ± 14%  perf-profile.children.cycles-pp.get_obj_cgroup_from_current
      0.11 ± 18%      -0.0        0.06 ± 48%  perf-profile.children.cycles-pp.__x64_sys_poll
      0.11 ± 18%      -0.0        0.06 ± 48%  perf-profile.children.cycles-pp.do_sys_poll
      0.15 ± 11%      -0.0        0.11 ± 12%  perf-profile.children.cycles-pp.copy_page_range
      0.10 ± 16%      -0.0        0.05 ± 46%  perf-profile.children.cycles-pp.do_dentry_open
      0.10 ±  6%      -0.0        0.06 ± 50%  perf-profile.children.cycles-pp.__cgroup_account_cputime
      0.16 ± 22%      -0.0        0.12 ± 17%  perf-profile.children.cycles-pp.update_rt_rq_load_avg
      0.07 ± 15%      -0.0        0.03 ±100%  perf-profile.children.cycles-pp.__dentry_kill
      0.09 ± 11%      -0.0        0.05 ± 46%  perf-profile.children.cycles-pp.__d_lookup_rcu
      0.11 ± 16%      -0.0        0.07 ± 20%  perf-profile.children.cycles-pp.enqueue_entity
      0.08 ± 20%      -0.0        0.04 ± 72%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.16 ± 11%      -0.0        0.12 ± 12%  perf-profile.children.cycles-pp.kcpustat_cpu_fetch
      0.10 ±  6%      -0.0        0.06 ± 13%  perf-profile.children.cycles-pp.vm_area_dup
      0.10 ± 15%      -0.0        0.06 ± 14%  perf-profile.children.cycles-pp.__might_sleep
      0.09 ± 10%      -0.0        0.06 ± 46%  perf-profile.children.cycles-pp.anon_vma_fork
      0.12 ±  8%      -0.0        0.08 ±  7%  perf-profile.children.cycles-pp.pte_alloc_one
      0.10 ± 18%      -0.0        0.07 ± 16%  perf-profile.children.cycles-pp.get_arg_page
      0.14 ±  8%      -0.0        0.10 ± 20%  perf-profile.children.cycles-pp.perf_event_mmap
      0.08 ± 17%      -0.0        0.04 ± 45%  perf-profile.children.cycles-pp.strnlen_user
      0.10 ± 17%      -0.0        0.07 ± 16%  perf-profile.children.cycles-pp.__get_user_pages_remote
      0.10 ± 18%      -0.0        0.06 ± 14%  perf-profile.children.cycles-pp.__get_user_pages
      0.08 ± 14%      -0.0        0.05 ± 45%  perf-profile.children.cycles-pp.create_elf_tables
      0.10 ± 10%      -0.0        0.07 ±  7%  perf-profile.children.cycles-pp.do_open_execat
      0.07 ±  7%      -0.0        0.04 ± 71%  perf-profile.children.cycles-pp.refill_obj_stock
      0.11 ± 10%      -0.0        0.08 ± 14%  perf-profile.children.cycles-pp.__clear_user
      0.09 ±  8%      -0.0        0.06 ± 11%  perf-profile.children.cycles-pp.lru_add_drain
      0.08 ± 13%      -0.0        0.06 ±  8%  perf-profile.children.cycles-pp.lru_add_drain_cpu
      0.08 ± 14%      -0.0        0.06 ± 13%  perf-profile.children.cycles-pp.__check_object_size
      0.15 ± 20%      +0.1        0.22 ± 17%  perf-profile.children.cycles-pp.memcpy_toio
      0.15 ± 20%      +0.1        0.24 ± 19%  perf-profile.children.cycles-pp.drm_fb_helper_damage_work
      0.18 ±  8%      +0.2        0.36 ± 57%  perf-profile.children.cycles-pp.do_idle
      0.18 ±  8%      +0.2        0.37 ± 57%  perf-profile.children.cycles-pp.secondary_startup_64_no_verify
      0.18 ±  8%      +0.2        0.37 ± 57%  perf-profile.children.cycles-pp.cpu_startup_entry
      0.17 ± 10%      +0.2        0.36 ± 54%  perf-profile.children.cycles-pp.start_secondary
      0.01 ±223%      +0.2        0.21 ± 63%  perf-profile.children.cycles-pp.cpuidle_enter
      0.01 ±223%      +0.2        0.21 ± 63%  perf-profile.children.cycles-pp.cpuidle_enter_state
      1.46 ±  9%      +0.3        1.74 ±  8%  perf-profile.children.cycles-pp.new_sync_read
      0.13 ±100%      +0.3        0.45 ± 16%  perf-profile.children.cycles-pp.collapse_huge_page
      0.10 ± 14%      +0.3        0.43 ±  7%  perf-profile.children.cycles-pp.get_cpu_idle_time_us
      0.29 ± 44%      +0.3        0.63 ± 34%  perf-profile.children.cycles-pp.process_interval
      0.10 ± 17%      +0.3        0.43 ±  7%  perf-profile.children.cycles-pp.get_idle_time
      0.13 ±103%      +0.3        0.48 ± 16%  perf-profile.children.cycles-pp.khugepaged_scan_mm_slot
      0.13 ±103%      +0.3        0.48 ± 16%  perf-profile.children.cycles-pp.khugepaged_scan_pmd
      0.14 ±100%      +0.3        0.49 ± 16%  perf-profile.children.cycles-pp.khugepaged
      0.04 ± 71%      +0.4        0.47 ± 10%  perf-profile.children.cycles-pp.get_iowait_time
      0.02 ±142%      +0.4        0.46 ± 11%  perf-profile.children.cycles-pp.get_cpu_iowait_time_us
      1.14 ±  9%      +0.5        1.67 ±  7%  perf-profile.children.cycles-pp.seq_read_iter
      0.80 ± 17%      +0.6        1.36 ± 38%  perf-profile.children.cycles-pp.rcu_core
      0.00            +0.7        0.67 ± 25%  perf-profile.children.cycles-pp.sched_clock_local
      0.71 ± 11%      +0.7        1.43 ±  9%  perf-profile.children.cycles-pp.show_stat
      3.58 ±  9%      +2.2        5.82 ± 43%  perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
      0.00            +7.6        7.59 ± 21%  perf-profile.children.cycles-pp.sched_clock_tick
     74.19 ±  2%     +11.5       85.71 ±  2%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
      3.43 ± 11%     +12.4       15.79 ± 21%  perf-profile.children.cycles-pp.ktime_get_update_offsets_now
     62.93 ±  2%     +15.6       78.51 ±  3%  perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
     62.43 ±  2%     +16.0       78.42 ±  3%  perf-profile.children.cycles-pp.hrtimer_interrupt
      6.89 ±  6%     +17.4       24.27 ± 13%  perf-profile.children.cycles-pp.clockevents_program_event
      3.42 ±  2%     +36.6       40.06 ±  6%  perf-profile.children.cycles-pp.ktime_get
      0.00           +50.2       50.16 ± 10%  perf-profile.children.cycles-pp.read_hpet
      4.88 ± 11%      -3.9        1.02 ± 35%  perf-profile.self.cycles-pp.ghes_notify_nmi
      3.79 ± 19%      -3.2        0.62 ± 10%  perf-profile.self.cycles-pp.perf_event_task_tick
      3.77 ± 16%      -2.5        1.30 ± 26%  perf-profile.self.cycles-pp.irqtime_account_irq
      2.27 ± 12%      -2.0        0.24 ± 21%  perf-profile.self.cycles-pp.__intel_pmu_enable_all
      2.98 ± 14%      -1.8        1.19 ± 15%  perf-profile.self.cycles-pp.ktime_get_update_offsets_now
      3.90 ± 16%      -1.7        2.21 ± 26%  perf-profile.self.cycles-pp.hrtimer_active
      3.49 ± 13%      -1.6        1.89 ± 26%  perf-profile.self.cycles-pp.update_curr
      2.06 ± 20%      -1.5        0.59 ± 18%  perf-profile.self.cycles-pp.rcu_sched_clock_irq
      2.42 ± 12%      -1.4        1.03 ± 34%  perf-profile.self.cycles-pp.native_sched_clock
      2.44 ± 13%      -1.4        1.07 ± 27%  perf-profile.self.cycles-pp.task_tick_fair
      1.61 ± 18%      -1.3        0.28 ± 23%  perf-profile.self.cycles-pp.update_process_times
      2.48 ± 22%      -1.2        1.27 ± 30%  perf-profile.self.cycles-pp.clockevents_program_event
      1.74 ±  7%      -1.1        0.67 ± 38%  perf-profile.self.cycles-pp.native_flush_tlb_one_user
      1.15 ± 30%      -1.0        0.17 ± 47%  perf-profile.self.cycles-pp.nmi_handle
      2.27 ± 11%      -0.9        1.36 ± 28%  perf-profile.self.cycles-pp.update_load_avg
      0.95 ± 25%      -0.8        0.16 ± 25%  perf-profile.self.cycles-pp.sync_regs
      1.04 ± 11%      -0.8        0.28 ± 18%  perf-profile.self.cycles-pp.trigger_load_balance
      1.15 ±  8%      -0.8        0.39 ±  5%  perf-profile.self.cycles-pp._raw_spin_lock
      1.74 ± 14%      -0.7        1.02 ± 27%  perf-profile.self.cycles-pp.update_cfs_group
      0.89 ±  8%      -0.6        0.26 ± 24%  perf-profile.self.cycles-pp.arch_scale_freq_tick
      0.84 ±  8%      -0.6        0.22 ± 49%  perf-profile.self.cycles-pp.exc_nmi
      0.49 ±  8%      -0.4        0.10 ± 21%  perf-profile.self.cycles-pp.account_process_tick
      0.72 ± 17%      -0.4        0.33 ± 20%  perf-profile.self.cycles-pp.hrtimer_interrupt
      0.46 ± 19%      -0.4        0.07 ± 10%  perf-profile.self.cycles-pp.native_irq_return_iret
      1.30 ±  6%      -0.4        0.92 ± 23%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.69 ±  9%      -0.4        0.33 ± 28%  perf-profile.self.cycles-pp.acpi_os_read_memory
      0.53 ± 15%      -0.3        0.20 ± 10%  perf-profile.self.cycles-pp.update_irq_load_avg
      0.51 ± 10%      -0.3        0.19 ± 13%  perf-profile.self.cycles-pp.__hrtimer_run_queues
      0.48 ±  7%      -0.3        0.17 ± 22%  perf-profile.self.cycles-pp.update_rq_clock
      0.50 ±  5%      -0.3        0.20 ± 27%  perf-profile.self.cycles-pp.scheduler_tick
      1.04 ± 14%      -0.3        0.74 ± 12%  perf-profile.self.cycles-pp.update_sd_lb_stats
      0.37 ± 29%      -0.3        0.08 ± 18%  perf-profile.self.cycles-pp.__sysvec_apic_timer_interrupt
      0.39 ± 12%      -0.3        0.11 ± 38%  perf-profile.self.cycles-pp.perf_mux_hrtimer_handler
      0.49 ± 41%      -0.3        0.21 ± 58%  perf-profile.self.cycles-pp.intel_pmu_handle_irq
      0.34 ±  2%      -0.3        0.07 ± 47%  perf-profile.self.cycles-pp.tick_sched_timer
      0.36 ± 10%      -0.3        0.10 ± 29%  perf-profile.self.cycles-pp.ghes_copy_tofrom_phys
      0.39 ±  5%      -0.3        0.14 ± 37%  perf-profile.self.cycles-pp.__native_set_fixmap
      0.34 ±  6%      -0.3        0.09 ± 59%  perf-profile.self.cycles-pp.default_do_nmi
      0.54 ±  7%      -0.2        0.29 ± 18%  perf-profile.self.cycles-pp.update_min_vruntime
      0.55 ± 14%      -0.2        0.34 ± 10%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.42 ±  9%      -0.2        0.22 ± 17%  perf-profile.self.cycles-pp.__hrtimer_next_event_base
      0.51 ±  4%      -0.2        0.31 ±  6%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.31 ± 12%      -0.2        0.11 ± 12%  perf-profile.self.cycles-pp.timerqueue_add
      0.48 ±  6%      -0.2        0.33 ± 13%  perf-profile.self.cycles-pp.update_blocked_averages
      0.20 ± 18%      -0.1        0.06 ± 72%  perf-profile.self.cycles-pp.native_set_fixmap
      0.20 ±  9%      -0.1        0.06 ± 47%  perf-profile.self.cycles-pp.account_user_time
      0.34 ±  8%      -0.1        0.21 ± 16%  perf-profile.self.cycles-pp.__hrtimer_get_next_event
      0.24 ± 12%      -0.1        0.11 ± 18%  perf-profile.self.cycles-pp.hrtimer_update_next_event
      0.22 ± 10%      -0.1        0.10 ± 11%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.18 ±  8%      -0.1        0.08 ± 12%  perf-profile.self.cycles-pp.hrtimer_forward
      0.22 ±  9%      -0.1        0.13 ± 46%  perf-profile.self.cycles-pp.calc_global_load_tick
      0.13 ± 12%      -0.1        0.05 ± 49%  perf-profile.self.cycles-pp.sched_slice
      0.12 ± 15%      -0.1        0.06 ±  8%  perf-profile.self.cycles-pp.rb_insert_color
      0.17 ± 13%      -0.1        0.11 ± 25%  perf-profile.self.cycles-pp.tick_program_event
      0.15 ± 17%      -0.1        0.10 ± 24%  perf-profile.self.cycles-pp.cpuacct_charge
      0.17 ±  2%      -0.1        0.12 ± 22%  perf-profile.self.cycles-pp.run_posix_cpu_timers
      0.09 ± 13%      -0.0        0.04 ± 45%  perf-profile.self.cycles-pp.finish_task_switch
      0.11 ± 10%      -0.0        0.06 ± 14%  perf-profile.self.cycles-pp.kmem_cache_alloc
      0.09 ± 15%      -0.0        0.05 ± 48%  perf-profile.self.cycles-pp.__cgroup_account_cputime_field
      0.09 ± 11%      -0.0        0.05 ± 49%  perf-profile.self.cycles-pp.__remove_hrtimer
      0.10 ± 12%      +0.1        0.18 ± 33%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      0.02 ±141%      +0.1        0.10 ± 11%  perf-profile.self.cycles-pp.memcpy_toio
      0.07 ± 19%      +0.1        0.19        perf-profile.self.cycles-pp.run_rebalance_domains
      0.00            +0.1        0.12 ± 35%  perf-profile.self.cycles-pp.rcu_core
      0.00            +0.3        0.33 ± 19%  perf-profile.self.cycles-pp.sched_clock_local
      0.17 ±  6%      +0.3        0.51 ± 11%  perf-profile.self.cycles-pp.__softirqentry_text_start
      2.38 ±  6%      +1.4        3.80 ±  7%  perf-profile.self.cycles-pp.ktime_get
      0.00           +47.1       47.13 ± 12%  perf-profile.self.cycles-pp.read_hpet


                                                                                
                             stress-ng.time.user_time                           
                                                                                
  5200 +--------------------------------------------------------------------+   
       | OO OO OO O  OO    O   O OO                                         |   
  5000 |-+         O    OO  O O     OO O  OO O  OO                          |   
  4800 |-+                              O     O                             |   
       |                                                                    |   
  4600 |-+                                                                  |   
  4400 |-+                                                                  |   
       |                                                                    |   
  4200 |-++.                                            .+            +     |   
  4000 |-+: ++          +                      .++     +  :      .+ .+ +  .+|   
       |.:    + +.+    : :    +         +    ++   + +. :  +.    +  +    ++  |   
  3800 |-+     +   :   : +.  + +.++.+   ::  +      +  +     ++  :           |   
  3600 |-+         +.++    ++        :.+ : +                  +:            |   
       |                             +    +                    +            |   
  3400 +--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                    stress-ng.time.percent_of_cpu_this_job_got                  
                                                                                
  9000 +--------------------------------------------------------------------+   
       |                                                                    |   
  8500 |-OO OO OO O  OO OO OO OO  O OO O   O    OO                          |   
       |           O                    O O  OO                             |   
  8000 |-+                       O                                          |   
       |                                                                    |   
  7500 |-+                                                                  |   
       |                                                                    |   
  7000 |-++.+                                           .+            +     |   
       |. :  +. +.      +                      .++  +  +  +     +.++.+ +  .+|   
  6500 |-+     +  +    + :   .++. +.+   +   .++   + :+ :   +    :       ++  |   
       |           +.++  +.++    +   :.+ + +       +  +     ++.:            |   
  6000 |-+                           +    +                    +            |   
       |                                                                    |   
  5500 +--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                stress-ng.lockbus.ops                           
                                                                                
  260000 +------------------------------------------------------------------+   
         |                     +                                            |   
  255000 |-+                  : :                     +.                    |   
  250000 |-.+   +. +.+   + .+ : +.  .++.  .+        .+  + .+ +.++.++.+   +. |   
         |+  +.+  +   +.+ +  +    ++    ++  ++.++.++     +  +         +.+  +|   
  245000 |-+                                                                |   
  240000 |-+                                                                |   
         |                                                                  |   
  235000 |-+                                 O                              |   
  230000 |-+         O            O     O      O                            |   
         |               OO  O O     O   O OO                               |   
  225000 |-+   O        O   O   O  O  O           O                         |   
  220000 |O+O   O OO  O                         O                           |   
         |   O                                                              |   
  215000 +------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                           stress-ng.lockbus.ops_per_sec                        
                                                                                
  4300 +--------------------------------------------------------------------+   
       |                      +                                             |   
  4200 |-+                   : :                      +                     |   
       |. +.  .++.++.  .++.+ : +.  .++.+ .++.        + +. +.++.++.++.+   +.+|   
  4100 |-+  ++       ++     +    ++     +    ++.++.++    +            +.+   |   
       |                                                                    |   
  4000 |-+                                                                  |   
       |                                                                    |   
  3900 |-+                                                                  |   
       |                                     O                              |   
  3800 |-+         O     O    O  O     OO  O  O                             |   
       |                O   O       O     O                                 |   
  3700 |-OO  O        O    O   O  O  O           O                          |   
       |    O  OO O  O                          O                           |   
  3600 +--------------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


---
0DAY/LKP+ Test Infrastructure                   Open Source Technology Center
https://lists.01.org/hyperkitty/list/lkp@lists.01.org       Intel Corporation

Thanks,
Oliver Sang


View attachment "config-5.12.0-rc5-00002-g8c4b798316c7" of type "text/plain" (172872 bytes)

View attachment "job-script" of type "text/plain" (8475 bytes)

View attachment "job.yaml" of type "text/plain" (5840 bytes)

View attachment "reproduce" of type "text/plain" (340 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ