lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Date:   Wed, 24 Nov 2021 16:34:35 +0800
From:   kernel test robot <oliver.sang@...el.com>
To:     Shakeel Butt <shakeelb@...gle.com>
Cc:     Linus Torvalds <torvalds@...ux-foundation.org>,
        Arnd Bergmann <arnd@...db.de>, Roman Gushchin <guro@...com>,
        Michal Hocko <mhocko@...e.com>,
        Muchun Song <songmuchun@...edance.com>,
        Vasily Averin <vvs@...tuozzo.com>,
        Johannes Weiner <hannes@...xchg.org>,
        Andrew Morton <akpm@...ux-foundation.org>,
        LKML <linux-kernel@...r.kernel.org>, lkp@...ts.01.org,
        lkp@...el.com, ying.huang@...el.com, feng.tang@...el.com,
        zhengjun.xing@...ux.intel.com, fengwei.yin@...el.com
Subject: [memcg, kmem]  58056f7750:  hackbench.throughput 10.3% improvement



Greeting,

FYI, we noticed a 10.3% improvement of hackbench.throughput due to commit:


commit: 58056f77502f3567b760c9a8fc8d2e9081515b2d ("memcg, kmem: further deprecate kmem.limit_in_bytes")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master

in testcase: hackbench
on test machine: 144 threads 4 sockets Intel(R) Xeon(R) Gold 5318H CPU @ 2.50GHz with 128G memory
with following parameters:

	nr_threads: 100%
	iterations: 4
	mode: process
	ipc: socket
	cpufreq_governor: performance
	ucode: 0x700001e

test-description: Hackbench is both a benchmark and a stress test for the Linux kernel scheduler.
test-url: https://github.com/linux-test-project/ltp/blob/master/testcases/kernel/sched/cfs-scheduler/hackbench.c





Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        sudo bin/lkp install job.yaml           # job file is attached in this email
        bin/lkp split-job --compatible job.yaml # generate the yaml file for lkp run
        sudo bin/lkp run generated-yaml-file

        # if come across any failure that blocks the test,
        # please remove ~/.lkp and /lkp dir to run from a clean state.

=========================================================================================
compiler/cpufreq_governor/ipc/iterations/kconfig/mode/nr_threads/rootfs/tbox_group/testcase/ucode:
  gcc-9/performance/socket/4/x86_64-rhel-8.3/process/100%/debian-10.4-x86_64-20200603.cgz/lkp-cpl-4sp1/hackbench/0x700001e

commit: 
  16f6bf266c ("mm/list_lru.c: prefer struct_size over open coded arithmetic")
  58056f7750 ("memcg, kmem: further deprecate kmem.limit_in_bytes")

16f6bf266c94017c 58056f77502f3567b760c9a8fc8 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
    124164           +10.3%     137012 ±  2%  hackbench.throughput
    553.20            -9.9%     498.51 ±  2%  hackbench.time.elapsed_time
    553.20            -9.9%     498.51 ±  2%  hackbench.time.elapsed_time.max
 3.519e+08 ±  4%     -56.3%  1.539e+08 ±  2%  hackbench.time.involuntary_context_switches
     76944           -10.0%      69266 ±  2%  hackbench.time.system_time
      1046           -15.9%     880.42 ±  2%  hackbench.time.user_time
 9.209e+08 ±  2%     -41.2%  5.414e+08        hackbench.time.voluntary_context_switches
    516011 ± 17%     -33.5%     343117 ± 32%  numa-numastat.node1.numa_hit
    260286 ±  8%     -16.7%     216841 ± 15%  numa-vmstat.node3.nr_shmem
    607.74            -8.8%     554.41        uptime.boot
    164520            +2.7%     168980        proc-vmstat.nr_slab_unreclaimable
    137719            -4.5%     131582        proc-vmstat.pgreuse
      0.42 ±  6%      +0.1        0.53 ±  6%  turbostat.C1%
 2.966e+08           -18.8%  2.408e+08 ±  2%  turbostat.IRQ
      2746           -12.3%       2408        vmstat.procs.r
   2336697 ±  3%     -38.4%    1439816        vmstat.system.cs
    534308            -9.9%     481527        vmstat.system.in
      5546 ±  5%     -12.6%       4846 ±  6%  slabinfo.kmalloc-cg-192.active_objs
      5546 ±  5%     -12.6%       4846 ±  6%  slabinfo.kmalloc-cg-192.num_objs
    157208            +9.5%     172170        slabinfo.kmalloc-cg-512.active_objs
      3128           +11.3%       3480        slabinfo.kmalloc-cg-512.active_slabs
    200236           +11.3%     222802        slabinfo.kmalloc-cg-512.num_objs
      3128           +11.3%       3480        slabinfo.kmalloc-cg-512.num_slabs
    157390            +9.5%     172289        slabinfo.skbuff_head_cache.active_objs
      3134           +11.2%       3485        slabinfo.skbuff_head_cache.active_slabs
    200626           +11.2%     223129        slabinfo.skbuff_head_cache.num_objs
      3134           +11.2%       3485        slabinfo.skbuff_head_cache.num_slabs
 1.669e+10            -5.2%  1.582e+10        perf-stat.i.branch-instructions
      0.80            -0.2        0.64 ±  2%  perf-stat.i.branch-miss-rate%
 1.312e+08           -24.3%   99283197 ±  2%  perf-stat.i.branch-misses
     27.80            +1.1       28.86        perf-stat.i.cache-miss-rate%
 8.619e+08            -1.5%  8.491e+08        perf-stat.i.cache-references
   2349252 ±  3%     -38.2%    1452325        perf-stat.i.context-switches
    143032 ±  2%     -14.7%     121977 ±  2%  perf-stat.i.cpu-migrations
      2001            -3.5%       1931        perf-stat.i.cycles-between-cache-misses
      0.07 ±  5%      -0.0        0.06 ±  5%  perf-stat.i.dTLB-load-miss-rate%
  16549027 ±  4%     -12.9%   14414299 ±  3%  perf-stat.i.dTLB-load-misses
 2.392e+10            -1.9%  2.348e+10        perf-stat.i.dTLB-loads
      0.03 ± 11%      +0.0        0.04 ±  6%  perf-stat.i.dTLB-store-miss-rate%
   4025883 ± 11%     +30.5%    5252098 ±  5%  perf-stat.i.dTLB-store-misses
  56000903           -35.7%   35989438 ±  8%  perf-stat.i.iTLB-load-misses
  23047129 ±  3%     -35.8%   14787249        perf-stat.i.iTLB-loads
 8.356e+10            -3.9%  8.027e+10        perf-stat.i.instructions
      1548           +47.8%       2288 ±  7%  perf-stat.i.instructions-per-iTLB-miss
      0.18            -4.4%       0.17        perf-stat.i.ipc
    869.08            +4.5%     908.26        perf-stat.i.metric.K/sec
    384.93            -2.7%     374.53        perf-stat.i.metric.M/sec
      7130            +3.7%       7398 ±  3%  perf-stat.i.minor-faults
     66.31            -6.3       60.05 ±  2%  perf-stat.i.node-load-miss-rate%
  39678105            -3.8%   38161927        perf-stat.i.node-load-misses
  20549470           +25.2%   25719712 ±  5%  perf-stat.i.node-loads
     43.76            -6.0       37.72 ±  5%  perf-stat.i.node-store-miss-rate%
  15613166 ±  2%     +10.0%   17180670 ±  4%  perf-stat.i.node-store-misses
  20301635           +40.5%   28518438 ±  4%  perf-stat.i.node-stores
      7172            +3.8%       7443 ±  3%  perf-stat.i.page-faults
      0.79            -0.2        0.62 ±  2%  perf-stat.overall.branch-miss-rate%
     28.13            +0.9       29.08        perf-stat.overall.cache-miss-rate%
      5.65            +2.5%       5.79        perf-stat.overall.cpi
      1942            -2.3%       1899        perf-stat.overall.cycles-between-cache-misses
      0.07 ±  5%      -0.0        0.06 ±  5%  perf-stat.overall.dTLB-load-miss-rate%
      0.03 ± 11%      +0.0        0.04 ±  6%  perf-stat.overall.dTLB-store-miss-rate%
      1500           +49.7%       2246 ±  7%  perf-stat.overall.instructions-per-iTLB-miss
      0.18            -2.4%       0.17        perf-stat.overall.ipc
     64.79            -5.9       58.87        perf-stat.overall.node-load-miss-rate%
     40.92            -5.0       35.89 ±  3%  perf-stat.overall.node-store-miss-rate%
 1.641e+10            -3.5%  1.584e+10        perf-stat.ps.branch-instructions
 1.288e+08           -23.6%   98487415 ±  2%  perf-stat.ps.branch-misses
   2337457 ±  3%     -38.5%    1437257        perf-stat.ps.context-switches
    130209 ±  2%     -13.6%     112490        perf-stat.ps.cpu-migrations
  16228057 ±  4%     -11.2%   14403294 ±  3%  perf-stat.ps.dTLB-load-misses
   3947999 ± 10%     +31.5%    5191308 ±  5%  perf-stat.ps.dTLB-store-misses
  54846930           -34.4%   35981347 ±  8%  perf-stat.ps.iTLB-load-misses
  22682918 ±  2%     -35.8%   14572556        perf-stat.ps.iTLB-loads
  8.23e+10            -2.4%  8.035e+10        perf-stat.ps.instructions
     20.85 ±  3%     +15.9%      24.16 ±  4%  perf-stat.ps.major-faults
      5283           +10.5%       5839        perf-stat.ps.minor-faults
  39315654            -4.2%   37669399        perf-stat.ps.node-load-misses
  21363970           +23.3%   26334557 ±  4%  perf-stat.ps.node-loads
  14602656           +11.5%   16280558 ±  2%  perf-stat.ps.node-store-misses
  21089334           +38.0%   29102191 ±  3%  perf-stat.ps.node-stores
      5304           +10.5%       5863        perf-stat.ps.page-faults
  4.56e+13           -12.0%  4.011e+13        perf-stat.total.instructions
      0.14 ±149%   +6903.8%      10.10 ± 90%  perf-sched.sch_delay.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault.[unknown]
    146.03 ± 20%     -59.7%      58.91 ± 32%  perf-sched.sch_delay.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
     20.00 ± 16%     -39.8%      12.05 ± 15%  perf-sched.sch_delay.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
     45.20 ± 21%     -45.1%      24.80 ± 17%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.__kmalloc_node_track_caller.kmalloc_reserve.__alloc_skb
    168.56 ± 19%     -77.4%      38.03 ± 34%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.aa_sk_perm.security_socket_sendmsg.sock_sendmsg
     87.74 ± 39%     -69.7%      26.57 ± 32%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc_node.__alloc_skb.alloc_skb_with_frags
    142.77 ± 20%     -39.2%      86.82 ± 21%  perf-sched.sch_delay.avg.ms.schedule_timeout.sock_alloc_send_pskb.unix_stream_sendmsg.sock_sendmsg
     22.06 ± 12%     -51.5%      10.70 ± 19%  perf-sched.sch_delay.avg.ms.schedule_timeout.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter
      6.08 ±146%  +12273.2%     752.85 ± 96%  perf-sched.sch_delay.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault.[unknown]
     22236 ± 14%     -62.5%       8341 ± 29%  perf-sched.sch_delay.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
     26048 ±  9%     -52.9%      12272 ± 28%  perf-sched.sch_delay.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
     24687 ±  8%     -61.5%       9499 ± 27%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.__kmalloc_node_track_caller.kmalloc_reserve.__alloc_skb
     24838 ±  8%     -70.9%       7234 ± 34%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.aa_sk_perm.security_socket_sendmsg.sock_sendmsg
     21611 ± 13%     -65.7%       7403 ± 44%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc_node.__alloc_skb.alloc_skb_with_frags
     26017 ± 10%     -53.3%      12149 ± 32%  perf-sched.sch_delay.max.ms.schedule_timeout.sock_alloc_send_pskb.unix_stream_sendmsg.sock_sendmsg
     26199 ±  9%     -57.5%      11127 ± 28%  perf-sched.sch_delay.max.ms.schedule_timeout.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter
     26.44 ± 13%     -36.5%      16.79 ± 16%  perf-sched.total_sch_delay.average.ms
     26690 ± 10%     -52.4%      12702 ± 29%  perf-sched.total_sch_delay.max.ms
     92.56 ± 13%     -34.8%      60.38 ± 16%  perf-sched.total_wait_and_delay.average.ms
     53236 ±  9%     -51.9%      25601 ± 30%  perf-sched.total_wait_and_delay.max.ms
     66.12 ± 13%     -34.1%      43.58 ± 17%  perf-sched.total_wait_time.average.ms
     26766 ±  8%     -47.8%      13965 ± 29%  perf-sched.total_wait_time.max.ms
    549.67 ± 18%     -64.8%     193.37 ± 28%  perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
     75.13 ± 14%     -41.6%      43.84 ± 14%  perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
    167.49 ± 17%     -45.5%      91.23 ± 16%  perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.__kmalloc_node_track_caller.kmalloc_reserve.__alloc_skb
    594.39 ± 14%     -76.9%     137.13 ± 29%  perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.aa_sk_perm.security_socket_sendmsg.sock_sendmsg
    317.87 ± 21%     -65.9%     108.34 ± 28%  perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc_node.__alloc_skb.alloc_skb_with_frags
      1953 ±115%     -90.1%     193.28 ±107%  perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.rmap_walk_anon.remove_migration_ptes.migrate_pages
    154.92 ± 33%     -82.5%      27.16 ±152%  perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.wait_for_completion.affine_move_task.__set_cpus_allowed_ptr_locked
      7364 ± 51%     -83.3%       1228 ±110%  perf-sched.wait_and_delay.avg.ms.rwsem_down_read_slowpath.rmap_walk_anon.remove_migration_ptes.migrate_pages
    532.84 ± 22%     -42.2%     308.22 ± 23%  perf-sched.wait_and_delay.avg.ms.schedule_timeout.sock_alloc_send_pskb.unix_stream_sendmsg.sock_sendmsg
     73.20 ± 12%     -48.1%      37.98 ± 19%  perf-sched.wait_and_delay.avg.ms.schedule_timeout.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter
    131.17 ± 12%     +54.1%     202.17 ± 14%  perf-sched.wait_and_delay.count.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
      1020 ± 16%     +30.8%       1335 ± 15%  perf-sched.wait_and_delay.count.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
     12199 ± 15%     +39.9%      17063 ± 19%  perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.__kmalloc_node_track_caller.kmalloc_reserve.__alloc_skb
      1423 ± 12%     +44.5%       2056 ± 17%  perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.kmem_cache_alloc_node.__alloc_skb.alloc_skb_with_frags
      6.00 ± 19%     +88.9%      11.33 ± 15%  perf-sched.wait_and_delay.count.schedule_timeout.kcompactd.kthread.ret_from_fork
     48952 ± 19%    +143.5%     119182 ± 12%  perf-sched.wait_and_delay.count.schedule_timeout.sock_alloc_send_pskb.unix_stream_sendmsg.sock_sendmsg
    240.00 ± 17%     +52.6%     366.17 ± 18%  perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork
    119.67 ± 28%     +60.3%     191.83 ± 17%  perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork
     45862 ± 11%     -63.5%      16731 ± 29%  perf-sched.wait_and_delay.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
     52359 ±  9%     -52.3%      24957 ± 30%  perf-sched.wait_and_delay.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
     49522 ±  8%     -61.6%      19021 ± 27%  perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.__kmalloc_node_track_caller.kmalloc_reserve.__alloc_skb
     49837 ±  8%     -70.5%      14699 ± 33%  perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.aa_sk_perm.security_socket_sendmsg.sock_sendmsg
     43808 ± 12%     -66.0%      14910 ± 44%  perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc_node.__alloc_skb.alloc_skb_with_frags
     36717 ± 23%     -82.8%       6317 ±122%  perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.rmap_walk_anon.remove_migration_ptes.migrate_pages
     31347 ± 40%     -75.5%       7689 ±112%  perf-sched.wait_and_delay.max.ms.rwsem_down_read_slowpath.rmap_walk_anon.remove_migration_ptes.migrate_pages
     52076 ± 10%     -52.5%      24745 ± 34%  perf-sched.wait_and_delay.max.ms.schedule_timeout.sock_alloc_send_pskb.unix_stream_sendmsg.sock_sendmsg
     52504 ±  9%     -57.3%      22437 ± 28%  perf-sched.wait_and_delay.max.ms.schedule_timeout.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter
      1.82 ±163%   +2252.9%      42.73 ±129%  perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault.[unknown]
    403.64 ± 24%     -66.7%     134.46 ± 28%  perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
     55.12 ± 14%     -42.3%      31.79 ± 14%  perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.02 ±108%    +534.1%       0.13 ± 85%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.__alloc_pages.alloc_pages_vma.do_anonymous_page
    122.29 ± 16%     -45.7%      66.43 ± 16%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.__kmalloc_node_track_caller.kmalloc_reserve.__alloc_skb
    425.82 ± 16%     -76.7%      99.10 ± 28%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.aa_sk_perm.security_socket_sendmsg.sock_sendmsg
    465.09 ±223%    -100.0%       0.04 ±117%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.change_p4d_range.change_protection.change_prot_numa
      9.43 ±125%    +219.2%      30.11 ±104%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.generic_perform_write.__generic_file_write_iter.generic_file_write_iter
    230.14 ± 17%     -64.5%      81.77 ± 28%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc_node.__alloc_skb.alloc_skb_with_frags
     70.21 ± 31%     -56.3%      30.65 ± 43%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.mutex_lock.unix_stream_read_generic.unix_stream_recvmsg
      1362 ±123%     -88.1%     161.89 ± 95%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.rmap_walk_anon.remove_migration_ptes.migrate_pages
      0.17 ±118%   +1194.7%       2.26 ± 80%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.sched_exec.bprm_execve
    154.92 ± 33%     -64.9%      54.45 ± 45%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.wait_for_completion.affine_move_task.__set_cpus_allowed_ptr_locked
      5277 ± 37%     -81.9%     953.75 ±103%  perf-sched.wait_time.avg.ms.rwsem_down_read_slowpath.rmap_walk_anon.remove_migration_ptes.migrate_pages
    390.06 ± 23%     -43.2%     221.40 ± 23%  perf-sched.wait_time.avg.ms.schedule_timeout.sock_alloc_send_pskb.unix_stream_sendmsg.sock_sendmsg
     51.14 ± 12%     -46.7%      27.27 ± 19%  perf-sched.wait_time.avg.ms.schedule_timeout.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter
     42.41 ±143%   +7008.4%       3014 ±126%  perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault.[unknown]
     17958 ± 24%     -60.2%       7144 ± 33%  perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
     24588 ±  7%     -65.6%       8461 ± 28%  perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
     26488 ±  8%     -51.3%      12906 ± 31%  perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
     20113 ± 16%     -45.9%      10885 ± 38%  perf-sched.wait_time.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
     25666 ±  9%     -57.7%      10860 ± 33%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.__kmalloc_node_track_caller.kmalloc_reserve.__alloc_skb
     19730 ± 11%     -59.6%       7972 ± 23%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.aa_sk_perm.security_socket_recvmsg.sock_recvmsg
     25151 ±  8%     -64.3%       8977 ± 31%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.aa_sk_perm.security_socket_sendmsg.sock_sendmsg
      1860 ±223%    -100.0%       0.06 ±105%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.change_p4d_range.change_protection.change_prot_numa
      4.30 ±194%     -97.7%       0.10 ±116%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.down_write_killable.vm_mmap_pgoff.elf_map
    320.22 ±175%   +1509.4%       5153 ± 71%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.generic_perform_write.__generic_file_write_iter.generic_file_write_iter
     24087 ± 12%     -61.8%       9191 ± 42%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc_node.__alloc_skb.alloc_skb_with_frags
     17911 ± 16%     -62.7%       6687 ± 29%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.mutex_lock.unix_stream_read_generic.unix_stream_recvmsg
     20724 ± 17%     -79.9%       4168 ± 82%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.rmap_walk_anon.remove_migration_ptes.migrate_pages
      1.20 ±151%   +2217.5%      27.84 ± 75%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.sched_exec.bprm_execve
     19696 ± 23%     -77.1%       4500 ± 99%  perf-sched.wait_time.max.ms.rwsem_down_read_slowpath.rmap_walk_anon.remove_migration_ptes.migrate_pages
     26233 ± 10%     -51.2%      12801 ± 36%  perf-sched.wait_time.max.ms.schedule_timeout.sock_alloc_send_pskb.unix_stream_sendmsg.sock_sendmsg
     26305 ±  9%     -56.7%      11393 ± 28%  perf-sched.wait_time.max.ms.schedule_timeout.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter
     24961 ± 11%     -45.5%      13607 ± 28%  perf-sched.wait_time.max.ms.smpboot_thread_fn.kthread.ret_from_fork
     25452 ± 10%     -47.3%      13401 ± 30%  perf-sched.wait_time.max.ms.worker_thread.kthread.ret_from_fork
     84420 ±  6%     -11.5%      74716 ±  8%  softirqs.CPU0.RCU
     86481 ±  5%     -11.8%      76240 ±  9%  softirqs.CPU1.RCU
     87185 ±  5%     -13.1%      75796 ±  9%  softirqs.CPU10.RCU
     92504 ±  6%     -14.8%      78845 ±  7%  softirqs.CPU100.RCU
     92105 ±  6%     -15.2%      78097 ±  7%  softirqs.CPU101.RCU
     92008 ±  6%     -14.9%      78278 ±  7%  softirqs.CPU102.RCU
     92262 ±  6%     -15.3%      78190 ±  6%  softirqs.CPU103.RCU
     92609 ±  6%     -14.8%      78921 ±  7%  softirqs.CPU104.RCU
     92568 ±  6%     -15.0%      78674 ±  7%  softirqs.CPU105.RCU
     93113 ±  6%     -15.2%      78921 ±  7%  softirqs.CPU106.RCU
     94050 ±  6%     -15.6%      79354 ±  8%  softirqs.CPU107.RCU
     86336 ±  6%     -12.2%      75835 ±  7%  softirqs.CPU108.RCU
     85713 ±  6%     -12.9%      74646 ±  9%  softirqs.CPU109.RCU
     85964 ±  5%     -12.0%      75619 ±  9%  softirqs.CPU11.RCU
     85393 ±  7%     -12.6%      74646 ±  9%  softirqs.CPU110.RCU
     85002 ±  6%     -12.3%      74549 ±  8%  softirqs.CPU111.RCU
     95689 ±  6%     -14.9%      81465 ± 10%  softirqs.CPU112.RCU
     95426 ±  6%     -13.5%      82531 ±  8%  softirqs.CPU113.RCU
     93843 ±  6%     -13.2%      81424 ±  9%  softirqs.CPU114.RCU
     94782 ±  6%     -13.7%      81765 ±  8%  softirqs.CPU115.RCU
     95329 ±  6%     -13.2%      82770 ±  9%  softirqs.CPU116.RCU
     95098 ±  6%     -13.3%      82488 ±  9%  softirqs.CPU117.RCU
     94061 ±  6%     -13.4%      81501 ±  8%  softirqs.CPU118.RCU
     94978 ±  6%     -13.7%      81931 ±  9%  softirqs.CPU119.RCU
     85245 ±  6%     -10.5%      76276 ±  9%  softirqs.CPU12.RCU
     94735 ±  6%     -13.3%      82148 ±  9%  softirqs.CPU120.RCU
     95239 ±  6%     -13.3%      82558 ±  9%  softirqs.CPU121.RCU
     94890 ±  6%     -13.5%      82113 ±  9%  softirqs.CPU122.RCU
     94265 ±  6%     -13.5%      81563 ±  8%  softirqs.CPU123.RCU
     95082 ±  6%     -14.6%      81160 ± 10%  softirqs.CPU124.RCU
     95351 ±  6%     -13.6%      82390 ±  9%  softirqs.CPU125.RCU
     81884 ±  5%     -11.7%      72288 ±  8%  softirqs.CPU126.RCU
     81461 ±  6%     -12.1%      71593 ±  8%  softirqs.CPU127.RCU
     88358 ±  5%     -13.9%      76053 ±  9%  softirqs.CPU128.RCU
     88086 ±  5%     -13.4%      76318 ±  8%  softirqs.CPU129.RCU
     14400            +9.9%      15821 ±  3%  softirqs.CPU13.SCHED
     88026 ±  5%     -13.4%      76206 ±  8%  softirqs.CPU130.RCU
     88351 ±  5%     -13.7%      76205 ±  9%  softirqs.CPU131.RCU
     87490 ±  5%     -13.3%      75876 ±  9%  softirqs.CPU132.RCU
     87529 ±  5%     -13.6%      75668 ±  9%  softirqs.CPU133.RCU
     87798 ±  5%     -12.5%      76813 ±  9%  softirqs.CPU134.RCU
     88136 ±  5%     -13.5%      76276 ±  9%  softirqs.CPU135.RCU
     88294 ±  5%     -13.6%      76251 ±  9%  softirqs.CPU136.RCU
     87208 ±  5%     -13.5%      75465 ±  9%  softirqs.CPU137.RCU
     87695 ±  5%     -13.3%      75994 ±  9%  softirqs.CPU138.RCU
     88320 ±  5%     -13.6%      76309 ±  9%  softirqs.CPU139.RCU
     86555 ±  6%     -12.3%      75887 ±  8%  softirqs.CPU14.RCU
     14298            +7.8%      15412 ±  4%  softirqs.CPU14.SCHED
     88184 ±  5%     -13.7%      76139 ±  9%  softirqs.CPU140.RCU
     87603 ±  5%     -13.1%      76136 ±  9%  softirqs.CPU141.RCU
     87625 ±  5%     -13.3%      75972 ±  9%  softirqs.CPU142.RCU
     88256 ±  5%     -13.3%      76557 ±  9%  softirqs.CPU143.RCU
     87050 ±  5%     -12.2%      76430 ±  8%  softirqs.CPU15.RCU
     81130 ±  6%     -12.1%      71332 ±  8%  softirqs.CPU16.RCU
     82146 ±  6%     -12.5%      71850 ±  9%  softirqs.CPU17.RCU
     14082           +10.2%      15524 ±  3%  softirqs.CPU17.SCHED
     93814 ±  5%     -15.8%      79008 ±  7%  softirqs.CPU18.RCU
     92946 ±  5%     -15.5%      78562 ±  7%  softirqs.CPU19.RCU
     86506 ±  5%      -9.5%      78315 ±  6%  softirqs.CPU2.RCU
     92407 ±  5%     -15.2%      78322 ±  7%  softirqs.CPU20.RCU
     92212 ±  6%     -15.7%      77774 ±  7%  softirqs.CPU21.RCU
     92309 ±  6%     -15.1%      78328 ±  7%  softirqs.CPU22.RCU
     92929 ±  5%     -15.4%      78577 ±  7%  softirqs.CPU23.RCU
     93052 ±  6%     -15.7%      78470 ±  7%  softirqs.CPU24.RCU
     92080 ±  6%     -15.3%      77996 ±  7%  softirqs.CPU25.RCU
     91828 ±  5%     -15.3%      77740 ±  7%  softirqs.CPU26.RCU
     92903 ±  5%     -15.6%      78432 ±  7%  softirqs.CPU27.RCU
     92628 ±  6%     -15.3%      78450 ±  7%  softirqs.CPU28.RCU
     92249 ±  6%     -16.2%      77285 ±  8%  softirqs.CPU29.RCU
     86877 ±  5%     -12.0%      76462 ±  8%  softirqs.CPU3.RCU
     91350 ±  6%     -14.7%      77958 ±  8%  softirqs.CPU30.RCU
     91443 ±  6%     -16.0%      76829 ±  5%  softirqs.CPU31.RCU
     86014 ±  6%     -14.2%      73818 ±  8%  softirqs.CPU32.RCU
     85380 ±  6%     -13.7%      73689 ±  8%  softirqs.CPU33.RCU
     86155 ±  5%     -14.3%      73854 ±  8%  softirqs.CPU34.RCU
     86462 ±  5%     -14.1%      74234 ±  8%  softirqs.CPU35.RCU
     92387 ±  6%     -13.1%      80287 ±  9%  softirqs.CPU36.RCU
     92117 ±  6%     -12.9%      80218 ±  8%  softirqs.CPU37.RCU
     90761 ±  8%     -11.9%      79957 ±  9%  softirqs.CPU38.RCU
     90705 ±  5%     -11.5%      80235 ±  9%  softirqs.CPU39.RCU
     86170 ±  5%     -11.2%      76515 ±  7%  softirqs.CPU4.RCU
     13627 ±  6%     +14.0%      15539 ±  3%  softirqs.CPU4.SCHED
     92368 ±  6%     -13.7%      79721 ±  9%  softirqs.CPU40.RCU
     92055 ±  6%     -13.0%      80123 ±  8%  softirqs.CPU41.RCU
     91447 ±  6%     -13.0%      79565 ±  9%  softirqs.CPU42.RCU
     91880 ±  6%     -13.2%      79751 ±  8%  softirqs.CPU43.RCU
     92254 ±  6%     -12.7%      80501 ±  8%  softirqs.CPU44.RCU
     91783 ±  6%     -12.8%      80015 ±  9%  softirqs.CPU45.RCU
     91421 ±  6%     -12.9%      79645 ±  9%  softirqs.CPU46.RCU
     91390 ±  6%     -12.8%      79722 ±  8%  softirqs.CPU47.RCU
     88932 ±  6%     -12.8%      77543 ±  8%  softirqs.CPU48.RCU
     88850 ±  6%     -12.4%      77848 ±  9%  softirqs.CPU49.RCU
     86897 ±  4%     -12.3%      76176 ±  8%  softirqs.CPU5.RCU
     88709 ±  6%     -12.5%      77631 ±  8%  softirqs.CPU50.RCU
     88614 ±  6%     -12.5%      77564 ±  8%  softirqs.CPU51.RCU
     88876 ±  6%     -13.3%      77084 ±  9%  softirqs.CPU52.RCU
     88779 ±  6%     -12.4%      77752 ±  8%  softirqs.CPU53.RCU
     89712 ±  5%     -13.5%      77589 ±  8%  softirqs.CPU54.RCU
     88569 ±  5%     -13.2%      76854 ±  8%  softirqs.CPU55.RCU
     88777 ±  5%     -13.1%      77130 ±  8%  softirqs.CPU56.RCU
     89211 ±  5%     -13.8%      76929 ±  8%  softirqs.CPU57.RCU
     88989 ±  5%     -13.0%      77400 ±  8%  softirqs.CPU58.RCU
     88940 ±  5%     -13.1%      77250 ±  8%  softirqs.CPU59.RCU
     86373 ±  5%     -12.3%      75749 ±  8%  softirqs.CPU6.RCU
     88320 ±  5%     -13.0%      76839 ±  9%  softirqs.CPU60.RCU
     88405 ±  5%     -13.2%      76706 ±  8%  softirqs.CPU61.RCU
     88722 ±  5%     -12.8%      77340 ±  8%  softirqs.CPU62.RCU
     88541 ±  5%     -12.9%      77089 ±  9%  softirqs.CPU63.RCU
     88636 ±  5%     -13.6%      76600 ±  8%  softirqs.CPU64.RCU
     88387 ±  5%     -13.4%      76579 ±  8%  softirqs.CPU65.RCU
     88139 ±  5%     -13.3%      76411 ±  8%  softirqs.CPU66.RCU
     88420 ±  5%     -13.3%      76702 ±  8%  softirqs.CPU67.RCU
     88484 ±  5%     -13.2%      76794 ±  8%  softirqs.CPU68.RCU
     88366 ±  5%     -12.7%      77154 ±  8%  softirqs.CPU69.RCU
     86334 ±  5%     -12.1%      75872 ±  7%  softirqs.CPU7.RCU
     88438 ±  5%     -13.4%      76599 ±  8%  softirqs.CPU70.RCU
     88125 ±  5%     -13.3%      76369 ±  8%  softirqs.CPU71.RCU
     87488 ±  6%     -12.1%      76899 ±  8%  softirqs.CPU72.RCU
     13664 ±  6%     +12.1%      15314 ±  3%  softirqs.CPU73.SCHED
     87669 ±  6%     -11.6%      77534 ±  7%  softirqs.CPU74.RCU
     87675 ±  5%     -11.5%      77573 ±  8%  softirqs.CPU75.RCU
     87948 ±  5%     -12.0%      77402 ±  8%  softirqs.CPU76.RCU
     88247 ±  5%     -12.2%      77474 ±  7%  softirqs.CPU77.RCU
     87612 ±  5%     -11.7%      77372 ±  8%  softirqs.CPU78.RCU
     87923 ±  6%     -12.4%      77023 ±  8%  softirqs.CPU79.RCU
     86391 ±  5%     -11.4%      76543 ±  8%  softirqs.CPU8.RCU
     88597 ±  6%     -12.0%      78003 ±  8%  softirqs.CPU80.RCU
     89101 ±  6%     -12.5%      77968 ±  8%  softirqs.CPU81.RCU
     14183 ±  2%      +8.3%      15363 ±  3%  softirqs.CPU81.SCHED
     88545 ±  6%     -12.4%      77527 ±  7%  softirqs.CPU82.RCU
     87757 ±  7%     -11.5%      77621 ±  8%  softirqs.CPU83.RCU
     88307 ±  6%     -11.5%      78128 ±  8%  softirqs.CPU84.RCU
     14211            +9.8%      15605 ±  3%  softirqs.CPU84.SCHED
     89048 ±  6%     -13.0%      77482 ±  8%  softirqs.CPU85.RCU
     89209 ±  5%     -12.3%      78258 ±  7%  softirqs.CPU86.RCU
     88691 ±  5%     -12.3%      77825 ±  8%  softirqs.CPU87.RCU
     88012 ±  5%     -11.8%      77588 ±  8%  softirqs.CPU88.RCU
     88781 ±  5%     -11.9%      78190 ±  8%  softirqs.CPU89.RCU
     86346 ±  6%     -10.6%      77225 ±  8%  softirqs.CPU9.RCU
     90192 ±  5%     -14.8%      76846 ±  7%  softirqs.CPU90.RCU
     89520 ±  6%     -14.7%      76378 ±  7%  softirqs.CPU91.RCU
     89174 ±  5%     -14.4%      76374 ±  7%  softirqs.CPU92.RCU
     88839 ±  6%     -14.4%      76030 ±  7%  softirqs.CPU93.RCU
     89065 ±  5%     -14.4%      76238 ±  7%  softirqs.CPU94.RCU
     89354 ±  6%     -14.3%      76619 ±  7%  softirqs.CPU95.RCU
     92662 ±  6%     -14.8%      78959 ±  7%  softirqs.CPU96.RCU
     91610 ±  6%     -14.8%      78044 ±  7%  softirqs.CPU97.RCU
     92230 ±  6%     -15.0%      78392 ±  7%  softirqs.CPU98.RCU
     92985 ±  6%     -14.9%      79138 ±  7%  softirqs.CPU99.RCU
  12886523 ±  6%     -13.3%   11171782 ±  8%  softirqs.RCU
    112751 ±  3%     -10.3%     101130 ±  7%  softirqs.TIMER
   6866613 ±  7%     -20.9%    5431401 ±  9%  interrupts.CAL:Function_call_interrupts
   1109865           -13.2%     963752 ±  7%  interrupts.CPU0.LOC:Local_timer_interrupts
    746403 ±  6%     -26.6%     547987 ±  6%  interrupts.CPU0.RES:Rescheduling_interrupts
   1109844           -11.5%     982545 ±  4%  interrupts.CPU1.LOC:Local_timer_interrupts
    784948 ±  6%     -26.8%     574218 ±  6%  interrupts.CPU1.RES:Rescheduling_interrupts
   1109908           -11.5%     982586 ±  4%  interrupts.CPU10.LOC:Local_timer_interrupts
    768936 ±  6%     -25.0%     576330 ±  7%  interrupts.CPU10.RES:Rescheduling_interrupts
   1109673           -11.3%     983929 ±  3%  interrupts.CPU100.LOC:Local_timer_interrupts
    933365 ±  9%     -33.5%     620758 ± 11%  interrupts.CPU100.RES:Rescheduling_interrupts
   1109648           -11.2%     985868 ±  3%  interrupts.CPU101.LOC:Local_timer_interrupts
    905179 ±  4%     -32.6%     610073 ±  8%  interrupts.CPU101.RES:Rescheduling_interrupts
     55524 ± 32%     -37.9%      34488 ± 22%  interrupts.CPU102.CAL:Function_call_interrupts
   1109644           -11.3%     983935 ±  3%  interrupts.CPU102.LOC:Local_timer_interrupts
    919161 ±  4%     -33.6%     610117 ± 10%  interrupts.CPU102.RES:Rescheduling_interrupts
   1109645           -11.3%     983934 ±  3%  interrupts.CPU103.LOC:Local_timer_interrupts
    913475 ±  5%     -33.8%     604948 ± 10%  interrupts.CPU103.RES:Rescheduling_interrupts
   1109644           -11.4%     983504 ±  3%  interrupts.CPU104.LOC:Local_timer_interrupts
    891411 ±  4%     -30.6%     618904 ± 11%  interrupts.CPU104.RES:Rescheduling_interrupts
   1109685           -11.3%     983840 ±  3%  interrupts.CPU105.LOC:Local_timer_interrupts
    916161 ±  6%     -33.4%     609904 ± 12%  interrupts.CPU105.RES:Rescheduling_interrupts
   1109651           -11.4%     983590 ±  3%  interrupts.CPU106.LOC:Local_timer_interrupts
    940033 ±  4%     -36.6%     596222 ±  8%  interrupts.CPU106.RES:Rescheduling_interrupts
     45963 ± 35%     -31.8%      31357 ± 20%  interrupts.CPU107.CAL:Function_call_interrupts
   1109713           -11.3%     983779 ±  3%  interrupts.CPU107.LOC:Local_timer_interrupts
    934019 ± 10%     -35.0%     606683 ±  9%  interrupts.CPU107.RES:Rescheduling_interrupts
   1109681           -11.2%     985738 ±  3%  interrupts.CPU108.LOC:Local_timer_interrupts
    907098 ± 10%     -26.3%     668850 ±  4%  interrupts.CPU108.RES:Rescheduling_interrupts
   1109685           -11.2%     985797 ±  3%  interrupts.CPU109.LOC:Local_timer_interrupts
    926079 ± 10%     -27.0%     675677 ±  2%  interrupts.CPU109.RES:Rescheduling_interrupts
   1109842           -11.5%     982553 ±  3%  interrupts.CPU11.LOC:Local_timer_interrupts
    777408 ±  7%     -24.9%     584102 ±  6%  interrupts.CPU11.RES:Rescheduling_interrupts
   1109698           -11.2%     985930 ±  3%  interrupts.CPU110.LOC:Local_timer_interrupts
    927794 ±  6%     -26.1%     685774 ±  3%  interrupts.CPU110.RES:Rescheduling_interrupts
   1109662           -11.1%     986214 ±  3%  interrupts.CPU111.LOC:Local_timer_interrupts
    910204 ±  4%     -26.3%     670493 ±  3%  interrupts.CPU111.RES:Rescheduling_interrupts
   1109686           -11.3%     984562 ±  3%  interrupts.CPU112.LOC:Local_timer_interrupts
    956972 ±  6%     -29.5%     674228 ±  3%  interrupts.CPU112.RES:Rescheduling_interrupts
   1109660           -11.1%     986024 ±  3%  interrupts.CPU113.LOC:Local_timer_interrupts
    911937 ±  6%     -25.3%     680865 ±  3%  interrupts.CPU113.RES:Rescheduling_interrupts
   1109640           -11.1%     986264 ±  3%  interrupts.CPU114.LOC:Local_timer_interrupts
    945548 ± 10%     -27.3%     687639 ±  2%  interrupts.CPU114.RES:Rescheduling_interrupts
   1109707           -11.1%     986350 ±  3%  interrupts.CPU115.LOC:Local_timer_interrupts
    965953 ±  5%     -28.4%     692092 ±  4%  interrupts.CPU115.RES:Rescheduling_interrupts
   1109635           -11.1%     985969 ±  3%  interrupts.CPU116.LOC:Local_timer_interrupts
    961063 ±  9%     -30.4%     669281 ±  3%  interrupts.CPU116.RES:Rescheduling_interrupts
   1109648           -11.1%     986070 ±  3%  interrupts.CPU117.LOC:Local_timer_interrupts
    953449 ±  5%     -29.3%     674189 ±  4%  interrupts.CPU117.RES:Rescheduling_interrupts
   1109688           -11.1%     986329 ±  3%  interrupts.CPU118.LOC:Local_timer_interrupts
    909710 ±  9%     -22.2%     707837 ±  2%  interrupts.CPU118.RES:Rescheduling_interrupts
   1109682           -11.1%     986282 ±  3%  interrupts.CPU119.LOC:Local_timer_interrupts
    951262 ±  7%     -27.6%     688722 ±  4%  interrupts.CPU119.RES:Rescheduling_interrupts
   1109883           -11.5%     982267 ±  4%  interrupts.CPU12.LOC:Local_timer_interrupts
    787531 ±  3%     -27.2%     573213 ±  7%  interrupts.CPU12.RES:Rescheduling_interrupts
   1109698           -11.1%     986222 ±  3%  interrupts.CPU120.LOC:Local_timer_interrupts
    943666 ±  5%     -28.5%     674336 ±  4%  interrupts.CPU120.RES:Rescheduling_interrupts
   1109696           -11.1%     986248 ±  3%  interrupts.CPU121.LOC:Local_timer_interrupts
    912017 ±  7%     -24.5%     688310        interrupts.CPU121.RES:Rescheduling_interrupts
   1109688           -11.1%     986359 ±  3%  interrupts.CPU122.LOC:Local_timer_interrupts
    962703 ±  8%     -29.7%     676523 ±  2%  interrupts.CPU122.RES:Rescheduling_interrupts
   1109653           -11.1%     986405 ±  3%  interrupts.CPU123.LOC:Local_timer_interrupts
    949945 ±  6%     -25.6%     706314 ±  4%  interrupts.CPU123.RES:Rescheduling_interrupts
   1109693           -11.1%     986142 ±  3%  interrupts.CPU124.LOC:Local_timer_interrupts
    926407 ±  8%     -25.0%     695173 ±  3%  interrupts.CPU124.RES:Rescheduling_interrupts
   1109690           -11.1%     986479 ±  3%  interrupts.CPU125.LOC:Local_timer_interrupts
    897337 ±  8%     -24.5%     677912 ±  4%  interrupts.CPU125.RES:Rescheduling_interrupts
   1109701           -11.2%     985053 ±  3%  interrupts.CPU126.LOC:Local_timer_interrupts
    870200 ±  5%     -27.7%     629561 ± 12%  interrupts.CPU126.RES:Rescheduling_interrupts
   1109638           -11.2%     985218 ±  3%  interrupts.CPU127.LOC:Local_timer_interrupts
    903216 ±  5%     -26.2%     667011 ± 11%  interrupts.CPU127.RES:Rescheduling_interrupts
   1109678           -11.2%     985554 ±  3%  interrupts.CPU128.LOC:Local_timer_interrupts
    898010 ±  3%     -27.7%     649239 ± 10%  interrupts.CPU128.RES:Rescheduling_interrupts
   1109640           -11.2%     985393 ±  3%  interrupts.CPU129.LOC:Local_timer_interrupts
    914045 ±  5%     -27.6%     661981 ± 13%  interrupts.CPU129.RES:Rescheduling_interrupts
   1109756           -11.5%     982011 ±  4%  interrupts.CPU13.LOC:Local_timer_interrupts
    770920 ±  8%     -26.9%     563555 ±  7%  interrupts.CPU13.RES:Rescheduling_interrupts
   1109642           -11.2%     985483 ±  3%  interrupts.CPU130.LOC:Local_timer_interrupts
    902719 ±  3%     -27.4%     655719 ± 12%  interrupts.CPU130.RES:Rescheduling_interrupts
   1109678           -11.2%     985411 ±  3%  interrupts.CPU131.LOC:Local_timer_interrupts
    927667 ±  2%     -29.9%     650000 ± 12%  interrupts.CPU131.RES:Rescheduling_interrupts
   1109655           -11.2%     985585 ±  3%  interrupts.CPU132.LOC:Local_timer_interrupts
    895668 ±  4%     -26.7%     656131 ±  9%  interrupts.CPU132.RES:Rescheduling_interrupts
   1109644           -11.2%     985607 ±  3%  interrupts.CPU133.LOC:Local_timer_interrupts
    932719 ±  8%     -28.6%     665823 ± 11%  interrupts.CPU133.RES:Rescheduling_interrupts
   1109654           -11.2%     985756 ±  3%  interrupts.CPU134.LOC:Local_timer_interrupts
    919080 ±  4%     -28.9%     653480 ± 10%  interrupts.CPU134.RES:Rescheduling_interrupts
   1109612           -11.2%     985559 ±  3%  interrupts.CPU135.LOC:Local_timer_interrupts
    919061 ±  6%     -29.3%     649968 ± 10%  interrupts.CPU135.RES:Rescheduling_interrupts
   1109671           -11.2%     985352 ±  3%  interrupts.CPU136.LOC:Local_timer_interrupts
    942893 ±  4%     -30.7%     653873 ± 10%  interrupts.CPU136.RES:Rescheduling_interrupts
   1109638           -11.2%     985586 ±  3%  interrupts.CPU137.LOC:Local_timer_interrupts
    909483 ±  4%     -26.2%     671290 ± 12%  interrupts.CPU137.RES:Rescheduling_interrupts
   1109667           -11.2%     985336 ±  3%  interrupts.CPU138.LOC:Local_timer_interrupts
    905979           -24.9%     680152 ± 12%  interrupts.CPU138.RES:Rescheduling_interrupts
   1109666           -11.2%     985457 ±  3%  interrupts.CPU139.LOC:Local_timer_interrupts
    914012 ±  2%     -30.0%     639753 ± 11%  interrupts.CPU139.RES:Rescheduling_interrupts
   1109769           -11.5%     982477 ±  4%  interrupts.CPU14.LOC:Local_timer_interrupts
    786901 ±  6%     -26.1%     581705 ±  5%  interrupts.CPU14.RES:Rescheduling_interrupts
   1109661           -11.2%     985582 ±  3%  interrupts.CPU140.LOC:Local_timer_interrupts
    891457 ±  4%     -25.1%     667637 ±  9%  interrupts.CPU140.RES:Rescheduling_interrupts
   1109567           -11.2%     985643 ±  3%  interrupts.CPU141.LOC:Local_timer_interrupts
    920671 ±  4%     -29.1%     652448 ± 10%  interrupts.CPU141.RES:Rescheduling_interrupts
   1109583           -11.2%     985732 ±  3%  interrupts.CPU142.LOC:Local_timer_interrupts
    901948 ±  3%     -25.7%     670496 ± 11%  interrupts.CPU142.RES:Rescheduling_interrupts
   1109693           -11.2%     985489 ±  3%  interrupts.CPU143.LOC:Local_timer_interrupts
    897848 ±  3%     -29.8%     630227 ± 11%  interrupts.CPU143.RES:Rescheduling_interrupts
   1109764           -11.5%     982288 ±  4%  interrupts.CPU15.LOC:Local_timer_interrupts
    769032 ±  4%     -24.6%     579800 ±  5%  interrupts.CPU15.RES:Rescheduling_interrupts
   1109749           -11.5%     981954 ±  4%  interrupts.CPU16.LOC:Local_timer_interrupts
    753284 ±  7%     -24.5%     568931 ±  7%  interrupts.CPU16.RES:Rescheduling_interrupts
   1109740           -11.5%     982067 ±  4%  interrupts.CPU17.LOC:Local_timer_interrupts
    762912 ±  3%     -24.3%     577864 ±  6%  interrupts.CPU17.RES:Rescheduling_interrupts
   1109715           -11.3%     983766 ±  3%  interrupts.CPU18.LOC:Local_timer_interrupts
    862584 ±  9%     -31.5%     591294 ±  8%  interrupts.CPU18.RES:Rescheduling_interrupts
   1109709           -11.4%     983571 ±  3%  interrupts.CPU19.LOC:Local_timer_interrupts
    914512 ±  5%     -33.8%     605096 ±  9%  interrupts.CPU19.RES:Rescheduling_interrupts
   1109849           -11.5%     982602 ±  4%  interrupts.CPU2.LOC:Local_timer_interrupts
    763288 ±  2%     -24.2%     578849 ±  8%  interrupts.CPU2.RES:Rescheduling_interrupts
   1109697           -11.3%     983869 ±  3%  interrupts.CPU20.LOC:Local_timer_interrupts
    935470 ±  7%     -33.8%     619417 ±  8%  interrupts.CPU20.RES:Rescheduling_interrupts
   1109689           -11.3%     983811 ±  3%  interrupts.CPU21.LOC:Local_timer_interrupts
    953999 ±  5%     -35.9%     611854 ±  9%  interrupts.CPU21.RES:Rescheduling_interrupts
   1109713           -11.3%     983859 ±  3%  interrupts.CPU22.LOC:Local_timer_interrupts
    949063 ±  4%     -34.9%     617867 ± 10%  interrupts.CPU22.RES:Rescheduling_interrupts
   1109677           -11.3%     983782 ±  3%  interrupts.CPU23.LOC:Local_timer_interrupts
    916942 ±  2%     -31.5%     627663 ±  9%  interrupts.CPU23.RES:Rescheduling_interrupts
   1109676           -11.3%     983778 ±  3%  interrupts.CPU24.LOC:Local_timer_interrupts
    946944 ±  7%     -34.4%     621109 ±  9%  interrupts.CPU24.RES:Rescheduling_interrupts
   1109671           -11.3%     983788 ±  3%  interrupts.CPU25.LOC:Local_timer_interrupts
    927026 ±  3%     -32.5%     625454 ± 10%  interrupts.CPU25.RES:Rescheduling_interrupts
   1109686           -11.3%     983799 ±  3%  interrupts.CPU26.LOC:Local_timer_interrupts
    925930 ±  4%     -31.8%     631366 ±  9%  interrupts.CPU26.RES:Rescheduling_interrupts
   1109675           -11.3%     983850 ±  3%  interrupts.CPU27.LOC:Local_timer_interrupts
    930402 ±  7%     -32.6%     627189 ±  8%  interrupts.CPU27.RES:Rescheduling_interrupts
   1109729           -11.4%     983703 ±  3%  interrupts.CPU28.LOC:Local_timer_interrupts
    928439 ±  4%     -32.0%     630948 ±  7%  interrupts.CPU28.RES:Rescheduling_interrupts
   1109731           -11.9%     977654 ±  4%  interrupts.CPU29.LOC:Local_timer_interrupts
    917894 ±  3%     -31.6%     627431 ±  7%  interrupts.CPU29.RES:Rescheduling_interrupts
   1109813           -11.5%     982425 ±  4%  interrupts.CPU3.LOC:Local_timer_interrupts
    763728 ±  4%     -25.8%     566752 ±  7%  interrupts.CPU3.RES:Rescheduling_interrupts
   1109702           -11.4%     983710 ±  3%  interrupts.CPU30.LOC:Local_timer_interrupts
    959814 ±  6%     -34.4%     630064 ± 11%  interrupts.CPU30.RES:Rescheduling_interrupts
   1109691           -11.3%     983822 ±  3%  interrupts.CPU31.LOC:Local_timer_interrupts
    953721 ±  7%     -34.2%     627837 ± 12%  interrupts.CPU31.RES:Rescheduling_interrupts
   1109716           -11.4%     983559 ±  3%  interrupts.CPU32.LOC:Local_timer_interrupts
    929251 ±  5%     -32.4%     628141 ± 10%  interrupts.CPU32.RES:Rescheduling_interrupts
   1109720           -11.4%     983527 ±  3%  interrupts.CPU33.LOC:Local_timer_interrupts
    955082 ±  8%     -34.0%     629974 ±  9%  interrupts.CPU33.RES:Rescheduling_interrupts
   1109706           -11.4%     983432 ±  3%  interrupts.CPU34.LOC:Local_timer_interrupts
    933432 ±  6%     -33.6%     619604 ±  9%  interrupts.CPU34.RES:Rescheduling_interrupts
   1109724           -11.4%     983426 ±  3%  interrupts.CPU35.LOC:Local_timer_interrupts
    904660 ±  3%     -31.4%     620967 ±  9%  interrupts.CPU35.RES:Rescheduling_interrupts
   1109716           -11.1%     986109 ±  3%  interrupts.CPU36.LOC:Local_timer_interrupts
    908916 ±  7%     -25.5%     677428 ±  2%  interrupts.CPU36.RES:Rescheduling_interrupts
   1109635           -11.1%     986076 ±  3%  interrupts.CPU37.LOC:Local_timer_interrupts
    926313 ±  8%     -24.1%     702863 ±  3%  interrupts.CPU37.RES:Rescheduling_interrupts
   1109685           -11.1%     986009 ±  3%  interrupts.CPU38.LOC:Local_timer_interrupts
    951056 ±  6%     -27.3%     691529 ±  4%  interrupts.CPU38.RES:Rescheduling_interrupts
   1109743           -11.1%     986389 ±  3%  interrupts.CPU39.LOC:Local_timer_interrupts
    927813 ±  7%     -24.0%     705228 ±  4%  interrupts.CPU39.RES:Rescheduling_interrupts
   1109737           -11.5%     982501 ±  3%  interrupts.CPU4.LOC:Local_timer_interrupts
    766534 ±  8%     -24.7%     577507 ±  6%  interrupts.CPU4.RES:Rescheduling_interrupts
   1109783           -11.1%     986790 ±  3%  interrupts.CPU40.LOC:Local_timer_interrupts
    929300 ±  3%     -24.5%     702028 ±  3%  interrupts.CPU40.RES:Rescheduling_interrupts
   1109714           -11.2%     985893 ±  3%  interrupts.CPU41.LOC:Local_timer_interrupts
    952762 ±  9%     -26.6%     698904 ±  3%  interrupts.CPU41.RES:Rescheduling_interrupts
   1109708           -11.1%     986368 ±  3%  interrupts.CPU42.LOC:Local_timer_interrupts
    961704 ±  6%     -26.4%     707713 ±  3%  interrupts.CPU42.RES:Rescheduling_interrupts
   1109757           -11.1%     986228 ±  3%  interrupts.CPU43.LOC:Local_timer_interrupts
    952246 ±  6%     -26.2%     702405 ±  2%  interrupts.CPU43.RES:Rescheduling_interrupts
   1109751           -11.1%     986140 ±  3%  interrupts.CPU44.LOC:Local_timer_interrupts
    980627 ±  2%     -29.8%     688585 ±  4%  interrupts.CPU44.RES:Rescheduling_interrupts
   1109689           -11.1%     985981 ±  3%  interrupts.CPU45.LOC:Local_timer_interrupts
    976440 ±  3%     -29.2%     691802 ±  3%  interrupts.CPU45.RES:Rescheduling_interrupts
   1109645           -11.1%     986013 ±  3%  interrupts.CPU46.LOC:Local_timer_interrupts
    952352 ±  2%     -26.7%     697811 ±  4%  interrupts.CPU46.RES:Rescheduling_interrupts
   1109682           -11.1%     986172 ±  3%  interrupts.CPU47.LOC:Local_timer_interrupts
    957266 ±  4%     -28.5%     684262        interrupts.CPU47.RES:Rescheduling_interrupts
     49803 ± 13%     -18.0%      40844 ± 12%  interrupts.CPU48.CAL:Function_call_interrupts
   1109702           -11.1%     986471 ±  3%  interrupts.CPU48.LOC:Local_timer_interrupts
    977427 ±  6%     -27.8%     705625 ±  3%  interrupts.CPU48.RES:Rescheduling_interrupts
   1109658           -11.1%     986172 ±  3%  interrupts.CPU49.LOC:Local_timer_interrupts
    937091 ±  7%     -26.2%     691409 ±  3%  interrupts.CPU49.RES:Rescheduling_interrupts
   1109735           -11.5%     982317 ±  4%  interrupts.CPU5.LOC:Local_timer_interrupts
    741049 ±  6%     -23.6%     566103 ±  6%  interrupts.CPU5.RES:Rescheduling_interrupts
   1109692           -11.1%     986149 ±  3%  interrupts.CPU50.LOC:Local_timer_interrupts
    942242 ±  7%     -26.2%     695845 ±  3%  interrupts.CPU50.RES:Rescheduling_interrupts
   1109688           -11.1%     986153 ±  3%  interrupts.CPU51.LOC:Local_timer_interrupts
    978237 ±  5%     -27.8%     706561 ±  2%  interrupts.CPU51.RES:Rescheduling_interrupts
   1109680           -11.1%     986101 ±  3%  interrupts.CPU52.LOC:Local_timer_interrupts
    942523 ±  6%     -25.9%     698844 ±  2%  interrupts.CPU52.RES:Rescheduling_interrupts
   1109709           -11.1%     986142 ±  3%  interrupts.CPU53.LOC:Local_timer_interrupts
    968770 ±  8%     -28.5%     692330 ±  4%  interrupts.CPU53.RES:Rescheduling_interrupts
   1109706           -11.2%     985372 ±  3%  interrupts.CPU54.LOC:Local_timer_interrupts
    892744 ±  3%     -27.4%     648201 ± 10%  interrupts.CPU54.RES:Rescheduling_interrupts
   1109711           -11.2%     985410 ±  3%  interrupts.CPU55.LOC:Local_timer_interrupts
    905314 ±  4%     -26.4%     666564 ± 10%  interrupts.CPU55.RES:Rescheduling_interrupts
   1109760           -11.2%     985583 ±  3%  interrupts.CPU56.LOC:Local_timer_interrupts
    923287 ±  5%     -27.0%     673797 ± 10%  interrupts.CPU56.RES:Rescheduling_interrupts
   1109750           -11.2%     985363 ±  3%  interrupts.CPU57.LOC:Local_timer_interrupts
    941749 ±  6%     -27.9%     678794 ±  9%  interrupts.CPU57.RES:Rescheduling_interrupts
   1109711           -11.2%     985560 ±  3%  interrupts.CPU58.LOC:Local_timer_interrupts
    911298 ±  3%     -27.4%     661172 ± 10%  interrupts.CPU58.RES:Rescheduling_interrupts
   1109694           -11.2%     985480 ±  3%  interrupts.CPU59.LOC:Local_timer_interrupts
    934580 ±  3%     -29.9%     654916 ±  9%  interrupts.CPU59.RES:Rescheduling_interrupts
   1109735           -11.5%     982399 ±  4%  interrupts.CPU6.LOC:Local_timer_interrupts
    740135 ±  6%     -24.3%     560193 ±  5%  interrupts.CPU6.RES:Rescheduling_interrupts
   1109690           -11.2%     985677 ±  3%  interrupts.CPU60.LOC:Local_timer_interrupts
    927684 ±  3%     -28.1%     667287 ± 11%  interrupts.CPU60.RES:Rescheduling_interrupts
   1109694           -11.2%     985524 ±  3%  interrupts.CPU61.LOC:Local_timer_interrupts
    958054 ±  4%     -30.3%     667940 ± 11%  interrupts.CPU61.RES:Rescheduling_interrupts
   1109679           -11.2%     985364 ±  3%  interrupts.CPU62.LOC:Local_timer_interrupts
    946262 ±  4%     -30.0%     662460 ± 11%  interrupts.CPU62.RES:Rescheduling_interrupts
   1109678           -11.2%     985301 ±  3%  interrupts.CPU63.LOC:Local_timer_interrupts
    931012 ±  4%     -26.6%     683774 ± 11%  interrupts.CPU63.RES:Rescheduling_interrupts
   1109691           -11.2%     985194 ±  3%  interrupts.CPU64.LOC:Local_timer_interrupts
    929091 ±  6%     -29.7%     653309 ± 11%  interrupts.CPU64.RES:Rescheduling_interrupts
   1109744           -11.2%     985320 ±  3%  interrupts.CPU65.LOC:Local_timer_interrupts
    933881 ±  3%     -27.6%     676297 ±  9%  interrupts.CPU65.RES:Rescheduling_interrupts
   1109710           -11.2%     985673 ±  3%  interrupts.CPU66.LOC:Local_timer_interrupts
    956361 ±  5%     -29.2%     676634 ±  9%  interrupts.CPU66.RES:Rescheduling_interrupts
   1109687           -11.2%     985315 ±  3%  interrupts.CPU67.LOC:Local_timer_interrupts
    909257 ±  3%     -26.6%     667439 ± 12%  interrupts.CPU67.RES:Rescheduling_interrupts
   1109693           -11.2%     985287 ±  3%  interrupts.CPU68.LOC:Local_timer_interrupts
    925288 ±  2%     -28.0%     665809 ±  9%  interrupts.CPU68.RES:Rescheduling_interrupts
   1109675           -11.2%     985486 ±  3%  interrupts.CPU69.LOC:Local_timer_interrupts
    958528 ±  4%     -29.0%     680393 ±  9%  interrupts.CPU69.RES:Rescheduling_interrupts
     31192 ±  5%     -17.8%      25645 ± 15%  interrupts.CPU7.CAL:Function_call_interrupts
   1109763           -11.4%     982851 ±  3%  interrupts.CPU7.LOC:Local_timer_interrupts
    766751 ±  7%     -25.8%     569219 ±  4%  interrupts.CPU7.RES:Rescheduling_interrupts
   1109572           -11.2%     985353 ±  3%  interrupts.CPU70.LOC:Local_timer_interrupts
    934309 ±  2%     -28.6%     666929 ± 11%  interrupts.CPU70.RES:Rescheduling_interrupts
   1109770           -11.2%     985376 ±  3%  interrupts.CPU71.LOC:Local_timer_interrupts
    923580 ±  5%     -26.7%     676802 ±  9%  interrupts.CPU71.RES:Rescheduling_interrupts
     48586 ± 18%     -31.1%      33495 ± 20%  interrupts.CPU72.CAL:Function_call_interrupts
   1109690           -11.4%     983437 ±  3%  interrupts.CPU72.LOC:Local_timer_interrupts
    732823 ±  6%     -25.1%     548780 ±  6%  interrupts.CPU72.RES:Rescheduling_interrupts
   1109780           -11.5%     982255 ±  4%  interrupts.CPU73.LOC:Local_timer_interrupts
    748198 ±  6%     -25.7%     555808 ±  2%  interrupts.CPU73.RES:Rescheduling_interrupts
   1109731           -11.5%     982361 ±  4%  interrupts.CPU74.LOC:Local_timer_interrupts
    732000 ±  5%     -24.4%     553248 ±  8%  interrupts.CPU74.RES:Rescheduling_interrupts
   1109689           -11.5%     982406 ±  4%  interrupts.CPU75.LOC:Local_timer_interrupts
    732437 ±  4%     -23.9%     557346 ±  5%  interrupts.CPU75.RES:Rescheduling_interrupts
   1109783           -11.5%     982253 ±  4%  interrupts.CPU76.LOC:Local_timer_interrupts
    727115           -22.8%     561035 ±  6%  interrupts.CPU76.RES:Rescheduling_interrupts
   1109749           -11.5%     981909 ±  4%  interrupts.CPU77.LOC:Local_timer_interrupts
    722320 ±  5%     -23.9%     549545 ±  7%  interrupts.CPU77.RES:Rescheduling_interrupts
   1109776           -11.5%     982201 ±  4%  interrupts.CPU78.LOC:Local_timer_interrupts
    774003 ±  8%     -27.3%     562990 ±  5%  interrupts.CPU78.RES:Rescheduling_interrupts
   1109771           -11.5%     982534 ±  3%  interrupts.CPU79.LOC:Local_timer_interrupts
    725034 ±  4%     -21.5%     569192 ±  5%  interrupts.CPU79.RES:Rescheduling_interrupts
   1109745           -11.5%     982390 ±  4%  interrupts.CPU8.LOC:Local_timer_interrupts
    766391 ±  6%     -26.4%     564224 ±  5%  interrupts.CPU8.RES:Rescheduling_interrupts
   1109765           -11.5%     982241 ±  4%  interrupts.CPU80.LOC:Local_timer_interrupts
    747959 ±  5%     -24.2%     566670 ±  8%  interrupts.CPU80.RES:Rescheduling_interrupts
   1109814           -11.5%     981873 ±  4%  interrupts.CPU81.LOC:Local_timer_interrupts
    747605 ±  7%     -24.3%     565728 ±  7%  interrupts.CPU81.RES:Rescheduling_interrupts
   1109714           -11.5%     982405 ±  4%  interrupts.CPU82.LOC:Local_timer_interrupts
    740632 ±  4%     -25.4%     552433 ±  7%  interrupts.CPU82.RES:Rescheduling_interrupts
   1109762           -11.5%     982189 ±  4%  interrupts.CPU83.LOC:Local_timer_interrupts
    752433 ±  6%     -23.9%     572430 ±  6%  interrupts.CPU83.RES:Rescheduling_interrupts
   1109801           -11.5%     982438 ±  4%  interrupts.CPU84.LOC:Local_timer_interrupts
    750543 ±  6%     -25.7%     557447 ±  7%  interrupts.CPU84.RES:Rescheduling_interrupts
   1109757           -11.5%     982181 ±  4%  interrupts.CPU85.LOC:Local_timer_interrupts
    710008 ±  5%     -20.3%     565565 ±  7%  interrupts.CPU85.RES:Rescheduling_interrupts
   1109819           -11.5%     982651 ±  3%  interrupts.CPU86.LOC:Local_timer_interrupts
    756402 ±  4%     -26.1%     559174 ±  6%  interrupts.CPU86.RES:Rescheduling_interrupts
   1109763           -11.5%     982634 ±  3%  interrupts.CPU87.LOC:Local_timer_interrupts
    763567 ±  5%     -27.5%     553678 ±  5%  interrupts.CPU87.RES:Rescheduling_interrupts
   1109795           -11.4%     982983 ±  3%  interrupts.CPU88.LOC:Local_timer_interrupts
    736222 ±  4%     -24.9%     552717 ±  6%  interrupts.CPU88.RES:Rescheduling_interrupts
   1109788           -11.4%     982800 ±  3%  interrupts.CPU89.LOC:Local_timer_interrupts
    729712 ±  5%     -24.6%     550308 ±  6%  interrupts.CPU89.RES:Rescheduling_interrupts
   1109951           -11.5%     982025 ±  4%  interrupts.CPU9.LOC:Local_timer_interrupts
    749632 ±  7%     -25.7%     556858 ±  7%  interrupts.CPU9.RES:Rescheduling_interrupts
   1109721           -11.4%     983595 ±  3%  interrupts.CPU90.LOC:Local_timer_interrupts
    889032 ±  8%     -34.4%     582776 ±  9%  interrupts.CPU90.RES:Rescheduling_interrupts
   1109668           -11.4%     983263 ±  3%  interrupts.CPU91.LOC:Local_timer_interrupts
    914855 ±  5%     -35.1%     593653 ±  8%  interrupts.CPU91.RES:Rescheduling_interrupts
   1109647           -11.3%     983819 ±  3%  interrupts.CPU92.LOC:Local_timer_interrupts
    948214 ±  8%     -35.0%     616661 ± 10%  interrupts.CPU92.RES:Rescheduling_interrupts
   1109694           -11.4%     983708 ±  3%  interrupts.CPU93.LOC:Local_timer_interrupts
    892302 ±  7%     -32.9%     598886 ±  9%  interrupts.CPU93.RES:Rescheduling_interrupts
   1109699           -11.4%     983475 ±  3%  interrupts.CPU94.LOC:Local_timer_interrupts
    913830 ±  4%     -33.8%     605237 ± 10%  interrupts.CPU94.RES:Rescheduling_interrupts
   1109687           -11.4%     983654 ±  3%  interrupts.CPU95.LOC:Local_timer_interrupts
    942816 ±  8%     -35.3%     610115 ±  9%  interrupts.CPU95.RES:Rescheduling_interrupts
   1109662           -11.3%     983930 ±  3%  interrupts.CPU96.LOC:Local_timer_interrupts
    921056 ±  7%     -33.8%     609373 ± 10%  interrupts.CPU96.RES:Rescheduling_interrupts
   1109663           -11.3%     983971 ±  3%  interrupts.CPU97.LOC:Local_timer_interrupts
    939132 ±  5%     -35.0%     610299 ± 10%  interrupts.CPU97.RES:Rescheduling_interrupts
   1109659           -11.3%     983854 ±  3%  interrupts.CPU98.LOC:Local_timer_interrupts
    905810 ±  6%     -32.7%     609848 ±  9%  interrupts.CPU98.RES:Rescheduling_interrupts
   1109697           -11.3%     983762 ±  3%  interrupts.CPU99.LOC:Local_timer_interrupts
    936646 ±  7%     -34.8%     610801 ± 10%  interrupts.CPU99.RES:Rescheduling_interrupts
 1.598e+08           -11.3%  1.417e+08 ±  3%  interrupts.LOC:Local_timer_interrupts
 1.274e+08 ±  2%     -28.5%   91108791        interrupts.RES:Rescheduling_interrupts
     21.07 ± 13%     -14.9        6.16 ± 35%  perf-profile.calltrace.cycles-pp.perf_prepare_sample.perf_event_output_forward.__perf_event_overflow.perf_swevent_overflow.perf_tp_event
     20.01 ± 13%     -14.3        5.71 ± 38%  perf-profile.calltrace.cycles-pp.perf_callchain.perf_prepare_sample.perf_event_output_forward.__perf_event_overflow.perf_swevent_overflow
     19.91 ± 13%     -14.2        5.68 ± 38%  perf-profile.calltrace.cycles-pp.get_perf_callchain.perf_callchain.perf_prepare_sample.perf_event_output_forward.__perf_event_overflow
     11.18 ± 11%     -11.2        0.00        perf-profile.calltrace.cycles-pp.page_counter_try_charge.obj_cgroup_charge_pages.obj_cgroup_charge.__kmalloc_node_track_caller.kmalloc_reserve
     17.91 ± 12%     -11.0        6.86 ± 22%  perf-profile.calltrace.cycles-pp.__wake_up_common_lock.sock_def_readable.unix_stream_sendmsg.sock_sendmsg.sock_write_iter
     17.29 ± 12%     -10.9        6.42 ± 23%  perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.sock_def_readable.unix_stream_sendmsg.sock_sendmsg
     17.15 ± 12%     -10.8        6.31 ± 23%  perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.sock_def_readable.unix_stream_sendmsg
     17.06 ± 12%     -10.8        6.26 ± 23%  perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.sock_def_readable
     18.74 ± 11%     -10.4        8.30 ± 17%  perf-profile.calltrace.cycles-pp.sock_def_readable.unix_stream_sendmsg.sock_sendmsg.sock_write_iter.new_sync_write
     12.96 ± 13%      -9.5        3.45 ± 49%  perf-profile.calltrace.cycles-pp.perf_callchain_kernel.get_perf_callchain.perf_callchain.perf_prepare_sample.perf_event_output_forward
     15.35 ± 11%      -9.5        5.88 ± 22%  perf-profile.calltrace.cycles-pp.schedule_timeout.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter.new_sync_read
     15.28 ± 11%      -9.4        5.84 ± 22%  perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter
     15.16 ± 11%      -9.4        5.77 ± 23%  perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_timeout.unix_stream_read_generic.unix_stream_recvmsg
     10.21 ± 12%      -7.0        3.21 ± 32%  perf-profile.calltrace.cycles-pp.__perf_event_overflow.perf_swevent_overflow.perf_tp_event.perf_trace_sched_stat_runtime.update_curr
     10.16 ± 12%      -7.0        3.19 ± 32%  perf-profile.calltrace.cycles-pp.perf_event_output_forward.__perf_event_overflow.perf_swevent_overflow.perf_tp_event.perf_trace_sched_stat_runtime
      5.82 ± 26%      -5.8        0.00        perf-profile.calltrace.cycles-pp.asm_exc_page_fault.__get_user_nocheck_8.perf_callchain_user.get_perf_callchain.perf_callchain
      5.63 ± 11%      -5.6        0.00        perf-profile.calltrace.cycles-pp.page_counter_try_charge.obj_cgroup_charge_pages.obj_cgroup_charge.kmem_cache_alloc_node.__alloc_skb
      6.42 ± 13%      -5.6        0.81 ±114%  perf-profile.calltrace.cycles-pp.perf_callchain_user.get_perf_callchain.perf_callchain.perf_prepare_sample.perf_event_output_forward
      6.28 ± 14%      -4.8        1.48 ± 46%  perf-profile.calltrace.cycles-pp.perf_swevent_overflow.perf_tp_event.perf_trace_sched_switch.__schedule.schedule
      6.27 ± 14%      -4.8        1.47 ± 46%  perf-profile.calltrace.cycles-pp.__perf_event_overflow.perf_swevent_overflow.perf_tp_event.perf_trace_sched_switch.__schedule
      5.06 ± 17%      -4.8        0.28 ±223%  perf-profile.calltrace.cycles-pp.__unwind_start.perf_callchain_kernel.get_perf_callchain.perf_callchain.perf_prepare_sample
      6.24 ± 14%      -4.8        1.46 ± 46%  perf-profile.calltrace.cycles-pp.perf_event_output_forward.__perf_event_overflow.perf_swevent_overflow.perf_tp_event.perf_trace_sched_switch
      6.92 ± 13%      -4.5        2.40 ± 24%  perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      4.80 ± 13%      -4.5        0.31 ±223%  perf-profile.calltrace.cycles-pp.unwind_next_frame.perf_callchain_kernel.get_perf_callchain.perf_callchain.perf_prepare_sample
      6.87 ± 13%      -4.5        2.38 ± 24%  perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common
     52.74            -4.5       48.28        perf-profile.calltrace.cycles-pp.__libc_write
     52.06            -4.3       47.73        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_write
      6.76 ± 11%      -4.3        2.44 ± 26%  perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.schedule_timeout.unix_stream_read_generic
     51.93            -4.3       47.62        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
      6.17 ± 13%      -4.1        2.04 ± 27%  perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function
      6.08 ± 11%      -4.0        2.11 ± 28%  perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.schedule_timeout
      5.56 ± 13%      -3.8        1.73 ± 30%  perf-profile.calltrace.cycles-pp.update_curr.enqueue_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up
      5.78 ± 11%      -3.8        1.95 ± 30%  perf-profile.calltrace.cycles-pp.update_curr.dequeue_entity.dequeue_task_fair.__schedule.schedule
      5.51 ± 11%      -3.7        1.83 ± 31%  perf-profile.calltrace.cycles-pp.perf_trace_sched_stat_runtime.update_curr.dequeue_entity.dequeue_task_fair.__schedule
      5.25 ± 13%      -3.7        1.59 ± 32%  perf-profile.calltrace.cycles-pp.perf_trace_sched_stat_runtime.update_curr.enqueue_entity.enqueue_task_fair.ttwu_do_activate
      5.45 ± 12%      -3.6        1.83 ± 30%  perf-profile.calltrace.cycles-pp.perf_trace_sched_wakeup_template.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      5.36 ± 11%      -3.6        1.77 ± 32%  perf-profile.calltrace.cycles-pp.perf_tp_event.perf_trace_sched_stat_runtime.update_curr.dequeue_entity.dequeue_task_fair
      5.08 ± 13%      -3.6        1.52 ± 33%  perf-profile.calltrace.cycles-pp.perf_tp_event.perf_trace_sched_stat_runtime.update_curr.enqueue_entity.enqueue_task_fair
      5.31 ± 12%      -3.6        1.76 ± 31%  perf-profile.calltrace.cycles-pp.perf_tp_event.perf_trace_sched_wakeup_template.try_to_wake_up.autoremove_wake_function.__wake_up_common
      5.25 ± 11%      -3.5        1.73 ± 31%  perf-profile.calltrace.cycles-pp.perf_swevent_overflow.perf_tp_event.perf_trace_sched_stat_runtime.update_curr.dequeue_entity
      4.98 ± 13%      -3.5        1.49 ± 33%  perf-profile.calltrace.cycles-pp.perf_swevent_overflow.perf_tp_event.perf_trace_sched_stat_runtime.update_curr.enqueue_entity
      5.10 ± 12%      -3.4        1.68 ± 31%  perf-profile.calltrace.cycles-pp.perf_swevent_overflow.perf_tp_event.perf_trace_sched_wakeup_template.try_to_wake_up.autoremove_wake_function
      5.09 ± 12%      -3.4        1.67 ± 31%  perf-profile.calltrace.cycles-pp.__perf_event_overflow.perf_swevent_overflow.perf_tp_event.perf_trace_sched_wakeup_template.try_to_wake_up
      5.07 ± 12%      -3.4        1.66 ± 32%  perf-profile.calltrace.cycles-pp.perf_event_output_forward.__perf_event_overflow.perf_swevent_overflow.perf_tp_event.perf_trace_sched_wakeup_template
      4.63 ± 13%      -3.2        1.45 ± 33%  perf-profile.calltrace.cycles-pp.perf_trace_sched_switch.__schedule.schedule.schedule_timeout.unix_stream_read_generic
      4.45 ± 13%      -3.1        1.38 ± 33%  perf-profile.calltrace.cycles-pp.perf_tp_event.perf_trace_sched_switch.__schedule.schedule.schedule_timeout
      4.07 ± 12%      -2.9        1.15 ± 27%  perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
      4.01 ± 12%      -2.9        1.11 ± 28%  perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
      3.78 ± 12%      -2.8        0.99 ± 30%  perf-profile.calltrace.cycles-pp.schedule.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
      3.74 ± 12%      -2.8        0.98 ± 30%  perf-profile.calltrace.cycles-pp.__schedule.schedule.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
      2.40 ± 12%      -1.5        0.90 ± 18%  perf-profile.calltrace.cycles-pp._raw_spin_lock.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      2.09 ± 13%      -1.4        0.66 ± 46%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.try_to_wake_up.autoremove_wake_function.__wake_up_common
     47.70            -1.3       46.38        perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
     47.45            -1.3       46.14        perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
     46.94            -1.3       45.63        perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     46.52            -1.2       45.29        perf-profile.calltrace.cycles-pp.sock_write_iter.new_sync_write.vfs_write.ksys_write.do_syscall_64
     46.20            -1.1       45.08        perf-profile.calltrace.cycles-pp.sock_sendmsg.sock_write_iter.new_sync_write.vfs_write.ksys_write
     45.87            -1.1       44.76        perf-profile.calltrace.cycles-pp.unix_stream_sendmsg.sock_sendmsg.sock_write_iter.new_sync_write.vfs_write
      1.33 ±  3%      -0.6        0.78 ±  6%  perf-profile.calltrace.cycles-pp.switch_mm_irqs_off.__schedule.schedule.schedule_timeout.unix_stream_read_generic
      0.62 ±  2%      +0.2        0.78        perf-profile.calltrace.cycles-pp.__check_object_size.simple_copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.unix_stream_read_actor
      0.65 ±  2%      +0.2        0.81        perf-profile.calltrace.cycles-pp.simple_copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.unix_stream_read_actor.unix_stream_read_generic
      0.75 ±  2%      +0.2        0.92 ±  4%  perf-profile.calltrace.cycles-pp.skb_copy_datagram_from_iter.unix_stream_sendmsg.sock_sendmsg.sock_write_iter.new_sync_write
      0.62 ±  3%      +0.3        0.90 ±  3%  perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter
      0.65 ±  2%      +0.3        0.94 ±  3%  perf-profile.calltrace.cycles-pp.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.unix_stream_read_actor
      0.81 ±  2%      +0.3        1.12 ±  3%  perf-profile.calltrace.cycles-pp._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.unix_stream_read_actor.unix_stream_read_generic
      0.58 ±  7%      +0.4        0.96 ±  4%  perf-profile.calltrace.cycles-pp.__slab_free.skb_release_data.consume_skb.unix_stream_read_generic.unix_stream_recvmsg
      0.18 ±141%      +0.4        0.57 ±  6%  perf-profile.calltrace.cycles-pp.mod_objcg_state.kmem_cache_alloc_node.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb
      1.06 ±  6%      +0.4        1.47 ±  5%  perf-profile.calltrace.cycles-pp._raw_spin_lock.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter.new_sync_read
      0.73 ±  7%      +0.4        1.18 ±  7%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.skb_queue_tail.unix_stream_sendmsg.sock_sendmsg.sock_write_iter
      0.75 ±  7%      +0.4        1.20 ±  7%  perf-profile.calltrace.cycles-pp.skb_queue_tail.unix_stream_sendmsg.sock_sendmsg.sock_write_iter.new_sync_write
      0.60 ± 12%      +0.5        1.06 ±  8%  perf-profile.calltrace.cycles-pp.skb_unlink.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter.new_sync_read
      0.36 ± 71%      +0.5        0.83 ±  9%  perf-profile.calltrace.cycles-pp.get_mem_cgroup_from_objcg.obj_cgroup_charge_pages.obj_cgroup_charge.__kmalloc_node_track_caller.kmalloc_reserve
      1.52 ±  2%      +0.5        1.99 ±  2%  perf-profile.calltrace.cycles-pp.skb_copy_datagram_iter.unix_stream_read_actor.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter
      1.50 ±  2%      +0.5        1.97 ±  2%  perf-profile.calltrace.cycles-pp.__skb_datagram_iter.skb_copy_datagram_iter.unix_stream_read_actor.unix_stream_read_generic.unix_stream_recvmsg
      0.37 ± 71%      +0.5        0.84 ±  8%  perf-profile.calltrace.cycles-pp.get_mem_cgroup_from_objcg.obj_cgroup_uncharge_pages.kfree.skb_release_data.consume_skb
      1.53            +0.5        2.01 ±  2%  perf-profile.calltrace.cycles-pp.unix_stream_read_actor.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter.new_sync_read
      0.09 ±223%      +0.5        0.58 ±  6%  perf-profile.calltrace.cycles-pp.mod_objcg_state.kfree.skb_release_data.consume_skb.unix_stream_read_generic
      0.65 ±  5%      +0.5        1.15 ±  6%  perf-profile.calltrace.cycles-pp.skb_set_owner_w.sock_alloc_send_pskb.unix_stream_sendmsg.sock_sendmsg.sock_write_iter
      0.08 ±223%      +0.5        0.59 ±  5%  perf-profile.calltrace.cycles-pp.mutex_lock.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter.new_sync_read
      0.41 ± 71%      +0.6        1.00 ±  7%  perf-profile.calltrace.cycles-pp.mutex_unlock.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter.new_sync_read
      0.41 ± 71%      +0.6        1.02 ±  8%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.skb_unlink.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter
      0.00            +0.7        0.67 ±  4%  perf-profile.calltrace.cycles-pp.unix_write_space.sock_wfree.unix_destruct_scm.skb_release_head_state.skb_release_all
      1.12 ±  7%      +0.7        1.81 ±  5%  perf-profile.calltrace.cycles-pp._raw_spin_lock.unix_stream_sendmsg.sock_sendmsg.sock_write_iter.new_sync_write
      0.88 ± 10%      +0.7        1.57 ±  5%  perf-profile.calltrace.cycles-pp.__slab_free.kmem_cache_free.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter
      0.96 ± 13%      +0.7        1.69 ±  7%  perf-profile.calltrace.cycles-pp.propagate_protected_usage.page_counter_uncharge.obj_cgroup_uncharge_pages.kmem_cache_free.unix_stream_read_generic
      1.16 ±  9%      +0.8        1.99 ±  4%  perf-profile.calltrace.cycles-pp.sock_wfree.unix_destruct_scm.skb_release_head_state.skb_release_all.consume_skb
      1.20 ±  8%      +0.8        2.03 ±  4%  perf-profile.calltrace.cycles-pp.unix_destruct_scm.skb_release_head_state.skb_release_all.consume_skb.unix_stream_read_generic
      1.22 ±  8%      +0.8        2.06 ±  4%  perf-profile.calltrace.cycles-pp.skb_release_head_state.skb_release_all.consume_skb.unix_stream_read_generic.unix_stream_recvmsg
      1.23 ±  8%      +0.8        2.07 ±  4%  perf-profile.calltrace.cycles-pp.skb_release_all.consume_skb.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter
      1.97 ± 12%      +1.4        3.41 ±  7%  perf-profile.calltrace.cycles-pp.propagate_protected_usage.page_counter_uncharge.obj_cgroup_uncharge_pages.kfree.skb_release_data
      0.00            +1.7        1.72 ±  7%  perf-profile.calltrace.cycles-pp.propagate_protected_usage.page_counter_charge.obj_cgroup_charge_pages.obj_cgroup_charge.kmem_cache_alloc_node
      3.75 ± 12%      +1.8        5.55 ±  6%  perf-profile.calltrace.cycles-pp.page_counter_cancel.page_counter_uncharge.obj_cgroup_uncharge_pages.kmem_cache_free.unix_stream_read_generic
      6.27 ± 10%      +2.2        8.45 ±  5%  perf-profile.calltrace.cycles-pp.obj_cgroup_charge_pages.obj_cgroup_charge.kmem_cache_alloc_node.__alloc_skb.alloc_skb_with_frags
      6.32 ± 10%      +2.2        8.50 ±  5%  perf-profile.calltrace.cycles-pp.obj_cgroup_charge.kmem_cache_alloc_node.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb
      7.85 ±  8%      +2.4       10.30 ±  4%  perf-profile.calltrace.cycles-pp.kmem_cache_alloc_node.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb.unix_stream_sendmsg
      4.76 ± 12%      +2.6        7.32 ±  5%  perf-profile.calltrace.cycles-pp.page_counter_uncharge.obj_cgroup_uncharge_pages.kmem_cache_free.unix_stream_read_generic.unix_stream_recvmsg
      5.38 ± 11%      +3.0        8.34 ±  5%  perf-profile.calltrace.cycles-pp.obj_cgroup_uncharge_pages.kmem_cache_free.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter
      7.80 ± 11%      +3.3       11.14 ±  5%  perf-profile.calltrace.cycles-pp.page_counter_cancel.page_counter_uncharge.obj_cgroup_uncharge_pages.kfree.skb_release_data
      0.00            +3.4        3.40 ±  6%  perf-profile.calltrace.cycles-pp.propagate_protected_usage.page_counter_charge.obj_cgroup_charge_pages.obj_cgroup_charge.__kmalloc_node_track_caller
     46.87            +3.5       50.41        perf-profile.calltrace.cycles-pp.__libc_read
     45.87            +3.7       49.56        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_read
     45.74            +3.7       49.46        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
     44.83            +3.9       48.69        perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
     45.02            +3.9       48.89        perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read
     43.59            +3.9       47.47        perf-profile.calltrace.cycles-pp.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter.new_sync_read.vfs_read
     43.68            +3.9       47.56        perf-profile.calltrace.cycles-pp.unix_stream_recvmsg.sock_read_iter.new_sync_read.vfs_read.ksys_read
     44.11            +3.9       48.00        perf-profile.calltrace.cycles-pp.sock_read_iter.new_sync_read.vfs_read.ksys_read.do_syscall_64
     44.32            +3.9       48.22        perf-profile.calltrace.cycles-pp.new_sync_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      7.54 ±  9%      +4.1       11.64 ±  4%  perf-profile.calltrace.cycles-pp.kmem_cache_free.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter.new_sync_read
     12.45 ± 10%      +4.3       16.71 ±  5%  perf-profile.calltrace.cycles-pp.obj_cgroup_charge_pages.obj_cgroup_charge.__kmalloc_node_track_caller.kmalloc_reserve.__alloc_skb
     12.50 ± 10%      +4.3       16.77 ±  5%  perf-profile.calltrace.cycles-pp.obj_cgroup_charge.__kmalloc_node_track_caller.kmalloc_reserve.__alloc_skb.alloc_skb_with_frags
     14.17 ±  9%      +4.5       18.70 ±  4%  perf-profile.calltrace.cycles-pp.__kmalloc_node_track_caller.kmalloc_reserve.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb
     14.23 ±  9%      +4.5       18.75 ±  4%  perf-profile.calltrace.cycles-pp.kmalloc_reserve.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb.unix_stream_sendmsg
      9.88 ± 11%      +4.8       14.72 ±  5%  perf-profile.calltrace.cycles-pp.page_counter_uncharge.obj_cgroup_uncharge_pages.kfree.skb_release_data.consume_skb
     11.16 ± 11%      +5.6       16.78 ±  4%  perf-profile.calltrace.cycles-pp.obj_cgroup_uncharge_pages.kfree.skb_release_data.consume_skb.unix_stream_read_generic
     12.06 ±  9%      +5.9       17.92 ±  4%  perf-profile.calltrace.cycles-pp.kfree.skb_release_data.consume_skb.unix_stream_read_generic.unix_stream_recvmsg
     13.11 ±  9%      +6.5       19.61 ±  4%  perf-profile.calltrace.cycles-pp.skb_release_data.consume_skb.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter
     22.76 ±  8%      +7.1       29.89 ±  4%  perf-profile.calltrace.cycles-pp.alloc_skb_with_frags.sock_alloc_send_pskb.unix_stream_sendmsg.sock_sendmsg.sock_write_iter
     22.70 ±  8%      +7.1       29.82 ±  4%  perf-profile.calltrace.cycles-pp.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb.unix_stream_sendmsg.sock_sendmsg
      0.00            +7.4        7.38 ±  6%  perf-profile.calltrace.cycles-pp.page_counter_charge.obj_cgroup_charge_pages.obj_cgroup_charge.kmem_cache_alloc_node.__alloc_skb
     14.56 ±  9%      +7.5       22.04 ±  4%  perf-profile.calltrace.cycles-pp.consume_skb.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter.new_sync_read
     23.80 ±  8%      +7.9       31.75 ±  4%  perf-profile.calltrace.cycles-pp.sock_alloc_send_pskb.unix_stream_sendmsg.sock_sendmsg.sock_write_iter.new_sync_write
      0.00           +14.6       14.60 ±  5%  perf-profile.calltrace.cycles-pp.page_counter_charge.obj_cgroup_charge_pages.obj_cgroup_charge.__kmalloc_node_track_caller.kmalloc_reserve
     17.18 ± 11%     -16.6        0.59 ±  7%  perf-profile.children.cycles-pp.page_counter_try_charge
     23.30 ± 12%     -15.5        7.83 ± 31%  perf-profile.children.cycles-pp.perf_tp_event
     22.67 ± 12%     -15.1        7.58 ± 31%  perf-profile.children.cycles-pp.perf_swevent_overflow
     22.63 ± 12%     -15.1        7.56 ± 31%  perf-profile.children.cycles-pp.__perf_event_overflow
     22.53 ± 12%     -15.0        7.52 ± 31%  perf-profile.children.cycles-pp.perf_event_output_forward
     22.12 ± 12%     -14.8        7.34 ± 31%  perf-profile.children.cycles-pp.perf_prepare_sample
     21.00 ± 12%     -14.1        6.88 ± 32%  perf-profile.children.cycles-pp.perf_callchain
     20.89 ± 12%     -14.0        6.85 ± 32%  perf-profile.children.cycles-pp.get_perf_callchain
     19.54 ± 10%     -12.0        7.51 ± 22%  perf-profile.children.cycles-pp.schedule
     19.55 ± 10%     -11.9        7.61 ± 21%  perf-profile.children.cycles-pp.__schedule
     18.11 ± 12%     -10.8        7.26 ± 21%  perf-profile.children.cycles-pp.__wake_up_common_lock
     17.47 ± 12%     -10.7        6.81 ± 21%  perf-profile.children.cycles-pp.__wake_up_common
     17.34 ± 12%     -10.6        6.70 ± 22%  perf-profile.children.cycles-pp.autoremove_wake_function
     17.24 ± 12%     -10.6        6.64 ± 22%  perf-profile.children.cycles-pp.try_to_wake_up
     18.74 ± 11%     -10.4        8.30 ± 17%  perf-profile.children.cycles-pp.sock_def_readable
     13.76 ± 13%      -9.3        4.50 ± 32%  perf-profile.children.cycles-pp.perf_callchain_kernel
     15.56 ± 10%      -9.2        6.32 ± 21%  perf-profile.children.cycles-pp.schedule_timeout
     12.23 ± 12%      -7.8        4.42 ± 28%  perf-profile.children.cycles-pp.update_curr
     11.26 ± 12%      -7.3        3.96 ± 30%  perf-profile.children.cycles-pp.perf_trace_sched_stat_runtime
     10.42 ± 13%      -7.0        3.38 ± 32%  perf-profile.children.cycles-pp.unwind_next_frame
      6.96 ± 13%      -4.8        2.12 ± 34%  perf-profile.children.cycles-pp.perf_trace_sched_switch
      6.72 ± 13%      -4.5        2.22 ± 31%  perf-profile.children.cycles-pp.perf_callchain_user
      7.06 ± 13%      -4.5        2.60 ± 24%  perf-profile.children.cycles-pp.ttwu_do_activate
     52.87            -4.4       48.42        perf-profile.children.cycles-pp.__libc_write
      7.00 ± 13%      -4.4        2.58 ± 24%  perf-profile.children.cycles-pp.enqueue_task_fair
      6.37 ± 12%      -4.3        2.11 ± 31%  perf-profile.children.cycles-pp.__get_user_nocheck_8
      6.86 ± 11%      -4.2        2.63 ± 24%  perf-profile.children.cycles-pp.dequeue_task_fair
      6.30 ± 13%      -4.1        2.21 ± 26%  perf-profile.children.cycles-pp.enqueue_entity
      6.17 ± 11%      -3.9        2.28 ± 27%  perf-profile.children.cycles-pp.dequeue_entity
      5.50 ± 12%      -3.7        1.76 ± 32%  perf-profile.children.cycles-pp.__unwind_start
      5.52 ± 11%      -3.6        1.94 ± 29%  perf-profile.children.cycles-pp.perf_trace_sched_wakeup_template
      4.70 ± 11%      -3.1        1.64 ± 19%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
      4.61 ± 11%      -3.0        1.56 ± 20%  perf-profile.children.cycles-pp.exit_to_user_mode_prepare
      4.40 ± 13%      -2.5        1.87 ± 28%  perf-profile.children.cycles-pp.asm_exc_page_fault
      2.99 ± 13%      -2.0        1.00 ± 31%  perf-profile.children.cycles-pp.__orc_find
      3.90 ± 10%      -1.5        2.38 ± 21%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
     47.70            -1.3       46.38        perf-profile.children.cycles-pp.ksys_write
     47.46            -1.3       46.14        perf-profile.children.cycles-pp.vfs_write
     46.95            -1.3       45.64        perf-profile.children.cycles-pp.new_sync_write
     46.55            -1.2       45.36        perf-profile.children.cycles-pp.sock_write_iter
      1.67 ± 13%      -1.1        0.55 ± 32%  perf-profile.children.cycles-pp.orc_find
     46.21            -1.1       45.09        perf-profile.children.cycles-pp.sock_sendmsg
     45.90            -1.1       44.79        perf-profile.children.cycles-pp.unix_stream_sendmsg
      1.66 ± 12%      -1.1        0.57 ± 31%  perf-profile.children.cycles-pp.unwind_get_return_address
      5.94 ±  5%      -1.1        4.86 ±  3%  perf-profile.children.cycles-pp._raw_spin_lock
      2.00 ±  3%      -0.9        1.12 ±  6%  perf-profile.children.cycles-pp.switch_mm_irqs_off
      1.30 ± 12%      -0.8        0.46 ± 30%  perf-profile.children.cycles-pp.__kernel_text_address
      1.20 ± 10%      -0.8        0.44 ± 28%  perf-profile.children.cycles-pp.native_irq_return_iret
      1.09 ± 13%      -0.8        0.34 ± 31%  perf-profile.children.cycles-pp.kernelmode_fixup_or_oops
      1.05 ± 14%      -0.7        0.32 ± 34%  perf-profile.children.cycles-pp.stack_access_ok
      1.01 ± 13%      -0.7        0.31 ± 31%  perf-profile.children.cycles-pp.fixup_exception
      1.00 ± 12%      -0.7        0.34 ± 30%  perf-profile.children.cycles-pp.kernel_text_address
      0.91 ± 13%      -0.6        0.28 ± 32%  perf-profile.children.cycles-pp.search_exception_tables
      0.88 ± 13%      -0.6        0.27 ± 32%  perf-profile.children.cycles-pp.search_extable
      0.84 ± 14%      -0.6        0.26 ± 31%  perf-profile.children.cycles-pp.bsearch
      1.44 ± 14%      -0.5        0.90 ± 29%  perf-profile.children.cycles-pp.exc_page_fault
     97.98            -0.5       97.46        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.95 ± 14%      -0.5        0.46 ± 13%  perf-profile.children.cycles-pp.select_task_rq_fair
     97.74            -0.5       97.26        perf-profile.children.cycles-pp.do_syscall_64
      0.82 ± 11%      -0.4        0.42 ±  8%  perf-profile.children.cycles-pp.pick_next_task_fair
      0.70 ± 14%      -0.4        0.31 ± 15%  perf-profile.children.cycles-pp.select_idle_sibling
      0.65 ±  9%      -0.4        0.27 ± 28%  perf-profile.children.cycles-pp.__perf_event_header__init_id
      0.87 ± 12%      -0.4        0.51 ±  9%  perf-profile.children.cycles-pp.update_load_avg
      0.51 ± 14%      -0.4        0.16 ± 31%  perf-profile.children.cycles-pp.cmp_ex_search
      0.46 ± 11%      -0.3        0.21 ± 13%  perf-profile.children.cycles-pp.reweight_entity
      0.44 ±  9%      -0.3        0.19 ± 10%  perf-profile.children.cycles-pp.load_new_mm_cr3
      0.41 ± 15%      -0.2        0.18 ± 16%  perf-profile.children.cycles-pp.select_idle_cpu
      0.46 ±  8%      -0.2        0.24 ±  7%  perf-profile.children.cycles-pp.switch_fpu_return
      0.37 ±  8%      -0.2        0.15 ± 26%  perf-profile.children.cycles-pp.perf_event_pid_type
      0.45 ±  3%      -0.2        0.25 ±  6%  perf-profile.children.cycles-pp.__switch_to
      0.31 ± 11%      -0.2        0.11 ± 32%  perf-profile.children.cycles-pp.ftrace_graph_ret_addr
      0.31 ± 12%      -0.2        0.12 ± 30%  perf-profile.children.cycles-pp.perf_output_begin_forward
      0.31 ±  9%      -0.2        0.13 ± 17%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.30 ±  8%      -0.2        0.12 ± 28%  perf-profile.children.cycles-pp.__task_pid_nr_ns
      0.32 ± 16%      -0.2        0.15 ± 31%  perf-profile.children.cycles-pp.perf_trace_sched_migrate_task
      0.37 ±  5%      -0.2        0.21 ±  7%  perf-profile.children.cycles-pp.ttwu_do_wakeup
      0.37 ± 16%      -0.2        0.21 ± 24%  perf-profile.children.cycles-pp.set_task_cpu
      0.27 ± 10%      -0.2        0.11 ± 18%  perf-profile.children.cycles-pp.native_sched_clock
      0.27 ± 17%      -0.2        0.11 ± 22%  perf-profile.children.cycles-pp.available_idle_cpu
      0.30 ±  9%      -0.2        0.15 ± 10%  perf-profile.children.cycles-pp.set_next_entity
      0.35 ±  5%      -0.2        0.20 ±  7%  perf-profile.children.cycles-pp.check_preempt_curr
      0.23 ± 13%      -0.2        0.08 ± 48%  perf-profile.children.cycles-pp.bad_get_user
      0.29 ± 10%      -0.1        0.14 ±  9%  perf-profile.children.cycles-pp.__switch_to_asm
      0.32 ±  9%      -0.1        0.18 ± 14%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.66 ±  5%      -0.1        0.52 ±  6%  perf-profile.children.cycles-pp.prepare_to_wait
      0.30 ±  4%      -0.1        0.16 ±  9%  perf-profile.children.cycles-pp.check_preempt_wakeup
      0.20 ± 11%      -0.1        0.06 ± 47%  perf-profile.children.cycles-pp.error_entry
      0.26 ± 11%      -0.1        0.12 ± 10%  perf-profile.children.cycles-pp.update_rq_clock
      0.21 ± 16%      -0.1        0.08 ±  8%  perf-profile.children.cycles-pp.__enqueue_entity
      0.30 ± 15%      -0.1        0.17 ± 22%  perf-profile.children.cycles-pp.finish_task_switch
      0.19 ± 14%      -0.1        0.07 ± 48%  perf-profile.children.cycles-pp.is_bpf_text_address
      0.24 ± 11%      -0.1        0.11 ± 13%  perf-profile.children.cycles-pp.__restore_fpregs_from_fpstate
      0.24 ± 12%      -0.1        0.12 ± 11%  perf-profile.children.cycles-pp.__update_load_avg_se
      0.21 ± 16%      -0.1        0.09 ± 20%  perf-profile.children.cycles-pp.cpuacct_charge
      0.22 ± 13%      -0.1        0.10 ± 11%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.13 ± 15%      -0.1        0.03 ±102%  perf-profile.children.cycles-pp.perf_tp_event_match
      0.16 ± 11%      -0.1        0.05 ± 45%  perf-profile.children.cycles-pp.perf_trace_buf_alloc
      0.16 ± 16%      -0.1        0.06 ± 11%  perf-profile.children.cycles-pp.put_prev_entity
      0.14 ± 14%      -0.1        0.04 ± 75%  perf-profile.children.cycles-pp.save_fpregs_to_fpstate
      0.13 ± 11%      -0.1        0.04 ± 73%  perf-profile.children.cycles-pp.get_callchain_entry
      0.15 ± 11%      -0.1        0.06 ± 46%  perf-profile.children.cycles-pp.perf_trace_buf_update
      0.14 ± 15%      -0.1        0.05 ± 51%  perf-profile.children.cycles-pp.bpf_ksym_find
      0.13 ± 11%      -0.1        0.05 ± 47%  perf-profile.children.cycles-pp.get_stack_info
      0.16 ±  4%      -0.1        0.08 ±  8%  perf-profile.children.cycles-pp.resched_curr
      0.11 ± 11%      -0.1        0.04 ± 71%  perf-profile.children.cycles-pp.perf_instruction_pointer
      0.11 ± 14%      -0.1        0.04 ± 47%  perf-profile.children.cycles-pp.___perf_sw_event
      0.10 ± 14%      -0.1        0.04 ± 71%  perf-profile.children.cycles-pp.tracing_gen_ctx_irq_test
      0.13 ± 15%      -0.1        0.07 ± 10%  perf-profile.children.cycles-pp.update_min_vruntime
      0.14 ± 14%      -0.1        0.08 ± 11%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.14 ± 11%      -0.0        0.09 ± 12%  perf-profile.children.cycles-pp.preempt_schedule_common
      0.10 ±  9%      -0.0        0.05 ±  8%  perf-profile.children.cycles-pp.__list_add_valid
      0.08 ±  6%      -0.0        0.03 ± 70%  perf-profile.children.cycles-pp.iov_iter_init
      0.10 ±  7%      -0.0        0.06 ±  7%  perf-profile.children.cycles-pp.__calc_delta
      0.17 ±  4%      -0.0        0.14 ±  5%  perf-profile.children.cycles-pp.asm_sysvec_reschedule_ipi
      0.12 ±  6%      -0.0        0.09 ±  6%  perf-profile.children.cycles-pp.finish_wait
      0.14 ±  3%      -0.0        0.10 ±  4%  perf-profile.children.cycles-pp.rcu_read_unlock_strict
      0.06            +0.0        0.07        perf-profile.children.cycles-pp.refill_obj_stock
      0.06            +0.0        0.07 ±  5%  perf-profile.children.cycles-pp.check_stack_object
      0.07            +0.0        0.09 ±  7%  perf-profile.children.cycles-pp.rcu_all_qs
      0.15 ±  3%      +0.0        0.17 ±  4%  perf-profile.children.cycles-pp.__ksize
      0.04 ± 45%      +0.0        0.06 ±  7%  perf-profile.children.cycles-pp.irq_exit_rcu
      0.10 ±  4%      +0.0        0.12 ±  3%  perf-profile.children.cycles-pp.wait_for_unix_gc
      0.13 ±  5%      +0.0        0.16 ±  4%  perf-profile.children.cycles-pp.__might_fault
      0.19 ±  3%      +0.0        0.22 ±  4%  perf-profile.children.cycles-pp.__virt_addr_valid
      0.18            +0.0        0.21 ±  4%  perf-profile.children.cycles-pp.__might_sleep
      0.19 ±  2%      +0.0        0.22 ±  2%  perf-profile.children.cycles-pp.security_socket_recvmsg
      0.22 ±  2%      +0.0        0.25        perf-profile.children.cycles-pp.sock_recvmsg
      0.26 ±  3%      +0.0        0.30 ±  2%  perf-profile.children.cycles-pp.copyin
      0.22 ±  2%      +0.1        0.27 ±  3%  perf-profile.children.cycles-pp.__check_heap_object
      0.00            +0.1        0.05 ±  7%  perf-profile.children.cycles-pp.load_balance
      0.37            +0.1        0.42 ±  2%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.33 ±  2%      +0.1        0.38 ±  2%  perf-profile.children.cycles-pp.aa_sk_perm
      0.04 ± 73%      +0.1        0.09 ± 13%  perf-profile.children.cycles-pp.generic_file_write_iter
      0.04 ± 73%      +0.1        0.09 ± 13%  perf-profile.children.cycles-pp.__generic_file_write_iter
      0.04 ± 73%      +0.1        0.09 ± 13%  perf-profile.children.cycles-pp.generic_perform_write
      0.00            +0.1        0.06 ±  6%  perf-profile.children.cycles-pp.__softirqentry_text_start
      0.42 ±  2%      +0.1        0.49 ±  4%  perf-profile.children.cycles-pp._copy_from_iter
      0.38            +0.1        0.45 ±  3%  perf-profile.children.cycles-pp.__entry_text_start
      0.00            +0.1        0.08 ± 17%  perf-profile.children.cycles-pp.schedule_idle
      0.26 ±  4%      +0.1        0.35 ±  3%  perf-profile.children.cycles-pp.__build_skb_around
      0.01 ±223%      +0.1        0.11 ± 41%  perf-profile.children.cycles-pp.try_to_migrate_one
      0.49 ±  3%      +0.1        0.60 ±  5%  perf-profile.children.cycles-pp.mutex_lock
      0.90 ±  3%      +0.1        1.01 ±  6%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
      0.01 ±223%      +0.1        0.14 ± 40%  perf-profile.children.cycles-pp.try_to_migrate
      0.65            +0.2        0.81        perf-profile.children.cycles-pp.simple_copy_to_iter
      0.75 ±  2%      +0.2        0.92 ±  4%  perf-profile.children.cycles-pp.skb_copy_datagram_from_iter
      0.02 ±223%      +0.2        0.22 ± 48%  perf-profile.children.cycles-pp.remove_migration_pte
      0.76 ±  7%      +0.2        0.99 ± 12%  perf-profile.children.cycles-pp.__slab_alloc
      0.76 ±  6%      +0.2        0.98 ± 11%  perf-profile.children.cycles-pp.___slab_alloc
      0.92            +0.2        1.15 ±  2%  perf-profile.children.cycles-pp.__check_object_size
      0.29 ± 15%      +0.2        0.53 ±  7%  perf-profile.children.cycles-pp.drain_stock
      0.49 ± 12%      +0.2        0.74 ± 19%  perf-profile.children.cycles-pp.__unfreeze_partials
      0.03 ±155%      +0.2        0.27 ± 47%  perf-profile.children.cycles-pp.remove_migration_ptes
      0.34 ± 13%      +0.3        0.60 ±  6%  perf-profile.children.cycles-pp.refill_stock
      0.40 ±  9%      +0.3        0.67 ±  4%  perf-profile.children.cycles-pp.unix_write_space
      0.34 ± 15%      +0.3        0.62 ±  7%  perf-profile.children.cycles-pp.try_charge_memcg
      0.03 ±161%      +0.3        0.32 ± 45%  perf-profile.children.cycles-pp.page_vma_mapped_walk
      0.65 ±  2%      +0.3        0.94 ±  3%  perf-profile.children.cycles-pp.copyout
      0.82 ±  2%      +0.3        1.12 ±  3%  perf-profile.children.cycles-pp._copy_to_iter
      0.17 ± 38%      +0.3        0.49 ± 39%  perf-profile.children.cycles-pp.do_user_addr_fault
      0.88 ±  2%      +0.3        1.20 ±  3%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      0.06 ±106%      +0.4        0.41 ± 44%  perf-profile.children.cycles-pp.rmap_walk_anon
      1.28 ±  9%      +0.4        1.63 ±  7%  perf-profile.children.cycles-pp.__mod_memcg_lruvec_state
      0.06 ±106%      +0.4        0.42 ± 44%  perf-profile.children.cycles-pp.migrate_misplaced_page
      0.06 ±106%      +0.4        0.42 ± 44%  perf-profile.children.cycles-pp.migrate_pages
      0.06 ±108%      +0.4        0.42 ± 45%  perf-profile.children.cycles-pp.do_numa_page
      0.08 ± 81%      +0.4        0.44 ± 43%  perf-profile.children.cycles-pp.__handle_mm_fault
      0.08 ± 84%      +0.4        0.45 ± 42%  perf-profile.children.cycles-pp.handle_mm_fault
      1.79 ±  8%      +0.4        2.22 ±  6%  perf-profile.children.cycles-pp.mod_objcg_state
      0.00            +0.4        0.43 ± 22%  perf-profile.children.cycles-pp.intel_idle
      0.58 ± 12%      +0.4        1.01 ±  7%  perf-profile.children.cycles-pp.mutex_unlock
      0.76 ±  7%      +0.4        1.21 ±  7%  perf-profile.children.cycles-pp.skb_queue_tail
      0.01 ±223%      +0.5        0.46 ± 22%  perf-profile.children.cycles-pp.cpuidle_enter_state
      0.01 ±223%      +0.5        0.46 ± 22%  perf-profile.children.cycles-pp.cpuidle_enter
      0.60 ± 12%      +0.5        1.06 ±  7%  perf-profile.children.cycles-pp.skb_unlink
      1.52            +0.5        1.99 ±  2%  perf-profile.children.cycles-pp.skb_copy_datagram_iter
      1.50 ±  2%      +0.5        1.98 ±  2%  perf-profile.children.cycles-pp.__skb_datagram_iter
      1.53            +0.5        2.01 ±  2%  perf-profile.children.cycles-pp.unix_stream_read_actor
      0.65 ±  5%      +0.5        1.15 ±  6%  perf-profile.children.cycles-pp.skb_set_owner_w
      0.02 ±144%      +0.6        0.58 ± 21%  perf-profile.children.cycles-pp.start_secondary
      0.02 ±144%      +0.6        0.59 ± 21%  perf-profile.children.cycles-pp.secondary_startup_64_no_verify
      0.02 ±144%      +0.6        0.59 ± 21%  perf-profile.children.cycles-pp.cpu_startup_entry
      0.02 ±144%      +0.6        0.59 ± 21%  perf-profile.children.cycles-pp.do_idle
      1.16 ±  9%      +0.8        1.99 ±  4%  perf-profile.children.cycles-pp.sock_wfree
      1.21 ±  8%      +0.8        2.04 ±  4%  perf-profile.children.cycles-pp.unix_destruct_scm
      1.22 ±  8%      +0.8        2.06 ±  4%  perf-profile.children.cycles-pp.skb_release_head_state
      1.23 ±  8%      +0.8        2.07 ±  4%  perf-profile.children.cycles-pp.skb_release_all
      1.58 ±  7%      +1.0        2.53 ±  8%  perf-profile.children.cycles-pp.get_mem_cgroup_from_objcg
      1.47 ±  9%      +1.1        2.54 ±  5%  perf-profile.children.cycles-pp.__slab_free
      2.99 ±  5%      +1.1        4.13 ±  4%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      7.88 ±  8%      +2.4       10.33 ±  4%  perf-profile.children.cycles-pp.kmem_cache_alloc_node
     47.00            +3.5       50.55        perf-profile.children.cycles-pp.__libc_read
      7.43 ± 12%      +3.7       11.08 ±  7%  perf-profile.children.cycles-pp.propagate_protected_usage
     44.84            +3.9       48.70        perf-profile.children.cycles-pp.vfs_read
     45.04            +3.9       48.91        perf-profile.children.cycles-pp.ksys_read
     43.69            +3.9       47.56        perf-profile.children.cycles-pp.unix_stream_recvmsg
     43.61            +3.9       47.49        perf-profile.children.cycles-pp.unix_stream_read_generic
     44.12            +3.9       48.00        perf-profile.children.cycles-pp.sock_read_iter
     44.34            +3.9       48.24        perf-profile.children.cycles-pp.new_sync_read
      7.55 ±  9%      +4.1       11.66 ±  4%  perf-profile.children.cycles-pp.kmem_cache_free
     14.24 ±  9%      +4.5       18.76 ±  4%  perf-profile.children.cycles-pp.kmalloc_reserve
     14.20 ±  9%      +4.5       18.72 ±  4%  perf-profile.children.cycles-pp.__kmalloc_node_track_caller
     11.62 ± 11%      +5.2       16.83 ±  6%  perf-profile.children.cycles-pp.page_counter_cancel
     12.06 ±  9%      +5.9       17.92 ±  4%  perf-profile.children.cycles-pp.kfree
     18.74 ± 10%      +6.4       25.17 ±  5%  perf-profile.children.cycles-pp.obj_cgroup_charge_pages
     18.83 ± 10%      +6.5       25.28 ±  5%  perf-profile.children.cycles-pp.obj_cgroup_charge
     13.12 ±  9%      +6.5       19.62 ±  4%  perf-profile.children.cycles-pp.skb_release_data
     22.71 ±  8%      +7.1       29.84 ±  4%  perf-profile.children.cycles-pp.__alloc_skb
     22.76 ±  8%      +7.1       29.89 ±  4%  perf-profile.children.cycles-pp.alloc_skb_with_frags
     14.56 ±  9%      +7.5       22.04 ±  4%  perf-profile.children.cycles-pp.consume_skb
     14.92 ± 11%      +7.6       22.56 ±  5%  perf-profile.children.cycles-pp.page_counter_uncharge
     23.80 ±  8%      +8.0       31.76 ±  4%  perf-profile.children.cycles-pp.sock_alloc_send_pskb
     16.54 ± 11%      +8.6       25.14 ±  5%  perf-profile.children.cycles-pp.obj_cgroup_uncharge_pages
      0.00           +22.0       22.04 ±  5%  perf-profile.children.cycles-pp.page_counter_charge
     12.79 ± 10%     -12.6        0.17 ±  9%  perf-profile.self.cycles-pp.page_counter_try_charge
      4.56 ± 13%      -3.1        1.44 ± 33%  perf-profile.self.cycles-pp.unwind_next_frame
      3.43 ± 14%      -2.3        1.14 ± 32%  perf-profile.self.cycles-pp.__get_user_nocheck_8
      2.98 ± 13%      -2.0        0.99 ± 32%  perf-profile.self.cycles-pp.__orc_find
      3.89 ± 10%      -1.5        2.38 ± 21%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      1.61 ± 13%      -1.1        0.53 ± 32%  perf-profile.self.cycles-pp.orc_find
      1.22 ± 12%      -0.8        0.39 ± 34%  perf-profile.self.cycles-pp.perf_callchain_kernel
      1.20 ± 10%      -0.8        0.44 ± 28%  perf-profile.self.cycles-pp.native_irq_return_iret
      0.92 ± 13%      -0.6        0.29 ± 33%  perf-profile.self.cycles-pp.stack_access_ok
      1.54 ±  5%      -0.6        0.92 ±  6%  perf-profile.self.cycles-pp.switch_mm_irqs_off
      0.82 ±  9%      -0.4        0.45 ±  9%  perf-profile.self.cycles-pp.__schedule
      0.50 ± 14%      -0.3        0.16 ± 30%  perf-profile.self.cycles-pp.cmp_ex_search
      0.50 ± 12%      -0.3        0.16 ± 30%  perf-profile.self.cycles-pp.__unwind_start
      0.48 ± 13%      -0.3        0.16 ± 31%  perf-profile.self.cycles-pp.kernel_text_address
      0.49 ± 13%      -0.3        0.22 ± 10%  perf-profile.self.cycles-pp.update_curr
      0.36 ± 12%      -0.3        0.11 ± 31%  perf-profile.self.cycles-pp.unwind_get_return_address
      0.44 ± 10%      -0.3        0.19 ± 10%  perf-profile.self.cycles-pp.load_new_mm_cr3
      0.34 ± 13%      -0.2        0.10 ± 49%  perf-profile.self.cycles-pp.bsearch
      0.30 ± 17%      -0.2        0.08 ± 47%  perf-profile.self.cycles-pp.perf_callchain_user
      0.45 ±  3%      -0.2        0.25 ±  7%  perf-profile.self.cycles-pp.__switch_to
      0.27 ±  9%      -0.2        0.08 ± 47%  perf-profile.self.cycles-pp.get_perf_callchain
      0.33 ± 10%      -0.2        0.14 ± 21%  perf-profile.self.cycles-pp.perf_tp_event
      0.28 ± 12%      -0.2        0.11 ± 30%  perf-profile.self.cycles-pp.perf_output_begin_forward
      0.28 ±  9%      -0.2        0.11 ± 27%  perf-profile.self.cycles-pp.__task_pid_nr_ns
      0.26 ± 12%      -0.2        0.09 ± 48%  perf-profile.self.cycles-pp.ftrace_graph_ret_addr
      0.26 ± 12%      -0.2        0.10 ± 46%  perf-profile.self.cycles-pp.__kernel_text_address
      0.26 ± 17%      -0.2        0.10 ± 21%  perf-profile.self.cycles-pp.available_idle_cpu
      0.27 ± 10%      -0.2        0.11 ± 27%  perf-profile.self.cycles-pp.perf_prepare_sample
      0.26 ±  9%      -0.2        0.10 ± 19%  perf-profile.self.cycles-pp.native_sched_clock
      0.22 ± 11%      -0.2        0.07 ± 48%  perf-profile.self.cycles-pp.asm_exc_page_fault
      0.30 ±  6%      -0.1        0.16 ±  3%  perf-profile.self.cycles-pp.new_sync_write
      0.28 ± 10%      -0.1        0.14 ±  9%  perf-profile.self.cycles-pp.__switch_to_asm
      0.22 ± 12%      -0.1        0.08 ± 49%  perf-profile.self.cycles-pp.perf_trace_sched_switch
      0.40 ± 12%      -0.1        0.27 ±  9%  perf-profile.self.cycles-pp.update_load_avg
      0.28 ± 11%      -0.1        0.14 ± 14%  perf-profile.self.cycles-pp.try_to_wake_up
      0.21 ± 16%      -0.1        0.08 ±  8%  perf-profile.self.cycles-pp.__enqueue_entity
      0.24 ± 11%      -0.1        0.11 ± 13%  perf-profile.self.cycles-pp.__restore_fpregs_from_fpstate
      0.24 ± 12%      -0.1        0.12 ± 10%  perf-profile.self.cycles-pp.__update_load_avg_se
      0.17 ± 13%      -0.1        0.05 ± 47%  perf-profile.self.cycles-pp.error_entry
      0.22 ± 14%      -0.1        0.11 ± 16%  perf-profile.self.cycles-pp.perf_trace_sched_stat_runtime
      0.20 ± 16%      -0.1        0.09 ± 16%  perf-profile.self.cycles-pp.cpuacct_charge
      0.21 ± 14%      -0.1        0.10 ± 14%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.22 ±  6%      -0.1        0.12 ±  8%  perf-profile.self.cycles-pp.switch_fpu_return
      0.13 ± 14%      -0.1        0.03 ±102%  perf-profile.self.cycles-pp.perf_tp_event_match
      0.28 ±  6%      -0.1        0.18 ±  5%  perf-profile.self.cycles-pp.prepare_to_wait
      0.17 ± 10%      -0.1        0.07 ± 12%  perf-profile.self.cycles-pp.reweight_entity
      0.13 ±  8%      -0.1        0.04 ± 71%  perf-profile.self.cycles-pp.get_callchain_entry
      0.13 ± 14%      -0.1        0.04 ± 75%  perf-profile.self.cycles-pp.save_fpregs_to_fpstate
      0.14 ± 15%      -0.1        0.05 ± 49%  perf-profile.self.cycles-pp.bpf_ksym_find
      0.19 ± 11%      -0.1        0.10 ±  5%  perf-profile.self.cycles-pp.pick_next_task_fair
      0.16 ± 14%      -0.1        0.08 ± 13%  perf-profile.self.cycles-pp.update_rq_clock
      0.16 ± 11%      -0.1        0.08 ±  5%  perf-profile.self.cycles-pp.enqueue_entity
      0.15 ± 13%      -0.1        0.07 ± 12%  perf-profile.self.cycles-pp.select_idle_sibling
      0.34 ±  5%      -0.1        0.26 ±  3%  perf-profile.self.cycles-pp.sock_write_iter
      0.15 ± 16%      -0.1        0.07 ± 18%  perf-profile.self.cycles-pp.select_idle_cpu
      0.16 ±  4%      -0.1        0.08 ±  8%  perf-profile.self.cycles-pp.resched_curr
      0.15 ± 10%      -0.1        0.08 ±  4%  perf-profile.self.cycles-pp.select_task_rq_fair
      0.14 ± 14%      -0.1        0.08 ± 16%  perf-profile.self.cycles-pp.enqueue_task_fair
      0.13 ±  8%      -0.1        0.06 ± 11%  perf-profile.self.cycles-pp.schedule
      0.10 ± 14%      -0.1        0.04 ± 71%  perf-profile.self.cycles-pp.tracing_gen_ctx_irq_test
      0.14 ± 10%      -0.1        0.07 ±  6%  perf-profile.self.cycles-pp.dequeue_task_fair
      0.12 ± 14%      -0.1        0.06 ±  7%  perf-profile.self.cycles-pp.update_min_vruntime
      0.13 ±  7%      -0.1        0.08 ±  7%  perf-profile.self.cycles-pp.do_syscall_64
      0.29 ±  6%      -0.0        0.24 ±  2%  perf-profile.self.cycles-pp.__libc_read
      0.13 ± 12%      -0.0        0.08 ±  8%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.10 ±  9%      -0.0        0.05 ±  8%  perf-profile.self.cycles-pp.__list_add_valid
      0.11 ± 14%      -0.0        0.06 ± 14%  perf-profile.self.cycles-pp.perf_trace_sched_wakeup_template
      0.10 ±  7%      -0.0        0.06 ±  6%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.08 ± 10%      -0.0        0.03 ± 70%  perf-profile.self.cycles-pp.iov_iter_init
      0.10 ±  7%      -0.0        0.06 ±  7%  perf-profile.self.cycles-pp.__calc_delta
      0.09 ±  7%      -0.0        0.05 ±  8%  perf-profile.self.cycles-pp.finish_task_switch
      0.24 ±  3%      -0.0        0.20 ±  3%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.08 ± 10%      -0.0        0.04 ± 45%  perf-profile.self.cycles-pp.dequeue_entity
      0.08 ±  8%      -0.0        0.04 ± 45%  perf-profile.self.cycles-pp.check_preempt_wakeup
      0.08 ± 11%      -0.0        0.05 ± 46%  perf-profile.self.cycles-pp.__perf_event_header__init_id
      0.07 ±  5%      -0.0        0.04 ± 44%  perf-profile.self.cycles-pp.sock_sendmsg
      0.11 ±  6%      -0.0        0.09 ±  5%  perf-profile.self.cycles-pp.exit_to_user_mode_prepare
      0.07 ± 10%      -0.0        0.05 ±  7%  perf-profile.self.cycles-pp.schedule_timeout
      0.21 ±  2%      -0.0        0.18 ±  4%  perf-profile.self.cycles-pp.sock_read_iter
      0.08 ±  4%      -0.0        0.06 ±  7%  perf-profile.self.cycles-pp.rcu_read_unlock_strict
      0.23 ±  3%      -0.0        0.21 ±  3%  perf-profile.self.cycles-pp.__libc_write
      0.13 ±  6%      -0.0        0.11 ±  6%  perf-profile.self.cycles-pp.security_file_permission
      0.06 ±  6%      +0.0        0.07        perf-profile.self.cycles-pp.refill_obj_stock
      0.05 ±  7%      +0.0        0.06 ±  7%  perf-profile.self.cycles-pp.check_stack_object
      0.09 ±  7%      +0.0        0.11 ±  6%  perf-profile.self.cycles-pp.obj_cgroup_charge
      0.05 ±  8%      +0.0        0.07        perf-profile.self.cycles-pp.__cond_resched
      0.05 ±  7%      +0.0        0.07 ±  8%  perf-profile.self.cycles-pp.rcu_all_qs
      0.10 ±  4%      +0.0        0.12 ±  6%  perf-profile.self.cycles-pp._copy_from_iter
      0.18 ±  2%      +0.0        0.20 ±  4%  perf-profile.self.cycles-pp.new_sync_read
      0.14 ±  3%      +0.0        0.16 ±  6%  perf-profile.self.cycles-pp.__ksize
      0.16            +0.0        0.19 ±  3%  perf-profile.self.cycles-pp.__might_sleep
      0.19 ±  4%      +0.0        0.22 ±  3%  perf-profile.self.cycles-pp.aa_sk_perm
      0.18 ±  4%      +0.0        0.21 ±  4%  perf-profile.self.cycles-pp.__virt_addr_valid
      0.17 ±  2%      +0.0        0.21 ±  3%  perf-profile.self.cycles-pp.__entry_text_start
      0.27 ±  2%      +0.0        0.31 ±  3%  perf-profile.self.cycles-pp.__alloc_skb
      0.34            +0.0        0.38 ±  4%  perf-profile.self.cycles-pp.kmem_cache_alloc_node
      0.04 ± 45%      +0.0        0.09 ±  9%  perf-profile.self.cycles-pp.skb_copy_datagram_from_iter
      0.21 ±  2%      +0.0        0.26 ±  4%  perf-profile.self.cycles-pp.__check_heap_object
      0.50 ±  2%      +0.1        0.54 ±  2%  perf-profile.self.cycles-pp.__kmalloc_node_track_caller
      0.46 ±  4%      +0.1        0.52 ±  5%  perf-profile.self.cycles-pp.unix_stream_sendmsg
      0.36            +0.1        0.42 ±  2%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.32 ±  2%      +0.1        0.38 ±  5%  perf-profile.self.cycles-pp.___slab_alloc
      0.22 ±  5%      +0.1        0.28 ±  6%  perf-profile.self.cycles-pp.unix_write_space
      0.00            +0.1        0.07 ±  7%  perf-profile.self.cycles-pp.refill_stock
      0.14 ± 16%      +0.1        0.21 ±  3%  perf-profile.self.cycles-pp.page_counter_uncharge
      0.19 ±  6%      +0.1        0.27 ±  5%  perf-profile.self.cycles-pp.__build_skb_around
      0.40 ±  2%      +0.1        0.50 ±  4%  perf-profile.self.cycles-pp.kfree
      0.37 ±  4%      +0.1        0.49 ±  6%  perf-profile.self.cycles-pp.mutex_lock
      0.45 ±  3%      +0.1        0.59 ±  2%  perf-profile.self.cycles-pp.__check_object_size
      0.21 ±  8%      +0.1        0.35 ±  9%  perf-profile.self.cycles-pp.consume_skb
      0.02 ±223%      +0.1        0.16 ± 45%  perf-profile.self.cycles-pp.page_vma_mapped_walk
      0.20 ± 11%      +0.2        0.36 ±  7%  perf-profile.self.cycles-pp.skb_release_data
      1.02 ±  3%      +0.2        1.22 ±  4%  perf-profile.self.cycles-pp.unix_stream_read_generic
      0.58 ±  2%      +0.2        0.78 ±  3%  perf-profile.self.cycles-pp.kmem_cache_free
      0.86 ±  2%      +0.3        1.19 ±  3%  perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      1.25 ±  9%      +0.4        1.60 ±  6%  perf-profile.self.cycles-pp.__mod_memcg_lruvec_state
      0.00            +0.4        0.43 ± 22%  perf-profile.self.cycles-pp.intel_idle
      0.57 ± 12%      +0.4        1.00 ±  7%  perf-profile.self.cycles-pp.mutex_unlock
      0.76 ±  8%      +0.4        1.20 ±  8%  perf-profile.self.cycles-pp.obj_cgroup_uncharge_pages
      0.64 ±  4%      +0.5        1.14 ±  6%  perf-profile.self.cycles-pp.skb_set_owner_w
      0.73 ±  8%      +0.5        1.25 ±  9%  perf-profile.self.cycles-pp.obj_cgroup_charge_pages
      0.74 ±  9%      +0.6        1.30 ±  7%  perf-profile.self.cycles-pp.sock_wfree
      0.81 ± 11%      +0.6        1.42 ±  7%  perf-profile.self.cycles-pp.sock_def_readable
      2.22 ±  4%      +0.7        2.88 ±  6%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      2.79 ±  4%      +0.9        3.68 ±  5%  perf-profile.self.cycles-pp._raw_spin_lock
      1.56 ±  7%      +0.9        2.50 ±  8%  perf-profile.self.cycles-pp.get_mem_cgroup_from_objcg
      1.45 ±  9%      +1.1        2.51 ±  5%  perf-profile.self.cycles-pp.__slab_free
      7.36 ± 12%      +3.6       10.97 ±  7%  perf-profile.self.cycles-pp.propagate_protected_usage
     11.50 ± 11%      +5.2       16.66 ±  6%  perf-profile.self.cycles-pp.page_counter_cancel
      0.00           +16.7       16.74 ±  6%  perf-profile.self.cycles-pp.page_counter_charge


                                                                                
                                hackbench.throughput                            
                                                                                
  145000 +------------------------------------------------------------------+   
         |                                                      O           |   
  140000 |-+                                               O   O         O  |   
         |                                 OO   OO   O  O O          O O  O |   
  135000 |-O          O O      OO O           O    O              O         |   
         |  O O  O   O       O         O              O                     |   
  130000 |-+    O                    O                       O      O       |   
         |                OO             O      ++     .+.+                 |   
  125000 |.++.+.+          +.+.++.+.++.+.+.++   : +   +    +.+.+   .+       |   
         |       +   ++.+  :        O        : :   +.+          +.+         |   
  120000 |-+      :O :   : :                 : :                            |   
         |        : :    ::                   :                             |   
  115000 |-+       ::     :                   +                             |   
         |         +      +                                                 |   
  110000 +------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                       hackbench.time.voluntary_context_switches                
                                                                                
  1.4e+09 +-----------------------------------------------------------------+   
          |           +                                                     |   
  1.3e+09 |-+     +  ::                                                     |   
  1.2e+09 |-+     :+ : :                                                    |   
          |      :  +  +.+.                                                 |   
  1.1e+09 |.++. .+         ++.++. .++.+.+ .+         +                      |   
    1e+09 |-+  +                 +       +  +       + :                     |   
          |                                  +     +  :    .++.  .+.        |   
    9e+08 |-+                                 +.+. :   ++.+    ++   +       |   
    8e+08 |-+                                     +                         |   
          |                                                                 |   
    7e+08 |-+                                                               |   
    6e+08 |-+                                                               |   
          | OO O OO O OO O OO OO O OO O OO O OO O OO O OO O  O OO O OO O OO |   
    5e+08 +-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                      hackbench.time.involuntary_context_switches               
                                                                                
  6.5e+08 +-----------------------------------------------------------------+   
    6e+08 |-+         +                                                     |   
          |       +   :                                                     |   
  5.5e+08 |-+     :: : :                                                    |   
    5e+08 |-+     :: : +.+.                                                 |   
          |.     :  +      ++.      +.                                      |   
  4.5e+08 |-++. .+            ++. .+  +.+ .+        .+                      |   
    4e+08 |-+  +                 +       +  +      +  :                     |   
  3.5e+08 |-+                                +     :  :    .++.  .+.        |   
          |                                   +.+.:    ++.+    ++   +       |   
    3e+08 |-+                                     +                         |   
  2.5e+08 |-+                                                               |   
          |                                                                 |   
    2e+08 |-+    O  O       O         O OO                                  |   
  1.5e+08 +-----------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


---
0DAY/LKP+ Test Infrastructure                   Open Source Technology Center
https://lists.01.org/hyperkitty/list/lkp@lists.01.org       Intel Corporation

Thanks,
Oliver Sang


View attachment "config-5.15.0-00059-g58056f77502f" of type "text/plain" (172640 bytes)

View attachment "job-script" of type "text/plain" (8136 bytes)

View attachment "job.yaml" of type "text/plain" (5583 bytes)

View attachment "reproduce" of type "text/plain" (530 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ