lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Message-ID: <202210121629.587ff60f-yujie.liu@intel.com>
Date:   Wed, 12 Oct 2022 16:30:20 +0800
From:   kernel test robot <yujie.liu@...el.com>
To:     Jiebin Sun <jiebin.sun@...el.com>
CC:     <lkp@...ts.01.org>, <lkp@...el.com>,
        <linux-kernel@...r.kernel.org>,
        "Tim Chen" <tim.c.chen@...ux.intel.com>,
        Alexander Mikhalitsyn <alexander.mikhalitsyn@...tuozzo.com>,
        Alexey Gladkov <legion@...nel.org>,
        Christoph Lameter <cl@...ux.com>,
        Davidlohr Bueso <dave@...olabs.net>,
        "Dennis Zhou" <dennis@...nel.org>,
        "Eric W . Biederman" <ebiederm@...ssion.com>,
        Manfred Spraul <manfred@...orfullife.com>,
        Shakeel Butt <shakeelb@...gle.com>, Tejun Heo <tj@...nel.org>,
        Vasily Averin <vasily.averin@...ux.dev>,
        Andrew Morton <akpm@...ux-foundation.org>,
        <ying.huang@...el.com>, <feng.tang@...el.com>,
        <zhengjun.xing@...ux.intel.com>, <fengwei.yin@...el.com>
Subject: [ipc/msg] 72d1e61108: stress-ng.msg.ops_per_sec 262.8% improvement

Greeting,

FYI, we noticed a 262.8% improvement of stress-ng.msg.ops_per_sec due to commit:

commit: 72d1e611082eda18689106a0c192f2827072713c ("ipc/msg: mitigate the lock contention with percpu counter")
https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything

in testcase: stress-ng
on test machine: 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz (Cascade Lake) with 512G memory
with following parameters:

	nr_threads: 100%
	testtime: 60s
	sc_pid_max: 4194304
	class: scheduler
	test: msg
	cpufreq_governor: performance


Details are as below:

=========================================================================================
class/compiler/cpufreq_governor/kconfig/nr_threads/rootfs/sc_pid_max/tbox_group/test/testcase/testtime:
  scheduler/gcc-11/performance/x86_64-rhel-8.3/100%/debian-11.1-x86_64-20220510.cgz/4194304/lkp-csl-2sp7/msg/stress-ng/60s

commit: 
  5d0ce3595a ("percpu: add percpu_counter_add_local and percpu_counter_sub_local")
  72d1e61108 ("ipc/msg: mitigate the lock contention with percpu counter")

5d0ce3595ab75330 72d1e611082eda18689106a0c19 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
 3.141e+08          +263.2%  1.141e+09        stress-ng.msg.ops
   5192613          +262.8%   18837790        stress-ng.msg.ops_per_sec
    260380           +42.8%     371816 ±  3%  stress-ng.time.involuntary_context_switches
     22498           -10.5%      20124        stress-ng.time.minor_page_faults
      6813           -31.5%       4669        stress-ng.time.percent_of_cpu_this_job_got
      4222           -33.8%       2794        stress-ng.time.system_time
     52.03          +162.4%     136.53 ±  3%  stress-ng.time.user_time
  47960019           -35.2%   31093209        stress-ng.time.voluntary_context_switches
     20.33            -0.8%      20.17        boot-time.dhcp
      4844           +27.3%       6167        uptime.idle
  1.75e+09           +77.6%  3.109e+09        cpuidle..time
  41428140           -36.2%   26440706        cpuidle..usage
     26987 ±  5%     -12.6%      23598 ±  5%  meminfo.Active
     26350 ±  6%     -12.9%      22961 ±  6%  meminfo.Active(anon)
     36104 ±  6%     -14.9%      30713 ±  5%  meminfo.Shmem
     28.42           +22.3       50.70        mpstat.cpu.all.idle%
      1.14 ±  2%      -0.1        1.00 ±  2%  mpstat.cpu.all.irq%
     69.22           -23.5       45.71        mpstat.cpu.all.sys%
      1.12            +1.4        2.49 ±  3%  mpstat.cpu.all.usr%
   2225561 ± 41%     -58.6%     920422 ±132%  numa-meminfo.node0.Unevictable
     25879 ±  6%     -15.6%      21834 ±  7%  numa-meminfo.node1.Active
     25243 ±  6%     -14.3%      21622 ±  7%  numa-meminfo.node1.Active(anon)
     33938 ±  7%     -19.4%      27346 ± 10%  numa-meminfo.node1.Shmem
     30.00           +72.2%      51.67        vmstat.cpu.id
     72.33 ±  2%     -40.3%      43.17 ±  7%  vmstat.procs.r
   1408523           -35.3%     910912        vmstat.system.cs
    363813           -15.3%     308282        vmstat.system.in
    556389 ± 41%     -58.6%     230105 ±132%  numa-vmstat.node0.nr_unevictable
    556389 ± 41%     -58.6%     230105 ±132%  numa-vmstat.node0.nr_zone_unevictable
      6310 ±  6%     -14.3%       5405 ±  7%  numa-vmstat.node1.nr_active_anon
      8484 ±  7%     -19.4%       6836 ± 10%  numa-vmstat.node1.nr_shmem
      6310 ±  6%     -14.3%       5405 ±  7%  numa-vmstat.node1.nr_zone_active_anon
      6587 ±  6%     -12.9%       5740 ±  6%  proc-vmstat.nr_active_anon
     10635 ±  2%      -6.1%       9987        proc-vmstat.nr_mapped
      9025 ±  6%     -14.9%       7677 ±  5%  proc-vmstat.nr_shmem
      6587 ±  6%     -12.9%       5740 ±  6%  proc-vmstat.nr_zone_active_anon
     17181 ±  6%     -16.6%      14323 ±  6%  proc-vmstat.pgactivate
      2066           -30.4%       1437        turbostat.Avg_MHz
     74.06           -22.5       51.58        turbostat.Busy%
  23899322           -74.2%    6171939 ± 21%  turbostat.C1
   5624190 ±  3%    +232.6%   18703687 ±  7%  turbostat.C1E
     16.26 ± 15%     +22.3       38.57 ±  8%  turbostat.C1E%
     25.52           +86.3%      47.55        turbostat.CPU%c1
      0.04          +225.0%       0.13        turbostat.IPC
  24328141           -15.3%   20609928        turbostat.IRQ
  11586443           -88.5%    1326977 ±  8%  turbostat.POLL
      0.82            -0.5        0.28 ±  9%  turbostat.POLL%
     56.33            +3.8%      58.50        turbostat.PkgTmp
    239.06            +5.8%     252.82        turbostat.PkgWatt
     38.95            +6.6%      41.53        turbostat.RAMWatt
    762.67 ± 29%     -29.3%     539.17        sched_debug.cfs_rq:/.load_avg.max
   1449885           -51.6%     701690        sched_debug.cfs_rq:/.min_vruntime.avg
   1555986           -52.4%     740207        sched_debug.cfs_rq:/.min_vruntime.max
   1350339           -49.0%     689030        sched_debug.cfs_rq:/.min_vruntime.min
     69465 ± 12%     -87.8%       8445 ± 12%  sched_debug.cfs_rq:/.min_vruntime.stddev
    535.56 ±  4%     -25.4%     399.38 ±  2%  sched_debug.cfs_rq:/.runnable_avg.avg
      1895 ± 12%     -33.0%       1270 ± 12%  sched_debug.cfs_rq:/.runnable_avg.max
    357.02 ±  7%     -23.8%     272.01 ±  5%  sched_debug.cfs_rq:/.runnable_avg.stddev
     53257 ± 18%    -109.8%      -5230        sched_debug.cfs_rq:/.spread0.avg
    159409 ± 16%     -79.1%      33308 ± 39%  sched_debug.cfs_rq:/.spread0.max
    -46234           -61.2%     -17921        sched_debug.cfs_rq:/.spread0.min
     69434 ± 12%     -87.8%       8441 ± 12%  sched_debug.cfs_rq:/.spread0.stddev
    495.72 ±  4%     -20.1%     396.07 ±  2%  sched_debug.cfs_rq:/.util_avg.avg
      1796 ± 12%     -29.7%       1261 ± 12%  sched_debug.cfs_rq:/.util_avg.max
    338.68 ±  7%     -20.1%     270.73 ±  5%  sched_debug.cfs_rq:/.util_avg.stddev
    129.36 ± 26%     -45.5%      70.54 ±  6%  sched_debug.cfs_rq:/.util_est_enqueued.avg
    897.17 ±  9%     -23.5%     686.00 ± 12%  sched_debug.cfs_rq:/.util_est_enqueued.max
    165.67 ±  5%     -21.9%     129.32 ±  3%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
      3063 ± 12%    +128.6%       7003 ± 38%  sched_debug.cpu.avg_idle.min
    203936 ±  5%     -17.7%     167885 ±  3%  sched_debug.cpu.avg_idle.stddev
      0.00 ± 10%     -25.3%       0.00 ± 11%  sched_debug.cpu.next_balance.stddev
    468755           -35.1%     304073        sched_debug.cpu.nr_switches.avg
    694515 ±  3%     -45.5%     378328        sched_debug.cpu.nr_switches.max
    328982 ±  3%     -16.6%     274253        sched_debug.cpu.nr_switches.min
    112645 ± 11%     -83.0%      19135 ±  9%  sched_debug.cpu.nr_switches.stddev
 2.185e+09 ±  4%     -23.0%  1.681e+09 ±  8%  sched_debug.cpu.nr_uninterruptible.avg
 5.392e+09          +138.5%  1.286e+10        perf-stat.i.branch-instructions
      1.17            -0.3        0.85        perf-stat.i.branch-miss-rate%
  58802448           +58.7%   93338979        perf-stat.i.branch-misses
     22.60            -2.2       20.40        perf-stat.i.cache-miss-rate%
  92976185          +122.5%  2.069e+08        perf-stat.i.cache-misses
  4.09e+08          +147.2%  1.011e+09        perf-stat.i.cache-references
   1487973           -35.3%     962275        perf-stat.i.context-switches
      7.73           -71.8%       2.18        perf-stat.i.cpi
 2.044e+11           -30.8%  1.415e+11        perf-stat.i.cpu-cycles
     55597          +101.4%     111973        perf-stat.i.cpu-migrations
      2209           -66.7%     735.14 ±  2%  perf-stat.i.cycles-between-cache-misses
      0.02 ± 10%      -0.0        0.01 ±  6%  perf-stat.i.dTLB-load-miss-rate%
   1268585 ±  7%     +81.9%    2308078 ±  8%  perf-stat.i.dTLB-load-misses
 6.963e+09          +146.5%  1.716e+10        perf-stat.i.dTLB-loads
      0.00 ±  8%      -0.0        0.00 ±  8%  perf-stat.i.dTLB-store-miss-rate%
    146770 ±  3%    +117.8%     319631 ±  4%  perf-stat.i.dTLB-store-misses
 3.623e+09          +177.1%  1.004e+10        perf-stat.i.dTLB-stores
     54.35           +16.9       71.22        perf-stat.i.iTLB-load-miss-rate%
   9731875          +119.0%   21313415 ±  4%  perf-stat.i.iTLB-load-misses
  2.59e+10          +147.2%  6.403e+10        perf-stat.i.instructions
      2979           +10.8%       3301 ±  3%  perf-stat.i.instructions-per-iTLB-miss
      0.16          +199.8%       0.47        perf-stat.i.ipc
      2.13           -30.8%       1.47        perf-stat.i.metric.GHz
    460.63          +126.4%       1042        perf-stat.i.metric.K/sec
    170.65          +150.7%     427.80        perf-stat.i.metric.M/sec
     95.23            +1.1       96.30        perf-stat.i.node-load-miss-rate%
  21792545          +192.0%   63637628        perf-stat.i.node-load-misses
    855255 ±  2%    +122.2%    1900158        perf-stat.i.node-loads
  10944429          +120.3%   24108332        perf-stat.i.node-store-misses
      1.09            -0.4        0.73        perf-stat.overall.branch-miss-rate%
     22.72            -2.3       20.46        perf-stat.overall.cache-miss-rate%
      7.89           -72.0%       2.21        perf-stat.overall.cpi
      2198           -68.9%     684.17        perf-stat.overall.cycles-between-cache-misses
      0.02 ±  7%      -0.0        0.01 ±  7%  perf-stat.overall.dTLB-load-miss-rate%
      0.00 ±  3%      -0.0        0.00 ±  5%  perf-stat.overall.dTLB-store-miss-rate%
     55.25           +17.5       72.73        perf-stat.overall.iTLB-load-miss-rate%
      2661           +13.1%       3009 ±  4%  perf-stat.overall.instructions-per-iTLB-miss
      0.13          +257.0%       0.45        perf-stat.overall.ipc
 5.309e+09          +138.4%  1.266e+10        perf-stat.ps.branch-instructions
  57807111           +58.9%   91875333        perf-stat.ps.branch-misses
  91543136          +122.4%  2.036e+08        perf-stat.ps.cache-misses
  4.03e+08          +147.0%  9.953e+08        perf-stat.ps.cache-references
   1466557           -35.4%     947266        perf-stat.ps.context-switches
 2.013e+11           -30.8%  1.393e+11        perf-stat.ps.cpu-cycles
     54749          +101.3%     110229        perf-stat.ps.cpu-migrations
   1251114 ±  7%     +81.5%    2271328 ±  8%  perf-stat.ps.dTLB-load-misses
 6.857e+09          +146.4%   1.69e+10        perf-stat.ps.dTLB-loads
    144598 ±  3%    +117.6%     314612 ±  4%  perf-stat.ps.dTLB-store-misses
 3.569e+09          +176.9%  9.883e+09        perf-stat.ps.dTLB-stores
   9587082          +118.8%   20981028 ±  4%  perf-stat.ps.iTLB-load-misses
 2.551e+10          +147.1%  6.303e+10        perf-stat.ps.instructions
  21457539          +192.0%   62646387        perf-stat.ps.node-load-misses
    843598 ±  2%    +121.8%    1870813        perf-stat.ps.node-loads
  10776219          +120.2%   23732746        perf-stat.ps.node-store-misses
 1.641e+12          +147.4%  4.059e+12        perf-stat.total.instructions
     83.06           -24.3       58.74        perf-profile.calltrace.cycles-pp.stress_run
     45.22           -21.0       24.23        perf-profile.calltrace.cycles-pp.do_msgsnd.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_msgsnd.stress_run
     45.66           -19.5       26.12        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_msgsnd.stress_run
     45.72           -19.2       26.54        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_msgsnd.stress_run
     46.10           -18.1       27.97        perf-profile.calltrace.cycles-pp.__libc_msgsnd.stress_run
     12.26            -8.9        3.34        perf-profile.calltrace.cycles-pp._raw_spin_lock.do_msgrcv.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_msgrcv
     35.66            -8.6       27.03        perf-profile.calltrace.cycles-pp.do_msgrcv.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_msgrcv.stress_run
     35.96            -8.5       27.48        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_msgrcv.stress_run
     36.02            -8.3       27.70        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_msgrcv.stress_run
      9.60            -7.8        1.79        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.do_msgrcv.do_syscall_64.entry_SYSCALL_64_after_hwframe
     36.48            -7.5       29.01        perf-profile.calltrace.cycles-pp.__libc_msgrcv.stress_run
     10.97            -7.0        3.93        perf-profile.calltrace.cycles-pp._raw_spin_lock.do_msgsnd.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_msgsnd
      8.64 ±  2%      -6.4        2.29        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.do_msgsnd.do_syscall_64.entry_SYSCALL_64_after_hwframe
      4.79 ±  8%      -1.8        3.02 ±  7%  perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.mwait_idle_with_hints.intel_idle_irq.cpuidle_enter_state.cpuidle_enter
      2.12            -0.2        1.95 ±  4%  perf-profile.calltrace.cycles-pp.schedule.do_msgrcv.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_msgrcv
      2.10            -0.2        1.93 ±  4%  perf-profile.calltrace.cycles-pp.__schedule.schedule.do_msgrcv.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.66            -0.1        0.56 ±  2%  perf-profile.calltrace.cycles-pp.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      1.01            +0.2        1.22 ±  4%  perf-profile.calltrace.cycles-pp.try_to_wake_up.wake_up_q.do_msgsnd.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.58 ±  4%      +0.4        1.00 ±  7%  perf-profile.calltrace.cycles-pp.update_sg_lb_stats.update_sd_lb_stats.find_busiest_group.load_balance.newidle_balance
      1.12            +0.4        1.54 ±  3%  perf-profile.calltrace.cycles-pp.wake_up_q.do_msgsnd.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_msgsnd
      0.63 ±  5%      +0.4        1.08 ±  6%  perf-profile.calltrace.cycles-pp.update_sd_lb_stats.find_busiest_group.load_balance.newidle_balance.pick_next_task_fair
      0.64 ±  5%      +0.4        1.09 ±  6%  perf-profile.calltrace.cycles-pp.find_busiest_group.load_balance.newidle_balance.pick_next_task_fair.__schedule
      0.93 ±  4%      +0.5        1.40 ±  6%  perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.do_msgrcv.do_syscall_64
      0.91 ±  4%      +0.5        1.39 ±  6%  perf-profile.calltrace.cycles-pp.newidle_balance.pick_next_task_fair.__schedule.schedule.do_msgrcv
      0.76 ±  5%      +0.5        1.26 ±  6%  perf-profile.calltrace.cycles-pp.load_balance.newidle_balance.pick_next_task_fair.__schedule.schedule
      0.00            +0.5        0.53 ±  2%  perf-profile.calltrace.cycles-pp.__check_heap_object.__check_object_size.store_msg.do_msg_fill.do_msgrcv
      0.00            +0.5        0.55 ±  4%  perf-profile.calltrace.cycles-pp.get_obj_cgroup_from_current.__kmalloc.alloc_msg.load_msg.do_msgsnd
      0.00            +0.6        0.56        perf-profile.calltrace.cycles-pp.__schedule.schedule.rwsem_down_write_slowpath.msgctl_down.ksys_msgctl
      0.00            +0.6        0.56        perf-profile.calltrace.cycles-pp.schedule.rwsem_down_write_slowpath.msgctl_down.ksys_msgctl.do_syscall_64
      0.00            +0.6        0.57 ±  3%  perf-profile.calltrace.cycles-pp.__radix_tree_lookup.ipc_obtain_object_check.do_msgsnd.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +0.6        0.58 ±  3%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irq.rwsem_down_read_slowpath.sysvipc_proc_start.seq_read_iter
      0.00            +0.6        0.60        perf-profile.calltrace.cycles-pp.__schedule.schedule.rwsem_down_read_slowpath.msgctl_info.ksys_msgctl
      0.00            +0.6        0.60 ±  3%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irq.rwsem_down_write_slowpath.msgctl_down.ksys_msgctl
      0.00            +0.6        0.60 ±  2%  perf-profile.calltrace.cycles-pp.schedule.rwsem_down_read_slowpath.msgctl_info.ksys_msgctl.do_syscall_64
      0.00            +0.6        0.62 ±  3%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irq.rwsem_down_read_slowpath.sysvipc_proc_start.seq_read_iter.seq_read
      0.00            +0.7        0.66        perf-profile.calltrace.cycles-pp.rwsem_wake.msgctl_down.ksys_msgctl.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +0.7        0.67 ±  2%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irq.rwsem_down_write_slowpath.msgctl_down.ksys_msgctl.do_syscall_64
      0.00            +0.7        0.71 ±  2%  perf-profile.calltrace.cycles-pp.__entry_text_start.__libc_msgsnd.stress_run
      0.00            +0.7        0.72        perf-profile.calltrace.cycles-pp.__entry_text_start.__libc_msgrcv.stress_run
      0.00            +0.8        0.81        perf-profile.calltrace.cycles-pp.ss_wakeup.do_msgrcv.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_msgrcv
      0.00            +0.9        0.86        perf-profile.calltrace.cycles-pp._copy_to_user.store_msg.do_msg_fill.do_msgrcv.do_syscall_64
      0.00            +0.9        0.90        perf-profile.calltrace.cycles-pp.stress_msg.stress_run
      0.00            +0.9        0.92        perf-profile.calltrace.cycles-pp.__x64_sys_msgsnd.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_msgsnd.stress_run
      0.00            +0.9        0.94 ±  3%  perf-profile.calltrace.cycles-pp.osq_lock.rwsem_optimistic_spin.rwsem_down_write_slowpath.msgctl_down.ksys_msgctl
      0.00            +1.0        1.04 ±  2%  perf-profile.calltrace.cycles-pp.__check_object_size.load_msg.do_msgsnd.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +1.1        1.07 ±  3%  perf-profile.calltrace.cycles-pp.rwsem_down_read_slowpath.sysvipc_proc_start.seq_read_iter.seq_read.vfs_read
      0.00            +1.1        1.09        perf-profile.calltrace.cycles-pp.__list_del_entry_valid.do_msgrcv.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_msgrcv
      0.00            +1.1        1.10        perf-profile.calltrace.cycles-pp.check_heap_object.__check_object_size.store_msg.do_msg_fill.do_msgrcv
      0.00            +1.1        1.11 ±  2%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irq.rwsem_down_read_slowpath.msgctl_info.ksys_msgctl
      0.00            +1.1        1.14 ±  2%  perf-profile.calltrace.cycles-pp.sysvipc_proc_start.seq_read_iter.seq_read.vfs_read.ksys_read
      0.00            +1.2        1.21        perf-profile.calltrace.cycles-pp._raw_spin_lock_irq.rwsem_down_read_slowpath.msgctl_info.ksys_msgctl.do_syscall_64
      0.00            +1.3        1.28 ±  3%  perf-profile.calltrace.cycles-pp.rwsem_optimistic_spin.rwsem_down_write_slowpath.msgctl_down.ksys_msgctl.do_syscall_64
      0.00            +1.3        1.29        perf-profile.calltrace.cycles-pp._copy_from_user.load_msg.do_msgsnd.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.08 ±  2%      +1.4        2.50        perf-profile.calltrace.cycles-pp.__slab_free.free_msg.do_msgrcv.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +1.6        1.64        perf-profile.calltrace.cycles-pp.percpu_counter_add_batch.do_msgrcv.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_msgrcv
      0.00            +1.7        1.73        perf-profile.calltrace.cycles-pp.percpu_counter_add_batch.do_msgsnd.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_msgsnd
      0.60            +1.7        2.34        perf-profile.calltrace.cycles-pp.kfree.free_msg.do_msgrcv.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +1.8        1.83        perf-profile.calltrace.cycles-pp.__check_object_size.store_msg.do_msg_fill.do_msgrcv.do_syscall_64
      0.00            +1.9        1.88 ±  2%  perf-profile.calltrace.cycles-pp.seq_read_iter.seq_read.vfs_read.ksys_read.do_syscall_64
      0.00            +1.9        1.88 ±  2%  perf-profile.calltrace.cycles-pp.seq_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +1.9        1.90 ±  2%  perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
      0.00            +1.9        1.90 ±  2%  perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
      0.00            +1.9        1.91 ±  2%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.read
      0.00            +1.9        1.91 ±  2%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
      0.00            +1.9        1.92 ±  2%  perf-profile.calltrace.cycles-pp.read
      0.51 ±  2%      +2.0        2.49        perf-profile.calltrace.cycles-pp.memcg_slab_post_alloc_hook.__kmalloc.alloc_msg.load_msg.do_msgsnd
      0.00            +2.1        2.13        perf-profile.calltrace.cycles-pp.rwsem_down_read_slowpath.msgctl_info.ksys_msgctl.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +2.7        2.69        perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.msgctl_down.ksys_msgctl.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +2.7        2.70        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.__percpu_counter_sum.msgctl_info.ksys_msgctl
      0.82            +2.7        3.56        perf-profile.calltrace.cycles-pp.store_msg.do_msg_fill.do_msgrcv.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +2.8        2.76        perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.__percpu_counter_sum.msgctl_info.ksys_msgctl.do_syscall_64
      0.53 ±  2%      +3.1        3.65        perf-profile.calltrace.cycles-pp.msgctl_down.ksys_msgctl.do_syscall_64.entry_SYSCALL_64_after_hwframe.msgctl
      1.77            +3.3        5.05        perf-profile.calltrace.cycles-pp.free_msg.do_msgrcv.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_msgrcv
      0.96            +3.3        4.26        perf-profile.calltrace.cycles-pp.__kmalloc.alloc_msg.load_msg.do_msgsnd.do_syscall_64
      1.09            +3.4        4.46        perf-profile.calltrace.cycles-pp.do_msg_fill.do_msgrcv.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_msgrcv
      0.99            +3.4        4.40        perf-profile.calltrace.cycles-pp.alloc_msg.load_msg.do_msgsnd.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +3.6        3.55        perf-profile.calltrace.cycles-pp.__percpu_counter_sum.msgctl_info.ksys_msgctl.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +3.9        3.91        perf-profile.calltrace.cycles-pp.idr_find.ipc_obtain_object_check.do_msgsnd.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +4.0        4.00        perf-profile.calltrace.cycles-pp.idr_find.ipc_obtain_object_check.do_msgrcv.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.66            +5.3        6.96        perf-profile.calltrace.cycles-pp.load_msg.do_msgsnd.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_msgsnd
      0.78            +5.5        6.28        perf-profile.calltrace.cycles-pp.ipc_obtain_object_check.do_msgrcv.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_msgrcv
      0.84            +5.5        6.37        perf-profile.calltrace.cycles-pp.ipc_obtain_object_check.do_msgsnd.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_msgsnd
      0.00            +6.2        6.19        perf-profile.calltrace.cycles-pp.msgctl_info.ksys_msgctl.do_syscall_64.entry_SYSCALL_64_after_hwframe.msgctl
      1.04 ±  2%      +9.0       10.05        perf-profile.calltrace.cycles-pp.ksys_msgctl.do_syscall_64.entry_SYSCALL_64_after_hwframe.msgctl
      1.06 ±  2%      +9.0       10.10        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.msgctl
      1.06 ±  2%      +9.1       10.12        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.msgctl
      1.09 ±  2%      +9.1       10.21        perf-profile.calltrace.cycles-pp.msgctl
     15.18 ±  4%     +13.4       28.63        perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     15.18 ±  4%     +13.4       28.63        perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
     15.17 ±  4%     +13.5       28.62        perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     15.37 ±  4%     +13.5       28.92        perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
     13.06 ±  5%     +14.0       27.01        perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
     13.45 ±  5%     +14.0       27.41        perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     13.07 ±  5%     +14.0       27.04        perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
      6.33 ±  8%     +15.3       21.61 ±  2%  perf-profile.calltrace.cycles-pp.mwait_idle_with_hints.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
      6.33 ±  8%     +15.3       21.62 ±  2%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
     83.06           -24.3       58.74        perf-profile.children.cycles-pp.stress_run
     45.26           -21.0       24.29        perf-profile.children.cycles-pp.do_msgsnd
     46.20           -17.8       28.40        perf-profile.children.cycles-pp.__libc_msgsnd
     83.01           -17.2       65.78        perf-profile.children.cycles-pp.do_syscall_64
     83.14           -16.8       66.36        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     23.90           -16.1        7.80        perf-profile.children.cycles-pp._raw_spin_lock
     18.40            -9.0        9.43        perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
     35.69            -8.6       27.10        perf-profile.children.cycles-pp.do_msgrcv
     36.59            -7.1       29.47        perf-profile.children.cycles-pp.__libc_msgrcv
      1.17            -1.1        0.07 ±  5%  perf-profile.children.cycles-pp.wake_q_add
      3.08 ±  7%      -0.8        2.26 ±  5%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
      0.98 ±  4%      -0.5        0.45 ±  9%  perf-profile.children.cycles-pp.poll_idle
      0.83 ±  2%      -0.3        0.50        perf-profile.children.cycles-pp.schedule_idle
      0.41 ±  3%      -0.2        0.21 ±  3%  perf-profile.children.cycles-pp.ttwu_queue_wakelist
      0.24 ±  3%      -0.1        0.10 ±  4%  perf-profile.children.cycles-pp.__switch_to
      0.31 ±  3%      -0.1        0.19 ±  4%  perf-profile.children.cycles-pp.asm_sysvec_call_function_single
      0.63            -0.1        0.52 ±  2%  perf-profile.children.cycles-pp.sched_ttwu_pending
      0.20 ±  2%      -0.1        0.08 ±  5%  perf-profile.children.cycles-pp.switch_fpu_return
      0.31 ±  3%      -0.1        0.20 ±  3%  perf-profile.children.cycles-pp.finish_task_switch
      0.26 ±  2%      -0.1        0.15 ±  4%  perf-profile.children.cycles-pp.sysvec_call_function_single
      0.23 ±  2%      -0.1        0.13 ±  3%  perf-profile.children.cycles-pp.__sysvec_call_function_single
      0.66            -0.1        0.57        perf-profile.children.cycles-pp.flush_smp_call_function_queue
      0.19 ±  3%      -0.1        0.12 ±  4%  perf-profile.children.cycles-pp.__flush_smp_call_function_queue
      0.17 ±  4%      -0.1        0.10 ±  6%  perf-profile.children.cycles-pp.llist_add_batch
      0.14 ±  4%      -0.1        0.08 ±  6%  perf-profile.children.cycles-pp.switch_mm_irqs_off
      0.17 ±  4%      -0.1        0.10 ±  4%  perf-profile.children.cycles-pp.__smp_call_single_queue
      0.17 ± 19%      -0.1        0.12 ±  8%  perf-profile.children.cycles-pp.task_tick_fair
      0.16 ±  4%      -0.0        0.11 ±  9%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.16 ±  5%      -0.0        0.12 ±  4%  perf-profile.children.cycles-pp.prepare_task_switch
      0.10 ±  4%      -0.0        0.07 ± 14%  perf-profile.children.cycles-pp.tick_nohz_next_event
      0.07 ±  5%      -0.0        0.06 ±  8%  perf-profile.children.cycles-pp.llist_reverse_order
      0.11 ±  3%      -0.0        0.10 ±  3%  perf-profile.children.cycles-pp.perf_trace_sched_wakeup_template
      0.12            -0.0        0.11 ±  4%  perf-profile.children.cycles-pp.__switch_to_asm
      0.06 ±  9%      +0.0        0.07 ±  6%  perf-profile.children.cycles-pp.rebalance_domains
      0.13 ±  2%      +0.0        0.15 ±  3%  perf-profile.children.cycles-pp.update_rq_clock
      0.45            +0.0        0.47        perf-profile.children.cycles-pp.dequeue_task_fair
      0.03 ± 70%      +0.0        0.06        perf-profile.children.cycles-pp.detach_tasks
      0.38            +0.0        0.40        perf-profile.children.cycles-pp.dequeue_entity
      0.08 ±  4%      +0.0        0.11        perf-profile.children.cycles-pp.update_blocked_averages
      0.31 ±  4%      +0.0        0.35        perf-profile.children.cycles-pp.select_task_rq
      0.10 ±  8%      +0.0        0.14 ±  3%  perf-profile.children.cycles-pp.__softirqentry_text_start
      0.05 ±  7%      +0.1        0.10 ±  6%  perf-profile.children.cycles-pp.select_idle_core
      0.10 ± 10%      +0.1        0.15 ±  5%  perf-profile.children.cycles-pp.ktime_get
      0.00            +0.1        0.05 ±  7%  perf-profile.children.cycles-pp.restore_fpregs_from_fpstate
      0.00            +0.1        0.05 ±  7%  perf-profile.children.cycles-pp.task_h_load
      0.00            +0.1        0.05 ±  7%  perf-profile.children.cycles-pp.__update_blocked_fair
      0.08 ± 12%      +0.1        0.13 ±  6%  perf-profile.children.cycles-pp.clockevents_program_event
      0.12 ±  7%      +0.1        0.18 ±  2%  perf-profile.children.cycles-pp.__irq_exit_rcu
      0.23 ±  3%      +0.1        0.29 ±  3%  perf-profile.children.cycles-pp.exit_to_user_mode_prepare
      0.34 ±  2%      +0.1        0.40 ±  3%  perf-profile.children.cycles-pp.update_load_avg
      0.00            +0.1        0.06 ±  9%  perf-profile.children.cycles-pp.is_vmalloc_addr
      0.10 ±  7%      +0.1        0.16 ±  4%  perf-profile.children.cycles-pp.select_idle_cpu
      0.26 ±  3%      +0.1        0.32 ±  2%  perf-profile.children.cycles-pp.select_task_rq_fair
      0.00            +0.1        0.06        perf-profile.children.cycles-pp.set_task_cpu
      0.00            +0.1        0.06 ±  6%  perf-profile.children.cycles-pp.security_msg_msg_alloc
      0.12 ±  6%      +0.1        0.19        perf-profile.children.cycles-pp.select_idle_sibling
      0.00            +0.1        0.07 ± 11%  perf-profile.children.cycles-pp.security_msg_queue_msgrcv
      0.07 ±  8%      +0.1        0.14 ±  5%  perf-profile.children.cycles-pp.msgctl_stat
      0.00            +0.1        0.07        perf-profile.children.cycles-pp.__x64_sys_msgrcv
      0.00            +0.1        0.07 ±  5%  perf-profile.children.cycles-pp.memcpy_erms
      0.00            +0.1        0.07 ±  5%  perf-profile.children.cycles-pp.kmalloc_slab
      0.06 ±  6%      +0.1        0.14 ±  5%  perf-profile.children.cycles-pp.___slab_alloc
      0.00            +0.1        0.08 ±  6%  perf-profile.children.cycles-pp.security_ipc_permission
      0.00            +0.1        0.08        perf-profile.children.cycles-pp.__cond_resched
      0.00            +0.1        0.08 ±  5%  perf-profile.children.cycles-pp.down_write
      0.00            +0.1        0.09 ±  4%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode_prepare
      0.00            +0.1        0.09 ±  4%  perf-profile.children.cycles-pp.rwsem_mark_wake
      0.01 ±223%      +0.1        0.11 ±  6%  perf-profile.children.cycles-pp.find_busiest_queue
      0.00            +0.1        0.10 ±  6%  perf-profile.children.cycles-pp.check_stack_object
      0.00            +0.1        0.11 ±  7%  perf-profile.children.cycles-pp.idr_get_next_ul
      0.00            +0.1        0.11 ±  6%  perf-profile.children.cycles-pp.idr_get_next
      0.04 ± 44%      +0.1        0.16 ±  3%  perf-profile.children.cycles-pp.security_msg_msg_free
      0.00            +0.1        0.12 ±  4%  perf-profile.children.cycles-pp.up_read
      0.60 ±  9%      +0.1        0.73 ±  2%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
      0.00            +0.1        0.13 ±  3%  perf-profile.children.cycles-pp.number
      0.07 ±  5%      +0.1        0.22 ±  6%  perf-profile.children.cycles-pp._find_next_bit
      0.00            +0.1        0.15 ±  2%  perf-profile.children.cycles-pp.refill_obj_stock
      0.03 ± 70%      +0.2        0.18 ±  6%  perf-profile.children.cycles-pp.obj_cgroup_charge
      0.05            +0.2        0.20 ±  3%  perf-profile.children.cycles-pp.syscall_enter_from_user_mode
      0.04 ± 44%      +0.2        0.20 ±  2%  perf-profile.children.cycles-pp.sysvipc_proc_next
      0.00            +0.2        0.16 ±  2%  perf-profile.children.cycles-pp.down_read
      0.00            +0.2        0.16 ±  3%  perf-profile.children.cycles-pp.format_decode
      0.08 ±  8%      +0.2        0.30 ±  5%  perf-profile.children.cycles-pp.idle_cpu
      0.00            +0.2        0.22 ±  2%  perf-profile.children.cycles-pp.rwsem_spin_on_owner
      0.09 ±  8%      +0.3        0.35        perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      0.06 ± 21%      +0.3        0.32 ±  7%  perf-profile.children.cycles-pp.__get_obj_cgroup_from_memcg
      0.14 ±  5%      +0.3        0.41 ±  2%  perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
      0.09 ±  6%      +0.3        0.38 ±  3%  perf-profile.children.cycles-pp.__get_user_8
      0.09            +0.3        0.38 ±  3%  perf-profile.children.cycles-pp.__put_user_nocheck_8
      0.11 ±  6%      +0.3        0.40 ±  5%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.14 ±  3%      +0.3        0.44        perf-profile.children.cycles-pp.ipcperms
      0.09 ±  9%      +0.3        0.39 ±  2%  perf-profile.children.cycles-pp.__list_add_valid
      0.10 ±  4%      +0.3        0.41 ±  2%  perf-profile.children.cycles-pp.__might_sleep
      0.27            +0.3        0.60        perf-profile.children.cycles-pp.syscall_exit_to_user_mode
      0.13 ±  2%      +0.4        0.49 ±  3%  perf-profile.children.cycles-pp.__virt_addr_valid
      0.10 ±  5%      +0.4        0.46 ±  2%  perf-profile.children.cycles-pp.vsnprintf
      0.10 ±  4%      +0.4        0.46        perf-profile.children.cycles-pp.seq_printf
      0.10 ±  7%      +0.4        0.48        perf-profile.children.cycles-pp.sysvipc_msg_proc_show
      0.14 ±  3%      +0.4        0.56 ±  2%  perf-profile.children.cycles-pp.mod_objcg_state
      0.10 ± 16%      +0.4        0.55 ±  4%  perf-profile.children.cycles-pp.get_obj_cgroup_from_current
      0.23 ±  2%      +0.5        0.70        perf-profile.children.cycles-pp.__might_resched
      1.18            +0.6        1.79 ±  2%  perf-profile.children.cycles-pp.try_to_wake_up
      3.34 ±  2%      +0.6        3.95 ±  2%  perf-profile.children.cycles-pp.__schedule
      0.29 ±  4%      +0.6        0.92        perf-profile.children.cycles-pp.stress_msg
      0.18 ±  2%      +0.6        0.82        perf-profile.children.cycles-pp.copy_user_short_string
      0.18 ±  5%      +0.6        0.82        perf-profile.children.cycles-pp.rwsem_wake
      0.21 ±  2%      +0.7        0.92        perf-profile.children.cycles-pp.entry_SYSRETQ_unsafe_stack
      0.09 ± 10%      +0.7        0.82        perf-profile.children.cycles-pp.ss_wakeup
      0.21 ±  3%      +0.7        0.94        perf-profile.children.cycles-pp._copy_to_user
      0.23 ±  2%      +0.7        0.96 ±  2%  perf-profile.children.cycles-pp.__check_heap_object
      0.21 ±  2%      +0.7        0.95        perf-profile.children.cycles-pp.__x64_sys_msgsnd
      0.22 ±  5%      +0.8        0.99 ±  2%  perf-profile.children.cycles-pp.__radix_tree_lookup
      0.29 ±  2%      +0.8        1.10        perf-profile.children.cycles-pp.__might_fault
      1.31            +0.9        2.22 ±  2%  perf-profile.children.cycles-pp.wake_up_q
      2.54            +0.9        3.47 ±  2%  perf-profile.children.cycles-pp.schedule
      0.38 ±  2%      +0.9        1.31        perf-profile.children.cycles-pp._copy_from_user
      0.20 ±  3%      +0.9        1.14        perf-profile.children.cycles-pp.__list_del_entry_valid
      0.00            +0.9        0.94 ±  3%  perf-profile.children.cycles-pp.osq_lock
      0.10 ±  4%      +1.0        1.14 ±  2%  perf-profile.children.cycles-pp.sysvipc_proc_start
      0.80 ±  5%      +1.1        1.86 ±  4%  perf-profile.children.cycles-pp.update_sg_lb_stats
      0.87 ±  4%      +1.1        1.99 ±  4%  perf-profile.children.cycles-pp.update_sd_lb_stats
      0.34 ±  2%      +1.1        1.46        perf-profile.children.cycles-pp.__entry_text_start
      0.88 ±  4%      +1.1        2.01 ±  4%  perf-profile.children.cycles-pp.find_busiest_group
      0.37 ±  2%      +1.2        1.56        perf-profile.children.cycles-pp.check_heap_object
      0.05 ±  7%      +1.2        1.29 ±  3%  perf-profile.children.cycles-pp.rwsem_optimistic_spin
      1.05 ±  5%      +1.3        2.31 ±  4%  perf-profile.children.cycles-pp.load_balance
      1.34 ±  4%      +1.3        2.63 ±  4%  perf-profile.children.cycles-pp.pick_next_task_fair
      1.20 ±  4%      +1.3        2.50 ±  4%  perf-profile.children.cycles-pp.newidle_balance
      1.09 ±  2%      +1.4        2.50        perf-profile.children.cycles-pp.__slab_free
      0.27 ±  4%      +1.6        1.88 ±  2%  perf-profile.children.cycles-pp.seq_read_iter
      0.27 ±  3%      +1.6        1.88 ±  2%  perf-profile.children.cycles-pp.seq_read
      0.28 ±  4%      +1.6        1.90 ±  2%  perf-profile.children.cycles-pp.vfs_read
      0.28 ±  4%      +1.6        1.90 ±  2%  perf-profile.children.cycles-pp.ksys_read
      0.28 ±  4%      +1.6        1.92 ±  2%  perf-profile.children.cycles-pp.read
      0.63            +1.8        2.42        perf-profile.children.cycles-pp.kfree
      0.51 ±  2%      +2.0        2.50        perf-profile.children.cycles-pp.memcg_slab_post_alloc_hook
      0.70            +2.2        2.94        perf-profile.children.cycles-pp.__check_object_size
      0.27 ±  3%      +2.4        2.69        perf-profile.children.cycles-pp.rwsem_down_write_slowpath
      0.07 ±  8%      +2.4        2.51        perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.83            +2.8        3.59        perf-profile.children.cycles-pp.store_msg
      0.31 ±  3%      +2.9        3.20        perf-profile.children.cycles-pp.rwsem_down_read_slowpath
      0.26            +3.0        3.28        perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.53 ±  2%      +3.1        3.65        perf-profile.children.cycles-pp.msgctl_down
      1.78            +3.3        5.07        perf-profile.children.cycles-pp.free_msg
      0.97            +3.4        4.34        perf-profile.children.cycles-pp.__kmalloc
      0.00            +3.4        3.38        perf-profile.children.cycles-pp.percpu_counter_add_batch
      0.99            +3.4        4.41        perf-profile.children.cycles-pp.alloc_msg
      1.10            +3.5        4.61        perf-profile.children.cycles-pp.do_msg_fill
      0.00            +3.6        3.56        perf-profile.children.cycles-pp.__percpu_counter_sum
      1.68            +5.3        7.01        perf-profile.children.cycles-pp.load_msg
      0.43 ±  2%      +5.8        6.20        perf-profile.children.cycles-pp.msgctl_info
      0.60 ±  2%      +7.4        7.97        perf-profile.children.cycles-pp.idr_find
      1.04 ±  2%      +9.0       10.05        perf-profile.children.cycles-pp.ksys_msgctl
      1.10 ±  2%      +9.1       10.23        perf-profile.children.cycles-pp.msgctl
      1.63           +11.1       12.72        perf-profile.children.cycles-pp.ipc_obtain_object_check
     15.18 ±  4%     +13.4       28.63        perf-profile.children.cycles-pp.start_secondary
     15.36 ±  4%     +13.5       28.91        perf-profile.children.cycles-pp.do_idle
     15.37 ±  4%     +13.5       28.92        perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     15.37 ±  4%     +13.5       28.92        perf-profile.children.cycles-pp.cpu_startup_entry
     13.61 ±  4%     +14.1       27.69        perf-profile.children.cycles-pp.cpuidle_idle_call
     13.23 ±  5%     +14.1       27.31        perf-profile.children.cycles-pp.cpuidle_enter
     13.23 ±  5%     +14.1       27.31        perf-profile.children.cycles-pp.cpuidle_enter_state
     12.02 ±  5%     +14.5       26.52        perf-profile.children.cycles-pp.mwait_idle_with_hints
      6.40 ±  8%     +15.4       21.83 ±  2%  perf-profile.children.cycles-pp.intel_idle
     29.09           -26.2        2.86        perf-profile.self.cycles-pp.do_msgsnd
     17.19           -15.2        1.98        perf-profile.self.cycles-pp.do_msgrcv
     18.27            -8.9        9.35        perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      5.53            -1.9        3.62        perf-profile.self.cycles-pp._raw_spin_lock
      1.17            -1.1        0.07 ±  5%  perf-profile.self.cycles-pp.wake_q_add
      0.94 ±  3%      -0.5        0.43 ±  8%  perf-profile.self.cycles-pp.poll_idle
      0.41            -0.2        0.18 ±  2%  perf-profile.self.cycles-pp.__schedule
      0.24 ±  3%      -0.1        0.10 ±  3%  perf-profile.self.cycles-pp.__switch_to
      0.17 ±  4%      -0.1        0.10 ±  6%  perf-profile.self.cycles-pp.llist_add_batch
      0.14 ±  4%      -0.1        0.08 ±  6%  perf-profile.self.cycles-pp.switch_mm_irqs_off
      0.12 ±  4%      -0.1        0.06 ±  7%  perf-profile.self.cycles-pp.__flush_smp_call_function_queue
      0.07 ±  6%      -0.0        0.03 ± 70%  perf-profile.self.cycles-pp.sched_ttwu_pending
      0.09 ±  4%      -0.0        0.05 ±  8%  perf-profile.self.cycles-pp.ttwu_queue_wakelist
      0.11 ±  6%      -0.0        0.08 ±  6%  perf-profile.self.cycles-pp.finish_task_switch
      0.14 ±  4%      -0.0        0.12 ±  3%  perf-profile.self.cycles-pp.msgctl_info
      0.11 ±  6%      -0.0        0.09 ±  5%  perf-profile.self.cycles-pp.prepare_task_switch
      0.12            -0.0        0.11 ±  4%  perf-profile.self.cycles-pp.__switch_to_asm
      0.08 ±  4%      -0.0        0.07 ±  5%  perf-profile.self.cycles-pp.enqueue_entity
      0.09 ±  7%      +0.0        0.11 ±  6%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.17 ±  4%      +0.0        0.20 ±  4%  perf-profile.self.cycles-pp.update_load_avg
      0.08            +0.0        0.11 ±  5%  perf-profile.self.cycles-pp.update_rq_clock
      0.04 ± 44%      +0.0        0.09 ±  4%  perf-profile.self.cycles-pp.update_sd_lb_stats
      0.07 ± 15%      +0.1        0.12 ±  6%  perf-profile.self.cycles-pp.ktime_get
      0.00            +0.1        0.05 ±  7%  perf-profile.self.cycles-pp.restore_fpregs_from_fpstate
      0.00            +0.1        0.05 ±  7%  perf-profile.self.cycles-pp.security_msg_msg_alloc
      0.00            +0.1        0.05 ±  7%  perf-profile.self.cycles-pp.task_h_load
      0.00            +0.1        0.06 ±  9%  perf-profile.self.cycles-pp.load_balance
      0.00            +0.1        0.06 ±  8%  perf-profile.self.cycles-pp.rwsem_mark_wake
      0.11 ±  6%      +0.1        0.17 ±  3%  perf-profile.self.cycles-pp.menu_select
      0.00            +0.1        0.06        perf-profile.self.cycles-pp.__x64_sys_msgrcv
      0.00            +0.1        0.06 ±  7%  perf-profile.self.cycles-pp.security_msg_queue_msgrcv
      0.00            +0.1        0.06 ±  7%  perf-profile.self.cycles-pp.kmalloc_slab
      0.00            +0.1        0.07 ±  8%  perf-profile.self.cycles-pp.memcpy_erms
      0.00            +0.1        0.07        perf-profile.self.cycles-pp.free_msg
      0.00            +0.1        0.07 ±  5%  perf-profile.self.cycles-pp.security_ipc_permission
      0.00            +0.1        0.07 ±  6%  perf-profile.self.cycles-pp.syscall_exit_to_user_mode_prepare
      0.00            +0.1        0.08        perf-profile.self.cycles-pp.security_msg_msg_free
      0.00            +0.1        0.08        perf-profile.self.cycles-pp.down_write
      0.00            +0.1        0.08 ±  7%  perf-profile.self.cycles-pp.alloc_msg
      0.04 ± 44%      +0.1        0.13 ±  6%  perf-profile.self.cycles-pp.___slab_alloc
      0.00            +0.1        0.09 ±  7%  perf-profile.self.cycles-pp.check_stack_object
      0.00            +0.1        0.10 ±  7%  perf-profile.self.cycles-pp.idr_get_next_ul
      0.00            +0.1        0.10 ±  4%  perf-profile.self.cycles-pp.vsnprintf
      0.00            +0.1        0.10 ±  4%  perf-profile.self.cycles-pp.find_busiest_queue
      0.00            +0.1        0.10 ±  4%  perf-profile.self.cycles-pp.rwsem_optimistic_spin
      0.24 ±  2%      +0.1        0.36        perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.00            +0.1        0.12 ±  3%  perf-profile.self.cycles-pp._copy_to_user
      0.00            +0.1        0.12 ±  4%  perf-profile.self.cycles-pp.number
      0.00            +0.1        0.12 ±  3%  perf-profile.self.cycles-pp.format_decode
      0.00            +0.1        0.12 ±  3%  perf-profile.self.cycles-pp.up_read
      0.06 ±  7%      +0.1        0.20 ±  5%  perf-profile.self.cycles-pp._find_next_bit
      0.00            +0.1        0.14 ±  2%  perf-profile.self.cycles-pp.rwsem_down_write_slowpath
      0.00            +0.1        0.14 ±  6%  perf-profile.self.cycles-pp.obj_cgroup_charge
      0.00            +0.1        0.15 ±  3%  perf-profile.self.cycles-pp.exit_to_user_mode_prepare
      0.00            +0.1        0.15 ±  3%  perf-profile.self.cycles-pp.refill_obj_stock
      0.00            +0.2        0.16 ±  4%  perf-profile.self.cycles-pp.syscall_enter_from_user_mode
      0.00            +0.2        0.16 ±  2%  perf-profile.self.cycles-pp.down_read
      0.03 ± 70%      +0.2        0.21 ±  3%  perf-profile.self.cycles-pp.load_msg
      0.04 ± 71%      +0.2        0.22 ±  3%  perf-profile.self.cycles-pp.__might_fault
      0.01 ±223%      +0.2        0.21 ±  2%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.00            +0.2        0.21 ±  8%  perf-profile.self.cycles-pp.get_obj_cgroup_from_current
      0.00            +0.2        0.21 ±  2%  perf-profile.self.cycles-pp.syscall_exit_to_user_mode
      0.00            +0.2        0.21 ±  2%  perf-profile.self.cycles-pp.rwsem_spin_on_owner
      0.08 ±  8%      +0.2        0.30 ±  4%  perf-profile.self.cycles-pp.idle_cpu
      0.07            +0.2        0.29 ±  2%  perf-profile.self.cycles-pp.__check_object_size
      0.08 ±  5%      +0.2        0.33        perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      0.06 ±  6%      +0.2        0.31 ±  2%  perf-profile.self.cycles-pp.__x64_sys_msgsnd
      0.06 ± 21%      +0.3        0.32 ±  8%  perf-profile.self.cycles-pp.__get_obj_cgroup_from_memcg
      0.14 ±  5%      +0.3        0.40 ±  2%  perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack
      0.09 ±  5%      +0.3        0.36 ±  2%  perf-profile.self.cycles-pp.__might_sleep
      0.11 ±  6%      +0.3        0.38 ±  5%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.08 ±  5%      +0.3        0.37 ±  3%  perf-profile.self.cycles-pp.__get_user_8
      0.14 ±  3%      +0.3        0.42        perf-profile.self.cycles-pp.ipcperms
      0.09            +0.3        0.38 ±  3%  perf-profile.self.cycles-pp.__put_user_nocheck_8
      0.13 ±  3%      +0.3        0.42        perf-profile.self.cycles-pp.wake_up_q
      0.08 ±  5%      +0.3        0.38 ±  3%  perf-profile.self.cycles-pp.__list_add_valid
      0.00            +0.3        0.30 ±  3%  perf-profile.self.cycles-pp.rwsem_down_read_slowpath
      0.12 ±  3%      +0.3        0.46 ±  4%  perf-profile.self.cycles-pp.__virt_addr_valid
      0.06 ±  6%      +0.3        0.41 ±  3%  perf-profile.self.cycles-pp.do_msg_fill
      0.19 ±  3%      +0.4        0.55        perf-profile.self.cycles-pp._copy_from_user
      0.13 ±  3%      +0.4        0.54        perf-profile.self.cycles-pp.mod_objcg_state
      0.14 ±  2%      +0.4        0.58        perf-profile.self.cycles-pp.__libc_msgsnd
      0.23 ±  2%      +0.5        0.69        perf-profile.self.cycles-pp.__might_resched
      0.12 ±  3%      +0.5        0.59        perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.14 ±  5%      +0.5        0.61        perf-profile.self.cycles-pp.__entry_text_start
      0.15 ±  4%      +0.5        0.63 ±  2%  perf-profile.self.cycles-pp.__libc_msgrcv
      0.14 ±  2%      +0.5        0.64 ±  2%  perf-profile.self.cycles-pp.__kmalloc
      0.17 ±  4%      +0.5        0.68        perf-profile.self.cycles-pp.do_syscall_64
      0.20 ±  2%      +0.5        0.71 ±  2%  perf-profile.self.cycles-pp.store_msg
      0.28 ±  4%      +0.6        0.87        perf-profile.self.cycles-pp.stress_msg
      0.17 ±  3%      +0.6        0.79        perf-profile.self.cycles-pp.copy_user_short_string
      0.20            +0.7        0.89        perf-profile.self.cycles-pp.entry_SYSRETQ_unsafe_stack
      0.08 ±  8%      +0.7        0.80        perf-profile.self.cycles-pp.ss_wakeup
      0.13 ±  5%      +0.7        0.85 ±  6%  perf-profile.self.cycles-pp.try_to_wake_up
      0.23 ±  2%      +0.7        0.96 ±  2%  perf-profile.self.cycles-pp.__check_heap_object
      0.00            +0.7        0.74        perf-profile.self.cycles-pp.__percpu_counter_sum
      0.21 ±  6%      +0.8        0.96        perf-profile.self.cycles-pp.__radix_tree_lookup
      0.67 ±  4%      +0.8        1.44 ±  4%  perf-profile.self.cycles-pp.update_sg_lb_stats
      0.22 ±  4%      +0.8        1.04 ±  2%  perf-profile.self.cycles-pp.check_heap_object
      0.00            +0.9        0.92 ±  3%  perf-profile.self.cycles-pp.osq_lock
      0.20 ±  4%      +0.9        1.12        perf-profile.self.cycles-pp.__list_del_entry_valid
      0.49 ±  2%      +1.4        1.85        perf-profile.self.cycles-pp.kfree
      1.08 ±  2%      +1.4        2.49        perf-profile.self.cycles-pp.__slab_free
      0.46 ±  3%      +1.8        2.28        perf-profile.self.cycles-pp.memcg_slab_post_alloc_hook
      0.81            +2.9        3.76        perf-profile.self.cycles-pp.ipc_obtain_object_check
      0.00            +3.3        3.34        perf-profile.self.cycles-pp.percpu_counter_add_batch
      0.59            +7.3        7.92        perf-profile.self.cycles-pp.idr_find
     11.93 ±  5%     +14.5       26.39        perf-profile.self.cycles-pp.mwait_idle_with_hints


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        sudo bin/lkp install job.yaml           # job file is attached in this email
        bin/lkp split-job --compatible job.yaml # generate the yaml file for lkp run
        sudo bin/lkp run generated-yaml-file

        # if come across any failure that blocks the test,
        # please remove ~/.lkp and /lkp dir to run from a clean state.


Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


-- 
0-DAY CI Kernel Test Service
https://01.org/lkp

View attachment "config-6.0.0-rc3-00057-g72d1e611082e" of type "text/plain" (163956 bytes)

View attachment "job-script" of type "text/plain" (8354 bytes)

View attachment "job.yaml" of type "text/plain" (5592 bytes)

View attachment "reproduce" of type "text/plain" (380 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ