lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Date:   Mon, 24 Jun 2019 13:45:11 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     Waiman Long <longman@...hat.com>
Cc:     Ingo Molnar <mingo@...nel.org>,
        Peter Zijlstra <peterz@...radead.org>,
        Borislav Petkov <bp@...en8.de>,
        Davidlohr Bueso <dave@...olabs.net>,
        "H. Peter Anvin" <hpa@...or.com>,
        Linus Torvalds <torvalds@...ux-foundation.org>,
        Thomas Gleixner <tglx@...utronix.de>,
        Tim Chen <tim.c.chen@...ux.intel.com>,
        Will Deacon <will.deacon@....com>,
        huang ying <huang.ying.caritas@...il.com>,
        LKML <linux-kernel@...r.kernel.org>, tipbuild@...or.com,
        lkp@...org
Subject: [locking/rwsem] 4f23dbc1e6: reaim.jobs_per_min -37.0% regression

Greeting,

FYI, we noticed a -37.0% regression of reaim.jobs_per_min due to commit:


commit: 4f23dbc1e657951e5d94c60369bc1db065961fb3 ("locking/rwsem: Implement lock handoff to prevent lock starvation")
https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git locking/core

in testcase: reaim
on test machine: 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 64G memory
with following parameters:

	runtime: 300s
	nr_task: 7000t
	test: shared_memory
	cpufreq_governor: performance
	ucode: 0x42e

test-description: REAIM is an updated and improved version of AIM 7 benchmark.
test-url: https://sourceforge.net/projects/re-aim-7/


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
compiler/cpufreq_governor/kconfig/nr_task/rootfs/runtime/tbox_group/test/testcase/ucode:
  gcc-7/performance/x86_64-rhel-7.6/7000t/debian-x86_64-2019-05-14.cgz/300s/lkp-ivb-2ep1/shared_memory/reaim/0x42e

commit: 
  3f6d517a3e ("locking/rwsem: Make rwsem_spin_on_owner() return owner state")
  4f23dbc1e6 ("locking/rwsem: Implement lock handoff to prevent lock starvation")

3f6d517a3ece6e6c 4f23dbc1e657951e5d94c60369b 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :4           25%           1:4     dmesg.WARNING:at#for_ip_interrupt_entry/0x
         %stddev     %change         %stddev
             \          |                \  
    226.02            +7.5%     243.00        reaim.child_systime
    117.37            -8.6%     107.28        reaim.child_utime
   5420413           -37.0%    3413821        reaim.jobs_per_min
    774.34           -37.0%     487.69        reaim.jobs_per_min_child
     50.22           +67.2%      83.96        reaim.jti
   5502800           -35.5%    3547918        reaim.max_jobs_per_min
      7.75           +58.8%      12.31        reaim.parent_time
     49.33           -68.5%      15.53 ±  2%  reaim.std_dev_percent
      2.30           -30.1%       1.61 ±  2%  reaim.std_dev_time
    306.44            +2.4%     313.76        reaim.time.elapsed_time
    306.44            +2.4%     313.76        reaim.time.elapsed_time.max
   1134292 ±  2%     -78.5%     243783 ±  2%  reaim.time.involuntary_context_switches
  32442242           -28.7%   23120658        reaim.time.minor_page_faults
      3343           -29.1%       2369        reaim.time.percent_of_cpu_this_job_got
      6782           -23.6%       5181        reaim.time.system_time
      3464           -34.9%       2254        reaim.time.user_time
   2427742 ±  6%    +602.1%   17045855        reaim.time.voluntary_context_switches
  20650000           -28.8%   14700000        reaim.workload
  16171248 ± 21%     -35.0%   10507950 ± 21%  numa-numastat.node0.local_node
  16182139 ± 21%     -35.0%   10516565 ± 21%  numa-numastat.node0.numa_hit
     24931 ± 37%     +48.0%      36905 ± 30%  numa-meminfo.node0.Shmem
      3119 ± 26%    +442.1%      16909 ± 50%  numa-meminfo.node1.Inactive
      3119 ± 26%    +442.1%      16909 ± 50%  numa-meminfo.node1.Inactive(anon)
     29.75           +63.9%      48.75        vmstat.cpu.id
     23.00           -34.8%      15.00        vmstat.cpu.us
     20649 ±  3%    +450.0%     113568        vmstat.system.cs
     29.82           +19.2       48.97        mpstat.cpu.all.idle%
      0.00 ±141%      +0.0        0.01 ± 59%  mpstat.cpu.all.soft%
     46.40           -11.0       35.41        mpstat.cpu.all.sys%
     23.78            -8.2       15.61        mpstat.cpu.all.usr%
      6233 ± 37%     +48.1%       9228 ± 30%  numa-vmstat.node0.nr_shmem
   8507002 ± 21%     -33.6%    5650272 ± 20%  numa-vmstat.node0.numa_hit
   8496124 ± 21%     -33.6%    5641461 ± 20%  numa-vmstat.node0.numa_local
    780.00 ± 26%    +442.4%       4230 ± 50%  numa-vmstat.node1.nr_inactive_anon
    780.00 ± 26%    +442.4%       4230 ± 50%  numa-vmstat.node1.nr_zone_inactive_anon
 1.723e+08 ± 56%    +253.7%  6.093e+08 ± 19%  cpuidle.C1.time
   2490308 ± 61%    +221.5%    8005497 ± 11%  cpuidle.C1.usage
 1.356e+08 ± 46%    +618.9%  9.746e+08 ± 12%  cpuidle.C1E.time
   1194640 ± 57%    +456.9%    6652522 ±  6%  cpuidle.C1E.usage
 5.544e+08 ±102%    +304.7%  2.243e+09 ± 35%  cpuidle.C3.time
   2306269 ± 89%    +306.1%    9364769 ± 29%  cpuidle.C3.usage
   4508751 ±  3%     +61.0%    7256974 ± 18%  cpuidle.C6.usage
    164390 ± 51%    +114.5%     352624        cpuidle.POLL.time
    572585           +30.4%     746803        meminfo.Active
    568609           +30.6%     742827        meminfo.Active(anon)
    546925           +30.1%     711800        meminfo.AnonPages
   4383767           +33.0%    5832436        meminfo.Committed_AS
     20744 ±  9%     +74.4%      36183        meminfo.Inactive
     19912 ± 10%     +77.7%      35375        meminfo.Inactive(anon)
     76491           +28.9%      98623        meminfo.KernelStack
     26140 ±  4%     +25.9%      32910        meminfo.Mapped
   2599230           +12.6%    2926477        meminfo.Memused
    169805           +33.8%     227281        meminfo.PageTables
     38612 ±  8%     +67.1%      64538        meminfo.Shmem
      2118           -25.4%       1579        turbostat.Avg_MHz
     71.13           -18.2       52.90        turbostat.Busy%
   2487326 ± 61%    +221.7%    8001963 ± 11%  turbostat.C1
      1.16 ± 55%      +2.9        4.03 ± 19%  turbostat.C1%
   1194469 ± 57%    +456.9%    6652394 ±  6%  turbostat.C1E
      0.91 ± 46%      +5.5        6.45 ± 12%  turbostat.C1E%
   2306085 ± 89%    +306.1%    9364606 ± 29%  turbostat.C3
      3.79 ±102%     +11.1       14.85 ± 35%  turbostat.C3%
   4506779 ±  3%     +60.9%    7253478 ± 18%  turbostat.C6
     15.89 ± 14%    +143.5%      38.68 ±  4%  turbostat.CPU%c1
      0.36 ±114%    +319.9%       1.53 ± 59%  turbostat.CPU%c3
     12.61 ± 22%     -45.4%       6.89 ± 29%  turbostat.CPU%c6
    141.75            -1.7%     139.37        turbostat.CorWatt
      7.33 ± 29%     -70.3%       2.18 ± 26%  turbostat.Pkg%pc2
      0.80 ± 45%    -100.0%       0.00        turbostat.Pkg%pc6
    170.75            -1.3%     168.54        turbostat.PkgWatt
      8.18 ±  2%     +10.4%       9.03        turbostat.RAMWatt
    142331           +30.4%     185610        proc-vmstat.nr_active_anon
    136930           +29.9%     177875        proc-vmstat.nr_anon_pages
    266116            +1.9%     271174        proc-vmstat.nr_file_pages
      4976 ± 10%     +77.6%       8837        proc-vmstat.nr_inactive_anon
     76529           +28.6%      98440        proc-vmstat.nr_kernel_stack
      6540 ±  4%     +25.9%       8231        proc-vmstat.nr_mapped
     42403           +33.8%      56719        proc-vmstat.nr_page_table_pages
      9648 ±  8%     +67.1%      16125        proc-vmstat.nr_shmem
     18684            +2.2%      19096        proc-vmstat.nr_slab_reclaimable
     77388            +5.7%      81780        proc-vmstat.nr_slab_unreclaimable
    142331           +30.4%     185610        proc-vmstat.nr_zone_active_anon
      4976 ± 10%     +77.6%       8837        proc-vmstat.nr_zone_inactive_anon
  32372140           -28.0%   23317838        proc-vmstat.numa_hit
  32356342           -28.0%   23302023        proc-vmstat.numa_local
      4069 ± 13%     +29.5%       5269 ±  5%  proc-vmstat.pgactivate
  33565451           -27.4%   24379811        proc-vmstat.pgalloc_normal
  33200833           -28.1%   23880025        proc-vmstat.pgfault
  33516003           -27.4%   24322043        proc-vmstat.pgfree
      1441            -9.7%       1302        slabinfo.Acpi-Parse.active_slabs
    105247            -9.7%      95071        slabinfo.Acpi-Parse.num_objs
      1441            -9.7%       1302        slabinfo.Acpi-Parse.num_slabs
    100839           +26.8%     127886        slabinfo.anon_vma.active_objs
      2465           +17.3%       2891        slabinfo.anon_vma.active_slabs
    113424           +17.3%     133011        slabinfo.anon_vma.num_objs
      2465           +17.3%       2891        slabinfo.anon_vma.num_slabs
    188835           +28.9%     243366        slabinfo.anon_vma_chain.active_objs
      3330           +18.9%       3958        slabinfo.anon_vma_chain.active_slabs
    213156           +18.9%     253390        slabinfo.anon_vma_chain.num_objs
      3330           +18.9%       3958        slabinfo.anon_vma_chain.num_slabs
     28182 ±  2%     -15.8%      23739        slabinfo.filp.active_objs
     29551 ±  2%     -16.0%      24818        slabinfo.filp.num_objs
     24561            -9.4%      22243        slabinfo.kmalloc-256.num_objs
     99611 ±  3%    +102.7%     201872 ±  6%  slabinfo.kmalloc-8.active_objs
    194.00 ±  3%    +103.4%     394.50 ±  6%  slabinfo.kmalloc-8.active_slabs
     99611 ±  3%    +103.0%     202245 ±  6%  slabinfo.kmalloc-8.num_objs
    194.00 ±  3%    +103.4%     394.50 ±  6%  slabinfo.kmalloc-8.num_slabs
      9649           +13.1%      10914        slabinfo.mm_struct.active_objs
     28364 ±  2%     +14.5%      32464 ±  2%  slabinfo.proc_inode_cache.num_objs
     20118 ±  2%     -13.2%      17457        slabinfo.shmem_inode_cache.active_objs
     20536 ±  2%     -13.6%      17742        slabinfo.shmem_inode_cache.num_objs
      7380           +18.1%       8720        slabinfo.sighand_cache.active_objs
     10471           +11.4%      11661        slabinfo.signal_cache.active_objs
      5032           +28.0%       6441        slabinfo.task_struct.active_objs
      2016           +13.9%       2297        slabinfo.task_struct.active_slabs
      6049           +13.9%       6892        slabinfo.task_struct.num_objs
      2016           +13.9%       2297        slabinfo.task_struct.num_slabs
    156078           +28.8%     201097        slabinfo.vm_area_struct.active_objs
      4350           +19.7%       5206        slabinfo.vm_area_struct.active_slabs
    174056           +19.7%     208265        slabinfo.vm_area_struct.num_objs
      4350           +19.7%       5206        slabinfo.vm_area_struct.num_slabs
      4594 ± 32%     -59.9%       1840 ±109%  interrupts.CPU0.NMI:Non-maskable_interrupts
      4594 ± 32%     -59.9%       1840 ±109%  interrupts.CPU0.PMI:Performance_monitoring_interrupts
      8707 ±  3%     +18.4%      10308 ±  9%  interrupts.CPU1.RES:Rescheduling_interrupts
      3822 ± 34%     -51.4%       1857 ± 57%  interrupts.CPU10.NMI:Non-maskable_interrupts
      3822 ± 34%     -51.4%       1857 ± 57%  interrupts.CPU10.PMI:Performance_monitoring_interrupts
      8226 ±  7%     +28.2%      10548 ±  6%  interrupts.CPU10.RES:Rescheduling_interrupts
      8711 ±  3%     +14.7%       9992 ±  3%  interrupts.CPU11.RES:Rescheduling_interrupts
      4571 ± 32%     -41.7%       2664 ± 70%  interrupts.CPU12.NMI:Non-maskable_interrupts
      4571 ± 32%     -41.7%       2664 ± 70%  interrupts.CPU12.PMI:Performance_monitoring_interrupts
      9045 ±  4%     +15.7%      10467 ±  8%  interrupts.CPU13.RES:Rescheduling_interrupts
      8495 ±  6%     +17.3%       9964 ±  8%  interrupts.CPU14.RES:Rescheduling_interrupts
      8527 ±  5%     +17.9%      10052 ±  6%  interrupts.CPU17.RES:Rescheduling_interrupts
      8833 ±  5%     +15.4%      10195        interrupts.CPU18.RES:Rescheduling_interrupts
      4575 ± 32%    -100.0%       0.50 ±100%  interrupts.CPU19.NMI:Non-maskable_interrupts
      4575 ± 32%    -100.0%       0.50 ±100%  interrupts.CPU19.PMI:Performance_monitoring_interrupts
      8669 ±  3%     +20.6%      10457 ±  6%  interrupts.CPU2.RES:Rescheduling_interrupts
      3019 ±  2%     +17.0%       3534 ± 16%  interrupts.CPU20.CAL:Function_call_interrupts
      8862 ±  7%     +19.5%      10591 ±  5%  interrupts.CPU20.RES:Rescheduling_interrupts
      8641 ±  6%     +15.7%      10000 ±  4%  interrupts.CPU21.RES:Rescheduling_interrupts
      4571 ± 33%     -67.1%       1502 ±172%  interrupts.CPU22.NMI:Non-maskable_interrupts
      4571 ± 33%     -67.1%       1502 ±172%  interrupts.CPU22.PMI:Performance_monitoring_interrupts
      8543 ±  3%     +18.0%      10077 ±  9%  interrupts.CPU24.RES:Rescheduling_interrupts
      8520 ±  5%     +18.2%      10067 ±  7%  interrupts.CPU25.RES:Rescheduling_interrupts
      8904 ±  5%     +19.8%      10666 ± 11%  interrupts.CPU26.RES:Rescheduling_interrupts
      8634 ±  5%     +17.7%      10164 ±  5%  interrupts.CPU27.RES:Rescheduling_interrupts
      8670 ±  3%     +13.5%       9844 ±  3%  interrupts.CPU28.RES:Rescheduling_interrupts
      2540 ± 90%     -75.4%     624.75 ±172%  interrupts.CPU29.NMI:Non-maskable_interrupts
      2540 ± 90%     -75.4%     624.75 ±172%  interrupts.CPU29.PMI:Performance_monitoring_interrupts
      8739 ±  3%     +12.9%       9864 ±  4%  interrupts.CPU29.RES:Rescheduling_interrupts
      9127 ±  6%     +18.7%      10837 ±  8%  interrupts.CPU3.RES:Rescheduling_interrupts
      8641 ±  5%     +22.7%      10604 ±  4%  interrupts.CPU30.RES:Rescheduling_interrupts
      8629 ±  4%     +20.0%      10359 ±  4%  interrupts.CPU31.RES:Rescheduling_interrupts
      8633 ±  4%     +21.9%      10525 ± 10%  interrupts.CPU32.RES:Rescheduling_interrupts
      8526 ±  5%     +14.3%       9745 ±  4%  interrupts.CPU33.RES:Rescheduling_interrupts
      8547 ±  4%     +18.9%      10162 ± 10%  interrupts.CPU34.RES:Rescheduling_interrupts
      8445 ±  3%     +27.5%      10769 ±  8%  interrupts.CPU35.RES:Rescheduling_interrupts
      8650 ±  6%     +13.3%       9805 ±  5%  interrupts.CPU38.RES:Rescheduling_interrupts
      3817 ± 34%     -49.1%       1944 ± 59%  interrupts.CPU39.NMI:Non-maskable_interrupts
      3817 ± 34%     -49.1%       1944 ± 59%  interrupts.CPU39.PMI:Performance_monitoring_interrupts
      8814 ±  4%     +18.3%      10430 ±  8%  interrupts.CPU4.RES:Rescheduling_interrupts
      8854 ±  7%     +13.4%      10043 ±  4%  interrupts.CPU41.RES:Rescheduling_interrupts
      8689 ±  4%     +18.7%      10311 ±  5%  interrupts.CPU43.RES:Rescheduling_interrupts
      8716 ±  2%     +15.4%      10055 ±  4%  interrupts.CPU44.RES:Rescheduling_interrupts
      8737 ±  6%     +12.0%       9784 ±  4%  interrupts.CPU45.RES:Rescheduling_interrupts
      8789 ±  4%     +15.6%      10162 ±  4%  interrupts.CPU46.RES:Rescheduling_interrupts
      8445 ±  3%     +14.6%       9682 ±  8%  interrupts.CPU47.RES:Rescheduling_interrupts
      8485 ±  6%     +20.1%      10190 ±  6%  interrupts.CPU5.RES:Rescheduling_interrupts
      3588 ± 16%     -82.6%     623.75 ±173%  interrupts.CPU6.NMI:Non-maskable_interrupts
      3588 ± 16%     -82.6%     623.75 ±173%  interrupts.CPU6.PMI:Performance_monitoring_interrupts
      8839 ±  2%     +14.4%      10113 ± 12%  interrupts.CPU7.RES:Rescheduling_interrupts
      8835 ±  2%     +19.7%      10572 ±  3%  interrupts.CPU8.RES:Rescheduling_interrupts
      4540 ± 33%     -73.7%       1193 ±172%  interrupts.CPU9.NMI:Non-maskable_interrupts
      4540 ± 33%     -73.7%       1193 ±172%  interrupts.CPU9.PMI:Performance_monitoring_interrupts
    128174 ± 13%     -25.6%      95338 ± 13%  interrupts.NMI:Non-maskable_interrupts
    128174 ± 13%     -25.6%      95338 ± 13%  interrupts.PMI:Performance_monitoring_interrupts
    419690 ±  3%     +16.0%     486755 ±  2%  interrupts.RES:Rescheduling_interrupts
     99759 ±  9%     -26.3%      73562 ±  9%  sched_debug.cfs_rq:/.exec_clock.avg
     98221 ±  9%     -27.6%      71078 ±  9%  sched_debug.cfs_rq:/.exec_clock.min
      1508 ± 18%    +100.1%       3018 ± 10%  sched_debug.cfs_rq:/.exec_clock.stddev
     29605 ± 16%     -38.3%      18276 ± 34%  sched_debug.cfs_rq:/.load.avg
      0.64 ± 21%     -73.2%       0.17 ± 58%  sched_debug.cfs_rq:/.nr_running.avg
      0.24 ± 13%     +32.5%       0.32 ± 13%  sched_debug.cfs_rq:/.nr_running.stddev
      3.63 ± 48%    +273.8%      13.59 ± 17%  sched_debug.cfs_rq:/.nr_spread_over.avg
     19.77 ± 44%    +476.6%     113.97 ± 30%  sched_debug.cfs_rq:/.nr_spread_over.max
      3.95 ± 38%    +524.5%      24.68 ± 22%  sched_debug.cfs_rq:/.nr_spread_over.stddev
     92.35 ± 38%     -48.9%      47.17 ± 64%  sched_debug.cfs_rq:/.removed.util_avg.max
     28931 ± 17%     -39.2%      17578 ± 36%  sched_debug.cfs_rq:/.runnable_weight.avg
    427.66 ± 17%     -34.7%     279.47 ± 26%  sched_debug.cfs_rq:/.util_avg.avg
    492.76 ± 14%    +101.9%     994.74        sched_debug.cfs_rq:/.util_est_enqueued.max
     91.64 ± 14%     +66.8%     152.83 ±  2%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
     87079 ± 51%    +312.9%     359582 ± 32%  sched_debug.cpu.avg_idle.min
    269134           -49.8%     135233 ± 10%  sched_debug.cpu.avg_idle.stddev
     24.86 ± 16%     -72.4%       6.87 ± 42%  sched_debug.cpu.clock.stddev
     24.86 ± 16%     -72.4%       6.87 ± 42%  sched_debug.cpu.clock_task.stddev
      1.55 ± 67%     -97.3%       0.04 ±173%  sched_debug.cpu.cpu_load[1].min
     16.87 ±  5%     -29.3%      11.92 ± 28%  sched_debug.cpu.cpu_load[2].avg
    148.69 ± 43%    +118.6%     325.07 ± 18%  sched_debug.cpu.cpu_load[2].max
      2.06 ± 69%     -87.9%       0.25 ±173%  sched_debug.cpu.cpu_load[2].min
     22.48 ± 40%    +118.8%      49.19 ± 18%  sched_debug.cpu.cpu_load[2].stddev
     16.77 ±  7%     -31.3%      11.52 ± 25%  sched_debug.cpu.cpu_load[3].avg
    119.88 ± 38%    +180.2%     335.89 ± 14%  sched_debug.cpu.cpu_load[3].max
      3.06 ± 46%     -86.4%       0.42 ±173%  sched_debug.cpu.cpu_load[3].min
     18.37 ± 35%    +168.0%      49.24 ± 14%  sched_debug.cpu.cpu_load[3].stddev
     16.97 ±  8%     -32.5%      11.46 ± 26%  sched_debug.cpu.cpu_load[4].avg
    126.53 ± 30%    +180.8%     355.26 ±  8%  sched_debug.cpu.cpu_load[4].max
      3.76 ± 41%     -88.9%       0.42 ±173%  sched_debug.cpu.cpu_load[4].min
     19.14 ± 28%    +166.9%      51.09 ±  8%  sched_debug.cpu.cpu_load[4].stddev
     41079 ± 25%     -85.5%       5957 ± 87%  sched_debug.cpu.curr->pid.avg
      0.00 ± 10%     -38.7%       0.00 ± 19%  sched_debug.cpu.next_balance.stddev
      2443 ± 23%     +91.9%       4688 ± 10%  sched_debug.cpu.nr_load_updates.stddev
      0.70 ± 19%     -74.4%       0.18 ± 52%  sched_debug.cpu.nr_running.avg
      1.78 ±  7%     -27.1%       1.30 ±  4%  sched_debug.cpu.nr_running.max
     62457 ±  9%    +440.6%     337627 ±  8%  sched_debug.cpu.nr_switches.avg
     71810 ± 10%    +406.1%     363399 ±  9%  sched_debug.cpu.nr_switches.max
     54873 ±  8%    +395.8%     272038 ± 12%  sched_debug.cpu.nr_switches.min
      3498 ± 11%    +420.3%      18202 ± 17%  sched_debug.cpu.nr_switches.stddev
     58.42 ± 26%     -95.1%       2.84 ±173%  sched_debug.cpu.nr_uninterruptible.avg
    392.04 ±  6%     -19.5%     315.41 ± 21%  sched_debug.cpu.nr_uninterruptible.max
     63598 ±  9%    +440.7%     343895 ±  8%  sched_debug.cpu.sched_count.avg
    108864 ± 14%    +265.4%     397816 ±  7%  sched_debug.cpu.sched_count.max
     55344 ±  9%    +399.4%     276364 ± 13%  sched_debug.cpu.sched_count.min
      8197 ± 12%    +142.2%      19856 ± 20%  sched_debug.cpu.sched_count.stddev
     16740 ±  9%    +875.7%     163328 ±  8%  sched_debug.cpu.sched_goidle.avg
     20511 ± 11%    +755.1%     175390 ±  9%  sched_debug.cpu.sched_goidle.max
     14666 ± 10%    +793.0%     130972 ± 12%  sched_debug.cpu.sched_goidle.min
      1107 ± 27%    +696.5%       8818 ± 17%  sched_debug.cpu.sched_goidle.stddev
     32998 ±  9%    +447.6%     180712 ±  8%  sched_debug.cpu.ttwu_count.avg
     67242 ± 18%    +200.8%     202251 ±  9%  sched_debug.cpu.ttwu_count.max
     29281 ±  9%    +416.0%     151103 ± 12%  sched_debug.cpu.ttwu_count.min
     11270 ±  9%     -41.3%       6611 ±  9%  sched_debug.cpu.ttwu_local.avg
     14202 ± 11%     -42.9%       8109 ± 11%  sched_debug.cpu.ttwu_local.max
     10204 ±  9%     -44.4%       5673 ± 13%  sched_debug.cpu.ttwu_local.min
      8.67           -20.3%       6.91 ±  5%  perf-stat.i.MPKI
  7.31e+09            -9.8%  6.594e+09        perf-stat.i.branch-instructions
      2.22 ±  2%      -0.5        1.74 ±  3%  perf-stat.i.branch-miss-rate%
 1.327e+08           -23.3%  1.018e+08        perf-stat.i.branch-misses
     13.83 ±  2%      +1.6       15.39        perf-stat.i.cache-miss-rate%
  18766269           +33.4%   25027081        perf-stat.i.cache-misses
 1.612e+08            +8.3%  1.746e+08        perf-stat.i.cache-references
     20802 ±  3%    +450.5%     114509        perf-stat.i.context-switches
      2.84           -19.8%       2.28        perf-stat.i.cpi
 1.015e+11           -25.4%  7.568e+10        perf-stat.i.cpu-cycles
      2898          +491.8%      17156        perf-stat.i.cpu-migrations
      5534           -43.8%       3108        perf-stat.i.cycles-between-cache-misses
      2.54 ±  5%      -0.2        2.32 ±  4%  perf-stat.i.dTLB-load-miss-rate%
  3.09e+08 ±  5%     -20.1%  2.469e+08 ±  4%  perf-stat.i.dTLB-load-misses
 1.097e+10           -10.8%  9.788e+09        perf-stat.i.dTLB-loads
  57275127           -29.2%   40541922        perf-stat.i.dTLB-store-misses
 7.974e+09           -26.5%   5.86e+09        perf-stat.i.dTLB-stores
 3.924e+10           -11.0%  3.493e+10        perf-stat.i.instructions
      1705 ±  2%      +4.7%       1785 ±  2%  perf-stat.i.instructions-per-iTLB-miss
      0.37           +22.8%       0.45        perf-stat.i.ipc
    108318           -29.7%      76116        perf-stat.i.minor-faults
     41.76            -2.8       38.93        perf-stat.i.node-load-miss-rate%
   8273526           +26.1%   10430774 ±  2%  perf-stat.i.node-load-misses
  11161421           +46.0%   16291820        perf-stat.i.node-loads
     32.61            -0.9       31.67        perf-stat.i.node-store-miss-rate%
   2525210           +23.1%    3109679 ±  2%  perf-stat.i.node-store-misses
   5649991 ±  2%     +19.8%    6771279 ±  2%  perf-stat.i.node-stores
    108319           -29.7%      76123        perf-stat.i.page-faults
      4.11           +21.5%       4.99        perf-stat.overall.MPKI
      1.82            -0.3        1.54        perf-stat.overall.branch-miss-rate%
     11.64            +2.7       14.34        perf-stat.overall.cache-miss-rate%
      2.59           -16.2%       2.17        perf-stat.overall.cpi
      5406           -44.0%       3026        perf-stat.overall.cycles-between-cache-misses
      2.74 ±  5%      -0.3        2.46 ±  4%  perf-stat.overall.dTLB-load-miss-rate%
      0.71            -0.0        0.69        perf-stat.overall.dTLB-store-miss-rate%
      1843            -9.9%       1661        perf-stat.overall.instructions-per-iTLB-miss
      0.39           +19.4%       0.46        perf-stat.overall.ipc
     42.57            -3.5       39.03        perf-stat.overall.node-load-miss-rate%
    581297           +27.8%     743013        perf-stat.overall.path-length
 7.284e+09            -9.8%  6.569e+09        perf-stat.ps.branch-instructions
 1.323e+08           -23.4%  1.013e+08        perf-stat.ps.branch-misses
  18702352           +33.2%   24912163        perf-stat.ps.cache-misses
 1.607e+08            +8.1%  1.738e+08        perf-stat.ps.cache-references
     20730 ±  3%    +449.6%     113947        perf-stat.ps.context-switches
 1.011e+11           -25.5%  7.538e+10        perf-stat.ps.cpu-cycles
      2889          +490.9%      17073        perf-stat.ps.cpu-migrations
 3.079e+08 ±  5%     -20.2%  2.457e+08 ±  4%  perf-stat.ps.dTLB-load-misses
 1.094e+10           -10.8%  9.752e+09        perf-stat.ps.dTLB-loads
  57072470           -29.3%   40339440        perf-stat.ps.dTLB-store-misses
 7.946e+09           -26.6%  5.832e+09        perf-stat.ps.dTLB-stores
  3.91e+10           -11.0%  3.479e+10        perf-stat.ps.instructions
    107948           -29.8%      75801        perf-stat.ps.minor-faults
   8244755           +25.9%   10379054 ±  2%  perf-stat.ps.node-load-misses
  11123083           +45.7%   16210863        perf-stat.ps.node-loads
   2516620           +23.0%    3094634 ±  2%  perf-stat.ps.node-store-misses
   5630180 ±  2%     +19.7%    6738969 ±  2%  perf-stat.ps.node-stores
    107947           -29.8%      75800        perf-stat.ps.page-faults
   1.2e+13            -9.0%  1.092e+13        perf-stat.total.instructions
    152656 ±  2%     -43.7%      85943 ±  2%  softirqs.CPU0.RCU
     21952 ±  4%    +169.0%      59043        softirqs.CPU0.SCHED
    154311 ±  2%     -43.0%      87983 ±  3%  softirqs.CPU1.RCU
     19434 ±  5%    +196.0%      57523        softirqs.CPU1.SCHED
    151161 ±  2%     -42.9%      86368 ±  2%  softirqs.CPU10.RCU
     18935 ±  6%    +205.0%      57751        softirqs.CPU10.SCHED
    154486 ±  2%     -44.5%      85668 ±  4%  softirqs.CPU11.RCU
     19781 ±  2%    +185.0%      56383 ±  4%  softirqs.CPU11.SCHED
    150442 ±  2%     -44.2%      83878 ±  3%  softirqs.CPU12.RCU
     20143 ±  4%    +182.8%      56965 ±  3%  softirqs.CPU12.SCHED
    150690 ±  2%     -47.0%      79823 ±  6%  softirqs.CPU13.RCU
     19317 ±  4%    +184.0%      54867 ±  5%  softirqs.CPU13.SCHED
    149872 ±  3%     -44.9%      82557 ±  3%  softirqs.CPU14.RCU
     18949 ±  7%    +193.4%      55592 ±  3%  softirqs.CPU14.SCHED
    149419 ±  2%     -45.4%      81598 ±  4%  softirqs.CPU15.RCU
     18862 ±  5%    +190.1%      54728 ±  5%  softirqs.CPU15.SCHED
    152179 ±  2%     -45.1%      83605 ±  4%  softirqs.CPU16.RCU
     18938 ±  6%    +194.0%      55681 ±  3%  softirqs.CPU16.SCHED
    149048 ±  4%     -42.4%      85794 ±  3%  softirqs.CPU17.RCU
     18738 ±  6%    +201.8%      56551        softirqs.CPU17.SCHED
    152870 ±  2%     -43.9%      85742 ±  2%  softirqs.CPU18.RCU
     19081 ±  4%    +197.7%      56805        softirqs.CPU18.SCHED
    151075 ±  3%     -43.6%      85268 ±  2%  softirqs.CPU19.RCU
     19100 ±  4%    +198.5%      57021        softirqs.CPU19.SCHED
    153619 ±  2%     -41.8%      89365 ±  4%  softirqs.CPU2.RCU
     18521 ±  4%    +205.8%      56633        softirqs.CPU2.SCHED
    153780 ±  2%     -44.3%      85608 ±  2%  softirqs.CPU20.RCU
     19273 ±  3%    +193.4%      56557        softirqs.CPU20.SCHED
    153434 ±  3%     -45.7%      83359 ±  4%  softirqs.CPU21.RCU
     19223 ±  4%    +186.2%      55023 ±  4%  softirqs.CPU21.SCHED
    152924 ±  2%     -45.7%      83110 ±  5%  softirqs.CPU22.RCU
     19407 ±  2%    +181.4%      54609 ±  6%  softirqs.CPU22.SCHED
    151596 ±  3%     -44.5%      84128 ±  5%  softirqs.CPU23.RCU
     19213 ±  3%    +191.0%      55910 ±  3%  softirqs.CPU23.SCHED
    153267 ±  2%     -43.8%      86128        softirqs.CPU24.RCU
     18863 ±  5%    +199.2%      56436        softirqs.CPU24.SCHED
    153544 ±  2%     -42.9%      87725 ±  2%  softirqs.CPU25.RCU
     18870 ±  5%    +200.7%      56738        softirqs.CPU25.SCHED
    156600 ±  4%     -42.0%      90775 ±  3%  softirqs.CPU26.RCU
     18876 ±  8%    +200.4%      56703        softirqs.CPU26.SCHED
    153052 ±  3%     -42.6%      87851 ±  2%  softirqs.CPU27.RCU
     19567 ±  4%    +190.3%      56799        softirqs.CPU27.SCHED
    150630 ±  2%     -41.9%      87457 ±  2%  softirqs.CPU28.RCU
     19392 ±  2%    +188.9%      56019 ±  2%  softirqs.CPU28.SCHED
    151381 ±  2%     -43.3%      85893 ±  3%  softirqs.CPU29.RCU
     19692 ±  2%    +185.8%      56276 ±  2%  softirqs.CPU29.SCHED
    154940           -42.5%      89095 ±  2%  softirqs.CPU3.RCU
     19722 ±  3%    +189.3%      57065        softirqs.CPU3.SCHED
    151714 ±  2%     -42.1%      87770        softirqs.CPU30.RCU
     19523 ±  2%    +192.4%      57079        softirqs.CPU30.SCHED
    152456 ±  3%     -42.7%      87341        softirqs.CPU31.RCU
     19717          +188.6%      56902        softirqs.CPU31.SCHED
    150641 ±  2%     -43.0%      85853        softirqs.CPU32.RCU
     19699 ±  2%    +188.9%      56912        softirqs.CPU32.SCHED
    151150 ±  2%     -45.4%      82532 ±  6%  softirqs.CPU33.RCU
     19783 ±  2%    +179.5%      55286 ±  7%  softirqs.CPU33.SCHED
    151936 ±  2%     -43.2%      86274        softirqs.CPU34.RCU
     19701          +190.4%      57208        softirqs.CPU34.SCHED
    151156 ±  3%     -44.3%      84120 ±  2%  softirqs.CPU35.RCU
     19406 ±  2%    +192.5%      56769 ±  2%  softirqs.CPU35.SCHED
    153943 ±  2%     -44.7%      85178 ±  3%  softirqs.CPU36.RCU
     19300 ±  4%    +189.9%      55956 ±  2%  softirqs.CPU36.SCHED
    153694 ±  2%     -46.9%      81682 ±  7%  softirqs.CPU37.RCU
     19295 ±  4%    +174.7%      53008 ±  7%  softirqs.CPU37.SCHED
    152869 ±  2%     -44.8%      84443 ±  4%  softirqs.CPU38.RCU
     19113 ±  5%    +191.3%      55671 ±  2%  softirqs.CPU38.SCHED
    152186 ±  3%     -44.0%      85182 ±  4%  softirqs.CPU39.RCU
     18800 ±  6%    +196.8%      55800 ±  2%  softirqs.CPU39.SCHED
    152529 ±  2%     -42.0%      88506        softirqs.CPU4.RCU
     19761 ±  2%    +188.7%      57044        softirqs.CPU4.SCHED
    116661 ±  7%      -9.1%     106031        softirqs.CPU4.TIMER
    152366 ±  2%     -44.8%      84079 ±  5%  softirqs.CPU40.RCU
     19252 ±  3%    +181.1%      54123 ±  3%  softirqs.CPU40.SCHED
    147145 ±  6%     -41.5%      86076 ±  4%  softirqs.CPU41.RCU
     18425 ±  8%    +200.6%      55392 ±  2%  softirqs.CPU41.SCHED
    152769 ±  2%     -42.7%      87592 ±  3%  softirqs.CPU42.RCU
     19272 ±  4%    +192.8%      56420        softirqs.CPU42.SCHED
    152636 ±  3%     -43.3%      86493 ±  4%  softirqs.CPU43.RCU
     19383 ±  4%    +188.1%      55841 ±  3%  softirqs.CPU43.SCHED
    154975 ±  2%     -43.9%      86914 ±  3%  softirqs.CPU44.RCU
     19301 ±  3%    +194.3%      56810        softirqs.CPU44.SCHED
    154172 ±  2%     -44.9%      84893 ±  7%  softirqs.CPU45.RCU
     19395 ±  3%    +182.6%      54804 ±  4%  softirqs.CPU45.SCHED
    154519 ±  3%     -44.8%      85277 ±  5%  softirqs.CPU46.RCU
     19466 ±  3%    +189.4%      56333        softirqs.CPU46.SCHED
    152846 ±  3%     -44.7%      84581 ±  3%  softirqs.CPU47.RCU
     19137 ±  2%    +189.0%      55313 ±  4%  softirqs.CPU47.SCHED
    152438 ±  2%     -44.3%      84952 ±  5%  softirqs.CPU5.RCU
     19462 ±  2%    +182.3%      54942 ±  7%  softirqs.CPU5.SCHED
    115983 ±  7%     -11.5%     102647 ±  2%  softirqs.CPU5.TIMER
    153231 ±  3%     -42.7%      87823        softirqs.CPU6.RCU
     19431          +190.0%      56353 ±  2%  softirqs.CPU6.SCHED
    119504 ±  6%     -12.7%     104295 ±  2%  softirqs.CPU6.TIMER
    153186 ±  2%     -43.7%      86317 ±  3%  softirqs.CPU7.RCU
     19492 ±  2%    +190.1%      56549        softirqs.CPU7.SCHED
    154320 ±  2%     -42.8%      88211 ±  3%  softirqs.CPU8.RCU
     19523 ±  2%    +193.6%      57311        softirqs.CPU8.SCHED
    154132 ±  2%     -45.5%      84032 ±  6%  softirqs.CPU9.RCU
     19705 ±  2%    +179.2%      55016 ±  8%  softirqs.CPU9.SCHED
      4981 ± 66%    +395.5%      24681        softirqs.NET_RX
   7320015 ±  2%     -43.8%    4110873        softirqs.RCU
    929219 ±  3%    +190.2%    2696549        softirqs.SCHED
     30.48 ± 33%     -28.1        2.39 ±148%  perf-profile.calltrace.cycles-pp.semop
     21.61 ± 32%     -19.9        1.68 ±149%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.semop
     21.05 ± 32%     -19.4        1.65 ±149%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.semop
     12.92 ± 31%     -12.1        0.84 ±173%  perf-profile.calltrace.cycles-pp.do_semtimedop.do_syscall_64.entry_SYSCALL_64_after_hwframe.semop
      4.63 ± 36%      -4.3        0.32 ±173%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.semop
      2.47 ±  6%      -1.3        1.22 ± 11%  perf-profile.calltrace.cycles-pp.ipc_has_perm.security_sem_semop.do_semtimedop.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.45 ± 21%      -1.1        0.30 ±100%  perf-profile.calltrace.cycles-pp.osq_lock.rwsem_down_write_slowpath.down_write.do_shmat.__x64_sys_shmat
      2.03 ± 10%      -1.1        0.89 ±  9%  perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write.do_shmat.__x64_sys_shmat.do_syscall_64
      2.06 ±  9%      -1.1        0.93 ±  9%  perf-profile.calltrace.cycles-pp.down_write.do_shmat.__x64_sys_shmat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.77 ± 13%      -1.0        0.79 ± 10%  perf-profile.calltrace.cycles-pp.idr_find.ipc_obtain_object_check.do_semtimedop.do_syscall_64.entry_SYSCALL_64_after_hwframe
      3.30 ± 13%      -0.9        2.37 ± 13%  perf-profile.calltrace.cycles-pp.ipc_has_perm.security_ipc_permission.do_semtimedop.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.71 ± 12%      -0.8        0.91 ±  5%  perf-profile.calltrace.cycles-pp.do_smart_wakeup_zero.do_smart_update.do_semtimedop.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.39 ± 13%      -0.6        0.77 ± 13%  perf-profile.calltrace.cycles-pp.avc_has_perm.ipc_has_perm.security_sem_semop.do_semtimedop.do_syscall_64
      3.45 ±  3%      -0.6        2.84 ±  8%  perf-profile.calltrace.cycles-pp.__radix_tree_lookup.ipc_obtain_object_check.do_semtimedop.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.35 ± 22%      -0.6        0.77 ±  7%  perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write.shmctl_down.ksys_shmctl.do_syscall_64
      1.39 ± 21%      -0.6        0.80 ±  7%  perf-profile.calltrace.cycles-pp.down_write.shmctl_down.ksys_shmctl.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.09 ± 11%      -0.4        1.67 ±  9%  perf-profile.calltrace.cycles-pp.shm_close.remove_vma.__do_munmap.ksys_shmdt.do_syscall_64
      0.86 ± 16%      -0.3        0.58 ±  7%  perf-profile.calltrace.cycles-pp.__might_fault._copy_from_user.do_semtimedop.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.09            -0.2        0.90 ±  2%  perf-profile.calltrace.cycles-pp.do_page_fault.page_fault
      1.09 ±  2%      -0.2        0.91 ±  2%  perf-profile.calltrace.cycles-pp.page_fault
      1.07            -0.2        0.88 ±  3%  perf-profile.calltrace.cycles-pp.__do_page_fault.do_page_fault.page_fault
      0.96 ±  3%      -0.2        0.78 ±  3%  perf-profile.calltrace.cycles-pp.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
      0.91 ±  2%      -0.2        0.74 ±  4%  perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
      1.84 ± 22%      +0.8        2.69 ±  8%  perf-profile.calltrace.cycles-pp.__do_munmap.ksys_shmdt.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.87 ± 22%      +0.9        2.73 ±  8%  perf-profile.calltrace.cycles-pp.ksys_shmdt.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.81 ± 25%      +6.5        8.29 ± 32%  perf-profile.calltrace.cycles-pp.__x64_sys_shmget.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.80 ± 25%      +6.5        8.28 ± 32%  perf-profile.calltrace.cycles-pp.ipcget.__x64_sys_shmget.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.04 ± 30%      +6.7        7.79 ± 47%  perf-profile.calltrace.cycles-pp.osq_lock.rwsem_down_write_slowpath.down_write.ipcget.__x64_sys_shmget
      1.58 ± 20%      +6.9        8.51 ± 41%  perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write.ipcget.__x64_sys_shmget.do_syscall_64
      1.61 ± 20%      +7.0        8.57 ± 41%  perf-profile.calltrace.cycles-pp.down_write.ipcget.__x64_sys_shmget.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.83 ±  7%     +21.0       26.86 ±  8%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
      5.87 ±  7%     +21.0       26.90 ±  8%  perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      5.12 ±  8%     +21.3       26.38 ±  8%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      6.17 ±  7%     +21.8       27.95 ±  8%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      6.18 ±  7%     +21.8       27.96 ±  8%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
      6.18 ±  7%     +21.8       27.96 ±  8%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
      6.40 ±  7%     +22.1       28.53 ±  8%  perf-profile.calltrace.cycles-pp.secondary_startup_64
     30.90 ± 32%     -28.4        2.46 ±146%  perf-profile.children.cycles-pp.semop
     69.19           -13.7       55.49 ±  4%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     68.07           -13.3       54.79 ±  4%  perf-profile.children.cycles-pp.do_syscall_64
     31.73           -12.3       19.45 ±  6%  perf-profile.children.cycles-pp.do_semtimedop
     10.99            -3.9        7.09 ±  4%  perf-profile.children.cycles-pp.syscall_return_via_sysret
     10.43            -3.5        6.94 ±  5%  perf-profile.children.cycles-pp.entry_SYSCALL_64
      5.93 ±  9%      -2.1        3.86 ±  9%  perf-profile.children.cycles-pp.ipc_has_perm
      6.53            -2.0        4.57 ±  6%  perf-profile.children.cycles-pp.ipc_obtain_object_check
      3.71 ±  4%      -1.9        1.85 ±  5%  perf-profile.children.cycles-pp.security_sem_semop
      3.51            -1.6        1.93 ±  4%  perf-profile.children.cycles-pp.do_smart_update
      4.76 ±  9%      -1.3        3.45 ± 16%  perf-profile.children.cycles-pp.security_ipc_permission
      1.36 ± 35%      -1.3        0.11 ±142%  perf-profile.children.cycles-pp.shmat
      1.37 ± 35%      -1.3        0.12 ±140%  perf-profile.children.cycles-pp.shmdt
      2.62 ±  2%      -1.1        1.47 ±  5%  perf-profile.children.cycles-pp._copy_from_user
      3.19 ±  6%      -1.1        2.10 ±  5%  perf-profile.children.cycles-pp.__x64_sys_shmat
      3.18 ±  6%      -1.1        2.10 ±  6%  perf-profile.children.cycles-pp.do_shmat
      1.95 ±  4%      -1.1        0.88 ±  4%  perf-profile.children.cycles-pp.idr_find
      3.33 ± 13%      -1.0        2.30 ±  8%  perf-profile.children.cycles-pp.avc_has_perm
      1.96            -0.9        1.04 ±  4%  perf-profile.children.cycles-pp.do_smart_wakeup_zero
      3.71 ±  2%      -0.7        3.02 ±  7%  perf-profile.children.cycles-pp.__radix_tree_lookup
      2.09 ±  8%      -0.6        1.49 ±  4%  perf-profile.children.cycles-pp.ksys_shmctl
      2.07 ±  8%      -0.6        1.49 ±  4%  perf-profile.children.cycles-pp.shmctl_down
      1.20 ±  2%      -0.5        0.67 ±  5%  perf-profile.children.cycles-pp.__might_fault
      0.80 ±  7%      -0.5        0.33 ±  9%  perf-profile.children.cycles-pp.selinux_sem_semop
      1.04 ±  2%      -0.5        0.57        perf-profile.children.cycles-pp.perform_atomic_semop
      0.94 ±  9%      -0.4        0.52 ±  3%  perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
      0.85 ±  2%      -0.4        0.46 ± 19%  perf-profile.children.cycles-pp.selinux_ipc_permission
      1.36 ±  3%      -0.4        0.97 ±  2%  perf-profile.children.cycles-pp.rwsem_spin_on_owner
      1.21 ±  3%      -0.4        0.83 ±  7%  perf-profile.children.cycles-pp._raw_spin_lock
      1.10 ±  2%      -0.4        0.73        perf-profile.children.cycles-pp.__x64_sys_semop
      0.80            -0.4        0.44 ±  7%  perf-profile.children.cycles-pp.wake_const_ops
      2.09 ± 12%      -0.4        1.74 ±  7%  perf-profile.children.cycles-pp.shm_close
      1.21            -0.3        0.86 ±  4%  perf-profile.children.cycles-pp.exit_to_usermode_loop
      0.84 ±  3%      -0.3        0.50 ±  6%  perf-profile.children.cycles-pp.copy_user_generic_unrolled
      1.12            -0.3        0.81 ±  5%  perf-profile.children.cycles-pp.task_work_run
      1.05            -0.3        0.76 ±  4%  perf-profile.children.cycles-pp.__fput
      1.02            -0.3        0.76 ±  2%  perf-profile.children.cycles-pp.newseg
      0.85            -0.3        0.59 ±  5%  perf-profile.children.cycles-pp.dput
      0.81            -0.2        0.56 ±  5%  perf-profile.children.cycles-pp.dentry_kill
      0.80            -0.2        0.56 ±  5%  perf-profile.children.cycles-pp.__dentry_kill
      1.27            -0.2        1.04 ±  6%  perf-profile.children.cycles-pp.page_fault
      0.58 ±  2%      -0.2        0.35 ±  5%  perf-profile.children.cycles-pp.___might_sleep
      1.25            -0.2        1.02 ±  6%  perf-profile.children.cycles-pp.do_page_fault
      0.47 ±  9%      -0.2        0.25 ±  4%  perf-profile.children.cycles-pp.fpregs_assert_state_consistent
      1.23            -0.2        1.01 ±  5%  perf-profile.children.cycles-pp.__do_page_fault
      0.58 ±  8%      -0.2        0.36 ±  9%  perf-profile.children.cycles-pp.ipcperms
      1.10 ±  2%      -0.2        0.89 ±  5%  perf-profile.children.cycles-pp.handle_mm_fault
      0.58 ±  2%      -0.2        0.39 ±  7%  perf-profile.children.cycles-pp.evict
      0.49 ±  3%      -0.2        0.29 ±  6%  perf-profile.children.cycles-pp.__might_sleep
      1.05 ±  2%      -0.2        0.86 ±  5%  perf-profile.children.cycles-pp.__handle_mm_fault
      0.55 ±  5%      -0.2        0.35 ±  5%  perf-profile.children.cycles-pp.ksys_semget
      0.52 ±  4%      -0.2        0.32 ±  7%  perf-profile.children.cycles-pp.ipc_addid
      0.56 ±  7%      -0.2        0.38 ± 12%  perf-profile.children.cycles-pp.tick_sched_timer
      0.46 ±  3%      -0.2        0.28 ±  6%  perf-profile.children.cycles-pp.__check_object_size
      0.49 ± 10%      -0.2        0.32 ± 10%  perf-profile.children.cycles-pp.update_process_times
      0.51            -0.2        0.34 ±  9%  perf-profile.children.cycles-pp.shmem_evict_inode
      0.51 ±  8%      -0.2        0.34 ± 11%  perf-profile.children.cycles-pp.tick_sched_handle
      0.45 ±  4%      -0.2        0.29 ±  8%  perf-profile.children.cycles-pp.update_queue
      0.53 ±  4%      -0.2        0.37 ±  5%  perf-profile.children.cycles-pp.ksys_semctl
      0.47            -0.2        0.32 ±  9%  perf-profile.children.cycles-pp.shmem_truncate_range
      0.35 ± 12%      -0.2        0.20 ±  6%  perf-profile.children.cycles-pp.scheduler_tick
      0.45            -0.1        0.30 ± 10%  perf-profile.children.cycles-pp.shmem_undo_range
      0.32 ±  5%      -0.1        0.17 ± 11%  perf-profile.children.cycles-pp.__raw_spin_unlock
      0.60 ±  2%      -0.1        0.46 ±  9%  perf-profile.children.cycles-pp.unmap_region
      0.24 ± 18%      -0.1        0.10 ± 15%  perf-profile.children.cycles-pp.task_tick_fair
      0.41 ±  4%      -0.1        0.28 ±  7%  perf-profile.children.cycles-pp.semctl_down
      0.38 ±  4%      -0.1        0.24 ±  7%  perf-profile.children.cycles-pp.newary
      0.75 ±  9%      -0.1        0.62 ± 12%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.53 ±  4%      -0.1        0.41 ±  5%  perf-profile.children.cycles-pp.__do_fault
      0.45            -0.1        0.33 ± 10%  perf-profile.children.cycles-pp.unmap_vmas
      0.51 ±  4%      -0.1        0.39 ±  4%  perf-profile.children.cycles-pp.shmem_fault
      0.50 ±  4%      -0.1        0.39 ±  4%  perf-profile.children.cycles-pp.shmem_getpage_gfp
      0.42 ±  2%      -0.1        0.32 ± 11%  perf-profile.children.cycles-pp.unmap_page_range
      0.60 ±  2%      -0.1        0.49 ±  6%  perf-profile.children.cycles-pp.do_mmap
      0.42 ±  2%      -0.1        0.32 ±  8%  perf-profile.children.cycles-pp.mmap_region
      0.50 ±  4%      -0.1        0.41 ±  4%  perf-profile.children.cycles-pp.__shmem_file_setup
      0.26 ±  3%      -0.1        0.17 ± 13%  perf-profile.children.cycles-pp.release_pages
      0.18 ±  8%      -0.1        0.11 ± 11%  perf-profile.children.cycles-pp.idr_alloc_cyclic
      0.24 ±  7%      -0.1        0.17 ±  4%  perf-profile.children.cycles-pp.shm_destroy
      0.18 ±  6%      -0.1        0.11 ±  3%  perf-profile.children.cycles-pp.up_write
      0.16 ±  2%      -0.1        0.10 ± 11%  perf-profile.children.cycles-pp.check_stack_object
      0.18 ±  6%      -0.1        0.11 ± 11%  perf-profile.children.cycles-pp.__pagevec_release
      0.18 ±  7%      -0.1        0.12 ± 11%  perf-profile.children.cycles-pp.freeary
      0.17 ±  7%      -0.1        0.11 ±  8%  perf-profile.children.cycles-pp.idr_alloc_u32
      0.24 ±  7%      -0.1        0.18 ±  3%  perf-profile.children.cycles-pp.ipc_rmid
      0.13 ±  9%      -0.1        0.07 ± 10%  perf-profile.children.cycles-pp.idr_replace
      0.12 ±  6%      -0.1        0.07 ± 13%  perf-profile.children.cycles-pp.ktime_get_real_seconds
      0.14 ±  3%      -0.1        0.08 ± 12%  perf-profile.children.cycles-pp.osq_unlock
      0.17 ±  4%      -0.1        0.12 ±  5%  perf-profile.children.cycles-pp.shmem_alloc_and_acct_page
      0.07 ± 10%      -0.0        0.03 ±100%  perf-profile.children.cycles-pp.shm_release
      0.17 ±  3%      -0.0        0.12 ±  6%  perf-profile.children.cycles-pp.lru_add_drain
      0.16 ±  4%      -0.0        0.12 ±  7%  perf-profile.children.cycles-pp.shmem_alloc_page
      0.10 ±  4%      -0.0        0.06 ± 60%  perf-profile.children.cycles-pp.tlb_flush_mmu
      0.12 ±  5%      -0.0        0.08 ±  5%  perf-profile.children.cycles-pp.sprintf
      0.14 ±  3%      -0.0        0.10 ± 11%  perf-profile.children.cycles-pp.truncate_inode_page
      0.14 ±  3%      -0.0        0.10 ± 18%  perf-profile.children.cycles-pp.swapgs_restore_regs_and_return_to_usermode
      0.07 ±  7%      -0.0        0.03 ±100%  perf-profile.children.cycles-pp.mem_cgroup_try_charge_delay
      0.16 ±  7%      -0.0        0.12 ±  6%  perf-profile.children.cycles-pp.alloc_pages_vma
      0.11 ±  7%      -0.0        0.08 ±  5%  perf-profile.children.cycles-pp.vsnprintf
      0.11 ±  8%      -0.0        0.07 ± 10%  perf-profile.children.cycles-pp.ipc_obtain_object_idr
      0.08 ± 10%      -0.0        0.05 ± 58%  perf-profile.children.cycles-pp.free_unref_page_list
      0.07 ±  5%      -0.0        0.04 ± 58%  perf-profile.children.cycles-pp.security_vm_enough_memory_mm
      0.16            -0.0        0.13 ±  8%  perf-profile.children.cycles-pp.lru_add_drain_cpu
      0.10 ± 10%      -0.0        0.07 ± 15%  perf-profile.children.cycles-pp.shmem_add_to_page_cache
      0.08 ±  5%      -0.0        0.05 ± 59%  perf-profile.children.cycles-pp.__destroy_inode
      0.21 ±  6%      -0.0        0.18 ±  2%  perf-profile.children.cycles-pp.shmem_get_inode
      0.18 ±  3%      -0.0        0.15 ± 10%  perf-profile.children.cycles-pp.alloc_file_pseudo
      0.15 ±  5%      -0.0        0.11 ±  4%  perf-profile.children.cycles-pp.ipc_kht_remove
      0.07 ± 12%      -0.0        0.04 ± 57%  perf-profile.children.cycles-pp.update_cfs_group
      0.14 ±  6%      -0.0        0.11 ± 11%  perf-profile.children.cycles-pp.perf_event_mmap
      0.10 ±  5%      -0.0        0.07 ± 19%  perf-profile.children.cycles-pp.prepare_exit_to_usermode
      0.09 ±  4%      -0.0        0.07 ±  7%  perf-profile.children.cycles-pp.delete_from_page_cache
      0.09 ±  5%      -0.0        0.06 ± 11%  perf-profile.children.cycles-pp.finish_fault
      0.11 ±  4%      -0.0        0.08 ±  5%  perf-profile.children.cycles-pp.semctl_setval
      0.08 ±  5%      -0.0        0.06 ± 11%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      0.09 ±  4%      -0.0        0.06 ±  6%  perf-profile.children.cycles-pp.shm_mmap
      0.08 ±  5%      -0.0        0.06 ±  7%  perf-profile.children.cycles-pp.__delete_from_page_cache
      0.09 ±  8%      -0.0        0.07 ± 12%  perf-profile.children.cycles-pp.idr_get_free
      0.14            -0.0        0.12 ±  5%  perf-profile.children.cycles-pp.pagevec_lru_move_fn
      0.10 ±  4%      -0.0        0.08 ± 14%  perf-profile.children.cycles-pp.alloc_set_pte
      0.18 ±  2%      -0.0        0.16 ±  7%  perf-profile.children.cycles-pp.kmem_cache_alloc
      0.16 ±  2%      -0.0        0.15        perf-profile.children.cycles-pp.new_inode
      0.06 ± 17%      +0.0        0.09 ± 12%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.01 ±173%      +0.0        0.06 ±  7%  perf-profile.children.cycles-pp.update_rq_clock
      0.01 ±173%      +0.1        0.06 ± 17%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.01 ±173%      +0.1        0.07 ±  6%  perf-profile.children.cycles-pp.__slab_free
      0.00            +0.1        0.06 ±  7%  perf-profile.children.cycles-pp.__unwind_start
      0.09            +0.1        0.15 ±  2%  perf-profile.children.cycles-pp.pick_next_task_fair
      0.00            +0.1        0.06 ± 11%  perf-profile.children.cycles-pp.__update_load_avg_se
      0.01 ±173%      +0.1        0.08 ±  6%  perf-profile.children.cycles-pp.enqueue_hrtimer
      0.01 ±173%      +0.1        0.07 ±  5%  perf-profile.children.cycles-pp.timerqueue_add
      0.00            +0.1        0.06 ± 13%  perf-profile.children.cycles-pp.finish_task_switch
      0.00            +0.1        0.07 ± 13%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.00            +0.1        0.07 ±  7%  perf-profile.children.cycles-pp.select_idle_sibling
      0.00            +0.1        0.07 ± 12%  perf-profile.children.cycles-pp.unmapped_area_topdown
      0.03 ±100%      +0.1        0.11 ±  8%  perf-profile.children.cycles-pp.update_curr
      0.03 ±173%      +0.1        0.11 ± 33%  perf-profile.children.cycles-pp.vfs_read
      0.03 ±173%      +0.1        0.11 ± 34%  perf-profile.children.cycles-pp.ksys_read
      0.00            +0.2        0.15 ±  4%  perf-profile.children.cycles-pp.switch_mm_irqs_off
      0.00            +0.2        0.16 ±  5%  perf-profile.children.cycles-pp.update_load_avg
      0.00            +0.2        0.18 ±  3%  perf-profile.children.cycles-pp.select_task_rq_fair
      0.00            +0.2        0.19 ±  9%  perf-profile.children.cycles-pp.dequeue_entity
      0.00            +0.2        0.20 ±  6%  perf-profile.children.cycles-pp.unwind_next_frame
      0.00            +0.2        0.22 ±  7%  perf-profile.children.cycles-pp.dequeue_task_fair
      0.00            +0.3        0.25 ±  5%  perf-profile.children.cycles-pp.sched_ttwu_pending
      0.03 ±100%      +0.3        0.28 ±  5%  perf-profile.children.cycles-pp.arch_stack_walk
      0.03 ±100%      +0.3        0.32 ±  7%  perf-profile.children.cycles-pp.stack_trace_save_tsk
      0.00            +0.3        0.29 ±  4%  perf-profile.children.cycles-pp.schedule_idle
      0.06 ± 14%      +0.3        0.37 ±  7%  perf-profile.children.cycles-pp.__account_scheduler_latency
      0.14 ±  5%      +0.3        0.46 ±  6%  perf-profile.children.cycles-pp.schedule
      0.45 ±  2%      +0.4        0.88 ±  5%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.09 ± 11%      +0.4        0.53 ±  7%  perf-profile.children.cycles-pp.enqueue_entity
      0.36 ±  3%      +0.5        0.82 ±  7%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      0.10 ±  8%      +0.5        0.56 ±  6%  perf-profile.children.cycles-pp.enqueue_task_fair
      0.10 ±  8%      +0.5        0.57 ±  6%  perf-profile.children.cycles-pp.activate_task
      0.10 ±  5%      +0.5        0.57 ±  7%  perf-profile.children.cycles-pp.ttwu_do_activate
      0.18 ±  4%      +0.6        0.75 ±  4%  perf-profile.children.cycles-pp.__schedule
      0.55 ±  3%      +0.9        1.47 ±  3%  perf-profile.children.cycles-pp.wake_up_q
      0.43 ±  3%      +1.0        1.39 ±  3%  perf-profile.children.cycles-pp.try_to_wake_up
      0.61 ±  2%      +1.0        1.58 ±  3%  perf-profile.children.cycles-pp.rwsem_wake
      3.63 ±  5%      +6.5       10.13 ± 34%  perf-profile.children.cycles-pp.ipcget
      3.09 ±  6%      +6.7        9.80 ± 35%  perf-profile.children.cycles-pp.__x64_sys_shmget
      5.31 ±  9%     +21.1       26.38 ±  8%  perf-profile.children.cycles-pp.intel_idle
      6.08 ±  7%     +21.4       27.44 ±  8%  perf-profile.children.cycles-pp.cpuidle_enter_state
      6.08 ±  7%     +21.4       27.45 ±  8%  perf-profile.children.cycles-pp.cpuidle_enter
      6.18 ±  7%     +21.8       27.96 ±  8%  perf-profile.children.cycles-pp.start_secondary
      6.40 ±  7%     +22.1       28.53 ±  8%  perf-profile.children.cycles-pp.secondary_startup_64
      6.40 ±  7%     +22.1       28.53 ±  8%  perf-profile.children.cycles-pp.cpu_startup_entry
      6.40 ±  7%     +22.1       28.53 ±  8%  perf-profile.children.cycles-pp.do_idle
     18.65            -5.0       13.62 ±  4%  perf-profile.self.cycles-pp.do_syscall_64
     10.97            -3.9        7.07 ±  4%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      7.38            -3.1        4.26 ±  4%  perf-profile.self.cycles-pp.do_semtimedop
      9.98            -3.1        6.90 ±  5%  perf-profile.self.cycles-pp.entry_SYSCALL_64
      1.88 ±  4%      -1.0        0.84 ±  4%  perf-profile.self.cycles-pp.idr_find
      2.59 ±  4%      -1.0        1.58 ±  9%  perf-profile.self.cycles-pp.ipc_has_perm
      3.24 ± 13%      -1.0        2.23 ±  8%  perf-profile.self.cycles-pp.avc_has_perm
      3.65 ±  2%      -0.7        2.98 ±  7%  perf-profile.self.cycles-pp.__radix_tree_lookup
      1.19            -0.6        0.63 ±  2%  perf-profile.self.cycles-pp.do_smart_wakeup_zero
      1.03 ±  3%      -0.5        0.57 ±  5%  perf-profile.self.cycles-pp.do_smart_update
      1.01 ±  2%      -0.5        0.56        perf-profile.self.cycles-pp.perform_atomic_semop
      1.34 ±  3%      -0.4        0.90 ±  3%  perf-profile.self.cycles-pp.rwsem_spin_on_owner
      1.17            -0.4        0.72 ±  4%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.70 ±  5%      -0.4        0.28 ±  8%  perf-profile.self.cycles-pp.selinux_sem_semop
      1.15 ±  3%      -0.4        0.78 ±  5%  perf-profile.self.cycles-pp._raw_spin_lock
      0.98 ±  2%      -0.4        0.62 ±  2%  perf-profile.self.cycles-pp.__x64_sys_semop
      0.73 ±  2%      -0.3        0.40 ± 18%  perf-profile.self.cycles-pp.selinux_ipc_permission
      0.80 ±  2%      -0.3        0.47 ±  6%  perf-profile.self.cycles-pp.copy_user_generic_unrolled
      0.71 ±  2%      -0.3        0.39 ±  9%  perf-profile.self.cycles-pp.wake_const_ops
      0.56 ±  5%      -0.3        0.29 ±  6%  perf-profile.self.cycles-pp._copy_from_user
      1.03 ±  7%      -0.3        0.77 ± 10%  perf-profile.self.cycles-pp.ipc_obtain_object_check
      0.56 ±  3%      -0.2        0.34 ±  4%  perf-profile.self.cycles-pp.___might_sleep
      0.45 ±  9%      -0.2        0.23 ±  3%  perf-profile.self.cycles-pp.fpregs_assert_state_consistent
      0.56 ±  8%      -0.2        0.35 ±  9%  perf-profile.self.cycles-pp.ipcperms
      0.54 ± 12%      -0.2        0.35 ±  3%  perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
      0.45 ±  3%      -0.2        0.26 ±  6%  perf-profile.self.cycles-pp.__might_sleep
      0.38 ± 13%      -0.2        0.21 ±  7%  perf-profile.self.cycles-pp.security_sem_semop
      0.42 ±  4%      -0.2        0.27 ± 12%  perf-profile.self.cycles-pp.update_queue
      0.28 ±  5%      -0.1        0.16 ± 10%  perf-profile.self.cycles-pp.__raw_spin_unlock
      0.27 ±  6%      -0.1        0.16 ±  6%  perf-profile.self.cycles-pp.__might_fault
      0.29 ±  5%      -0.1        0.18 ±  8%  perf-profile.self.cycles-pp.__check_object_size
      0.17 ±  2%      -0.1        0.11 ± 13%  perf-profile.self.cycles-pp.ipc_addid
      0.17 ±  4%      -0.1        0.11 ±  3%  perf-profile.self.cycles-pp.up_write
      0.13 ±  3%      -0.1        0.08 ± 12%  perf-profile.self.cycles-pp.osq_unlock
      0.18 ±  3%      -0.1        0.13 ± 14%  perf-profile.self.cycles-pp.unmap_page_range
      0.14 ±  6%      -0.1        0.09 ± 12%  perf-profile.self.cycles-pp.check_stack_object
      0.13 ±  8%      -0.0        0.10 ±  4%  perf-profile.self.cycles-pp.ipc_kht_remove
      0.07 ± 12%      -0.0        0.04 ± 57%  perf-profile.self.cycles-pp.update_cfs_group
      0.18 ±  4%      -0.0        0.16 ±  4%  perf-profile.self.cycles-pp.down_write
      0.10 ± 12%      -0.0        0.08        perf-profile.self.cycles-pp.kmem_cache_alloc
      0.09 ±  9%      -0.0        0.07 ± 12%  perf-profile.self.cycles-pp.idr_get_free
      0.07 ±  6%      -0.0        0.05 ±  8%  perf-profile.self.cycles-pp.__handle_mm_fault
      0.01 ±173%      +0.1        0.06 ± 13%  perf-profile.self.cycles-pp.___perf_sw_event
      0.01 ±173%      +0.1        0.07 ±  7%  perf-profile.self.cycles-pp.timerqueue_add
      0.00            +0.1        0.06 ±  9%  perf-profile.self.cycles-pp.__update_load_avg_se
      0.01 ±173%      +0.1        0.07 ±  6%  perf-profile.self.cycles-pp.__slab_free
      0.00            +0.1        0.06 ± 14%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.00            +0.1        0.06        perf-profile.self.cycles-pp.update_curr
      0.06 ±  9%      +0.1        0.12 ±  9%  perf-profile.self.cycles-pp.try_to_wake_up
      0.00            +0.1        0.06 ±  6%  perf-profile.self.cycles-pp.__alloc_file
      0.00            +0.1        0.07 ±  7%  perf-profile.self.cycles-pp.update_load_avg
      0.00            +0.1        0.07 ± 13%  perf-profile.self.cycles-pp.unmapped_area_topdown
      0.00            +0.1        0.08 ±  5%  perf-profile.self.cycles-pp.select_task_rq_fair
      0.08 ±  5%      +0.1        0.16 ± 16%  perf-profile.self.cycles-pp.menu_select
      0.00            +0.1        0.10 ±  5%  perf-profile.self.cycles-pp.__schedule
      0.00            +0.1        0.10 ±  5%  perf-profile.self.cycles-pp.switch_mm_irqs_off
      0.00            +0.1        0.13 ±  6%  perf-profile.self.cycles-pp.unwind_next_frame
      0.15 ±  9%      +0.4        0.53 ± 10%  perf-profile.self.cycles-pp.rwsem_down_write_slowpath
      0.36 ±  4%      +0.5        0.82 ±  7%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      5.31 ±  9%     +21.0       26.34 ±  8%  perf-profile.self.cycles-pp.intel_idle


                                                                                
                               reaim.time.user_time                             
                                                                                
  4000 +-+------------------------------------------------------------------+   
       |                                                                    |   
  3500 +-++.+..+.+..+.+..+.+..+..+    +.+..+.+    +     +.+..+.+..+.+..+.+..|   
  3000 +-+                       :    :      :    :     :                   |   
       |                         :    :       :   ::    :                   |   
  2500 +-+                        :  :        :   ::   :                    |   
       O  O O  O O    O  O    O   :O :     O O: O:O: O :O O  O O  O O  O O  |   
  2000 +-+                        :  :        :  : :   :                    |   
       |                          :  :         : :  :  :                    |   
  1500 +-+                        : :          : :  : :                     |   
  1000 +-+                        : :          : :  : :                     |   
       |                           ::          : :  : :                     |   
   500 +-+                         ::           :    ::                     |   
       |                           :            :    :                      |   
     0 +-+----------O------O-----O----O-O-----------------------------------+   
                                                                                
                                                                                                                                                                
                              reaim.time.system_time                            
                                                                                
  7000 +-+------------------------------------------------------------------+   
       |..+.+..+      +.   +.    :    :    +.+    :     :         +    +    |   
  6000 +-+                       :    :      :    :     :                   |   
       |                         :   :       :    ::   :            O       |   
  5000 O-+O O  O O    O  O    O   :O :     O O: O:O: O :O O  O O  O    O O  |   
       |                          :  :        :  : :   :                    |   
  4000 +-+                        :  :        :  : :   :                    |   
       |                          :  :        :  :  :  :                    |   
  3000 +-+                        : :          : :  : :                     |   
       |                          : :          : :  : :                     |   
  2000 +-+                        : :          : :  : :                     |   
       |                           ::          ::   : :                     |   
  1000 +-+                         :            :    :                      |   
       |                           :            :    :                      |   
     0 +-+----------O------O-----O----O-O-----------------------------------+   
                                                                                
                                                                                                                                                                
                      reaim.time.percent_of_cpu_this_job_got                    
                                                                                
  3500 +-+------------------------------------------------------------------+   
       |..+.+..+.+..+.+..+.+..+..+    +.+..+.+    +     +.+..+.+..+.+..+.+..|   
  3000 +-+                       :    :      :    :     :                   |   
       |                         :    :      :    :     :                   |   
  2500 +-+            O          : O :     O O: O O: O :  O         O       |   
       O  O O  O O       O    O   :  :        :  : :   :O    O O  O    O O  |   
  2000 +-+                        :  :        :  : :   :                    |   
       |                          :  :        :  : :   :                    |   
  1500 +-+                        : :          : :  : :                     |   
       |                          : :          : :  : :                     |   
  1000 +-+                        : :          : :  : :                     |   
       |                           ::          ::   : :                     |   
   500 +-+                         :            :    :                      |   
       |                           :            :    :                      |   
     0 +-+----------O------O-----O----O-O-----------------------------------+   
                                                                                
                                                                                                                                                                
                             reaim.time.minor_page_faults                       
                                                                                
  3.5e+07 +-+---------------------------------------------------------------+   
          |.+..+.+..+.+..+.+..+.+.+    +..+.+..+    +   +..+.+..+.+..+.+..+.|   
    3e+07 +-+                     :    :       :    :   :                   |   
          |                       :    :       :    :   :                   |   
  2.5e+07 +-+                      :   :       :   ::   :                   |   
          O O  O O  O    O O    O  : O:     O  O:O :O:O:O  O O  O O  O O  O |   
    2e+07 +-+                      :  :         :  : : :                    |   
          |                        :  :         :  : : :                    |   
  1.5e+07 +-+                       : :         : :  : :                    |   
          |                         : :         : :  : :                    |   
    1e+07 +-+                       : :         : :  : :                    |   
          |                         ::           ::   :                     |   
    5e+06 +-+                        :           :    :                     |   
          |                          :           :    :                     |   
        0 +-+---------O-------O---O----O--O---------------------------------+   
                                                                                
                                                                                                                                                                
                         reaim.time.voluntary_context_switches                  
                                                                                
  1.8e+07 +-+---------------------------------------------------------------+   
          O O  O O  O    O O    O    O      O  O O  O O    O O  O O  O O  O |   
  1.6e+07 +-+                                           O                   |   
  1.4e+07 +-+                                                               |   
          |                                                                 |   
  1.2e+07 +-+                                                               |   
    1e+07 +-+                                                               |   
          |                                                                 |   
    8e+06 +-+                                                               |   
    6e+06 +-+                                                               |   
          |                                                                 |   
    4e+06 +-+                                                               |   
    2e+06 +-+..+.+..+.+..+.+..+.+.+..  +..+.+..+    +   +..+.+..+.+..+.+..+.|   
          |                           +         + .. + +                    |   
        0 +-+---------O-------O---O----O--O---------------------------------+   
                                                                                
                                                                                                                                                                
                        reaim.time.involuntary_context_switches                 
                                                                                
  1.2e+06 +-+---------------------------------------------------------------+   
          |.  .+.+..+.+..+.+..+.+.+      .+.  .+    +   +..+.+..+.+.   +..+.|   
    1e+06 +-+.                    :    +.   +. :    :   :                   |   
          |                       :    :       :    :   :                   |   
          |                        :   :       :   ::   :                   |   
   800000 +-+                      :   :        :  : : :                    |   
          |                        :  :         :  : : :                    |   
   600000 +-+                      :  :         :  : : :                    |   
          |                         : :         : :  : :                    |   
   400000 +-+                       : :         : :  : :                    |   
          |                         : :         : :  : :                    |   
          O O  O O  O    O O    O   :O:     O  O O: O O O  O O  O O  O O  O |   
   200000 +-+                        :           :    :                     |   
          |                          :           :    :                     |   
        0 +-+---------O-------O---O----O--O---------------------------------+   
                                                                                
                                                                                                                                                                
                                reaim.parent_time                               
                                                                                
  14 +-+--------------------------------------------------------------------+   
     |                                                                      |   
  12 O-+O O  O  O    O O     O    O       O O  O  O O  O  O O  O O  O  O O  |   
     |                                                                      |   
  10 +-+                                                                    |   
     |                                                                      |   
   8 +-++.+..+..+.+..+.+..+..+.+     +.+..+.+     +    +..+.+..+.+..+..+.+..|   
     |                         :     :      :     :    :                    |   
   6 +-+                        :   :        :   : :  :                     |   
     |                          :   :        :   : :  :                     |   
   4 +-+                         :  :         :  : :  :                     |   
     |                           : :          : :  : :                      |   
   2 +-+                         : :          : :   ::                      |   
     |                            :            :    :                       |   
   0 +-+----------O-------O----O-----O-O------------------------------------+   
                                                                                
                                                                                                                                                                
                                 reaim.child_utime                              
                                                                                
  120 +-+-------------------------------------------------------------------+   
      |    O  O O  +. O O       +    +..+.O. O O  O  O O. O    O       O O  |   
  100 O-+O                   O  :  O :       :    :    :    O     O O       |   
      |                         :    :       :    :    :                    |   
      |                          :   :        :  : :   :                    |   
   80 +-+                        :  :         :  : :  :                     |   
      |                          :  :         :  : :  :                     |   
   60 +-+                        :  :         :  : :  :                     |   
      |                           : :         : :   : :                     |   
   40 +-+                         : :         : :   : :                     |   
      |                           : :         : :   : :                     |   
      |                           ::           ::   ::                      |   
   20 +-+                          :           :     :                      |   
      |                            :           :     :                      |   
    0 +-+----------O-------O----O----O--O-----------------------------------+   
                                                                                
                                                                                                                                                                
                                 reaim.jobs_per_min                             
                                                                                
  6e+06 +-+-----------------------------------------------------------------+   
        |..+.+..+.+..+.+.. .+..+.+    +..+.+..+          .+.. .+..+.+..+.+..|   
  5e+06 +-+               +      :    :       :    +    +    +              |   
        |                        :    :       :    :    :                   |   
        |                         :   :       :    :    :                   |   
  4e+06 +-+                       :  :         :  : :  :                    |   
        O  O O  O O    O  O    O  : O:     O  O:O :O:O :O O  O O  O O  O O  |   
  3e+06 +-+                       :  :         :  : :  :                    |   
        |                          : :         :  : :  :                    |   
  2e+06 +-+                        : :         : :  : :                     |   
        |                          : :         : :  : :                     |   
        |                          ::           ::   ::                     |   
  1e+06 +-+                         :           ::   ::                     |   
        |                           :           :    :                      |   
      0 +-+----------O------O----O----O--O----------------------------------+   
                                                                                
                                                                                                                                                                
                             reaim.jobs_per_min_child                           
                                                                                
  800 +-+-------------------------------------------------------------------+   
      |.        +..+..+.+..+.+..+    :       +    +    +.   +..+.           |   
  700 +-+                       :    :       :    :    :                    |   
  600 +-+                       :    :       :    :    :                    |   
      |                          :  :        :   : :   :                    |   
  500 O-+O O  O O     O O    O   : O:     O  O:O :O: O:     O  O  O    O O  |   
      |                          :  :         :  : :  :O  O         O       |   
  400 +-+                        :  :         :  : :  :                     |   
      |                           : :         : :   : :                     |   
  300 +-+                         : :         : :   : :                     |   
  200 +-+                         : :         : :   : :                     |   
      |                           ::           ::   ::                      |   
  100 +-+                          :           :     :                      |   
      |                            :           :     :                      |   
    0 +-+----------O-------O----O----O--O-----------------------------------+   
                                                                                
                                                                                                                                                                
                                reaim.std_dev_time                              
                                                                                
  2.5 +-+-------------------------------------------------------------------+   
      |..+.+..+.+..+..+.+..+.+..+    +..+.+..+    +    +..+.+..+..+.+..+.+..|   
      |                         :    :       :    :    :                    |   
    2 +-+                       :    :       :    :    :                    |   
      |         O     O          :   :       O   :O:   :                    |   
      O  O O  O         O    O   : O:     O   :O : : O:O  O O  O  O O  O O  |   
  1.5 +-+                        :  :         :  : :  :                     |   
      |                          :  :         :  : :  :                     |   
    1 +-+                         : :         : :   : :                     |   
      |                           : :         : :   : :                     |   
      |                           : :         : :   : :                     |   
  0.5 +-+                         ::           ::   ::                      |   
      |                            :           :     :                      |   
      |                            :           :     :                      |   
    0 +-+----------O-------O----O----O--O-----------------------------------+   
                                                                                
                                                                                                                                                                
                              reaim.std_dev_percent                             
                                                                                
  60 +-+--------------------------------------------------------------------+   
     |                                                                      |   
  50 +-++.+..+..+.  .+.     .+.+     +.+..+.+            .+.+..       .+.  .|   
     |.           +.   +..+.   :     :      :     +    +.      +.+..+.   +. |   
     |                          :    :       :    :    :                    |   
  40 +-+                        :   :        :   ::   :                     |   
     |                          :   :        :   : :  :                     |   
  30 +-+                        :   :        :   : :  :                     |   
     |                           :  :         :  : :  :                     |   
  20 +-+                         : :          : :  : :                      |   
     |  O O  O  O    O O     O   :O:      O O :O: O:O: O  O O  O O  O  O O  |   
     O                           : :          : :  : :                      |   
  10 +-+                          ::           ::   ::                      |   
     |                            :            :    :                       |   
   0 +-+----------O-------O----O-----O-O------------------------------------+   
                                                                                
                                                                                                                                                                
                                    reaim.jti                                   
                                                                                
  90 +-+--------------------------------------------------------------------+   
     O  O O  O  O    O O     O    O       O O  O  O O  O  O O  O O  O  O O  |   
  80 +-+                                                                    |   
  70 +-+                                                                    |   
     |                                                                      |   
  60 +-+                                                                    |   
  50 +-++.+..+..+.+..+.+..+..+.+     +.+..+.+     +    +..+.+..+.+..+..+.+..|   
     |                         :     :      :     :    :                    |   
  40 +-+                        :   :        :   : :  :                     |   
  30 +-+                        :   :        :   : :  :                     |   
     |                           :  :         :  : :  :                     |   
  20 +-+                         : :          : :  : :                      |   
  10 +-+                         : :          : :   ::                      |   
     |                            :            :    :                       |   
   0 +-+----------O-------O----O-----O-O------------------------------------+   
                                                                                
                                                                                                                                                                
                               reaim.max_jobs_per_min                           
                                                                                
  6e+06 +-+-----------------------------------------------------------------+   
        |..+.+..+.+..+.+..+.+..+.+    +..+.+..+    +    +.+..+.+..+.+..+.+..|   
  5e+06 +-+                      :    :       :    :    :                   |   
        |                        :    :       :    :    :                   |   
        |                         :   :       :   ::   :                    |   
  4e+06 +-+                       :  :         :  : :  :                    |   
        O  O O  O O    O  O    O  : O:     O  O:O :O:O :O O  O O  O O  O O  |   
  3e+06 +-+                       :  :         :  : :  :                    |   
        |                          : :         : :  : :                     |   
  2e+06 +-+                        : :         : :  : :                     |   
        |                          : :         : :  : :                     |   
        |                          ::           ::   ::                     |   
  1e+06 +-+                         :           :    :                      |   
        |                           :           :    :                      |   
      0 +-+----------O------O----O----O--O----------------------------------+   
                                                                                
                                                                                                                                                                
                                    reaim.workload                              
                                                                                
  2.5e+07 +-+---------------------------------------------------------------+   
          |                                                                 |   
          |                            +..+.              .+.       .+.  .+.|   
    2e+07 +-+..+.+..+.+..+.+..+.+.+    :    +..+    +   +.   +..+.+.   +.   |   
          |                       :    :       :    :   :                   |   
          |                        :   :       :   ::   :                   |   
  1.5e+07 O-O  O O  O    O O    O  : O:     O  O:O :O:O:O  O O  O O  O O  O |   
          |                        :  :         :  : : :                    |   
    1e+07 +-+                      :  :         :  : : :                    |   
          |                         : :         : :  : :                    |   
          |                         : :         : :  : :                    |   
    5e+06 +-+                       : :         : :  : :                    |   
          |                         ::           ::   :                     |   
          |                          :           :    :                     |   
        0 +-+---------O-------O---O----O--O---------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample

***************************************************************************************************
lkp-ivb-2ep1: 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 64G memory




Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.2.0-rc5-00066-g4f23dbc" of type "text/plain" (196393 bytes)

View attachment "job-script" of type "text/plain" (7502 bytes)

View attachment "job.yaml" of type "text/plain" (5166 bytes)

View attachment "reproduce" of type "text/plain" (2797 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ