lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Message-ID: <20200409135029.GA2072@xsang-OptiPlex-9020>
Date:   Thu, 9 Apr 2020 21:50:29 +0800
From:   kernel test robot <oliver.sang@...el.com>
To:     Alexander Potapenko <glider@...gle.com>
Cc:     Linus Torvalds <torvalds@...ux-foundation.org>,
        Kees Cook <keescook@...omium.org>,
        Christoph Lameter <cl@...ux.com>,
        Masahiro Yamada <yamada.masahiro@...ionext.com>,
        "Serge E. Hallyn" <serge@...lyn.com>,
        Nick Desaulniers <ndesaulniers@...gle.com>,
        Kostya Serebryany <kcc@...gle.com>,
        Dmitry Vyukov <dvyukov@...gle.com>,
        Sandeep Patil <sspatil@...roid.com>,
        Laura Abbott <labbott@...hat.com>,
        Randy Dunlap <rdunlap@...radead.org>,
        Jann Horn <jannh@...gle.com>,
        Mark Rutland <mark.rutland@....com>,
        Marco Elver <elver@...gle.com>,
        Andrew Morton <akpm@...ux-foundation.org>,
        LKML <linux-kernel@...r.kernel.org>, lkp@...ts.01.org,
        OTC LSE PnP <otc.lse.pnp@...el.com>
Subject: [mm]  6471384af2:
 phoronix-test-suite.darktable.Masskrug.CPU-only.seconds 10.2% regression

Greeting,

FYI, we noticed a 10.2% regression of phoronix-test-suite.darktable.Masskrug.CPU-only.seconds due to commit:


commit: 6471384af2a6530696fc0203bafe4de41a23c9ef ("mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master

in testcase: phoronix-test-suite
on test machine: 12 threads Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz with 8G memory
with following parameters:

	test: darktable-1.0.4
	cpufreq_governor: performance

test-description: The Phoronix Test Suite is the most comprehensive testing and benchmarking platform available that provides an extensible framework for which new tests can be easily added.
test-url: http://www.phoronix-test-suite.com/

In addition to that, the commit also has significant impact on the following tests:
(Please be noted there was a previous report about this - https://lists.01.org/hyperkitty/list/lkp@lists.01.org/thread/H7PLLOT5ACTCUV3AO25F3OORLOWTXUD7/  [mm] 6471384af2: hackbench.throughput -4.9% regression)

+------------------+-------------------------------------------------------------------+
| testcase: change | hackbench: hackbench.throughput -4.9% regression                  |
| test machine     | 16 threads Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz with 32G memory |
| test parameters  | cpufreq_governor=performance                                      |
|                  | ipc=socket                                                        |
|                  | mode=process                                                      |
|                  | nr_threads=100%                                                   |
|                  | ucode=0xb8                                                        |
+------------------+-------------------------------------------------------------------+
| testcase: change | hackbench: hackbench.throughput -4.3% regression                  |
| test machine     | 16 threads Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz with 32G memory |
| test parameters  | cpufreq_governor=performance                                      |
|                  | mode=process                                                      |
|                  | nr_threads=100%                                                   |
|                  | ucode=0xb8                                                        |
+------------------+-------------------------------------------------------------------+


If you fix the issue, kindly add following tag
Reported-by: kernel test robot <oliver.sang@...el.com>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
compiler/cpufreq_governor/kconfig/rootfs/tbox_group/test/testcase:
  gcc-7/performance/x86_64-lck-7668-clear/clear-x86_64-phoronix-30140/lkp-cfl-d1/darktable-1.0.4/phoronix-test-suite

commit: 
  ba5c5e4a5d ("arm64: move jump_label_init() before parse_early_param()")
  6471384af2 ("mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options")

ba5c5e4a5da443e8 6471384af2a6530696fc0203baf 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          0:4           14%           0:8     perf-profile.children.cycles-pp.error_entry
          0:4            5%           0:8     perf-profile.self.cycles-pp.error_entry
         %stddev     %change         %stddev
             \          |                \  
     13.91           +10.2%      15.33        phoronix-test-suite.darktable.Masskrug.CPU-only.seconds
     10382          +246.4%      35966 ± 87%  phoronix-test-suite.time.involuntary_context_switches
    216.50 ±  5%    +115.1%     465.60 ± 47%  phoronix-test-suite.time.percent_of_cpu_this_job_got
     32.03          +183.6%      90.82 ± 72%  phoronix-test-suite.time.system_time
    463.82          +162.7%       1218 ± 73%  phoronix-test-suite.time.user_time
      6771 ±  2%    +172.8%      18471 ± 75%  phoronix-test-suite.time.voluntary_context_switches
     21.19           +14.2%      24.19 ± 12%  boot-time.boot
    205.62            +8.0%     222.02 ± 11%  boot-time.idle
     78.00           -26.2%      57.60 ± 31%  vmstat.cpu.id
     18.00 ±  5%    +104.4%      36.80 ± 45%  vmstat.cpu.us
     78.47           -20.6       57.89 ± 30%  mpstat.cpu.all.idle%
      1.90 ±  4%      +1.6        3.46 ± 33%  mpstat.cpu.all.sys%
     18.58 ±  5%     +18.9       37.50 ± 44%  mpstat.cpu.all.usr%
     70704            -4.9%      67246 ±  5%  proc-vmstat.nr_dirty_background_threshold
    678976            -4.1%     651286 ±  3%  proc-vmstat.nr_dirty_threshold
    813037            -8.1%     747488 ±  8%  proc-vmstat.nr_free_pages
      2746            +4.4%       2867 ±  3%  proc-vmstat.nr_page_table_pages
      2673 ± 10%     -46.8%       1423 ± 48%  proc-vmstat.pgmajfault
    392682 ±  7%     -26.0%     290401 ± 19%  proc-vmstat.pgpgin
 1.573e+08 ± 49%     -44.0%   88078468 ± 79%  cpuidle.C1E.time
   1405592 ± 55%     -70.1%     420174 ± 46%  cpuidle.C1E.usage
 1.542e+09 ± 10%     -42.8%   8.82e+08 ± 35%  cpuidle.C6.time
     43436 ± 20%     -50.4%      21536 ± 78%  cpuidle.C7s.time
    204.50 ±  3%     -43.9%     114.80 ± 57%  cpuidle.C7s.usage
    266262 ± 23%     -36.1%     170084 ± 40%  cpuidle.POLL.time
     49762 ± 16%     -33.9%      32908 ± 14%  cpuidle.POLL.usage
      8168 ±  3%     -11.8%       7202 ±  2%  slabinfo.dmaengine-unmap-2.active_objs
      8355 ±  3%     -11.2%       7418        slabinfo.dmaengine-unmap-2.num_objs
    831.00 ±  7%      -8.2%     763.00 ±  8%  slabinfo.fs_cache.active_objs
     36567           +24.6%      45552 ±  6%  slabinfo.kmalloc-32.active_objs
     38100           +28.5%      48943 ±  6%  slabinfo.kmalloc-32.num_objs
    710.50 ± 12%     -20.6%     564.00 ±  9%  slabinfo.pde_opener.num_objs
      1186 ±  2%     -21.1%     936.20 ± 18%  slabinfo.skbuff_head_cache.active_objs
      2688           -23.1%       2068 ±  9%  slabinfo.skbuff_head_cache.num_objs
     39.82 ±  4%     +50.2%      59.82 ± 23%  turbostat.Any%C0
    935.50 ±  4%     +92.8%       1803 ± 41%  turbostat.Avg_MHz
     21.82 ±  4%     +20.5       42.27 ± 41%  turbostat.Busy%
   1405286 ± 55%     -70.1%     419869 ± 46%  turbostat.C1E
     56.14 ± 15%     -25.4       30.74 ± 40%  turbostat.C6%
     46.41 ± 12%     -50.4%      23.02 ± 36%  turbostat.CPU%c6
     21.79 ±  6%     +60.4%      34.96 ± 29%  turbostat.CorWatt
     44.67 ± 11%     -42.4%      25.71 ± 33%  turbostat.Pkg%pc2
     22.13 ±  6%     +59.7%      35.36 ± 29%  turbostat.PkgWatt
      0.60           +51.1%       0.91 ± 30%  turbostat.RAMWatt
    149.84 ±  4%     +81.6%     272.05 ± 36%  turbostat.Totl%C0
     16562 ±  2%     -16.8%      13774 ± 19%  interrupts.127:IR-PCI-MSI.376832-edge.ahci[0000:00:17.0]
      1003 ± 21%     -33.3%     669.00 ± 21%  interrupts.CPU0.RES:Rescheduling_interrupts
    795.00           -25.9%     589.20 ± 28%  interrupts.CPU11.RES:Rescheduling_interrupts
    265.50 ± 29%   +1049.8%       3052 ± 96%  interrupts.CPU2.NMI:Non-maskable_interrupts
    265.50 ± 29%   +1049.8%       3052 ± 96%  interrupts.CPU2.PMI:Performance_monitoring_interrupts
      1019 ± 31%     -40.6%     605.60 ± 10%  interrupts.CPU2.RES:Rescheduling_interrupts
      1084 ± 21%     -25.7%     805.00 ± 18%  interrupts.CPU3.RES:Rescheduling_interrupts
     16562 ±  2%     -16.8%      13774 ± 19%  interrupts.CPU6.127:IR-PCI-MSI.376832-edge.ahci[0000:00:17.0]
    965.00 ±  9%     -27.3%     701.20 ± 21%  interrupts.CPU7.RES:Rescheduling_interrupts
    277.50 ± 26%   +1001.0%       3055 ± 97%  interrupts.CPU8.NMI:Non-maskable_interrupts
    277.50 ± 26%   +1001.0%       3055 ± 97%  interrupts.CPU8.PMI:Performance_monitoring_interrupts
    263.00 ± 38%   +1062.9%       3058 ± 97%  interrupts.CPU9.NMI:Non-maskable_interrupts
    263.00 ± 38%   +1062.9%       3058 ± 97%  interrupts.CPU9.PMI:Performance_monitoring_interrupts
     10560 ±  9%     -16.0%       8868        interrupts.RES:Rescheduling_interrupts
     28013           -12.4%      24539 ±  5%  softirqs.BLOCK
     29359 ±  7%     -23.1%      22569 ± 17%  softirqs.CPU0.SCHED
     28572 ±  3%     -22.2%      22216 ± 17%  softirqs.CPU1.SCHED
     28808 ±  6%     -19.7%      23129 ± 17%  softirqs.CPU10.SCHED
     28751 ±  5%     -18.1%      23544 ± 18%  softirqs.CPU11.SCHED
     28344 ±  5%     -19.1%      22936 ± 19%  softirqs.CPU2.SCHED
     28708 ±  4%     -18.5%      23401 ± 16%  softirqs.CPU3.SCHED
     27998 ±  7%     -16.3%      23438 ± 17%  softirqs.CPU4.SCHED
     28625 ±  3%     -17.7%      23553 ± 18%  softirqs.CPU5.SCHED
     27929           -12.4%      24458 ±  5%  softirqs.CPU6.BLOCK
     29319 ±  6%     -22.7%      22651 ± 17%  softirqs.CPU6.SCHED
     29015 ±  4%     -23.3%      22251 ± 18%  softirqs.CPU7.SCHED
     29141 ±  6%     -20.3%      23229 ± 17%  softirqs.CPU8.SCHED
     28629 ±  8%     -18.4%      23351 ± 17%  softirqs.CPU9.SCHED
    345274 ±  5%     -20.0%     276276 ± 17%  softirqs.SCHED
    123.62 ± 16%     -34.6%      80.79 ± 33%  perf-stat.i.MPKI
      6.40 ±  3%      -2.2        4.17 ± 23%  perf-stat.i.branch-miss-rate%
  37646369          +102.6%   76279695 ± 44%  perf-stat.i.cache-misses
 1.376e+08 ±  4%     +52.0%   2.09e+08 ± 24%  perf-stat.i.cache-references
      3.26 ± 16%     -18.1%       2.67 ± 12%  perf-stat.i.cpi
 1.109e+10 ±  4%     +94.3%  2.156e+10 ± 41%  perf-stat.i.cpu-cycles
     20.80           +56.7%      32.59 ± 21%  perf-stat.i.cpu-migrations
    872884 ±  7%     +26.4%    1103664 ± 10%  perf-stat.i.dTLB-load-misses
    170919 ±  6%     +51.6%     259124 ± 12%  perf-stat.i.dTLB-store-misses
   5747307 ±  3%    +204.8%   17519442 ± 44%  perf-stat.i.node-stores
      1.48            +9.0%       1.62 ±  2%  perf-stat.overall.cpi
     41.90 ±  2%      +4.6       46.51 ±  7%  perf-stat.overall.iTLB-load-miss-rate%
      0.67            -8.2%       0.62 ±  2%  perf-stat.overall.ipc
      0.00 ± 10%      -0.0        0.00 ± 54%  perf-stat.overall.node-load-miss-rate%
      0.00 ±  2%      -0.0        0.00 ± 53%  perf-stat.overall.node-store-miss-rate%
  37500511          +102.7%   76000098 ± 44%  perf-stat.ps.cache-misses
  1.37e+08 ±  4%     +52.0%  2.082e+08 ± 24%  perf-stat.ps.cache-references
 1.105e+10 ±  4%     +94.3%  2.148e+10 ± 41%  perf-stat.ps.cpu-cycles
     20.71           +56.7%      32.46 ± 21%  perf-stat.ps.cpu-migrations
    868910 ±  7%     +26.5%    1098800 ± 10%  perf-stat.ps.dTLB-load-misses
    170138 ±  6%     +51.6%     257986 ± 11%  perf-stat.ps.dTLB-store-misses
   5724997 ±  3%    +204.9%   17455501 ± 44%  perf-stat.ps.node-stores
     12327 ± 17%    +341.7%      54454 ± 84%  sched_debug.cfs_rq:/.exec_clock.avg
     11152 ± 16%    +376.1%      53103 ± 86%  sched_debug.cfs_rq:/.exec_clock.min
      1737 ± 32%     -31.6%       1188 ± 38%  sched_debug.cfs_rq:/.exec_clock.stddev
    142450 ±  2%     -19.0%     115440 ± 13%  sched_debug.cfs_rq:/.load.avg
    911295 ±  7%     -31.3%     625987 ± 21%  sched_debug.cfs_rq:/.load.max
    273423 ±  6%     -34.1%     180109 ± 19%  sched_debug.cfs_rq:/.load.stddev
    247.81 ±  7%     -19.8%     198.69 ± 11%  sched_debug.cfs_rq:/.load_avg.avg
    930.00 ±  8%     -22.0%     725.19 ± 21%  sched_debug.cfs_rq:/.load_avg.max
    291.63 ±  8%     -29.4%     205.75 ± 20%  sched_debug.cfs_rq:/.load_avg.stddev
    111830 ± 22%    +452.9%     618271 ± 88%  sched_debug.cfs_rq:/.min_vruntime.avg
    119158 ± 22%    +428.3%     629514 ± 87%  sched_debug.cfs_rq:/.min_vruntime.max
    107831 ± 22%    +464.1%     608276 ± 89%  sched_debug.cfs_rq:/.min_vruntime.min
      1.03 ±  4%    +114.2%       2.21 ± 64%  sched_debug.cfs_rq:/.nr_spread_over.avg
    732.38 ±  3%     -34.2%     481.71 ± 21%  sched_debug.cfs_rq:/.runnable_load_avg.max
    208.65 ±  5%     -37.6%     130.30 ± 26%  sched_debug.cfs_rq:/.runnable_load_avg.stddev
    911168 ±  7%     -32.1%     618393 ± 22%  sched_debug.cfs_rq:/.runnable_weight.max
    274250 ±  6%     -35.1%     178090 ± 19%  sched_debug.cfs_rq:/.runnable_weight.stddev
     -5527           +99.2%     -11008        sched_debug.cfs_rq:/.spread0.min
    254.75 ± 30%    +110.6%     536.44 ± 27%  sched_debug.cfs_rq:/.util_avg.min
    214.48 ±  7%     -26.5%     157.58 ± 22%  sched_debug.cfs_rq:/.util_avg.stddev
      7077           -15.8%       5955 ± 12%  sched_debug.cpu.nr_switches.stddev
    -29.75           +51.8%     -45.15        sched_debug.cpu.nr_uninterruptible.min
     19316 ±  6%     -16.3%      16170 ± 12%  sched_debug.cpu.sched_goidle.max
      2080 ±  9%     -23.6%       1589 ± 20%  sched_debug.cpu.sched_goidle.stddev
     78.60           -26.0       52.62 ± 57%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.call_cpuidle.do_idle.cpu_startup_entry
     28.69 ± 10%     -16.4       12.28 ± 48%  perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.call_cpuidle.do_idle
     28.29 ± 11%     -16.2       12.07 ± 48%  perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.call_cpuidle
     15.98 ± 10%      -9.5        6.50 ± 51%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
     12.21 ± 18%      -7.7        4.51 ± 49%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
      7.04 ± 23%      -4.4        2.68 ± 39%  perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      7.73 ± 18%      -3.9        3.81 ± 49%  perf-profile.calltrace.cycles-pp.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      6.01 ± 23%      -3.7        2.27 ± 38%  perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt
      5.34 ± 23%      -3.3        2.03 ± 40%  perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
      5.21 ± 29%      -2.8        2.42 ± 42%  perf-profile.calltrace.cycles-pp.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
      6.14 ± 10%      -2.4        3.76 ± 61%  perf-profile.calltrace.cycles-pp.cpuidle_enter.call_cpuidle.do_idle.cpu_startup_entry.rest_init
      6.14 ± 10%      -2.4        3.77 ± 61%  perf-profile.calltrace.cycles-pp.call_cpuidle.do_idle.cpu_startup_entry.rest_init.arch_call_rest_init
      5.24 ± 19%      -2.3        2.91 ± 71%  perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.cpuidle_select.do_idle.cpu_startup_entry
      3.20 ±  3%      -2.1        1.07 ± 32%  perf-profile.calltrace.cycles-pp.irq_enter.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      3.31 ± 31%      -1.8        1.55 ± 44%  perf-profile.calltrace.cycles-pp.run_rebalance_domains.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt
      3.46 ± 15%      -1.5        2.01 ± 71%  perf-profile.calltrace.cycles-pp.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.cpuidle_select.do_idle
      2.05 ± 29%      -1.3        0.80 ± 53%  perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
      2.62 ±  8%      -1.2        1.39 ± 83%  perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter.call_cpuidle.do_idle.cpu_startup_entry
      2.01 ± 28%      -1.2        0.82 ± 64%  perf-profile.calltrace.cycles-pp.rebalance_domains.run_rebalance_domains.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt
      1.65 ± 20%      -0.9        0.72 ± 63%  perf-profile.calltrace.cycles-pp.load_balance.rebalance_domains.run_rebalance_domains.__softirqentry_text_start.irq_exit
      1.31 ± 26%      -0.8        0.52 ± 82%  perf-profile.calltrace.cycles-pp.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      1.56 ± 14%      -0.8        0.80 ± 85%  perf-profile.calltrace.cycles-pp.tick_program_event.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
      1.19 ± 43%      -0.7        0.45 ± 84%  perf-profile.calltrace.cycles-pp.update_blocked_averages.run_rebalance_domains.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt
      1.16 ± 19%      -0.6        0.60 ± 83%  perf-profile.calltrace.cycles-pp.find_next_bit.get_next_timer_interrupt.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select
      0.83 ± 13%      -0.4        0.44 ± 86%  perf-profile.calltrace.cycles-pp.enqueue_hrtimer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      0.93 ± 14%      -0.4        0.55 ± 83%  perf-profile.calltrace.cycles-pp.__next_timer_interrupt.get_next_timer_interrupt.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select
     79.88           -26.4       53.48 ± 56%  perf-profile.children.cycles-pp.cpuidle_enter
     80.06           -26.3       53.75 ± 56%  perf-profile.children.cycles-pp.call_cpuidle
     79.23           -26.2       53.05 ± 56%  perf-profile.children.cycles-pp.cpuidle_enter_state
     30.27 ± 10%     -15.4       14.88 ± 33%  perf-profile.children.cycles-pp.apic_timer_interrupt
     28.33 ± 11%     -14.6       13.72 ± 32%  perf-profile.children.cycles-pp.smp_apic_timer_interrupt
     16.20 ± 10%      -8.3        7.85 ± 30%  perf-profile.children.cycles-pp.hrtimer_interrupt
     12.29 ± 18%      -6.6        5.70 ± 28%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      7.33 ± 20%      -4.1        3.19 ± 25%  perf-profile.children.cycles-pp.tick_sched_timer
      6.42 ± 21%      -3.7        2.71 ± 25%  perf-profile.children.cycles-pp.tick_sched_handle
      7.78 ± 18%      -3.6        4.16 ± 40%  perf-profile.children.cycles-pp.irq_exit
      5.76 ± 22%      -3.3        2.48 ± 25%  perf-profile.children.cycles-pp.update_process_times
      5.80 ± 27%      -2.9        2.90 ± 35%  perf-profile.children.cycles-pp.__softirqentry_text_start
      5.71 ± 18%      -2.5        3.19 ± 64%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      3.44 ±  3%      -2.2        1.25 ± 25%  perf-profile.children.cycles-pp.irq_enter
      3.65 ± 32%      -1.9        1.70 ± 41%  perf-profile.children.cycles-pp.run_rebalance_domains
      2.25 ±  8%      -1.5        0.80 ± 41%  perf-profile.children.cycles-pp.ktime_get
      2.25 ± 27%      -1.3        0.99 ± 48%  perf-profile.children.cycles-pp.rebalance_domains
      1.77 ± 30%      -1.2        0.57 ± 35%  perf-profile.children.cycles-pp.tick_irq_enter
      2.33 ± 23%      -1.1        1.24 ± 27%  perf-profile.children.cycles-pp.scheduler_tick
      1.81 ± 22%      -0.9        0.86 ± 49%  perf-profile.children.cycles-pp.load_balance
      1.23 ± 45%      -0.8        0.42 ± 21%  perf-profile.children.cycles-pp.update_rq_clock
      1.52 ± 23%      -0.7        0.82 ± 69%  perf-profile.children.cycles-pp.find_next_bit
      1.14 ± 11%      -0.7        0.44 ± 33%  perf-profile.children.cycles-pp.__remove_hrtimer
      1.28 ± 35%      -0.7        0.59 ± 51%  perf-profile.children.cycles-pp.read_tsc
      1.56 ± 32%      -0.7        0.90 ± 34%  perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
      1.69 ± 16%      -0.6        1.07 ± 52%  perf-profile.children.cycles-pp.tick_program_event
      0.99 ±  2%      -0.6        0.37 ± 26%  perf-profile.children.cycles-pp.timerqueue_del
      1.12 ±  5%      -0.6        0.51 ± 60%  perf-profile.children.cycles-pp.find_busiest_group
      0.89 ± 34%      -0.6        0.32 ± 33%  perf-profile.children.cycles-pp.update_irq_load_avg
      0.78 ±  5%      -0.6        0.22 ± 22%  perf-profile.children.cycles-pp.run_local_timers
      1.19 ±  3%      -0.5        0.64 ± 47%  perf-profile.children.cycles-pp.irqtime_account_irq
      0.96 ±  5%      -0.5        0.44 ± 26%  perf-profile.children.cycles-pp.run_timer_softirq
      0.97 ±  5%      -0.5        0.51 ± 45%  perf-profile.children.cycles-pp.timerqueue_add
      0.69 ±  8%      -0.4        0.26 ± 49%  perf-profile.children.cycles-pp.rcu_sched_clock_irq
      0.67 ± 48%      -0.4        0.27 ± 27%  perf-profile.children.cycles-pp.rcu_irq_enter
      1.21 ± 15%      -0.4        0.83 ± 50%  perf-profile.children.cycles-pp.clockevents_program_event
      0.58 ± 32%      -0.4        0.22 ± 67%  perf-profile.children.cycles-pp.__hrtimer_get_next_event
      0.60 ± 20%      -0.4        0.24 ± 35%  perf-profile.children.cycles-pp.rcu_cleanup_after_idle
      1.09 ± 19%      -0.4        0.74 ± 21%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.94 ± 12%      -0.3        0.59 ± 52%  perf-profile.children.cycles-pp.enqueue_hrtimer
      0.53 ± 26%      -0.3        0.21 ± 56%  perf-profile.children.cycles-pp.perf_event_task_tick
      0.46 ± 45%      -0.3        0.19 ± 65%  perf-profile.children.cycles-pp.ktime_get_update_offsets_now
      0.39 ± 37%      -0.3        0.13 ± 65%  perf-profile.children.cycles-pp.rb_erase
      0.34 ± 26%      -0.3        0.09 ± 81%  perf-profile.children.cycles-pp.raise_softirq
      0.57 ± 18%      -0.2        0.32 ± 62%  perf-profile.children.cycles-pp.rcu_idle_enter
      0.47 ±  2%      -0.2        0.23 ± 41%  perf-profile.children.cycles-pp.nr_iowait_cpu
      0.41 ±  4%      -0.2        0.18 ± 44%  perf-profile.children.cycles-pp.irq_work_tick
      0.65 ± 25%      -0.2        0.42 ± 31%  perf-profile.children.cycles-pp._raw_spin_lock
      0.36 ± 47%      -0.2        0.14 ± 32%  perf-profile.children.cycles-pp.ksoftirqd_running
      0.46 ±  6%      -0.2        0.25 ± 55%  perf-profile.children.cycles-pp.tick_sched_do_timer
      0.39 ± 12%      -0.2        0.18 ± 31%  perf-profile.children.cycles-pp.rcu_try_advance_all_cbs
      0.36 ±  6%      -0.2        0.15 ± 79%  perf-profile.children.cycles-pp.leave_mm
      0.27 ± 29%      -0.2        0.06 ±101%  perf-profile.children.cycles-pp.collect_expired_timers
      0.43 ± 12%      -0.2        0.23 ± 51%  perf-profile.children.cycles-pp.tick_do_update_jiffies64
      0.26 ± 45%      -0.2        0.05 ± 90%  perf-profile.children.cycles-pp.tick_check_oneshot_broadcast_this_cpu
      0.53 ±  9%      -0.2        0.33 ± 55%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.48            -0.2        0.28 ± 45%  perf-profile.children.cycles-pp.rb_next
      0.36 ±  6%      -0.2        0.17 ± 46%  perf-profile.children.cycles-pp.timekeeping_advance
      0.41 ±  4%      -0.2        0.21 ± 51%  perf-profile.children.cycles-pp.arch_cpu_idle_enter
      0.38            -0.2        0.19 ± 48%  perf-profile.children.cycles-pp.update_wall_time
      0.21 ± 52%      -0.2        0.03 ± 81%  perf-profile.children.cycles-pp.__bitmap_and
      0.24 ± 19%      -0.2        0.06 ± 81%  perf-profile.children.cycles-pp.rcu_qs
      0.40 ±  7%      -0.2        0.23 ± 72%  perf-profile.children.cycles-pp.tick_nohz_tick_stopped
      0.47 ± 27%      -0.2        0.30 ± 18%  perf-profile.children.cycles-pp.interrupt_entry
      0.48 ±  9%      -0.2        0.32 ± 21%  perf-profile.children.cycles-pp.kthread
      0.32 ±  6%      -0.2        0.16 ± 69%  perf-profile.children.cycles-pp.find_next_and_bit
      0.25 ± 52%      -0.2        0.09 ± 91%  perf-profile.children.cycles-pp.sched_clock_idle_sleep_event
      0.38            -0.1        0.23 ± 50%  perf-profile.children.cycles-pp.x86_pmu_enable
      0.48 ±  9%      -0.1        0.34 ± 23%  perf-profile.children.cycles-pp.ret_from_fork
      0.35 ±  4%      -0.1        0.21 ± 47%  perf-profile.children.cycles-pp.__intel_pmu_enable_all
      0.35 ±  4%      -0.1        0.22 ± 47%  perf-profile.children.cycles-pp.intel_pmu_enable_all
      0.44 ± 20%      -0.1        0.30 ± 28%  perf-profile.children.cycles-pp.try_to_wake_up
      0.15 ± 31%      -0.1        0.05 ± 84%  perf-profile.children.cycles-pp.__open64_nocancel
      0.17 ± 19%      -0.1        0.08 ± 95%  perf-profile.children.cycles-pp.check_tsc_unstable
      0.15 ± 22%      -0.1        0.06 ± 95%  perf-profile.children.cycles-pp.entry_SYSCALL_64
      0.21 ± 34%      -0.1        0.12 ± 49%  perf-profile.children.cycles-pp.expire_timers
      0.21 ± 34%      -0.1        0.12 ± 49%  perf-profile.children.cycles-pp.call_timer_fn
      0.24 ± 19%      -0.1        0.14 ± 48%  perf-profile.children.cycles-pp.perf_duration_warn
      0.24 ± 19%      -0.1        0.14 ± 48%  perf-profile.children.cycles-pp.printk
      0.24 ± 19%      -0.1        0.14 ± 48%  perf-profile.children.cycles-pp.vprintk_func
      0.24 ± 19%      -0.1        0.14 ± 48%  perf-profile.children.cycles-pp.vprintk_default
      0.24 ± 19%      -0.1        0.15 ± 46%  perf-profile.children.cycles-pp.irq_work_interrupt
      0.24 ± 19%      -0.1        0.15 ± 46%  perf-profile.children.cycles-pp.smp_irq_work_interrupt
      0.24 ± 19%      -0.1        0.15 ± 46%  perf-profile.children.cycles-pp.irq_work_run
      0.13 ± 23%      -0.1        0.05 ± 90%  perf-profile.children.cycles-pp.can_stop_idle_tick
      0.23 ± 39%      -0.1        0.16 ± 49%  perf-profile.children.cycles-pp.command_substitute
      0.14 ± 35%      -0.1        0.07 ± 54%  perf-profile.children.cycles-pp.default_wake_function
      0.14 ± 35%      -0.1        0.07 ± 69%  perf-profile.children.cycles-pp.update_rt_rq_load_avg
      0.15 ± 31%      -0.1        0.08 ± 60%  perf-profile.children.cycles-pp.setlocale
      0.13 ±  7%      -0.1        0.07 ± 60%  perf-profile.children.cycles-pp.wp_page_copy
      0.12 ± 21%      -0.1        0.06 ± 84%  perf-profile.children.cycles-pp.task_tick_idle
      0.09 ± 17%      -0.1        0.03 ± 82%  perf-profile.children.cycles-pp.wait_for
      0.13 ±  7%      -0.0        0.09 ± 30%  perf-profile.children.cycles-pp.intel_atomic_commit_work
      0.13 ±  7%      -0.0        0.09 ± 30%  perf-profile.children.cycles-pp.intel_atomic_commit_tail
      0.17 ±  8%      +0.3        0.43 ± 48%  perf-profile.children.cycles-pp.__vfs_read
      0.20 ±  4%      +0.3        0.46 ± 49%  perf-profile.children.cycles-pp.__x64_sys_read
      0.20 ±  4%      +0.3        0.46 ± 49%  perf-profile.children.cycles-pp.ksys_read
      0.17 ± 19%      +0.3        0.44 ± 48%  perf-profile.children.cycles-pp.vfs_read
      0.03 ±100%      +2.5        2.49 ±109%  perf-profile.children.cycles-pp.__do_munmap
      2.96 ±  3%      -1.5        1.49 ± 54%  perf-profile.self.cycles-pp.cpuidle_enter_state
      1.64 ±  2%      -1.1        0.49 ± 35%  perf-profile.self.cycles-pp.ktime_get
      1.23 ± 33%      -0.7        0.54 ± 53%  perf-profile.self.cycles-pp.read_tsc
      1.19 ± 37%      -0.6        0.55 ± 66%  perf-profile.self.cycles-pp.__hrtimer_next_event_base
      1.36 ± 23%      -0.6        0.74 ± 69%  perf-profile.self.cycles-pp.find_next_bit
      0.89 ± 34%      -0.6        0.32 ± 33%  perf-profile.self.cycles-pp.update_irq_load_avg
      0.84 ± 54%      -0.5        0.32 ± 22%  perf-profile.self.cycles-pp.update_blocked_averages
      0.83 ± 19%      -0.4        0.41 ± 33%  perf-profile.self.cycles-pp.__hrtimer_run_queues
      0.56 ±  6%      -0.4        0.15 ± 33%  perf-profile.self.cycles-pp.run_local_timers
      0.85 ±  9%      -0.4        0.44 ± 45%  perf-profile.self.cycles-pp.timerqueue_add
      0.64 ±  8%      -0.4        0.24 ± 43%  perf-profile.self.cycles-pp.rcu_sched_clock_irq
      0.75 ± 36%      -0.4        0.35 ± 15%  perf-profile.self.cycles-pp.hrtimer_interrupt
      0.75 ± 10%      -0.4        0.37 ± 64%  perf-profile.self.cycles-pp.find_busiest_group
      0.74 ±  2%      -0.4        0.36 ± 38%  perf-profile.self.cycles-pp.irqtime_account_irq
      1.09 ± 19%      -0.4        0.74 ± 21%  perf-profile.self.cycles-pp.native_irq_return_iret
      0.54 ± 32%      -0.3        0.22 ± 55%  perf-profile.self.cycles-pp.get_next_timer_interrupt
      0.53 ± 26%      -0.3        0.21 ± 56%  perf-profile.self.cycles-pp.perf_event_task_tick
      0.45 ± 46%      -0.3        0.13 ± 40%  perf-profile.self.cycles-pp.rebalance_domains
      0.48            -0.3        0.19 ± 27%  perf-profile.self.cycles-pp.run_timer_softirq
      0.43 ± 33%      -0.3        0.15 ± 22%  perf-profile.self.cycles-pp.smp_apic_timer_interrupt
      0.48 ± 41%      -0.3        0.22 ± 63%  perf-profile.self.cycles-pp.tick_nohz_next_event
      0.38 ± 36%      -0.2        0.13 ± 66%  perf-profile.self.cycles-pp.rb_erase
      0.38 ± 49%      -0.2        0.13 ± 46%  perf-profile.self.cycles-pp.cpuidle_governor_latency_req
      0.31 ±  9%      -0.2        0.07 ± 66%  perf-profile.self.cycles-pp.timerqueue_del
      0.47 ±  2%      -0.2        0.23 ± 41%  perf-profile.self.cycles-pp.nr_iowait_cpu
      0.36 ± 47%      -0.2        0.13 ± 27%  perf-profile.self.cycles-pp.ksoftirqd_running
      0.59 ± 27%      -0.2        0.37 ± 30%  perf-profile.self.cycles-pp._raw_spin_lock
      0.30 ±  4%      -0.2        0.09 ± 76%  perf-profile.self.cycles-pp.leave_mm
      0.27 ± 29%      -0.2        0.06 ±101%  perf-profile.self.cycles-pp.collect_expired_timers
      0.37 ± 43%      -0.2        0.16 ± 55%  perf-profile.self.cycles-pp.ktime_get_update_offsets_now
      0.26 ± 45%      -0.2        0.05 ± 90%  perf-profile.self.cycles-pp.tick_check_oneshot_broadcast_this_cpu
      0.47 ±  2%      -0.2        0.27 ± 46%  perf-profile.self.cycles-pp.rb_next
      0.36 ±  4%      -0.2        0.16 ± 49%  perf-profile.self.cycles-pp.irq_work_tick
      0.36 ± 20%      -0.2        0.17 ± 29%  perf-profile.self.cycles-pp.rcu_try_advance_all_cbs
      0.35 ±  8%      -0.2        0.16 ± 83%  perf-profile.self.cycles-pp.tick_nohz_tick_stopped
      0.49 ± 13%      -0.2        0.31 ± 63%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.21 ± 52%      -0.2        0.03 ± 81%  perf-profile.self.cycles-pp.__bitmap_and
      0.22 ± 13%      -0.2        0.05 ± 87%  perf-profile.self.cycles-pp.rcu_qs
      0.27 ± 47%      -0.2        0.09 ± 34%  perf-profile.self.cycles-pp.rcu_cleanup_after_idle
      0.23 ± 47%      -0.2        0.06 ± 86%  perf-profile.self.cycles-pp.raise_softirq
      0.47 ± 27%      -0.2        0.30 ± 18%  perf-profile.self.cycles-pp.interrupt_entry
      0.31 ±  9%      -0.2        0.15 ± 70%  perf-profile.self.cycles-pp.find_next_and_bit
      0.20 ± 50%      -0.1        0.05 ± 55%  perf-profile.self.cycles-pp.update_process_times
      0.35 ±  4%      -0.1        0.21 ± 47%  perf-profile.self.cycles-pp.__intel_pmu_enable_all
      0.19            -0.1        0.05 ± 88%  perf-profile.self.cycles-pp.rcu_eqs_exit
      0.23 ±  2%      -0.1        0.10 ± 70%  perf-profile.self.cycles-pp.tick_program_event
      0.19            -0.1        0.08 ± 82%  perf-profile.self.cycles-pp.rcu_irq_exit
      0.15 ±  6%      -0.1        0.04 ± 86%  perf-profile.self.cycles-pp.account_process_tick
      0.13 ± 46%      -0.1        0.03 ± 84%  perf-profile.self.cycles-pp.tick_nohz_irq_exit
      0.17 ± 15%      -0.1        0.08 ± 58%  perf-profile.self.cycles-pp.__remove_hrtimer
      0.20 ±  4%      -0.1        0.12 ± 45%  perf-profile.self.cycles-pp.tick_irq_enter
      0.13 ± 23%      -0.1        0.05 ± 90%  perf-profile.self.cycles-pp.can_stop_idle_tick
      0.14 ± 35%      -0.1        0.07 ± 69%  perf-profile.self.cycles-pp.update_rt_rq_load_avg
      0.12 ± 16%      -0.1        0.07 ± 20%  perf-profile.self.cycles-pp.tick_sched_timer
      0.12 ± 21%      -0.1        0.06 ± 84%  perf-profile.self.cycles-pp.task_tick_idle


                                                                                
             phoronix-test-suite.darktable.Masskrug.CPU-only.seconds            
                                                                                
  16 +----------------------------------------------------------------------+   
     |    O    O    O    O    O    O     O    O    O    O    O    O    O    |   
  14 |-+  +    +    +    +    +    +     +    +    +    +    +    +         |   
  12 |-+  :    :    :    :    :    :     :    :    :    :    :    :         |   
     |:   ::   ::   ::   ::   ::   ::    :    :    :    :    :    :         |   
  10 |:+  ::   ::   ::   ::   ::   ::   : :  : :  : :  : :  : :  :          |   
     |:  : :  : :  : :  : :  : :  : :   : :  : :  : :  : :  : :  :          |   
   8 |:+ : :  : :  : :  : :  : :  : :   : :  : :  : :  : :  : :  :          |   
     | : :  : :  : :  : :  : :  : :  :  : :  : :  : :  : :  : :  :          |   
   6 |-: :  : :  : :  : :  : :  : :  : :  : :  : :  : :  : :  : :           |   
   4 |-: :  : :  : :  : :  : :  : :  : :  : :  : :  : :  : :  : :           |   
     | : :  : :  : :  : :  : :  : :  : :   ::   ::   ::   ::   ::           |   
   2 |-+:    :    :    :    :    :    ::   ::   ::   ::   ::   ::           |   
     |  :    :    :    :    :    :    :    :    :    :    :    :            |   
   0 +----------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                        phoronix-test-suite.time.user_time                      
                                                                                
  600 +---------------------------------------------------------------------+   
      |                                                                     |   
  500 |-+  O    O    O    O    O    O    O    O    O    O    O    O    O    |   
      |    +    +    +    +    +    +    +    +    +    +    +    +         |   
      |    :    :    :    :    :    :    :    :    :    :    :    :         |   
  400 |:+  ::   ::   ::   ::   ::   ::   ::   ::   ::   ::   ::   :         |   
      |:  : :  : :  : :  : :  : :  : :  : :  : :  : :  : :  : :  :          |   
  300 |:+ : :  : :  : :  : :  : :  : :  : :  : :  : :  : :  : :  :          |   
      |:  : :  : :  : :  : :  : :  : :  : :  : :  : :  : :  : :  :          |   
  200 |-: :  : :  : :  : :  : :  : :  : :  : :  : :  : :  : :  : :          |   
      | : :  : :  : :  : :  : :  : :  : :  : :  : :  : :  : :  : :          |   
      | : :  : :  : :  : :  : :  : :  : :  : :  : :  : :  : :  : :          |   
  100 |-::   ::   ::   ::   ::   ::   ::   ::   ::   ::   ::   ::           |   
      |  :    :    :    :    :    :    :    :    :    :    :    :           |   
    0 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                      phoronix-test-suite.time.system_time                      
                                                                                
  40 +----------------------------------------------------------------------+   
     |    O    O    O    O    O    O     O    O    O    O    O    O    O    |   
  35 |-+                                                                    |   
  30 |-+  +    +    +    +    +    +     +    +    +    +    +    +         |   
     |    :    :    :    :    :    :     :    :    :    :    :    :         |   
  25 |:+  ::   ::   ::   ::   ::   ::   ::   ::   ::   ::   ::   :          |   
     |:  : :  : :  : :  : :  : :  : :   : :  : :  : :  : :  : :  :          |   
  20 |:+ : :  : :  : :  : :  : :  : :   : :  : :  : :  : :  : :  :          |   
     |:  : :  : :  : :  : :  : :  : :   : :  : :  : :  : :  : :  :          |   
  15 |-: :  : :  : :  : :  : :  : :  : :  : :  : :  : :  : :  : :           |   
  10 |-: :  : :  : :  : :  : :  : :  : :  : :  : :  : :  : :  : :           |   
     | : :  : :  : :  : :  : :  : :  : :  : :  : :  : :  : :  : :           |   
   5 |-::   ::   ::   ::   ::   ::   : :   ::   ::   ::   ::   ::           |   
     |  :    :    :    :    :    :    :    :    :    :    :    :            |   
   0 +----------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
               phoronix-test-suite.time.percent_of_cpu_this_job_got             
                                                                                
  250 +---------------------------------------------------------------------+   
      |    +    O    +    +    +    +    +    +    +    +         +         |   
      |    :    +    :    :    :    :    :    :    :    :         :         |   
  200 |-+  :    :    :    :    :    :    :    :    :    :    +    :         |   
      |:   ::   ::   ::   ::   ::   ::   ::   ::   ::   ::   :    :         |   
      |:  : :   ::  : :  : :  : :  : :  : :  : :  : :  : :   ::  :          |   
  150 |:+ : :  : :  : :  : :  : :  : :  : :  : :  : :  : :  : :  :          |   
      |:  : :  : :  : :  : :  : :  : :  : :  : :  : :  : :  : :  :          |   
  100 |-: :  : :  : :  : :  : :  : :  : :  : :  : :  : :  : : :  :          |   
      | : :  : :  : :  : :  : :  : :  : :  : :  : :  : :  : :  : :          |   
      | : :  : :  : :  : :  : :  : :  : :  : :  : :  : :  : :  : :          |   
   50 |-::   : :  ::   ::   ::   ::   ::   ::   ::   ::   : :  ::           |   
      |  :    :    :    :    :    :    :    :    :    :    :   ::           |   
      |  :    :    :    :    :    :    :    :    :    :    :    :           |   
    0 +---------------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample

***************************************************************************************************
lkp-cfl-e1: 16 threads Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz with 32G memory
=========================================================================================
compiler/cpufreq_governor/ipc/kconfig/mode/nr_threads/rootfs/tbox_group/testcase/ucode:
  gcc-7/performance/socket/x86_64-rhel-7.2-clear_lck_7595/process/100%/clear-ota-25590-x86_64-2018-10-18.cgz/lkp-cfl-e1/hackbench/0xb8

commit: 
  ba5c5e4a5d ("arm64: move jump_label_init() before parse_early_param()")
  6471384af2 ("mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options")

ba5c5e4a5da443e8 6471384af2a6530696fc0203baf 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
     90709            -4.9%      86276        hackbench.throughput
    108209           +42.0%     153632        hackbench.time.involuntary_context_switches
    825777            -3.9%     793179        hackbench.time.minor_page_faults
    712.02            -3.9%     684.38        hackbench.time.user_time
 5.616e+08            -4.3%  5.376e+08        hackbench.workload
      0.84            +0.1        0.95 ±  2%  mpstat.cpu.-1.irq%
      1347 ±  3%     +12.7%       1518 ±  2%  slabinfo.kmalloc-rcl-64.num_objs
      3.49            +4.7%       3.65        turbostat.RAMWatt
    396367            -1.3%     391171        vmstat.system.cs
  14755342            -3.8%   14194624        proc-vmstat.numa_hit
  14755342            -3.8%   14194624        proc-vmstat.numa_local
  14815709            -3.7%   14261066        proc-vmstat.pgalloc_normal
  14779482            -3.8%   14224922        proc-vmstat.pgfree
      8627 ±172%    +682.5%      67514 ± 65%  sched_debug.cfs_rq:/.MIN_vruntime.avg
    137821 ±172%    +585.5%     944775 ± 59%  sched_debug.cfs_rq:/.MIN_vruntime.max
     33372 ±172%    +620.8%     240555 ± 61%  sched_debug.cfs_rq:/.MIN_vruntime.stddev
      8627 ±172%    +682.5%      67514 ± 65%  sched_debug.cfs_rq:/.max_vruntime.avg
    137821 ±172%    +585.5%     944775 ± 59%  sched_debug.cfs_rq:/.max_vruntime.max
     33372 ±172%    +620.8%     240555 ± 61%  sched_debug.cfs_rq:/.max_vruntime.stddev
    556.14 ±  4%      -9.3%     504.21 ±  5%  sched_debug.cpu.clock_task.stddev
      1883 ± 36%     -48.0%     979.75 ± 26%  interrupts.133:IR-PCI-MSI.2097153-edge.eth1-TxRx-0
     46373 ±  4%     -10.1%      41692 ±  4%  interrupts.CPU0.RES:Rescheduling_interrupts
      1883 ± 36%     -48.0%     979.75 ± 26%  interrupts.CPU1.133:IR-PCI-MSI.2097153-edge.eth1-TxRx-0
     45230 ±  5%     -11.7%      39943 ±  4%  interrupts.CPU1.RES:Rescheduling_interrupts
     57716 ±  3%     -12.3%      50601        interrupts.CPU12.RES:Rescheduling_interrupts
     53566 ±  3%     -11.5%      47399        interrupts.CPU13.RES:Rescheduling_interrupts
     52007 ±  4%     -14.0%      44710 ±  5%  interrupts.CPU15.RES:Rescheduling_interrupts
     46887 ±  3%     -12.3%      41140 ±  6%  interrupts.CPU2.RES:Rescheduling_interrupts
     52269 ±  5%     -12.1%      45958 ±  5%  interrupts.CPU3.RES:Rescheduling_interrupts
     57120 ±  7%     -12.0%      50281 ±  4%  interrupts.CPU4.RES:Rescheduling_interrupts
     49253 ±  3%     -11.5%      43586 ±  2%  interrupts.CPU7.RES:Rescheduling_interrupts
    820390           -10.2%     736882        interrupts.RES:Rescheduling_interrupts
     44.93            -2.4%      43.85        perf-stat.i.MPKI
 9.635e+09            -3.5%    9.3e+09        perf-stat.i.branch-instructions
 1.106e+08            -4.2%   1.06e+08        perf-stat.i.branch-misses
     13.44            +0.3       13.72        perf-stat.i.cache-miss-rate%
 2.912e+08            -4.3%  2.786e+08        perf-stat.i.cache-misses
 2.171e+09            -6.3%  2.034e+09        perf-stat.i.cache-references
    397614            -1.4%     391925        perf-stat.i.context-switches
      1.26            +4.2%       1.31        perf-stat.i.cpi
     62650            -3.9%      60222        perf-stat.i.cpu-migrations
    209.19            +4.6%     218.81        perf-stat.i.cycles-between-cache-misses
 1.459e+10            -4.0%  1.401e+10        perf-stat.i.dTLB-loads
 1.025e+10            -4.2%  9.817e+09        perf-stat.i.dTLB-stores
  86435026            -2.5%   84314445        perf-stat.i.iTLB-load-misses
 4.833e+10            -4.0%  4.639e+10        perf-stat.i.instructions
    562.26            -1.5%     553.60        perf-stat.i.instructions-per-iTLB-miss
      0.79            -4.0%       0.76        perf-stat.i.ipc
      0.00 ± 35%      +0.0        0.00 ± 49%  perf-stat.i.node-load-miss-rate%
      0.64 ± 41%     +40.2%       0.90 ± 32%  perf-stat.i.node-load-misses
  21652012            -2.2%   21184023        perf-stat.i.node-loads
      0.57 ±  7%     +76.9%       1.00 ± 33%  perf-stat.i.node-store-misses
  26105714           +79.9%   46970933        perf-stat.i.node-stores
     44.93            -2.4%      43.85        perf-stat.overall.MPKI
     13.41            +0.3       13.69        perf-stat.overall.cache-miss-rate%
      1.26            +4.2%       1.31        perf-stat.overall.cpi
    208.98            +4.6%     218.59        perf-stat.overall.cycles-between-cache-misses
    559.11            -1.6%     550.27        perf-stat.overall.instructions-per-iTLB-miss
      0.79            -4.0%       0.76        perf-stat.overall.ipc
      0.00 ± 42%      +0.0        0.00 ± 33%  perf-stat.overall.node-load-miss-rate%
 9.619e+09            -3.5%  9.285e+09        perf-stat.ps.branch-instructions
 1.104e+08            -4.2%  1.058e+08        perf-stat.ps.branch-misses
 2.907e+08            -4.3%  2.781e+08        perf-stat.ps.cache-misses
 2.168e+09            -6.3%  2.031e+09        perf-stat.ps.cache-references
    396957            -1.4%     391281        perf-stat.ps.context-switches
     62546            -3.9%      60123        perf-stat.ps.cpu-migrations
 1.457e+10            -4.0%  1.399e+10        perf-stat.ps.dTLB-loads
 1.023e+10            -4.2%  9.801e+09        perf-stat.ps.dTLB-stores
  86292583            -2.5%   84176193        perf-stat.ps.iTLB-load-misses
 4.825e+10            -4.0%  4.632e+10        perf-stat.ps.instructions
      0.64 ± 41%     +40.2%       0.90 ± 32%  perf-stat.ps.node-load-misses
  21616330            -2.2%   21149342        perf-stat.ps.node-loads
      0.56 ±  7%     +76.9%       1.00 ± 33%  perf-stat.ps.node-store-misses
  26062702           +79.9%   46894016        perf-stat.ps.node-stores
 2.929e+13            -3.5%  2.827e+13        perf-stat.total.instructions
     39.46            -1.3       38.13        perf-profile.calltrace.cycles-pp.__x64_sys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
     39.37            -1.3       38.05        perf-profile.calltrace.cycles-pp.ksys_read.__x64_sys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
     38.18            -1.3       36.91        perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.__x64_sys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
     35.37            -1.2       34.19        perf-profile.calltrace.cycles-pp.__vfs_read.vfs_read.ksys_read.__x64_sys_read.do_syscall_64
     34.98            -1.2       33.81        perf-profile.calltrace.cycles-pp.new_sync_read.__vfs_read.vfs_read.ksys_read.__x64_sys_read
     34.12            -1.1       33.00        perf-profile.calltrace.cycles-pp.sock_read_iter.new_sync_read.__vfs_read.vfs_read.ksys_read
     32.88            -1.0       31.84        perf-profile.calltrace.cycles-pp.sock_recvmsg.sock_read_iter.new_sync_read.__vfs_read.vfs_read
     32.46            -1.0       31.43        perf-profile.calltrace.cycles-pp.unix_stream_recvmsg.sock_recvmsg.sock_read_iter.new_sync_read.__vfs_read
     32.04            -1.0       31.03        perf-profile.calltrace.cycles-pp.unix_stream_read_generic.unix_stream_recvmsg.sock_recvmsg.sock_read_iter.new_sync_read
      2.70 ±  2%      -0.8        1.93        perf-profile.calltrace.cycles-pp._raw_spin_lock.unix_stream_sendmsg.sock_sendmsg.sock_write_iter.new_sync_write
      2.16 ±  2%      -0.8        1.41        perf-profile.calltrace.cycles-pp.refcount_add_checked.skb_set_owner_w.sock_alloc_send_pskb.unix_stream_sendmsg.sock_sendmsg
      2.70            -0.7        1.95        perf-profile.calltrace.cycles-pp.skb_set_owner_w.sock_alloc_send_pskb.unix_stream_sendmsg.sock_sendmsg.sock_write_iter
      2.16 ±  2%      -0.7        1.41        perf-profile.calltrace.cycles-pp.refcount_add_not_zero_checked.refcount_add_checked.skb_set_owner_w.sock_alloc_send_pskb.unix_stream_sendmsg
     11.94            -0.4       11.50        perf-profile.calltrace.cycles-pp.consume_skb.unix_stream_read_generic.unix_stream_recvmsg.sock_recvmsg.sock_read_iter
      9.59            -0.4        9.18        perf-profile.calltrace.cycles-pp.unix_stream_read_actor.unix_stream_read_generic.unix_stream_recvmsg.sock_recvmsg.sock_read_iter
      9.49            -0.4        9.09        perf-profile.calltrace.cycles-pp.skb_copy_datagram_iter.unix_stream_read_actor.unix_stream_read_generic.unix_stream_recvmsg.sock_recvmsg
      9.20            -0.4        8.85        perf-profile.calltrace.cycles-pp.__skb_datagram_iter.skb_copy_datagram_iter.unix_stream_read_actor.unix_stream_read_generic.unix_stream_recvmsg
      8.85            -0.3        8.54        perf-profile.calltrace.cycles-pp.simple_copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.unix_stream_read_actor.unix_stream_read_generic
      6.51            -0.3        6.21        perf-profile.calltrace.cycles-pp._copy_to_iter.simple_copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.unix_stream_read_actor
      7.29            -0.3        7.00        perf-profile.calltrace.cycles-pp.skb_release_all.consume_skb.unix_stream_read_generic.unix_stream_recvmsg.sock_recvmsg
      5.44            -0.2        5.20        perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string._copy_to_iter.simple_copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter
      3.99            -0.2        3.79        perf-profile.calltrace.cycles-pp.syscall_slow_exit_work.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.44 ±  2%      -0.2        2.24 ±  4%  perf-profile.calltrace.cycles-pp.___cache_free.kfree.skb_free_head.skb_release_data.skb_release_all
      3.23            -0.2        3.05        perf-profile.calltrace.cycles-pp.skb_release_head_state.skb_release_all.consume_skb.unix_stream_read_generic.unix_stream_recvmsg
      3.28            -0.2        3.10        perf-profile.calltrace.cycles-pp.__audit_syscall_exit.syscall_slow_exit_work.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.74            -0.2        2.56        perf-profile.calltrace.cycles-pp.skb_queue_tail.unix_stream_sendmsg.sock_sendmsg.sock_write_iter.new_sync_write
      3.05            -0.2        2.88        perf-profile.calltrace.cycles-pp.unix_destruct_scm.skb_release_head_state.skb_release_all.consume_skb.unix_stream_read_generic
      1.53 ±  3%      -0.2        1.36        perf-profile.calltrace.cycles-pp.syscall_return_via_sysret
      4.44            -0.2        4.27        perf-profile.calltrace.cycles-pp.skb_copy_datagram_from_iter.unix_stream_sendmsg.sock_sendmsg.sock_write_iter.new_sync_write
      2.62            -0.2        2.46        perf-profile.calltrace.cycles-pp.sock_wfree.unix_destruct_scm.skb_release_head_state.skb_release_all.consume_skb
      1.95            -0.2        1.79        perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.skb_queue_tail.unix_stream_sendmsg.sock_sendmsg.sock_write_iter
      1.11            -0.1        0.97 ±  3%  perf-profile.calltrace.cycles-pp._raw_spin_lock.___cache_free.kfree.skb_free_head.skb_release_data
      1.53 ±  2%      -0.1        1.39 ±  2%  perf-profile.calltrace.cycles-pp.cache_alloc_refill.kmem_cache_alloc_node_trace.__kmalloc_node_track_caller.__kmalloc_reserve.__alloc_skb
      1.02 ±  2%      -0.1        0.89 ±  3%  perf-profile.calltrace.cycles-pp.queued_spin_lock_slowpath._raw_spin_lock.___cache_free.kfree.skb_free_head
      3.69            -0.1        3.56        perf-profile.calltrace.cycles-pp.kmem_cache_free.kfree_skbmem.consume_skb.unix_stream_read_generic.unix_stream_recvmsg
      3.83            -0.1        3.70        perf-profile.calltrace.cycles-pp.kfree_skbmem.consume_skb.unix_stream_read_generic.unix_stream_recvmsg.sock_recvmsg
      1.60            -0.1        1.47 ±  2%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64
      1.85            -0.1        1.73        perf-profile.calltrace.cycles-pp.syscall_trace_enter.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.60            -0.1        1.48 ±  2%  perf-profile.calltrace.cycles-pp.skb_unlink.unix_stream_read_generic.unix_stream_recvmsg.sock_recvmsg.sock_read_iter
      2.70            -0.1        2.59        perf-profile.calltrace.cycles-pp._copy_from_iter.skb_copy_datagram_from_iter.unix_stream_sendmsg.sock_sendmsg.sock_write_iter
      0.69 ±  3%      -0.1        0.58        perf-profile.calltrace.cycles-pp.queued_spin_lock_slowpath._raw_spin_lock.cache_alloc_refill.kmem_cache_alloc_node_trace.__kmalloc_node_track_caller
      1.77            -0.1        1.66        perf-profile.calltrace.cycles-pp.___cache_free.kmem_cache_free.kfree_skbmem.consume_skb.unix_stream_read_generic
      0.74 ±  2%      -0.1        0.64        perf-profile.calltrace.cycles-pp._raw_spin_lock.cache_alloc_refill.kmem_cache_alloc_node_trace.__kmalloc_node_track_caller.__kmalloc_reserve
      1.82            -0.1        1.72        perf-profile.calltrace.cycles-pp.__fget_light.__fdget_pos.ksys_write.__x64_sys_write.do_syscall_64
      1.90            -0.1        1.81        perf-profile.calltrace.cycles-pp.__fdget_pos.ksys_write.__x64_sys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.98 ±  2%      -0.1        0.89 ±  3%  perf-profile.calltrace.cycles-pp.unroll_tree_refs.__audit_syscall_exit.syscall_slow_exit_work.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.05            -0.1        0.99 ±  3%  perf-profile.calltrace.cycles-pp.unix_write_space.sock_wfree.unix_destruct_scm.skb_release_head_state.skb_release_all
      0.97 ±  2%      -0.1        0.92 ±  2%  perf-profile.calltrace.cycles-pp.__audit_syscall_entry.syscall_trace_enter.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.65            -0.1        0.59 ±  3%  perf-profile.calltrace.cycles-pp._raw_spin_lock.___cache_free.kmem_cache_free.kfree_skbmem.consume_skb
      0.60            -0.1        0.55 ±  3%  perf-profile.calltrace.cycles-pp.queued_spin_lock_slowpath._raw_spin_lock.___cache_free.kmem_cache_free.kfree_skbmem
      0.70            -0.1        0.64        perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.skb_unlink.unix_stream_read_generic.unix_stream_recvmsg.sock_recvmsg
      0.86            -0.1        0.81        perf-profile.calltrace.cycles-pp.free_block.___cache_free.kmem_cache_free.kfree_skbmem.consume_skb
      2.03            -0.1        1.98        perf-profile.calltrace.cycles-pp.schedule_timeout.unix_stream_read_generic.unix_stream_recvmsg.sock_recvmsg.sock_read_iter
      0.66 ±  2%      -0.1        0.60        perf-profile.calltrace.cycles-pp.refcount_inc_not_zero_checked.refcount_inc_checked.unix_stream_read_generic.unix_stream_recvmsg.sock_recvmsg
      1.98            -0.0        1.94        perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.unix_stream_read_generic.unix_stream_recvmsg.sock_recvmsg
      0.69 ±  3%      -0.0        0.64 ±  2%  perf-profile.calltrace.cycles-pp.refcount_inc_checked.unix_stream_read_generic.unix_stream_recvmsg.sock_recvmsg.sock_read_iter
      1.94            -0.0        1.89        perf-profile.calltrace.cycles-pp.__sched_text_start.schedule.schedule_timeout.unix_stream_read_generic.unix_stream_recvmsg
      0.75 ±  3%      -0.0        0.71        perf-profile.calltrace.cycles-pp.deactivate_task.__sched_text_start.schedule.schedule_timeout.unix_stream_read_generic
      0.70 ±  2%      -0.0        0.67        perf-profile.calltrace.cycles-pp.cache_alloc_refill.kmem_cache_alloc_node.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb
      1.86 ±  2%      +0.2        2.08        perf-profile.calltrace.cycles-pp.refcount_inc_not_zero_checked.unix_stream_read_generic.unix_stream_recvmsg.sock_recvmsg.sock_read_iter
     92.65            +0.4       93.01        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
     90.21            +0.6       90.79        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +1.0        1.00 ±  4%  perf-profile.calltrace.cycles-pp.memset_erms.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb.unix_stream_sendmsg
      0.00            +1.1        1.06        perf-profile.calltrace.cycles-pp.memset_erms.__kmalloc_node_track_caller.__kmalloc_reserve.__alloc_skb.alloc_skb_with_frags
      4.41            +1.7        6.13        perf-profile.calltrace.cycles-pp.kmem_cache_alloc_node_trace.__kmalloc_node_track_caller.__kmalloc_reserve.__alloc_skb.alloc_skb_with_frags
     42.73            +2.2       44.95        perf-profile.calltrace.cycles-pp.__x64_sys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     42.64            +2.2       44.86        perf-profile.calltrace.cycles-pp.ksys_write.__x64_sys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     40.20            +2.4       42.55        perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.__x64_sys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     37.99            +2.4       40.41        perf-profile.calltrace.cycles-pp.__vfs_write.vfs_write.ksys_write.__x64_sys_write.do_syscall_64
     37.73            +2.4       40.18        perf-profile.calltrace.cycles-pp.new_sync_write.__vfs_write.vfs_write.ksys_write.__x64_sys_write
     36.93            +2.5       39.41        perf-profile.calltrace.cycles-pp.sock_write_iter.new_sync_write.__vfs_write.vfs_write.ksys_write
     35.68            +2.5       38.19        perf-profile.calltrace.cycles-pp.sock_sendmsg.sock_write_iter.new_sync_write.__vfs_write.vfs_write
     34.85            +2.5       37.36        perf-profile.calltrace.cycles-pp.unix_stream_sendmsg.sock_sendmsg.sock_write_iter.new_sync_write.__vfs_write
      4.99            +2.8        7.79        perf-profile.calltrace.cycles-pp.__kmalloc_node_track_caller.__kmalloc_reserve.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb
      5.22            +3.0        8.21        perf-profile.calltrace.cycles-pp.__kmalloc_reserve.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb.unix_stream_sendmsg
     16.62            +4.0       20.60        perf-profile.calltrace.cycles-pp.sock_alloc_send_pskb.unix_stream_sendmsg.sock_sendmsg.sock_write_iter.new_sync_write
     12.22            +4.7       16.95        perf-profile.calltrace.cycles-pp.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb.unix_stream_sendmsg.sock_sendmsg
     12.55            +4.7       17.29        perf-profile.calltrace.cycles-pp.alloc_skb_with_frags.sock_alloc_send_pskb.unix_stream_sendmsg.sock_sendmsg.sock_write_iter
     39.76            -1.3       38.41        perf-profile.children.cycles-pp.__x64_sys_read
     39.48            -1.3       38.16        perf-profile.children.cycles-pp.ksys_read
     38.25            -1.3       36.99        perf-profile.children.cycles-pp.vfs_read
     35.48            -1.2       34.28        perf-profile.children.cycles-pp.__vfs_read
     35.06            -1.2       33.89        perf-profile.children.cycles-pp.new_sync_read
     34.37            -1.1       33.23        perf-profile.children.cycles-pp.sock_read_iter
      6.75            -1.1        5.62        perf-profile.children.cycles-pp._raw_spin_lock
     32.74            -1.0       31.70        perf-profile.children.cycles-pp.unix_stream_recvmsg
     33.00            -1.0       31.95        perf-profile.children.cycles-pp.sock_recvmsg
     32.16            -1.0       31.14        perf-profile.children.cycles-pp.unix_stream_read_generic
      2.74            -0.8        1.98        perf-profile.children.cycles-pp.skb_set_owner_w
      2.21 ±  2%      -0.7        1.46        perf-profile.children.cycles-pp.refcount_add_checked
      2.33            -0.7        1.59        perf-profile.children.cycles-pp.refcount_add_not_zero_checked
     12.06            -0.4       11.61        perf-profile.children.cycles-pp.consume_skb
      9.63            -0.4        9.22        perf-profile.children.cycles-pp.unix_stream_read_actor
      9.56            -0.4        9.15        perf-profile.children.cycles-pp.skb_copy_datagram_iter
      9.26            -0.4        8.90        perf-profile.children.cycles-pp.__skb_datagram_iter
      8.99            -0.4        8.63        perf-profile.children.cycles-pp.simple_copy_to_iter
      4.35            -0.4        3.99        perf-profile.children.cycles-pp.queued_spin_lock_slowpath
      7.37            -0.3        7.06        perf-profile.children.cycles-pp.skb_release_all
      4.26 ±  2%      -0.3        3.95        perf-profile.children.cycles-pp.___cache_free
      6.56            -0.3        6.26        perf-profile.children.cycles-pp._copy_to_iter
      7.13            -0.3        6.83        perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      4.06            -0.3        3.79        perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      3.06            -0.3        2.80        perf-profile.children.cycles-pp.syscall_return_via_sysret
      4.12            -0.2        3.91        perf-profile.children.cycles-pp.syscall_slow_exit_work
      2.78            -0.2        2.59        perf-profile.children.cycles-pp.skb_queue_tail
      2.46 ±  2%      -0.2        2.27        perf-profile.children.cycles-pp.__might_sleep
      3.28            -0.2        3.10        perf-profile.children.cycles-pp.skb_release_head_state
      2.74            -0.2        2.57        perf-profile.children.cycles-pp.sock_wfree
      3.31            -0.2        3.14 ±  2%  perf-profile.children.cycles-pp.unix_destruct_scm
      4.49            -0.2        4.32        perf-profile.children.cycles-pp.skb_copy_datagram_from_iter
      3.48            -0.2        3.31        perf-profile.children.cycles-pp.__audit_syscall_exit
      2.24            -0.2        2.08        perf-profile.children.cycles-pp.cache_alloc_refill
      3.79            -0.2        3.64        perf-profile.children.cycles-pp.kmem_cache_free
      3.90            -0.1        3.77        perf-profile.children.cycles-pp.kfree_skbmem
      1.62            -0.1        1.49        perf-profile.children.cycles-pp.entry_SYSCALL_64
      2.77            -0.1        2.65        perf-profile.children.cycles-pp._copy_from_iter
      1.94            -0.1        1.82        perf-profile.children.cycles-pp.syscall_trace_enter
      2.52            -0.1        2.40        perf-profile.children.cycles-pp.__fget_light
      3.14 ±  2%      -0.1        3.02        perf-profile.children.cycles-pp.__check_object_size
      1.58 ±  3%      -0.1        1.46        perf-profile.children.cycles-pp.___might_sleep
      1.61            -0.1        1.50 ±  2%  perf-profile.children.cycles-pp.skb_unlink
      2.67            -0.1        2.56        perf-profile.children.cycles-pp.__fdget_pos
      1.77 ±  2%      -0.1        1.67        perf-profile.children.cycles-pp.free_block
      1.10 ±  3%      -0.1        1.01 ±  3%  perf-profile.children.cycles-pp.__audit_syscall_entry
      0.57 ±  5%      -0.1        0.48 ±  9%  perf-profile.children.cycles-pp.memcg_kmem_put_cache
      1.08 ±  2%      -0.1        1.00 ±  2%  perf-profile.children.cycles-pp.unroll_tree_refs
      0.93 ±  4%      -0.1        0.84        perf-profile.children.cycles-pp.__might_fault
      0.18 ±  7%      -0.1        0.11 ±  4%  perf-profile.children.cycles-pp.should_failslab
      0.56 ±  6%      -0.1        0.50        perf-profile.children.cycles-pp.rcu_all_qs
      2.70            -0.1        2.64        perf-profile.children.cycles-pp.schedule
      2.75            -0.1        2.69        perf-profile.children.cycles-pp.schedule_timeout
      2.83            -0.1        2.78        perf-profile.children.cycles-pp.__sched_text_start
      1.07            -0.1        1.02        perf-profile.children.cycles-pp.__list_del_entry_valid
      0.70 ±  3%      -0.1        0.65 ±  2%  perf-profile.children.cycles-pp.refcount_inc_checked
      0.30            -0.0        0.25 ±  6%  perf-profile.children.cycles-pp.copyin
      0.67            -0.0        0.62        perf-profile.children.cycles-pp._cond_resched
      1.19            -0.0        1.15 ±  2%  perf-profile.children.cycles-pp.security_file_permission
      0.74 ±  2%      -0.0        0.70 ±  2%  perf-profile.children.cycles-pp.wait_for_unix_gc
      1.14            -0.0        1.10 ±  2%  perf-profile.children.cycles-pp.unix_write_space
      0.32 ±  2%      -0.0        0.29 ±  3%  perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
      0.98            -0.0        0.94        perf-profile.children.cycles-pp.deactivate_task
      0.22            -0.0        0.20 ±  5%  perf-profile.children.cycles-pp.update_rq_clock
      0.73            -0.0        0.71 ±  2%  perf-profile.children.cycles-pp.dequeue_task_fair
      0.16 ±  2%      -0.0        0.15        perf-profile.children.cycles-pp.set_next_entity
      0.08 ±  8%      +0.0        0.10 ±  4%  perf-profile.children.cycles-pp.cache_grow_begin
      0.09 ±  4%      +0.0        0.12 ± 11%  perf-profile.children.cycles-pp.maybe_add_creds
      0.00            +0.1        0.08 ±  6%  perf-profile.children.cycles-pp.memset
      0.97 ±  5%      +0.2        1.13 ±  3%  perf-profile.children.cycles-pp.__virt_addr_valid
      2.52 ±  2%      +0.2        2.69        perf-profile.children.cycles-pp.refcount_inc_not_zero_checked
     92.69            +0.4       93.05        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     90.39            +0.6       90.97        perf-profile.children.cycles-pp.do_syscall_64
      4.64            +1.8        6.49        perf-profile.children.cycles-pp.kmem_cache_alloc_node_trace
      0.00            +2.1        2.11 ±  2%  perf-profile.children.cycles-pp.memset_erms
     43.03            +2.2       45.23        perf-profile.children.cycles-pp.__x64_sys_write
     42.76            +2.2       44.96        perf-profile.children.cycles-pp.ksys_write
     40.25            +2.3       42.59        perf-profile.children.cycles-pp.vfs_write
     38.06            +2.4       40.48        perf-profile.children.cycles-pp.__vfs_write
     37.80            +2.4       40.24        perf-profile.children.cycles-pp.new_sync_write
     37.11            +2.5       39.57        perf-profile.children.cycles-pp.sock_write_iter
     35.75            +2.5       38.24        perf-profile.children.cycles-pp.sock_sendmsg
     35.18            +2.5       37.70        perf-profile.children.cycles-pp.unix_stream_sendmsg
      5.05            +2.8        7.89        perf-profile.children.cycles-pp.__kmalloc_node_track_caller
      5.26            +3.0        8.30        perf-profile.children.cycles-pp.__kmalloc_reserve
     16.66            +4.0       20.62        perf-profile.children.cycles-pp.sock_alloc_send_pskb
     12.30            +4.7       17.02        perf-profile.children.cycles-pp.__alloc_skb
     12.59            +4.7       17.33        perf-profile.children.cycles-pp.alloc_skb_with_frags
      3.43 ±  2%      -0.8        2.61        perf-profile.self.cycles-pp._raw_spin_lock
      2.31            -0.7        1.57        perf-profile.self.cycles-pp.refcount_add_not_zero_checked
      4.33            -0.3        3.98        perf-profile.self.cycles-pp.queued_spin_lock_slowpath
      7.10            -0.3        6.80        perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      3.03            -0.3        2.77        perf-profile.self.cycles-pp.syscall_return_via_sysret
      1.87            -0.2        1.62 ±  2%  perf-profile.self.cycles-pp.__check_object_size
      3.01            -0.2        2.80        perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      1.76            -0.2        1.59        perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      1.37 ±  2%      -0.1        1.24 ±  2%  perf-profile.self.cycles-pp.sock_def_readable
      1.62            -0.1        1.49        perf-profile.self.cycles-pp.entry_SYSCALL_64
      2.46            -0.1        2.34        perf-profile.self.cycles-pp.__fget_light
      1.80            -0.1        1.69        perf-profile.self.cycles-pp.unix_stream_read_generic
      1.49 ±  2%      -0.1        1.38 ±  2%  perf-profile.self.cycles-pp.___might_sleep
      0.90 ±  3%      -0.1        0.81 ±  2%  perf-profile.self.cycles-pp.__audit_syscall_entry
      1.07 ±  2%      -0.1        0.98 ±  3%  perf-profile.self.cycles-pp.unroll_tree_refs
      1.14 ±  2%      -0.1        1.06        perf-profile.self.cycles-pp.sock_read_iter
      0.66 ±  3%      -0.1        0.58 ±  2%  perf-profile.self.cycles-pp.sock_wfree
      1.04 ±  3%      -0.1        0.98        perf-profile.self.cycles-pp.__might_sleep
      1.04            -0.1        0.98        perf-profile.self.cycles-pp.__list_del_entry_valid
      1.86            -0.1        1.80        perf-profile.self.cycles-pp.__audit_syscall_exit
      0.46 ±  8%      -0.1        0.40 ±  3%  perf-profile.self.cycles-pp.rcu_all_qs
      0.98 ±  2%      -0.1        0.93 ±  4%  perf-profile.self.cycles-pp.free_block
      0.74 ±  2%      -0.1        0.69 ±  2%  perf-profile.self.cycles-pp.vfs_read
      0.58            -0.0        0.54 ±  2%  perf-profile.self.cycles-pp.syscall_slow_exit_work
      0.11 ±  9%      -0.0        0.07 ±  5%  perf-profile.self.cycles-pp.should_failslab
      0.74            -0.0        0.70        perf-profile.self.cycles-pp.new_sync_read
      0.37 ±  3%      -0.0        0.34 ±  2%  perf-profile.self.cycles-pp._cond_resched
      0.64            -0.0        0.60        perf-profile.self.cycles-pp._copy_from_iter
      0.18 ±  2%      -0.0        0.15 ±  5%  perf-profile.self.cycles-pp.copyin
      0.30 ±  5%      -0.0        0.27 ±  5%  perf-profile.self.cycles-pp.__skb_datagram_iter
      0.25 ±  4%      -0.0        0.23 ±  6%  perf-profile.self.cycles-pp.simple_copy_to_iter
      0.23            -0.0        0.21 ±  5%  perf-profile.self.cycles-pp.sock_sendmsg
      0.14 ±  5%      -0.0        0.12        perf-profile.self.cycles-pp.__list_add_valid
      0.18            -0.0        0.16 ±  4%  perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
      0.12 ±  4%      -0.0        0.11 ±  6%  perf-profile.self.cycles-pp.security_socket_sendmsg
      0.09 ±  4%      +0.0        0.11 ±  9%  perf-profile.self.cycles-pp.maybe_add_creds
      0.11 ± 21%      +0.0        0.15 ± 24%  perf-profile.self.cycles-pp.kmalloc_slab
      0.19 ±  5%      +0.1        0.26 ± 12%  perf-profile.self.cycles-pp.__kmalloc_reserve
      1.11 ±  2%      +0.1        1.19        perf-profile.self.cycles-pp.kfree
      0.13 ± 11%      +0.1        0.22 ±  8%  perf-profile.self.cycles-pp.__kmalloc_node_track_caller
      0.94 ±  5%      +0.1        1.09 ±  3%  perf-profile.self.cycles-pp.__virt_addr_valid
      2.51 ±  2%      +0.2        2.68        perf-profile.self.cycles-pp.refcount_inc_not_zero_checked
      1.66 ±  5%      +0.8        2.42        perf-profile.self.cycles-pp.__alloc_skb
      1.26            +1.9        3.15 ±  2%  perf-profile.self.cycles-pp.kmem_cache_alloc_node_trace
      0.00            +2.1        2.07 ±  2%  perf-profile.self.cycles-pp.memset_erms



***************************************************************************************************
lkp-cfl-e1: 16 threads Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz with 32G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_threads/rootfs/tbox_group/testcase/ucode:
  gcc-7/performance/x86_64-rhel-7.2-clear_lck_7595/process/100%/clear-ota-25590-x86_64-2018-10-18.cgz/lkp-cfl-e1/hackbench/0xb8

commit: 
  ba5c5e4a5d ("arm64: move jump_label_init() before parse_early_param()")
  6471384af2 ("mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options")

ba5c5e4a5da443e8 6471384af2a6530696fc0203baf 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
     88494            -4.3%      84657        hackbench.throughput
    138438 ±  4%     +31.9%     182656 ±  7%  hackbench.time.involuntary_context_switches
    694.79            -5.9%     653.85        hackbench.time.user_time
 5.568e+08            -4.3%  5.328e+08        hackbench.workload
   8544005 ± 38%     -45.2%    4685694 ±  6%  cpuidle.C1E.time
      5.31 ± 11%     +29.2%       6.86 ± 16%  sched_debug.cpu.nr_uninterruptible.stddev
      3.48            +4.6%       3.64        turbostat.RAMWatt
      0.84 ±  2%      +0.1        0.95        mpstat.cpu.-1.irq%
      0.05 ±  3%      +0.0        0.06 ± 10%  mpstat.cpu.-1.soft%
  15632269            -4.1%   14984464 ±  3%  proc-vmstat.numa_hit
  15632269            -4.1%   14984464 ±  3%  proc-vmstat.numa_local
  15715101            -4.2%   15056704 ±  3%  proc-vmstat.pgalloc_normal
  15686075            -4.2%   15022294 ±  3%  proc-vmstat.pgfree
     45.36            -2.9%      44.03        perf-stat.i.MPKI
 9.663e+09            -2.7%  9.402e+09        perf-stat.i.branch-instructions
 1.088e+08            -2.7%  1.058e+08        perf-stat.i.branch-misses
     13.34            +0.3       13.62        perf-stat.i.cache-miss-rate%
 2.908e+08            -4.1%  2.787e+08        perf-stat.i.cache-misses
 2.179e+09            -6.0%  2.048e+09        perf-stat.i.cache-references
      1.26            +3.4%       1.31        perf-stat.i.cpi
     61720            -2.9%      59924        perf-stat.i.cpu-migrations
    210.15            +4.0%     218.46        perf-stat.i.cycles-between-cache-misses
 1.449e+10            -3.3%  1.401e+10        perf-stat.i.dTLB-loads
 1.015e+10            -3.5%   9.79e+09        perf-stat.i.dTLB-stores
 4.808e+10            -3.3%  4.651e+10        perf-stat.i.instructions
      0.79            -3.3%       0.77        perf-stat.i.ipc
  20904568            -2.2%   20436868        perf-stat.i.node-loads
  25841390           +79.8%   46459814        perf-stat.i.node-stores
     45.31            -2.8%      44.03        perf-stat.overall.MPKI
     13.35            +0.3       13.61        perf-stat.overall.cache-miss-rate%
      1.26            +3.5%       1.31        perf-stat.overall.cpi
    208.67            +4.4%     217.94        perf-stat.overall.cycles-between-cache-misses
      0.79            -3.4%       0.77        perf-stat.overall.ipc
 9.647e+09            -2.7%  9.387e+09        perf-stat.ps.branch-instructions
 1.086e+08            -2.7%  1.056e+08        perf-stat.ps.branch-misses
 2.903e+08            -4.1%  2.783e+08        perf-stat.ps.cache-misses
 2.175e+09            -6.0%  2.045e+09        perf-stat.ps.cache-references
     61620            -2.9%      59827        perf-stat.ps.cpu-migrations
 1.447e+10            -3.3%  1.399e+10        perf-stat.ps.dTLB-loads
 1.013e+10            -3.5%  9.774e+09        perf-stat.ps.dTLB-stores
   4.8e+10            -3.3%  4.643e+10        perf-stat.ps.instructions
  20870674            -2.2%   20403728        perf-stat.ps.node-loads
  25799504           +79.8%   46384429        perf-stat.ps.node-stores
 2.964e+13            -3.4%  2.863e+13        perf-stat.total.instructions
     38.02            -1.3       36.70        perf-profile.calltrace.cycles-pp.ksys_read.__x64_sys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
     38.11            -1.3       36.80        perf-profile.calltrace.cycles-pp.__x64_sys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
     36.86            -1.3       35.58        perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.__x64_sys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
     34.13            -1.2       32.97        perf-profile.calltrace.cycles-pp.__vfs_read.vfs_read.ksys_read.__x64_sys_read.do_syscall_64
     33.76            -1.1       32.61        perf-profile.calltrace.cycles-pp.new_sync_read.__vfs_read.vfs_read.ksys_read.__x64_sys_read
     32.94            -1.1       31.84        perf-profile.calltrace.cycles-pp.sock_read_iter.new_sync_read.__vfs_read.vfs_read.ksys_read
     31.73            -1.0       30.70        perf-profile.calltrace.cycles-pp.sock_recvmsg.sock_read_iter.new_sync_read.__vfs_read.vfs_read
     31.32            -1.0       30.30        perf-profile.calltrace.cycles-pp.unix_stream_recvmsg.sock_recvmsg.sock_read_iter.new_sync_read.__vfs_read
     30.92            -1.0       29.92        perf-profile.calltrace.cycles-pp.unix_stream_read_generic.unix_stream_recvmsg.sock_recvmsg.sock_read_iter.new_sync_read
      2.61 ±  3%      -0.6        1.97        perf-profile.calltrace.cycles-pp.skb_set_owner_w.sock_alloc_send_pskb.unix_stream_sendmsg.sock_sendmsg.sock_write_iter
      2.02 ±  4%      -0.6        1.39        perf-profile.calltrace.cycles-pp.refcount_add_not_zero_checked.refcount_add_checked.skb_set_owner_w.sock_alloc_send_pskb.unix_stream_sendmsg
      2.03 ±  4%      -0.6        1.39        perf-profile.calltrace.cycles-pp.refcount_add_checked.skb_set_owner_w.sock_alloc_send_pskb.unix_stream_sendmsg.sock_sendmsg
      2.68            -0.6        2.05 ±  2%  perf-profile.calltrace.cycles-pp._raw_spin_lock.unix_stream_sendmsg.sock_sendmsg.sock_write_iter.new_sync_write
      6.64 ±  3%      -0.4        6.25 ±  2%  perf-profile.calltrace.cycles-pp.sock_def_readable.unix_stream_sendmsg.sock_sendmsg.sock_write_iter.new_sync_write
      9.06            -0.4        8.68 ±  2%  perf-profile.calltrace.cycles-pp.unix_stream_read_actor.unix_stream_read_generic.unix_stream_recvmsg.sock_recvmsg.sock_read_iter
      8.98            -0.4        8.59 ±  2%  perf-profile.calltrace.cycles-pp.skb_copy_datagram_iter.unix_stream_read_actor.unix_stream_read_generic.unix_stream_recvmsg.sock_recvmsg
      8.74            -0.4        8.36 ±  2%  perf-profile.calltrace.cycles-pp.__skb_datagram_iter.skb_copy_datagram_iter.unix_stream_read_actor.unix_stream_read_generic.unix_stream_recvmsg
      8.41            -0.4        8.05 ±  2%  perf-profile.calltrace.cycles-pp.simple_copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.unix_stream_read_actor.unix_stream_read_generic
     11.82            -0.3       11.47        perf-profile.calltrace.cycles-pp.consume_skb.unix_stream_read_generic.unix_stream_recvmsg.sock_recvmsg.sock_read_iter
      6.11            -0.3        5.79        perf-profile.calltrace.cycles-pp._copy_to_iter.simple_copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.unix_stream_read_actor
      5.04            -0.3        4.79 ±  2%  perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string._copy_to_iter.simple_copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter
      3.55 ±  3%      -0.2        3.36 ±  2%  perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.__wake_up_sync_key.sock_def_readable.unix_stream_sendmsg
      3.57            -0.2        3.38        perf-profile.calltrace.cycles-pp.kmem_cache_free.kfree_skbmem.consume_skb.unix_stream_read_generic.unix_stream_recvmsg
      7.34            -0.2        7.16        perf-profile.calltrace.cycles-pp.skb_release_all.consume_skb.unix_stream_read_generic.unix_stream_recvmsg.sock_recvmsg
      3.67            -0.2        3.50        perf-profile.calltrace.cycles-pp.kfree_skbmem.consume_skb.unix_stream_read_generic.unix_stream_recvmsg.sock_recvmsg
      3.47            -0.2        3.31        perf-profile.calltrace.cycles-pp.skb_release_head_state.skb_release_all.consume_skb.unix_stream_read_generic.unix_stream_recvmsg
      2.90            -0.2        2.75        perf-profile.calltrace.cycles-pp.sock_wfree.unix_destruct_scm.skb_release_head_state.skb_release_all.consume_skb
      1.85 ±  3%      -0.1        1.70 ±  4%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.__wake_up_common_lock.__wake_up_sync_key.sock_def_readable.unix_stream_sendmsg
      1.53 ±  3%      -0.1        1.39 ±  4%  perf-profile.calltrace.cycles-pp.syscall_return_via_sysret
      2.32 ±  2%      -0.1        2.18        perf-profile.calltrace.cycles-pp.___cache_free.kfree.skb_free_head.skb_release_data.skb_release_all
      4.39            -0.1        4.25        perf-profile.calltrace.cycles-pp.skb_copy_datagram_from_iter.unix_stream_sendmsg.sock_sendmsg.sock_write_iter.new_sync_write
      1.77 ±  3%      -0.1        1.63 ±  4%  perf-profile.calltrace.cycles-pp.queued_spin_lock_slowpath._raw_spin_lock_irqsave.__wake_up_common_lock.__wake_up_sync_key.sock_def_readable
      3.28            -0.1        3.15        perf-profile.calltrace.cycles-pp.unix_destruct_scm.skb_release_head_state.skb_release_all.consume_skb.unix_stream_read_generic
      2.04 ±  2%      -0.1        1.92 ±  2%  perf-profile.calltrace.cycles-pp.schedule_timeout.unix_stream_read_generic.unix_stream_recvmsg.sock_recvmsg.sock_read_iter
      2.67            -0.1        2.55        perf-profile.calltrace.cycles-pp.skb_queue_tail.unix_stream_sendmsg.sock_sendmsg.sock_write_iter.new_sync_write
      1.96 ±  2%      -0.1        1.84 ±  2%  perf-profile.calltrace.cycles-pp.__sched_text_start.schedule.schedule_timeout.unix_stream_read_generic.unix_stream_recvmsg
      2.00 ±  2%      -0.1        1.88 ±  2%  perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.unix_stream_read_generic.unix_stream_recvmsg.sock_recvmsg
      2.68 ±  2%      -0.1        2.57 ±  2%  perf-profile.calltrace.cycles-pp._copy_from_iter.skb_copy_datagram_from_iter.unix_stream_sendmsg.sock_sendmsg.sock_write_iter
      1.43 ±  2%      -0.1        1.33        perf-profile.calltrace.cycles-pp.cache_alloc_refill.kmem_cache_alloc_node_trace.__kmalloc_node_track_caller.__kmalloc_reserve.__alloc_skb
      1.01 ±  3%      -0.1        0.91 ±  2%  perf-profile.calltrace.cycles-pp._raw_spin_lock.___cache_free.kfree.skb_free_head.skb_release_data
      3.91            -0.1        3.81        perf-profile.calltrace.cycles-pp.syscall_slow_exit_work.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.40 ±  4%      -0.1        1.29 ±  2%  perf-profile.calltrace.cycles-pp.skb_unlink.unix_stream_read_generic.unix_stream_recvmsg.sock_recvmsg.sock_read_iter
      1.71 ±  2%      -0.1        1.61        perf-profile.calltrace.cycles-pp.___cache_free.kmem_cache_free.kfree_skbmem.consume_skb.unix_stream_read_generic
      0.93 ±  3%      -0.1        0.83 ±  2%  perf-profile.calltrace.cycles-pp.queued_spin_lock_slowpath._raw_spin_lock.___cache_free.kfree.skb_free_head
      3.24            -0.1        3.15        perf-profile.calltrace.cycles-pp.__audit_syscall_exit.syscall_slow_exit_work.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.38 ±  2%      -0.1        1.29        perf-profile.calltrace.cycles-pp.rw_verify_area.vfs_read.ksys_read.__x64_sys_read.do_syscall_64
      3.74            -0.1        3.66        perf-profile.calltrace.cycles-pp.kmem_cache_alloc_node.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb.unix_stream_sendmsg
      0.96 ±  4%      -0.1        0.89 ±  2%  perf-profile.calltrace.cycles-pp.stack_trace_save_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.activate_task
      0.67 ±  3%      -0.1        0.60        perf-profile.calltrace.cycles-pp._raw_spin_lock.cache_alloc_refill.kmem_cache_alloc_node_trace.__kmalloc_node_track_caller.__kmalloc_reserve
      0.97            -0.1        0.90        perf-profile.calltrace.cycles-pp.security_file_permission.rw_verify_area.vfs_read.ksys_read.__x64_sys_read
      0.61 ±  3%      -0.1        0.55        perf-profile.calltrace.cycles-pp.queued_spin_lock_slowpath._raw_spin_lock.cache_alloc_refill.kmem_cache_alloc_node_trace.__kmalloc_node_track_caller
      0.81 ±  4%      -0.1        0.75        perf-profile.calltrace.cycles-pp.arch_stack_walk.stack_trace_save_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair
      0.57 ±  5%      -0.0        0.52 ±  2%  perf-profile.calltrace.cycles-pp._raw_spin_unlock_irqrestore.skb_queue_tail.unix_stream_sendmsg.sock_sendmsg.sock_write_iter
      0.61 ±  2%      -0.0        0.57 ±  3%  perf-profile.calltrace.cycles-pp._raw_spin_lock.unix_stream_read_generic.unix_stream_recvmsg.sock_recvmsg.sock_read_iter
      0.66            -0.0        0.61 ±  3%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.skb_unlink.unix_stream_read_generic.unix_stream_recvmsg.sock_recvmsg
      0.65            -0.0        0.61 ±  4%  perf-profile.calltrace.cycles-pp.wait_for_unix_gc.unix_stream_sendmsg.sock_sendmsg.sock_write_iter.new_sync_write
      1.13 ±  2%      +0.1        1.23        perf-profile.calltrace.cycles-pp.ksize.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb.unix_stream_sendmsg
      1.10            +0.1        1.21        perf-profile.calltrace.cycles-pp.__ksize.ksize.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb
      0.38 ± 57%      +0.1        0.53        perf-profile.calltrace.cycles-pp.refcount_dec_and_test_checked.consume_skb.unix_stream_read_generic.unix_stream_recvmsg.sock_recvmsg
      1.80 ±  2%      +0.2        1.96        perf-profile.calltrace.cycles-pp.refcount_inc_not_zero_checked.unix_stream_read_generic.unix_stream_recvmsg.sock_recvmsg.sock_read_iter
     92.56            +0.4       92.96        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
     90.14            +0.7       90.81        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +0.9        0.86 ±  6%  perf-profile.calltrace.cycles-pp.memset_erms.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb.unix_stream_sendmsg
      0.00            +0.9        0.89 ±  8%  perf-profile.calltrace.cycles-pp.memset_erms.__kmalloc_node_track_caller.__kmalloc_reserve.__alloc_skb.alloc_skb_with_frags
      4.35            +1.5        5.85        perf-profile.calltrace.cycles-pp.kmem_cache_alloc_node_trace.__kmalloc_node_track_caller.__kmalloc_reserve.__alloc_skb.alloc_skb_with_frags
     44.05            +2.2       46.24        perf-profile.calltrace.cycles-pp.ksys_write.__x64_sys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     44.14            +2.2       46.34        perf-profile.calltrace.cycles-pp.__x64_sys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     41.66            +2.3       43.94        perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.__x64_sys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     39.46            +2.4       41.82        perf-profile.calltrace.cycles-pp.__vfs_write.vfs_write.ksys_write.__x64_sys_write.do_syscall_64
     39.21            +2.4       41.58        perf-profile.calltrace.cycles-pp.new_sync_write.__vfs_write.vfs_write.ksys_write.__x64_sys_write
     38.46            +2.4       40.85        perf-profile.calltrace.cycles-pp.sock_write_iter.new_sync_write.__vfs_write.vfs_write.ksys_write
      4.91            +2.4        7.30        perf-profile.calltrace.cycles-pp.__kmalloc_node_track_caller.__kmalloc_reserve.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb
     37.20            +2.4       39.63        perf-profile.calltrace.cycles-pp.sock_sendmsg.sock_write_iter.new_sync_write.__vfs_write.vfs_write
     36.36            +2.4       38.80        perf-profile.calltrace.cycles-pp.unix_stream_sendmsg.sock_sendmsg.sock_write_iter.new_sync_write.__vfs_write
      5.15            +2.6        7.74        perf-profile.calltrace.cycles-pp.__kmalloc_reserve.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb.unix_stream_sendmsg
     16.15            +3.9       20.05        perf-profile.calltrace.cycles-pp.sock_alloc_send_pskb.unix_stream_sendmsg.sock_sendmsg.sock_write_iter.new_sync_write
     11.73            +4.5       16.21        perf-profile.calltrace.cycles-pp.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb.unix_stream_sendmsg.sock_sendmsg
     12.06            +4.5       16.55        perf-profile.calltrace.cycles-pp.alloc_skb_with_frags.sock_alloc_send_pskb.unix_stream_sendmsg.sock_sendmsg.sock_write_iter
     38.39            -1.3       37.06        perf-profile.children.cycles-pp.__x64_sys_read
     38.13            -1.3       36.81        perf-profile.children.cycles-pp.ksys_read
     36.94            -1.3       35.66        perf-profile.children.cycles-pp.vfs_read
     34.24            -1.2       33.07        perf-profile.children.cycles-pp.__vfs_read
     33.84            -1.2       32.68        perf-profile.children.cycles-pp.new_sync_read
     33.17            -1.1       32.06        perf-profile.children.cycles-pp.sock_read_iter
     31.60            -1.0       30.55        perf-profile.children.cycles-pp.unix_stream_recvmsg
     31.85            -1.0       30.81        perf-profile.children.cycles-pp.sock_recvmsg
     31.02            -1.0       30.02        perf-profile.children.cycles-pp.unix_stream_read_generic
      6.85            -1.0        5.88        perf-profile.children.cycles-pp._raw_spin_lock
      2.65 ±  3%      -0.6        2.00        perf-profile.children.cycles-pp.skb_set_owner_w
      2.08 ±  4%      -0.6        1.44        perf-profile.children.cycles-pp.refcount_add_checked
      2.23 ±  4%      -0.6        1.62        perf-profile.children.cycles-pp.refcount_add_not_zero_checked
      6.85 ±  2%      -0.4        6.44 ±  2%  perf-profile.children.cycles-pp.sock_def_readable
      9.10            -0.4        8.70 ±  2%  perf-profile.children.cycles-pp.unix_stream_read_actor
      9.04            -0.4        8.65 ±  2%  perf-profile.children.cycles-pp.skb_copy_datagram_iter
      8.79            -0.4        8.41 ±  2%  perf-profile.children.cycles-pp.__skb_datagram_iter
      8.50            -0.4        8.14 ±  2%  perf-profile.children.cycles-pp.simple_copy_to_iter
     11.93            -0.3       11.59        perf-profile.children.cycles-pp.consume_skb
      5.62            -0.3        5.29        perf-profile.children.cycles-pp.queued_spin_lock_slowpath
      6.17            -0.3        5.84        perf-profile.children.cycles-pp._copy_to_iter
      6.71            -0.3        6.42        perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      4.08            -0.2        3.84        perf-profile.children.cycles-pp.___cache_free
      3.02            -0.2        2.80 ±  2%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      5.15            -0.2        4.93        perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      3.66            -0.2        3.46        perf-profile.children.cycles-pp.kmem_cache_free
      7.41            -0.2        7.22        perf-profile.children.cycles-pp.skb_release_all
      3.74            -0.2        3.58        perf-profile.children.cycles-pp.kfree_skbmem
      3.01            -0.2        2.85        perf-profile.children.cycles-pp.sock_wfree
      3.51            -0.2        3.36        perf-profile.children.cycles-pp.skb_release_head_state
      2.38 ±  2%      -0.1        2.23        perf-profile.children.cycles-pp.__might_sleep
      4.44            -0.1        4.30        perf-profile.children.cycles-pp.skb_copy_datagram_from_iter
      3.53            -0.1        3.40        perf-profile.children.cycles-pp.unix_destruct_scm
      1.55 ±  3%      -0.1        1.42 ±  2%  perf-profile.children.cycles-pp.___might_sleep
      2.10 ±  2%      -0.1        1.97        perf-profile.children.cycles-pp.cache_alloc_refill
      2.70            -0.1        2.58        perf-profile.children.cycles-pp.skb_queue_tail
      2.75 ±  2%      -0.1        2.64 ±  2%  perf-profile.children.cycles-pp._copy_from_iter
      1.84            -0.1        1.74        perf-profile.children.cycles-pp.rw_verify_area
      4.03            -0.1        3.93        perf-profile.children.cycles-pp.syscall_slow_exit_work
      1.06 ±  4%      -0.1        0.96 ±  3%  perf-profile.children.cycles-pp.unwind_next_frame
      1.41 ±  4%      -0.1        1.31 ±  2%  perf-profile.children.cycles-pp.skb_unlink
      2.71            -0.1        2.61        perf-profile.children.cycles-pp.schedule_timeout
      2.83            -0.1        2.73        perf-profile.children.cycles-pp.__sched_text_start
      3.43            -0.1        3.34        perf-profile.children.cycles-pp.__audit_syscall_exit
      2.65            -0.1        2.57        perf-profile.children.cycles-pp.schedule
      1.22 ±  4%      -0.1        1.14        perf-profile.children.cycles-pp.stack_trace_save_tsk
      1.15            -0.1        1.07        perf-profile.children.cycles-pp.security_file_permission
      0.58 ±  5%      -0.1        0.50 ±  3%  perf-profile.children.cycles-pp.memcg_kmem_put_cache
      2.65            -0.1        2.58 ±  2%  perf-profile.children.cycles-pp.__fdget_pos
      1.06 ±  3%      -0.1        0.99        perf-profile.children.cycles-pp.arch_stack_walk
      0.90 ±  3%      -0.1        0.83 ±  3%  perf-profile.children.cycles-pp.__might_fault
      1.42 ±  2%      -0.1        1.37        perf-profile.children.cycles-pp.__fsnotify_parent
      0.17 ±  7%      -0.1        0.12 ± 10%  perf-profile.children.cycles-pp.should_failslab
      1.03            -0.1        0.98        perf-profile.children.cycles-pp.__list_del_entry_valid
      0.89 ±  2%      -0.0        0.85 ±  2%  perf-profile.children.cycles-pp.deactivate_task
      0.30            -0.0        0.26 ±  4%  perf-profile.children.cycles-pp.copyin
      1.07 ±  2%      -0.0        1.04        perf-profile.children.cycles-pp.unroll_tree_refs
      1.01            -0.0        0.98        perf-profile.children.cycles-pp.mutex_lock
      0.35 ±  2%      -0.0        0.32 ±  5%  perf-profile.children.cycles-pp.dequeue_entity
      0.36 ±  3%      -0.0        0.34 ±  2%  perf-profile.children.cycles-pp.select_idle_sibling
      0.25 ±  2%      -0.0        0.23 ±  2%  perf-profile.children.cycles-pp.copyout
      0.33 ±  3%      -0.0        0.31 ±  3%  perf-profile.children.cycles-pp.path_put
      0.56            +0.0        0.59 ±  2%  perf-profile.children.cycles-pp.refcount_dec_and_test_checked
      0.08 ± 10%      +0.0        0.11 ±  7%  perf-profile.children.cycles-pp.cache_grow_begin
      0.11 ±  6%      +0.0        0.15 ± 14%  perf-profile.children.cycles-pp.kmalloc_slab
      0.00            +0.1        0.08 ± 10%  perf-profile.children.cycles-pp.memset
      1.15            +0.1        1.26        perf-profile.children.cycles-pp.__ksize
      1.18 ±  2%      +0.1        1.30        perf-profile.children.cycles-pp.ksize
      2.52 ±  2%      +0.2        2.69        perf-profile.children.cycles-pp.refcount_inc_not_zero_checked
     92.60            +0.4       93.00        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     90.32            +0.7       90.98        perf-profile.children.cycles-pp.do_syscall_64
      4.57            +1.6        6.21        perf-profile.children.cycles-pp.kmem_cache_alloc_node_trace
      0.00            +1.8        1.77 ±  8%  perf-profile.children.cycles-pp.memset_erms
     44.45            +2.2       46.62        perf-profile.children.cycles-pp.__x64_sys_write
     44.16            +2.2       46.36        perf-profile.children.cycles-pp.ksys_write
     41.70            +2.3       43.98        perf-profile.children.cycles-pp.vfs_write
     39.54            +2.4       41.89        perf-profile.children.cycles-pp.__vfs_write
     39.29            +2.4       41.66        perf-profile.children.cycles-pp.new_sync_write
     38.63            +2.4       41.02        perf-profile.children.cycles-pp.sock_write_iter
      4.98            +2.4        7.41        perf-profile.children.cycles-pp.__kmalloc_node_track_caller
     37.26            +2.4       39.70        perf-profile.children.cycles-pp.sock_sendmsg
     36.68            +2.4       39.13        perf-profile.children.cycles-pp.unix_stream_sendmsg
      5.18            +2.6        7.80        perf-profile.children.cycles-pp.__kmalloc_reserve
     16.19            +3.9       20.09        perf-profile.children.cycles-pp.sock_alloc_send_pskb
     11.81            +4.5       16.28        perf-profile.children.cycles-pp.__alloc_skb
     12.11            +4.5       16.59        perf-profile.children.cycles-pp.alloc_skb_with_frags
      3.37            -0.7        2.63        perf-profile.self.cycles-pp._raw_spin_lock
      2.21 ±  3%      -0.6        1.60        perf-profile.self.cycles-pp.refcount_add_not_zero_checked
      5.60            -0.3        5.27        perf-profile.self.cycles-pp.queued_spin_lock_slowpath
      6.68            -0.3        6.39        perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      1.77            -0.2        1.55        perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      2.98            -0.2        2.77 ±  2%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      1.83 ±  4%      -0.2        1.65 ±  6%  perf-profile.self.cycles-pp.__check_object_size
      1.45 ±  3%      -0.1        1.34 ±  2%  perf-profile.self.cycles-pp.___might_sleep
      2.98            -0.1        2.88        perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.65            -0.1        0.55 ±  2%  perf-profile.self.cycles-pp.sock_wfree
      1.71            -0.1        1.61 ±  2%  perf-profile.self.cycles-pp.unix_stream_read_generic
      2.94            -0.1        2.85        perf-profile.self.cycles-pp.unix_stream_sendmsg
      1.85            -0.1        1.78        perf-profile.self.cycles-pp.kmem_cache_free
      0.92 ±  3%      -0.1        0.85 ±  4%  perf-profile.self.cycles-pp.__audit_syscall_entry
      1.00 ±  2%      -0.1        0.94        perf-profile.self.cycles-pp.__list_del_entry_valid
      1.10 ±  3%      -0.1        1.05 ±  2%  perf-profile.self.cycles-pp.sock_read_iter
      0.50 ±  4%      -0.1        0.45 ±  4%  perf-profile.self.cycles-pp.memcg_kmem_put_cache
      0.71 ±  3%      -0.1        0.66        perf-profile.self.cycles-pp.new_sync_read
      1.00 ±  2%      -0.0        0.95 ±  3%  perf-profile.self.cycles-pp.__might_sleep
      1.83            -0.0        1.79        perf-profile.self.cycles-pp.__audit_syscall_exit
      0.34 ±  5%      -0.0        0.30 ±  4%  perf-profile.self.cycles-pp.skb_copy_datagram_from_iter
      0.18 ±  4%      -0.0        0.14 ±  5%  perf-profile.self.cycles-pp.copyin
      0.40 ±  2%      -0.0        0.37 ±  5%  perf-profile.self.cycles-pp.skb_set_owner_w
      0.10 ±  8%      -0.0        0.08 ±  5%  perf-profile.self.cycles-pp.should_failslab
      0.23 ±  3%      -0.0        0.21 ±  3%  perf-profile.self.cycles-pp.skb_copy_datagram_iter
      0.13 ±  6%      -0.0        0.11 ±  7%  perf-profile.self.cycles-pp.update_rq_clock
      0.14 ±  6%      -0.0        0.12 ±  3%  perf-profile.self.cycles-pp.stack_trace_consume_entry_nosched
      0.11 ±  4%      -0.0        0.10 ±  4%  perf-profile.self.cycles-pp.__update_load_avg_se
      0.11            +0.0        0.12 ±  4%  perf-profile.self.cycles-pp.kfree_skbmem
      0.31 ±  2%      +0.0        0.34 ±  2%  perf-profile.self.cycles-pp.skb_release_data
      0.10 ± 11%      +0.0        0.13 ± 14%  perf-profile.self.cycles-pp.kmalloc_slab
      0.18 ±  2%      +0.0        0.23 ±  4%  perf-profile.self.cycles-pp.__kmalloc_reserve
      0.15 ±  3%      +0.1        0.21 ±  2%  perf-profile.self.cycles-pp.__kmalloc_node_track_caller
      1.07 ±  2%      +0.1        1.18 ±  2%  perf-profile.self.cycles-pp.kfree
      1.12            +0.1        1.24        perf-profile.self.cycles-pp.__ksize
      2.51 ±  2%      +0.2        2.67        perf-profile.self.cycles-pp.refcount_inc_not_zero_checked
      1.38 ± 10%      +1.0        2.39        perf-profile.self.cycles-pp.__alloc_skb
      0.00            +1.7        1.73 ±  8%  perf-profile.self.cycles-pp.memset_erms
      1.26            +1.8        3.08        perf-profile.self.cycles-pp.kmem_cache_alloc_node_trace





Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Oliver Sang


View attachment "config-5.2.0-05754-g6471384af2a65" of type "text/plain" (202142 bytes)

View attachment "job-script" of type "text/plain" (7255 bytes)

View attachment "job.yaml" of type "text/plain" (4647 bytes)

View attachment "reproduce" of type "text/plain" (301 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ