lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Date:   Wed, 4 Mar 2020 15:43:47 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     Linus Torvalds <torvalds@...ux-foundation.org>
Cc:     Feng Tang <feng.tang@...el.com>,
        "Eric W. Biederman" <ebiederm@...ssion.com>, Huang@...o2-debian,
        Ying <ying.huang@...el.com>, Philip Li <philip.li@...el.com>,
        Andi Kleen <andi.kleen@...el.com>,
        Jiri Olsa <jolsa@...hat.com>,
        Peter Zijlstra <peterz@...radead.org>,
        LKML <linux-kernel@...r.kernel.org>,
        Linus Torvalds <torvalds@...ux-foundation.org>,
        lkp@...ts.01.org
Subject: [signal] fda31c5029: will-it-scale.per_process_ops 58.4% improvement

Greeting,

FYI, we noticed a 58.4% improvement of will-it-scale.per_process_ops due to commit:


commit: fda31c50292a5062332fa0343c084bd9f46604d9 ("signal: avoid double atomic counter increments for user accounting")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master

in testcase: will-it-scale
on test machine: 192 threads Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory
with following parameters:

	nr_task: 50%
	mode: process
	test: signal1
	cpufreq_governor: performance
	ucode: 0x500002c

test-description: Will It Scale takes a testcase and runs it from 1 through to n parallel copies to see if the testcase will scale. It builds both a process and threads based test in order to see any differences between the two.
test-url: https://github.com/antonblanchard/will-it-scale

In addition to that, the commit also has significant impact on the following tests:

+------------------+---------------------------------------------------------------------------+
| testcase: change | will-it-scale: will-it-scale.per_process_ops 110.2% improvement           |
| test machine     | 72 threads Intel(R) Xeon(R) Gold 6139 CPU @ 2.30GHz with 128G memory      |
| test parameters  | cpufreq_governor=performance                                              |
|                  | mode=process                                                              |
|                  | nr_task=100%                                                              |
|                  | test=signal1                                                              |
|                  | ucode=0x2000065                                                           |
+------------------+---------------------------------------------------------------------------+
| testcase: change | netperf: netperf.Throughput_total_tps 10.2% improvement                   |
| test machine     | 192 threads Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory |
| test parameters  | cluster=cs-localhost                                                      |
|                  | cpufreq_governor=performance                                              |
|                  | ip=ipv4                                                                   |
|                  | nr_threads=16                                                             |
|                  | runtime=300s                                                              |
|                  | test=TCP_CRR                                                              |
|                  | ucode=0x500002c                                                           |
+------------------+---------------------------------------------------------------------------+
| testcase: change | will-it-scale: will-it-scale.per_process_ops 552.8% improvement           |
| test machine     | 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory      |
| test parameters  | cpufreq_governor=performance                                              |
|                  | mode=process                                                              |
|                  | nr_task=100%                                                              |
|                  | test=signal1                                                              |
|                  | ucode=0x11                                                                |
+------------------+---------------------------------------------------------------------------+




Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode:
  gcc-7/performance/x86_64-rhel-7.6/process/50%/debian-x86_64-20191114.cgz/lkp-csl-2ap3/signal1/will-it-scale/0x500002c

commit: 
  c5f8689118 (" RISC-V Fixes for 5.6-rc4")
  fda31c5029 ("signal: avoid double atomic counter increments for user accounting")

c5f86891185c408b fda31c50292a5062332fa0343c0 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
     44576           +58.4%      70601        will-it-scale.per_process_ops
   4279339           +58.4%    6777813        will-it-scale.workload
      1790           +16.0%       2076        vmstat.system.cs
     46699 ± 78%     -61.8%      17845 ± 20%  cpuidle.POLL.time
      8341 ± 51%     -48.4%       4307 ± 19%  cpuidle.POLL.usage
      0.00 ± 40%      -0.0        0.00 ± 50%  mpstat.cpu.all.soft%
      1.19            +0.7        1.88        mpstat.cpu.all.usr%
     40006 ± 15%     -27.2%      29109 ±  3%  meminfo.Inactive
     39419 ± 15%     -27.7%      28515 ±  3%  meminfo.Inactive(anon)
     57649 ± 13%     -34.9%      37538 ±  4%  meminfo.Mapped
    328365 ±  7%     -34.6%     214804 ± 18%  numa-numastat.node1.local_node
    351656 ± 11%     -30.1%     245828 ± 15%  numa-numastat.node1.numa_hit
    202334 ± 40%     +57.5%     318712 ± 15%  numa-numastat.node3.local_node
     31045           -89.7%       3205 ±167%  numa-numastat.node3.other_node
      9863 ± 15%     -26.9%       7209 ±  3%  proc-vmstat.nr_inactive_anon
     14568 ± 13%     -33.9%       9625 ±  4%  proc-vmstat.nr_mapped
      9863 ± 15%     -26.9%       7209 ±  3%  proc-vmstat.nr_zone_inactive_anon
     10776 ± 49%     -73.3%       2880 ±106%  proc-vmstat.numa_hint_faults
     11673 ± 18%     -31.2%       8028 ± 20%  numa-meminfo.node0.Mapped
      3411 ± 16%     -20.7%       2705 ± 18%  numa-meminfo.node0.PageTables
     12604 ± 10%     -42.2%       7290 ± 14%  numa-meminfo.node2.Mapped
    200125 ± 23%     -30.4%     139206 ± 16%  numa-meminfo.node3.Active
    199214 ± 23%     -30.1%     139206 ± 16%  numa-meminfo.node3.Active(anon)
    120134 ± 42%     -63.8%      43520 ± 30%  numa-meminfo.node3.AnonPages
     20962 ± 25%     -49.5%      10591 ± 22%  numa-meminfo.node3.Mapped
      2943 ± 17%     -29.5%       2076 ± 22%  numa-vmstat.node0.nr_mapped
    851.75 ± 16%     -20.6%     676.50 ± 18%  numa-vmstat.node0.nr_page_table_pages
      3209 ± 11%     -42.0%       1861 ± 16%  numa-vmstat.node2.nr_mapped
     49797 ± 23%     -30.2%      34782 ± 16%  numa-vmstat.node3.nr_active_anon
     30036 ± 42%     -63.7%      10913 ± 29%  numa-vmstat.node3.nr_anon_pages
      5298 ± 25%     -48.8%       2710 ± 21%  numa-vmstat.node3.nr_mapped
     49797 ± 23%     -30.2%      34782 ± 16%  numa-vmstat.node3.nr_zone_active_anon
    118430           -23.3%      90848 ±  5%  numa-vmstat.node3.numa_other
      3600 ± 19%    +141.3%       8689        interrupts.CPU170.NMI:Non-maskable_interrupts
      3600 ± 19%    +141.3%       8689        interrupts.CPU170.PMI:Performance_monitoring_interrupts
      5480 ± 38%     +58.5%       8688        interrupts.CPU184.NMI:Non-maskable_interrupts
      5480 ± 38%     +58.5%       8688        interrupts.CPU184.PMI:Performance_monitoring_interrupts
      5360 ± 27%     -35.2%       3476 ± 43%  interrupts.CPU191.NMI:Non-maskable_interrupts
      5360 ± 27%     -35.2%       3476 ± 43%  interrupts.CPU191.PMI:Performance_monitoring_interrupts
    372.50 ± 71%     -84.4%      58.00 ± 42%  interrupts.CPU26.RES:Rescheduling_interrupts
    153.50 ±105%     -82.6%      26.75 ± 36%  interrupts.CPU27.RES:Rescheduling_interrupts
    323.25 ±148%     -95.7%      13.75 ± 37%  interrupts.CPU40.RES:Rescheduling_interrupts
      3575 ± 33%    +122.3%       7946 ± 16%  interrupts.CPU50.NMI:Non-maskable_interrupts
      3575 ± 33%    +122.3%       7946 ± 16%  interrupts.CPU50.PMI:Performance_monitoring_interrupts
      5569 ± 48%     -42.3%       3215 ± 48%  interrupts.CPU88.NMI:Non-maskable_interrupts
      5569 ± 48%     -42.3%       3215 ± 48%  interrupts.CPU88.PMI:Performance_monitoring_interrupts
      6266 ± 27%     +38.6%       8685        interrupts.CPU95.NMI:Non-maskable_interrupts
      6266 ± 27%     +38.6%       8685        interrupts.CPU95.PMI:Performance_monitoring_interrupts
     13586 ± 69%     -99.8%      32.06 ±133%  sched_debug.cfs_rq:/.MIN_vruntime.avg
   1213223 ± 49%     -99.6%       4676 ±126%  sched_debug.cfs_rq:/.MIN_vruntime.max
    125620 ± 56%     -99.7%     384.84 ±130%  sched_debug.cfs_rq:/.MIN_vruntime.stddev
      9914 ±  7%     -22.2%       7709 ± 11%  sched_debug.cfs_rq:/.load.avg
     29354 ± 21%     -50.8%      14445 ± 73%  sched_debug.cfs_rq:/.load.stddev
    346.92 ± 22%     -38.2%     214.42 ± 28%  sched_debug.cfs_rq:/.load_avg.max
     13586 ± 69%     -99.8%      32.07 ±133%  sched_debug.cfs_rq:/.max_vruntime.avg
   1213225 ± 49%     -99.6%       4677 ±126%  sched_debug.cfs_rq:/.max_vruntime.max
    125620 ± 56%     -99.7%     384.88 ±130%  sched_debug.cfs_rq:/.max_vruntime.stddev
      0.34 ± 75%     +93.1%       0.66 ± 36%  sched_debug.cfs_rq:/.nr_spread_over.avg
      9884 ±  7%     -22.1%       7697 ± 11%  sched_debug.cfs_rq:/.runnable_weight.avg
     29282 ± 22%     -50.9%      14373 ± 74%  sched_debug.cfs_rq:/.runnable_weight.stddev
      0.02           -77.0%       0.00 ±173%  sched_debug.cfs_rq:/.spread.avg
      0.18           -76.0%       0.04 ±173%  sched_debug.cfs_rq:/.spread.stddev
    426861 ± 18%     -57.3%     182108 ± 39%  sched_debug.cpu.avg_idle.min
     28.06 ±  4%     -54.6%      12.73 ± 18%  sched_debug.cpu.clock.stddev
     28.06 ±  4%     -54.6%      12.73 ± 18%  sched_debug.cpu.clock_task.stddev
    872983 ±  9%     -30.2%     609336 ±  6%  sched_debug.cpu.max_idle_balance_cost.max
     35455 ± 17%     -67.5%      11534 ± 42%  sched_debug.cpu.max_idle_balance_cost.stddev
      1450           +14.5%       1661        sched_debug.cpu.sched_count.avg
    459.75 ±  2%      +9.5%     503.42        sched_debug.cpu.sched_count.min
    573.27           +16.1%     665.57        sched_debug.cpu.sched_goidle.avg
    638.47           +16.0%     740.32        sched_debug.cpu.ttwu_count.avg
    166.54 ±  2%      +9.9%     183.04        sched_debug.cpu.ttwu_count.min
    160.17           +11.2%     178.17        sched_debug.cpu.ttwu_local.min
      4.62            +5.3%       4.86        perf-stat.i.MPKI
 4.414e+09           +51.7%  6.698e+09        perf-stat.i.branch-instructions
      0.91            -0.0        0.89        perf-stat.i.branch-miss-rate%
  39959035           +48.5%   59348514        perf-stat.i.branch-misses
  38375649 ±  2%     +57.3%   60353472        perf-stat.i.cache-misses
 1.001e+08           +60.6%  1.608e+08        perf-stat.i.cache-references
      1802           +13.3%       2042        perf-stat.i.context-switches
     14.06           -35.7%       9.04        perf-stat.i.cpi
 3.055e+11            -1.8%      3e+11        perf-stat.i.cpu-cycles
    179.00            +1.5%     181.67        perf-stat.i.cpu-migrations
      8007 ±  2%     -37.8%       4982        perf-stat.i.cycles-between-cache-misses
      0.00 ± 21%      +0.0        0.00 ±  9%  perf-stat.i.dTLB-load-miss-rate%
     77491 ± 30%    +126.0%     175135 ±  6%  perf-stat.i.dTLB-load-misses
 6.591e+09           +52.7%  1.006e+10        perf-stat.i.dTLB-loads
 4.005e+09           +53.2%  6.138e+09        perf-stat.i.dTLB-stores
     88.59            +3.6       92.17        perf-stat.i.iTLB-load-miss-rate%
  31297204           +57.9%   49408821        perf-stat.i.iTLB-load-misses
   3996196            +3.3%    4127542        perf-stat.i.iTLB-loads
  2.17e+10           +52.6%  3.312e+10        perf-stat.i.instructions
    696.80            -3.2%     674.72        perf-stat.i.instructions-per-iTLB-miss
      0.07           +55.7%       0.11        perf-stat.i.ipc
     94.13            -1.3       92.83        perf-stat.i.node-load-miss-rate%
   3766232            -6.6%    3517476        perf-stat.i.node-load-misses
    237318 ±  4%     +16.3%     276111 ±  8%  perf-stat.i.node-loads
   6669507           +33.6%    8911446        perf-stat.i.node-store-misses
      4.61            +5.2%       4.85        perf-stat.overall.MPKI
      0.90            -0.0        0.89        perf-stat.overall.branch-miss-rate%
     38.41            -0.9       37.54        perf-stat.overall.cache-miss-rate%
     14.09           -35.7%       9.06        perf-stat.overall.cpi
      7950 ±  2%     -37.4%       4975        perf-stat.overall.cycles-between-cache-misses
      0.00 ± 27%      +0.0        0.00 ±  6%  perf-stat.overall.dTLB-load-miss-rate%
      0.00 ±  3%      -0.0        0.00 ±  4%  perf-stat.overall.dTLB-store-miss-rate%
     88.70            +3.6       92.30        perf-stat.overall.iTLB-load-miss-rate%
    692.54            -3.2%     670.19        perf-stat.overall.instructions-per-iTLB-miss
      0.07           +55.5%       0.11        perf-stat.overall.ipc
     94.07            -1.3       92.73        perf-stat.overall.node-load-miss-rate%
   1533068            -3.9%    1473264        perf-stat.overall.path-length
 4.397e+09           +51.8%  6.676e+09        perf-stat.ps.branch-instructions
  39608863           +49.2%   59109187        perf-stat.ps.branch-misses
  38317702 ±  2%     +56.9%   60130399        perf-stat.ps.cache-misses
  99756959           +60.6%  1.602e+08        perf-stat.ps.cache-references
      1729           +16.8%       2020        perf-stat.ps.context-switches
 3.045e+11            -1.8%  2.991e+11        perf-stat.ps.cpu-cycles
    158.55 ±  3%     +12.1%     177.81        perf-stat.ps.cpu-migrations
     70109 ± 27%    +144.4%     171364 ±  6%  perf-stat.ps.dTLB-load-misses
 6.566e+09           +52.8%  1.003e+10        perf-stat.ps.dTLB-loads
     11636 ±  3%     +11.6%      12980 ±  4%  perf-stat.ps.dTLB-store-misses
 3.992e+09           +53.3%  6.118e+09        perf-stat.ps.dTLB-stores
  31214750           +57.8%   49260047        perf-stat.ps.iTLB-load-misses
   3974993            +3.4%    4111693        perf-stat.ps.iTLB-loads
 2.162e+10           +52.7%  3.301e+10        perf-stat.ps.instructions
   3751065            -6.6%    3504931        perf-stat.ps.node-load-misses
   6647605           +33.6%    8881930        perf-stat.ps.node-store-misses
 6.561e+12           +52.2%  9.986e+12        perf-stat.total.instructions
    121407           +13.3%     137596 ±  3%  softirqs.CPU0.TIMER
    117580 ± 17%     -16.9%      97759        softirqs.CPU100.RCU
    109909 ±  5%     -11.4%      97342        softirqs.CPU109.RCU
    107219            -9.1%      97448        softirqs.CPU110.RCU
    106157 ±  2%      -7.2%      98471        softirqs.CPU117.RCU
    107670           +16.8%     125810 ±  4%  softirqs.CPU120.RCU
    108291           +11.8%     121084        softirqs.CPU121.RCU
    123400           -11.9%     108711        softirqs.CPU121.TIMER
    106835           +13.8%     121530        softirqs.CPU122.RCU
    122489 ±  2%     -11.1%     108949 ±  2%  softirqs.CPU122.TIMER
    107742           +23.2%     132701 ±  9%  softirqs.CPU123.RCU
    114533 ± 11%     +19.7%     137067 ± 11%  softirqs.CPU124.RCU
    118272 ± 15%     +16.2%     137401 ± 11%  softirqs.CPU125.RCU
    123722 ±  3%     -11.6%     109416 ±  2%  softirqs.CPU126.TIMER
    122287            -9.3%     110860 ±  4%  softirqs.CPU127.TIMER
    122210            -9.1%     111043 ±  3%  softirqs.CPU129.TIMER
    108445           +27.4%     138162 ± 10%  softirqs.CPU130.RCU
    108077 ±  2%     +13.2%     122298        softirqs.CPU132.RCU
    122582           -10.6%     109537        softirqs.CPU132.TIMER
    121761           -11.1%     108260        softirqs.CPU133.TIMER
    106883           +12.8%     120534        softirqs.CPU134.RCU
    121741           -10.6%     108844        softirqs.CPU134.TIMER
    122110           -11.2%     108475        softirqs.CPU135.TIMER
     16219 ± 80%     -76.8%       3765 ±  2%  softirqs.CPU136.SCHED
    123635 ±  2%     -11.7%     109221 ±  2%  softirqs.CPU136.TIMER
    123450 ±  2%     -11.1%     109740 ±  2%  softirqs.CPU137.TIMER
    107653           +20.2%     129379 ± 10%  softirqs.CPU139.RCU
      3819 ±  2%    +226.0%      12454 ±116%  softirqs.CPU139.SCHED
    122150           -11.2%     108432        softirqs.CPU140.TIMER
    110775           +12.6%     124694 ±  4%  softirqs.CPU141.RCU
    122493           -10.5%     109640 ±  2%  softirqs.CPU141.TIMER
    122378           -10.7%     109334 ±  2%  softirqs.CPU142.TIMER
    122130            -9.1%     111060 ±  3%  softirqs.CPU143.TIMER
    105017           +24.6%     130841 ±  9%  softirqs.CPU144.RCU
    121320            -8.5%     111050 ±  3%  softirqs.CPU144.TIMER
    105072           +23.9%     130187 ±  9%  softirqs.CPU145.RCU
    121297            -9.3%     110057 ±  4%  softirqs.CPU145.TIMER
    104493           +36.9%     143009 ±  7%  softirqs.CPU146.RCU
      3778          +612.0%      26906 ± 51%  softirqs.CPU146.SCHED
    108147 ±  3%     +21.8%     131725 ±  9%  softirqs.CPU147.RCU
    105527           +24.1%     130999 ±  9%  softirqs.CPU149.RCU
      3799 ±  3%    +229.9%      12534 ±117%  softirqs.CPU149.SCHED
    122606 ±  2%     -10.4%     109903 ±  4%  softirqs.CPU151.TIMER
    122742 ±  2%     -12.0%     107991        softirqs.CPU152.TIMER
    121230           -10.4%     108591 ±  2%  softirqs.CPU153.TIMER
    106908           +21.3%     129639 ±  9%  softirqs.CPU154.RCU
    121328            -9.4%     109963 ±  4%  softirqs.CPU154.TIMER
    105635           +30.1%     137414 ± 10%  softirqs.CPU155.RCU
    105735           +20.8%     127735 ±  6%  softirqs.CPU156.RCU
    121271            -9.6%     109675 ±  2%  softirqs.CPU156.TIMER
    121328            -9.1%     110241 ±  3%  softirqs.CPU158.TIMER
    104725           +19.2%     124792 ±  2%  softirqs.CPU159.RCU
    121045           -10.7%     108148        softirqs.CPU159.TIMER
    104917           +24.3%     130388 ±  9%  softirqs.CPU160.RCU
    105583           +29.9%     137156 ± 10%  softirqs.CPU161.RCU
      3785          +449.6%      20804 ± 81%  softirqs.CPU161.SCHED
    104604           +18.0%     123390        softirqs.CPU162.RCU
    120928           -10.9%     107711        softirqs.CPU162.TIMER
    121305            -9.4%     109852 ±  4%  softirqs.CPU163.TIMER
    104715           +17.7%     123295        softirqs.CPU164.RCU
    121108           -11.1%     107628        softirqs.CPU164.TIMER
    121235            -9.6%     109543 ±  4%  softirqs.CPU165.TIMER
    105333           +23.1%     129700 ±  9%  softirqs.CPU167.RCU
    121734            -9.3%     110408 ±  3%  softirqs.CPU167.TIMER
     20965 ± 61%     -81.4%       3899        softirqs.CPU170.SCHED
    123907 ±  2%     -11.3%     109868        softirqs.CPU170.TIMER
     80419 ± 10%     +54.9%     124537 ±  7%  softirqs.CPU171.RCU
    125000 ±  2%     -11.1%     111087 ±  2%  softirqs.CPU171.TIMER
     81184 ± 11%     +56.5%     127065 ± 11%  softirqs.CPU172.RCU
    124065           -10.1%     111568 ±  3%  softirqs.CPU172.TIMER
     89620 ± 28%     +51.6%     135901 ± 12%  softirqs.CPU173.RCU
    124277 ±  2%      -9.6%     112358 ±  3%  softirqs.CPU174.TIMER
     74486 ±  2%     +76.2%     131238 ±  9%  softirqs.CPU175.RCU
    123147            -9.9%     110980 ±  2%  softirqs.CPU176.TIMER
     86630 ± 24%     +49.2%     129262 ± 10%  softirqs.CPU177.RCU
    123460 ±  2%      -9.7%     111516 ±  3%  softirqs.CPU177.TIMER
     74989 ±  3%     +68.2%     126121 ±  9%  softirqs.CPU178.RCU
    123656 ±  2%     -10.3%     110946 ±  2%  softirqs.CPU178.TIMER
     75002 ±  3%     +58.5%     118904        softirqs.CPU179.RCU
    124162 ±  2%     -12.0%     109245        softirqs.CPU179.TIMER
    124031           -10.2%     111386 ±  3%  softirqs.CPU180.TIMER
     77779 ±  4%     +74.5%     135752 ± 12%  softirqs.CPU181.RCU
      4453 ± 26%    +368.6%      20868 ± 81%  softirqs.CPU181.SCHED
     75281 ±  5%     +80.0%     135519 ± 11%  softirqs.CPU182.RCU
     86198 ± 19%     +50.4%     129625 ± 10%  softirqs.CPU183.RCU
    123871 ±  2%      -9.5%     112134 ±  3%  softirqs.CPU183.TIMER
    124761 ±  2%     -11.7%     110148        softirqs.CPU184.TIMER
     88837 ± 24%     +53.0%     135926 ± 11%  softirqs.CPU185.RCU
    123449           -11.5%     109196        softirqs.CPU186.TIMER
     74532 ±  2%     +71.3%     127646 ± 11%  softirqs.CPU187.RCU
    123890 ±  2%     -10.0%     111482 ±  3%  softirqs.CPU187.TIMER
     85095 ± 21%     +55.9%     132661 ± 11%  softirqs.CPU188.RCU
    123244           -10.8%     109925        softirqs.CPU189.TIMER
    122570           -10.8%     109284        softirqs.CPU190.TIMER
    115157 ±  9%     +26.1%     145195 ±  4%  softirqs.CPU191.RCU
     38965           -48.7%      19995 ± 76%  softirqs.CPU34.SCHED
    154328           -14.6%     131808 ±  7%  softirqs.CPU50.RCU
    123075            -8.8%     112236 ±  3%  softirqs.CPU72.TIMER
     94479 ± 10%     +30.3%     123151 ±  5%  softirqs.CPU95.RCU
    125651           -11.4%     111284        softirqs.CPU95.TIMER
     31.74 ± 12%     -24.6        7.16 ± 10%  perf-profile.calltrace.cycles-pp.get_signal.do_signal.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
     31.37 ± 12%     -24.5        6.84 ± 10%  perf-profile.calltrace.cycles-pp.__sigqueue_free.__dequeue_signal.dequeue_signal.get_signal.do_signal
     31.42 ± 12%     -24.5        6.91 ± 10%  perf-profile.calltrace.cycles-pp.__dequeue_signal.dequeue_signal.get_signal.do_signal.exit_to_usermode_loop
     31.46 ± 12%     -24.5        6.97 ± 10%  perf-profile.calltrace.cycles-pp.dequeue_signal.get_signal.do_signal.exit_to_usermode_loop.do_syscall_64
     32.63 ± 12%     -24.1        8.57 ± 10%  perf-profile.calltrace.cycles-pp.do_signal.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.raise
     32.64 ± 12%     -24.1        8.59 ± 10%  perf-profile.calltrace.cycles-pp.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.raise
     29.48 ± 12%     -22.3        7.22 ± 10%  perf-profile.calltrace.cycles-pp.do_send_sig_info.do_send_specific.do_tkill.__x64_sys_tgkill.do_syscall_64
     29.19 ± 12%     -22.3        6.93 ± 10%  perf-profile.calltrace.cycles-pp.__sigqueue_alloc.__send_signal.do_send_sig_info.do_send_specific.do_tkill
     29.23 ± 12%     -22.2        7.01 ± 10%  perf-profile.calltrace.cycles-pp.__send_signal.do_send_sig_info.do_send_specific.do_tkill.__x64_sys_tgkill
     14.79 ± 11%     -14.8        0.00        perf-profile.calltrace.cycles-pp.free_uid.__sigqueue_free.__dequeue_signal.dequeue_signal.get_signal
     14.79 ± 11%     -14.8        0.00        perf-profile.calltrace.cycles-pp.refcount_dec_and_lock_irqsave.free_uid.__sigqueue_free.__dequeue_signal.dequeue_signal
     14.78 ± 11%     -14.8        0.00        perf-profile.calltrace.cycles-pp.refcount_dec_not_one.refcount_dec_and_lock_irqsave.free_uid.__sigqueue_free.__dequeue_signal
      0.39 ± 57%      +0.4        0.77 ±  9%  perf-profile.calltrace.cycles-pp.__fpu__restore_sig.__x64_sys_rt_sigreturn.do_syscall_64.entry_SYSCALL_64_after_hwframe.raise
      0.44 ± 57%      +0.4        0.84 ±  9%  perf-profile.calltrace.cycles-pp.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.42 ± 57%      +0.4        0.83 ±  9%  perf-profile.calltrace.cycles-pp.do_signal.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.87 ± 13%      +0.5        1.34 ±  8%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.89 ± 13%      +0.5        1.37 ±  8%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.49 ± 57%      +0.5        0.99 ±  9%  perf-profile.calltrace.cycles-pp.__x64_sys_rt_sigreturn.do_syscall_64.entry_SYSCALL_64_after_hwframe.raise
      0.50 ± 57%      +0.5        1.01 ±  9%  perf-profile.calltrace.cycles-pp.copy_fpstate_to_sigframe.do_signal.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +0.7        0.66 ±  8%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.raise
      0.00            +0.7        0.67 ±  9%  perf-profile.calltrace.cycles-pp.fpu__clear.do_signal.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +0.7        0.73 ±  8%  perf-profile.calltrace.cycles-pp.__x64_sys_rt_sigprocmask.do_syscall_64.entry_SYSCALL_64_after_hwframe.raise
      0.98 ± 13%     +20.0       21.00 ±  8%  perf-profile.calltrace.cycles-pp.aa_get_task_label.apparmor_task_kill.security_task_kill.do_send_specific.do_tkill
     32.51 ± 12%     +23.6       56.11 ±  8%  perf-profile.calltrace.cycles-pp.do_send_specific.do_tkill.__x64_sys_tgkill.do_syscall_64.entry_SYSCALL_64_after_hwframe
     32.73 ± 12%     +23.7       56.46 ±  8%  perf-profile.calltrace.cycles-pp.do_tkill.__x64_sys_tgkill.do_syscall_64.entry_SYSCALL_64_after_hwframe.raise
     32.75 ± 12%     +23.7       56.48 ±  8%  perf-profile.calltrace.cycles-pp.__x64_sys_tgkill.do_syscall_64.entry_SYSCALL_64_after_hwframe.raise
      2.94 ± 12%     +45.8       48.70 ±  8%  perf-profile.calltrace.cycles-pp.apparmor_task_kill.security_task_kill.do_send_specific.do_tkill.__x64_sys_tgkill
      2.95 ± 13%     +45.8       48.73 ±  8%  perf-profile.calltrace.cycles-pp.security_task_kill.do_send_specific.do_tkill.__x64_sys_tgkill.do_syscall_64
     31.77 ± 12%     -24.6        7.16 ± 10%  perf-profile.children.cycles-pp.get_signal
     31.37 ± 12%     -24.5        6.84 ± 10%  perf-profile.children.cycles-pp.__sigqueue_free
     31.42 ± 12%     -24.5        6.91 ± 10%  perf-profile.children.cycles-pp.__dequeue_signal
     31.46 ± 12%     -24.5        6.97 ± 10%  perf-profile.children.cycles-pp.dequeue_signal
     33.19 ± 12%     -23.8        9.43 ±  9%  perf-profile.children.cycles-pp.exit_to_usermode_loop
     33.16 ± 12%     -23.8        9.41 ± 10%  perf-profile.children.cycles-pp.do_signal
     29.19 ± 12%     -22.3        6.93 ± 10%  perf-profile.children.cycles-pp.__sigqueue_alloc
     29.48 ± 12%     -22.2        7.22 ± 10%  perf-profile.children.cycles-pp.do_send_sig_info
     29.24 ± 12%     -22.2        7.02 ± 10%  perf-profile.children.cycles-pp.__send_signal
     14.80 ± 11%     -14.8        0.00        perf-profile.children.cycles-pp.free_uid
     14.79 ± 11%     -14.8        0.00        perf-profile.children.cycles-pp.refcount_dec_and_lock_irqsave
     14.79 ± 11%     -14.8        0.00        perf-profile.children.cycles-pp.refcount_dec_not_one
      0.17 ± 14%      -0.1        0.09 ± 12%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.04 ± 57%      +0.0        0.07 ± 11%  perf-profile.children.cycles-pp.__x64_sys_gettid
      0.04 ± 57%      +0.0        0.08 ± 10%  perf-profile.children.cycles-pp.__clear_user
      0.07 ± 13%      +0.0        0.10 ±  8%  perf-profile.children.cycles-pp.setup_sigcontext
      0.07 ± 13%      +0.0        0.10 ±  8%  perf-profile.children.cycles-pp._copy_to_user
      0.04 ± 58%      +0.0        0.08 ± 10%  perf-profile.children.cycles-pp.__x64_sys_getpid
      0.04 ± 58%      +0.0        0.08 ± 10%  perf-profile.children.cycles-pp.__might_sleep
      0.07 ± 17%      +0.0        0.12 ± 10%  perf-profile.children.cycles-pp.copy_init_pkru_to_fpregs
      0.10 ± 14%      +0.0        0.15 ±  9%  perf-profile.children.cycles-pp.___might_sleep
      0.09 ± 14%      +0.1        0.15 ± 11%  perf-profile.children.cycles-pp.restore_altstack
      0.09 ± 14%      +0.1        0.15 ±  7%  perf-profile.children.cycles-pp.signal_setup_done
      0.07 ± 16%      +0.1        0.12 ± 15%  perf-profile.children.cycles-pp.memset_erms
      0.00            +0.1        0.06 ±  7%  perf-profile.children.cycles-pp.__local_bh_enable_ip
      0.00            +0.1        0.06 ±  7%  perf-profile.children.cycles-pp.__lock_task_sighand
      0.07 ± 17%      +0.1        0.13 ± 14%  perf-profile.children.cycles-pp.fpstate_init
      0.00            +0.1        0.06 ± 13%  perf-profile.children.cycles-pp.recalc_sigpending_tsk
      0.01 ±173%      +0.1        0.08 ±  8%  perf-profile.children.cycles-pp.clockevents_program_event
      0.00            +0.1        0.07 ± 22%  perf-profile.children.cycles-pp.__radix_tree_lookup
      0.12 ± 14%      +0.1        0.20 ± 10%  perf-profile.children.cycles-pp.__task_pid_nr_ns
      0.00            +0.1        0.08 ±  6%  perf-profile.children.cycles-pp.ktime_get
      0.00            +0.1        0.08 ± 23%  perf-profile.children.cycles-pp.find_task_by_vpid
      0.14 ± 13%      +0.1        0.23 ±  8%  perf-profile.children.cycles-pp.recalc_sigpending
      0.16 ± 15%      +0.1        0.26 ±  7%  perf-profile.children.cycles-pp.__set_task_blocked
      0.17 ± 14%      +0.1        0.28 ±  9%  perf-profile.children.cycles-pp.sigprocmask
      0.18 ± 13%      +0.1        0.29 ±  8%  perf-profile.children.cycles-pp.copy_user_generic_unrolled
      0.17 ± 14%      +0.1        0.29 ±  8%  perf-profile.children.cycles-pp.__might_fault
      0.21 ± 12%      +0.1        0.35 ±  9%  perf-profile.children.cycles-pp._copy_from_user
      0.32 ± 13%      +0.2        0.51 ±  7%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.31 ± 14%      +0.2        0.51 ±  7%  perf-profile.children.cycles-pp.__set_current_blocked
      0.29 ± 11%      +0.2        0.51 ±  8%  perf-profile.children.cycles-pp.aa_may_signal
      0.42 ± 13%      +0.2        0.67 ±  9%  perf-profile.children.cycles-pp.fpu__clear
      0.45 ± 11%      +0.3        0.72 ±  9%  perf-profile.children.cycles-pp.entry_SYSCALL_64
      0.45 ± 14%      +0.3        0.74 ±  8%  perf-profile.children.cycles-pp.__x64_sys_rt_sigprocmask
      0.47 ± 14%      +0.3        0.77 ±  9%  perf-profile.children.cycles-pp.__fpu__restore_sig
      0.63 ± 13%      +0.4        1.02 ±  9%  perf-profile.children.cycles-pp.copy_fpstate_to_sigframe
      0.88 ± 14%      +0.6        1.44 ±  8%  perf-profile.children.cycles-pp.__x64_sys_rt_sigreturn
      0.93 ± 13%      +0.8        1.74 ± 10%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.98 ± 13%     +20.0       21.02 ±  8%  perf-profile.children.cycles-pp.aa_get_task_label
     32.51 ± 12%     +23.6       56.12 ±  8%  perf-profile.children.cycles-pp.do_send_specific
     32.75 ± 12%     +23.7       56.49 ±  8%  perf-profile.children.cycles-pp.__x64_sys_tgkill
     32.73 ± 12%     +23.7       56.46 ±  8%  perf-profile.children.cycles-pp.do_tkill
      2.94 ± 12%     +45.8       48.73 ±  8%  perf-profile.children.cycles-pp.apparmor_task_kill
      2.95 ± 13%     +45.8       48.74 ±  8%  perf-profile.children.cycles-pp.security_task_kill
     29.16 ± 12%     -22.3        6.89 ± 10%  perf-profile.self.cycles-pp.__sigqueue_alloc
     14.79 ± 11%     -14.8        0.00        perf-profile.self.cycles-pp.refcount_dec_not_one
     16.58 ± 12%      -9.7        6.84 ± 10%  perf-profile.self.cycles-pp.__sigqueue_free
      0.07 ± 10%      +0.0        0.09 ±  7%  perf-profile.self.cycles-pp.fpregs_mark_activate
      0.04 ± 58%      +0.0        0.07 ± 12%  perf-profile.self.cycles-pp.__set_task_blocked
      0.04 ± 57%      +0.0        0.07 ± 11%  perf-profile.self.cycles-pp.do_tkill
      0.04 ± 57%      +0.0        0.08 ±  8%  perf-profile.self.cycles-pp.__clear_user
      0.07 ± 16%      +0.0        0.10 ±  8%  perf-profile.self.cycles-pp.__x64_sys_rt_sigprocmask
      0.04 ± 57%      +0.0        0.07 ± 11%  perf-profile.self.cycles-pp.__might_sleep
      0.07 ± 13%      +0.0        0.10 ±  8%  perf-profile.self.cycles-pp.setup_sigcontext
      0.04 ± 57%      +0.0        0.08 ± 10%  perf-profile.self.cycles-pp.copy_init_pkru_to_fpregs
      0.03 ±100%      +0.0        0.07 ± 11%  perf-profile.self.cycles-pp.__set_current_blocked
      0.10 ± 14%      +0.0        0.15 ±  9%  perf-profile.self.cycles-pp.___might_sleep
      0.07 ± 16%      +0.0        0.12 ± 13%  perf-profile.self.cycles-pp.memset_erms
      0.10 ± 12%      +0.1        0.16 ± 10%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.10 ± 12%      +0.1        0.16 ± 10%  perf-profile.self.cycles-pp.recalc_sigpending
      0.01 ±173%      +0.1        0.07 ± 11%  perf-profile.self.cycles-pp.__might_fault
      0.00            +0.1        0.06 ± 13%  perf-profile.self.cycles-pp.recalc_sigpending_tsk
      0.12 ± 13%      +0.1        0.19 ±  8%  perf-profile.self.cycles-pp.__task_pid_nr_ns
      0.00            +0.1        0.07 ± 10%  perf-profile.self.cycles-pp.ktime_get
      0.00            +0.1        0.07 ± 22%  perf-profile.self.cycles-pp.__radix_tree_lookup
      0.12 ± 14%      +0.1        0.20 ±  9%  perf-profile.self.cycles-pp.do_syscall_64
      0.15 ± 16%      +0.1        0.25 ±  9%  perf-profile.self.cycles-pp.do_signal
      0.17 ± 14%      +0.1        0.28 ±  8%  perf-profile.self.cycles-pp.copy_user_generic_unrolled
      0.20 ± 14%      +0.1        0.33 ±  7%  perf-profile.self.cycles-pp.__x64_sys_rt_sigreturn
      0.22 ± 11%      +0.1        0.36 ±  8%  perf-profile.self.cycles-pp.fpu__clear
      0.25 ± 11%      +0.1        0.39 ±  9%  perf-profile.self.cycles-pp.entry_SYSCALL_64
      0.32 ± 13%      +0.2        0.51 ±  7%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.29 ± 12%      +0.2        0.48 ±  8%  perf-profile.self.cycles-pp.raise
      0.29 ± 11%      +0.2        0.50 ±  9%  perf-profile.self.cycles-pp.aa_may_signal
      0.37 ± 15%      +0.2        0.60 ±  9%  perf-profile.self.cycles-pp.__fpu__restore_sig
      0.50 ± 14%      +0.3        0.81 ±  9%  perf-profile.self.cycles-pp.copy_fpstate_to_sigframe
      0.93 ± 13%      +0.8        1.74 ± 10%  perf-profile.self.cycles-pp.native_irq_return_iret
      0.98 ± 13%     +19.9       20.90 ±  8%  perf-profile.self.cycles-pp.aa_get_task_label
      1.67 ± 13%     +25.4       27.05 ±  9%  perf-profile.self.cycles-pp.apparmor_task_kill


                                                                                
                            will-it-scale.per_process_ops                       
                                                                                
  75000 +-------------------------------------------------------------------+   
        |                              O                                    |   
  70000 |-+O  O  O  O  O   O  O  O  O     O  O  O  O  O  O   O  O  O  O  O  |   
        |                                                                   |   
  65000 |-+                                                                 |   
        |                                                                   |   
  60000 |-+                                                                 |   
        |                                                                   |   
  55000 |-+                                                                 |   
        |                                                                   |   
  50000 |-+                                                                 |   
        |                                                                   |   
  45000 |..+..+..+..+..+...+..+..+                                          |   
        |                                                                   |   
  40000 +-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                will-it-scale.workload                          
                                                                                
    7e+06 +-----------------------------------------------------------------+   
          |  O  O  O  O  O  O  O  O  O  O  O  O  O  O  O     O  O  O  O  O  |   
  6.5e+06 |-+                                             O                 |   
          |                                                                 |   
          |                                                                 |   
    6e+06 |-+                                                               |   
          |                                                                 |   
  5.5e+06 |-+                                                               |   
          |                                                                 |   
    5e+06 |-+                                                               |   
          |                                                                 |   
          |                                                                 |   
  4.5e+06 |-+                                                               |   
          |..+..+..+..+..+..+..+..+                                         |   
    4e+06 +-----------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample

***************************************************************************************************
lkp-skl-2sp7: 72 threads Intel(R) Xeon(R) Gold 6139 CPU @ 2.30GHz with 128G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode:
  gcc-7/performance/x86_64-rhel-7.6/process/100%/debian-x86_64-20191114.cgz/lkp-skl-2sp7/signal1/will-it-scale/0x2000065

commit: 
  c5f8689118 (" RISC-V Fixes for 5.6-rc4")
  fda31c5029 ("signal: avoid double atomic counter increments for user accounting")

c5f86891185c408b fda31c50292a5062332fa0343c0 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
     54904          +110.2%     115404        will-it-scale.per_process_ops
   3953154          +110.2%    8309119        will-it-scale.workload
   1010582 ±  5%     -11.4%     895562 ±  8%  numa-meminfo.node0.MemUsed
     78.89            -8.5       70.34        mpstat.cpu.all.sys%
     20.75            +8.5       29.21        mpstat.cpu.all.usr%
      1419           +13.1%       1604 ±  6%  slabinfo.skbuff_ext_cache.active_objs
      1419           +13.1%       1604 ±  6%  slabinfo.skbuff_ext_cache.num_objs
     78.00           -10.6%      69.75        vmstat.cpu.sy
     20.00           +42.5%      28.50        vmstat.cpu.us
      1495            +3.5%       1547        vmstat.system.cs
    118.24 ±  3%      -8.0%     108.76        sched_debug.cfs_rq:/.util_est_enqueued.stddev
      1853 ±  7%     -14.5%       1583 ±  4%  sched_debug.cpu.nr_switches.min
      1009 ±  4%     -36.5%     641.25 ±  7%  sched_debug.cpu.sched_count.min
     64.12 ±  5%     -12.0%      56.43 ±  5%  sched_debug.cpu.sched_goidle.avg
    408.00 ±  3%     -37.2%     256.12 ± 10%  sched_debug.cpu.ttwu_count.min
    400.17 ±  3%     -41.6%     233.79 ± 10%  sched_debug.cpu.ttwu_local.min
    677.25 ± 51%     -50.5%     335.50 ±  3%  interrupts.CPU21.RES:Rescheduling_interrupts
    338.50 ±  5%      -5.8%     318.75        interrupts.CPU40.RES:Rescheduling_interrupts
    458.00 ± 24%     -30.1%     320.25 ±  2%  interrupts.CPU44.RES:Rescheduling_interrupts
      6240 ± 31%     +39.6%       8711        interrupts.CPU55.NMI:Non-maskable_interrupts
      6240 ± 31%     +39.6%       8711        interrupts.CPU55.PMI:Performance_monitoring_interrupts
      6238 ± 31%     +39.6%       8709        interrupts.CPU56.NMI:Non-maskable_interrupts
      6238 ± 31%     +39.6%       8709        interrupts.CPU56.PMI:Performance_monitoring_interrupts
      6238 ± 31%     +39.6%       8710        interrupts.CPU58.NMI:Non-maskable_interrupts
      6238 ± 31%     +39.6%       8710        interrupts.CPU58.PMI:Performance_monitoring_interrupts
    445.25 ± 32%     -25.7%     331.00 ±  8%  interrupts.CPU58.RES:Rescheduling_interrupts
      6236 ± 31%     +39.7%       8709        interrupts.CPU59.NMI:Non-maskable_interrupts
      6236 ± 31%     +39.7%       8709        interrupts.CPU59.PMI:Performance_monitoring_interrupts
      6238 ± 31%     +39.6%       8711        interrupts.CPU60.NMI:Non-maskable_interrupts
      6238 ± 31%     +39.6%       8711        interrupts.CPU60.PMI:Performance_monitoring_interrupts
      6241 ± 31%     +39.6%       8714        interrupts.CPU61.NMI:Non-maskable_interrupts
      6241 ± 31%     +39.6%       8714        interrupts.CPU61.PMI:Performance_monitoring_interrupts
      4.55           -26.4%       3.35        perf-stat.i.MPKI
 3.894e+09          +103.0%  7.903e+09        perf-stat.i.branch-instructions
      1.02            -0.1        0.91        perf-stat.i.branch-miss-rate%
  39579112           +78.7%   70734489        perf-stat.i.branch-misses
     36.65            +3.0       39.65        perf-stat.i.cache-miss-rate%
  32774536 ±  2%     +62.7%   53311773        perf-stat.i.cache-misses
  89326733 ±  2%     +50.3%  1.343e+08        perf-stat.i.cache-references
      1468            +3.7%       1523        perf-stat.i.context-switches
     11.23           -51.4%       5.46        perf-stat.i.cpi
    103.83            -6.2%      97.37 ±  2%  perf-stat.i.cpu-migrations
      6752 ±  2%     -38.9%       4126        perf-stat.i.cycles-between-cache-misses
      0.39            +0.0        0.40        perf-stat.i.dTLB-load-miss-rate%
  23680840          +110.0%   49731808        perf-stat.i.dTLB-load-misses
 5.968e+09          +105.5%  1.227e+10        perf-stat.i.dTLB-loads
      0.00 ±  5%      -0.0        0.00 ± 12%  perf-stat.i.dTLB-store-miss-rate%
 3.636e+09          +105.3%  7.465e+09        perf-stat.i.dTLB-stores
     97.67           -12.4       85.31        perf-stat.i.iTLB-load-miss-rate%
  25427480          +109.6%   53290587        perf-stat.i.iTLB-load-misses
    557904 ± 25%   +1536.0%    9127411 ±  7%  perf-stat.i.iTLB-loads
 1.965e+10          +104.7%  4.022e+10        perf-stat.i.instructions
    797.76            -3.2%     772.24        perf-stat.i.instructions-per-iTLB-miss
      0.09          +103.8%       0.18        perf-stat.i.ipc
     87.09            -2.8       84.28        perf-stat.i.node-load-miss-rate%
   2494881           +39.3%    3476540        perf-stat.i.node-load-misses
    369358           +75.2%     647260        perf-stat.i.node-loads
   6092757           +50.1%    9144245        perf-stat.i.node-store-misses
      4.55           -26.6%       3.34        perf-stat.overall.MPKI
      1.02            -0.1        0.90        perf-stat.overall.branch-miss-rate%
     36.69            +3.0       39.71        perf-stat.overall.cache-miss-rate%
     11.24           -51.4%       5.47        perf-stat.overall.cpi
      6743 ±  2%     -38.8%       4126        perf-stat.overall.cycles-between-cache-misses
      0.40            +0.0        0.40        perf-stat.overall.dTLB-load-miss-rate%
      0.00            -0.0        0.00 ±  2%  perf-stat.overall.dTLB-store-miss-rate%
     97.86           -12.5       85.39        perf-stat.overall.iTLB-load-miss-rate%
    772.79            -2.3%     754.70        perf-stat.overall.instructions-per-iTLB-miss
      0.09          +105.6%       0.18        perf-stat.overall.ipc
     87.08            -2.8       84.29        perf-stat.overall.node-load-miss-rate%
   1496927            -2.7%    1457192        perf-stat.overall.path-length
 3.881e+09          +102.9%  7.877e+09        perf-stat.ps.branch-instructions
  39447391           +78.7%   70499455        perf-stat.ps.branch-misses
  32665932 ±  2%     +62.7%   53131306        perf-stat.ps.cache-misses
  89041613 ±  2%     +50.3%  1.338e+08        perf-stat.ps.cache-references
      1463            +3.8%       1518        perf-stat.ps.context-switches
    103.58            -6.2%      97.14 ±  2%  perf-stat.ps.cpu-migrations
  23603337          +110.0%   49565150        perf-stat.ps.dTLB-load-misses
 5.949e+09          +105.5%  1.223e+10        perf-stat.ps.dTLB-loads
 3.624e+09          +105.3%   7.44e+09        perf-stat.ps.dTLB-stores
  25342778          +109.6%   53110947        perf-stat.ps.iTLB-load-misses
    556203 ± 25%   +1535.4%    9096275 ±  7%  perf-stat.ps.iTLB-loads
 1.958e+10          +104.7%  4.008e+10        perf-stat.ps.instructions
   2486652           +39.3%    3464791        perf-stat.ps.node-load-misses
    368954           +75.1%     645988        perf-stat.ps.node-loads
   6072500           +50.1%    9113393        perf-stat.ps.node-store-misses
 5.918e+12          +104.6%  1.211e+13        perf-stat.total.instructions
     94878 ±  5%     +33.6%     126748 ±  4%  softirqs.CPU0.RCU
    113937            -9.6%     102988        softirqs.CPU0.TIMER
     94125 ±  3%     +33.9%     126058 ±  4%  softirqs.CPU1.RCU
    112762           -11.3%     100071        softirqs.CPU1.TIMER
     94771 ±  3%     +32.5%     125590 ±  3%  softirqs.CPU10.RCU
    112676           -11.4%      99858        softirqs.CPU10.TIMER
     94400 ±  3%     +32.5%     125097 ±  3%  softirqs.CPU11.RCU
    111102           -10.5%      99464        softirqs.CPU11.TIMER
     93242 ±  4%     +35.3%     126147 ±  5%  softirqs.CPU12.RCU
    112269           -11.3%      99636        softirqs.CPU12.TIMER
     93727 ±  4%     +33.4%     125048 ±  4%  softirqs.CPU13.RCU
    112538           -11.9%      99188        softirqs.CPU13.TIMER
     94061 ±  3%     +33.5%     125571 ±  4%  softirqs.CPU14.RCU
    112794           -11.6%      99676        softirqs.CPU14.TIMER
     94539 ±  3%     +32.9%     125633 ±  4%  softirqs.CPU15.RCU
    112213           -11.1%      99752        softirqs.CPU15.TIMER
     94430 ±  4%     +32.3%     124930 ±  3%  softirqs.CPU16.RCU
    112073           -10.9%      99806        softirqs.CPU16.TIMER
     93861 ±  4%     +32.9%     124749 ±  4%  softirqs.CPU17.RCU
    111659           -11.4%      98981        softirqs.CPU17.TIMER
     93831 ±  4%     +34.3%     126054 ±  3%  softirqs.CPU18.RCU
    124477 ± 16%     -18.1%     101983        softirqs.CPU18.TIMER
     93028 ±  4%     +34.9%     125540 ±  3%  softirqs.CPU19.RCU
    112566            -9.9%     101377        softirqs.CPU19.TIMER
     94813 ±  3%     +31.8%     124972 ±  3%  softirqs.CPU2.RCU
    112172           -11.3%      99445        softirqs.CPU2.TIMER
     95150 ±  7%     +38.5%     131782 ±  4%  softirqs.CPU20.RCU
    114632 ±  2%      -8.9%     104389        softirqs.CPU20.TIMER
     99411           +26.2%     125474 ±  4%  softirqs.CPU21.RCU
    114529           -10.6%     102342        softirqs.CPU21.TIMER
     94332 ±  4%     +33.7%     126113 ±  3%  softirqs.CPU22.RCU
     93175 ±  4%     +34.7%     125490 ±  4%  softirqs.CPU23.RCU
    112469            -9.5%     101759        softirqs.CPU23.TIMER
     94210 ±  4%     +34.4%     126617 ±  4%  softirqs.CPU24.RCU
     94144 ±  4%     +34.0%     126146 ±  4%  softirqs.CPU25.RCU
     93241 ±  4%     +34.3%     125256 ±  3%  softirqs.CPU26.RCU
     93358 ±  4%     +34.4%     125512 ±  3%  softirqs.CPU27.RCU
    112502            -9.6%     101651        softirqs.CPU27.TIMER
     92663 ±  4%     +35.1%     125162 ±  4%  softirqs.CPU28.RCU
    112470           -10.5%     100680        softirqs.CPU28.TIMER
     94244 ±  3%     +33.6%     125887 ±  3%  softirqs.CPU29.RCU
     93663 ±  3%     +33.6%     125127 ±  3%  softirqs.CPU3.RCU
    112367           -11.1%      99922        softirqs.CPU3.TIMER
     93149 ±  4%     +34.3%     125061 ±  4%  softirqs.CPU30.RCU
    112546            -9.9%     101363        softirqs.CPU30.TIMER
     93175 ±  3%     +34.6%     125411 ±  3%  softirqs.CPU31.RCU
    112412           -10.3%     100802        softirqs.CPU31.TIMER
     93824 ±  3%     +34.6%     126289 ±  2%  softirqs.CPU32.RCU
    112925           -10.5%     101084        softirqs.CPU32.TIMER
     93606 ±  4%     +35.2%     126601 ±  4%  softirqs.CPU33.RCU
     95943 ±  3%     +31.5%     126185 ±  4%  softirqs.CPU34.RCU
     94187 ±  4%     +34.8%     126990 ±  4%  softirqs.CPU35.RCU
    112148            -9.2%     101778        softirqs.CPU35.TIMER
     94392 ±  4%     +32.8%     125321 ±  4%  softirqs.CPU36.RCU
    112299           -11.8%      99078        softirqs.CPU36.TIMER
     94411 ±  4%     +32.6%     125155 ±  4%  softirqs.CPU37.RCU
    112362           -11.7%      99222        softirqs.CPU37.TIMER
     94535 ±  2%     +32.1%     124871 ±  3%  softirqs.CPU38.RCU
    111602           -11.2%      99125        softirqs.CPU38.TIMER
     93738 ±  3%     +33.2%     124879 ±  4%  softirqs.CPU39.RCU
    111940           -11.4%      99173        softirqs.CPU39.TIMER
     93883 ±  3%     +33.4%     125217 ±  3%  softirqs.CPU4.RCU
    112271           -11.6%      99260        softirqs.CPU4.TIMER
     93664 ±  3%     +33.5%     125001 ±  4%  softirqs.CPU40.RCU
    111796           -11.6%      98876        softirqs.CPU40.TIMER
     95132 ±  3%     +32.5%     126040 ±  4%  softirqs.CPU41.RCU
    112010           -11.3%      99407        softirqs.CPU41.TIMER
     93600 ±  3%     +35.0%     126337 ±  3%  softirqs.CPU42.RCU
    111813           -10.5%     100023        softirqs.CPU42.TIMER
     96063 ±  5%     +30.2%     125067 ±  3%  softirqs.CPU43.RCU
    110609           -10.4%      99063        softirqs.CPU43.TIMER
     94699 ±  4%     +31.7%     124702 ±  3%  softirqs.CPU44.RCU
    112163           -10.9%      99906        softirqs.CPU44.TIMER
     94374 ±  3%     +33.2%     125687 ±  4%  softirqs.CPU45.RCU
    111496           -10.3%     100047        softirqs.CPU45.TIMER
     95463 ±  4%     +32.0%     125980 ±  4%  softirqs.CPU46.RCU
    113314           -11.7%     100022        softirqs.CPU46.TIMER
     94648 ±  4%     +32.4%     125320 ±  4%  softirqs.CPU47.RCU
    122901 ± 16%     -18.8%      99738        softirqs.CPU47.TIMER
     93679 ±  4%     +35.4%     126887 ±  5%  softirqs.CPU48.RCU
    112180           -10.8%     100079        softirqs.CPU48.TIMER
     94101 ±  4%     +32.9%     125025 ±  4%  softirqs.CPU49.RCU
    112337           -12.0%      98866        softirqs.CPU49.TIMER
     95279 ±  3%     +32.8%     126546 ±  5%  softirqs.CPU5.RCU
    112079           -11.0%      99774        softirqs.CPU5.TIMER
     94319 ±  4%     +32.4%     124919 ±  4%  softirqs.CPU50.RCU
    112444           -11.7%      99317        softirqs.CPU50.TIMER
     94101 ±  4%     +32.9%     125091 ±  4%  softirqs.CPU51.RCU
    111990           -11.4%      99206        softirqs.CPU51.TIMER
     94527 ±  4%     +32.4%     125115 ±  4%  softirqs.CPU52.RCU
    112146           -11.0%      99785        softirqs.CPU52.TIMER
     94674 ±  3%     +32.2%     125117 ±  4%  softirqs.CPU53.RCU
    112652           -12.2%      98956        softirqs.CPU53.TIMER
     93587 ±  4%     +33.7%     125123 ±  4%  softirqs.CPU54.RCU
    124229 ± 15%     -18.6%     101165        softirqs.CPU54.TIMER
     93475 ±  4%     +33.8%     125045 ±  3%  softirqs.CPU55.RCU
    112560           -10.7%     100563        softirqs.CPU55.TIMER
     93442 ±  4%     +33.5%     124770 ±  3%  softirqs.CPU56.RCU
     95943 ±  3%     +30.5%     125208 ±  3%  softirqs.CPU57.RCU
    111885            -9.1%     101710        softirqs.CPU57.TIMER
     93661 ±  4%     +33.4%     124988 ±  3%  softirqs.CPU58.RCU
     93550 ±  4%     +34.1%     125483 ±  3%  softirqs.CPU59.RCU
    112478            -9.6%     101716        softirqs.CPU59.TIMER
     94064 ±  3%     +34.3%     126351 ±  4%  softirqs.CPU6.RCU
    112367           -10.8%     100261        softirqs.CPU6.TIMER
     93270 ±  3%     +33.9%     124934 ±  3%  softirqs.CPU60.RCU
    112103           -10.3%     100558        softirqs.CPU60.TIMER
     93206 ±  4%     +34.5%     125334 ±  3%  softirqs.CPU61.RCU
     92573 ±  4%     +34.8%     124784 ±  3%  softirqs.CPU62.RCU
     93028 ±  4%     +34.8%     125360 ±  3%  softirqs.CPU63.RCU
     92747 ±  5%     +34.7%     124938 ±  3%  softirqs.CPU64.RCU
    112226           -10.5%     100479        softirqs.CPU64.TIMER
     94153 ±  4%     +32.9%     125098 ±  3%  softirqs.CPU65.RCU
    110969            -9.0%     100953        softirqs.CPU65.TIMER
     93058 ±  4%     +34.0%     124731 ±  3%  softirqs.CPU66.RCU
    112302            -9.9%     101140        softirqs.CPU66.TIMER
     93859 ±  5%     +33.8%     125587 ±  4%  softirqs.CPU67.RCU
    113034           -10.7%     100984        softirqs.CPU67.TIMER
     93164 ±  4%     +34.8%     125599 ±  2%  softirqs.CPU68.RCU
    112064            -9.9%     100932        softirqs.CPU68.TIMER
     93580 ±  5%     +34.0%     125441 ±  4%  softirqs.CPU69.RCU
     96134 ±  4%     +30.2%     125191 ±  3%  softirqs.CPU7.RCU
    123181 ± 16%     -19.5%      99220        softirqs.CPU7.TIMER
     94940 ±  4%     +31.0%     124398 ±  4%  softirqs.CPU70.RCU
    111495           -10.0%     100395        softirqs.CPU70.TIMER
     94607 ±  5%     +33.1%     125955 ±  3%  softirqs.CPU71.RCU
     95153 ±  4%     +31.2%     124844 ±  4%  softirqs.CPU8.RCU
    113482           -11.7%     100224        softirqs.CPU8.TIMER
     94211 ±  3%     +33.8%     126073 ±  4%  softirqs.CPU9.RCU
    111974           -10.7%      99948        softirqs.CPU9.TIMER
   6779912 ±  4%     +33.3%    9040799 ±  4%  softirqs.RCU
   8127301           -10.2%    7296878        softirqs.TIMER
     33.47           -26.4        7.04 ±  2%  perf-profile.calltrace.cycles-pp.get_signal.do_signal.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
     31.97           -25.8        6.12 ±  2%  perf-profile.calltrace.cycles-pp.dequeue_signal.get_signal.do_signal.exit_to_usermode_loop.do_syscall_64
     35.42           -24.4       11.03        perf-profile.calltrace.cycles-pp.do_signal.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.raise
     35.45           -24.4       11.08        perf-profile.calltrace.cycles-pp.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.raise
     29.81           -24.3        5.50 ±  2%  perf-profile.calltrace.cycles-pp.__sigqueue_free.__dequeue_signal.dequeue_signal.get_signal.do_signal
     30.02           -24.2        5.77 ±  2%  perf-profile.calltrace.cycles-pp.__dequeue_signal.dequeue_signal.get_signal.do_signal.exit_to_usermode_loop
     20.11           -20.1        0.00        perf-profile.calltrace.cycles-pp.free_uid.__sigqueue_free.__dequeue_signal.dequeue_signal.get_signal
     20.09           -20.1        0.00        perf-profile.calltrace.cycles-pp.refcount_dec_and_lock_irqsave.free_uid.__sigqueue_free.__dequeue_signal.dequeue_signal
     20.07           -20.1        0.00        perf-profile.calltrace.cycles-pp.refcount_dec_not_one.refcount_dec_and_lock_irqsave.free_uid.__sigqueue_free.__dequeue_signal
     22.28           -14.3        7.99 ±  2%  perf-profile.calltrace.cycles-pp.do_send_sig_info.do_send_specific.do_tkill.__x64_sys_tgkill.do_syscall_64
     20.35           -13.9        6.50 ±  2%  perf-profile.calltrace.cycles-pp.__sigqueue_alloc.__send_signal.do_send_sig_info.do_send_specific.do_tkill
     20.56           -13.7        6.87 ±  2%  perf-profile.calltrace.cycles-pp.__send_signal.do_send_sig_info.do_send_specific.do_tkill.__x64_sys_tgkill
     82.09           -12.4       69.66        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.raise
     82.31           -12.2       70.14        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.raise
     89.83            -2.2       87.61        perf-profile.calltrace.cycles-pp.raise
      4.38            -0.9        3.51        perf-profile.calltrace.cycles-pp.do_signal.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
      4.42            -0.9        3.55        perf-profile.calltrace.cycles-pp.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.36 ±  2%      -0.8        0.55        perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.__lock_task_sighand.do_send_sig_info.do_send_specific.do_tkill
      1.37 ±  2%      -0.8        0.58        perf-profile.calltrace.cycles-pp.__lock_task_sighand.do_send_sig_info.do_send_specific.do_tkill.__x64_sys_tgkill
      1.18            -0.8        0.40 ± 57%  perf-profile.calltrace.cycles-pp.signal_setup_done.do_signal.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.19 ±  4%      -0.7        0.54 ±  3%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irq.get_signal.do_signal.exit_to_usermode_loop.do_syscall_64
      3.18            -0.2        2.93        perf-profile.calltrace.cycles-pp.fpu__clear.do_signal.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.17 ±  2%      -0.2        0.93 ±  5%  perf-profile.calltrace.cycles-pp.fpregs_mark_activate.fpu__clear.do_signal.exit_to_usermode_loop.do_syscall_64
      7.57            +0.1        7.70        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.03 ±  3%      +0.1        1.16 ±  5%  perf-profile.calltrace.cycles-pp.recalc_sigpending_tsk.recalc_sigpending.__set_task_blocked.__set_current_blocked.sigprocmask
      7.62            +0.2        7.80        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.52            +0.6        1.09        perf-profile.calltrace.cycles-pp._copy_from_user.__x64_sys_rt_sigprocmask.do_syscall_64.entry_SYSCALL_64_after_hwframe.raise
      2.22            +0.6        2.80        perf-profile.calltrace.cycles-pp.__fpu__restore_sig.__x64_sys_rt_sigreturn.do_syscall_64.entry_SYSCALL_64_after_hwframe.raise
      0.75            +0.7        1.41        perf-profile.calltrace.cycles-pp.swapgs_restore_regs_and_return_to_usermode
      0.87            +0.8        1.71        perf-profile.calltrace.cycles-pp.__x64_sys_rt_sigreturn.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.62            +1.0        1.58        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64
      0.56 ±  2%      +1.0        1.58        perf-profile.calltrace.cycles-pp.aa_may_signal.apparmor_task_kill.security_task_kill.do_send_specific.do_tkill
      2.58            +1.0        3.62        perf-profile.calltrace.cycles-pp.__x64_sys_rt_sigreturn.do_syscall_64.entry_SYSCALL_64_after_hwframe.raise
      2.80 ±  2%      +1.3        4.10        perf-profile.calltrace.cycles-pp.__x64_sys_rt_sigprocmask.do_syscall_64.entry_SYSCALL_64_after_hwframe.raise
      1.34 ±  2%      +1.4        2.78        perf-profile.calltrace.cycles-pp.copy_fpstate_to_sigframe.do_signal.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
      3.59            +4.6        8.17        perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.raise
      3.72            +5.2        8.92        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.raise
     27.51            +7.0       34.49        perf-profile.calltrace.cycles-pp.do_send_specific.do_tkill.__x64_sys_tgkill.do_syscall_64.entry_SYSCALL_64_after_hwframe
     27.80            +7.3       35.09        perf-profile.calltrace.cycles-pp.do_tkill.__x64_sys_tgkill.do_syscall_64.entry_SYSCALL_64_after_hwframe.raise
     27.86            +7.4       35.22        perf-profile.calltrace.cycles-pp.__x64_sys_tgkill.do_syscall_64.entry_SYSCALL_64_after_hwframe.raise
      1.08            +9.4       10.53        perf-profile.calltrace.cycles-pp.aa_get_task_label.apparmor_task_kill.security_task_kill.do_send_specific.do_tkill
      4.91           +20.8       25.71        perf-profile.calltrace.cycles-pp.apparmor_task_kill.security_task_kill.do_send_specific.do_tkill.__x64_sys_tgkill
      4.94           +20.9       25.81        perf-profile.calltrace.cycles-pp.security_task_kill.do_send_specific.do_tkill.__x64_sys_tgkill.do_syscall_64
     33.49           -26.4        7.06 ±  2%  perf-profile.children.cycles-pp.get_signal
     31.97           -25.8        6.13 ±  2%  perf-profile.children.cycles-pp.dequeue_signal
     39.82           -25.3       14.56        perf-profile.children.cycles-pp.do_signal
     39.87           -25.2       14.63        perf-profile.children.cycles-pp.exit_to_usermode_loop
     29.82           -24.3        5.50 ±  2%  perf-profile.children.cycles-pp.__sigqueue_free
     30.02           -24.2        5.78 ±  2%  perf-profile.children.cycles-pp.__dequeue_signal
     20.11           -20.1        0.00        perf-profile.children.cycles-pp.free_uid
     20.09           -20.1        0.00        perf-profile.children.cycles-pp.refcount_dec_and_lock_irqsave
     20.08           -20.1        0.00        perf-profile.children.cycles-pp.refcount_dec_not_one
     22.29           -14.3        8.00 ±  2%  perf-profile.children.cycles-pp.do_send_sig_info
     20.35           -13.8        6.50 ±  2%  perf-profile.children.cycles-pp.__sigqueue_alloc
     20.57           -13.7        6.89 ±  2%  perf-profile.children.cycles-pp.__send_signal
     89.75           -12.3       77.47        perf-profile.children.cycles-pp.do_syscall_64
     90.00           -12.0       78.00        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     90.24            -2.6       87.68        perf-profile.children.cycles-pp.raise
      3.73            -1.8        1.94 ±  2%  perf-profile.children.cycles-pp.recalc_sigpending
      2.57 ±  3%      -1.2        1.32 ±  2%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      1.37 ±  2%      -0.8        0.58        perf-profile.children.cycles-pp.__lock_task_sighand
      1.43 ±  2%      -0.7        0.76 ±  3%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      1.18            -0.7        0.52 ±  4%  perf-profile.children.cycles-pp.signal_setup_done
      3.34 ±  3%      -0.6        2.74        perf-profile.children.cycles-pp.__set_current_blocked
      1.57            -0.5        1.09 ±  4%  perf-profile.children.cycles-pp.fpregs_mark_activate
      3.19            -0.2        2.95        perf-profile.children.cycles-pp.fpu__clear
      0.10            -0.0        0.09 ±  4%  perf-profile.children.cycles-pp.kmem_cache_free
      0.07 ±  5%      +0.0        0.09 ±  4%  perf-profile.children.cycles-pp.task_tick_fair
      0.09 ±  4%      +0.0        0.11 ±  4%  perf-profile.children.cycles-pp.scheduler_tick
      0.12 ±  5%      +0.0        0.16 ±  2%  perf-profile.children.cycles-pp.tick_sched_handle
      0.12 ±  5%      +0.0        0.16 ±  2%  perf-profile.children.cycles-pp.update_process_times
      0.06 ± 14%      +0.0        0.10 ±  4%  perf-profile.children.cycles-pp.clockevents_program_event
      0.18 ±  4%      +0.0        0.23 ±  2%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.14 ±  7%      +0.0        0.18 ±  2%  perf-profile.children.cycles-pp.tick_sched_timer
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.fpu__alloc_mathframe
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.complete_signal
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.do_sigaltstack
      0.32 ±  4%      +0.1        0.37 ±  2%  perf-profile.children.cycles-pp.hrtimer_interrupt
      0.00            +0.1        0.06 ± 15%  perf-profile.children.cycles-pp._cond_resched
      0.00            +0.1        0.06 ±  7%  perf-profile.children.cycles-pp.fpu__restore_sig
      0.05            +0.1        0.11 ±  4%  perf-profile.children.cycles-pp.kmem_cache_alloc
      0.03 ±100%      +0.1        0.09 ±  4%  perf-profile.children.cycles-pp.ktime_get
      0.03 ±100%      +0.1        0.09 ±  4%  perf-profile.children.cycles-pp.check_kill_permission
      0.11 ±  7%      +0.1        0.18        perf-profile.children.cycles-pp.send_signal
      0.07            +0.1        0.14 ±  3%  perf-profile.children.cycles-pp.__indirect_thunk_start
      0.20 ±  3%      +0.1        0.27 ±  7%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.04 ± 57%      +0.1        0.11 ±  8%  perf-profile.children.cycles-pp.prepare_signal
      0.09 ±  4%      +0.1        0.19 ±  2%  perf-profile.children.cycles-pp.__local_bh_enable_ip
      0.14 ±  3%      +0.1        0.25 ±  2%  perf-profile.children.cycles-pp.fpregs_assert_state_consistent
      0.14 ± 11%      +0.1        0.27 ±  7%  perf-profile.children.cycles-pp.__radix_tree_lookup
      0.16 ±  7%      +0.1        0.31 ±  6%  perf-profile.children.cycles-pp.find_task_by_vpid
      0.15 ±  3%      +0.2        0.30        perf-profile.children.cycles-pp.setup_sigcontext
      0.00            +0.2        0.16 ± 14%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      1.09 ±  3%      +0.2        1.26 ±  4%  perf-profile.children.cycles-pp.recalc_sigpending_tsk
      0.16 ±  2%      +0.2        0.33        perf-profile.children.cycles-pp.copy_init_pkru_to_fpregs
      0.15 ±  2%      +0.2        0.32 ±  2%  perf-profile.children.cycles-pp.__might_sleep
      0.08 ±  5%      +0.2        0.27 ±  4%  perf-profile.children.cycles-pp.irq_exit
      0.17 ±  2%      +0.2        0.36        perf-profile.children.cycles-pp.__x64_sys_gettid
      0.08 ±  5%      +0.2        0.28 ±  4%  perf-profile.children.cycles-pp.__softirqentry_text_start
      0.06 ± 11%      +0.2        0.26 ±  4%  perf-profile.children.cycles-pp.rcu_core
      0.23            +0.2        0.45        perf-profile.children.cycles-pp.memset_erms
      0.40 ±  5%      +0.2        0.64 ±  2%  perf-profile.children.cycles-pp.smp_apic_timer_interrupt
      0.24            +0.2        0.47        perf-profile.children.cycles-pp.fpstate_init
      0.45 ±  4%      +0.2        0.68 ±  3%  perf-profile.children.cycles-pp.apic_timer_interrupt
      0.20 ±  3%      +0.2        0.45        perf-profile.children.cycles-pp.__x64_sys_getpid
      0.24 ±  2%      +0.3        0.51        perf-profile.children.cycles-pp.restore_altstack
      0.29            +0.3        0.59        perf-profile.children.cycles-pp.___might_sleep
      0.19 ±  3%      +0.3        0.53        perf-profile.children.cycles-pp._copy_to_user
      0.30            +0.3        0.64        perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      0.13            +0.4        0.49        perf-profile.children.cycles-pp.__clear_user
      0.44            +0.5        0.90        perf-profile.children.cycles-pp.__task_pid_nr_ns
      2.23            +0.6        2.83        perf-profile.children.cycles-pp.__fpu__restore_sig
      0.57            +0.6        1.22        perf-profile.children.cycles-pp.copy_user_generic_unrolled
      0.56            +0.7        1.23        perf-profile.children.cycles-pp.__might_fault
      0.76            +0.7        1.43        perf-profile.children.cycles-pp.swapgs_restore_regs_and_return_to_usermode
      0.67            +0.7        1.42        perf-profile.children.cycles-pp.native_irq_return_iret
      0.71            +0.8        1.49        perf-profile.children.cycles-pp._copy_from_user
      0.56 ±  2%      +1.0        1.59 ±  2%  perf-profile.children.cycles-pp.aa_may_signal
      2.81 ±  2%      +1.3        4.12        perf-profile.children.cycles-pp.__x64_sys_rt_sigprocmask
      1.34            +1.4        2.79        perf-profile.children.cycles-pp.copy_fpstate_to_sigframe
      3.46            +1.9        5.36        perf-profile.children.cycles-pp.__x64_sys_rt_sigreturn
      3.98            +5.0        8.97        perf-profile.children.cycles-pp.syscall_return_via_sysret
      4.35            +5.3        9.68        perf-profile.children.cycles-pp.entry_SYSCALL_64
     27.52            +7.0       34.53        perf-profile.children.cycles-pp.do_send_specific
     27.80            +7.3       35.10        perf-profile.children.cycles-pp.do_tkill
     27.86            +7.4       35.23        perf-profile.children.cycles-pp.__x64_sys_tgkill
      1.09            +9.4       10.53        perf-profile.children.cycles-pp.aa_get_task_label
      4.92           +20.8       25.73        perf-profile.children.cycles-pp.apparmor_task_kill
      4.94           +20.9       25.81        perf-profile.children.cycles-pp.security_task_kill
     20.07           -20.1        0.00        perf-profile.self.cycles-pp.refcount_dec_not_one
     20.30           -13.9        6.39 ±  2%  perf-profile.self.cycles-pp.__sigqueue_alloc
      9.71            -4.2        5.50 ±  2%  perf-profile.self.cycles-pp.__sigqueue_free
      2.64            -2.0        0.67 ±  2%  perf-profile.self.cycles-pp.recalc_sigpending
      2.56 ±  3%      -1.3        1.30 ±  2%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      1.39 ±  2%      -0.8        0.60        perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      1.56            -0.5        1.08 ±  4%  perf-profile.self.cycles-pp.fpregs_mark_activate
      1.58            -0.4        1.16        perf-profile.self.cycles-pp.fpu__clear
      0.10            -0.0        0.09 ±  4%  perf-profile.self.cycles-pp.kmem_cache_free
      0.11 ±  4%      +0.1        0.16 ±  2%  perf-profile.self.cycles-pp.get_signal
      0.05            +0.1        0.10        perf-profile.self.cycles-pp.send_signal
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.dequeue_signal
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.fpu__restore_sig
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.complete_signal
      0.06            +0.1        0.11 ±  3%  perf-profile.self.cycles-pp.__x64_sys_gettid
      0.03 ±100%      +0.1        0.08 ±  5%  perf-profile.self.cycles-pp.ktime_get
      0.06 ±  6%      +0.1        0.12        perf-profile.self.cycles-pp.__indirect_thunk_start
      0.01 ±173%      +0.1        0.07        perf-profile.self.cycles-pp.exit_to_usermode_loop
      0.07 ±  5%      +0.1        0.14        perf-profile.self.cycles-pp.__local_bh_enable_ip
      0.06            +0.1        0.13        perf-profile.self.cycles-pp.__x64_sys_tgkill
      0.06 ±  9%      +0.1        0.12 ±  4%  perf-profile.self.cycles-pp.__set_task_blocked
      0.06 ± 11%      +0.1        0.13        perf-profile.self.cycles-pp.__x64_sys_getpid
      0.03 ±100%      +0.1        0.10 ±  7%  perf-profile.self.cycles-pp.prepare_signal
      0.03 ±100%      +0.1        0.10        perf-profile.self.cycles-pp.kmem_cache_alloc
      0.00            +0.1        0.08 ±  5%  perf-profile.self.cycles-pp.restore_altstack
      0.00            +0.1        0.08        perf-profile.self.cycles-pp.security_task_kill
      0.01 ±173%      +0.1        0.10 ±  5%  perf-profile.self.cycles-pp.sigprocmask
      0.08            +0.1        0.17 ±  3%  perf-profile.self.cycles-pp.__dequeue_signal
      0.14            +0.1        0.23        perf-profile.self.cycles-pp.__send_signal
      0.09 ±  4%      +0.1        0.18        perf-profile.self.cycles-pp._copy_from_user
      0.10 ±  4%      +0.1        0.21 ±  4%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.00            +0.1        0.10 ±  8%  perf-profile.self.cycles-pp.do_send_sig_info
      0.00            +0.1        0.11 ±  4%  perf-profile.self.cycles-pp._copy_to_user
      0.10 ±  4%      +0.1        0.21 ±  2%  perf-profile.self.cycles-pp.__set_current_blocked
      0.13 ±  3%      +0.1        0.24        perf-profile.self.cycles-pp.fpregs_assert_state_consistent
      0.11 ±  6%      +0.1        0.23        perf-profile.self.cycles-pp.copy_init_pkru_to_fpregs
      0.13 ±  9%      +0.1        0.27 ±  7%  perf-profile.self.cycles-pp.__radix_tree_lookup
      0.15 ±  3%      +0.1        0.29        perf-profile.self.cycles-pp.setup_sigcontext
      0.00            +0.2        0.16 ± 14%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      0.13 ±  3%      +0.2        0.29 ±  2%  perf-profile.self.cycles-pp.__might_sleep
      1.08 ±  3%      +0.2        1.25 ±  4%  perf-profile.self.cycles-pp.recalc_sigpending_tsk
      0.13            +0.2        0.31 ±  2%  perf-profile.self.cycles-pp.do_tkill
      0.05            +0.2        0.25 ±  4%  perf-profile.self.cycles-pp.do_send_specific
      0.23 ±  2%      +0.2        0.45        perf-profile.self.cycles-pp.memset_erms
      0.16 ±  5%      +0.3        0.43 ±  3%  perf-profile.self.cycles-pp.__might_fault
      0.26 ±  3%      +0.3        0.54        perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.22            +0.3        0.51 ±  2%  perf-profile.self.cycles-pp.__x64_sys_rt_sigprocmask
      0.29 ±  2%      +0.3        0.58        perf-profile.self.cycles-pp.___might_sleep
      0.13 ±  3%      +0.3        0.43        perf-profile.self.cycles-pp.__clear_user
      0.29            +0.3        0.62        perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      0.39 ±  2%      +0.4        0.77        perf-profile.self.cycles-pp.do_signal
      0.42            +0.4        0.87        perf-profile.self.cycles-pp.__task_pid_nr_ns
      0.46            +0.5        0.99        perf-profile.self.cycles-pp.copy_user_generic_unrolled
      0.49            +0.6        1.04        perf-profile.self.cycles-pp.raise
      1.60            +0.6        2.24        perf-profile.self.cycles-pp.__fpu__restore_sig
      0.74            +0.7        1.41        perf-profile.self.cycles-pp.swapgs_restore_regs_and_return_to_usermode
      0.67            +0.7        1.41        perf-profile.self.cycles-pp.native_irq_return_iret
      0.53            +0.8        1.36        perf-profile.self.cycles-pp.__x64_sys_rt_sigreturn
      1.01            +0.9        1.93        perf-profile.self.cycles-pp.copy_fpstate_to_sigframe
      0.56 ±  2%      +1.0        1.57        perf-profile.self.cycles-pp.aa_may_signal
     15.16            +1.8       16.95        perf-profile.self.cycles-pp.do_syscall_64
      3.97            +4.9        8.86        perf-profile.self.cycles-pp.entry_SYSCALL_64
      3.96            +5.0        8.96        perf-profile.self.cycles-pp.syscall_return_via_sysret
      1.08            +9.4       10.45        perf-profile.self.cycles-pp.aa_get_task_label
      3.25           +10.3       13.52        perf-profile.self.cycles-pp.apparmor_task_kill



***************************************************************************************************
lkp-csl-2ap3: 192 threads Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory
=========================================================================================
cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/tbox_group/test/testcase/ucode:
  cs-localhost/gcc-7/performance/ipv4/x86_64-rhel-7.6/16/debian-x86_64-20191114.cgz/300s/lkp-csl-2ap3/TCP_CRR/netperf/0x500002c

commit: 
  c5f8689118 (" RISC-V Fixes for 5.6-rc4")
  fda31c5029 ("signal: avoid double atomic counter increments for user accounting")

c5f86891185c408b fda31c50292a5062332fa0343c0 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :4           25%           1:4     dmesg.WARNING:at_ip_tcp_get_metrics/0x
           :4           25%           1:4     dmesg.WARNING:stack_recursion
         %stddev     %change         %stddev
             \          |                \  
    322509           +10.2%     355460        netperf.Throughput_total_tps
     20156           +10.2%      22216        netperf.Throughput_tps
      7382 ±  7%     +26.3%       9322 ±  9%  netperf.time.minor_page_faults
     87.75           +11.1%      97.49        netperf.time.user_time
 1.853e+08           +11.3%  2.062e+08        netperf.time.voluntary_context_switches
  96752796           +10.2%  1.066e+08        netperf.workload
 1.703e+08 ±  2%     +15.9%  1.973e+08        cpuidle.POLL.usage
   2222403            +6.7%    2370426        vmstat.system.cs
      3365 ± 22%     -46.5%       1798 ± 59%  numa-vmstat.node1.nr_shmem
    442.50 ± 27%    +258.2%       1585 ± 53%  numa-vmstat.node2.nr_shmem
     13462 ± 22%     -46.5%       7196 ± 59%  numa-meminfo.node1.Shmem
    992.75 ± 27%    +467.0%       5628 ± 65%  numa-meminfo.node2.Inactive
      1771 ± 27%    +258.0%       6341 ± 53%  numa-meminfo.node2.Shmem
    218532 ± 10%     +29.9%     283805 ±  4%  sched_debug.cfs_rq:/.min_vruntime.stddev
    218541 ± 10%     +29.9%     283828 ±  4%  sched_debug.cfs_rq:/.spread0.stddev
   1009329 ±  8%     +36.8%    1380435        sched_debug.cpu.nr_switches.stddev
    282041 ± 16%     -35.6%     181597 ± 33%  numa-numastat.node1.local_node
    305341 ± 11%     -30.4%     212613 ± 28%  numa-numastat.node1.numa_hit
    276974 ± 28%     +37.1%     379821 ±  8%  numa-numastat.node3.local_node
    292564 ± 28%     +35.2%     395417 ±  4%  numa-numastat.node3.numa_hit
     65742            -9.6%      59430        slabinfo.sock_inode_cache.active_objs
      1688            -9.6%       1526        slabinfo.sock_inode_cache.active_slabs
     65852            -9.5%      59567        slabinfo.sock_inode_cache.num_objs
      1688            -9.6%       1526        slabinfo.sock_inode_cache.num_slabs
     37612            -3.0%      36475        proc-vmstat.nr_slab_reclaimable
     10575 ± 69%     +72.1%      18196 ± 44%  proc-vmstat.numa_hint_faults
      3415 ± 28%    +118.2%       7450 ± 37%  proc-vmstat.numa_hint_faults_local
   1193490 ±  2%     +10.4%    1318175        proc-vmstat.numa_hit
   1100296 ±  2%     +11.3%    1224975        proc-vmstat.numa_local
   2391504 ±  8%     +39.3%    3330174 ±  2%  proc-vmstat.pgalloc_normal
   2448804 ±  8%     +38.4%    3389302 ±  2%  proc-vmstat.pgfree
     16.94 ±  3%      -6.1%      15.91        perf-stat.i.MPKI
 7.854e+09            +9.3%  8.586e+09        perf-stat.i.branch-instructions
  1.64e+08            +8.5%   1.78e+08        perf-stat.i.branch-misses
     16.69            -2.0       14.68        perf-stat.i.cache-miss-rate%
 1.096e+08            -8.9%   99855110        perf-stat.i.cache-misses
 6.567e+08            +3.8%   6.82e+08        perf-stat.i.cache-references
   2245859            +6.8%    2398766        perf-stat.i.context-switches
      2.00            -6.1%       1.88        perf-stat.i.cpi
 7.748e+10 ±  2%      +3.1%  7.991e+10        perf-stat.i.cpu-cycles
    338.22 ±  2%      +6.8%     361.18 ±  3%  perf-stat.i.cpu-migrations
    724.52           +12.7%     816.64        perf-stat.i.cycles-between-cache-misses
 1.183e+10            +9.6%  1.296e+10        perf-stat.i.dTLB-loads
      0.02 ±  9%      -0.0        0.00 ± 22%  perf-stat.i.dTLB-store-miss-rate%
   1195259 ±  2%     -81.3%     223982 ±  2%  perf-stat.i.dTLB-store-misses
 6.933e+09            +9.6%  7.597e+09        perf-stat.i.dTLB-stores
 1.323e+08            +9.3%  1.446e+08        perf-stat.i.iTLB-load-misses
 3.941e+10            +9.4%  4.311e+10        perf-stat.i.instructions
      0.51 ±  2%      +6.1%       0.54        perf-stat.i.ipc
  20549408            -6.6%   19188189 ±  2%  perf-stat.i.node-load-misses
   1819402 ±  4%      -6.1%    1707901        perf-stat.i.node-loads
   7248084            +5.8%    7671217        perf-stat.i.node-store-misses
     16.67            -5.1%      15.82        perf-stat.overall.MPKI
     16.68            -2.0       14.64        perf-stat.overall.cache-miss-rate%
      1.97 ±  2%      -5.7%       1.85        perf-stat.overall.cpi
    707.17           +13.2%     800.43        perf-stat.overall.cycles-between-cache-misses
      0.02 ±  2%      -0.0        0.00 ±  3%  perf-stat.overall.dTLB-store-miss-rate%
      0.51 ±  2%      +6.0%       0.54        perf-stat.overall.ipc
 7.827e+09            +9.3%  8.558e+09        perf-stat.ps.branch-instructions
 1.635e+08            +8.5%  1.774e+08        perf-stat.ps.branch-misses
 1.092e+08            -8.9%   99521756        perf-stat.ps.cache-misses
 6.545e+08            +3.8%  6.797e+08        perf-stat.ps.cache-references
   2238328            +6.8%    2390749        perf-stat.ps.context-switches
 7.722e+10 ±  2%      +3.1%  7.964e+10        perf-stat.ps.cpu-cycles
    337.30 ±  2%      +6.8%     360.28 ±  3%  perf-stat.ps.cpu-migrations
 1.179e+10            +9.6%  1.292e+10        perf-stat.ps.dTLB-loads
   1191307 ±  2%     -81.3%     223300 ±  2%  perf-stat.ps.dTLB-store-misses
  6.91e+09            +9.6%  7.572e+09        perf-stat.ps.dTLB-stores
 1.318e+08            +9.3%  1.441e+08        perf-stat.ps.iTLB-load-misses
 3.927e+10            +9.4%  4.296e+10        perf-stat.ps.instructions
  20479931            -6.6%   19123045 ±  2%  perf-stat.ps.node-load-misses
   1813547 ±  4%      -6.1%    1702635        perf-stat.ps.node-loads
   7223611            +5.8%    7645156        perf-stat.ps.node-store-misses
 1.201e+13            +9.2%  1.312e+13        perf-stat.total.instructions
      1055 ± 20%    +128.1%       2407 ± 45%  interrupts.CPU10.NMI:Non-maskable_interrupts
      1055 ± 20%    +128.1%       2407 ± 45%  interrupts.CPU10.PMI:Performance_monitoring_interrupts
      8570 ±  3%     -31.1%       5903 ± 22%  interrupts.CPU100.RES:Rescheduling_interrupts
      8521 ± 20%     -39.3%       5171 ± 32%  interrupts.CPU101.RES:Rescheduling_interrupts
      7475 ± 14%     -44.4%       4157 ± 50%  interrupts.CPU102.RES:Rescheduling_interrupts
      7131 ± 11%     -47.7%       3732 ± 26%  interrupts.CPU103.RES:Rescheduling_interrupts
      7606 ± 10%     -34.8%       4960 ± 25%  interrupts.CPU104.RES:Rescheduling_interrupts
      8024 ± 11%     -38.7%       4917 ± 21%  interrupts.CPU105.RES:Rescheduling_interrupts
      9968 ± 17%     -52.8%       4700 ± 10%  interrupts.CPU107.RES:Rescheduling_interrupts
      8589 ± 17%     -44.4%       4779 ± 26%  interrupts.CPU108.RES:Rescheduling_interrupts
      7968 ± 10%     -44.3%       4439 ± 26%  interrupts.CPU109.RES:Rescheduling_interrupts
      7399 ±  8%     -46.4%       3963 ± 11%  interrupts.CPU111.RES:Rescheduling_interrupts
      6432 ± 13%     -52.5%       3053 ± 18%  interrupts.CPU115.RES:Rescheduling_interrupts
      5560 ± 20%     -39.8%       3347 ± 15%  interrupts.CPU116.RES:Rescheduling_interrupts
      5974 ± 12%     -44.5%       3313 ± 23%  interrupts.CPU117.RES:Rescheduling_interrupts
      1179 ± 35%     -55.3%     526.75 ± 27%  interrupts.CPU118.NMI:Non-maskable_interrupts
      1179 ± 35%     -55.3%     526.75 ± 27%  interrupts.CPU118.PMI:Performance_monitoring_interrupts
      7565 ± 14%     -25.5%       5639 ± 16%  interrupts.CPU123.RES:Rescheduling_interrupts
      8549 ± 18%     -51.0%       4186 ± 16%  interrupts.CPU125.RES:Rescheduling_interrupts
      7080 ± 22%     -46.5%       3791 ± 37%  interrupts.CPU127.RES:Rescheduling_interrupts
      6232 ± 11%     -40.1%       3732 ± 29%  interrupts.CPU128.RES:Rescheduling_interrupts
    889.25 ± 48%    +111.6%       1881 ± 29%  interrupts.CPU130.NMI:Non-maskable_interrupts
    889.25 ± 48%    +111.6%       1881 ± 29%  interrupts.CPU130.PMI:Performance_monitoring_interrupts
      5799 ± 23%     -34.3%       3808 ± 13%  interrupts.CPU130.RES:Rescheduling_interrupts
      7009 ± 31%     -37.4%       4385 ± 14%  interrupts.CPU131.RES:Rescheduling_interrupts
      6116 ± 22%     -44.2%       3411 ± 33%  interrupts.CPU133.RES:Rescheduling_interrupts
      7620 ±  5%     -45.0%       4191 ± 42%  interrupts.CPU134.RES:Rescheduling_interrupts
      6764 ± 17%     -36.4%       4300 ± 31%  interrupts.CPU135.RES:Rescheduling_interrupts
      7088 ± 21%     -36.8%       4478 ±  6%  interrupts.CPU136.RES:Rescheduling_interrupts
      6892 ±  7%     -28.0%       4961 ± 20%  interrupts.CPU137.RES:Rescheduling_interrupts
    896.50 ± 41%     -51.8%     432.25 ± 43%  interrupts.CPU140.NMI:Non-maskable_interrupts
    896.50 ± 41%     -51.8%     432.25 ± 43%  interrupts.CPU140.PMI:Performance_monitoring_interrupts
      6098 ± 18%     -43.7%       3433 ± 27%  interrupts.CPU140.RES:Rescheduling_interrupts
      1603 ± 20%     -62.4%     602.25 ± 49%  interrupts.CPU141.NMI:Non-maskable_interrupts
      1603 ± 20%     -62.4%     602.25 ± 49%  interrupts.CPU141.PMI:Performance_monitoring_interrupts
      9304 ± 13%     -49.2%       4730 ± 33%  interrupts.CPU149.RES:Rescheduling_interrupts
      6589 ± 11%     -45.9%       3567 ± 14%  interrupts.CPU150.RES:Rescheduling_interrupts
      6926 ± 16%     -45.5%       3774 ± 13%  interrupts.CPU151.RES:Rescheduling_interrupts
      7982 ± 30%     -54.8%       3611 ± 35%  interrupts.CPU152.RES:Rescheduling_interrupts
      7352 ± 18%     -41.4%       4311 ± 35%  interrupts.CPU153.RES:Rescheduling_interrupts
      7475 ± 11%     -39.4%       4530 ± 15%  interrupts.CPU154.RES:Rescheduling_interrupts
      7749 ± 15%     -53.2%       3626 ± 10%  interrupts.CPU155.RES:Rescheduling_interrupts
      6817 ± 23%     -30.3%       4753 ± 23%  interrupts.CPU156.RES:Rescheduling_interrupts
      7093 ± 18%     -46.4%       3802 ± 33%  interrupts.CPU157.RES:Rescheduling_interrupts
      6179 ± 27%     -38.4%       3805 ± 23%  interrupts.CPU159.RES:Rescheduling_interrupts
      1042 ± 28%     +75.2%       1826 ± 32%  interrupts.CPU16.NMI:Non-maskable_interrupts
      1042 ± 28%     +75.2%       1826 ± 32%  interrupts.CPU16.PMI:Performance_monitoring_interrupts
      6360 ± 17%     -45.8%       3448 ± 24%  interrupts.CPU160.RES:Rescheduling_interrupts
      2159 ± 12%     -66.4%     726.50 ± 37%  interrupts.CPU161.NMI:Non-maskable_interrupts
      2159 ± 12%     -66.4%     726.50 ± 37%  interrupts.CPU161.PMI:Performance_monitoring_interrupts
      7674 ±  9%     -38.7%       4707 ± 18%  interrupts.CPU161.RES:Rescheduling_interrupts
      7093 ± 20%     -38.6%       4355 ± 23%  interrupts.CPU162.RES:Rescheduling_interrupts
      6247 ± 21%     -44.5%       3470 ± 19%  interrupts.CPU163.RES:Rescheduling_interrupts
      5874 ± 17%     -47.0%       3115 ± 19%  interrupts.CPU164.RES:Rescheduling_interrupts
      6423 ± 22%     -46.0%       3467 ± 14%  interrupts.CPU166.RES:Rescheduling_interrupts
    940.25 ± 41%     +74.9%       1644 ± 12%  interrupts.CPU17.NMI:Non-maskable_interrupts
    940.25 ± 41%     +74.9%       1644 ± 12%  interrupts.CPU17.PMI:Performance_monitoring_interrupts
      8566 ±  7%     -47.9%       4459 ± 31%  interrupts.CPU173.RES:Rescheduling_interrupts
      7573 ± 21%     -51.2%       3697 ± 25%  interrupts.CPU174.RES:Rescheduling_interrupts
      7335 ± 25%     -37.3%       4596 ± 11%  interrupts.CPU175.RES:Rescheduling_interrupts
      7736 ± 11%     -34.2%       5091 ± 14%  interrupts.CPU177.RES:Rescheduling_interrupts
      8237 ± 14%     -39.0%       5022 ± 11%  interrupts.CPU178.RES:Rescheduling_interrupts
      9074 ± 15%     -33.5%       6032 ± 12%  interrupts.CPU182.RES:Rescheduling_interrupts
      7147 ± 27%     -44.4%       3975 ± 22%  interrupts.CPU185.RES:Rescheduling_interrupts
      8989 ± 20%     -59.1%       3673 ± 23%  interrupts.CPU186.RES:Rescheduling_interrupts
      8485 ± 12%     -38.9%       5181 ± 20%  interrupts.CPU187.RES:Rescheduling_interrupts
      7586 ± 10%     -28.4%       5430 ± 33%  interrupts.CPU188.RES:Rescheduling_interrupts
    642.75 ± 59%    +148.4%       1596 ± 53%  interrupts.CPU19.NMI:Non-maskable_interrupts
    642.75 ± 59%    +148.4%       1596 ± 53%  interrupts.CPU19.PMI:Performance_monitoring_interrupts
      9094 ± 13%     -37.7%       5669 ± 31%  interrupts.CPU190.RES:Rescheduling_interrupts
    884.25 ± 35%     +70.1%       1504 ± 27%  interrupts.CPU23.NMI:Non-maskable_interrupts
    884.25 ± 35%     +70.1%       1504 ± 27%  interrupts.CPU23.PMI:Performance_monitoring_interrupts
     38525 ±  7%     -17.7%      31724 ±  8%  interrupts.CPU24.RES:Rescheduling_interrupts
    871.75 ± 37%    +100.4%       1747 ± 29%  interrupts.CPU31.NMI:Non-maskable_interrupts
    871.75 ± 37%    +100.4%       1747 ± 29%  interrupts.CPU31.PMI:Performance_monitoring_interrupts
     11805 ±  9%     +29.4%      15276 ± 12%  interrupts.CPU32.RES:Rescheduling_interrupts
      1058 ± 51%    +119.8%       2327 ± 44%  interrupts.CPU33.NMI:Non-maskable_interrupts
      1058 ± 51%    +119.8%       2327 ± 44%  interrupts.CPU33.PMI:Performance_monitoring_interrupts
     38118 ± 14%     -23.5%      29177 ±  6%  interrupts.CPU48.RES:Rescheduling_interrupts
     30784 ±  7%     -23.4%      23592 ±  8%  interrupts.CPU49.RES:Rescheduling_interrupts
     19693 ±  7%     +13.5%      22354 ±  6%  interrupts.CPU5.RES:Rescheduling_interrupts
     10794 ± 16%     -39.1%       6569 ± 30%  interrupts.CPU58.RES:Rescheduling_interrupts
      7622 ± 13%     -19.8%       6112 ± 11%  interrupts.CPU61.RES:Rescheduling_interrupts
      8081 ± 17%     -31.8%       5507 ± 12%  interrupts.CPU69.RES:Rescheduling_interrupts
      1061 ± 68%     -69.0%     329.25 ± 51%  interrupts.CPU70.NMI:Non-maskable_interrupts
      1061 ± 68%     -69.0%     329.25 ± 51%  interrupts.CPU70.PMI:Performance_monitoring_interrupts
      7918 ± 12%     -41.8%       4604 ±  3%  interrupts.CPU70.RES:Rescheduling_interrupts
      8417 ± 16%     -40.2%       5032 ± 19%  interrupts.CPU71.RES:Rescheduling_interrupts
      1466 ± 20%     +75.2%       2568 ± 32%  interrupts.CPU76.NMI:Non-maskable_interrupts
      1466 ± 20%     +75.2%       2568 ± 32%  interrupts.CPU76.PMI:Performance_monitoring_interrupts
    814.50 ± 46%    +214.7%       2563 ± 63%  interrupts.CPU77.NMI:Non-maskable_interrupts
    814.50 ± 46%    +214.7%       2563 ± 63%  interrupts.CPU77.PMI:Performance_monitoring_interrupts
      8621 ± 18%     -28.8%       6136 ± 19%  interrupts.CPU86.RES:Rescheduling_interrupts
      8983 ± 14%     -34.0%       5931 ± 32%  interrupts.CPU87.RES:Rescheduling_interrupts
     11835 ± 15%     -45.1%       6497 ± 25%  interrupts.CPU88.RES:Rescheduling_interrupts
      4059 ± 20%     -52.2%       1940 ± 12%  interrupts.CPU9.NMI:Non-maskable_interrupts
      4059 ± 20%     -52.2%       1940 ± 12%  interrupts.CPU9.PMI:Performance_monitoring_interrupts
      8488 ± 24%     -33.7%       5632 ± 21%  interrupts.CPU90.RES:Rescheduling_interrupts
      9135 ±  8%     -37.5%       5711 ± 26%  interrupts.CPU93.RES:Rescheduling_interrupts
   2025835 ±  2%     -14.3%    1735783        interrupts.RES:Rescheduling_interrupts
   3401595 ± 10%     +14.8%    3903805 ±  7%  softirqs.CPU0.NET_RX
   4622138 ±  8%     +16.5%    5384961 ±  4%  softirqs.CPU1.NET_RX
    375709 ±  4%     +10.5%     415117 ±  2%  softirqs.CPU1.RCU
   1833804 ±  7%     +49.3%    2737951 ± 15%  softirqs.CPU10.NET_RX
    242358 ±  2%     +18.6%     287422 ±  7%  softirqs.CPU10.RCU
     46964 ±  2%      +7.1%      50308 ±  3%  softirqs.CPU10.SCHED
   1260318 ±  4%     -22.8%     972590 ± 21%  softirqs.CPU100.NET_RX
   1033703 ± 11%     -35.9%     662702 ± 32%  softirqs.CPU103.NET_RX
    200335 ±  3%      -9.1%     182036 ±  6%  softirqs.CPU103.RCU
   1422191 ± 14%     -37.7%     885854 ± 10%  softirqs.CPU107.NET_RX
    220237 ±  4%     -12.4%     192929 ±  2%  softirqs.CPU107.RCU
   1160807 ± 10%     -26.8%     849210 ± 26%  softirqs.CPU109.NET_RX
   1900896 ±  2%     +35.5%    2575457 ± 11%  softirqs.CPU11.NET_RX
    244903           +14.2%     279748 ±  5%  softirqs.CPU11.RCU
   1071385 ±  8%     -27.7%     775088 ±  9%  softirqs.CPU111.NET_RX
    979515 ± 14%     -39.2%     595214 ± 14%  softirqs.CPU115.NET_RX
    197903 ±  3%      -9.3%     179482 ±  2%  softirqs.CPU115.RCU
    882082 ± 17%     -29.7%     620432 ± 14%  softirqs.CPU116.NET_RX
    867298 ± 12%     -25.5%     646134 ± 17%  softirqs.CPU117.NET_RX
   1794166 ± 15%     +29.1%    2316141 ± 12%  softirqs.CPU12.NET_RX
    239087 ±  6%     +11.4%     266391 ±  5%  softirqs.CPU12.RCU
    848536 ± 11%     +28.0%    1086122 ± 10%  softirqs.CPU120.NET_RX
   1123406 ± 18%     -34.6%     734889 ± 18%  softirqs.CPU125.NET_RX
    204518 ±  4%      -9.2%     185653 ±  3%  softirqs.CPU125.RCU
   1018785 ± 18%     -34.7%     665177 ± 34%  softirqs.CPU127.NET_RX
   1000752 ± 13%     -29.6%     704748 ± 23%  softirqs.CPU128.NET_RX
     44458 ±  2%      -7.1%      41297 ±  4%  softirqs.CPU134.SCHED
   1542490 ± 10%     +40.0%    2158872 ±  8%  softirqs.CPU14.NET_RX
    228004 ±  3%     +13.6%     258992 ±  3%  softirqs.CPU14.RCU
    985123 ± 18%     -28.8%     700948 ± 26%  softirqs.CPU140.NET_RX
   1159842 ± 13%     +57.6%    1827624 ± 13%  softirqs.CPU143.NET_RX
    207874 ±  3%     +14.6%     238308 ±  4%  softirqs.CPU143.RCU
   1286990 ±  9%     -31.7%     878963 ± 20%  softirqs.CPU149.NET_RX
   1060469 ±  9%     -31.8%     723313 ± 14%  softirqs.CPU150.NET_RX
    202027 ±  2%      -8.3%     185175 ±  2%  softirqs.CPU150.RCU
   1033042 ± 15%     -28.1%     742587 ± 16%  softirqs.CPU151.NET_RX
   1186178 ± 10%     -39.0%     723314 ± 10%  softirqs.CPU155.NET_RX
    207907 ±  3%     -10.6%     185862 ±  2%  softirqs.CPU155.RCU
   1524416 ± 16%     +32.4%    2017860 ± 11%  softirqs.CPU16.NET_RX
    227281 ±  5%     +10.5%     251113 ±  4%  softirqs.CPU16.RCU
    924942 ± 11%     -28.5%     661677 ± 30%  softirqs.CPU160.NET_RX
    910616 ± 18%     -34.2%     599277 ± 15%  softirqs.CPU164.NET_RX
    194859 ±  5%      -7.0%     181264        softirqs.CPU166.RCU
   1279731 ± 13%     +18.8%    1520309 ±  7%  softirqs.CPU167.NET_RX
   1249019 ±  2%     -35.6%     804451 ± 26%  softirqs.CPU173.NET_RX
    211274           -10.6%     188789 ±  5%  softirqs.CPU173.RCU
   1200880 ± 16%     -20.7%     952011 ± 14%  softirqs.CPU178.NET_RX
   1236223 ± 19%     -41.5%     723685 ± 22%  softirqs.CPU186.NET_RX
    209369 ±  4%     -11.0%     186308 ±  4%  softirqs.CPU186.RCU
     45103 ±  2%      -9.4%      40881        softirqs.CPU186.SCHED
   1490168 ±  8%     +39.0%    2071447 ± 13%  softirqs.CPU19.NET_RX
    224087 ±  3%     +12.9%     253013 ±  5%  softirqs.CPU19.RCU
   3959103 ±  4%     +23.5%    4888600 ±  8%  softirqs.CPU2.NET_RX
    345221 ±  2%     +13.5%     391654 ±  5%  softirqs.CPU2.RCU
   2888234 ±  9%     +51.0%    4362408 ± 15%  softirqs.CPU28.NET_RX
    292016 ±  4%     +26.4%     369052 ±  8%  softirqs.CPU28.RCU
   2724145 ± 15%     +56.0%    4251002 ± 12%  softirqs.CPU29.NET_RX
    285780 ±  6%     +27.1%     363294 ±  7%  softirqs.CPU29.RCU
     51503 ±  3%     +11.5%      57450 ±  4%  softirqs.CPU29.SCHED
   4069535 ±  6%     +22.5%    4986134 ±  7%  softirqs.CPU3.NET_RX
    349534 ±  3%     +13.8%     397896 ±  4%  softirqs.CPU3.RCU
   2239096 ±  7%     +58.5%    3549052 ± 14%  softirqs.CPU30.NET_RX
    262465 ±  2%     +24.9%     327859 ±  8%  softirqs.CPU30.RCU
   1903616 ± 16%     +66.3%    3165849 ± 11%  softirqs.CPU31.NET_RX
    245631 ±  6%     +25.3%     307687 ±  6%  softirqs.CPU31.RCU
   1872898 ±  5%     +55.8%    2918098 ±  9%  softirqs.CPU32.NET_RX
    245264 ±  2%     +20.7%     296097 ±  4%  softirqs.CPU32.RCU
    238892 ±  5%     +13.0%     269868 ±  5%  softirqs.CPU33.RCU
   1625479 ± 14%     +42.0%    2307455 ± 12%  softirqs.CPU34.NET_RX
    230172 ±  5%     +15.6%     265984 ±  5%  softirqs.CPU34.RCU
   1431425 ± 27%     +50.7%    2156596 ± 10%  softirqs.CPU35.NET_RX
    222408 ±  8%     +15.0%     255877 ±  3%  softirqs.CPU35.RCU
   1425888 ± 10%     +57.9%    2251562 ± 16%  softirqs.CPU36.NET_RX
    220910 ±  3%     +18.4%     261540 ±  6%  softirqs.CPU36.RCU
   1390282 ± 20%     +39.5%    1938774 ± 21%  softirqs.CPU37.NET_RX
    218899 ±  6%     +12.7%     246597 ±  7%  softirqs.CPU37.RCU
   3466299 ±  7%     +33.1%    4613618 ± 11%  softirqs.CPU4.NET_RX
    319333 ±  3%     +20.5%     384704 ±  7%  softirqs.CPU4.RCU
    217199 ±  2%     +13.2%     245942 ±  7%  softirqs.CPU40.RCU
   1318730 ± 14%     +49.4%    1969608 ± 25%  softirqs.CPU43.NET_RX
    210877 ±  4%     +10.7%     233360 ±  4%  softirqs.CPU46.RCU
   3130163 ±  4%     +39.1%    4355289 ±  8%  softirqs.CPU5.NET_RX
    304762 ±  2%     +20.5%     367320 ±  5%  softirqs.CPU5.RCU
   2684258 ±  7%     +41.9%    3808765 ± 16%  softirqs.CPU6.NET_RX
    284887 ±  2%     +20.4%     342882 ±  9%  softirqs.CPU6.RCU
   1102761 ± 16%     +26.1%    1390745 ± 13%  softirqs.CPU66.NET_RX
   2636500 ± 12%     +39.5%    3676666 ±  8%  softirqs.CPU7.NET_RX
    280346 ±  5%     +19.1%     333768 ±  4%  softirqs.CPU7.RCU
   1179367 ± 13%     -22.6%     912396 ±  7%  softirqs.CPU70.NET_RX
   1224610 ± 13%     -20.7%     971311 ± 16%  softirqs.CPU71.NET_RX
   4206117 ± 15%     +28.8%    5418348 ±  7%  softirqs.CPU72.NET_RX
    353633 ±  8%     +17.2%     414446 ±  4%  softirqs.CPU72.RCU
   3827474 ± 16%     +36.7%    5231591 ±  7%  softirqs.CPU73.NET_RX
    337017 ±  8%     +20.1%     404857 ±  4%  softirqs.CPU73.RCU
   3098547 ± 17%     +43.7%    4453958 ±  5%  softirqs.CPU74.NET_RX
    303181 ±  8%     +22.5%     371474 ±  3%  softirqs.CPU74.RCU
   2477806 ± 15%     +35.9%    3367981 ± 12%  softirqs.CPU76.NET_RX
    274071 ±  6%     +16.7%     319970 ±  6%  softirqs.CPU76.RCU
   2097179 ± 25%     +59.3%    3341034 ± 11%  softirqs.CPU77.NET_RX
    253023 ± 10%     +24.4%     314882 ±  5%  softirqs.CPU77.RCU
     48472 ±  4%     +10.6%      53602 ±  3%  softirqs.CPU77.SCHED
   1994037 ± 15%     +34.6%    2683848 ± 10%  softirqs.CPU78.NET_RX
    248165 ±  5%     +14.4%     283969 ±  5%  softirqs.CPU78.RCU
   1862157 ± 14%     +68.0%    3128025 ± 18%  softirqs.CPU8.NET_RX
    243359 ±  5%     +25.8%     306238 ±  8%  softirqs.CPU8.RCU
   2100938 ± 10%     +38.3%    2906103 ± 16%  softirqs.CPU9.NET_RX
    253470 ±  4%     +16.3%     294755 ±  8%  softirqs.CPU9.RCU
  2.96e+08           +10.5%  3.272e+08        softirqs.NET_RX
     20.33            -1.7       18.59 ±  5%  perf-profile.calltrace.cycles-pp.ip_rcv.__netif_receive_skb_one_core.process_backlog.net_rx_action.__softirqentry_text_start
     20.57            -1.7       18.84 ±  5%  perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.net_rx_action.__softirqentry_text_start.do_softirq_own_stack
     20.89            -1.7       19.17 ±  5%  perf-profile.calltrace.cycles-pp.process_backlog.net_rx_action.__softirqentry_text_start.do_softirq_own_stack.do_softirq
     21.07            -1.7       19.36 ±  5%  perf-profile.calltrace.cycles-pp.net_rx_action.__softirqentry_text_start.do_softirq_own_stack.do_softirq.__local_bh_enable_ip
     21.39            -1.7       19.69 ±  5%  perf-profile.calltrace.cycles-pp.do_softirq_own_stack.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_output
     21.34            -1.7       19.63 ±  5%  perf-profile.calltrace.cycles-pp.__softirqentry_text_start.do_softirq_own_stack.do_softirq.__local_bh_enable_ip.ip_finish_output2
     21.56            -1.7       19.89 ±  5%  perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb
     21.49            -1.7       19.83 ±  5%  perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_output.__ip_queue_xmit
     18.96            -1.5       17.50 ±  5%  perf-profile.calltrace.cycles-pp.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core
     19.04            -1.5       17.59 ±  5%  perf-profile.calltrace.cycles-pp.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_backlog.net_rx_action
     18.98            -1.4       17.54 ±  5%  perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_backlog
     18.36            -1.3       17.09 ±  5%  perf-profile.calltrace.cycles-pp.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver.ip_rcv
     14.66            -1.2       13.41 ±  6%  perf-profile.calltrace.cycles-pp.__inet_stream_connect.inet_stream_connect.__sys_connect.__x64_sys_connect.do_syscall_64
     14.72            -1.2       13.47 ±  6%  perf-profile.calltrace.cycles-pp.inet_stream_connect.__sys_connect.__x64_sys_connect.do_syscall_64.entry_SYSCALL_64_after_hwframe
     14.89 ±  2%      -1.2       13.67 ±  6%  perf-profile.calltrace.cycles-pp.__x64_sys_connect.do_syscall_64.entry_SYSCALL_64_after_hwframe
     14.87 ±  2%      -1.2       13.65 ±  6%  perf-profile.calltrace.cycles-pp.__sys_connect.__x64_sys_connect.do_syscall_64.entry_SYSCALL_64_after_hwframe
     13.24            -1.0       12.20 ±  5%  perf-profile.calltrace.cycles-pp.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames
     13.06            -1.0       12.03 ±  5%  perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit
     10.64            -1.0        9.66 ±  5%  perf-profile.calltrace.cycles-pp.tcp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64
     10.93            -1.0        9.95 ±  5%  perf-profile.calltrace.cycles-pp.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
     11.04            -1.0       10.06 ±  5%  perf-profile.calltrace.cycles-pp.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
     11.07            -1.0       10.09 ±  5%  perf-profile.calltrace.cycles-pp.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
      8.93 ±  2%      -0.7        8.18 ±  5%  perf-profile.calltrace.cycles-pp.tcp_write_xmit.__tcp_push_pending_frames.tcp_close.inet_release.__sock_release
      9.17            -0.7        8.42 ±  5%  perf-profile.calltrace.cycles-pp.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto
      8.95 ±  2%      -0.7        8.20 ±  5%  perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.tcp_close.inet_release.__sock_release.sock_close
      8.12            -0.7        7.38 ±  5%  perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto
      8.10            -0.7        7.36 ±  5%  perf-profile.calltrace.cycles-pp.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg
     13.10 ±  2%      -0.7       12.38 ±  5%  perf-profile.calltrace.cycles-pp.sock_close.__fput.task_work_run.exit_to_usermode_loop.do_syscall_64
     13.09 ±  2%      -0.7       12.37 ±  5%  perf-profile.calltrace.cycles-pp.__sock_release.sock_close.__fput.task_work_run.exit_to_usermode_loop
     12.97 ±  2%      -0.7       12.25 ±  5%  perf-profile.calltrace.cycles-pp.inet_release.__sock_release.sock_close.__fput.task_work_run
     12.26 ±  2%      -0.7       11.56 ±  5%  perf-profile.calltrace.cycles-pp.tcp_close.inet_release.__sock_release.sock_close.__fput
      7.26            -0.7        6.58 ±  6%  perf-profile.calltrace.cycles-pp.tcp_v4_connect.__inet_stream_connect.inet_stream_connect.__sys_connect.__x64_sys_connect
      7.26 ±  2%      -0.6        6.68 ±  6%  perf-profile.calltrace.cycles-pp.release_sock.__inet_stream_connect.inet_stream_connect.__sys_connect.__x64_sys_connect
      7.22 ±  2%      -0.6        6.64 ±  6%  perf-profile.calltrace.cycles-pp.__release_sock.release_sock.__inet_stream_connect.inet_stream_connect.__sys_connect
      7.17 ±  2%      -0.6        6.60 ±  6%  perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.__release_sock.release_sock.__inet_stream_connect.inet_stream_connect
      7.74 ±  2%      -0.6        7.17 ±  5%  perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_close.inet_release
      6.08            -0.5        5.54 ±  5%  perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked
      9.95            -0.5        9.43 ±  5%  perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver
      6.26            -0.5        5.74 ±  5%  perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg
      7.40 ±  2%      -0.5        6.89 ±  5%  perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_close
      5.72            -0.5        5.25 ±  6%  perf-profile.calltrace.cycles-pp.tcp_connect.tcp_v4_connect.__inet_stream_connect.inet_stream_connect.__sys_connect
      6.29 ±  2%      -0.4        5.84 ±  6%  perf-profile.calltrace.cycles-pp.tcp_rcv_state_process.tcp_v4_do_rcv.__release_sock.release_sock.__inet_stream_connect
      4.80            -0.4        4.37 ±  6%  perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_connect.tcp_v4_connect.__inet_stream_connect.inet_stream_connect
      4.48            -0.4        4.06 ±  6%  perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_connect.tcp_v4_connect.__inet_stream_connect
      4.07            -0.4        3.67 ±  6%  perf-profile.calltrace.cycles-pp.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_connect.tcp_v4_connect
      4.04            -0.4        3.65 ±  6%  perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_connect
      0.95 ± 48%      -0.4        0.56 ±  4%  perf-profile.calltrace.cycles-pp.__inet_lookup_established.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver
      5.66            -0.3        5.31 ±  6%  perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_rcv_state_process.tcp_v4_do_rcv.__release_sock.release_sock
      5.46            -0.3        5.12 ±  6%  perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_rcv_state_process.tcp_v4_do_rcv.__release_sock
      5.29            -0.3        4.96 ±  6%  perf-profile.calltrace.cycles-pp.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_rcv_state_process.tcp_v4_do_rcv
      5.16            -0.3        4.83 ±  5%  perf-profile.calltrace.cycles-pp.tcp_rcv_state_process.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
      1.36 ±  3%      -0.2        1.11 ±  5%  perf-profile.calltrace.cycles-pp.__release_sock.release_sock.tcp_sendmsg.sock_sendmsg.__sys_sendto
      1.30 ±  2%      -0.2        1.06 ±  5%  perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.__release_sock.release_sock.tcp_sendmsg.sock_sendmsg
      1.41 ±  3%      -0.2        1.18 ±  5%  perf-profile.calltrace.cycles-pp.release_sock.tcp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto
      1.22 ±  2%      -0.2        0.99 ±  5%  perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock.tcp_sendmsg
      0.99 ±  2%      -0.2        0.77 ±  4%  perf-profile.calltrace.cycles-pp.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock
      1.66 ±  3%      -0.2        1.47 ±  6%  perf-profile.calltrace.cycles-pp.__release_sock.tcp_close.inet_release.__sock_release.sock_close
      1.63 ±  3%      -0.2        1.45 ±  6%  perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.__release_sock.tcp_close.inet_release.__sock_release
      1.62 ±  3%      -0.2        1.45 ±  6%  perf-profile.calltrace.cycles-pp.tcp_rcv_state_process.tcp_v4_do_rcv.__release_sock.tcp_close.inet_release
      2.14            -0.2        1.97 ±  5%  perf-profile.calltrace.cycles-pp.tcp_conn_request.tcp_rcv_state_process.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu
      1.09            -0.2        0.93 ±  4%  perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu
      2.17            -0.1        2.03 ±  5%  perf-profile.calltrace.cycles-pp.tcp_fin.tcp_data_queue.tcp_rcv_state_process.tcp_v4_do_rcv.tcp_v4_rcv
      2.15 ±  3%      -0.1        2.02 ±  6%  perf-profile.calltrace.cycles-pp.tcp_v4_syn_recv_sock.tcp_check_req.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
      2.19            -0.1        2.06 ±  5%  perf-profile.calltrace.cycles-pp.tcp_data_queue.tcp_rcv_state_process.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu
      0.86 ±  2%      -0.1        0.74 ±  6%  perf-profile.calltrace.cycles-pp.tcp_rcv_state_process.tcp_child_process.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
      1.65            -0.1        1.53 ±  7%  perf-profile.calltrace.cycles-pp.__sched_text_start.schedule_idle.do_idle.cpu_startup_entry.start_secondary
      1.69            -0.1        1.58 ±  6%  perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      0.93 ±  2%      -0.1        0.85 ±  5%  perf-profile.calltrace.cycles-pp.inet_create.__sock_create.__sys_socket.__x64_sys_socket.do_syscall_64
      0.70 ±  3%      -0.1        0.62 ±  5%  perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv
      0.64 ±  5%      +0.1        0.74 ±  5%  perf-profile.calltrace.cycles-pp.tcp_send_fin.tcp_close.inet_release.__sock_release.sock_close
      0.26 ±100%      +0.3        0.56 ±  5%  perf-profile.calltrace.cycles-pp.security_file_free.__fput.task_work_run.exit_to_usermode_loop.do_syscall_64
      0.13 ±173%      +0.4        0.55 ±  6%  perf-profile.calltrace.cycles-pp.apparmor_file_free_security.security_file_free.__fput.task_work_run.exit_to_usermode_loop
     26.31            -2.1       24.23 ±  5%  perf-profile.children.cycles-pp.__tcp_transmit_skb
     24.71            -1.9       22.79 ±  5%  perf-profile.children.cycles-pp.__ip_queue_xmit
     23.70            -1.8       21.90 ±  5%  perf-profile.children.cycles-pp.ip_output
     20.48            -1.7       18.76 ±  5%  perf-profile.children.cycles-pp.ip_rcv
     20.73            -1.7       19.02 ±  5%  perf-profile.children.cycles-pp.__netif_receive_skb_one_core
     22.71            -1.7       21.00 ±  5%  perf-profile.children.cycles-pp.__softirqentry_text_start
     21.04            -1.7       19.34 ±  5%  perf-profile.children.cycles-pp.process_backlog
     21.24            -1.7       19.55 ±  5%  perf-profile.children.cycles-pp.net_rx_action
     21.55            -1.7       19.87 ±  5%  perf-profile.children.cycles-pp.do_softirq_own_stack
     23.02            -1.7       21.37 ±  5%  perf-profile.children.cycles-pp.ip_finish_output2
     21.69            -1.6       20.05 ±  5%  perf-profile.children.cycles-pp.do_softirq
     21.86            -1.6       20.23 ±  5%  perf-profile.children.cycles-pp.__local_bh_enable_ip
     17.04            -1.5       15.55 ±  5%  perf-profile.children.cycles-pp.tcp_write_xmit
     17.06            -1.5       15.57 ±  5%  perf-profile.children.cycles-pp.__tcp_push_pending_frames
     20.36            -1.5       18.91 ±  5%  perf-profile.children.cycles-pp.tcp_v4_do_rcv
     19.12            -1.4       17.69 ±  5%  perf-profile.children.cycles-pp.ip_local_deliver_finish
     19.09            -1.4       17.66 ±  5%  perf-profile.children.cycles-pp.ip_protocol_deliver_rcu
     19.17            -1.4       17.75 ±  5%  perf-profile.children.cycles-pp.ip_local_deliver
     14.66            -1.2       13.41 ±  6%  perf-profile.children.cycles-pp.__inet_stream_connect
     18.50            -1.2       17.25 ±  5%  perf-profile.children.cycles-pp.tcp_v4_rcv
     14.72            -1.2       13.48 ±  6%  perf-profile.children.cycles-pp.inet_stream_connect
     14.89 ±  2%      -1.2       13.67 ±  6%  perf-profile.children.cycles-pp.__x64_sys_connect
     14.88            -1.2       13.65 ±  6%  perf-profile.children.cycles-pp.__sys_connect
     13.95            -1.1       12.87 ±  6%  perf-profile.children.cycles-pp.tcp_rcv_state_process
     11.08            -1.0       10.10 ±  5%  perf-profile.children.cycles-pp.__x64_sys_sendto
     10.64            -1.0        9.67 ±  5%  perf-profile.children.cycles-pp.tcp_sendmsg
     10.93            -1.0        9.95 ±  5%  perf-profile.children.cycles-pp.sock_sendmsg
     11.04            -1.0       10.06 ±  5%  perf-profile.children.cycles-pp.__sys_sendto
     10.54 ±  2%      -0.9        9.61 ±  6%  perf-profile.children.cycles-pp.__release_sock
      9.17            -0.7        8.43 ±  5%  perf-profile.children.cycles-pp.tcp_sendmsg_locked
     13.11 ±  2%      -0.7       12.38 ±  5%  perf-profile.children.cycles-pp.sock_close
     13.09 ±  2%      -0.7       12.37 ±  5%  perf-profile.children.cycles-pp.__sock_release
     12.97 ±  2%      -0.7       12.25 ±  5%  perf-profile.children.cycles-pp.inet_release
      9.25 ±  2%      -0.7        8.54 ±  6%  perf-profile.children.cycles-pp.release_sock
     12.26 ±  2%      -0.7       11.56 ±  5%  perf-profile.children.cycles-pp.tcp_close
      7.26            -0.7        6.58 ±  6%  perf-profile.children.cycles-pp.tcp_v4_connect
      1.14 ±  5%      -0.5        0.61 ±  4%  perf-profile.children.cycles-pp.inet_ehashfn
      5.73            -0.5        5.26 ±  6%  perf-profile.children.cycles-pp.tcp_connect
      2.66 ±  2%      -0.4        2.26 ±  5%  perf-profile.children.cycles-pp.tcp_ack
      1.81 ±  4%      -0.4        1.42 ±  5%  perf-profile.children.cycles-pp.__inet_lookup_established
      6.20            -0.3        5.85 ±  5%  perf-profile.children.cycles-pp.tcp_rcv_established
      1.46 ±  3%      -0.3        1.20 ±  6%  perf-profile.children.cycles-pp.ip_route_output_key_hash_rcu
      1.50 ±  3%      -0.3        1.25 ±  6%  perf-profile.children.cycles-pp.ip_route_output_key_hash
      0.81 ±  3%      -0.2        0.59 ±  6%  perf-profile.children.cycles-pp.tcp_get_metrics
      1.09 ±  3%      -0.2        0.89 ±  6%  perf-profile.children.cycles-pp.ip_route_output_flow
      1.04 ±  2%      -0.2        0.86 ±  7%  perf-profile.children.cycles-pp.tcp_init_transfer
      2.14            -0.2        1.97 ±  5%  perf-profile.children.cycles-pp.tcp_conn_request
      0.59 ±  2%      -0.2        0.43 ±  6%  perf-profile.children.cycles-pp.tcp_update_metrics
      0.57            -0.2        0.40 ±  9%  perf-profile.children.cycles-pp.ip_rcv_core
      0.30 ±  4%      -0.2        0.14 ±  7%  perf-profile.children.cycles-pp.inet_lookup_ifaddr_rcu
      0.31 ±  5%      -0.2        0.15 ±  8%  perf-profile.children.cycles-pp.__ip_dev_find
      0.76 ±  2%      -0.2        0.61 ±  3%  perf-profile.children.cycles-pp.tcp_set_state
      0.58 ±  2%      -0.1        0.43 ±  8%  perf-profile.children.cycles-pp.tcp_init_metrics
      0.21 ±  5%      -0.1        0.07 ±  6%  perf-profile.children.cycles-pp.sock_prot_inuse_add
      0.67 ±  2%      -0.1        0.53 ±  7%  perf-profile.children.cycles-pp.ip_rcv_finish_core
      0.41 ±  2%      -0.1        0.27 ±  4%  perf-profile.children.cycles-pp.__tcp_select_window
      2.15 ±  3%      -0.1        2.02 ±  6%  perf-profile.children.cycles-pp.tcp_v4_syn_recv_sock
      0.69 ±  2%      -0.1        0.56 ±  7%  perf-profile.children.cycles-pp.ip_rcv_finish
      1.72            -0.1        1.59 ±  6%  perf-profile.children.cycles-pp.schedule_idle
      0.31 ±  5%      -0.1        0.20 ±  8%  perf-profile.children.cycles-pp.tcp_sync_mss
      0.58 ±  3%      -0.1        0.47 ±  7%  perf-profile.children.cycles-pp.tcp_time_wait
      0.27 ±  2%      -0.1        0.16 ±  2%  perf-profile.children.cycles-pp.tcp_rack_update_reo_wnd
      0.55 ±  2%      -0.1        0.44 ±  6%  perf-profile.children.cycles-pp.__inet_hash_connect
      0.34 ±  2%      -0.1        0.23 ±  5%  perf-profile.children.cycles-pp.tcp_schedule_loss_probe
      0.29 ±  2%      -0.1        0.18 ±  5%  perf-profile.children.cycles-pp.tcp_newly_delivered
      0.17 ±  7%      -0.1        0.06 ±  6%  perf-profile.children.cycles-pp.__sk_free
      0.57 ±  4%      -0.1        0.47 ±  7%  perf-profile.children.cycles-pp.tcp_finish_connect
      0.31 ±  4%      -0.1        0.21 ±  3%  perf-profile.children.cycles-pp.__ip_finish_output
      0.32 ±  3%      -0.1        0.24 ±  4%  perf-profile.children.cycles-pp.inet_unhash
      0.93 ±  2%      -0.1        0.85 ±  6%  perf-profile.children.cycles-pp.inet_create
      0.20 ±  8%      -0.1        0.13 ±  5%  perf-profile.children.cycles-pp.inet_sock_destruct
      0.74 ±  2%      -0.1        0.68 ±  4%  perf-profile.children.cycles-pp.tcp_done
      0.28            -0.1        0.21 ± 11%  perf-profile.children.cycles-pp.tcp_parse_options
      0.29 ±  5%      -0.1        0.23        perf-profile.children.cycles-pp.__inet_inherit_port
      0.44 ±  2%      -0.1        0.38 ±  5%  perf-profile.children.cycles-pp.inet_csk_route_child_sock
      0.32 ±  5%      -0.1        0.26 ± 10%  perf-profile.children.cycles-pp.__inet_check_established
      0.45 ±  5%      -0.1        0.38 ±  6%  perf-profile.children.cycles-pp.inet_csk_route_req
      0.57 ±  3%      -0.1        0.52 ±  6%  perf-profile.children.cycles-pp.dst_release
      0.17 ±  5%      -0.1        0.11 ±  9%  perf-profile.children.cycles-pp.tcp_mtup_init
      0.22 ±  5%      -0.1        0.16 ± 17%  perf-profile.children.cycles-pp.__inet_lookup_listener
      0.46 ±  2%      -0.1        0.40 ±  5%  perf-profile.children.cycles-pp.sk_alloc
      0.28 ±  4%      -0.1        0.23 ±  5%  perf-profile.children.cycles-pp.inet_put_port
      0.08 ±  5%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.inet_ehash_nolisten
      0.32            -0.0        0.28 ± 10%  perf-profile.children.cycles-pp.tcp_ack_update_rtt
      0.29 ±  5%      -0.0        0.25 ±  4%  perf-profile.children.cycles-pp.inet_ehash_insert
      0.17 ±  2%      -0.0        0.12 ±  4%  perf-profile.children.cycles-pp.tcp_stream_memory_free
      0.13 ±  5%      -0.0        0.09 ±  8%  perf-profile.children.cycles-pp.tcp_select_initial_window
      0.32 ±  6%      -0.0        0.29 ±  3%  perf-profile.children.cycles-pp.inet_csk_reqsk_queue_hash_add
      0.27 ±  4%      -0.0        0.23 ±  8%  perf-profile.children.cycles-pp.rebalance_domains
      0.26 ±  3%      -0.0        0.23 ±  7%  perf-profile.children.cycles-pp.tcp_update_pacing_rate
      0.15 ±  7%      -0.0        0.12 ±  6%  perf-profile.children.cycles-pp.secure_tcp_ts_off
      0.10 ±  4%      -0.0        0.07 ± 10%  perf-profile.children.cycles-pp.ip_build_and_send_pkt
      0.20 ±  5%      -0.0        0.17 ±  6%  perf-profile.children.cycles-pp._raw_spin_trylock
      0.09 ±  4%      -0.0        0.06 ± 13%  perf-profile.children.cycles-pp.tcp_push
      0.11 ±  4%      -0.0        0.09 ±  4%  perf-profile.children.cycles-pp.sock_alloc_inode
      0.08 ±  5%      -0.0        0.07 ± 13%  perf-profile.children.cycles-pp.tcp_sndbuf_expand
      0.13 ±  3%      -0.0        0.12 ±  3%  perf-profile.children.cycles-pp.tick_nohz_idle_enter
      0.16 ±  2%      +0.0        0.18 ±  2%  perf-profile.children.cycles-pp.__ip_local_out
      0.29 ±  6%      +0.0        0.33 ±  5%  perf-profile.children.cycles-pp.update_curr
      0.16 ±  2%      +0.0        0.20 ±  9%  perf-profile.children.cycles-pp.sk_stream_kill_queues
      0.11 ± 12%      +0.0        0.15 ±  8%  perf-profile.children.cycles-pp.tcp_check_oom
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.tcp_rearm_rto
      0.00            +0.1        0.05 ±  8%  perf-profile.children.cycles-pp.woken_wake_function
      0.00            +0.1        0.06 ±  9%  perf-profile.children.cycles-pp.deactivate_task
      0.36 ±  3%      +0.1        0.42 ±  6%  perf-profile.children.cycles-pp.apparmor_socket_post_create
      0.27            +0.1        0.33 ±  8%  perf-profile.children.cycles-pp.tcp_write_queue_purge
      0.50 ±  3%      +0.1        0.56 ±  6%  perf-profile.children.cycles-pp.security_file_free
      0.48 ±  3%      +0.1        0.55 ±  6%  perf-profile.children.cycles-pp.apparmor_file_free_security
      0.46 ±  2%      +0.1        0.53 ±  6%  perf-profile.children.cycles-pp.sk_stream_alloc_skb
      0.64 ±  5%      +0.1        0.74 ±  5%  perf-profile.children.cycles-pp.tcp_send_fin
      0.80 ±  5%      +0.1        0.93 ±  8%  perf-profile.children.cycles-pp.percpu_counter_add_batch
      0.69            +0.2        0.87 ±  8%  perf-profile.children.cycles-pp.__sk_mem_reduce_allocated
      0.65 ±  6%      +0.2        0.83 ±  7%  perf-profile.children.cycles-pp.sk_forced_mem_schedule
      1.12 ±  5%      -0.5        0.60 ±  5%  perf-profile.self.cycles-pp.inet_ehashfn
      0.36 ±  8%      -0.2        0.15 ±  7%  perf-profile.self.cycles-pp.tcp_get_metrics
      1.22            -0.2        1.03 ±  4%  perf-profile.self.cycles-pp.__tcp_transmit_skb
      0.56 ±  4%      -0.2        0.38 ±  4%  perf-profile.self.cycles-pp.ip_protocol_deliver_rcu
      0.71 ±  2%      -0.2        0.53 ±  7%  perf-profile.self.cycles-pp.__ip_queue_xmit
      0.56            -0.2        0.40 ±  8%  perf-profile.self.cycles-pp.ip_rcv_core
      0.29 ±  6%      -0.2        0.14 ±  7%  perf-profile.self.cycles-pp.inet_lookup_ifaddr_rcu
      0.61 ±  4%      -0.2        0.46 ±  7%  perf-profile.self.cycles-pp.ip_output
      0.21 ±  5%      -0.1        0.07 ±  7%  perf-profile.self.cycles-pp.sock_prot_inuse_add
      0.41 ±  2%      -0.1        0.27 ±  4%  perf-profile.self.cycles-pp.__tcp_select_window
      0.66 ±  2%      -0.1        0.53 ±  8%  perf-profile.self.cycles-pp.ip_rcv_finish_core
      0.99 ±  3%      -0.1        0.86 ±  5%  perf-profile.self.cycles-pp.ip_route_output_key_hash_rcu
      0.31 ±  6%      -0.1        0.19 ±  7%  perf-profile.self.cycles-pp.tcp_sync_mss
      0.89 ±  3%      -0.1        0.77 ±  4%  perf-profile.self.cycles-pp.__sched_text_start
      0.27 ±  2%      -0.1        0.16 ±  4%  perf-profile.self.cycles-pp.tcp_rack_update_reo_wnd
      0.16 ± 10%      -0.1        0.05 ±  8%  perf-profile.self.cycles-pp.__sk_free
      0.30 ±  2%      -0.1        0.20 ±  5%  perf-profile.self.cycles-pp.tcp_schedule_loss_probe
      0.28            -0.1        0.18 ±  8%  perf-profile.self.cycles-pp.tcp_newly_delivered
      0.26 ±  3%      -0.1        0.16 ±  5%  perf-profile.self.cycles-pp.__ip_finish_output
      0.28            -0.1        0.20 ± 10%  perf-profile.self.cycles-pp.tcp_parse_options
      0.23 ±  3%      -0.1        0.16 ±  7%  perf-profile.self.cycles-pp.tcp_set_state
      0.10 ±  9%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.inet_put_port
      0.21 ±  8%      -0.1        0.14 ±  9%  perf-profile.self.cycles-pp.tcp_update_metrics
      0.24 ±  6%      -0.1        0.17 ±  4%  perf-profile.self.cycles-pp.sk_clone_lock
      0.09 ±  4%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.__inet_lookup_listener
      0.87            -0.1        0.80 ±  6%  perf-profile.self.cycles-pp.ipv4_dst_check
      0.18 ±  4%      -0.1        0.12 ±  6%  perf-profile.self.cycles-pp.__inet_hash_connect
      0.63            -0.1        0.57 ±  6%  perf-profile.self.cycles-pp.tcp_ack
      0.26 ±  3%      -0.1        0.20 ±  5%  perf-profile.self.cycles-pp.tcp_event_new_data_sent
      0.58 ±  2%      -0.1        0.52 ±  4%  perf-profile.self.cycles-pp.tcp_v4_rcv
      0.57 ±  3%      -0.1        0.51 ±  6%  perf-profile.self.cycles-pp.dst_release
      0.17 ±  5%      -0.1        0.11 ±  7%  perf-profile.self.cycles-pp.tcp_mtup_init
      0.16            -0.1        0.10 ±  4%  perf-profile.self.cycles-pp.inet_csk_get_port
      0.30 ±  3%      -0.0        0.25 ± 10%  perf-profile.self.cycles-pp.tcp_conn_request
      0.28 ±  3%      -0.0        0.23 ±  9%  perf-profile.self.cycles-pp.tcp_ack_update_rtt
      0.29 ±  5%      -0.0        0.25 ±  5%  perf-profile.self.cycles-pp.sk_alloc
      0.13 ±  5%      -0.0        0.09 ±  7%  perf-profile.self.cycles-pp.tcp_select_initial_window
      0.09 ±  7%      -0.0        0.05 ± 58%  perf-profile.self.cycles-pp.inet_twsk_kill
      0.16 ±  2%      -0.0        0.12 ±  4%  perf-profile.self.cycles-pp.tcp_stream_memory_free
      0.13 ±  5%      -0.0        0.10 ±  9%  perf-profile.self.cycles-pp.secure_tcp_ts_off
      0.07 ± 10%      -0.0        0.04 ± 57%  perf-profile.self.cycles-pp.inet_create
      0.15 ±  4%      -0.0        0.12 ± 13%  perf-profile.self.cycles-pp.tcp_init_metrics
      0.41            -0.0        0.38 ±  5%  perf-profile.self.cycles-pp.tcp_write_xmit
      0.25 ±  3%      -0.0        0.23 ±  6%  perf-profile.self.cycles-pp.tcp_update_pacing_rate
      0.20 ±  5%      -0.0        0.17 ±  6%  perf-profile.self.cycles-pp._raw_spin_trylock
      0.10 ±  5%      -0.0        0.07 ± 10%  perf-profile.self.cycles-pp.ip_build_and_send_pkt
      0.14 ±  6%      -0.0        0.12 ±  9%  perf-profile.self.cycles-pp.tcp_init_sock
      0.08 ±  5%      -0.0        0.06 ± 13%  perf-profile.self.cycles-pp.tcp_push
      0.15 ±  8%      -0.0        0.13 ±  8%  perf-profile.self.cycles-pp.__inet_bind
      0.08 ±  5%      -0.0        0.07 ± 13%  perf-profile.self.cycles-pp.tcp_sndbuf_expand
      0.09 ±  4%      -0.0        0.07 ± 10%  perf-profile.self.cycles-pp.exit_to_usermode_loop
      0.14 ±  3%      -0.0        0.12 ±  5%  perf-profile.self.cycles-pp.dequeue_entity
      0.16 ±  2%      +0.0        0.18 ±  2%  perf-profile.self.cycles-pp.process_backlog
      0.12 ±  3%      +0.0        0.14 ±  3%  perf-profile.self.cycles-pp.ip_send_check
      0.06            +0.0        0.08 ±  6%  perf-profile.self.cycles-pp.tcp_data_queue
      0.06 ±  6%      +0.0        0.08 ±  5%  perf-profile.self.cycles-pp.tcp_queue_rcv
      0.08            +0.0        0.10 ±  7%  perf-profile.self.cycles-pp.tcp_openreq_init_rwin
      0.15 ±  7%      +0.0        0.17 ±  4%  perf-profile.self.cycles-pp.update_curr
      0.32 ±  2%      +0.0        0.35 ±  3%  perf-profile.self.cycles-pp.__dev_queue_xmit
      0.11 ± 12%      +0.0        0.15 ±  8%  perf-profile.self.cycles-pp.tcp_check_oom
      0.00            +0.1        0.05 ±  8%  perf-profile.self.cycles-pp.deactivate_task
      0.48 ±  3%      +0.1        0.55 ±  6%  perf-profile.self.cycles-pp.apparmor_file_free_security
      0.79 ±  5%      +0.1        0.93 ±  8%  perf-profile.self.cycles-pp.percpu_counter_add_batch
      0.69 ±  2%      +0.2        0.86 ±  7%  perf-profile.self.cycles-pp.__sk_mem_reduce_allocated
      0.64 ±  6%      +0.2        0.82 ±  7%  perf-profile.self.cycles-pp.sk_forced_mem_schedule



***************************************************************************************************
lkp-knm01: 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode:
  gcc-7/performance/x86_64-rhel-7.6/process/100%/debian-x86_64-20191114.cgz/lkp-knm01/signal1/will-it-scale/0x11

commit: 
  c5f8689118 (" RISC-V Fixes for 5.6-rc4")
  fda31c5029 ("signal: avoid double atomic counter increments for user accounting")

c5f86891185c408b fda31c50292a5062332fa0343c0 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
      3067          +552.8%      20025        will-it-scale.per_process_ops
    340.21            -7.5%     314.76        will-it-scale.time.elapsed_time
    340.21            -7.5%     314.76        will-it-scale.time.elapsed_time.max
      1355           -39.6%     818.25 ±  6%  will-it-scale.time.involuntary_context_switches
      9.00           -63.9%       3.25 ± 13%  will-it-scale.time.percent_of_cpu_this_job_got
     25.68           -73.4%       6.82 ±  8%  will-it-scale.time.system_time
    883653          +552.7%    5767476        will-it-scale.workload
     26.46            +3.6%      27.42 ±  3%  boot-time.dhcp
 1.657e+09           -30.6%   1.15e+09        cpuidle.C1.time
   3193373 ± 14%     -25.0%    2396004 ±  2%  cpuidle.C1.usage
      1.73 ±  4%      -0.5        1.27 ±  3%  mpstat.cpu.all.idle%
      0.04 ±  4%      -0.0        0.00 ± 31%  mpstat.cpu.all.soft%
     95.77           -10.6       85.17        mpstat.cpu.all.sys%
      2.46           +11.1       13.56        mpstat.cpu.all.usr%
     95.00           -11.3%      84.25        vmstat.cpu.sy
      2.00          +550.0%      13.00        vmstat.cpu.us
      1944           +16.2%       2260 ±  2%  vmstat.system.cs
    470108 ± 15%     +24.8%     586837        vmstat.system.in
    123551           +16.8%     144321        numa-vmstat.node0.nr_active_anon
     36751 ±  7%     -85.1%       5494        numa-vmstat.node0.nr_inactive_anon
     46933           -65.5%      16169 ± 42%  numa-vmstat.node0.nr_mapped
     41995 ±  2%     -11.3%      37270        numa-vmstat.node0.nr_shmem
    123552           +16.8%     144323        numa-vmstat.node0.nr_zone_active_anon
     36750 ±  7%     -85.0%       5494        numa-vmstat.node0.nr_zone_inactive_anon
    493669           +16.9%     577281        meminfo.Active
    493599           +16.9%     577219        meminfo.Active(anon)
    115818 ±  7%     +57.5%     182439 ±  6%  meminfo.AnonHugePages
    148593 ±  7%     -85.1%      22201        meminfo.Inactive
    148358 ±  7%     -85.2%      21980        meminfo.Inactive(anon)
    200638           -62.5%      75186 ± 37%  meminfo.Mapped
    169522 ±  2%     -12.0%     149183        meminfo.Shmem
    493298           +16.9%     576732        numa-meminfo.node0.Active
    493228           +16.9%     576670        numa-meminfo.node0.Active(anon)
    115754 ±  7%     +57.5%     182348 ±  6%  numa-meminfo.node0.AnonHugePages
    147885 ±  7%     -85.0%      22200        numa-meminfo.node0.Inactive
    147650 ±  7%     -85.1%      21978        numa-meminfo.node0.Inactive(anon)
    188036           -65.8%      64244 ± 42%  numa-meminfo.node0.Mapped
   2596576            -9.8%    2340996 ±  2%  numa-meminfo.node0.MemUsed
    168709 ±  2%     -11.7%     148991        numa-meminfo.node0.Shmem
      1642 ± 15%     -21.1%       1295 ± 10%  slabinfo.Acpi-Parse.active_objs
      1642 ± 15%     -21.1%       1295 ± 10%  slabinfo.Acpi-Parse.num_objs
      1761 ±  4%      -8.0%       1620 ±  8%  slabinfo.file_lock_cache.active_objs
      1761 ±  4%      -8.0%       1620 ±  8%  slabinfo.file_lock_cache.num_objs
     57952 ±  3%      -8.9%      52785 ±  5%  slabinfo.filp.active_objs
     57966 ±  3%      -8.8%      52839 ±  5%  slabinfo.filp.num_objs
      1147 ±  6%     -14.4%     981.75 ± 11%  slabinfo.mnt_cache.active_objs
      1147 ±  6%     -14.4%     981.75 ± 11%  slabinfo.mnt_cache.num_objs
      1233 ± 10%     -12.6%       1078 ±  3%  slabinfo.task_group.active_objs
      1233 ± 10%     -12.6%       1078 ±  3%  slabinfo.task_group.num_objs
    123402           +17.0%     144354        proc-vmstat.nr_active_anon
    118969            -4.5%     113609        proc-vmstat.nr_anon_pages
    311079            -1.7%     305856        proc-vmstat.nr_file_pages
     37156 ±  7%     -85.2%       5494        proc-vmstat.nr_inactive_anon
     50253           -62.1%      19066 ± 37%  proc-vmstat.nr_mapped
    433.00           -49.8%     217.25 ± 96%  proc-vmstat.nr_mlock
     42455 ±  2%     -12.0%      37362        proc-vmstat.nr_shmem
     29831            -1.4%      29426        proc-vmstat.nr_slab_reclaimable
     78999            -2.4%      77135        proc-vmstat.nr_slab_unreclaimable
    123402           +17.0%     144354        proc-vmstat.nr_zone_active_anon
     37156 ±  7%     -85.2%       5494        proc-vmstat.nr_zone_inactive_anon
      3085 ± 13%     -46.9%       1638 ± 75%  proc-vmstat.numa_hint_faults
      3085 ± 13%     -46.9%       1638 ± 75%  proc-vmstat.numa_hint_faults_local
    902639            +2.4%     924149        proc-vmstat.numa_hit
    902639            +2.4%     924149        proc-vmstat.numa_local
    124435 ±  7%     -12.6%     108771 ± 10%  proc-vmstat.numa_pte_updates
    686.50 ± 24%    +443.0%       3727 ±  6%  proc-vmstat.pgactivate
    966850            +3.7%    1002458        proc-vmstat.pgalloc_normal
    833907            +3.5%     863467        proc-vmstat.pgfault
    110012 ± 22%     -72.3%      30477 ± 38%  sched_debug.cfs_rq:/.MIN_vruntime.avg
  28555041 ± 13%     -69.3%    8777414 ± 38%  sched_debug.cfs_rq:/.MIN_vruntime.max
   1729376 ± 16%     -70.1%     516315 ± 38%  sched_debug.cfs_rq:/.MIN_vruntime.stddev
    157262           -18.1%     128746        sched_debug.cfs_rq:/.exec_clock.avg
    161920           -18.3%     132304        sched_debug.cfs_rq:/.exec_clock.max
     80370 ±  2%     +28.9%     103558 ±  8%  sched_debug.cfs_rq:/.exec_clock.min
      5902           -62.1%       2234 ± 24%  sched_debug.cfs_rq:/.exec_clock.stddev
      6889 ±  3%     -35.6%       4434 ± 11%  sched_debug.cfs_rq:/.load.avg
   1010490 ±  7%     -65.0%     354092 ± 32%  sched_debug.cfs_rq:/.load.max
     60658 ±  7%     -63.5%      22142 ± 33%  sched_debug.cfs_rq:/.load.stddev
      6.31 ±  2%     -34.2%       4.15 ± 13%  sched_debug.cfs_rq:/.load_avg.avg
    802.83 ±  4%     -61.6%     308.55 ± 32%  sched_debug.cfs_rq:/.load_avg.max
     48.94 ±  4%     -59.3%      19.93 ± 32%  sched_debug.cfs_rq:/.load_avg.stddev
    110013 ± 22%     -72.3%      30477 ± 38%  sched_debug.cfs_rq:/.max_vruntime.avg
  28555439 ± 13%     -69.3%    8777417 ± 38%  sched_debug.cfs_rq:/.max_vruntime.max
   1729399 ± 16%     -70.1%     516315 ± 38%  sched_debug.cfs_rq:/.max_vruntime.stddev
  52675531           -17.0%   43694414        sched_debug.cfs_rq:/.min_vruntime.avg
  54455320           -17.6%   44888232        sched_debug.cfs_rq:/.min_vruntime.max
  25078264 ±  4%     +36.4%   34197680 ±  8%  sched_debug.cfs_rq:/.min_vruntime.min
   2299146           -55.8%    1015351 ± 17%  sched_debug.cfs_rq:/.min_vruntime.stddev
      1.92 ±  4%     -32.2%       1.30 ±  7%  sched_debug.cfs_rq:/.nr_running.max
      0.67           +20.0%       0.80        sched_debug.cfs_rq:/.nr_running.min
      0.08 ±  8%     -47.9%       0.04 ± 16%  sched_debug.cfs_rq:/.nr_running.stddev
      4.82 ±  7%     -64.4%       1.71 ± 19%  sched_debug.cfs_rq:/.nr_spread_over.avg
    178.50           -45.7%      96.95 ± 18%  sched_debug.cfs_rq:/.nr_spread_over.max
     15.91 ±  2%     -50.2%       7.92 ± 19%  sched_debug.cfs_rq:/.nr_spread_over.stddev
      5.48 ±  2%     -32.7%       3.69 ± 12%  sched_debug.cfs_rq:/.runnable_load_avg.avg
    800.58 ±  4%     -62.9%     297.00 ± 33%  sched_debug.cfs_rq:/.runnable_load_avg.max
     47.32 ±  4%     -60.9%      18.52 ± 34%  sched_debug.cfs_rq:/.runnable_load_avg.stddev
      6874 ±  3%     -35.5%       4432 ± 11%  sched_debug.cfs_rq:/.runnable_weight.avg
   1009164 ±  7%     -64.9%     354035 ± 32%  sched_debug.cfs_rq:/.runnable_weight.max
     60580 ±  7%     -63.5%      22139 ± 33%  sched_debug.cfs_rq:/.runnable_weight.stddev
      1.39 ± 97%     -99.2%       0.01 ± 34%  sched_debug.cfs_rq:/.spread.avg
    399.69 ± 97%     -99.2%       3.00 ± 34%  sched_debug.cfs_rq:/.spread.max
     23.51 ± 97%     -99.2%       0.18 ± 34%  sched_debug.cfs_rq:/.spread.stddev
   1435730 ± 50%     -99.7%       4004 ±4085%  sched_debug.cfs_rq:/.spread0.avg
   2590437 ± 27%     -81.9%     468538 ± 30%  sched_debug.cfs_rq:/.spread0.max
 -26578845           -62.0%  -10090079        sched_debug.cfs_rq:/.spread0.min
   2318545           -61.7%     887260 ± 20%  sched_debug.cfs_rq:/.spread0.stddev
    211.50 ± 49%    +104.9%     433.30 ± 10%  sched_debug.cfs_rq:/.util_avg.min
     74.21 ±  4%     -18.1%      60.76 ±  3%  sched_debug.cfs_rq:/.util_avg.stddev
    897.67 ±  5%     -21.0%     709.55 ± 15%  sched_debug.cfs_rq:/.util_est_enqueued.max
    172.54           -13.1%     149.94 ±  2%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
     79667 ± 12%    +914.5%     808208        sched_debug.cpu.avg_idle.min
    212992           -13.6%     184095        sched_debug.cpu.clock.avg
    215395           -13.2%     186974        sched_debug.cpu.clock.max
    210678           -14.0%     181244        sched_debug.cpu.clock.min
      1344 ±  4%     +23.5%       1660 ±  8%  sched_debug.cpu.clock.stddev
    212992           -13.6%     184095        sched_debug.cpu.clock_task.avg
    215395           -13.2%     186974        sched_debug.cpu.clock_task.max
    210678           -14.0%     181244        sched_debug.cpu.clock_task.min
      1344 ±  4%     +23.5%       1660 ±  8%  sched_debug.cpu.clock_task.stddev
      7154           -21.4%       5623 ± 13%  sched_debug.cpu.curr->pid.max
      2170 ±  6%     +25.5%       2724 ±  2%  sched_debug.cpu.curr->pid.min
    434.69 ±  8%     -36.1%     277.74 ± 17%  sched_debug.cpu.curr->pid.stddev
    847958 ±  6%      -6.1%     796043 ±  6%  sched_debug.cpu.max_idle_balance_cost.avg
   3727165 ±  5%     -16.6%    3109400 ± 12%  sched_debug.cpu.max_idle_balance_cost.max
    532179 ±  2%     -38.6%     326898 ±  3%  sched_debug.cpu.max_idle_balance_cost.stddev
      0.00 ±  2%     -58.6%       0.00 ± 18%  sched_debug.cpu.next_balance.stddev
      2.67           -19.4%       2.15 ±  7%  sched_debug.cpu.nr_running.max
      0.16           -31.6%       0.11 ± 14%  sched_debug.cpu.nr_running.stddev
     23319 ±  9%     +53.8%      35861 ± 17%  sched_debug.cpu.nr_switches.max
    496.92 ±  8%     +90.3%     945.60 ± 35%  sched_debug.cpu.nr_switches.min
      1266           -11.0%       1127        sched_debug.cpu.sched_count.avg
     18347 ± 18%     +73.7%      31876 ± 18%  sched_debug.cpu.sched_count.max
      2500 ±  3%     +16.6%       2915 ±  9%  sched_debug.cpu.sched_count.stddev
     69.50           -29.8%      48.80 ±  3%  sched_debug.cpu.sched_goidle.avg
    225.86 ±  3%      -8.7%     206.16        sched_debug.cpu.sched_goidle.stddev
      8740 ± 15%     +67.9%      14676 ± 21%  sched_debug.cpu.ttwu_count.max
    106.50           +14.0%     121.40 ±  5%  sched_debug.cpu.ttwu_count.min
      1127 ±  4%     +20.7%       1360 ± 10%  sched_debug.cpu.ttwu_count.stddev
      8094 ± 19%     +75.2%      14185 ± 22%  sched_debug.cpu.ttwu_local.max
    101.33           +16.3%     117.80 ±  5%  sched_debug.cpu.ttwu_local.min
      1033 ±  5%     +22.5%       1266 ± 11%  sched_debug.cpu.ttwu_local.stddev
    210660           -14.0%     181223        sched_debug.cpu_clk
    209992           -14.0%     180528        sched_debug.ktime
      0.00           -62.5%       0.00 ± 34%  sched_debug.rt_rq:/.rt_nr_migratory.avg
      0.67           -62.5%       0.25 ± 34%  sched_debug.rt_rq:/.rt_nr_migratory.max
      0.04           -62.5%       0.01 ± 34%  sched_debug.rt_rq:/.rt_nr_migratory.stddev
      0.00           -62.5%       0.00 ± 34%  sched_debug.rt_rq:/.rt_nr_running.avg
      0.67           -62.5%       0.25 ± 34%  sched_debug.rt_rq:/.rt_nr_running.max
      0.04           -62.5%       0.01 ± 34%  sched_debug.rt_rq:/.rt_nr_running.stddev
      2.03           -10.0%       1.82        sched_debug.rt_rq:/.rt_runtime.stddev
      1.05 ±  6%     -54.4%       0.48 ± 89%  sched_debug.rt_rq:/.rt_time.avg
    213376           -14.0%     183399        sched_debug.sched_clk
    126907           -15.5%     107271 ±  2%  softirqs.CPU0.TIMER
    127602           -19.4%     102862 ±  2%  softirqs.CPU10.TIMER
    128215           -15.3%     108633 ±  4%  softirqs.CPU11.TIMER
    125593            +8.1%     135781 ±  4%  softirqs.CPU112.TIMER
    126428           -12.4%     110694 ±  8%  softirqs.CPU118.TIMER
    127198           -17.5%     104927        softirqs.CPU12.TIMER
    127387           -18.2%     104211        softirqs.CPU13.TIMER
    127702           -20.7%     101213 ±  4%  softirqs.CPU14.TIMER
    126108           -14.4%     107987 ±  4%  softirqs.CPU140.TIMER
    151683 ±  2%      -6.5%     141860        softirqs.CPU143.RCU
    125664           -10.7%     112215 ±  7%  softirqs.CPU147.TIMER
    127409           -17.8%     104706 ±  2%  softirqs.CPU15.TIMER
    126694           -13.3%     109860 ±  3%  softirqs.CPU16.TIMER
    125411           -13.0%     109148 ±  4%  softirqs.CPU163.TIMER
    126918           -15.7%     106978 ±  2%  softirqs.CPU17.TIMER
    125827           +13.6%     142953 ± 10%  softirqs.CPU175.TIMER
    126114           -10.3%     113071 ±  5%  softirqs.CPU179.TIMER
    127109           -10.8%     113366 ±  7%  softirqs.CPU18.TIMER
    125761           -14.7%     107275 ±  5%  softirqs.CPU182.TIMER
    125964           -14.6%     107604 ±  6%  softirqs.CPU183.TIMER
    125292           -16.3%     104829 ±  7%  softirqs.CPU188.TIMER
    124946           -15.3%     105819 ±  5%  softirqs.CPU189.TIMER
    126958           -18.8%     103141        softirqs.CPU19.TIMER
    129119           -12.2%     113425 ±  5%  softirqs.CPU2.TIMER
    128004           -14.1%     109891 ±  4%  softirqs.CPU20.TIMER
    125490           -10.6%     112249 ±  4%  softirqs.CPU201.TIMER
    125498           -12.7%     109598 ±  3%  softirqs.CPU204.TIMER
    125217           -17.2%     103714 ±  3%  softirqs.CPU205.TIMER
    149348 ±  3%      -7.0%     138827 ±  2%  softirqs.CPU209.RCU
    128280           -13.0%     111590 ±  6%  softirqs.CPU21.TIMER
    124326           -10.4%     111455 ±  4%  softirqs.CPU211.TIMER
    124159           -10.3%     111419 ±  5%  softirqs.CPU217.TIMER
    126809           -12.2%     111343 ±  5%  softirqs.CPU22.TIMER
    127223           -14.5%     108804 ±  2%  softirqs.CPU23.TIMER
    125104           -17.6%     103046 ±  2%  softirqs.CPU230.TIMER
    125513           -17.3%     103833 ±  4%  softirqs.CPU231.TIMER
    145353 ±  2%     -11.5%     128639 ±  3%  softirqs.CPU232.RCU
    148102 ±  3%     -10.9%     132008        softirqs.CPU233.RCU
    124656           -11.6%     110217 ±  4%  softirqs.CPU238.TIMER
    125885           -11.3%     111674 ±  6%  softirqs.CPU239.TIMER
    126925           -12.3%     111331 ±  6%  softirqs.CPU24.TIMER
    126896           -18.9%     102913        softirqs.CPU25.TIMER
    126926           -13.9%     109332 ±  6%  softirqs.CPU26.TIMER
    124843           -14.4%     106864 ±  8%  softirqs.CPU264.TIMER
    124548           -19.9%      99763 ±  2%  softirqs.CPU265.TIMER
    124505           -21.0%      98321 ±  3%  softirqs.CPU267.TIMER
    126787           -16.6%     105683 ±  2%  softirqs.CPU27.TIMER
    124670           -11.9%     109797 ±  5%  softirqs.CPU271.TIMER
    125126           -15.7%     105434 ±  7%  softirqs.CPU272.TIMER
    124914           -18.0%     102384        softirqs.CPU276.TIMER
    124637           -15.2%     105734 ±  6%  softirqs.CPU277.TIMER
    126165           -19.3%     101875 ±  3%  softirqs.CPU278.TIMER
    127070           -15.2%     107787 ±  2%  softirqs.CPU28.TIMER
    147682 ±  3%      -7.4%     136816 ±  2%  softirqs.CPU281.RCU
    156309 ±  6%     -11.7%     138002 ±  4%  softirqs.CPU283.RCU
    131076 ±  4%     -12.1%     115207 ±  6%  softirqs.CPU283.TIMER
    124947           -20.2%      99737 ±  4%  softirqs.CPU284.TIMER
    125886           -16.9%     104560 ±  5%  softirqs.CPU285.TIMER
    123762           -15.9%     104068        softirqs.CPU286.TIMER
     48425 ±  8%    +157.2%     124544 ± 10%  softirqs.CPU287.RCU
     49910           +84.0%      91844 ± 12%  softirqs.CPU287.TIMER
    127042           -13.4%     110005 ±  3%  softirqs.CPU29.TIMER
    126872           -20.0%     101525 ±  4%  softirqs.CPU3.TIMER
    126487           -14.7%     107872 ±  5%  softirqs.CPU30.TIMER
    126758           -18.0%     103990 ±  3%  softirqs.CPU31.TIMER
    126973           -21.0%     100312 ±  2%  softirqs.CPU32.TIMER
    127155           -17.8%     104473 ±  2%  softirqs.CPU33.TIMER
    128219           -17.6%     105609        softirqs.CPU34.TIMER
    154527 ±  7%     -10.1%     138967 ±  2%  softirqs.CPU35.RCU
    160380 ± 20%     -28.5%     114686 ±  8%  softirqs.CPU35.TIMER
    126861           -13.7%     109531 ±  5%  softirqs.CPU37.TIMER
    126074           -16.8%     104881        softirqs.CPU38.TIMER
    134258 ±  5%     -20.5%     106757        softirqs.CPU39.TIMER
    131209 ±  2%     -18.3%     107174 ±  4%  softirqs.CPU4.TIMER
    126418           -11.2%     112198 ±  2%  softirqs.CPU42.TIMER
    126097           -13.5%     109046 ±  6%  softirqs.CPU43.TIMER
    126490           -18.2%     103508 ±  4%  softirqs.CPU44.TIMER
    127144           -20.0%     101704 ±  3%  softirqs.CPU45.TIMER
    128063           -15.3%     108424 ±  7%  softirqs.CPU46.TIMER
    127621           -16.6%     106471 ±  2%  softirqs.CPU47.TIMER
    126693           -19.6%     101863        softirqs.CPU48.TIMER
    126353           -17.6%     104065 ±  6%  softirqs.CPU49.TIMER
    127498           -17.2%     105628 ±  2%  softirqs.CPU5.TIMER
    126467           -17.0%     105026        softirqs.CPU50.TIMER
    126653           -14.0%     108865 ±  7%  softirqs.CPU51.TIMER
    126557           -12.9%     110205 ±  3%  softirqs.CPU52.TIMER
    160876 ± 20%     -28.7%     114698 ±  7%  softirqs.CPU53.TIMER
    127285           -19.3%     102718 ±  4%  softirqs.CPU56.TIMER
    126855           -20.8%     100422 ±  3%  softirqs.CPU58.TIMER
    126652           -15.1%     107573 ±  6%  softirqs.CPU59.TIMER
    126898           -16.0%     106626        softirqs.CPU6.TIMER
    126363           -16.9%     105028 ±  3%  softirqs.CPU68.TIMER
    126186           -19.0%     102237        softirqs.CPU69.TIMER
    127210           -14.3%     109029 ±  3%  softirqs.CPU7.TIMER
    125925           -15.1%     106863 ±  6%  softirqs.CPU70.TIMER
    125862           -15.2%     106706 ±  2%  softirqs.CPU71.TIMER
    126231           -13.2%     109602 ±  9%  softirqs.CPU74.TIMER
    126338           -15.6%     106674 ±  3%  softirqs.CPU75.TIMER
    127023           -13.9%     109365 ±  5%  softirqs.CPU8.TIMER
    126866           -12.7%     110803        softirqs.CPU84.TIMER
    126578            -8.8%     115406 ±  3%  softirqs.CPU86.TIMER
    146718 ±  2%     -14.3%     125687 ±  4%  softirqs.CPU89.RCU
    126854           -13.3%     109959 ±  4%  softirqs.CPU9.TIMER
    126028            -9.9%     113531 ±  3%  softirqs.CPU94.TIMER
    127092           -16.5%     106081 ±  6%  softirqs.CPU97.TIMER
      1187 ± 60%     -74.1%     307.50 ± 26%  interrupts.32:IR-PCI-MSI.2097155-edge.eth0-TxRx-2
      3273           +13.0%       3699        interrupts.CPU1.NMI:Non-maskable_interrupts
      3273           +13.0%       3699        interrupts.CPU1.PMI:Performance_monitoring_interrupts
      3347           +64.6%       5508 ± 33%  interrupts.CPU10.NMI:Non-maskable_interrupts
      3347           +64.6%       5508 ± 33%  interrupts.CPU10.PMI:Performance_monitoring_interrupts
     14.50 ± 10%   +5079.3%     751.00 ± 73%  interrupts.CPU10.RES:Rescheduling_interrupts
      3362           +36.8%       4599 ± 35%  interrupts.CPU11.NMI:Non-maskable_interrupts
      3362           +36.8%       4599 ± 35%  interrupts.CPU11.PMI:Performance_monitoring_interrupts
     11.50 ± 13%   +2456.5%     294.00 ± 58%  interrupts.CPU11.RES:Rescheduling_interrupts
      3.00         +4825.0%     147.75 ±167%  interrupts.CPU111.RES:Rescheduling_interrupts
    113.00 ± 28%     -90.5%      10.75 ± 71%  interrupts.CPU113.RES:Rescheduling_interrupts
      3339           +57.5%       5259 ± 32%  interrupts.CPU114.NMI:Non-maskable_interrupts
      3339           +57.5%       5259 ± 32%  interrupts.CPU114.PMI:Performance_monitoring_interrupts
      3333           +83.3%       6109 ± 24%  interrupts.CPU115.NMI:Non-maskable_interrupts
      3333           +83.3%       6109 ± 24%  interrupts.CPU115.PMI:Performance_monitoring_interrupts
      1187 ± 60%     -74.1%     307.50 ± 26%  interrupts.CPU12.32:IR-PCI-MSI.2097155-edge.eth0-TxRx-2
     17.00 ± 52%   +1208.8%     222.50 ±  5%  interrupts.CPU12.RES:Rescheduling_interrupts
    290.00 ± 94%     -55.1%     130.25 ±166%  interrupts.CPU122.RES:Rescheduling_interrupts
      4.00 ± 25%   +8300.0%     336.00 ± 45%  interrupts.CPU124.RES:Rescheduling_interrupts
      6587           -47.6%       3453        interrupts.CPU125.NMI:Non-maskable_interrupts
      6587           -47.6%       3453        interrupts.CPU125.PMI:Performance_monitoring_interrupts
      3.00         +2408.3%      75.25 ±161%  interrupts.CPU125.RES:Rescheduling_interrupts
      9.00 ± 22%   +2138.9%     201.50 ±  7%  interrupts.CPU13.RES:Rescheduling_interrupts
      3347           +55.9%       5218 ± 32%  interrupts.CPU131.NMI:Non-maskable_interrupts
      3347           +55.9%       5218 ± 32%  interrupts.CPU131.PMI:Performance_monitoring_interrupts
      3336           +56.2%       5209 ± 32%  interrupts.CPU133.NMI:Non-maskable_interrupts
      3336           +56.2%       5209 ± 32%  interrupts.CPU133.PMI:Performance_monitoring_interrupts
      3312           +57.5%       5218 ± 33%  interrupts.CPU134.NMI:Non-maskable_interrupts
      3312           +57.5%       5218 ± 33%  interrupts.CPU134.PMI:Performance_monitoring_interrupts
      3297           +56.7%       5168 ± 33%  interrupts.CPU138.NMI:Non-maskable_interrupts
      3297           +56.7%       5168 ± 33%  interrupts.CPU138.PMI:Performance_monitoring_interrupts
    269.50 ± 70%     -79.7%      54.75 ± 90%  interrupts.CPU138.RES:Rescheduling_interrupts
    193.00 ± 82%     -98.7%       2.50 ± 60%  interrupts.CPU138.TLB:TLB_shootdowns
    192.50 ± 86%     -82.9%      33.00 ±131%  interrupts.CPU139.RES:Rescheduling_interrupts
     10.50 ± 33%   +2471.4%     270.00 ± 44%  interrupts.CPU14.RES:Rescheduling_interrupts
      3315           +57.2%       5212 ± 32%  interrupts.CPU140.NMI:Non-maskable_interrupts
      3315           +57.2%       5212 ± 32%  interrupts.CPU140.PMI:Performance_monitoring_interrupts
    163.00 ± 65%     -92.6%      12.00 ± 70%  interrupts.CPU141.RES:Rescheduling_interrupts
      3152           +57.3%       4958 ± 32%  interrupts.CPU142.NMI:Non-maskable_interrupts
      3152           +57.3%       4958 ± 32%  interrupts.CPU142.PMI:Performance_monitoring_interrupts
      3150           +79.6%       5656 ± 23%  interrupts.CPU143.NMI:Non-maskable_interrupts
      3150           +79.6%       5656 ± 23%  interrupts.CPU143.PMI:Performance_monitoring_interrupts
      1200 ± 81%     -90.6%     112.75 ±131%  interrupts.CPU144.RES:Rescheduling_interrupts
      3352           +66.6%       5585 ± 33%  interrupts.CPU15.NMI:Non-maskable_interrupts
      3352           +66.6%       5585 ± 33%  interrupts.CPU15.PMI:Performance_monitoring_interrupts
      8.50 ± 17%   +2335.3%     207.00 ± 12%  interrupts.CPU15.RES:Rescheduling_interrupts
      3237 ±  3%     +61.0%       5212 ± 32%  interrupts.CPU150.NMI:Non-maskable_interrupts
      3237 ±  3%     +61.0%       5212 ± 32%  interrupts.CPU150.PMI:Performance_monitoring_interrupts
      3338           +54.6%       5159 ± 33%  interrupts.CPU154.NMI:Non-maskable_interrupts
      3338           +54.6%       5159 ± 33%  interrupts.CPU154.PMI:Performance_monitoring_interrupts
    191.50 ± 88%     -95.4%       8.75 ± 40%  interrupts.CPU160.RES:Rescheduling_interrupts
      3247           +88.3%       6116 ± 24%  interrupts.CPU161.NMI:Non-maskable_interrupts
      3247           +88.3%       6116 ± 24%  interrupts.CPU161.PMI:Performance_monitoring_interrupts
     62.00 ± 87%     -89.9%       6.25 ± 30%  interrupts.CPU164.RES:Rescheduling_interrupts
      3335           +58.8%       5294 ± 33%  interrupts.CPU168.NMI:Non-maskable_interrupts
      3335           +58.8%       5294 ± 33%  interrupts.CPU168.PMI:Performance_monitoring_interrupts
    203.00 ± 95%     -96.3%       7.50 ± 20%  interrupts.CPU168.RES:Rescheduling_interrupts
      3327           +54.3%       5133 ± 32%  interrupts.CPU171.NMI:Non-maskable_interrupts
      3327           +54.3%       5133 ± 32%  interrupts.CPU171.PMI:Performance_monitoring_interrupts
      3311           +58.0%       5233 ± 32%  interrupts.CPU175.NMI:Non-maskable_interrupts
      3311           +58.0%       5233 ± 32%  interrupts.CPU175.PMI:Performance_monitoring_interrupts
      3324           +59.3%       5294 ± 33%  interrupts.CPU179.NMI:Non-maskable_interrupts
      3324           +59.3%       5294 ± 33%  interrupts.CPU179.PMI:Performance_monitoring_interrupts
      3353           +60.8%       5392 ± 32%  interrupts.CPU18.NMI:Non-maskable_interrupts
      3353           +60.8%       5392 ± 32%  interrupts.CPU18.PMI:Performance_monitoring_interrupts
     13.50 ± 25%   +1307.4%     190.00 ± 12%  interrupts.CPU18.RES:Rescheduling_interrupts
      3337           +78.0%       5939 ± 24%  interrupts.CPU181.NMI:Non-maskable_interrupts
      3337           +78.0%       5939 ± 24%  interrupts.CPU181.PMI:Performance_monitoring_interrupts
      3341           +85.0%       6179 ± 24%  interrupts.CPU182.NMI:Non-maskable_interrupts
      3341           +85.0%       6179 ± 24%  interrupts.CPU182.PMI:Performance_monitoring_interrupts
      3320           +59.2%       5285 ± 33%  interrupts.CPU183.NMI:Non-maskable_interrupts
      3320           +59.2%       5285 ± 33%  interrupts.CPU183.PMI:Performance_monitoring_interrupts
     63.50 ± 85%     -85.4%       9.25 ± 55%  interrupts.CPU185.RES:Rescheduling_interrupts
      3356           +65.9%       5566 ± 33%  interrupts.CPU19.NMI:Non-maskable_interrupts
      3356           +65.9%       5566 ± 33%  interrupts.CPU19.PMI:Performance_monitoring_interrupts
     15.50 ± 29%    +814.5%     141.75 ± 42%  interrupts.CPU19.RES:Rescheduling_interrupts
      3345           +76.7%       5909 ± 23%  interrupts.CPU192.NMI:Non-maskable_interrupts
      3345           +76.7%       5909 ± 23%  interrupts.CPU192.PMI:Performance_monitoring_interrupts
      3354           +55.8%       5224 ± 33%  interrupts.CPU194.NMI:Non-maskable_interrupts
      3354           +55.8%       5224 ± 33%  interrupts.CPU194.PMI:Performance_monitoring_interrupts
      6540           -46.6%       3494        interrupts.CPU195.NMI:Non-maskable_interrupts
      6540           -46.6%       3494        interrupts.CPU195.PMI:Performance_monitoring_interrupts
      6.00        +18154.2%       1095 ±133%  interrupts.CPU196.RES:Rescheduling_interrupts
      3335           +60.1%       5340 ± 32%  interrupts.CPU198.NMI:Non-maskable_interrupts
      3335           +60.1%       5340 ± 32%  interrupts.CPU198.PMI:Performance_monitoring_interrupts
      5.50 ± 27%   +2436.4%     139.50 ±109%  interrupts.CPU198.RES:Rescheduling_interrupts
      3329           +60.6%       5348 ± 33%  interrupts.CPU199.NMI:Non-maskable_interrupts
      3329           +60.6%       5348 ± 33%  interrupts.CPU199.PMI:Performance_monitoring_interrupts
      3341           +80.1%       6018 ± 23%  interrupts.CPU200.NMI:Non-maskable_interrupts
      3341           +80.1%       6018 ± 23%  interrupts.CPU200.PMI:Performance_monitoring_interrupts
      3351           +53.5%       5145 ± 34%  interrupts.CPU202.NMI:Non-maskable_interrupts
      3351           +53.5%       5145 ± 34%  interrupts.CPU202.PMI:Performance_monitoring_interrupts
     74.50 ± 85%     -90.9%       6.75 ± 36%  interrupts.CPU207.RES:Rescheduling_interrupts
    270.00 ± 81%     -65.3%      93.75 ±149%  interrupts.CPU208.RES:Rescheduling_interrupts
    100.00 ± 38%     -38.2%      61.75 ± 62%  interrupts.CPU215.RES:Rescheduling_interrupts
    264.50 ± 14%     -92.2%      20.50 ± 84%  interrupts.CPU216.RES:Rescheduling_interrupts
      3325           +78.2%       5925 ± 24%  interrupts.CPU218.NMI:Non-maskable_interrupts
      3325           +78.2%       5925 ± 24%  interrupts.CPU218.PMI:Performance_monitoring_interrupts
      3346           +55.5%       5202 ± 33%  interrupts.CPU219.NMI:Non-maskable_interrupts
      3346           +55.5%       5202 ± 33%  interrupts.CPU219.PMI:Performance_monitoring_interrupts
      3321           +60.7%       5335 ± 32%  interrupts.CPU221.NMI:Non-maskable_interrupts
      3321           +60.7%       5335 ± 32%  interrupts.CPU221.PMI:Performance_monitoring_interrupts
      3305           +54.6%       5108 ± 33%  interrupts.CPU223.NMI:Non-maskable_interrupts
      3305           +54.6%       5108 ± 33%  interrupts.CPU223.PMI:Performance_monitoring_interrupts
      3314           +55.2%       5143 ± 35%  interrupts.CPU226.NMI:Non-maskable_interrupts
      3314           +55.2%       5143 ± 35%  interrupts.CPU226.PMI:Performance_monitoring_interrupts
      4.50 ± 11%   +3355.6%     155.50 ±163%  interrupts.CPU226.RES:Rescheduling_interrupts
      3326           +96.0%       6519 ± 25%  interrupts.CPU23.NMI:Non-maskable_interrupts
      3326           +96.0%       6519 ± 25%  interrupts.CPU23.PMI:Performance_monitoring_interrupts
      6.50 ± 38%   +2665.4%     179.75 ±160%  interrupts.CPU230.RES:Rescheduling_interrupts
      3069 ±  8%     +68.2%       5164 ± 32%  interrupts.CPU238.NMI:Non-maskable_interrupts
      3069 ±  8%     +68.2%       5164 ± 32%  interrupts.CPU238.PMI:Performance_monitoring_interrupts
      3328           +64.6%       5477 ± 32%  interrupts.CPU24.NMI:Non-maskable_interrupts
      3328           +64.6%       5477 ± 32%  interrupts.CPU24.PMI:Performance_monitoring_interrupts
    438.00 ± 67%     -94.0%      26.25 ± 74%  interrupts.CPU24.RES:Rescheduling_interrupts
    508.00 ± 97%     -98.6%       7.25 ± 53%  interrupts.CPU240.RES:Rescheduling_interrupts
      3318           +53.8%       5101 ± 32%  interrupts.CPU246.NMI:Non-maskable_interrupts
      3318           +53.8%       5101 ± 32%  interrupts.CPU246.PMI:Performance_monitoring_interrupts
    131.50 ± 92%     -95.1%       6.50 ± 25%  interrupts.CPU246.RES:Rescheduling_interrupts
      3339           +55.1%       5178 ± 32%  interrupts.CPU248.NMI:Non-maskable_interrupts
      3339           +55.1%       5178 ± 32%  interrupts.CPU248.PMI:Performance_monitoring_interrupts
      3328           +11.8%       3721        interrupts.CPU25.NMI:Non-maskable_interrupts
      3328           +11.8%       3721        interrupts.CPU25.PMI:Performance_monitoring_interrupts
      3319           +55.0%       5145 ± 32%  interrupts.CPU250.NMI:Non-maskable_interrupts
      3319           +55.0%       5145 ± 32%  interrupts.CPU250.PMI:Performance_monitoring_interrupts
      3310           +57.7%       5221 ± 33%  interrupts.CPU251.NMI:Non-maskable_interrupts
      3310           +57.7%       5221 ± 33%  interrupts.CPU251.PMI:Performance_monitoring_interrupts
      3330           +60.8%       5354 ± 32%  interrupts.CPU256.NMI:Non-maskable_interrupts
      3330           +60.8%       5354 ± 32%  interrupts.CPU256.PMI:Performance_monitoring_interrupts
    153.00 ± 90%     -78.1%      33.50 ±132%  interrupts.CPU256.RES:Rescheduling_interrupts
    147.00 ± 79%     -89.5%      15.50 ± 59%  interrupts.CPU257.RES:Rescheduling_interrupts
      4.00 ± 25%   +3237.5%     133.50 ± 89%  interrupts.CPU262.RES:Rescheduling_interrupts
      3318           +10.6%       3670        interrupts.CPU267.NMI:Non-maskable_interrupts
      3318           +10.6%       3670        interrupts.CPU267.PMI:Performance_monitoring_interrupts
     68.00 ± 63%     -59.9%      27.25 ±126%  interrupts.CPU267.RES:Rescheduling_interrupts
      3330           +39.7%       4653 ± 34%  interrupts.CPU27.NMI:Non-maskable_interrupts
      3330           +39.7%       4653 ± 34%  interrupts.CPU27.PMI:Performance_monitoring_interrupts
      3314           +60.8%       5330 ± 33%  interrupts.CPU270.NMI:Non-maskable_interrupts
      3314           +60.8%       5330 ± 33%  interrupts.CPU270.PMI:Performance_monitoring_interrupts
      3344           +84.2%       6160 ± 23%  interrupts.CPU274.NMI:Non-maskable_interrupts
      3344           +84.2%       6160 ± 23%  interrupts.CPU274.PMI:Performance_monitoring_interrupts
      3346          +114.6%       7179        interrupts.CPU276.NMI:Non-maskable_interrupts
      3346          +114.6%       7179        interrupts.CPU276.PMI:Performance_monitoring_interrupts
      4.50 ± 33%   +2111.1%      99.50 ± 97%  interrupts.CPU277.RES:Rescheduling_interrupts
    251.50 ± 94%     -84.4%      39.25 ±148%  interrupts.CPU281.RES:Rescheduling_interrupts
    206.50 ± 86%     -96.4%       7.50 ± 52%  interrupts.CPU283.RES:Rescheduling_interrupts
      3339          +116.1%       7216        interrupts.CPU284.NMI:Non-maskable_interrupts
      3339          +116.1%       7216        interrupts.CPU284.PMI:Performance_monitoring_interrupts
      1655           -22.9%       1276 ±  4%  interrupts.CPU287.CAL:Function_call_interrupts
    386.00           -90.7%      35.75 ±103%  interrupts.CPU287.TLB:TLB_shootdowns
      3343           +67.1%       5586 ± 33%  interrupts.CPU29.NMI:Non-maskable_interrupts
      3343           +67.1%       5586 ± 33%  interrupts.CPU29.PMI:Performance_monitoring_interrupts
      8.00 ± 12%    +418.8%      41.50 ± 88%  interrupts.CPU29.RES:Rescheduling_interrupts
      3354           +10.7%       3712        interrupts.CPU3.NMI:Non-maskable_interrupts
      3354           +10.7%       3712        interrupts.CPU3.PMI:Performance_monitoring_interrupts
     86.00 ± 12%    +201.7%     259.50 ± 29%  interrupts.CPU3.RES:Rescheduling_interrupts
     68.00 ± 64%     -87.1%       8.75 ± 40%  interrupts.CPU30.RES:Rescheduling_interrupts
      3325           +39.7%       4645 ± 34%  interrupts.CPU34.NMI:Non-maskable_interrupts
      3325           +39.7%       4645 ± 34%  interrupts.CPU34.PMI:Performance_monitoring_interrupts
     65.50 ± 64%     -81.3%      12.25 ± 50%  interrupts.CPU36.RES:Rescheduling_interrupts
      3354           +36.2%       4570 ± 32%  interrupts.CPU38.NMI:Non-maskable_interrupts
      3354           +36.2%       4570 ± 32%  interrupts.CPU38.PMI:Performance_monitoring_interrupts
      3351           +38.8%       4650 ± 34%  interrupts.CPU4.NMI:Non-maskable_interrupts
      3351           +38.8%       4650 ± 34%  interrupts.CPU4.PMI:Performance_monitoring_interrupts
     73.50 ±  2%    +225.5%     239.25 ± 27%  interrupts.CPU4.RES:Rescheduling_interrupts
      3307           +34.6%       4452 ± 33%  interrupts.CPU43.NMI:Non-maskable_interrupts
      3307           +34.6%       4452 ± 33%  interrupts.CPU43.PMI:Performance_monitoring_interrupts
    191.00 ± 96%     -97.1%       5.50 ± 15%  interrupts.CPU43.TLB:TLB_shootdowns
      3357           +61.4%       5417 ± 32%  interrupts.CPU46.NMI:Non-maskable_interrupts
      3357           +61.4%       5417 ± 32%  interrupts.CPU46.PMI:Performance_monitoring_interrupts
      8.00         +2384.4%     198.75 ± 90%  interrupts.CPU46.RES:Rescheduling_interrupts
      3362           +35.9%       4569 ± 35%  interrupts.CPU47.NMI:Non-maskable_interrupts
      3362           +35.9%       4569 ± 35%  interrupts.CPU47.PMI:Performance_monitoring_interrupts
      5.50 ± 45%   +8100.0%     451.00 ± 85%  interrupts.CPU47.RES:Rescheduling_interrupts
     58.50          +236.8%     197.00 ± 10%  interrupts.CPU5.RES:Rescheduling_interrupts
      3349           +61.5%       5408 ± 32%  interrupts.CPU51.NMI:Non-maskable_interrupts
      3349           +61.5%       5408 ± 32%  interrupts.CPU51.PMI:Performance_monitoring_interrupts
      6.00 ± 16%  +15120.8%     913.25 ±100%  interrupts.CPU52.RES:Rescheduling_interrupts
      8.00 ± 12%   +3215.6%     265.25 ±109%  interrupts.CPU53.RES:Rescheduling_interrupts
     10.50 ±  4%   +1283.3%     145.25 ±104%  interrupts.CPU54.RES:Rescheduling_interrupts
      3374           +34.8%       4547 ± 33%  interrupts.CPU58.NMI:Non-maskable_interrupts
      3374           +34.8%       4547 ± 33%  interrupts.CPU58.PMI:Performance_monitoring_interrupts
     30.00          +585.0%     205.50 ± 15%  interrupts.CPU6.RES:Rescheduling_interrupts
      6.50 ±  7%    +938.5%      67.50 ± 87%  interrupts.CPU60.RES:Rescheduling_interrupts
    322.00 ± 83%     -87.0%      42.00 ± 95%  interrupts.CPU66.RES:Rescheduling_interrupts
    255.00 ± 77%     -86.5%      34.50 ± 88%  interrupts.CPU67.RES:Rescheduling_interrupts
     72.50 ± 40%     -68.6%      22.75 ±107%  interrupts.CPU68.RES:Rescheduling_interrupts
    123.00 ± 18%     -86.2%      17.00 ± 36%  interrupts.CPU69.RES:Rescheduling_interrupts
      7.50 ± 20%   +2496.7%     194.75 ± 10%  interrupts.CPU7.RES:Rescheduling_interrupts
      3131           +96.7%       6160 ± 24%  interrupts.CPU70.NMI:Non-maskable_interrupts
      3131           +96.7%       6160 ± 24%  interrupts.CPU70.PMI:Performance_monitoring_interrupts
     83.00 ± 73%     -82.8%      14.25 ± 76%  interrupts.CPU70.RES:Rescheduling_interrupts
      3208           +36.8%       4390 ± 34%  interrupts.CPU71.NMI:Non-maskable_interrupts
      3208           +36.8%       4390 ± 34%  interrupts.CPU71.PMI:Performance_monitoring_interrupts
    136.00 ±  6%     -60.5%      53.75 ± 49%  interrupts.CPU71.RES:Rescheduling_interrupts
    760.50 ± 63%     -94.7%      40.00 ±103%  interrupts.CPU72.RES:Rescheduling_interrupts
    160.00 ± 17%     -96.4%       5.75 ± 18%  interrupts.CPU73.RES:Rescheduling_interrupts
      6593           -45.8%       3575        interrupts.CPU74.NMI:Non-maskable_interrupts
      6593           -45.8%       3575        interrupts.CPU74.PMI:Performance_monitoring_interrupts
      3.50 ± 14%   +3635.7%     130.75 ±154%  interrupts.CPU77.RES:Rescheduling_interrupts
     25.50 ±  9%    +712.7%     207.25 ± 14%  interrupts.CPU8.RES:Rescheduling_interrupts
      3339           +58.0%       5275 ± 33%  interrupts.CPU87.NMI:Non-maskable_interrupts
      3339           +58.0%       5275 ± 33%  interrupts.CPU87.PMI:Performance_monitoring_interrupts
      5.50 ± 27%   +3804.5%     214.75 ± 21%  interrupts.CPU9.RES:Rescheduling_interrupts
      3344           +54.0%       5150 ± 32%  interrupts.CPU92.NMI:Non-maskable_interrupts
      3344           +54.0%       5150 ± 32%  interrupts.CPU92.PMI:Performance_monitoring_interrupts
      3348           +54.8%       5184 ± 33%  interrupts.CPU93.NMI:Non-maskable_interrupts
      3348           +54.8%       5184 ± 33%  interrupts.CPU93.PMI:Performance_monitoring_interrupts
      3310           +86.2%       6165 ± 24%  interrupts.CPU98.NMI:Non-maskable_interrupts
      3310           +86.2%       6165 ± 24%  interrupts.CPU98.PMI:Performance_monitoring_interrupts
    390.50           -49.4%     197.75 ± 54%  interrupts.IWI:IRQ_work_interrupts
   1266237            +8.0%    1367845 ±  2%  interrupts.NMI:Non-maskable_interrupts
   1266237            +8.0%    1367845 ±  2%  interrupts.PMI:Performance_monitoring_interrupts
     21036           +61.3%      33926 ± 10%  interrupts.RES:Rescheduling_interrupts
      2292 ±  3%     -36.5%       1456 ±  6%  interrupts.TLB:TLB_shootdowns





Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.6.0-rc3-00043-gfda31c50292a5" of type "text/plain" (203539 bytes)

View attachment "job-script" of type "text/plain" (7449 bytes)

View attachment "job.yaml" of type "text/plain" (5124 bytes)

View attachment "reproduce" of type "text/plain" (311 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ