| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <CAKfTPtB3jQWd52FTyKNk5w8mmoDnu+jwYgkFBOiOKjb_BjxqTw@mail.gmail.com>
Date: Thu, 14 May 2020 19:09:35 +0200
From: Vincent Guittot <vincent.guittot@...aro.org>
To: kernel test robot <oliver.sang@...el.com>
Cc: Ingo Molnar <mingo@...nel.org>, Ben Segall <bsegall@...gle.com>,
Dietmar Eggemann <dietmar.eggemann@....com>,
Juri Lelli <juri.lelli@...hat.com>,
Linus Torvalds <torvalds@...ux-foundation.org>,
Mel Gorman <mgorman@...e.de>, Mike Galbraith <efault@....de>,
Peter Zijlstra <peterz@...radead.org>,
Steven Rostedt <rostedt@...dmis.org>,
Thomas Gleixner <tglx@...utronix.de>,
LKML <linux-kernel@...r.kernel.org>, lkp@...ts.01.org,
OTC LSE PnP <otc.lse.pnp@...el.com>
Subject: Re: [sched/fair] 0b0695f2b3: phoronix-test-suite.compress-gzip.0.seconds
19.8% regression
Hi Oliver,
On Thu, 14 May 2020 at 16:05, kernel test robot <oliver.sang@...el.com> wrote:
>
> Hi Vincent Guittot,
>
> Below report FYI.
> Last year, we actually reported an improvement "[sched/fair] 0b0695f2b3:
> vm-scalability.median 3.1% improvement" on link [1].
> but now we found the regression on pts.compress-gzip.
> This seems align with what showed in "[v4,00/10] sched/fair: rework the CFS
> load balance" (link [2]), where showed the reworked load balance could have
> both positive and negative effect for different test suites.
We have tried to run all possible use cases but it's impossible to
covers all so there were a possibility that one that is not covered,
would regressed.
> And also from link [3], the patch set risks regressions.
>
> We also confirmed this regression on another platform
> (Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz with 8G memory),
> below is the data (lower is better).
> v5.4 4.1
> fcf0553db6f4c79387864f6e4ab4a891601f395e 4.01
> 0b0695f2b34a4afa3f6e9aa1ff0e5336d8dad912 4.89
> v5.5 5.18
> v5.6 4.62
> v5.7-rc2 4.53
> v5.7-rc3 4.59
>
> It seems there are some recovery on latest kernels, but not fully back.
> We were just wondering whether you could share some lights the further works
> on the load balance after patch set [2] which could cause the performance
> change?
> And whether you have plan to refine the load balance algorithm further?
I'm going to have a look at your regression to understand what is
going wrong and how it can be fixed
Thanks
Vincent
> thanks
>
> [1] https://lists.01.org/hyperkitty/list/lkp@lists.01.org/thread/SANC7QLYZKUNMM6O7UNR3OAQAKS5BESE/
> [2] https://lore.kernel.org/patchwork/cover/1141687/
> [3] https://www.phoronix.com/scan.php?page=news_item&px=Linux-5.5-Scheduler
>
>
>
> Below is the detail regression report FYI.
>
> Greeting,
>
> FYI, we noticed a 19.8% regression of phoronix-test-suite.compress-gzip.0.seconds due to commit:
>
>
> commit: 0b0695f2b34a4afa3f6e9aa1ff0e5336d8dad912 ("sched/fair: Rework load_balance()")
> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
>
> in testcase: phoronix-test-suite
> on test machine: 16 threads Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz with 32G memory
> with following parameters:
>
> test: compress-gzip-1.2.0
> cpufreq_governor: performance
> ucode: 0xca
>
> test-description: The Phoronix Test Suite is the most comprehensive testing and benchmarking platform available that provides an extensible framework for which new tests can be easily added.
> test-url: http://www.phoronix-test-suite.com/
>
> In addition to that, the commit also has significant impact on the following tests:
>
> +------------------+-----------------------------------------------------------------------+
> | testcase: change | phoronix-test-suite: |
> | test machine | 12 threads Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz with 8G memory |
> | test parameters | cpufreq_governor=performance |
> | | test=compress-gzip-1.2.0 |
> +------------------+-----------------------------------------------------------------------+
> | testcase: change | vm-scalability: vm-scalability.median 3.1% improvement |
> | test machine | 104 threads Skylake with 192G memory |
> | test parameters | cpufreq_governor=performance |
> | | runtime=300s |
> | | size=8T |
> | | test=anon-cow-seq |
> | | ucode=0x2000064 |
> +------------------+-----------------------------------------------------------------------+
> | testcase: change | stress-ng: stress-ng.fault.ops_per_sec -23.1% regression |
> | test machine | 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory |
> | test parameters | class=scheduler |
> | | cpufreq_governor=performance |
> | | disk=1HDD |
> | | nr_threads=100% |
> | | sc_pid_max=4194304 |
> | | testtime=1s |
> | | ucode=0xb000038 |
> +------------------+-----------------------------------------------------------------------+
> | testcase: change | stress-ng: stress-ng.schedpolicy.ops_per_sec -33.3% regression |
> | test machine | 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 192G memory |
> | test parameters | class=interrupt |
> | | cpufreq_governor=performance |
> | | disk=1HDD |
> | | nr_threads=100% |
> | | testtime=1s |
> | | ucode=0x500002c |
> +------------------+-----------------------------------------------------------------------+
> | testcase: change | stress-ng: stress-ng.schedpolicy.ops_per_sec 42.3% improvement |
> | test machine | 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory |
> | test parameters | class=interrupt |
> | | cpufreq_governor=performance |
> | | disk=1HDD |
> | | nr_threads=100% |
> | | testtime=30s |
> | | ucode=0xb000038 |
> +------------------+-----------------------------------------------------------------------+
> | testcase: change | stress-ng: stress-ng.schedpolicy.ops_per_sec 35.1% improvement |
> | test machine | 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory |
> | test parameters | class=interrupt |
> | | cpufreq_governor=performance |
> | | disk=1HDD |
> | | nr_threads=100% |
> | | testtime=1s |
> | | ucode=0xb000038 |
> +------------------+-----------------------------------------------------------------------+
> | testcase: change | stress-ng: stress-ng.ioprio.ops_per_sec -20.7% regression |
> | test machine | 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 192G memory |
> | test parameters | class=os |
> | | cpufreq_governor=performance |
> | | disk=1HDD |
> | | fs=ext4 |
> | | nr_threads=100% |
> | | testtime=1s |
> | | ucode=0x500002b |
> +------------------+-----------------------------------------------------------------------+
> | testcase: change | stress-ng: stress-ng.schedpolicy.ops_per_sec 43.0% improvement |
> | test machine | 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory |
> | test parameters | class=interrupt |
> | | cpufreq_governor=performance |
> | | disk=1HDD |
> | | nr_threads=100% |
> | | testtime=30s |
> | | ucode=0xb000038 |
> +------------------+-----------------------------------------------------------------------+
>
>
> If you fix the issue, kindly add following tag
> Reported-by: kernel test robot <oliver.sang@...el.com>
>
>
> Details are as below:
> -------------------------------------------------------------------------------------------------->
>
>
> To reproduce:
>
> git clone https://github.com/intel/lkp-tests.git
> cd lkp-tests
> bin/lkp install job.yaml # job file is attached in this email
> bin/lkp run job.yaml
>
> =========================================================================================
> compiler/cpufreq_governor/kconfig/rootfs/tbox_group/test/testcase/ucode:
> gcc-7/performance/x86_64-lck-7983/clear-x86_64-phoronix-30140/lkp-cfl-e1/compress-gzip-1.2.0/phoronix-test-suite/0xca
>
> commit:
> fcf0553db6 ("sched/fair: Remove meaningless imbalance calculation")
> 0b0695f2b3 ("sched/fair: Rework load_balance()")
>
> fcf0553db6f4c793 0b0695f2b34a4afa3f6e9aa1ff0
> ---------------- ---------------------------
> fail:runs %reproduction fail:runs
> | | |
> :4 4% 0:7 perf-profile.children.cycles-pp.error_entry
> %stddev %change %stddev
> \ | \
> 6.01 +19.8% 7.20 phoronix-test-suite.compress-gzip.0.seconds
> 147.57 ą 8% +25.1% 184.54 phoronix-test-suite.time.elapsed_time
> 147.57 ą 8% +25.1% 184.54 phoronix-test-suite.time.elapsed_time.max
> 52926 ą 8% -23.8% 40312 meminfo.max_used_kB
> 0.11 ą 7% -0.0 0.09 ą 3% mpstat.cpu.all.soft%
> 242384 -1.4% 238931 proc-vmstat.nr_inactive_anon
> 242384 -1.4% 238931 proc-vmstat.nr_zone_inactive_anon
> 1.052e+08 ą 27% +56.5% 1.647e+08 ą 10% cpuidle.C1E.time
> 1041078 ą 22% +54.7% 1610786 ą 7% cpuidle.C1E.usage
> 3.414e+08 ą 6% +57.6% 5.381e+08 ą 28% cpuidle.C6.time
> 817897 ą 3% +50.1% 1227607 ą 11% cpuidle.C6.usage
> 2884 -4.2% 2762 turbostat.Avg_MHz
> 1041024 ą 22% +54.7% 1610657 ą 7% turbostat.C1E
> 817802 ą 3% +50.1% 1227380 ą 11% turbostat.C6
> 66.75 -2.0% 65.42 turbostat.CorWatt
> 67.28 -2.0% 65.94 turbostat.PkgWatt
> 32.50 +6.2% 34.50 vmstat.cpu.id
> 62.50 -2.4% 61.00 vmstat.cpu.us
> 2443 ą 2% -28.9% 1738 ą 2% vmstat.io.bi
> 23765 ą 4% +16.5% 27685 vmstat.system.cs
> 37860 -7.1% 35180 ą 2% vmstat.system.in
> 3.474e+09 ą 3% -12.7% 3.032e+09 perf-stat.i.branch-instructions
> 1.344e+08 ą 2% -11.6% 1.188e+08 perf-stat.i.branch-misses
> 13033225 ą 4% -19.0% 10561032 perf-stat.i.cache-misses
> 5.105e+08 ą 3% -15.3% 4.322e+08 perf-stat.i.cache-references
> 24205 ą 4% +16.3% 28161 perf-stat.i.context-switches
> 30.25 ą 2% +39.7% 42.27 ą 2% perf-stat.i.cpi
> 4.63e+10 -4.7% 4.412e+10 perf-stat.i.cpu-cycles
> 3147 ą 4% -8.4% 2882 ą 2% perf-stat.i.cpu-migrations
> 16724 ą 2% +45.9% 24406 ą 5% perf-stat.i.cycles-between-cache-misses
> 0.18 ą 13% -0.1 0.12 ą 4% perf-stat.i.dTLB-load-miss-rate%
> 4.822e+09 ą 3% -11.9% 4.248e+09 perf-stat.i.dTLB-loads
> 0.07 ą 8% -0.0 0.05 ą 16% perf-stat.i.dTLB-store-miss-rate%
> 1.623e+09 ą 2% -11.5% 1.436e+09 perf-stat.i.dTLB-stores
> 1007120 ą 3% -8.9% 917854 ą 2% perf-stat.i.iTLB-load-misses
> 1.816e+10 ą 3% -12.2% 1.594e+10 perf-stat.i.instructions
> 2.06 ą 54% -66.0% 0.70 perf-stat.i.major-faults
> 29896 ą 13% -35.2% 19362 ą 8% perf-stat.i.minor-faults
> 0.00 ą 9% -0.0 0.00 ą 6% perf-stat.i.node-load-miss-rate%
> 1295134 ą 3% -14.2% 1111173 perf-stat.i.node-loads
> 3064949 ą 4% -18.7% 2491063 ą 2% perf-stat.i.node-stores
> 29898 ą 13% -35.2% 19363 ą 8% perf-stat.i.page-faults
> 28.10 -3.5% 27.12 perf-stat.overall.MPKI
> 2.55 -0.1 2.44 ą 2% perf-stat.overall.cache-miss-rate%
> 2.56 ą 3% +8.5% 2.77 perf-stat.overall.cpi
> 3567 ą 5% +17.3% 4186 perf-stat.overall.cycles-between-cache-misses
> 0.02 ą 3% +0.0 0.02 ą 3% perf-stat.overall.dTLB-load-miss-rate%
> 18031 -3.6% 17375 ą 2% perf-stat.overall.instructions-per-iTLB-miss
> 0.39 ą 3% -7.9% 0.36 perf-stat.overall.ipc
> 3.446e+09 ą 3% -12.6% 3.011e+09 perf-stat.ps.branch-instructions
> 1.333e+08 ą 2% -11.5% 1.18e+08 perf-stat.ps.branch-misses
> 12927998 ą 4% -18.8% 10491818 perf-stat.ps.cache-misses
> 5.064e+08 ą 3% -15.2% 4.293e+08 perf-stat.ps.cache-references
> 24011 ą 4% +16.5% 27973 perf-stat.ps.context-switches
> 4.601e+10 -4.6% 4.391e+10 perf-stat.ps.cpu-cycles
> 3121 ą 4% -8.3% 2863 ą 2% perf-stat.ps.cpu-migrations
> 4.783e+09 ą 3% -11.8% 4.219e+09 perf-stat.ps.dTLB-loads
> 1.61e+09 ą 2% -11.4% 1.426e+09 perf-stat.ps.dTLB-stores
> 999100 ą 3% -8.7% 911974 ą 2% perf-stat.ps.iTLB-load-misses
> 1.802e+10 ą 3% -12.1% 1.584e+10 perf-stat.ps.instructions
> 2.04 ą 54% -65.9% 0.70 perf-stat.ps.major-faults
> 29656 ą 13% -35.1% 19237 ą 8% perf-stat.ps.minor-faults
> 1284601 ą 3% -14.1% 1103823 perf-stat.ps.node-loads
> 3039931 ą 4% -18.6% 2474451 ą 2% perf-stat.ps.node-stores
> 29658 ą 13% -35.1% 19238 ą 8% perf-stat.ps.page-faults
> 50384 ą 2% +16.5% 58713 ą 4% softirqs.CPU0.RCU
> 33143 ą 2% +19.9% 39731 ą 2% softirqs.CPU0.SCHED
> 72672 +24.0% 90109 softirqs.CPU0.TIMER
> 22182 ą 4% +26.3% 28008 ą 4% softirqs.CPU1.SCHED
> 74465 ą 4% +26.3% 94027 ą 3% softirqs.CPU1.TIMER
> 18680 ą 7% +29.2% 24135 ą 3% softirqs.CPU10.SCHED
> 75941 ą 2% +21.8% 92486 ą 7% softirqs.CPU10.TIMER
> 48991 ą 4% +22.7% 60105 ą 5% softirqs.CPU11.RCU
> 18666 ą 6% +28.4% 23976 ą 4% softirqs.CPU11.SCHED
> 74896 ą 6% +24.4% 93173 ą 3% softirqs.CPU11.TIMER
> 49490 +20.5% 59659 ą 2% softirqs.CPU12.RCU
> 18973 ą 7% +26.0% 23909 ą 3% softirqs.CPU12.SCHED
> 50620 +19.9% 60677 ą 6% softirqs.CPU13.RCU
> 19136 ą 6% +23.2% 23577 ą 4% softirqs.CPU13.SCHED
> 74812 +33.3% 99756 ą 7% softirqs.CPU13.TIMER
> 50824 +15.9% 58881 ą 3% softirqs.CPU14.RCU
> 19550 ą 5% +24.1% 24270 ą 4% softirqs.CPU14.SCHED
> 76801 +22.8% 94309 ą 4% softirqs.CPU14.TIMER
> 51844 +11.5% 57795 ą 3% softirqs.CPU15.RCU
> 19204 ą 8% +28.4% 24662 ą 2% softirqs.CPU15.SCHED
> 74751 +29.9% 97127 ą 3% softirqs.CPU15.TIMER
> 50307 +17.4% 59062 ą 4% softirqs.CPU2.RCU
> 22150 +12.2% 24848 softirqs.CPU2.SCHED
> 79653 ą 2% +21.6% 96829 ą 10% softirqs.CPU2.TIMER
> 50833 +21.1% 61534 ą 4% softirqs.CPU3.RCU
> 18935 ą 2% +32.0% 25002 ą 3% softirqs.CPU3.SCHED
> 50569 +15.8% 58570 ą 4% softirqs.CPU4.RCU
> 20509 ą 5% +18.3% 24271 softirqs.CPU4.SCHED
> 80942 ą 2% +15.3% 93304 ą 3% softirqs.CPU4.TIMER
> 50692 +16.5% 59067 ą 4% softirqs.CPU5.RCU
> 20237 ą 3% +18.2% 23914 ą 3% softirqs.CPU5.SCHED
> 78963 +21.8% 96151 ą 2% softirqs.CPU5.TIMER
> 19709 ą 7% +20.1% 23663 softirqs.CPU6.SCHED
> 81250 +15.9% 94185 softirqs.CPU6.TIMER
> 51379 +15.0% 59108 softirqs.CPU7.RCU
> 19642 ą 5% +28.4% 25227 ą 3% softirqs.CPU7.SCHED
> 78299 ą 2% +30.3% 102021 ą 4% softirqs.CPU7.TIMER
> 49723 +19.0% 59169 ą 4% softirqs.CPU8.RCU
> 20138 ą 6% +21.7% 24501 ą 2% softirqs.CPU8.SCHED
> 75256 ą 3% +22.8% 92419 ą 2% softirqs.CPU8.TIMER
> 50406 ą 2% +17.4% 59178 ą 4% softirqs.CPU9.RCU
> 19182 ą 9% +24.2% 23831 ą 6% softirqs.CPU9.SCHED
> 73572 ą 5% +30.4% 95951 ą 8% softirqs.CPU9.TIMER
> 812363 +16.6% 946858 ą 3% softirqs.RCU
> 330042 ą 4% +23.5% 407533 softirqs.SCHED
> 1240046 +22.5% 1519539 softirqs.TIMER
> 251015 ą 21% -84.2% 39587 ą106% sched_debug.cfs_rq:/.MIN_vruntime.avg
> 537847 ą 4% -44.8% 297100 ą 66% sched_debug.cfs_rq:/.MIN_vruntime.max
> 257990 ą 5% -63.4% 94515 ą 82% sched_debug.cfs_rq:/.MIN_vruntime.stddev
> 38935 +47.9% 57601 sched_debug.cfs_rq:/.exec_clock.avg
> 44119 +40.6% 62013 sched_debug.cfs_rq:/.exec_clock.max
> 37622 +49.9% 56404 sched_debug.cfs_rq:/.exec_clock.min
> 47287 ą 7% -70.3% 14036 ą 88% sched_debug.cfs_rq:/.load.min
> 67.17 -52.9% 31.62 ą 31% sched_debug.cfs_rq:/.load_avg.min
> 251015 ą 21% -84.2% 39588 ą106% sched_debug.cfs_rq:/.max_vruntime.avg
> 537847 ą 4% -44.8% 297103 ą 66% sched_debug.cfs_rq:/.max_vruntime.max
> 257991 ą 5% -63.4% 94516 ą 82% sched_debug.cfs_rq:/.max_vruntime.stddev
> 529078 ą 3% +45.2% 768398 sched_debug.cfs_rq:/.min_vruntime.avg
> 547175 ą 2% +44.1% 788582 sched_debug.cfs_rq:/.min_vruntime.max
> 496420 +48.3% 736148 ą 2% sched_debug.cfs_rq:/.min_vruntime.min
> 1.33 ą 15% -44.0% 0.75 ą 32% sched_debug.cfs_rq:/.nr_running.avg
> 0.83 ą 20% -70.0% 0.25 ą 70% sched_debug.cfs_rq:/.nr_running.min
> 0.54 ą 8% -15.9% 0.45 ą 7% sched_debug.cfs_rq:/.nr_running.stddev
> 0.33 +62.9% 0.54 ą 8% sched_debug.cfs_rq:/.nr_spread_over.avg
> 1.33 +54.7% 2.06 ą 17% sched_debug.cfs_rq:/.nr_spread_over.max
> 0.44 ą 7% +56.4% 0.69 ą 6% sched_debug.cfs_rq:/.nr_spread_over.stddev
> 130.83 ą 14% -25.6% 97.37 ą 15% sched_debug.cfs_rq:/.runnable_load_avg.avg
> 45.33 ą 6% -79.3% 9.38 ą 70% sched_debug.cfs_rq:/.runnable_load_avg.min
> 47283 ą 7% -70.9% 13741 ą 89% sched_debug.cfs_rq:/.runnable_weight.min
> 1098 ą 8% -27.6% 795.02 ą 24% sched_debug.cfs_rq:/.util_avg.avg
> 757.50 ą 9% -51.3% 369.25 ą 10% sched_debug.cfs_rq:/.util_avg.min
> 762.39 ą 11% -44.4% 424.04 ą 46% sched_debug.cfs_rq:/.util_est_enqueued.avg
> 314.00 ą 18% -78.5% 67.38 ą100% sched_debug.cfs_rq:/.util_est_enqueued.min
> 142951 ą 5% +22.8% 175502 ą 3% sched_debug.cpu.avg_idle.avg
> 72112 -18.3% 58937 ą 13% sched_debug.cpu.avg_idle.stddev
> 127638 ą 7% +39.3% 177858 ą 5% sched_debug.cpu.clock.avg
> 127643 ą 7% +39.3% 177862 ą 5% sched_debug.cpu.clock.max
> 127633 ą 7% +39.3% 177855 ą 5% sched_debug.cpu.clock.min
> 126720 ą 7% +39.4% 176681 ą 5% sched_debug.cpu.clock_task.avg
> 127168 ą 7% +39.3% 177179 ą 5% sched_debug.cpu.clock_task.max
> 125240 ą 7% +39.5% 174767 ą 5% sched_debug.cpu.clock_task.min
> 563.60 ą 2% +25.9% 709.78 ą 9% sched_debug.cpu.clock_task.stddev
> 1.66 ą 18% -37.5% 1.04 ą 32% sched_debug.cpu.nr_running.avg
> 0.83 ą 20% -62.5% 0.31 ą 87% sched_debug.cpu.nr_running.min
> 127617 ą 3% +52.9% 195080 sched_debug.cpu.nr_switches.avg
> 149901 ą 6% +45.2% 217652 sched_debug.cpu.nr_switches.max
> 108182 ą 5% +61.6% 174808 sched_debug.cpu.nr_switches.min
> 0.20 ą 5% -62.5% 0.07 ą 67% sched_debug.cpu.nr_uninterruptible.avg
> -29.33 -13.5% -25.38 sched_debug.cpu.nr_uninterruptible.min
> 92666 ą 8% +66.8% 154559 sched_debug.cpu.sched_count.avg
> 104565 ą 11% +57.2% 164374 sched_debug.cpu.sched_count.max
> 80272 ą 10% +77.2% 142238 sched_debug.cpu.sched_count.min
> 38029 ą 10% +80.4% 68608 sched_debug.cpu.sched_goidle.avg
> 43413 ą 11% +68.5% 73149 sched_debug.cpu.sched_goidle.max
> 32420 ą 11% +94.5% 63069 sched_debug.cpu.sched_goidle.min
> 51567 ą 8% +60.7% 82878 sched_debug.cpu.ttwu_count.avg
> 79015 ą 9% +45.2% 114717 ą 4% sched_debug.cpu.ttwu_count.max
> 42919 ą 9% +63.3% 70086 sched_debug.cpu.ttwu_count.min
> 127632 ą 7% +39.3% 177854 ą 5% sched_debug.cpu_clk
> 125087 ą 7% +40.1% 175285 ą 5% sched_debug.ktime
> 127882 ą 6% +39.3% 178163 ą 5% sched_debug.sched_clk
> 146.00 ą 13% +902.9% 1464 ą143% interrupts.133:IR-PCI-MSI.2097153-edge.eth1-TxRx-0
> 3375 ą 93% -94.8% 174.75 ą 26% interrupts.134:IR-PCI-MSI.2097154-edge.eth1-TxRx-1
> 297595 ą 8% +22.8% 365351 ą 2% interrupts.CPU0.LOC:Local_timer_interrupts
> 8402 -21.7% 6577 ą 25% interrupts.CPU0.NMI:Non-maskable_interrupts
> 8402 -21.7% 6577 ą 25% interrupts.CPU0.PMI:Performance_monitoring_interrupts
> 937.00 ą 2% +18.1% 1106 ą 11% interrupts.CPU0.RES:Rescheduling_interrupts
> 146.00 ą 13% +902.9% 1464 ą143% interrupts.CPU1.133:IR-PCI-MSI.2097153-edge.eth1-TxRx-0
> 297695 ą 8% +22.7% 365189 ą 2% interrupts.CPU1.LOC:Local_timer_interrupts
> 8412 -20.9% 6655 ą 25% interrupts.CPU1.NMI:Non-maskable_interrupts
> 8412 -20.9% 6655 ą 25% interrupts.CPU1.PMI:Performance_monitoring_interrupts
> 297641 ą 8% +22.7% 365268 ą 2% interrupts.CPU10.LOC:Local_timer_interrupts
> 8365 -10.9% 7455 ą 3% interrupts.CPU10.NMI:Non-maskable_interrupts
> 8365 -10.9% 7455 ą 3% interrupts.CPU10.PMI:Performance_monitoring_interrupts
> 297729 ą 8% +22.7% 365238 ą 2% interrupts.CPU11.LOC:Local_timer_interrupts
> 8376 -21.8% 6554 ą 26% interrupts.CPU11.NMI:Non-maskable_interrupts
> 8376 -21.8% 6554 ą 26% interrupts.CPU11.PMI:Performance_monitoring_interrupts
> 297394 ą 8% +22.8% 365274 ą 2% interrupts.CPU12.LOC:Local_timer_interrupts
> 8393 -10.5% 7512 ą 3% interrupts.CPU12.NMI:Non-maskable_interrupts
> 8393 -10.5% 7512 ą 3% interrupts.CPU12.PMI:Performance_monitoring_interrupts
> 297744 ą 8% +22.7% 365243 ą 2% interrupts.CPU13.LOC:Local_timer_interrupts
> 8353 -10.6% 7469 ą 3% interrupts.CPU13.NMI:Non-maskable_interrupts
> 8353 -10.6% 7469 ą 3% interrupts.CPU13.PMI:Performance_monitoring_interrupts
> 148.50 ą 17% -24.2% 112.50 ą 8% interrupts.CPU13.TLB:TLB_shootdowns
> 297692 ą 8% +22.7% 365311 ą 2% interrupts.CPU14.LOC:Local_timer_interrupts
> 8374 -10.4% 7501 ą 4% interrupts.CPU14.NMI:Non-maskable_interrupts
> 8374 -10.4% 7501 ą 4% interrupts.CPU14.PMI:Performance_monitoring_interrupts
> 297453 ą 8% +22.8% 365311 ą 2% interrupts.CPU15.LOC:Local_timer_interrupts
> 8336 -22.8% 6433 ą 26% interrupts.CPU15.NMI:Non-maskable_interrupts
> 8336 -22.8% 6433 ą 26% interrupts.CPU15.PMI:Performance_monitoring_interrupts
> 699.50 ą 21% +51.3% 1058 ą 10% interrupts.CPU15.RES:Rescheduling_interrupts
> 3375 ą 93% -94.8% 174.75 ą 26% interrupts.CPU2.134:IR-PCI-MSI.2097154-edge.eth1-TxRx-1
> 297685 ą 8% +22.7% 365273 ą 2% interrupts.CPU2.LOC:Local_timer_interrupts
> 8357 -21.2% 6584 ą 25% interrupts.CPU2.NMI:Non-maskable_interrupts
> 8357 -21.2% 6584 ą 25% interrupts.CPU2.PMI:Performance_monitoring_interrupts
> 164.00 ą 30% -23.0% 126.25 ą 32% interrupts.CPU2.TLB:TLB_shootdowns
> 297352 ą 8% +22.9% 365371 ą 2% interrupts.CPU3.LOC:Local_timer_interrupts
> 8383 -10.6% 7493 ą 4% interrupts.CPU3.NMI:Non-maskable_interrupts
> 8383 -10.6% 7493 ą 4% interrupts.CPU3.PMI:Performance_monitoring_interrupts
> 780.50 ą 3% +32.7% 1035 ą 6% interrupts.CPU3.RES:Rescheduling_interrupts
> 297595 ą 8% +22.8% 365415 ą 2% interrupts.CPU4.LOC:Local_timer_interrupts
> 8382 -21.4% 6584 ą 25% interrupts.CPU4.NMI:Non-maskable_interrupts
> 8382 -21.4% 6584 ą 25% interrupts.CPU4.PMI:Performance_monitoring_interrupts
> 297720 ą 8% +22.7% 365347 ą 2% interrupts.CPU5.LOC:Local_timer_interrupts
> 8353 -32.0% 5679 ą 34% interrupts.CPU5.NMI:Non-maskable_interrupts
> 8353 -32.0% 5679 ą 34% interrupts.CPU5.PMI:Performance_monitoring_interrupts
> 727.00 ą 16% +98.3% 1442 ą 47% interrupts.CPU5.RES:Rescheduling_interrupts
> 297620 ą 8% +22.8% 365343 ą 2% interrupts.CPU6.LOC:Local_timer_interrupts
> 8388 -10.3% 7526 ą 4% interrupts.CPU6.NMI:Non-maskable_interrupts
> 8388 -10.3% 7526 ą 4% interrupts.CPU6.PMI:Performance_monitoring_interrupts
> 156.50 ą 3% -27.6% 113.25 ą 16% interrupts.CPU6.TLB:TLB_shootdowns
> 297690 ą 8% +22.7% 365363 ą 2% interrupts.CPU7.LOC:Local_timer_interrupts
> 8390 -23.1% 6449 ą 25% interrupts.CPU7.NMI:Non-maskable_interrupts
> 8390 -23.1% 6449 ą 25% interrupts.CPU7.PMI:Performance_monitoring_interrupts
> 918.00 ą 16% +49.4% 1371 ą 7% interrupts.CPU7.RES:Rescheduling_interrupts
> 120.00 ą 35% +70.8% 205.00 ą 17% interrupts.CPU7.TLB:TLB_shootdowns
> 297731 ą 8% +22.7% 365368 ą 2% interrupts.CPU8.LOC:Local_timer_interrupts
> 8393 -32.5% 5668 ą 35% interrupts.CPU8.NMI:Non-maskable_interrupts
> 8393 -32.5% 5668 ą 35% interrupts.CPU8.PMI:Performance_monitoring_interrupts
> 297779 ą 8% +22.7% 365399 ą 2% interrupts.CPU9.LOC:Local_timer_interrupts
> 8430 -10.8% 7517 ą 2% interrupts.CPU9.NMI:Non-maskable_interrupts
> 8430 -10.8% 7517 ą 2% interrupts.CPU9.PMI:Performance_monitoring_interrupts
> 956.50 +13.5% 1085 ą 4% interrupts.CPU9.RES:Rescheduling_interrupts
> 4762118 ą 8% +22.7% 5845069 ą 2% interrupts.LOC:Local_timer_interrupts
> 134093 -18.2% 109662 ą 11% interrupts.NMI:Non-maskable_interrupts
> 134093 -18.2% 109662 ą 11% interrupts.PMI:Performance_monitoring_interrupts
> 66.97 ą 9% -29.9 37.12 ą 49% perf-profile.calltrace.cycles-pp.deflate
> 66.67 ą 9% -29.7 36.97 ą 50% perf-profile.calltrace.cycles-pp.deflate_medium.deflate
> 43.24 ą 9% -18.6 24.61 ą 49% perf-profile.calltrace.cycles-pp.longest_match.deflate_medium.deflate
> 2.29 ą 14% -1.2 1.05 ą 58% perf-profile.calltrace.cycles-pp.deflateSetDictionary
> 0.74 ą 6% -0.5 0.27 ą100% perf-profile.calltrace.cycles-pp.read.__libc_start_main
> 0.74 ą 7% -0.5 0.27 ą100% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.read.__libc_start_main
> 0.73 ą 7% -0.5 0.27 ą100% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.read.__libc_start_main
> 0.73 ą 7% -0.5 0.27 ą100% perf-profile.calltrace.cycles-pp.__x64_sys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read.__libc_start_main
> 0.73 ą 7% -0.5 0.27 ą100% perf-profile.calltrace.cycles-pp.ksys_read.__x64_sys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
> 0.26 ą100% +0.6 0.88 ą 45% perf-profile.calltrace.cycles-pp.vfs_statx.__do_sys_newfstatat.__x64_sys_newfstatat.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 0.34 ą100% +0.7 1.02 ą 42% perf-profile.calltrace.cycles-pp.do_sys_open.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 0.28 ą100% +0.7 0.96 ą 44% perf-profile.calltrace.cycles-pp.__do_sys_newfstatat.__x64_sys_newfstatat.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 0.28 ą100% +0.7 0.96 ą 44% perf-profile.calltrace.cycles-pp.__x64_sys_newfstatat.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 0.34 ą100% +0.7 1.03 ą 42% perf-profile.calltrace.cycles-pp.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 0.00 +0.8 0.77 ą 35% perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
> 0.56 ą 9% +0.8 1.40 ą 45% perf-profile.calltrace.cycles-pp.__schedule.schedule.futex_wait_queue_me.futex_wait.do_futex
> 0.58 ą 10% +0.9 1.43 ą 45% perf-profile.calltrace.cycles-pp.schedule.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex
> 0.33 ą100% +0.9 1.21 ą 28% perf-profile.calltrace.cycles-pp.menu_select.cpuidle_select.do_idle.cpu_startup_entry.start_secondary
> 0.34 ą 99% +0.9 1.27 ą 30% perf-profile.calltrace.cycles-pp.cpuidle_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
> 0.00 +1.0 0.96 ą 62% perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
> 0.62 ą 9% +1.0 1.60 ą 52% perf-profile.calltrace.cycles-pp.futex_wait_queue_me.futex_wait.do_futex.__x64_sys_futex.do_syscall_64
> 0.68 ą 10% +1.0 1.73 ą 51% perf-profile.calltrace.cycles-pp.futex_wait.do_futex.__x64_sys_futex.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 0.46 ą100% +1.1 1.60 ą 43% perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle
> 0.47 ą100% +1.2 1.62 ą 43% perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
> 17.73 ą 21% +19.1 36.84 ą 25% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
> 17.75 ą 20% +19.9 37.63 ą 26% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
> 17.84 ą 20% +20.0 37.82 ą 26% perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
> 18.83 ą 20% +21.2 40.05 ą 27% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
> 18.89 ą 20% +21.2 40.11 ą 27% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
> 18.89 ą 20% +21.2 40.12 ą 27% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
> 20.14 ą 20% +22.5 42.66 ą 27% perf-profile.calltrace.cycles-pp.secondary_startup_64
> 66.97 ą 9% -29.9 37.12 ą 49% perf-profile.children.cycles-pp.deflate
> 66.83 ą 9% -29.8 37.06 ą 49% perf-profile.children.cycles-pp.deflate_medium
> 43.58 ą 9% -18.8 24.80 ą 49% perf-profile.children.cycles-pp.longest_match
> 2.29 ą 14% -1.2 1.12 ą 43% perf-profile.children.cycles-pp.deflateSetDictionary
> 0.84 -0.3 0.58 ą 19% perf-profile.children.cycles-pp.read
> 0.52 ą 13% -0.2 0.27 ą 43% perf-profile.children.cycles-pp.fill_window
> 0.06 +0.0 0.08 ą 13% perf-profile.children.cycles-pp.update_stack_state
> 0.07 ą 14% +0.0 0.11 ą 24% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
> 0.03 ą100% +0.1 0.09 ą 19% perf-profile.children.cycles-pp.find_next_and_bit
> 0.00 +0.1 0.06 ą 15% perf-profile.children.cycles-pp.refcount_inc_not_zero_checked
> 0.03 ą100% +0.1 0.08 ą 33% perf-profile.children.cycles-pp.free_pcppages_bulk
> 0.07 ą 7% +0.1 0.12 ą 36% perf-profile.children.cycles-pp.syscall_return_via_sysret
> 0.00 +0.1 0.06 ą 28% perf-profile.children.cycles-pp.rb_erase
> 0.03 ą100% +0.1 0.09 ą 24% perf-profile.children.cycles-pp.shmem_undo_range
> 0.03 ą100% +0.1 0.09 ą 28% perf-profile.children.cycles-pp.unlinkat
> 0.03 ą100% +0.1 0.09 ą 28% perf-profile.children.cycles-pp.__x64_sys_unlinkat
> 0.03 ą100% +0.1 0.09 ą 28% perf-profile.children.cycles-pp.do_unlinkat
> 0.03 ą100% +0.1 0.09 ą 28% perf-profile.children.cycles-pp.ovl_destroy_inode
> 0.03 ą100% +0.1 0.09 ą 28% perf-profile.children.cycles-pp.shmem_evict_inode
> 0.03 ą100% +0.1 0.09 ą 28% perf-profile.children.cycles-pp.shmem_truncate_range
> 0.05 +0.1 0.12 ą 38% perf-profile.children.cycles-pp.unmap_vmas
> 0.00 +0.1 0.07 ą 30% perf-profile.children.cycles-pp.timerqueue_del
> 0.00 +0.1 0.07 ą 26% perf-profile.children.cycles-pp.idle_cpu
> 0.09 ą 17% +0.1 0.15 ą 19% perf-profile.children.cycles-pp.__update_load_avg_se
> 0.00 +0.1 0.07 ą 33% perf-profile.children.cycles-pp.unmap_region
> 0.00 +0.1 0.07 ą 33% perf-profile.children.cycles-pp.__alloc_fd
> 0.03 ą100% +0.1 0.10 ą 31% perf-profile.children.cycles-pp.destroy_inode
> 0.03 ą100% +0.1 0.10 ą 30% perf-profile.children.cycles-pp.evict
> 0.06 ą 16% +0.1 0.13 ą 35% perf-profile.children.cycles-pp.ovl_override_creds
> 0.00 +0.1 0.07 ą 26% perf-profile.children.cycles-pp.kernel_text_address
> 0.00 +0.1 0.07 ą 41% perf-profile.children.cycles-pp.file_remove_privs
> 0.07 ą 23% +0.1 0.14 ą 47% perf-profile.children.cycles-pp.hrtimer_next_event_without
> 0.03 ą100% +0.1 0.10 ą 24% perf-profile.children.cycles-pp.__dentry_kill
> 0.03 ą100% +0.1 0.10 ą 29% perf-profile.children.cycles-pp.dentry_unlink_inode
> 0.03 ą100% +0.1 0.10 ą 29% perf-profile.children.cycles-pp.iput
> 0.03 ą100% +0.1 0.10 ą 54% perf-profile.children.cycles-pp.__close_fd
> 0.08 ą 25% +0.1 0.15 ą 35% perf-profile.children.cycles-pp.__switch_to
> 0.00 +0.1 0.07 ą 29% perf-profile.children.cycles-pp.__switch_to_asm
> 0.00 +0.1 0.08 ą 24% perf-profile.children.cycles-pp.__kernel_text_address
> 0.03 ą100% +0.1 0.11 ą 51% perf-profile.children.cycles-pp.enqueue_hrtimer
> 0.03 ą100% +0.1 0.11 ą 33% perf-profile.children.cycles-pp.rcu_gp_kthread_wake
> 0.03 ą100% +0.1 0.11 ą 33% perf-profile.children.cycles-pp.swake_up_one
> 0.03 ą100% +0.1 0.11 ą 33% perf-profile.children.cycles-pp.swake_up_locked
> 0.10 ą 30% +0.1 0.18 ą 17% perf-profile.children.cycles-pp.irqtime_account_irq
> 0.03 ą100% +0.1 0.11 ą 38% perf-profile.children.cycles-pp.unmap_page_range
> 0.00 +0.1 0.09 ą 37% perf-profile.children.cycles-pp.putname
> 0.03 ą100% +0.1 0.11 ą 28% perf-profile.children.cycles-pp.filemap_map_pages
> 0.07 ą 28% +0.1 0.16 ą 35% perf-profile.children.cycles-pp.getname
> 0.03 ą100% +0.1 0.11 ą 40% perf-profile.children.cycles-pp.unmap_single_vma
> 0.08 ą 29% +0.1 0.17 ą 38% perf-profile.children.cycles-pp.queued_spin_lock_slowpath
> 0.03 ą100% +0.1 0.12 ą 54% perf-profile.children.cycles-pp.setlocale
> 0.03 ą100% +0.1 0.12 ą 60% perf-profile.children.cycles-pp.__open64_nocancel
> 0.00 +0.1 0.09 ą 31% perf-profile.children.cycles-pp.__hrtimer_get_next_event
> 0.00 +0.1 0.10 ą 28% perf-profile.children.cycles-pp.get_unused_fd_flags
> 0.00 +0.1 0.10 ą 65% perf-profile.children.cycles-pp.timerqueue_add
> 0.07 ą 28% +0.1 0.17 ą 42% perf-profile.children.cycles-pp.__hrtimer_next_event_base
> 0.03 ą100% +0.1 0.12 ą 51% perf-profile.children.cycles-pp.__x64_sys_close
> 0.00 +0.1 0.10 ą 38% perf-profile.children.cycles-pp.do_lookup_x
> 0.03 ą100% +0.1 0.12 ą 23% perf-profile.children.cycles-pp.kmem_cache_free
> 0.04 ą100% +0.1 0.14 ą 53% perf-profile.children.cycles-pp.__do_munmap
> 0.00 +0.1 0.10 ą 35% perf-profile.children.cycles-pp.unwind_get_return_address
> 0.00 +0.1 0.10 ą 49% perf-profile.children.cycles-pp.shmem_add_to_page_cache
> 0.07 ą 20% +0.1 0.18 ą 25% perf-profile.children.cycles-pp.find_next_bit
> 0.08 ą 25% +0.1 0.18 ą 34% perf-profile.children.cycles-pp.dput
> 0.11 ą 33% +0.1 0.21 ą 37% perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
> 0.08 ą 5% +0.1 0.19 ą 27% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
> 0.00 +0.1 0.11 ą 52% perf-profile.children.cycles-pp.rcu_idle_exit
> 0.03 ą100% +0.1 0.14 ą 18% perf-profile.children.cycles-pp.entry_SYSCALL_64
> 0.08 +0.1 0.19 ą 43% perf-profile.children.cycles-pp.exit_mmap
> 0.09 ą 22% +0.1 0.20 ą 57% perf-profile.children.cycles-pp.set_next_entity
> 0.07 ą 7% +0.1 0.18 ą 60% perf-profile.children.cycles-pp.switch_mm_irqs_off
> 0.10 ą 26% +0.1 0.21 ą 32% perf-profile.children.cycles-pp.sched_clock
> 0.12 ą 25% +0.1 0.23 ą 39% perf-profile.children.cycles-pp.update_cfs_group
> 0.07 ą 14% +0.1 0.18 ą 45% perf-profile.children.cycles-pp.lapic_next_deadline
> 0.08 ą 5% +0.1 0.20 ą 44% perf-profile.children.cycles-pp.mmput
> 0.11 ą 18% +0.1 0.23 ą 41% perf-profile.children.cycles-pp.clockevents_program_event
> 0.07 ą 7% +0.1 0.18 ą 40% perf-profile.children.cycles-pp.strncpy_from_user
> 0.00 +0.1 0.12 ą 48% perf-profile.children.cycles-pp.flush_old_exec
> 0.11 ą 18% +0.1 0.23 ą 37% perf-profile.children.cycles-pp.native_sched_clock
> 0.09 ą 17% +0.1 0.21 ą 46% perf-profile.children.cycles-pp._dl_sysdep_start
> 0.12 ą 19% +0.1 0.26 ą 37% perf-profile.children.cycles-pp.tick_program_event
> 0.09 ą 33% +0.1 0.23 ą 61% perf-profile.children.cycles-pp.mmap_region
> 0.14 ą 21% +0.1 0.28 ą 39% perf-profile.children.cycles-pp.sched_clock_cpu
> 0.11 ą 27% +0.1 0.25 ą 56% perf-profile.children.cycles-pp.do_mmap
> 0.11 ą 36% +0.1 0.26 ą 57% perf-profile.children.cycles-pp.ksys_mmap_pgoff
> 0.09 ą 17% +0.1 0.23 ą 48% perf-profile.children.cycles-pp.lookup_fast
> 0.04 ą100% +0.2 0.19 ą 48% perf-profile.children.cycles-pp.open_path
> 0.11 ą 30% +0.2 0.27 ą 58% perf-profile.children.cycles-pp.vm_mmap_pgoff
> 0.11 ą 27% +0.2 0.28 ą 37% perf-profile.children.cycles-pp.update_blocked_averages
> 0.11 +0.2 0.29 ą 38% perf-profile.children.cycles-pp.search_binary_handler
> 0.11 ą 4% +0.2 0.29 ą 39% perf-profile.children.cycles-pp.load_elf_binary
> 0.11 ą 30% +0.2 0.30 ą 50% perf-profile.children.cycles-pp.inode_permission
> 0.04 ą100% +0.2 0.24 ą 55% perf-profile.children.cycles-pp.__GI___open64_nocancel
> 0.15 ą 29% +0.2 0.35 ą 34% perf-profile.children.cycles-pp.getname_flags
> 0.16 ą 25% +0.2 0.38 ą 26% perf-profile.children.cycles-pp.get_next_timer_interrupt
> 0.18 ą 11% +0.2 0.41 ą 39% perf-profile.children.cycles-pp.execve
> 0.19 ą 5% +0.2 0.42 ą 37% perf-profile.children.cycles-pp.__x64_sys_execve
> 0.18 ą 2% +0.2 0.42 ą 37% perf-profile.children.cycles-pp.__do_execve_file
> 0.32 ą 18% +0.3 0.57 ą 33% perf-profile.children.cycles-pp.__account_scheduler_latency
> 0.21 ą 17% +0.3 0.48 ą 47% perf-profile.children.cycles-pp.schedule_idle
> 0.20 ą 19% +0.3 0.49 ą 33% perf-profile.children.cycles-pp.tick_nohz_next_event
> 0.21 ą 26% +0.3 0.55 ą 41% perf-profile.children.cycles-pp.find_busiest_group
> 0.32 ą 26% +0.3 0.67 ą 52% perf-profile.children.cycles-pp.__handle_mm_fault
> 0.22 ą 25% +0.4 0.57 ą 49% perf-profile.children.cycles-pp.filename_lookup
> 0.34 ą 27% +0.4 0.72 ą 50% perf-profile.children.cycles-pp.handle_mm_fault
> 0.42 ą 32% +0.4 0.80 ą 43% perf-profile.children.cycles-pp.shmem_getpage_gfp
> 0.36 ą 23% +0.4 0.77 ą 41% perf-profile.children.cycles-pp.load_balance
> 0.41 ą 30% +0.4 0.82 ą 50% perf-profile.children.cycles-pp.do_page_fault
> 0.39 ą 30% +0.4 0.80 ą 50% perf-profile.children.cycles-pp.__do_page_fault
> 0.28 ą 22% +0.4 0.70 ą 37% perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
> 0.43 ą 31% +0.4 0.86 ą 49% perf-profile.children.cycles-pp.page_fault
> 0.31 ą 25% +0.5 0.77 ą 46% perf-profile.children.cycles-pp.user_path_at_empty
> 0.36 ą 20% +0.5 0.84 ą 34% perf-profile.children.cycles-pp.newidle_balance
> 0.45 ą 21% +0.5 0.95 ą 44% perf-profile.children.cycles-pp.vfs_statx
> 0.46 ą 20% +0.5 0.97 ą 43% perf-profile.children.cycles-pp.__do_sys_newfstatat
> 0.47 ą 20% +0.5 0.98 ą 44% perf-profile.children.cycles-pp.__x64_sys_newfstatat
> 0.29 ą 37% +0.5 0.81 ą 32% perf-profile.children.cycles-pp.io_serial_in
> 0.53 ą 25% +0.5 1.06 ą 49% perf-profile.children.cycles-pp.path_openat
> 0.55 ą 24% +0.5 1.09 ą 50% perf-profile.children.cycles-pp.do_filp_open
> 0.35 ą 2% +0.5 0.90 ą 60% perf-profile.children.cycles-pp.uart_console_write
> 0.35 ą 4% +0.6 0.91 ą 60% perf-profile.children.cycles-pp.console_unlock
> 0.35 ą 4% +0.6 0.91 ą 60% perf-profile.children.cycles-pp.univ8250_console_write
> 0.35 ą 4% +0.6 0.91 ą 60% perf-profile.children.cycles-pp.serial8250_console_write
> 0.82 ą 23% +0.6 1.42 ą 31% perf-profile.children.cycles-pp.__hrtimer_run_queues
> 0.47 ą 28% +0.6 1.10 ą 39% perf-profile.children.cycles-pp.irq_work_interrupt
> 0.47 ą 28% +0.6 1.10 ą 39% perf-profile.children.cycles-pp.smp_irq_work_interrupt
> 0.47 ą 28% +0.6 1.10 ą 39% perf-profile.children.cycles-pp.irq_work_run
> 0.47 ą 28% +0.6 1.10 ą 39% perf-profile.children.cycles-pp.perf_duration_warn
> 0.47 ą 28% +0.6 1.10 ą 39% perf-profile.children.cycles-pp.printk
> 0.47 ą 28% +0.6 1.10 ą 39% perf-profile.children.cycles-pp.vprintk_func
> 0.47 ą 28% +0.6 1.10 ą 39% perf-profile.children.cycles-pp.vprintk_default
> 0.47 ą 28% +0.6 1.11 ą 39% perf-profile.children.cycles-pp.irq_work_run_list
> 0.49 ą 31% +0.6 1.13 ą 39% perf-profile.children.cycles-pp.vprintk_emit
> 0.54 ą 19% +0.6 1.17 ą 38% perf-profile.children.cycles-pp.pick_next_task_fair
> 0.32 ą 7% +0.7 1.02 ą 56% perf-profile.children.cycles-pp.poll_idle
> 0.60 ą 15% +0.7 1.31 ą 29% perf-profile.children.cycles-pp.menu_select
> 0.65 ą 27% +0.7 1.36 ą 45% perf-profile.children.cycles-pp.do_sys_open
> 0.62 ą 15% +0.7 1.36 ą 31% perf-profile.children.cycles-pp.cpuidle_select
> 0.66 ą 26% +0.7 1.39 ą 44% perf-profile.children.cycles-pp.__x64_sys_openat
> 1.11 ą 22% +0.9 2.03 ą 31% perf-profile.children.cycles-pp.hrtimer_interrupt
> 0.89 ą 24% +0.9 1.81 ą 42% perf-profile.children.cycles-pp.futex_wait_queue_me
> 1.16 ą 27% +1.0 2.13 ą 36% perf-profile.children.cycles-pp.schedule
> 0.97 ą 23% +1.0 1.97 ą 42% perf-profile.children.cycles-pp.futex_wait
> 1.33 ą 25% +1.2 2.55 ą 39% perf-profile.children.cycles-pp.__schedule
> 1.84 ą 26% +1.6 3.42 ą 36% perf-profile.children.cycles-pp.smp_apic_timer_interrupt
> 1.76 ą 22% +1.6 3.41 ą 40% perf-profile.children.cycles-pp.do_futex
> 1.79 ą 22% +1.7 3.49 ą 41% perf-profile.children.cycles-pp.__x64_sys_futex
> 2.23 ą 20% +1.7 3.98 ą 37% perf-profile.children.cycles-pp.apic_timer_interrupt
> 17.73 ą 21% +19.1 36.86 ą 25% perf-profile.children.cycles-pp.intel_idle
> 19.00 ą 21% +21.1 40.13 ą 26% perf-profile.children.cycles-pp.cpuidle_enter_state
> 19.02 ą 21% +21.2 40.19 ą 26% perf-profile.children.cycles-pp.cpuidle_enter
> 18.89 ą 20% +21.2 40.12 ą 27% perf-profile.children.cycles-pp.start_secondary
> 20.14 ą 20% +22.5 42.65 ą 27% perf-profile.children.cycles-pp.cpu_startup_entry
> 20.08 ą 20% +22.5 42.59 ą 27% perf-profile.children.cycles-pp.do_idle
> 20.14 ą 20% +22.5 42.66 ą 27% perf-profile.children.cycles-pp.secondary_startup_64
> 43.25 ą 9% -18.6 24.63 ą 49% perf-profile.self.cycles-pp.longest_match
> 22.74 ą 11% -10.8 11.97 ą 50% perf-profile.self.cycles-pp.deflate_medium
> 2.26 ą 14% -1.2 1.11 ą 44% perf-profile.self.cycles-pp.deflateSetDictionary
> 0.51 ą 12% -0.3 0.24 ą 57% perf-profile.self.cycles-pp.fill_window
> 0.07 ą 7% +0.0 0.10 ą 24% perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
> 0.07 ą 7% +0.1 0.12 ą 36% perf-profile.self.cycles-pp.syscall_return_via_sysret
> 0.08 ą 12% +0.1 0.14 ą 15% perf-profile.self.cycles-pp.__update_load_avg_se
> 0.06 +0.1 0.13 ą 27% perf-profile.self.cycles-pp._raw_spin_lock_irqsave
> 0.08 ą 25% +0.1 0.15 ą 37% perf-profile.self.cycles-pp.__switch_to
> 0.06 ą 16% +0.1 0.13 ą 29% perf-profile.self.cycles-pp.get_page_from_freelist
> 0.00 +0.1 0.07 ą 29% perf-profile.self.cycles-pp.__switch_to_asm
> 0.05 +0.1 0.13 ą 57% perf-profile.self.cycles-pp.switch_mm_irqs_off
> 0.00 +0.1 0.08 ą 41% perf-profile.self.cycles-pp.interrupt_entry
> 0.00 +0.1 0.08 ą 61% perf-profile.self.cycles-pp.run_timer_softirq
> 0.07 ą 23% +0.1 0.15 ą 43% perf-profile.self.cycles-pp.__hrtimer_next_event_base
> 0.03 ą100% +0.1 0.12 ą 43% perf-profile.self.cycles-pp.update_cfs_group
> 0.08 ą 29% +0.1 0.17 ą 38% perf-profile.self.cycles-pp.queued_spin_lock_slowpath
> 0.00 +0.1 0.09 ą 29% perf-profile.self.cycles-pp.strncpy_from_user
> 0.06 ą 16% +0.1 0.15 ą 24% perf-profile.self.cycles-pp.find_next_bit
> 0.00 +0.1 0.10 ą 35% perf-profile.self.cycles-pp.do_lookup_x
> 0.00 +0.1 0.10 ą 13% perf-profile.self.cycles-pp.kmem_cache_free
> 0.06 ą 16% +0.1 0.16 ą 30% perf-profile.self.cycles-pp.do_idle
> 0.03 ą100% +0.1 0.13 ą 18% perf-profile.self.cycles-pp.entry_SYSCALL_64
> 0.03 ą100% +0.1 0.14 ą 41% perf-profile.self.cycles-pp.update_blocked_averages
> 0.11 ą 18% +0.1 0.22 ą 37% perf-profile.self.cycles-pp.native_sched_clock
> 0.07 ą 14% +0.1 0.18 ą 46% perf-profile.self.cycles-pp.lapic_next_deadline
> 0.00 +0.1 0.12 ą 65% perf-profile.self.cycles-pp.set_next_entity
> 0.12 ą 28% +0.1 0.27 ą 32% perf-profile.self.cycles-pp.cpuidle_enter_state
> 0.15 ą 3% +0.2 0.32 ą 39% perf-profile.self.cycles-pp.io_serial_out
> 0.25 ą 4% +0.2 0.48 ą 19% perf-profile.self.cycles-pp.menu_select
> 0.15 ą 22% +0.3 0.42 ą 46% perf-profile.self.cycles-pp.find_busiest_group
> 0.29 ą 37% +0.4 0.71 ą 42% perf-profile.self.cycles-pp.io_serial_in
> 0.32 ą 7% +0.7 1.02 ą 56% perf-profile.self.cycles-pp.poll_idle
> 17.73 ą 21% +19.1 36.79 ą 25% perf-profile.self.cycles-pp.intel_idle
>
>
>
> phoronix-test-suite.compress-gzip.0.seconds
>
> 8 +-----------------------------------------------------------------------+
> | O O O O O O O O |
> 7 |-+ O O O O O O O O O |
> 6 |-+ + + + |
> | + : + + : + + + : |
> 5 |-+ : : : : :: : : : : |
> | :: : : : :: : : : :: : : |
> 4 |:+: : : : : : : : : : : : : : : : : |
> |: : : : : : : : : + + : : + : : : : : : : |
> 3 |:+: : : : : : : : : : : : : : : : : : : : |
> 2 |:+: : : : : : : : : : : : : : : : : : : : : : : |
> |: : : : : : : : : : : : : : : : : : : : : : : : |
> 1 |-: :: : : : : : : : : :: :: :: : : |
> | : : : : : : : : : : : : |
> 0 +-----------------------------------------------------------------------+
>
>
> [*] bisect-good sample
> [O] bisect-bad sample
>
> ***************************************************************************************************
> lkp-cfl-d1: 12 threads Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz with 8G memory
>
>
> ***************************************************************************************************
> lkp-skl-fpga01: 104 threads Skylake with 192G memory
> =========================================================================================
> compiler/cpufreq_governor/kconfig/rootfs/runtime/size/tbox_group/test/testcase/ucode:
> gcc-7/performance/x86_64-rhel-7.6/debian-x86_64-2019-09-23.cgz/300s/8T/lkp-skl-fpga01/anon-cow-seq/vm-scalability/0x2000064
>
> commit:
> fcf0553db6 ("sched/fair: Remove meaningless imbalance calculation")
> 0b0695f2b3 ("sched/fair: Rework load_balance()")
>
> fcf0553db6f4c793 0b0695f2b34a4afa3f6e9aa1ff0
> ---------------- ---------------------------
> %stddev %change %stddev
> \ | \
> 413301 +3.1% 426103 vm-scalability.median
> 0.04 ą 2% -34.0% 0.03 ą 12% vm-scalability.median_stddev
> 43837589 +2.4% 44902458 vm-scalability.throughput
> 181085 -18.7% 147221 vm-scalability.time.involuntary_context_switches
> 12762365 ą 2% +3.9% 13262025 vm-scalability.time.minor_page_faults
> 7773 +2.9% 7997 vm-scalability.time.percent_of_cpu_this_job_got
> 11449 +1.2% 11589 vm-scalability.time.system_time
> 12024 +4.7% 12584 vm-scalability.time.user_time
> 439194 ą 2% +46.0% 641402 ą 2% vm-scalability.time.voluntary_context_switches
> 1.148e+10 +5.0% 1.206e+10 vm-scalability.workload
> 0.00 ą 54% +0.0 0.00 ą 17% mpstat.cpu.all.iowait%
> 4767597 +52.5% 7268430 ą 41% numa-numastat.node1.local_node
> 4781030 +52.3% 7280347 ą 41% numa-numastat.node1.numa_hit
> 24.75 -9.1% 22.50 ą 2% vmstat.cpu.id
> 37.50 +4.7% 39.25 vmstat.cpu.us
> 6643 ą 3% +15.1% 7647 vmstat.system.cs
> 12220504 +33.4% 16298593 ą 4% cpuidle.C1.time
> 260215 ą 6% +55.3% 404158 ą 3% cpuidle.C1.usage
> 4986034 ą 3% +56.2% 7786811 ą 2% cpuidle.POLL.time
> 145941 ą 3% +61.2% 235218 ą 2% cpuidle.POLL.usage
> 1990 +3.0% 2049 turbostat.Avg_MHz
> 254633 ą 6% +56.7% 398892 ą 4% turbostat.C1
> 0.04 +0.0 0.05 turbostat.C1%
> 309.99 +1.5% 314.75 turbostat.RAMWatt
> 1688 ą 11% +17.4% 1983 ą 5% slabinfo.UNIX.active_objs
> 1688 ą 11% +17.4% 1983 ą 5% slabinfo.UNIX.num_objs
> 2460 ą 3% -15.8% 2072 ą 11% slabinfo.dmaengine-unmap-16.active_objs
> 2460 ą 3% -15.8% 2072 ą 11% slabinfo.dmaengine-unmap-16.num_objs
> 2814 ą 9% +14.6% 3225 ą 4% slabinfo.sock_inode_cache.active_objs
> 2814 ą 9% +14.6% 3225 ą 4% slabinfo.sock_inode_cache.num_objs
> 0.67 ą 5% +0.1 0.73 ą 3% perf-profile.calltrace.cycles-pp.__alloc_pages_nodemask.alloc_pages_vma.do_huge_pmd_wp_page.__handle_mm_fault.handle_mm_fault
> 0.68 ą 6% +0.1 0.74 ą 2% perf-profile.calltrace.cycles-pp.alloc_pages_vma.do_huge_pmd_wp_page.__handle_mm_fault.handle_mm_fault.__do_page_fault
> 0.05 +0.0 0.07 ą 7% perf-profile.children.cycles-pp.schedule
> 0.06 +0.0 0.08 ą 6% perf-profile.children.cycles-pp.__wake_up_common
> 0.06 ą 7% +0.0 0.08 ą 6% perf-profile.children.cycles-pp.wake_up_page_bit
> 0.23 ą 7% +0.0 0.28 ą 5% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
> 0.00 +0.1 0.05 perf-profile.children.cycles-pp.drm_fb_helper_sys_imageblit
> 0.00 +0.1 0.05 perf-profile.children.cycles-pp.sys_imageblit
> 29026 ą 3% -26.7% 21283 ą 44% numa-vmstat.node0.nr_inactive_anon
> 30069 ą 3% -20.5% 23905 ą 26% numa-vmstat.node0.nr_shmem
> 12120 ą 2% -15.5% 10241 ą 12% numa-vmstat.node0.nr_slab_reclaimable
> 29026 ą 3% -26.7% 21283 ą 44% numa-vmstat.node0.nr_zone_inactive_anon
> 4010893 +16.1% 4655889 ą 9% numa-vmstat.node1.nr_active_anon
> 3982581 +16.3% 4632344 ą 9% numa-vmstat.node1.nr_anon_pages
> 6861 +16.1% 7964 ą 8% numa-vmstat.node1.nr_anon_transparent_hugepages
> 2317 ą 42% +336.9% 10125 ą 93% numa-vmstat.node1.nr_inactive_anon
> 6596 ą 4% +18.2% 7799 ą 14% numa-vmstat.node1.nr_kernel_stack
> 9629 ą 8% +66.4% 16020 ą 41% numa-vmstat.node1.nr_shmem
> 7558 ą 3% +26.5% 9561 ą 14% numa-vmstat.node1.nr_slab_reclaimable
> 4010227 +16.1% 4655056 ą 9% numa-vmstat.node1.nr_zone_active_anon
> 2317 ą 42% +336.9% 10125 ą 93% numa-vmstat.node1.nr_zone_inactive_anon
> 2859663 ą 2% +46.2% 4179500 ą 36% numa-vmstat.node1.numa_hit
> 2680260 ą 2% +49.3% 4002218 ą 37% numa-vmstat.node1.numa_local
> 116661 ą 3% -26.3% 86010 ą 44% numa-meminfo.node0.Inactive
> 116192 ą 3% -26.7% 85146 ą 44% numa-meminfo.node0.Inactive(anon)
> 48486 ą 2% -15.5% 40966 ą 12% numa-meminfo.node0.KReclaimable
> 48486 ą 2% -15.5% 40966 ą 12% numa-meminfo.node0.SReclaimable
> 120367 ą 3% -20.5% 95642 ą 26% numa-meminfo.node0.Shmem
> 16210528 +15.2% 18673368 ą 6% numa-meminfo.node1.Active
> 16210394 +15.2% 18673287 ą 6% numa-meminfo.node1.Active(anon)
> 14170064 +15.6% 16379835 ą 7% numa-meminfo.node1.AnonHugePages
> 16113351 +15.3% 18577254 ą 7% numa-meminfo.node1.AnonPages
> 10534 ą 33% +293.8% 41480 ą 92% numa-meminfo.node1.Inactive
> 9262 ą 42% +338.2% 40589 ą 93% numa-meminfo.node1.Inactive(anon)
> 30235 ą 3% +26.5% 38242 ą 14% numa-meminfo.node1.KReclaimable
> 6594 ą 4% +18.3% 7802 ą 14% numa-meminfo.node1.KernelStack
> 17083646 +15.1% 19656922 ą 7% numa-meminfo.node1.MemUsed
> 30235 ą 3% +26.5% 38242 ą 14% numa-meminfo.node1.SReclaimable
> 38540 ą 8% +66.4% 64117 ą 42% numa-meminfo.node1.Shmem
> 106342 +19.8% 127451 ą 11% numa-meminfo.node1.Slab
> 9479688 +4.5% 9905902 proc-vmstat.nr_active_anon
> 9434298 +4.5% 9856978 proc-vmstat.nr_anon_pages
> 16194 +4.3% 16895 proc-vmstat.nr_anon_transparent_hugepages
> 276.75 +3.6% 286.75 proc-vmstat.nr_dirtied
> 3888633 -1.1% 3845882 proc-vmstat.nr_dirty_background_threshold
> 7786774 -1.1% 7701168 proc-vmstat.nr_dirty_threshold
> 39168820 -1.1% 38741444 proc-vmstat.nr_free_pages
> 50391 +1.0% 50904 proc-vmstat.nr_slab_unreclaimable
> 257.50 +3.6% 266.75 proc-vmstat.nr_written
> 9479678 +4.5% 9905895 proc-vmstat.nr_zone_active_anon
> 1501517 -5.9% 1412958 proc-vmstat.numa_hint_faults
> 1075936 -13.1% 934706 proc-vmstat.numa_hint_faults_local
> 17306395 +4.8% 18141722 proc-vmstat.numa_hit
> 5211079 +4.2% 5427541 proc-vmstat.numa_huge_pte_updates
> 17272620 +4.8% 18107691 proc-vmstat.numa_local
> 33774 +0.8% 34031 proc-vmstat.numa_other
> 690793 ą 3% -13.7% 596166 ą 2% proc-vmstat.numa_pages_migrated
> 2.669e+09 +4.2% 2.78e+09 proc-vmstat.numa_pte_updates
> 2.755e+09 +5.6% 2.909e+09 proc-vmstat.pgalloc_normal
> 13573227 ą 2% +3.6% 14060842 proc-vmstat.pgfault
> 2.752e+09 +5.6% 2.906e+09 proc-vmstat.pgfree
> 1.723e+08 ą 2% +14.3% 1.97e+08 ą 8% proc-vmstat.pgmigrate_fail
> 690793 ą 3% -13.7% 596166 ą 2% proc-vmstat.pgmigrate_success
> 5015265 +5.0% 5266730 proc-vmstat.thp_deferred_split_page
> 5019661 +5.0% 5271482 proc-vmstat.thp_fault_alloc
> 18284 ą 62% -79.9% 3681 ą172% sched_debug.cfs_rq:/.MIN_vruntime.avg
> 1901618 ą 62% -89.9% 192494 ą172% sched_debug.cfs_rq:/.MIN_vruntime.max
> 185571 ą 62% -85.8% 26313 ą172% sched_debug.cfs_rq:/.MIN_vruntime.stddev
> 15241 ą 6% -36.6% 9655 ą 6% sched_debug.cfs_rq:/.exec_clock.stddev
> 18284 ą 62% -79.9% 3681 ą172% sched_debug.cfs_rq:/.max_vruntime.avg
> 1901618 ą 62% -89.9% 192494 ą172% sched_debug.cfs_rq:/.max_vruntime.max
> 185571 ą 62% -85.8% 26313 ą172% sched_debug.cfs_rq:/.max_vruntime.stddev
> 898812 ą 7% -31.2% 618552 ą 5% sched_debug.cfs_rq:/.min_vruntime.stddev
> 10.30 ą 12% +34.5% 13.86 ą 6% sched_debug.cfs_rq:/.nr_spread_over.avg
> 34.75 ą 8% +95.9% 68.08 ą 4% sched_debug.cfs_rq:/.nr_spread_over.max
> 9.12 ą 11% +82.3% 16.62 ą 9% sched_debug.cfs_rq:/.nr_spread_over.stddev
> -1470498 -31.9% -1000709 sched_debug.cfs_rq:/.spread0.min
> 899820 ą 7% -31.2% 618970 ą 5% sched_debug.cfs_rq:/.spread0.stddev
> 1589 ą 9% -19.2% 1284 ą 9% sched_debug.cfs_rq:/.util_avg.max
> 0.54 ą 39% +7484.6% 41.08 ą 92% sched_debug.cfs_rq:/.util_est_enqueued.min
> 238.84 ą 8% -33.2% 159.61 ą 26% sched_debug.cfs_rq:/.util_est_enqueued.stddev
> 10787 ą 2% +13.8% 12274 sched_debug.cpu.nr_switches.avg
> 35242 ą 9% +32.3% 46641 ą 25% sched_debug.cpu.nr_switches.max
> 9139 ą 3% +16.4% 10636 sched_debug.cpu.sched_count.avg
> 32025 ą 10% +34.6% 43091 ą 27% sched_debug.cpu.sched_count.max
> 4016 ą 2% +14.7% 4606 ą 5% sched_debug.cpu.sched_count.min
> 2960 +38.3% 4093 sched_debug.cpu.sched_goidle.avg
> 11201 ą 24% +75.8% 19691 ą 26% sched_debug.cpu.sched_goidle.max
> 1099 ą 6% +56.9% 1725 ą 6% sched_debug.cpu.sched_goidle.min
> 1877 ą 10% +32.5% 2487 ą 17% sched_debug.cpu.sched_goidle.stddev
> 4348 ą 3% +19.3% 5188 sched_debug.cpu.ttwu_count.avg
> 17832 ą 11% +78.6% 31852 ą 29% sched_debug.cpu.ttwu_count.max
> 1699 ą 6% +28.2% 2178 ą 7% sched_debug.cpu.ttwu_count.min
> 1357 ą 10% -22.6% 1050 ą 4% sched_debug.cpu.ttwu_local.avg
> 11483 ą 5% -25.0% 8614 ą 15% sched_debug.cpu.ttwu_local.max
> 1979 ą 12% -36.8% 1251 ą 10% sched_debug.cpu.ttwu_local.stddev
> 3.941e+10 +5.0% 4.137e+10 perf-stat.i.branch-instructions
> 0.02 ą 50% -0.0 0.02 ą 5% perf-stat.i.branch-miss-rate%
> 67.94 -3.9 63.99 perf-stat.i.cache-miss-rate%
> 8.329e+08 -1.9% 8.17e+08 perf-stat.i.cache-misses
> 1.224e+09 +4.5% 1.28e+09 perf-stat.i.cache-references
> 6650 ą 3% +15.5% 7678 perf-stat.i.context-switches
> 1.64 -1.8% 1.61 perf-stat.i.cpi
> 2.037e+11 +2.8% 2.095e+11 perf-stat.i.cpu-cycles
> 257.56 -4.0% 247.13 perf-stat.i.cpu-migrations
> 244.94 +4.5% 255.91 perf-stat.i.cycles-between-cache-misses
> 1189446 ą 2% +3.2% 1227527 perf-stat.i.dTLB-load-misses
> 2.669e+10 +4.7% 2.794e+10 perf-stat.i.dTLB-loads
> 0.00 ą 7% -0.0 0.00 perf-stat.i.dTLB-store-miss-rate%
> 337782 +4.5% 353044 perf-stat.i.dTLB-store-misses
> 9.096e+09 +4.7% 9.526e+09 perf-stat.i.dTLB-stores
> 39.50 +2.1 41.64 perf-stat.i.iTLB-load-miss-rate%
> 296305 ą 2% +9.0% 323020 perf-stat.i.iTLB-load-misses
> 1.238e+11 +4.9% 1.299e+11 perf-stat.i.instructions
> 428249 ą 2% -4.4% 409553 perf-stat.i.instructions-per-iTLB-miss
> 0.61 +1.6% 0.62 perf-stat.i.ipc
> 44430 +3.8% 46121 perf-stat.i.minor-faults
> 54.82 +3.9 58.73 perf-stat.i.node-load-miss-rate%
> 68519419 ą 4% -11.7% 60479057 ą 6% perf-stat.i.node-load-misses
> 49879161 ą 3% -20.7% 39554915 ą 4% perf-stat.i.node-loads
> 44428 +3.8% 46119 perf-stat.i.page-faults
> 0.02 -0.0 0.01 ą 5% perf-stat.overall.branch-miss-rate%
> 68.03 -4.2 63.83 perf-stat.overall.cache-miss-rate%
> 1.65 -2.0% 1.61 perf-stat.overall.cpi
> 244.61 +4.8% 256.41 perf-stat.overall.cycles-between-cache-misses
> 30.21 +2.2 32.38 perf-stat.overall.iTLB-load-miss-rate%
> 417920 ą 2% -3.7% 402452 perf-stat.overall.instructions-per-iTLB-miss
> 0.61 +2.1% 0.62 perf-stat.overall.ipc
> 57.84 +2.6 60.44 perf-stat.overall.node-load-miss-rate%
> 3.925e+10 +5.1% 4.124e+10 perf-stat.ps.branch-instructions
> 8.295e+08 -1.8% 8.144e+08 perf-stat.ps.cache-misses
> 1.219e+09 +4.6% 1.276e+09 perf-stat.ps.cache-references
> 6625 ą 3% +15.4% 7648 perf-stat.ps.context-switches
> 2.029e+11 +2.9% 2.088e+11 perf-stat.ps.cpu-cycles
> 256.82 -4.2% 246.09 perf-stat.ps.cpu-migrations
> 1184763 ą 2% +3.3% 1223366 perf-stat.ps.dTLB-load-misses
> 2.658e+10 +4.8% 2.786e+10 perf-stat.ps.dTLB-loads
> 336658 +4.5% 351710 perf-stat.ps.dTLB-store-misses
> 9.059e+09 +4.8% 9.497e+09 perf-stat.ps.dTLB-stores
> 295140 ą 2% +9.0% 321824 perf-stat.ps.iTLB-load-misses
> 1.233e+11 +5.0% 1.295e+11 perf-stat.ps.instructions
> 44309 +3.7% 45933 perf-stat.ps.minor-faults
> 68208972 ą 4% -11.6% 60272675 ą 6% perf-stat.ps.node-load-misses
> 49689740 ą 3% -20.7% 39401789 ą 4% perf-stat.ps.node-loads
> 44308 +3.7% 45932 perf-stat.ps.page-faults
> 3.732e+13 +5.1% 3.922e+13 perf-stat.total.instructions
> 14949 ą 2% +14.5% 17124 ą 11% softirqs.CPU0.SCHED
> 9940 +37.8% 13700 ą 24% softirqs.CPU1.SCHED
> 9370 ą 2% +28.2% 12014 ą 16% softirqs.CPU10.SCHED
> 17637 ą 2% -16.5% 14733 ą 16% softirqs.CPU101.SCHED
> 17846 ą 3% -17.4% 14745 ą 16% softirqs.CPU103.SCHED
> 9552 +24.7% 11916 ą 17% softirqs.CPU11.SCHED
> 9210 ą 5% +27.9% 11784 ą 16% softirqs.CPU12.SCHED
> 9378 ą 3% +27.7% 11974 ą 16% softirqs.CPU13.SCHED
> 9164 ą 2% +29.4% 11856 ą 18% softirqs.CPU14.SCHED
> 9215 +21.2% 11170 ą 19% softirqs.CPU15.SCHED
> 9118 ą 2% +29.1% 11772 ą 16% softirqs.CPU16.SCHED
> 9413 +29.2% 12165 ą 18% softirqs.CPU17.SCHED
> 9309 ą 2% +29.9% 12097 ą 17% softirqs.CPU18.SCHED
> 9423 +26.1% 11880 ą 15% softirqs.CPU19.SCHED
> 9010 ą 7% +37.8% 12420 ą 18% softirqs.CPU2.SCHED
> 9382 ą 3% +27.0% 11916 ą 15% softirqs.CPU20.SCHED
> 9102 ą 4% +30.0% 11830 ą 16% softirqs.CPU21.SCHED
> 9543 ą 3% +23.4% 11780 ą 18% softirqs.CPU22.SCHED
> 8998 ą 5% +29.2% 11630 ą 18% softirqs.CPU24.SCHED
> 9254 ą 2% +23.9% 11462 ą 19% softirqs.CPU25.SCHED
> 18450 ą 4% -16.9% 15341 ą 16% softirqs.CPU26.SCHED
> 17551 ą 4% -14.8% 14956 ą 13% softirqs.CPU27.SCHED
> 17575 ą 4% -14.6% 15010 ą 14% softirqs.CPU28.SCHED
> 17515 ą 5% -14.2% 15021 ą 13% softirqs.CPU29.SCHED
> 17715 ą 2% -16.1% 14856 ą 13% softirqs.CPU30.SCHED
> 17754 ą 4% -16.1% 14904 ą 13% softirqs.CPU31.SCHED
> 17675 ą 2% -17.0% 14679 ą 21% softirqs.CPU32.SCHED
> 17625 ą 2% -16.0% 14813 ą 13% softirqs.CPU34.SCHED
> 17619 ą 2% -14.7% 15024 ą 14% softirqs.CPU35.SCHED
> 17887 ą 3% -17.0% 14841 ą 14% softirqs.CPU36.SCHED
> 17658 ą 3% -16.3% 14771 ą 12% softirqs.CPU38.SCHED
> 17501 ą 2% -15.3% 14816 ą 14% softirqs.CPU39.SCHED
> 9360 ą 2% +25.4% 11740 ą 14% softirqs.CPU4.SCHED
> 17699 ą 4% -16.2% 14827 ą 14% softirqs.CPU42.SCHED
> 17580 ą 3% -16.5% 14679 ą 15% softirqs.CPU43.SCHED
> 17658 ą 3% -17.1% 14644 ą 14% softirqs.CPU44.SCHED
> 17452 ą 4% -14.0% 15001 ą 15% softirqs.CPU46.SCHED
> 17599 ą 4% -17.4% 14544 ą 14% softirqs.CPU47.SCHED
> 17792 ą 3% -16.5% 14864 ą 14% softirqs.CPU48.SCHED
> 17333 ą 2% -16.7% 14445 ą 14% softirqs.CPU49.SCHED
> 9483 +32.3% 12547 ą 24% softirqs.CPU5.SCHED
> 17842 ą 3% -15.9% 14997 ą 16% softirqs.CPU51.SCHED
> 9051 ą 2% +23.3% 11160 ą 13% softirqs.CPU52.SCHED
> 9385 ą 3% +25.2% 11752 ą 16% softirqs.CPU53.SCHED
> 9446 ą 6% +24.9% 11798 ą 14% softirqs.CPU54.SCHED
> 10006 ą 6% +22.4% 12249 ą 14% softirqs.CPU55.SCHED
> 9657 +22.0% 11780 ą 16% softirqs.CPU57.SCHED
> 9399 +27.5% 11980 ą 15% softirqs.CPU58.SCHED
> 9234 ą 3% +27.7% 11795 ą 14% softirqs.CPU59.SCHED
> 9726 ą 6% +24.0% 12062 ą 16% softirqs.CPU6.SCHED
> 9165 ą 2% +23.7% 11342 ą 14% softirqs.CPU60.SCHED
> 9357 ą 2% +25.8% 11774 ą 15% softirqs.CPU61.SCHED
> 9406 ą 3% +25.2% 11780 ą 16% softirqs.CPU62.SCHED
> 9489 +23.2% 11688 ą 15% softirqs.CPU63.SCHED
> 9399 ą 2% +23.5% 11604 ą 16% softirqs.CPU65.SCHED
> 8950 ą 2% +31.6% 11774 ą 16% softirqs.CPU66.SCHED
> 9260 +21.7% 11267 ą 19% softirqs.CPU67.SCHED
> 9187 +27.1% 11672 ą 17% softirqs.CPU68.SCHED
> 9443 ą 2% +25.5% 11847 ą 17% softirqs.CPU69.SCHED
> 9144 ą 3% +28.0% 11706 ą 16% softirqs.CPU7.SCHED
> 9276 ą 2% +28.0% 11871 ą 17% softirqs.CPU70.SCHED
> 9494 +21.4% 11526 ą 14% softirqs.CPU71.SCHED
> 9124 ą 3% +27.8% 11657 ą 17% softirqs.CPU72.SCHED
> 9189 ą 3% +25.9% 11568 ą 16% softirqs.CPU73.SCHED
> 9392 ą 2% +23.7% 11619 ą 16% softirqs.CPU74.SCHED
> 17821 ą 3% -14.7% 15197 ą 17% softirqs.CPU78.SCHED
> 17581 ą 2% -15.7% 14827 ą 15% softirqs.CPU79.SCHED
> 9123 +28.2% 11695 ą 15% softirqs.CPU8.SCHED
> 17524 ą 2% -16.7% 14601 ą 14% softirqs.CPU80.SCHED
> 17644 ą 3% -16.2% 14782 ą 14% softirqs.CPU81.SCHED
> 17705 ą 3% -18.6% 14414 ą 22% softirqs.CPU84.SCHED
> 17679 ą 2% -14.1% 15185 ą 11% softirqs.CPU85.SCHED
> 17434 ą 3% -15.5% 14724 ą 14% softirqs.CPU86.SCHED
> 17409 ą 2% -15.0% 14794 ą 13% softirqs.CPU87.SCHED
> 17470 ą 3% -15.7% 14730 ą 13% softirqs.CPU88.SCHED
> 17748 ą 4% -17.1% 14721 ą 12% softirqs.CPU89.SCHED
> 9323 +28.0% 11929 ą 17% softirqs.CPU9.SCHED
> 17471 ą 2% -16.9% 14525 ą 13% softirqs.CPU90.SCHED
> 17900 ą 3% -17.0% 14850 ą 14% softirqs.CPU94.SCHED
> 17599 ą 4% -17.4% 14544 ą 15% softirqs.CPU95.SCHED
> 17697 ą 4% -17.7% 14569 ą 13% softirqs.CPU96.SCHED
> 17561 ą 3% -15.1% 14901 ą 13% softirqs.CPU97.SCHED
> 17404 ą 3% -16.1% 14601 ą 13% softirqs.CPU98.SCHED
> 17802 ą 3% -19.4% 14344 ą 15% softirqs.CPU99.SCHED
> 1310 ą 10% -17.0% 1088 ą 5% interrupts.CPU1.RES:Rescheduling_interrupts
> 3427 +13.3% 3883 ą 9% interrupts.CPU10.CAL:Function_call_interrupts
> 736.50 ą 20% +34.4% 989.75 ą 17% interrupts.CPU100.RES:Rescheduling_interrupts
> 3421 ą 3% +14.6% 3921 ą 9% interrupts.CPU101.CAL:Function_call_interrupts
> 4873 ą 8% +16.2% 5662 ą 7% interrupts.CPU101.NMI:Non-maskable_interrupts
> 4873 ą 8% +16.2% 5662 ą 7% interrupts.CPU101.PMI:Performance_monitoring_interrupts
> 629.50 ą 19% +83.2% 1153 ą 46% interrupts.CPU101.RES:Rescheduling_interrupts
> 661.75 ą 14% +25.7% 832.00 ą 13% interrupts.CPU102.RES:Rescheduling_interrupts
> 4695 ą 5% +15.5% 5420 ą 9% interrupts.CPU103.NMI:Non-maskable_interrupts
> 4695 ą 5% +15.5% 5420 ą 9% interrupts.CPU103.PMI:Performance_monitoring_interrupts
> 3460 +12.1% 3877 ą 9% interrupts.CPU11.CAL:Function_call_interrupts
> 691.50 ą 7% +41.0% 975.00 ą 32% interrupts.CPU19.RES:Rescheduling_interrupts
> 3413 ą 2% +13.4% 3870 ą 10% interrupts.CPU20.CAL:Function_call_interrupts
> 3413 ą 2% +13.4% 3871 ą 10% interrupts.CPU22.CAL:Function_call_interrupts
> 863.00 ą 36% +45.3% 1254 ą 24% interrupts.CPU23.RES:Rescheduling_interrupts
> 659.75 ą 12% +83.4% 1209 ą 20% interrupts.CPU26.RES:Rescheduling_interrupts
> 615.00 ą 10% +87.8% 1155 ą 14% interrupts.CPU27.RES:Rescheduling_interrupts
> 663.75 ą 5% +67.9% 1114 ą 7% interrupts.CPU28.RES:Rescheduling_interrupts
> 3421 ą 4% +13.4% 3879 ą 9% interrupts.CPU29.CAL:Function_call_interrupts
> 805.25 ą 16% +33.0% 1071 ą 15% interrupts.CPU29.RES:Rescheduling_interrupts
> 3482 ą 3% +11.0% 3864 ą 8% interrupts.CPU3.CAL:Function_call_interrupts
> 819.75 ą 19% +48.4% 1216 ą 12% interrupts.CPU30.RES:Rescheduling_interrupts
> 777.25 ą 8% +31.6% 1023 ą 6% interrupts.CPU31.RES:Rescheduling_interrupts
> 844.50 ą 25% +41.7% 1196 ą 20% interrupts.CPU32.RES:Rescheduling_interrupts
> 722.75 ą 14% +94.2% 1403 ą 26% interrupts.CPU33.RES:Rescheduling_interrupts
> 3944 ą 25% +36.8% 5394 ą 9% interrupts.CPU34.NMI:Non-maskable_interrupts
> 3944 ą 25% +36.8% 5394 ą 9% interrupts.CPU34.PMI:Performance_monitoring_interrupts
> 781.75 ą 9% +45.3% 1136 ą 27% interrupts.CPU34.RES:Rescheduling_interrupts
> 735.50 ą 9% +33.3% 980.75 ą 4% interrupts.CPU35.RES:Rescheduling_interrupts
> 691.75 ą 10% +41.6% 979.50 ą 13% interrupts.CPU36.RES:Rescheduling_interrupts
> 727.00 ą 16% +47.7% 1074 ą 15% interrupts.CPU37.RES:Rescheduling_interrupts
> 4413 ą 7% +24.9% 5511 ą 9% interrupts.CPU38.NMI:Non-maskable_interrupts
> 4413 ą 7% +24.9% 5511 ą 9% interrupts.CPU38.PMI:Performance_monitoring_interrupts
> 708.75 ą 25% +62.6% 1152 ą 22% interrupts.CPU38.RES:Rescheduling_interrupts
> 666.50 ą 7% +57.8% 1052 ą 13% interrupts.CPU39.RES:Rescheduling_interrupts
> 765.75 ą 11% +25.2% 958.75 ą 14% interrupts.CPU4.RES:Rescheduling_interrupts
> 3395 ą 2% +15.1% 3908 ą 10% interrupts.CPU40.CAL:Function_call_interrupts
> 770.00 ą 16% +45.3% 1119 ą 18% interrupts.CPU40.RES:Rescheduling_interrupts
> 740.50 ą 26% +61.9% 1198 ą 19% interrupts.CPU41.RES:Rescheduling_interrupts
> 3459 ą 2% +12.9% 3905 ą 11% interrupts.CPU42.CAL:Function_call_interrupts
> 4530 ą 5% +22.8% 5564 ą 9% interrupts.CPU42.NMI:Non-maskable_interrupts
> 4530 ą 5% +22.8% 5564 ą 9% interrupts.CPU42.PMI:Performance_monitoring_interrupts
> 3330 ą 25% +60.0% 5328 ą 10% interrupts.CPU44.NMI:Non-maskable_interrupts
> 3330 ą 25% +60.0% 5328 ą 10% interrupts.CPU44.PMI:Performance_monitoring_interrupts
> 686.25 ą 9% +48.4% 1018 ą 10% interrupts.CPU44.RES:Rescheduling_interrupts
> 702.00 ą 15% +38.6% 973.25 ą 5% interrupts.CPU45.RES:Rescheduling_interrupts
> 4742 ą 7% +19.3% 5657 ą 8% interrupts.CPU46.NMI:Non-maskable_interrupts
> 4742 ą 7% +19.3% 5657 ą 8% interrupts.CPU46.PMI:Performance_monitoring_interrupts
> 732.75 ą 6% +51.9% 1113 ą 7% interrupts.CPU46.RES:Rescheduling_interrupts
> 775.50 ą 17% +41.3% 1095 ą 6% interrupts.CPU47.RES:Rescheduling_interrupts
> 670.75 ą 5% +60.7% 1078 ą 6% interrupts.CPU48.RES:Rescheduling_interrupts
> 4870 ą 8% +16.5% 5676 ą 7% interrupts.CPU49.NMI:Non-maskable_interrupts
> 4870 ą 8% +16.5% 5676 ą 7% interrupts.CPU49.PMI:Performance_monitoring_interrupts
> 694.75 ą 12% +25.8% 874.00 ą 11% interrupts.CPU49.RES:Rescheduling_interrupts
> 686.00 ą 9% +52.0% 1042 ą 20% interrupts.CPU50.RES:Rescheduling_interrupts
> 3361 +17.2% 3938 ą 9% interrupts.CPU51.CAL:Function_call_interrupts
> 4707 ą 6% +16.0% 5463 ą 8% interrupts.CPU51.NMI:Non-maskable_interrupts
> 4707 ą 6% +16.0% 5463 ą 8% interrupts.CPU51.PMI:Performance_monitoring_interrupts
> 638.75 ą 12% +28.6% 821.25 ą 15% interrupts.CPU54.RES:Rescheduling_interrupts
> 677.50 ą 8% +51.8% 1028 ą 29% interrupts.CPU58.RES:Rescheduling_interrupts
> 3465 ą 2% +12.0% 3880 ą 9% interrupts.CPU6.CAL:Function_call_interrupts
> 641.25 ą 2% +26.1% 808.75 ą 10% interrupts.CPU60.RES:Rescheduling_interrupts
> 599.75 ą 2% +45.6% 873.50 ą 8% interrupts.CPU62.RES:Rescheduling_interrupts
> 661.50 ą 9% +52.4% 1008 ą 27% interrupts.CPU63.RES:Rescheduling_interrupts
> 611.00 ą 12% +31.1% 801.00 ą 13% interrupts.CPU69.RES:Rescheduling_interrupts
> 3507 ą 2% +10.8% 3888 ą 9% interrupts.CPU7.CAL:Function_call_interrupts
> 664.00 ą 5% +32.3% 878.50 ą 23% interrupts.CPU70.RES:Rescheduling_interrupts
> 5780 ą 9% -38.8% 3540 ą 37% interrupts.CPU73.NMI:Non-maskable_interrupts
> 5780 ą 9% -38.8% 3540 ą 37% interrupts.CPU73.PMI:Performance_monitoring_interrupts
> 5787 ą 9% -26.7% 4243 ą 28% interrupts.CPU76.NMI:Non-maskable_interrupts
> 5787 ą 9% -26.7% 4243 ą 28% interrupts.CPU76.PMI:Performance_monitoring_interrupts
> 751.50 ą 15% +88.0% 1413 ą 37% interrupts.CPU78.RES:Rescheduling_interrupts
> 725.50 ą 12% +82.9% 1327 ą 36% interrupts.CPU79.RES:Rescheduling_interrupts
> 714.00 ą 18% +33.2% 951.00 ą 15% interrupts.CPU80.RES:Rescheduling_interrupts
> 706.25 ą 19% +55.6% 1098 ą 27% interrupts.CPU82.RES:Rescheduling_interrupts
> 4524 ą 6% +19.6% 5409 ą 8% interrupts.CPU83.NMI:Non-maskable_interrupts
> 4524 ą 6% +19.6% 5409 ą 8% interrupts.CPU83.PMI:Performance_monitoring_interrupts
> 666.75 ą 15% +37.3% 915.50 ą 4% interrupts.CPU83.RES:Rescheduling_interrupts
> 782.50 ą 26% +57.6% 1233 ą 21% interrupts.CPU84.RES:Rescheduling_interrupts
> 622.75 ą 12% +77.8% 1107 ą 17% interrupts.CPU85.RES:Rescheduling_interrupts
> 3465 ą 3% +13.5% 3933 ą 9% interrupts.CPU86.CAL:Function_call_interrupts
> 714.75 ą 14% +47.0% 1050 ą 10% interrupts.CPU86.RES:Rescheduling_interrupts
> 3519 ą 2% +11.7% 3929 ą 9% interrupts.CPU87.CAL:Function_call_interrupts
> 582.75 ą 10% +54.2% 898.75 ą 11% interrupts.CPU87.RES:Rescheduling_interrupts
> 713.00 ą 10% +36.6% 974.25 ą 11% interrupts.CPU88.RES:Rescheduling_interrupts
> 690.50 ą 13% +53.0% 1056 ą 13% interrupts.CPU89.RES:Rescheduling_interrupts
> 3477 +11.0% 3860 ą 8% interrupts.CPU9.CAL:Function_call_interrupts
> 684.50 ą 14% +39.7% 956.25 ą 11% interrupts.CPU90.RES:Rescheduling_interrupts
> 3946 ą 21% +39.8% 5516 ą 10% interrupts.CPU91.NMI:Non-maskable_interrupts
> 3946 ą 21% +39.8% 5516 ą 10% interrupts.CPU91.PMI:Performance_monitoring_interrupts
> 649.00 ą 13% +54.3% 1001 ą 6% interrupts.CPU91.RES:Rescheduling_interrupts
> 674.25 ą 21% +39.5% 940.25 ą 11% interrupts.CPU92.RES:Rescheduling_interrupts
> 3971 ą 26% +41.2% 5606 ą 8% interrupts.CPU94.NMI:Non-maskable_interrupts
> 3971 ą 26% +41.2% 5606 ą 8% interrupts.CPU94.PMI:Performance_monitoring_interrupts
> 4129 ą 22% +33.2% 5499 ą 9% interrupts.CPU95.NMI:Non-maskable_interrupts
> 4129 ą 22% +33.2% 5499 ą 9% interrupts.CPU95.PMI:Performance_monitoring_interrupts
> 685.75 ą 14% +38.0% 946.50 ą 9% interrupts.CPU96.RES:Rescheduling_interrupts
> 4630 ą 11% +18.3% 5477 ą 8% interrupts.CPU97.NMI:Non-maskable_interrupts
> 4630 ą 11% +18.3% 5477 ą 8% interrupts.CPU97.PMI:Performance_monitoring_interrupts
> 4835 ą 9% +16.3% 5622 ą 9% interrupts.CPU98.NMI:Non-maskable_interrupts
> 4835 ą 9% +16.3% 5622 ą 9% interrupts.CPU98.PMI:Performance_monitoring_interrupts
> 596.25 ą 11% +81.8% 1083 ą 9% interrupts.CPU98.RES:Rescheduling_interrupts
> 674.75 ą 17% +43.7% 969.50 ą 5% interrupts.CPU99.RES:Rescheduling_interrupts
> 78.25 ą 13% +21.4% 95.00 ą 10% interrupts.IWI:IRQ_work_interrupts
> 85705 ą 6% +26.0% 107990 ą 6% interrupts.RES:Rescheduling_interrupts
>
>
>
> ***************************************************************************************************
> lkp-bdw-ep6: 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory
> =========================================================================================
> class/compiler/cpufreq_governor/disk/kconfig/nr_threads/rootfs/sc_pid_max/tbox_group/testcase/testtime/ucode:
> scheduler/gcc-7/performance/1HDD/x86_64-rhel-7.6/100%/debian-x86_64-2019-11-14.cgz/4194304/lkp-bdw-ep6/stress-ng/1s/0xb000038
>
> commit:
> fcf0553db6 ("sched/fair: Remove meaningless imbalance calculation")
> 0b0695f2b3 ("sched/fair: Rework load_balance()")
>
> fcf0553db6f4c793 0b0695f2b34a4afa3f6e9aa1ff0
> ---------------- ---------------------------
> %stddev %change %stddev
> \ | \
> 887157 ą 4% -23.1% 682080 ą 3% stress-ng.fault.ops
> 887743 ą 4% -23.1% 682337 ą 3% stress-ng.fault.ops_per_sec
> 9537184 ą 10% -21.2% 7518352 ą 14% stress-ng.hrtimers.ops_per_sec
> 360922 ą 13% -21.1% 284734 ą 6% stress-ng.kill.ops
> 361115 ą 13% -21.1% 284810 ą 6% stress-ng.kill.ops_per_sec
> 23260649 -26.9% 17006477 ą 24% stress-ng.mq.ops
> 23255884 -26.9% 17004540 ą 24% stress-ng.mq.ops_per_sec
> 3291588 ą 3% +42.5% 4690316 ą 2% stress-ng.schedpolicy.ops
> 3327913 ą 3% +41.5% 4709770 ą 2% stress-ng.schedpolicy.ops_per_sec
> 48.14 -2.2% 47.09 stress-ng.time.elapsed_time
> 48.14 -2.2% 47.09 stress-ng.time.elapsed_time.max
> 5480 +3.7% 5681 stress-ng.time.percent_of_cpu_this_job_got
> 2249 +1.3% 2278 stress-ng.time.system_time
> 902759 ą 4% -22.6% 698616 ą 3% proc-vmstat.unevictable_pgs_culled
> 98767954 ą 7% +16.4% 1.15e+08 ą 7% cpuidle.C1.time
> 1181676 ą 12% -43.2% 671022 ą 37% cpuidle.C6.usage
> 2.21 ą 7% +0.4 2.62 ą 10% turbostat.C1%
> 1176838 ą 12% -43.2% 668921 ą 37% turbostat.C6
> 3961223 ą 4% +12.8% 4469620 ą 5% vmstat.memory.cache
> 439.50 ą 3% +14.7% 504.00 ą 9% vmstat.procs.r
> 0.42 ą 7% -15.6% 0.35 ą 13% sched_debug.cfs_rq:/.nr_running.stddev
> 0.00 ą 4% -18.1% 0.00 ą 16% sched_debug.cpu.next_balance.stddev
> 0.41 ą 7% -15.1% 0.35 ą 13% sched_debug.cpu.nr_running.stddev
> 9367 ą 9% -12.8% 8166 ą 2% softirqs.CPU1.SCHED
> 35143 ą 6% -12.0% 30930 ą 2% softirqs.CPU22.TIMER
> 31997 ą 4% -7.5% 29595 ą 2% softirqs.CPU27.TIMER
> 3.64 ą173% -100.0% 0.00 iostat.sda.await.max
> 3.64 ą173% -100.0% 0.00 iostat.sda.r_await.max
> 3.90 ą173% -100.0% 0.00 iostat.sdc.await.max
> 3.90 ą173% -100.0% 0.00 iostat.sdc.r_await.max
> 12991737 ą 10% +61.5% 20979642 ą 8% numa-numastat.node0.local_node
> 13073590 ą 10% +61.1% 21059448 ą 8% numa-numastat.node0.numa_hit
> 20903562 ą 3% -32.2% 14164789 ą 3% numa-numastat.node1.local_node
> 20993788 ą 3% -32.1% 14245636 ą 3% numa-numastat.node1.numa_hit
> 90229 ą 4% -10.4% 80843 ą 9% numa-numastat.node1.other_node
> 50.75 ą 90% +1732.0% 929.75 ą147% interrupts.CPU23.IWI:IRQ_work_interrupts
> 40391 ą 59% -57.0% 17359 ą 11% interrupts.CPU24.RES:Rescheduling_interrupts
> 65670 ą 11% -48.7% 33716 ą 54% interrupts.CPU42.RES:Rescheduling_interrupts
> 42201 ą 46% -57.1% 18121 ą 35% interrupts.CPU49.RES:Rescheduling_interrupts
> 293869 ą 44% +103.5% 598082 ą 23% interrupts.CPU52.LOC:Local_timer_interrupts
> 17367 ą 8% +120.5% 38299 ą 44% interrupts.CPU55.RES:Rescheduling_interrupts
> 1.127e+08 +3.8% 1.17e+08 ą 2% perf-stat.i.branch-misses
> 11.10 +1.2 12.26 ą 6% perf-stat.i.cache-miss-rate%
> 4.833e+10 ą 3% +4.7% 5.06e+10 perf-stat.i.instructions
> 15009442 ą 4% +14.3% 17150138 ą 3% perf-stat.i.node-load-misses
> 47.12 ą 5% +3.2 50.37 ą 5% perf-stat.i.node-store-miss-rate%
> 6016833 ą 7% +17.0% 7036803 ą 3% perf-stat.i.node-store-misses
> 1.044e+10 ą 2% +4.0% 1.086e+10 perf-stat.ps.branch-instructions
> 1.364e+10 ą 3% +4.0% 1.418e+10 perf-stat.ps.dTLB-loads
> 4.804e+10 ą 2% +4.1% 5.003e+10 perf-stat.ps.instructions
> 14785608 ą 5% +11.3% 16451530 ą 3% perf-stat.ps.node-load-misses
> 5968712 ą 7% +13.4% 6769847 ą 3% perf-stat.ps.node-store-misses
> 13588 ą 4% +29.4% 17585 ą 9% slabinfo.Acpi-State.active_objs
> 13588 ą 4% +29.4% 17585 ą 9% slabinfo.Acpi-State.num_objs
> 20859 ą 3% -8.6% 19060 ą 4% slabinfo.kmalloc-192.num_objs
> 488.00 ą 25% +41.0% 688.00 ą 5% slabinfo.kmalloc-rcl-128.active_objs
> 488.00 ą 25% +41.0% 688.00 ą 5% slabinfo.kmalloc-rcl-128.num_objs
> 39660 ą 3% +11.8% 44348 ą 2% slabinfo.radix_tree_node.active_objs
> 44284 ą 3% +12.3% 49720 slabinfo.radix_tree_node.num_objs
> 5811 ą 15% +16.1% 6746 ą 14% slabinfo.sighand_cache.active_objs
> 402.00 ą 15% +17.5% 472.50 ą 14% slabinfo.sighand_cache.active_slabs
> 6035 ą 15% +17.5% 7091 ą 14% slabinfo.sighand_cache.num_objs
> 402.00 ą 15% +17.5% 472.50 ą 14% slabinfo.sighand_cache.num_slabs
> 10282 ą 10% +12.9% 11604 ą 9% slabinfo.signal_cache.active_objs
> 11350 ą 10% +12.8% 12808 ą 9% slabinfo.signal_cache.num_objs
> 732920 ą 9% +162.0% 1919987 ą 11% numa-meminfo.node0.Active
> 732868 ą 9% +162.0% 1919814 ą 11% numa-meminfo.node0.Active(anon)
> 545019 ą 6% +61.0% 877443 ą 17% numa-meminfo.node0.AnonHugePages
> 695015 ą 10% +46.8% 1020150 ą 14% numa-meminfo.node0.AnonPages
> 638322 ą 4% +448.2% 3499399 ą 5% numa-meminfo.node0.FilePages
> 81008 ą 14% +2443.4% 2060329 ą 3% numa-meminfo.node0.Inactive
> 80866 ą 14% +2447.4% 2060022 ą 3% numa-meminfo.node0.Inactive(anon)
> 86504 ą 10% +2287.3% 2065084 ą 3% numa-meminfo.node0.Mapped
> 2010104 +160.8% 5242366 ą 5% numa-meminfo.node0.MemUsed
> 16453 ą 15% +159.2% 42640 numa-meminfo.node0.PageTables
> 112769 ą 13% +2521.1% 2955821 ą 7% numa-meminfo.node0.Shmem
> 1839527 ą 4% -60.2% 732645 ą 23% numa-meminfo.node1.Active
> 1839399 ą 4% -60.2% 732637 ą 23% numa-meminfo.node1.Active(anon)
> 982237 ą 7% -45.9% 531445 ą 27% numa-meminfo.node1.AnonHugePages
> 1149348 ą 8% -41.2% 676067 ą 25% numa-meminfo.node1.AnonPages
> 3170649 ą 4% -77.2% 723230 ą 7% numa-meminfo.node1.FilePages
> 1960718 ą 4% -91.8% 160773 ą 31% numa-meminfo.node1.Inactive
> 1960515 ą 4% -91.8% 160722 ą 31% numa-meminfo.node1.Inactive(anon)
> 118489 ą 11% -20.2% 94603 ą 3% numa-meminfo.node1.KReclaimable
> 1966065 ą 4% -91.5% 166789 ą 29% numa-meminfo.node1.Mapped
> 5034310 ą 3% -60.2% 2003121 ą 9% numa-meminfo.node1.MemUsed
> 42684 ą 10% -64.2% 15283 ą 21% numa-meminfo.node1.PageTables
> 118489 ą 11% -20.2% 94603 ą 3% numa-meminfo.node1.SReclaimable
> 2644708 ą 5% -91.9% 214268 ą 24% numa-meminfo.node1.Shmem
> 147513 ą 20% +244.2% 507737 ą 7% numa-vmstat.node0.nr_active_anon
> 137512 ą 21% +105.8% 282999 ą 3% numa-vmstat.node0.nr_anon_pages
> 210.25 ą 33% +124.7% 472.50 ą 11% numa-vmstat.node0.nr_anon_transparent_hugepages
> 158008 ą 4% +454.7% 876519 ą 6% numa-vmstat.node0.nr_file_pages
> 18416 ą 27% +2711.4% 517747 ą 3% numa-vmstat.node0.nr_inactive_anon
> 26255 ą 22% +34.3% 35251 ą 10% numa-vmstat.node0.nr_kernel_stack
> 19893 ą 23% +2509.5% 519129 ą 3% numa-vmstat.node0.nr_mapped
> 3928 ą 22% +179.4% 10976 ą 4% numa-vmstat.node0.nr_page_table_pages
> 26623 ą 18% +2681.9% 740635 ą 7% numa-vmstat.node0.nr_shmem
> 147520 ą 20% +244.3% 507885 ą 7% numa-vmstat.node0.nr_zone_active_anon
> 18415 ą 27% +2711.5% 517739 ą 3% numa-vmstat.node0.nr_zone_inactive_anon
> 6937137 ą 8% +55.9% 10814957 ą 7% numa-vmstat.node0.numa_hit
> 6860210 ą 8% +56.6% 10739902 ą 7% numa-vmstat.node0.numa_local
> 425559 ą 13% -52.9% 200300 ą 17% numa-vmstat.node1.nr_active_anon
> 786341 ą 4% -76.6% 183664 ą 7% numa-vmstat.node1.nr_file_pages
> 483646 ą 4% -90.8% 44606 ą 29% numa-vmstat.node1.nr_inactive_anon
> 485120 ą 4% -90.5% 46130 ą 27% numa-vmstat.node1.nr_mapped
> 10471 ą 6% -61.3% 4048 ą 18% numa-vmstat.node1.nr_page_table_pages
> 654852 ą 5% -91.4% 56439 ą 25% numa-vmstat.node1.nr_shmem
> 29681 ą 11% -20.3% 23669 ą 3% numa-vmstat.node1.nr_slab_reclaimable
> 425556 ą 13% -52.9% 200359 ą 17% numa-vmstat.node1.nr_zone_active_anon
> 483649 ą 4% -90.8% 44600 ą 29% numa-vmstat.node1.nr_zone_inactive_anon
> 10527487 ą 5% -31.3% 7233899 ą 6% numa-vmstat.node1.numa_hit
> 10290625 ą 5% -31.9% 7006050 ą 7% numa-vmstat.node1.numa_local
>
>
>
> ***************************************************************************************************
> lkp-csl-2sp5: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 192G memory
> =========================================================================================
> class/compiler/cpufreq_governor/disk/kconfig/nr_threads/rootfs/tbox_group/testcase/testtime/ucode:
> interrupt/gcc-7/performance/1HDD/x86_64-fedora-25/100%/debian-x86_64-2019-11-14.cgz/lkp-csl-2sp5/stress-ng/1s/0x500002c
>
> commit:
> fcf0553db6 ("sched/fair: Remove meaningless imbalance calculation")
> 0b0695f2b3 ("sched/fair: Rework load_balance()")
>
> fcf0553db6f4c793 0b0695f2b34a4afa3f6e9aa1ff0
> ---------------- ---------------------------
> %stddev %change %stddev
> \ | \
> 6684836 -33.3% 4457559 ą 4% stress-ng.schedpolicy.ops
> 6684766 -33.3% 4457633 ą 4% stress-ng.schedpolicy.ops_per_sec
> 19978129 -28.8% 14231813 ą 16% stress-ng.time.involuntary_context_switches
> 82.49 ą 2% -5.2% 78.23 stress-ng.time.user_time
> 106716 ą 29% +40.3% 149697 ą 2% meminfo.max_used_kB
> 4.07 ą 22% +1.2 5.23 ą 5% mpstat.cpu.all.irq%
> 2721317 ą 10% +66.5% 4531100 ą 22% cpuidle.POLL.time
> 71470 ą 18% +41.1% 100822 ą 11% cpuidle.POLL.usage
> 841.00 ą 41% -50.4% 417.25 ą 17% numa-meminfo.node0.Dirty
> 7096 ą 7% +25.8% 8930 ą 9% numa-meminfo.node1.KernelStack
> 68752 ą 90% -45.9% 37169 ą143% sched_debug.cfs_rq:/.runnable_weight.stddev
> 654.93 ą 11% +19.3% 781.09 ą 2% sched_debug.cpu.clock_task.stddev
> 183.06 ą 83% -76.9% 42.20 ą 17% iostat.sda.await.max
> 627.47 ą102% -96.7% 20.52 ą 38% iostat.sda.r_await.max
> 183.08 ą 83% -76.9% 42.24 ą 17% iostat.sda.w_await.max
> 209.00 ą 41% -50.2% 104.00 ą 17% numa-vmstat.node0.nr_dirty
> 209.50 ą 41% -50.4% 104.00 ą 17% numa-vmstat.node0.nr_zone_write_pending
> 6792 ą 8% +34.4% 9131 ą 7% numa-vmstat.node1.nr_kernel_stack
> 3.57 ą173% +9.8 13.38 ą 25% perf-profile.calltrace.cycles-pp.proc_reg_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 3.57 ą173% +9.8 13.38 ą 25% perf-profile.calltrace.cycles-pp.seq_read.proc_reg_read.vfs_read.ksys_read.do_syscall_64
> 3.57 ą173% +9.8 13.39 ą 25% perf-profile.children.cycles-pp.proc_reg_read
> 3.57 ą173% +12.6 16.16 ą 28% perf-profile.children.cycles-pp.seq_read
> 7948 ą 56% -53.1% 3730 ą 5% softirqs.CPU25.RCU
> 6701 ą 33% -46.7% 3570 ą 5% softirqs.CPU34.RCU
> 8232 ą 89% -60.5% 3247 softirqs.CPU50.RCU
> 326269 ą 16% -27.4% 236940 softirqs.RCU
> 68066 +7.9% 73438 proc-vmstat.nr_active_anon
> 67504 +7.8% 72783 proc-vmstat.nr_anon_pages
> 7198 ą 19% +34.2% 9658 ą 2% proc-vmstat.nr_page_table_pages
> 40664 ą 8% +10.1% 44766 proc-vmstat.nr_slab_unreclaimable
> 68066 +7.9% 73438 proc-vmstat.nr_zone_active_anon
> 1980169 ą 4% -5.3% 1875307 proc-vmstat.numa_hit
> 1960247 ą 4% -5.4% 1855033 proc-vmstat.numa_local
> 956008 ą 16% -17.8% 786247 proc-vmstat.pgfault
> 26598 ą 76% +301.2% 106716 ą 45% interrupts.CPU1.RES:Rescheduling_interrupts
> 151212 ą 39% -67.3% 49451 ą 57% interrupts.CPU26.RES:Rescheduling_interrupts
> 1013586 ą 2% -10.9% 903528 ą 7% interrupts.CPU27.LOC:Local_timer_interrupts
> 1000980 ą 2% -11.4% 886740 ą 8% interrupts.CPU31.LOC:Local_timer_interrupts
> 1021043 ą 3% -9.9% 919686 ą 6% interrupts.CPU32.LOC:Local_timer_interrupts
> 125222 ą 51% -86.0% 17483 ą106% interrupts.CPU33.RES:Rescheduling_interrupts
> 1003735 ą 2% -11.1% 891833 ą 8% interrupts.CPU34.LOC:Local_timer_interrupts
> 1021799 ą 2% -13.2% 886665 ą 8% interrupts.CPU38.LOC:Local_timer_interrupts
> 997788 ą 2% -13.2% 866427 ą 10% interrupts.CPU42.LOC:Local_timer_interrupts
> 1001618 -11.6% 885490 ą 9% interrupts.CPU45.LOC:Local_timer_interrupts
> 22321 ą 58% +550.3% 145153 ą 22% interrupts.CPU9.RES:Rescheduling_interrupts
> 3151 ą 53% +67.3% 5273 ą 8% slabinfo.avc_xperms_data.active_objs
> 3151 ą 53% +67.3% 5273 ą 8% slabinfo.avc_xperms_data.num_objs
> 348.75 ą 13% +39.8% 487.50 ą 5% slabinfo.biovec-128.active_objs
> 348.75 ą 13% +39.8% 487.50 ą 5% slabinfo.biovec-128.num_objs
> 13422 ą 97% +121.1% 29678 ą 2% slabinfo.btrfs_extent_map.active_objs
> 14638 ą 98% +117.8% 31888 ą 2% slabinfo.btrfs_extent_map.num_objs
> 3835 ą 18% +40.9% 5404 ą 7% slabinfo.dmaengine-unmap-16.active_objs
> 3924 ą 18% +39.9% 5490 ą 8% slabinfo.dmaengine-unmap-16.num_objs
> 3482 ą 96% +119.1% 7631 ą 10% slabinfo.khugepaged_mm_slot.active_objs
> 3573 ą 96% +119.4% 7839 ą 10% slabinfo.khugepaged_mm_slot.num_objs
> 8629 ą 52% -49.2% 4384 slabinfo.kmalloc-rcl-64.active_objs
> 8629 ą 52% -49.2% 4384 slabinfo.kmalloc-rcl-64.num_objs
> 2309 ą 57% +82.1% 4206 ą 5% slabinfo.mnt_cache.active_objs
> 2336 ą 57% +80.8% 4224 ą 5% slabinfo.mnt_cache.num_objs
> 5320 ą 48% +69.1% 8999 ą 23% slabinfo.pool_workqueue.active_objs
> 165.75 ą 48% +69.4% 280.75 ą 23% slabinfo.pool_workqueue.active_slabs
> 5320 ą 48% +69.2% 8999 ą 23% slabinfo.pool_workqueue.num_objs
> 165.75 ą 48% +69.4% 280.75 ą 23% slabinfo.pool_workqueue.num_slabs
> 3306 ą 15% +27.0% 4199 ą 3% slabinfo.task_group.active_objs
> 3333 ą 16% +30.1% 4336 ą 3% slabinfo.task_group.num_objs
> 14.74 ą 2% +1.8 16.53 ą 2% perf-stat.i.cache-miss-rate%
> 22459727 ą 20% +46.7% 32955572 ą 4% perf-stat.i.cache-misses
> 33575 ą 19% +68.8% 56658 ą 13% perf-stat.i.cpu-migrations
> 0.03 ą 20% +0.0 0.05 ą 8% perf-stat.i.dTLB-load-miss-rate%
> 6351703 ą 33% +47.2% 9352532 ą 9% perf-stat.i.dTLB-load-misses
> 0.45 ą 3% -3.0% 0.44 perf-stat.i.ipc
> 4711345 ą 18% +43.9% 6780944 ą 7% perf-stat.i.node-load-misses
> 82.51 +4.5 86.97 perf-stat.i.node-store-miss-rate%
> 2861142 ą 31% +60.8% 4601146 ą 5% perf-stat.i.node-store-misses
> 0.92 ą 6% -0.1 0.85 ą 2% perf-stat.overall.branch-miss-rate%
> 0.02 ą 3% +0.0 0.02 ą 4% perf-stat.overall.dTLB-store-miss-rate%
> 715.05 ą 5% +9.9% 785.50 ą 4% perf-stat.overall.instructions-per-iTLB-miss
> 0.44 ą 2% -5.4% 0.42 ą 2% perf-stat.overall.ipc
> 79.67 +2.1 81.80 ą 2% perf-stat.overall.node-store-miss-rate%
> 22237897 ą 19% +46.4% 32560557 ą 5% perf-stat.ps.cache-misses
> 32491 ą 18% +70.5% 55390 ą 13% perf-stat.ps.cpu-migrations
> 6071108 ą 31% +45.0% 8804767 ą 9% perf-stat.ps.dTLB-load-misses
> 1866 ą 98% -91.9% 150.48 ą 2% perf-stat.ps.major-faults
> 4593546 ą 16% +42.4% 6541402 ą 7% perf-stat.ps.node-load-misses
> 2757176 ą 29% +58.4% 4368169 ą 5% perf-stat.ps.node-store-misses
> 1.303e+12 ą 3% -9.8% 1.175e+12 ą 3% perf-stat.total.instructions
>
>
>
> ***************************************************************************************************
> lkp-bdw-ep6: 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory
> =========================================================================================
> class/compiler/cpufreq_governor/disk/kconfig/nr_threads/rootfs/tbox_group/testcase/testtime/ucode:
> interrupt/gcc-7/performance/1HDD/x86_64-rhel-7.6/100%/debian-x86_64-2019-11-14.cgz/lkp-bdw-ep6/stress-ng/30s/0xb000038
>
> commit:
> fcf0553db6 ("sched/fair: Remove meaningless imbalance calculation")
> 0b0695f2b3 ("sched/fair: Rework load_balance()")
>
> fcf0553db6f4c793 0b0695f2b34a4afa3f6e9aa1ff0
> ---------------- ---------------------------
> fail:runs %reproduction fail:runs
> | | |
> 1:4 -25% :4 dmesg.WARNING:at#for_ip_interrupt_entry/0x
> %stddev %change %stddev
> \ | \
> 98245522 +42.3% 1.398e+08 stress-ng.schedpolicy.ops
> 3274860 +42.3% 4661027 stress-ng.schedpolicy.ops_per_sec
> 3.473e+08 -9.7% 3.137e+08 stress-ng.sigq.ops
> 11576537 -9.7% 10454846 stress-ng.sigq.ops_per_sec
> 38097605 ą 6% +10.3% 42011440 ą 4% stress-ng.sigrt.ops
> 1269646 ą 6% +10.3% 1400024 ą 4% stress-ng.sigrt.ops_per_sec
> 3.628e+08 ą 4% -21.5% 2.848e+08 ą 10% stress-ng.time.involuntary_context_switches
> 7040 +2.9% 7245 stress-ng.time.percent_of_cpu_this_job_got
> 15.09 ą 3% -13.4% 13.07 ą 5% iostat.cpu.idle
> 14.82 ą 3% -2.0 12.80 ą 5% mpstat.cpu.all.idle%
> 3.333e+08 ą 17% +59.9% 5.331e+08 ą 22% cpuidle.C1.time
> 5985148 ą 23% +112.5% 12719679 ą 20% cpuidle.C1E.usage
> 14.50 ą 3% -12.1% 12.75 ą 6% vmstat.cpu.id
> 1113131 ą 2% -10.5% 996285 ą 3% vmstat.system.cs
> 2269 +2.4% 2324 turbostat.Avg_MHz
> 0.64 ą 17% +0.4 1.02 ą 23% turbostat.C1%
> 5984799 ą 23% +112.5% 12719086 ą 20% turbostat.C1E
> 4.17 ą 32% -46.0% 2.25 ą 38% turbostat.Pkg%pc2
> 216.57 +2.1% 221.12 turbostat.PkgWatt
> 13.33 ą 3% +3.9% 13.84 turbostat.RAMWatt
> 99920 +13.6% 113486 ą 15% proc-vmstat.nr_active_anon
> 5738 +1.2% 5806 proc-vmstat.nr_inactive_anon
> 46788 +2.1% 47749 proc-vmstat.nr_slab_unreclaimable
> 99920 +13.6% 113486 ą 15% proc-vmstat.nr_zone_active_anon
> 5738 +1.2% 5806 proc-vmstat.nr_zone_inactive_anon
> 3150 ą 2% +35.4% 4265 ą 33% proc-vmstat.numa_huge_pte_updates
> 1641223 +34.3% 2203844 ą 32% proc-vmstat.numa_pte_updates
> 13575 ą 18% +62.1% 21999 ą 4% slabinfo.ext4_extent_status.active_objs
> 13954 ą 17% +57.7% 21999 ą 4% slabinfo.ext4_extent_status.num_objs
> 2527 ą 4% +9.8% 2774 ą 2% slabinfo.khugepaged_mm_slot.active_objs
> 2527 ą 4% +9.8% 2774 ą 2% slabinfo.khugepaged_mm_slot.num_objs
> 57547 ą 8% -15.3% 48743 ą 9% slabinfo.kmalloc-rcl-64.active_objs
> 898.75 ą 8% -15.3% 761.00 ą 9% slabinfo.kmalloc-rcl-64.active_slabs
> 57547 ą 8% -15.3% 48743 ą 9% slabinfo.kmalloc-rcl-64.num_objs
> 898.75 ą 8% -15.3% 761.00 ą 9% slabinfo.kmalloc-rcl-64.num_slabs
> 1.014e+10 +1.7% 1.031e+10 perf-stat.i.branch-instructions
> 13.37 ą 4% +2.0 15.33 ą 3% perf-stat.i.cache-miss-rate%
> 1.965e+11 +2.6% 2.015e+11 perf-stat.i.cpu-cycles
> 20057708 ą 4% +13.9% 22841468 ą 4% perf-stat.i.iTLB-loads
> 4.973e+10 +1.4% 5.042e+10 perf-stat.i.instructions
> 3272 ą 2% +2.9% 3366 perf-stat.i.minor-faults
> 4500892 ą 3% +18.9% 5351518 ą 6% perf-stat.i.node-store-misses
> 3.91 +1.3% 3.96 perf-stat.overall.cpi
> 69.62 -1.5 68.11 perf-stat.overall.iTLB-load-miss-rate%
> 1.047e+10 +1.3% 1.061e+10 perf-stat.ps.branch-instructions
> 1117454 ą 2% -10.6% 999467 ą 3% perf-stat.ps.context-switches
> 1.986e+11 +2.4% 2.033e+11 perf-stat.ps.cpu-cycles
> 19614413 ą 4% +13.6% 22288555 ą 4% perf-stat.ps.iTLB-loads
> 3493 -1.1% 3453 perf-stat.ps.minor-faults
> 4546636 ą 3% +17.0% 5321658 ą 5% perf-stat.ps.node-store-misses
> 0.64 ą 3% -0.2 0.44 ą 57% perf-profile.calltrace.cycles-pp.common_timer_get.do_timer_gettime.__x64_sys_timer_gettime.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 0.66 ą 3% -0.1 0.58 ą 7% perf-profile.children.cycles-pp.common_timer_get
> 0.44 ą 4% -0.1 0.39 ą 5% perf-profile.children.cycles-pp.posix_ktime_get_ts
> 0.39 ą 5% -0.0 0.34 ą 6% perf-profile.children.cycles-pp.ktime_get_ts64
> 0.07 ą 17% +0.0 0.10 ą 8% perf-profile.children.cycles-pp.task_tick_fair
> 0.08 ą 15% +0.0 0.11 ą 7% perf-profile.children.cycles-pp.scheduler_tick
> 0.46 ą 5% +0.1 0.54 ą 6% perf-profile.children.cycles-pp.__might_sleep
> 0.69 ą 8% +0.2 0.85 ą 12% perf-profile.children.cycles-pp.___might_sleep
> 0.90 ą 5% -0.2 0.73 ą 9% perf-profile.self.cycles-pp.__might_fault
> 0.40 ą 6% -0.1 0.33 ą 9% perf-profile.self.cycles-pp.do_timer_gettime
> 0.50 ą 4% -0.1 0.45 ą 7% perf-profile.self.cycles-pp.put_itimerspec64
> 0.32 ą 2% -0.0 0.27 ą 9% perf-profile.self.cycles-pp.update_curr_fair
> 0.20 ą 6% -0.0 0.18 ą 2% perf-profile.self.cycles-pp.ktime_get_ts64
> 0.08 ą 23% +0.0 0.12 ą 8% perf-profile.self.cycles-pp._raw_spin_trylock
> 0.42 ą 5% +0.1 0.50 ą 6% perf-profile.self.cycles-pp.__might_sleep
> 0.66 ą 9% +0.2 0.82 ą 12% perf-profile.self.cycles-pp.___might_sleep
> 47297 ą 13% +19.7% 56608 ą 5% softirqs.CPU13.SCHED
> 47070 ą 3% +20.5% 56735 ą 7% softirqs.CPU2.SCHED
> 55443 ą 9% -20.2% 44250 ą 2% softirqs.CPU28.SCHED
> 56633 ą 3% -12.6% 49520 ą 7% softirqs.CPU34.SCHED
> 56599 ą 11% -18.0% 46384 ą 2% softirqs.CPU36.SCHED
> 56909 ą 9% -18.4% 46438 ą 6% softirqs.CPU40.SCHED
> 45062 ą 9% +28.1% 57709 ą 9% softirqs.CPU45.SCHED
> 43959 +28.7% 56593 ą 9% softirqs.CPU49.SCHED
> 46235 ą 10% +22.2% 56506 ą 11% softirqs.CPU5.SCHED
> 44779 ą 12% +22.5% 54859 ą 11% softirqs.CPU57.SCHED
> 46739 ą 10% +21.1% 56579 ą 8% softirqs.CPU6.SCHED
> 53129 ą 4% -13.1% 46149 ą 8% softirqs.CPU70.SCHED
> 55822 ą 7% -20.5% 44389 ą 8% softirqs.CPU73.SCHED
> 56011 ą 5% -11.4% 49610 ą 7% softirqs.CPU77.SCHED
> 55263 ą 9% -13.2% 47942 ą 12% softirqs.CPU78.SCHED
> 58792 ą 14% -21.3% 46291 ą 9% softirqs.CPU81.SCHED
> 53341 ą 7% -13.7% 46041 ą 10% softirqs.CPU83.SCHED
> 59096 ą 15% -23.9% 44998 ą 6% softirqs.CPU85.SCHED
> 36647 -98.5% 543.00 ą 61% numa-meminfo.node0.Active(file)
> 620922 ą 4% -10.4% 556566 ą 5% numa-meminfo.node0.FilePages
> 21243 ą 3% -36.2% 13543 ą 41% numa-meminfo.node0.Inactive
> 20802 ą 3% -35.3% 13455 ą 42% numa-meminfo.node0.Inactive(anon)
> 15374 ą 9% -27.2% 11193 ą 8% numa-meminfo.node0.KernelStack
> 21573 -34.7% 14084 ą 14% numa-meminfo.node0.Mapped
> 1136795 ą 5% -12.4% 995965 ą 6% numa-meminfo.node0.MemUsed
> 16420 ą 6% -66.0% 5580 ą 18% numa-meminfo.node0.PageTables
> 108182 ą 2% -18.5% 88150 ą 3% numa-meminfo.node0.SUnreclaim
> 166467 ą 2% -15.8% 140184 ą 4% numa-meminfo.node0.Slab
> 181705 ą 36% +63.8% 297623 ą 10% numa-meminfo.node1.Active
> 320.75 ą 27% +11187.0% 36203 numa-meminfo.node1.Active(file)
> 2208 ą 38% +362.1% 10207 ą 54% numa-meminfo.node1.Inactive
> 2150 ą 39% +356.0% 9804 ą 58% numa-meminfo.node1.Inactive(anon)
> 41819 ą 10% +17.3% 49068 ą 6% numa-meminfo.node1.KReclaimable
> 11711 ą 5% +47.2% 17238 ą 22% numa-meminfo.node1.KernelStack
> 10642 +68.3% 17911 ą 11% numa-meminfo.node1.Mapped
> 952520 ą 6% +20.3% 1146337 ą 3% numa-meminfo.node1.MemUsed
> 12342 ą 15% +92.4% 23741 ą 9% numa-meminfo.node1.PageTables
> 41819 ą 10% +17.3% 49068 ą 6% numa-meminfo.node1.SReclaimable
> 80394 ą 3% +27.1% 102206 ą 3% numa-meminfo.node1.SUnreclaim
> 122214 ą 3% +23.8% 151275 ą 3% numa-meminfo.node1.Slab
> 9160 -98.5% 135.25 ą 61% numa-vmstat.node0.nr_active_file
> 155223 ą 4% -10.4% 139122 ą 5% numa-vmstat.node0.nr_file_pages
> 5202 ą 3% -35.4% 3362 ą 42% numa-vmstat.node0.nr_inactive_anon
> 109.50 ą 14% -80.1% 21.75 ą160% numa-vmstat.node0.nr_inactive_file
> 14757 ą 3% -34.4% 9676 ą 12% numa-vmstat.node0.nr_kernel_stack
> 5455 -34.9% 3549 ą 12% numa-vmstat.node0.nr_mapped
> 4069 ą 6% -68.3% 1289 ą 24% numa-vmstat.node0.nr_page_table_pages
> 26943 ą 2% -19.2% 21761 ą 3% numa-vmstat.node0.nr_slab_unreclaimable
> 2240 ą 6% -97.8% 49.00 ą 69% numa-vmstat.node0.nr_written
> 9160 -98.5% 135.25 ą 61% numa-vmstat.node0.nr_zone_active_file
> 5202 ą 3% -35.4% 3362 ą 42% numa-vmstat.node0.nr_zone_inactive_anon
> 109.50 ą 14% -80.1% 21.75 ą160% numa-vmstat.node0.nr_zone_inactive_file
> 79.75 ą 28% +11247.0% 9049 numa-vmstat.node1.nr_active_file
> 542.25 ą 41% +352.1% 2451 ą 58% numa-vmstat.node1.nr_inactive_anon
> 14.00 ą140% +617.9% 100.50 ą 35% numa-vmstat.node1.nr_inactive_file
> 11182 ą 4% +28.9% 14415 ą 4% numa-vmstat.node1.nr_kernel_stack
> 2728 ą 3% +67.7% 4576 ą 9% numa-vmstat.node1.nr_mapped
> 3056 ą 15% +88.2% 5754 ą 8% numa-vmstat.node1.nr_page_table_pages
> 10454 ą 10% +17.3% 12262 ą 7% numa-vmstat.node1.nr_slab_reclaimable
> 20006 ą 3% +25.0% 25016 ą 3% numa-vmstat.node1.nr_slab_unreclaimable
> 19.00 ą 52% +11859.2% 2272 ą 2% numa-vmstat.node1.nr_written
> 79.75 ą 28% +11247.0% 9049 numa-vmstat.node1.nr_zone_active_file
> 542.25 ą 41% +352.1% 2451 ą 58% numa-vmstat.node1.nr_zone_inactive_anon
> 14.00 ą140% +617.9% 100.50 ą 35% numa-vmstat.node1.nr_zone_inactive_file
> 173580 ą 21% +349.5% 780280 ą 7% sched_debug.cfs_rq:/.MIN_vruntime.avg
> 6891819 ą 37% +109.1% 14412817 ą 9% sched_debug.cfs_rq:/.MIN_vruntime.max
> 1031500 ą 25% +189.1% 2982452 ą 8% sched_debug.cfs_rq:/.MIN_vruntime.stddev
> 149079 +13.6% 169354 ą 2% sched_debug.cfs_rq:/.exec_clock.min
> 8550 ą 3% -59.7% 3442 ą 32% sched_debug.cfs_rq:/.exec_clock.stddev
> 4.95 ą 6% -15.2% 4.20 ą 10% sched_debug.cfs_rq:/.load_avg.min
> 173580 ą 21% +349.5% 780280 ą 7% sched_debug.cfs_rq:/.max_vruntime.avg
> 6891819 ą 37% +109.1% 14412817 ą 9% sched_debug.cfs_rq:/.max_vruntime.max
> 1031500 ą 25% +189.1% 2982452 ą 8% sched_debug.cfs_rq:/.max_vruntime.stddev
> 16144141 +27.9% 20645199 ą 6% sched_debug.cfs_rq:/.min_vruntime.avg
> 17660392 +27.7% 22546402 ą 4% sched_debug.cfs_rq:/.min_vruntime.max
> 13747718 +36.8% 18802595 ą 5% sched_debug.cfs_rq:/.min_vruntime.min
> 0.17 ą 11% +35.0% 0.22 ą 15% sched_debug.cfs_rq:/.nr_running.stddev
> 10.64 ą 14% -26.4% 7.83 ą 12% sched_debug.cpu.clock.stddev
> 10.64 ą 14% -26.4% 7.83 ą 12% sched_debug.cpu.clock_task.stddev
> 7093 ą 42% -65.9% 2420 ą120% sched_debug.cpu.curr->pid.min
> 2434979 ą 2% -18.6% 1981697 ą 3% sched_debug.cpu.nr_switches.avg
> 3993189 ą 6% -22.2% 3104832 ą 5% sched_debug.cpu.nr_switches.max
> -145.03 -42.8% -82.90 sched_debug.cpu.nr_uninterruptible.min
> 2097122 ą 6% +38.7% 2908923 ą 6% sched_debug.cpu.sched_count.min
> 809684 ą 13% -30.5% 562929 ą 17% sched_debug.cpu.sched_count.stddev
> 307565 ą 4% -15.1% 261231 ą 3% sched_debug.cpu.ttwu_count.min
> 207286 ą 6% -16.4% 173387 ą 3% sched_debug.cpu.ttwu_local.min
> 125963 ą 23% +53.1% 192849 ą 2% sched_debug.cpu.ttwu_local.stddev
> 2527246 +10.8% 2800959 ą 3% sched_debug.cpu.yld_count.avg
> 1294266 ą 4% +53.7% 1989264 ą 2% sched_debug.cpu.yld_count.min
> 621332 ą 9% -38.4% 382813 ą 22% sched_debug.cpu.yld_count.stddev
> 899.50 ą 28% -48.2% 465.75 ą 42% interrupts.37:IR-PCI-MSI.1572868-edge.eth0-TxRx-3
> 372.50 ą 7% +169.5% 1004 ą 40% interrupts.40:IR-PCI-MSI.1572871-edge.eth0-TxRx-6
> 6201 ą 8% +17.9% 7309 ą 3% interrupts.CPU0.CAL:Function_call_interrupts
> 653368 ą 47% +159.4% 1695029 ą 17% interrupts.CPU0.RES:Rescheduling_interrupts
> 7104 ą 7% +13.6% 8067 interrupts.CPU1.CAL:Function_call_interrupts
> 2094 ą 59% +89.1% 3962 ą 10% interrupts.CPU10.TLB:TLB_shootdowns
> 7309 ą 8% +11.2% 8125 interrupts.CPU11.CAL:Function_call_interrupts
> 2089 ą 62% +86.2% 3890 ą 11% interrupts.CPU13.TLB:TLB_shootdowns
> 7068 ą 8% +15.2% 8144 ą 2% interrupts.CPU14.CAL:Function_call_interrupts
> 7112 ą 7% +13.6% 8079 ą 3% interrupts.CPU15.CAL:Function_call_interrupts
> 1950 ą 61% +103.5% 3968 ą 11% interrupts.CPU15.TLB:TLB_shootdowns
> 899.50 ą 28% -48.2% 465.75 ą 42% interrupts.CPU16.37:IR-PCI-MSI.1572868-edge.eth0-TxRx-3
> 2252 ą 47% +62.6% 3664 ą 15% interrupts.CPU16.TLB:TLB_shootdowns
> 7111 ą 8% +14.8% 8167 ą 3% interrupts.CPU18.CAL:Function_call_interrupts
> 1972 ą 60% +96.3% 3872 ą 9% interrupts.CPU18.TLB:TLB_shootdowns
> 372.50 ą 7% +169.5% 1004 ą 40% interrupts.CPU19.40:IR-PCI-MSI.1572871-edge.eth0-TxRx-6
> 2942 ą 12% -57.5% 1251 ą 22% interrupts.CPU22.TLB:TLB_shootdowns
> 7819 -12.2% 6861 ą 3% interrupts.CPU23.CAL:Function_call_interrupts
> 3327 ą 12% -62.7% 1241 ą 29% interrupts.CPU23.TLB:TLB_shootdowns
> 7767 ą 3% -14.0% 6683 ą 5% interrupts.CPU24.CAL:Function_call_interrupts
> 3185 ą 21% -63.8% 1154 ą 14% interrupts.CPU24.TLB:TLB_shootdowns
> 7679 ą 4% -11.3% 6812 ą 2% interrupts.CPU25.CAL:Function_call_interrupts
> 3004 ą 28% -63.4% 1100 ą 7% interrupts.CPU25.TLB:TLB_shootdowns
> 3187 ą 17% -61.3% 1232 ą 35% interrupts.CPU26.TLB:TLB_shootdowns
> 3193 ą 16% -59.3% 1299 ą 34% interrupts.CPU27.TLB:TLB_shootdowns
> 3059 ą 21% -58.0% 1285 ą 32% interrupts.CPU28.TLB:TLB_shootdowns
> 7798 ą 4% -13.8% 6719 ą 7% interrupts.CPU29.CAL:Function_call_interrupts
> 3122 ą 20% -62.3% 1178 ą 37% interrupts.CPU29.TLB:TLB_shootdowns
> 7727 ą 2% -11.6% 6827 ą 5% interrupts.CPU30.CAL:Function_call_interrupts
> 3102 ą 18% -59.4% 1259 ą 33% interrupts.CPU30.TLB:TLB_shootdowns
> 3269 ą 24% -58.1% 1371 ą 48% interrupts.CPU31.TLB:TLB_shootdowns
> 7918 ą 3% -14.5% 6771 interrupts.CPU32.CAL:Function_call_interrupts
> 3324 ą 18% -70.7% 973.50 ą 18% interrupts.CPU32.TLB:TLB_shootdowns
> 2817 ą 27% -60.2% 1121 ą 26% interrupts.CPU33.TLB:TLB_shootdowns
> 7956 ą 3% -11.8% 7018 ą 4% interrupts.CPU34.CAL:Function_call_interrupts
> 3426 ą 21% -70.3% 1018 ą 29% interrupts.CPU34.TLB:TLB_shootdowns
> 3121 ą 17% -70.3% 926.75 ą 22% interrupts.CPU35.TLB:TLB_shootdowns
> 7596 ą 4% -10.6% 6793 ą 3% interrupts.CPU36.CAL:Function_call_interrupts
> 2900 ą 30% -62.3% 1094 ą 34% interrupts.CPU36.TLB:TLB_shootdowns
> 7863 -13.1% 6833 ą 2% interrupts.CPU37.CAL:Function_call_interrupts
> 3259 ą 15% -65.9% 1111 ą 20% interrupts.CPU37.TLB:TLB_shootdowns
> 3230 ą 26% -64.0% 1163 ą 39% interrupts.CPU38.TLB:TLB_shootdowns
> 7728 ą 5% -13.8% 6662 ą 7% interrupts.CPU39.CAL:Function_call_interrupts
> 2950 ą 29% -61.6% 1133 ą 26% interrupts.CPU39.TLB:TLB_shootdowns
> 6864 ą 3% +18.7% 8147 interrupts.CPU4.CAL:Function_call_interrupts
> 1847 ą 59% +118.7% 4039 ą 7% interrupts.CPU4.TLB:TLB_shootdowns
> 7951 ą 6% -15.0% 6760 ą 2% interrupts.CPU40.CAL:Function_call_interrupts
> 3200 ą 30% -72.3% 886.50 ą 39% interrupts.CPU40.TLB:TLB_shootdowns
> 7819 ą 6% -11.3% 6933 ą 2% interrupts.CPU41.CAL:Function_call_interrupts
> 3149 ą 28% -62.9% 1169 ą 24% interrupts.CPU41.TLB:TLB_shootdowns
> 7884 ą 4% -11.0% 7019 ą 2% interrupts.CPU42.CAL:Function_call_interrupts
> 3248 ą 16% -63.4% 1190 ą 23% interrupts.CPU42.TLB:TLB_shootdowns
> 7659 ą 5% -12.7% 6690 ą 3% interrupts.CPU43.CAL:Function_call_interrupts
> 490732 ą 20% +114.5% 1052606 ą 47% interrupts.CPU43.RES:Rescheduling_interrupts
> 1432688 ą 34% -67.4% 467217 ą 43% interrupts.CPU47.RES:Rescheduling_interrupts
> 7122 ą 8% +16.0% 8259 ą 3% interrupts.CPU48.CAL:Function_call_interrupts
> 1868 ą 65% +118.4% 4079 ą 8% interrupts.CPU48.TLB:TLB_shootdowns
> 7165 ą 8% +11.3% 7977 ą 5% interrupts.CPU49.CAL:Function_call_interrupts
> 1961 ą 59% +98.4% 3891 ą 4% interrupts.CPU49.TLB:TLB_shootdowns
> 461807 ą 47% +190.8% 1342990 ą 48% interrupts.CPU5.RES:Rescheduling_interrupts
> 7167 ą 7% +15.4% 8273 interrupts.CPU50.CAL:Function_call_interrupts
> 2027 ą 51% +103.9% 4134 ą 8% interrupts.CPU50.TLB:TLB_shootdowns
> 7163 ą 9% +16.3% 8328 interrupts.CPU51.CAL:Function_call_interrupts
> 660073 ą 33% +74.0% 1148640 ą 25% interrupts.CPU51.RES:Rescheduling_interrupts
> 2043 ą 64% +95.8% 4000 ą 5% interrupts.CPU51.TLB:TLB_shootdowns
> 7428 ą 9% +13.5% 8434 ą 2% interrupts.CPU52.CAL:Function_call_interrupts
> 2280 ą 61% +85.8% 4236 ą 9% interrupts.CPU52.TLB:TLB_shootdowns
> 7144 ą 11% +17.8% 8413 interrupts.CPU53.CAL:Function_call_interrupts
> 1967 ą 67% +104.7% 4026 ą 5% interrupts.CPU53.TLB:TLB_shootdowns
> 7264 ą 10% +15.6% 8394 ą 4% interrupts.CPU54.CAL:Function_call_interrupts
> 7045 ą 11% +18.7% 8365 ą 2% interrupts.CPU56.CAL:Function_call_interrupts
> 2109 ą 59% +91.6% 4041 ą 10% interrupts.CPU56.TLB:TLB_shootdowns
> 7307 ą 9% +15.3% 8428 ą 2% interrupts.CPU57.CAL:Function_call_interrupts
> 2078 ą 64% +96.5% 4085 ą 6% interrupts.CPU57.TLB:TLB_shootdowns
> 6834 ą 12% +19.8% 8190 ą 3% interrupts.CPU58.CAL:Function_call_interrupts
> 612496 ą 85% +122.5% 1362815 ą 27% interrupts.CPU58.RES:Rescheduling_interrupts
> 1884 ą 69% +112.0% 3995 ą 8% interrupts.CPU58.TLB:TLB_shootdowns
> 7185 ą 8% +15.9% 8329 interrupts.CPU59.CAL:Function_call_interrupts
> 1982 ą 58% +101.1% 3986 ą 5% interrupts.CPU59.TLB:TLB_shootdowns
> 7051 ą 6% +13.1% 7975 interrupts.CPU6.CAL:Function_call_interrupts
> 1831 ą 49% +102.1% 3701 ą 8% interrupts.CPU6.TLB:TLB_shootdowns
> 7356 ą 8% +16.2% 8548 interrupts.CPU60.CAL:Function_call_interrupts
> 2124 ą 57% +92.8% 4096 ą 5% interrupts.CPU60.TLB:TLB_shootdowns
> 7243 ą 9% +15.1% 8334 interrupts.CPU61.CAL:Function_call_interrupts
> 572423 ą 71% +110.0% 1201919 ą 40% interrupts.CPU61.RES:Rescheduling_interrupts
> 7295 ą 9% +14.7% 8369 interrupts.CPU63.CAL:Function_call_interrupts
> 2139 ą 57% +85.7% 3971 ą 3% interrupts.CPU63.TLB:TLB_shootdowns
> 7964 ą 2% -15.6% 6726 ą 5% interrupts.CPU66.CAL:Function_call_interrupts
> 3198 ą 21% -65.0% 1119 ą 24% interrupts.CPU66.TLB:TLB_shootdowns
> 8103 ą 2% -17.5% 6687 ą 9% interrupts.CPU67.CAL:Function_call_interrupts
> 3357 ą 18% -62.9% 1244 ą 32% interrupts.CPU67.TLB:TLB_shootdowns
> 7772 ą 2% -14.0% 6687 ą 8% interrupts.CPU68.CAL:Function_call_interrupts
> 2983 ą 17% -59.2% 1217 ą 15% interrupts.CPU68.TLB:TLB_shootdowns
> 7986 ą 4% -13.8% 6887 ą 4% interrupts.CPU69.CAL:Function_call_interrupts
> 3192 ą 24% -65.0% 1117 ą 30% interrupts.CPU69.TLB:TLB_shootdowns
> 7070 ą 6% +14.6% 8100 ą 2% interrupts.CPU7.CAL:Function_call_interrupts
> 697891 ą 32% +54.4% 1077890 ą 18% interrupts.CPU7.RES:Rescheduling_interrupts
> 1998 ą 55% +97.1% 3938 ą 10% interrupts.CPU7.TLB:TLB_shootdowns
> 8085 -13.4% 7002 ą 3% interrupts.CPU70.CAL:Function_call_interrupts
> 1064985 ą 35% -62.5% 398986 ą 29% interrupts.CPU70.RES:Rescheduling_interrupts
> 3347 ą 12% -61.7% 1280 ą 24% interrupts.CPU70.TLB:TLB_shootdowns
> 2916 ą 16% -58.8% 1201 ą 39% interrupts.CPU71.TLB:TLB_shootdowns
> 3314 ą 19% -61.3% 1281 ą 26% interrupts.CPU72.TLB:TLB_shootdowns
> 3119 ą 18% -61.5% 1200 ą 39% interrupts.CPU73.TLB:TLB_shootdowns
> 7992 ą 4% -12.6% 6984 ą 3% interrupts.CPU74.CAL:Function_call_interrupts
> 3187 ą 21% -56.8% 1378 ą 40% interrupts.CPU74.TLB:TLB_shootdowns
> 7953 ą 4% -12.0% 6999 ą 4% interrupts.CPU75.CAL:Function_call_interrupts
> 3072 ą 26% -56.8% 1327 ą 34% interrupts.CPU75.TLB:TLB_shootdowns
> 8119 ą 5% -12.4% 7109 ą 7% interrupts.CPU76.CAL:Function_call_interrupts
> 3418 ą 20% -67.5% 1111 ą 31% interrupts.CPU76.TLB:TLB_shootdowns
> 7804 ą 5% -11.4% 6916 ą 4% interrupts.CPU77.CAL:Function_call_interrupts
> 7976 ą 5% -14.4% 6826 ą 3% interrupts.CPU78.CAL:Function_call_interrupts
> 3209 ą 27% -71.8% 904.75 ą 28% interrupts.CPU78.TLB:TLB_shootdowns
> 8187 ą 4% -14.6% 6991 ą 3% interrupts.CPU79.CAL:Function_call_interrupts
> 3458 ą 20% -67.5% 1125 ą 36% interrupts.CPU79.TLB:TLB_shootdowns
> 7122 ą 7% +14.2% 8136 ą 2% interrupts.CPU8.CAL:Function_call_interrupts
> 2096 ą 63% +87.4% 3928 ą 8% interrupts.CPU8.TLB:TLB_shootdowns
> 8130 ą 5% -17.2% 6728 ą 5% interrupts.CPU81.CAL:Function_call_interrupts
> 3253 ą 24% -70.6% 955.00 ą 38% interrupts.CPU81.TLB:TLB_shootdowns
> 7940 ą 5% -13.9% 6839 ą 5% interrupts.CPU82.CAL:Function_call_interrupts
> 2952 ą 26% -66.3% 996.00 ą 51% interrupts.CPU82.TLB:TLB_shootdowns
> 7900 ą 6% -13.4% 6844 ą 3% interrupts.CPU83.CAL:Function_call_interrupts
> 3012 ą 34% -68.3% 956.00 ą 17% interrupts.CPU83.TLB:TLB_shootdowns
> 7952 ą 6% -15.8% 6695 ą 2% interrupts.CPU84.CAL:Function_call_interrupts
> 3049 ą 31% -75.5% 746.50 ą 27% interrupts.CPU84.TLB:TLB_shootdowns
> 8065 ą 6% -15.7% 6798 interrupts.CPU85.CAL:Function_call_interrupts
> 3222 ą 23% -69.7% 976.00 ą 13% interrupts.CPU85.TLB:TLB_shootdowns
> 8049 ą 5% -13.2% 6983 ą 4% interrupts.CPU86.CAL:Function_call_interrupts
> 3159 ą 19% -61.9% 1202 ą 27% interrupts.CPU86.TLB:TLB_shootdowns
> 8154 ą 8% -16.9% 6773 ą 3% interrupts.CPU87.CAL:Function_call_interrupts
> 1432962 ą 21% -48.5% 737989 ą 30% interrupts.CPU87.RES:Rescheduling_interrupts
> 3186 ą 33% -72.3% 881.75 ą 21% interrupts.CPU87.TLB:TLB_shootdowns
>
>
>
> ***************************************************************************************************
> lkp-bdw-ep6: 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory
> =========================================================================================
> class/compiler/cpufreq_governor/disk/kconfig/nr_threads/rootfs/tbox_group/testcase/testtime/ucode:
> interrupt/gcc-7/performance/1HDD/x86_64-rhel-7.6/100%/debian-x86_64-2019-11-14.cgz/lkp-bdw-ep6/stress-ng/1s/0xb000038
>
> commit:
> fcf0553db6 ("sched/fair: Remove meaningless imbalance calculation")
> 0b0695f2b3 ("sched/fair: Rework load_balance()")
>
> fcf0553db6f4c793 0b0695f2b34a4afa3f6e9aa1ff0
> ---------------- ---------------------------
> %stddev %change %stddev
> \ | \
> 3345449 +35.1% 4518187 ą 5% stress-ng.schedpolicy.ops
> 3347036 +35.1% 4520740 ą 5% stress-ng.schedpolicy.ops_per_sec
> 11464910 ą 6% -23.3% 8796455 ą 11% stress-ng.sigq.ops
> 11452565 ą 6% -23.3% 8786844 ą 11% stress-ng.sigq.ops_per_sec
> 228736 +20.7% 276087 ą 20% stress-ng.sleep.ops
> 157479 +23.0% 193722 ą 21% stress-ng.sleep.ops_per_sec
> 14584704 -5.8% 13744640 ą 4% stress-ng.timerfd.ops
> 14546032 -5.7% 13718862 ą 4% stress-ng.timerfd.ops_per_sec
> 27.24 ą105% +283.9% 104.58 ą109% iostat.sdb.r_await.max
> 122324 ą 35% +63.9% 200505 ą 21% meminfo.AnonHugePages
> 47267 ą 26% +155.2% 120638 ą 45% numa-meminfo.node1.AnonHugePages
> 22880 ą 6% -9.9% 20605 ą 3% softirqs.CPU57.TIMER
> 636196 ą 24% +38.5% 880847 ą 7% cpuidle.C1.usage
> 55936214 ą 20% +63.9% 91684673 ą 18% cpuidle.C1E.time
> 1.175e+08 ą 22% +101.8% 2.372e+08 ą 29% cpuidle.C3.time
> 4.242e+08 ą 6% -39.1% 2.584e+08 ą 39% cpuidle.C6.time
> 59.50 ą 34% +66.0% 98.75 ą 22% proc-vmstat.nr_anon_transparent_hugepages
> 25612 ą 10% +13.8% 29146 ą 4% proc-vmstat.nr_kernel_stack
> 2783465 ą 9% +14.5% 3187157 ą 9% proc-vmstat.pgalloc_normal
> 1743 ą 28% +43.8% 2507 ą 23% proc-vmstat.thp_deferred_split_page
> 1765 ą 30% +43.2% 2529 ą 22% proc-vmstat.thp_fault_alloc
> 811.00 ą 3% -13.8% 699.00 ą 7% slabinfo.kmem_cache_node.active_objs
> 864.00 ą 3% -13.0% 752.00 ą 7% slabinfo.kmem_cache_node.num_objs
> 8686 ą 7% +13.6% 9869 ą 3% slabinfo.pid.active_objs
> 8690 ą 7% +13.8% 9890 ą 3% slabinfo.pid.num_objs
> 9813 ą 6% +15.7% 11352 ą 3% slabinfo.task_delay_info.active_objs
> 9813 ą 6% +15.7% 11352 ą 3% slabinfo.task_delay_info.num_objs
> 79.22 ą 10% -41.1% 46.68 ą 22% sched_debug.cfs_rq:/.load_avg.avg
> 242.49 ą 6% -29.6% 170.70 ą 17% sched_debug.cfs_rq:/.load_avg.stddev
> 43.14 ą 29% -67.1% 14.18 ą 66% sched_debug.cfs_rq:/.removed.load_avg.avg
> 201.73 ą 15% -50.1% 100.68 ą 60% sched_debug.cfs_rq:/.removed.load_avg.stddev
> 1987 ą 28% -67.3% 650.09 ą 66% sched_debug.cfs_rq:/.removed.runnable_sum.avg
> 9298 ą 15% -50.3% 4616 ą 60% sched_debug.cfs_rq:/.removed.runnable_sum.stddev
> 18.17 ą 27% -68.6% 5.70 ą 63% sched_debug.cfs_rq:/.removed.util_avg.avg
> 87.61 ą 13% -52.6% 41.48 ą 59% sched_debug.cfs_rq:/.removed.util_avg.stddev
> 633327 ą 24% +38.4% 876596 ą 7% turbostat.C1
> 2.75 ą 22% +1.8 4.52 ą 17% turbostat.C1E%
> 5.76 ą 22% +6.1 11.82 ą 30% turbostat.C3%
> 20.69 ą 5% -8.1 12.63 ą 38% turbostat.C6%
> 15.62 ą 6% +18.4% 18.50 ą 8% turbostat.CPU%c1
> 1.56 ą 16% +208.5% 4.82 ą 38% turbostat.CPU%c3
> 12.81 ą 4% -48.1% 6.65 ą 43% turbostat.CPU%c6
> 5.02 ą 8% -34.6% 3.28 ą 14% turbostat.Pkg%pc2
> 0.85 ą 57% -84.7% 0.13 ą173% turbostat.Pkg%pc6
> 88.25 ą 13% +262.6% 320.00 ą 71% interrupts.CPU10.TLB:TLB_shootdowns
> 116.25 ą 36% +151.6% 292.50 ą 68% interrupts.CPU19.TLB:TLB_shootdowns
> 109.25 ą 8% +217.4% 346.75 ą106% interrupts.CPU2.TLB:TLB_shootdowns
> 15180 ą111% +303.9% 61314 ą 32% interrupts.CPU23.RES:Rescheduling_interrupts
> 111.50 ą 26% +210.3% 346.00 ą 79% interrupts.CPU3.TLB:TLB_shootdowns
> 86.50 ą 35% +413.0% 443.75 ą 66% interrupts.CPU33.TLB:TLB_shootdowns
> 728.00 ą 8% +29.6% 943.50 ą 16% interrupts.CPU38.CAL:Function_call_interrupts
> 1070 ą 72% +84.9% 1979 ą 9% interrupts.CPU54.76:IR-PCI-MSI.512000-edge.ahci[0000:00:1f.2]
> 41429 ą 64% -73.7% 10882 ą 73% interrupts.CPU59.RES:Rescheduling_interrupts
> 26330 ą 85% -73.3% 7022 ą 86% interrupts.CPU62.RES:Rescheduling_interrupts
> 103.00 ą 22% +181.3% 289.75 ą 92% interrupts.CPU65.TLB:TLB_shootdowns
> 100.00 ą 40% +365.0% 465.00 ą 71% interrupts.CPU70.TLB:TLB_shootdowns
> 110.25 ą 18% +308.4% 450.25 ą 71% interrupts.CPU80.TLB:TLB_shootdowns
> 93.50 ą 42% +355.1% 425.50 ą 82% interrupts.CPU84.TLB:TLB_shootdowns
> 104.50 ą 18% +289.7% 407.25 ą 68% interrupts.CPU87.TLB:TLB_shootdowns
> 1.76 ą 3% -0.1 1.66 ą 4% perf-stat.i.branch-miss-rate%
> 8.08 ą 6% +2.0 10.04 perf-stat.i.cache-miss-rate%
> 18031213 ą 4% +27.2% 22939937 ą 3% perf-stat.i.cache-misses
> 4.041e+08 -1.9% 3.965e+08 perf-stat.i.cache-references
> 31764 ą 26% -40.6% 18859 ą 10% perf-stat.i.cycles-between-cache-misses
> 66.18 -1.5 64.71 perf-stat.i.iTLB-load-miss-rate%
> 4503482 ą 8% +19.5% 5382698 ą 5% perf-stat.i.node-load-misses
> 3892859 ą 2% +16.6% 4538750 ą 4% perf-stat.i.node-store-misses
> 1526815 ą 13% +25.8% 1921178 ą 9% perf-stat.i.node-stores
> 4.72 ą 4% +1.3 6.00 ą 3% perf-stat.overall.cache-miss-rate%
> 9120 ą 6% -18.9% 7394 ą 2% perf-stat.overall.cycles-between-cache-misses
> 18237318 ą 4% +25.4% 22866104 ą 3% perf-stat.ps.cache-misses
> 4392089 ą 8% +18.1% 5189251 ą 5% perf-stat.ps.node-load-misses
> 1629766 ą 2% +17.9% 1920947 ą 13% perf-stat.ps.node-loads
> 3694566 ą 2% +16.1% 4288126 ą 4% perf-stat.ps.node-store-misses
> 1536866 ą 12% +23.7% 1901141 ą 7% perf-stat.ps.node-stores
> 38.20 ą 18% -13.2 24.96 ą 10% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
> 38.20 ą 18% -13.2 24.96 ą 10% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 7.98 ą 67% -7.2 0.73 ą173% perf-profile.calltrace.cycles-pp.smp_call_function_single.event_function_call.perf_remove_from_context.perf_event_release_kernel.perf_release
> 7.98 ą 67% -7.2 0.73 ą173% perf-profile.calltrace.cycles-pp.event_function_call.perf_remove_from_context.perf_event_release_kernel.perf_release.__fput
> 7.98 ą 67% -7.2 0.73 ą173% perf-profile.calltrace.cycles-pp.perf_remove_from_context.perf_event_release_kernel.perf_release.__fput.task_work_run
> 11.86 ą 41% -6.8 5.07 ą 62% perf-profile.calltrace.cycles-pp.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 11.86 ą 41% -6.8 5.07 ą 62% perf-profile.calltrace.cycles-pp.do_signal.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 11.86 ą 41% -6.8 5.07 ą 62% perf-profile.calltrace.cycles-pp.get_signal.do_signal.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 11.86 ą 41% -6.8 5.07 ą 62% perf-profile.calltrace.cycles-pp.do_group_exit.get_signal.do_signal.exit_to_usermode_loop.do_syscall_64
> 11.86 ą 41% -6.8 5.07 ą 62% perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.get_signal.do_signal.exit_to_usermode_loop
> 11.86 ą 41% -6.8 5.07 ą 62% perf-profile.calltrace.cycles-pp.task_work_run.do_exit.do_group_exit.get_signal.do_signal
> 4.27 ą 66% -3.5 0.73 ą173% perf-profile.calltrace.cycles-pp.read
> 4.05 ą 71% -3.3 0.73 ą173% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.read
> 4.05 ą 71% -3.3 0.73 ą173% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
> 13.30 ą 38% -8.2 5.07 ą 62% perf-profile.children.cycles-pp.task_work_run
> 12.47 ą 46% -7.4 5.07 ą 62% perf-profile.children.cycles-pp.exit_to_usermode_loop
> 12.47 ą 46% -7.4 5.07 ą 62% perf-profile.children.cycles-pp.__fput
> 7.98 ą 67% -7.2 0.73 ą173% perf-profile.children.cycles-pp.perf_remove_from_context
> 11.86 ą 41% -6.8 5.07 ą 62% perf-profile.children.cycles-pp.do_signal
> 11.86 ą 41% -6.8 5.07 ą 62% perf-profile.children.cycles-pp.get_signal
> 9.43 ą 21% -4.7 4.72 ą 67% perf-profile.children.cycles-pp.ksys_read
> 9.43 ą 21% -4.7 4.72 ą 67% perf-profile.children.cycles-pp.vfs_read
> 4.27 ą 66% -3.5 0.73 ą173% perf-profile.children.cycles-pp.read
> 3.86 ą101% -3.1 0.71 ą173% perf-profile.children.cycles-pp._raw_spin_lock
> 3.86 ą101% -3.1 0.71 ą173% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
> 3.86 ą101% -3.1 0.71 ą173% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
>
>
>
> ***************************************************************************************************
> lkp-csl-2sp5: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 192G memory
> =========================================================================================
> class/compiler/cpufreq_governor/disk/fs/kconfig/nr_threads/rootfs/tbox_group/testcase/testtime/ucode:
> os/gcc-7/performance/1HDD/ext4/x86_64-rhel-7.6/100%/debian-x86_64-2019-09-23.cgz/lkp-csl-2sp5/stress-ng/1s/0x500002b
>
> commit:
> fcf0553db6 ("sched/fair: Remove meaningless imbalance calculation")
> 0b0695f2b3 ("sched/fair: Rework load_balance()")
>
> fcf0553db6f4c793 0b0695f2b34a4afa3f6e9aa1ff0
> ---------------- ---------------------------
> fail:runs %reproduction fail:runs
> | | |
> :2 50% 1:8 dmesg.WARNING:at_ip_selinux_file_ioctl/0x
> %stddev %change %stddev
> \ | \
> 122451 ą 11% -19.9% 98072 ą 15% stress-ng.ioprio.ops
> 116979 ą 11% -20.7% 92815 ą 16% stress-ng.ioprio.ops_per_sec
> 274187 ą 21% -26.7% 201013 ą 11% stress-ng.kill.ops
> 274219 ą 21% -26.7% 201040 ą 11% stress-ng.kill.ops_per_sec
> 3973765 -10.1% 3570462 ą 5% stress-ng.lockf.ops
> 3972581 -10.2% 3568935 ą 5% stress-ng.lockf.ops_per_sec
> 10719 ą 8% -39.9% 6442 ą 22% stress-ng.procfs.ops
> 9683 ą 3% -39.3% 5878 ą 22% stress-ng.procfs.ops_per_sec
> 6562721 -35.1% 4260609 ą 8% stress-ng.schedpolicy.ops
> 6564233 -35.1% 4261479 ą 8% stress-ng.schedpolicy.ops_per_sec
> 1070988 +21.4% 1299977 ą 7% stress-ng.sigrt.ops
> 1061773 +21.2% 1286618 ą 7% stress-ng.sigrt.ops_per_sec
> 1155684 ą 5% -14.8% 984531 ą 16% stress-ng.symlink.ops
> 991624 ą 4% -23.8% 755147 ą 41% stress-ng.symlink.ops_per_sec
> 6925 -12.1% 6086 ą 27% stress-ng.time.percent_of_cpu_this_job_got
> 24.68 +9.3 33.96 ą 52% mpstat.cpu.all.idle%
> 171.00 ą 2% -55.3% 76.50 ą 60% numa-vmstat.node1.nr_inactive_file
> 171.00 ą 2% -55.3% 76.50 ą 60% numa-vmstat.node1.nr_zone_inactive_file
> 2.032e+11 -12.5% 1.777e+11 ą 27% perf-stat.i.cpu-cycles
> 2.025e+11 -12.0% 1.782e+11 ą 27% perf-stat.ps.cpu-cycles
> 25.00 +37.5% 34.38 ą 51% vmstat.cpu.id
> 68.00 -13.2% 59.00 ą 27% vmstat.cpu.sy
> 25.24 +37.0% 34.57 ą 51% iostat.cpu.idle
> 68.21 -12.7% 59.53 ą 27% iostat.cpu.system
> 4.31 ą100% +200.6% 12.96 ą 63% iostat.sda.r_await.max
> 1014 ą 2% -17.1% 841.00 ą 10% meminfo.Inactive(file)
> 30692 ą 12% -20.9% 24280 ą 30% meminfo.Mlocked
> 103627 ą 27% -32.7% 69720 meminfo.Percpu
> 255.50 ą 2% -18.1% 209.25 ą 10% proc-vmstat.nr_inactive_file
> 255.50 ą 2% -18.1% 209.25 ą 10% proc-vmstat.nr_zone_inactive_file
> 185035 ą 22% -22.2% 143917 ą 25% proc-vmstat.pgmigrate_success
> 2107 -12.3% 1848 ą 27% turbostat.Avg_MHz
> 69.00 -7.1% 64.12 ą 8% turbostat.PkgTmp
> 94.63 -2.2% 92.58 ą 4% turbostat.RAMWatt
> 96048 +26.8% 121800 ą 8% softirqs.CPU10.NET_RX
> 96671 ą 4% +34.2% 129776 ą 6% softirqs.CPU15.NET_RX
> 171243 ą 3% -12.9% 149135 ą 8% softirqs.CPU25.NET_RX
> 165317 ą 4% -11.4% 146494 ą 9% softirqs.CPU27.NET_RX
> 139558 -24.5% 105430 ą 14% softirqs.CPU58.NET_RX
> 147836 -15.8% 124408 ą 6% softirqs.CPU63.NET_RX
> 129568 -13.8% 111624 ą 10% softirqs.CPU66.NET_RX
> 1050 ą 2% +14.2% 1198 ą 9% slabinfo.biovec-128.active_objs
> 1050 ą 2% +14.2% 1198 ą 9% slabinfo.biovec-128.num_objs
> 23129 +19.6% 27668 ą 6% slabinfo.kmalloc-512.active_objs
> 766.50 +17.4% 899.75 ą 6% slabinfo.kmalloc-512.active_slabs
> 24535 +17.4% 28806 ą 6% slabinfo.kmalloc-512.num_objs
> 766.50 +17.4% 899.75 ą 6% slabinfo.kmalloc-512.num_slabs
> 1039 ą 4% -4.3% 994.12 ą 6% slabinfo.sock_inode_cache.active_slabs
> 40527 ą 4% -4.3% 38785 ą 6% slabinfo.sock_inode_cache.num_objs
> 1039 ą 4% -4.3% 994.12 ą 6% slabinfo.sock_inode_cache.num_slabs
> 1549456 -43.6% 873443 ą 24% sched_debug.cfs_rq:/.min_vruntime.stddev
> 73.25 ą 5% +74.8% 128.03 ą 31% sched_debug.cfs_rq:/.nr_spread_over.stddev
> 18.60 ą 57% -63.8% 6.73 ą 64% sched_debug.cfs_rq:/.removed.load_avg.avg
> 79.57 ą 44% -44.1% 44.52 ą 55% sched_debug.cfs_rq:/.removed.load_avg.stddev
> 857.10 ą 57% -63.8% 310.09 ą 64% sched_debug.cfs_rq:/.removed.runnable_sum.avg
> 3664 ą 44% -44.1% 2049 ą 55% sched_debug.cfs_rq:/.removed.runnable_sum.stddev
> 4.91 ą 42% -45.3% 2.69 ą 61% sched_debug.cfs_rq:/.removed.util_avg.avg
> 1549544 -43.6% 874006 ą 24% sched_debug.cfs_rq:/.spread0.stddev
> 786.14 ą 6% -20.1% 628.46 ą 23% sched_debug.cfs_rq:/.util_avg.avg
> 1415 ą 8% -16.7% 1178 ą 18% sched_debug.cfs_rq:/.util_avg.max
> 467435 ą 15% +46.7% 685829 ą 15% sched_debug.cpu.avg_idle.avg
> 17972 ą 8% +631.2% 131410 ą 34% sched_debug.cpu.avg_idle.min
> 7.66 ą 26% +209.7% 23.72 ą 54% sched_debug.cpu.clock.stddev
> 7.66 ą 26% +209.7% 23.72 ą 54% sched_debug.cpu.clock_task.stddev
> 618063 ą 5% -17.0% 513085 ą 5% sched_debug.cpu.max_idle_balance_cost.max
> 12083 ą 28% -85.4% 1768 ą231% sched_debug.cpu.max_idle_balance_cost.stddev
> 12857 ą 16% +2117.7% 285128 ą106% sched_debug.cpu.yld_count.min
> 0.55 ą 6% -0.2 0.37 ą 51% perf-profile.children.cycles-pp.fpregs_assert_state_consistent
> 0.30 ą 21% -0.2 0.14 ą105% perf-profile.children.cycles-pp.yield_task_fair
> 0.32 ą 6% -0.2 0.16 ą 86% perf-profile.children.cycles-pp.rmap_walk_anon
> 0.19 -0.1 0.10 ą 86% perf-profile.children.cycles-pp.page_mapcount_is_zero
> 0.19 -0.1 0.10 ą 86% perf-profile.children.cycles-pp.total_mapcount
> 0.14 -0.1 0.09 ą 29% perf-profile.children.cycles-pp.start_kernel
> 0.11 ą 9% -0.0 0.07 ą 47% perf-profile.children.cycles-pp.__switch_to
> 0.10 ą 14% -0.0 0.06 ą 45% perf-profile.children.cycles-pp.switch_fpu_return
> 0.08 ą 6% -0.0 0.04 ą 79% perf-profile.children.cycles-pp.__update_load_avg_se
> 0.12 ą 13% -0.0 0.09 ą 23% perf-profile.children.cycles-pp.native_write_msr
> 0.31 ą 6% -0.2 0.15 ą 81% perf-profile.self.cycles-pp.poll_idle
> 0.50 ą 6% -0.2 0.35 ą 50% perf-profile.self.cycles-pp.fpregs_assert_state_consistent
> 0.18 ą 2% -0.1 0.10 ą 86% perf-profile.self.cycles-pp.total_mapcount
> 0.10 ą 14% -0.0 0.06 ą 45% perf-profile.self.cycles-pp.switch_fpu_return
> 0.10 ą 10% -0.0 0.06 ą 47% perf-profile.self.cycles-pp.__switch_to
> 0.07 ą 7% -0.0 0.03 ą100% perf-profile.self.cycles-pp.prep_new_page
> 0.07 ą 7% -0.0 0.03 ą100% perf-profile.self.cycles-pp.llist_add_batch
> 0.07 ą 14% -0.0 0.04 ą 79% perf-profile.self.cycles-pp.__update_load_avg_se
> 0.12 ą 13% -0.0 0.09 ą 23% perf-profile.self.cycles-pp.native_write_msr
> 66096 ą 99% -99.8% 148.50 ą 92% interrupts.109:PCI-MSI.31981642-edge.i40e-eth0-TxRx-73
> 543.50 ą 39% -73.3% 145.38 ą 81% interrupts.112:PCI-MSI.31981645-edge.i40e-eth0-TxRx-76
> 169.00 ą 28% -55.3% 75.50 ą 83% interrupts.116:PCI-MSI.31981649-edge.i40e-eth0-TxRx-80
> 224.00 ą 14% -57.6% 95.00 ą 87% interrupts.121:PCI-MSI.31981654-edge.i40e-eth0-TxRx-85
> 680.00 ą 28% -80.5% 132.75 ą 82% interrupts.43:PCI-MSI.31981576-edge.i40e-eth0-TxRx-7
> 327.50 ą 31% -39.0% 199.62 ą 60% interrupts.60:PCI-MSI.31981593-edge.i40e-eth0-TxRx-24
> 217.50 ą 19% -51.7% 105.12 ą 79% interrupts.63:PCI-MSI.31981596-edge.i40e-eth0-TxRx-27
> 375.00 ą 46% -78.5% 80.50 ą 82% interrupts.70:PCI-MSI.31981603-edge.i40e-eth0-TxRx-34
> 196.50 ą 3% -51.6% 95.12 ą 74% interrupts.72:PCI-MSI.31981605-edge.i40e-eth0-TxRx-36
> 442.50 ą 45% -73.1% 118.88 ą 90% interrupts.79:PCI-MSI.31981612-edge.i40e-eth0-TxRx-43
> 271.00 ą 8% -53.2% 126.88 ą 75% interrupts.82:PCI-MSI.31981615-edge.i40e-eth0-TxRx-46
> 145448 ą 4% -41.6% 84975 ą 42% interrupts.CPU1.RES:Rescheduling_interrupts
> 11773 ą 19% -38.1% 7290 ą 52% interrupts.CPU13.TLB:TLB_shootdowns
> 24177 ą 15% +356.5% 110368 ą 58% interrupts.CPU16.RES:Rescheduling_interrupts
> 3395 ą 3% +78.3% 6055 ą 18% interrupts.CPU17.NMI:Non-maskable_interrupts
> 3395 ą 3% +78.3% 6055 ą 18% interrupts.CPU17.PMI:Performance_monitoring_interrupts
> 106701 ą 41% -55.6% 47425 ą 56% interrupts.CPU18.RES:Rescheduling_interrupts
> 327.50 ą 31% -39.3% 198.88 ą 60% interrupts.CPU24.60:PCI-MSI.31981593-edge.i40e-eth0-TxRx-24
> 411618 +53.6% 632283 ą 77% interrupts.CPU25.LOC:Local_timer_interrupts
> 16189 ą 26% -53.0% 7611 ą 66% interrupts.CPU25.TLB:TLB_shootdowns
> 407253 +54.4% 628596 ą 78% interrupts.CPU26.LOC:Local_timer_interrupts
> 216.50 ą 19% -51.8% 104.25 ą 80% interrupts.CPU27.63:PCI-MSI.31981596-edge.i40e-eth0-TxRx-27
> 7180 -20.9% 5682 ą 25% interrupts.CPU29.NMI:Non-maskable_interrupts
> 7180 -20.9% 5682 ą 25% interrupts.CPU29.PMI:Performance_monitoring_interrupts
> 15186 ą 12% -45.5% 8276 ą 49% interrupts.CPU3.TLB:TLB_shootdowns
> 13092 ą 19% -29.5% 9231 ą 35% interrupts.CPU30.TLB:TLB_shootdowns
> 13204 ą 26% -29.3% 9336 ą 19% interrupts.CPU31.TLB:TLB_shootdowns
> 374.50 ą 46% -78.7% 79.62 ą 83% interrupts.CPU34.70:PCI-MSI.31981603-edge.i40e-eth0-TxRx-34
> 7188 -25.6% 5345 ą 26% interrupts.CPU35.NMI:Non-maskable_interrupts
> 7188 -25.6% 5345 ą 26% interrupts.CPU35.PMI:Performance_monitoring_interrupts
> 196.00 ą 4% -52.0% 94.12 ą 75% interrupts.CPU36.72:PCI-MSI.31981605-edge.i40e-eth0-TxRx-36
> 12170 ą 20% -34.3% 7998 ą 32% interrupts.CPU39.TLB:TLB_shootdowns
> 442.00 ą 45% -73.3% 118.12 ą 91% interrupts.CPU43.79:PCI-MSI.31981612-edge.i40e-eth0-TxRx-43
> 12070 ą 15% -37.2% 7581 ą 49% interrupts.CPU43.TLB:TLB_shootdowns
> 7177 -27.6% 5195 ą 26% interrupts.CPU45.NMI:Non-maskable_interrupts
> 7177 -27.6% 5195 ą 26% interrupts.CPU45.PMI:Performance_monitoring_interrupts
> 271.00 ą 8% -53.4% 126.38 ą 75% interrupts.CPU46.82:PCI-MSI.31981615-edge.i40e-eth0-TxRx-46
> 3591 +84.0% 6607 ą 12% interrupts.CPU46.NMI:Non-maskable_interrupts
> 3591 +84.0% 6607 ą 12% interrupts.CPU46.PMI:Performance_monitoring_interrupts
> 57614 ą 30% -34.0% 38015 ą 28% interrupts.CPU46.RES:Rescheduling_interrupts
> 149154 ą 41% -47.2% 78808 ą 51% interrupts.CPU51.RES:Rescheduling_interrupts
> 30366 ą 28% +279.5% 115229 ą 42% interrupts.CPU52.RES:Rescheduling_interrupts
> 29690 +355.5% 135237 ą 57% interrupts.CPU54.RES:Rescheduling_interrupts
> 213106 ą 2% -66.9% 70545 ą 43% interrupts.CPU59.RES:Rescheduling_interrupts
> 225753 ą 7% -72.9% 61212 ą 72% interrupts.CPU60.RES:Rescheduling_interrupts
> 12430 ą 14% -41.5% 7276 ą 52% interrupts.CPU61.TLB:TLB_shootdowns
> 44552 ą 22% +229.6% 146864 ą 36% interrupts.CPU65.RES:Rescheduling_interrupts
> 126088 ą 56% -35.3% 81516 ą 73% interrupts.CPU66.RES:Rescheduling_interrupts
> 170880 ą 15% -62.9% 63320 ą 52% interrupts.CPU68.RES:Rescheduling_interrupts
> 186033 ą 10% -39.8% 112012 ą 41% interrupts.CPU69.RES:Rescheduling_interrupts
> 679.50 ą 29% -80.5% 132.25 ą 82% interrupts.CPU7.43:PCI-MSI.31981576-edge.i40e-eth0-TxRx-7
> 124750 ą 18% -39.4% 75553 ą 43% interrupts.CPU7.RES:Rescheduling_interrupts
> 158500 ą 47% -52.1% 75915 ą 67% interrupts.CPU71.RES:Rescheduling_interrupts
> 11846 ą 11% -32.5% 8001 ą 47% interrupts.CPU72.TLB:TLB_shootdowns
> 66095 ą 99% -99.8% 147.62 ą 93% interrupts.CPU73.109:PCI-MSI.31981642-edge.i40e-eth0-TxRx-73
> 7221 ą 2% -31.0% 4982 ą 35% interrupts.CPU73.NMI:Non-maskable_interrupts
> 7221 ą 2% -31.0% 4982 ą 35% interrupts.CPU73.PMI:Performance_monitoring_interrupts
> 15304 ą 14% -47.9% 7972 ą 31% interrupts.CPU73.TLB:TLB_shootdowns
> 10918 ą 3% -31.9% 7436 ą 36% interrupts.CPU74.TLB:TLB_shootdowns
> 543.00 ą 39% -73.3% 144.75 ą 81% interrupts.CPU76.112:PCI-MSI.31981645-edge.i40e-eth0-TxRx-76
> 12214 ą 14% -40.9% 7220 ą 38% interrupts.CPU79.TLB:TLB_shootdowns
> 168.00 ą 29% -55.7% 74.50 ą 85% interrupts.CPU80.116:PCI-MSI.31981649-edge.i40e-eth0-TxRx-80
> 28619 ą 3% +158.4% 73939 ą 44% interrupts.CPU80.RES:Rescheduling_interrupts
> 12258 -34.3% 8056 ą 29% interrupts.CPU80.TLB:TLB_shootdowns
> 7214 -19.5% 5809 ą 24% interrupts.CPU82.NMI:Non-maskable_interrupts
> 7214 -19.5% 5809 ą 24% interrupts.CPU82.PMI:Performance_monitoring_interrupts
> 13522 ą 11% -41.2% 7949 ą 29% interrupts.CPU84.TLB:TLB_shootdowns
> 223.50 ą 14% -57.8% 94.25 ą 88% interrupts.CPU85.121:PCI-MSI.31981654-edge.i40e-eth0-TxRx-85
> 11989 ą 2% -31.7% 8194 ą 22% interrupts.CPU85.TLB:TLB_shootdowns
> 121153 ą 29% -41.4% 70964 ą 58% interrupts.CPU86.RES:Rescheduling_interrupts
> 11731 ą 8% -40.7% 6957 ą 36% interrupts.CPU86.TLB:TLB_shootdowns
> 12192 ą 22% -35.8% 7824 ą 43% interrupts.CPU87.TLB:TLB_shootdowns
> 11603 ą 19% -31.8% 7915 ą 41% interrupts.CPU89.TLB:TLB_shootdowns
> 10471 ą 5% -27.0% 7641 ą 31% interrupts.CPU91.TLB:TLB_shootdowns
> 7156 -20.9% 5658 ą 23% interrupts.CPU92.NMI:Non-maskable_interrupts
> 7156 -20.9% 5658 ą 23% interrupts.CPU92.PMI:Performance_monitoring_interrupts
> 99802 ą 20% -43.6% 56270 ą 47% interrupts.CPU92.RES:Rescheduling_interrupts
> 109162 ą 18% -28.7% 77839 ą 26% interrupts.CPU93.RES:Rescheduling_interrupts
> 15044 ą 29% -44.4% 8359 ą 30% interrupts.CPU93.TLB:TLB_shootdowns
> 110749 ą 19% -47.3% 58345 ą 48% interrupts.CPU94.RES:Rescheduling_interrupts
> 7245 -21.4% 5697 ą 25% interrupts.CPU95.NMI:Non-maskable_interrupts
> 7245 -21.4% 5697 ą 25% interrupts.CPU95.PMI:Performance_monitoring_interrupts
> 1969 ą 5% +491.7% 11653 ą 81% interrupts.IWI:IRQ_work_interrupts
>
>
>
> ***************************************************************************************************
> lkp-bdw-ep6: 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory
> =========================================================================================
> class/compiler/cpufreq_governor/disk/kconfig/nr_threads/rootfs/tbox_group/testcase/testtime/ucode:
> interrupt/gcc-7/performance/1HDD/x86_64-rhel-7.6/100%/debian-x86_64-2019-09-23.cgz/lkp-bdw-ep6/stress-ng/30s/0xb000038
>
> commit:
> fcf0553db6 ("sched/fair: Remove meaningless imbalance calculation")
> 0b0695f2b3 ("sched/fair: Rework load_balance()")
>
> fcf0553db6f4c793 0b0695f2b34a4afa3f6e9aa1ff0
> ---------------- ---------------------------
> %stddev %change %stddev
> \ | \
> 98318389 +43.0% 1.406e+08 stress-ng.schedpolicy.ops
> 3277346 +43.0% 4685146 stress-ng.schedpolicy.ops_per_sec
> 3.506e+08 ą 4% -10.3% 3.146e+08 ą 3% stress-ng.sigq.ops
> 11684738 ą 4% -10.3% 10485353 ą 3% stress-ng.sigq.ops_per_sec
> 3.628e+08 ą 6% -19.4% 2.925e+08 ą 6% stress-ng.time.involuntary_context_switches
> 29456 +2.8% 30285 stress-ng.time.system_time
> 7636655 ą 9% +46.6% 11197377 ą 27% cpuidle.C1E.usage
> 1111483 ą 3% -9.5% 1005829 vmstat.system.cs
> 22638222 ą 4% +16.5% 26370816 ą 11% meminfo.Committed_AS
> 28908 ą 6% +24.6% 36020 ą 16% meminfo.KernelStack
> 7636543 ą 9% +46.6% 11196090 ą 27% turbostat.C1E
> 3.46 ą 16% -61.2% 1.35 ą 7% turbostat.Pkg%pc2
> 217.54 +1.7% 221.33 turbostat.PkgWatt
> 13.34 ą 2% +5.8% 14.11 turbostat.RAMWatt
> 525.50 ą 8% -15.7% 443.00 ą 12% slabinfo.biovec-128.active_objs
> 525.50 ą 8% -15.7% 443.00 ą 12% slabinfo.biovec-128.num_objs
> 28089 ą 12% -33.0% 18833 ą 22% slabinfo.pool_workqueue.active_objs
> 877.25 ą 12% -32.6% 591.00 ą 21% slabinfo.pool_workqueue.active_slabs
> 28089 ą 12% -32.6% 18925 ą 21% slabinfo.pool_workqueue.num_objs
> 877.25 ą 12% -32.6% 591.00 ą 21% slabinfo.pool_workqueue.num_slabs
> 846.75 ą 6% -18.0% 694.75 ą 9% slabinfo.skbuff_fclone_cache.active_objs
> 846.75 ą 6% -18.0% 694.75 ą 9% slabinfo.skbuff_fclone_cache.num_objs
> 63348 ą 6% -20.7% 50261 ą 4% softirqs.CPU14.SCHED
> 44394 ą 4% +21.4% 53880 ą 8% softirqs.CPU42.SCHED
> 52246 ą 7% -15.1% 44352 softirqs.CPU47.SCHED
> 58350 ą 4% -11.0% 51914 ą 7% softirqs.CPU6.SCHED
> 58009 ą 7% -23.8% 44206 ą 4% softirqs.CPU63.SCHED
> 49166 ą 6% +23.4% 60683 ą 9% softirqs.CPU68.SCHED
> 44594 ą 7% +14.3% 50951 ą 8% softirqs.CPU78.SCHED
> 46407 ą 9% +19.6% 55515 ą 8% softirqs.CPU84.SCHED
> 55555 ą 8% -15.5% 46933 ą 4% softirqs.CPU9.SCHED
> 198757 ą 18% +44.1% 286316 ą 9% numa-meminfo.node0.Active
> 189280 ą 19% +37.1% 259422 ą 7% numa-meminfo.node0.Active(anon)
> 110438 ą 33% +68.3% 185869 ą 16% numa-meminfo.node0.AnonHugePages
> 143458 ą 28% +67.7% 240547 ą 13% numa-meminfo.node0.AnonPages
> 12438 ą 16% +61.9% 20134 ą 37% numa-meminfo.node0.KernelStack
> 1004379 ą 7% +16.4% 1168764 ą 4% numa-meminfo.node0.MemUsed
> 357111 ą 24% -41.6% 208655 ą 29% numa-meminfo.node1.Active
> 330094 ą 22% -39.6% 199339 ą 32% numa-meminfo.node1.Active(anon)
> 265924 ą 25% -52.2% 127138 ą 46% numa-meminfo.node1.AnonHugePages
> 314059 ą 22% -49.6% 158305 ą 36% numa-meminfo.node1.AnonPages
> 15386 ą 16% -25.1% 11525 ą 15% numa-meminfo.node1.KernelStack
> 1200805 ą 11% -18.6% 977595 ą 7% numa-meminfo.node1.MemUsed
> 965.50 ą 15% -29.3% 682.25 ą 43% numa-meminfo.node1.Mlocked
> 46762 ą 18% +37.8% 64452 ą 8% numa-vmstat.node0.nr_active_anon
> 35393 ą 27% +68.9% 59793 ą 12% numa-vmstat.node0.nr_anon_pages
> 52.75 ą 33% +71.1% 90.25 ą 15% numa-vmstat.node0.nr_anon_transparent_hugepages
> 15.00 ą 96% +598.3% 104.75 ą 15% numa-vmstat.node0.nr_inactive_file
> 11555 ą 22% +68.9% 19513 ą 41% numa-vmstat.node0.nr_kernel_stack
> 550.25 ą162% +207.5% 1691 ą 48% numa-vmstat.node0.nr_written
> 46762 ą 18% +37.8% 64452 ą 8% numa-vmstat.node0.nr_zone_active_anon
> 15.00 ą 96% +598.3% 104.75 ą 15% numa-vmstat.node0.nr_zone_inactive_file
> 82094 ą 22% -39.5% 49641 ą 32% numa-vmstat.node1.nr_active_anon
> 78146 ą 23% -49.5% 39455 ą 37% numa-vmstat.node1.nr_anon_pages
> 129.00 ą 25% -52.3% 61.50 ą 47% numa-vmstat.node1.nr_anon_transparent_hugepages
> 107.75 ą 12% -85.4% 15.75 ą103% numa-vmstat.node1.nr_inactive_file
> 14322 ą 11% -21.1% 11304 ą 11% numa-vmstat.node1.nr_kernel_stack
> 241.00 ą 15% -29.5% 170.00 ą 43% numa-vmstat.node1.nr_mlock
> 82094 ą 22% -39.5% 49641 ą 32% numa-vmstat.node1.nr_zone_active_anon
> 107.75 ą 12% -85.4% 15.75 ą103% numa-vmstat.node1.nr_zone_inactive_file
> 0.81 ą 5% +0.2 0.99 ą 10% perf-profile.calltrace.cycles-pp.task_rq_lock.task_sched_runtime.cpu_clock_sample.posix_cpu_timer_get.do_timer_gettime
> 0.60 ą 11% +0.2 0.83 ą 9% perf-profile.calltrace.cycles-pp.___might_sleep.__might_fault._copy_to_user.put_itimerspec64.__x64_sys_timer_gettime
> 1.73 ą 9% +0.3 2.05 ą 8% perf-profile.calltrace.cycles-pp.__might_fault._copy_to_user.put_itimerspec64.__x64_sys_timer_gettime.do_syscall_64
> 3.92 ą 5% +0.6 4.49 ą 7% perf-profile.calltrace.cycles-pp.task_sched_runtime.cpu_clock_sample.posix_cpu_timer_get.do_timer_gettime.__x64_sys_timer_gettime
> 4.17 ą 4% +0.6 4.78 ą 7% perf-profile.calltrace.cycles-pp.cpu_clock_sample.posix_cpu_timer_get.do_timer_gettime.__x64_sys_timer_gettime.do_syscall_64
> 5.72 ą 3% +0.7 6.43 ą 7% perf-profile.calltrace.cycles-pp.posix_cpu_timer_get.do_timer_gettime.__x64_sys_timer_gettime.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 0.24 ą 54% -0.2 0.07 ą131% perf-profile.children.cycles-pp.ext4_inode_csum_set
> 0.45 ą 3% +0.1 0.56 ą 4% perf-profile.children.cycles-pp.__might_sleep
> 0.84 ą 5% +0.2 1.03 ą 9% perf-profile.children.cycles-pp.task_rq_lock
> 0.66 ą 8% +0.2 0.88 ą 7% perf-profile.children.cycles-pp.___might_sleep
> 1.83 ą 9% +0.3 2.16 ą 8% perf-profile.children.cycles-pp.__might_fault
> 4.04 ą 5% +0.6 4.62 ą 7% perf-profile.children.cycles-pp.task_sched_runtime
> 4.24 ą 4% +0.6 4.87 ą 7% perf-profile.children.cycles-pp.cpu_clock_sample
> 5.77 ą 3% +0.7 6.48 ą 7% perf-profile.children.cycles-pp.posix_cpu_timer_get
> 0.22 ą 11% +0.1 0.28 ą 15% perf-profile.self.cycles-pp.cpu_clock_sample
> 0.47 ą 7% +0.1 0.55 ą 5% perf-profile.self.cycles-pp.update_curr
> 0.28 ą 5% +0.1 0.38 ą 14% perf-profile.self.cycles-pp.task_rq_lock
> 0.42 ą 3% +0.1 0.53 ą 4% perf-profile.self.cycles-pp.__might_sleep
> 0.50 ą 5% +0.1 0.61 ą 11% perf-profile.self.cycles-pp.task_sched_runtime
> 0.63 ą 9% +0.2 0.85 ą 7% perf-profile.self.cycles-pp.___might_sleep
> 9180611 ą 5% +40.1% 12859327 ą 14% sched_debug.cfs_rq:/.MIN_vruntime.max
> 1479571 ą 6% +57.6% 2331469 ą 14% sched_debug.cfs_rq:/.MIN_vruntime.stddev
> 7951 ą 6% -52.5% 3773 ą 17% sched_debug.cfs_rq:/.exec_clock.stddev
> 321306 ą 39% -44.2% 179273 sched_debug.cfs_rq:/.load.max
> 9180613 ą 5% +40.1% 12859327 ą 14% sched_debug.cfs_rq:/.max_vruntime.max
> 1479571 ą 6% +57.6% 2331469 ą 14% sched_debug.cfs_rq:/.max_vruntime.stddev
> 16622378 +20.0% 19940069 ą 7% sched_debug.cfs_rq:/.min_vruntime.avg
> 18123901 +19.7% 21686545 ą 6% sched_debug.cfs_rq:/.min_vruntime.max
> 14338218 ą 3% +27.4% 18267927 ą 7% sched_debug.cfs_rq:/.min_vruntime.min
> 0.17 ą 16% +23.4% 0.21 ą 11% sched_debug.cfs_rq:/.nr_running.stddev
> 319990 ą 39% -44.6% 177347 sched_debug.cfs_rq:/.runnable_weight.max
> -2067420 -33.5% -1375445 sched_debug.cfs_rq:/.spread0.min
> 1033 ą 8% -13.7% 891.85 ą 3% sched_debug.cfs_rq:/.util_est_enqueued.max
> 93676 ą 16% -29.0% 66471 ą 17% sched_debug.cpu.avg_idle.min
> 10391 ą 52% +118.9% 22750 ą 15% sched_debug.cpu.curr->pid.avg
> 14393 ą 35% +113.2% 30689 ą 17% sched_debug.cpu.curr->pid.max
> 3041 ą 38% +161.8% 7963 ą 11% sched_debug.cpu.curr->pid.stddev
> 3.38 ą 6% -16.3% 2.83 ą 5% sched_debug.cpu.nr_running.max
> 2412687 ą 4% -16.0% 2027251 ą 3% sched_debug.cpu.nr_switches.avg
> 4038819 ą 3% -20.2% 3223112 ą 5% sched_debug.cpu.nr_switches.max
> 834203 ą 17% -37.8% 518798 ą 27% sched_debug.cpu.nr_switches.stddev
> 45.85 ą 13% +41.2% 64.75 ą 18% sched_debug.cpu.nr_uninterruptible.max
> 1937209 ą 2% +58.5% 3070891 ą 3% sched_debug.cpu.sched_count.min
> 1074023 ą 13% -57.9% 451958 ą 12% sched_debug.cpu.sched_count.stddev
> 1283769 ą 7% +65.1% 2118907 ą 7% sched_debug.cpu.yld_count.min
> 714244 ą 5% -51.9% 343373 ą 22% sched_debug.cpu.yld_count.stddev
> 12.54 ą 9% -18.8% 10.18 ą 15% perf-stat.i.MPKI
> 1.011e+10 +2.6% 1.038e+10 perf-stat.i.branch-instructions
> 13.22 ą 5% +2.5 15.75 ą 3% perf-stat.i.cache-miss-rate%
> 21084021 ą 6% +33.9% 28231058 ą 6% perf-stat.i.cache-misses
> 1143861 ą 5% -12.1% 1005721 ą 6% perf-stat.i.context-switches
> 1.984e+11 +1.8% 2.02e+11 perf-stat.i.cpu-cycles
> 1.525e+10 +1.3% 1.544e+10 perf-stat.i.dTLB-loads
> 65.46 -2.7 62.76 ą 3% perf-stat.i.iTLB-load-miss-rate%
> 20360883 ą 4% +10.5% 22500874 ą 4% perf-stat.i.iTLB-loads
> 4.963e+10 +2.0% 5.062e+10 perf-stat.i.instructions
> 181557 -2.4% 177113 perf-stat.i.msec
> 5350122 ą 8% +26.5% 6765332 ą 7% perf-stat.i.node-load-misses
> 4264320 ą 3% +24.8% 5321600 ą 4% perf-stat.i.node-store-misses
> 6.12 ą 5% +1.5 7.60 ą 2% perf-stat.overall.cache-miss-rate%
> 7646 ą 6% -17.7% 6295 ą 3% perf-stat.overall.cycles-between-cache-misses
> 69.29 -1.1 68.22 perf-stat.overall.iTLB-load-miss-rate%
> 61.11 ą 2% +6.6 67.71 ą 5% perf-stat.overall.node-load-miss-rate%
> 74.82 +1.8 76.58 perf-stat.overall.node-store-miss-rate%
> 1.044e+10 +1.8% 1.063e+10 perf-stat.ps.branch-instructions
> 26325951 ą 6% +22.9% 32366684 ą 2% perf-stat.ps.cache-misses
> 1115530 ą 3% -9.5% 1009780 perf-stat.ps.context-switches
> 1.536e+10 +1.0% 1.552e+10 perf-stat.ps.dTLB-loads
> 44718416 ą 2% +5.8% 47308605 ą 3% perf-stat.ps.iTLB-load-misses
> 19831973 ą 4% +11.1% 22040029 ą 4% perf-stat.ps.iTLB-loads
> 5.064e+10 +1.4% 5.137e+10 perf-stat.ps.instructions
> 5454694 ą 9% +26.4% 6892365 ą 6% perf-stat.ps.node-load-misses
> 4263688 ą 4% +24.9% 5325279 ą 4% perf-stat.ps.node-store-misses
> 3.001e+13 +1.7% 3.052e+13 perf-stat.total.instructions
> 18550 -74.9% 4650 ą173% interrupts.76:IR-PCI-MSI.512000-edge.ahci[0000:00:1f.2]
> 7642 ą 9% -20.4% 6086 ą 2% interrupts.CPU0.CAL:Function_call_interrupts
> 4376 ą 22% -75.4% 1077 ą 41% interrupts.CPU0.TLB:TLB_shootdowns
> 8402 ą 5% -19.0% 6806 interrupts.CPU1.CAL:Function_call_interrupts
> 4559 ą 20% -73.7% 1199 ą 15% interrupts.CPU1.TLB:TLB_shootdowns
> 8423 ą 4% -20.2% 6725 ą 2% interrupts.CPU10.CAL:Function_call_interrupts
> 4536 ą 14% -75.0% 1135 ą 20% interrupts.CPU10.TLB:TLB_shootdowns
> 8303 ą 3% -18.2% 6795 ą 2% interrupts.CPU11.CAL:Function_call_interrupts
> 4404 ą 11% -71.6% 1250 ą 35% interrupts.CPU11.TLB:TLB_shootdowns
> 8491 ą 6% -21.3% 6683 interrupts.CPU12.CAL:Function_call_interrupts
> 4723 ą 20% -77.2% 1077 ą 17% interrupts.CPU12.TLB:TLB_shootdowns
> 8403 ą 5% -20.3% 6700 ą 2% interrupts.CPU13.CAL:Function_call_interrupts
> 4557 ą 19% -74.2% 1175 ą 22% interrupts.CPU13.TLB:TLB_shootdowns
> 8459 ą 4% -18.6% 6884 interrupts.CPU14.CAL:Function_call_interrupts
> 4559 ą 18% -69.8% 1376 ą 13% interrupts.CPU14.TLB:TLB_shootdowns
> 8305 ą 7% -17.7% 6833 ą 2% interrupts.CPU15.CAL:Function_call_interrupts
> 4261 ą 25% -67.6% 1382 ą 24% interrupts.CPU15.TLB:TLB_shootdowns
> 8277 ą 5% -19.1% 6696 ą 3% interrupts.CPU16.CAL:Function_call_interrupts
> 4214 ą 22% -69.6% 1282 ą 8% interrupts.CPU16.TLB:TLB_shootdowns
> 8258 ą 5% -18.9% 6694 ą 3% interrupts.CPU17.CAL:Function_call_interrupts
> 4461 ą 19% -74.1% 1155 ą 21% interrupts.CPU17.TLB:TLB_shootdowns
> 8457 ą 6% -20.6% 6717 interrupts.CPU18.CAL:Function_call_interrupts
> 4889 ą 34% +60.0% 7822 interrupts.CPU18.NMI:Non-maskable_interrupts
> 4889 ą 34% +60.0% 7822 interrupts.CPU18.PMI:Performance_monitoring_interrupts
> 4731 ą 22% -77.2% 1078 ą 10% interrupts.CPU18.TLB:TLB_shootdowns
> 8160 ą 5% -18.1% 6684 interrupts.CPU19.CAL:Function_call_interrupts
> 4311 ą 20% -74.2% 1114 ą 13% interrupts.CPU19.TLB:TLB_shootdowns
> 8464 ą 2% -18.2% 6927 ą 3% interrupts.CPU2.CAL:Function_call_interrupts
> 4938 ą 14% -70.5% 1457 ą 18% interrupts.CPU2.TLB:TLB_shootdowns
> 8358 ą 6% -19.7% 6715 ą 3% interrupts.CPU20.CAL:Function_call_interrupts
> 4567 ą 24% -74.6% 1160 ą 35% interrupts.CPU20.TLB:TLB_shootdowns
> 8460 ą 4% -22.3% 6577 ą 2% interrupts.CPU21.CAL:Function_call_interrupts
> 4514 ą 18% -76.0% 1084 ą 22% interrupts.CPU21.TLB:TLB_shootdowns
> 6677 ą 6% +19.6% 7988 ą 9% interrupts.CPU22.CAL:Function_call_interrupts
> 1288 ą 14% +209.1% 3983 ą 35% interrupts.CPU22.TLB:TLB_shootdowns
> 6751 ą 2% +24.0% 8370 ą 9% interrupts.CPU23.CAL:Function_call_interrupts
> 1037 ą 29% +323.0% 4388 ą 36% interrupts.CPU23.TLB:TLB_shootdowns
> 6844 +20.6% 8251 ą 9% interrupts.CPU24.CAL:Function_call_interrupts
> 1205 ą 17% +229.2% 3967 ą 40% interrupts.CPU24.TLB:TLB_shootdowns
> 6880 +21.9% 8389 ą 7% interrupts.CPU25.CAL:Function_call_interrupts
> 1228 ą 19% +245.2% 4240 ą 35% interrupts.CPU25.TLB:TLB_shootdowns
> 6494 ą 8% +25.1% 8123 ą 9% interrupts.CPU26.CAL:Function_call_interrupts
> 1141 ą 13% +262.5% 4139 ą 32% interrupts.CPU26.TLB:TLB_shootdowns
> 6852 +19.2% 8166 ą 7% interrupts.CPU27.CAL:Function_call_interrupts
> 1298 ą 8% +197.1% 3857 ą 31% interrupts.CPU27.TLB:TLB_shootdowns
> 6563 ą 6% +25.2% 8214 ą 8% interrupts.CPU28.CAL:Function_call_interrupts
> 1176 ą 8% +237.1% 3964 ą 33% interrupts.CPU28.TLB:TLB_shootdowns
> 6842 ą 2% +21.4% 8308 ą 8% interrupts.CPU29.CAL:Function_call_interrupts
> 1271 ą 11% +223.8% 4118 ą 33% interrupts.CPU29.TLB:TLB_shootdowns
> 8418 ą 3% -21.1% 6643 ą 2% interrupts.CPU3.CAL:Function_call_interrupts
> 4677 ą 11% -75.1% 1164 ą 16% interrupts.CPU3.TLB:TLB_shootdowns
> 6798 ą 3% +21.8% 8284 ą 7% interrupts.CPU30.CAL:Function_call_interrupts
> 1219 ą 12% +236.3% 4102 ą 30% interrupts.CPU30.TLB:TLB_shootdowns
> 6503 ą 4% +25.9% 8186 ą 6% interrupts.CPU31.CAL:Function_call_interrupts
> 1046 ą 15% +289.1% 4072 ą 32% interrupts.CPU31.TLB:TLB_shootdowns
> 6949 ą 3% +17.2% 8141 ą 8% interrupts.CPU32.CAL:Function_call_interrupts
> 1241 ą 23% +210.6% 3854 ą 34% interrupts.CPU32.TLB:TLB_shootdowns
> 1487 ą 26% +161.6% 3889 ą 46% interrupts.CPU33.TLB:TLB_shootdowns
> 1710 ą 44% +140.1% 4105 ą 36% interrupts.CPU34.TLB:TLB_shootdowns
> 6957 ą 2% +15.2% 8012 ą 9% interrupts.CPU35.CAL:Function_call_interrupts
> 1165 ą 8% +223.1% 3765 ą 38% interrupts.CPU35.TLB:TLB_shootdowns
> 1423 ą 24% +173.4% 3892 ą 33% interrupts.CPU36.TLB:TLB_shootdowns
> 1279 ą 29% +224.2% 4148 ą 39% interrupts.CPU37.TLB:TLB_shootdowns
> 1301 ą 20% +226.1% 4244 ą 35% interrupts.CPU38.TLB:TLB_shootdowns
> 6906 ą 2% +18.5% 8181 ą 8% interrupts.CPU39.CAL:Function_call_interrupts
> 368828 ą 20% +96.2% 723710 ą 7% interrupts.CPU39.RES:Rescheduling_interrupts
> 1438 ą 12% +174.8% 3951 ą 33% interrupts.CPU39.TLB:TLB_shootdowns
> 8399 ą 5% -19.2% 6788 ą 2% interrupts.CPU4.CAL:Function_call_interrupts
> 4567 ą 18% -72.7% 1245 ą 28% interrupts.CPU4.TLB:TLB_shootdowns
> 6895 +22.4% 8439 ą 9% interrupts.CPU40.CAL:Function_call_interrupts
> 1233 ą 11% +247.1% 4280 ą 36% interrupts.CPU40.TLB:TLB_shootdowns
> 6819 ą 2% +21.3% 8274 ą 9% interrupts.CPU41.CAL:Function_call_interrupts
> 1260 ą 14% +207.1% 3871 ą 38% interrupts.CPU41.TLB:TLB_shootdowns
> 1301 ą 9% +204.7% 3963 ą 36% interrupts.CPU42.TLB:TLB_shootdowns
> 6721 ą 3% +22.3% 8221 ą 7% interrupts.CPU43.CAL:Function_call_interrupts
> 1237 ą 19% +224.8% 4017 ą 35% interrupts.CPU43.TLB:TLB_shootdowns
> 8422 ą 8% -22.7% 6506 ą 5% interrupts.CPU44.CAL:Function_call_interrupts
> 15261375 ą 7% -7.8% 14064176 interrupts.CPU44.LOC:Local_timer_interrupts
> 4376 ą 25% -75.7% 1063 ą 26% interrupts.CPU44.TLB:TLB_shootdowns
> 8451 ą 5% -23.7% 6448 ą 6% interrupts.CPU45.CAL:Function_call_interrupts
> 4351 ą 18% -74.9% 1094 ą 12% interrupts.CPU45.TLB:TLB_shootdowns
> 8705 ą 6% -21.2% 6860 ą 2% interrupts.CPU46.CAL:Function_call_interrupts
> 4787 ą 20% -69.5% 1462 ą 16% interrupts.CPU46.TLB:TLB_shootdowns
> 8334 ą 3% -18.9% 6763 interrupts.CPU47.CAL:Function_call_interrupts
> 4126 ą 10% -71.3% 1186 ą 18% interrupts.CPU47.TLB:TLB_shootdowns
> 8578 ą 4% -21.7% 6713 interrupts.CPU48.CAL:Function_call_interrupts
> 4520 ą 15% -74.5% 1154 ą 23% interrupts.CPU48.TLB:TLB_shootdowns
> 8450 ą 8% -18.8% 6863 ą 3% interrupts.CPU49.CAL:Function_call_interrupts
> 4494 ą 24% -66.5% 1505 ą 22% interrupts.CPU49.TLB:TLB_shootdowns
> 8307 ą 4% -18.0% 6816 ą 2% interrupts.CPU5.CAL:Function_call_interrupts
> 7845 -37.4% 4908 ą 34% interrupts.CPU5.NMI:Non-maskable_interrupts
> 7845 -37.4% 4908 ą 34% interrupts.CPU5.PMI:Performance_monitoring_interrupts
> 4429 ą 17% -69.8% 1339 ą 20% interrupts.CPU5.TLB:TLB_shootdowns
> 8444 ą 4% -21.7% 6613 interrupts.CPU50.CAL:Function_call_interrupts
> 4282 ą 16% -76.0% 1029 ą 17% interrupts.CPU50.TLB:TLB_shootdowns
> 8750 ą 6% -22.2% 6803 interrupts.CPU51.CAL:Function_call_interrupts
> 4755 ą 20% -73.1% 1277 ą 15% interrupts.CPU51.TLB:TLB_shootdowns
> 8478 ą 6% -20.2% 6766 ą 2% interrupts.CPU52.CAL:Function_call_interrupts
> 4337 ą 20% -72.6% 1190 ą 22% interrupts.CPU52.TLB:TLB_shootdowns
> 8604 ą 7% -21.5% 6750 ą 4% interrupts.CPU53.CAL:Function_call_interrupts
> 4649 ą 17% -74.3% 1193 ą 23% interrupts.CPU53.TLB:TLB_shootdowns
> 8317 ą 9% -19.4% 6706 ą 3% interrupts.CPU54.CAL:Function_call_interrupts
> 4372 ą 12% -75.4% 1076 ą 29% interrupts.CPU54.TLB:TLB_shootdowns
> 8439 ą 3% -18.5% 6876 interrupts.CPU55.CAL:Function_call_interrupts
> 4415 ą 11% -71.6% 1254 ą 17% interrupts.CPU55.TLB:TLB_shootdowns
> 8869 ą 6% -22.6% 6864 ą 2% interrupts.CPU56.CAL:Function_call_interrupts
> 517594 ą 13% +123.3% 1155539 ą 25% interrupts.CPU56.RES:Rescheduling_interrupts
> 5085 ą 22% -74.9% 1278 ą 17% interrupts.CPU56.TLB:TLB_shootdowns
> 8682 ą 4% -21.7% 6796 ą 2% interrupts.CPU57.CAL:Function_call_interrupts
> 4808 ą 19% -74.1% 1243 ą 13% interrupts.CPU57.TLB:TLB_shootdowns
> 8626 ą 7% -21.8% 6746 ą 2% interrupts.CPU58.CAL:Function_call_interrupts
> 4816 ą 20% -79.1% 1007 ą 28% interrupts.CPU58.TLB:TLB_shootdowns
> 8759 ą 8% -20.3% 6984 interrupts.CPU59.CAL:Function_call_interrupts
> 4840 ą 22% -70.6% 1423 ą 14% interrupts.CPU59.TLB:TLB_shootdowns
> 8167 ą 6% -19.0% 6615 ą 2% interrupts.CPU6.CAL:Function_call_interrupts
> 4129 ą 21% -75.4% 1017 ą 24% interrupts.CPU6.TLB:TLB_shootdowns
> 8910 ą 4% -23.7% 6794 ą 3% interrupts.CPU60.CAL:Function_call_interrupts
> 5017 ą 12% -77.8% 1113 ą 15% interrupts.CPU60.TLB:TLB_shootdowns
> 8689 ą 5% -21.6% 6808 interrupts.CPU61.CAL:Function_call_interrupts
> 4715 ą 20% -77.6% 1055 ą 19% interrupts.CPU61.TLB:TLB_shootdowns
> 8574 ą 4% -18.9% 6953 ą 2% interrupts.CPU62.CAL:Function_call_interrupts
> 4494 ą 17% -72.3% 1244 ą 7% interrupts.CPU62.TLB:TLB_shootdowns
> 8865 ą 3% -25.4% 6614 ą 7% interrupts.CPU63.CAL:Function_call_interrupts
> 4870 ą 12% -76.8% 1130 ą 12% interrupts.CPU63.TLB:TLB_shootdowns
> 8724 ą 7% -20.2% 6958 ą 3% interrupts.CPU64.CAL:Function_call_interrupts
> 4736 ą 16% -72.6% 1295 ą 7% interrupts.CPU64.TLB:TLB_shootdowns
> 8717 ą 6% -23.7% 6653 ą 4% interrupts.CPU65.CAL:Function_call_interrupts
> 4626 ą 19% -76.5% 1087 ą 21% interrupts.CPU65.TLB:TLB_shootdowns
> 6671 +24.7% 8318 ą 9% interrupts.CPU66.CAL:Function_call_interrupts
> 1091 ą 8% +249.8% 3819 ą 32% interrupts.CPU66.TLB:TLB_shootdowns
> 6795 ą 2% +26.9% 8624 ą 9% interrupts.CPU67.CAL:Function_call_interrupts
> 1098 ą 24% +299.5% 4388 ą 39% interrupts.CPU67.TLB:TLB_shootdowns
> 6704 ą 5% +25.8% 8431 ą 8% interrupts.CPU68.CAL:Function_call_interrupts
> 1214 ą 15% +236.1% 4083 ą 36% interrupts.CPU68.TLB:TLB_shootdowns
> 1049 ą 15% +326.2% 4473 ą 33% interrupts.CPU69.TLB:TLB_shootdowns
> 8554 ą 6% -19.6% 6874 ą 2% interrupts.CPU7.CAL:Function_call_interrupts
> 4753 ą 19% -71.7% 1344 ą 16% interrupts.CPU7.TLB:TLB_shootdowns
> 1298 ą 13% +227.4% 4249 ą 38% interrupts.CPU70.TLB:TLB_shootdowns
> 6976 +19.9% 8362 ą 7% interrupts.CPU71.CAL:Function_call_interrupts
> 1232748 ą 18% -57.3% 525824 ą 33% interrupts.CPU71.RES:Rescheduling_interrupts
> 1253 ą 9% +211.8% 3909 ą 31% interrupts.CPU71.TLB:TLB_shootdowns
> 1316 ą 22% +188.7% 3800 ą 33% interrupts.CPU72.TLB:TLB_shootdowns
> 6665 ą 5% +26.5% 8429 ą 8% interrupts.CPU73.CAL:Function_call_interrupts
> 1202 ą 13% +234.1% 4017 ą 37% interrupts.CPU73.TLB:TLB_shootdowns
> 6639 ą 5% +27.0% 8434 ą 8% interrupts.CPU74.CAL:Function_call_interrupts
> 1079 ą 16% +269.4% 3986 ą 36% interrupts.CPU74.TLB:TLB_shootdowns
> 1055 ą 12% +301.2% 4235 ą 34% interrupts.CPU75.TLB:TLB_shootdowns
> 7011 ą 3% +21.6% 8522 ą 8% interrupts.CPU76.CAL:Function_call_interrupts
> 1223 ą 13% +230.7% 4047 ą 35% interrupts.CPU76.TLB:TLB_shootdowns
> 6886 ą 7% +25.6% 8652 ą 10% interrupts.CPU77.CAL:Function_call_interrupts
> 1316 ą 16% +229.8% 4339 ą 36% interrupts.CPU77.TLB:TLB_shootdowns
> 7343 ą 5% +19.1% 8743 ą 9% interrupts.CPU78.CAL:Function_call_interrupts
> 1699 ą 37% +144.4% 4152 ą 31% interrupts.CPU78.TLB:TLB_shootdowns
> 7136 ą 4% +21.4% 8666 ą 9% interrupts.CPU79.CAL:Function_call_interrupts
> 1094 ą 13% +276.2% 4118 ą 34% interrupts.CPU79.TLB:TLB_shootdowns
> 8531 ą 5% -19.5% 6869 ą 2% interrupts.CPU8.CAL:Function_call_interrupts
> 4764 ą 16% -71.0% 1382 ą 14% interrupts.CPU8.TLB:TLB_shootdowns
> 1387 ą 29% +181.8% 3910 ą 38% interrupts.CPU80.TLB:TLB_shootdowns
> 1114 ą 30% +259.7% 4007 ą 36% interrupts.CPU81.TLB:TLB_shootdowns
> 7012 +23.9% 8685 ą 8% interrupts.CPU82.CAL:Function_call_interrupts
> 1274 ą 12% +255.4% 4530 ą 27% interrupts.CPU82.TLB:TLB_shootdowns
> 6971 ą 3% +23.8% 8628 ą 9% interrupts.CPU83.CAL:Function_call_interrupts
> 1156 ą 18% +260.1% 4162 ą 34% interrupts.CPU83.TLB:TLB_shootdowns
> 7030 ą 4% +21.0% 8504 ą 8% interrupts.CPU84.CAL:Function_call_interrupts
> 1286 ą 23% +224.0% 4166 ą 31% interrupts.CPU84.TLB:TLB_shootdowns
> 7059 +22.4% 8644 ą 11% interrupts.CPU85.CAL:Function_call_interrupts
> 1421 ą 22% +208.8% 4388 ą 33% interrupts.CPU85.TLB:TLB_shootdowns
> 7018 ą 2% +22.8% 8615 ą 9% interrupts.CPU86.CAL:Function_call_interrupts
> 1258 ą 8% +231.1% 4167 ą 34% interrupts.CPU86.TLB:TLB_shootdowns
> 1338 ą 3% +217.9% 4255 ą 31% interrupts.CPU87.TLB:TLB_shootdowns
> 8376 ą 4% -19.0% 6787 ą 2% interrupts.CPU9.CAL:Function_call_interrupts
> 4466 ą 17% -71.2% 1286 ą 18% interrupts.CPU9.TLB:TLB_shootdowns
>
>
>
>
>
> Disclaimer:
> Results have been estimated based on internal Intel analysis and are provided
> for informational purposes only. Any difference in system hardware or software
> design or configuration may affect actual performance.
>
>
> Thanks,
> Oliver Sang
>
Powered by blists - more mailing lists