lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Message-ID: <20210520052511.GA14190@xsang-OptiPlex-9020>
Date:   Thu, 20 May 2021 13:25:11 +0800
From:   kernel test robot <oliver.sang@...el.com>
To:     Peter Zijlstra <peterz@...radead.org>
Cc:     Greg Kroah-Hartman <gregkh@...uxfoundation.org>,
        Valentin Schneider <valentin.schneider@....com>,
        LKML <linux-kernel@...r.kernel.org>, lkp@...ts.01.org,
        lkp@...el.com, ying.huang@...el.com, feng.tang@...el.com,
        zhengjun.xing@...el.com, aubrey.li@...ux.intel.com,
        yu.c.chen@...el.com
Subject: [sched]  d27e9ae2f2:  aim9.sync_disk_rw.ops_per_sec -2.1% regression



Greeting,

FYI, we noticed a -2.1% regression of aim9.sync_disk_rw.ops_per_sec due to commit:


commit: d27e9ae2f244805bbdc730d85fba28685d2471e5 ("sched: Move /proc/sched_debug to debugfs")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master


in testcase: aim9
on test machine: 256 threads 2 sockets Intel(R) Genuine Intel(R) CPU 0000 @ 1.30GHz with 112G memory
with following parameters:

	testtime: 300s
	test: sync_disk_rw
	cpufreq_governor: performance
	ucode: 0xffff0190

test-description: Suite IX is the "AIM Independent Resource Benchmark:" the famous synthetic benchmark.
test-url: https://sourceforge.net/projects/aimbench/files/aim-suite9/

In addition to that, the commit also has significant impact on the following tests:

+------------------+--------------------------------------------------------------------------------+
| testcase: change | stress-ng: boot-time.dhcp 7.3% regression                                      |
| test machine     | 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 512G memory |
| test parameters  | class=filesystem                                                               |
|                  | cpufreq_governor=performance                                                   |
|                  | disk=1HDD                                                                      |
|                  | fs=ext4                                                                        |
|                  | nr_threads=10%                                                                 |
|                  | test=procfs                                                                    |
|                  | testtime=60s                                                                   |
|                  | ucode=0x5003006                                                                |
+------------------+--------------------------------------------------------------------------------+


If you fix the issue, kindly add following tag
Reported-by: kernel test robot <oliver.sang@...el.com>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install                job.yaml  # job file is attached in this email
        bin/lkp split-job --compatible job.yaml  # generate the yaml file for lkp run
        bin/lkp run                    generated-yaml-file

=========================================================================================
compiler/cpufreq_governor/kconfig/rootfs/tbox_group/test/testcase/testtime/ucode:
  gcc-9/performance/x86_64-rhel-8.3/debian-10.4-x86_64-20200603.cgz/lkp-knl-f1/sync_disk_rw/aim9/300s/0xffff0190

commit: 
  3b87f136f8 ("sched,debug: Convert sysctl sched_domains to debugfs")
  d27e9ae2f2 ("sched: Move /proc/sched_debug to debugfs")

3b87f136f8fccddf d27e9ae2f244805bbdc730d85fb 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
    100217            -2.1%      98162        aim9.sync_disk_rw.ops_per_sec
      0.08 ± 46%     -47.4%       0.04 ±  4%  perf-sched.sch_delay.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
     17079 ± 40%     +56.8%      26788        softirqs.CPU144.SCHED
     63033            -0.7%      62586        proc-vmstat.nr_anon_pages
      1059            -1.4%       1044        proc-vmstat.nr_page_table_pages
      3019 ±  3%     +27.9%       3861        proc-vmstat.numa_hint_faults
      3019 ±  3%     +27.9%       3861        proc-vmstat.numa_hint_faults_local
   8637459            -1.8%    8481509        proc-vmstat.numa_hit
   8637455            -1.8%    8481506        proc-vmstat.numa_local
    499305            -2.0%     489390        proc-vmstat.pgactivate
   8726852            -1.9%    8562994        proc-vmstat.pgalloc_normal
   8701259            -1.9%    8537254        proc-vmstat.pgfree
      0.47 ±  7%      -0.1        0.39 ±  8%  perf-profile.children.cycles-pp.tick_program_event
      0.88 ±  2%      -0.0        0.84 ±  2%  perf-profile.children.cycles-pp.__x86_retpoline_rax
      0.08 ± 24%      -0.0        0.05 ± 48%  perf-profile.children.cycles-pp.update_load_avg
      0.14 ±  9%      -0.0        0.12 ±  9%  perf-profile.children.cycles-pp.profile_tick
      0.46 ±  7%      -0.1        0.39 ±  9%  perf-profile.self.cycles-pp.tick_program_event
      0.40 ± 20%      -0.1        0.33 ±  7%  perf-profile.self.cycles-pp.rb_insert_color
      0.14 ±  9%      -0.0        0.12 ±  8%  perf-profile.self.cycles-pp.profile_tick
      0.11 ±  3%      +0.0        0.13 ±  8%  perf-profile.self.cycles-pp.restore_regs_and_return_to_kernel
     45.91 ±223%    -100.0%       0.00        sched_debug.cfs_rq:/.MIN_vruntime.avg
     10421 ±223%    -100.0%       0.00        sched_debug.cfs_rq:/.MIN_vruntime.max
      0.00          -100.0%       0.00        sched_debug.cfs_rq:/.MIN_vruntime.min
     18174 ± 25%    -100.0%       0.00        sched_debug.cfs_rq:/.load.avg
    458000 ± 17%    -100.0%       0.00        sched_debug.cfs_rq:/.load.max
     81111 ± 13%    -100.0%       0.00        sched_debug.cfs_rq:/.load.stddev
     16.00 ± 14%    -100.0%       0.00        sched_debug.cfs_rq:/.load_avg.avg
    477.53 ± 11%    -100.0%       0.00        sched_debug.cfs_rq:/.load_avg.max
     76.27 ±  5%    -100.0%       0.00        sched_debug.cfs_rq:/.load_avg.stddev
     45.91 ±223%    -100.0%       0.00        sched_debug.cfs_rq:/.max_vruntime.avg
     10421 ±223%    -100.0%       0.00        sched_debug.cfs_rq:/.max_vruntime.max
      0.00          -100.0%       0.00        sched_debug.cfs_rq:/.max_vruntime.min
    364404 ± 13%    -100.0%       0.00        sched_debug.cfs_rq:/.min_vruntime.avg
    445882 ± 13%    -100.0%       0.00        sched_debug.cfs_rq:/.min_vruntime.max
    114201 ± 31%    -100.0%       0.00        sched_debug.cfs_rq:/.min_vruntime.min
     52485 ± 14%    -100.0%       0.00        sched_debug.cfs_rq:/.min_vruntime.stddev
      0.05 ± 25%    -100.0%       0.00        sched_debug.cfs_rq:/.nr_running.avg
      1.00          -100.0%       0.00        sched_debug.cfs_rq:/.nr_running.max
      0.21 ± 11%    -100.0%       0.00        sched_debug.cfs_rq:/.nr_running.stddev
     43.91 ± 13%    -100.0%       0.00        sched_debug.cfs_rq:/.runnable_avg.avg
      1025          -100.0%       0.00        sched_debug.cfs_rq:/.runnable_avg.max
    186.43 ±  5%    -100.0%       0.00        sched_debug.cfs_rq:/.runnable_avg.stddev
    115058 ± 48%    -100.0%       0.00        sched_debug.cfs_rq:/.spread0.avg
    196547 ± 26%    -100.0%       0.00        sched_debug.cfs_rq:/.spread0.max
   -135131          -100.0%       0.00        sched_debug.cfs_rq:/.spread0.min
     52485 ± 14%    -100.0%       0.00        sched_debug.cfs_rq:/.spread0.stddev
     43.82 ± 13%    -100.0%       0.00        sched_debug.cfs_rq:/.util_avg.avg
      1023          -100.0%       0.00        sched_debug.cfs_rq:/.util_avg.max
    186.23 ±  5%    -100.0%       0.00        sched_debug.cfs_rq:/.util_avg.stddev
     13.43 ± 28%    -100.0%       0.00        sched_debug.cfs_rq:/.util_est_enqueued.avg
    675.72 ±  6%    -100.0%       0.00        sched_debug.cfs_rq:/.util_est_enqueued.max
     87.51 ± 12%    -100.0%       0.00        sched_debug.cfs_rq:/.util_est_enqueued.stddev
   1097307 ±  2%    -100.0%       0.00        sched_debug.cpu.avg_idle.avg
   2935820 ± 14%    -100.0%       0.00        sched_debug.cpu.avg_idle.max
    349655 ± 30%    -100.0%       0.00        sched_debug.cpu.avg_idle.min
    267237 ± 10%    -100.0%       0.00        sched_debug.cpu.avg_idle.stddev
    207137 ±  4%    -100.0%       0.00        sched_debug.cpu.clock.avg
    207324 ±  4%    -100.0%       0.00        sched_debug.cpu.clock.max
    206944 ±  4%    -100.0%       0.00        sched_debug.cpu.clock.min
    108.63          -100.0%       0.00        sched_debug.cpu.clock.stddev
    200958 ±  4%    -100.0%       0.00        sched_debug.cpu.clock_task.avg
    202840 ±  4%    -100.0%       0.00        sched_debug.cpu.clock_task.max
    196364 ±  4%    -100.0%       0.00        sched_debug.cpu.clock_task.min
    832.94 ±  5%    -100.0%       0.00        sched_debug.cpu.clock_task.stddev
     68.56 ± 19%    -100.0%       0.00        sched_debug.cpu.curr->pid.avg
      7076 ±  3%    -100.0%       0.00        sched_debug.cpu.curr->pid.max
    597.52 ±  8%    -100.0%       0.00        sched_debug.cpu.curr->pid.stddev
    564484 ±  2%    -100.0%       0.00        sched_debug.cpu.max_idle_balance_cost.avg
   1462314 ± 14%    -100.0%       0.00        sched_debug.cpu.max_idle_balance_cost.max
    500000          -100.0%       0.00        sched_debug.cpu.max_idle_balance_cost.min
    123218 ± 12%    -100.0%       0.00        sched_debug.cpu.max_idle_balance_cost.stddev
      4294          -100.0%       0.00        sched_debug.cpu.next_balance.avg
      4294          -100.0%       0.00        sched_debug.cpu.next_balance.max
      4294          -100.0%       0.00        sched_debug.cpu.next_balance.min
      0.00          -100.0%       0.00        sched_debug.cpu.next_balance.stddev
      0.01 ± 18%    -100.0%       0.00        sched_debug.cpu.nr_running.avg
      1.00          -100.0%       0.00        sched_debug.cpu.nr_running.max
      0.12 ±  7%    -100.0%       0.00        sched_debug.cpu.nr_running.stddev
      3047 ±  3%    -100.0%       0.00        sched_debug.cpu.nr_switches.avg
     31444 ± 23%    -100.0%       0.00        sched_debug.cpu.nr_switches.max
      1033 ±  8%    -100.0%       0.00        sched_debug.cpu.nr_switches.min
      3226 ± 13%    -100.0%       0.00        sched_debug.cpu.nr_switches.stddev
    213.83          -100.0%       0.00        sched_debug.cpu.nr_uninterruptible.max
    -28.70          -100.0%       0.00        sched_debug.cpu.nr_uninterruptible.min
     14.04 ±  2%    -100.0%       0.00        sched_debug.cpu.nr_uninterruptible.stddev
    206943 ±  4%    -100.0%       0.00        sched_debug.cpu_clk
    996147          -100.0%       0.00        sched_debug.dl_rq:.dl_bw->bw.avg
    996147          -100.0%       0.00        sched_debug.dl_rq:.dl_bw->bw.max
    996147          -100.0%       0.00        sched_debug.dl_rq:.dl_bw->bw.min
 4.295e+09          -100.0%       0.00        sched_debug.jiffies
    206285 ±  4%    -100.0%       0.00        sched_debug.ktime
    950.00          -100.0%       0.00        sched_debug.rt_rq:/.rt_runtime.avg
    950.00          -100.0%       0.00        sched_debug.rt_rq:/.rt_runtime.max
    950.00          -100.0%       0.00        sched_debug.rt_rq:/.rt_runtime.min
    207759 ±  4%    -100.0%       0.00        sched_debug.sched_clk
      1.00          -100.0%       0.00        sched_debug.sched_clock_stable()
   4140603          -100.0%       0.00        sched_debug.sysctl_sched.sysctl_sched_features
     24.00          -100.0%       0.00        sched_debug.sysctl_sched.sysctl_sched_latency
      3.00          -100.0%       0.00        sched_debug.sysctl_sched.sysctl_sched_min_granularity
      1.00          -100.0%       0.00        sched_debug.sysctl_sched.sysctl_sched_tunable_scaling
      4.00          -100.0%       0.00        sched_debug.sysctl_sched.sysctl_sched_wakeup_granularity
    614734           -42.7%     352291        interrupts.CAL:Function_call_interrupts
      2370 ±  5%     -65.9%     807.17 ±  6%  interrupts.CPU0.CAL:Function_call_interrupts
      2591 ± 13%     -40.3%       1547 ± 24%  interrupts.CPU1.CAL:Function_call_interrupts
      2576 ± 18%     -36.9%       1625 ±  8%  interrupts.CPU10.CAL:Function_call_interrupts
      2331 ±  5%     -41.4%       1366 ± 20%  interrupts.CPU100.CAL:Function_call_interrupts
      2277           -44.7%       1258 ±  4%  interrupts.CPU101.CAL:Function_call_interrupts
      2299           -44.1%       1285 ±  3%  interrupts.CPU102.CAL:Function_call_interrupts
      2300           -43.9%       1290 ±  8%  interrupts.CPU104.CAL:Function_call_interrupts
      2297 ±  3%     -45.7%       1247 ±  3%  interrupts.CPU105.CAL:Function_call_interrupts
      2343 ±  4%     -39.2%       1423 ± 19%  interrupts.CPU106.CAL:Function_call_interrupts
      2283 ±  2%     -39.0%       1393 ± 17%  interrupts.CPU107.CAL:Function_call_interrupts
      2255           -43.0%       1286 ±  8%  interrupts.CPU109.CAL:Function_call_interrupts
      2368 ±  6%     -45.6%       1289 ±  3%  interrupts.CPU11.CAL:Function_call_interrupts
      2305           -43.4%       1306 ±  6%  interrupts.CPU110.CAL:Function_call_interrupts
      2267           -43.9%       1272 ±  6%  interrupts.CPU111.CAL:Function_call_interrupts
      2276 ±  3%     -46.2%       1224        interrupts.CPU113.CAL:Function_call_interrupts
      2304 ±  2%     -45.4%       1257 ±  3%  interrupts.CPU114.CAL:Function_call_interrupts
      2245           -43.2%       1274 ±  9%  interrupts.CPU115.CAL:Function_call_interrupts
    188.33 ± 17%     +61.9%     305.00 ± 30%  interrupts.CPU115.NMI:Non-maskable_interrupts
    188.33 ± 17%     +61.9%     305.00 ± 30%  interrupts.CPU115.PMI:Performance_monitoring_interrupts
      2518 ± 21%     -49.3%       1277 ±  6%  interrupts.CPU116.CAL:Function_call_interrupts
    197.83 ± 17%     +63.3%     323.00 ± 29%  interrupts.CPU116.NMI:Non-maskable_interrupts
    197.83 ± 17%     +63.3%     323.00 ± 29%  interrupts.CPU116.PMI:Performance_monitoring_interrupts
      2267           -36.8%       1433 ± 23%  interrupts.CPU117.CAL:Function_call_interrupts
      2358 ± 11%     -45.8%       1277 ±  2%  interrupts.CPU118.CAL:Function_call_interrupts
      2655 ± 15%     -52.1%       1270 ±  5%  interrupts.CPU119.CAL:Function_call_interrupts
      2370 ±  7%     -45.0%       1303 ±  2%  interrupts.CPU120.CAL:Function_call_interrupts
      2253           -43.0%       1283 ±  6%  interrupts.CPU121.CAL:Function_call_interrupts
      2255           -43.1%       1284 ±  3%  interrupts.CPU122.CAL:Function_call_interrupts
      2254           -45.0%       1240 ±  2%  interrupts.CPU123.CAL:Function_call_interrupts
      2373 ±  8%     -43.9%       1331 ±  4%  interrupts.CPU124.CAL:Function_call_interrupts
      2256           -42.7%       1292 ±  9%  interrupts.CPU125.CAL:Function_call_interrupts
      2443 ±  5%     -46.9%       1297 ±  3%  interrupts.CPU126.CAL:Function_call_interrupts
    199.83 ± 16%     +64.5%     328.67 ± 32%  interrupts.CPU126.NMI:Non-maskable_interrupts
    199.83 ± 16%     +64.5%     328.67 ± 32%  interrupts.CPU126.PMI:Performance_monitoring_interrupts
      2285 ±  2%     -41.4%       1339 ± 11%  interrupts.CPU127.CAL:Function_call_interrupts
      2243           -42.9%       1280 ± 10%  interrupts.CPU128.CAL:Function_call_interrupts
      2235           -42.7%       1280 ± 10%  interrupts.CPU129.CAL:Function_call_interrupts
      2252           -45.2%       1234        interrupts.CPU130.CAL:Function_call_interrupts
      2269 ±  2%     -46.0%       1226        interrupts.CPU131.CAL:Function_call_interrupts
      2254           -42.4%       1299 ±  6%  interrupts.CPU132.CAL:Function_call_interrupts
      2239           -44.9%       1234        interrupts.CPU133.CAL:Function_call_interrupts
      2260           -45.6%       1229        interrupts.CPU134.CAL:Function_call_interrupts
      2237           -44.6%       1239 ±  2%  interrupts.CPU135.CAL:Function_call_interrupts
      2291 ±  3%     -46.6%       1224        interrupts.CPU136.CAL:Function_call_interrupts
      2304 ±  4%     -47.3%       1213        interrupts.CPU137.CAL:Function_call_interrupts
      2245           -45.6%       1222        interrupts.CPU138.CAL:Function_call_interrupts
      2238           -45.3%       1224        interrupts.CPU139.CAL:Function_call_interrupts
      2649 ±  3%     -37.4%       1659 ± 23%  interrupts.CPU14.CAL:Function_call_interrupts
      2320 ±  6%     -45.6%       1263 ±  6%  interrupts.CPU140.CAL:Function_call_interrupts
      2462 ± 19%     -50.4%       1221        interrupts.CPU141.CAL:Function_call_interrupts
      2250           -42.5%       1295 ± 10%  interrupts.CPU142.CAL:Function_call_interrupts
      2262 ±  2%     -44.7%       1251 ±  3%  interrupts.CPU143.CAL:Function_call_interrupts
      2255           -45.5%       1228        interrupts.CPU144.CAL:Function_call_interrupts
      2247           -45.1%       1234        interrupts.CPU145.CAL:Function_call_interrupts
      2258           -45.6%       1227        interrupts.CPU146.CAL:Function_call_interrupts
      2237           -45.2%       1226        interrupts.CPU147.CAL:Function_call_interrupts
      2244           -41.9%       1304 ± 11%  interrupts.CPU148.CAL:Function_call_interrupts
      2343 ±  4%     -45.9%       1267 ±  6%  interrupts.CPU149.CAL:Function_call_interrupts
      2343 ±  2%     -41.7%       1366 ±  2%  interrupts.CPU15.CAL:Function_call_interrupts
      2266           -38.1%       1402 ± 24%  interrupts.CPU150.CAL:Function_call_interrupts
      2262           -46.3%       1215        interrupts.CPU151.CAL:Function_call_interrupts
      2280           -45.3%       1247 ±  2%  interrupts.CPU152.CAL:Function_call_interrupts
      2262           -43.9%       1268 ±  8%  interrupts.CPU153.CAL:Function_call_interrupts
      2481 ± 22%     -50.5%       1227        interrupts.CPU154.CAL:Function_call_interrupts
      2236           -41.1%       1318 ± 12%  interrupts.CPU155.CAL:Function_call_interrupts
      2284 ±  3%     -46.6%       1219        interrupts.CPU156.CAL:Function_call_interrupts
      2252           -43.5%       1272 ±  8%  interrupts.CPU157.CAL:Function_call_interrupts
      2250           -45.1%       1234        interrupts.CPU158.CAL:Function_call_interrupts
      2243           -44.7%       1239 ±  3%  interrupts.CPU159.CAL:Function_call_interrupts
      2683 ±  4%     -37.0%       1689 ± 12%  interrupts.CPU16.CAL:Function_call_interrupts
      2131 ± 19%     -42.0%       1235        interrupts.CPU160.CAL:Function_call_interrupts
      2254           -45.8%       1221        interrupts.CPU161.CAL:Function_call_interrupts
      2246           -45.0%       1235 ±  2%  interrupts.CPU162.CAL:Function_call_interrupts
      2251           -45.7%       1221        interrupts.CPU163.CAL:Function_call_interrupts
      2246           -41.4%       1315 ±  9%  interrupts.CPU164.CAL:Function_call_interrupts
      2253           -39.9%       1354 ± 13%  interrupts.CPU165.CAL:Function_call_interrupts
      2260           -45.5%       1231        interrupts.CPU166.CAL:Function_call_interrupts
      2241           -45.7%       1217        interrupts.CPU167.CAL:Function_call_interrupts
      2247           -43.9%       1261 ±  4%  interrupts.CPU168.CAL:Function_call_interrupts
      2249           -45.1%       1235 ±  2%  interrupts.CPU169.CAL:Function_call_interrupts
      2381 ±  2%     -43.8%       1338 ±  8%  interrupts.CPU17.CAL:Function_call_interrupts
      2273           -44.7%       1256 ±  3%  interrupts.CPU170.CAL:Function_call_interrupts
      2252           -45.8%       1221        interrupts.CPU171.CAL:Function_call_interrupts
      2271 ±  2%     -43.6%       1280 ±  6%  interrupts.CPU172.CAL:Function_call_interrupts
      2240           -45.4%       1223        interrupts.CPU173.CAL:Function_call_interrupts
      2260           -44.4%       1257 ±  5%  interrupts.CPU174.CAL:Function_call_interrupts
      2242           -45.1%       1231 ±  3%  interrupts.CPU175.CAL:Function_call_interrupts
      2242           -45.5%       1221        interrupts.CPU176.CAL:Function_call_interrupts
      2267           -45.7%       1231 ±  2%  interrupts.CPU177.CAL:Function_call_interrupts
      2239           -44.2%       1249        interrupts.CPU178.CAL:Function_call_interrupts
      2097 ± 18%     -40.4%       1250 ±  3%  interrupts.CPU179.CAL:Function_call_interrupts
      2633 ±  2%     -40.1%       1578 ±  8%  interrupts.CPU18.CAL:Function_call_interrupts
      2307 ±  4%     -45.6%       1256 ±  5%  interrupts.CPU180.CAL:Function_call_interrupts
      2168 ± 11%     -43.4%       1227        interrupts.CPU181.CAL:Function_call_interrupts
      2297 ±  3%     -46.9%       1220        interrupts.CPU182.CAL:Function_call_interrupts
      2327 ±  5%     -47.6%       1219        interrupts.CPU183.CAL:Function_call_interrupts
      2311 ±  2%     -47.0%       1225        interrupts.CPU184.CAL:Function_call_interrupts
      2271 ±  3%     -46.4%       1218        interrupts.CPU185.CAL:Function_call_interrupts
      2244           -44.6%       1243 ±  2%  interrupts.CPU186.CAL:Function_call_interrupts
      2145 ± 10%     -41.2%       1261 ±  6%  interrupts.CPU187.CAL:Function_call_interrupts
      2346 ±  7%     -47.6%       1228        interrupts.CPU188.CAL:Function_call_interrupts
      2157 ±  9%     -43.3%       1223        interrupts.CPU189.CAL:Function_call_interrupts
      2371 ±  5%     -45.0%       1304 ±  3%  interrupts.CPU19.CAL:Function_call_interrupts
      2269           -45.7%       1233        interrupts.CPU190.CAL:Function_call_interrupts
      2258           -44.9%       1245 ±  2%  interrupts.CPU191.CAL:Function_call_interrupts
      2234           -45.1%       1225        interrupts.CPU192.CAL:Function_call_interrupts
      2430 ± 17%     -49.9%       1218        interrupts.CPU193.CAL:Function_call_interrupts
      2265 ±  2%     -46.1%       1220        interrupts.CPU194.CAL:Function_call_interrupts
      2330 ±  4%     -47.3%       1228        interrupts.CPU195.CAL:Function_call_interrupts
      2272 ±  2%     -44.3%       1265 ±  7%  interrupts.CPU196.CAL:Function_call_interrupts
      2259           -45.5%       1232 ±  2%  interrupts.CPU197.CAL:Function_call_interrupts
      2249           -45.6%       1223        interrupts.CPU198.CAL:Function_call_interrupts
      2243           -42.8%       1284 ±  5%  interrupts.CPU199.CAL:Function_call_interrupts
      2648 ±  6%     -35.8%       1700 ± 15%  interrupts.CPU2.CAL:Function_call_interrupts
      2687 ±  6%     -39.9%       1616 ± 14%  interrupts.CPU20.CAL:Function_call_interrupts
      2358 ±  8%     -48.4%       1216        interrupts.CPU200.CAL:Function_call_interrupts
      2289 ±  4%     -41.7%       1335 ± 12%  interrupts.CPU201.CAL:Function_call_interrupts
      2291 ±  4%     -46.3%       1230 ±  2%  interrupts.CPU202.CAL:Function_call_interrupts
      2268 ±  2%     -49.6%       1143 ± 16%  interrupts.CPU203.CAL:Function_call_interrupts
      2251           -42.3%       1299 ± 10%  interrupts.CPU204.CAL:Function_call_interrupts
      2250           -43.5%       1272 ±  7%  interrupts.CPU205.CAL:Function_call_interrupts
      2252           -43.0%       1284 ±  8%  interrupts.CPU206.CAL:Function_call_interrupts
      2252           -45.5%       1226        interrupts.CPU207.CAL:Function_call_interrupts
      2257           -45.8%       1222 ±  2%  interrupts.CPU208.CAL:Function_call_interrupts
      2261           -42.8%       1293 ±  7%  interrupts.CPU209.CAL:Function_call_interrupts
      2376 ±  3%     -42.3%       1371 ±  5%  interrupts.CPU21.CAL:Function_call_interrupts
      2240           -45.6%       1219        interrupts.CPU210.CAL:Function_call_interrupts
      2252           -45.5%       1228        interrupts.CPU211.CAL:Function_call_interrupts
      2322 ±  6%     -46.9%       1232        interrupts.CPU212.CAL:Function_call_interrupts
      2260           -45.2%       1237        interrupts.CPU213.CAL:Function_call_interrupts
      2248           -45.3%       1231        interrupts.CPU214.CAL:Function_call_interrupts
      2247           -44.6%       1245 ±  3%  interrupts.CPU215.CAL:Function_call_interrupts
      2271           -45.5%       1237        interrupts.CPU216.CAL:Function_call_interrupts
      2276           -43.6%       1283 ±  9%  interrupts.CPU217.CAL:Function_call_interrupts
      2248           -44.4%       1249 ±  3%  interrupts.CPU219.CAL:Function_call_interrupts
      2590 ±  4%     -38.4%       1596 ±  7%  interrupts.CPU22.CAL:Function_call_interrupts
      2245           -45.7%       1220        interrupts.CPU220.CAL:Function_call_interrupts
      2254           -45.6%       1227        interrupts.CPU221.CAL:Function_call_interrupts
      2295 ±  4%     -46.6%       1225        interrupts.CPU222.CAL:Function_call_interrupts
      2263 ±  2%     -45.4%       1237        interrupts.CPU223.CAL:Function_call_interrupts
      2236           -44.5%       1240 ±  2%  interrupts.CPU224.CAL:Function_call_interrupts
      2249           -44.3%       1252 ±  3%  interrupts.CPU225.CAL:Function_call_interrupts
      2284 ±  2%     -45.6%       1243 ±  4%  interrupts.CPU226.CAL:Function_call_interrupts
      2437 ± 15%     -50.1%       1216 ±  2%  interrupts.CPU227.CAL:Function_call_interrupts
      2256           -44.5%       1251 ±  3%  interrupts.CPU228.CAL:Function_call_interrupts
      2248           -45.1%       1233        interrupts.CPU229.CAL:Function_call_interrupts
      2365 ±  2%     -44.9%       1304 ±  3%  interrupts.CPU23.CAL:Function_call_interrupts
      2336 ±  6%     -47.1%       1236        interrupts.CPU230.CAL:Function_call_interrupts
      2268           -44.9%       1249 ±  2%  interrupts.CPU231.CAL:Function_call_interrupts
      2238           -41.7%       1305 ±  8%  interrupts.CPU232.CAL:Function_call_interrupts
      2246           -42.9%       1283 ±  6%  interrupts.CPU233.CAL:Function_call_interrupts
      2240           -44.8%       1237 ±  2%  interrupts.CPU234.CAL:Function_call_interrupts
      2246           -43.5%       1268 ±  6%  interrupts.CPU235.CAL:Function_call_interrupts
      2243           -43.2%       1274 ±  7%  interrupts.CPU236.CAL:Function_call_interrupts
      2250           -43.2%       1279 ±  7%  interrupts.CPU237.CAL:Function_call_interrupts
      2266           -45.2%       1242 ±  4%  interrupts.CPU238.CAL:Function_call_interrupts
      2262           -43.6%       1275 ±  5%  interrupts.CPU239.CAL:Function_call_interrupts
      2894 ± 18%     -38.4%       1782 ± 16%  interrupts.CPU24.CAL:Function_call_interrupts
      2266           -45.4%       1237 ±  2%  interrupts.CPU240.CAL:Function_call_interrupts
      2252           -44.8%       1243 ±  2%  interrupts.CPU241.CAL:Function_call_interrupts
      2264           -45.7%       1228        interrupts.CPU242.CAL:Function_call_interrupts
      2325 ±  5%     -46.6%       1241 ±  2%  interrupts.CPU243.CAL:Function_call_interrupts
      2277 ±  2%     -46.5%       1217        interrupts.CPU244.CAL:Function_call_interrupts
      2253           -45.2%       1234        interrupts.CPU245.CAL:Function_call_interrupts
      2299 ±  6%     -45.5%       1253 ±  2%  interrupts.CPU246.CAL:Function_call_interrupts
      2254           -44.4%       1252 ±  2%  interrupts.CPU247.CAL:Function_call_interrupts
      2294 ±  2%     -45.0%       1261 ±  4%  interrupts.CPU248.CAL:Function_call_interrupts
      2398 ± 13%     -48.0%       1247 ±  2%  interrupts.CPU249.CAL:Function_call_interrupts
      2699 ± 28%     -51.9%       1299 ±  4%  interrupts.CPU25.CAL:Function_call_interrupts
      2242           -44.4%       1246 ±  2%  interrupts.CPU250.CAL:Function_call_interrupts
      2283 ±  2%     -45.9%       1235        interrupts.CPU251.CAL:Function_call_interrupts
      2280           -45.6%       1239 ±  3%  interrupts.CPU252.CAL:Function_call_interrupts
      2251           -44.7%       1246        interrupts.CPU253.CAL:Function_call_interrupts
      2239           -45.5%       1219        interrupts.CPU254.CAL:Function_call_interrupts
      2252 ±  4%     -44.8%       1242 ±  5%  interrupts.CPU255.CAL:Function_call_interrupts
      2695 ±  6%     -41.3%       1581 ±  6%  interrupts.CPU28.CAL:Function_call_interrupts
      2408 ±  5%     -44.3%       1341 ±  4%  interrupts.CPU29.CAL:Function_call_interrupts
      2444 ±  5%     -43.2%       1389 ± 10%  interrupts.CPU3.CAL:Function_call_interrupts
      2329           -42.2%       1345 ±  4%  interrupts.CPU31.CAL:Function_call_interrupts
      2665 ±  7%     -43.1%       1516 ±  3%  interrupts.CPU32.CAL:Function_call_interrupts
      2341 ±  2%     -40.6%       1390 ±  4%  interrupts.CPU33.CAL:Function_call_interrupts
      2553 ±  4%     -33.6%       1695 ±  3%  interrupts.CPU34.CAL:Function_call_interrupts
    166.17 ± 15%     +82.7%     303.67 ± 29%  interrupts.CPU34.NMI:Non-maskable_interrupts
    166.17 ± 15%     +82.7%     303.67 ± 29%  interrupts.CPU34.PMI:Performance_monitoring_interrupts
      2333 ±  2%     -43.9%       1309 ±  4%  interrupts.CPU35.CAL:Function_call_interrupts
    161.83 ± 13%    +102.1%     327.00 ± 21%  interrupts.CPU35.NMI:Non-maskable_interrupts
    161.83 ± 13%    +102.1%     327.00 ± 21%  interrupts.CPU35.PMI:Performance_monitoring_interrupts
      2365 ±  2%     -38.7%       1450 ± 19%  interrupts.CPU37.CAL:Function_call_interrupts
      2664 ±  3%     -34.8%       1736 ± 19%  interrupts.CPU38.CAL:Function_call_interrupts
      2402 ±  4%     -45.1%       1320 ±  7%  interrupts.CPU39.CAL:Function_call_interrupts
      2620 ±  3%     -34.5%       1716 ± 18%  interrupts.CPU4.CAL:Function_call_interrupts
     24.83 ± 40%    +214.8%      78.17 ±121%  interrupts.CPU41.RES:Rescheduling_interrupts
      2384 ±  4%     -36.7%       1509 ± 17%  interrupts.CPU45.CAL:Function_call_interrupts
      2664 ±  3%     -35.7%       1714 ± 16%  interrupts.CPU46.CAL:Function_call_interrupts
      2364 ±  2%     -39.5%       1431 ±  9%  interrupts.CPU47.CAL:Function_call_interrupts
      2697           -44.5%       1496 ±  7%  interrupts.CPU48.CAL:Function_call_interrupts
      2363 ±  2%     -40.2%       1412 ±  9%  interrupts.CPU49.CAL:Function_call_interrupts
      2386           -45.4%       1301 ±  3%  interrupts.CPU5.CAL:Function_call_interrupts
      2653 ±  4%     -38.7%       1627 ±  8%  interrupts.CPU50.CAL:Function_call_interrupts
    229.17 ± 36%     +68.2%     385.50 ± 19%  interrupts.CPU50.NMI:Non-maskable_interrupts
    229.17 ± 36%     +68.2%     385.50 ± 19%  interrupts.CPU50.PMI:Performance_monitoring_interrupts
      3318 ± 64%     -60.7%       1303 ± 21%  interrupts.CPU51.CAL:Function_call_interrupts
      2914 ± 16%     -45.7%       1582 ± 13%  interrupts.CPU52.CAL:Function_call_interrupts
      2487 ± 15%     -53.5%       1157 ± 21%  interrupts.CPU53.CAL:Function_call_interrupts
      2541 ±  6%     -36.8%       1605 ±  7%  interrupts.CPU54.CAL:Function_call_interrupts
    232.83 ± 35%     +56.5%     364.33 ± 28%  interrupts.CPU54.NMI:Non-maskable_interrupts
    232.83 ± 35%     +56.5%     364.33 ± 28%  interrupts.CPU54.PMI:Performance_monitoring_interrupts
      4491 ± 97%     -69.8%       1354 ±  5%  interrupts.CPU55.CAL:Function_call_interrupts
      2711 ± 15%     -44.1%       1515 ±  7%  interrupts.CPU56.CAL:Function_call_interrupts
      2613 ± 23%     -48.4%       1348 ±  6%  interrupts.CPU57.CAL:Function_call_interrupts
      2505 ±  9%     -47.1%       1325 ±  4%  interrupts.CPU59.CAL:Function_call_interrupts
      2651 ±  7%     -40.6%       1575 ± 12%  interrupts.CPU6.CAL:Function_call_interrupts
      3231 ± 45%     -53.1%       1516 ±  6%  interrupts.CPU60.CAL:Function_call_interrupts
      2437 ±  7%     -45.5%       1327 ±  6%  interrupts.CPU61.CAL:Function_call_interrupts
      2407 ±  4%     -46.0%       1300 ±  4%  interrupts.CPU62.CAL:Function_call_interrupts
      2236           -45.0%       1229        interrupts.CPU64.CAL:Function_call_interrupts
      2241           -42.3%       1293 ± 11%  interrupts.CPU65.CAL:Function_call_interrupts
      2300 ±  3%     -42.7%       1318 ±  4%  interrupts.CPU66.CAL:Function_call_interrupts
      2305 ±  5%     -43.8%       1296 ±  6%  interrupts.CPU67.CAL:Function_call_interrupts
      2357 ±  5%     -44.6%       1306 ±  5%  interrupts.CPU68.CAL:Function_call_interrupts
      2256           -44.7%       1247 ±  2%  interrupts.CPU69.CAL:Function_call_interrupts
      2211 ± 14%     -40.2%       1323 ±  7%  interrupts.CPU70.CAL:Function_call_interrupts
      2300 ±  3%     -40.3%       1374 ± 23%  interrupts.CPU71.CAL:Function_call_interrupts
      2370 ±  6%     -43.0%       1350 ± 12%  interrupts.CPU72.CAL:Function_call_interrupts
      2298 ±  4%     -44.6%       1273 ±  4%  interrupts.CPU73.CAL:Function_call_interrupts
      2304           -39.3%       1399 ±  8%  interrupts.CPU74.CAL:Function_call_interrupts
      2317 ±  3%     -46.2%       1247 ±  2%  interrupts.CPU75.CAL:Function_call_interrupts
      2273 ±  7%     -41.8%       1322 ±  9%  interrupts.CPU76.CAL:Function_call_interrupts
      2246           -44.4%       1248 ±  3%  interrupts.CPU77.CAL:Function_call_interrupts
      2266           -42.9%       1294 ±  4%  interrupts.CPU78.CAL:Function_call_interrupts
      2210 ±  2%     -42.6%       1268 ±  6%  interrupts.CPU79.CAL:Function_call_interrupts
      3611 ± 38%     -57.3%       1542 ± 13%  interrupts.CPU8.CAL:Function_call_interrupts
      2280           -42.1%       1321 ± 10%  interrupts.CPU80.CAL:Function_call_interrupts
      2305 ±  2%     -46.4%       1234 ±  2%  interrupts.CPU81.CAL:Function_call_interrupts
      2347 ±  2%     -46.3%       1260        interrupts.CPU82.CAL:Function_call_interrupts
      2258           -42.4%       1300 ±  6%  interrupts.CPU83.CAL:Function_call_interrupts
      2275           -44.3%       1266 ±  2%  interrupts.CPU84.CAL:Function_call_interrupts
      2287           -44.8%       1263 ±  4%  interrupts.CPU85.CAL:Function_call_interrupts
      2268           -42.7%       1300 ±  4%  interrupts.CPU86.CAL:Function_call_interrupts
      2265           -45.5%       1234 ±  2%  interrupts.CPU87.CAL:Function_call_interrupts
      2385 ± 11%     -43.4%       1351 ±  7%  interrupts.CPU88.CAL:Function_call_interrupts
      2343 ±  8%     -46.1%       1262 ±  5%  interrupts.CPU89.CAL:Function_call_interrupts
      2555 ± 14%     -49.4%       1292 ±  5%  interrupts.CPU9.CAL:Function_call_interrupts
      2482 ± 18%     -43.7%       1396 ± 18%  interrupts.CPU90.CAL:Function_call_interrupts
      2282 ±  2%     -41.3%       1340 ± 17%  interrupts.CPU91.CAL:Function_call_interrupts
      2284 ±  2%     -42.3%       1317 ±  8%  interrupts.CPU92.CAL:Function_call_interrupts
      2296 ±  4%     -43.7%       1292 ±  6%  interrupts.CPU93.CAL:Function_call_interrupts
      2271           -43.4%       1285        interrupts.CPU94.CAL:Function_call_interrupts
      2261           -44.5%       1256 ±  3%  interrupts.CPU95.CAL:Function_call_interrupts
      2281           -43.6%       1286 ±  8%  interrupts.CPU96.CAL:Function_call_interrupts
      2252           -44.3%       1255 ±  3%  interrupts.CPU97.CAL:Function_call_interrupts
      2277           -41.1%       1341 ±  9%  interrupts.CPU98.CAL:Function_call_interrupts


                                                                                
                            aim9.sync_disk_rw.ops_per_sec                       
                                                                                
  104000 +------------------------------------------------------------------+   
  103000 |-+                        +                                       |   
         |                          :   +   .+                              |   
  102000 |.+  +.+                  : :  :+ +  + .+                        + |   
  101000 |-++ :  :  +    +.        : : :  +    +  :    .+.               : :|   
         |   +   : + :  +  +       :  ::          :   +   +.+.    .+. .+ : :|   
  100000 |-+      +  : +    +   .+ :  +            : +        + .+   +  +   |   
   99000 |-+          +      +.+  :                ++          +            |   
   98000 |-+                 O O  + O   O  O O O   OO                       |   
         |                                O      O    O                     |   
   97000 |-+ O                                                              |   
   96000 |-+  O     O OO O O     O    O                                     |   
         | O      O                                                         |   
   95000 |-+    O                                                           |   
   94000 +------------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample

***************************************************************************************************
lkp-csl-2sp7: 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 512G memory
=========================================================================================
class/compiler/cpufreq_governor/disk/fs/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime/ucode:
  filesystem/gcc-9/performance/1HDD/ext4/x86_64-rhel-8.3/10%/debian-10.4-x86_64-20200603.cgz/lkp-csl-2sp7/procfs/stress-ng/60s/0x5003006

commit: 
  3b87f136f8 ("sched,debug: Convert sysctl sched_domains to debugfs")
  d27e9ae2f2 ("sched: Move /proc/sched_debug to debugfs")

3b87f136f8fccddf d27e9ae2f244805bbdc730d85fb 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
    115034 ±  2%      -9.5%     104080 ±  5%  stress-ng.time.minor_page_faults
     15.69 ±  8%      +7.3%      16.83        boot-time.dhcp
  35702184 ±181%     -93.3%    2406274 ± 26%  cpuidle.C1.time
    492358 ±124%     -91.0%      44488 ± 53%  cpuidle.C1.usage
      0.17 ± 53%      +0.1        0.27        perf-profile.children.cycles-pp.pagetypeinfo_show
      0.18 ± 53%      +0.1        0.28        perf-profile.children.cycles-pp.walk_zones_in_node
    902.17 ±  8%     +19.3%       1076 ±  9%  slabinfo.mnt_cache.active_objs
    902.17 ±  8%     +19.3%       1076 ±  9%  slabinfo.mnt_cache.num_objs
      8847 ± 10%     -17.6%       7288 ± 13%  softirqs.CPU1.SCHED
    163494 ±  9%     +22.7%     200569        softirqs.RCU
     10483 ±  3%     -17.5%       8652 ± 11%  numa-meminfo.node0.KernelStack
      3924 ±  6%     -27.5%       2844 ± 26%  numa-meminfo.node0.PageTables
      7325 ±  5%     +25.5%       9194 ± 10%  numa-meminfo.node1.KernelStack
      0.01 ± 12%     +62.8%       0.01 ± 14%  perf-sched.sch_delay.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
      1.38 ±  7%     -42.2%       0.80 ± 70%  perf-sched.wait_and_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      2.76 ±  7%     -42.2%       1.59 ± 70%  perf-sched.wait_and_delay.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
    758224 ±  4%      -5.6%     715591 ±  3%  proc-vmstat.numa_hit
    671579 ±  4%      -6.4%     628907 ±  4%  proc-vmstat.numa_local
    347739            -3.0%     337408        proc-vmstat.pgfault
    230479 ± 18%     +64.7%     379605 ±  7%  numa-numastat.node0.local_node
    270668 ± 19%     +50.9%     408487 ±  8%  numa-numastat.node0.numa_hit
    409683 ± 10%     -47.5%     215143        numa-numastat.node1.local_node
    456129 ±  8%     -40.2%     272921 ±  7%  numa-numastat.node1.numa_hit
    264583           -66.7%      88202 ±141%  numa-vmstat.node0.nr_dirtied
     10486 ±  3%     -17.5%       8651 ± 11%  numa-vmstat.node0.nr_kernel_stack
    981.83 ±  6%     -27.6%     711.33 ± 27%  numa-vmstat.node0.nr_page_table_pages
    264582           -66.7%      88202 ±141%  numa-vmstat.node0.nr_written
   1096517 ± 12%     -20.3%     873421 ± 13%  numa-vmstat.node0.numa_local
     24.50 ±223%  +7.2e+05%     176406 ± 70%  numa-vmstat.node1.nr_dirtied
      7330 ±  5%     +25.5%       9196 ± 10%  numa-vmstat.node1.nr_kernel_stack
     24.50 ±223%  +7.2e+05%     176405 ± 70%  numa-vmstat.node1.nr_written
    649365 ± 21%     +35.5%     879722 ± 12%  numa-vmstat.node1.numa_local
  31793019 ±  5%      -6.9%   29603311 ±  3%  perf-stat.i.cache-misses
 1.004e+08 ±  3%      -3.5%   96964748 ±  2%  perf-stat.i.cache-references
      3794 ±  4%      +7.4%       4076 ±  3%  perf-stat.i.cycles-between-cache-misses
     33021 ±  6%     +25.7%      41508 ± 12%  perf-stat.i.dTLB-store-misses
      5081            -3.1%       4921        perf-stat.i.minor-faults
     41683 ±  7%     -13.5%      36065 ±  2%  perf-stat.i.node-stores
      5093            -3.1%       4934        perf-stat.i.page-faults
      3809 ±  5%      +7.3%       4088 ±  3%  perf-stat.overall.cycles-between-cache-misses
  31305655 ±  5%      -6.9%   29134943 ±  3%  perf-stat.ps.cache-misses
     31556 ±  5%     +24.5%      39295 ± 12%  perf-stat.ps.dTLB-store-misses
      4947            -3.3%       4781        perf-stat.ps.minor-faults
     39817 ±  7%     -13.8%      34317 ±  2%  perf-stat.ps.node-stores
      4958            -3.3%       4792        perf-stat.ps.page-faults
    464635 ±  3%     -10.3%     416791 ±  3%  interrupts.CAL:Function_call_interrupts
      3721 ± 22%     -41.2%       2188 ± 31%  interrupts.CPU0.NMI:Non-maskable_interrupts
      3721 ± 22%     -41.2%       2188 ± 31%  interrupts.CPU0.PMI:Performance_monitoring_interrupts
      6017 ± 18%     -26.0%       4452 ± 20%  interrupts.CPU13.NMI:Non-maskable_interrupts
      6017 ± 18%     -26.0%       4452 ± 20%  interrupts.CPU13.PMI:Performance_monitoring_interrupts
      2082 ± 49%    +305.7%       8449 ± 61%  interrupts.CPU17.CAL:Function_call_interrupts
      1749 ± 78%    +474.1%      10040 ± 65%  interrupts.CPU17.TLB:TLB_shootdowns
      5210 ± 11%     -42.7%       2984 ± 64%  interrupts.CPU20.NMI:Non-maskable_interrupts
      5210 ± 11%     -42.7%       2984 ± 64%  interrupts.CPU20.PMI:Performance_monitoring_interrupts
      3955 ± 18%     +48.0%       5854 ± 24%  interrupts.CPU30.NMI:Non-maskable_interrupts
      3955 ± 18%     +48.0%       5854 ± 24%  interrupts.CPU30.PMI:Performance_monitoring_interrupts
      5266 ± 43%     -62.2%       1989 ± 38%  interrupts.CPU32.CAL:Function_call_interrupts
      5605 ± 47%     -68.9%       1741 ± 56%  interrupts.CPU32.TLB:TLB_shootdowns
      4104 ± 20%     +46.6%       6018 ± 20%  interrupts.CPU37.NMI:Non-maskable_interrupts
      4104 ± 20%     +46.6%       6018 ± 20%  interrupts.CPU37.PMI:Performance_monitoring_interrupts
      5979 ± 18%     +23.1%       7363 ±  4%  interrupts.CPU48.NMI:Non-maskable_interrupts
      5979 ± 18%     +23.1%       7363 ±  4%  interrupts.CPU48.PMI:Performance_monitoring_interrupts
      4162 ± 40%     -60.7%       1634 ± 12%  interrupts.CPU50.CAL:Function_call_interrupts
      4377 ± 42%     -69.6%       1329 ± 19%  interrupts.CPU50.TLB:TLB_shootdowns
      5890 ±  8%     -27.9%       4247 ± 26%  interrupts.CPU53.NMI:Non-maskable_interrupts
      5890 ±  8%     -27.9%       4247 ± 26%  interrupts.CPU53.PMI:Performance_monitoring_interrupts
      5251 ± 39%     -70.2%       1566 ± 58%  interrupts.CPU62.NMI:Non-maskable_interrupts
      5251 ± 39%     -70.2%       1566 ± 58%  interrupts.CPU62.PMI:Performance_monitoring_interrupts
      4791 ± 23%     -27.0%       3499 ± 24%  interrupts.CPU64.NMI:Non-maskable_interrupts
      4791 ± 23%     -27.0%       3499 ± 24%  interrupts.CPU64.PMI:Performance_monitoring_interrupts
      6595 ± 82%     -57.4%       2808 ± 15%  interrupts.CPU78.CAL:Function_call_interrupts
      5832 ± 21%     -34.0%       3852 ± 32%  interrupts.CPU78.NMI:Non-maskable_interrupts
      5832 ± 21%     -34.0%       3852 ± 32%  interrupts.CPU78.PMI:Performance_monitoring_interrupts
      7480 ± 90%     -63.0%       2768 ± 19%  interrupts.CPU78.TLB:TLB_shootdowns
     11479 ± 69%     -77.2%       2614 ±  8%  interrupts.CPU80.CAL:Function_call_interrupts
      6588 ± 13%     -28.4%       4718 ± 26%  interrupts.CPU85.NMI:Non-maskable_interrupts
      6588 ± 13%     -28.4%       4718 ± 26%  interrupts.CPU85.PMI:Performance_monitoring_interrupts
      4290 ± 27%     +44.9%       6217 ± 21%  interrupts.CPU90.NMI:Non-maskable_interrupts
      4290 ± 27%     +44.9%       6217 ± 21%  interrupts.CPU90.PMI:Performance_monitoring_interrupts
      3834 ± 34%     +78.7%       6851 ±  8%  interrupts.CPU91.NMI:Non-maskable_interrupts
      3834 ± 34%     +78.7%       6851 ±  8%  interrupts.CPU91.PMI:Performance_monitoring_interrupts
      2262 ± 59%     +97.0%       4455 ± 16%  interrupts.CPU95.NMI:Non-maskable_interrupts
      2262 ± 59%     +97.0%       4455 ± 16%  interrupts.CPU95.PMI:Performance_monitoring_interrupts
      3624 ±  5%     -10.8%       3233        interrupts.RES:Rescheduling_interrupts
    504703 ±  2%     -12.1%     443495 ±  5%  interrupts.TLB:TLB_shootdowns
    183.58 ± 71%    -100.0%       0.00        sched_debug.cfs_rq:/.MIN_vruntime.avg
     14724 ± 71%    -100.0%       0.00        sched_debug.cfs_rq:/.MIN_vruntime.max
      0.00          -100.0%       0.00        sched_debug.cfs_rq:/.MIN_vruntime.min
      2808 ± 10%    -100.0%       0.00        sched_debug.cfs_rq:/.load.avg
     36770 ± 27%    -100.0%       0.00        sched_debug.cfs_rq:/.load.max
      7615 ± 13%    -100.0%       0.00        sched_debug.cfs_rq:/.load.stddev
    154.89 ± 26%    -100.0%       0.00        sched_debug.cfs_rq:/.load_avg.avg
      6749 ± 20%    -100.0%       0.00        sched_debug.cfs_rq:/.load_avg.max
    811.86 ± 26%    -100.0%       0.00        sched_debug.cfs_rq:/.load_avg.stddev
    183.58 ± 71%    -100.0%       0.00        sched_debug.cfs_rq:/.max_vruntime.avg
     14724 ± 71%    -100.0%       0.00        sched_debug.cfs_rq:/.max_vruntime.max
      0.00          -100.0%       0.00        sched_debug.cfs_rq:/.max_vruntime.min
     22653 ±  9%    -100.0%       0.00        sched_debug.cfs_rq:/.min_vruntime.avg
     38638 ±  7%    -100.0%       0.00        sched_debug.cfs_rq:/.min_vruntime.max
     16264 ±  6%    -100.0%       0.00        sched_debug.cfs_rq:/.min_vruntime.min
      3513 ± 12%    -100.0%       0.00        sched_debug.cfs_rq:/.min_vruntime.stddev
      0.13 ±  4%    -100.0%       0.00        sched_debug.cfs_rq:/.nr_running.avg
      1.00          -100.0%       0.00        sched_debug.cfs_rq:/.nr_running.max
      0.34 ±  2%    -100.0%       0.00        sched_debug.cfs_rq:/.nr_running.stddev
    361.15 ±  2%    -100.0%       0.00        sched_debug.cfs_rq:/.runnable_avg.avg
      1288 ± 24%    -100.0%       0.00        sched_debug.cfs_rq:/.runnable_avg.max
    300.95 ±  7%    -100.0%       0.00        sched_debug.cfs_rq:/.runnable_avg.stddev
     13707 ± 32%    -100.0%       0.00        sched_debug.cfs_rq:/.spread0.max
     -8665          -100.0%       0.00        sched_debug.cfs_rq:/.spread0.min
      3514 ± 12%    -100.0%       0.00        sched_debug.cfs_rq:/.spread0.stddev
    360.57 ±  2%    -100.0%       0.00        sched_debug.cfs_rq:/.util_avg.avg
      1283 ± 24%    -100.0%       0.00        sched_debug.cfs_rq:/.util_avg.max
    300.48 ±  7%    -100.0%       0.00        sched_debug.cfs_rq:/.util_avg.stddev
     27.85 ± 18%    -100.0%       0.00        sched_debug.cfs_rq:/.util_est_enqueued.avg
    510.00 ±  5%    -100.0%       0.00        sched_debug.cfs_rq:/.util_est_enqueued.max
    105.65 ±  7%    -100.0%       0.00        sched_debug.cfs_rq:/.util_est_enqueued.stddev
    709530 ±  2%    -100.0%       0.00        sched_debug.cpu.avg_idle.avg
   1091974 ±  9%    -100.0%       0.00        sched_debug.cpu.avg_idle.max
      2305 ± 17%    -100.0%       0.00        sched_debug.cpu.avg_idle.min
    291703 ±  5%    -100.0%       0.00        sched_debug.cpu.avg_idle.stddev
     36937 ±  3%    -100.0%       0.00        sched_debug.cpu.clock.avg
     36942 ±  3%    -100.0%       0.00        sched_debug.cpu.clock.max
     36932 ±  3%    -100.0%       0.00        sched_debug.cpu.clock.min
      3.05 ± 35%    -100.0%       0.00        sched_debug.cpu.clock.stddev
     36831 ±  3%    -100.0%       0.00        sched_debug.cpu.clock_task.avg
     36927 ±  3%    -100.0%       0.00        sched_debug.cpu.clock_task.max
     33217 ±  4%    -100.0%       0.00        sched_debug.cpu.clock_task.min
    444.03 ± 37%    -100.0%       0.00        sched_debug.cpu.clock_task.stddev
    273.56 ± 12%    -100.0%       0.00        sched_debug.cpu.curr->pid.avg
      2450          -100.0%       0.00        sched_debug.cpu.curr->pid.max
    747.29 ±  5%    -100.0%       0.00        sched_debug.cpu.curr->pid.stddev
    504404          -100.0%       0.00        sched_debug.cpu.max_idle_balance_cost.avg
    763707 ± 14%    -100.0%       0.00        sched_debug.cpu.max_idle_balance_cost.max
    500000          -100.0%       0.00        sched_debug.cpu.max_idle_balance_cost.min
     30655 ± 48%    -100.0%       0.00        sched_debug.cpu.max_idle_balance_cost.stddev
      4294          -100.0%       0.00        sched_debug.cpu.next_balance.avg
      4294          -100.0%       0.00        sched_debug.cpu.next_balance.max
      4294          -100.0%       0.00        sched_debug.cpu.next_balance.min
      0.00 ±  9%    -100.0%       0.00        sched_debug.cpu.next_balance.stddev
      0.12 ± 11%    -100.0%       0.00        sched_debug.cpu.nr_running.avg
      1.00          -100.0%       0.00        sched_debug.cpu.nr_running.max
      0.32 ±  4%    -100.0%       0.00        sched_debug.cpu.nr_running.stddev
      1562          -100.0%       0.00        sched_debug.cpu.nr_switches.avg
      8499 ± 16%    -100.0%       0.00        sched_debug.cpu.nr_switches.max
    499.33 ± 21%    -100.0%       0.00        sched_debug.cpu.nr_switches.min
      1322 ± 13%    -100.0%       0.00        sched_debug.cpu.nr_switches.stddev
      0.02 ± 20%    -100.0%       0.00        sched_debug.cpu.nr_uninterruptible.avg
     37.00 ± 19%    -100.0%       0.00        sched_debug.cpu.nr_uninterruptible.max
    -20.33          -100.0%       0.00        sched_debug.cpu.nr_uninterruptible.min
      7.91 ±  6%    -100.0%       0.00        sched_debug.cpu.nr_uninterruptible.stddev
     36934 ±  3%    -100.0%       0.00        sched_debug.cpu_clk
    996147          -100.0%       0.00        sched_debug.dl_rq:.dl_bw->bw.avg
    996147          -100.0%       0.00        sched_debug.dl_rq:.dl_bw->bw.max
    996147          -100.0%       0.00        sched_debug.dl_rq:.dl_bw->bw.min
 4.295e+09          -100.0%       0.00        sched_debug.jiffies
     36440 ±  3%    -100.0%       0.00        sched_debug.ktime
    950.00          -100.0%       0.00        sched_debug.rt_rq:/.rt_runtime.avg
    950.00          -100.0%       0.00        sched_debug.rt_rq:/.rt_runtime.max
    950.00          -100.0%       0.00        sched_debug.rt_rq:/.rt_runtime.min
     38291 ±  4%    -100.0%       0.00        sched_debug.sched_clk
      1.00          -100.0%       0.00        sched_debug.sched_clock_stable()
   4140603          -100.0%       0.00        sched_debug.sysctl_sched.sysctl_sched_features
     24.00          -100.0%       0.00        sched_debug.sysctl_sched.sysctl_sched_latency
      3.00          -100.0%       0.00        sched_debug.sysctl_sched.sysctl_sched_min_granularity
      1.00          -100.0%       0.00        sched_debug.sysctl_sched.sysctl_sched_tunable_scaling
      4.00          -100.0%       0.00        sched_debug.sysctl_sched.sysctl_sched_wakeup_granularity





Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


---
0DAY/LKP+ Test Infrastructure                   Open Source Technology Center
https://lists.01.org/hyperkitty/list/lkp@lists.01.org       Intel Corporation

Thanks,
Oliver Sang


View attachment "config-5.12.0-rc2-00060-gd27e9ae2f244" of type "text/plain" (172872 bytes)

View attachment "job-script" of type "text/plain" (7833 bytes)

View attachment "job.yaml" of type "text/plain" (5260 bytes)

View attachment "reproduce" of type "text/plain" (254 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ