lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Date:   Mon, 18 Nov 2019 16:25:59 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     Hillf Danton <hdanton@...a.com>
Cc:     linux-mm <linux-mm@...ck.org>,
        Andrew Morton <akpm@...ux-foundation.org>,
        Rong Chen <rong.a.chen@...el.com>,
        Fengguang Wu <fengguang.wu@...el.com>,
        Tejun Heo <tj@...nel.org>, Shakeel Butt <shakeelb@...gle.com>,
        Minchan Kim <minchan@...nel.org>, Mel Gorman <mgorman@...e.de>,
        linux-kernel <linux-kernel@...r.kernel.org>,
        Hillf Danton <hdanton@...a.com>, lkp@...ts.01.org
Subject: [writeback] e4f9eb2268:  fsmark.files_per_sec 48.8% improvement

Greeting,

FYI, we noticed a 48.8% improvement of fsmark.files_per_sec due to commit:


commit: e4f9eb2268da0eb6b706fe582ef34cbfc359c799 ("[RFC v3] writeback: add elastic bdi in cgwb bdp")
url: https://github.com/0day-ci/linux/commits/Hillf-Danton/writeback-add-elastic-bdi-in-cgwb-bdp/20191113-060336


in testcase: fsmark
on test machine: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 256G memory
with following parameters:

	iterations: 1x
	nr_threads: 32t
	disk: 1HDD
	fs: f2fs
	filesize: 16MB
	test_size: 60G
	sync_method: NoSync
	nr_directories: 16d
	nr_files_per_directory: 256fpd
	cpufreq_governor: performance
	ucode: 0x500002b

test-description: The fsmark is a file system benchmark to test synchronous write workloads, for example, mail servers workload.
test-url: https://sourceforge.net/projects/fsmark/

In addition to that, the commit also has significant impact on the following tests:

+------------------+----------------------------------------------------------------------+
| testcase: change | fsmark: fsmark.files_per_sec 84.3% improvement                       |
| test machine     | 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 256G memory |
| test parameters  | cpufreq_governor=performance                                         |
|                  | disk=1SSD                                                            |
|                  | filesize=16MB                                                        |
|                  | fs=btrfs                                                             |
|                  | iterations=1x                                                        |
|                  | nr_directories=16d                                                   |
|                  | nr_files_per_directory=256fpd                                        |
|                  | nr_threads=32t                                                       |
|                  | sync_method=NoSync                                                   |
|                  | test_size=60G                                                        |
|                  | ucode=0x500002b                                                      |
+------------------+----------------------------------------------------------------------+
| testcase: change | fsmark: fsmark.files_per_sec 39.4% improvement                       |
| test machine     | 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 256G memory |
| test parameters  | cpufreq_governor=performance                                         |
|                  | disk=1SSD                                                            |
|                  | filesize=16MB                                                        |
|                  | fs=f2fs                                                              |
|                  | iterations=1x                                                        |
|                  | nr_directories=16d                                                   |
|                  | nr_files_per_directory=256fpd                                        |
|                  | nr_threads=32t                                                       |
|                  | sync_method=NoSync                                                   |
|                  | test_size=60G                                                        |
|                  | ucode=0x500002b                                                      |
+------------------+----------------------------------------------------------------------+




Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
compiler/cpufreq_governor/disk/filesize/fs/iterations/kconfig/nr_directories/nr_files_per_directory/nr_threads/rootfs/sync_method/tbox_group/test_size/testcase/ucode:
  gcc-7/performance/1HDD/16MB/f2fs/1x/x86_64-rhel-7.6/16d/256fpd/32t/debian-x86_64-2019-09-23.cgz/NoSync/lkp-csl-2sp7/60G/fsmark/0x500002b

commit: 
  eb094f0696 ("Merge branch 'x86-pti-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip")
  e4f9eb2268 ("writeback: add elastic bdi in cgwb bdp")

eb094f06963bb0fd e4f9eb2268da0eb6b706fe582ef 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :4           25%           1:4     dmesg.WARNING:stack_recursion
         %stddev     %change         %stddev
             \          |                \  
    206181 ±  3%     -24.0%     156619        fsmark.app_overhead
     36.85           +48.8%      54.85 ±  2%  fsmark.files_per_sec
    347.27            +5.2%     365.18 ±  4%  fsmark.time.elapsed_time
    347.27            +5.2%     365.18 ±  4%  fsmark.time.elapsed_time.max
     82.25 ±  5%     +28.9%     106.00 ±  7%  fsmark.time.percent_of_cpu_this_job_got
    286.02 ±  5%     +35.5%     387.44 ±  4%  fsmark.time.system_time
    136907           -38.4%      84268        fsmark.time.voluntary_context_switches
 5.109e+08 ±102%     -74.3%  1.314e+08 ±168%  cpuidle.C1.time
   6376448            +7.8%    6874624 ±  6%  meminfo.DirectMap2M
      9.35 ±  3%      -8.4        0.95        mpstat.cpu.all.iowait%
      0.92 ±  4%      +0.3        1.23 ±  7%  mpstat.cpu.all.sys%
      1.52 ±102%      -1.2        0.37 ±168%  turbostat.C1%
     42.25 ±  3%      -6.5%      39.50 ±  2%  turbostat.PkgTmp
     27542 ± 24%     +34.9%      37156 ±  7%  numa-vmstat.node1.nr_active_anon
     27275 ± 25%     +34.4%      36654 ±  8%  numa-vmstat.node1.nr_anon_pages
     27542 ± 24%     +34.9%      37156 ±  7%  numa-vmstat.node1.nr_zone_active_anon
     89.25            +8.7%      97.00        vmstat.cpu.id
      9.00          -100.0%       0.00        vmstat.procs.b
      2180 ±  2%     -15.6%       1841        vmstat.system.cs
     89.63            +9.0%      97.68        iostat.cpu.idle
      9.29 ±  3%     -89.9%       0.94        iostat.cpu.iowait
      0.97 ±  5%     +31.0%       1.27 ±  7%  iostat.cpu.system
    222.75 ± 57%    +436.4%       1194 ± 57%  iostat.sda.r_await.max
    331.50 ±173%     -83.3%      55.25 ±173%  iostat.sdb.r_await.max
    229996 ±  8%     -18.9%     186634 ± 12%  numa-meminfo.node0.Active
    130566 ± 17%     -21.1%     103049 ± 17%  numa-meminfo.node0.AnonHugePages
    176121 ± 10%     +25.1%     220255 ± 10%  numa-meminfo.node1.Active
    110144 ± 24%     +34.9%     148629 ±  7%  numa-meminfo.node1.Active(anon)
    109070 ± 25%     +34.4%     146622 ±  8%  numa-meminfo.node1.AnonPages
      2752 ±  8%     +32.5%       3646 ±  4%  sched_debug.cfs_rq:/.exec_clock.avg
      3340 ±  6%     +18.3%       3953 ±  4%  sched_debug.cfs_rq:/.exec_clock.stddev
     84751 ± 14%     +32.3%     112168 ±  6%  sched_debug.cfs_rq:/.min_vruntime.avg
    254376 ±  4%     +26.3%     321225 ±  4%  sched_debug.cfs_rq:/.min_vruntime.max
     85628 ±  7%     +29.8%     111155 ±  7%  sched_debug.cfs_rq:/.min_vruntime.stddev
      0.08 ±  4%      -9.0%       0.07 ±  9%  sched_debug.cfs_rq:/.nr_running.avg
     85629 ±  7%     +29.8%     111155 ±  7%  sched_debug.cfs_rq:/.spread0.stddev
      6282            +6.8%       6708 ±  4%  sched_debug.cpu.curr->pid.max
      0.07 ±  2%     -73.9%       0.02 ±  9%  sched_debug.cpu.nr_uninterruptible.avg
      2274           -12.6%       1987 ±  5%  sched_debug.cpu.sched_goidle.avg
      1129           -46.0%     610.16 ±  9%  sched_debug.cpu.ttwu_local.avg
      2157 ±  3%     -16.0%       1811        perf-stat.i.context-switches
     53.45            -2.9       50.56 ±  2%  perf-stat.i.iTLB-load-miss-rate%
     83.08            +4.7       87.75 ±  2%  perf-stat.i.node-load-miss-rate%
    231939 ± 10%     +28.7%     298559 ±  7%  perf-stat.i.node-load-misses
     54.36 ± 17%     +29.7       84.08 ±  7%  perf-stat.i.node-store-miss-rate%
    206288 ±  8%     +43.9%     296880 ±  9%  perf-stat.i.node-store-misses
    145852 ± 10%     -17.7%     120031 ± 12%  perf-stat.i.node-stores
     54.51            -2.8       51.70 ±  2%  perf-stat.overall.iTLB-load-miss-rate%
     83.92            +3.5       87.37        perf-stat.overall.node-load-miss-rate%
     58.58 ±  4%     +12.6       71.22 ±  2%  perf-stat.overall.node-store-miss-rate%
      2150 ±  3%     -16.0%       1805        perf-stat.ps.context-switches
    231136 ± 10%     +28.7%     297572 ±  7%  perf-stat.ps.node-load-misses
    205545 ±  8%     +43.9%     295793 ±  9%  perf-stat.ps.node-store-misses
    145378 ± 10%     -17.6%     119740 ± 12%  perf-stat.ps.node-stores
     26510            +2.1%      27062        proc-vmstat.nr_active_file
   7110246            +4.5%    7433121 ±  2%  proc-vmstat.nr_dirty
  14943233            +2.5%   15312986        proc-vmstat.nr_file_pages
  14641262            +2.5%   15010784        proc-vmstat.nr_inactive_file
      1231            -2.5%       1200        proc-vmstat.nr_page_table_pages
     53598            +1.7%      54512        proc-vmstat.nr_slab_reclaimable
     26510            +2.1%      27062        proc-vmstat.nr_zone_active_file
  14641262            +2.5%   15010784        proc-vmstat.nr_zone_inactive_file
   7132293            +4.5%    7455224 ±  2%  proc-vmstat.nr_zone_write_pending
   3382991 ± 16%     +36.8%    4627644 ±  2%  proc-vmstat.numa_foreign
  13296658 ±  4%      -9.2%   12077073        proc-vmstat.numa_hit
  13265384 ±  4%      -9.2%   12045787        proc-vmstat.numa_local
   3382991 ± 16%     +36.8%    4627644 ±  2%  proc-vmstat.numa_miss
   3414265 ± 16%     +36.5%    4658931 ±  2%  proc-vmstat.numa_other
     58871 ± 26%     -45.2%      32279 ± 31%  proc-vmstat.numa_pte_updates
    347.75 ± 30%     -49.7%     174.75 ± 73%  interrupts.CPU22.NMI:Non-maskable_interrupts
    347.75 ± 30%     -49.7%     174.75 ± 73%  interrupts.CPU22.PMI:Performance_monitoring_interrupts
    384.25 ± 25%     -53.4%     179.00 ± 76%  interrupts.CPU23.NMI:Non-maskable_interrupts
    384.25 ± 25%     -53.4%     179.00 ± 76%  interrupts.CPU23.PMI:Performance_monitoring_interrupts
    314.25 ± 27%     -46.5%     168.00 ± 67%  interrupts.CPU25.NMI:Non-maskable_interrupts
    314.25 ± 27%     -46.5%     168.00 ± 67%  interrupts.CPU25.PMI:Performance_monitoring_interrupts
      3620 ±  2%     +88.1%       6810 ± 78%  interrupts.CPU35.CAL:Function_call_interrupts
      3603 ±  2%     +33.3%       4801 ± 38%  interrupts.CPU44.CAL:Function_call_interrupts
      3516 ±  2%      +7.1%       3764 ±  4%  interrupts.CPU5.CAL:Function_call_interrupts
    130.75 ±158%    +871.7%       1270 ± 65%  interrupts.CPU5.RES:Rescheduling_interrupts
    336.25 ± 33%     -43.3%     190.50 ± 65%  interrupts.CPU52.NMI:Non-maskable_interrupts
    336.25 ± 33%     -43.3%     190.50 ± 65%  interrupts.CPU52.PMI:Performance_monitoring_interrupts
      3494 ±  2%      +7.5%       3757 ±  4%  interrupts.CPU6.CAL:Function_call_interrupts
    457.25 ± 16%     -54.8%     206.75 ± 64%  interrupts.CPU60.NMI:Non-maskable_interrupts
    457.25 ± 16%     -54.8%     206.75 ± 64%  interrupts.CPU60.PMI:Performance_monitoring_interrupts
    359.75 ± 31%     -44.6%     199.25 ± 59%  interrupts.CPU62.NMI:Non-maskable_interrupts
    359.75 ± 31%     -44.6%     199.25 ± 59%  interrupts.CPU62.PMI:Performance_monitoring_interrupts
      3595 ±  3%      +5.7%       3799 ±  3%  interrupts.CPU77.CAL:Function_call_interrupts
      3554            +9.8%       3902 ±  6%  interrupts.CPU79.CAL:Function_call_interrupts
      3571 ±  3%      +6.3%       3797 ±  3%  interrupts.CPU81.CAL:Function_call_interrupts
      3568 ±  2%      +6.6%       3802 ±  4%  interrupts.CPU86.CAL:Function_call_interrupts
      7.75 ± 93%   +5029.0%     397.50 ±127%  interrupts.CPU9.RES:Rescheduling_interrupts
      7.25 ± 29%   +2024.1%     154.00 ±152%  interrupts.CPU92.RES:Rescheduling_interrupts
    141969 ±  8%     -15.8%     119489 ±  4%  softirqs.CPU10.TIMER
    144804 ±  7%     -15.2%     122776 ±  5%  softirqs.CPU14.TIMER
    141762 ±  6%     -15.9%     119169 ±  4%  softirqs.CPU16.TIMER
    141326 ±  4%     -12.0%     124357 ±  7%  softirqs.CPU17.TIMER
    145957 ±  6%     -15.2%     123724 ±  5%  softirqs.CPU20.TIMER
    142813 ±  6%     -13.1%     124176 ±  6%  softirqs.CPU21.TIMER
    142851 ±  7%     -13.6%     123373 ±  4%  softirqs.CPU22.TIMER
    149498 ±  6%     -16.0%     125543 ±  8%  softirqs.CPU4.TIMER
     29806 ±  8%     -18.1%      24404 ±  5%  softirqs.CPU5.RCU
    146049 ±  5%     -17.3%     120793 ±  2%  softirqs.CPU5.TIMER
    149260 ±  7%     -18.7%     121342 ±  5%  softirqs.CPU51.TIMER
    148165 ±  6%     -17.4%     122323 ±  8%  softirqs.CPU52.TIMER
    144746 ±  5%     -18.0%     118649 ±  3%  softirqs.CPU53.TIMER
    141387 ±  7%     -16.4%     118229 ±  4%  softirqs.CPU54.TIMER
    147445 ±  7%     -17.3%     121986 ±  8%  softirqs.CPU56.TIMER
    140489 ±  9%     -17.3%     116198 ±  4%  softirqs.CPU58.TIMER
    142617 ±  7%     -15.2%     120996 ±  3%  softirqs.CPU6.TIMER
    141828 ±  9%     -14.9%     120757 ±  5%  softirqs.CPU62.TIMER
    140300 ±  7%     -17.0%     116455 ±  4%  softirqs.CPU64.TIMER
    145431 ±  9%     -15.5%     122932 ±  9%  softirqs.CPU65.TIMER
    134905 ±  9%     -15.7%     113750 ±  4%  softirqs.CPU67.TIMER
    144366 ±  6%     -16.2%     121011 ±  7%  softirqs.CPU68.TIMER
    142688 ±  6%     -15.0%     121309 ±  6%  softirqs.CPU69.TIMER
    141441 ±  7%     -15.8%     119130 ±  5%  softirqs.CPU70.TIMER
    150355 ± 12%     -23.1%     115612 ±  4%  softirqs.CPU71.TIMER
    148522 ±  6%     -16.0%     124752 ±  7%  softirqs.CPU8.TIMER
     55.67 ± 14%     -12.8       42.92 ± 21%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      1.13 ± 14%      +0.5        1.59 ± 17%  perf-profile.calltrace.cycles-pp.rebalance_domains.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt
      0.81 ±100%      +1.5        2.27 ± 42%  perf-profile.calltrace.cycles-pp.ktime_get.tick_nohz_irq_exit.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt
      0.84 ± 97%      +1.5        2.31 ± 42%  perf-profile.calltrace.cycles-pp.tick_nohz_irq_exit.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
      2.16 ± 69%      +3.0        5.17 ± 40%  perf-profile.calltrace.cycles-pp.ktime_get.clockevents_program_event.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
     55.83 ± 14%     -12.6       43.21 ± 21%  perf-profile.children.cycles-pp.intel_idle
      0.59 ± 21%      -0.2        0.39 ± 34%  perf-profile.children.cycles-pp.enqueue_hrtimer
      0.48 ± 22%      -0.2        0.28 ± 37%  perf-profile.children.cycles-pp.timerqueue_del
      0.52 ± 24%      -0.2        0.34 ± 36%  perf-profile.children.cycles-pp.timerqueue_add
      0.60 ± 11%      -0.2        0.45 ± 17%  perf-profile.children.cycles-pp.cpuidle_governor_latency_req
      0.32 ± 25%      -0.1        0.19 ± 42%  perf-profile.children.cycles-pp.update_ts_time_stats
      0.19 ± 29%      -0.1        0.10 ± 33%  perf-profile.children.cycles-pp.run_posix_cpu_timers
      0.20 ± 24%      -0.1        0.11 ± 46%  perf-profile.children.cycles-pp.pm_qos_request
      0.11 ± 17%      -0.1        0.04 ±107%  perf-profile.children.cycles-pp.cpuidle_not_available
      0.21 ± 23%      -0.1        0.14 ± 23%  perf-profile.children.cycles-pp.rb_insert_color
      0.15 ± 14%      -0.1        0.10 ± 15%  perf-profile.children.cycles-pp.rb_erase
      0.12 ± 26%      +0.0        0.17 ± 14%  perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
      0.01 ±173%      +0.1        0.07 ± 23%  perf-profile.children.cycles-pp.timer_clear_idle
      0.25 ±  2%      +0.1        0.37 ± 13%  perf-profile.children.cycles-pp.rcu_eqs_enter
      0.41 ± 16%      +0.2        0.57 ± 14%  perf-profile.children.cycles-pp.arch_cpu_idle_enter
      0.39 ± 15%      +0.2        0.56 ± 14%  perf-profile.children.cycles-pp.tsc_verify_tsc_adjust
      0.12 ± 11%      +0.2        0.30 ± 39%  perf-profile.children.cycles-pp.hrtimer_forward
      0.29 ± 54%      +0.4        0.68 ± 36%  perf-profile.children.cycles-pp._raw_spin_trylock
      0.98 ± 73%      +1.4        2.35 ± 41%  perf-profile.children.cycles-pp.tick_nohz_irq_exit
      0.78 ± 99%      +1.4        2.21 ± 46%  perf-profile.children.cycles-pp.timekeeping_max_deferment
      5.04 ± 47%      +5.0       10.07 ± 34%  perf-profile.children.cycles-pp.ktime_get
     55.70 ± 13%     -12.5       43.17 ± 21%  perf-profile.self.cycles-pp.intel_idle
      0.19 ± 29%      -0.1        0.10 ± 33%  perf-profile.self.cycles-pp.run_posix_cpu_timers
      0.11 ± 20%      -0.1        0.04 ±107%  perf-profile.self.cycles-pp.cpuidle_not_available
      0.19 ± 22%      -0.1        0.12 ± 25%  perf-profile.self.cycles-pp.rb_insert_color
      0.13 ± 19%      -0.0        0.08 ± 17%  perf-profile.self.cycles-pp.rb_erase
      0.01 ±173%      +0.1        0.07 ± 23%  perf-profile.self.cycles-pp.timer_clear_idle
      0.21 ± 10%      +0.1        0.27 ± 18%  perf-profile.self.cycles-pp.get_next_timer_interrupt
      0.24            +0.1        0.35 ± 13%  perf-profile.self.cycles-pp.rcu_eqs_enter
      0.12 ± 11%      +0.2        0.29 ± 39%  perf-profile.self.cycles-pp.hrtimer_forward
      0.32 ± 30%      +0.2        0.53 ± 19%  perf-profile.self.cycles-pp.tsc_verify_tsc_adjust
      0.57 ± 34%      +0.3        0.82 ± 17%  perf-profile.self.cycles-pp.tick_nohz_next_event
      0.25 ± 36%      +0.4        0.68 ± 36%  perf-profile.self.cycles-pp._raw_spin_trylock
      0.77 ±102%      +1.4        2.21 ± 47%  perf-profile.self.cycles-pp.timekeeping_max_deferment
      4.14 ± 55%      +4.7        8.84 ± 36%  perf-profile.self.cycles-pp.ktime_get


                                                                                
                              fsmark.time.system_time                           
                                                                                
  450 +-+-------------------------------------------------------------------+   
      |                      O       O                                      |   
  400 O-O  O O   O OO     O O  O O O  O                                     |   
  350 +-+ O           O O                                                   |   
      |         .+.           .+.                                           |   
  300 +-+.++. .+   ++.+.+.+.++   +.+.++.+.+.++.+. .+.++.+.+. .++.+.+.+.++.+.|   
  250 +-+    +                                   +          +               |   
      |                                                                     |   
  200 +-+                                                                   |   
  150 +-+                                                                   |   
      |                                                                     |   
  100 +-+                                                                   |   
   50 +-+                                                                   |   
      |                                                                     |   
    0 +-+------O------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                       fsmark.time.voluntary_context_switches                   
                                                                                
  160000 +-+----------------------------------------------------------------+   
         |                                                                  |   
  140000 +-++.+.+.++.+.++.+.+.++.+.+  +.+.++.+.+.++.+.++.+.+.++.+.++.+.+.++.|   
  120000 +-+                        :+                                      |   
         |                          +                                       |   
  100000 +-+                                                                |   
         |                                                                  |   
   80000 O-OO O O  O   OO O O OO O OO O O                                   |   
         |                                                                  |   
   60000 +-+                                                                |   
   40000 +-+                                                                |   
         |           O                                                      |   
   20000 +-+                                                                |   
         |                                                                  |   
       0 +-+------O---------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                              fsmark.files_per_sec                              
                                                                                
  60 +-+----------O---------------------------------------------------------+   
     O O OO O   O   OO O   O OO   O O O                                     |   
  50 +-+                 O      O                                           |   
     |                                                                      |   
     |                                                                      |   
  40 +-+   .+.+.+.+.++.+.+.+.+ .+.+.+.          .+.+       .++.+.+.+.+.++. .|   
     |.+.++                   +       +.++.+.+.+    +.+.+.+               + |   
  30 +-+                                                                    |   
     |                                                                      |   
  20 +-+                                                                    |   
     |                                                                      |   
     |                                                                      |   
  10 +-+                                                                    |   
     |                                                                      |   
   0 +-+------O-------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample

***************************************************************************************************
lkp-csl-2sp7: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 256G memory
=========================================================================================
compiler/cpufreq_governor/disk/filesize/fs/iterations/kconfig/nr_directories/nr_files_per_directory/nr_threads/rootfs/sync_method/tbox_group/test_size/testcase/ucode:
  gcc-7/performance/1SSD/16MB/btrfs/1x/x86_64-rhel-7.6/16d/256fpd/32t/debian-x86_64-2019-09-23.cgz/NoSync/lkp-csl-2sp7/60G/fsmark/0x500002b

commit: 
  eb094f0696 ("Merge branch 'x86-pti-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip")
  e4f9eb2268 ("writeback: add elastic bdi in cgwb bdp")

eb094f06963bb0fd e4f9eb2268da0eb6b706fe582ef 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
     98.50           +84.3%     181.50 ±  3%  fsmark.files_per_sec
    201.25 ± 24%    +235.9%     676.00 ± 52%  fsmark.time.involuntary_context_switches
     67.00 ±  2%    +310.4%     275.00 ±  3%  fsmark.time.percent_of_cpu_this_job_got
     92.12 ±  2%    +311.2%     378.78 ±  3%  fsmark.time.system_time
    764632 ± 18%    +669.6%    5884310        fsmark.time.voluntary_context_switches
      0.98 ± 11%      -5.7%       0.92 ± 10%  boot-time.smp_boot
   4639725           -54.5%    2110392        meminfo.Dirty
  22793620           +16.9%   26639911        meminfo.Writeback
      9.44            -7.4        2.01 ±  2%  mpstat.cpu.all.iowait%
      0.82            +2.3        3.17 ±  3%  mpstat.cpu.all.sys%
     89.45            +5.7%      94.51        iostat.cpu.idle
      9.31           -78.8%       1.98 ±  2%  iostat.cpu.iowait
      0.95 ±  4%    +239.5%       3.23 ±  3%  iostat.cpu.system
   2364563 ±  6%     -58.4%     983026 ± 12%  numa-meminfo.node0.Dirty
     15165 ±  8%     +15.6%      17534 ±  6%  numa-meminfo.node0.Shmem
   2279950 ±  5%     -50.6%    1126676 ± 10%  numa-meminfo.node1.Dirty
      5408 ± 25%     -47.0%       2869 ± 41%  numa-meminfo.node1.Shmem
     89.00            +5.6%      94.00        vmstat.cpu.id
      9.00           -88.9%       1.00        vmstat.procs.b
      1.00          +200.0%       3.00        vmstat.procs.r
     14071 ± 14%    +843.8%     132804 ±  2%  vmstat.system.cs
   7965182 ± 51%     -65.6%    2737782 ±137%  numa-numastat.node0.local_node
   7993076 ± 51%     -65.7%    2745575 ±137%  numa-numastat.node0.numa_hit
   1192997 ±173%    +294.2%    4703020 ± 54%  numa-numastat.node0.numa_miss
   1220902 ±168%    +285.8%    4710820 ± 54%  numa-numastat.node0.other_node
   1192997 ±173%    +294.2%    4703020 ± 54%  numa-numastat.node1.numa_foreign
  29194348 ± 65%    +156.8%   74982446 ±  3%  cpuidle.C1.time
    614163 ± 72%    +548.6%    3983752 ±  8%  cpuidle.C1.usage
 3.703e+09 ± 51%    +187.6%  1.065e+10 ± 22%  cpuidle.C1E.time
  15222120 ± 13%     +63.5%   24881567 ± 14%  cpuidle.C1E.usage
 8.998e+09 ± 20%     -79.9%  1.809e+09 ±134%  cpuidle.C6.time
   9904301 ± 10%     +66.0%   16441007 ±  6%  cpuidle.POLL.time
    675155 ± 17%    +392.9%    3327749 ±  6%  cpuidle.POLL.usage
      1591 ±  2%     +16.6%       1855 ±  4%  slabinfo.biovec-max.active_objs
      1653 ±  2%     +14.9%       1900 ±  4%  slabinfo.biovec-max.num_objs
      4963 ±  2%     +16.3%       5770 ±  3%  slabinfo.btrfs_delayed_node.active_objs
      4987 ±  2%     +16.1%       5791 ±  3%  slabinfo.btrfs_delayed_node.num_objs
      2834 ±  4%     +13.8%       3224 ±  2%  slabinfo.btrfs_ordered_extent.active_objs
      2933 ±  3%     +13.2%       3322 ±  2%  slabinfo.btrfs_ordered_extent.num_objs
     22120 ±  3%      +9.4%      24207 ±  5%  slabinfo.filp.num_objs
    590993 ±  6%     -58.5%     245356 ± 12%  numa-vmstat.node0.nr_dirty
      3791 ±  8%     +15.6%       4383 ±  6%  numa-vmstat.node0.nr_shmem
   8101322 ± 48%     -60.9%    3166607 ±115%  numa-vmstat.node0.numa_hit
   8074295 ± 48%     -60.9%    3158895 ±115%  numa-vmstat.node0.numa_local
   1163378 ±173%    +293.6%    4578545 ± 55%  numa-vmstat.node0.numa_miss
   1190408 ±168%    +285.3%    4586258 ± 54%  numa-vmstat.node0.numa_other
    569934 ±  5%     -50.7%     281036 ± 10%  numa-vmstat.node1.nr_dirty
      1350 ± 25%     -46.9%     717.25 ± 41%  numa-vmstat.node1.nr_shmem
   1163392 ±173%    +293.6%    4578595 ± 55%  numa-vmstat.node1.numa_foreign
     86.50 ±  7%     +60.4%     138.75 ±  2%  turbostat.Avg_MHz
      5.62 ± 10%      +0.9        6.56 ±  3%  turbostat.Busy%
      1549 ±  3%     +37.2%       2125 ±  2%  turbostat.Bzy_MHz
    615790 ± 72%    +547.0%    3984026 ±  8%  turbostat.C1
      0.22 ± 66%      +0.3        0.56 ±  3%  turbostat.C1%
  15221796 ± 13%     +63.5%   24881064 ± 14%  turbostat.C1E
     27.76 ± 51%     +52.0       79.78 ± 22%  turbostat.C1E%
     67.49 ± 20%     -54.0       13.51 ±134%  turbostat.C6%
     96.43            +6.6%     102.80        turbostat.PkgWatt
   1160706           -54.6%     527074        proc-vmstat.nr_dirty
  14826379            +2.1%   15135031        proc-vmstat.nr_file_pages
  14543661            +2.1%   14852549        proc-vmstat.nr_inactive_file
      1241            -2.7%       1207        proc-vmstat.nr_page_table_pages
      5143            -0.8%       5100        proc-vmstat.nr_shmem
     52491            +1.6%      53314        proc-vmstat.nr_slab_reclaimable
     57094            +4.3%      59578        proc-vmstat.nr_slab_unreclaimable
   5704228           +16.6%    6653630        proc-vmstat.nr_writeback
  14543661            +2.1%   14852549        proc-vmstat.nr_zone_inactive_file
   6865018            +4.6%    7180790        proc-vmstat.nr_zone_write_pending
   4757461 ±  2%     +34.0%    6373060 ±  2%  proc-vmstat.numa_foreign
  11471142           -14.1%    9856783        proc-vmstat.numa_hit
  11439835           -14.1%    9825555        proc-vmstat.numa_local
   4757461 ±  2%     +34.0%    6373060 ±  2%  proc-vmstat.numa_miss
   4788768 ±  2%     +33.7%    6404288 ±  2%  proc-vmstat.numa_other
      5368 ±  3%      -7.9%       4943        proc-vmstat.pgactivate
    763.18 ± 24%    +277.2%       2878 ± 24%  sched_debug.cfs_rq:/.exec_clock.avg
      3908 ± 63%    +150.5%       9793 ±  8%  sched_debug.cfs_rq:/.exec_clock.max
      1041 ± 36%    +174.6%       2859 ±  3%  sched_debug.cfs_rq:/.exec_clock.stddev
     51.08 ± 12%     -23.5%      39.10 ± 19%  sched_debug.cfs_rq:/.load_avg.avg
     28955 ± 21%    +156.1%      74144 ± 22%  sched_debug.cfs_rq:/.min_vruntime.avg
     77632 ±  4%    +163.1%     204261 ±  8%  sched_debug.cfs_rq:/.min_vruntime.max
     23904 ±  5%    +160.8%      62349 ±  6%  sched_debug.cfs_rq:/.min_vruntime.stddev
    -18011          -164.1%      11547 ±189%  sched_debug.cfs_rq:/.spread0.avg
     30657 ± 61%    +362.1%     141676 ± 13%  sched_debug.cfs_rq:/.spread0.max
     23911 ±  5%    +160.8%      62353 ±  6%  sched_debug.cfs_rq:/.spread0.stddev
      8971 ± 13%   +1286.6%     124400 ±  2%  sched_debug.cpu.nr_switches.avg
    326674 ± 42%    +188.8%     943564 ± 18%  sched_debug.cpu.nr_switches.max
     36613 ± 34%    +365.8%     170553 ± 10%  sched_debug.cpu.nr_switches.stddev
      7385 ± 16%   +1850.7%     144072 ±  2%  sched_debug.cpu.sched_count.avg
    323205 ± 43%    +200.8%     972192 ± 18%  sched_debug.cpu.sched_count.max
     36207 ± 35%    +412.3%     185505 ±  9%  sched_debug.cpu.sched_count.stddev
      3624 ± 16%   +1592.5%      61342 ±  2%  sched_debug.cpu.sched_goidle.avg
    161191 ± 43%    +191.6%     470040 ± 18%  sched_debug.cpu.sched_goidle.max
     18051 ± 35%    +371.2%      85059 ± 10%  sched_debug.cpu.sched_goidle.stddev
      3672 ± 16%   +2154.6%      82799 ±  3%  sched_debug.cpu.ttwu_count.avg
    180691 ± 42%    +400.9%     905080 ±  2%  sched_debug.cpu.ttwu_count.max
     18925 ± 38%    +722.1%     155592 ±  6%  sched_debug.cpu.ttwu_count.stddev
    662.08           -38.8%     404.94 ±  2%  sched_debug.cpu.ttwu_local.avg
 5.829e+08 ±  5%     +46.2%   8.52e+08 ±  2%  perf-stat.i.branch-instructions
      4.91 ± 32%      -3.2        1.75        perf-stat.i.branch-miss-rate%
  10726947 ±  2%      +9.3%   11729895 ±  8%  perf-stat.i.cache-misses
     14453 ± 14%    +852.2%     137618 ±  2%  perf-stat.i.context-switches
 6.705e+09 ±  6%     +77.1%  1.188e+10 ±  3%  perf-stat.i.cpu-cycles
      2.25 ± 29%    +133.2%       5.25 ± 22%  perf-stat.i.cpu-migrations
      2108 ±  9%     +76.5%       3721 ± 25%  perf-stat.i.cycles-between-cache-misses
      0.37 ± 22%      -0.3        0.08 ±156%  perf-stat.i.dTLB-load-miss-rate%
   1231579 ± 20%     -77.4%     278161 ±141%  perf-stat.i.dTLB-load-misses
 7.319e+08 ±  5%     +40.1%  1.026e+09 ±  2%  perf-stat.i.dTLB-loads
      0.10 ± 21%      -0.1        0.02 ±141%  perf-stat.i.dTLB-store-miss-rate%
    179997 ± 19%     -74.9%      45267 ±129%  perf-stat.i.dTLB-store-misses
 3.336e+08 ±  5%     +32.2%   4.41e+08 ±  2%  perf-stat.i.dTLB-stores
   1340141 ± 21%     +46.2%    1959021 ±  4%  perf-stat.i.iTLB-load-misses
   1028018 ± 13%     +63.2%    1677371 ±  2%  perf-stat.i.iTLB-loads
 2.893e+09 ±  5%     +39.1%  4.026e+09 ±  2%  perf-stat.i.instructions
     76.46 ±  2%      +4.8       81.30 ±  3%  perf-stat.i.node-load-miss-rate%
    616311 ±  4%    +105.7%    1267509 ± 20%  perf-stat.i.node-load-misses
     53.09 ± 10%     +25.4       78.49 ±  4%  perf-stat.i.node-store-miss-rate%
    248843 ± 26%    +111.1%     525396 ± 17%  perf-stat.i.node-store-misses
    212587 ±  4%     -39.3%     129033 ± 15%  perf-stat.i.node-stores
      4.15 ± 19%      -2.0        2.13        perf-stat.overall.branch-miss-rate%
      2.33 ±  9%     +26.8%       2.95 ±  2%  perf-stat.overall.cpi
    625.64 ±  7%     +62.9%       1018 ±  8%  perf-stat.overall.cycles-between-cache-misses
      0.17 ± 25%      -0.1        0.03 ±141%  perf-stat.overall.dTLB-load-miss-rate%
      0.05 ± 24%      -0.0        0.01 ±128%  perf-stat.overall.dTLB-store-miss-rate%
      0.43 ± 10%     -21.8%       0.34 ±  2%  perf-stat.overall.ipc
     43.85 ±  2%     +18.1       61.93 ±  7%  perf-stat.overall.node-load-miss-rate%
     52.98 ± 15%     +27.0       79.93 ±  4%  perf-stat.overall.node-store-miss-rate%
 5.786e+08 ±  5%     +46.1%  8.456e+08 ±  2%  perf-stat.ps.branch-instructions
  10655938 ±  2%      +9.2%   11641373 ±  8%  perf-stat.ps.cache-misses
     14347 ± 14%    +851.9%     136576 ±  2%  perf-stat.ps.context-switches
 6.657e+09 ±  6%     +77.1%  1.179e+10 ±  3%  perf-stat.ps.cpu-cycles
      2.23 ± 29%    +133.3%       5.21 ± 22%  perf-stat.ps.cpu-migrations
   1222682 ± 20%     -77.4%     276165 ±141%  perf-stat.ps.dTLB-load-misses
 7.267e+08 ±  5%     +40.1%  1.018e+09 ±  2%  perf-stat.ps.dTLB-loads
    178699 ± 19%     -74.9%      44941 ±129%  perf-stat.ps.dTLB-store-misses
 3.312e+08 ±  4%     +32.2%  4.378e+08 ±  2%  perf-stat.ps.dTLB-stores
   1330582 ± 21%     +46.2%    1944678 ±  4%  perf-stat.ps.iTLB-load-misses
   1020385 ± 13%     +63.2%    1664945 ±  2%  perf-stat.ps.iTLB-loads
 2.872e+09 ±  5%     +39.1%  3.996e+09 ±  2%  perf-stat.ps.instructions
    611263 ±  4%    +105.7%    1257464 ± 20%  perf-stat.ps.node-load-misses
    247453 ± 26%    +111.0%     522107 ± 17%  perf-stat.ps.node-store-misses
    210950 ±  4%     -39.3%     128103 ± 15%  perf-stat.ps.node-stores
 3.907e+11 ±  5%     +39.3%  5.442e+11 ±  2%  perf-stat.total.instructions
     12342 ± 13%     -13.4%      10692 ± 13%  softirqs.CPU0.RCU
     12466 ± 16%     -13.6%      10765 ± 11%  softirqs.CPU11.RCU
     12356 ± 15%     -14.6%      10554 ±  9%  softirqs.CPU13.RCU
     12632 ± 12%     -16.6%      10537 ±  9%  softirqs.CPU14.RCU
     18944 ±  3%     +17.8%      22313 ±  7%  softirqs.CPU15.SCHED
     65343 ± 13%     -18.5%      53276 ±  8%  softirqs.CPU2.TIMER
     11755 ±  7%     -21.1%       9278 ± 14%  softirqs.CPU23.RCU
     18741 ±  4%     +13.1%      21196        softirqs.CPU24.SCHED
     12699 ± 12%     -17.8%      10445 ±  9%  softirqs.CPU27.RCU
     67824 ± 10%     -24.7%      51099 ±  5%  softirqs.CPU27.TIMER
     18466           +14.4%      21133 ±  6%  softirqs.CPU3.SCHED
     12543 ± 11%     -16.5%      10467 ±  8%  softirqs.CPU31.RCU
     13209 ± 16%     -20.0%      10569 ±  9%  softirqs.CPU32.RCU
     13391 ± 13%     -20.9%      10595 ±  9%  softirqs.CPU34.RCU
     12978 ± 14%     -19.0%      10514 ±  9%  softirqs.CPU35.RCU
     12716 ± 12%     -17.6%      10480 ±  8%  softirqs.CPU37.RCU
     13130 ± 17%     -20.4%      10457 ±  8%  softirqs.CPU41.RCU
     12755 ± 11%     -17.5%      10525 ±  9%  softirqs.CPU43.RCU
     13506 ± 13%     -21.5%      10600 ±  9%  softirqs.CPU44.RCU
     13047 ± 14%     -18.4%      10650 ±  7%  softirqs.CPU45.RCU
     13383 ± 14%     -21.8%      10461 ±  8%  softirqs.CPU46.RCU
     13084 ± 14%     -19.6%      10514 ±  8%  softirqs.CPU47.RCU
     13283 ± 11%     -28.5%       9491 ± 31%  softirqs.CPU50.RCU
     13232 ± 12%     -20.6%      10503 ±  5%  softirqs.CPU51.RCU
     12804 ±  8%     -15.7%      10796 ± 10%  softirqs.CPU53.RCU
     12941 ± 12%     -17.0%      10742 ± 10%  softirqs.CPU54.RCU
     12749 ± 11%     -16.6%      10638 ± 10%  softirqs.CPU56.RCU
     13099 ± 14%     -17.0%      10877 ±  9%  softirqs.CPU58.RCU
     13363 ±  9%     -19.8%      10713 ± 10%  softirqs.CPU6.RCU
     12488 ± 13%     -14.0%      10743 ± 10%  softirqs.CPU60.RCU
     12485 ± 13%     -12.6%      10907 ±  9%  softirqs.CPU61.RCU
     12534 ± 14%     -14.8%      10673 ± 10%  softirqs.CPU7.RCU
     11454 ±  7%     -15.2%       9712 ±  9%  softirqs.CPU72.RCU
     13026 ± 28%     -21.9%      10169 ±  7%  softirqs.CPU76.RCU
     12586 ± 13%     -18.0%      10323 ± 10%  softirqs.CPU8.RCU
     12873 ± 17%     -20.8%      10195 ±  9%  softirqs.CPU80.RCU
     12497 ± 14%     -18.5%      10189 ±  9%  softirqs.CPU81.RCU
     12851 ± 15%     -19.7%      10319 ±  9%  softirqs.CPU82.RCU
     12866 ± 15%     -20.5%      10230 ±  8%  softirqs.CPU83.RCU
     12558 ± 12%     -16.9%      10432 ±  7%  softirqs.CPU85.RCU
     12641 ± 16%     -32.6%       8526 ± 37%  softirqs.CPU86.RCU
     12856 ± 16%     -20.3%      10248 ±  8%  softirqs.CPU88.RCU
     13192 ± 19%     -22.0%      10286 ±  7%  softirqs.CPU89.RCU
     11716 ±  8%     -12.9%      10199 ±  9%  softirqs.CPU90.RCU
     12620 ± 12%     -18.0%      10345 ±  7%  softirqs.CPU91.RCU
     12933 ± 16%     -21.9%      10095 ±  8%  softirqs.CPU92.RCU
     13185 ± 16%     -23.0%      10146 ±  9%  softirqs.CPU93.RCU
     13066 ± 16%     -22.3%      10156 ±  8%  softirqs.CPU94.RCU
     12889 ± 16%     -21.2%      10156 ±  8%  softirqs.CPU95.RCU
   5699200 ±  8%     -13.7%    4919656        softirqs.TIMER
    954.75 ± 39%    +183.2%       2703 ± 54%  interrupts.CPU0.RES:Rescheduling_interrupts
    281.00 ± 21%     -43.7%     158.25 ± 20%  interrupts.CPU11.NMI:Non-maskable_interrupts
    281.00 ± 21%     -43.7%     158.25 ± 20%  interrupts.CPU11.PMI:Performance_monitoring_interrupts
    183.25 ±103%   +4491.8%       8414 ± 66%  interrupts.CPU15.RES:Rescheduling_interrupts
    310.00 ± 55%     -44.4%     172.25 ± 11%  interrupts.CPU2.NMI:Non-maskable_interrupts
    310.00 ± 55%     -44.4%     172.25 ± 11%  interrupts.CPU2.PMI:Performance_monitoring_interrupts
      1.50 ±137%  +24083.3%     362.75 ± 98%  interrupts.CPU22.RES:Rescheduling_interrupts
    248.25 ± 23%     -36.3%     158.25 ± 17%  interrupts.CPU24.NMI:Non-maskable_interrupts
    248.25 ± 23%     -36.3%     158.25 ± 17%  interrupts.CPU24.PMI:Performance_monitoring_interrupts
      7.25 ± 97%  +21217.2%       1545 ± 96%  interrupts.CPU24.RES:Rescheduling_interrupts
    274.25 ± 16%     -41.7%     160.00 ± 16%  interrupts.CPU25.NMI:Non-maskable_interrupts
    274.25 ± 16%     -41.7%     160.00 ± 16%  interrupts.CPU25.PMI:Performance_monitoring_interrupts
    249.50 ± 25%     -32.1%     169.50 ± 27%  interrupts.CPU26.NMI:Non-maskable_interrupts
    249.50 ± 25%     -32.1%     169.50 ± 27%  interrupts.CPU26.PMI:Performance_monitoring_interrupts
    277.25 ± 26%     -43.1%     157.75 ± 16%  interrupts.CPU27.NMI:Non-maskable_interrupts
    277.25 ± 26%     -43.1%     157.75 ± 16%  interrupts.CPU27.PMI:Performance_monitoring_interrupts
     15.25 ±139%  +73360.7%      11202 ±108%  interrupts.CPU28.RES:Rescheduling_interrupts
    258.75 ± 26%     -40.9%     153.00 ± 30%  interrupts.CPU29.NMI:Non-maskable_interrupts
    258.75 ± 26%     -40.9%     153.00 ± 30%  interrupts.CPU29.PMI:Performance_monitoring_interrupts
      9.25 ± 92%  +89291.9%       8268 ±111%  interrupts.CPU29.RES:Rescheduling_interrupts
      9.00 ± 65%  +1.6e+05%      14544 ± 72%  interrupts.CPU3.RES:Rescheduling_interrupts
    315.50 ± 12%     -44.3%     175.75 ± 15%  interrupts.CPU30.NMI:Non-maskable_interrupts
    315.50 ± 12%     -44.3%     175.75 ± 15%  interrupts.CPU30.PMI:Performance_monitoring_interrupts
      1.25 ± 87%  +3.3e+05%       4066 ± 85%  interrupts.CPU30.RES:Rescheduling_interrupts
    528.50 ± 66%     -70.1%     158.00 ± 11%  interrupts.CPU31.NMI:Non-maskable_interrupts
    528.50 ± 66%     -70.1%     158.00 ± 11%  interrupts.CPU31.PMI:Performance_monitoring_interrupts
      1558 ±172%    +216.0%       4923 ± 74%  interrupts.CPU31.RES:Rescheduling_interrupts
    581.75 ± 48%     -63.3%     213.50 ± 44%  interrupts.CPU32.NMI:Non-maskable_interrupts
    581.75 ± 48%     -63.3%     213.50 ± 44%  interrupts.CPU32.PMI:Performance_monitoring_interrupts
    295.25 ± 21%     -27.9%     213.00 ± 34%  interrupts.CPU33.NMI:Non-maskable_interrupts
    295.25 ± 21%     -27.9%     213.00 ± 34%  interrupts.CPU33.PMI:Performance_monitoring_interrupts
    327.75 ± 25%     -49.8%     164.50 ± 17%  interrupts.CPU34.NMI:Non-maskable_interrupts
    327.75 ± 25%     -49.8%     164.50 ± 17%  interrupts.CPU34.PMI:Performance_monitoring_interrupts
     11.00 ± 61%  +65011.4%       7162 ±137%  interrupts.CPU34.RES:Rescheduling_interrupts
    315.00 ± 24%     -51.8%     151.75 ± 33%  interrupts.CPU35.NMI:Non-maskable_interrupts
    315.00 ± 24%     -51.8%     151.75 ± 33%  interrupts.CPU35.PMI:Performance_monitoring_interrupts
    293.25 ± 22%     -45.1%     161.00 ± 14%  interrupts.CPU36.NMI:Non-maskable_interrupts
    293.25 ± 22%     -45.1%     161.00 ± 14%  interrupts.CPU36.PMI:Performance_monitoring_interrupts
    281.75 ± 29%     -38.4%     173.50 ± 15%  interrupts.CPU38.NMI:Non-maskable_interrupts
    281.75 ± 29%     -38.4%     173.50 ± 15%  interrupts.CPU38.PMI:Performance_monitoring_interrupts
    298.75 ± 20%     -43.3%     169.50 ± 17%  interrupts.CPU39.NMI:Non-maskable_interrupts
    298.75 ± 20%     -43.3%     169.50 ± 17%  interrupts.CPU39.PMI:Performance_monitoring_interrupts
      9.50 ± 86%  +45005.3%       4285 ± 92%  interrupts.CPU4.RES:Rescheduling_interrupts
    297.50 ± 20%     -42.4%     171.25 ± 13%  interrupts.CPU40.NMI:Non-maskable_interrupts
    297.50 ± 20%     -42.4%     171.25 ± 13%  interrupts.CPU40.PMI:Performance_monitoring_interrupts
    298.00 ± 18%     -42.2%     172.25 ± 14%  interrupts.CPU41.NMI:Non-maskable_interrupts
    298.00 ± 18%     -42.2%     172.25 ± 14%  interrupts.CPU41.PMI:Performance_monitoring_interrupts
    379.50 ± 24%     -50.0%     189.75 ± 23%  interrupts.CPU42.NMI:Non-maskable_interrupts
    379.50 ± 24%     -50.0%     189.75 ± 23%  interrupts.CPU42.PMI:Performance_monitoring_interrupts
    325.75 ± 20%     -48.9%     166.50 ± 18%  interrupts.CPU44.NMI:Non-maskable_interrupts
    325.75 ± 20%     -48.9%     166.50 ± 18%  interrupts.CPU44.PMI:Performance_monitoring_interrupts
      1.50 ±100%  +67400.0%       1012 ± 96%  interrupts.CPU45.RES:Rescheduling_interrupts
    314.25 ± 21%     -46.3%     168.75 ± 12%  interrupts.CPU46.NMI:Non-maskable_interrupts
    314.25 ± 21%     -46.3%     168.75 ± 12%  interrupts.CPU46.PMI:Performance_monitoring_interrupts
    261.00 ± 18%     -42.6%     149.75 ± 34%  interrupts.CPU49.NMI:Non-maskable_interrupts
    261.00 ± 18%     -42.6%     149.75 ± 34%  interrupts.CPU49.PMI:Performance_monitoring_interrupts
    342.75 ± 42%     -56.8%     148.00 ± 32%  interrupts.CPU50.NMI:Non-maskable_interrupts
    342.75 ± 42%     -56.8%     148.00 ± 32%  interrupts.CPU50.PMI:Performance_monitoring_interrupts
      2.25 ±110%  +23233.3%     525.00 ±100%  interrupts.CPU50.RES:Rescheduling_interrupts
    270.50 ± 22%     -37.3%     169.50 ± 14%  interrupts.CPU51.NMI:Non-maskable_interrupts
    270.50 ± 22%     -37.3%     169.50 ± 14%  interrupts.CPU51.PMI:Performance_monitoring_interrupts
      3.00 ± 74%  +12616.7%     381.50 ±104%  interrupts.CPU51.RES:Rescheduling_interrupts
    274.50 ± 25%     -39.4%     166.25 ± 27%  interrupts.CPU52.NMI:Non-maskable_interrupts
    274.50 ± 25%     -39.4%     166.25 ± 27%  interrupts.CPU52.PMI:Performance_monitoring_interrupts
    358.00 ± 37%     -56.6%     155.25 ± 34%  interrupts.CPU53.NMI:Non-maskable_interrupts
    358.00 ± 37%     -56.6%     155.25 ± 34%  interrupts.CPU53.PMI:Performance_monitoring_interrupts
    301.00 ± 29%     -47.0%     159.50 ± 33%  interrupts.CPU54.NMI:Non-maskable_interrupts
    301.00 ± 29%     -47.0%     159.50 ± 33%  interrupts.CPU54.PMI:Performance_monitoring_interrupts
    248.75 ± 26%     -34.8%     162.25 ± 25%  interrupts.CPU55.NMI:Non-maskable_interrupts
    248.75 ± 26%     -34.8%     162.25 ± 25%  interrupts.CPU55.PMI:Performance_monitoring_interrupts
    275.75 ± 26%     -45.7%     149.75 ± 35%  interrupts.CPU56.NMI:Non-maskable_interrupts
    275.75 ± 26%     -45.7%     149.75 ± 35%  interrupts.CPU56.PMI:Performance_monitoring_interrupts
    256.00 ± 15%     -38.0%     158.75 ± 31%  interrupts.CPU57.NMI:Non-maskable_interrupts
    256.00 ± 15%     -38.0%     158.75 ± 31%  interrupts.CPU57.PMI:Performance_monitoring_interrupts
    430.50 ± 59%     -64.8%     151.50 ± 32%  interrupts.CPU59.NMI:Non-maskable_interrupts
    430.50 ± 59%     -64.8%     151.50 ± 32%  interrupts.CPU59.PMI:Performance_monitoring_interrupts
    260.25 ± 16%     -53.2%     121.75 ± 30%  interrupts.CPU60.NMI:Non-maskable_interrupts
    260.25 ± 16%     -53.2%     121.75 ± 30%  interrupts.CPU60.PMI:Performance_monitoring_interrupts
    273.75 ± 17%     -36.7%     173.25 ± 13%  interrupts.CPU62.NMI:Non-maskable_interrupts
    273.75 ± 17%     -36.7%     173.25 ± 13%  interrupts.CPU62.PMI:Performance_monitoring_interrupts
    265.75 ± 27%     -38.6%     163.25 ± 14%  interrupts.CPU63.NMI:Non-maskable_interrupts
    265.75 ± 27%     -38.6%     163.25 ± 14%  interrupts.CPU63.PMI:Performance_monitoring_interrupts
    275.50 ± 24%     -36.4%     175.25 ± 14%  interrupts.CPU64.NMI:Non-maskable_interrupts
    275.50 ± 24%     -36.4%     175.25 ± 14%  interrupts.CPU64.PMI:Performance_monitoring_interrupts
    232.50 ± 15%     -20.4%     185.00 ± 14%  interrupts.CPU68.NMI:Non-maskable_interrupts
    232.50 ± 15%     -20.4%     185.00 ± 14%  interrupts.CPU68.PMI:Performance_monitoring_interrupts
      0.75 ±110%  +9.8e+05%       7387 ±147%  interrupts.CPU71.RES:Rescheduling_interrupts
    276.00 ± 22%     -34.0%     182.25 ± 17%  interrupts.CPU78.NMI:Non-maskable_interrupts
    276.00 ± 22%     -34.0%     182.25 ± 17%  interrupts.CPU78.PMI:Performance_monitoring_interrupts
      2634 ± 43%     -32.0%       1791 ±  8%  interrupts.CPU79.CAL:Function_call_interrupts
    399.50 ± 22%     -50.8%     196.75 ± 27%  interrupts.CPU80.NMI:Non-maskable_interrupts
    399.50 ± 22%     -50.8%     196.75 ± 27%  interrupts.CPU80.PMI:Performance_monitoring_interrupts
    257.25 ± 29%     -28.6%     183.75 ± 23%  interrupts.CPU81.NMI:Non-maskable_interrupts
    257.25 ± 29%     -28.6%     183.75 ± 23%  interrupts.CPU81.PMI:Performance_monitoring_interrupts
      0.00       +1.2e+105%       1217 ±173%  interrupts.CPU83.TLB:TLB_shootdowns
      1.25 ±173%  +1.4e+05%       1787 ± 77%  interrupts.CPU86.RES:Rescheduling_interrupts
    256.00 ± 28%     -36.0%     163.75 ± 21%  interrupts.CPU87.NMI:Non-maskable_interrupts
    256.00 ± 28%     -36.0%     163.75 ± 21%  interrupts.CPU87.PMI:Performance_monitoring_interrupts
      9.25 ±138%  +52624.3%       4877 ±155%  interrupts.CPU87.RES:Rescheduling_interrupts
    274.00 ± 31%     -38.7%     168.00 ± 16%  interrupts.CPU89.NMI:Non-maskable_interrupts
    274.00 ± 31%     -38.7%     168.00 ± 16%  interrupts.CPU89.PMI:Performance_monitoring_interrupts
     36.25 ±107%   +3699.3%       1377 ±100%  interrupts.CPU9.RES:Rescheduling_interrupts
    376.50 ±  8%     -48.1%     195.50 ± 22%  interrupts.CPU90.NMI:Non-maskable_interrupts
    376.50 ±  8%     -48.1%     195.50 ± 22%  interrupts.CPU90.PMI:Performance_monitoring_interrupts
    300.75 ± 37%     -44.1%     168.00 ± 16%  interrupts.CPU92.NMI:Non-maskable_interrupts
    300.75 ± 37%     -44.1%     168.00 ± 16%  interrupts.CPU92.PMI:Performance_monitoring_interrupts
     26762 ± 15%     -30.6%      18563 ± 12%  interrupts.NMI:Non-maskable_interrupts
     26762 ± 15%     -30.6%      18563 ± 12%  interrupts.PMI:Performance_monitoring_interrupts
     35167 ±  4%    +613.2%     250825 ± 10%  interrupts.RES:Rescheduling_interrupts
     54.58 ±  4%     -12.9       41.68 ± 12%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     84.09            -6.4       77.67 ±  3%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
     84.87            -5.9       78.93 ±  3%  perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     97.14            -1.9       95.20        perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
     97.14            -1.9       95.20        perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
     97.06            -1.9       95.12        perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     97.91            -0.8       97.10        perf-profile.calltrace.cycles-pp.secondary_startup_64
      0.92 ± 14%      +0.2        1.17 ± 11%  perf-profile.calltrace.cycles-pp.ktime_get.tick_irq_enter.irq_enter.smp_apic_timer_interrupt.apic_timer_interrupt
      1.11 ±  4%      +0.3        1.44 ± 13%  perf-profile.calltrace.cycles-pp.native_write_msr.lapic_next_deadline.clockevents_program_event.hrtimer_interrupt.smp_apic_timer_interrupt
      1.35 ±  2%      +0.3        1.68 ± 12%  perf-profile.calltrace.cycles-pp.lapic_next_deadline.clockevents_program_event.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      0.15 ±173%      +0.5        0.66 ± 17%  perf-profile.calltrace.cycles-pp.ktime_get.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt
      0.56 ± 61%      +0.5        1.09 ± 20%  perf-profile.calltrace.cycles-pp.ktime_get_update_offsets_now.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
      0.16 ±173%      +0.6        0.71 ± 12%  perf-profile.calltrace.cycles-pp.calc_global_load_tick.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer
      0.18 ±173%      +0.7        0.92 ± 19%  perf-profile.calltrace.cycles-pp.run_local_timers.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
      3.78 ±  8%      +1.4        5.15 ± 12%  perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      0.75 ± 93%      +1.5        2.21 ± 14%  perf-profile.calltrace.cycles-pp.ktime_get.tick_nohz_irq_exit.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt
      0.42 ±173%      +1.5        1.89 ±  7%  perf-profile.calltrace.cycles-pp.timekeeping_max_deferment.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle
      0.77 ± 91%      +1.5        2.25 ± 14%  perf-profile.calltrace.cycles-pp.tick_nohz_irq_exit.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
      3.93 ± 19%      +1.9        5.85 ±  2%  perf-profile.calltrace.cycles-pp.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry
      4.89 ± 14%      +2.0        6.90 ±  2%  perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry.start_secondary
      4.40 ± 13%      +2.0        6.45 ± 11%  perf-profile.calltrace.cycles-pp.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      1.85 ± 74%      +2.8        4.66 ± 15%  perf-profile.calltrace.cycles-pp.ktime_get.clockevents_program_event.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      3.33 ± 40%      +3.2        6.51 ± 14%  perf-profile.calltrace.cycles-pp.clockevents_program_event.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
      9.27 ±  5%      +4.0       13.26 ± 14%  perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     12.25 ± 11%      +4.3       16.56 ± 12%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
     19.44 ±  7%      +5.9       25.33 ± 11%  perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle
     23.39 ±  7%      +7.8       31.24 ± 10%  perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     54.85 ±  4%     -12.9       41.90 ± 12%  perf-profile.children.cycles-pp.intel_idle
     85.51            -4.9       80.64 ±  2%  perf-profile.children.cycles-pp.cpuidle_enter_state
     85.56            -4.8       80.71 ±  2%  perf-profile.children.cycles-pp.cpuidle_enter
     97.14            -1.9       95.20        perf-profile.children.cycles-pp.start_secondary
     97.91            -0.8       97.10        perf-profile.children.cycles-pp.secondary_startup_64
     97.91            -0.8       97.10        perf-profile.children.cycles-pp.cpu_startup_entry
     97.99            -0.8       97.22        perf-profile.children.cycles-pp.do_idle
      0.55 ± 22%      -0.3        0.27 ± 13%  perf-profile.children.cycles-pp.__remove_hrtimer
      0.68 ± 18%      -0.2        0.46 ± 12%  perf-profile.children.cycles-pp.irq_work_interrupt
      0.68 ± 18%      -0.2        0.46 ± 12%  perf-profile.children.cycles-pp.smp_irq_work_interrupt
      0.68 ± 18%      -0.2        0.46 ± 12%  perf-profile.children.cycles-pp.irq_work_run
      0.68 ± 18%      -0.2        0.46 ± 12%  perf-profile.children.cycles-pp.printk
      0.68 ± 18%      -0.2        0.46 ± 12%  perf-profile.children.cycles-pp.vprintk_emit
      0.43 ± 22%      -0.2        0.23 ±  3%  perf-profile.children.cycles-pp.timerqueue_del
      0.67 ± 11%      -0.2        0.49 ± 21%  perf-profile.children.cycles-pp.serial8250_console_putchar
      0.69 ± 12%      -0.2        0.51 ± 20%  perf-profile.children.cycles-pp.uart_console_write
      0.68 ± 11%      -0.2        0.51 ± 20%  perf-profile.children.cycles-pp.wait_for_xmitr
      0.27 ± 22%      -0.1        0.15 ±  8%  perf-profile.children.cycles-pp.rb_next
      0.30 ± 19%      -0.1        0.19 ± 13%  perf-profile.children.cycles-pp.__hrtimer_get_next_event
      0.51 ±  6%      -0.1        0.39 ± 15%  perf-profile.children.cycles-pp.io_serial_in
      0.16 ± 30%      -0.1        0.06 ±  6%  perf-profile.children.cycles-pp.run_posix_cpu_timers
      0.17 ± 23%      -0.1        0.10 ± 10%  perf-profile.children.cycles-pp.account_process_tick
      0.14 ±  6%      -0.1        0.08 ± 17%  perf-profile.children.cycles-pp.rb_erase
      0.11 ± 24%      +0.0        0.14 ± 18%  perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
      0.11 ±  7%      +0.0        0.15 ±  7%  perf-profile.children.cycles-pp.raise_softirq
      0.08 ± 17%      +0.0        0.12 ± 18%  perf-profile.children.cycles-pp.smp_call_function_single
      0.01 ±173%      +0.0        0.06 ±  9%  perf-profile.children.cycles-pp.search_binary_handler
      0.01 ±173%      +0.0        0.06 ±  9%  perf-profile.children.cycles-pp.load_elf_binary
      0.07 ± 11%      +0.0        0.12 ±  5%  perf-profile.children.cycles-pp.__handle_mm_fault
      0.07 ± 14%      +0.0        0.12 ±  8%  perf-profile.children.cycles-pp.handle_mm_fault
      0.09 ± 16%      +0.1        0.14 ±  3%  perf-profile.children.cycles-pp.__do_page_fault
      0.10 ± 22%      +0.1        0.15 ±  4%  perf-profile.children.cycles-pp.page_fault
      0.10 ±  7%      +0.1        0.15 ± 18%  perf-profile.children.cycles-pp.find_next_and_bit
      0.09 ± 16%      +0.1        0.14 ±  5%  perf-profile.children.cycles-pp.do_page_fault
      0.01 ±173%      +0.1        0.07 ± 31%  perf-profile.children.cycles-pp.mmput
      0.01 ±173%      +0.1        0.07 ± 31%  perf-profile.children.cycles-pp.exit_mmap
      0.01 ±173%      +0.1        0.08 ± 27%  perf-profile.children.cycles-pp.__libc_fork
      0.00            +0.1        0.07 ± 28%  perf-profile.children.cycles-pp.__x64_sys_clone
      0.00            +0.1        0.07 ± 28%  perf-profile.children.cycles-pp._do_fork
      0.00            +0.1        0.08        perf-profile.children.cycles-pp.read
      0.07 ± 20%      +0.1        0.15 ± 14%  perf-profile.children.cycles-pp.tick_check_broadcast_expired
      0.17 ± 15%      +0.1        0.26 ±  7%  perf-profile.children.cycles-pp.ksys_read
      0.17 ± 15%      +0.1        0.26 ±  7%  perf-profile.children.cycles-pp.vfs_read
      0.01 ±173%      +0.1        0.10 ± 19%  perf-profile.children.cycles-pp.timekeeping_advance
      0.03 ±100%      +0.1        0.14 ± 16%  perf-profile.children.cycles-pp.tick_sched_do_timer
      0.46 ±  8%      +0.1        0.57 ±  8%  perf-profile.children.cycles-pp.update_sd_lb_stats
      0.19 ± 23%      +0.1        0.30 ±  8%  perf-profile.children.cycles-pp.call_cpuidle
      0.36 ± 23%      +0.1        0.48 ± 10%  perf-profile.children.cycles-pp.__intel_pmu_enable_all
      0.18 ± 27%      +0.1        0.31 ±  7%  perf-profile.children.cycles-pp.trigger_load_balance
      0.41 ± 14%      +0.1        0.55 ±  6%  perf-profile.children.cycles-pp.rcu_idle_exit
      0.09 ± 19%      +0.2        0.33 ± 17%  perf-profile.children.cycles-pp.hrtimer_forward
      1.39 ±  2%      +0.3        1.71 ± 12%  perf-profile.children.cycles-pp.lapic_next_deadline
      0.30 ± 48%      +0.3        0.63 ± 22%  perf-profile.children.cycles-pp._raw_spin_trylock
      0.93 ± 18%      +0.4        1.30 ±  9%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.74 ±  5%      +0.4        1.13 ± 31%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      1.23 ± 20%      +0.4        1.63 ± 12%  perf-profile.children.cycles-pp.do_syscall_64
      0.70 ± 23%      +0.4        1.10 ± 21%  perf-profile.children.cycles-pp.ktime_get_update_offsets_now
      0.32 ± 63%      +0.4        0.73 ± 12%  perf-profile.children.cycles-pp.calc_global_load_tick
      1.23 ± 20%      +0.4        1.63 ± 11%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
      1.55 ±  6%      +0.5        2.02 ± 12%  perf-profile.children.cycles-pp.native_write_msr
      0.44 ± 37%      +0.5        0.94 ± 19%  perf-profile.children.cycles-pp.run_local_timers
      1.20 ± 11%      +0.6        1.76 ± 13%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.57 ±113%      +1.3        1.91 ±  7%  perf-profile.children.cycles-pp.timekeeping_max_deferment
      0.93 ± 63%      +1.4        2.28 ± 14%  perf-profile.children.cycles-pp.tick_nohz_irq_exit
      3.83 ±  8%      +1.4        5.21 ± 12%  perf-profile.children.cycles-pp.tick_sched_timer
      4.01 ± 18%      +1.9        5.94 ±  2%  perf-profile.children.cycles-pp.tick_nohz_next_event
      4.96 ± 14%      +2.0        6.98 ±  2%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      6.77 ± 12%      +2.7        9.46 ± 12%  perf-profile.children.cycles-pp.irq_exit
      3.39 ± 39%      +3.2        6.59 ± 14%  perf-profile.children.cycles-pp.clockevents_program_event
      9.43 ±  5%      +4.0       13.42 ± 14%  perf-profile.children.cycles-pp.menu_select
     12.45 ± 11%      +4.3       16.79 ± 12%  perf-profile.children.cycles-pp.hrtimer_interrupt
      4.44 ± 45%      +4.8        9.22 ± 11%  perf-profile.children.cycles-pp.ktime_get
     19.69 ±  7%      +6.0       25.66 ± 11%  perf-profile.children.cycles-pp.smp_apic_timer_interrupt
     21.81 ±  7%      +7.0       28.80 ± 10%  perf-profile.children.cycles-pp.apic_timer_interrupt
     54.70 ±  4%     -12.8       41.87 ± 12%  perf-profile.self.cycles-pp.intel_idle
      0.51 ±  6%      -0.1        0.39 ± 15%  perf-profile.self.cycles-pp.io_serial_in
      0.16 ± 30%      -0.1        0.06 ±  6%  perf-profile.self.cycles-pp.run_posix_cpu_timers
      0.17 ± 23%      -0.1        0.10 ± 12%  perf-profile.self.cycles-pp.account_process_tick
      0.12 ±  7%      -0.0        0.07 ± 22%  perf-profile.self.cycles-pp.rb_erase
      0.10 ±  8%      +0.0        0.14 ±  6%  perf-profile.self.cycles-pp.raise_softirq
      0.05 ±  8%      +0.0        0.09 ± 19%  perf-profile.self.cycles-pp.stack_trace_save_tsk
      0.04 ± 58%      +0.0        0.08 ± 19%  perf-profile.self.cycles-pp.end_page_writeback
      0.10 ± 11%      +0.1        0.15 ± 19%  perf-profile.self.cycles-pp.find_next_and_bit
      0.05 ± 62%      +0.1        0.14 ± 15%  perf-profile.self.cycles-pp.tick_check_broadcast_expired
      0.23 ± 17%      +0.1        0.34 ±  4%  perf-profile.self.cycles-pp.rcu_idle_exit
      0.18 ± 23%      +0.1        0.30 ±  6%  perf-profile.self.cycles-pp.call_cpuidle
      0.15 ± 32%      +0.1        0.28 ±  6%  perf-profile.self.cycles-pp.trigger_load_balance
      0.69 ±  8%      +0.2        0.92 ± 14%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.08 ± 17%      +0.2        0.32 ± 15%  perf-profile.self.cycles-pp.hrtimer_forward
      0.86 ± 17%      +0.2        1.11 ±  2%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.33 ± 19%      +0.3        0.59 ±  7%  perf-profile.self.cycles-pp.__softirqentry_text_start
      0.23 ± 56%      +0.3        0.52 ± 12%  perf-profile.self.cycles-pp.tick_sched_timer
      0.51 ± 25%      +0.3        0.81 ±  6%  perf-profile.self.cycles-pp.tick_nohz_next_event
      0.30 ± 48%      +0.3        0.63 ± 22%  perf-profile.self.cycles-pp._raw_spin_trylock
      0.54 ± 28%      +0.4        0.93 ± 20%  perf-profile.self.cycles-pp.ktime_get_update_offsets_now
      0.31 ± 69%      +0.4        0.71 ± 13%  perf-profile.self.cycles-pp.calc_global_load_tick
      1.54 ±  6%      +0.5        2.01 ± 12%  perf-profile.self.cycles-pp.native_write_msr
      0.42 ± 40%      +0.5        0.91 ± 18%  perf-profile.self.cycles-pp.run_local_timers
      1.20 ± 11%      +0.6        1.75 ± 13%  perf-profile.self.cycles-pp.native_irq_return_iret
      0.56 ±119%      +1.3        1.90 ±  7%  perf-profile.self.cycles-pp.timekeeping_max_deferment
      2.71 ±  2%      +1.5        4.19 ± 31%  perf-profile.self.cycles-pp.cpuidle_enter_state
      3.75 ±  5%      +2.0        5.80 ± 31%  perf-profile.self.cycles-pp.menu_select
      3.47 ± 56%      +4.6        8.04 ± 11%  perf-profile.self.cycles-pp.ktime_get



***************************************************************************************************
lkp-csl-2sp7: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 256G memory
=========================================================================================
compiler/cpufreq_governor/disk/filesize/fs/iterations/kconfig/nr_directories/nr_files_per_directory/nr_threads/rootfs/sync_method/tbox_group/test_size/testcase/ucode:
  gcc-7/performance/1SSD/16MB/f2fs/1x/x86_64-rhel-7.6/16d/256fpd/32t/debian-x86_64-2019-09-23.cgz/NoSync/lkp-csl-2sp7/60G/fsmark/0x500002b

commit: 
  eb094f0696 ("Merge branch 'x86-pti-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip")
  e4f9eb2268 ("writeback: add elastic bdi in cgwb bdp")

eb094f06963bb0fd e4f9eb2268da0eb6b706fe582ef 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
    181861 ±  2%     -15.6%     153543        fsmark.app_overhead
     85.92           +39.4%     119.80        fsmark.files_per_sec
    198.50 ±  2%     +28.3%     254.75 ±  3%  fsmark.time.percent_of_cpu_this_job_got
    286.52 ±  2%     +29.5%     370.98 ±  3%  fsmark.time.system_time
    102252           -19.7%      82096 ±  2%  fsmark.time.voluntary_context_switches
     41480 ± 17%     -10.7%      37022 ± 15%  numa-meminfo.node0.Writeback
   6836583 ± 35%     -38.8%    4184696 ± 55%  numa-numastat.node0.local_node
   6852179 ± 35%     -38.7%    4197165 ± 55%  numa-numastat.node0.numa_hit
   6824137 ± 35%     -34.6%    4465626 ± 52%  numa-vmstat.node0.numa_hit
   6808740 ± 35%     -34.6%    4453309 ± 52%  numa-vmstat.node0.numa_local
     89.04            +7.9%      96.06        iostat.cpu.idle
      8.47 ±  3%     -90.8%       0.78        iostat.cpu.iowait
      2.25 ±  2%     +29.7%       2.92 ±  3%  iostat.cpu.system
      8.58 ±  3%      -7.8        0.78        mpstat.cpu.all.iowait%
      0.10 ± 30%      -0.0        0.05 ± 19%  mpstat.cpu.all.soft%
      2.18 ±  2%      +0.7        2.90 ±  3%  mpstat.cpu.all.sys%
     26823           +11.0%      29767 ±  5%  slabinfo.nat_entry.active_objs
     27331           +10.9%      30313 ±  5%  slabinfo.nat_entry.num_objs
      1235 ±  7%     +13.8%       1405 ±  7%  slabinfo.task_group.active_objs
      1235 ±  7%     +13.8%       1405 ±  7%  slabinfo.task_group.num_objs
  15229559 ± 58%     +67.5%   25512150 ±  9%  turbostat.C1E
     40.66 ± 77%     +50.7       91.40 ±  2%  turbostat.C1E%
   9576498 ± 46%     -98.0%     190269 ± 40%  turbostat.C6
     52.09 ± 55%     -49.8        2.30 ± 19%  turbostat.C6%
     88.75            +8.2%      96.00        vmstat.cpu.id
      8.00          -100.0%       0.00        vmstat.procs.b
      2.00           +50.0%       3.00        vmstat.procs.r
      3502            -7.3%       3246        vmstat.system.cs
  5.72e+09 ± 77%    +125.2%  1.288e+10        cpuidle.C1E.time
  15230307 ± 58%     +67.5%   25512110 ±  9%  cpuidle.C1E.usage
 7.309e+09 ± 55%     -95.5%  3.319e+08 ± 19%  cpuidle.C6.time
   9587605 ± 46%     -97.9%     202885 ± 37%  cpuidle.C6.usage
  1.79e+08 ±170%     -98.7%    2281628 ± 26%  cpuidle.POLL.time
    204680 ±  6%     +25.6%     257009 ±  6%  sched_debug.cfs_rq:/.min_vruntime.max
      2844            -9.2%       2582        sched_debug.cpu.sched_count.avg
      1375 ±  2%      -9.8%       1240        sched_debug.cpu.sched_goidle.avg
      1436            -9.1%       1306        sched_debug.cpu.ttwu_count.avg
    662.09 ±  2%     -40.9%     391.06 ±  2%  sched_debug.cpu.ttwu_local.avg
     27940            +2.1%      28536        proc-vmstat.nr_active_file
   6852781            +5.6%    7234155        proc-vmstat.nr_dirty
  14437609            +2.5%   14803468        proc-vmstat.nr_file_pages
  14134934            +2.6%   14500568        proc-vmstat.nr_inactive_file
      6784            -0.7%       6738        proc-vmstat.nr_mapped
     52131            +1.6%      52983        proc-vmstat.nr_slab_reclaimable
     27940            +2.1%      28536        proc-vmstat.nr_zone_active_file
  14134934            +2.6%   14500568        proc-vmstat.nr_zone_inactive_file
   6873487            +5.5%    7254778        proc-vmstat.nr_zone_write_pending
   3081233 ±  6%     +57.1%    4839628 ±  5%  proc-vmstat.numa_foreign
  13183834           -13.4%   11419080 ±  2%  proc-vmstat.numa_hit
  13152450           -13.4%   11387779 ±  2%  proc-vmstat.numa_local
   3081233 ±  6%     +57.1%    4839628 ±  5%  proc-vmstat.numa_miss
   3112617 ±  5%     +56.5%    4870930 ±  5%  proc-vmstat.numa_other
    197.50 ± 25%     -41.8%     115.00 ± 25%  interrupts.CPU11.NMI:Non-maskable_interrupts
    197.50 ± 25%     -41.8%     115.00 ± 25%  interrupts.CPU11.PMI:Performance_monitoring_interrupts
    191.25 ± 23%     -42.6%     109.75 ± 26%  interrupts.CPU12.NMI:Non-maskable_interrupts
    191.25 ± 23%     -42.6%     109.75 ± 26%  interrupts.CPU12.PMI:Performance_monitoring_interrupts
    173.50 ± 18%     -31.7%     118.50 ± 23%  interrupts.CPU13.NMI:Non-maskable_interrupts
    173.50 ± 18%     -31.7%     118.50 ± 23%  interrupts.CPU13.PMI:Performance_monitoring_interrupts
    191.50 ± 22%     -34.5%     125.50 ± 26%  interrupts.CPU15.NMI:Non-maskable_interrupts
    191.50 ± 22%     -34.5%     125.50 ± 26%  interrupts.CPU15.PMI:Performance_monitoring_interrupts
    202.50 ± 26%     -37.9%     125.75 ± 25%  interrupts.CPU17.NMI:Non-maskable_interrupts
    202.50 ± 26%     -37.9%     125.75 ± 25%  interrupts.CPU17.PMI:Performance_monitoring_interrupts
      2753 ±100%     -99.9%       4.00 ± 39%  interrupts.CPU20.RES:Rescheduling_interrupts
      9094 ±117%     -78.7%       1940 ± 16%  interrupts.CPU33.CAL:Function_call_interrupts
      2836 ± 35%     -31.5%       1943 ± 16%  interrupts.CPU38.CAL:Function_call_interrupts
      6869 ±138%     -99.9%       5.75 ± 85%  interrupts.CPU38.RES:Rescheduling_interrupts
      2347 ± 14%     -17.2%       1943 ± 16%  interrupts.CPU39.CAL:Function_call_interrupts
      3990 ± 75%     -51.4%       1941 ± 16%  interrupts.CPU41.CAL:Function_call_interrupts
      2353 ± 14%     -17.6%       1940 ± 16%  interrupts.CPU45.CAL:Function_call_interrupts
      2325 ± 14%     -16.6%       1938 ± 16%  interrupts.CPU49.CAL:Function_call_interrupts
    955.00 ± 44%     -90.3%      93.00 ±149%  interrupts.CPU5.RES:Rescheduling_interrupts
     29.75 ± 99%   +1516.0%     480.75 ± 94%  interrupts.CPU6.RES:Rescheduling_interrupts
    159.50 ± 15%     -33.5%     106.00 ± 25%  interrupts.CPU63.NMI:Non-maskable_interrupts
    159.50 ± 15%     -33.5%     106.00 ± 25%  interrupts.CPU63.PMI:Performance_monitoring_interrupts
      2322 ± 14%     -17.1%       1924 ± 17%  interrupts.CPU65.CAL:Function_call_interrupts
    281.50 ± 50%     -53.5%     131.00 ± 12%  interrupts.CPU7.NMI:Non-maskable_interrupts
    281.50 ± 50%     -53.5%     131.00 ± 12%  interrupts.CPU7.PMI:Performance_monitoring_interrupts
      2.50 ± 91%   +5730.0%     145.75 ±158%  interrupts.CPU73.RES:Rescheduling_interrupts
    159.75 ± 84%    +145.4%     392.00 ± 36%  interrupts.TLB:TLB_shootdowns
      2.38 ± 42%      -0.9        1.53        perf-stat.i.branch-miss-rate%
      3487            -7.4%       3229        perf-stat.i.context-switches
      1.98 ± 10%     -22.6%       1.53 ± 16%  perf-stat.i.cpu-migrations
      0.14 ± 52%      -0.1        0.01 ± 18%  perf-stat.i.dTLB-load-miss-rate%
    659751 ± 52%     -92.7%      48076 ±  8%  perf-stat.i.dTLB-load-misses
      0.04 ± 55%      -0.0        0.00 ± 12%  perf-stat.i.dTLB-store-miss-rate%
     94475 ± 55%     -90.7%       8798 ± 11%  perf-stat.i.dTLB-store-misses
     55.83 ±  4%      -4.9       50.93        perf-stat.i.iTLB-load-miss-rate%
      2649            -2.3%       2589        perf-stat.i.minor-faults
     81.20 ±  2%      +5.5       86.71        perf-stat.i.node-load-miss-rate%
    484257 ±  7%     +29.8%     628546 ±  7%  perf-stat.i.node-load-misses
     62.30 ± 17%     +20.4       82.69 ±  4%  perf-stat.i.node-store-miss-rate%
    514471 ±  3%     +31.6%     677042 ±  3%  perf-stat.i.node-store-misses
    358788 ±  4%     -18.9%     290959 ±  4%  perf-stat.i.node-stores
      2648            -2.2%       2589        perf-stat.i.page-faults
      0.08 ± 58%      -0.1        0.01 ±  8%  perf-stat.overall.dTLB-load-miss-rate%
      0.03 ± 55%      -0.0        0.00 ± 11%  perf-stat.overall.dTLB-store-miss-rate%
     57.62 ±  3%      -4.4       53.26        perf-stat.overall.iTLB-load-miss-rate%
     84.13            +3.6       87.71        perf-stat.overall.node-load-miss-rate%
     58.90 ±  2%     +11.0       69.93        perf-stat.overall.node-store-miss-rate%
      3462            -7.4%       3206        perf-stat.ps.context-switches
      1.96 ± 10%     -22.6%       1.52 ± 16%  perf-stat.ps.cpu-migrations
    655168 ± 52%     -92.7%      47743 ±  8%  perf-stat.ps.dTLB-load-misses
     93819 ± 55%     -90.7%       8737 ± 11%  perf-stat.ps.dTLB-store-misses
      2630            -2.3%       2571        perf-stat.ps.minor-faults
    480779 ±  7%     +29.8%     624079 ±  7%  perf-stat.ps.node-load-misses
    510737 ±  3%     +31.6%     672190 ±  3%  perf-stat.ps.node-store-misses
    356350 ±  4%     -18.9%     289023 ±  4%  perf-stat.ps.node-stores
      2630            -2.2%       2571        perf-stat.ps.page-faults
     14573 ±  8%     -23.4%      11161 ±  2%  softirqs.CPU0.RCU
     14485 ± 13%     -24.9%      10882 ±  4%  softirqs.CPU1.RCU
     14537 ± 10%     -26.0%      10762 ±  4%  softirqs.CPU10.RCU
     14502 ± 10%     -26.1%      10723 ±  4%  softirqs.CPU11.RCU
     14234 ±  9%     -19.1%      11517        softirqs.CPU12.RCU
     14620 ± 12%     -25.2%      10942        softirqs.CPU13.RCU
     16181 ±  9%     -39.4%       9804 ± 26%  softirqs.CPU14.RCU
     14249 ±  9%     -23.1%      10962 ±  7%  softirqs.CPU15.RCU
     16345 ± 36%     -36.0%      10457 ±  7%  softirqs.CPU16.RCU
     13746 ±  6%     -18.0%      11276 ±  9%  softirqs.CPU17.RCU
     12917 ±  5%     -14.1%      11093 ±  6%  softirqs.CPU18.RCU
     13142 ± 10%     -17.3%      10874 ±  2%  softirqs.CPU19.RCU
     14030 ±  7%     -21.7%      10991        softirqs.CPU2.RCU
     14155 ±  4%     -16.0%      11894 ±  8%  softirqs.CPU20.RCU
     20588 ±  5%      -6.6%      19220        softirqs.CPU20.SCHED
     13883 ±  9%     -21.9%      10842 ±  3%  softirqs.CPU21.RCU
     13766 ±  6%     -17.9%      11298 ±  5%  softirqs.CPU22.RCU
     13611 ±  4%     -19.3%      10988 ±  5%  softirqs.CPU23.RCU
     15069 ±  9%     -16.9%      12527 ± 12%  softirqs.CPU24.RCU
     13603 ±  5%     -19.0%      11015 ±  5%  softirqs.CPU25.RCU
     13213 ±  3%     -19.3%      10659 ±  5%  softirqs.CPU26.RCU
     13608 ±  3%     -21.2%      10719 ±  5%  softirqs.CPU27.RCU
     13584 ±  3%     -20.6%      10783 ±  5%  softirqs.CPU28.RCU
     13707           -19.4%      11043 ±  9%  softirqs.CPU29.RCU
     13609 ±  2%     -19.5%      10949 ±  5%  softirqs.CPU30.RCU
     13826 ±  6%     -19.4%      11137 ±  7%  softirqs.CPU31.RCU
     14447 ±  4%     -23.4%      11060 ±  6%  softirqs.CPU32.RCU
     14353 ±  3%     -21.3%      11291 ±  8%  softirqs.CPU33.RCU
     14954 ±  5%     -24.1%      11357 ±  8%  softirqs.CPU34.RCU
     14775 ±  6%     -24.9%      11102 ±  5%  softirqs.CPU35.RCU
     14473 ±  6%     -23.1%      11125 ±  5%  softirqs.CPU36.RCU
     14422 ±  4%     -23.7%      10997 ±  6%  softirqs.CPU37.RCU
     14698 ±  6%     -23.2%      11293 ±  9%  softirqs.CPU38.RCU
     14324 ±  8%     -19.5%      11537 ±  9%  softirqs.CPU39.RCU
     14767 ±  9%     -21.6%      11582 ±  9%  softirqs.CPU4.RCU
     14756 ±  6%     -24.7%      11116 ±  6%  softirqs.CPU41.RCU
     14452 ±  7%     -22.9%      11143 ±  8%  softirqs.CPU42.RCU
     14440 ±  4%     -22.3%      11220 ±  6%  softirqs.CPU43.RCU
     14323 ± 10%     -22.6%      11081 ±  6%  softirqs.CPU44.RCU
     15165 ±  7%     -26.9%      11090 ±  6%  softirqs.CPU45.RCU
     15026 ±  8%     -26.1%      11108 ±  6%  softirqs.CPU46.RCU
     14623 ±  6%     -21.2%      11521 ± 11%  softirqs.CPU47.RCU
     14161 ±  7%     -23.3%      10861 ±  5%  softirqs.CPU48.RCU
     14237 ± 12%     -21.1%      11227 ±  6%  softirqs.CPU49.RCU
     14550 ±  9%     -23.8%      11082 ±  2%  softirqs.CPU5.RCU
     16170 ±  7%     -27.9%      11654 ± 11%  softirqs.CPU51.RCU
     67978 ± 12%     -21.4%      53439 ± 13%  softirqs.CPU51.TIMER
     15390 ± 17%     -29.1%      10912 ±  5%  softirqs.CPU52.RCU
     14734 ±  9%     -24.5%      11122 ±  2%  softirqs.CPU53.RCU
     14260 ±  9%     -26.3%      10511 ±  6%  softirqs.CPU54.RCU
     14341 ± 10%     -24.2%      10863 ±  5%  softirqs.CPU56.RCU
     14111 ±  9%     -23.5%      10794 ±  5%  softirqs.CPU57.RCU
     14339 ±  9%     -24.8%      10778 ±  6%  softirqs.CPU58.RCU
     14423 ± 10%     -25.2%      10785 ±  5%  softirqs.CPU59.RCU
     14253 ±  8%     -19.3%      11501 ± 18%  softirqs.CPU6.RCU
     14423 ± 10%     -25.1%      10798 ±  4%  softirqs.CPU60.RCU
     14437 ± 10%     -22.5%      11189 ±  7%  softirqs.CPU61.RCU
     14208 ± 11%     -29.0%      10084 ± 12%  softirqs.CPU62.RCU
     13388 ±  7%     -19.9%      10717 ±  2%  softirqs.CPU64.RCU
     13585 ±  7%     -22.9%      10473 ±  4%  softirqs.CPU65.RCU
     12562 ±  6%     -14.6%      10731 ± 11%  softirqs.CPU66.RCU
     12721 ±  6%     -18.9%      10320 ±  5%  softirqs.CPU67.RCU
     13154 ±  5%     -20.2%      10499 ±  3%  softirqs.CPU68.RCU
     13112 ±  6%     -19.9%      10503 ±  4%  softirqs.CPU69.RCU
     14259 ±  8%     -27.6%      10321 ±  9%  softirqs.CPU7.RCU
     13513 ±  6%     -22.1%      10525 ±  3%  softirqs.CPU70.RCU
     13559 ±  6%     -20.5%      10783 ±  3%  softirqs.CPU71.RCU
     13422 ±  5%     -22.9%      10352 ±  3%  softirqs.CPU72.RCU
     13196 ±  4%     -22.1%      10274 ±  5%  softirqs.CPU74.RCU
     13132 ±  4%     -20.6%      10427 ±  4%  softirqs.CPU75.RCU
     13502 ±  2%     -23.3%      10359 ±  6%  softirqs.CPU77.RCU
     13393 ±  4%     -22.2%      10414 ±  5%  softirqs.CPU78.RCU
     13119 ±  3%     -22.6%      10158 ±  6%  softirqs.CPU79.RCU
     14239 ± 10%     -25.4%      10625 ±  5%  softirqs.CPU8.RCU
     14006 ±  5%     -21.6%      10983 ±  8%  softirqs.CPU80.RCU
     13848 ±  5%     -20.5%      11012 ±  8%  softirqs.CPU81.RCU
     14386 ±  4%     -22.4%      11169 ±  6%  softirqs.CPU82.RCU
     14429 ± 10%     -24.4%      10902 ±  6%  softirqs.CPU83.RCU
     14024 ±  6%     -20.2%      11198 ± 11%  softirqs.CPU84.RCU
     14124 ±  5%     -21.4%      11100 ±  8%  softirqs.CPU85.RCU
     14448 ±  6%     -25.5%      10769 ±  7%  softirqs.CPU86.RCU
     13965 ±  7%     -23.5%      10681 ±  6%  softirqs.CPU87.RCU
     13792 ±  8%     -21.7%      10802 ±  5%  softirqs.CPU88.RCU
     14454 ±  6%     -23.8%      11021 ±  8%  softirqs.CPU89.RCU
     14117 ±  8%     -22.7%      10910 ±  4%  softirqs.CPU9.RCU
     13597 ±  6%     -21.3%      10696 ±  8%  softirqs.CPU90.RCU
     13972 ±  6%     -23.5%      10693 ±  5%  softirqs.CPU91.RCU
     13956 ± 12%     -23.2%      10713 ±  7%  softirqs.CPU92.RCU
     66146 ± 12%     -18.1%      54161 ±  5%  softirqs.CPU92.TIMER
     14347 ±  7%     -25.6%      10668 ±  7%  softirqs.CPU93.RCU
     14787 ±  7%     -26.9%      10810 ±  6%  softirqs.CPU94.RCU
     14463 ±  5%     -25.1%      10838 ±  8%  softirqs.CPU95.RCU
   1353112 ±  5%     -22.3%    1050993 ±  4%  softirqs.RCU
     47.02 ±  9%      -9.1       37.95        perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     79.13 ±  3%      -5.9       73.22        perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
     80.01 ±  2%      -5.6       74.36        perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      1.11 ± 18%      +0.4        1.47 ±  8%  perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
      1.27 ± 20%      +0.4        1.69 ±  5%  perf-profile.calltrace.cycles-pp.native_write_msr.lapic_next_deadline.clockevents_program_event.hrtimer_interrupt.smp_apic_timer_interrupt
      0.32 ±101%      +0.4        0.76 ± 11%  perf-profile.calltrace.cycles-pp.calc_global_load_tick.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer
      0.51 ± 63%      +0.5        0.99 ± 17%  perf-profile.calltrace.cycles-pp.ktime_get.tick_irq_enter.irq_enter.smp_apic_timer_interrupt.apic_timer_interrupt
      0.14 ±173%      +0.5        0.64 ± 11%  perf-profile.calltrace.cycles-pp._raw_spin_trylock.rebalance_domains.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt
      1.52 ± 20%      +0.5        2.03 ±  5%  perf-profile.calltrace.cycles-pp.lapic_next_deadline.clockevents_program_event.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      0.52 ± 62%      +0.5        1.04 ± 16%  perf-profile.calltrace.cycles-pp.ktime_get_update_offsets_now.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
      3.18 ±  8%      +0.6        3.74 ±  8%  perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt
      2.97 ± 10%      +0.6        3.61 ±  8%  perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
      0.00            +0.7        0.66 ± 14%  perf-profile.calltrace.cycles-pp.ktime_get.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt
      0.97 ± 74%      +1.3        2.22 ± 22%  perf-profile.calltrace.cycles-pp.ktime_get.tick_nohz_irq_exit.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt
      1.01 ± 72%      +1.3        2.27 ± 22%  perf-profile.calltrace.cycles-pp.tick_nohz_irq_exit.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
      7.13 ±  8%      +1.4        8.48 ±  4%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
      4.00 ±  8%      +1.4        5.42 ±  7%  perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      5.37 ± 11%      +1.6        6.94 ±  8%  perf-profile.calltrace.cycles-pp.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      5.57 ± 17%      +1.6        7.18 ±  8%  perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry.start_secondary
      4.38 ± 19%      +1.6        5.99 ± 11%  perf-profile.calltrace.cycles-pp.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry
      9.86 ± 16%      +3.1       12.96 ±  2%  perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     13.67 ± 19%      +4.5       18.16 ±  2%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
     21.15 ± 12%      +6.4       27.60        perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle
     26.11 ± 15%      +7.8       33.89        perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     47.35 ±  9%      -9.0       38.31        perf-profile.children.cycles-pp.intel_idle
     80.89 ±  3%      -5.7       75.19        perf-profile.children.cycles-pp.cpuidle_enter
     80.81 ±  3%      -5.7       75.12        perf-profile.children.cycles-pp.cpuidle_enter_state
      0.49 ± 45%      -0.2        0.30 ± 12%  perf-profile.children.cycles-pp.timerqueue_add
      0.22 ± 32%      -0.1        0.13 ± 15%  perf-profile.children.cycles-pp.rb_insert_color
      0.18 ± 12%      -0.1        0.11 ± 29%  perf-profile.children.cycles-pp.account_process_tick
      0.17 ± 25%      -0.1        0.11 ± 13%  perf-profile.children.cycles-pp.run_posix_cpu_timers
      0.20 ± 14%      -0.1        0.14 ± 11%  perf-profile.children.cycles-pp.drm_fb_helper_dirty_work
      0.15 ± 18%      -0.0        0.10 ± 18%  perf-profile.children.cycles-pp.rb_erase
      0.06 ± 15%      +0.0        0.07 ± 14%  perf-profile.children.cycles-pp.schedule
      0.04 ± 60%      +0.0        0.08 ± 23%  perf-profile.children.cycles-pp._raw_write_lock
      0.07 ± 13%      +0.0        0.10 ± 28%  perf-profile.children.cycles-pp.execve
      0.07 ± 17%      +0.0        0.11 ± 18%  perf-profile.children.cycles-pp.tick_check_broadcast_expired
      0.03 ±100%      +0.0        0.07 ± 17%  perf-profile.children.cycles-pp.__fprop_inc_percpu_max
      0.10 ± 12%      +0.0        0.15 ±  5%  perf-profile.children.cycles-pp.__might_sleep
      0.03 ±100%      +0.1        0.08 ± 23%  perf-profile.children.cycles-pp.__libc_fork
      0.10 ± 36%      +0.1        0.16 ± 17%  perf-profile.children.cycles-pp.up_read
      0.22 ± 14%      +0.1        0.28 ±  9%  perf-profile.children.cycles-pp.idle_cpu
      0.12 ± 21%      +0.1        0.20 ± 14%  perf-profile.children.cycles-pp.raise_softirq
      0.16 ± 15%      +0.1        0.23 ± 16%  perf-profile.children.cycles-pp.down_read
      0.01 ±173%      +0.1        0.09 ± 45%  perf-profile.children.cycles-pp.timekeeping_advance
      0.22 ± 17%      +0.1        0.30 ±  7%  perf-profile.children.cycles-pp.menu_reflect
      0.06 ± 11%      +0.1        0.16 ± 77%  perf-profile.children.cycles-pp.mmput
      0.15 ± 36%      +0.1        0.26 ± 15%  perf-profile.children.cycles-pp.read_node_page
      0.19 ± 33%      +0.1        0.29 ± 18%  perf-profile.children.cycles-pp.___might_sleep
      0.36 ± 20%      +0.1        0.46 ±  6%  perf-profile.children.cycles-pp.rcu_core
      0.19 ± 23%      +0.1        0.29 ±  7%  perf-profile.children.cycles-pp.trigger_load_balance
      0.43 ± 18%      +0.1        0.54 ±  6%  perf-profile.children.cycles-pp.__intel_pmu_enable_all
      0.02 ±173%      +0.1        0.13 ± 28%  perf-profile.children.cycles-pp.tick_sched_do_timer
      0.37 ± 25%      +0.1        0.48 ± 15%  perf-profile.children.cycles-pp.clear_page_dirty_for_io
      0.16 ± 18%      +0.1        0.28 ± 10%  perf-profile.children.cycles-pp.mutex_lock
      0.09 ±108%      +0.1        0.23 ± 26%  perf-profile.children.cycles-pp.wake_up_klogd_work_func
      0.37 ± 20%      +0.2        0.56 ± 11%  perf-profile.children.cycles-pp.f2fs_submit_page_write
      0.43 ± 31%      +0.3        0.71 ±  9%  perf-profile.children.cycles-pp._raw_spin_trylock
      0.11 ± 23%      +0.3        0.40 ± 32%  perf-profile.children.cycles-pp.hrtimer_forward
      0.90 ± 18%      +0.3        1.21 ±  9%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      1.15 ± 18%      +0.4        1.52 ±  8%  perf-profile.children.cycles-pp.scheduler_tick
      0.40 ± 58%      +0.4        0.77 ± 12%  perf-profile.children.cycles-pp.calc_global_load_tick
      0.62 ± 29%      +0.4        1.05 ± 16%  perf-profile.children.cycles-pp.ktime_get_update_offsets_now
      1.18 ± 17%      +0.4        1.61 ±  8%  perf-profile.children.cycles-pp.read_tsc
      1.56 ± 18%      +0.5        2.07 ±  6%  perf-profile.children.cycles-pp.lapic_next_deadline
      1.75 ± 19%      +0.5        2.29 ±  5%  perf-profile.children.cycles-pp.native_write_msr
      3.23 ±  8%      +0.6        3.88 ±  7%  perf-profile.children.cycles-pp.tick_sched_handle
      3.04 ± 10%      +0.7        3.76 ±  7%  perf-profile.children.cycles-pp.update_process_times
      1.15 ± 49%      +1.2        2.31 ± 21%  perf-profile.children.cycles-pp.tick_nohz_irq_exit
      7.28 ±  8%      +1.4        8.72 ±  4%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      4.07 ±  8%      +1.5        5.59 ±  6%  perf-profile.children.cycles-pp.tick_sched_timer
      5.64 ± 17%      +1.6        7.29 ±  8%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      4.46 ± 19%      +1.6        6.10 ± 10%  perf-profile.children.cycles-pp.tick_nohz_next_event
      6.47 ± 16%      +1.8        8.28 ±  7%  perf-profile.children.cycles-pp.irq_exit
     10.02 ± 16%      +3.2       13.17 ±  2%  perf-profile.children.cycles-pp.menu_select
      5.46 ± 48%      +3.9        9.37 ±  5%  perf-profile.children.cycles-pp.ktime_get
     13.91 ± 18%      +4.6       18.54 ±  2%  perf-profile.children.cycles-pp.hrtimer_interrupt
     21.48 ± 12%      +6.5       27.95        perf-profile.children.cycles-pp.smp_apic_timer_interrupt
     24.16 ± 13%      +7.2       31.34        perf-profile.children.cycles-pp.apic_timer_interrupt
     47.25 ±  9%      -9.0       38.30        perf-profile.self.cycles-pp.intel_idle
      0.19 ± 36%      -0.1        0.11 ± 25%  perf-profile.self.cycles-pp.rb_insert_color
      0.24 ± 36%      -0.1        0.15 ± 14%  perf-profile.self.cycles-pp.rb_next
      0.18 ± 11%      -0.1        0.11 ± 29%  perf-profile.self.cycles-pp.account_process_tick
      0.14 ± 23%      -0.1        0.08 ± 21%  perf-profile.self.cycles-pp.rb_erase
      0.08 ± 20%      -0.0        0.03 ±100%  perf-profile.self.cycles-pp.cpuidle_reflect
      0.04 ± 60%      +0.0        0.08 ± 23%  perf-profile.self.cycles-pp._raw_write_lock
      0.10 ± 22%      +0.0        0.14 ± 11%  perf-profile.self.cycles-pp.cpuidle_governor_latency_req
      0.10 ± 15%      +0.0        0.14 ±  7%  perf-profile.self.cycles-pp.__might_sleep
      0.06 ± 11%      +0.0        0.11 ± 16%  perf-profile.self.cycles-pp.tick_check_broadcast_expired
      0.10 ± 36%      +0.1        0.16 ± 14%  perf-profile.self.cycles-pp.up_read
      0.15 ± 23%      +0.1        0.22 ± 10%  perf-profile.self.cycles-pp.menu_reflect
      0.08 ± 15%      +0.1        0.15 ± 14%  perf-profile.self.cycles-pp.mutex_lock
      0.12 ± 22%      +0.1        0.19 ± 19%  perf-profile.self.cycles-pp.raise_softirq
      0.15 ± 36%      +0.1        0.25 ± 13%  perf-profile.self.cycles-pp.trigger_load_balance
      0.17 ± 31%      +0.1        0.27 ± 19%  perf-profile.self.cycles-pp.f2fs_do_write_data_page
      0.11 ± 36%      +0.1        0.21 ± 18%  perf-profile.self.cycles-pp.read_node_page
      0.10 ± 15%      +0.1        0.21 ± 18%  perf-profile.self.cycles-pp.pagecache_get_page
      0.19 ± 32%      +0.1        0.29 ± 16%  perf-profile.self.cycles-pp.___might_sleep
      0.33 ± 20%      +0.1        0.44 ±  8%  perf-profile.self.cycles-pp.hrtimer_interrupt
      0.33 ± 24%      +0.2        0.54 ± 16%  perf-profile.self.cycles-pp.tick_sched_timer
      0.92 ± 22%      +0.2        1.17 ±  4%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.38 ±  8%      +0.3        0.64 ± 17%  perf-profile.self.cycles-pp.__softirqentry_text_start
      0.11 ± 23%      +0.3        0.39 ± 32%  perf-profile.self.cycles-pp.hrtimer_forward
      0.43 ± 31%      +0.3        0.71 ±  9%  perf-profile.self.cycles-pp._raw_spin_trylock
      0.55 ± 10%      +0.3        0.84 ± 21%  perf-profile.self.cycles-pp.tick_nohz_next_event
      0.45 ± 25%      +0.3        0.79 ± 22%  perf-profile.self.cycles-pp.ktime_get_update_offsets_now
      0.40 ± 61%      +0.3        0.74 ± 12%  perf-profile.self.cycles-pp.calc_global_load_tick
      1.14 ± 17%      +0.4        1.57 ±  7%  perf-profile.self.cycles-pp.read_tsc
      1.75 ± 19%      +0.5        2.28 ±  5%  perf-profile.self.cycles-pp.native_write_msr
      3.62 ± 23%      +1.6        5.21 ±  4%  perf-profile.self.cycles-pp.menu_select
      4.41 ± 58%      +3.6        7.99 ±  7%  perf-profile.self.cycles-pp.ktime_get





Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.4.0-rc7-00040-ge4f9eb2268da0" of type "text/plain" (200680 bytes)

View attachment "job-script" of type "text/plain" (8158 bytes)

View attachment "job.yaml" of type "text/plain" (5812 bytes)

View attachment "reproduce" of type "text/plain" (1051 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ