lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite for Android: free password hash cracker in your pocket
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date:   Sat, 30 Nov 2019 20:23:07 +0000
From:   Mariusz Ceier <mceier@...il.com>
To:     kernel test robot <rong.a.chen@...el.com>
Cc:     Davidlohr Bueso <dave@...olabs.net>,
        Ingo Molnar <mingo@...nel.org>,
        Davidlohr Bueso <dbueso@...e.de>,
        Thomas Gleixner <tglx@...utronix.de>,
        Peter Zijlstra <peterz@...radead.org>,
        Borislav Petkov <bp@...en8.de>,
        Linus Torvalds <torvalds@...ux-foundation.org>,
        LKML <linux-kernel@...r.kernel.org>, lkp@...ts.01.org
Subject: Re: [x86/mm/pat] 8d04a5f97a: phoronix-test-suite.glmark2.0.score
 -23.7% regression

I can also confirm this - just bisected framebuffer rendering
performance regression on amdgpu and
8d04a5f97a5fa9d7afdf46eda3a5ceaa973a1bcc is the first bad commit
(leading to drop from around 260-300fps to about 60fps in CS:GO on
Fury X).

PS. Sorry for duplicate email (sending plain text only now, hopefully).

On Wed, 27 Nov 2019 at 00:56, kernel test robot <rong.a.chen@...el.com> wrote:
>
> Greeting,
>
> FYI, we noticed a -23.7% regression of phoronix-test-suite.glmark2.0.score due to commit:
>
>
> commit: 8d04a5f97a5fa9d7afdf46eda3a5ceaa973a1bcc ("x86/mm/pat: Convert the PAT tree to a generic interval tree")
> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
>
> in testcase: phoronix-test-suite
> on test machine: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 192G memory
> with following parameters:
>
>         need_x: true
>         test: glmark2-1.1.0
>         cpufreq_governor: performance
>         ucode: 0x500002b
>
> test-description: The Phoronix Test Suite is the most comprehensive testing and benchmarking platform available that provides an extensible framework for which new tests can be easily added.
> test-url: http://www.phoronix-test-suite.com/
>
> In addition to that, the commit also has significant impact on the following tests:
>
> +------------------+-----------------------------------------------------------------------------+
> | testcase: change | phoronix-test-suite: phoronix-test-suite.gtkperf.0.seconds 26.8% regression |
> | test machine     | 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 192G memory        |
> | test parameters  | cpufreq_governor=performance                                                |
> |                  | need_x=true                                                                 |
> |                  | test=gtkperf-1.2.1                                                          |
> |                  | ucode=0x500002b                                                             |
> +------------------+-----------------------------------------------------------------------------+
>
>
> If you fix the issue, kindly add following tag
> Reported-by: kernel test robot <rong.a.chen@...el.com>
>
>
> Details are as below:
> -------------------------------------------------------------------------------------------------->
>
>
> To reproduce:
>
>         git clone https://github.com/intel/lkp-tests.git
>         cd lkp-tests
>         bin/lkp install job.yaml  # job file is attached in this email
>         bin/lkp run     job.yaml
>
> =========================================================================================
> compiler/cpufreq_governor/kconfig/need_x/rootfs/tbox_group/test/testcase/ucode:
>   gcc-7/performance/x86_64-rhel-7.6/true/debian-x86_64-phoronix/lkp-csl-2sp8/glmark2-1.1.0/phoronix-test-suite/0x500002b
>
> commit:
>   9f4813b531 (" Linux 5.4-rc8")
>   8d04a5f97a ("x86/mm/pat: Convert the PAT tree to a generic interval tree")
>
> 9f4813b531a0b8cc 8d04a5f97a5fa9d7afdf46eda3a
> ---------------- ---------------------------
>        fail:runs  %reproduction    fail:runs
>            |             |             |
>           1:4          -25%            :4     dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x
>           2:4          -50%            :4     dmesg.WARNING:stack_recursion
>          %stddev     %change         %stddev
>              \          |                \
>     290.50           -23.7%     221.75 ±  4%  phoronix-test-suite.glmark2.0.score
>     210.25           -24.1%     159.50 ±  4%  phoronix-test-suite.glmark2.1.score
>      35084           -19.8%      28122 ±  3%  vmstat.system.cs
>      16157            +1.6%      16415        proc-vmstat.nr_kernel_stack
>      17551            +2.0%      17909        proc-vmstat.nr_slab_reclaimable
>      46846            +2.4%      47962        proc-vmstat.nr_slab_unreclaimable
>     156401 ± 15%     -19.5%     125917 ± 11%  proc-vmstat.numa_hint_faults_local
>    2091859            -2.4%    2041250 ±  2%  proc-vmstat.pgfault
>      72.57 ±  2%     -10.0       62.61 ± 19%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
>       1.25 ±  8%      -0.5        0.79 ± 58%  perf-profile.calltrace.cycles-pp.irq_enter.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
>       1.04 ± 10%      -0.4        0.66 ± 58%  perf-profile.calltrace.cycles-pp.tick_irq_enter.irq_enter.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
>      72.70 ±  2%      -9.9       62.81 ± 19%  perf-profile.children.cycles-pp.intel_idle
>       1.28 ±  8%      -0.3        0.94 ± 30%  perf-profile.children.cycles-pp.irq_enter
>       1.08 ±  9%      -0.3        0.78 ± 28%  perf-profile.children.cycles-pp.tick_irq_enter
>       0.47 ± 14%      -0.2        0.32 ± 39%  perf-profile.children.cycles-pp.rcu_sched_clock_irq
>       0.48 ±  4%      -0.1        0.37 ± 32%  perf-profile.children.cycles-pp.futex_wait_queue_me
>       0.16 ± 11%      -0.1        0.06 ± 63%  perf-profile.children.cycles-pp.perf_event_task_tick
>       0.20 ±  5%      -0.1        0.10 ± 62%  perf-profile.children.cycles-pp.rb_next
>       0.11 ± 15%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.switch_mm_irqs_off
>       0.18 ±  7%      -0.1        0.10 ± 61%  perf-profile.children.cycles-pp.update_ts_time_stats
>      72.59 ±  2%      -9.8       62.77 ± 19%  perf-profile.self.cycles-pp.intel_idle
>       0.16 ± 14%      -0.1        0.06 ± 63%  perf-profile.self.cycles-pp.perf_event_task_tick
>       0.18 ±  6%      -0.1        0.10 ± 65%  perf-profile.self.cycles-pp.rb_next
>   26427524           -30.2%   18439861 ± 11%  perf-stat.i.branch-misses
>  1.264e+08           -32.7%   85112641 ± 15%  perf-stat.i.cache-references
>      35249           -19.8%      28261 ±  3%  perf-stat.i.context-switches
>    1663983 ± 10%     -47.0%     881904 ± 41%  perf-stat.i.dTLB-load-misses
>   2.55e+09 ±  3%     -29.5%  1.799e+09 ± 13%  perf-stat.i.dTLB-loads
>  1.123e+09 ±  2%      -6.9%  1.046e+09 ±  3%  perf-stat.i.dTLB-stores
>  1.112e+10           -15.7%  9.376e+09 ±  4%  perf-stat.i.instructions
>       6700           -21.1%       5289 ±  7%  perf-stat.i.instructions-per-iTLB-miss
>      92755 ±  4%      -5.2%      87901        perf-stat.i.node-loads
>       6929           -20.3%       5525 ±  7%  perf-stat.overall.instructions-per-iTLB-miss
>   26388462           -30.2%   18414499 ± 11%  perf-stat.ps.branch-misses
>  1.262e+08           -32.7%   84986397 ± 15%  perf-stat.ps.cache-references
>      35191           -19.8%      28218 ±  3%  perf-stat.ps.context-switches
>    1661390 ± 10%     -47.0%     880613 ± 41%  perf-stat.ps.dTLB-load-misses
>  2.546e+09 ±  3%     -29.4%  1.796e+09 ± 13%  perf-stat.ps.dTLB-loads
>  1.122e+09 ±  2%      -6.8%  1.045e+09 ±  3%  perf-stat.ps.dTLB-stores
>  1.111e+10           -15.7%  9.362e+09 ±  4%  perf-stat.ps.instructions
>      92587 ±  4%      -5.2%      87778        perf-stat.ps.node-loads
>  7.805e+12           -16.3%  6.535e+12 ±  4%  perf-stat.total.instructions
>     421893 ± 13%     -33.5%     280495 ± 17%  numa-meminfo.node0.Active
>     301962 ± 18%     -24.7%     227271 ±  3%  numa-meminfo.node0.Active(anon)
>     119930           -55.6%      53224 ± 75%  numa-meminfo.node0.Active(file)
>     289371 ± 19%     -23.6%     220986 ±  3%  numa-meminfo.node0.AnonPages
>    1599232           -46.1%     861252 ± 63%  numa-meminfo.node0.FilePages
>    1467294           -45.5%     800320 ± 63%  numa-meminfo.node0.Inactive
>     406871           -60.9%     159255 ± 92%  numa-meminfo.node0.Inactive(file)
>      10042 ±  2%     -14.2%       8615 ± 12%  numa-meminfo.node0.KernelStack
>     109027           -67.1%      35838 ± 78%  numa-meminfo.node0.Mapped
>    2488468 ±  2%     -36.4%    1582090 ± 41%  numa-meminfo.node0.MemUsed
>       4081 ±  3%     -53.1%       1913 ± 56%  numa-meminfo.node0.PageTables
>     169752 ±  3%     -15.8%     143010 ± 19%  numa-meminfo.node0.Slab
>     183294 ± 31%     +78.6%     327336 ± 16%  numa-meminfo.node1.Active
>     176459 ± 32%     +44.1%     254351 ±  4%  numa-meminfo.node1.Active(anon)
>       6834 ± 11%    +967.9%      72984 ± 57%  numa-meminfo.node1.Active(file)
>     171600 ± 33%     +42.0%     243637 ±  2%  numa-meminfo.node1.AnonPages
>      18539 ±  5%   +3986.1%     757532 ± 72%  numa-meminfo.node1.FilePages
>       6284 ± 14%  +10623.2%     673848 ± 75%  numa-meminfo.node1.Inactive
>       3812 ± 13%   +6513.3%     252099 ± 58%  numa-meminfo.node1.Inactive(file)
>       4526 ± 18%   +1622.5%      77962 ± 35%  numa-meminfo.node1.Mapped
>     477729 ± 11%    +190.4%    1387370 ± 48%  numa-meminfo.node1.MemUsed
>     704.00 ± 20%    +307.6%       2869 ± 37%  numa-meminfo.node1.PageTables
>      75486 ± 18%     -24.7%      56818 ±  3%  numa-vmstat.node0.nr_active_anon
>      29985           -55.6%      13304 ± 75%  numa-vmstat.node0.nr_active_file
>      72338 ± 19%     -23.6%      55246 ±  3%  numa-vmstat.node0.nr_anon_pages
>     399807           -46.1%     215312 ± 63%  numa-vmstat.node0.nr_file_pages
>     101715           -60.9%      39814 ± 92%  numa-vmstat.node0.nr_inactive_file
>      10043 ±  2%     -14.2%       8614 ± 12%  numa-vmstat.node0.nr_kernel_stack
>      27254           -67.1%       8959 ± 78%  numa-vmstat.node0.nr_mapped
>       1019 ±  3%     -53.2%     477.50 ± 56%  numa-vmstat.node0.nr_page_table_pages
>      75486 ± 18%     -24.7%      56818 ±  3%  numa-vmstat.node0.nr_zone_active_anon
>      29985           -55.6%      13304 ± 75%  numa-vmstat.node0.nr_zone_active_file
>     101715           -60.9%      39814 ± 92%  numa-vmstat.node0.nr_zone_inactive_file
>    1483935 ±  6%     -18.2%    1214502 ± 21%  numa-vmstat.node0.numa_local
>      44115 ± 32%     +44.2%      63599 ±  4%  numa-vmstat.node1.nr_active_anon
>       1708 ± 11%    +968.1%      18243 ± 57%  numa-vmstat.node1.nr_active_file
>      42900 ± 33%     +42.0%      60922 ±  2%  numa-vmstat.node1.nr_anon_pages
>       4635 ±  5%   +3985.9%     189382 ± 72%  numa-vmstat.node1.nr_file_pages
>     952.75 ± 13%   +6515.2%      63026 ± 58%  numa-vmstat.node1.nr_inactive_file
>       1131 ± 18%   +1623.5%      19492 ± 35%  numa-vmstat.node1.nr_mapped
>     176.00 ± 20%    +307.4%     717.00 ± 37%  numa-vmstat.node1.nr_page_table_pages
>      44115 ± 32%     +44.2%      63599 ±  4%  numa-vmstat.node1.nr_zone_active_anon
>       1708 ± 11%    +968.1%      18243 ± 57%  numa-vmstat.node1.nr_zone_active_file
>     952.75 ± 13%   +6515.2%      63026 ± 58%  numa-vmstat.node1.nr_zone_inactive_file
>       1546 ±  3%     +34.6%       2081 ±  6%  slabinfo.UNIX.active_objs
>       1546 ±  3%     +34.6%       2081 ±  6%  slabinfo.UNIX.num_objs
>       2089 ±  4%     +11.9%       2337 ±  7%  slabinfo.dmaengine-unmap-16.active_objs
>       2089 ±  4%     +11.9%       2337 ±  7%  slabinfo.dmaengine-unmap-16.num_objs
>      21286 ±  4%     +10.5%      23516 ±  5%  slabinfo.filp.active_objs
>      21594 ±  4%     +10.8%      23932 ±  4%  slabinfo.filp.num_objs
>       3632 ±  7%     +15.4%       4190 ±  7%  slabinfo.mnt_cache.active_objs
>       3632 ±  7%     +15.4%       4190 ±  7%  slabinfo.mnt_cache.num_objs
>       3343 ±  4%     +23.1%       4116 ±  7%  slabinfo.numa_policy.active_objs
>       3343 ±  4%     +23.1%       4116 ±  7%  slabinfo.numa_policy.num_objs
>       5711 ±  3%     +10.0%       6282 ±  4%  slabinfo.ovl_inode.active_objs
>       5711 ±  3%     +10.0%       6282 ±  4%  slabinfo.ovl_inode.num_objs
>       2143 ±  4%     +15.8%       2481 ±  7%  slabinfo.pool_workqueue.active_objs
>       2143 ±  4%     +15.8%       2481 ±  7%  slabinfo.pool_workqueue.num_objs
>       2309 ±  8%     +14.7%       2648 ±  6%  slabinfo.skbuff_ext_cache.active_objs
>       2309 ±  8%     +14.7%       2648 ±  6%  slabinfo.skbuff_ext_cache.num_objs
>       2785 ±  3%     +21.8%       3390 ±  4%  slabinfo.sock_inode_cache.active_objs
>       2785 ±  3%     +21.8%       3390 ±  4%  slabinfo.sock_inode_cache.num_objs
>       1300 ±  3%     +15.3%       1500 ±  8%  slabinfo.task_group.active_objs
>       1300 ±  3%     +15.3%       1500 ±  8%  slabinfo.task_group.num_objs
>     471.00 ± 12%     -33.1%     315.25 ± 31%  slabinfo.xfrm_state.active_objs
>     471.00 ± 12%     -33.1%     315.25 ± 31%  slabinfo.xfrm_state.num_objs
>      15189 ±  4%     -16.9%      12626 ±  4%  sched_debug.cfs_rq:/.exec_clock.avg
>      54802 ±  5%     -20.8%      43426 ±  7%  sched_debug.cfs_rq:/.load.avg
>      59.91 ± 51%     -38.0%      37.16 ±  5%  sched_debug.cfs_rq:/.load_avg.avg
>       2493 ±112%     -68.5%     784.19 ±  3%  sched_debug.cfs_rq:/.load_avg.max
>     320.29 ± 88%     -55.3%     143.18 ±  2%  sched_debug.cfs_rq:/.load_avg.stddev
>      68001 ±  4%     -12.7%      59359 ±  3%  sched_debug.cfs_rq:/.min_vruntime.avg
>      67912           -18.0%      55681 ±  8%  sched_debug.cfs_rq:/.min_vruntime.stddev
>       0.11 ±  5%     -26.6%       0.08 ±  7%  sched_debug.cfs_rq:/.nr_running.avg
>       0.30 ±  2%     -11.0%       0.27 ±  3%  sched_debug.cfs_rq:/.nr_running.stddev
>      28.47 ±  6%     -16.1%      23.89 ±  6%  sched_debug.cfs_rq:/.runnable_load_avg.avg
>     754.35 ±  4%      -8.9%     687.54 ±  6%  sched_debug.cfs_rq:/.runnable_load_avg.max
>     127.97 ±  4%     -10.0%     115.24 ±  6%  sched_debug.cfs_rq:/.runnable_load_avg.stddev
>      54698 ±  5%     -20.6%      43419 ±  7%  sched_debug.cfs_rq:/.runnable_weight.avg
>      44849 ± 15%     -52.5%      21314 ± 60%  sched_debug.cfs_rq:/.spread0.avg
>      67914           -18.0%      55682 ±  8%  sched_debug.cfs_rq:/.spread0.stddev
>     110.05 ±  2%     -20.3%      87.72 ±  2%  sched_debug.cfs_rq:/.util_avg.avg
>      22.41 ±  9%     -27.7%      16.21 ± 10%  sched_debug.cfs_rq:/.util_est_enqueued.avg
>       0.05 ±  4%     -10.7%       0.05 ±  4%  sched_debug.cpu.nr_running.avg
>     134130           -20.1%     107139 ±  3%  sched_debug.cpu.nr_switches.avg
>     764365 ±  4%     -25.1%     572854 ±  5%  sched_debug.cpu.nr_switches.max
>     212569 ±  2%     -22.1%     165622 ±  8%  sched_debug.cpu.nr_switches.stddev
>     131734           -20.5%     104682 ±  3%  sched_debug.cpu.sched_count.avg
>     760935 ±  5%     -25.2%     569239 ±  5%  sched_debug.cpu.sched_count.max
>     212271 ±  2%     -22.0%     165483 ±  8%  sched_debug.cpu.sched_count.stddev
>      65857           -20.5%      52334 ±  3%  sched_debug.cpu.sched_goidle.avg
>     380419 ±  5%     -25.2%     284602 ±  5%  sched_debug.cpu.sched_goidle.max
>     106128 ±  2%     -22.0%      82735 ±  8%  sched_debug.cpu.sched_goidle.stddev
>      65779           -20.6%      52253 ±  3%  sched_debug.cpu.ttwu_count.avg
>     164575 ±  4%     -15.1%     139787 ± 12%  sched_debug.cpu.ttwu_count.stddev
>     925.77           +14.7%       1061 ±  4%  sched_debug.cpu.ttwu_local.avg
>      95.00 ± 81%     -96.3%       3.50 ± 91%  interrupts.48:PCI-MSI.31981581-edge.i40e-eth0-TxRx-12
>      81.00 ±150%    -100.0%       0.00        interrupts.59:PCI-MSI.31981592-edge.i40e-eth0-TxRx-23
>       0.00       +4.9e+105%       4915 ±172%  interrupts.76:PCI-MSI.31981609-edge.i40e-eth0-TxRx-40
>      71.75 ±167%     -99.7%       0.25 ±173%  interrupts.82:PCI-MSI.31981615-edge.i40e-eth0-TxRx-46
>       1278 ± 29%   +1192.4%      16520 ±127%  interrupts.CPU1.RES:Rescheduling_interrupts
>     410.50 ± 26%     -94.5%      22.50 ±173%  interrupts.CPU10.TLB:TLB_shootdowns
>      95.00 ± 81%     -97.4%       2.50 ±128%  interrupts.CPU12.48:PCI-MSI.31981581-edge.i40e-eth0-TxRx-12
>     218.50 ± 82%     -91.5%      18.50 ±173%  interrupts.CPU12.TLB:TLB_shootdowns
>       1142           -64.1%     409.75 ± 92%  interrupts.CPU13.NMI:Non-maskable_interrupts
>       1142           -64.1%     409.75 ± 92%  interrupts.CPU13.PMI:Performance_monitoring_interrupts
>      11287 ±167%     -98.6%     154.00 ±163%  interrupts.CPU13.RES:Rescheduling_interrupts
>       1141 ±  2%     -49.4%     578.00 ± 72%  interrupts.CPU15.NMI:Non-maskable_interrupts
>       1141 ±  2%     -49.4%     578.00 ± 72%  interrupts.CPU15.PMI:Performance_monitoring_interrupts
>       1124 ±  2%     -67.4%     366.75 ±115%  interrupts.CPU16.NMI:Non-maskable_interrupts
>       1124 ±  2%     -67.4%     366.75 ±115%  interrupts.CPU16.PMI:Performance_monitoring_interrupts
>     315.50 ± 62%     -69.3%      96.75 ±173%  interrupts.CPU17.TLB:TLB_shootdowns
>     137.25 ± 63%     -96.5%       4.75 ±161%  interrupts.CPU18.TLB:TLB_shootdowns
>     160.00 ±170%     -99.5%       0.75 ±110%  interrupts.CPU2.38:PCI-MSI.31981571-edge.i40e-eth0-TxRx-2
>     506.25 ± 30%     -77.4%     114.50 ± 84%  interrupts.CPU2.TLB:TLB_shootdowns
>     241.25 ± 78%     -99.1%       2.25 ±173%  interrupts.CPU21.TLB:TLB_shootdowns
>     295.00 ± 56%     -79.1%      61.75 ±169%  interrupts.CPU22.TLB:TLB_shootdowns
>     723.25 ± 63%     -84.0%     116.00 ±167%  interrupts.CPU23.RES:Rescheduling_interrupts
>      24258 ± 60%     -96.5%     848.00 ±  5%  interrupts.CPU24.RES:Rescheduling_interrupts
>     254.75 ± 27%    +109.6%     534.00 ± 54%  interrupts.CPU24.TLB:TLB_shootdowns
>     124.25 ± 75%    +178.5%     346.00 ± 32%  interrupts.CPU25.TLB:TLB_shootdowns
>      70.00 ±173%    +362.9%     324.00 ± 89%  interrupts.CPU27.TLB:TLB_shootdowns
>      41.75 ± 84%  +1.9e+05%      77527 ±171%  interrupts.CPU28.RES:Rescheduling_interrupts
>       1088 ± 22%  +12337.9%     135324 ± 82%  interrupts.CPU3.RES:Rescheduling_interrupts
>     546.00 ± 39%     -87.0%      71.25 ± 68%  interrupts.CPU4.TLB:TLB_shootdowns
>      26.25 ± 65%   +2721.0%     740.50 ± 82%  interrupts.CPU43.RES:Rescheduling_interrupts
>       4.50 ±104%   +8927.8%     406.25 ± 92%  interrupts.CPU47.TLB:TLB_shootdowns
>       1154 ±172%     -99.9%       1.50 ±137%  interrupts.CPU48.84:PCI-MSI.31981617-edge.i40e-eth0-TxRx-48
>     435.50 ± 70%     -98.5%       6.75 ± 92%  interrupts.CPU48.RES:Rescheduling_interrupts
>     504.50 ± 62%     -99.8%       1.25 ±131%  interrupts.CPU49.RES:Rescheduling_interrupts
>     180.25 ± 72%    -100.0%       0.00        interrupts.CPU49.TLB:TLB_shootdowns
>       1605 ± 69%     -79.6%     327.75 ± 70%  interrupts.CPU5.NMI:Non-maskable_interrupts
>       1605 ± 69%     -79.6%     327.75 ± 70%  interrupts.CPU5.PMI:Performance_monitoring_interrupts
>     527.00 ±  8%     -96.6%      17.75 ±173%  interrupts.CPU5.TLB:TLB_shootdowns
>       1248 ± 44%     -77.9%     276.25 ± 84%  interrupts.CPU53.NMI:Non-maskable_interrupts
>       1248 ± 44%     -77.9%     276.25 ± 84%  interrupts.CPU53.PMI:Performance_monitoring_interrupts
>     613.00 ±100%     -99.5%       3.25 ± 88%  interrupts.CPU54.RES:Rescheduling_interrupts
>     165.50 ± 98%     -96.5%       5.75 ±173%  interrupts.CPU54.TLB:TLB_shootdowns
>     577.75 ±111%     -99.7%       1.75 ±116%  interrupts.CPU55.RES:Rescheduling_interrupts
>     668.25 ± 76%     -46.1%     360.50 ±123%  interrupts.CPU56.NMI:Non-maskable_interrupts
>     668.25 ± 76%     -46.1%     360.50 ±123%  interrupts.CPU56.PMI:Performance_monitoring_interrupts
>     390.50 ±111%     -87.8%      47.75 ±168%  interrupts.CPU56.RES:Rescheduling_interrupts
>     134.50 ± 64%     -74.2%      34.75 ±173%  interrupts.CPU56.TLB:TLB_shootdowns
>     356.50 ± 96%     -94.5%      19.50 ± 73%  interrupts.CPU59.RES:Rescheduling_interrupts
>      77.25 ± 57%     -99.0%       0.75 ±173%  interrupts.CPU59.TLB:TLB_shootdowns
>       1004 ± 24%     -62.6%     376.25 ±106%  interrupts.CPU61.NMI:Non-maskable_interrupts
>       1004 ± 24%     -62.6%     376.25 ±106%  interrupts.CPU61.PMI:Performance_monitoring_interrupts
>       1181 ± 34%     -96.8%      38.25 ±143%  interrupts.CPU61.RES:Rescheduling_interrupts
>     379.00 ± 65%     -97.8%       8.50 ±173%  interrupts.CPU61.TLB:TLB_shootdowns
>       2301 ±140%     -99.7%       6.50 ± 86%  interrupts.CPU62.RES:Rescheduling_interrupts
>      95.75 ± 59%     -92.4%       7.25 ±173%  interrupts.CPU62.TLB:TLB_shootdowns
>     433.75 ± 83%     -98.4%       6.75 ±139%  interrupts.CPU63.RES:Rescheduling_interrupts
>     103.00 ± 56%     -92.5%       7.75 ±173%  interrupts.CPU63.TLB:TLB_shootdowns
>     994.50 ± 23%     -62.2%     376.25 ±113%  interrupts.CPU64.NMI:Non-maskable_interrupts
>     994.50 ± 23%     -62.2%     376.25 ±113%  interrupts.CPU64.PMI:Performance_monitoring_interrupts
>     512.75 ± 65%     -98.2%       9.25 ±137%  interrupts.CPU64.RES:Rescheduling_interrupts
>     151.00 ± 61%     -99.7%       0.50 ±173%  interrupts.CPU64.TLB:TLB_shootdowns
>       1008 ± 44%     -94.9%      51.50 ±100%  interrupts.CPU67.RES:Rescheduling_interrupts
>     242.25 ± 79%     -96.0%       9.75 ±155%  interrupts.CPU67.TLB:TLB_shootdowns
>     779.50 ± 75%     -98.8%       9.50 ±126%  interrupts.CPU68.RES:Rescheduling_interrupts
>       1059 ± 43%     -93.1%      73.25 ±109%  interrupts.CPU69.RES:Rescheduling_interrupts
>     246.00 ±104%     -93.1%      17.00 ±173%  interrupts.CPU69.TLB:TLB_shootdowns
>       1054 ±152%     -98.1%      20.00 ±161%  interrupts.CPU7.RES:Rescheduling_interrupts
>      66.00 ± 66%     -90.9%       6.00 ±173%  interrupts.CPU7.TLB:TLB_shootdowns
>       1056 ± 24%     -94.3%      59.75 ±125%  interrupts.CPU70.RES:Rescheduling_interrupts
>     224.75 ± 69%     -89.5%      23.50 ±163%  interrupts.CPU70.TLB:TLB_shootdowns
>       0.00       +1.1e+104%     107.00 ± 93%  interrupts.CPU72.TLB:TLB_shootdowns
>       9.00 ±137%   +3747.2%     346.25 ± 96%  interrupts.CPU73.RES:Rescheduling_interrupts
>       0.00       +2.3e+104%     230.00 ±133%  interrupts.CPU74.TLB:TLB_shootdowns
>       0.00       +1.6e+104%     165.00 ± 51%  interrupts.CPU75.TLB:TLB_shootdowns
>     537.75 ± 26%     -90.2%      52.50 ±113%  interrupts.CPU8.TLB:TLB_shootdowns
>       0.00       +9.8e+103%      97.50 ± 67%  interrupts.CPU83.TLB:TLB_shootdowns
>       0.00         +2e+104%     198.00 ± 49%  interrupts.CPU85.TLB:TLB_shootdowns
>      13.25 ± 59%    +809.4%     120.50 ± 75%  interrupts.CPU88.RES:Rescheduling_interrupts
>       5593 ± 51%     -93.2%     380.50 ±160%  interrupts.CPU9.RES:Rescheduling_interrupts
>       0.75 ± 57%  +31033.3%     233.50 ± 89%  interrupts.CPU93.RES:Rescheduling_interrupts
>       0.50 ±100%  +47100.0%     236.00 ±115%  interrupts.CPU94.RES:Rescheduling_interrupts
>      16.25 ±169%   +3118.5%     523.00 ± 78%  interrupts.CPU95.RES:Rescheduling_interrupts
>       2.00 ±173%   +8037.5%     162.75 ± 67%  interrupts.CPU95.TLB:TLB_shootdowns
>      49869 ±  7%     +94.1%      96790 ± 11%  softirqs.CPU0.RCU
>      51718 ±  6%     +67.2%      86459 ± 11%  softirqs.CPU1.RCU
>      51281 ±  5%     +92.7%      98810 ±  7%  softirqs.CPU10.RCU
>      49686 ± 12%     +95.1%      96913 ± 11%  softirqs.CPU11.RCU
>      51222 ±  6%     +95.4%     100111 ± 11%  softirqs.CPU12.RCU
>      50980 ±  6%     +99.1%     101505 ±  6%  softirqs.CPU13.RCU
>      51046 ±  7%     +98.3%     101207 ±  6%  softirqs.CPU14.RCU
>      45578 ± 21%    +116.5%      98669 ±  8%  softirqs.CPU15.RCU
>      44168 ±  7%     +99.2%      87963 ± 11%  softirqs.CPU16.RCU
>      41610 ±  9%    +119.4%      91301 ± 13%  softirqs.CPU17.RCU
>      43759 ±  8%    +110.7%      92217 ± 11%  softirqs.CPU18.RCU
>      42902 ±  8%    +109.1%      89698 ± 10%  softirqs.CPU19.RCU
>      52217 ±  5%     +94.3%     101454 ±  5%  softirqs.CPU2.RCU
>      42704 ±  5%    +114.8%      91744 ± 10%  softirqs.CPU20.RCU
>      44910 ±  7%    +102.1%      90783 ± 10%  softirqs.CPU22.RCU
>      44991 ±  7%    +104.7%      92096 ± 11%  softirqs.CPU23.RCU
>      47098 ± 12%     +92.3%      90567 ±  9%  softirqs.CPU24.RCU
>     106586 ±  6%     -11.9%      93907 ±  3%  softirqs.CPU24.SCHED
>      42456 ± 12%     +99.8%      84830 ±  8%  softirqs.CPU25.RCU
>      47709 ±  6%     +82.9%      87250 ± 10%  softirqs.CPU26.RCU
>      50970 ± 13%     +73.4%      88408 ± 12%  softirqs.CPU27.RCU
>      51241 ± 13%     +67.2%      85676 ± 15%  softirqs.CPU28.RCU
>      50293 ± 13%     +80.0%      90528 ± 10%  softirqs.CPU29.RCU
>      53868 ±  4%     +84.6%      99454 ± 10%  softirqs.CPU3.RCU
>      44947 ± 14%     +98.9%      89421 ±  8%  softirqs.CPU31.RCU
>      54210 ± 19%     +86.6%     101147 ±  5%  softirqs.CPU32.RCU
>      58728 ± 15%     +70.0%      99864 ±  9%  softirqs.CPU33.RCU
>      55716 ± 18%     +87.2%     104278 ±  5%  softirqs.CPU34.RCU
>      58583 ± 15%     +77.6%     104065 ±  5%  softirqs.CPU35.RCU
>      58080 ± 14%     +82.0%     105718 ±  5%  softirqs.CPU36.RCU
>      50595 ± 12%     +95.7%      99025 ±  4%  softirqs.CPU37.RCU
>      53939 ± 13%     +82.0%      98153 ±  4%  softirqs.CPU38.RCU
>      53424 ± 14%     +90.6%     101817 ±  6%  softirqs.CPU39.RCU
>      52327 ±  5%     +96.8%     102971 ±  4%  softirqs.CPU4.RCU
>      54501 ± 14%     +87.8%     102330 ±  7%  softirqs.CPU40.RCU
>      53036 ± 35%     +92.4%     102050 ±  4%  softirqs.CPU41.RCU
>      60492 ± 14%     +69.9%     102761 ±  2%  softirqs.CPU42.RCU
>      57222 ± 15%     +69.7%      97101 ±  5%  softirqs.CPU43.RCU
>      58535 ± 14%     +76.2%     103127 ±  5%  softirqs.CPU44.RCU
>      57025 ± 14%     +76.3%     100515 ±  6%  softirqs.CPU45.RCU
>      60244 ± 17%     +63.4%      98447 ±  7%  softirqs.CPU46.RCU
>      57985 ± 14%     +68.9%      97910 ± 12%  softirqs.CPU47.RCU
>      49462 ±  6%     +95.9%      96879 ± 11%  softirqs.CPU48.RCU
>      49436 ±  4%     +92.7%      95249 ± 12%  softirqs.CPU49.RCU
>      52413 ±  5%     +88.8%      98965 ± 11%  softirqs.CPU5.RCU
>      51032 ±  6%    +104.5%     104382 ±  7%  softirqs.CPU50.RCU
>      45684 ± 30%    +117.7%      99464 ± 15%  softirqs.CPU51.RCU
>      51919 ±  5%    +103.2%     105507 ±  6%  softirqs.CPU52.RCU
>      52420 ±  4%     +95.6%     102530 ± 12%  softirqs.CPU53.RCU
>      49538 ±  9%    +112.3%     105146 ±  7%  softirqs.CPU54.RCU
>      49883 ±  7%    +103.9%     101712 ±  9%  softirqs.CPU55.RCU
>      49604 ±  6%     +95.9%      97166 ±  9%  softirqs.CPU56.RCU
>      51798 ±  4%     +98.5%     102805 ±  6%  softirqs.CPU57.RCU
>      49777 ±  8%     +99.7%      99405 ±  7%  softirqs.CPU58.RCU
>      49338 ± 14%    +105.1%     101169 ± 11%  softirqs.CPU59.RCU
>      49653 ± 10%    +106.2%     102399 ±  6%  softirqs.CPU6.RCU
>      51134 ±  6%     +98.8%     101668 ± 11%  softirqs.CPU60.RCU
>      50488 ±  7%    +102.4%     102180 ±  7%  softirqs.CPU61.RCU
>      50078 ±  6%    +102.7%     101522 ±  6%  softirqs.CPU62.RCU
>      51026 ±  5%     +99.6%     101832 ±  8%  softirqs.CPU63.RCU
>      42895 ±  7%    +107.0%      88783 ± 13%  softirqs.CPU64.RCU
>      42387 ± 11%    +113.3%      90430 ± 11%  softirqs.CPU65.RCU
>      43746 ±  8%    +108.8%      91341 ± 12%  softirqs.CPU66.RCU
>      40324 ± 12%    +121.3%      89240 ± 11%  softirqs.CPU67.RCU
>      41900 ±  6%    +114.6%      89917 ± 11%  softirqs.CPU68.RCU
>      44157 ±  7%    +104.3%      90228 ± 11%  softirqs.CPU69.RCU
>      50551 ±  9%     +97.0%      99596 ±  8%  softirqs.CPU7.RCU
>      44094 ±  7%    +107.9%      91683 ± 13%  softirqs.CPU70.RCU
>      44086 ±  7%    +106.8%      91179 ± 11%  softirqs.CPU71.RCU
>      46488 ± 15%     +90.2%      88422 ±  8%  softirqs.CPU72.RCU
>      42653 ± 12%     +99.2%      84953 ±  9%  softirqs.CPU73.RCU
>      48559 ± 12%     +76.5%      85713 ± 12%  softirqs.CPU74.RCU
>      51770 ± 16%     +73.0%      89572 ±  9%  softirqs.CPU75.RCU
>      50182 ± 14%     +72.4%      86513 ± 14%  softirqs.CPU76.RCU
>      50446 ± 14%     +77.7%      89651 ± 10%  softirqs.CPU77.RCU
>      49119 ± 13%     +84.9%      90818 ±  8%  softirqs.CPU78.RCU
>      45781 ± 15%     +93.5%      88595 ±  8%  softirqs.CPU79.RCU
>      50229 ±  6%     +96.4%      98632 ±  7%  softirqs.CPU8.RCU
>      45155 ± 36%    +128.7%     103272 ±  5%  softirqs.CPU80.RCU
>      58487 ± 14%     +77.0%     103508 ±  6%  softirqs.CPU81.RCU
>      55666 ± 18%     +87.0%     104094 ±  5%  softirqs.CPU82.RCU
>      58506 ± 14%     +77.7%     103963 ±  5%  softirqs.CPU83.RCU
>      58416 ± 14%     +80.3%     105297 ±  6%  softirqs.CPU84.RCU
>      51291 ± 12%     +95.8%     100447 ±  4%  softirqs.CPU85.RCU
>      54557 ± 13%     +82.1%      99356 ±  4%  softirqs.CPU86.RCU
>      54147 ± 13%     +89.8%     102754 ±  6%  softirqs.CPU87.RCU
>      55853 ± 14%     +82.1%     101688 ±  6%  softirqs.CPU88.RCU
>      59176 ± 16%     +70.8%     101077 ±  5%  softirqs.CPU89.RCU
>      52291 ±  6%     +93.8%     101344 ±  5%  softirqs.CPU9.RCU
>      60854 ± 15%     +69.0%     102864 ±  2%  softirqs.CPU90.RCU
>      57902 ± 12%     +70.1%      98501 ±  4%  softirqs.CPU91.RCU
>      59683 ± 15%     +73.3%     103419 ±  5%  softirqs.CPU92.RCU
>      57459 ± 14%     +78.5%     102584 ±  6%  softirqs.CPU93.RCU
>      59398 ± 13%     +63.2%      96927 ±  8%  softirqs.CPU94.RCU
>      56576 ± 15%     +76.2%      99664 ± 11%  softirqs.CPU95.RCU
>       5123 ± 16%    +168.1%      13739 ±  7%  softirqs.NET_RX
>    4882876 ± 10%     +90.1%    9282598 ±  6%  softirqs.RCU
>
>
>
>                         phoronix-test-suite.glmark2.0.score
>
>   300 +-+-------------------------------------------------------------------+
>       |  :     +.    +..+    +.       +..+.             +.      +           |
>   250 +-+:                                                                  |
>       O :   O  O  O  O    O  O  O  O  O  O     O     O     O                O
>       | :O              O                   O     O     O    O  O  O  O  O  |
>   200 +-+                                                                   |
>       | :                                                                   |
>   150 +-+                                                                   |
>       |:                                                                    |
>   100 +-+                                                                   |
>       |:                                                                    |
>       |:                                                                    |
>    50 +-+                                                                   |
>       |                                                                     |
>     0 +-+-------------------------------------------------------------------+
>
>
>                         phoronix-test-suite.glmark2.1.score
>
>   250 +-+-------------------------------------------------------------------+
>       |                                                                     |
>       |  +..+..+..+..+..+.+..+..+..  .+..+..+..+..+..+..+..+.+..+           |
>   200 +-+:                         +.                                       |
>       |  :                               O                 O                O
>       O :O  O  O  O  O  O O  O  O  O  O     O  O  O  O  O    O  O  O  O  O  |
>   150 +-+                                                                   |
>       | :                                                                   |
>   100 +-+                                                                   |
>       |:                                                                    |
>       |:                                                                    |
>    50 +-+                                                                   |
>       |:                                                                    |
>       |                                                                     |
>     0 +-+-------------------------------------------------------------------+
>
>
> [*] bisect-good sample
> [O] bisect-bad  sample
>
> ***************************************************************************************************
> lkp-csl-2sp8: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 192G memory
> =========================================================================================
> compiler/cpufreq_governor/kconfig/need_x/rootfs/tbox_group/test/testcase/ucode:
>   gcc-7/performance/x86_64-rhel-7.6/true/debian-x86_64-phoronix/lkp-csl-2sp8/gtkperf-1.2.1/phoronix-test-suite/0x500002b
>
> commit:
>   9f4813b531 (" Linux 5.4-rc8")
>   8d04a5f97a ("x86/mm/pat: Convert the PAT tree to a generic interval tree")
>
> 9f4813b531a0b8cc 8d04a5f97a5fa9d7afdf46eda3a
> ---------------- ---------------------------
>          %stddev     %change         %stddev
>              \          |                \
>      10.34           +26.8%      13.11 ±  5%  phoronix-test-suite.gtkperf.0.seconds
>      62.29           +13.3%      70.56 ±  2%  phoronix-test-suite.time.elapsed_time
>      62.29           +13.3%      70.56 ±  2%  phoronix-test-suite.time.elapsed_time.max
>      46875           -12.5%      41002 ±  3%  meminfo.max_used_kB
>      29.85           +39.6%      41.68 ± 22%  boot-time.boot
>       2433           +48.7%       3617 ± 25%  boot-time.idle
>       1628           -11.4%       1442 ±  4%  vmstat.io.bi
>       9046           -11.6%       7993 ±  2%  vmstat.system.cs
>  1.954e+09 ± 69%    +226.7%  6.384e+09 ±  6%  cpuidle.C1E.time
>    6278082 ± 33%     +94.4%   12207082 ± 16%  cpuidle.C1E.usage
>  3.958e+09 ± 32%     -93.8%  2.444e+08 ± 10%  cpuidle.C6.time
>    5684464           -95.4%     262692 ± 13%  cpuidle.C6.usage
>       8.34 ±100%      -8.3        0.00        perf-profile.calltrace.cycles-pp.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
>       8.34 ±100%      -8.3        0.00        perf-profile.calltrace.cycles-pp.do_filp_open.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
>       8.34 ±100%      -8.3        0.00        perf-profile.calltrace.cycles-pp.path_openat.do_filp_open.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
>       5.28 ±100%      -3.2        2.08 ±173%  perf-profile.calltrace.cycles-pp.mmput.do_exit.do_group_exit.get_signal.do_signal
>       5.28 ±100%      -3.2        2.08 ±173%  perf-profile.calltrace.cycles-pp.exit_mmap.mmput.do_exit.do_group_exit.get_signal
>       8.34 ±100%      -8.3        0.00        perf-profile.children.cycles-pp.do_sys_open
>       8.34 ±100%      -8.3        0.00        perf-profile.children.cycles-pp.do_filp_open
>       8.34 ±100%      -8.3        0.00        perf-profile.children.cycles-pp.path_openat
>       5.28 ±100%      -3.2        2.08 ±173%  perf-profile.children.cycles-pp.mmput
>       5.28 ±100%      -3.2        2.08 ±173%  perf-profile.children.cycles-pp.exit_mmap
>      17.00 ± 13%    +230.9%      56.25 ± 29%  sched_debug.cfs_rq:/.runnable_load_avg.max
>       3.87 ± 17%     +73.5%       6.71 ± 22%  sched_debug.cfs_rq:/.runnable_load_avg.stddev
>      25.25 ± 12%     -22.4%      19.58 ± 20%  sched_debug.cfs_rq:/.util_est_enqueued.avg
>      34397           +44.4%      49658 ± 29%  sched_debug.cpu.clock.avg
>      34402           +44.4%      49663 ± 29%  sched_debug.cpu.clock.max
>      34391           +44.4%      49654 ± 29%  sched_debug.cpu.clock.min
>      34397           +44.4%      49658 ± 29%  sched_debug.cpu.clock_task.avg
>      34402           +44.4%      49663 ± 29%  sched_debug.cpu.clock_task.max
>      34391           +44.4%      49654 ± 29%  sched_debug.cpu.clock_task.min
>       9.74 ±  5%     -11.5%       8.62 ±  4%  sched_debug.cpu.nr_uninterruptible.stddev
>      34393           +44.4%      49654 ± 29%  sched_debug.cpu_clk
>      31537           +48.4%      46795 ± 31%  sched_debug.ktime
>      34725           +44.0%      49988 ± 29%  sched_debug.sched_clk
>      96262            +2.1%      98322        proc-vmstat.nr_active_anon
>      29673            +1.4%      30100        proc-vmstat.nr_active_file
>      94407            +2.3%      96570        proc-vmstat.nr_anon_pages
>       1082            +1.5%       1099        proc-vmstat.nr_page_table_pages
>      96262            +2.1%      98322        proc-vmstat.nr_zone_active_anon
>      29673            +1.4%      30100        proc-vmstat.nr_zone_active_file
>       7664 ± 27%    +172.9%      20912 ± 38%  proc-vmstat.numa_hint_faults
>       7507 ± 27%    +124.1%      16824 ± 38%  proc-vmstat.numa_hint_faults_local
>     328182            +5.9%     347387 ±  4%  proc-vmstat.numa_hit
>     309363            +6.2%     328630 ±  4%  proc-vmstat.numa_local
>      39400 ± 35%     +90.3%      74971 ± 42%  proc-vmstat.numa_pte_updates
>     370297            +7.5%     398082 ±  4%  proc-vmstat.pgalloc_normal
>     278665           +12.0%     312075 ±  4%  proc-vmstat.pgfault
>     269968 ±  2%     +10.9%     299343 ±  5%  proc-vmstat.pgfree
>      16.26 ± 53%     -51.0%       7.96 ±  2%  perf-stat.i.MPKI
>  8.359e+08 ±  2%     -12.0%  7.355e+08 ±  5%  perf-stat.i.branch-instructions
>       3.78 ± 21%      -1.0        2.76 ±  9%  perf-stat.i.branch-miss-rate%
>   30272546 ± 11%     -25.3%   22600376 ± 15%  perf-stat.i.branch-misses
>    1870084 ±  6%     -59.7%     752958 ± 15%  perf-stat.i.cache-misses
>   47650854 ± 37%     -46.3%   25593693 ±  2%  perf-stat.i.cache-references
>       9360           -12.3%       8208 ±  2%  perf-stat.i.context-switches
>  6.312e+09 ±  6%     -15.0%  5.364e+09 ±  2%  perf-stat.i.cpu-cycles
>       3783 ±  9%    +176.2%      10448 ± 18%  perf-stat.i.cycles-between-cache-misses
>       0.16 ± 40%      -0.1        0.01 ± 16%  perf-stat.i.dTLB-load-miss-rate%
>     921731 ± 27%     -87.9%     111653 ± 10%  perf-stat.i.dTLB-load-misses
>       0.05 ± 32%      -0.0        0.01 ± 21%  perf-stat.i.dTLB-store-miss-rate%
>     157895 ± 26%     -82.3%      28003 ± 22%  perf-stat.i.dTLB-store-misses
>      41.55 ±  4%      -3.5       38.06        perf-stat.i.iTLB-load-miss-rate%
>  3.826e+09 ±  2%     -11.9%  3.372e+09 ±  6%  perf-stat.i.instructions
>       9.33           -13.3%       8.09 ±  3%  perf-stat.i.major-faults
>      12.54 ± 38%     -39.3%       7.61 ±  5%  perf-stat.overall.MPKI
>       3.63 ± 13%      -0.6        3.06 ± 10%  perf-stat.overall.branch-miss-rate%
>       4.51 ± 35%      -1.6        2.94 ± 14%  perf-stat.overall.cache-miss-rate%
>       3391 ± 10%    +113.9%       7255 ± 12%  perf-stat.overall.cycles-between-cache-misses
>       0.10 ± 33%      -0.1        0.01 ± 14%  perf-stat.overall.dTLB-load-miss-rate%
>       0.04 ± 26%      -0.0        0.01 ± 23%  perf-stat.overall.dTLB-store-miss-rate%
>   8.23e+08 ±  2%     -11.9%  7.253e+08 ±  5%  perf-stat.ps.branch-instructions
>   29792047 ± 11%     -25.2%   22290398 ± 15%  perf-stat.ps.branch-misses
>    1841339 ±  6%     -59.6%     743171 ± 15%  perf-stat.ps.cache-misses
>   46876977 ± 37%     -46.2%   25232671 ±  2%  perf-stat.ps.cache-references
>       9205           -12.1%       8091 ±  2%  perf-stat.ps.context-switches
>  6.211e+09 ±  6%     -14.8%  5.289e+09 ±  2%  perf-stat.ps.cpu-cycles
>     907015 ± 27%     -87.9%     110123 ± 10%  perf-stat.ps.dTLB-load-misses
>     155379 ± 26%     -82.2%      27623 ± 22%  perf-stat.ps.dTLB-store-misses
>  3.766e+09 ±  2%     -11.7%  3.325e+09 ±  6%  perf-stat.ps.instructions
>       9.21           -13.4%       7.98 ±  3%  perf-stat.ps.major-faults
>      16348 ±  5%     +33.4%      21802 ±  5%  softirqs.CPU0.SCHED
>      11186 ± 15%     +37.7%      15406 ± 16%  softirqs.CPU1.SCHED
>       9914 ±  8%     +48.9%      14762 ± 30%  softirqs.CPU10.SCHED
>       9645 ±  2%     +21.1%      11681 ±  4%  softirqs.CPU11.SCHED
>       9779 ±  3%     +17.1%      11450 ±  7%  softirqs.CPU13.SCHED
>       9805 ±  3%     +12.6%      11040 ±  6%  softirqs.CPU14.SCHED
>       9658 ±  2%     +25.3%      12104 ± 10%  softirqs.CPU15.SCHED
>      31797 ±  7%     +23.0%      39108 ± 14%  softirqs.CPU16.TIMER
>       9330 ±  6%     +29.9%      12116 ±  4%  softirqs.CPU17.SCHED
>      31554 ± 10%     +16.1%      36648 ±  6%  softirqs.CPU17.TIMER
>       9493 ±  4%     +21.9%      11574 ±  6%  softirqs.CPU18.SCHED
>       9700 ±  2%     +61.1%      15631 ± 47%  softirqs.CPU19.SCHED
>      10097 ±  4%     +20.7%      12186        softirqs.CPU2.SCHED
>       9731 ±  2%     +18.3%      11516 ±  3%  softirqs.CPU20.SCHED
>       9664 ±  3%     +18.1%      11413 ±  4%  softirqs.CPU21.SCHED
>       9810 ±  3%     +12.3%      11013 ±  6%  softirqs.CPU22.SCHED
>       9509 ±  3%     +20.0%      11407 ±  4%  softirqs.CPU23.SCHED
>       9355 ±  9%     +23.8%      11581 ±  8%  softirqs.CPU26.SCHED
>      34990           +24.1%      43413 ±  6%  softirqs.CPU26.TIMER
>       9577 ±  8%     +20.8%      11569 ±  4%  softirqs.CPU27.SCHED
>       9920 ±  2%     +15.5%      11462 ±  6%  softirqs.CPU28.SCHED
>      10084           +13.1%      11409 ±  4%  softirqs.CPU29.SCHED
>      30741 ± 10%     +27.7%      39262 ±  8%  softirqs.CPU3.TIMER
>       9843           +18.7%      11688 ±  6%  softirqs.CPU30.SCHED
>       8609 ± 21%     +33.6%      11503 ±  4%  softirqs.CPU32.SCHED
>       9714 ±  2%     +22.8%      11927 ±  7%  softirqs.CPU33.SCHED
>       9695           +18.1%      11445 ±  5%  softirqs.CPU34.SCHED
>       8242 ± 30%     +39.4%      11488 ±  6%  softirqs.CPU35.SCHED
>       8035 ± 30%     +40.4%      11280 ±  5%  softirqs.CPU37.SCHED
>       9880           +17.7%      11627 ±  6%  softirqs.CPU46.SCHED
>       8892 ±  9%     +26.4%      11243 ±  8%  softirqs.CPU47.SCHED
>       9244 ±  7%     +16.8%      10799 ±  3%  softirqs.CPU49.SCHED
>       9617           +18.4%      11389 ±  3%  softirqs.CPU50.SCHED
>      31719 ±  7%     +12.5%      35691 ±  5%  softirqs.CPU50.TIMER
>       9366 ±  5%     +16.5%      10911 ±  3%  softirqs.CPU51.SCHED
>       9929 ±  3%     +22.9%      12207 ±  7%  softirqs.CPU53.SCHED
>       8616 ± 13%     +41.5%      12194 ± 10%  softirqs.CPU54.SCHED
>       9500 ±  3%     +15.4%      10962        softirqs.CPU55.SCHED
>      32014 ±  9%     +20.3%      38529 ± 15%  softirqs.CPU56.TIMER
>       9732 ±  6%     +16.2%      11310 ±  4%  softirqs.CPU59.SCHED
>       9628 ±  2%     +21.5%      11699 ± 10%  softirqs.CPU60.SCHED
>      10026 ±  5%     +10.9%      11124 ±  6%  softirqs.CPU62.SCHED
>       9585           +14.5%      10972 ±  8%  softirqs.CPU63.SCHED
>       9541 ±  3%     +21.4%      11586 ± 16%  softirqs.CPU64.SCHED
>       9035 ± 10%     +17.1%      10577 ±  7%  softirqs.CPU65.SCHED
>       9483 ±  2%     +20.2%      11401 ±  5%  softirqs.CPU66.SCHED
>       9026 ± 12%     +23.4%      11134 ±  4%  softirqs.CPU68.SCHED
>       9482 ±  2%     +13.8%      10787 ±  5%  softirqs.CPU69.SCHED
>       9658 ±  2%     +20.4%      11624 ±  4%  softirqs.CPU7.SCHED
>       9764 ±  3%     +11.0%      10836 ±  5%  softirqs.CPU70.SCHED
>       9316 ± 11%     +18.8%      11072 ± 12%  softirqs.CPU74.SCHED
>       9963           +12.3%      11189 ±  6%  softirqs.CPU75.SCHED
>      10017           +14.1%      11425 ±  7%  softirqs.CPU76.SCHED
>       9788           +18.4%      11588 ±  5%  softirqs.CPU77.SCHED
>       9968           +16.3%      11591 ±  7%  softirqs.CPU78.SCHED
>       9911 ±  2%     +16.6%      11555 ±  5%  softirqs.CPU79.SCHED
>       9408 ±  2%     +19.7%      11258 ±  7%  softirqs.CPU80.SCHED
>      10027 ±  4%     +11.3%      11164 ±  5%  softirqs.CPU81.SCHED
>       9638 ±  3%     +14.3%      11019 ±  7%  softirqs.CPU82.SCHED
>       9551 ±  2%     +15.0%      10981 ±  8%  softirqs.CPU83.SCHED
>       9583 ±  4%     +18.8%      11389 ±  6%  softirqs.CPU84.SCHED
>       8173 ± 29%     +34.4%      10989 ±  7%  softirqs.CPU86.SCHED
>       9936 ±  4%     +14.8%      11410 ±  6%  softirqs.CPU9.SCHED
>       8179 ± 30%     +37.6%      11257 ± 10%  softirqs.CPU91.SCHED
>       8077 ± 30%     +39.1%      11233 ±  8%  softirqs.CPU95.SCHED
>     919351 ±  7%     +19.9%    1102697 ±  4%  softirqs.SCHED
>    3193255 ±  2%     +11.6%    3563915 ±  6%  softirqs.TIMER
>      33.00 ±147%    -100.0%       0.00        interrupts.52:PCI-MSI.31981585-edge.i40e-eth0-TxRx-16
>     127.00           +15.4%     146.50 ±  3%  interrupts.9:IO-APIC.9-fasteoi.acpi
>     124159 ±  3%     +16.5%     144587 ±  3%  interrupts.CPU0.LOC:Local_timer_interrupts
>     127.00           +15.4%     146.50 ±  3%  interrupts.CPU1.9:IO-APIC.9-fasteoi.acpi
>     123924 ±  3%     +16.7%     144656 ±  2%  interrupts.CPU1.LOC:Local_timer_interrupts
>     123637 ±  3%     +16.2%     143635 ±  4%  interrupts.CPU10.LOC:Local_timer_interrupts
>     123785 ±  3%     +15.3%     142755 ±  3%  interrupts.CPU11.LOC:Local_timer_interrupts
>     123691 ±  3%     +16.3%     143829 ±  3%  interrupts.CPU12.LOC:Local_timer_interrupts
>     123545 ±  3%     +15.7%     142970 ±  3%  interrupts.CPU13.LOC:Local_timer_interrupts
>     123726 ±  3%     +16.2%     143718 ±  3%  interrupts.CPU14.LOC:Local_timer_interrupts
>     123727 ±  3%     +15.2%     142493 ±  2%  interrupts.CPU15.LOC:Local_timer_interrupts
>     123761 ±  3%     +16.7%     144390 ±  3%  interrupts.CPU16.LOC:Local_timer_interrupts
>     123749 ±  3%     +16.9%     144632 ±  3%  interrupts.CPU17.LOC:Local_timer_interrupts
>     123660 ±  3%     +16.9%     144533 ±  3%  interrupts.CPU18.LOC:Local_timer_interrupts
>     123976 ±  3%     +17.0%     145106 ±  3%  interrupts.CPU19.LOC:Local_timer_interrupts
>     123983 ±  3%     +16.8%     144764 ±  3%  interrupts.CPU2.LOC:Local_timer_interrupts
>     123783 ±  3%     +16.7%     144400 ±  3%  interrupts.CPU20.LOC:Local_timer_interrupts
>     123924 ±  3%     +16.5%     144405 ±  3%  interrupts.CPU21.LOC:Local_timer_interrupts
>     124054 ±  3%     +15.9%     143768 ±  3%  interrupts.CPU22.LOC:Local_timer_interrupts
>     124085 ±  3%     +15.7%     143543 ±  3%  interrupts.CPU23.LOC:Local_timer_interrupts
>     110004 ± 22%     +30.5%     143553 ±  4%  interrupts.CPU24.LOC:Local_timer_interrupts
>     110068 ± 22%     +29.7%     142727 ±  3%  interrupts.CPU25.LOC:Local_timer_interrupts
>     110942 ± 22%     +29.6%     143774 ±  2%  interrupts.CPU26.LOC:Local_timer_interrupts
>     111894 ± 22%     +28.3%     143555 ±  4%  interrupts.CPU27.LOC:Local_timer_interrupts
>     111050 ± 22%     +29.3%     143625 ±  4%  interrupts.CPU28.LOC:Local_timer_interrupts
>     111805 ± 22%     +27.5%     142570 ±  3%  interrupts.CPU29.LOC:Local_timer_interrupts
>     122901 ±  3%     +17.6%     144537 ±  3%  interrupts.CPU3.LOC:Local_timer_interrupts
>     111025 ± 22%     +29.5%     143752 ±  4%  interrupts.CPU30.LOC:Local_timer_interrupts
>     110372 ± 21%     +29.3%     142764 ±  3%  interrupts.CPU31.LOC:Local_timer_interrupts
>     110041 ± 22%     +30.2%     143294 ±  3%  interrupts.CPU32.LOC:Local_timer_interrupts
>     110289 ± 21%     +29.4%     142737 ±  3%  interrupts.CPU33.LOC:Local_timer_interrupts
>       0.75 ±110%   +4200.0%      32.25 ±143%  interrupts.CPU33.RES:Rescheduling_interrupts
>     111042 ± 22%     +28.7%     142872 ±  4%  interrupts.CPU34.LOC:Local_timer_interrupts
>     110820 ± 22%     +29.6%     143676 ±  4%  interrupts.CPU35.LOC:Local_timer_interrupts
>     110849 ± 22%     +29.9%     143999 ±  4%  interrupts.CPU36.LOC:Local_timer_interrupts
>     110376 ± 22%     +30.4%     143915 ±  3%  interrupts.CPU37.LOC:Local_timer_interrupts
>     111593 ± 22%     +29.0%     143915 ±  4%  interrupts.CPU38.LOC:Local_timer_interrupts
>     111949 ± 22%     +27.1%     142241 ±  4%  interrupts.CPU39.LOC:Local_timer_interrupts
>     124261 ±  3%     +15.7%     143828 ±  3%  interrupts.CPU4.LOC:Local_timer_interrupts
>     111554 ± 22%     +27.5%     142286 ±  4%  interrupts.CPU40.LOC:Local_timer_interrupts
>     110802 ± 22%     +29.1%     143047 ±  4%  interrupts.CPU41.LOC:Local_timer_interrupts
>     111642 ± 22%     +29.4%     144476 ±  4%  interrupts.CPU42.LOC:Local_timer_interrupts
>     110980 ± 22%     +29.3%     143450 ±  4%  interrupts.CPU43.LOC:Local_timer_interrupts
>     111595 ± 22%     +27.4%     142158 ±  4%  interrupts.CPU44.LOC:Local_timer_interrupts
>     110926 ± 22%     +28.8%     142914 ±  4%  interrupts.CPU45.LOC:Local_timer_interrupts
>     111788 ± 22%     +28.5%     143694 ±  4%  interrupts.CPU46.LOC:Local_timer_interrupts
>     110789 ± 22%     +29.2%     143182 ±  4%  interrupts.CPU47.LOC:Local_timer_interrupts
>     123853 ±  3%     +16.0%     143633 ±  3%  interrupts.CPU48.LOC:Local_timer_interrupts
>     123954 ±  3%     +16.1%     143915 ±  3%  interrupts.CPU49.LOC:Local_timer_interrupts
>     125940 ±  5%     +14.2%     143844 ±  2%  interrupts.CPU5.LOC:Local_timer_interrupts
>     122996 ±  3%     +16.9%     143771 ±  3%  interrupts.CPU50.LOC:Local_timer_interrupts
>     123469 ±  3%     +16.3%     143600 ±  3%  interrupts.CPU51.LOC:Local_timer_interrupts
>     122963 ±  2%     +16.8%     143584 ±  3%  interrupts.CPU52.LOC:Local_timer_interrupts
>     123920 ±  3%     +16.7%     144558 ±  3%  interrupts.CPU53.LOC:Local_timer_interrupts
>     123791 ±  3%     +16.7%     144460 ±  3%  interrupts.CPU54.LOC:Local_timer_interrupts
>     123877 ±  3%     +16.6%     144392 ±  3%  interrupts.CPU55.LOC:Local_timer_interrupts
>     123809 ±  3%     +16.4%     144159 ±  3%  interrupts.CPU56.LOC:Local_timer_interrupts
>     123506 ±  3%     +16.6%     144033 ±  3%  interrupts.CPU57.LOC:Local_timer_interrupts
>     123672 ±  3%     +17.7%     145596 ±  4%  interrupts.CPU58.LOC:Local_timer_interrupts
>     123454 ±  3%     +16.8%     144178 ±  3%  interrupts.CPU59.LOC:Local_timer_interrupts
>     123108 ±  4%     +17.5%     144649 ±  3%  interrupts.CPU6.LOC:Local_timer_interrupts
>     123692 ±  3%     +16.1%     143634 ±  3%  interrupts.CPU60.LOC:Local_timer_interrupts
>     123855 ±  3%     +15.5%     143073 ±  3%  interrupts.CPU61.LOC:Local_timer_interrupts
>     123714 ±  3%     +16.2%     143797 ±  3%  interrupts.CPU62.LOC:Local_timer_interrupts
>     123863 ±  3%     +16.0%     143707 ±  3%  interrupts.CPU63.LOC:Local_timer_interrupts
>     123875 ±  3%     +15.9%     143543 ±  3%  interrupts.CPU64.LOC:Local_timer_interrupts
>     124440 ±  2%     +16.8%     145383 ±  2%  interrupts.CPU65.LOC:Local_timer_interrupts
>     124284 ±  2%     +15.9%     144070 ±  3%  interrupts.CPU66.LOC:Local_timer_interrupts
>     123799 ±  3%     +15.8%     143331 ±  3%  interrupts.CPU67.LOC:Local_timer_interrupts
>     123400 ±  3%     +17.0%     144332 ±  3%  interrupts.CPU68.LOC:Local_timer_interrupts
>     125156 ±  4%     +14.5%     143289 ±  3%  interrupts.CPU69.LOC:Local_timer_interrupts
>     124195 ±  3%     +16.3%     144402 ±  3%  interrupts.CPU7.LOC:Local_timer_interrupts
>     123791 ±  3%     +16.0%     143551 ±  3%  interrupts.CPU70.LOC:Local_timer_interrupts
>     123826 ±  3%     +15.7%     143277 ±  3%  interrupts.CPU71.LOC:Local_timer_interrupts
>     110101 ± 22%     +29.0%     141983 ±  4%  interrupts.CPU72.LOC:Local_timer_interrupts
>     110774 ± 22%     +28.3%     142100 ±  4%  interrupts.CPU73.LOC:Local_timer_interrupts
>     111785 ± 22%     +27.6%     142623 ±  3%  interrupts.CPU74.LOC:Local_timer_interrupts
>     111709 ± 22%     +29.2%     144354 ±  3%  interrupts.CPU75.LOC:Local_timer_interrupts
>     111739 ± 22%     +28.9%     143992 ±  3%  interrupts.CPU76.LOC:Local_timer_interrupts
>     111017 ± 22%     +29.7%     143954 ±  3%  interrupts.CPU77.LOC:Local_timer_interrupts
>     111625 ± 22%     +28.7%     143651 ±  3%  interrupts.CPU78.LOC:Local_timer_interrupts
>     110970 ± 22%     +29.5%     143695 ±  4%  interrupts.CPU79.LOC:Local_timer_interrupts
>     122666 ±  4%     +18.0%     144790 ±  3%  interrupts.CPU8.LOC:Local_timer_interrupts
>     109397 ± 21%     +32.0%     144381 ±  3%  interrupts.CPU80.LOC:Local_timer_interrupts
>     111101 ± 22%     +29.2%     143593 ±  3%  interrupts.CPU81.LOC:Local_timer_interrupts
>     110996 ± 22%     +28.0%     142033 ±  4%  interrupts.CPU82.LOC:Local_timer_interrupts
>     110733 ± 22%     +28.4%     142147 ±  4%  interrupts.CPU83.LOC:Local_timer_interrupts
>     110217 ± 22%     +30.5%     143787 ±  4%  interrupts.CPU84.LOC:Local_timer_interrupts
>     109986 ± 22%     +31.0%     144082 ±  3%  interrupts.CPU85.LOC:Local_timer_interrupts
>     110129 ± 22%     +30.2%     143365 ±  3%  interrupts.CPU86.LOC:Local_timer_interrupts
>     110749 ± 22%     +28.8%     142696 ±  4%  interrupts.CPU87.LOC:Local_timer_interrupts
>     110778 ± 22%     +29.8%     143804 ±  4%  interrupts.CPU88.LOC:Local_timer_interrupts
>     111348 ± 22%     +28.5%     143096 ±  4%  interrupts.CPU89.LOC:Local_timer_interrupts
>     123749 ±  3%     +16.1%     143677 ±  3%  interrupts.CPU9.LOC:Local_timer_interrupts
>     110728 ± 22%     +29.0%     142841 ±  4%  interrupts.CPU90.LOC:Local_timer_interrupts
>     110097 ± 22%     +30.1%     143207 ±  4%  interrupts.CPU91.LOC:Local_timer_interrupts
>     111533 ± 22%     +28.3%     143083 ±  4%  interrupts.CPU92.LOC:Local_timer_interrupts
>     110777 ± 22%     +29.0%     142893 ±  4%  interrupts.CPU93.LOC:Local_timer_interrupts
>     109394 ± 21%     +31.1%     143427 ±  4%  interrupts.CPU94.LOC:Local_timer_interrupts
>     110000 ± 22%     +30.0%     143027 ±  4%  interrupts.CPU95.LOC:Local_timer_interrupts
>   11264859 ± 12%     +22.4%   13788804 ±  3%  interrupts.LOC:Local_timer_interrupts
>
>
>
>
>
> Disclaimer:
> Results have been estimated based on internal Intel analysis and are provided
> for informational purposes only. Any difference in system hardware or software
> design or configuration may affect actual performance.
>
>
> Thanks,
> Rong Chen
>

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ