lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Date:   Fri, 28 Jun 2019 08:50:32 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     Waiman Long <longman@...hat.com>
Cc:     Ingo Molnar <mingo@...nel.org>,
        Peter Zijlstra <peterz@...radead.org>,
        Borislav Petkov <bp@...en8.de>,
        Davidlohr Bueso <dave@...olabs.net>,
        "H. Peter Anvin" <hpa@...or.com>,
        Linus Torvalds <torvalds@...ux-foundation.org>,
        Thomas Gleixner <tglx@...utronix.de>,
        Tim Chen <tim.c.chen@...ux.intel.com>,
        Will Deacon <will.deacon@....com>,
        huang ying <huang.ying.caritas@...il.com>,
        LKML <linux-kernel@...r.kernel.org>,
        Stephen Rothwell <sfr@...b.auug.org.au>, lkp@...org
Subject: [locking/rwsem] 7d43f1ce9d: vm-scalability.median -16.5% regression

Greeting,

FYI, we noticed a -16.5% regression of vm-scalability.median due to commit:


commit: 7d43f1ce9dd075d8b2aa3ad1f3970ef386a5c358 ("locking/rwsem: Enable time-based spinning on reader-owned rwsem")
https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git master

in testcase: vm-scalability
on test machine: 192 threads Intel(R) Xeon(R) CPU @ 2.20GHz with 192G memory
with following parameters:

	runtime: 300s
	size: 8T
	test: anon-w-seq-mt
	cpufreq_governor: performance

test-description: The motivation behind this suite is to exercise functions and regions of the mm/ of the Linux kernel which are of interest to us.
test-url: https://git.kernel.org/cgit/linux/kernel/git/wfg/vm-scalability.git/

In addition to that, the commit also has significant impact on the following tests:

+------------------+-----------------------------------------------------------------------+
| testcase: change | vm-scalability: vm-scalability.median 14.0% improvement               |
| test machine     | 4 threads Intel(R) Core(TM) i3-3220 CPU @ 3.30GHz with 8G memory      |
| test parameters  | cpufreq_governor=performance                                          |
|                  | runtime=300s                                                          |
|                  | test=small-allocs-mt                                                  |
|                  | ucode=0x21                                                            |
+------------------+-----------------------------------------------------------------------+


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
compiler/cpufreq_governor/kconfig/rootfs/runtime/size/tbox_group/test/testcase:
  gcc-7/performance/x86_64-rhel-7.6/debian-x86_64-2019-05-14.cgz/300s/8T/lkp-csl-2ap1/anon-w-seq-mt/vm-scalability

commit: 
  94a9717b3c ("locking/rwsem: Make rwsem->owner an atomic_long_t")
  7d43f1ce9d ("locking/rwsem: Enable time-based spinning on reader-owned rwsem")

94a9717b3c40e77a 7d43f1ce9dd075d8b2aa3ad1f39 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :4           25%           1:4     dmesg.WARNING:at#for_ip_interrupt_entry/0x
         %stddev     %change         %stddev
             \          |                \  
      0.01 ±  2%    +166.9%       0.02 ±  6%  vm-scalability.free_time
    191759           -16.5%     160104        vm-scalability.median
      0.04 ±  4%    +163.7%       0.11 ±  8%  vm-scalability.median_stddev
  39109084            -7.1%   36338578        vm-scalability.throughput
    306656 ±  2%     -64.3%     109562 ±  5%  vm-scalability.time.involuntary_context_switches
  89394993            -8.2%   82031710 ±  4%  vm-scalability.time.maximum_resident_set_size
  13786907           -23.1%   10603356 ±  2%  vm-scalability.time.minor_page_faults
     15225           -29.2%      10777 ±  2%  vm-scalability.time.percent_of_cpu_this_job_got
     22047           -23.6%      16835        vm-scalability.time.system_time
     23929           -34.3%      15715 ±  2%  vm-scalability.time.user_time
   1555880           -47.0%     825313 ±  2%  vm-scalability.time.voluntary_context_switches
 1.037e+10           -23.1%   7.98e+09 ±  2%  vm-scalability.workload
     37.83            +1.0%      38.20        boot-time.boot
  18197810 ±138%     -83.6%    2988363 ±  7%  cpuidle.C1.time
  21000317 ± 14%    +109.1%   43903618 ± 21%  cpuidle.C1E.usage
    119868 ± 42%     +54.1%     184715 ± 19%  cpuidle.POLL.usage
     20.00 ±  3%    +113.8%      42.75 ±  4%  vmstat.cpu.id
     40.75           -34.4%      26.75 ±  3%  vmstat.cpu.us
    158.50           -31.1%     109.25        vmstat.procs.r
     15578           -34.5%      10207        vmstat.system.cs
     19.72 ±  2%     +23.2       42.90 ±  3%  mpstat.cpu.all.idle%
      0.07 ±  5%      -0.1        0.01 ± 36%  mpstat.cpu.all.iowait%
      0.00 ±  9%      -0.0        0.00 ± 12%  mpstat.cpu.all.soft%
     38.55            -9.0       29.59        mpstat.cpu.all.sys%
     41.67           -14.2       27.49 ±  2%  mpstat.cpu.all.usr%
   3990675           -22.9%    3074963 ±  2%  numa-numastat.node1.local_node
   4021819           -23.1%    3092032 ±  2%  numa-numastat.node1.numa_hit
     31144           -45.2%      17068 ± 64%  numa-numastat.node1.other_node
   4722883 ± 29%     -33.2%    3155351 ±  3%  numa-numastat.node2.local_node
   4747771 ± 29%     -32.9%    3186380 ±  3%  numa-numastat.node2.numa_hit
      2404           -28.1%       1728 ±  2%  turbostat.Avg_MHz
     80.61           -22.4       58.22 ±  2%  turbostat.Busy%
  20996179 ± 15%    +109.1%   43902033 ± 21%  turbostat.C1E
     18.85 ±  2%    +115.1%      40.56 ±  3%  turbostat.CPU%c1
    477.24           -11.9%     420.57        turbostat.PkgWatt
     17150 ±  3%     +13.7%      19503 ±  3%  slabinfo.cred_jar.active_objs
     17150 ±  3%     +13.7%      19503 ±  3%  slabinfo.cred_jar.num_objs
      1757 ±  5%     -11.9%       1547 ±  4%  slabinfo.kmalloc-rcl-96.active_objs
      1757 ±  5%     -11.9%       1547 ±  4%  slabinfo.kmalloc-rcl-96.num_objs
     17269           -11.2%      15337 ±  2%  slabinfo.task_delay_info.active_objs
     17269           -11.2%      15337 ±  2%  slabinfo.task_delay_info.num_objs
  45090908           -30.3%   31436724 ±  2%  meminfo.Active
  45090677           -30.3%   31436484 ±  2%  meminfo.Active(anon)
  33671307           -30.5%   23416058        meminfo.AnonHugePages
  44973813           -30.4%   31321842 ±  2%  meminfo.AnonPages
  90656336           -19.4%   73097429        meminfo.Committed_AS
    568845 ± 10%     -23.9%     432780 ±  4%  meminfo.DirectMap4k
  47516120           -28.9%   33772543 ±  2%  meminfo.Memused
     79828           -28.6%      56964 ±  2%  meminfo.PageTables
    303260 ±  2%     -10.9%     270236 ±  4%  meminfo.max_used_kB
  11010213 ±  3%     -27.8%    7949775 ± 22%  numa-meminfo.node0.Active
  11010184 ±  3%     -27.8%    7949699 ± 22%  numa-meminfo.node0.Active(anon)
  10955056 ±  3%     -27.7%    7918969 ± 22%  numa-meminfo.node0.AnonPages
  11634902 ±  3%     -26.3%    8570261 ± 21%  numa-meminfo.node0.MemUsed
  11216510 ±  3%     -40.5%    6673714 ±  5%  numa-meminfo.node1.Active
  11216510 ±  3%     -40.5%    6673685 ±  5%  numa-meminfo.node1.Active(anon)
   8326268 ±  4%     -40.7%    4934615 ±  5%  numa-meminfo.node1.AnonHugePages
  11198925 ±  3%     -40.6%    6650406 ±  5%  numa-meminfo.node1.AnonPages
  37759405           +12.1%   42320486        numa-meminfo.node1.MemFree
  11767954 ±  3%     -38.8%    7206873 ±  5%  numa-meminfo.node1.MemUsed
     18759 ±  2%     -37.9%      11650 ±  5%  numa-meminfo.node1.PageTables
  11526969 ±  9%     -41.4%    6752594 ±  9%  numa-meminfo.node2.Active
  11526927 ±  9%     -41.4%    6752552 ±  9%  numa-meminfo.node2.Active(anon)
   8576117 ± 10%     -41.9%    4982950 ± 10%  numa-meminfo.node2.AnonHugePages
  11505263 ±  9%     -41.5%    6735650 ±  9%  numa-meminfo.node2.AnonPages
  37426528 ±  2%     +12.9%   42249731        numa-meminfo.node2.MemFree
  12100840 ±  9%     -39.9%    7277636 ±  8%  numa-meminfo.node2.MemUsed
     19912 ± 13%     -37.6%      12427 ± 13%  numa-meminfo.node2.PageTables
      7507 ± 25%     -25.9%       5563 ±  5%  numa-meminfo.node3.Mapped
   2653570 ±  3%     -25.9%    1965712 ± 22%  numa-vmstat.node0.nr_active_anon
   2650970 ±  3%     -25.9%    1964507 ± 22%  numa-vmstat.node0.nr_zone_active_anon
   2682266 ±  4%     -37.6%    1672693 ±  5%  numa-vmstat.node1.nr_active_anon
   2677270 ±  4%     -37.8%    1666244 ±  5%  numa-vmstat.node1.nr_anon_pages
      3905 ±  6%     -38.4%       2405 ±  5%  numa-vmstat.node1.nr_anon_transparent_hugepages
   9564136           +10.6%   10577037        numa-vmstat.node1.nr_free_pages
      4608 ±  3%     -37.0%       2901 ±  5%  numa-vmstat.node1.nr_page_table_pages
   2680195 ±  4%     -37.6%    1671790 ±  5%  numa-vmstat.node1.nr_zone_active_anon
   2295337 ±  5%     -20.7%    1821302 ±  5%  numa-vmstat.node1.numa_hit
   2180885 ±  5%     -21.1%    1720231 ±  5%  numa-vmstat.node1.numa_local
   2839093 ± 10%     -39.9%    1707571 ± 11%  numa-vmstat.node2.nr_active_anon
   2833805 ± 10%     -39.9%    1703266 ± 11%  numa-vmstat.node2.nr_anon_pages
      4158 ± 12%     -40.7%       2465 ± 12%  numa-vmstat.node2.nr_anon_transparent_hugepages
     25.50 ±116%    +204.9%      77.75 ± 80%  numa-vmstat.node2.nr_dirtied
   9402050 ±  3%     +12.1%   10544232        numa-vmstat.node2.nr_free_pages
     34.50 ±131%    +145.7%      84.75 ± 76%  numa-vmstat.node2.nr_inactive_file
      4980 ± 14%     -37.3%       3120 ± 12%  numa-vmstat.node2.nr_page_table_pages
     24.25 ±123%    +193.8%      71.25 ± 79%  numa-vmstat.node2.nr_written
   2836803 ± 10%     -39.8%    1706487 ± 11%  numa-vmstat.node2.nr_zone_active_anon
     34.50 ±131%    +145.7%      84.75 ± 76%  numa-vmstat.node2.nr_zone_inactive_file
   2612725 ± 30%     -27.7%    1887879 ±  6%  numa-vmstat.node2.numa_hit
   2504430 ± 31%     -29.2%    1773066 ±  7%  numa-vmstat.node2.numa_local
  11388953           -31.7%    7777830        proc-vmstat.nr_active_anon
  11358111           -31.8%    7748673        proc-vmstat.nr_anon_pages
     16635           -31.6%      11372        proc-vmstat.nr_anon_transparent_hugepages
    441.75           +20.8%     533.75        proc-vmstat.nr_dirtied
   3718102            +9.8%    4082210        proc-vmstat.nr_dirty_background_threshold
   7448721            +9.8%    8175520        proc-vmstat.nr_dirty_threshold
  37434915            +9.7%   41069758        proc-vmstat.nr_free_pages
    529.50            +8.3%     573.25        proc-vmstat.nr_inactive_file
     28930            -1.0%      28637        proc-vmstat.nr_kernel_stack
      9789 ±  2%      -3.2%       9472        proc-vmstat.nr_mapped
     20077           -29.2%      14204        proc-vmstat.nr_page_table_pages
     46878            -5.0%      44552 ±  2%  proc-vmstat.nr_shmem
    416.00           +20.5%     501.25        proc-vmstat.nr_written
  11388927           -31.7%    7777820        proc-vmstat.nr_zone_active_anon
    529.50            +8.3%     573.25        proc-vmstat.nr_zone_inactive_file
  19364148           -21.9%   15132755 ±  2%  proc-vmstat.numa_hit
     14619 ±  5%     -43.5%       8257 ± 10%  proc-vmstat.numa_huge_pte_updates
  19270325           -22.0%   15039253 ±  2%  proc-vmstat.numa_local
   7544881 ±  5%     -43.3%    4275462 ± 10%  proc-vmstat.numa_pte_updates
     27144 ±  5%     -30.7%      18821 ±  3%  proc-vmstat.pgactivate
 2.311e+09           -23.1%  1.776e+09 ±  2%  proc-vmstat.pgalloc_normal
  14816778           -21.5%   11633019 ±  2%  proc-vmstat.pgfault
 2.308e+09           -23.4%  1.767e+09 ±  2%  proc-vmstat.pgfree
   4478662           -23.4%    3428568 ±  2%  proc-vmstat.thp_deferred_split_page
   4483698           -23.1%    3446054 ±  2%  proc-vmstat.thp_fault_alloc
    108799 ± 11%     -36.7%      68886 ±  3%  sched_debug.cfs_rq:/.exec_clock.avg
    117668 ± 11%     -22.3%      91468 ±  2%  sched_debug.cfs_rq:/.exec_clock.max
     98854 ± 11%     -48.6%      50806 ±  4%  sched_debug.cfs_rq:/.exec_clock.min
      4923 ± 19%    +124.0%      11028 ± 11%  sched_debug.cfs_rq:/.exec_clock.stddev
    285.55 ± 16%     -17.5%     235.65 ± 20%  sched_debug.cfs_rq:/.load_avg.max
      1.28 ± 35%    -100.0%       0.00        sched_debug.cfs_rq:/.load_avg.min
  20824703 ± 11%     -49.5%   10507532 ±  3%  sched_debug.cfs_rq:/.min_vruntime.avg
  22103859 ± 12%     -43.6%   12472189 ±  3%  sched_debug.cfs_rq:/.min_vruntime.max
  18960410 ± 12%     -58.0%    7969687 ±  5%  sched_debug.cfs_rq:/.min_vruntime.min
    649563 ± 30%     +62.7%    1056596 ± 12%  sched_debug.cfs_rq:/.min_vruntime.stddev
      0.64 ± 13%     -24.2%       0.48 ±  9%  sched_debug.cfs_rq:/.nr_running.avg
      0.31 ± 19%     +28.6%       0.40 ±  8%  sched_debug.cfs_rq:/.nr_running.stddev
      9.18 ± 10%     -33.1%       6.15 ±  4%  sched_debug.cfs_rq:/.nr_spread_over.avg
     65.36 ± 20%     -35.4%      42.25 ± 33%  sched_debug.cfs_rq:/.nr_spread_over.max
      7.30 ± 15%     -33.5%       4.85 ± 13%  sched_debug.cfs_rq:/.nr_spread_over.stddev
   1791102 ± 26%     +35.2%    2420709 ± 10%  sched_debug.cfs_rq:/.spread0.avg
   3067181 ± 20%     +42.7%    4377704 ±  7%  sched_debug.cfs_rq:/.spread0.max
    651962 ± 30%     +62.0%    1055918 ± 12%  sched_debug.cfs_rq:/.spread0.stddev
    773.26 ±  7%     -20.2%     617.26 ±  4%  sched_debug.cfs_rq:/.util_avg.avg
    188.92 ± 19%     -99.4%       1.05 ±105%  sched_debug.cfs_rq:/.util_avg.min
    184.74 ± 23%     +58.9%     293.65 ± 12%  sched_debug.cfs_rq:/.util_avg.stddev
      1166 ± 10%     -17.1%     966.80 ±  8%  sched_debug.cfs_rq:/.util_est_enqueued.max
    126.46 ± 15%     -46.2%      67.98 ± 29%  sched_debug.cpu.clock.stddev
    126.46 ± 15%     -46.2%      67.98 ± 29%  sched_debug.cpu.clock_task.stddev
     10644 ± 12%     -37.8%       6622 ±  7%  sched_debug.cpu.curr->pid.avg
     15239 ±  9%     -20.2%      12163        sched_debug.cpu.curr->pid.max
      0.00 ± 15%     -49.1%       0.00 ± 28%  sched_debug.cpu.next_balance.stddev
    144767 ± 10%     -11.1%     128756        sched_debug.cpu.nr_load_updates.avg
    125750 ±  9%     -17.2%     104090 ± 10%  sched_debug.cpu.nr_load_updates.min
      0.64 ± 13%     -28.8%       0.45 ±  7%  sched_debug.cpu.nr_running.avg
      1.51 ±  7%     -23.8%       1.15 ± 14%  sched_debug.cpu.nr_running.max
     11047 ± 12%     -41.7%       6438        sched_debug.cpu.nr_switches.avg
      6783 ± 10%     -44.3%       3775 ±  3%  sched_debug.cpu.nr_switches.min
      0.16 ± 51%     +91.3%       0.30 ± 25%  sched_debug.cpu.nr_uninterruptible.avg
    -23.28           +27.6%     -29.70        sched_debug.cpu.nr_uninterruptible.min
     14.58 ±  2%     +46.8%      21.41        sched_debug.cpu.nr_uninterruptible.stddev
     10564 ± 12%     -44.1%       5908        sched_debug.cpu.sched_count.avg
      6543 ± 11%     -45.8%       3546 ±  2%  sched_debug.cpu.sched_count.min
      4217 ± 11%     -37.3%       2643        sched_debug.cpu.sched_goidle.avg
      2673 ± 12%     -47.3%       1409 ± 15%  sched_debug.cpu.sched_goidle.min
      5061 ± 12%     -42.6%       2902 ±  2%  sched_debug.cpu.ttwu_count.avg
      2587 ± 14%     -53.7%       1197 ±  3%  sched_debug.cpu.ttwu_count.min
    818.87 ± 18%     -52.2%     391.10 ±  2%  sched_debug.cpu.ttwu_local.avg
     17144 ± 28%     -64.4%       6106 ± 25%  sched_debug.cpu.ttwu_local.max
    228.22 ± 12%     -20.1%     182.40 ±  2%  sched_debug.cpu.ttwu_local.min
      1796 ± 28%     -72.7%     489.75 ± 17%  sched_debug.cpu.ttwu_local.stddev
      5.02            -4.6%       4.79 ±  2%  perf-stat.i.MPKI
 3.584e+10           -20.8%  2.839e+10 ±  2%  perf-stat.i.branch-instructions
      0.02 ±  5%      +0.0        0.03 ± 25%  perf-stat.i.branch-miss-rate%
     70.15           -10.3       59.89 ±  2%  perf-stat.i.cache-miss-rate%
 3.988e+08           -30.7%  2.762e+08 ±  2%  perf-stat.i.cache-misses
 5.652e+08           -21.8%  4.419e+08        perf-stat.i.cache-references
     15914           -35.2%      10304        perf-stat.i.context-switches
      3.81           -14.1%       3.27        perf-stat.i.cpi
 4.361e+11           -28.3%  3.126e+11        perf-stat.i.cpu-cycles
    305.95 ±  3%     -16.3%     255.98 ±  2%  perf-stat.i.cpu-migrations
      1080           +18.5%       1280 ±  6%  perf-stat.i.cycles-between-cache-misses
 2.472e+10           -18.0%  2.026e+10        perf-stat.i.dTLB-loads
      0.00 ±  2%      +0.0        0.00 ± 16%  perf-stat.i.dTLB-store-miss-rate%
    301820 ±  2%     -15.5%     255096 ±  9%  perf-stat.i.dTLB-store-misses
 8.332e+09           -22.0%  6.497e+09 ±  2%  perf-stat.i.dTLB-stores
     77.38           -14.2       63.16        perf-stat.i.iTLB-load-miss-rate%
   2037871            +6.8%    2177306 ±  2%  perf-stat.i.iTLB-load-misses
    762134           +85.5%    1413938 ±  4%  perf-stat.i.iTLB-loads
 1.129e+11           -19.1%  9.132e+10 ±  2%  perf-stat.i.instructions
     59923           -25.4%      44683        perf-stat.i.instructions-per-iTLB-miss
      0.27           +22.8%       0.33        perf-stat.i.ipc
     49321           -21.9%      38543 ±  2%  perf-stat.i.minor-faults
     18.01            +4.5       22.49        perf-stat.i.node-load-miss-rate%
   1432415           -12.0%    1260792        perf-stat.i.node-load-misses
   6775281           -28.9%    4818204 ±  2%  perf-stat.i.node-loads
 1.205e+08           -26.7%   88302418 ±  2%  perf-stat.i.node-stores
     49298           -21.8%      38548 ±  2%  perf-stat.i.page-faults
      5.00            -3.4%       4.84        perf-stat.overall.MPKI
      0.02 ±  5%      +0.0        0.03 ± 20%  perf-stat.overall.branch-miss-rate%
     70.65            -8.0       62.65 ±  2%  perf-stat.overall.cache-miss-rate%
      3.87           -11.1%       3.44        perf-stat.overall.cpi
      1094            +3.8%       1135        perf-stat.overall.cycles-between-cache-misses
     72.93           -12.1       60.85        perf-stat.overall.iTLB-load-miss-rate%
     55490           -23.8%      42278        perf-stat.overall.instructions-per-iTLB-miss
      0.26           +12.5%       0.29        perf-stat.overall.ipc
     17.41            +3.4       20.83        perf-stat.overall.node-load-miss-rate%
      3179            +6.5%       3387        perf-stat.overall.path-length
 3.561e+10           -20.3%  2.836e+10 ±  2%  perf-stat.ps.branch-instructions
 3.966e+08           -30.3%  2.764e+08 ±  2%  perf-stat.ps.cache-misses
 5.614e+08           -21.4%  4.412e+08        perf-stat.ps.cache-references
     15636           -35.3%      10116        perf-stat.ps.context-switches
 4.339e+11           -27.7%  3.139e+11        perf-stat.ps.cpu-cycles
    304.71 ±  3%     -15.8%     256.64 ±  2%  perf-stat.ps.cpu-migrations
 2.457e+10           -17.6%  2.025e+10        perf-stat.ps.dTLB-loads
    299980 ±  2%     -15.6%     253225 ±  9%  perf-stat.ps.dTLB-store-misses
 8.285e+09           -21.6%  6.491e+09 ±  2%  perf-stat.ps.dTLB-stores
   2021880            +6.8%    2158636 ±  2%  perf-stat.ps.iTLB-load-misses
    750297           +85.3%    1389972 ±  4%  perf-stat.ps.iTLB-loads
 1.122e+11           -18.7%  9.126e+10 ±  2%  perf-stat.ps.instructions
     49039           -21.8%      38340 ±  2%  perf-stat.ps.minor-faults
   1411907           -11.6%    1248139        perf-stat.ps.node-load-misses
   6699062           -29.2%    4743984 ±  2%  perf-stat.ps.node-loads
   1.2e+08           -26.2%   88495173 ±  2%  perf-stat.ps.node-stores
     49041           -21.8%      38339 ±  2%  perf-stat.ps.page-faults
 3.298e+13           -18.0%  2.704e+13 ±  2%  perf-stat.total.instructions
     69.87 ±  6%     -11.9       57.98 ± 16%  perf-profile.calltrace.cycles-pp.do_access
     42.62 ±  6%      -9.0       33.64 ± 17%  perf-profile.calltrace.cycles-pp.handle_mm_fault.__do_page_fault.do_page_fault.page_fault.do_access
     30.80 ±  6%      -7.6       23.21 ± 18%  perf-profile.calltrace.cycles-pp.do_rw_once
      1.48 ±  6%      -0.4        1.07 ± 28%  perf-profile.calltrace.cycles-pp.rcu_all_qs._cond_resched.clear_huge_page.do_huge_pmd_anonymous_page.__handle_mm_fault
      0.72 ±  4%      -0.3        0.47 ± 59%  perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.do_rw_once
      0.84 ±  6%      -0.2        0.65 ± 11%  perf-profile.calltrace.cycles-pp._cond_resched.clear_huge_page.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault
      0.86 ±  5%      -0.2        0.70 ± 18%  perf-profile.calltrace.cycles-pp.apic_timer_interrupt.do_rw_once
      0.00            +0.7        0.72 ± 20%  perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle
      0.00            +0.8        0.79 ± 21%  perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      0.00            +4.4        4.43 ± 12%  perf-profile.calltrace.cycles-pp.rwsem_down_read_slowpath.__do_page_fault.do_page_fault.page_fault.do_access
      0.00            +4.8        4.81 ± 22%  perf-profile.calltrace.cycles-pp.osq_lock.rwsem_optimistic_spin.rwsem_down_read_slowpath.__do_page_fault.do_page_fault
      0.00            +5.0        4.98 ± 23%  perf-profile.calltrace.cycles-pp.rwsem_optimistic_spin.rwsem_down_read_slowpath.__do_page_fault.do_page_fault.page_fault
      2.90 ±  9%      +7.8       10.71 ± 16%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      3.22 ±  9%      +8.3       11.56 ± 16%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
      3.24 ±  9%      +8.4       11.64 ± 16%  perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      3.43 ±  9%      +8.6       12.08 ± 16%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
      3.43 ±  9%      +8.6       12.08 ± 16%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
      3.43 ±  9%      +8.7       12.08 ± 16%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      3.47 ±  9%      +8.7       12.21 ± 16%  perf-profile.calltrace.cycles-pp.secondary_startup_64
     40.01 ±  6%      -9.8       30.18 ± 18%  perf-profile.children.cycles-pp.do_rw_once
      0.24 ±  5%      -0.0        0.20 ± 10%  perf-profile.children.cycles-pp.do_anonymous_page
      0.19 ±  6%      -0.0        0.14 ±  9%  perf-profile.children.cycles-pp.alloc_pages_vma
      0.07 ± 17%      +0.0        0.09 ±  4%  perf-profile.children.cycles-pp.update_sd_lb_stats
      0.07 ± 17%      +0.0        0.10 ±  5%  perf-profile.children.cycles-pp.find_busiest_group
      0.07 ± 10%      +0.0        0.11 ± 28%  perf-profile.children.cycles-pp.rcu_sched_clock_irq
      0.08 ± 13%      +0.0        0.12 ±  4%  perf-profile.children.cycles-pp.load_balance
      0.01 ±173%      +0.0        0.06 ±  9%  perf-profile.children.cycles-pp.unmap_vmas
      0.01 ±173%      +0.0        0.06 ±  9%  perf-profile.children.cycles-pp.unmap_page_range
      0.17 ± 14%      +0.1        0.22 ± 21%  perf-profile.children.cycles-pp.munmap
      0.00            +0.1        0.06 ± 22%  perf-profile.children.cycles-pp.lapic_next_deadline
      0.06 ± 13%      +0.1        0.12 ±  6%  perf-profile.children.cycles-pp.rebalance_domains
      0.22 ±  5%      +0.1        0.30 ±  9%  perf-profile.children.cycles-pp.__softirqentry_text_start
      0.13 ± 14%      +0.1        0.22 ±  7%  perf-profile.children.cycles-pp.__free_pages_ok
      0.06 ± 11%      +0.1        0.16 ± 14%  perf-profile.children.cycles-pp.tick_nohz_next_event
      0.17 ± 13%      +0.1        0.27 ±  5%  perf-profile.children.cycles-pp.release_pages
      0.27 ±  5%      +0.1        0.38 ±  8%  perf-profile.children.cycles-pp.irq_exit
      0.17 ± 13%      +0.1        0.27 ±  7%  perf-profile.children.cycles-pp.tlb_finish_mmu
      0.17 ± 13%      +0.1        0.27 ±  7%  perf-profile.children.cycles-pp.tlb_flush_mmu
      0.07 ± 10%      +0.1        0.18 ± 17%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.21 ± 14%      +0.1        0.33 ±  8%  perf-profile.children.cycles-pp.__do_munmap
      0.21 ± 15%      +0.1        0.34 ±  7%  perf-profile.children.cycles-pp.__vm_munmap
      0.19 ± 13%      +0.1        0.32 ±  7%  perf-profile.children.cycles-pp.unmap_region
      0.21 ± 14%      +0.1        0.34 ±  7%  perf-profile.children.cycles-pp.__x64_sys_munmap
      0.23 ± 19%      +0.2        0.40 ± 16%  perf-profile.children.cycles-pp.update_cfs_group
      0.11 ± 12%      +0.2        0.29 ± 22%  perf-profile.children.cycles-pp.menu_select
      0.81 ± 13%      +0.2        1.00 ±  7%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.69 ±  5%      +0.2        0.92 ±  7%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.69 ±  5%      +0.2        0.92 ±  7%  perf-profile.children.cycles-pp.do_syscall_64
      0.00            +4.8        4.82 ± 21%  perf-profile.children.cycles-pp.osq_lock
      0.00            +5.0        5.00 ± 23%  perf-profile.children.cycles-pp.rwsem_optimistic_spin
      0.10 ± 13%      +5.0        5.12 ± 22%  perf-profile.children.cycles-pp.rwsem_down_read_slowpath
      2.95 ± 10%      +7.9       10.83 ± 16%  perf-profile.children.cycles-pp.intel_idle
      3.29 ±  9%      +8.5       11.76 ± 16%  perf-profile.children.cycles-pp.cpuidle_enter
      3.29 ±  9%      +8.5       11.76 ± 16%  perf-profile.children.cycles-pp.cpuidle_enter_state
      3.43 ±  9%      +8.6       12.08 ± 16%  perf-profile.children.cycles-pp.start_secondary
      3.47 ±  9%      +8.7       12.21 ± 16%  perf-profile.children.cycles-pp.do_idle
      3.47 ±  9%      +8.7       12.21 ± 16%  perf-profile.children.cycles-pp.secondary_startup_64
      3.47 ±  9%      +8.7       12.21 ± 16%  perf-profile.children.cycles-pp.cpu_startup_entry
     38.36 ±  6%      -9.6       28.80 ± 18%  perf-profile.self.cycles-pp.do_rw_once
      7.60 ±  6%      -2.5        5.07 ± 18%  perf-profile.self.cycles-pp.do_access
      0.10 ±  5%      -0.0        0.06 ± 11%  perf-profile.self.cycles-pp.__handle_mm_fault
      0.04 ± 59%      +0.0        0.08 ±  6%  perf-profile.self.cycles-pp.update_sd_lb_stats
      0.06 ± 13%      +0.0        0.11 ± 25%  perf-profile.self.cycles-pp.rcu_sched_clock_irq
      0.11 ± 13%      +0.1        0.19 ±  8%  perf-profile.self.cycles-pp.__free_pages_ok
      0.00            +0.1        0.09 ± 28%  perf-profile.self.cycles-pp.menu_select
      0.00            +0.1        0.10 ± 22%  perf-profile.self.cycles-pp.cpuidle_enter_state
      0.00            +0.2        0.15 ± 57%  perf-profile.self.cycles-pp.rwsem_optimistic_spin
      0.23 ± 19%      +0.2        0.40 ± 16%  perf-profile.self.cycles-pp.update_cfs_group
      0.00            +4.8        4.77 ± 21%  perf-profile.self.cycles-pp.osq_lock
      2.94 ± 10%      +7.9       10.83 ± 16%  perf-profile.self.cycles-pp.intel_idle
     30379 ±  4%     -12.5%      26581        softirqs.CPU0.RCU
     17842 ±  4%     +62.7%      29021 ±  8%  softirqs.CPU0.SCHED
     12748 ± 10%     +62.8%      20757 ± 13%  softirqs.CPU1.SCHED
    112729 ±  4%      +8.4%     122207        softirqs.CPU1.TIMER
     32056 ±  3%     -14.0%      27582 ±  2%  softirqs.CPU10.RCU
     13372 ±  7%     +48.7%      19891 ± 15%  softirqs.CPU10.SCHED
     13163 ± 13%     +56.7%      20628 ± 17%  softirqs.CPU100.SCHED
     13220 ±  8%     +56.5%      20689 ± 16%  softirqs.CPU101.SCHED
    111761 ±  5%      +8.5%     121220        softirqs.CPU101.TIMER
     13280 ± 10%     +56.7%      20806 ± 14%  softirqs.CPU102.SCHED
     31322 ±  5%     -11.2%      27817        softirqs.CPU103.RCU
     13245 ±  8%     +54.5%      20461 ± 14%  softirqs.CPU103.SCHED
     13470 ±  9%     +55.2%      20910 ± 17%  softirqs.CPU104.SCHED
     31043 ±  5%     -11.0%      27631        softirqs.CPU105.RCU
     13171 ±  8%     +57.6%      20753 ± 16%  softirqs.CPU105.SCHED
     32204 ±  5%     -14.8%      27443        softirqs.CPU106.RCU
     13560 ±  4%     +51.8%      20590 ± 17%  softirqs.CPU106.SCHED
     31484 ±  5%      -8.8%      28711 ±  4%  softirqs.CPU107.RCU
     13376 ±  9%     +52.6%      20415 ± 17%  softirqs.CPU107.SCHED
     13179 ±  9%     +53.2%      20196 ± 15%  softirqs.CPU108.SCHED
     30932 ±  5%      -9.5%      28009 ±  3%  softirqs.CPU109.RCU
     13315 ±  9%     +53.1%      20390 ± 16%  softirqs.CPU109.SCHED
     31762 ±  4%     -10.3%      28485        softirqs.CPU11.RCU
     13133 ±  8%     +56.5%      20559 ± 15%  softirqs.CPU11.SCHED
     31055 ±  4%     -11.0%      27630        softirqs.CPU110.RCU
     13271 ±  9%     +54.0%      20443 ± 17%  softirqs.CPU110.SCHED
     31417 ±  4%     -12.9%      27357        softirqs.CPU111.RCU
     13352 ±  9%     +53.5%      20500 ± 17%  softirqs.CPU111.SCHED
     31743 ±  4%     -14.0%      27295        softirqs.CPU112.RCU
     13334 ±  9%     +53.4%      20461 ± 16%  softirqs.CPU112.SCHED
     31649 ±  4%     -12.2%      27796        softirqs.CPU113.RCU
     13586 ±  9%     +51.2%      20542 ± 16%  softirqs.CPU113.SCHED
     32529 ±  8%     -13.3%      28213 ±  2%  softirqs.CPU114.RCU
     13423 ±  8%     +51.9%      20388 ± 16%  softirqs.CPU114.SCHED
     31879 ±  6%     -13.3%      27646        softirqs.CPU115.RCU
     13357 ±  9%     +53.0%      20442 ± 17%  softirqs.CPU115.SCHED
     31622 ±  4%      -9.7%      28569 ±  5%  softirqs.CPU116.RCU
     13626 ±  9%     +52.6%      20789 ± 14%  softirqs.CPU116.SCHED
     31089 ±  5%     -11.8%      27412        softirqs.CPU117.RCU
     12525 ± 14%     +66.6%      20866 ± 17%  softirqs.CPU117.SCHED
     13348 ±  9%     +55.9%      20815 ± 17%  softirqs.CPU118.SCHED
    112643 ±  4%      +8.5%     122261        softirqs.CPU118.TIMER
     31601 ±  4%     -12.8%      27540        softirqs.CPU119.RCU
     13363 ±  8%     +49.6%      19995 ± 16%  softirqs.CPU119.SCHED
     33443 ± 12%     -14.6%      28564        softirqs.CPU12.RCU
     13720 ± 12%     +49.4%      20492 ± 14%  softirqs.CPU12.SCHED
     31831 ±  5%     -15.2%      26996 ±  2%  softirqs.CPU120.RCU
     12720 ±  9%     +76.8%      22491 ±  2%  softirqs.CPU120.SCHED
     32010 ±  3%     -15.3%      27101 ±  2%  softirqs.CPU121.RCU
     12802 ± 11%     +75.2%      22431 ±  3%  softirqs.CPU121.SCHED
     30635 ±  5%      -8.7%      27972 ±  6%  softirqs.CPU122.RCU
     12666 ± 12%     +80.5%      22856 ±  4%  softirqs.CPU122.SCHED
     31969 ±  2%     -15.4%      27041 ±  3%  softirqs.CPU123.RCU
     12565 ± 12%     +79.2%      22518 ±  3%  softirqs.CPU123.SCHED
     31849 ±  3%     -15.4%      26938 ±  2%  softirqs.CPU124.RCU
     12703 ± 10%     +75.7%      22323 ±  3%  softirqs.CPU124.SCHED
     32084 ±  3%     -15.6%      27087 ±  2%  softirqs.CPU125.RCU
     12639 ± 10%     +76.3%      22280 ±  2%  softirqs.CPU125.SCHED
     31975 ±  3%     -15.7%      26959 ±  2%  softirqs.CPU126.RCU
     12668 ±  9%     +72.4%      21836 ±  5%  softirqs.CPU126.SCHED
     31527 ±  5%     -14.3%      27020 ±  2%  softirqs.CPU127.RCU
     12542 ±  9%     +79.4%      22495 ±  3%  softirqs.CPU127.SCHED
     31366 ±  2%     -12.8%      27348 ±  2%  softirqs.CPU128.RCU
     12644 ± 11%     +76.6%      22330 ±  3%  softirqs.CPU128.SCHED
     31189 ±  2%     -11.8%      27499 ±  2%  softirqs.CPU129.RCU
     12696 ±  9%     +77.1%      22480        softirqs.CPU129.SCHED
     13020 ±  8%     +56.6%      20385 ± 15%  softirqs.CPU13.SCHED
     31251 ±  3%     -11.5%      27664 ±  2%  softirqs.CPU130.RCU
     12720 ±  9%     +75.6%      22337 ±  3%  softirqs.CPU130.SCHED
     31795           -13.0%      27663 ±  2%  softirqs.CPU131.RCU
     12715 ± 11%     +75.1%      22266 ±  2%  softirqs.CPU131.SCHED
     31456 ±  3%     -12.1%      27662 ±  3%  softirqs.CPU132.RCU
     12661 ± 10%     +76.7%      22370 ±  2%  softirqs.CPU132.SCHED
     30812 ±  4%     -12.6%      26922 ±  3%  softirqs.CPU133.RCU
     12542 ± 12%     +82.2%      22851 ±  5%  softirqs.CPU133.SCHED
     31150 ±  2%     -12.7%      27197 ±  3%  softirqs.CPU134.RCU
     12669 ± 11%     +74.1%      22063 ±  3%  softirqs.CPU134.SCHED
     31286 ±  3%     -11.6%      27648 ±  2%  softirqs.CPU135.RCU
     12685 ±  9%     +80.7%      22921 ±  3%  softirqs.CPU135.SCHED
     31466 ±  2%     -12.3%      27610 ±  3%  softirqs.CPU136.RCU
     12643 ± 10%     +78.4%      22560 ±  2%  softirqs.CPU136.SCHED
     31550 ±  2%     -12.9%      27464 ±  2%  softirqs.CPU137.RCU
     12744 ±  9%     +75.0%      22309 ±  2%  softirqs.CPU137.SCHED
     31343 ±  3%     -12.5%      27418 ±  2%  softirqs.CPU138.RCU
     12741 ± 10%     +68.8%      21506 ± 12%  softirqs.CPU138.SCHED
     31628 ±  2%     -13.0%      27505 ±  2%  softirqs.CPU139.RCU
     12659 ± 11%     +76.9%      22398 ±  6%  softirqs.CPU139.SCHED
     33226 ±  9%     -16.0%      27898        softirqs.CPU14.RCU
     13074 ±  7%     +51.3%      19779 ± 16%  softirqs.CPU14.SCHED
     31554 ±  3%     -11.3%      27993 ±  3%  softirqs.CPU140.RCU
     12730 ±  9%     +78.0%      22663 ±  4%  softirqs.CPU140.SCHED
     12497 ± 13%     +82.2%      22769 ±  3%  softirqs.CPU141.SCHED
     31195 ±  4%     -12.3%      27362 ±  4%  softirqs.CPU142.RCU
     12640 ± 12%     +81.9%      22989 ±  6%  softirqs.CPU142.SCHED
     31194 ±  4%     -13.2%      27074 ±  4%  softirqs.CPU143.RCU
     12716 ± 11%     +72.5%      21937 ± 13%  softirqs.CPU143.SCHED
     31035 ±  4%      -9.9%      27958 ±  2%  softirqs.CPU144.RCU
     13145 ±  8%     +72.0%      22605 ±  7%  softirqs.CPU144.SCHED
     31116 ±  3%     -11.4%      27567        softirqs.CPU145.RCU
     13086 ±  8%     +70.4%      22297 ±  6%  softirqs.CPU145.SCHED
     30996 ±  3%     -13.6%      26791 ±  2%  softirqs.CPU146.RCU
     13084 ±  8%     +58.7%      20761 ± 19%  softirqs.CPU146.SCHED
     30902 ±  3%     -10.1%      27769        softirqs.CPU147.RCU
     13209 ±  8%     +69.5%      22392 ±  6%  softirqs.CPU147.SCHED
     30816 ±  3%      -9.3%      27938        softirqs.CPU148.RCU
     13056 ±  7%     +74.1%      22731 ±  7%  softirqs.CPU148.SCHED
     30876 ±  3%      -9.6%      27926        softirqs.CPU149.RCU
     13104 ±  7%     +66.5%      21813 ± 11%  softirqs.CPU149.SCHED
     32201 ±  6%     -13.2%      27952 ±  2%  softirqs.CPU15.RCU
     13135 ±  6%     +57.2%      20646 ± 14%  softirqs.CPU15.SCHED
     30836 ±  3%     -10.0%      27752        softirqs.CPU150.RCU
     13288 ±  6%     +65.0%      21919 ± 10%  softirqs.CPU150.SCHED
     31074 ±  3%     -11.4%      27531        softirqs.CPU151.RCU
     13178 ±  8%     +70.8%      22506 ±  9%  softirqs.CPU151.SCHED
     30955 ±  3%     -10.2%      27786        softirqs.CPU152.RCU
     13182 ±  8%     +75.6%      23155 ±  2%  softirqs.CPU152.SCHED
     31032 ±  3%     -13.3%      26914 ±  4%  softirqs.CPU153.RCU
     13156 ±  8%     +69.5%      22293 ±  7%  softirqs.CPU153.SCHED
     31190 ±  3%     -11.5%      27609        softirqs.CPU154.RCU
     13279 ±  7%     +67.4%      22223 ±  7%  softirqs.CPU154.SCHED
     31230 ±  2%     -10.7%      27889        softirqs.CPU155.RCU
     13298 ±  5%     +69.1%      22485 ±  6%  softirqs.CPU155.SCHED
     31098 ±  2%     -10.9%      27722        softirqs.CPU156.RCU
     13306 ±  6%     +71.6%      22834 ±  7%  softirqs.CPU156.SCHED
     31711 ±  6%     -16.7%      26402 ±  3%  softirqs.CPU157.RCU
     13212 ±  7%     +74.3%      23031 ±  8%  softirqs.CPU157.SCHED
     30982 ±  3%     -11.6%      27383        softirqs.CPU158.RCU
     13159 ±  7%     +71.6%      22585 ±  8%  softirqs.CPU158.SCHED
     30935 ±  3%      -9.9%      27859        softirqs.CPU159.RCU
     13305 ±  8%     +71.5%      22816 ±  6%  softirqs.CPU159.SCHED
     32162 ±  4%     -12.6%      28095        softirqs.CPU16.RCU
     12923 ±  8%     +54.4%      19953 ± 16%  softirqs.CPU16.SCHED
     31177 ±  3%     -10.7%      27829 ±  4%  softirqs.CPU160.RCU
     13060 ±  8%     +75.9%      22970 ±  7%  softirqs.CPU160.SCHED
     31499 ±  3%     -12.5%      27562 ±  3%  softirqs.CPU161.RCU
     13111 ±  8%     +76.9%      23190 ±  8%  softirqs.CPU161.SCHED
     30994 ±  2%     -10.1%      27854 ±  4%  softirqs.CPU162.RCU
     13231 ±  9%     +74.1%      23036 ±  8%  softirqs.CPU162.SCHED
     30973 ±  2%     -11.4%      27428 ±  3%  softirqs.CPU163.RCU
     13239 ±  8%     +73.0%      22902 ±  8%  softirqs.CPU163.SCHED
     30916 ±  3%     -10.8%      27588 ±  4%  softirqs.CPU164.RCU
     13376 ±  9%     +74.0%      23279 ±  8%  softirqs.CPU164.SCHED
     30922 ±  3%     -11.9%      27245 ±  3%  softirqs.CPU165.RCU
     13124 ±  8%     +72.9%      22694 ±  9%  softirqs.CPU165.SCHED
     30990 ±  2%     -11.2%      27508 ±  3%  softirqs.CPU166.RCU
     13235 ±  8%     +74.7%      23116 ±  8%  softirqs.CPU166.SCHED
     30940 ±  3%     -11.4%      27423 ±  3%  softirqs.CPU167.RCU
     13241 ±  8%     +68.6%      22319 ±  6%  softirqs.CPU167.SCHED
     31837 ±  7%     -15.2%      27013 ±  2%  softirqs.CPU168.RCU
     11975 ±  8%     +51.5%      18143 ± 19%  softirqs.CPU168.SCHED
     32905 ±  9%     -17.5%      27162 ±  2%  softirqs.CPU169.RCU
     12075 ± 10%     +46.4%      17674 ± 20%  softirqs.CPU169.SCHED
     32180 ±  4%     -11.8%      28372        softirqs.CPU17.RCU
     12701 ±  8%     +58.9%      20186 ± 16%  softirqs.CPU17.SCHED
     32128 ±  8%     -16.3%      26892 ±  3%  softirqs.CPU170.RCU
     12379 ± 11%     +43.9%      17817 ± 19%  softirqs.CPU170.SCHED
     31840 ±  6%     -15.0%      27057 ±  2%  softirqs.CPU171.RCU
     12030 ± 10%     +48.8%      17901 ± 19%  softirqs.CPU171.SCHED
     31612 ±  7%     -13.1%      27469 ±  4%  softirqs.CPU172.RCU
     12139 ± 10%     +48.1%      17974 ± 20%  softirqs.CPU172.SCHED
     31799 ±  7%     -14.4%      27217 ±  3%  softirqs.CPU173.RCU
     12137 ± 12%     +47.7%      17923 ± 19%  softirqs.CPU173.SCHED
     31879 ±  6%     -15.2%      27038 ±  3%  softirqs.CPU174.RCU
     12235 ± 12%     +47.0%      17985 ± 20%  softirqs.CPU174.SCHED
     32162 ±  6%     -14.4%      27533 ±  3%  softirqs.CPU175.RCU
     12298 ± 15%     +46.7%      18043 ± 18%  softirqs.CPU175.SCHED
     30774 ±  6%     -11.1%      27366 ±  2%  softirqs.CPU176.RCU
     12026 ± 11%     +48.0%      17803 ± 19%  softirqs.CPU176.SCHED
     30679 ±  4%     -11.5%      27156 ±  2%  softirqs.CPU177.RCU
     12176 ± 13%     +45.5%      17714 ± 20%  softirqs.CPU177.SCHED
     12140 ± 11%     +48.8%      18069 ± 19%  softirqs.CPU178.SCHED
     12211 ± 12%     +47.4%      18003 ± 19%  softirqs.CPU179.SCHED
     32382 ±  4%     -12.2%      28441        softirqs.CPU18.RCU
     12902 ±  6%     +57.4%      20303 ± 15%  softirqs.CPU18.SCHED
     31075 ±  5%     -11.9%      27369 ±  3%  softirqs.CPU180.RCU
     12165 ± 12%     +46.6%      17839 ± 20%  softirqs.CPU180.SCHED
     30964 ±  4%     -12.0%      27238 ±  2%  softirqs.CPU181.RCU
     12248 ± 12%     +47.8%      18098 ± 22%  softirqs.CPU181.SCHED
     31103 ±  4%     -11.8%      27425        softirqs.CPU182.RCU
     12424 ± 12%     +43.3%      17805 ± 19%  softirqs.CPU182.SCHED
     30938 ±  5%     -10.5%      27675 ±  2%  softirqs.CPU183.RCU
     12480 ± 16%     +42.5%      17782 ± 18%  softirqs.CPU183.SCHED
     12169 ± 12%     +47.7%      17972 ± 20%  softirqs.CPU184.SCHED
     30878 ±  5%     -11.6%      27312 ±  2%  softirqs.CPU185.RCU
     12060 ± 13%     +49.6%      18037 ± 20%  softirqs.CPU185.SCHED
     31185 ±  6%     -13.0%      27132 ±  4%  softirqs.CPU186.RCU
     12442 ± 11%     +44.3%      17958 ± 20%  softirqs.CPU186.SCHED
     30816 ±  6%      -9.9%      27753 ±  2%  softirqs.CPU187.RCU
     11967 ± 13%     +51.8%      18169 ± 19%  softirqs.CPU187.SCHED
     30741 ±  5%     -10.3%      27590        softirqs.CPU188.RCU
     12301 ± 11%     +47.0%      18077 ± 18%  softirqs.CPU188.SCHED
     30878 ±  5%     -11.1%      27437        softirqs.CPU189.RCU
     12403 ± 11%     +46.1%      18125 ± 19%  softirqs.CPU189.SCHED
     33549 ±  9%     -15.5%      28337        softirqs.CPU19.RCU
     13145 ±  8%     +56.0%      20502 ± 15%  softirqs.CPU19.SCHED
     12217 ± 12%     +45.2%      17734 ± 20%  softirqs.CPU190.SCHED
     31599 ±  4%     -12.2%      27731 ±  2%  softirqs.CPU191.RCU
     12201 ± 13%     +46.1%      17832 ± 20%  softirqs.CPU191.SCHED
     13107 ±  9%     +57.7%      20673 ± 15%  softirqs.CPU2.SCHED
     32171 ±  4%      -8.8%      29343 ±  4%  softirqs.CPU20.RCU
     13049 ±  7%     +57.4%      20544 ± 15%  softirqs.CPU20.SCHED
     31950 ±  4%     -12.1%      28099        softirqs.CPU21.RCU
     13103 ±  7%     +56.7%      20536 ± 16%  softirqs.CPU21.SCHED
    112050 ±  5%      +9.1%     122248        softirqs.CPU21.TIMER
     32375 ±  8%     -12.1%      28462        softirqs.CPU22.RCU
     13126 ±  8%     +57.0%      20615 ± 15%  softirqs.CPU22.SCHED
     32092 ±  4%     -11.6%      28371        softirqs.CPU23.RCU
     12677 ± 12%     +65.3%      20955 ± 17%  softirqs.CPU23.SCHED
     32554 ±  7%     -12.0%      28659        softirqs.CPU24.RCU
     12382 ± 10%     +80.6%      22358 ±  3%  softirqs.CPU24.SCHED
     32092 ±  3%     -13.9%      27628 ±  2%  softirqs.CPU25.RCU
     12271 ± 10%     +81.6%      22280 ±  5%  softirqs.CPU25.SCHED
     31747 ±  4%     -14.1%      27267 ±  4%  softirqs.CPU26.RCU
     12617 ±  7%     +76.2%      22233 ±  4%  softirqs.CPU26.SCHED
     32013 ±  3%     -13.6%      27673 ±  2%  softirqs.CPU27.RCU
     12501 ±  9%     +80.0%      22496 ±  3%  softirqs.CPU27.SCHED
     32129 ±  3%     -12.1%      28232 ±  2%  softirqs.CPU28.RCU
     12719 ±  9%     +76.8%      22482 ±  4%  softirqs.CPU28.SCHED
     32415 ±  3%     -14.2%      27797 ±  2%  softirqs.CPU29.RCU
     12529 ±  9%     +77.8%      22284 ±  3%  softirqs.CPU29.SCHED
     31703 ±  5%      -9.7%      28624        softirqs.CPU3.RCU
     13071 ±  7%     +56.5%      20460 ± 18%  softirqs.CPU3.SCHED
     32136 ±  3%     -13.5%      27802        softirqs.CPU30.RCU
     12408 ± 11%     +82.4%      22632 ±  3%  softirqs.CPU30.SCHED
     32766 ±  2%     -14.7%      27942 ±  2%  softirqs.CPU31.RCU
     12782 ± 10%     +69.8%      21700 ±  3%  softirqs.CPU31.SCHED
     32114 ±  2%     -13.2%      27885 ±  2%  softirqs.CPU32.RCU
     12565 ± 10%     +76.9%      22227 ±  3%  softirqs.CPU32.SCHED
     31908           -10.9%      28428 ±  5%  softirqs.CPU33.RCU
     12470 ± 11%     +81.9%      22679        softirqs.CPU33.SCHED
     31862 ±  2%     -12.2%      27972 ±  3%  softirqs.CPU34.RCU
     12441 ± 11%     +79.4%      22324 ±  3%  softirqs.CPU34.SCHED
     32334 ±  2%     -13.0%      28127 ±  2%  softirqs.CPU35.RCU
     12571 ± 10%     +78.0%      22380 ±  2%  softirqs.CPU35.SCHED
     31808 ±  3%     -11.3%      28213 ±  3%  softirqs.CPU36.RCU
     12634 ±  9%     +76.1%      22245 ±  2%  softirqs.CPU36.SCHED
     31475 ±  4%     -12.0%      27691 ±  2%  softirqs.CPU37.RCU
     12674 ±  9%     +75.3%      22217 ±  2%  softirqs.CPU37.SCHED
     31654           -11.4%      28031 ±  3%  softirqs.CPU38.RCU
     12380 ±  9%     +77.6%      21990 ±  2%  softirqs.CPU38.SCHED
     31846 ±  3%     -11.1%      28313 ±  3%  softirqs.CPU39.RCU
     12426 ±  9%     +80.6%      22446 ±  3%  softirqs.CPU39.SCHED
     12972 ±  6%     +53.7%      19940 ± 17%  softirqs.CPU4.SCHED
    112980 ±  5%      +7.3%     121215 ±  4%  softirqs.CPU4.TIMER
     31924 ±  2%     -12.5%      27919 ±  3%  softirqs.CPU40.RCU
     12607 ±  8%     +76.4%      22236 ±  4%  softirqs.CPU40.SCHED
     31979 ±  2%     -13.2%      27768 ±  2%  softirqs.CPU41.RCU
     12515 ±  9%     +83.2%      22925 ±  6%  softirqs.CPU41.SCHED
     31649 ±  2%     -11.4%      28056 ±  2%  softirqs.CPU42.RCU
     12409 ± 10%     +82.5%      22643 ±  2%  softirqs.CPU42.SCHED
     31892 ±  2%     -12.3%      27980 ±  3%  softirqs.CPU43.RCU
     12459 ±  9%     +77.6%      22123 ±  5%  softirqs.CPU43.SCHED
     31735 ±  2%     -11.7%      28017 ±  2%  softirqs.CPU44.RCU
     12510 ±  9%     +79.4%      22438 ±  2%  softirqs.CPU44.SCHED
     31322 ±  2%     -11.7%      27656 ±  3%  softirqs.CPU45.RCU
     12601 ±  9%     +73.3%      21837 ±  2%  softirqs.CPU45.SCHED
     31661 ±  3%     -11.6%      27979 ±  4%  softirqs.CPU46.RCU
     12508 ±  9%     +79.2%      22411 ±  3%  softirqs.CPU46.SCHED
     31532 ±  3%     -11.8%      27802 ±  2%  softirqs.CPU47.RCU
     12568 ±  8%     +78.5%      22440 ±  2%  softirqs.CPU47.SCHED
     32597 ±  3%     -13.9%      28075        softirqs.CPU48.RCU
     12838 ±  6%     +74.5%      22404 ±  6%  softirqs.CPU48.SCHED
     31583 ±  3%     -11.8%      27858        softirqs.CPU49.RCU
     12826 ±  6%     +74.7%      22407 ±  4%  softirqs.CPU49.SCHED
     32127 ±  7%     -11.5%      28427        softirqs.CPU5.RCU
     13560 ± 10%     +50.6%      20420 ± 15%  softirqs.CPU5.SCHED
     31596 ±  3%     -12.9%      27527        softirqs.CPU50.RCU
     12964 ±  7%     +73.7%      22515 ±  4%  softirqs.CPU50.SCHED
     31453 ±  3%     -10.5%      28160        softirqs.CPU51.RCU
     12739 ±  6%     +73.9%      22153 ±  7%  softirqs.CPU51.SCHED
     12804 ±  6%     +75.9%      22528 ±  6%  softirqs.CPU52.SCHED
     31450 ±  2%     -10.4%      28167        softirqs.CPU53.RCU
     12942 ±  6%     +75.0%      22651 ±  7%  softirqs.CPU53.SCHED
     31441 ±  3%     -10.1%      28251        softirqs.CPU54.RCU
     13105 ±  3%     +71.3%      22453 ±  7%  softirqs.CPU54.SCHED
     31603 ±  3%     -11.0%      28132        softirqs.CPU55.RCU
     12777 ±  7%     +77.2%      22637 ±  8%  softirqs.CPU55.SCHED
     31555 ±  3%     -11.1%      28066        softirqs.CPU56.RCU
     12777 ±  6%     +75.1%      22376 ±  6%  softirqs.CPU56.SCHED
     31436 ±  3%     -12.0%      27671 ±  3%  softirqs.CPU57.RCU
     12781 ±  6%     +77.2%      22649 ±  6%  softirqs.CPU57.SCHED
     31503 ±  3%     -10.7%      28139        softirqs.CPU58.RCU
     12954 ±  6%     +75.3%      22709 ±  7%  softirqs.CPU58.SCHED
     31839 ±  3%     -10.9%      28371 ±  2%  softirqs.CPU59.RCU
     13236 ±  7%     +71.1%      22653 ±  6%  softirqs.CPU59.SCHED
     13223 ±  8%     +52.4%      20148 ± 16%  softirqs.CPU6.SCHED
     32616           -13.7%      28163        softirqs.CPU60.RCU
     13139           +73.8%      22841 ±  7%  softirqs.CPU60.SCHED
     31250 ±  3%     -11.5%      27660 ±  3%  softirqs.CPU61.RCU
     12835 ±  6%     +76.8%      22694 ±  7%  softirqs.CPU61.SCHED
     31480 ±  3%     -11.3%      27922        softirqs.CPU62.RCU
     12962 ±  6%     +72.4%      22345 ±  6%  softirqs.CPU62.SCHED
     31567 ±  3%     -10.2%      28342 ±  2%  softirqs.CPU63.RCU
     12909 ±  7%     +74.7%      22556 ±  6%  softirqs.CPU63.SCHED
     30655 ±  5%     -14.7%      26163        softirqs.CPU64.RCU
     12793 ±  6%     +75.5%      22452 ±  7%  softirqs.CPU64.SCHED
     30643 ±  5%     -13.8%      26417        softirqs.CPU65.RCU
     12839 ±  7%     +73.3%      22251 ±  7%  softirqs.CPU65.SCHED
     30720 ±  4%     -14.6%      26249        softirqs.CPU66.RCU
     12754 ±  6%     +75.7%      22415 ±  7%  softirqs.CPU66.SCHED
     30650 ±  4%     -14.2%      26294        softirqs.CPU67.RCU
     12760 ±  6%     +77.6%      22659 ±  8%  softirqs.CPU67.SCHED
     30537 ±  5%     -13.3%      26480 ±  2%  softirqs.CPU68.RCU
     12927 ±  7%     +75.6%      22699 ±  7%  softirqs.CPU68.SCHED
     30702 ±  5%     -14.5%      26253 ±  2%  softirqs.CPU69.RCU
     12814 ±  7%     +79.8%      23040 ±  9%  softirqs.CPU69.SCHED
     12743 ± 10%     +61.6%      20591 ± 13%  softirqs.CPU7.SCHED
     30578 ±  4%     -13.8%      26345        softirqs.CPU70.RCU
     12985 ±  8%     +74.9%      22711 ±  8%  softirqs.CPU70.SCHED
     30596 ±  4%     -14.2%      26259 ±  2%  softirqs.CPU71.RCU
     12903 ±  7%     +75.1%      22588 ±  7%  softirqs.CPU71.SCHED
     31963 ± 10%     -17.8%      26279        softirqs.CPU72.RCU
     12353 ±  6%     +52.5%      18835 ± 10%  softirqs.CPU72.SCHED
     32117 ± 12%     -18.9%      26039        softirqs.CPU73.RCU
     12342 ±  9%     +45.4%      17944 ± 16%  softirqs.CPU73.SCHED
     32473 ± 12%     -19.9%      26000 ±  2%  softirqs.CPU74.RCU
     12308 ±  8%     +47.5%      18160 ± 16%  softirqs.CPU74.SCHED
     31632 ± 10%     -16.9%      26275 ±  2%  softirqs.CPU75.RCU
     12134 ±  9%     +46.8%      17808 ± 17%  softirqs.CPU75.SCHED
     31589 ± 10%     -17.2%      26141 ±  2%  softirqs.CPU76.RCU
     11922 ± 11%     +65.2%      19694 ± 33%  softirqs.CPU76.SCHED
     31630 ± 11%     -17.2%      26191 ±  2%  softirqs.CPU77.RCU
     12062 ±  9%     +48.6%      17923 ± 15%  softirqs.CPU77.SCHED
     32390 ±  9%     -19.5%      26088 ±  2%  softirqs.CPU78.RCU
     12168 ±  9%     +47.0%      17883 ± 16%  softirqs.CPU78.SCHED
     31792 ± 10%     -16.7%      26488 ±  4%  softirqs.CPU79.RCU
     12295 ± 11%     +44.8%      17807 ± 17%  softirqs.CPU79.SCHED
     31541 ±  5%     -10.3%      28297        softirqs.CPU8.RCU
     13076 ±  7%     +58.5%      20721 ± 15%  softirqs.CPU8.SCHED
     12180 ± 10%     +45.3%      17693 ± 17%  softirqs.CPU80.SCHED
     31450 ±  7%     -12.7%      27447 ±  2%  softirqs.CPU81.RCU
     12339 ± 12%     +42.2%      17544 ± 17%  softirqs.CPU81.SCHED
     12207 ±  9%     +45.6%      17777 ± 18%  softirqs.CPU82.SCHED
     12064 ±  9%     +48.7%      17943 ± 16%  softirqs.CPU83.SCHED
     12084 ±  8%     +46.6%      17712 ± 18%  softirqs.CPU84.SCHED
     31146 ±  4%     -10.9%      27756 ±  2%  softirqs.CPU85.RCU
     12156 ±  8%     +47.3%      17911 ± 16%  softirqs.CPU85.SCHED
     31536 ±  4%     -12.1%      27718 ±  3%  softirqs.CPU86.RCU
     12371 ±  8%     +42.9%      17673 ± 18%  softirqs.CPU86.SCHED
     12263 ± 10%     +44.0%      17662 ± 18%  softirqs.CPU87.SCHED
     12216 ±  9%     +45.7%      17804 ± 18%  softirqs.CPU88.SCHED
     31424 ±  5%     -11.8%      27724 ±  2%  softirqs.CPU89.RCU
     12379 ±  8%     +44.2%      17853 ± 16%  softirqs.CPU89.SCHED
     31423 ±  5%     -10.3%      28191        softirqs.CPU9.RCU
     13188 ±  5%     +58.2%      20864 ± 14%  softirqs.CPU9.SCHED
    113341 ±  4%      +8.2%     122664        softirqs.CPU9.TIMER
     31525 ±  5%     -12.5%      27585 ±  4%  softirqs.CPU90.RCU
     12345 ±  8%     +45.0%      17895 ± 20%  softirqs.CPU90.SCHED
     11971 ±  8%     +49.7%      17922 ± 18%  softirqs.CPU91.SCHED
     12212 ±  9%     +47.2%      17979 ± 16%  softirqs.CPU92.SCHED
     12305 ± 10%     +46.2%      17988 ± 18%  softirqs.CPU93.SCHED
     31246 ±  5%     -11.2%      27734 ±  2%  softirqs.CPU94.RCU
     12173 ±  9%     +48.9%      18124 ± 19%  softirqs.CPU94.SCHED
     31258 ±  5%     -11.0%      27824 ±  2%  softirqs.CPU95.RCU
     12187 ± 11%     +47.6%      17988 ± 19%  softirqs.CPU95.SCHED
     31677 ±  4%     -15.7%      26719 ±  2%  softirqs.CPU96.RCU
     13001 ±  7%     +57.1%      20420 ± 15%  softirqs.CPU96.SCHED
     31379 ±  6%     -11.8%      27678 ±  2%  softirqs.CPU97.RCU
     12869 ±  9%     +62.5%      20909 ± 14%  softirqs.CPU97.SCHED
    110944 ±  5%     +10.3%     122392        softirqs.CPU97.TIMER
     33271 ±  7%     -16.0%      27945        softirqs.CPU98.RCU
     13545 ±  7%     +56.5%      21193 ± 15%  softirqs.CPU98.SCHED
     31522 ±  5%     -10.9%      28072        softirqs.CPU99.RCU
     13760 ±  7%     +50.1%      20650 ± 16%  softirqs.CPU99.SCHED
   6052777 ±  4%     -12.3%    5310987        softirqs.RCU
   2454586           +63.5%    4012658 ±  2%  softirqs.SCHED
   5164839 ±  4%     -44.0%    2891677 ±  4%  interrupts.CAL:Function_call_interrupts
     23211 ±  5%     -49.8%      11662 ± 14%  interrupts.CPU0.CAL:Function_call_interrupts
     19793 ±  4%     -56.4%       8622 ± 20%  interrupts.CPU0.TLB:TLB_shootdowns
     25756 ±  6%     -48.9%      13159 ±  6%  interrupts.CPU1.CAL:Function_call_interrupts
      6216 ±  6%     -29.9%       4356 ± 21%  interrupts.CPU1.NMI:Non-maskable_interrupts
      6216 ±  6%     -29.9%       4356 ± 21%  interrupts.CPU1.PMI:Performance_monitoring_interrupts
     22359 ±  5%     -54.3%      10221 ±  8%  interrupts.CPU1.TLB:TLB_shootdowns
     25979 ±  5%     -44.8%      14345 ± 10%  interrupts.CPU10.CAL:Function_call_interrupts
      6067 ±  5%     -29.7%       4265 ± 36%  interrupts.CPU10.NMI:Non-maskable_interrupts
      6067 ±  5%     -29.7%       4265 ± 36%  interrupts.CPU10.PMI:Performance_monitoring_interrupts
     22666 ±  4%     -49.6%      11419 ± 11%  interrupts.CPU10.TLB:TLB_shootdowns
     26575 ±  9%     -45.8%      14399 ±  6%  interrupts.CPU100.CAL:Function_call_interrupts
     23233 ± 11%     -50.7%      11457 ±  8%  interrupts.CPU100.TLB:TLB_shootdowns
     27400 ±  6%     -45.7%      14890 ±  8%  interrupts.CPU101.CAL:Function_call_interrupts
     24034 ±  7%     -50.1%      11999 ± 10%  interrupts.CPU101.TLB:TLB_shootdowns
     27415 ±  9%     -44.7%      15151 ±  6%  interrupts.CPU102.CAL:Function_call_interrupts
      6287 ±  4%     -26.4%       4625 ± 16%  interrupts.CPU102.NMI:Non-maskable_interrupts
      6287 ±  4%     -26.4%       4625 ± 16%  interrupts.CPU102.PMI:Performance_monitoring_interrupts
     24027 ± 11%     -49.2%      12215 ±  8%  interrupts.CPU102.TLB:TLB_shootdowns
     27495 ±  7%     -47.1%      14548 ±  7%  interrupts.CPU103.CAL:Function_call_interrupts
     24192 ±  6%     -52.3%      11541 ±  9%  interrupts.CPU103.TLB:TLB_shootdowns
     27461 ±  7%     -44.9%      15142 ±  9%  interrupts.CPU104.CAL:Function_call_interrupts
      6105 ±  6%     -43.1%       3471 ± 39%  interrupts.CPU104.NMI:Non-maskable_interrupts
      6105 ±  6%     -43.1%       3471 ± 39%  interrupts.CPU104.PMI:Performance_monitoring_interrupts
     24138 ±  6%     -49.7%      12153 ± 11%  interrupts.CPU104.TLB:TLB_shootdowns
     26900 ±  7%     -45.0%      14796 ±  7%  interrupts.CPU105.CAL:Function_call_interrupts
     23655 ±  7%     -50.1%      11793 ±  8%  interrupts.CPU105.TLB:TLB_shootdowns
     26032 ±  9%     -42.6%      14934 ± 10%  interrupts.CPU106.CAL:Function_call_interrupts
      6015 ±  6%     -33.6%       3994 ± 18%  interrupts.CPU106.NMI:Non-maskable_interrupts
      6015 ±  6%     -33.6%       3994 ± 18%  interrupts.CPU106.PMI:Performance_monitoring_interrupts
     22884 ±  8%     -47.4%      12037 ± 12%  interrupts.CPU106.TLB:TLB_shootdowns
     27375 ±  5%     -43.6%      15441 ±  9%  interrupts.CPU107.CAL:Function_call_interrupts
     24034 ±  6%     -48.1%      12472 ± 12%  interrupts.CPU107.TLB:TLB_shootdowns
     27473 ±  6%     -45.5%      14969 ±  8%  interrupts.CPU108.CAL:Function_call_interrupts
     24119 ±  6%     -50.4%      11962 ± 10%  interrupts.CPU108.TLB:TLB_shootdowns
     27681 ±  9%     -43.6%      15612 ±  8%  interrupts.CPU109.CAL:Function_call_interrupts
     24298 ±  9%     -48.0%      12638 ± 10%  interrupts.CPU109.TLB:TLB_shootdowns
     27016 ±  7%     -44.6%      14956 ± 10%  interrupts.CPU11.CAL:Function_call_interrupts
      6075 ±  6%     -26.1%       4487 ± 22%  interrupts.CPU11.NMI:Non-maskable_interrupts
      6075 ±  6%     -26.1%       4487 ± 22%  interrupts.CPU11.PMI:Performance_monitoring_interrupts
     23697 ±  8%     -49.4%      12001 ± 13%  interrupts.CPU11.TLB:TLB_shootdowns
     27375 ±  6%     -44.3%      15248 ±  9%  interrupts.CPU110.CAL:Function_call_interrupts
      6121 ±  5%     -28.7%       4364 ± 34%  interrupts.CPU110.NMI:Non-maskable_interrupts
      6121 ±  5%     -28.7%       4364 ± 34%  interrupts.CPU110.PMI:Performance_monitoring_interrupts
     24007 ±  7%     -48.9%      12263 ± 12%  interrupts.CPU110.TLB:TLB_shootdowns
     27096 ±  5%     -45.0%      14899 ±  9%  interrupts.CPU111.CAL:Function_call_interrupts
     23791 ±  5%     -49.9%      11920 ± 12%  interrupts.CPU111.TLB:TLB_shootdowns
     27578 ±  7%     -46.7%      14694 ± 10%  interrupts.CPU112.CAL:Function_call_interrupts
      6310 ±  7%     -29.2%       4468 ± 17%  interrupts.CPU112.NMI:Non-maskable_interrupts
      6310 ±  7%     -29.2%       4468 ± 17%  interrupts.CPU112.PMI:Performance_monitoring_interrupts
     24277 ±  7%     -51.7%      11726 ± 12%  interrupts.CPU112.TLB:TLB_shootdowns
     28568 ±  4%     -48.0%      14870 ±  7%  interrupts.CPU113.CAL:Function_call_interrupts
      6171 ±  7%     -43.9%       3464 ± 39%  interrupts.CPU113.NMI:Non-maskable_interrupts
      6171 ±  7%     -43.9%       3464 ± 39%  interrupts.CPU113.PMI:Performance_monitoring_interrupts
     25339 ±  5%     -53.0%      11908 ±  9%  interrupts.CPU113.TLB:TLB_shootdowns
     27869 ±  7%     -47.3%      14690 ± 11%  interrupts.CPU114.CAL:Function_call_interrupts
     24553 ±  6%     -52.5%      11664 ± 14%  interrupts.CPU114.TLB:TLB_shootdowns
     27168 ±  5%     -44.2%      15162 ±  9%  interrupts.CPU115.CAL:Function_call_interrupts
     23784 ±  5%     -48.7%      12203 ± 10%  interrupts.CPU115.TLB:TLB_shootdowns
     27674 ±  7%     -44.4%      15395 ± 10%  interrupts.CPU116.CAL:Function_call_interrupts
     24359 ±  7%     -49.2%      12386 ± 13%  interrupts.CPU116.TLB:TLB_shootdowns
     25881 ± 15%     -41.8%      15054 ± 11%  interrupts.CPU117.CAL:Function_call_interrupts
     22516 ± 18%     -46.2%      12105 ± 13%  interrupts.CPU117.TLB:TLB_shootdowns
     27172 ±  6%     -43.8%      15279 ±  7%  interrupts.CPU118.CAL:Function_call_interrupts
     23916 ±  6%     -48.6%      12298 ±  9%  interrupts.CPU118.TLB:TLB_shootdowns
     27108 ±  7%     -46.3%      14548 ± 13%  interrupts.CPU119.CAL:Function_call_interrupts
      6164 ±  4%     -28.8%       4388 ± 21%  interrupts.CPU119.NMI:Non-maskable_interrupts
      6164 ±  4%     -28.8%       4388 ± 21%  interrupts.CPU119.PMI:Performance_monitoring_interrupts
     23830 ±  7%     -51.4%      11571 ± 17%  interrupts.CPU119.TLB:TLB_shootdowns
     26907 ±  5%     -43.4%      15217 ±  8%  interrupts.CPU12.CAL:Function_call_interrupts
      6205 ±  5%     -32.9%       4164 ± 34%  interrupts.CPU12.NMI:Non-maskable_interrupts
      6205 ±  5%     -32.9%       4164 ± 34%  interrupts.CPU12.PMI:Performance_monitoring_interrupts
     23637 ±  6%     -48.1%      12262 ± 11%  interrupts.CPU12.TLB:TLB_shootdowns
     25698 ±  9%     -36.6%      16298 ±  5%  interrupts.CPU120.CAL:Function_call_interrupts
     22429 ±  9%     -40.2%      13422 ±  6%  interrupts.CPU120.TLB:TLB_shootdowns
     25824 ±  9%     -38.1%      15984 ±  2%  interrupts.CPU121.CAL:Function_call_interrupts
      6254 ±  4%     -34.0%       4127 ± 24%  interrupts.CPU121.NMI:Non-maskable_interrupts
      6254 ±  4%     -34.0%       4127 ± 24%  interrupts.CPU121.PMI:Performance_monitoring_interrupts
     22548 ±  8%     -42.3%      13012 ±  3%  interrupts.CPU121.TLB:TLB_shootdowns
     25386 ± 12%     -36.5%      16116 ±  4%  interrupts.CPU122.CAL:Function_call_interrupts
     21996 ± 11%     -39.5%      13312 ±  4%  interrupts.CPU122.TLB:TLB_shootdowns
     25944 ± 11%     -37.5%      16217 ±  5%  interrupts.CPU123.CAL:Function_call_interrupts
      6261 ±  3%     -49.5%       3161 ± 42%  interrupts.CPU123.NMI:Non-maskable_interrupts
      6261 ±  3%     -49.5%       3161 ± 42%  interrupts.CPU123.PMI:Performance_monitoring_interrupts
     22614 ± 10%     -41.4%      13258 ±  7%  interrupts.CPU123.TLB:TLB_shootdowns
     26203 ±  9%     -41.2%      15417 ±  5%  interrupts.CPU124.CAL:Function_call_interrupts
     22819 ±  9%     -45.7%      12387 ±  6%  interrupts.CPU124.TLB:TLB_shootdowns
     26658 ±  9%     -37.8%      16571 ±  3%  interrupts.CPU125.CAL:Function_call_interrupts
      6317 ±  4%     -38.0%       3914 ± 35%  interrupts.CPU125.NMI:Non-maskable_interrupts
      6317 ±  4%     -38.0%       3914 ± 35%  interrupts.CPU125.PMI:Performance_monitoring_interrupts
     23411 ±  9%     -42.1%      13551 ±  4%  interrupts.CPU125.TLB:TLB_shootdowns
     26357 ± 11%     -39.0%      16090        interrupts.CPU126.CAL:Function_call_interrupts
     22964 ± 11%     -42.7%      13155 ±  2%  interrupts.CPU126.TLB:TLB_shootdowns
     26279 ± 10%     -37.8%      16357 ±  2%  interrupts.CPU127.CAL:Function_call_interrupts
      6414 ±  4%     -31.6%       4389 ± 19%  interrupts.CPU127.NMI:Non-maskable_interrupts
      6414 ±  4%     -31.6%       4389 ± 19%  interrupts.CPU127.PMI:Performance_monitoring_interrupts
     22928 ±  9%     -42.0%      13304 ±  3%  interrupts.CPU127.TLB:TLB_shootdowns
     26403 ±  7%     -38.7%      16195 ±  5%  interrupts.CPU128.CAL:Function_call_interrupts
      6320 ±  3%     -48.7%       3241 ± 41%  interrupts.CPU128.NMI:Non-maskable_interrupts
      6320 ±  3%     -48.7%       3241 ± 41%  interrupts.CPU128.PMI:Performance_monitoring_interrupts
     23017 ±  6%     -42.9%      13151 ±  7%  interrupts.CPU128.TLB:TLB_shootdowns
     26062 ± 11%     -38.4%      16041 ±  2%  interrupts.CPU129.CAL:Function_call_interrupts
     22712 ± 11%     -42.6%      13045 ±  2%  interrupts.CPU129.TLB:TLB_shootdowns
     27023 ±  8%     -46.6%      14438 ±  9%  interrupts.CPU13.CAL:Function_call_interrupts
     23836 ±  9%     -51.8%      11478 ± 11%  interrupts.CPU13.TLB:TLB_shootdowns
     25782 ± 11%     -37.9%      16023 ±  4%  interrupts.CPU130.CAL:Function_call_interrupts
     22383 ± 11%     -41.5%      13102 ±  6%  interrupts.CPU130.TLB:TLB_shootdowns
     25769 ± 10%     -37.1%      16197 ±  3%  interrupts.CPU131.CAL:Function_call_interrupts
     22457 ±  9%     -41.1%      13226 ±  3%  interrupts.CPU131.TLB:TLB_shootdowns
     25939 ±  8%     -36.8%      16394 ±  3%  interrupts.CPU132.CAL:Function_call_interrupts
      6245 ±  3%     -39.8%       3761 ± 35%  interrupts.CPU132.NMI:Non-maskable_interrupts
      6245 ±  3%     -39.8%       3761 ± 35%  interrupts.CPU132.PMI:Performance_monitoring_interrupts
     22535 ±  8%     -40.5%      13418 ±  5%  interrupts.CPU132.TLB:TLB_shootdowns
     24855 ± 17%     -37.7%      15479 ±  8%  interrupts.CPU133.CAL:Function_call_interrupts
     21434 ± 17%     -41.9%      12446 ± 10%  interrupts.CPU133.TLB:TLB_shootdowns
     26463 ± 12%     -39.6%      15994 ±  5%  interrupts.CPU134.CAL:Function_call_interrupts
      6319 ±  2%     -31.5%       4328 ± 18%  interrupts.CPU134.NMI:Non-maskable_interrupts
      6319 ±  2%     -31.5%       4328 ± 18%  interrupts.CPU134.PMI:Performance_monitoring_interrupts
     23156 ± 11%     -44.1%      12953 ±  7%  interrupts.CPU134.TLB:TLB_shootdowns
     25511 ± 10%     -38.3%      15740 ±  5%  interrupts.CPU135.CAL:Function_call_interrupts
     22206 ±  9%     -42.3%      12803 ±  6%  interrupts.CPU135.TLB:TLB_shootdowns
     26114 ± 12%     -37.6%      16304 ±  2%  interrupts.CPU136.CAL:Function_call_interrupts
      6348 ±  4%     -35.5%       4096 ± 17%  interrupts.CPU136.NMI:Non-maskable_interrupts
      6348 ±  4%     -35.5%       4096 ± 17%  interrupts.CPU136.PMI:Performance_monitoring_interrupts
     22716 ± 12%     -41.4%      13306 ±  3%  interrupts.CPU136.TLB:TLB_shootdowns
     25715 ± 11%     -38.7%      15752 ±  4%  interrupts.CPU137.CAL:Function_call_interrupts
      6314 ±  3%     -43.2%       3586 ± 38%  interrupts.CPU137.NMI:Non-maskable_interrupts
      6314 ±  3%     -43.2%       3586 ± 38%  interrupts.CPU137.PMI:Performance_monitoring_interrupts
     22325 ± 10%     -43.0%      12718 ±  5%  interrupts.CPU137.TLB:TLB_shootdowns
     25832 ± 14%     -40.7%      15324 ± 10%  interrupts.CPU138.CAL:Function_call_interrupts
      6262 ±  5%     -57.8%       2644 ± 36%  interrupts.CPU138.NMI:Non-maskable_interrupts
      6262 ±  5%     -57.8%       2644 ± 36%  interrupts.CPU138.PMI:Performance_monitoring_interrupts
     22456 ± 14%     -45.4%      12269 ± 13%  interrupts.CPU138.TLB:TLB_shootdowns
     26008 ± 10%     -39.9%      15625 ±  7%  interrupts.CPU139.CAL:Function_call_interrupts
     22688 ± 10%     -44.4%      12617 ±  9%  interrupts.CPU139.TLB:TLB_shootdowns
     27079 ±  7%     -47.2%      14294 ±  6%  interrupts.CPU14.CAL:Function_call_interrupts
     23830 ±  7%     -52.5%      11308 ±  7%  interrupts.CPU14.TLB:TLB_shootdowns
     25986 ±  9%     -38.9%      15865 ±  5%  interrupts.CPU140.CAL:Function_call_interrupts
      6388 ±  4%     -41.6%       3733 ± 34%  interrupts.CPU140.NMI:Non-maskable_interrupts
      6388 ±  4%     -41.6%       3733 ± 34%  interrupts.CPU140.PMI:Performance_monitoring_interrupts
     22603 ±  9%     -43.3%      12818 ±  6%  interrupts.CPU140.TLB:TLB_shootdowns
     25088 ± 17%     -37.9%      15590 ±  5%  interrupts.CPU141.CAL:Function_call_interrupts
      6327 ±  4%     -34.5%       4144 ± 34%  interrupts.CPU141.NMI:Non-maskable_interrupts
      6327 ±  4%     -34.5%       4144 ± 34%  interrupts.CPU141.PMI:Performance_monitoring_interrupts
     21633 ± 17%     -41.9%      12562 ±  6%  interrupts.CPU141.TLB:TLB_shootdowns
     25905 ± 15%     -40.1%      15514 ±  8%  interrupts.CPU142.CAL:Function_call_interrupts
     22513 ± 16%     -44.6%      12479 ± 10%  interrupts.CPU142.TLB:TLB_shootdowns
     26096 ± 12%     -42.5%      15005 ±  5%  interrupts.CPU143.CAL:Function_call_interrupts
     22777 ± 11%     -47.1%      12059 ±  6%  interrupts.CPU143.TLB:TLB_shootdowns
     27603 ±  3%     -46.2%      14854 ±  8%  interrupts.CPU144.CAL:Function_call_interrupts
     24314 ±  5%     -51.0%      11902 ± 10%  interrupts.CPU144.TLB:TLB_shootdowns
     28108 ±  3%     -45.8%      15236 ±  6%  interrupts.CPU145.CAL:Function_call_interrupts
     24783 ±  5%     -50.8%      12200 ±  8%  interrupts.CPU145.TLB:TLB_shootdowns
     27705 ±  2%     -48.2%      14342 ±  7%  interrupts.CPU146.CAL:Function_call_interrupts
     24528 ±  4%     -54.1%      11263 ± 10%  interrupts.CPU146.TLB:TLB_shootdowns
     27894           -44.5%      15469 ±  5%  interrupts.CPU147.CAL:Function_call_interrupts
      6127 ±  7%     -37.3%       3843 ± 20%  interrupts.CPU147.NMI:Non-maskable_interrupts
      6127 ±  7%     -37.3%       3843 ± 20%  interrupts.CPU147.PMI:Performance_monitoring_interrupts
     24626 ±  4%     -49.5%      12436 ±  6%  interrupts.CPU147.TLB:TLB_shootdowns
     28319 ±  4%     -44.3%      15773 ±  7%  interrupts.CPU148.CAL:Function_call_interrupts
     25004 ±  5%     -48.9%      12776 ±  9%  interrupts.CPU148.TLB:TLB_shootdowns
     27991 ±  2%     -44.4%      15576 ±  3%  interrupts.CPU149.CAL:Function_call_interrupts
      6236 ±  3%     -30.5%       4336 ± 24%  interrupts.CPU149.NMI:Non-maskable_interrupts
      6236 ±  3%     -30.5%       4336 ± 24%  interrupts.CPU149.PMI:Performance_monitoring_interrupts
     24647 ±  3%     -49.2%      12518 ±  4%  interrupts.CPU149.TLB:TLB_shootdowns
     26702 ±  8%     -43.9%      14975 ±  9%  interrupts.CPU15.CAL:Function_call_interrupts
      6223 ±  4%     -26.3%       4587 ± 17%  interrupts.CPU15.NMI:Non-maskable_interrupts
      6223 ±  4%     -26.3%       4587 ± 17%  interrupts.CPU15.PMI:Performance_monitoring_interrupts
     23511 ±  9%     -48.8%      12038 ± 11%  interrupts.CPU15.TLB:TLB_shootdowns
     27200 ±  4%     -42.7%      15580 ±  7%  interrupts.CPU150.CAL:Function_call_interrupts
     23912 ±  4%     -47.7%      12512 ± 10%  interrupts.CPU150.TLB:TLB_shootdowns
     28217 ±  3%     -45.3%      15433 ±  8%  interrupts.CPU151.CAL:Function_call_interrupts
      6289 ±  2%     -34.3%       4132 ± 29%  interrupts.CPU151.NMI:Non-maskable_interrupts
      6289 ±  2%     -34.3%       4132 ± 29%  interrupts.CPU151.PMI:Performance_monitoring_interrupts
     24912 ±  3%     -50.3%      12379 ± 11%  interrupts.CPU151.TLB:TLB_shootdowns
     27892 ±  4%     -43.5%      15747 ±  5%  interrupts.CPU152.CAL:Function_call_interrupts
     24655 ±  5%     -48.4%      12730 ±  7%  interrupts.CPU152.TLB:TLB_shootdowns
     27985 ±  3%     -43.6%      15781 ±  3%  interrupts.CPU153.CAL:Function_call_interrupts
     24700 ±  5%     -48.2%      12785 ±  4%  interrupts.CPU153.TLB:TLB_shootdowns
     28258           -45.4%      15424 ±  6%  interrupts.CPU154.CAL:Function_call_interrupts
     24984 ±  2%     -50.4%      12395 ±  9%  interrupts.CPU154.TLB:TLB_shootdowns
     28729 ±  4%     -46.4%      15412 ±  5%  interrupts.CPU155.CAL:Function_call_interrupts
     25466 ±  3%     -51.5%      12348 ±  7%  interrupts.CPU155.TLB:TLB_shootdowns
     28471 ±  2%     -46.4%      15270 ±  8%  interrupts.CPU156.CAL:Function_call_interrupts
     25189 ±  4%     -51.5%      12218 ± 10%  interrupts.CPU156.TLB:TLB_shootdowns
     28339 ±  2%     -46.9%      15043 ±  2%  interrupts.CPU157.CAL:Function_call_interrupts
     25019           -51.6%      12097        interrupts.CPU157.TLB:TLB_shootdowns
     27386 ±  3%     -42.3%      15802 ±  3%  interrupts.CPU158.CAL:Function_call_interrupts
     24077 ±  6%     -47.1%      12748 ±  3%  interrupts.CPU158.TLB:TLB_shootdowns
     28242 ±  3%     -43.6%      15939 ±  4%  interrupts.CPU159.CAL:Function_call_interrupts
      6371 ±  2%     -31.3%       4379 ± 22%  interrupts.CPU159.NMI:Non-maskable_interrupts
      6371 ±  2%     -31.3%       4379 ± 22%  interrupts.CPU159.PMI:Performance_monitoring_interrupts
     24953 ±  5%     -48.4%      12885 ±  6%  interrupts.CPU159.TLB:TLB_shootdowns
     27056 ±  7%     -46.3%      14540 ±  7%  interrupts.CPU16.CAL:Function_call_interrupts
     23823 ±  7%     -51.5%      11562 ± 10%  interrupts.CPU16.TLB:TLB_shootdowns
     28362           -43.8%      15926 ±  6%  interrupts.CPU160.CAL:Function_call_interrupts
      6236 ±  4%     -39.2%       3788 ± 38%  interrupts.CPU160.NMI:Non-maskable_interrupts
      6236 ±  4%     -39.2%       3788 ± 38%  interrupts.CPU160.PMI:Performance_monitoring_interrupts
     25119 ±  4%     -48.9%      12826 ±  8%  interrupts.CPU160.TLB:TLB_shootdowns
     28858 ±  3%     -45.7%      15668 ±  9%  interrupts.CPU161.CAL:Function_call_interrupts
      6292 ±  3%     -43.9%       3530 ± 21%  interrupts.CPU161.NMI:Non-maskable_interrupts
      6292 ±  3%     -43.9%       3530 ± 21%  interrupts.CPU161.PMI:Performance_monitoring_interrupts
     25535 ±  4%     -50.5%      12638 ± 11%  interrupts.CPU161.TLB:TLB_shootdowns
     28903           -47.3%      15233 ± 11%  interrupts.CPU162.CAL:Function_call_interrupts
      6140 ±  6%     -41.1%       3616 ± 16%  interrupts.CPU162.NMI:Non-maskable_interrupts
      6140 ±  6%     -41.1%       3616 ± 16%  interrupts.CPU162.PMI:Performance_monitoring_interrupts
     25603 ±  3%     -52.5%      12159 ± 13%  interrupts.CPU162.TLB:TLB_shootdowns
     28874 ±  2%     -48.4%      14894 ±  7%  interrupts.CPU163.CAL:Function_call_interrupts
     25490 ±  3%     -53.5%      11857 ±  9%  interrupts.CPU163.TLB:TLB_shootdowns
     29304 ±  3%     -48.4%      15117 ±  9%  interrupts.CPU164.CAL:Function_call_interrupts
     25985 ±  5%     -53.5%      12092 ± 12%  interrupts.CPU164.TLB:TLB_shootdowns
     28915 ±  2%     -49.4%      14641 ±  7%  interrupts.CPU165.CAL:Function_call_interrupts
      6260 ±  3%     -37.2%       3933 ± 36%  interrupts.CPU165.NMI:Non-maskable_interrupts
      6260 ±  3%     -37.2%       3933 ± 36%  interrupts.CPU165.PMI:Performance_monitoring_interrupts
     25555 ±  4%     -54.8%      11548 ±  9%  interrupts.CPU165.TLB:TLB_shootdowns
     28013 ±  3%     -47.5%      14719 ±  7%  interrupts.CPU166.CAL:Function_call_interrupts
      6299           -38.1%       3896 ± 37%  interrupts.CPU166.NMI:Non-maskable_interrupts
      6299           -38.1%       3896 ± 37%  interrupts.CPU166.PMI:Performance_monitoring_interrupts
     24656 ±  7%     -52.4%      11733 ± 10%  interrupts.CPU166.TLB:TLB_shootdowns
     29233 ±  5%     -47.6%      15321 ±  6%  interrupts.CPU167.CAL:Function_call_interrupts
      6154 ±  4%     -27.6%       4454 ± 14%  interrupts.CPU167.NMI:Non-maskable_interrupts
      6154 ±  4%     -27.6%       4454 ± 14%  interrupts.CPU167.PMI:Performance_monitoring_interrupts
     25980 ±  7%     -52.6%      12311 ±  7%  interrupts.CPU167.TLB:TLB_shootdowns
     26911 ± 10%     -45.7%      14624 ± 10%  interrupts.CPU168.CAL:Function_call_interrupts
      6336 ±  5%     -32.3%       4287 ± 27%  interrupts.CPU168.NMI:Non-maskable_interrupts
      6336 ±  5%     -32.3%       4287 ± 27%  interrupts.CPU168.PMI:Performance_monitoring_interrupts
     23567 ± 10%     -50.9%      11568 ± 13%  interrupts.CPU168.TLB:TLB_shootdowns
     27296 ±  9%     -46.2%      14687 ± 11%  interrupts.CPU169.CAL:Function_call_interrupts
      6293 ±  5%     -41.3%       3694 ± 36%  interrupts.CPU169.NMI:Non-maskable_interrupts
      6293 ±  5%     -41.3%       3694 ± 36%  interrupts.CPU169.PMI:Performance_monitoring_interrupts
     24002 ±  9%     -51.4%      11676 ± 15%  interrupts.CPU169.TLB:TLB_shootdowns
     26859 ±  5%     -46.1%      14479 ±  7%  interrupts.CPU17.CAL:Function_call_interrupts
      6134 ±  6%     -26.5%       4509 ± 18%  interrupts.CPU17.NMI:Non-maskable_interrupts
      6134 ±  6%     -26.5%       4509 ± 18%  interrupts.CPU17.PMI:Performance_monitoring_interrupts
     23653 ±  6%     -51.2%      11535 ±  9%  interrupts.CPU17.TLB:TLB_shootdowns
     27483 ± 10%     -45.7%      14933 ± 13%  interrupts.CPU170.CAL:Function_call_interrupts
     24190 ± 10%     -51.1%      11827 ± 16%  interrupts.CPU170.TLB:TLB_shootdowns
     27788 ±  8%     -46.7%      14812 ± 12%  interrupts.CPU171.CAL:Function_call_interrupts
     24457 ±  8%     -51.9%      11762 ± 15%  interrupts.CPU171.TLB:TLB_shootdowns
     27544 ±  9%     -45.5%      15011 ±  8%  interrupts.CPU172.CAL:Function_call_interrupts
      6348 ±  7%     -33.9%       4194 ± 26%  interrupts.CPU172.NMI:Non-maskable_interrupts
      6348 ±  7%     -33.9%       4194 ± 26%  interrupts.CPU172.PMI:Performance_monitoring_interrupts
     24149 ±  9%     -50.5%      11961 ± 10%  interrupts.CPU172.TLB:TLB_shootdowns
     27362 ±  9%     -45.6%      14880 ±  8%  interrupts.CPU173.CAL:Function_call_interrupts
     23971 ± 10%     -50.8%      11805 ± 10%  interrupts.CPU173.TLB:TLB_shootdowns
     27355 ± 10%     -46.2%      14722 ±  8%  interrupts.CPU174.CAL:Function_call_interrupts
      6334 ±  6%     -22.9%       4882 ± 26%  interrupts.CPU174.NMI:Non-maskable_interrupts
      6334 ±  6%     -22.9%       4882 ± 26%  interrupts.CPU174.PMI:Performance_monitoring_interrupts
     24098 ± 10%     -51.5%      11690 ± 10%  interrupts.CPU174.TLB:TLB_shootdowns
     27909 ± 10%     -44.5%      15487 ±  9%  interrupts.CPU175.CAL:Function_call_interrupts
      6305 ±  6%     -33.4%       4196 ± 31%  interrupts.CPU175.NMI:Non-maskable_interrupts
      6305 ±  6%     -33.4%       4196 ± 31%  interrupts.CPU175.PMI:Performance_monitoring_interrupts
     24615 ± 10%     -49.7%      12380 ± 12%  interrupts.CPU175.TLB:TLB_shootdowns
     27866 ±  8%     -46.0%      15044 ±  8%  interrupts.CPU176.CAL:Function_call_interrupts
     24526 ±  8%     -51.4%      11927 ± 11%  interrupts.CPU176.TLB:TLB_shootdowns
     27709 ± 11%     -43.8%      15585 ± 11%  interrupts.CPU177.CAL:Function_call_interrupts
     24404 ± 12%     -48.7%      12521 ± 15%  interrupts.CPU177.TLB:TLB_shootdowns
     28578 ±  8%     -47.0%      15143 ±  9%  interrupts.CPU178.CAL:Function_call_interrupts
     25259 ±  8%     -52.2%      12066 ± 11%  interrupts.CPU178.TLB:TLB_shootdowns
     27908 ±  8%     -46.2%      15014 ± 11%  interrupts.CPU179.CAL:Function_call_interrupts
     24608 ±  8%     -51.5%      11937 ± 15%  interrupts.CPU179.TLB:TLB_shootdowns
     27059 ±  7%     -45.1%      14845 ±  9%  interrupts.CPU18.CAL:Function_call_interrupts
     23765 ±  6%     -50.0%      11880 ± 11%  interrupts.CPU18.TLB:TLB_shootdowns
     27930 ±  9%     -45.6%      15199 ± 13%  interrupts.CPU180.CAL:Function_call_interrupts
     24730 ±  7%     -50.9%      12139 ± 16%  interrupts.CPU180.TLB:TLB_shootdowns
     27442 ±  6%     -48.0%      14279 ± 11%  interrupts.CPU181.CAL:Function_call_interrupts
     24097 ±  6%     -52.7%      11387 ± 15%  interrupts.CPU181.TLB:TLB_shootdowns
     27579 ±  7%     -46.7%      14708 ± 10%  interrupts.CPU182.CAL:Function_call_interrupts
     24311 ±  6%     -52.1%      11633 ± 14%  interrupts.CPU182.TLB:TLB_shootdowns
     27735 ±  9%     -46.0%      14966 ± 11%  interrupts.CPU183.CAL:Function_call_interrupts
     24351 ±  9%     -51.1%      11909 ± 13%  interrupts.CPU183.TLB:TLB_shootdowns
     27962 ± 10%     -45.6%      15200 ± 12%  interrupts.CPU184.CAL:Function_call_interrupts
     24675 ± 10%     -50.5%      12209 ± 15%  interrupts.CPU184.TLB:TLB_shootdowns
     27476 ±  6%     -45.1%      15087 ± 12%  interrupts.CPU185.CAL:Function_call_interrupts
     24052 ±  7%     -49.6%      12116 ± 15%  interrupts.CPU185.TLB:TLB_shootdowns
     27700 ±  7%     -46.8%      14723 ± 10%  interrupts.CPU186.CAL:Function_call_interrupts
     24339 ±  8%     -52.1%      11654 ± 13%  interrupts.CPU186.TLB:TLB_shootdowns
     27587 ±  8%     -45.8%      14949 ± 11%  interrupts.CPU187.CAL:Function_call_interrupts
      6344 ±  5%     -23.0%       4888 ± 25%  interrupts.CPU187.NMI:Non-maskable_interrupts
      6344 ±  5%     -23.0%       4888 ± 25%  interrupts.CPU187.PMI:Performance_monitoring_interrupts
     24155 ± 10%     -51.0%      11829 ± 15%  interrupts.CPU187.TLB:TLB_shootdowns
     28173 ± 10%     -48.0%      14658 ± 12%  interrupts.CPU188.CAL:Function_call_interrupts
     24829 ± 10%     -53.6%      11517 ± 15%  interrupts.CPU188.TLB:TLB_shootdowns
     28397 ±  8%     -49.6%      14307 ± 13%  interrupts.CPU189.CAL:Function_call_interrupts
     25047 ±  8%     -55.2%      11217 ± 17%  interrupts.CPU189.TLB:TLB_shootdowns
     27106 ±  8%     -44.4%      15067 ±  9%  interrupts.CPU19.CAL:Function_call_interrupts
     23886 ±  7%     -49.2%      12128 ± 11%  interrupts.CPU19.TLB:TLB_shootdowns
     28041 ± 10%     -47.2%      14810 ± 12%  interrupts.CPU190.CAL:Function_call_interrupts
     24695 ± 10%     -52.7%      11687 ± 16%  interrupts.CPU190.TLB:TLB_shootdowns
     28185 ±  9%     -47.3%      14851 ± 10%  interrupts.CPU191.CAL:Function_call_interrupts
      6298 ±  6%     -50.1%       3141 ± 49%  interrupts.CPU191.NMI:Non-maskable_interrupts
      6298 ±  6%     -50.1%       3141 ± 49%  interrupts.CPU191.PMI:Performance_monitoring_interrupts
     24829 ±  9%     -52.7%      11748 ± 13%  interrupts.CPU191.TLB:TLB_shootdowns
     25802 ±  6%     -49.2%      13105 ±  5%  interrupts.CPU2.CAL:Function_call_interrupts
     22493 ±  6%     -55.1%      10089 ±  7%  interrupts.CPU2.TLB:TLB_shootdowns
     26384 ±  9%     -43.1%      15007 ±  8%  interrupts.CPU20.CAL:Function_call_interrupts
     23142 ±  8%     -48.0%      12028 ± 10%  interrupts.CPU20.TLB:TLB_shootdowns
     26128 ±  9%     -43.2%      14833 ±  9%  interrupts.CPU21.CAL:Function_call_interrupts
      6251 ±  5%     -33.4%       4162 ± 33%  interrupts.CPU21.NMI:Non-maskable_interrupts
      6251 ±  5%     -33.4%       4162 ± 33%  interrupts.CPU21.PMI:Performance_monitoring_interrupts
     22823 ± 10%     -48.1%      11854 ± 11%  interrupts.CPU21.TLB:TLB_shootdowns
     26900 ±  8%     -45.7%      14604 ±  6%  interrupts.CPU22.CAL:Function_call_interrupts
      6279 ±  6%     -26.0%       4643 ± 17%  interrupts.CPU22.NMI:Non-maskable_interrupts
      6279 ±  6%     -26.0%       4643 ± 17%  interrupts.CPU22.PMI:Performance_monitoring_interrupts
     23712 ± 10%     -50.8%      11667 ±  7%  interrupts.CPU22.TLB:TLB_shootdowns
     26228 ±  9%     -44.3%      14618 ±  7%  interrupts.CPU23.CAL:Function_call_interrupts
      6251 ±  4%     -40.9%       3697 ± 45%  interrupts.CPU23.NMI:Non-maskable_interrupts
      6251 ±  4%     -40.9%       3697 ± 45%  interrupts.CPU23.PMI:Performance_monitoring_interrupts
     22955 ± 10%     -48.5%      11824 ±  9%  interrupts.CPU23.TLB:TLB_shootdowns
     24749 ± 12%     -42.9%      14142 ±  5%  interrupts.CPU24.CAL:Function_call_interrupts
      6406 ±  2%     -39.9%       3847 ± 34%  interrupts.CPU24.NMI:Non-maskable_interrupts
      6406 ±  2%     -39.9%       3847 ± 34%  interrupts.CPU24.PMI:Performance_monitoring_interrupts
     21366 ± 12%     -47.8%      11161 ±  7%  interrupts.CPU24.TLB:TLB_shootdowns
     24735 ±  9%     -41.2%      14545 ±  5%  interrupts.CPU25.CAL:Function_call_interrupts
      6006 ±  5%     -32.6%       4045 ± 19%  interrupts.CPU25.NMI:Non-maskable_interrupts
      6006 ±  5%     -32.6%       4045 ± 19%  interrupts.CPU25.PMI:Performance_monitoring_interrupts
     21376 ±  8%     -45.8%      11588 ±  6%  interrupts.CPU25.TLB:TLB_shootdowns
     24910 ±  9%     -39.7%      15012 ±  5%  interrupts.CPU26.CAL:Function_call_interrupts
      6311 ±  3%     -34.7%       4123 ± 23%  interrupts.CPU26.NMI:Non-maskable_interrupts
      6311 ±  3%     -34.7%       4123 ± 23%  interrupts.CPU26.PMI:Performance_monitoring_interrupts
     21534 ±  9%     -43.7%      12131 ±  8%  interrupts.CPU26.TLB:TLB_shootdowns
     25255 ±  8%     -41.0%      14892 ±  4%  interrupts.CPU27.CAL:Function_call_interrupts
     21957 ±  7%     -45.5%      11959 ±  5%  interrupts.CPU27.TLB:TLB_shootdowns
     25496 ±  8%     -41.6%      14890 ±  2%  interrupts.CPU28.CAL:Function_call_interrupts
      6340 ±  2%     -44.1%       3546 ± 37%  interrupts.CPU28.NMI:Non-maskable_interrupts
      6340 ±  2%     -44.1%       3546 ± 37%  interrupts.CPU28.PMI:Performance_monitoring_interrupts
     22200 ±  7%     -46.0%      11986 ±  3%  interrupts.CPU28.TLB:TLB_shootdowns
     25768 ±  8%     -41.9%      14977 ±  4%  interrupts.CPU29.CAL:Function_call_interrupts
     22356 ±  8%     -46.5%      11961 ±  4%  interrupts.CPU29.TLB:TLB_shootdowns
     26674 ±  7%     -46.9%      14173 ±  8%  interrupts.CPU3.CAL:Function_call_interrupts
      6329 ±  5%     -26.1%       4680 ± 20%  interrupts.CPU3.NMI:Non-maskable_interrupts
      6329 ±  5%     -26.1%       4680 ± 20%  interrupts.CPU3.PMI:Performance_monitoring_interrupts
     23313 ±  7%     -52.0%      11198 ± 11%  interrupts.CPU3.TLB:TLB_shootdowns
     24951 ±  7%     -37.8%      15520 ±  3%  interrupts.CPU30.CAL:Function_call_interrupts
     21567 ±  6%     -41.9%      12538 ±  3%  interrupts.CPU30.TLB:TLB_shootdowns
     25942 ± 10%     -38.6%      15925 ±  2%  interrupts.CPU31.CAL:Function_call_interrupts
      6355 ±  4%     -31.7%       4343 ± 18%  interrupts.CPU31.NMI:Non-maskable_interrupts
      6355 ±  4%     -31.7%       4343 ± 18%  interrupts.CPU31.PMI:Performance_monitoring_interrupts
     22629 ± 10%     -42.8%      12954 ±  2%  interrupts.CPU31.TLB:TLB_shootdowns
     25946 ±  6%     -40.2%      15516 ±  2%  interrupts.CPU32.CAL:Function_call_interrupts
      6418 ±  4%     -41.7%       3744 ± 31%  interrupts.CPU32.NMI:Non-maskable_interrupts
      6418 ±  4%     -41.7%       3744 ± 31%  interrupts.CPU32.PMI:Performance_monitoring_interrupts
     22602 ±  5%     -44.1%      12631 ±  2%  interrupts.CPU32.TLB:TLB_shootdowns
     25218 ±  8%     -40.5%      15014        interrupts.CPU33.CAL:Function_call_interrupts
     21993 ±  6%     -44.1%      12292        interrupts.CPU33.TLB:TLB_shootdowns
     24915 ± 10%     -36.8%      15737 ±  6%  interrupts.CPU34.CAL:Function_call_interrupts
     21629 ±  9%     -41.0%      12751 ±  7%  interrupts.CPU34.TLB:TLB_shootdowns
     25492 ± 10%     -39.2%      15504 ±  4%  interrupts.CPU35.CAL:Function_call_interrupts
     22134 ±  9%     -43.2%      12579 ±  5%  interrupts.CPU35.TLB:TLB_shootdowns
     25612 ±  7%     -38.6%      15723 ±  5%  interrupts.CPU36.CAL:Function_call_interrupts
      6202 ±  5%     -56.3%       2713 ± 47%  interrupts.CPU36.NMI:Non-maskable_interrupts
      6202 ±  5%     -56.3%       2713 ± 47%  interrupts.CPU36.PMI:Performance_monitoring_interrupts
     22216 ±  6%     -42.6%      12741 ±  7%  interrupts.CPU36.TLB:TLB_shootdowns
     24492 ± 14%     -35.7%      15750 ±  4%  interrupts.CPU37.CAL:Function_call_interrupts
      5455 ± 24%     -46.0%       2945 ± 35%  interrupts.CPU37.NMI:Non-maskable_interrupts
      5455 ± 24%     -46.0%       2945 ± 35%  interrupts.CPU37.PMI:Performance_monitoring_interrupts
     21124 ± 15%     -39.2%      12849 ±  5%  interrupts.CPU37.TLB:TLB_shootdowns
     25436 ±  9%     -37.7%      15841 ±  3%  interrupts.CPU38.CAL:Function_call_interrupts
      6297 ±  3%     -31.3%       4327 ± 21%  interrupts.CPU38.NMI:Non-maskable_interrupts
      6297 ±  3%     -31.3%       4327 ± 21%  interrupts.CPU38.PMI:Performance_monitoring_interrupts
     22093 ±  8%     -41.7%      12888 ±  3%  interrupts.CPU38.TLB:TLB_shootdowns
     24543 ±  9%     -35.4%      15843 ±  5%  interrupts.CPU39.CAL:Function_call_interrupts
      6360 ±  5%     -43.0%       3623 ± 32%  interrupts.CPU39.NMI:Non-maskable_interrupts
      6360 ±  5%     -43.0%       3623 ± 32%  interrupts.CPU39.PMI:Performance_monitoring_interrupts
     21208 ±  8%     -39.1%      12907 ±  6%  interrupts.CPU39.TLB:TLB_shootdowns
     26453 ±  8%     -47.2%      13959 ±  7%  interrupts.CPU4.CAL:Function_call_interrupts
     23163 ±  8%     -52.3%      11052 ± 10%  interrupts.CPU4.TLB:TLB_shootdowns
     24773 ± 12%     -36.3%      15769 ±  3%  interrupts.CPU40.CAL:Function_call_interrupts
     21535 ± 11%     -40.5%      12806 ±  4%  interrupts.CPU40.TLB:TLB_shootdowns
     25594 ± 10%     -39.8%      15414 ±  2%  interrupts.CPU41.CAL:Function_call_interrupts
     22313 ±  8%     -44.4%      12401 ±  2%  interrupts.CPU41.TLB:TLB_shootdowns
     25198 ± 13%     -38.7%      15458 ±  4%  interrupts.CPU42.CAL:Function_call_interrupts
     21796 ± 13%     -42.4%      12565 ±  5%  interrupts.CPU42.TLB:TLB_shootdowns
     25525 ±  7%     -38.7%      15639 ±  6%  interrupts.CPU43.CAL:Function_call_interrupts
      6398 ±  3%     -32.6%       4310 ± 21%  interrupts.CPU43.NMI:Non-maskable_interrupts
      6398 ±  3%     -32.6%       4310 ± 21%  interrupts.CPU43.PMI:Performance_monitoring_interrupts
     22240 ±  6%     -43.1%      12660 ±  7%  interrupts.CPU43.TLB:TLB_shootdowns
     25386 ± 12%     -38.1%      15723 ±  5%  interrupts.CPU44.CAL:Function_call_interrupts
      6338 ±  5%     -44.1%       3545 ± 24%  interrupts.CPU44.NMI:Non-maskable_interrupts
      6338 ±  5%     -44.1%       3545 ± 24%  interrupts.CPU44.PMI:Performance_monitoring_interrupts
     22061 ± 11%     -42.0%      12799 ±  7%  interrupts.CPU44.TLB:TLB_shootdowns
     24287 ± 13%     -33.6%      16117 ±  2%  interrupts.CPU45.CAL:Function_call_interrupts
      6357 ±  5%     -27.5%       4607 ± 21%  interrupts.CPU45.NMI:Non-maskable_interrupts
      6357 ±  5%     -27.5%       4607 ± 21%  interrupts.CPU45.PMI:Performance_monitoring_interrupts
     20932 ± 12%     -37.4%      13099 ±  2%  interrupts.CPU45.TLB:TLB_shootdowns
     25436 ± 10%     -37.1%      15993        interrupts.CPU46.CAL:Function_call_interrupts
      6270 ±  5%     -30.6%       4350 ± 23%  interrupts.CPU46.NMI:Non-maskable_interrupts
      6270 ±  5%     -30.6%       4350 ± 23%  interrupts.CPU46.PMI:Performance_monitoring_interrupts
    747.75 ± 30%     -38.1%     462.50 ± 25%  interrupts.CPU46.RES:Rescheduling_interrupts
     22069 ±  9%     -41.1%      12997 ±  2%  interrupts.CPU46.TLB:TLB_shootdowns
     25670 ± 10%     -38.7%      15735 ±  8%  interrupts.CPU47.CAL:Function_call_interrupts
     22454 ± 10%     -43.3%      12733 ± 10%  interrupts.CPU47.TLB:TLB_shootdowns
     27053 ±  3%     -47.8%      14113 ±  6%  interrupts.CPU48.CAL:Function_call_interrupts
      6279 ±  2%     -34.2%       4134 ± 25%  interrupts.CPU48.NMI:Non-maskable_interrupts
      6279 ±  2%     -34.2%       4134 ± 25%  interrupts.CPU48.PMI:Performance_monitoring_interrupts
     23836 ±  4%     -53.3%      11123 ±  7%  interrupts.CPU48.TLB:TLB_shootdowns
     26789           -49.1%      13642 ±  7%  interrupts.CPU49.CAL:Function_call_interrupts
      6202           -35.4%       4006 ± 38%  interrupts.CPU49.NMI:Non-maskable_interrupts
      6202           -35.4%       4006 ± 38%  interrupts.CPU49.PMI:Performance_monitoring_interrupts
     23620 ±  4%     -54.9%      10645 ±  9%  interrupts.CPU49.TLB:TLB_shootdowns
     26950 ±  4%     -45.1%      14808 ±  7%  interrupts.CPU5.CAL:Function_call_interrupts
      6268 ±  5%     -35.6%       4034 ± 32%  interrupts.CPU5.NMI:Non-maskable_interrupts
      6268 ±  5%     -35.6%       4034 ± 32%  interrupts.CPU5.PMI:Performance_monitoring_interrupts
     23633 ±  5%     -49.9%      11849 ±  9%  interrupts.CPU5.TLB:TLB_shootdowns
     26621           -46.0%      14377 ±  2%  interrupts.CPU50.CAL:Function_call_interrupts
      6283 ±  3%     -31.5%       4304 ± 20%  interrupts.CPU50.NMI:Non-maskable_interrupts
      6283 ±  3%     -31.5%       4304 ± 20%  interrupts.CPU50.PMI:Performance_monitoring_interrupts
     23351 ±  3%     -51.4%      11359 ±  3%  interrupts.CPU50.TLB:TLB_shootdowns
     26585           -44.9%      14661 ±  5%  interrupts.CPU51.CAL:Function_call_interrupts
      6309 ±  4%     -36.1%       4031 ± 38%  interrupts.CPU51.NMI:Non-maskable_interrupts
      6309 ±  4%     -36.1%       4031 ± 38%  interrupts.CPU51.PMI:Performance_monitoring_interrupts
     23286 ±  5%     -49.8%      11685 ±  6%  interrupts.CPU51.TLB:TLB_shootdowns
     26514 ±  4%     -42.4%      15272 ±  7%  interrupts.CPU52.CAL:Function_call_interrupts
      6147 ±  5%     -48.6%       3159 ± 33%  interrupts.CPU52.NMI:Non-maskable_interrupts
      6147 ±  5%     -48.6%       3159 ± 33%  interrupts.CPU52.PMI:Performance_monitoring_interrupts
     23414 ±  7%     -47.6%      12270 ±  9%  interrupts.CPU52.TLB:TLB_shootdowns
     27328 ±  3%     -46.5%      14624 ±  5%  interrupts.CPU53.CAL:Function_call_interrupts
     24033 ±  7%     -51.6%      11632 ±  6%  interrupts.CPU53.TLB:TLB_shootdowns
     26150 ±  3%     -43.3%      14824 ±  8%  interrupts.CPU54.CAL:Function_call_interrupts
     22887 ±  3%     -48.1%      11879 ± 10%  interrupts.CPU54.TLB:TLB_shootdowns
     27196 ±  2%     -45.0%      14952 ±  6%  interrupts.CPU55.CAL:Function_call_interrupts
      6356 ±  2%     -32.9%       4263 ± 23%  interrupts.CPU55.NMI:Non-maskable_interrupts
      6356 ±  2%     -32.9%       4263 ± 23%  interrupts.CPU55.PMI:Performance_monitoring_interrupts
     23944 ±  4%     -49.7%      12051 ±  7%  interrupts.CPU55.TLB:TLB_shootdowns
     26840 ±  2%     -43.1%      15283 ±  6%  interrupts.CPU56.CAL:Function_call_interrupts
     23570 ±  5%     -47.7%      12319 ±  7%  interrupts.CPU56.TLB:TLB_shootdowns
     27239 ±  2%     -43.1%      15511 ±  8%  interrupts.CPU57.CAL:Function_call_interrupts
    393.00 ± 12%     +21.9%     479.25 ± 13%  interrupts.CPU57.RES:Rescheduling_interrupts
     23900 ±  5%     -47.6%      12525 ± 10%  interrupts.CPU57.TLB:TLB_shootdowns
     27064 ±  2%     -43.0%      15437 ±  5%  interrupts.CPU58.CAL:Function_call_interrupts
     23828 ±  4%     -47.7%      12455 ±  7%  interrupts.CPU58.TLB:TLB_shootdowns
     27483           -43.8%      15433 ± 10%  interrupts.CPU59.CAL:Function_call_interrupts
     24266 ±  3%     -48.6%      12463 ± 12%  interrupts.CPU59.TLB:TLB_shootdowns
     26361 ±  5%     -46.2%      14179 ±  7%  interrupts.CPU6.CAL:Function_call_interrupts
      6265 ±  4%     -24.6%       4723 ± 18%  interrupts.CPU6.NMI:Non-maskable_interrupts
      6265 ±  4%     -24.6%       4723 ± 18%  interrupts.CPU6.PMI:Performance_monitoring_interrupts
     23127 ±  6%     -51.5%      11222 ±  9%  interrupts.CPU6.TLB:TLB_shootdowns
     26757 ±  4%     -43.3%      15169 ±  7%  interrupts.CPU60.CAL:Function_call_interrupts
     24077 ±  4%     -49.3%      12205 ±  9%  interrupts.CPU60.TLB:TLB_shootdowns
     27530           -43.7%      15489 ±  6%  interrupts.CPU61.CAL:Function_call_interrupts
     24260 ±  2%     -48.4%      12528 ±  9%  interrupts.CPU61.TLB:TLB_shootdowns
     27161           -44.6%      15044 ±  5%  interrupts.CPU62.CAL:Function_call_interrupts
     23894 ±  3%     -49.2%      12132 ±  7%  interrupts.CPU62.TLB:TLB_shootdowns
     26635 ±  3%     -42.9%      15214 ±  5%  interrupts.CPU63.CAL:Function_call_interrupts
     23534 ±  3%     -47.9%      12263 ±  6%  interrupts.CPU63.TLB:TLB_shootdowns
     27482 ±  3%     -45.4%      15004 ±  4%  interrupts.CPU64.CAL:Function_call_interrupts
     24240 ±  5%     -50.5%      12010 ±  5%  interrupts.CPU64.TLB:TLB_shootdowns
     27551 ±  2%     -43.2%      15661 ±  4%  interrupts.CPU65.CAL:Function_call_interrupts
     24288 ±  3%     -47.8%      12668 ±  6%  interrupts.CPU65.TLB:TLB_shootdowns
     27620 ±  4%     -44.8%      15257 ±  5%  interrupts.CPU66.CAL:Function_call_interrupts
     24313 ±  5%     -49.6%      12248 ±  7%  interrupts.CPU66.TLB:TLB_shootdowns
     27420 ±  2%     -43.7%      15438 ±  7%  interrupts.CPU67.CAL:Function_call_interrupts
      6064 ±  8%     -27.8%       4378 ± 18%  interrupts.CPU67.NMI:Non-maskable_interrupts
      6064 ±  8%     -27.8%       4378 ± 18%  interrupts.CPU67.PMI:Performance_monitoring_interrupts
     24187 ±  4%     -48.5%      12455 ±  9%  interrupts.CPU67.TLB:TLB_shootdowns
     27980 ±  3%     -46.0%      15105 ±  6%  interrupts.CPU68.CAL:Function_call_interrupts
      6223 ±  4%     -30.8%       4303 ± 20%  interrupts.CPU68.NMI:Non-maskable_interrupts
      6223 ±  4%     -30.8%       4303 ± 20%  interrupts.CPU68.PMI:Performance_monitoring_interrupts
     24702 ±  6%     -50.9%      12120 ±  8%  interrupts.CPU68.TLB:TLB_shootdowns
     27578 ±  5%     -45.0%      15181 ±  7%  interrupts.CPU69.CAL:Function_call_interrupts
      6112 ±  4%     -40.5%       3638 ± 16%  interrupts.CPU69.NMI:Non-maskable_interrupts
      6112 ±  4%     -40.5%       3638 ± 16%  interrupts.CPU69.PMI:Performance_monitoring_interrupts
     24269 ±  6%     -49.7%      12205 ±  8%  interrupts.CPU69.TLB:TLB_shootdowns
     26146 ±  7%     -44.8%      14424 ±  7%  interrupts.CPU7.CAL:Function_call_interrupts
     22825 ±  6%     -49.7%      11483 ± 10%  interrupts.CPU7.TLB:TLB_shootdowns
     27118 ±  5%     -46.2%      14601 ±  8%  interrupts.CPU70.CAL:Function_call_interrupts
      6198 ±  4%     -39.0%       3780 ± 32%  interrupts.CPU70.NMI:Non-maskable_interrupts
      6198 ±  4%     -39.0%       3780 ± 32%  interrupts.CPU70.PMI:Performance_monitoring_interrupts
     23866 ±  8%     -51.3%      11619 ± 10%  interrupts.CPU70.TLB:TLB_shootdowns
     27347 ±  6%     -45.7%      14855 ±  7%  interrupts.CPU71.CAL:Function_call_interrupts
     24244 ± 10%     -51.0%      11883 ±  9%  interrupts.CPU71.TLB:TLB_shootdowns
     26505 ±  9%     -45.8%      14367 ±  6%  interrupts.CPU72.CAL:Function_call_interrupts
     23166 ±  9%     -50.8%      11391 ±  8%  interrupts.CPU72.TLB:TLB_shootdowns
     27462 ±  8%     -47.5%      14410 ± 13%  interrupts.CPU73.CAL:Function_call_interrupts
      6234 ±  3%     -32.7%       4197 ± 25%  interrupts.CPU73.NMI:Non-maskable_interrupts
      6234 ±  3%     -32.7%       4197 ± 25%  interrupts.CPU73.PMI:Performance_monitoring_interrupts
     24197 ±  8%     -52.9%      11396 ± 17%  interrupts.CPU73.TLB:TLB_shootdowns
     26916 ±  8%     -47.1%      14237 ±  9%  interrupts.CPU74.CAL:Function_call_interrupts
     23636 ±  8%     -52.1%      11321 ± 11%  interrupts.CPU74.TLB:TLB_shootdowns
     27194 ±  8%     -46.6%      14528 ±  8%  interrupts.CPU75.CAL:Function_call_interrupts
      6285 ±  6%     -33.9%       4153 ± 31%  interrupts.CPU75.NMI:Non-maskable_interrupts
      6285 ±  6%     -33.9%       4153 ± 31%  interrupts.CPU75.PMI:Performance_monitoring_interrupts
     23889 ±  8%     -51.9%      11497 ± 11%  interrupts.CPU75.TLB:TLB_shootdowns
     27127 ±  9%     -45.3%      14833 ± 11%  interrupts.CPU76.CAL:Function_call_interrupts
     23874 ± 11%     -50.4%      11841 ± 14%  interrupts.CPU76.TLB:TLB_shootdowns
     27266 ±  9%     -46.1%      14706 ± 10%  interrupts.CPU77.CAL:Function_call_interrupts
     23940 ± 10%     -51.3%      11653 ± 13%  interrupts.CPU77.TLB:TLB_shootdowns
     27483 ±  9%     -46.9%      14595 ± 11%  interrupts.CPU78.CAL:Function_call_interrupts
     24199 ±  9%     -52.0%      11605 ± 15%  interrupts.CPU78.TLB:TLB_shootdowns
     27843 ± 11%     -48.8%      14261 ±  9%  interrupts.CPU79.CAL:Function_call_interrupts
      6445 ±  4%     -24.5%       4868 ± 22%  interrupts.CPU79.NMI:Non-maskable_interrupts
      6445 ±  4%     -24.5%       4868 ± 22%  interrupts.CPU79.PMI:Performance_monitoring_interrupts
     24643 ± 12%     -54.0%      11330 ± 10%  interrupts.CPU79.TLB:TLB_shootdowns
     26776 ± 11%     -46.1%      14430 ±  6%  interrupts.CPU8.CAL:Function_call_interrupts
     23541 ± 11%     -51.4%      11431 ±  8%  interrupts.CPU8.TLB:TLB_shootdowns
     27139 ± 10%     -45.4%      14826 ± 10%  interrupts.CPU80.CAL:Function_call_interrupts
     23925 ± 10%     -50.4%      11866 ± 13%  interrupts.CPU80.TLB:TLB_shootdowns
     26416 ± 12%     -45.7%      14354 ± 10%  interrupts.CPU81.CAL:Function_call_interrupts
     23185 ± 12%     -50.8%      11399 ± 13%  interrupts.CPU81.TLB:TLB_shootdowns
     28161 ± 11%     -47.9%      14665 ± 10%  interrupts.CPU82.CAL:Function_call_interrupts
      6338 ±  5%     -34.3%       4164 ± 29%  interrupts.CPU82.NMI:Non-maskable_interrupts
      6338 ±  5%     -34.3%       4164 ± 29%  interrupts.CPU82.PMI:Performance_monitoring_interrupts
     24870 ± 11%     -52.7%      11756 ± 13%  interrupts.CPU82.TLB:TLB_shootdowns
     27751 ± 11%     -46.6%      14821 ± 10%  interrupts.CPU83.CAL:Function_call_interrupts
     24497 ± 11%     -51.7%      11823 ± 13%  interrupts.CPU83.TLB:TLB_shootdowns
     27812 ±  9%     -47.2%      14675 ± 11%  interrupts.CPU84.CAL:Function_call_interrupts
     24536 ±  9%     -52.5%      11650 ± 14%  interrupts.CPU84.TLB:TLB_shootdowns
     27537 ±  8%     -44.6%      15254 ± 10%  interrupts.CPU85.CAL:Function_call_interrupts
     24260 ±  8%     -49.0%      12380 ± 13%  interrupts.CPU85.TLB:TLB_shootdowns
     27193 ± 11%     -47.4%      14313 ± 12%  interrupts.CPU86.CAL:Function_call_interrupts
      6416 ±  4%     -45.7%       3485 ± 27%  interrupts.CPU86.NMI:Non-maskable_interrupts
      6416 ±  4%     -45.7%       3485 ± 27%  interrupts.CPU86.PMI:Performance_monitoring_interrupts
     24032 ± 11%     -52.7%      11370 ± 15%  interrupts.CPU86.TLB:TLB_shootdowns
     27092 ±  9%     -45.6%      14750 ±  7%  interrupts.CPU87.CAL:Function_call_interrupts
     23820 ± 10%     -50.7%      11754 ± 10%  interrupts.CPU87.TLB:TLB_shootdowns
     27593 ±  9%     -45.8%      14961 ±  9%  interrupts.CPU88.CAL:Function_call_interrupts
     24289 ±  9%     -50.8%      11950 ± 11%  interrupts.CPU88.TLB:TLB_shootdowns
     27852 ±  9%     -47.1%      14746 ±  8%  interrupts.CPU89.CAL:Function_call_interrupts
     24606 ± 10%     -52.4%      11718 ± 11%  interrupts.CPU89.TLB:TLB_shootdowns
     27251 ±  4%     -46.2%      14670 ±  2%  interrupts.CPU9.CAL:Function_call_interrupts
      6241 ±  4%     -40.6%       3710 ± 30%  interrupts.CPU9.NMI:Non-maskable_interrupts
      6241 ±  4%     -40.6%       3710 ± 30%  interrupts.CPU9.PMI:Performance_monitoring_interrupts
     23911 ±  5%     -50.8%      11756 ±  2%  interrupts.CPU9.TLB:TLB_shootdowns
     26653 ±  8%     -44.2%      14876 ± 10%  interrupts.CPU90.CAL:Function_call_interrupts
      6351 ±  3%     -31.2%       4370 ± 32%  interrupts.CPU90.NMI:Non-maskable_interrupts
      6351 ±  3%     -31.2%       4370 ± 32%  interrupts.CPU90.PMI:Performance_monitoring_interrupts
     23402 ±  8%     -49.3%      11858 ± 13%  interrupts.CPU90.TLB:TLB_shootdowns
     27116 ± 10%     -46.6%      14491 ± 10%  interrupts.CPU91.CAL:Function_call_interrupts
      6467 ±  4%     -33.5%       4301 ± 34%  interrupts.CPU91.NMI:Non-maskable_interrupts
      6467 ±  4%     -33.5%       4301 ± 34%  interrupts.CPU91.PMI:Performance_monitoring_interrupts
     23824 ± 11%     -51.6%      11536 ± 12%  interrupts.CPU91.TLB:TLB_shootdowns
     26961 ± 11%     -46.1%      14532 ±  8%  interrupts.CPU92.CAL:Function_call_interrupts
     23679 ± 13%     -51.4%      11497 ± 11%  interrupts.CPU92.TLB:TLB_shootdowns
     27548 ± 10%     -46.2%      14817 ± 11%  interrupts.CPU93.CAL:Function_call_interrupts
      6512 ±  5%     -30.4%       4531 ± 38%  interrupts.CPU93.NMI:Non-maskable_interrupts
      6512 ±  5%     -30.4%       4531 ± 38%  interrupts.CPU93.PMI:Performance_monitoring_interrupts
     24280 ± 11%     -51.2%      11841 ± 14%  interrupts.CPU93.TLB:TLB_shootdowns
     27800 ± 11%     -47.4%      14610 ±  9%  interrupts.CPU94.CAL:Function_call_interrupts
      6433 ±  4%     -41.1%       3789 ± 45%  interrupts.CPU94.NMI:Non-maskable_interrupts
      6433 ±  4%     -41.1%       3789 ± 45%  interrupts.CPU94.PMI:Performance_monitoring_interrupts
     24567 ± 11%     -52.6%      11637 ± 11%  interrupts.CPU94.TLB:TLB_shootdowns
     27488 ±  9%     -44.3%      15305 ±  7%  interrupts.CPU95.CAL:Function_call_interrupts
     24215 ± 10%     -48.8%      12388 ±  9%  interrupts.CPU95.TLB:TLB_shootdowns
     24706 ±  3%     -47.9%      12861 ± 10%  interrupts.CPU96.CAL:Function_call_interrupts
     21352 ±  3%     -54.2%       9779 ± 12%  interrupts.CPU96.TLB:TLB_shootdowns
     26692 ±  6%     -46.5%      14274 ±  6%  interrupts.CPU97.CAL:Function_call_interrupts
      6129 ±  4%     -35.1%       3975 ± 30%  interrupts.CPU97.NMI:Non-maskable_interrupts
      6129 ±  4%     -35.1%       3975 ± 30%  interrupts.CPU97.PMI:Performance_monitoring_interrupts
     23379 ±  4%     -51.5%      11344 ±  8%  interrupts.CPU97.TLB:TLB_shootdowns
     26419 ±  6%     -46.1%      14244 ±  7%  interrupts.CPU98.CAL:Function_call_interrupts
     23076 ±  7%     -50.9%      11339 ± 10%  interrupts.CPU98.TLB:TLB_shootdowns
     26972 ±  7%     -44.4%      14989 ± 12%  interrupts.CPU99.CAL:Function_call_interrupts
      6269 ±  5%     -46.7%       3344 ± 27%  interrupts.CPU99.NMI:Non-maskable_interrupts
      6269 ±  5%     -46.7%       3344 ± 27%  interrupts.CPU99.PMI:Performance_monitoring_interrupts
     23671 ±  8%     -49.2%      12023 ± 16%  interrupts.CPU99.TLB:TLB_shootdowns
    170.25           -44.5%      94.50 ± 47%  interrupts.IWI:IRQ_work_interrupts
   1103722           -28.1%     794110 ± 14%  interrupts.NMI:Non-maskable_interrupts
   1103722           -28.1%     794110 ± 14%  interrupts.PMI:Performance_monitoring_interrupts
   4531323 ±  2%     -48.8%    2317840 ±  6%  interrupts.TLB:TLB_shootdowns


                                                                                
                            vm-scalability.time.user_time                       
                                                                                
  25000 +-+-----------------------------------------------------------------+   
        |   +.+.+.+.+   +.+.+.+.+     +.+.+.+.+.+.+.+   +.+     +.+.+.+.+.+.|   
        |                       O                                           |   
  20000 O-+       O O O O O O O                                             |   
        |                                                                   |   
        |                         O O O O                                   |   
  15000 +-+                                                                 |   
        |                                                                   |   
  10000 +-+                                                                 |   
        |                                                                   |   
        |                                                                   |   
   5000 +-+                                                                 |   
        |                                                                   |   
        |                                                                   |   
      0 +-O-O-O-O-----------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                           vm-scalability.time.system_time                      
                                                                                
  25000 +-+-----------------------------------------------------------------+   
        |. .+.+.+.+.               .+.             .+.     .+.              |   
        | +         +.+.+.+.+.+.+.+   +.+.+.+.+.+.+   +.+.+   +.+.+.+.+.+.+.|   
  20000 O-+       O O O O O O O O                                           |   
        |                           O O                                     |   
        |                         O     O                                   |   
  15000 +-+                                                                 |   
        |                                                                   |   
  10000 +-+                                                                 |   
        |                                                                   |   
        |                                                                   |   
   5000 +-+                                                                 |   
        |                                                                   |   
        |                                                                   |   
      0 +-O-O-O-O-----------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                   vm-scalability.time.percent_of_cpu_this_job_got              
                                                                                
  16000 +-+-----------------------------------------------------------------+   
        |.+.+.+.+.+.+.+.+.+.+.+.+.+   +.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.|   
  14000 O-+         O           O                                           |   
  12000 +-+       O   O O O O O                                             |   
        |                           O O                                     |   
  10000 +-+                       O     O                                   |   
        |                                                                   |   
   8000 +-+                                                                 |   
        |                                                                   |   
   6000 +-+                                                                 |   
   4000 +-+                                                                 |   
        |                                                                   |   
   2000 +-+                                                                 |   
        |                                                                   |   
      0 +-O-O-O-O-----------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                         vm-scalability.time.minor_page_faults                  
                                                                                
  1.6e+07 +-+---------------------------------------------------------------+   
          |                                                                 |   
  1.4e+07 +-+.+.+.+.+.+.+.++.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.++.+.+.+.+.+.+.+.|   
  1.2e+07 O-+       O O O OO O O O                                          |   
          |                                                                 |   
    1e+07 +-+                      O O O O                                  |   
          |                                                                 |   
    8e+06 +-+                                                               |   
          |                                                                 |   
    6e+06 +-+                                                               |   
    4e+06 +-+                                                               |   
          |                                                                 |   
    2e+06 +-+                                                               |   
          |                                                                 |   
        0 +-O-O-O-O---------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                    vm-scalability.time.voluntary_context_switches              
                                                                                
  1.6e+06 +-+---------------------------------------------------------------+   
          |.+. .+.   .+. .++. .+. .+.+.     .+. .+.+.     .+ .+. +        + |   
  1.4e+06 O-+ +   +.O O O  O + O O     +.+.+   +     +.+.+  +   +           |   
          |               O  O                                              |   
  1.2e+06 +-+                                                               |   
    1e+06 +-+                                                               |   
          |                                                                 |   
   800000 +-+                      O O O O                                  |   
          |                                                                 |   
   600000 +-+                                                               |   
   400000 +-+                                                               |   
          |                                                                 |   
   200000 +-+                                                               |   
          |                                                                 |   
        0 +-O-O-O-O---------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                  vm-scalability.time.involuntary_context_switches              
                                                                                
  350000 +-+----------------------------------------------------------------+   
         |            .+.+.   .+.           +. .+.     .+.+.     .+.+. .+.  |   
  300000 +-+.+.+.+.+.+     +.+   +.+.+.+.+.+  +   +.+.+     +.+.+     +   +.|   
         |                                                                  |   
  250000 +-+                                                                |   
         |                                                                  |   
  200000 +-+                                                                |   
         O           O           O                                          |   
  150000 +-+       O   O O O O O                                            |   
         |                             O                                    |   
  100000 +-+                       O O   O                                  |   
         |                                                                  |   
   50000 +-+                                                                |   
         |                                                                  |   
       0 +-O-O-O-O----------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                              vm-scalability.free_time                          
                                                                                
  0.025 +-+-----------------------O-----O-----------------------------------+   
        |                                                                   |   
        |                           O O                                     |   
   0.02 +-+                                                                 |   
        |                                                                   |   
        |                                                                   |   
  0.015 +-+                                                                 |   
        |         O     O   O O O                                           |   
   0.01 O-+         O O   O                                                 |   
        |.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.|   
        |                                                                   |   
  0.005 +-+                                                                 |   
        |                                                                   |   
        |                                                                   |   
      0 +-O-O-O-O-----------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                vm-scalability.median                           
                                                                                
  200000 +-+----------------------------------------------------------------+   
  180000 +-+ +     +.+.O O O O +           +  +     +.+     +         +   + |   
         O         O O         O O                                          |   
  160000 +-+                       O O O O                                  |   
  140000 +-+                                                                |   
         |                                                                  |   
  120000 +-+                                                                |   
  100000 +-+                                                                |   
   80000 +-+                                                                |   
         |                                                                  |   
   60000 +-+                                                                |   
   40000 +-+                                                                |   
         |                                                                  |   
   20000 +-+                                                                |   
       0 +-O-O-O-O----------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                           vm-scalability.median_stddev                         
                                                                                
  0.12 +-+------------------------------------------------------------------+   
       |                         O O                                        |   
   0.1 +-+                           O                                      |   
       |                               O                                    |   
       O           O O   O   O                                              |   
  0.08 +-+             O   O   O                                            |   
       |         O                                                          |   
  0.06 +-+                                                                  |   
       |                                                            +.      |   
  0.04 +-+  .+.+. .+.+. .+. .+.+.+.   .+.. .+. .+. .+.+. .+.       +  +. .+.|   
       |.+.+     +     +   +       +.+    +   +   +     +   +.+.+.+     +   |   
       |                                                                    |   
  0.02 +-+                                                                  |   
       |                                                                    |   
     0 +-O-O-O-O------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                vm-scalability.workload                         
                                                                                
  1.2e+10 +-+---------------------------------------------------------------+   
          |                                                                 |   
    1e+10 +-+.+.+.+.+.+.+.++.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.++.+.+.+.+.+.+.+.|   
          O         O O O OO O O O                                          |   
          |                                                                 |   
    8e+09 +-+                      O O O O                                  |   
          |                                                                 |   
    6e+09 +-+                                                               |   
          |                                                                 |   
    4e+09 +-+                                                               |   
          |                                                                 |   
          |                                                                 |   
    2e+09 +-+                                                               |   
          |                                                                 |   
        0 +-O-O-O-O---------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample

***************************************************************************************************
lkp-ivb-ep01: 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz with 384G memory
=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/load/rootfs/tbox_group/test/testcase:
  gcc-7/performance/1BRD_48G/f2fs/x86_64-rhel-7.6/3000/debian-x86_64-2019-05-14.cgz/lkp-ivb-ep01/disk_src/aim7

commit: 
  94a9717b3c ("locking/rwsem: Make rwsem->owner an atomic_long_t")
  7d43f1ce9d ("locking/rwsem: Enable time-based spinning on reader-owned rwsem")

94a9717b3c40e77a 7d43f1ce9dd075d8b2aa3ad1f39 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :2           50%           1:2     dmesg.WARNING:at_ip_do_filp_open/0x



***************************************************************************************************
lkp-ivb-d02: 4 threads Intel(R) Core(TM) i3-3220 CPU @ 3.30GHz with 8G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/rootfs/runtime/tbox_group/test/testcase/ucode:
  gcc-7/performance/x86_64-rhel-7.6/debian-x86_64-2019-05-14.cgz/300s/lkp-ivb-d02/small-allocs-mt/vm-scalability/0x21

commit: 
  94a9717b3c ("locking/rwsem: Make rwsem->owner an atomic_long_t")
  7d43f1ce9d ("locking/rwsem: Enable time-based spinning on reader-owned rwsem")

94a9717b3c40e77a 7d43f1ce9dd075d8b2aa3ad1f39 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
    793660           +14.0%     904895        vm-scalability.median
   3173995           +14.0%    3617058        vm-scalability.throughput
    151.84           -11.9%     133.72        vm-scalability.time.elapsed_time
    151.84           -11.9%     133.72        vm-scalability.time.elapsed_time.max
    713096 ±  2%     +20.1%     856760 ±  2%  vm-scalability.time.involuntary_context_switches
    360.25            +4.9%     378.00        vm-scalability.time.percent_of_cpu_this_job_got
    200.83           -19.5%     161.71        vm-scalability.time.system_time
   7120853 ±  3%     -97.5%     180425        vm-scalability.time.voluntary_context_switches
     21.06            +5.8%      22.29 ±  4%  boot-time.boot
     54.91            +9.2%      59.95 ±  7%  boot-time.idle
    124534 ±  3%      -9.1%     113147        meminfo.AnonHugePages
     28262           +14.7%      32431        meminfo.max_used_kB
      8.74 ±  4%      -5.4        3.37 ±  3%  mpstat.cpu.all.idle%
     58.13            +7.7       65.86        mpstat.cpu.all.usr%
      1207 ±  4%     +14.8%       1385 ±  2%  slabinfo.kmalloc-96.active_objs
      1207 ±  4%     +14.8%       1385 ±  2%  slabinfo.kmalloc-96.num_objs
      9.25 ±  4%     -56.8%       4.00        vmstat.cpu.id
     57.25           +12.7%      64.50        vmstat.cpu.us
    101684 ±  2%     -83.4%      16863        vmstat.system.cs
     13871            +1.5%      14082        vmstat.system.in
      5053            -1.2%       4991        proc-vmstat.nr_inactive_anon
      5512            +1.4%       5588        proc-vmstat.nr_mapped
    260063            +1.6%     264256        proc-vmstat.nr_slab_unreclaimable
      5053            -1.2%       4991        proc-vmstat.nr_zone_inactive_anon
      1113            -1.6%       1095        proc-vmstat.pgactivate
     62845 ±  2%     -74.7%      15909 ± 10%  softirqs.CPU0.SCHED
     61686           -72.7%      16834 ±  6%  softirqs.CPU1.SCHED
     62727           -76.0%      15049 ±  9%  softirqs.CPU2.SCHED
     60562 ±  2%     -72.5%      16673 ±  4%  softirqs.CPU3.SCHED
    247822           -74.0%      64466 ±  2%  softirqs.SCHED
  30358831 ±  4%     -94.8%    1581973 ±  4%  cpuidle.C1.time
   6764050 ±  3%     -97.5%     169367        cpuidle.C1.usage
   1648723 ± 27%    +212.2%    5147756 ± 11%  cpuidle.C1E.time
     37741 ± 13%    +132.6%      87804 ± 18%  cpuidle.C1E.usage
    104315 ±  7%    +653.2%     785684 ± 76%  cpuidle.C3.time
      9526 ±  4%     -71.0%       2761 ± 64%  cpuidle.C3.usage
    196095 ±  2%     -98.8%       2432 ±  3%  cpuidle.POLL.time
     88406 ±  2%     -97.4%       2325 ±  4%  cpuidle.POLL.usage
      3059            +3.8%       3176        turbostat.Avg_MHz
   6764093 ±  3%     -97.5%     169364        turbostat.C1
      4.96 ±  3%      -4.7        0.29 ±  3%  turbostat.C1%
     37886 ± 13%    +131.7%      87799 ± 18%  turbostat.C1E
      0.27 ± 27%      +0.7        0.95 ± 10%  turbostat.C1E%
      9529 ±  4%     -71.1%       2754 ± 64%  turbostat.C3
      0.02 ± 24%      +0.1        0.14 ± 72%  turbostat.C3%
      1.83 ±  3%      +0.3        2.15 ±  2%  turbostat.C6%
      6.00 ±  3%     -61.8%       2.29 ±  6%  turbostat.CPU%c1
      0.00 ±173%  +25100.0%       0.63 ± 98%  turbostat.CPU%c3
   2138754           -10.3%    1918451        turbostat.IRQ
      4014           -10.5%       3593 ±  2%  interrupts.CAL:Function_call_interrupts
     18427 ±  6%     -95.1%     895.00 ± 11%  interrupts.CPU0.RES:Rescheduling_interrupts
    895.75 ±144%    -100.0%       0.00        interrupts.CPU1.NMI:Non-maskable_interrupts
    895.75 ±144%    -100.0%       0.00        interrupts.CPU1.PMI:Performance_monitoring_interrupts
     19299 ± 11%     -94.1%       1139 ± 20%  interrupts.CPU1.RES:Rescheduling_interrupts
     58.00 ± 73%    -100.0%       0.00        interrupts.CPU2.NMI:Non-maskable_interrupts
     58.00 ± 73%    -100.0%       0.00        interrupts.CPU2.PMI:Performance_monitoring_interrupts
     19013 ±  8%     -94.4%       1062 ± 24%  interrupts.CPU2.RES:Rescheduling_interrupts
    509962 ±  3%     -20.9%     403302 ±  4%  interrupts.CPU3.LOC:Local_timer_interrupts
    631.75 ±163%    -100.0%       0.00        interrupts.CPU3.NMI:Non-maskable_interrupts
    631.75 ±163%    -100.0%       0.00        interrupts.CPU3.PMI:Performance_monitoring_interrupts
     18040 ± 12%     -93.7%       1129 ± 14%  interrupts.CPU3.RES:Rescheduling_interrupts
      1607 ± 81%     -99.6%       6.00        interrupts.NMI:Non-maskable_interrupts
      1607 ± 81%     -99.6%       6.00        interrupts.PMI:Performance_monitoring_interrupts
     74780 ±  4%     -94.3%       4226 ± 13%  interrupts.RES:Rescheduling_interrupts
      2.25 ± 23%     +75.9%       3.96 ±  5%  sched_debug.cfs_rq:/.nr_spread_over.avg
      4.50 ± 44%    +103.7%       9.17 ±  5%  sched_debug.cfs_rq:/.nr_spread_over.max
      1.64 ± 49%    +101.5%       3.31 ±  3%  sched_debug.cfs_rq:/.nr_spread_over.stddev
    422.17 ± 14%     +29.7%     547.50 ±  3%  sched_debug.cfs_rq:/.runnable_load_avg.max
    135.56 ± 13%     +34.4%     182.15 ±  8%  sched_debug.cfs_rq:/.runnable_load_avg.stddev
      1047 ± 12%     +33.7%       1400 ±  3%  sched_debug.cfs_rq:/.util_est_enqueued.max
    416.51 ± 11%     +24.1%     516.73        sched_debug.cfs_rq:/.util_est_enqueued.stddev
    273573 ±  4%     +14.6%     313548 ±  9%  sched_debug.cpu.avg_idle.avg
      0.68 ± 22%    +155.0%       1.73 ± 14%  sched_debug.cpu.clock.stddev
      0.68 ± 22%    +155.0%       1.73 ± 14%  sched_debug.cpu.clock_task.stddev
    216.10 ± 10%     +34.3%     290.33 ± 12%  sched_debug.cpu.cpu_load[0].avg
    401.50 ± 23%     +40.5%     564.17 ±  4%  sched_debug.cpu.cpu_load[0].max
    112.08 ± 39%     +65.0%     184.96 ±  7%  sched_debug.cpu.cpu_load[0].stddev
    211.88 ±  9%     +17.9%     249.83 ±  4%  sched_debug.cpu.cpu_load[1].avg
    325.50 ± 15%     +47.4%     479.83 ±  7%  sched_debug.cpu.cpu_load[1].max
     72.16 ± 25%    +102.9%     146.40 ± 12%  sched_debug.cpu.cpu_load[1].stddev
    205.62 ±  6%     +11.2%     228.71 ±  3%  sched_debug.cpu.cpu_load[2].avg
    273.50 ±  9%     +47.5%     403.33        sched_debug.cpu.cpu_load[2].max
     47.18 ± 17%    +128.6%     107.86        sched_debug.cpu.cpu_load[2].stddev
    202.62 ±  4%      +7.5%     217.75 ±  3%  sched_debug.cpu.cpu_load[3].avg
    246.67 ±  5%     +38.0%     340.50 ±  7%  sched_debug.cpu.cpu_load[3].max
     32.83 ±  7%    +127.3%      74.63 ± 18%  sched_debug.cpu.cpu_load[3].stddev
    236.42 ±  4%     +26.2%     298.33 ± 11%  sched_debug.cpu.cpu_load[4].max
     25.55 ± 13%    +105.2%      52.42 ± 30%  sched_debug.cpu.cpu_load[4].stddev
      0.00 ± 14%     -50.1%       0.00        sched_debug.cpu.next_balance.stddev
   1905769 ±  2%     -79.6%     387826        sched_debug.cpu.nr_switches.avg
   1972844 ±  2%     -73.6%     520051 ±  5%  sched_debug.cpu.nr_switches.max
   1744240           -84.6%     269070 ±  3%  sched_debug.cpu.nr_switches.min
     23.92 ± 55%    +130.7%      55.17 ± 45%  sched_debug.cpu.nr_uninterruptible.max
     18.08 ± 60%    +118.2%      39.45 ± 26%  sched_debug.cpu.nr_uninterruptible.stddev
   1895971 ±  2%     -80.1%     377492        sched_debug.cpu.sched_count.avg
   1962479 ±  2%     -73.9%     512285 ±  5%  sched_debug.cpu.sched_count.max
   1734135           -84.9%     261092 ±  2%  sched_debug.cpu.sched_count.min
    794818 ±  2%     -94.8%      41267 ±  8%  sched_debug.cpu.sched_goidle.avg
    825986 ±  3%     -93.5%      54059 ± 12%  sched_debug.cpu.sched_goidle.max
    738516 ±  2%     -96.2%      28020        sched_debug.cpu.sched_goidle.min
     33586 ± 11%     -71.6%       9536 ± 24%  sched_debug.cpu.sched_goidle.stddev
    986934 ±  2%     -80.6%     191360        sched_debug.cpu.ttwu_count.avg
   1048238 ±  2%     -74.3%     269323 ±  5%  sched_debug.cpu.ttwu_count.max
    882027 ±  2%     -86.0%     123850 ±  5%  sched_debug.cpu.ttwu_count.min
    179043 ±  2%     +29.0%     230973 ±  6%  sched_debug.cpu.ttwu_local.max
    130127 ±  4%     -22.1%     101364 ±  6%  sched_debug.cpu.ttwu_local.min
     18651 ± 12%    +174.2%      51141 ±  9%  sched_debug.cpu.ttwu_local.stddev
      2.32 ± 16%     -62.1%       0.88        perf-stat.i.MPKI
 3.793e+09           +10.4%  4.187e+09        perf-stat.i.branch-instructions
      2.64 ± 14%      +2.8        5.39 ±  6%  perf-stat.i.cache-miss-rate%
    499435 ± 17%     +37.6%     687301 ±  9%  perf-stat.i.cache-misses
  24288477 ±  3%     -51.9%   11692134        perf-stat.i.cache-references
    103479 ±  3%     -83.4%      17136        perf-stat.i.context-switches
      1.02            -6.7%       0.95        perf-stat.i.cpi
  1.23e+10            +3.9%  1.278e+10        perf-stat.i.cpu-cycles
      1998 ±  2%     -96.2%      76.32 ± 10%  perf-stat.i.cpu-migrations
 3.219e+09            +9.8%  3.533e+09        perf-stat.i.dTLB-loads
      0.28            +0.0        0.29        perf-stat.i.dTLB-store-miss-rate%
   2078000            +4.5%    2171902        perf-stat.i.dTLB-store-misses
 7.347e+08            +2.0%  7.492e+08        perf-stat.i.dTLB-stores
     84.00           +14.2       98.21        perf-stat.i.iTLB-load-miss-rate%
   1805387 ±  6%     -31.7%    1232340        perf-stat.i.iTLB-load-misses
    353772 ±  4%     -95.3%      16649        perf-stat.i.iTLB-loads
 1.226e+10            +9.6%  1.344e+10        perf-stat.i.instructions
      8575 ± 12%     +69.5%      14535 ±  3%  perf-stat.i.instructions-per-iTLB-miss
      1.00            +6.2%       1.06        perf-stat.i.ipc
    690868           +13.2%     781827        perf-stat.i.minor-faults
    690868           +13.2%     781827        perf-stat.i.page-faults
      1.98 ±  3%     -56.1%       0.87        perf-stat.overall.MPKI
      2.06 ± 18%      +3.8        5.89 ± 11%  perf-stat.overall.cache-miss-rate%
      1.00            -5.2%       0.95        perf-stat.overall.cpi
      0.28            +0.0        0.29        perf-stat.overall.dTLB-store-miss-rate%
     83.57           +15.1       98.67        perf-stat.overall.iTLB-load-miss-rate%
      6818 ±  6%     +59.9%      10905        perf-stat.overall.instructions-per-iTLB-miss
      1.00            +5.5%       1.05        perf-stat.overall.ipc
      3951            -3.2%       3826        perf-stat.overall.path-length
 3.771e+09           +10.4%  4.164e+09        perf-stat.ps.branch-instructions
    496503 ± 17%     +37.7%     683817 ±  9%  perf-stat.ps.cache-misses
  24145521 ±  3%     -51.8%   11629801        perf-stat.ps.cache-references
    102871 ±  3%     -83.4%      17045        perf-stat.ps.context-switches
 1.223e+10            +3.9%  1.271e+10        perf-stat.ps.cpu-cycles
      1986 ±  2%     -96.2%      75.93 ± 10%  perf-stat.ps.cpu-migrations
   3.2e+09            +9.8%  3.514e+09        perf-stat.ps.dTLB-loads
   2065804            +4.6%    2160396        perf-stat.ps.dTLB-store-misses
 7.304e+08            +2.0%  7.453e+08        perf-stat.ps.dTLB-stores
   1794803 ±  6%     -31.7%    1225789        perf-stat.ps.iTLB-load-misses
    351694 ±  4%     -95.3%      16561        perf-stat.ps.iTLB-loads
 1.219e+10            +9.7%  1.337e+10        perf-stat.ps.instructions
    686822           +13.2%     777664        perf-stat.ps.minor-faults
    686822           +13.2%     777664        perf-stat.ps.page-faults
 1.861e+12            -3.2%  1.801e+12        perf-stat.total.instructions
     56.68 ± 69%     -56.7        0.00        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
     56.68 ± 69%     -56.7        0.00        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
     56.32 ± 70%     -56.3        0.00        perf-profile.calltrace.cycles-pp.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
     56.32 ± 70%     -56.3        0.00        perf-profile.calltrace.cycles-pp.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
     56.32 ± 70%     -56.3        0.00        perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
     56.11 ± 71%     -56.1        0.00        perf-profile.calltrace.cycles-pp.mmput.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
     56.07 ± 71%     -56.1        0.00        perf-profile.calltrace.cycles-pp.exit_mmap.mmput.do_exit.do_group_exit.__x64_sys_exit_group
     36.23 ±101%     -36.2        0.00        perf-profile.calltrace.cycles-pp.secondary_startup_64
     35.47 ±105%     -35.5        0.00        perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
     35.47 ±105%     -35.5        0.00        perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
     35.47 ±105%     -35.5        0.00        perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     34.78 ±106%     -34.8        0.00        perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     34.75 ±106%     -34.7        0.00        perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
     29.00 ±110%     -29.0        0.00        perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     19.43 ± 97%     -19.4        0.00        perf-profile.calltrace.cycles-pp.free_pgtables.exit_mmap.mmput.do_exit.do_group_exit
     18.70 ± 59%     -18.7        0.00        perf-profile.calltrace.cycles-pp.remove_vma.exit_mmap.mmput.do_exit.do_group_exit
     14.99 ± 96%     -15.0        0.00        perf-profile.calltrace.cycles-pp.unlink_file_vma.free_pgtables.exit_mmap.mmput.do_exit
     14.52 ±108%     -14.5        0.00        perf-profile.calltrace.cycles-pp.unmap_vmas.exit_mmap.mmput.do_exit.do_group_exit
     12.42 ±108%     -12.4        0.00        perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.exit_mmap.mmput.do_exit
     10.42 ± 59%     -10.4        0.00        perf-profile.calltrace.cycles-pp.kmem_cache_free.remove_vma.exit_mmap.mmput.do_exit
      7.54 ± 95%      -7.5        0.00        perf-profile.calltrace.cycles-pp.__rb_erase_color.unlink_file_vma.free_pgtables.exit_mmap.mmput
      6.58 ± 61%      -6.6        0.00        perf-profile.calltrace.cycles-pp.__slab_free.kmem_cache_free.remove_vma.exit_mmap.mmput
      6.29 ±111%      -6.3        0.00        perf-profile.calltrace.cycles-pp.con_scroll.lf.vt_console_print.console_unlock.vprintk_emit
      6.29 ±111%      -6.3        0.00        perf-profile.calltrace.cycles-pp.fbcon_scroll.con_scroll.lf.vt_console_print.console_unlock
      6.29 ±111%      -6.3        0.00        perf-profile.calltrace.cycles-pp.fbcon_redraw.fbcon_scroll.con_scroll.lf.vt_console_print
      6.12 ±113%      -6.1        0.00        perf-profile.calltrace.cycles-pp.fbcon_putcs.fbcon_redraw.fbcon_scroll.con_scroll.lf
      6.12 ±113%      -6.1        0.00        perf-profile.calltrace.cycles-pp.bit_putcs.fbcon_putcs.fbcon_redraw.fbcon_scroll.con_scroll
      5.22 ±113%      -5.2        0.00        perf-profile.calltrace.cycles-pp.drm_fb_helper_cfb_imageblit.bit_putcs.fbcon_putcs.fbcon_redraw.fbcon_scroll
      5.22 ±113%      -5.2        0.00        perf-profile.calltrace.cycles-pp.cfb_imageblit.drm_fb_helper_cfb_imageblit.bit_putcs.fbcon_putcs.fbcon_redraw
      3.95 ±103%      -3.9        0.00        perf-profile.calltrace.cycles-pp.vprintk_emit.printk.irq_work_run_list.irq_work_run.smp_irq_work_interrupt
      3.95 ±103%      -3.9        0.00        perf-profile.calltrace.cycles-pp.console_unlock.vprintk_emit.printk.irq_work_run_list.irq_work_run
      3.95 ±103%      -3.9        0.00        perf-profile.calltrace.cycles-pp.vt_console_print.console_unlock.vprintk_emit.printk.irq_work_run_list
      3.95 ±103%      -3.9        0.00        perf-profile.calltrace.cycles-pp.lf.vt_console_print.console_unlock.vprintk_emit.printk
      3.95 ±103%      -3.9        0.00        perf-profile.calltrace.cycles-pp.printk.irq_work_run_list.irq_work_run.smp_irq_work_interrupt.irq_work_interrupt
      3.76 ± 95%      -3.8        0.00        perf-profile.calltrace.cycles-pp.vma_interval_tree_remove.unlink_file_vma.free_pgtables.exit_mmap.mmput
      3.00 ±101%      -3.0        0.00        perf-profile.calltrace.cycles-pp.down_write.unlink_file_vma.free_pgtables.exit_mmap.mmput
     61.00 ± 61%     -61.0        0.00        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     60.99 ± 61%     -61.0        0.00        perf-profile.children.cycles-pp.do_syscall_64
     56.33 ± 70%     -56.3        0.00        perf-profile.children.cycles-pp.do_group_exit
     56.33 ± 70%     -56.3        0.00        perf-profile.children.cycles-pp.do_exit
     56.32 ± 70%     -56.3        0.00        perf-profile.children.cycles-pp.__x64_sys_exit_group
     56.12 ± 71%     -56.1        0.00        perf-profile.children.cycles-pp.mmput
     56.12 ± 71%     -56.1        0.00        perf-profile.children.cycles-pp.exit_mmap
     36.23 ±101%     -36.2        0.00        perf-profile.children.cycles-pp.secondary_startup_64
     36.23 ±101%     -36.2        0.00        perf-profile.children.cycles-pp.cpu_startup_entry
     36.23 ±101%     -36.2        0.00        perf-profile.children.cycles-pp.do_idle
     35.48 ±102%     -35.5        0.00        perf-profile.children.cycles-pp.cpuidle_enter
     35.48 ±102%     -35.5        0.00        perf-profile.children.cycles-pp.cpuidle_enter_state
     35.47 ±105%     -35.5        0.00        perf-profile.children.cycles-pp.start_secondary
     29.21 ±110%     -29.2        0.00        perf-profile.children.cycles-pp.intel_idle
     19.51 ± 97%     -19.5        0.00        perf-profile.children.cycles-pp.free_pgtables
     19.14 ± 59%     -19.1        0.00        perf-profile.children.cycles-pp.remove_vma
     15.08 ± 96%     -15.1        0.00        perf-profile.children.cycles-pp.unlink_file_vma
     14.52 ±108%     -14.5        0.00        perf-profile.children.cycles-pp.unmap_vmas
     12.67 ±107%     -12.7        0.00        perf-profile.children.cycles-pp.unmap_page_range
     10.58 ± 58%     -10.6        0.00        perf-profile.children.cycles-pp.kmem_cache_free
      7.55 ± 95%      -7.6        0.00        perf-profile.children.cycles-pp.__rb_erase_color
      7.39 ±102%      -7.4        0.00        perf-profile.children.cycles-pp.console_unlock
      7.36 ±103%      -7.4        0.00        perf-profile.children.cycles-pp.vprintk_emit
      7.17 ±101%      -7.2        0.00        perf-profile.children.cycles-pp.vt_console_print
      6.99 ±103%      -7.0        0.00        perf-profile.children.cycles-pp.fbcon_putcs
      6.99 ±103%      -7.0        0.00        perf-profile.children.cycles-pp.bit_putcs
      6.75 ± 97%      -6.7        0.00        perf-profile.children.cycles-pp.lf
      6.75 ± 97%      -6.7        0.00        perf-profile.children.cycles-pp.con_scroll
      6.75 ± 97%      -6.7        0.00        perf-profile.children.cycles-pp.fbcon_scroll
      6.74 ± 98%      -6.7        0.00        perf-profile.children.cycles-pp.fbcon_redraw
      6.63 ± 60%      -6.6        0.00        perf-profile.children.cycles-pp.__slab_free
      5.98 ±105%      -6.0        0.00        perf-profile.children.cycles-pp.drm_fb_helper_cfb_imageblit
      5.98 ±105%      -6.0        0.00        perf-profile.children.cycles-pp.cfb_imageblit
      4.29 ± 87%      -4.3        0.00        perf-profile.children.cycles-pp.irq_work_interrupt
      4.29 ± 87%      -4.3        0.00        perf-profile.children.cycles-pp.smp_irq_work_interrupt
      4.29 ± 87%      -4.3        0.00        perf-profile.children.cycles-pp.irq_work_run_list
      4.29 ± 87%      -4.3        0.00        perf-profile.children.cycles-pp.irq_work_run
      4.29 ± 87%      -4.3        0.00        perf-profile.children.cycles-pp.printk
      4.05 ± 55%      -4.0        0.00        perf-profile.children.cycles-pp.___might_sleep
      3.82 ± 95%      -3.8        0.00        perf-profile.children.cycles-pp.vma_interval_tree_remove
      3.19 ± 98%      -3.2        0.00        perf-profile.children.cycles-pp.down_write
     29.21 ±110%     -29.2        0.00        perf-profile.self.cycles-pp.intel_idle
      7.47 ±108%      -7.5        0.00        perf-profile.self.cycles-pp.unmap_page_range
      6.47 ± 59%      -6.5        0.00        perf-profile.self.cycles-pp.__slab_free
      5.98 ±105%      -6.0        0.00        perf-profile.self.cycles-pp.cfb_imageblit
      5.36 ± 97%      -5.4        0.00        perf-profile.self.cycles-pp.__rb_erase_color
      3.97 ± 55%      -4.0        0.00        perf-profile.self.cycles-pp.___might_sleep
      3.73 ± 95%      -3.7        0.00        perf-profile.self.cycles-pp.vma_interval_tree_remove





Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.2.0-rc5-00073-g7d43f1ce9dd07" of type "text/plain" (196393 bytes)

View attachment "job-script" of type "text/plain" (7442 bytes)

View attachment "job.yaml" of type "text/plain" (5053 bytes)

View attachment "reproduce" of type "text/plain" (5055 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ