lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Date:   Thu, 28 Feb 2019 20:54:42 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     "Darrick J. Wong" <darrick.wong@...cle.com>
Cc:     LKML <linux-kernel@...r.kernel.org>,
        "Darrick J. Wong" <darrick.wong@...cle.com>, lkp@...org
Subject: [LKP] [xfs]  4ff1f5b3d9:  aim7.jobs-per-min -30.4% regression

Greeting,

FYI, we noticed a -30.4% regression of aim7.jobs-per-min due to commit:


commit: 4ff1f5b3d991dd50d49e7772952c53f947f833fb ("xfs: parallelize inode inactivation")
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git djwong-wtf

in testcase: aim7
on test machine: 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz with 384G memory
with following parameters:

	disk: 1BRD_48G
	fs: xfs
	test: disk_src
	load: 3000
	cpufreq_governor: performance

test-description: AIM7 is a traditional UNIX system level benchmark suite which is used to test and measure the performance of multiuser system.
test-url: https://sourceforge.net/projects/aimbench/files/aim-suite7/



Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/load/rootfs/tbox_group/test/testcase:
  gcc-7/performance/1BRD_48G/xfs/x86_64-rhel-7.2/3000/debian-x86_64-2018-04-03.cgz/lkp-ivb-ep01/disk_src/aim7

commit: 
  d21621840e ("xfs: force inactivation before fallocate when space is low")
  4ff1f5b3d9 ("xfs: parallelize inode inactivation")

d21621840e601d57 4ff1f5b3d991dd50d49e7772952 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          1:4            0%           1:4     dmesg.WARNING:at#for_ip_interrupt_entry/0x
           :4           25%           1:4     dmesg.WARNING:at_ip___perf_sw_event/0x
           :4          100%           4:4     kmsg.ACPI_Error:Method_parse/execution_failed~_SB.PMI0._GHL,AE_NOT_EXIST(#/psparse-#)
           :4          100%           4:4     kmsg.ACPI_Error:Method_parse/execution_failed~_SB.PMI0._PMC,AE_NOT_EXIST(#/psparse-#)
           :4          100%           4:4     kmsg.ACPI_Error:No_handler_for_Region[SYSI](#)[IPMI](#/evregion-#)
           :4          100%           4:4     kmsg.ACPI_Error:Region_IPMI(ID=#)has_no_handler(#/exfldio-#)
          2:4          -50%            :4     kmsg.XFS(ram#):xlog_verify_grant_tail:space>BBTOB(tail_blocks)
         %stddev     %change         %stddev
             \          |                \  
     14529           -30.4%      10116        aim7.jobs-per-min
      1239           +43.6%       1780        aim7.time.elapsed_time
      1239           +43.6%       1780        aim7.time.elapsed_time.max
      5548          +101.5%      11181 ±  3%  aim7.time.involuntary_context_switches
      3435           +48.7%       5108        aim7.time.system_time
    104.79           +14.0%     119.45        aim7.time.user_time
  27741617           +48.9%   41302729        aim7.time.voluntary_context_switches
     92.23            -1.2%      91.12        iostat.cpu.idle
      7.52           +15.2%       8.66 ±  2%  iostat.cpu.system
      7.51            +1.1        8.66 ±  2%  mpstat.cpu.sys%
      0.25            -0.0        0.21        mpstat.cpu.usr%
   3004482 ±  6%     -20.3%    2394055        numa-numastat.node1.local_node
   3006284 ±  6%     -20.2%    2398052        numa-numastat.node1.numa_hit
     92.00            -1.4%      90.75        vmstat.cpu.id
     12421           -25.6%       9239        vmstat.io.bo
   3949806           -71.5%    1126257        vmstat.memory.cache
     54607           +10.1%      60109        vmstat.system.cs
 1.072e+09           +24.6%  1.336e+09 ± 11%  cpuidle.C1.time
  12450429           +36.1%   16942123 ±  8%  cpuidle.C1.usage
 1.458e+09          +285.5%  5.621e+09 ± 52%  cpuidle.C1E.time
   9648938          +272.1%   35902807 ± 61%  cpuidle.C1E.usage
 3.099e+10 ±  6%     +34.4%  4.164e+10 ±  9%  cpuidle.C6.time
   1079386           +70.6%    1841065 ±  2%  cpuidle.POLL.time
     70217           +56.5%     109919        cpuidle.POLL.usage
   2976541           -94.9%     150749        meminfo.KReclaimable
  10894926           -70.8%    3186076        meminfo.Memused
   2976541           -94.9%     150749        meminfo.SReclaimable
   2189071           -90.3%     212524        meminfo.SUnreclaim
     34172           +17.5%      40138        meminfo.Shmem
   5165613           -93.0%     363274        meminfo.Slab
     16354           -86.5%       2214        meminfo.max_used_kB
      2759 ± 27%     +69.4%       4673 ± 31%  numa-vmstat.node0.nr_shmem
    373298           -94.5%      20349 ±  7%  numa-vmstat.node0.nr_slab_reclaimable
    278964 ±  4%     -90.3%      27152 ± 38%  numa-vmstat.node0.nr_slab_unreclaimable
     45797 ± 24%     +22.2%      55956 ± 18%  numa-vmstat.node1.nr_active_anon
     43827 ± 26%     +22.1%      53502 ± 19%  numa-vmstat.node1.nr_anon_pages
    370653           -95.3%      17336 ±  8%  numa-vmstat.node1.nr_slab_reclaimable
    268165 ±  4%     -90.3%      25976 ± 39%  numa-vmstat.node1.nr_slab_unreclaimable
     45797 ± 24%     +22.2%      55956 ± 18%  numa-vmstat.node1.nr_zone_active_anon
   1776154 ±  4%     -15.0%    1509746 ±  6%  numa-vmstat.node1.numa_hit
    159.67            +4.0%     166.00        turbostat.Avg_MHz
  12442054           +36.1%   16934916 ±  8%  turbostat.C1
   9648658          +272.1%   35902542 ± 61%  turbostat.C1E
      2.94            +4.9        7.85 ± 51%  turbostat.C1E%
      8.25 ± 15%     -63.4%       3.01 ± 85%  turbostat.CPU%c3
     56.33 ±  2%     -13.0%      49.00        turbostat.CoreTmp
 1.008e+08           +42.9%  1.441e+08        turbostat.IRQ
      0.67 ±  9%     -56.9%       0.29 ± 11%  turbostat.Pkg%pc2
     56.67 ±  2%     -13.5%      49.00        turbostat.PkgTmp
     99306           +43.5%     142520        turbostat.SMI
   1493417           -94.5%      81402 ±  7%  numa-meminfo.node0.KReclaimable
   5538082           -70.1%    1658246 ± 13%  numa-meminfo.node0.MemUsed
   1493417           -94.5%      81402 ±  7%  numa-meminfo.node0.SReclaimable
   1116040 ±  4%     -90.3%     108602 ± 38%  numa-meminfo.node0.SUnreclaim
     11039 ± 27%     +69.4%      18697 ± 31%  numa-meminfo.node0.Shmem
   2609458           -92.7%     190004 ± 21%  numa-meminfo.node0.Slab
    183176 ± 24%     +22.2%     223893 ± 18%  numa-meminfo.node1.Active
    183172 ± 24%     +22.2%     223825 ± 18%  numa-meminfo.node1.Active(anon)
    175292 ± 26%     +22.1%     214007 ± 19%  numa-meminfo.node1.AnonPages
   1482818           -95.3%      69351 ±  8%  numa-meminfo.node1.KReclaimable
   5356088           -71.5%    1527854 ± 13%  numa-meminfo.node1.MemUsed
   1482818           -95.3%      69351 ±  8%  numa-meminfo.node1.SReclaimable
   1072807 ±  4%     -90.3%     103903 ± 39%  numa-meminfo.node1.SUnreclaim
   2555625           -93.2%     173254 ± 24%  numa-meminfo.node1.Slab
    113079            +3.9%     117521        proc-vmstat.nr_active_anon
    109959            +2.7%     112943        proc-vmstat.nr_anon_pages
     98.00            +6.1%     104.00        proc-vmstat.nr_anon_transparent_hugepages
    354.67 ±  2%      +3.4%     366.75        proc-vmstat.nr_dirtied
   9577429            +2.0%    9769871        proc-vmstat.nr_dirty_background_threshold
  19178276            +2.0%   19563630        proc-vmstat.nr_dirty_threshold
  96329054            +2.0%   98256289        proc-vmstat.nr_free_pages
      5250            +5.5%       5541        proc-vmstat.nr_inactive_anon
    340.00            +1.5%     345.00        proc-vmstat.nr_inactive_file
      8543           +17.5%      10034        proc-vmstat.nr_shmem
    744139           -94.9%      37686        proc-vmstat.nr_slab_reclaimable
    547263           -90.3%      53129        proc-vmstat.nr_slab_unreclaimable
    353.67 ±  2%      +3.6%     366.25        proc-vmstat.nr_written
    113079            +3.9%     117521        proc-vmstat.nr_zone_active_anon
      5250            +5.5%       5541        proc-vmstat.nr_zone_inactive_anon
    340.00            +1.5%     345.00        proc-vmstat.nr_zone_inactive_file
      2284 ±  8%    +166.9%       6098 ±  2%  proc-vmstat.numa_hint_faults_local
   5647278           -16.5%    4715166        proc-vmstat.numa_hit
   5639393           -16.5%    4707260        proc-vmstat.numa_local
     27823 ± 31%     -33.6%      18466 ±  4%  proc-vmstat.numa_pages_migrated
      1036 ±  8%     +30.9%       1356 ±  8%  proc-vmstat.pgactivate
   8609506           -37.7%    5365970        proc-vmstat.pgalloc_normal
   3296783           +40.3%    4625166        proc-vmstat.pgfault
   4012655           +22.4%    4913301        proc-vmstat.pgfree
     27823 ± 31%     -33.6%      18466 ±  4%  proc-vmstat.pgmigrate_success
  15372540 ±  2%      +7.1%   16457304        proc-vmstat.pgpgout
     25.92           -33.7%      17.19 ±  3%  perf-stat.i.MPKI
 7.471e+08 ±  6%     +27.6%   9.53e+08        perf-stat.i.branch-instructions
  30969748 ±  2%     +23.0%   38088600 ±  2%  perf-stat.i.branch-misses
     14.89 ±  8%      -3.6       11.33        perf-stat.i.cache-miss-rate%
  13934723 ± 15%     -34.8%    9086049 ±  2%  perf-stat.i.cache-misses
  91416041 ±  5%     -11.8%   80671348 ±  2%  perf-stat.i.cache-references
     54668           +10.0%      60134        perf-stat.i.context-switches
      2.03 ±  2%     -27.0%       1.48 ±  2%  perf-stat.i.cpi
    483.98 ±  3%    +168.0%       1297 ±  2%  perf-stat.i.cpu-migrations
    560.95 ±  6%     +36.1%     763.38        perf-stat.i.cycles-between-cache-misses
      0.78 ±  4%      -0.3        0.44 ±  7%  perf-stat.i.dTLB-load-miss-rate%
   7006483 ±  7%     -19.6%    5631048 ±  6%  perf-stat.i.dTLB-load-misses
 9.075e+08 ±  7%     +41.7%  1.286e+09        perf-stat.i.dTLB-loads
      0.11 ±  3%      -0.0        0.07 ±  9%  perf-stat.i.dTLB-store-miss-rate%
    682889 ±  2%     -26.8%     499664 ±  7%  perf-stat.i.dTLB-store-misses
 6.047e+08 ±  6%     +24.7%  7.542e+08 ±  7%  perf-stat.i.dTLB-stores
     50.96 ±  5%      +6.9       57.84        perf-stat.i.iTLB-load-miss-rate%
   1725697           +16.0%    2001978        perf-stat.i.iTLB-load-misses
 3.605e+09 ±  6%     +32.8%  4.786e+09        perf-stat.i.instructions
      2080 ±  4%     +14.9%       2389 ±  2%  perf-stat.i.instructions-per-iTLB-miss
      0.50 ±  2%     +36.6%       0.69 ±  2%  perf-stat.i.ipc
      2598            -2.1%       2542        perf-stat.i.minor-faults
     43.11 ±  2%      +2.5       45.60        perf-stat.i.node-load-miss-rate%
   5915330 ± 36%     -53.1%    2776569        perf-stat.i.node-loads
     39.85            +1.8       41.64        perf-stat.i.node-store-miss-rate%
   1498790 ±  6%     -10.5%    1341835 ±  2%  perf-stat.i.node-store-misses
   2255272 ±  7%     -17.0%    1872796        perf-stat.i.node-stores
      2598            -2.1%       2542        perf-stat.i.page-faults
     25.38           -33.6%      16.86 ±  2%  perf-stat.overall.MPKI
     15.15 ± 10%      -3.9       11.26        perf-stat.overall.cache-miss-rate%
      1.97 ±  3%     -26.4%       1.45 ±  2%  perf-stat.overall.cpi
    518.24 ± 12%     +47.2%     762.76        perf-stat.overall.cycles-between-cache-misses
      0.77 ±  4%      -0.3        0.44 ±  7%  perf-stat.overall.dTLB-load-miss-rate%
      0.11 ±  3%      -0.0        0.07 ±  9%  perf-stat.overall.dTLB-store-miss-rate%
     50.39 ±  5%      +7.2       57.62        perf-stat.overall.iTLB-load-miss-rate%
      2087 ±  5%     +14.6%       2390 ±  2%  perf-stat.overall.instructions-per-iTLB-miss
      0.51 ±  3%     +35.9%       0.69 ±  2%  perf-stat.overall.ipc
     40.95 ±  6%      +4.6       45.59        perf-stat.overall.node-load-miss-rate%
     39.94            +1.8       41.74        perf-stat.overall.node-store-miss-rate%
 7.465e+08 ±  6%     +27.6%  9.525e+08        perf-stat.ps.branch-instructions
  30944263 ±  2%     +23.0%   38067137 ±  2%  perf-stat.ps.branch-misses
  13922480 ± 15%     -34.8%    9080937 ±  2%  perf-stat.ps.cache-misses
  91340090 ±  5%     -11.7%   80625818 ±  2%  perf-stat.ps.cache-references
     54623           +10.0%      60100        perf-stat.ps.context-switches
    483.58 ±  3%    +168.1%       1296 ±  2%  perf-stat.ps.cpu-migrations
   7000707 ±  7%     -19.6%    5627875 ±  6%  perf-stat.ps.dTLB-load-misses
 9.067e+08 ±  7%     +41.8%  1.286e+09        perf-stat.ps.dTLB-loads
    682332 ±  2%     -26.8%     499382 ±  7%  perf-stat.ps.dTLB-store-misses
 6.042e+08 ±  6%     +24.8%  7.538e+08 ±  7%  perf-stat.ps.dTLB-stores
   1724289           +16.0%    2000848        perf-stat.ps.iTLB-load-misses
 3.602e+09 ±  6%     +32.8%  4.783e+09        perf-stat.ps.instructions
      2596            -2.1%       2541        perf-stat.ps.minor-faults
   5909580 ± 36%     -53.0%    2775010        perf-stat.ps.node-loads
   1497558 ±  6%     -10.4%    1341077 ±  2%  perf-stat.ps.node-store-misses
   2253406 ±  7%     -16.9%    1871742        perf-stat.ps.node-stores
      2596            -2.1%       2541        perf-stat.ps.page-faults
 4.467e+12 ±  6%     +90.7%  8.517e+12        perf-stat.total.instructions
     51255           +64.5%      84302        sched_debug.cfs_rq:/.exec_clock.avg
    209012           -57.0%      89792 ±  2%  sched_debug.cfs_rq:/.exec_clock.max
     41408           +95.1%      80786        sched_debug.cfs_rq:/.exec_clock.min
     25327           -90.5%       2417 ± 12%  sched_debug.cfs_rq:/.exec_clock.stddev
      2420 ±  8%     -40.9%       1429 ± 34%  sched_debug.cfs_rq:/.load_avg.max
      0.25 ± 61%     -73.8%       0.07 ±173%  sched_debug.cfs_rq:/.load_avg.min
    412.79 ±  4%     -35.4%     266.72 ± 31%  sched_debug.cfs_rq:/.load_avg.stddev
    216580           +42.3%     308167        sched_debug.cfs_rq:/.min_vruntime.avg
    360515 ± 11%     +34.9%     486486 ±  5%  sched_debug.cfs_rq:/.min_vruntime.max
    186432           +53.7%     286556        sched_debug.cfs_rq:/.min_vruntime.min
      0.14 ±  2%     +11.2%       0.15 ±  5%  sched_debug.cfs_rq:/.nr_running.avg
     14.77 ±  8%     +26.0%      18.61 ± 18%  sched_debug.cfs_rq:/.util_est_enqueued.avg
    611829 ±  2%     +10.4%     675598        sched_debug.cpu.avg_idle.avg
    164110 ±  7%     +19.1%     195481 ±  9%  sched_debug.cpu.avg_idle.min
    236480           -11.0%     210366 ±  2%  sched_debug.cpu.avg_idle.stddev
    635922           +42.7%     907749        sched_debug.cpu.clock.avg
    635962           +42.7%     907796        sched_debug.cpu.clock.max
    635881           +42.7%     907703        sched_debug.cpu.clock.min
     23.70           +15.9%      27.46 ±  9%  sched_debug.cpu.clock.stddev
    635922           +42.7%     907749        sched_debug.cpu.clock_task.avg
    635962           +42.7%     907796        sched_debug.cpu.clock_task.max
    635881           +42.7%     907703        sched_debug.cpu.clock_task.min
     23.70           +15.9%      27.46 ±  9%  sched_debug.cpu.clock_task.stddev
    668.98 ± 37%     -57.0%     287.96 ± 23%  sched_debug.cpu.cpu_load[1].max
    753.68 ± 40%     -62.3%     284.23 ± 21%  sched_debug.cpu.cpu_load[2].max
    121.44 ± 39%     -56.6%      52.70 ± 18%  sched_debug.cpu.cpu_load[2].stddev
     23.72 ± 25%     -34.5%      15.54 ± 13%  sched_debug.cpu.cpu_load[3].avg
    708.89 ± 36%     -62.4%     266.19 ± 16%  sched_debug.cpu.cpu_load[3].max
    114.92 ± 34%     -56.5%      49.97 ± 14%  sched_debug.cpu.cpu_load[3].stddev
     21.19 ± 15%     -30.5%      14.73 ± 14%  sched_debug.cpu.cpu_load[4].avg
    626.62 ± 23%     -60.2%     249.32 ± 14%  sched_debug.cpu.cpu_load[4].max
    101.81 ± 22%     -54.0%      46.80 ± 14%  sched_debug.cpu.cpu_load[4].stddev
    780.25 ±  2%     +32.5%       1034 ±  5%  sched_debug.cpu.curr->pid.avg
     19307           +34.1%      25896        sched_debug.cpu.curr->pid.max
      3186           +37.5%       4380        sched_debug.cpu.curr->pid.stddev
    608234           +44.5%     878900        sched_debug.cpu.nr_load_updates.avg
    615952           +43.8%     885845        sched_debug.cpu.nr_load_updates.max
    605651           +44.7%     876407        sched_debug.cpu.nr_load_updates.min
    839549           +55.0%    1301467        sched_debug.cpu.nr_switches.avg
   4512683           -69.5%    1377858        sched_debug.cpu.nr_switches.max
    622663 ±  2%     +97.4%    1228956        sched_debug.cpu.nr_switches.min
    588922           -90.6%      55523 ±  6%  sched_debug.cpu.nr_switches.stddev
    163.90 ±  4%     +20.2%     196.95 ±  5%  sched_debug.cpu.nr_uninterruptible.max
    837970           +55.1%    1299788        sched_debug.cpu.sched_count.avg
   4511470           -69.5%    1376198        sched_debug.cpu.sched_count.max
    621018 ±  2%     +97.6%    1227438        sched_debug.cpu.sched_count.min
    588959           -90.6%      55619 ±  7%  sched_debug.cpu.sched_count.stddev
    405140           +57.7%     638835        sched_debug.cpu.sched_goidle.avg
   2057532           -67.1%     677064        sched_debug.cpu.sched_goidle.max
    304696           +98.0%     603265        sched_debug.cpu.sched_goidle.min
    264975           -89.4%      28187 ±  8%  sched_debug.cpu.sched_goidle.stddev
    419432           +55.2%     651053        sched_debug.cpu.ttwu_count.avg
   2222748           -68.8%     692406        sched_debug.cpu.ttwu_count.max
    322383 ±  2%     +90.2%     613211        sched_debug.cpu.ttwu_count.min
    289153           -89.7%      29888 ± 10%  sched_debug.cpu.ttwu_count.stddev
     13408           +16.4%      15611        sched_debug.cpu.ttwu_local.avg
     30957 ±  3%     -28.7%      22087 ±  4%  sched_debug.cpu.ttwu_local.max
      6697 ± 15%     +57.8%      10567 ±  5%  sched_debug.cpu.ttwu_local.min
      3892 ±  2%     -32.0%       2648 ± 10%  sched_debug.cpu.ttwu_local.stddev
    635879           +42.7%     907700        sched_debug.cpu_clk
    633439           +42.9%     905254        sched_debug.ktime
    636354           +42.7%     908173        sched_debug.sched_clk
      1549           +45.0%       2246        slabinfo.Acpi-ParseExt.active_objs
      1549           +45.0%       2246        slabinfo.Acpi-ParseExt.num_objs
    269.67 ±  6%     +26.1%     340.00 ± 15%  slabinfo.biovec-128.active_objs
    269.67 ±  6%     +26.1%     340.00 ± 15%  slabinfo.biovec-128.num_objs
    272.00           +32.3%     359.75 ±  8%  slabinfo.buffer_head.active_objs
    272.00           +32.3%     359.75 ±  8%  slabinfo.buffer_head.num_objs
   3018310           -98.6%      40945        slabinfo.dmaengine-unmap-16.active_objs
     72008           -98.3%       1211        slabinfo.dmaengine-unmap-16.active_slabs
   3024351           -98.3%      50889        slabinfo.dmaengine-unmap-16.num_objs
     72008           -98.3%       1211        slabinfo.dmaengine-unmap-16.num_slabs
     16983 ±  2%     -71.8%       4785        slabinfo.kmalloc-128.active_objs
    532.33 ±  2%     -72.0%     149.25        slabinfo.kmalloc-128.active_slabs
     17058 ±  2%     -71.9%       4785        slabinfo.kmalloc-128.num_objs
    532.33 ±  2%     -72.0%     149.25        slabinfo.kmalloc-128.num_slabs
   3024970           -96.7%     100606        slabinfo.kmalloc-16.active_objs
     11827           -96.6%     396.25        slabinfo.kmalloc-16.active_slabs
   3028004           -96.6%     101626        slabinfo.kmalloc-16.num_objs
     11827           -96.6%     396.25        slabinfo.kmalloc-16.num_slabs
      8063           -33.1%       5397 ±  2%  slabinfo.kmalloc-1k.active_objs
      8178           -34.0%       5399 ±  2%  slabinfo.kmalloc-1k.num_objs
      4370           -13.2%       3793        slabinfo.kmalloc-2k.active_objs
      4441           -13.1%       3858        slabinfo.kmalloc-2k.num_objs
    108697           -82.1%      19496 ± 11%  slabinfo.kmalloc-32.active_objs
    849.67           -81.9%     153.75 ± 10%  slabinfo.kmalloc-32.active_slabs
    108850           -81.9%      19730 ± 10%  slabinfo.kmalloc-32.num_objs
    849.67           -81.9%     153.75 ± 10%  slabinfo.kmalloc-32.num_slabs
   2664947           -98.6%      36570        slabinfo.kmalloc-512.active_objs
     83752           -98.2%       1471        slabinfo.kmalloc-512.active_slabs
   2680087           -98.2%      47097        slabinfo.kmalloc-512.num_objs
     83752           -98.2%       1471        slabinfo.kmalloc-512.num_slabs
      1205           -22.7%     931.50        slabinfo.kmalloc-8k.active_objs
      1411           -25.3%       1054        slabinfo.kmalloc-8k.num_objs
     10001 ±  2%     +10.5%      11055 ±  2%  slabinfo.kmalloc-96.active_objs
     10508           +10.0%      11556 ±  2%  slabinfo.kmalloc-96.num_objs
     95480           -95.4%       4387        slabinfo.mnt_cache.active_objs
      2276           -95.4%     104.00        slabinfo.mnt_cache.active_slabs
     95629           -95.4%       4387        slabinfo.mnt_cache.num_objs
      2276           -95.4%     104.00        slabinfo.mnt_cache.num_slabs
    528.00 ±  2%     +27.1%     671.25 ±  6%  slabinfo.nfs_commit_data.active_objs
    528.00 ±  2%     +27.1%     671.25 ±  6%  slabinfo.nfs_commit_data.num_objs
    410.00 ±  9%     +45.5%     596.75 ±  4%  slabinfo.nfs_read_data.active_objs
    410.00 ±  9%     +45.5%     596.75 ±  4%  slabinfo.nfs_read_data.num_objs
     69156           -75.7%      16774        slabinfo.radix_tree_node.active_objs
      2472           -75.7%     601.50        slabinfo.radix_tree_node.active_slabs
     69238           -75.7%      16850        slabinfo.radix_tree_node.num_objs
      2472           -75.7%     601.50        slabinfo.radix_tree_node.num_slabs
    429.67 ± 16%     +18.8%     510.25 ± 15%  slabinfo.skbuff_ext_cache.active_objs
    429.67 ± 16%     +18.8%     510.25 ± 15%  slabinfo.skbuff_ext_cache.num_objs
    540.33 ± 12%     +31.3%     709.25 ± 18%  slabinfo.skbuff_fclone_cache.active_objs
    540.33 ± 12%     +31.3%     709.25 ± 18%  slabinfo.skbuff_fclone_cache.num_objs
      9615           -34.8%       6268        slabinfo.xfs_buf_item.active_objs
    325.00           -34.8%     212.00        slabinfo.xfs_buf_item.active_slabs
      9760           -34.6%       6378        slabinfo.xfs_buf_item.num_objs
    325.00           -34.8%     212.00        slabinfo.xfs_buf_item.num_slabs
     10325 ±  5%     -75.8%       2503 ±  6%  slabinfo.xfs_efd_item.active_objs
    280.33 ±  5%     -76.1%      67.00 ±  7%  slabinfo.xfs_efd_item.active_slabs
     10394 ±  5%     -75.9%       2503 ±  6%  slabinfo.xfs_efd_item.num_objs
    280.33 ±  5%     -76.1%      67.00 ±  7%  slabinfo.xfs_efd_item.num_slabs
   3018096           -98.7%      40718        slabinfo.xfs_inode.active_objs
     88945           -98.3%       1490        slabinfo.xfs_inode.active_slabs
   3024161           -98.3%      50678        slabinfo.xfs_inode.num_objs
     88945           -98.3%       1490        slabinfo.xfs_inode.num_slabs
     69.15 ±  2%      -5.9       63.20        perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
     69.15 ±  2%      -5.9       63.20        perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
     69.07 ±  2%      -5.9       63.16        perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     62.33 ±  3%      -5.5       56.87        perf-profile.calltrace.cycles-pp.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     70.98 ±  3%      -5.0       65.99        perf-profile.calltrace.cycles-pp.secondary_startup_64
     51.64 ±  3%      -2.5       49.13        perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary
      9.20 ±  4%      -2.4        6.84 ±  2%  perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary
      8.71 ±  4%      -2.3        6.41 ±  3%  perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry
      3.03 ±  4%      -1.0        2.04 ± 14%  perf-profile.calltrace.cycles-pp.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle
      4.53 ±  9%      -0.9        3.58 ±  7%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle
      2.88 ±  4%      -0.9        1.99 ± 11%  perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      2.55 ±  3%      -0.8        1.70 ± 19%  perf-profile.calltrace.cycles-pp.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
      1.07 ±  7%      -0.7        0.40 ± 57%  perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry.start_secondary
      3.07 ±  8%      -0.6        2.44 ±  5%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
      0.80 ±  3%      -0.5        0.31 ±101%  perf-profile.calltrace.cycles-pp.irq_enter.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle
      0.80 ±  7%      -0.5        0.31 ±102%  perf-profile.calltrace.cycles-pp.run_rebalance_domains.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt
      0.77 ±  9%      -0.5        0.30 ±102%  perf-profile.calltrace.cycles-pp.update_blocked_averages.run_rebalance_domains.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt
      0.68 ±  3%      -0.4        0.30 ±100%  perf-profile.calltrace.cycles-pp.rebalance_domains.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt
      1.60 ±  9%      -0.3        1.32 ±  5%  perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      1.05 ±  9%      -0.3        0.78 ±  7%  perf-profile.calltrace.cycles-pp.xfs_dir2_node_removename.xfs_dir_removename.xfs_remove.xfs_vn_unlink.vfs_unlink
      1.04 ±  7%      -0.3        0.79 ±  7%  perf-profile.calltrace.cycles-pp.xfs_dir2_block_addname.xfs_dir_createname.xfs_create.xfs_generic_create.path_openat
      0.83 ±  8%      -0.2        0.59 ±  9%  perf-profile.calltrace.cycles-pp.__xfs_dir3_data_check.xfs_dir3_data_check.xfs_dir2_block_addname.xfs_dir_createname.xfs_create
      0.84 ±  9%      -0.2        0.61 ± 10%  perf-profile.calltrace.cycles-pp.xfs_dir3_data_check.xfs_dir2_block_addname.xfs_dir_createname.xfs_create.xfs_generic_create
      1.22 ±  3%      -0.2        0.99 ±  5%  perf-profile.calltrace.cycles-pp.down_write.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.86 ±  7%      -0.2        0.63 ±  9%  perf-profile.calltrace.cycles-pp.rwsem_spin_on_owner.rwsem_down_write_failed.call_rwsem_down_write_failed.down_write.do_unlinkat
      1.20 ±  3%      -0.2        0.98 ±  5%  perf-profile.calltrace.cycles-pp.rwsem_down_write_failed.call_rwsem_down_write_failed.down_write.do_unlinkat.do_syscall_64
      1.20 ±  3%      -0.2        0.98 ±  5%  perf-profile.calltrace.cycles-pp.call_rwsem_down_write_failed.down_write.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.75 ±  3%      -0.2        1.54 ±  9%  perf-profile.calltrace.cycles-pp.xfs_dir2_block_removename.xfs_dir_removename.xfs_remove.xfs_vn_unlink.vfs_unlink
      1.29 ±  9%      -0.2        1.08 ±  6%  perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt
      1.13 ±  7%      -0.1        0.99 ±  4%  perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
      0.57 ±  5%      +0.1        0.69 ± 10%  perf-profile.calltrace.cycles-pp.xfs_trans_free_items.xfs_log_commit_cil.__xfs_trans_commit.xfs_remove.xfs_vn_unlink
      1.10 ±  8%      +0.2        1.25 ±  8%  perf-profile.calltrace.cycles-pp.xfs_iunlink.xfs_remove.xfs_vn_unlink.vfs_unlink.do_unlinkat
      0.67 ± 17%      +0.3        0.92 ±  4%  perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary
      0.68 ± 15%      +0.3        0.95 ±  4%  perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      0.37 ± 71%      +0.3        0.63 ±  5%  perf-profile.calltrace.cycles-pp.xfs_trans_free_items.xfs_log_commit_cil.__xfs_trans_commit.xfs_create.xfs_generic_create
      6.59 ±  3%      +0.3        6.88 ±  2%  perf-profile.calltrace.cycles-pp.xfs_remove.xfs_vn_unlink.vfs_unlink.do_unlinkat.do_syscall_64
      6.60 ±  3%      +0.3        6.89 ±  2%  perf-profile.calltrace.cycles-pp.xfs_vn_unlink.vfs_unlink.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.77 ± 14%      +0.3        1.08 ±  5%  perf-profile.calltrace.cycles-pp.__save_stack_trace.save_stack_trace_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair
      6.77 ±  3%      +0.3        7.09 ±  2%  perf-profile.calltrace.cycles-pp.vfs_unlink.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.79 ± 14%      +0.3        1.11 ±  5%  perf-profile.calltrace.cycles-pp.save_stack_trace_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.ttwu_do_activate
      0.69 ±  9%      +0.3        1.03 ±  4%  perf-profile.calltrace.cycles-pp.xfs_btree_check_sblock.xfs_btree_get_rec.xfs_inobt_get_rec.xfs_check_agi_freecount.xfs_dialloc_ag
      1.84 ±  4%      +0.4        2.20 ± 22%  perf-profile.calltrace.cycles-pp.rwsem_down_write_failed.call_rwsem_down_write_failed.down_write.path_openat.do_filp_open
      0.89 ± 17%      +0.4        1.25 ±  6%  perf-profile.calltrace.cycles-pp.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.ttwu_do_activate.sched_ttwu_pending
      1.86 ±  4%      +0.4        2.22 ± 22%  perf-profile.calltrace.cycles-pp.down_write.path_openat.do_filp_open.do_sys_open.do_syscall_64
      1.85 ±  4%      +0.4        2.21 ± 22%  perf-profile.calltrace.cycles-pp.call_rwsem_down_write_failed.down_write.path_openat.do_filp_open.do_sys_open
      0.70 ±  6%      +0.4        1.08 ±  3%  perf-profile.calltrace.cycles-pp.xfs_btree_check_sblock.xfs_btree_increment.xfs_check_agi_freecount.xfs_dialloc_ag.xfs_dialloc
      0.18 ±141%      +0.4        0.56 ±  8%  perf-profile.calltrace.cycles-pp.xfs_buf_item_unlock.xfs_trans_free_items.xfs_log_commit_cil.__xfs_trans_commit.xfs_remove
      0.43 ± 75%      +0.4        0.82 ±  9%  perf-profile.calltrace.cycles-pp.xfs_iunlink_remove.xfs_ifree.xfs_inactive_ifree.xfs_inactive.xfs_inactive_inode
      0.18 ±141%      +0.4        0.58 ±  8%  perf-profile.calltrace.cycles-pp.xfs_trans_read_buf_map.xfs_read_agi.xfs_iunlink.xfs_remove.xfs_vn_unlink
      0.51 ± 74%      +0.4        0.93 ±  9%  perf-profile.calltrace.cycles-pp.xfs_btree_check_sblock.xfs_btree_increment.xfs_check_agi_freecount.xfs_difree_inobt.xfs_difree
      0.37 ± 70%      +0.4        0.80 ±  7%  perf-profile.calltrace.cycles-pp.__xfs_btree_check_sblock.xfs_btree_check_sblock.xfs_btree_increment.xfs_check_agi_freecount.xfs_dialloc_ag
      0.17 ±141%      +0.5        0.62 ±  5%  perf-profile.calltrace.cycles-pp.xfs_ialloc_read_agi.xfs_dialloc.xfs_ialloc.xfs_dir_ialloc.xfs_create
      1.11 ± 15%      +0.5        1.57 ±  4%  perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.ttwu_do_activate.sched_ttwu_pending.do_idle
      0.22 ±141%      +0.5        0.68 ± 16%  perf-profile.calltrace.cycles-pp.__xfs_btree_check_sblock.xfs_btree_check_sblock.xfs_btree_increment.xfs_check_agi_freecount.xfs_difree_inobt
      1.15 ± 13%      +0.5        1.64 ±  5%  perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.sched_ttwu_pending.do_idle.cpu_startup_entry
      1.17 ± 13%      +0.5        1.66 ±  5%  perf-profile.calltrace.cycles-pp.ttwu_do_activate.sched_ttwu_pending.do_idle.cpu_startup_entry.start_secondary
      1.02 ±  2%      +0.5        1.56 ±  4%  perf-profile.calltrace.cycles-pp.xfs_btree_increment.xfs_check_agi_freecount.xfs_dialloc_ag.xfs_dialloc.xfs_ialloc
      1.30 ± 11%      +0.5        1.85 ±  5%  perf-profile.calltrace.cycles-pp.sched_ttwu_pending.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      0.00            +0.6        0.55 ±  6%  perf-profile.calltrace.cycles-pp.xfs_trans_read_buf_map.xfs_read_agi.xfs_ialloc_read_agi.xfs_dialloc.xfs_ialloc
      1.00 ±  7%      +0.6        1.57 ±  6%  perf-profile.calltrace.cycles-pp.xfs_btree_get_rec.xfs_inobt_get_rec.xfs_check_agi_freecount.xfs_dialloc_ag.xfs_dialloc
      0.00            +0.6        0.58 ±  6%  perf-profile.calltrace.cycles-pp.xfs_read_agi.xfs_ialloc_read_agi.xfs_dialloc.xfs_ialloc.xfs_dir_ialloc
      0.00            +0.6        0.64 ±  9%  perf-profile.calltrace.cycles-pp.unwind_next_frame.__save_stack_trace.save_stack_trace_tsk.__account_scheduler_latency.enqueue_entity
      0.19 ±141%      +0.7        0.85 ± 25%  perf-profile.calltrace.cycles-pp.xfs_dir2_leaf_removename.xfs_dir_removename.xfs_remove.xfs_vn_unlink.vfs_unlink
      0.59 ± 79%      +0.7        1.33 ±  4%  perf-profile.calltrace.cycles-pp.xfs_log_commit_cil.__xfs_trans_commit.xfs_inactive_ifree.xfs_inactive.xfs_inactive_inode
      0.60 ± 79%      +0.8        1.39 ±  4%  perf-profile.calltrace.cycles-pp.__xfs_trans_commit.xfs_inactive_ifree.xfs_inactive.xfs_inactive_inode.xfs_reclaim_inodes_pag
      1.65 ±  8%      +0.8        2.48 ±  5%  perf-profile.calltrace.cycles-pp.xfs_inobt_get_rec.xfs_check_agi_freecount.xfs_dialloc_ag.xfs_dialloc.xfs_ialloc
      0.46 ± 74%      +1.1        1.52 ±  6%  perf-profile.calltrace.cycles-pp.__xfs_btree_check_sblock.xfs_btree_check_sblock.xfs_btree_get_rec.xfs_inobt_get_rec.xfs_check_agi_freecount
      3.73 ±  4%      +1.4        5.15 ±  2%  perf-profile.calltrace.cycles-pp.xfs_dialloc_ag.xfs_dialloc.xfs_ialloc.xfs_dir_ialloc.xfs_create
      8.77 ±  3%      +1.4       10.20        perf-profile.calltrace.cycles-pp.xfs_generic_create.path_openat.do_filp_open.do_sys_open.do_syscall_64
      8.57 ±  3%      +1.4       10.01        perf-profile.calltrace.cycles-pp.xfs_create.xfs_generic_create.path_openat.do_filp_open.do_sys_open
      4.82 ±  4%      +1.5        6.30 ±  2%  perf-profile.calltrace.cycles-pp.xfs_dir_ialloc.xfs_create.xfs_generic_create.path_openat.do_filp_open
      4.79 ±  4%      +1.5        6.29 ±  2%  perf-profile.calltrace.cycles-pp.xfs_ialloc.xfs_dir_ialloc.xfs_create.xfs_generic_create.path_openat
      3.09 ±  4%      +1.5        4.61 ±  3%  perf-profile.calltrace.cycles-pp.xfs_check_agi_freecount.xfs_dialloc_ag.xfs_dialloc.xfs_ialloc.xfs_dir_ialloc
      4.33 ±  3%      +1.5        5.85 ±  2%  perf-profile.calltrace.cycles-pp.xfs_dialloc.xfs_ialloc.xfs_dir_ialloc.xfs_create.xfs_generic_create
      2.36 ± 75%      +1.7        4.08 ±  7%  perf-profile.calltrace.cycles-pp.xfs_difree_inobt.xfs_difree.xfs_ifree.xfs_inactive_ifree.xfs_inactive
     13.23 ±  2%      +1.8       15.04 ±  2%  perf-profile.calltrace.cycles-pp.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.49 ± 75%      +1.8        4.31 ±  6%  perf-profile.calltrace.cycles-pp.xfs_difree.xfs_ifree.xfs_inactive_ifree.xfs_inactive.xfs_inactive_inode
     13.07 ±  2%      +1.8       14.89 ±  2%  perf-profile.calltrace.cycles-pp.path_openat.do_filp_open.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
     13.08 ±  2%      +1.8       14.91 ±  2%  perf-profile.calltrace.cycles-pp.do_filp_open.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
     23.05            +2.0       25.04        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
     23.07            +2.0       25.07        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
      2.94 ± 75%      +2.2        5.19 ±  4%  perf-profile.calltrace.cycles-pp.xfs_ifree.xfs_inactive_ifree.xfs_inactive.xfs_inactive_inode.xfs_reclaim_inodes_pag
      4.48 ± 59%      +3.0        7.48 ±  3%  perf-profile.calltrace.cycles-pp.ret_from_fork
      4.47 ± 59%      +3.0        7.48 ±  3%  perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
      4.25 ± 61%      +3.0        7.28 ±  3%  perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork
      4.23 ± 61%      +3.0        7.27 ±  3%  perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork
      3.68 ± 76%      +3.1        6.81 ±  2%  perf-profile.calltrace.cycles-pp.xfs_inactive_worker.process_one_work.worker_thread.kthread.ret_from_fork
      0.00            +6.7        6.68 ±  3%  perf-profile.calltrace.cycles-pp.xfs_inactive_ifree.xfs_inactive.xfs_inactive_inode.xfs_reclaim_inodes_pag.xfs_inactive_worker
      0.00            +6.7        6.70 ±  2%  perf-profile.calltrace.cycles-pp.xfs_inactive.xfs_inactive_inode.xfs_reclaim_inodes_pag.xfs_inactive_worker.process_one_work
      0.00            +6.8        6.79 ±  2%  perf-profile.calltrace.cycles-pp.xfs_inactive_inode.xfs_reclaim_inodes_pag.xfs_inactive_worker.process_one_work.worker_thread
      0.00            +6.8        6.81 ±  2%  perf-profile.calltrace.cycles-pp.xfs_reclaim_inodes_pag.xfs_inactive_worker.process_one_work.worker_thread.kthread
    350472 ±  2%     +47.0%     515216 ±  4%  softirqs.CPU0.RCU
    178137 ±  4%     +45.2%     258701 ±  3%  softirqs.CPU0.SCHED
    475912 ±  9%     +26.6%     602627 ±  8%  softirqs.CPU0.TIMER
    351031 ±  2%     +46.9%     515648 ±  3%  softirqs.CPU1.RCU
    177205 ±  4%     +43.6%     254440 ±  3%  softirqs.CPU1.SCHED
    477011 ± 10%     +29.5%     617834 ±  9%  softirqs.CPU1.TIMER
    355813 ±  2%     +46.1%     519913 ±  5%  softirqs.CPU10.RCU
    176887 ±  4%     +45.7%     257643 ±  2%  softirqs.CPU10.SCHED
    483096 ±  9%     +34.2%     648174 ±  9%  softirqs.CPU10.TIMER
    351205 ±  2%     +48.0%     519792 ±  3%  softirqs.CPU11.RCU
    178176 ±  3%     +44.5%     257381 ±  2%  softirqs.CPU11.SCHED
    509243 ±  5%     +22.5%     623578 ±  9%  softirqs.CPU11.TIMER
    354284 ±  3%     +46.6%     519224 ±  5%  softirqs.CPU12.RCU
    178183 ±  3%     +44.5%     257434 ±  2%  softirqs.CPU12.SCHED
    485506 ±  9%     +33.8%     649681 ± 10%  softirqs.CPU12.TIMER
    350181 ±  3%     +49.2%     522540 ±  4%  softirqs.CPU13.RCU
    178505 ±  3%     +44.5%     257948 ±  2%  softirqs.CPU13.SCHED
    511564 ±  5%     +23.4%     631349 ±  7%  softirqs.CPU13.TIMER
    358014 ±  2%     +46.7%     525091 ±  5%  softirqs.CPU14.RCU
    178876 ±  3%     +44.4%     258277 ±  2%  softirqs.CPU14.SCHED
    487637 ±  9%     +36.3%     664878 ±  6%  softirqs.CPU14.TIMER
    376767 ±  8%     +37.9%     519729 ±  3%  softirqs.CPU15.RCU
    179212 ±  3%     +46.0%     261688 ±  3%  softirqs.CPU15.SCHED
    501815 ±  6%     +19.0%     597149 ± 13%  softirqs.CPU15.TIMER
    342231           +51.1%     517281 ±  6%  softirqs.CPU16.RCU
    174376           +48.1%     258219 ±  2%  softirqs.CPU16.SCHED
    439278 ± 10%     +51.2%     664047 ±  8%  softirqs.CPU16.TIMER
    349494 ±  3%     +47.5%     515663 ±  5%  softirqs.CPU17.RCU
    178311 ±  3%     +45.1%     258733 ±  2%  softirqs.CPU17.SCHED
    513865 ±  5%     +27.3%     654392 ±  8%  softirqs.CPU17.TIMER
    350783 ±  3%     +47.9%     518746 ±  5%  softirqs.CPU18.RCU
    182358 ±  2%     +41.9%     258705 ±  2%  softirqs.CPU18.SCHED
    459578 ± 18%     +49.3%     686336 ±  2%  softirqs.CPU18.TIMER
    351545 ±  2%     +48.1%     520706 ±  4%  softirqs.CPU19.RCU
    178437 ±  3%     +45.0%     258724 ±  2%  softirqs.CPU19.SCHED
    512878 ±  4%     +28.2%     657397 ±  8%  softirqs.CPU19.TIMER
    357605 ±  2%     +45.9%     521686 ±  4%  softirqs.CPU2.RCU
    176320 ±  4%     +48.1%     261181 ±  2%  softirqs.CPU2.SCHED
    477369 ±  9%     +44.0%     687253 ± 18%  softirqs.CPU2.TIMER
    353070 ±  2%     +48.5%     524194 ±  5%  softirqs.CPU20.RCU
    174940 ±  4%     +45.2%     253945 ±  3%  softirqs.CPU20.SCHED
    475767 ±  9%     +28.4%     610744 ±  6%  softirqs.CPU20.TIMER
    353397 ±  2%     +47.8%     522147 ±  4%  softirqs.CPU21.RCU
    176012 ±  4%     +44.1%     253708 ±  3%  softirqs.CPU21.SCHED
    476021 ± 10%     +30.8%     622474 ±  7%  softirqs.CPU21.TIMER
    362004 ±  2%     +46.6%     530709 ±  5%  softirqs.CPU22.RCU
    176322 ±  4%     +48.1%     261159 ±  2%  softirqs.CPU22.SCHED
    478634 ± 10%     +19.9%     574057 ± 13%  softirqs.CPU22.TIMER
    363725           +46.6%     533242 ±  3%  softirqs.CPU23.RCU
    177800 ±  3%     +43.4%     254985 ±  3%  softirqs.CPU23.SCHED
    478565 ± 11%     +29.4%     619240 ±  8%  softirqs.CPU23.TIMER
    365199           +46.8%     536212 ±  4%  softirqs.CPU24.RCU
    176725 ±  4%     +45.5%     257046 ±  2%  softirqs.CPU24.SCHED
    478986 ± 10%     +29.6%     620820 ±  8%  softirqs.CPU24.TIMER
    359975           +48.7%     535443 ±  2%  softirqs.CPU25.RCU
    181025 ±  4%     +40.6%     254450 ±  3%  softirqs.CPU25.SCHED
    447316 ± 19%     +37.7%     615960 ±  9%  softirqs.CPU25.TIMER
    361455 ±  2%     +47.8%     534276 ±  5%  softirqs.CPU26.RCU
    176157 ±  4%     +47.9%     260536 ±  3%  softirqs.CPU26.SCHED
    479646 ±  9%     +25.9%     603755 ± 12%  softirqs.CPU26.TIMER
    360462           +47.0%     529894 ±  2%  softirqs.CPU27.RCU
    178260 ±  3%     +43.3%     255441 ±  3%  softirqs.CPU27.SCHED
    505255 ±  6%     +23.1%     621841 ±  8%  softirqs.CPU27.TIMER
    348842 ±  6%     +53.1%     533960 ±  5%  softirqs.CPU28.RCU
    172723 ±  7%     +48.9%     257257 ±  2%  softirqs.CPU28.SCHED
    500520 ±  5%     +32.2%     661543 ±  5%  softirqs.CPU28.TIMER
    358679 ±  2%     +48.1%     531097 ±  3%  softirqs.CPU29.RCU
    178070 ±  3%     +43.4%     255300 ±  3%  softirqs.CPU29.SCHED
    504558 ±  5%     +24.0%     625466 ±  7%  softirqs.CPU29.TIMER
    357266 ±  2%     +47.2%     526003 ±  3%  softirqs.CPU3.RCU
    178003 ±  3%     +43.2%     254968 ±  3%  softirqs.CPU3.SCHED
    475849 ± 10%     +32.1%     628681 ±  6%  softirqs.CPU3.TIMER
    359014 ±  2%     +47.3%     528798 ±  6%  softirqs.CPU30.RCU
    177534 ±  3%     +45.2%     257845 ±  2%  softirqs.CPU30.SCHED
    480857 ±  9%     +36.9%     658480 ±  7%  softirqs.CPU30.TIMER
    352751 ±  2%     +49.4%     527040 ±  4%  softirqs.CPU31.RCU
    178325 ±  3%     +44.2%     257065 ±  3%  softirqs.CPU31.SCHED
    506996 ±  5%     +24.2%     629661 ±  7%  softirqs.CPU31.TIMER
    309246           +46.3%     452366 ±  3%  softirqs.CPU32.RCU
    178408 ±  3%     +44.6%     258044 ±  2%  softirqs.CPU32.SCHED
    487386 ±  9%     +35.9%     662460 ±  7%  softirqs.CPU32.TIMER
    307793 ±  2%     +45.3%     447116        softirqs.CPU33.RCU
    178346 ±  3%     +44.6%     257893 ±  2%  softirqs.CPU33.SCHED
    510065 ±  5%     +22.5%     624959 ±  9%  softirqs.CPU33.TIMER
    317885           +45.5%     462518        softirqs.CPU34.RCU
    179003 ±  3%     +44.1%     257979 ±  2%  softirqs.CPU34.SCHED
    491374 ±  9%     +32.7%     652163 ± 10%  softirqs.CPU34.TIMER
    310841 ±  2%     +46.3%     454843        softirqs.CPU35.RCU
    171543 ±  7%     +52.8%     262108 ±  3%  softirqs.CPU35.SCHED
    503129 ±  5%     +38.4%     696471 ± 19%  softirqs.CPU35.TIMER
    342548 ± 10%     +35.8%     465062        softirqs.CPU36.RCU
    182025 ±  4%     +41.7%     257873 ±  2%  softirqs.CPU36.SCHED
    522374 ± 17%     +24.9%     652676 ± 11%  softirqs.CPU36.TIMER
    317881 ±  2%     +46.2%     464732        softirqs.CPU37.RCU
    179258 ±  3%     +44.0%     258173 ±  2%  softirqs.CPU37.SCHED
    515687 ±  5%     +25.4%     646835 ± 10%  softirqs.CPU37.TIMER
    315664 ±  2%     +46.9%     463821        softirqs.CPU38.RCU
    182092 ±  2%     +41.9%     258304 ±  2%  softirqs.CPU38.SCHED
    555470 ±  8%     +34.1%     745113 ± 11%  softirqs.CPU38.TIMER
    318718 ±  2%     +44.6%     460755        softirqs.CPU39.RCU
    178974 ±  3%     +44.3%     258318 ±  2%  softirqs.CPU39.SCHED
    515014 ±  4%     +26.0%     649046 ± 11%  softirqs.CPU39.TIMER
    363141           +46.5%     532039 ±  4%  softirqs.CPU4.RCU
    175995 ±  4%     +46.1%     257188 ±  2%  softirqs.CPU4.SCHED
    477697 ±  9%     +32.2%     631353 ±  6%  softirqs.CPU4.TIMER
    355561 ±  2%     +49.6%     531834 ±  2%  softirqs.CPU5.RCU
    181315 ±  4%     +40.2%     254271 ±  3%  softirqs.CPU5.SCHED
    544589 ± 12%     +14.4%     622872 ±  7%  softirqs.CPU5.TIMER
    359996           +46.9%     528818 ±  5%  softirqs.CPU6.RCU
    176407 ±  4%     +48.2%     261494 ±  3%  softirqs.CPU6.SCHED
    478671 ±  9%     +48.3%     709673 ± 16%  softirqs.CPU6.TIMER
    360493           +47.7%     532348 ±  2%  softirqs.CPU7.RCU
    178141 ±  3%     +43.4%     255425 ±  3%  softirqs.CPU7.SCHED
    503929 ±  5%     +25.2%     631075 ±  5%  softirqs.CPU7.TIMER
    376377 ±  4%     +40.4%     528491 ±  4%  softirqs.CPU8.RCU
    176203 ±  4%     +45.8%     256922 ±  2%  softirqs.CPU8.SCHED
    498117 ±  5%     +29.3%     644298 ±  9%  softirqs.CPU8.TIMER
    356558 ±  2%     +48.3%     528683 ±  3%  softirqs.CPU9.RCU
    177928 ±  3%     +43.6%     255576 ±  3%  softirqs.CPU9.SCHED
    505291 ±  6%     +24.6%     629841 ±  6%  softirqs.CPU9.TIMER
  13967990 ±  2%     +46.7%   20497699 ±  3%  softirqs.RCU
   7112543 ±  3%     +44.8%   10302369 ±  2%  softirqs.SCHED
  19732539 ±  7%     +30.1%   25676278 ±  7%  softirqs.TIMER
      1965           +19.4%       2345        interrupts.10:IR-IO-APIC.10-edge.ipmi_si
      1775 ± 31%     +45.8%       2589 ± 26%  interrupts.37:IR-PCI-MSI.524289-edge.eth0-TxRx-0
    715.33          +205.9%       2188 ±  3%  interrupts.41:IR-PCI-MSI.524293-edge.eth0-TxRx-4
    606.00           +43.5%     869.75        interrupts.76:IR-PCI-MSI.512000-edge.ahci[0000:00:1f.2]
    399993           +43.8%     575091        interrupts.CAL:Function_call_interrupts
     10146 ±  8%     +46.2%      14830 ±  6%  interrupts.CPU0.CAL:Function_call_interrupts
   2482784           +43.5%    3563654        interrupts.CPU0.LOC:Local_timer_interrupts
     13988 ± 13%     +91.3%      26758 ± 18%  interrupts.CPU0.RES:Rescheduling_interrupts
     10395 ±  6%     +38.8%      14425 ±  6%  interrupts.CPU1.CAL:Function_call_interrupts
   2482642           +43.5%    3563164        interrupts.CPU1.LOC:Local_timer_interrupts
     14273 ± 16%     +68.4%      24036 ± 21%  interrupts.CPU1.RES:Rescheduling_interrupts
      9224 ±  3%     +62.1%      14955 ±  4%  interrupts.CPU10.CAL:Function_call_interrupts
   2482714           +43.5%    3563439        interrupts.CPU10.LOC:Local_timer_interrupts
     10652 ±  3%    +144.4%      26030 ± 16%  interrupts.CPU10.RES:Rescheduling_interrupts
      9328 ±  2%     +60.4%      14965 ±  5%  interrupts.CPU11.CAL:Function_call_interrupts
   2482726           +43.5%    3563349        interrupts.CPU11.LOC:Local_timer_interrupts
    158.33 ±141%    +269.3%     584.75 ± 71%  interrupts.CPU11.NMI:Non-maskable_interrupts
    158.33 ±141%    +269.3%     584.75 ± 71%  interrupts.CPU11.PMI:Performance_monitoring_interrupts
     10654          +155.4%      27214 ± 16%  interrupts.CPU11.RES:Rescheduling_interrupts
     10732 ±  2%     +41.7%      15206 ±  4%  interrupts.CPU12.CAL:Function_call_interrupts
   2482697           +43.5%    3563237        interrupts.CPU12.LOC:Local_timer_interrupts
     16311 ±  2%     +77.1%      28895 ±  3%  interrupts.CPU12.RES:Rescheduling_interrupts
     10469 ±  5%     +40.6%      14722 ±  5%  interrupts.CPU13.CAL:Function_call_interrupts
   2482942           +43.5%    3563379        interrupts.CPU13.LOC:Local_timer_interrupts
     14521 ± 19%     +73.2%      25149 ± 21%  interrupts.CPU13.RES:Rescheduling_interrupts
     10235 ±  4%     +44.9%      14835 ±  7%  interrupts.CPU14.CAL:Function_call_interrupts
   2482777           +43.5%    3563414        interrupts.CPU14.LOC:Local_timer_interrupts
     14459 ± 16%     +83.8%      26575 ± 17%  interrupts.CPU14.RES:Rescheduling_interrupts
     10475 ±  5%     +42.2%      14893 ±  5%  interrupts.CPU15.CAL:Function_call_interrupts
   2482263           +43.5%    3563256        interrupts.CPU15.LOC:Local_timer_interrupts
     10270 ±  4%     +39.2%      14297 ±  8%  interrupts.CPU16.CAL:Function_call_interrupts
   2482715           +43.5%    3563527        interrupts.CPU16.LOC:Local_timer_interrupts
     13353 ± 17%     +81.0%      24169 ± 20%  interrupts.CPU16.RES:Rescheduling_interrupts
     10596 ±  3%     +32.4%      14031 ±  5%  interrupts.CPU17.CAL:Function_call_interrupts
   2482547           +43.5%    3563381        interrupts.CPU17.LOC:Local_timer_interrupts
     15715 ±  8%     +41.7%      22268 ± 17%  interrupts.CPU17.RES:Rescheduling_interrupts
     10162 ±  4%     +38.0%      14021 ±  8%  interrupts.CPU18.CAL:Function_call_interrupts
   2482670           +43.5%    3563564        interrupts.CPU18.LOC:Local_timer_interrupts
     15192 ± 17%     +59.1%      24167 ± 21%  interrupts.CPU18.RES:Rescheduling_interrupts
     10490 ±  3%     +33.9%      14051 ±  5%  interrupts.CPU19.CAL:Function_call_interrupts
   2482487           +43.5%    3563418        interrupts.CPU19.LOC:Local_timer_interrupts
     16413 ±  6%     +51.3%      24830 ± 19%  interrupts.CPU19.RES:Rescheduling_interrupts
     10153 ±  4%     +41.1%      14330 ±  6%  interrupts.CPU2.CAL:Function_call_interrupts
   2482585           +43.5%    3563309        interrupts.CPU2.LOC:Local_timer_interrupts
     14386 ± 19%     +65.2%      23766 ± 22%  interrupts.CPU2.RES:Rescheduling_interrupts
      9830 ±  6%     +40.7%      13834 ±  4%  interrupts.CPU20.CAL:Function_call_interrupts
   2482474           +43.6%    3563715        interrupts.CPU20.LOC:Local_timer_interrupts
     12446 ± 20%     +67.4%      20840 ± 19%  interrupts.CPU20.RES:Rescheduling_interrupts
     10022 ±  4%     +46.2%      14658 ±  6%  interrupts.CPU21.CAL:Function_call_interrupts
   2482761           +43.5%    3563322        interrupts.CPU21.LOC:Local_timer_interrupts
     12414 ± 19%    +110.6%      26145 ± 15%  interrupts.CPU21.RES:Rescheduling_interrupts
      9641 ± 10%     +48.7%      14334 ±  8%  interrupts.CPU22.CAL:Function_call_interrupts
   2482862           +43.5%    3563405        interrupts.CPU22.LOC:Local_timer_interrupts
     12562 ± 18%     +93.4%      24296 ± 24%  interrupts.CPU22.RES:Rescheduling_interrupts
      9796 ±  4%     +50.0%      14698 ±  7%  interrupts.CPU23.CAL:Function_call_interrupts
   2482838           +43.5%    3563157        interrupts.CPU23.LOC:Local_timer_interrupts
     11819 ± 19%    +109.1%      24709 ± 18%  interrupts.CPU23.RES:Rescheduling_interrupts
     10259 ±  6%     +41.5%      14515 ±  6%  interrupts.CPU24.CAL:Function_call_interrupts
   2482791           +43.5%    3563576        interrupts.CPU24.LOC:Local_timer_interrupts
     15199 ± 19%     +53.5%      23324 ± 20%  interrupts.CPU24.RES:Rescheduling_interrupts
      9824 ±  4%     +53.3%      15063 ±  6%  interrupts.CPU25.CAL:Function_call_interrupts
   2482696           +43.5%    3563590        interrupts.CPU25.LOC:Local_timer_interrupts
    157.67 ±141%    +301.5%     633.00 ± 65%  interrupts.CPU25.NMI:Non-maskable_interrupts
    157.67 ±141%    +301.5%     633.00 ± 65%  interrupts.CPU25.PMI:Performance_monitoring_interrupts
     12009 ± 16%    +120.3%      26455 ± 15%  interrupts.CPU25.RES:Rescheduling_interrupts
      1775 ± 31%     +45.8%       2589 ± 26%  interrupts.CPU26.37:IR-PCI-MSI.524289-edge.eth0-TxRx-0
      9659 ± 10%     +47.7%      14267 ±  5%  interrupts.CPU26.CAL:Function_call_interrupts
   2482909           +43.5%    3563595        interrupts.CPU26.LOC:Local_timer_interrupts
     12858 ± 16%     +85.8%      23889 ± 20%  interrupts.CPU26.RES:Rescheduling_interrupts
      9768 ±  6%     +40.6%      13739 ±  8%  interrupts.CPU27.CAL:Function_call_interrupts
   2482824           +43.5%    3562529        interrupts.CPU27.LOC:Local_timer_interrupts
     12876 ± 22%     +73.5%      22335 ± 14%  interrupts.CPU27.RES:Rescheduling_interrupts
      9889 ±  8%     +36.8%      13526 ±  7%  interrupts.CPU28.CAL:Function_call_interrupts
   2482643           +43.5%    3563431        interrupts.CPU28.LOC:Local_timer_interrupts
     11749 ± 27%     +81.9%      21373 ± 23%  interrupts.CPU28.RES:Rescheduling_interrupts
      9717 ± 10%     +52.3%      14797 ±  5%  interrupts.CPU29.CAL:Function_call_interrupts
   2482876           +43.5%    3563145        interrupts.CPU29.LOC:Local_timer_interrupts
    182.33 ±107%    +191.0%     530.50 ± 22%  interrupts.CPU29.NMI:Non-maskable_interrupts
    182.33 ±107%    +191.0%     530.50 ± 22%  interrupts.CPU29.PMI:Performance_monitoring_interrupts
     12399 ± 19%    +111.8%      26258 ± 15%  interrupts.CPU29.RES:Rescheduling_interrupts
      9915 ±  8%     +45.1%      14388 ±  5%  interrupts.CPU3.CAL:Function_call_interrupts
   2482914           +43.5%    3563108        interrupts.CPU3.LOC:Local_timer_interrupts
     12900 ± 21%     +87.5%      24189 ± 19%  interrupts.CPU3.RES:Rescheduling_interrupts
     10273 ±  9%     +33.7%      13735 ± 10%  interrupts.CPU30.CAL:Function_call_interrupts
   2482440           +43.5%    3563278        interrupts.CPU30.LOC:Local_timer_interrupts
     14696 ± 12%     +46.5%      21527 ± 16%  interrupts.CPU30.RES:Rescheduling_interrupts
     10671 ±  2%     +30.3%      13903 ±  5%  interrupts.CPU31.CAL:Function_call_interrupts
   2482690           +43.5%    3563421        interrupts.CPU31.LOC:Local_timer_interrupts
     15897           +36.0%      21625 ± 19%  interrupts.CPU31.RES:Rescheduling_interrupts
      9336 ±  2%     +42.8%      13334 ±  2%  interrupts.CPU32.CAL:Function_call_interrupts
   2482536           +43.5%    3563218        interrupts.CPU32.LOC:Local_timer_interrupts
     11007 ±  4%     +68.1%      18500 ±  2%  interrupts.CPU32.RES:Rescheduling_interrupts
      9807 ±  9%     +47.2%      14438 ±  8%  interrupts.CPU33.CAL:Function_call_interrupts
   2482381           +43.5%    3563052        interrupts.CPU33.LOC:Local_timer_interrupts
    159.33 ±141%    +372.3%     752.50 ± 37%  interrupts.CPU33.NMI:Non-maskable_interrupts
    159.33 ±141%    +372.3%     752.50 ± 37%  interrupts.CPU33.PMI:Performance_monitoring_interrupts
     12320 ± 19%     +93.3%      23809 ± 18%  interrupts.CPU33.RES:Rescheduling_interrupts
    715.33          +205.9%       2188 ±  3%  interrupts.CPU34.41:IR-PCI-MSI.524293-edge.eth0-TxRx-4
      9817 ±  8%     +39.7%      13714 ±  4%  interrupts.CPU34.CAL:Function_call_interrupts
   2482766           +43.5%    3563310        interrupts.CPU34.LOC:Local_timer_interrupts
     12476 ± 18%     +66.0%      20710 ± 18%  interrupts.CPU34.RES:Rescheduling_interrupts
      9850 ±  9%     +41.9%      13976 ±  6%  interrupts.CPU35.CAL:Function_call_interrupts
   2482956           +43.5%    3563268        interrupts.CPU35.LOC:Local_timer_interrupts
     11311 ± 29%     +87.9%      21257 ± 17%  interrupts.CPU35.RES:Rescheduling_interrupts
      9784 ±  9%     +45.2%      14206 ±  5%  interrupts.CPU36.CAL:Function_call_interrupts
   2482486           +43.5%    3563409        interrupts.CPU36.LOC:Local_timer_interrupts
      9305 ±  3%     +55.2%      14440 ±  8%  interrupts.CPU37.CAL:Function_call_interrupts
   2482564           +43.5%    3562982        interrupts.CPU37.LOC:Local_timer_interrupts
     11000 ±  7%    +117.0%      23870 ± 21%  interrupts.CPU37.RES:Rescheduling_interrupts
      9502 ± 11%     +51.0%      14352 ±  6%  interrupts.CPU38.CAL:Function_call_interrupts
   2482635           +43.5%    3563178        interrupts.CPU38.LOC:Local_timer_interrupts
     11903 ± 21%     +92.5%      22913 ± 19%  interrupts.CPU38.RES:Rescheduling_interrupts
      9345 ±  2%     +55.4%      14520 ±  8%  interrupts.CPU39.CAL:Function_call_interrupts
   2482870           +43.5%    3563092        interrupts.CPU39.LOC:Local_timer_interrupts
     96.33 ±141%    +224.4%     312.50 ± 98%  interrupts.CPU39.NMI:Non-maskable_interrupts
     96.33 ±141%    +224.4%     312.50 ± 98%  interrupts.CPU39.PMI:Performance_monitoring_interrupts
     10384 ±  3%    +127.6%      23634 ± 21%  interrupts.CPU39.RES:Rescheduling_interrupts
      9743 ±  4%     +46.2%      14242 ± 10%  interrupts.CPU4.CAL:Function_call_interrupts
   2482637           +43.5%    3563660        interrupts.CPU4.LOC:Local_timer_interrupts
     12474 ± 16%     +91.7%      23919 ± 19%  interrupts.CPU4.RES:Rescheduling_interrupts
     10284 ±  7%     +35.7%      13954 ±  6%  interrupts.CPU5.CAL:Function_call_interrupts
   2482985           +43.5%    3563234        interrupts.CPU5.LOC:Local_timer_interrupts
     14377 ± 16%     +51.1%      21730 ± 20%  interrupts.CPU5.RES:Rescheduling_interrupts
    606.00           +43.5%     869.75        interrupts.CPU6.76:IR-PCI-MSI.512000-edge.ahci[0000:00:1f.2]
     10159 ±  4%     +45.7%      14803 ±  7%  interrupts.CPU6.CAL:Function_call_interrupts
   2482577           +43.5%    3563712        interrupts.CPU6.LOC:Local_timer_interrupts
     14706 ± 18%     +79.6%      26418 ± 17%  interrupts.CPU6.RES:Rescheduling_interrupts
     10361 ±  7%     +43.1%      14831 ±  5%  interrupts.CPU7.CAL:Function_call_interrupts
   2482952           +43.5%    3563417        interrupts.CPU7.LOC:Local_timer_interrupts
     14163 ± 18%     +89.0%      26764 ± 16%  interrupts.CPU7.RES:Rescheduling_interrupts
      1965           +19.4%       2345        interrupts.CPU8.10:IR-IO-APIC.10-edge.ipmi_si
     10416 ±  5%     +42.8%      14873 ±  7%  interrupts.CPU8.CAL:Function_call_interrupts
   2482862           +43.5%    3563671        interrupts.CPU8.LOC:Local_timer_interrupts
     10333 ±  5%     +39.1%      14374 ±  7%  interrupts.CPU9.CAL:Function_call_interrupts
   2482570           +43.5%    3563526        interrupts.CPU9.LOC:Local_timer_interrupts
     14052 ± 18%     +71.2%      24058 ± 20%  interrupts.CPU9.RES:Rescheduling_interrupts
  99308053           +43.5%  1.425e+08        interrupts.LOC:Local_timer_interrupts
    120.00           +66.7%     200.00        interrupts.MCP:Machine_check_polls
   1078626           -10.5%     964955        interrupts.RES:Rescheduling_interrupts
    336.67 ±  5%     +51.2%     509.00 ±  4%  interrupts.TLB:TLB_shootdowns


                                                                                
                                  aim7.jobs-per-min                             
                                                                                
  16000 +-+-----------------------------------------------------------------+   
        |                                                      +.+.+ .+   +.|   
  14000 +-+                              .+.                  +     + :   : |   
  12000 +-+.+.+.++.+.+   +.+.+.+.++.+.+.+   +.+.+.++.+.+. .+.+        :   : |   
        |            :   :                               +            :   : |   
  10000 O-O O O O  O O O O O O O OO O O                                : :  |   
        |             : :                                              : :  |   
   8000 +-+           : :                                              : :  |   
        |             : :                                              : :  |   
   6000 +-+           : :                                              : :  |   
   4000 +-+           : :                                              : :  |   
        |             : :                                               :   |   
   2000 +-+            :                                                :   |   
        |              :                                                :   |   
      0 +-+------O----------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.0.0-rc5-00051-g4ff1f5b" of type "text/plain" (169016 bytes)

View attachment "job-script" of type "text/plain" (7803 bytes)

View attachment "job.yaml" of type "text/plain" (5327 bytes)

View attachment "reproduce" of type "text/plain" (703 bytes)

Powered by blists - more mailing lists