lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Date:   Mon, 14 Jun 2021 22:55:36 +0800
From:   kernel test robot <oliver.sang@...el.com>
To:     "Aneesh Kumar K.V" <aneesh.kumar@...ux.ibm.com>
Cc:     0day robot <lkp@...el.com>, LKML <linux-kernel@...r.kernel.org>,
        lkp@...ts.01.org, ying.huang@...el.com, feng.tang@...el.com,
        zhengjun.xing@...ux.intel.com, linux-mm@...ck.org,
        akpm@...ux-foundation.org, mpe@...erman.id.au,
        linuxppc-dev@...ts.ozlabs.org, kaleshsingh@...gle.com,
        npiggin@...il.com, joel@...lfernandes.org,
        Christophe Leroy <christophe.leroy@...roup.eu>,
        Linus Torvalds <torvalds@...ux-foundation.org>,
        "Kirill A . Shutemov" <kirill@...temov.name>,
        "Aneesh Kumar K.V" <aneesh.kumar@...ux.ibm.com>
Subject: [mm/mremap]  ecf8443e51:  vm-scalability.throughput -29.4% regression



Greeting,

FYI, we noticed a -29.4% regression of vm-scalability.throughput due to commit:


commit: ecf8443e51a862b261313c2319ab4e4aed9e6b7e ("[PATCH v7 02/11] mm/mremap: Fix race between MOVE_PUD mremap and pageout")
url: https://github.com/0day-ci/linux/commits/Aneesh-Kumar-K-V/Speedup-mremap-on-ppc64/20210607-135424
base: https://git.kernel.org/cgit/linux/kernel/git/powerpc/linux.git next

in testcase: vm-scalability
on test machine: 192 threads 4 sockets Intel(R) Xeon(R) CPU @ 2.20GHz with 192G memory
with following parameters:

	runtime: 300
	thp_enabled: never
	thp_defrag: always
	nr_task: 8
	nr_ssd: 1
	test: swap-w-seq-mt
	cpufreq_governor: performance
	ucode: 0x4003006

test-description: The motivation behind this suite is to exercise functions and regions of the mm/ of the Linux kernel which are of interest to us.
test-url: https://git.kernel.org/cgit/linux/kernel/git/wfg/vm-scalability.git/



If you fix the issue, kindly add following tag
Reported-by: kernel test robot <oliver.sang@...el.com>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install                job.yaml  # job file is attached in this email
        bin/lkp split-job --compatible job.yaml  # generate the yaml file for lkp run
        bin/lkp run                    generated-yaml-file

=========================================================================================
compiler/cpufreq_governor/kconfig/nr_ssd/nr_task/rootfs/runtime/tbox_group/test/testcase/thp_defrag/thp_enabled/ucode:
  gcc-9/performance/x86_64-rhel-8.3/1/8/debian-10.4-x86_64-20200603.cgz/300/lkp-csl-2ap1/swap-w-seq-mt/vm-scalability/always/never/0x4003006

commit: 
  5f80ee2fc0 ("mm/mremap: Fix race between MOVE_PMD mremap and pageout")
  ecf8443e51 ("mm/mremap: Fix race between MOVE_PUD mremap and pageout")

5f80ee2fc08b3613 ecf8443e51a862b261313c2319a 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
      7.93 ±  4%     +14.6%       9.08 ±  5%  vm-scalability.free_time
    373210           -29.4%     263582 ±  3%  vm-scalability.median
     43.45 ±  5%      -8.1       35.34 ±  7%  vm-scalability.stddev%
   2954757           -29.4%    2085715 ±  3%  vm-scalability.throughput
    220.03           +38.3%     304.37 ±  2%  vm-scalability.time.elapsed_time
    220.03           +38.3%     304.37 ±  2%  vm-scalability.time.elapsed_time.max
     32410 ±  2%     +29.9%      42117 ± 10%  vm-scalability.time.involuntary_context_switches
    528.33           +13.1%     597.67        vm-scalability.time.percent_of_cpu_this_job_got
    913.19 ±  2%     +70.7%       1558 ±  3%  vm-scalability.time.system_time
    250.22            +4.7%     261.96        vm-scalability.time.user_time
     34211 ± 11%     -95.9%       1400 ± 38%  vm-scalability.time.voluntary_context_switches
      4.82 ±  3%     +15.8%       5.57 ±  3%  iostat.cpu.system
  3.77e+08 ± 24%     +46.4%   5.52e+08 ± 19%  turbostat.IRQ
    271.03           +30.2%     352.88 ±  2%  uptime.boot
     48496           +30.1%      63095 ±  2%  uptime.idle
      0.66 ± 12%      -0.7        0.00 ± 57%  mpstat.cpu.all.iowait%
      0.06 ±  4%      -0.0        0.06 ±  4%  mpstat.cpu.all.soft%
      3.09 ±  2%      +0.9        4.03 ±  3%  mpstat.cpu.all.sys%
      0.60            -0.1        0.48 ±  4%  mpstat.cpu.all.usr%
     28255 ±  5%     +77.2%      50067 ±  6%  meminfo.Active
     27866 ±  5%     +78.2%      49655 ±  6%  meminfo.Active(anon)
  28180965           -20.0%   22548765 ±  3%  meminfo.MemAvailable
  28675444           -19.7%   23016466 ±  3%  meminfo.MemFree
     38693 ±  5%     +56.1%      60412 ±  2%  meminfo.Shmem
    496106 ±  7%   +2180.0%   11311305 ± 12%  meminfo.SwapCached
     60640 ±  3%     -97.9%       1245 ± 46%  meminfo.Writeback
     87575 ±  3%     -43.8%      49246 ±  8%  slabinfo.dmaengine-unmap-16.active_objs
      2123 ±  3%     -44.5%       1179 ±  8%  slabinfo.dmaengine-unmap-16.active_slabs
     89210 ±  3%     -44.5%      49532 ±  8%  slabinfo.dmaengine-unmap-16.num_objs
      2123 ±  3%     -44.5%       1179 ±  8%  slabinfo.dmaengine-unmap-16.num_slabs
      4866           -20.4%       3874 ±  3%  slabinfo.kmalloc-4k.active_objs
    623.83           -20.9%     493.50 ±  3%  slabinfo.kmalloc-4k.active_slabs
      4992           -20.8%       3952 ±  3%  slabinfo.kmalloc-4k.num_objs
    623.83           -20.9%     493.50 ±  3%  slabinfo.kmalloc-4k.num_slabs
    374.17 ±  2%     -47.8%     195.17 ± 53%  vmstat.io.bi
   1341570           -23.6%    1025527 ±  2%  vmstat.io.bo
  28897122           -21.2%   22757749 ±  2%  vmstat.memory.free
 1.238e+08            +8.1%  1.339e+08 ±  2%  vmstat.memory.swpd
      1.00          -100.0%       0.00        vmstat.procs.b
      8.00           +29.2%      10.33 ±  7%  vmstat.procs.r
    364.17 ±  2%     -48.3%     188.17 ± 55%  vmstat.swap.si
   1341561           -23.6%    1025520 ±  2%  vmstat.swap.so
      3328 ±  2%     -19.9%       2666        vmstat.system.cs
   1511984           +17.7%    1780129        vmstat.system.in
     14215 ± 15%     -98.5%     209.67 ± 84%  numa-meminfo.node0.Writeback
    308351 ±116%   +1024.4%    3467064 ± 11%  numa-meminfo.node1.FilePages
    168771 ±  6%     -15.1%     143367 ± 12%  numa-meminfo.node1.KReclaimable
    168771 ±  6%     -15.1%     143367 ± 12%  numa-meminfo.node1.SReclaimable
    127034 ± 32%   +2618.6%    3453577 ± 10%  numa-meminfo.node1.SwapCached
     14382 ±  9%     -97.7%     332.00 ± 34%  numa-meminfo.node1.Writeback
    124583 ± 35%   +2602.3%    3366554 ± 14%  numa-meminfo.node2.FilePages
     77286 ± 26%     +55.8%     120430 ± 18%  numa-meminfo.node2.KReclaimable
     77286 ± 26%     +55.8%     120430 ± 18%  numa-meminfo.node2.SReclaimable
    138348 ± 20%     +28.1%     177225 ± 13%  numa-meminfo.node2.Slab
    113067 ± 23%   +2874.7%    3363404 ± 14%  numa-meminfo.node2.SwapCached
     14263 ±  5%     -97.4%     373.17 ± 37%  numa-meminfo.node2.Writeback
     13835 ± 13%    +155.7%      35375 ± 18%  numa-meminfo.node3.Active
     13835 ± 13%    +155.2%      35308 ± 18%  numa-meminfo.node3.Active(anon)
    261289 ±137%   +1236.3%    3491534 ±  7%  numa-meminfo.node3.FilePages
     16541 ± 12%    +130.9%      38192 ± 21%  numa-meminfo.node3.Shmem
     81973 ± 30%   +3903.0%    3281379 ±  9%  numa-meminfo.node3.SwapCached
     16321 ±  6%     -97.7%     372.33 ± 42%  numa-meminfo.node3.Writeback
      3643 ± 14%     -98.5%      55.00 ±102%  numa-vmstat.node0.nr_writeback
      3619 ± 14%     -98.6%      49.67 ±107%  numa-vmstat.node0.nr_zone_write_pending
     77012 ±116%   +1039.9%     877855 ± 10%  numa-vmstat.node1.nr_file_pages
     42097 ±  6%     -14.0%      36208 ± 10%  numa-vmstat.node1.nr_slab_reclaimable
     31679 ± 32%   +2660.4%     874478 ± 10%  numa-vmstat.node1.nr_swapcached
  11014726 ±  2%     +12.6%   12401236 ±  4%  numa-vmstat.node1.nr_vmscan_write
      3577 ± 10%     -97.5%      89.83 ± 41%  numa-vmstat.node1.nr_writeback
  11011253 ±  2%     +12.6%   12401174 ±  4%  numa-vmstat.node1.nr_written
      3502 ± 10%     -97.7%      82.17 ± 49%  numa-vmstat.node1.nr_zone_write_pending
     31302 ± 35%   +2617.2%     850555 ± 13%  numa-vmstat.node2.nr_file_pages
     19271 ± 26%     +57.8%      30407 ± 17%  numa-vmstat.node2.nr_slab_reclaimable
     28421 ± 24%   +2889.9%     849766 ± 12%  numa-vmstat.node2.nr_swapcached
  11041559 ±  3%     +19.1%   13149436 ± 10%  numa-vmstat.node2.nr_vmscan_write
      3782 ±  4%     -97.6%      92.00 ± 34%  numa-vmstat.node2.nr_writeback
  11037863 ±  3%     +19.1%   13149374 ± 10%  numa-vmstat.node2.nr_written
      3724 ±  4%     -97.9%      79.00 ± 39%  numa-vmstat.node2.nr_zone_write_pending
      3443 ± 14%    +160.0%       8954 ± 20%  numa-vmstat.node3.nr_active_anon
     65304 ±137%   +1252.7%     883364 ±  6%  numa-vmstat.node3.nr_file_pages
      4129 ± 12%    +134.5%       9682 ± 23%  numa-vmstat.node3.nr_shmem
     20479 ± 30%   +3956.2%     830690 ±  8%  numa-vmstat.node3.nr_swapcached
  10829696 ±  5%     +14.0%   12346461 ±  4%  numa-vmstat.node3.nr_vmscan_write
      4124 ±  4%     -97.6%      97.00 ± 39%  numa-vmstat.node3.nr_writeback
  10825671 ±  5%     +14.0%   12346386 ±  4%  numa-vmstat.node3.nr_written
      3444 ± 14%    +160.0%       8955 ± 20%  numa-vmstat.node3.nr_zone_active_anon
      4055 ±  4%     -97.9%      83.67 ± 43%  numa-vmstat.node3.nr_zone_write_pending
      8412 ±  3%     +28.0%      10768 ± 13%  proc-vmstat.allocstall_movable
   5996587 ± 46%    +183.4%   16993010 ± 32%  proc-vmstat.compact_free_scanned
   3451053 ± 29%     +75.7%    6064315 ± 15%  proc-vmstat.compact_isolated
   3259087 ± 27%     +89.7%    6184085 ± 15%  proc-vmstat.compact_migrate_scanned
      6961 ±  5%     +78.5%      12425 ±  6%  proc-vmstat.nr_active_anon
  40993758            -3.1%   39718983        proc-vmstat.nr_anon_pages
    698799           -19.8%     560310 ±  3%  proc-vmstat.nr_dirty_background_threshold
   1399308           -19.8%    1121992 ±  3%  proc-vmstat.nr_dirty_threshold
    388625 ±  2%    +695.9%    3092997 ± 11%  proc-vmstat.nr_file_pages
   7165507           -19.3%    5779541 ±  3%  proc-vmstat.nr_free_pages
  41119695            +3.5%   42542747        proc-vmstat.nr_inactive_anon
    145036            +3.1%     149568        proc-vmstat.nr_page_table_pages
      9675 ±  5%     +56.0%      15094 ±  2%  proc-vmstat.nr_shmem
    123879 ±  8%   +2178.6%    2822769 ± 12%  proc-vmstat.nr_swapcached
  38677277            +7.7%   41656286 ±  3%  proc-vmstat.nr_vmscan_write
     14819 ±  3%     -97.9%     307.33 ± 50%  proc-vmstat.nr_writeback
  74789620            +5.5%   78882933 ±  2%  proc-vmstat.nr_written
      6962 ±  5%     +78.5%      12426 ±  6%  proc-vmstat.nr_zone_active_anon
  41119448            +3.5%   42542544        proc-vmstat.nr_zone_inactive_anon
     14582 ±  3%     -98.1%     274.50 ± 47%  proc-vmstat.nr_zone_write_pending
  20967960 ± 16%     +27.7%   26777776 ± 10%  proc-vmstat.numa_pte_updates
     16061 ± 13%     +34.4%      21592 ±  7%  proc-vmstat.pgactivate
   1726355 ± 29%     +75.7%    3033559 ± 15%  proc-vmstat.pgmigrate_success
 2.992e+08            +5.5%  3.155e+08 ±  2%  proc-vmstat.pgpgout
     48003 ±  2%     +32.3%      63514 ±  3%  proc-vmstat.pgreuse
  74393768           -12.6%   65057070 ±  4%  proc-vmstat.pgrotated
  2.36e+08            +8.4%  2.558e+08        proc-vmstat.pgscan_anon
 1.048e+08 ±  2%     +24.6%  1.305e+08 ±  6%  proc-vmstat.pgscan_direct
  74792004            +5.5%   78884465 ±  2%  proc-vmstat.pswpout
     50067 ±  7%     +17.0%      58567 ±  4%  proc-vmstat.slabs_scanned
 4.724e+09           -18.5%   3.85e+09        perf-stat.i.branch-instructions
      3231 ±  2%     -19.3%       2608        perf-stat.i.context-switches
      1.86 ±  2%     +47.3%       2.74 ±  5%  perf-stat.i.cpi
 3.281e+10 ±  2%     +15.9%  3.803e+10 ±  4%  perf-stat.i.cpu-cycles
    206.24            -1.9%     202.23        perf-stat.i.cpu-migrations
    642.02 ±  6%     +14.2%     732.98 ±  8%  perf-stat.i.cycles-between-cache-misses
 4.851e+09           -17.3%  4.014e+09        perf-stat.i.dTLB-loads
   3304908           -19.0%    2677570 ±  4%  perf-stat.i.dTLB-store-misses
 2.027e+09           -19.4%  1.633e+09        perf-stat.i.dTLB-stores
   5484115 ±  3%     -14.5%    4691022 ±  3%  perf-stat.i.iTLB-load-misses
 1.857e+10           -17.5%  1.533e+10        perf-stat.i.instructions
      5181 ±  3%     -16.9%       4306 ±  2%  perf-stat.i.instructions-per-iTLB-miss
      0.59 ±  3%     -24.6%       0.44 ±  3%  perf-stat.i.ipc
     23.06 ±  6%     -49.2%      11.72 ± 54%  perf-stat.i.major-faults
      0.17 ±  2%     +15.9%       0.20 ±  4%  perf-stat.i.metric.GHz
     60.43           -18.1%      49.46        perf-stat.i.metric.M/sec
    585912           -26.9%     428491 ±  2%  perf-stat.i.minor-faults
     87.62            +3.0       90.65        perf-stat.i.node-load-miss-rate%
   1450164 ±  6%     -29.1%    1027918 ±  3%  perf-stat.i.node-loads
     83.34            +4.1       87.48        perf-stat.i.node-store-miss-rate%
   1130049 ± 10%     -24.5%     853398 ± 11%  perf-stat.i.node-stores
    585935           -26.9%     428503 ±  2%  perf-stat.i.page-faults
      1.77 ±  2%     +42.5%       2.52 ±  5%  perf-stat.overall.cpi
    560.05 ±  6%     +22.0%     683.28 ± 11%  perf-stat.overall.cycles-between-cache-misses
      0.56 ±  2%     -29.7%       0.40 ±  5%  perf-stat.overall.ipc
     91.72            +2.0       93.70        perf-stat.overall.node-load-miss-rate%
     86.56            +3.1       89.65        perf-stat.overall.node-store-miss-rate%
      7112           +13.7%       8086        perf-stat.overall.path-length
 4.703e+09           -18.7%  3.821e+09 ±  2%  perf-stat.ps.branch-instructions
      3220 ±  2%     -19.7%       2587        perf-stat.ps.context-switches
 3.277e+10 ±  2%     +17.3%  3.843e+10 ±  5%  perf-stat.ps.cpu-cycles
    205.01            -2.7%     199.55 ±  2%  perf-stat.ps.cpu-migrations
 4.834e+09           -17.5%  3.987e+09        perf-stat.ps.dTLB-loads
   3283502           -19.6%    2639256 ±  4%  perf-stat.ps.dTLB-store-misses
  2.02e+09           -19.9%  1.618e+09        perf-stat.ps.dTLB-stores
   5471166 ±  3%     -15.0%    4652576 ±  2%  perf-stat.ps.iTLB-load-misses
  1.85e+10           -17.7%  1.523e+10        perf-stat.ps.instructions
     22.79 ±  6%     -49.2%      11.57 ± 54%  perf-stat.ps.major-faults
    581983           -27.5%     422213 ±  2%  perf-stat.ps.minor-faults
   1442428 ±  6%     -29.8%    1013002 ±  3%  perf-stat.ps.node-loads
   1123088 ± 10%     -25.3%     839225 ± 10%  perf-stat.ps.node-stores
    582006           -27.5%     422225 ±  2%  perf-stat.ps.page-faults
 4.085e+12           +13.6%  4.642e+12        perf-stat.total.instructions
      0.01 ± 23%     -49.2%       0.01 ± 13%  perf-sched.sch_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      0.55 ±221%     -99.5%       0.00        perf-sched.sch_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      0.02 ± 32%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.kswapd.kthread.ret_from_fork
      0.01 ± 62%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.rwsem_down_write_slowpath.down_write_killable.__vm_munmap.__x64_sys_munmap
      0.01 ± 17%     -47.2%       0.00 ± 10%  perf-sched.sch_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      0.02 ± 21%     -49.1%       0.01 ± 22%  perf-sched.sch_delay.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      0.02 ± 37%     -60.0%       0.01 ± 34%  perf-sched.sch_delay.max.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
    166.76 ±223%    -100.0%       0.01 ± 10%  perf-sched.sch_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      0.03 ± 67%     -62.5%       0.01 ± 51%  perf-sched.sch_delay.max.ms.io_schedule.__lock_page_or_retry.do_swap_page.__handle_mm_fault
      0.03 ±130%     -77.5%       0.01 ± 61%  perf-sched.sch_delay.max.ms.io_schedule.swap_readpage.read_swap_cache_async.swap_cluster_readahead
      0.02 ± 39%    -100.0%       0.00        perf-sched.sch_delay.max.ms.kswapd.kthread.ret_from_fork
      0.09 ± 59%     -85.5%       0.01 ± 10%  perf-sched.sch_delay.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      0.05 ± 77%     -89.3%       0.01 ± 56%  perf-sched.sch_delay.max.ms.rcu_gp_kthread.kthread.ret_from_fork
      0.02 ± 96%    -100.0%       0.00        perf-sched.sch_delay.max.ms.rwsem_down_write_slowpath.down_write_killable.__vm_munmap.__x64_sys_munmap
      0.02 ± 36%     -62.8%       0.01 ± 21%  perf-sched.sch_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      1018 ± 14%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      1018 ± 14%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.do_syslog.part.0.kmsg_read.vfs_read
     15.36 ± 84%     -97.3%       0.42 ± 78%  perf-sched.wait_and_delay.avg.ms.io_schedule.swap_readpage.read_swap_cache_async.swapin_readahead
      2042 ± 37%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
      8.00 ± 28%    -100.0%       0.00        perf-sched.wait_and_delay.count.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      8.00 ± 28%    -100.0%       0.00        perf-sched.wait_and_delay.count.do_syslog.part.0.kmsg_read.vfs_read
     93.67 ±133%    +261.4%     338.50 ± 12%  perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.__alloc_pages_slowpath.constprop.0
     20.17 ±223%    +547.1%     130.50 ± 10%  perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.mempool_alloc.bio_alloc_bioset.__swap_writepage
    346.50 ± 74%    +194.0%       1018 ± 10%  perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.shrink_node.do_try_to_free_pages.try_to_free_pages
      9.00 ± 19%    -100.0%       0.00        perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
     23.00 ±  5%     +24.6%      28.67 ± 10%  perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
      5996 ± 31%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      5996 ± 31%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.do_syslog.part.0.kmsg_read.vfs_read
      4492 ± 79%     -99.9%       3.21 ± 71%  perf-sched.wait_and_delay.max.ms.io_schedule.swap_readpage.read_swap_cache_async.swapin_readahead
      6025 ± 32%     -82.0%       1082 ±  4%  perf-sched.wait_and_delay.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      6024 ± 32%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
      1018 ± 14%     -99.9%       1.33 ± 15%  perf-sched.wait_time.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      1018 ± 14%     -99.8%       1.60 ± 57%  perf-sched.wait_time.avg.ms.do_syslog.part.0.kmsg_read.vfs_read
    309.68 ±143%    -100.0%       0.01 ± 27%  perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
    224.46 ±171%     -99.9%       0.13 ±110%  perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
    378.18 ± 74%     -99.8%       0.69 ± 48%  perf-sched.wait_time.avg.ms.io_schedule.swap_readpage.read_swap_cache_async.swap_cluster_readahead
     15.36 ± 84%     -97.3%       0.42 ± 78%  perf-sched.wait_time.avg.ms.io_schedule.swap_readpage.read_swap_cache_async.swapin_readahead
    105.18          -100.0%       0.00        perf-sched.wait_time.avg.ms.kswapd.kthread.ret_from_fork
     57.26 ± 71%    -100.0%       0.00        perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.zap_pte_range.unmap_page_range.unmap_vmas
      2.81 ± 22%     -68.2%       0.89 ± 45%  perf-sched.wait_time.avg.ms.rcu_gp_kthread.kthread.ret_from_fork
    754.86 ±100%    -100.0%       0.00        perf-sched.wait_time.avg.ms.rwsem_down_read_slowpath.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
    298.39 ±141%    -100.0%       0.00        perf-sched.wait_time.avg.ms.rwsem_down_write_slowpath.down_write_killable.__vm_munmap.__x64_sys_munmap
      2042 ± 37%    -100.0%       0.37 ± 99%  perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
      0.00 ± 10%    -100.0%       0.00        perf-sched.wait_time.avg.ms.schedule_timeout.kswapd.kthread.ret_from_fork
      5996 ± 31%    -100.0%       2.66 ± 15%  perf-sched.wait_time.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      5996 ± 31%    -100.0%       2.56 ± 22%  perf-sched.wait_time.max.ms.do_syslog.part.0.kmsg_read.vfs_read
      1179 ±142%    -100.0%       0.03 ± 89%  perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
    818.62 ±149%    -100.0%       0.13 ±110%  perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
      4236 ± 54%     -99.9%       3.55 ± 58%  perf-sched.wait_time.max.ms.io_schedule.swap_readpage.read_swap_cache_async.swap_cluster_readahead
      4492 ± 79%     -99.9%       3.21 ± 71%  perf-sched.wait_time.max.ms.io_schedule.swap_readpage.read_swap_cache_async.swapin_readahead
    107.44          -100.0%       0.00        perf-sched.wait_time.max.ms.kswapd.kthread.ret_from_fork
      6025 ± 32%     -82.0%       1082 ±  4%  perf-sched.wait_time.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      2367 ± 71%    -100.0%       0.00        perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.zap_pte_range.unmap_page_range.unmap_vmas
      5.00           -56.8%       2.16 ± 93%  perf-sched.wait_time.max.ms.rcu_gp_kthread.kthread.ret_from_fork
      1888 ±101%    -100.0%       0.00        perf-sched.wait_time.max.ms.rwsem_down_read_slowpath.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
      1240 ±140%    -100.0%       0.00        perf-sched.wait_time.max.ms.rwsem_down_write_slowpath.down_write_killable.__vm_munmap.__x64_sys_munmap
      6024 ± 32%    -100.0%       0.74 ±107%  perf-sched.wait_time.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
      0.01 ± 13%    -100.0%       0.00        perf-sched.wait_time.max.ms.schedule_timeout.kswapd.kthread.ret_from_fork
    443.33           +38.2%     612.67 ±  2%  interrupts.9:IO-APIC.9-fasteoi.acpi
 2.506e+08 ±  2%     +71.0%  4.285e+08 ±  2%  interrupts.CAL:Function_call_interrupts
    436770           +39.6%     609678 ±  3%  interrupts.CPU0.LOC:Local_timer_interrupts
    443.33           +38.2%     612.67 ±  2%  interrupts.CPU1.9:IO-APIC.9-fasteoi.acpi
    436473           +39.6%     609534 ±  3%  interrupts.CPU1.LOC:Local_timer_interrupts
    436112           +39.7%     609460 ±  3%  interrupts.CPU10.LOC:Local_timer_interrupts
    435885           +39.8%     609476 ±  3%  interrupts.CPU100.LOC:Local_timer_interrupts
    435912           +39.8%     609519 ±  3%  interrupts.CPU101.LOC:Local_timer_interrupts
    435983           +39.8%     609496 ±  3%  interrupts.CPU102.LOC:Local_timer_interrupts
    436229           +39.7%     609544 ±  3%  interrupts.CPU103.LOC:Local_timer_interrupts
    152.33 ± 49%    +944.5%       1591 ± 79%  interrupts.CPU103.NMI:Non-maskable_interrupts
    152.33 ± 49%    +944.5%       1591 ± 79%  interrupts.CPU103.PMI:Performance_monitoring_interrupts
    436056           +39.8%     609461 ±  3%  interrupts.CPU104.LOC:Local_timer_interrupts
    435861           +39.8%     609478 ±  3%  interrupts.CPU105.LOC:Local_timer_interrupts
    436021           +39.8%     609379 ±  3%  interrupts.CPU106.LOC:Local_timer_interrupts
    435843           +39.9%     609554 ±  3%  interrupts.CPU107.LOC:Local_timer_interrupts
    435999           +39.8%     609466 ±  3%  interrupts.CPU108.LOC:Local_timer_interrupts
    436372           +39.7%     609463 ±  3%  interrupts.CPU109.LOC:Local_timer_interrupts
    435946           +39.8%     609652 ±  3%  interrupts.CPU11.LOC:Local_timer_interrupts
    436096           +39.8%     609463 ±  3%  interrupts.CPU110.LOC:Local_timer_interrupts
    435856           +39.8%     609436 ±  3%  interrupts.CPU111.LOC:Local_timer_interrupts
    436180           +39.7%     609470 ±  3%  interrupts.CPU112.LOC:Local_timer_interrupts
    435902           +39.8%     609466 ±  3%  interrupts.CPU113.LOC:Local_timer_interrupts
    435823           +39.9%     609521 ±  3%  interrupts.CPU114.LOC:Local_timer_interrupts
    435831           +39.8%     609416 ±  3%  interrupts.CPU115.LOC:Local_timer_interrupts
    435716           +39.9%     609492 ±  3%  interrupts.CPU116.LOC:Local_timer_interrupts
    435934           +39.8%     609518 ±  3%  interrupts.CPU117.LOC:Local_timer_interrupts
    435946           +39.8%     609459 ±  3%  interrupts.CPU118.LOC:Local_timer_interrupts
    435735           +39.9%     609492 ±  3%  interrupts.CPU119.LOC:Local_timer_interrupts
    436018           +39.8%     609544 ±  3%  interrupts.CPU12.LOC:Local_timer_interrupts
   1730294 ± 54%    +231.1%    5729689 ± 73%  interrupts.CPU120.CAL:Function_call_interrupts
    436513           +38.6%     605176 ±  4%  interrupts.CPU120.LOC:Local_timer_interrupts
    437125           +38.4%     605185 ±  4%  interrupts.CPU121.LOC:Local_timer_interrupts
    119.00 ± 49%   +2582.6%       3192 ± 78%  interrupts.CPU121.NMI:Non-maskable_interrupts
    119.00 ± 49%   +2582.6%       3192 ± 78%  interrupts.CPU121.PMI:Performance_monitoring_interrupts
    436783           +38.6%     605196 ±  4%  interrupts.CPU122.LOC:Local_timer_interrupts
    436134           +38.8%     605202 ±  4%  interrupts.CPU123.LOC:Local_timer_interrupts
    436402           +38.7%     605173 ±  4%  interrupts.CPU124.LOC:Local_timer_interrupts
    436347           +38.7%     605194 ±  4%  interrupts.CPU125.LOC:Local_timer_interrupts
    436510           +38.7%     605240 ±  4%  interrupts.CPU126.LOC:Local_timer_interrupts
    436805           +38.5%     605169 ±  4%  interrupts.CPU127.LOC:Local_timer_interrupts
    436659           +38.6%     605187 ±  4%  interrupts.CPU128.LOC:Local_timer_interrupts
    436231           +38.7%     605179 ±  4%  interrupts.CPU129.LOC:Local_timer_interrupts
    436500           +39.6%     609536 ±  3%  interrupts.CPU13.LOC:Local_timer_interrupts
    436424           +38.7%     605184 ±  4%  interrupts.CPU130.LOC:Local_timer_interrupts
    436297           +38.7%     605194 ±  4%  interrupts.CPU131.LOC:Local_timer_interrupts
    436562           +38.6%     605189 ±  4%  interrupts.CPU132.LOC:Local_timer_interrupts
    436922           +38.5%     605198 ±  4%  interrupts.CPU133.LOC:Local_timer_interrupts
    436581           +38.6%     605204 ±  4%  interrupts.CPU134.LOC:Local_timer_interrupts
    436103           +38.8%     605178 ±  4%  interrupts.CPU135.LOC:Local_timer_interrupts
    436818           +38.5%     605171 ±  4%  interrupts.CPU136.LOC:Local_timer_interrupts
    436327           +38.7%     605173 ±  4%  interrupts.CPU137.LOC:Local_timer_interrupts
    436190           +38.7%     605185 ±  4%  interrupts.CPU138.LOC:Local_timer_interrupts
    436439           +38.7%     605194 ±  4%  interrupts.CPU139.LOC:Local_timer_interrupts
    436084           +39.8%     609534 ±  3%  interrupts.CPU14.LOC:Local_timer_interrupts
    436238           +38.7%     605188 ±  4%  interrupts.CPU140.LOC:Local_timer_interrupts
    436692           +38.6%     605188 ±  4%  interrupts.CPU141.LOC:Local_timer_interrupts
    436428           +38.7%     605187 ±  4%  interrupts.CPU142.LOC:Local_timer_interrupts
    436618           +38.6%     605181 ±  4%  interrupts.CPU143.LOC:Local_timer_interrupts
   2395060 ± 21%    +210.7%    7440320 ± 40%  interrupts.CPU144.CAL:Function_call_interrupts
    437076 ±  2%     +38.3%     604600 ±  4%  interrupts.CPU144.LOC:Local_timer_interrupts
   4849080 ± 21%    +113.2%   10339751 ± 39%  interrupts.CPU144.TLB:TLB_shootdowns
    437303 ±  2%     +38.3%     604631 ±  4%  interrupts.CPU145.LOC:Local_timer_interrupts
    437337 ±  2%     +38.3%     604650 ±  4%  interrupts.CPU146.LOC:Local_timer_interrupts
   1628946 ± 35%    +144.0%    3974501 ± 26%  interrupts.CPU147.CAL:Function_call_interrupts
    436987 ±  2%     +38.4%     604641 ±  4%  interrupts.CPU147.LOC:Local_timer_interrupts
   3313139 ± 38%     +62.6%    5387495 ± 26%  interrupts.CPU147.TLB:TLB_shootdowns
    437013 ±  2%     +38.4%     604647 ±  4%  interrupts.CPU148.LOC:Local_timer_interrupts
    437047 ±  2%     +38.3%     604641 ±  4%  interrupts.CPU149.LOC:Local_timer_interrupts
    435764           +39.9%     609486 ±  3%  interrupts.CPU15.LOC:Local_timer_interrupts
    437104 ±  2%     +38.3%     604641 ±  4%  interrupts.CPU150.LOC:Local_timer_interrupts
    437148 ±  2%     +38.3%     604674 ±  4%  interrupts.CPU151.LOC:Local_timer_interrupts
    437106 ±  2%     +38.3%     604644 ±  4%  interrupts.CPU152.LOC:Local_timer_interrupts
    437065 ±  2%     +38.4%     604688 ±  4%  interrupts.CPU153.LOC:Local_timer_interrupts
    437182 ±  2%     +38.3%     604648 ±  4%  interrupts.CPU154.LOC:Local_timer_interrupts
    437068 ±  2%     +38.3%     604649 ±  4%  interrupts.CPU155.LOC:Local_timer_interrupts
    437126 ±  2%     +38.3%     604638 ±  4%  interrupts.CPU156.LOC:Local_timer_interrupts
    437340 ±  2%     +38.3%     604639 ±  4%  interrupts.CPU157.LOC:Local_timer_interrupts
    437229 ±  2%     +38.3%     604631 ±  4%  interrupts.CPU158.LOC:Local_timer_interrupts
    437031 ±  2%     +38.4%     604646 ±  4%  interrupts.CPU159.LOC:Local_timer_interrupts
    436217           +39.7%     609502 ±  3%  interrupts.CPU16.LOC:Local_timer_interrupts
    437281 ±  2%     +38.3%     604671 ±  4%  interrupts.CPU160.LOC:Local_timer_interrupts
    437174 ±  2%     +38.3%     604664 ±  4%  interrupts.CPU161.LOC:Local_timer_interrupts
    437093 ±  2%     +38.3%     604644 ±  4%  interrupts.CPU162.LOC:Local_timer_interrupts
    437080 ±  2%     +38.3%     604617 ±  4%  interrupts.CPU163.LOC:Local_timer_interrupts
    437014 ±  2%     +38.4%     604623 ±  4%  interrupts.CPU164.LOC:Local_timer_interrupts
    437064 ±  2%     +38.3%     604664 ±  4%  interrupts.CPU165.LOC:Local_timer_interrupts
    437060 ±  2%     +38.3%     604645 ±  4%  interrupts.CPU166.LOC:Local_timer_interrupts
    437133 ±  2%     +38.3%     604647 ±  4%  interrupts.CPU167.LOC:Local_timer_interrupts
    437691           +38.8%     607710 ±  3%  interrupts.CPU168.LOC:Local_timer_interrupts
    437778           +38.8%     607719 ±  3%  interrupts.CPU169.LOC:Local_timer_interrupts
    333.83 ±130%    +630.9%       2440 ±113%  interrupts.CPU169.NMI:Non-maskable_interrupts
    333.83 ±130%    +630.9%       2440 ±113%  interrupts.CPU169.PMI:Performance_monitoring_interrupts
    435893           +39.8%     609457 ±  3%  interrupts.CPU17.LOC:Local_timer_interrupts
    437845           +38.8%     607745 ±  3%  interrupts.CPU170.LOC:Local_timer_interrupts
    972.00 ±199%    +287.4%       3765 ± 75%  interrupts.CPU170.NMI:Non-maskable_interrupts
    972.00 ±199%    +287.4%       3765 ± 75%  interrupts.CPU170.PMI:Performance_monitoring_interrupts
    437428           +38.9%     607740 ±  3%  interrupts.CPU171.LOC:Local_timer_interrupts
    437494           +38.9%     607739 ±  3%  interrupts.CPU172.LOC:Local_timer_interrupts
   1590512 ± 27%    +139.8%    3814104 ± 44%  interrupts.CPU173.CAL:Function_call_interrupts
    437559           +38.9%     607732 ±  3%  interrupts.CPU173.LOC:Local_timer_interrupts
    437721           +38.8%     607747 ±  3%  interrupts.CPU174.LOC:Local_timer_interrupts
    437316           +39.0%     607744 ±  3%  interrupts.CPU175.LOC:Local_timer_interrupts
    437481           +38.9%     607717 ±  3%  interrupts.CPU176.LOC:Local_timer_interrupts
    437671           +38.9%     607816 ±  3%  interrupts.CPU177.LOC:Local_timer_interrupts
    437549           +38.9%     607776 ±  3%  interrupts.CPU178.LOC:Local_timer_interrupts
    437791           +38.8%     607753 ±  3%  interrupts.CPU179.LOC:Local_timer_interrupts
    435869           +39.8%     609482 ±  3%  interrupts.CPU18.LOC:Local_timer_interrupts
    437851           +38.8%     607737 ±  3%  interrupts.CPU180.LOC:Local_timer_interrupts
    437762           +38.8%     607765 ±  3%  interrupts.CPU181.LOC:Local_timer_interrupts
    437506           +38.9%     607755 ±  3%  interrupts.CPU182.LOC:Local_timer_interrupts
    437825           +38.8%     607732 ±  3%  interrupts.CPU183.LOC:Local_timer_interrupts
    437482           +38.9%     607739 ±  3%  interrupts.CPU184.LOC:Local_timer_interrupts
    437502           +38.9%     607760 ±  3%  interrupts.CPU185.LOC:Local_timer_interrupts
    437904           +38.8%     607748 ±  3%  interrupts.CPU186.LOC:Local_timer_interrupts
    437601           +38.9%     607731 ±  3%  interrupts.CPU187.LOC:Local_timer_interrupts
    437425           +38.9%     607746 ±  3%  interrupts.CPU188.LOC:Local_timer_interrupts
    437660           +38.9%     607742 ±  3%  interrupts.CPU189.LOC:Local_timer_interrupts
    435885           +39.8%     609492 ±  3%  interrupts.CPU19.LOC:Local_timer_interrupts
    658863 ± 99%    +449.3%    3618986 ± 71%  interrupts.CPU190.CAL:Function_call_interrupts
    437525           +38.9%     607724 ±  3%  interrupts.CPU190.LOC:Local_timer_interrupts
    437714           +38.9%     607775 ±  3%  interrupts.CPU191.LOC:Local_timer_interrupts
    436340           +39.7%     609513 ±  3%  interrupts.CPU2.LOC:Local_timer_interrupts
    435793           +39.9%     609516 ±  3%  interrupts.CPU20.LOC:Local_timer_interrupts
    435929           +39.8%     609516 ±  3%  interrupts.CPU21.LOC:Local_timer_interrupts
    435897           +39.8%     609589 ±  3%  interrupts.CPU22.LOC:Local_timer_interrupts
    436105           +39.7%     609457 ±  3%  interrupts.CPU23.LOC:Local_timer_interrupts
    436542           +38.6%     605219 ±  4%  interrupts.CPU24.LOC:Local_timer_interrupts
    437153           +38.4%     605215 ±  4%  interrupts.CPU25.LOC:Local_timer_interrupts
    436932           +38.5%     605126 ±  4%  interrupts.CPU26.LOC:Local_timer_interrupts
    436176           +38.8%     605266 ±  4%  interrupts.CPU27.LOC:Local_timer_interrupts
    436415           +38.7%     605223 ±  4%  interrupts.CPU28.LOC:Local_timer_interrupts
    436413           +38.7%     605249 ±  4%  interrupts.CPU29.LOC:Local_timer_interrupts
    435892           +39.8%     609496 ±  3%  interrupts.CPU3.LOC:Local_timer_interrupts
      1884 ± 93%     -90.6%     177.17 ± 79%  interrupts.CPU3.NMI:Non-maskable_interrupts
      1884 ± 93%     -90.6%     177.17 ± 79%  interrupts.CPU3.PMI:Performance_monitoring_interrupts
   3717304 ± 27%     -51.0%    1822336 ± 39%  interrupts.CPU3.TLB:TLB_shootdowns
    436566           +38.6%     605260 ±  4%  interrupts.CPU30.LOC:Local_timer_interrupts
    436820           +38.5%     605209 ±  4%  interrupts.CPU31.LOC:Local_timer_interrupts
    436693           +38.6%     605210 ±  4%  interrupts.CPU32.LOC:Local_timer_interrupts
    436239           +38.7%     605207 ±  4%  interrupts.CPU33.LOC:Local_timer_interrupts
    436425           +38.7%     605210 ±  4%  interrupts.CPU34.LOC:Local_timer_interrupts
    436281           +38.7%     605214 ±  4%  interrupts.CPU35.LOC:Local_timer_interrupts
    436621           +38.6%     605203 ±  4%  interrupts.CPU36.LOC:Local_timer_interrupts
    436949           +38.5%     605220 ±  4%  interrupts.CPU37.LOC:Local_timer_interrupts
    436637           +38.6%     605224 ±  4%  interrupts.CPU38.LOC:Local_timer_interrupts
    436124           +38.8%     605199 ±  4%  interrupts.CPU39.LOC:Local_timer_interrupts
    435874           +39.8%     609500 ±  3%  interrupts.CPU4.LOC:Local_timer_interrupts
    436826           +38.5%     605202 ±  4%  interrupts.CPU40.LOC:Local_timer_interrupts
    436386           +38.7%     605204 ±  4%  interrupts.CPU41.LOC:Local_timer_interrupts
    436202           +38.7%     605125 ±  4%  interrupts.CPU42.LOC:Local_timer_interrupts
    436444           +38.7%     605206 ±  4%  interrupts.CPU43.LOC:Local_timer_interrupts
    436315           +38.7%     605211 ±  4%  interrupts.CPU44.LOC:Local_timer_interrupts
    436710           +38.6%     605216 ±  4%  interrupts.CPU45.LOC:Local_timer_interrupts
    436448           +38.7%     605224 ±  4%  interrupts.CPU46.LOC:Local_timer_interrupts
    436683           +38.6%     605205 ±  4%  interrupts.CPU47.LOC:Local_timer_interrupts
    437175 ±  2%     +38.3%     604605 ±  4%  interrupts.CPU48.LOC:Local_timer_interrupts
    428230 ± 65%    +243.5%    1470847 ± 61%  interrupts.CPU49.CAL:Function_call_interrupts
    437324 ±  2%     +38.3%     604629 ±  4%  interrupts.CPU49.LOC:Local_timer_interrupts
    435982           +39.8%     609551 ±  3%  interrupts.CPU5.LOC:Local_timer_interrupts
    437347 ±  2%     +38.3%     604679 ±  4%  interrupts.CPU50.LOC:Local_timer_interrupts
    436995 ±  2%     +38.4%     604678 ±  4%  interrupts.CPU51.LOC:Local_timer_interrupts
    437070 ±  2%     +38.3%     604662 ±  4%  interrupts.CPU52.LOC:Local_timer_interrupts
    437067 ±  2%     +38.3%     604680 ±  4%  interrupts.CPU53.LOC:Local_timer_interrupts
    437130 ±  2%     +38.3%     604671 ±  4%  interrupts.CPU54.LOC:Local_timer_interrupts
    437155 ±  2%     +38.3%     604700 ±  4%  interrupts.CPU55.LOC:Local_timer_interrupts
    437141 ±  2%     +38.3%     604682 ±  4%  interrupts.CPU56.LOC:Local_timer_interrupts
    437048 ±  2%     +38.4%     604671 ±  4%  interrupts.CPU57.LOC:Local_timer_interrupts
    437189 ±  2%     +38.3%     604701 ±  4%  interrupts.CPU58.LOC:Local_timer_interrupts
    437069 ±  2%     +38.4%     604688 ±  4%  interrupts.CPU59.LOC:Local_timer_interrupts
    436032           +39.8%     609555 ±  3%  interrupts.CPU6.LOC:Local_timer_interrupts
    437190 ±  2%     +38.3%     604670 ±  4%  interrupts.CPU60.LOC:Local_timer_interrupts
    437388 ±  2%     +38.2%     604673 ±  4%  interrupts.CPU61.LOC:Local_timer_interrupts
    437227 ±  2%     +38.3%     604666 ±  4%  interrupts.CPU62.LOC:Local_timer_interrupts
    437036 ±  2%     +38.4%     604662 ±  4%  interrupts.CPU63.LOC:Local_timer_interrupts
    437304 ±  2%     +38.3%     604662 ±  4%  interrupts.CPU64.LOC:Local_timer_interrupts
    437177 ±  2%     +38.3%     604654 ±  4%  interrupts.CPU65.LOC:Local_timer_interrupts
    437090 ±  2%     +38.3%     604659 ±  4%  interrupts.CPU66.LOC:Local_timer_interrupts
    437100 ±  2%     +38.3%     604602 ±  4%  interrupts.CPU67.LOC:Local_timer_interrupts
    437019 ±  2%     +38.4%     604618 ±  4%  interrupts.CPU68.LOC:Local_timer_interrupts
    437048 ±  2%     +38.4%     604675 ±  4%  interrupts.CPU69.LOC:Local_timer_interrupts
    436254           +39.7%     609555 ±  3%  interrupts.CPU7.LOC:Local_timer_interrupts
    140.83 ± 25%    +500.0%     845.00 ± 93%  interrupts.CPU7.NMI:Non-maskable_interrupts
    140.83 ± 25%    +500.0%     845.00 ± 93%  interrupts.CPU7.PMI:Performance_monitoring_interrupts
    437047 ±  2%     +38.4%     604663 ±  4%  interrupts.CPU70.LOC:Local_timer_interrupts
    437164 ±  2%     +38.3%     604658 ±  4%  interrupts.CPU71.LOC:Local_timer_interrupts
    437731           +38.8%     607745 ±  3%  interrupts.CPU72.LOC:Local_timer_interrupts
    437796           +38.8%     607772 ±  3%  interrupts.CPU73.LOC:Local_timer_interrupts
    437891           +38.8%     607769 ±  3%  interrupts.CPU74.LOC:Local_timer_interrupts
    488.67 ±180%    +653.0%       3679 ± 75%  interrupts.CPU74.NMI:Non-maskable_interrupts
    488.67 ±180%    +653.0%       3679 ± 75%  interrupts.CPU74.PMI:Performance_monitoring_interrupts
    456635 ± 60%    +233.5%    1522817 ± 41%  interrupts.CPU75.CAL:Function_call_interrupts
    437451           +38.9%     607764 ±  3%  interrupts.CPU75.LOC:Local_timer_interrupts
    437531           +38.9%     607754 ±  3%  interrupts.CPU76.LOC:Local_timer_interrupts
    437603           +38.9%     607768 ±  3%  interrupts.CPU77.LOC:Local_timer_interrupts
    437752           +38.8%     607791 ±  3%  interrupts.CPU78.LOC:Local_timer_interrupts
    437335           +39.0%     607782 ±  3%  interrupts.CPU79.LOC:Local_timer_interrupts
    436062           +39.8%     609477 ±  3%  interrupts.CPU8.LOC:Local_timer_interrupts
    437520           +38.9%     607783 ±  3%  interrupts.CPU80.LOC:Local_timer_interrupts
    437687           +38.9%     607803 ±  3%  interrupts.CPU81.LOC:Local_timer_interrupts
    437567           +38.9%     607820 ±  3%  interrupts.CPU82.LOC:Local_timer_interrupts
    437815           +38.8%     607773 ±  3%  interrupts.CPU83.LOC:Local_timer_interrupts
    437892           +38.8%     607756 ±  3%  interrupts.CPU84.LOC:Local_timer_interrupts
    437781           +38.8%     607806 ±  3%  interrupts.CPU85.LOC:Local_timer_interrupts
    437537           +38.9%     607769 ±  3%  interrupts.CPU86.LOC:Local_timer_interrupts
    437848           +38.8%     607766 ±  3%  interrupts.CPU87.LOC:Local_timer_interrupts
    437543           +38.9%     607809 ±  3%  interrupts.CPU88.LOC:Local_timer_interrupts
    437531           +38.9%     607776 ±  3%  interrupts.CPU89.LOC:Local_timer_interrupts
    435850           +39.8%     609496 ±  3%  interrupts.CPU9.LOC:Local_timer_interrupts
    437953           +38.8%     607759 ±  3%  interrupts.CPU90.LOC:Local_timer_interrupts
    437616           +38.9%     607766 ±  3%  interrupts.CPU91.LOC:Local_timer_interrupts
    437458           +38.9%     607762 ±  3%  interrupts.CPU92.LOC:Local_timer_interrupts
    437697           +38.9%     607787 ±  3%  interrupts.CPU93.LOC:Local_timer_interrupts
    437556           +38.9%     607804 ±  3%  interrupts.CPU94.LOC:Local_timer_interrupts
    437772           +38.8%     607836 ±  3%  interrupts.CPU95.LOC:Local_timer_interrupts
    436537           +39.6%     609467 ±  3%  interrupts.CPU96.LOC:Local_timer_interrupts
    436496           +39.6%     609495 ±  3%  interrupts.CPU97.LOC:Local_timer_interrupts
    436305           +39.7%     609468 ±  3%  interrupts.CPU98.LOC:Local_timer_interrupts
    435736           +39.9%     609454 ±  3%  interrupts.CPU99.LOC:Local_timer_interrupts
  83872531           +38.9%  1.165e+08 ±  4%  interrupts.LOC:Local_timer_interrupts
      0.00       +1.9e+104%     192.00        interrupts.MCP:Machine_check_polls
    122140 ±  6%     +15.2%     140671 ±  5%  interrupts.NMI:Non-maskable_interrupts
    122140 ±  6%     +15.2%     140671 ±  5%  interrupts.PMI:Performance_monitoring_interrupts
 5.166e+08 ±  2%     +15.1%  5.948e+08 ±  2%  interrupts.TLB:TLB_shootdowns
     17.91 ±  6%      -7.3       10.57 ±  2%  perf-profile.calltrace.cycles-pp.try_to_unmap_flush_dirty.shrink_page_list.shrink_inactive_list.shrink_lruvec.shrink_node
     17.90 ±  6%      -7.3       10.56 ±  2%  perf-profile.calltrace.cycles-pp.arch_tlbbatch_flush.try_to_unmap_flush_dirty.shrink_page_list.shrink_inactive_list.shrink_lruvec
     17.22 ±  6%      -7.0       10.22 ±  2%  perf-profile.calltrace.cycles-pp.on_each_cpu_cond_mask.arch_tlbbatch_flush.try_to_unmap_flush_dirty.shrink_page_list.shrink_inactive_list
     16.87 ±  6%      -6.8       10.07 ±  2%  perf-profile.calltrace.cycles-pp.smp_call_function_many_cond.on_each_cpu_cond_mask.arch_tlbbatch_flush.try_to_unmap_flush_dirty.shrink_page_list
     40.74 ±  7%      -4.6       36.13 ±  4%  perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
     40.30 ±  8%      -4.5       35.84 ±  4%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     40.30 ±  8%      -4.5       35.84 ±  4%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
     40.30 ±  8%      -4.5       35.83 ±  4%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     38.05 ±  9%      -4.4       33.60 ±  4%  perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     37.65 ±  9%      -4.4       33.26 ±  4%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
      4.33 ±  9%      -3.2        1.13 ± 22%  perf-profile.calltrace.cycles-pp.__remove_mapping.shrink_page_list.shrink_inactive_list.shrink_lruvec.shrink_node
      5.57 ±  6%      -2.7        2.85 ±  4%  perf-profile.calltrace.cycles-pp.pageout.shrink_page_list.shrink_inactive_list.shrink_lruvec.shrink_node
      3.96 ±  5%      -2.2        1.76 ±  7%  perf-profile.calltrace.cycles-pp.add_to_swap.shrink_page_list.shrink_inactive_list.shrink_lruvec.shrink_node
      3.72 ±  6%      -1.8        1.90 ±  8%  perf-profile.calltrace.cycles-pp.__swap_writepage.pageout.shrink_page_list.shrink_inactive_list.shrink_lruvec
      3.91 ±  6%      -1.8        2.13 ±  3%  perf-profile.calltrace.cycles-pp.llist_add_batch.smp_call_function_many_cond.on_each_cpu_cond_mask.arch_tlbbatch_flush.try_to_unmap_flush_dirty
      1.83 ±  6%      -1.4        0.41 ± 71%  perf-profile.calltrace.cycles-pp.asm_sysvec_call_function.smp_call_function_many_cond.on_each_cpu_cond_mask.arch_tlbbatch_flush.try_to_unmap_flush_dirty
      3.04 ± 11%      -0.9        2.17 ± 11%  perf-profile.calltrace.cycles-pp.mem_cgroup_charge.do_anonymous_page.__handle_mm_fault.handle_mm_fault.do_user_addr_fault
      5.24 ±  7%      -0.8        4.43 ±  4%  perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      3.40 ±  6%      -0.8        2.62 ± 10%  perf-profile.calltrace.cycles-pp.do_rw_once
      4.81 ±  7%      -0.7        4.06 ±  5%  perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle
      1.94 ± 11%      -0.7        1.19 ±  9%  perf-profile.calltrace.cycles-pp.__mem_cgroup_charge.mem_cgroup_charge.do_anonymous_page.__handle_mm_fault.handle_mm_fault
      3.31 ±  8%      -0.5        2.78 ±  5%  perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      3.29 ±  8%      -0.5        2.76 ±  5%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      0.61 ±  5%      -0.3        0.28 ±100%  perf-profile.calltrace.cycles-pp.__softirqentry_text_start.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      1.15 ± 17%      -0.3        0.88 ±  8%  perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages_slowpath.__alloc_pages.alloc_pages_vma.do_anonymous_page
      0.73 ±  6%      -0.1        0.62 ±  6%  perf-profile.calltrace.cycles-pp.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      0.46 ± 44%      +0.4        0.83 ± 17%  perf-profile.calltrace.cycles-pp.asm_sysvec_call_function.do_access
      1.58 ±  8%      +0.5        2.07 ±  6%  perf-profile.calltrace.cycles-pp.__default_send_IPI_dest_field.default_send_IPI_mask_sequence_phys.smp_call_function_many_cond.on_each_cpu_cond_mask.arch_tlbbatch_flush
      0.00            +2.2        2.25 ±  4%  perf-profile.calltrace.cycles-pp.asm_sysvec_call_function.native_queued_spin_lock_slowpath._raw_spin_lock.page_vma_mapped_walk.try_to_unmap_one
      0.00            +2.4        2.38 ±  8%  perf-profile.calltrace.cycles-pp.asm_sysvec_call_function.native_queued_spin_lock_slowpath._raw_spin_lock.page_vma_mapped_walk.page_referenced_one
      0.00            +3.4        3.37 ±  9%  perf-profile.calltrace.cycles-pp.flush_smp_call_function_queue.__sysvec_call_function.sysvec_call_function.asm_sysvec_call_function.native_queued_spin_lock_slowpath
      0.00            +3.4        3.41 ±  9%  perf-profile.calltrace.cycles-pp.__sysvec_call_function.sysvec_call_function.asm_sysvec_call_function.native_queued_spin_lock_slowpath._raw_spin_lock
      0.00            +3.7        3.72 ±  6%  perf-profile.calltrace.cycles-pp.sysvec_call_function.asm_sysvec_call_function.native_queued_spin_lock_slowpath._raw_spin_lock.page_vma_mapped_walk
      5.56 ±  8%     +10.7       16.22 ±  3%  perf-profile.calltrace.cycles-pp.rmap_walk_anon.try_to_unmap.shrink_page_list.shrink_inactive_list.shrink_lruvec
      5.84 ±  8%     +10.8       16.68 ±  3%  perf-profile.calltrace.cycles-pp.try_to_unmap.shrink_page_list.shrink_inactive_list.shrink_lruvec.shrink_node
      4.96 ±  8%     +10.9       15.85 ±  3%  perf-profile.calltrace.cycles-pp.try_to_unmap_one.rmap_walk_anon.try_to_unmap.shrink_page_list.shrink_inactive_list
      0.10 ±223%     +12.3       12.40 ±  4%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.page_vma_mapped_walk.try_to_unmap_one.rmap_walk_anon
      0.10 ±223%     +12.5       12.62 ±  3%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.page_vma_mapped_walk.page_referenced_one.rmap_walk_anon
      0.34 ±109%     +13.2       13.56 ±  3%  perf-profile.calltrace.cycles-pp._raw_spin_lock.page_vma_mapped_walk.try_to_unmap_one.rmap_walk_anon.try_to_unmap
      2.00 ± 12%     +13.4       15.36 ±  3%  perf-profile.calltrace.cycles-pp.rmap_walk_anon.page_referenced.shrink_page_list.shrink_inactive_list.shrink_lruvec
      2.13 ± 12%     +13.4       15.48 ±  3%  perf-profile.calltrace.cycles-pp.page_referenced.shrink_page_list.shrink_inactive_list.shrink_lruvec.shrink_node
      0.32 ±105%     +13.4       13.74 ±  3%  perf-profile.calltrace.cycles-pp._raw_spin_lock.page_vma_mapped_walk.page_referenced_one.rmap_walk_anon.page_referenced
      0.50 ± 77%     +13.5       14.04 ±  3%  perf-profile.calltrace.cycles-pp.page_vma_mapped_walk.try_to_unmap_one.rmap_walk_anon.try_to_unmap.shrink_page_list
      0.51 ± 76%     +13.9       14.39 ±  3%  perf-profile.calltrace.cycles-pp.page_vma_mapped_walk.page_referenced_one.rmap_walk_anon.page_referenced.shrink_page_list
      0.87 ± 22%     +14.0       14.83 ±  3%  perf-profile.calltrace.cycles-pp.page_referenced_one.rmap_walk_anon.page_referenced.shrink_page_list.shrink_inactive_list
     18.00 ±  6%      -7.4       10.64 ±  2%  perf-profile.children.cycles-pp.try_to_unmap_flush_dirty
     17.99 ±  6%      -7.4       10.64 ±  2%  perf-profile.children.cycles-pp.arch_tlbbatch_flush
     17.31 ±  6%      -7.0       10.29 ±  2%  perf-profile.children.cycles-pp.on_each_cpu_cond_mask
     17.30 ±  6%      -7.0       10.29 ±  3%  perf-profile.children.cycles-pp.smp_call_function_many_cond
     40.74 ±  7%      -4.6       36.13 ±  4%  perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     40.74 ±  7%      -4.6       36.13 ±  4%  perf-profile.children.cycles-pp.cpu_startup_entry
     40.74 ±  7%      -4.6       36.13 ±  4%  perf-profile.children.cycles-pp.do_idle
     38.48 ±  8%      -4.6       33.90 ±  4%  perf-profile.children.cycles-pp.cpuidle_enter
     38.46 ±  8%      -4.6       33.89 ±  4%  perf-profile.children.cycles-pp.cpuidle_enter_state
     40.30 ±  8%      -4.5       35.84 ±  4%  perf-profile.children.cycles-pp.start_secondary
      4.34 ±  9%      -3.0        1.33 ±  8%  perf-profile.children.cycles-pp.__remove_mapping
      5.60 ±  6%      -2.7        2.87 ±  4%  perf-profile.children.cycles-pp.pageout
      3.99 ±  5%      -2.1        1.84 ±  3%  perf-profile.children.cycles-pp.add_to_swap
      4.12 ±  5%      -1.8        2.29 ±  6%  perf-profile.children.cycles-pp.llist_add_batch
      3.74 ±  6%      -1.8        1.98 ±  4%  perf-profile.children.cycles-pp.__swap_writepage
      1.78 ± 12%      -1.4        0.38 ± 13%  perf-profile.children.cycles-pp.mem_cgroup_swapout
      1.64 ±  6%      -1.1        0.55 ±  4%  perf-profile.children.cycles-pp.get_swap_page
      3.25 ± 11%      -1.0        2.28 ± 11%  perf-profile.children.cycles-pp.mem_cgroup_charge
      1.57 ±  6%      -0.9        0.66 ±  6%  perf-profile.children.cycles-pp.swap_writepage
      3.90 ±  6%      -0.9        3.01 ± 11%  perf-profile.children.cycles-pp.do_rw_once
      5.60 ±  7%      -0.8        4.78 ±  4%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
      1.09 ± 19%      -0.8        0.29 ±  6%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      1.35 ±  6%      -0.8        0.57 ±  5%  perf-profile.children.cycles-pp._swap_info_get
      1.16 ±  5%      -0.8        0.39 ±  8%  perf-profile.children.cycles-pp.page_swap_info
      5.12 ±  7%      -0.7        4.37 ±  5%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
      1.14 ±  5%      -0.7        0.39 ±  8%  perf-profile.children.cycles-pp.swap_set_page_dirty
      2.02 ± 11%      -0.7        1.30 ±  9%  perf-profile.children.cycles-pp.__mem_cgroup_charge
      0.88 ± 13%      -0.7        0.19 ± 18%  perf-profile.children.cycles-pp.page_counter_uncharge
      2.00 ± 10%      -0.6        1.36 ± 11%  perf-profile.children.cycles-pp.get_page_from_freelist
      0.79 ± 13%      -0.6        0.16 ± 18%  perf-profile.children.cycles-pp.page_counter_cancel
      1.26 ±  2%      -0.6        0.64 ±  7%  perf-profile.children.cycles-pp.handle_irq_event
      1.27 ±  2%      -0.6        0.65 ±  7%  perf-profile.children.cycles-pp.__common_interrupt
      1.25 ±  2%      -0.6        0.64 ±  7%  perf-profile.children.cycles-pp.handle_irq_event_percpu
      1.26 ±  2%      -0.6        0.65 ±  7%  perf-profile.children.cycles-pp.handle_edge_irq
      1.25 ±  2%      -0.6        0.64 ±  7%  perf-profile.children.cycles-pp.__handle_irq_event_percpu
      1.24 ±  2%      -0.6        0.63 ±  7%  perf-profile.children.cycles-pp.nvme_irq
      1.28 ±  2%      -0.6        0.67 ±  6%  perf-profile.children.cycles-pp.common_interrupt
      1.28 ±  2%      -0.6        0.67 ±  6%  perf-profile.children.cycles-pp.asm_common_interrupt
      0.93 ±  5%      -0.6        0.34 ±  5%  perf-profile.children.cycles-pp.swap_page_sector
      1.05 ±  7%      -0.6        0.48 ±  4%  perf-profile.children.cycles-pp.try_to_free_swap
      1.02 ±  7%      -0.6        0.46 ±  5%  perf-profile.children.cycles-pp.page_swapcount
      3.59 ±  8%      -0.5        3.05 ±  5%  perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
      3.56 ±  8%      -0.5        3.04 ±  5%  perf-profile.children.cycles-pp.hrtimer_interrupt
      1.10 ±  2%      -0.5        0.58 ±  8%  perf-profile.children.cycles-pp.blk_update_request
      1.11 ±  2%      -0.5        0.59 ±  7%  perf-profile.children.cycles-pp.blk_mq_end_request
      0.70 ±  9%      -0.5        0.20 ± 10%  perf-profile.children.cycles-pp.swap_duplicate
      0.70 ± 10%      -0.5        0.19 ± 11%  perf-profile.children.cycles-pp.__swap_duplicate
      1.32 ±  5%      -0.5        0.83 ±  2%  perf-profile.children.cycles-pp.add_to_swap_cache
      0.69 ±  9%      -0.5        0.23 ±  5%  perf-profile.children.cycles-pp.get_swap_pages
      1.21 ± 11%      -0.4        0.77 ± 12%  perf-profile.children.cycles-pp.prep_new_page
      1.13 ± 11%      -0.4        0.71 ± 12%  perf-profile.children.cycles-pp.clear_page_erms
      0.93 ± 12%      -0.4        0.52 ± 11%  perf-profile.children.cycles-pp.try_charge
      0.84 ±  5%      -0.4        0.44 ±  4%  perf-profile.children.cycles-pp.bio_alloc_bioset
      1.12 ±  8%      -0.4        0.74 ±  8%  perf-profile.children.cycles-pp.submit_bio
      0.45 ± 22%      -0.4        0.06 ± 11%  perf-profile.children.cycles-pp.swap_cgroup_record
      1.11 ±  7%      -0.4        0.72 ±  8%  perf-profile.children.cycles-pp.submit_bio_noacct
      0.66 ±  9%      -0.4        0.29 ±  8%  perf-profile.children.cycles-pp.put_swap_page
      0.67 ±  6%      -0.4        0.30 ±  7%  perf-profile.children.cycles-pp.__delete_from_swap_cache
      0.84 ±  8%      -0.4        0.47 ±  8%  perf-profile.children.cycles-pp.page_lock_anon_vma_read
      0.57 ±  8%      -0.4        0.20 ±  5%  perf-profile.children.cycles-pp.scan_swap_map_slots
      0.51 ±  3%      -0.3        0.18 ± 11%  perf-profile.children.cycles-pp.__frontswap_store
      0.65 ±  3%      -0.3        0.32 ±  9%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.72 ± 15%      -0.3        0.39 ± 11%  perf-profile.children.cycles-pp._find_next_bit
      0.59 ±  8%      -0.3        0.28 ±  7%  perf-profile.children.cycles-pp.asm_sysvec_call_function_single
      0.80 ±  4%      -0.3        0.48 ±  8%  perf-profile.children.cycles-pp.end_swap_bio_write
      0.79 ±  4%      -0.3        0.48 ±  8%  perf-profile.children.cycles-pp.end_page_writeback
      0.61 ±  5%      -0.3        0.31 ±  7%  perf-profile.children.cycles-pp.native_flush_tlb_multi
      0.38 ±  8%      -0.3        0.09 ± 12%  perf-profile.children.cycles-pp.mem_cgroup_id_get_online
      0.72 ±  7%      -0.3        0.43 ±  5%  perf-profile.children.cycles-pp.blk_mq_submit_bio
      0.64 ± 11%      -0.3        0.35 ±  7%  perf-profile.children.cycles-pp.mem_cgroup_charge_statistics
      0.47 ±  8%      -0.3        0.21 ±  7%  perf-profile.children.cycles-pp.sysvec_call_function_single
      0.44 ±  7%      -0.3        0.19 ±  7%  perf-profile.children.cycles-pp.__sysvec_call_function_single
      0.52 ±  9%      -0.2        0.27 ±  4%  perf-profile.children.cycles-pp.cgroup_rstat_updated
      0.56 ±  4%      -0.2        0.33 ±  6%  perf-profile.children.cycles-pp.bio_init
      0.28 ± 13%      -0.2        0.06 ± 13%  perf-profile.children.cycles-pp.get_swap_device
      0.36 ± 26%      -0.2        0.15 ± 26%  perf-profile.children.cycles-pp.cpumask_any_but
      0.54 ±  5%      -0.2        0.36 ±  4%  perf-profile.children.cycles-pp.__mod_lruvec_page_state
      0.27 ± 11%      -0.2        0.09 ± 12%  perf-profile.children.cycles-pp.scan_swap_map_try_ssd_cluster
      0.44 ±  8%      -0.2        0.26 ±  4%  perf-profile.children.cycles-pp.isolate_lru_pages
      0.38 ±  8%      -0.2        0.20 ±  8%  perf-profile.children.cycles-pp.__set_page_dirty_no_writeback
      0.42 ±  7%      -0.2        0.25 ± 10%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.27 ±  8%      -0.2        0.11 ± 10%  perf-profile.children.cycles-pp.mempool_alloc
      0.43 ±  6%      -0.2        0.26 ± 10%  perf-profile.children.cycles-pp.rotate_reclaimable_page
      0.30 ±  8%      -0.2        0.14 ±  8%  perf-profile.children.cycles-pp.blk_mq_flush_plug_list
      0.29 ±  8%      -0.2        0.13 ±  5%  perf-profile.children.cycles-pp.blk_mq_sched_insert_requests
      0.39 ± 10%      -0.2        0.23 ±  4%  perf-profile.children.cycles-pp.__test_set_page_writeback
      0.30 ±  8%      -0.2        0.14 ±  6%  perf-profile.children.cycles-pp.blk_flush_plug_list
      0.29 ±  9%      -0.2        0.13 ±  6%  perf-profile.children.cycles-pp.nvme_queue_rq
      0.28 ±  8%      -0.2        0.13 ±  5%  perf-profile.children.cycles-pp.blk_mq_try_issue_list_directly
      0.28 ±  8%      -0.2        0.13 ±  6%  perf-profile.children.cycles-pp.__blk_mq_try_issue_directly
      0.28 ±  9%      -0.1        0.13 ±  5%  perf-profile.children.cycles-pp.blk_mq_request_issue_directly
      0.48 ±  7%      -0.1        0.33 ±  4%  perf-profile.children.cycles-pp.cpumask_next
      0.25 ±  8%      -0.1        0.10 ±  9%  perf-profile.children.cycles-pp.nvme_map_data
      0.38 ±  5%      -0.1        0.24 ±  9%  perf-profile.children.cycles-pp.pagevec_lru_move_fn
      0.40 ±  7%      -0.1        0.26 ±  5%  perf-profile.children.cycles-pp.rmqueue
      0.21 ±  8%      -0.1        0.08 ± 20%  perf-profile.children.cycles-pp.kmem_cache_alloc
      0.34 ±  7%      -0.1        0.23 ±  3%  perf-profile.children.cycles-pp.free_unref_page_list
      0.34 ±  5%      -0.1        0.23 ±  7%  perf-profile.children.cycles-pp.__mod_memcg_lruvec_state
      0.46 ± 11%      -0.1        0.35 ±  9%  perf-profile.children.cycles-pp.lru_cache_add
      0.30 ± 13%      -0.1        0.20 ± 10%  perf-profile.children.cycles-pp.rcu_read_unlock_strict
      0.23 ±  6%      -0.1        0.12 ±  9%  perf-profile.children.cycles-pp.rmqueue_bulk
      0.42 ± 10%      -0.1        0.32 ± 10%  perf-profile.children.cycles-pp.__pagevec_lru_add
      0.62 ±  5%      -0.1        0.53 ±  7%  perf-profile.children.cycles-pp.__softirqentry_text_start
      0.12 ±  9%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.kmem_cache_free
      0.21 ±  5%      -0.1        0.13 ±  8%  perf-profile.children.cycles-pp.xas_store
      0.23 ±  7%      -0.1        0.14 ± 14%  perf-profile.children.cycles-pp.pagevec_move_tail_fn
      0.22 ±  5%      -0.1        0.14 ±  3%  perf-profile.children.cycles-pp.__mod_node_page_state
      0.24 ±  3%      -0.1        0.16 ±  7%  perf-profile.children.cycles-pp.xas_create
      0.23 ± 10%      -0.1        0.15 ±  4%  perf-profile.children.cycles-pp.free_pcppages_bulk
      0.33 ±  8%      -0.1        0.25 ± 13%  perf-profile.children.cycles-pp.submit_bio_checks
      0.24 ±  8%      -0.1        0.17 ±  9%  perf-profile.children.cycles-pp.blk_attempt_plug_merge
      0.21 ±  5%      -0.1        0.14 ±  9%  perf-profile.children.cycles-pp.test_clear_page_writeback
      0.20 ±  6%      -0.1        0.14 ±  8%  perf-profile.children.cycles-pp.__mod_lruvec_state
      0.18 ±  2%      -0.1        0.11 ±  4%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.17 ±  8%      -0.1        0.11 ± 11%  perf-profile.children.cycles-pp.page_add_new_anon_rmap
      0.09 ± 14%      -0.1        0.03 ±101%  perf-profile.children.cycles-pp.__count_memcg_events
      0.19 ± 11%      -0.1        0.13 ±  6%  perf-profile.children.cycles-pp.page_mapping
      0.20 ±  6%      -0.1        0.14 ± 11%  perf-profile.children.cycles-pp.load_balance
      0.15 ± 10%      -0.1        0.10 ±  5%  perf-profile.children.cycles-pp.page_remove_rmap
      0.16 ±  2%      -0.1        0.10 ± 16%  perf-profile.children.cycles-pp.update_sd_lb_stats
      0.18 ± 11%      -0.1        0.12 ± 10%  perf-profile.children.cycles-pp.bio_attempt_back_merge
      0.16 ±  4%      -0.0        0.11 ± 14%  perf-profile.children.cycles-pp.find_busiest_group
      0.13 ±  8%      -0.0        0.08 ±  8%  perf-profile.children.cycles-pp.mutex_unlock
      0.07 ±  9%      -0.0        0.02 ± 99%  perf-profile.children.cycles-pp.__mod_memcg_state
      0.07 ±  7%      -0.0        0.02 ± 99%  perf-profile.children.cycles-pp.release_pages
      0.10 ± 17%      -0.0        0.06 ± 11%  perf-profile.children.cycles-pp.lock_page_lruvec_irqsave
      0.06 ± 11%      -0.0        0.02 ± 99%  perf-profile.children.cycles-pp.mutex_lock
      0.11 ±  6%      -0.0        0.07 ±  8%  perf-profile.children.cycles-pp.memset_erms
      0.14 ± 10%      -0.0        0.10 ± 19%  perf-profile.children.cycles-pp.page_counter_try_charge
      0.10 ± 11%      -0.0        0.07 ± 11%  perf-profile.children.cycles-pp.___might_sleep
      0.10 ±  9%      -0.0        0.07 ±  5%  perf-profile.children.cycles-pp.__hrtimer_next_event_base
      0.10 ± 10%      -0.0        0.07 ±  8%  perf-profile.children.cycles-pp.blk_throtl_bio
      0.08 ± 15%      -0.0        0.04 ± 45%  perf-profile.children.cycles-pp.rcu_idle_exit
      0.08 ±  6%      -0.0        0.04 ± 45%  perf-profile.children.cycles-pp.__blk_queue_split
      0.11 ±  9%      -0.0        0.08 ± 15%  perf-profile.children.cycles-pp.__perf_sw_event
      0.11 ±  7%      -0.0        0.08 ±  8%  perf-profile.children.cycles-pp.ll_back_merge_fn
      0.09 ±  7%      -0.0        0.06 ± 11%  perf-profile.children.cycles-pp.__cond_resched
      0.12 ±  7%      -0.0        0.09 ± 12%  perf-profile.children.cycles-pp.update_rq_clock
      0.08 ± 16%      -0.0        0.05 ± 47%  perf-profile.children.cycles-pp.bio_associate_blkg_from_css
      0.07 ± 10%      -0.0        0.04 ± 45%  perf-profile.children.cycles-pp.___perf_sw_event
      0.10 ± 10%      -0.0        0.08 ±  4%  perf-profile.children.cycles-pp.hrtimer_update_next_event
      0.09 ±  7%      -0.0        0.07 ± 16%  perf-profile.children.cycles-pp.get_next_timer_interrupt
      0.07 ±  7%      -0.0        0.04 ± 45%  perf-profile.children.cycles-pp.update_irq_load_avg
      0.09 ±  7%      -0.0        0.07 ±  5%  perf-profile.children.cycles-pp.hrtimer_next_event_without
      0.08 ± 10%      -0.0        0.06 ± 11%  perf-profile.children.cycles-pp.update_blocked_averages
      1.01 ±  7%      +0.2        1.20 ±  6%  perf-profile.children.cycles-pp.native_flush_tlb_local
      2.16 ±  7%      +0.4        2.54 ±  2%  perf-profile.children.cycles-pp.default_send_IPI_mask_sequence_phys
      1.59 ±  8%      +0.6        2.15 ±  3%  perf-profile.children.cycles-pp.__default_send_IPI_dest_field
     42.06 ±  5%      +8.5       50.53 ±  2%  perf-profile.children.cycles-pp.shrink_node
     42.00 ±  5%      +8.5       50.49 ±  2%  perf-profile.children.cycles-pp.shrink_lruvec
     41.98 ±  5%      +8.5       50.48 ±  2%  perf-profile.children.cycles-pp.shrink_inactive_list
     40.95 ±  5%      +8.9       49.83 ±  2%  perf-profile.children.cycles-pp.shrink_page_list
      5.08 ±  8%     +11.0       16.05 ±  3%  perf-profile.children.cycles-pp.try_to_unmap_one
      5.86 ±  8%     +11.0       16.84 ±  3%  perf-profile.children.cycles-pp.try_to_unmap
      2.15 ± 12%     +13.5       15.64 ±  3%  perf-profile.children.cycles-pp.page_referenced
      1.30 ± 17%     +13.7       15.00 ±  3%  perf-profile.children.cycles-pp.page_referenced_one
      7.62 ±  9%     +24.3       31.90 ±  3%  perf-profile.children.cycles-pp.rmap_walk_anon
      1.58 ± 18%     +24.6       26.23 ±  3%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      2.42 ± 19%     +25.7       28.14 ±  3%  perf-profile.children.cycles-pp._raw_spin_lock
      1.80 ± 27%     +27.0       28.78 ±  3%  perf-profile.children.cycles-pp.page_vma_mapped_walk
      8.26 ±  8%      -4.0        4.26 ±  5%  perf-profile.self.cycles-pp.smp_call_function_many_cond
      3.24 ±  6%      -1.1        2.10 ±  9%  perf-profile.self.cycles-pp.do_rw_once
      2.97 ±  8%      -1.0        1.96 ± 10%  perf-profile.self.cycles-pp.do_access
      2.96 ±  6%      -1.0        1.95 ±  6%  perf-profile.self.cycles-pp.llist_add_batch
      1.73 ±  9%      -0.7        0.98 ±  8%  perf-profile.self.cycles-pp.try_to_unmap_one
      0.79 ± 13%      -0.6        0.16 ± 18%  perf-profile.self.cycles-pp.page_counter_cancel
      1.10 ±  6%      -0.6        0.50 ±  5%  perf-profile.self.cycles-pp._swap_info_get
      0.89 ±  6%      -0.6        0.33 ±  8%  perf-profile.self.cycles-pp.page_swap_info
      0.73 ±  6%      -0.5        0.27 ±  5%  perf-profile.self.cycles-pp.swap_page_sector
      0.98 ± 12%      -0.4        0.53 ± 11%  perf-profile.self.cycles-pp.clear_page_erms
      0.59 ±  6%      -0.4        0.17 ±  6%  perf-profile.self.cycles-pp.get_swap_page
      0.69 ± 13%      -0.4        0.31 ± 12%  perf-profile.self.cycles-pp.try_charge
      2.74 ±  7%      -0.3        2.40 ±  7%  perf-profile.self.cycles-pp.flush_tlb_func
      0.38 ±  8%      -0.3        0.09 ± 10%  perf-profile.self.cycles-pp.mem_cgroup_id_get_online
      0.74 ±  7%      -0.3        0.46 ±  4%  perf-profile.self.cycles-pp.add_to_swap_cache
      0.45 ±  9%      -0.3        0.18 ± 10%  perf-profile.self.cycles-pp.__delete_from_swap_cache
      0.52 ± 10%      -0.3        0.26 ±  5%  perf-profile.self.cycles-pp.cgroup_rstat_updated
      0.42 ± 10%      -0.2        0.19 ± 12%  perf-profile.self.cycles-pp.mem_cgroup_charge
      0.55 ± 16%      -0.2        0.33 ± 12%  perf-profile.self.cycles-pp._find_next_bit
      0.32 ±  4%      -0.2        0.11 ±  9%  perf-profile.self.cycles-pp.__frontswap_store
      0.46 ±  4%      -0.2        0.27 ±  8%  perf-profile.self.cycles-pp.bio_init
      0.32 ±  5%      -0.2        0.16 ±  9%  perf-profile.self.cycles-pp.native_flush_tlb_multi
      0.40 ±  7%      -0.2        0.24 ± 10%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.31 ± 16%      -0.1        0.17 ± 21%  perf-profile.self.cycles-pp.page_lock_anon_vma_read
      0.19 ± 13%      -0.1        0.07 ±  5%  perf-profile.self.cycles-pp.scan_swap_map_slots
      0.28 ±  9%      -0.1        0.17 ± 11%  perf-profile.self.cycles-pp.__set_page_dirty_no_writeback
      0.36 ±  5%      -0.1        0.25 ±  3%  perf-profile.self.cycles-pp.shrink_page_list
      0.33 ±  8%      -0.1        0.22 ±  6%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.26 ±  8%      -0.1        0.16 ±  4%  perf-profile.self.cycles-pp.isolate_lru_pages
      0.56 ±  6%      -0.1        0.47 ±  4%  perf-profile.self.cycles-pp.cpuidle_enter_state
      0.29 ± 12%      -0.1        0.21 ± 17%  perf-profile.self.cycles-pp.get_page_from_freelist
      0.16 ±  6%      -0.1        0.08 ±  9%  perf-profile.self.cycles-pp.end_page_writeback
      0.29 ± 11%      -0.1        0.21 ± 12%  perf-profile.self.cycles-pp.__mem_cgroup_charge
      0.20 ±  5%      -0.1        0.13 ±  6%  perf-profile.self.cycles-pp.__mod_node_page_state
      0.22 ±  3%      -0.1        0.15 ±  6%  perf-profile.self.cycles-pp.xas_create
      0.23 ±  8%      -0.1        0.16 ±  7%  perf-profile.self.cycles-pp.__mod_memcg_lruvec_state
      0.13 ±  9%      -0.1        0.06 ±  7%  perf-profile.self.cycles-pp.rmqueue_bulk
      0.17 ±  9%      -0.1        0.10 ±  9%  perf-profile.self.cycles-pp.__mod_lruvec_page_state
      0.14 ±  8%      -0.1        0.08 ±  7%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.15 ±  8%      -0.1        0.09 ±  7%  perf-profile.self.cycles-pp.__test_set_page_writeback
      0.08 ±  9%      -0.1        0.02 ± 99%  perf-profile.self.cycles-pp.__swap_writepage
      0.13 ± 12%      -0.1        0.08 ±  9%  perf-profile.self.cycles-pp.free_pcppages_bulk
      0.11 ± 12%      -0.0        0.06 ± 47%  perf-profile.self.cycles-pp.page_counter_try_charge
      0.15 ± 11%      -0.0        0.11 ±  7%  perf-profile.self.cycles-pp.page_mapping
      0.12 ± 12%      -0.0        0.08 ± 19%  perf-profile.self.cycles-pp.__pagevec_lru_add
      0.08 ±  8%      -0.0        0.04 ± 45%  perf-profile.self.cycles-pp.pagevec_lru_move_fn
      0.12 ±  6%      -0.0        0.08 ±  8%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.10 ± 10%      -0.0        0.07 ± 16%  perf-profile.self.cycles-pp.pagevec_move_tail_fn
      0.10 ± 10%      -0.0        0.06 ± 14%  perf-profile.self.cycles-pp.mutex_unlock
      0.08 ± 19%      -0.0        0.04 ± 45%  perf-profile.self.cycles-pp.___might_sleep
      0.11 ±  8%      -0.0        0.08 ± 12%  perf-profile.self.cycles-pp.update_sd_lb_stats
      0.09 ± 13%      -0.0        0.06 ±  7%  perf-profile.self.cycles-pp.__hrtimer_next_event_base
      0.08 ±  7%      -0.0        0.06 ±  8%  perf-profile.self.cycles-pp.memset_erms
      0.07 ± 11%      -0.0        0.04 ± 45%  perf-profile.self.cycles-pp.rotate_reclaimable_page
      0.06 ±  7%      -0.0        0.04 ± 45%  perf-profile.self.cycles-pp.update_irq_load_avg
      0.08 ±  4%      -0.0        0.06 ±  7%  perf-profile.self.cycles-pp.ll_back_merge_fn
      0.16 ±  5%      +0.1        0.21 ±  5%  perf-profile.self.cycles-pp.asm_sysvec_call_function
      1.01 ±  7%      +0.2        1.20 ±  6%  perf-profile.self.cycles-pp.native_flush_tlb_local
      2.22 ±  6%      +0.4        2.59 ±  4%  perf-profile.self.cycles-pp.flush_smp_call_function_queue
      1.28 ±  5%      +0.4        1.72        perf-profile.self.cycles-pp._raw_spin_lock
      1.59 ±  8%      +0.6        2.15 ±  3%  perf-profile.self.cycles-pp.__default_send_IPI_dest_field
      0.34 ± 35%      +0.6        0.93 ±  8%  perf-profile.self.cycles-pp.page_vma_mapped_walk
      1.35 ± 14%     +20.2       21.54 ±  4%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
     11952 ± 12%     -27.8%       8625 ± 13%  softirqs.CPU0.RCU
     32374 ±  3%     +30.1%      42116 ±  2%  softirqs.CPU0.SCHED
     10010 ± 15%     -33.0%       6705 ± 17%  softirqs.CPU1.RCU
     29521 ±  3%     +25.7%      37120 ±  8%  softirqs.CPU1.SCHED
      9560 ± 18%     -35.2%       6197 ± 12%  softirqs.CPU10.RCU
      9641 ± 15%     -42.8%       5512 ± 13%  softirqs.CPU100.RCU
     27690 ±  2%     +31.8%      36488 ±  3%  softirqs.CPU100.SCHED
      9404 ± 16%     -37.4%       5884 ± 14%  softirqs.CPU101.RCU
     26702 ±  4%     +39.4%      37210 ±  4%  softirqs.CPU101.SCHED
      9636 ± 16%     -40.6%       5728 ± 15%  softirqs.CPU102.RCU
     26856 ±  6%     +37.4%      36889 ±  4%  softirqs.CPU102.SCHED
      9761 ± 16%     -42.7%       5591 ± 14%  softirqs.CPU103.RCU
     26740 ±  3%     +32.4%      35414 ±  8%  softirqs.CPU103.SCHED
      9279 ± 16%     -34.8%       6049 ± 14%  softirqs.CPU104.RCU
     26363 ±  5%     +36.1%      35872 ±  9%  softirqs.CPU104.SCHED
      9410 ± 16%     -38.5%       5787 ± 15%  softirqs.CPU105.RCU
     26533 ±  9%     +41.5%      37539 ±  3%  softirqs.CPU105.SCHED
      9764 ± 26%     -39.6%       5894 ±  9%  softirqs.CPU106.RCU
     26912 ±  4%     +38.8%      37354 ±  5%  softirqs.CPU106.SCHED
      9551 ± 18%     -39.0%       5824 ± 12%  softirqs.CPU107.RCU
     27262 ±  5%     +28.8%      35113 ±  4%  softirqs.CPU107.SCHED
      9607 ± 17%     -37.9%       5968 ± 16%  softirqs.CPU108.RCU
     26904 ±  3%     +34.2%      36105 ±  3%  softirqs.CPU108.SCHED
      9485 ± 18%     -43.7%       5341 ±  7%  softirqs.CPU109.RCU
     26209 ±  6%     +38.5%      36295 ±  2%  softirqs.CPU109.SCHED
     28056 ±  6%     +36.9%      38396 ±  5%  softirqs.CPU11.SCHED
      9807 ± 14%     -39.4%       5942 ± 11%  softirqs.CPU110.RCU
     26349 ±  4%     +36.5%      35954 ±  5%  softirqs.CPU110.SCHED
      9271 ± 15%     -40.6%       5506 ± 11%  softirqs.CPU111.RCU
     26559 ±  8%     +36.5%      36243 ±  4%  softirqs.CPU111.SCHED
      9830 ± 20%     -41.0%       5804 ± 14%  softirqs.CPU112.RCU
     26422 ±  5%     +36.5%      36058 ±  7%  softirqs.CPU112.SCHED
      9658 ± 19%     -37.0%       6088 ± 20%  softirqs.CPU113.RCU
     26912 ±  2%     +38.2%      37204 ±  5%  softirqs.CPU113.SCHED
      9835 ± 17%     -41.0%       5798 ± 13%  softirqs.CPU114.RCU
     27485 ±  2%     +32.6%      36438 ±  7%  softirqs.CPU114.SCHED
      9878 ± 16%     -38.1%       6114 ± 16%  softirqs.CPU115.RCU
     27028 ±  3%     +34.2%      36262 ±  4%  softirqs.CPU115.SCHED
      9542 ± 17%     -37.5%       5967 ± 11%  softirqs.CPU116.RCU
     26216 ±  7%     +43.6%      37650 ±  4%  softirqs.CPU116.SCHED
      9457 ± 18%     -39.8%       5691 ± 12%  softirqs.CPU117.RCU
     27605 ±  5%     +34.0%      36982 ±  3%  softirqs.CPU117.SCHED
      9567 ± 17%     -41.5%       5594 ± 15%  softirqs.CPU118.RCU
     28212 ±  2%     +31.3%      37047 ±  5%  softirqs.CPU118.SCHED
      9858 ± 16%     -39.7%       5943 ± 13%  softirqs.CPU119.RCU
     28244 ±  4%     +32.8%      37516 ±  4%  softirqs.CPU119.SCHED
     10429 ± 16%     -31.2%       7175 ± 16%  softirqs.CPU12.RCU
     27864 ±  3%     +33.1%      37090 ± 11%  softirqs.CPU12.SCHED
      9672 ± 15%     -38.5%       5944 ± 13%  softirqs.CPU120.RCU
     27469 ±  3%     +25.6%      34512 ±  9%  softirqs.CPU120.SCHED
      9419 ± 17%     -37.5%       5884 ± 16%  softirqs.CPU121.RCU
     26369 ±  7%     +24.5%      32820 ±  7%  softirqs.CPU121.SCHED
      9627 ± 17%     -41.6%       5626 ± 14%  softirqs.CPU122.RCU
     27261 ±  4%     +37.2%      37408 ±  5%  softirqs.CPU122.SCHED
      9753 ± 17%     -40.5%       5799 ± 12%  softirqs.CPU123.RCU
     27689 ±  2%     +32.5%      36691 ±  7%  softirqs.CPU123.SCHED
     10024 ± 18%     -40.7%       5942 ± 11%  softirqs.CPU124.RCU
     27738 ±  2%     +31.2%      36403 ±  6%  softirqs.CPU124.SCHED
      9669 ± 15%     -38.7%       5930 ± 14%  softirqs.CPU125.RCU
     27931 ±  2%     +29.1%      36046 ±  7%  softirqs.CPU125.SCHED
      9931 ± 20%     -41.1%       5849 ± 13%  softirqs.CPU126.RCU
     27307 ±  3%     +33.4%      36414 ±  7%  softirqs.CPU126.SCHED
      9518 ± 14%     -39.6%       5745 ± 17%  softirqs.CPU127.RCU
     27125 ±  5%     +32.5%      35941 ±  9%  softirqs.CPU127.SCHED
      9714 ± 16%     -39.9%       5837 ± 10%  softirqs.CPU128.RCU
     27767 ±  3%     +32.7%      36842 ±  4%  softirqs.CPU128.SCHED
      9659 ± 14%     -40.8%       5718 ± 10%  softirqs.CPU129.RCU
     27759 ±  3%     +30.9%      36346 ±  4%  softirqs.CPU129.SCHED
     10789 ± 26%     -46.1%       5820 ±  9%  softirqs.CPU13.RCU
     26247 ± 15%     +45.6%      38218 ±  4%  softirqs.CPU13.SCHED
      9600 ± 15%     -38.2%       5930 ± 16%  softirqs.CPU130.RCU
     27262 ±  4%     +34.1%      36562 ±  6%  softirqs.CPU130.SCHED
     10393 ± 20%     -45.2%       5699 ± 14%  softirqs.CPU131.RCU
     27624 ±  4%     +33.4%      36844 ±  3%  softirqs.CPU131.SCHED
      9593 ± 14%     -40.2%       5737 ± 14%  softirqs.CPU132.RCU
     26882 ±  9%     +34.5%      36151 ±  2%  softirqs.CPU132.SCHED
      9524 ± 17%     -39.6%       5749 ± 12%  softirqs.CPU133.RCU
     27165 ±  5%     +31.0%      35586 ±  7%  softirqs.CPU133.SCHED
      9714 ± 12%     -41.1%       5726 ± 14%  softirqs.CPU134.RCU
     27285 ±  3%     +29.3%      35284 ±  5%  softirqs.CPU134.SCHED
      9561 ± 13%     -38.3%       5901 ± 19%  softirqs.CPU135.RCU
     27567 ±  3%     +28.9%      35546 ±  7%  softirqs.CPU135.SCHED
      9577 ± 14%     -40.4%       5711 ± 12%  softirqs.CPU136.RCU
     28030 ±  3%     +31.5%      36846 ±  5%  softirqs.CPU136.SCHED
      9717 ± 16%     -40.6%       5770 ± 15%  softirqs.CPU137.RCU
     28225 ±  3%     +31.7%      37183 ±  6%  softirqs.CPU137.SCHED
      9676 ± 14%     -41.9%       5624 ± 13%  softirqs.CPU138.RCU
     27095 ±  3%     +38.9%      37642 ±  6%  softirqs.CPU138.SCHED
      9678 ± 17%     -40.0%       5805 ± 16%  softirqs.CPU139.RCU
     27639 ±  2%     +29.6%      35811 ± 10%  softirqs.CPU139.SCHED
      9842 ± 16%     -40.1%       5894 ± 11%  softirqs.CPU14.RCU
     25932 ± 13%     +47.9%      38346 ±  3%  softirqs.CPU14.SCHED
      9769 ± 14%     -40.6%       5807 ± 14%  softirqs.CPU140.RCU
     28046 ±  3%     +29.1%      36200 ±  8%  softirqs.CPU140.SCHED
      9640 ± 15%     -39.3%       5847 ± 15%  softirqs.CPU141.RCU
      9629 ± 13%     -38.5%       5924 ± 12%  softirqs.CPU142.RCU
     27220 ±  3%     +34.6%      36638 ±  5%  softirqs.CPU142.SCHED
      9691 ± 16%     -40.7%       5750 ± 13%  softirqs.CPU143.RCU
     27774 ±  3%     +31.3%      36455 ±  5%  softirqs.CPU143.SCHED
      9922 ± 17%     -38.1%       6143 ± 12%  softirqs.CPU144.RCU
     26996 ±  4%     +27.2%      34330 ±  4%  softirqs.CPU144.SCHED
      9626 ± 17%     -40.9%       5693 ± 13%  softirqs.CPU145.RCU
      9475 ± 15%     -41.8%       5519 ± 11%  softirqs.CPU146.RCU
     26918 ±  9%     +35.7%      36521 ±  6%  softirqs.CPU146.SCHED
     10026 ± 13%     -43.5%       5660 ± 14%  softirqs.CPU147.RCU
     27369 ±  3%     +33.8%      36627 ±  5%  softirqs.CPU147.SCHED
     10079 ± 12%     -42.3%       5813 ± 11%  softirqs.CPU148.RCU
     26659 ±  2%     +41.8%      37791 ±  2%  softirqs.CPU148.SCHED
      9605 ± 14%     -39.7%       5796 ± 12%  softirqs.CPU149.RCU
     27454 ±  2%     +36.1%      37356 ±  7%  softirqs.CPU149.SCHED
      9534 ± 17%     -36.3%       6071 ± 10%  softirqs.CPU15.RCU
     28826 ±  2%     +33.7%      38554 ±  4%  softirqs.CPU15.SCHED
      9821 ± 14%     -35.8%       6309 ± 29%  softirqs.CPU150.RCU
     27391 ±  4%     +34.6%      36880 ±  6%  softirqs.CPU150.SCHED
      9559 ± 15%     -41.6%       5585 ± 11%  softirqs.CPU151.RCU
     26371 ±  2%     +37.9%      36354 ±  6%  softirqs.CPU151.SCHED
      9662 ± 14%     -38.1%       5984 ± 16%  softirqs.CPU152.RCU
     26885 ±  4%     +30.6%      35100 ± 10%  softirqs.CPU152.SCHED
      9520 ± 14%     -40.5%       5664 ± 13%  softirqs.CPU153.RCU
     27614 ±  4%     +33.9%      36988 ±  5%  softirqs.CPU153.SCHED
      9494 ± 14%     -40.8%       5622 ± 13%  softirqs.CPU154.RCU
     27383           +31.7%      36070 ±  7%  softirqs.CPU154.SCHED
      9623 ± 13%     -40.8%       5699 ± 12%  softirqs.CPU155.RCU
     28108 ±  2%     +28.0%      35984 ±  8%  softirqs.CPU155.SCHED
      9297 ± 15%     -38.2%       5747 ± 14%  softirqs.CPU156.RCU
     28227 ±  4%     +28.6%      36312 ±  6%  softirqs.CPU156.SCHED
      9421 ± 13%     -41.2%       5535 ± 15%  softirqs.CPU157.RCU
     26196 ± 11%     +38.9%      36389 ±  6%  softirqs.CPU157.SCHED
      9572 ± 15%     -42.3%       5518 ± 14%  softirqs.CPU158.RCU
     27901 ±  2%     +31.3%      36647 ±  7%  softirqs.CPU158.SCHED
      9195 ± 17%     -38.7%       5634 ± 17%  softirqs.CPU159.RCU
     27962 ±  2%     +30.4%      36469 ±  6%  softirqs.CPU159.SCHED
      9822 ± 15%     -32.6%       6617 ± 18%  softirqs.CPU16.RCU
     26952 ±  5%     +39.9%      37703 ±  8%  softirqs.CPU16.SCHED
      9209 ± 20%     -41.9%       5355 ± 13%  softirqs.CPU160.RCU
     27793 ±  2%     +29.8%      36073 ±  9%  softirqs.CPU160.SCHED
      9543 ± 16%     -40.5%       5678 ± 13%  softirqs.CPU161.RCU
     28608 ±  2%     +28.7%      36806 ±  3%  softirqs.CPU161.SCHED
      9399 ± 17%     -39.8%       5662 ± 12%  softirqs.CPU162.RCU
     28119 ±  2%     +31.9%      37092 ±  8%  softirqs.CPU162.SCHED
      9334 ± 17%     -40.3%       5568 ± 13%  softirqs.CPU163.RCU
     28083           +32.2%      37136 ±  5%  softirqs.CPU163.SCHED
      9558 ± 14%     -43.1%       5438 ± 11%  softirqs.CPU164.RCU
     28339 ±  2%     +30.8%      37066 ±  7%  softirqs.CPU164.SCHED
      9522 ± 15%     -42.4%       5485 ± 12%  softirqs.CPU165.RCU
     28064 ±  3%     +32.6%      37204 ±  7%  softirqs.CPU165.SCHED
      9509 ± 17%     -41.0%       5609 ± 15%  softirqs.CPU166.RCU
     27120 ±  8%     +33.2%      36135 ±  7%  softirqs.CPU166.SCHED
      9335 ± 16%     -41.2%       5485 ± 11%  softirqs.CPU167.RCU
     28405 ±  2%     +27.8%      36306 ±  3%  softirqs.CPU167.SCHED
     25148 ±  4%     +38.7%      34890 ±  5%  softirqs.CPU168.SCHED
      8770 ± 17%     -40.3%       5237 ± 10%  softirqs.CPU169.RCU
     10241 ± 25%     -36.9%       6461 ± 16%  softirqs.CPU17.RCU
     26802 ±  6%     +34.2%      35979 ± 10%  softirqs.CPU17.SCHED
      9208 ± 15%     -42.2%       5318 ± 11%  softirqs.CPU170.RCU
     27330 ±  4%     +23.0%      33604 ± 10%  softirqs.CPU170.SCHED
      8924 ± 17%     -39.5%       5400 ±  9%  softirqs.CPU171.RCU
     28100 ±  3%     +31.5%      36959 ±  4%  softirqs.CPU171.SCHED
      8924 ± 17%     -38.0%       5529 ± 10%  softirqs.CPU172.RCU
     28259 ±  3%     +31.1%      37040 ±  4%  softirqs.CPU172.SCHED
      8875 ± 16%     -41.0%       5232 ± 13%  softirqs.CPU173.RCU
     28254 ±  2%     +32.8%      37523 ±  4%  softirqs.CPU173.SCHED
      9176 ± 15%     -42.1%       5317 ± 13%  softirqs.CPU174.RCU
     27499 ±  3%     +38.5%      38081 ±  3%  softirqs.CPU174.SCHED
      8654 ± 15%     -38.4%       5333 ± 13%  softirqs.CPU175.RCU
     27692 ±  2%     +35.4%      37507 ±  3%  softirqs.CPU175.SCHED
      9225 ± 14%     -41.6%       5389 ± 15%  softirqs.CPU176.RCU
     27375 ±  4%     +34.9%      36923 ±  8%  softirqs.CPU176.SCHED
      9289 ± 15%     -43.5%       5249 ± 13%  softirqs.CPU177.RCU
     28263 ±  3%     +32.0%      37296 ±  6%  softirqs.CPU177.SCHED
      9471 ± 14%     -41.8%       5510 ± 15%  softirqs.CPU178.RCU
     28202 ±  3%     +29.6%      36560 ±  5%  softirqs.CPU178.SCHED
      9263 ± 13%     -40.8%       5486 ± 15%  softirqs.CPU179.RCU
     27974 ±  4%     +32.7%      37116 ±  4%  softirqs.CPU179.SCHED
      9755 ± 20%     -38.9%       5961 ± 14%  softirqs.CPU18.RCU
     28714 ±  2%     +33.8%      38408 ±  5%  softirqs.CPU18.SCHED
      9532 ± 17%     -42.7%       5461 ± 17%  softirqs.CPU180.RCU
     28337 ±  2%     +29.8%      36778 ±  3%  softirqs.CPU180.SCHED
      8762 ± 15%     -39.3%       5319 ± 14%  softirqs.CPU181.RCU
     27484 ±  3%     +33.9%      36801 ±  2%  softirqs.CPU181.SCHED
      9105 ± 14%     -39.8%       5483 ± 13%  softirqs.CPU182.RCU
     27679 ±  4%     +33.4%      36912 ±  4%  softirqs.CPU182.SCHED
      9146 ± 16%     -41.3%       5367 ± 15%  softirqs.CPU183.RCU
      9449 ± 14%     -38.4%       5822 ± 25%  softirqs.CPU184.RCU
     28179 ±  2%     +34.7%      37964 ±  3%  softirqs.CPU184.SCHED
      9703 ± 17%     -42.4%       5590 ± 21%  softirqs.CPU185.RCU
     28451           +35.1%      38440 ±  3%  softirqs.CPU185.SCHED
      8740 ± 17%     -40.4%       5210 ± 11%  softirqs.CPU186.RCU
     27970 ±  3%     +33.9%      37441 ±  3%  softirqs.CPU186.SCHED
      9322 ± 15%     -42.1%       5401 ± 14%  softirqs.CPU187.RCU
      9517 ± 11%     -44.2%       5313 ± 14%  softirqs.CPU188.RCU
     27852 ±  3%     +34.4%      37436 ±  5%  softirqs.CPU188.SCHED
      9375 ± 12%     -42.7%       5375 ± 16%  softirqs.CPU189.RCU
     27896 ±  3%     +34.7%      37573 ±  4%  softirqs.CPU189.SCHED
      9548 ± 17%     -38.0%       5921 ± 12%  softirqs.CPU19.RCU
     28340 ±  2%     +34.4%      38104 ±  5%  softirqs.CPU19.SCHED
      9329 ± 14%     -41.7%       5437 ± 16%  softirqs.CPU190.RCU
     28512 ±  3%     +30.6%      37248 ±  3%  softirqs.CPU190.SCHED
      9956 ± 12%     -38.6%       6112 ± 20%  softirqs.CPU191.RCU
      9920 ± 17%     -35.0%       6447 ± 12%  softirqs.CPU2.RCU
      9792 ± 16%     -38.1%       6058 ± 24%  softirqs.CPU20.RCU
      9552 ± 17%     -37.0%       6016 ± 18%  softirqs.CPU21.RCU
     28148           +36.7%      38489 ±  6%  softirqs.CPU21.SCHED
     10016 ± 14%     -39.9%       6023 ± 17%  softirqs.CPU22.RCU
     10216 ± 25%     -42.6%       5863 ± 11%  softirqs.CPU23.RCU
     28979 ±  3%     +32.4%      38368 ±  8%  softirqs.CPU23.SCHED
     10358 ± 15%     -38.4%       6381 ± 11%  softirqs.CPU24.RCU
     26220 ±  4%     +37.7%      36110 ±  8%  softirqs.CPU24.SCHED
      9726 ± 15%     -35.9%       6235 ± 14%  softirqs.CPU25.RCU
     10261 ± 14%     -41.7%       5986 ± 14%  softirqs.CPU26.RCU
     27998 ±  6%     +36.6%      38241 ±  6%  softirqs.CPU26.SCHED
      9785 ± 14%     -35.1%       6354 ± 13%  softirqs.CPU27.RCU
     28204 ±  4%     +31.8%      37162 ±  7%  softirqs.CPU27.SCHED
     10151 ± 16%     -39.1%       6183 ± 17%  softirqs.CPU28.RCU
     27983 ±  4%     +35.9%      38026 ±  6%  softirqs.CPU28.SCHED
     10092 ± 14%     -40.2%       6036 ± 15%  softirqs.CPU29.RCU
     28593 ±  3%     +34.5%      38465 ±  4%  softirqs.CPU29.SCHED
     10086 ± 18%     -34.8%       6575 ± 26%  softirqs.CPU3.RCU
     27482 ±  7%     +40.4%      38591 ±  3%  softirqs.CPU3.SCHED
      9883 ± 16%     -37.0%       6229 ± 15%  softirqs.CPU30.RCU
     29738 ±  7%     +28.3%      38155 ±  4%  softirqs.CPU30.SCHED
      9925 ± 17%     -38.8%       6077 ± 16%  softirqs.CPU31.RCU
     10305 ± 14%     -39.1%       6275 ± 12%  softirqs.CPU32.RCU
     28109 ±  3%     +34.8%      37891 ±  4%  softirqs.CPU32.SCHED
     10590 ± 19%     -40.9%       6259 ± 17%  softirqs.CPU33.RCU
     28261 ±  2%     +26.6%      35778 ±  7%  softirqs.CPU33.SCHED
     10001 ± 16%     -38.6%       6142 ± 12%  softirqs.CPU34.RCU
     28560           +35.0%      38550 ±  3%  softirqs.CPU34.SCHED
     10201 ± 15%     -38.8%       6247 ± 12%  softirqs.CPU35.RCU
     28600 ±  3%     +33.1%      38059 ±  3%  softirqs.CPU35.SCHED
     10116 ± 13%     -38.1%       6260 ± 13%  softirqs.CPU36.RCU
     10122 ± 15%     -40.3%       6039 ± 14%  softirqs.CPU37.RCU
     28116 ±  3%     +37.2%      38569 ±  3%  softirqs.CPU37.SCHED
     10111 ± 13%     -39.1%       6158 ± 14%  softirqs.CPU38.RCU
     28486           +33.3%      37962 ±  4%  softirqs.CPU38.SCHED
     10005 ± 14%     -38.3%       6174 ± 15%  softirqs.CPU39.RCU
      9741 ± 16%     -41.7%       5676 ±  7%  softirqs.CPU4.RCU
     28352 ±  3%     +35.1%      38298 ±  4%  softirqs.CPU4.SCHED
     10184 ± 12%     -40.0%       6115 ± 13%  softirqs.CPU40.RCU
     28728 ±  2%     +33.1%      38233 ±  4%  softirqs.CPU40.SCHED
     10194 ± 13%     -39.6%       6162 ± 13%  softirqs.CPU41.RCU
     28822           +31.2%      37804 ±  5%  softirqs.CPU41.SCHED
     10618 ± 18%     -41.7%       6195 ± 13%  softirqs.CPU42.RCU
     28518           +36.7%      38997 ±  3%  softirqs.CPU42.SCHED
     10139 ± 15%     -38.7%       6218 ± 14%  softirqs.CPU43.RCU
     28442 ±  2%     +31.5%      37398 ±  8%  softirqs.CPU43.SCHED
     10142 ± 13%     -39.8%       6108 ± 11%  softirqs.CPU44.RCU
     10157 ± 14%     -35.7%       6529 ± 19%  softirqs.CPU45.RCU
     28301 ±  2%     +31.6%      37252 ±  8%  softirqs.CPU45.SCHED
     10304 ± 16%     -40.1%       6174 ± 18%  softirqs.CPU46.RCU
     28728 ±  2%     +34.4%      38599 ±  4%  softirqs.CPU46.SCHED
     10163 ± 14%     -39.6%       6136 ± 14%  softirqs.CPU47.RCU
     28510 ±  5%     +33.3%      37991 ±  3%  softirqs.CPU47.SCHED
     10751 ± 15%     -39.5%       6504 ± 18%  softirqs.CPU48.RCU
     24789 ± 10%     +37.8%      34152 ±  5%  softirqs.CPU48.SCHED
     10001 ± 16%     -39.8%       6017 ± 12%  softirqs.CPU49.RCU
     27400 ±  4%     +28.0%      35059 ± 10%  softirqs.CPU49.SCHED
      9899 ± 16%     -37.8%       6155 ± 11%  softirqs.CPU5.RCU
     25043 ± 26%     +52.9%      38283 ±  4%  softirqs.CPU5.SCHED
      9941 ± 15%     -40.9%       5880 ± 13%  softirqs.CPU50.RCU
     27850 ±  5%     +33.9%      37300 ±  5%  softirqs.CPU50.SCHED
     10580 ± 13%     -42.0%       6131 ± 13%  softirqs.CPU51.RCU
     28440 ±  2%     +34.7%      38294 ±  4%  softirqs.CPU51.SCHED
     10532 ± 11%     -41.7%       6144 ± 12%  softirqs.CPU52.RCU
     27522 ±  3%     +40.1%      38549 ±  3%  softirqs.CPU52.SCHED
     10169 ± 14%     -39.9%       6116 ± 11%  softirqs.CPU53.RCU
     28015 ±  2%     +38.0%      38675 ±  6%  softirqs.CPU53.SCHED
     27943 ±  3%     +36.6%      38162 ±  5%  softirqs.CPU54.SCHED
      9711 ± 15%     -39.8%       5848 ± 12%  softirqs.CPU55.RCU
     28057 ±  2%     +32.3%      37105 ±  6%  softirqs.CPU55.SCHED
      9893 ± 14%     -39.3%       6004 ± 11%  softirqs.CPU56.RCU
     28221 ±  2%     +34.3%      37914 ±  6%  softirqs.CPU56.SCHED
     10168 ± 16%     -40.7%       6028 ± 12%  softirqs.CPU57.RCU
     28823 ±  2%     +32.2%      38113 ±  6%  softirqs.CPU57.SCHED
     10014 ± 14%     -39.8%       6033 ± 14%  softirqs.CPU58.RCU
     28179 ±  2%     +33.0%      37485 ±  6%  softirqs.CPU58.SCHED
     10303 ± 13%     -41.0%       6083 ± 15%  softirqs.CPU59.RCU
     29026           +32.1%      38345 ±  4%  softirqs.CPU59.SCHED
     10178 ± 14%     -34.1%       6709 ± 13%  softirqs.CPU6.RCU
     29174 ± 10%     +32.6%      38682 ±  3%  softirqs.CPU6.SCHED
      9753 ± 15%     -37.9%       6057 ± 13%  softirqs.CPU60.RCU
     28862 ±  2%     +32.8%      38320 ±  4%  softirqs.CPU60.SCHED
      9816 ± 13%     -40.9%       5805 ± 14%  softirqs.CPU61.RCU
     28572 ±  2%     +35.2%      38638 ±  3%  softirqs.CPU61.SCHED
     10000 ± 14%     -41.2%       5876 ± 13%  softirqs.CPU62.RCU
     28515 ±  2%     +34.6%      38377 ±  4%  softirqs.CPU62.SCHED
      9722 ± 16%     -35.3%       6293 ± 18%  softirqs.CPU63.RCU
     28871           +32.9%      38371 ±  3%  softirqs.CPU63.SCHED
      9335 ± 17%     -39.8%       5620 ± 13%  softirqs.CPU64.RCU
      9671 ± 16%     -38.8%       5918 ± 11%  softirqs.CPU65.RCU
      9635 ± 16%     -38.9%       5884 ± 10%  softirqs.CPU66.RCU
      9620 ± 17%     -40.7%       5702 ± 11%  softirqs.CPU67.RCU
     23386 ± 35%     +66.2%      38859 ±  3%  softirqs.CPU67.SCHED
      9840 ± 16%     -42.5%       5661 ± 11%  softirqs.CPU68.RCU
     28877 ±  3%     +31.9%      38083 ±  7%  softirqs.CPU68.SCHED
      9621 ± 16%     -40.2%       5750 ± 10%  softirqs.CPU69.RCU
     28471 ±  4%     +34.2%      38218 ±  4%  softirqs.CPU69.SCHED
      9500 ± 19%     -37.7%       5920 ± 11%  softirqs.CPU7.RCU
     27971 ±  2%     +27.6%      35679 ± 12%  softirqs.CPU7.SCHED
      9950 ± 19%     -42.6%       5713 ± 11%  softirqs.CPU70.RCU
     28378 ±  4%     +36.4%      38698 ±  4%  softirqs.CPU70.SCHED
      9769 ± 17%     -41.5%       5712 ± 11%  softirqs.CPU71.RCU
     28400 ±  3%     +35.0%      38334 ±  2%  softirqs.CPU71.SCHED
     10186 ± 23%     -36.0%       6516 ± 11%  softirqs.CPU72.RCU
     29528 ±  5%     +19.4%      35245 ±  3%  softirqs.CPU72.SCHED
      9117 ± 16%     -37.7%       5682 ± 11%  softirqs.CPU73.RCU
      8864 ± 16%     -37.6%       5527 ± 10%  softirqs.CPU74.RCU
      9191 ± 15%     -40.4%       5474 ± 10%  softirqs.CPU75.RCU
     28423 ±  4%     +28.2%      36424 ±  8%  softirqs.CPU75.SCHED
      9557 ± 20%     -44.1%       5341 ± 10%  softirqs.CPU76.RCU
     28240 ±  3%     +38.6%      39129 ±  3%  softirqs.CPU76.SCHED
      9108 ± 14%     -40.6%       5408 ± 11%  softirqs.CPU77.RCU
     28923 ±  2%     +35.5%      39203 ±  2%  softirqs.CPU77.SCHED
      9485 ± 16%     -41.3%       5571 ± 13%  softirqs.CPU78.RCU
     28339 ±  2%     +38.9%      39369 ±  3%  softirqs.CPU78.SCHED
      8889 ± 15%     -39.2%       5401 ± 12%  softirqs.CPU79.RCU
     28802 ±  2%     +35.8%      39116 ±  3%  softirqs.CPU79.SCHED
     28293 ±  2%     +32.1%      37372 ±  7%  softirqs.CPU8.SCHED
      9386 ± 13%     -40.2%       5612 ± 14%  softirqs.CPU80.RCU
     28492 ±  3%     +36.6%      38911 ±  2%  softirqs.CPU80.SCHED
      9460 ± 14%     -41.5%       5535 ± 13%  softirqs.CPU81.RCU
     28808           +33.5%      38470 ±  5%  softirqs.CPU81.SCHED
      9799 ± 14%     -41.7%       5717 ± 15%  softirqs.CPU82.RCU
     28638 ±  3%     +33.8%      38305 ±  3%  softirqs.CPU82.SCHED
      9574 ± 13%     -40.1%       5731 ± 14%  softirqs.CPU83.RCU
     28754           +34.1%      38564 ±  3%  softirqs.CPU83.SCHED
      9488 ± 18%     -41.6%       5541 ± 15%  softirqs.CPU84.RCU
     28843           +34.3%      38733 ±  4%  softirqs.CPU84.SCHED
      9072 ± 15%     -39.8%       5465 ± 13%  softirqs.CPU85.RCU
     28412 ±  3%     +37.4%      39033 ±  2%  softirqs.CPU85.SCHED
      9680 ± 17%     -41.2%       5696 ± 15%  softirqs.CPU86.RCU
     28536 ±  2%     +35.4%      38644 ±  2%  softirqs.CPU86.SCHED
      9251 ± 16%     -37.9%       5749 ± 13%  softirqs.CPU87.RCU
     28704           +34.3%      38561 ±  3%  softirqs.CPU87.SCHED
      9761 ± 14%     -40.2%       5842 ± 16%  softirqs.CPU88.RCU
     28799 ±  2%     +36.4%      39277 ±  4%  softirqs.CPU88.SCHED
      9979 ± 14%     -42.2%       5770 ± 14%  softirqs.CPU89.RCU
     28862 ±  3%     +36.8%      39497 ±  3%  softirqs.CPU89.SCHED
      9789 ± 16%     -40.8%       5798 ± 12%  softirqs.CPU9.RCU
     27851 ±  5%     +39.6%      38881 ±  5%  softirqs.CPU9.SCHED
      8905 ± 14%     -39.5%       5388 ± 11%  softirqs.CPU90.RCU
     28731 ±  2%     +36.1%      39093 ±  4%  softirqs.CPU90.SCHED
      9570 ± 13%     -38.1%       5924 ± 18%  softirqs.CPU91.RCU
      9375 ± 12%     -41.8%       5455 ± 14%  softirqs.CPU92.RCU
     28905           +34.1%      38760 ±  5%  softirqs.CPU92.SCHED
      9198 ± 15%     -35.9%       5893 ± 17%  softirqs.CPU93.RCU
      9560 ± 13%     -37.5%       5976 ± 13%  softirqs.CPU94.RCU
     29218 ±  2%     +28.9%      37660 ±  9%  softirqs.CPU94.SCHED
      9535 ± 13%     -40.6%       5667 ± 14%  softirqs.CPU95.RCU
      9324 ± 15%     -35.0%       6063 ± 21%  softirqs.CPU96.RCU
     25757 ±  5%     +25.0%      32203 ±  9%  softirqs.CPU96.SCHED
      9674 ± 14%     -40.8%       5729 ± 18%  softirqs.CPU97.RCU
      9371 ± 15%     -37.7%       5840 ± 13%  softirqs.CPU98.RCU
     26068 ±  4%     +38.4%      36065 ±  8%  softirqs.CPU98.SCHED
      9497 ± 15%     -41.0%       5602 ± 13%  softirqs.CPU99.RCU
     26556 ±  6%     +36.5%      36239 ±  3%  softirqs.CPU99.SCHED
   1862214 ± 15%     -39.5%    1127214 ± 12%  softirqs.RCU
   5342380           +32.5%    7080895 ±  3%  softirqs.SCHED
     42480 ±  3%     +28.2%      54463        softirqs.TIMER


                                                                                
                          vm-scalability.time.system_time                       
                                                                                
  1800 +--------------------------------------------------------------------+   
  1700 |-+         O                                  O        O            |   
       |        O     O        O OO  O    O     OO O     OO O     O      O  |   
  1600 |-OO  O O  O      O    O         O    OO     O  O     O       OO O   |   
  1500 |-+           O  O  OO       O  O   O                               O|   
       |    O                                                   O  O        |   
  1400 |-+                                                                  |   
  1300 |-+                                                                  |   
  1200 |-+                                                                  |   
       |                                                                    |   
  1100 |-+                                                                  |   
  1000 |-+                                                                  |   
       |.++. +.+ .++.++.  .+   +.    +   .++.+ .+                           |   
   900 |-+  +   +       ++  +.+  ++.+ + +     +  +                          |   
   800 +--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                  vm-scalability.time.percent_of_cpu_this_job_got               
                                                                                
  640 +---------------------------------------------------------------------+   
      |                         O                                           |   
  620 |-+                                                                O  |   
  600 |-O   O OO OO  O  O  O  O  O    OO O  O    OO  O    OO  O  O          |   
      |  O                O  O     OO         OO       OO           O OO  O |   
  580 |-+  O        O  O                   O        O        O  O           |   
      |                                                            O        |   
  560 |-+                                                                   |   
      |                                                                     |   
  540 |-+            +                  .+.    +.                           |   
  520 |.+   +.+ .++.+ +   +   +        +   ++.+  +                          |   
      |  +  :  +       + + +. :+ +.+ .+                                     |   
  500 |-+ +:            +    +  +   +                                       |   
      |    +                                                                |   
  480 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                         vm-scalability.time.elapsed_time                       
                                                                                
  340 +---------------------------------------------------------------------+   
      |                             O          O        O                   |   
  320 |-+      O  O  O           O               OO OO    OO OO       O     |   
      | OO  O    O      O    OO O      O O    O        O         O  O  O O  |   
  300 |-+  O  O     O  O  O        O  O    OO                      O      O |   
      |                    O                                    O           |   
  280 |-+                                                                   |   
      |                                                                     |   
  260 |-+                                                                   |   
      |                                                                     |   
  240 |-+                                                                   |   
      |  +.++.          +.  .+ .+   +       +.                              |   
  220 |.+     ++.++.++.+  ++  +  +.+ + +.+.+  ++.+                          |   
      |                               +                                     |   
  200 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                       vm-scalability.time.elapsed_time.max                     
                                                                                
  340 +---------------------------------------------------------------------+   
      |                             O          O        O                   |   
  320 |-+      O  O  O           O               OO OO    OO OO       O     |   
      | OO  O    O      O    OO O      O O    O        O         O  O  O O  |   
  300 |-+  O  O     O  O  O        O  O    OO                      O      O |   
      |                    O                                    O           |   
  280 |-+                                                                   |   
      |                                                                     |   
  260 |-+                                                                   |   
      |                                                                     |   
  240 |-+                                                                   |   
      |  +.++.          +.  .+ .+   +       +.                              |   
  220 |.+     ++.++.++.+  ++  +  +.+ + +.+.+  ++.+                          |   
      |                               +                                     |   
  200 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                               vm-scalability.throughput                        
                                                                                
  3.4e+06 +-----------------------------------------------------------------+   
          |                                                                 |   
  3.2e+06 |-+                           +                                   |   
          |       .+                 +  :+      .+                          |   
    3e+06 |+.    +  ++.++.+ .++. + .+ +:  +++. +  +                         |   
  2.8e+06 |-+++.+          +    + +    +      +                             |   
          |                                                                 |   
  2.6e+06 |-+                                                               |   
          |                                                                 |   
  2.4e+06 |-+                                                               |   
  2.2e+06 |-+                                                               |   
          |   O  O     O  O  OO      O  O   O O                  O O       O|   
    2e+06 |O+O  O  OO      O    OOO O     OO   O     O  O     O   O  OO OO  |   
          |          O  O              O         OO O O  O OO  O            |   
  1.8e+06 +-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                vm-scalability.median                           
                                                                                
  420000 +------------------------------------------------------------------+   
         |                              +                                   |   
  400000 |-+                            :                                   |   
  380000 |-+     .+                +.+ : :    .++.+                         |   
         |+.  .++  +.++.++ .++. +. :  :: +.+++                              |   
  360000 |-+++            +    +  +   +                                     |   
  340000 |-+                                                                |   
         |                                                                  |   
  320000 |-+                                                                |   
  300000 |-+                                                                |   
         |                                                                  |   
  280000 |-+          O      O                                  O           |   
  260000 |-+ O OO        OO O  O  O  O  OO  OO                     O O  OO O|   
         |O O     OO O  O       O  O       O   O  OO O  O  OOO    O   O     |   
  240000 +------------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


---
0DAY/LKP+ Test Infrastructure                   Open Source Technology Center
https://lists.01.org/hyperkitty/list/lkp@lists.01.org       Intel Corporation

Thanks,
Oliver Sang


View attachment "config-5.13.0-rc2-00032-gecf8443e51a8" of type "text/plain" (174118 bytes)

View attachment "job-script" of type "text/plain" (8608 bytes)

View attachment "job.yaml" of type "text/plain" (5967 bytes)

View attachment "reproduce" of type "text/plain" (944 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ