lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite for Android: free password hash cracker in your pocket
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date:	Sun, 14 Aug 2016 22:50:53 +0800
From:	Fengguang Wu <fengguang.wu@...el.com>
To:	Christoph Hellwig <hch@....de>
Cc:	Dave Chinner <david@...morbit.com>,
	Ye Xiaolong <xiaolong.ye@...el.com>,
	Linus Torvalds <torvalds@...ux-foundation.org>,
	LKML <linux-kernel@...r.kernel.org>,
	Bob Peterson <rpeterso@...hat.com>, LKP <lkp@...org>
Subject: Re: [LKP] [lkp] [xfs] 68a9f5e700: aim7.jobs-per-min -13.6% regression

Hi Christoph,

On Sun, Aug 14, 2016 at 06:51:28AM +0800, Fengguang Wu wrote:
>Hi Christoph,
>
>On Sun, Aug 14, 2016 at 12:15:08AM +0200, Christoph Hellwig wrote:
>>Hi Fengguang,
>>
>>feel free to try this git tree:
>>
>>   git://git.infradead.org/users/hch/vfs.git iomap-fixes
>
>I just queued some test jobs for it.
>
>% queue -q vip -t ivb44 -b hch-vfs/iomap-fixes aim7-fs-1brd.yaml fs=xfs -r3 -k fe9c2c81ed073878768785a985295cbacc349e42 -k ca2edab2e1d8f30dda874b7f717c2d4664991e9b -k 99091700659f4df965e138b38b4fa26a29b7eade
>
>That job file can be found here:
>
>        https://git.kernel.org/cgit/linux/kernel/git/wfg/lkp-tests.git/tree/jobs/aim7-fs-1brd.yaml
>
>It specifies a matrix of the below atom tests:
>
>        wfg /c/lkp-tests% split-job jobs/aim7-fs-1brd.yaml -s 'fs: xfs'
>
>        jobs/aim7-fs-1brd.yaml => ./aim7-fs-1brd-1BRD_48G-xfs-disk_src-3000-performance.yaml
>        jobs/aim7-fs-1brd.yaml => ./aim7-fs-1brd-1BRD_48G-xfs-disk_rr-3000-performance.yaml
>        jobs/aim7-fs-1brd.yaml => ./aim7-fs-1brd-1BRD_48G-xfs-disk_rw-3000-performance.yaml
>        jobs/aim7-fs-1brd.yaml => ./aim7-fs-1brd-1BRD_48G-xfs-disk_cp-3000-performance.yaml
>        jobs/aim7-fs-1brd.yaml => ./aim7-fs-1brd-1BRD_48G-xfs-disk_wrt-3000-performance.yaml
>        jobs/aim7-fs-1brd.yaml => ./aim7-fs-1brd-1BRD_48G-xfs-sync_disk_rw-600-performance.yaml
>        jobs/aim7-fs-1brd.yaml => ./aim7-fs-1brd-1BRD_48G-xfs-creat-clo-1500-performance.yaml
>        jobs/aim7-fs-1brd.yaml => ./aim7-fs-1brd-1BRD_48G-xfs-disk_rd-9000-performance.yaml

I got some results now. The several finished aim7 tests have some
performance regressions for commit fe9c2c81 ("xfs: rewrite and
optimize the delalloc write path") comparing to its parent commit
ca2edab2e and their base mainline commit 990917006 ("Merge tag
'nfs-for-4.8-2' of git://git.linux-nfs.org/projects/trondmy/linux-nfs").

wfg@inn ~% compare -g aim7 -ai 99091700659f4df965e138b38b4fa26a29b7eade ca2edab2e1d8f30dda874b7f717c2d4664991e9b fe9c2c81ed073878768785a985295cbacc349e42
tests: 4
    60      perf-index  fe9c2c81ed073878768785a985295cbacc349e42
    97     power-index  fe9c2c81ed073878768785a985295cbacc349e42

99091700659f4df9  ca2edab2e1d8f30dda874b7f71  fe9c2c81ed073878768785a985  testcase/testparams/testbox
----------------  --------------------------  --------------------------  ---------------------------
         %stddev     %change         %stddev     %change         %stddev
             \          |                \          |                \
    270459                      272267 ±  3%       -48%     139834 ±  3%  aim7/1BRD_48G-xfs-disk_cp-3000-performance/ivb44
    473257                      468546               5%     497512        aim7/1BRD_48G-xfs-disk_rd-9000-performance/ivb44
    360578             -18%     296589             -60%     144974        aim7/1BRD_48G-xfs-disk_rr-3000-performance/ivb44
    358701              -6%     335712             -40%     216057        GEO-MEAN aim7.jobs-per-min

99091700659f4df9  ca2edab2e1d8f30dda874b7f71  fe9c2c81ed073878768785a985
----------------  --------------------------  --------------------------
     48.48                       48.15              36%      65.85        aim7/1BRD_48G-xfs-disk_cp-3000-performance/ivb44
     89.50                       89.76                       88.75        aim7/1BRD_48G-xfs-disk_rd-9000-performance/ivb44
     35.78              23%      43.93              76%      63.09        aim7/1BRD_48G-xfs-disk_rr-3000-performance/ivb44
     53.75               7%      57.48              33%      71.71        GEO-MEAN turbostat.%Busy

99091700659f4df9  ca2edab2e1d8f30dda874b7f71  fe9c2c81ed073878768785a985
----------------  --------------------------  --------------------------
      1439                        1431              36%       1964        aim7/1BRD_48G-xfs-disk_cp-3000-performance/ivb44
      2671                        2674                        2650        aim7/1BRD_48G-xfs-disk_rd-9000-performance/ivb44
      1057              23%       1303              78%       1883        aim7/1BRD_48G-xfs-disk_rr-3000-performance/ivb44
      1595               7%       1708              34%       2139        GEO-MEAN turbostat.Avg_MHz

99091700659f4df9  ca2edab2e1d8f30dda874b7f71  fe9c2c81ed073878768785a985
----------------  --------------------------  --------------------------
       167                         167               6%        177        aim7/1BRD_48G-xfs-disk_cp-3000-performance/ivb44
       175                         175                         176        aim7/1BRD_48G-xfs-disk_rd-9000-performance/ivb44
       150               8%        162              19%        178        aim7/1BRD_48G-xfs-disk_rr-3000-performance/ivb44
       164                         168               8%        177        GEO-MEAN turbostat.PkgWatt

99091700659f4df9  ca2edab2e1d8f30dda874b7f71  fe9c2c81ed073878768785a985
----------------  --------------------------  --------------------------
     10.27                       10.43             -14%       8.79        aim7/1BRD_48G-xfs-disk_cp-3000-performance/ivb44
      6.85                        6.66                        6.88        aim7/1BRD_48G-xfs-disk_rd-9000-performance/ivb44
      9.96              14%      11.36              -7%       9.23        aim7/1BRD_48G-xfs-disk_rr-3000-performance/ivb44
      8.88               4%       9.24              -7%       8.23        GEO-MEAN turbostat.RAMWatt

Here are the detailed numbers for each test case. The perf-profile and
latency_stats numbers are sorted by absolute change in each sub-category
now. perf-profile numbers > 5 are all shown.

It may be more pleasant to view the long trace.call.funcs lines with
vim ":set nowrap" option.

aim7/1BRD_48G-xfs-disk_rr-3000-performance/ivb44

99091700659f4df9  ca2edab2e1d8f30dda874b7f71  fe9c2c81ed073878768785a985  
----------------  --------------------------  --------------------------  
         %stddev     %change         %stddev     %change         %stddev
             \          |                \          |                \  
    360578             -18%     294351             -60%     144974        aim7.jobs-per-min
     12835             458%      71658             480%      74445        aim7.time.involuntary_context_switches
       755              50%       1136             373%       3570        aim7.time.system_time
    155970                      152810              73%     269438        aim7.time.minor_page_faults
     50.15              22%      61.39             148%     124.39        aim7.time.elapsed_time
     50.15              22%      61.39             148%     124.39        aim7.time.elapsed_time.max
    438660                      428601              -7%     407807        aim7.time.voluntary_context_switches
      2452                        2480               5%       2584        aim7.time.maximum_resident_set_size
     34293 ±  4%        70%      58129 ± 19%       213%     107483        interrupts.CAL:Function_call_interrupts
     79.70 ±  6%        16%      92.63 ±  6%        89%     150.33        uptime.boot
      2890 ±  8%         6%       3077 ±  8%        15%       3329        uptime.idle
    150186 ±  9%        41%     212090             122%     333727        softirqs.RCU
    161166               9%     176318              16%     186527        softirqs.SCHED
    648051              33%     864346             222%    2089349        softirqs.TIMER
     50.15              22%      61.39             148%     124.39        time.elapsed_time
     50.15              22%      61.39             148%     124.39        time.elapsed_time.max
     12835             458%      71658             480%      74445        time.involuntary_context_switches
    155970                      152810              73%     269438        time.minor_page_faults
      1563              21%       1898              85%       2895        time.percent_of_cpu_this_job_got
       755              50%       1136             373%       3570        time.system_time
   4564660 ±  4%        68%    7651587              79%    8159302        numa-numastat.node0.numa_foreign
   3929898              81%    7129718              46%    5733813        numa-numastat.node0.numa_miss
         0                           2 ± 20%                     2        numa-numastat.node1.other_node
   4569811 ±  4%        68%    7654689              79%    8163206        numa-numastat.node1.numa_miss
   3935075              81%    7132850              46%    5737410        numa-numastat.node1.numa_foreign
  34767917               4%   36214694              11%   38627727        numa-numastat.node1.numa_hit
  34767917               4%   36214691              11%   38627725        numa-numastat.node1.local_node
     12377 ± 18%      3615%     459790            2848%     364868        vmstat.io.bo
       119              -8%        110 ±  4%       -16%        101        vmstat.memory.buff
  18826454             -16%   15748045             -37%   11882562        vmstat.memory.free
        16              25%         20             106%         33        vmstat.procs.r
     19407             469%     110509             520%     120350        vmstat.system.cs
     48215              10%      52977               3%      49819        vmstat.system.in
    142459             -11%     126667             -23%     109481        cpuidle.C1-IVT.usage
  29494441 ±  3%       -18%   24206809             -36%   18889149        cpuidle.C1-IVT.time
   5736732              28%    7315830             525%   35868316        cpuidle.C1E-IVT.time
     51148               9%      55743              98%     101021        cpuidle.C1E-IVT.usage
  18347890              27%   23243942              21%   22154105        cpuidle.C3-IVT.time
     96127               9%     104487             -29%      68552        cpuidle.C3-IVT.usage
 1.525e+09               6%  1.617e+09              41%  2.147e+09        cpuidle.C6-IVT.time
   1805218              11%    1998052              33%    2397285        cpuidle.C6-IVT.usage
       286 ± 11%        14%        328 ±  7%       389%       1402        cpuidle.POLL.usage
   1013526 ± 41%        98%    2003264 ± 20%       272%    3774675        cpuidle.POLL.time
     35.78              24%      44.22              76%      63.09        turbostat.%Busy
      1057              24%       1312              78%       1883        turbostat.Avg_MHz
     34.80              -3%      33.63             -22%      27.18        turbostat.CPU%c1
      0.34              -5%       0.33             -77%       0.08        turbostat.CPU%c3
     29.07             -25%      21.82             -67%       9.65        turbostat.CPU%c6
       118              11%        130              23%        145        turbostat.CorWatt
      9.39 ± 13%       -19%       7.61 ±  6%       -61%       3.67        turbostat.Pkg%pc2
      3.04 ± 33%       -49%       1.55 ± 14%       -76%       0.72        turbostat.Pkg%pc6
       150               9%        164              19%        178        turbostat.PkgWatt
      9.96              14%      11.34              -7%       9.23        turbostat.RAMWatt
     18232 ±  8%        -8%      16747 ± 10%        11%      20267        meminfo.AnonHugePages
     80723                       78330             -24%      61572        meminfo.CmaFree
   4690642 ± 10%       -15%    3981312             -15%    3983392        meminfo.DirectMap2M
   1060897             -21%     834807             -22%     828755        meminfo.Dirty
   2362330              26%    2983603              44%    3391287        meminfo.Inactive
   2353250              26%    2974520              44%    3382139        meminfo.Inactive(file)
  19388991             -18%   15966408             -38%   12038822        meminfo.MemFree
   1186231               4%    1236627              13%    1341728        meminfo.SReclaimable
    179570               3%     185696              14%     204382        meminfo.SUnreclaim
   1365802               4%    1422323              13%    1546111        meminfo.Slab
    318863              10%     352026              16%     368386        meminfo.Unevictable
      0.00                        0.00                        9.15        perf-profile.cycles-pp.xfs_file_iomap_begin_delay.isra.9.xfs_file_iomap_begin.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
      0.00                        0.00                        8.90        perf-profile.cycles-pp.xfs_inode_set_eofblocks_tag.xfs_file_iomap_begin_delay.isra.9.xfs_file_iomap_begin.iomap_apply.iomap_file_buffered_write
      0.00                        0.00                        8.61        perf-profile.cycles-pp._raw_spin_lock.xfs_inode_set_eofblocks_tag.xfs_file_iomap_begin_delay.isra.9.xfs_file_iomap_begin.iomap_apply
      0.00                        0.00                        8.50        perf-profile.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.xfs_inode_set_eofblocks_tag.xfs_file_iomap_begin_delay.isra.9.xfs_file_iomap_begin
      6.05             -11%       5.42 ±  4%       -15%       5.14        perf-profile.cycles-pp.hrtimer_interrupt.local_apic_timer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter
      6.54             -11%       5.80 ±  4%       -16%       5.51        perf-profile.cycles-pp.local_apic_timer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter.call_cpuidle
     16.78              -9%      15.34 ±  9%       -11%      14.90        perf-profile.cycles-pp.apic_timer_interrupt.cpuidle_enter.call_cpuidle.cpu_startup_entry.start_secondary
     16.51 ±  3%        -9%      14.99 ±  9%       -12%      14.49        perf-profile.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter.call_cpuidle.cpu_startup_entry
      0.23 ± 23%        20%       0.28 ± 12%      3683%       8.70        perf-profile.func.cycles-pp.native_queued_spin_lock_slowpath
 4.369e+11 ±  4%        20%  5.239e+11              97%  8.601e+11        perf-stat.branch-instructions
      0.38               5%       0.40             -27%       0.28        perf-stat.branch-miss-rate
 1.678e+09 ±  3%        26%  2.117e+09              44%  2.413e+09        perf-stat.branch-misses
     42.30              -7%      39.31              -5%      40.38        perf-stat.cache-miss-rate
 6.874e+09 ±  4%        19%   8.21e+09              51%  1.041e+10        perf-stat.cache-misses
 1.625e+10 ±  3%        29%  2.089e+10              59%  2.578e+10        perf-stat.cache-references
   1017846             588%    7005227            1401%   15273586        perf-stat.context-switches
 2.757e+12 ±  4%        48%  4.092e+12             318%  1.151e+13        perf-stat.cpu-cycles
    177918              15%     204776              35%     241051        perf-stat.cpu-migrations
      0.37 ± 14%        60%       0.60 ±  3%        45%       0.54        perf-stat.dTLB-load-miss-rate
 2.413e+09 ± 14%        97%  4.757e+09 ±  4%       149%  6.001e+09        perf-stat.dTLB-load-misses
 6.438e+11              23%  7.893e+11              71%  1.103e+12        perf-stat.dTLB-loads
      0.06 ± 38%       100%       0.11 ±  6%       207%       0.17        perf-stat.dTLB-store-miss-rate
 2.656e+08 ± 34%       123%   5.91e+08 ±  7%       203%  8.038e+08        perf-stat.dTLB-store-misses
     45.99 ±  5%         8%      49.56 ± 11%        14%      52.61        perf-stat.iTLB-load-miss-rate
  45151945                    45832755              72%   77697494        perf-stat.iTLB-load-misses
  53205262 ±  7%       -10%   47792612 ± 21%        32%   69997751        perf-stat.iTLB-loads
 2.457e+12 ±  4%        16%  2.851e+12              66%  4.084e+12        perf-stat.instructions
      0.89             -22%       0.70             -60%       0.35        perf-stat.ipc
    286640               8%     310690              99%     571225        perf-stat.minor-faults
     29.16               7%      31.25               8%      31.42        perf-stat.node-load-miss-rate
  4.86e+08 ±  3%       123%  1.084e+09             250%    1.7e+09        perf-stat.node-load-misses
  1.18e+09             102%  2.385e+09             214%  3.711e+09        perf-stat.node-loads
     21.51              30%      27.95              62%      34.86        perf-stat.node-store-miss-rate
 1.262e+09              58%  1.989e+09             177%  3.499e+09        perf-stat.node-store-misses
 4.606e+09              11%  5.126e+09              42%  6.539e+09        perf-stat.node-stores
    286617               8%     310730              99%     571253        perf-stat.page-faults
   1166432              23%    1429828              42%    1653754        numa-meminfo.node0.Inactive(file)
   1175123              22%    1434274              41%    1662351        numa-meminfo.node0.Inactive
    513534             -23%     394773             -24%     392567        numa-meminfo.node0.Dirty
   9717968             -17%    8082393             -37%    6159862        numa-meminfo.node0.MemFree
    159470              11%     176717              16%     184229        numa-meminfo.node0.Unevictable
  23148226               7%   24783802              15%   26706333        numa-meminfo.node0.MemUsed
    103531 ± 32%       -10%      93669 ± 40%        40%     144469        numa-meminfo.node0.SUnreclaim
   1187035              30%    1549075              46%    1727751        numa-meminfo.node1.Inactive
   1186646              30%    1544438              46%    1727201        numa-meminfo.node1.Inactive(file)
  21000905               3%   21647702              13%   23741428        numa-meminfo.node1.Active(file)
  21083707               3%   21748741              13%   23822391        numa-meminfo.node1.Active
    547021             -20%     438525             -21%     433706        numa-meminfo.node1.Dirty
   9663240             -19%    7870896             -39%    5869977        numa-meminfo.node1.MemFree
    561241              12%     625903              21%     679671        numa-meminfo.node1.SReclaimable
    637259 ±  4%        13%     717863 ±  5%        16%     739482        numa-meminfo.node1.Slab
  23329350               8%   25121687              16%   27122606        numa-meminfo.node1.MemUsed
    159394              10%     175315              16%     184159        numa-meminfo.node1.Unevictable
    521615              33%     695562             267%    1916159        latency_stats.avg.call_rwsem_down_write_failed.do_unlinkat.SyS_unlink.entry_SYSCALL_64_fastpath
    500644              33%     667614             261%    1805608        latency_stats.avg.call_rwsem_down_write_failed.path_openat.do_filp_open.do_sys_open.SyS_creat.entry_SYSCALL_64_fastpath
      8932 ± 46%       -70%       2717 ±  4%       -95%        464        latency_stats.avg.wait_on_page_bit.truncate_inode_pages_range.truncate_inode_pages_final.evict.iput.dentry_unlink_inode.__dentry_kill.dput.__fput.____fput.task_work_run.exit_to_usermode_loop
         0                           0                       73327        latency_stats.hits.wait_on_page_bit.__migration_entry_wait.migration_entry_wait.do_swap_page.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
        43 ± 37%      7923%       3503 ±  4%     31792%      13926        latency_stats.hits.down.xfs_buf_lock._xfs_buf_find.xfs_buf_get_map.xfs_buf_read_map.xfs_trans_read_buf_map.xfs_read_agf.xfs_alloc_read_agf.xfs_alloc_fix_freelist.xfs_free_extent_fix_freelist.xfs_free_extent.xfs_trans_free_extent
   1422573              30%    1852368 ±  5%       228%    4672496        latency_stats.max.call_rwsem_down_write_failed.path_openat.do_filp_open.do_sys_open.SyS_creat.entry_SYSCALL_64_fastpath
   1423130              30%    1851873 ±  5%       228%    4661765        latency_stats.max.call_rwsem_down_write_failed.do_unlinkat.SyS_unlink.entry_SYSCALL_64_fastpath
       627 ± 66%      3788%      24404 ± 17%      6254%      39883        latency_stats.max.down.xfs_buf_lock._xfs_buf_find.xfs_buf_get_map.xfs_buf_read_map.xfs_trans_read_buf_map.xfs_read_agf.xfs_alloc_read_agf.xfs_alloc_fix_freelist.xfs_free_extent_fix_freelist.xfs_free_extent.xfs_trans_free_extent
      3922 ± 18%        56%       6134 ± 29%       634%      28786        latency_stats.max.down.xfs_buf_lock._xfs_buf_find.xfs_buf_get_map.xfs_buf_read_map.xfs_trans_read_buf_map.xfs_read_agi.xfs_ialloc_read_agi.xfs_dialloc.xfs_ialloc.xfs_dir_ialloc.xfs_create
         0                           0                       16665        latency_stats.max.wait_on_page_bit.__migration_entry_wait.migration_entry_wait.do_swap_page.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
  5.15e+10              25%  6.454e+10             220%  1.649e+11        latency_stats.sum.call_rwsem_down_write_failed.do_unlinkat.SyS_unlink.entry_SYSCALL_64_fastpath
         0                           0                   1.385e+08        latency_stats.sum.wait_on_page_bit.__migration_entry_wait.migration_entry_wait.do_swap_page.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
  11666476              45%   16905624             755%   99756088        latency_stats.sum.down.xfs_buf_lock._xfs_buf_find.xfs_buf_get_map.xfs_buf_read_map.xfs_trans_read_buf_map.xfs_read_agi.xfs_iunlink_remove.xfs_ifree.xfs_inactive_ifree.xfs_inactive.xfs_fs_destroy_inode
      2216 ± 69%     80030%    1775681 ±  4%     3e+06%   67521154        latency_stats.sum.down.xfs_buf_lock._xfs_buf_find.xfs_buf_get_map.xfs_buf_read_map.xfs_trans_read_buf_map.xfs_read_agf.xfs_alloc_read_agf.xfs_alloc_fix_freelist.xfs_free_extent_fix_freelist.xfs_free_extent.xfs_trans_free_extent
   1601815              28%    2053992             288%    6213577        latency_stats.sum.down.xfs_buf_lock._xfs_buf_find.xfs_buf_get_map.xfs_buf_read_map.xfs_trans_read_buf_map.xfs_read_agi.xfs_ialloc_read_agi.xfs_dialloc.xfs_ialloc.xfs_dir_ialloc.xfs_create
   1774397              20%    2120576             244%    6099374        latency_stats.sum.down.xfs_buf_lock._xfs_buf_find.xfs_buf_get_map.xfs_buf_read_map.xfs_trans_read_buf_map.xfs_read_agi.xfs_iunlink.xfs_droplink.xfs_remove.xfs_vn_unlink.vfs_unlink
       628 ±141%       125%       1416 ±  5%     4e+05%    2677036        latency_stats.sum.xfs_iget.xfs_ialloc.xfs_dir_ialloc.xfs_create.xfs_generic_create.xfs_vn_mknod.xfs_vn_create.path_openat.do_filp_open.do_sys_open.SyS_creat.entry_SYSCALL_64_fastpath
      6087 ± 92%      1277%      83839 ±  3%     11105%     682063        latency_stats.sum.call_rwsem_down_read_failed.xfs_log_commit_cil.__xfs_trans_commit.xfs_trans_commit.xfs_vn_update_time.file_update_time.xfs_file_aio_write_checks.xfs_file_buffered_aio_write.xfs_file_write_iter.__vfs_write.vfs_write.SyS_write
         0                           0                      116108        latency_stats.sum.xlog_grant_head_wait.xlog_grant_head_check.xfs_log_reserve.xfs_trans_reserve.xfs_trans_alloc.xfs_vn_update_time.file_update_time.xfs_file_aio_write_checks.xfs_file_buffered_aio_write.xfs_file_write_iter.__vfs_write.vfs_write
      1212 ± 59%      1842%      23546 ±  7%      4861%      60149        latency_stats.sum.call_rwsem_down_read_failed.xfs_log_commit_cil.__xfs_trans_commit.xfs_trans_commit.xfs_vn_update_time.touch_atime.generic_file_read_iter.xfs_file_buffered_aio_read.xfs_file_read_iter.__vfs_read.vfs_read.SyS_read
      1624 ± 22%      1356%      23637 ±  3%      1596%      27545        latency_stats.sum.call_rwsem_down_read_failed.xfs_log_commit_cil.__xfs_trans_commit.__xfs_trans_roll.xfs_trans_roll.xfs_itruncate_extents.xfs_free_eofblocks.xfs_release.xfs_file_release.__fput.____fput.task_work_run
      2068 ± 27%       834%      19319 ± 23%      1125%      25334        latency_stats.sum.call_rwsem_down_read_failed.xfs_log_commit_cil.__xfs_trans_commit.__xfs_trans_roll.xfs_trans_roll.xfs_itruncate_extents.xfs_inactive_truncate.xfs_inactive.xfs_fs_destroy_inode.destroy_inode.evict.iput
         0                           0                       22155        latency_stats.sum.xlog_grant_head_wait.xlog_grant_head_check.xfs_log_reserve.xfs_trans_reserve.xfs_trans_alloc.xfs_inactive_truncate.xfs_inactive.xfs_fs_destroy_inode.destroy_inode.evict.iput.dentry_unlink_inode
        39 ± 71%     41280%      16414 ± 14%     51951%      20647        latency_stats.sum.call_rwsem_down_read_failed.xfs_log_commit_cil.__xfs_trans_commit.__xfs_trans_roll.xfs_trans_roll.xfs_defer_trans_roll.xfs_defer_finish.xfs_itruncate_extents.xfs_inactive_truncate.xfs_inactive.xfs_fs_destroy_inode.destroy_inode
         0                           0                       15600        latency_stats.sum.xlog_grant_head_wait.xlog_grant_head_check.xfs_log_reserve.xfs_trans_reserve.xfs_trans_alloc.xfs_inactive_ifree.xfs_inactive.xfs_fs_destroy_inode.destroy_inode.evict.iput.dentry_unlink_inode
        10 ±141%      6795%        689 ± 70%     1e+05%      10637        latency_stats.sum.call_rwsem_down_read_failed.xfs_log_commit_cil.__xfs_trans_commit.xfs_trans_commit.xfs_inactive_ifree.xfs_inactive.xfs_fs_destroy_inode.destroy_inode.evict.iput.dentry_unlink_inode.__dentry_kill
        99 ±112%        86%        185 ± 80%      9978%      10011        latency_stats.sum.down.xfs_buf_lock._xfs_buf_find.xfs_buf_get_map.xfs_buf_read_map.xfs_trans_read_buf_map.xfs_imap_to_bp.xfs_iunlink_remove.xfs_ifree.xfs_inactive_ifree.xfs_inactive.xfs_fs_destroy_inode
     18232 ±134%       -16%      15260 ± 54%       -40%      10975        latency_stats.sum.xfs_lock_two_inodes.xfs_remove.xfs_vn_unlink.vfs_unlink.do_unlinkat.SyS_unlink.entry_SYSCALL_64_fastpath
       647 ±  3%       -97%         21 ± 19%        34%        868        proc-vmstat.kswapd_high_wmark_hit_quickly
      1091             -97%         36 ±  9%        29%       1411        proc-vmstat.kswapd_low_wmark_hit_quickly
    265066             -21%     208142             -22%     206344        proc-vmstat.nr_dirty
     20118                       19574             -23%      15432        proc-vmstat.nr_free_cma
   4844108             -18%    3988031             -38%    3008251        proc-vmstat.nr_free_pages
    588262              26%     743537              44%     845765        proc-vmstat.nr_inactive_file
        50 ± 25%       192%        148 ± 15%       103%        103        proc-vmstat.nr_pages_scanned
    296623               4%     309201              13%     335474        proc-vmstat.nr_slab_reclaimable
     44880               3%      46405              14%      51078        proc-vmstat.nr_slab_unreclaimable
     79716              10%      88008              16%      92097        proc-vmstat.nr_unevictable
       167 ±  9%     9e+06%   14513434           2e+06%    3569348        proc-vmstat.nr_vmscan_immediate_reclaim
    162380 ± 18%      4392%    7294622            7024%   11567602        proc-vmstat.nr_written
    588257              26%     743537              44%     845784        proc-vmstat.nr_zone_inactive_file
     79716              10%      88008              16%      92097        proc-vmstat.nr_zone_unevictable
    265092             -21%     208154             -22%     206388        proc-vmstat.nr_zone_write_pending
   8507451 ±  3%        74%   14784261              64%   13918067        proc-vmstat.numa_foreign
        10 ±  4%                    10 ±  4%     6e+05%      57855        proc-vmstat.numa_hint_faults
   8507451 ±  3%        74%   14784187              64%   13918067        proc-vmstat.numa_miss
        72                          72           3e+05%     213175        proc-vmstat.numa_pte_updates
      1740             -97%         59 ± 12%        33%       2306        proc-vmstat.pageoutrun
   5322372            1068%   62167111            1024%   59824114        proc-vmstat.pgactivate
   2816355              27%    3575784              14%    3203214        proc-vmstat.pgalloc_dma32
  74392338              11%   82333943              14%   84954110        proc-vmstat.pgalloc_normal
  60958397             -18%   49976330             -26%   45055885        proc-vmstat.pgdeactivate
    302790               9%     329088              94%     586116        proc-vmstat.pgfault
  61061205              14%   69758545              18%   72000453        proc-vmstat.pgfree
    655652 ± 18%      4352%   29190304            6967%   46338056        proc-vmstat.pgpgout
  60965725             -18%   49983704             -26%   45063375        proc-vmstat.pgrefill
         2 ± 17%     4e+07%     985929 ±  8%     7e+07%    1952629        proc-vmstat.pgrotated
     82046 ± 36%     50634%   41625211            5397%    4510385        proc-vmstat.pgscan_direct
  60128369             -37%   38068394              10%   66306637        proc-vmstat.pgscan_kswapd
      2030 ± 46%     1e+06%   27038054 ±  3%     78642%    1598733        proc-vmstat.pgsteal_direct
         0                     2414551 ±  3%               3694833        proc-vmstat.workingset_activate
         0                     2414551 ±  3%               3694833        proc-vmstat.workingset_refault
        26 ± 39%     1e+07%    2657286           3e+06%     658792        numa-vmstat.node0.nr_vmscan_immediate_reclaim
     40449 ± 22%      3135%    1308601 ±  4%      4723%    1950670        numa-vmstat.node0.nr_written
    291648              22%     357059              42%     413612        numa-vmstat.node0.nr_zone_inactive_file
    291655              22%     357053              42%     413596        numa-vmstat.node0.nr_inactive_file
   1542314 ±  5%        77%    2731911              98%    3056411        numa-vmstat.node0.numa_foreign
   1366073 ±  4%       103%    2766780 ±  3%        68%    2293117        numa-vmstat.node0.numa_miss
    128634             -23%      99104             -24%      98062        numa-vmstat.node0.nr_dirty
    128663             -23%      99130             -24%      98051        numa-vmstat.node0.nr_zone_write_pending
   2424918             -16%    2033425             -37%    1537826        numa-vmstat.node0.nr_free_pages
  14037168              10%   15473174              20%   16883787        numa-vmstat.node0.numa_local
  14037172              10%   15473174              20%   16883790        numa-vmstat.node0.numa_hit
     39867              10%      44022              16%      46058        numa-vmstat.node0.nr_zone_unevictable
     39867              10%      44022              16%      46058        numa-vmstat.node0.nr_unevictable
     25871 ± 32%        -9%      23414 ± 40%        40%      36094        numa-vmstat.node0.nr_slab_unreclaimable
  14851187               6%   15749527              11%   16497187        numa-vmstat.node0.nr_dirtied
         0                     1225299 ±  4%               2008478        numa-vmstat.node1.workingset_refault
         0                     1225299 ±  4%               2008478        numa-vmstat.node1.workingset_activate
        23 ± 35%     1e+07%    2974198 ±  3%     3e+06%     683002        numa-vmstat.node1.nr_vmscan_immediate_reclaim
     40769 ± 26%      3264%    1371611 ±  3%      5569%    2311374        numa-vmstat.node1.nr_written
        25 ±  8%       216%         81 ±  3%       356%        117        numa-vmstat.node1.nr_pages_scanned
    296681              30%     385708              45%     431591        numa-vmstat.node1.nr_zone_inactive_file
    296681              30%     385709              45%     431591        numa-vmstat.node1.nr_inactive_file
   5252547                     5401234              13%    5936151        numa-vmstat.node1.nr_zone_active_file
   5252547                     5401238              13%    5936151        numa-vmstat.node1.nr_active_file
    136060             -19%     110021             -21%     107114        numa-vmstat.node1.nr_zone_write_pending
    136060             -19%     110019             -21%     107107        numa-vmstat.node1.nr_dirty
   1520682 ±  3%        76%    2681012              98%    3008493        numa-vmstat.node1.numa_miss
   2413468             -18%    1980184             -39%    1466738        numa-vmstat.node1.nr_free_pages
   1344474 ±  3%       102%    2715690 ±  4%        67%    2245159        numa-vmstat.node1.numa_foreign
     20160                       19698             -22%      15673        numa-vmstat.node1.nr_free_cma
  14350439              12%   16005551              27%   18257157        numa-vmstat.node1.numa_local
  14350440              12%   16005552              27%   18257158        numa-vmstat.node1.numa_hit
  15381788               9%   16829619              21%   18645441        numa-vmstat.node1.nr_dirtied
    140354              11%     156202              21%     169950        numa-vmstat.node1.nr_slab_reclaimable
     39848              10%      43676              16%      46041        numa-vmstat.node1.nr_zone_unevictable
     39848              10%      43676              16%      46041        numa-vmstat.node1.nr_unevictable
       377 ±  9%                   370 ±  5%        24%        468        slabinfo.bdev_cache.active_objs
       377 ±  9%                   370 ±  5%        24%        468        slabinfo.bdev_cache.num_objs
       389 ± 13%       604%       2737 ± 23%      3371%      13501        slabinfo.bio-1.active_objs
       389 ± 13%       612%       2770 ± 24%      3441%      13774        slabinfo.bio-1.num_objs
         7 ± 17%      1039%         83 ± 24%      3623%        273        slabinfo.bio-1.active_slabs
         7 ± 17%      1039%         83 ± 24%      3623%        273        slabinfo.bio-1.num_slabs
       978 ±  4%        10%       1075              17%       1144        slabinfo.blkdev_requests.active_objs
       978 ±  4%        10%       1075              17%       1144        slabinfo.blkdev_requests.num_objs
  10942119               3%   11286505              13%   12389701        slabinfo.buffer_head.num_objs
    280566               3%     289397              13%     317684        slabinfo.buffer_head.active_slabs
    280566               3%     289397              13%     317684        slabinfo.buffer_head.num_slabs
  10941627                    10693692              11%   12140372        slabinfo.buffer_head.active_objs
      7436 ±  3%                  7558              20%       8922        slabinfo.cred_jar.active_objs
      7436 ±  3%                  7558              20%       8922        slabinfo.cred_jar.num_objs
      4734              85%       8767 ±  8%        60%       7554        slabinfo.kmalloc-128.num_objs
      4734              78%       8418 ±  8%        45%       6848        slabinfo.kmalloc-128.active_objs
     17074             -11%      15121             -10%      15379        slabinfo.kmalloc-256.num_objs
      3105               4%       3216              14%       3527        slabinfo.kmalloc-4096.num_objs
      3061               4%       3170              12%       3419        slabinfo.kmalloc-4096.active_objs
     13131 ±  3%        17%      15379              12%      14714        slabinfo.kmalloc-512.num_objs
      1623 ±  3%                  1664 ±  3%        16%       1889        slabinfo.mnt_cache.active_objs
      1623 ±  3%                  1664 ±  3%        16%       1889        slabinfo.mnt_cache.num_objs
      2670               6%       2821              19%       3178        slabinfo.nsproxy.active_objs
      2670               6%       2821              19%       3178        slabinfo.nsproxy.num_objs
      2532               5%       2656              17%       2959        slabinfo.posix_timers_cache.active_objs
      2532               5%       2656              17%       2959        slabinfo.posix_timers_cache.num_objs
     20689              87%      38595 ± 13%        47%      30452        slabinfo.radix_tree_node.active_objs
       399              83%        730 ± 13%        47%        587        slabinfo.radix_tree_node.active_slabs
       399              83%        730 ± 13%        47%        587        slabinfo.radix_tree_node.num_slabs
     22379              83%      40931 ± 13%        47%      32872        slabinfo.radix_tree_node.num_objs
      4688                        4706              22%       5712        slabinfo.sigqueue.active_objs
      4688                        4706              22%       5712        slabinfo.sigqueue.num_objs
       979 ±  4%         7%       1046 ±  3%       -15%        833        slabinfo.task_group.active_objs
       979 ±  4%         7%       1046 ±  3%       -15%        833        slabinfo.task_group.num_objs
      1344               5%       1410              17%       1570        slabinfo.xfs_btree_cur.active_objs
      1344               5%       1410              17%       1570        slabinfo.xfs_btree_cur.num_objs
      2500               5%       2632              18%       2946        slabinfo.xfs_da_state.active_objs
      2500               5%       2632              18%       2946        slabinfo.xfs_da_state.num_objs
      1299             279%       4917 ± 17%       134%       3035        slabinfo.xfs_efd_item.num_objs
      1299             278%       4911 ± 17%       126%       2940        slabinfo.xfs_efd_item.active_objs
      1904 ±  3%         4%       1982              42%       2703        slabinfo.xfs_inode.num_objs
      1904 ±  3%         4%       1982              39%       2644        slabinfo.xfs_inode.active_objs
      1659             113%       3538 ± 27%      1360%      24227        slabinfo.xfs_log_ticket.active_objs
      1659             116%       3588 ± 27%      1369%      24383        slabinfo.xfs_log_ticket.num_objs
        37             169%         99 ± 29%      1405%        557        slabinfo.xfs_log_ticket.active_slabs
        37             169%         99 ± 29%      1405%        557        slabinfo.xfs_log_ticket.num_slabs
      2615              84%       4821 ± 28%      1549%      43132        slabinfo.xfs_trans.active_objs
      2615              86%       4860 ± 28%      1551%      43171        slabinfo.xfs_trans.num_objs
        37             162%         97 ± 30%      1614%        634        slabinfo.xfs_trans.active_slabs
        37             162%         97 ± 30%      1614%        634        slabinfo.xfs_trans.num_slabs
      3255 ± 12%      9210%     303094           38966%    1271810        sched_debug.cfs_rq:/.min_vruntime.avg
      8273 ± 10%       382%      39836 ± 17%       309%      33806        sched_debug.cfs_rq:/.load.avg
       716 ± 34%     28783%     206899           1e+05%    1034000        sched_debug.cfs_rq:/.min_vruntime.min
      1830 ±  5%      4365%      81731           10579%     195502        sched_debug.cfs_rq:/.min_vruntime.stddev
      1845 ±  4%      4330%      81754           10503%     195683        sched_debug.cfs_rq:/.spread0.stddev
     73578 ± 34%      1043%     841209 ± 34%       452%     405848        sched_debug.cfs_rq:/.load.max
     12.67 ± 35%      3999%     519.25            1979%     263.33        sched_debug.cfs_rq:/.runnable_load_avg.max
      2.34 ± 33%      4268%     102.01            1854%      45.63        sched_debug.cfs_rq:/.runnable_load_avg.stddev
     10284 ± 12%      4107%     432665 ±  7%     15350%    1588973        sched_debug.cfs_rq:/.min_vruntime.max
      1.05 ± 20%      2335%      25.54            1631%      18.15        sched_debug.cfs_rq:/.runnable_load_avg.avg
     44.06 ± 28%       254%     155.90 ± 16%       310%     180.49        sched_debug.cfs_rq:/.util_avg.stddev
     15448 ± 19%       831%     143829 ± 22%       422%      80585        sched_debug.cfs_rq:/.load.stddev
       597 ± 13%       -39%        367 ± 17%       -49%        303        sched_debug.cfs_rq:/.util_avg.min
      1464 ± 23%       -55%        664 ± 30%       -63%        546        sched_debug.cfs_rq:/.load_avg.min
      1830 ±  3%       -50%        911 ±  5%       -65%        642        sched_debug.cfs_rq:/.load_avg.avg
      0.30 ± 13%        22%       0.36 ± 11%        86%       0.56        sched_debug.cfs_rq:/.nr_running.avg
      2302 ± 11%       -31%       1589             -50%       1157        sched_debug.cfs_rq:/.load_avg.max
       819 ±  3%        36%       1116              15%        940        sched_debug.cfs_rq:/.util_avg.max
       728             -14%        630              -9%        664        sched_debug.cfs_rq:/.util_avg.avg
     73578 ± 34%      1043%     841209 ± 34%       452%     405848        sched_debug.cpu.load.max
      1.81 ± 11%        77%       3.22             395%       8.98        sched_debug.cpu.clock.stddev
      1.81 ± 11%        77%       3.22             395%       8.98        sched_debug.cpu.clock_task.stddev
      8278 ± 10%       379%      39671 ± 18%       305%      33517        sched_debug.cpu.load.avg
      3600             385%      17452            1023%      40419        sched_debug.cpu.nr_load_updates.min
      5446             305%      22069             754%      46492        sched_debug.cpu.nr_load_updates.avg
      8627 ±  5%       217%      27314             517%      53222        sched_debug.cpu.nr_load_updates.max
      6221 ±  3%      2137%     139191            3486%     223092        sched_debug.cpu.nr_switches.max
     15.67 ± 40%      3187%     515.00            1579%     263.00        sched_debug.cpu.cpu_load[0].max
      2.55 ± 33%      3886%     101.45            1697%      45.73        sched_debug.cpu.cpu_load[0].stddev
     15452 ± 19%       831%     143937 ± 22%       421%      80431        sched_debug.cpu.load.stddev
      1144             236%       3839             329%       4911        sched_debug.cpu.nr_load_updates.stddev
     23.67 ± 41%       709%     191.50 ±  6%       637%     174.33        sched_debug.cpu.nr_uninterruptible.max
       978            7241%      71831 ±  3%     13746%     135493        sched_debug.cpu.nr_switches.avg
      0.96 ± 19%      2503%      24.95            1720%      17.44        sched_debug.cpu.cpu_load[0].avg
       957 ±  4%      3406%      33568            3626%      35679        sched_debug.cpu.nr_switches.stddev
     29644 ± 16%       107%      61350 ±  8%       190%      86111        sched_debug.cpu.clock.max
     29644 ± 16%       107%      61350 ±  8%       190%      86111        sched_debug.cpu.clock_task.max
     29640 ± 16%       107%      61344 ±  8%       190%      86096        sched_debug.cpu.clock.avg
     29640 ± 16%       107%      61344 ±  8%       190%      86096        sched_debug.cpu.clock_task.avg
     29635 ± 16%       107%      61338 ±  8%       190%      86079        sched_debug.cpu.clock.min
     29635 ± 16%       107%      61338 ±  8%       190%      86079        sched_debug.cpu.clock_task.min
       335 ±  4%      7948%      27014           22596%      76183        sched_debug.cpu.nr_switches.min
      1.62 ± 32%      1784%      30.61 ±  3%      1100%      19.51        sched_debug.cpu.cpu_load[4].avg
      5.46 ± 15%      2325%     132.40            1031%      61.73        sched_debug.cpu.nr_uninterruptible.stddev
       424 ± 11%       106%        875 ± 13%       263%       1541        sched_debug.cpu.curr->pid.avg
      1400             166%       3721             264%       5100        sched_debug.cpu.curr->pid.max
       610 ±  3%       108%       1269             126%       1380        sched_debug.cpu.curr->pid.stddev
      0.43 ± 15%         4%       0.45 ± 16%        48%       0.64        sched_debug.cpu.nr_running.avg
    253789 ± 13%        -5%     241499 ±  3%       -22%     198383        sched_debug.cpu.avg_idle.stddev
     29638 ± 16%       107%      61339 ±  8%       190%      86079        sched_debug.cpu_clk
     28529 ± 17%       111%      60238 ±  8%       198%      84957        sched_debug.ktime
      0.17             -74%       0.04 ±  8%       -83%       0.03        sched_debug.rt_rq:/.rt_time.avg
      0.85 ±  3%       -74%       0.22 ±  8%       -83%       0.14        sched_debug.rt_rq:/.rt_time.stddev
      5.14 ± 10%       -75%       1.28 ±  6%       -83%       0.88        sched_debug.rt_rq:/.rt_time.max
     29638 ± 16%       107%      61339 ±  8%       190%      86079        sched_debug.sched_clk

aim7/1BRD_48G-xfs-disk_rd-9000-performance/ivb44

99091700659f4df9  ca2edab2e1d8f30dda874b7f71  fe9c2c81ed073878768785a985  
----------------  --------------------------  --------------------------  
    473257                      468546               5%     497512        aim7.jobs-per-min
    613996              11%     681283              -7%     571701        aim7.time.involuntary_context_switches
      4914                        4977              -6%       4634        aim7.time.system_time
    114.83                      115.98              -5%     109.23        aim7.time.elapsed_time
    114.83                      115.98              -5%     109.23        aim7.time.elapsed_time.max
     60711 ±  8%        20%      73007              -9%      55449        aim7.time.voluntary_context_switches
      2509              -6%       2360              -4%       2416        aim7.time.maximum_resident_set_size
    362268              19%     430263              -8%     332046        softirqs.RCU
       352 ±  7%       -32%        238             -35%        230        vmstat.procs.r
         5 ± 16%        80%          9             -40%          3        vmstat.procs.b
      9584               7%      10255             -10%       8585        vmstat.system.cs
     20442 ±  5%        38%      28201             -40%      12270        cpuidle.C1-IVT.usage
      3.95              -3%       3.81               9%       4.29        turbostat.CPU%c1
      0.81 ± 14%        44%       1.17              28%       1.04        turbostat.Pkg%pc6
     19711 ±  5%        -7%      18413             -17%      16384        meminfo.AnonHugePages
   3974485                     3977216              27%    5046310        meminfo.DirectMap2M
    139742 ±  4%                137012             -17%     116493        meminfo.DirectMap4k
    244933 ±  4%        -7%     228626              15%     280670        meminfo.PageTables
     12.47 ± 39%        84%      22.89              64%      20.46        perf-profile.func.cycles-pp.poll_idle
     57.44 ±  6%       -10%      51.55             -13%      50.13        perf-profile.func.cycles-pp.intel_idle
      0.20               3%       0.20              -5%       0.19        perf-stat.branch-miss-rate
 5.356e+08               4%  5.552e+08              -6%  5.046e+08        perf-stat.branch-misses
   1113549               7%    1187535             -15%     951607        perf-stat.context-switches
  1.48e+13                   1.491e+13              -6%  1.397e+13        perf-stat.cpu-cycles
    101697 ±  3%         9%     111167              -3%      98319        perf-stat.cpu-migrations
      0.69 ± 20%       -17%       0.57             139%       1.65        perf-stat.dTLB-load-miss-rate
 3.264e+09 ± 19%       -17%  2.712e+09             148%  8.084e+09        perf-stat.dTLB-load-misses
 4.695e+11                   4.718e+11                   4.818e+11        perf-stat.dTLB-loads
 3.276e+11 ±  3%             3.303e+11               8%  3.528e+11        perf-stat.dTLB-stores
     56.47 ± 19%        41%      79.48             -58%      23.96        perf-stat.iTLB-load-miss-rate
  48864487 ±  4%         7%   52183944             -12%   43166037        perf-stat.iTLB-load-misses
  40455495 ± 41%       -67%   13468883             239%   1.37e+08        perf-stat.iTLB-loads
     29278 ±  4%        -6%      27480              12%      32844        perf-stat.instructions-per-iTLB-miss
      0.10                        0.10               5%       0.10        perf-stat.ipc
     47.16                       46.36                       46.51        perf-stat.node-store-miss-rate
      6568 ± 44%       -59%       2721             -71%       1916        numa-meminfo.node0.Shmem
    194395               7%     207086              15%     224164        numa-meminfo.node0.Active
     10218 ± 24%       -37%       6471             -36%       6494        numa-meminfo.node0.Mapped
      7496 ± 34%       -97%        204              37%      10278        numa-meminfo.node0.AnonHugePages
    178888               6%     188799              16%     208213        numa-meminfo.node0.AnonPages
    179468               6%     191062              17%     209704        numa-meminfo.node0.Active(anon)
    256890             -15%     219489             -15%     219503        numa-meminfo.node1.Active
     12213 ± 24%        49%      18208             -50%       6105        numa-meminfo.node1.AnonHugePages
     45080 ± 23%       -33%      30138              87%      84468        numa-meminfo.node1.PageTables
    241623             -15%     204604             -16%     203913        numa-meminfo.node1.Active(anon)
    240637             -15%     204491             -15%     203847        numa-meminfo.node1.AnonPages
  23782392 ±139%       673%  1.838e+08            -100%          0        latency_stats.sum.wait_on_page_bit.__migration_entry_wait.migration_entry_wait.do_swap_page.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
     61157 ±  4%        -6%      57187              14%      69751        proc-vmstat.nr_page_table_pages
      1641 ± 44%       -59%        679             -71%        478        numa-vmstat.node0.nr_shmem
      2655 ± 23%       -35%       1715             -35%       1726        numa-vmstat.node0.nr_mapped
     44867               5%      47231              16%      52261        numa-vmstat.node0.nr_anon_pages
     45014               6%      47793              17%      52636        numa-vmstat.node0.nr_zone_active_anon
     45014               6%      47793              17%      52636        numa-vmstat.node0.nr_active_anon
     11300 ± 23%       -33%       7542              88%      21209        numa-vmstat.node1.nr_page_table_pages
     60581             -16%      51156             -15%      51193        numa-vmstat.node1.nr_zone_active_anon
     60581             -16%      51156             -15%      51193        numa-vmstat.node1.nr_active_anon
     60328             -15%      51127             -15%      51174        numa-vmstat.node1.nr_anon_pages
     13671                       13608              11%      15190        slabinfo.cred_jar.active_objs
     13707                       13608              11%      15231        slabinfo.cred_jar.num_objs
     24109                       24386             -11%      21574        slabinfo.kmalloc-16.active_objs
     24109                       24386             -11%      21574        slabinfo.kmalloc-16.num_objs
     13709 ±  6%                 13391             -15%      11600        slabinfo.kmalloc-512.active_objs
     13808 ±  6%                 13454             -16%      11657        slabinfo.kmalloc-512.num_objs
   1456658               4%    1511260              15%    1675984        sched_debug.cfs_rq:/.min_vruntime.min
    441613 ±  3%       -28%     316751             -76%     105734        sched_debug.cfs_rq:/.min_vruntime.stddev
    443999 ±  3%       -28%     318033             -76%     106909        sched_debug.cfs_rq:/.spread0.stddev
   2657974                     2625551             -19%    2158111        sched_debug.cfs_rq:/.min_vruntime.max
      0.22 ± 23%        96%       0.43             109%       0.46        sched_debug.cfs_rq:/.nr_spread_over.stddev
      1.50             100%       3.00             133%       3.50        sched_debug.cfs_rq:/.nr_spread_over.max
    111.95 ± 26%        15%     128.92             128%     254.81        sched_debug.cfs_rq:/.exec_clock.stddev
       802               3%        829             -16%        671        sched_debug.cfs_rq:/.load_avg.min
       874                         879             -11%        780        sched_debug.cfs_rq:/.load_avg.avg
      1256 ± 17%       -20%       1011             -24%        957        sched_debug.cfs_rq:/.load_avg.max
      1.33 ± 35%      -100%       0.00             200%       4.00        sched_debug.cpu.cpu_load[4].min
      4.56 ±  6%       -11%       4.07             -27%       3.33        sched_debug.cpu.cpu_load[4].stddev
      4.76 ±  3%       -13%       4.14             -30%       3.35        sched_debug.cpu.cpu_load[3].stddev
     25.17 ± 12%       -26%      18.50             -21%      20.00        sched_debug.cpu.cpu_load[3].max
     25.67 ±  9%       -32%      17.50             -24%      19.50        sched_debug.cpu.cpu_load[0].max
      4.67 ±  3%       -17%       3.90             -22%       3.62        sched_debug.cpu.cpu_load[0].stddev
      4.88             -15%       4.14             -31%       3.39        sched_debug.cpu.cpu_load[2].stddev
     26.17 ± 10%       -29%      18.50             -25%      19.50        sched_debug.cpu.cpu_load[2].max
      7265               4%       7556             -12%       6419        sched_debug.cpu.nr_switches.avg
      9.41 ± 10%                  9.67              21%      11.38        sched_debug.cpu.cpu_load[1].avg
      9.03 ± 12%         3%       9.32              23%      11.09        sched_debug.cpu.cpu_load[0].avg
      4140 ±  4%       -11%       3698             -11%       3703        sched_debug.cpu.nr_switches.stddev
      9.41 ± 10%         3%       9.71              22%      11.49        sched_debug.cpu.cpu_load[3].avg
      4690                        4821              -9%       4257        sched_debug.cpu.nr_switches.min
      9.39 ±  9%         3%       9.69              23%      11.52        sched_debug.cpu.cpu_load[4].avg
      9.43 ± 10%                  9.71              21%      11.44        sched_debug.cpu.cpu_load[2].avg
     57.92 ± 18%        -4%      55.55             -23%      44.50        sched_debug.cpu.nr_uninterruptible.stddev
      3002 ±  3%        10%       3288              31%       3919        sched_debug.cpu.curr->pid.avg
      6666                        6652             -10%       6025        sched_debug.cpu.curr->pid.max
      1379                        1361             -19%       1118        sched_debug.cpu.ttwu_local.avg
      1849 ±  3%       -12%       1628             -18%       1517        sched_debug.cpu.ttwu_local.stddev
      1679 ±  8%         4%       1738             -15%       1423        sched_debug.cpu.curr->pid.stddev
      1.58 ± 33%       -11%       1.41              65%       2.60        sched_debug.cpu.nr_running.avg
      1767               6%       1880             -16%       1489        sched_debug.cpu.ttwu_count.avg
       506 ±  6%       -15%        430             -17%        419        sched_debug.cpu.ttwu_count.min
      7139               8%       7745             -11%       6355        sched_debug.cpu.sched_count.avg
      4355               6%       4625             -11%       3884        sched_debug.cpu.sched_count.min
      4.91 ±  3%       -16%       4.13             -28%       3.52        sched_debug.cpu.cpu_load[1].stddev
     26.67 ±  9%       -29%      19.00             -27%      19.50        sched_debug.cpu.cpu_load[1].max
       209 ±  8%        19%        247             -15%        178        sched_debug.cpu.sched_goidle.avg
      5.67 ± 27%       -12%       5.00              50%       8.50        sched_debug.cpu.nr_running.max
     36072 ±  7%        70%      61152              17%      42236        sched_debug.cpu.sched_count.max
      2008              -8%       1847             -18%       1645        sched_debug.cpu.ttwu_count.stddev
      0.07 ± 19%       -20%       0.06             186%       0.21        sched_debug.rt_rq:/.rt_time.avg
      0.36 ± 17%       -23%       0.28             142%       0.88        sched_debug.rt_rq:/.rt_time.stddev
      2.33 ± 15%       -27%       1.70              87%       4.35        sched_debug.rt_rq:/.rt_time.max

aim7/1BRD_48G-xfs-disk_cp-3000-performance/ivb44

99091700659f4df9  ca2edab2e1d8f30dda874b7f71  fe9c2c81ed073878768785a985  
----------------  --------------------------  --------------------------  
    270459                      272267 ±  3%       -48%     139834 ±  3%  aim7.jobs-per-min
     21229 ±  5%                 20896 ±  3%       449%     116516 ±  6%  aim7.time.involuntary_context_switches
      1461 ±  5%                  1454 ±  5%       174%       3998 ±  3%  aim7.time.system_time
    155368                      153149             149%     386164        aim7.time.minor_page_faults
     66.84                       66.41 ±  3%        93%     129.07 ±  3%  aim7.time.elapsed_time
     66.84                       66.41 ±  3%        93%     129.07 ±  3%  aim7.time.elapsed_time.max
    328369               3%     339077              96%     644393        aim7.time.voluntary_context_switches
     49489 ± 11%        -8%      45459              39%      68941 ±  4%  interrupts.CAL:Function_call_interrupts
     96.62 ±  7%                 97.09              61%     155.12        uptime.boot
    186640 ± 10%                186707             127%     424522 ±  4%  softirqs.RCU
    146596                      147043              37%     201373        softirqs.SCHED
   1005660 ±  3%                991053 ±  4%       118%    2196513        softirqs.TIMER
     66.84                       66.41 ±  3%        93%     129.07 ±  3%  time.elapsed_time
     66.84                       66.41 ±  3%        93%     129.07 ±  3%  time.elapsed_time.max
     21229 ±  5%                 20896 ±  3%       449%     116516 ±  6%  time.involuntary_context_switches
    155368                      153149             149%     386164        time.minor_page_faults
      2212                        2215              41%       3112        time.percent_of_cpu_this_job_got
      1461 ±  5%                  1454 ±  5%       174%       3998 ±  3%  time.system_time
    328369               3%     339077              96%     644393        time.voluntary_context_switches
   1197810 ± 16%       -67%     393936 ± 40%       -56%     530668 ± 43%  numa-numastat.node0.numa_miss
   1196269 ± 16%       -68%     387751 ± 40%       -55%     533013 ± 42%  numa-numastat.node1.numa_foreign
       112             159%        292 ±  4%       146%        277        vmstat.memory.buff
  16422228                    16461619             -28%   11832310        vmstat.memory.free
        22              -3%         22              87%         42 ±  3%  vmstat.procs.r
     48853                       48768                       50273        vmstat.system.in
    125202               8%     135626              51%     189515 ±  4%  cpuidle.C1-IVT.usage
  28088338 ±  3%        11%   31082173              17%   32997314 ±  5%  cpuidle.C1-IVT.time
   3471814              27%    4422338 ± 15%      2877%  1.034e+08 ±  3%  cpuidle.C1E-IVT.time
     33353               8%      36128             703%     267725        cpuidle.C1E-IVT.usage
  11371800               9%   12381174             244%   39113028        cpuidle.C3-IVT.time
     64048               5%      67490              62%     103940        cpuidle.C3-IVT.usage
 1.637e+09                   1.631e+09              20%  1.959e+09        cpuidle.C6-IVT.time
   1861259               4%    1931551              19%    2223599        cpuidle.C6-IVT.usage
       230 ±  9%        42%        326            1631%       3986        cpuidle.POLL.usage
   1724995 ± 41%        54%    2656939 ± 10%       112%    3662791        cpuidle.POLL.time
     48.48                       48.15              36%      65.85        turbostat.%Busy
      1439                        1431              36%       1964        turbostat.Avg_MHz
     33.28                       33.45             -25%      24.85        turbostat.CPU%c1
     18.09 ±  3%                 18.24 ±  4%       -49%       9.16        turbostat.CPU%c6
       134                         133               8%        144        turbostat.CorWatt
      5.39 ± 17%         4%       5.63 ±  8%       -34%       3.54        turbostat.Pkg%pc2
      2.97 ± 44%       -17%       2.48 ± 32%       -70%       0.91 ± 22%  turbostat.Pkg%pc6
       167                         167               6%        177        turbostat.PkgWatt
     10.27                       10.43             -14%       8.79        turbostat.RAMWatt
  44376005            -100%     205734            -100%     214640        meminfo.Active
  44199835            -100%      30412            -100%      30241        meminfo.Active(file)
    103029 ±  3%        27%     130507 ±  6%        29%     133114 ±  8%  meminfo.CmaFree
    124701 ±  4%                123685 ± 14%        16%     144180 ±  3%  meminfo.DirectMap4k
      7886 ±  4%                  7993 ±  5%       144%      19231 ±  7%  meminfo.Dirty
   2472446            1791%   46747572            1976%   51320420        meminfo.Inactive
   2463353            1797%   46738477            1983%   51311261        meminfo.Inactive(file)
  16631615                    16664565             -28%   11936074        meminfo.MemFree
 4.125e+11              -5%  3.927e+11             103%   8.36e+11        perf-stat.branch-instructions
      0.41             -20%       0.33             -43%       0.23        perf-stat.branch-miss-rate
 1.671e+09             -23%   1.28e+09              16%  1.946e+09        perf-stat.branch-misses
 7.138e+09              -3%  6.917e+09              23%  8.746e+09        perf-stat.cache-misses
 2.036e+10              -4%  1.956e+10              22%  2.476e+10        perf-stat.cache-references
    821470               4%     851532              88%    1548125 ±  3%  perf-stat.context-switches
  4.93e+12 ±  3%        -4%  4.755e+12 ±  4%       154%   1.25e+13        perf-stat.cpu-cycles
    125073               4%     129993             167%     333599        perf-stat.cpu-migrations
 3.595e+09 ± 16%       -19%  2.895e+09 ± 17%        39%  4.987e+09 ± 10%  perf-stat.dTLB-load-misses
 6.411e+11                   6.339e+11 ±  3%        57%  1.004e+12        perf-stat.dTLB-loads
      0.06 ±  3%       -42%       0.04              87%       0.12 ±  3%  perf-stat.dTLB-store-miss-rate
 2.738e+08             -39%  1.675e+08              64%  4.502e+08 ±  5%  perf-stat.dTLB-store-misses
 4.321e+11               5%  4.552e+11             -12%   3.81e+11 ±  8%  perf-stat.dTLB-stores
 2.343e+12              -5%  2.229e+12              67%  3.918e+12        perf-stat.instructions
     46162 ± 41%                 46733 ±  3%        55%      71500        perf-stat.instructions-per-iTLB-miss
      0.48 ±  4%                  0.47 ±  5%       -34%       0.31        perf-stat.ipc
    325877                      322934             115%     699924        perf-stat.minor-faults
     42.88               3%      44.33                       43.65        perf-stat.node-load-miss-rate
 9.499e+08                   9.578e+08              66%  1.581e+09        perf-stat.node-load-misses
 1.266e+09              -5%  1.203e+09              61%   2.04e+09        perf-stat.node-loads
     39.17                       40.00               8%      42.12        perf-stat.node-store-miss-rate
 3.198e+09               4%  3.318e+09              36%  4.344e+09        perf-stat.node-store-misses
 4.966e+09                   4.977e+09              20%  5.968e+09        perf-stat.node-stores
    325852                      322963             115%     699918        perf-stat.page-faults
  21719324            -100%      15215 ±  3%      -100%      14631        numa-meminfo.node0.Active(file)
   1221037            1806%   23278263            1969%   25269114        numa-meminfo.node0.Inactive(file)
   1223564            1803%   23286857            1965%   25269597        numa-meminfo.node0.Inactive
  21811771            -100%     102448            -100%     104424        numa-meminfo.node0.Active
      2971 ± 13%        -8%       2734 ±  3%       157%       7626 ±  4%  numa-meminfo.node0.Dirty
   8476780                     8356206             -27%    6162743        numa-meminfo.node0.MemFree
    617361                      611434              11%     687829        numa-meminfo.node0.SReclaimable
   1249068            1779%   23471025            1985%   26046948        numa-meminfo.node1.Inactive
   1242501            1789%   23470523            1996%   26038272        numa-meminfo.node1.Inactive(file)
  22500867            -100%      15202 ±  4%      -100%      15613        numa-meminfo.node1.Active(file)
  22584509            -100%     103192 ±  6%      -100%     109976        numa-meminfo.node1.Active
      4814 ± 13%                  4957 ±  5%       135%      11335        numa-meminfo.node1.Dirty
   8132889                     8297084 ±  3%       -29%    5777419 ±  3%  numa-meminfo.node1.MemFree
     83641 ±  7%         5%      87990 ±  7%        13%      94363        numa-meminfo.node1.Active(anon)
     82877 ±  7%         4%      86528 ±  6%        13%      93620        numa-meminfo.node1.AnonPages
         0                           0                      842360 ±100%  latency_stats.avg.call_rwsem_down_write_failed.do_unlinkat.SyS_unlink.do_syscall_64.return_from_SYSCALL_64
       113 ±173%       232%        376 ±100%     2e+05%     203269 ±  4%  latency_stats.hits.wait_on_page_bit.__migration_entry_wait.migration_entry_wait.do_swap_page.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
      5245 ± 14%                  5325 ±  3%       535%      33286 ± 23%  latency_stats.max.down.xfs_buf_lock._xfs_buf_find.xfs_buf_get_map.xfs_buf_read_map.xfs_trans_read_buf_map.xfs_read_agi.xfs_ialloc_read_agi.xfs_dialloc.xfs_ialloc.xfs_dir_ialloc.xfs_create
      1133 ±173%       113%       2416 ±100%      1351%      16434        latency_stats.max.wait_on_page_bit.__migration_entry_wait.migration_entry_wait.do_swap_page.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
         0                           0                      842360 ±100%  latency_stats.max.call_rwsem_down_write_failed.do_unlinkat.SyS_unlink.do_syscall_64.return_from_SYSCALL_64
      7813 ± 13%       -33%       5197 ±  9%       403%      39305 ± 18%  latency_stats.max.down.xfs_buf_lock._xfs_buf_find.xfs_buf_get_map.xfs_buf_read_map.xfs_trans_read_buf_map.xfs_read_agi.xfs_iunlink_remove.xfs_ifree.xfs_inactive_ifree.xfs_inactive.xfs_fs_destroy_inode
      5271 ± 13%        -3%       5091 ±  5%       288%      20467        latency_stats.max.down.xfs_buf_lock._xfs_buf_find.xfs_buf_get_map.xfs_buf_read_map.xfs_trans_read_buf_map.xfs_read_agi.xfs_iunlink.xfs_droplink.xfs_remove.xfs_vn_unlink.vfs_unlink
     10369 ± 17%       -41%       6086 ± 21%       -96%        385 ±100%  latency_stats.max.wait_on_page_bit.truncate_inode_pages_range.truncate_inode_pages_final.evict.iput.dentry_unlink_inode.__dentry_kill.dput.__fput.____fput.task_work_run.exit_to_usermode_loop
     94417 ±173%       556%     619712 ±100%     3e+05%  3.061e+08 ±  5%  latency_stats.sum.wait_on_page_bit.__migration_entry_wait.migration_entry_wait.do_swap_page.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
  22126648 ±  4%              22776886            1311%  3.123e+08 ±  7%  latency_stats.sum.down.xfs_buf_lock._xfs_buf_find.xfs_buf_get_map.xfs_buf_read_map.xfs_trans_read_buf_map.xfs_read_agi.xfs_iunlink_remove.xfs_ifree.xfs_inactive_ifree.xfs_inactive.xfs_fs_destroy_inode
      2536 ±117%       -98%         48 ± 43%      2059%      54765 ±100%  latency_stats.sum.down.xfs_buf_lock._xfs_buf_find.xfs_buf_get_map.xfs_buf_read_map.xfs_trans_read_buf_map.xfs_read_agf.xfs_alloc_read_agf.xfs_alloc_fix_freelist.xfs_free_extent_fix_freelist.xfs_free_extent.xfs_trans_free_extent
   1702264 ±  3%         5%    1790192             509%   10359205 ±  6%  latency_stats.sum.down.xfs_buf_lock._xfs_buf_find.xfs_buf_get_map.xfs_buf_read_map.xfs_trans_read_buf_map.xfs_read_agi.xfs_ialloc_read_agi.xfs_dialloc.xfs_ialloc.xfs_dir_ialloc.xfs_create
   1180839 ±  3%         5%    1238547             453%    6527115 ±  5%  latency_stats.sum.down.xfs_buf_lock._xfs_buf_find.xfs_buf_get_map.xfs_buf_read_map.xfs_trans_read_buf_map.xfs_read_agi.xfs_iunlink.xfs_droplink.xfs_remove.xfs_vn_unlink.vfs_unlink
       467 ±173%       680%       3644 ±  4%     7e+05%    3196407 ±  3%  latency_stats.sum.xfs_iget.xfs_ialloc.xfs_dir_ialloc.xfs_create.xfs_generic_create.xfs_vn_mknod.xfs_vn_create.path_openat.do_filp_open.do_sys_open.SyS_creat.entry_SYSCALL_64_fastpath
         0                           0                      842360 ±100%  latency_stats.sum.call_rwsem_down_write_failed.do_unlinkat.SyS_unlink.do_syscall_64.return_from_SYSCALL_64
    159018 ± 43%       -49%      81514 ± 19%       -99%        999 ±100%  latency_stats.sum.wait_on_page_bit.truncate_inode_pages_range.truncate_inode_pages_final.evict.iput.dentry_unlink_inode.__dentry_kill.dput.__fput.____fput.task_work_run.exit_to_usermode_loop
      1084 ±  5%        21%       1313 ±  3%        86%       2018        proc-vmstat.kswapd_high_wmark_hit_quickly
      1817 ±  3%        38%       2511 ±  3%       175%       4989        proc-vmstat.kswapd_low_wmark_hit_quickly
  11055004            -100%       7603            -100%       7559        proc-vmstat.nr_active_file
      1993                        2013 ±  4%       128%       4553 ±  5%  proc-vmstat.nr_dirty
     25746 ±  3%        26%      32494 ±  6%        29%      33319 ±  8%  proc-vmstat.nr_free_cma
   4152484                     4162399             -28%    2984494        proc-vmstat.nr_free_pages
    615907            1798%   11688190            1983%   12827366        proc-vmstat.nr_inactive_file
  11055042            -100%       7603            -100%       7559        proc-vmstat.nr_zone_active_file
    615904            1798%   11688234            1983%   12827434        proc-vmstat.nr_zone_inactive_file
      2016 ±  3%                  2025 ±  4%       127%       4582 ±  4%  proc-vmstat.nr_zone_write_pending
      2912 ±  3%        32%       3834 ±  3%       141%       7009        proc-vmstat.pageoutrun
   5380414            -100%       2502 ±  3%      -100%       2602 ±  3%  proc-vmstat.pgactivate
  61925072            -100%          0            -100%          0        proc-vmstat.pgdeactivate
    348105                      343315             108%     723517        proc-vmstat.pgfault
  61932469            -100%          0            -100%          0        proc-vmstat.pgrefill
   5432311            -100%       3802 ±  3%      -100%       3657        numa-vmstat.node0.nr_zone_active_file
   5432276            -100%       3802 ±  3%      -100%       3657        numa-vmstat.node0.nr_active_file
    305236            1802%    5806215            1969%    6314975        numa-vmstat.node0.nr_zone_inactive_file
    305239            1802%    5806170            1969%    6314910        numa-vmstat.node0.nr_inactive_file
       748 ±  7%       -20%        597 ± 10%       114%       1602        numa-vmstat.node0.nr_dirty
       775 ±  7%       -21%        610 ± 12%       112%       1642        numa-vmstat.node0.nr_zone_write_pending
   2116796                     2102494 ±  3%       -27%    1543100        numa-vmstat.node0.nr_free_pages
    154392                      152538              11%     171898        numa-vmstat.node0.nr_slab_reclaimable
    310642            1784%    5853811            1995%    6507801        numa-vmstat.node1.nr_zone_inactive_file
    310642            1784%    5853814            1995%    6507801        numa-vmstat.node1.nr_inactive_file
   5627293            -100%       3799 ±  4%      -100%       3903        numa-vmstat.node1.nr_zone_active_file
   5627293            -100%       3799 ±  4%      -100%       3903        numa-vmstat.node1.nr_active_file
      1206 ± 16%        14%       1373             129%       2758 ± 10%  numa-vmstat.node1.nr_zone_write_pending
      1205 ± 16%        14%       1373             129%       2757 ± 10%  numa-vmstat.node1.nr_dirty
   2031121                     2088592 ±  3%       -29%    1446172 ±  3%  numa-vmstat.node1.nr_free_pages
     25743 ±  3%        27%      32608 ±  7%        30%      33415 ±  8%  numa-vmstat.node1.nr_free_cma
     20877 ±  7%         6%      22077 ±  6%        13%      23620        numa-vmstat.node1.nr_zone_active_anon
     20877 ±  7%         6%      22077 ±  6%        13%      23620        numa-vmstat.node1.nr_active_anon
     20684 ±  7%         5%      21709 ±  6%        13%      23431        numa-vmstat.node1.nr_anon_pages
      4687                        4704              11%       5205 ±  3%  slabinfo.kmalloc-128.num_objs
      4687                        4704              11%       5205 ±  3%  slabinfo.kmalloc-128.active_objs
      1401             -19%       1142               8%       1516 ±  6%  slabinfo.xfs_efd_item.num_objs
      1401             -19%       1142               8%       1516 ±  6%  slabinfo.xfs_efd_item.active_objs
      1725 ±  5%        -8%       1589             -12%       1518        slabinfo.xfs_inode.num_objs
      1725 ±  5%        -8%       1589             -12%       1518        slabinfo.xfs_inode.active_objs
    382810 ±  4%                383813 ±  3%       301%    1535378        sched_debug.cfs_rq:/.min_vruntime.avg
    249011 ±  6%                245840 ±  3%       420%    1294704        sched_debug.cfs_rq:/.min_vruntime.min
    105216                      106278              79%     188096        sched_debug.cfs_rq:/.min_vruntime.stddev
    105260                      106358              79%     188314        sched_debug.cfs_rq:/.spread0.stddev
      9414 ±  4%                  9361 ±  4%       230%      31092        sched_debug.cfs_rq:/.exec_clock.min
    541056 ±  9%                540188 ±  3%       236%    1820030        sched_debug.cfs_rq:/.min_vruntime.max
    150.87 ± 11%       -21%     119.80 ± 10%        34%     202.73 ±  7%  sched_debug.cfs_rq:/.util_avg.stddev
     13783                       13656             170%      37192        sched_debug.cfs_rq:/.exec_clock.avg
     17625                       17508             141%      42564        sched_debug.cfs_rq:/.exec_clock.max
   3410.74 ±  3%               3458.30              38%    4706.14        sched_debug.cfs_rq:/.exec_clock.stddev
       732 ± 11%        11%        809 ±  3%       -34%        480 ±  7%  sched_debug.cfs_rq:/.load_avg.min
       844 ±  8%         7%        901             -33%        569 ±  4%  sched_debug.cfs_rq:/.load_avg.avg
      0.41 ±  7%        11%       0.46 ± 11%        21%       0.50 ±  5%  sched_debug.cfs_rq:/.nr_running.avg
      1339 ±  5%                  1338             -32%        909        sched_debug.cfs_rq:/.load_avg.max
      0.53 ±  4%        -4%       0.51              32%       0.70        sched_debug.cfs_rq:/.nr_spread_over.avg
      0.50                        0.50              33%       0.67        sched_debug.cfs_rq:/.nr_spread_over.min
    355.00 ± 26%       -67%     118.75 ±  4%       -82%      64.83 ± 20%  sched_debug.cpu.cpu_load[4].max
     18042                       17697             135%      42380        sched_debug.cpu.nr_load_updates.min
     51.83 ± 22%       -66%      17.44             -78%      11.18 ±  5%  sched_debug.cpu.cpu_load[4].stddev
     22708                       22546             111%      47986        sched_debug.cpu.nr_load_updates.avg
     29633 ±  7%        -7%      27554              83%      54243        sched_debug.cpu.nr_load_updates.max
     48.83 ± 29%       -65%      16.91 ± 29%       -73%      13.34 ± 13%  sched_debug.cpu.cpu_load[3].stddev
    329.25 ± 34%       -65%     113.75 ± 30%       -76%      79.67 ± 28%  sched_debug.cpu.cpu_load[3].max
     17106              14%      19541 ± 19%        34%      22978 ±  6%  sched_debug.cpu.nr_switches.max
      1168 ±  4%        -3%       1131 ±  4%       144%       2846 ± 21%  sched_debug.cpu.ttwu_local.max
      3826 ±  3%                  3766              17%       4487        sched_debug.cpu.nr_load_updates.stddev
     19.73 ± 12%        -4%      18.86 ± 14%        59%      31.42 ±  8%  sched_debug.cpu.nr_uninterruptible.avg
    149.75 ±  8%                150.00 ± 11%        42%     212.50        sched_debug.cpu.nr_uninterruptible.max
     98147 ± 34%                 97985 ± 42%        59%     156085 ±  8%  sched_debug.cpu.avg_idle.min
      8554 ±  3%         4%       8896 ±  5%        62%      13822        sched_debug.cpu.nr_switches.avg
      2582 ±  3%        11%       2857 ± 11%        19%       3083 ±  3%  sched_debug.cpu.nr_switches.stddev
     60029 ±  9%                 60817 ±  7%        44%      86205        sched_debug.cpu.clock.max
     60029 ±  9%                 60817 ±  7%        44%      86205        sched_debug.cpu.clock_task.max
     60020 ±  9%                 60807 ±  7%        44%      86188        sched_debug.cpu.clock.avg
     60020 ±  9%                 60807 ±  7%        44%      86188        sched_debug.cpu.clock_task.avg
     60008 ±  9%                 60793 ±  7%        44%      86169        sched_debug.cpu.clock.min
     60008 ±  9%                 60793 ±  7%        44%      86169        sched_debug.cpu.clock_task.min
     18.36 ±  7%       -37%      11.60 ±  5%       -33%      12.21        sched_debug.cpu.cpu_load[3].avg
      5577 ±  6%         3%       5772 ±  6%        81%      10121        sched_debug.cpu.nr_switches.min
     19.14 ±  3%       -36%      12.24             -36%      12.33        sched_debug.cpu.cpu_load[4].avg
     17.21 ± 14%       -31%      11.90 ± 18%       -27%      12.56 ±  6%  sched_debug.cpu.cpu_load[2].avg
     83.49 ±  7%         5%      87.64 ±  3%        17%      97.56 ±  4%  sched_debug.cpu.nr_uninterruptible.stddev
      3729                        3735              18%       4409 ± 13%  sched_debug.cpu.curr->pid.max
       374 ±  9%        -4%        360 ±  9%       157%        962        sched_debug.cpu.ttwu_local.min
       665                         671             122%       1479        sched_debug.cpu.ttwu_local.avg
       196 ±  7%         5%        207 ±  8%        88%        369 ± 14%  sched_debug.cpu.ttwu_local.stddev
      1196 ±  4%         5%       1261 ±  6%        11%       1333 ± 10%  sched_debug.cpu.curr->pid.stddev
      0.45 ±  7%        17%       0.53 ± 16%        29%       0.58 ± 16%  sched_debug.cpu.nr_running.avg
      6738 ± 16%         8%       7296 ± 20%        52%      10236        sched_debug.cpu.ttwu_count.max
      3952 ±  4%         5%       4150 ±  5%        75%       6917        sched_debug.cpu.ttwu_count.avg
       913              22%       1117 ± 18%        42%       1302 ±  3%  sched_debug.cpu.sched_goidle.stddev
      2546 ±  4%         4%       2653 ±  7%        89%       4816        sched_debug.cpu.ttwu_count.min
      5301 ±  6%        36%       7190 ± 33%        61%       8513 ±  8%  sched_debug.cpu.sched_goidle.max
      4683 ± 16%        14%       5355 ± 25%        52%       7125        sched_debug.cpu.sched_count.stddev
      8262 ±  3%         6%       8746 ±  7%        68%      13912        sched_debug.cpu.sched_count.avg
      5139 ±  5%         4%       5362 ±  6%        90%       9773        sched_debug.cpu.sched_count.min
      2088 ±  6%         7%       2229 ±  5%        55%       3242        sched_debug.cpu.sched_goidle.min
      3258 ±  4%         6%       3445 ±  6%        44%       4706        sched_debug.cpu.sched_goidle.avg
     37088 ± 17%        12%      41540 ± 23%        60%      59447        sched_debug.cpu.sched_count.max
      1007 ±  7%        13%       1139 ± 14%        38%       1386 ±  3%  sched_debug.cpu.ttwu_count.stddev
    262591 ±  4%        -3%     253748 ±  4%       -11%     232974        sched_debug.cpu.avg_idle.stddev
     60009 ±  9%                 60795 ±  7%        44%      86169        sched_debug.cpu_clk
     58763 ±  9%                 59673 ±  7%        45%      85068        sched_debug.ktime
     60009 ±  9%                 60795 ±  7%        44%      86169        sched_debug.sched_clk

aim7/1BRD_48G-xfs-creat-clo-1500-performance/ivb44

99091700659f4df9  fe9c2c81ed073878768785a985  
----------------  --------------------------  
     69789               5%      73162        aim7.jobs-per-min
     81603              -7%      75897 ±  5%  aim7.time.involuntary_context_switches
      3825              -6%       3583        aim7.time.system_time
    129.08              -5%     123.16        aim7.time.elapsed_time
    129.08              -5%     123.16        aim7.time.elapsed_time.max
      2536              -4%       2424        aim7.time.maximum_resident_set_size
      3145             131%       7253 ± 20%  numa-numastat.node1.numa_miss
      3145             131%       7253 ± 20%  numa-numastat.node1.numa_foreign
      7059               4%       7362        vmstat.system.cs
   7481848              40%   10487336 ±  8%  cpuidle.C1-IVT.time
   1491314              75%    2607219 ± 10%  cpuidle.POLL.time
        67              10%         73 ±  4%  turbostat.CoreTmp
        66              12%         73 ±  4%  turbostat.PkgTmp
   5025792             -21%    3973802        meminfo.DirectMap2M
     49098              12%      54859        meminfo.PageTables
      3.94              97%       7.76 ± 18%  perf-profile.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.call_cpuidle.cpu_startup_entry
     11.88             -24%       8.99 ± 14%  perf-profile.cycles-pp.apic_timer_interrupt.cpuidle_enter.call_cpuidle.cpu_startup_entry.start_secondary
     11.63             -25%       8.78 ± 13%  perf-profile.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter.call_cpuidle.cpu_startup_entry
 8.412e+11              -7%   7.83e+11        perf-stat.branch-instructions
      0.30                        0.29        perf-stat.branch-miss-rate
 2.495e+09              -8%  2.292e+09        perf-stat.branch-misses
 4.277e+09              -6%  4.003e+09        perf-stat.cache-misses
 1.396e+10              -5%  1.327e+10        perf-stat.cache-references
 1.224e+13              -8%   1.12e+13        perf-stat.cpu-cycles
      0.58             -57%       0.25 ± 16%  perf-stat.dTLB-load-miss-rate
 5.407e+09             -60%  2.175e+09 ± 18%  perf-stat.dTLB-load-misses
 9.243e+11              -6%  8.708e+11        perf-stat.dTLB-loads
      0.17             -58%       0.07 ±  4%  perf-stat.dTLB-store-miss-rate
 4.368e+08             -50%  2.177e+08 ±  3%  perf-stat.dTLB-store-misses
 2.549e+11              19%  3.041e+11        perf-stat.dTLB-stores
 3.737e+12              -6%  3.498e+12        perf-stat.instructions
      0.31                        0.31        perf-stat.ipc
    439716                      426816        perf-stat.minor-faults
 2.164e+09              -7%  2.012e+09        perf-stat.node-load-misses
 2.417e+09              -7%  2.259e+09        perf-stat.node-loads
  1.24e+09              -3%  1.198e+09        perf-stat.node-store-misses
 1.556e+09              -4%  1.501e+09        perf-stat.node-stores
    439435                      426823        perf-stat.page-faults
     51452              14%      58403 ±  8%  numa-meminfo.node0.Active(anon)
     10472             -36%       6692 ± 45%  numa-meminfo.node1.Shmem
      7665              74%      13316        numa-meminfo.node1.PageTables
      6724             144%      16416 ± 43%  latency_stats.avg.perf_event_alloc.SYSC_perf_event_open.SyS_perf_event_open.entry_SYSCALL_64_fastpath
      6724             144%      16416 ± 43%  latency_stats.max.perf_event_alloc.SYSC_perf_event_open.SyS_perf_event_open.entry_SYSCALL_64_fastpath
      6724             144%      16416 ± 43%  latency_stats.sum.perf_event_alloc.SYSC_perf_event_open.SyS_perf_event_open.entry_SYSCALL_64_fastpath
     12237              12%      13693        proc-vmstat.nr_page_table_pages
     12824              14%      14578 ±  8%  numa-vmstat.node0.nr_zone_active_anon
     12824              14%      14578 ±  8%  numa-vmstat.node0.nr_active_anon
      2618             -36%       1672 ± 45%  numa-vmstat.node1.nr_shmem
     17453              24%      21726 ±  6%  numa-vmstat.node1.numa_miss
      1909              74%       3323        numa-vmstat.node1.nr_page_table_pages
     17453              24%      21726 ±  6%  numa-vmstat.node1.numa_foreign
       922              24%       1143 ±  6%  slabinfo.blkdev_requests.active_objs
       922              24%       1143 ±  6%  slabinfo.blkdev_requests.num_objs
       569              21%        686 ± 11%  slabinfo.file_lock_cache.active_objs
       569              21%        686 ± 11%  slabinfo.file_lock_cache.num_objs
      9.07              16%      10.56 ±  9%  sched_debug.cfs_rq:/.runnable_load_avg.avg
     18406             -14%      15835 ± 10%  sched_debug.cfs_rq:/.load.stddev
      0.67             150%       1.67 ± 43%  sched_debug.cfs_rq:/.nr_spread_over.max
       581             -11%        517 ±  4%  sched_debug.cfs_rq:/.load_avg.min
       659             -10%        596 ±  4%  sched_debug.cfs_rq:/.load_avg.avg
       784             -12%        692 ±  4%  sched_debug.cfs_rq:/.load_avg.max
     18086             -12%      15845 ±  9%  sched_debug.cpu.load.stddev
     18.72             -17%      15.49 ±  8%  sched_debug.cpu.nr_uninterruptible.avg
     69.33              42%      98.67 ±  7%  sched_debug.cpu.nr_uninterruptible.max
    317829             -12%     280218 ±  4%  sched_debug.cpu.avg_idle.min
      9.80              18%      11.54 ± 10%  sched_debug.cpu.cpu_load[1].avg
      8.91              15%      10.28 ±  9%  sched_debug.cpu.cpu_load[0].avg
      9.53              22%      11.64 ± 10%  sched_debug.cpu.cpu_load[3].avg
      7083              11%       7853        sched_debug.cpu.nr_switches.min
      9.73              22%      11.90 ±  7%  sched_debug.cpu.cpu_load[4].avg
      9.68              20%      11.59 ± 11%  sched_debug.cpu.cpu_load[2].avg
     24.59              49%      36.53 ± 17%  sched_debug.cpu.nr_uninterruptible.stddev
      1176              12%       1319 ±  4%  sched_debug.cpu.curr->pid.avg
       373              35%        502 ±  6%  sched_debug.cpu.ttwu_local.min
      3644              13%       4120 ±  3%  sched_debug.cpu.ttwu_count.min
      4855              13%       5463 ±  6%  sched_debug.cpu.sched_goidle.max
      7019              10%       7745        sched_debug.cpu.sched_count.min
      2305              10%       2529 ±  3%  sched_debug.cpu.sched_goidle.min
      0.00             -19%       0.00 ±  7%  sched_debug.cpu.next_balance.stddev
      0.68             -17%       0.57 ± 11%  sched_debug.cpu.nr_running.stddev
      0.05              27%       0.06 ± 14%  sched_debug.rt_rq:/.rt_nr_running.stddev

Thanks,
Fengguang

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ