lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Date:   Sun, 14 Feb 2021 21:25:47 +0800
From:   kernel test robot <oliver.sang@...el.com>
To:     "Darrick J. Wong" <djwong@...nel.org>
Cc:     LKML <linux-kernel@...r.kernel.org>,
        "Darrick J. Wong" <darrick.wong@...cle.com>, lkp@...ts.01.org,
        lkp@...el.com, ying.huang@...el.com, feng.tang@...el.com,
        zhengjun.xing@...el.com
Subject: [xfs]  f4da454e15:  stress-ng.dentry.ops_per_sec -36.8% regression


Greeting,

FYI, we noticed a -36.8% regression of stress-ng.dentry.ops_per_sec due to commit:


commit: f4da454e1584a4d1d248733b75e84f8bd1717c04 ("xfs: deferred inode inactivation")
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git reserve-rt-metadata-space


in testcase: stress-ng
on test machine: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 512G memory
with following parameters:

	nr_threads: 10%
	disk: 1HDD
	testtime: 60s
	fs: xfs
	class: filesystem
	test: dentry
	cpufreq_governor: performance
	ucode: 0x5003003




If you fix the issue, kindly add following tag
Reported-by: kernel test robot <oliver.sang@...el.com>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install                job.yaml  # job file is attached in this email
        bin/lkp split-job --compatible job.yaml
        bin/lkp run                    compatible-job.yaml

=========================================================================================
class/compiler/cpufreq_governor/disk/fs/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime/ucode:
  filesystem/gcc-9/performance/1HDD/xfs/x86_64-rhel-8.3/10%/debian-10.4-x86_64-20200603.cgz/lkp-csl-2sp7/dentry/stress-ng/60s/0x5003003

commit: 
  3a66a7bb7f ("xfs: decide if inode needs inactivation")
  f4da454e15 ("xfs: deferred inode inactivation")

3a66a7bb7fc1db25 f4da454e1584a4d1d248733b75e 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          0:6            6%           0:6     perf-profile.children.cycles-pp.error_entry
          0:6            7%           0:6     perf-profile.self.cycles-pp.error_entry
         %stddev     %change         %stddev
             \          |                \  
   1369615           -36.8%     865183 ±  4%  stress-ng.dentry.ops
     22689           -36.8%      14344 ±  4%  stress-ng.dentry.ops_per_sec
      3424 ±  6%     -47.9%       1784 ± 10%  stress-ng.time.involuntary_context_switches
    155.33 ±  2%     -54.6%      70.50 ±  4%  stress-ng.time.percent_of_cpu_this_job_got
     93.52 ±  2%     -55.4%      41.76 ±  3%  stress-ng.time.system_time
   3708024           -56.0%    1630527 ±  3%  stress-ng.time.voluntary_context_switches
    649772 ± 10%     +20.4%     782242 ±  7%  numa-vmstat.node1.numa_local
    252.11            -8.9%     229.76        pmeter.Average_Active_Power
    357411           -39.4%     216570 ±  8%  cpuidle.POLL.time
     32024 ±  3%     -22.8%      24713 ±  9%  cpuidle.POLL.usage
     11365           +56.8%      17817        vmstat.io.bo
   1285188           +28.2%    1646990 ±  2%  vmstat.memory.cache
    115332           -50.3%      57331 ±  2%  vmstat.system.cs
     76735 ±  6%     -18.4%      62592 ± 10%  interrupts.CAL:Function_call_interrupts
     27490 ±  3%     -28.1%      19770 ± 10%  interrupts.NMI:Non-maskable_interrupts
     27490 ±  3%     -28.1%      19770 ± 10%  interrupts.PMI:Performance_monitoring_interrupts
      7730 ± 11%     -48.7%       3962 ±  5%  interrupts.RES:Rescheduling_interrupts
      0.00 ± 57%      +0.0        0.05 ±  7%  mpstat.cpu.all.iowait%
      1.27            +0.4        1.70 ± 18%  mpstat.cpu.all.irq%
      1.53 ±  2%      -0.7        0.80 ±  2%  mpstat.cpu.all.sys%
      0.17 ±  7%      -0.0        0.14 ± 12%  mpstat.cpu.all.usr%
      3996           -10.2%       3589        meminfo.Active
      3732           -10.7%       3334        meminfo.Active(anon)
    221392          +160.1%     575734 ±  7%  meminfo.KReclaimable
   2952006           +19.9%    3540247        meminfo.Memused
    221392          +160.1%     575734 ±  7%  meminfo.SReclaimable
    188686           +38.9%     262104 ±  7%  meminfo.SUnreclaim
    410079          +104.3%     837838 ±  7%  meminfo.Slab
   2999998           +18.5%    3555063 ±  2%  meminfo.max_used_kB
    928.67           -10.4%     831.83        proc-vmstat.nr_active_anon
      3843            -2.8%       3736        proc-vmstat.nr_shmem
     55432          +159.4%     143789 ±  7%  proc-vmstat.nr_slab_reclaimable
     47169           +38.9%      65510 ±  7%  proc-vmstat.nr_slab_unreclaimable
    928.67           -10.4%     831.83        proc-vmstat.nr_zone_active_anon
    581683 ±  2%     +24.2%     722572 ±  5%  proc-vmstat.numa_hit
    494979 ±  2%     +28.5%     635895 ±  6%  proc-vmstat.numa_local
      1170 ±  5%     -26.6%     858.83 ±  7%  proc-vmstat.pgactivate
    640090 ±  3%     +53.6%     983382 ±  8%  proc-vmstat.pgalloc_normal
    750072           +58.3%    1187409 ±  4%  proc-vmstat.pgpgout
     14221 ±  9%     -37.3%       8921 ± 13%  softirqs.CPU0.RCU
      9115 ± 16%     -29.3%       6445 ± 18%  softirqs.CPU10.RCU
      8920 ± 18%     -27.3%       6488 ± 11%  softirqs.CPU12.RCU
      9366 ± 19%     -30.8%       6477 ±  7%  softirqs.CPU13.RCU
      9508 ±  9%     -23.1%       7309 ± 11%  softirqs.CPU33.RCU
      9207 ±  8%     -18.5%       7504 ± 16%  softirqs.CPU44.RCU
      9369 ± 11%     -29.2%       6637 ± 13%  softirqs.CPU54.RCU
      9104 ± 16%     -27.8%       6575 ± 15%  softirqs.CPU55.RCU
      9031 ± 17%     -23.3%       6925 ± 14%  softirqs.CPU57.RCU
      9250 ± 14%     -27.8%       6679 ± 17%  softirqs.CPU58.RCU
      9051 ± 14%     -23.3%       6940 ±  8%  softirqs.CPU59.RCU
      9349 ± 20%     -31.7%       6388 ± 10%  softirqs.CPU6.RCU
      9253 ± 16%     -30.9%       6398 ± 13%  softirqs.CPU60.RCU
      8870 ± 14%     -28.1%       6381 ±  6%  softirqs.CPU61.RCU
      9024 ± 15%     -26.4%       6641 ± 12%  softirqs.CPU63.RCU
      9291 ± 15%     -26.8%       6805 ± 12%  softirqs.CPU7.RCU
      9341 ± 18%     -28.0%       6724 ± 12%  softirqs.CPU9.RCU
      9408 ± 11%     -20.3%       7494 ± 14%  softirqs.CPU91.RCU
      9031 ± 10%     -17.7%       7436 ± 11%  softirqs.CPU95.RCU
      4.15          +211.0%      12.90 ± 48%  perf-stat.i.MPKI
 1.821e+09           -53.9%  8.403e+08 ±  2%  perf-stat.i.branch-instructions
      0.77 ±  2%      +0.9        1.68 ± 39%  perf-stat.i.branch-miss-rate%
   2991406 ± 14%     +46.6%    4384047 ± 10%  perf-stat.i.cache-misses
    120216           -50.4%      59609 ±  2%  perf-stat.i.context-switches
      0.84 ±  3%     +92.5%       1.62 ± 13%  perf-stat.i.cpi
 7.973e+09 ±  4%     -21.7%  6.243e+09 ± 11%  perf-stat.i.cpu-cycles
      2783 ± 10%     -44.6%       1542 ± 10%  perf-stat.i.cycles-between-cache-misses
      0.02 ± 71%      +0.1        0.09 ± 85%  perf-stat.i.dTLB-load-miss-rate%
 2.545e+09 ±  2%     -57.3%  1.086e+09 ±  3%  perf-stat.i.dTLB-loads
      0.00 ± 75%      +0.0        0.02 ± 85%  perf-stat.i.dTLB-store-miss-rate%
 1.295e+09           -58.4%   5.38e+08 ±  3%  perf-stat.i.dTLB-stores
   4857590 ±  4%     -36.1%    3104446 ±  8%  perf-stat.i.iTLB-load-misses
   7419477           -37.9%    4606414 ± 10%  perf-stat.i.iTLB-loads
 9.536e+09           -55.8%  4.219e+09 ±  2%  perf-stat.i.instructions
      2000 ±  3%     -31.0%       1380 ±  6%  perf-stat.i.instructions-per-iTLB-miss
      1.20 ±  3%     -44.6%       0.66 ± 12%  perf-stat.i.ipc
      0.08 ±  4%     -21.7%       0.07 ± 11%  perf-stat.i.metric.GHz
     59.47 ±  2%     -55.9%      26.22 ±  2%  perf-stat.i.metric.M/sec
    265802 ±  7%     +39.8%     371625 ± 29%  perf-stat.i.node-loads
    227014 ± 10%     +58.7%     360306 ± 12%  perf-stat.i.node-stores
      4.13          +170.3%      11.15 ± 46%  perf-stat.overall.MPKI
      0.79            +1.0        1.74 ± 33%  perf-stat.overall.branch-miss-rate%
      0.84 ±  3%     +77.3%       1.48 ± 12%  perf-stat.overall.cpi
      2703 ±  9%     -47.1%       1431 ± 11%  perf-stat.overall.cycles-between-cache-misses
      0.02 ± 71%      +0.1        0.08 ± 83%  perf-stat.overall.dTLB-load-miss-rate%
      0.00 ± 78%      +0.0        0.02 ± 84%  perf-stat.overall.dTLB-store-miss-rate%
      1965 ±  3%     -30.4%       1368 ±  7%  perf-stat.overall.instructions-per-iTLB-miss
      1.20 ±  3%     -42.7%       0.69 ± 12%  perf-stat.overall.ipc
 1.792e+09           -53.8%  8.274e+08 ±  2%  perf-stat.ps.branch-instructions
   2944376 ± 14%     +46.6%    4315869 ± 10%  perf-stat.ps.cache-misses
    118262           -50.4%      58653 ±  2%  perf-stat.ps.context-switches
 7.847e+09 ±  4%     -21.7%  6.146e+09 ± 11%  perf-stat.ps.cpu-cycles
 2.504e+09 ±  2%     -57.3%  1.069e+09 ±  3%  perf-stat.ps.dTLB-loads
 1.274e+09           -58.4%  5.297e+08 ±  3%  perf-stat.ps.dTLB-stores
   4779380 ±  4%     -36.1%    3055664 ±  9%  perf-stat.ps.iTLB-load-misses
   7299542           -37.9%    4533500 ± 10%  perf-stat.ps.iTLB-loads
 9.384e+09           -55.7%  4.154e+09 ±  2%  perf-stat.ps.instructions
    261579 ±  7%     +39.8%     365777 ± 29%  perf-stat.ps.node-loads
    223469 ± 10%     +58.7%     354627 ± 12%  perf-stat.ps.node-stores
 5.943e+11           -54.7%  2.692e+11 ±  6%  perf-stat.total.instructions
    758118           -32.0%     515635        slabinfo.dentry.active_objs
     36107           -32.0%      24564        slabinfo.dentry.active_slabs
    758258           -32.0%     515861        slabinfo.dentry.num_objs
     36107           -32.0%      24564        slabinfo.dentry.num_slabs
      1958 ±  8%    +107.3%       4059 ± 33%  slabinfo.dmaengine-unmap-16.active_objs
      1958 ±  8%    +108.7%       4088 ± 32%  slabinfo.dmaengine-unmap-16.num_objs
     36181          +939.4%     376082 ±  6%  slabinfo.kmalloc-16.active_objs
    141.33          +941.2%       1471 ±  6%  slabinfo.kmalloc-16.active_slabs
     36181          +941.7%     376884 ±  6%  slabinfo.kmalloc-16.num_objs
    141.33          +941.2%       1471 ±  6%  slabinfo.kmalloc-16.num_slabs
     40092           -28.6%      28609 ±  2%  slabinfo.kmalloc-192.active_objs
      1911           -28.6%       1364 ±  2%  slabinfo.kmalloc-192.active_slabs
     40142           -28.6%      28667 ±  2%  slabinfo.kmalloc-192.num_objs
      1911           -28.6%       1364 ±  2%  slabinfo.kmalloc-192.num_slabs
      4793 ±  2%     +11.3%       5336 ±  2%  slabinfo.kmalloc-1k.active_objs
      4818 ±  2%     +11.6%       5379 ±  2%  slabinfo.kmalloc-1k.num_objs
      5797 ±  2%    +177.3%      16076 ±  7%  slabinfo.kmalloc-256.active_objs
    364.00 ±  2%    +176.4%       1006 ±  7%  slabinfo.kmalloc-256.active_slabs
      5831 ±  2%    +176.1%      16100 ±  7%  slabinfo.kmalloc-256.num_objs
    364.00 ±  2%    +176.4%       1006 ±  7%  slabinfo.kmalloc-256.num_slabs
     72078 ±  2%     +12.1%      80830 ±  2%  slabinfo.kmalloc-32.active_objs
     72161 ±  2%     +12.1%      80906 ±  2%  slabinfo.kmalloc-32.num_objs
      2262 ±  4%     +14.3%       2586 ±  3%  slabinfo.kmalloc-4k.num_objs
     22582 ±  4%    +171.3%      61261 ± 45%  slabinfo.kmalloc-512.active_objs
      3271 ±  3%    +137.5%       7769 ± 45%  slabinfo.kmalloc-512.active_slabs
     26173 ±  3%    +137.5%      62157 ± 45%  slabinfo.kmalloc-512.num_objs
      3271 ±  3%    +137.5%       7769 ± 45%  slabinfo.kmalloc-512.num_slabs
      1188 ±  7%    +345.6%       5296 ± 13%  slabinfo.kmalloc-8k.active_objs
    306.00 ±  6%    +359.3%       1405 ± 12%  slabinfo.kmalloc-8k.active_slabs
      1226 ±  6%    +358.5%       5622 ± 12%  slabinfo.kmalloc-8k.num_objs
    306.00 ±  6%    +359.3%       1405 ± 12%  slabinfo.kmalloc-8k.num_slabs
      2337 ± 46%    +162.7%       6140 ± 11%  slabinfo.numa_policy.active_objs
    157.83 ± 45%    +169.5%     425.33 ± 10%  slabinfo.numa_policy.active_slabs
      2375 ± 45%    +168.9%       6387 ± 10%  slabinfo.numa_policy.num_objs
    157.83 ± 45%    +169.5%     425.33 ± 10%  slabinfo.numa_policy.num_slabs
     14793           +33.8%      19788 ±  2%  slabinfo.radix_tree_node.active_objs
      1057           +33.7%       1413 ±  2%  slabinfo.radix_tree_node.active_slabs
     14811           +33.6%      19795 ±  2%  slabinfo.radix_tree_node.num_objs
      1057           +33.7%       1413 ±  2%  slabinfo.radix_tree_node.num_slabs
      4388 ± 50%    +191.1%      12773 ± 10%  slabinfo.xfs_buf.active_objs
    106.50 ± 47%    +186.1%     304.67 ± 10%  slabinfo.xfs_buf.active_slabs
      4484 ± 47%    +186.0%      12825 ± 10%  slabinfo.xfs_buf.num_objs
    106.50 ± 47%    +186.1%     304.67 ± 10%  slabinfo.xfs_buf.num_slabs
      2058 ±  7%     +39.7%       2876 ± 10%  slabinfo.xfs_icr.active_objs
      2058 ±  7%     +39.7%       2876 ± 10%  slabinfo.xfs_icr.num_objs
     15764 ±  2%   +2097.1%     346362 ± 10%  slabinfo.xfs_ili.active_objs
    382.00 ±  2%   +2058.9%       8247 ± 10%  slabinfo.xfs_ili.active_slabs
     16064 ±  2%   +2056.3%     346396 ± 10%  slabinfo.xfs_ili.num_objs
    382.00 ±  2%   +2058.9%       8247 ± 10%  slabinfo.xfs_ili.num_slabs
     15023         +2202.3%     345896 ± 10%  slabinfo.xfs_inode.active_objs
    476.17         +2170.1%      10809 ± 10%  slabinfo.xfs_inode.active_slabs
     15259         +2167.0%     345923 ± 10%  slabinfo.xfs_inode.num_objs
    476.17         +2170.1%      10809 ± 10%  slabinfo.xfs_inode.num_slabs
     38.47 ±  4%     -13.5       24.97 ±  9%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
     38.84 ±  3%     -13.4       25.43 ±  9%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
     10.76 ±  4%     -10.8        0.00        perf-profile.calltrace.cycles-pp.destroy_inode.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe
     10.61 ±  4%     -10.6        0.00        perf-profile.calltrace.cycles-pp.xfs_fs_destroy_inode.destroy_inode.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe
     10.57 ±  4%     -10.6        0.00        perf-profile.calltrace.cycles-pp.xfs_inactive.xfs_fs_destroy_inode.destroy_inode.do_unlinkat.do_syscall_64
     10.55 ±  4%     -10.6        0.00        perf-profile.calltrace.cycles-pp.xfs_inactive_ifree.xfs_inactive.xfs_fs_destroy_inode.destroy_inode.do_unlinkat
     19.27 ±  3%      -9.2       10.03 ±  8%  perf-profile.calltrace.cycles-pp.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      8.36 ±  5%      -8.4        0.00        perf-profile.calltrace.cycles-pp.xfs_ifree.xfs_inactive_ifree.xfs_inactive.xfs_fs_destroy_inode.destroy_inode
      7.07 ±  5%      -7.1        0.00        perf-profile.calltrace.cycles-pp.xfs_difree.xfs_ifree.xfs_inactive_ifree.xfs_inactive.xfs_fs_destroy_inode
      6.26 ±  5%      -5.7        0.53 ± 47%  perf-profile.calltrace.cycles-pp.xfs_check_agi_freecount.xfs_dialloc_ag.xfs_dir_ialloc.xfs_create.xfs_generic_create
      5.36 ±  5%      -5.4        0.00        perf-profile.calltrace.cycles-pp.xfs_difree_inobt.xfs_difree.xfs_ifree.xfs_inactive_ifree.xfs_inactive
      7.07 ±  5%      -5.0        2.06 ± 12%  perf-profile.calltrace.cycles-pp.xfs_dialloc_ag.xfs_dir_ialloc.xfs_create.xfs_generic_create.path_openat
      4.96 ±  5%      -5.0        0.00        perf-profile.calltrace.cycles-pp.xfs_check_agi_freecount.xfs_difree_inobt.xfs_difree.xfs_ifree.xfs_inactive_ifree
      8.86 ±  4%      -4.4        4.50 ±  9%  perf-profile.calltrace.cycles-pp.xfs_dir_ialloc.xfs_create.xfs_generic_create.path_openat.do_filp_open
     15.40 ±  5%      -4.3       11.14 ± 10%  perf-profile.calltrace.cycles-pp.xfs_create.xfs_generic_create.path_openat.do_filp_open.do_sys_openat2
     15.53 ±  5%      -4.2       11.31 ± 10%  perf-profile.calltrace.cycles-pp.xfs_generic_create.path_openat.do_filp_open.do_sys_openat2.do_sys_open
     16.56 ±  4%      -4.2       12.38 ± 10%  perf-profile.calltrace.cycles-pp.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
     16.55 ±  4%      -4.2       12.37 ± 10%  perf-profile.calltrace.cycles-pp.do_sys_openat2.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
     16.39 ±  4%      -4.2       12.21 ± 10%  perf-profile.calltrace.cycles-pp.do_filp_open.do_sys_openat2.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
     16.34 ±  5%      -4.2       12.18 ± 10%  perf-profile.calltrace.cycles-pp.path_openat.do_filp_open.do_sys_openat2.do_sys_open.do_syscall_64
      2.03 ±  6%      -0.7        1.37 ±  6%  perf-profile.calltrace.cycles-pp.xfs_buf_find.xfs_buf_get_map.xfs_buf_read_map.xfs_trans_read_buf_map.xfs_read_agi
      0.54 ± 48%      +0.3        0.81 ±  7%  perf-profile.calltrace.cycles-pp.rebalance_domains.__softirqentry_text_start.asm_call_sysvec_on_stack.do_softirq_own_stack.irq_exit_rcu
      1.18 ± 12%      +0.4        1.62 ±  5%  perf-profile.calltrace.cycles-pp.__softirqentry_text_start.asm_call_sysvec_on_stack.do_softirq_own_stack.irq_exit_rcu.sysvec_apic_timer_interrupt
      1.18 ± 13%      +0.4        1.63 ±  4%  perf-profile.calltrace.cycles-pp.asm_call_sysvec_on_stack.do_softirq_own_stack.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      1.18 ± 13%      +0.5        1.64 ±  5%  perf-profile.calltrace.cycles-pp.do_softirq_own_stack.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      0.73 ±  8%      +0.5        1.20 ±  8%  perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
      0.92 ±  5%      +0.5        1.40 ± 11%  perf-profile.calltrace.cycles-pp.xfs_iunlink.xfs_remove.xfs_vn_unlink.vfs_unlink.do_unlinkat
      1.17 ±  8%      +0.5        1.66 ±  9%  perf-profile.calltrace.cycles-pp.xlog_cil_insert_items.xfs_log_commit_cil.__xfs_trans_commit.xfs_remove.xfs_vn_unlink
      0.52 ± 45%      +0.5        1.02 ±  9%  perf-profile.calltrace.cycles-pp.xfs_buf_item_format.xlog_cil_insert_items.xfs_log_commit_cil.__xfs_trans_commit.xfs_remove
      1.50 ± 12%      +0.6        2.11 ±  3%  perf-profile.calltrace.cycles-pp.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      1.60 ±  9%      +0.6        2.23 ± 18%  perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry.start_secondary
      0.29 ±100%      +0.7        0.95 ± 13%  perf-profile.calltrace.cycles-pp.xfs_init_new_inode.xfs_dir_ialloc.xfs_create.xfs_generic_create.path_openat
      0.00            +0.7        0.72 ± 16%  perf-profile.calltrace.cycles-pp.xfs_btree_lookup.xfs_dialloc_ag_update_inobt.xfs_dialloc_ag.xfs_dir_ialloc.xfs_create
      0.00            +0.8        0.75 ± 18%  perf-profile.calltrace.cycles-pp.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack
      0.00            +0.8        0.82 ± 15%  perf-profile.calltrace.cycles-pp.xfs_dialloc_ag_update_inobt.xfs_dialloc_ag.xfs_dir_ialloc.xfs_create.xfs_generic_create
      2.77 ±  6%      +0.9        3.69 ±  7%  perf-profile.calltrace.cycles-pp.__xfs_trans_commit.xfs_remove.xfs_vn_unlink.vfs_unlink.do_unlinkat
      2.68 ±  5%      +0.9        3.61 ±  7%  perf-profile.calltrace.cycles-pp.xfs_log_commit_cil.__xfs_trans_commit.xfs_remove.xfs_vn_unlink.vfs_unlink
      1.38 ±  7%      +1.0        2.38 ± 11%  perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
      1.42 ±  8%      +1.1        2.56 ± 15%  perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
      1.71 ± 10%      +1.2        2.90 ± 12%  perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack
      7.82 ±  4%      +1.3        9.13 ±  8%  perf-profile.calltrace.cycles-pp.xfs_vn_unlink.vfs_unlink.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      7.95 ±  4%      +1.3        9.27 ±  8%  perf-profile.calltrace.cycles-pp.vfs_unlink.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +2.0        1.96 ± 39%  perf-profile.calltrace.cycles-pp.xfs_inactive_ifree.xfs_inactive.xfs_inactive_inode.xfs_inode_walk_ag.xfs_inode_walk
      0.00            +2.0        1.99 ± 39%  perf-profile.calltrace.cycles-pp.xfs_inactive.xfs_inactive_inode.xfs_inode_walk_ag.xfs_inode_walk.xfs_inactive_worker
      2.57 ±  7%      +2.0        4.57 ± 17%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack.sysvec_apic_timer_interrupt
      0.00            +2.0        2.04 ± 39%  perf-profile.calltrace.cycles-pp.xfs_inactive_inode.xfs_inode_walk_ag.xfs_inode_walk.xfs_inactive_worker.process_one_work
      0.00            +2.1        2.05 ± 40%  perf-profile.calltrace.cycles-pp.xfs_inactive_worker.process_one_work.worker_thread.kthread.ret_from_fork
      0.00            +2.1        2.05 ± 40%  perf-profile.calltrace.cycles-pp.xfs_inode_walk.xfs_inactive_worker.process_one_work.worker_thread.kthread
      0.00            +2.1        2.05 ± 40%  perf-profile.calltrace.cycles-pp.xfs_inode_walk_ag.xfs_inode_walk.xfs_inactive_worker.process_one_work.worker_thread
      0.58 ±  3%      +2.4        3.02 ± 27%  perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork
      0.62 ±  2%      +2.4        3.06 ± 27%  perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork
      4.99 ± 11%      +2.5        7.52 ±  6%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.77 ±  3%      +2.5        3.30 ± 26%  perf-profile.calltrace.cycles-pp.ret_from_fork
      0.77 ±  3%      +2.5        3.30 ± 26%  perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
      5.08 ± 11%      +2.6        7.73 ±  7%  perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      5.11 ± 11%      +2.7        7.77 ±  7%  perf-profile.calltrace.cycles-pp.asm_call_sysvec_on_stack.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      8.04 ± 13%      +4.0       12.00 ±  6%  perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle
     11.17 ± 11%      +4.6       15.79 ±  6%  perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     51.45 ±  3%      +8.5       60.00 ±  6%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
     52.54 ±  3%      +8.8       61.38 ±  5%  perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     58.82 ±  2%     +10.4       69.24 ±  3%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     58.87 ±  2%     +10.4       69.31 ±  3%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
     58.87 ±  2%     +10.4       69.30 ±  3%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     59.47 ±  2%     +10.6       70.03 ±  3%  perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
     38.71 ±  4%     -13.4       25.34 ±  9%  perf-profile.children.cycles-pp.do_syscall_64
     39.10 ±  3%     -13.3       25.82 ±  9%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     12.39 ±  5%     -11.7        0.65 ± 18%  perf-profile.children.cycles-pp.xfs_check_agi_freecount
     10.61 ±  4%     -10.6        0.02 ±142%  perf-profile.children.cycles-pp.xfs_fs_destroy_inode
     10.76 ±  4%     -10.6        0.19 ± 13%  perf-profile.children.cycles-pp.destroy_inode
     19.27 ±  3%      -9.2       10.04 ±  8%  perf-profile.children.cycles-pp.do_unlinkat
     10.55 ±  4%      -8.6        1.96 ± 39%  perf-profile.children.cycles-pp.xfs_inactive_ifree
     10.57 ±  4%      -8.6        1.99 ± 39%  perf-profile.children.cycles-pp.xfs_inactive
      8.36 ±  5%      -7.3        1.02 ± 36%  perf-profile.children.cycles-pp.xfs_ifree
      7.25 ±  7%      -7.0        0.23 ± 30%  perf-profile.children.cycles-pp.xfs_inobt_get_rec
      7.07 ±  5%      -6.5        0.54 ± 35%  perf-profile.children.cycles-pp.xfs_difree
      5.36 ±  5%      -5.0        0.34 ± 28%  perf-profile.children.cycles-pp.xfs_difree_inobt
      7.07 ±  5%      -5.0        2.07 ± 12%  perf-profile.children.cycles-pp.xfs_dialloc_ag
      8.86 ±  4%      -4.4        4.50 ±  9%  perf-profile.children.cycles-pp.xfs_dir_ialloc
     15.41 ±  5%      -4.3       11.14 ± 10%  perf-profile.children.cycles-pp.xfs_create
     15.53 ±  5%      -4.2       11.31 ± 10%  perf-profile.children.cycles-pp.xfs_generic_create
     16.57 ±  4%      -4.2       12.39 ± 10%  perf-profile.children.cycles-pp.do_sys_open
     16.56 ±  4%      -4.2       12.39 ± 10%  perf-profile.children.cycles-pp.do_sys_openat2
     16.40 ±  4%      -4.2       12.23 ± 10%  perf-profile.children.cycles-pp.do_filp_open
     16.36 ±  4%      -4.2       12.20 ± 10%  perf-profile.children.cycles-pp.path_openat
      4.22 ±  4%      -4.1        0.16 ± 15%  perf-profile.children.cycles-pp.xfs_btree_check_sblock
      3.76 ±  3%      -3.7        0.07 ± 50%  perf-profile.children.cycles-pp.xfs_btree_increment
      3.64 ±  8%      -3.5        0.13 ± 27%  perf-profile.children.cycles-pp.xfs_btree_get_rec
      2.61 ±  5%      -2.5        0.10 ± 18%  perf-profile.children.cycles-pp.__xfs_btree_check_sblock
      1.65 ±  7%      -1.5        0.18 ± 47%  perf-profile.children.cycles-pp.xfs_difree_finobt
      1.19 ±  6%      -1.2        0.03 ±102%  perf-profile.children.cycles-pp.xfs_verify_agino
      1.27 ±  5%      -0.8        0.43 ±  9%  perf-profile.children.cycles-pp.xfs_agino_range
      1.07 ±  6%      -0.8        0.26 ± 12%  perf-profile.children.cycles-pp.xfs_errortag_test
      1.21 ±  5%      -0.8        0.40 ± 37%  perf-profile.children.cycles-pp.xfs_iunlink_remove
      2.69 ±  5%      -0.6        2.07 ±  9%  perf-profile.children.cycles-pp.xfs_read_agi
      1.16 ±  8%      -0.5        0.68 ± 13%  perf-profile.children.cycles-pp.xfs_next_bit
      2.14 ±  8%      -0.4        1.70 ±  9%  perf-profile.children.cycles-pp.__sched_text_start
      1.47 ±  6%      -0.4        1.05 ±  8%  perf-profile.children.cycles-pp.xfs_buf_lock
      2.10 ±  7%      -0.4        1.70 ±  8%  perf-profile.children.cycles-pp.xfs_buf_item_release
      1.38 ±  6%      -0.4        0.99 ±  9%  perf-profile.children.cycles-pp.down
      1.35 ±  6%      -0.4        0.96 ±  9%  perf-profile.children.cycles-pp.__down
      1.98 ±  8%      -0.4        1.62 ± 10%  perf-profile.children.cycles-pp.xfs_buf_unlock
      1.22 ±  6%      -0.4        0.87 ±  9%  perf-profile.children.cycles-pp.schedule_timeout
      1.96 ±  6%      -0.3        1.62 ± 12%  perf-profile.children.cycles-pp.xfs_btree_lookup
      1.25 ±  7%      -0.3        0.92 ± 12%  perf-profile.children.cycles-pp.schedule
      1.55 ±  7%      -0.3        1.23 ± 13%  perf-profile.children.cycles-pp.try_to_wake_up
      0.76 ±  6%      -0.2        0.54 ± 17%  perf-profile.children.cycles-pp.ttwu_do_activate
      0.42 ±  8%      -0.2        0.20 ± 17%  perf-profile.children.cycles-pp.xfs_ag_block_count
      0.73 ±  7%      -0.2        0.52 ± 17%  perf-profile.children.cycles-pp.enqueue_task_fair
      0.29 ±  9%      -0.2        0.10 ± 17%  perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
      0.58 ±  9%      -0.2        0.40 ±  4%  perf-profile.children.cycles-pp.dequeue_task_fair
      0.57 ±  6%      -0.2        0.39 ± 18%  perf-profile.children.cycles-pp.enqueue_entity
      0.51 ±  8%      -0.2        0.35 ±  7%  perf-profile.children.cycles-pp.dequeue_entity
      0.46 ±  8%      -0.1        0.32 ±  8%  perf-profile.children.cycles-pp.update_load_avg
      0.24 ± 16%      -0.1        0.13 ± 16%  perf-profile.children.cycles-pp.xfs_lookup_get_search_key
      0.40 ±  7%      -0.1        0.30 ± 17%  perf-profile.children.cycles-pp.xfs_trans_log_buf
      0.33 ±  7%      -0.1        0.24 ± 17%  perf-profile.children.cycles-pp.xfs_trans_dirty_buf
      0.28 ±  8%      -0.1        0.20 ± 17%  perf-profile.children.cycles-pp.update_curr
      0.10 ± 23%      -0.1        0.03 ±103%  perf-profile.children.cycles-pp.reweight_entity
      0.19 ±  7%      -0.0        0.15 ±  8%  perf-profile.children.cycles-pp.xfs_inobt_init_cursor
      0.14 ± 10%      -0.0        0.11 ± 13%  perf-profile.children.cycles-pp.poll_idle
      0.06 ± 48%      +0.0        0.09 ± 21%  perf-profile.children.cycles-pp.xfs_trans_roll
      0.02 ±141%      +0.0        0.07 ± 19%  perf-profile.children.cycles-pp.__x64_sys_exit_group
      0.02 ±141%      +0.0        0.07 ± 19%  perf-profile.children.cycles-pp.do_group_exit
      0.02 ±141%      +0.0        0.07 ± 19%  perf-profile.children.cycles-pp.do_exit
      0.07 ± 52%      +0.0        0.11 ± 13%  perf-profile.children.cycles-pp.handle_mm_fault
      0.07 ± 51%      +0.1        0.13 ± 16%  perf-profile.children.cycles-pp.do_user_addr_fault
      0.07 ± 51%      +0.1        0.13 ± 16%  perf-profile.children.cycles-pp.exc_page_fault
      0.05 ± 76%      +0.1        0.11 ± 14%  perf-profile.children.cycles-pp.__handle_mm_fault
      0.00            +0.1        0.06 ± 21%  perf-profile.children.cycles-pp.rcu_needs_cpu
      0.24 ±  7%      +0.1        0.31 ±  8%  perf-profile.children.cycles-pp.exit_to_user_mode_prepare
      0.06 ± 51%      +0.1        0.12 ± 23%  perf-profile.children.cycles-pp.get_cpu_device
      0.08 ± 51%      +0.1        0.14 ± 11%  perf-profile.children.cycles-pp.asm_exc_page_fault
      0.05 ± 48%      +0.1        0.12 ± 23%  perf-profile.children.cycles-pp.bprm_execve
      0.00            +0.1        0.07 ± 15%  perf-profile.children.cycles-pp.switch_fpu_return
      0.14 ± 20%      +0.1        0.21 ± 17%  perf-profile.children.cycles-pp.kmem_alloc_large
      0.06 ± 13%      +0.1        0.14 ± 24%  perf-profile.children.cycles-pp.rb_next
      0.04 ± 72%      +0.1        0.12 ± 39%  perf-profile.children.cycles-pp.__hrtimer_get_next_event
      0.02 ±144%      +0.1        0.10 ± 22%  perf-profile.children.cycles-pp.exec_binprm
      0.19 ± 14%      +0.1        0.27 ± 14%  perf-profile.children.cycles-pp.nr_iowait_cpu
      0.02 ±142%      +0.1        0.10 ± 22%  perf-profile.children.cycles-pp.load_elf_binary
      0.14 ± 21%      +0.1        0.22 ±  6%  perf-profile.children.cycles-pp.tsc_verify_tsc_adjust
      0.01 ±223%      +0.1        0.09 ± 29%  perf-profile.children.cycles-pp.rb_insert_color
      0.00            +0.1        0.09 ± 35%  perf-profile.children.cycles-pp.xfs_ialloc_inode_init
      0.14 ± 21%      +0.1        0.23 ±  6%  perf-profile.children.cycles-pp.arch_cpu_idle_enter
      0.28 ±  9%      +0.1        0.38 ± 12%  perf-profile.children.cycles-pp.hrtimer_next_event_without
      0.00            +0.1        0.09 ± 30%  perf-profile.children.cycles-pp.xfs_trans_get_buf_map
      0.03 ± 99%      +0.1        0.12 ± 13%  perf-profile.children.cycles-pp.__alloc_pages_nodemask
      0.00            +0.1        0.10 ± 18%  perf-profile.children.cycles-pp.get_page_from_freelist
      0.18 ±  7%      +0.1        0.28 ± 17%  perf-profile.children.cycles-pp.__intel_pmu_enable_all
      0.17 ± 17%      +0.1        0.27 ± 19%  perf-profile.children.cycles-pp.update_ts_time_stats
      0.44 ±  7%      +0.1        0.55 ±  7%  perf-profile.children.cycles-pp.native_sched_clock
      0.24 ± 15%      +0.1        0.35 ± 16%  perf-profile.children.cycles-pp.kmem_alloc
      0.47 ±  8%      +0.1        0.58 ±  8%  perf-profile.children.cycles-pp.sched_clock
      0.00            +0.1        0.12 ± 20%  perf-profile.children.cycles-pp.xfs_fs_inode_init_once
      0.02 ±141%      +0.1        0.14 ± 16%  perf-profile.children.cycles-pp.xfs_btree_del_cursor
      0.17 ± 20%      +0.1        0.30 ± 21%  perf-profile.children.cycles-pp.xlog_write
      0.11 ± 18%      +0.1        0.24 ± 36%  perf-profile.children.cycles-pp.timerqueue_del
      0.00            +0.1        0.13 ± 17%  perf-profile.children.cycles-pp.xfs_iunlink_insert_backref
      0.03 ± 99%      +0.1        0.16 ± 28%  perf-profile.children.cycles-pp.xfs_ialloc_ag_alloc
      0.21 ± 18%      +0.1        0.35 ± 16%  perf-profile.children.cycles-pp.xlog_cil_push_work
      0.18 ±  9%      +0.1        0.32 ± 12%  perf-profile.children.cycles-pp.update_blocked_averages
      0.08 ± 22%      +0.1        0.23 ± 44%  perf-profile.children.cycles-pp.timerqueue_add
      0.10 ± 23%      +0.1        0.25 ± 45%  perf-profile.children.cycles-pp.enqueue_hrtimer
      0.16 ± 13%      +0.2        0.31 ± 20%  perf-profile.children.cycles-pp.__hrtimer_next_event_base
      0.19 ±  7%      +0.2        0.34 ± 13%  perf-profile.children.cycles-pp.run_rebalance_domains
      0.52 ±  8%      +0.2        0.68 ±  6%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.46 ±  9%      +0.2        0.64 ± 12%  perf-profile.children.cycles-pp.read_tsc
      0.26 ±  5%      +0.2        0.44 ± 18%  perf-profile.children.cycles-pp.xfs_trans_log_inode
      0.13 ± 20%      +0.2        0.31 ± 38%  perf-profile.children.cycles-pp.__remove_hrtimer
      0.49 ± 15%      +0.2        0.67 ± 16%  perf-profile.children.cycles-pp.irqtime_account_irq
      0.28 ± 16%      +0.2        0.48 ± 15%  perf-profile.children.cycles-pp.xfs_iget
      0.64 ± 19%      +0.2        0.84 ±  7%  perf-profile.children.cycles-pp.rebalance_domains
      0.04 ± 71%      +0.2        0.25 ± 13%  perf-profile.children.cycles-pp.allocate_slab
      0.12 ± 19%      +0.2        0.35 ± 10%  perf-profile.children.cycles-pp.__slab_alloc
      0.12 ± 21%      +0.2        0.34 ± 11%  perf-profile.children.cycles-pp.___slab_alloc
      0.00            +0.2        0.24 ± 16%  perf-profile.children.cycles-pp.xfs_inode_alloc
      0.25 ±  9%      +0.2        0.49 ± 17%  perf-profile.children.cycles-pp.xfs_imap_to_bp
      0.28 ±  4%      +0.3        0.57 ± 27%  perf-profile.children.cycles-pp.serial8250_console_putchar
      0.29 ±  3%      +0.3        0.59 ± 26%  perf-profile.children.cycles-pp.wait_for_xmitr
      0.29 ±  3%      +0.3        0.60 ± 26%  perf-profile.children.cycles-pp.uart_console_write
      0.30 ±  4%      +0.3        0.62 ± 26%  perf-profile.children.cycles-pp.serial8250_console_write
      0.45 ±  4%      +0.3        0.77 ± 17%  perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
      0.31 ±  3%      +0.3        0.65 ± 26%  perf-profile.children.cycles-pp.asm_sysvec_irq_work
      0.31 ±  3%      +0.3        0.65 ± 26%  perf-profile.children.cycles-pp.sysvec_irq_work
      0.31 ±  3%      +0.3        0.65 ± 26%  perf-profile.children.cycles-pp.__sysvec_irq_work
      0.31 ±  3%      +0.3        0.65 ± 26%  perf-profile.children.cycles-pp.irq_work_run
      0.31 ±  3%      +0.3        0.65 ± 26%  perf-profile.children.cycles-pp.irq_work_single
      0.31 ±  3%      +0.3        0.65 ± 26%  perf-profile.children.cycles-pp.printk
      0.31 ±  3%      +0.3        0.65 ± 26%  perf-profile.children.cycles-pp.vprintk_emit
      0.31 ±  3%      +0.3        0.65 ± 26%  perf-profile.children.cycles-pp.console_unlock
      0.33 ±  2%      +0.3        0.67 ± 24%  perf-profile.children.cycles-pp.irq_work_run_list
      0.52 ± 13%      +0.4        0.95 ± 13%  perf-profile.children.cycles-pp.xfs_init_new_inode
      0.79 ±  7%      +0.5        1.27 ±  8%  perf-profile.children.cycles-pp.scheduler_tick
      1.33 ± 11%      +0.5        1.81 ±  5%  perf-profile.children.cycles-pp.__softirqentry_text_start
      1.29 ± 13%      +0.5        1.77 ±  6%  perf-profile.children.cycles-pp.do_softirq_own_stack
      0.92 ±  5%      +0.5        1.40 ± 11%  perf-profile.children.cycles-pp.xfs_iunlink
      1.63 ± 13%      +0.6        2.25 ±  4%  perf-profile.children.cycles-pp.irq_exit_rcu
      0.19 ± 10%      +0.6        0.83 ± 16%  perf-profile.children.cycles-pp.xfs_dialloc_ag_update_inobt
      1.62 ±  9%      +0.6        2.27 ± 17%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      1.48 ±  7%      +1.0        2.49 ± 12%  perf-profile.children.cycles-pp.update_process_times
      1.52 ±  7%      +1.1        2.66 ± 15%  perf-profile.children.cycles-pp.tick_sched_handle
      1.82 ± 10%      +1.2        3.01 ± 12%  perf-profile.children.cycles-pp.tick_sched_timer
      7.96 ±  4%      +1.3        9.28 ±  8%  perf-profile.children.cycles-pp.vfs_unlink
      2.71 ±  6%      +2.0        4.73 ± 17%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.00            +2.0        2.04 ± 39%  perf-profile.children.cycles-pp.xfs_inactive_inode
      0.00            +2.1        2.05 ± 40%  perf-profile.children.cycles-pp.xfs_inactive_worker
      0.00            +2.1        2.05 ± 40%  perf-profile.children.cycles-pp.xfs_inode_walk
      0.00            +2.1        2.05 ± 40%  perf-profile.children.cycles-pp.xfs_inode_walk_ag
      0.58 ±  3%      +2.4        3.02 ± 27%  perf-profile.children.cycles-pp.process_one_work
      0.62 ±  2%      +2.4        3.06 ± 27%  perf-profile.children.cycles-pp.worker_thread
      0.77 ±  3%      +2.5        3.30 ± 26%  perf-profile.children.cycles-pp.kthread
      0.77 ±  3%      +2.5        3.31 ± 26%  perf-profile.children.cycles-pp.ret_from_fork
      5.20 ± 11%      +2.5        7.74 ±  6%  perf-profile.children.cycles-pp.hrtimer_interrupt
      5.30 ± 11%      +2.6        7.94 ±  7%  perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
      6.96 ± 11%      +3.5       10.43 ±  7%  perf-profile.children.cycles-pp.asm_call_sysvec_on_stack
      8.37 ± 13%      +4.0       12.37 ±  7%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
     10.06 ± 11%      +4.4       14.42 ±  5%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
     53.02 ±  2%      +9.0       62.01 ±  5%  perf-profile.children.cycles-pp.cpuidle_enter_state
     53.05 ±  2%      +9.0       62.03 ±  5%  perf-profile.children.cycles-pp.cpuidle_enter
     58.87 ±  2%     +10.4       69.31 ±  3%  perf-profile.children.cycles-pp.start_secondary
     59.47 ±  2%     +10.6       70.03 ±  3%  perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     59.47 ±  2%     +10.6       70.03 ±  3%  perf-profile.children.cycles-pp.cpu_startup_entry
     59.46 ±  2%     +10.6       70.02 ±  3%  perf-profile.children.cycles-pp.do_idle
      2.13 ±  6%      -2.1        0.06 ± 54%  perf-profile.self.cycles-pp.__xfs_btree_check_sblock
      1.03 ±  7%      -0.8        0.23 ±  8%  perf-profile.self.cycles-pp.xfs_errortag_test
      0.87 ± 10%      -0.6        0.27 ±  9%  perf-profile.self.cycles-pp.xfs_agino_range
      1.11 ±  7%      -0.5        0.65 ± 12%  perf-profile.self.cycles-pp.xfs_next_bit
      0.46 ± 11%      -0.3        0.15 ± 22%  perf-profile.self.cycles-pp.xfs_btree_lookup
      0.39 ±  8%      -0.2        0.19 ± 14%  perf-profile.self.cycles-pp.xfs_ag_block_count
      0.21 ±  8%      -0.1        0.06 ± 52%  perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
      0.32 ±  6%      -0.1        0.23 ± 17%  perf-profile.self.cycles-pp.xfs_trans_dirty_buf
      0.10 ± 22%      -0.1        0.03 ±103%  perf-profile.self.cycles-pp.reweight_entity
      0.05 ± 51%      +0.0        0.10 ± 13%  perf-profile.self.cycles-pp.update_blocked_averages
      0.05 ± 45%      +0.1        0.10 ± 23%  perf-profile.self.cycles-pp.irq_exit_rcu
      0.10 ± 18%      +0.1        0.16 ± 15%  perf-profile.self.cycles-pp.hrtimer_interrupt
      0.00            +0.1        0.06 ± 21%  perf-profile.self.cycles-pp.rcu_needs_cpu
      0.04 ± 72%      +0.1        0.11 ± 16%  perf-profile.self.cycles-pp.xlog_write
      0.19 ± 15%      +0.1        0.26 ± 12%  perf-profile.self.cycles-pp.update_sd_lb_stats
      0.05 ±  8%      +0.1        0.12 ± 27%  perf-profile.self.cycles-pp.rb_next
      0.00            +0.1        0.07 ± 15%  perf-profile.self.cycles-pp.switch_fpu_return
      0.03 ±102%      +0.1        0.10 ± 23%  perf-profile.self.cycles-pp.sched_clock_cpu
      0.05 ± 74%      +0.1        0.12 ± 21%  perf-profile.self.cycles-pp.get_cpu_device
      0.01 ±223%      +0.1        0.09 ± 33%  perf-profile.self.cycles-pp.rb_insert_color
      0.19 ± 14%      +0.1        0.26 ± 14%  perf-profile.self.cycles-pp.nr_iowait_cpu
      0.02 ±142%      +0.1        0.10 ± 29%  perf-profile.self.cycles-pp.timerqueue_del
      0.00            +0.1        0.09 ± 37%  perf-profile.self.cycles-pp.tick_irq_enter
      0.36 ±  6%      +0.1        0.45 ± 12%  perf-profile.self.cycles-pp.do_idle
      0.18 ±  7%      +0.1        0.28 ± 17%  perf-profile.self.cycles-pp.__intel_pmu_enable_all
      0.42 ±  6%      +0.1        0.53 ±  8%  perf-profile.self.cycles-pp.native_sched_clock
      0.04 ± 72%      +0.1        0.15 ± 51%  perf-profile.self.cycles-pp.timerqueue_add
      0.00            +0.1        0.12 ± 15%  perf-profile.self.cycles-pp.xfs_iunlink_insert_backref
      0.09 ± 28%      +0.1        0.22 ± 38%  perf-profile.self.cycles-pp.__hrtimer_run_queues
      0.13 ± 13%      +0.1        0.26 ± 25%  perf-profile.self.cycles-pp.__hrtimer_next_event_base
      0.45 ±  8%      +0.2        0.62 ± 13%  perf-profile.self.cycles-pp.read_tsc
      0.25 ± 28%      +0.2        0.44 ± 15%  perf-profile.self.cycles-pp.update_process_times


                                                                                
                            pmeter.Average_Active_Power                         
                                                                                
  270 +---------------------------------------------------------------------+   
  265 |-+                                   :+         :                    |   
      |                                     : +        ::                   |   
  260 |-+                                  :  :       : :                   |   
  255 |-+    .+.+      .+                  :   :.+.   :  :     +.       .+  |   
      |. +.++    +.+ .+  +. +. +.+.++. +.+.+   +   ++.+  +  +.+  + .+.++  +.|   
  250 |-+           +      +  +       +                   + :     +         |   
  245 |-+                                                  +                |   
  240 |-+                                     O                             |   
      |                                                                     |   
  235 |-+          O               O       O     O O  OO   O   O            |   
  230 |-O   O   O   O O  O OO OO O  O O  O  O                               |   
      |  O O  O  O      O              O       O    O    O  O    O          |   
  225 |-+                                                     O             |   
  220 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                            stress-ng.time.system_time                          
                                                                                
  100 +---------------------------------------------------------------------+   
      |.++.++.+.++.+ .+.+ .++.++.+.++.++.+.+ .++.+.+ .++.+.++. +.  .+. +.++.|   
   90 |-+           +    +                  +       +         +  ++   +     |   
      |                                                                     |   
   80 |-+                                                                   |   
      |                                                                     |   
   70 |-+                                                                   |   
      |                                                                     |   
   60 |-+                                                                   |   
      |                                                                     |   
   50 |-+                                                                   |   
      |                                                  O  O               |   
   40 |-O   O O OO    O     O         OO O    OO O  O O    O  OO O          |   
      |  O O       OO   OO O  OO O OO      OO      O   O                    |   
   30 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                    stress-ng.time.percent_of_cpu_this_job_got                  
                                                                                
  180 +---------------------------------------------------------------------+   
      |                                                                     |   
  160 |.++.++.+.++.+ .+.+ .+ .++.+.+ .++.+.+   +.+.  .++.+.+   +.  .+. +.+  |   
      |             +    +  +       +       +.+    ++       +.+  ++   +   +.|   
  140 |-+                                                                   |   
      |                                                                     |   
  120 |-+                                                                   |   
      |                                                                     |   
  100 |-+                                                                   |   
      |                                                                     |   
   80 |-+                                                   O               |   
      | O   O O OO          O         O  O     O O  O O  O O  OO O          |   
   60 |-+O O       OO O OO O  OO O OO  O   OO O    O   O                    |   
      |                                                                     |   
   40 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                       stress-ng.time.voluntary_context_switches                
                                                                                
    4e+06 +-----------------------------------------------------------------+   
          |.++.++.++.++.  .+ .++.++.++.++.++.++.++.++. +.++.       +.++.++.+|   
  3.5e+06 |-+           ++  +                         +     ++.++.+         |   
          |                                                                 |   
          |                                                                 |   
    3e+06 |-+                                                               |   
          |                                                                 |   
  2.5e+06 |-+                                                               |   
          |                                                                 |   
    2e+06 |-+                                                               |   
          |                                                                 |   
          |                                O              O OO OO O         |   
  1.5e+06 |-OO OO OO OO OO OO OO OO OO OO O  OO OO OO OO O                  |   
          |                                                                 |   
    1e+06 +-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                    stress-ng.time.involuntary_context_switches                 
                                                                                
  4000 +--------------------------------------------------------------------+   
       |                                                            +       |   
  3500 |.+  +           +.    .+  +   +   +                +        :+ +    |   
       |  :+ +.++. .+ .+  ++.+  :+ +.+ + + +.++. +  +.+.+ + :.+ .+.+  + + +.|   
       |  +       +  +          +       +       + + :    +  +  +         +  |   
  3000 |-+                                         +                        |   
       |                                                                    |   
  2500 |-+                                                                  |   
       |                                                                    |   
  2000 |-+                                                       O          |   
       |                                                 O  O               |   
       |       O           O         OO   OO  O  O    O    O  O             |   
  1500 |-O  OO  O O  O OO O  O O  O     O    O                 O            |   
       |  O         O           O  O            O  OO   O                   |   
  1000 +--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                 stress-ng.dentry.ops                           
                                                                                
  1.5e+06 +-----------------------------------------------------------------+   
          |                                                                 |   
  1.4e+06 |.++.++.++.++. +.++.++.++.++.++.++.++.++.++. +.++.++.++. +.++.++.+|   
  1.3e+06 |-+           +                             +           +         |   
          |                                                                 |   
  1.2e+06 |-+                                                               |   
  1.1e+06 |-+                                                               |   
          |                                                                 |   
    1e+06 |-+                                                               |   
   900000 |-+                                                O              |   
          |                                               O O  OO O         |   
   800000 |-+     O           O         O  O    O                           |   
   700000 |-OO OO  O OO OO OO  O O  OO O  O  OO  O OO OO O                  |   
          |                       O                                         |   
   600000 +-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                            stress-ng.dentry.ops_per_sec                        
                                                                                
  24000 +-------------------------------------------------------------------+   
        |.++.++.++.++.  .+. +.++.++.++.++.+.++.++.++. +.++.+.     .++. +.++.|   
  22000 |-+           ++   +                         +       ++.++    +     |   
        |                                                                   |   
  20000 |-+                                                                 |   
        |                                                                   |   
  18000 |-+                                                                 |   
        |                                                                   |   
  16000 |-+                                                                 |   
        |                                                    O   O          |   
  14000 |-+                                              O O  O O           |   
        |       O  O   O    O          O  O    O                            |   
  12000 |-OO OO  O  O O  O O  OO OO OO  O   OO  O OO OO O                   |   
        |                                                                   |   
  10000 +-------------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Oliver Sang


View attachment "config-5.11.0-rc7-00104-gf4da454e1584" of type "text/plain" (174007 bytes)

View attachment "job-script" of type "text/plain" (8408 bytes)

View attachment "job.yaml" of type "text/plain" (5745 bytes)

View attachment "reproduce" of type "text/plain" (553 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ