lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <20210301073255.GC12822@xsang-OptiPlex-9020>
Date:   Mon, 1 Mar 2021 15:32:55 +0800
From:   kernel test robot <oliver.sang@...el.com>
To:     Alexey Gladkov <gladkov.alexey@...il.com>
Cc:     0day robot <lkp@...el.com>,
        kernel test robot <oliver.sang@...el.com>,
        LKML <linux-kernel@...r.kernel.org>, lkp@...ts.01.org,
        ying.huang@...el.com, feng.tang@...el.com, zhengjun.xing@...el.com,
        io-uring@...r.kernel.org,
        Kernel Hardening <kernel-hardening@...ts.openwall.com>,
        Linux Containers <containers@...ts.linux-foundation.org>,
        linux-mm@...ck.org, Alexey Gladkov <legion@...nel.org>,
        Andrew Morton <akpm@...ux-foundation.org>,
        Christian Brauner <christian.brauner@...ntu.com>,
        "Eric W . Biederman" <ebiederm@...ssion.com>,
        Jann Horn <jannh@...gle.com>, Jens Axboe <axboe@...nel.dk>,
        Kees Cook <keescook@...omium.org>,
        Linus Torvalds <torvalds@...ux-foundation.org>,
        Oleg Nesterov <oleg@...hat.com>
Subject: e1e57d56fe:  stress-ng.access.ops_per_sec -41.6% regression


Greeting,

FYI, we noticed a -41.6% regression of stress-ng.access.ops_per_sec due to commit:


commit: e1e57d56fef0dd06daf3743f0948da86dfaf2f6f ("[PATCH v7 2/7] Add a reference to ucounts for each cred")
url: https://github.com/0day-ci/linux/commits/Alexey-Gladkov/Count-rlimits-in-each-user-namespace/20210222-175836
base: https://git.kernel.org/cgit/linux/kernel/git/shuah/linux-kselftest.git next

in testcase: stress-ng
on test machine: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 512G memory
with following parameters:

	nr_threads: 10%
	disk: 1HDD
	testtime: 60s
	fs: ext4
	class: filesystem
	test: access
	cpufreq_governor: performance
	ucode: 0x5003006


In addition to that, the commit also has significant impact on the following tests:

+------------------+----------------------------------------------------------------------+
| testcase: change | stress-ng: stress-ng.access.ops_per_sec -42.3% regression            |
| test machine     | 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 512G memory |
| test parameters  | class=filesystem                                                     |
|                  | cpufreq_governor=performance                                         |
|                  | disk=1HDD                                                            |
|                  | fs=xfs                                                               |
|                  | nr_threads=10%                                                       |
|                  | test=access                                                          |
|                  | testtime=60s                                                         |
|                  | ucode=0x5003006                                                      |
+------------------+----------------------------------------------------------------------+
| testcase: change | stress-ng: stress-ng.access.ops_per_sec -44.5% regression            |
| test machine     | 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 512G memory |
| test parameters  | class=filesystem                                                     |
|                  | cpufreq_governor=performance                                         |
|                  | disk=1HDD                                                            |
|                  | fs=f2fs                                                              |
|                  | nr_threads=10%                                                       |
|                  | test=access                                                          |
|                  | testtime=60s                                                         |
|                  | ucode=0x5003006                                                      |
+------------------+----------------------------------------------------------------------+


If you fix the issue, kindly add following tag
Reported-by: kernel test robot <oliver.sang@...el.com>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install                job.yaml  # job file is attached in this email
        bin/lkp split-job --compatible job.yaml
        bin/lkp run                    compatible-job.yaml

=========================================================================================
class/compiler/cpufreq_governor/disk/fs/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime/ucode:
  filesystem/gcc-9/performance/1HDD/ext4/x86_64-rhel-8.3/10%/debian-10.4-x86_64-20200603.cgz/lkp-csl-2sp7/access/stress-ng/60s/0x5003006

commit: 
  a88f997967 ("Increase size of ucounts to atomic_long_t")
  e1e57d56fe ("Add a reference to ucounts for each cred")

a88f9979677c6f55 e1e57d56fef0dd06daf3743f094 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
   2158451           -41.6%    1260149 ±  3%  stress-ng.access.ops
     35974           -41.6%      21002 ±  3%  stress-ng.access.ops_per_sec
     10.85 ± 10%     -46.4%       5.82        stress-ng.time.user_time
      0.21 ±  8%      -0.1        0.12        mpstat.cpu.all.usr%
      0.16 ±  4%     -18.5%       0.13 ± 12%  sched_debug.cfs_rq:/.nr_running.avg
    884086 ± 13%     -22.7%     683381 ±  8%  sched_debug.cpu.max_idle_balance_cost.max
     46438 ± 21%     -55.1%      20855 ± 32%  sched_debug.cpu.max_idle_balance_cost.stddev
     81265            +4.1%      84595 ±  2%  interrupts.CAL:Function_call_interrupts
     83.33 ± 34%   +4735.4%       4029 ± 96%  interrupts.CPU42.NMI:Non-maskable_interrupts
     83.33 ± 34%   +4735.4%       4029 ± 96%  interrupts.CPU42.PMI:Performance_monitoring_interrupts
     93.17 ± 26%   +4031.1%       3848 ± 96%  interrupts.CPU43.NMI:Non-maskable_interrupts
     93.17 ± 26%   +4031.1%       3848 ± 96%  interrupts.CPU43.PMI:Performance_monitoring_interrupts
     93.33 ± 27%   +2585.9%       2506 ±133%  interrupts.CPU44.NMI:Non-maskable_interrupts
     93.33 ± 27%   +2585.9%       2506 ±133%  interrupts.CPU44.PMI:Performance_monitoring_interrupts
     93.33 ± 26%   +2141.6%       2092 ±126%  interrupts.CPU45.NMI:Non-maskable_interrupts
     93.33 ± 26%   +2141.6%       2092 ±126%  interrupts.CPU45.PMI:Performance_monitoring_interrupts
    106.00 ± 23%   +2191.5%       2429 ±132%  interrupts.CPU46.NMI:Non-maskable_interrupts
    106.00 ± 23%   +2191.5%       2429 ±132%  interrupts.CPU46.PMI:Performance_monitoring_interrupts
    115.33 ± 23%   +1392.6%       1721 ±144%  interrupts.CPU47.NMI:Non-maskable_interrupts
    115.33 ± 23%   +1392.6%       1721 ±144%  interrupts.CPU47.PMI:Performance_monitoring_interrupts
      1585 ±133%    +201.4%       4778 ± 69%  interrupts.CPU48.NMI:Non-maskable_interrupts
      1585 ±133%    +201.4%       4778 ± 69%  interrupts.CPU48.PMI:Performance_monitoring_interrupts
    101.17 ± 18%     +53.7%     155.50 ±  6%  interrupts.CPU51.NMI:Non-maskable_interrupts
    101.17 ± 18%     +53.7%     155.50 ±  6%  interrupts.CPU51.PMI:Performance_monitoring_interrupts
    112.83 ±  6%    +883.3%       1109 ±193%  interrupts.CPU54.NMI:Non-maskable_interrupts
    112.83 ±  6%    +883.3%       1109 ±193%  interrupts.CPU54.PMI:Performance_monitoring_interrupts
    101.00 ± 22%   +1390.3%       1505 ±139%  interrupts.CPU79.NMI:Non-maskable_interrupts
    101.00 ± 22%   +1390.3%       1505 ±139%  interrupts.CPU79.PMI:Performance_monitoring_interrupts
    111.00 ± 25%    +344.6%     493.50 ±155%  interrupts.CPU80.NMI:Non-maskable_interrupts
    111.00 ± 25%    +344.6%     493.50 ±155%  interrupts.CPU80.PMI:Performance_monitoring_interrupts
      8.32 ±  7%     +54.7%      12.88        perf-stat.i.MPKI
 2.865e+09           -30.8%  1.984e+09 ±  2%  perf-stat.i.branch-instructions
      0.76 ±  2%      +0.1        0.84        perf-stat.i.branch-miss-rate%
  20772880           -21.8%   16252845        perf-stat.i.branch-misses
      1.85 ±  2%     +52.3%       2.81 ±  2%  perf-stat.i.cpi
 2.734e+10            +3.6%  2.832e+10        perf-stat.i.cpu-cycles
 4.122e+09           -33.7%  2.733e+09        perf-stat.i.dTLB-loads
 2.299e+09           -36.4%  1.463e+09 ±  2%  perf-stat.i.dTLB-stores
     86.57            -5.3       81.28        perf-stat.i.iTLB-load-miss-rate%
  15778062 ±  3%     -39.1%    9607502 ±  3%  perf-stat.i.iTLB-load-misses
 1.479e+10           -32.4%  9.991e+09 ±  2%  perf-stat.i.instructions
    977.00 ±  2%      +9.0%       1064 ±  3%  perf-stat.i.instructions-per-iTLB-miss
      0.55 ±  2%     -33.4%       0.36 ±  2%  perf-stat.i.ipc
      0.28            +3.6%       0.29        perf-stat.i.metric.GHz
      0.41 ± 42%     -28.7%       0.29 ±  2%  perf-stat.i.metric.K/sec
     98.22           -32.9%      65.92 ±  2%  perf-stat.i.metric.M/sec
     88.10 ±  2%      +7.9       96.03        perf-stat.i.node-load-miss-rate%
   3379054 ± 14%     +82.1%    6154210 ±  7%  perf-stat.i.node-load-misses
    441950 ± 29%     -54.3%     202107 ± 15%  perf-stat.i.node-loads
  12643063 ±  9%     -23.6%    9657658 ±  7%  perf-stat.i.node-store-misses
      8.36 ±  7%     +55.9%      13.04        perf-stat.overall.MPKI
      0.73            +0.1        0.82        perf-stat.overall.branch-miss-rate%
      1.85 ±  2%     +53.3%       2.84 ±  2%  perf-stat.overall.cpi
     87.69            -5.6       82.07        perf-stat.overall.iTLB-load-miss-rate%
    937.91 ±  2%     +11.0%       1041 ±  3%  perf-stat.overall.instructions-per-iTLB-miss
      0.54 ±  2%     -34.8%       0.35 ±  2%  perf-stat.overall.ipc
     88.63 ±  2%      +8.2       96.83        perf-stat.overall.node-load-miss-rate%
 2.819e+09           -30.8%  1.952e+09 ±  2%  perf-stat.ps.branch-instructions
  20439060           -21.8%   15990109        perf-stat.ps.branch-misses
  2.69e+10            +3.6%  2.786e+10        perf-stat.ps.cpu-cycles
 4.057e+09           -33.7%  2.689e+09        perf-stat.ps.dTLB-loads
 2.263e+09           -36.4%  1.439e+09 ±  2%  perf-stat.ps.dTLB-stores
  15526322 ±  3%     -39.1%    9453069 ±  3%  perf-stat.ps.iTLB-load-misses
 1.455e+10           -32.4%   9.83e+09 ±  2%  perf-stat.ps.instructions
   3325252 ± 14%     +82.1%    6055830 ±  7%  perf-stat.ps.node-load-misses
    434905 ± 29%     -54.3%     198876 ± 15%  perf-stat.ps.node-loads
  12441875 ±  9%     -23.6%    9503199 ±  7%  perf-stat.ps.node-store-misses
 9.193e+11           -32.7%  6.187e+11 ±  2%  perf-stat.total.instructions
      6.30 ± 10%      -5.6        0.74 ± 12%  perf-profile.calltrace.cycles-pp.free_uid.put_cred_rcu.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      6.25 ± 10%      -5.5        0.71 ± 13%  perf-profile.calltrace.cycles-pp.refcount_dec_and_lock_irqsave.free_uid.put_cred_rcu.do_faccessat.do_syscall_64
      6.23 ± 10%      -5.5        0.71 ± 13%  perf-profile.calltrace.cycles-pp.refcount_dec_not_one.refcount_dec_and_lock_irqsave.free_uid.put_cred_rcu.do_faccessat
      5.64 ± 12%      -4.1        1.52 ± 11%  perf-profile.calltrace.cycles-pp.key_put.put_cred_rcu.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      6.59 ± 19%      -3.7        2.89 ± 12%  perf-profile.calltrace.cycles-pp.__x64_sys_fchmod.do_syscall_64.entry_SYSCALL_64_after_hwframe
      6.50 ± 19%      -3.7        2.84 ± 12%  perf-profile.calltrace.cycles-pp.chmod_common.__x64_sys_fchmod.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.83 ± 20%      -3.3        2.50 ± 13%  perf-profile.calltrace.cycles-pp.notify_change.chmod_common.__x64_sys_fchmod.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.63 ± 21%      -3.2        2.38 ± 13%  perf-profile.calltrace.cycles-pp.ext4_setattr.notify_change.chmod_common.__x64_sys_fchmod.do_syscall_64
      5.39 ± 19%      -3.1        2.29 ± 13%  perf-profile.calltrace.cycles-pp.__mark_inode_dirty.ext4_setattr.notify_change.chmod_common.__x64_sys_fchmod
      4.48 ± 21%      -2.6        1.87 ± 13%  perf-profile.calltrace.cycles-pp.ext4_dirty_inode.__mark_inode_dirty.ext4_setattr.notify_change.chmod_common
      4.53 ± 18%      -2.4        2.09 ± 10%  perf-profile.calltrace.cycles-pp.security_prepare_creds.prepare_creds.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.98 ± 22%      -1.8        1.22 ±  9%  perf-profile.calltrace.cycles-pp.apparmor_cred_prepare.security_prepare_creds.prepare_creds.do_faccessat.do_syscall_64
      2.65 ± 17%      -1.6        1.05 ± 14%  perf-profile.calltrace.cycles-pp.__ext4_journal_start_sb.ext4_dirty_inode.__mark_inode_dirty.ext4_setattr.notify_change
      3.02 ± 12%      -1.5        1.50 ±  8%  perf-profile.calltrace.cycles-pp.security_cred_free.put_cred_rcu.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.19 ± 14%      -1.3        0.85 ± 13%  perf-profile.calltrace.cycles-pp.jbd2__journal_start.__ext4_journal_start_sb.ext4_dirty_inode.__mark_inode_dirty.ext4_setattr
      2.03 ± 15%      -1.3        0.76 ± 13%  perf-profile.calltrace.cycles-pp.start_this_handle.jbd2__journal_start.__ext4_journal_start_sb.ext4_dirty_inode.__mark_inode_dirty
      2.77 ± 11%      -1.2        1.57 ±  8%  perf-profile.calltrace.cycles-pp.filename_lookup.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.38 ± 11%      -1.1        1.29 ±  7%  perf-profile.calltrace.cycles-pp.__do_sys_newfstatat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.44 ± 11%      -1.1        1.39 ±  8%  perf-profile.calltrace.cycles-pp.path_lookupat.filename_lookup.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.01 ± 14%      -1.0        0.96 ±  6%  perf-profile.calltrace.cycles-pp.apparmor_cred_free.security_cred_free.put_cred_rcu.do_faccessat.do_syscall_64
      1.78 ± 31%      -1.0        0.78 ± 12%  perf-profile.calltrace.cycles-pp.__ext4_mark_inode_dirty.ext4_dirty_inode.__mark_inode_dirty.ext4_setattr.notify_change
      2.08 ± 12%      -1.0        1.12 ±  7%  perf-profile.calltrace.cycles-pp.vfs_statx.__do_sys_newfstatat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.55 ± 11%      -0.7        0.88 ± 11%  perf-profile.calltrace.cycles-pp.user_path_at_empty.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.52 ± 11%      -0.7        0.87 ± 11%  perf-profile.calltrace.cycles-pp.getname_flags.user_path_at_empty.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.94 ±  9%      -0.7        0.29 ±100%  perf-profile.calltrace.cycles-pp.kfree.security_cred_free.put_cred_rcu.do_faccessat.do_syscall_64
      1.43 ± 10%      -0.6        0.80 ± 11%  perf-profile.calltrace.cycles-pp.kmem_cache_alloc.prepare_creds.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.44 ±  9%      -0.6        0.81 ± 13%  perf-profile.calltrace.cycles-pp.__kmalloc.security_prepare_creds.prepare_creds.do_faccessat.do_syscall_64
      0.88 ±  9%      -0.6        0.27 ±100%  perf-profile.calltrace.cycles-pp.kmem_cache_free.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.33 ± 10%      -0.6        0.74 ± 10%  perf-profile.calltrace.cycles-pp.link_path_walk.path_lookupat.filename_lookup.do_faccessat.do_syscall_64
      1.20 ± 11%      -0.5        0.66 ± 12%  perf-profile.calltrace.cycles-pp.strncpy_from_user.getname_flags.user_path_at_empty.do_faccessat.do_syscall_64
      1.33 ± 27%      +1.3        2.65 ± 12%  perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      0.00            +1.4        1.38 ± 10%  perf-profile.calltrace.cycles-pp.override_creds.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
     20.13 ± 10%      +3.3       23.38 ±  9%  perf-profile.calltrace.cycles-pp.put_cred_rcu.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
     19.93 ± 11%      +3.7       23.61 ±  9%  perf-profile.calltrace.cycles-pp.prepare_creds.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00           +14.5       14.53 ±  8%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.get_ucounts.prepare_creds.do_faccessat
      0.00           +15.0       15.01 ±  8%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.put_ucounts.put_cred_rcu.do_faccessat
      0.00           +16.1       16.09 ±  8%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.get_ucounts.prepare_creds.do_faccessat.do_syscall_64
      0.00           +16.4       16.42 ±  8%  perf-profile.calltrace.cycles-pp.get_ucounts.prepare_creds.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00           +16.5       16.55 ±  8%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.put_ucounts.put_cred_rcu.do_faccessat.do_syscall_64
      0.00           +16.9       16.88 ±  8%  perf-profile.calltrace.cycles-pp.put_ucounts.put_cred_rcu.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      6.30 ± 10%      -5.6        0.74 ± 12%  perf-profile.children.cycles-pp.free_uid
      6.26 ± 10%      -5.5        0.72 ± 13%  perf-profile.children.cycles-pp.refcount_dec_and_lock_irqsave
      6.23 ± 10%      -5.5        0.71 ± 13%  perf-profile.children.cycles-pp.refcount_dec_not_one
      5.66 ± 12%      -4.1        1.54 ± 11%  perf-profile.children.cycles-pp.key_put
      6.59 ± 19%      -3.7        2.89 ± 12%  perf-profile.children.cycles-pp.__x64_sys_fchmod
      6.50 ± 19%      -3.7        2.84 ± 12%  perf-profile.children.cycles-pp.chmod_common
      5.83 ± 20%      -3.3        2.50 ± 12%  perf-profile.children.cycles-pp.notify_change
      5.64 ± 21%      -3.3        2.38 ± 13%  perf-profile.children.cycles-pp.ext4_setattr
      5.39 ± 19%      -3.1        2.29 ± 13%  perf-profile.children.cycles-pp.__mark_inode_dirty
      4.48 ± 21%      -2.6        1.87 ± 13%  perf-profile.children.cycles-pp.ext4_dirty_inode
      4.54 ± 18%      -2.4        2.09 ± 10%  perf-profile.children.cycles-pp.security_prepare_creds
      2.98 ± 22%      -1.8        1.22 ±  9%  perf-profile.children.cycles-pp.apparmor_cred_prepare
      2.65 ± 17%      -1.6        1.05 ± 14%  perf-profile.children.cycles-pp.__ext4_journal_start_sb
      3.64 ± 11%      -1.6        2.05 ±  8%  perf-profile.children.cycles-pp.filename_lookup
      3.02 ± 12%      -1.5        1.50 ±  8%  perf-profile.children.cycles-pp.security_cred_free
      3.24 ± 11%      -1.4        1.83 ±  8%  perf-profile.children.cycles-pp.path_lookupat
      2.19 ± 14%      -1.3        0.85 ± 13%  perf-profile.children.cycles-pp.jbd2__journal_start
      2.04 ± 15%      -1.3        0.77 ± 14%  perf-profile.children.cycles-pp.start_this_handle
      2.38 ± 11%      -1.1        1.29 ±  6%  perf-profile.children.cycles-pp.__do_sys_newfstatat
      2.04 ± 14%      -1.1        0.96 ±  6%  perf-profile.children.cycles-pp.apparmor_cred_free
      1.79 ± 31%      -1.0        0.78 ± 13%  perf-profile.children.cycles-pp.__ext4_mark_inode_dirty
      2.08 ± 12%      -1.0        1.12 ±  8%  perf-profile.children.cycles-pp.vfs_statx
      2.02 ± 12%      -0.9        1.14 ± 10%  perf-profile.children.cycles-pp.user_path_at_empty
      1.98 ± 12%      -0.9        1.12 ± 10%  perf-profile.children.cycles-pp.getname_flags
      1.89 ± 10%      -0.8        1.08 ± 11%  perf-profile.children.cycles-pp.kmem_cache_alloc
      1.79 ± 10%      -0.8        1.00 ± 10%  perf-profile.children.cycles-pp.link_path_walk
      1.53 ± 11%      -0.7        0.85 ± 11%  perf-profile.children.cycles-pp.strncpy_from_user
      1.49 ±  9%      -0.7        0.83 ± 12%  perf-profile.children.cycles-pp.__kmalloc
      1.05 ± 26%      -0.6        0.48 ± 13%  perf-profile.children.cycles-pp.ext4_mark_iloc_dirty
      1.19 ± 10%      -0.5        0.67 ±  8%  perf-profile.children.cycles-pp.kmem_cache_free
      0.90 ± 23%      -0.5        0.42 ± 12%  perf-profile.children.cycles-pp.ext4_do_update_inode
      0.86 ± 13%      -0.5        0.40 ± 14%  perf-profile.children.cycles-pp.__ext4_journal_stop
      0.76 ± 14%      -0.4        0.34 ± 15%  perf-profile.children.cycles-pp.jbd2_journal_stop
      0.95 ±  9%      -0.4        0.54 ± 12%  perf-profile.children.cycles-pp.kfree
      0.84 ± 12%      -0.4        0.43 ± 12%  perf-profile.children.cycles-pp.common_perm_cond
      0.68 ± 37%      -0.4        0.29 ± 13%  perf-profile.children.cycles-pp.ext4_reserve_inode_write
      0.74 ± 15%      -0.4        0.35 ± 11%  perf-profile.children.cycles-pp._raw_read_lock
      0.79 ± 12%      -0.4        0.40 ± 12%  perf-profile.children.cycles-pp.common_perm
      0.77 ± 12%      -0.4        0.39 ± 11%  perf-profile.children.cycles-pp.inode_permission
      0.83 ± 12%      -0.4        0.48 ± 10%  perf-profile.children.cycles-pp.walk_component
      0.45 ± 15%      -0.3        0.13 ± 20%  perf-profile.children.cycles-pp.add_transaction_credits
      0.58 ± 12%      -0.3        0.27 ±  8%  perf-profile.children.cycles-pp.generic_permission
      0.67 ± 11%      -0.3        0.37 ± 12%  perf-profile.children.cycles-pp.__check_object_size
      0.59 ± 13%      -0.3        0.34 ± 10%  perf-profile.children.cycles-pp.complete_walk
      0.56 ±  9%      -0.2        0.31 ± 11%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.50 ± 10%      -0.2        0.26 ±  9%  perf-profile.children.cycles-pp.vfs_getattr
      0.47 ± 12%      -0.2        0.23 ± 15%  perf-profile.children.cycles-pp.stop_this_handle
      0.56 ± 13%      -0.2        0.32 ± 10%  perf-profile.children.cycles-pp.unlazy_walk
      0.49 ± 11%      -0.2        0.25 ±  9%  perf-profile.children.cycles-pp.security_inode_getattr
      0.42 ± 28%      -0.2        0.19 ± 18%  perf-profile.children.cycles-pp.ext4_journal_check_start
      0.43 ± 12%      -0.2        0.22 ± 14%  perf-profile.children.cycles-pp.security_path_chmod
      0.45 ± 10%      -0.2        0.24 ± 13%  perf-profile.children.cycles-pp.obj_cgroup_charge
      0.38 ± 36%      -0.2        0.17 ± 10%  perf-profile.children.cycles-pp.ext4_get_inode_loc
      0.44 ±  6%      -0.2        0.24 ± 16%  perf-profile.children.cycles-pp.refill_obj_stock
      0.42 ± 10%      -0.2        0.22 ±  9%  perf-profile.children.cycles-pp.get_obj_cgroup_from_current
      0.37 ± 36%      -0.2        0.17 ± 12%  perf-profile.children.cycles-pp.__ext4_get_inode_loc
      0.46 ± 10%      -0.2        0.27 ± 10%  perf-profile.children.cycles-pp.lookup_fast
      0.44 ± 13%      -0.2        0.25 ± 13%  perf-profile.children.cycles-pp.__legitimize_path
      0.41 ± 11%      -0.2        0.24 ± 17%  perf-profile.children.cycles-pp.___might_sleep
      0.41 ±  9%      -0.2        0.23 ±  6%  perf-profile.children.cycles-pp.__mod_memcg_lruvec_state
      0.26 ± 57%      -0.2        0.09 ±  9%  perf-profile.children.cycles-pp.ext4_inode_csum_set
      0.27 ± 17%      -0.2        0.10 ±  6%  perf-profile.children.cycles-pp.capable_wrt_inode_uidgid
      0.32 ± 36%      -0.2        0.16 ± 12%  perf-profile.children.cycles-pp.revert_creds
      0.27 ± 33%      -0.2        0.10 ± 14%  perf-profile.children.cycles-pp.__ext4_journal_get_write_access
      0.43 ± 10%      -0.2        0.27 ± 12%  perf-profile.children.cycles-pp.crc32c_pcl_intel_update
      0.26 ± 17%      -0.2        0.10 ±  6%  perf-profile.children.cycles-pp.ns_capable_common
      0.26 ± 16%      -0.2        0.10 ±  6%  perf-profile.children.cycles-pp.security_capable
      0.23 ± 18%      -0.2        0.08 ±  8%  perf-profile.children.cycles-pp.apparmor_capable
      0.36 ± 11%      -0.2        0.21 ±  9%  perf-profile.children.cycles-pp.path_put
      0.32 ± 14%      -0.1        0.17 ±  9%  perf-profile.children.cycles-pp.__check_heap_object
      0.35 ± 12%      -0.1        0.21 ±  9%  perf-profile.children.cycles-pp.dput
      0.34 ± 11%      -0.1        0.19 ±  9%  perf-profile.children.cycles-pp.__entry_text_start
      0.32 ± 10%      -0.1        0.18 ± 12%  perf-profile.children.cycles-pp.__d_lookup_rcu
      0.19 ± 14%      -0.1        0.07 ± 11%  perf-profile.children.cycles-pp.__ext4_handle_dirty_metadata
      0.30 ± 10%      -0.1        0.18 ±  7%  perf-profile.children.cycles-pp.path_init
      0.28 ± 13%      -0.1        0.16 ±  6%  perf-profile.children.cycles-pp.cp_new_stat
      0.27 ± 11%      -0.1        0.16 ± 15%  perf-profile.children.cycles-pp.__might_sleep
      0.19 ± 34%      -0.1        0.08 ± 10%  perf-profile.children.cycles-pp.ext4_inode_csum
      0.26 ± 27%      -0.1        0.16 ± 17%  perf-profile.children.cycles-pp.map_id_up
      0.22 ± 14%      -0.1        0.12 ± 14%  perf-profile.children.cycles-pp._cond_resched
      0.15 ± 17%      -0.1        0.05 ± 45%  perf-profile.children.cycles-pp.jbd2_journal_get_write_access
      0.22 ± 15%      -0.1        0.13 ± 15%  perf-profile.children.cycles-pp.__legitimize_mnt
      0.22 ± 11%      -0.1        0.13 ± 14%  perf-profile.children.cycles-pp.__wake_up_common_lock
      0.19 ± 13%      -0.1        0.10 ± 13%  perf-profile.children.cycles-pp.__mod_memcg_state
      0.17 ± 12%      -0.1        0.09 ± 13%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
      0.18 ± 11%      -0.1        0.10 ± 15%  perf-profile.children.cycles-pp.step_into
      0.17 ± 13%      -0.1        0.09 ± 10%  perf-profile.children.cycles-pp.rcu_read_unlock_strict
      0.17 ± 11%      -0.1        0.10 ± 16%  perf-profile.children.cycles-pp.lockref_get_not_dead
      0.17 ± 13%      -0.1        0.10 ± 17%  perf-profile.children.cycles-pp.memset_erms
      0.15 ± 15%      -0.1        0.08 ± 13%  perf-profile.children.cycles-pp._copy_to_user
      0.16 ± 12%      -0.1        0.10 ± 15%  perf-profile.children.cycles-pp.__might_fault
      0.10 ± 16%      -0.1        0.05 ± 47%  perf-profile.children.cycles-pp.__put_cred
      0.13 ± 17%      -0.1        0.08 ±  9%  perf-profile.children.cycles-pp.lockref_put_return
      0.08 ± 13%      -0.1        0.03 ± 99%  perf-profile.children.cycles-pp.__find_get_block
      0.12 ± 14%      -0.1        0.06 ± 14%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      0.11 ± 16%      -0.1        0.06 ± 11%  perf-profile.children.cycles-pp.rcu_all_qs
      0.28 ±  7%      -0.1        0.23 ±  5%  perf-profile.children.cycles-pp.scheduler_tick
      0.09 ± 19%      -0.0        0.05 ± 45%  perf-profile.children.cycles-pp.security_inode_permission
      0.07 ±  9%      -0.0        0.03 ± 99%  perf-profile.children.cycles-pp.terminate_walk
      0.11 ± 12%      -0.0        0.06 ± 11%  perf-profile.children.cycles-pp.__virt_addr_valid
      0.11 ±  9%      -0.0        0.07 ± 18%  perf-profile.children.cycles-pp.__getblk_gfp
      0.10 ± 13%      -0.0        0.06 ± 13%  perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
      0.11 ±  9%      -0.0        0.08 ±  9%  perf-profile.children.cycles-pp.calc_global_load_tick
      0.09 ± 12%      -0.0        0.06 ± 13%  perf-profile.children.cycles-pp.__fget_light
      0.08 ±  8%      +0.3        0.39 ±  9%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.14 ± 12%      +1.2        1.39 ± 10%  perf-profile.children.cycles-pp.override_creds
      1.34 ± 27%      +1.3        2.66 ± 12%  perf-profile.children.cycles-pp.menu_select
     20.16 ± 10%      +3.2       23.40 ±  9%  perf-profile.children.cycles-pp.put_cred_rcu
     19.94 ± 11%      +3.7       23.62 ±  9%  perf-profile.children.cycles-pp.prepare_creds
      0.00           +16.4       16.43 ±  8%  perf-profile.children.cycles-pp.get_ucounts
      0.00           +16.9       16.88 ±  8%  perf-profile.children.cycles-pp.put_ucounts
      0.00           +29.5       29.55 ±  8%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      0.22 ±  8%     +32.6       32.80 ±  8%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
     13.87 ± 11%      -9.6        4.27 ± 13%  perf-profile.self.cycles-pp.prepare_creds
      6.20 ± 10%      -5.5        0.70 ± 13%  perf-profile.self.cycles-pp.refcount_dec_not_one
      5.62 ± 12%      -4.1        1.52 ± 11%  perf-profile.self.cycles-pp.key_put
      5.18 ±  9%      -2.4        2.73 ± 10%  perf-profile.self.cycles-pp.put_cred_rcu
      2.96 ± 22%      -1.7        1.22 ± 10%  perf-profile.self.cycles-pp.apparmor_cred_prepare
      2.02 ± 15%      -1.1        0.96 ±  6%  perf-profile.self.cycles-pp.apparmor_cred_free
      0.85 ± 16%      -0.6        0.28 ± 18%  perf-profile.self.cycles-pp.start_this_handle
      0.96 ±  9%      -0.4        0.51 ±  9%  perf-profile.self.cycles-pp.link_path_walk
      0.96 ± 10%      -0.4        0.57 ± 12%  perf-profile.self.cycles-pp.kmem_cache_alloc
      0.78 ± 12%      -0.4        0.40 ± 12%  perf-profile.self.cycles-pp.common_perm
      0.73 ± 15%      -0.4        0.35 ± 11%  perf-profile.self.cycles-pp._raw_read_lock
      0.83 ± 10%      -0.4        0.48 ±  8%  perf-profile.self.cycles-pp.kmem_cache_free
      0.45 ± 15%      -0.3        0.13 ± 18%  perf-profile.self.cycles-pp.add_transaction_credits
      0.71 ± 11%      -0.3        0.39 ±  8%  perf-profile.self.cycles-pp.strncpy_from_user
      0.73 ± 11%      -0.3        0.43 ± 14%  perf-profile.self.cycles-pp.__kmalloc
      0.62 ± 11%      -0.3        0.36 ± 14%  perf-profile.self.cycles-pp.kfree
      0.56 ±  9%      -0.2        0.31 ± 11%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.38 ± 30%      -0.2        0.17 ± 19%  perf-profile.self.cycles-pp.ext4_journal_check_start
      0.44 ± 11%      -0.2        0.23 ± 14%  perf-profile.self.cycles-pp.obj_cgroup_charge
      0.43 ±  6%      -0.2        0.23 ± 14%  perf-profile.self.cycles-pp.refill_obj_stock
      0.41 ± 11%      -0.2        0.23 ± 17%  perf-profile.self.cycles-pp.___might_sleep
      0.36 ±  8%      -0.2        0.19 ±  8%  perf-profile.self.cycles-pp.get_obj_cgroup_from_current
      0.32 ± 36%      -0.2        0.15 ± 11%  perf-profile.self.cycles-pp.revert_creds
      0.24 ± 18%      -0.2        0.07 ± 18%  perf-profile.self.cycles-pp.jbd2_journal_stop
      0.42 ± 11%      -0.2        0.26 ± 12%  perf-profile.self.cycles-pp.crc32c_pcl_intel_update
      0.39 ± 13%      -0.2        0.23 ± 12%  perf-profile.self.cycles-pp.do_faccessat
      0.23 ± 18%      -0.2        0.08 ±  8%  perf-profile.self.cycles-pp.apparmor_capable
      0.25 ± 16%      -0.2        0.10 ± 23%  perf-profile.self.cycles-pp.stop_this_handle
      0.31 ± 14%      -0.1        0.16 ± 10%  perf-profile.self.cycles-pp.__check_heap_object
      0.34 ± 11%      -0.1        0.19 ± 10%  perf-profile.self.cycles-pp.__entry_text_start
      0.31 ±  9%      -0.1        0.18 ± 12%  perf-profile.self.cycles-pp.__d_lookup_rcu
      0.28 ±  9%      -0.1        0.15 ± 11%  perf-profile.self.cycles-pp.generic_permission
      0.26 ± 11%      -0.1        0.15 ±  9%  perf-profile.self.cycles-pp.path_init
      0.24 ± 13%      -0.1        0.13 ± 16%  perf-profile.self.cycles-pp.__check_object_size
      0.22 ± 21%      -0.1        0.12 ± 19%  perf-profile.self.cycles-pp.ext4_do_update_inode
      0.19 ± 39%      -0.1        0.09 ± 13%  perf-profile.self.cycles-pp.__ext4_get_inode_loc
      0.26 ± 29%      -0.1        0.16 ± 18%  perf-profile.self.cycles-pp.map_id_up
      0.25 ± 11%      -0.1        0.15 ± 14%  perf-profile.self.cycles-pp.__might_sleep
      0.22 ± 14%      -0.1        0.13 ± 15%  perf-profile.self.cycles-pp.__legitimize_mnt
      0.20 ± 15%      -0.1        0.12 ± 18%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.19 ± 13%      -0.1        0.10 ± 13%  perf-profile.self.cycles-pp.__mod_memcg_state
      0.19 ± 16%      -0.1        0.11 ± 12%  perf-profile.self.cycles-pp.walk_component
      0.20 ±  7%      -0.1        0.12 ± 14%  perf-profile.self.cycles-pp.__mod_memcg_lruvec_state
      0.21 ± 13%      -0.1        0.13 ± 23%  perf-profile.self.cycles-pp.inode_permission
      0.17 ± 11%      -0.1        0.10 ± 16%  perf-profile.self.cycles-pp.step_into
      0.17 ± 11%      -0.1        0.10 ± 16%  perf-profile.self.cycles-pp.lockref_get_not_dead
      0.14 ± 17%      -0.1        0.08 ± 14%  perf-profile.self.cycles-pp.lookup_fast
      0.16 ± 14%      -0.1        0.09 ± 19%  perf-profile.self.cycles-pp.memset_erms
      0.11 ± 12%      -0.1        0.05 ± 70%  perf-profile.self.cycles-pp._cond_resched
      0.10 ± 19%      -0.1        0.04 ± 72%  perf-profile.self.cycles-pp.__put_cred
      0.13 ± 16%      -0.1        0.07 ±  6%  perf-profile.self.cycles-pp.getname_flags
      0.13 ± 19%      -0.1        0.08 ± 22%  perf-profile.self.cycles-pp.filename_lookup
      0.12 ± 12%      -0.1        0.07 ± 14%  perf-profile.self.cycles-pp.rcu_read_unlock_strict
      0.12 ± 14%      -0.1        0.06 ± 14%  perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      0.12 ± 18%      -0.1        0.07 ±  5%  perf-profile.self.cycles-pp.lockref_put_return
      0.10 ± 10%      -0.0        0.05 ± 46%  perf-profile.self.cycles-pp.__virt_addr_valid
      0.09 ± 19%      -0.0        0.05 ± 45%  perf-profile.self.cycles-pp.security_inode_permission
      0.08 ± 13%      -0.0        0.04 ± 73%  perf-profile.self.cycles-pp.__fget_light
      0.10 ± 18%      -0.0        0.06 ± 18%  perf-profile.self.cycles-pp.notify_change
      0.10 ± 13%      -0.0        0.06 ± 13%  perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack
      0.10 ± 18%      -0.0        0.06 ± 11%  perf-profile.self.cycles-pp.path_lookupat
      0.11 ±  9%      -0.0        0.08 ±  9%  perf-profile.self.cycles-pp.calc_global_load_tick
      0.07 ± 10%      +0.2        0.23 ± 11%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.00            +0.2        0.18 ± 16%  perf-profile.self.cycles-pp.get_ucounts
      0.00            +0.2        0.18 ± 16%  perf-profile.self.cycles-pp.put_ucounts
      0.65 ± 23%      +0.8        1.48 ± 14%  perf-profile.self.cycles-pp.cpuidle_enter_state
      0.13 ± 11%      +1.2        1.38 ± 10%  perf-profile.self.cycles-pp.override_creds
      0.79 ± 43%      +1.3        2.12 ± 16%  perf-profile.self.cycles-pp.menu_select
      0.22 ±  7%      +3.0        3.25 ± 10%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.00           +29.5       29.55 ±  8%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath


                                                                                
                                 stress-ng.access.ops                           
                                                                                
  2.6e+06 +-----------------------------------------------------------------+   
          |           +.                                                    |   
  2.4e+06 |-+         : ++.         +. +.++.+ .+                            |   
          |        +  :    ++.+ .+.+  +      +  +.++. .++. +.               |   
  2.2e+06 |.++.++. :+:         +                     +    +  ++. +.++.++.++.|   
    2e+06 |-+     +  +                                          +           |   
          |                                                                 |   
  1.8e+06 |-+                                                               |   
          |                                                                 |   
  1.6e+06 |-+                                                               |   
  1.4e+06 |-+                                                               |   
          |    O   O             O        O            OO                   |   
  1.2e+06 |-OO  O O  OO OO OO OO   OO OO O  OO OO OO O    O                 |   
          |                                                                 |   
    1e+06 +-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                            stress-ng.access.ops_per_sec                        
                                                                                
  45000 +-------------------------------------------------------------------+   
        |                                                                   |   
  40000 |-+          ++.+                                                   |   
        |           :    :     .++.++.+.++.++.+.++.  .+                     |   
        |.+   +. +. :    +.+.++                    ++  +.+.++.+ .+.++.  .++.|   
  35000 |-++.+  +  +                                           +      ++    |   
        |                                                                   |   
  30000 |-+                                                                 |   
        |                                                                   |   
  25000 |-+                                                                 |   
        |                                                                   |   
        | OO OO OO O OO OO O O  O     O  O      OO OO OO                    |   
  20000 |-+                   O  O OO   O  OO O          O                  |   
        |                                                                   |   
  15000 +-------------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample

***************************************************************************************************
lkp-csl-2sp7: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 512G memory
=========================================================================================
class/compiler/cpufreq_governor/disk/fs/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime/ucode:
  filesystem/gcc-9/performance/1HDD/xfs/x86_64-rhel-8.3/10%/debian-10.4-x86_64-20200603.cgz/lkp-csl-2sp7/access/stress-ng/60s/0x5003006

commit: 
  a88f997967 ("Increase size of ucounts to atomic_long_t")
  e1e57d56fe ("Add a reference to ucounts for each cred")

a88f9979677c6f55 e1e57d56fef0dd06daf3743f094 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
   2101125           -42.3%    1213160 ±  4%  stress-ng.access.ops
     35018           -42.3%      20219 ±  4%  stress-ng.access.ops_per_sec
     10.19           -41.9%       5.92 ±  4%  stress-ng.time.user_time
      0.27 ±  6%      -0.1        0.20 ±  2%  mpstat.cpu.all.usr%
    303.99            -1.3%     300.00        pmeter.Average_Active_Power
      1136 ± 10%     +35.9%       1545 ± 19%  sched_debug.cfs_rq:/.runnable_avg.max
      1135 ± 10%     +36.0%       1545 ± 19%  sched_debug.cfs_rq:/.util_avg.max
     31.83 ±  9%     -15.7%      26.83 ±  7%  sched_debug.cpu.nr_uninterruptible.max
     16112 ± 11%     +23.7%      19924 ± 10%  numa-meminfo.node0.Mapped
      2090 ± 19%     +38.8%       2900 ± 17%  numa-meminfo.node0.PageTables
      3068 ± 98%    +180.5%       8607 ± 42%  numa-meminfo.node0.Shmem
     21447 ±  9%     -18.2%      17535 ± 11%  numa-meminfo.node1.Mapped
     16587 ± 20%     -33.9%      10971 ± 31%  numa-meminfo.node1.Shmem
      4093 ± 11%     +23.8%       5069 ± 10%  numa-vmstat.node0.nr_mapped
    521.00 ± 19%     +37.5%     716.17 ± 16%  numa-vmstat.node0.nr_page_table_pages
    766.83 ± 99%    +180.5%       2151 ± 42%  numa-vmstat.node0.nr_shmem
      5432 ±  9%     -18.4%       4433 ± 11%  numa-vmstat.node1.nr_mapped
      4147 ± 20%     -33.8%       2744 ± 31%  numa-vmstat.node1.nr_shmem
    106.33 ± 24%   +1398.7%       1593 ±178%  interrupts.CPU15.NMI:Non-maskable_interrupts
    106.33 ± 24%   +1398.7%       1593 ±178%  interrupts.CPU15.PMI:Performance_monitoring_interrupts
    101.33 ± 32%   +1158.9%       1275 ±196%  interrupts.CPU16.NMI:Non-maskable_interrupts
    101.33 ± 32%   +1158.9%       1275 ±196%  interrupts.CPU16.PMI:Performance_monitoring_interrupts
    105.00 ± 23%   +1277.5%       1446 ±199%  interrupts.CPU17.NMI:Non-maskable_interrupts
    105.00 ± 23%   +1277.5%       1446 ±199%  interrupts.CPU17.PMI:Performance_monitoring_interrupts
    117.33 ±  7%    +960.7%       1244 ±195%  interrupts.CPU21.NMI:Non-maskable_interrupts
    117.33 ±  7%    +960.7%       1244 ±195%  interrupts.CPU21.PMI:Performance_monitoring_interrupts
    114.33 ±  9%   +2296.1%       2739 ±133%  interrupts.CPU22.NMI:Non-maskable_interrupts
    114.33 ±  9%   +2296.1%       2739 ±133%  interrupts.CPU22.PMI:Performance_monitoring_interrupts
    122.33 ±  4%   +1530.1%       1994 ±139%  interrupts.CPU3.NMI:Non-maskable_interrupts
    122.33 ±  4%   +1530.1%       1994 ±139%  interrupts.CPU3.PMI:Performance_monitoring_interrupts
    118.00 ±  9%   +1711.9%       2138 ±130%  interrupts.CPU46.NMI:Non-maskable_interrupts
    118.00 ±  9%   +1711.9%       2138 ±130%  interrupts.CPU46.PMI:Performance_monitoring_interrupts
    120.17 ± 23%    +964.5%       1279 ±182%  interrupts.CPU52.NMI:Non-maskable_interrupts
    120.17 ± 23%    +964.5%       1279 ±182%  interrupts.CPU52.PMI:Performance_monitoring_interrupts
     86.50 ± 36%   +1641.4%       1506 ±190%  interrupts.CPU58.NMI:Non-maskable_interrupts
     86.50 ± 36%   +1641.4%       1506 ±190%  interrupts.CPU58.PMI:Performance_monitoring_interrupts
    109.83 ± 21%    +873.1%       1068 ±191%  interrupts.CPU69.NMI:Non-maskable_interrupts
    109.83 ± 21%    +873.1%       1068 ±191%  interrupts.CPU69.PMI:Performance_monitoring_interrupts
    103.33 ± 22%    +490.3%     610.00 ±164%  interrupts.CPU87.NMI:Non-maskable_interrupts
    103.33 ± 22%    +490.3%     610.00 ±164%  interrupts.CPU87.PMI:Performance_monitoring_interrupts
    109.17 ± 27%   +1859.8%       2139 ±139%  interrupts.CPU94.NMI:Non-maskable_interrupts
    109.17 ± 27%   +1859.8%       2139 ±139%  interrupts.CPU94.PMI:Performance_monitoring_interrupts
      8.31 ±  7%     +53.2%      12.74 ±  2%  perf-stat.i.MPKI
 2.845e+09           -30.2%  1.985e+09 ±  2%  perf-stat.i.branch-instructions
      0.84 ±  3%      +0.1        0.95        perf-stat.i.branch-miss-rate%
  22369243 ±  4%     -16.3%   18726571 ±  2%  perf-stat.i.branch-misses
      1.86           +53.6%       2.86 ±  3%  perf-stat.i.cpi
 2.766e+10            +3.9%  2.873e+10        perf-stat.i.cpu-cycles
 4.076e+09 ±  2%     -33.0%  2.732e+09 ±  4%  perf-stat.i.dTLB-loads
 2.322e+09           -36.8%  1.467e+09 ±  3%  perf-stat.i.dTLB-stores
     83.49            -7.8       75.66        perf-stat.i.iTLB-load-miss-rate%
  14193692           -41.6%    8290248 ±  2%  perf-stat.i.iTLB-load-misses
 1.479e+10           -32.0%  1.005e+10 ±  2%  perf-stat.i.instructions
      1146           +12.8%       1293        perf-stat.i.instructions-per-iTLB-miss
      0.55           -33.1%       0.37 ±  3%  perf-stat.i.ipc
      0.29            +3.9%       0.30        perf-stat.i.metric.GHz
     97.79           -32.5%      65.96 ±  3%  perf-stat.i.metric.M/sec
     93.66            +3.2       96.82        perf-stat.i.node-load-miss-rate%
   5224551 ±  8%     +42.9%    7464161 ±  5%  perf-stat.i.node-load-misses
    319707 ± 17%     -42.2%     184671 ±  9%  perf-stat.i.node-loads
  13233767 ±  9%     -23.4%   10143303 ±  8%  perf-stat.i.node-store-misses
      8.40 ±  7%     +52.0%      12.77 ±  2%  perf-stat.overall.MPKI
      0.79 ±  3%      +0.2        0.94        perf-stat.overall.branch-miss-rate%
      1.87           +52.9%       2.86 ±  3%  perf-stat.overall.cpi
      0.00 ± 41%      +0.0        0.00 ± 30%  perf-stat.overall.dTLB-load-miss-rate%
      0.00 ± 12%      +0.0        0.00 ± 16%  perf-stat.overall.dTLB-store-miss-rate%
     84.65            -8.3       76.34        perf-stat.overall.iTLB-load-miss-rate%
      1041           +16.4%       1212        perf-stat.overall.instructions-per-iTLB-miss
      0.53           -34.5%       0.35 ±  3%  perf-stat.overall.ipc
     94.24            +3.3       97.58        perf-stat.overall.node-load-miss-rate%
   2.8e+09           -30.2%  1.953e+09 ±  2%  perf-stat.ps.branch-instructions
  21989609 ±  4%     -16.2%   18425367 ±  2%  perf-stat.ps.branch-misses
 2.722e+10            +3.8%  2.827e+10        perf-stat.ps.cpu-cycles
  4.01e+09 ±  2%     -33.0%  2.688e+09 ±  4%  perf-stat.ps.dTLB-loads
 2.285e+09           -36.8%  1.443e+09 ±  3%  perf-stat.ps.dTLB-stores
  13967002           -41.6%    8156895 ±  2%  perf-stat.ps.iTLB-load-misses
 1.455e+10           -32.0%  9.888e+09 ±  2%  perf-stat.ps.instructions
   5142072 ±  8%     +42.8%    7344021 ±  5%  perf-stat.ps.node-load-misses
    314656 ± 17%     -42.2%     181724 ±  9%  perf-stat.ps.node-loads
  13025851 ±  9%     -23.4%    9980421 ±  8%  perf-stat.ps.node-store-misses
  9.19e+11           -32.1%  6.244e+11 ±  3%  perf-stat.total.instructions
      5.77 ±  6%      -5.0        0.73 ± 10%  perf-profile.calltrace.cycles-pp.free_uid.put_cred_rcu.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.73 ±  6%      -5.0        0.71 ± 10%  perf-profile.calltrace.cycles-pp.refcount_dec_and_lock_irqsave.free_uid.put_cred_rcu.do_faccessat.do_syscall_64
      5.71 ±  6%      -5.0        0.70 ± 10%  perf-profile.calltrace.cycles-pp.refcount_dec_not_one.refcount_dec_and_lock_irqsave.free_uid.put_cred_rcu.do_faccessat
      8.88 ± 11%      -4.7        4.16 ±  8%  perf-profile.calltrace.cycles-pp.__x64_sys_fchmod.do_syscall_64.entry_SYSCALL_64_after_hwframe
      8.78 ± 11%      -4.7        4.10 ±  8%  perf-profile.calltrace.cycles-pp.chmod_common.__x64_sys_fchmod.do_syscall_64.entry_SYSCALL_64_after_hwframe
      8.05 ± 11%      -4.3        3.74 ±  8%  perf-profile.calltrace.cycles-pp.notify_change.chmod_common.__x64_sys_fchmod.do_syscall_64.entry_SYSCALL_64_after_hwframe
      7.84 ± 11%      -4.2        3.60 ±  8%  perf-profile.calltrace.cycles-pp.xfs_vn_setattr.notify_change.chmod_common.__x64_sys_fchmod.do_syscall_64
      7.77 ± 11%      -4.2        3.56 ±  8%  perf-profile.calltrace.cycles-pp.xfs_setattr_nonsize.xfs_vn_setattr.notify_change.chmod_common.__x64_sys_fchmod
      5.72 ± 11%      -4.2        1.56 ± 12%  perf-profile.calltrace.cycles-pp.key_put.put_cred_rcu.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      4.83 ± 13%      -2.8        1.99 ± 10%  perf-profile.calltrace.cycles-pp.__xfs_trans_commit.xfs_setattr_nonsize.xfs_vn_setattr.notify_change.chmod_common
      4.70 ± 13%      -2.8        1.92 ± 10%  perf-profile.calltrace.cycles-pp.xfs_log_commit_cil.__xfs_trans_commit.xfs_setattr_nonsize.xfs_vn_setattr.notify_change
      4.90 ± 14%      -2.7        2.19 ±  9%  perf-profile.calltrace.cycles-pp.security_prepare_creds.prepare_creds.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      3.29 ± 16%      -2.0        1.32 ± 10%  perf-profile.calltrace.cycles-pp.apparmor_cred_prepare.security_prepare_creds.prepare_creds.do_faccessat.do_syscall_64
      3.16 ±  9%      -1.5        1.62 ± 10%  perf-profile.calltrace.cycles-pp.security_cred_free.put_cred_rcu.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.14 ± 12%      -1.3        0.82 ± 11%  perf-profile.calltrace.cycles-pp.xlog_cil_insert_items.xfs_log_commit_cil.__xfs_trans_commit.xfs_setattr_nonsize.xfs_vn_setattr
      2.85 ± 10%      -1.3        1.59 ±  8%  perf-profile.calltrace.cycles-pp.filename_lookup.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.46 ±  8%      -1.2        1.30 ± 10%  perf-profile.calltrace.cycles-pp.__do_sys_newfstatat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.53 ± 10%      -1.1        1.40 ±  8%  perf-profile.calltrace.cycles-pp.path_lookupat.filename_lookup.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.14 ±  8%      -1.1        1.09 ± 11%  perf-profile.calltrace.cycles-pp.apparmor_cred_free.security_cred_free.put_cred_rcu.do_faccessat.do_syscall_64
      2.17 ±  9%      -1.0        1.13 ± 10%  perf-profile.calltrace.cycles-pp.vfs_statx.__do_sys_newfstatat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.66 ± 13%      -0.8        0.84 ±  6%  perf-profile.calltrace.cycles-pp.xfs_trans_alloc.xfs_setattr_nonsize.xfs_vn_setattr.notify_change.chmod_common
      1.43 ± 13%      -0.7        0.71 ±  7%  perf-profile.calltrace.cycles-pp.xfs_trans_reserve.xfs_trans_alloc.xfs_setattr_nonsize.xfs_vn_setattr.notify_change
      1.39 ± 14%      -0.7        0.69 ±  7%  perf-profile.calltrace.cycles-pp.xfs_log_reserve.xfs_trans_reserve.xfs_trans_alloc.xfs_setattr_nonsize.xfs_vn_setattr
      1.50 ± 10%      -0.7        0.81 ± 10%  perf-profile.calltrace.cycles-pp.__kmalloc.security_prepare_creds.prepare_creds.do_faccessat.do_syscall_64
      1.46 ± 10%      -0.7        0.80 ± 10%  perf-profile.calltrace.cycles-pp.kmem_cache_alloc.prepare_creds.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.92 ± 10%      -0.7        0.26 ±100%  perf-profile.calltrace.cycles-pp.filename_lookup.vfs_statx.__do_sys_newfstatat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.55 ± 10%      -0.6        0.91 ±  8%  perf-profile.calltrace.cycles-pp.user_path_at_empty.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.52 ± 10%      -0.6        0.89 ±  8%  perf-profile.calltrace.cycles-pp.getname_flags.user_path_at_empty.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.38 ±  9%      -0.6        0.75 ±  9%  perf-profile.calltrace.cycles-pp.link_path_walk.path_lookupat.filename_lookup.do_faccessat.do_syscall_64
      0.96 ± 12%      -0.6        0.37 ± 70%  perf-profile.calltrace.cycles-pp.kfree.security_cred_free.put_cred_rcu.do_faccessat.do_syscall_64
      0.89 ±  9%      -0.5        0.36 ± 70%  perf-profile.calltrace.cycles-pp.kmem_cache_free.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.18 ± 10%      -0.5        0.67 ±  9%  perf-profile.calltrace.cycles-pp.strncpy_from_user.getname_flags.user_path_at_empty.do_faccessat.do_syscall_64
      0.00            +1.5        1.52 ± 17%  perf-profile.calltrace.cycles-pp.override_creds.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.47 ± 38%      +2.4        3.89 ± 57%  perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      0.00           +15.2       15.20 ± 12%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.get_ucounts.prepare_creds.do_faccessat
      0.00           +15.6       15.63 ± 12%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.put_ucounts.put_cred_rcu.do_faccessat
      0.00           +16.8       16.78 ± 12%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.get_ucounts.prepare_creds.do_faccessat.do_syscall_64
      0.00           +17.1       17.12 ± 12%  perf-profile.calltrace.cycles-pp.get_ucounts.prepare_creds.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00           +17.2       17.23 ± 12%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.put_ucounts.put_cred_rcu.do_faccessat.do_syscall_64
      0.00           +17.6       17.57 ± 12%  perf-profile.calltrace.cycles-pp.put_ucounts.put_cred_rcu.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.77 ±  6%      -5.0        0.73 ± 10%  perf-profile.children.cycles-pp.free_uid
      5.74 ±  6%      -5.0        0.72 ±  9%  perf-profile.children.cycles-pp.refcount_dec_and_lock_irqsave
      5.71 ±  6%      -5.0        0.70 ± 10%  perf-profile.children.cycles-pp.refcount_dec_not_one
      8.88 ± 11%      -4.7        4.16 ±  8%  perf-profile.children.cycles-pp.__x64_sys_fchmod
      8.78 ± 11%      -4.7        4.10 ±  8%  perf-profile.children.cycles-pp.chmod_common
      8.05 ± 11%      -4.3        3.74 ±  8%  perf-profile.children.cycles-pp.notify_change
      7.85 ± 11%      -4.2        3.60 ±  8%  perf-profile.children.cycles-pp.xfs_vn_setattr
      7.78 ± 11%      -4.2        3.56 ±  8%  perf-profile.children.cycles-pp.xfs_setattr_nonsize
      5.74 ± 11%      -4.2        1.57 ± 13%  perf-profile.children.cycles-pp.key_put
      4.83 ± 13%      -2.8        1.99 ± 10%  perf-profile.children.cycles-pp.__xfs_trans_commit
      4.71 ± 13%      -2.8        1.92 ± 10%  perf-profile.children.cycles-pp.xfs_log_commit_cil
      4.90 ± 14%      -2.7        2.19 ± 10%  perf-profile.children.cycles-pp.security_prepare_creds
      3.28 ± 48%      -2.6        0.71 ±113%  perf-profile.children.cycles-pp.start_kernel
      3.29 ± 16%      -2.0        1.32 ± 10%  perf-profile.children.cycles-pp.apparmor_cred_prepare
      3.79 ± 10%      -1.7        2.08 ±  8%  perf-profile.children.cycles-pp.filename_lookup
      3.39 ± 10%      -1.5        1.85 ±  9%  perf-profile.children.cycles-pp.path_lookupat
      3.16 ±  9%      -1.5        1.62 ± 10%  perf-profile.children.cycles-pp.security_cred_free
      2.15 ± 12%      -1.3        0.83 ± 10%  perf-profile.children.cycles-pp.xlog_cil_insert_items
      2.47 ±  8%      -1.2        1.30 ± 10%  perf-profile.children.cycles-pp.__do_sys_newfstatat
      2.16 ±  9%      -1.1        1.09 ± 11%  perf-profile.children.cycles-pp.apparmor_cred_free
      2.17 ± 10%      -1.0        1.14 ±  9%  perf-profile.children.cycles-pp.vfs_statx
      1.62 ± 13%      -1.0        0.60 ± 13%  perf-profile.children.cycles-pp._raw_spin_lock
      2.11 ± 10%      -0.9        1.22 ± 10%  perf-profile.children.cycles-pp.kmem_cache_alloc
      1.87 ±  9%      -0.9        1.01 ±  9%  perf-profile.children.cycles-pp.link_path_walk
      2.02 ± 10%      -0.8        1.17 ±  8%  perf-profile.children.cycles-pp.user_path_at_empty
      1.99 ± 10%      -0.8        1.15 ±  7%  perf-profile.children.cycles-pp.getname_flags
      1.67 ± 13%      -0.8        0.84 ±  6%  perf-profile.children.cycles-pp.xfs_trans_alloc
      1.43 ± 13%      -0.7        0.71 ±  7%  perf-profile.children.cycles-pp.xfs_trans_reserve
      1.40 ± 14%      -0.7        0.69 ±  6%  perf-profile.children.cycles-pp.xfs_log_reserve
      1.54 ± 10%      -0.7        0.84 ± 10%  perf-profile.children.cycles-pp.__kmalloc
      1.52 ± 10%      -0.7        0.85 ±  7%  perf-profile.children.cycles-pp.strncpy_from_user
      1.06 ± 18%      -0.6        0.44 ± 12%  perf-profile.children.cycles-pp.xfs_log_ticket_ungrant
      1.21 ±  9%      -0.5        0.69 ±  9%  perf-profile.children.cycles-pp.kmem_cache_free
      0.94 ±  9%      -0.5        0.43 ±  8%  perf-profile.children.cycles-pp.common_perm_cond
      0.89 ± 10%      -0.5        0.39 ±  8%  perf-profile.children.cycles-pp.common_perm
      0.97 ± 12%      -0.4        0.54 ± 11%  perf-profile.children.cycles-pp.kfree
      0.80 ±  8%      -0.4        0.40 ± 10%  perf-profile.children.cycles-pp.inode_permission
      0.86 ±  8%      -0.4        0.48 ± 10%  perf-profile.children.cycles-pp.walk_component
      0.56 ± 22%      -0.3        0.21 ± 10%  perf-profile.children.cycles-pp.xlog_grant_add_space
      0.61 ±  8%      -0.3        0.29 ± 11%  perf-profile.children.cycles-pp.generic_permission
      0.52 ± 26%      -0.3        0.19 ± 48%  perf-profile.children.cycles-pp.xfs_trans_ijoin
      0.67 ±  9%      -0.3        0.37 ±  7%  perf-profile.children.cycles-pp.__check_object_size
      0.54 ± 10%      -0.3        0.25 ± 11%  perf-profile.children.cycles-pp.vfs_getattr
      0.64 ± 11%      -0.3        0.34 ± 12%  perf-profile.children.cycles-pp.complete_walk
      0.54 ±  9%      -0.3        0.24 ± 10%  perf-profile.children.cycles-pp.security_inode_getattr
      0.61 ± 12%      -0.3        0.33 ±  7%  perf-profile.children.cycles-pp.down_read
      0.59 ± 12%      -0.3        0.33 ± 11%  perf-profile.children.cycles-pp.unlazy_walk
      0.48 ±  9%      -0.3        0.22 ±  9%  perf-profile.children.cycles-pp.security_path_chmod
      0.56 ± 12%      -0.2        0.31 ±  7%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.45 ± 14%      -0.2        0.23 ±  9%  perf-profile.children.cycles-pp.get_obj_cgroup_from_current
      0.45 ± 12%      -0.2        0.23 ± 12%  perf-profile.children.cycles-pp.obj_cgroup_charge
      0.47 ±  7%      -0.2        0.26 ±  8%  perf-profile.children.cycles-pp.lookup_fast
      0.46 ± 10%      -0.2        0.26 ± 13%  perf-profile.children.cycles-pp.__legitimize_path
      0.44 ±  9%      -0.2        0.24 ±  8%  perf-profile.children.cycles-pp.xlog_grant_push_ail
      0.43 ±  9%      -0.2        0.24 ±  8%  perf-profile.children.cycles-pp.xlog_grant_push_threshold
      0.43 ±  9%      -0.2        0.24 ±  8%  perf-profile.children.cycles-pp.xlog_space_left
      0.33 ± 23%      -0.2        0.14 ± 13%  perf-profile.children.cycles-pp.revert_creds
      0.42 ±  7%      -0.2        0.24 ± 11%  perf-profile.children.cycles-pp.__mod_memcg_lruvec_state
      0.29 ± 11%      -0.2        0.11 ± 13%  perf-profile.children.cycles-pp.capable_wrt_inode_uidgid
      0.41 ± 11%      -0.2        0.24 ±  8%  perf-profile.children.cycles-pp.___might_sleep
      0.28 ± 11%      -0.2        0.11 ± 10%  perf-profile.children.cycles-pp.ns_capable_common
      0.28 ± 11%      -0.2        0.10 ± 10%  perf-profile.children.cycles-pp.security_capable
      0.42 ± 13%      -0.2        0.25 ± 16%  perf-profile.children.cycles-pp.refill_obj_stock
      0.26 ± 12%      -0.2        0.09 ± 11%  perf-profile.children.cycles-pp.apparmor_capable
      0.36 ± 10%      -0.2        0.20 ± 11%  perf-profile.children.cycles-pp.path_put
      0.36 ±  9%      -0.2        0.20 ± 10%  perf-profile.children.cycles-pp.dput
      0.34 ± 12%      -0.2        0.19 ± 10%  perf-profile.children.cycles-pp.__entry_text_start
      0.31 ± 11%      -0.1        0.17 ±  7%  perf-profile.children.cycles-pp.path_init
      0.33 ±  7%      -0.1        0.18 ± 10%  perf-profile.children.cycles-pp.__check_heap_object
      0.31 ±  7%      -0.1        0.17 ±  7%  perf-profile.children.cycles-pp.__d_lookup_rcu
      0.28 ±  6%      -0.1        0.16 ± 18%  perf-profile.children.cycles-pp.xfs_inode_item_format
      0.27 ± 10%      -0.1        0.16 ± 11%  perf-profile.children.cycles-pp.__might_sleep
      0.24 ± 13%      -0.1        0.13 ±  8%  perf-profile.children.cycles-pp.__legitimize_mnt
      0.23 ± 11%      -0.1        0.11 ± 12%  perf-profile.children.cycles-pp._cond_resched
      0.27 ± 14%      -0.1        0.16 ± 14%  perf-profile.children.cycles-pp.cp_new_stat
      0.17 ± 20%      -0.1        0.06 ± 48%  perf-profile.children.cycles-pp.up_read
      0.23 ± 17%      -0.1        0.13 ± 11%  perf-profile.children.cycles-pp.memset_erms
      0.20 ±  9%      -0.1        0.10 ± 15%  perf-profile.children.cycles-pp.__mod_memcg_state
      0.27 ± 15%      -0.1        0.17 ± 11%  perf-profile.children.cycles-pp.xlog_ticket_alloc
      0.20 ± 13%      -0.1        0.11 ± 14%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.12 ± 12%      -0.1        0.03 ±101%  perf-profile.children.cycles-pp.__list_add_valid
      0.19 ± 14%      -0.1        0.11 ± 14%  perf-profile.children.cycles-pp.step_into
      0.17 ±  9%      -0.1        0.09 ± 18%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
      0.17 ± 14%      -0.1        0.09 ± 18%  perf-profile.children.cycles-pp.__might_fault
      0.10 ± 10%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.security_inode_permission
      0.18 ± 10%      -0.1        0.10 ± 16%  perf-profile.children.cycles-pp.lockref_get_not_dead
      0.17 ±  8%      -0.1        0.10 ±  6%  perf-profile.children.cycles-pp.rcu_read_unlock_strict
      0.15 ± 12%      -0.1        0.09 ± 12%  perf-profile.children.cycles-pp._copy_to_user
      0.13 ±  6%      -0.1        0.07 ± 15%  perf-profile.children.cycles-pp.lockref_put_return
      0.11 ± 13%      -0.1        0.05 ± 47%  perf-profile.children.cycles-pp.rcu_all_qs
      0.13 ±  8%      -0.1        0.07 ± 19%  perf-profile.children.cycles-pp.down_write
      0.09 ± 18%      -0.1        0.04 ± 72%  perf-profile.children.cycles-pp.mnt_want_write
      0.11 ±  9%      -0.0        0.06 ± 15%  perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
      0.09 ± 14%      -0.0        0.04 ± 72%  perf-profile.children.cycles-pp.up_write
      0.12 ± 11%      -0.0        0.07 ± 11%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      0.09 ± 18%      -0.0        0.05 ± 45%  perf-profile.children.cycles-pp.__fget_light
      0.10 ± 14%      -0.0        0.06 ± 21%  perf-profile.children.cycles-pp.__put_cred
      0.08 ±  9%      -0.0        0.04 ± 72%  perf-profile.children.cycles-pp.terminate_walk
      0.07 ± 16%      -0.0        0.03 ± 70%  perf-profile.children.cycles-pp.syscall_enter_from_user_mode
      0.10 ± 12%      -0.0        0.06 ± 11%  perf-profile.children.cycles-pp.__virt_addr_valid
      0.05 ± 45%      +0.3        0.40 ± 12%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.16 ± 11%      +1.4        1.52 ± 17%  perf-profile.children.cycles-pp.override_creds
      1.48 ± 38%      +2.4        3.91 ± 57%  perf-profile.children.cycles-pp.menu_select
      0.00           +17.1       17.13 ± 12%  perf-profile.children.cycles-pp.get_ucounts
      0.00           +17.6       17.57 ± 12%  perf-profile.children.cycles-pp.put_ucounts
      0.76 ± 16%     +30.2       30.99 ± 12%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      0.06 ± 13%     +34.0       34.08 ± 12%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
     13.33 ±  8%      -9.2        4.18 ± 10%  perf-profile.self.cycles-pp.prepare_creds
      5.68 ±  6%      -5.0        0.70 ± 10%  perf-profile.self.cycles-pp.refcount_dec_not_one
      5.70 ± 11%      -4.1        1.55 ± 13%  perf-profile.self.cycles-pp.key_put
      4.91 ±  6%      -2.1        2.77 ± 11%  perf-profile.self.cycles-pp.put_cred_rcu
      3.27 ± 16%      -2.0        1.31 ± 10%  perf-profile.self.cycles-pp.apparmor_cred_prepare
      2.15 ±  9%      -1.1        1.08 ± 11%  perf-profile.self.cycles-pp.apparmor_cred_free
      0.99 ± 17%      -0.6        0.42 ± 12%  perf-profile.self.cycles-pp.xfs_log_ticket_ungrant
      0.88 ± 10%      -0.5        0.39 ±  8%  perf-profile.self.cycles-pp.common_perm
      0.99 ± 10%      -0.5        0.54 ±  9%  perf-profile.self.cycles-pp.link_path_walk
      1.07 ±  9%      -0.4        0.63 ± 11%  perf-profile.self.cycles-pp.kmem_cache_alloc
      0.86 ± 13%      -0.4        0.44 ±  7%  perf-profile.self.cycles-pp._raw_spin_lock
      0.86 ±  8%      -0.4        0.48 ±  7%  perf-profile.self.cycles-pp.kmem_cache_free
      0.55 ± 22%      -0.3        0.21 ± 10%  perf-profile.self.cycles-pp.xlog_grant_add_space
      0.76 ±  9%      -0.3        0.43 ± 10%  perf-profile.self.cycles-pp.__kmalloc
      0.50 ± 16%      -0.3        0.17 ± 22%  perf-profile.self.cycles-pp.xfs_log_commit_cil
      0.65 ± 15%      -0.3        0.35 ± 10%  perf-profile.self.cycles-pp.kfree
      0.69 ± 12%      -0.3        0.40 ±  8%  perf-profile.self.cycles-pp.strncpy_from_user
      0.58 ± 11%      -0.3        0.31 ±  7%  perf-profile.self.cycles-pp.down_read
      0.56 ± 12%      -0.2        0.31 ±  8%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.47 ±  9%      -0.2        0.24 ±  7%  perf-profile.self.cycles-pp.xlog_cil_insert_items
      0.43 ± 11%      -0.2        0.22 ± 12%  perf-profile.self.cycles-pp.obj_cgroup_charge
      0.43 ±  9%      -0.2        0.23 ±  7%  perf-profile.self.cycles-pp.xlog_space_left
      0.33 ± 23%      -0.2        0.14 ± 13%  perf-profile.self.cycles-pp.revert_creds
      0.38 ± 15%      -0.2        0.19 ±  9%  perf-profile.self.cycles-pp.get_obj_cgroup_from_current
      0.40 ± 12%      -0.2        0.23 ±  9%  perf-profile.self.cycles-pp.___might_sleep
      0.26 ± 11%      -0.2        0.09 ± 11%  perf-profile.self.cycles-pp.apparmor_capable
      0.41 ± 13%      -0.2        0.25 ± 15%  perf-profile.self.cycles-pp.refill_obj_stock
      0.34 ± 12%      -0.2        0.19 ± 10%  perf-profile.self.cycles-pp.__entry_text_start
      0.40 ±  6%      -0.1        0.25 ± 14%  perf-profile.self.cycles-pp.do_faccessat
      0.22 ± 31%      -0.1        0.07 ± 60%  perf-profile.self.cycles-pp.xfs_trans_ijoin
      0.32 ±  7%      -0.1        0.17 ± 10%  perf-profile.self.cycles-pp.__check_heap_object
      0.31 ±  6%      -0.1        0.17 ±  9%  perf-profile.self.cycles-pp.__d_lookup_rcu
      0.28 ± 12%      -0.1        0.15 ±  9%  perf-profile.self.cycles-pp.path_init
      0.28 ± 11%      -0.1        0.16 ± 13%  perf-profile.self.cycles-pp.generic_permission
      0.24 ± 13%      -0.1        0.13 ±  6%  perf-profile.self.cycles-pp.__check_object_size
      0.17 ± 20%      -0.1        0.06 ± 49%  perf-profile.self.cycles-pp.up_read
      0.23 ± 13%      -0.1        0.13 ±  8%  perf-profile.self.cycles-pp.__legitimize_mnt
      0.25 ± 10%      -0.1        0.15 ± 12%  perf-profile.self.cycles-pp.__might_sleep
      0.22 ± 18%      -0.1        0.12 ± 10%  perf-profile.self.cycles-pp.memset_erms
      0.22 ±  8%      -0.1        0.12 ± 20%  perf-profile.self.cycles-pp.xfs_inode_item_format
      0.20 ±  9%      -0.1        0.10 ± 15%  perf-profile.self.cycles-pp.__mod_memcg_state
      0.22 ±  9%      -0.1        0.12 ± 14%  perf-profile.self.cycles-pp.inode_permission
      0.20 ±  8%      -0.1        0.11 ± 19%  perf-profile.self.cycles-pp.walk_component
      0.21 ± 10%      -0.1        0.11 ±  9%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.20 ± 14%      -0.1        0.10 ± 14%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.18 ± 12%      -0.1        0.10 ± 14%  perf-profile.self.cycles-pp.xfs_trans_log_inode
      0.11 ± 11%      -0.1        0.03 ±101%  perf-profile.self.cycles-pp.__list_add_valid
      0.18 ± 16%      -0.1        0.10 ± 15%  perf-profile.self.cycles-pp.step_into
      0.20 ±  7%      -0.1        0.13 ± 13%  perf-profile.self.cycles-pp.__mod_memcg_lruvec_state
      0.17 ± 10%      -0.1        0.10 ± 18%  perf-profile.self.cycles-pp.lockref_get_not_dead
      0.10 ±  7%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.security_inode_permission
      0.15 ± 10%      -0.1        0.08 ± 13%  perf-profile.self.cycles-pp.lookup_fast
      0.14 ± 11%      -0.1        0.07 ± 10%  perf-profile.self.cycles-pp.getname_flags
      0.14 ± 17%      -0.1        0.08 ± 12%  perf-profile.self.cycles-pp.filename_lookup
      0.12 ±  7%      -0.1        0.07 ± 16%  perf-profile.self.cycles-pp.lockref_put_return
      0.08 ± 16%      -0.1        0.03 ±102%  perf-profile.self.cycles-pp.up_write
      0.08 ± 23%      -0.1        0.03 ± 99%  perf-profile.self.cycles-pp.unlazy_walk
      0.11 ±  9%      -0.0        0.06 ± 15%  perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack
      0.11 ± 13%      -0.0        0.06 ± 15%  perf-profile.self.cycles-pp._cond_resched
      0.12 ±  7%      -0.0        0.07 ± 15%  perf-profile.self.cycles-pp.rcu_read_unlock_strict
      0.11 ±  9%      -0.0        0.07 ± 13%  perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      0.10 ±  6%      -0.0        0.06 ±  6%  perf-profile.self.cycles-pp.path_lookupat
      0.09 ± 17%      -0.0        0.05 ± 45%  perf-profile.self.cycles-pp.__fget_light
      0.12 ± 12%      -0.0        0.07 ± 16%  perf-profile.self.cycles-pp.notify_change
      0.07 ± 11%      -0.0        0.03 ± 99%  perf-profile.self.cycles-pp.syscall_enter_from_user_mode
      0.09 ± 14%      -0.0        0.06 ±  8%  perf-profile.self.cycles-pp.__virt_addr_valid
      0.09 ± 13%      -0.0        0.06 ± 16%  perf-profile.self.cycles-pp.__put_cred
      0.00            +0.2        0.18 ± 15%  perf-profile.self.cycles-pp.get_ucounts
      0.00            +0.2        0.18 ± 11%  perf-profile.self.cycles-pp.put_ucounts
      0.05 ± 45%      +0.2        0.24 ± 17%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.67 ± 24%      +1.3        1.94 ± 39%  perf-profile.self.cycles-pp.cpuidle_enter_state
      0.15 ± 10%      +1.4        1.51 ± 17%  perf-profile.self.cycles-pp.override_creds
      0.88 ± 42%      +2.4        3.25 ± 59%  perf-profile.self.cycles-pp.menu_select
      0.06 ± 15%      +3.2        3.25 ±  9%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.75 ± 16%     +30.2       30.98 ± 12%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath



***************************************************************************************************
lkp-csl-2sp7: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 512G memory
=========================================================================================
class/compiler/cpufreq_governor/disk/fs/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime/ucode:
  filesystem/gcc-9/performance/1HDD/f2fs/x86_64-rhel-8.3/10%/debian-10.4-x86_64-20200603.cgz/lkp-csl-2sp7/access/stress-ng/60s/0x5003006

commit: 
  a88f997967 ("Increase size of ucounts to atomic_long_t")
  e1e57d56fe ("Add a reference to ucounts for each cred")

a88f9979677c6f55 e1e57d56fef0dd06daf3743f094 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
   2266573           -44.5%    1258748 ±  4%  stress-ng.access.ops
     37776           -44.5%      20979 ±  4%  stress-ng.access.ops_per_sec
     10.37 ±  2%     -45.1%       5.70 ±  5%  stress-ng.time.user_time
      0.30            -0.1        0.21 ±  3%  mpstat.cpu.all.usr%
    304.83            -1.6%     299.99        pmeter.Average_Active_Power
    116.83 ±  7%     +31.1%     153.17 ±  4%  interrupts.CPU37.NMI:Non-maskable_interrupts
    116.83 ±  7%     +31.1%     153.17 ±  4%  interrupts.CPU37.PMI:Performance_monitoring_interrupts
     86.50 ± 38%   +1387.1%       1286 ±123%  interrupts.CPU60.NMI:Non-maskable_interrupts
     86.50 ± 38%   +1387.1%       1286 ±123%  interrupts.CPU60.PMI:Performance_monitoring_interrupts
     93.17 ± 38%     +62.3%     151.17 ±  2%  interrupts.CPU62.NMI:Non-maskable_interrupts
     93.17 ± 38%     +62.3%     151.17 ±  2%  interrupts.CPU62.PMI:Performance_monitoring_interrupts
     95.00 ± 32%   +1047.9%       1090 ±189%  interrupts.CPU77.NMI:Non-maskable_interrupts
     95.00 ± 32%   +1047.9%       1090 ±189%  interrupts.CPU77.PMI:Performance_monitoring_interrupts
    113.67 ±  9%     +33.1%     151.33 ±  4%  interrupts.CPU85.NMI:Non-maskable_interrupts
    113.67 ±  9%     +33.1%     151.33 ±  4%  interrupts.CPU85.PMI:Performance_monitoring_interrupts
    104.17 ± 31%    +897.1%       1038 ±188%  interrupts.CPU88.NMI:Non-maskable_interrupts
    104.17 ± 31%    +897.1%       1038 ±188%  interrupts.CPU88.PMI:Performance_monitoring_interrupts
    107.17 ± 33%     +45.9%     156.33 ±  4%  interrupts.CPU90.NMI:Non-maskable_interrupts
    107.17 ± 33%     +45.9%     156.33 ±  4%  interrupts.CPU90.PMI:Performance_monitoring_interrupts
    103.17 ± 35%   +1114.1%       1252 ±195%  interrupts.CPU91.NMI:Non-maskable_interrupts
    103.17 ± 35%   +1114.1%       1252 ±195%  interrupts.CPU91.PMI:Performance_monitoring_interrupts
    103.33 ± 33%    +650.2%     775.17 ±133%  interrupts.CPU92.NMI:Non-maskable_interrupts
    103.33 ± 33%    +650.2%     775.17 ±133%  interrupts.CPU92.PMI:Performance_monitoring_interrupts
    109.83 ± 23%    +567.4%     733.00 ±176%  interrupts.CPU93.NMI:Non-maskable_interrupts
    109.83 ± 23%    +567.4%     733.00 ±176%  interrupts.CPU93.PMI:Performance_monitoring_interrupts
    112.83 ± 21%    +656.3%     853.33 ±182%  interrupts.CPU94.NMI:Non-maskable_interrupts
    112.83 ± 21%    +656.3%     853.33 ±182%  interrupts.CPU94.PMI:Performance_monitoring_interrupts
      8.22 ±  5%     +57.2%      12.91        perf-stat.i.MPKI
 2.751e+09           -31.3%   1.89e+09 ±  3%  perf-stat.i.branch-instructions
      0.85            +0.1        0.95        perf-stat.i.branch-miss-rate%
  22385388           -19.2%   18095507 ±  2%  perf-stat.i.branch-misses
      1.95 ±  2%     +56.0%       3.04 ±  3%  perf-stat.i.cpi
  2.79e+10            +3.2%   2.88e+10        perf-stat.i.cpu-cycles
 3.898e+09 ±  2%     -34.1%  2.569e+09 ±  4%  perf-stat.i.dTLB-loads
      0.00 ± 12%      +0.0        0.00 ± 18%  perf-stat.i.dTLB-store-miss-rate%
 2.176e+09           -38.4%   1.34e+09 ±  4%  perf-stat.i.dTLB-stores
     83.86            -7.8       76.11        perf-stat.i.iTLB-load-miss-rate%
  12992520 ±  4%     -43.6%    7326831 ±  3%  perf-stat.i.iTLB-load-misses
 1.427e+10           -33.3%  9.513e+09 ±  3%  perf-stat.i.instructions
      1195 ±  3%     +14.8%       1372        perf-stat.i.instructions-per-iTLB-miss
      0.53           -34.2%       0.35 ±  3%  perf-stat.i.ipc
      0.29            +3.2%       0.30        perf-stat.i.metric.GHz
     93.35           -33.7%      61.88 ±  4%  perf-stat.i.metric.M/sec
     89.51 ±  2%      +6.9       96.40        perf-stat.i.node-load-miss-rate%
   2600729 ±  3%    +135.6%    6126728 ±  7%  perf-stat.i.node-load-misses
    292982 ± 19%     -38.5%     180109 ± 16%  perf-stat.i.node-loads
  13416978 ±  7%     -27.9%    9679639 ±  9%  perf-stat.i.node-store-misses
      8.29 ±  5%     +55.7%      12.90        perf-stat.overall.MPKI
      0.81            +0.1        0.96        perf-stat.overall.branch-miss-rate%
      1.96           +54.9%       3.03 ±  3%  perf-stat.overall.cpi
      0.00 ± 20%      +0.0        0.00 ± 18%  perf-stat.overall.dTLB-load-miss-rate%
      0.00 ± 10%      +0.0        0.00 ± 19%  perf-stat.overall.dTLB-store-miss-rate%
     84.89            -8.2       76.69        perf-stat.overall.iTLB-load-miss-rate%
      1099 ±  3%     +18.1%       1298        perf-stat.overall.instructions-per-iTLB-miss
      0.51           -35.4%       0.33 ±  3%  perf-stat.overall.ipc
     89.87 ±  2%      +7.3       97.15        perf-stat.overall.node-load-miss-rate%
 2.707e+09           -31.3%   1.86e+09 ±  3%  perf-stat.ps.branch-instructions
  22015761           -19.1%   17801305 ±  2%  perf-stat.ps.branch-misses
 2.745e+10            +3.2%  2.833e+10        perf-stat.ps.cpu-cycles
 3.835e+09 ±  2%     -34.1%  2.528e+09 ±  4%  perf-stat.ps.dTLB-loads
 2.141e+09           -38.4%  1.318e+09 ±  4%  perf-stat.ps.dTLB-stores
  12784377 ±  4%     -43.6%    7209165 ±  3%  perf-stat.ps.iTLB-load-misses
 1.404e+10           -33.3%   9.36e+09 ±  3%  perf-stat.ps.instructions
   2559028 ±  3%    +135.6%    6028475 ±  7%  perf-stat.ps.node-load-misses
    288289 ± 19%     -38.5%     177214 ± 16%  perf-stat.ps.node-loads
  13202172 ±  7%     -27.9%    9524529 ±  9%  perf-stat.ps.node-store-misses
 8.819e+11           -33.1%    5.9e+11 ±  3%  perf-stat.total.instructions
      7.41 ±  9%      -6.7        0.75 ± 12%  perf-profile.calltrace.cycles-pp.free_uid.put_cred_rcu.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      7.36 ±  9%      -6.6        0.73 ± 11%  perf-profile.calltrace.cycles-pp.refcount_dec_and_lock_irqsave.free_uid.put_cred_rcu.do_faccessat.do_syscall_64
      7.33 ±  9%      -6.6        0.71 ± 11%  perf-profile.calltrace.cycles-pp.refcount_dec_not_one.refcount_dec_and_lock_irqsave.free_uid.put_cred_rcu.do_faccessat
      6.89 ± 12%      -5.2        1.72 ± 12%  perf-profile.calltrace.cycles-pp.key_put.put_cred_rcu.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.28 ± 14%      -3.0        2.29 ±  5%  perf-profile.calltrace.cycles-pp.security_prepare_creds.prepare_creds.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      3.52 ± 17%      -2.2        1.34 ±  5%  perf-profile.calltrace.cycles-pp.apparmor_cred_prepare.security_prepare_creds.prepare_creds.do_faccessat.do_syscall_64
      3.50 ±  9%      -1.8        1.68 ±  5%  perf-profile.calltrace.cycles-pp.security_cred_free.put_cred_rcu.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      3.20 ±  7%      -1.5        1.70 ±  6%  perf-profile.calltrace.cycles-pp.filename_lookup.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.73 ±  8%      -1.3        1.38 ±  7%  perf-profile.calltrace.cycles-pp.__do_sys_newfstatat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.40 ± 11%      -1.3        1.07 ±  6%  perf-profile.calltrace.cycles-pp.apparmor_cred_free.security_cred_free.put_cred_rcu.do_faccessat.do_syscall_64
      2.79 ±  7%      -1.3        1.51 ±  6%  perf-profile.calltrace.cycles-pp.path_lookupat.filename_lookup.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.44 ± 10%      -1.2        1.19 ± 11%  perf-profile.calltrace.cycles-pp.__x64_sys_fchmod.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.34 ± 10%      -1.2        1.13 ± 10%  perf-profile.calltrace.cycles-pp.chmod_common.__x64_sys_fchmod.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.39 ±  9%      -1.2        1.22 ±  7%  perf-profile.calltrace.cycles-pp.vfs_statx.__do_sys_newfstatat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.76 ±  7%      -0.8        0.97 ±  7%  perf-profile.calltrace.cycles-pp.user_path_at_empty.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.64 ±  8%      -0.8        0.86 ±  9%  perf-profile.calltrace.cycles-pp.kmem_cache_alloc.prepare_creds.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.72 ±  7%      -0.8        0.95 ±  8%  perf-profile.calltrace.cycles-pp.getname_flags.user_path_at_empty.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.53 ± 10%      -0.8        0.76 ± 12%  perf-profile.calltrace.cycles-pp.notify_change.chmod_common.__x64_sys_fchmod.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.64 ± 10%      -0.8        0.88 ±  6%  perf-profile.calltrace.cycles-pp.__kmalloc.security_prepare_creds.prepare_creds.do_faccessat.do_syscall_64
      1.51 ±  6%      -0.7        0.81 ±  5%  perf-profile.calltrace.cycles-pp.link_path_walk.path_lookupat.filename_lookup.do_faccessat.do_syscall_64
      1.35 ± 11%      -0.7        0.67 ± 13%  perf-profile.calltrace.cycles-pp.f2fs_setattr.notify_change.chmod_common.__x64_sys_fchmod.do_syscall_64
      1.00 ±  7%      -0.6        0.37 ± 70%  perf-profile.calltrace.cycles-pp.kmem_cache_free.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.33 ±  7%      -0.6        0.73 ±  9%  perf-profile.calltrace.cycles-pp.strncpy_from_user.getname_flags.user_path_at_empty.do_faccessat.do_syscall_64
      1.05 ±  7%      -0.6        0.49 ± 45%  perf-profile.calltrace.cycles-pp.kfree.security_cred_free.put_cred_rcu.do_faccessat.do_syscall_64
      0.97 ±  9%      -0.5        0.45 ± 45%  perf-profile.calltrace.cycles-pp.filename_lookup.vfs_statx.__do_sys_newfstatat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.75 ± 29%      +1.3        3.05 ±  8%  perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      0.00            +1.6        1.57 ± 11%  perf-profile.calltrace.cycles-pp.override_creds.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00           +17.0       17.03 ±  7%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.get_ucounts.prepare_creds.do_faccessat
      0.00           +17.5       17.51 ±  9%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.put_ucounts.put_cred_rcu.do_faccessat
      0.00           +18.7       18.69 ±  7%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.get_ucounts.prepare_creds.do_faccessat.do_syscall_64
      0.00           +19.0       19.04 ±  7%  perf-profile.calltrace.cycles-pp.get_ucounts.prepare_creds.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00           +19.2       19.19 ±  8%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.put_ucounts.put_cred_rcu.do_faccessat.do_syscall_64
      0.00           +19.5       19.54 ±  8%  perf-profile.calltrace.cycles-pp.put_ucounts.put_cred_rcu.do_faccessat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      7.41 ±  9%      -6.7        0.75 ± 11%  perf-profile.children.cycles-pp.free_uid
      7.37 ±  9%      -6.6        0.73 ± 12%  perf-profile.children.cycles-pp.refcount_dec_and_lock_irqsave
      7.34 ±  9%      -6.6        0.72 ± 11%  perf-profile.children.cycles-pp.refcount_dec_not_one
      6.92 ± 12%      -5.2        1.73 ± 12%  perf-profile.children.cycles-pp.key_put
      5.29 ± 14%      -3.0        2.29 ±  5%  perf-profile.children.cycles-pp.security_prepare_creds
      3.53 ± 17%      -2.2        1.34 ±  5%  perf-profile.children.cycles-pp.apparmor_cred_prepare
      4.19 ±  7%      -1.9        2.24 ±  6%  perf-profile.children.cycles-pp.filename_lookup
      3.50 ±  9%      -1.8        1.68 ±  5%  perf-profile.children.cycles-pp.security_cred_free
      3.69 ±  7%      -1.7        1.99 ±  6%  perf-profile.children.cycles-pp.path_lookupat
      2.73 ±  8%      -1.3        1.38 ±  7%  perf-profile.children.cycles-pp.__do_sys_newfstatat
      2.41 ± 11%      -1.3        1.08 ±  5%  perf-profile.children.cycles-pp.apparmor_cred_free
      2.44 ± 10%      -1.2        1.19 ± 11%  perf-profile.children.cycles-pp.__x64_sys_fchmod
      2.35 ± 10%      -1.2        1.14 ± 10%  perf-profile.children.cycles-pp.chmod_common
      2.40 ±  9%      -1.2        1.23 ±  7%  perf-profile.children.cycles-pp.vfs_statx
      2.28 ±  8%      -1.0        1.26 ±  7%  perf-profile.children.cycles-pp.user_path_at_empty
      2.24 ±  8%      -1.0        1.24 ±  8%  perf-profile.children.cycles-pp.getname_flags
      2.03 ±  7%      -0.9        1.09 ±  6%  perf-profile.children.cycles-pp.link_path_walk
      2.04 ±  8%      -0.9        1.10 ±  8%  perf-profile.children.cycles-pp.kmem_cache_alloc
      1.69 ± 10%      -0.8        0.92 ±  6%  perf-profile.children.cycles-pp.__kmalloc
      1.54 ± 10%      -0.8        0.76 ± 11%  perf-profile.children.cycles-pp.notify_change
      1.71 ±  8%      -0.8        0.94 ±  8%  perf-profile.children.cycles-pp.strncpy_from_user
      1.35 ± 11%      -0.7        0.67 ± 13%  perf-profile.children.cycles-pp.f2fs_setattr
      1.31 ±  8%      -0.6        0.68 ±  7%  perf-profile.children.cycles-pp.kmem_cache_free
      1.09 ± 11%      -0.6        0.46 ±  8%  perf-profile.children.cycles-pp.common_perm_cond
      1.02 ± 11%      -0.6        0.42 ±  9%  perf-profile.children.cycles-pp.common_perm
      0.87 ±  6%      -0.5        0.41 ± 10%  perf-profile.children.cycles-pp.inode_permission
      1.06 ±  7%      -0.5        0.59 ±  8%  perf-profile.children.cycles-pp.kfree
      0.94 ± 10%      -0.4        0.52 ±  7%  perf-profile.children.cycles-pp.walk_component
      0.65 ±  8%      -0.4        0.29 ±  7%  perf-profile.children.cycles-pp.generic_permission
      0.63 ±  9%      -0.4        0.27 ±  8%  perf-profile.children.cycles-pp.vfs_getattr
      0.62 ±  9%      -0.4        0.27 ±  8%  perf-profile.children.cycles-pp.security_inode_getattr
      0.75 ±  6%      -0.4        0.40 ± 10%  perf-profile.children.cycles-pp.__check_object_size
      0.55 ± 12%      -0.3        0.23 ± 11%  perf-profile.children.cycles-pp.security_path_chmod
      0.68 ± 10%      -0.3        0.36 ±  6%  perf-profile.children.cycles-pp.complete_walk
      0.65 ± 10%      -0.3        0.34 ±  6%  perf-profile.children.cycles-pp.unlazy_walk
      0.62 ±  9%      -0.3        0.33 ±  8%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.46 ± 11%      -0.2        0.21 ± 15%  perf-profile.children.cycles-pp.f2fs_mark_inode_dirty_sync
      0.49 ±  9%      -0.2        0.25 ±  8%  perf-profile.children.cycles-pp.__mod_memcg_lruvec_state
      0.47 ± 15%      -0.2        0.24 ±  9%  perf-profile.children.cycles-pp.get_obj_cgroup_from_current
      0.50 ± 10%      -0.2        0.27 ±  9%  perf-profile.children.cycles-pp.__legitimize_path
      0.52 ± 11%      -0.2        0.29 ±  8%  perf-profile.children.cycles-pp.lookup_fast
      0.49 ±  9%      -0.2        0.27 ±  7%  perf-profile.children.cycles-pp.obj_cgroup_charge
      0.40 ± 11%      -0.2        0.18 ± 15%  perf-profile.children.cycles-pp.f2fs_inode_dirtied
      0.43 ±  9%      -0.2        0.22 ± 10%  perf-profile.children.cycles-pp.path_put
      0.47 ± 10%      -0.2        0.26 ± 11%  perf-profile.children.cycles-pp.refill_obj_stock
      0.42 ±  9%      -0.2        0.21 ±  8%  perf-profile.children.cycles-pp.___might_sleep
      0.32 ± 13%      -0.2        0.11 ± 11%  perf-profile.children.cycles-pp.capable_wrt_inode_uidgid
      0.31 ± 13%      -0.2        0.11 ± 13%  perf-profile.children.cycles-pp.ns_capable_common
      0.31 ± 13%      -0.2        0.11 ± 13%  perf-profile.children.cycles-pp.security_capable
      0.42 ±  9%      -0.2        0.22 ± 11%  perf-profile.children.cycles-pp.dput
      0.42 ± 11%      -0.2        0.21 ±  8%  perf-profile.children.cycles-pp._raw_spin_lock
      0.29 ± 15%      -0.2        0.09 ± 13%  perf-profile.children.cycles-pp.apparmor_capable
      0.38 ±  7%      -0.2        0.20 ±  8%  perf-profile.children.cycles-pp.__entry_text_start
      0.31 ± 28%      -0.2        0.14 ± 18%  perf-profile.children.cycles-pp.revert_creds
      0.32 ± 10%      -0.2        0.14 ±  9%  perf-profile.children.cycles-pp.cp_new_stat
      0.36 ±  7%      -0.2        0.19 ±  8%  perf-profile.children.cycles-pp.__check_heap_object
      0.35 ± 11%      -0.2        0.20 ±  7%  perf-profile.children.cycles-pp.__d_lookup_rcu
      0.34 ±  9%      -0.1        0.20 ± 10%  perf-profile.children.cycles-pp.path_init
      0.26 ± 16%      -0.1        0.12 ± 24%  perf-profile.children.cycles-pp.f2fs_balance_fs
      0.26 ± 12%      -0.1        0.13 ±  7%  perf-profile.children.cycles-pp.__legitimize_mnt
      0.23 ±  9%      -0.1        0.11 ± 10%  perf-profile.children.cycles-pp.__mod_memcg_state
      0.27 ± 11%      -0.1        0.15 ±  8%  perf-profile.children.cycles-pp.__might_sleep
      0.20 ± 13%      -0.1        0.09 ±  7%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
      0.21 ± 11%      -0.1        0.11 ± 16%  perf-profile.children.cycles-pp.step_into
      0.19 ± 11%      -0.1        0.10 ± 14%  perf-profile.children.cycles-pp._cond_resched
      0.18 ±  6%      -0.1        0.09 ± 14%  perf-profile.children.cycles-pp._copy_to_user
      0.20 ± 12%      -0.1        0.11 ± 13%  perf-profile.children.cycles-pp.__might_fault
      0.19 ±  8%      -0.1        0.11 ± 10%  perf-profile.children.cycles-pp.rcu_read_unlock_strict
      0.16 ±  9%      -0.1        0.08 ± 13%  perf-profile.children.cycles-pp.lockref_put_return
      0.19 ± 13%      -0.1        0.11 ±  9%  perf-profile.children.cycles-pp.lockref_get_not_dead
      0.16 ± 18%      -0.1        0.08 ±  9%  perf-profile.children.cycles-pp.memset_erms
      0.14 ±  7%      -0.1        0.07 ± 10%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      0.09 ± 10%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.exit_to_user_mode_prepare
      0.09 ± 14%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.rcu_all_qs
      0.09 ± 10%      -0.1        0.03 ±101%  perf-profile.children.cycles-pp.terminate_walk
      0.12 ±  7%      -0.1        0.06 ± 17%  perf-profile.children.cycles-pp.__put_cred
      0.09 ±  6%      -0.1        0.04 ± 71%  perf-profile.children.cycles-pp.mnt_want_write
      0.10 ± 10%      -0.0        0.06 ±  8%  perf-profile.children.cycles-pp.security_inode_permission
      0.12 ± 13%      -0.0        0.07 ±  9%  perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
      0.09 ± 12%      -0.0        0.05 ± 46%  perf-profile.children.cycles-pp.__fget_light
      0.06 ±  9%      +0.3        0.40 ±  9%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      1.76 ± 29%      +1.3        3.08 ±  8%  perf-profile.children.cycles-pp.menu_select
      0.18 ±  8%      +1.4        1.57 ± 10%  perf-profile.children.cycles-pp.override_creds
      0.00           +19.0       19.04 ±  7%  perf-profile.children.cycles-pp.get_ucounts
      0.00           +19.5       19.54 ±  8%  perf-profile.children.cycles-pp.put_ucounts
      0.07 ± 11%     +34.5       34.55 ±  8%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      0.06 ± 14%     +37.9       37.96 ±  8%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
     16.68 ± 10%     -12.2        4.45 ± 11%  perf-profile.self.cycles-pp.prepare_creds
      7.30 ±  9%      -6.6        0.71 ± 11%  perf-profile.self.cycles-pp.refcount_dec_not_one
      6.87 ± 12%      -5.2        1.71 ± 12%  perf-profile.self.cycles-pp.key_put
      6.25 ±  8%      -3.3        2.90 ±  8%  perf-profile.self.cycles-pp.put_cred_rcu
      3.50 ± 17%      -2.2        1.33 ±  5%  perf-profile.self.cycles-pp.apparmor_cred_prepare
      2.39 ± 11%      -1.3        1.08 ±  5%  perf-profile.self.cycles-pp.apparmor_cred_free
      1.01 ± 11%      -0.6        0.41 ±  8%  perf-profile.self.cycles-pp.common_perm
      1.08 ±  8%      -0.5        0.57 ±  6%  perf-profile.self.cycles-pp.link_path_walk
      1.03 ±  8%      -0.5        0.58 ±  7%  perf-profile.self.cycles-pp.kmem_cache_alloc
      0.92 ±  8%      -0.5        0.46 ±  7%  perf-profile.self.cycles-pp.kmem_cache_free
      0.84 ±  8%      -0.4        0.46 ± 10%  perf-profile.self.cycles-pp.__kmalloc
      0.78 ±  9%      -0.3        0.45 ±  6%  perf-profile.self.cycles-pp.strncpy_from_user
      0.69 ±  7%      -0.3        0.39 ±  8%  perf-profile.self.cycles-pp.kfree
      0.62 ±  9%      -0.3        0.33 ±  8%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.58 ± 11%      -0.3        0.30 ± 12%  perf-profile.self.cycles-pp.f2fs_setattr
      0.47 ± 10%      -0.2        0.26 ±  7%  perf-profile.self.cycles-pp.obj_cgroup_charge
      0.41 ±  9%      -0.2        0.21 ±  7%  perf-profile.self.cycles-pp.___might_sleep
      0.46 ± 11%      -0.2        0.25 ± 10%  perf-profile.self.cycles-pp.refill_obj_stock
      0.29 ± 15%      -0.2        0.09 ± 13%  perf-profile.self.cycles-pp.apparmor_capable
      0.45 ±  7%      -0.2        0.26 ±  5%  perf-profile.self.cycles-pp.do_faccessat
      0.38 ±  7%      -0.2        0.20 ±  8%  perf-profile.self.cycles-pp.__entry_text_start
      0.38 ± 14%      -0.2        0.21 ± 10%  perf-profile.self.cycles-pp.get_obj_cgroup_from_current
      0.31 ± 29%      -0.2        0.14 ± 18%  perf-profile.self.cycles-pp.revert_creds
      0.35 ±  7%      -0.2        0.18 ±  8%  perf-profile.self.cycles-pp.__check_heap_object
      0.35 ± 12%      -0.2        0.20 ±  8%  perf-profile.self.cycles-pp.__d_lookup_rcu
      0.35 ± 12%      -0.1        0.20 ±  9%  perf-profile.self.cycles-pp._raw_spin_lock
      0.31 ±  8%      -0.1        0.17 ± 10%  perf-profile.self.cycles-pp.path_init
      0.29 ±  6%      -0.1        0.16 ±  8%  perf-profile.self.cycles-pp.generic_permission
      0.27 ±  6%      -0.1        0.14 ± 16%  perf-profile.self.cycles-pp.__check_object_size
      0.25 ± 16%      -0.1        0.12 ± 25%  perf-profile.self.cycles-pp.f2fs_balance_fs
      0.25 ± 10%      -0.1        0.13 ±  8%  perf-profile.self.cycles-pp.__legitimize_mnt
      0.23 ±  9%      -0.1        0.11 ± 10%  perf-profile.self.cycles-pp.__mod_memcg_state
      0.25 ± 10%      -0.1        0.13 ±  9%  perf-profile.self.cycles-pp.__might_sleep
      0.24 ±  5%      -0.1        0.13 ± 16%  perf-profile.self.cycles-pp.inode_permission
      0.22 ± 12%      -0.1        0.12 ±  4%  perf-profile.self.cycles-pp.walk_component
      0.20 ± 12%      -0.1        0.10 ± 18%  perf-profile.self.cycles-pp.step_into
      0.22 ±  4%      -0.1        0.12 ±  9%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.23 ± 13%      -0.1        0.13 ±  8%  perf-profile.self.cycles-pp.__mod_memcg_lruvec_state
      0.16 ± 11%      -0.1        0.08 ± 12%  perf-profile.self.cycles-pp.getname_flags
      0.19 ± 13%      -0.1        0.11 ±  9%  perf-profile.self.cycles-pp.lockref_get_not_dead
      0.16 ± 11%      -0.1        0.08 ± 12%  perf-profile.self.cycles-pp.lockref_put_return
      0.15 ± 19%      -0.1        0.08 ± 10%  perf-profile.self.cycles-pp.memset_erms
      0.13 ±  8%      -0.1        0.06 ± 11%  perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      0.16 ± 12%      -0.1        0.09 ±  6%  perf-profile.self.cycles-pp.filename_lookup
      0.12 ± 11%      -0.1        0.06 ±  8%  perf-profile.self.cycles-pp.path_lookupat
      0.15 ± 11%      -0.1        0.09 ± 18%  perf-profile.self.cycles-pp.lookup_fast
      0.15 ± 10%      -0.1        0.09 ±  8%  perf-profile.self.cycles-pp.rcu_read_unlock_strict
      0.11 ±  9%      -0.1        0.06 ± 11%  perf-profile.self.cycles-pp.__put_cred
      0.10 ± 10%      -0.0        0.06 ±  8%  perf-profile.self.cycles-pp.security_inode_permission
      0.12 ± 13%      -0.0        0.07 ± 11%  perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack
      0.09 ± 11%      -0.0        0.04 ± 45%  perf-profile.self.cycles-pp.notify_change
      0.10 ± 17%      -0.0        0.06 ± 14%  perf-profile.self.cycles-pp.__virt_addr_valid
      0.09 ± 12%      -0.0        0.05 ± 46%  perf-profile.self.cycles-pp.__fget_light
      0.00            +0.2        0.16 ± 11%  perf-profile.self.cycles-pp.put_ucounts
      0.00            +0.2        0.17 ± 10%  perf-profile.self.cycles-pp.get_ucounts
      0.06 ± 11%      +0.2        0.23 ±  8%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.84 ± 23%      +0.8        1.69 ±  7%  perf-profile.self.cycles-pp.cpuidle_enter_state
      1.20 ± 31%      +1.2        2.44 ±  8%  perf-profile.self.cycles-pp.menu_select
      0.17 ±  9%      +1.4        1.56 ± 10%  perf-profile.self.cycles-pp.override_creds
      0.06 ± 14%      +3.3        3.41 ±  7%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.07 ± 11%     +34.5       34.55 ±  8%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath





Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Oliver Sang


View attachment "config-5.11.0-rc7-00013-ge1e57d56fef0" of type "text/plain" (172449 bytes)

View attachment "job-script" of type "text/plain" (8497 bytes)

View attachment "job.yaml" of type "text/plain" (5750 bytes)

View attachment "reproduce" of type "text/plain" (534 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ