lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Date:   Thu, 26 Nov 2020 08:12:02 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     Chuck Lever <chuck.lever@...cle.com>
Cc:     0day robot <lkp@...el.com>, Daire Byrne <daire@...g.com>,
        LKML <linux-kernel@...r.kernel.org>, lkp@...ts.01.org,
        ying.huang@...el.com, feng.tang@...el.com, zhengjun.xing@...el.com,
        netdev@...r.kernel.org, linux-nfs@...r.kernel.org
Subject: [SUNRPC] 3bc6a407d1: fsmark.files_per_sec -12.9% regression

Greeting,

FYI, we noticed a -12.9% regression of fsmark.files_per_sec due to commit:


commit: 3bc6a407d1a090417218ecdb0dd4a92a5d75704b ("[PATCH RFC] SUNRPC: Use zero-copy to perform socket send operations")
url: https://github.com/0day-ci/linux/commits/Chuck-Lever/SUNRPC-Use-zero-copy-to-perform-socket-send-operations/20201110-000351
base: git://git.linux-nfs.org/projects/trondmy/linux-nfs.git linux-next

in testcase: fsmark
on test machine: 192 threads Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory
with following parameters:

	iterations: 1x
	nr_threads: 64t
	disk: 1BRD_48G
	fs: xfs
	fs2: nfsv4
	filesize: 4M
	test_size: 40G
	sync_method: fsyncBeforeClose
	cpufreq_governor: performance
	ucode: 0x5003003

test-description: The fsmark is a file system benchmark to test synchronous write workloads, for example, mail servers workload.
test-url: https://sourceforge.net/projects/fsmark/

In addition to that, the commit also has significant impact on the following tests:

+------------------+---------------------------------------------------------------------------+
| testcase: change | fsmark: fsmark.files_per_sec -13.3% regression                            |
| test machine     | 192 threads Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory |
| test parameters  | cpufreq_governor=performance                                              |
|                  | disk=1BRD_48G                                                             |
|                  | filesize=4M                                                               |
|                  | fs2=nfsv4                                                                 |
|                  | fs=xfs                                                                    |
|                  | iterations=1x                                                             |
|                  | nr_threads=64t                                                            |
|                  | sync_method=NoSync                                                        |
|                  | test_size=40G                                                             |
|                  | ucode=0x5003003                                                           |
+------------------+---------------------------------------------------------------------------+


If you fix the issue, kindly add following tag
Reported-by: kernel test robot <rong.a.chen@...el.com>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
compiler/cpufreq_governor/disk/filesize/fs2/fs/iterations/kconfig/nr_threads/rootfs/sync_method/tbox_group/test_size/testcase/ucode:
  gcc-9/performance/1BRD_48G/4M/nfsv4/xfs/1x/x86_64-rhel-8.3/64t/debian-10.4-x86_64-20200603.cgz/fsyncBeforeClose/lkp-csl-2ap2/40G/fsmark/0x5003003

commit: 
  v5.10-rc2
  3bc6a407d1 ("SUNRPC: Use zero-copy to perform socket send operations")

       v5.10-rc2 3bc6a407d1a090417218ecdb0dd 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
  13020954           +12.4%   14633883 ±  3%  fsmark.app_overhead
    558.15           -12.9%     485.88        fsmark.files_per_sec
     21.42           +12.5%      24.10        fsmark.time.elapsed_time
     21.42           +12.5%      24.10        fsmark.time.elapsed_time.max
      1788 ±  5%     -13.3%       1551 ±  4%  fsmark.time.involuntary_context_switches
    245.25           -12.4%     214.75        fsmark.time.percent_of_cpu_this_job_got
   1495010            -3.3%    1446006        fsmark.time.voluntary_context_switches
      2.54 ±  3%     +16.7%       2.97        iostat.cpu.iowait
      3965 ±  7%     +63.2%       6471 ± 44%  softirqs.CPU45.RCU
     12441 ±  3%      +8.1%      13448 ±  4%  meminfo.Active
     12332 ±  3%      +8.2%      13339 ±  4%  meminfo.Active(anon)
     27337 ±  7%     +41.3%      38640 ± 18%  sched_debug.cfs_rq:/.min_vruntime.avg
     14415 ± 19%     +52.1%      21925 ± 14%  sched_debug.cfs_rq:/.min_vruntime.min
   1637052            -8.0%    1505871 ±  2%  vmstat.io.bo
    195349 ±  2%     -12.7%     170450 ±  2%  vmstat.system.cs
      2.74 ±  4%      +0.4        3.18        mpstat.cpu.all.iowait%
      1.09 ±  7%      -0.1        0.94 ±  2%  mpstat.cpu.all.irq%
      0.14 ±  5%      +0.3        0.41 ±  2%  mpstat.cpu.all.soft%
      3.45 ±  2%      -0.5        2.99 ±  2%  mpstat.cpu.all.sys%
      6813 ± 40%     -55.1%       3059 ± 57%  numa-meminfo.node0.PageTables
    119106 ±  4%     -11.5%     105394 ± 11%  numa-meminfo.node0.SUnreclaim
     12837 ± 28%     +30.5%      16748 ± 22%  numa-meminfo.node3.Shmem
    248191            +8.2%     268539 ±  4%  numa-meminfo.node3.Unevictable
  26140871 ±  2%     -13.6%   22585817 ±  4%  cpuidle.C1.time
    432703           -24.4%     327005 ±  2%  cpuidle.C1.usage
 2.216e+09 ± 53%    +108.7%  4.624e+09 ±  3%  cpuidle.C1E.time
   6704163 ± 23%     +53.8%   10312626 ±  3%  cpuidle.C1E.usage
 2.113e+09 ± 57%     -94.9%  1.081e+08 ± 94%  cpuidle.C6.time
      1700 ± 40%     -55.1%     764.25 ± 57%  numa-vmstat.node0.nr_page_table_pages
     29779 ±  4%     -11.5%      26356 ± 11%  numa-vmstat.node0.nr_slab_unreclaimable
      3176 ± 29%     +32.8%       4219 ± 22%  numa-vmstat.node3.nr_shmem
     62047            +8.2%      67134 ±  4%  numa-vmstat.node3.nr_unevictable
     62047            +8.2%      67134 ±  4%  numa-vmstat.node3.nr_zone_unevictable
      3082 ±  3%      +8.2%       3335 ±  4%  proc-vmstat.nr_active_anon
      3082 ±  3%      +8.2%       3335 ±  4%  proc-vmstat.nr_zone_active_anon
     33095 ± 20%     -30.7%      22929 ± 25%  proc-vmstat.numa_pte_updates
  76453330           -12.6%   66854033        proc-vmstat.pgalloc_normal
  55754409           -17.7%   45900156        proc-vmstat.pgfree
     10550            +7.1%      11301 ±  2%  proc-vmstat.pgreuse
     21843           -16.2%      18302        slabinfo.kmalloc-1k.active_objs
    689.50           -15.6%     582.00        slabinfo.kmalloc-1k.active_slabs
     22080           -15.6%      18635        slabinfo.kmalloc-1k.num_objs
    689.50           -15.6%     582.00        slabinfo.kmalloc-1k.num_slabs
     12057 ±  7%     -12.6%      10544 ±  4%  slabinfo.nfsd_file.active_objs
     12250 ±  6%     -12.0%      10779 ±  4%  slabinfo.nfsd_file.num_objs
 3.726e+09            -6.0%  3.502e+09        perf-stat.i.branch-instructions
 3.236e+08 ±  2%      -8.1%  2.974e+08        perf-stat.i.cache-references
    223562 ±  3%     -13.9%     192428 ±  2%  perf-stat.i.context-switches
 4.738e+09            -6.5%  4.428e+09        perf-stat.i.dTLB-loads
 2.344e+09 ±  2%      -6.8%  2.185e+09        perf-stat.i.dTLB-stores
 1.885e+10            -5.8%  1.776e+10        perf-stat.i.instructions
     58.38            -6.5%      54.61        perf-stat.i.metric.M/sec
      9172 ±  3%      -7.8%       8458        perf-stat.i.minor-faults
  37419450 ±  3%      -9.0%   34056149 ±  2%  perf-stat.i.node-load-misses
  10313099 ±  2%     -11.4%    9134750 ±  5%  perf-stat.i.node-loads
      9181 ±  3%      -7.8%       8467        perf-stat.i.page-faults
 3.565e+09            -5.6%  3.365e+09        perf-stat.ps.branch-instructions
 3.098e+08 ±  2%      -7.7%  2.858e+08        perf-stat.ps.cache-references
    214060 ±  3%     -13.6%     184935 ±  2%  perf-stat.ps.context-switches
 4.534e+09            -6.2%  4.255e+09        perf-stat.ps.dTLB-loads
 2.243e+09            -6.4%  2.099e+09        perf-stat.ps.dTLB-stores
 1.803e+10            -5.4%  1.706e+10        perf-stat.ps.instructions
      8793 ±  3%      -7.2%       8164 ±  2%  perf-stat.ps.minor-faults
  35826191 ±  3%      -8.7%   32716417 ±  2%  perf-stat.ps.node-load-misses
   9874765 ±  2%     -11.0%    8786772 ±  5%  perf-stat.ps.node-loads
      8802 ±  3%      -7.2%       8172 ±  2%  perf-stat.ps.page-faults
 4.219e+11            +3.9%  4.384e+11        perf-stat.total.instructions
      1283 ± 44%     -41.5%     750.50 ± 16%  interrupts.CPU100.NMI:Non-maskable_interrupts
      1283 ± 44%     -41.5%     750.50 ± 16%  interrupts.CPU100.PMI:Performance_monitoring_interrupts
    603.75 ± 19%     +82.7%       1103 ± 33%  interrupts.CPU105.NMI:Non-maskable_interrupts
    603.75 ± 19%     +82.7%       1103 ± 33%  interrupts.CPU105.PMI:Performance_monitoring_interrupts
     83.00 ± 15%     -43.1%      47.25 ± 29%  interrupts.CPU115.RES:Rescheduling_interrupts
      2540 ± 11%     -24.4%       1919 ± 24%  interrupts.CPU144.CAL:Function_call_interrupts
      2703 ± 14%     -40.1%       1619 ± 29%  interrupts.CPU152.CAL:Function_call_interrupts
      2791 ± 10%     -31.7%       1905 ± 28%  interrupts.CPU156.CAL:Function_call_interrupts
      2660 ±  6%     -37.7%       1657 ± 32%  interrupts.CPU163.CAL:Function_call_interrupts
      2.50 ± 72%   +4810.0%     122.75 ±164%  interrupts.CPU163.TLB:TLB_shootdowns
      2499 ± 11%     -34.5%       1636 ± 35%  interrupts.CPU167.CAL:Function_call_interrupts
    980.75 ± 18%     -36.3%     624.50 ± 17%  interrupts.CPU172.NMI:Non-maskable_interrupts
    980.75 ± 18%     -36.3%     624.50 ± 17%  interrupts.CPU172.PMI:Performance_monitoring_interrupts
      2671 ± 14%     -24.8%       2009 ± 19%  interrupts.CPU173.CAL:Function_call_interrupts
      2.25 ± 57%   +5244.4%     120.25 ±164%  interrupts.CPU173.TLB:TLB_shootdowns
      1048 ± 17%     -40.2%     627.25 ± 35%  interrupts.CPU175.NMI:Non-maskable_interrupts
      1048 ± 17%     -40.2%     627.25 ± 35%  interrupts.CPU175.PMI:Performance_monitoring_interrupts
    848.25 ± 24%     -39.9%     509.75 ± 20%  interrupts.CPU176.NMI:Non-maskable_interrupts
    848.25 ± 24%     -39.9%     509.75 ± 20%  interrupts.CPU176.PMI:Performance_monitoring_interrupts
      3.50 ± 58%   +3421.4%     123.25 ±157%  interrupts.CPU178.TLB:TLB_shootdowns
      2913 ±  4%     -29.5%       2055 ± 19%  interrupts.CPU184.CAL:Function_call_interrupts
    862.00 ± 20%     -34.3%     566.75 ± 14%  interrupts.CPU184.NMI:Non-maskable_interrupts
    862.00 ± 20%     -34.3%     566.75 ± 14%  interrupts.CPU184.PMI:Performance_monitoring_interrupts
    931.25 ± 34%     -45.2%     510.25 ± 20%  interrupts.CPU186.NMI:Non-maskable_interrupts
    931.25 ± 34%     -45.2%     510.25 ± 20%  interrupts.CPU186.PMI:Performance_monitoring_interrupts
    840.25 ± 17%     -27.8%     606.75 ± 20%  interrupts.CPU21.NMI:Non-maskable_interrupts
    840.25 ± 17%     -27.8%     606.75 ± 20%  interrupts.CPU21.PMI:Performance_monitoring_interrupts
      2851 ± 16%     -25.1%       2135 ± 14%  interrupts.CPU29.CAL:Function_call_interrupts
    560.75 ± 13%     +67.8%     941.00 ± 17%  interrupts.CPU30.NMI:Non-maskable_interrupts
    560.75 ± 13%     +67.8%     941.00 ± 17%  interrupts.CPU30.PMI:Performance_monitoring_interrupts
    615.25 ± 16%     +44.9%     891.75 ± 16%  interrupts.CPU31.NMI:Non-maskable_interrupts
    615.25 ± 16%     +44.9%     891.75 ± 16%  interrupts.CPU31.PMI:Performance_monitoring_interrupts
     34.25 ± 28%    +156.2%      87.75 ± 26%  interrupts.CPU34.RES:Rescheduling_interrupts
      1.50 ±137%   +7700.0%     117.00 ±166%  interrupts.CPU39.TLB:TLB_shootdowns
      2394 ±  4%     +18.6%       2840 ±  9%  interrupts.CPU41.CAL:Function_call_interrupts
    610.75 ± 21%     -41.3%     358.75 ± 33%  interrupts.CPU44.NMI:Non-maskable_interrupts
    610.75 ± 21%     -41.3%     358.75 ± 33%  interrupts.CPU44.PMI:Performance_monitoring_interrupts
      2608 ±  4%     -36.5%       1655 ± 35%  interrupts.CPU48.CAL:Function_call_interrupts
      2820 ±  8%     -34.7%       1841 ± 32%  interrupts.CPU55.CAL:Function_call_interrupts
      1249 ± 47%     -49.6%     630.00 ± 44%  interrupts.CPU56.NMI:Non-maskable_interrupts
      1249 ± 47%     -49.6%     630.00 ± 44%  interrupts.CPU56.PMI:Performance_monitoring_interrupts
    609.50 ± 29%    +167.1%       1628 ± 78%  interrupts.CPU67.NMI:Non-maskable_interrupts
    609.50 ± 29%    +167.1%       1628 ± 78%  interrupts.CPU67.PMI:Performance_monitoring_interrupts
      3176 ± 31%     -35.7%       2043 ± 31%  interrupts.CPU72.CAL:Function_call_interrupts
      2671 ±  7%     -40.8%       1582 ± 31%  interrupts.CPU73.CAL:Function_call_interrupts
      2626 ±  5%     -32.6%       1770 ± 22%  interrupts.CPU74.CAL:Function_call_interrupts
      2674 ±  6%     -35.1%       1735 ± 22%  interrupts.CPU77.CAL:Function_call_interrupts
    966.50 ± 17%     -49.0%     492.75 ± 18%  interrupts.CPU80.NMI:Non-maskable_interrupts
    966.50 ± 17%     -49.0%     492.75 ± 18%  interrupts.CPU80.PMI:Performance_monitoring_interrupts
      2659 ± 19%     -29.4%       1878 ± 26%  interrupts.CPU84.CAL:Function_call_interrupts
    824.00 ± 11%     -33.1%     551.25 ± 21%  interrupts.CPU88.NMI:Non-maskable_interrupts
    824.00 ± 11%     -33.1%     551.25 ± 21%  interrupts.CPU88.PMI:Performance_monitoring_interrupts
      1012 ± 39%     -41.9%     588.00 ± 37%  interrupts.CPU98.NMI:Non-maskable_interrupts
      1012 ± 39%     -41.9%     588.00 ± 37%  interrupts.CPU98.PMI:Performance_monitoring_interrupts
     32.98 ±  3%      -3.4       29.60        perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
     32.98 ±  3%      -3.4       29.61        perf-profile.calltrace.cycles-pp.ret_from_fork
     21.71 ±  3%      -2.5       19.25 ±  2%  perf-profile.calltrace.cycles-pp.nfsd.kthread.ret_from_fork
     16.07 ±  3%      -2.2       13.84        perf-profile.calltrace.cycles-pp.svc_process.nfsd.kthread.ret_from_fork
     16.07 ±  3%      -2.2       13.84        perf-profile.calltrace.cycles-pp.svc_process_common.svc_process.nfsd.kthread.ret_from_fork
     16.01 ±  3%      -2.2       13.78        perf-profile.calltrace.cycles-pp.nfsd_dispatch.svc_process_common.svc_process.nfsd.kthread
     15.96 ±  3%      -2.2       13.74        perf-profile.calltrace.cycles-pp.nfsd4_proc_compound.nfsd_dispatch.svc_process_common.svc_process.nfsd
     16.78 ±  4%      -1.6       15.17 ±  2%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
     16.68 ±  4%      -1.6       15.08        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
      9.34 ±  3%      -1.3        8.09 ±  2%  perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      9.41 ±  3%      -1.2        8.16 ±  2%  perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      9.39 ±  3%      -1.2        8.14 ±  2%  perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      9.33 ±  3%      -1.2        8.09 ±  2%  perf-profile.calltrace.cycles-pp.nfs_file_write.new_sync_write.vfs_write.ksys_write.do_syscall_64
      9.01 ±  3%      -1.2        7.80 ±  2%  perf-profile.calltrace.cycles-pp.generic_perform_write.nfs_file_write.new_sync_write.vfs_write.ksys_write
      9.75 ±  3%      -1.1        8.66        perf-profile.calltrace.cycles-pp.nfsd_commit.nfsd4_proc_compound.nfsd_dispatch.svc_process_common.svc_process
      9.73 ±  3%      -1.1        8.64        perf-profile.calltrace.cycles-pp.xfs_file_fsync.nfsd_commit.nfsd4_proc_compound.nfsd_dispatch.svc_process_common
      9.68 ±  3%      -1.1        8.59        perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync.nfsd_commit.nfsd4_proc_compound
      9.68 ±  3%      -1.1        8.59        perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync.nfsd_commit
      9.68 ±  3%      -1.1        8.59        perf-profile.calltrace.cycles-pp.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync
      9.69 ±  3%      -1.1        8.61        perf-profile.calltrace.cycles-pp.file_write_and_wait_range.xfs_file_fsync.nfsd_commit.nfsd4_proc_compound.nfsd_dispatch
      5.84 ±  4%      -1.1        4.78 ±  2%  perf-profile.calltrace.cycles-pp.nfsd4_write.nfsd4_proc_compound.nfsd_dispatch.svc_process_common.svc_process
      5.81 ±  4%      -1.1        4.75 ±  2%  perf-profile.calltrace.cycles-pp.nfsd_vfs_write.nfsd4_write.nfsd4_proc_compound.nfsd_dispatch.svc_process_common
      5.80 ±  4%      -1.1        4.75 ±  2%  perf-profile.calltrace.cycles-pp.do_iter_write.nfsd_vfs_write.nfsd4_write.nfsd4_proc_compound.nfsd_dispatch
      5.79 ±  4%      -1.0        4.74 ±  2%  perf-profile.calltrace.cycles-pp.do_iter_readv_writev.do_iter_write.nfsd_vfs_write.nfsd4_write.nfsd4_proc_compound
      5.79 ±  4%      -1.0        4.74 ±  2%  perf-profile.calltrace.cycles-pp.xfs_file_buffered_aio_write.do_iter_readv_writev.do_iter_write.nfsd_vfs_write.nfsd4_write
      5.54 ±  4%      -0.9        4.59 ±  2%  perf-profile.calltrace.cycles-pp.iomap_file_buffered_write.xfs_file_buffered_aio_write.do_iter_readv_writev.do_iter_write.nfsd_vfs_write
      5.54 ±  4%      -0.9        4.59 ±  2%  perf-profile.calltrace.cycles-pp.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.do_iter_readv_writev.do_iter_write
      5.51 ±  4%      -0.9        4.57 ±  2%  perf-profile.calltrace.cycles-pp.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.do_iter_readv_writev
     11.19 ±  3%      -0.9       10.29 ±  2%  perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork
     10.69 ±  3%      -0.8        9.86 ±  2%  perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork
      1.78 ±  6%      -0.7        1.09 ±  3%  perf-profile.calltrace.cycles-pp.iomap_writepages.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range
      1.77 ±  6%      -0.7        1.08 ±  3%  perf-profile.calltrace.cycles-pp.write_cache_pages.iomap_writepages.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range
      5.28 ±  3%      -0.6        4.64 ±  2%  perf-profile.calltrace.cycles-pp.__x64_sys_fsync.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.28 ±  3%      -0.6        4.64 ±  2%  perf-profile.calltrace.cycles-pp.do_fsync.__x64_sys_fsync.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.28 ±  3%      -0.6        4.64 ±  2%  perf-profile.calltrace.cycles-pp.nfs_file_fsync.do_fsync.__x64_sys_fsync.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.23 ±  7%      -0.6        0.62 ±  4%  perf-profile.calltrace.cycles-pp.iomap_writepage_map.write_cache_pages.iomap_writepages.xfs_vm_writepages.do_writepages
      4.30 ±  4%      -0.6        3.69 ±  3%  perf-profile.calltrace.cycles-pp.rpc_free_task.rpc_async_release.process_one_work.worker_thread.kthread
      4.30 ±  4%      -0.6        3.69 ±  3%  perf-profile.calltrace.cycles-pp.rpc_async_release.process_one_work.worker_thread.kthread.ret_from_fork
      4.37 ±  4%      -0.6        3.75 ±  5%  perf-profile.calltrace.cycles-pp.nfs_write_end.generic_perform_write.nfs_file_write.new_sync_write.vfs_write
      4.75 ±  3%      -0.6        4.19 ±  2%  perf-profile.calltrace.cycles-pp.file_write_and_wait_range.nfs_file_fsync.do_fsync.__x64_sys_fsync.do_syscall_64
      3.42 ±  5%      -0.6        2.86 ±  3%  perf-profile.calltrace.cycles-pp.nfs_write_completion.rpc_free_task.rpc_async_release.process_one_work.worker_thread
      2.69            -0.5        2.21 ±  3%  perf-profile.calltrace.cycles-pp.iov_iter_copy_from_user_atomic.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
      2.65            -0.5        2.18 ±  3%  perf-profile.calltrace.cycles-pp.memcpy_erms.iov_iter_copy_from_user_atomic.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      3.16 ±  5%      -0.4        2.73 ±  7%  perf-profile.calltrace.cycles-pp.nfs_updatepage.nfs_write_end.generic_perform_write.nfs_file_write.new_sync_write
      2.34 ±  5%      -0.4        1.92 ±  4%  perf-profile.calltrace.cycles-pp.nfs_end_page_writeback.nfs_write_completion.rpc_free_task.rpc_async_release.process_one_work
      3.59 ±  3%      -0.4        3.19 ±  2%  perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.file_write_and_wait_range.nfs_file_fsync.do_fsync.__x64_sys_fsync
      3.59 ±  3%      -0.4        3.19 ±  2%  perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range.nfs_file_fsync.do_fsync
      3.59 ±  3%      -0.4        3.19 ±  2%  perf-profile.calltrace.cycles-pp.nfs_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range.nfs_file_fsync
      3.56 ±  3%      -0.4        3.16 ±  2%  perf-profile.calltrace.cycles-pp.write_cache_pages.nfs_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range
      7.90 ±  4%      -0.4        7.51        perf-profile.calltrace.cycles-pp.submit_bio.iomap_submit_ioend.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range
      7.90 ±  4%      -0.4        7.51        perf-profile.calltrace.cycles-pp.iomap_submit_ioend.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range
      2.96 ±  4%      -0.3        2.63 ±  2%  perf-profile.calltrace.cycles-pp.nfs_writepages_callback.write_cache_pages.nfs_writepages.do_writepages.__filemap_fdatawrite_range
      2.23 ±  3%      -0.3        1.91 ±  2%  perf-profile.calltrace.cycles-pp.iov_iter_copy_from_user_atomic.generic_perform_write.nfs_file_write.new_sync_write.vfs_write
      0.58 ±  3%      -0.3        0.27 ±100%  perf-profile.calltrace.cycles-pp.mem_cgroup_charge.__add_to_page_cache_locked.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin
      2.53            -0.3        2.21 ±  4%  perf-profile.calltrace.cycles-pp.memcpy_to_page._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg
      2.52 ±  2%      -0.3        2.20 ±  4%  perf-profile.calltrace.cycles-pp.memcpy_erms.memcpy_to_page._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter
      2.92 ±  4%      -0.3        2.61 ±  2%  perf-profile.calltrace.cycles-pp.nfs_do_writepage.nfs_writepages_callback.write_cache_pages.nfs_writepages.do_writepages
      2.18 ±  3%      -0.3        1.88 ±  2%  perf-profile.calltrace.cycles-pp.copyin.iov_iter_copy_from_user_atomic.generic_perform_write.nfs_file_write.new_sync_write
      2.17 ±  3%      -0.3        1.86 ±  2%  perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyin.iov_iter_copy_from_user_atomic.generic_perform_write.nfs_file_write
      0.55 ±  4%      -0.3        0.25 ±100%  perf-profile.calltrace.cycles-pp.__test_set_page_writeback.nfs_do_writepage.nfs_writepages_callback.write_cache_pages.nfs_writepages
      2.55 ±  2%      -0.3        2.25 ±  3%  perf-profile.calltrace.cycles-pp._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg.inet6_recvmsg
      2.23 ±  4%      -0.3        1.95 ±  7%  perf-profile.calltrace.cycles-pp.nfs_writepage_setup.nfs_updatepage.nfs_write_end.generic_perform_write.nfs_file_write
      2.07 ±  3%      -0.3        1.82 ±  2%  perf-profile.calltrace.cycles-pp.grab_cache_page_write_begin.nfs_write_begin.generic_perform_write.nfs_file_write.new_sync_write
      2.13 ±  3%      -0.2        1.88 ±  2%  perf-profile.calltrace.cycles-pp.nfs_write_begin.generic_perform_write.nfs_file_write.new_sync_write.vfs_write
      2.05 ±  3%      -0.2        1.80 ±  2%  perf-profile.calltrace.cycles-pp.pagecache_get_page.grab_cache_page_write_begin.nfs_write_begin.generic_perform_write.nfs_file_write
      0.66 ±  3%      -0.2        0.42 ± 57%  perf-profile.calltrace.cycles-pp.io_schedule.wait_on_page_bit.__filemap_fdatawait_range.file_write_and_wait_range.nfs_file_fsync
      0.62 ±  4%      -0.2        0.39 ± 57%  perf-profile.calltrace.cycles-pp.try_to_wake_up.wake_page_function.__wake_up_common.wake_up_page_bit.nfs_end_page_writeback
      1.25 ±  7%      -0.2        1.02 ±  2%  perf-profile.calltrace.cycles-pp.iomap_write_end.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
      0.64 ±  3%      -0.2        0.41 ± 57%  perf-profile.calltrace.cycles-pp.schedule.io_schedule.wait_on_page_bit.__filemap_fdatawait_range.file_write_and_wait_range
      0.62 ±  3%      -0.2        0.40 ± 57%  perf-profile.calltrace.cycles-pp.__schedule.schedule.io_schedule.wait_on_page_bit.__filemap_fdatawait_range
      1.40 ±  6%      -0.2        1.20 ±  3%  perf-profile.calltrace.cycles-pp.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
      1.33 ±  6%      -0.2        1.13 ±  3%  perf-profile.calltrace.cycles-pp.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      1.32 ±  6%      -0.2        1.12 ±  3%  perf-profile.calltrace.cycles-pp.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor.iomap_apply
      1.17 ±  5%      -0.2        1.00        perf-profile.calltrace.cycles-pp.__filemap_fdatawait_range.file_write_and_wait_range.nfs_file_fsync.do_fsync.__x64_sys_fsync
      1.48 ±  3%      -0.2        1.31 ±  2%  perf-profile.calltrace.cycles-pp.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.nfs_write_begin.generic_perform_write
      0.80 ±  4%      -0.1        0.66 ±  3%  perf-profile.calltrace.cycles-pp.wake_up_page_bit.nfs_end_page_writeback.nfs_write_completion.rpc_free_task.rpc_async_release
      0.72 ±  5%      -0.1        0.58 ±  4%  perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      0.89 ±  7%      -0.1        0.76 ±  6%  perf-profile.calltrace.cycles-pp.end_page_writeback.nfs_end_page_writeback.nfs_write_completion.rpc_free_task.rpc_async_release
      1.04 ±  3%      -0.1        0.91 ±  3%  perf-profile.calltrace.cycles-pp.__add_to_page_cache_locked.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.nfs_write_begin
      0.80 ±  7%      -0.1        0.67 ±  3%  perf-profile.calltrace.cycles-pp.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor
      0.85 ±  8%      -0.1        0.73 ±  6%  perf-profile.calltrace.cycles-pp.test_clear_page_writeback.end_page_writeback.nfs_end_page_writeback.nfs_write_completion.rpc_free_task
      0.74 ±  4%      -0.1        0.62 ±  4%  perf-profile.calltrace.cycles-pp.wait_on_page_bit.__filemap_fdatawait_range.file_write_and_wait_range.nfs_file_fsync.do_fsync
      0.67 ±  4%      -0.1        0.55 ±  4%  perf-profile.calltrace.cycles-pp.__wake_up_common.wake_up_page_bit.nfs_end_page_writeback.nfs_write_completion.rpc_free_task
      0.86 ±  4%      -0.1        0.75 ±  5%  perf-profile.calltrace.cycles-pp.__set_page_dirty_nobuffers.nfs_writepage_setup.nfs_updatepage.nfs_write_end.generic_perform_write
      0.63 ±  4%      -0.1        0.52 ±  3%  perf-profile.calltrace.cycles-pp.wake_page_function.__wake_up_common.wake_up_page_bit.nfs_end_page_writeback.nfs_write_completion
      0.66 ±  3%      -0.1        0.56 ±  7%  perf-profile.calltrace.cycles-pp.iomap_set_range_uptodate.iomap_write_end.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      0.61 ±  5%      -0.1        0.56 ±  5%  perf-profile.calltrace.cycles-pp.xfs_end_io.process_one_work.worker_thread.kthread.ret_from_fork
      0.61 ±  5%      -0.1        0.56 ±  5%  perf-profile.calltrace.cycles-pp.xfs_end_ioend.xfs_end_io.process_one_work.worker_thread.kthread
      0.59 ±  6%      -0.1        0.54 ±  5%  perf-profile.calltrace.cycles-pp.iomap_finish_ioends.xfs_end_ioend.xfs_end_io.process_one_work.worker_thread
      0.83 ±  2%      -0.1        0.77        perf-profile.calltrace.cycles-pp.__alloc_pages_nodemask.brd_insert_page.brd_do_bvec.brd_submit_bio.submit_bio_noacct
      0.77 ±  2%      -0.0        0.72        perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages_nodemask.brd_insert_page.brd_do_bvec.brd_submit_bio
      3.13 ±  2%      +0.4        3.50 ±  4%  perf-profile.calltrace.cycles-pp.inet6_recvmsg.svc_tcp_recvfrom.svc_handle_xprt.svc_recv.nfsd
      3.12 ±  2%      +0.4        3.50 ±  4%  perf-profile.calltrace.cycles-pp.tcp_recvmsg.inet6_recvmsg.svc_tcp_recvfrom.svc_handle_xprt.svc_recv
      0.00            +0.6        0.61 ±  6%  perf-profile.calltrace.cycles-pp.__kfree_skb.tcp_recvmsg.inet6_recvmsg.svc_tcp_recvfrom.svc_handle_xprt
      0.00            +3.0        2.98        perf-profile.calltrace.cycles-pp.memcpy_erms.skb_copy_ubufs.__netif_receive_skb_core.__netif_receive_skb_one_core.process_backlog
      0.00            +3.7        3.71        perf-profile.calltrace.cycles-pp.skb_copy_ubufs.__netif_receive_skb_core.__netif_receive_skb_one_core.process_backlog.net_rx_action
      0.00            +3.7        3.72        perf-profile.calltrace.cycles-pp.__netif_receive_skb_core.__netif_receive_skb_one_core.process_backlog.net_rx_action.__softirqentry_text_start
      0.00            +3.9        3.88 ±  2%  perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.net_rx_action.__softirqentry_text_start.asm_call_sysvec_on_stack
      0.00            +3.9        3.89 ±  2%  perf-profile.calltrace.cycles-pp.process_backlog.net_rx_action.__softirqentry_text_start.asm_call_sysvec_on_stack.do_softirq_own_stack
      0.00            +3.9        3.89 ±  2%  perf-profile.calltrace.cycles-pp.net_rx_action.__softirqentry_text_start.asm_call_sysvec_on_stack.do_softirq_own_stack.do_softirq
      0.00            +3.9        3.90 ±  2%  perf-profile.calltrace.cycles-pp.asm_call_sysvec_on_stack.do_softirq_own_stack.do_softirq.__local_bh_enable_ip.ip6_finish_output2
      0.00            +3.9        3.90 ±  2%  perf-profile.calltrace.cycles-pp.do_softirq_own_stack.do_softirq.__local_bh_enable_ip.ip6_finish_output2.ip6_output
      0.00            +3.9        3.90 ±  2%  perf-profile.calltrace.cycles-pp.__softirqentry_text_start.asm_call_sysvec_on_stack.do_softirq_own_stack.do_softirq.__local_bh_enable_ip
      0.00            +3.9        3.90 ±  2%  perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.ip6_finish_output2.ip6_output.ip6_xmit.inet6_csk_xmit
      0.00            +3.9        3.90 ±  2%  perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.ip6_finish_output2.ip6_output.ip6_xmit
      0.00            +3.9        3.94 ±  2%  perf-profile.calltrace.cycles-pp.ip6_finish_output2.ip6_output.ip6_xmit.inet6_csk_xmit.__tcp_transmit_skb
      0.00            +4.0        3.96        perf-profile.calltrace.cycles-pp.ip6_output.ip6_xmit.inet6_csk_xmit.__tcp_transmit_skb.tcp_write_xmit
      0.00            +4.0        3.96        perf-profile.calltrace.cycles-pp.ip6_xmit.inet6_csk_xmit.__tcp_transmit_skb.tcp_write_xmit.tcp_sendmsg_locked
      0.00            +4.0        3.99        perf-profile.calltrace.cycles-pp.inet6_csk_xmit.__tcp_transmit_skb.tcp_write_xmit.tcp_sendmsg_locked.tcp_sendmsg
      0.00            +4.0        4.03        perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg
      0.00            +4.1        4.09        perf-profile.calltrace.cycles-pp.tcp_write_xmit.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.xprt_sock_sendmsg
     42.90 ±  5%      +4.1       47.02        perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
     43.95 ±  4%      +4.2       48.13        perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     49.22 ±  2%      +4.9       54.12        perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     49.23 ±  2%      +4.9       54.14        perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
     49.22 ±  2%      +4.9       54.13        perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     49.50 ±  2%      +5.0       54.45        perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
     32.98 ±  3%      -3.4       29.60        perf-profile.children.cycles-pp.kthread
     32.98 ±  3%      -3.4       29.61        perf-profile.children.cycles-pp.ret_from_fork
     21.71 ±  3%      -2.5       19.25 ±  2%  perf-profile.children.cycles-pp.nfsd
     16.07 ±  3%      -2.2       13.84        perf-profile.children.cycles-pp.svc_process
     16.07 ±  3%      -2.2       13.84        perf-profile.children.cycles-pp.svc_process_common
     16.01 ±  3%      -2.2       13.78        perf-profile.children.cycles-pp.nfsd_dispatch
     15.96 ±  3%      -2.2       13.74        perf-profile.children.cycles-pp.nfsd4_proc_compound
     14.45 ±  2%      -1.7       12.79        perf-profile.children.cycles-pp.file_write_and_wait_range
     17.18 ±  4%      -1.6       15.62 ±  2%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     17.07 ±  4%      -1.5       15.53 ±  2%  perf-profile.children.cycles-pp.do_syscall_64
     13.26 ±  2%      -1.5       11.78        perf-profile.children.cycles-pp.__filemap_fdatawrite_range
     13.26 ±  2%      -1.5       11.78        perf-profile.children.cycles-pp.do_writepages
      9.34 ±  3%      -1.3        8.09 ±  2%  perf-profile.children.cycles-pp.nfs_file_write
      9.60 ±  3%      -1.2        8.37 ±  2%  perf-profile.children.cycles-pp.new_sync_write
      9.66 ±  3%      -1.2        8.44 ±  2%  perf-profile.children.cycles-pp.ksys_write
      9.64 ±  3%      -1.2        8.43 ±  2%  perf-profile.children.cycles-pp.vfs_write
      9.01 ±  3%      -1.2        7.81 ±  2%  perf-profile.children.cycles-pp.generic_perform_write
      9.75 ±  3%      -1.1        8.66        perf-profile.children.cycles-pp.nfsd_commit
      9.73 ±  3%      -1.1        8.64        perf-profile.children.cycles-pp.xfs_file_fsync
      5.34 ±  2%      -1.1        4.25 ±  2%  perf-profile.children.cycles-pp.write_cache_pages
      9.68 ±  3%      -1.1        8.59        perf-profile.children.cycles-pp.xfs_vm_writepages
      5.84 ±  4%      -1.1        4.78 ±  2%  perf-profile.children.cycles-pp.nfsd4_write
      5.81 ±  4%      -1.1        4.75 ±  2%  perf-profile.children.cycles-pp.nfsd_vfs_write
      5.80 ±  4%      -1.1        4.75 ±  2%  perf-profile.children.cycles-pp.do_iter_write
      5.79 ±  4%      -1.0        4.74 ±  2%  perf-profile.children.cycles-pp.do_iter_readv_writev
      5.79 ±  4%      -1.0        4.74 ±  2%  perf-profile.children.cycles-pp.xfs_file_buffered_aio_write
      5.54 ±  4%      -0.9        4.59 ±  2%  perf-profile.children.cycles-pp.iomap_file_buffered_write
      5.54 ±  4%      -0.9        4.59 ±  2%  perf-profile.children.cycles-pp.iomap_apply
      5.52 ±  3%      -0.9        4.57 ±  2%  perf-profile.children.cycles-pp.iomap_write_actor
      8.46 ±  3%      -0.9        7.56        perf-profile.children.cycles-pp.submit_bio_noacct
      8.46 ±  3%      -0.9        7.56        perf-profile.children.cycles-pp.brd_submit_bio
     11.19 ±  3%      -0.9       10.29 ±  2%  perf-profile.children.cycles-pp.worker_thread
      8.46 ±  3%      -0.9        7.56        perf-profile.children.cycles-pp.submit_bio
      8.42 ±  3%      -0.9        7.53        perf-profile.children.cycles-pp.brd_do_bvec
      8.40 ±  3%      -0.9        7.51        perf-profile.children.cycles-pp.iomap_submit_ioend
     10.69 ±  3%      -0.8        9.86 ±  2%  perf-profile.children.cycles-pp.process_one_work
      4.91 ±  2%      -0.8        4.12 ±  2%  perf-profile.children.cycles-pp.iov_iter_copy_from_user_atomic
      1.78 ±  6%      -0.7        1.09 ±  3%  perf-profile.children.cycles-pp.iomap_writepages
      4.50 ±  4%      -0.6        3.85 ±  3%  perf-profile.children.cycles-pp.rpc_free_task
      5.28 ±  3%      -0.6        4.64 ±  2%  perf-profile.children.cycles-pp.__x64_sys_fsync
      5.28 ±  3%      -0.6        4.64 ±  2%  perf-profile.children.cycles-pp.do_fsync
      5.28 ±  3%      -0.6        4.64 ±  2%  perf-profile.children.cycles-pp.nfs_file_fsync
      1.23 ±  7%      -0.6        0.62 ±  4%  perf-profile.children.cycles-pp.iomap_writepage_map
      4.30 ±  4%      -0.6        3.69 ±  3%  perf-profile.children.cycles-pp.rpc_async_release
      4.37 ±  4%      -0.6        3.76 ±  5%  perf-profile.children.cycles-pp.nfs_write_end
      3.42 ±  5%      -0.6        2.87 ±  3%  perf-profile.children.cycles-pp.nfs_write_completion
      5.67 ±  4%      -0.6        5.12 ±  2%  perf-profile.children.cycles-pp.brd_insert_page
      5.11 ±  4%      -0.5        4.66 ±  3%  perf-profile.children.cycles-pp._raw_spin_lock
      3.40 ±  4%      -0.5        2.94        perf-profile.children.cycles-pp.grab_cache_page_write_begin
      3.37 ±  4%      -0.4        2.92        perf-profile.children.cycles-pp.pagecache_get_page
      3.16 ±  4%      -0.4        2.73 ±  7%  perf-profile.children.cycles-pp.nfs_updatepage
      2.34 ±  6%      -0.4        1.92 ±  4%  perf-profile.children.cycles-pp.nfs_end_page_writeback
      3.59 ±  3%      -0.4        3.19 ±  2%  perf-profile.children.cycles-pp.nfs_writepages
      1.84 ±  2%      -0.3        1.52 ±  4%  perf-profile.children.cycles-pp.__schedule
      2.96 ±  4%      -0.3        2.63 ±  2%  perf-profile.children.cycles-pp.nfs_writepages_callback
      2.53            -0.3        2.21 ±  4%  perf-profile.children.cycles-pp.memcpy_to_page
      2.93 ±  4%      -0.3        2.62 ±  2%  perf-profile.children.cycles-pp.nfs_do_writepage
      2.18 ±  3%      -0.3        1.88 ±  2%  perf-profile.children.cycles-pp.copyin
      2.29 ±  4%      -0.3        1.98        perf-profile.children.cycles-pp.add_to_page_cache_lru
      2.18 ±  3%      -0.3        1.88 ±  2%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      2.56 ±  2%      -0.3        2.26 ±  3%  perf-profile.children.cycles-pp._copy_to_iter
      2.24 ±  4%      -0.3        1.95 ±  6%  perf-profile.children.cycles-pp.nfs_writepage_setup
      2.14 ±  3%      -0.3        1.89 ±  2%  perf-profile.children.cycles-pp.nfs_write_begin
      1.49 ±  2%      -0.2        1.24 ±  5%  perf-profile.children.cycles-pp.schedule
      1.25 ±  7%      -0.2        1.03 ±  2%  perf-profile.children.cycles-pp.iomap_write_end
      1.53 ±  5%      -0.2        1.31 ±  2%  perf-profile.children.cycles-pp.__add_to_page_cache_locked
      1.41 ±  5%      -0.2        1.20 ±  3%  perf-profile.children.cycles-pp.iomap_write_begin
      1.17 ±  3%      -0.2        0.97 ±  2%  perf-profile.children.cycles-pp.try_to_wake_up
      1.07            -0.2        0.88 ±  5%  perf-profile.children.cycles-pp.pick_next_task_fair
      1.19 ±  5%      -0.2        1.01        perf-profile.children.cycles-pp.__filemap_fdatawait_range
      0.97 ±  4%      -0.2        0.80 ±  2%  perf-profile.children.cycles-pp.enqueue_task_fair
      0.98 ±  4%      -0.2        0.81 ±  2%  perf-profile.children.cycles-pp.ttwu_do_activate
      1.43 ±  6%      -0.2        1.26 ±  5%  perf-profile.children.cycles-pp.end_page_writeback
      1.41 ±  6%      -0.2        1.24 ±  4%  perf-profile.children.cycles-pp.test_clear_page_writeback
      0.91 ±  3%      -0.2        0.74 ±  3%  perf-profile.children.cycles-pp.enqueue_entity
      0.94            -0.2        0.78 ±  5%  perf-profile.children.cycles-pp.newidle_balance
      0.57 ±  3%      -0.2        0.42 ±  6%  perf-profile.children.cycles-pp.tcp_rcv_established
      0.82 ±  4%      -0.2        0.67 ±  3%  perf-profile.children.cycles-pp.wake_up_page_bit
      1.04 ±  5%      -0.2        0.89 ±  3%  perf-profile.children.cycles-pp.percpu_counter_add_batch
      0.58 ±  3%      -0.1        0.43 ±  6%  perf-profile.children.cycles-pp.tcp_v6_do_rcv
      0.86 ±  3%      -0.1        0.72 ±  3%  perf-profile.children.cycles-pp.__wake_up_common
      0.73 ±  8%      -0.1        0.59 ±  6%  perf-profile.children.cycles-pp.account_page_dirtied
      1.05 ±  2%      -0.1        0.91 ±  4%  perf-profile.children.cycles-pp.load_balance
      0.72 ±  5%      -0.1        0.59 ±  3%  perf-profile.children.cycles-pp.poll_idle
      0.78 ±  6%      -0.1        0.65 ±  6%  perf-profile.children.cycles-pp.__mod_memcg_lruvec_state
      0.61 ±  4%      -0.1        0.47 ±  4%  perf-profile.children.cycles-pp.sched_ttwu_pending
      0.69 ±  3%      -0.1        0.56 ±  3%  perf-profile.children.cycles-pp.__account_scheduler_latency
      0.44 ±  3%      -0.1        0.31 ±  4%  perf-profile.children.cycles-pp.__release_sock
      0.55 ±  4%      -0.1        0.42 ±  5%  perf-profile.children.cycles-pp.release_sock
      0.29 ±  3%      -0.1        0.16 ±  6%  perf-profile.children.cycles-pp.tcp_ack
      0.27 ±  3%      -0.1        0.15 ±  4%  perf-profile.children.cycles-pp.tcp_clean_rtx_queue
      0.56 ±  4%      -0.1        0.43 ±  6%  perf-profile.children.cycles-pp.flush_smp_call_function_from_idle
      0.97 ±  4%      -0.1        0.85 ±  2%  perf-profile.children.cycles-pp.__test_set_page_writeback
      0.92 ±  4%      -0.1        0.80 ±  5%  perf-profile.children.cycles-pp.__set_page_dirty_nobuffers
      0.75 ±  4%      -0.1        0.64 ±  5%  perf-profile.children.cycles-pp.wait_on_page_bit
      0.94 ±  2%      -0.1        0.83 ±  5%  perf-profile.children.cycles-pp.find_busiest_group
      0.65 ±  3%      -0.1        0.53 ±  2%  perf-profile.children.cycles-pp.wake_page_function
      0.73 ±  4%      -0.1        0.62 ±  5%  perf-profile.children.cycles-pp.mem_cgroup_charge
      0.54 ±  7%      -0.1        0.42 ±  6%  perf-profile.children.cycles-pp.__mod_memcg_state
      0.93 ±  2%      -0.1        0.82 ±  5%  perf-profile.children.cycles-pp.update_sd_lb_stats
      0.66 ±  3%      -0.1        0.55 ±  5%  perf-profile.children.cycles-pp.io_schedule
      0.89 ±  4%      -0.1        0.79 ±  3%  perf-profile.children.cycles-pp.__nfs_commit_inode
      0.66 ±  3%      -0.1        0.57 ±  7%  perf-profile.children.cycles-pp.iomap_set_range_uptodate
      0.50 ±  8%      -0.1        0.40 ±  4%  perf-profile.children.cycles-pp.nfs_ctx_key_to_expire
      0.38 ± 15%      -0.1        0.28 ± 13%  perf-profile.children.cycles-pp.__set_page_dirty
      0.28 ±  5%      -0.1        0.18 ± 13%  perf-profile.children.cycles-pp.rwsem_down_write_slowpath
      0.27 ± 17%      -0.1        0.17 ± 13%  perf-profile.children.cycles-pp.ktime_get_update_offsets_now
      0.20 ±  9%      -0.1        0.11 ± 24%  perf-profile.children.cycles-pp.xfs_ilock
      0.27 ±  5%      -0.1        0.17 ± 14%  perf-profile.children.cycles-pp.rwsem_optimistic_spin
      0.22 ±  7%      -0.1        0.13 ± 19%  perf-profile.children.cycles-pp.rwsem_spin_on_owner
      0.53 ±  2%      -0.1        0.44 ±  2%  perf-profile.children.cycles-pp.stack_trace_save_tsk
      1.06 ±  3%      -0.1        0.98        perf-profile.children.cycles-pp.nfs_commit_release
      1.06 ±  3%      -0.1        0.98        perf-profile.children.cycles-pp.nfs_commit_release_pages
      0.60 ±  3%      -0.1        0.52 ±  3%  perf-profile.children.cycles-pp.prep_new_page
      0.72 ±  3%      -0.1        0.64        perf-profile.children.cycles-pp.lru_cache_add
      0.55 ±  5%      -0.1        0.47 ±  2%  perf-profile.children.cycles-pp.__radix_tree_lookup
      0.41 ±  4%      -0.1        0.33 ±  2%  perf-profile.children.cycles-pp.schedule_idle
      0.67 ±  3%      -0.1        0.60 ±  2%  perf-profile.children.cycles-pp.pagevec_lru_move_fn
      0.39 ±  4%      -0.1        0.32 ±  6%  perf-profile.children.cycles-pp.ip6_protocol_deliver_rcu
      0.39 ±  3%      -0.1        0.32 ±  6%  perf-profile.children.cycles-pp.ip6_input
      0.38 ±  4%      -0.1        0.31 ±  7%  perf-profile.children.cycles-pp.tcp_v6_rcv
      0.55 ±  3%      -0.1        0.48 ±  2%  perf-profile.children.cycles-pp.brd_lookup_page
      0.51 ±  3%      -0.1        0.44 ±  5%  perf-profile.children.cycles-pp.clear_page_erms
      0.46 ±  4%      -0.1        0.39 ±  3%  perf-profile.children.cycles-pp.arch_stack_walk
      0.39 ±  4%      -0.1        0.32 ±  6%  perf-profile.children.cycles-pp.ip6_input_finish
      0.40 ±  3%      -0.1        0.34 ±  6%  perf-profile.children.cycles-pp.ipv6_rcv
      0.22 ±  5%      -0.1        0.15 ±  7%  perf-profile.children.cycles-pp.svc_xprt_do_enqueue
      0.49 ±  3%      -0.1        0.43 ±  2%  perf-profile.children.cycles-pp.__pagevec_lru_add_fn
      0.61 ±  5%      -0.1        0.56 ±  5%  perf-profile.children.cycles-pp.xfs_end_io
      0.61 ±  5%      -0.1        0.56 ±  5%  perf-profile.children.cycles-pp.xfs_end_ioend
      0.59 ±  6%      -0.1        0.54 ±  5%  perf-profile.children.cycles-pp.iomap_finish_ioends
      0.59 ±  6%      -0.1        0.54 ±  5%  perf-profile.children.cycles-pp.iomap_finish_ioend
      0.47 ±  4%      -0.1        0.42 ±  4%  perf-profile.children.cycles-pp.nfs_page_group_sync_on_bit
      0.33 ±  4%      -0.0        0.28 ±  3%  perf-profile.children.cycles-pp.unwind_next_frame
      0.18 ±  3%      -0.0        0.13 ±  3%  perf-profile.children.cycles-pp.svc_xprt_received
      0.18 ±  5%      -0.0        0.14 ±  6%  perf-profile.children.cycles-pp.do_nfsd_create
      0.39 ±  3%      -0.0        0.35 ±  8%  perf-profile.children.cycles-pp.nfs_request_add_commit_list
      0.37 ±  3%      -0.0        0.32 ±  3%  perf-profile.children.cycles-pp.xas_load
      0.21 ±  4%      -0.0        0.16 ±  6%  perf-profile.children.cycles-pp.nfsd4_open
      0.15 ±  5%      -0.0        0.11 ± 10%  perf-profile.children.cycles-pp.schedule_timeout
      0.21 ±  5%      -0.0        0.17 ±  6%  perf-profile.children.cycles-pp.nfs_page_group_lock
      0.21 ±  3%      -0.0        0.17 ± 11%  perf-profile.children.cycles-pp.dec_zone_page_state
      0.14 ±  6%      -0.0        0.10 ±  8%  perf-profile.children.cycles-pp.clear_bdi_congested
      0.32 ±  5%      -0.0        0.28 ±  4%  perf-profile.children.cycles-pp.pagevec_lookup_range_tag
      0.32 ±  5%      -0.0        0.28 ±  4%  perf-profile.children.cycles-pp.find_get_pages_range_tag
      0.17 ±  7%      -0.0        0.13 ±  5%  perf-profile.children.cycles-pp.asm_sysvec_call_function_single
      0.34 ±  3%      -0.0        0.30 ±  3%  perf-profile.children.cycles-pp.__mod_lruvec_state
      0.21            -0.0        0.17 ±  8%  perf-profile.children.cycles-pp.try_charge
      0.11 ±  9%      -0.0        0.08 ±  6%  perf-profile.children.cycles-pp.xfs_generic_create
      0.11 ±  9%      -0.0        0.08 ±  6%  perf-profile.children.cycles-pp.vfs_create
      0.27 ±  3%      -0.0        0.24 ±  3%  perf-profile.children.cycles-pp.__mod_node_page_state
      0.18 ±  5%      -0.0        0.15 ±  2%  perf-profile.children.cycles-pp.alloc_pages_current
      0.22 ±  5%      -0.0        0.18 ±  8%  perf-profile.children.cycles-pp.___might_sleep
      0.15 ±  5%      -0.0        0.12 ±  9%  perf-profile.children.cycles-pp.sysvec_call_function_single
      0.14 ±  6%      -0.0        0.10 ± 10%  perf-profile.children.cycles-pp.__sysvec_call_function_single
      0.20 ±  3%      -0.0        0.17 ±  6%  perf-profile.children.cycles-pp.nfs_initiate_commit
      0.20 ±  5%      -0.0        0.17 ±  6%  perf-profile.children.cycles-pp.nfs_page_set_headlock
      0.11 ±  7%      -0.0        0.08 ±  6%  perf-profile.children.cycles-pp.xfs_create
      0.37 ±  5%      -0.0        0.34 ±  4%  perf-profile.children.cycles-pp.update_rq_clock
      0.14 ±  9%      -0.0        0.11 ±  3%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.18 ±  8%      -0.0        0.15 ± 12%  perf-profile.children.cycles-pp.dequeue_entity
      0.14 ±  6%      -0.0        0.11 ± 10%  perf-profile.children.cycles-pp.find_get_entry
      0.15 ±  2%      -0.0        0.12 ±  8%  perf-profile.children.cycles-pp.finish_task_switch
      0.16 ±  5%      -0.0        0.13 ± 10%  perf-profile.children.cycles-pp.node_dirty_ok
      0.14 ±  8%      -0.0        0.11 ±  8%  perf-profile.children.cycles-pp.ttwu_queue_wakelist
      0.33 ±  3%      -0.0        0.30 ±  3%  perf-profile.children.cycles-pp.nfs_page_group_destroy
      0.15 ± 10%      -0.0        0.12 ±  4%  perf-profile.children.cycles-pp.lock_sock_nested
      0.10 ±  4%      -0.0        0.07 ± 11%  perf-profile.children.cycles-pp.rpc_wait_bit_killable
      0.11 ±  7%      -0.0        0.08 ±  5%  perf-profile.children.cycles-pp.xfs_map_blocks
      0.12 ±  8%      -0.0        0.09 ±  8%  perf-profile.children.cycles-pp.update_curr
      0.19 ±  8%      -0.0        0.17 ±  4%  perf-profile.children.cycles-pp.do_mkdirat
      0.19 ±  8%      -0.0        0.17 ±  4%  perf-profile.children.cycles-pp.filename_create
      0.18 ±  4%      -0.0        0.15 ±  6%  perf-profile.children.cycles-pp.xs_stream_data_receive_workfn
      0.18 ±  4%      -0.0        0.15 ±  6%  perf-profile.children.cycles-pp.xs_stream_data_receive
      0.15 ±  4%      -0.0        0.12 ±  6%  perf-profile.children.cycles-pp.lookup_dcache
      0.18 ±  2%      -0.0        0.15 ± 10%  perf-profile.children.cycles-pp.mutex_lock
      0.08 ±  5%      -0.0        0.06 ± 11%  perf-profile.children.cycles-pp._raw_spin_lock_bh
      0.11 ±  6%      -0.0        0.09 ± 10%  perf-profile.children.cycles-pp.unlock_page
      0.15 ±  5%      -0.0        0.13 ±  5%  perf-profile.children.cycles-pp.autoremove_wake_function
      0.11 ±  7%      -0.0        0.08 ± 10%  perf-profile.children.cycles-pp.xas_start
      0.10 ±  7%      -0.0        0.08 ± 10%  perf-profile.children.cycles-pp.__orc_find
      0.11 ±  4%      -0.0        0.08 ± 10%  perf-profile.children.cycles-pp.out_of_line_wait_on_bit
      0.11 ±  4%      -0.0        0.08 ± 10%  perf-profile.children.cycles-pp.__wait_on_bit
      0.08 ± 11%      -0.0        0.05 ±  8%  perf-profile.children.cycles-pp.__switch_to_asm
      0.10 ± 10%      -0.0        0.08 ±  8%  perf-profile.children.cycles-pp.nfs_key_timeout_notify
      0.10 ±  9%      -0.0        0.07 ±  5%  perf-profile.children.cycles-pp.set_next_entity
      0.11 ± 14%      -0.0        0.09 ±  4%  perf-profile.children.cycles-pp.__lock_sock
      0.21 ±  3%      -0.0        0.19 ±  4%  perf-profile.children.cycles-pp.__queue_work
      0.12 ±  4%      -0.0        0.11 ±  4%  perf-profile.children.cycles-pp.nfs_unlock_and_release_request
      0.10            -0.0        0.08 ±  8%  perf-profile.children.cycles-pp.__pagevec_release
      0.08 ±  5%      -0.0        0.07 ±  7%  perf-profile.children.cycles-pp.tick_nohz_idle_exit
      0.19 ±  4%      -0.0        0.17 ±  4%  perf-profile.children.cycles-pp.queue_work_on
      0.16 ±  2%      -0.0        0.14 ±  5%  perf-profile.children.cycles-pp.__wake_up_common_lock
      0.09 ±  7%      -0.0        0.07 ±  5%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
      0.08 ±  5%      -0.0        0.07 ±  6%  perf-profile.children.cycles-pp.__switch_to
      0.11 ±  7%      -0.0        0.09 ±  4%  perf-profile.children.cycles-pp.iov_iter_advance
      0.08 ±  5%      -0.0        0.07 ±  6%  perf-profile.children.cycles-pp.xas_set_mark
      0.08            -0.0        0.07 ±  6%  perf-profile.children.cycles-pp.nfs_request_add_commit_list_locked
      0.01 ±173%      +0.0        0.06 ± 11%  perf-profile.children.cycles-pp.osq_lock
      0.27 ±  4%      +0.0        0.32 ±  5%  perf-profile.children.cycles-pp.__tcp_push_pending_frames
      0.01 ±173%      +0.1        0.06 ±  6%  perf-profile.children.cycles-pp.kfree
      0.28 ±  7%      +0.1        0.35 ±  4%  perf-profile.children.cycles-pp.svc_tcp_sendto
      0.30 ±  7%      +0.1        0.38 ±  3%  perf-profile.children.cycles-pp.svc_send
      0.42 ± 10%      +0.1        0.51 ±  4%  perf-profile.children.cycles-pp.rebalance_domains
      0.00            +0.1        0.11 ± 13%  perf-profile.children.cycles-pp.pskb_expand_head
      0.00            +0.1        0.13        perf-profile.children.cycles-pp.iov_iter_get_pages
      1.22 ±  5%      +0.2        1.37 ±  2%  perf-profile.children.cycles-pp.rmqueue
      0.09 ±  4%      +0.2        0.24 ± 10%  perf-profile.children.cycles-pp.__check_object_size
      0.07            +0.2        0.24 ±  7%  perf-profile.children.cycles-pp.simple_copy_to_iter
      0.00            +0.2        0.18 ±  2%  perf-profile.children.cycles-pp.__zerocopy_sg_from_iter
      0.00            +0.2        0.18 ±  2%  perf-profile.children.cycles-pp.skb_zerocopy_iter_stream
      0.50 ±  4%      +0.3        0.76 ±  3%  perf-profile.children.cycles-pp.rmqueue_bulk
      0.43 ±  5%      +0.3        0.72 ±  4%  perf-profile.children.cycles-pp.free_unref_page
      0.24 ±  8%      +0.3        0.53 ±  7%  perf-profile.children.cycles-pp.free_pcppages_bulk
      0.78 ±  5%      +0.3        1.11 ±  4%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.23 ±  6%      +0.4        0.59 ±  5%  perf-profile.children.cycles-pp.skb_release_data
      3.28 ±  2%      +0.4        3.65 ±  4%  perf-profile.children.cycles-pp.inet6_recvmsg
      3.27 ±  2%      +0.4        3.64 ±  4%  perf-profile.children.cycles-pp.tcp_recvmsg
      0.31 ±  4%      +0.4        0.71 ±  7%  perf-profile.children.cycles-pp.__kfree_skb
      5.52            +2.3        7.82        perf-profile.children.cycles-pp.memcpy_erms
      5.13 ± 10%      +3.3        8.47 ±  3%  perf-profile.children.cycles-pp.asm_call_sysvec_on_stack
      0.73 ±  5%      +3.7        4.43        perf-profile.children.cycles-pp.tcp_write_xmit
      0.79 ±  4%      +3.7        4.51        perf-profile.children.cycles-pp.__tcp_transmit_skb
      0.69 ±  5%      +3.7        4.44        perf-profile.children.cycles-pp.inet6_csk_xmit
      0.62 ±  5%      +3.8        4.40        perf-profile.children.cycles-pp.ip6_xmit
      0.60 ±  4%      +3.8        4.38        perf-profile.children.cycles-pp.ip6_output
      0.56 ±  3%      +3.8        4.34 ±  2%  perf-profile.children.cycles-pp.ip6_finish_output2
      0.48 ±  4%      +3.8        4.26 ±  2%  perf-profile.children.cycles-pp.__local_bh_enable_ip
      0.47 ±  4%      +3.8        4.26 ±  2%  perf-profile.children.cycles-pp.do_softirq
      0.42 ±  3%      +3.8        4.21 ±  2%  perf-profile.children.cycles-pp.__netif_receive_skb_one_core
      0.44 ±  4%      +3.8        4.23 ±  2%  perf-profile.children.cycles-pp.net_rx_action
      0.43 ±  4%      +3.8        4.22 ±  2%  perf-profile.children.cycles-pp.process_backlog
      1.38 ±  8%      +3.8        5.20        perf-profile.children.cycles-pp.do_softirq_own_stack
      1.38 ±  8%      +3.8        5.20        perf-profile.children.cycles-pp.__softirqentry_text_start
      0.00            +3.9        3.86 ±  2%  perf-profile.children.cycles-pp.skb_copy_ubufs
      0.00            +3.9        3.87 ±  2%  perf-profile.children.cycles-pp.__netif_receive_skb_core
     44.19 ±  4%      +4.2       48.41        perf-profile.children.cycles-pp.cpuidle_enter
     44.18 ±  4%      +4.2       48.40        perf-profile.children.cycles-pp.cpuidle_enter_state
     49.23 ±  2%      +4.9       54.14        perf-profile.children.cycles-pp.start_secondary
     49.50 ±  2%      +5.0       54.45        perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     49.50 ±  2%      +5.0       54.45        perf-profile.children.cycles-pp.cpu_startup_entry
     49.49 ±  2%      +5.0       54.44        perf-profile.children.cycles-pp.do_idle
      2.92 ±  3%      -0.4        2.54 ±  3%  perf-profile.self.cycles-pp._raw_spin_lock
      2.16 ±  3%      -0.3        1.86 ±  2%  perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      2.19 ±  3%      -0.3        1.92 ±  2%  perf-profile.self.cycles-pp.brd_do_bvec
      1.85 ±  6%      -0.2        1.66 ±  4%  perf-profile.self.cycles-pp.nfs_do_writepage
      0.92 ±  5%      -0.2        0.75 ±  5%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.92 ±  6%      -0.1        0.79 ±  5%  perf-profile.self.cycles-pp.percpu_counter_add_batch
      0.69 ±  5%      -0.1        0.56 ±  4%  perf-profile.self.cycles-pp.poll_idle
      0.53 ±  8%      -0.1        0.41 ±  6%  perf-profile.self.cycles-pp.__mod_memcg_state
      0.46 ± 10%      -0.1        0.35 ±  9%  perf-profile.self.cycles-pp.nfs_end_page_writeback
      0.64 ±  2%      -0.1        0.53 ±  4%  perf-profile.self.cycles-pp.update_sd_lb_stats
      0.82 ±  2%      -0.1        0.72 ±  3%  perf-profile.self.cycles-pp.nfs_write_end
      0.24 ± 18%      -0.1        0.14 ± 13%  perf-profile.self.cycles-pp.ktime_get_update_offsets_now
      0.36 ±  6%      -0.1        0.26 ±  4%  perf-profile.self.cycles-pp.svc_tcp_recvfrom
      0.22 ±  7%      -0.1        0.12 ± 15%  perf-profile.self.cycles-pp.rwsem_spin_on_owner
      0.48 ±  8%      -0.1        0.39 ±  4%  perf-profile.self.cycles-pp.nfs_ctx_key_to_expire
      0.46 ±  6%      -0.1        0.37 ±  2%  perf-profile.self.cycles-pp.rmqueue
      0.65 ±  3%      -0.1        0.56 ±  7%  perf-profile.self.cycles-pp.iomap_set_range_uptodate
      0.54 ±  4%      -0.1        0.47 ±  2%  perf-profile.self.cycles-pp.__radix_tree_lookup
      0.50 ±  2%      -0.1        0.44 ±  5%  perf-profile.self.cycles-pp.clear_page_erms
      0.30 ± 11%      -0.1        0.24 ±  9%  perf-profile.self.cycles-pp.clear_page_dirty_for_io
      0.16 ±  5%      -0.0        0.11 ±  4%  perf-profile.self.cycles-pp.try_to_wake_up
      0.26 ±  5%      -0.0        0.22 ±  5%  perf-profile.self.cycles-pp.find_get_pages_range_tag
      0.14 ±  6%      -0.0        0.10 ±  8%  perf-profile.self.cycles-pp.clear_bdi_congested
      0.19 ±  5%      -0.0        0.15 ± 12%  perf-profile.self.cycles-pp.dec_zone_page_state
      0.20 ±  5%      -0.0        0.16 ±  5%  perf-profile.self.cycles-pp.nfs_page_set_headlock
      0.28 ±  4%      -0.0        0.24 ±  2%  perf-profile.self.cycles-pp.xas_load
      0.26 ±  4%      -0.0        0.23 ±  3%  perf-profile.self.cycles-pp.__mod_node_page_state
      0.21 ±  5%      -0.0        0.18 ±  7%  perf-profile.self.cycles-pp.___might_sleep
      0.21 ±  6%      -0.0        0.18 ±  4%  perf-profile.self.cycles-pp.__nfs_create_request
      0.22            -0.0        0.19 ±  5%  perf-profile.self.cycles-pp.__pagevec_lru_add_fn
      0.07 ± 12%      -0.0        0.04 ± 57%  perf-profile.self.cycles-pp.svc_recv
      0.14 ±  9%      -0.0        0.11 ±  6%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.07 ±  7%      -0.0        0.04 ± 58%  perf-profile.self.cycles-pp.iomap_writepage_map
      0.11 ±  6%      -0.0        0.09 ± 10%  perf-profile.self.cycles-pp.unlock_page
      0.10 ±  7%      -0.0        0.08 ± 10%  perf-profile.self.cycles-pp.__orc_find
      0.08 ± 11%      -0.0        0.05 ±  8%  perf-profile.self.cycles-pp.__switch_to_asm
      0.15 ±  4%      -0.0        0.13 ±  5%  perf-profile.self.cycles-pp.do_idle
      0.11 ±  4%      -0.0        0.09 ±  5%  perf-profile.self.cycles-pp.iov_iter_advance
      0.10 ± 10%      -0.0        0.08 ±  5%  perf-profile.self.cycles-pp.alloc_pages_current
      0.17 ±  2%      -0.0        0.15 ±  5%  perf-profile.self.cycles-pp.unwind_next_frame
      0.15 ±  2%      -0.0        0.13 ±  5%  perf-profile.self.cycles-pp.release_pages
      0.10 ±  5%      -0.0        0.08 ± 14%  perf-profile.self.cycles-pp.xas_start
      0.07 ± 12%      -0.0        0.05        perf-profile.self.cycles-pp.set_next_entity
      0.08            -0.0        0.07 ±  7%  perf-profile.self.cycles-pp.xas_set_mark
      0.08 ±  6%      -0.0        0.06 ±  6%  perf-profile.self.cycles-pp.enqueue_entity
      0.14 ±  6%      +0.0        0.16 ±  6%  perf-profile.self.cycles-pp.skb_release_data
      0.01 ±173%      +0.0        0.06 ± 11%  perf-profile.self.cycles-pp.osq_lock
      0.00            +0.1        0.06 ±  6%  perf-profile.self.cycles-pp.kfree
      0.00            +0.1        0.09 ± 13%  perf-profile.self.cycles-pp.pskb_expand_head
      0.00            +0.1        0.12 ±  6%  perf-profile.self.cycles-pp.skb_copy_ubufs
      0.10 ± 12%      +0.1        0.23 ±  8%  perf-profile.self.cycles-pp.free_pcppages_bulk
      0.00            +0.1        0.13        perf-profile.self.cycles-pp.iov_iter_get_pages
      0.07 ±  7%      +0.1        0.21 ± 10%  perf-profile.self.cycles-pp.__check_object_size
      0.77 ±  4%      +0.3        1.10 ±  4%  perf-profile.self.cycles-pp.__list_del_entry_valid
      5.46            +2.3        7.73        perf-profile.self.cycles-pp.memcpy_erms


                                                                                
                               fsmark.files_per_sec                             
                                                                                
  580 +---------------------------------------------------------------------+   
      |         .+                                                          |   
  560 |..+.  .+.  +                          .+..+..    .+..+               |   
      |    +.      +..      +    +      .+..+       +.+.                    |   
      |               +.. .. + .. +   +.                                    |   
  540 |-+                +    +    + +                                      |   
      |                             +                                       |   
  520 |-+                                                                   |   
      |                                                                     |   
  500 |-+                                                                   |   
      |    O                     O  O                                       |   
      |  O       O O  O       O               O     O O  O  O  O O  O  O O  |   
  480 |-+     O          O  O         O     O    O                          |   
      |                                  O                                  |   
  460 +---------------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample

***************************************************************************************************
lkp-csl-2ap2: 192 threads Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory
=========================================================================================
compiler/cpufreq_governor/disk/filesize/fs2/fs/iterations/kconfig/nr_threads/rootfs/sync_method/tbox_group/test_size/testcase/ucode:
  gcc-9/performance/1BRD_48G/4M/nfsv4/xfs/1x/x86_64-rhel-8.3/64t/debian-10.4-x86_64-20200603.cgz/NoSync/lkp-csl-2ap2/40G/fsmark/0x5003003

commit: 
  v5.10-rc2
  3bc6a407d1 ("SUNRPC: Use zero-copy to perform socket send operations")

       v5.10-rc2 3bc6a407d1a090417218ecdb0dd 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
  12868393           +17.9%   15177801 ±  2%  fsmark.app_overhead
    559.38           -13.3%     485.20        fsmark.files_per_sec
     21.35           +13.1%      24.16        fsmark.time.elapsed_time
     21.35           +13.1%      24.16        fsmark.time.elapsed_time.max
      1664 ±  4%     -13.9%       1433 ±  3%  fsmark.time.involuntary_context_switches
    251.75 ±  2%     -13.1%     218.75        fsmark.time.percent_of_cpu_this_job_got
     25.96            -2.0%      25.44        boot-time.dhcp
      2.47 ±  2%     +19.0%       2.95 ±  2%  iostat.cpu.iowait
    671458 ±  2%      +8.6%     728893 ±  3%  softirqs.RCU
   1669499            -8.2%    1532757        vmstat.io.bo
    202132           -13.7%     174380 ±  2%  vmstat.system.cs
    937363 ± 62%     -61.6%     359933 ±  9%  cpuidle.C1.usage
 2.258e+09 ± 51%     +86.4%  4.209e+09 ±  9%  cpuidle.C1E.time
   6449059 ± 20%     +51.7%    9781611 ±  5%  cpuidle.C1E.usage
      2.67 ±  2%      +0.5        3.16 ±  2%  mpstat.cpu.all.iowait%
      0.14 ±  3%      +0.3        0.42        mpstat.cpu.all.soft%
      3.55 ±  4%      -0.5        3.05 ±  2%  mpstat.cpu.all.sys%
     10140 ± 26%     -31.3%       6970 ±  3%  numa-meminfo.node2.Mapped
     16956 ±  7%     -17.1%      14052 ±  3%  numa-meminfo.node2.Writeback
    196488 ± 12%     -17.5%     162160 ± 11%  numa-meminfo.node3.Slab
      3006 ±  4%      +7.2%       3223 ±  4%  proc-vmstat.nr_active_anon
      3006 ±  4%      +7.2%       3223 ±  4%  proc-vmstat.nr_zone_active_anon
  77623491 ±  3%     -13.4%   67184071        proc-vmstat.pgalloc_normal
  56668889 ±  4%     -18.7%   46043533        proc-vmstat.pgfree
     10476            +4.4%      10935        proc-vmstat.pgreuse
      4779 ± 13%     -27.6%       3460 ±  6%  sched_debug.cfs_rq:/.min_vruntime.stddev
     27500 ± 37%     -78.2%       5982 ± 93%  sched_debug.cfs_rq:/.spread0.max
      4780 ± 13%     -25.8%       3546 ±  7%  sched_debug.cfs_rq:/.spread0.stddev
   1730403 ± 10%     +41.9%    2456125 ± 26%  sched_debug.cpu.avg_idle.max
     13907 ± 18%     -28.8%       9897 ± 21%  sched_debug.cpu.nr_switches.max
      1151 ±  9%     -15.7%     970.60 ±  8%  sched_debug.cpu.nr_switches.stddev
     21696           -15.9%      18236 ±  2%  slabinfo.kmalloc-1k.active_objs
    684.25           -15.1%     580.75 ±  2%  slabinfo.kmalloc-1k.active_slabs
     21907           -15.1%      18591 ±  2%  slabinfo.kmalloc-1k.num_objs
    684.25           -15.1%     580.75 ±  2%  slabinfo.kmalloc-1k.num_slabs
     11971 ±  3%     -11.5%      10592 ±  7%  slabinfo.nfsd_file.active_objs
      3314 ±  5%     -20.9%       2621 ± 13%  slabinfo.xfs_efd_item.active_objs
      3355 ±  4%     -20.1%       2680 ± 12%  slabinfo.xfs_efd_item.num_objs
      3275 ±  5%     -20.4%       2607 ± 13%  slabinfo.xfs_efi_item.active_objs
      3318 ±  5%     -19.7%       2663 ± 13%  slabinfo.xfs_efi_item.num_objs
 3.817e+09            -7.6%  3.526e+09        perf-stat.i.branch-instructions
 3.316e+08 ±  2%      -9.4%  3.003e+08 ±  2%  perf-stat.i.cache-references
    234663 ±  2%     -16.7%     195399        perf-stat.i.context-switches
 3.805e+10 ±  2%      -6.6%  3.552e+10 ±  2%  perf-stat.i.cpu-cycles
      1382 ±  3%      -7.2%       1283 ±  3%  perf-stat.i.cpu-migrations
 4.805e+09            -7.6%  4.438e+09        perf-stat.i.dTLB-loads
 2.407e+09            -8.7%  2.198e+09        perf-stat.i.dTLB-stores
   7989114 ±  2%      -4.2%    7652291        perf-stat.i.iTLB-load-misses
 1.929e+10            -7.4%  1.787e+10        perf-stat.i.instructions
      2469 ±  2%      -4.9%       2348        perf-stat.i.instructions-per-iTLB-miss
      0.20 ±  2%      -6.6%       0.19 ±  2%  perf-stat.i.metric.GHz
     59.57            -7.9%      54.88        perf-stat.i.metric.M/sec
  38311030 ±  5%     -10.6%   34257932 ±  2%  perf-stat.i.node-load-misses
  10676843 ±  7%     -13.7%    9213801        perf-stat.i.node-loads
  10297341 ± 21%     -21.5%    8080955 ±  3%  perf-stat.i.node-stores
 3.647e+09            -7.2%  3.386e+09        perf-stat.ps.branch-instructions
 3.169e+08 ±  2%      -9.0%  2.884e+08        perf-stat.ps.cache-references
    224244           -16.3%     187638        perf-stat.ps.context-switches
 3.636e+10 ±  2%      -6.2%  3.411e+10 ±  2%  perf-stat.ps.cpu-cycles
  4.59e+09            -7.1%  4.262e+09        perf-stat.ps.dTLB-loads
 2.299e+09            -8.2%   2.11e+09        perf-stat.ps.dTLB-stores
   7634292 ±  2%      -3.7%    7349222        perf-stat.ps.iTLB-load-misses
 1.843e+10            -6.9%  1.716e+10        perf-stat.ps.instructions
  36618483 ±  5%     -10.1%   32904334 ±  2%  perf-stat.ps.node-load-misses
  10197765 ±  7%     -13.3%    8839824        perf-stat.ps.node-loads
   9840509 ± 21%     -21.1%    7760538 ±  3%  perf-stat.ps.node-stores
 4.173e+11            +4.7%  4.368e+11 ±  2%  perf-stat.total.instructions
     27.50 ± 35%   +1444.5%     424.75 ± 95%  interrupts.33:PCI-MSI.524291-edge.eth0-TxRx-2
     46517 ±  2%     +12.0%      52079        interrupts.CPU0.LOC:Local_timer_interrupts
      2.00 ± 35%   +2775.0%      57.50 ±157%  interrupts.CPU0.TLB:TLB_shootdowns
     46340 ±  3%     +12.3%      52034 ±  2%  interrupts.CPU1.LOC:Local_timer_interrupts
     46533 ±  3%     +11.7%      51972        interrupts.CPU10.LOC:Local_timer_interrupts
     46371 ±  3%     +11.9%      51871 ±  2%  interrupts.CPU100.LOC:Local_timer_interrupts
     46334 ±  3%     +12.1%      51963        interrupts.CPU101.LOC:Local_timer_interrupts
     46349 ±  3%     +11.7%      51764        interrupts.CPU102.LOC:Local_timer_interrupts
     46306 ±  3%     +12.0%      51860        interrupts.CPU103.LOC:Local_timer_interrupts
     46351 ±  3%     +11.7%      51795 ±  2%  interrupts.CPU104.LOC:Local_timer_interrupts
     46348 ±  3%     +11.9%      51847 ±  2%  interrupts.CPU105.LOC:Local_timer_interrupts
     46414 ±  2%     +11.8%      51900        interrupts.CPU106.LOC:Local_timer_interrupts
     46379 ±  3%     +11.9%      51902        interrupts.CPU107.LOC:Local_timer_interrupts
     46358 ±  3%     +11.9%      51883 ±  2%  interrupts.CPU108.LOC:Local_timer_interrupts
     46420 ±  2%     +11.5%      51766        interrupts.CPU109.LOC:Local_timer_interrupts
     46404 ±  3%     +12.0%      51974        interrupts.CPU11.LOC:Local_timer_interrupts
     46349 ±  3%     +11.9%      51882 ±  2%  interrupts.CPU110.LOC:Local_timer_interrupts
     46346 ±  3%     +11.9%      51882 ±  2%  interrupts.CPU111.LOC:Local_timer_interrupts
     46377 ±  3%     +11.9%      51878 ±  2%  interrupts.CPU112.LOC:Local_timer_interrupts
     46357 ±  3%     +11.8%      51827        interrupts.CPU113.LOC:Local_timer_interrupts
     46221 ±  3%     +12.2%      51862        interrupts.CPU114.LOC:Local_timer_interrupts
     46360 ±  3%     +11.9%      51876 ±  2%  interrupts.CPU115.LOC:Local_timer_interrupts
     46382 ±  3%     +11.9%      51881 ±  2%  interrupts.CPU116.LOC:Local_timer_interrupts
     46363 ±  3%     +12.0%      51921 ±  2%  interrupts.CPU117.LOC:Local_timer_interrupts
     46362 ±  3%     +12.1%      51961        interrupts.CPU118.LOC:Local_timer_interrupts
     46375 ±  3%     +11.9%      51882 ±  2%  interrupts.CPU119.LOC:Local_timer_interrupts
     27.50 ± 35%   +1444.5%     424.75 ± 95%  interrupts.CPU12.33:PCI-MSI.524291-edge.eth0-TxRx-2
     46330 ±  2%     +12.4%      52054        interrupts.CPU12.LOC:Local_timer_interrupts
     46146 ±  3%     +12.5%      51905 ±  2%  interrupts.CPU120.LOC:Local_timer_interrupts
     46155 ±  3%     +12.5%      51904 ±  2%  interrupts.CPU121.LOC:Local_timer_interrupts
     46153 ±  3%     +12.5%      51922 ±  2%  interrupts.CPU122.LOC:Local_timer_interrupts
     46147 ±  3%     +12.5%      51908 ±  2%  interrupts.CPU123.LOC:Local_timer_interrupts
     46163 ±  3%     +12.5%      51942        interrupts.CPU124.LOC:Local_timer_interrupts
     46238 ±  3%     +12.3%      51921        interrupts.CPU125.LOC:Local_timer_interrupts
     46340 ±  3%     +12.0%      51886 ±  2%  interrupts.CPU126.LOC:Local_timer_interrupts
     46208 ±  3%     +12.1%      51822 ±  2%  interrupts.CPU127.LOC:Local_timer_interrupts
     46153 ±  3%     +12.5%      51910 ±  2%  interrupts.CPU128.LOC:Local_timer_interrupts
     46179 ±  3%     +12.4%      51916 ±  2%  interrupts.CPU129.LOC:Local_timer_interrupts
     46554 ±  3%     +11.4%      51855        interrupts.CPU13.LOC:Local_timer_interrupts
     46246 ±  3%     +12.3%      51947 ±  2%  interrupts.CPU130.LOC:Local_timer_interrupts
     45950 ±  3%     +13.0%      51904 ±  2%  interrupts.CPU131.LOC:Local_timer_interrupts
     46146 ±  3%     +12.4%      51845 ±  2%  interrupts.CPU132.LOC:Local_timer_interrupts
     46125 ±  3%     +12.6%      51920 ±  2%  interrupts.CPU133.LOC:Local_timer_interrupts
     46236 ±  3%     +12.3%      51912 ±  2%  interrupts.CPU134.LOC:Local_timer_interrupts
     46232 ±  3%     +12.3%      51902 ±  2%  interrupts.CPU135.LOC:Local_timer_interrupts
     46196 ±  3%     +12.5%      51954        interrupts.CPU136.LOC:Local_timer_interrupts
     46239 ±  3%     +11.9%      51745 ±  2%  interrupts.CPU137.LOC:Local_timer_interrupts
     46415 ±  3%     +11.8%      51892        interrupts.CPU138.LOC:Local_timer_interrupts
     46236 ±  3%     +12.2%      51895 ±  2%  interrupts.CPU139.LOC:Local_timer_interrupts
     46508 ±  3%     +11.7%      51938        interrupts.CPU14.LOC:Local_timer_interrupts
     46279 ±  3%     +12.1%      51896        interrupts.CPU140.LOC:Local_timer_interrupts
     46142 ±  3%     +12.5%      51917 ±  2%  interrupts.CPU141.LOC:Local_timer_interrupts
    482.50 ± 18%     +28.0%     617.50 ± 13%  interrupts.CPU141.NMI:Non-maskable_interrupts
    482.50 ± 18%     +28.0%     617.50 ± 13%  interrupts.CPU141.PMI:Performance_monitoring_interrupts
     46295 ±  3%     +11.6%      51670 ±  2%  interrupts.CPU142.LOC:Local_timer_interrupts
    617.75 ±  8%     +29.8%     801.75 ± 15%  interrupts.CPU142.NMI:Non-maskable_interrupts
    617.75 ±  8%     +29.8%     801.75 ± 15%  interrupts.CPU142.PMI:Performance_monitoring_interrupts
      1.00 ± 70%  +20750.0%     208.50 ±169%  interrupts.CPU142.TLB:TLB_shootdowns
     46152 ±  3%     +12.4%      51862 ±  2%  interrupts.CPU143.LOC:Local_timer_interrupts
     46417 ±  3%     +12.1%      52022        interrupts.CPU144.LOC:Local_timer_interrupts
     46496 ±  3%     +11.8%      51987        interrupts.CPU145.LOC:Local_timer_interrupts
     46498 ±  2%     +11.9%      52022        interrupts.CPU146.LOC:Local_timer_interrupts
     46507 ±  2%     +11.8%      51978        interrupts.CPU147.LOC:Local_timer_interrupts
     46499 ±  2%     +11.8%      51994        interrupts.CPU148.LOC:Local_timer_interrupts
     46515 ±  2%     +11.7%      51971        interrupts.CPU149.LOC:Local_timer_interrupts
     46469 ±  2%     +11.8%      51941        interrupts.CPU15.LOC:Local_timer_interrupts
     46568 ±  3%     +11.7%      52007        interrupts.CPU150.LOC:Local_timer_interrupts
     46562 ±  3%     +11.7%      52002        interrupts.CPU151.LOC:Local_timer_interrupts
     46510 ±  2%     +11.8%      51979        interrupts.CPU152.LOC:Local_timer_interrupts
     46661 ±  2%     +11.4%      51971        interrupts.CPU153.LOC:Local_timer_interrupts
     46532 ±  2%     +11.7%      51986        interrupts.CPU154.LOC:Local_timer_interrupts
     46514 ±  2%     +11.9%      52042        interrupts.CPU155.LOC:Local_timer_interrupts
     46415 ±  2%     +12.2%      52069        interrupts.CPU156.LOC:Local_timer_interrupts
      2565 ± 12%     -24.0%       1949 ± 17%  interrupts.CPU157.CAL:Function_call_interrupts
     46532 ±  2%     +11.7%      51958        interrupts.CPU157.LOC:Local_timer_interrupts
     46494 ±  2%     +11.8%      51999        interrupts.CPU158.LOC:Local_timer_interrupts
     46949 ±  3%     +10.8%      52016        interrupts.CPU159.LOC:Local_timer_interrupts
     46480 ±  2%     +11.8%      51958        interrupts.CPU16.LOC:Local_timer_interrupts
     46492 ±  2%     +11.8%      51997        interrupts.CPU160.LOC:Local_timer_interrupts
     46601 ±  3%     +11.6%      51984        interrupts.CPU161.LOC:Local_timer_interrupts
    524.75 ± 52%    +105.4%       1078 ± 23%  interrupts.CPU161.NMI:Non-maskable_interrupts
    524.75 ± 52%    +105.4%       1078 ± 23%  interrupts.CPU161.PMI:Performance_monitoring_interrupts
     46573 ±  3%     +11.6%      51992        interrupts.CPU162.LOC:Local_timer_interrupts
     46516 ±  3%     +11.8%      51994        interrupts.CPU163.LOC:Local_timer_interrupts
     46552 ±  3%     +11.7%      52016        interrupts.CPU164.LOC:Local_timer_interrupts
     46556 ±  2%     +11.6%      51968        interrupts.CPU165.LOC:Local_timer_interrupts
     46509 ±  2%     +11.9%      52023        interrupts.CPU166.LOC:Local_timer_interrupts
     46486 ±  2%     +11.8%      51954        interrupts.CPU167.LOC:Local_timer_interrupts
      2735 ± 17%     -22.1%       2131 ±  7%  interrupts.CPU168.CAL:Function_call_interrupts
     46657 ±  3%     +11.2%      51869 ±  2%  interrupts.CPU168.LOC:Local_timer_interrupts
      2577 ± 10%     -31.5%       1765 ± 11%  interrupts.CPU169.CAL:Function_call_interrupts
     46474 ±  2%     +11.8%      51975        interrupts.CPU169.LOC:Local_timer_interrupts
      1882 ± 13%     +29.5%       2438 ± 11%  interrupts.CPU17.CAL:Function_call_interrupts
     46241 ±  3%     +12.3%      51932        interrupts.CPU17.LOC:Local_timer_interrupts
     46536 ±  3%     +11.6%      51918        interrupts.CPU170.LOC:Local_timer_interrupts
     46614 ±  3%     +11.4%      51912        interrupts.CPU171.LOC:Local_timer_interrupts
     46464 ±  3%     +11.8%      51967        interrupts.CPU172.LOC:Local_timer_interrupts
     46584 ±  3%     +11.5%      51924        interrupts.CPU173.LOC:Local_timer_interrupts
     46526 ±  3%     +11.5%      51868 ±  2%  interrupts.CPU174.LOC:Local_timer_interrupts
     46603 ±  3%     +11.4%      51905        interrupts.CPU175.LOC:Local_timer_interrupts
     46588 ±  3%     +11.4%      51878 ±  2%  interrupts.CPU176.LOC:Local_timer_interrupts
     46476 ±  2%     +11.7%      51930 ±  2%  interrupts.CPU177.LOC:Local_timer_interrupts
     46529 ±  2%     +11.4%      51821        interrupts.CPU178.LOC:Local_timer_interrupts
     46519 ±  3%     +11.6%      51902 ±  2%  interrupts.CPU179.LOC:Local_timer_interrupts
      1137 ± 12%     -41.7%     662.50 ± 40%  interrupts.CPU179.NMI:Non-maskable_interrupts
      1137 ± 12%     -41.7%     662.50 ± 40%  interrupts.CPU179.PMI:Performance_monitoring_interrupts
     46228 ±  3%     +12.3%      51928        interrupts.CPU18.LOC:Local_timer_interrupts
      2552 ± 15%     -19.6%       2052 ±  8%  interrupts.CPU180.CAL:Function_call_interrupts
     46592 ±  3%     +11.4%      51889        interrupts.CPU180.LOC:Local_timer_interrupts
     46437 ±  3%     +11.4%      51749 ±  2%  interrupts.CPU181.LOC:Local_timer_interrupts
     46574 ±  2%     +11.4%      51877        interrupts.CPU182.LOC:Local_timer_interrupts
    877.50 ± 21%     -52.0%     421.50 ± 28%  interrupts.CPU182.NMI:Non-maskable_interrupts
    877.50 ± 21%     -52.0%     421.50 ± 28%  interrupts.CPU182.PMI:Performance_monitoring_interrupts
     46513 ±  3%     +11.5%      51881        interrupts.CPU183.LOC:Local_timer_interrupts
    803.00 ± 17%     -35.1%     521.25 ± 11%  interrupts.CPU183.NMI:Non-maskable_interrupts
    803.00 ± 17%     -35.1%     521.25 ± 11%  interrupts.CPU183.PMI:Performance_monitoring_interrupts
     46487 ±  3%     +11.6%      51868 ±  2%  interrupts.CPU184.LOC:Local_timer_interrupts
    748.50 ± 16%     -36.4%     476.00 ± 24%  interrupts.CPU184.NMI:Non-maskable_interrupts
    748.50 ± 16%     -36.4%     476.00 ± 24%  interrupts.CPU184.PMI:Performance_monitoring_interrupts
     46583 ±  3%     +11.3%      51870 ±  2%  interrupts.CPU185.LOC:Local_timer_interrupts
     46465 ±  3%     +11.6%      51870 ±  2%  interrupts.CPU186.LOC:Local_timer_interrupts
     46465 ±  3%     +11.8%      51937 ±  2%  interrupts.CPU187.LOC:Local_timer_interrupts
     46555 ±  3%     +11.4%      51878 ±  2%  interrupts.CPU188.LOC:Local_timer_interrupts
      2884 ± 25%     -29.4%       2037 ± 10%  interrupts.CPU189.CAL:Function_call_interrupts
     46581 ±  3%     +11.4%      51886        interrupts.CPU189.LOC:Local_timer_interrupts
     46447 ±  3%     +11.8%      51919        interrupts.CPU19.LOC:Local_timer_interrupts
     46576 ±  3%     +11.4%      51870 ±  2%  interrupts.CPU190.LOC:Local_timer_interrupts
     46478 ±  2%     +11.6%      51889        interrupts.CPU191.LOC:Local_timer_interrupts
     46453 ±  3%     +11.9%      51986        interrupts.CPU2.LOC:Local_timer_interrupts
     46377 ±  3%     +12.2%      52015        interrupts.CPU20.LOC:Local_timer_interrupts
      1365 ± 85%     -67.8%     439.50 ± 41%  interrupts.CPU20.NMI:Non-maskable_interrupts
      1365 ± 85%     -67.8%     439.50 ± 41%  interrupts.CPU20.PMI:Performance_monitoring_interrupts
     46389 ±  3%     +12.1%      52019        interrupts.CPU21.LOC:Local_timer_interrupts
     46439 ±  3%     +11.9%      51967        interrupts.CPU22.LOC:Local_timer_interrupts
    750.00 ±  6%     -43.7%     422.25 ± 27%  interrupts.CPU22.NMI:Non-maskable_interrupts
    750.00 ±  6%     -43.7%     422.25 ± 27%  interrupts.CPU22.PMI:Performance_monitoring_interrupts
     46396 ±  3%     +12.1%      52020        interrupts.CPU23.LOC:Local_timer_interrupts
     46080 ±  3%     +12.8%      51965        interrupts.CPU24.LOC:Local_timer_interrupts
     46183 ±  3%     +12.6%      51990 ±  2%  interrupts.CPU25.LOC:Local_timer_interrupts
     46226 ±  3%     +12.4%      51979 ±  2%  interrupts.CPU26.LOC:Local_timer_interrupts
     46238 ±  3%     +12.4%      51984 ±  2%  interrupts.CPU27.LOC:Local_timer_interrupts
     46288 ±  3%     +12.3%      51999        interrupts.CPU28.LOC:Local_timer_interrupts
     46173 ±  3%     +12.5%      51962        interrupts.CPU29.LOC:Local_timer_interrupts
     46412 ±  3%     +11.9%      51944        interrupts.CPU3.LOC:Local_timer_interrupts
     46318 ±  3%     +12.2%      51967 ±  2%  interrupts.CPU30.LOC:Local_timer_interrupts
     46311 ±  3%     +12.2%      51981        interrupts.CPU31.LOC:Local_timer_interrupts
     46262 ±  3%     +12.4%      51981 ±  2%  interrupts.CPU32.LOC:Local_timer_interrupts
     46201 ±  3%     +12.1%      51800 ±  2%  interrupts.CPU33.LOC:Local_timer_interrupts
     46263 ±  3%     +12.3%      51957        interrupts.CPU34.LOC:Local_timer_interrupts
     46287 ±  3%     +12.1%      51870 ±  2%  interrupts.CPU35.LOC:Local_timer_interrupts
     46200 ±  3%     +11.8%      51638 ±  2%  interrupts.CPU36.LOC:Local_timer_interrupts
     46224 ±  3%     +12.5%      52006        interrupts.CPU37.LOC:Local_timer_interrupts
     46163 ±  3%     +12.5%      51950 ±  2%  interrupts.CPU38.LOC:Local_timer_interrupts
     46328 ±  3%     +12.2%      51993        interrupts.CPU39.LOC:Local_timer_interrupts
     46435 ±  3%     +12.1%      52052        interrupts.CPU4.LOC:Local_timer_interrupts
     46343 ±  3%     +12.2%      52012        interrupts.CPU40.LOC:Local_timer_interrupts
    484.00 ± 25%     +52.1%     736.00 ± 23%  interrupts.CPU40.NMI:Non-maskable_interrupts
    484.00 ± 25%     +52.1%     736.00 ± 23%  interrupts.CPU40.PMI:Performance_monitoring_interrupts
     46267 ±  3%     +12.3%      51973        interrupts.CPU41.LOC:Local_timer_interrupts
     46223 ±  3%     +12.5%      51997 ±  2%  interrupts.CPU42.LOC:Local_timer_interrupts
     46251 ±  3%     +12.5%      52055        interrupts.CPU43.LOC:Local_timer_interrupts
     46293 ±  3%     +12.2%      51951        interrupts.CPU44.LOC:Local_timer_interrupts
      0.50 ±100%  +22800.0%     114.50 ±169%  interrupts.CPU44.TLB:TLB_shootdowns
     46318 ±  3%     +12.2%      51951        interrupts.CPU45.LOC:Local_timer_interrupts
     46190 ±  3%     +12.5%      51954 ±  2%  interrupts.CPU46.LOC:Local_timer_interrupts
     46195 ±  3%     +12.4%      51912 ±  2%  interrupts.CPU47.LOC:Local_timer_interrupts
     46573 ±  3%     +11.6%      51993        interrupts.CPU48.LOC:Local_timer_interrupts
     46525 ±  2%     +11.9%      52043        interrupts.CPU49.LOC:Local_timer_interrupts
     46481 ±  2%     +11.8%      51970        interrupts.CPU5.LOC:Local_timer_interrupts
     46676 ±  2%     +11.5%      52021        interrupts.CPU50.LOC:Local_timer_interrupts
     46614 ±  3%     +11.6%      52002        interrupts.CPU51.LOC:Local_timer_interrupts
     46510 ±  3%     +11.8%      52005        interrupts.CPU52.LOC:Local_timer_interrupts
     46599 ±  3%     +11.6%      52027        interrupts.CPU53.LOC:Local_timer_interrupts
     46678 ±  2%     +11.5%      52032        interrupts.CPU54.LOC:Local_timer_interrupts
     46576 ±  3%     +11.6%      51997        interrupts.CPU55.LOC:Local_timer_interrupts
     46546 ±  3%     +11.6%      51960 ±  2%  interrupts.CPU56.LOC:Local_timer_interrupts
     46574 ±  3%     +11.7%      52019        interrupts.CPU57.LOC:Local_timer_interrupts
     46620 ±  3%     +11.6%      52046        interrupts.CPU58.LOC:Local_timer_interrupts
     46589 ±  3%     +11.6%      51978        interrupts.CPU59.LOC:Local_timer_interrupts
     46399 ±  3%     +12.1%      52033        interrupts.CPU6.LOC:Local_timer_interrupts
     46580 ±  3%     +11.6%      52004        interrupts.CPU60.LOC:Local_timer_interrupts
     46567 ±  3%     +11.8%      52042        interrupts.CPU61.LOC:Local_timer_interrupts
     46574 ±  3%     +11.7%      52011        interrupts.CPU62.LOC:Local_timer_interrupts
     46504 ±  2%     +11.9%      52021        interrupts.CPU63.LOC:Local_timer_interrupts
     88.75 ± 20%     -39.2%      54.00 ± 23%  interrupts.CPU63.RES:Rescheduling_interrupts
     46584 ±  3%     +11.7%      52012        interrupts.CPU64.LOC:Local_timer_interrupts
     46539 ±  2%     +11.8%      52026        interrupts.CPU65.LOC:Local_timer_interrupts
    492.00 ± 45%    +123.2%       1098 ± 15%  interrupts.CPU65.NMI:Non-maskable_interrupts
    492.00 ± 45%    +123.2%       1098 ± 15%  interrupts.CPU65.PMI:Performance_monitoring_interrupts
     46516 ±  2%     +12.0%      52087        interrupts.CPU66.LOC:Local_timer_interrupts
     46574 ±  2%     +11.8%      52048        interrupts.CPU67.LOC:Local_timer_interrupts
     46532 ±  2%     +11.8%      52025        interrupts.CPU68.LOC:Local_timer_interrupts
     46544 ±  3%     +11.8%      52025        interrupts.CPU69.LOC:Local_timer_interrupts
     46360 ±  3%     +12.2%      52031        interrupts.CPU7.LOC:Local_timer_interrupts
     46617 ±  3%     +11.5%      51988        interrupts.CPU70.LOC:Local_timer_interrupts
     46629 ±  3%     +11.5%      52000        interrupts.CPU71.LOC:Local_timer_interrupts
     46559 ±  2%     +11.5%      51891        interrupts.CPU72.LOC:Local_timer_interrupts
     46387 ±  3%     +12.1%      51993        interrupts.CPU73.LOC:Local_timer_interrupts
     46619 ±  3%     +11.6%      52046        interrupts.CPU74.LOC:Local_timer_interrupts
     46617 ±  3%     +11.3%      51907 ±  2%  interrupts.CPU75.LOC:Local_timer_interrupts
     46621 ±  3%     +11.3%      51904 ±  2%  interrupts.CPU76.LOC:Local_timer_interrupts
    942.75 ± 22%     -47.9%     491.00 ± 27%  interrupts.CPU76.NMI:Non-maskable_interrupts
    942.75 ± 22%     -47.9%     491.00 ± 27%  interrupts.CPU76.PMI:Performance_monitoring_interrupts
     46584 ±  3%     +11.6%      51993        interrupts.CPU77.LOC:Local_timer_interrupts
     85.75 ± 15%     -28.3%      61.50 ± 20%  interrupts.CPU77.RES:Rescheduling_interrupts
      3195 ± 11%     -38.2%       1974 ± 16%  interrupts.CPU78.CAL:Function_call_interrupts
     46556 ±  3%     +11.5%      51905 ±  2%  interrupts.CPU78.LOC:Local_timer_interrupts
      1102 ± 51%     -58.9%     453.00 ± 34%  interrupts.CPU78.NMI:Non-maskable_interrupts
      1102 ± 51%     -58.9%     453.00 ± 34%  interrupts.CPU78.PMI:Performance_monitoring_interrupts
    130.75 ± 51%     -62.1%      49.50 ± 22%  interrupts.CPU78.RES:Rescheduling_interrupts
      3086 ± 16%     -28.7%       2202 ± 15%  interrupts.CPU79.CAL:Function_call_interrupts
     46568 ±  3%     +11.4%      51897 ±  2%  interrupts.CPU79.LOC:Local_timer_interrupts
    934.25 ± 34%     -40.8%     552.75 ± 33%  interrupts.CPU79.NMI:Non-maskable_interrupts
    934.25 ± 34%     -40.8%     552.75 ± 33%  interrupts.CPU79.PMI:Performance_monitoring_interrupts
     95.75 ± 15%     -47.8%      50.00 ± 18%  interrupts.CPU79.RES:Rescheduling_interrupts
     46388 ±  3%     +12.1%      52009        interrupts.CPU8.LOC:Local_timer_interrupts
     46580 ±  3%     +11.4%      51896        interrupts.CPU80.LOC:Local_timer_interrupts
    854.75 ± 29%     -39.9%     513.50 ± 38%  interrupts.CPU80.NMI:Non-maskable_interrupts
    854.75 ± 29%     -39.9%     513.50 ± 38%  interrupts.CPU80.PMI:Performance_monitoring_interrupts
     46569 ±  3%     +11.5%      51906 ±  2%  interrupts.CPU81.LOC:Local_timer_interrupts
    947.75 ± 16%     -37.4%     593.25 ± 34%  interrupts.CPU81.NMI:Non-maskable_interrupts
    947.75 ± 16%     -37.4%     593.25 ± 34%  interrupts.CPU81.PMI:Performance_monitoring_interrupts
     80.75 ± 26%     -36.8%      51.00 ± 24%  interrupts.CPU81.RES:Rescheduling_interrupts
     46580 ±  3%     +11.7%      52016        interrupts.CPU82.LOC:Local_timer_interrupts
     99.50 ± 49%     -50.5%      49.25 ± 51%  interrupts.CPU82.RES:Rescheduling_interrupts
     46554 ±  3%     +11.7%      51996        interrupts.CPU83.LOC:Local_timer_interrupts
      1112 ± 16%     -50.6%     549.50 ± 46%  interrupts.CPU83.NMI:Non-maskable_interrupts
      1112 ± 16%     -50.6%     549.50 ± 46%  interrupts.CPU83.PMI:Performance_monitoring_interrupts
     46552 ±  3%     +11.7%      51995        interrupts.CPU84.LOC:Local_timer_interrupts
     46559 ±  3%     +11.8%      52052        interrupts.CPU85.LOC:Local_timer_interrupts
    985.00 ± 31%     -48.5%     507.25 ±  5%  interrupts.CPU85.NMI:Non-maskable_interrupts
    985.00 ± 31%     -48.5%     507.25 ±  5%  interrupts.CPU85.PMI:Performance_monitoring_interrupts
     46561 ±  3%     +11.7%      51995        interrupts.CPU86.LOC:Local_timer_interrupts
    807.25 ±  6%     -45.9%     436.50 ± 28%  interrupts.CPU86.NMI:Non-maskable_interrupts
    807.25 ±  6%     -45.9%     436.50 ± 28%  interrupts.CPU86.PMI:Performance_monitoring_interrupts
     46540 ±  3%     +11.7%      51995        interrupts.CPU87.LOC:Local_timer_interrupts
    834.50 ± 20%     -45.2%     457.50 ± 57%  interrupts.CPU87.NMI:Non-maskable_interrupts
    834.50 ± 20%     -45.2%     457.50 ± 57%  interrupts.CPU87.PMI:Performance_monitoring_interrupts
     46604 ±  3%     +11.5%      51950 ±  2%  interrupts.CPU88.LOC:Local_timer_interrupts
    814.50 ± 15%     -52.0%     391.25 ± 30%  interrupts.CPU88.NMI:Non-maskable_interrupts
    814.50 ± 15%     -52.0%     391.25 ± 30%  interrupts.CPU88.PMI:Performance_monitoring_interrupts
     46563 ±  3%     +11.5%      51929 ±  2%  interrupts.CPU89.LOC:Local_timer_interrupts
    881.75 ± 39%     -51.2%     430.50 ± 22%  interrupts.CPU89.NMI:Non-maskable_interrupts
    881.75 ± 39%     -51.2%     430.50 ± 22%  interrupts.CPU89.PMI:Performance_monitoring_interrupts
     46354 ±  3%     +12.0%      51935        interrupts.CPU9.LOC:Local_timer_interrupts
     46561 ±  3%     +11.4%      51891        interrupts.CPU90.LOC:Local_timer_interrupts
      1047 ± 41%     -56.5%     455.75 ± 25%  interrupts.CPU90.NMI:Non-maskable_interrupts
      1047 ± 41%     -56.5%     455.75 ± 25%  interrupts.CPU90.PMI:Performance_monitoring_interrupts
     46529 ±  3%     +11.5%      51896        interrupts.CPU91.LOC:Local_timer_interrupts
    856.00 ± 23%     -47.2%     452.25 ± 42%  interrupts.CPU91.NMI:Non-maskable_interrupts
    856.00 ± 23%     -47.2%     452.25 ± 42%  interrupts.CPU91.PMI:Performance_monitoring_interrupts
     46565 ±  3%     +11.7%      51995        interrupts.CPU92.LOC:Local_timer_interrupts
    889.75 ± 34%     -45.3%     486.50 ± 50%  interrupts.CPU92.NMI:Non-maskable_interrupts
    889.75 ± 34%     -45.3%     486.50 ± 50%  interrupts.CPU92.PMI:Performance_monitoring_interrupts
     46577 ±  3%     +11.4%      51892 ±  2%  interrupts.CPU93.LOC:Local_timer_interrupts
    797.50 ±  8%     -38.2%     492.75 ± 35%  interrupts.CPU93.NMI:Non-maskable_interrupts
    797.50 ±  8%     -38.2%     492.75 ± 35%  interrupts.CPU93.PMI:Performance_monitoring_interrupts
     46564 ±  3%     +11.5%      51903        interrupts.CPU94.LOC:Local_timer_interrupts
    749.75 ± 23%     -34.6%     490.00 ± 22%  interrupts.CPU94.NMI:Non-maskable_interrupts
    749.75 ± 23%     -34.6%     490.00 ± 22%  interrupts.CPU94.PMI:Performance_monitoring_interrupts
      2895 ± 17%     -25.5%       2158 ± 18%  interrupts.CPU95.CAL:Function_call_interrupts
     46572 ±  3%     +11.4%      51890        interrupts.CPU95.LOC:Local_timer_interrupts
     46376 ±  3%     +11.9%      51882 ±  2%  interrupts.CPU96.LOC:Local_timer_interrupts
     46357 ±  3%     +12.0%      51916        interrupts.CPU97.LOC:Local_timer_interrupts
     46397 ±  3%     +12.1%      51999        interrupts.CPU98.LOC:Local_timer_interrupts
     46448 ±  2%     +11.8%      51930 ±  2%  interrupts.CPU99.LOC:Local_timer_interrupts
   8914517 ±  3%     +11.9%    9973298        interrupts.LOC:Local_timer_interrupts
     33.74            -2.4       31.38 ±  4%  perf-profile.calltrace.cycles-pp.ret_from_fork
     33.74            -2.4       31.38 ±  4%  perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
     17.77 ±  2%      -2.2       15.55 ±  5%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
     17.53 ±  2%      -2.1       15.47 ±  5%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
     22.36            -2.0       20.38 ±  4%  perf-profile.calltrace.cycles-pp.nfsd.kthread.ret_from_fork
     16.52 ±  2%      -1.9       14.63 ±  4%  perf-profile.calltrace.cycles-pp.svc_process.nfsd.kthread.ret_from_fork
     16.52 ±  2%      -1.9       14.63 ±  4%  perf-profile.calltrace.cycles-pp.svc_process_common.svc_process.nfsd.kthread.ret_from_fork
     16.44 ±  2%      -1.9       14.57 ±  3%  perf-profile.calltrace.cycles-pp.nfsd_dispatch.svc_process_common.svc_process.nfsd.kthread
     16.38 ±  3%      -1.8       14.54 ±  4%  perf-profile.calltrace.cycles-pp.nfsd4_proc_compound.nfsd_dispatch.svc_process_common.svc_process.nfsd
      9.78 ±  2%      -1.2        8.60 ±  5%  perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      9.76 ±  2%      -1.2        8.58 ±  5%  perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      9.69 ±  2%      -1.2        8.53 ±  5%  perf-profile.calltrace.cycles-pp.nfs_file_write.new_sync_write.vfs_write.ksys_write.do_syscall_64
      9.70 ±  2%      -1.2        8.54 ±  5%  perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      9.34 ±  2%      -1.1        8.23 ±  5%  perf-profile.calltrace.cycles-pp.generic_perform_write.nfs_file_write.new_sync_write.vfs_write.ksys_write
     10.04 ±  3%      -1.0        9.04 ±  3%  perf-profile.calltrace.cycles-pp.nfsd_commit.nfsd4_proc_compound.nfsd_dispatch.svc_process_common.svc_process
     10.02 ±  3%      -1.0        9.03 ±  3%  perf-profile.calltrace.cycles-pp.xfs_file_fsync.nfsd_commit.nfsd4_proc_compound.nfsd_dispatch.svc_process_common
      9.98 ±  3%      -1.0        9.00 ±  3%  perf-profile.calltrace.cycles-pp.file_write_and_wait_range.xfs_file_fsync.nfsd_commit.nfsd4_proc_compound.nfsd_dispatch
      9.97 ±  3%      -1.0        8.98 ±  3%  perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync.nfsd_commit.nfsd4_proc_compound
      9.96 ±  3%      -1.0        8.98 ±  3%  perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync.nfsd_commit
      9.96 ±  3%      -1.0        8.98 ±  3%  perf-profile.calltrace.cycles-pp.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync
      5.98 ±  2%      -0.8        5.17 ±  5%  perf-profile.calltrace.cycles-pp.nfsd4_write.nfsd4_proc_compound.nfsd_dispatch.svc_process_common.svc_process
      5.95 ±  2%      -0.8        5.14 ±  5%  perf-profile.calltrace.cycles-pp.nfsd_vfs_write.nfsd4_write.nfsd4_proc_compound.nfsd_dispatch.svc_process_common
      5.94 ±  2%      -0.8        5.13 ±  5%  perf-profile.calltrace.cycles-pp.do_iter_write.nfsd_vfs_write.nfsd4_write.nfsd4_proc_compound.nfsd_dispatch
      5.92 ±  2%      -0.8        5.12 ±  5%  perf-profile.calltrace.cycles-pp.do_iter_readv_writev.do_iter_write.nfsd_vfs_write.nfsd4_write.nfsd4_proc_compound
      5.92 ±  2%      -0.8        5.12 ±  5%  perf-profile.calltrace.cycles-pp.xfs_file_buffered_aio_write.do_iter_readv_writev.do_iter_write.nfsd_vfs_write.nfsd4_write
      8.55 ±  3%      -0.8        7.76 ±  3%  perf-profile.calltrace.cycles-pp.brd_do_bvec.brd_submit_bio.submit_bio_noacct.submit_bio.iomap_submit_ioend
      5.67 ±  2%      -0.7        4.96 ±  5%  perf-profile.calltrace.cycles-pp.iomap_file_buffered_write.xfs_file_buffered_aio_write.do_iter_readv_writev.do_iter_write.nfsd_vfs_write
      5.67 ±  2%      -0.7        4.96 ±  5%  perf-profile.calltrace.cycles-pp.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.do_iter_readv_writev.do_iter_write
      5.63 ±  2%      -0.7        4.93 ±  5%  perf-profile.calltrace.cycles-pp.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.do_iter_readv_writev
      1.86 ±  4%      -0.7        1.19 ±  4%  perf-profile.calltrace.cycles-pp.iomap_writepages.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range
      1.85 ±  4%      -0.7        1.19 ±  4%  perf-profile.calltrace.cycles-pp.write_cache_pages.iomap_writepages.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range
      1.30 ±  7%      -0.6        0.66 ±  5%  perf-profile.calltrace.cycles-pp.iomap_writepage_map.write_cache_pages.iomap_writepages.xfs_vm_writepages.do_writepages
      4.55 ±  3%      -0.6        3.99 ±  4%  perf-profile.calltrace.cycles-pp.nfs_write_end.generic_perform_write.nfs_file_write.new_sync_write.vfs_write
      5.47 ±  3%      -0.5        4.93 ±  4%  perf-profile.calltrace.cycles-pp.nfs4_file_flush.filp_close.__x64_sys_close.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.47 ±  3%      -0.5        4.94 ±  4%  perf-profile.calltrace.cycles-pp.__x64_sys_close.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.47 ±  3%      -0.5        4.94 ±  4%  perf-profile.calltrace.cycles-pp.filp_close.__x64_sys_close.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.47 ±  3%      -0.5        4.93 ±  4%  perf-profile.calltrace.cycles-pp.nfs_wb_all.nfs4_file_flush.filp_close.__x64_sys_close.do_syscall_64
      4.44 ±  2%      -0.5        3.95 ±  3%  perf-profile.calltrace.cycles-pp.rpc_free_task.rpc_async_release.process_one_work.worker_thread.kthread
      4.44 ±  2%      -0.5        3.95 ±  3%  perf-profile.calltrace.cycles-pp.rpc_async_release.process_one_work.worker_thread.kthread.ret_from_fork
      3.58 ±  2%      -0.4        3.13 ±  3%  perf-profile.calltrace.cycles-pp.nfs_write_completion.rpc_free_task.rpc_async_release.process_one_work.worker_thread
      3.35 ±  4%      -0.4        2.90 ±  5%  perf-profile.calltrace.cycles-pp.nfs_updatepage.nfs_write_end.generic_perform_write.nfs_file_write.new_sync_write
      2.43 ±  3%      -0.3        2.09 ±  3%  perf-profile.calltrace.cycles-pp.nfs_end_page_writeback.nfs_write_completion.rpc_free_task.rpc_async_release.process_one_work
      2.37 ±  3%      -0.3        2.05 ±  4%  perf-profile.calltrace.cycles-pp.nfs_writepage_setup.nfs_updatepage.nfs_write_end.generic_perform_write.nfs_file_write
      0.57 ±  5%      -0.3        0.26 ±100%  perf-profile.calltrace.cycles-pp.__test_set_page_writeback.nfs_do_writepage.nfs_writepages_callback.write_cache_pages.nfs_writepages
      2.73 ±  4%      -0.3        2.43 ±  5%  perf-profile.calltrace.cycles-pp.iov_iter_copy_from_user_atomic.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
      2.69 ±  4%      -0.3        2.39 ±  5%  perf-profile.calltrace.cycles-pp.memcpy_erms.iov_iter_copy_from_user_atomic.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      2.20            -0.3        1.93 ±  4%  perf-profile.calltrace.cycles-pp.nfs_write_begin.generic_perform_write.nfs_file_write.new_sync_write.vfs_write
      2.10            -0.2        1.85 ±  4%  perf-profile.calltrace.cycles-pp.pagecache_get_page.grab_cache_page_write_begin.nfs_write_begin.generic_perform_write.nfs_file_write
      2.11            -0.2        1.86 ±  4%  perf-profile.calltrace.cycles-pp.grab_cache_page_write_begin.nfs_write_begin.generic_perform_write.nfs_file_write.new_sync_write
      0.66 ±  4%      -0.2        0.42 ± 57%  perf-profile.calltrace.cycles-pp.schedule.io_schedule.wait_on_page_bit.__filemap_fdatawait_range.filemap_write_and_wait_range
      0.65 ±  4%      -0.2        0.41 ± 57%  perf-profile.calltrace.cycles-pp.__schedule.schedule.io_schedule.wait_on_page_bit.__filemap_fdatawait_range
      1.48            -0.2        1.28 ±  5%  perf-profile.calltrace.cycles-pp.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
      0.60 ±  5%      -0.2        0.40 ± 57%  perf-profile.calltrace.cycles-pp.__nfs_commit_inode.nfs_wb_all.nfs4_file_flush.filp_close.__x64_sys_close
      0.83 ± 13%      -0.2        0.64 ±  2%  perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      1.40            -0.2        1.21 ±  5%  perf-profile.calltrace.cycles-pp.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      1.38            -0.2        1.20 ±  4%  perf-profile.calltrace.cycles-pp.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor.iomap_apply
      1.19 ±  3%      -0.2        1.02 ±  4%  perf-profile.calltrace.cycles-pp.__filemap_fdatawait_range.filemap_write_and_wait_range.nfs_wb_all.nfs4_file_flush.filp_close
      1.52            -0.2        1.35 ±  5%  perf-profile.calltrace.cycles-pp.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.nfs_write_begin.generic_perform_write
      0.78 ±  3%      -0.1        0.65 ±  4%  perf-profile.calltrace.cycles-pp.wait_on_page_bit.__filemap_fdatawait_range.filemap_write_and_wait_range.nfs_wb_all.nfs4_file_flush
      0.85 ±  2%      -0.1        0.72 ±  5%  perf-profile.calltrace.cycles-pp.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor
      0.67 ±  4%      -0.1        0.54 ±  5%  perf-profile.calltrace.cycles-pp.io_schedule.wait_on_page_bit.__filemap_fdatawait_range.filemap_write_and_wait_range.nfs_wb_all
      0.91 ±  5%      -0.1        0.79 ±  4%  perf-profile.calltrace.cycles-pp.__set_page_dirty_nobuffers.nfs_writepage_setup.nfs_updatepage.nfs_write_end.generic_perform_write
      0.83 ±  5%      -0.1        0.71 ±  4%  perf-profile.calltrace.cycles-pp.wake_up_page_bit.nfs_end_page_writeback.nfs_write_completion.rpc_free_task.rpc_async_release
      1.05 ±  2%      -0.1        0.94 ±  6%  perf-profile.calltrace.cycles-pp.__add_to_page_cache_locked.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.nfs_write_begin
      0.90 ±  4%      -0.1        0.80 ±  3%  perf-profile.calltrace.cycles-pp.end_page_writeback.nfs_end_page_writeback.nfs_write_completion.rpc_free_task.rpc_async_release
      0.88 ±  4%      -0.1        0.78 ±  3%  perf-profile.calltrace.cycles-pp.test_clear_page_writeback.end_page_writeback.nfs_end_page_writeback.nfs_write_completion.rpc_free_task
      0.70 ±  5%      -0.1        0.61 ±  5%  perf-profile.calltrace.cycles-pp.__wake_up_common.wake_up_page_bit.nfs_end_page_writeback.nfs_write_completion.rpc_free_task
      0.66 ±  4%      -0.1        0.58 ±  4%  perf-profile.calltrace.cycles-pp.wake_page_function.__wake_up_common.wake_up_page_bit.nfs_end_page_writeback.nfs_write_completion
      0.66 ±  4%      -0.1        0.57 ±  5%  perf-profile.calltrace.cycles-pp.try_to_wake_up.wake_page_function.__wake_up_common.wake_up_page_bit.nfs_end_page_writeback
      0.87 ±  5%      +0.1        0.97 ±  8%  perf-profile.calltrace.cycles-pp.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry
      0.96 ±  5%      +0.1        1.06 ±  7%  perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry.start_secondary
      0.83 ±  5%      +0.1        0.94 ±  5%  perf-profile.calltrace.cycles-pp.do_softirq_own_stack.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      0.83 ±  5%      +0.1        0.94 ±  5%  perf-profile.calltrace.cycles-pp.asm_call_sysvec_on_stack.do_softirq_own_stack.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.83 ±  5%      +0.1        0.94 ±  5%  perf-profile.calltrace.cycles-pp.__softirqentry_text_start.asm_call_sysvec_on_stack.do_softirq_own_stack.irq_exit_rcu.sysvec_apic_timer_interrupt
      1.02 ±  6%      +0.1        1.15 ±  6%  perf-profile.calltrace.cycles-pp.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      4.18 ±  2%      +0.3        4.50 ±  5%  perf-profile.calltrace.cycles-pp.svc_tcp_recvfrom.svc_handle_xprt.svc_recv.nfsd.kthread
      0.00            +0.6        0.55 ±  6%  perf-profile.calltrace.cycles-pp.skb_release_data.__kfree_skb.tcp_recvmsg.inet6_recvmsg.svc_tcp_recvfrom
      3.21            +0.6        3.80 ±  4%  perf-profile.calltrace.cycles-pp.inet6_recvmsg.svc_tcp_recvfrom.svc_handle_xprt.svc_recv.nfsd
      3.20 ±  2%      +0.6        3.79 ±  4%  perf-profile.calltrace.cycles-pp.tcp_recvmsg.inet6_recvmsg.svc_tcp_recvfrom.svc_handle_xprt.svc_recv
      0.00            +0.7        0.66 ±  5%  perf-profile.calltrace.cycles-pp.__kfree_skb.tcp_recvmsg.inet6_recvmsg.svc_tcp_recvfrom.svc_handle_xprt
      4.16 ±  5%      +1.7        5.85 ± 12%  perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      0.00            +3.1        3.13 ±  6%  perf-profile.calltrace.cycles-pp.memcpy_erms.skb_copy_ubufs.__netif_receive_skb_core.__netif_receive_skb_one_core.process_backlog
     41.73            +3.3       44.98 ±  5%  perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      0.00            +3.9        3.94 ±  6%  perf-profile.calltrace.cycles-pp.skb_copy_ubufs.__netif_receive_skb_core.__netif_receive_skb_one_core.process_backlog.net_rx_action
      0.00            +4.0        3.95 ±  6%  perf-profile.calltrace.cycles-pp.__netif_receive_skb_core.__netif_receive_skb_one_core.process_backlog.net_rx_action.__softirqentry_text_start
      0.00            +4.1        4.13 ±  6%  perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.net_rx_action.__softirqentry_text_start.asm_call_sysvec_on_stack
      0.00            +4.1        4.14 ±  6%  perf-profile.calltrace.cycles-pp.process_backlog.net_rx_action.__softirqentry_text_start.asm_call_sysvec_on_stack.do_softirq_own_stack
      0.00            +4.1        4.14 ±  6%  perf-profile.calltrace.cycles-pp.net_rx_action.__softirqentry_text_start.asm_call_sysvec_on_stack.do_softirq_own_stack.do_softirq
      0.00            +4.2        4.16 ±  6%  perf-profile.calltrace.cycles-pp.asm_call_sysvec_on_stack.do_softirq_own_stack.do_softirq.__local_bh_enable_ip.ip6_finish_output2
      0.00            +4.2        4.16 ±  6%  perf-profile.calltrace.cycles-pp.do_softirq_own_stack.do_softirq.__local_bh_enable_ip.ip6_finish_output2.ip6_output
      0.00            +4.2        4.16 ±  6%  perf-profile.calltrace.cycles-pp.__softirqentry_text_start.asm_call_sysvec_on_stack.do_softirq_own_stack.do_softirq.__local_bh_enable_ip
      0.00            +4.2        4.16 ±  6%  perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.ip6_finish_output2.ip6_output.ip6_xmit.inet6_csk_xmit
      0.00            +4.2        4.16 ±  6%  perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.ip6_finish_output2.ip6_output.ip6_xmit
      0.00            +4.2        4.20 ±  6%  perf-profile.calltrace.cycles-pp.ip6_finish_output2.ip6_output.ip6_xmit.inet6_csk_xmit.__tcp_transmit_skb
      0.00            +4.2        4.22 ±  6%  perf-profile.calltrace.cycles-pp.ip6_output.ip6_xmit.inet6_csk_xmit.__tcp_transmit_skb.tcp_write_xmit
      0.00            +4.2        4.22 ±  6%  perf-profile.calltrace.cycles-pp.ip6_xmit.inet6_csk_xmit.__tcp_transmit_skb.tcp_write_xmit.tcp_sendmsg_locked
      0.00            +4.2        4.25 ±  6%  perf-profile.calltrace.cycles-pp.inet6_csk_xmit.__tcp_transmit_skb.tcp_write_xmit.tcp_sendmsg_locked.tcp_sendmsg
      0.00            +4.3        4.30 ±  6%  perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg
      0.00            +4.4        4.36 ±  6%  perf-profile.calltrace.cycles-pp.tcp_write_xmit.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.xprt_sock_sendmsg
     47.71            +4.6       52.32 ±  4%  perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
     47.45            +4.7       52.16 ±  4%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
     47.43            +4.7       52.15 ±  4%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     47.44            +4.7       52.16 ±  4%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     33.74            -2.4       31.38 ±  4%  perf-profile.children.cycles-pp.ret_from_fork
     33.74            -2.4       31.38 ±  4%  perf-profile.children.cycles-pp.kthread
     18.18 ±  2%      -2.2       15.97 ±  5%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     17.93 ±  2%      -2.0       15.89 ±  5%  perf-profile.children.cycles-pp.do_syscall_64
     22.36            -2.0       20.38 ±  4%  perf-profile.children.cycles-pp.nfsd
     16.52 ±  2%      -1.9       14.63 ±  4%  perf-profile.children.cycles-pp.svc_process
     16.52 ±  2%      -1.9       14.63 ±  4%  perf-profile.children.cycles-pp.svc_process_common
     16.44 ±  2%      -1.9       14.57 ±  3%  perf-profile.children.cycles-pp.nfsd_dispatch
     16.38 ±  3%      -1.8       14.54 ±  4%  perf-profile.children.cycles-pp.nfsd4_proc_compound
     13.64 ±  3%      -1.3       12.38 ±  3%  perf-profile.children.cycles-pp.__filemap_fdatawrite_range
     13.64 ±  3%      -1.3       12.38 ±  3%  perf-profile.children.cycles-pp.do_writepages
     10.04 ±  2%      -1.2        8.87 ±  5%  perf-profile.children.cycles-pp.ksys_write
     10.02 ±  2%      -1.2        8.85 ±  5%  perf-profile.children.cycles-pp.vfs_write
      9.70 ±  2%      -1.2        8.53 ±  5%  perf-profile.children.cycles-pp.nfs_file_write
      9.96 ±  2%      -1.2        8.81 ±  5%  perf-profile.children.cycles-pp.new_sync_write
      9.35 ±  2%      -1.1        8.23 ±  5%  perf-profile.children.cycles-pp.generic_perform_write
     10.04 ±  3%      -1.0        9.04 ±  3%  perf-profile.children.cycles-pp.nfsd_commit
     10.02 ±  3%      -1.0        9.03 ±  3%  perf-profile.children.cycles-pp.xfs_file_fsync
      9.98 ±  3%      -1.0        9.00 ±  3%  perf-profile.children.cycles-pp.file_write_and_wait_range
      9.96 ±  3%      -1.0        8.98 ±  3%  perf-profile.children.cycles-pp.xfs_vm_writepages
      5.51 ±  3%      -0.9        4.56 ±  4%  perf-profile.children.cycles-pp.write_cache_pages
      8.74 ±  3%      -0.9        7.84 ±  3%  perf-profile.children.cycles-pp.submit_bio
      8.74 ±  3%      -0.9        7.84 ±  3%  perf-profile.children.cycles-pp.submit_bio_noacct
      8.68 ±  3%      -0.9        7.79 ±  3%  perf-profile.children.cycles-pp.iomap_submit_ioend
      8.73 ±  3%      -0.9        7.84 ±  3%  perf-profile.children.cycles-pp.brd_submit_bio
      8.70 ±  3%      -0.9        7.81 ±  3%  perf-profile.children.cycles-pp.brd_do_bvec
      5.98 ±  2%      -0.8        5.17 ±  5%  perf-profile.children.cycles-pp.nfsd4_write
      5.95 ±  2%      -0.8        5.14 ±  5%  perf-profile.children.cycles-pp.nfsd_vfs_write
      5.94 ±  2%      -0.8        5.13 ±  5%  perf-profile.children.cycles-pp.do_iter_write
      5.92 ±  2%      -0.8        5.12 ±  5%  perf-profile.children.cycles-pp.do_iter_readv_writev
      5.92 ±  2%      -0.8        5.12 ±  5%  perf-profile.children.cycles-pp.xfs_file_buffered_aio_write
      5.67 ±  2%      -0.7        4.96 ±  5%  perf-profile.children.cycles-pp.iomap_file_buffered_write
      5.67 ±  2%      -0.7        4.96 ±  5%  perf-profile.children.cycles-pp.iomap_apply
      5.63 ±  2%      -0.7        4.94 ±  5%  perf-profile.children.cycles-pp.iomap_write_actor
      1.86 ±  4%      -0.7        1.19 ±  4%  perf-profile.children.cycles-pp.iomap_writepages
      1.31 ±  7%      -0.6        0.67 ±  5%  perf-profile.children.cycles-pp.iomap_writepage_map
      5.90 ±  4%      -0.6        5.30 ±  3%  perf-profile.children.cycles-pp.brd_insert_page
      5.00 ±  3%      -0.6        4.44 ±  6%  perf-profile.children.cycles-pp.iov_iter_copy_from_user_atomic
      4.56 ±  3%      -0.6        4.00 ±  4%  perf-profile.children.cycles-pp.nfs_write_end
      5.47 ±  3%      -0.5        4.94 ±  4%  perf-profile.children.cycles-pp.filp_close
      5.47 ±  3%      -0.5        4.94 ±  4%  perf-profile.children.cycles-pp.__x64_sys_close
      5.47 ±  3%      -0.5        4.93 ±  4%  perf-profile.children.cycles-pp.nfs4_file_flush
      5.47 ±  3%      -0.5        4.93 ±  4%  perf-profile.children.cycles-pp.nfs_wb_all
      4.69 ±  2%      -0.5        4.16 ±  3%  perf-profile.children.cycles-pp.rpc_free_task
      5.35 ±  5%      -0.5        4.83 ±  2%  perf-profile.children.cycles-pp._raw_spin_lock
      4.44 ±  2%      -0.5        3.95 ±  3%  perf-profile.children.cycles-pp.rpc_async_release
      3.58 ±  2%      -0.4        3.13 ±  3%  perf-profile.children.cycles-pp.nfs_write_completion
      3.35 ±  4%      -0.4        2.91 ±  5%  perf-profile.children.cycles-pp.nfs_updatepage
      3.51            -0.4        3.08 ±  4%  perf-profile.children.cycles-pp.grab_cache_page_write_begin
      3.48            -0.4        3.06 ±  4%  perf-profile.children.cycles-pp.pagecache_get_page
      2.43 ±  3%      -0.3        2.09 ±  3%  perf-profile.children.cycles-pp.nfs_end_page_writeback
      1.91 ±  2%      -0.3        1.58 ±  3%  perf-profile.children.cycles-pp.__schedule
      2.38 ±  3%      -0.3        2.06 ±  4%  perf-profile.children.cycles-pp.nfs_writepage_setup
      2.37            -0.3        2.07 ±  5%  perf-profile.children.cycles-pp.add_to_page_cache_lru
      1.54            -0.3        1.27 ±  3%  perf-profile.children.cycles-pp.schedule
      2.20            -0.3        1.93 ±  4%  perf-profile.children.cycles-pp.nfs_write_begin
      1.49            -0.2        1.29 ±  5%  perf-profile.children.cycles-pp.iomap_write_begin
      0.84 ± 13%      -0.2        0.64 ±  2%  perf-profile.children.cycles-pp.poll_idle
      1.11 ±  2%      -0.2        0.92 ±  4%  perf-profile.children.cycles-pp.pick_next_task_fair
      1.56 ±  2%      -0.2        1.38 ±  6%  perf-profile.children.cycles-pp.__add_to_page_cache_locked
      1.21 ±  2%      -0.2        1.04 ±  5%  perf-profile.children.cycles-pp.__filemap_fdatawait_range
      0.98 ±  2%      -0.2        0.81 ±  3%  perf-profile.children.cycles-pp.newidle_balance
      1.05            -0.2        0.88 ±  4%  perf-profile.children.cycles-pp.ttwu_do_activate
      0.23 ±102%      -0.2        0.06 ±  6%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
      1.04 ±  2%      -0.2        0.88 ±  5%  perf-profile.children.cycles-pp.enqueue_task_fair
      1.08 ±  7%      -0.2        0.93 ±  3%  perf-profile.children.cycles-pp.percpu_counter_add_batch
      1.21 ±  2%      -0.2        1.05 ±  5%  perf-profile.children.cycles-pp.try_to_wake_up
      1.10            -0.2        0.94 ±  3%  perf-profile.children.cycles-pp.load_balance
      0.60 ±  6%      -0.2        0.45 ±  7%  perf-profile.children.cycles-pp.tcp_rcv_established
      0.61 ±  6%      -0.1        0.46 ±  7%  perf-profile.children.cycles-pp.tcp_v6_do_rcv
      1.00            -0.1        0.86 ±  3%  perf-profile.children.cycles-pp.find_busiest_group
      0.96            -0.1        0.82 ±  6%  perf-profile.children.cycles-pp.enqueue_entity
      0.79 ±  3%      -0.1        0.65 ±  5%  perf-profile.children.cycles-pp.wait_on_page_bit
      0.99            -0.1        0.85 ±  2%  perf-profile.children.cycles-pp.update_sd_lb_stats
      0.30 ±  5%      -0.1        0.16 ± 10%  perf-profile.children.cycles-pp.tcp_ack
      0.61 ±  3%      -0.1        0.48 ±  3%  perf-profile.children.cycles-pp.flush_smp_call_function_from_idle
      0.28 ±  5%      -0.1        0.15 ± 12%  perf-profile.children.cycles-pp.tcp_clean_rtx_queue
      1.45 ±  3%      -0.1        1.32 ±  3%  perf-profile.children.cycles-pp.end_page_writeback
      0.65 ±  2%      -0.1        0.52 ±  3%  perf-profile.children.cycles-pp.sched_ttwu_pending
      0.76 ±  7%      -0.1        0.64 ±  4%  perf-profile.children.cycles-pp.account_page_dirtied
      1.43 ±  3%      -0.1        1.31 ±  3%  perf-profile.children.cycles-pp.test_clear_page_writeback
      0.97 ±  5%      -0.1        0.84 ±  4%  perf-profile.children.cycles-pp.__set_page_dirty_nobuffers
      0.68 ±  3%      -0.1        0.55 ±  5%  perf-profile.children.cycles-pp.io_schedule
      0.74            -0.1        0.62 ±  6%  perf-profile.children.cycles-pp.__account_scheduler_latency
      0.45 ±  5%      -0.1        0.33 ±  7%  perf-profile.children.cycles-pp.__release_sock
      0.85 ±  5%      -0.1        0.73 ±  4%  perf-profile.children.cycles-pp.wake_up_page_bit
      0.56 ±  3%      -0.1        0.44 ±  7%  perf-profile.children.cycles-pp.release_sock
      0.89 ±  3%      -0.1        0.78 ±  4%  perf-profile.children.cycles-pp.__wake_up_common
      0.98 ±  2%      -0.1        0.87 ±  4%  perf-profile.children.cycles-pp.__test_set_page_writeback
      0.77            -0.1        0.66 ±  5%  perf-profile.children.cycles-pp.lru_cache_add
      0.55 ±  8%      -0.1        0.44 ±  4%  perf-profile.children.cycles-pp.iomap_set_page_dirty
      0.99 ±  4%      -0.1        0.88 ±  5%  perf-profile.children.cycles-pp.__nfs_commit_inode
      0.23 ±  8%      -0.1        0.14 ± 11%  perf-profile.children.cycles-pp.rwsem_spin_on_owner
      0.21 ±  9%      -0.1        0.12 ± 17%  perf-profile.children.cycles-pp.xfs_ilock
      0.80 ±  3%      -0.1        0.71 ±  5%  perf-profile.children.cycles-pp.__mod_memcg_lruvec_state
      0.71 ±  2%      -0.1        0.62 ±  5%  perf-profile.children.cycles-pp.pagevec_lru_move_fn
      0.68 ±  5%      -0.1        0.59 ±  4%  perf-profile.children.cycles-pp.wake_page_function
      0.58            -0.1        0.48 ±  6%  perf-profile.children.cycles-pp.stack_trace_save_tsk
      0.29 ± 11%      -0.1        0.20 ± 13%  perf-profile.children.cycles-pp.rwsem_down_write_slowpath
      0.52 ±  4%      -0.1        0.43 ±  6%  perf-profile.children.cycles-pp.arch_stack_walk
      0.28 ± 11%      -0.1        0.20 ± 15%  perf-profile.children.cycles-pp.rwsem_optimistic_spin
      0.54 ±  2%      -0.1        0.46 ±  4%  perf-profile.children.cycles-pp.__pagevec_lru_add_fn
      0.55 ±  7%      -0.1        0.48 ±  7%  perf-profile.children.cycles-pp.__mod_memcg_state
      0.72 ±  3%      -0.1        0.65 ±  5%  perf-profile.children.cycles-pp.nfs_scan_commit
      0.75            -0.1        0.68 ±  5%  perf-profile.children.cycles-pp.mem_cgroup_charge
      0.38 ± 13%      -0.1        0.31 ±  6%  perf-profile.children.cycles-pp.__set_page_dirty
      0.37 ±  2%      -0.1        0.30 ±  3%  perf-profile.children.cycles-pp.unwind_next_frame
      0.21 ± 10%      -0.1        0.15 ± 18%  perf-profile.children.cycles-pp.svc_xprt_do_enqueue
      0.41 ±  5%      -0.1        0.35 ±  4%  perf-profile.children.cycles-pp.schedule_idle
      0.51 ±  2%      -0.1        0.45 ±  9%  perf-profile.children.cycles-pp.nfs_ctx_key_to_expire
      0.49 ±  5%      -0.1        0.43 ±  7%  perf-profile.children.cycles-pp.nfs_create_request
      0.18 ±  5%      -0.1        0.12 ± 16%  perf-profile.children.cycles-pp.svc_xprt_received
      0.39 ±  4%      -0.1        0.34 ±  6%  perf-profile.children.cycles-pp.xas_load
      0.23 ±  6%      -0.1        0.18 ±  2%  perf-profile.children.cycles-pp.dequeue_task_fair
      0.75            -0.1        0.70        perf-profile.children.cycles-pp.clear_page_dirty_for_io
      0.21 ±  7%      -0.0        0.16 ±  4%  perf-profile.children.cycles-pp.dequeue_entity
      0.33            -0.0        0.28 ±  2%  perf-profile.children.cycles-pp.nfs_lock_and_join_requests
      0.15 ±  3%      -0.0        0.11 ±  3%  perf-profile.children.cycles-pp.clear_bdi_congested
      0.16 ± 12%      -0.0        0.12 ±  3%  perf-profile.children.cycles-pp.schedule_timeout
      0.12 ± 11%      -0.0        0.08 ±  8%  perf-profile.children.cycles-pp.iov_iter_advance
      0.35 ±  3%      -0.0        0.32 ±  2%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.16 ±  5%      -0.0        0.12 ±  3%  perf-profile.children.cycles-pp.lock_sock_nested
      0.06            -0.0        0.03 ±100%  perf-profile.children.cycles-pp.__inc_numa_state
      0.38            -0.0        0.35 ±  4%  perf-profile.children.cycles-pp.nfs_scan_commit_list
      0.25 ±  7%      -0.0        0.22 ±  8%  perf-profile.children.cycles-pp.nfs_initiate_commit
      0.25 ±  3%      -0.0        0.22 ±  8%  perf-profile.children.cycles-pp.get_mem_cgroup_from_mm
      0.11 ±  4%      -0.0        0.08 ±  6%  perf-profile.children.cycles-pp.__wait_on_bit
      0.39 ±  2%      -0.0        0.36 ±  3%  perf-profile.children.cycles-pp.nfs_io_completion_release
      0.33 ±  3%      -0.0        0.30 ±  4%  perf-profile.children.cycles-pp.find_get_pages_range_tag
      0.13 ± 10%      -0.0        0.10 ±  8%  perf-profile.children.cycles-pp.update_curr
      0.11 ±  4%      -0.0        0.08 ±  5%  perf-profile.children.cycles-pp.out_of_line_wait_on_bit
      0.33 ±  3%      -0.0        0.30 ±  2%  perf-profile.children.cycles-pp.pagevec_lookup_range_tag
      0.18 ±  3%      -0.0        0.15 ±  5%  perf-profile.children.cycles-pp.xs_stream_data_receive_workfn
      0.18 ±  3%      -0.0        0.15 ±  5%  perf-profile.children.cycles-pp.xs_stream_data_receive
      0.13 ±  8%      -0.0        0.10 ±  7%  perf-profile.children.cycles-pp.__x64_sys_gettimeofday
      0.28 ±  2%      -0.0        0.25 ±  5%  perf-profile.children.cycles-pp.kmem_cache_alloc
      0.15 ±  6%      -0.0        0.12 ±  6%  perf-profile.children.cycles-pp.nfs_page_find_private_request
      0.10            -0.0        0.08 ±  6%  perf-profile.children.cycles-pp.rpc_wait_bit_killable
      0.08 ±  5%      -0.0        0.06 ±  7%  perf-profile.children.cycles-pp.exit_to_user_mode_prepare
      0.14            -0.0        0.12 ±  9%  perf-profile.children.cycles-pp.sysvec_call_function_single
      0.16 ±  2%      -0.0        0.14 ±  8%  perf-profile.children.cycles-pp.xas_store
      0.15 ±  7%      -0.0        0.12 ±  6%  perf-profile.children.cycles-pp.find_get_entry
      0.11 ±  4%      -0.0        0.09 ±  4%  perf-profile.children.cycles-pp.xs_sock_recvmsg
      0.11 ±  7%      -0.0        0.09 ±  9%  perf-profile.children.cycles-pp.__xa_set_mark
      0.10 ±  7%      -0.0        0.08 ±  8%  perf-profile.children.cycles-pp.set_next_entity
      0.10 ±  7%      -0.0        0.08        perf-profile.children.cycles-pp.__fput
      0.08 ± 10%      -0.0        0.06 ±  7%  perf-profile.children.cycles-pp._raw_spin_lock_bh
      0.07 ± 11%      -0.0        0.05 ±  8%  perf-profile.children.cycles-pp.task_work_run
      0.09 ±  4%      -0.0        0.07 ±  6%  perf-profile.children.cycles-pp.tick_nohz_idle_exit
      0.09 ±  9%      -0.0        0.07 ± 10%  perf-profile.children.cycles-pp.xas_set_mark
      0.08 ±  5%      -0.0        0.07 ± 12%  perf-profile.children.cycles-pp.__switch_to
      0.10 ±  5%      -0.0        0.08 ±  8%  perf-profile.children.cycles-pp.xa_get_order
      0.08 ± 10%      -0.0        0.06 ±  6%  perf-profile.children.cycles-pp.refcount_dec_and_lock
      0.11 ±  4%      -0.0        0.10 ±  7%  perf-profile.children.cycles-pp._cond_resched
      0.07            -0.0        0.06 ±  7%  perf-profile.children.cycles-pp.nfsd_set_fh_dentry
      0.06 ± 11%      +0.0        0.08 ±  5%  perf-profile.children.cycles-pp.sk_stream_alloc_skb
      0.11 ±  8%      +0.0        0.14 ±  7%  perf-profile.children.cycles-pp.__slab_free
      0.29 ±  5%      +0.0        0.34 ±  6%  perf-profile.children.cycles-pp.__tcp_push_pending_frames
      0.44 ±  5%      +0.1        0.49 ±  5%  perf-profile.children.cycles-pp.rebalance_domains
      0.26 ±  4%      +0.1        0.31 ±  7%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.01 ±173%      +0.1        0.07 ±  7%  perf-profile.children.cycles-pp.kfree
      0.47 ±  5%      +0.1        0.54 ±  7%  perf-profile.children.cycles-pp.timekeeping_max_deferment
      0.30 ±  6%      +0.1        0.38 ± 10%  perf-profile.children.cycles-pp.svc_tcp_sendto
      0.33 ±  6%      +0.1        0.41 ± 10%  perf-profile.children.cycles-pp.svc_send
      0.97 ±  5%      +0.1        1.07 ±  7%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.00            +0.1        0.13 ±  6%  perf-profile.children.cycles-pp.pskb_expand_head
      0.00            +0.1        0.14 ± 16%  perf-profile.children.cycles-pp.iov_iter_get_pages
      0.10 ± 15%      +0.2        0.26 ±  9%  perf-profile.children.cycles-pp.__check_object_size
      0.08 ± 13%      +0.2        0.26 ± 10%  perf-profile.children.cycles-pp.simple_copy_to_iter
      0.00            +0.2        0.18 ± 12%  perf-profile.children.cycles-pp.__zerocopy_sg_from_iter
      0.00            +0.2        0.19 ± 12%  perf-profile.children.cycles-pp.skb_zerocopy_iter_stream
      0.28 ± 17%      +0.3        0.55 ±  7%  perf-profile.children.cycles-pp.free_pcppages_bulk
      0.50 ±  6%      +0.3        0.79 ±  9%  perf-profile.children.cycles-pp.rmqueue_bulk
      0.46 ±  5%      +0.3        0.77 ±  7%  perf-profile.children.cycles-pp.free_unref_page
      4.19 ±  2%      +0.3        4.50 ±  5%  perf-profile.children.cycles-pp.svc_tcp_recvfrom
      0.79 ±  2%      +0.4        1.18 ±  6%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.24 ±  6%      +0.4        0.63 ±  6%  perf-profile.children.cycles-pp.skb_release_data
      0.33 ±  6%      +0.4        0.76 ±  5%  perf-profile.children.cycles-pp.__kfree_skb
      3.37            +0.6        3.95 ±  4%  perf-profile.children.cycles-pp.inet6_recvmsg
      3.36 ±  2%      +0.6        3.94 ±  4%  perf-profile.children.cycles-pp.tcp_recvmsg
      4.19 ±  5%      +1.7        5.87 ± 12%  perf-profile.children.cycles-pp.menu_select
      5.57 ±  2%      +2.8        8.38 ±  5%  perf-profile.children.cycles-pp.memcpy_erms
     41.96            +3.1       45.11 ±  5%  perf-profile.children.cycles-pp.cpuidle_enter
     41.96            +3.1       45.11 ±  5%  perf-profile.children.cycles-pp.cpuidle_enter_state
      0.74 ±  5%      +4.0        4.72 ±  5%  perf-profile.children.cycles-pp.tcp_write_xmit
      0.80 ±  6%      +4.0        4.83 ±  5%  perf-profile.children.cycles-pp.__tcp_transmit_skb
      0.71 ±  6%      +4.0        4.74 ±  5%  perf-profile.children.cycles-pp.inet6_csk_xmit
      0.63 ±  5%      +4.0        4.68 ±  5%  perf-profile.children.cycles-pp.ip6_output
      0.59 ±  7%      +4.0        4.64 ±  6%  perf-profile.children.cycles-pp.ip6_finish_output2
      0.65 ±  6%      +4.1        4.70 ±  5%  perf-profile.children.cycles-pp.ip6_xmit
      0.43 ±  8%      +4.1        4.49 ±  6%  perf-profile.children.cycles-pp.__netif_receive_skb_one_core
      0.50 ±  8%      +4.1        4.56 ±  6%  perf-profile.children.cycles-pp.__local_bh_enable_ip
      0.45 ±  9%      +4.1        4.51 ±  6%  perf-profile.children.cycles-pp.process_backlog
      0.46 ±  8%      +4.1        4.53 ±  6%  perf-profile.children.cycles-pp.net_rx_action
      0.49 ±  8%      +4.1        4.56 ±  6%  perf-profile.children.cycles-pp.do_softirq
      0.00            +4.1        4.10 ±  6%  perf-profile.children.cycles-pp.skb_copy_ubufs
      0.00            +4.1        4.12 ±  5%  perf-profile.children.cycles-pp.__netif_receive_skb_core
      1.40 ±  6%      +4.2        5.56 ±  4%  perf-profile.children.cycles-pp.__softirqentry_text_start
      1.40 ±  6%      +4.2        5.56 ±  4%  perf-profile.children.cycles-pp.do_softirq_own_stack
      5.04 ±  5%      +4.2        9.28 ±  7%  perf-profile.children.cycles-pp.asm_call_sysvec_on_stack
     47.71            +4.6       52.32 ±  4%  perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     47.71            +4.6       52.32 ±  4%  perf-profile.children.cycles-pp.cpu_startup_entry
     47.71            +4.6       52.31 ±  4%  perf-profile.children.cycles-pp.do_idle
     47.45            +4.7       52.16 ±  4%  perf-profile.children.cycles-pp.start_secondary
      1.83 ±  6%      -0.3        1.53 ± 13%  perf-profile.self.cycles-pp.ktime_get
      2.21 ±  2%      -0.2        1.97 ±  2%  perf-profile.self.cycles-pp.brd_do_bvec
      2.93 ±  2%      -0.2        2.70 ±  3%  perf-profile.self.cycles-pp._raw_spin_lock
      0.81 ± 12%      -0.2        0.62 ±  2%  perf-profile.self.cycles-pp.poll_idle
      0.97 ±  7%      -0.1        0.82 ±  3%  perf-profile.self.cycles-pp.percpu_counter_add_batch
      0.80 ±  7%      -0.1        0.69 ±  7%  perf-profile.self.cycles-pp.nfs_writepage_setup
      0.67            -0.1        0.56 ±  2%  perf-profile.self.cycles-pp.update_sd_lb_stats
      0.50 ±  6%      -0.1        0.39 ±  4%  perf-profile.self.cycles-pp.nfs_end_page_writeback
      0.92 ±  3%      -0.1        0.81 ±  6%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.47 ±  6%      -0.1        0.37 ±  9%  perf-profile.self.cycles-pp.rmqueue
      0.36 ±  7%      -0.1        0.27 ± 10%  perf-profile.self.cycles-pp.svc_tcp_recvfrom
      0.23 ±  8%      -0.1        0.14 ± 11%  perf-profile.self.cycles-pp.rwsem_spin_on_owner
      0.83 ±  3%      -0.1        0.75 ±  4%  perf-profile.self.cycles-pp.nfs_write_end
      0.54 ±  6%      -0.1        0.46 ±  7%  perf-profile.self.cycles-pp.__mod_memcg_state
      0.25 ±  6%      -0.1        0.18 ± 22%  perf-profile.self.cycles-pp.ktime_get_update_offsets_now
      0.08 ±  5%      -0.1        0.03 ±102%  perf-profile.self.cycles-pp.svc_recv
      0.30 ±  4%      -0.0        0.26 ±  8%  perf-profile.self.cycles-pp.xas_load
      0.15 ±  5%      -0.0        0.11 ±  3%  perf-profile.self.cycles-pp.clear_bdi_congested
      0.19 ±  2%      -0.0        0.15 ±  5%  perf-profile.self.cycles-pp.unwind_next_frame
      0.11 ± 14%      -0.0        0.08 ±  6%  perf-profile.self.cycles-pp.iov_iter_advance
      0.25 ±  4%      -0.0        0.22 ±  8%  perf-profile.self.cycles-pp.__pagevec_lru_add_fn
      0.08 ±  5%      -0.0        0.06 ±  9%  perf-profile.self.cycles-pp.enqueue_task_fair
      0.29 ±  2%      -0.0        0.26 ±  4%  perf-profile.self.cycles-pp.clear_page_dirty_for_io
      0.24 ±  2%      -0.0        0.22 ±  6%  perf-profile.self.cycles-pp.get_mem_cgroup_from_mm
      0.21 ±  3%      -0.0        0.18 ±  5%  perf-profile.self.cycles-pp.get_page_from_freelist
      0.19 ±  6%      -0.0        0.17 ±  3%  perf-profile.self.cycles-pp.__schedule
      0.16 ±  4%      -0.0        0.13 ±  8%  perf-profile.self.cycles-pp.try_to_wake_up
      0.11 ± 11%      -0.0        0.09 ± 13%  perf-profile.self.cycles-pp.alloc_pages_current
      0.10 ±  8%      -0.0        0.08 ±  5%  perf-profile.self.cycles-pp.brd_insert_page
      0.26 ±  2%      -0.0        0.24 ±  2%  perf-profile.self.cycles-pp.find_get_pages_range_tag
      0.08 ±  6%      -0.0        0.05 ±  9%  perf-profile.self.cycles-pp.select_task_rq_fair
      0.07 ±  5%      -0.0        0.05 ±  8%  perf-profile.self.cycles-pp.iomap_writepage_map
      0.08 ±  5%      -0.0        0.06 ± 11%  perf-profile.self.cycles-pp.__switch_to
      0.15 ±  5%      -0.0        0.13 ±  3%  perf-profile.self.cycles-pp.release_pages
      0.09 ±  4%      -0.0        0.07 ±  5%  perf-profile.self.cycles-pp.nfs_scan_commit_list
      0.08 ±  5%      -0.0        0.07 ± 10%  perf-profile.self.cycles-pp.xas_set_mark
      0.14 ±  7%      +0.0        0.17 ±  9%  perf-profile.self.cycles-pp.skb_release_data
      0.11 ±  8%      +0.0        0.14 ±  7%  perf-profile.self.cycles-pp.__slab_free
      0.26 ±  4%      +0.1        0.31 ±  7%  perf-profile.self.cycles-pp.native_irq_return_iret
      0.01 ±173%      +0.1        0.07 ±  7%  perf-profile.self.cycles-pp.kfree
      0.47 ±  5%      +0.1        0.54 ±  7%  perf-profile.self.cycles-pp.timekeeping_max_deferment
      0.00            +0.1        0.10 ±  5%  perf-profile.self.cycles-pp.pskb_expand_head
      0.12 ±  5%      +0.1        0.24 ±  5%  perf-profile.self.cycles-pp.free_pcppages_bulk
      0.00            +0.1        0.14 ± 16%  perf-profile.self.cycles-pp.iov_iter_get_pages
      0.00            +0.1        0.14 ±  5%  perf-profile.self.cycles-pp.skb_copy_ubufs
      0.07 ± 15%      +0.2        0.23 ±  6%  perf-profile.self.cycles-pp.__check_object_size
      0.78 ±  2%      +0.4        1.16 ±  6%  perf-profile.self.cycles-pp.__list_del_entry_valid
      2.82 ±  7%      +1.1        3.88 ± 18%  perf-profile.self.cycles-pp.cpuidle_enter_state
      3.14 ±  7%      +1.6        4.71 ± 15%  perf-profile.self.cycles-pp.menu_select
      5.50 ±  2%      +2.8        8.30 ±  5%  perf-profile.self.cycles-pp.memcpy_erms





Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.10.0-rc2-00001-g3bc6a407d1a0" of type "text/plain" (170174 bytes)

View attachment "job-script" of type "text/plain" (8397 bytes)

View attachment "job.yaml" of type "text/plain" (5773 bytes)

View attachment "reproduce" of type "text/plain" (2206 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ