lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Message-ID: <20200227022539.GD6548@shao2-debian>
Date:   Thu, 27 Feb 2020 10:25:39 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     Mel Gorman <mgorman@...hsingularity.net>
Cc:     Ingo Molnar <mingo@...nel.org>,
        Peter Zijlstra <peterz@...radead.org>,
        LKML <linux-kernel@...r.kernel.org>,
        Linus Torvalds <torvalds@...ux-foundation.org>,
        lkp@...ts.01.org
Subject: [sched/fair] 52262ee567: fsmark.files_per_sec 15.9% improvement

Greeting,

FYI, we noticed a 15.9% improvement of fsmark.files_per_sec due to commit:


commit: 52262ee567ad14c9606be25f3caddcefa3c514e4 ("sched/fair: Allow a per-CPU kthread waking a task to stack on the same CPU, to fix XFS performance regression")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master

in testcase: fsmark
on test machine: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 192G memory
with following parameters:

	iterations: 1x
	nr_threads: 1t
	disk: 1BRD_32G
	fs: xfs
	fs2: nfsv4
	filesize: 4K
	test_size: 4G
	sync_method: fsyncBeforeClose
	nr_files_per_directory: 1fpd
	cpufreq_governor: performance
	ucode: 0x500002c

test-description: The fsmark is a file system benchmark to test synchronous write workloads, for example, mail servers workload.
test-url: https://sourceforge.net/projects/fsmark/





Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
compiler/cpufreq_governor/disk/filesize/fs2/fs/iterations/kconfig/nr_files_per_directory/nr_threads/rootfs/sync_method/tbox_group/test_size/testcase/ucode:
  gcc-7/performance/1BRD_32G/4K/nfsv4/xfs/1x/x86_64-rhel-7.6/1fpd/1t/debian-x86_64-20191114.cgz/fsyncBeforeClose/lkp-csl-2sp5/4G/fsmark/0x500002c

commit: 
  2a4b03ffc6 ("sched/fair: Prevent unlimited runtime on throttled group")
  52262ee567 ("sched/fair: Allow a per-CPU kthread waking a task to stack on the same CPU, to fix XFS performance regression")

2a4b03ffc69f2ded 52262ee567ad14c9606be25f3ca 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :4           50%           2:4     dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x
           :4           50%           2:4     dmesg.WARNING:stack_recursion
         %stddev     %change         %stddev
             \          |                \  
 1.089e+08           -12.5%   95356120        fsmark.app_overhead
      2576           +15.9%       2984        fsmark.files_per_sec
    392.21           -13.6%     338.83        fsmark.time.elapsed_time
    392.21           -13.6%     338.83        fsmark.time.elapsed_time.max
      2992 ± 10%     +48.3%       4437 ±  5%  fsmark.time.involuntary_context_switches
     19.00           +10.5%      21.00        fsmark.time.percent_of_cpu_this_job_got
     70.75            -3.3%      68.40        fsmark.time.system_time
     59795           +15.9%      69283        meminfo.max_used_kB
      0.03 ±  8%      -0.0        0.02 ±  2%  mpstat.cpu.all.iowait%
      0.11 ±  4%      +0.0        0.12 ±  4%  mpstat.cpu.all.soft%
     25453 ± 25%    +523.7%     158757 ±  9%  numa-vmstat.node0.numa_other
    177274 ±  3%     -75.1%      44201 ± 34%  numa-vmstat.node1.numa_other
     80285           +15.4%      92649        vmstat.io.bo
    199671           +10.0%     219574        vmstat.system.cs
  87811891 ±  5%     -16.2%   73542543 ± 12%  cpuidle.C1E.usage
   3465501 ±  2%     -23.0%    2667878        cpuidle.POLL.time
    617284 ±  3%     -29.6%     434395        cpuidle.POLL.usage
     78.25            -5.1%      74.25 ±  2%  turbostat.Avg_MHz
  87810415 ±  5%     -16.3%   73540232 ± 12%  turbostat.C1E
  75847290           -13.5%   65570722        turbostat.IRQ
    142.00            -1.9%     139.25        proc-vmstat.nr_anon_transparent_hugepages
      7516            -6.0%       7066        proc-vmstat.nr_shmem
      1675 ±  4%     -19.2%       1354 ±  2%  proc-vmstat.pgactivate
   1134011           -13.5%     980821        proc-vmstat.pgfault
      6.00           -16.7%       5.00        nfsstat.Client.nfs.v3.commit
     14.00           -14.3%      12.00        nfsstat.Client.nfs.v3.commit.percent
      6.00           -16.7%       5.00        nfsstat.Client.nfs.v3.write
     14.00           -14.3%      12.00        nfsstat.Client.nfs.v3.write.percent
      6.00           -16.7%       5.00        nfsstat.Client.nfs.v4.getattr
     12981 ±  7%    +477.5%      74964 ± 23%  slabinfo.avc_xperms_data.active_objs
    101.00 ±  7%    +483.2%     589.00 ± 23%  slabinfo.avc_xperms_data.active_slabs
     12981 ±  7%    +481.2%      75447 ± 23%  slabinfo.avc_xperms_data.num_objs
    101.00 ±  7%    +483.2%     589.00 ± 23%  slabinfo.avc_xperms_data.num_slabs
    352.00 ± 30%     -63.6%     128.00        slabinfo.dnotify_struct.active_objs
    352.00 ± 30%     -63.6%     128.00        slabinfo.dnotify_struct.num_objs
    114094 ±  9%     +96.4%     224115 ± 19%  slabinfo.kmalloc-32.active_objs
    910.75 ±  9%     +96.8%       1792 ± 18%  slabinfo.kmalloc-32.active_slabs
    116638 ±  9%     +96.8%     229557 ± 18%  slabinfo.kmalloc-32.num_objs
    910.75 ±  9%     +96.8%       1792 ± 18%  slabinfo.kmalloc-32.num_slabs
     54300          +196.0%     160750 ± 13%  slabinfo.kmalloc-8.active_objs
    105.25          +197.9%     313.50 ± 13%  slabinfo.kmalloc-8.active_slabs
     54300          +196.0%     160750 ± 13%  slabinfo.kmalloc-8.num_objs
    105.25          +197.9%     313.50 ± 13%  slabinfo.kmalloc-8.num_slabs
     25021          +463.2%     140908 ± 17%  slabinfo.lsm_file_cache.active_objs
     96.75          +470.5%     552.00 ± 17%  slabinfo.lsm_file_cache.active_slabs
     25021          +465.3%     141452 ± 17%  slabinfo.lsm_file_cache.num_objs
     96.75          +470.5%     552.00 ± 17%  slabinfo.lsm_file_cache.num_slabs
      3289 ±  6%     -38.6%       2020 ± 14%  slabinfo.nfs_read_data.active_objs
      3289 ±  6%     -38.6%       2020 ± 14%  slabinfo.nfs_read_data.num_objs
      5397 ±  4%    +155.1%      13766 ± 20%  slabinfo.nfsd4_files.active_objs
     95.75 ±  4%    +157.2%     246.25 ± 20%  slabinfo.nfsd4_files.active_slabs
      5397 ±  4%    +156.0%      13814 ± 20%  slabinfo.nfsd4_files.num_objs
     95.75 ±  4%    +157.2%     246.25 ± 20%  slabinfo.nfsd4_files.num_slabs
      4531           -21.3%       3567 ± 15%  slabinfo.nfsd4_stateids.active_objs
      4531           -21.3%       3567 ± 15%  slabinfo.nfsd4_stateids.num_objs
      3424           -10.1%       3077 ±  4%  slabinfo.xfs_btree_cur.active_objs
      3424           -10.1%       3077 ±  4%  slabinfo.xfs_btree_cur.num_objs
      5670           -16.5%       4733 ± 13%  slabinfo.xfs_buf_item.active_objs
      5670           -16.5%       4733 ± 13%  slabinfo.xfs_buf_item.num_objs
      3207           -23.4%       2456 ± 16%  slabinfo.xfs_da_state.active_objs
      3207           -23.4%       2456 ± 16%  slabinfo.xfs_da_state.num_objs
      2812 ± 10%     -28.3%       2016 ±  7%  sched_debug.cfs_rq:/.exec_clock.avg
    162.85 ±  6%     +60.3%     261.00 ± 12%  sched_debug.cfs_rq:/.load_avg.avg
    695.55 ± 12%     +53.6%       1068 ±  5%  sched_debug.cfs_rq:/.load_avg.stddev
      0.05 ± 16%     +46.3%       0.08 ± 10%  sched_debug.cfs_rq:/.nr_running.avg
      0.22 ±  6%     +21.1%       0.26 ±  5%  sched_debug.cfs_rq:/.nr_running.stddev
     12.52 ± 29%     +95.1%      24.42 ± 35%  sched_debug.cfs_rq:/.runnable_load_avg.avg
    136.41 ±  5%     +17.5%     160.29 ±  5%  sched_debug.cfs_rq:/.util_avg.stddev
    247.98 ± 20%     +96.3%     486.88 ± 17%  sched_debug.cfs_rq:/.util_est_enqueued.max
     39.78 ± 32%     +73.2%      68.92 ± 18%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
     30761 ± 24%     -55.0%      13854 ± 14%  sched_debug.cpu.avg_idle.min
    245111           +13.6%     278377 ±  2%  sched_debug.cpu.avg_idle.stddev
    206234 ±  5%     -14.8%     175687 ±  6%  sched_debug.cpu.clock.avg
    206238 ±  5%     -14.8%     175690 ±  6%  sched_debug.cpu.clock.max
    206230 ±  5%     -14.8%     175682 ±  6%  sched_debug.cpu.clock.min
      2.31 ±  2%     -10.4%       2.07 ±  2%  sched_debug.cpu.clock.stddev
    206234 ±  5%     -14.8%     175687 ±  6%  sched_debug.cpu.clock_task.avg
    206238 ±  5%     -14.8%     175690 ±  6%  sched_debug.cpu.clock_task.max
    206230 ±  5%     -14.8%     175682 ±  6%  sched_debug.cpu.clock_task.min
      2.31 ±  2%     -10.4%       2.07 ±  2%  sched_debug.cpu.clock_task.stddev
    899.44 ±  3%     -10.2%     808.09 ±  4%  sched_debug.cpu.curr->pid.stddev
    856405 ±  6%     +19.2%    1021054 ±  3%  sched_debug.cpu.nr_switches.max
    506.49 ± 10%     -89.1%      55.10 ±  9%  sched_debug.cpu.nr_uninterruptible.max
   -573.16           -92.3%     -44.41        sched_debug.cpu.nr_uninterruptible.min
    160.90 ±  6%     -91.6%      13.58 ±  3%  sched_debug.cpu.nr_uninterruptible.stddev
    853839 ±  7%     +19.3%    1018542 ±  3%  sched_debug.cpu.sched_count.max
    398223 ±  7%     +15.4%     459371 ±  2%  sched_debug.cpu.sched_goidle.max
    414329 ±  7%     +41.6%     586567 ± 10%  sched_debug.cpu.ttwu_count.max
     24210 ±  7%     +69.0%      40921 ±  8%  sched_debug.cpu.ttwu_local.avg
     49327 ±  7%    +196.1%     146061 ± 12%  sched_debug.cpu.ttwu_local.max
     20763 ±  9%     +85.5%      38511 ± 21%  sched_debug.cpu.ttwu_local.stddev
    206230 ±  5%     -14.8%     175683 ±  6%  sched_debug.cpu_clk
    203356 ±  5%     -15.0%     172812 ±  7%  sched_debug.ktime
    206600 ±  5%     -14.8%     176042 ±  6%  sched_debug.sched_clk
     24.30           -38.4%      14.97        perf-stat.i.MPKI
 7.854e+08            +6.5%  8.366e+08        perf-stat.i.branch-instructions
      1.50            -0.4        1.06        perf-stat.i.branch-miss-rate%
  12115396           -22.8%    9356693        perf-stat.i.branch-misses
      4.14 ±  8%      +1.2        5.32 ± 12%  perf-stat.i.cache-miss-rate%
  91548188           -36.1%   58460096        perf-stat.i.cache-references
    201140           +10.2%     221607        perf-stat.i.context-switches
      1.82            -7.7%       1.67 ±  2%  perf-stat.i.cpi
 6.871e+09            -4.3%  6.577e+09 ±  3%  perf-stat.i.cpu-cycles
     11237           -99.0%     113.42        perf-stat.i.cpu-migrations
 1.047e+09            +1.3%   1.06e+09        perf-stat.i.dTLB-loads
 5.526e+08            +6.2%  5.871e+08        perf-stat.i.dTLB-stores
     40.71            -3.0       37.75 ±  2%  perf-stat.i.iTLB-load-miss-rate%
   4834507           +11.4%    5385007        perf-stat.i.iTLB-loads
 3.807e+09            +3.8%  3.953e+09        perf-stat.i.instructions
      1154            +5.7%       1220 ±  2%  perf-stat.i.instructions-per-iTLB-miss
      0.55            +8.5%       0.60 ±  2%  perf-stat.i.ipc
    718104 ± 15%     -39.4%     434939 ± 23%  perf-stat.i.node-load-misses
    318025 ± 15%     -31.0%     219510 ± 15%  perf-stat.i.node-store-misses
    484962 ±  8%     -35.1%     314912 ±  2%  perf-stat.i.node-stores
     24.05           -38.5%      14.79        perf-stat.overall.MPKI
      1.54            -0.4        1.12        perf-stat.overall.branch-miss-rate%
      3.90 ±  9%      +1.3        5.23 ± 11%  perf-stat.overall.cache-miss-rate%
      1.80            -7.8%       1.66 ±  2%  perf-stat.overall.cpi
     40.69            -3.0       37.72 ±  2%  perf-stat.overall.iTLB-load-miss-rate%
      1147            +5.7%       1212 ±  2%  perf-stat.overall.instructions-per-iTLB-miss
      0.55            +8.5%       0.60 ±  2%  perf-stat.overall.ipc
 7.835e+08            +6.5%  8.342e+08        perf-stat.ps.branch-instructions
  12087209           -22.8%    9332571        perf-stat.ps.branch-misses
  91312838           -36.2%   58286286        perf-stat.ps.cache-references
    200622           +10.1%     220944        perf-stat.ps.context-switches
 6.853e+09            -4.3%  6.558e+09 ±  3%  perf-stat.ps.cpu-cycles
     11208           -99.0%     113.09        perf-stat.ps.cpu-migrations
 1.044e+09            +1.2%  1.057e+09        perf-stat.ps.dTLB-loads
 5.512e+08            +6.2%  5.853e+08        perf-stat.ps.dTLB-stores
   4822085           +11.3%    5368970        perf-stat.ps.iTLB-loads
 3.797e+09            +3.8%  3.941e+09        perf-stat.ps.instructions
    716278 ± 15%     -39.5%     433664 ± 23%  perf-stat.ps.node-load-misses
    317214 ± 15%     -31.0%     218863 ± 15%  perf-stat.ps.node-store-misses
    483718 ±  8%     -35.1%     313980 ±  2%  perf-stat.ps.node-stores
 1.492e+12           -10.3%  1.338e+12        perf-stat.total.instructions
     57773 ±  4%     -13.6%      49940 ±  4%  softirqs.CPU0.SCHED
    195517            -9.7%     176459 ±  6%  softirqs.CPU10.RCU
     50414 ±  4%     -21.3%      39678 ± 13%  softirqs.CPU10.SCHED
    195777            -9.5%     177251 ±  6%  softirqs.CPU11.RCU
     50746 ±  2%     -16.1%      42556 ±  7%  softirqs.CPU11.SCHED
    196092           -11.1%     174303 ±  4%  softirqs.CPU13.RCU
    196393           -11.6%     173524 ±  5%  softirqs.CPU14.RCU
     50321 ±  4%     -40.5%      29965 ± 37%  softirqs.CPU14.SCHED
    195416           -10.9%     174210 ±  5%  softirqs.CPU15.RCU
    195557           -10.9%     174149 ±  4%  softirqs.CPU16.RCU
    194883           -11.2%     173088 ±  4%  softirqs.CPU17.RCU
     50552 ±  4%     -14.0%      43464 ±  5%  softirqs.CPU17.SCHED
    194406           -10.2%     174639 ±  4%  softirqs.CPU18.RCU
     50538 ±  3%     -12.8%      44055 ±  3%  softirqs.CPU18.SCHED
    196377            -9.8%     177130 ±  5%  softirqs.CPU19.RCU
     49252 ±  2%     -13.8%      42462 ±  8%  softirqs.CPU19.SCHED
     51044 ±  3%     -21.9%      39849 ± 22%  softirqs.CPU2.SCHED
    196095           -11.0%     174602 ±  5%  softirqs.CPU20.RCU
    195870           -10.3%     175600 ±  6%  softirqs.CPU21.RCU
    194904           -10.5%     174526 ±  5%  softirqs.CPU22.RCU
     50142 ±  3%     -20.6%      39819 ± 13%  softirqs.CPU22.SCHED
    194750            -9.5%     176281 ±  5%  softirqs.CPU23.RCU
     49972 ±  3%     -21.5%      39215 ± 20%  softirqs.CPU23.SCHED
     50470           -13.8%      43525        softirqs.CPU24.SCHED
     50619           -24.5%      38227 ± 21%  softirqs.CPU26.SCHED
     50043 ±  2%     -19.5%      40296 ± 10%  softirqs.CPU28.SCHED
     50766 ±  4%     -14.6%      43343 ±  4%  softirqs.CPU29.SCHED
    196076            -9.3%     177747 ±  6%  softirqs.CPU3.RCU
     50251 ±  3%     -15.0%      42688 ±  6%  softirqs.CPU3.SCHED
     49628           -15.0%      42186 ±  6%  softirqs.CPU30.SCHED
     50027           -14.3%      42866 ±  7%  softirqs.CPU31.SCHED
     49620           -13.1%      43136 ±  4%  softirqs.CPU32.SCHED
     49416           -10.7%      44136 ±  2%  softirqs.CPU34.SCHED
     48341           -12.2%      42467 ±  6%  softirqs.CPU36.SCHED
     49978 ±  2%     -13.4%      43302 ±  5%  softirqs.CPU37.SCHED
     49460           -14.6%      42221 ±  6%  softirqs.CPU38.SCHED
     50113 ±  4%     -12.2%      43990 ±  6%  softirqs.CPU4.SCHED
     50157 ±  2%     -12.4%      43929 ±  5%  softirqs.CPU40.SCHED
     49434 ±  3%     -14.4%      42305 ±  5%  softirqs.CPU41.SCHED
     49831 ±  2%     -12.2%      43765 ±  4%  softirqs.CPU42.SCHED
     49536           -12.2%      43474 ±  3%  softirqs.CPU44.SCHED
     49991           -12.7%      43626 ±  5%  softirqs.CPU45.SCHED
     50649 ±  2%     -14.5%      43312 ±  4%  softirqs.CPU46.SCHED
    195158            -9.9%     175740 ±  6%  softirqs.CPU48.RCU
    196900           -10.6%     176114 ±  6%  softirqs.CPU49.RCU
     50469 ±  6%     -15.2%      42802 ±  6%  softirqs.CPU49.SCHED
    195738            -9.8%     176510 ±  6%  softirqs.CPU5.RCU
     49552 ±  4%     -10.9%      44146 ±  4%  softirqs.CPU5.SCHED
    196601           -11.5%     174031 ±  5%  softirqs.CPU50.RCU
     50013 ±  4%     -24.7%      37669 ± 19%  softirqs.CPU50.SCHED
    195947           -10.8%     174695 ±  5%  softirqs.CPU51.RCU
     50173 ±  5%     -14.8%      42724 ±  4%  softirqs.CPU51.SCHED
    195712           -11.5%     173148 ±  4%  softirqs.CPU52.RCU
     50012 ±  4%     -12.8%      43613 ±  4%  softirqs.CPU52.SCHED
    195130           -11.1%     173462 ±  5%  softirqs.CPU53.RCU
    195751           -11.7%     172866 ±  5%  softirqs.CPU54.RCU
     49754 ±  4%     -23.5%      38064 ± 10%  softirqs.CPU54.SCHED
    194633           -10.7%     173727 ±  5%  softirqs.CPU55.RCU
    194996           -10.9%     173818 ±  6%  softirqs.CPU56.RCU
     50760 ±  3%     -14.4%      43462 ±  4%  softirqs.CPU56.SCHED
    194868           -11.6%     172360 ±  4%  softirqs.CPU57.RCU
     50598 ±  3%     -15.2%      42888 ±  6%  softirqs.CPU57.SCHED
    194479           -11.3%     172569 ±  4%  softirqs.CPU58.RCU
    195721           -12.4%     171514 ±  3%  softirqs.CPU59.RCU
     50362 ±  3%     -13.8%      43387 ±  9%  softirqs.CPU59.SCHED
    195061           -11.6%     172498 ±  4%  softirqs.CPU60.RCU
     50548 ±  4%     -24.3%      38279 ± 19%  softirqs.CPU60.SCHED
    196237           -11.8%     172999 ±  4%  softirqs.CPU61.RCU
     50726 ±  4%     -16.1%      42583 ±  5%  softirqs.CPU61.SCHED
    195045           -12.0%     171546 ±  4%  softirqs.CPU62.RCU
    194599           -11.9%     171472 ±  4%  softirqs.CPU63.RCU
     50531 ±  3%     -33.7%      33508 ± 25%  softirqs.CPU63.SCHED
    194767           -11.5%     172381 ±  2%  softirqs.CPU64.RCU
     49945 ±  4%     -25.9%      37031 ± 26%  softirqs.CPU64.SCHED
    194233           -11.3%     172193 ±  4%  softirqs.CPU65.RCU
    194329           -10.4%     174168 ±  5%  softirqs.CPU66.RCU
     50670 ±  3%     -17.8%      41647 ±  9%  softirqs.CPU67.SCHED
    194551           -10.2%     174670 ±  5%  softirqs.CPU68.RCU
     49211 ±  5%     -14.1%      42273 ±  7%  softirqs.CPU68.SCHED
    194004           -10.5%     173556 ±  4%  softirqs.CPU69.RCU
    194034            -9.7%     175215 ±  5%  softirqs.CPU70.RCU
     49913 ±  3%     -13.7%      43052 ±  5%  softirqs.CPU70.SCHED
    193816            -9.8%     174886 ±  4%  softirqs.CPU71.RCU
     50740           -15.6%      42840 ±  6%  softirqs.CPU72.SCHED
    195973            -9.2%     177883 ±  3%  softirqs.CPU73.RCU
     48402           -15.0%      41159 ±  7%  softirqs.CPU73.SCHED
     50208 ±  2%     -12.2%      44087 ±  3%  softirqs.CPU75.SCHED
     48890 ±  3%     -11.2%      43422 ±  4%  softirqs.CPU77.SCHED
    195451           -10.5%     174998 ±  5%  softirqs.CPU8.RCU
     49821           -11.2%      44227 ±  3%  softirqs.CPU81.SCHED
     49475 ±  2%     -25.0%      37130 ± 19%  softirqs.CPU84.SCHED
     48407 ±  3%     -10.5%      43324 ±  5%  softirqs.CPU86.SCHED
     49557 ±  2%     -14.2%      42532 ±  6%  softirqs.CPU87.SCHED
    196354           -10.8%     175148 ±  5%  softirqs.CPU9.RCU
     49789 ±  3%     -13.7%      42973 ±  6%  softirqs.CPU9.SCHED
     49402           -13.3%      42842 ±  6%  softirqs.CPU92.SCHED
     50088           -15.8%      42152 ±  3%  softirqs.CPU93.SCHED
   4672675 ±  3%     -13.3%    4052204        softirqs.SCHED
    346.50 ±  4%     -11.4%     307.00 ±  7%  interrupts.35:PCI-MSI.31981568-edge.i40e-0000:3d:00.0:misc
      1.50 ±137%  +12866.7%     194.50 ± 88%  interrupts.44:PCI-MSI.31981577-edge.i40e-eth0-TxRx-8
    199.25 ±162%     -96.2%       7.50 ±173%  interrupts.94:PCI-MSI.31981627-edge.i40e-eth0-TxRx-58
    787.50           -13.7%     680.00        interrupts.9:IO-APIC.9-fasteoi.acpi
     46288            -1.6%      45538        interrupts.CAL:Function_call_interrupts
    787259           -13.6%     680255        interrupts.CPU0.LOC:Local_timer_interrupts
    787.50           -13.7%     680.00        interrupts.CPU1.9:IO-APIC.9-fasteoi.acpi
    787219           -13.6%     680249        interrupts.CPU1.LOC:Local_timer_interrupts
    787370           -13.6%     680275        interrupts.CPU10.LOC:Local_timer_interrupts
    787283           -13.6%     680287        interrupts.CPU11.LOC:Local_timer_interrupts
    787301           -13.6%     680169        interrupts.CPU12.LOC:Local_timer_interrupts
    787314           -13.6%     680324        interrupts.CPU13.LOC:Local_timer_interrupts
    787250           -13.6%     680192        interrupts.CPU14.LOC:Local_timer_interrupts
      1839 ± 37%     -68.0%     588.75 ±124%  interrupts.CPU14.RES:Rescheduling_interrupts
    787217           -13.6%     680356        interrupts.CPU15.LOC:Local_timer_interrupts
    787205           -13.6%     680227        interrupts.CPU16.LOC:Local_timer_interrupts
    787284           -13.6%     680258        interrupts.CPU17.LOC:Local_timer_interrupts
      1784 ± 38%     -63.3%     655.75 ±122%  interrupts.CPU17.RES:Rescheduling_interrupts
    787390           -13.6%     680270        interrupts.CPU18.LOC:Local_timer_interrupts
    787266           -13.6%     680261        interrupts.CPU19.LOC:Local_timer_interrupts
    787223           -13.6%     680318        interrupts.CPU2.LOC:Local_timer_interrupts
    787229           -13.6%     680399        interrupts.CPU20.LOC:Local_timer_interrupts
    787237           -13.6%     680195        interrupts.CPU21.LOC:Local_timer_interrupts
    787394           -13.6%     680394        interrupts.CPU22.LOC:Local_timer_interrupts
    787225           -13.6%     680254        interrupts.CPU23.LOC:Local_timer_interrupts
    787155           -13.6%     680271        interrupts.CPU24.LOC:Local_timer_interrupts
    787209           -13.6%     680268        interrupts.CPU25.LOC:Local_timer_interrupts
    787152           -13.6%     680465        interrupts.CPU26.LOC:Local_timer_interrupts
      2861 ± 24%     -48.9%       1461 ± 58%  interrupts.CPU26.RES:Rescheduling_interrupts
    787062           -13.6%     680296        interrupts.CPU27.LOC:Local_timer_interrupts
    787087           -13.6%     680303        interrupts.CPU28.LOC:Local_timer_interrupts
    786970           -13.6%     680250        interrupts.CPU29.LOC:Local_timer_interrupts
    787249           -13.6%     680260        interrupts.CPU3.LOC:Local_timer_interrupts
    787165           -13.6%     680254        interrupts.CPU30.LOC:Local_timer_interrupts
    787128           -13.6%     680154        interrupts.CPU31.LOC:Local_timer_interrupts
    786971           -13.6%     680267        interrupts.CPU32.LOC:Local_timer_interrupts
    787076           -13.6%     680236        interrupts.CPU33.LOC:Local_timer_interrupts
    787176           -13.6%     680261        interrupts.CPU34.LOC:Local_timer_interrupts
    787245           -13.6%     680374        interrupts.CPU35.LOC:Local_timer_interrupts
    787434           -13.6%     680347        interrupts.CPU36.LOC:Local_timer_interrupts
    787179           -13.5%     680547        interrupts.CPU37.LOC:Local_timer_interrupts
    787201           -13.6%     680274        interrupts.CPU38.LOC:Local_timer_interrupts
    787200           -13.6%     680279        interrupts.CPU39.LOC:Local_timer_interrupts
    787239           -13.6%     680052        interrupts.CPU4.LOC:Local_timer_interrupts
    787199           -13.6%     680273        interrupts.CPU40.LOC:Local_timer_interrupts
    787207           -13.6%     680281        interrupts.CPU41.LOC:Local_timer_interrupts
    787171           -13.6%     680278        interrupts.CPU42.LOC:Local_timer_interrupts
    787016           -13.6%     680365        interrupts.CPU43.LOC:Local_timer_interrupts
    787357           -13.6%     680316        interrupts.CPU44.LOC:Local_timer_interrupts
    787163           -13.6%     680285        interrupts.CPU45.LOC:Local_timer_interrupts
    787186           -13.6%     680303        interrupts.CPU46.LOC:Local_timer_interrupts
    787205           -13.6%     680277        interrupts.CPU47.LOC:Local_timer_interrupts
    787211           -13.6%     680346        interrupts.CPU48.LOC:Local_timer_interrupts
    787157           -13.6%     680243        interrupts.CPU49.LOC:Local_timer_interrupts
    787268           -13.6%     680239        interrupts.CPU5.LOC:Local_timer_interrupts
      1780 ± 40%     -56.1%     781.75 ±101%  interrupts.CPU5.RES:Rescheduling_interrupts
    787199           -13.6%     680320        interrupts.CPU50.LOC:Local_timer_interrupts
    787103           -13.6%     680372        interrupts.CPU51.LOC:Local_timer_interrupts
    787165           -13.6%     680292        interrupts.CPU52.LOC:Local_timer_interrupts
    787197           -13.6%     680290        interrupts.CPU53.LOC:Local_timer_interrupts
    787220           -13.6%     680501        interrupts.CPU54.LOC:Local_timer_interrupts
    787200           -13.6%     680295        interrupts.CPU55.LOC:Local_timer_interrupts
    787181           -13.6%     680361        interrupts.CPU56.LOC:Local_timer_interrupts
    787250           -13.6%     680281        interrupts.CPU57.LOC:Local_timer_interrupts
    787196           -13.6%     680343        interrupts.CPU58.LOC:Local_timer_interrupts
    787211           -13.6%     680389        interrupts.CPU59.LOC:Local_timer_interrupts
    787320           -13.6%     680306        interrupts.CPU6.LOC:Local_timer_interrupts
    787227           -13.6%     680322        interrupts.CPU60.LOC:Local_timer_interrupts
    787215           -13.6%     680309        interrupts.CPU61.LOC:Local_timer_interrupts
    787467           -13.6%     680307        interrupts.CPU62.LOC:Local_timer_interrupts
    787213           -13.6%     680344        interrupts.CPU63.LOC:Local_timer_interrupts
    787245           -13.6%     680278        interrupts.CPU64.LOC:Local_timer_interrupts
    787183           -13.6%     680261        interrupts.CPU65.LOC:Local_timer_interrupts
    787290           -13.6%     680257        interrupts.CPU66.LOC:Local_timer_interrupts
    787195           -13.6%     680217        interrupts.CPU67.LOC:Local_timer_interrupts
    787185           -13.6%     680279        interrupts.CPU68.LOC:Local_timer_interrupts
    508.75 ± 48%     -57.7%     215.00 ± 94%  interrupts.CPU68.NMI:Non-maskable_interrupts
    508.75 ± 48%     -57.7%     215.00 ± 94%  interrupts.CPU68.PMI:Performance_monitoring_interrupts
    787182           -13.6%     680211        interrupts.CPU69.LOC:Local_timer_interrupts
    474.25 ± 46%     -63.1%     175.00 ± 77%  interrupts.CPU69.NMI:Non-maskable_interrupts
    474.25 ± 46%     -63.1%     175.00 ± 77%  interrupts.CPU69.PMI:Performance_monitoring_interrupts
    346.50 ±  4%     -11.4%     307.00 ±  7%  interrupts.CPU7.35:PCI-MSI.31981568-edge.i40e-0000:3d:00.0:misc
    787291           -13.6%     680272        interrupts.CPU7.LOC:Local_timer_interrupts
      1714 ± 41%     -60.9%     670.50 ±124%  interrupts.CPU7.RES:Rescheduling_interrupts
    787178           -13.6%     680134        interrupts.CPU70.LOC:Local_timer_interrupts
    787167           -13.6%     680214        interrupts.CPU71.LOC:Local_timer_interrupts
    787188           -13.6%     680270        interrupts.CPU72.LOC:Local_timer_interrupts
    787128           -13.6%     680259        interrupts.CPU73.LOC:Local_timer_interrupts
    787198           -13.6%     680293        interrupts.CPU74.LOC:Local_timer_interrupts
    787051           -13.6%     680291        interrupts.CPU75.LOC:Local_timer_interrupts
    787169           -13.6%     680284        interrupts.CPU76.LOC:Local_timer_interrupts
    787246           -13.6%     680263        interrupts.CPU77.LOC:Local_timer_interrupts
    787150           -13.6%     680251        interrupts.CPU78.LOC:Local_timer_interrupts
    787193           -13.6%     680265        interrupts.CPU79.LOC:Local_timer_interrupts
      1.00 ±173%  +19300.0%     194.00 ± 88%  interrupts.CPU8.44:PCI-MSI.31981577-edge.i40e-eth0-TxRx-8
    787223           -13.6%     680386        interrupts.CPU8.LOC:Local_timer_interrupts
      1830 ± 40%     -60.8%     717.75 ±126%  interrupts.CPU8.RES:Rescheduling_interrupts
    787173           -13.6%     680246        interrupts.CPU80.LOC:Local_timer_interrupts
    787186           -13.6%     680292        interrupts.CPU81.LOC:Local_timer_interrupts
    787162           -13.6%     680266        interrupts.CPU82.LOC:Local_timer_interrupts
    787173           -13.6%     680281        interrupts.CPU83.LOC:Local_timer_interrupts
    787239           -13.6%     680260        interrupts.CPU84.LOC:Local_timer_interrupts
    787140           -13.6%     680257        interrupts.CPU85.LOC:Local_timer_interrupts
    787332           -13.6%     680232        interrupts.CPU86.LOC:Local_timer_interrupts
    787254           -13.6%     680443        interrupts.CPU87.LOC:Local_timer_interrupts
    787322           -13.6%     680287        interrupts.CPU88.LOC:Local_timer_interrupts
    787265           -13.6%     680304        interrupts.CPU89.LOC:Local_timer_interrupts
    787278           -13.6%     680301        interrupts.CPU9.LOC:Local_timer_interrupts
      1769 ± 41%     -56.4%     772.00 ± 98%  interrupts.CPU9.RES:Rescheduling_interrupts
    787244           -13.6%     680235        interrupts.CPU90.LOC:Local_timer_interrupts
    787153           -13.6%     680253        interrupts.CPU91.LOC:Local_timer_interrupts
    787182           -13.6%     680326        interrupts.CPU92.LOC:Local_timer_interrupts
    787183           -13.6%     680258        interrupts.CPU93.LOC:Local_timer_interrupts
    787188           -13.6%     680303        interrupts.CPU94.LOC:Local_timer_interrupts
    787583           -13.6%     680553        interrupts.CPU95.LOC:Local_timer_interrupts
  75572622           -13.6%   65307895        interrupts.LOC:Local_timer_interrupts
     43832 ±  7%     -17.5%      36167 ±  2%  interrupts.NMI:Non-maskable_interrupts
     43832 ±  7%     -17.5%      36167 ±  2%  interrupts.PMI:Performance_monitoring_interrupts
     15.86 ±  2%      -5.0       10.82 ±  4%  perf-profile.calltrace.cycles-pp.nfsd.kthread.ret_from_fork
     23.11 ±  2%      -4.7       18.39 ±  4%  perf-profile.calltrace.cycles-pp.ret_from_fork
     23.11 ±  2%      -4.7       18.39 ±  4%  perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
     10.35 ±  3%      -3.1        7.21 ±  5%  perf-profile.calltrace.cycles-pp.svc_process.nfsd.kthread.ret_from_fork
     10.34 ±  3%      -3.1        7.20 ±  4%  perf-profile.calltrace.cycles-pp.svc_process_common.svc_process.nfsd.kthread.ret_from_fork
      9.94 ±  4%      -3.0        6.97 ±  5%  perf-profile.calltrace.cycles-pp.nfsd_dispatch.svc_process_common.svc_process.nfsd.kthread
      9.49 ±  4%      -2.8        6.70 ±  5%  perf-profile.calltrace.cycles-pp.nfsd4_proc_compound.nfsd_dispatch.svc_process_common.svc_process.nfsd
      2.99 ±  2%      -1.3        1.67 ±  6%  perf-profile.calltrace.cycles-pp.nfsd4_write.nfsd4_proc_compound.nfsd_dispatch.svc_process_common.svc_process
      2.93 ±  2%      -1.3        1.63 ±  6%  perf-profile.calltrace.cycles-pp.nfsd_vfs_write.nfsd4_write.nfsd4_proc_compound.nfsd_dispatch.svc_process_common
      2.89 ±  2%      -1.3        1.62 ±  6%  perf-profile.calltrace.cycles-pp.do_iter_write.nfsd_vfs_write.nfsd4_write.nfsd4_proc_compound.nfsd_dispatch
      2.84            -1.2        1.59 ±  6%  perf-profile.calltrace.cycles-pp.do_iter_readv_writev.do_iter_write.nfsd_vfs_write.nfsd4_write.nfsd4_proc_compound
      2.83            -1.2        1.58 ±  6%  perf-profile.calltrace.cycles-pp.xfs_file_buffered_aio_write.do_iter_readv_writev.do_iter_write.nfsd_vfs_write.nfsd4_write
      2.39 ±  2%      -1.2        1.23 ±  6%  perf-profile.calltrace.cycles-pp.xfs_file_fsync.xfs_file_buffered_aio_write.do_iter_readv_writev.do_iter_write.nfsd_vfs_write
      2.83            -1.2        1.68 ±  3%  perf-profile.calltrace.cycles-pp.svc_send.nfsd.kthread.ret_from_fork
      2.71            -1.1        1.59 ±  2%  perf-profile.calltrace.cycles-pp.svc_tcp_sendto.svc_send.nfsd.kthread.ret_from_fork
      2.71            -1.1        1.59 ±  2%  perf-profile.calltrace.cycles-pp.svc_sendto.svc_tcp_sendto.svc_send.nfsd.kthread
      2.68            -1.1        1.57 ±  2%  perf-profile.calltrace.cycles-pp.kernel_sendpage.svc_send_common.svc_sendto.svc_tcp_sendto.svc_send
      2.70            -1.1        1.59 ±  2%  perf-profile.calltrace.cycles-pp.svc_send_common.svc_sendto.svc_tcp_sendto.svc_send.nfsd
      1.66 ±  4%      -1.1        0.55 ±  3%  perf-profile.calltrace.cycles-pp.__sched_text_start.schedule.worker_thread.kthread.ret_from_fork
      1.69 ±  4%      -1.1        0.58 ±  5%  perf-profile.calltrace.cycles-pp.schedule.worker_thread.kthread.ret_from_fork
      2.67            -1.1        1.57 ±  2%  perf-profile.calltrace.cycles-pp.inet_sendpage.kernel_sendpage.svc_send_common.svc_sendto.svc_tcp_sendto
      2.64            -1.1        1.55 ±  3%  perf-profile.calltrace.cycles-pp.tcp_sendpage.inet_sendpage.kernel_sendpage.svc_send_common.svc_sendto
      2.55            -1.1        1.50 ±  3%  perf-profile.calltrace.cycles-pp.do_tcp_sendpages.tcp_sendpage_locked.tcp_sendpage.inet_sendpage.kernel_sendpage
      2.57            -1.1        1.52 ±  3%  perf-profile.calltrace.cycles-pp.tcp_sendpage_locked.tcp_sendpage.inet_sendpage.kernel_sendpage.svc_send_common
      2.27            -0.9        1.38 ±  2%  perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.do_tcp_sendpages.tcp_sendpage_locked.tcp_sendpage.inet_sendpage
      2.26            -0.9        1.38 ±  2%  perf-profile.calltrace.cycles-pp.tcp_write_xmit.__tcp_push_pending_frames.do_tcp_sendpages.tcp_sendpage_locked.tcp_sendpage
      2.16            -0.8        1.31 ±  2%  perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.do_tcp_sendpages.tcp_sendpage_locked
      2.02            -0.8        1.23 ±  3%  perf-profile.calltrace.cycles-pp.inet6_csk_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.do_tcp_sendpages
      2.66            -0.7        1.93 ±  6%  perf-profile.calltrace.cycles-pp.svc_recv.nfsd.kthread.ret_from_fork
      2.45 ±  4%      -0.7        1.73 ±  4%  perf-profile.calltrace.cycles-pp.nfsd4_open.nfsd4_proc_compound.nfsd_dispatch.svc_process_common.svc_process
      2.54            -0.4        2.10 ± 12%  perf-profile.calltrace.cycles-pp.ip6_xmit.inet6_csk_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames
      1.86 ±  4%      -0.4        1.45 ±  4%  perf-profile.calltrace.cycles-pp.do_nfsd_create.nfsd4_open.nfsd4_proc_compound.nfsd_dispatch.svc_process_common
      1.12 ±  4%      -0.3        0.83 ±  4%  perf-profile.calltrace.cycles-pp.vfs_create.do_nfsd_create.nfsd4_open.nfsd4_proc_compound.nfsd_dispatch
      1.09 ±  4%      -0.3        0.80 ±  4%  perf-profile.calltrace.cycles-pp.xfs_generic_create.vfs_create.do_nfsd_create.nfsd4_open.nfsd4_proc_compound
      1.16 ±  5%      -0.3        0.88 ±  6%  perf-profile.calltrace.cycles-pp.file_write_and_wait_range.xfs_file_fsync.xfs_file_buffered_aio_write.do_iter_readv_writev.do_iter_write
      1.05 ±  5%      -0.3        0.77 ±  5%  perf-profile.calltrace.cycles-pp.xfs_create.xfs_generic_create.vfs_create.do_nfsd_create.nfsd4_open
      0.74 ±  4%      -0.3        0.46 ± 58%  perf-profile.calltrace.cycles-pp.xfs_dir_ialloc.xfs_create.xfs_generic_create.vfs_mkdir.nfsd_create_locked
      0.73 ±  5%      -0.3        0.45 ± 58%  perf-profile.calltrace.cycles-pp.xfs_ialloc.xfs_dir_ialloc.xfs_create.xfs_generic_create.vfs_mkdir
      0.67 ±  8%      -0.3        0.41 ± 57%  perf-profile.calltrace.cycles-pp.xfs_map_blocks.iomap_do_writepage.write_cache_pages.iomap_writepages.xfs_vm_writepages
      0.65 ±  8%      -0.3        0.39 ± 57%  perf-profile.calltrace.cycles-pp.xfs_bmapi_convert_delalloc.xfs_map_blocks.iomap_do_writepage.write_cache_pages.iomap_writepages
      1.02 ±  5%      -0.2        0.81 ±  5%  perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync.xfs_file_buffered_aio_write.do_iter_readv_writev
      1.01 ±  5%      -0.2        0.81 ±  5%  perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync.xfs_file_buffered_aio_write
      1.01 ±  5%      -0.2        0.81 ±  5%  perf-profile.calltrace.cycles-pp.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync
      0.75 ± 17%      -0.2        0.56 ±  7%  perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
      0.79 ±  9%      -0.2        0.62 ±  3%  perf-profile.calltrace.cycles-pp.write_cache_pages.iomap_writepages.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range
      0.79 ±  9%      -0.2        0.63 ±  3%  perf-profile.calltrace.cycles-pp.iomap_writepages.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range
      0.72 ±  8%      -0.2        0.57 ±  5%  perf-profile.calltrace.cycles-pp.iomap_do_writepage.write_cache_pages.iomap_writepages.xfs_vm_writepages.do_writepages
      0.82 ±  9%      -0.1        0.69 ±  4%  perf-profile.calltrace.cycles-pp.xlog_ioend_work.process_one_work.worker_thread.kthread.ret_from_fork
      0.80 ±  8%      -0.1        0.68 ±  4%  perf-profile.calltrace.cycles-pp.xlog_state_do_callback.xlog_ioend_work.process_one_work.worker_thread.kthread
      0.60 ±  6%      +0.1        0.66 ±  3%  perf-profile.calltrace.cycles-pp.native_write_msr.lapic_next_deadline.clockevents_program_event.hrtimer_interrupt.smp_apic_timer_interrupt
      0.69 ±  4%      +0.1        0.76 ±  3%  perf-profile.calltrace.cycles-pp.lapic_next_deadline.clockevents_program_event.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      0.57 ±  7%      +0.2        0.79 ±  8%  perf-profile.calltrace.cycles-pp._nfs4_proc_access.nfs4_proc_access.nfs_do_access.nfs_permission.inode_permission
      0.57 ±  8%      +0.2        0.79 ±  8%  perf-profile.calltrace.cycles-pp.nfs4_proc_access.nfs_do_access.nfs_permission.inode_permission.link_path_walk
      0.58 ±  7%      +0.2        0.82 ±  7%  perf-profile.calltrace.cycles-pp.nfs_do_access.nfs_permission.inode_permission.link_path_walk.path_openat
      0.58 ±  7%      +0.2        0.83 ±  8%  perf-profile.calltrace.cycles-pp.nfs_permission.inode_permission.link_path_walk.path_openat.do_filp_open
      0.59 ±  6%      +0.3        0.84 ±  8%  perf-profile.calltrace.cycles-pp.inode_permission.link_path_walk.path_openat.do_filp_open.do_sys_open
      0.63 ±  6%      +0.3        0.90 ±  8%  perf-profile.calltrace.cycles-pp.link_path_walk.path_openat.do_filp_open.do_sys_open.do_syscall_64
      0.59 ±  9%      +0.3        0.89 ±  8%  perf-profile.calltrace.cycles-pp.__rpc_do_wake_up_task_on_wq.rpc_wake_up_queued_task.xs_stream_data_receive.xs_stream_data_receive_workfn.process_one_work
      0.87 ±  2%      +0.3        1.17 ±  8%  perf-profile.calltrace.cycles-pp.call_transmit.__rpc_execute.rpc_async_schedule.process_one_work.worker_thread
      0.64 ±  9%      +0.3        0.94 ± 14%  perf-profile.calltrace.cycles-pp.__rpc_execute.rpc_run_task.nfs4_call_sync_custom.nfs4_call_sync_sequence.nfs4_do_create
      0.84 ±  4%      +0.3        1.15 ±  8%  perf-profile.calltrace.cycles-pp.xs_tcp_send_request.xprt_transmit.call_transmit.__rpc_execute.rpc_async_schedule
      0.59 ±  9%      +0.3        0.90 ±  8%  perf-profile.calltrace.cycles-pp.rpc_wake_up_queued_task.xs_stream_data_receive.xs_stream_data_receive_workfn.process_one_work.worker_thread
      0.86 ±  3%      +0.3        1.17 ±  8%  perf-profile.calltrace.cycles-pp.xprt_transmit.call_transmit.__rpc_execute.rpc_async_schedule.process_one_work
      0.67 ±  8%      +0.3        0.98 ± 13%  perf-profile.calltrace.cycles-pp.rpc_run_task.nfs4_call_sync_custom.nfs4_call_sync_sequence.nfs4_do_create.nfs4_proc_mkdir
      0.67 ±  9%      +0.3        0.98 ± 13%  perf-profile.calltrace.cycles-pp.nfs4_call_sync_sequence.nfs4_do_create.nfs4_proc_mkdir.nfs_mkdir.vfs_mkdir
      0.67 ±  8%      +0.3        0.98 ± 13%  perf-profile.calltrace.cycles-pp.nfs4_call_sync_custom.nfs4_call_sync_sequence.nfs4_do_create.nfs4_proc_mkdir.nfs_mkdir
      0.67 ±  9%      +0.3        0.99 ± 13%  perf-profile.calltrace.cycles-pp.nfs4_do_create.nfs4_proc_mkdir.nfs_mkdir.vfs_mkdir.do_mkdirat
      0.42 ± 58%      +0.3        0.75 ±  9%  perf-profile.calltrace.cycles-pp.rpc_run_task.nfs4_call_sync_custom.nfs4_call_sync_sequence._nfs4_proc_access.nfs4_proc_access
      0.42 ± 58%      +0.3        0.76 ±  8%  perf-profile.calltrace.cycles-pp.nfs4_call_sync_sequence._nfs4_proc_access.nfs4_proc_access.nfs_do_access.nfs_permission
      0.42 ± 58%      +0.3        0.76 ±  8%  perf-profile.calltrace.cycles-pp.nfs4_call_sync_custom.nfs4_call_sync_sequence._nfs4_proc_access.nfs4_proc_access.nfs_do_access
      0.60 ±  6%      +0.3        0.94 ± 25%  perf-profile.calltrace.cycles-pp.inet6_csk_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked
      0.76 ±  8%      +0.4        1.11 ± 14%  perf-profile.calltrace.cycles-pp.nfs_mkdir.vfs_mkdir.do_mkdirat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.75 ±  8%      +0.4        1.11 ± 14%  perf-profile.calltrace.cycles-pp.nfs4_proc_mkdir.nfs_mkdir.vfs_mkdir.do_mkdirat.do_syscall_64
      0.77 ±  9%      +0.4        1.14 ± 14%  perf-profile.calltrace.cycles-pp.vfs_mkdir.do_mkdirat.do_syscall_64.entry_SYSCALL_64_after_hwframe.mkdir
      0.63 ±  5%      +0.4        1.00 ± 25%  perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg
      0.67 ±  5%      +0.4        1.03 ± 25%  perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.xs_sendpages
      0.67 ±  6%      +0.4        1.03 ± 25%  perf-profile.calltrace.cycles-pp.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg
      0.89 ± 10%      +0.4        1.28 ± 12%  perf-profile.calltrace.cycles-pp.do_mkdirat.do_syscall_64.entry_SYSCALL_64_after_hwframe.mkdir
      0.89 ±  9%      +0.4        1.29 ± 12%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.mkdir
      0.89 ±  9%      +0.4        1.29 ± 12%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.mkdir
      0.90 ±  9%      +0.4        1.31 ± 12%  perf-profile.calltrace.cycles-pp.mkdir
      1.06 ±  4%      +0.4        1.49 ±  8%  perf-profile.calltrace.cycles-pp.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe.open64
      1.03 ±  4%      +0.4        1.46 ±  7%  perf-profile.calltrace.cycles-pp.path_openat.do_filp_open.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.06 ±  4%      +0.4        1.50 ±  8%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.open64
      1.06 ±  4%      +0.4        1.50 ±  8%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.open64
      1.03 ±  4%      +0.4        1.47 ±  7%  perf-profile.calltrace.cycles-pp.do_filp_open.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe.open64
      1.08 ±  5%      +0.4        1.52 ±  7%  perf-profile.calltrace.cycles-pp.open64
      0.28 ±100%      +0.4        0.72 ±  9%  perf-profile.calltrace.cycles-pp.__rpc_execute.rpc_run_task.nfs4_call_sync_custom.nfs4_call_sync_sequence._nfs4_proc_access
      0.14 ±173%      +0.5        0.60 ±  9%  perf-profile.calltrace.cycles-pp.xfs_log_force_lsn.nfsd_create_setattr.nfsd_create_locked.nfsd4_create.nfsd4_proc_compound
      0.14 ±173%      +0.5        0.61 ±  9%  perf-profile.calltrace.cycles-pp.nfsd_create_setattr.nfsd_create_locked.nfsd4_create.nfsd4_proc_compound.nfsd_dispatch
      1.05 ±  4%      +0.5        1.52 ±  9%  perf-profile.calltrace.cycles-pp.xs_stream_data_receive.xs_stream_data_receive_workfn.process_one_work.worker_thread.kthread
      1.05 ±  4%      +0.5        1.53 ±  9%  perf-profile.calltrace.cycles-pp.xs_stream_data_receive_workfn.process_one_work.worker_thread.kthread.ret_from_fork
      1.71 ±  3%      +0.6        2.27 ±  8%  perf-profile.calltrace.cycles-pp.__rpc_execute.rpc_async_schedule.process_one_work.worker_thread.kthread
      1.72 ±  3%      +0.6        2.27 ±  8%  perf-profile.calltrace.cycles-pp.rpc_async_schedule.process_one_work.worker_thread.kthread.ret_from_fork
      0.79 ±  3%      +0.6        1.35 ± 27%  perf-profile.calltrace.cycles-pp.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.xs_sendpages.xs_tcp_send_request
      0.81 ±  3%      +0.6        1.39 ± 27%  perf-profile.calltrace.cycles-pp.tcp_sendmsg.sock_sendmsg.xs_sendpages.xs_tcp_send_request.xprt_transmit
      0.82 ±  4%      +0.6        1.42 ± 26%  perf-profile.calltrace.cycles-pp.sock_sendmsg.xs_sendpages.xs_tcp_send_request.xprt_transmit.call_transmit
      0.83 ±  4%      +0.6        1.43 ± 26%  perf-profile.calltrace.cycles-pp.xs_sendpages.xs_tcp_send_request.xprt_transmit.call_transmit.__rpc_execute
      0.00            +0.7        0.74 ±  7%  perf-profile.calltrace.cycles-pp.__next_timer_interrupt.get_next_timer_interrupt.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select
      0.46 ± 58%      +0.8        1.28 ±  6%  perf-profile.calltrace.cycles-pp.get_next_timer_interrupt.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle
      1.11 ±  7%      +1.2        2.26 ± 32%  perf-profile.calltrace.cycles-pp.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry
      5.16 ±  3%      +1.2        6.35 ±  3%  perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork
      3.85 ±  4%      +1.3        5.18 ± 14%  perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      1.40 ±  7%      +1.4        2.82 ± 25%  perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry.start_secondary
     47.03            +1.6       48.62 ±  2%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     65.35            +1.9       67.21        perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     64.66            +1.9       66.56        perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
     72.19            +3.3       75.53        perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     73.05            +3.3       76.39        perf-profile.calltrace.cycles-pp.secondary_startup_64
     72.25            +3.4       75.60        perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
     72.25            +3.4       75.61        perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
     15.86 ±  2%      -5.0       10.82 ±  4%  perf-profile.children.cycles-pp.nfsd
     23.11 ±  2%      -4.7       18.39 ±  4%  perf-profile.children.cycles-pp.ret_from_fork
     23.11 ±  2%      -4.7       18.39 ±  4%  perf-profile.children.cycles-pp.kthread
     10.35 ±  3%      -3.1        7.20 ±  4%  perf-profile.children.cycles-pp.svc_process_common
     10.35 ±  3%      -3.1        7.21 ±  5%  perf-profile.children.cycles-pp.svc_process
      9.94 ±  4%      -3.0        6.98 ±  5%  perf-profile.children.cycles-pp.nfsd_dispatch
      9.50 ±  4%      -2.8        6.70 ±  5%  perf-profile.children.cycles-pp.nfsd4_proc_compound
      4.46 ±  4%      -2.2        2.23 ±  5%  perf-profile.children.cycles-pp.__sched_text_start
      2.83 ±  2%      -2.2        0.60 ±  9%  perf-profile.children.cycles-pp.pick_next_task_fair
      2.40 ±  2%      -2.2        0.20 ± 11%  perf-profile.children.cycles-pp.newidle_balance
      3.54 ±  3%      -2.1        1.41 ±  4%  perf-profile.children.cycles-pp.schedule
      2.53            -2.0        0.53 ±  9%  perf-profile.children.cycles-pp.load_balance
      2.36 ±  2%      -2.0        0.39 ±  7%  perf-profile.children.cycles-pp.find_busiest_group
      2.99 ±  2%      -1.3        1.67 ±  6%  perf-profile.children.cycles-pp.nfsd4_write
      2.93 ±  2%      -1.3        1.63 ±  6%  perf-profile.children.cycles-pp.nfsd_vfs_write
      2.89 ±  2%      -1.3        1.62 ±  6%  perf-profile.children.cycles-pp.do_iter_write
      2.84            -1.2        1.59 ±  6%  perf-profile.children.cycles-pp.do_iter_readv_writev
      2.83            -1.2        1.58 ±  6%  perf-profile.children.cycles-pp.xfs_file_buffered_aio_write
      2.39 ±  2%      -1.2        1.23 ±  6%  perf-profile.children.cycles-pp.xfs_file_fsync
      2.83            -1.2        1.68 ±  3%  perf-profile.children.cycles-pp.svc_send
      2.71            -1.1        1.59 ±  2%  perf-profile.children.cycles-pp.svc_tcp_sendto
      2.71            -1.1        1.59 ±  2%  perf-profile.children.cycles-pp.svc_sendto
      2.70            -1.1        1.59 ±  2%  perf-profile.children.cycles-pp.svc_send_common
      2.69            -1.1        1.57 ±  2%  perf-profile.children.cycles-pp.kernel_sendpage
      2.67            -1.1        1.57 ±  2%  perf-profile.children.cycles-pp.inet_sendpage
      2.65            -1.1        1.55 ±  3%  perf-profile.children.cycles-pp.tcp_sendpage
      1.62            -1.1        0.55 ±  8%  perf-profile.children.cycles-pp.schedule_timeout
      2.55            -1.1        1.50 ±  3%  perf-profile.children.cycles-pp.do_tcp_sendpages
      2.57            -1.1        1.52 ±  3%  perf-profile.children.cycles-pp.tcp_sendpage_locked
      0.98 ±  9%      -0.8        0.22 ±  8%  perf-profile.children.cycles-pp.rcu_do_batch
      2.67            -0.7        1.93 ±  6%  perf-profile.children.cycles-pp.svc_recv
      2.45 ±  4%      -0.7        1.73 ±  4%  perf-profile.children.cycles-pp.nfsd4_open
      0.79 ± 10%      -0.7        0.11 ± 19%  perf-profile.children.cycles-pp.put_cred_rcu
      2.02 ±  9%      -0.6        1.40 ±  7%  perf-profile.children.cycles-pp.xfs_log_force_lsn
      1.04 ±  5%      -0.6        0.45 ±  7%  perf-profile.children.cycles-pp.xlog_cil_force_lsn
      0.68 ±  4%      -0.6        0.12 ± 12%  perf-profile.children.cycles-pp.wait_for_completion
      0.72 ±  5%      -0.6        0.16 ± 11%  perf-profile.children.cycles-pp.__flush_work
      0.72 ±  4%      -0.5        0.22 ±  8%  perf-profile.children.cycles-pp.idle_cpu
      1.29 ±  4%      -0.5        0.83 ±  5%  perf-profile.children.cycles-pp.xfs_dialloc
      1.48 ±  5%      -0.4        1.03 ±  6%  perf-profile.children.cycles-pp.xfs_ialloc
      3.52            -0.4        3.08 ±  5%  perf-profile.children.cycles-pp.__tcp_push_pending_frames
      1.14 ±  6%      -0.4        0.71 ±  7%  perf-profile.children.cycles-pp.xfs_dialloc_ag
      3.51            -0.4        3.08 ±  5%  perf-profile.children.cycles-pp.tcp_write_xmit
      1.48 ±  4%      -0.4        1.05 ±  6%  perf-profile.children.cycles-pp.xfs_dir_ialloc
      3.35            -0.4        2.94 ±  6%  perf-profile.children.cycles-pp.__tcp_transmit_skb
      1.86 ±  4%      -0.4        1.45 ±  4%  perf-profile.children.cycles-pp.do_nfsd_create
      3.15            -0.4        2.76 ±  6%  perf-profile.children.cycles-pp.inet6_csk_xmit
      0.84 ±  4%      -0.4        0.46 ± 10%  perf-profile.children.cycles-pp.xfs_check_agi_freecount
      3.06            -0.4        2.71 ±  6%  perf-profile.children.cycles-pp.ip6_xmit
      2.48 ±  3%      -0.3        2.14 ±  4%  perf-profile.children.cycles-pp.xfs_generic_create
      2.41 ±  4%      -0.3        2.08 ±  5%  perf-profile.children.cycles-pp.xfs_create
      2.94            -0.3        2.61 ±  6%  perf-profile.children.cycles-pp.ip6_output
      0.91 ±  3%      -0.3        0.61 ±  6%  perf-profile.children.cycles-pp.fh_verify
      1.12 ±  4%      -0.3        0.83 ±  4%  perf-profile.children.cycles-pp.vfs_create
      2.78            -0.3        2.49 ±  6%  perf-profile.children.cycles-pp.ip6_finish_output2
      2.38            -0.2        2.15 ±  7%  perf-profile.children.cycles-pp.net_rx_action
      0.67 ±  4%      -0.2        0.44        perf-profile.children.cycles-pp.xs_data_ready
      2.44            -0.2        2.20 ±  7%  perf-profile.children.cycles-pp.do_softirq_own_stack
      2.31            -0.2        2.08 ±  7%  perf-profile.children.cycles-pp.process_backlog
      2.14            -0.2        1.91 ±  7%  perf-profile.children.cycles-pp.ip6_input_finish
      2.27            -0.2        2.04 ±  7%  perf-profile.children.cycles-pp.__netif_receive_skb_one_core
      2.21            -0.2        1.98 ±  7%  perf-profile.children.cycles-pp.ipv6_rcv
      2.14            -0.2        1.91 ±  7%  perf-profile.children.cycles-pp.ip6_protocol_deliver_rcu
      2.10            -0.2        1.88 ±  7%  perf-profile.children.cycles-pp.tcp_v6_rcv
      2.46            -0.2        2.24 ±  6%  perf-profile.children.cycles-pp.do_softirq
      2.49            -0.2        2.27 ±  7%  perf-profile.children.cycles-pp.__local_bh_enable_ip
      2.16            -0.2        1.94 ±  7%  perf-profile.children.cycles-pp.ip6_input
      0.36 ±  9%      -0.2        0.15 ± 19%  perf-profile.children.cycles-pp.nfsd4_process_open2
      0.44 ±  9%      -0.2        0.24 ±  6%  perf-profile.children.cycles-pp.nfsd4_encode_operation
      1.01 ±  5%      -0.2        0.81 ±  5%  perf-profile.children.cycles-pp.xfs_vm_writepages
      0.77 ± 17%      -0.2        0.59 ±  6%  perf-profile.children.cycles-pp.scheduler_tick
      0.28 ±  7%      -0.2        0.09 ± 26%  perf-profile.children.cycles-pp.cpumask_next_and
      0.28 ± 10%      -0.2        0.11 ± 24%  perf-profile.children.cycles-pp.xfs_btree_increment
      0.34 ± 11%      -0.2        0.17 ± 12%  perf-profile.children.cycles-pp.xfs_btree_check_sblock
      0.46 ± 11%      -0.2        0.29 ± 13%  perf-profile.children.cycles-pp.xfs_inobt_get_rec
      0.28 ± 12%      -0.2        0.11 ± 19%  perf-profile.children.cycles-pp.nfs4_get_vfs_file
      0.28 ± 11%      -0.2        0.11 ± 23%  perf-profile.children.cycles-pp.nfsd_file_acquire
      0.86 ±  6%      -0.2        0.70 ±  5%  perf-profile.children.cycles-pp.write_cache_pages
      0.79 ±  9%      -0.2        0.63 ±  3%  perf-profile.children.cycles-pp.iomap_writepages
      0.72 ±  8%      -0.2        0.57 ±  5%  perf-profile.children.cycles-pp.iomap_do_writepage
      0.33 ±  9%      -0.2        0.18 ±  3%  perf-profile.children.cycles-pp.nfsd4_encode_fattr
      0.33 ± 10%      -0.2        0.18 ±  4%  perf-profile.children.cycles-pp.nfsd4_encode_getattr
      1.41 ±  2%      -0.1        1.26 ±  4%  perf-profile.children.cycles-pp.file_write_and_wait_range
      0.43 ±  7%      -0.1        0.29 ± 20%  perf-profile.children.cycles-pp.xfs_trans_read_buf_map
      0.67 ±  8%      -0.1        0.53 ±  5%  perf-profile.children.cycles-pp.xfs_map_blocks
      0.65 ±  8%      -0.1        0.51 ±  4%  perf-profile.children.cycles-pp.xfs_bmapi_convert_delalloc
      0.60 ±  3%      -0.1        0.46 ±  8%  perf-profile.children.cycles-pp.svc_xprt_received
      0.50 ±  9%      -0.1        0.36 ±  7%  perf-profile.children.cycles-pp.xfs_bmapi_allocate
      0.82 ±  9%      -0.1        0.69 ±  4%  perf-profile.children.cycles-pp.xlog_ioend_work
      0.46 ±  7%      -0.1        0.33 ±  9%  perf-profile.children.cycles-pp.xfs_bmap_btalloc
      0.46 ±  4%      -0.1        0.33 ±  7%  perf-profile.children.cycles-pp.nfsd_setuser_and_check_port
      0.80 ±  8%      -0.1        0.68 ±  4%  perf-profile.children.cycles-pp.xlog_state_do_callback
      0.38 ± 10%      -0.1        0.26 ±  5%  perf-profile.children.cycles-pp.update_blocked_averages
      0.43 ±  4%      -0.1        0.32 ±  9%  perf-profile.children.cycles-pp.nfsd_setuser
      0.22 ±  8%      -0.1        0.10 ±  8%  perf-profile.children.cycles-pp.nfsd4_getattr
      0.20 ±  7%      -0.1        0.08 ± 23%  perf-profile.children.cycles-pp.find_next_and_bit
      0.43 ±  7%      -0.1        0.33 ± 16%  perf-profile.children.cycles-pp.xfs_alloc_vextent
      1.21 ±  2%      -0.1        1.10 ±  5%  perf-profile.children.cycles-pp.__filemap_fdatawrite_range
      1.20 ±  2%      -0.1        1.09 ±  4%  perf-profile.children.cycles-pp.do_writepages
      0.27 ± 15%      -0.1        0.17 ± 34%  perf-profile.children.cycles-pp.xfs_btree_get_rec
      0.20 ±  9%      -0.1        0.11 ± 33%  perf-profile.children.cycles-pp.__xfs_btree_check_sblock
      0.45 ± 12%      -0.1        0.36 ±  6%  perf-profile.children.cycles-pp.xlog_state_clean_iclog
      0.29 ±  6%      -0.1        0.21 ±  8%  perf-profile.children.cycles-pp.xfs_alloc_ag_vextent_near
      0.13 ±  3%      -0.1        0.04 ± 58%  perf-profile.children.cycles-pp._xfs_trans_bjoin
      0.15 ±  7%      -0.1        0.06 ± 13%  perf-profile.children.cycles-pp.xfs_fs_fh_to_dentry
      0.21 ±  4%      -0.1        0.12 ± 12%  perf-profile.children.cycles-pp.kfree
      0.17 ±  4%      -0.1        0.09 ±  9%  perf-profile.children.cycles-pp.exportfs_decode_fh
      0.36 ±  4%      -0.1        0.28 ±  6%  perf-profile.children.cycles-pp.lookup_one_len
      0.35 ±  9%      -0.1        0.27 ± 10%  perf-profile.children.cycles-pp.xfs_btree_lookup
      0.17 ± 14%      -0.1        0.10 ±  9%  perf-profile.children.cycles-pp.__radix_tree_lookup
      0.14 ±  8%      -0.1        0.06 ± 11%  perf-profile.children.cycles-pp.xfs_nfs_get_inode
      0.27 ± 11%      -0.1        0.20 ± 16%  perf-profile.children.cycles-pp.xfs_buf_get_map
      0.27 ± 11%      -0.1        0.20 ± 20%  perf-profile.children.cycles-pp.xfs_buf_read_map
      0.10 ± 13%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.fh_put
      0.11 ± 17%      -0.1        0.04 ± 58%  perf-profile.children.cycles-pp.xfs_perag_get
      0.09 ± 15%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.dentry_open
      0.12 ± 10%      -0.1        0.05 ± 59%  perf-profile.children.cycles-pp.nfsd4_close
      0.24 ±  8%      -0.1        0.18 ± 15%  perf-profile.children.cycles-pp.__slab_free
      0.28 ±  6%      -0.1        0.21 ± 10%  perf-profile.children.cycles-pp.prepare_creds
      0.30 ±  7%      -0.1        0.24 ±  7%  perf-profile.children.cycles-pp.__lookup_slow
      0.25 ±  5%      -0.1        0.19 ±  6%  perf-profile.children.cycles-pp.xfs_iget
      0.11 ± 17%      -0.1        0.04 ± 58%  perf-profile.children.cycles-pp.__nfsd_open
      0.32 ±  7%      -0.1        0.25 ±  8%  perf-profile.children.cycles-pp.nfsd_create
      0.25 ± 10%      -0.1        0.19 ± 16%  perf-profile.children.cycles-pp.xfs_btree_lookup_get_block
      0.10 ±  8%      -0.1        0.04 ± 58%  perf-profile.children.cycles-pp.exp_find
      0.10 ± 15%      -0.1        0.04 ± 57%  perf-profile.children.cycles-pp.svc_reserve
      0.11 ± 13%      -0.1        0.05 ± 57%  perf-profile.children.cycles-pp.rqst_exp_find
      0.15 ± 13%      -0.1        0.09 ± 11%  perf-profile.children.cycles-pp.nfs4svc_decode_compoundargs
      0.14 ±  5%      -0.1        0.08 ±  5%  perf-profile.children.cycles-pp.dput
      0.13 ± 21%      -0.1        0.07 ± 14%  perf-profile.children.cycles-pp.finish_task_switch
      0.26 ±  7%      -0.1        0.20 ±  9%  perf-profile.children.cycles-pp.xfs_vn_lookup
      0.18 ±  6%      -0.1        0.13 ± 17%  perf-profile.children.cycles-pp.sk_stream_alloc_skb
      0.09 ± 12%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.security_prepare_creds
      0.08 ±  8%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.xfs_read_agi
      0.23 ±  7%      -0.1        0.18 ±  5%  perf-profile.children.cycles-pp.xfs_lookup
      0.28 ±  9%      -0.1        0.23 ±  8%  perf-profile.children.cycles-pp.iomap_apply
      0.17 ± 11%      -0.1        0.12 ± 15%  perf-profile.children.cycles-pp.nfsd_cache_lookup
      0.08 ±  8%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.xfs_log_reserve
      0.17 ±  4%      -0.1        0.12 ± 25%  perf-profile.children.cycles-pp.__alloc_skb
      0.28 ±  9%      -0.1        0.23 ±  9%  perf-profile.children.cycles-pp.iomap_file_buffered_write
      0.22 ±  9%      -0.0        0.17 ± 20%  perf-profile.children.cycles-pp.xfs_buf_find
      0.23 ±  6%      -0.0        0.18 ±  5%  perf-profile.children.cycles-pp.xfs_dir_lookup
      0.09 ±  7%      -0.0        0.04 ± 58%  perf-profile.children.cycles-pp.xfs_ialloc_read_agi
      0.07 ± 14%      -0.0        0.03 ±100%  perf-profile.children.cycles-pp.inet6_csk_route_socket
      0.25 ±  5%      -0.0        0.20 ±  5%  perf-profile.children.cycles-pp.__dev_queue_xmit
      0.14 ±  3%      -0.0        0.09 ± 13%  perf-profile.children.cycles-pp.xfs_trans_alloc
      0.07 ± 11%      -0.0        0.03 ±100%  perf-profile.children.cycles-pp.down_read
      0.12 ±  9%      -0.0        0.07 ± 10%  perf-profile.children.cycles-pp.write_bytes_to_xdr_buf
      0.17 ±  7%      -0.0        0.13 ±  3%  perf-profile.children.cycles-pp.xfs_dir2_node_lookup
      0.09 ± 11%      -0.0        0.05 ± 60%  perf-profile.children.cycles-pp.svcauth_unix_set_client
      0.16 ±  9%      -0.0        0.12 ± 12%  perf-profile.children.cycles-pp.xfs_dialloc_ag_update_inobt
      0.17 ± 10%      -0.0        0.13 ± 11%  perf-profile.children.cycles-pp.xfs_da_read_buf
      0.09 ± 17%      -0.0        0.04 ± 59%  perf-profile.children.cycles-pp.security_cred_free
      0.21 ±  9%      -0.0        0.17 ± 12%  perf-profile.children.cycles-pp.iomap_submit_ioend
      0.21 ±  4%      -0.0        0.18 ±  2%  perf-profile.children.cycles-pp.__kmalloc
      0.20 ±  7%      -0.0        0.17 ±  8%  perf-profile.children.cycles-pp.iomap_write_actor
      0.08 ± 10%      -0.0        0.06 ± 14%  perf-profile.children.cycles-pp.xfs_trans_reserve
      0.12 ±  7%      -0.0        0.09 ±  7%  perf-profile.children.cycles-pp.nf_hook_slow
      0.08 ±  8%      -0.0        0.06 ±  7%  perf-profile.children.cycles-pp.brd_lookup_page
      0.09 ±  7%      -0.0        0.07 ± 12%  perf-profile.children.cycles-pp.___perf_sw_event
      0.09 ± 11%      -0.0        0.07 ±  5%  perf-profile.children.cycles-pp.selinux_ip_postroute
      0.07 ±  6%      -0.0        0.05 ±  8%  perf-profile.children.cycles-pp.xfs_btree_update
      0.09 ±  7%      +0.0        0.11 ±  6%  perf-profile.children.cycles-pp.xfs_trans_unreserve_and_mod_sb
      0.12 ±  8%      +0.0        0.14 ±  9%  perf-profile.children.cycles-pp.raise_softirq
      0.08 ± 14%      +0.0        0.10 ± 10%  perf-profile.children.cycles-pp.__percpu_counter_sum
      0.05 ±  9%      +0.0        0.08 ±  8%  perf-profile.children.cycles-pp.drm_fb_helper_dirty_work
      0.08 ± 15%      +0.0        0.11 ±  4%  perf-profile.children.cycles-pp.__percpu_counter_compare
      0.04 ± 57%      +0.0        0.07 ± 13%  perf-profile.children.cycles-pp.rpc_task_set_transport
      0.06 ±  6%      +0.0        0.09 ± 20%  perf-profile.children.cycles-pp.xfs_buf_item_size_segment
      0.07 ±  7%      +0.0        0.10 ± 15%  perf-profile.children.cycles-pp.rpcauth_wrap_req_encode
      0.11 ± 10%      +0.0        0.14 ±  7%  perf-profile.children.cycles-pp.xprt_release
      0.18 ±  8%      +0.0        0.21 ±  8%  perf-profile.children.cycles-pp._raw_spin_lock_bh
      0.08 ± 21%      +0.0        0.12 ± 17%  perf-profile.children.cycles-pp.filename_create
      0.07 ± 20%      +0.0        0.11 ±  4%  perf-profile.children.cycles-pp.decode_getfattr_attrs
      0.07 ± 13%      +0.0        0.10 ± 15%  perf-profile.children.cycles-pp.xfs_cil_prepare_item
      0.04 ± 57%      +0.0        0.07 ± 22%  perf-profile.children.cycles-pp.xprt_free_slot
      0.09 ± 19%      +0.0        0.13 ± 14%  perf-profile.children.cycles-pp.decode_getfattr_generic
      0.06 ± 15%      +0.0        0.10 ± 18%  perf-profile.children.cycles-pp._nfs4_opendata_to_nfs4_state
      0.07 ± 11%      +0.0        0.11 ± 14%  perf-profile.children.cycles-pp.iget5_locked
      0.16 ±  5%      +0.0        0.20 ±  4%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.28 ±  6%      +0.0        0.33 ±  8%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.16 ±  6%      +0.1        0.21 ±  8%  perf-profile.children.cycles-pp.rpc_release_resources_task
      0.00            +0.1        0.05 ±  8%  perf-profile.children.cycles-pp.xprt_iter_get_next
      0.52 ±  2%      +0.1        0.57 ±  5%  perf-profile.children.cycles-pp.native_sched_clock
      0.11 ± 10%      +0.1        0.16 ±  7%  perf-profile.children.cycles-pp.xfs_buf_item_format
      0.00            +0.1        0.06 ±  9%  perf-profile.children.cycles-pp.native_apic_mem_write
      0.18 ± 16%      +0.1        0.24 ± 10%  perf-profile.children.cycles-pp.nfs4_do_close
      0.54 ±  2%      +0.1        0.60 ±  5%  perf-profile.children.cycles-pp.sched_clock
      0.15 ±  7%      +0.1        0.20 ± 16%  perf-profile.children.cycles-pp.generic_perform_write
      0.01 ±173%      +0.1        0.07 ± 17%  perf-profile.children.cycles-pp.rcu_gp_kthread
      0.01 ±173%      +0.1        0.07 ± 17%  perf-profile.children.cycles-pp.xprt_do_reserve
      0.17 ± 13%      +0.1        0.23 ± 16%  perf-profile.children.cycles-pp.rpc_async_release
      0.71 ±  4%      +0.1        0.78 ±  2%  perf-profile.children.cycles-pp.lapic_next_deadline
      0.16 ±  7%      +0.1        0.22 ± 16%  perf-profile.children.cycles-pp.nfs_file_write
      0.16 ± 15%      +0.1        0.23 ± 16%  perf-profile.children.cycles-pp.nfs_write_completion
      0.25            +0.1        0.32 ± 11%  perf-profile.children.cycles-pp.__switch_to_asm
      0.85 ±  2%      +0.1        0.92 ±  3%  perf-profile.children.cycles-pp.native_write_msr
      0.24 ±  4%      +0.1        0.30        perf-profile.children.cycles-pp.irq_work_run
      0.24 ±  4%      +0.1        0.30        perf-profile.children.cycles-pp.printk
      0.24 ±  4%      +0.1        0.30        perf-profile.children.cycles-pp.vprintk_emit
      0.21 ± 11%      +0.1        0.28 ± 10%  perf-profile.children.cycles-pp.__put_nfs_open_context
      0.09 ± 28%      +0.1        0.16 ±  7%  perf-profile.children.cycles-pp.nfs_initiate_pgio
      0.24 ±  4%      +0.1        0.31 ±  2%  perf-profile.children.cycles-pp.irq_work_interrupt
      0.24 ±  4%      +0.1        0.31 ±  2%  perf-profile.children.cycles-pp.smp_irq_work_interrupt
      0.20 ± 10%      +0.1        0.27 ± 13%  perf-profile.children.cycles-pp.rpc_free_task
      0.21 ± 11%      +0.1        0.28 ± 10%  perf-profile.children.cycles-pp.nfs_file_release
      0.41 ±  9%      +0.1        0.48 ±  3%  perf-profile.children.cycles-pp.dequeue_task_fair
      0.16 ±  4%      +0.1        0.23 ±  7%  perf-profile.children.cycles-pp.call_encode
      0.30 ± 13%      +0.1        0.38 ±  5%  perf-profile.children.cycles-pp.dequeue_entity
      0.16 ±  4%      +0.1        0.24 ±  6%  perf-profile.children.cycles-pp.rpc_wait_bit_killable
      0.26 ± 11%      +0.1        0.34 ± 10%  perf-profile.children.cycles-pp.__fput
      0.17 ± 14%      +0.1        0.25 ± 11%  perf-profile.children.cycles-pp.call_decode
      0.41 ±  9%      +0.1        0.49 ±  8%  perf-profile.children.cycles-pp.xlog_cil_push
      0.14 ± 18%      +0.1        0.23 ± 30%  perf-profile.children.cycles-pp.nfs4_run_open_task
      0.11 ± 23%      +0.1        0.19 ±  5%  perf-profile.children.cycles-pp.nfs_pageio_complete
      0.10 ± 21%      +0.1        0.19 ±  3%  perf-profile.children.cycles-pp.nfs_pageio_doio
      0.10 ± 21%      +0.1        0.19 ±  3%  perf-profile.children.cycles-pp.nfs_generic_pg_pgios
      0.24 ± 12%      +0.1        0.33 ±  7%  perf-profile.children.cycles-pp.task_work_run
      0.19 ±  6%      +0.1        0.28 ±  3%  perf-profile.children.cycles-pp.out_of_line_wait_on_bit
      0.19 ±  6%      +0.1        0.28 ±  3%  perf-profile.children.cycles-pp.__wait_on_bit
      0.30 ±  8%      +0.1        0.39 ±  9%  perf-profile.children.cycles-pp.xs_sock_recvmsg
      0.17 ±  7%      +0.1        0.27 ± 13%  perf-profile.children.cycles-pp.new_sync_write
      0.24 ± 12%      +0.1        0.34 ±  6%  perf-profile.children.cycles-pp.exit_to_usermode_loop
      0.18 ± 20%      +0.1        0.28 ±  7%  perf-profile.children.cycles-pp.nfs_writepages
      0.19 ±  8%      +0.1        0.29 ± 14%  perf-profile.children.cycles-pp.write
      0.27 ± 11%      +0.1        0.37 ±  6%  perf-profile.children.cycles-pp.close
      0.18 ±  8%      +0.1        0.29 ± 15%  perf-profile.children.cycles-pp.ksys_write
      0.18 ± 10%      +0.1        0.29 ± 15%  perf-profile.children.cycles-pp.vfs_write
      0.16 ± 11%      +0.1        0.27 ± 13%  perf-profile.children.cycles-pp.io_serial_in
      0.23 ± 11%      +0.1        0.34 ± 12%  perf-profile.children.cycles-pp.serial8250_console_putchar
      0.23 ± 14%      +0.1        0.35 ± 12%  perf-profile.children.cycles-pp.uart_console_write
      0.12 ± 23%      +0.1        0.24 ± 10%  perf-profile.children.cycles-pp.hrtimer_get_next_event
      0.00            +0.1        0.11 ±  4%  perf-profile.children.cycles-pp.run_ksoftirqd
      0.23 ± 11%      +0.1        0.35 ± 12%  perf-profile.children.cycles-pp.wait_for_xmitr
      0.22 ±  8%      +0.1        0.34 ±  3%  perf-profile.children.cycles-pp.__hrtimer_next_event_base
      0.25 ± 13%      +0.1        0.38 ± 11%  perf-profile.children.cycles-pp.console_unlock
      0.24 ± 14%      +0.1        0.36 ± 11%  perf-profile.children.cycles-pp.serial8250_console_write
      0.27 ± 14%      +0.1        0.39 ± 11%  perf-profile.children.cycles-pp.irq_work_run_list
      0.01 ±173%      +0.1        0.14 ±  6%  perf-profile.children.cycles-pp.smpboot_thread_fn
      0.30 ± 14%      +0.1        0.43 ± 16%  perf-profile.children.cycles-pp.nfs4_do_open
      0.67 ±  5%      +0.1        0.81 ±  3%  perf-profile.children.cycles-pp.xfs_log_commit_cil
      0.32 ± 14%      +0.1        0.46 ± 13%  perf-profile.children.cycles-pp.nfs_atomic_open
      0.30 ± 14%      +0.1        0.43 ± 15%  perf-profile.children.cycles-pp.nfs4_atomic_open
      0.25 ± 18%      +0.1        0.39 ±  2%  perf-profile.children.cycles-pp.nfs_file_fsync
      0.21 ± 14%      +0.1        0.35 ±  8%  perf-profile.children.cycles-pp.rpc_execute
      0.26 ± 18%      +0.1        0.40 ±  2%  perf-profile.children.cycles-pp.do_fsync
      0.24 ± 14%      +0.1        0.39 ± 16%  perf-profile.children.cycles-pp._nfs4_open_and_get_state
      0.26 ± 18%      +0.1        0.40        perf-profile.children.cycles-pp.__x64_sys_fsync
      0.27 ± 18%      +0.2        0.42        perf-profile.children.cycles-pp.fsync
      0.71 ±  5%      +0.2        0.86 ±  3%  perf-profile.children.cycles-pp.__xfs_trans_commit
      1.18 ±  2%      +0.2        1.35 ±  3%  perf-profile.children.cycles-pp.unwind_next_frame
      0.30 ±  8%      +0.2        0.47 ± 12%  perf-profile.children.cycles-pp.__wake_up_bit
      1.63 ±  3%      +0.2        1.82 ±  5%  perf-profile.children.cycles-pp.__wake_up_common
      1.62            +0.2        1.82 ±  5%  perf-profile.children.cycles-pp.arch_stack_walk
      0.53 ±  7%      +0.2        0.74 ±  6%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.57 ±  7%      +0.2        0.79 ±  8%  perf-profile.children.cycles-pp._nfs4_proc_access
      0.57 ±  8%      +0.2        0.79 ±  8%  perf-profile.children.cycles-pp.nfs4_proc_access
      0.85 ± 11%      +0.2        1.07 ± 10%  perf-profile.children.cycles-pp.nfsd_create_setattr
      1.84 ±  2%      +0.2        2.08 ±  5%  perf-profile.children.cycles-pp.stack_trace_save_tsk
      0.59 ±  6%      +0.2        0.83 ±  8%  perf-profile.children.cycles-pp.nfs_do_access
      0.64 ±  6%      +0.3        0.90 ±  9%  perf-profile.children.cycles-pp.inode_permission
      0.60 ±  6%      +0.3        0.86 ±  9%  perf-profile.children.cycles-pp.nfs_permission
      0.75 ±  6%      +0.3        1.02 ±  4%  perf-profile.children.cycles-pp.autoremove_wake_function
      2.61 ±  3%      +0.3        2.89 ±  4%  perf-profile.children.cycles-pp.activate_task
      0.19 ± 11%      +0.3        0.46 ±  2%  perf-profile.children.cycles-pp.hrtimer_next_event_without
      2.60 ±  3%      +0.3        2.88 ±  4%  perf-profile.children.cycles-pp.enqueue_task_fair
      2.62 ±  3%      +0.3        2.90 ±  4%  perf-profile.children.cycles-pp.ttwu_do_activate
      0.27 ± 19%      +0.3        0.55 ±  6%  perf-profile.children.cycles-pp.find_next_bit
      2.09 ±  3%      +0.3        2.38 ±  5%  perf-profile.children.cycles-pp.__account_scheduler_latency
      0.59 ±  9%      +0.3        0.89 ±  8%  perf-profile.children.cycles-pp.__rpc_do_wake_up_task_on_wq
      0.68 ±  5%      +0.3        0.98 ±  9%  perf-profile.children.cycles-pp.link_path_walk
      0.59 ±  9%      +0.3        0.90 ±  8%  perf-profile.children.cycles-pp.rpc_wake_up_queued_task
      2.47 ±  3%      +0.3        2.79 ±  4%  perf-profile.children.cycles-pp.enqueue_entity
      0.67 ±  9%      +0.3        0.99 ± 13%  perf-profile.children.cycles-pp.nfs4_do_create
      0.76 ±  8%      +0.4        1.11 ± 14%  perf-profile.children.cycles-pp.nfs_mkdir
      0.75 ±  8%      +0.4        1.11 ± 14%  perf-profile.children.cycles-pp.nfs4_proc_mkdir
      0.89 ± 10%      +0.4        1.28 ± 12%  perf-profile.children.cycles-pp.do_mkdirat
      0.90 ±  9%      +0.4        1.31 ± 12%  perf-profile.children.cycles-pp.mkdir
      1.07 ±  5%      +0.4        1.50 ±  7%  perf-profile.children.cycles-pp.do_sys_open
      1.05 ±  5%      +0.4        1.48 ±  7%  perf-profile.children.cycles-pp.do_filp_open
      1.04 ±  5%      +0.4        1.48 ±  8%  perf-profile.children.cycles-pp.path_openat
      1.08 ±  4%      +0.4        1.52 ±  7%  perf-profile.children.cycles-pp.open64
      0.39 ± 11%      +0.5        0.85 ±  5%  perf-profile.children.cycles-pp.__next_timer_interrupt
      1.05 ±  4%      +0.5        1.53 ±  9%  perf-profile.children.cycles-pp.xs_stream_data_receive_workfn
      1.05 ±  4%      +0.5        1.53 ±  9%  perf-profile.children.cycles-pp.xs_stream_data_receive
      1.44 ±  4%      +0.5        1.96 ± 10%  perf-profile.children.cycles-pp.tcp_sendmsg_locked
      1.52 ±  3%      +0.5        2.03 ± 10%  perf-profile.children.cycles-pp.sock_sendmsg
      1.22 ±  6%      +0.5        1.74 ±  8%  perf-profile.children.cycles-pp.nfs4_call_sync_custom
      1.22 ±  6%      +0.5        1.75 ±  8%  perf-profile.children.cycles-pp.nfs4_call_sync_sequence
      1.47 ±  4%      +0.5        2.00 ±  9%  perf-profile.children.cycles-pp.tcp_sendmsg
      1.52 ±  3%      +0.5        2.06 ±  9%  perf-profile.children.cycles-pp.xs_sendpages
      1.55 ±  3%      +0.5        2.10 ±  9%  perf-profile.children.cycles-pp.xs_tcp_send_request
      1.72 ±  3%      +0.6        2.27 ±  8%  perf-profile.children.cycles-pp.rpc_async_schedule
      1.60 ±  3%      +0.6        2.15 ±  9%  perf-profile.children.cycles-pp.call_transmit
      1.58 ±  3%      +0.6        2.14 ±  9%  perf-profile.children.cycles-pp.xprt_transmit
      0.01 ±173%      +0.6        0.58 ± 93%  perf-profile.children.cycles-pp.timekeeping_max_deferment
      1.46 ±  6%      +0.7        2.14 ±  7%  perf-profile.children.cycles-pp.rpc_run_task
      0.59 ± 11%      +0.7        1.29 ±  5%  perf-profile.children.cycles-pp.get_next_timer_interrupt
      2.88 ±  4%      +1.1        3.94 ±  8%  perf-profile.children.cycles-pp.__rpc_execute
      1.12 ±  7%      +1.2        2.29 ± 31%  perf-profile.children.cycles-pp.tick_nohz_next_event
      5.16 ±  3%      +1.2        6.36 ±  3%  perf-profile.children.cycles-pp.process_one_work
      2.86 ±  6%      +1.2        4.08 ±  5%  perf-profile.children.cycles-pp.do_syscall_64
      2.87 ±  6%      +1.2        4.09 ±  5%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
      3.90 ±  4%      +1.4        5.26 ± 14%  perf-profile.children.cycles-pp.menu_select
      1.42 ±  7%      +1.4        2.85 ± 25%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
     47.06            +1.6       48.70 ±  2%  perf-profile.children.cycles-pp.intel_idle
     66.08            +1.9       67.93        perf-profile.children.cycles-pp.cpuidle_enter
     66.05            +1.9       67.90        perf-profile.children.cycles-pp.cpuidle_enter_state
     73.05            +3.3       76.39        perf-profile.children.cycles-pp.secondary_startup_64
     73.05            +3.3       76.39        perf-profile.children.cycles-pp.cpu_startup_entry
     73.08            +3.3       76.42        perf-profile.children.cycles-pp.do_idle
     72.25            +3.4       75.61        perf-profile.children.cycles-pp.start_secondary
      1.58 ±  2%      -1.3        0.26 ±  4%  perf-profile.self.cycles-pp.find_busiest_group
      0.72 ±  4%      -0.5        0.22 ±  8%  perf-profile.self.cycles-pp.idle_cpu
      0.28 ±  4%      -0.1        0.16 ± 11%  perf-profile.self.cycles-pp.try_to_wake_up
      0.18 ±  8%      -0.1        0.08 ± 23%  perf-profile.self.cycles-pp.find_next_and_bit
      0.17 ± 12%      -0.1        0.08 ± 20%  perf-profile.self.cycles-pp.__xfs_btree_check_sblock
      0.20 ±  4%      -0.1        0.12 ± 15%  perf-profile.self.cycles-pp.kfree
      0.17 ± 14%      -0.1        0.09 ± 11%  perf-profile.self.cycles-pp.__radix_tree_lookup
      0.14 ± 12%      -0.1        0.07 ± 20%  perf-profile.self.cycles-pp.nfsd4_proc_compound
      0.10 ± 15%      -0.1        0.04 ± 57%  perf-profile.self.cycles-pp.svc_reserve
      0.09 ± 31%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.nfsd4_encode_fattr
      0.33 ± 10%      -0.1        0.28        perf-profile.self.cycles-pp.update_rq_clock
      0.23 ±  2%      -0.1        0.17 ± 16%  perf-profile.self.cycles-pp.__slab_free
      0.07 ± 10%      -0.0        0.03 ±100%  perf-profile.self.cycles-pp.__tcp_transmit_skb
      0.07 ±  5%      -0.0        0.03 ±100%  perf-profile.self.cycles-pp.avc_has_perm
      0.11 ±  7%      -0.0        0.07 ±  5%  perf-profile.self.cycles-pp.__kmalloc
      0.11 ±  7%      -0.0        0.08 ±  6%  perf-profile.self.cycles-pp.update_blocked_averages
      0.12 ± 10%      -0.0        0.08 ±  5%  perf-profile.self.cycles-pp.enqueue_task_fair
      0.11 ±  9%      -0.0        0.08 ± 14%  perf-profile.self.cycles-pp.xfs_buf_find
      0.09 ±  5%      -0.0        0.05 ± 58%  perf-profile.self.cycles-pp.tcp_v6_rcv
      0.13 ± 11%      -0.0        0.10 ± 18%  perf-profile.self.cycles-pp.load_balance
      0.11 ±  4%      -0.0        0.08 ±  5%  perf-profile.self.cycles-pp.skb_release_data
      0.06 ± 11%      +0.0        0.09 ± 17%  perf-profile.self.cycles-pp.worker_thread
      0.05 ± 60%      +0.0        0.09 ± 13%  perf-profile.self.cycles-pp.apic_timer_interrupt
      0.16 ±  5%      +0.0        0.20 ±  4%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.27 ±  5%      +0.0        0.32 ±  6%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.49 ±  2%      +0.1        0.55 ±  5%  perf-profile.self.cycles-pp.native_sched_clock
      0.01 ±173%      +0.1        0.07 ± 11%  perf-profile.self.cycles-pp.decode_getfattr_attrs
      0.65 ±  4%      +0.1        0.71 ±  2%  perf-profile.self.cycles-pp.unwind_next_frame
      0.25            +0.1        0.32 ± 11%  perf-profile.self.cycles-pp.__switch_to_asm
      0.01 ±173%      +0.1        0.08 ± 16%  perf-profile.self.cycles-pp.dequeue_entity
      0.85 ±  2%      +0.1        0.92 ±  3%  perf-profile.self.cycles-pp.native_write_msr
      0.10 ± 10%      +0.1        0.18 ± 10%  perf-profile.self.cycles-pp.process_one_work
      0.34 ± 14%      +0.1        0.42 ± 12%  perf-profile.self.cycles-pp.__softirqentry_text_start
      0.01 ±173%      +0.1        0.11 ± 10%  perf-profile.self.cycles-pp.get_next_timer_interrupt
      0.20 ±  5%      +0.1        0.29 ±  2%  perf-profile.self.cycles-pp.__hrtimer_next_event_base
      0.54 ±  3%      +0.1        0.64 ±  9%  perf-profile.self.cycles-pp.do_idle
      0.16 ± 11%      +0.1        0.27 ± 13%  perf-profile.self.cycles-pp.io_serial_in
      0.30 ± 27%      +0.2        0.45 ± 14%  perf-profile.self.cycles-pp.brd_do_bvec
      0.45 ±  3%      +0.2        0.61 ±  5%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.24 ± 19%      +0.2        0.44 ±  7%  perf-profile.self.cycles-pp.find_next_bit
      0.18 ± 10%      +0.2        0.43 ±  5%  perf-profile.self.cycles-pp.__next_timer_interrupt
      1.34 ±  2%      +0.3        1.65 ±  4%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.01 ±173%      +0.6        0.58 ± 93%  perf-profile.self.cycles-pp.timekeeping_max_deferment
     47.03            +1.7       48.69 ±  2%  perf-profile.self.cycles-pp.intel_idle


                                                                                
                      fsmark.time.percent_of_cpu_this_job_got                   
                                                                                
    21 +--------------------------------------------------------------------+   
       |                                                                    |   
  20.5 |-+                                                                  |   
       |                                                                    |   
       |                                                                    |   
    20 |-+                                                                  |   
       |                                                                    |   
  19.5 |-+                                                                  |   
       |                                                                    |   
    19 |-+  +..+      +    +..+       +.+..+.+..+.+..+..+.+..+.+..+.+..+.+..|   
       |:   :  :      ::   :   :     :                                      |   
       |:  :    :    : :  :    :     :                                      |   
  18.5 |-: :    :    :  : :     :   :                                       |   
       | ::      :  :   ::      :   :                                       |   
    18 +--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                             fsmark.time.elapsed_time                           
                                                                                
  410 +---------------------------------------------------------------------+   
      | .+.+..+.+..+..+.+..+.+..+..+                                        |   
  400 |.+                           +  .+.  .+.                             |   
  390 |-+                            +.   +.   +..+..+.+..+.+..+..+.+..+.+..|   
      |                                                                     |   
  380 |-+                                                                   |   
      |                                                                     |   
  370 |-+                                                                   |   
      |                                                                     |   
  360 |-+                                                                   |   
  350 |-+                                                                   |   
      |       O       O      O     O      O                                 |   
  340 |-+O O    O  O    O  O    O    O  O                 O                 |   
      |                                      O O  O  O O    O  O            |   
  330 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                           fsmark.time.elapsed_time.max                         
                                                                                
  410 +---------------------------------------------------------------------+   
      | .+.+..+.+..+..+.+..+.+..+..+                                        |   
  400 |.+                           +  .+.  .+.                             |   
  390 |-+                            +.   +.   +..+..+.+..+.+..+..+.+..+.+..|   
      |                                                                     |   
  380 |-+                                                                   |   
      |                                                                     |   
  370 |-+                                                                   |   
      |                                                                     |   
  360 |-+                                                                   |   
  350 |-+                                                                   |   
      |       O       O      O     O      O                                 |   
  340 |-+O O    O  O    O  O    O    O  O                 O                 |   
      |                                      O O  O  O O    O  O            |   
  330 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                               fsmark.files_per_sec                             
                                                                                
  3100 +--------------------------------------------------------------------+   
       |                                        O    O                      |   
  3000 |-+                              O    O    O     O    O O            |   
       |  O O    O  O O  O O     O O  O                   O                 |   
  2900 |-+     O              O            O                                |   
       |                                                                    |   
  2800 |-+                                                                  |   
       |                                                                    |   
  2700 |-+                                                                  |   
       |                                                                    |   
  2600 |-+                                       .+..  .+.  .+.+.. .+..+.+..|   
       |                             .+.+..+.+..+    +.   +.      +         |   
  2500 |..+.+..+.+..+.+..+.+..+..+.+.                                       |   
       |                                                                    |   
  2400 +--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                  fsmark.app_overhead                           
                                                                                
  1.14e+08 +----------------------------------------------------------------+   
  1.12e+08 |.+..    .+.+.  .+.+..+.+.+..                                    |   
           |    +.+.     +.              .+.. .+..+. .+.. .+..              |   
   1.1e+08 |-+                          +    +      +    +    +. .+.. .+..+.|   
  1.08e+08 |-+                                                  +    +      |   
  1.06e+08 |-+                                                              |   
  1.04e+08 |-+                                                              |   
           |                                                                |   
  1.02e+08 |-+                                                              |   
     1e+08 |-+                                                              |   
   9.8e+07 |-+    O                                                         |   
   9.6e+07 |-+  O    O O O    O  O   O       O             O                |   
           | O              O      O    O                O    O O           |   
   9.4e+07 |-+                            O    O  O O O                     |   
   9.2e+07 +----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                        nfsstat.Client.nfs.v3.write.percent                     
                                                                                
    14 +--------------------------------------------------------------------+   
       |                                                                    |   
       |                                                                    |   
  13.5 |-+                                                                  |   
       |                                                                    |   
       |                                                                    |   
       |                                                                    |   
    13 |-+                                                                  |   
       |                                                                    |   
       |                                                                    |   
  12.5 |-+                                                                  |   
       |                                                                    |   
       |                                                                    |   
       |                                                                    |   
    12 +--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                       nfsstat.Client.nfs.v3.commit.percent                     
                                                                                
    14 +--------------------------------------------------------------------+   
       |                                                                    |   
       |                                                                    |   
  13.5 |-+                                                                  |   
       |                                                                    |   
       |                                                                    |   
       |                                                                    |   
    13 |-+                                                                  |   
       |                                                                    |   
       |                                                                    |   
  12.5 |-+                                                                  |   
       |                                                                    |   
       |                                                                    |   
       |                                                                    |   
    12 +--------------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.5.0-rc3-00067-g52262ee567ad1" of type "text/plain" (202322 bytes)

View attachment "job-script" of type "text/plain" (8392 bytes)

View attachment "job.yaml" of type "text/plain" (5940 bytes)

View attachment "reproduce" of type "text/plain" (979 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ