lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite for Android: free password hash cracker in your pocket
[<prev] [next>] [day] [month] [year] [list]
Message-ID: <20191230004909.GG2760@shao2-debian>
Date:   Mon, 30 Dec 2019 08:49:10 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     "Rafael J. Wysocki" <rafael.j.wysocki@...el.com>
Cc:     LKML <linux-kernel@...r.kernel.org>,
        "Rafael J. Wysocki" <rafael.j.wysocki@...el.com>,
        linux-acpi@...r.kernel.org, devel@...ica.org,
        linux-pm@...r.kernel.org, lkp@...ts.01.org
Subject: [intel_idle]  bf5a9506c5:  fsmark.files_per_sec 12.6% improvement

Greeting,

FYI, we noticed a 12.6% improvement of fsmark.files_per_sec due to commit:


commit: bf5a9506c59abd8e06049ccd2b634e28d62e0b63 ("intel_idle: Use ACPI _CST on server systems")
https://git.kernel.org/cgit/linux/kernel/git/rafael/linux-pm.git intel_idle+acpi

in testcase: fsmark
on test machine: 72 threads Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz with 256G memory
with following parameters:

	iterations: 4
	disk: 1SSD
	nr_threads: 4
	fs: ext4
	fs2: nfsv4
	filesize: 8K
	test_size: 20G
	sync_method: fsyncBeforeClose
	nr_directories: 16d
	nr_files_per_directory: 256fpd
	cpufreq_governor: performance
	ucode: 0x43

test-description: The fsmark is a file system benchmark to test synchronous write workloads, for example, mail servers workload.
test-url: https://sourceforge.net/projects/fsmark/

In addition to that, the commit also has significant impact on the following tests:



Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
compiler/cpufreq_governor/disk/filesize/fs2/fs/iterations/kconfig/nr_directories/nr_files_per_directory/nr_threads/rootfs/sync_method/tbox_group/test_size/testcase/ucode:
  gcc-7/performance/1SSD/8K/nfsv4/ext4/4/x86_64-rhel-7.6/16d/256fpd/4/debian-x86_64-2019-11-14.cgz/fsyncBeforeClose/lkp-hsw-ep4/20G/fsmark/0x43

commit: 
  706ae86d1a ("intel_idle: Add module parameter to prevent ACPI _CST from being used")
  bf5a9506c5 ("intel_idle: Use ACPI _CST on server systems")

706ae86d1a0a737a bf5a9506c59abd8e06049ccd2b6 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
  45681661           -11.1%   40588454        fsmark.app_overhead
      3853           +12.6%       4340        fsmark.files_per_sec
    690.80           -11.1%     614.31        fsmark.time.elapsed_time
    690.80           -11.1%     614.31        fsmark.time.elapsed_time.max
     44.50           -10.1%      40.00        fsmark.time.percent_of_cpu_this_job_got
    280.61           -19.2%     226.63        fsmark.time.system_time
     30.02 ±  4%     -19.4%      24.20        fsmark.time.user_time
  10912602            +3.3%   11276074        fsmark.time.voluntary_context_switches
      0.27 ±  5%      -0.0        0.24 ±  2%  mpstat.cpu.all.soft%
     28.75 ±  2%     -14.8%      24.50 ±  2%  nfsstat.Client.nfs.v4.getattr
    287362 ±  4%      +9.6%     314949 ±  4%  numa-meminfo.node1.SUnreclaim
     23399           -39.8%      14080 ± 40%  numa-numastat.node1.other_node
      1591 ± 24%     -18.5%       1296 ± 29%  meminfo.Mlocked
     81767 ±  2%     +14.1%      93268        meminfo.max_used_kB
      9295 ±  8%     -14.8%       7918 ±  5%  slabinfo.dmaengine-unmap-16.active_objs
      9487 ±  8%     -14.8%       8081 ±  5%  slabinfo.dmaengine-unmap-16.num_objs
     15470           +12.1%      17337        iostat.nvme0n1.w/s
    207292           +13.8%     235962        iostat.nvme0n1.wkB/s
     28555 ±  2%     +14.0%      32552        iostat.nvme0n1.wrqm/s
    190858 ±  2%     +13.4%     216477        vmstat.io.bo
    245617 ±  2%     +13.4%     278642        vmstat.system.cs
    158327            +2.5%     162220        vmstat.system.in
   3086195 ±  5%     -12.7%    2693297 ±  9%  numa-vmstat.node0.nr_dirtied
   3075412 ±  5%     -12.8%    2683107 ±  9%  numa-vmstat.node0.nr_written
   7942592 ±  3%     -11.3%    7046146 ±  4%  numa-vmstat.node0.numa_hit
   7931373 ±  3%     -11.7%    7005377 ±  3%  numa-vmstat.node0.numa_local
     71846 ±  4%      +9.6%      78737 ±  4%  numa-vmstat.node1.nr_slab_unreclaimable
 4.088e+09 ± 21%    +903.1%  4.101e+10        cpuidle.C1E.time
  63122122 ± 27%    +106.8%  1.305e+08 ± 13%  cpuidle.C1E.usage
 2.371e+10 ± 53%    -100.0%       0.00        cpuidle.C3.time
  65576000 ± 49%    -100.0%       0.00        cpuidle.C3.usage
 1.913e+10 ± 66%    -100.0%       0.00        cpuidle.C6.time
  21868884 ± 61%    -100.0%       0.00        cpuidle.C6.usage
  40449960           +71.5%   69366001 ± 60%  cpuidle.POLL.time
   9131387           +28.8%   11758900 ± 22%  cpuidle.POLL.usage
    397.50 ± 24%     -18.6%     323.75 ± 29%  proc-vmstat.nr_mlock
      8303            -7.6%       7676        proc-vmstat.nr_shmem
    399.75 ±  3%     -12.4%     350.00 ±  2%  proc-vmstat.nr_writeback
  27176203            -1.1%   26866156        proc-vmstat.numa_hit
  27152741            -1.1%   26847345        proc-vmstat.numa_local
    302056 ±  9%     -12.2%     265309 ±  7%  proc-vmstat.numa_pte_updates
     16405            +4.9%      17212        proc-vmstat.pgactivate
  37413395            -0.8%   37103102        proc-vmstat.pgalloc_normal
   1988730 ±  2%     -11.9%    1751515        proc-vmstat.pgfault
  30074093            -0.9%   29800436        proc-vmstat.pgfree
    144.75            -5.4%     137.00        turbostat.Avg_MHz
  63121438 ± 27%    +106.7%  1.305e+08 ± 13%  turbostat.C1E
      8.00 ± 18%     +84.0       92.04        turbostat.C1E%
  65575105 ± 49%    -100.0%       0.00        turbostat.C3
     47.17 ± 54%     -47.2        0.00        turbostat.C3%
  21858411 ± 61%    -100.0%       0.00        turbostat.C6
     37.06 ± 65%     -37.1        0.00        turbostat.C6%
     56.91 ± 12%     +66.7%      94.90        turbostat.CPU%c1
     22.48 ± 23%    -100.0%       0.00        turbostat.CPU%c3
     15.33 ± 78%    -100.0%       0.00        turbostat.CPU%c6
     51.75 ±  5%     +13.5%      58.75 ±  2%  turbostat.CoreTmp
  1.12e+08           -10.2%  1.005e+08        turbostat.IRQ
      2.04 ± 31%    -100.0%       0.00        turbostat.Pkg%pc2
      0.04 ± 45%    -100.0%       0.00        turbostat.Pkg%pc3
     56.00 ±  4%     +12.9%      63.25        turbostat.PkgTmp
    145.21 ±  5%     +18.3%     171.74        turbostat.PkgWatt
     15.58            +2.2%      15.92        turbostat.RAMWatt
     10322 ±  2%     -18.1%       8454 ±  5%  sched_debug.cfs_rq:/.exec_clock.avg
     15292 ±  4%     -15.6%      12901 ±  5%  sched_debug.cfs_rq:/.exec_clock.max
      5837 ±  7%     -28.9%       4152 ± 10%  sched_debug.cfs_rq:/.exec_clock.min
    312.34 ± 22%     -27.8%     225.44 ±  8%  sched_debug.cfs_rq:/.load_avg.avg
     26337 ±  5%     -12.0%      23188 ±  7%  sched_debug.cfs_rq:/.min_vruntime.avg
     43710 ±  8%     -18.5%      35630 ±  7%  sched_debug.cfs_rq:/.min_vruntime.max
     22598 ± 18%     -20.3%      18011 ± 10%  sched_debug.cfs_rq:/.spread0.max
    306.03 ± 10%     +15.7%     354.13 ±  5%  sched_debug.cfs_rq:/.util_est_enqueued.max
    389982 ±  3%     -15.6%     329303 ±  4%  sched_debug.cpu.clock.avg
    389985 ±  3%     -15.6%     329306 ±  4%  sched_debug.cpu.clock.max
    389979 ±  3%     -15.6%     329300 ±  4%  sched_debug.cpu.clock.min
    389982 ±  3%     -15.6%     329303 ±  4%  sched_debug.cpu.clock_task.avg
    389985 ±  3%     -15.6%     329306 ±  4%  sched_debug.cpu.clock_task.max
    389979 ±  3%     -15.6%     329300 ±  4%  sched_debug.cpu.clock_task.min
     11587 ±  3%     -14.5%       9910 ±  3%  sched_debug.cpu.curr->pid.max
      1554 ±  4%      -8.2%       1426 ±  2%  sched_debug.cpu.curr->pid.stddev
      0.00 ± 13%     +25.0%       0.00 ±  6%  sched_debug.cpu.next_balance.stddev
      0.06 ±  9%     +14.7%       0.07 ±  5%  sched_debug.cpu.nr_running.avg
    622358 ±  6%     -18.8%     505182 ± 11%  sched_debug.cpu.nr_switches.min
    616408 ±  6%     -18.8%     500660 ± 12%  sched_debug.cpu.sched_count.min
    297865 ±  7%     -19.4%     240061 ± 11%  sched_debug.cpu.sched_goidle.min
    289069 ±  3%     -18.1%     236863 ± 13%  sched_debug.cpu.ttwu_count.min
     61893 ±  3%     -11.7%      54636 ±  3%  sched_debug.cpu.ttwu_local.avg
     12424 ±  6%     -28.2%       8924 ± 20%  sched_debug.cpu.ttwu_local.min
     49190 ±  3%      -9.8%      44376 ±  3%  sched_debug.cpu.ttwu_local.stddev
    389979 ±  3%     -15.6%     329301 ±  4%  sched_debug.cpu_clk
    386296 ±  3%     -15.7%     325620 ±  4%  sched_debug.ktime
    390471 ±  3%     -15.5%     329772 ±  4%  sched_debug.sched_clk
     53.34           -20.0%      42.67        perf-stat.i.MPKI
 8.428e+08           +11.1%  9.364e+08        perf-stat.i.branch-instructions
      2.66 ± 17%      -0.7        1.99        perf-stat.i.branch-miss-rate%
      3.37            +1.4        4.80 ±  2%  perf-stat.i.cache-miss-rate%
   6624263           +10.7%    7334586        perf-stat.i.cache-misses
 2.055e+08 ±  2%      -9.3%  1.863e+08        perf-stat.i.cache-references
    246753 ±  2%     +13.5%     280186        perf-stat.i.context-switches
      2.88 ±  5%     -29.2%       2.04        perf-stat.i.cpi
 1.072e+10 ±  2%     -17.6%  8.831e+09        perf-stat.i.cpu-cycles
      2714 ± 33%     -48.7%       1392 ±  4%  perf-stat.i.cycles-between-cache-misses
      0.49 ± 11%      -0.3        0.21 ±  6%  perf-stat.i.dTLB-load-miss-rate%
   5282248 ± 12%     -51.1%    2580959 ±  6%  perf-stat.i.dTLB-load-misses
  1.11e+09            +9.8%  1.219e+09        perf-stat.i.dTLB-loads
      0.13 ±  7%      -0.1        0.08 ±  3%  perf-stat.i.dTLB-store-miss-rate%
   1061338 ± 11%     -49.9%     532150 ±  3%  perf-stat.i.dTLB-store-misses
  8.21e+08 ±  4%     -14.9%  6.988e+08        perf-stat.i.dTLB-stores
     19.69 ± 15%     -16.6        3.10 ± 27%  perf-stat.i.iTLB-load-miss-rate%
    995023 ± 19%     -82.5%     173956 ± 28%  perf-stat.i.iTLB-load-misses
   4426091 ±  4%     +22.8%    5435517        perf-stat.i.iTLB-loads
 3.916e+09           +11.1%   4.35e+09        perf-stat.i.instructions
      4050 ± 15%    +592.3%      28041 ± 24%  perf-stat.i.instructions-per-iTLB-miss
      0.36 ±  2%     +35.8%       0.49        perf-stat.i.ipc
   4039828           +10.6%    4467418        perf-stat.i.node-load-misses
    424975           +10.7%     470461        perf-stat.i.node-loads
   1331640           +10.9%    1477425        perf-stat.i.node-store-misses
    795700            +7.8%     857866        perf-stat.i.node-stores
     52.46           -18.4%      42.82        perf-stat.overall.MPKI
      2.50 ± 12%      -0.5        2.01        perf-stat.overall.branch-miss-rate%
      3.23 ±  2%      +0.7        3.94        perf-stat.overall.cache-miss-rate%
      2.74 ±  2%     -25.8%       2.03        perf-stat.overall.cpi
      1618 ±  2%     -25.6%       1203        perf-stat.overall.cycles-between-cache-misses
      0.47 ± 12%      -0.3        0.21 ±  6%  perf-stat.overall.dTLB-load-miss-rate%
      0.13 ±  7%      -0.1        0.08 ±  3%  perf-stat.overall.dTLB-store-miss-rate%
     18.33 ± 17%     -15.2        3.10 ± 28%  perf-stat.overall.iTLB-load-miss-rate%
      4054 ± 15%    +563.2%      26888 ± 24%  perf-stat.overall.instructions-per-iTLB-miss
      0.37 ±  2%     +34.8%       0.49        perf-stat.overall.ipc
     62.59            +0.7       63.26        perf-stat.overall.node-store-miss-rate%
 8.415e+08           +11.1%  9.348e+08        perf-stat.ps.branch-instructions
   6615317           +10.7%    7323201        perf-stat.ps.cache-misses
 2.051e+08 ±  2%      -9.4%   1.86e+08        perf-stat.ps.cache-references
    246355 ±  2%     +13.5%     279653        perf-stat.ps.context-switches
  1.07e+10 ±  2%     -17.6%  8.816e+09        perf-stat.ps.cpu-cycles
   5274582 ± 12%     -51.2%    2576527 ±  6%  perf-stat.ps.dTLB-load-misses
 1.108e+09            +9.7%  1.216e+09        perf-stat.ps.dTLB-loads
   1059913 ± 11%     -49.9%     531355 ±  3%  perf-stat.ps.dTLB-store-misses
 8.198e+08 ±  4%     -14.9%  6.976e+08        perf-stat.ps.dTLB-stores
    993569 ± 19%     -82.5%     173648 ± 28%  perf-stat.ps.iTLB-load-misses
   4419022 ±  4%     +22.8%    5425497        perf-stat.ps.iTLB-loads
  3.91e+09           +11.0%  4.342e+09        perf-stat.ps.instructions
   4033979           +10.6%    4459868        perf-stat.ps.node-load-misses
    425006           +10.7%     470577 ±  2%  perf-stat.ps.node-loads
   1329648           +10.9%    1474849        perf-stat.ps.node-store-misses
    794588            +7.8%     856495        perf-stat.ps.node-stores
  2.76e+12            -2.8%  2.683e+12        perf-stat.total.instructions
     89677 ±  8%     +11.6%     100064 ±  9%  softirqs.CPU0.NET_RX
    353563           -11.8%     311687        softirqs.CPU0.RCU
     99894 ±  3%     -15.5%      84375        softirqs.CPU0.SCHED
    374430           -11.9%     329898 ±  2%  softirqs.CPU1.RCU
     95213 ±  3%     -13.6%      82224 ±  2%  softirqs.CPU1.SCHED
    368642 ±  2%     -12.4%     322833 ±  2%  softirqs.CPU10.RCU
     92664 ±  4%     -16.8%      77065 ±  4%  softirqs.CPU10.SCHED
    364651           -10.6%     325857        softirqs.CPU11.RCU
     95412 ±  3%     -18.9%      77426 ±  3%  softirqs.CPU11.SCHED
    367813           -11.6%     324982        softirqs.CPU12.RCU
     94069 ±  2%     -16.9%      78140        softirqs.CPU12.SCHED
    368558           -13.0%     320597 ±  2%  softirqs.CPU13.RCU
     93391 ±  2%     -15.6%      78805        softirqs.CPU13.SCHED
    366121           -11.6%     323476        softirqs.CPU14.RCU
     95396 ±  3%     -20.8%      75553 ±  3%  softirqs.CPU14.SCHED
     93373 ±  5%     -16.8%      77692        softirqs.CPU15.SCHED
    365265           -11.1%     324857        softirqs.CPU16.RCU
     93707 ±  4%     -17.8%      76980 ±  3%  softirqs.CPU16.SCHED
    366272           -10.4%     328061        softirqs.CPU17.RCU
     93696 ±  4%     -15.6%      79045 ±  2%  softirqs.CPU17.SCHED
    366621           -12.0%     322746        softirqs.CPU18.RCU
     95783 ±  2%     -17.1%      79434        softirqs.CPU18.SCHED
    243492 ± 13%     -19.7%     195468 ±  3%  softirqs.CPU18.TIMER
    367659           -10.9%     327720 ±  2%  softirqs.CPU19.RCU
     92035 ±  2%     -14.8%      78422 ±  2%  softirqs.CPU19.SCHED
    234011 ± 16%     -17.9%     192099 ±  4%  softirqs.CPU19.TIMER
    367630           -11.4%     325762        softirqs.CPU2.RCU
     94477 ±  2%     -18.1%      77362        softirqs.CPU2.SCHED
    369559 ±  2%     -11.1%     328660 ±  2%  softirqs.CPU20.RCU
     92781 ±  3%     -16.5%      77464        softirqs.CPU20.SCHED
    236181 ± 16%     -20.0%     189055 ±  2%  softirqs.CPU20.TIMER
    369195            -9.8%     332891        softirqs.CPU21.RCU
    233505 ± 17%     -19.2%     188593 ±  2%  softirqs.CPU21.TIMER
    365867           -10.3%     328169        softirqs.CPU22.RCU
     92425 ±  2%     -15.2%      78360 ±  2%  softirqs.CPU22.SCHED
    234254 ± 15%     -20.1%     187076 ±  3%  softirqs.CPU22.TIMER
    366974            -9.7%     331484        softirqs.CPU23.RCU
     91883 ±  2%     -14.3%      78736 ±  2%  softirqs.CPU23.SCHED
    238610 ± 14%     -20.1%     190606 ±  2%  softirqs.CPU23.TIMER
    368741           -11.0%     328085        softirqs.CPU24.RCU
     93540 ±  3%     -15.8%      78779 ±  2%  softirqs.CPU24.SCHED
    367436           -11.0%     327111        softirqs.CPU25.RCU
     92572 ±  2%     -16.1%      77652        softirqs.CPU25.SCHED
    232583 ± 16%     -18.3%     189916 ±  2%  softirqs.CPU25.TIMER
    368839 ±  2%     -12.7%     322116        softirqs.CPU26.RCU
     92484 ±  5%     -15.9%      77824        softirqs.CPU26.SCHED
    233106 ± 16%     -18.3%     190342 ±  2%  softirqs.CPU26.TIMER
    366326           -10.4%     328180        softirqs.CPU27.RCU
     91542 ±  4%     -17.3%      75680 ±  3%  softirqs.CPU27.SCHED
    232319 ± 16%     -18.5%     189310 ±  2%  softirqs.CPU27.TIMER
    173278 ± 12%     +22.3%     211841 ±  8%  softirqs.CPU28.NET_RX
     93293 ±  2%     -17.7%      76792 ±  2%  softirqs.CPU28.SCHED
    231483 ± 16%     -17.9%     190047 ±  3%  softirqs.CPU28.TIMER
    165270 ± 11%     +33.3%     220379 ± 12%  softirqs.CPU29.NET_RX
    361393            -8.9%     329361        softirqs.CPU29.RCU
     93247 ±  2%     -16.4%      77934        softirqs.CPU29.SCHED
    236961 ± 15%     -19.0%     191876        softirqs.CPU29.TIMER
    364649           -10.8%     325288        softirqs.CPU3.RCU
     93159 ±  3%     -14.8%      79418 ±  3%  softirqs.CPU3.SCHED
    364012            -9.7%     328562        softirqs.CPU30.RCU
     92147 ±  3%     -15.4%      77913 ±  2%  softirqs.CPU30.SCHED
    235135 ± 15%     -18.4%     191945 ±  2%  softirqs.CPU30.TIMER
    181244 ±  4%     +27.9%     231821 ±  9%  softirqs.CPU31.NET_RX
     92779 ±  3%     -15.0%      78852        softirqs.CPU31.SCHED
    237701 ± 14%     -19.0%     192576 ±  2%  softirqs.CPU31.TIMER
     90576 ±  5%     -13.6%      78282 ±  2%  softirqs.CPU32.SCHED
    231799 ± 16%     -17.6%     190890        softirqs.CPU32.TIMER
    365996 ±  2%      -9.6%     330918        softirqs.CPU33.RCU
     90828 ±  3%     -13.9%      78245        softirqs.CPU33.SCHED
    236323 ± 15%     -19.6%     189979 ±  2%  softirqs.CPU33.TIMER
    366467           -10.7%     327239        softirqs.CPU34.RCU
     91983 ±  4%     -12.7%      80311 ±  5%  softirqs.CPU34.SCHED
    363376           -10.4%     325603        softirqs.CPU35.RCU
    233975 ± 16%     -19.6%     188231 ±  2%  softirqs.CPU35.TIMER
    357557           -11.8%     315281        softirqs.CPU36.RCU
     93135 ±  4%     -22.3%      72355        softirqs.CPU36.SCHED
    363068           -11.5%     321387        softirqs.CPU37.RCU
     92109 ±  2%     -14.7%      78572 ±  3%  softirqs.CPU37.SCHED
    363307           -12.4%     318401        softirqs.CPU38.RCU
     91515 ±  2%     -16.1%      76750 ±  2%  softirqs.CPU38.SCHED
    359287 ±  2%     -12.4%     314871        softirqs.CPU39.RCU
     92667 ±  4%     -17.3%      76620        softirqs.CPU39.SCHED
    363670           -11.2%     322838 ±  2%  softirqs.CPU4.RCU
     92228 ±  2%     -15.3%      78096        softirqs.CPU4.SCHED
    360650           -12.6%     315039        softirqs.CPU40.RCU
     91816 ±  2%     -17.2%      75987 ±  2%  softirqs.CPU40.SCHED
    362998           -11.3%     321804        softirqs.CPU41.RCU
     92346 ±  6%     -17.0%      76617        softirqs.CPU41.SCHED
    359105            -9.3%     325535 ±  2%  softirqs.CPU42.RCU
     92512 ±  3%     -15.9%      77803        softirqs.CPU42.SCHED
    358281           -11.6%     316663        softirqs.CPU43.RCU
     93441 ±  2%     -16.9%      77608 ±  2%  softirqs.CPU43.SCHED
    358618           -10.8%     319904        softirqs.CPU44.RCU
     91356 ±  2%     -14.7%      77921        softirqs.CPU44.SCHED
    361020           -12.1%     317402        softirqs.CPU45.RCU
     92331 ±  3%     -16.9%      76770        softirqs.CPU45.SCHED
    357817           -11.3%     317386        softirqs.CPU46.RCU
    355837           -11.2%     315881        softirqs.CPU47.RCU
     92875           -17.1%      76964        softirqs.CPU47.SCHED
    356814           -11.2%     316991        softirqs.CPU48.RCU
     92868 ±  3%     -15.7%      78250        softirqs.CPU48.SCHED
    359131           -12.8%     313113        softirqs.CPU49.RCU
     91355 ±  4%     -16.5%      76241 ±  3%  softirqs.CPU49.SCHED
    367629           -11.9%     323772        softirqs.CPU5.RCU
     91727           -15.5%      77464 ±  5%  softirqs.CPU5.SCHED
    235517 ± 12%     -17.1%     195238 ±  6%  softirqs.CPU5.TIMER
    357412           -10.6%     319665        softirqs.CPU50.RCU
     91137 ±  3%     -17.3%      75325 ±  4%  softirqs.CPU50.SCHED
    361302           -12.5%     316254        softirqs.CPU51.RCU
     92884 ±  4%     -18.3%      75879 ±  2%  softirqs.CPU51.SCHED
    364328           -13.2%     316094        softirqs.CPU52.RCU
     93505 ±  4%     -18.3%      76410 ±  2%  softirqs.CPU52.SCHED
    355176           -11.7%     313652        softirqs.CPU53.RCU
     91944 ±  4%     -19.0%      74484 ± 10%  softirqs.CPU53.SCHED
    361924            -9.6%     327038        softirqs.CPU54.RCU
     93005 ±  5%     -13.8%      80156        softirqs.CPU54.SCHED
    362788            -9.7%     327549        softirqs.CPU55.RCU
     92072 ±  3%     -14.6%      78663        softirqs.CPU55.SCHED
    231778 ± 16%     -18.8%     188142 ±  3%  softirqs.CPU55.TIMER
    363707            -9.6%     328893        softirqs.CPU56.RCU
     91605 ±  4%     -15.0%      77850        softirqs.CPU56.SCHED
    231946 ± 16%     -17.8%     190677 ±  4%  softirqs.CPU56.TIMER
     90426 ±  4%     -14.7%      77138        softirqs.CPU57.SCHED
    231293 ± 16%     -18.4%     188705 ±  2%  softirqs.CPU57.TIMER
    359002            -9.4%     325359        softirqs.CPU58.RCU
     90609 ±  3%     -14.8%      77211 ±  2%  softirqs.CPU58.SCHED
     90963 ±  3%     -14.7%      77561        softirqs.CPU59.SCHED
    232667 ± 16%     -18.8%     189020 ±  2%  softirqs.CPU59.TIMER
    367066           -11.7%     324066        softirqs.CPU6.RCU
     91908 ±  2%     -11.6%      81286 ±  4%  softirqs.CPU6.SCHED
     92504           -14.7%      78898        softirqs.CPU60.SCHED
    138292 ± 13%     +33.3%     184363 ±  8%  softirqs.CPU61.NET_RX
    357797           -10.0%     322183        softirqs.CPU61.RCU
     90633 ±  4%     -15.5%      76540        softirqs.CPU61.SCHED
    359430           -10.9%     320236        softirqs.CPU62.RCU
     90003           -15.7%      75839 ±  3%  softirqs.CPU62.SCHED
    230456 ± 17%     -18.0%     188998 ±  3%  softirqs.CPU62.TIMER
    360280           -11.8%     317866        softirqs.CPU63.RCU
     89729 ±  5%     -16.0%      75394 ±  4%  softirqs.CPU63.SCHED
    135420 ±  7%     +37.8%     186614 ± 11%  softirqs.CPU64.NET_RX
    358914           -10.4%     321608        softirqs.CPU64.RCU
     92825 ±  2%     -18.9%      75317 ±  2%  softirqs.CPU64.SCHED
    123845 ±  8%     +32.5%     164047 ± 10%  softirqs.CPU65.NET_RX
    354012           -10.1%     318199        softirqs.CPU65.RCU
     91672 ±  2%     -16.3%      76696 ±  2%  softirqs.CPU65.SCHED
    360304           -11.5%     318840        softirqs.CPU66.RCU
     89248 ±  4%     -12.5%      78089        softirqs.CPU66.SCHED
    233035 ± 16%     -18.1%     190915 ±  3%  softirqs.CPU66.TIMER
    358272           -10.4%     320983        softirqs.CPU67.RCU
     90877           -15.6%      76693        softirqs.CPU67.SCHED
    359796           -11.8%     317394        softirqs.CPU68.RCU
     91998 ±  3%     -21.4%      72341 ±  6%  softirqs.CPU68.SCHED
    357151           -11.2%     317284        softirqs.CPU69.RCU
     91929 ±  3%     -17.3%      76011 ±  2%  softirqs.CPU69.SCHED
    367641           -10.8%     327775        softirqs.CPU7.RCU
     93048 ±  4%     -15.2%      78884        softirqs.CPU7.SCHED
    261268 ± 21%     -24.8%     196492 ±  6%  softirqs.CPU7.TIMER
    356170           -10.2%     319947 ±  2%  softirqs.CPU70.RCU
     90769 ±  4%     -13.8%      78201        softirqs.CPU70.SCHED
    233512 ± 16%     -20.1%     186689 ±  4%  softirqs.CPU70.TIMER
    355056           -11.4%     314405        softirqs.CPU71.RCU
     86620 ±  7%     -13.9%      74604        softirqs.CPU71.SCHED
    230153 ± 17%     -18.0%     188711 ±  3%  softirqs.CPU71.TIMER
    370893           -11.3%     329103        softirqs.CPU8.RCU
     91414 ±  4%     -18.1%      74887 ±  4%  softirqs.CPU8.SCHED
    368617           -11.7%     325356        softirqs.CPU9.RCU
     91965 ±  5%     -14.8%      78358        softirqs.CPU9.SCHED
  26123416           -10.9%   23277653        softirqs.RCU
   6636684 ±  3%     -16.0%    5573973        softirqs.SCHED
    346.75 ±  2%     -12.1%     304.75        interrupts.100:IR-PCI-MSI.1572923-edge.eth0-TxRx-59
    346.25 ±  2%     -12.5%     303.00        interrupts.101:IR-PCI-MSI.1572924-edge.eth0-TxRx-60
    350.75 ±  3%     -13.5%     303.25        interrupts.102:IR-PCI-MSI.1572925-edge.eth0-TxRx-61
    350.50 ±  2%     -13.5%     303.25        interrupts.103:IR-PCI-MSI.1572926-edge.eth0-TxRx-62
    189468 ± 58%     +42.3%     269641 ±  9%  interrupts.238:IR-PCI-MSI.68681742-edge.nvme0q14
    397.00 ± 10%     -19.6%     319.00 ±  2%  interrupts.49:IR-PCI-MSI.1572874-edge.eth0-TxRx-10
    552.00 ± 25%     -32.1%     375.00 ± 19%  interrupts.50:IR-PCI-MSI.1572875-edge.eth0-TxRx-11
    347.75           -12.1%     305.50        interrupts.55:IR-PCI-MSI.1572880-edge.eth0-TxRx-16
    354.25 ±  3%     -13.7%     305.75        interrupts.56:IR-PCI-MSI.1572881-edge.eth0-TxRx-17
    346.50 ±  2%     -12.0%     305.00        interrupts.57:IR-PCI-MSI.1572882-edge.eth0-TxRx-18
    350.50 ±  2%     -13.3%     304.00        interrupts.58:IR-PCI-MSI.1572883-edge.eth0-TxRx-19
    350.75 ±  2%     -13.5%     303.25        interrupts.59:IR-PCI-MSI.1572884-edge.eth0-TxRx-20
    348.00           -12.0%     306.25        interrupts.60:IR-PCI-MSI.1572885-edge.eth0-TxRx-21
    355.75 ±  2%     -15.0%     302.25        interrupts.61:IR-PCI-MSI.1572886-edge.eth0-TxRx-22
    384.50 ± 18%     -20.7%     305.00        interrupts.62:IR-PCI-MSI.1572887-edge.eth0-TxRx-23
    346.75 ±  2%     -11.9%     305.50        interrupts.63:IR-PCI-MSI.1572888-edge.eth0-TxRx-24
    347.75 ±  2%     -12.7%     303.50        interrupts.64:IR-PCI-MSI.1572889-edge.eth0-TxRx-25
    349.75 ±  3%     -12.2%     307.25 ±  2%  interrupts.67:IR-PCI-MSI.1572892-edge.eth0-TxRx-28
    353.50 ±  3%     -11.5%     312.75 ±  5%  interrupts.68:IR-PCI-MSI.1572893-edge.eth0-TxRx-29
    382.00 ± 15%     -20.0%     305.75        interrupts.69:IR-PCI-MSI.1572894-edge.eth0-TxRx-30
    353.25 ±  2%     -13.6%     305.25        interrupts.71:IR-PCI-MSI.1572896-edge.eth0-TxRx-32
    350.00 ±  2%     -12.6%     306.00        interrupts.74:IR-PCI-MSI.1572897-edge.eth0-TxRx-33
    360.25 ±  5%     -14.9%     306.50        interrupts.75:IR-PCI-MSI.1572898-edge.eth0-TxRx-34
    347.50 ±  2%     -12.2%     305.25        interrupts.76:IR-PCI-MSI.1572899-edge.eth0-TxRx-35
    381.25 ± 17%     -20.2%     304.25        interrupts.78:IR-PCI-MSI.1572901-edge.eth0-TxRx-37
    361.50 ±  9%     -16.3%     302.50        interrupts.79:IR-PCI-MSI.1572902-edge.eth0-TxRx-38
    345.50 ±  2%     -12.2%     303.25        interrupts.80:IR-PCI-MSI.1572903-edge.eth0-TxRx-39
    347.50 ±  3%     -12.7%     303.50        interrupts.81:IR-PCI-MSI.1572904-edge.eth0-TxRx-40
    346.50 ±  2%     -11.4%     307.00 ±  2%  interrupts.82:IR-PCI-MSI.1572905-edge.eth0-TxRx-41
    345.75 ±  2%     -12.1%     303.75        interrupts.83:IR-PCI-MSI.1572906-edge.eth0-TxRx-42
    347.50 ±  3%     -12.9%     302.50        interrupts.84:IR-PCI-MSI.1572907-edge.eth0-TxRx-43
    350.00 ±  4%     -12.1%     307.75 ±  2%  interrupts.85:IR-PCI-MSI.1572908-edge.eth0-TxRx-44
    345.75 ±  2%     -10.8%     308.50 ±  2%  interrupts.86:IR-PCI-MSI.1572909-edge.eth0-TxRx-45
    345.25 ±  2%     -11.9%     304.00        interrupts.87:IR-PCI-MSI.1572910-edge.eth0-TxRx-46
    349.25 ±  3%     -12.7%     304.75        interrupts.88:IR-PCI-MSI.1572911-edge.eth0-TxRx-47
    345.25 ±  2%     -11.9%     304.00        interrupts.89:IR-PCI-MSI.1572912-edge.eth0-TxRx-48
    345.25 ±  2%     -12.0%     303.75        interrupts.90:IR-PCI-MSI.1572913-edge.eth0-TxRx-49
    348.75 ±  3%     -12.7%     304.50        interrupts.91:IR-PCI-MSI.1572914-edge.eth0-TxRx-50
    345.25 ±  2%     -12.5%     302.25        interrupts.92:IR-PCI-MSI.1572915-edge.eth0-TxRx-51
    346.00 ±  2%     -12.4%     303.00        interrupts.93:IR-PCI-MSI.1572916-edge.eth0-TxRx-52
    345.25 ±  2%     -11.5%     305.50        interrupts.94:IR-PCI-MSI.1572917-edge.eth0-TxRx-53
    348.50 ±  3%     -13.1%     303.00        interrupts.95:IR-PCI-MSI.1572918-edge.eth0-TxRx-54
    346.25 ±  2%     -12.4%     303.25        interrupts.96:IR-PCI-MSI.1572919-edge.eth0-TxRx-55
    345.25 ±  2%     -12.2%     303.25        interrupts.97:IR-PCI-MSI.1572920-edge.eth0-TxRx-56
    345.25 ±  2%     -12.0%     303.75        interrupts.98:IR-PCI-MSI.1572921-edge.eth0-TxRx-57
    346.00 ±  2%     -12.5%     302.75        interrupts.99:IR-PCI-MSI.1572922-edge.eth0-TxRx-58
      1575 ±  2%     -34.3%       1036 ± 57%  interrupts.9:IR-IO-APIC.9-fasteoi.acpi
   1394297           -11.3%    1236680        interrupts.CPU0.LOC:Local_timer_interrupts
      1575 ±  2%     -34.3%       1036 ± 57%  interrupts.CPU1.9:IR-IO-APIC.9-fasteoi.acpi
   1394529           -11.4%    1235991        interrupts.CPU1.LOC:Local_timer_interrupts
    397.00 ± 10%     -19.6%     319.00 ±  2%  interrupts.CPU10.49:IR-PCI-MSI.1572874-edge.eth0-TxRx-10
   1394211           -11.4%    1235851        interrupts.CPU10.LOC:Local_timer_interrupts
    552.00 ± 25%     -32.1%     375.00 ± 19%  interrupts.CPU11.50:IR-PCI-MSI.1572875-edge.eth0-TxRx-11
   1394049           -11.3%    1236106        interrupts.CPU11.LOC:Local_timer_interrupts
   1393832           -11.3%    1236228        interrupts.CPU12.LOC:Local_timer_interrupts
   1394077           -11.3%    1236410        interrupts.CPU13.LOC:Local_timer_interrupts
      1.75 ±142%   +2642.9%      48.00 ±117%  interrupts.CPU13.TLB:TLB_shootdowns
   1394271           -11.3%    1236186        interrupts.CPU14.LOC:Local_timer_interrupts
   1394009           -11.3%    1236288        interrupts.CPU15.LOC:Local_timer_interrupts
    347.75           -12.1%     305.50        interrupts.CPU16.55:IR-PCI-MSI.1572880-edge.eth0-TxRx-16
   1393748           -11.3%    1236036        interrupts.CPU16.LOC:Local_timer_interrupts
    354.25 ±  3%     -13.7%     305.75        interrupts.CPU17.56:IR-PCI-MSI.1572881-edge.eth0-TxRx-17
   1394347           -11.4%    1235697        interrupts.CPU17.LOC:Local_timer_interrupts
   1391577           -11.2%    1236414        interrupts.CPU18.LOC:Local_timer_interrupts
    381.25 ± 17%     -20.2%     304.25        interrupts.CPU19.78:IR-PCI-MSI.1572901-edge.eth0-TxRx-37
   1391791           -11.2%    1236205        interrupts.CPU19.LOC:Local_timer_interrupts
      1012 ± 26%     -55.8%     447.75 ± 31%  interrupts.CPU19.NMI:Non-maskable_interrupts
      1012 ± 26%     -55.8%     447.75 ± 31%  interrupts.CPU19.PMI:Performance_monitoring_interrupts
   1394823           -11.4%    1236359        interrupts.CPU2.LOC:Local_timer_interrupts
    361.50 ±  9%     -16.3%     302.50        interrupts.CPU20.79:IR-PCI-MSI.1572902-edge.eth0-TxRx-38
   1392093           -11.1%    1236903        interrupts.CPU20.LOC:Local_timer_interrupts
    345.50 ±  2%     -12.2%     303.25        interrupts.CPU21.80:IR-PCI-MSI.1572903-edge.eth0-TxRx-39
   1391725           -11.1%    1236718        interrupts.CPU21.LOC:Local_timer_interrupts
    347.50 ±  3%     -12.7%     303.50        interrupts.CPU22.81:IR-PCI-MSI.1572904-edge.eth0-TxRx-40
   1391695           -11.1%    1236856        interrupts.CPU22.LOC:Local_timer_interrupts
      1087 ± 20%     -65.5%     375.25 ± 30%  interrupts.CPU22.NMI:Non-maskable_interrupts
      1087 ± 20%     -65.5%     375.25 ± 30%  interrupts.CPU22.PMI:Performance_monitoring_interrupts
    346.50 ±  2%     -11.4%     307.00 ±  2%  interrupts.CPU23.82:IR-PCI-MSI.1572905-edge.eth0-TxRx-41
   1391389           -11.2%    1236235        interrupts.CPU23.LOC:Local_timer_interrupts
    814.00 ± 18%     -39.7%     490.75 ± 14%  interrupts.CPU23.NMI:Non-maskable_interrupts
    814.00 ± 18%     -39.7%     490.75 ± 14%  interrupts.CPU23.PMI:Performance_monitoring_interrupts
    345.75 ±  2%     -12.1%     303.75        interrupts.CPU24.83:IR-PCI-MSI.1572906-edge.eth0-TxRx-42
   1391633           -11.2%    1236428        interrupts.CPU24.LOC:Local_timer_interrupts
    347.50 ±  3%     -12.9%     302.50        interrupts.CPU25.84:IR-PCI-MSI.1572907-edge.eth0-TxRx-43
   1391734           -11.1%    1237187        interrupts.CPU25.LOC:Local_timer_interrupts
    350.00 ±  4%     -12.1%     307.75 ±  2%  interrupts.CPU26.85:IR-PCI-MSI.1572908-edge.eth0-TxRx-44
   1391682           -11.1%    1236720        interrupts.CPU26.LOC:Local_timer_interrupts
    345.75 ±  2%     -10.8%     308.50 ±  2%  interrupts.CPU27.86:IR-PCI-MSI.1572909-edge.eth0-TxRx-45
   1391590           -11.1%    1237122        interrupts.CPU27.LOC:Local_timer_interrupts
      1033 ± 18%     -58.6%     428.00 ± 54%  interrupts.CPU27.NMI:Non-maskable_interrupts
      1033 ± 18%     -58.6%     428.00 ± 54%  interrupts.CPU27.PMI:Performance_monitoring_interrupts
    345.25 ±  2%     -11.9%     304.00        interrupts.CPU28.87:IR-PCI-MSI.1572910-edge.eth0-TxRx-46
   1391512           -11.1%    1237095        interrupts.CPU28.LOC:Local_timer_interrupts
    349.25 ±  3%     -12.7%     304.75        interrupts.CPU29.88:IR-PCI-MSI.1572911-edge.eth0-TxRx-47
   1391540           -11.1%    1237046        interrupts.CPU29.LOC:Local_timer_interrupts
   1394691           -11.4%    1235965        interrupts.CPU3.LOC:Local_timer_interrupts
    345.25 ±  2%     -11.9%     304.00        interrupts.CPU30.89:IR-PCI-MSI.1572912-edge.eth0-TxRx-48
   1391319           -11.1%    1236639        interrupts.CPU30.LOC:Local_timer_interrupts
    345.25 ±  2%     -12.0%     303.75        interrupts.CPU31.90:IR-PCI-MSI.1572913-edge.eth0-TxRx-49
   1391299           -11.1%    1236878        interrupts.CPU31.LOC:Local_timer_interrupts
      8480 ± 13%     +37.3%      11645 ± 17%  interrupts.CPU31.RES:Rescheduling_interrupts
    348.75 ±  3%     -12.7%     304.50        interrupts.CPU32.91:IR-PCI-MSI.1572914-edge.eth0-TxRx-50
   1391630           -11.1%    1236541        interrupts.CPU32.LOC:Local_timer_interrupts
    345.25 ±  2%     -12.5%     302.25        interrupts.CPU33.92:IR-PCI-MSI.1572915-edge.eth0-TxRx-51
   1391302           -11.1%    1236797        interrupts.CPU33.LOC:Local_timer_interrupts
    346.00 ±  2%     -12.4%     303.00        interrupts.CPU34.93:IR-PCI-MSI.1572916-edge.eth0-TxRx-52
   1391611           -11.1%    1236715        interrupts.CPU34.LOC:Local_timer_interrupts
    345.25 ±  2%     -11.5%     305.50        interrupts.CPU35.94:IR-PCI-MSI.1572917-edge.eth0-TxRx-53
   1391648           -11.1%    1236760        interrupts.CPU35.LOC:Local_timer_interrupts
    346.50 ±  2%     -12.0%     305.00        interrupts.CPU36.57:IR-PCI-MSI.1572882-edge.eth0-TxRx-18
   1394548           -11.4%    1235992        interrupts.CPU36.LOC:Local_timer_interrupts
    904.75 ± 25%     -65.7%     310.25 ± 33%  interrupts.CPU36.NMI:Non-maskable_interrupts
    904.75 ± 25%     -65.7%     310.25 ± 33%  interrupts.CPU36.PMI:Performance_monitoring_interrupts
    350.50 ±  2%     -13.3%     304.00        interrupts.CPU37.58:IR-PCI-MSI.1572883-edge.eth0-TxRx-19
   1394450           -11.4%    1235822        interrupts.CPU37.LOC:Local_timer_interrupts
    350.75 ±  2%     -13.5%     303.25        interrupts.CPU38.59:IR-PCI-MSI.1572884-edge.eth0-TxRx-20
   1394863           -11.4%    1235961        interrupts.CPU38.LOC:Local_timer_interrupts
    839.00 ± 29%     -47.5%     440.75 ± 30%  interrupts.CPU38.NMI:Non-maskable_interrupts
    839.00 ± 29%     -47.5%     440.75 ± 30%  interrupts.CPU38.PMI:Performance_monitoring_interrupts
    348.00           -12.0%     306.25        interrupts.CPU39.60:IR-PCI-MSI.1572885-edge.eth0-TxRx-21
   1394367           -11.4%    1235937        interrupts.CPU39.LOC:Local_timer_interrupts
   1394567           -11.3%    1236485        interrupts.CPU4.LOC:Local_timer_interrupts
    355.75 ±  2%     -15.0%     302.25        interrupts.CPU40.61:IR-PCI-MSI.1572886-edge.eth0-TxRx-22
   1394441           -11.4%    1235766        interrupts.CPU40.LOC:Local_timer_interrupts
      1006 ± 22%     -50.0%     502.75 ±  3%  interrupts.CPU40.NMI:Non-maskable_interrupts
      1006 ± 22%     -50.0%     502.75 ±  3%  interrupts.CPU40.PMI:Performance_monitoring_interrupts
      8749 ± 21%     -36.8%       5525 ± 12%  interrupts.CPU40.RES:Rescheduling_interrupts
    384.50 ± 18%     -20.7%     305.00        interrupts.CPU41.62:IR-PCI-MSI.1572887-edge.eth0-TxRx-23
   1394268           -11.4%    1235731        interrupts.CPU41.LOC:Local_timer_interrupts
    346.75 ±  2%     -11.9%     305.50        interrupts.CPU42.63:IR-PCI-MSI.1572888-edge.eth0-TxRx-24
   1394021           -11.3%    1236286        interrupts.CPU42.LOC:Local_timer_interrupts
    347.75 ±  2%     -12.7%     303.50        interrupts.CPU43.64:IR-PCI-MSI.1572889-edge.eth0-TxRx-25
   1394221           -11.3%    1235978        interrupts.CPU43.LOC:Local_timer_interrupts
   1394343           -11.3%    1236324        interrupts.CPU44.LOC:Local_timer_interrupts
   1393992           -11.3%    1235971        interrupts.CPU45.LOC:Local_timer_interrupts
    349.75 ±  3%     -12.2%     307.25 ±  2%  interrupts.CPU46.67:IR-PCI-MSI.1572892-edge.eth0-TxRx-28
   1394175           -11.3%    1236769        interrupts.CPU46.LOC:Local_timer_interrupts
      6661 ± 14%     -22.9%       5138 ± 14%  interrupts.CPU46.RES:Rescheduling_interrupts
    353.50 ±  3%     -11.5%     312.75 ±  5%  interrupts.CPU47.68:IR-PCI-MSI.1572893-edge.eth0-TxRx-29
   1393568           -11.4%    1235225        interrupts.CPU47.LOC:Local_timer_interrupts
    382.00 ± 15%     -20.0%     305.75        interrupts.CPU48.69:IR-PCI-MSI.1572894-edge.eth0-TxRx-30
   1393742           -11.3%    1235991        interrupts.CPU48.LOC:Local_timer_interrupts
   1393582           -11.3%    1236072        interrupts.CPU49.LOC:Local_timer_interrupts
      7134 ± 24%     -28.2%       5118 ± 17%  interrupts.CPU49.RES:Rescheduling_interrupts
   1394310           -11.3%    1236531        interrupts.CPU5.LOC:Local_timer_interrupts
      1.25 ± 34%  +18920.0%     237.75 ±159%  interrupts.CPU5.TLB:TLB_shootdowns
    353.25 ±  2%     -13.6%     305.25        interrupts.CPU50.71:IR-PCI-MSI.1572896-edge.eth0-TxRx-32
   1394096           -11.3%    1235969        interrupts.CPU50.LOC:Local_timer_interrupts
    350.00 ±  2%     -12.6%     306.00        interrupts.CPU51.74:IR-PCI-MSI.1572897-edge.eth0-TxRx-33
   1393510           -11.3%    1236160        interrupts.CPU51.LOC:Local_timer_interrupts
    360.25 ±  5%     -14.9%     306.50        interrupts.CPU52.75:IR-PCI-MSI.1572898-edge.eth0-TxRx-34
   1393543           -11.3%    1235831        interrupts.CPU52.LOC:Local_timer_interrupts
    347.50 ±  2%     -12.2%     305.25        interrupts.CPU53.76:IR-PCI-MSI.1572899-edge.eth0-TxRx-35
   1393609           -11.3%    1236303        interrupts.CPU53.LOC:Local_timer_interrupts
    348.50 ±  3%     -13.1%     303.00        interrupts.CPU54.95:IR-PCI-MSI.1572918-edge.eth0-TxRx-54
   1391373           -11.1%    1237096        interrupts.CPU54.LOC:Local_timer_interrupts
    346.25 ±  2%     -12.4%     303.25        interrupts.CPU55.96:IR-PCI-MSI.1572919-edge.eth0-TxRx-55
   1391656           -11.1%    1236514        interrupts.CPU55.LOC:Local_timer_interrupts
    345.25 ±  2%     -12.2%     303.25        interrupts.CPU56.97:IR-PCI-MSI.1572920-edge.eth0-TxRx-56
   1391694           -11.1%    1237258        interrupts.CPU56.LOC:Local_timer_interrupts
    345.25 ±  2%     -12.0%     303.75        interrupts.CPU57.98:IR-PCI-MSI.1572921-edge.eth0-TxRx-57
   1391523           -11.1%    1236648        interrupts.CPU57.LOC:Local_timer_interrupts
    346.00 ±  2%     -12.5%     302.75        interrupts.CPU58.99:IR-PCI-MSI.1572922-edge.eth0-TxRx-58
   1391416           -11.1%    1236639        interrupts.CPU58.LOC:Local_timer_interrupts
    990.00 ± 22%     -59.0%     405.50 ± 20%  interrupts.CPU58.NMI:Non-maskable_interrupts
    990.00 ± 22%     -59.0%     405.50 ± 20%  interrupts.CPU58.PMI:Performance_monitoring_interrupts
    346.75 ±  2%     -12.1%     304.75        interrupts.CPU59.100:IR-PCI-MSI.1572923-edge.eth0-TxRx-59
    189468 ± 58%     +42.3%     269641 ±  9%  interrupts.CPU59.238:IR-PCI-MSI.68681742-edge.nvme0q14
   1391397           -11.1%    1236642        interrupts.CPU59.LOC:Local_timer_interrupts
   1394149           -11.3%    1236395        interrupts.CPU6.LOC:Local_timer_interrupts
    346.25 ±  2%     -12.5%     303.00        interrupts.CPU60.101:IR-PCI-MSI.1572924-edge.eth0-TxRx-60
   1391340           -11.2%    1236043        interrupts.CPU60.LOC:Local_timer_interrupts
    350.75 ±  3%     -13.5%     303.25        interrupts.CPU61.102:IR-PCI-MSI.1572925-edge.eth0-TxRx-61
   1391652           -11.1%    1236511        interrupts.CPU61.LOC:Local_timer_interrupts
    350.50 ±  2%     -13.5%     303.25        interrupts.CPU62.103:IR-PCI-MSI.1572926-edge.eth0-TxRx-62
   1391515           -11.1%    1236905        interrupts.CPU62.LOC:Local_timer_interrupts
   1391513           -11.1%    1236897        interrupts.CPU63.LOC:Local_timer_interrupts
   1391196           -11.1%    1236536        interrupts.CPU64.LOC:Local_timer_interrupts
   1391407           -11.1%    1236682        interrupts.CPU65.LOC:Local_timer_interrupts
   1390924           -11.1%    1236339        interrupts.CPU66.LOC:Local_timer_interrupts
   1390923           -11.1%    1236693        interrupts.CPU67.LOC:Local_timer_interrupts
   1391465           -11.1%    1236392        interrupts.CPU68.LOC:Local_timer_interrupts
   1391080           -11.1%    1236169        interrupts.CPU69.LOC:Local_timer_interrupts
   1394504           -11.4%    1235843        interrupts.CPU7.LOC:Local_timer_interrupts
   1392100           -11.2%    1236038        interrupts.CPU70.LOC:Local_timer_interrupts
   1392017           -11.1%    1236955        interrupts.CPU71.LOC:Local_timer_interrupts
      5426 ±  8%     -24.4%       4103 ±  7%  interrupts.CPU71.RES:Rescheduling_interrupts
   1394324           -11.3%    1236282        interrupts.CPU8.LOC:Local_timer_interrupts
   1394500           -11.3%    1236390        interrupts.CPU9.LOC:Local_timer_interrupts
 1.003e+08           -11.2%   89020052        interrupts.LOC:Local_timer_interrupts
     56009 ±  7%     -23.2%      42995 ±  2%  interrupts.NMI:Non-maskable_interrupts
     56009 ±  7%     -23.2%      42995 ±  2%  interrupts.PMI:Performance_monitoring_interrupts
    219.75 ±  8%    +665.4%       1682 ± 19%  interrupts.TLB:TLB_shootdowns
     71.73            -9.3       62.45 ±  3%  perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     71.06            -9.2       61.84 ±  3%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
     59.98 ±  2%      -9.2       50.81 ±  5%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     78.91            -8.9       70.04 ±  2%  perf-profile.calltrace.cycles-pp.secondary_startup_64
     77.67            -8.3       69.41 ±  3%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     77.70            -8.3       69.45 ±  3%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
     77.71            -8.3       69.45 ±  3%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
      1.21 ± 36%      -0.7        0.52 ± 68%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_kernel.secondary_startup_64
      1.21 ± 36%      -0.7        0.52 ± 68%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_kernel.secondary_startup_64
      1.21 ± 36%      -0.7        0.52 ± 68%  perf-profile.calltrace.cycles-pp.start_kernel.secondary_startup_64
      2.25 ±  4%      -0.3        1.95 ± 13%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
      0.64 ±  7%      +0.2        0.84 ±  4%  perf-profile.calltrace.cycles-pp._nfs4_open_and_get_state.nfs4_do_open.nfs4_atomic_open.nfs_atomic_open.path_openat
      0.72 ±  6%      +0.2        0.93 ±  3%  perf-profile.calltrace.cycles-pp.nfs4_do_open.nfs4_atomic_open.nfs_atomic_open.path_openat.do_filp_open
      0.72 ±  7%      +0.2        0.94 ±  3%  perf-profile.calltrace.cycles-pp.nfs4_atomic_open.nfs_atomic_open.path_openat.do_filp_open.do_sys_open
      0.77 ±  7%      +0.3        1.03 ±  3%  perf-profile.calltrace.cycles-pp.nfs_atomic_open.path_openat.do_filp_open.do_sys_open.do_syscall_64
      0.70 ±  6%      +0.3        0.95        perf-profile.calltrace.cycles-pp.__sched_text_start.schedule_idle.do_idle.cpu_startup_entry.start_secondary
      0.53 ±  5%      +0.3        0.79 ± 16%  perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.file_write_and_wait_range.ext4_sync_file.ext4_buffered_write_iter.ext4_file_write_iter
      0.72 ±  6%      +0.3        0.98        perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      0.59 ±  9%      +0.3        0.87 ± 10%  perf-profile.calltrace.cycles-pp.svc_recvfrom.svc_tcp_recvfrom.svc_recv.nfsd.kthread
      0.64 ±  9%      +0.3        0.93 ± 10%  perf-profile.calltrace.cycles-pp.svc_tcp_recvfrom.svc_recv.nfsd.kthread.ret_from_fork
      1.16 ± 11%      +0.3        1.46 ±  4%  perf-profile.calltrace.cycles-pp.__rpc_execute.rpc_run_task.nfs4_call_sync_custom.nfs4_call_sync_sequence._nfs4_proc_lookup
      1.21 ± 10%      +0.3        1.51 ±  3%  perf-profile.calltrace.cycles-pp.rpc_run_task.nfs4_call_sync_custom.nfs4_call_sync_sequence._nfs4_proc_lookup.nfs4_proc_lookup_common
      1.21 ± 10%      +0.3        1.52 ±  3%  perf-profile.calltrace.cycles-pp.nfs4_call_sync_custom.nfs4_call_sync_sequence._nfs4_proc_lookup.nfs4_proc_lookup_common.nfs4_proc_lookup
      1.21 ± 10%      +0.3        1.53 ±  3%  perf-profile.calltrace.cycles-pp.nfs4_call_sync_sequence._nfs4_proc_lookup.nfs4_proc_lookup_common.nfs4_proc_lookup.nfs_lookup_revalidate_dentry
      1.22 ± 10%      +0.3        1.54 ±  4%  perf-profile.calltrace.cycles-pp._nfs4_proc_lookup.nfs4_proc_lookup_common.nfs4_proc_lookup.nfs_lookup_revalidate_dentry.nfs_do_lookup_revalidate
      0.61 ±  7%      +0.3        0.94 ± 15%  perf-profile.calltrace.cycles-pp.file_write_and_wait_range.ext4_sync_file.ext4_buffered_write_iter.ext4_file_write_iter.do_iter_readv_writev
      0.61 ± 13%      +0.3        0.93 ±  3%  perf-profile.calltrace.cycles-pp.handle_irq_event.handle_edge_irq.do_IRQ.ret_from_intr.cpuidle_enter_state
      0.60 ± 13%      +0.3        0.93 ±  3%  perf-profile.calltrace.cycles-pp.handle_irq_event_percpu.handle_irq_event.handle_edge_irq.do_IRQ.ret_from_intr
      1.23 ± 10%      +0.3        1.56 ±  4%  perf-profile.calltrace.cycles-pp.nfs4_proc_lookup.nfs_lookup_revalidate_dentry.nfs_do_lookup_revalidate.__nfs_lookup_revalidate.lookup_dcache
      1.23 ± 10%      +0.3        1.56 ±  4%  perf-profile.calltrace.cycles-pp.nfs4_proc_lookup_common.nfs4_proc_lookup.nfs_lookup_revalidate_dentry.nfs_do_lookup_revalidate.__nfs_lookup_revalidate
      0.64 ± 12%      +0.3        0.97 ±  3%  perf-profile.calltrace.cycles-pp.handle_edge_irq.do_IRQ.ret_from_intr.cpuidle_enter_state.cpuidle_enter
      1.28 ± 10%      +0.3        1.61 ±  3%  perf-profile.calltrace.cycles-pp.nfs_lookup_revalidate_dentry.nfs_do_lookup_revalidate.__nfs_lookup_revalidate.lookup_dcache.__lookup_hash
      0.63 ± 12%      +0.3        0.97 ± 16%  perf-profile.calltrace.cycles-pp.vfs_create.do_nfsd_create.nfsd4_open.nfsd4_proc_compound.nfsd_dispatch
      1.28 ± 10%      +0.3        1.62 ±  3%  perf-profile.calltrace.cycles-pp.nfs_do_lookup_revalidate.__nfs_lookup_revalidate.lookup_dcache.__lookup_hash.filename_create
      1.29 ± 10%      +0.3        1.63 ±  3%  perf-profile.calltrace.cycles-pp.__nfs_lookup_revalidate.lookup_dcache.__lookup_hash.filename_create.do_mkdirat
      0.67 ± 12%      +0.3        1.02        perf-profile.calltrace.cycles-pp.do_IRQ.ret_from_intr.cpuidle_enter_state.cpuidle_enter.do_idle
      1.30 ± 10%      +0.3        1.65 ±  3%  perf-profile.calltrace.cycles-pp.__lookup_hash.filename_create.do_mkdirat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.23 ±  6%      +0.4        1.58 ±  7%  perf-profile.calltrace.cycles-pp.xs_stream_data_receive_workfn.process_one_work.worker_thread.kthread.ret_from_fork
      1.01 ±  8%      +0.4        1.36 ±  4%  perf-profile.calltrace.cycles-pp.path_openat.do_filp_open.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.30 ± 10%      +0.4        1.65 ±  3%  perf-profile.calltrace.cycles-pp.lookup_dcache.__lookup_hash.filename_create.do_mkdirat.do_syscall_64
      0.69 ± 11%      +0.4        1.04 ±  2%  perf-profile.calltrace.cycles-pp.ret_from_intr.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      0.42 ± 58%      +0.4        0.78 ± 11%  perf-profile.calltrace.cycles-pp.tcp_recvmsg.inet6_recvmsg.svc_recvfrom.svc_tcp_recvfrom.svc_recv
      1.22 ±  6%      +0.4        1.58 ±  7%  perf-profile.calltrace.cycles-pp.xs_stream_data_receive.xs_stream_data_receive_workfn.process_one_work.worker_thread.kthread
      0.43 ± 57%      +0.4        0.79 ± 10%  perf-profile.calltrace.cycles-pp.inet6_recvmsg.svc_recvfrom.svc_tcp_recvfrom.svc_recv.nfsd
      0.40 ± 58%      +0.4        0.78 ± 16%  perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range.ext4_sync_file.ext4_buffered_write_iter
      0.40 ± 58%      +0.4        0.78 ± 16%  perf-profile.calltrace.cycles-pp.ext4_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range.ext4_sync_file
      0.96 ±  8%      +0.4        1.34 ±  4%  perf-profile.calltrace.cycles-pp.inet6_csk_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked
      0.74 ±  5%      +0.4        1.14 ± 14%  perf-profile.calltrace.cycles-pp.ext4_sync_file.ext4_buffered_write_iter.ext4_file_write_iter.do_iter_readv_writev.do_iter_write
      0.25 ±100%      +0.4        0.65 ±  5%  perf-profile.calltrace.cycles-pp.__sched_text_start.schedule.worker_thread.kthread.ret_from_fork
      1.31 ± 11%      +0.4        1.71 ±  4%  perf-profile.calltrace.cycles-pp.tcp_v6_rcv.ip6_protocol_deliver_rcu.ip6_input_finish.ip6_input.ipv6_rcv
      0.26 ±100%      +0.4        0.67 ±  5%  perf-profile.calltrace.cycles-pp.schedule.worker_thread.kthread.ret_from_fork
      1.35 ± 10%      +0.4        1.76 ±  4%  perf-profile.calltrace.cycles-pp.ip6_input_finish.ip6_input.ipv6_rcv.__netif_receive_skb_one_core.process_backlog
      1.34 ± 10%      +0.4        1.76 ±  4%  perf-profile.calltrace.cycles-pp.ip6_protocol_deliver_rcu.ip6_input_finish.ip6_input.ipv6_rcv.__netif_receive_skb_one_core
      1.36 ± 10%      +0.4        1.78 ±  4%  perf-profile.calltrace.cycles-pp.ip6_input.ipv6_rcv.__netif_receive_skb_one_core.process_backlog.net_rx_action
      1.41 ±  9%      +0.4        1.84 ±  4%  perf-profile.calltrace.cycles-pp.ipv6_rcv.__netif_receive_skb_one_core.process_backlog.net_rx_action.__softirqentry_text_start
      0.44 ± 58%      +0.4        0.87 ±  4%  perf-profile.calltrace.cycles-pp.blk_mq_complete_request.nvme_irq.__handle_irq_event_percpu.handle_irq_event_percpu.handle_irq_event
      0.48 ± 57%      +0.4        0.93 ± 16%  perf-profile.calltrace.cycles-pp.ext4_create.vfs_create.do_nfsd_create.nfsd4_open.nfsd4_proc_compound
      0.47 ± 58%      +0.4        0.91 ±  3%  perf-profile.calltrace.cycles-pp.__handle_irq_event_percpu.handle_irq_event_percpu.handle_irq_event.handle_edge_irq.do_IRQ
      0.46 ± 59%      +0.4        0.91 ±  3%  perf-profile.calltrace.cycles-pp.nvme_irq.__handle_irq_event_percpu.handle_irq_event_percpu.handle_irq_event.handle_edge_irq
      0.44 ± 58%      +0.4        0.89 ± 18%  perf-profile.calltrace.cycles-pp.generic_perform_write.ext4_buffered_write_iter.ext4_file_write_iter.do_iter_readv_writev.do_iter_write
      1.10 ±  7%      +0.5        1.55 ±  9%  perf-profile.calltrace.cycles-pp.inet6_csk_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.do_tcp_sendpages
      0.28 ±100%      +0.5        0.75 ±  6%  perf-profile.calltrace.cycles-pp.blk_update_request.blk_mq_end_request.blk_mq_complete_request.nvme_irq.__handle_irq_event_percpu
      1.45 ±  9%      +0.5        1.93 ±  5%  perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.net_rx_action.__softirqentry_text_start.do_softirq_own_stack
      1.49 ±  9%      +0.5        1.98 ±  5%  perf-profile.calltrace.cycles-pp.process_backlog.net_rx_action.__softirqentry_text_start.do_softirq_own_stack.do_softirq
      1.22 ±  7%      +0.5        1.71 ±  9%  perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.do_tcp_sendpages.tcp_sendpage_locked
      0.28 ±100%      +0.5        0.78 ±  6%  perf-profile.calltrace.cycles-pp.blk_mq_end_request.blk_mq_complete_request.nvme_irq.__handle_irq_event_percpu.handle_irq_event_percpu
      1.52 ±  8%      +0.5        2.02 ±  5%  perf-profile.calltrace.cycles-pp.net_rx_action.__softirqentry_text_start.do_softirq_own_stack.do_softirq.__local_bh_enable_ip
      1.38 ±  9%      +0.5        1.90 ±  7%  perf-profile.calltrace.cycles-pp.jbd2_journal_commit_transaction.kjournald2.kthread.ret_from_fork
      1.36 ±  7%      +0.5        1.88 ±  9%  perf-profile.calltrace.cycles-pp.tcp_write_xmit.__tcp_push_pending_frames.do_tcp_sendpages.tcp_sendpage_locked.tcp_sendpage
      1.36 ±  7%      +0.5        1.88 ±  8%  perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.do_tcp_sendpages.tcp_sendpage_locked.tcp_sendpage.inet_sendpage
      1.55 ±  8%      +0.5        2.08 ±  4%  perf-profile.calltrace.cycles-pp.__softirqentry_text_start.do_softirq_own_stack.do_softirq.__local_bh_enable_ip.ip6_finish_output2
      1.55 ±  9%      +0.5        2.08 ±  4%  perf-profile.calltrace.cycles-pp.do_softirq_own_stack.do_softirq.__local_bh_enable_ip.ip6_finish_output2.ip6_output
      1.58 ±  8%      +0.5        2.11 ±  5%  perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.ip6_finish_output2.ip6_output.ip6_xmit
      1.04 ±  9%      +0.5        1.58 ± 14%  perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg
      1.58 ±  9%      +0.5        2.12 ±  5%  perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.ip6_finish_output2.ip6_output.ip6_xmit.inet6_csk_xmit
      1.43 ±  8%      +0.6        1.98 ±  8%  perf-profile.calltrace.cycles-pp.kjournald2.kthread.ret_from_fork
      1.12 ±  6%      +0.6        1.67 ± 15%  perf-profile.calltrace.cycles-pp.do_nfsd_create.nfsd4_open.nfsd4_proc_compound.nfsd_dispatch.svc_process_common
      0.14 ±173%      +0.6        0.70 ±  6%  perf-profile.calltrace.cycles-pp.file_write_and_wait_range.nfs_file_fsync.do_fsync.__x64_sys_fsync.do_syscall_64
      0.14 ±173%      +0.6        0.72 ±  5%  perf-profile.calltrace.cycles-pp.nfs_file_fsync.do_fsync.__x64_sys_fsync.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.54 ±  7%      +0.6        2.15 ±  9%  perf-profile.calltrace.cycles-pp.do_tcp_sendpages.tcp_sendpage_locked.tcp_sendpage.inet_sendpage.kernel_sendpage
      0.00            +0.6        0.60 ±  4%  perf-profile.calltrace.cycles-pp.generic_perform_write.nfs_file_write.new_sync_write.vfs_write.ksys_write
      1.55 ±  7%      +0.6        2.17 ±  9%  perf-profile.calltrace.cycles-pp.tcp_sendpage_locked.tcp_sendpage.inet_sendpage.kernel_sendpage.svc_send_common
      0.00            +0.7        0.65 ±  6%  perf-profile.calltrace.cycles-pp.nfs_file_write.new_sync_write.vfs_write.ksys_write.do_syscall_64
      1.66 ±  8%      +0.7        2.32 ±  5%  perf-profile.calltrace.cycles-pp.xs_tcp_send_request.xprt_transmit.call_transmit.__rpc_execute.rpc_async_schedule
      0.00            +0.7        0.66 ±  6%  perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.79 ±  8%      +0.7        2.47 ±  5%  perf-profile.calltrace.cycles-pp.ip6_finish_output2.ip6_output.ip6_xmit.inet6_csk_xmit.__tcp_transmit_skb
      1.44 ±  7%      +0.7        2.13 ± 15%  perf-profile.calltrace.cycles-pp.nfsd4_open.nfsd4_proc_compound.nfsd_dispatch.svc_process_common.svc_process
      1.92 ±  8%      +0.7        2.65 ±  5%  perf-profile.calltrace.cycles-pp.ip6_output.ip6_xmit.inet6_csk_xmit.__tcp_transmit_skb.tcp_write_xmit
      1.73 ±  8%      +0.7        2.47 ±  5%  perf-profile.calltrace.cycles-pp.xprt_transmit.call_transmit.__rpc_execute.rpc_async_schedule.process_one_work
      1.63 ±  5%      +0.7        2.38 ±  7%  perf-profile.calltrace.cycles-pp.svc_recv.nfsd.kthread.ret_from_fork
      1.74 ±  8%      +0.7        2.48 ±  5%  perf-profile.calltrace.cycles-pp.call_transmit.__rpc_execute.rpc_async_schedule.process_one_work.worker_thread
      1.79 ±  6%      +0.8        2.55 ±  9%  perf-profile.calltrace.cycles-pp.tcp_sendpage.inet_sendpage.kernel_sendpage.svc_send_common.svc_sendto
      1.89 ± 12%      +0.8        2.65 ±  3%  perf-profile.calltrace.cycles-pp.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.xs_sendpages.xs_tcp_send_request
      1.82 ±  6%      +0.8        2.58 ±  9%  perf-profile.calltrace.cycles-pp.kernel_sendpage.svc_send_common.svc_sendto.svc_tcp_sendto.svc_send
      1.39 ±  5%      +0.8        2.16 ± 16%  perf-profile.calltrace.cycles-pp.ext4_buffered_write_iter.ext4_file_write_iter.do_iter_readv_writev.do_iter_write.nfsd_vfs_write
      1.99 ±  8%      +0.8        2.76 ±  5%  perf-profile.calltrace.cycles-pp.ip6_xmit.inet6_csk_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames
      1.81 ±  6%      +0.8        2.58 ±  9%  perf-profile.calltrace.cycles-pp.inet_sendpage.kernel_sendpage.svc_send_common.svc_sendto.svc_tcp_sendto
      1.39 ±  5%      +0.8        2.17 ± 16%  perf-profile.calltrace.cycles-pp.ext4_file_write_iter.do_iter_readv_writev.do_iter_write.nfsd_vfs_write.nfsd4_write
      1.40 ±  5%      +0.8        2.17 ± 16%  perf-profile.calltrace.cycles-pp.do_iter_readv_writev.do_iter_write.nfsd_vfs_write.nfsd4_write.nfsd4_proc_compound
      1.82 ±  6%      +0.8        2.60 ±  9%  perf-profile.calltrace.cycles-pp.svc_send_common.svc_sendto.svc_tcp_sendto.svc_send.nfsd
      1.44 ±  5%      +0.8        2.22 ± 16%  perf-profile.calltrace.cycles-pp.nfsd_vfs_write.nfsd4_write.nfsd4_proc_compound.nfsd_dispatch.svc_process_common
      1.42 ±  5%      +0.8        2.20 ± 16%  perf-profile.calltrace.cycles-pp.do_iter_write.nfsd_vfs_write.nfsd4_write.nfsd4_proc_compound.nfsd_dispatch
      1.82 ±  6%      +0.8        2.61 ±  9%  perf-profile.calltrace.cycles-pp.svc_sendto.svc_tcp_sendto.svc_send.nfsd.kthread
      1.83 ±  6%      +0.8        2.62 ±  9%  perf-profile.calltrace.cycles-pp.svc_tcp_sendto.svc_send.nfsd.kthread.ret_from_fork
      1.49 ±  5%      +0.8        2.30 ± 15%  perf-profile.calltrace.cycles-pp.nfsd4_write.nfsd4_proc_compound.nfsd_dispatch.svc_process_common.svc_process
      2.04 ± 11%      +0.8        2.85 ±  3%  perf-profile.calltrace.cycles-pp.tcp_sendmsg.sock_sendmsg.xs_sendpages.xs_tcp_send_request.xprt_transmit
      2.08 ± 11%      +0.8        2.91 ±  2%  perf-profile.calltrace.cycles-pp.sock_sendmsg.xs_sendpages.xs_tcp_send_request.xprt_transmit.call_transmit
      2.09 ± 11%      +0.8        2.94 ±  2%  perf-profile.calltrace.cycles-pp.xs_sendpages.xs_tcp_send_request.xprt_transmit.call_transmit.__rpc_execute
      0.15 ±173%      +0.9        1.03 ± 37%  perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      2.05 ±  6%      +0.9        2.95 ±  9%  perf-profile.calltrace.cycles-pp.svc_send.nfsd.kthread.ret_from_fork
      0.45 ±114%      +0.9        1.38 ±  4%  perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v6_do_rcv.tcp_v6_rcv.ip6_protocol_deliver_rcu.ip6_input_finish
      0.45 ±114%      +0.9        1.39 ±  4%  perf-profile.calltrace.cycles-pp.tcp_v6_do_rcv.tcp_v6_rcv.ip6_protocol_deliver_rcu.ip6_input_finish.ip6_input
      3.30 ±  8%      +1.6        4.85 ±  3%  perf-profile.calltrace.cycles-pp.__rpc_execute.rpc_async_schedule.process_one_work.worker_thread.kthread
      3.31 ±  8%      +1.6        4.87 ±  3%  perf-profile.calltrace.cycles-pp.rpc_async_schedule.process_one_work.worker_thread.kthread.ret_from_fork
      3.94 ±  4%      +2.1        6.03 ± 14%  perf-profile.calltrace.cycles-pp.nfsd4_proc_compound.nfsd_dispatch.svc_process_common.svc_process.nfsd
      5.18 ±  5%      +2.2        7.36 ±  3%  perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork
      4.24 ±  4%      +2.2        6.45 ± 14%  perf-profile.calltrace.cycles-pp.nfsd_dispatch.svc_process_common.svc_process.nfsd.kthread
      4.55 ±  4%      +2.3        6.89 ± 14%  perf-profile.calltrace.cycles-pp.svc_process_common.svc_process.nfsd.kthread.ret_from_fork
      4.56 ±  4%      +2.3        6.91 ± 14%  perf-profile.calltrace.cycles-pp.svc_process.nfsd.kthread.ret_from_fork
      5.81 ±  5%      +2.4        8.18 ±  2%  perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork
      8.25 ±  4%      +4.0       12.24 ± 11%  perf-profile.calltrace.cycles-pp.nfsd.kthread.ret_from_fork
     15.63 ±  3%      +7.0       22.61 ±  7%  perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
     15.63 ±  3%      +7.0       22.61 ±  7%  perf-profile.calltrace.cycles-pp.ret_from_fork
     72.86            -9.9       62.95 ±  3%  perf-profile.children.cycles-pp.cpuidle_enter_state
     72.88            -9.9       62.98 ±  3%  perf-profile.children.cycles-pp.cpuidle_enter
     60.09 ±  2%      -9.1       51.01 ±  5%  perf-profile.children.cycles-pp.intel_idle
     78.91            -8.9       70.04 ±  2%  perf-profile.children.cycles-pp.secondary_startup_64
     78.91            -8.9       70.04 ±  2%  perf-profile.children.cycles-pp.cpu_startup_entry
     78.94            -8.9       70.07 ±  2%  perf-profile.children.cycles-pp.do_idle
     77.71            -8.3       69.45 ±  3%  perf-profile.children.cycles-pp.start_secondary
      1.21 ± 36%      -0.6        0.59 ± 45%  perf-profile.children.cycles-pp.start_kernel
      0.71 ± 16%      -0.3        0.43 ± 13%  perf-profile.children.cycles-pp.rebalance_domains
      0.44 ±  9%      -0.2        0.21 ± 12%  perf-profile.children.cycles-pp.run_rebalance_domains
      0.42 ± 11%      -0.2        0.21 ± 10%  perf-profile.children.cycles-pp.update_blocked_averages
      0.58 ± 12%      -0.2        0.40 ± 13%  perf-profile.children.cycles-pp.load_balance
      0.44 ± 37%      -0.2        0.27 ±  7%  perf-profile.children.cycles-pp.tick_irq_enter
      0.56 ± 29%      -0.2        0.41 ±  5%  perf-profile.children.cycles-pp.irq_enter
      0.28 ± 15%      -0.1        0.16 ± 22%  perf-profile.children.cycles-pp.run_timer_softirq
      0.16 ± 12%      -0.1        0.10 ±  7%  perf-profile.children.cycles-pp.__remove_hrtimer
      0.08 ± 15%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.update_rt_rq_load_avg
      0.08 ± 32%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.pm_qos_read_value
      0.24 ±  7%      -0.0        0.20 ± 10%  perf-profile.children.cycles-pp.update_ts_time_stats
      0.11 ± 17%      -0.0        0.07 ± 14%  perf-profile.children.cycles-pp.timerqueue_del
      0.12 ±  6%      -0.0        0.09 ±  8%  perf-profile.children.cycles-pp.enqueue_hrtimer
      0.06 ±  7%      +0.0        0.08 ± 11%  perf-profile.children.cycles-pp.selinux_parse_skb
      0.12 ±  7%      +0.0        0.14 ±  8%  perf-profile.children.cycles-pp.tcp_event_new_data_sent
      0.06 ± 15%      +0.0        0.08 ± 23%  perf-profile.children.cycles-pp.svc_xprt_release
      0.07 ± 14%      +0.0        0.10 ±  5%  perf-profile.children.cycles-pp.lookup_fast
      0.06 ± 11%      +0.0        0.09 ± 13%  perf-profile.children.cycles-pp.security_prepare_creds
      0.09 ±  7%      +0.0        0.12 ± 11%  perf-profile.children.cycles-pp.__update_load_avg_se
      0.08 ± 16%      +0.0        0.11 ± 10%  perf-profile.children.cycles-pp.do_dentry_open
      0.08 ± 19%      +0.0        0.11 ±  4%  perf-profile.children.cycles-pp.nfs4_xdr_dec_lookup
      0.04 ± 57%      +0.0        0.07 ±  7%  perf-profile.children.cycles-pp.nfsd_file_delayed_close
      0.04 ± 58%      +0.0        0.07 ± 12%  perf-profile.children.cycles-pp.prepare_to_wait
      0.07 ± 14%      +0.0        0.10 ±  4%  perf-profile.children.cycles-pp.walk_component
      0.05 ±  9%      +0.0        0.08 ± 10%  perf-profile.children.cycles-pp.tcp_v6_send_check
      0.08 ± 19%      +0.0        0.11 ± 10%  perf-profile.children.cycles-pp.crc32c_pcl_intel_update
      0.17 ±  9%      +0.0        0.20 ± 10%  perf-profile.children.cycles-pp.ext4_nfs_commit_metadata
      0.17 ±  9%      +0.0        0.20 ± 10%  perf-profile.children.cycles-pp.ext4_write_inode
      0.08 ± 10%      +0.0        0.11 ± 17%  perf-profile.children.cycles-pp.jbd2_journal_write_metadata_buffer
      0.09 ±  8%      +0.0        0.12 ±  8%  perf-profile.children.cycles-pp.selinux_ip_postroute_compat
      0.07 ± 12%      +0.0        0.10 ±  8%  perf-profile.children.cycles-pp.xprt_free_slot
      0.06 ±  7%      +0.0        0.09 ± 16%  perf-profile.children.cycles-pp.dput
      0.11 ± 11%      +0.0        0.14 ±  7%  perf-profile.children.cycles-pp.nfs_fhget
      0.09 ± 17%      +0.0        0.12 ± 13%  perf-profile.children.cycles-pp.__check_object_size
      0.16 ±  7%      +0.0        0.20 ± 10%  perf-profile.children.cycles-pp.nfsd_create_setattr
      0.09 ± 13%      +0.0        0.12 ± 12%  perf-profile.children.cycles-pp.memcpy_from_page
      0.06 ± 13%      +0.0        0.10 ± 11%  perf-profile.children.cycles-pp.xas_load
      0.06 ± 11%      +0.0        0.09 ± 15%  perf-profile.children.cycles-pp.xprt_request_enqueue_receive
      0.06 ± 11%      +0.0        0.09 ± 23%  perf-profile.children.cycles-pp.memcmp
      0.13 ±  6%      +0.0        0.16 ±  6%  perf-profile.children.cycles-pp.decode_getfattr_attrs
      0.06 ± 22%      +0.0        0.09 ± 23%  perf-profile.children.cycles-pp.mpage_submit_page
      0.09 ±  4%      +0.0        0.13 ± 15%  perf-profile.children.cycles-pp.security_inode_permission
      0.07 ± 16%      +0.0        0.10 ± 14%  perf-profile.children.cycles-pp.tcp_schedule_loss_probe
      0.11 ± 12%      +0.0        0.14 ±  5%  perf-profile.children.cycles-pp.rpcauth_refreshcred
      0.11 ±  6%      +0.0        0.15 ± 14%  perf-profile.children.cycles-pp.selinux_ip_postroute
      0.04 ± 58%      +0.0        0.08 ± 16%  perf-profile.children.cycles-pp.svc_generic_init_request
      0.16 ±  9%      +0.0        0.20 ±  5%  perf-profile.children.cycles-pp.__kmalloc
      0.10 ±  7%      +0.0        0.14 ±  7%  perf-profile.children.cycles-pp.__intel_pmu_enable_all
      0.08 ± 31%      +0.0        0.12 ± 13%  perf-profile.children.cycles-pp.nfsd4_close
      0.11 ± 14%      +0.0        0.14 ±  5%  perf-profile.children.cycles-pp._copy_to_iter
      0.05            +0.0        0.09 ± 21%  perf-profile.children.cycles-pp.enqueue_to_backlog
      0.06 ± 14%      +0.0        0.10 ± 11%  perf-profile.children.cycles-pp.unx_marshal
      0.06 ± 13%      +0.0        0.10 ±  7%  perf-profile.children.cycles-pp.ext4_io_submit
      0.11 ± 11%      +0.0        0.15 ± 13%  perf-profile.children.cycles-pp.jbd2_complete_transaction
      0.18 ±  4%      +0.0        0.21 ±  6%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.07 ± 12%      +0.0        0.11 ± 16%  perf-profile.children.cycles-pp.mpage_map_and_submit_buffers
      0.04 ± 57%      +0.0        0.08 ± 19%  perf-profile.children.cycles-pp.selinux_sock_rcv_skb_compat
      0.04 ± 58%      +0.0        0.08 ± 23%  perf-profile.children.cycles-pp.page_fault
      0.04 ± 58%      +0.0        0.08 ±  8%  perf-profile.children.cycles-pp.generic_block_bmap
      0.06 ±  9%      +0.0        0.10 ±  5%  perf-profile.children.cycles-pp.rpc_malloc
      0.03 ±100%      +0.0        0.07 ±  7%  perf-profile.children.cycles-pp.__jbd2_journal_temp_unlink_buffer
      0.12 ±  4%      +0.0        0.17 ±  3%  perf-profile.children.cycles-pp.path_parentat
      0.11 ± 10%      +0.0        0.15 ±  2%  perf-profile.children.cycles-pp.unx_lookup_cred
      0.06 ± 11%      +0.0        0.10 ±  7%  perf-profile.children.cycles-pp.jbd2_journal_bmap
      0.07 ±  7%      +0.0        0.11 ±  4%  perf-profile.children.cycles-pp.call_allocate
      0.05 ± 58%      +0.0        0.09 ±  9%  perf-profile.children.cycles-pp.svc_xprt_dequeue
      0.04 ± 58%      +0.0        0.08 ±  5%  perf-profile.children.cycles-pp.xdr_reserve_space
      0.14 ±  3%      +0.0        0.18 ±  3%  perf-profile.children.cycles-pp.mempool_alloc
      0.04 ± 57%      +0.0        0.08 ± 23%  perf-profile.children.cycles-pp.do_page_fault
      0.04 ± 60%      +0.0        0.09 ± 19%  perf-profile.children.cycles-pp.__block_commit_write
      0.03 ±100%      +0.0        0.07 ± 12%  perf-profile.children.cycles-pp.bio_alloc_bioset
      0.13 ±  7%      +0.0        0.17 ±  2%  perf-profile.children.cycles-pp.filename_parentat
      0.09 ±  8%      +0.0        0.14 ±  8%  perf-profile.children.cycles-pp.alloc_inode
      0.04 ± 58%      +0.0        0.09 ± 24%  perf-profile.children.cycles-pp.ext4_es_lookup_extent
      0.06 ± 20%      +0.0        0.11 ± 10%  perf-profile.children.cycles-pp.__ext4_handle_dirty_metadata
      0.04 ± 58%      +0.0        0.08 ± 13%  perf-profile.children.cycles-pp.nfs_writeback_done
      0.04 ± 60%      +0.0        0.09 ± 16%  perf-profile.children.cycles-pp.block_write_end
      0.09 ± 12%      +0.0        0.13 ± 11%  perf-profile.children.cycles-pp.blk_mq_request_issue_directly
      0.06 ± 11%      +0.0        0.10 ± 15%  perf-profile.children.cycles-pp.__netif_receive_skb_core
      0.10 ± 12%      +0.0        0.15 ± 14%  perf-profile.children.cycles-pp.lock_timer_base
      0.04 ± 59%      +0.0        0.09 ± 23%  perf-profile.children.cycles-pp.pagevec_lru_move_fn
      0.08 ± 16%      +0.0        0.12 ± 14%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      0.08 ± 16%      +0.0        0.12 ± 14%  perf-profile.children.cycles-pp.copyin
      0.08 ±  6%      +0.0        0.12 ±  8%  perf-profile.children.cycles-pp.selinux_socket_sock_rcv_skb
      0.08 ± 19%      +0.0        0.13 ±  9%  perf-profile.children.cycles-pp.__might_sleep
      0.07 ± 14%      +0.0        0.12 ± 15%  perf-profile.children.cycles-pp._cond_resched
      0.09 ± 12%      +0.0        0.14 ± 12%  perf-profile.children.cycles-pp.nvme_queue_rq
      0.06 ± 11%      +0.0        0.11 ± 21%  perf-profile.children.cycles-pp.nfs_writepages_callback
      0.04 ± 59%      +0.0        0.09 ± 14%  perf-profile.children.cycles-pp.nfs_pgio_result
      0.01 ±173%      +0.0        0.06 ± 20%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.01 ±173%      +0.0        0.06 ± 11%  perf-profile.children.cycles-pp.__lock_sock
      0.01 ±173%      +0.0        0.06 ± 20%  perf-profile.children.cycles-pp.__x64_sys_close
      0.01 ±173%      +0.0        0.06 ± 16%  perf-profile.children.cycles-pp.find_get_pages_range_tag
      0.13 ± 11%      +0.0        0.18 ±  4%  perf-profile.children.cycles-pp.memset_erms
      0.08 ± 23%      +0.1        0.13 ± 21%  perf-profile.children.cycles-pp.dx_probe
      0.06 ±  7%      +0.1        0.11 ± 21%  perf-profile.children.cycles-pp.nfs_do_writepage
      0.12 ±  8%      +0.1        0.17 ± 10%  perf-profile.children.cycles-pp.svcauth_unix_set_client
      0.11 ±  3%      +0.1        0.16 ±  6%  perf-profile.children.cycles-pp.nfs4svc_decode_compoundargs
      0.10 ± 11%      +0.1        0.15 ±  2%  perf-profile.children.cycles-pp.rpcauth_wrap_req_encode
      0.10 ± 15%      +0.1        0.16 ±  7%  perf-profile.children.cycles-pp._copy_from_iter_full
      0.10 ± 11%      +0.1        0.15 ±  7%  perf-profile.children.cycles-pp.sk_filter_trim_cap
      0.15 ±  9%      +0.1        0.21 ± 15%  perf-profile.children.cycles-pp.nf_hook_slow
      0.15 ± 11%      +0.1        0.20 ± 11%  perf-profile.children.cycles-pp.journal_submit_commit_record
      0.11 ± 10%      +0.1        0.16 ± 11%  perf-profile.children.cycles-pp.inet6_csk_route_socket
      0.09 ± 20%      +0.1        0.14 ± 19%  perf-profile.children.cycles-pp.__pagevec_release
      0.10 ± 11%      +0.1        0.15 ±  7%  perf-profile.children.cycles-pp.menu_reflect
      0.03 ±100%      +0.1        0.08 ± 26%  perf-profile.children.cycles-pp.__do_page_fault
      0.08 ± 17%      +0.1        0.13 ± 12%  perf-profile.children.cycles-pp.tick_nohz_idle_enter
      0.09 ± 14%      +0.1        0.15 ± 10%  perf-profile.children.cycles-pp.__blk_mq_try_issue_directly
      0.08 ± 10%      +0.1        0.14 ±  8%  perf-profile.children.cycles-pp.security_sock_rcv_skb
      0.09 ± 12%      +0.1        0.14 ± 10%  perf-profile.children.cycles-pp.blk_mq_try_issue_list_directly
      0.01 ±173%      +0.1        0.07 ± 17%  perf-profile.children.cycles-pp.cache_check
      0.15 ±  9%      +0.1        0.21 ± 14%  perf-profile.children.cycles-pp.add_to_page_cache_lru
      0.24 ±  3%      +0.1        0.30 ± 11%  perf-profile.children.cycles-pp.newidle_balance
      0.18 ± 18%      +0.1        0.23 ±  6%  perf-profile.children.cycles-pp.lock_sock_nested
      0.06 ±  9%      +0.1        0.11 ± 11%  perf-profile.children.cycles-pp.resched_curr
      0.16 ± 13%      +0.1        0.21 ± 14%  perf-profile.children.cycles-pp.nfs4_get_vfs_file
      0.11 ±  4%      +0.1        0.16 ± 15%  perf-profile.children.cycles-pp.put_cred_rcu
      0.01 ±173%      +0.1        0.07 ±  6%  perf-profile.children.cycles-pp.d_lookup
      0.00            +0.1        0.06 ± 15%  perf-profile.children.cycles-pp.kmalloc_order_trace
      0.00            +0.1        0.06 ± 15%  perf-profile.children.cycles-pp.kmalloc_order
      0.35 ±  5%      +0.1        0.41 ±  7%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.18 ±  6%      +0.1        0.24 ± 12%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.16 ± 13%      +0.1        0.22 ±  5%  perf-profile.children.cycles-pp.blk_mq_make_request
      0.15 ± 14%      +0.1        0.20 ± 15%  perf-profile.children.cycles-pp.nfsd_file_acquire
      0.07 ± 22%      +0.1        0.13 ± 19%  perf-profile.children.cycles-pp.nfsd4_getattr
      0.08 ±  5%      +0.1        0.14 ± 24%  perf-profile.children.cycles-pp.bit_wait_io
      0.07 ± 19%      +0.1        0.12 ± 31%  perf-profile.children.cycles-pp.finish_task_switch
      0.00            +0.1        0.06 ± 14%  perf-profile.children.cycles-pp.pick_next_task_idle
      0.00            +0.1        0.06 ± 14%  perf-profile.children.cycles-pp.tcp_mstamp_refresh
      0.10 ± 12%      +0.1        0.16 ± 22%  perf-profile.children.cycles-pp.__ext4_read_dirblock
      0.11 ± 10%      +0.1        0.17 ± 13%  perf-profile.children.cycles-pp.__jbd2_journal_file_buffer
      0.10 ± 11%      +0.1        0.16 ±  9%  perf-profile.children.cycles-pp.blk_mq_sched_insert_requests
      0.14 ±  8%      +0.1        0.20 ±  3%  perf-profile.children.cycles-pp.nfs_write_begin
      0.11 ± 17%      +0.1        0.17 ± 19%  perf-profile.children.cycles-pp.exportfs_decode_fh
      0.03 ±100%      +0.1        0.09 ±  4%  perf-profile.children.cycles-pp.write_bytes_to_xdr_buf
      0.01 ±173%      +0.1        0.07 ± 20%  perf-profile.children.cycles-pp.end_buffer_write_sync
      0.00            +0.1        0.06 ± 11%  perf-profile.children.cycles-pp.__d_lookup
      0.01 ±173%      +0.1        0.07 ± 31%  perf-profile.children.cycles-pp.handle_mm_fault
      0.01 ±173%      +0.1        0.07 ± 15%  perf-profile.children.cycles-pp.d_splice_alias
      0.15 ±  8%      +0.1        0.21 ±  3%  perf-profile.children.cycles-pp.decode_getfattr_generic
      0.09 ±  4%      +0.1        0.15 ±  8%  perf-profile.children.cycles-pp.new_slab
      0.05 ± 58%      +0.1        0.11 ± 22%  perf-profile.children.cycles-pp.smpboot_thread_fn
      0.21 ± 12%      +0.1        0.27 ±  8%  perf-profile.children.cycles-pp.skb_copy_datagram_iter
      0.20 ± 11%      +0.1        0.26 ±  9%  perf-profile.children.cycles-pp.__skb_datagram_iter
      0.21 ± 13%      +0.1        0.27 ± 18%  perf-profile.children.cycles-pp.irq_work_interrupt
      0.21 ± 13%      +0.1        0.27 ± 18%  perf-profile.children.cycles-pp.smp_irq_work_interrupt
      0.21 ± 13%      +0.1        0.27 ± 18%  perf-profile.children.cycles-pp.irq_work_run
      0.21 ± 13%      +0.1        0.27 ± 18%  perf-profile.children.cycles-pp.printk
      0.21 ± 13%      +0.1        0.27 ± 18%  perf-profile.children.cycles-pp.vprintk_emit
      0.12 ± 16%      +0.1        0.19 ±  7%  perf-profile.children.cycles-pp.kfree
      0.08 ±  5%      +0.1        0.15 ± 19%  perf-profile.children.cycles-pp.netif_rx
      0.10 ± 14%      +0.1        0.16 ± 10%  perf-profile.children.cycles-pp.blk_finish_plug
      0.10 ± 10%      +0.1        0.17 ±  9%  perf-profile.children.cycles-pp.blk_flush_plug_list
      0.10 ± 10%      +0.1        0.17 ±  9%  perf-profile.children.cycles-pp.blk_mq_flush_plug_list
      0.05 ± 60%      +0.1        0.11 ± 15%  perf-profile.children.cycles-pp.__set_page_dirty_nobuffers
      0.03 ±100%      +0.1        0.09 ± 20%  perf-profile.children.cycles-pp.new_inode
      0.00            +0.1        0.06 ± 26%  perf-profile.children.cycles-pp.__pagevec_lru_add_fn
      0.00            +0.1        0.06 ± 13%  perf-profile.children.cycles-pp.nfs4_sequence_done
      0.00            +0.1        0.06 ± 17%  perf-profile.children.cycles-pp.__kmalloc_node_track_caller
      0.03 ±100%      +0.1        0.09 ± 15%  perf-profile.children.cycles-pp.jbd2_journal_dirty_metadata
      0.01 ±173%      +0.1        0.08 ± 11%  perf-profile.children.cycles-pp.nfs4_xdr_dec_write
      0.16 ± 18%      +0.1        0.22 ± 20%  perf-profile.children.cycles-pp.jbd2_log_wait_commit
      0.02 ±173%      +0.1        0.08 ± 26%  perf-profile.children.cycles-pp.wb_workfn
      0.14 ±  6%      +0.1        0.20 ± 12%  perf-profile.children.cycles-pp.ext4_dirty_inode
      0.07 ±  6%      +0.1        0.13 ± 19%  perf-profile.children.cycles-pp.sunrpc_cache_lookup_rcu
      0.00            +0.1        0.07 ±  7%  perf-profile.children.cycles-pp._ext4_get_block
      0.00            +0.1        0.07 ± 17%  perf-profile.children.cycles-pp.__kmalloc_reserve
      0.11 ± 12%      +0.1        0.17 ±  8%  perf-profile.children.cycles-pp.mutex_spin_on_owner
      0.18 ±  8%      +0.1        0.25 ±  2%  perf-profile.children.cycles-pp.link_path_walk
      0.14 ± 15%      +0.1        0.21 ± 10%  perf-profile.children.cycles-pp.sk_stream_alloc_skb
      0.17 ± 13%      +0.1        0.24 ± 20%  perf-profile.children.cycles-pp.__find_get_block
      0.12 ±  8%      +0.1        0.19 ± 12%  perf-profile.children.cycles-pp.__mutex_lock
      0.14 ± 12%      +0.1        0.21 ± 12%  perf-profile.children.cycles-pp.nfs_initiate_pgio
      0.00            +0.1        0.07 ± 21%  perf-profile.children.cycles-pp.jbd2_journal_stop
      0.08 ±  5%      +0.1        0.15 ± 19%  perf-profile.children.cycles-pp.netif_rx_internal
      0.00            +0.1        0.07 ± 33%  perf-profile.children.cycles-pp.__handle_mm_fault
      0.16 ±  5%      +0.1        0.23 ± 26%  perf-profile.children.cycles-pp.__ext4_find_entry
      0.01 ±173%      +0.1        0.08 ± 21%  perf-profile.children.cycles-pp.percpu_counter_add_batch
      0.19 ± 12%      +0.1        0.27 ± 16%  perf-profile.children.cycles-pp.nfsd4_process_open2
      0.15 ±  3%      +0.1        0.23 ± 27%  perf-profile.children.cycles-pp.ext4_dx_find_entry
      0.18 ±  8%      +0.1        0.26 ±  7%  perf-profile.children.cycles-pp.nfs_pageio_complete
      0.07 ±  6%      +0.1        0.14 ± 22%  perf-profile.children.cycles-pp.exp_find
      0.00            +0.1        0.07 ± 20%  perf-profile.children.cycles-pp.xdr_encode_opaque_fixed
      0.01 ±173%      +0.1        0.09 ± 28%  perf-profile.children.cycles-pp.timekeeping_max_deferment
      0.00            +0.1        0.07 ± 11%  perf-profile.children.cycles-pp.nfs_update_inode
      0.29 ±  5%      +0.1        0.36 ±  7%  perf-profile.children.cycles-pp.native_sched_clock
      0.17 ± 10%      +0.1        0.25 ±  9%  perf-profile.children.cycles-pp.nfs_end_page_writeback
      0.15 ± 10%      +0.1        0.22 ± 11%  perf-profile.children.cycles-pp.__alloc_skb
      0.18 ±  7%      +0.1        0.25 ±  9%  perf-profile.children.cycles-pp.nfs_generic_pg_pgios
      0.00            +0.1        0.07 ± 11%  perf-profile.children.cycles-pp.nfs_refresh_inode_locked
      0.07 ± 10%      +0.1        0.15 ± 22%  perf-profile.children.cycles-pp.rqst_exp_find
      0.23 ±  9%      +0.1        0.31 ±  5%  perf-profile.children.cycles-pp.nfs4_run_open_task
      0.16 ± 15%      +0.1        0.24 ± 11%  perf-profile.children.cycles-pp.wait_on_page_bit
      0.15 ±  9%      +0.1        0.23 ± 17%  perf-profile.children.cycles-pp.generic_write_end
      0.00            +0.1        0.08 ± 14%  perf-profile.children.cycles-pp.new_inode_pseudo
      0.00            +0.1        0.08 ± 19%  perf-profile.children.cycles-pp.rpc_wake_up_queued_task_set_status
      0.13 ±  5%      +0.1        0.21 ± 13%  perf-profile.children.cycles-pp.ext4_mark_iloc_dirty
      0.12 ± 13%      +0.1        0.20 ±  4%  perf-profile.children.cycles-pp.nfs4_close_done
      0.30 ±  6%      +0.1        0.38 ±  8%  perf-profile.children.cycles-pp.sched_clock
      0.18 ±  7%      +0.1        0.26 ±  7%  perf-profile.children.cycles-pp.nfs_pageio_doio
      0.00            +0.1        0.08 ± 15%  perf-profile.children.cycles-pp.exp_find_key
      0.00            +0.1        0.08 ±  8%  perf-profile.children.cycles-pp.nfs40_sequence_free_slot
      0.12 ±  9%      +0.1        0.20 ±  7%  perf-profile.children.cycles-pp.check_preempt_curr
      0.17 ±  6%      +0.1        0.25 ± 13%  perf-profile.children.cycles-pp.ext4_da_write_end
      0.14 ± 13%      +0.1        0.22 ±  6%  perf-profile.children.cycles-pp.xprt_release
      0.20 ±  9%      +0.1        0.28 ± 15%  perf-profile.children.cycles-pp.kernel_text_address
      0.29 ±  6%      +0.1        0.38 ±  4%  perf-profile.children.cycles-pp._nfs4_opendata_to_nfs4_state
      0.08 ±  5%      +0.1        0.17 ± 25%  perf-profile.children.cycles-pp.do_get_write_access
      0.03 ±100%      +0.1        0.11 ±  7%  perf-profile.children.cycles-pp.in_sched_functions
      0.27 ± 14%      +0.1        0.35 ±  5%  perf-profile.children.cycles-pp.submit_bh_wbc
      0.23 ±  6%      +0.1        0.31 ±  8%  perf-profile.children.cycles-pp.___might_sleep
      0.12 ± 11%      +0.1        0.21 ± 13%  perf-profile.children.cycles-pp.nfs_updatepage
      0.10 ±  5%      +0.1        0.18 ± 15%  perf-profile.children.cycles-pp.add_dirent_to_buf
      0.34 ± 10%      +0.1        0.42 ±  7%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.14 ±  9%      +0.1        0.22 ±  9%  perf-profile.children.cycles-pp.___slab_alloc
      0.13 ± 12%      +0.1        0.22 ± 37%  perf-profile.children.cycles-pp.kmem_cache_free
      0.03 ±100%      +0.1        0.11 ± 35%  perf-profile.children.cycles-pp.__jbd2_log_wait_for_space
      0.25 ±  7%      +0.1        0.34 ±  3%  perf-profile.children.cycles-pp.__release_sock
      0.22 ±  8%      +0.1        0.31 ±  7%  perf-profile.children.cycles-pp.__unwind_start
      0.22 ±  5%      +0.1        0.31 ± 13%  perf-profile.children.cycles-pp.__kernel_text_address
      0.18 ± 10%      +0.1        0.27 ± 10%  perf-profile.children.cycles-pp.prepare_creds
      0.16 ± 15%      +0.1        0.25 ±  5%  perf-profile.children.cycles-pp.set_next_entity
      0.23 ± 12%      +0.1        0.33 ±  6%  perf-profile.children.cycles-pp.generic_make_request
      0.17 ±  2%      +0.1        0.26 ± 22%  perf-profile.children.cycles-pp.ext4_lookup
      0.14 ±  7%      +0.1        0.23 ±  9%  perf-profile.children.cycles-pp.__slab_alloc
      0.21 ± 12%      +0.1        0.30 ± 10%  perf-profile.children.cycles-pp.uart_console_write
      0.15 ± 16%      +0.1        0.24 ±  6%  perf-profile.children.cycles-pp.write_cache_pages
      0.03 ±100%      +0.1        0.12 ± 34%  perf-profile.children.cycles-pp.add_transaction_credits
      0.23 ± 10%      +0.1        0.33 ± 12%  perf-profile.children.cycles-pp.irq_work_run_list
      0.24 ± 13%      +0.1        0.33 ±  5%  perf-profile.children.cycles-pp.submit_bio
      0.36 ±  8%      +0.1        0.46 ±  2%  perf-profile.children.cycles-pp.dequeue_entity
      0.31 ±  4%      +0.1        0.40 ±  5%  perf-profile.children.cycles-pp.tcp_clean_rtx_queue
      0.18 ± 27%      +0.1        0.27 ± 14%  perf-profile.children.cycles-pp.wait_for_xmitr
      0.15 ± 14%      +0.1        0.25 ±  6%  perf-profile.children.cycles-pp.iov_iter_copy_from_user_atomic
      0.00            +0.1        0.10 ± 36%  perf-profile.children.cycles-pp.jbd2_log_do_checkpoint
      0.22 ± 10%      +0.1        0.32 ± 20%  perf-profile.children.cycles-pp.ext4_finish_bio
      0.17 ±  2%      +0.1        0.27 ±  5%  perf-profile.children.cycles-pp.__slab_free
      0.16 ±  6%      +0.1        0.26 ±  6%  perf-profile.children.cycles-pp.ttwu_do_wakeup
      0.23 ±  9%      +0.1        0.33 ± 20%  perf-profile.children.cycles-pp.ext4_ext_map_blocks
      0.15 ± 10%      +0.1        0.25 ± 14%  perf-profile.children.cycles-pp.loopback_xmit
      0.20 ± 10%      +0.1        0.30 ±  9%  perf-profile.children.cycles-pp.__mark_inode_dirty
      0.14 ±  5%      +0.1        0.24 ± 11%  perf-profile.children.cycles-pp.nfs_write_end
      0.21 ± 12%      +0.1        0.31 ±  9%  perf-profile.children.cycles-pp.serial8250_console_write
      0.29 ± 13%      +0.1        0.39 ±  5%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.22 ± 11%      +0.1        0.32 ± 10%  perf-profile.children.cycles-pp.console_unlock
      0.17 ±  4%      +0.1        0.27 ±  6%  perf-profile.children.cycles-pp.__switch_to
      0.26 ±  2%      +0.1        0.36 ±  9%  perf-profile.children.cycles-pp.__kfree_skb
      0.19 ±  2%      +0.1        0.30 ± 20%  perf-profile.children.cycles-pp.__lookup_slow
      0.22 ±  7%      +0.1        0.32 ±  2%  perf-profile.children.cycles-pp.nfs_release_seqid
      0.24 ±  5%      +0.1        0.35 ± 12%  perf-profile.children.cycles-pp.unwind_get_return_address
      0.23 ±  6%      +0.1        0.34 ±  7%  perf-profile.children.cycles-pp.__switch_to_asm
      0.24 ±  9%      +0.1        0.35 ± 18%  perf-profile.children.cycles-pp.ext4_end_bio
      0.13 ±  5%      +0.1        0.24 ± 18%  perf-profile.children.cycles-pp.jbd2_journal_get_write_access
      0.32 ±  8%      +0.1        0.43 ± 10%  perf-profile.children.cycles-pp.select_task_rq_fair
      0.33 ± 14%      +0.1        0.43 ± 15%  perf-profile.children.cycles-pp.ext4_mark_inode_dirty
      0.21 ± 13%      +0.1        0.33 ± 14%  perf-profile.children.cycles-pp.io_schedule
      0.35 ± 13%      +0.1        0.46 ± 13%  perf-profile.children.cycles-pp.pagecache_get_page
      0.41 ± 16%      +0.1        0.52 ±  5%  perf-profile.children.cycles-pp.native_write_msr
      0.34 ±  5%      +0.1        0.45 ± 14%  perf-profile.children.cycles-pp.schedule_timeout
      0.23 ±  6%      +0.1        0.35 ± 12%  perf-profile.children.cycles-pp.grab_cache_page_write_begin
      0.14 ±  6%      +0.1        0.25 ± 18%  perf-profile.children.cycles-pp.__ext4_journal_get_write_access
      0.21 ± 16%      +0.1        0.33 ± 11%  perf-profile.children.cycles-pp.__filemap_fdatawait_range
      0.23 ± 25%      +0.1        0.34 ± 13%  perf-profile.children.cycles-pp.ext4_add_nondir
      0.21 ± 11%      +0.1        0.32 ± 10%  perf-profile.children.cycles-pp.nfsd_setuser
      0.16 ±  8%      +0.1        0.28 ± 12%  perf-profile.children.cycles-pp.dev_hard_start_xmit
      0.21 ± 13%      +0.1        0.33 ±  7%  perf-profile.children.cycles-pp.rpc_release_resources_task
      0.23 ±  2%      +0.1        0.35 ± 19%  perf-profile.children.cycles-pp.lookup_one_len
      0.38 ± 17%      +0.1        0.50 ±  9%  perf-profile.children.cycles-pp.xs_data_ready
      0.22 ± 11%      +0.1        0.34 ±  8%  perf-profile.children.cycles-pp.nfsd_setuser_and_check_port
      0.16 ± 12%      +0.1        0.29 ± 19%  perf-profile.children.cycles-pp.nfsd4_encode_fattr
      0.16 ±  6%      +0.1        0.29 ± 15%  perf-profile.children.cycles-pp.ext4_add_entry
      0.28 ± 13%      +0.1        0.41 ±  6%  perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
      0.28 ±  8%      +0.1        0.40 ± 18%  perf-profile.children.cycles-pp.rpc_async_release
      0.27 ±  8%      +0.1        0.39 ± 18%  perf-profile.children.cycles-pp.nfs_write_completion
      0.13 ± 10%      +0.1        0.26 ±  4%  perf-profile.children.cycles-pp.stack_trace_consume_entry_nosched
      0.16 ±  5%      +0.1        0.28 ± 14%  perf-profile.children.cycles-pp.ext4_dx_add_entry
      0.35 ± 10%      +0.1        0.47 ± 12%  perf-profile.children.cycles-pp.__wait_on_bit
      0.35 ± 10%      +0.1        0.48 ± 10%  perf-profile.children.cycles-pp.nfs_file_release
      0.17 ± 12%      +0.1        0.30 ± 19%  perf-profile.children.cycles-pp.nfsd4_encode_getattr
      0.41 ±  6%      +0.1        0.55 ±  3%  perf-profile.children.cycles-pp.tcp_ack
      0.30 ±  7%      +0.1        0.43 ± 14%  perf-profile.children.cycles-pp.wake_up_page_bit
      0.36 ±  9%      +0.1        0.49 ± 13%  perf-profile.children.cycles-pp.out_of_line_wait_on_bit
      0.32 ± 10%      +0.1        0.45 ±  8%  perf-profile.children.cycles-pp.update_load_avg
      0.38 ±  4%      +0.1        0.51 ±  8%  perf-profile.children.cycles-pp.xs_read_stream_request
      0.23 ±  7%      +0.1        0.37 ± 26%  perf-profile.children.cycles-pp.rcu_do_batch
      0.31 ±  6%      +0.1        0.45 ± 13%  perf-profile.children.cycles-pp.kmem_cache_alloc
      0.29 ± 10%      +0.1        0.43 ±  7%  perf-profile.children.cycles-pp.svc_xprt_received
      0.27 ± 15%      +0.1        0.42 ± 14%  perf-profile.children.cycles-pp.end_bio_bh_io_sync
      0.27 ±  5%      +0.1        0.42 ± 11%  perf-profile.children.cycles-pp.__dev_queue_xmit
      0.46 ±  6%      +0.2        0.61 ±  4%  perf-profile.children.cycles-pp.dequeue_task_fair
      0.29 ±  9%      +0.2        0.44 ±  7%  perf-profile.children.cycles-pp.call_encode
      0.34 ± 12%      +0.2        0.49 ±  8%  perf-profile.children.cycles-pp.call_decode
      0.34 ±  6%      +0.2        0.49 ±  6%  perf-profile.children.cycles-pp.memcpy_erms
      0.42 ± 10%      +0.2        0.58 ±  9%  perf-profile.children.cycles-pp.exit_to_usermode_loop
      0.41 ±  9%      +0.2        0.57 ± 10%  perf-profile.children.cycles-pp.__fput
      0.28 ±  3%      +0.2        0.44 ±  6%  perf-profile.children.cycles-pp.rpc_execute
      0.39 ±  5%      +0.2        0.55 ±  5%  perf-profile.children.cycles-pp.release_sock
      0.35 ± 10%      +0.2        0.52 ± 14%  perf-profile.children.cycles-pp.__put_nfs_open_context
      0.23 ± 10%      +0.2        0.39 ± 14%  perf-profile.children.cycles-pp.nfsd4_encode_operation
      0.41 ±  9%      +0.2        0.57 ±  9%  perf-profile.children.cycles-pp.task_work_run
      0.32 ±  4%      +0.2        0.48 ± 10%  perf-profile.children.cycles-pp.__orc_find
      0.60 ±  6%      +0.2        0.77 ±  7%  perf-profile.children.cycles-pp.xs_sock_recvmsg
      0.23 ± 10%      +0.2        0.41 ±  5%  perf-profile.children.cycles-pp.rpc_exit_task
      0.29 ± 23%      +0.2        0.47 ± 10%  perf-profile.children.cycles-pp.tick_nohz_irq_exit
      0.31 ± 18%      +0.2        0.49 ± 22%  perf-profile.children.cycles-pp.ext4_da_write_begin
      0.35 ±  9%      +0.2        0.53 ±  5%  perf-profile.children.cycles-pp.nfs_writepages
      0.32 ±  7%      +0.2        0.50 ±  5%  perf-profile.children.cycles-pp.orc_find
      0.55 ±  6%      +0.2        0.75 ±  5%  perf-profile.children.cycles-pp.pick_next_task_fair
      0.64 ±  7%      +0.2        0.84 ±  4%  perf-profile.children.cycles-pp._nfs4_open_and_get_state
      0.41 ±  9%      +0.2        0.62 ± 12%  perf-profile.children.cycles-pp.__wake_up_bit
      0.72 ±  6%      +0.2        0.93 ±  3%  perf-profile.children.cycles-pp.nfs4_do_open
      0.72 ±  7%      +0.2        0.94 ±  3%  perf-profile.children.cycles-pp.nfs4_atomic_open
      0.45 ±  8%      +0.2        0.69 ±  5%  perf-profile.children.cycles-pp.read_tsc
      0.48 ±  9%      +0.2        0.72 ±  5%  perf-profile.children.cycles-pp.nfs_file_fsync
      0.48 ± 10%      +0.2        0.72 ±  6%  perf-profile.children.cycles-pp.__x64_sys_fsync
      0.64 ± 14%      +0.2        0.88 ±  6%  perf-profile.children.cycles-pp.rpc_wake_up_queued_task
      0.48 ±  9%      +0.2        0.72 ±  6%  perf-profile.children.cycles-pp.do_fsync
      0.41 ±  8%      +0.2        0.66 ±  5%  perf-profile.children.cycles-pp.nfs_file_write
      0.45 ±  7%      +0.2        0.70 ±  6%  perf-profile.children.cycles-pp.vfs_write
      0.42 ±  7%      +0.2        0.67 ±  6%  perf-profile.children.cycles-pp.new_sync_write
      0.45 ±  8%      +0.3        0.71 ±  7%  perf-profile.children.cycles-pp.ksys_write
      0.77 ±  7%      +0.3        1.03 ±  3%  perf-profile.children.cycles-pp.nfs_atomic_open
      0.52 ±  6%      +0.3        0.79 ± 16%  perf-profile.children.cycles-pp.ext4_writepages
      0.73 ±  5%      +0.3        1.00 ±  2%  perf-profile.children.cycles-pp.schedule_idle
      0.65 ± 14%      +0.3        0.92 ±  6%  perf-profile.children.cycles-pp.__rpc_do_wake_up_task_on_wq
      0.59 ±  9%      +0.3        0.87 ± 10%  perf-profile.children.cycles-pp.svc_recvfrom
      0.67 ±  7%      +0.3        0.95 ±  2%  perf-profile.children.cycles-pp.svc_xprt_do_enqueue
      0.56 ± 13%      +0.3        0.85 ±  6%  perf-profile.children.cycles-pp.blk_update_request
      0.64 ±  9%      +0.3        0.93 ± 10%  perf-profile.children.cycles-pp.svc_tcp_recvfrom
      0.54 ±  5%      +0.3        0.85 ± 12%  perf-profile.children.cycles-pp.fh_verify
      1.21 ± 10%      +0.3        1.52 ±  3%  perf-profile.children.cycles-pp.nfs4_call_sync_custom
      0.58 ± 12%      +0.3        0.89 ±  7%  perf-profile.children.cycles-pp.blk_mq_end_request
      1.21 ± 10%      +0.3        1.53 ±  3%  perf-profile.children.cycles-pp.nfs4_call_sync_sequence
      0.61 ± 11%      +0.3        0.93 ± 16%  perf-profile.children.cycles-pp.ext4_create
      1.22 ± 10%      +0.3        1.54 ±  4%  perf-profile.children.cycles-pp._nfs4_proc_lookup
      1.23 ± 10%      +0.3        1.56 ±  4%  perf-profile.children.cycles-pp.nfs4_proc_lookup
      1.23 ± 10%      +0.3        1.56 ±  4%  perf-profile.children.cycles-pp.nfs4_proc_lookup_common
      1.28 ± 10%      +0.3        1.61 ±  3%  perf-profile.children.cycles-pp.nfs_lookup_revalidate_dentry
      0.63 ± 12%      +0.3        0.97 ± 16%  perf-profile.children.cycles-pp.vfs_create
      1.31 ±  9%      +0.3        1.65 ±  3%  perf-profile.children.cycles-pp.nfs_do_lookup_revalidate
      1.03 ±  6%      +0.3        1.38 ±  3%  perf-profile.children.cycles-pp.do_filp_open
      0.64 ± 13%      +0.3        0.99 ±  5%  perf-profile.children.cycles-pp.blk_mq_complete_request
      1.02 ±  7%      +0.3        1.37 ±  3%  perf-profile.children.cycles-pp.path_openat
      1.30 ± 10%      +0.3        1.65 ±  3%  perf-profile.children.cycles-pp.__lookup_hash
      1.07 ±  6%      +0.3        1.42 ±  3%  perf-profile.children.cycles-pp.do_sys_open
      1.23 ±  6%      +0.4        1.58 ±  7%  perf-profile.children.cycles-pp.xs_stream_data_receive_workfn
      1.22 ±  6%      +0.4        1.58 ±  7%  perf-profile.children.cycles-pp.xs_stream_data_receive
      0.68 ± 13%      +0.4        1.03 ±  4%  perf-profile.children.cycles-pp.nvme_irq
      0.80 ±  5%      +0.4        1.16 ± 10%  perf-profile.children.cycles-pp.__wake_up_common_lock
      1.32 ± 10%      +0.4        1.68 ±  3%  perf-profile.children.cycles-pp.__nfs_lookup_revalidate
      1.32 ± 10%      +0.4        1.68 ±  3%  perf-profile.children.cycles-pp.lookup_dcache
      0.68 ± 13%      +0.4        1.04 ±  4%  perf-profile.children.cycles-pp.__handle_irq_event_percpu
      0.69 ± 12%      +0.4        1.06 ±  4%  perf-profile.children.cycles-pp.handle_irq_event_percpu
      0.70 ± 12%      +0.4        1.07 ±  4%  perf-profile.children.cycles-pp.handle_irq_event
      0.73 ± 13%      +0.4        1.11 ±  4%  perf-profile.children.cycles-pp.handle_edge_irq
      0.77 ± 12%      +0.4        1.16 ±  2%  perf-profile.children.cycles-pp.do_IRQ
      0.74 ±  5%      +0.4        1.14 ± 15%  perf-profile.children.cycles-pp.ext4_sync_file
      0.79 ± 11%      +0.4        1.18 ±  3%  perf-profile.children.cycles-pp.ret_from_intr
      1.46 ±  9%      +0.4        1.87 ±  3%  perf-profile.children.cycles-pp.filename_create
      1.49 ±  9%      +0.4        1.90 ±  3%  perf-profile.children.cycles-pp.do_mkdirat
      1.08 ±  5%      +0.4        1.48 ±  8%  perf-profile.children.cycles-pp.tcp_recvmsg
      1.09 ±  5%      +0.4        1.51 ±  8%  perf-profile.children.cycles-pp.inet6_recvmsg
      0.88 ±  7%      +0.4        1.31 ± 11%  perf-profile.children.cycles-pp.__filemap_fdatawrite_range
      0.88 ±  7%      +0.5        1.33 ± 10%  perf-profile.children.cycles-pp.do_writepages
      0.96 ±  5%      +0.5        1.42 ±  6%  perf-profile.children.cycles-pp.unwind_next_frame
      1.13 ±  7%      +0.5        1.59 ±  9%  perf-profile.children.cycles-pp._raw_spin_lock
      1.49 ±  7%      +0.5        1.95 ±  3%  perf-profile.children.cycles-pp.tcp_rcv_established
      1.50 ±  7%      +0.5        1.97 ±  3%  perf-profile.children.cycles-pp.tcp_v6_do_rcv
      1.26 ±  5%      +0.5        1.74 ±  5%  perf-profile.children.cycles-pp.schedule
      1.12 ±  2%      +0.5        1.60 ±  7%  perf-profile.children.cycles-pp.queue_work_on
      1.66 ±  7%      +0.5        2.14 ±  3%  perf-profile.children.cycles-pp.tcp_v6_rcv
      1.07 ±  3%      +0.5        1.56 ±  7%  perf-profile.children.cycles-pp.__queue_work
      1.70 ±  8%      +0.5        2.20 ±  3%  perf-profile.children.cycles-pp.ip6_protocol_deliver_rcu
      1.71 ±  7%      +0.5        2.21 ±  3%  perf-profile.children.cycles-pp.ip6_input_finish
      1.72 ±  7%      +0.5        2.23 ±  4%  perf-profile.children.cycles-pp.ip6_input
      1.39 ±  9%      +0.5        1.90 ±  7%  perf-profile.children.cycles-pp.jbd2_journal_commit_transaction
      1.79 ±  7%      +0.5        2.31 ±  4%  perf-profile.children.cycles-pp.ipv6_rcv
      1.58 ±  9%      +0.5        2.12 ±  4%  perf-profile.children.cycles-pp.rpc_run_task
      1.09 ±  8%      +0.5        1.64 ± 11%  perf-profile.children.cycles-pp.file_write_and_wait_range
      0.50 ± 15%      +0.5        1.05 ± 37%  perf-profile.children.cycles-pp.poll_idle
      1.44 ±  8%      +0.5        1.98 ±  8%  perf-profile.children.cycles-pp.kjournald2
      1.12 ±  6%      +0.6        1.68 ± 15%  perf-profile.children.cycles-pp.do_nfsd_create
      0.94 ±  7%      +0.6        1.49 ± 12%  perf-profile.children.cycles-pp.generic_perform_write
      1.17 ±  6%      +0.6        1.75 ±  9%  perf-profile.children.cycles-pp.autoremove_wake_function
      1.86 ±  7%      +0.6        2.43 ±  4%  perf-profile.children.cycles-pp.__netif_receive_skb_one_core
      1.91 ±  7%      +0.6        2.50 ±  4%  perf-profile.children.cycles-pp.process_backlog
      1.23 ±  6%      +0.6        1.83 ±  9%  perf-profile.children.cycles-pp.__wake_up_common
      1.54 ±  7%      +0.6        2.15 ±  9%  perf-profile.children.cycles-pp.do_tcp_sendpages
      1.55 ±  7%      +0.6        2.17 ±  9%  perf-profile.children.cycles-pp.tcp_sendpage_locked
      1.95 ±  7%      +0.6        2.57 ±  4%  perf-profile.children.cycles-pp.net_rx_action
      2.02 ±  7%      +0.6        2.66 ±  4%  perf-profile.children.cycles-pp.do_softirq
      1.99 ±  7%      +0.6        2.63 ±  4%  perf-profile.children.cycles-pp.do_softirq_own_stack
      2.04 ±  7%      +0.7        2.70 ±  4%  perf-profile.children.cycles-pp.__local_bh_enable_ip
      1.99 ±  7%      +0.7        2.65 ±  3%  perf-profile.children.cycles-pp.tcp_sendmsg_locked
      1.44 ±  7%      +0.7        2.13 ± 15%  perf-profile.children.cycles-pp.nfsd4_open
      2.15 ±  5%      +0.7        2.84 ±  3%  perf-profile.children.cycles-pp.tcp_sendmsg
      2.19 ±  5%      +0.7        2.91 ±  2%  perf-profile.children.cycles-pp.sock_sendmsg
      2.21 ±  6%      +0.7        2.94 ±  2%  perf-profile.children.cycles-pp.xs_sendpages
      1.43 ±  4%      +0.7        2.17 ±  6%  perf-profile.children.cycles-pp.arch_stack_walk
      2.25 ±  6%      +0.7        2.98 ±  2%  perf-profile.children.cycles-pp.xs_tcp_send_request
      1.97 ±  4%      +0.7        2.71 ±  3%  perf-profile.children.cycles-pp.__sched_text_start
      1.64 ±  5%      +0.7        2.38 ±  7%  perf-profile.children.cycles-pp.svc_recv
      1.79 ±  6%      +0.8        2.55 ±  9%  perf-profile.children.cycles-pp.tcp_sendpage
      1.39 ±  5%      +0.8        2.16 ± 16%  perf-profile.children.cycles-pp.ext4_buffered_write_iter
      1.82 ±  6%      +0.8        2.59 ±  9%  perf-profile.children.cycles-pp.kernel_sendpage
      1.81 ±  6%      +0.8        2.58 ±  9%  perf-profile.children.cycles-pp.inet_sendpage
      1.40 ±  5%      +0.8        2.17 ± 16%  perf-profile.children.cycles-pp.ext4_file_write_iter
      1.40 ±  5%      +0.8        2.17 ± 16%  perf-profile.children.cycles-pp.do_iter_readv_writev
      1.82 ±  6%      +0.8        2.60 ±  9%  perf-profile.children.cycles-pp.svc_send_common
      1.44 ±  5%      +0.8        2.22 ± 16%  perf-profile.children.cycles-pp.nfsd_vfs_write
      1.42 ±  5%      +0.8        2.20 ± 16%  perf-profile.children.cycles-pp.do_iter_write
      1.82 ±  6%      +0.8        2.61 ±  9%  perf-profile.children.cycles-pp.svc_sendto
      1.83 ±  6%      +0.8        2.62 ±  9%  perf-profile.children.cycles-pp.svc_tcp_sendto
      1.49 ±  5%      +0.8        2.30 ± 15%  perf-profile.children.cycles-pp.nfsd4_write
      2.33 ±  6%      +0.8        3.14 ±  2%  perf-profile.children.cycles-pp.xprt_transmit
      2.35 ±  6%      +0.8        3.17 ±  5%  perf-profile.children.cycles-pp.ip6_finish_output2
      1.60 ±  4%      +0.8        2.43 ±  6%  perf-profile.children.cycles-pp.stack_trace_save_tsk
      2.35 ±  6%      +0.8        3.18 ±  2%  perf-profile.children.cycles-pp.call_transmit
      2.53 ±  6%      +0.9        3.42 ±  5%  perf-profile.children.cycles-pp.ip6_output
      2.05 ±  6%      +0.9        2.96 ±  9%  perf-profile.children.cycles-pp.svc_send
      2.65 ±  6%      +0.9        3.58 ±  5%  perf-profile.children.cycles-pp.ip6_xmit
      1.87 ±  4%      +0.9        2.81 ±  6%  perf-profile.children.cycles-pp.__account_scheduler_latency
      2.77 ±  6%      +1.0        3.77 ±  5%  perf-profile.children.cycles-pp.inet6_csk_xmit
      2.99 ±  6%      +1.0        4.00 ±  4%  perf-profile.children.cycles-pp.tcp_write_xmit
      2.99 ±  6%      +1.0        4.00 ±  4%  perf-profile.children.cycles-pp.__tcp_push_pending_frames
      3.04 ±  6%      +1.1        4.15 ±  5%  perf-profile.children.cycles-pp.__tcp_transmit_skb
      2.30 ±  3%      +1.1        3.42 ±  7%  perf-profile.children.cycles-pp.enqueue_entity
      2.42 ±  3%      +1.2        3.59 ±  7%  perf-profile.children.cycles-pp.enqueue_task_fair
      2.45 ±  4%      +1.2        3.62 ±  7%  perf-profile.children.cycles-pp.ttwu_do_activate
      2.44 ±  4%      +1.2        3.61 ±  7%  perf-profile.children.cycles-pp.activate_task
      2.64 ±  2%      +1.2        3.87 ±  6%  perf-profile.children.cycles-pp.try_to_wake_up
      4.22 ±  7%      +1.5        5.74 ±  4%  perf-profile.children.cycles-pp.do_syscall_64
      4.23 ±  7%      +1.5        5.75 ±  4%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
      3.31 ±  8%      +1.6        4.87 ±  3%  perf-profile.children.cycles-pp.rpc_async_schedule
      4.47 ±  6%      +1.8        6.32 ±  2%  perf-profile.children.cycles-pp.__rpc_execute
      3.95 ±  4%      +2.1        6.03 ± 14%  perf-profile.children.cycles-pp.nfsd4_proc_compound
      5.18 ±  5%      +2.2        7.36 ±  3%  perf-profile.children.cycles-pp.process_one_work
      4.24 ±  4%      +2.2        6.45 ± 14%  perf-profile.children.cycles-pp.nfsd_dispatch
      4.55 ±  4%      +2.3        6.89 ± 14%  perf-profile.children.cycles-pp.svc_process_common
      4.56 ±  4%      +2.3        6.91 ± 14%  perf-profile.children.cycles-pp.svc_process
      5.81 ±  5%      +2.4        8.18 ±  2%  perf-profile.children.cycles-pp.worker_thread
      8.25 ±  4%      +4.0       12.24 ± 11%  perf-profile.children.cycles-pp.nfsd
     15.63 ±  3%      +7.0       22.61 ±  7%  perf-profile.children.cycles-pp.kthread
     15.63 ±  3%      +7.0       22.61 ±  7%  perf-profile.children.cycles-pp.ret_from_fork
     59.98 ±  2%      -9.0       51.00 ±  5%  perf-profile.self.cycles-pp.intel_idle
      0.21 ± 23%      -0.1        0.12 ± 19%  perf-profile.self.cycles-pp.run_timer_softirq
      0.13 ± 12%      -0.1        0.04 ± 59%  perf-profile.self.cycles-pp.load_balance
      0.18 ± 11%      -0.1        0.11 ± 22%  perf-profile.self.cycles-pp.rcu_core
      0.29 ±  7%      -0.1        0.22 ± 16%  perf-profile.self.cycles-pp.find_busiest_group
      0.08 ± 32%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.pm_qos_read_value
      0.05 ±  8%      +0.0        0.07 ±  7%  perf-profile.self.cycles-pp.inet6_csk_route_socket
      0.05 ±  8%      +0.0        0.07 ± 11%  perf-profile.self.cycles-pp.pick_next_entity
      0.06 ± 11%      +0.0        0.08 ± 13%  perf-profile.self.cycles-pp.ext4_mark_iloc_dirty
      0.06 ± 13%      +0.0        0.09 ±  5%  perf-profile.self.cycles-pp.tcp_clean_rtx_queue
      0.06 ± 14%      +0.0        0.08 ± 10%  perf-profile.self.cycles-pp.crc32c_pcl_intel_update
      0.07 ± 11%      +0.0        0.10 ± 17%  perf-profile.self.cycles-pp.decode_getfattr_attrs
      0.08 ± 13%      +0.0        0.11 ± 11%  perf-profile.self.cycles-pp.update_curr
      0.08 ± 10%      +0.0        0.11 ±  9%  perf-profile.self.cycles-pp.__update_load_avg_se
      0.09 ±  7%      +0.0        0.12 ±  5%  perf-profile.self.cycles-pp.process_one_work
      0.06 ±  9%      +0.0        0.09 ± 19%  perf-profile.self.cycles-pp._raw_read_lock
      0.07 ± 13%      +0.0        0.10 ± 18%  perf-profile.self.cycles-pp.__jbd2_journal_file_buffer
      0.09 ± 13%      +0.0        0.12 ± 12%  perf-profile.self.cycles-pp.memcpy_from_page
      0.12 ± 14%      +0.0        0.16 ±  4%  perf-profile.self.cycles-pp.memset_erms
      0.06 ±  9%      +0.0        0.09 ± 19%  perf-profile.self.cycles-pp.memcmp
      0.07 ± 22%      +0.0        0.11 ± 19%  perf-profile.self.cycles-pp.tcp_write_xmit
      0.16 ±  6%      +0.0        0.20 ±  4%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.04 ± 58%      +0.0        0.08 ± 12%  perf-profile.self.cycles-pp.check_preempt_curr
      0.07 ± 22%      +0.0        0.11 ± 15%  perf-profile.self.cycles-pp.__might_sleep
      0.03 ±100%      +0.0        0.07 ±  7%  perf-profile.self.cycles-pp.__jbd2_journal_temp_unlink_buffer
      0.10 ± 15%      +0.0        0.14 ± 10%  perf-profile.self.cycles-pp.update_load_avg
      0.04 ± 58%      +0.0        0.08 ± 15%  perf-profile.self.cycles-pp.ip6_finish_output2
      0.09 ± 17%      +0.0        0.14 ± 13%  perf-profile.self.cycles-pp.tcp_sendmsg_locked
      0.08 ± 16%      +0.0        0.12 ± 15%  perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      0.06 ± 11%      +0.0        0.10 ± 15%  perf-profile.self.cycles-pp.__netif_receive_skb_core
      0.10 ±  4%      +0.0        0.14 ± 12%  perf-profile.self.cycles-pp.do_syscall_64
      0.08 ±  6%      +0.0        0.12 ± 14%  perf-profile.self.cycles-pp.prepare_creds
      0.09 ± 16%      +0.0        0.14 ±  8%  perf-profile.self.cycles-pp.newidle_balance
      0.01 ±173%      +0.0        0.06 ± 11%  perf-profile.self.cycles-pp.__alloc_skb
      0.03 ±100%      +0.0        0.07 ±  5%  perf-profile.self.cycles-pp.xdr_reserve_space
      0.12 ± 15%      +0.1        0.17 ± 14%  perf-profile.self.cycles-pp.enqueue_task_fair
      0.18 ±  7%      +0.1        0.23 ± 11%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.01 ±173%      +0.1        0.07 ± 12%  perf-profile.self.cycles-pp.___slab_alloc
      0.10 ±  5%      +0.1        0.15 ±  8%  perf-profile.self.cycles-pp.dequeue_task_fair
      0.06 ±  9%      +0.1        0.11 ± 11%  perf-profile.self.cycles-pp.resched_curr
      0.03 ±100%      +0.1        0.08 ± 19%  perf-profile.self.cycles-pp.get_page_from_freelist
      0.03 ±100%      +0.1        0.08 ± 19%  perf-profile.self.cycles-pp.nfsd4_encode_fattr
      0.02 ±173%      +0.1        0.07 ± 11%  perf-profile.self.cycles-pp.finish_task_switch
      0.12 ± 11%      +0.1        0.18 ± 17%  perf-profile.self.cycles-pp.tcp_recvmsg
      0.12 ± 17%      +0.1        0.18 ±  7%  perf-profile.self.cycles-pp.kfree
      0.05 ± 60%      +0.1        0.11 ± 10%  perf-profile.self.cycles-pp.menu_reflect
      0.15 ±  5%      +0.1        0.21 ± 15%  perf-profile.self.cycles-pp.enqueue_entity
      0.08 ±  8%      +0.1        0.14 ±  8%  perf-profile.self.cycles-pp.stack_trace_consume_entry_nosched
      0.00            +0.1        0.06 ± 11%  perf-profile.self.cycles-pp.__wake_up_common
      0.00            +0.1        0.06 ± 20%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.11 ± 19%      +0.1        0.17 ±  9%  perf-profile.self.cycles-pp.__account_scheduler_latency
      0.21 ±  8%      +0.1        0.27 ±  8%  perf-profile.self.cycles-pp.try_to_wake_up
      0.14 ±  8%      +0.1        0.20 ±  4%  perf-profile.self.cycles-pp._raw_spin_lock_bh
      0.11 ± 11%      +0.1        0.17 ±  2%  perf-profile.self.cycles-pp.set_next_entity
      0.11 ± 12%      +0.1        0.17 ± 11%  perf-profile.self.cycles-pp.mutex_spin_on_owner
      0.25 ± 12%      +0.1        0.32 ±  9%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.01 ±173%      +0.1        0.08 ± 19%  perf-profile.self.cycles-pp.percpu_counter_add_batch
      0.14 ± 15%      +0.1        0.21 ± 10%  perf-profile.self.cycles-pp.io_serial_in
      0.14 ± 13%      +0.1        0.21 ±  7%  perf-profile.self.cycles-pp.orc_find
      0.00            +0.1        0.07 ± 10%  perf-profile.self.cycles-pp.in_sched_functions
      0.39 ±  4%      +0.1        0.46 ±  3%  perf-profile.self.cycles-pp.do_idle
      0.28 ±  8%      +0.1        0.35 ±  7%  perf-profile.self.cycles-pp.native_sched_clock
      0.01 ±173%      +0.1        0.09 ± 31%  perf-profile.self.cycles-pp.timekeeping_max_deferment
      0.13 ± 16%      +0.1        0.21 ± 14%  perf-profile.self.cycles-pp.jbd2_journal_commit_transaction
      0.22 ±  7%      +0.1        0.31 ±  7%  perf-profile.self.cycles-pp.___might_sleep
      0.34 ± 10%      +0.1        0.42 ±  7%  perf-profile.self.cycles-pp.native_irq_return_iret
      0.15 ±  4%      +0.1        0.24 ±  6%  perf-profile.self.cycles-pp.stack_trace_save_tsk
      0.22 ± 11%      +0.1        0.31 ± 12%  perf-profile.self.cycles-pp.select_task_rq_fair
      0.17 ±  2%      +0.1        0.27 ±  4%  perf-profile.self.cycles-pp.__slab_free
      0.17 ±  2%      +0.1        0.27 ±  6%  perf-profile.self.cycles-pp.__switch_to
      0.19 ±  8%      +0.1        0.29 ±  6%  perf-profile.self.cycles-pp.svc_recv
      0.23 ±  6%      +0.1        0.34 ±  7%  perf-profile.self.cycles-pp.__switch_to_asm
      0.41 ± 16%      +0.1        0.52 ±  4%  perf-profile.self.cycles-pp.native_write_msr
      0.22 ± 14%      +0.1        0.34 ±  8%  perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
      0.38 ±  2%      +0.2        0.53 ±  4%  perf-profile.self.cycles-pp.__sched_text_start
      0.33 ±  6%      +0.2        0.48 ±  7%  perf-profile.self.cycles-pp.memcpy_erms
      0.31 ±  6%      +0.2        0.48 ±  9%  perf-profile.self.cycles-pp.__orc_find
      0.41 ±  5%      +0.2        0.59 ± 11%  perf-profile.self.cycles-pp.unwind_next_frame
      0.43 ±  6%      +0.2        0.66 ±  6%  perf-profile.self.cycles-pp.read_tsc
      1.04 ±  7%      +0.4        1.43 ±  7%  perf-profile.self.cycles-pp._raw_spin_lock
      0.35 ± 12%      +0.4        0.77 ± 32%  perf-profile.self.cycles-pp.poll_idle


                                                                                
                              fsmark.time.system_time                           
                                                                                
  300 +-+-------------------------------------------------------------------+   
      |.++.+.++.+.++.+.++   ++.+.++.+.++.+.++.+.++.+.++.+.++.+.++.+.++.+.++.|   
  250 +-+                                                                   |   
      O OO O O  O  O O    O OO O OO O OO O OO O OO                          |   
      |       O        OO                                                   |   
  200 +-+                                                                   |   
      |                                                                     |   
  150 +-+                                                                   |   
      |                                                                     |   
  100 +-+                                                                   |   
      |                                                                     |   
      |                                                                     |   
   50 +-+                                                                   |   
      |                                                                     |   
    0 +-+---------O---------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                             fsmark.time.elapsed_time                           
                                                                                
  800 +-+-------------------------------------------------------------------+   
      |                                                                     |   
  700 +-++.+.++.+.++.+.++.+.++.+.++.+.++.+.++.+.++.+.++.+.++.+.++.+.++.+.++.|   
  600 O-OO O OO O  O O OO O OO O OO O OO O OO O OO                          |   
      |                                                                     |   
  500 +-+                                                                   |   
      |                                                                     |   
  400 +-+                                                                   |   
      |                                                                     |   
  300 +-+                                                                   |   
  200 +-+                                                                   |   
      |                                                                     |   
  100 +-+                                                                   |   
      |                                                                     |   
    0 +-+---------O---------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                           fsmark.time.elapsed_time.max                         
                                                                                
  800 +-+-------------------------------------------------------------------+   
      |                                                                     |   
  700 +-++.+.++.+.++.+.++.+.++.+.++.+.++.+.++.+.++.+.++.+.++.+.++.+.++.+.++.|   
  600 O-OO O OO O  O O OO O OO O OO O OO O OO O OO                          |   
      |                                                                     |   
  500 +-+                                                                   |   
      |                                                                     |   
  400 +-+                                                                   |   
      |                                                                     |   
  300 +-+                                                                   |   
  200 +-+                                                                   |   
      |                                                                     |   
  100 +-+                                                                   |   
      |                                                                     |   
    0 +-+---------O---------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                               fsmark.files_per_sec                             
                                                                                
  4500 +-+----------O--O---------O---O-----O------O-------------------------+   
       O OO O OO O    O  O OO O O  O  O O O  OO O                           |   
  4000 +-++.+.++.+.++.++.+.++.+.++.+.++.+.++.++.+.++.+.++.+.++.+.++.++.+.++.|   
  3500 +-+                                                                  |   
       |                                                                    |   
  3000 +-+                                                                  |   
  2500 +-+                                                                  |   
       |                                                                    |   
  2000 +-+                                                                  |   
  1500 +-+                                                                  |   
       |                                                                    |   
  1000 +-+                                                                  |   
   500 +-+                                                                  |   
       |                                                                    |   
     0 +-+---------O--------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                  fsmark.app_overhead                           
                                                                                
    5e+07 +-+---------------------------------------------------------------+   
  4.5e+07 +-++.++.++. .++.++.+ .+.++.+ .++.+.++.+ .++.+.++.++.++.+.++.++.++.|   
          O OO OO    +        +   O  O+          +                          |   
    4e+07 +-+     OO   OO OO OO O  O  O OO O OO OO O                        |   
  3.5e+07 +-+                                                               |   
          |                                                                 |   
    3e+07 +-+                                                               |   
  2.5e+07 +-+                                                               |   
    2e+07 +-+                                                               |   
          |                                                                 |   
  1.5e+07 +-+                                                               |   
    1e+07 +-+                                                               |   
          |                                                                 |   
    5e+06 +-+                                                               |   
        0 +-+--------O------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                          nfsstat.Client.nfs.v4.getattr                         
                                                                                
  35 +-+--------------------------------------------------------------------+   
     |                                                +. +.                 |   
  30 +-++.+.        +.        +.+.    .+.+   .++.+.+ +  +  +. +.+.        +.|   
     |.+    ++.+.+.+  +.++.+.+    ++.+    +.+       +        +    +.++.+.+  |   
  25 O-OO O OO O   OO O OO O OO O OO O O OO O OO O                          |   
     |                                                                      |   
  20 +-+                                                                    |   
     |                                                                      |   
  15 +-+                                                                    |   
     |                                                                      |   
  10 +-+                                                                    |   
     |                                                                      |   
   5 +-+                                                                    |   
     |                                                                      |   
   0 +-+---------O----------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample

***************************************************************************************************
lkp-hsw-ep4: 72 threads Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz with 256G memory
=========================================================================================
bs/compiler/cpufreq_governor/disk/fs2/fs/ioengine/kconfig/nr_task/rootfs/runtime/rw/tbox_group/test_size/testcase/ucode:
  4k/gcc-7/performance/1SSD/nfsv4/ext4/sync/x86_64-rhel-7.6/8/debian-x86_64-2019-11-14.cgz/300s/write/lkp-hsw-ep4/512g/fio-basic/0x43

commit: 
  706ae86d1a ("intel_idle: Add module parameter to prevent ACPI _CST from being used")
  bf5a9506c5 ("intel_idle: Use ACPI _CST on server systems")

706ae86d1a0a737a bf5a9506c59abd8e06049ccd2b6 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          3:4          -75%            :4     dmesg.WARNING:at#for_ip_interrupt_entry/0x
         %stddev     %change         %stddev
             \          |                \  
      5.47 ±  5%      +0.4        5.91 ±  5%  fio.latency_10us%
      1.42 ± 16%      -1.0        0.39 ±  3%  fio.latency_20us%
      2.68 ± 37%      -2.7        0.01        fio.latency_2us%
     90.26            +3.4       93.67        fio.latency_4us%
      0.16 ± 57%      -0.1        0.02 ± 18%  fio.latency_50us%
   2840881 ±  3%      +5.5%    2996400 ±  3%  fio.time.voluntary_context_switches
      4864           -10.9%       4336        fio.write_clat_95%_us
     13600 ±  4%     -47.6%       7120        fio.write_clat_99%_us
    475.75 ±155%    +225.2%       1547 ± 30%  meminfo.Mlocked
      0.04 ± 51%      -0.0        0.01 ± 12%  mpstat.cpu.all.soft%
 1.763e+09 ±  7%     -15.1%  1.496e+09 ±  6%  nfsstat.Server.io.write
    205.25 ±151%    +321.8%     865.75 ± 37%  numa-meminfo.node0.Mlocked
      7025 ±  7%     -14.0%       6038 ±  5%  numa-meminfo.node1.KernelStack
     51.50 ±150%    +319.9%     216.25 ± 37%  numa-vmstat.node0.nr_mlock
      7021 ±  7%     -14.0%       6037 ±  5%  numa-vmstat.node1.nr_kernel_stack
      2881 ±  6%     +19.2%       3435 ±  5%  slabinfo.eventpoll_pwq.active_objs
      2881 ±  6%     +19.2%       3435 ±  5%  slabinfo.eventpoll_pwq.num_objs
      5160 ± 25%     -34.3%       3392 ±  8%  slabinfo.skbuff_head_cache.active_objs
      5220 ± 25%     -35.0%       3392 ±  8%  slabinfo.skbuff_head_cache.num_objs
    837037           -12.9%     729056 ± 13%  proc-vmstat.nr_dirty
     13858            -2.4%      13522        proc-vmstat.nr_kernel_stack
    118.75 ±155%    +225.3%     386.25 ± 30%  proc-vmstat.nr_mlock
     87659            -5.2%      83066 ±  4%  proc-vmstat.nr_slab_unreclaimable
     10677 ± 41%     -80.3%       2099 ±106%  proc-vmstat.numa_hint_faults
 9.777e+08 ± 56%   +2354.5%    2.4e+10 ± 16%  cpuidle.C1E.time
   5831691 ± 92%    +792.2%   52027458 ± 16%  cpuidle.C1E.usage
 7.705e+09 ± 74%    -100.0%       0.00        cpuidle.C3.time
  18972999 ± 68%    -100.0%       0.00        cpuidle.C3.usage
 1.221e+10 ± 47%    -100.0%       0.00        cpuidle.C6.time
  15556061 ± 40%    -100.0%       0.00        cpuidle.C6.usage
      2.74 ±  3%     -13.1%       2.38 ± 13%  iostat.cpu.system
      6884           -12.3%       6040 ± 14%  iostat.nvme0n1.avgqu-sz
    950.80           -12.5%     832.10 ± 14%  iostat.nvme0n1.await
     92.35           -10.2%      82.97 ± 13%  iostat.nvme0n1.util
    950.80           -12.5%     832.10 ± 14%  iostat.nvme0n1.w_await
     17.59 ±  5%     -16.4%      14.70 ± 11%  iostat.nvme0n1.wrqm/s
    162150 ±130%    +150.2%     405670 ± 89%  sched_debug.cfs_rq:/.load.avg
    161902 ±130%    +150.0%     404718 ± 89%  sched_debug.cfs_rq:/.runnable_weight.avg
      0.15 ±  5%     -16.2%       0.13 ±  7%  sched_debug.cpu.nr_uninterruptible.avg
    -18.79           +26.6%     -23.78        sched_debug.cpu.nr_uninterruptible.min
      9616 ± 10%     +25.4%      12058 ± 12%  sched_debug.cpu.ttwu_local.max
      2020 ±  8%     +22.4%       2473 ±  9%  sched_debug.cpu.ttwu_local.stddev
     15278 ± 18%     -35.6%       9839 ± 19%  softirqs.CPU24.NET_RX
     15392 ± 19%     -33.0%      10317 ± 10%  softirqs.CPU26.NET_RX
     16316 ± 25%     -42.5%       9381 ± 40%  softirqs.CPU30.NET_RX
     17458 ± 10%     -35.9%      11197 ± 22%  softirqs.CPU39.NET_RX
     13289 ± 32%     +46.4%      19461 ± 13%  softirqs.CPU44.NET_RX
     14350 ± 21%     -34.9%       9341 ±  5%  softirqs.CPU58.NET_RX
     14579 ± 15%     -43.6%       8222 ± 39%  softirqs.CPU62.NET_RX
    152.75 ±  3%     -21.1%     120.50 ±  8%  turbostat.Avg_MHz
      6.02 ±  4%      -0.8        5.20 ±  4%  turbostat.Busy%
   5831092 ± 92%    +792.1%   52019885 ± 16%  turbostat.C1E
      4.39 ± 57%     +90.2       94.59        turbostat.C1E%
  18971941 ± 68%    -100.0%       0.00        turbostat.C3
     34.62 ± 74%     -34.6        0.00        turbostat.C3%
  15546645 ± 40%    -100.0%       0.00        turbostat.C6
     54.75 ± 46%     -54.7        0.00        turbostat.C6%
     47.98 ± 23%     +97.6%      94.80        turbostat.CPU%c1
     16.91 ± 50%    -100.0%       0.00        turbostat.CPU%c3
     29.08 ± 61%    -100.0%       0.00        turbostat.CPU%c6
      7.73 ± 59%    -100.0%       0.00        turbostat.Pkg%pc2
      0.98 ± 73%    -100.0%       0.00        turbostat.Pkg%pc3
     15.62 ±  3%      +5.6%      16.50        turbostat.RAMWatt
     20.11 ± 13%     -72.2%       5.59 ±  2%  perf-stat.i.MPKI
      2.15 ± 21%      -1.0        1.18 ± 13%  perf-stat.i.branch-miss-rate%
  16670364 ± 17%     -44.4%    9268690 ±  8%  perf-stat.i.branch-misses
      8.11 ± 11%     +10.3       18.38 ±  7%  perf-stat.i.cache-miss-rate%
   8334917 ±  3%     -11.9%    7344584 ± 10%  perf-stat.i.cache-misses
  78617949 ±  9%     -58.0%   33023951 ± 11%  perf-stat.i.cache-references
      2.62 ±  7%     -22.8%       2.02 ± 11%  perf-stat.i.cpi
 1.117e+10 ±  4%     -28.3%  8.013e+09 ± 10%  perf-stat.i.cpu-cycles
      0.43 ± 21%      -0.2        0.19 ± 46%  perf-stat.i.dTLB-load-miss-rate%
   5177186 ± 26%     -45.9%    2801702 ± 47%  perf-stat.i.dTLB-load-misses
  1.65e+09 ±  2%     -11.5%   1.46e+09 ± 12%  perf-stat.i.dTLB-loads
      0.13 ±  8%      -0.1        0.06 ± 16%  perf-stat.i.dTLB-store-miss-rate%
   1281525 ±  7%     -54.6%     582052 ± 20%  perf-stat.i.dTLB-store-misses
 1.036e+09 ±  4%     -29.7%  7.279e+08 ± 11%  perf-stat.i.dTLB-stores
     66.25 ± 10%     -41.2       25.03 ± 13%  perf-stat.i.iTLB-load-miss-rate%
   1916947 ± 13%     -57.5%     813866 ± 13%  perf-stat.i.iTLB-load-misses
    933037 ± 25%     +90.1%    1773776        perf-stat.i.iTLB-loads
      2854 ± 15%    +480.7%      16577 ±  4%  perf-stat.i.instructions-per-iTLB-miss
      0.44 ±  5%     +26.9%       0.56 ±  7%  perf-stat.i.ipc
      3121            -2.3%       3048        perf-stat.i.minor-faults
   3206042 ±  8%     -18.9%    2600446 ± 10%  perf-stat.i.node-load-misses
      3121            -2.3%       3048        perf-stat.i.page-faults
     14.33 ± 11%     -54.1%       6.58        perf-stat.overall.MPKI
      1.49 ± 17%      -0.6        0.91 ±  3%  perf-stat.overall.branch-miss-rate%
     10.74 ± 12%     +11.6       22.30 ±  3%  perf-stat.overall.cache-miss-rate%
      2.03 ±  5%     -21.3%       1.60 ±  2%  perf-stat.overall.cpi
      1342 ±  6%     -18.7%       1091 ±  2%  perf-stat.overall.cycles-between-cache-misses
      0.31 ± 28%      -0.1        0.19 ± 41%  perf-stat.overall.dTLB-load-miss-rate%
      0.12 ±  6%      -0.0        0.08 ± 13%  perf-stat.overall.dTLB-store-miss-rate%
     67.50 ±  9%     -36.2       31.32 ±  8%  perf-stat.overall.iTLB-load-miss-rate%
      2924 ± 14%    +111.5%       6185 ±  3%  perf-stat.overall.instructions-per-iTLB-miss
      0.49 ±  5%     +26.7%       0.63 ±  2%  perf-stat.overall.ipc
  16621101 ± 17%     -44.4%    9245588 ±  8%  perf-stat.ps.branch-misses
   8313227 ±  3%     -11.9%    7324913 ± 10%  perf-stat.ps.cache-misses
  78386575 ±  9%     -58.0%   32934256 ± 11%  perf-stat.ps.cache-references
 1.114e+10 ±  4%     -28.2%  7.991e+09 ±  9%  perf-stat.ps.cpu-cycles
   5160694 ± 26%     -45.9%    2794047 ± 47%  perf-stat.ps.dTLB-load-misses
 1.645e+09 ±  2%     -11.5%  1.456e+09 ± 12%  perf-stat.ps.dTLB-loads
   1277482 ±  7%     -54.6%     580540 ± 20%  perf-stat.ps.dTLB-store-misses
 1.033e+09 ±  4%     -29.7%   7.26e+08 ± 11%  perf-stat.ps.dTLB-stores
   1910701 ± 13%     -57.5%     811716 ± 13%  perf-stat.ps.iTLB-load-misses
    930257 ± 25%     +90.1%    1768609        perf-stat.ps.iTLB-loads
      3115            -2.3%       3043        perf-stat.ps.minor-faults
   3198179 ±  8%     -18.9%    2593793 ± 10%  perf-stat.ps.node-load-misses
      3115            -2.3%       3043        perf-stat.ps.page-faults
     93074 ± 17%     +41.1%     131332 ± 14%  interrupts.226:IR-PCI-MSI.68681730-edge.nvme0q2
     23811 ± 74%     -65.4%       8244 ±100%  interrupts.239:IR-PCI-MSI.68681743-edge.nvme0q15
    152.75           +29.1%     197.25 ± 20%  interrupts.59:IR-PCI-MSI.1572884-edge.eth0-TxRx-20
    152.50           +21.0%     184.50 ± 15%  interrupts.61:IR-PCI-MSI.1572886-edge.eth0-TxRx-22
    152.50           +15.1%     175.50 ± 15%  interrupts.63:IR-PCI-MSI.1572888-edge.eth0-TxRx-24
    152.00           +18.6%     180.25 ± 14%  interrupts.64:IR-PCI-MSI.1572889-edge.eth0-TxRx-25
    153.25           +19.6%     183.25 ± 16%  interrupts.74:IR-PCI-MSI.1572897-edge.eth0-TxRx-33
      1292 ± 35%     +95.6%       2527 ± 29%  interrupts.CPU19.RES:Rescheduling_interrupts
     93074 ± 17%     +41.1%     131332 ± 14%  interrupts.CPU21.226:IR-PCI-MSI.68681730-edge.nvme0q2
    952.00 ± 45%    +293.3%       3744 ± 58%  interrupts.CPU22.RES:Rescheduling_interrupts
    688.50 ± 27%    +190.8%       2002 ± 48%  interrupts.CPU25.RES:Rescheduling_interrupts
    815.25 ± 31%     -61.4%     314.50 ±  3%  interrupts.CPU29.NMI:Non-maskable_interrupts
    815.25 ± 31%     -61.4%     314.50 ±  3%  interrupts.CPU29.PMI:Performance_monitoring_interrupts
    664.00 ± 28%     -48.6%     341.25 ± 45%  interrupts.CPU34.NMI:Non-maskable_interrupts
    664.00 ± 28%     -48.6%     341.25 ± 45%  interrupts.CPU34.PMI:Performance_monitoring_interrupts
      1473 ± 82%     -64.0%     530.50 ±104%  interrupts.CPU34.RES:Rescheduling_interrupts
    489.25 ± 40%    +150.3%       1224 ± 35%  interrupts.CPU37.RES:Rescheduling_interrupts
    152.75           +29.1%     197.25 ± 20%  interrupts.CPU38.59:IR-PCI-MSI.1572884-edge.eth0-TxRx-20
    152.50           +21.0%     184.50 ± 15%  interrupts.CPU40.61:IR-PCI-MSI.1572886-edge.eth0-TxRx-22
    152.50           +15.1%     175.50 ± 15%  interrupts.CPU42.63:IR-PCI-MSI.1572888-edge.eth0-TxRx-24
    152.00           +18.6%     180.25 ± 14%  interrupts.CPU43.64:IR-PCI-MSI.1572889-edge.eth0-TxRx-25
    677.25 ± 27%     -56.4%     295.50 ± 71%  interrupts.CPU50.NMI:Non-maskable_interrupts
    677.25 ± 27%     -56.4%     295.50 ± 71%  interrupts.CPU50.PMI:Performance_monitoring_interrupts
    153.25           +19.6%     183.25 ± 16%  interrupts.CPU51.74:IR-PCI-MSI.1572897-edge.eth0-TxRx-33
    972.75 ± 53%     -66.3%     328.00 ± 48%  interrupts.CPU57.NMI:Non-maskable_interrupts
    972.75 ± 53%     -66.3%     328.00 ± 48%  interrupts.CPU57.PMI:Performance_monitoring_interrupts
    674.75 ± 24%     -63.5%     246.25 ± 52%  interrupts.CPU58.NMI:Non-maskable_interrupts
    674.75 ± 24%     -63.5%     246.25 ± 52%  interrupts.CPU58.PMI:Performance_monitoring_interrupts
    518.75 ± 20%     -54.9%     233.75 ± 48%  interrupts.CPU59.NMI:Non-maskable_interrupts
    518.75 ± 20%     -54.9%     233.75 ± 48%  interrupts.CPU59.PMI:Performance_monitoring_interrupts
      1708 ± 51%     -85.1%     254.50 ± 42%  interrupts.CPU59.RES:Rescheduling_interrupts
    718.00 ± 45%     -63.2%     264.50 ± 62%  interrupts.CPU61.RES:Rescheduling_interrupts
    622.25 ± 23%     -57.3%     266.00 ± 31%  interrupts.CPU62.NMI:Non-maskable_interrupts
    622.25 ± 23%     -57.3%     266.00 ± 31%  interrupts.CPU62.PMI:Performance_monitoring_interrupts
    934.25 ± 41%     -64.4%     333.00 ± 83%  interrupts.CPU62.RES:Rescheduling_interrupts
     23811 ± 74%     -65.4%       8244 ±100%  interrupts.CPU63.239:IR-PCI-MSI.68681743-edge.nvme0q15
    544.00 ± 16%     -27.1%     396.75 ± 32%  interrupts.CPU64.NMI:Non-maskable_interrupts
    544.00 ± 16%     -27.1%     396.75 ± 32%  interrupts.CPU64.PMI:Performance_monitoring_interrupts
    700.25 ± 44%     -63.4%     256.00 ± 28%  interrupts.CPU66.NMI:Non-maskable_interrupts
    700.25 ± 44%     -63.4%     256.00 ± 28%  interrupts.CPU66.PMI:Performance_monitoring_interrupts
      1385 ± 63%     -65.0%     485.50 ±112%  interrupts.CPU66.RES:Rescheduling_interrupts
    753.00 ± 27%     -58.6%     311.50 ± 55%  interrupts.CPU67.NMI:Non-maskable_interrupts
    753.00 ± 27%     -58.6%     311.50 ± 55%  interrupts.CPU67.PMI:Performance_monitoring_interrupts
    747.00 ± 52%     -64.9%     262.00 ± 62%  interrupts.CPU70.NMI:Non-maskable_interrupts
    747.00 ± 52%     -64.9%     262.00 ± 62%  interrupts.CPU70.PMI:Performance_monitoring_interrupts
    985.00 ± 60%     -62.6%     368.00 ± 21%  interrupts.CPU9.NMI:Non-maskable_interrupts
    985.00 ± 60%     -62.6%     368.00 ± 21%  interrupts.CPU9.PMI:Performance_monitoring_interrupts
     47824 ±  5%     -29.7%      33623 ±  9%  interrupts.NMI:Non-maskable_interrupts
     47824 ±  5%     -29.7%      33623 ±  9%  interrupts.PMI:Performance_monitoring_interrupts
     71.08 ±  6%     -11.8       59.31 ±  9%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
     71.08 ±  6%     -11.8       59.31 ±  9%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
     71.05 ±  6%     -11.8       59.28 ±  9%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     71.92 ±  6%     -11.5       60.38 ±  9%  perf-profile.calltrace.cycles-pp.secondary_startup_64
     52.53 ±  8%     -11.2       41.34 ± 10%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     65.93 ±  6%     -10.9       55.02 ± 10%  perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     65.16 ±  6%     -10.7       54.47 ± 10%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
      3.64 ±  8%      -0.7        2.95 ± 10%  perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      0.71 ± 17%      -0.3        0.41 ± 58%  perf-profile.calltrace.cycles-pp.irq_enter.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      0.66 ± 14%      +0.3        0.93 ±  8%  perf-profile.calltrace.cycles-pp.mpage_prepare_extent_to_map.ext4_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range
      0.69 ± 14%      +0.3        0.95 ±  8%  perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range.ext4_sync_file.nfsd_commit
      0.69 ± 14%      +0.3        0.95 ±  8%  perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.file_write_and_wait_range.ext4_sync_file.nfsd_commit.nfsd4_proc_compound
      0.69 ± 14%      +0.3        0.95 ±  8%  perf-profile.calltrace.cycles-pp.ext4_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range.ext4_sync_file
      0.71 ± 16%      +0.3        1.00 ±  9%  perf-profile.calltrace.cycles-pp.nfsd_commit.nfsd4_proc_compound.nfsd_dispatch.svc_process_common.svc_process
      0.71 ± 16%      +0.3        1.00 ±  9%  perf-profile.calltrace.cycles-pp.ext4_sync_file.nfsd_commit.nfsd4_proc_compound.nfsd_dispatch.svc_process_common
      0.71 ± 16%      +0.3        1.00 ±  9%  perf-profile.calltrace.cycles-pp.file_write_and_wait_range.ext4_sync_file.nfsd_commit.nfsd4_proc_compound.nfsd_dispatch
      0.90 ± 21%      +0.3        1.20 ± 10%  perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyin.iov_iter_copy_from_user_atomic.generic_perform_write.ext4_buffered_write_iter
      0.90 ± 20%      +0.3        1.21 ± 10%  perf-profile.calltrace.cycles-pp.copyin.iov_iter_copy_from_user_atomic.generic_perform_write.ext4_buffered_write_iter.ext4_file_write_iter
      0.92 ± 21%      +0.3        1.25 ± 10%  perf-profile.calltrace.cycles-pp.iov_iter_copy_from_user_atomic.generic_perform_write.ext4_buffered_write_iter.ext4_file_write_iter.new_sync_write
      0.78 ± 24%      +0.4        1.15 ± 24%  perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyin.iov_iter_copy_from_user_atomic.generic_perform_write.nfs_file_write
      0.78 ± 24%      +0.4        1.16 ± 24%  perf-profile.calltrace.cycles-pp.copyin.iov_iter_copy_from_user_atomic.generic_perform_write.nfs_file_write.new_sync_write
      0.80 ± 24%      +0.4        1.18 ± 24%  perf-profile.calltrace.cycles-pp.iov_iter_copy_from_user_atomic.generic_perform_write.nfs_file_write.new_sync_write.vfs_write
      0.92 ± 18%      +0.4        1.32 ± 10%  perf-profile.calltrace.cycles-pp.ext4_da_write_end.generic_perform_write.ext4_buffered_write_iter.ext4_file_write_iter.new_sync_write
      0.75 ±  3%      +0.4        1.17 ± 19%  perf-profile.calltrace.cycles-pp.nfs_end_page_writeback.nfs_write_completion.rpc_free_task.rpc_async_release.process_one_work
      1.29 ± 24%      +0.4        1.71 ±  6%  perf-profile.calltrace.cycles-pp.ext4_da_write_begin.generic_perform_write.ext4_buffered_write_iter.ext4_file_write_iter.new_sync_write
      0.13 ±173%      +0.4        0.57 ±  5%  perf-profile.calltrace.cycles-pp.lapic_next_deadline.clockevents_program_event.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      0.14 ±173%      +0.4        0.58 ± 11%  perf-profile.calltrace.cycles-pp.nfs_commit_release_pages.nfs_commit_release.rpc_free_task.rpc_async_release.process_one_work
      0.14 ±173%      +0.4        0.58 ± 11%  perf-profile.calltrace.cycles-pp.nfs_commit_release.rpc_free_task.rpc_async_release.process_one_work.worker_thread
      0.14 ±173%      +0.5        0.66 ± 14%  perf-profile.calltrace.cycles-pp.tick_nohz_irq_exit.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
      1.11 ± 14%      +0.6        1.67 ± 29%  perf-profile.calltrace.cycles-pp.__x64_sys_close.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.11 ± 14%      +0.6        1.67 ± 29%  perf-profile.calltrace.cycles-pp.filp_close.__x64_sys_close.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.11 ± 14%      +0.6        1.67 ± 29%  perf-profile.calltrace.cycles-pp.nfs_wb_all.filp_close.__x64_sys_close.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.32 ±102%      +0.6        0.89 ± 26%  perf-profile.calltrace.cycles-pp.pagecache_get_page.grab_cache_page_write_begin.nfs_write_begin.generic_perform_write.nfs_file_write
      0.32 ±102%      +0.6        0.90 ± 27%  perf-profile.calltrace.cycles-pp.grab_cache_page_write_begin.nfs_write_begin.generic_perform_write.nfs_file_write.new_sync_write
      1.23 ± 14%      +0.6        1.81 ± 20%  perf-profile.calltrace.cycles-pp.memcpy_erms.iov_iter_copy_from_user_atomic.generic_perform_write.ext4_buffered_write_iter.ext4_file_write_iter
      1.50 ±  6%      +0.6        2.09 ± 14%  perf-profile.calltrace.cycles-pp.memcpy_from_page._copy_from_iter_full.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg
      1.24 ± 14%      +0.6        1.84 ± 20%  perf-profile.calltrace.cycles-pp.iov_iter_copy_from_user_atomic.generic_perform_write.ext4_buffered_write_iter.ext4_file_write_iter.do_iter_readv_writev
      0.95 ± 58%      +0.6        1.55 ±  6%  perf-profile.calltrace.cycles-pp.mpage_submit_page.mpage_process_page_bufs.mpage_prepare_extent_to_map.ext4_writepages.do_writepages
      1.53 ±  6%      +0.6        2.13 ± 14%  perf-profile.calltrace.cycles-pp._copy_from_iter_full.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.xs_sendpages
      0.98 ± 58%      +0.6        1.59 ±  6%  perf-profile.calltrace.cycles-pp.mpage_process_page_bufs.mpage_prepare_extent_to_map.ext4_writepages.do_writepages.__filemap_fdatawrite_range
      1.00 ± 20%      +0.6        1.61 ± 23%  perf-profile.calltrace.cycles-pp.nfs_write_end.generic_perform_write.nfs_file_write.new_sync_write.vfs_write
      0.50 ± 58%      +0.6        1.12 ± 33%  perf-profile.calltrace.cycles-pp.block_write_end.generic_write_end.ext4_da_write_end.generic_perform_write.ext4_buffered_write_iter
      0.34 ±102%      +0.6        0.99 ± 28%  perf-profile.calltrace.cycles-pp.nfs_write_begin.generic_perform_write.nfs_file_write.new_sync_write.vfs_write
      1.03 ± 14%      +0.7        1.73 ± 10%  perf-profile.calltrace.cycles-pp.invalidate_mapping_pages.generic_fadvise.ksys_fadvise64_64.__x64_sys_fadvise64.do_syscall_64
      0.94 ± 36%      +0.7        1.64 ± 10%  perf-profile.calltrace.cycles-pp.clockevents_program_event.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
      0.26 ±173%      +0.7        1.00 ± 14%  perf-profile.calltrace.cycles-pp.ktime_get.clockevents_program_event.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      2.03 ±  9%      +0.8        2.82 ± 15%  perf-profile.calltrace.cycles-pp.rpc_async_schedule.process_one_work.worker_thread.kthread.ret_from_fork
      2.03 ±  9%      +0.8        2.82 ± 15%  perf-profile.calltrace.cycles-pp.__rpc_execute.rpc_async_schedule.process_one_work.worker_thread.kthread
      0.86 ± 65%      +0.8        1.67 ± 14%  perf-profile.calltrace.cycles-pp.generic_write_end.ext4_da_write_end.generic_perform_write.ext4_buffered_write_iter.ext4_file_write_iter
      1.77 ± 17%      +0.9        2.72 ±  7%  perf-profile.calltrace.cycles-pp.__x64_sys_fadvise64.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.77 ± 17%      +0.9        2.72 ±  7%  perf-profile.calltrace.cycles-pp.ksys_fadvise64_64.__x64_sys_fadvise64.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.77 ± 17%      +0.9        2.72 ±  7%  perf-profile.calltrace.cycles-pp.generic_fadvise.ksys_fadvise64_64.__x64_sys_fadvise64.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.30 ±101%      +1.0        1.26 ±  7%  perf-profile.calltrace.cycles-pp.ext4_bio_write_page.mpage_submit_page.mpage_process_page_bufs.mpage_prepare_extent_to_map.ext4_writepages
      0.30 ±173%      +1.0        1.29 ± 19%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.rcu_core.__softirqentry_text_start.irq_exit
      1.19 ± 48%      +1.1        2.30 ± 17%  perf-profile.calltrace.cycles-pp.rcu_core.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt
      0.51 ±119%      +1.2        1.68 ± 19%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.rcu_core.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt
      3.23 ± 21%      +1.2        4.42 ±  8%  perf-profile.calltrace.cycles-pp.generic_perform_write.ext4_buffered_write_iter.ext4_file_write_iter.new_sync_write.vfs_write
      3.30 ± 21%      +1.3        4.58 ±  8%  perf-profile.calltrace.cycles-pp.ext4_buffered_write_iter.ext4_file_write_iter.new_sync_write.vfs_write.ksys_write
      3.33 ± 21%      +1.3        4.62 ±  8%  perf-profile.calltrace.cycles-pp.ext4_file_write_iter.new_sync_write.vfs_write.ksys_write.do_syscall_64
      2.46 ± 21%      +1.4        3.89 ± 24%  perf-profile.calltrace.cycles-pp.generic_perform_write.nfs_file_write.new_sync_write.vfs_write.ksys_write
      2.61 ± 20%      +1.5        4.10 ± 24%  perf-profile.calltrace.cycles-pp.nfs_file_write.new_sync_write.vfs_write.ksys_write.do_syscall_64
      5.96 ± 19%      +2.8        8.79 ± 14%  perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      6.19 ± 19%      +3.0        9.15 ± 14%  perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      6.29 ± 19%      +3.0        9.29 ± 14%  perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     10.02 ± 17%      +4.9       14.95 ± 14%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
     10.04 ± 17%      +4.9       14.99 ± 14%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
     71.08 ±  6%     -11.8       59.31 ±  9%  perf-profile.children.cycles-pp.start_secondary
     71.92 ±  6%     -11.5       60.38 ±  9%  perf-profile.children.cycles-pp.secondary_startup_64
     71.92 ±  6%     -11.5       60.38 ±  9%  perf-profile.children.cycles-pp.cpu_startup_entry
     71.95 ±  6%     -11.5       60.41 ±  9%  perf-profile.children.cycles-pp.do_idle
     52.66 ±  8%     -11.2       41.45 ± 10%  perf-profile.children.cycles-pp.intel_idle
     66.67 ±  6%     -10.7       56.01 ±  9%  perf-profile.children.cycles-pp.cpuidle_enter_state
     66.69 ±  6%     -10.7       56.04 ±  9%  perf-profile.children.cycles-pp.cpuidle_enter
      3.71 ±  8%      -0.7        3.00 ± 10%  perf-profile.children.cycles-pp.menu_select
      0.43 ± 29%      -0.3        0.17 ± 13%  perf-profile.children.cycles-pp.cpuidle_governor_latency_req
      0.63 ± 16%      -0.2        0.41 ± 10%  perf-profile.children.cycles-pp.tick_irq_enter
      0.30 ± 30%      -0.2        0.09 ± 17%  perf-profile.children.cycles-pp.enqueue_hrtimer
      0.44 ± 29%      -0.2        0.25 ±  9%  perf-profile.children.cycles-pp.run_rebalance_domains
      0.26 ± 32%      -0.2        0.08 ± 14%  perf-profile.children.cycles-pp.timerqueue_add
      0.42 ± 28%      -0.2        0.24 ±  7%  perf-profile.children.cycles-pp.update_blocked_averages
      0.75 ± 16%      -0.2        0.58 ± 10%  perf-profile.children.cycles-pp.irq_enter
      0.35 ± 21%      -0.1        0.20 ±  2%  perf-profile.children.cycles-pp.run_timer_softirq
      0.26 ± 23%      -0.1        0.11 ±  3%  perf-profile.children.cycles-pp.update_ts_time_stats
      0.25 ± 21%      -0.1        0.12 ±  8%  perf-profile.children.cycles-pp.__remove_hrtimer
      0.23 ± 13%      -0.1        0.11 ± 11%  perf-profile.children.cycles-pp.nr_iowait_cpu
      0.19 ± 26%      -0.1        0.09 ± 13%  perf-profile.children.cycles-pp.timerqueue_del
      0.12 ± 32%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.pm_qos_request
      0.11 ± 17%      -0.1        0.03 ±102%  perf-profile.children.cycles-pp.run_posix_cpu_timers
      0.18 ± 21%      -0.1        0.10 ±  4%  perf-profile.children.cycles-pp.arch_cpu_idle_enter
      0.16 ± 21%      -0.1        0.10 ±  4%  perf-profile.children.cycles-pp.tsc_verify_tsc_adjust
      0.08 ± 13%      +0.0        0.10 ± 10%  perf-profile.children.cycles-pp.dequeue_task_fair
      0.07 ± 23%      +0.0        0.10 ± 21%  perf-profile.children.cycles-pp.__fget_light
      0.04 ± 57%      +0.0        0.07 ±  6%  perf-profile.children.cycles-pp.iov_iter_advance
      0.10 ±  8%      +0.0        0.13 ± 18%  perf-profile.children.cycles-pp.rcu_dynticks_eqs_exit
      0.12 ±  8%      +0.0        0.16 ±  9%  perf-profile.children.cycles-pp.newidle_balance
      0.08 ±  5%      +0.0        0.12 ± 15%  perf-profile.children.cycles-pp.__queue_work
      0.09 ± 22%      +0.0        0.13 ± 13%  perf-profile.children.cycles-pp.blk_mq_get_tag
      0.07 ± 24%      +0.0        0.11 ± 17%  perf-profile.children.cycles-pp.__fdget_pos
      0.07 ± 20%      +0.0        0.11 ± 15%  perf-profile.children.cycles-pp.raise_softirq
      0.04 ± 57%      +0.0        0.09 ± 17%  perf-profile.children.cycles-pp.queue_work_on
      0.11 ± 19%      +0.0        0.15 ±  8%  perf-profile.children.cycles-pp.blk_mq_get_request
      0.08 ± 21%      +0.0        0.12 ± 19%  perf-profile.children.cycles-pp.__mark_inode_dirty
      0.10 ± 14%      +0.0        0.14 ± 13%  perf-profile.children.cycles-pp.xas_clear_mark
      0.10 ±  7%      +0.0        0.14 ± 25%  perf-profile.children.cycles-pp.security_file_permission
      0.04 ± 58%      +0.0        0.08 ± 19%  perf-profile.children.cycles-pp.trigger_load_balance
      0.08 ± 19%      +0.0        0.12 ± 10%  perf-profile.children.cycles-pp.nfs_scan_commit_list
      0.14 ± 19%      +0.0        0.19 ±  8%  perf-profile.children.cycles-pp.__intel_pmu_enable_all
      0.14 ±  8%      +0.0        0.19 ± 17%  perf-profile.children.cycles-pp.kmem_cache_free
      0.08 ± 19%      +0.0        0.12 ± 12%  perf-profile.children.cycles-pp.nfs_io_completion_put
      0.03 ±100%      +0.0        0.07 ± 27%  perf-profile.children.cycles-pp.nfs_key_timeout_notify
      0.01 ±173%      +0.0        0.06 ± 11%  perf-profile.children.cycles-pp.__check_object_size
      0.01 ±173%      +0.0        0.06 ± 20%  perf-profile.children.cycles-pp.skb_page_frag_refill
      0.05 ± 64%      +0.1        0.10 ± 10%  perf-profile.children.cycles-pp.xas_init_marks
      0.01 ±173%      +0.1        0.06 ± 20%  perf-profile.children.cycles-pp.select_task_rq_fair
      0.09 ± 17%      +0.1        0.14 ± 18%  perf-profile.children.cycles-pp.nfs_page_group_unlock
      0.04 ± 57%      +0.1        0.09 ± 23%  perf-profile.children.cycles-pp.__fprop_inc_percpu_max
      0.09 ± 15%      +0.1        0.15 ± 12%  perf-profile.children.cycles-pp._cond_resched
      0.10 ±  8%      +0.1        0.16 ± 12%  perf-profile.children.cycles-pp.mutex_spin_on_owner
      0.15 ±  3%      +0.1        0.20 ±  3%  perf-profile.children.cycles-pp.dec_zone_page_state
      0.03 ±100%      +0.1        0.08 ± 13%  perf-profile.children.cycles-pp.inc_zone_page_state
      0.15 ± 19%      +0.1        0.20 ± 11%  perf-profile.children.cycles-pp.create_empty_buffers
      0.04 ± 58%      +0.1        0.10 ± 13%  perf-profile.children.cycles-pp.ext4_write_checks
      0.17 ±  6%      +0.1        0.23 ± 12%  perf-profile.children.cycles-pp.pick_next_task_fair
      0.27 ±  6%      +0.1        0.33 ± 10%  perf-profile.children.cycles-pp.native_sched_clock
      0.07 ± 24%      +0.1        0.14 ± 20%  perf-profile.children.cycles-pp.ext4_da_get_block_prep
      0.18 ± 16%      +0.1        0.24 ±  7%  perf-profile.children.cycles-pp.update_rq_clock
      0.28 ±  7%      +0.1        0.34 ±  9%  perf-profile.children.cycles-pp.sched_clock
      0.09 ± 24%      +0.1        0.15 ± 28%  perf-profile.children.cycles-pp.stop_this_handle
      0.24 ± 12%      +0.1        0.30 ± 13%  perf-profile.children.cycles-pp.__account_scheduler_latency
      0.22 ± 17%      +0.1        0.29 ± 11%  perf-profile.children.cycles-pp.percpu_counter_add_batch
      0.17 ± 22%      +0.1        0.24 ± 17%  perf-profile.children.cycles-pp.___might_sleep
      0.15 ± 18%      +0.1        0.22 ± 11%  perf-profile.children.cycles-pp.__mod_node_page_state
      0.17 ±  7%      +0.1        0.24 ± 17%  perf-profile.children.cycles-pp.__release_sock
      0.03 ±100%      +0.1        0.10 ± 15%  perf-profile.children.cycles-pp.lru_deactivate_file_fn
      0.11 ±  9%      +0.1        0.18 ± 15%  perf-profile.children.cycles-pp.pagevec_lookup_entries
      0.12 ± 11%      +0.1        0.19 ± 12%  perf-profile.children.cycles-pp.wait_on_page_bit
      0.11 ±  7%      +0.1        0.18 ± 15%  perf-profile.children.cycles-pp.find_get_entries
      0.18 ± 18%      +0.1        0.26 ± 11%  perf-profile.children.cycles-pp.xas_store
      0.15 ± 16%      +0.1        0.23 ± 12%  perf-profile.children.cycles-pp.nfs_inode_remove_request
      0.07 ± 21%      +0.1        0.14 ± 35%  perf-profile.children.cycles-pp.xs_stream_data_receive_workfn
      0.07 ± 21%      +0.1        0.14 ± 35%  perf-profile.children.cycles-pp.xs_stream_data_receive
      0.03 ±100%      +0.1        0.11 ± 48%  perf-profile.children.cycles-pp.xs_sock_recvmsg
      0.02 ±173%      +0.1        0.10 ± 23%  perf-profile.children.cycles-pp.__sb_start_write
      0.06 ± 66%      +0.1        0.14 ± 23%  perf-profile.children.cycles-pp.deactivate_file_page
      0.32 ± 11%      +0.1        0.41 ± 14%  perf-profile.children.cycles-pp.enqueue_entity
      0.20 ±  5%      +0.1        0.29 ± 16%  perf-profile.children.cycles-pp.release_sock
      0.18 ± 12%      +0.1        0.27 ± 18%  perf-profile.children.cycles-pp.new_slab
      0.24 ±  6%      +0.1        0.33 ± 12%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.07 ± 19%      +0.1        0.16 ± 57%  perf-profile.children.cycles-pp.tcp_sendpage_locked
      0.12 ± 16%      +0.1        0.22 ±  7%  perf-profile.children.cycles-pp.io_schedule
      0.06 ± 17%      +0.1        0.15 ± 55%  perf-profile.children.cycles-pp.do_tcp_sendpages
      0.00            +0.1        0.10 ± 70%  perf-profile.children.cycles-pp.__libc_fork
      0.08 ± 16%      +0.1        0.17 ± 54%  perf-profile.children.cycles-pp.kernel_sendpage
      0.08 ± 16%      +0.1        0.17 ± 54%  perf-profile.children.cycles-pp.inet_sendpage
      0.29 ± 10%      +0.1        0.39 ±  9%  perf-profile.children.cycles-pp.schedule
      0.21 ± 14%      +0.1        0.30 ± 21%  perf-profile.children.cycles-pp.__slab_alloc
      0.14 ± 18%      +0.1        0.24 ± 30%  perf-profile.children.cycles-pp.__pagevec_lru_add_fn
      0.08 ± 16%      +0.1        0.18 ± 53%  perf-profile.children.cycles-pp.svc_tcp_sendto
      0.08 ± 16%      +0.1        0.18 ± 53%  perf-profile.children.cycles-pp.svc_sendto
      0.08 ± 16%      +0.1        0.18 ± 53%  perf-profile.children.cycles-pp.svc_send_common
      0.08 ± 20%      +0.1        0.17 ± 54%  perf-profile.children.cycles-pp.tcp_sendpage
      0.20 ± 12%      +0.1        0.30 ± 20%  perf-profile.children.cycles-pp.___slab_alloc
      0.16 ±  8%      +0.1        0.26 ±  7%  perf-profile.children.cycles-pp.nfs_scan_commit
      0.14 ± 19%      +0.1        0.24 ± 35%  perf-profile.children.cycles-pp.tcp_v6_rcv
      0.12 ± 60%      +0.1        0.22 ± 26%  perf-profile.children.cycles-pp.timekeeping_max_deferment
      0.25 ± 18%      +0.1        0.36 ± 19%  perf-profile.children.cycles-pp.entry_SYSCALL_64
      0.12 ± 39%      +0.1        0.23 ± 12%  perf-profile.children.cycles-pp._raw_spin_trylock
      0.15 ± 17%      +0.1        0.26 ± 34%  perf-profile.children.cycles-pp.ip6_input
      0.18 ± 13%      +0.1        0.29 ± 29%  perf-profile.children.cycles-pp.net_rx_action
      0.13 ± 12%      +0.1        0.25 ± 13%  perf-profile.children.cycles-pp.free_pcppages_bulk
      0.21 ± 22%      +0.1        0.32 ± 22%  perf-profile.children.cycles-pp.jbd2_journal_stop
      0.43 ±  6%      +0.1        0.55 ± 14%  perf-profile.children.cycles-pp._raw_spin_lock
      0.23 ± 10%      +0.1        0.35 ± 13%  perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
      0.32 ± 23%      +0.1        0.43 ± 12%  perf-profile.children.cycles-pp.__add_to_page_cache_locked
      0.20 ± 21%      +0.1        0.32 ± 14%  perf-profile.children.cycles-pp.__delete_from_page_cache
      0.15 ± 18%      +0.1        0.27 ± 33%  perf-profile.children.cycles-pp.ipv6_rcv
      0.45 ± 20%      +0.1        0.57 ±  7%  perf-profile.children.cycles-pp.ext4_io_submit
      0.18 ± 20%      +0.1        0.30 ± 28%  perf-profile.children.cycles-pp.__nfs_create_request
      0.17 ± 17%      +0.1        0.29 ± 29%  perf-profile.children.cycles-pp.do_softirq_own_stack
      0.16 ± 16%      +0.1        0.28 ± 31%  perf-profile.children.cycles-pp.__netif_receive_skb_one_core
      0.23 ± 25%      +0.1        0.35 ± 22%  perf-profile.children.cycles-pp.__ext4_journal_stop
      0.45 ± 20%      +0.1        0.57 ±  6%  perf-profile.children.cycles-pp.submit_bio
      0.45 ± 20%      +0.1        0.57 ±  6%  perf-profile.children.cycles-pp.generic_make_request
      0.42 ± 21%      +0.1        0.55 ±  6%  perf-profile.children.cycles-pp.blk_mq_make_request
      0.17 ± 16%      +0.1        0.30 ± 30%  perf-profile.children.cycles-pp.__local_bh_enable_ip
      0.17 ± 16%      +0.1        0.29 ± 30%  perf-profile.children.cycles-pp.do_softirq
      0.16 ± 17%      +0.1        0.29 ± 31%  perf-profile.children.cycles-pp.process_backlog
      0.18 ± 24%      +0.1        0.31 ±  8%  perf-profile.children.cycles-pp.native_apic_msr_eoi_write
      0.24 ±  7%      +0.1        0.37 ± 29%  perf-profile.children.cycles-pp.tcp_rcv_established
      0.14 ± 30%      +0.1        0.28 ± 32%  perf-profile.children.cycles-pp.__tcp_push_pending_frames
      0.22 ± 22%      +0.1        0.35 ± 27%  perf-profile.children.cycles-pp.nfs_create_request
      0.20 ± 16%      +0.1        0.33 ± 28%  perf-profile.children.cycles-pp.ip6_finish_output2
      0.23 ± 25%      +0.1        0.37 ± 16%  perf-profile.children.cycles-pp.__filemap_fdatawait_range
      0.24 ±  7%      +0.1        0.38 ± 29%  perf-profile.children.cycles-pp.tcp_v6_do_rcv
      0.21 ± 21%      +0.1        0.35 ± 22%  perf-profile.children.cycles-pp.__lru_cache_add
      0.19 ± 25%      +0.1        0.34 ± 34%  perf-profile.children.cycles-pp.find_get_pages_range_tag
      0.45 ± 10%      +0.1        0.60 ±  5%  perf-profile.children.cycles-pp.lapic_next_deadline
      0.44 ±  7%      +0.1        0.59 ± 13%  perf-profile.children.cycles-pp.__sched_text_start
      0.34 ± 10%      +0.1        0.49 ±  7%  perf-profile.children.cycles-pp.clear_page_dirty_for_io
      0.23 ± 17%      +0.1        0.38 ± 27%  perf-profile.children.cycles-pp.ip6_xmit
      0.22 ± 17%      +0.2        0.37 ± 28%  perf-profile.children.cycles-pp.ip6_output
      0.29 ± 17%      +0.2        0.44 ± 13%  perf-profile.children.cycles-pp.__remove_mapping
      0.29 ± 18%      +0.2        0.45 ± 14%  perf-profile.children.cycles-pp.remove_mapping
      0.18 ± 24%      +0.2        0.33 ± 26%  perf-profile.children.cycles-pp.__nfs_commit_inode
      0.18 ± 12%      +0.2        0.34 ± 17%  perf-profile.children.cycles-pp.free_unref_page_list
      0.26 ± 23%      +0.2        0.42 ± 20%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.38 ± 14%      +0.2        0.54 ± 12%  perf-profile.children.cycles-pp.__mod_lruvec_state
      0.33 ± 20%      +0.2        0.49 ± 17%  perf-profile.children.cycles-pp.start_this_handle
      0.35 ± 12%      +0.2        0.51 ± 17%  perf-profile.children.cycles-pp.kmem_cache_alloc
      0.25 ± 20%      +0.2        0.41 ± 25%  perf-profile.children.cycles-pp.inet6_csk_xmit
      0.24 ± 18%      +0.2        0.40 ± 28%  perf-profile.children.cycles-pp.tcp_write_xmit
      0.38 ±  2%      +0.2        0.54 ±  7%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.44 ± 14%      +0.2        0.60 ± 19%  perf-profile.children.cycles-pp.mark_buffer_dirty
      0.27 ± 18%      +0.2        0.43 ± 16%  perf-profile.children.cycles-pp.ext4_block_write_begin
      0.28 ± 21%      +0.2        0.45 ± 23%  perf-profile.children.cycles-pp.__tcp_transmit_skb
      0.08 ± 16%      +0.2        0.25 ± 82%  perf-profile.children.cycles-pp.svc_send
      0.47 ± 12%      +0.2        0.65 ± 12%  perf-profile.children.cycles-pp.nfs_commit_release
      0.47 ± 12%      +0.2        0.65 ± 12%  perf-profile.children.cycles-pp.nfs_commit_release_pages
      0.29 ± 14%      +0.2        0.47 ±  8%  perf-profile.children.cycles-pp.io_serial_in
      0.51 ±  5%      +0.2        0.70 ±  5%  perf-profile.children.cycles-pp.native_write_msr
      0.14 ± 18%      +0.2        0.33 ± 17%  perf-profile.children.cycles-pp.ktime_get_update_offsets_now
      0.33 ± 10%      +0.2        0.52 ± 15%  perf-profile.children.cycles-pp.autoremove_wake_function
      0.87 ± 17%      +0.2        1.07 ±  8%  perf-profile.children.cycles-pp.end_page_writeback
      0.83 ± 17%      +0.2        1.03 ±  8%  perf-profile.children.cycles-pp.test_clear_page_writeback
      0.46 ± 15%      +0.2        0.65 ± 16%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.56 ± 22%      +0.2        0.76 ±  9%  perf-profile.children.cycles-pp.jbd2__journal_start
      0.34 ±  8%      +0.2        0.54 ± 14%  perf-profile.children.cycles-pp.__wake_up_common
      0.26 ± 15%      +0.2        0.46 ± 19%  perf-profile.children.cycles-pp.pagevec_lru_move_fn
      0.48 ± 10%      +0.2        0.69 ± 16%  perf-profile.children.cycles-pp.try_to_wake_up
      0.24 ± 12%      +0.2        0.45 ± 15%  perf-profile.children.cycles-pp.wake_up_page_bit
      0.28 ± 36%      +0.2        0.50 ± 12%  perf-profile.children.cycles-pp.note_gp_changes
      0.44 ± 10%      +0.2        0.66 ±  6%  perf-profile.children.cycles-pp.read_tsc
      0.56 ± 14%      +0.2        0.79 ± 17%  perf-profile.children.cycles-pp.__test_set_page_writeback
      0.69 ± 21%      +0.2        0.93 ±  9%  perf-profile.children.cycles-pp.__ext4_journal_start_sb
      0.54 ± 21%      +0.3        0.80 ± 14%  perf-profile.children.cycles-pp.add_to_page_cache_lru
      0.71 ± 16%      +0.3        1.00 ±  9%  perf-profile.children.cycles-pp.nfsd_commit
      0.71 ± 16%      +0.3        1.00 ±  9%  perf-profile.children.cycles-pp.ext4_sync_file
      0.71 ± 16%      +0.3        1.00 ±  9%  perf-profile.children.cycles-pp.file_write_and_wait_range
      0.35 ± 13%      +0.3        0.65 ± 12%  perf-profile.children.cycles-pp.__pagevec_release
      0.39 ± 13%      +0.3        0.71 ± 12%  perf-profile.children.cycles-pp.release_pages
      0.97 ± 17%      +0.3        1.31 ±  3%  perf-profile.children.cycles-pp.ext4_bio_write_page
      0.91 ± 15%      +0.3        1.26 ± 16%  perf-profile.children.cycles-pp.__block_commit_write
      0.94 ± 15%      +0.4        1.30 ± 15%  perf-profile.children.cycles-pp.block_write_end
      1.16 ±  7%      +0.4        1.53 ± 16%  perf-profile.children.cycles-pp.inet6_recvmsg
      0.29 ± 56%      +0.4        0.67 ± 13%  perf-profile.children.cycles-pp.tick_nohz_irq_exit
      1.16 ±  6%      +0.4        1.53 ± 16%  perf-profile.children.cycles-pp.tcp_recvmsg
      0.57 ± 22%      +0.4        0.99 ± 28%  perf-profile.children.cycles-pp.nfs_write_begin
      0.76 ±  2%      +0.4        1.17 ± 19%  perf-profile.children.cycles-pp.nfs_end_page_writeback
      1.20 ± 16%      +0.4        1.62 ±  2%  perf-profile.children.cycles-pp.mpage_submit_page
      1.23 ± 16%      +0.4        1.66 ±  2%  perf-profile.children.cycles-pp.mpage_process_page_bufs
      1.21 ± 15%      +0.5        1.68 ± 14%  perf-profile.children.cycles-pp.generic_write_end
      1.45 ± 16%      +0.5        1.98 ±  4%  perf-profile.children.cycles-pp.mpage_prepare_extent_to_map
      1.49 ± 15%      +0.5        2.03 ±  4%  perf-profile.children.cycles-pp.ext4_writepages
      1.21 ± 20%      +0.5        1.76 ± 14%  perf-profile.children.cycles-pp.pagecache_get_page
      1.24 ± 20%      +0.6        1.79 ± 14%  perf-profile.children.cycles-pp.grab_cache_page_write_begin
      1.11 ± 14%      +0.6        1.67 ± 29%  perf-profile.children.cycles-pp.nfs_wb_all
      1.50 ±  5%      +0.6        2.09 ± 14%  perf-profile.children.cycles-pp.memcpy_from_page
      1.53 ±  6%      +0.6        2.13 ± 14%  perf-profile.children.cycles-pp._copy_from_iter_full
      1.00 ± 19%      +0.6        1.61 ± 23%  perf-profile.children.cycles-pp.nfs_write_end
      1.89 ± 22%      +0.6        2.53 ±  6%  perf-profile.children.cycles-pp.ext4_da_write_begin
      1.58 ± 14%      +0.7        2.24 ± 10%  perf-profile.children.cycles-pp.ext4_da_write_end
      1.69 ± 19%      +0.7        2.37 ± 16%  perf-profile.children.cycles-pp.copyin
      1.68 ± 19%      +0.7        2.37 ± 16%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      1.03 ± 15%      +0.7        1.74 ± 11%  perf-profile.children.cycles-pp.invalidate_mapping_pages
      0.97 ± 35%      +0.7        1.70 ±  9%  perf-profile.children.cycles-pp.clockevents_program_event
      2.03 ±  9%      +0.8        2.82 ± 15%  perf-profile.children.cycles-pp.rpc_async_schedule
      2.03 ±  9%      +0.8        2.82 ± 15%  perf-profile.children.cycles-pp.__rpc_execute
      2.24 ± 15%      +0.8        3.07 ± 17%  perf-profile.children.cycles-pp.__filemap_fdatawrite_range
      0.63 ± 71%      +0.8        1.48 ± 14%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      2.30 ± 14%      +0.9        3.16 ± 15%  perf-profile.children.cycles-pp.do_writepages
      1.77 ± 17%      +0.9        2.72 ±  7%  perf-profile.children.cycles-pp.__x64_sys_fadvise64
      1.77 ± 17%      +0.9        2.72 ±  7%  perf-profile.children.cycles-pp.ksys_fadvise64_64
      1.77 ± 17%      +0.9        2.72 ±  7%  perf-profile.children.cycles-pp.generic_fadvise
      1.14 ± 45%      +1.1        2.23 ± 12%  perf-profile.children.cycles-pp.ktime_get
      1.22 ± 48%      +1.2        2.38 ± 16%  perf-profile.children.cycles-pp.rcu_core
      1.34 ± 36%      +1.2        2.54 ± 10%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      2.97 ± 14%      +1.3        4.28 ± 16%  perf-profile.children.cycles-pp.iov_iter_copy_from_user_atomic
      2.61 ± 20%      +1.5        4.11 ± 24%  perf-profile.children.cycles-pp.nfs_file_write
      5.98 ± 19%      +2.8        8.81 ± 14%  perf-profile.children.cycles-pp.new_sync_write
      6.21 ± 19%      +3.0        9.17 ± 14%  perf-profile.children.cycles-pp.vfs_write
      6.30 ± 19%      +3.0        9.31 ± 14%  perf-profile.children.cycles-pp.ksys_write
      8.30 ± 16%      +3.7       12.01 ± 13%  perf-profile.children.cycles-pp.generic_perform_write
     10.25 ± 16%      +5.0       15.29 ± 14%  perf-profile.children.cycles-pp.do_syscall_64
     10.28 ± 16%      +5.1       15.33 ± 14%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     52.58 ±  8%     -11.1       41.43 ± 10%  perf-profile.self.cycles-pp.intel_idle
      1.42 ±  7%      -0.3        1.10 ± 19%  perf-profile.self.cycles-pp.menu_select
      0.18 ± 28%      -0.2        0.03 ±100%  perf-profile.self.cycles-pp.timerqueue_add
      0.27 ± 22%      -0.1        0.14 ±  5%  perf-profile.self.cycles-pp.run_timer_softirq
      0.23 ± 14%      -0.1        0.11 ± 13%  perf-profile.self.cycles-pp.nr_iowait_cpu
      0.20 ± 20%      -0.1        0.10 ± 25%  perf-profile.self.cycles-pp.rcu_core
      0.12 ± 32%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.pm_qos_request
      0.11 ± 17%      -0.1        0.03 ±102%  perf-profile.self.cycles-pp.run_posix_cpu_timers
      0.16 ± 13%      -0.1        0.09 ± 20%  perf-profile.self.cycles-pp.perf_mux_hrtimer_handler
      0.13 ± 20%      -0.1        0.06 ± 13%  perf-profile.self.cycles-pp.update_blocked_averages
      0.12 ± 19%      -0.1        0.07 ± 16%  perf-profile.self.cycles-pp.rb_next
      0.11 ± 25%      -0.1        0.06 ± 13%  perf-profile.self.cycles-pp.smp_apic_timer_interrupt
      0.11 ± 22%      -0.1        0.06 ± 17%  perf-profile.self.cycles-pp.load_balance
      0.11 ± 28%      -0.0        0.06 ± 13%  perf-profile.self.cycles-pp.get_next_timer_interrupt
      0.11 ± 15%      -0.0        0.06 ± 26%  perf-profile.self.cycles-pp.scheduler_tick
      0.10 ±  8%      +0.0        0.13 ± 18%  perf-profile.self.cycles-pp.rcu_dynticks_eqs_exit
      0.07 ± 11%      +0.0        0.11 ±  8%  perf-profile.self.cycles-pp.rcu_idle_exit
      0.08 ± 23%      +0.0        0.11 ±  7%  perf-profile.self.cycles-pp.kmem_cache_alloc
      0.06 ± 20%      +0.0        0.10 ± 21%  perf-profile.self.cycles-pp.__fget_light
      0.14 ± 15%      +0.0        0.18 ±  4%  perf-profile.self.cycles-pp.update_rq_clock
      0.07 ± 21%      +0.0        0.11 ± 21%  perf-profile.self.cycles-pp.ext4_journal_check_start
      0.10 ± 18%      +0.0        0.13 ± 14%  perf-profile.self.cycles-pp.xas_clear_mark
      0.07 ± 17%      +0.0        0.11 ± 12%  perf-profile.self.cycles-pp.raise_softirq
      0.07 ± 23%      +0.0        0.11 ± 22%  perf-profile.self.cycles-pp.stop_this_handle
      0.04 ± 60%      +0.0        0.09 ± 25%  perf-profile.self.cycles-pp.nfs_request_add_commit_list
      0.06 ± 22%      +0.0        0.10 ± 14%  perf-profile.self.cycles-pp.invalidate_mapping_pages
      0.03 ±102%      +0.0        0.07 ± 11%  perf-profile.self.cycles-pp.xas_set_mark
      0.04 ± 59%      +0.0        0.09 ± 13%  perf-profile.self.cycles-pp.iov_iter_copy_from_user_atomic
      0.12 ± 10%      +0.0        0.17 ±  6%  perf-profile.self.cycles-pp.ext4_bio_write_page
      0.09 ± 17%      +0.1        0.14 ± 18%  perf-profile.self.cycles-pp.nfs_page_group_unlock
      0.10 ±  8%      +0.1        0.16 ± 13%  perf-profile.self.cycles-pp.mutex_spin_on_owner
      0.01 ±173%      +0.1        0.07 ± 23%  perf-profile.self.cycles-pp.__filemap_fdatawait_range
      0.03 ±100%      +0.1        0.08 ± 19%  perf-profile.self.cycles-pp.nfs_inode_remove_request
      0.12 ±  6%      +0.1        0.18 ±  6%  perf-profile.self.cycles-pp.dec_zone_page_state
      0.17 ± 18%      +0.1        0.23 ±  7%  perf-profile.self.cycles-pp.clear_page_dirty_for_io
      0.09 ± 15%      +0.1        0.15 ± 19%  perf-profile.self.cycles-pp.__mod_memcg_state
      0.08 ± 14%      +0.1        0.13 ± 14%  perf-profile.self.cycles-pp.try_to_wake_up
      0.01 ±173%      +0.1        0.07 ± 26%  perf-profile.self.cycles-pp.memset_erms
      0.01 ±173%      +0.1        0.07 ± 20%  perf-profile.self.cycles-pp.pagecache_get_page
      0.01 ±173%      +0.1        0.07 ± 14%  perf-profile.self.cycles-pp.account_page_dirtied
      0.01 ±173%      +0.1        0.08 ± 14%  perf-profile.self.cycles-pp.trigger_load_balance
      0.15 ± 16%      +0.1        0.21 ± 11%  perf-profile.self.cycles-pp.__mod_node_page_state
      0.00            +0.1        0.07 ±  7%  perf-profile.self.cycles-pp.inc_zone_page_state
      0.09 ± 43%      +0.1        0.15 ± 14%  perf-profile.self.cycles-pp.calc_global_load_tick
      0.25 ±  5%      +0.1        0.32 ±  9%  perf-profile.self.cycles-pp.native_sched_clock
      0.10 ±  8%      +0.1        0.17 ± 11%  perf-profile.self.cycles-pp.find_get_entries
      0.15 ± 12%      +0.1        0.22 ± 15%  perf-profile.self.cycles-pp.start_this_handle
      0.15 ± 22%      +0.1        0.22 ±  9%  perf-profile.self.cycles-pp.percpu_counter_add_batch
      0.17 ±  9%      +0.1        0.24 ± 18%  perf-profile.self.cycles-pp.__test_set_page_writeback
      0.15 ± 25%      +0.1        0.22 ± 13%  perf-profile.self.cycles-pp.ext4_da_write_end
      0.09 ± 23%      +0.1        0.17 ± 23%  perf-profile.self.cycles-pp.tick_sched_timer
      0.10 ± 15%      +0.1        0.18 ± 12%  perf-profile.self.cycles-pp.free_pcppages_bulk
      0.00            +0.1        0.08 ± 17%  perf-profile.self.cycles-pp.irq_enter
      0.16 ± 15%      +0.1        0.24 ± 20%  perf-profile.self.cycles-pp.new_slab
      0.24 ±  5%      +0.1        0.33 ± 13%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.14 ±  3%      +0.1        0.23 ± 17%  perf-profile.self.cycles-pp.___might_sleep
      0.12 ± 21%      +0.1        0.22 ± 19%  perf-profile.self.cycles-pp.release_pages
      0.41 ±  7%      +0.1        0.51 ±  6%  perf-profile.self.cycles-pp._raw_spin_lock
      0.20 ±  9%      +0.1        0.30 ± 15%  perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
      0.25 ± 18%      +0.1        0.36 ± 19%  perf-profile.self.cycles-pp.entry_SYSCALL_64
      0.12 ± 39%      +0.1        0.23 ± 14%  perf-profile.self.cycles-pp._raw_spin_trylock
      0.10 ± 88%      +0.1        0.21 ± 27%  perf-profile.self.cycles-pp.timekeeping_max_deferment
      0.17 ± 23%      +0.1        0.30 ±  8%  perf-profile.self.cycles-pp.native_apic_msr_eoi_write
      0.40 ± 12%      +0.1        0.53 ±  5%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.08 ± 17%      +0.1        0.22 ± 27%  perf-profile.self.cycles-pp.ktime_get_update_offsets_now
      0.26 ± 23%      +0.2        0.42 ± 20%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.38 ±  2%      +0.2        0.54 ±  7%  perf-profile.self.cycles-pp.native_irq_return_iret
      0.51 ±  5%      +0.2        0.69 ±  6%  perf-profile.self.cycles-pp.native_write_msr
      0.20 ± 34%      +0.2        0.38 ± 11%  perf-profile.self.cycles-pp.note_gp_changes
      0.42 ± 19%      +0.2        0.62 ± 29%  perf-profile.self.cycles-pp.nfs_write_end
      0.42 ± 10%      +0.2        0.62 ±  8%  perf-profile.self.cycles-pp.read_tsc
      0.70 ± 16%      +0.4        1.05 ± 15%  perf-profile.self.cycles-pp.do_syscall_64
      0.73 ± 10%      +0.4        1.12 ±  3%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      1.49 ±  5%      +0.6        2.07 ± 15%  perf-profile.self.cycles-pp.memcpy_from_page
      1.64 ± 22%      +0.7        2.34 ± 16%  perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      0.63 ± 71%      +0.8        1.47 ± 14%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      0.74 ± 71%      +0.9        1.66 ± 13%  perf-profile.self.cycles-pp.ktime_get





Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.5.0-rc2-00011-gbf5a9506c59ab" of type "text/plain" (202395 bytes)

View attachment "job-script" of type "text/plain" (8368 bytes)

View attachment "job.yaml" of type "text/plain" (6005 bytes)

View attachment "reproduce" of type "text/plain" (1041 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ