lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Message-ID: <202308241229.68396422-oliver.sang@intel.com>
Date:   Thu, 24 Aug 2023 13:59:06 +0800
From:   kernel test robot <oliver.sang@...el.com>
To:     Chuck Lever <chuck.lever@...cle.com>
CC:     <oe-lkp@...ts.linux.dev>, <lkp@...el.com>,
        <linux-kernel@...r.kernel.org>, Jeff Layton <jlayton@...nel.org>,
        <linux-nfs@...r.kernel.org>, <ying.huang@...el.com>,
        <feng.tang@...el.com>, <fengwei.yin@...el.com>,
        <oliver.sang@...el.com>
Subject: [linus:master] [NFSD]  39d432fc76:  fsmark.files_per_sec -100.0%
 regression



hi, Chuck Lever,

Fengwei (CCed) helped us review this astonishing finding by fsmark tests,
and doubt below part:
-			nfsd4_end_grace(nn);
+			trace_nfsd_end_grace(netns(file));

and confirmed if adding back:
			nfsd4_end_grace(nn);

the regression is gone and files_per_sec restore to 61.93.

FYI.


Hello,

kernel test robot noticed a -100.0% regression of fsmark.files_per_sec on:


commit: 39d432fc76301cf0a0c454022117601994ca9397 ("NFSD: trace nfsctl operations")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master

testcase: fsmark
test machine: 64 threads 2 sockets Intel(R) Xeon(R) Gold 6346 CPU @ 3.10GHz (Ice Lake) with 256G memory
parameters:

	iterations: 1x
	nr_threads: 32t
	disk: 1SSD
	fs: btrfs
	fs2: nfsv4
	filesize: 16MB
	test_size: 20G
	sync_method: NoSync
	nr_directories: 16d
	nr_files_per_directory: 256fpd
	cpufreq_governor: performance


In addition to that, the commit also has significant impact on the following tests:

+------------------+-------------------------------------------------------------------------------------------+
| testcase: change | fsmark: fsmark.files_per_sec -100.0% regression                                           |
| test machine     | 64 threads 2 sockets Intel(R) Xeon(R) Gold 6346 CPU @ 3.10GHz (Ice Lake) with 256G memory |
| test parameters  | cpufreq_governor=performance                                                              |
|                  | debug-setup=no-monitor                                                                    |
|                  | disk=1SSD                                                                                 |
|                  | filesize=16MB                                                                             |
|                  | fs2=nfsv4                                                                                 |
|                  | fs=btrfs                                                                                  |
|                  | iterations=1x                                                                             |
|                  | nr_directories=16d                                                                        |
|                  | nr_files_per_directory=256fpd                                                             |
|                  | nr_threads=32t                                                                            |
|                  | sync_method=NoSync                                                                        |
|                  | test_size=20G                                                                             |
+------------------+-------------------------------------------------------------------------------------------+


If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <oliver.sang@...el.com>
| Closes: https://lore.kernel.org/oe-lkp/202308241229.68396422-oliver.sang@intel.com


Details are as below:
-------------------------------------------------------------------------------------------------->


The kernel config and materials to reproduce are available at:
https://download.01.org/0day-ci/archive/20230824/202308241229.68396422-oliver.sang@intel.com

=========================================================================================
compiler/cpufreq_governor/disk/filesize/fs2/fs/iterations/kconfig/nr_directories/nr_files_per_directory/nr_threads/rootfs/sync_method/tbox_group/test_size/testcase:
  gcc-12/performance/1SSD/16MB/nfsv4/btrfs/1x/x86_64-rhel-8.3/16d/256fpd/32t/debian-11.1-x86_64-20220510.cgz/NoSync/lkp-icl-2sp7/20G/fsmark

commit: 
  3434d7aa77 ("NFSD: Clean up nfsctl_transaction_write()")
  39d432fc76 ("NFSD: trace nfsctl operations")

3434d7aa77d24c5c 39d432fc76301cf0a0c45402211 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
     51.67 ± 25%     -90.8%       4.75 ± 45%  perf-c2c.DRAM.local
     35.36 ± 16%     +35.2%      47.81 ± 16%  boot-time.dhcp
      1.81 ± 41%     +90.9%       3.46 ± 31%  boot-time.smp_boot
 1.455e+09 ±  6%  +12542.8%  1.839e+11 ± 20%  cpuidle..time
   3013365 ±  2%   +6257.5%  1.916e+08 ± 20%  cpuidle..usage
     81.11 ±  7%   +3758.6%       3129 ± 18%  uptime.boot
      4349 ±  8%   +4266.2%     189897 ± 19%  uptime.idle
     71.95           +31.7%      94.75        iostat.cpu.idle
     18.35 ±  3%     -84.5%       2.85 ± 56%  iostat.cpu.iowait
      7.98 ±  5%     -70.3%       2.37        iostat.cpu.system
      1.73 ±  3%     -98.0%       0.03 ± 15%  iostat.cpu.user
   7176382 ±  6%     +37.6%    9873307 ±  6%  numa-numastat.node0.local_node
   7210227 ±  6%     +37.5%    9914765 ±  6%  numa-numastat.node0.numa_hit
   6958614 ±  6%     +29.1%    8981736 ±  4%  numa-numastat.node1.local_node
   6990967 ±  6%     +28.8%    9006618 ±  4%  numa-numastat.node1.numa_hit
   8749533 ±  4%   +7050.4%  6.256e+08 ± 18%  fsmark.app_overhead
     61.92 ±  5%    -100.0%       0.00        fsmark.files_per_sec
     22.77 ±  4%  +12864.1%       2951 ± 20%  fsmark.time.elapsed_time
     22.77 ±  4%  +12864.1%       2951 ± 20%  fsmark.time.elapsed_time.max
    189.33 ±  7%     -99.5%       1.00        fsmark.time.percent_of_cpu_this_job_got
     70.09           +24.7       94.75        mpstat.cpu.all.idle%
     19.80 ±  4%     -16.9        2.86 ± 56%  mpstat.cpu.all.iowait%
      1.18 ±  4%      +1.0        2.20        mpstat.cpu.all.irq%
      0.13 ±  2%      -0.1        0.07 ±  3%  mpstat.cpu.all.soft%
      7.10 ±  7%      -7.0        0.09 ± 12%  mpstat.cpu.all.sys%
      1.70 ±  5%      -1.7        0.03 ± 15%  mpstat.cpu.all.usr%
     71.33           +32.5%      94.50        vmstat.cpu.id
     18.00 ±  3%     -87.5%       2.25 ± 79%  vmstat.cpu.wa
    784537 ±  5%     -99.0%       7461 ± 21%  vmstat.io.bo
     10.17 ± 23%     -85.2%       1.50 ± 74%  vmstat.procs.b
      6.00 ± 25%    -100.0%       0.00        vmstat.procs.r
    101767 ±  7%     -91.5%       8688 ±  8%  vmstat.system.cs
     70688            -9.1%      64225        vmstat.system.in
     43996 ±  5%    +121.0%      97246 ±  9%  meminfo.Active
      4516 ± 14%   +1588.6%      76266 ±  6%  meminfo.Active(anon)
     39479 ±  4%     -46.9%      20978 ± 24%  meminfo.Active(file)
     34922 ± 23%    +551.6%     227542        meminfo.AnonHugePages
    397478           -24.1%     301733        meminfo.AnonPages
    653045           -19.5%     525838        meminfo.Committed_AS
    154701 ± 15%     -42.0%      89789 ±  4%  meminfo.Dirty
    401768           -23.5%     307192        meminfo.Inactive(anon)
     33307           -11.3%      29548        meminfo.Mapped
      6837 ±  7%     -27.6%       4950        meminfo.PageTables
      9016 ±  7%    +814.1%      82421 ±  6%  meminfo.Shmem
    324959 ±  7%     +25.1%     406488 ±  2%  meminfo.Writeback
      1940 ± 58%    +683.0%      15195 ± 36%  numa-meminfo.node0.Active(anon)
     18757 ± 14%     -33.6%      12450 ± 30%  numa-meminfo.node0.Active(file)
     18975 ± 47%    +872.5%     184526 ± 25%  numa-meminfo.node0.AnonHugePages
     79966 ± 22%     -38.8%      48963 ±  8%  numa-meminfo.node0.Dirty
      5833 ± 21%    +235.6%      19573 ± 31%  numa-meminfo.node0.Shmem
     23325 ± 16%    +198.4%      69601 ±  9%  numa-meminfo.node1.Active
      2576 ± 43%   +2270.7%      61072 ±  5%  numa-meminfo.node1.Active(anon)
     20749 ± 15%     -58.9%       8528 ± 46%  numa-meminfo.node1.Active(file)
    183812 ± 19%     -67.5%      59692 ± 75%  numa-meminfo.node1.AnonPages
     67153 ± 14%     -39.2%      40835 ± 14%  numa-meminfo.node1.Dirty
    184305 ± 19%     -66.8%      61274 ± 73%  numa-meminfo.node1.Inactive(anon)
      3143 ± 25%     -37.2%       1973 ± 21%  numa-meminfo.node1.PageTables
      3184 ± 36%   +1873.9%      62849 ±  5%  numa-meminfo.node1.Shmem
    339.17 ±  5%     -91.4%      29.25        turbostat.Avg_MHz
     11.43 ±  5%      -8.2        3.23        turbostat.Busy%
      3004           -69.7%     909.25        turbostat.Bzy_MHz
     52538 ±  7%    +834.8%     491105 ± 18%  turbostat.C1
      0.41 ±  7%      -0.4        0.01 ± 57%  turbostat.C1%
    849439 ±  2%    +679.9%    6624934 ± 48%  turbostat.C1E
     23.10 ±  3%     -20.1        3.02 ± 53%  turbostat.C1E%
   1067521 ±  8%  +17084.8%  1.835e+08 ± 20%  turbostat.C6
     64.56 ±  2%     +29.6       94.19        turbostat.C6%
     87.74           +10.3%      96.76        turbostat.CPU%c1
      0.83 ± 45%     -98.5%       0.01 ± 34%  turbostat.CPU%c6
    128.85 ±152%     -99.9%       0.09        turbostat.IPC
   1858116 ±  5%  +10114.2%  1.898e+08 ± 20%  turbostat.IRQ
      0.64 ±  6%      -0.6        0.01 ±100%  turbostat.POLL%
    154.16           -23.3%     118.17        turbostat.PkgWatt
     63.10           -15.5%      53.29        turbostat.RAMWatt
    484.83 ± 58%    +683.5%       3798 ± 36%  numa-vmstat.node0.nr_active_anon
      4690 ± 14%     -33.6%       3112 ± 30%  numa-vmstat.node0.nr_active_file
      8.83 ± 51%    +916.0%      89.75 ± 25%  numa-vmstat.node0.nr_anon_transparent_hugepages
     19775 ± 24%     -38.3%      12194 ±  8%  numa-vmstat.node0.nr_dirty
      1457 ± 21%    +235.7%       4893 ± 31%  numa-vmstat.node0.nr_shmem
    484.83 ± 58%    +683.5%       3798 ± 36%  numa-vmstat.node0.nr_zone_active_anon
      4690 ± 14%     -33.6%       3112 ± 30%  numa-vmstat.node0.nr_zone_active_file
     36457 ± 22%     -59.5%      14783 ± 10%  numa-vmstat.node0.nr_zone_write_pending
   7210556 ±  6%     +37.5%    9914914 ±  6%  numa-vmstat.node0.numa_hit
   7176711 ±  6%     +37.6%    9873455 ±  6%  numa-vmstat.node0.numa_local
    643.83 ± 43%   +2271.4%      15268 ±  5%  numa-vmstat.node1.nr_active_anon
      5189 ± 15%     -58.9%       2132 ± 46%  numa-vmstat.node1.nr_active_file
     45952 ± 19%     -67.5%      14923 ± 75%  numa-vmstat.node1.nr_anon_pages
     16398 ± 15%     -37.8%      10204 ± 14%  numa-vmstat.node1.nr_dirty
     46074 ± 19%     -66.8%      15318 ± 73%  numa-vmstat.node1.nr_inactive_anon
    785.33 ± 25%     -37.3%     492.75 ± 21%  numa-vmstat.node1.nr_page_table_pages
    795.83 ± 36%   +1874.3%      15712 ±  5%  numa-vmstat.node1.nr_shmem
    643.83 ± 43%   +2271.4%      15268 ±  5%  numa-vmstat.node1.nr_zone_active_anon
      5189 ± 15%     -58.9%       2132 ± 46%  numa-vmstat.node1.nr_zone_active_file
     46074 ± 19%     -66.8%      15318 ± 73%  numa-vmstat.node1.nr_zone_inactive_anon
     34148 ± 11%     -62.3%      12864 ± 19%  numa-vmstat.node1.nr_zone_write_pending
   6991051 ±  6%     +28.8%    9006790 ±  4%  numa-vmstat.node1.numa_hit
   6958699 ±  6%     +29.1%    8981908 ±  4%  numa-vmstat.node1.numa_local
      1128 ± 14%   +1589.1%      19066 ±  6%  proc-vmstat.nr_active_anon
      9870 ±  4%     -46.9%       5244 ± 24%  proc-vmstat.nr_active_file
     99377           -24.1%      75431        proc-vmstat.nr_anon_pages
     16.67 ± 23%    +564.5%     110.75        proc-vmstat.nr_anon_transparent_hugepages
     38368 ± 15%     -41.6%      22410 ±  4%  proc-vmstat.nr_dirty
    100443           -23.5%      76796        proc-vmstat.nr_inactive_anon
     15586            -6.6%      14560        proc-vmstat.nr_kernel_stack
      8326           -11.3%       7386        proc-vmstat.nr_mapped
      1709 ±  7%     -27.6%       1236        proc-vmstat.nr_page_table_pages
      2254 ±  7%    +814.2%      20605 ±  6%  proc-vmstat.nr_shmem
     81381 ±  7%     +24.9%     101605 ±  2%  proc-vmstat.nr_writeback
      1128 ± 14%   +1589.1%      19066 ±  6%  proc-vmstat.nr_zone_active_anon
      9870 ±  4%     -46.9%       5244 ± 24%  proc-vmstat.nr_zone_active_file
    100443           -23.5%      76796        proc-vmstat.nr_zone_inactive_anon
     71853 ± 11%     -61.5%      27641 ± 12%  proc-vmstat.nr_zone_write_pending
    404.67 ±  6%   +1276.3%       5569 ± 13%  proc-vmstat.numa_hint_faults
     69.67 ± 27%   +4299.9%       3065 ± 77%  proc-vmstat.numa_hint_faults_local
  14202988           +33.2%   18923999 ±  5%  proc-vmstat.numa_hit
  14136791           +33.4%   18857658 ±  5%  proc-vmstat.numa_local
    335.00 ± 10%   +1203.3%       4366 ± 82%  proc-vmstat.numa_pages_migrated
    698.50 ±  8%   +5553.1%      39487 ± 15%  proc-vmstat.numa_pte_updates
     11408 ±  2%     +84.5%      21048 ±  5%  proc-vmstat.pgactivate
  18935279           +26.2%   23897622 ±  4%  proc-vmstat.pgalloc_normal
    225757         +2641.0%    6188094 ± 19%  proc-vmstat.pgfault
  13477492           +37.7%   18558535 ±  5%  proc-vmstat.pgfree
    335.00 ± 10%   +1203.3%       4366 ± 82%  proc-vmstat.pgmigrate_success
      8658 ±  2%   +3345.5%     298318 ± 19%  proc-vmstat.pgreuse
    637184 ± 40%   +3336.2%   21894720 ± 20%  proc-vmstat.unevictable_pgs_scanned
      0.23 ± 10%     -76.3%       0.05        sched_debug.cfs_rq:/.h_nr_running.avg
      0.43 ±  5%     -48.4%       0.22        sched_debug.cfs_rq:/.h_nr_running.stddev
     13720 ± 57%    +127.8%      31261 ±  2%  sched_debug.cfs_rq:/.load.avg
    109.64 ± 26%     -72.0%      30.67 ± 14%  sched_debug.cfs_rq:/.load_avg.avg
      1053 ±  2%     -20.8%     834.32 ±  3%  sched_debug.cfs_rq:/.load_avg.max
    274.91 ± 14%     -48.8%     140.77 ±  7%  sched_debug.cfs_rq:/.load_avg.stddev
     13475 ± 20%     +63.8%      22072 ± 17%  sched_debug.cfs_rq:/.min_vruntime.avg
     39896 ± 14%     +40.9%      56195 ± 15%  sched_debug.cfs_rq:/.min_vruntime.max
      5378 ± 23%    +108.5%      11216 ± 14%  sched_debug.cfs_rq:/.min_vruntime.min
      5503 ± 11%     +57.4%       8663 ± 13%  sched_debug.cfs_rq:/.min_vruntime.stddev
      0.23 ± 10%     -76.3%       0.05        sched_debug.cfs_rq:/.nr_running.avg
      0.43 ±  5%     -48.4%       0.22        sched_debug.cfs_rq:/.nr_running.stddev
     65.15 ± 36%     -93.0%       4.53 ± 76%  sched_debug.cfs_rq:/.removed.load_avg.avg
      1024           -87.2%     131.45 ±119%  sched_debug.cfs_rq:/.removed.load_avg.max
    242.33 ± 17%     -90.8%      22.40 ±101%  sched_debug.cfs_rq:/.removed.load_avg.stddev
     23.97 ± 36%     -91.6%       2.00 ± 84%  sched_debug.cfs_rq:/.removed.runnable_avg.avg
    504.33 ±  3%     -86.8%      66.39 ±118%  sched_debug.cfs_rq:/.removed.runnable_avg.max
     93.28 ± 17%     -88.6%      10.66 ±106%  sched_debug.cfs_rq:/.removed.runnable_avg.stddev
     23.97 ± 36%     -91.6%       2.00 ± 84%  sched_debug.cfs_rq:/.removed.util_avg.avg
    504.33 ±  3%     -86.8%      66.39 ±118%  sched_debug.cfs_rq:/.removed.util_avg.max
     93.28 ± 17%     -88.6%      10.66 ±106%  sched_debug.cfs_rq:/.removed.util_avg.stddev
    447.97 ±  3%     -93.2%      30.55 ± 10%  sched_debug.cfs_rq:/.runnable_avg.avg
      1652 ± 21%     -65.3%     573.91        sched_debug.cfs_rq:/.runnable_avg.max
    361.15 ± 10%     -72.7%      98.67 ±  4%  sched_debug.cfs_rq:/.runnable_avg.stddev
      5515 ± 10%     +57.1%       8663 ± 13%  sched_debug.cfs_rq:/.spread0.stddev
    447.20 ±  3%     -93.2%      30.54 ± 10%  sched_debug.cfs_rq:/.util_avg.avg
      1651 ± 21%     -65.2%     573.90        sched_debug.cfs_rq:/.util_avg.max
    360.75 ± 10%     -72.7%      98.65 ±  4%  sched_debug.cfs_rq:/.util_avg.stddev
     35.87 ± 32%     -92.8%       2.57 ± 23%  sched_debug.cfs_rq:/.util_est_enqueued.avg
    627.00 ± 32%     -88.2%      73.92 ± 11%  sched_debug.cfs_rq:/.util_est_enqueued.max
    125.31 ± 21%     -90.6%      11.82 ± 13%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
    773027 ±  4%     +25.1%     966950        sched_debug.cpu.avg_idle.avg
      3844 ± 16%   +9867.6%     383238 ± 36%  sched_debug.cpu.avg_idle.min
    284379 ± 22%     -62.7%     106134 ± 15%  sched_debug.cpu.avg_idle.stddev
     56501 ± 11%   +2798.3%    1637565 ± 18%  sched_debug.cpu.clock.avg
     56505 ± 11%   +2798.1%    1637568 ± 18%  sched_debug.cpu.clock.max
     56493 ± 11%   +2798.7%    1637561 ± 18%  sched_debug.cpu.clock.min
      2.84 ± 14%     -30.1%       1.98 ±  3%  sched_debug.cpu.clock.stddev
     55933 ± 11%   +2766.0%    1603047 ± 18%  sched_debug.cpu.clock_task.avg
     56388 ± 11%   +2747.3%    1605532 ± 18%  sched_debug.cpu.clock_task.max
     43272 ±  5%   +3548.5%    1578791 ± 18%  sched_debug.cpu.clock_task.min
      1611 ± 45%    +102.9%       3269 ± 31%  sched_debug.cpu.clock_task.stddev
      3920          +950.0%      41168 ± 18%  sched_debug.cpu.curr->pid.max
      1512 ±  2%    +241.7%       5166 ± 17%  sched_debug.cpu.curr->pid.stddev
      0.00 ± 28%     -66.7%       0.00 ±  4%  sched_debug.cpu.next_balance.stddev
      0.22 ±  7%     -87.6%       0.03 ±  4%  sched_debug.cpu.nr_running.avg
      0.42 ±  5%     -63.9%       0.15 ±  2%  sched_debug.cpu.nr_running.stddev
      6681         +2922.6%     201964 ± 20%  sched_debug.cpu.nr_switches.avg
    203120          +243.0%     696661 ± 25%  sched_debug.cpu.nr_switches.max
    347.50 ± 19%  +12362.1%      43305 ± 29%  sched_debug.cpu.nr_switches.min
     26003 ±  2%    +367.6%     121580 ± 15%  sched_debug.cpu.nr_switches.stddev
     56496 ± 11%   +2798.5%    1637561 ± 18%  sched_debug.cpu_clk
     55771 ± 11%   +2834.9%    1636836 ± 18%  sched_debug.ktime
     51473 ±  5%   +3047.3%    1620032 ± 18%  sched_debug.sched_clk
     17.02 ±  3%     -78.0%       3.74        perf-stat.i.MPKI
 1.839e+09 ±  6%     -94.8%   96458674 ±  5%  perf-stat.i.branch-instructions
      1.20 ±  7%      -0.4        0.80        perf-stat.i.branch-miss-rate%
  22120063 ± 10%     -95.7%     961876 ±  9%  perf-stat.i.branch-misses
     59.22 ±  3%     -50.6        8.66        perf-stat.i.cache-miss-rate%
  91195048 ±  7%     -99.1%     801389 ± 18%  perf-stat.i.cache-misses
 1.507e+08 ±  7%     -98.3%    2538095 ± 10%  perf-stat.i.cache-references
    119037 ±  8%     -92.7%       8685 ±  8%  perf-stat.i.context-switches
      2.52 ±  3%      -7.7%       2.33        perf-stat.i.cpi
 2.236e+10 ±  6%     -95.2%  1.081e+09 ±  3%  perf-stat.i.cpu-cycles
    264.18 ± 12%     -71.9%      74.16        perf-stat.i.cpu-migrations
    331.37 ± 29%   +2939.3%      10071 ±  2%  perf-stat.i.cycles-between-cache-misses
      0.03 ± 14%      +0.1        0.09 ± 23%  perf-stat.i.dTLB-load-miss-rate%
    734621 ± 10%     -86.2%     101019 ± 22%  perf-stat.i.dTLB-load-misses
 2.249e+09 ±  5%     -94.3%  1.281e+08 ±  4%  perf-stat.i.dTLB-loads
      0.01 ± 10%      +0.0        0.04 ± 13%  perf-stat.i.dTLB-store-miss-rate%
    122554 ±  8%     -80.7%      23688 ± 11%  perf-stat.i.dTLB-store-misses
 9.999e+08 ±  5%     -93.2%   67811239 ±  3%  perf-stat.i.dTLB-stores
 9.066e+09 ±  6%     -94.6%  4.866e+08 ±  4%  perf-stat.i.instructions
      0.47 ± 64%     -97.2%       0.01 ± 42%  perf-stat.i.major-faults
      0.35 ±  6%     -95.2%       0.02 ±  3%  perf-stat.i.metric.GHz
      1229 ±  6%     -34.7%     802.68        perf-stat.i.metric.K/sec
     81.72 ±  5%     -95.3%       3.81 ±  5%  perf-stat.i.metric.M/sec
      3477 ±  6%     -42.5%       1998        perf-stat.i.minor-faults
     52.18 ±  4%     +41.9       94.08        perf-stat.i.node-load-miss-rate%
  18404682 ±  7%     -99.0%     176907 ± 14%  perf-stat.i.node-load-misses
  16742387 ±  9%     -99.1%     144408 ± 22%  perf-stat.i.node-loads
     12.77 ±  9%     +18.7       31.45 ± 27%  perf-stat.i.node-store-miss-rate%
   4850218 ± 12%     -99.0%      49478 ±  9%  perf-stat.i.node-store-misses
  36531812 ±  6%     -99.1%     320605 ± 19%  perf-stat.i.node-stores
      3478 ±  6%     -42.5%       1998        perf-stat.i.page-faults
     16.63 ±  3%     -68.7%       5.21 ±  5%  perf-stat.overall.MPKI
      1.20 ±  7%      -0.2        1.00 ±  4%  perf-stat.overall.branch-miss-rate%
     60.51 ±  2%     -29.2       31.34 ±  8%  perf-stat.overall.cache-miss-rate%
      2.47 ±  3%      -9.9%       2.22        perf-stat.overall.cpi
    245.31          +464.9%       1385 ± 15%  perf-stat.overall.cycles-between-cache-misses
      0.03 ± 13%      +0.0        0.08 ± 22%  perf-stat.overall.dTLB-load-miss-rate%
      0.01 ±  9%      +0.0        0.03 ± 12%  perf-stat.overall.dTLB-store-miss-rate%
      0.41 ±  3%     +10.9%       0.45        perf-stat.overall.ipc
  1.76e+09 ±  5%     -94.5%   96442481 ±  5%  perf-stat.ps.branch-instructions
  21170965 ± 10%     -95.5%     961854 ±  9%  perf-stat.ps.branch-misses
  87299736 ±  6%     -99.1%     801893 ± 18%  perf-stat.ps.cache-misses
 1.443e+08 ±  6%     -98.2%    2538635 ± 10%  perf-stat.ps.cache-references
    113951 ±  8%     -92.4%       8683 ±  8%  perf-stat.ps.context-switches
     61339            +4.3%      63977        perf-stat.ps.cpu-clock
 2.141e+10 ±  6%     -95.0%  1.081e+09 ±  3%  perf-stat.ps.cpu-cycles
    252.91 ± 12%     -70.7%      74.14        perf-stat.ps.cpu-migrations
    703511 ± 10%     -85.6%     100987 ± 22%  perf-stat.ps.dTLB-load-misses
 2.153e+09 ±  5%     -94.1%  1.281e+08 ±  4%  perf-stat.ps.dTLB-loads
    117341 ±  8%     -79.8%      23681 ± 11%  perf-stat.ps.dTLB-store-misses
 9.573e+08 ±  5%     -92.9%   67795911 ±  3%  perf-stat.ps.dTLB-stores
 8.678e+09 ±  5%     -94.4%  4.866e+08 ±  4%  perf-stat.ps.instructions
      0.45 ± 64%     -97.1%       0.01 ± 42%  perf-stat.ps.major-faults
      3324 ±  5%     -39.9%       1998        perf-stat.ps.minor-faults
  17622482 ±  7%     -99.0%     177005 ± 14%  perf-stat.ps.node-load-misses
  16025463 ±  9%     -99.1%     144476 ± 22%  perf-stat.ps.node-loads
   4642492 ± 12%     -98.9%      49511 ± 10%  perf-stat.ps.node-store-misses
  34969456 ±  6%     -99.1%     320794 ± 19%  perf-stat.ps.node-stores
      3324 ±  5%     -39.9%       1998        perf-stat.ps.page-faults
     61339            +4.3%      63977        perf-stat.ps.task-clock
 2.065e+11 ±  3%    +589.5%  1.424e+12 ± 16%  perf-stat.total.instructions
      0.01 ± 60%    +113.6%       0.02 ± 20%  perf-sched.sch_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      0.00 ±157%    +425.0%       0.01 ± 14%  perf-sched.sch_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep
      0.03 ± 16%     +71.9%       0.05 ±  3%  perf-sched.sch_delay.avg.ms.irq_thread.kthread.ret_from_fork
      0.01 ± 38%     -65.4%       0.00 ± 19%  perf-sched.sch_delay.avg.ms.pipe_read.vfs_read.ksys_read.do_syscall_64
      0.58 ±151%     -96.5%       0.02 ±  7%  perf-sched.sch_delay.avg.ms.worker_thread.kthread.ret_from_fork
      0.00 ±108%    +218.0%       0.01 ± 13%  perf-sched.sch_delay.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      0.03 ± 21%     +92.0%       0.07        perf-sched.sch_delay.max.ms.irq_thread.kthread.ret_from_fork
      0.00 ±120%    +267.2%       0.02 ± 12%  perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll
      7.44 ±112%     -99.6%       0.03 ± 25%  perf-sched.sch_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork
      0.04 ± 91%     -81.6%       0.01        perf-sched.total_sch_delay.average.ms
     55.05 ±169%     -94.2%       3.20 ± 12%  perf-sched.total_sch_delay.max.ms
      9.41 ± 27%    +263.5%      34.20 ±  5%  perf-sched.total_wait_and_delay.average.ms
      2099 ± 13%    +875.6%      20478 ±  4%  perf-sched.total_wait_and_delay.count.ms
    309.85 ±  2%   +1511.3%       4992        perf-sched.total_wait_and_delay.max.ms
      9.37 ± 28%    +265.1%      34.20 ±  5%  perf-sched.total_wait_time.average.ms
    309.85 ±  2%   +1511.3%       4992        perf-sched.total_wait_time.max.ms
      1.83 ± 24%   +1179.9%      23.43        perf-sched.wait_and_delay.avg.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity
     68.28 ± 38%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      0.93 ± 30%  +26685.0%     250.13 ±  3%  perf-sched.wait_and_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
     69.46           +46.7%     101.87        perf-sched.wait_and_delay.avg.ms.irq_thread.kthread.ret_from_fork
      1.64 ±  6%   +2380.5%      40.63 ±  2%  perf-sched.wait_and_delay.avg.ms.pipe_read.vfs_read.ksys_read.do_syscall_64
     39.18 ± 14%    +863.6%     377.57        perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      4.35 ±  6%    +576.4%      29.42 ±  6%  perf-sched.wait_and_delay.avg.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
     32.32 ± 47%   +1966.0%     667.82        perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
     49.09 ± 11%    +739.5%     412.11 ±  4%  perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork
    245.00 ± 39%     +56.7%     384.00        perf-sched.wait_and_delay.count.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity
      3.83 ± 23%    -100.0%       0.00        perf-sched.wait_and_delay.count.devkmsg_read.vfs_read.ksys_read.do_syscall_64
    507.67 ±  9%     -75.6%     124.00        perf-sched.wait_and_delay.count.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
    449.83 ±  9%     -72.7%     123.00        perf-sched.wait_and_delay.count.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      3.00         +1500.0%      48.00        perf-sched.wait_and_delay.count.irq_thread.kthread.ret_from_fork
    343.83 ±  9%    +416.7%       1776 ±  2%  perf-sched.wait_and_delay.count.pipe_read.vfs_read.ksys_read.do_syscall_64
     73.33 ±  8%    +121.9%     162.75 ±  6%  perf-sched.wait_and_delay.count.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
    302.83 ± 30%     +71.9%     520.50        perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork
    117.50 ± 18%    +212.1%     366.75 ±  4%  perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork
    299.23 ±  3%   +1568.5%       4992        perf-sched.wait_and_delay.max.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity
    236.11 ± 27%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
     69.82 ±139%   +2069.6%       1514 ± 31%  perf-sched.wait_and_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
    238.02 ± 28%    +322.4%       1005        perf-sched.wait_and_delay.max.ms.pipe_read.vfs_read.ksys_read.do_syscall_64
    292.85 ±  3%     +70.7%     499.98        perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      5.04         +6666.8%     340.75 ± 18%  perf-sched.wait_and_delay.max.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
    305.61 ±  2%    +800.7%       2752 ± 29%  perf-sched.wait_and_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork
    303.33 ±  4%    +705.6%       2443 ± 13%  perf-sched.wait_and_delay.max.ms.worker_thread.kthread.ret_from_fork
      1.83 ± 24%   +1180.3%      23.43        perf-sched.wait_time.avg.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity
     68.27 ± 38%     -97.3%       1.82 ± 12%  perf-sched.wait_time.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      0.00 ±223%  +7.7e+07%     766.71 ±  7%  perf-sched.wait_time.avg.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep
      0.93 ± 30%  +26838.2%     250.12 ±  3%  perf-sched.wait_time.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      0.29 ± 69%    -100.0%       0.00        perf-sched.wait_time.avg.ms.do_task_dead.do_exit.do_group_exit.get_signal.arch_do_signal_or_restart
     69.43           +46.6%     101.82        perf-sched.wait_time.avg.ms.irq_thread.kthread.ret_from_fork
      1.63 ±  6%   +2390.3%      40.62 ±  2%  perf-sched.wait_time.avg.ms.pipe_read.vfs_read.ksys_read.do_syscall_64
     38.67 ± 13%    +876.3%     377.56        perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      0.07 ±223%    +576.2%       0.49 ±  7%  perf-sched.wait_time.avg.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone
      4.34 ±  6%    +578.4%      29.41 ±  6%  perf-sched.wait_time.avg.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
     32.27 ± 47%   +1969.2%     667.81        perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
     48.50 ± 11%    +749.6%     412.09 ±  4%  perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork
    299.06 ±  3%   +1569.4%       4992        perf-sched.wait_time.max.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity
    236.10 ± 28%     -98.5%       3.65 ± 12%  perf-sched.wait_time.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      0.00 ±223%  +5.5e+07%       1000        perf-sched.wait_time.max.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep
     69.82 ±139%   +2069.6%       1514 ± 31%  perf-sched.wait_time.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      0.55 ± 77%    -100.0%       0.00        perf-sched.wait_time.max.ms.do_task_dead.do_exit.do_group_exit.get_signal.arch_do_signal_or_restart
    238.01 ± 28%    +322.4%       1005        perf-sched.wait_time.max.ms.pipe_read.vfs_read.ksys_read.do_syscall_64
    292.84 ±  3%     +70.7%     499.97        perf-sched.wait_time.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      0.15 ±223%    +647.1%       1.09 ± 12%  perf-sched.wait_time.max.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone
      5.03         +6675.8%     340.74 ± 18%  perf-sched.wait_time.max.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
    305.61 ±  2%    +800.7%       2752 ± 29%  perf-sched.wait_time.max.ms.smpboot_thread_fn.kthread.ret_from_fork
    303.32 ±  4%    +705.6%       2443 ± 13%  perf-sched.wait_time.max.ms.worker_thread.kthread.ret_from_fork
     37.90 ± 63%     -36.6        1.29 ±  9%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
     37.90 ± 63%     -36.6        1.29 ±  8%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
     27.62 ± 86%     -27.5        0.16 ±173%  perf-profile.calltrace.cycles-pp.exit_mmap.__mmput.exit_mm.do_exit.do_group_exit
     24.63 ± 88%     -24.6        0.00        perf-profile.calltrace.cycles-pp.unmap_vmas.exit_mmap.__mmput.exit_mm.do_exit
     24.63 ± 88%     -24.6        0.00        perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.exit_mmap.__mmput.exit_mm
     24.63 ± 88%     -24.6        0.00        perf-profile.calltrace.cycles-pp.zap_pmd_range.unmap_page_range.unmap_vmas.exit_mmap.__mmput
     24.63 ± 88%     -24.6        0.00        perf-profile.calltrace.cycles-pp.zap_pte_range.zap_pmd_range.unmap_page_range.unmap_vmas.exit_mmap
     23.24 ± 82%     -23.2        0.00        perf-profile.calltrace.cycles-pp.do_group_exit.get_signal.arch_do_signal_or_restart.exit_to_user_mode_loop.exit_to_user_mode_prepare
     23.24 ± 82%     -23.2        0.00        perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.get_signal.arch_do_signal_or_restart.exit_to_user_mode_loop
     23.24 ± 82%     -23.2        0.00        perf-profile.calltrace.cycles-pp.exit_mm.do_exit.do_group_exit.get_signal.arch_do_signal_or_restart
     23.24 ± 82%     -23.2        0.00        perf-profile.calltrace.cycles-pp.__mmput.exit_mm.do_exit.do_group_exit.get_signal
     19.58 ±103%     -19.6        0.00        perf-profile.calltrace.cycles-pp.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
     18.70 ±111%     -18.7        0.00        perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
     18.70 ±111%     -18.7        0.00        perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
     18.70 ±111%     -18.7        0.00        perf-profile.calltrace.cycles-pp.arch_do_signal_or_restart.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
     18.70 ±111%     -18.7        0.00        perf-profile.calltrace.cycles-pp.get_signal.arch_do_signal_or_restart.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode
     10.10 ± 88%     -10.1        0.00        perf-profile.calltrace.cycles-pp.page_remove_rmap.zap_pte_range.zap_pmd_range.unmap_page_range.unmap_vmas
     10.23 ± 79%     -10.1        0.15 ±173%  perf-profile.calltrace.cycles-pp.__libc_start_main
     10.23 ± 79%     -10.1        0.15 ±173%  perf-profile.calltrace.cycles-pp.main.__libc_start_main
     10.23 ± 79%     -10.1        0.15 ±173%  perf-profile.calltrace.cycles-pp.run_builtin.main.__libc_start_main
      9.52 ± 97%      -9.3        0.18 ±173%  perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
      9.52 ± 97%      -9.3        0.19 ±173%  perf-profile.calltrace.cycles-pp.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
      9.52 ± 97%      -9.3        0.19 ±173%  perf-profile.calltrace.cycles-pp.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
      6.39 ± 87%      -6.4        0.00        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_write.writen.record__pushfn.perf_mmap__push
      6.39 ± 87%      -6.4        0.00        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write.writen.record__pushfn
      6.39 ± 87%      -6.4        0.00        perf-profile.calltrace.cycles-pp.__generic_file_write_iter.generic_file_write_iter.vfs_write.ksys_write.do_syscall_64
      6.39 ± 87%      -6.4        0.00        perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write.writen
      6.39 ± 87%      -6.4        0.00        perf-profile.calltrace.cycles-pp.generic_file_write_iter.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      6.39 ± 87%      -6.4        0.00        perf-profile.calltrace.cycles-pp.generic_perform_write.__generic_file_write_iter.generic_file_write_iter.vfs_write.ksys_write
      6.39 ± 87%      -6.4        0.00        perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
      6.39 ± 87%      -6.4        0.00        perf-profile.calltrace.cycles-pp.record__pushfn.perf_mmap__push.record__mmap_read_evlist.__cmd_record.cmd_record
      6.39 ± 87%      -6.4        0.00        perf-profile.calltrace.cycles-pp.writen.record__pushfn.perf_mmap__push.record__mmap_read_evlist.__cmd_record
      6.39 ± 87%      -6.4        0.00        perf-profile.calltrace.cycles-pp.__libc_write.writen.record__pushfn.perf_mmap__push.record__mmap_read_evlist
      6.15 ±101%      -6.1        0.00        perf-profile.calltrace.cycles-pp.cmd_record.run_builtin.main.__libc_start_main
      6.15 ±101%      -6.1        0.00        perf-profile.calltrace.cycles-pp.__cmd_record.cmd_record.run_builtin.main.__libc_start_main
      6.15 ±101%      -6.1        0.00        perf-profile.calltrace.cycles-pp.record__mmap_read_evlist.__cmd_record.cmd_record.run_builtin.main
      6.08 ± 79%      -6.1        0.00        perf-profile.calltrace.cycles-pp.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
      6.08 ± 79%      -6.1        0.00        perf-profile.calltrace.cycles-pp.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64
      4.63 ± 80%      -4.6        0.00        perf-profile.calltrace.cycles-pp.perf_mmap__push.record__mmap_read_evlist.__cmd_record.cmd_record.run_builtin
      3.84 ±104%      -3.8        0.00        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__mmap
      3.84 ±104%      -3.8        0.00        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__mmap
      3.84 ±104%      -3.8        0.00        perf-profile.calltrace.cycles-pp.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe.__mmap
      3.84 ±104%      -3.8        0.00        perf-profile.calltrace.cycles-pp.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe.__mmap
      3.84 ±104%      -3.8        0.00        perf-profile.calltrace.cycles-pp.__mmap
      0.00            +0.5        0.52 ±  3%  perf-profile.calltrace.cycles-pp.__run_timers.run_timer_softirq.__do_softirq.__irq_exit_rcu.sysvec_apic_timer_interrupt
      0.00            +0.5        0.52 ±  3%  perf-profile.calltrace.cycles-pp.run_timer_softirq.__do_softirq.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.00            +0.6        0.59 ±  9%  perf-profile.calltrace.cycles-pp.hrtimer_next_event_without.tick_nohz_get_sleep_length.menu_select.cpuidle_idle_call.do_idle
      0.00            +0.7        0.71 ± 15%  perf-profile.calltrace.cycles-pp._raw_spin_trylock.rebalance_domains.__do_softirq.__irq_exit_rcu.sysvec_apic_timer_interrupt
      0.00            +0.8        0.77 ± 25%  perf-profile.calltrace.cycles-pp.update_rq_clock_task.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer
      0.00            +0.8        0.79 ± 12%  perf-profile.calltrace.cycles-pp.update_blocked_averages.run_rebalance_domains.__do_softirq.__irq_exit_rcu.sysvec_apic_timer_interrupt
      0.00            +0.8        0.80 ± 23%  perf-profile.calltrace.cycles-pp.__break_lease.do_dentry_open.dentry_open.__nfsd_open.nfsd_file_do_acquire
      0.00            +0.8        0.80 ± 24%  perf-profile.calltrace.cycles-pp.__nfsd_open.nfsd_file_do_acquire.nfsd4_commit.nfsd4_proc_compound.nfsd_dispatch
      0.00            +0.8        0.80 ± 24%  perf-profile.calltrace.cycles-pp.dentry_open.__nfsd_open.nfsd_file_do_acquire.nfsd4_commit.nfsd4_proc_compound
      0.00            +0.8        0.80 ± 24%  perf-profile.calltrace.cycles-pp.do_dentry_open.dentry_open.__nfsd_open.nfsd_file_do_acquire.nfsd4_commit
      0.00            +0.8        0.80 ± 24%  perf-profile.calltrace.cycles-pp.nfsd.kthread.ret_from_fork
      0.00            +0.8        0.80 ± 24%  perf-profile.calltrace.cycles-pp.svc_process.nfsd.kthread.ret_from_fork
      0.00            +0.8        0.80 ± 24%  perf-profile.calltrace.cycles-pp.svc_process_common.svc_process.nfsd.kthread.ret_from_fork
      0.00            +0.8        0.80 ± 24%  perf-profile.calltrace.cycles-pp.nfsd_dispatch.svc_process_common.svc_process.nfsd.kthread
      0.00            +0.8        0.80 ± 24%  perf-profile.calltrace.cycles-pp.nfsd4_proc_compound.nfsd_dispatch.svc_process_common.svc_process.nfsd
      0.00            +0.8        0.80 ± 24%  perf-profile.calltrace.cycles-pp.nfsd4_commit.nfsd4_proc_compound.nfsd_dispatch.svc_process_common.svc_process
      0.00            +0.8        0.80 ± 24%  perf-profile.calltrace.cycles-pp.nfsd_file_do_acquire.nfsd4_commit.nfsd4_proc_compound.nfsd_dispatch.svc_process_common
      0.00            +0.8        0.81 ± 12%  perf-profile.calltrace.cycles-pp.cpuidle_governor_latency_req.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry
      0.00            +0.8        0.84 ± 12%  perf-profile.calltrace.cycles-pp.run_rebalance_domains.__do_softirq.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.00            +0.9        0.86 ±  8%  perf-profile.calltrace.cycles-pp.update_sg_lb_stats.update_sd_lb_stats.find_busiest_group.load_balance.rebalance_domains
      0.00            +1.1        1.10 ± 20%  perf-profile.calltrace.cycles-pp.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.cpuidle_idle_call.do_idle
      0.00            +1.1        1.11 ±  2%  perf-profile.calltrace.cycles-pp.update_sd_lb_stats.find_busiest_group.load_balance.rebalance_domains.__do_softirq
      0.00            +1.2        1.18 ±  5%  perf-profile.calltrace.cycles-pp.find_busiest_group.load_balance.rebalance_domains.__do_softirq.__irq_exit_rcu
      0.00            +1.3        1.27 ± 28%  perf-profile.calltrace.cycles-pp.tick_sched_do_timer.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
      0.00            +1.4        1.38 ± 49%  perf-profile.calltrace.cycles-pp.tick_irq_enter.irq_enter_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      0.00            +1.4        1.44 ± 46%  perf-profile.calltrace.cycles-pp.irq_enter_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      0.00            +1.4        1.45 ± 14%  perf-profile.calltrace.cycles-pp.__intel_pmu_enable_all.perf_rotate_context.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt
      0.00            +1.5        1.51 ± 12%  perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.rest_init
      0.00            +1.6        1.56        perf-profile.calltrace.cycles-pp.load_balance.rebalance_domains.__do_softirq.__irq_exit_rcu.sysvec_apic_timer_interrupt
      0.00            +1.6        1.59 ± 12%  perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.rest_init.arch_call_rest_init
      0.00            +1.6        1.59 ±  5%  perf-profile.calltrace.cycles-pp.lapic_next_deadline.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      0.00            +1.6        1.61 ± 13%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.rest_init.arch_call_rest_init.start_kernel
      0.00            +1.6        1.62 ± 12%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.rest_init.arch_call_rest_init.start_kernel.x86_64_start_reservations
      0.00            +1.6        1.62 ± 12%  perf-profile.calltrace.cycles-pp.x86_64_start_kernel.secondary_startup_64_no_verify
      0.00            +1.6        1.62 ± 12%  perf-profile.calltrace.cycles-pp.x86_64_start_reservations.x86_64_start_kernel.secondary_startup_64_no_verify
      0.00            +1.6        1.62 ± 12%  perf-profile.calltrace.cycles-pp.start_kernel.x86_64_start_reservations.x86_64_start_kernel.secondary_startup_64_no_verify
      0.00            +1.6        1.62 ± 12%  perf-profile.calltrace.cycles-pp.arch_call_rest_init.start_kernel.x86_64_start_reservations.x86_64_start_kernel.secondary_startup_64_no_verify
      0.00            +1.6        1.62 ± 12%  perf-profile.calltrace.cycles-pp.rest_init.arch_call_rest_init.start_kernel.x86_64_start_reservations.x86_64_start_kernel
      0.00            +1.7        1.72 ±  8%  perf-profile.calltrace.cycles-pp.ret_from_fork
      0.00            +1.7        1.72 ±  8%  perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
      0.00            +1.9        1.86 ± 10%  perf-profile.calltrace.cycles-pp.perf_rotate_context.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
      0.00            +2.0        1.97 ± 11%  perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry
      0.00            +2.2        2.22 ±  4%  perf-profile.calltrace.cycles-pp.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.00            +2.6        2.56 ±  5%  perf-profile.calltrace.cycles-pp.rebalance_domains.__do_softirq.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.00            +2.8        2.85 ±  7%  perf-profile.calltrace.cycles-pp.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      0.00            +3.1        3.10 ±  8%  perf-profile.calltrace.cycles-pp.arch_scale_freq_tick.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer
      0.00            +4.4        4.40 ±  6%  perf-profile.calltrace.cycles-pp.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
      0.00            +4.6        4.62 ±  5%  perf-profile.calltrace.cycles-pp.__do_softirq.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      0.88 ±223%      +4.7        5.61 ± 11%  perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
      0.00            +5.5        5.51 ±  4%  perf-profile.calltrace.cycles-pp.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      2.36 ±100%      +5.6        7.96 ±  9%  perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
      2.36 ±100%      +5.9        8.22 ±  9%  perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
      2.36 ±100%      +8.6       10.93 ±  6%  perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      2.36 ±100%     +13.5       15.87 ±  4%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.88 ±223%     +21.2       22.07 ±  4%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      0.88 ±223%     +21.3       22.22 ±  4%  perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      0.88 ±223%     +30.2       31.04 ±  5%  perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
      0.88 ±223%     +38.9       39.74        perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
      0.00           +42.2       42.16        perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
      0.88 ±223%     +81.5       82.34        perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
      0.88 ±223%     +82.4       83.24        perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
      2.48 ±158%     +87.7       90.16        perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      2.48 ±158%     +87.9       90.37        perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      2.48 ±158%     +87.9       90.38        perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
      0.88 ±223%     +88.1       88.95        perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      2.48 ±158%     +89.5       92.00        perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
     74.91 ± 19%     -71.0        3.92 ±  2%  perf-profile.children.cycles-pp.do_syscall_64
     74.91 ± 19%     -71.0        3.94 ±  2%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     28.80 ± 65%     -28.5        0.29 ± 33%  perf-profile.children.cycles-pp.zap_pte_range
     28.80 ± 65%     -28.5        0.31 ± 36%  perf-profile.children.cycles-pp.unmap_page_range
     28.80 ± 65%     -28.5        0.31 ± 36%  perf-profile.children.cycles-pp.zap_pmd_range
     28.80 ± 65%     -28.5        0.33 ± 29%  perf-profile.children.cycles-pp.unmap_vmas
     23.24 ± 82%     -23.2        0.00        perf-profile.children.cycles-pp.arch_do_signal_or_restart
     23.24 ± 82%     -23.2        0.00        perf-profile.children.cycles-pp.get_signal
      7.90 ± 90%      -7.9        0.01 ±173%  perf-profile.children.cycles-pp.record__mmap_read_evlist
      6.39 ± 87%      -6.4        0.01 ±173%  perf-profile.children.cycles-pp.generic_perform_write
      6.39 ± 87%      -6.4        0.01 ±173%  perf-profile.children.cycles-pp.perf_mmap__push
      6.39 ± 87%      -6.4        0.01 ±173%  perf-profile.children.cycles-pp.record__pushfn
      6.39 ± 87%      -6.4        0.01 ±173%  perf-profile.children.cycles-pp.writen
      6.39 ± 87%      -6.4        0.01 ±173%  perf-profile.children.cycles-pp.__libc_write
      6.39 ± 87%      -6.4        0.02 ±173%  perf-profile.children.cycles-pp.__generic_file_write_iter
      6.39 ± 87%      -6.4        0.02 ±173%  perf-profile.children.cycles-pp.generic_file_write_iter
      0.00            +0.1        0.06 ± 17%  perf-profile.children.cycles-pp.unlink_anon_vmas
      0.00            +0.1        0.06 ± 31%  perf-profile.children.cycles-pp.__get_user_pages
      0.00            +0.1        0.07 ± 28%  perf-profile.children.cycles-pp.cpuidle_not_available
      0.00            +0.1        0.07 ± 28%  perf-profile.children.cycles-pp.__lookup_slow
      0.00            +0.1        0.07 ± 26%  perf-profile.children.cycles-pp.check_move_unevictable_folios
      0.00            +0.1        0.07 ± 26%  perf-profile.children.cycles-pp.check_move_unevictable_pages
      0.00            +0.1        0.07 ± 14%  perf-profile.children.cycles-pp.__percpu_counter_init
      0.00            +0.1        0.07 ± 26%  perf-profile.children.cycles-pp.get_arg_page
      0.00            +0.1        0.07 ± 26%  perf-profile.children.cycles-pp.get_user_pages_remote
      0.00            +0.1        0.08 ± 26%  perf-profile.children.cycles-pp.drm_gem_shmem_put_pages_locked
      0.00            +0.1        0.08 ± 26%  perf-profile.children.cycles-pp.drm_gem_put_pages
      0.00            +0.1        0.08 ± 26%  perf-profile.children.cycles-pp.drm_gem_check_release_pagevec
      0.00            +0.1        0.08 ± 23%  perf-profile.children.cycles-pp.hrtimer_run_queues
      0.00            +0.1        0.08 ± 45%  perf-profile.children.cycles-pp.__x64_sys_mprotect
      0.00            +0.1        0.08 ± 45%  perf-profile.children.cycles-pp.do_mprotect_pkey
      0.00            +0.1        0.09 ± 33%  perf-profile.children.cycles-pp.lookup_fast
      0.00            +0.1        0.09 ± 29%  perf-profile.children.cycles-pp.rcu_do_batch
      0.00            +0.1        0.09 ± 32%  perf-profile.children.cycles-pp.can_stop_idle_tick
      0.00            +0.1        0.09 ± 32%  perf-profile.children.cycles-pp.__vmalloc_node_range
      0.00            +0.1        0.09 ± 32%  perf-profile.children.cycles-pp.alloc_thread_stack_node
      0.00            +0.1        0.09 ± 20%  perf-profile.children.cycles-pp.perf_read
      0.00            +0.1        0.10 ± 26%  perf-profile.children.cycles-pp.perf_evsel__read
      0.00            +0.1        0.10 ± 21%  perf-profile.children.cycles-pp.menu_reflect
      0.00            +0.1        0.10 ± 21%  perf-profile.children.cycles-pp.alloc_bprm
      0.00            +0.1        0.10 ± 12%  perf-profile.children.cycles-pp.vma_interval_tree_insert
      0.00            +0.1        0.10 ± 27%  perf-profile.children.cycles-pp.nohz_balancer_kick
      0.00            +0.1        0.10 ± 18%  perf-profile.children.cycles-pp.mm_init
      0.00            +0.1        0.10 ± 14%  perf-profile.children.cycles-pp.note_gp_changes
      0.00            +0.1        0.11 ± 21%  perf-profile.children.cycles-pp.cpuidle_reflect
      0.00            +0.1        0.11 ±  9%  perf-profile.children.cycles-pp.dup_task_struct
      0.00            +0.1        0.11 ± 52%  perf-profile.children.cycles-pp.find_idlest_cpu
      0.00            +0.1        0.11 ± 52%  perf-profile.children.cycles-pp.find_idlest_group
      0.00            +0.1        0.12 ± 11%  perf-profile.children.cycles-pp.drm_gem_vunmap_unlocked
      0.00            +0.1        0.12 ± 11%  perf-profile.children.cycles-pp.drm_gem_vunmap
      0.00            +0.1        0.12 ± 11%  perf-profile.children.cycles-pp.drm_gem_shmem_vunmap
      0.00            +0.1        0.12 ± 38%  perf-profile.children.cycles-pp.irq_work_run_list
      0.00            +0.1        0.12 ± 13%  perf-profile.children.cycles-pp.__switch_to_asm
      0.00            +0.1        0.13 ± 11%  perf-profile.children.cycles-pp.pipe_read
      0.00            +0.1        0.14 ±  8%  perf-profile.children.cycles-pp.exec_mmap
      0.00            +0.1        0.14 ± 44%  perf-profile.children.cycles-pp.tsc_verify_tsc_adjust
      0.00            +0.1        0.14 ± 14%  perf-profile.children.cycles-pp.irqentry_exit
      0.00            +0.1        0.14 ± 14%  perf-profile.children.cycles-pp.walk_component
      0.00            +0.1        0.14 ± 39%  perf-profile.children.cycles-pp.pick_next_task_fair
      0.00            +0.1        0.15 ± 25%  perf-profile.children.cycles-pp.smpboot_thread_fn
      0.00            +0.2        0.15 ± 51%  perf-profile.children.cycles-pp.update_wall_time
      0.00            +0.2        0.15 ± 51%  perf-profile.children.cycles-pp.timekeeping_advance
      0.00            +0.2        0.16 ± 47%  perf-profile.children.cycles-pp.copy_strings
      0.00            +0.2        0.16 ± 16%  perf-profile.children.cycles-pp.ct_kernel_exit
      0.00            +0.2        0.16 ± 40%  perf-profile.children.cycles-pp.select_task_rq_fair
      0.00            +0.2        0.16 ± 33%  perf-profile.children.cycles-pp.perf_event_task_tick
      0.00            +0.2        0.16 ±  4%  perf-profile.children.cycles-pp.__libc_read
      0.00            +0.2        0.16 ± 28%  perf-profile.children.cycles-pp.__x64_sys_vfork
      0.00            +0.2        0.16 ± 19%  perf-profile.children.cycles-pp.intel_idle_irq
      0.00            +0.2        0.16 ± 22%  perf-profile.children.cycles-pp.___perf_sw_event
      0.00            +0.2        0.16 ± 28%  perf-profile.children.cycles-pp.update_rt_rq_load_avg
      0.00            +0.2        0.16 ± 39%  perf-profile.children.cycles-pp.arch_cpu_idle_enter
      0.00            +0.2        0.16 ± 16%  perf-profile.children.cycles-pp._find_next_and_bit
      0.00            +0.2        0.16 ± 24%  perf-profile.children.cycles-pp.evsel__read_counter
      0.00            +0.2        0.17 ± 13%  perf-profile.children.cycles-pp.irqtime_account_process_tick
      0.00            +0.2        0.17 ± 49%  perf-profile.children.cycles-pp.shmem_read_folio_gfp
      0.00            +0.2        0.17 ± 33%  perf-profile.children.cycles-pp.rb_erase
      0.00            +0.2        0.18 ± 31%  perf-profile.children.cycles-pp.drm_atomic_commit
      0.00            +0.2        0.18 ± 31%  perf-profile.children.cycles-pp.drm_atomic_helper_commit
      0.00            +0.2        0.18 ± 16%  perf-profile.children.cycles-pp.hrtimer_forward
      0.00            +0.2        0.18 ±  4%  perf-profile.children.cycles-pp.readn
      0.00            +0.2        0.18 ± 30%  perf-profile.children.cycles-pp.sched_clock
      0.00            +0.2        0.18 ± 27%  perf-profile.children.cycles-pp.drm_atomic_helper_dirtyfb
      0.00            +0.2        0.19 ± 28%  perf-profile.children.cycles-pp.check_cpu_stall
      0.00            +0.2        0.19 ± 30%  perf-profile.children.cycles-pp.begin_new_exec
      0.00            +0.2        0.19 ± 20%  perf-profile.children.cycles-pp.tick_nohz_tick_stopped
      0.00            +0.2        0.19 ± 19%  perf-profile.children.cycles-pp.__vfork
      0.00            +0.2        0.20 ± 34%  perf-profile.children.cycles-pp.drm_gem_shmem_get_pages
      0.00            +0.2        0.20 ± 34%  perf-profile.children.cycles-pp.drm_gem_shmem_get_pages_locked
      0.00            +0.2        0.20 ± 34%  perf-profile.children.cycles-pp.drm_gem_get_pages
      0.00            +0.2        0.20 ± 34%  perf-profile.children.cycles-pp.shmem_read_mapping_page_gfp
      0.00            +0.2        0.20 ± 18%  perf-profile.children.cycles-pp.rb_next
      0.00            +0.2        0.20 ± 39%  perf-profile.children.cycles-pp.drm_gem_vmap_unlocked
      0.00            +0.2        0.20 ± 39%  perf-profile.children.cycles-pp.drm_gem_vmap
      0.00            +0.2        0.20 ± 39%  perf-profile.children.cycles-pp.drm_gem_shmem_object_vmap
      0.00            +0.2        0.20 ± 39%  perf-profile.children.cycles-pp.drm_gem_shmem_vmap_locked
      0.00            +0.2        0.21 ± 45%  perf-profile.children.cycles-pp.run_posix_cpu_timers
      0.00            +0.2        0.21 ± 20%  perf-profile.children.cycles-pp.tick_program_event
      0.00            +0.2        0.22 ±102%  perf-profile.children.cycles-pp.tick_check_oneshot_broadcast_this_cpu
      0.00            +0.2        0.23 ± 23%  perf-profile.children.cycles-pp.ttwu_do_activate
      0.00            +0.2        0.23 ± 25%  perf-profile.children.cycles-pp.arch_cpu_idle_exit
      0.00            +0.2        0.24 ± 15%  perf-profile.children.cycles-pp.rb_insert_color
      0.00            +0.2        0.24 ± 45%  perf-profile.children.cycles-pp.irqentry_enter
      0.00            +0.3        0.26 ±  7%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.00            +0.3        0.27 ± 16%  perf-profile.children.cycles-pp.rcu_core
      0.00            +0.3        0.29 ±  9%  perf-profile.children.cycles-pp.error_entry
      0.00            +0.3        0.31 ± 15%  perf-profile.children.cycles-pp.call_cpuidle
      0.00            +0.3        0.33 ± 22%  perf-profile.children.cycles-pp.drm_fb_helper_damage_work
      0.00            +0.3        0.33 ± 22%  perf-profile.children.cycles-pp.drm_fbdev_generic_helper_fb_dirty
      0.00            +0.3        0.33 ± 27%  perf-profile.children.cycles-pp.sched_clock_idle_wakeup_event
      0.00            +0.3        0.33 ± 18%  perf-profile.children.cycles-pp.ct_kernel_exit_state
      0.00            +0.3        0.35 ± 47%  perf-profile.children.cycles-pp.ct_kernel_enter
      0.00            +0.3        0.35 ± 17%  perf-profile.children.cycles-pp.hrtimer_update_next_event
      0.00            +0.4        0.37 ± 16%  perf-profile.children.cycles-pp.idle_cpu
      0.00            +0.4        0.40 ± 30%  perf-profile.children.cycles-pp.hrtimer_get_next_event
      0.00            +0.4        0.40 ± 15%  perf-profile.children.cycles-pp.irq_work_tick
      0.00            +0.4        0.40 ± 20%  perf-profile.children.cycles-pp.timerqueue_del
      0.00            +0.4        0.43 ± 10%  perf-profile.children.cycles-pp.call_timer_fn
      0.00            +0.4        0.44 ±  7%  perf-profile.children.cycles-pp.read_counters
      0.00            +0.4        0.44 ± 45%  perf-profile.children.cycles-pp.ct_idle_exit
      0.00            +0.5        0.48 ±  8%  perf-profile.children.cycles-pp.cmd_stat
      0.00            +0.5        0.48 ±  8%  perf-profile.children.cycles-pp.dispatch_events
      0.00            +0.5        0.48 ±  8%  perf-profile.children.cycles-pp.process_interval
      0.00            +0.5        0.50 ± 18%  perf-profile.children.cycles-pp.process_one_work
      0.00            +0.5        0.51 ± 23%  perf-profile.children.cycles-pp.__hrtimer_next_event_base
      0.00            +0.5        0.51 ± 10%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.00            +0.5        0.52 ±  3%  perf-profile.children.cycles-pp.run_timer_softirq
      0.00            +0.5        0.52 ± 27%  perf-profile.children.cycles-pp.schedule_timeout
      0.00            +0.5        0.53 ± 16%  perf-profile.children.cycles-pp.worker_thread
      0.00            +0.5        0.53 ± 28%  perf-profile.children.cycles-pp.timerqueue_add
      0.00            +0.6        0.56 ± 18%  perf-profile.children.cycles-pp.schedule
      0.00            +0.6        0.58 ±  3%  perf-profile.children.cycles-pp.__run_timers
      0.00            +0.6        0.58 ± 18%  perf-profile.children.cycles-pp.get_cpu_device
      0.00            +0.6        0.59 ± 14%  perf-profile.children.cycles-pp.tick_nohz_irq_exit
      0.00            +0.6        0.60 ± 16%  perf-profile.children.cycles-pp.update_rq_clock
      0.00            +0.6        0.62 ± 61%  perf-profile.children.cycles-pp.calc_global_load_tick
      0.00            +0.7        0.66 ± 21%  perf-profile.children.cycles-pp.enqueue_hrtimer
      0.00            +0.7        0.66 ± 17%  perf-profile.children.cycles-pp.local_clock
      0.00            +0.7        0.68 ±  7%  perf-profile.children.cycles-pp.hrtimer_next_event_without
      0.00            +0.7        0.70 ± 24%  perf-profile.children.cycles-pp.get_next_timer_interrupt
      0.00            +0.7        0.74 ± 15%  perf-profile.children.cycles-pp._raw_spin_trylock
      0.00            +0.7        0.75 ± 29%  perf-profile.children.cycles-pp.update_irq_load_avg
      0.00            +0.8        0.80 ± 24%  perf-profile.children.cycles-pp.__nfsd_open
      0.00            +0.8        0.80 ± 24%  perf-profile.children.cycles-pp.dentry_open
      0.00            +0.8        0.80 ± 24%  perf-profile.children.cycles-pp.__break_lease
      0.00            +0.8        0.80 ± 24%  perf-profile.children.cycles-pp.nfsd
      0.00            +0.8        0.80 ± 24%  perf-profile.children.cycles-pp.svc_process
      0.00            +0.8        0.80 ± 24%  perf-profile.children.cycles-pp.svc_process_common
      0.00            +0.8        0.80 ± 24%  perf-profile.children.cycles-pp.nfsd_dispatch
      0.00            +0.8        0.80 ± 24%  perf-profile.children.cycles-pp.nfsd4_proc_compound
      0.00            +0.8        0.80 ± 24%  perf-profile.children.cycles-pp.nfsd4_commit
      0.00            +0.8        0.80 ± 24%  perf-profile.children.cycles-pp.nfsd_file_do_acquire
      0.00            +0.8        0.81 ± 13%  perf-profile.children.cycles-pp.update_blocked_averages
      0.00            +0.8        0.82 ± 25%  perf-profile.children.cycles-pp.do_dentry_open
      0.00            +0.8        0.84 ± 12%  perf-profile.children.cycles-pp.run_rebalance_domains
      0.00            +0.9        0.89 ± 15%  perf-profile.children.cycles-pp.cpuidle_governor_latency_req
      0.00            +1.0        0.98 ±  8%  perf-profile.children.cycles-pp.update_sg_lb_stats
      0.00            +1.0        0.98 ± 24%  perf-profile.children.cycles-pp.update_rq_clock_task
      0.00            +1.1        1.08 ±  9%  perf-profile.children.cycles-pp.irqtime_account_irq
      0.00            +1.2        1.16 ± 17%  perf-profile.children.cycles-pp.tick_nohz_next_event
      0.00            +1.2        1.17 ± 14%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.00            +1.3        1.25 ±  2%  perf-profile.children.cycles-pp.update_sd_lb_stats
      0.00            +1.3        1.31 ±  2%  perf-profile.children.cycles-pp.find_busiest_group
      0.00            +1.4        1.35 ± 28%  perf-profile.children.cycles-pp.tick_sched_do_timer
      0.00            +1.5        1.46 ± 48%  perf-profile.children.cycles-pp.tick_irq_enter
      0.00            +1.5        1.46 ±  3%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.00            +1.5        1.51 ± 12%  perf-profile.children.cycles-pp.__intel_pmu_enable_all
      0.00            +1.5        1.53 ± 42%  perf-profile.children.cycles-pp.irq_enter_rcu
      0.00            +1.6        1.62 ± 12%  perf-profile.children.cycles-pp.x86_64_start_kernel
      0.00            +1.6        1.62 ± 12%  perf-profile.children.cycles-pp.x86_64_start_reservations
      0.00            +1.6        1.62 ± 12%  perf-profile.children.cycles-pp.start_kernel
      0.00            +1.6        1.62 ± 12%  perf-profile.children.cycles-pp.arch_call_rest_init
      0.00            +1.6        1.62 ± 12%  perf-profile.children.cycles-pp.rest_init
      0.00            +1.7        1.70 ±  7%  perf-profile.children.cycles-pp.native_sched_clock
      0.00            +1.7        1.70 ±  3%  perf-profile.children.cycles-pp.load_balance
      0.00            +1.7        1.71 ±  5%  perf-profile.children.cycles-pp.lapic_next_deadline
      0.00            +1.7        1.72 ±  8%  perf-profile.children.cycles-pp.kthread
      0.00            +1.8        1.76 ±  7%  perf-profile.children.cycles-pp.ret_from_fork
      0.00            +1.9        1.95 ± 16%  perf-profile.children.cycles-pp.read_tsc
      0.00            +2.0        1.98 ±  9%  perf-profile.children.cycles-pp.perf_rotate_context
      0.00            +2.0        2.04 ± 11%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.00            +2.3        2.30 ±  5%  perf-profile.children.cycles-pp.clockevents_program_event
      0.00            +2.6        2.60 ±  6%  perf-profile.children.cycles-pp.rebalance_domains
      0.00            +3.0        2.99 ±  7%  perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
      0.00            +3.2        3.18 ±  7%  perf-profile.children.cycles-pp.arch_scale_freq_tick
      0.00            +3.9        3.92 ± 31%  perf-profile.children.cycles-pp.ktime_get
      0.00            +4.5        4.53 ±  6%  perf-profile.children.cycles-pp.menu_select
      0.00            +4.7        4.71 ±  6%  perf-profile.children.cycles-pp.__do_softirq
      0.88 ±223%      +5.0        5.86 ± 11%  perf-profile.children.cycles-pp.scheduler_tick
      0.00            +5.6        5.64 ±  4%  perf-profile.children.cycles-pp.__irq_exit_rcu
      2.36 ±100%      +5.8        8.15 ±  9%  perf-profile.children.cycles-pp.update_process_times
      2.36 ±100%      +6.0        8.35 ±  9%  perf-profile.children.cycles-pp.tick_sched_handle
      2.36 ±100%      +8.8       11.21 ±  6%  perf-profile.children.cycles-pp.tick_sched_timer
      2.36 ±100%     +13.9       16.30 ±  3%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      3.24 ±117%     +19.3       22.55 ±  4%  perf-profile.children.cycles-pp.hrtimer_interrupt
      3.24 ±117%     +19.4       22.66 ±  4%  perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
      3.24 ±117%     +28.0       31.28 ±  4%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
      3.24 ±117%     +32.7       35.97 ±  2%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
      0.00           +42.2       42.16        perf-profile.children.cycles-pp.intel_idle
      0.88 ±223%     +83.5       84.34        perf-profile.children.cycles-pp.cpuidle_enter_state
      0.88 ±223%     +83.9       84.78        perf-profile.children.cycles-pp.cpuidle_enter
      2.48 ±158%     +87.9       90.38        perf-profile.children.cycles-pp.start_secondary
      2.48 ±158%     +89.5       92.00        perf-profile.children.cycles-pp.secondary_startup_64_no_verify
      2.48 ±158%     +89.5       92.00        perf-profile.children.cycles-pp.cpu_startup_entry
      2.48 ±158%     +89.5       92.00        perf-profile.children.cycles-pp.do_idle
      0.88 ±223%     +89.8       90.65        perf-profile.children.cycles-pp.cpuidle_idle_call
      0.00            +0.1        0.06 ± 20%  perf-profile.self.cycles-pp.shmem_get_folio_gfp
      0.00            +0.1        0.07 ± 19%  perf-profile.self.cycles-pp.tick_irq_enter
      0.00            +0.1        0.07 ± 26%  perf-profile.self.cycles-pp.check_move_unevictable_folios
      0.00            +0.1        0.07 ± 33%  perf-profile.self.cycles-pp.perf_event_task_tick
      0.00            +0.1        0.08 ± 19%  perf-profile.self.cycles-pp.hrtimer_run_queues
      0.00            +0.1        0.08 ± 14%  perf-profile.self.cycles-pp.irqentry_exit
      0.00            +0.1        0.08 ± 21%  perf-profile.self.cycles-pp.hrtimer_get_next_event
      0.00            +0.1        0.08 ± 26%  perf-profile.self.cycles-pp.can_stop_idle_tick
      0.00            +0.1        0.09 ± 29%  perf-profile.self.cycles-pp.irqentry_enter
      0.00            +0.1        0.09 ± 43%  perf-profile.self.cycles-pp.update_blocked_averages
      0.00            +0.1        0.10 ± 12%  perf-profile.self.cycles-pp.vma_interval_tree_insert
      0.00            +0.1        0.10 ± 34%  perf-profile.self.cycles-pp.irq_work_run_list
      0.00            +0.1        0.10 ± 43%  perf-profile.self.cycles-pp.ct_kernel_exit
      0.00            +0.1        0.10 ± 41%  perf-profile.self.cycles-pp.__sysvec_apic_timer_interrupt
      0.00            +0.1        0.11 ± 13%  perf-profile.self.cycles-pp.tick_sched_timer
      0.00            +0.1        0.11 ± 42%  perf-profile.self.cycles-pp.filemap_map_pages
      0.00            +0.1        0.12 ± 26%  perf-profile.self.cycles-pp.rb_erase
      0.00            +0.1        0.12 ± 56%  perf-profile.self.cycles-pp.tsc_verify_tsc_adjust
      0.00            +0.1        0.12 ± 13%  perf-profile.self.cycles-pp.__switch_to_asm
      0.00            +0.1        0.13 ± 21%  perf-profile.self.cycles-pp.hrtimer_update_next_event
      0.00            +0.1        0.13 ± 36%  perf-profile.self.cycles-pp.hrtimer_forward
      0.00            +0.1        0.13 ± 25%  perf-profile.self.cycles-pp.rb_next
      0.00            +0.1        0.13 ± 42%  perf-profile.self.cycles-pp.update_rq_clock
      0.00            +0.1        0.14 ± 34%  perf-profile.self.cycles-pp.sysvec_apic_timer_interrupt
      0.00            +0.1        0.14 ± 23%  perf-profile.self.cycles-pp.enqueue_hrtimer
      0.00            +0.1        0.14 ± 32%  perf-profile.self.cycles-pp.tick_nohz_get_sleep_length
      0.00            +0.1        0.14 ± 15%  perf-profile.self.cycles-pp.rcu_sched_clock_irq
      0.00            +0.1        0.15 ± 40%  perf-profile.self.cycles-pp.clockevents_program_event
      0.00            +0.1        0.15 ± 23%  perf-profile.self.cycles-pp.tick_nohz_tick_stopped
      0.00            +0.2        0.15 ± 21%  perf-profile.self.cycles-pp.___perf_sw_event
      0.00            +0.2        0.16 ± 14%  perf-profile.self.cycles-pp.irqtime_account_process_tick
      0.00            +0.2        0.16 ± 19%  perf-profile.self.cycles-pp._find_next_and_bit
      0.00            +0.2        0.16 ± 23%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.00            +0.2        0.16 ± 25%  perf-profile.self.cycles-pp.update_rt_rq_load_avg
      0.00            +0.2        0.16 ± 19%  perf-profile.self.cycles-pp.intel_idle_irq
      0.00            +0.2        0.16 ± 17%  perf-profile.self.cycles-pp.hrtimer_next_event_without
      0.00            +0.2        0.18 ± 45%  perf-profile.self.cycles-pp.timerqueue_del
      0.00            +0.2        0.18 ± 27%  perf-profile.self.cycles-pp.check_cpu_stall
      0.00            +0.2        0.18 ± 36%  perf-profile.self.cycles-pp.update_sd_lb_stats
      0.00            +0.2        0.19 ± 44%  perf-profile.self.cycles-pp.run_posix_cpu_timers
      0.00            +0.2        0.19 ± 18%  perf-profile.self.cycles-pp.perf_mux_hrtimer_handler
      0.00            +0.2        0.20 ± 47%  perf-profile.self.cycles-pp.__do_softirq
      0.00            +0.2        0.20 ± 23%  perf-profile.self.cycles-pp.tick_program_event
      0.00            +0.2        0.20 ± 26%  perf-profile.self.cycles-pp.rb_insert_color
      0.00            +0.2        0.20 ±  8%  perf-profile.self.cycles-pp.sched_clock_cpu
      0.00            +0.2        0.21 ± 41%  perf-profile.self.cycles-pp.get_next_timer_interrupt
      0.00            +0.2        0.21 ± 60%  perf-profile.self.cycles-pp.ct_kernel_enter
      0.00            +0.2        0.22 ±102%  perf-profile.self.cycles-pp.tick_check_oneshot_broadcast_this_cpu
      0.00            +0.2        0.25 ± 26%  perf-profile.self.cycles-pp.update_rq_clock_task
      0.00            +0.2        0.25 ± 26%  perf-profile.self.cycles-pp.__irq_exit_rcu
      0.00            +0.3        0.26 ± 38%  perf-profile.self.cycles-pp.rebalance_domains
      0.00            +0.3        0.26 ± 33%  perf-profile.self.cycles-pp.cpuidle_governor_latency_req
      0.00            +0.3        0.26 ± 25%  perf-profile.self.cycles-pp.perf_rotate_context
      0.00            +0.3        0.28 ± 20%  perf-profile.self.cycles-pp.tick_nohz_next_event
      0.00            +0.3        0.29 ±  9%  perf-profile.self.cycles-pp.error_entry
      0.00            +0.3        0.29 ± 38%  perf-profile.self.cycles-pp.hrtimer_interrupt
      0.00            +0.3        0.30 ± 18%  perf-profile.self.cycles-pp.call_cpuidle
      0.00            +0.3        0.30 ± 16%  perf-profile.self.cycles-pp.load_balance
      0.00            +0.3        0.31 ± 22%  perf-profile.self.cycles-pp.ct_kernel_exit_state
      0.00            +0.3        0.32 ± 20%  perf-profile.self.cycles-pp.irqtime_account_irq
      0.00            +0.3        0.32 ± 31%  perf-profile.self.cycles-pp.timerqueue_add
      0.00            +0.3        0.32 ± 28%  perf-profile.self.cycles-pp.sched_clock_idle_wakeup_event
      0.00            +0.4        0.35 ± 20%  perf-profile.self.cycles-pp.scheduler_tick
      0.00            +0.4        0.36 ± 16%  perf-profile.self.cycles-pp.idle_cpu
      0.00            +0.4        0.38 ± 17%  perf-profile.self.cycles-pp.irq_work_tick
      0.00            +0.4        0.39 ± 20%  perf-profile.self.cycles-pp.cpuidle_enter
      0.00            +0.4        0.40 ± 18%  perf-profile.self.cycles-pp.__hrtimer_next_event_base
      0.00            +0.4        0.45 ± 36%  perf-profile.self.cycles-pp.do_idle
      0.00            +0.5        0.50 ±  7%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.00            +0.5        0.51 ± 18%  perf-profile.self.cycles-pp.__hrtimer_run_queues
      0.00            +0.5        0.53 ± 17%  perf-profile.self.cycles-pp.get_cpu_device
      0.00            +0.5        0.54 ±  8%  perf-profile.self.cycles-pp.update_process_times
      0.00            +0.6        0.56 ± 27%  perf-profile.self.cycles-pp.asm_sysvec_apic_timer_interrupt
      0.00            +0.6        0.62 ± 61%  perf-profile.self.cycles-pp.calc_global_load_tick
      0.00            +0.7        0.69 ± 16%  perf-profile.self.cycles-pp.cpuidle_idle_call
      0.00            +0.7        0.73 ± 14%  perf-profile.self.cycles-pp._raw_spin_trylock
      0.00            +0.7        0.73 ± 29%  perf-profile.self.cycles-pp.update_irq_load_avg
      0.00            +0.7        0.74 ±  9%  perf-profile.self.cycles-pp.update_sg_lb_stats
      0.00            +1.1        1.10 ± 14%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.00            +1.2        1.18 ± 31%  perf-profile.self.cycles-pp.tick_sched_do_timer
      0.00            +1.4        1.45 ±  9%  perf-profile.self.cycles-pp.menu_select
      0.00            +1.5        1.51 ± 12%  perf-profile.self.cycles-pp.__intel_pmu_enable_all
      0.00            +1.6        1.63 ±  9%  perf-profile.self.cycles-pp.native_sched_clock
      0.00            +1.7        1.70 ±  4%  perf-profile.self.cycles-pp.lapic_next_deadline
      0.00            +1.9        1.93 ± 16%  perf-profile.self.cycles-pp.read_tsc
      0.00            +2.3        2.30 ± 52%  perf-profile.self.cycles-pp.ktime_get
      0.00            +3.2        3.18 ±  7%  perf-profile.self.cycles-pp.arch_scale_freq_tick
      0.00            +5.8        5.84 ±  9%  perf-profile.self.cycles-pp.cpuidle_enter_state
      0.00           +42.2       42.16        perf-profile.self.cycles-pp.intel_idle


***************************************************************************************************
lkp-icl-2sp7: 64 threads 2 sockets Intel(R) Xeon(R) Gold 6346 CPU @ 3.10GHz (Ice Lake) with 256G memory
=========================================================================================
compiler/cpufreq_governor/debug-setup/disk/filesize/fs2/fs/iterations/kconfig/nr_directories/nr_files_per_directory/nr_threads/rootfs/sync_method/tbox_group/test_size/testcase:
  gcc-12/performance/no-monitor/1SSD/16MB/nfsv4/btrfs/1x/x86_64-rhel-8.3/16d/256fpd/32t/debian-11.1-x86_64-20220510.cgz/NoSync/lkp-icl-2sp7/20G/fsmark

commit: 
  3434d7aa77 ("NFSD: Clean up nfsctl_transaction_write()")
  39d432fc76 ("NFSD: trace nfsctl operations")

3434d7aa77d24c5c 39d432fc76301cf0a0c45402211 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
   7488368 ±  2%   +7979.7%   6.05e+08 ± 47%  fsmark.app_overhead
     62.42 ±  2%    -100.0%       0.00        fsmark.files_per_sec
     22.46 ±  2%  +15766.1%       3564 ±  5%  fsmark.time.elapsed_time
     22.46 ±  2%  +15766.1%       3564 ±  5%  fsmark.time.elapsed_time.max
    192.33           -99.5%       1.00        fsmark.time.percent_of_cpu_this_job_got
     43.05           +13.7%      48.94 ±  4%  fsmark.time.system_time
    978884            +2.9%    1007574 ±  2%  fsmark.time.voluntary_context_switches



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ