lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Message-ID: <20211202084407.GB30817@xsang-OptiPlex-9020>
Date:   Thu, 2 Dec 2021 16:44:07 +0800
From:   kernel test robot <oliver.sang@...el.com>
To:     Jaegeuk Kim <jaegeuk@...nel.org>
Cc:     LKML <linux-kernel@...r.kernel.org>,
        Linux Memory Management List <linux-mm@...ck.org>,
        lkp@...ts.01.org, lkp@...el.com, ying.huang@...el.com,
        feng.tang@...el.com, zhengjun.xing@...ux.intel.com,
        fengwei.yin@...el.com
Subject: [f2fs]  4fa18391ae:  aim7.jobs-per-min -35.5% regression



Greeting,

FYI, we noticed a -35.5% regression of aim7.jobs-per-min due to commit:


commit: 4fa18391ae8cf8f1e498d54b46c571aebf3aa806 ("f2fs: do not expose unwritten blocks to user by DIO")
https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git master

in testcase: aim7
on test machine: 144 threads 4 sockets Intel(R) Xeon(R) Gold 5318H CPU @ 2.50GHz with 128G memory
with following parameters:

	disk: 1BRD_48G
	fs: f2fs
	test: disk_cp
	load: 3000
	cpufreq_governor: performance
	ucode: 0x700001e

test-description: AIM7 is a traditional UNIX system level benchmark suite which is used to test and measure the performance of multiuser system.
test-url: https://sourceforge.net/projects/aimbench/files/aim-suite7/

In addition to that, the commit also has significant impact on the following tests:

+------------------+---------------------------------------------------------------------------------+
| testcase: change | aim7: aim7.jobs-per-min -53.2% regression                                       |
| test machine     | 88 threads 2 sockets Intel(R) Xeon(R) Gold 6238M CPU @ 2.10GHz with 128G memory |
| test parameters  | cpufreq_governor=performance                                                    |
|                  | disk=4BRD_12G                                                                   |
|                  | fs=f2fs                                                                         |
|                  | load=3000                                                                       |
|                  | md=RAID1                                                                        |
|                  | test=disk_rw                                                                    |
|                  | ucode=0x5003006                                                                 |
+------------------+---------------------------------------------------------------------------------+


If you fix the issue, kindly add following tag
Reported-by: kernel test robot <oliver.sang@...el.com>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        sudo bin/lkp install job.yaml           # job file is attached in this email
        bin/lkp split-job --compatible job.yaml # generate the yaml file for lkp run
        sudo bin/lkp run generated-yaml-file

        # if come across any failure that blocks the test,
        # please remove ~/.lkp and /lkp dir to run from a clean state.

=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/load/rootfs/tbox_group/test/testcase/ucode:
  gcc-9/performance/1BRD_48G/f2fs/x86_64-rhel-8.3/3000/debian-10.4-x86_64-20200603.cgz/lkp-cpl-4sp1/disk_cp/aim7/0x700001e

commit: 
  3d697a4a6b ("f2fs: rework write preallocations")
  4fa18391ae ("f2fs: do not expose unwritten blocks to user by DIO")

3d697a4a6b7dab8f 4fa18391ae8cf8f1e498d54b46c 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
     74254           -35.5%      47877 ±  2%  aim7.jobs-per-min
    242.67           +55.1%     376.33 ±  2%  aim7.time.elapsed_time
    242.67           +55.1%     376.33 ±  2%  aim7.time.elapsed_time.max
   2145257 ±  5%    +193.1%    6287243 ±  5%  aim7.time.involuntary_context_switches
    297540 ±  2%     +25.9%     374643 ±  3%  aim7.time.minor_page_faults
     32890           +52.7%      50236 ±  2%  aim7.time.system_time
     27.09           +27.3%      34.49 ±  2%  aim7.time.user_time
    421880 ± 21%   +2657.0%   11631366 ± 11%  aim7.time.voluntary_context_switches
      6.03 ±  2%     +12.9%       6.81 ±  4%  iostat.cpu.idle
 1.847e+09 ±  2%     +85.7%  3.428e+09 ±  4%  cpuidle..time
   3876146 ± 14%    +226.0%   12637810 ± 12%  cpuidle..usage
   2826103 ± 46%    +274.8%   10593320 ± 21%  turbostat.C1E
  73187009           +55.6%  1.139e+08 ±  4%  turbostat.IRQ
    299.26           +44.6%     432.71 ±  2%  uptime.boot
      9119 ±  4%     +17.1%      10680 ±  3%  uptime.idle
      5.28 ±  2%      +1.0        6.32 ±  4%  mpstat.cpu.all.idle%
      0.00 ± 65%      -0.0        0.00 ± 83%  mpstat.cpu.all.iowait%
      0.02 ±  6%      +0.0        0.03 ±  6%  mpstat.cpu.all.soft%
      5206           -17.8%       4279 ±  2%  vmstat.io.bo
   3449254           +55.8%    5375641        vmstat.memory.cache
    267.50 ±  3%     +55.8%     416.67 ±  3%  vmstat.procs.r
     13082 ±  3%    +396.8%      64991 ±  7%  vmstat.system.cs
      4053 ±  2%     +65.0%       6688 ±  2%  slabinfo.f2fs_inode_cache.active_objs
      4053 ±  2%     +68.9%       6845 ±  2%  slabinfo.f2fs_inode_cache.num_objs
     30617 ±  2%      -7.8%      28224 ±  4%  slabinfo.lsm_file_cache.active_objs
     43090           +16.7%      50281        slabinfo.radix_tree_node.active_objs
    775.17           +22.9%     952.50        slabinfo.radix_tree_node.active_slabs
     43439           +22.9%      53368        slabinfo.radix_tree_node.num_objs
    775.17           +22.9%     952.50        slabinfo.radix_tree_node.num_slabs
      5863 ± 10%     +14.7%       6728 ±  3%  slabinfo.shmem_inode_cache.active_objs
      5863 ± 10%     +14.7%       6728 ±  3%  slabinfo.shmem_inode_cache.num_objs
    510198 ±  3%    +286.5%    1971756 ±  2%  meminfo.Active
    504595 ±  3%    +289.3%    1964178 ±  2%  meminfo.Active(anon)
      5602 ±  3%     +35.2%       7577        meminfo.Active(file)
   3279800           +58.5%    5199801        meminfo.Cached
   3235772           +50.3%    4863430        meminfo.Committed_AS
    127149 ±  2%    +249.3%     444125 ±  6%  meminfo.Dirty
    569998           +80.8%    1030479 ±  3%  meminfo.Inactive
    445430           +32.0%     587968 ±  6%  meminfo.Inactive(anon)
    124568 ±  2%    +255.2%     442511 ±  6%  meminfo.Inactive(file)
    116537 ±  2%     +95.7%     228065 ± 22%  meminfo.Mapped
   6080907           +42.7%    8677900 ±  7%  meminfo.Memused
    578794 ±  2%    +276.3%    2178122        meminfo.Shmem
   6531876           +58.6%   10359908 ±  5%  meminfo.max_used_kB
     32624 ±  3%    +243.2%     111961 ±  6%  numa-meminfo.node0.Dirty
     32163 ±  3%    +247.1%     111637 ±  6%  numa-meminfo.node0.Inactive(file)
     39026 ±114%    +415.2%     201066 ± 25%  numa-meminfo.node1.Active
     38236 ±117%    +420.7%     199109 ± 25%  numa-meminfo.node1.Active(anon)
     31896 ±  3%    +252.2%     112348 ±  6%  numa-meminfo.node1.Dirty
    128417 ± 83%    +265.7%     469606 ± 35%  numa-meminfo.node1.FilePages
     31133 ±  3%    +259.4%     111876 ±  6%  numa-meminfo.node1.Inactive(file)
     17929 ± 33%    +104.0%      36582 ± 16%  numa-meminfo.node1.Mapped
    668841 ± 34%    +107.6%    1388183 ± 37%  numa-meminfo.node1.MemUsed
     47191 ±117%    +378.5%     225818 ± 22%  numa-meminfo.node1.Shmem
     14889 ± 91%   +1353.8%     216463 ± 16%  numa-meminfo.node2.Active
     13790 ± 96%   +1458.0%     214844 ± 17%  numa-meminfo.node2.Active(anon)
     32130 ±  3%    +249.6%     112328 ±  6%  numa-meminfo.node2.Dirty
     31451 ±  4%    +255.5%     111802 ±  6%  numa-meminfo.node2.Inactive(file)
     18670 ± 86%   +1222.0%     246811 ± 20%  numa-meminfo.node2.Shmem
    345708 ± 20%    +278.7%    1309082 ±  9%  numa-meminfo.node3.Active
    344607 ± 20%    +279.4%    1307362 ±  9%  numa-meminfo.node3.Active(anon)
     32034 ±  4%    +249.0%     111794 ±  6%  numa-meminfo.node3.Dirty
    483217 ± 26%    +310.2%    1982305 ± 42%  numa-meminfo.node3.FilePages
     31339 ±  4%    +255.6%     111429 ±  6%  numa-meminfo.node3.Inactive(file)
     30679 ± 28%    +218.0%      97565 ± 37%  numa-meminfo.node3.Mapped
  31871824           -17.7%   26220660 ± 15%  numa-meminfo.node3.MemFree
   1105046 ± 36%    +137.8%    2627440 ± 33%  numa-meminfo.node3.MemUsed
    377293 ± 21%    +280.9%    1437212 ±  9%  numa-meminfo.node3.Shmem
    126093 ±  3%    +289.2%     490726 ±  2%  proc-vmstat.nr_active_anon
      1399 ±  3%     +35.4%       1895        proc-vmstat.nr_active_file
     32182 ±  3%    +245.4%     111167 ±  6%  proc-vmstat.nr_dirty
   3124644           -34.7%    2040439 ± 51%  proc-vmstat.nr_dirty_background_threshold
   6256929           -34.7%    4085867 ± 51%  proc-vmstat.nr_dirty_threshold
    820380           +58.4%    1299662        proc-vmstat.nr_file_pages
  31392561           -34.9%   20422143 ± 51%  proc-vmstat.nr_free_pages
    111459           +31.8%     146918 ±  6%  proc-vmstat.nr_inactive_anon
     31518 ±  3%    +251.3%     110738 ±  6%  proc-vmstat.nr_inactive_file
     29350 ±  2%     +93.5%      56798 ± 22%  proc-vmstat.nr_mapped
    144728 ±  2%    +276.0%     544126        proc-vmstat.nr_shmem
     41915            +6.0%      44432        proc-vmstat.nr_slab_reclaimable
    319571 ±  2%     +27.0%     405943 ±  4%  proc-vmstat.nr_written
    126093 ±  3%    +289.2%     490726 ±  2%  proc-vmstat.nr_zone_active_anon
      1399 ±  3%     +35.4%       1895        proc-vmstat.nr_zone_active_file
    111459           +31.8%     146918 ±  6%  proc-vmstat.nr_zone_inactive_anon
     31518 ±  3%    +251.3%     110738 ±  6%  proc-vmstat.nr_zone_inactive_file
     32190 ±  3%    +245.4%     111176 ±  6%  proc-vmstat.nr_zone_write_pending
    160622 ±  3%    +214.7%     505422 ±  5%  proc-vmstat.numa_hint_faults
    107007 ±  9%    +119.8%     235173 ±  3%  proc-vmstat.numa_hint_faults_local
  78831382            +1.2%   79756673        proc-vmstat.numa_hit
  78645280            +1.2%   79565383        proc-vmstat.numa_local
    348125 ±  3%    +105.9%     716714 ±  5%  proc-vmstat.numa_pte_updates
    252370 ±  2%    +131.6%     584436        proc-vmstat.pgactivate
  78837491            +1.2%   79765368        proc-vmstat.pgalloc_normal
   1481521           +51.5%    2244332        proc-vmstat.pgfault
   1279742 ±  2%     +26.9%    1624069 ±  4%  proc-vmstat.pgpgout
     77086 ±  2%     +45.2%     111961 ±  2%  proc-vmstat.pgreuse
      8204 ±  2%    +243.2%      28153 ±  6%  numa-vmstat.node0.nr_dirty
      8086 ±  2%    +247.3%      28083 ±  6%  numa-vmstat.node0.nr_inactive_file
      8086 ±  3%    +247.3%      28083 ±  6%  numa-vmstat.node0.nr_zone_inactive_file
      8209 ±  2%    +243.0%      28154 ±  6%  numa-vmstat.node0.nr_zone_write_pending
      9580 ±117%    +419.5%      49767 ± 25%  numa-vmstat.node1.nr_active_anon
      8035 ±  3%    +252.0%      28282 ±  6%  numa-vmstat.node1.nr_dirty
     32176 ± 82%    +265.5%     117617 ± 35%  numa-vmstat.node1.nr_file_pages
      7844 ±  3%    +259.3%      28180 ±  6%  numa-vmstat.node1.nr_inactive_file
      4436 ± 33%    +106.0%       9137 ± 16%  numa-vmstat.node1.nr_mapped
     11807 ±118%    +378.2%      56456 ± 22%  numa-vmstat.node1.nr_shmem
      9580 ±117%    +419.5%      49767 ± 25%  numa-vmstat.node1.nr_zone_active_anon
      7842 ±  3%    +259.4%      28182 ±  6%  numa-vmstat.node1.nr_zone_inactive_file
      8037 ±  3%    +251.9%      28286 ±  6%  numa-vmstat.node1.nr_zone_write_pending
      3453 ± 96%   +1454.2%      53675 ± 17%  numa-vmstat.node2.nr_active_anon
      8063 ±  3%    +251.7%      28356 ±  6%  numa-vmstat.node2.nr_dirty
      7891 ±  3%    +258.0%      28246 ±  6%  numa-vmstat.node2.nr_inactive_file
      4674 ± 86%   +1219.2%      61670 ± 20%  numa-vmstat.node2.nr_shmem
      3453 ± 96%   +1454.2%      53675 ± 17%  numa-vmstat.node2.nr_zone_active_anon
      7891 ±  3%    +257.9%      28245 ±  6%  numa-vmstat.node2.nr_zone_inactive_file
      8064 ±  3%    +251.6%      28358 ±  6%  numa-vmstat.node2.nr_zone_write_pending
     86302 ± 20%    +278.5%     326666 ±  9%  numa-vmstat.node3.nr_active_anon
      8089 ±  4%    +248.2%      28168 ±  6%  numa-vmstat.node3.nr_dirty
    120914 ± 26%    +310.0%     495749 ± 42%  numa-vmstat.node3.nr_file_pages
   7967855           -17.7%    6555056 ± 15%  numa-vmstat.node3.nr_free_pages
      7914 ±  4%    +254.9%      28087 ±  6%  numa-vmstat.node3.nr_inactive_file
      7570 ± 28%    +223.7%      24509 ± 36%  numa-vmstat.node3.nr_mapped
     94354 ± 21%    +280.7%     359225 ±  9%  numa-vmstat.node3.nr_shmem
     86302 ± 20%    +278.5%     326665 ±  9%  numa-vmstat.node3.nr_zone_active_anon
      7916 ±  4%    +254.8%      28086 ±  6%  numa-vmstat.node3.nr_zone_inactive_file
      8093 ±  4%    +248.1%      28171 ±  6%  numa-vmstat.node3.nr_zone_write_pending
 4.368e+09           -12.4%  3.828e+09        perf-stat.i.branch-instructions
  62313172 ±  3%     -14.0%   53571269 ±  2%  perf-stat.i.cache-misses
  1.23e+08 ±  2%      -7.5%  1.138e+08 ±  2%  perf-stat.i.cache-references
     13039 ±  3%    +425.8%      68561 ±  7%  perf-stat.i.context-switches
     22.22           +17.6%      26.14        perf-stat.i.cpi
 4.464e+11            -1.1%  4.415e+11        perf-stat.i.cpu-cycles
      1253 ± 11%    +769.8%      10902 ±  9%  perf-stat.i.cpu-migrations
      7048 ±  3%     +20.9%       8521 ±  2%  perf-stat.i.cycles-between-cache-misses
    903165 ± 29%    +120.2%    1988492 ± 11%  perf-stat.i.dTLB-load-misses
  5.24e+09           -14.4%  4.487e+09        perf-stat.i.dTLB-loads
      0.02 ± 37%      +0.0        0.03 ± 25%  perf-stat.i.dTLB-store-miss-rate%
 1.494e+09           -21.5%  1.172e+09        perf-stat.i.dTLB-stores
     91.97           -11.7       80.23        perf-stat.i.iTLB-load-miss-rate%
   8340455 ±  4%     -25.2%    6235004 ±  9%  perf-stat.i.iTLB-load-misses
    565944 ±  7%    +313.3%    2339210 ±  6%  perf-stat.i.iTLB-loads
 1.948e+10           -13.8%  1.679e+10        perf-stat.i.instructions
      0.06 ±  4%     -19.5%       0.05 ±  2%  perf-stat.i.ipc
      3.10            -1.1%       3.07        perf-stat.i.metric.GHz
      1006 ±  2%     -28.8%     716.09 ±  4%  perf-stat.i.metric.K/sec
     77.11           -14.2%      66.14        perf-stat.i.metric.M/sec
     91.04            +0.9       91.97        perf-stat.i.node-load-miss-rate%
  12487874 ±  2%     +10.0%   13732587 ±  3%  perf-stat.i.node-load-misses
     85.06            +2.2       87.26        perf-stat.i.node-store-miss-rate%
   9673418            -8.0%    8902435        perf-stat.i.node-store-misses
   1618007           -23.9%    1231200        perf-stat.i.node-stores
      6.31 ±  2%      +6.1%       6.70 ±  2%  perf-stat.overall.MPKI
      0.40 ±  2%      +0.0        0.44 ±  2%  perf-stat.overall.branch-miss-rate%
     50.71 ±  2%      -3.3       47.39        perf-stat.overall.cache-miss-rate%
     22.94           +15.2%      26.43        perf-stat.overall.cpi
      7176 ±  3%     +16.0%       8327 ±  2%  perf-stat.overall.cycles-between-cache-misses
      0.02 ± 25%      +0.0        0.04 ± 10%  perf-stat.overall.dTLB-load-miss-rate%
      0.02 ± 23%      +0.0        0.03 ± 20%  perf-stat.overall.dTLB-store-miss-rate%
     93.81           -20.2       73.60 ±  2%  perf-stat.overall.iTLB-load-miss-rate%
      0.04           -13.2%       0.04        perf-stat.overall.ipc
     85.66            +2.2       87.83        perf-stat.overall.node-store-miss-rate%
 4.373e+09           -12.6%  3.821e+09        perf-stat.ps.branch-instructions
  62406854 ±  3%     -14.8%   53156258 ±  2%  perf-stat.ps.cache-misses
 1.231e+08 ±  2%      -8.9%  1.122e+08 ±  2%  perf-stat.ps.cache-references
     13029 ±  3%    +400.6%      65217 ±  7%  perf-stat.ps.context-switches
 4.473e+11            -1.1%  4.424e+11        perf-stat.ps.cpu-cycles
      1247 ± 11%    +718.0%      10208 ±  9%  perf-stat.ps.cpu-migrations
    957219 ± 25%    +103.1%    1944276 ± 10%  perf-stat.ps.dTLB-load-misses
 5.246e+09           -14.7%  4.473e+09        perf-stat.ps.dTLB-loads
 1.495e+09           -22.2%  1.163e+09        perf-stat.ps.dTLB-stores
   8350584 ±  4%     -25.5%    6222734 ±  9%  perf-stat.ps.iTLB-load-misses
    550897 ±  7%    +303.4%    2222438 ±  6%  perf-stat.ps.iTLB-loads
  1.95e+10           -14.1%  1.674e+10        perf-stat.ps.instructions
  12506886 ±  2%      +7.9%   13489702 ±  3%  perf-stat.ps.node-load-misses
   9691776            -8.7%    8848452        perf-stat.ps.node-store-misses
   1621745           -24.4%    1226078        perf-stat.ps.node-stores
 4.748e+12           +33.1%  6.318e+12        perf-stat.total.instructions
     94.68           -35.4       59.23        perf-profile.calltrace.cycles-pp.__generic_file_write_iter.f2fs_file_write_iter.new_sync_write.vfs_write.ksys_write
     94.64           -35.4       59.21        perf-profile.calltrace.cycles-pp.generic_perform_write.__generic_file_write_iter.f2fs_file_write_iter.new_sync_write.vfs_write
     62.29           -23.3       38.95        perf-profile.calltrace.cycles-pp.f2fs_write_end.generic_perform_write.__generic_file_write_iter.f2fs_file_write_iter.new_sync_write
     61.76           -23.1       38.64        perf-profile.calltrace.cycles-pp.f2fs_inode_dirtied.f2fs_mark_inode_dirty_sync.f2fs_write_end.generic_perform_write.__generic_file_write_iter
     61.76           -23.1       38.64        perf-profile.calltrace.cycles-pp.f2fs_mark_inode_dirty_sync.f2fs_write_end.generic_perform_write.__generic_file_write_iter.f2fs_file_write_iter
     61.74           -23.1       38.63        perf-profile.calltrace.cycles-pp._raw_spin_lock.f2fs_inode_dirtied.f2fs_mark_inode_dirty_sync.f2fs_write_end.generic_perform_write
     61.49           -23.0       38.48        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.f2fs_inode_dirtied.f2fs_mark_inode_dirty_sync.f2fs_write_end
     32.17           -12.0       20.14        perf-profile.calltrace.cycles-pp.f2fs_write_begin.generic_perform_write.__generic_file_write_iter.f2fs_file_write_iter.new_sync_write
     31.08           -11.6       19.45        perf-profile.calltrace.cycles-pp.f2fs_get_block.f2fs_write_begin.generic_perform_write.__generic_file_write_iter.f2fs_file_write_iter
     31.03           -11.6       19.41        perf-profile.calltrace.cycles-pp.f2fs_reserve_block.f2fs_get_block.f2fs_write_begin.generic_perform_write.__generic_file_write_iter
     31.02           -11.6       19.41        perf-profile.calltrace.cycles-pp.f2fs_reserve_new_blocks.f2fs_reserve_block.f2fs_get_block.f2fs_write_begin.generic_perform_write
     15.52            -5.9        9.66        perf-profile.calltrace.cycles-pp.__mark_inode_dirty.f2fs_reserve_new_blocks.f2fs_reserve_block.f2fs_get_block.f2fs_write_begin
     15.44            -5.8        9.62        perf-profile.calltrace.cycles-pp._raw_spin_lock.f2fs_inode_dirtied.__mark_inode_dirty.f2fs_reserve_new_blocks.f2fs_reserve_block
     15.45            -5.8        9.62        perf-profile.calltrace.cycles-pp.f2fs_inode_dirtied.__mark_inode_dirty.f2fs_reserve_new_blocks.f2fs_reserve_block.f2fs_get_block
     15.39            -5.8        9.58        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.f2fs_inode_dirtied.__mark_inode_dirty.f2fs_reserve_new_blocks
     15.38            -5.7        9.68        perf-profile.calltrace.cycles-pp._raw_spin_lock.f2fs_inode_dirtied.f2fs_mark_inode_dirty_sync.f2fs_reserve_new_blocks.f2fs_reserve_block
     15.38            -5.7        9.68        perf-profile.calltrace.cycles-pp.f2fs_inode_dirtied.f2fs_mark_inode_dirty_sync.f2fs_reserve_new_blocks.f2fs_reserve_block.f2fs_get_block
     15.38            -5.7        9.68        perf-profile.calltrace.cycles-pp.f2fs_mark_inode_dirty_sync.f2fs_reserve_new_blocks.f2fs_reserve_block.f2fs_get_block.f2fs_write_begin
     15.36            -5.7        9.66        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.f2fs_inode_dirtied.f2fs_mark_inode_dirty_sync.f2fs_reserve_new_blocks
     95.40            +3.0       98.39        perf-profile.calltrace.cycles-pp.write
     95.33            +3.0       98.33        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
     95.31            +3.0       98.32        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     95.30            +3.0       98.31        perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     95.28            +3.0       98.30        perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     95.24            +3.0       98.27        perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     95.23            +3.0       98.27        perf-profile.calltrace.cycles-pp.f2fs_file_write_iter.new_sync_write.vfs_write.ksys_write.do_syscall_64
      0.00           +38.5       38.49        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.f2fs_inode_dirtied.f2fs_mark_inode_dirty_sync.f2fs_file_write_iter
      0.00           +38.6       38.56        perf-profile.calltrace.cycles-pp._raw_spin_lock.f2fs_inode_dirtied.f2fs_mark_inode_dirty_sync.f2fs_file_write_iter.new_sync_write
      0.00           +38.6       38.59        perf-profile.calltrace.cycles-pp.f2fs_inode_dirtied.f2fs_mark_inode_dirty_sync.f2fs_file_write_iter.new_sync_write.vfs_write
      0.00           +38.6       38.59        perf-profile.calltrace.cycles-pp.f2fs_mark_inode_dirty_sync.f2fs_file_write_iter.new_sync_write.vfs_write.ksys_write
     94.69           -35.4       59.24        perf-profile.children.cycles-pp.__generic_file_write_iter
     94.66           -35.4       59.22        perf-profile.children.cycles-pp.generic_perform_write
     62.29           -23.3       38.95        perf-profile.children.cycles-pp.f2fs_write_end
     32.18           -12.0       20.14        perf-profile.children.cycles-pp.f2fs_write_begin
     31.15           -11.7       19.49        perf-profile.children.cycles-pp.f2fs_reserve_block
     31.14           -11.7       19.48        perf-profile.children.cycles-pp.f2fs_reserve_new_blocks
     31.08           -11.6       19.45        perf-profile.children.cycles-pp.f2fs_get_block
     15.78            -6.0        9.82        perf-profile.children.cycles-pp.__mark_inode_dirty
      3.04 ± 12%      -2.4        0.62 ± 26%  perf-profile.children.cycles-pp.rwsem_down_write_slowpath
      2.58 ± 14%      -2.2        0.41 ± 36%  perf-profile.children.cycles-pp.osq_lock
      1.97 ± 11%      -1.5        0.50 ± 20%  perf-profile.children.cycles-pp.unlink
      1.97 ± 11%      -1.5        0.50 ± 20%  perf-profile.children.cycles-pp.__x64_sys_unlink
      1.97 ± 11%      -1.5        0.50 ± 20%  perf-profile.children.cycles-pp.do_unlinkat
      1.52 ± 10%      -1.1        0.40 ± 15%  perf-profile.children.cycles-pp.creat64
      1.52 ± 10%      -1.1        0.40 ± 15%  perf-profile.children.cycles-pp.do_filp_open
      1.52 ± 10%      -1.1        0.40 ± 15%  perf-profile.children.cycles-pp.path_openat
      1.52 ± 10%      -1.1        0.41 ± 15%  perf-profile.children.cycles-pp.do_sys_open
      1.52 ± 10%      -1.1        0.41 ± 15%  perf-profile.children.cycles-pp.do_sys_openat2
      0.45 ±  2%      -0.2        0.20 ±  6%  perf-profile.children.cycles-pp.rwsem_spin_on_owner
      0.61 ±  3%      -0.2        0.39 ±  3%  perf-profile.children.cycles-pp.read
      0.53 ±  4%      -0.2        0.34 ±  3%  perf-profile.children.cycles-pp.ksys_read
      0.51 ±  3%      -0.2        0.33 ±  3%  perf-profile.children.cycles-pp.vfs_read
      0.47 ±  3%      -0.2        0.30 ±  2%  perf-profile.children.cycles-pp.new_sync_read
      0.46 ±  4%      -0.2        0.29 ±  3%  perf-profile.children.cycles-pp.f2fs_file_read_iter
      0.42 ±  3%      -0.1        0.28 ±  3%  perf-profile.children.cycles-pp.filemap_read
      0.40            -0.1        0.25        perf-profile.children.cycles-pp.__close
      0.39            -0.1        0.25        perf-profile.children.cycles-pp.task_work_run
      0.41            -0.1        0.26        perf-profile.children.cycles-pp.syscall_exit_to_user_mode
      0.40            -0.1        0.26        perf-profile.children.cycles-pp.exit_to_user_mode_prepare
      0.39            -0.1        0.25        perf-profile.children.cycles-pp.dput
      0.39            -0.1        0.25        perf-profile.children.cycles-pp.__fput
      0.39            -0.1        0.25        perf-profile.children.cycles-pp.__dentry_kill
      0.39            -0.1        0.25        perf-profile.children.cycles-pp.f2fs_evict_inode
      0.39            -0.1        0.25        perf-profile.children.cycles-pp.evict
      0.29            -0.1        0.17 ±  2%  perf-profile.children.cycles-pp.folio_unlock
      0.32            -0.1        0.19 ±  2%  perf-profile.children.cycles-pp.f2fs_convert_inline_inode
      0.31 ±  2%      -0.1        0.19 ±  2%  perf-profile.children.cycles-pp.f2fs_convert_inline_page
      0.26            -0.1        0.15 ±  2%  perf-profile.children.cycles-pp.folio_mark_dirty
      0.24 ±  2%      -0.1        0.14 ±  3%  perf-profile.children.cycles-pp.f2fs_set_data_page_dirty
      0.28 ±  5%      -0.1        0.18 ±  4%  perf-profile.children.cycles-pp.filemap_get_pages
      0.26 ±  4%      -0.1        0.17 ±  2%  perf-profile.children.cycles-pp.__filemap_get_folio
      0.25            -0.1        0.16 ±  3%  perf-profile.children.cycles-pp.f2fs_create
      0.26 ±  4%      -0.1        0.17 ±  4%  perf-profile.children.cycles-pp.pagecache_get_page
      0.24 ±  2%      -0.1        0.16 ±  3%  perf-profile.children.cycles-pp.f2fs_do_add_link
      0.24 ±  2%      -0.1        0.16 ±  3%  perf-profile.children.cycles-pp.f2fs_add_dentry
      0.24 ±  2%      -0.1        0.16 ±  3%  perf-profile.children.cycles-pp.f2fs_add_regular_entry
      0.27 ±  6%      -0.1        0.18 ±  3%  perf-profile.children.cycles-pp.filemap_get_read_batch
      0.22 ±  3%      -0.1        0.14 ±  3%  perf-profile.children.cycles-pp.f2fs_lookup_extent_cache
      0.18 ±  2%      -0.1        0.10 ±  3%  perf-profile.children.cycles-pp.f2fs_update_dirty_page
      0.18 ±  2%      -0.1        0.11        perf-profile.children.cycles-pp.f2fs_unlink
      0.20 ±  4%      -0.1        0.13 ±  5%  perf-profile.children.cycles-pp.__get_node_page
      0.19 ±  3%      -0.1        0.12 ±  3%  perf-profile.children.cycles-pp.f2fs_truncate
      0.18 ±  2%      -0.1        0.12 ±  4%  perf-profile.children.cycles-pp.vfs_unlink
      0.18            -0.1        0.11 ±  3%  perf-profile.children.cycles-pp.f2fs_inode_synced
      0.40 ±  6%      -0.1        0.33 ±  6%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.36 ±  6%      -0.1        0.30 ±  6%  perf-profile.children.cycles-pp.tick_sched_handle
      0.36 ±  6%      -0.1        0.30 ±  6%  perf-profile.children.cycles-pp.update_process_times
      0.18            -0.1        0.12 ±  4%  perf-profile.children.cycles-pp.f2fs_update_parent_metadata
      0.30 ±  7%      -0.1        0.23 ±  6%  perf-profile.children.cycles-pp.scheduler_tick
      0.37 ±  6%      -0.1        0.31 ±  6%  perf-profile.children.cycles-pp.tick_sched_timer
      0.27 ±  7%      -0.1        0.21 ±  8%  perf-profile.children.cycles-pp.task_tick_fair
      0.16 ±  9%      -0.1        0.10 ± 14%  perf-profile.children.cycles-pp.update_cfs_group
      0.13 ±  2%      -0.1        0.08 ±  4%  perf-profile.children.cycles-pp.f2fs_do_truncate_blocks
      0.13 ±  3%      -0.0        0.08 ±  4%  perf-profile.children.cycles-pp.f2fs_truncate_data_blocks_range
      0.12 ±  4%      -0.0        0.08        perf-profile.children.cycles-pp.truncate_inode_pages_range
      0.12            -0.0        0.08 ±  6%  perf-profile.children.cycles-pp.f2fs_update_inode
      0.12 ±  3%      -0.0        0.07 ±  6%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      0.07 ±  5%      -0.0        0.02 ± 99%  perf-profile.children.cycles-pp.xas_load
      0.06 ±  7%      -0.0        0.02 ± 99%  perf-profile.children.cycles-pp.__filemap_add_folio
      0.08            -0.0        0.04 ± 44%  perf-profile.children.cycles-pp.copy_page_to_iter
      0.10 ±  4%      -0.0        0.07 ±  7%  perf-profile.children.cycles-pp.copy_page_from_iter_atomic
      0.09 ±  4%      -0.0        0.06        perf-profile.children.cycles-pp.filemap_add_folio
      0.08 ±  6%      -0.0        0.05 ±  7%  perf-profile.children.cycles-pp._raw_read_lock
     99.78            +0.1       99.86        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     99.77            +0.1       99.85        perf-profile.children.cycles-pp.do_syscall_64
     95.42            +3.0       98.40        perf-profile.children.cycles-pp.write
     95.30            +3.0       98.32        perf-profile.children.cycles-pp.ksys_write
     95.29            +3.0       98.30        perf-profile.children.cycles-pp.vfs_write
     95.25            +3.0       98.28        perf-profile.children.cycles-pp.new_sync_write
     95.23            +3.0       98.27        perf-profile.children.cycles-pp.f2fs_file_write_iter
     93.92            +3.4       97.33        perf-profile.children.cycles-pp._raw_spin_lock
     93.48            +3.5       97.00        perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
     93.60            +3.6       97.16        perf-profile.children.cycles-pp.f2fs_inode_dirtied
     77.91            +9.5       87.39        perf-profile.children.cycles-pp.f2fs_mark_inode_dirty_sync
      2.57 ± 14%      -2.2        0.41 ± 36%  perf-profile.self.cycles-pp.osq_lock
      0.44 ±  2%      -0.2        0.20 ±  6%  perf-profile.self.cycles-pp.rwsem_spin_on_owner
      0.29 ±  2%      -0.1        0.17 ±  2%  perf-profile.self.cycles-pp.folio_unlock
      0.44            -0.1        0.33        perf-profile.self.cycles-pp._raw_spin_lock
      0.25 ±  7%      -0.1        0.17 ±  5%  perf-profile.self.cycles-pp.filemap_get_read_batch
      0.19            -0.1        0.13        perf-profile.self.cycles-pp.f2fs_write_begin
      0.14 ±  7%      -0.1        0.08 ±  5%  perf-profile.self.cycles-pp.f2fs_lookup_extent_cache
      0.16 ±  9%      -0.1        0.10 ± 14%  perf-profile.self.cycles-pp.update_cfs_group
      0.08 ±  5%      -0.0        0.04 ± 44%  perf-profile.self.cycles-pp.f2fs_update_dirty_page
      0.12 ±  4%      -0.0        0.07 ±  6%  perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      0.09 ±  4%      -0.0        0.06 ±  9%  perf-profile.self.cycles-pp.__get_node_page
      0.08 ±  6%      -0.0        0.05 ±  7%  perf-profile.self.cycles-pp._raw_read_lock
      0.16 ±  7%      +0.1        0.22 ±  2%  perf-profile.self.cycles-pp.f2fs_file_write_iter
     92.94            +3.6       96.50        perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      0.16 ± 87%     -90.4%       0.01 ± 25%  perf-sched.sch_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      4.85 ±103%     -99.7%       0.02 ± 98%  perf-sched.sch_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      0.06 ± 18%     -72.1%       0.02 ± 52%  perf-sched.sch_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
      0.54 ± 33%     -97.9%       0.01 ± 34%  perf-sched.sch_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      0.64 ±186%     -97.5%       0.02 ± 25%  perf-sched.sch_delay.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      0.22 ± 72%     -83.6%       0.04 ± 13%  perf-sched.sch_delay.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.15 ±140%     -96.1%       0.01 ± 46%  perf-sched.sch_delay.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      2.86 ± 55%     -99.3%       0.02 ±158%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.down_read.f2fs_getxattr.__f2fs_get_acl
      0.07 ± 58%    +190.8%       0.19 ± 34%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.dput.path_openat.do_filp_open
      0.35 ± 61%     -74.6%       0.09 ±  6%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.generic_perform_write.__generic_file_write_iter.f2fs_file_write_iter
      8.61 ± 25%     -98.0%       0.17 ±137%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.__alloc_nat_entry.constprop
      4.91 ±158%     -98.5%       0.07 ± 12%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.migrate_task_to.task_numa_migrate
     10.70 ±  7%     -99.5%       0.05 ± 49%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.sched_exec.bprm_execve
      0.86 ± 10%     -91.0%       0.08 ± 18%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.wait_for_completion.stop_two_cpus.migrate_swap
      3.41 ±194%     -93.4%       0.23 ± 34%  perf-sched.sch_delay.avg.ms.rwsem_down_read_slowpath.f2fs_get_node_info.f2fs_convert_inline_page.f2fs_convert_inline_inode
      0.00 ±117%   +6386.7%       0.16 ± 46%  perf-sched.sch_delay.avg.ms.rwsem_down_read_slowpath.f2fs_get_node_info.truncate_node.f2fs_remove_inode_page
      0.00 ±223%  +55000.0%       0.09 ± 13%  perf-sched.sch_delay.avg.ms.rwsem_down_read_slowpath.f2fs_write_begin.generic_perform_write.__generic_file_write_iter
      0.00 ±223%   +4125.0%       0.03 ± 26%  perf-sched.sch_delay.avg.ms.rwsem_down_write_slowpath.block_operations.f2fs_write_checkpoint.__write_checkpoint_sync
      0.98 ± 96%     -96.4%       0.04 ± 17%  perf-sched.sch_delay.avg.ms.rwsem_down_write_slowpath.do_unlinkat.__x64_sys_unlink.do_syscall_64
      0.40 ± 49%     -74.7%       0.10 ± 18%  perf-sched.sch_delay.avg.ms.rwsem_down_write_slowpath.f2fs_submit_page_write.do_write_page.f2fs_outplace_write_data
      0.32 ± 17%     -88.6%       0.04 ± 28%  perf-sched.sch_delay.avg.ms.rwsem_down_write_slowpath.path_openat.do_filp_open.do_sys_openat2
      1.00 ± 14%     -92.4%       0.08 ± 59%  perf-sched.sch_delay.avg.ms.rwsem_down_write_slowpath.set_node_addr.f2fs_new_node_page.f2fs_new_inode_page
      1.11 ±  9%     -97.4%       0.03 ± 38%  perf-sched.sch_delay.avg.ms.rwsem_down_write_slowpath.set_node_addr.truncate_node.f2fs_remove_inode_page
      0.31 ±167%     -94.5%       0.02 ± 29%  perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      0.05 ± 38%     -83.9%       0.01 ± 16%  perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
      0.02 ± 92%     -99.3%       0.00 ±223%  perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
      0.07 ± 14%     -74.5%       0.02 ± 20%  perf-sched.sch_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      0.07 ±  2%     -70.2%       0.02 ± 15%  perf-sched.sch_delay.avg.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
      0.13 ± 91%     -85.2%       0.02 ±  5%  perf-sched.sch_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
      5.59 ± 98%     -99.7%       0.02 ±114%  perf-sched.sch_delay.avg.ms.syslog_print.do_syslog.part.0.kmsg_read
      2.86 ± 20%     -99.3%       0.02 ± 75%  perf-sched.sch_delay.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
      0.07 ±  4%     -38.9%       0.04 ±  9%  perf-sched.sch_delay.avg.ms.worker_thread.kthread.ret_from_fork
      1.97 ±128%     -96.5%       0.07 ± 38%  perf-sched.sch_delay.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      4.87 ±102%     -99.6%       0.02 ± 99%  perf-sched.sch_delay.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      0.18 ± 12%     -60.3%       0.07 ± 68%  perf-sched.sch_delay.max.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
     13.10 ± 19%     -93.3%       0.88 ± 43%  perf-sched.sch_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
    175.40 ±212%     -99.7%       0.56 ± 43%  perf-sched.sch_delay.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
    348.26 ±137%     -99.8%       0.72 ± 30%  perf-sched.sch_delay.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      9.26 ± 43%     -87.0%       1.21 ± 28%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.copy_page_to_iter.filemap_read.f2fs_file_read_iter
     11.10 ± 49%     -99.6%       0.04 ±146%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.down_read.f2fs_getxattr.__f2fs_get_acl
      9.45 ± 23%     -82.4%       1.67 ± 14%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.dput.path_openat.do_filp_open
      3966 ±  5%     -99.4%      23.21 ± 30%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.generic_perform_write.__generic_file_write_iter.f2fs_file_write_iter
     18.16 ± 11%     -98.1%       0.34 ±137%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.__alloc_nat_entry.constprop
    687.44 ±219%     -99.7%       2.05 ± 24%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.migrate_task_to.task_numa_migrate
     47.23 ± 19%     -97.6%       1.13 ± 37%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.sched_exec.bprm_execve
     22.99 ± 97%     -91.0%       2.08 ± 18%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.wait_for_completion.stop_two_cpus.migrate_swap
    690.65 ±220%     -99.6%       2.85 ± 94%  perf-sched.sch_delay.max.ms.rwsem_down_read_slowpath.f2fs_get_node_info.f2fs_convert_inline_page.f2fs_convert_inline_inode
      0.01 ±223%  +6.1e+05%      31.52 ± 18%  perf-sched.sch_delay.max.ms.rwsem_down_read_slowpath.f2fs_write_begin.generic_perform_write.__generic_file_write_iter
      0.05 ±123%   +1835.7%       1.01 ± 25%  perf-sched.sch_delay.max.ms.rwsem_down_write_slowpath.__submit_merged_write_cond.f2fs_wait_on_page_writeback.f2fs_convert_inline_page
      0.00 ±223%  +1.1e+05%       1.23 ± 98%  perf-sched.sch_delay.max.ms.rwsem_down_write_slowpath.block_operations.f2fs_write_checkpoint.__write_checkpoint_sync
      1256 ±140%     -99.8%       2.10 ± 45%  perf-sched.sch_delay.max.ms.rwsem_down_write_slowpath.do_unlinkat.__x64_sys_unlink.do_syscall_64
      9.78 ± 20%     -85.3%       1.43 ±  9%  perf-sched.sch_delay.max.ms.rwsem_down_write_slowpath.f2fs_submit_page_write.do_write_page.f2fs_outplace_write_data
     40.64 ± 78%     -95.3%       1.91 ± 25%  perf-sched.sch_delay.max.ms.rwsem_down_write_slowpath.path_openat.do_filp_open.do_sys_openat2
     19.70 ± 35%     -96.6%       0.67 ± 76%  perf-sched.sch_delay.max.ms.rwsem_down_write_slowpath.set_node_addr.f2fs_new_node_page.f2fs_new_inode_page
     26.77 ± 81%     -95.9%       1.11 ± 55%  perf-sched.sch_delay.max.ms.rwsem_down_write_slowpath.set_node_addr.truncate_node.f2fs_remove_inode_page
      2.03 ±200%     -95.5%       0.09 ± 46%  perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      0.10 ± 60%     -89.7%       0.01 ± 18%  perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select
      0.21 ±  3%     -33.3%       0.14 ± 34%  perf-sched.sch_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      5.59 ± 98%     -99.6%       0.02 ±100%  perf-sched.sch_delay.max.ms.syslog_print.do_syslog.part.0.kmsg_read
     19.24 ± 60%     -96.6%       0.65 ± 63%  perf-sched.sch_delay.max.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
      0.36 ± 51%     -75.0%       0.09 ± 12%  perf-sched.total_sch_delay.average.ms
      3968 ±  5%     -99.2%      31.62 ± 18%  perf-sched.total_sch_delay.max.ms
    174.33 ±  6%     -90.8%      15.99        perf-sched.total_wait_and_delay.average.ms
    125363 ±  5%   +1515.1%    2024743        perf-sched.total_wait_and_delay.count.ms
    173.97 ±  6%     -90.9%      15.90        perf-sched.total_wait_time.average.ms
    226.65 ±  2%     -32.6%     152.83 ± 13%  perf-sched.wait_and_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
     39.90 ± 29%    +159.1%     103.40 ± 62%  perf-sched.wait_and_delay.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read
    375.82 ± 36%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.down_write.f2fs_drop_nlink.f2fs_unlink
    274.64 ± 25%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.dput.do_unlinkat.__x64_sys_unlink
    162.15 ± 11%     -91.0%      14.67 ±  2%  perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.generic_perform_write.__generic_file_write_iter.f2fs_file_write_iter
      1079 ± 16%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.migrate_task_to.task_numa_migrate
      6.64 ±  4%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.wait_for_completion.affine_move_task.__set_cpus_allowed_ptr_locked
    456.33 ± 13%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.preempt_schedule_common.__cond_resched.wait_for_completion.stop_two_cpus.migrate_swap
    818.49 ± 30%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.rwsem_down_read_slowpath.f2fs_get_node_info.f2fs_convert_inline_page.f2fs_convert_inline_inode
     19.96 ± 30%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.rwsem_down_read_slowpath.f2fs_get_node_info.truncate_node.f2fs_remove_inode_page
    691.67 ± 11%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.rwsem_down_write_slowpath.do_unlinkat.__x64_sys_unlink.do_syscall_64
      1141 ± 60%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.rwsem_down_write_slowpath.f2fs_submit_page_write.do_write_page.f2fs_outplace_write_data
     79.17 ±  5%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.rwsem_down_write_slowpath.path_openat.do_filp_open.do_sys_openat2
    617.75 ±  7%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.rwsem_down_write_slowpath.set_node_addr.f2fs_new_node_page.f2fs_new_inode_page
     15.65 ±  9%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.rwsem_down_write_slowpath.set_node_addr.truncate_node.f2fs_remove_inode_page
      4.65          -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
    607.44 ±  2%     -31.1%     418.23 ±  3%  perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
      4093 ± 43%    +687.5%      32234 ±  6%  perf-sched.wait_and_delay.count.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
      4147 ± 42%     -55.0%       1864 ± 30%  perf-sched.wait_and_delay.count.pipe_read.new_sync_read.vfs_read.ksys_read
     39.17 ± 12%    -100.0%       0.00        perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.down_write.f2fs_drop_nlink.f2fs_unlink
    175.33 ± 15%    -100.0%       0.00        perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.dput.do_unlinkat.__x64_sys_unlink
     93296 ±  8%     -50.3%      46324 ±  7%  perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.generic_perform_write.__generic_file_write_iter.f2fs_file_write_iter
    168.50 ± 11%    -100.0%       0.00        perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.stop_one_cpu.migrate_task_to.task_numa_migrate
      1343 ±  5%    -100.0%       0.00        perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.wait_for_completion.affine_move_task.__set_cpus_allowed_ptr_locked
    388.83 ±  3%    -100.0%       0.00        perf-sched.wait_and_delay.count.preempt_schedule_common.__cond_resched.wait_for_completion.stop_two_cpus.migrate_swap
    150.00 ± 32%    -100.0%       0.00        perf-sched.wait_and_delay.count.rwsem_down_read_slowpath.f2fs_get_node_info.f2fs_convert_inline_page.f2fs_convert_inline_inode
      1785 ± 23%    -100.0%       0.00        perf-sched.wait_and_delay.count.rwsem_down_read_slowpath.f2fs_get_node_info.truncate_node.f2fs_remove_inode_page
     29.67 ±223%  +6.5e+06%    1918472 ±  2%  perf-sched.wait_and_delay.count.rwsem_down_read_slowpath.f2fs_write_begin.generic_perform_write.__generic_file_write_iter
      1975 ±  5%    -100.0%       0.00        perf-sched.wait_and_delay.count.rwsem_down_write_slowpath.do_unlinkat.__x64_sys_unlink.do_syscall_64
    124.67 ± 56%    -100.0%       0.00        perf-sched.wait_and_delay.count.rwsem_down_write_slowpath.f2fs_submit_page_write.do_write_page.f2fs_outplace_write_data
      5226 ±  2%    -100.0%       0.00        perf-sched.wait_and_delay.count.rwsem_down_write_slowpath.path_openat.do_filp_open.do_sys_openat2
     94.00 ± 24%    -100.0%       0.00        perf-sched.wait_and_delay.count.rwsem_down_write_slowpath.set_node_addr.f2fs_new_node_page.f2fs_new_inode_page
      1676 ± 23%    -100.0%       0.00        perf-sched.wait_and_delay.count.rwsem_down_write_slowpath.set_node_addr.truncate_node.f2fs_remove_inode_page
      2177          -100.0%       0.00        perf-sched.wait_and_delay.count.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
      3150 ±  3%     +21.0%       3813 ±  3%  perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork
      3045 ± 34%     -67.1%       1000        perf-sched.wait_and_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      4155 ±  2%     -99.0%      41.14 ± 12%  perf-sched.wait_and_delay.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1254 ±  2%     -17.0%       1041        perf-sched.wait_and_delay.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      4055 ±  4%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.down_write.f2fs_drop_nlink.f2fs_unlink
      4156 ±  4%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.dput.do_unlinkat.__x64_sys_unlink
      4713 ± 21%     -99.2%      38.66 ± 31%  perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.generic_perform_write.__generic_file_write_iter.f2fs_file_write_iter
      4219 ±  3%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.migrate_task_to.task_numa_migrate
      1002          -100.0%       0.00        perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.wait_for_completion.affine_move_task.__set_cpus_allowed_ptr_locked
      4230 ±  3%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.preempt_schedule_common.__cond_resched.wait_for_completion.stop_two_cpus.migrate_swap
      4207 ±  3%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.rwsem_down_read_slowpath.f2fs_get_node_info.f2fs_convert_inline_page.f2fs_convert_inline_inode
      3391 ± 44%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.rwsem_down_read_slowpath.f2fs_get_node_info.truncate_node.f2fs_remove_inode_page
      4274 ±  3%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.rwsem_down_write_slowpath.do_unlinkat.__x64_sys_unlink.do_syscall_64
      3922 ±  5%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.rwsem_down_write_slowpath.f2fs_submit_page_write.do_write_page.f2fs_outplace_write_data
      4225 ±  3%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.rwsem_down_write_slowpath.path_openat.do_filp_open.do_sys_openat2
      4203 ±  4%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.rwsem_down_write_slowpath.set_node_addr.f2fs_new_node_page.f2fs_new_inode_page
    787.44 ±200%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.rwsem_down_write_slowpath.set_node_addr.truncate_node.f2fs_remove_inode_page
      5.42          -100.0%       0.00        perf-sched.wait_and_delay.max.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
      0.81 ±100%    +110.3%       1.70 ±  4%  perf-sched.wait_time.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
    226.11 ±  2%     -32.4%     152.82 ± 13%  perf-sched.wait_time.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
     53.90 ± 24%     -71.9%      15.12 ± 28%  perf-sched.wait_time.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
    112.48 ± 63%     -88.2%      13.32 ±  5%  perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
     39.75 ± 30%    +160.1%     103.39 ± 62%  perf-sched.wait_time.avg.ms.pipe_read.new_sync_read.vfs_read.ksys_read
    148.34 ± 35%     -90.1%      14.71 ±  2%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.__alloc_pages.folio_alloc.__filemap_get_folio
     96.54 ± 41%     -85.8%      13.67        perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.__filemap_get_folio.pagecache_get_page.__get_node_page
    194.05 ± 14%     -93.0%      13.67 ±  3%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.__filemap_get_folio.pagecache_get_page.f2fs_write_begin
     15.14 ±  8%     -25.6%      11.26 ± 17%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.__invalidate_mapping_pages.truncate_node.f2fs_remove_inode_page
    159.26 ± 30%     -90.9%      14.47 ±  2%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.copy_page_to_iter.filemap_read.f2fs_file_read_iter
    433.64 ±135%     -97.1%      12.45 ± 82%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
    188.13 ± 35%     -93.7%      11.85 ± 11%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.down_read.__submit_merged_write_cond.f2fs_wait_on_page_writeback
    277.46 ± 35%     -95.1%      13.48 ±  8%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.down_read.f2fs_get_node_info.f2fs_convert_inline_page
     12.47 ± 22%     -74.3%       3.21 ±141%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.down_read.f2fs_getxattr.__f2fs_get_acl
    528.68 ±151%     -99.0%       5.18 ±133%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.down_read.f2fs_unlink.vfs_unlink
    186.00 ± 22%     -92.2%      14.59 ±  4%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.down_read.f2fs_write_begin.generic_perform_write
    375.67 ± 36%     -96.9%      11.75 ± 14%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.down_write.f2fs_drop_nlink.f2fs_unlink
    156.96 ± 63%     -90.9%      14.36 ±  2%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.down_write.f2fs_file_write_iter.new_sync_write
      4.43 ± 13%     -91.5%       0.38 ±146%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.dput.__fput.task_work_run
    271.05 ± 23%     -96.5%       9.44 ± 10%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.dput.do_unlinkat.__x64_sys_unlink
     89.30 ± 47%     -86.8%      11.75 ±  4%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.dput.path_openat.do_filp_open
    167.73 ± 55%     -91.7%      13.95 ±  4%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.filemap_read.f2fs_file_read_iter.new_sync_read
    161.80 ± 10%     -91.0%      14.58 ±  2%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.generic_perform_write.__generic_file_write_iter.f2fs_file_write_iter
    585.90 ± 64%     -98.3%      10.19 ± 16%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.generic_perform_write.__generic_file_write_iter.generic_file_write_iter
    164.56 ± 47%     -93.0%      11.60 ± 14%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.__add_ino_entry.f2fs_add_orphan_inode
     64.17 ±174%     -88.5%       7.40 ± 58%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.__alloc_nat_entry.constprop
    966.48 ±145%    -100.0%       0.00        perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.mutex_lock.f2fs_build_free_nids.f2fs_balance_fs_bg
    240.61 ± 49%     -95.9%       9.89 ±  5%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.rmap_walk_anon.remove_migration_ptes.migrate_pages
      1074 ± 16%     -98.8%      12.85        perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.migrate_task_to.task_numa_migrate
    105.36 ± 85%     -88.1%      12.55 ±  3%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.vfs_write.ksys_write.do_syscall_64
      6.64 ±  4%     -13.2%       5.77 ±  2%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.wait_for_completion.affine_move_task.__set_cpus_allowed_ptr_locked
    455.47 ± 14%     -97.1%      13.29 ±  3%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.wait_for_completion.stop_two_cpus.migrate_swap
      0.00 ±223%  +1.2e+06%      14.02        perf-sched.wait_time.avg.ms.rwsem_down_read_slowpath.f2fs_do_truncate_blocks.f2fs_truncate.f2fs_evict_inode
      1.96 ±223%    +436.4%      10.52 ± 34%  perf-sched.wait_time.avg.ms.rwsem_down_read_slowpath.f2fs_evict_inode.evict.__dentry_kill
    815.08 ± 31%     -99.1%       6.95 ±  5%  perf-sched.wait_time.avg.ms.rwsem_down_read_slowpath.f2fs_get_node_info.f2fs_convert_inline_page.f2fs_convert_inline_inode
     19.96 ± 30%     -61.6%       7.66 ±  7%  perf-sched.wait_time.avg.ms.rwsem_down_read_slowpath.f2fs_get_node_info.truncate_node.f2fs_remove_inode_page
      0.00 ±223%  +1.1e+05%       5.30 ±  3%  perf-sched.wait_time.avg.ms.rwsem_down_write_slowpath.block_operations.f2fs_write_checkpoint.__write_checkpoint_sync
    690.69 ± 11%     -99.5%       3.14 ± 17%  perf-sched.wait_time.avg.ms.rwsem_down_write_slowpath.do_unlinkat.__x64_sys_unlink.do_syscall_64
      1140 ± 60%     -99.4%       7.01 ±  5%  perf-sched.wait_time.avg.ms.rwsem_down_write_slowpath.f2fs_submit_page_write.do_write_page.f2fs_outplace_write_data
     78.85 ±  5%     -96.6%       2.71 ±  6%  perf-sched.wait_time.avg.ms.rwsem_down_write_slowpath.path_openat.do_filp_open.do_sys_openat2
    616.75 ±  7%     -98.1%      11.80 ±  3%  perf-sched.wait_time.avg.ms.rwsem_down_write_slowpath.set_node_addr.f2fs_new_node_page.f2fs_new_inode_page
     14.54 ± 10%     -84.8%       2.20 ± 10%  perf-sched.wait_time.avg.ms.rwsem_down_write_slowpath.set_node_addr.truncate_node.f2fs_remove_inode_page
     63.82 ±  8%     -77.4%      14.45 ±  6%  perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll
      3.16 ± 36%     -87.4%       0.40 ± 72%  perf-sched.wait_time.avg.ms.schedule_timeout.khugepaged.kthread.ret_from_fork
      4.58           -14.4%       3.93        perf-sched.wait_time.avg.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
      0.00 ±223%   +5360.0%       0.14 ± 41%  perf-sched.wait_time.avg.ms.schedule_timeout.wait_for_completion.stop_two_cpus.migrate_swap
    607.31 ±  2%     -31.1%     418.21 ±  3%  perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
      1.46 ± 17%     -98.7%       0.02 ± 73%  perf-sched.wait_time.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
      1.61 ±100%    +110.3%       3.39 ±  4%  perf-sched.wait_time.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      3045 ± 34%     -67.2%       1000        perf-sched.wait_time.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      3269 ± 44%     -99.5%      15.50 ±  3%  perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
      3.75 ±155%    +617.4%      26.87 ± 57%  perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
      4155 ±  2%     -99.3%      30.92 ± 24%  perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1254 ±  2%     -17.0%       1041        perf-sched.wait_time.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      3909 ±  3%     -99.6%      16.33 ±  5%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.__alloc_pages.folio_alloc.__filemap_get_folio
      4072 ±  4%     -99.6%      17.93 ±  2%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.__filemap_get_folio.pagecache_get_page.__get_node_page
      4130 ±  2%     -99.6%      16.33 ±  4%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.__filemap_get_folio.pagecache_get_page.f2fs_write_begin
     45.57 ± 22%     -64.7%      16.08 ±  4%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.__invalidate_mapping_pages.truncate_node.f2fs_remove_inode_page
      4128 ±  4%     -99.6%      16.32 ±  2%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.copy_page_to_iter.filemap_read.f2fs_file_read_iter
      3883 ±  3%     -99.6%      15.85 ±  2%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.down_read.__submit_merged_write_cond.f2fs_wait_on_page_writeback
     37.90 ± 22%     -56.2%      16.60 ±  3%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.down_read.f2fs_evict_inode.evict
      4057 ±  4%     -99.6%      16.11 ±  2%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.down_read.f2fs_get_node_info.f2fs_convert_inline_page
     20.56 ± 16%     -74.2%       5.30 ±140%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.down_read.f2fs_getxattr.__f2fs_get_acl
      1364 ±140%     -99.6%       5.18 ±133%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.down_read.f2fs_unlink.vfs_unlink
      4109 ±  3%     -99.6%      15.96        perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.down_read.f2fs_write_begin.generic_perform_write
      4054 ±  4%     -99.6%      15.96 ±  4%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.down_write.f2fs_drop_nlink.f2fs_unlink
      4001 ±  6%     -99.6%      16.21 ±  4%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.down_write.f2fs_file_write_iter.new_sync_write
     21.79 ± 11%     -83.3%       3.65 ±160%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.dput.__fput.task_work_run
      4156 ±  4%     -99.6%      18.65 ± 21%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.dput.do_unlinkat.__x64_sys_unlink
      4102 ±  5%     -99.6%      16.85 ±  5%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.dput.path_openat.do_filp_open
      4004 ±  7%     -99.6%      16.49 ±  4%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.filemap_read.f2fs_file_read_iter.new_sync_read
      4285 ±  3%     -99.4%      26.93 ± 21%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.generic_perform_write.__generic_file_write_iter.f2fs_file_write_iter
      3224 ± 44%     -99.5%      16.99 ±  3%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.__add_ino_entry.f2fs_add_orphan_inode
    767.10 ±216%     -98.3%      12.95 ± 44%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc.__alloc_nat_entry.constprop
      1326 ±139%    -100.0%       0.00        perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.mutex_lock.f2fs_build_free_nids.f2fs_balance_fs_bg
      2.09 ±218%   +1554.5%      34.55 ± 44%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.mutex_lock.perf_poll.do_sys_poll
      3909 ±  7%     -99.6%      16.64 ±  3%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.rmap_walk_anon.remove_migration_ptes.migrate_pages
      1297 ±138%     -98.8%      15.88 ±  2%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.rmap_walk_anon.try_to_migrate.migrate_pages
      4216 ±  3%     -99.2%      32.99 ± 19%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.stop_one_cpu.migrate_task_to.task_numa_migrate
    732.06 ±209%     -97.8%      16.19 ±  2%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.truncate_inode_pages_range.f2fs_evict_inode.evict
      2115 ± 96%     -98.8%      24.82 ± 41%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.vfs_write.ksys_write.do_syscall_64
      4228 ±  3%     -95.5%     189.67 ±191%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.wait_for_completion.stop_two_cpus.migrate_swap
      3292 ± 44%     -99.5%      17.11 ±  2%  perf-sched.wait_time.max.ms.rwsem_down_read_slowpath.__submit_merged_write_cond.f2fs_wait_on_page_writeback.f2fs_convert_inline_page
      0.00 ±223%  +7.3e+05%      18.13 ±  3%  perf-sched.wait_time.max.ms.rwsem_down_read_slowpath.f2fs_do_truncate_blocks.f2fs_truncate.f2fs_evict_inode
      1.96 ±223%    +699.0%      15.68 ±  2%  perf-sched.wait_time.max.ms.rwsem_down_read_slowpath.f2fs_evict_inode.evict.__dentry_kill
      4204 ±  4%     -99.6%      18.38 ± 13%  perf-sched.wait_time.max.ms.rwsem_down_read_slowpath.f2fs_get_node_info.f2fs_convert_inline_page.f2fs_convert_inline_inode
      3389 ± 44%     -99.5%      17.36        perf-sched.wait_time.max.ms.rwsem_down_read_slowpath.f2fs_get_node_info.truncate_node.f2fs_remove_inode_page
      0.02 ±223%  +69711.4%      15.36 ±  2%  perf-sched.wait_time.max.ms.rwsem_down_write_slowpath.block_operations.f2fs_write_checkpoint.__write_checkpoint_sync
      4272 ±  3%     -99.4%      27.57 ± 27%  perf-sched.wait_time.max.ms.rwsem_down_write_slowpath.do_unlinkat.__x64_sys_unlink.do_syscall_64
      3919 ±  5%     -99.6%      16.94 ±  3%  perf-sched.wait_time.max.ms.rwsem_down_write_slowpath.f2fs_submit_page_write.do_write_page.f2fs_outplace_write_data
      4221 ±  3%     -99.4%      23.69 ± 32%  perf-sched.wait_time.max.ms.rwsem_down_write_slowpath.path_openat.do_filp_open.do_sys_openat2
      4202 ±  4%     -99.6%      17.16 ±  5%  perf-sched.wait_time.max.ms.rwsem_down_write_slowpath.set_node_addr.f2fs_new_node_page.f2fs_new_inode_page
    783.28 ±201%     -98.0%      15.67 ±  4%  perf-sched.wait_time.max.ms.rwsem_down_write_slowpath.set_node_addr.truncate_node.f2fs_remove_inode_page
      3.16 ± 36%     -87.4%       0.40 ± 72%  perf-sched.wait_time.max.ms.schedule_timeout.khugepaged.kthread.ret_from_fork
      5.20          +206.7%      15.95 ± 49%  perf-sched.wait_time.max.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
      0.00 ±223%  +22853.3%       0.57 ± 34%  perf-sched.wait_time.max.ms.schedule_timeout.wait_for_completion.stop_two_cpus.migrate_swap
     10.65 ± 29%     -91.9%       0.87 ± 77%  perf-sched.wait_time.max.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
     10815 ±  7%     +75.0%      18927 ±  5%  softirqs.CPU0.RCU
     12622 ±  7%    +125.4%      28452 ±  8%  softirqs.CPU0.SCHED
      8626 ± 13%    +101.7%      17398 ±  4%  softirqs.CPU1.RCU
      9883 ± 12%    +151.8%      24890 ±  7%  softirqs.CPU1.SCHED
      8171 ±  8%    +110.5%      17200 ±  5%  softirqs.CPU10.RCU
      7984 ±  8%    +198.5%      23832 ±  8%  softirqs.CPU10.SCHED
      7374 ±  7%    +150.5%      18472 ±  7%  softirqs.CPU100.RCU
      7018 ±  4%    +236.7%      23633 ±  9%  softirqs.CPU100.SCHED
      7413 ±  8%    +150.2%      18551 ±  7%  softirqs.CPU101.RCU
      6968 ±  4%    +240.8%      23748 ±  8%  softirqs.CPU101.SCHED
      7340 ±  5%    +156.1%      18799 ±  9%  softirqs.CPU102.RCU
      7126 ±  5%    +235.2%      23889 ±  9%  softirqs.CPU102.SCHED
      7206 ±  4%    +162.1%      18890 ±  9%  softirqs.CPU103.RCU
      7009 ±  4%    +239.7%      23809 ±  9%  softirqs.CPU103.SCHED
      7100 ±  3%    +165.9%      18879 ±  7%  softirqs.CPU104.RCU
      7007 ±  5%    +238.2%      23702 ±  9%  softirqs.CPU104.SCHED
      7139 ±  3%    +165.4%      18948 ±  7%  softirqs.CPU105.RCU
      6958 ±  3%    +242.5%      23831 ±  9%  softirqs.CPU105.SCHED
      7339 ±  5%    +156.2%      18803 ±  6%  softirqs.CPU106.RCU
      7021 ±  5%    +235.1%      23530 ±  8%  softirqs.CPU106.SCHED
      7745 ± 13%    +143.8%      18885 ±  7%  softirqs.CPU107.RCU
      7877 ± 16%    +215.8%      24878 ± 12%  softirqs.CPU107.SCHED
      6032 ±  5%    +162.7%      15850 ±  7%  softirqs.CPU108.RCU
      6971 ±  6%    +238.8%      23620 ±  9%  softirqs.CPU108.SCHED
      6203 ±  7%    +144.2%      15148 ±  6%  softirqs.CPU109.RCU
      7302 ±  9%    +223.4%      23616 ±  9%  softirqs.CPU109.SCHED
      8270 ±  8%    +126.1%      18702 ±  7%  softirqs.CPU11.RCU
      7439 ±  7%    +241.6%      25410 ± 11%  softirqs.CPU11.SCHED
      6065 ±  5%    +147.6%      15020 ±  6%  softirqs.CPU110.RCU
      6857 ± 12%    +243.6%      23565 ±  9%  softirqs.CPU110.SCHED
      6014 ±  5%    +158.0%      15519 ±  4%  softirqs.CPU111.RCU
      7015 ±  6%    +237.0%      23643 ±  8%  softirqs.CPU111.SCHED
      7635 ±  9%    +139.0%      18249 ±  8%  softirqs.CPU112.RCU
      6969 ±  4%    +240.0%      23694 ± 10%  softirqs.CPU112.SCHED
      7341 ±  2%    +156.3%      18815 ±  7%  softirqs.CPU113.RCU
      6881 ±  7%    +245.2%      23751 ±  9%  softirqs.CPU113.SCHED
      7364          +155.5%      18813 ±  8%  softirqs.CPU114.RCU
      7038 ±  6%    +234.8%      23569 ±  9%  softirqs.CPU114.SCHED
      7295 ±  4%    +150.6%      18283 ± 11%  softirqs.CPU115.RCU
      7022 ±  6%    +235.9%      23588 ±  9%  softirqs.CPU115.SCHED
      7437 ±  2%    +150.6%      18636 ±  7%  softirqs.CPU116.RCU
      7108 ±  7%    +232.8%      23655 ±  8%  softirqs.CPU116.SCHED
      7306 ±  2%    +155.6%      18674 ±  8%  softirqs.CPU117.RCU
      7077 ±  7%    +232.9%      23561 ±  9%  softirqs.CPU117.SCHED
      7401          +149.9%      18498 ±  7%  softirqs.CPU118.RCU
      7331 ± 11%    +223.2%      23697 ±  8%  softirqs.CPU118.SCHED
      7300 ±  2%    +155.6%      18661 ±  7%  softirqs.CPU119.RCU
      7040 ±  6%    +236.0%      23657 ±  9%  softirqs.CPU119.SCHED
      8881 ± 13%    +120.1%      19543 ± 15%  softirqs.CPU12.RCU
      8651 ± 19%    +177.6%      24016 ±  7%  softirqs.CPU12.SCHED
      7347 ±  3%    +152.3%      18541 ±  8%  softirqs.CPU120.RCU
      7111 ±  6%    +233.3%      23699 ±  9%  softirqs.CPU120.SCHED
      8354 ± 18%    +121.8%      18529 ±  7%  softirqs.CPU121.RCU
      6872 ±  4%    +245.7%      23759 ±  8%  softirqs.CPU121.SCHED
      7318 ±  6%    +153.8%      18570 ±  7%  softirqs.CPU122.RCU
      7038 ±  7%    +235.9%      23642 ±  9%  softirqs.CPU122.SCHED
      7418 ±  6%    +152.8%      18758 ±  6%  softirqs.CPU123.RCU
      7122 ±  7%    +234.4%      23819 ± 10%  softirqs.CPU123.SCHED
      7279 ±  3%    +155.2%      18574 ±  7%  softirqs.CPU124.RCU
      7001 ±  5%    +236.5%      23560 ±  9%  softirqs.CPU124.SCHED
      7619 ± 10%    +142.5%      18479 ±  6%  softirqs.CPU125.RCU
      7293 ± 23%    +232.4%      24245 ±  6%  softirqs.CPU125.SCHED
      5313 ±  5%    +156.7%      13641 ±  4%  softirqs.CPU126.RCU
      6947 ±  3%    +238.1%      23492 ±  9%  softirqs.CPU126.SCHED
      5554 ± 13%    +145.1%      13613 ±  4%  softirqs.CPU127.RCU
      7076 ±  4%    +231.2%      23441 ±  8%  softirqs.CPU127.SCHED
      7414 ± 15%    +127.4%      16863 ±  4%  softirqs.CPU128.RCU
      7113 ±  4%    +226.5%      23229 ±  9%  softirqs.CPU128.SCHED
      6822 ±  6%    +152.0%      17195 ±  4%  softirqs.CPU129.RCU
      6907 ±  4%    +243.6%      23730 ±  8%  softirqs.CPU129.SCHED
      7912 ± 10%    +126.8%      17947 ±  6%  softirqs.CPU13.RCU
      7374 ±  7%    +226.0%      24042 ±  7%  softirqs.CPU13.SCHED
      7122 ± 10%    +137.3%      16899 ±  4%  softirqs.CPU130.RCU
      6981 ±  3%    +234.3%      23335 ± 10%  softirqs.CPU130.SCHED
      7111 ± 10%    +141.1%      17145 ±  4%  softirqs.CPU131.RCU
      6993 ±  4%    +236.9%      23563 ± 10%  softirqs.CPU131.SCHED
      6976 ±  6%    +140.6%      16788 ±  4%  softirqs.CPU132.RCU
      6974 ±  4%    +237.5%      23543 ±  8%  softirqs.CPU132.SCHED
      7088 ±  6%    +139.5%      16974 ±  3%  softirqs.CPU133.RCU
      7051 ±  3%    +233.9%      23546 ±  8%  softirqs.CPU133.SCHED
      7095 ± 11%    +138.9%      16952 ±  4%  softirqs.CPU134.RCU
      7257 ± 10%    +224.5%      23548 ±  8%  softirqs.CPU134.SCHED
      6911 ±  5%    +147.8%      17126 ±  3%  softirqs.CPU135.RCU
      7030 ±  3%    +241.0%      23971 ±  9%  softirqs.CPU135.SCHED
      6996 ±  5%    +141.1%      16869 ±  4%  softirqs.CPU136.RCU
      7070 ±  3%    +232.2%      23491 ±  8%  softirqs.CPU136.SCHED
      7047 ±  6%    +138.4%      16799 ±  4%  softirqs.CPU137.RCU
      6998 ±  5%    +234.8%      23431 ±  9%  softirqs.CPU137.SCHED
      6901 ±  6%    +153.5%      17492 ± 11%  softirqs.CPU138.RCU
      7036 ±  4%    +235.3%      23591 ±  9%  softirqs.CPU138.SCHED
      7007 ±  6%    +141.2%      16905 ±  3%  softirqs.CPU139.RCU
      7077 ±  3%    +233.0%      23568 ±  8%  softirqs.CPU139.SCHED
      7657 ±  8%    +125.6%      17272 ±  7%  softirqs.CPU14.RCU
      7596 ±  9%    +211.7%      23676 ±  8%  softirqs.CPU14.SCHED
      6971 ±  8%    +142.8%      16927 ±  3%  softirqs.CPU140.RCU
      7455 ± 13%    +216.3%      23580 ±  8%  softirqs.CPU140.SCHED
      7003 ±  6%    +138.2%      16681 ±  5%  softirqs.CPU141.RCU
      7055 ±  4%    +232.3%      23447 ±  8%  softirqs.CPU141.SCHED
      6841 ±  6%    +150.5%      17134 ±  4%  softirqs.CPU142.RCU
      7045 ±  4%    +238.1%      23823 ±  9%  softirqs.CPU142.SCHED
      6916 ±  4%    +146.1%      17022 ±  3%  softirqs.CPU143.RCU
      6635 ±  6%    +247.5%      23056 ±  8%  softirqs.CPU143.SCHED
      9055 ± 17%     +91.7%      17362 ±  5%  softirqs.CPU15.RCU
      7529 ±  8%    +214.9%      23708 ±  8%  softirqs.CPU15.SCHED
      6375 ± 11%    +125.6%      14385 ±  8%  softirqs.CPU16.RCU
      7352 ±  5%    +222.4%      23699 ±  9%  softirqs.CPU16.SCHED
      6021 ±  2%    +138.4%      14358 ±  7%  softirqs.CPU17.RCU
      7199 ±  5%    +230.7%      23808 ±  8%  softirqs.CPU17.SCHED
      7547 ±  3%    +152.9%      19085 ±  7%  softirqs.CPU18.RCU
      7469 ± 12%    +220.0%      23903 ±  9%  softirqs.CPU18.SCHED
      7405 ±  3%    +156.1%      18965 ±  7%  softirqs.CPU19.RCU
      6945 ±  8%    +243.2%      23840 ±  9%  softirqs.CPU19.SCHED
      8323 ±  9%    +111.6%      17609 ±  6%  softirqs.CPU2.RCU
      7990 ±  8%    +205.1%      24379 ±  9%  softirqs.CPU2.SCHED
      7603 ±  3%    +148.0%      18861 ±  6%  softirqs.CPU20.RCU
      7076 ±  4%    +233.0%      23565 ±  8%  softirqs.CPU20.SCHED
      7536 ±  4%    +149.0%      18763 ±  6%  softirqs.CPU21.RCU
      6956 ±  4%    +242.3%      23811 ±  8%  softirqs.CPU21.SCHED
      7603 ±  4%    +151.3%      19104 ±  6%  softirqs.CPU22.RCU
      7005 ±  4%    +239.4%      23774 ±  8%  softirqs.CPU22.SCHED
      7815 ±  7%    +148.3%      19406 ±  7%  softirqs.CPU23.RCU
      6980 ±  4%    +240.3%      23752 ±  8%  softirqs.CPU23.SCHED
      7503 ±  3%    +153.4%      19012 ±  9%  softirqs.CPU24.RCU
      6976 ±  4%    +242.5%      23891 ±  9%  softirqs.CPU24.SCHED
      7936 ±  7%    +138.8%      18955 ±  7%  softirqs.CPU25.RCU
      7154 ±  6%    +233.1%      23832 ±  9%  softirqs.CPU25.SCHED
      7663 ±  4%    +147.7%      18984 ±  7%  softirqs.CPU26.RCU
      7036 ±  4%    +238.2%      23793 ±  8%  softirqs.CPU26.SCHED
      7585 ±  3%    +149.5%      18929 ±  7%  softirqs.CPU27.RCU
      7018 ±  4%    +242.7%      24050 ± 10%  softirqs.CPU27.SCHED
      7643 ±  3%    +149.7%      19084 ±  8%  softirqs.CPU28.RCU
      6952 ±  3%    +243.2%      23859 ±  9%  softirqs.CPU28.SCHED
      7751 ±  7%    +147.1%      19155 ±  9%  softirqs.CPU29.RCU
      6893 ±  8%    +248.0%      23991 ±  9%  softirqs.CPU29.SCHED
      7684 ±  7%    +125.5%      17326 ±  7%  softirqs.CPU3.RCU
      7587 ±  6%    +212.2%      23691 ±  8%  softirqs.CPU3.SCHED
      7667 ±  4%    +148.4%      19044 ± 10%  softirqs.CPU30.RCU
      7012 ±  4%    +241.5%      23945 ±  9%  softirqs.CPU30.SCHED
      7625 ±  3%    +151.3%      19163 ±  8%  softirqs.CPU31.RCU
      7049 ±  5%    +235.9%      23676 ±  8%  softirqs.CPU31.SCHED
      6267 ±  5%    +152.3%      15810 ± 10%  softirqs.CPU32.RCU
      6983 ±  4%    +240.3%      23763 ±  8%  softirqs.CPU32.SCHED
      6379 ±  7%    +148.2%      15831 ±  8%  softirqs.CPU33.RCU
      7070 ±  5%    +238.0%      23901 ±  9%  softirqs.CPU33.SCHED
      6283 ±  3%    +154.8%      16014 ±  9%  softirqs.CPU34.RCU
      6955 ±  4%    +245.2%      24012 ± 10%  softirqs.CPU34.SCHED
      6348 ±  5%    +152.1%      16002 ±  8%  softirqs.CPU35.RCU
      7025 ±  4%    +245.7%      24285 ±  9%  softirqs.CPU35.SCHED
      7711 ±  5%    +138.3%      18373 ±  7%  softirqs.CPU36.RCU
      6930 ±  5%    +239.2%      23506 ±  8%  softirqs.CPU36.SCHED
      7671 ±  3%    +138.1%      18266 ±  7%  softirqs.CPU37.RCU
      7109 ±  7%    +237.6%      24000 ±  8%  softirqs.CPU37.SCHED
      7452 ±  3%    +142.6%      18081 ±  7%  softirqs.CPU38.RCU
      6928 ±  7%    +244.6%      23875 ±  8%  softirqs.CPU38.SCHED
      8280 ± 16%    +127.0%      18799 ±  6%  softirqs.CPU39.RCU
      7003 ±  5%    +247.4%      24332 ±  8%  softirqs.CPU39.SCHED
      7541 ±  9%    +136.1%      17804 ±  6%  softirqs.CPU4.RCU
      7236 ±  4%    +229.4%      23835 ±  8%  softirqs.CPU4.SCHED
      8123 ± 15%    +123.9%      18190 ± 10%  softirqs.CPU40.RCU
      6932 ±  4%    +246.6%      24026 ±  9%  softirqs.CPU40.SCHED
      7735 ±  8%    +136.8%      18317 ±  7%  softirqs.CPU41.RCU
      7111 ±  7%    +233.6%      23722 ±  9%  softirqs.CPU41.SCHED
      7462 ±  2%    +147.5%      18466 ±  7%  softirqs.CPU42.RCU
      7114 ±  6%    +237.0%      23979 ±  9%  softirqs.CPU42.SCHED
      7477 ±  2%    +144.6%      18292 ±  8%  softirqs.CPU43.RCU
      7080 ±  5%    +236.4%      23815 ±  9%  softirqs.CPU43.SCHED
      7580 ±  2%    +144.7%      18546 ±  7%  softirqs.CPU44.RCU
      7048 ±  5%    +238.8%      23877 ±  9%  softirqs.CPU44.SCHED
      7373 ±  3%    +148.8%      18343 ±  7%  softirqs.CPU45.RCU
      6959 ±  4%    +243.8%      23928 ±  8%  softirqs.CPU45.SCHED
      7679 ±  5%    +138.3%      18297 ±  7%  softirqs.CPU46.RCU
      7037 ±  6%    +237.9%      23782 ±  9%  softirqs.CPU46.SCHED
      7521 ±  2%    +148.2%      18664 ±  6%  softirqs.CPU47.RCU
      7028 ±  6%    +239.8%      23886 ±  9%  softirqs.CPU47.SCHED
      6951 ±  4%    +143.4%      16918 ±  7%  softirqs.CPU48.RCU
      7081 ±  5%    +235.0%      23724 ±  9%  softirqs.CPU48.SCHED
      7224 ±  9%    +137.5%      17159 ±  7%  softirqs.CPU49.RCU
      7211 ±  8%    +231.9%      23935 ±  7%  softirqs.CPU49.SCHED
      7993 ± 13%    +118.1%      17438 ±  5%  softirqs.CPU5.RCU
      7338 ±  4%    +229.2%      24154 ±  7%  softirqs.CPU5.SCHED
      6685 ±  2%    +147.8%      16569 ±  7%  softirqs.CPU50.RCU
      6994 ±  5%    +239.6%      23751 ±  8%  softirqs.CPU50.SCHED
      6915 ±  4%    +143.2%      16821 ±  7%  softirqs.CPU51.RCU
      7008 ±  5%    +236.8%      23605 ±  9%  softirqs.CPU51.SCHED
      6976 ±  4%    +143.3%      16970 ±  7%  softirqs.CPU52.RCU
      7033 ±  6%    +232.9%      23415 ± 10%  softirqs.CPU52.SCHED
      6957 ±  6%    +144.1%      16981 ±  7%  softirqs.CPU53.RCU
      7009 ±  6%    +240.3%      23852 ±  8%  softirqs.CPU53.SCHED
      7206 ±  6%    +135.5%      16971 ±  4%  softirqs.CPU54.RCU
      7143 ±  3%    +228.0%      23431 ±  9%  softirqs.CPU54.SCHED
      7055 ±  3%    +143.1%      17152 ±  4%  softirqs.CPU55.RCU
      7268 ± 12%    +218.2%      23126 ± 10%  softirqs.CPU55.SCHED
      6974 ±  4%    +145.9%      17149 ±  4%  softirqs.CPU56.RCU
      6974 ±  4%    +239.0%      23644 ±  8%  softirqs.CPU56.SCHED
      7307 ±  7%    +135.0%      17171 ±  4%  softirqs.CPU57.RCU
      7077 ±  4%    +232.8%      23555 ±  8%  softirqs.CPU57.SCHED
      7012 ±  4%    +148.6%      17432 ±  4%  softirqs.CPU58.RCU
      6992 ±  3%    +235.4%      23452 ±  8%  softirqs.CPU58.SCHED
      7011 ±  8%    +142.9%      17029 ±  5%  softirqs.CPU59.RCU
      6979 ±  4%    +236.4%      23478 ±  9%  softirqs.CPU59.SCHED
      7519 ±  5%    +129.5%      17255 ±  5%  softirqs.CPU6.RCU
      7198 ±  5%    +226.2%      23479 ±  9%  softirqs.CPU6.SCHED
      6924 ±  4%    +149.2%      17253 ±  4%  softirqs.CPU60.RCU
      6984 ±  3%    +238.3%      23628 ±  9%  softirqs.CPU60.SCHED
      7311 ±  6%    +141.1%      17623 ±  4%  softirqs.CPU61.RCU
      7254 ± 10%    +223.4%      23458 ±  8%  softirqs.CPU61.SCHED
      7365 ±  8%    +137.6%      17496 ±  4%  softirqs.CPU62.RCU
      7073 ±  3%    +233.8%      23614 ±  8%  softirqs.CPU62.SCHED
      7169 ± 10%    +137.2%      17007 ±  4%  softirqs.CPU63.RCU
      7141 ±  3%    +226.7%      23331 ± 10%  softirqs.CPU63.SCHED
      6942 ±  5%    +136.6%      16427 ±  4%  softirqs.CPU64.RCU
      7338 ±  8%    +221.8%      23617 ±  8%  softirqs.CPU64.SCHED
      6722 ±  2%    +145.5%      16504 ±  4%  softirqs.CPU65.RCU
      6997 ±  4%    +236.6%      23552 ±  8%  softirqs.CPU65.SCHED
      7092 ±  7%    +135.1%      16673 ±  4%  softirqs.CPU66.RCU
      7137 ±  4%    +229.5%      23521 ±  9%  softirqs.CPU66.SCHED
      7049 ±  9%    +137.0%      16708 ±  4%  softirqs.CPU67.RCU
      6987 ±  4%    +241.8%      23887 ± 10%  softirqs.CPU67.SCHED
      6846 ±  4%    +143.9%      16695 ±  4%  softirqs.CPU68.RCU
      7191 ±  4%    +229.1%      23666 ±  8%  softirqs.CPU68.SCHED
      6809 ±  6%    +138.5%      16237 ±  5%  softirqs.CPU69.RCU
      6984 ±  3%    +235.4%      23426 ±  9%  softirqs.CPU69.SCHED
      7592 ±  7%    +129.3%      17406 ±  7%  softirqs.CPU7.RCU
      7198 ±  3%    +226.9%      23529 ±  7%  softirqs.CPU7.SCHED
      6869 ±  4%    +139.2%      16434 ±  4%  softirqs.CPU70.RCU
      7063 ±  3%    +235.0%      23661 ±  8%  softirqs.CPU70.SCHED
      6903 ±  6%    +140.4%      16594 ±  4%  softirqs.CPU71.RCU
      7067 ±  3%    +232.9%      23528 ±  8%  softirqs.CPU71.SCHED
      7060 ±  4%    +139.3%      16895 ±  8%  softirqs.CPU72.RCU
      6673 ± 10%    +251.9%      23484 ±  9%  softirqs.CPU72.SCHED
      6712 ±  6%    +152.2%      16930 ±  7%  softirqs.CPU73.RCU
      7038 ±  5%    +239.0%      23862 ±  9%  softirqs.CPU73.SCHED
      6858 ±  4%    +151.0%      17218 ±  6%  softirqs.CPU74.RCU
      6761 ± 13%    +250.1%      23672 ±  8%  softirqs.CPU74.SCHED
      7050 ±  4%    +145.5%      17311 ±  6%  softirqs.CPU75.RCU
      7119 ±  5%    +232.1%      23640 ±  8%  softirqs.CPU75.SCHED
      7115 ±  6%    +140.1%      17088 ±  6%  softirqs.CPU76.RCU
      7207 ±  5%    +221.1%      23142 ±  9%  softirqs.CPU76.SCHED
      7311 ±  7%    +135.4%      17210 ±  6%  softirqs.CPU77.RCU
      7264 ± 12%    +231.6%      24086 ± 10%  softirqs.CPU77.SCHED
      6991 ±  3%    +143.4%      17016 ±  6%  softirqs.CPU78.RCU
      6384 ± 10%    +269.7%      23603 ±  9%  softirqs.CPU78.SCHED
      6827 ±  4%    +148.5%      16965 ±  5%  softirqs.CPU79.RCU
      7175 ±  5%    +229.4%      23636 ±  8%  softirqs.CPU79.SCHED
      7790 ±  4%    +132.3%      18093 ±  8%  softirqs.CPU8.RCU
      7776 ± 10%    +204.8%      23700 ±  9%  softirqs.CPU8.SCHED
      7173 ±  8%    +147.9%      17782 ±  5%  softirqs.CPU80.RCU
      7024 ±  5%    +236.4%      23629 ±  8%  softirqs.CPU80.SCHED
      6849 ±  7%    +159.8%      17796 ±  6%  softirqs.CPU81.RCU
      7089 ±  5%    +235.3%      23768 ±  8%  softirqs.CPU81.SCHED
      7285 ±  4%    +143.7%      17755 ±  6%  softirqs.CPU82.RCU
      7120 ±  4%    +232.0%      23640 ±  8%  softirqs.CPU82.SCHED
      7064 ±  4%    +149.7%      17643 ±  5%  softirqs.CPU83.RCU
      7046 ±  4%    +233.3%      23485 ±  8%  softirqs.CPU83.SCHED
      7375 ±  7%    +140.9%      17770 ±  7%  softirqs.CPU84.RCU
      7171 ±  4%    +230.9%      23729 ±  8%  softirqs.CPU84.SCHED
      7274 ±  9%    +142.5%      17640 ±  5%  softirqs.CPU85.RCU
      7155 ±  5%    +229.2%      23553 ±  8%  softirqs.CPU85.SCHED
      8084 ± 21%    +124.9%      18179 ±  7%  softirqs.CPU86.RCU
      7262 ±  4%    +226.1%      23683 ±  8%  softirqs.CPU86.SCHED
      7756 ± 10%    +130.7%      17892 ±  6%  softirqs.CPU87.RCU
      7348 ±  5%    +222.1%      23666 ±  8%  softirqs.CPU87.SCHED
      7336 ±  4%    +146.4%      18076 ±  7%  softirqs.CPU88.RCU
      7183 ±  4%    +230.0%      23702 ±  8%  softirqs.CPU88.SCHED
      7696 ±  9%    +142.6%      18670 ±  8%  softirqs.CPU89.RCU
      7821 ±  9%    +205.4%      23884 ±  7%  softirqs.CPU89.SCHED
     10128 ± 18%     +75.5%      17773 ±  8%  softirqs.CPU9.RCU
     10395 ± 17%    +130.4%      23951 ±  7%  softirqs.CPU9.SCHED
      6410 ±  2%    +171.3%      17394 ± 10%  softirqs.CPU90.RCU
      6999 ±  4%    +240.6%      23839 ±  9%  softirqs.CPU90.SCHED
      6715 ± 11%    +154.3%      17077 ± 10%  softirqs.CPU91.RCU
      7277 ±  7%    +223.5%      23546 ±  8%  softirqs.CPU91.SCHED
      6445 ±  2%    +162.1%      16895 ±  8%  softirqs.CPU92.RCU
      6858 ±  4%    +247.1%      23806 ±  9%  softirqs.CPU92.SCHED
      6511 ±  2%    +156.4%      16697 ±  7%  softirqs.CPU93.RCU
      6897 ±  4%    +244.0%      23724 ±  8%  softirqs.CPU93.SCHED
      6447          +163.8%      17008 ±  7%  softirqs.CPU94.RCU
      7024 ±  4%    +236.4%      23633 ±  8%  softirqs.CPU94.SCHED
      6573 ±  4%    +155.3%      16785 ±  8%  softirqs.CPU95.RCU
      7115 ±  5%    +233.9%      23761 ±  8%  softirqs.CPU95.SCHED
      7281 ±  4%    +155.0%      18568 ±  9%  softirqs.CPU96.RCU
      7036 ±  4%    +237.7%      23762 ±  9%  softirqs.CPU96.SCHED
      7311 ±  5%    +158.7%      18918 ±  6%  softirqs.CPU97.RCU
      6970 ±  3%    +244.0%      23979 ±  9%  softirqs.CPU97.SCHED
      7275 ±  5%    +155.6%      18594 ±  7%  softirqs.CPU98.RCU
      6978 ±  4%    +241.9%      23857 ±  9%  softirqs.CPU98.SCHED
      7223 ±  3%    +158.7%      18690 ±  8%  softirqs.CPU99.RCU
      6980 ±  3%    +240.0%      23734 ±  8%  softirqs.CPU99.SCHED
   1047424 ±  3%    +141.9%    2534003 ±  5%  softirqs.RCU
   1035968 ±  4%    +230.4%    3422827 ±  8%  softirqs.SCHED
     62978 ±  3%     +39.5%      87842 ±  2%  softirqs.TIMER
    445184 ± 17%    +827.6%    4129649 ± 13%  interrupts.CAL:Function_call_interrupts
      3492 ± 31%    +739.6%      29320 ± 16%  interrupts.CPU0.CAL:Function_call_interrupts
    487082           +52.0%     740155 ±  5%  interrupts.CPU0.LOC:Local_timer_interrupts
    869.50 ± 14%    +233.1%       2896 ± 14%  interrupts.CPU0.RES:Rescheduling_interrupts
      3490 ± 30%    +715.0%      28448 ± 15%  interrupts.CPU1.CAL:Function_call_interrupts
    487092           +52.2%     741504 ±  4%  interrupts.CPU1.LOC:Local_timer_interrupts
    714.33 ± 13%    +249.9%       2499 ±  7%  interrupts.CPU1.RES:Rescheduling_interrupts
      3023 ± 21%    +833.5%      28220 ± 15%  interrupts.CPU10.CAL:Function_call_interrupts
    487061           +52.3%     741612 ±  4%  interrupts.CPU10.LOC:Local_timer_interrupts
    736.50 ± 16%    +217.9%       2341 ±  7%  interrupts.CPU10.RES:Rescheduling_interrupts
      3101 ± 18%    +827.1%      28754 ± 12%  interrupts.CPU100.CAL:Function_call_interrupts
    487296           +52.5%     743050 ±  4%  interrupts.CPU100.LOC:Local_timer_interrupts
    428.67 ±  6%    +420.0%       2229 ± 10%  interrupts.CPU100.RES:Rescheduling_interrupts
      3075 ± 16%    +834.5%      28737 ± 12%  interrupts.CPU101.CAL:Function_call_interrupts
    487344           +52.4%     742943 ±  4%  interrupts.CPU101.LOC:Local_timer_interrupts
    449.17 ±  6%    +387.3%       2188 ±  9%  interrupts.CPU101.RES:Rescheduling_interrupts
      3089 ± 16%    +839.1%      29014 ± 12%  interrupts.CPU102.CAL:Function_call_interrupts
    487370           +52.4%     742976 ±  4%  interrupts.CPU102.LOC:Local_timer_interrupts
    451.83 ±  6%    +392.4%       2224 ± 12%  interrupts.CPU102.RES:Rescheduling_interrupts
      3106 ± 18%    +829.1%      28861 ± 11%  interrupts.CPU103.CAL:Function_call_interrupts
    487340           +52.5%     742952 ±  4%  interrupts.CPU103.LOC:Local_timer_interrupts
    454.00 ±  8%    +386.8%       2210 ±  9%  interrupts.CPU103.RES:Rescheduling_interrupts
      3101 ± 17%    +829.2%      28821 ± 11%  interrupts.CPU104.CAL:Function_call_interrupts
    487360           +52.4%     742860 ±  4%  interrupts.CPU104.LOC:Local_timer_interrupts
    441.00 ±  4%    +400.6%       2207 ± 11%  interrupts.CPU104.RES:Rescheduling_interrupts
      3074 ± 17%    +829.6%      28575 ± 11%  interrupts.CPU105.CAL:Function_call_interrupts
    487347           +52.4%     742943 ±  4%  interrupts.CPU105.LOC:Local_timer_interrupts
    425.67 ±  3%    +415.0%       2192 ± 11%  interrupts.CPU105.RES:Rescheduling_interrupts
      3099 ± 15%    +825.3%      28684 ± 11%  interrupts.CPU106.CAL:Function_call_interrupts
    487373           +52.4%     742836 ±  4%  interrupts.CPU106.LOC:Local_timer_interrupts
    453.83 ±  6%    +394.5%       2244 ± 12%  interrupts.CPU106.RES:Rescheduling_interrupts
      3085 ± 17%    +835.0%      28851 ± 11%  interrupts.CPU107.CAL:Function_call_interrupts
    487353           +52.4%     742823 ±  4%  interrupts.CPU107.LOC:Local_timer_interrupts
    446.67 ±  7%    +392.0%       2197 ± 11%  interrupts.CPU107.RES:Rescheduling_interrupts
      3029 ± 20%    +848.3%      28729 ± 14%  interrupts.CPU108.CAL:Function_call_interrupts
    486780           +52.7%     743168 ±  4%  interrupts.CPU108.LOC:Local_timer_interrupts
    453.33 ± 16%    +368.8%       2125 ±  8%  interrupts.CPU108.RES:Rescheduling_interrupts
      3065 ± 20%    +843.4%      28916 ± 14%  interrupts.CPU109.CAL:Function_call_interrupts
    486706           +52.7%     743121 ±  4%  interrupts.CPU109.LOC:Local_timer_interrupts
    486.67 ± 12%    +349.9%       2189 ±  7%  interrupts.CPU109.RES:Rescheduling_interrupts
      3027 ± 17%    +835.6%      28321 ± 14%  interrupts.CPU11.CAL:Function_call_interrupts
    487015           +52.2%     741464 ±  4%  interrupts.CPU11.LOC:Local_timer_interrupts
    713.17 ± 11%    +239.8%       2423 ±  7%  interrupts.CPU11.RES:Rescheduling_interrupts
      3152 ± 14%    +818.6%      28959 ± 15%  interrupts.CPU110.CAL:Function_call_interrupts
    486728           +52.7%     743215 ±  4%  interrupts.CPU110.LOC:Local_timer_interrupts
    497.50 ± 31%    +330.4%       2141 ±  8%  interrupts.CPU110.RES:Rescheduling_interrupts
      2994 ± 21%    +867.7%      28980 ± 14%  interrupts.CPU111.CAL:Function_call_interrupts
    486750           +52.7%     743088 ±  4%  interrupts.CPU111.LOC:Local_timer_interrupts
    452.33 ±  7%    +380.0%       2171 ±  5%  interrupts.CPU111.RES:Rescheduling_interrupts
      3081 ± 19%    +832.4%      28728 ± 14%  interrupts.CPU112.CAL:Function_call_interrupts
    486734           +50.8%     733783 ±  7%  interrupts.CPU112.LOC:Local_timer_interrupts
    460.33 ± 16%    +362.6%       2129 ±  7%  interrupts.CPU112.RES:Rescheduling_interrupts
      3073 ± 20%    +840.5%      28902 ± 14%  interrupts.CPU113.CAL:Function_call_interrupts
    486755           +52.7%     743229 ±  4%  interrupts.CPU113.LOC:Local_timer_interrupts
    479.67 ± 12%    +343.7%       2128 ± 10%  interrupts.CPU113.RES:Rescheduling_interrupts
      3070 ± 20%    +841.3%      28902 ± 14%  interrupts.CPU114.CAL:Function_call_interrupts
    486791           +52.7%     743096 ±  4%  interrupts.CPU114.LOC:Local_timer_interrupts
    499.67 ± 24%    +332.3%       2160 ±  8%  interrupts.CPU114.RES:Rescheduling_interrupts
      3114 ± 19%    +829.7%      28954 ± 14%  interrupts.CPU115.CAL:Function_call_interrupts
    486756           +52.7%     743215 ±  4%  interrupts.CPU115.LOC:Local_timer_interrupts
    477.33 ± 18%    +350.4%       2149 ±  9%  interrupts.CPU115.RES:Rescheduling_interrupts
      3030 ± 19%    +847.1%      28703 ± 14%  interrupts.CPU116.CAL:Function_call_interrupts
    486791           +52.7%     743179 ±  4%  interrupts.CPU116.LOC:Local_timer_interrupts
    467.50 ± 17%    +352.2%       2113 ±  9%  interrupts.CPU116.RES:Rescheduling_interrupts
      3055 ± 20%    +842.1%      28784 ± 15%  interrupts.CPU117.CAL:Function_call_interrupts
    486795           +52.7%     743347 ±  4%  interrupts.CPU117.LOC:Local_timer_interrupts
    462.17 ± 20%    +357.7%       2115 ±  7%  interrupts.CPU117.RES:Rescheduling_interrupts
      3094 ± 21%    +833.8%      28900 ± 14%  interrupts.CPU118.CAL:Function_call_interrupts
    486828           +52.7%     743378 ±  4%  interrupts.CPU118.LOC:Local_timer_interrupts
    479.17 ± 22%    +342.0%       2117 ±  6%  interrupts.CPU118.RES:Rescheduling_interrupts
      3122 ± 16%    +831.0%      29071 ± 14%  interrupts.CPU119.CAL:Function_call_interrupts
    486797           +52.7%     743293 ±  4%  interrupts.CPU119.LOC:Local_timer_interrupts
    478.50 ± 19%    +348.6%       2146 ± 10%  interrupts.CPU119.RES:Rescheduling_interrupts
      2949 ± 20%    +858.9%      28280 ± 15%  interrupts.CPU12.CAL:Function_call_interrupts
    487292           +52.2%     741434 ±  4%  interrupts.CPU12.LOC:Local_timer_interrupts
    712.83 ± 14%    +238.1%       2410 ±  8%  interrupts.CPU12.RES:Rescheduling_interrupts
      3068 ± 18%    +846.8%      29047 ± 14%  interrupts.CPU120.CAL:Function_call_interrupts
    486780           +52.7%     743502 ±  4%  interrupts.CPU120.LOC:Local_timer_interrupts
    469.00 ± 16%    +347.3%       2097 ±  7%  interrupts.CPU120.RES:Rescheduling_interrupts
      3089 ± 18%    +836.3%      28921 ± 14%  interrupts.CPU121.CAL:Function_call_interrupts
    486752           +52.7%     743238 ±  4%  interrupts.CPU121.LOC:Local_timer_interrupts
    462.50 ± 18%    +359.4%       2124 ±  7%  interrupts.CPU121.RES:Rescheduling_interrupts
      3082 ± 18%    +834.4%      28798 ± 14%  interrupts.CPU122.CAL:Function_call_interrupts
    486770           +52.7%     743206 ±  4%  interrupts.CPU122.LOC:Local_timer_interrupts
    473.67 ± 27%    +334.8%       2059 ±  6%  interrupts.CPU122.RES:Rescheduling_interrupts
      3087 ± 19%    +836.5%      28917 ± 14%  interrupts.CPU123.CAL:Function_call_interrupts
    486767           +52.7%     743240 ±  4%  interrupts.CPU123.LOC:Local_timer_interrupts
    464.83 ± 25%    +347.4%       2079 ±  8%  interrupts.CPU123.RES:Rescheduling_interrupts
      3055 ± 18%    +843.7%      28833 ± 14%  interrupts.CPU124.CAL:Function_call_interrupts
    486694           +52.7%     743032 ±  4%  interrupts.CPU124.LOC:Local_timer_interrupts
    443.50 ± 19%    +373.3%       2099 ±  9%  interrupts.CPU124.RES:Rescheduling_interrupts
      3090 ± 19%    +838.9%      29015 ± 13%  interrupts.CPU125.CAL:Function_call_interrupts
    486797           +52.6%     743008 ±  4%  interrupts.CPU125.LOC:Local_timer_interrupts
    472.83 ± 24%    +346.5%       2111 ±  6%  interrupts.CPU125.RES:Rescheduling_interrupts
      3046 ± 17%    +839.8%      28624 ± 13%  interrupts.CPU126.CAL:Function_call_interrupts
    486711           +52.7%     743080 ±  4%  interrupts.CPU126.LOC:Local_timer_interrupts
    440.00 ± 12%    +403.9%       2217 ±  9%  interrupts.CPU126.RES:Rescheduling_interrupts
      3081 ± 17%    +826.6%      28555 ± 14%  interrupts.CPU127.CAL:Function_call_interrupts
    486703           +52.7%     743033 ±  4%  interrupts.CPU127.LOC:Local_timer_interrupts
    444.17 ± 14%    +400.3%       2222 ± 10%  interrupts.CPU127.RES:Rescheduling_interrupts
      3082 ± 16%    +825.7%      28534 ± 13%  interrupts.CPU128.CAL:Function_call_interrupts
    486688           +52.7%     743207 ±  4%  interrupts.CPU128.LOC:Local_timer_interrupts
    460.50 ± 14%    +377.2%       2197 ± 12%  interrupts.CPU128.RES:Rescheduling_interrupts
      3040 ± 17%    +846.4%      28774 ± 13%  interrupts.CPU129.CAL:Function_call_interrupts
    486683           +52.7%     743097 ±  4%  interrupts.CPU129.LOC:Local_timer_interrupts
    440.67 ± 12%    +410.0%       2247 ± 11%  interrupts.CPU129.RES:Rescheduling_interrupts
      2942 ± 20%    +854.7%      28090 ± 15%  interrupts.CPU13.CAL:Function_call_interrupts
    487053           +52.2%     741255 ±  5%  interrupts.CPU13.LOC:Local_timer_interrupts
    684.17 ± 13%    +244.9%       2359 ±  5%  interrupts.CPU13.RES:Rescheduling_interrupts
      3083 ± 19%    +818.8%      28334 ± 13%  interrupts.CPU130.CAL:Function_call_interrupts
    486682           +52.7%     743009 ±  4%  interrupts.CPU130.LOC:Local_timer_interrupts
    446.17 ± 12%    +391.6%       2193 ± 10%  interrupts.CPU130.RES:Rescheduling_interrupts
      3078 ± 17%    +810.2%      28018 ± 13%  interrupts.CPU131.CAL:Function_call_interrupts
    486681           +52.7%     743341 ±  4%  interrupts.CPU131.LOC:Local_timer_interrupts
    429.00 ± 12%    +400.9%       2148 ± 10%  interrupts.CPU131.RES:Rescheduling_interrupts
      3076 ± 16%    +829.4%      28592 ± 13%  interrupts.CPU132.CAL:Function_call_interrupts
    486702           +52.7%     743190 ±  4%  interrupts.CPU132.LOC:Local_timer_interrupts
    459.00 ±  8%    +380.2%       2204 ±  9%  interrupts.CPU132.RES:Rescheduling_interrupts
      3112 ± 17%    +819.5%      28614 ± 14%  interrupts.CPU133.CAL:Function_call_interrupts
    486690           +52.7%     743126 ±  4%  interrupts.CPU133.LOC:Local_timer_interrupts
    452.17 ± 13%    +406.0%       2288 ±  7%  interrupts.CPU133.RES:Rescheduling_interrupts
      3111 ± 16%    +824.4%      28764 ± 14%  interrupts.CPU134.CAL:Function_call_interrupts
    486723           +52.6%     742953 ±  4%  interrupts.CPU134.LOC:Local_timer_interrupts
    445.33 ± 11%    +398.3%       2219 ± 10%  interrupts.CPU134.RES:Rescheduling_interrupts
      3085 ± 17%    +834.2%      28820 ± 13%  interrupts.CPU135.CAL:Function_call_interrupts
    486699           +52.7%     743192 ±  4%  interrupts.CPU135.LOC:Local_timer_interrupts
    436.50 ±  7%    +412.9%       2239 ± 11%  interrupts.CPU135.RES:Rescheduling_interrupts
      3092 ± 18%    +821.1%      28482 ± 13%  interrupts.CPU136.CAL:Function_call_interrupts
    486679           +52.7%     742986 ±  4%  interrupts.CPU136.LOC:Local_timer_interrupts
    433.83 ± 11%    +420.3%       2257 ±  9%  interrupts.CPU136.RES:Rescheduling_interrupts
      3096 ± 17%    +821.6%      28534 ± 13%  interrupts.CPU137.CAL:Function_call_interrupts
    486675           +52.7%     743227 ±  4%  interrupts.CPU137.LOC:Local_timer_interrupts
    448.17 ± 10%    +398.1%       2232 ± 10%  interrupts.CPU137.RES:Rescheduling_interrupts
      3023 ± 18%    +850.2%      28725 ± 14%  interrupts.CPU138.CAL:Function_call_interrupts
    486711           +52.7%     742996 ±  4%  interrupts.CPU138.LOC:Local_timer_interrupts
    440.50 ±  8%    +400.0%       2202 ±  9%  interrupts.CPU138.RES:Rescheduling_interrupts
      3033 ± 19%    +848.2%      28767 ± 14%  interrupts.CPU139.CAL:Function_call_interrupts
    486719           +52.7%     742997 ±  4%  interrupts.CPU139.LOC:Local_timer_interrupts
    442.17 ± 13%    +412.9%       2267 ±  9%  interrupts.CPU139.RES:Rescheduling_interrupts
      2961 ± 20%    +867.7%      28655 ± 14%  interrupts.CPU14.CAL:Function_call_interrupts
    486965           +52.3%     741610 ±  4%  interrupts.CPU14.LOC:Local_timer_interrupts
    686.83 ± 11%    +250.3%       2406 ±  6%  interrupts.CPU14.RES:Rescheduling_interrupts
      3023 ± 15%    +852.5%      28798 ± 13%  interrupts.CPU140.CAL:Function_call_interrupts
    486753           +52.7%     743181 ±  4%  interrupts.CPU140.LOC:Local_timer_interrupts
    461.00 ± 14%    +381.8%       2221 ±  8%  interrupts.CPU140.RES:Rescheduling_interrupts
      3042 ± 18%    +839.3%      28572 ± 12%  interrupts.CPU141.CAL:Function_call_interrupts
    486715           +52.7%     743148 ±  4%  interrupts.CPU141.LOC:Local_timer_interrupts
    437.67 ± 13%    +399.3%       2185 ±  9%  interrupts.CPU141.RES:Rescheduling_interrupts
      3100 ± 15%    +823.1%      28620 ± 13%  interrupts.CPU142.CAL:Function_call_interrupts
    486722           +52.7%     742995 ±  4%  interrupts.CPU142.LOC:Local_timer_interrupts
    461.50 ± 13%    +384.1%       2234 ±  9%  interrupts.CPU142.RES:Rescheduling_interrupts
      3131 ± 16%    +813.3%      28595 ± 13%  interrupts.CPU143.CAL:Function_call_interrupts
    486720           +52.7%     742985 ±  4%  interrupts.CPU143.LOC:Local_timer_interrupts
    503.83 ± 13%    +346.3%       2248 ±  9%  interrupts.CPU143.RES:Rescheduling_interrupts
      2983 ± 21%    +854.1%      28466 ± 14%  interrupts.CPU15.CAL:Function_call_interrupts
    487042           +52.2%     741464 ±  4%  interrupts.CPU15.LOC:Local_timer_interrupts
    722.33 ± 16%    +230.0%       2383 ±  6%  interrupts.CPU15.RES:Rescheduling_interrupts
      2940 ± 19%    +861.2%      28259 ± 15%  interrupts.CPU16.CAL:Function_call_interrupts
    487099           +52.2%     741364 ±  4%  interrupts.CPU16.LOC:Local_timer_interrupts
    674.00 ± 14%    +250.5%       2362 ±  5%  interrupts.CPU16.RES:Rescheduling_interrupts
      2858 ± 20%    +897.5%      28508 ± 15%  interrupts.CPU17.CAL:Function_call_interrupts
    487086           +52.2%     741199 ±  5%  interrupts.CPU17.LOC:Local_timer_interrupts
    695.50 ± 17%    +246.8%       2411 ±  7%  interrupts.CPU17.RES:Rescheduling_interrupts
      3268 ± 22%    +781.3%      28805 ± 12%  interrupts.CPU18.CAL:Function_call_interrupts
    487377           +52.4%     742840 ±  4%  interrupts.CPU18.LOC:Local_timer_interrupts
      1052 ±  2%    +165.1%       2789 ±  8%  interrupts.CPU18.RES:Rescheduling_interrupts
      3324 ± 14%    +767.1%      28827 ± 11%  interrupts.CPU19.CAL:Function_call_interrupts
    487369           +52.4%     742896 ±  4%  interrupts.CPU19.LOC:Local_timer_interrupts
    611.50 ±  2%    +290.2%       2386 ±  9%  interrupts.CPU19.RES:Rescheduling_interrupts
      3171 ± 11%    +791.5%      28275 ± 15%  interrupts.CPU2.CAL:Function_call_interrupts
    487065           +52.2%     741466 ±  4%  interrupts.CPU2.LOC:Local_timer_interrupts
    717.50 ± 19%    +227.4%       2349 ±  7%  interrupts.CPU2.RES:Rescheduling_interrupts
      3153 ± 17%    +818.4%      28958 ± 11%  interrupts.CPU20.CAL:Function_call_interrupts
    487382           +52.4%     742811 ±  4%  interrupts.CPU20.LOC:Local_timer_interrupts
    564.83 ±  4%    +315.8%       2348 ± 11%  interrupts.CPU20.RES:Rescheduling_interrupts
      3155 ± 18%    +811.4%      28757 ± 11%  interrupts.CPU21.CAL:Function_call_interrupts
    487271           +52.4%     742756 ±  4%  interrupts.CPU21.LOC:Local_timer_interrupts
    589.67 ±  4%    +293.0%       2317 ±  9%  interrupts.CPU21.RES:Rescheduling_interrupts
      3263 ± 14%    +781.8%      28776 ± 11%  interrupts.CPU22.CAL:Function_call_interrupts
    487401           +52.4%     742929 ±  4%  interrupts.CPU22.LOC:Local_timer_interrupts
    588.67 ±  5%    +294.2%       2320 ±  9%  interrupts.CPU22.RES:Rescheduling_interrupts
      3123 ± 16%    +815.3%      28583 ± 11%  interrupts.CPU23.CAL:Function_call_interrupts
    487357           +52.4%     742935 ±  4%  interrupts.CPU23.LOC:Local_timer_interrupts
    575.50 ±  3%    +299.9%       2301 ± 10%  interrupts.CPU23.RES:Rescheduling_interrupts
      3094 ± 17%    +835.4%      28941 ± 11%  interrupts.CPU24.CAL:Function_call_interrupts
    487248           +52.5%     742909 ±  4%  interrupts.CPU24.LOC:Local_timer_interrupts
    584.67 ±  7%    +295.0%       2309 ±  9%  interrupts.CPU24.RES:Rescheduling_interrupts
      3189 ± 16%    +803.9%      28830 ± 12%  interrupts.CPU25.CAL:Function_call_interrupts
    487361           +52.4%     742807 ±  4%  interrupts.CPU25.LOC:Local_timer_interrupts
    598.33 ±  8%    +292.5%       2348 ±  8%  interrupts.CPU25.RES:Rescheduling_interrupts
      2958 ± 17%    +871.5%      28737 ± 11%  interrupts.CPU26.CAL:Function_call_interrupts
    487369           +52.4%     742721 ±  4%  interrupts.CPU26.LOC:Local_timer_interrupts
    573.83 ±  5%    +303.9%       2317 ±  9%  interrupts.CPU26.RES:Rescheduling_interrupts
      3081 ± 17%    +838.8%      28928 ± 12%  interrupts.CPU27.CAL:Function_call_interrupts
    487395           +52.4%     742997 ±  4%  interrupts.CPU27.LOC:Local_timer_interrupts
    575.33 ±  4%    +299.8%       2300 ± 10%  interrupts.CPU27.RES:Rescheduling_interrupts
      3103 ± 17%    +823.2%      28650 ± 12%  interrupts.CPU28.CAL:Function_call_interrupts
    487388           +52.4%     742886 ±  4%  interrupts.CPU28.LOC:Local_timer_interrupts
    578.67 ±  4%    +300.5%       2317 ± 11%  interrupts.CPU28.RES:Rescheduling_interrupts
      2929 ± 17%    +873.1%      28509 ± 10%  interrupts.CPU29.CAL:Function_call_interrupts
    487445           +52.4%     742893 ±  4%  interrupts.CPU29.LOC:Local_timer_interrupts
    564.50 ±  6%    +312.8%       2330 ±  9%  interrupts.CPU29.RES:Rescheduling_interrupts
      2985 ± 19%    +851.1%      28397 ± 14%  interrupts.CPU3.CAL:Function_call_interrupts
    487074           +52.2%     741389 ±  4%  interrupts.CPU3.LOC:Local_timer_interrupts
    695.33 ± 13%    +238.2%       2351 ±  7%  interrupts.CPU3.RES:Rescheduling_interrupts
      3114 ± 16%    +827.8%      28896 ± 12%  interrupts.CPU30.CAL:Function_call_interrupts
    487375           +52.4%     742816 ±  4%  interrupts.CPU30.LOC:Local_timer_interrupts
    585.00 ±  4%    +306.9%       2380 ± 11%  interrupts.CPU30.RES:Rescheduling_interrupts
      3127 ± 17%    +813.9%      28582 ± 12%  interrupts.CPU31.CAL:Function_call_interrupts
    487393           +52.4%     742865 ±  4%  interrupts.CPU31.LOC:Local_timer_interrupts
    579.50 ±  3%    +305.3%       2348 ± 10%  interrupts.CPU31.RES:Rescheduling_interrupts
      2980 ± 18%    +862.4%      28686 ± 12%  interrupts.CPU32.CAL:Function_call_interrupts
    487370           +52.4%     742989 ±  4%  interrupts.CPU32.LOC:Local_timer_interrupts
    563.50 ±  6%    +308.7%       2303 ± 10%  interrupts.CPU32.RES:Rescheduling_interrupts
      3119 ± 18%    +820.7%      28719 ± 11%  interrupts.CPU33.CAL:Function_call_interrupts
    487360           +52.4%     742960 ±  4%  interrupts.CPU33.LOC:Local_timer_interrupts
    587.00 ±  5%    +296.8%       2329 ± 11%  interrupts.CPU33.RES:Rescheduling_interrupts
      3083 ± 17%    +833.8%      28793 ± 11%  interrupts.CPU34.CAL:Function_call_interrupts
    487354           +52.4%     742690 ±  4%  interrupts.CPU34.LOC:Local_timer_interrupts
    576.17 ±  4%    +303.2%       2322 ± 10%  interrupts.CPU34.RES:Rescheduling_interrupts
      3135 ± 16%    +822.2%      28919 ± 12%  interrupts.CPU35.CAL:Function_call_interrupts
    487390           +52.4%     742785 ±  4%  interrupts.CPU35.LOC:Local_timer_interrupts
    579.67 ±  5%    +301.5%       2327 ± 11%  interrupts.CPU35.RES:Rescheduling_interrupts
      3075 ± 18%    +833.3%      28705 ± 14%  interrupts.CPU36.CAL:Function_call_interrupts
    486770           +52.6%     742982 ±  4%  interrupts.CPU36.LOC:Local_timer_interrupts
      1100 ± 10%    +150.2%       2752 ±  9%  interrupts.CPU36.RES:Rescheduling_interrupts
      2971 ± 20%    +869.5%      28806 ± 13%  interrupts.CPU37.CAL:Function_call_interrupts
    486730           +52.7%     743040 ±  4%  interrupts.CPU37.LOC:Local_timer_interrupts
    597.17 ±  9%    +289.5%       2326 ±  8%  interrupts.CPU37.RES:Rescheduling_interrupts
      3131 ± 19%    +822.8%      28895 ± 14%  interrupts.CPU38.CAL:Function_call_interrupts
    486743           +52.6%     742891 ±  4%  interrupts.CPU38.LOC:Local_timer_interrupts
    596.17 ± 14%    +287.0%       2307 ±  5%  interrupts.CPU38.RES:Rescheduling_interrupts
      3121 ± 17%    +833.1%      29125 ± 14%  interrupts.CPU39.CAL:Function_call_interrupts
    486833           +52.6%     742976 ±  4%  interrupts.CPU39.LOC:Local_timer_interrupts
    601.00 ± 11%    +279.5%       2280 ±  7%  interrupts.CPU39.RES:Rescheduling_interrupts
      2985 ± 22%    +851.0%      28388 ± 15%  interrupts.CPU4.CAL:Function_call_interrupts
    486986           +52.2%     741386 ±  4%  interrupts.CPU4.LOC:Local_timer_interrupts
    682.00 ± 13%    +257.7%       2439 ±  6%  interrupts.CPU4.RES:Rescheduling_interrupts
      3080 ± 19%    +834.0%      28774 ± 14%  interrupts.CPU40.CAL:Function_call_interrupts
    486717           +53.1%     745155 ±  4%  interrupts.CPU40.LOC:Local_timer_interrupts
    579.33 ± 12%    +296.8%       2298 ±  7%  interrupts.CPU40.RES:Rescheduling_interrupts
      3077 ± 20%    +837.3%      28849 ± 14%  interrupts.CPU41.CAL:Function_call_interrupts
    486774           +52.7%     743289 ±  4%  interrupts.CPU41.LOC:Local_timer_interrupts
    608.50 ± 18%    +269.9%       2250 ±  8%  interrupts.CPU41.RES:Rescheduling_interrupts
      3089 ± 19%    +833.6%      28846 ± 14%  interrupts.CPU42.CAL:Function_call_interrupts
    486799           +52.7%     743424 ±  4%  interrupts.CPU42.LOC:Local_timer_interrupts
    602.17 ± 12%    +281.9%       2299 ±  9%  interrupts.CPU42.RES:Rescheduling_interrupts
      3086 ± 16%    +839.6%      28999 ± 14%  interrupts.CPU43.CAL:Function_call_interrupts
    486768           +52.7%     743297 ±  4%  interrupts.CPU43.LOC:Local_timer_interrupts
    618.17 ± 17%    +267.7%       2272 ±  8%  interrupts.CPU43.RES:Rescheduling_interrupts
      3001 ± 21%    +864.7%      28951 ± 14%  interrupts.CPU44.CAL:Function_call_interrupts
    486814           +52.6%     743114 ±  4%  interrupts.CPU44.LOC:Local_timer_interrupts
    587.67 ±  9%    +287.1%       2274 ±  7%  interrupts.CPU44.RES:Rescheduling_interrupts
      3123 ± 19%    +827.1%      28955 ± 14%  interrupts.CPU45.CAL:Function_call_interrupts
    486764           +52.7%     743329 ±  4%  interrupts.CPU45.LOC:Local_timer_interrupts
    609.00 ± 12%    +278.8%       2307 ±  6%  interrupts.CPU45.RES:Rescheduling_interrupts
      3093 ± 16%    +834.7%      28916 ± 13%  interrupts.CPU46.CAL:Function_call_interrupts
    486788           +52.7%     743194 ±  4%  interrupts.CPU46.LOC:Local_timer_interrupts
    638.33 ± 19%    +256.8%       2277 ±  7%  interrupts.CPU46.RES:Rescheduling_interrupts
      3065 ± 18%    +847.8%      29052 ± 14%  interrupts.CPU47.CAL:Function_call_interrupts
    486799           +52.6%     743046 ±  4%  interrupts.CPU47.LOC:Local_timer_interrupts
    612.67 ± 15%    +266.2%       2243 ±  6%  interrupts.CPU47.RES:Rescheduling_interrupts
      3033 ± 20%    +852.5%      28893 ± 14%  interrupts.CPU48.CAL:Function_call_interrupts
    486791           +52.7%     743286 ±  4%  interrupts.CPU48.LOC:Local_timer_interrupts
    594.67 ±  9%    +274.6%       2227 ±  8%  interrupts.CPU48.RES:Rescheduling_interrupts
      3094 ± 20%    +831.8%      28836 ± 14%  interrupts.CPU49.CAL:Function_call_interrupts
    486791           +52.7%     743219 ±  4%  interrupts.CPU49.LOC:Local_timer_interrupts
    620.67 ± 14%    +263.2%       2254 ±  7%  interrupts.CPU49.RES:Rescheduling_interrupts
      2959 ± 18%    +860.9%      28442 ± 13%  interrupts.CPU5.CAL:Function_call_interrupts
    487094           +52.2%     741600 ±  4%  interrupts.CPU5.LOC:Local_timer_interrupts
    697.17 ±  8%    +246.5%       2416 ±  7%  interrupts.CPU5.RES:Rescheduling_interrupts
      3046 ± 20%    +847.0%      28851 ± 13%  interrupts.CPU50.CAL:Function_call_interrupts
    486780           +52.7%     743459 ±  4%  interrupts.CPU50.LOC:Local_timer_interrupts
    606.17 ± 10%    +269.2%       2238 ±  7%  interrupts.CPU50.RES:Rescheduling_interrupts
      3051 ± 20%    +850.8%      29018 ± 14%  interrupts.CPU51.CAL:Function_call_interrupts
    486794           +52.7%     743396 ±  4%  interrupts.CPU51.LOC:Local_timer_interrupts
    604.00 ±  9%    +271.9%       2246 ±  6%  interrupts.CPU51.RES:Rescheduling_interrupts
      3031 ± 22%    +844.3%      28622 ± 13%  interrupts.CPU52.CAL:Function_call_interrupts
    486730           +52.7%     743019 ±  4%  interrupts.CPU52.LOC:Local_timer_interrupts
    602.00 ±  9%    +269.8%       2226 ±  7%  interrupts.CPU52.RES:Rescheduling_interrupts
      3086 ± 19%    +836.8%      28911 ± 14%  interrupts.CPU53.CAL:Function_call_interrupts
    486796           +52.6%     743093 ±  4%  interrupts.CPU53.LOC:Local_timer_interrupts
    604.17 ± 11%    +278.2%       2284 ±  7%  interrupts.CPU53.RES:Rescheduling_interrupts
      3066 ± 18%    +820.5%      28228 ± 14%  interrupts.CPU54.CAL:Function_call_interrupts
    486721           +52.7%     743023 ±  4%  interrupts.CPU54.LOC:Local_timer_interrupts
      1017 ± 13%    +188.5%       2933 ± 14%  interrupts.CPU54.RES:Rescheduling_interrupts
      3068 ± 17%    +807.3%      27838 ± 16%  interrupts.CPU55.CAL:Function_call_interrupts
    486751           +52.6%     743015 ±  4%  interrupts.CPU55.LOC:Local_timer_interrupts
    627.33 ±  8%    +274.0%       2346 ± 11%  interrupts.CPU55.RES:Rescheduling_interrupts
      3139 ± 16%    +814.3%      28705 ± 12%  interrupts.CPU56.CAL:Function_call_interrupts
    486627           +52.7%     742990 ±  4%  interrupts.CPU56.LOC:Local_timer_interrupts
    605.00 ± 13%    +301.1%       2426 ±  8%  interrupts.CPU56.RES:Rescheduling_interrupts
      3092 ± 19%    +823.4%      28557 ± 13%  interrupts.CPU57.CAL:Function_call_interrupts
    486697           +52.7%     742981 ±  4%  interrupts.CPU57.LOC:Local_timer_interrupts
    601.83 ±  9%    +304.0%       2431 ±  9%  interrupts.CPU57.RES:Rescheduling_interrupts
      2869 ± 19%    +884.7%      28260 ± 14%  interrupts.CPU58.CAL:Function_call_interrupts
    486728           +52.7%     743011 ±  4%  interrupts.CPU58.LOC:Local_timer_interrupts
    589.00 ±  9%    +305.3%       2387 ±  8%  interrupts.CPU58.RES:Rescheduling_interrupts
      3087 ± 17%    +816.4%      28298 ± 14%  interrupts.CPU59.CAL:Function_call_interrupts
    486638           +52.7%     743122 ±  4%  interrupts.CPU59.LOC:Local_timer_interrupts
    606.83 ±  7%    +295.5%       2400 ± 10%  interrupts.CPU59.RES:Rescheduling_interrupts
      2950 ± 16%    +853.7%      28134 ± 13%  interrupts.CPU6.CAL:Function_call_interrupts
    487085           +52.2%     741518 ±  4%  interrupts.CPU6.LOC:Local_timer_interrupts
    688.33 ± 11%    +250.0%       2409 ±  5%  interrupts.CPU6.RES:Rescheduling_interrupts
      3046 ± 18%    +837.8%      28569 ± 14%  interrupts.CPU60.CAL:Function_call_interrupts
    486713           +52.7%     743209 ±  4%  interrupts.CPU60.LOC:Local_timer_interrupts
    575.83 ±  7%    +329.1%       2471 ±  9%  interrupts.CPU60.RES:Rescheduling_interrupts
      3099 ± 17%    +819.3%      28495 ± 13%  interrupts.CPU61.CAL:Function_call_interrupts
    486750           +52.6%     742985 ±  4%  interrupts.CPU61.LOC:Local_timer_interrupts
    590.83 ± 11%    +303.3%       2382 ±  9%  interrupts.CPU61.RES:Rescheduling_interrupts
      3091 ± 17%    +834.6%      28888 ± 13%  interrupts.CPU62.CAL:Function_call_interrupts
    486710           +52.6%     742904 ±  4%  interrupts.CPU62.LOC:Local_timer_interrupts
    607.17 ±  8%    +299.4%       2425 ±  9%  interrupts.CPU62.RES:Rescheduling_interrupts
      3082 ± 17%    +817.5%      28284 ± 15%  interrupts.CPU63.CAL:Function_call_interrupts
    486691           +52.7%     743057 ±  4%  interrupts.CPU63.LOC:Local_timer_interrupts
    585.17 ±  3%    +307.8%       2386 ± 12%  interrupts.CPU63.RES:Rescheduling_interrupts
      3104 ± 19%    +824.1%      28686 ± 13%  interrupts.CPU64.CAL:Function_call_interrupts
    486692           +52.7%     743020 ±  4%  interrupts.CPU64.LOC:Local_timer_interrupts
    586.00 ±  8%    +302.1%       2356 ± 10%  interrupts.CPU64.RES:Rescheduling_interrupts
      3134 ± 16%    +812.4%      28597 ± 13%  interrupts.CPU65.CAL:Function_call_interrupts
    486641           +52.7%     743055 ±  4%  interrupts.CPU65.LOC:Local_timer_interrupts
    616.33 ±  8%    +285.0%       2373 ± 10%  interrupts.CPU65.RES:Rescheduling_interrupts
      2841 ± 18%    +901.8%      28470 ± 14%  interrupts.CPU66.CAL:Function_call_interrupts
    486753           +52.6%     742991 ±  4%  interrupts.CPU66.LOC:Local_timer_interrupts
    596.67 ±  7%    +295.9%       2362 ±  9%  interrupts.CPU66.RES:Rescheduling_interrupts
      2992 ± 18%    +848.8%      28386 ± 13%  interrupts.CPU67.CAL:Function_call_interrupts
    486726           +52.6%     742794 ±  4%  interrupts.CPU67.LOC:Local_timer_interrupts
    587.83 ±  7%    +305.9%       2386 ±  8%  interrupts.CPU67.RES:Rescheduling_interrupts
      3105 ± 17%    +819.7%      28559 ± 14%  interrupts.CPU68.CAL:Function_call_interrupts
    486718           +52.7%     743179 ±  4%  interrupts.CPU68.LOC:Local_timer_interrupts
    604.17 ±  9%    +299.0%       2410 ± 10%  interrupts.CPU68.RES:Rescheduling_interrupts
      3073 ± 15%    +826.6%      28478 ± 13%  interrupts.CPU69.CAL:Function_call_interrupts
    486691           +52.7%     743035 ±  4%  interrupts.CPU69.LOC:Local_timer_interrupts
    601.67 ± 11%    +304.1%       2431 ±  8%  interrupts.CPU69.RES:Rescheduling_interrupts
      3019 ± 21%    +832.0%      28139 ± 13%  interrupts.CPU7.CAL:Function_call_interrupts
    486988           +52.3%     741486 ±  4%  interrupts.CPU7.LOC:Local_timer_interrupts
    729.83 ± 11%    +224.9%       2371 ±  4%  interrupts.CPU7.RES:Rescheduling_interrupts
      3058 ± 16%    +829.7%      28432 ± 13%  interrupts.CPU70.CAL:Function_call_interrupts
    486702           +52.6%     742802 ±  4%  interrupts.CPU70.LOC:Local_timer_interrupts
    583.83 ±  5%    +310.9%       2399 ± 10%  interrupts.CPU70.RES:Rescheduling_interrupts
      6390 ±  9%    +416.6%      33010 ± 11%  interrupts.CPU71.CAL:Function_call_interrupts
    486784           +52.6%     742926 ±  4%  interrupts.CPU71.LOC:Local_timer_interrupts
      1423 ±  9%    +157.9%       3670 ±  5%  interrupts.CPU71.RES:Rescheduling_interrupts
      3021 ± 17%    +830.6%      28117 ± 15%  interrupts.CPU72.CAL:Function_call_interrupts
    487036           +52.4%     742341 ±  4%  interrupts.CPU72.LOC:Local_timer_interrupts
    658.17 ± 12%    +258.0%       2356 ±  4%  interrupts.CPU72.RES:Rescheduling_interrupts
      3140 ± 19%    +808.0%      28519 ± 14%  interrupts.CPU73.CAL:Function_call_interrupts
    487006           +52.2%     741396 ±  4%  interrupts.CPU73.LOC:Local_timer_interrupts
    741.83 ± 12%    +223.5%       2400 ±  7%  interrupts.CPU73.RES:Rescheduling_interrupts
      3032 ± 16%    +834.2%      28332 ± 15%  interrupts.CPU74.CAL:Function_call_interrupts
    487026           +52.3%     741506 ±  4%  interrupts.CPU74.LOC:Local_timer_interrupts
    720.50 ± 15%    +233.9%       2405 ±  7%  interrupts.CPU74.RES:Rescheduling_interrupts
      3003 ± 16%    +846.3%      28425 ± 14%  interrupts.CPU75.CAL:Function_call_interrupts
    487028           +52.3%     741605 ±  4%  interrupts.CPU75.LOC:Local_timer_interrupts
    694.17 ± 15%    +245.4%       2397 ±  6%  interrupts.CPU75.RES:Rescheduling_interrupts
      2925 ± 20%    +858.7%      28050 ± 15%  interrupts.CPU76.CAL:Function_call_interrupts
    486998           +52.2%     741381 ±  4%  interrupts.CPU76.LOC:Local_timer_interrupts
    677.67 ± 12%    +251.1%       2379 ±  8%  interrupts.CPU76.RES:Rescheduling_interrupts
      3011 ± 22%    +847.9%      28545 ± 14%  interrupts.CPU77.CAL:Function_call_interrupts
    487138           +52.2%     741649 ±  4%  interrupts.CPU77.LOC:Local_timer_interrupts
    662.67 ± 12%    +257.7%       2370 ±  8%  interrupts.CPU77.RES:Rescheduling_interrupts
      3097 ± 20%    +824.4%      28630 ± 14%  interrupts.CPU78.CAL:Function_call_interrupts
    487107           +52.2%     741528 ±  4%  interrupts.CPU78.LOC:Local_timer_interrupts
    691.50 ± 11%    +248.5%       2409 ±  8%  interrupts.CPU78.RES:Rescheduling_interrupts
      3001 ± 17%    +849.2%      28494 ± 14%  interrupts.CPU79.CAL:Function_call_interrupts
    487080           +52.2%     741446 ±  4%  interrupts.CPU79.LOC:Local_timer_interrupts
    672.50 ± 12%    +262.3%       2436 ±  7%  interrupts.CPU79.RES:Rescheduling_interrupts
      3077 ± 16%    +807.1%      27914 ± 15%  interrupts.CPU8.CAL:Function_call_interrupts
    487014           +52.2%     741404 ±  4%  interrupts.CPU8.LOC:Local_timer_interrupts
    699.67 ± 14%    +240.1%       2379 ±  8%  interrupts.CPU8.RES:Rescheduling_interrupts
      3076 ± 15%    +829.5%      28592 ± 14%  interrupts.CPU80.CAL:Function_call_interrupts
    487016           +52.2%     741388 ±  4%  interrupts.CPU80.LOC:Local_timer_interrupts
    716.17 ± 21%    +238.4%       2423 ±  9%  interrupts.CPU80.RES:Rescheduling_interrupts
      2922 ± 17%    +880.8%      28658 ± 14%  interrupts.CPU81.CAL:Function_call_interrupts
    486965           +52.3%     741508 ±  4%  interrupts.CPU81.LOC:Local_timer_interrupts
    641.33 ± 17%    +282.6%       2453 ±  6%  interrupts.CPU81.RES:Rescheduling_interrupts
      2930 ± 18%    +869.8%      28416 ± 15%  interrupts.CPU82.CAL:Function_call_interrupts
    487018           +52.3%     741612 ±  4%  interrupts.CPU82.LOC:Local_timer_interrupts
    633.17 ± 15%    +279.7%       2404 ±  6%  interrupts.CPU82.RES:Rescheduling_interrupts
      3025 ± 17%    +844.9%      28592 ± 14%  interrupts.CPU83.CAL:Function_call_interrupts
    486935           +52.3%     741530 ±  4%  interrupts.CPU83.LOC:Local_timer_interrupts
    665.67 ± 15%    +258.8%       2388 ±  7%  interrupts.CPU83.RES:Rescheduling_interrupts
      3044 ± 18%    +833.9%      28435 ± 15%  interrupts.CPU84.CAL:Function_call_interrupts
    487201           +52.2%     741427 ±  4%  interrupts.CPU84.LOC:Local_timer_interrupts
    628.00 ± 12%    +279.8%       2385 ±  8%  interrupts.CPU84.RES:Rescheduling_interrupts
      2942 ± 16%    +863.4%      28347 ± 14%  interrupts.CPU85.CAL:Function_call_interrupts
    487042           +52.2%     741305 ±  5%  interrupts.CPU85.LOC:Local_timer_interrupts
    596.00 ± 16%    +303.1%       2402 ±  4%  interrupts.CPU85.RES:Rescheduling_interrupts
      3044 ± 17%    +837.6%      28542 ± 14%  interrupts.CPU86.CAL:Function_call_interrupts
    486993           +52.2%     741355 ±  4%  interrupts.CPU86.LOC:Local_timer_interrupts
    647.67 ± 19%    +272.2%       2410 ±  7%  interrupts.CPU86.RES:Rescheduling_interrupts
      2936 ± 19%    +865.2%      28338 ± 14%  interrupts.CPU87.CAL:Function_call_interrupts
    487005           +52.2%     741411 ±  4%  interrupts.CPU87.LOC:Local_timer_interrupts
    602.00 ± 14%    +290.0%       2347 ±  6%  interrupts.CPU87.RES:Rescheduling_interrupts
      2912 ± 16%    +881.2%      28575 ± 14%  interrupts.CPU88.CAL:Function_call_interrupts
    487115           +52.2%     741612 ±  4%  interrupts.CPU88.LOC:Local_timer_interrupts
    597.67 ± 16%    +299.4%       2386 ±  7%  interrupts.CPU88.RES:Rescheduling_interrupts
      2979 ± 18%    +855.5%      28473 ± 15%  interrupts.CPU89.CAL:Function_call_interrupts
    487078           +52.2%     741416 ±  4%  interrupts.CPU89.LOC:Local_timer_interrupts
    646.17 ± 16%    +271.6%       2401 ±  7%  interrupts.CPU89.RES:Rescheduling_interrupts
      3046 ± 14%    +823.8%      28139 ± 15%  interrupts.CPU9.CAL:Function_call_interrupts
    487114           +52.2%     741616 ±  4%  interrupts.CPU9.LOC:Local_timer_interrupts
    712.33 ± 18%    +239.6%       2419 ±  7%  interrupts.CPU9.RES:Rescheduling_interrupts
      3034 ± 16%    +845.5%      28687 ± 11%  interrupts.CPU90.CAL:Function_call_interrupts
    487326           +52.5%     742970 ±  4%  interrupts.CPU90.LOC:Local_timer_interrupts
    460.50 ± 10%    +397.9%       2292 ±  9%  interrupts.CPU90.RES:Rescheduling_interrupts
      3126 ± 18%    +818.9%      28729 ± 11%  interrupts.CPU91.CAL:Function_call_interrupts
    487370           +52.4%     742848 ±  4%  interrupts.CPU91.LOC:Local_timer_interrupts
    460.00 ±  9%    +391.1%       2259 ±  9%  interrupts.CPU91.RES:Rescheduling_interrupts
      3094 ± 19%    +829.8%      28767 ± 12%  interrupts.CPU92.CAL:Function_call_interrupts
    487351           +52.4%     742922 ±  4%  interrupts.CPU92.LOC:Local_timer_interrupts
    493.17 ±  8%    +367.0%       2303 ± 12%  interrupts.CPU92.RES:Rescheduling_interrupts
      3143 ± 16%    +807.0%      28515 ± 11%  interrupts.CPU93.CAL:Function_call_interrupts
    487320           +52.4%     742795 ±  4%  interrupts.CPU93.LOC:Local_timer_interrupts
    472.67 ±  6%    +381.7%       2276 ± 10%  interrupts.CPU93.RES:Rescheduling_interrupts
      3072 ± 17%    +828.5%      28531 ± 11%  interrupts.CPU94.CAL:Function_call_interrupts
    487359           +52.4%     742843 ±  4%  interrupts.CPU94.LOC:Local_timer_interrupts
    463.83 ±  8%    +387.7%       2262 ± 10%  interrupts.CPU94.RES:Rescheduling_interrupts
      3071 ± 19%    +839.8%      28860 ± 10%  interrupts.CPU95.CAL:Function_call_interrupts
    487324           +52.4%     742821 ±  4%  interrupts.CPU95.LOC:Local_timer_interrupts
    462.83 ± 10%    +395.6%       2293 ±  9%  interrupts.CPU95.RES:Rescheduling_interrupts
      3088 ± 18%    +833.7%      28835 ± 12%  interrupts.CPU96.CAL:Function_call_interrupts
    487363           +52.4%     742915 ±  4%  interrupts.CPU96.LOC:Local_timer_interrupts
    456.17 ±  3%    +397.0%       2267 ± 11%  interrupts.CPU96.RES:Rescheduling_interrupts
      3138 ± 16%    +817.0%      28784 ± 11%  interrupts.CPU97.CAL:Function_call_interrupts
    487336           +52.4%     742851 ±  4%  interrupts.CPU97.LOC:Local_timer_interrupts
    485.33 ±  7%    +375.3%       2306 ±  8%  interrupts.CPU97.RES:Rescheduling_interrupts
      3111 ± 19%    +825.8%      28806 ± 12%  interrupts.CPU98.CAL:Function_call_interrupts
    487340           +52.4%     742828 ±  4%  interrupts.CPU98.LOC:Local_timer_interrupts
    426.67 ±  4%    +422.8%       2230 ± 10%  interrupts.CPU98.RES:Rescheduling_interrupts
      3147 ± 14%    +817.4%      28876 ± 12%  interrupts.CPU99.CAL:Function_call_interrupts
    487380           +52.4%     742982 ±  4%  interrupts.CPU99.LOC:Local_timer_interrupts
    451.00 ±  3%    +410.4%       2302 ± 13%  interrupts.CPU99.RES:Rescheduling_interrupts
    617.83 ±  9%    +165.3%       1639        interrupts.IWI:IRQ_work_interrupts
  70124050           +52.5%  1.069e+08 ±  4%  interrupts.LOC:Local_timer_interrupts
      0.00       +1.4e+104%     144.00        interrupts.MCP:Machine_check_polls
     83786 ±  2%    +299.0%     334310 ±  7%  interrupts.RES:Rescheduling_interrupts
    207.17 ± 58%    +120.5%     456.83 ± 22%  interrupts.TLB:TLB_shootdowns


                                                                                
                                  aim7.jobs-per-min                             
                                                                                
  80000 +-------------------------------------------------------------------+   
        |                                                                   |   
  75000 |-++.+ .+   +. +.             +  +.++. +. +. +.+ .+ .+ .++. +. +.++.|   
        |.+   +  +.+  +  + .++.++.++.+ + :    +  +  +   +  +  +    +  +     |   
  70000 |-+               +             +                                   |   
        |                                                                   |   
  65000 |-+                                                                 |   
        |                                                                   |   
  60000 |-+                                                                 |   
        |                                                                   |   
  55000 |-+                                                                 |   
        |                                                                   |   
  50000 |-+                                                                 |   
        | OO OO OO OO OO OO OO  O O  O  OO OO O                             |   
  45000 +-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                       perf-sched.total_wait_time.average.ms                    
                                                                                
  200 +---------------------------------------------------------------------+   
  180 |-+   +.       .+   +       +.      +        +.+        +     +. +.   |   
      |.+ .+  + .++.+  :  :+ +.  +  ++.  + +.+ .+. :  +.+ .+ + + +. : +  + .|   
  160 |-++     +       +.+  +  ++      ++     +   +      +  +   +  +      + |   
  140 |-+                                                                   |   
      |                                                                     |   
  120 |-+                                                                   |   
  100 |-+                                                                   |   
   80 |-+                                                                   |   
      |                                                                     |   
   60 |-+                                                                   |   
   40 |-+                                                                   |   
      |                                                                     |   
   20 |-OO OO OO OO O OO OO OO OO O OO OO OO O                              |   
    0 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                       perf-sched.total_wait_and_delay.count.ms                 
                                                                                
  2.2e+06 +-----------------------------------------------------------------+   
    2e+06 |O+O  OO OO OO OO OO O O O  O  OO OO O                            |   
          |   O                 O   O  O                                    |   
  1.8e+06 |-+                                                               |   
  1.6e+06 |-+                                                               |   
  1.4e+06 |-+                                                               |   
  1.2e+06 |-+                                                               |   
          |                                                                 |   
    1e+06 |-+                                                               |   
   800000 |-+                                                               |   
   600000 |-+                                                               |   
   400000 |-+                                                               |   
          |                                                                 |   
   200000 |+.++.++.++.++.++.++.+++.++.++.++.++.++.++.+++.++.++.++.++.++.++.+|   
        0 +-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                    perf-sched.total_wait_and_delay.average.ms                  
                                                                                
  200 +---------------------------------------------------------------------+   
  180 |-+   +.       .+   +  +    +.      +        +.+        +     +. +.   |   
      |.+ .+  + .++.+  :  :+ :+  +  ++.  + +.+ .+. :  +.+ .+ + + +. : +  + .|   
  160 |-++     +       +.+  +  ++      ++     +   +      +  +   +  +      + |   
  140 |-+                                                                   |   
      |                                                                     |   
  120 |-+                                                                   |   
  100 |-+                                                                   |   
   80 |-+                                                                   |   
      |                                                                     |   
   60 |-+                                                                   |   
   40 |-+                                                                   |   
      |                                                                     |   
   20 |-OO OO OO OO O OO OO OO OO O OO OO OO O                              |   
    0 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                aim7.time.system_time                           
                                                                                
  55000 +-------------------------------------------------------------------+   
        |                      O   O  O                                     |   
        | OO OO O   O    OO OO  O O  O                                      |   
  50000 |-+      O O  OO                OO OO O                             |   
        |                                                                   |   
        |                                                                   |   
  45000 |-+                                                                 |   
        |                                                                   |   
  40000 |-+                                                                 |   
        |                                                                   |   
        |                                                                   |   
  35000 |-+               +.            +                                   |   
        |.+   +. +.+ .+ .+  ++.++.++.+ + :   .+ .+ .+   +. +. +. +.++.+ .++.|   
        |  +.+  +   +  +              +  +.++  +  +  +.+  +  +  +      +    |   
  30000 +-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                              aim7.time.elapsed_time                            
                                                                                
  400 +---------------------------------------------------------------------+   
      | OO OO O   O    O OO    O    OO                                      |   
  380 |-+      O O  O O     O   O      OO OO O                              |   
  360 |-+                                                                   |   
      |                                                                     |   
  340 |-+                                                                   |   
  320 |-+                                                                   |   
      |                                                                     |   
  300 |-+                                                                   |   
  280 |-+                                                                   |   
      |                                                                     |   
  260 |-+               .+            .+                                    |   
  240 |.+   +. +.+ .+. +  +.++.++.+.++  +.++.++.+.++.++.++.++.+.++.++.+ .++.|   
      |  +.+  +   +   +                                                +    |   
  220 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                            aim7.time.elapsed_time.max                          
                                                                                
  400 +---------------------------------------------------------------------+   
      | OO OO O   O    O OO    O    OO                                      |   
  380 |-+      O O  O O     O   O      OO OO O                              |   
  360 |-+                                                                   |   
      |                                                                     |   
  340 |-+                                                                   |   
  320 |-+                                                                   |   
      |                                                                     |   
  300 |-+                                                                   |   
  280 |-+                                                                   |   
      |                                                                     |   
  260 |-+               .+            .+                                    |   
  240 |.+   +. +.+ .+. +  +.++.++.+.++  +.++.++.+.++.++.++.++.+.++.++.+ .++.|   
      |  +.+  +   +   +                                                +    |   
  220 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                             aim7.time.minor_page_faults                        
                                                                                
  420000 +------------------------------------------------------------------+   
         |                                                                  |   
  400000 |-+   O     O   O     O    O                                       |   
  380000 |O+O     OO  O O   O O  OO  O O      O                             |   
         |   O             O              OO                                |   
  360000 |-+    O                            O                              |   
         |                              O                                   |   
  340000 |-+                                                                |   
         |                                                                  |   
  320000 |++                                                                |   
  300000 |:+       +. +   .+ .++.++. +. +  +    ++   .+   +.+       .++.  .+|   
         |  + .+   : + + +  +       +  + + :+  +  + +  :  :  :.+   +    ++  |   
  280000 |-+ +  :.+     +                 +  ++    +   +.+   +  +.+         |   
         |      +                                                           |   
  260000 +------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                         aim7.time.voluntary_context_switches                   
                                                                                
  1.4e+07 +-----------------------------------------------------------------+   
          |O     O  O     O     OO  O OO    O  O                            |   
  1.2e+07 |-+   O  O  O                                                     |   
          |   O              O           OO                                 |   
    1e+07 |-+                                                               |   
          |            O                     O                              |   
    8e+06 |-+                  O   O                                        |   
          |                                                                 |   
    6e+06 |-+                                                               |   
          |                                                                 |   
    4e+06 |-+                                                               |   
          |                                                                 |   
    2e+06 |-+                                                               |   
          |+.+ .+ .+      +.      .+   +.  .++.  .+ .+++.  .+ .+   +.    +.+|   
        0 +-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                        aim7.time.involuntary_context_switches                  
                                                                                
    7e+06 +-----------------------------------------------------------------+   
  6.5e+06 |-+O        O   O O  OO  OO          O                            |   
          |O  O OO OO    O       O    O  OO  O                              |   
    6e+06 |-+          O     O              O                               |   
  5.5e+06 |-+                                                               |   
    5e+06 |-+                                                               |   
  4.5e+06 |-+                                                               |   
          |                                                                 |   
    4e+06 |-+                                                               |   
  3.5e+06 |-+                                                               |   
    3e+06 |-+                                                               |   
  2.5e+06 |-+                                                               |   
          |+.     .+ .++.++.++.+++.++.+ .++.  .++.++.+  .++.++.++.++.++.  .+|   
    2e+06 |-+++.++  +                  +    ++        ++                ++  |   
  1.5e+06 +-----------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample

***************************************************************************************************
lkp-csl-2sp9: 88 threads 2 sockets Intel(R) Xeon(R) Gold 6238M CPU @ 2.10GHz with 128G memory
=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/load/md/rootfs/tbox_group/test/testcase/ucode:
  gcc-9/performance/4BRD_12G/f2fs/x86_64-rhel-8.3/3000/RAID1/debian-10.4-x86_64-20200603.cgz/lkp-csl-2sp9/disk_rw/aim7/0x5003006

commit: 
  3d697a4a6b ("f2fs: rework write preallocations")
  4fa18391ae ("f2fs: do not expose unwritten blocks to user by DIO")

3d697a4a6b7dab8f 4fa18391ae8cf8f1e498d54b46c 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
     90929 ±  2%     -53.2%      42552 ±  3%  aim7.jobs-per-min
    198.30 ±  2%    +113.7%     423.70 ±  3%  aim7.time.elapsed_time
    198.30 ±  2%    +113.7%     423.70 ±  3%  aim7.time.elapsed_time.max
   2150286 ±  3%    +185.9%    6146859 ± 13%  aim7.time.involuntary_context_switches
    228185           +25.7%     286784 ±  2%  aim7.time.minor_page_faults
     16152 ±  2%    +116.7%      35004 ±  2%  aim7.time.system_time
    874119          +943.2%    9119021 ±104%  aim7.time.voluntary_context_switches
  1.14e+09           +63.2%  1.861e+09 ± 38%  cpuidle..time
    235.35 ±  2%     +95.9%     461.05 ±  3%  uptime.boot
      0.00 ±  8%      -0.0        0.00 ± 21%  mpstat.cpu.all.iowait%
      0.38 ±  7%      -0.2        0.22 ± 21%  mpstat.cpu.all.usr%
      8418 ±  8%     -55.8%       3717 ±  8%  vmstat.io.bo
    256.40 ±  2%     +66.6%     427.20 ±  6%  vmstat.procs.r
      3.04 ± 72%      -2.2        0.80 ±134%  turbostat.C6%
  36502392 ±  6%    +118.9%   79908293 ±  7%  turbostat.IRQ
     57.23            -2.9%      55.58        turbostat.RAMWatt
      4482 ±  4%     +33.8%       5994 ± 18%  slabinfo.ext4_fc_dentry_update.active_objs
      4482 ±  4%     +33.8%       5994 ± 18%  slabinfo.ext4_fc_dentry_update.num_objs
      5381 ±  2%     +12.6%       6061 ±  9%  slabinfo.ext4_io_end.active_objs
      5381 ±  2%     +12.6%       6061 ±  9%  slabinfo.ext4_io_end.num_objs
    158482           +16.8%     185095        meminfo.AnonHugePages
   1953315           -16.4%    1633057        meminfo.Dirty
   2356102           -13.1%    2048324        meminfo.Inactive
   1947087           -16.4%    1628551        meminfo.Inactive(file)
    110665 ±  5%      +3.5%     114560 ±  5%  meminfo.PageTables
      2971 ± 15%   +4198.0%     127712 ±191%  numa-meminfo.node0.Active(anon)
    977407           -16.4%     816829        numa-meminfo.node0.Dirty
   1282403 ±  2%     -12.6%    1120784        numa-meminfo.node0.Inactive
    974889           -16.4%     815389        numa-meminfo.node0.Inactive(file)
    109623 ±  5%      +3.6%     113568 ±  5%  numa-meminfo.node0.PageTables
    978964           -16.8%     814786        numa-meminfo.node1.Dirty
   1077260           -14.0%     926684 ±  2%  numa-meminfo.node1.Inactive
    975291           -16.8%     811876        numa-meminfo.node1.Inactive(file)
    770.40 ± 12%   +4039.6%      31891 ±191%  numa-vmstat.node0.nr_active_anon
    244188           -16.4%     204212        numa-vmstat.node0.nr_dirty
    243567           -16.3%     203829        numa-vmstat.node0.nr_inactive_file
     27386 ±  4%      +3.6%      28381 ±  5%  numa-vmstat.node0.nr_page_table_pages
    770.40 ± 12%   +4039.6%      31891 ±191%  numa-vmstat.node0.nr_zone_active_anon
    243564           -16.3%     203831        numa-vmstat.node0.nr_zone_inactive_file
    244226           -16.4%     204240        numa-vmstat.node0.nr_zone_write_pending
    244559           -16.7%     203834        numa-vmstat.node1.nr_dirty
    243622           -16.6%     203115        numa-vmstat.node1.nr_inactive_file
    243621           -16.6%     203117        numa-vmstat.node1.nr_zone_inactive_file
    244596           -16.7%     203855        numa-vmstat.node1.nr_zone_write_pending
    488730           -16.4%     408353        proc-vmstat.nr_dirty
    487181           -16.4%     407214        proc-vmstat.nr_inactive_file
     27654 ±  5%      +3.6%      28653 ±  5%  proc-vmstat.nr_page_table_pages
    487181           -16.4%     407214        proc-vmstat.nr_zone_inactive_file
    488798           -16.4%     408400        proc-vmstat.nr_zone_write_pending
     78832 ±  3%    +143.0%     191563 ± 40%  proc-vmstat.numa_hint_faults
     49727 ±  4%    +137.0%     117845 ± 36%  proc-vmstat.numa_hint_faults_local
    220945 ±  6%     +46.9%     324636 ± 25%  proc-vmstat.numa_pte_updates
    143107          +116.1%     309219 ± 64%  proc-vmstat.pgactivate
    997125           +64.2%    1636904 ±  7%  proc-vmstat.pgfault
     51933 ±  3%     +92.8%     100129 ±  2%  proc-vmstat.pgreuse
      1734 ±  2%      +7.5%       1865        proc-vmstat.unevictable_pgs_culled
 6.804e+09           -15.9%   5.72e+09        perf-stat.i.branch-instructions
  24882928 ±  2%     -33.1%   16658305 ±  7%  perf-stat.i.branch-misses
  66850630 ±  2%     -31.6%   45743150 ±  5%  perf-stat.i.cache-misses
 1.701e+08 ±  2%     -31.2%   1.17e+08 ±  8%  perf-stat.i.cache-references
      7.54           +25.9%       9.49        perf-stat.i.cpi
      3472 ±  4%     +50.2%       5213 ±  4%  perf-stat.i.cycles-between-cache-misses
 8.008e+09           -20.6%  6.357e+09        perf-stat.i.dTLB-loads
 2.261e+09 ±  2%     -46.7%  1.205e+09 ±  3%  perf-stat.i.dTLB-stores
     92.94            -7.0       85.98 ±  6%  perf-stat.i.iTLB-load-miss-rate%
  13564259 ±  3%     -46.2%    7302051 ±  2%  perf-stat.i.iTLB-load-misses
 2.975e+10           -18.6%  2.422e+10        perf-stat.i.instructions
      2241 ±  3%     +48.9%       3337 ±  2%  perf-stat.i.instructions-per-iTLB-miss
      0.15 ±  2%     -22.4%       0.11 ±  2%  perf-stat.i.ipc
    310.75 ±  7%     -22.8%     240.04 ± 12%  perf-stat.i.metric.K/sec
    195.92           -22.3%     152.23        perf-stat.i.metric.M/sec
      4409           -19.5%       3547 ±  4%  perf-stat.i.minor-faults
  10313917 ±  2%     -18.9%    8365261 ± 13%  perf-stat.i.node-load-misses
     75.30            +4.5       79.79        perf-stat.i.node-store-miss-rate%
   9204873           -24.4%    6954712 ±  3%  perf-stat.i.node-store-misses
   2871224 ±  3%     -40.2%    1716899 ±  3%  perf-stat.i.node-stores
      4416           -19.6%       3550 ±  4%  perf-stat.i.page-faults
      5.72 ±  2%     -15.5%       4.83 ±  8%  perf-stat.overall.MPKI
      0.37            -0.1        0.29 ±  8%  perf-stat.overall.branch-miss-rate%
      7.72           +24.7%       9.62        perf-stat.overall.cpi
      3436 ±  2%     +48.9%       5115 ±  6%  perf-stat.overall.cycles-between-cache-misses
      0.01 ± 31%      +0.0        0.01 ± 31%  perf-stat.overall.dTLB-store-miss-rate%
     95.11           -11.1       84.03 ± 10%  perf-stat.overall.iTLB-load-miss-rate%
      2196 ±  3%     +51.2%       3320 ±  3%  perf-stat.overall.instructions-per-iTLB-miss
      0.13           -19.8%       0.10        perf-stat.overall.ipc
     76.23            +4.0       80.19        perf-stat.overall.node-store-miss-rate%
 6.778e+09           -15.8%  5.708e+09        perf-stat.ps.branch-instructions
  24742062 ±  2%     -32.8%   16614599 ±  7%  perf-stat.ps.branch-misses
  66579716 ±  2%     -31.4%   45648425 ±  5%  perf-stat.ps.cache-misses
 1.694e+08 ±  2%     -31.1%  1.167e+08 ±  8%  perf-stat.ps.cache-references
 7.978e+09           -20.5%  6.343e+09        perf-stat.ps.dTLB-loads
 2.252e+09 ±  2%     -46.6%  1.203e+09 ±  3%  perf-stat.ps.dTLB-stores
  13509109 ±  3%     -46.1%    7286635 ±  2%  perf-stat.ps.iTLB-load-misses
 2.963e+10           -18.4%  2.417e+10        perf-stat.ps.instructions
      4361           -19.0%       3533 ±  4%  perf-stat.ps.minor-faults
  10270443 ±  2%     -18.7%    8348155 ± 13%  perf-stat.ps.node-load-misses
   9169754           -24.3%    6940506 ±  3%  perf-stat.ps.node-store-misses
   2861078 ±  3%     -40.1%    1713680 ±  3%  perf-stat.ps.node-stores
      4368           -19.0%       3536 ±  4%  perf-stat.ps.page-faults
 5.904e+12           +73.9%  1.026e+13 ±  2%  perf-stat.total.instructions
     95.90           -52.9       43.01        perf-profile.calltrace.cycles-pp.__generic_file_write_iter.f2fs_file_write_iter.new_sync_write.vfs_write.ksys_write
     95.77           -52.8       42.94        perf-profile.calltrace.cycles-pp.generic_perform_write.__generic_file_write_iter.f2fs_file_write_iter.new_sync_write.vfs_write
     61.32           -33.6       27.71        perf-profile.calltrace.cycles-pp.f2fs_write_end.generic_perform_write.__generic_file_write_iter.f2fs_file_write_iter.new_sync_write
     60.09           -32.9       27.18        perf-profile.calltrace.cycles-pp.f2fs_mark_inode_dirty_sync.f2fs_write_end.generic_perform_write.__generic_file_write_iter.f2fs_file_write_iter
     60.09           -32.9       27.17        perf-profile.calltrace.cycles-pp.f2fs_inode_dirtied.f2fs_mark_inode_dirty_sync.f2fs_write_end.generic_perform_write.__generic_file_write_iter
     60.07           -32.9       27.16        perf-profile.calltrace.cycles-pp._raw_spin_lock.f2fs_inode_dirtied.f2fs_mark_inode_dirty_sync.f2fs_write_end.generic_perform_write
     59.70           -32.7       27.01        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.f2fs_inode_dirtied.f2fs_mark_inode_dirty_sync.f2fs_write_end
     33.71           -18.9       14.84        perf-profile.calltrace.cycles-pp.f2fs_write_begin.generic_perform_write.__generic_file_write_iter.f2fs_file_write_iter.new_sync_write
     30.55           -17.0       13.55        perf-profile.calltrace.cycles-pp.f2fs_get_block.f2fs_write_begin.generic_perform_write.__generic_file_write_iter.f2fs_file_write_iter
     30.44           -16.9       13.51        perf-profile.calltrace.cycles-pp.f2fs_reserve_block.f2fs_get_block.f2fs_write_begin.generic_perform_write.__generic_file_write_iter
     30.42           -16.9       13.51        perf-profile.calltrace.cycles-pp.f2fs_reserve_new_blocks.f2fs_reserve_block.f2fs_get_block.f2fs_write_begin.generic_perform_write
     15.20            -8.5        6.74        perf-profile.calltrace.cycles-pp.__mark_inode_dirty.f2fs_reserve_new_blocks.f2fs_reserve_block.f2fs_get_block.f2fs_write_begin
     15.09            -8.4        6.69        perf-profile.calltrace.cycles-pp._raw_spin_lock.f2fs_inode_dirtied.__mark_inode_dirty.f2fs_reserve_new_blocks.f2fs_reserve_block
     15.09            -8.4        6.69        perf-profile.calltrace.cycles-pp.f2fs_inode_dirtied.__mark_inode_dirty.f2fs_reserve_new_blocks.f2fs_reserve_block.f2fs_get_block
     15.00            -8.3        6.66        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.f2fs_inode_dirtied.__mark_inode_dirty.f2fs_reserve_new_blocks
     14.96            -8.3        6.68        perf-profile.calltrace.cycles-pp.f2fs_mark_inode_dirty_sync.f2fs_reserve_new_blocks.f2fs_reserve_block.f2fs_get_block.f2fs_write_begin
     14.96            -8.3        6.67        perf-profile.calltrace.cycles-pp._raw_spin_lock.f2fs_inode_dirtied.f2fs_mark_inode_dirty_sync.f2fs_reserve_new_blocks.f2fs_reserve_block
     14.96            -8.3        6.68        perf-profile.calltrace.cycles-pp.f2fs_inode_dirtied.f2fs_mark_inode_dirty_sync.f2fs_reserve_new_blocks.f2fs_reserve_block.f2fs_get_block
     14.92            -8.3        6.66        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.f2fs_inode_dirtied.f2fs_mark_inode_dirty_sync.f2fs_reserve_new_blocks
     97.87            +1.3       99.22        perf-profile.calltrace.cycles-pp.write
     97.65            +1.5       99.11        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
     97.62            +1.5       99.09        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     97.54            +1.5       99.04        perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     97.45            +1.5       99.00        perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     97.26            +1.7       98.91        perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     97.20            +1.7       98.88        perf-profile.calltrace.cycles-pp.f2fs_file_write_iter.new_sync_write.vfs_write.ksys_write.do_syscall_64
      0.00           +54.8       54.77        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.f2fs_inode_dirtied.f2fs_mark_inode_dirty_sync.f2fs_file_write_iter
      0.00           +55.0       55.03        perf-profile.calltrace.cycles-pp._raw_spin_lock.f2fs_inode_dirtied.f2fs_mark_inode_dirty_sync.f2fs_file_write_iter.new_sync_write
      0.00           +55.1       55.11        perf-profile.calltrace.cycles-pp.f2fs_inode_dirtied.f2fs_mark_inode_dirty_sync.f2fs_file_write_iter.new_sync_write.vfs_write
      0.00           +55.1       55.12        perf-profile.calltrace.cycles-pp.f2fs_mark_inode_dirty_sync.f2fs_file_write_iter.new_sync_write.vfs_write.ksys_write
     95.91           -52.9       43.02        perf-profile.children.cycles-pp.__generic_file_write_iter
     95.79           -52.8       42.96        perf-profile.children.cycles-pp.generic_perform_write
     61.32           -33.6       27.71        perf-profile.children.cycles-pp.f2fs_write_end
     33.71           -18.9       14.84        perf-profile.children.cycles-pp.f2fs_write_begin
     30.55           -17.0       13.55        perf-profile.children.cycles-pp.f2fs_get_block
     30.55           -17.0       13.57        perf-profile.children.cycles-pp.f2fs_reserve_block
     30.54           -17.0       13.56        perf-profile.children.cycles-pp.f2fs_reserve_new_blocks
     15.40            -8.5        6.86        perf-profile.children.cycles-pp.__mark_inode_dirty
      0.98 ±  7%      -0.6        0.33 ±  6%  perf-profile.children.cycles-pp.f2fs_lookup_extent_cache
      0.60 ±  4%      -0.5        0.07 ± 17%  perf-profile.children.cycles-pp.rwsem_down_write_slowpath
      0.82 ±  4%      -0.5        0.32 ±  2%  perf-profile.children.cycles-pp.__get_node_page
      0.62 ±  2%      -0.5        0.17 ±  6%  perf-profile.children.cycles-pp.do_sys_open
      0.62 ±  2%      -0.5        0.17 ±  6%  perf-profile.children.cycles-pp.do_sys_openat2
      0.62 ±  2%      -0.5        0.17 ±  6%  perf-profile.children.cycles-pp.do_filp_open
      0.62 ±  2%      -0.5        0.17 ±  6%  perf-profile.children.cycles-pp.path_openat
      0.61 ±  3%      -0.4        0.16 ±  4%  perf-profile.children.cycles-pp.creat64
      0.85 ±  3%      -0.4        0.43 ±  3%  perf-profile.children.cycles-pp.pagecache_get_page
      0.83 ±  3%      -0.4        0.42 ±  3%  perf-profile.children.cycles-pp.__filemap_get_folio
      0.65 ±  2%      -0.4        0.27 ±  5%  perf-profile.children.cycles-pp.folio_unlock
      0.67 ±  2%      -0.4        0.31 ±  3%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
      0.62 ±  2%      -0.4        0.27 ±  4%  perf-profile.children.cycles-pp.dput
      0.48 ±  4%      -0.4        0.12 ±  3%  perf-profile.children.cycles-pp.unlink
      0.48 ±  4%      -0.4        0.12 ±  3%  perf-profile.children.cycles-pp.__x64_sys_unlink
      0.48 ±  4%      -0.4        0.12 ±  3%  perf-profile.children.cycles-pp.do_unlinkat
      0.62 ±  2%      -0.4        0.27 ±  4%  perf-profile.children.cycles-pp.__close
      0.62 ±  2%      -0.4        0.27 ±  4%  perf-profile.children.cycles-pp.task_work_run
      0.62 ±  2%      -0.4        0.27 ±  4%  perf-profile.children.cycles-pp.__fput
      0.62 ±  2%      -0.4        0.27 ±  3%  perf-profile.children.cycles-pp.__dentry_kill
      0.65 ±  3%      -0.3        0.30 ±  3%  perf-profile.children.cycles-pp.exit_to_user_mode_prepare
      0.61 ±  2%      -0.3        0.27 ±  3%  perf-profile.children.cycles-pp.evict
      0.61 ±  2%      -0.3        0.27 ±  3%  perf-profile.children.cycles-pp.f2fs_evict_inode
      0.53 ±  3%      -0.3        0.23 ±  2%  perf-profile.children.cycles-pp.folio_mark_dirty
      0.46            -0.3        0.18 ±  4%  perf-profile.children.cycles-pp.f2fs_set_data_page_dirty
      0.28 ±  2%      -0.2        0.03 ± 81%  perf-profile.children.cycles-pp.rwsem_spin_on_owner
      0.33 ±  2%      -0.2        0.12 ±  4%  perf-profile.children.cycles-pp.f2fs_update_dirty_page
      0.46 ±  3%      -0.2        0.26 ±  2%  perf-profile.children.cycles-pp.copy_page_from_iter_atomic
      0.32 ±  3%      -0.2        0.13 ±  5%  perf-profile.children.cycles-pp.truncate_inode_pages_range
      0.33            -0.2        0.15 ±  3%  perf-profile.children.cycles-pp.f2fs_convert_inline_inode
      0.24 ±  5%      -0.2        0.06 ±  6%  perf-profile.children.cycles-pp.read_node_page
      0.32            -0.2        0.15        perf-profile.children.cycles-pp.f2fs_convert_inline_page
      0.31 ± 10%      -0.2        0.14 ± 12%  perf-profile.children.cycles-pp._raw_read_lock
      0.35 ±  2%      -0.2        0.20 ±  3%  perf-profile.children.cycles-pp.copyin
      0.35 ±  2%      -0.1        0.20 ±  3%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      0.26 ±  2%      -0.1        0.12 ±  3%  perf-profile.children.cycles-pp.llseek
      0.26 ±  2%      -0.1        0.12 ±  4%  perf-profile.children.cycles-pp.f2fs_create
      0.25 ±  3%      -0.1        0.11 ±  3%  perf-profile.children.cycles-pp.f2fs_do_add_link
      0.25 ±  3%      -0.1        0.11 ±  3%  perf-profile.children.cycles-pp.f2fs_add_dentry
      0.25 ±  3%      -0.1        0.11 ±  3%  perf-profile.children.cycles-pp.f2fs_add_regular_entry
      0.21 ±  3%      -0.1        0.09 ±  4%  perf-profile.children.cycles-pp.f2fs_truncate
      0.19 ±  4%      -0.1        0.09 ±  4%  perf-profile.children.cycles-pp.vfs_unlink
      0.19 ±  2%      -0.1        0.09        perf-profile.children.cycles-pp.fault_in_iov_iter_readable
      0.19 ±  3%      -0.1        0.09 ±  4%  perf-profile.children.cycles-pp.f2fs_unlink
      0.16 ±  6%      -0.1        0.05 ±  9%  perf-profile.children.cycles-pp.truncate_cleanup_page
      0.18 ±  4%      -0.1        0.08 ±  4%  perf-profile.children.cycles-pp.filemap_add_folio
      0.18 ±  4%      -0.1        0.08        perf-profile.children.cycles-pp.f2fs_update_parent_metadata
      0.17 ±  2%      -0.1        0.08 ±  4%  perf-profile.children.cycles-pp.f2fs_inode_synced
      0.16 ±  3%      -0.1        0.07 ±  7%  perf-profile.children.cycles-pp.f2fs_do_truncate_blocks
      0.15 ±  5%      -0.1        0.07 ±  7%  perf-profile.children.cycles-pp.f2fs_truncate_data_blocks_range
      0.16 ±  2%      -0.1        0.08 ±  5%  perf-profile.children.cycles-pp.fault_in_readable
      0.18 ±  3%      -0.1        0.09 ±  8%  perf-profile.children.cycles-pp.xas_load
      0.14 ±  3%      -0.1        0.06        perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.14 ±  3%      -0.1        0.06        perf-profile.children.cycles-pp.__entry_text_start
      0.13 ±  3%      -0.1        0.06 ±  6%  perf-profile.children.cycles-pp.__get_user_nocheck_1
      0.12 ±  8%      -0.1        0.05        perf-profile.children.cycles-pp.__might_resched
      0.12 ±  3%      -0.1        0.06 ±  8%  perf-profile.children.cycles-pp.__filemap_add_folio
      0.11 ±  3%      -0.1        0.05        perf-profile.children.cycles-pp.folio_alloc
      0.12 ±  3%      -0.1        0.06        perf-profile.children.cycles-pp.f2fs_update_inode
      0.09 ±  5%      -0.1        0.03 ± 81%  perf-profile.children.cycles-pp.__fdget_pos
      0.10            -0.1        0.05        perf-profile.children.cycles-pp.security_file_permission
      0.10 ±  5%      -0.0        0.05        perf-profile.children.cycles-pp.__alloc_pages
     99.52            +0.2       99.76        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     99.47            +0.3       99.73        perf-profile.children.cycles-pp.do_syscall_64
     97.94            +1.3       99.25        perf-profile.children.cycles-pp.write
     97.55            +1.5       99.05        perf-profile.children.cycles-pp.ksys_write
     97.47            +1.5       99.01        perf-profile.children.cycles-pp.vfs_write
     97.27            +1.7       98.92        perf-profile.children.cycles-pp.new_sync_write
     97.21            +1.7       98.89        perf-profile.children.cycles-pp.f2fs_file_write_iter
     91.52            +4.7       96.21        perf-profile.children.cycles-pp._raw_spin_lock
     90.82            +4.9       95.68        perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
     91.08            +5.0       96.11        perf-profile.children.cycles-pp.f2fs_inode_dirtied
     75.81           +13.5       89.31        perf-profile.children.cycles-pp.f2fs_mark_inode_dirty_sync
      0.66 ±  6%      -0.5        0.19 ±  3%  perf-profile.self.cycles-pp.f2fs_lookup_extent_cache
      0.65 ±  2%      -0.4        0.27 ±  5%  perf-profile.self.cycles-pp.folio_unlock
      0.48 ±  4%      -0.3        0.20 ±  3%  perf-profile.self.cycles-pp.f2fs_write_begin
      0.68 ±  6%      -0.2        0.47        perf-profile.self.cycles-pp.f2fs_file_write_iter
      0.32 ±  5%      -0.2        0.14 ±  5%  perf-profile.self.cycles-pp.__get_node_page
      0.70            -0.2        0.53 ±  3%  perf-profile.self.cycles-pp._raw_spin_lock
      0.30 ± 11%      -0.2        0.14 ± 11%  perf-profile.self.cycles-pp._raw_read_lock
      0.35 ±  2%      -0.1        0.20 ±  3%  perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      0.29 ±  3%      -0.1        0.16 ±  6%  perf-profile.self.cycles-pp.__filemap_get_folio
      0.17 ±  2%      -0.1        0.06 ± 10%  perf-profile.self.cycles-pp.f2fs_update_dirty_page
      0.15 ±  5%      -0.1        0.07 ± 12%  perf-profile.self.cycles-pp.f2fs_write_end
      0.14 ±  6%      -0.1        0.06        perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.14 ±  3%      -0.1        0.07 ±  9%  perf-profile.self.cycles-pp.xas_load
      0.13 ±  3%      -0.1        0.06        perf-profile.self.cycles-pp.__get_user_nocheck_1
      0.11 ±  7%      -0.1        0.05        perf-profile.self.cycles-pp.__might_resched
      0.10 ±  4%      -0.1        0.04 ± 50%  perf-profile.self.cycles-pp.write
      0.11 ±  7%      -0.1        0.05        perf-profile.self.cycles-pp.copy_page_from_iter_atomic
      0.00            +0.1        0.08 ±  7%  perf-profile.self.cycles-pp.f2fs_inode_dirtied
     90.34            +4.9       95.21        perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
     16070 ±  7%    +142.5%      38965 ± 13%  softirqs.CPU0.RCU
     11696 ±  7%    +112.8%      24896 ± 51%  softirqs.CPU0.SCHED
     15170 ±  7%    +155.3%      38726 ± 12%  softirqs.CPU1.RCU
      7900 ±  4%    +167.5%      21130 ± 58%  softirqs.CPU1.SCHED
     14327 ±  7%    +174.8%      39372 ± 11%  softirqs.CPU10.RCU
      6520 ±  7%    +212.5%      20379 ± 61%  softirqs.CPU10.SCHED
     13839 ±  3%    +176.7%      38295 ± 12%  softirqs.CPU11.RCU
      6543 ±  8%    +204.6%      19932 ± 64%  softirqs.CPU11.SCHED
     13977 ±  4%    +175.1%      38456 ±  9%  softirqs.CPU12.RCU
      6912 ±  4%    +194.9%      20387 ± 59%  softirqs.CPU12.SCHED
     14092 ±  5%    +170.9%      38181 ± 12%  softirqs.CPU13.RCU
      6888 ±  4%    +192.4%      20144 ± 61%  softirqs.CPU13.SCHED
     14325 ±  5%    +173.4%      39165 ± 13%  softirqs.CPU14.RCU
      6510 ±  5%    +213.4%      20402 ± 63%  softirqs.CPU14.SCHED
     15663 ± 15%    +163.9%      41339 ± 11%  softirqs.CPU15.RCU
      6997 ±  8%    +185.7%      19990 ± 63%  softirqs.CPU15.SCHED
     14391 ±  7%    +190.4%      41788 ± 11%  softirqs.CPU16.RCU
      6446 ±  6%    +213.1%      20183 ± 61%  softirqs.CPU16.SCHED
     15202 ± 11%    +173.9%      41641 ± 11%  softirqs.CPU17.RCU
      7060 ± 12%    +164.8%      18694 ± 58%  softirqs.CPU17.SCHED
     15071 ±  8%    +175.1%      41462 ± 11%  softirqs.CPU18.RCU
      6624 ±  4%    +210.5%      20567 ± 62%  softirqs.CPU18.SCHED
     14443 ±  5%    +192.7%      42275 ± 10%  softirqs.CPU19.RCU
      6410          +215.4%      20217 ± 61%  softirqs.CPU19.SCHED
     15142 ±  8%    +158.4%      39135 ± 11%  softirqs.CPU2.RCU
      7840 ±  2%    +175.1%      21565 ± 58%  softirqs.CPU2.SCHED
     15396 ± 13%    +171.5%      41798 ± 11%  softirqs.CPU20.RCU
      6552 ±  3%    +206.7%      20099 ± 63%  softirqs.CPU20.SCHED
     14915 ±  5%    +179.7%      41716 ± 10%  softirqs.CPU21.RCU
      6546 ±  3%    +202.5%      19804 ± 61%  softirqs.CPU21.SCHED
     14300 ±  5%    +180.0%      40039 ± 11%  softirqs.CPU22.RCU
      6519          +211.5%      20310 ± 61%  softirqs.CPU22.SCHED
     14439 ±  4%    +176.6%      39935 ± 11%  softirqs.CPU23.RCU
      6749 ±  9%    +194.5%      19878 ± 63%  softirqs.CPU23.SCHED
     14086 ±  4%    +180.7%      39546 ± 10%  softirqs.CPU24.RCU
      6388          +214.4%      20089 ± 62%  softirqs.CPU24.SCHED
     14159 ±  5%    +178.1%      39373 ± 11%  softirqs.CPU25.RCU
      6470          +209.0%      19994 ± 63%  softirqs.CPU25.SCHED
     14261 ±  3%    +178.2%      39680 ± 11%  softirqs.CPU26.RCU
      6562 ±  2%    +210.6%      20380 ± 61%  softirqs.CPU26.SCHED
     14147 ±  4%    +181.8%      39865 ± 10%  softirqs.CPU27.RCU
      6405          +215.6%      20216 ± 62%  softirqs.CPU27.SCHED
     14153 ±  3%    +181.1%      39787 ± 12%  softirqs.CPU28.RCU
      6580 ±  4%    +203.9%      19996 ± 63%  softirqs.CPU28.SCHED
     14191 ±  6%    +179.4%      39648 ± 11%  softirqs.CPU29.RCU
      6431          +213.0%      20131 ± 63%  softirqs.CPU29.SCHED
     14550 ±  7%    +169.4%      39198 ± 12%  softirqs.CPU3.RCU
      7132 ±  8%    +191.3%      20773 ± 61%  softirqs.CPU3.SCHED
     14235 ±  4%    +182.9%      40272 ± 11%  softirqs.CPU30.RCU
      6378          +213.2%      19979 ± 63%  softirqs.CPU30.SCHED
     14434 ±  4%    +173.0%      39407 ±  8%  softirqs.CPU31.RCU
      6429          +210.8%      19986 ± 63%  softirqs.CPU31.SCHED
     14471 ±  5%    +179.6%      40463 ± 11%  softirqs.CPU32.RCU
      6386          +212.0%      19929 ± 63%  softirqs.CPU32.SCHED
     14262 ±  4%    +185.2%      40677 ± 10%  softirqs.CPU33.RCU
      6391          +220.1%      20460 ± 57%  softirqs.CPU33.SCHED
     14194 ±  4%    +184.8%      40423 ± 10%  softirqs.CPU34.RCU
      6332 ±  2%    +227.1%      20712 ± 62%  softirqs.CPU34.SCHED
     14306 ±  4%    +180.8%      40169 ± 11%  softirqs.CPU35.RCU
      6391          +217.2%      20273 ± 63%  softirqs.CPU35.SCHED
     14541 ±  5%    +174.3%      39883 ± 11%  softirqs.CPU36.RCU
      6595 ±  5%    +204.5%      20083 ± 62%  softirqs.CPU36.SCHED
     14453 ±  5%    +179.1%      40334 ± 11%  softirqs.CPU37.RCU
      6363          +213.3%      19936 ± 63%  softirqs.CPU37.SCHED
     14375 ±  4%    +181.8%      40509 ± 10%  softirqs.CPU38.RCU
      6328          +220.5%      20284 ± 61%  softirqs.CPU38.SCHED
     14357 ±  7%    +171.7%      39008 ± 12%  softirqs.CPU39.RCU
      6438 ±  5%    +207.3%      19787 ± 63%  softirqs.CPU39.SCHED
     14691 ±  5%    +163.3%      38678 ± 11%  softirqs.CPU4.RCU
      6922 ±  6%    +197.7%      20610 ± 60%  softirqs.CPU4.SCHED
     14464 ±  5%    +175.3%      39815 ± 11%  softirqs.CPU40.RCU
      6462 ±  4%    +209.0%      19970 ± 63%  softirqs.CPU40.SCHED
     14198 ±  5%    +180.7%      39861 ± 11%  softirqs.CPU41.RCU
      6204 ±  2%    +219.8%      19840 ± 63%  softirqs.CPU41.SCHED
     14391 ±  5%    +178.4%      40063 ± 10%  softirqs.CPU42.RCU
      6340 ±  4%    +216.7%      20078 ± 63%  softirqs.CPU42.SCHED
     14408 ±  5%    +179.8%      40314 ± 10%  softirqs.CPU43.RCU
      6238 ±  6%    +219.1%      19908 ± 63%  softirqs.CPU43.SCHED
     11472 ±  4%    +176.1%      31673 ±  9%  softirqs.CPU44.RCU
      6632 ± 11%    +208.7%      20476 ± 59%  softirqs.CPU44.SCHED
     13094 ±  4%    +184.5%      37257 ± 15%  softirqs.CPU45.RCU
      6868 ±  9%    +192.1%      20062 ± 64%  softirqs.CPU45.SCHED
     14034 ±  4%    +175.5%      38667 ±  9%  softirqs.CPU46.RCU
      6149 ±  6%    +227.8%      20155 ± 63%  softirqs.CPU46.SCHED
     14116 ±  7%    +170.0%      38111 ± 12%  softirqs.CPU47.RCU
      6489 ±  3%    +206.9%      19915 ± 65%  softirqs.CPU47.SCHED
     14489 ±  5%    +167.5%      38753 ± 12%  softirqs.CPU48.RCU
      6972 ± 12%    +191.9%      20355 ± 64%  softirqs.CPU48.SCHED
     14832 ±  8%    +166.0%      39448 ± 11%  softirqs.CPU49.RCU
      6842 ±  5%    +199.6%      20499 ± 64%  softirqs.CPU49.SCHED
     14218 ±  6%    +175.1%      39116 ± 10%  softirqs.CPU5.RCU
      7016 ±  8%    +189.9%      20337 ± 60%  softirqs.CPU5.SCHED
     14533 ±  6%    +160.9%      37916 ± 13%  softirqs.CPU50.RCU
      6278 ±  2%    +223.5%      20312 ± 63%  softirqs.CPU50.SCHED
     14426 ±  4%    +166.5%      38442 ± 12%  softirqs.CPU51.RCU
      6720          +203.2%      20375 ± 63%  softirqs.CPU51.SCHED
     15761 ± 17%    +147.9%      39064 ± 14%  softirqs.CPU52.RCU
      6518 ±  2%    +210.9%      20265 ± 63%  softirqs.CPU52.SCHED
     14310 ±  4%    +172.4%      38986 ± 11%  softirqs.CPU53.RCU
      6566 ±  2%    +210.0%      20356 ± 62%  softirqs.CPU53.SCHED
     14328 ±  4%    +172.6%      39058 ± 12%  softirqs.CPU54.RCU
      6688 ±  2%    +204.4%      20360 ± 63%  softirqs.CPU54.SCHED
     14659 ±  3%    +167.0%      39146 ± 12%  softirqs.CPU55.RCU
      6616          +208.7%      20426 ± 64%  softirqs.CPU55.SCHED
     14198 ±  4%    +174.9%      39037 ± 12%  softirqs.CPU56.RCU
      6233 ±  7%    +228.6%      20484 ± 64%  softirqs.CPU56.SCHED
     14251 ±  4%    +170.4%      38541 ± 11%  softirqs.CPU57.RCU
      6678          +208.1%      20577 ± 60%  softirqs.CPU57.SCHED
     14153 ±  5%    +163.5%      37296 ± 14%  softirqs.CPU58.RCU
      6641          +203.1%      20129 ± 63%  softirqs.CPU58.SCHED
     14565 ±  4%    +168.5%      39112 ± 11%  softirqs.CPU59.RCU
      6515 ±  5%    +216.0%      20590 ± 63%  softirqs.CPU59.SCHED
     14546 ±  4%    +163.9%      38382 ± 10%  softirqs.CPU6.RCU
      7020 ±  5%    +189.7%      20333 ± 61%  softirqs.CPU6.SCHED
     14367 ±  5%    +185.7%      41044 ± 12%  softirqs.CPU60.RCU
      6673 ±  2%    +209.5%      20652 ± 62%  softirqs.CPU60.SCHED
     14467 ±  7%    +186.0%      41373 ± 10%  softirqs.CPU61.RCU
      6428 ±  8%    +211.1%      19995 ± 61%  softirqs.CPU61.SCHED
     14646 ±  6%    +182.2%      41332 ± 11%  softirqs.CPU62.RCU
      6728 ±  2%    +201.0%      20255 ± 61%  softirqs.CPU62.SCHED
     14798 ±  8%    +177.4%      41048 ± 11%  softirqs.CPU63.RCU
      6695 ±  3%    +206.0%      20488 ± 62%  softirqs.CPU63.SCHED
     14997 ±  8%    +175.0%      41236 ± 12%  softirqs.CPU64.RCU
      6747 ±  4%    +200.9%      20307 ± 62%  softirqs.CPU64.SCHED
     14849 ±  8%    +176.8%      41096 ± 10%  softirqs.CPU65.RCU
      7213 ± 13%    +182.2%      20353 ± 61%  softirqs.CPU65.SCHED
     14197 ±  6%    +186.6%      40691 ± 11%  softirqs.CPU66.RCU
      6424          +214.6%      20212 ± 62%  softirqs.CPU66.SCHED
     14416 ±  4%    +178.8%      40186 ± 11%  softirqs.CPU67.RCU
      6834 ± 16%    +190.3%      19836 ± 64%  softirqs.CPU67.SCHED
     14052 ±  5%    +186.2%      40216 ± 10%  softirqs.CPU68.RCU
      6434          +212.5%      20106 ± 63%  softirqs.CPU68.SCHED
     14142 ±  5%    +184.0%      40169 ± 11%  softirqs.CPU69.RCU
      6261 ±  4%    +222.1%      20171 ± 64%  softirqs.CPU69.SCHED
     14347 ±  5%    +173.8%      39277 ± 13%  softirqs.CPU7.RCU
      6802 ±  5%    +197.2%      20218 ± 63%  softirqs.CPU7.SCHED
     14211 ±  4%    +181.7%      40035 ± 11%  softirqs.CPU70.RCU
      6337          +216.9%      20081 ± 63%  softirqs.CPU70.SCHED
     14131 ±  5%    +182.5%      39917 ± 12%  softirqs.CPU71.RCU
      6444 ±  2%    +208.6%      19885 ± 63%  softirqs.CPU71.SCHED
     14058 ±  4%    +186.9%      40334 ± 11%  softirqs.CPU72.RCU
      6160 ±  4%    +224.3%      19978 ± 64%  softirqs.CPU72.SCHED
     14369 ±  5%    +179.5%      40164 ± 11%  softirqs.CPU73.RCU
      6372 ±  3%    +215.1%      20078 ± 64%  softirqs.CPU73.SCHED
     14151 ±  4%    +182.3%      39946 ± 11%  softirqs.CPU74.RCU
      6388          +213.2%      20006 ± 64%  softirqs.CPU74.SCHED
     13433 ±  4%    +170.6%      36347 ± 10%  softirqs.CPU75.RCU
      6345          +212.9%      19852 ± 64%  softirqs.CPU75.SCHED
     13655 ±  4%    +167.3%      36501 ± 11%  softirqs.CPU76.RCU
      6414          +211.9%      20008 ± 63%  softirqs.CPU76.SCHED
     13500 ±  4%    +175.0%      37126 ± 11%  softirqs.CPU77.RCU
      6478 ±  2%    +208.4%      19976 ± 64%  softirqs.CPU77.SCHED
     13530 ±  3%    +170.9%      36652 ± 11%  softirqs.CPU78.RCU
      6350 ±  2%    +215.2%      20016 ± 64%  softirqs.CPU78.SCHED
     13783 ±  6%    +166.3%      36708 ± 11%  softirqs.CPU79.RCU
      6359          +215.5%      20063 ± 64%  softirqs.CPU79.SCHED
     14775 ±  4%    +162.1%      38732 ± 14%  softirqs.CPU8.RCU
      6879 ±  8%    +203.2%      20861 ± 62%  softirqs.CPU8.SCHED
     13722 ±  2%    +165.6%      36443 ± 11%  softirqs.CPU80.RCU
      6914 ±  8%    +189.4%      20011 ± 64%  softirqs.CPU80.SCHED
     13773 ±  5%    +166.1%      36647 ± 12%  softirqs.CPU81.RCU
      6894 ±  7%    +186.0%      19719 ± 65%  softirqs.CPU81.SCHED
     13420 ±  4%    +169.6%      36180 ± 11%  softirqs.CPU82.RCU
      6363          +213.2%      19930 ± 64%  softirqs.CPU82.SCHED
     13462 ±  4%    +172.6%      36691 ± 11%  softirqs.CPU83.RCU
      6345          +210.1%      19678 ± 66%  softirqs.CPU83.SCHED
     13822 ±  6%    +166.0%      36765 ± 11%  softirqs.CPU84.RCU
      6436 ±  2%    +209.4%      19916 ± 64%  softirqs.CPU84.SCHED
     13587 ±  6%    +168.9%      36536 ± 11%  softirqs.CPU85.RCU
      6646 ±  8%    +200.6%      19976 ± 64%  softirqs.CPU85.SCHED
     13790 ±  7%    +164.4%      36454 ± 12%  softirqs.CPU86.RCU
      6366          +218.5%      20275 ± 65%  softirqs.CPU86.SCHED
     14165 ±  4%    +159.3%      36733 ± 11%  softirqs.CPU87.RCU
      6303 ±  5%    +216.3%      19933 ± 63%  softirqs.CPU87.SCHED
     15184 ±  9%    +152.6%      38350 ± 12%  softirqs.CPU9.RCU
      6635 ±  7%    +205.1%      20245 ± 62%  softirqs.CPU9.SCHED
   1259928 ±  4%    +173.7%    3448251 ± 11%  softirqs.RCU
    585046          +204.4%    1781134 ± 62%  softirqs.SCHED
     43092 ±  3%     +92.6%      82983 ±  3%  softirqs.TIMER
    615536          +423.4%    3221540 ± 91%  interrupts.CAL:Function_call_interrupts
      7642 ± 10%    +383.6%      36955 ± 92%  interrupts.CPU0.CAL:Function_call_interrupts
    387376 ±  7%    +119.2%     849204 ±  3%  interrupts.CPU0.LOC:Local_timer_interrupts
      3620 ± 20%    +111.8%       7669 ± 49%  interrupts.CPU0.RES:Rescheduling_interrupts
      6424 ± 11%    +463.5%      36199 ± 92%  interrupts.CPU1.CAL:Function_call_interrupts
    380497 ± 11%    +123.2%     849354 ±  3%  interrupts.CPU1.LOC:Local_timer_interrupts
      3178 ± 17%    +126.2%       7188 ± 45%  interrupts.CPU1.RES:Rescheduling_interrupts
      7335 ± 16%    +407.3%      37212 ± 91%  interrupts.CPU10.CAL:Function_call_interrupts
    388720 ±  7%    +118.5%     849232 ±  3%  interrupts.CPU10.LOC:Local_timer_interrupts
      6838 ± 11%    +439.1%      36865 ± 94%  interrupts.CPU11.CAL:Function_call_interrupts
    389069 ±  7%    +118.2%     849037 ±  3%  interrupts.CPU11.LOC:Local_timer_interrupts
      3410 ± 21%    +114.6%       7318 ± 52%  interrupts.CPU11.RES:Rescheduling_interrupts
      7341 ± 16%    +398.5%      36598 ± 93%  interrupts.CPU12.CAL:Function_call_interrupts
    388730 ±  7%    +118.4%     849137 ±  3%  interrupts.CPU12.LOC:Local_timer_interrupts
      6642 ± 16%    +457.8%      37048 ± 92%  interrupts.CPU13.CAL:Function_call_interrupts
    388874 ±  7%    +118.4%     849167 ±  3%  interrupts.CPU13.LOC:Local_timer_interrupts
      3519 ± 28%    +123.5%       7864 ± 50%  interrupts.CPU13.RES:Rescheduling_interrupts
      6402 ±  7%    +494.1%      38031 ± 91%  interrupts.CPU14.CAL:Function_call_interrupts
    389043 ±  7%    +118.3%     849216 ±  3%  interrupts.CPU14.LOC:Local_timer_interrupts
      3192 ± 16%    +155.5%       8156 ± 44%  interrupts.CPU14.RES:Rescheduling_interrupts
      7059 ± 12%    +419.5%      36669 ± 93%  interrupts.CPU15.CAL:Function_call_interrupts
    388754 ±  7%    +118.4%     849016 ±  3%  interrupts.CPU15.LOC:Local_timer_interrupts
      3472 ± 15%    +108.9%       7255 ± 50%  interrupts.CPU15.RES:Rescheduling_interrupts
      6818 ±  8%    +456.7%      37956 ± 94%  interrupts.CPU16.CAL:Function_call_interrupts
    388739 ±  7%    +118.5%     849227 ±  3%  interrupts.CPU16.LOC:Local_timer_interrupts
      3848 ± 12%    +116.5%       8331 ± 56%  interrupts.CPU16.RES:Rescheduling_interrupts
      6371 ±  5%    +405.0%      32178 ± 92%  interrupts.CPU17.CAL:Function_call_interrupts
    388878 ±  7%    +118.4%     849333 ±  3%  interrupts.CPU17.LOC:Local_timer_interrupts
      3170 ± 14%    +108.6%       6615 ± 43%  interrupts.CPU17.RES:Rescheduling_interrupts
      7237 ± 11%    +406.2%      36636 ± 94%  interrupts.CPU18.CAL:Function_call_interrupts
    388818 ±  7%    +118.4%     849168 ±  3%  interrupts.CPU18.LOC:Local_timer_interrupts
      6703 ± 12%    +447.8%      36722 ± 95%  interrupts.CPU19.CAL:Function_call_interrupts
    388737 ±  7%    +118.5%     849230 ±  3%  interrupts.CPU19.LOC:Local_timer_interrupts
      7211 ± 13%    +413.6%      37042 ± 93%  interrupts.CPU2.CAL:Function_call_interrupts
    388716 ±  7%    +118.5%     849295 ±  3%  interrupts.CPU2.LOC:Local_timer_interrupts
      3359 ± 15%    +112.9%       7153 ± 51%  interrupts.CPU2.RES:Rescheduling_interrupts
      6591 ±  7%    +461.9%      37036 ± 92%  interrupts.CPU20.CAL:Function_call_interrupts
    388559 ±  7%    +118.5%     849091 ±  3%  interrupts.CPU20.LOC:Local_timer_interrupts
      3421 ± 12%    +128.5%       7817 ± 47%  interrupts.CPU20.RES:Rescheduling_interrupts
      6817 ±  8%    +435.5%      36507 ± 95%  interrupts.CPU21.CAL:Function_call_interrupts
    388842 ±  7%    +118.4%     849238 ±  3%  interrupts.CPU21.LOC:Local_timer_interrupts
      7216 ± 11%    +406.6%      36559 ± 89%  interrupts.CPU22.CAL:Function_call_interrupts
    390366 ±  6%    +117.5%     849170 ±  3%  interrupts.CPU22.LOC:Local_timer_interrupts
      3918 ± 20%    +118.7%       8569 ± 53%  interrupts.CPU22.RES:Rescheduling_interrupts
      7698 ±  7%    +365.6%      35846 ± 88%  interrupts.CPU23.CAL:Function_call_interrupts
    390440 ±  6%    +117.5%     849167 ±  3%  interrupts.CPU23.LOC:Local_timer_interrupts
      4061 ± 12%     +84.1%       7476 ± 52%  interrupts.CPU23.RES:Rescheduling_interrupts
      7153 ±  7%    +404.3%      36078 ± 88%  interrupts.CPU24.CAL:Function_call_interrupts
    390475 ±  6%    +117.5%     849162 ±  3%  interrupts.CPU24.LOC:Local_timer_interrupts
      3495 ± 15%    +117.8%       7612 ± 49%  interrupts.CPU24.RES:Rescheduling_interrupts
      7266 ±  7%    +389.2%      35552 ± 92%  interrupts.CPU25.CAL:Function_call_interrupts
    390503 ±  6%    +117.5%     849300 ±  3%  interrupts.CPU25.LOC:Local_timer_interrupts
      6754 ± 13%    +440.9%      36531 ± 89%  interrupts.CPU26.CAL:Function_call_interrupts
    390568 ±  6%    +117.4%     849144 ±  3%  interrupts.CPU26.LOC:Local_timer_interrupts
      3125 ± 26%    +147.2%       7727 ± 58%  interrupts.CPU26.RES:Rescheduling_interrupts
      6823 ± 12%    +433.5%      36404 ± 90%  interrupts.CPU27.CAL:Function_call_interrupts
    390444 ±  6%    +117.5%     849273 ±  3%  interrupts.CPU27.LOC:Local_timer_interrupts
      6731 ±  7%    +439.2%      36294 ± 89%  interrupts.CPU28.CAL:Function_call_interrupts
    390455 ±  6%    +117.5%     849149 ±  3%  interrupts.CPU28.LOC:Local_timer_interrupts
      6430 ± 11%    +476.0%      37036 ± 88%  interrupts.CPU29.CAL:Function_call_interrupts
    390516 ±  6%    +117.4%     849097 ±  3%  interrupts.CPU29.LOC:Local_timer_interrupts
      3095 ± 17%    +159.4%       8029 ± 51%  interrupts.CPU29.RES:Rescheduling_interrupts
      6434 ±  5%    +471.7%      36786 ± 93%  interrupts.CPU3.CAL:Function_call_interrupts
    388744 ±  7%    +118.5%     849247 ±  3%  interrupts.CPU3.LOC:Local_timer_interrupts
      3230 ±  7%    +125.6%       7286 ± 50%  interrupts.CPU3.RES:Rescheduling_interrupts
      7285 ± 10%    +396.8%      36198 ± 91%  interrupts.CPU30.CAL:Function_call_interrupts
    390345 ±  6%    +117.5%     849103 ±  3%  interrupts.CPU30.LOC:Local_timer_interrupts
      7124 ± 10%    +418.7%      36952 ± 87%  interrupts.CPU31.CAL:Function_call_interrupts
    390540 ±  6%    +117.4%     849133 ±  3%  interrupts.CPU31.LOC:Local_timer_interrupts
      3535 ± 20%    +124.4%       7935 ± 49%  interrupts.CPU31.RES:Rescheduling_interrupts
      7139 ±  8%    +406.7%      36171 ± 90%  interrupts.CPU32.CAL:Function_call_interrupts
    390387 ±  6%    +117.5%     849102 ±  3%  interrupts.CPU32.LOC:Local_timer_interrupts
      7225 ± 11%    +395.6%      35812 ± 90%  interrupts.CPU33.CAL:Function_call_interrupts
    390604 ±  6%    +117.4%     849209 ±  3%  interrupts.CPU33.LOC:Local_timer_interrupts
      7198 ±  3%    +400.2%      36009 ± 88%  interrupts.CPU34.CAL:Function_call_interrupts
    390498 ±  6%    +117.5%     849142 ±  3%  interrupts.CPU34.LOC:Local_timer_interrupts
      3652 ±  9%    +112.9%       7775 ± 53%  interrupts.CPU34.RES:Rescheduling_interrupts
      6965 ±  6%    +413.1%      35736 ± 90%  interrupts.CPU35.CAL:Function_call_interrupts
    390625 ±  6%    +117.4%     849189 ±  3%  interrupts.CPU35.LOC:Local_timer_interrupts
      7451 ± 17%    +385.4%      36166 ± 89%  interrupts.CPU36.CAL:Function_call_interrupts
    390622 ±  6%    +117.4%     849106 ±  3%  interrupts.CPU36.LOC:Local_timer_interrupts
      6702 ±  8%    +445.2%      36541 ± 87%  interrupts.CPU37.CAL:Function_call_interrupts
    390478 ±  6%    +117.5%     849116 ±  3%  interrupts.CPU37.LOC:Local_timer_interrupts
      3040 ± 15%    +165.0%       8058 ± 48%  interrupts.CPU37.RES:Rescheduling_interrupts
      7555 ±  6%    +375.5%      35924 ± 90%  interrupts.CPU38.CAL:Function_call_interrupts
    390510 ±  6%    +117.5%     849202 ±  3%  interrupts.CPU38.LOC:Local_timer_interrupts
      7351 ± 11%    +387.0%      35804 ± 92%  interrupts.CPU39.CAL:Function_call_interrupts
    390552 ±  6%    +117.4%     849179 ±  3%  interrupts.CPU39.LOC:Local_timer_interrupts
      7614 ± 12%    +384.1%      36865 ± 92%  interrupts.CPU4.CAL:Function_call_interrupts
    389005 ±  7%    +118.2%     848944 ±  3%  interrupts.CPU4.LOC:Local_timer_interrupts
      7111 ± 12%    +416.6%      36737 ± 88%  interrupts.CPU40.CAL:Function_call_interrupts
    390368 ±  6%    +117.5%     849130 ±  3%  interrupts.CPU40.LOC:Local_timer_interrupts
      3551 ± 25%    +122.9%       7915 ± 51%  interrupts.CPU40.RES:Rescheduling_interrupts
      7215 ±  9%    +396.0%      35791 ± 90%  interrupts.CPU41.CAL:Function_call_interrupts
    390468 ±  6%    +117.5%     849083 ±  3%  interrupts.CPU41.LOC:Local_timer_interrupts
      7134 ±  6%    +403.5%      35918 ± 90%  interrupts.CPU42.CAL:Function_call_interrupts
    390266 ±  6%    +117.6%     849122 ±  3%  interrupts.CPU42.LOC:Local_timer_interrupts
      8083 ±  4%    +376.7%      38528 ± 86%  interrupts.CPU43.CAL:Function_call_interrupts
    390606 ±  6%    +117.4%     849146 ±  3%  interrupts.CPU43.LOC:Local_timer_interrupts
      4303 ± 11%    +121.2%       9519 ± 55%  interrupts.CPU43.RES:Rescheduling_interrupts
      6523 ±  3%    +458.3%      36418 ± 94%  interrupts.CPU44.CAL:Function_call_interrupts
    389317 ±  6%    +118.2%     849345 ±  3%  interrupts.CPU44.LOC:Local_timer_interrupts
      6358 ± 12%    +471.0%      36306 ± 94%  interrupts.CPU45.CAL:Function_call_interrupts
    389508 ±  6%    +118.0%     849135 ±  3%  interrupts.CPU45.LOC:Local_timer_interrupts
      3130 ± 24%    +126.8%       7099 ± 50%  interrupts.CPU45.RES:Rescheduling_interrupts
      6516 ±  7%    +468.8%      37063 ± 94%  interrupts.CPU46.CAL:Function_call_interrupts
    388793 ±  7%    +118.4%     849154 ±  3%  interrupts.CPU46.LOC:Local_timer_interrupts
      3426 ± 15%    +113.0%       7296 ± 49%  interrupts.CPU46.RES:Rescheduling_interrupts
      6581 ±  6%    +458.0%      36723 ± 94%  interrupts.CPU47.CAL:Function_call_interrupts
    388699 ±  7%    +118.5%     849204 ±  3%  interrupts.CPU47.LOC:Local_timer_interrupts
      3433 ± 14%    +108.0%       7143 ± 49%  interrupts.CPU47.RES:Rescheduling_interrupts
      7456 ± 13%    +408.6%      37926 ± 93%  interrupts.CPU48.CAL:Function_call_interrupts
    388895 ±  7%    +118.3%     849032 ±  3%  interrupts.CPU48.LOC:Local_timer_interrupts
      6408 ± 11%    +481.7%      37273 ± 92%  interrupts.CPU49.CAL:Function_call_interrupts
    388926 ±  7%    +118.4%     849291 ±  3%  interrupts.CPU49.LOC:Local_timer_interrupts
      3200 ± 21%    +133.1%       7460 ± 43%  interrupts.CPU49.RES:Rescheduling_interrupts
      6725 ±  8%    +451.1%      37061 ± 93%  interrupts.CPU5.CAL:Function_call_interrupts
    388665 ±  7%    +118.5%     849243 ±  3%  interrupts.CPU5.LOC:Local_timer_interrupts
      3178 ± 12%    +137.8%       7558 ± 52%  interrupts.CPU5.RES:Rescheduling_interrupts
      6486 ±  6%    +476.0%      37363 ± 93%  interrupts.CPU50.CAL:Function_call_interrupts
    388802 ±  7%    +118.4%     849225 ±  3%  interrupts.CPU50.LOC:Local_timer_interrupts
      3410 ± 15%    +120.9%       7533 ± 45%  interrupts.CPU50.RES:Rescheduling_interrupts
      6693 ±  8%    +452.6%      36982 ± 94%  interrupts.CPU51.CAL:Function_call_interrupts
    388939 ±  7%    +118.3%     849245 ±  3%  interrupts.CPU51.LOC:Local_timer_interrupts
    388889 ±  7%    +118.4%     849245 ±  3%  interrupts.CPU52.LOC:Local_timer_interrupts
      6706 ±  8%    +461.8%      37676 ± 94%  interrupts.CPU53.CAL:Function_call_interrupts
    388872 ±  7%    +118.4%     849194 ±  3%  interrupts.CPU53.LOC:Local_timer_interrupts
      3536 ± 15%    +118.0%       7709 ± 53%  interrupts.CPU53.RES:Rescheduling_interrupts
      6881 ±  7%    +440.7%      37208 ± 92%  interrupts.CPU54.CAL:Function_call_interrupts
    388838 ±  7%    +118.4%     849273 ±  3%  interrupts.CPU54.LOC:Local_timer_interrupts
      3688 ± 14%    +101.3%       7424 ± 42%  interrupts.CPU54.RES:Rescheduling_interrupts
      6766 ±  6%    +439.3%      36492 ± 94%  interrupts.CPU55.CAL:Function_call_interrupts
    388854 ±  7%    +118.4%     849434 ±  3%  interrupts.CPU55.LOC:Local_timer_interrupts
      3552 ± 13%     +96.2%       6968 ± 49%  interrupts.CPU55.RES:Rescheduling_interrupts
      7038 ± 16%    +430.0%      37300 ± 93%  interrupts.CPU56.CAL:Function_call_interrupts
    388685 ±  7%    +118.5%     849330 ±  3%  interrupts.CPU56.LOC:Local_timer_interrupts
      6216 ±  7%    +485.1%      36371 ± 94%  interrupts.CPU57.CAL:Function_call_interrupts
    388824 ±  7%    +118.5%     849427 ±  3%  interrupts.CPU57.LOC:Local_timer_interrupts
      3073 ± 12%    +123.7%       6875 ± 49%  interrupts.CPU57.RES:Rescheduling_interrupts
      6798 ±  9%    +453.6%      37640 ± 94%  interrupts.CPU58.CAL:Function_call_interrupts
    388740 ±  7%    +118.5%     849206 ±  3%  interrupts.CPU58.LOC:Local_timer_interrupts
      6742 ± 12%    +465.8%      38147 ± 93%  interrupts.CPU59.CAL:Function_call_interrupts
    388837 ±  7%    +118.4%     849245 ±  3%  interrupts.CPU59.LOC:Local_timer_interrupts
      3702 ± 24%    +115.4%       7976 ± 46%  interrupts.CPU59.RES:Rescheduling_interrupts
      7071 ±  5%    +415.7%      36466 ± 93%  interrupts.CPU6.CAL:Function_call_interrupts
    388599 ±  7%    +118.6%     849384 ±  3%  interrupts.CPU6.LOC:Local_timer_interrupts
      3756 ± 13%     +88.9%       7094 ± 46%  interrupts.CPU6.RES:Rescheduling_interrupts
      6960 ± 12%    +429.0%      36826 ± 95%  interrupts.CPU60.CAL:Function_call_interrupts
    388631 ±  7%    +118.5%     849268 ±  3%  interrupts.CPU60.LOC:Local_timer_interrupts
      6342 ±  3%    +478.8%      36709 ± 92%  interrupts.CPU61.CAL:Function_call_interrupts
    388764 ±  7%    +118.5%     849308 ±  3%  interrupts.CPU61.LOC:Local_timer_interrupts
      3132 ±  8%    +128.2%       7147 ± 39%  interrupts.CPU61.RES:Rescheduling_interrupts
      7166 ±  8%    +415.7%      36960 ± 93%  interrupts.CPU62.CAL:Function_call_interrupts
    388714 ±  7%    +118.4%     849100 ±  3%  interrupts.CPU62.LOC:Local_timer_interrupts
      6830 ±  7%    +431.5%      36306 ± 93%  interrupts.CPU63.CAL:Function_call_interrupts
    388727 ±  7%    +118.5%     849278 ±  3%  interrupts.CPU63.LOC:Local_timer_interrupts
      3606 ± 11%     +97.2%       7110 ± 47%  interrupts.CPU63.RES:Rescheduling_interrupts
      6157 ±  4%    +504.8%      37241 ± 94%  interrupts.CPU64.CAL:Function_call_interrupts
    388562 ±  7%    +118.6%     849206 ±  3%  interrupts.CPU64.LOC:Local_timer_interrupts
      3021 ±  6%    +159.5%       7841 ± 55%  interrupts.CPU64.RES:Rescheduling_interrupts
    388542 ±  7%    +118.6%     849230 ±  3%  interrupts.CPU65.LOC:Local_timer_interrupts
      7508 ± 14%    +380.9%      36113 ± 90%  interrupts.CPU66.CAL:Function_call_interrupts
    390568 ±  6%    +117.4%     849172 ±  3%  interrupts.CPU66.LOC:Local_timer_interrupts
      7378 ± 10%    +392.7%      36354 ± 90%  interrupts.CPU67.CAL:Function_call_interrupts
    390659 ±  6%    +117.4%     849167 ±  3%  interrupts.CPU67.LOC:Local_timer_interrupts
      6935 ±  7%    +427.9%      36612 ± 87%  interrupts.CPU68.CAL:Function_call_interrupts
    390734 ±  6%    +117.3%     849231 ±  3%  interrupts.CPU68.LOC:Local_timer_interrupts
      3319 ± 16%    +135.0%       7801 ± 44%  interrupts.CPU68.RES:Rescheduling_interrupts
      7400 ± 11%    +405.5%      37410 ± 88%  interrupts.CPU69.CAL:Function_call_interrupts
    390590 ±  6%    +117.4%     849186 ±  3%  interrupts.CPU69.LOC:Local_timer_interrupts
      3730 ± 19%    +109.6%       7820 ± 53%  interrupts.CPU69.RES:Rescheduling_interrupts
      6899 ± 15%    +438.8%      37177 ± 92%  interrupts.CPU7.CAL:Function_call_interrupts
    388810 ±  7%    +118.4%     849273 ±  3%  interrupts.CPU7.LOC:Local_timer_interrupts
      3700 ± 28%    +113.8%       7909 ± 50%  interrupts.CPU7.RES:Rescheduling_interrupts
      7103 ±  2%    +409.1%      36160 ± 91%  interrupts.CPU70.CAL:Function_call_interrupts
    390675 ±  6%    +117.4%     849167 ±  3%  interrupts.CPU70.LOC:Local_timer_interrupts
      7491 ±  7%    +377.7%      35780 ± 90%  interrupts.CPU71.CAL:Function_call_interrupts
    390620 ±  6%    +117.4%     849196 ±  3%  interrupts.CPU71.LOC:Local_timer_interrupts
      7438 ± 11%    +392.3%      36619 ± 89%  interrupts.CPU72.CAL:Function_call_interrupts
    390519 ±  6%    +117.5%     849200 ±  3%  interrupts.CPU72.LOC:Local_timer_interrupts
      6536 ±  8%    +469.0%      37197 ± 88%  interrupts.CPU73.CAL:Function_call_interrupts
    390615 ±  6%    +117.4%     849177 ±  3%  interrupts.CPU73.LOC:Local_timer_interrupts
      3003 ± 19%    +166.1%       7993 ± 49%  interrupts.CPU73.RES:Rescheduling_interrupts
      6692 ±  8%    +450.3%      36828 ± 88%  interrupts.CPU74.CAL:Function_call_interrupts
    390533 ±  6%    +117.4%     849163 ±  3%  interrupts.CPU74.LOC:Local_timer_interrupts
      3099 ± 17%    +152.2%       7818 ± 49%  interrupts.CPU74.RES:Rescheduling_interrupts
      7087 ±  7%    +418.7%      36760 ± 89%  interrupts.CPU75.CAL:Function_call_interrupts
    390754 ±  6%    +117.3%     849249 ±  3%  interrupts.CPU75.LOC:Local_timer_interrupts
      3499 ± 13%    +126.3%       7919 ± 52%  interrupts.CPU75.RES:Rescheduling_interrupts
      7517 ± 11%    +381.3%      36180 ± 90%  interrupts.CPU76.CAL:Function_call_interrupts
    390640 ±  6%    +117.4%     849185 ±  3%  interrupts.CPU76.LOC:Local_timer_interrupts
      7406 ±  5%    +386.6%      36042 ± 91%  interrupts.CPU77.CAL:Function_call_interrupts
    390601 ±  6%    +117.4%     849253 ±  3%  interrupts.CPU77.LOC:Local_timer_interrupts
      6893 ±  5%    +429.4%      36494 ± 90%  interrupts.CPU78.CAL:Function_call_interrupts
    390719 ±  6%    +117.3%     849096 ±  3%  interrupts.CPU78.LOC:Local_timer_interrupts
      3361 ±  7%    +121.8%       7455 ± 54%  interrupts.CPU78.RES:Rescheduling_interrupts
      6978 ±  7%    +420.2%      36296 ± 91%  interrupts.CPU79.CAL:Function_call_interrupts
    390688 ±  6%    +117.4%     849222 ±  3%  interrupts.CPU79.LOC:Local_timer_interrupts
      6997 ±  8%    +426.7%      36856 ± 92%  interrupts.CPU8.CAL:Function_call_interrupts
    388855 ±  7%    +118.4%     849266 ±  3%  interrupts.CPU8.LOC:Local_timer_interrupts
      3779 ± 14%    +104.9%       7744 ± 45%  interrupts.CPU8.RES:Rescheduling_interrupts
      6539 ±  8%    +453.8%      36214 ± 92%  interrupts.CPU80.CAL:Function_call_interrupts
    390742 ±  6%    +117.3%     849168 ±  3%  interrupts.CPU80.LOC:Local_timer_interrupts
      3010 ± 17%    +136.6%       7124 ± 64%  interrupts.CPU80.RES:Rescheduling_interrupts
      6863 ± 10%    +425.8%      36083 ± 89%  interrupts.CPU81.CAL:Function_call_interrupts
    390652 ±  6%    +117.4%     849154 ±  3%  interrupts.CPU81.LOC:Local_timer_interrupts
      3419 ± 15%    +116.8%       7413 ± 49%  interrupts.CPU81.RES:Rescheduling_interrupts
      7915 ±  8%    +359.6%      36378 ± 91%  interrupts.CPU82.CAL:Function_call_interrupts
    390600 ±  6%    +117.4%     849219 ±  3%  interrupts.CPU82.LOC:Local_timer_interrupts
      7126 ±  7%    +408.0%      36204 ± 90%  interrupts.CPU83.CAL:Function_call_interrupts
    390593 ±  6%    +117.4%     849229 ±  3%  interrupts.CPU83.LOC:Local_timer_interrupts
      6958 ±  4%    +435.5%      37262 ± 88%  interrupts.CPU84.CAL:Function_call_interrupts
    390708 ±  6%    +117.4%     849259 ±  3%  interrupts.CPU84.LOC:Local_timer_interrupts
      3378 ± 10%    +139.9%       8102 ± 47%  interrupts.CPU84.RES:Rescheduling_interrupts
      7234 ±  8%    +402.3%      36335 ± 91%  interrupts.CPU85.CAL:Function_call_interrupts
    390560 ±  6%    +117.4%     849212 ±  3%  interrupts.CPU85.LOC:Local_timer_interrupts
      7163 ± 10%    +402.6%      36002 ± 92%  interrupts.CPU86.CAL:Function_call_interrupts
    390301 ±  6%    +117.6%     849259 ±  3%  interrupts.CPU86.LOC:Local_timer_interrupts
      7377 ± 11%    +396.9%      36656 ± 89%  interrupts.CPU87.CAL:Function_call_interrupts
    390784 ±  6%    +117.3%     849128 ±  3%  interrupts.CPU87.LOC:Local_timer_interrupts
      6735 ± 11%    +449.9%      37035 ± 93%  interrupts.CPU9.CAL:Function_call_interrupts
    388678 ±  7%    +118.5%     849289 ±  3%  interrupts.CPU9.LOC:Local_timer_interrupts
      3564 ± 18%    +113.7%       7617 ± 49%  interrupts.CPU9.RES:Rescheduling_interrupts
  34282586 ±  6%    +118.0%   74729713 ±  3%  interrupts.LOC:Local_timer_interrupts
    317089          +109.1%     662999 ± 51%  interrupts.RES:Rescheduling_interrupts





Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


---
0DAY/LKP+ Test Infrastructure                   Open Source Technology Center
https://lists.01.org/hyperkitty/list/lkp@lists.01.org       Intel Corporation

Thanks,
Oliver Sang


View attachment "config-5.16.0-rc1-00006-g4fa18391ae8c" of type "text/plain" (173486 bytes)

View attachment "job-script" of type "text/plain" (8196 bytes)

View attachment "job.yaml" of type "text/plain" (5650 bytes)

View attachment "reproduce" of type "text/plain" (694 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ