lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Message-ID: <20190527031556.GQ19312@shao2-debian>
Date:   Mon, 27 May 2019 11:15:56 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     Jan Kara <jack@...e.cz>
Cc:     Theodore Ts'o <tytso@....edu>, Ira Weiny <ira.weiny@...el.com>,
        LKML <linux-kernel@...r.kernel.org>, linux-ext4@...r.kernel.org,
        lkp@...org
Subject: [ext4]  079f9927c7:  aim7.jobs-per-min 376.0% improvement

Greeting,

FYI, we noticed a 376.0% improvement of aim7.jobs-per-min due to commit:


commit: 079f9927c7bfa026d963db1455197159ebe5b534 ("ext4: gracefully handle ext4_break_layouts() failure during truncate")
https://git.kernel.org/cgit/linux/kernel/git/tytso/ext4.git dev

in testcase: aim7
on test machine: 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz with 384G memory
with following parameters:

	disk: 1BRD_48G
	fs: ext4
	test: creat-clo
	load: 1000
	cpufreq_governor: performance

test-description: AIM7 is a traditional UNIX system level benchmark suite which is used to test and measure the performance of multiuser system.
test-url: https://sourceforge.net/projects/aimbench/files/aim-suite7/



Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/load/rootfs/tbox_group/test/testcase:
  gcc-7/performance/1BRD_48G/ext4/x86_64-rhel-7.6/1000/debian-x86_64-2018-04-03.cgz/lkp-ivb-ep01/creat-clo/aim7

commit: 
  ee0ed02ca9 ("ext4: do not delete unlinked inode from orphan list on failed truncate")
  079f9927c7 ("ext4: gracefully handle ext4_break_layouts() failure during truncate")

ee0ed02ca93ef1ec 079f9927c7bfa026d963db14551 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          1:4          -25%            :4     dmesg.WARNING:at#for_ip_interrupt_entry/0x
           :4           25%           1:4     dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x
           :4           25%           1:4     dmesg.WARNING:stack_recursion
         %stddev     %change         %stddev
             \          |                \  
     16534          +376.0%      78703 ±  4%  aim7.jobs-per-min
    363.16           -78.9%      76.58 ±  5%  aim7.time.elapsed_time
    363.16           -78.9%      76.58 ±  5%  aim7.time.elapsed_time.max
    594747           -82.5%     103989 ±  5%  aim7.time.involuntary_context_switches
     97265 ±  3%     -37.4%      60892        aim7.time.minor_page_faults
     13398           -80.7%       2581 ±  5%  aim7.time.system_time
     33.72 ±  3%      -9.2%      30.61        aim7.time.user_time
   1221874           -70.9%     355969 ± 23%  aim7.time.voluntary_context_switches
     42.74 ±  4%      -7.2%      39.67 ±  2%  boot-time.boot
      1510 ±  5%      -7.9%       1391 ±  3%  boot-time.idle
      8.54          +125.9%      19.30 ±  7%  iostat.cpu.idle
     91.18           -12.7%      79.61        iostat.cpu.system
      0.27 ±  3%    +307.2%       1.09 ±  3%  iostat.cpu.user
      8.04 ±  2%      +9.3       17.30 ±  8%  mpstat.cpu.all.idle%
      0.01 ± 18%      -0.0        0.00 ± 42%  mpstat.cpu.all.iowait%
     91.68           -10.1       81.60        mpstat.cpu.all.sys%
      0.27 ±  3%      +0.8        1.10 ±  3%  mpstat.cpu.all.usr%
   1127458 ±  8%     +26.5%    1426646 ±  6%  numa-numastat.node0.local_node
   1129163 ±  8%     +26.8%    1431830 ±  6%  numa-numastat.node0.numa_hit
   1087298 ±  8%     +37.4%    1494060 ±  6%  numa-numastat.node1.local_node
   1098684 ±  8%     +36.7%    1501858 ±  6%  numa-numastat.node1.numa_hit
  38000204 ± 89%     -83.4%    6302127 ± 13%  cpuidle.C1.time
    441337 ± 94%     -64.7%     155896 ± 21%  cpuidle.C1.usage
  7.57e+08 ± 15%     -44.7%   4.19e+08 ± 18%  cpuidle.C6.time
   1339886 ±  6%     -56.4%     584695 ±  6%  cpuidle.C6.usage
     15308 ± 21%    +388.7%      74812 ± 36%  cpuidle.POLL.time
      8842 ± 23%    +925.3%      90663 ± 38%  cpuidle.POLL.usage
      8.00          +134.4%      18.75 ±  7%  vmstat.cpu.id
     91.00           -13.2%      79.00 ±  2%  vmstat.cpu.sy
      2201          +339.2%       9669 ±  5%  vmstat.io.bo
    739.25           -35.9%     474.00        vmstat.memory.buff
     40.75           -17.2%      33.75        vmstat.procs.r
     81511            +1.7%      82928        vmstat.system.in
      2166           -13.1%       1883        meminfo.Active(file)
    176619           -57.3%      75501 ±  3%  meminfo.AnonHugePages
    986.50 ± 75%     -96.1%      38.75 ±173%  meminfo.Mlocked
    110576           +39.9%     154732        meminfo.SUnreclaim
     46793 ±  7%     -25.0%      35102 ±  5%  meminfo.Shmem
    190857           +21.5%     231855        meminfo.Slab
      8496          +368.5%      39805 ±  3%  meminfo.max_used_kB
     10510 ± 19%     +15.5%      12138 ± 19%  numa-meminfo.node0.Mapped
    502.75 ± 74%     -96.8%      16.25 ±173%  numa-meminfo.node0.Mlocked
     64390 ± 14%     +40.0%      90175 ±  4%  numa-meminfo.node0.SUnreclaim
     32715 ± 24%     -60.4%      12939 ± 74%  numa-meminfo.node0.Shmem
    108863 ±  7%     +20.2%     130824 ±  8%  numa-meminfo.node0.Slab
    120127 ± 10%     -69.6%      36463 ± 31%  numa-meminfo.node1.AnonHugePages
    171951 ± 11%     -19.5%     138407 ± 13%  numa-meminfo.node1.AnonPages
     46192 ± 21%     +38.6%      64015 ±  5%  numa-meminfo.node1.SUnreclaim
     82005 ± 10%     +22.6%     100526 ± 10%  numa-meminfo.node1.Slab
      2620 ± 20%     +15.8%       3033 ± 19%  numa-vmstat.node0.nr_mapped
    125.25 ± 75%     -96.8%       4.00 ±173%  numa-vmstat.node0.nr_mlock
      8177 ± 24%     -60.5%       3232 ± 74%  numa-vmstat.node0.nr_shmem
     16098 ± 14%     +39.9%      22524 ±  4%  numa-vmstat.node0.nr_slab_unreclaimable
     42987 ± 11%     -19.5%      34596 ± 13%  numa-vmstat.node1.nr_anon_pages
    120.25 ± 81%     -95.4%       5.50 ±173%  numa-vmstat.node1.nr_mlock
     11547 ± 21%     +38.4%      15983 ±  6%  numa-vmstat.node1.nr_slab_unreclaimable
    995499 ± 12%     +13.7%    1132178 ±  7%  numa-vmstat.node1.numa_hit
    818394 ± 15%     +17.2%     958772 ±  8%  numa-vmstat.node1.numa_local
      2749            -7.0%       2556        turbostat.Avg_MHz
    433741 ± 96%     -65.9%     148066 ± 23%  turbostat.C1
   1337165 ±  6%     -56.5%     581717 ±  6%  turbostat.C6
      5.16 ± 15%      +8.0       13.12 ± 18%  turbostat.C6%
      6.00 ±  4%     +30.3%       7.82 ±  9%  turbostat.CPU%c1
      1.56 ± 35%    +389.9%       7.66 ± 24%  turbostat.CPU%c6
  29977039           -77.6%    6718772 ±  3%  turbostat.IRQ
      1.04 ± 16%    +451.8%       5.72 ± 17%  turbostat.Pkg%pc2
      0.01 ±100%   +8000.0%       0.41 ±101%  turbostat.Pkg%pc3
      0.07 ± 97%    +755.2%       0.62 ± 52%  turbostat.Pkg%pc6
     29420           -84.9%       4430 ±  5%  turbostat.SMI
     76258            -7.8%      70292        proc-vmstat.nr_active_anon
    541.25           -13.1%     470.50        proc-vmstat.nr_active_file
     69429            -4.1%      66607        proc-vmstat.nr_anon_pages
      1774 ± 24%     -68.2%     564.75 ± 11%  proc-vmstat.nr_dirtied
    276750            -1.1%     273776        proc-vmstat.nr_file_pages
      4975            +1.9%       5068        proc-vmstat.nr_inactive_anon
    486.25            -1.9%     477.25        proc-vmstat.nr_inactive_file
     23712            -6.5%      22169        proc-vmstat.nr_kernel_stack
      5858            +3.8%       6079        proc-vmstat.nr_mapped
    246.50 ± 75%     -96.1%       9.50 ±173%  proc-vmstat.nr_mlock
     11698 ±  7%     -25.0%       8775 ±  5%  proc-vmstat.nr_shmem
     20069            -3.9%      19281        proc-vmstat.nr_slab_reclaimable
     27647           +40.1%      38730        proc-vmstat.nr_slab_unreclaimable
      1724 ± 26%     -69.6%     524.50 ± 10%  proc-vmstat.nr_written
     76258            -7.8%      70292        proc-vmstat.nr_zone_active_anon
    541.25           -13.1%     470.50        proc-vmstat.nr_zone_active_file
      4975            +1.9%       5068        proc-vmstat.nr_zone_inactive_anon
    486.25            -1.9%     477.25        proc-vmstat.nr_zone_inactive_file
     48176 ±  4%     -75.1%      12005 ± 27%  proc-vmstat.numa_hint_faults
     31019 ±  6%     -84.0%       4952 ± 55%  proc-vmstat.numa_hint_faults_local
   2253964           +31.3%    2958574        proc-vmstat.numa_hit
   2240865           +31.4%    2945589        proc-vmstat.numa_local
     60592 ± 21%     -63.0%      22390 ± 53%  proc-vmstat.numa_pte_updates
   3620485           +51.5%    5486303        proc-vmstat.pgalloc_normal
    919928           -71.5%     262105 ±  4%  proc-vmstat.pgfault
   3395671           +55.1%    5267921        proc-vmstat.pgfree
    809823            -3.2%     783689        proc-vmstat.pgpgout
      2267 ± 11%     -27.2%       1651 ±  4%  slabinfo.buffer_head.active_objs
      2269 ± 11%     -26.7%       1663 ±  5%  slabinfo.buffer_head.num_objs
      1211           -25.9%     898.00 ±  9%  slabinfo.dquot.active_objs
      1211           -25.9%     898.00 ±  9%  slabinfo.dquot.num_objs
      3575 ±  3%     -13.6%       3090        slabinfo.ext4_inode_cache.active_objs
      3654 ±  3%     -14.8%       3114        slabinfo.ext4_inode_cache.num_objs
     34698 ±  2%    +582.6%     236857 ±  3%  slabinfo.filp.active_objs
      1192          +524.6%       7445 ±  3%  slabinfo.filp.active_slabs
     38161          +524.4%     238278 ±  3%  slabinfo.filp.num_objs
      1192          +524.6%       7445 ±  3%  slabinfo.filp.num_slabs
      3296           -11.0%       2934        slabinfo.fscrypt_ctx.active_objs
      3296           -11.0%       2934        slabinfo.fscrypt_ctx.num_objs
      3140           -16.3%       2627 ±  3%  slabinfo.kmalloc-128.active_objs
      3140           -16.3%       2627 ±  3%  slabinfo.kmalloc-128.num_objs
      5591 ±  3%      -9.0%       5088        slabinfo.kmalloc-96.active_objs
      5605 ±  3%      -9.1%       5095        slabinfo.kmalloc-96.num_objs
      2595 ±  5%     +10.7%       2873 ±  2%  slabinfo.kmalloc-rcl-64.active_objs
      2595 ±  5%     +10.7%       2873 ±  2%  slabinfo.kmalloc-rcl-64.num_objs
      1016 ±  3%     -12.6%     888.00 ±  8%  slabinfo.kmem_cache_node.active_objs
      1056 ±  3%     -12.1%     928.00 ±  7%  slabinfo.kmem_cache_node.num_objs
      1162           -11.5%       1028        slabinfo.names_cache.active_objs
      1195           -11.4%       1059        slabinfo.names_cache.num_objs
    287.75 ± 12%     -51.6%     139.25 ± 19%  slabinfo.nfs_commit_data.active_objs
    287.75 ± 12%     -51.6%     139.25 ± 19%  slabinfo.nfs_commit_data.num_objs
    225.00 ± 12%     -59.6%      91.00        slabinfo.nfs_read_data.active_objs
    225.00 ± 12%     -59.6%      91.00        slabinfo.nfs_read_data.num_objs
     13453           -11.3%      11935        slabinfo.proc_inode_cache.active_objs
      1067 ±  6%     -19.2%     862.25 ± 12%  slabinfo.task_group.active_objs
      1067 ±  6%     -19.2%     862.25 ± 12%  slabinfo.task_group.num_objs
      2.99 ± 14%    +158.5%       7.73 ±  7%  perf-stat.i.MPKI
 1.129e+10           -10.5%   1.01e+10        perf-stat.i.branch-instructions
      0.54 ±  6%      +1.2        1.73 ±  7%  perf-stat.i.branch-miss-rate%
  23864805           +87.5%   44737022 ±  4%  perf-stat.i.branch-misses
     27.92            +1.8       29.71        perf-stat.i.cache-miss-rate%
  23518787 ±  4%     +25.8%   29586748        perf-stat.i.cache-misses
  84242323 ±  5%     +18.4%   99771184        perf-stat.i.cache-references
      1.98 ±  2%     +25.3%       2.48 ±  2%  perf-stat.i.cpi
 1.098e+11            -5.8%  1.035e+11        perf-stat.i.cpu-cycles
      1819           -54.8%     822.64 ±  4%  perf-stat.i.cpu-migrations
      4591 ±  4%     -31.3%       3153        perf-stat.i.cycles-between-cache-misses
      0.19 ± 20%      +0.5        0.68 ± 29%  perf-stat.i.dTLB-load-miss-rate%
  31335134 ± 23%    +254.2%   1.11e+08 ± 32%  perf-stat.i.dTLB-load-misses
 1.843e+10           -11.9%  1.624e+10        perf-stat.i.dTLB-loads
   2954542 ± 22%     +95.9%    5788569 ± 24%  perf-stat.i.dTLB-store-misses
 1.264e+09          +131.4%  2.925e+09 ±  4%  perf-stat.i.dTLB-stores
     64.60 ± 34%     +29.0       93.61 ±  2%  perf-stat.i.iTLB-load-miss-rate%
   1328625 ± 20%    +276.4%    5001400 ± 16%  perf-stat.i.iTLB-load-misses
 5.944e+10           -11.2%  5.281e+10        perf-stat.i.instructions
     46008 ± 17%     -78.8%       9761 ± 18%  perf-stat.i.instructions-per-iTLB-miss
      0.53           -10.6%       0.48 ±  2%  perf-stat.i.ipc
      2433           +28.5%       3127        perf-stat.i.minor-faults
     47.71            -4.8       42.88        perf-stat.i.node-load-miss-rate%
  13410002 ±  7%     -13.0%   11663161 ±  4%  perf-stat.i.node-load-misses
     40.76            -2.0       38.75        perf-stat.i.node-store-miss-rate%
   6166336 ±  2%     +57.1%    9689578 ±  3%  perf-stat.i.node-store-misses
   8743609 ±  2%     +55.2%   13568157 ±  3%  perf-stat.i.node-stores
      2433           +28.5%       3127        perf-stat.i.page-faults
      1.42 ±  6%     +33.3%       1.89        perf-stat.overall.MPKI
      0.21            +0.2        0.44 ±  4%  perf-stat.overall.branch-miss-rate%
     27.94            +1.7       29.66        perf-stat.overall.cache-miss-rate%
      1.85            +6.1%       1.96        perf-stat.overall.cpi
      4678 ±  4%     -25.2%       3497        perf-stat.overall.cycles-between-cache-misses
      0.17 ± 23%      +0.5        0.68 ± 33%  perf-stat.overall.dTLB-load-miss-rate%
     63.84 ± 35%     +30.3       94.15 ±  2%  perf-stat.overall.iTLB-load-miss-rate%
     46330 ± 16%     -76.5%      10887 ± 18%  perf-stat.overall.instructions-per-iTLB-miss
      0.54            -5.7%       0.51        perf-stat.overall.ipc
     47.98            -4.4       43.56        perf-stat.overall.node-load-miss-rate%
 1.126e+10           -11.5%  9.964e+09        perf-stat.ps.branch-instructions
  23803167           +85.5%   44150955 ±  4%  perf-stat.ps.branch-misses
  23452823 ±  4%     +24.5%   29196539        perf-stat.ps.cache-misses
  84006313 ±  5%     +17.2%   98455946        perf-stat.ps.cache-references
 1.095e+11            -6.8%  1.021e+11        perf-stat.ps.cpu-cycles
      1814           -55.3%     811.78 ±  4%  perf-stat.ps.cpu-migrations
  31246611 ± 23%    +250.4%  1.095e+08 ± 32%  perf-stat.ps.dTLB-load-misses
 1.837e+10           -12.8%  1.602e+10        perf-stat.ps.dTLB-loads
   2946207 ± 22%     +93.8%    5711211 ± 24%  perf-stat.ps.dTLB-store-misses
 1.261e+09          +128.9%  2.886e+09 ±  4%  perf-stat.ps.dTLB-stores
   1324897 ± 20%    +272.5%    4935070 ± 16%  perf-stat.ps.iTLB-load-misses
 5.927e+10           -12.1%   5.21e+10        perf-stat.ps.instructions
      2426           +27.4%       3092        perf-stat.ps.minor-faults
     79779            -1.0%      78978        perf-stat.ps.msec
  13371979 ±  7%     -13.9%   11507768 ±  4%  perf-stat.ps.node-load-misses
   6148850 ±  2%     +55.5%    9560388 ±  3%  perf-stat.ps.node-store-misses
   8718962 ±  2%     +53.5%   13387860 ±  3%  perf-stat.ps.node-stores
      2426           +27.4%       3092        perf-stat.ps.page-faults
 2.168e+13           -81.0%  4.115e+12 ±  4%  perf-stat.total.instructions
    155607 ±  7%     -84.5%      24158        sched_debug.cfs_rq:/.exec_clock.avg
    158019 ±  7%     -83.9%      25412        sched_debug.cfs_rq:/.exec_clock.max
    154697 ±  7%     -84.5%      23983        sched_debug.cfs_rq:/.exec_clock.min
    499.00 ±  6%     -54.7%     226.10 ± 28%  sched_debug.cfs_rq:/.exec_clock.stddev
     26441 ±  2%     -27.3%      19235 ±  3%  sched_debug.cfs_rq:/.load.avg
     77245 ± 19%     -28.8%      54981 ± 16%  sched_debug.cfs_rq:/.load.max
      4994 ± 71%    +148.9%      12431        sched_debug.cfs_rq:/.load.min
     16095 ± 28%     -31.8%      10978 ± 11%  sched_debug.cfs_rq:/.load.stddev
     44.75 ± 18%     +92.2%      86.02 ± 11%  sched_debug.cfs_rq:/.load_avg.avg
    300.77 ± 19%    +124.0%     673.88 ± 31%  sched_debug.cfs_rq:/.load_avg.max
      9.08 ± 10%     -54.6%       4.12 ± 66%  sched_debug.cfs_rq:/.load_avg.min
     63.85 ± 20%    +165.7%     169.68 ± 15%  sched_debug.cfs_rq:/.load_avg.stddev
   5946728 ±  7%     -83.4%     989775        sched_debug.cfs_rq:/.min_vruntime.avg
   6057788 ±  7%     -82.2%    1075477 ±  3%  sched_debug.cfs_rq:/.min_vruntime.max
   5666459 ±  8%     -83.9%     915130        sched_debug.cfs_rq:/.min_vruntime.min
     61219 ±  7%     -34.2%      40297 ± 37%  sched_debug.cfs_rq:/.min_vruntime.stddev
      0.83 ±  4%     -18.3%       0.68        sched_debug.cfs_rq:/.nr_running.avg
      0.22 ± 71%    +127.0%       0.50        sched_debug.cfs_rq:/.nr_running.min
      0.57 ±  9%     -87.3%       0.07 ± 39%  sched_debug.cfs_rq:/.nr_spread_over.avg
      6.20 ± 17%     -83.9%       1.00 ± 35%  sched_debug.cfs_rq:/.nr_spread_over.max
      1.10 ± 10%     -78.9%       0.23 ± 35%  sched_debug.cfs_rq:/.nr_spread_over.stddev
      7.72 ±103%    +348.0%      34.59 ± 30%  sched_debug.cfs_rq:/.removed.load_avg.avg
     79.24 ±100%    +544.7%     510.88        sched_debug.cfs_rq:/.removed.load_avg.max
     23.18 ±100%    +442.4%     125.74 ± 13%  sched_debug.cfs_rq:/.removed.load_avg.stddev
    356.27 ±103%    +346.6%       1591 ± 29%  sched_debug.cfs_rq:/.removed.runnable_sum.avg
      3682 ±100%    +538.4%      23507        sched_debug.cfs_rq:/.removed.runnable_sum.max
      1069 ±100%    +440.7%       5785 ± 13%  sched_debug.cfs_rq:/.removed.runnable_sum.stddev
      2.15 ±102%    +333.0%       9.32 ± 18%  sched_debug.cfs_rq:/.removed.util_avg.avg
     25.76 ±100%    +543.4%     165.75 ±  8%  sched_debug.cfs_rq:/.removed.util_avg.max
      6.54 ±100%    +450.1%      35.99 ± 10%  sched_debug.cfs_rq:/.removed.util_avg.stddev
     20.86 ±  3%     -28.3%      14.95 ±  2%  sched_debug.cfs_rq:/.runnable_load_avg.avg
     64.47 ± 26%     -43.8%      36.25 ± 14%  sched_debug.cfs_rq:/.runnable_load_avg.max
     13.48 ± 33%     -45.3%       7.37 ± 14%  sched_debug.cfs_rq:/.runnable_load_avg.stddev
     23012 ±  4%     -29.4%      16252 ±  2%  sched_debug.cfs_rq:/.runnable_weight.avg
     65365 ± 27%     -37.8%      40658 ± 19%  sched_debug.cfs_rq:/.runnable_weight.max
      4983 ± 72%    +149.2%      12418        sched_debug.cfs_rq:/.runnable_weight.min
     13411 ± 36%     -48.3%       6932 ± 14%  sched_debug.cfs_rq:/.runnable_weight.stddev
    279326 ± 14%     -75.3%      69126 ± 16%  sched_debug.cfs_rq:/.spread0.avg
    390244 ± 10%     -60.3%     154863 ± 20%  sched_debug.cfs_rq:/.spread0.max
     61180 ±  7%     -34.1%      40294 ± 37%  sched_debug.cfs_rq:/.spread0.stddev
    382.72 ± 12%     -56.6%     166.00 ± 56%  sched_debug.cfs_rq:/.util_avg.min
    233.66 ± 17%     -28.1%     167.97 ±  6%  sched_debug.cfs_rq:/.util_est_enqueued.avg
    562074 ±  5%     -41.3%     330076 ±  9%  sched_debug.cpu.avg_idle.avg
    153410 ± 17%     -92.0%      12251 ±118%  sched_debug.cpu.avg_idle.min
    227868 ±  5%     +21.8%     277629 ±  5%  sched_debug.cpu.avg_idle.stddev
    222329 ±  5%     -65.5%      76724        sched_debug.cpu.clock.avg
    222336 ±  5%     -65.5%      76728        sched_debug.cpu.clock.max
    222321 ±  5%     -65.5%      76719        sched_debug.cpu.clock.min
      4.39 ± 15%     -45.6%       2.39 ± 24%  sched_debug.cpu.clock.stddev
    222329 ±  5%     -65.5%      76724        sched_debug.cpu.clock_task.avg
    222336 ±  5%     -65.5%      76728        sched_debug.cpu.clock_task.max
    222321 ±  5%     -65.5%      76719        sched_debug.cpu.clock_task.min
      4.39 ± 15%     -45.6%       2.39 ± 24%  sched_debug.cpu.clock_task.stddev
     21.54 ±  4%     -28.9%      15.32 ±  5%  sched_debug.cpu.cpu_load[0].avg
     68.51 ± 24%     -53.1%      32.12 ±  7%  sched_debug.cpu.cpu_load[0].max
     13.77 ± 32%     -49.0%       7.02 ±  4%  sched_debug.cpu.cpu_load[0].stddev
     22.15 ±  4%     -26.3%      16.33 ±  4%  sched_debug.cpu.cpu_load[1].avg
     78.08 ± 32%     -53.1%      36.62 ± 15%  sched_debug.cpu.cpu_load[1].max
      6.70 ± 19%     -45.9%       3.62 ± 40%  sched_debug.cpu.cpu_load[1].min
     14.37 ± 33%     -55.3%       6.42 ±  4%  sched_debug.cpu.cpu_load[1].stddev
     22.47 ±  4%     -23.1%      17.28 ±  4%  sched_debug.cpu.cpu_load[2].avg
     22.77 ±  4%     -19.5%      18.34 ±  2%  sched_debug.cpu.cpu_load[3].avg
     22.98 ±  2%     -13.8%      19.81        sched_debug.cpu.cpu_load[4].avg
     93.10 ± 14%     +70.0%     158.25 ± 13%  sched_debug.cpu.cpu_load[4].max
     15.03 ± 24%     +57.8%      23.72 ± 14%  sched_debug.cpu.cpu_load[4].stddev
      1838 ±  4%     -29.5%       1297 ±  3%  sched_debug.cpu.curr->pid.avg
      6582 ±  4%     -58.1%       2761        sched_debug.cpu.curr->pid.max
    361.45 ± 80%    +106.9%     747.88 ±  8%  sched_debug.cpu.curr->pid.min
      1008 ± 10%     -47.2%     531.98        sched_debug.cpu.curr->pid.stddev
     26427 ±  2%     -27.0%      19302 ±  2%  sched_debug.cpu.load.avg
     77266 ± 19%     -29.1%      54762 ± 14%  sched_debug.cpu.load.max
     16043 ± 27%     -31.3%      11015 ± 10%  sched_debug.cpu.load.stddev
      0.00 ±  3%     -11.5%       0.00 ±  4%  sched_debug.cpu.next_balance.stddev
    182784 ±  7%     -77.9%      40349        sched_debug.cpu.nr_load_updates.avg
    192548 ±  6%     -75.2%      47823 ±  4%  sched_debug.cpu.nr_load_updates.max
    179866 ±  7%     -79.1%      37656        sched_debug.cpu.nr_load_updates.min
      2037 ±  2%     -15.1%       1729 ± 11%  sched_debug.cpu.nr_load_updates.stddev
      0.95 ±  6%     -24.7%       0.72        sched_debug.cpu.nr_running.avg
      0.47 ± 11%     -20.2%       0.37 ± 12%  sched_debug.cpu.nr_running.stddev
     40112 ±  7%     -76.4%       9447 ± 12%  sched_debug.cpu.nr_switches.avg
     52113 ±  3%     -65.3%      18084 ±  6%  sched_debug.cpu.nr_switches.max
     36473 ±  8%     -81.5%       6753 ± 17%  sched_debug.cpu.nr_switches.min
      3224 ±  6%     -20.8%       2553 ±  7%  sched_debug.cpu.nr_switches.stddev
     18.87 ±  5%     -43.6%      10.64 ±  4%  sched_debug.cpu.nr_uninterruptible.avg
    852.94 ±  4%     -89.8%      86.88 ± 16%  sched_debug.cpu.nr_uninterruptible.max
   -286.66           -85.2%     -42.38        sched_debug.cpu.nr_uninterruptible.min
    194.19 ±  4%     -86.0%      27.19 ±  4%  sched_debug.cpu.nr_uninterruptible.stddev
     38344 ±  7%     -76.8%       8892 ± 14%  sched_debug.cpu.sched_count.avg
     44285 ±  5%     -72.3%      12276 ± 11%  sched_debug.cpu.sched_count.max
     35777 ±  8%     -80.9%       6843 ± 18%  sched_debug.cpu.sched_count.min
      1748 ±  7%     -26.6%       1282 ±  9%  sched_debug.cpu.sched_count.stddev
     10951 ±  7%     -77.0%       2523 ± 22%  sched_debug.cpu.sched_goidle.avg
     13008 ±  7%     -69.8%       3926 ± 14%  sched_debug.cpu.sched_goidle.max
     10335 ±  7%     -82.9%       1764 ± 27%  sched_debug.cpu.sched_goidle.min
    619.45 ±  7%     -22.1%     482.74 ± 12%  sched_debug.cpu.sched_goidle.stddev
     25678 ±  7%     -78.9%       5421 ± 13%  sched_debug.cpu.ttwu_count.avg
     33687 ± 14%     -80.6%       6542 ± 11%  sched_debug.cpu.ttwu_count.max
     23414 ±  8%     -78.8%       4973 ± 14%  sched_debug.cpu.ttwu_count.min
      2076 ± 25%     -82.8%     356.21 ± 10%  sched_debug.cpu.ttwu_count.stddev
      5437 ±  7%     -82.6%     945.25 ±  5%  sched_debug.cpu.ttwu_local.avg
      7134 ±  7%     -76.9%       1651 ± 13%  sched_debug.cpu.ttwu_local.max
      4648 ±  7%     -84.4%     726.00 ± 10%  sched_debug.cpu.ttwu_local.min
    486.67 ± 14%     -67.0%     160.43 ± 18%  sched_debug.cpu.ttwu_local.stddev
    222322 ±  5%     -65.5%      76718        sched_debug.cpu_clk
    218751 ±  5%     -66.6%      73147        sched_debug.ktime
    222764 ±  5%     -65.4%      77160        sched_debug.sched_clk
    125327           -76.6%      29327 ± 17%  softirqs.CPU0.RCU
     20714 ±  3%     -57.8%       8743 ±  6%  softirqs.CPU0.SCHED
    153988           -73.1%      41422 ±  5%  softirqs.CPU0.TIMER
    124970           -76.3%      29675 ± 22%  softirqs.CPU1.RCU
     15792           -64.5%       5602 ±  8%  softirqs.CPU1.SCHED
    154998 ±  3%     -74.0%      40293 ± 10%  softirqs.CPU1.TIMER
    123582           -77.2%      28183 ± 19%  softirqs.CPU10.RCU
     15646           -73.4%       4156 ±  7%  softirqs.CPU10.SCHED
    149793 ±  3%     -75.6%      36510 ±  4%  softirqs.CPU10.TIMER
    124322           -77.6%      27802 ± 21%  softirqs.CPU11.RCU
     15923           -69.4%       4872 ±  3%  softirqs.CPU11.SCHED
    150655 ±  3%     -74.7%      38147 ±  3%  softirqs.CPU11.TIMER
    126447           -74.8%      31910 ± 22%  softirqs.CPU12.RCU
     16156           -66.5%       5413 ± 18%  softirqs.CPU12.SCHED
    154057 ±  2%     -73.2%      41265 ±  8%  softirqs.CPU12.TIMER
    126457           -76.5%      29743 ± 21%  softirqs.CPU13.RCU
     15979           -69.9%       4808        softirqs.CPU13.SCHED
    154436 ±  7%     -73.3%      41274 ±  8%  softirqs.CPU13.TIMER
    125940 ±  2%     -77.1%      28826 ± 22%  softirqs.CPU14.RCU
     16497 ±  3%     -70.2%       4916        softirqs.CPU14.SCHED
    153940 ±  2%     -74.8%      38789 ±  4%  softirqs.CPU14.TIMER
    124842           -78.1%      27353 ± 22%  softirqs.CPU15.RCU
     15957           -69.9%       4806 ±  2%  softirqs.CPU15.SCHED
    156045 ±  3%     -75.0%      39007 ±  3%  softirqs.CPU15.TIMER
    124602           -77.3%      28319 ± 20%  softirqs.CPU16.RCU
     16843 ±  6%     -71.1%       4874        softirqs.CPU16.SCHED
    153923 ±  3%     -74.3%      39521 ±  3%  softirqs.CPU16.TIMER
    123307           -76.9%      28498 ± 21%  softirqs.CPU17.RCU
     15719 ±  2%     -68.9%       4886 ±  3%  softirqs.CPU17.SCHED
    153920 ±  5%     -74.4%      39333 ±  3%  softirqs.CPU17.TIMER
    123677           -77.2%      28196 ± 21%  softirqs.CPU18.RCU
     16079           -69.4%       4922 ±  4%  softirqs.CPU18.SCHED
    148412           -73.7%      39028 ±  5%  softirqs.CPU18.TIMER
    124392           -78.1%      27286 ± 21%  softirqs.CPU19.RCU
     15882 ±  2%     -69.4%       4855 ±  6%  softirqs.CPU19.SCHED
    152830 ±  6%     -74.9%      38385 ±  3%  softirqs.CPU19.TIMER
    125331           -76.5%      29431 ± 21%  softirqs.CPU2.RCU
     15735           -70.6%       4624 ±  7%  softirqs.CPU2.SCHED
    148379           -71.0%      43101 ± 14%  softirqs.CPU2.TIMER
    123961           -76.0%      29706 ± 20%  softirqs.CPU20.RCU
     16273 ±  3%     -68.3%       5157 ±  9%  softirqs.CPU20.SCHED
    151943 ±  2%     -74.5%      38701 ±  3%  softirqs.CPU20.TIMER
    125243 ±  2%     -76.2%      29763 ± 21%  softirqs.CPU21.RCU
     15652 ±  2%     -69.6%       4760 ±  5%  softirqs.CPU21.SCHED
    161256 ± 12%     -75.9%      38857 ±  4%  softirqs.CPU21.TIMER
    124479           -75.9%      30023 ± 19%  softirqs.CPU22.RCU
     16100           -68.2%       5119 ±  2%  softirqs.CPU22.SCHED
    150035           -73.4%      39884 ±  4%  softirqs.CPU22.TIMER
    125176           -77.3%      28403 ± 19%  softirqs.CPU23.RCU
     15832           -68.6%       4969 ±  7%  softirqs.CPU23.SCHED
    167218 ± 11%     -76.4%      39544 ±  6%  softirqs.CPU23.TIMER
    124460           -76.9%      28810 ± 23%  softirqs.CPU24.RCU
     15964           -69.2%       4912 ±  4%  softirqs.CPU24.SCHED
    153473           -74.8%      38717 ±  3%  softirqs.CPU24.TIMER
    125379 ±  2%     -77.8%      27828 ± 20%  softirqs.CPU25.RCU
     15938           -69.3%       4887 ±  6%  softirqs.CPU25.SCHED
    153816 ±  4%     -74.6%      39145 ±  4%  softirqs.CPU25.TIMER
    124840           -76.9%      28822 ± 21%  softirqs.CPU26.RCU
     16065           -69.4%       4920        softirqs.CPU26.SCHED
    151409           -74.1%      39241 ±  7%  softirqs.CPU26.TIMER
    124537           -76.9%      28727 ± 23%  softirqs.CPU27.RCU
     16079           -70.2%       4794 ±  3%  softirqs.CPU27.SCHED
    154904 ±  2%     -74.1%      40193 ± 14%  softirqs.CPU27.TIMER
    124537           -76.6%      29154 ± 17%  softirqs.CPU28.RCU
     16165           -69.1%       4992 ±  2%  softirqs.CPU28.SCHED
    149455 ±  2%     -74.1%      38671 ±  5%  softirqs.CPU28.TIMER
    124603           -77.7%      27737 ± 17%  softirqs.CPU29.RCU
     16574 ±  7%     -70.8%       4837 ±  4%  softirqs.CPU29.SCHED
    154094 ±  7%     -74.9%      38727 ±  4%  softirqs.CPU29.TIMER
    125497 ±  2%     -77.6%      28139 ± 21%  softirqs.CPU3.RCU
     17110 ±  5%     -66.6%       5713 ± 15%  softirqs.CPU3.SCHED
    159439           -74.4%      40765 ±  4%  softirqs.CPU3.TIMER
    125858           -76.7%      29336 ± 20%  softirqs.CPU30.RCU
     15553           -70.8%       4539 ±  3%  softirqs.CPU30.SCHED
    149468 ±  2%     -74.8%      37703 ±  6%  softirqs.CPU30.TIMER
    124502           -78.2%      27131 ± 21%  softirqs.CPU31.RCU
     15962           -69.6%       4850 ±  4%  softirqs.CPU31.SCHED
    150482 ±  4%     -74.9%      37815 ±  4%  softirqs.CPU31.TIMER
    122223 ±  2%     -74.1%      31635 ± 16%  softirqs.CPU32.RCU
     15947 ±  2%     -68.6%       5015 ±  4%  softirqs.CPU32.SCHED
    161780 ±  9%     -73.7%      42498 ±  7%  softirqs.CPU32.TIMER
    121276           -77.5%      27260 ± 19%  softirqs.CPU33.RCU
     15742           -69.1%       4858 ±  2%  softirqs.CPU33.SCHED
    154469 ±  7%     -72.2%      42993 ± 15%  softirqs.CPU33.TIMER
    120980           -78.5%      26031 ± 19%  softirqs.CPU34.RCU
     15924 ±  2%     -68.7%       4987        softirqs.CPU34.SCHED
    151995 ±  2%     -74.4%      38894 ±  5%  softirqs.CPU34.TIMER
    121999           -76.2%      28990 ± 23%  softirqs.CPU35.RCU
     15665 ±  2%     -68.4%       4954 ±  5%  softirqs.CPU35.SCHED
    155429 ±  2%     -75.1%      38760 ±  2%  softirqs.CPU35.TIMER
    121908           -77.5%      27433 ± 24%  softirqs.CPU36.RCU
     15870           -69.2%       4893 ±  2%  softirqs.CPU36.SCHED
    152401 ±  3%     -74.2%      39339 ±  4%  softirqs.CPU36.TIMER
    120644           -78.0%      26489 ± 19%  softirqs.CPU37.RCU
     15743 ±  2%     -69.6%       4789 ±  5%  softirqs.CPU37.SCHED
    152409 ±  4%     -74.5%      38878 ±  4%  softirqs.CPU37.TIMER
    121224           -75.9%      29208 ± 19%  softirqs.CPU38.RCU
     15934           -67.4%       5198 ±  8%  softirqs.CPU38.SCHED
    148974 ±  2%     -73.7%      39206 ±  7%  softirqs.CPU38.TIMER
    120939           -77.8%      26904 ± 22%  softirqs.CPU39.RCU
     15635           -69.9%       4702 ±  6%  softirqs.CPU39.SCHED
    163698 ± 15%     -76.9%      37831 ±  4%  softirqs.CPU39.TIMER
    126050           -76.9%      29174 ± 21%  softirqs.CPU4.RCU
     16129           -69.3%       4954 ±  3%  softirqs.CPU4.SCHED
    153560           -74.5%      39176 ±  4%  softirqs.CPU4.TIMER
    125855           -78.0%      27679 ± 22%  softirqs.CPU5.RCU
     16649 ±  3%     -70.9%       4839        softirqs.CPU5.SCHED
    154576 ±  5%     -74.7%      39120 ±  4%  softirqs.CPU5.TIMER
    124711           -77.2%      28483 ± 20%  softirqs.CPU6.RCU
     16290 ±  2%     -68.6%       5118 ±  6%  softirqs.CPU6.SCHED
    151528           -72.5%      41640 ± 13%  softirqs.CPU6.TIMER
    126001           -77.2%      28694 ± 21%  softirqs.CPU7.RCU
     16072           -68.8%       5013 ±  6%  softirqs.CPU7.SCHED
    155073 ±  2%     -75.4%      38089 ±  4%  softirqs.CPU7.TIMER
    125571           -77.2%      28640 ± 21%  softirqs.CPU8.RCU
     16083           -69.6%       4887 ±  2%  softirqs.CPU8.SCHED
    148865 ±  2%     -74.1%      38616 ±  5%  softirqs.CPU8.TIMER
    125533 ±  2%     -77.8%      27916 ± 18%  softirqs.CPU9.RCU
     16028           -69.4%       4905 ±  3%  softirqs.CPU9.SCHED
    152644 ±  6%     -74.9%      38343 ±  4%  softirqs.CPU9.TIMER
   4971208           -77.0%    1142693 ± 20%  softirqs.RCU
    645925           -68.9%     201062 ±  2%  softirqs.SCHED
   6149792 ±  2%     -74.3%    1577446 ±  3%  softirqs.TIMER
    617.50 ± 18%     -77.9%     136.75 ± 43%  interrupts.37:IR-PCI-MSI.524289-edge.eth0-TxRx-0
    306.75 ± 20%     -76.3%      72.75 ± 33%  interrupts.38:IR-PCI-MSI.524290-edge.eth0-TxRx-1
    447.00 ± 35%     -86.4%      61.00 ± 37%  interrupts.39:IR-PCI-MSI.524291-edge.eth0-TxRx-2
    392.50 ± 26%     -86.8%      51.75 ±  7%  interrupts.40:IR-PCI-MSI.524292-edge.eth0-TxRx-3
    251.50 ± 13%     -74.0%      65.50 ± 57%  interrupts.41:IR-PCI-MSI.524293-edge.eth0-TxRx-4
    385.75 ± 50%     -79.5%      79.00 ± 38%  interrupts.42:IR-PCI-MSI.524294-edge.eth0-TxRx-5
    297.00 ± 36%     -68.9%      92.50 ± 57%  interrupts.43:IR-PCI-MSI.524295-edge.eth0-TxRx-6
    404.00 ± 34%     -80.8%      77.50 ± 31%  interrupts.44:IR-PCI-MSI.524296-edge.eth0-TxRx-7
    140895 ±  5%     -67.3%      46076 ±  9%  interrupts.CAL:Function_call_interrupts
      3751 ±  5%     -69.6%       1141 ±  9%  interrupts.CPU0.CAL:Function_call_interrupts
    732484           -78.3%     158839 ±  3%  interrupts.CPU0.LOC:Local_timer_interrupts
      5101 ± 35%     -79.9%       1027 ±173%  interrupts.CPU0.NMI:Non-maskable_interrupts
      5101 ± 35%     -79.9%       1027 ±173%  interrupts.CPU0.PMI:Performance_monitoring_interrupts
      7691 ± 10%     -67.0%       2537 ± 22%  interrupts.CPU0.RES:Rescheduling_interrupts
      3323 ±  8%     -65.4%       1150 ±  9%  interrupts.CPU1.CAL:Function_call_interrupts
    732357           -78.4%     158544 ±  3%  interrupts.CPU1.LOC:Local_timer_interrupts
      7105 ±  7%     -54.5%       3231 ± 28%  interrupts.CPU1.RES:Rescheduling_interrupts
      3609 ±  6%     -66.8%       1199 ±  8%  interrupts.CPU10.CAL:Function_call_interrupts
    732159           -78.4%     158400 ±  4%  interrupts.CPU10.LOC:Local_timer_interrupts
      6721 ±  8%     -69.3%       2065 ± 10%  interrupts.CPU10.RES:Rescheduling_interrupts
      3638 ±  8%     -69.4%       1114 ± 12%  interrupts.CPU11.CAL:Function_call_interrupts
    732161           -78.4%     158371 ±  3%  interrupts.CPU11.LOC:Local_timer_interrupts
      6935 ±  6%     -68.0%       2219 ± 21%  interrupts.CPU11.RES:Rescheduling_interrupts
      3571 ±  8%     -66.5%       1196 ±  9%  interrupts.CPU12.CAL:Function_call_interrupts
    732160           -78.4%     158295 ±  3%  interrupts.CPU12.LOC:Local_timer_interrupts
      6735 ±  9%     -68.7%       2109 ± 10%  interrupts.CPU12.RES:Rescheduling_interrupts
      3586 ±  9%     -67.8%       1155 ±  8%  interrupts.CPU13.CAL:Function_call_interrupts
    732074           -78.4%     158256 ±  3%  interrupts.CPU13.LOC:Local_timer_interrupts
      6965 ±  6%     -72.6%       1910 ±  8%  interrupts.CPU13.RES:Rescheduling_interrupts
      3399 ±  5%     -68.3%       1077 ± 18%  interrupts.CPU14.CAL:Function_call_interrupts
    732544           -78.3%     158727 ±  4%  interrupts.CPU14.LOC:Local_timer_interrupts
      6709 ±  7%     -69.1%       2076 ± 12%  interrupts.CPU14.RES:Rescheduling_interrupts
      3554 ± 11%     -65.9%       1213 ±  9%  interrupts.CPU15.CAL:Function_call_interrupts
    731864           -78.4%     158337 ±  4%  interrupts.CPU15.LOC:Local_timer_interrupts
      6902 ±  7%     -72.2%       1917 ± 16%  interrupts.CPU15.RES:Rescheduling_interrupts
      3287 ±  5%     -64.4%       1171 ±  8%  interrupts.CPU16.CAL:Function_call_interrupts
    732135           -78.4%     158356 ±  3%  interrupts.CPU16.LOC:Local_timer_interrupts
      6625 ±  9%     -69.0%       2053 ± 10%  interrupts.CPU16.RES:Rescheduling_interrupts
      3585 ± 10%     -69.5%       1094 ± 16%  interrupts.CPU17.CAL:Function_call_interrupts
    730713           -78.3%     158566 ±  3%  interrupts.CPU17.LOC:Local_timer_interrupts
      6874 ±  7%     -72.2%       1910 ±  7%  interrupts.CPU17.RES:Rescheduling_interrupts
      3655 ±  9%     -68.4%       1156 ± 12%  interrupts.CPU18.CAL:Function_call_interrupts
    732401           -78.3%     158690 ±  4%  interrupts.CPU18.LOC:Local_timer_interrupts
      6716 ± 10%     -69.9%       2021 ±  8%  interrupts.CPU18.RES:Rescheduling_interrupts
      3337 ± 10%     -66.7%       1110 ± 13%  interrupts.CPU19.CAL:Function_call_interrupts
    731334           -78.4%     158031 ±  4%  interrupts.CPU19.LOC:Local_timer_interrupts
      7403 ±  6%     -72.7%       2024 ± 14%  interrupts.CPU19.RES:Rescheduling_interrupts
    404.00 ± 34%     -80.8%      77.50 ± 31%  interrupts.CPU2.44:IR-PCI-MSI.524296-edge.eth0-TxRx-7
      3530           -69.9%       1063 ± 10%  interrupts.CPU2.CAL:Function_call_interrupts
    732052           -78.4%     158430 ±  3%  interrupts.CPU2.LOC:Local_timer_interrupts
      6823 ±  7%     -72.3%       1888 ±  7%  interrupts.CPU2.RES:Rescheduling_interrupts
      3376 ±  5%     -64.6%       1195 ±  6%  interrupts.CPU20.CAL:Function_call_interrupts
    732255           -78.4%     158361 ±  4%  interrupts.CPU20.LOC:Local_timer_interrupts
      6615 ±  8%     -67.6%       2146 ± 10%  interrupts.CPU20.RES:Rescheduling_interrupts
      3514 ± 10%     -67.8%       1131 ±  6%  interrupts.CPU21.CAL:Function_call_interrupts
    732132           -78.4%     158381 ±  3%  interrupts.CPU21.LOC:Local_timer_interrupts
      6889 ±  6%     -70.9%       2008 ± 16%  interrupts.CPU21.RES:Rescheduling_interrupts
      3519 ± 11%     -65.7%       1208 ±  9%  interrupts.CPU22.CAL:Function_call_interrupts
    732144           -78.4%     158339 ±  4%  interrupts.CPU22.LOC:Local_timer_interrupts
      6662 ±  9%     -68.1%       2124 ± 15%  interrupts.CPU22.RES:Rescheduling_interrupts
      3470 ±  6%     -66.4%       1165 ±  8%  interrupts.CPU23.CAL:Function_call_interrupts
    732050           -78.3%     158538 ±  4%  interrupts.CPU23.LOC:Local_timer_interrupts
      6803 ±  8%     -65.3%       2361 ± 36%  interrupts.CPU23.RES:Rescheduling_interrupts
      3551 ±  6%     -67.2%       1165 ±  3%  interrupts.CPU24.CAL:Function_call_interrupts
    732140           -78.3%     158510 ±  4%  interrupts.CPU24.LOC:Local_timer_interrupts
      6626 ±  9%     -68.1%       2113 ±  9%  interrupts.CPU24.RES:Rescheduling_interrupts
      3697 ±  5%     -68.3%       1171 ± 11%  interrupts.CPU25.CAL:Function_call_interrupts
    732078           -78.3%     158523 ±  3%  interrupts.CPU25.LOC:Local_timer_interrupts
      6899 ±  6%     -69.4%       2110 ±  7%  interrupts.CPU25.RES:Rescheduling_interrupts
    617.50 ± 18%     -77.9%     136.75 ± 43%  interrupts.CPU26.37:IR-PCI-MSI.524289-edge.eth0-TxRx-0
      3624 ±  4%     -68.2%       1152 ±  6%  interrupts.CPU26.CAL:Function_call_interrupts
    732096           -78.4%     158339 ±  4%  interrupts.CPU26.LOC:Local_timer_interrupts
      6622 ±  8%     -69.3%       2035 ± 12%  interrupts.CPU26.RES:Rescheduling_interrupts
      3581 ± 12%     -67.1%       1179 ± 11%  interrupts.CPU27.CAL:Function_call_interrupts
    732145           -78.4%     158379 ±  3%  interrupts.CPU27.LOC:Local_timer_interrupts
      6845 ±  6%     -70.9%       1994 ± 11%  interrupts.CPU27.RES:Rescheduling_interrupts
    306.75 ± 20%     -76.3%      72.75 ± 33%  interrupts.CPU28.38:IR-PCI-MSI.524290-edge.eth0-TxRx-1
      3353 ±  7%     -66.5%       1123 ±  3%  interrupts.CPU28.CAL:Function_call_interrupts
    732259           -78.4%     158392 ±  4%  interrupts.CPU28.LOC:Local_timer_interrupts
      6668 ±  8%     -70.1%       1996 ±  8%  interrupts.CPU28.RES:Rescheduling_interrupts
      3535 ±  8%     -65.6%       1215 ±  8%  interrupts.CPU29.CAL:Function_call_interrupts
    732060           -78.4%     158465 ±  4%  interrupts.CPU29.LOC:Local_timer_interrupts
      6888 ±  6%     -71.2%       1986 ± 11%  interrupts.CPU29.RES:Rescheduling_interrupts
      3596 ±  5%     -66.8%       1194 ± 12%  interrupts.CPU3.CAL:Function_call_interrupts
    732432           -78.4%     158445 ±  4%  interrupts.CPU3.LOC:Local_timer_interrupts
      7473 ±  9%     -73.2%       2004 ± 20%  interrupts.CPU3.RES:Rescheduling_interrupts
    447.00 ± 35%     -86.4%      61.00 ± 37%  interrupts.CPU30.39:IR-PCI-MSI.524291-edge.eth0-TxRx-2
      3491 ±  9%     -66.3%       1176 ± 10%  interrupts.CPU30.CAL:Function_call_interrupts
    732194           -78.4%     158307 ±  4%  interrupts.CPU30.LOC:Local_timer_interrupts
      6652 ±  9%     -69.4%       2033 ±  8%  interrupts.CPU30.RES:Rescheduling_interrupts
      3428 ±  3%     -66.3%       1156 ± 13%  interrupts.CPU31.CAL:Function_call_interrupts
    732419           -78.4%     158523 ±  4%  interrupts.CPU31.LOC:Local_timer_interrupts
      6850 ±  8%     -70.8%       1998 ±  8%  interrupts.CPU31.RES:Rescheduling_interrupts
    392.50 ± 26%     -86.8%      51.75 ±  7%  interrupts.CPU32.40:IR-PCI-MSI.524292-edge.eth0-TxRx-3
      3432 ±  3%     -66.2%       1159 ±  8%  interrupts.CPU32.CAL:Function_call_interrupts
    732243           -78.4%     158318 ±  4%  interrupts.CPU32.LOC:Local_timer_interrupts
      6604 ±  7%     -71.0%       1918 ±  9%  interrupts.CPU32.RES:Rescheduling_interrupts
      3538           -67.8%       1138 ± 17%  interrupts.CPU33.CAL:Function_call_interrupts
    732395           -78.4%     158278 ±  4%  interrupts.CPU33.LOC:Local_timer_interrupts
      6785 ±  9%     -69.5%       2069 ± 19%  interrupts.CPU33.RES:Rescheduling_interrupts
    251.50 ± 13%     -74.0%      65.50 ± 57%  interrupts.CPU34.41:IR-PCI-MSI.524293-edge.eth0-TxRx-4
      3734 ±  7%     -68.7%       1167 ±  8%  interrupts.CPU34.CAL:Function_call_interrupts
    732181           -78.4%     158515 ±  4%  interrupts.CPU34.LOC:Local_timer_interrupts
      6577 ±  8%     -68.5%       2073 ±  6%  interrupts.CPU34.RES:Rescheduling_interrupts
      3546 ±  2%     -67.7%       1147 ±  8%  interrupts.CPU35.CAL:Function_call_interrupts
    732115           -78.4%     158122 ±  3%  interrupts.CPU35.LOC:Local_timer_interrupts
      6784 ±  9%     -67.4%       2209 ±  8%  interrupts.CPU35.RES:Rescheduling_interrupts
    385.75 ± 50%     -79.5%      79.00 ± 38%  interrupts.CPU36.42:IR-PCI-MSI.524294-edge.eth0-TxRx-5
      3620 ±  3%     -66.9%       1198 ±  9%  interrupts.CPU36.CAL:Function_call_interrupts
    732242           -78.4%     158318 ±  4%  interrupts.CPU36.LOC:Local_timer_interrupts
      6537 ±  7%     -69.7%       1983 ±  9%  interrupts.CPU36.RES:Rescheduling_interrupts
      3466 ±  3%     -68.0%       1108 ± 16%  interrupts.CPU37.CAL:Function_call_interrupts
    732076           -78.4%     158298 ±  4%  interrupts.CPU37.LOC:Local_timer_interrupts
      6727 ±  7%     -69.0%       2086 ± 18%  interrupts.CPU37.RES:Rescheduling_interrupts
    297.00 ± 36%     -68.9%      92.50 ± 57%  interrupts.CPU38.43:IR-PCI-MSI.524295-edge.eth0-TxRx-6
      3454 ±  4%     -66.0%       1173 ±  3%  interrupts.CPU38.CAL:Function_call_interrupts
    731970           -78.4%     158222 ±  4%  interrupts.CPU38.LOC:Local_timer_interrupts
      6481 ±  9%     -69.2%       1994 ±  7%  interrupts.CPU38.RES:Rescheduling_interrupts
      3626 ±  6%     -69.1%       1122 ± 16%  interrupts.CPU39.CAL:Function_call_interrupts
    732075           -78.5%     157421 ±  4%  interrupts.CPU39.LOC:Local_timer_interrupts
      6762 ±  7%     -71.2%       1945 ±  3%  interrupts.CPU39.RES:Rescheduling_interrupts
      3578 ±  7%     -67.5%       1161 ± 12%  interrupts.CPU4.CAL:Function_call_interrupts
    732402           -78.4%     158505 ±  4%  interrupts.CPU4.LOC:Local_timer_interrupts
      6713 ±  8%     -68.3%       2126 ± 15%  interrupts.CPU4.RES:Rescheduling_interrupts
      3331 ±  6%     -64.4%       1187 ± 10%  interrupts.CPU5.CAL:Function_call_interrupts
    732138           -78.4%     158438 ±  4%  interrupts.CPU5.LOC:Local_timer_interrupts
      1468 ±113%    +254.6%       5208 ± 33%  interrupts.CPU5.NMI:Non-maskable_interrupts
      1468 ±113%    +254.6%       5208 ± 33%  interrupts.CPU5.PMI:Performance_monitoring_interrupts
      7136 ±  8%     -72.1%       1989 ±  9%  interrupts.CPU5.RES:Rescheduling_interrupts
      3461 ±  5%     -69.5%       1056 ±  3%  interrupts.CPU6.CAL:Function_call_interrupts
    732368           -78.3%     158758 ±  4%  interrupts.CPU6.LOC:Local_timer_interrupts
      6924 ± 13%     -70.1%       2068 ±  5%  interrupts.CPU6.RES:Rescheduling_interrupts
      3401 ± 10%     -66.7%       1132 ± 13%  interrupts.CPU7.CAL:Function_call_interrupts
    732061           -78.4%     158412 ±  4%  interrupts.CPU7.LOC:Local_timer_interrupts
      7002 ±  4%     -65.2%       2437 ± 20%  interrupts.CPU7.RES:Rescheduling_interrupts
      3596 ±  9%     -68.5%       1132 ±  8%  interrupts.CPU8.CAL:Function_call_interrupts
    732166           -78.4%     158317 ±  4%  interrupts.CPU8.LOC:Local_timer_interrupts
      6761 ±  8%     -71.0%       1963 ±  3%  interrupts.CPU8.RES:Rescheduling_interrupts
      3537 ±  4%     -68.5%       1112 ± 11%  interrupts.CPU9.CAL:Function_call_interrupts
    731979           -78.4%     158238 ±  4%  interrupts.CPU9.LOC:Local_timer_interrupts
      5093 ± 34%    -100.0%       0.75 ±110%  interrupts.CPU9.NMI:Non-maskable_interrupts
      5093 ± 34%    -100.0%       0.75 ±110%  interrupts.CPU9.PMI:Performance_monitoring_interrupts
      7101 ±  7%     -70.9%       2063 ± 12%  interrupts.CPU9.RES:Rescheduling_interrupts
  29285268           -78.4%    6335519 ±  4%  interrupts.LOC:Local_timer_interrupts
    273604           -69.4%      83805 ±  9%  interrupts.RES:Rescheduling_interrupts
     90.95           -90.9        0.00        perf-profile.calltrace.cycles-pp.ext4_truncate.ext4_setattr.notify_change.do_truncate.path_openat
     92.48           -83.6        8.89 ±  9%  perf-profile.calltrace.cycles-pp.ext4_setattr.notify_change.do_truncate.path_openat.do_filp_open
     92.67           -82.9        9.72 ±  8%  perf-profile.calltrace.cycles-pp.notify_change.do_truncate.path_openat.do_filp_open.do_sys_open
     93.03           -82.3       10.76 ±  8%  perf-profile.calltrace.cycles-pp.do_truncate.path_openat.do_filp_open.do_sys_open.do_syscall_64
     44.25           -44.2        0.00        perf-profile.calltrace.cycles-pp.ext4_orphan_del.ext4_truncate.ext4_setattr.notify_change.do_truncate
     43.72           -43.7        0.00        perf-profile.calltrace.cycles-pp.ext4_orphan_add.ext4_truncate.ext4_setattr.notify_change.do_truncate
     42.08           -42.1        0.00        perf-profile.calltrace.cycles-pp.__mutex_lock.ext4_orphan_add.ext4_truncate.ext4_setattr.notify_change
     41.71           -41.7        0.00        perf-profile.calltrace.cycles-pp.__mutex_lock.ext4_orphan_del.ext4_truncate.ext4_setattr.notify_change
     40.77           -40.8        0.00        perf-profile.calltrace.cycles-pp.osq_lock.__mutex_lock.ext4_orphan_add.ext4_truncate.ext4_setattr
     40.47           -40.5        0.00        perf-profile.calltrace.cycles-pp.osq_lock.__mutex_lock.ext4_orphan_del.ext4_truncate.ext4_setattr
     95.68            -5.2       90.47        perf-profile.calltrace.cycles-pp.path_openat.do_filp_open.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
     95.72            -5.1       90.62        perf-profile.calltrace.cycles-pp.do_filp_open.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe.creat
     96.07            -3.9       92.17        perf-profile.calltrace.cycles-pp.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe.creat
     96.11            -3.7       92.37        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.creat
     96.13            -3.7       92.43        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.creat
     97.30            -3.1       94.21        perf-profile.calltrace.cycles-pp.creat
      0.76            -0.3        0.42 ± 57%  perf-profile.calltrace.cycles-pp.crc32c_pcl_intel_update.creat
      0.00            +0.6        0.61 ±  4%  perf-profile.calltrace.cycles-pp.__ext4_get_inode_loc.ext4_reserve_inode_write.ext4_mark_inode_dirty.ext4_dirty_inode.__mark_inode_dirty
      0.00            +0.7        0.65 ±  2%  perf-profile.calltrace.cycles-pp.__vfs_getxattr.cap_inode_need_killpriv.security_inode_need_killpriv.dentry_needs_remove_privs.do_truncate
      0.00            +0.7        0.67 ±  8%  perf-profile.calltrace.cycles-pp.selinux_inode_permission.security_inode_permission.link_path_walk.path_openat.do_filp_open
      0.00            +0.7        0.71 ±  2%  perf-profile.calltrace.cycles-pp.cap_inode_need_killpriv.security_inode_need_killpriv.dentry_needs_remove_privs.do_truncate.path_openat
      0.00            +0.7        0.72 ±  7%  perf-profile.calltrace.cycles-pp.security_inode_permission.link_path_walk.path_openat.do_filp_open.do_sys_open
      0.00            +0.8        0.75        perf-profile.calltrace.cycles-pp.security_inode_need_killpriv.dentry_needs_remove_privs.do_truncate.path_openat.do_filp_open
      0.00            +0.8        0.77 ±  4%  perf-profile.calltrace.cycles-pp.lockref_put_return.dput.path_openat.do_filp_open.do_sys_open
      0.00            +0.8        0.79        perf-profile.calltrace.cycles-pp.dentry_needs_remove_privs.do_truncate.path_openat.do_filp_open.do_sys_open
      0.00            +0.8        0.81 ±  7%  perf-profile.calltrace.cycles-pp.getname_flags.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe.creat
      0.00            +0.8        0.82 ± 15%  perf-profile.calltrace.cycles-pp.try_to_wake_up.wake_up_q.rwsem_wake.call_rwsem_wake.up_write
      0.00            +0.9        0.86 ±  7%  perf-profile.calltrace.cycles-pp.ext4_reserve_inode_write.ext4_mark_inode_dirty.ext4_dirty_inode.__mark_inode_dirty.ext4_setattr
      0.00            +0.9        0.88 ± 14%  perf-profile.calltrace.cycles-pp.wake_up_q.rwsem_wake.call_rwsem_wake.up_write.path_openat
      0.00            +0.9        0.89 ±  5%  perf-profile.calltrace.cycles-pp.lockref_get_not_zero.dget_parent.fscrypt_file_open.ext4_file_open.do_dentry_open
      0.00            +0.9        0.91 ±  5%  perf-profile.calltrace.cycles-pp.dget_parent.fscrypt_file_open.ext4_file_open.do_dentry_open.path_openat
      0.00            +0.9        0.91 ±  6%  perf-profile.calltrace.cycles-pp.rwsem_spin_on_owner.rwsem_down_write_failed.call_rwsem_down_write_failed.down_write.path_openat
      0.00            +0.9        0.94 ±  7%  perf-profile.calltrace.cycles-pp.lockref_get_not_dead.legitimize_path.unlazy_walk.complete_walk.path_openat
      0.00            +1.0        1.04 ±  3%  perf-profile.calltrace.cycles-pp.dput.path_openat.do_filp_open.do_sys_open.do_syscall_64
      0.00            +1.1        1.13 ±  8%  perf-profile.calltrace.cycles-pp.legitimize_path.unlazy_walk.complete_walk.path_openat.do_filp_open
      0.00            +1.2        1.17 ±  7%  perf-profile.calltrace.cycles-pp.unlazy_walk.complete_walk.path_openat.do_filp_open.do_sys_open
      0.00            +1.2        1.17 ±  6%  perf-profile.calltrace.cycles-pp.__alloc_file.alloc_empty_file.path_openat.do_filp_open.do_sys_open
      0.00            +1.2        1.20 ± 11%  perf-profile.calltrace.cycles-pp.jbd2_journal_stop.__ext4_journal_stop.__mark_inode_dirty.ext4_setattr.notify_change
      0.00            +1.2        1.22 ±  7%  perf-profile.calltrace.cycles-pp.complete_walk.path_openat.do_filp_open.do_sys_open.do_syscall_64
      0.00            +1.2        1.23 ±  7%  perf-profile.calltrace.cycles-pp.alloc_empty_file.path_openat.do_filp_open.do_sys_open.do_syscall_64
      0.00            +1.3        1.25 ± 11%  perf-profile.calltrace.cycles-pp.__ext4_journal_stop.__mark_inode_dirty.ext4_setattr.notify_change.do_truncate
      0.00            +1.4        1.37 ±  4%  perf-profile.calltrace.cycles-pp.__fput.task_work_run.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +1.5        1.53 ± 13%  perf-profile.calltrace.cycles-pp.fscrypt_file_open.ext4_file_open.do_dentry_open.path_openat.do_filp_open
      0.00            +1.6        1.60 ±  4%  perf-profile.calltrace.cycles-pp.ext4_do_update_inode.ext4_mark_iloc_dirty.ext4_mark_inode_dirty.ext4_dirty_inode.__mark_inode_dirty
      0.00            +1.6        1.63 ±  7%  perf-profile.calltrace.cycles-pp.rwsem_wake.call_rwsem_wake.up_write.path_openat.do_filp_open
      0.00            +1.7        1.65 ±  7%  perf-profile.calltrace.cycles-pp.call_rwsem_wake.up_write.path_openat.do_filp_open.do_sys_open
      0.00            +1.7        1.72 ±  5%  perf-profile.calltrace.cycles-pp.ext4_mark_iloc_dirty.ext4_mark_inode_dirty.ext4_dirty_inode.__mark_inode_dirty.ext4_setattr
      0.00            +1.7        1.73 ± 11%  perf-profile.calltrace.cycles-pp.ext4_file_open.do_dentry_open.path_openat.do_filp_open.do_sys_open
      0.00            +1.8        1.82 ±  5%  perf-profile.calltrace.cycles-pp.link_path_walk.path_openat.do_filp_open.do_sys_open.do_syscall_64
      0.00            +1.8        1.83 ±  3%  perf-profile.calltrace.cycles-pp.task_work_run.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.close
      0.00            +2.0        1.98 ±  7%  perf-profile.calltrace.cycles-pp.up_write.path_openat.do_filp_open.do_sys_open.do_syscall_64
      0.00            +2.0        1.99 ±  3%  perf-profile.calltrace.cycles-pp.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.close
      0.62 ±  4%      +2.0        2.66 ±  3%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.close
      0.63 ±  4%      +2.1        2.71 ±  3%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.close
      0.54 ±  4%      +2.1        2.66 ±  5%  perf-profile.calltrace.cycles-pp.ext4_mark_inode_dirty.ext4_dirty_inode.__mark_inode_dirty.ext4_setattr.notify_change
      0.00            +2.7        2.65 ±  6%  perf-profile.calltrace.cycles-pp.do_dentry_open.path_openat.do_filp_open.do_sys_open.do_syscall_64
      0.00            +2.7        2.69 ± 14%  perf-profile.calltrace.cycles-pp.start_this_handle.jbd2__journal_start.ext4_dirty_inode.__mark_inode_dirty.ext4_setattr
      0.83 ±  5%      +2.9        3.69 ±  3%  perf-profile.calltrace.cycles-pp.close
      0.00            +3.2        3.18 ± 12%  perf-profile.calltrace.cycles-pp.jbd2__journal_start.ext4_dirty_inode.__mark_inode_dirty.ext4_setattr.notify_change
      0.99            +5.4        6.38 ±  9%  perf-profile.calltrace.cycles-pp.ext4_dirty_inode.__mark_inode_dirty.ext4_setattr.notify_change.do_truncate
      1.09            +6.6        7.72 ±  9%  perf-profile.calltrace.cycles-pp.__mark_inode_dirty.ext4_setattr.notify_change.do_truncate.path_openat
      0.60 ±  5%     +63.4       64.03        perf-profile.calltrace.cycles-pp.osq_lock.rwsem_down_write_failed.call_rwsem_down_write_failed.down_write.path_openat
      0.67 ±  4%     +65.6       66.25        perf-profile.calltrace.cycles-pp.rwsem_down_write_failed.call_rwsem_down_write_failed.down_write.path_openat.do_filp_open
      0.67 ±  4%     +65.6       66.27        perf-profile.calltrace.cycles-pp.call_rwsem_down_write_failed.down_write.path_openat.do_filp_open.do_sys_open
      0.75 ±  4%     +66.2       66.91        perf-profile.calltrace.cycles-pp.down_write.path_openat.do_filp_open.do_sys_open.do_syscall_64
     90.96           -91.0        0.00        perf-profile.children.cycles-pp.ext4_truncate
     83.85           -83.8        0.00        perf-profile.children.cycles-pp.__mutex_lock
     92.49           -83.6        8.91 ±  9%  perf-profile.children.cycles-pp.ext4_setattr
     92.68           -82.9        9.74 ±  8%  perf-profile.children.cycles-pp.notify_change
     93.03           -82.3       10.77 ±  8%  perf-profile.children.cycles-pp.do_truncate
     44.28           -44.3        0.00        perf-profile.children.cycles-pp.ext4_orphan_del
     43.76           -43.8        0.00        perf-profile.children.cycles-pp.ext4_orphan_add
     81.93           -17.8       64.14        perf-profile.children.cycles-pp.osq_lock
     95.69            -5.2       90.51        perf-profile.children.cycles-pp.path_openat
     95.72            -5.1       90.64        perf-profile.children.cycles-pp.do_filp_open
     96.08            -3.9       92.19        perf-profile.children.cycles-pp.do_sys_open
     97.32            -3.0       94.29        perf-profile.children.cycles-pp.creat
     96.88            -1.6       95.26        perf-profile.children.cycles-pp.do_syscall_64
     96.91            -1.5       95.37        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
      2.71            -1.0        1.73 ±  5%  perf-profile.children.cycles-pp.ext4_mark_iloc_dirty
      1.54 ±  3%      -0.7        0.87 ±  7%  perf-profile.children.cycles-pp.ext4_reserve_inode_write
      2.13 ±  2%      -0.5        1.62 ±  5%  perf-profile.children.cycles-pp.ext4_do_update_inode
      0.78            -0.2        0.56 ±  5%  perf-profile.children.cycles-pp.crc32c_pcl_intel_update
      0.79 ±  7%      -0.2        0.62 ±  4%  perf-profile.children.cycles-pp.__ext4_get_inode_loc
      0.19 ±  7%      -0.1        0.05 ± 58%  perf-profile.children.cycles-pp.__brelse
      0.43 ± 11%      -0.1        0.36        perf-profile.children.cycles-pp.ext4_inode_csum_set
      0.47 ±  3%      -0.1        0.41 ±  5%  perf-profile.children.cycles-pp.__getblk_gfp
      0.26 ±  7%      -0.1        0.20 ± 16%  perf-profile.children.cycles-pp.__ext4_journal_get_write_access
      0.39 ±  4%      -0.0        0.34 ±  6%  perf-profile.children.cycles-pp.__find_get_block
      0.47 ±  3%      -0.0        0.43 ±  3%  perf-profile.children.cycles-pp.hrtimer_interrupt
      0.14 ±  9%      -0.0        0.11 ±  9%  perf-profile.children.cycles-pp.jbd2_journal_get_write_access
      0.07 ±  7%      -0.0        0.04 ± 58%  perf-profile.children.cycles-pp.schedule
      0.10 ±  4%      -0.0        0.08 ±  5%  perf-profile.children.cycles-pp.jbd2_write_access_granted
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.lockref_get
      0.00            +0.1        0.05 ±  9%  perf-profile.children.cycles-pp.security_file_free
      0.00            +0.1        0.06 ±  9%  perf-profile.children.cycles-pp.inode_has_perm
      0.00            +0.1        0.06 ±  9%  perf-profile.children.cycles-pp.mntput_no_expire
      0.00            +0.1        0.06 ± 15%  perf-profile.children.cycles-pp.__fd_install
      0.00            +0.1        0.06 ± 11%  perf-profile.children.cycles-pp.setattr_prepare
      0.00            +0.1        0.06 ± 11%  perf-profile.children.cycles-pp.rcu_cblist_dequeue
      0.00            +0.1        0.06 ± 11%  perf-profile.children.cycles-pp.__x64_sys_creat
      0.14 ±  5%      +0.1        0.20 ±  9%  perf-profile.children.cycles-pp.rcu_all_qs
      0.00            +0.1        0.06 ± 17%  perf-profile.children.cycles-pp.__wake_up_common_lock
      0.09 ±  4%      +0.1        0.15 ±  2%  perf-profile.children.cycles-pp.do_unlinkat
      0.00            +0.1        0.07 ±  7%  perf-profile.children.cycles-pp.__check_heap_object
      0.00            +0.1        0.07 ± 11%  perf-profile.children.cycles-pp.setattr_copy
      0.00            +0.1        0.07 ± 22%  perf-profile.children.cycles-pp.__lookup_mnt
      0.00            +0.1        0.07 ± 20%  perf-profile.children.cycles-pp.__sb_end_write
      0.00            +0.1        0.08 ± 11%  perf-profile.children.cycles-pp.selinux_task_getsecid
      0.35 ±  4%      +0.1        0.43 ±  7%  perf-profile.children.cycles-pp.jbd2_journal_dirty_metadata
      0.09 ±  4%      +0.1        0.17 ±  4%  perf-profile.children.cycles-pp.unlink
      0.00            +0.1        0.08 ± 21%  perf-profile.children.cycles-pp.percpu_counter_add_batch
      0.00            +0.1        0.09 ±  9%  perf-profile.children.cycles-pp.current_time
      0.01 ±173%      +0.1        0.10 ±  4%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.00            +0.1        0.10 ±  5%  perf-profile.children.cycles-pp.selinux_file_alloc_security
      0.00            +0.1        0.10 ± 13%  perf-profile.children.cycles-pp.security_task_getsecid
      0.00            +0.1        0.10 ±  8%  perf-profile.children.cycles-pp.__close_fd
      0.00            +0.1        0.10 ± 12%  perf-profile.children.cycles-pp.creat_clo
      0.07 ±  6%      +0.1        0.17 ±  8%  perf-profile.children.cycles-pp.ext4_discard_preallocations
      0.00            +0.1        0.10 ± 21%  perf-profile.children.cycles-pp.set_root
      0.00            +0.1        0.10 ±  8%  perf-profile.children.cycles-pp.__slab_free
      0.00            +0.1        0.11 ±  7%  perf-profile.children.cycles-pp.ext4_break_layouts
      0.11 ±  4%      +0.1        0.22 ±  7%  perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
      0.00            +0.1        0.11 ± 14%  perf-profile.children.cycles-pp.__legitimize_mnt
      0.00            +0.1        0.12 ± 12%  perf-profile.children.cycles-pp.generic_permission
      0.00            +0.1        0.12 ±  8%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.00            +0.1        0.12 ± 13%  perf-profile.children.cycles-pp.__fsnotify_parent
      0.00            +0.1        0.12 ± 16%  perf-profile.children.cycles-pp.file_free_rcu
      0.00            +0.1        0.13 ± 15%  perf-profile.children.cycles-pp.down_read
      0.03 ±102%      +0.1        0.16 ±  6%  perf-profile.children.cycles-pp.memset_erms
      0.06 ±  9%      +0.1        0.19 ± 10%  perf-profile.children.cycles-pp.path_init
      0.00            +0.1        0.14 ±  9%  perf-profile.children.cycles-pp.avc_has_perm
      0.00            +0.1        0.14 ±  3%  perf-profile.children.cycles-pp.__call_rcu
      0.00            +0.1        0.14 ±  5%  perf-profile.children.cycles-pp.task_work_add
      0.04 ± 59%      +0.1        0.19 ± 16%  perf-profile.children.cycles-pp.__sb_start_write
      0.01 ±173%      +0.2        0.17 ±  8%  perf-profile.children.cycles-pp.ima_file_check
      0.03 ±100%      +0.2        0.19 ±  7%  perf-profile.children.cycles-pp.xattr_resolve_name
      0.04 ±103%      +0.2        0.20 ± 13%  perf-profile.children.cycles-pp.__mnt_want_write
      0.05 ±  8%      +0.2        0.22 ± 10%  perf-profile.children.cycles-pp.fput_many
      0.03 ±100%      +0.2        0.20 ± 11%  perf-profile.children.cycles-pp.inode_security_rcu
      0.01 ±173%      +0.2        0.19 ±  6%  perf-profile.children.cycles-pp.__d_lookup_rcu
      0.06            +0.2        0.23 ±  7%  perf-profile.children.cycles-pp.__alloc_fd
      0.23 ±  7%      +0.2        0.41 ±  8%  perf-profile.children.cycles-pp._cond_resched
      0.04 ± 57%      +0.2        0.23 ± 10%  perf-profile.children.cycles-pp.terminate_walk
      0.21 ±  6%      +0.2        0.40 ±  3%  perf-profile.children.cycles-pp.osq_unlock
      0.00            +0.2        0.19 ±  5%  perf-profile.children.cycles-pp._raw_spin_trylock
      0.00            +0.2        0.19 ± 19%  perf-profile.children.cycles-pp.__virt_addr_valid
      0.17 ±  2%      +0.2        0.37 ±  2%  perf-profile.children.cycles-pp.ext4_xattr_get
      0.00            +0.2        0.20 ± 44%  perf-profile.children.cycles-pp.__follow_mount_rcu
      0.10 ±  5%      +0.2        0.30 ±  4%  perf-profile.children.cycles-pp.wake_q_add
      0.34 ±  3%      +0.2        0.54 ± 14%  perf-profile.children.cycles-pp._raw_spin_lock
      0.00            +0.2        0.21 ±  3%  perf-profile.children.cycles-pp.new_slab
      0.06 ± 14%      +0.2        0.27 ±  3%  perf-profile.children.cycles-pp.selinux_inode_setattr
      0.07 ±  5%      +0.2        0.29 ±  3%  perf-profile.children.cycles-pp.selinux_file_open
      0.08 ±  5%      +0.2        0.30 ±  9%  perf-profile.children.cycles-pp.truncate_pagecache
      0.07 ± 13%      +0.2        0.29 ±  5%  perf-profile.children.cycles-pp.avc_has_perm_noaudit
      0.08 ± 11%      +0.2        0.30 ±  8%  perf-profile.children.cycles-pp.fsnotify
      0.06 ± 11%      +0.2        0.29 ±  9%  perf-profile.children.cycles-pp.security_file_alloc
      0.07 ± 13%      +0.2        0.30 ±  2%  perf-profile.children.cycles-pp.security_inode_setattr
      0.07 ± 13%      +0.2        0.30 ±  7%  perf-profile.children.cycles-pp.__inode_security_revalidate
      0.07 ± 11%      +0.2        0.32 ±  3%  perf-profile.children.cycles-pp.inode_permission
      0.00            +0.2        0.25 ±  2%  perf-profile.children.cycles-pp.___slab_alloc
      0.10 ± 24%      +0.3        0.35 ± 12%  perf-profile.children.cycles-pp.mnt_want_write
      0.00            +0.3        0.26        perf-profile.children.cycles-pp.__slab_alloc
      0.09 ± 13%      +0.3        0.34 ±  5%  perf-profile.children.cycles-pp.filp_close
      0.07 ± 12%      +0.3        0.34 ± 34%  perf-profile.children.cycles-pp.ret_from_fork
      0.07 ± 12%      +0.3        0.34 ± 34%  perf-profile.children.cycles-pp.kthread
      0.09 ±  8%      +0.3        0.38 ±  2%  perf-profile.children.cycles-pp.security_file_open
      0.08 ± 10%      +0.3        0.38 ±  5%  perf-profile.children.cycles-pp.unmap_mapping_pages
      0.07 ± 11%      +0.3        0.37 ± 13%  perf-profile.children.cycles-pp.__check_object_size
      0.00            +0.3        0.32 ± 34%  perf-profile.children.cycles-pp.run_ksoftirqd
      0.00            +0.3        0.33 ± 35%  perf-profile.children.cycles-pp.smpboot_thread_fn
      0.10 ± 13%      +0.4        0.45 ± 11%  perf-profile.children.cycles-pp.__softirqentry_text_start
      0.12 ±  3%      +0.4        0.48 ±  7%  perf-profile.children.cycles-pp.ext4_release_file
      0.06 ± 14%      +0.4        0.41 ± 10%  perf-profile.children.cycles-pp.rcu_core
      0.24 ±  6%      +0.4        0.61 ± 13%  perf-profile.children.cycles-pp.__might_sleep
      0.30 ±  7%      +0.4        0.67 ±  3%  perf-profile.children.cycles-pp.___might_sleep
      0.10 ±  8%      +0.4        0.49 ±  2%  perf-profile.children.cycles-pp.kmem_cache_free
      0.11 ± 10%      +0.4        0.49 ± 19%  perf-profile.children.cycles-pp.lookup_fast
      0.10 ±  4%      +0.4        0.49 ± 11%  perf-profile.children.cycles-pp.strncpy_from_user
      0.11 ± 11%      +0.4        0.51 ±  6%  perf-profile.children.cycles-pp.may_open
      0.12 ±  8%      +0.4        0.53 ±  4%  perf-profile.children.cycles-pp.__x64_sys_close
      0.25            +0.4        0.66 ±  2%  perf-profile.children.cycles-pp.__vfs_getxattr
      0.18 ±  3%      +0.4        0.60 ± 13%  perf-profile.children.cycles-pp._raw_read_lock
      2.26 ±  2%      +0.4        2.68 ±  5%  perf-profile.children.cycles-pp.ext4_mark_inode_dirty
      0.13 ±  7%      +0.5        0.58 ± 16%  perf-profile.children.cycles-pp.walk_component
      0.26 ±  3%      +0.5        0.72 ±  2%  perf-profile.children.cycles-pp.cap_inode_need_killpriv
      0.27 ±  3%      +0.5        0.75        perf-profile.children.cycles-pp.security_inode_need_killpriv
      0.00            +0.5        0.49 ± 16%  perf-profile.children.cycles-pp.__d_lookup
      0.33 ±  5%      +0.5        0.82 ± 15%  perf-profile.children.cycles-pp.try_to_wake_up
      0.00            +0.5        0.51 ± 15%  perf-profile.children.cycles-pp.d_lookup
      0.28            +0.5        0.82 ±  2%  perf-profile.children.cycles-pp.dentry_needs_remove_privs
      0.34 ±  6%      +0.5        0.89 ± 14%  perf-profile.children.cycles-pp.wake_up_q
      0.10 ± 15%      +0.6        0.70 ± 18%  perf-profile.children.cycles-pp.add_transaction_credits
      0.27 ±  4%      +0.6        0.90 ± 12%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.17 ±  3%      +0.6        0.81 ±  7%  perf-profile.children.cycles-pp.getname_flags
      0.15 ± 14%      +0.7        0.83 ± 13%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      0.17 ±  7%      +0.7        0.85 ±  3%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.19 ±  3%      +0.8        0.96 ±  6%  perf-profile.children.cycles-pp.entry_SYSCALL_64
      0.23 ±  5%      +0.8        1.02 ±  7%  perf-profile.children.cycles-pp.selinux_inode_permission
      0.24 ±  5%      +0.9        1.09 ±  7%  perf-profile.children.cycles-pp.security_inode_permission
      0.06 ±  7%      +0.9        0.92 ±  7%  perf-profile.children.cycles-pp.rwsem_spin_on_owner
      0.00            +0.9        0.89 ±  5%  perf-profile.children.cycles-pp.lockref_get_not_zero
      0.04 ± 58%      +0.9        0.94 ±  7%  perf-profile.children.cycles-pp.lockref_get_not_dead
      0.24 ±  6%      +0.9        1.14 ±  5%  perf-profile.children.cycles-pp.kmem_cache_alloc
      0.00            +0.9        0.91 ±  5%  perf-profile.children.cycles-pp.dget_parent
      0.26 ±  6%      +1.0        1.21 ± 11%  perf-profile.children.cycles-pp.jbd2_journal_stop
      0.29 ±  4%      +1.0        1.26 ± 11%  perf-profile.children.cycles-pp.__ext4_journal_stop
      0.19 ±  7%      +1.0        1.18 ±  6%  perf-profile.children.cycles-pp.__alloc_file
      0.21 ±  5%      +1.0        1.24 ±  7%  perf-profile.children.cycles-pp.alloc_empty_file
      0.08 ±  5%      +1.0        1.13 ±  8%  perf-profile.children.cycles-pp.legitimize_path
      0.33 ±  3%      +1.1        1.40 ±  4%  perf-profile.children.cycles-pp.__fput
      0.09 ± 11%      +1.1        1.19 ±  7%  perf-profile.children.cycles-pp.unlazy_walk
      0.10 ± 12%      +1.1        1.22 ±  7%  perf-profile.children.cycles-pp.complete_walk
      0.00            +1.3        1.29 ±  4%  perf-profile.children.cycles-pp.lockref_put_return
      0.43 ±  3%      +1.4        1.84 ±  3%  perf-profile.children.cycles-pp.task_work_run
      0.41 ±  5%      +1.4        1.84 ±  4%  perf-profile.children.cycles-pp.link_path_walk
      0.05            +1.5        1.53 ± 13%  perf-profile.children.cycles-pp.fscrypt_file_open
      0.21 ±  4%      +1.5        1.74 ± 11%  perf-profile.children.cycles-pp.ext4_file_open
      0.12 ±  3%      +1.5        1.65 ±  7%  perf-profile.children.cycles-pp.rwsem_wake
      0.12 ±  4%      +1.5        1.66 ±  7%  perf-profile.children.cycles-pp.call_rwsem_wake
      0.48 ±  2%      +1.5        2.01 ±  3%  perf-profile.children.cycles-pp.exit_to_usermode_loop
      0.12 ±  5%      +1.8        1.90 ±  5%  perf-profile.children.cycles-pp.dput
      0.33 ±  7%      +1.9        2.25 ±  6%  perf-profile.children.cycles-pp.up_write
      0.58 ±  6%      +2.1        2.69 ± 14%  perf-profile.children.cycles-pp.start_this_handle
      0.45 ±  2%      +2.2        2.66 ±  6%  perf-profile.children.cycles-pp.do_dentry_open
      0.69 ±  5%      +2.5        3.18 ± 12%  perf-profile.children.cycles-pp.jbd2__journal_start
      0.85 ±  5%      +2.9        3.75 ±  3%  perf-profile.children.cycles-pp.close
      0.99            +5.4        6.39 ±  9%  perf-profile.children.cycles-pp.ext4_dirty_inode
      1.09            +6.6        7.72 ±  9%  perf-profile.children.cycles-pp.__mark_inode_dirty
      0.67 ±  4%     +65.7       66.34        perf-profile.children.cycles-pp.rwsem_down_write_failed
      0.67 ±  4%     +65.7       66.34        perf-profile.children.cycles-pp.call_rwsem_down_write_failed
      0.87 ±  4%     +66.7       67.56        perf-profile.children.cycles-pp.down_write
     81.35           -17.7       63.68        perf-profile.self.cycles-pp.osq_lock
      0.53 ±  3%      -0.5        0.03 ±100%  perf-profile.self.cycles-pp.ext4_reserve_inode_write
      0.57            -0.5        0.11 ±  4%  perf-profile.self.cycles-pp.ext4_mark_iloc_dirty
      0.90 ±  3%      -0.4        0.52 ±  7%  perf-profile.self.cycles-pp.ext4_do_update_inode
      0.18 ±  6%      -0.1        0.05 ± 58%  perf-profile.self.cycles-pp.__brelse
      0.65 ±  2%      -0.1        0.52 ±  4%  perf-profile.self.cycles-pp.crc32c_pcl_intel_update
      0.21 ± 24%      -0.1        0.13 ± 10%  perf-profile.self.cycles-pp.__ext4_get_inode_loc
      0.12 ± 27%      -0.1        0.07 ±  7%  perf-profile.self.cycles-pp.ext4_inode_csum_set
      0.10 ±  7%      -0.0        0.08 ±  6%  perf-profile.self.cycles-pp.jbd2_write_access_granted
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.lockref_get
      0.00            +0.1        0.05 ±  8%  perf-profile.self.cycles-pp.__vfs_getxattr
      0.00            +0.1        0.05 ±  8%  perf-profile.self.cycles-pp.mntput_no_expire
      0.11 ±  4%      +0.1        0.16 ± 11%  perf-profile.self.cycles-pp.rcu_all_qs
      0.00            +0.1        0.05 ±  9%  perf-profile.self.cycles-pp.inode_security_rcu
      0.00            +0.1        0.06 ±  9%  perf-profile.self.cycles-pp.inode_has_perm
      0.00            +0.1        0.06 ± 15%  perf-profile.self.cycles-pp.__fd_install
      0.00            +0.1        0.06 ±  7%  perf-profile.self.cycles-pp.try_to_wake_up
      0.00            +0.1        0.06 ±  7%  perf-profile.self.cycles-pp.may_open
      0.00            +0.1        0.06 ± 14%  perf-profile.self.cycles-pp.d_lookup
      0.00            +0.1        0.06 ± 11%  perf-profile.self.cycles-pp.security_file_open
      0.00            +0.1        0.06        perf-profile.self.cycles-pp.filp_close
      0.00            +0.1        0.06 ± 11%  perf-profile.self.cycles-pp.ext4_break_layouts
      0.00            +0.1        0.06 ± 11%  perf-profile.self.cycles-pp.rcu_cblist_dequeue
      0.00            +0.1        0.06 ± 11%  perf-profile.self.cycles-pp.do_truncate
      0.00            +0.1        0.06 ±  6%  perf-profile.self.cycles-pp.cap_inode_need_killpriv
      0.00            +0.1        0.06 ±  6%  perf-profile.self.cycles-pp.__check_heap_object
      0.00            +0.1        0.07 ±  7%  perf-profile.self.cycles-pp.wake_up_q
      0.00            +0.1        0.07 ± 17%  perf-profile.self.cycles-pp.creat_clo
      0.00            +0.1        0.07 ±  6%  perf-profile.self.cycles-pp.path_init
      0.00            +0.1        0.07 ± 10%  perf-profile.self.cycles-pp.selinux_task_getsecid
      0.00            +0.1        0.07 ± 26%  perf-profile.self.cycles-pp.__lookup_mnt
      0.00            +0.1        0.07 ± 11%  perf-profile.self.cycles-pp.__x64_sys_close
      0.00            +0.1        0.07 ± 15%  perf-profile.self.cycles-pp.__mark_inode_dirty
      0.35 ±  4%      +0.1        0.42 ±  8%  perf-profile.self.cycles-pp.jbd2_journal_dirty_metadata
      0.00            +0.1        0.07 ± 11%  perf-profile.self.cycles-pp.security_inode_permission
      0.00            +0.1        0.07 ± 20%  perf-profile.self.cycles-pp.__sb_end_write
      0.00            +0.1        0.08 ± 14%  perf-profile.self.cycles-pp.getname_flags
      0.00            +0.1        0.08 ± 19%  perf-profile.self.cycles-pp.percpu_counter_add_batch
      0.00            +0.1        0.08 ±  5%  perf-profile.self.cycles-pp.unmap_mapping_pages
      0.00            +0.1        0.08 ±  8%  perf-profile.self.cycles-pp.do_filp_open
      0.00            +0.1        0.08 ±  8%  perf-profile.self.cycles-pp.close
      0.00            +0.1        0.09 ±  4%  perf-profile.self.cycles-pp.walk_component
      0.00            +0.1        0.09 ±  4%  perf-profile.self.cycles-pp.__call_rcu
      0.00            +0.1        0.09 ± 21%  perf-profile.self.cycles-pp.__sb_start_write
      0.11 ± 14%      +0.1        0.20 ±  7%  perf-profile.self.cycles-pp._cond_resched
      0.00            +0.1        0.09 ±  7%  perf-profile.self.cycles-pp.selinux_file_alloc_security
      0.00            +0.1        0.09 ±  4%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.00            +0.1        0.09 ± 15%  perf-profile.self.cycles-pp.selinux_inode_setattr
      0.00            +0.1        0.09 ± 23%  perf-profile.self.cycles-pp.set_root
      0.00            +0.1        0.10 ± 17%  perf-profile.self.cycles-pp.ext4_discard_preallocations
      0.00            +0.1        0.10        perf-profile.self.cycles-pp.exit_to_usermode_loop
      0.11 ±  6%      +0.1        0.21 ±  5%  perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
      0.00            +0.1        0.10 ±  8%  perf-profile.self.cycles-pp.__slab_free
      0.00            +0.1        0.10 ± 10%  perf-profile.self.cycles-pp.creat
      0.00            +0.1        0.11 ±  4%  perf-profile.self.cycles-pp.__check_object_size
      0.00            +0.1        0.11 ±  6%  perf-profile.self.cycles-pp.lookup_fast
      0.00            +0.1        0.11 ± 17%  perf-profile.self.cycles-pp.__legitimize_mnt
      0.00            +0.1        0.11 ± 19%  perf-profile.self.cycles-pp.ext4_release_file
      0.00            +0.1        0.11 ± 13%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.00            +0.1        0.11 ±  7%  perf-profile.self.cycles-pp.__alloc_fd
      0.00            +0.1        0.11 ±  4%  perf-profile.self.cycles-pp.task_work_add
      0.00            +0.1        0.11 ±  7%  perf-profile.self.cycles-pp.selinux_file_open
      0.00            +0.1        0.12 ±  9%  perf-profile.self.cycles-pp.strncpy_from_user
      0.00            +0.1        0.12 ± 12%  perf-profile.self.cycles-pp.generic_permission
      0.00            +0.1        0.12 ± 12%  perf-profile.self.cycles-pp.__fsnotify_parent
      0.00            +0.1        0.12 ± 16%  perf-profile.self.cycles-pp.file_free_rcu
      0.00            +0.1        0.12 ± 65%  perf-profile.self.cycles-pp.__follow_mount_rcu
      0.00            +0.1        0.12 ±  6%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.03 ±100%      +0.1        0.15 ±  7%  perf-profile.self.cycles-pp.memset_erms
      0.20 ±  4%      +0.1        0.32 ±  6%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.00            +0.1        0.13 ± 10%  perf-profile.self.cycles-pp.task_work_run
      0.00            +0.1        0.13 ± 11%  perf-profile.self.cycles-pp.__inode_security_revalidate
      0.00            +0.1        0.14 ± 11%  perf-profile.self.cycles-pp.avc_has_perm
      0.04 ± 57%      +0.1        0.18 ± 12%  perf-profile.self.cycles-pp.do_sys_open
      0.15 ±  7%      +0.1        0.30 ±  6%  perf-profile.self.cycles-pp.ext4_setattr
      0.04 ± 57%      +0.1        0.18 ±  4%  perf-profile.self.cycles-pp.inode_permission
      0.00            +0.1        0.15        perf-profile.self.cycles-pp.new_slab
      0.00            +0.2        0.16 ± 45%  perf-profile.self.cycles-pp.dput
      0.04 ±102%      +0.2        0.19 ± 14%  perf-profile.self.cycles-pp.__mnt_want_write
      0.03 ±100%      +0.2        0.19 ±  7%  perf-profile.self.cycles-pp.xattr_resolve_name
      0.05 ±  8%      +0.2        0.22 ±  8%  perf-profile.self.cycles-pp.do_dentry_open
      0.01 ±173%      +0.2        0.18 ±  4%  perf-profile.self.cycles-pp.__d_lookup_rcu
      0.01 ±173%      +0.2        0.18 ± 13%  perf-profile.self.cycles-pp.__fput
      0.00            +0.2        0.18 ± 11%  perf-profile.self.cycles-pp.rwsem_wake
      0.00            +0.2        0.19 ±  4%  perf-profile.self.cycles-pp._raw_spin_trylock
      0.21 ±  6%      +0.2        0.40 ±  3%  perf-profile.self.cycles-pp.osq_unlock
      0.00            +0.2        0.19 ± 19%  perf-profile.self.cycles-pp.__virt_addr_valid
      0.06 ±  9%      +0.2        0.25 ±  9%  perf-profile.self.cycles-pp.do_syscall_64
      0.06 ± 13%      +0.2        0.26 ±  4%  perf-profile.self.cycles-pp.notify_change
      0.10 ±  5%      +0.2        0.30 ±  4%  perf-profile.self.cycles-pp.wake_q_add
      0.07 ± 13%      +0.2        0.28 ±  7%  perf-profile.self.cycles-pp.avc_has_perm_noaudit
      0.08 ± 11%      +0.2        0.30 ±  9%  perf-profile.self.cycles-pp.fsnotify
      0.07 ±  7%      +0.2        0.31 ± 16%  perf-profile.self.cycles-pp.link_path_walk
      0.00            +0.3        0.26        perf-profile.self.cycles-pp.jbd2__journal_start
      0.08 ± 10%      +0.3        0.35 ±  2%  perf-profile.self.cycles-pp.kmem_cache_free
      0.23 ±  7%      +0.3        0.55 ± 16%  perf-profile.self.cycles-pp.__might_sleep
      0.06 ± 13%      +0.3        0.39 ±  6%  perf-profile.self.cycles-pp.__alloc_file
      0.29 ±  6%      +0.4        0.65 ±  5%  perf-profile.self.cycles-pp.___might_sleep
      0.21 ± 11%      +0.4        0.58 ±  4%  perf-profile.self.cycles-pp.up_write
      0.00            +0.4        0.40 ± 16%  perf-profile.self.cycles-pp.__d_lookup
      0.11 ±  7%      +0.4        0.52 ±  7%  perf-profile.self.cycles-pp.selinux_inode_permission
      0.11 ±  6%      +0.4        0.52 ±  7%  perf-profile.self.cycles-pp.kmem_cache_alloc
      0.18 ±  3%      +0.4        0.60 ± 13%  perf-profile.self.cycles-pp._raw_read_lock
      0.13 ± 11%      +0.6        0.69 ± 18%  perf-profile.self.cycles-pp.path_openat
      0.10 ± 15%      +0.6        0.70 ± 18%  perf-profile.self.cycles-pp.add_transaction_credits
      0.15 ± 12%      +0.7        0.83 ± 13%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      0.17 ±  4%      +0.7        0.85 ±  6%  perf-profile.self.cycles-pp.entry_SYSCALL_64
      0.17 ±  7%      +0.7        0.85 ±  3%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.13 ±  6%      +0.7        0.85 ±  4%  perf-profile.self.cycles-pp.down_write
      0.00            +0.8        0.82 ±  7%  perf-profile.self.cycles-pp.lockref_get_not_zero
      0.04 ± 58%      +0.8        0.88 ±  6%  perf-profile.self.cycles-pp.lockref_get_not_dead
      0.00            +0.8        0.84 ±  2%  perf-profile.self.cycles-pp.rwsem_down_write_failed
      0.06 ±  7%      +0.8        0.90 ±  6%  perf-profile.self.cycles-pp.rwsem_spin_on_owner
      0.23 ±  5%      +0.9        1.09 ± 13%  perf-profile.self.cycles-pp.jbd2_journal_stop
      0.27 ±  8%      +1.1        1.38 ± 13%  perf-profile.self.cycles-pp.start_this_handle
      0.00            +1.3        1.27 ±  4%  perf-profile.self.cycles-pp.lockref_put_return


                                                                                
                                  aim7.jobs-per-min                             
                                                                                
  90000 +-+-----------------------------------------------------------------+   
        |                                 O O     O O                       |   
  80000 +-+                             O       O                           |   
  70000 O-O     O O O     O O O     O O       O                             |   
        |   O O       O O       O O                                         |   
  60000 +-+                                                                 |   
  50000 +-+                                                                 |   
        |                                                                   |   
  40000 +-+                                                                 |   
  30000 +-+                                                                 |   
        |                                                                   |   
  20000 +-+.+.+.+.+.+.+.+.+.+.+.+   +   +   +   +.+.+.+.+.+.+.+.+.+.+.+.+.+.|   
  10000 +-+                      : : : : : : : :                            |   
        |                        : : : : : : : :                            |   
      0 +-+-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                aim7.time.system_time                           
                                                                                
  14000 +-+-----------------------------------------------------------------+   
        |         +.+.+.+.+   +.+   +   :   +   : +   +.+.+.+.+.+.+.+.+     |   
  12000 +-+                     :   :   :   :   :                           |   
        |                       :   :   :   :   :                           |   
  10000 +-+                     :   :  : :  :  :                            |   
        |                        : : : : : : : :                            |   
   8000 +-+                      : : : : : : : :                            |   
        |                        : : : : : : : :                            |   
   6000 +-+                      : : : : : : : :                            |   
        |                        : : : : : : : :                            |   
   4000 +-+                      : : : : : : : :                            |   
        O O O O O O O O O O O O O O O O O :   O O                           |   
   2000 +-+                       :   :   O O :   O O                       |   
        |                         :   :   :   :                             |   
      0 +-+-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                              aim7.time.elapsed_time                            
                                                                                
  400 +-+-------------------------------------------------------------------+   
      |.+.+.+.+. .+.+.+..+.+.+.+   +   +   +   +.+.+. .+.+..+.+.+.+.+. .+.+.|   
  350 +-+       +              :   :   :   :   :     +                +     |   
  300 +-+                      :   :   :   :   :                            |   
      |                        :   :   :   :   :                            |   
  250 +-+                       : : : : : : : :                             |   
      |                         : : : : : : : :                             |   
  200 +-+                       : : : : : : : :                             |   
      |                         : : : : : : : :                             |   
  150 +-+                       : : : : : : : :                             |   
  100 +-+                       : : : : : : : :                             |   
      O O O O O O O O O  O O O O O O O O O O O O O O                        |   
   50 +-+                        :   :   :   :                              |   
      |                          :   :   :   :                              |   
    0 +-+-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                            aim7.time.elapsed_time.max                          
                                                                                
  400 +-+-------------------------------------------------------------------+   
      |.+.+.+.+. .+.+.+..+.+.+.+   +   +   +   +.+.+. .+.+..+.+.+.+.+. .+.+.|   
  350 +-+       +              :   :   :   :   :     +                +     |   
  300 +-+                      :   :   :   :   :                            |   
      |                        :   :   :   :   :                            |   
  250 +-+                       : : : : : : : :                             |   
      |                         : : : : : : : :                             |   
  200 +-+                       : : : : : : : :                             |   
      |                         : : : : : : : :                             |   
  150 +-+                       : : : : : : : :                             |   
  100 +-+                       : : : : : : : :                             |   
      O O O O O O O O O  O O O O O O O O O O O O O O                        |   
   50 +-+                        :   :   :   :                              |   
      |                          :   :   :   :                              |   
    0 +-+-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                             aim7.time.minor_page_faults                        
                                                                                
  120000 +-+----------------------------------------------------------------+   
         |                                                                  |   
  100000 +-+.+.+.+.+. .+.     .+.+   +   +  +   +.+.+. .+. .+.+.+. .+.   .+.|   
         |           +   +.+.+   :   :   :  :   :     +   +       +   +.+   |   
         |                       :   :   :  :   :                           |   
   80000 +-+                      :  ::  :: ::  :                           |   
         |               O   O   O: : : : : :: :                            |   
   60000 O-O O O O O O O   O   O  :O:O:O:O:OO:O:O O O                       |   
         |                        : : : : : :: :                            |   
   40000 +-+                      : : : : :: : :                            |   
         |                        : : : : :: : :                            |   
         |                         ::  ::  :  ::                            |   
   20000 +-+                       :   :   :  :                             |   
         |                         :   :   :  :                             |   
       0 +-+----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                         aim7.time.voluntary_context_switches                   
                                                                                
  1.4e+06 +-+---------------------------------------------------------------+   
          |.+.+.     .+. .+ .+. .+                      .+.+                |   
  1.2e+06 +-+   +.+.+   +  +   + :   +       +    .+.+.+    +.+.+.+.+.+.+.+.|   
          |                      :   :   +   :   +                          |   
    1e+06 +-+                    :   :   :   :   :                          |   
          |                       :  ::  :   ::  :                          |   
   800000 +-+                     : : : : : : : :                           |   
          |                       : : : : : : : :                           |   
   600000 +-+   O                 : : : : : : : :                           |   
          |   O         O O      O:O: :O: : : :O:                           |   
   400000 O-+         O      O    : : : : : : : :                           |   
          | O     O O      O   O   ::O ::O:O:O ::O O O                      |   
   200000 +-+                      :   :   :   :                            |   
          |                        :   :   :   :                            |   
        0 +-+---------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                       aim7.time.involuntary_context_switches                   
                                                                                
  700000 +-+----------------------------------------------------------------+   
         |                                                                  |   
  600000 +-+.+.+.+.+.+.+.+.+.+.+.+   +   +  +   +.+.+.+.+.+.+.+.+.+.+.+.+.+.|   
         |                       :   :   :  :   :                           |   
  500000 +-+                     :   :   :  :   :                           |   
         |                        :  ::  :: ::  :                           |   
  400000 +-+                      : : : : : :: :                            |   
         |                        : : : : : :: :                            |   
  300000 +-+                      : : : : : :: :                            |   
         |                        : : : : :: : :                            |   
  200000 +-+                      : : : : :: : :                            |   
         |                         ::  ::  :  ::                            |   
  100000 O-O O O O O O O O O O O O O O O O OO O O O O                       |   
         |                         :   :   :  :                             |   
       0 +-+----------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.1.0-rc3-00039-g079f992" of type "text/plain" (188903 bytes)

View attachment "job-script" of type "text/plain" (7890 bytes)

View attachment "job.yaml" of type "text/plain" (5690 bytes)

View attachment "reproduce" of type "text/plain" (702 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ