lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Message-ID: <20210831031921.GD4286@xsang-OptiPlex-9020>
Date:   Tue, 31 Aug 2021 11:19:21 +0800
From:   kernel test robot <oliver.sang@...el.com>
To:     Tetsuo Handa <penguin-kernel@...ove.sakura.ne.jp>
Cc:     0day robot <lkp@...el.com>,
        syzbot <syzbot+f61766d5763f9e7a118f@...kaller.appspotmail.com>,
        Tetsuo Handa <penguin-kernel@...ove.SAKURA.ne.jp>,
        LKML <linux-kernel@...r.kernel.org>, lkp@...ts.01.org,
        ying.huang@...el.com, feng.tang@...el.com,
        zhengjun.xing@...ux.intel.com, Christoph Hellwig <hch@....de>,
        Jens Axboe <axboe@...nel.dk>, Hillf Danton <hdanton@...a.com>,
        Pavel Tatashin <pasha.tatashin@...een.com>,
        Tyler Hicks <tyhicks@...ux.microsoft.com>,
        linux-block <linux-block@...r.kernel.org>
Subject: [loop]  8883efd909:  stress-ng.loop.ops_per_sec 148.4% improvement



Greeting,

FYI, we noticed a 148.4% improvement of stress-ng.loop.ops_per_sec due to commit:


commit: 8883efd909da6648570c698567fd8923bf66a852 ("[PATCH] loop: replace loop_ctl_mutex with loop_idr_spinlock")
url: https://github.com/0day-ci/linux/commits/Tetsuo-Handa/loop-replace-loop_ctl_mutex-with-loop_idr_spinlock/20210827-000519
base: git://git.infradead.org/users/hch/configfs.git for-next

in testcase: stress-ng
on test machine: 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 512G memory
with following parameters:

	nr_threads: 100%
	disk: 1HDD
	testtime: 60s
	class: device
	test: loop
	cpufreq_governor: performance
	ucode: 0x5003006






Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install                job.yaml  # job file is attached in this email
        bin/lkp split-job --compatible job.yaml  # generate the yaml file for lkp run
        bin/lkp run                    generated-yaml-file

=========================================================================================
class/compiler/cpufreq_governor/disk/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime/ucode:
  device/gcc-9/performance/1HDD/x86_64-rhel-8.3/100%/debian-10.4-x86_64-20200603.cgz/lkp-csl-2sp7/loop/stress-ng/60s/0x5003006

commit: 
  c42dd069be ("configfs: fix a race in configfs_lookup()")
  8883efd909 ("loop: replace loop_ctl_mutex with loop_idr_spinlock")

c42dd069be8dfc9b 8883efd909da6648570c698567f 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
      1967          +148.1%       4882        stress-ng.loop.ops
     32.23          +148.4%      80.06        stress-ng.loop.ops_per_sec
      4439           +49.5%       6637        stress-ng.time.involuntary_context_switches
     12363            +5.7%      13062        stress-ng.time.minor_page_faults
     59.17 ±  2%    +275.0%     221.86 ±  3%  stress-ng.time.percent_of_cpu_this_job_got
     37.89          +271.7%     140.82 ±  3%  stress-ng.time.system_time
     19176          +619.5%     137970        stress-ng.time.voluntary_context_switches
     97.63            -1.8%      95.87        iostat.cpu.idle
      2.21 ±  2%     +71.0%       3.78 ±  3%  iostat.cpu.system
      2506 ± 24%     +56.8%       3930 ± 10%  numa-meminfo.node1.Active
      2459 ± 21%     +58.1%       3887 ±  9%  numa-meminfo.node1.Active(anon)
    614.33 ± 21%     +58.2%     971.71 ±  9%  numa-vmstat.node1.nr_active_anon
    614.33 ± 21%     +58.2%     971.71 ±  9%  numa-vmstat.node1.nr_zone_active_anon
      1.14 ±  2%      -0.3        0.88 ± 10%  mpstat.cpu.all.irq%
      0.99            +1.9        2.85 ±  3%  mpstat.cpu.all.sys%
      0.14 ±  7%      +0.2        0.33        mpstat.cpu.all.usr%
    220892 ± 22%     +58.7%     350634 ± 14%  numa-numastat.node0.local_node
    267158 ± 12%     +55.8%     416300 ± 14%  numa-numastat.node0.numa_hit
    256360 ± 19%     +54.0%     394696 ± 14%  numa-numastat.node1.local_node
    299440 ± 12%     +41.8%     424652 ± 15%  numa-numastat.node1.numa_hit
      2627          +149.2%       6548        vmstat.io.bi
     23.17 ± 20%    +257.7%      82.86 ± 17%  vmstat.memory.buff
      1.33 ± 35%    +317.9%       5.57 ± 13%  vmstat.procs.r
     10361          +100.6%      20779        vmstat.system.cs
      0.01 ± 22%     -75.9%       0.00 ±159%  perf-sched.sch_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      1.78 ± 12%     -73.8%       0.47 ±158%  perf-sched.wait_and_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      3.55 ± 12%     -73.8%       0.93 ±158%  perf-sched.wait_and_delay.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      1.77 ± 12%     -73.8%       0.46 ±158%  perf-sched.wait_time.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      3.54 ± 12%     -73.8%       0.93 ±158%  perf-sched.wait_time.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
     79.00 ±  3%     +94.6%     153.71 ±  2%  turbostat.Avg_MHz
      3.81 ±  3%      +2.1        5.94 ±  3%  turbostat.Busy%
      2075           +25.0%       2594        turbostat.Bzy_MHz
     91336 ±  5%    +134.3%     213980 ± 12%  turbostat.C1
      0.14 ±  3%      +0.1        0.27 ± 21%  turbostat.C1%
     50.33            +4.2%      52.43        turbostat.PkgTmp
    136.43           +10.9%     151.37        turbostat.PkgWatt
      4082           +38.7%       5663 ±  3%  meminfo.Active
      3836           +40.7%       5397 ±  4%  meminfo.Active(anon)
    278033           +17.0%     325254        meminfo.AnonPages
   1423893           +12.9%    1608169        meminfo.Committed_AS
    290333           +17.1%     339985        meminfo.Inactive
    290180           +17.1%     339777        meminfo.Inactive(anon)
    140265           +13.5%     159191        meminfo.KReclaimable
     10899 ±  2%     +39.4%      15189 ±  4%  meminfo.PageTables
     54413           +19.4%      64976        meminfo.Percpu
    140265           +13.5%     159191        meminfo.SReclaimable
     15761           +27.8%      20142        meminfo.Shmem
     23366          +132.0%      54205        softirqs.BLOCK
     10108 ±  5%     -44.7%       5587 ± 13%  softirqs.CPU0.RCU
      7932 ± 13%     +17.4%       9314 ±  4%  softirqs.CPU14.SCHED
      9034 ± 20%     -55.4%       4031 ± 17%  softirqs.CPU2.RCU
      9678 ±  7%     -12.5%       8464 ±  7%  softirqs.CPU24.SCHED
      7465 ± 16%     -64.6%       2641 ±  6%  softirqs.CPU28.RCU
      8599 ± 16%     -49.9%       4307 ± 36%  softirqs.CPU3.RCU
      7737 ± 20%     -59.3%       3147 ±  8%  softirqs.CPU64.RCU
      8819 ± 20%     -65.6%       3033 ± 17%  softirqs.CPU93.RCU
    720713 ±  2%     -58.4%     299744        softirqs.RCU
     27527           +18.6%      32659        softirqs.TIMER
    956.67           +41.0%       1349 ±  4%  proc-vmstat.nr_active_anon
     69520           +17.0%      81314        proc-vmstat.nr_anon_pages
     72558           +17.1%      84945        proc-vmstat.nr_inactive_anon
     19957            +7.1%      21365        proc-vmstat.nr_kernel_stack
      9759            +4.0%      10145        proc-vmstat.nr_mapped
      2728 ±  2%     +39.1%       3796 ±  4%  proc-vmstat.nr_page_table_pages
      3939           +27.8%       5035        proc-vmstat.nr_shmem
     35060           +13.5%      39798        proc-vmstat.nr_slab_reclaimable
     55263            +9.5%      60522        proc-vmstat.nr_slab_unreclaimable
    956.67           +41.0%       1349 ±  4%  proc-vmstat.nr_zone_active_anon
     72558           +17.1%      84945        proc-vmstat.nr_zone_inactive_anon
    854.83 ± 25%     +88.3%       1610 ± 21%  proc-vmstat.numa_hint_faults
    568291           +48.2%     842456        proc-vmstat.numa_hit
    478946           +55.9%     746875        proc-vmstat.numa_local
     89355            +7.0%      95647        proc-vmstat.numa_other
    565.00 ± 37%     +64.3%     928.43 ± 15%  proc-vmstat.numa_pages_migrated
     12531          +155.2%      31985        proc-vmstat.pgactivate
    618054           +65.2%    1021005        proc-vmstat.pgalloc_normal
    343106            +5.6%     362461        proc-vmstat.pgfault
    509485           +74.9%     891020        proc-vmstat.pgfree
    565.00 ± 37%     +64.3%     928.43 ± 15%  proc-vmstat.pgmigrate_success
    178270          +146.6%     439638        proc-vmstat.pgpgin
     19761 ±  3%     +13.9%      22508        proc-vmstat.pgreuse
     17570 ±  3%    +174.8%      48289 ±  2%  proc-vmstat.unevictable_pgs_culled
 4.988e+08 ±  2%    +141.7%  1.206e+09 ±  3%  perf-stat.i.branch-instructions
   8933620 ± 15%     +95.1%   17430543 ±  8%  perf-stat.i.branch-misses
     15.20 ± 16%      +7.9       23.15 ±  9%  perf-stat.i.cache-miss-rate%
   2892116 ±  9%    +275.9%   10871506 ±  2%  perf-stat.i.cache-misses
  19918505 ± 27%    +141.7%   48133737 ± 12%  perf-stat.i.cache-references
     10259          +134.2%      24030 ±  3%  perf-stat.i.context-switches
     96024           +10.0%     105602 ±  3%  perf-stat.i.cpu-clock
 7.036e+09 ±  4%    +134.4%  1.649e+10 ±  3%  perf-stat.i.cpu-cycles
    245.11 ±  5%    +201.7%     739.59 ±  3%  perf-stat.i.cpu-migrations
      2548 ±  7%     -32.4%       1722 ±  4%  perf-stat.i.cycles-between-cache-misses
 5.805e+08          +132.4%  1.349e+09 ±  2%  perf-stat.i.dTLB-loads
      0.03 ± 23%      -0.0        0.02 ± 17%  perf-stat.i.dTLB-store-miss-rate%
 2.689e+08          +114.5%  5.766e+08 ±  2%  perf-stat.i.dTLB-stores
     39.31 ±  3%      +5.6       44.91 ±  3%  perf-stat.i.iTLB-load-miss-rate%
   1097403 ±  5%     +50.6%    1652224 ±  3%  perf-stat.i.iTLB-load-misses
  2.41e+09 ±  2%    +137.9%  5.733e+09 ±  3%  perf-stat.i.instructions
      2197 ±  6%     +59.6%       3507 ±  2%  perf-stat.i.instructions-per-iTLB-miss
      0.34 ±  3%      +7.1%       0.36 ±  2%  perf-stat.i.ipc
     11.88           +63.1%      19.37 ±  6%  perf-stat.i.major-faults
      0.07 ±  4%    +112.7%       0.16 ±  2%  perf-stat.i.metric.GHz
    237.42 ± 24%    +115.5%     511.75 ± 11%  perf-stat.i.metric.K/sec
     14.03 ±  2%    +113.1%      29.90 ±  2%  perf-stat.i.metric.M/sec
      3788           +41.6%       5363 ±  3%  perf-stat.i.minor-faults
     72.25 ±  2%      +3.7       75.93        perf-stat.i.node-load-miss-rate%
    505210 ±  5%    +370.2%    2375302 ±  3%  perf-stat.i.node-load-misses
    192263 ±  3%    +270.9%     713032 ±  5%  perf-stat.i.node-loads
     67.34 ±  3%      +3.7       71.01        perf-stat.i.node-store-miss-rate%
    319202 ±  7%    +438.6%    1719185 ±  3%  perf-stat.i.node-store-misses
    148309 ±  8%    +326.2%     632114 ±  3%  perf-stat.i.node-stores
      3849           +43.4%       5520 ±  3%  perf-stat.i.page-faults
     96024           +10.0%     105602 ±  3%  perf-stat.i.task-clock
      1.78 ± 13%      -0.4        1.36 ±  9%  perf-stat.overall.branch-miss-rate%
     15.14 ± 16%      +7.7       22.85 ± 12%  perf-stat.overall.cache-miss-rate%
      2448 ±  7%     -38.1%       1516 ±  3%  perf-stat.overall.cycles-between-cache-misses
      0.03 ± 22%      -0.0        0.02 ± 16%  perf-stat.overall.dTLB-store-miss-rate%
     39.36 ±  3%      +5.6       44.95 ±  2%  perf-stat.overall.iTLB-load-miss-rate%
      2201 ±  6%     +50.2%       3305 ±  2%  perf-stat.overall.instructions-per-iTLB-miss
     72.39 ±  2%      +5.1       77.46        perf-stat.overall.node-load-miss-rate%
     68.27 ±  2%      +4.8       73.04        perf-stat.overall.node-store-miss-rate%
 4.907e+08 ±  2%    +105.0%  1.006e+09 ±  2%  perf-stat.ps.branch-instructions
   8778182 ± 15%     +56.2%   13711296 ±  8%  perf-stat.ps.branch-misses
   2847343 ±  9%    +226.5%    9295762 ±  2%  perf-stat.ps.cache-misses
  19608519 ± 27%    +110.8%   41328950 ± 13%  perf-stat.ps.cache-references
     10102          +106.1%      20820        perf-stat.ps.context-switches
 6.928e+09 ±  4%    +103.4%  1.409e+10 ±  2%  perf-stat.ps.cpu-cycles
    241.52 ±  5%    +161.3%     630.99        perf-stat.ps.cpu-migrations
 5.712e+08           +98.1%  1.132e+09 ±  2%  perf-stat.ps.dTLB-loads
 2.646e+08           +85.1%  4.899e+08        perf-stat.ps.dTLB-stores
   1080453 ±  5%     +33.6%    1443503 ±  2%  perf-stat.ps.iTLB-load-misses
 2.371e+09 ±  2%    +101.2%   4.77e+09        perf-stat.ps.instructions
      3719            +7.9%       4011        perf-stat.ps.minor-faults
    497421 ±  5%    +322.4%    2101134 ±  2%  perf-stat.ps.node-load-misses
    189239 ±  3%    +223.0%     611269 ±  3%  perf-stat.ps.node-loads
    314328 ±  7%    +362.1%    1452522 ±  3%  perf-stat.ps.node-store-misses
    145941 ±  8%    +267.3%     535987 ±  2%  perf-stat.ps.node-stores
      3779            +9.6%       4143        perf-stat.ps.page-faults
 1.551e+11 ±  2%    +100.0%  3.101e+11        perf-stat.total.instructions
      4763 ±  6%     +37.5%       6549        slabinfo.Acpi-Parse.active_objs
      4763 ±  6%     +37.5%       6549        slabinfo.Acpi-Parse.num_objs
      1632 ±  2%     +15.1%       1878        slabinfo.bdev_cache.active_objs
      1632 ±  2%     +15.1%       1878        slabinfo.bdev_cache.num_objs
    158287 ±  2%     +14.4%     181134        slabinfo.dentry.active_objs
      3789 ±  2%     +14.6%       4343        slabinfo.dentry.active_slabs
    159181 ±  2%     +14.6%     182438        slabinfo.dentry.num_objs
      3789 ±  2%     +14.6%       4343        slabinfo.dentry.num_slabs
      2803 ±  3%     +20.5%       3376        slabinfo.file_lock_cache.active_objs
      2803 ±  3%     +20.5%       3376        slabinfo.file_lock_cache.num_objs
     29992 ±  3%     +50.6%      45158 ±  3%  slabinfo.filp.active_objs
    956.17 ±  2%     +48.2%       1417 ±  3%  slabinfo.filp.active_slabs
     30617 ±  2%     +48.1%      45357 ±  3%  slabinfo.filp.num_objs
    956.17 ±  2%     +48.2%       1417 ±  3%  slabinfo.filp.num_slabs
      9265 ±  3%     +25.3%      11605        slabinfo.fsnotify_mark_connector.active_objs
      9265 ±  3%     +25.3%      11605        slabinfo.fsnotify_mark_connector.num_objs
    132409           +14.3%     151389        slabinfo.inode_cache.active_objs
      2413           +15.3%       2782        slabinfo.inode_cache.active_slabs
    132752           +15.3%     153089        slabinfo.inode_cache.num_objs
      2413           +15.3%       2782        slabinfo.inode_cache.num_slabs
      6772           +20.6%       8170        slabinfo.kmalloc-1k.active_objs
      7495           +21.7%       9121        slabinfo.kmalloc-1k.num_objs
     11467 ±  3%     +31.7%      15100 ±  3%  slabinfo.kmalloc-256.active_objs
    358.00 ±  3%     +33.0%     476.14 ±  3%  slabinfo.kmalloc-256.active_slabs
     11469 ±  3%     +33.0%      15250 ±  3%  slabinfo.kmalloc-256.num_objs
    358.00 ±  3%     +33.0%     476.14 ±  3%  slabinfo.kmalloc-256.num_slabs
     75029           +25.2%      93955        slabinfo.kmalloc-8.active_objs
     75415           +24.9%      94226        slabinfo.kmalloc-8.num_objs
      3490 ±  5%     +70.9%       5965 ±  4%  slabinfo.kmalloc-cg-1k.active_objs
      3490 ±  5%     +75.8%       6135 ±  4%  slabinfo.kmalloc-cg-1k.num_objs
    850.83 ±  8%    +165.4%       2257 ± 11%  slabinfo.kmalloc-cg-2k.active_objs
    850.83 ±  8%    +171.6%       2310 ±  9%  slabinfo.kmalloc-cg-2k.num_objs
     12149 ±  5%    +100.9%      24402 ±  7%  slabinfo.kmalloc-rcl-64.active_objs
    189.17 ±  5%    +101.3%     380.86 ±  7%  slabinfo.kmalloc-rcl-64.active_slabs
     12155 ±  5%    +100.8%      24412 ±  7%  slabinfo.kmalloc-rcl-64.num_objs
    189.17 ±  5%    +101.3%     380.86 ±  7%  slabinfo.kmalloc-rcl-64.num_slabs
      3547 ±  4%     +65.9%       5884 ±  4%  slabinfo.kmalloc-rcl-96.active_objs
      3547 ±  4%     +65.9%       5884 ±  4%  slabinfo.kmalloc-rcl-96.num_objs
      2926           +10.2%       3226        slabinfo.pool_workqueue.active_objs
      1157 ±  2%     +15.6%       1338        slabinfo.request_queue.active_objs
      1157 ±  2%     +15.6%       1338        slabinfo.request_queue.num_objs
      6093 ±  2%     +26.9%       7731        slabinfo.shmem_inode_cache.active_objs
      6093 ±  2%     +26.9%       7731        slabinfo.shmem_inode_cache.num_objs
      3321 ±  6%     +19.1%       3957 ±  4%  slabinfo.skbuff_ext_cache.active_objs
      3321 ±  6%     +19.1%       3957 ±  4%  slabinfo.skbuff_ext_cache.num_objs
      4430 ±  3%    +102.0%       8948 ±  8%  slabinfo.skbuff_head_cache.active_objs
    138.00 ±  3%    +105.3%     283.29 ±  8%  slabinfo.skbuff_head_cache.active_slabs
      4430 ±  3%    +104.9%       9079 ±  8%  slabinfo.skbuff_head_cache.num_objs
    138.00 ±  3%    +105.3%     283.29 ±  8%  slabinfo.skbuff_head_cache.num_slabs
      3857 ±  3%     +24.1%       4787        slabinfo.task_delay_info.active_objs
      3857 ±  3%     +24.1%       4787        slabinfo.task_delay_info.num_objs
      4758           +23.7%       5884        slabinfo.trace_event_file.active_objs
      4758           +23.7%       5884        slabinfo.trace_event_file.num_objs
     63627 ±  3%     +12.1%      71328 ±  4%  slabinfo.vm_area_struct.active_objs
      1590 ±  3%     +12.2%       1784 ±  4%  slabinfo.vm_area_struct.active_slabs
     63655 ±  3%     +12.2%      71398 ±  4%  slabinfo.vm_area_struct.num_objs
      1590 ±  3%     +12.2%       1784 ±  4%  slabinfo.vm_area_struct.num_slabs
     68516 ±  2%     +33.1%      91195        interrupts.CAL:Function_call_interrupts
    699.67 ± 29%     +81.2%       1267 ± 45%  interrupts.CPU0.CAL:Function_call_interrupts
     42.00 ± 19%    +127.2%      95.43 ± 15%  interrupts.CPU0.RES:Rescheduling_interrupts
    261.67 ± 38%     +82.2%     476.86 ± 24%  interrupts.CPU1.NMI:Non-maskable_interrupts
    261.67 ± 38%     +82.2%     476.86 ± 24%  interrupts.CPU1.PMI:Performance_monitoring_interrupts
    664.33 ±  9%     +47.1%     977.43 ±  6%  interrupts.CPU10.CAL:Function_call_interrupts
    668.67 ±  6%     +46.9%     982.14 ±  6%  interrupts.CPU11.CAL:Function_call_interrupts
     41.00 ± 43%    +128.6%      93.71 ± 19%  interrupts.CPU11.RES:Rescheduling_interrupts
    664.33 ±  5%     +43.3%     952.14 ±  5%  interrupts.CPU12.CAL:Function_call_interrupts
    652.33 ±  6%     +49.3%     973.71 ±  7%  interrupts.CPU13.CAL:Function_call_interrupts
    669.33 ± 10%     +41.7%     948.14 ±  5%  interrupts.CPU14.CAL:Function_call_interrupts
    682.17 ±  8%     +40.5%     958.57 ±  6%  interrupts.CPU15.CAL:Function_call_interrupts
    704.50 ± 14%     +45.8%       1027 ±  7%  interrupts.CPU16.CAL:Function_call_interrupts
    691.17 ± 10%     +40.0%     967.57 ±  5%  interrupts.CPU17.CAL:Function_call_interrupts
    709.33 ±  8%     +36.8%     970.57 ±  4%  interrupts.CPU18.CAL:Function_call_interrupts
    692.33 ± 14%     +36.0%     941.29 ±  2%  interrupts.CPU19.CAL:Function_call_interrupts
    683.17 ±  9%     +32.4%     904.29 ±  6%  interrupts.CPU20.CAL:Function_call_interrupts
    689.50 ± 14%     +41.2%     973.71 ±  5%  interrupts.CPU21.CAL:Function_call_interrupts
    666.17 ±  7%     +45.0%     966.00 ±  6%  interrupts.CPU22.CAL:Function_call_interrupts
    199.33 ± 37%     +94.5%     387.71 ± 29%  interrupts.CPU22.NMI:Non-maskable_interrupts
    199.33 ± 37%     +94.5%     387.71 ± 29%  interrupts.CPU22.PMI:Performance_monitoring_interrupts
    688.17 ± 12%     +44.1%     991.86 ±  5%  interrupts.CPU23.CAL:Function_call_interrupts
    660.67 ±  7%     +45.6%     961.71 ±  3%  interrupts.CPU24.CAL:Function_call_interrupts
    709.17 ± 10%     +33.6%     947.43 ±  3%  interrupts.CPU25.CAL:Function_call_interrupts
    179.00 ± 37%    +102.0%     361.57 ± 15%  interrupts.CPU26.NMI:Non-maskable_interrupts
    179.00 ± 37%    +102.0%     361.57 ± 15%  interrupts.CPU26.PMI:Performance_monitoring_interrupts
    646.83 ±  6%     +41.8%     917.14 ±  4%  interrupts.CPU27.CAL:Function_call_interrupts
    173.17 ± 36%    +188.8%     500.14 ± 69%  interrupts.CPU27.NMI:Non-maskable_interrupts
    173.17 ± 36%    +188.8%     500.14 ± 69%  interrupts.CPU27.PMI:Performance_monitoring_interrupts
     22.67 ± 58%    +292.6%      89.00 ± 18%  interrupts.CPU27.RES:Rescheduling_interrupts
    230.33 ± 34%    +114.0%     492.86 ± 31%  interrupts.CPU28.NMI:Non-maskable_interrupts
    230.33 ± 34%    +114.0%     492.86 ± 31%  interrupts.CPU28.PMI:Performance_monitoring_interrupts
     26.50 ± 74%    +203.5%      80.43 ± 23%  interrupts.CPU28.RES:Rescheduling_interrupts
     33.00 ± 55%    +168.8%      88.71 ± 23%  interrupts.CPU3.RES:Rescheduling_interrupts
    238.83 ± 34%    +148.8%     594.29 ± 63%  interrupts.CPU30.NMI:Non-maskable_interrupts
    238.83 ± 34%    +148.8%     594.29 ± 63%  interrupts.CPU30.PMI:Performance_monitoring_interrupts
     14.00 ± 37%    +388.8%      68.43 ± 25%  interrupts.CPU30.RES:Rescheduling_interrupts
    681.17 ±  8%     +28.6%     876.29 ±  7%  interrupts.CPU31.CAL:Function_call_interrupts
     22.67 ± 41%    +230.9%      75.00 ± 25%  interrupts.CPU31.RES:Rescheduling_interrupts
    692.33 ±  9%     +26.8%     877.71 ±  5%  interrupts.CPU33.CAL:Function_call_interrupts
    210.00 ± 36%    +241.6%     717.43 ± 57%  interrupts.CPU33.NMI:Non-maskable_interrupts
    210.00 ± 36%    +241.6%     717.43 ± 57%  interrupts.CPU33.PMI:Performance_monitoring_interrupts
    689.50 ±  8%     +34.3%     925.71 ±  6%  interrupts.CPU35.CAL:Function_call_interrupts
     18.50 ± 12%    +305.4%      75.00 ± 27%  interrupts.CPU35.RES:Rescheduling_interrupts
    224.00 ± 10%     +78.5%     399.86 ± 34%  interrupts.CPU36.NMI:Non-maskable_interrupts
    224.00 ± 10%     +78.5%     399.86 ± 34%  interrupts.CPU36.PMI:Performance_monitoring_interrupts
    628.50 ±  5%     +46.3%     919.43 ±  2%  interrupts.CPU38.CAL:Function_call_interrupts
    655.83 ±  7%     +40.5%     921.14 ±  4%  interrupts.CPU41.CAL:Function_call_interrupts
    202.67 ± 26%    +228.3%     665.43 ± 70%  interrupts.CPU41.NMI:Non-maskable_interrupts
    202.67 ± 26%    +228.3%     665.43 ± 70%  interrupts.CPU41.PMI:Performance_monitoring_interrupts
    708.67 ± 12%     +29.3%     916.57 ±  6%  interrupts.CPU45.CAL:Function_call_interrupts
    202.67 ± 23%    +105.3%     416.14 ± 21%  interrupts.CPU45.NMI:Non-maskable_interrupts
    202.67 ± 23%    +105.3%     416.14 ± 21%  interrupts.CPU45.PMI:Performance_monitoring_interrupts
    711.00 ± 11%     +28.4%     913.14 ±  4%  interrupts.CPU46.CAL:Function_call_interrupts
    764.50 ± 12%     +25.4%     958.71 ±  4%  interrupts.CPU47.CAL:Function_call_interrupts
    185.67 ± 48%    +129.0%     425.14 ± 24%  interrupts.CPU47.NMI:Non-maskable_interrupts
    185.67 ± 48%    +129.0%     425.14 ± 24%  interrupts.CPU47.PMI:Performance_monitoring_interrupts
    695.33 ±  7%     +30.1%     904.86 ±  3%  interrupts.CPU48.CAL:Function_call_interrupts
    681.83 ± 10%     +34.8%     919.14 ±  6%  interrupts.CPU49.CAL:Function_call_interrupts
    254.83 ± 32%    +118.5%     556.71 ± 23%  interrupts.CPU49.NMI:Non-maskable_interrupts
    254.83 ± 32%    +118.5%     556.71 ± 23%  interrupts.CPU49.PMI:Performance_monitoring_interrupts
    654.17 ±  7%     +34.6%     880.57 ±  6%  interrupts.CPU50.CAL:Function_call_interrupts
    697.17 ±  9%     +31.4%     916.00 ±  3%  interrupts.CPU51.CAL:Function_call_interrupts
    656.50 ±  6%     +43.9%     944.57 ±  4%  interrupts.CPU52.CAL:Function_call_interrupts
    659.67 ±  4%     +39.7%     921.43 ±  3%  interrupts.CPU53.CAL:Function_call_interrupts
    692.67 ±  6%     +30.0%     900.14 ±  3%  interrupts.CPU54.CAL:Function_call_interrupts
    634.83 ±  3%     +39.2%     883.86 ±  3%  interrupts.CPU55.CAL:Function_call_interrupts
    692.50 ± 14%     +35.7%     939.57 ± 10%  interrupts.CPU56.CAL:Function_call_interrupts
    717.00 ± 10%     +26.5%     907.29 ±  6%  interrupts.CPU57.CAL:Function_call_interrupts
    683.83 ±  6%     +30.0%     888.71 ±  4%  interrupts.CPU58.CAL:Function_call_interrupts
    669.00 ±  9%     +35.5%     906.43 ±  4%  interrupts.CPU59.CAL:Function_call_interrupts
    639.17 ±  4%     +42.1%     908.43 ±  4%  interrupts.CPU60.CAL:Function_call_interrupts
    661.00 ±  5%     +37.6%     909.29 ±  5%  interrupts.CPU61.CAL:Function_call_interrupts
    635.67 ±  5%     +46.7%     932.57 ±  4%  interrupts.CPU62.CAL:Function_call_interrupts
    639.83 ±  8%     +43.6%     919.00 ±  2%  interrupts.CPU63.CAL:Function_call_interrupts
    647.83 ±  6%     +39.2%     902.00 ±  3%  interrupts.CPU64.CAL:Function_call_interrupts
    633.83 ±  2%     +49.6%     948.00 ±  4%  interrupts.CPU65.CAL:Function_call_interrupts
    619.33 ±  5%     +56.2%     967.57 ±  5%  interrupts.CPU66.CAL:Function_call_interrupts
    615.33 ±  4%     +53.5%     944.57 ±  7%  interrupts.CPU67.CAL:Function_call_interrupts
     25.67 ± 41%    +198.3%      76.57 ± 28%  interrupts.CPU67.RES:Rescheduling_interrupts
    590.67 ±  6%     +68.6%     995.57 ± 15%  interrupts.CPU68.CAL:Function_call_interrupts
    649.33 ± 11%     +34.3%     872.14 ±  6%  interrupts.CPU69.CAL:Function_call_interrupts
    623.50 ± 10%     +52.1%     948.14        interrupts.CPU7.CAL:Function_call_interrupts
    679.17 ±  7%     +33.3%     905.29 ±  3%  interrupts.CPU70.CAL:Function_call_interrupts
     28.00 ± 21%    +195.9%      82.86 ± 26%  interrupts.CPU70.RES:Rescheduling_interrupts
    639.17 ±  8%     +39.9%     894.00 ±  8%  interrupts.CPU71.CAL:Function_call_interrupts
    677.67 ±  8%     +31.8%     893.14 ±  4%  interrupts.CPU72.CAL:Function_call_interrupts
    694.17 ± 12%     +28.5%     891.71 ±  3%  interrupts.CPU73.CAL:Function_call_interrupts
     17.00 ± 40%    +345.4%      75.71 ± 31%  interrupts.CPU73.RES:Rescheduling_interrupts
    649.17 ± 16%     +38.2%     897.14 ±  5%  interrupts.CPU74.CAL:Function_call_interrupts
    148.00 ± 19%     +82.3%     269.86 ± 20%  interrupts.CPU74.NMI:Non-maskable_interrupts
    148.00 ± 19%     +82.3%     269.86 ± 20%  interrupts.CPU74.PMI:Performance_monitoring_interrupts
    220.83 ± 23%    +103.1%     448.43 ± 37%  interrupts.CPU75.NMI:Non-maskable_interrupts
    220.83 ± 23%    +103.1%     448.43 ± 37%  interrupts.CPU75.PMI:Performance_monitoring_interrupts
    709.50 ± 10%     +32.7%     941.29 ±  3%  interrupts.CPU76.CAL:Function_call_interrupts
    663.33 ±  8%     +33.1%     883.00 ±  6%  interrupts.CPU78.CAL:Function_call_interrupts
    610.00 ±  7%     +44.2%     879.43 ±  4%  interrupts.CPU79.CAL:Function_call_interrupts
    189.00 ± 12%     +80.0%     340.29 ± 29%  interrupts.CPU79.NMI:Non-maskable_interrupts
    189.00 ± 12%     +80.0%     340.29 ± 29%  interrupts.CPU79.PMI:Performance_monitoring_interrupts
    643.33 ±  5%     +42.2%     915.00 ± 10%  interrupts.CPU8.CAL:Function_call_interrupts
    694.50 ± 14%     +41.2%     980.86 ±  4%  interrupts.CPU80.CAL:Function_call_interrupts
     21.83 ± 33%    +274.9%      81.86 ± 26%  interrupts.CPU80.RES:Rescheduling_interrupts
    638.33 ± 19%     +52.3%     972.29 ±  3%  interrupts.CPU81.CAL:Function_call_interrupts
    194.33 ± 27%    +365.5%     904.57 ± 93%  interrupts.CPU81.NMI:Non-maskable_interrupts
    194.33 ± 27%    +365.5%     904.57 ± 93%  interrupts.CPU81.PMI:Performance_monitoring_interrupts
    664.00 ±  6%     +44.0%     956.29 ±  3%  interrupts.CPU82.CAL:Function_call_interrupts
    222.33 ± 22%    +106.7%     459.57 ± 55%  interrupts.CPU82.NMI:Non-maskable_interrupts
    222.33 ± 22%    +106.7%     459.57 ± 55%  interrupts.CPU82.PMI:Performance_monitoring_interrupts
    660.83 ± 14%     +49.2%     985.71 ±  6%  interrupts.CPU83.CAL:Function_call_interrupts
    668.50 ± 19%     +46.3%     977.86 ±  5%  interrupts.CPU85.CAL:Function_call_interrupts
    656.00 ±  9%     +47.0%     964.57 ±  4%  interrupts.CPU86.CAL:Function_call_interrupts
    188.17 ± 35%    +139.6%     450.86 ± 44%  interrupts.CPU86.NMI:Non-maskable_interrupts
    188.17 ± 35%    +139.6%     450.86 ± 44%  interrupts.CPU86.PMI:Performance_monitoring_interrupts
    699.83 ± 11%     +42.1%     994.29 ±  3%  interrupts.CPU87.CAL:Function_call_interrupts
    228.67 ± 34%    +369.7%       1074 ±107%  interrupts.CPU87.NMI:Non-maskable_interrupts
    228.67 ± 34%    +369.7%       1074 ±107%  interrupts.CPU87.PMI:Performance_monitoring_interrupts
    677.17 ± 11%     +43.6%     972.71 ±  4%  interrupts.CPU89.CAL:Function_call_interrupts
    149.67 ± 20%    +569.4%       1001 ±116%  interrupts.CPU89.NMI:Non-maskable_interrupts
    149.67 ± 20%    +569.4%       1001 ±116%  interrupts.CPU89.PMI:Performance_monitoring_interrupts
    683.83 ± 11%     +44.1%     985.43 ± 12%  interrupts.CPU9.CAL:Function_call_interrupts
    673.50 ± 22%     +45.8%     981.71 ±  3%  interrupts.CPU90.CAL:Function_call_interrupts
     23.83 ± 71%    +233.3%      79.43 ± 21%  interrupts.CPU90.RES:Rescheduling_interrupts
    691.67 ± 12%     +37.4%     950.14 ±  4%  interrupts.CPU91.CAL:Function_call_interrupts
    755.17 ± 10%     +31.9%     996.14 ±  5%  interrupts.CPU92.CAL:Function_call_interrupts
    679.83 ± 19%     +38.5%     941.57 ±  8%  interrupts.CPU93.CAL:Function_call_interrupts
    173.67 ± 31%    +150.9%     435.71 ± 43%  interrupts.CPU93.NMI:Non-maskable_interrupts
    173.67 ± 31%    +150.9%     435.71 ± 43%  interrupts.CPU93.PMI:Performance_monitoring_interrupts
    704.83 ± 12%     +32.3%     932.14 ±  3%  interrupts.CPU95.CAL:Function_call_interrupts
     24037 ±  5%    +106.6%      49655 ±  8%  interrupts.NMI:Non-maskable_interrupts
     24037 ±  5%    +106.6%      49655 ±  8%  interrupts.PMI:Performance_monitoring_interrupts
      2853 ±  5%    +148.0%       7076 ±  4%  interrupts.RES:Rescheduling_interrupts
     56.16 ±  3%     -15.3       40.90 ±  6%  perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
     55.55 ±  3%     -15.1       40.42 ±  5%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     55.55 ±  3%     -15.1       40.42 ±  5%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
     55.52 ±  3%     -15.1       40.41 ±  5%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     51.46 ±  4%     -12.8       38.70 ±  6%  perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     50.89 ±  4%     -12.7       38.18 ±  6%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
     38.90 ±  7%      -6.8       32.07 ±  8%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     11.55 ±  7%      -5.8        5.73 ±  8%  perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      9.40 ±  7%      -4.4        5.02 ± 10%  perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle
      6.16 ±  8%      -3.0        3.20 ±  8%  perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      6.07 ±  8%      -2.9        3.15 ±  8%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      3.17 ± 10%      -2.0        1.13 ± 19%  perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      2.65 ± 20%      -1.8        0.82 ± 19%  perf-profile.calltrace.cycles-pp.mutex_spin_on_owner.__mutex_lock.lo_ioctl.blkdev_ioctl.block_ioctl
      2.65 ± 20%      -1.8        0.83 ± 19%  perf-profile.calltrace.cycles-pp.__mutex_lock.lo_ioctl.blkdev_ioctl.block_ioctl.__x64_sys_ioctl
      3.25 ±  8%      -1.4        1.84 ±  9%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      1.80 ± 24%      -1.1        0.67 ± 79%  perf-profile.calltrace.cycles-pp.printk.lo_ioctl.cold.blkdev_ioctl.block_ioctl.__x64_sys_ioctl
      1.80 ± 24%      -1.1        0.67 ± 79%  perf-profile.calltrace.cycles-pp.vprintk_emit.printk.lo_ioctl.cold.blkdev_ioctl.block_ioctl
      1.80 ± 24%      -1.1        0.67 ± 79%  perf-profile.calltrace.cycles-pp.lo_ioctl.cold.blkdev_ioctl.block_ioctl.__x64_sys_ioctl.do_syscall_64
      1.90 ±  9%      -1.1        0.84 ± 13%  perf-profile.calltrace.cycles-pp.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      2.29 ±  7%      -1.0        1.29 ± 11%  perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      1.99 ±  7%      -0.8        1.17 ± 11%  perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
      1.92 ±  6%      -0.8        1.13 ± 12%  perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
      1.01 ±  4%      -0.3        0.67 ± 19%  perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
      0.00            +0.6        0.61 ± 11%  perf-profile.calltrace.cycles-pp.__dentry_kill.dput.simple_recursive_removal.debugfs_remove.blk_mq_debugfs_unregister_hctx
      0.00            +0.6        0.64 ± 14%  perf-profile.calltrace.cycles-pp.kernfs_remove.__kobject_del.kobject_del.blk_mq_unregister_hctx.blk_mq_unregister_dev
      0.00            +0.7        0.68 ± 13%  perf-profile.calltrace.cycles-pp.__kobject_del.kobject_del.blk_mq_unregister_hctx.blk_mq_unregister_dev.blk_unregister_queue
      0.00            +0.7        0.68 ± 14%  perf-profile.calltrace.cycles-pp.kobject_del.blk_mq_unregister_hctx.blk_mq_unregister_dev.blk_unregister_queue.del_gendisk
      0.00            +0.7        0.68 ± 13%  perf-profile.calltrace.cycles-pp.blk_mq_unregister_hctx.blk_mq_unregister_dev.blk_unregister_queue.del_gendisk.loop_remove
      0.68 ± 12%      +0.7        1.38 ± 15%  perf-profile.calltrace.cycles-pp.unit_file_find_dropin_paths
      0.00            +0.7        0.71 ± 13%  perf-profile.calltrace.cycles-pp.blk_mq_unregister_dev.blk_unregister_queue.del_gendisk.loop_remove.loop_control_ioctl
      0.00            +0.7        0.73 ± 19%  perf-profile.calltrace.cycles-pp.new_inode.debugfs_get_inode.__debugfs_create_file.debugfs_create_files.blk_mq_debugfs_register_hctx
      0.00            +0.7        0.73 ± 19%  perf-profile.calltrace.cycles-pp.debugfs_get_inode.__debugfs_create_file.debugfs_create_files.blk_mq_debugfs_register_hctx.blk_mq_debugfs_register
      0.00            +0.7        0.75 ±  9%  perf-profile.calltrace.cycles-pp.dput.simple_recursive_removal.debugfs_remove.blk_mq_debugfs_unregister_hctx.blk_mq_exit_queue
      0.00            +0.8        0.80 ± 13%  perf-profile.calltrace.cycles-pp.blk_unregister_queue.del_gendisk.loop_remove.loop_control_ioctl.__x64_sys_ioctl
      0.17 ±141%      +0.8        0.99 ± 13%  perf-profile.calltrace.cycles-pp.openat64.unit_file_find_dropin_paths
      0.00            +0.9        0.90 ± 12%  perf-profile.calltrace.cycles-pp.do_sys_openat2.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe.openat64
      0.00            +0.9        0.90 ± 12%  perf-profile.calltrace.cycles-pp.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe.openat64.unit_file_find_dropin_paths
      0.00            +0.9        0.95 ± 13%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.openat64.unit_file_find_dropin_paths
      0.00            +1.0        0.95 ± 12%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.openat64.unit_file_find_dropin_paths
      0.00            +1.0        1.03 ± 12%  perf-profile.calltrace.cycles-pp.del_gendisk.loop_remove.loop_control_ioctl.__x64_sys_ioctl.do_syscall_64
      0.09 ±223%      +1.1        1.22 ± 19%  perf-profile.calltrace.cycles-pp.io_serial_out.uart_console_write.serial8250_console_write.console_unlock.vprintk_emit
      8.56 ±  4%      +1.2        9.80 ±  6%  perf-profile.calltrace.cycles-pp.io_serial_in.wait_for_xmitr.serial8250_console_putchar.uart_console_write.serial8250_console_write
     11.20 ±  6%      +1.4       12.56 ±  6%  perf-profile.calltrace.cycles-pp.uart_console_write.serial8250_console_write.console_unlock.vprintk_emit.printk
      0.00            +1.6        1.60 ± 90%  perf-profile.calltrace.cycles-pp.start_creating.__debugfs_create_file.debugfs_create_files.blk_mq_debugfs_register_hctx.blk_mq_debugfs_register
      0.00            +2.5        2.45 ± 57%  perf-profile.calltrace.cycles-pp.__debugfs_create_file.debugfs_create_files.blk_mq_debugfs_register_hctx.blk_mq_debugfs_register.blk_register_queue
      0.00            +2.5        2.45 ± 57%  perf-profile.calltrace.cycles-pp.debugfs_create_files.blk_mq_debugfs_register_hctx.blk_mq_debugfs_register.blk_register_queue.__device_add_disk
     10.15 ±  6%      +2.6       12.72 ±  8%  perf-profile.calltrace.cycles-pp.serial8250_console_write.console_unlock.vprintk_emit.printk.set_capacity_and_notify.cold
     11.12 ±  6%      +2.7       13.81 ±  8%  perf-profile.calltrace.cycles-pp.console_unlock.vprintk_emit.printk.set_capacity_and_notify.cold.loop_set_size
      0.00            +3.1        3.15 ± 73%  perf-profile.calltrace.cycles-pp.blk_mq_debugfs_register_hctx.blk_mq_debugfs_register.blk_register_queue.__device_add_disk.loop_add
      0.00            +3.2        3.23 ± 71%  perf-profile.calltrace.cycles-pp.blk_mq_debugfs_register.blk_register_queue.__device_add_disk.loop_add.loop_control_ioctl
      0.09 ±223%      +3.6        3.72 ± 61%  perf-profile.calltrace.cycles-pp.__mutex_lock.blkdev_get_by_dev.blkdev_open.do_dentry_open.do_open
      2.21 ±  9%      +3.7        5.93 ± 50%  perf-profile.calltrace.cycles-pp.blkdev_open.do_dentry_open.do_open.path_openat.do_filp_open
      2.21 ±  9%      +3.7        5.93 ± 50%  perf-profile.calltrace.cycles-pp.blkdev_get_by_dev.blkdev_open.do_dentry_open.do_open.path_openat
      2.22 ±  9%      +3.7        5.95 ± 49%  perf-profile.calltrace.cycles-pp.do_dentry_open.do_open.path_openat.do_filp_open.do_sys_openat2
      2.23 ±  8%      +3.7        5.97 ± 49%  perf-profile.calltrace.cycles-pp.do_open.path_openat.do_filp_open.do_sys_openat2.do_sys_open
      0.00            +3.8        3.78 ± 65%  perf-profile.calltrace.cycles-pp.blk_register_queue.__device_add_disk.loop_add.loop_control_ioctl.__x64_sys_ioctl
      0.36 ± 70%      +3.9        4.24 ± 59%  perf-profile.calltrace.cycles-pp.loop_add.loop_control_ioctl.__x64_sys_ioctl.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +4.0        4.02 ± 62%  perf-profile.calltrace.cycles-pp.__device_add_disk.loop_add.loop_control_ioctl.__x64_sys_ioctl.do_syscall_64
      2.42 ±  8%      +4.2        6.60 ± 44%  perf-profile.calltrace.cycles-pp.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.42 ±  8%      +4.2        6.60 ± 44%  perf-profile.calltrace.cycles-pp.do_sys_openat2.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.40 ±  8%      +4.9        7.34 ± 39%  perf-profile.calltrace.cycles-pp.path_openat.do_filp_open.do_sys_openat2.do_sys_open.do_syscall_64
      2.40 ±  8%      +5.0        7.35 ± 39%  perf-profile.calltrace.cycles-pp.do_filp_open.do_sys_openat2.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +5.7        5.66 ± 15%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.mntput_no_expire.simple_recursive_removal.debugfs_remove
      0.00            +5.8        5.75 ± 15%  perf-profile.calltrace.cycles-pp._raw_spin_lock.mntput_no_expire.simple_recursive_removal.debugfs_remove.blk_mq_debugfs_unregister_hctx
      0.00            +6.5        6.53 ± 13%  perf-profile.calltrace.cycles-pp.mntput_no_expire.simple_recursive_removal.debugfs_remove.blk_mq_debugfs_unregister_hctx.blk_mq_exit_queue
      0.64 ± 17%      +7.8        8.41 ± 13%  perf-profile.calltrace.cycles-pp.simple_recursive_removal.debugfs_remove.blk_mq_debugfs_unregister_hctx.blk_mq_exit_queue.blk_cleanup_queue
      0.64 ± 17%      +7.8        8.41 ± 13%  perf-profile.calltrace.cycles-pp.debugfs_remove.blk_mq_debugfs_unregister_hctx.blk_mq_exit_queue.blk_cleanup_queue.blk_cleanup_disk
      0.64 ± 17%      +7.8        8.42 ± 13%  perf-profile.calltrace.cycles-pp.blk_mq_debugfs_unregister_hctx.blk_mq_exit_queue.blk_cleanup_queue.blk_cleanup_disk.loop_remove
      0.64 ± 17%      +7.8        8.42 ± 13%  perf-profile.calltrace.cycles-pp.blk_mq_exit_queue.blk_cleanup_queue.blk_cleanup_disk.loop_remove.loop_control_ioctl
      0.66 ± 18%      +7.9        8.54 ± 13%  perf-profile.calltrace.cycles-pp.blk_cleanup_disk.loop_remove.loop_control_ioctl.__x64_sys_ioctl.do_syscall_64
      0.66 ± 18%      +7.9        8.54 ± 13%  perf-profile.calltrace.cycles-pp.blk_cleanup_queue.blk_cleanup_disk.loop_remove.loop_control_ioctl.__x64_sys_ioctl
      0.78 ± 16%      +9.3       10.04 ± 11%  perf-profile.calltrace.cycles-pp.loop_remove.loop_control_ioctl.__x64_sys_ioctl.do_syscall_64.entry_SYSCALL_64_after_hwframe
     19.72 ±  5%     +11.6       31.37 ± 10%  perf-profile.calltrace.cycles-pp.__x64_sys_ioctl.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.46 ±  9%     +13.0       14.51 ± 23%  perf-profile.calltrace.cycles-pp.loop_control_ioctl.__x64_sys_ioctl.do_syscall_64.entry_SYSCALL_64_after_hwframe
     24.55 ±  3%     +15.1       39.62 ±  7%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
     24.55 ±  3%     +15.1       39.63 ±  7%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
     56.16 ±  3%     -15.3       40.90 ±  6%  perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     56.16 ±  3%     -15.3       40.90 ±  6%  perf-profile.children.cycles-pp.cpu_startup_entry
     56.16 ±  3%     -15.3       40.90 ±  6%  perf-profile.children.cycles-pp.do_idle
     55.55 ±  3%     -15.1       40.42 ±  5%  perf-profile.children.cycles-pp.start_secondary
     52.03 ±  4%     -12.9       39.16 ±  6%  perf-profile.children.cycles-pp.cpuidle_enter
     52.00 ±  4%     -12.9       39.14 ±  6%  perf-profile.children.cycles-pp.cpuidle_enter_state
     39.12 ±  7%      -6.8       32.29 ±  8%  perf-profile.children.cycles-pp.intel_idle
     11.31 ±  7%      -4.8        6.53 ±  8%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
      9.72 ±  7%      -4.3        5.38 ± 10%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
      6.40 ±  7%      -3.0        3.42 ±  7%  perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
      6.31 ±  8%      -2.9        3.37 ±  7%  perf-profile.children.cycles-pp.hrtimer_interrupt
      3.22 ± 10%      -2.1        1.15 ± 19%  perf-profile.children.cycles-pp.menu_select
      1.90 ±  7%      -1.7        0.23 ± 44%  perf-profile.children.cycles-pp.lo_release
      1.94 ±  7%      -1.7        0.27 ± 39%  perf-profile.children.cycles-pp.blkdev_put
      1.94 ±  7%      -1.7        0.28 ± 38%  perf-profile.children.cycles-pp.blkdev_close
      2.04 ±  7%      -1.6        0.39 ± 27%  perf-profile.children.cycles-pp.__fput
      2.09 ±  7%      -1.6        0.46 ± 25%  perf-profile.children.cycles-pp.task_work_run
      2.13 ±  7%      -1.6        0.50 ± 23%  perf-profile.children.cycles-pp.exit_to_user_mode_prepare
      2.14 ±  7%      -1.6        0.52 ± 23%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
      2.63 ± 10%      -1.5        1.13 ± 13%  perf-profile.children.cycles-pp.ktime_get
      3.41 ±  8%      -1.4        1.98 ±  9%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      1.77 ± 10%      -1.3        0.46 ± 28%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      1.51 ± 12%      -1.1        0.40 ± 31%  perf-profile.children.cycles-pp.tick_nohz_next_event
      1.95 ± 10%      -1.1        0.89 ± 13%  perf-profile.children.cycles-pp.clockevents_program_event
      2.40 ±  6%      -1.0        1.40 ± 11%  perf-profile.children.cycles-pp.tick_sched_timer
      1.80 ± 24%      -1.0        0.80 ± 48%  perf-profile.children.cycles-pp.lo_ioctl.cold
      2.09 ±  6%      -0.8        1.27 ± 11%  perf-profile.children.cycles-pp.tick_sched_handle
      0.87 ± 16%      -0.8        0.06 ± 70%  perf-profile.children.cycles-pp.timekeeping_max_deferment
      2.03 ±  6%      -0.8        1.22 ± 12%  perf-profile.children.cycles-pp.update_process_times
      0.86 ± 11%      -0.5        0.32 ± 15%  perf-profile.children.cycles-pp.tick_nohz_irq_exit
      1.82 ± 16%      -0.4        1.41 ± 15%  perf-profile.children.cycles-pp.delay_tsc
      0.66 ± 10%      -0.3        0.31 ±  8%  perf-profile.children.cycles-pp.lapic_next_deadline
      1.08 ±  3%      -0.3        0.74 ± 18%  perf-profile.children.cycles-pp.scheduler_tick
      0.58 ±  8%      -0.3        0.30 ± 18%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.54 ± 12%      -0.3        0.28 ±  6%  perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
      0.50 ± 15%      -0.3        0.25 ±  6%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.47 ± 14%      -0.2        0.23 ± 11%  perf-profile.children.cycles-pp.read_tsc
      0.45 ±  9%      -0.2        0.21 ± 29%  perf-profile.children.cycles-pp.ktime_get_update_offsets_now
      0.48 ± 11%      -0.2        0.24 ± 21%  perf-profile.children.cycles-pp.irq_enter_rcu
      0.36 ± 12%      -0.2        0.13 ± 17%  perf-profile.children.cycles-pp.rcu_sched_clock_irq
      0.40 ±  3%      -0.2        0.17 ±  5%  perf-profile.children.cycles-pp.irqtime_account_irq
      0.44 ± 12%      -0.2        0.22 ± 22%  perf-profile.children.cycles-pp.tick_irq_enter
      0.42 ± 16%      -0.2        0.21 ±  5%  perf-profile.children.cycles-pp.native_sched_clock
      0.40 ± 47%      -0.2        0.23 ± 15%  perf-profile.children.cycles-pp.__close
      0.35 ± 13%      -0.2        0.19 ± 15%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.20 ± 12%      -0.1        0.07 ± 11%  perf-profile.children.cycles-pp.__hrtimer_next_event_base
      0.22 ± 21%      -0.1        0.11 ± 13%  perf-profile.children.cycles-pp.__intel_pmu_enable_all
      0.21 ± 15%      -0.1        0.09 ± 24%  perf-profile.children.cycles-pp.rcu_idle_exit
      0.42 ± 15%      -0.1        0.31 ± 12%  perf-profile.children.cycles-pp.poll_idle
      0.20 ± 10%      -0.1        0.10 ± 17%  perf-profile.children.cycles-pp._raw_spin_trylock
      0.15 ± 11%      -0.1        0.06 ± 45%  perf-profile.children.cycles-pp.rcu_eqs_exit
      0.23 ± 21%      -0.1        0.14 ± 11%  perf-profile.children.cycles-pp.hrtimer_update_next_event
      0.24 ± 11%      -0.1        0.15 ± 14%  perf-profile.children.cycles-pp.update_rq_clock
      0.19 ±  7%      -0.1        0.10 ± 33%  perf-profile.children.cycles-pp.arch_scale_freq_tick
      0.15 ± 14%      -0.1        0.08 ± 21%  perf-profile.children.cycles-pp.update_irq_load_avg
      0.11 ± 16%      -0.1        0.05 ± 64%  perf-profile.children.cycles-pp.trigger_load_balance
      0.15 ± 18%      -0.1        0.10 ± 13%  perf-profile.children.cycles-pp.read
      0.08 ± 14%      -0.1        0.03 ± 89%  perf-profile.children.cycles-pp.exc_page_fault
      0.08 ± 14%      -0.0        0.03 ± 86%  perf-profile.children.cycles-pp.epoll_wait
      0.09 ± 18%      -0.0        0.05 ± 43%  perf-profile.children.cycles-pp.asm_exc_page_fault
      0.09 ± 13%      +0.0        0.13 ±  9%  perf-profile.children.cycles-pp.__d_lookup
      0.06 ± 46%      +0.0        0.10 ±  6%  perf-profile.children.cycles-pp.lookup_dcache
      0.06 ± 46%      +0.0        0.10 ±  6%  perf-profile.children.cycles-pp.d_lookup
      0.02 ± 99%      +0.0        0.07 ± 15%  perf-profile.children.cycles-pp.mutex_unlock
      0.02 ±141%      +0.1        0.07 ± 13%  perf-profile.children.cycles-pp.d_add
      0.08 ± 18%      +0.1        0.13 ± 14%  perf-profile.children.cycles-pp.__d_alloc
      0.02 ±141%      +0.1        0.07 ±  9%  perf-profile.children.cycles-pp.simple_lookup
      0.07 ± 23%      +0.1        0.12 ± 26%  perf-profile.children.cycles-pp.vfs_fstat
      0.11 ± 13%      +0.1        0.16 ± 22%  perf-profile.children.cycles-pp.__do_sys_newfstat
      0.02 ±144%      +0.1        0.08 ± 19%  perf-profile.children.cycles-pp.__alloc_pages
      0.00            +0.1        0.06 ± 10%  perf-profile.children.cycles-pp.blk_throtl_init
      0.00            +0.1        0.06 ± 16%  perf-profile.children.cycles-pp.blk_mq_init_sched
      0.00            +0.1        0.06 ± 13%  perf-profile.children.cycles-pp.__percpu_counter_init
      0.05 ± 76%      +0.1        0.11 ± 17%  perf-profile.children.cycles-pp.__d_drop
      0.04 ± 74%      +0.1        0.11 ± 17%  perf-profile.children.cycles-pp.___d_drop
      0.00            +0.1        0.07 ± 15%  perf-profile.children.cycles-pp.kernfs_put
      0.00            +0.1        0.07 ± 17%  perf-profile.children.cycles-pp.d_instantiate
      0.06 ± 58%      +0.1        0.12 ± 17%  perf-profile.children.cycles-pp.mutex_lock
      0.02 ±141%      +0.1        0.09 ± 11%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.00            +0.1        0.07 ± 18%  perf-profile.children.cycles-pp.obj_cgroup_uncharge_pages
      0.11 ± 29%      +0.1        0.18 ± 10%  perf-profile.children.cycles-pp.smpboot_thread_fn
      0.00            +0.1        0.07 ± 25%  perf-profile.children.cycles-pp.internal_create_groups
      0.00            +0.1        0.07 ± 17%  perf-profile.children.cycles-pp.__x64_sys_nanosleep
      0.00            +0.1        0.07 ± 10%  perf-profile.children.cycles-pp.page_counter_uncharge
      0.00            +0.1        0.07 ± 17%  perf-profile.children.cycles-pp.hrtimer_nanosleep
      0.00            +0.1        0.07 ± 17%  perf-profile.children.cycles-pp.do_nanosleep
      0.00            +0.1        0.07 ±  8%  perf-profile.children.cycles-pp.elevator_init_mq
      0.00            +0.1        0.07 ± 34%  perf-profile.children.cycles-pp.obj_cgroup_charge_pages
      0.03 ±100%      +0.1        0.10 ± 21%  perf-profile.children.cycles-pp.__slab_free
      0.06 ± 18%      +0.1        0.13 ± 18%  perf-profile.children.cycles-pp.memset_erms
      0.00            +0.1        0.07 ± 23%  perf-profile.children.cycles-pp.kernfs_iop_lookup
      0.00            +0.1        0.07 ± 17%  perf-profile.children.cycles-pp.up_write
      0.00            +0.1        0.07 ±  6%  perf-profile.children.cycles-pp.rcu_cblist_dequeue
      0.00            +0.1        0.08 ±  9%  perf-profile.children.cycles-pp.__list_add_valid
      0.00            +0.1        0.08 ± 11%  perf-profile.children.cycles-pp.blkcg_init_queue
      0.00            +0.1        0.08 ± 20%  perf-profile.children.cycles-pp.remove_files
      0.00            +0.1        0.08 ± 25%  perf-profile.children.cycles-pp.kernfs_iop_getattr
      0.03 ±100%      +0.1        0.11 ± 15%  perf-profile.children.cycles-pp.run_ksoftirqd
      0.08 ± 33%      +0.1        0.16 ± 13%  perf-profile.children.cycles-pp.d_invalidate
      0.00            +0.1        0.09 ±  8%  perf-profile.children.cycles-pp.remove_one
      0.09 ± 14%      +0.1        0.18 ± 13%  perf-profile.children.cycles-pp.alloc_inode
      0.00            +0.1        0.09 ± 18%  perf-profile.children.cycles-pp.path_parentat
      0.00            +0.1        0.09 ± 20%  perf-profile.children.cycles-pp.sysfs_unmerge_group
      0.00            +0.1        0.09 ± 25%  perf-profile.children.cycles-pp.obj_cgroup_charge
      0.18 ± 13%      +0.1        0.26 ± 13%  perf-profile.children.cycles-pp.filemap_read
      0.00            +0.1        0.09 ± 20%  perf-profile.children.cycles-pp.filename_parentat
      0.00            +0.1        0.09 ± 22%  perf-profile.children.cycles-pp.do_renameat2
      0.00            +0.1        0.09 ± 23%  perf-profile.children.cycles-pp.__x64_sys_rename
      0.00            +0.1        0.09 ± 21%  perf-profile.children.cycles-pp.__srcu_read_lock
      0.00            +0.1        0.09 ± 19%  perf-profile.children.cycles-pp.sysfs_remove_group
      0.00            +0.1        0.10 ± 29%  perf-profile.children.cycles-pp.rq_qos_exit
      0.00            +0.1        0.10 ± 29%  perf-profile.children.cycles-pp.blk_mq_debugfs_unregister_queue_rqos
      0.00            +0.1        0.10 ± 16%  perf-profile.children.cycles-pp.__destroy_inode
      0.17 ± 19%      +0.1        0.27 ± 12%  perf-profile.children.cycles-pp._find_next_bit
      0.00            +0.1        0.10 ± 12%  perf-profile.children.cycles-pp.destroy_inode
      0.00            +0.1        0.10 ± 11%  perf-profile.children.cycles-pp.device_unregister
      0.12 ± 12%      +0.1        0.23 ± 14%  perf-profile.children.cycles-pp.d_alloc
      0.00            +0.1        0.10 ± 12%  perf-profile.children.cycles-pp.blk_alloc_queue
      0.01 ±223%      +0.1        0.11 ± 17%  perf-profile.children.cycles-pp.fsnotify_grab_connector
      0.00            +0.1        0.11 ± 17%  perf-profile.children.cycles-pp.dpm_sysfs_remove
      0.00            +0.1        0.11 ± 44%  perf-profile.children.cycles-pp.kernfs_activate
      0.01 ±223%      +0.1        0.12 ± 18%  perf-profile.children.cycles-pp.fsnotify_destroy_marks
      0.10 ± 21%      +0.1        0.22 ± 18%  perf-profile.children.cycles-pp.do_faccessat
      0.04 ± 72%      +0.1        0.16 ± 14%  perf-profile.children.cycles-pp.blk_mq_init_queue_data
      0.31 ±  6%      +0.1        0.43 ± 14%  perf-profile.children.cycles-pp.kmem_cache_alloc
      0.01 ±223%      +0.1        0.13 ± 17%  perf-profile.children.cycles-pp.dentry_unlink_inode
      0.00            +0.1        0.13 ± 24%  perf-profile.children.cycles-pp.__kernfs_new_node
      0.00            +0.1        0.13 ± 29%  perf-profile.children.cycles-pp.device_add
      0.16 ± 23%      +0.1        0.29 ± 10%  perf-profile.children.cycles-pp.cpumask_next
      0.04 ± 72%      +0.1        0.17 ± 11%  perf-profile.children.cycles-pp.__blk_mq_alloc_disk
      0.14 ±  7%      +0.1        0.28 ± 15%  perf-profile.children.cycles-pp.d_alloc_parallel
      0.00            +0.1        0.14 ± 23%  perf-profile.children.cycles-pp.lookup_mnt
      0.00            +0.1        0.14 ± 11%  perf-profile.children.cycles-pp.lockref_put_or_lock
      0.01 ±223%      +0.1        0.15 ± 13%  perf-profile.children.cycles-pp.complete_walk
      0.20 ± 13%      +0.1        0.34 ± 18%  perf-profile.children.cycles-pp.kmem_cache_free
      0.04 ± 71%      +0.1        0.18 ± 14%  perf-profile.children.cycles-pp.pcpu_alloc
      0.00            +0.1        0.14 ± 22%  perf-profile.children.cycles-pp.__traverse_mounts
      0.12 ±  7%      +0.1        0.27 ±  9%  perf-profile.children.cycles-pp.new_inode_pseudo
      0.00            +0.2        0.15 ± 21%  perf-profile.children.cycles-pp.kernfs_new_node
      0.05 ± 46%      +0.2        0.21 ±  9%  perf-profile.children.cycles-pp.try_to_unlazy
      0.00            +0.2        0.16 ±  9%  perf-profile.children.cycles-pp.bdi_unregister
      0.04 ± 72%      +0.2        0.20 ± 15%  perf-profile.children.cycles-pp.down_write
      0.02 ±144%      +0.2        0.18 ± 26%  perf-profile.children.cycles-pp.kernfs_dop_revalidate
      0.08 ± 50%      +0.2        0.24 ± 13%  perf-profile.children.cycles-pp.step_into
      0.02 ±223%      +0.2        0.18 ± 23%  perf-profile.children.cycles-pp.wait_rcu_exp_gp
      0.02 ±223%      +0.2        0.18 ± 23%  perf-profile.children.cycles-pp.sync_rcu_exp_select_cpus
      0.35 ± 11%      +0.2        0.53 ± 18%  perf-profile.children.cycles-pp.rcu_core
      0.02 ±146%      +0.2        0.20 ± 22%  perf-profile.children.cycles-pp.__do_sys_newstat
      0.02 ±144%      +0.2        0.20 ±  8%  perf-profile.children.cycles-pp.__legitimize_path
      0.18 ± 11%      +0.2        0.37 ± 12%  perf-profile.children.cycles-pp.lookup_one_len
      0.20 ±  6%      +0.2        0.39 ± 15%  perf-profile.children.cycles-pp.__lookup_slow
      0.00            +0.2        0.20 ± 15%  perf-profile.children.cycles-pp.kernfs_remove_by_name_ns
      0.02 ±146%      +0.2        0.22 ± 23%  perf-profile.children.cycles-pp.vfs_statx
      0.12 ± 18%      +0.2        0.32 ± 15%  perf-profile.children.cycles-pp.lookup_fast
      0.00            +0.2        0.22 ± 27%  perf-profile.children.cycles-pp.internal_create_group
      0.00            +0.2        0.22 ± 19%  perf-profile.children.cycles-pp.blk_mq_debugfs_unregister_sched
      0.08 ± 15%      +0.2        0.30 ± 15%  perf-profile.children.cycles-pp.inode_permission
      0.00            +0.2        0.23 ± 19%  perf-profile.children.cycles-pp.__elevator_exit
      0.00            +0.2        0.23 ± 19%  perf-profile.children.cycles-pp.blk_mq_exit_sched
      0.08 ± 12%      +0.2        0.31 ± 17%  perf-profile.children.cycles-pp.evict
      0.01 ±223%      +0.2        0.25 ± 19%  perf-profile.children.cycles-pp.kernfs_iop_permission
      0.00            +0.2        0.24 ± 12%  perf-profile.children.cycles-pp.__legitimize_mnt
      0.00            +0.2        0.24 ± 15%  perf-profile.children.cycles-pp.device_del
      0.00            +0.2        0.25 ± 29%  perf-profile.children.cycles-pp.__kernfs_create_file
      0.00            +0.2        0.25 ± 29%  perf-profile.children.cycles-pp.sysfs_add_file_mode_ns
      0.00            +0.3        0.27 ± 26%  perf-profile.children.cycles-pp.sysfs_create_dir_ns
      0.01 ±223%      +0.3        0.28 ± 27%  perf-profile.children.cycles-pp.blk_mq_register_hctx
      0.01 ±223%      +0.3        0.28 ± 14%  perf-profile.children.cycles-pp.__kernfs_remove
      0.01 ±223%      +0.3        0.28 ± 26%  perf-profile.children.cycles-pp.__blk_mq_register_dev
      0.00            +0.3        0.28 ± 26%  perf-profile.children.cycles-pp.kernfs_create_dir_ns
      0.22 ± 15%      +0.3        0.51 ± 18%  perf-profile.children.cycles-pp.rcu_do_batch
      0.01 ±223%      +0.3        0.29 ± 26%  perf-profile.children.cycles-pp.kobject_add
      0.00            +0.3        0.29 ± 25%  perf-profile.children.cycles-pp.kobject_add_internal
      0.01 ±223%      +0.3        0.32 ± 17%  perf-profile.children.cycles-pp.disk_release
      0.01 ±223%      +0.3        0.32 ± 17%  perf-profile.children.cycles-pp.blk_release_queue
      0.01 ±223%      +0.3        0.33 ± 17%  perf-profile.children.cycles-pp.device_release
      0.01 ±223%      +0.3        0.35 ± 16%  perf-profile.children.cycles-pp.kobject_release
      0.28 ± 22%      +0.4        0.65 ±  8%  perf-profile.children.cycles-pp.mnt_get_count
      0.14 ± 26%      +0.4        0.50 ± 18%  perf-profile.children.cycles-pp.filename_lookup
      0.30 ± 15%      +0.4        0.68 ± 13%  perf-profile.children.cycles-pp.walk_component
      0.00            +0.4        0.41 ± 32%  perf-profile.children.cycles-pp.kernfs_add_one
      0.60 ± 12%      +0.4        1.05 ± 13%  perf-profile.children.cycles-pp.openat64
      0.21 ± 15%      +0.5        0.71 ± 11%  perf-profile.children.cycles-pp.__dentry_kill
      0.22 ± 17%      +0.5        0.73 ± 15%  perf-profile.children.cycles-pp.link_path_walk
      0.96 ± 10%      +0.6        1.51 ±  6%  perf-profile.children.cycles-pp.io_serial_out
      0.02 ±143%      +0.6        0.60 ± 19%  perf-profile.children.cycles-pp.path_init
      0.32 ± 16%      +0.6        0.96 ±  9%  perf-profile.children.cycles-pp.dput
      0.00            +0.6        0.64 ± 25%  perf-profile.children.cycles-pp.inode_sb_list_add
      0.04 ± 72%      +0.7        0.70 ±130%  perf-profile.children.cycles-pp.debugfs_create_dir
      0.00            +0.7        0.68 ± 13%  perf-profile.children.cycles-pp.blk_mq_unregister_hctx
      0.68 ± 12%      +0.7        1.38 ± 15%  perf-profile.children.cycles-pp.unit_file_find_dropin_paths
      0.01 ±223%      +0.7        0.71 ± 13%  perf-profile.children.cycles-pp.blk_mq_unregister_dev
      0.00            +0.7        0.74 ±147%  perf-profile.children.cycles-pp.simple_release_fs
      0.00            +0.8        0.77 ± 14%  perf-profile.children.cycles-pp.kernfs_remove
      0.13 ±  9%      +0.8        0.91 ± 18%  perf-profile.children.cycles-pp.new_inode
      0.12 ± 10%      +0.8        0.91 ± 18%  perf-profile.children.cycles-pp.debugfs_get_inode
      0.01 ±223%      +0.8        0.80 ± 13%  perf-profile.children.cycles-pp.blk_unregister_queue
      0.01 ±223%      +0.8        0.82 ± 14%  perf-profile.children.cycles-pp.kobject_del
      0.01 ±223%      +0.8        0.84 ± 13%  perf-profile.children.cycles-pp.__kobject_del
      0.05 ± 47%      +1.0        1.03 ± 12%  perf-profile.children.cycles-pp.del_gendisk
      0.38 ± 12%      +1.0        1.37 ± 15%  perf-profile.children.cycles-pp.path_lookupat
      9.05 ±  5%      +1.4       10.46 ±  6%  perf-profile.children.cycles-pp.io_serial_in
     11.41 ±  6%      +1.4       12.83 ±  6%  perf-profile.children.cycles-pp.uart_console_write
     11.87 ±  6%      +1.5       13.40 ±  6%  perf-profile.children.cycles-pp.serial8250_console_write
     12.97 ±  7%      +1.6       14.54 ±  6%  perf-profile.children.cycles-pp.console_unlock
      0.00            +1.8        1.80 ±141%  perf-profile.children.cycles-pp.simple_pin_fs
      0.20 ±  9%      +2.1        2.28 ±111%  perf-profile.children.cycles-pp.start_creating
      0.30 ± 11%      +2.3        2.64 ± 60%  perf-profile.children.cycles-pp.__debugfs_create_file
      0.30 ± 11%      +2.3        2.64 ± 60%  perf-profile.children.cycles-pp.debugfs_create_files
      0.34 ±  8%      +2.8        3.15 ± 73%  perf-profile.children.cycles-pp.blk_mq_debugfs_register_hctx
      0.35 ±  9%      +2.9        3.23 ± 71%  perf-profile.children.cycles-pp.blk_mq_debugfs_register
      0.41 ±  7%      +3.4        3.78 ± 65%  perf-profile.children.cycles-pp.blk_register_queue
      0.44 ±  5%      +3.6        4.02 ± 62%  perf-profile.children.cycles-pp.__device_add_disk
      2.28 ±  8%      +3.7        5.93 ± 49%  perf-profile.children.cycles-pp.blkdev_open
      2.28 ±  8%      +3.7        5.93 ± 49%  perf-profile.children.cycles-pp.blkdev_get_by_dev
      2.30 ±  8%      +3.7        5.96 ± 49%  perf-profile.children.cycles-pp.do_dentry_open
      2.30 ±  8%      +3.7        5.97 ± 49%  perf-profile.children.cycles-pp.do_open
      0.55 ±  6%      +3.7        4.24 ± 58%  perf-profile.children.cycles-pp.loop_add
      0.10 ± 73%      +4.2        4.33 ± 35%  perf-profile.children.cycles-pp.osq_lock
      2.97 ±  7%      +4.7        7.65 ± 37%  perf-profile.children.cycles-pp.path_openat
      2.98 ±  7%      +4.7        7.66 ± 37%  perf-profile.children.cycles-pp.do_filp_open
      3.17 ±  6%      +4.7        7.86 ± 36%  perf-profile.children.cycles-pp.do_sys_open
      3.16 ±  6%      +4.7        7.85 ± 36%  perf-profile.children.cycles-pp.do_sys_openat2
      0.30 ± 21%      +6.6        6.89 ± 13%  perf-profile.children.cycles-pp.mntput_no_expire
      0.64 ± 17%      +7.8        8.42 ± 13%  perf-profile.children.cycles-pp.blk_mq_debugfs_unregister_hctx
      0.64 ± 17%      +7.8        8.42 ± 13%  perf-profile.children.cycles-pp.blk_mq_exit_queue
      0.66 ± 18%      +7.9        8.54 ± 13%  perf-profile.children.cycles-pp.blk_cleanup_disk
      0.66 ± 18%      +7.9        8.54 ± 13%  perf-profile.children.cycles-pp.blk_cleanup_queue
      0.68 ± 18%      +8.1        8.81 ± 12%  perf-profile.children.cycles-pp.simple_recursive_removal
      0.68 ± 19%      +8.1        8.81 ± 12%  perf-profile.children.cycles-pp.debugfs_remove
      0.78 ± 16%      +9.3       10.04 ± 11%  perf-profile.children.cycles-pp.loop_remove
      0.00            +9.3        9.27 ± 36%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      0.56 ± 12%      +9.6       10.14 ± 33%  perf-profile.children.cycles-pp._raw_spin_lock
     19.85 ±  6%     +11.5       31.37 ± 10%  perf-profile.children.cycles-pp.__x64_sys_ioctl
      1.46 ±  9%     +13.0       14.51 ± 23%  perf-profile.children.cycles-pp.loop_control_ioctl
     26.59 ±  5%     +15.1       41.71 ±  6%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     26.56 ±  5%     +15.1       41.69 ±  6%  perf-profile.children.cycles-pp.do_syscall_64
     39.12 ±  7%      -6.8       32.29 ±  8%  perf-profile.self.cycles-pp.intel_idle
      2.25 ±  9%      -1.3        0.95 ± 14%  perf-profile.self.cycles-pp.ktime_get
      2.78 ±  6%      -1.3        1.52 ± 15%  perf-profile.self.cycles-pp.vprintk_emit
      0.87 ± 16%      -0.8        0.06 ± 70%  perf-profile.self.cycles-pp.timekeeping_max_deferment
      1.48 ± 11%      -0.7        0.74 ±  3%  perf-profile.self.cycles-pp.cpuidle_enter_state
      1.29 ± 19%      -0.7        0.62 ± 19%  perf-profile.self.cycles-pp.menu_select
      1.82 ± 16%      -0.4        1.41 ± 15%  perf-profile.self.cycles-pp.delay_tsc
      0.66 ± 10%      -0.3        0.31 ±  8%  perf-profile.self.cycles-pp.lapic_next_deadline
      0.58 ±  8%      -0.3        0.30 ± 18%  perf-profile.self.cycles-pp.native_irq_return_iret
      0.45 ± 14%      -0.2        0.22 ± 12%  perf-profile.self.cycles-pp.read_tsc
      0.33 ± 11%      -0.2        0.12 ± 16%  perf-profile.self.cycles-pp.rcu_sched_clock_irq
      0.41 ± 16%      -0.2        0.20 ±  4%  perf-profile.self.cycles-pp.native_sched_clock
      0.37 ± 11%      -0.2        0.17 ± 35%  perf-profile.self.cycles-pp.ktime_get_update_offsets_now
      0.34 ± 14%      -0.2        0.15 ± 14%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.39 ± 16%      -0.2        0.22 ± 33%  perf-profile.self.cycles-pp.update_process_times
      0.19 ±  6%      -0.1        0.07 ± 15%  perf-profile.self.cycles-pp.irqtime_account_irq
      0.22 ± 21%      -0.1        0.11 ± 13%  perf-profile.self.cycles-pp.__intel_pmu_enable_all
      0.42 ± 15%      -0.1        0.31 ± 11%  perf-profile.self.cycles-pp.poll_idle
      0.18 ± 15%      -0.1        0.07 ± 13%  perf-profile.self.cycles-pp.__hrtimer_next_event_base
      0.24 ± 14%      -0.1        0.13 ± 22%  perf-profile.self.cycles-pp.do_idle
      0.20 ± 11%      -0.1        0.10 ± 17%  perf-profile.self.cycles-pp._raw_spin_trylock
      0.19 ±  7%      -0.1        0.10 ± 33%  perf-profile.self.cycles-pp.arch_scale_freq_tick
      0.12 ± 16%      -0.1        0.04 ± 88%  perf-profile.self.cycles-pp.asm_sysvec_apic_timer_interrupt
      0.15 ± 14%      -0.1        0.08 ± 21%  perf-profile.self.cycles-pp.update_irq_load_avg
      0.13 ± 17%      -0.1        0.07 ± 18%  perf-profile.self.cycles-pp.hrtimer_interrupt
      0.10 ± 11%      -0.1        0.04 ± 68%  perf-profile.self.cycles-pp.__softirqentry_text_start
      0.10 ± 13%      -0.1        0.05 ± 65%  perf-profile.self.cycles-pp.trigger_load_balance
      0.09 ± 21%      +0.0        0.12 ± 14%  perf-profile.self.cycles-pp.kmem_cache_alloc
      0.02 ± 99%      +0.0        0.07 ± 15%  perf-profile.self.cycles-pp.mutex_unlock
      0.08 ± 21%      +0.1        0.14 ± 11%  perf-profile.self.cycles-pp.cpumask_next
      0.06 ± 18%      +0.1        0.12 ± 11%  perf-profile.self.cycles-pp.memset_erms
      0.04 ± 74%      +0.1        0.11 ± 17%  perf-profile.self.cycles-pp.___d_drop
      0.02 ±141%      +0.1        0.08 ± 10%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.15 ± 18%      +0.1        0.21 ± 13%  perf-profile.self.cycles-pp._find_next_bit
      0.00            +0.1        0.07 ± 22%  perf-profile.self.cycles-pp.lookup_mnt
      0.00            +0.1        0.07 ± 17%  perf-profile.self.cycles-pp.up_write
      0.00            +0.1        0.07 ±  9%  perf-profile.self.cycles-pp.__list_add_valid
      0.00            +0.1        0.07 ±  6%  perf-profile.self.cycles-pp.rcu_cblist_dequeue
      0.02 ±141%      +0.1        0.10 ± 21%  perf-profile.self.cycles-pp.__slab_free
      0.00            +0.1        0.08 ± 22%  perf-profile.self.cycles-pp.__destroy_inode
      0.00            +0.1        0.09 ±  8%  perf-profile.self.cycles-pp.remove_one
      0.01 ±223%      +0.1        0.10 ± 16%  perf-profile.self.cycles-pp.mutex_lock
      0.00            +0.1        0.09 ± 21%  perf-profile.self.cycles-pp.__srcu_read_lock
      0.00            +0.1        0.10 ± 17%  perf-profile.self.cycles-pp.__mutex_lock
      0.00            +0.1        0.13 ± 14%  perf-profile.self.cycles-pp.lockref_put_or_lock
      0.02 ±141%      +0.2        0.18 ± 12%  perf-profile.self.cycles-pp.down_write
      0.00            +0.2        0.19 ± 10%  perf-profile.self.cycles-pp.mntput_no_expire
      0.15 ± 27%      +0.3        0.43 ± 10%  perf-profile.self.cycles-pp.mnt_get_count
      0.54 ± 12%      +0.3        0.88 ±  9%  perf-profile.self.cycles-pp._raw_spin_lock
      0.02 ±144%      +0.5        0.54 ± 19%  perf-profile.self.cycles-pp.path_init
      0.96 ± 10%      +0.6        1.51 ±  6%  perf-profile.self.cycles-pp.io_serial_out
      9.05 ±  5%      +1.4       10.46 ±  6%  perf-profile.self.cycles-pp.io_serial_in
      0.10 ± 73%      +4.2        4.30 ± 36%  perf-profile.self.cycles-pp.osq_lock
      0.00            +9.2        9.20 ± 35%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath


                                                                                
                            stress-ng.time.system_time                          
                                                                                
  180 +---------------------------------------------------------------------+   
      |            O                                                        |   
  160 |-+  O        O     O OO                                              |   
  140 |-O O  O          O      O O   O  O   OO   O   O                      |   
      |        O      O            O  O   O    O   O  O                     |   
  120 |-+        O                                                          |   
      |                                                                     |   
  100 |-+                                                                   |   
      |                                                                     |   
   80 |-+                                                                   |   
   60 |-+                                                                   |   
      |                                                                     |   
   40 |.+.++.+.+.+.++.+.+.+.++.+.+.+.++.+.+.++.+.+.+.++.+.+.+.++.+.+.+. +.+.|   
      |                                                                +    |   
   20 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                    stress-ng.time.percent_of_cpu_this_job_got                  
                                                                                
  300 +---------------------------------------------------------------------+   
      |                                                                     |   
      |            O                                                        |   
  250 |-+  O        O       OO                                              |   
      | O O  O          O O    O O   O  O   OO   O   O                      |   
      |        O      O            O  O   O    O   O  O                     |   
  200 |-+        O                                                          |   
      |                                                                     |   
  150 |-+                                                                   |   
      |                                                                     |   
      |                                                                     |   
  100 |-+                                                                   |   
      |                                                                     |   
      |.+.++.+.+.+.++.+.+.+.++.+.+.+.++.+.+.++.+.+.+.++.   .+.++.+.+.+. +.+.|   
   50 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                      stress-ng.time.voluntary_context_switches                 
                                                                                
  160000 +------------------------------------------------------------------+   
         | O  O                                                             |   
  140000 |-+O   O O  O O  O O OO O OO O O OO O O OO O OO                    |   
  120000 |-+       O    O                                                   |   
         |                                                                  |   
  100000 |-+                                                                |   
         |                                                                  |   
   80000 |-+                                                                |   
         |                                                                  |   
   60000 |-+                                                                |   
   40000 |-+                                                                |   
         |                                                                  |   
   20000 |.++.+.+.++.+.++.+.+.++.+.++.+.+.++.+.+.++.+.++.+.+.++.+.++.+.+.++.|   
         |                                                                  |   
       0 +------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                    stress-ng.time.involuntary_context_switches                 
                                                                                
  7500 +--------------------------------------------------------------------+   
       |                  O                                                 |   
  7000 |-+                                                                  |   
       |                      O O O    O OO O O    O O O                    |   
  6500 |-+ OO O O  O O O O  O      O O          OO                          |   
       | O        O                                                         |   
  6000 |-+                                                                  |   
       |                                                                    |   
  5500 |-+                                                                  |   
       |                                                                    |   
  5000 |.+                                                                  |   
       | +.                           .+. +.     +.              +          |   
  4500 |-+ ++.+.+. +. .+.++.+.+.+.++.+   +  +.+.+  +.+.+.++.+.+.+ + .+.++.+.|   
       |          +  +                                             +        |   
  4000 +--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                stress-ng.loop.ops                              
                                                                                
  5500 +--------------------------------------------------------------------+   
       |                      O                                             |   
  5000 |-O OO O O  O O   OO O   O  O O O OO O   OO O O O                    |   
  4500 |-+        O    O          O           O                             |   
       |                                                                    |   
  4000 |-+                                                                  |   
       |                                                                    |   
  3500 |-+                                                                  |   
       |                                                                    |   
  3000 |-+                                                                  |   
  2500 |-+                                                                  |   
       |                                                                    |   
  2000 |.+.++.+.+.++. .+.++.+.+.+. +.+.+. +.+.+.++.+.+.+. +.+.+.++.+.+.+ .+.|   
       |             +            +      +               +              +   |   
  1500 +--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                           stress-ng.loop.ops_per_sec                           
                                                                                
  90 +----------------------------------------------------------------------+   
     | O                                                                    |   
  80 |-+ OO O     O    O O   OO   O O O  O O   O O O  O                     |   
     |        O O   OO     O    O       O    O      O                       |   
     |                                                                      |   
  70 |-+                                                                    |   
     |                                                                      |   
  60 |-+                                                                    |   
     |                                                                      |   
  50 |-+                                                                    |   
     |                                                                      |   
     |                                                                      |   
  40 |-+                                                                    |   
     |.+.+ .+.+.+.   +.+.     +. .+.+.+. +.+.+. .+.++.     .+ .+.+. .+.    .|   
  30 +----------------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


---
0DAY/LKP+ Test Infrastructure                   Open Source Technology Center
https://lists.01.org/hyperkitty/list/lkp@lists.01.org       Intel Corporation

Thanks,
Oliver Sang


View attachment "config-5.14.0-rc5-00006-g8883efd909da" of type "text/plain" (175472 bytes)

View attachment "job-script" of type "text/plain" (8516 bytes)

View attachment "job.yaml" of type "text/plain" (5625 bytes)

View attachment "reproduce" of type "text/plain" (337 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ