lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite for Android: free password hash cracker in your pocket
[<prev] [next>] [day] [month] [year] [list]
Message-ID: <20210901034658.GF14661@xsang-OptiPlex-9020>
Date:   Wed, 1 Sep 2021 11:46:58 +0800
From:   kernel test robot <oliver.sang@...el.com>
To:     Tetsuo Handa <penguin-kernel@...ove.sakura.ne.jp>
Cc:     0day robot <lkp@...el.com>,
        syzbot <syzbot+f61766d5763f9e7a118f@...kaller.appspotmail.com>,
        Tetsuo Handa <penguin-kernel@...ove.SAKURA.ne.jp>,
        LKML <linux-kernel@...r.kernel.org>, lkp@...ts.01.org,
        ying.huang@...el.com, feng.tang@...el.com,
        zhengjun.xing@...ux.intel.com
Subject: [loop]  acd1746478:  stress-ng.loop.ops_per_sec 140.9% improvement



Greeting,

FYI, we noticed a 140.9% improvement of stress-ng.loop.ops_per_sec due to commit:


commit: acd17464785f83a65c057d7a1a4918dd585e85e5 ("loop: reduce the loop_ctl_mutex scope")
url: https://github.com/0day-ci/linux/commits/UPDATE-20210829-214852/Tetsuo-Handa/loop-replace-loop_ctl_mutex-with-loop_idr_spinlock/20210827-000519


in testcase: stress-ng
on test machine: 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 512G memory
with following parameters:

	nr_threads: 100%
	disk: 1HDD
	testtime: 60s
	class: device
	test: loop
	cpufreq_governor: performance
	ucode: 0x5003006






Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install                job.yaml  # job file is attached in this email
        bin/lkp split-job --compatible job.yaml  # generate the yaml file for lkp run
        bin/lkp run                    generated-yaml-file

=========================================================================================
class/compiler/cpufreq_governor/disk/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime/ucode:
  device/gcc-9/performance/1HDD/x86_64-rhel-8.3/100%/debian-10.4-x86_64-20200603.cgz/lkp-csl-2sp7/loop/stress-ng/60s/0x5003006

commit: 
  c42dd069be ("configfs: fix a race in configfs_lookup()")
  acd1746478 ("loop: reduce the loop_ctl_mutex scope")

c42dd069be8dfc9b acd17464785f83a65c057d7a1a4 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
      1962          +140.1%       4710        stress-ng.loop.ops
     32.14          +140.9%      77.41        stress-ng.loop.ops_per_sec
      4420 ±  2%     +48.4%       6559        stress-ng.time.involuntary_context_switches
     12367            +5.6%      13066        stress-ng.time.minor_page_faults
     59.00 ±  2%    +284.0%     226.57 ±  5%  stress-ng.time.percent_of_cpu_this_job_got
     37.76          +280.0%     143.50 ±  5%  stress-ng.time.system_time
     19161          +532.8%     121256        stress-ng.time.voluntary_context_switches
     97.63            -2.0%      95.68        iostat.cpu.idle
      2.21 ±  2%     +77.8%       3.92 ±  4%  iostat.cpu.system
      2495 ± 22%     +55.4%       3878 ± 13%  numa-meminfo.node1.Active
      2454 ± 20%     +53.3%       3762 ± 12%  numa-meminfo.node1.Active(anon)
    613.14 ± 20%     +53.3%     940.14 ± 12%  numa-vmstat.node1.nr_active_anon
    613.14 ± 20%     +53.3%     940.14 ± 12%  numa-vmstat.node1.nr_zone_active_anon
    206179 ± 28%     +92.2%     396268 ±  5%  numa-numastat.node0.local_node
    258419 ± 14%     +72.3%     445222 ±  9%  numa-numastat.node0.numa_hit
    309280 ± 13%     +26.8%     392288 ± 12%  numa-numastat.node1.numa_hit
      1.14 ±  2%      -0.2        0.93 ±  7%  mpstat.cpu.all.irq%
      0.08 ±  5%      +0.0        0.09 ±  3%  mpstat.cpu.all.soft%
      0.99            +2.0        2.94 ±  5%  mpstat.cpu.all.sys%
      0.15 ±  9%      +0.2        0.39        mpstat.cpu.all.usr%
      2635          +149.9%       6585        vmstat.io.bi
     23.00 ± 19%    +251.6%      80.86 ± 21%  vmstat.memory.buff
      1.43 ± 34%    +290.0%       5.57 ± 18%  vmstat.procs.r
     10370           +95.7%      20296        vmstat.system.cs
     79.00 ±  2%    +102.5%     160.00 ±  2%  turbostat.Avg_MHz
      3.80 ±  3%      +2.4        6.19 ±  3%  turbostat.Busy%
      2081           +24.4%       2588        turbostat.Bzy_MHz
     90485 ±  5%    +209.2%     279779 ± 47%  turbostat.C1
      0.13 ±  3%      +0.2        0.34 ± 36%  turbostat.C1%
     50.57 ±  2%      +6.8%      54.00        turbostat.PkgTmp
    136.77           +12.1%     153.32        turbostat.PkgWatt
     23392          +129.2%      53623        softirqs.BLOCK
     10071 ±  4%     -45.8%       5462 ±  9%  softirqs.CPU0.RCU
      9338 ± 20%     -61.6%       3582 ± 14%  softirqs.CPU2.RCU
      7359 ± 15%     -61.4%       2841 ± 12%  softirqs.CPU28.RCU
      8626 ± 15%     -59.2%       3519 ± 20%  softirqs.CPU3.RCU
      7909 ± 19%     -64.7%       2789 ± 12%  softirqs.CPU64.RCU
      9509 ±  8%     -18.9%       7710 ± 12%  softirqs.CPU75.SCHED
      8593 ± 20%     -66.4%       2884 ±  9%  softirqs.CPU93.RCU
    718759 ±  2%     -59.0%     294408        softirqs.RCU
     27494           +21.3%      33342        softirqs.TIMER
      4090           +32.3%       5412 ±  6%  meminfo.Active
      3846           +33.8%       5145 ±  6%  meminfo.Active(anon)
    277937           +18.9%     330394 ±  2%  meminfo.AnonPages
   1425626           +13.9%    1624227        meminfo.Committed_AS
    290196           +19.1%     345650 ±  2%  meminfo.Inactive
    290048           +19.1%     345453 ±  2%  meminfo.Inactive(anon)
    140159           +14.7%     160788        meminfo.KReclaimable
     10869 ±  2%     +41.5%      15384 ±  2%  meminfo.PageTables
     54407           +20.5%      65575        meminfo.Percpu
    140159           +14.7%     160788        meminfo.SReclaimable
    220411           +10.1%     242578        meminfo.SUnreclaim
     15777           +25.7%      19839        meminfo.Shmem
    360571           +11.9%     403367        meminfo.Slab
    959.43           +34.0%       1286 ±  6%  proc-vmstat.nr_active_anon
     69494           +18.9%      82600 ±  2%  proc-vmstat.nr_anon_pages
     72522           +19.1%      86370 ±  2%  proc-vmstat.nr_inactive_anon
     19951            +7.4%      21423        proc-vmstat.nr_kernel_stack
      9753            +4.3%      10174        proc-vmstat.nr_mapped
      2719 ±  2%     +41.4%       3845 ±  2%  proc-vmstat.nr_page_table_pages
      3943           +25.8%       4962        proc-vmstat.nr_shmem
     35035           +14.7%      40199        proc-vmstat.nr_slab_reclaimable
     55101           +10.1%      60648        proc-vmstat.nr_slab_unreclaimable
    959.43           +34.0%       1286 ±  6%  proc-vmstat.nr_zone_active_anon
     72522           +19.1%      86370 ±  2%  proc-vmstat.nr_zone_inactive_anon
    569464           +47.3%     838926        proc-vmstat.numa_hit
    480108           +54.8%     743358        proc-vmstat.numa_local
     89364            +7.0%      95599        proc-vmstat.numa_other
    553.71 ± 35%    +274.9%       2075 ±111%  proc-vmstat.numa_pages_migrated
     12540          +151.9%      31595        proc-vmstat.pgactivate
    619253           +64.0%    1015469        proc-vmstat.pgalloc_normal
    343681            +4.9%     360630        proc-vmstat.pgfault
    510868           +73.5%     886253        proc-vmstat.pgfree
    553.71 ± 35%    +274.9%       2075 ±111%  proc-vmstat.pgmigrate_success
    178480          +145.1%     437452        proc-vmstat.pgpgin
     19754 ±  2%     +11.6%      22043        proc-vmstat.pgreuse
     17576 ±  3%    +162.0%      46054 ±  2%  proc-vmstat.unevictable_pgs_culled
      4704 ±  6%     +38.4%       6511        slabinfo.Acpi-Parse.active_objs
      4704 ±  6%     +38.4%       6511        slabinfo.Acpi-Parse.num_objs
      1633 ±  2%     +15.0%       1878        slabinfo.bdev_cache.active_objs
      1633 ±  2%     +15.0%       1878        slabinfo.bdev_cache.num_objs
    158044 ±  2%     +16.4%     184039        slabinfo.dentry.active_objs
      3783 ±  2%     +16.3%       4399        slabinfo.dentry.active_slabs
    158900 ±  2%     +16.3%     184790        slabinfo.dentry.num_objs
      3783 ±  2%     +16.3%       4399        slabinfo.dentry.num_slabs
      2779 ±  3%     +20.8%       3356        slabinfo.file_lock_cache.active_objs
      2779 ±  3%     +20.8%       3356        slabinfo.file_lock_cache.num_objs
     29944 ±  2%     +58.2%      47361        slabinfo.filp.active_objs
    953.86 ±  2%     +55.7%       1485        slabinfo.filp.active_slabs
     30540 ±  2%     +55.7%      47540        slabinfo.filp.num_objs
    953.86 ±  2%     +55.7%       1485        slabinfo.filp.num_slabs
      9178 ±  4%     +26.4%      11604        slabinfo.fsnotify_mark_connector.active_objs
      9178 ±  4%     +26.4%      11604        slabinfo.fsnotify_mark_connector.num_objs
    132554           +16.9%     154898        slabinfo.inode_cache.active_objs
      2415           +17.2%       2830        slabinfo.inode_cache.active_slabs
    132889           +17.2%     155722        slabinfo.inode_cache.num_objs
      2415           +17.2%       2830        slabinfo.inode_cache.num_slabs
      6800           +22.2%       8312        slabinfo.kmalloc-1k.active_objs
      7530           +23.0%       9263        slabinfo.kmalloc-1k.num_objs
     11435 ±  2%     +26.6%      14476 ±  5%  slabinfo.kmalloc-256.active_objs
    357.00 ±  2%     +28.1%     457.43 ±  5%  slabinfo.kmalloc-256.active_slabs
     11437 ±  2%     +28.1%      14651 ±  5%  slabinfo.kmalloc-256.num_objs
    357.00 ±  2%     +28.1%     457.43 ±  5%  slabinfo.kmalloc-256.num_slabs
     18404 ±  3%      +8.7%      19997 ±  2%  slabinfo.kmalloc-512.num_objs
     75207           +27.3%      95720 ±  2%  slabinfo.kmalloc-8.active_objs
     75607           +26.9%      95961 ±  2%  slabinfo.kmalloc-8.num_objs
      3496 ±  4%     +67.2%       5846 ±  5%  slabinfo.kmalloc-cg-1k.active_objs
      3496 ±  4%     +70.7%       5970 ±  5%  slabinfo.kmalloc-cg-1k.num_objs
    855.29 ±  7%    +136.2%       2019 ± 10%  slabinfo.kmalloc-cg-2k.active_objs
    855.29 ±  7%    +141.4%       2064 ± 10%  slabinfo.kmalloc-cg-2k.num_objs
     12227 ±  5%     +88.6%      23057 ± 11%  slabinfo.kmalloc-rcl-64.active_objs
    190.43 ±  5%     +89.0%     359.86 ± 11%  slabinfo.kmalloc-rcl-64.active_slabs
     12233 ±  5%     +88.6%      23067 ± 11%  slabinfo.kmalloc-rcl-64.num_objs
    190.43 ±  5%     +89.0%     359.86 ± 11%  slabinfo.kmalloc-rcl-64.num_slabs
      3583 ±  5%     +59.0%       5696 ± 10%  slabinfo.kmalloc-rcl-96.active_objs
      3583 ±  5%     +59.0%       5696 ± 10%  slabinfo.kmalloc-rcl-96.num_objs
      2923           +10.1%       3217        slabinfo.pool_workqueue.active_objs
      1158 ±  2%     +15.4%       1337        slabinfo.request_queue.active_objs
      1158 ±  2%     +15.4%       1337        slabinfo.request_queue.num_objs
      6066 ±  2%     +26.0%       7645        slabinfo.shmem_inode_cache.active_objs
      6066 ±  2%     +26.0%       7645        slabinfo.shmem_inode_cache.num_objs
      3264 ±  7%     +29.3%       4222 ±  8%  slabinfo.skbuff_ext_cache.active_objs
      3264 ±  7%     +30.3%       4252 ±  9%  slabinfo.skbuff_ext_cache.num_objs
      4393 ±  4%     +94.2%       8532 ±  8%  slabinfo.skbuff_head_cache.active_objs
    136.86 ±  4%     +97.2%     269.86 ±  8%  slabinfo.skbuff_head_cache.active_slabs
      4393 ±  4%     +96.9%       8652 ±  8%  slabinfo.skbuff_head_cache.num_objs
    136.86 ±  4%     +97.2%     269.86 ±  8%  slabinfo.skbuff_head_cache.num_slabs
      3845 ±  3%     +24.9%       4802        slabinfo.task_delay_info.active_objs
      3845 ±  3%     +24.9%       4802        slabinfo.task_delay_info.num_objs
      4763           +23.9%       5902        slabinfo.trace_event_file.active_objs
      4763           +23.9%       5902        slabinfo.trace_event_file.num_objs
     62874 ±  4%     +15.9%      72865 ±  5%  slabinfo.vm_area_struct.active_objs
      1572 ±  4%     +16.0%       1822 ±  5%  slabinfo.vm_area_struct.active_slabs
     62899 ±  4%     +16.0%      72934 ±  5%  slabinfo.vm_area_struct.num_objs
      1572 ±  4%     +16.0%       1822 ±  5%  slabinfo.vm_area_struct.num_slabs
 5.019e+08 ±  3%    +155.5%  1.283e+09 ±  2%  perf-stat.i.branch-instructions
   9083563 ± 14%    +123.5%   20298593 ±  6%  perf-stat.i.branch-misses
     15.24 ± 15%      +6.4       21.65 ±  8%  perf-stat.i.cache-miss-rate%
   2868120 ±  9%    +276.3%   10793991 ±  8%  perf-stat.i.cache-misses
  19611364 ± 26%    +166.4%   52242037 ± 11%  perf-stat.i.cache-references
     10276          +127.6%      23389 ±  4%  perf-stat.i.context-switches
      3.18 ±  3%      -7.1%       2.95 ±  2%  perf-stat.i.cpi
     96024           +10.0%     105591 ±  3%  perf-stat.i.cpu-clock
 7.028e+09 ±  3%    +143.3%   1.71e+10 ±  4%  perf-stat.i.cpu-cycles
    242.69 ±  5%    +206.5%     743.77 ±  3%  perf-stat.i.cpu-migrations
      2567 ±  7%     -30.8%       1777 ±  5%  perf-stat.i.cycles-between-cache-misses
    267918 ± 68%    +218.0%     851998 ± 16%  perf-stat.i.dTLB-load-misses
 5.822e+08          +144.1%  1.421e+09 ±  2%  perf-stat.i.dTLB-loads
      0.03 ± 22%      -0.0        0.02 ± 12%  perf-stat.i.dTLB-store-miss-rate%
     82103 ± 22%     +47.6%     121151 ± 12%  perf-stat.i.dTLB-store-misses
 2.691e+08          +123.3%   6.01e+08 ±  4%  perf-stat.i.dTLB-stores
     39.18 ±  3%      +6.1       45.28 ±  2%  perf-stat.i.iTLB-load-miss-rate%
   1094785 ±  4%     +50.8%    1650968 ±  4%  perf-stat.i.iTLB-load-misses
   1694815 ±  3%     +15.7%    1960345 ±  4%  perf-stat.i.iTLB-loads
 2.426e+09 ±  3%    +150.9%  6.088e+09 ±  3%  perf-stat.i.instructions
      2217 ±  6%     +68.9%       3744 ±  3%  perf-stat.i.instructions-per-iTLB-miss
      0.34 ±  3%      +9.2%       0.37 ±  2%  perf-stat.i.ipc
     11.86           +67.2%      19.83 ±  6%  perf-stat.i.major-faults
      0.07 ±  3%    +119.6%       0.16 ±  3%  perf-stat.i.metric.GHz
    234.23 ± 22%    +138.3%     558.28 ±  8%  perf-stat.i.metric.K/sec
     14.08 ±  2%    +122.6%      31.35        perf-stat.i.metric.M/sec
      3805           +41.8%       5396 ±  3%  perf-stat.i.minor-faults
     72.35            +3.6       75.98        perf-stat.i.node-load-miss-rate%
    505280 ±  5%    +368.9%    2369224 ±  7%  perf-stat.i.node-load-misses
    191531 ±  3%    +267.9%     704691 ±  6%  perf-stat.i.node-loads
     67.49 ±  2%      +3.0       70.46        perf-stat.i.node-store-miss-rate%
    320007 ±  7%    +429.4%    1694177 ±  7%  perf-stat.i.node-store-misses
    147930 ±  7%    +334.5%     642742 ±  5%  perf-stat.i.node-stores
      3866           +43.6%       5553 ±  3%  perf-stat.i.page-faults
     96024           +10.0%     105591 ±  3%  perf-stat.i.task-clock
      1.80 ± 12%      -0.3        1.49 ±  6%  perf-stat.overall.branch-miss-rate%
     15.16 ± 15%      +5.4       20.58 ± 12%  perf-stat.overall.cache-miss-rate%
      2464 ±  6%     -35.7%       1585 ±  5%  perf-stat.overall.cycles-between-cache-misses
      0.03 ± 22%      -0.0        0.02 ± 12%  perf-stat.overall.dTLB-store-miss-rate%
     39.24 ±  3%      +6.3       45.49 ±  3%  perf-stat.overall.iTLB-load-miss-rate%
      2220 ±  6%     +58.7%       3524 ±  3%  perf-stat.overall.instructions-per-iTLB-miss
     72.48            +4.8       77.30        perf-stat.overall.node-load-miss-rate%
     68.38 ±  2%      +4.2       72.55        perf-stat.overall.node-store-miss-rate%
 4.938e+08 ±  3%    +115.9%  1.066e+09 ±  2%  perf-stat.ps.branch-instructions
   8925789 ± 14%     +78.2%   15904236 ±  5%  perf-stat.ps.branch-misses
   2823831 ±  9%    +225.1%    9178906 ±  4%  perf-stat.ps.cache-misses
  19305594 ± 26%    +133.9%   45146871 ±  9%  perf-stat.ps.cache-references
     10118           +99.4%      20178        perf-stat.ps.context-switches
     94536            -1.1%      93528        perf-stat.ps.cpu-clock
  6.92e+09 ±  3%    +109.8%  1.452e+10 ±  3%  perf-stat.ps.cpu-cycles
    239.13 ±  5%    +166.2%     636.50 ±  2%  perf-stat.ps.cpu-migrations
    263726 ± 68%    +179.2%     736415 ± 17%  perf-stat.ps.dTLB-load-misses
 5.728e+08          +107.3%  1.187e+09 ±  2%  perf-stat.ps.dTLB-loads
 2.648e+08           +91.2%  5.065e+08        perf-stat.ps.dTLB-stores
   1077869 ±  4%     +32.9%    1432036 ±  2%  perf-stat.ps.iTLB-load-misses
 2.387e+09 ±  3%    +111.3%  5.044e+09 ±  2%  perf-stat.ps.instructions
      3735            +6.9%       3994        perf-stat.ps.minor-faults
    497490 ±  5%    +313.3%    2056087 ±  5%  perf-stat.ps.node-load-misses
    188519 ±  3%    +219.7%     602721 ±  2%  perf-stat.ps.node-loads
    315109 ±  7%    +355.8%    1436208 ±  3%  perf-stat.ps.node-store-misses
    145564 ±  7%    +272.9%     542765        perf-stat.ps.node-stores
      3795            +8.7%       4124        perf-stat.ps.page-faults
     94536            -1.1%      93528        perf-stat.ps.task-clock
  1.56e+11 ±  2%    +106.8%  3.226e+11 ±  3%  perf-stat.total.instructions
     68672 ±  2%     +32.1%      90720        interrupts.CAL:Function_call_interrupts
    662.14 ±  8%     +40.0%     927.29 ±  3%  interrupts.CPU10.CAL:Function_call_interrupts
    668.00 ±  5%     +40.1%     936.14 ±  4%  interrupts.CPU11.CAL:Function_call_interrupts
    661.43 ±  4%     +40.9%     931.71 ±  5%  interrupts.CPU12.CAL:Function_call_interrupts
     26.14 ± 33%    +235.0%      87.57 ± 13%  interrupts.CPU12.RES:Rescheduling_interrupts
    648.29 ±  6%     +48.9%     965.00 ±  5%  interrupts.CPU13.CAL:Function_call_interrupts
    665.29 ± 10%     +43.4%     953.86 ±  7%  interrupts.CPU14.CAL:Function_call_interrupts
    678.00 ±  7%     +38.3%     937.57 ±  3%  interrupts.CPU15.CAL:Function_call_interrupts
    693.00 ± 14%     +34.1%     929.43 ±  5%  interrupts.CPU16.CAL:Function_call_interrupts
     39.29 ± 65%    +136.7%      93.00 ± 28%  interrupts.CPU16.RES:Rescheduling_interrupts
    695.29 ±  9%     +34.3%     933.57 ±  3%  interrupts.CPU17.CAL:Function_call_interrupts
    702.86 ±  8%     +39.3%     978.86 ±  6%  interrupts.CPU18.CAL:Function_call_interrupts
    677.57 ± 15%     +38.2%     936.14 ±  6%  interrupts.CPU19.CAL:Function_call_interrupts
    672.00 ±  9%     +38.0%     927.43 ±  4%  interrupts.CPU20.CAL:Function_call_interrupts
    680.14 ± 14%     +39.2%     946.71 ±  6%  interrupts.CPU21.CAL:Function_call_interrupts
    656.57 ±  8%     +43.6%     942.57 ±  6%  interrupts.CPU22.CAL:Function_call_interrupts
    203.43 ± 33%    +184.5%     578.71 ± 89%  interrupts.CPU22.NMI:Non-maskable_interrupts
    203.43 ± 33%    +184.5%     578.71 ± 89%  interrupts.CPU22.PMI:Performance_monitoring_interrupts
    679.43 ± 12%     +39.1%     944.86 ±  6%  interrupts.CPU23.CAL:Function_call_interrupts
     34.00 ± 34%    +150.0%      85.00 ± 13%  interrupts.CPU23.RES:Rescheduling_interrupts
    657.86 ±  7%     +45.2%     955.29 ±  3%  interrupts.CPU24.CAL:Function_call_interrupts
    192.71 ± 44%    +253.7%     681.71 ± 49%  interrupts.CPU24.NMI:Non-maskable_interrupts
    192.71 ± 44%    +253.7%     681.71 ± 49%  interrupts.CPU24.PMI:Performance_monitoring_interrupts
    698.43 ± 10%     +31.9%     921.14 ±  4%  interrupts.CPU25.CAL:Function_call_interrupts
    181.14 ± 34%    +280.5%     689.29 ±119%  interrupts.CPU26.NMI:Non-maskable_interrupts
    181.14 ± 34%    +280.5%     689.29 ±119%  interrupts.CPU26.PMI:Performance_monitoring_interrupts
    636.43 ±  7%     +43.6%     913.86 ±  6%  interrupts.CPU27.CAL:Function_call_interrupts
     20.86 ± 63%    +273.3%      77.86 ± 29%  interrupts.CPU27.RES:Rescheduling_interrupts
     18.00 ± 52%    +345.2%      80.14 ± 11%  interrupts.CPU29.RES:Rescheduling_interrupts
     31.43 ± 55%    +173.6%      86.00 ± 16%  interrupts.CPU3.RES:Rescheduling_interrupts
     16.57 ± 47%    +392.2%      81.57 ± 22%  interrupts.CPU30.RES:Rescheduling_interrupts
    666.86 ±  9%     +43.8%     959.14 ± 18%  interrupts.CPU31.CAL:Function_call_interrupts
    206.43 ± 33%    +604.6%       1454 ±187%  interrupts.CPU32.NMI:Non-maskable_interrupts
    206.43 ± 33%    +604.6%       1454 ±187%  interrupts.CPU32.PMI:Performance_monitoring_interrupts
    681.86 ±  9%     +37.1%     934.71 ±  4%  interrupts.CPU33.CAL:Function_call_interrupts
    206.29 ± 34%     +81.7%     374.86 ± 50%  interrupts.CPU33.NMI:Non-maskable_interrupts
    206.29 ± 34%     +81.7%     374.86 ± 50%  interrupts.CPU33.PMI:Performance_monitoring_interrupts
     19.43 ± 32%    +311.8%      80.00 ± 24%  interrupts.CPU33.RES:Rescheduling_interrupts
    684.29 ±  7%     +34.5%     920.14 ±  7%  interrupts.CPU35.CAL:Function_call_interrupts
    220.00 ± 24%    +138.8%     525.29 ± 67%  interrupts.CPU35.NMI:Non-maskable_interrupts
    220.00 ± 24%    +138.8%     525.29 ± 67%  interrupts.CPU35.PMI:Performance_monitoring_interrupts
    700.43 ± 15%     +32.7%     929.43 ±  4%  interrupts.CPU36.CAL:Function_call_interrupts
     22.57 ± 16%    +241.8%      77.14 ± 23%  interrupts.CPU36.RES:Rescheduling_interrupts
    642.00 ±  7%     +47.5%     947.14 ±  3%  interrupts.CPU38.CAL:Function_call_interrupts
     20.29 ± 23%    +316.2%      84.43 ± 21%  interrupts.CPU38.RES:Rescheduling_interrupts
    649.86 ±  7%     +37.9%     895.86 ±  5%  interrupts.CPU41.CAL:Function_call_interrupts
     22.43 ± 23%    +242.7%      76.86 ± 19%  interrupts.CPU41.RES:Rescheduling_interrupts
     25.14 ± 55%    +226.1%      82.00 ± 20%  interrupts.CPU44.RES:Rescheduling_interrupts
     30.29 ± 67%    +178.8%      84.43 ± 17%  interrupts.CPU46.RES:Rescheduling_interrupts
    689.00 ±  6%     +36.4%     939.71 ±  4%  interrupts.CPU48.CAL:Function_call_interrupts
    676.71 ± 10%     +34.2%     908.14 ±  4%  interrupts.CPU49.CAL:Function_call_interrupts
    647.14 ±  7%     +44.2%     933.43 ±  4%  interrupts.CPU50.CAL:Function_call_interrupts
    686.29 ±  9%     +33.2%     914.29 ±  3%  interrupts.CPU51.CAL:Function_call_interrupts
    656.43 ±  6%     +42.1%     932.57 ±  7%  interrupts.CPU52.CAL:Function_call_interrupts
    651.43 ±  5%     +42.3%     926.86 ±  5%  interrupts.CPU53.CAL:Function_call_interrupts
    688.29 ±  6%     +31.9%     908.00 ±  5%  interrupts.CPU54.CAL:Function_call_interrupts
    631.43 ±  3%     +44.3%     911.29 ±  5%  interrupts.CPU55.CAL:Function_call_interrupts
    286.29 ± 33%     +61.6%     462.57 ± 28%  interrupts.CPU56.NMI:Non-maskable_interrupts
    286.29 ± 33%     +61.6%     462.57 ± 28%  interrupts.CPU56.PMI:Performance_monitoring_interrupts
    722.14 ± 10%     +27.9%     923.86 ±  8%  interrupts.CPU57.CAL:Function_call_interrupts
    218.14 ± 52%    +252.5%     768.86 ± 69%  interrupts.CPU57.NMI:Non-maskable_interrupts
    218.14 ± 52%    +252.5%     768.86 ± 69%  interrupts.CPU57.PMI:Performance_monitoring_interrupts
    678.14 ±  6%     +33.1%     902.57 ±  8%  interrupts.CPU58.CAL:Function_call_interrupts
    664.29 ±  8%     +32.8%     882.00        interrupts.CPU59.CAL:Function_call_interrupts
    641.14 ±  4%     +38.9%     890.43 ±  2%  interrupts.CPU60.CAL:Function_call_interrupts
    251.57 ± 35%    +259.4%     904.14 ± 94%  interrupts.CPU60.NMI:Non-maskable_interrupts
    251.57 ± 35%    +259.4%     904.14 ± 94%  interrupts.CPU60.PMI:Performance_monitoring_interrupts
    654.14 ±  5%     +38.5%     906.00 ±  8%  interrupts.CPU61.CAL:Function_call_interrupts
    633.57 ±  5%     +48.8%     942.71 ±  2%  interrupts.CPU62.CAL:Function_call_interrupts
    631.86 ±  8%     +39.7%     882.43 ±  3%  interrupts.CPU63.CAL:Function_call_interrupts
    643.86 ±  6%     +43.6%     924.29 ±  3%  interrupts.CPU64.CAL:Function_call_interrupts
    635.86 ±  2%     +45.5%     925.43 ±  4%  interrupts.CPU65.CAL:Function_call_interrupts
    620.00 ±  4%     +47.2%     912.43 ±  6%  interrupts.CPU66.CAL:Function_call_interrupts
    614.14 ±  4%     +47.4%     905.29 ±  7%  interrupts.CPU67.CAL:Function_call_interrupts
    596.14 ±  6%     +54.5%     921.29 ±  2%  interrupts.CPU68.CAL:Function_call_interrupts
    645.29 ± 11%     +43.2%     924.14 ±  4%  interrupts.CPU69.CAL:Function_call_interrupts
    631.71 ±  9%     +93.3%       1221 ± 42%  interrupts.CPU7.CAL:Function_call_interrupts
    672.43 ±  7%     +35.2%     909.00 ±  6%  interrupts.CPU70.CAL:Function_call_interrupts
    635.86 ±  7%     +44.9%     921.57 ±  4%  interrupts.CPU71.CAL:Function_call_interrupts
    686.57 ± 19%     +32.0%     906.14 ±  3%  interrupts.CPU74.CAL:Function_call_interrupts
    157.29 ± 22%    +772.0%       1371 ±171%  interrupts.CPU74.NMI:Non-maskable_interrupts
    157.29 ± 22%    +772.0%       1371 ±171%  interrupts.CPU74.PMI:Performance_monitoring_interrupts
    730.86 ± 12%     +22.6%     896.29 ±  3%  interrupts.CPU76.CAL:Function_call_interrupts
     19.00 ± 74%    +318.0%      79.43 ± 21%  interrupts.CPU79.RES:Rescheduling_interrupts
    633.43 ±  6%     +46.7%     929.14 ±  7%  interrupts.CPU8.CAL:Function_call_interrupts
    655.57 ± 18%     +50.8%     988.43 ±  4%  interrupts.CPU81.CAL:Function_call_interrupts
    195.43 ± 25%     +90.4%     372.00 ± 34%  interrupts.CPU81.NMI:Non-maskable_interrupts
    195.43 ± 25%     +90.4%     372.00 ± 34%  interrupts.CPU81.PMI:Performance_monitoring_interrupts
    685.57 ±  9%     +39.7%     957.86 ±  5%  interrupts.CPU82.CAL:Function_call_interrupts
    672.86 ± 13%     +44.2%     970.14 ±  7%  interrupts.CPU83.CAL:Function_call_interrupts
    223.86 ± 34%     +73.1%     387.43 ± 32%  interrupts.CPU83.NMI:Non-maskable_interrupts
    223.86 ± 34%     +73.1%     387.43 ± 32%  interrupts.CPU83.PMI:Performance_monitoring_interrupts
     18.29 ± 66%    +328.9%      78.43 ± 19%  interrupts.CPU83.RES:Rescheduling_interrupts
    683.71 ± 12%     +39.3%     952.71 ±  5%  interrupts.CPU86.CAL:Function_call_interrupts
    194.43 ± 32%    +154.4%     494.57 ± 77%  interrupts.CPU86.NMI:Non-maskable_interrupts
    194.43 ± 32%    +154.4%     494.57 ± 77%  interrupts.CPU86.PMI:Performance_monitoring_interrupts
    713.43 ± 11%     +31.4%     937.29 ±  5%  interrupts.CPU87.CAL:Function_call_interrupts
    226.29 ± 25%    +101.3%     455.43 ± 50%  interrupts.CPU88.NMI:Non-maskable_interrupts
    226.29 ± 25%    +101.3%     455.43 ± 50%  interrupts.CPU88.PMI:Performance_monitoring_interrupts
     25.29 ± 41%    +205.6%      77.29 ± 24%  interrupts.CPU88.RES:Rescheduling_interrupts
    688.00 ± 10%     +42.1%     977.86 ±  9%  interrupts.CPU89.CAL:Function_call_interrupts
    666.71 ± 12%     +40.8%     938.71 ±  5%  interrupts.CPU9.CAL:Function_call_interrupts
    237.86 ± 52%    +431.7%       1264 ±119%  interrupts.CPU9.NMI:Non-maskable_interrupts
    237.86 ± 52%    +431.7%       1264 ±119%  interrupts.CPU9.PMI:Performance_monitoring_interrupts
    680.86 ± 20%     +40.8%     958.57 ±  5%  interrupts.CPU90.CAL:Function_call_interrupts
    688.86 ± 11%     +39.3%     959.57 ±  4%  interrupts.CPU91.CAL:Function_call_interrupts
    734.29 ± 12%     +31.1%     962.57 ±  6%  interrupts.CPU92.CAL:Function_call_interrupts
    718.86 ± 21%     +34.3%     965.57 ±  5%  interrupts.CPU93.CAL:Function_call_interrupts
    692.14 ± 12%     +36.0%     941.57 ±  9%  interrupts.CPU95.CAL:Function_call_interrupts
     24378 ±  5%    +104.1%      49762 ± 13%  interrupts.NMI:Non-maskable_interrupts
     24378 ±  5%    +104.1%      49762 ± 13%  interrupts.PMI:Performance_monitoring_interrupts
      2814 ±  6%    +153.3%       7129 ±  2%  interrupts.RES:Rescheduling_interrupts
     56.20 ±  3%     -14.8       41.37 ±  7%  perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
     55.59 ±  3%     -14.6       40.95 ±  7%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     55.59 ±  3%     -14.6       40.95 ±  7%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
     55.56 ±  3%     -14.6       40.94 ±  7%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     51.55 ±  3%     -12.3       39.25 ±  8%  perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     51.00 ±  4%     -12.2       38.76 ±  8%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
     39.18 ±  7%      -6.4       32.76 ± 10%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     11.35 ±  8%      -5.7        5.60 ± 13%  perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      9.24 ±  8%      -4.3        4.91 ±  9%  perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle
      6.04 ±  9%      -2.9        3.17 ± 11%  perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      5.96 ±  9%      -2.8        3.11 ± 11%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      3.15 ± 10%      -2.0        1.11 ± 12%  perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      2.72 ± 19%      -1.7        0.99 ± 16%  perf-profile.calltrace.cycles-pp.mutex_spin_on_owner.__mutex_lock.lo_ioctl.blkdev_ioctl.block_ioctl
      2.72 ± 19%      -1.7        1.00 ± 16%  perf-profile.calltrace.cycles-pp.__mutex_lock.lo_ioctl.blkdev_ioctl.block_ioctl.__x64_sys_ioctl
      3.19 ±  9%      -1.3        1.87 ± 10%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      1.86 ± 10%      -1.0        0.82 ± 22%  perf-profile.calltrace.cycles-pp.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      2.24 ±  8%      -0.9        1.33 ± 12%  perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
      1.95 ±  8%      -0.7        1.22 ± 13%  perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
      1.88 ±  8%      -0.7        1.18 ± 13%  perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
      1.56 ± 10%      -0.6        1.00 ±  5%  perf-profile.calltrace.cycles-pp.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      1.25 ± 11%      -0.4        0.84 ±  4%  perf-profile.calltrace.cycles-pp.__softirqentry_text_start.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      1.00 ±  5%      -0.3        0.67 ±  6%  perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
      0.00            +0.6        0.64 ±  6%  perf-profile.calltrace.cycles-pp.__dentry_kill.dput.simple_recursive_removal.debugfs_remove.blk_mq_debugfs_unregister_hctx
      0.00            +0.7        0.68 ±  8%  perf-profile.calltrace.cycles-pp.mnt_get_count.mntput_no_expire.simple_recursive_removal.debugfs_remove.blk_mq_debugfs_unregister_hctx
      0.00            +0.7        0.73 ± 13%  perf-profile.calltrace.cycles-pp.kernfs_remove.__kobject_del.kobject_del.blk_mq_unregister_hctx.blk_mq_unregister_dev
      0.00            +0.7        0.74 ± 20%  perf-profile.calltrace.cycles-pp.debugfs_get_inode.__debugfs_create_file.debugfs_create_files.blk_mq_debugfs_register_hctx.blk_mq_debugfs_register
      0.68 ± 11%      +0.8        1.44 ± 21%  perf-profile.calltrace.cycles-pp.unit_file_find_dropin_paths
      0.00            +0.8        0.77 ± 13%  perf-profile.calltrace.cycles-pp.__kobject_del.kobject_del.blk_mq_unregister_hctx.blk_mq_unregister_dev.blk_unregister_queue
      0.00            +0.8        0.78 ± 13%  perf-profile.calltrace.cycles-pp.kobject_del.blk_mq_unregister_hctx.blk_mq_unregister_dev.blk_unregister_queue.del_gendisk
      0.00            +0.8        0.78 ± 13%  perf-profile.calltrace.cycles-pp.blk_mq_unregister_hctx.blk_mq_unregister_dev.blk_unregister_queue.del_gendisk.loop_remove
      0.00            +0.8        0.79 ±  7%  perf-profile.calltrace.cycles-pp.dput.simple_recursive_removal.debugfs_remove.blk_mq_debugfs_unregister_hctx.blk_mq_exit_queue
      0.00            +0.8        0.81 ± 12%  perf-profile.calltrace.cycles-pp.blk_mq_unregister_dev.blk_unregister_queue.del_gendisk.loop_remove.loop_control_ioctl
      0.14 ±158%      +0.9        1.04 ± 22%  perf-profile.calltrace.cycles-pp.openat64.unit_file_find_dropin_paths
      0.00            +0.9        0.91 ± 12%  perf-profile.calltrace.cycles-pp.blk_unregister_queue.del_gendisk.loop_remove.loop_control_ioctl.__x64_sys_ioctl
      0.32 ± 87%      +0.9        1.25 ± 44%  perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.memcpy_toio.drm_fb_helper_damage_work.process_one_work.worker_thread
      0.00            +0.9        0.94 ± 22%  perf-profile.calltrace.cycles-pp.do_sys_openat2.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe.openat64
      0.00            +0.9        0.94 ± 22%  perf-profile.calltrace.cycles-pp.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe.openat64.unit_file_find_dropin_paths
      0.00            +1.0        0.99 ± 22%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.openat64.unit_file_find_dropin_paths
      0.00            +1.0        0.99 ± 22%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.openat64.unit_file_find_dropin_paths
      0.00            +1.1        1.07 ± 24%  perf-profile.calltrace.cycles-pp.start_creating.__debugfs_create_file.debugfs_create_files.blk_mq_debugfs_register_hctx.blk_mq_debugfs_register
      0.08 ±244%      +1.2        1.24 ± 24%  perf-profile.calltrace.cycles-pp.io_serial_out.uart_console_write.serial8250_console_write.console_unlock.vprintk_emit
      0.00            +1.2        1.17 ± 11%  perf-profile.calltrace.cycles-pp.del_gendisk.loop_remove.loop_control_ioctl.__x64_sys_ioctl.do_syscall_64
     11.19 ±  5%      +1.7       12.89 ± 11%  perf-profile.calltrace.cycles-pp.uart_console_write.serial8250_console_write.console_unlock.vprintk_emit.printk
      0.00            +1.9        1.93 ± 15%  perf-profile.calltrace.cycles-pp.__debugfs_create_file.debugfs_create_files.blk_mq_debugfs_register_hctx.blk_mq_debugfs_register.blk_register_queue
      0.00            +1.9        1.93 ± 15%  perf-profile.calltrace.cycles-pp.debugfs_create_files.blk_mq_debugfs_register_hctx.blk_mq_debugfs_register.blk_register_queue.__device_add_disk
      0.00            +2.3        2.27 ± 15%  perf-profile.calltrace.cycles-pp.blk_mq_debugfs_register_hctx.blk_mq_debugfs_register.blk_register_queue.__device_add_disk.loop_add
      0.00            +2.4        2.37 ± 15%  perf-profile.calltrace.cycles-pp.blk_mq_debugfs_register.blk_register_queue.__device_add_disk.loop_add.loop_control_ioctl
      0.00            +2.8        2.81 ± 13%  perf-profile.calltrace.cycles-pp.blk_register_queue.__device_add_disk.loop_add.loop_control_ioctl.__x64_sys_ioctl
      0.39 ± 63%      +2.9        3.29 ± 11%  perf-profile.calltrace.cycles-pp.loop_add.loop_control_ioctl.__x64_sys_ioctl.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +3.1        3.09 ± 12%  perf-profile.calltrace.cycles-pp.__device_add_disk.loop_add.loop_control_ioctl.__x64_sys_ioctl.do_syscall_64
      0.00            +6.7        6.67 ± 18%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.mntput_no_expire.simple_recursive_removal.debugfs_remove
      0.00            +6.8        6.79 ± 18%  perf-profile.calltrace.cycles-pp._raw_spin_lock.mntput_no_expire.simple_recursive_removal.debugfs_remove.blk_mq_debugfs_unregister_hctx
      0.00            +7.7        7.66 ± 16%  perf-profile.calltrace.cycles-pp.mntput_no_expire.simple_recursive_removal.debugfs_remove.blk_mq_debugfs_unregister_hctx.blk_mq_exit_queue
      0.63 ± 18%      +8.6        9.21 ± 14%  perf-profile.calltrace.cycles-pp.simple_recursive_removal.debugfs_remove.blk_mq_debugfs_unregister_hctx.blk_mq_exit_queue.blk_cleanup_queue
      0.63 ± 18%      +8.6        9.21 ± 14%  perf-profile.calltrace.cycles-pp.debugfs_remove.blk_mq_debugfs_unregister_hctx.blk_mq_exit_queue.blk_cleanup_queue.blk_cleanup_disk
      0.63 ± 18%      +8.6        9.22 ± 14%  perf-profile.calltrace.cycles-pp.blk_mq_debugfs_unregister_hctx.blk_mq_exit_queue.blk_cleanup_queue.blk_cleanup_disk.loop_remove
      0.63 ± 18%      +8.6        9.22 ± 14%  perf-profile.calltrace.cycles-pp.blk_mq_exit_queue.blk_cleanup_queue.blk_cleanup_disk.loop_remove.loop_control_ioctl
      0.64 ± 18%      +8.7        9.37 ± 14%  perf-profile.calltrace.cycles-pp.blk_cleanup_disk.loop_remove.loop_control_ioctl.__x64_sys_ioctl.do_syscall_64
      0.64 ± 18%      +8.7        9.37 ± 14%  perf-profile.calltrace.cycles-pp.blk_cleanup_queue.blk_cleanup_disk.loop_remove.loop_control_ioctl.__x64_sys_ioctl
      0.75 ± 17%     +10.3       11.04 ± 13%  perf-profile.calltrace.cycles-pp.loop_remove.loop_control_ioctl.__x64_sys_ioctl.do_syscall_64.entry_SYSCALL_64_after_hwframe
     19.86 ±  5%     +11.8       31.65 ±  9%  perf-profile.calltrace.cycles-pp.__x64_sys_ioctl.do_syscall_64.entry_SYSCALL_64_after_hwframe
     24.70 ±  3%     +12.5       37.24 ± 10%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
     24.71 ±  3%     +12.5       37.24 ± 10%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
      1.44 ±  9%     +13.0       14.45 ± 13%  perf-profile.calltrace.cycles-pp.loop_control_ioctl.__x64_sys_ioctl.do_syscall_64.entry_SYSCALL_64_after_hwframe
     56.20 ±  3%     -14.8       41.37 ±  7%  perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     56.20 ±  3%     -14.8       41.37 ±  7%  perf-profile.children.cycles-pp.cpu_startup_entry
     56.20 ±  3%     -14.8       41.37 ±  7%  perf-profile.children.cycles-pp.do_idle
     55.59 ±  3%     -14.6       40.95 ±  7%  perf-profile.children.cycles-pp.start_secondary
     52.12 ±  3%     -12.5       39.66 ±  7%  perf-profile.children.cycles-pp.cpuidle_enter
     52.09 ±  3%     -12.5       39.64 ±  7%  perf-profile.children.cycles-pp.cpuidle_enter_state
     39.44 ±  7%      -6.5       32.93 ±  9%  perf-profile.children.cycles-pp.intel_idle
     11.13 ±  7%      -4.6        6.51 ± 10%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
      9.56 ±  8%      -4.3        5.29 ±  8%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
      7.05 ±  8%      -3.6        3.45 ± 25%  perf-profile.children.cycles-pp.mutex_spin_on_owner
      6.29 ±  8%      -2.9        3.39 ± 10%  perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
      6.21 ±  8%      -2.9        3.33 ± 10%  perf-profile.children.cycles-pp.hrtimer_interrupt
      3.20 ± 10%      -2.1        1.14 ± 12%  perf-profile.children.cycles-pp.menu_select
      1.91 ±  7%      -1.7        0.22 ± 19%  perf-profile.children.cycles-pp.lo_release
      1.94 ±  7%      -1.7        0.27 ± 16%  perf-profile.children.cycles-pp.blkdev_close
      1.94 ±  7%      -1.7        0.27 ± 16%  perf-profile.children.cycles-pp.blkdev_put
      2.04 ±  7%      -1.6        0.40 ± 14%  perf-profile.children.cycles-pp.__fput
      2.10 ±  6%      -1.6        0.47 ± 15%  perf-profile.children.cycles-pp.task_work_run
      2.15 ±  6%      -1.6        0.53 ± 15%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
      2.14 ±  6%      -1.6        0.52 ± 15%  perf-profile.children.cycles-pp.exit_to_user_mode_prepare
      2.58 ± 11%      -1.5        1.10 ± 21%  perf-profile.children.cycles-pp.ktime_get
      1.79 ±  9%      -1.4        0.40 ± 13%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      3.35 ±  8%      -1.3        2.02 ±  8%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      1.53 ± 11%      -1.2        0.35 ± 14%  perf-profile.children.cycles-pp.tick_nohz_next_event
      1.91 ± 10%      -1.0        0.88 ± 21%  perf-profile.children.cycles-pp.clockevents_program_event
      2.36 ±  7%      -0.9        1.44 ± 11%  perf-profile.children.cycles-pp.tick_sched_timer
      0.89 ± 15%      -0.9        0.03 ± 87%  perf-profile.children.cycles-pp.timekeeping_max_deferment
      2.05 ±  7%      -0.7        1.32 ± 11%  perf-profile.children.cycles-pp.tick_sched_handle
      2.00 ±  7%      -0.7        1.28 ± 11%  perf-profile.children.cycles-pp.update_process_times
      0.85 ± 10%      -0.5        0.32 ± 22%  perf-profile.children.cycles-pp.tick_nohz_irq_exit
      1.61 ± 10%      -0.5        1.13 ±  5%  perf-profile.children.cycles-pp.irq_exit_rcu
      0.65 ±  9%      -0.3        0.31 ±  6%  perf-profile.children.cycles-pp.lapic_next_deadline
      1.07 ±  3%      -0.3        0.75 ±  4%  perf-profile.children.cycles-pp.scheduler_tick
      0.57 ±  8%      -0.3        0.30 ± 11%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.54 ± 11%      -0.3        0.28 ±  7%  perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
      0.44 ±  9%      -0.3        0.19 ± 11%  perf-profile.children.cycles-pp.ktime_get_update_offsets_now
      0.50 ± 14%      -0.2        0.25 ±  6%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.51 ± 15%      -0.2        0.28 ± 12%  perf-profile.children.cycles-pp.rebalance_domains
      0.46 ± 14%      -0.2        0.23 ±  9%  perf-profile.children.cycles-pp.read_tsc
      0.40 ±  3%      -0.2        0.18 ± 12%  perf-profile.children.cycles-pp.irqtime_account_irq
      0.42 ± 15%      -0.2        0.21 ±  7%  perf-profile.children.cycles-pp.native_sched_clock
      0.46 ± 14%      -0.2        0.26 ± 29%  perf-profile.children.cycles-pp.irq_enter_rcu
      0.36 ± 12%      -0.2        0.16 ± 17%  perf-profile.children.cycles-pp.rcu_sched_clock_irq
      0.20 ± 12%      -0.2        0.04 ± 66%  perf-profile.children.cycles-pp.__hrtimer_next_event_base
      0.35 ± 12%      -0.1        0.20 ± 10%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.42 ± 14%      -0.1        0.30 ± 12%  perf-profile.children.cycles-pp.poll_idle
      0.23 ± 20%      -0.1        0.11 ± 16%  perf-profile.children.cycles-pp.hrtimer_update_next_event
      0.20 ± 16%      -0.1        0.08 ± 14%  perf-profile.children.cycles-pp.rcu_idle_exit
      0.22 ± 20%      -0.1        0.11 ± 16%  perf-profile.children.cycles-pp.__intel_pmu_enable_all
      0.15 ± 12%      -0.1        0.04 ± 65%  perf-profile.children.cycles-pp.rcu_eqs_exit
      0.20 ± 14%      -0.1        0.10 ± 12%  perf-profile.children.cycles-pp._raw_spin_trylock
      0.24 ± 22%      -0.1        0.14 ± 23%  perf-profile.children.cycles-pp.run_rebalance_domains
      0.24 ± 20%      -0.1        0.15 ± 24%  perf-profile.children.cycles-pp.update_blocked_averages
      0.23 ± 11%      -0.1        0.15 ± 15%  perf-profile.children.cycles-pp.update_rq_clock
      0.19 ±  6%      -0.1        0.11 ± 11%  perf-profile.children.cycles-pp.arch_scale_freq_tick
      0.11 ± 24%      -0.1        0.04 ± 64%  perf-profile.children.cycles-pp.__hrtimer_get_next_event
      0.23 ± 19%      -0.1        0.16 ± 11%  perf-profile.children.cycles-pp.arch_cpu_idle_enter
      0.14 ± 15%      -0.1        0.08 ± 14%  perf-profile.children.cycles-pp.update_irq_load_avg
      0.22 ± 17%      -0.1        0.16 ± 10%  perf-profile.children.cycles-pp.tsc_verify_tsc_adjust
      0.08 ± 15%      -0.1        0.03 ± 87%  perf-profile.children.cycles-pp.exc_page_fault
      0.08 ± 15%      -0.1        0.03 ± 87%  perf-profile.children.cycles-pp.do_user_addr_fault
      0.15 ± 17%      -0.1        0.10 ± 22%  perf-profile.children.cycles-pp.read
      0.09 ± 18%      -0.0        0.04 ± 65%  perf-profile.children.cycles-pp.asm_exc_page_fault
      0.10 ± 20%      -0.0        0.06 ± 14%  perf-profile.children.cycles-pp.trigger_load_balance
      0.12 ± 21%      -0.0        0.08 ± 17%  perf-profile.children.cycles-pp.__remove_hrtimer
      0.10 ± 23%      -0.0        0.06 ± 14%  perf-profile.children.cycles-pp.timerqueue_add
      0.09 ±  8%      +0.0        0.12 ± 10%  perf-profile.children.cycles-pp.____sys_sendmsg
      0.10 ± 11%      +0.0        0.13 ± 12%  perf-profile.children.cycles-pp.sock_sendmsg
      0.09 ±  7%      +0.0        0.12 ± 13%  perf-profile.children.cycles-pp.___sys_sendmsg
      0.07 ± 22%      +0.0        0.11 ± 18%  perf-profile.children.cycles-pp.vfs_fstat
      0.02 ±116%      +0.1        0.08 ± 23%  perf-profile.children.cycles-pp.d_add
      0.02 ±116%      +0.1        0.08 ± 23%  perf-profile.children.cycles-pp.simple_lookup
      0.09 ± 13%      +0.1        0.14 ±  8%  perf-profile.children.cycles-pp.__d_lookup
      0.03 ± 86%      +0.1        0.08 ± 10%  perf-profile.children.cycles-pp.mutex_unlock
      0.01 ±158%      +0.1        0.07 ± 17%  perf-profile.children.cycles-pp.put_and_wait_on_page_locked
      0.00            +0.1        0.06 ±  7%  perf-profile.children.cycles-pp.page_counter_cancel
      0.18 ± 22%      +0.1        0.24 ± 12%  perf-profile.children.cycles-pp.loop_process_work
      0.00            +0.1        0.06 ± 15%  perf-profile.children.cycles-pp.__x64_sys_nanosleep
      0.00            +0.1        0.06 ± 15%  perf-profile.children.cycles-pp.hrtimer_nanosleep
      0.00            +0.1        0.06 ± 15%  perf-profile.children.cycles-pp.do_nanosleep
      0.05 ± 65%      +0.1        0.11 ± 15%  perf-profile.children.cycles-pp.d_lookup
      0.05 ± 65%      +0.1        0.11 ± 15%  perf-profile.children.cycles-pp.lookup_dcache
      0.00            +0.1        0.06 ± 21%  perf-profile.children.cycles-pp.sysfs_remove_groups
      0.00            +0.1        0.07 ± 15%  perf-profile.children.cycles-pp.device_create
      0.00            +0.1        0.07 ± 15%  perf-profile.children.cycles-pp.device_create_groups_vargs
      0.06 ± 53%      +0.1        0.12 ± 11%  perf-profile.children.cycles-pp.mutex_lock
      0.06 ± 18%      +0.1        0.12 ± 15%  perf-profile.children.cycles-pp.memset_erms
      0.02 ±118%      +0.1        0.09 ± 16%  perf-profile.children.cycles-pp.__alloc_pages
      0.00            +0.1        0.07 ±  9%  perf-profile.children.cycles-pp.obj_cgroup_uncharge_pages
      0.00            +0.1        0.07 ± 10%  perf-profile.children.cycles-pp.rcu_cblist_dequeue
      0.00            +0.1        0.07 ± 15%  perf-profile.children.cycles-pp.page_counter_uncharge
      0.00            +0.1        0.07 ± 15%  perf-profile.children.cycles-pp.inode_init_always
      0.00            +0.1        0.07 ± 17%  perf-profile.children.cycles-pp.up_write
      0.00            +0.1        0.07 ± 19%  perf-profile.children.cycles-pp.obj_cgroup_charge_pages
      0.00            +0.1        0.07 ± 14%  perf-profile.children.cycles-pp.kernfs_iop_getattr
      0.00            +0.1        0.08 ± 18%  perf-profile.children.cycles-pp.bdi_register
      0.00            +0.1        0.08 ± 18%  perf-profile.children.cycles-pp.bdi_register_va
      0.00            +0.1        0.08 ± 11%  perf-profile.children.cycles-pp.d_instantiate
      0.00            +0.1        0.08 ± 27%  perf-profile.children.cycles-pp.__list_add_valid
      0.00            +0.1        0.08 ± 11%  perf-profile.children.cycles-pp.elevator_init_mq
      0.00            +0.1        0.08 ± 17%  perf-profile.children.cycles-pp.legitimize_mnt
      0.05 ± 69%      +0.1        0.13 ± 13%  perf-profile.children.cycles-pp.__d_drop
      0.08 ± 17%      +0.1        0.15 ± 12%  perf-profile.children.cycles-pp.__d_alloc
      0.05 ± 66%      +0.1        0.13 ± 13%  perf-profile.children.cycles-pp.___d_drop
      0.00            +0.1        0.08 ± 14%  perf-profile.children.cycles-pp.get_page_from_freelist
      0.03 ± 87%      +0.1        0.11 ±  9%  perf-profile.children.cycles-pp.__slab_free
      0.01 ±244%      +0.1        0.09 ± 21%  perf-profile.children.cycles-pp.obj_cgroup_charge
      0.00            +0.1        0.08 ± 17%  perf-profile.children.cycles-pp.kernfs_put
      0.00            +0.1        0.08 ± 25%  perf-profile.children.cycles-pp.remove_files
      0.02 ±158%      +0.1        0.10 ±  9%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.00            +0.1        0.08 ± 28%  perf-profile.children.cycles-pp.internal_create_groups
      0.15 ± 10%      +0.1        0.24 ± 11%  perf-profile.children.cycles-pp.blkdev_read_iter
      0.15 ±  9%      +0.1        0.24 ± 12%  perf-profile.children.cycles-pp.filemap_get_pages
      0.00            +0.1        0.09 ± 20%  perf-profile.children.cycles-pp.remove_one
      0.00            +0.1        0.09 ± 15%  perf-profile.children.cycles-pp.do_renameat2
      0.10 ± 18%      +0.1        0.19 ± 12%  perf-profile.children.cycles-pp.alloc_inode
      0.00            +0.1        0.10 ± 13%  perf-profile.children.cycles-pp.__x64_sys_rename
      0.00            +0.1        0.10 ± 13%  perf-profile.children.cycles-pp.blk_alloc_queue
      0.00            +0.1        0.10 ± 35%  perf-profile.children.cycles-pp.kernfs_activate
      0.11 ± 27%      +0.1        0.21 ± 11%  perf-profile.children.cycles-pp.smpboot_thread_fn
      0.10 ± 19%      +0.1        0.20 ± 11%  perf-profile.children.cycles-pp.do_faccessat
      0.04 ± 87%      +0.1        0.13 ± 12%  perf-profile.children.cycles-pp.run_ksoftirqd
      0.00            +0.1        0.10 ± 17%  perf-profile.children.cycles-pp.filename_parentat
      0.00            +0.1        0.10 ± 17%  perf-profile.children.cycles-pp.path_parentat
      0.00            +0.1        0.10 ± 17%  perf-profile.children.cycles-pp.sysfs_unmerge_group
      0.00            +0.1        0.10 ± 20%  perf-profile.children.cycles-pp.sysfs_remove_group
      0.00            +0.1        0.10 ± 12%  perf-profile.children.cycles-pp.__srcu_read_lock
      0.18 ± 12%      +0.1        0.28 ± 11%  perf-profile.children.cycles-pp.filemap_read
      0.17 ± 19%      +0.1        0.27 ± 12%  perf-profile.children.cycles-pp._find_next_bit
      0.04 ± 65%      +0.1        0.15 ±  8%  perf-profile.children.cycles-pp.blk_mq_init_queue_data
      0.08 ± 31%      +0.1        0.19 ±  9%  perf-profile.children.cycles-pp.d_invalidate
      0.00            +0.1        0.11 ± 13%  perf-profile.children.cycles-pp.device_unregister
      0.00            +0.1        0.11 ± 16%  perf-profile.children.cycles-pp.dpm_sysfs_remove
      0.01 ±244%      +0.1        0.12 ± 11%  perf-profile.children.cycles-pp.fsnotify_grab_connector
      0.00            +0.1        0.12 ± 12%  perf-profile.children.cycles-pp.__destroy_inode
      0.00            +0.1        0.12 ± 12%  perf-profile.children.cycles-pp.destroy_inode
      0.04 ± 65%      +0.1        0.16 ± 10%  perf-profile.children.cycles-pp.__blk_mq_alloc_disk
      0.30 ±  8%      +0.1        0.41 ±  8%  perf-profile.children.cycles-pp.newidle_balance
      0.00            +0.1        0.12 ± 20%  perf-profile.children.cycles-pp.blk_mq_debugfs_unregister_queue_rqos
      0.01 ±244%      +0.1        0.13 ± 10%  perf-profile.children.cycles-pp.fsnotify_destroy_marks
      0.00            +0.1        0.12 ± 20%  perf-profile.children.cycles-pp.rq_qos_exit
      0.31 ±  6%      +0.1        0.43 ± 15%  perf-profile.children.cycles-pp.kmem_cache_alloc
      0.04 ± 63%      +0.1        0.17 ± 12%  perf-profile.children.cycles-pp.pcpu_alloc
      0.31 ± 11%      +0.1        0.44 ±  7%  perf-profile.children.cycles-pp.pick_next_task_fair
      0.12 ± 14%      +0.1        0.25 ± 13%  perf-profile.children.cycles-pp.d_alloc
      0.01 ±244%      +0.1        0.14 ±  7%  perf-profile.children.cycles-pp.dentry_unlink_inode
      0.00            +0.1        0.14 ±  7%  perf-profile.children.cycles-pp.__kernfs_new_node
      0.49 ±  9%      +0.1        0.63 ±  6%  perf-profile.children.cycles-pp.__schedule
      0.39 ±  8%      +0.1        0.54 ±  7%  perf-profile.children.cycles-pp.schedule
      0.15 ± 24%      +0.1        0.30 ± 11%  perf-profile.children.cycles-pp.cpumask_next
      0.01 ±244%      +0.1        0.15 ± 17%  perf-profile.children.cycles-pp.complete_walk
      0.00            +0.2        0.15 ± 17%  perf-profile.children.cycles-pp.lockref_put_or_lock
      0.00            +0.2        0.15 ±  7%  perf-profile.children.cycles-pp.kernfs_new_node
      0.01 ±244%      +0.2        0.17 ± 20%  perf-profile.children.cycles-pp.wait_rcu_exp_gp
      0.01 ±244%      +0.2        0.17 ± 20%  perf-profile.children.cycles-pp.sync_rcu_exp_select_cpus
      0.02 ±161%      +0.2        0.17 ± 12%  perf-profile.children.cycles-pp.kernfs_dop_revalidate
      0.14 ±  7%      +0.2        0.31 ±  9%  perf-profile.children.cycles-pp.d_alloc_parallel
      0.00            +0.2        0.16 ± 11%  perf-profile.children.cycles-pp.lookup_mnt
      0.00            +0.2        0.17 ± 17%  perf-profile.children.cycles-pp.device_add
      0.00            +0.2        0.17 ± 11%  perf-profile.children.cycles-pp.__traverse_mounts
      0.20 ± 12%      +0.2        0.37 ±  8%  perf-profile.children.cycles-pp.kmem_cache_free
      0.03 ±120%      +0.2        0.20 ±  9%  perf-profile.children.cycles-pp.__do_sys_newstat
      0.00            +0.2        0.18 ± 14%  perf-profile.children.cycles-pp.bdi_unregister
      0.12 ± 13%      +0.2        0.30 ±  9%  perf-profile.children.cycles-pp.new_inode_pseudo
      0.05 ± 42%      +0.2        0.23 ± 17%  perf-profile.children.cycles-pp.try_to_unlazy
      0.04 ± 65%      +0.2        0.22 ±  8%  perf-profile.children.cycles-pp.down_write
      0.08 ± 45%      +0.2        0.26 ±  9%  perf-profile.children.cycles-pp.step_into
      0.35 ± 11%      +0.2        0.54 ±  6%  perf-profile.children.cycles-pp.rcu_core
      0.03 ±120%      +0.2        0.23 ±  6%  perf-profile.children.cycles-pp.vfs_statx
      0.21 ±  7%      +0.2        0.41 ± 11%  perf-profile.children.cycles-pp.__lookup_slow
      0.08 ± 19%      +0.2        0.29 ± 16%  perf-profile.children.cycles-pp.inode_permission
      0.02 ±161%      +0.2        0.23 ± 16%  perf-profile.children.cycles-pp.__legitimize_path
      0.12 ± 17%      +0.2        0.33 ± 11%  perf-profile.children.cycles-pp.lookup_fast
      0.18 ± 10%      +0.2        0.40 ±  6%  perf-profile.children.cycles-pp.lookup_one_len
      0.00            +0.2        0.22 ± 14%  perf-profile.children.cycles-pp.kernfs_remove_by_name_ns
      0.01 ±244%      +0.2        0.23 ± 14%  perf-profile.children.cycles-pp.kernfs_iop_permission
      0.00            +0.2        0.23 ± 17%  perf-profile.children.cycles-pp.internal_create_group
      0.07 ± 12%      +0.2        0.31 ±  9%  perf-profile.children.cycles-pp.evict
      0.01 ±158%      +0.2        0.25 ± 21%  perf-profile.children.cycles-pp.blk_mq_register_hctx
      0.00            +0.2        0.24 ± 19%  perf-profile.children.cycles-pp.blk_mq_debugfs_unregister_sched
      0.00            +0.2        0.24 ± 18%  perf-profile.children.cycles-pp.__elevator_exit
      0.00            +0.2        0.24 ± 18%  perf-profile.children.cycles-pp.blk_mq_exit_sched
      0.01 ±158%      +0.2        0.26 ± 22%  perf-profile.children.cycles-pp.__blk_mq_register_dev
      0.01 ±244%      +0.2        0.26 ± 22%  perf-profile.children.cycles-pp.sysfs_create_dir_ns
      0.01 ±244%      +0.3        0.27 ± 22%  perf-profile.children.cycles-pp.kernfs_create_dir_ns
      0.00            +0.3        0.26 ± 10%  perf-profile.children.cycles-pp.device_del
      0.01 ±158%      +0.3        0.28 ± 21%  perf-profile.children.cycles-pp.kobject_add
      0.00            +0.3        0.27 ± 17%  perf-profile.children.cycles-pp.__legitimize_mnt
      0.01 ±244%      +0.3        0.28 ± 21%  perf-profile.children.cycles-pp.kobject_add_internal
      0.00            +0.3        0.28 ± 18%  perf-profile.children.cycles-pp.sysfs_add_file_mode_ns
      0.00            +0.3        0.28 ± 18%  perf-profile.children.cycles-pp.__kernfs_create_file
      0.22 ± 14%      +0.3        0.52 ±  6%  perf-profile.children.cycles-pp.rcu_do_batch
      0.03 ± 87%      +0.3        0.35 ± 15%  perf-profile.children.cycles-pp.debugfs_create_dir
      0.00            +0.3        0.31 ± 24%  perf-profile.children.cycles-pp.simple_release_fs
      0.01 ±244%      +0.3        0.34 ± 11%  perf-profile.children.cycles-pp.__kernfs_remove
      0.01 ±244%      +0.3        0.35 ± 18%  perf-profile.children.cycles-pp.disk_release
      0.01 ±244%      +0.3        0.35 ± 18%  perf-profile.children.cycles-pp.blk_release_queue
      0.01 ±244%      +0.3        0.35 ± 19%  perf-profile.children.cycles-pp.device_release
      0.13 ± 25%      +0.4        0.49 ±  9%  perf-profile.children.cycles-pp.filename_lookup
      0.01 ±244%      +0.4        0.38 ± 18%  perf-profile.children.cycles-pp.kobject_release
      0.31 ± 15%      +0.4        0.72 ± 12%  perf-profile.children.cycles-pp.walk_component
      0.00            +0.4        0.42 ± 25%  perf-profile.children.cycles-pp.kernfs_add_one
      0.27 ± 25%      +0.5        0.72 ±  7%  perf-profile.children.cycles-pp.mnt_get_count
      0.61 ± 11%      +0.5        1.08 ± 22%  perf-profile.children.cycles-pp.openat64
      0.23 ± 18%      +0.5        0.74 ±  7%  perf-profile.children.cycles-pp.link_path_walk
      0.20 ± 17%      +0.5        0.74 ±  7%  perf-profile.children.cycles-pp.__dentry_kill
      0.02 ±160%      +0.6        0.61 ± 15%  perf-profile.children.cycles-pp.path_init
      0.00            +0.6        0.62 ± 27%  perf-profile.children.cycles-pp.inode_sb_list_add
      0.95 ± 10%      +0.7        1.62 ± 11%  perf-profile.children.cycles-pp.io_serial_out
      0.31 ± 17%      +0.7        0.99 ±  7%  perf-profile.children.cycles-pp.dput
      0.68 ± 11%      +0.8        1.44 ± 21%  perf-profile.children.cycles-pp.unit_file_find_dropin_paths
      0.00            +0.8        0.78 ± 13%  perf-profile.children.cycles-pp.blk_mq_unregister_hctx
      0.00            +0.8        0.78 ± 35%  perf-profile.children.cycles-pp.simple_pin_fs
      0.14 ± 12%      +0.8        0.93 ± 18%  perf-profile.children.cycles-pp.new_inode
      0.13 ± 13%      +0.8        0.93 ± 19%  perf-profile.children.cycles-pp.debugfs_get_inode
      0.01 ±244%      +0.8        0.81 ± 12%  perf-profile.children.cycles-pp.blk_mq_unregister_dev
      0.00            +0.9        0.88 ± 12%  perf-profile.children.cycles-pp.kernfs_remove
      0.01 ±244%      +0.9        0.91 ± 12%  perf-profile.children.cycles-pp.blk_unregister_queue
      0.01 ±244%      +0.9        0.93 ± 11%  perf-profile.children.cycles-pp.kobject_del
      0.01 ±244%      +0.9        0.95 ± 12%  perf-profile.children.cycles-pp.__kobject_del
      0.38 ± 11%      +1.0        1.40 ± 13%  perf-profile.children.cycles-pp.path_lookupat
      0.05 ± 43%      +1.1        1.17 ± 11%  perf-profile.children.cycles-pp.del_gendisk
      0.19 ±  9%      +1.1        1.32 ± 23%  perf-profile.children.cycles-pp.start_creating
      9.04 ±  4%      +1.4       10.49 ± 10%  perf-profile.children.cycles-pp.io_serial_in
      0.30 ± 10%      +1.7        2.04 ± 15%  perf-profile.children.cycles-pp.debugfs_create_files
      0.30 ± 10%      +1.7        2.04 ± 15%  perf-profile.children.cycles-pp.__debugfs_create_file
      0.34 ±  8%      +1.9        2.27 ± 15%  perf-profile.children.cycles-pp.blk_mq_debugfs_register_hctx
      0.35 ±  8%      +2.0        2.37 ± 15%  perf-profile.children.cycles-pp.blk_mq_debugfs_register
      0.41 ±  7%      +2.4        2.81 ± 13%  perf-profile.children.cycles-pp.blk_register_queue
      0.10 ± 74%      +2.6        2.73 ± 38%  perf-profile.children.cycles-pp.osq_lock
      0.45 ±  6%      +2.6        3.09 ± 12%  perf-profile.children.cycles-pp.__device_add_disk
      0.56 ±  6%      +2.7        3.29 ± 11%  perf-profile.children.cycles-pp.loop_add
      0.29 ± 24%      +7.8        8.06 ± 16%  perf-profile.children.cycles-pp.mntput_no_expire
      0.63 ± 18%      +8.6        9.22 ± 14%  perf-profile.children.cycles-pp.blk_mq_debugfs_unregister_hctx
      0.63 ± 18%      +8.6        9.22 ± 14%  perf-profile.children.cycles-pp.blk_mq_exit_queue
      0.64 ± 18%      +8.7        9.37 ± 14%  perf-profile.children.cycles-pp.blk_cleanup_disk
      0.64 ± 18%      +8.7        9.37 ± 14%  perf-profile.children.cycles-pp.blk_cleanup_queue
      0.00            +8.9        8.87 ± 18%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      0.66 ± 19%      +9.0        9.65 ± 14%  perf-profile.children.cycles-pp.simple_recursive_removal
      0.66 ± 19%      +9.0        9.66 ± 14%  perf-profile.children.cycles-pp.debugfs_remove
      0.56 ± 11%      +9.2        9.79 ± 16%  perf-profile.children.cycles-pp._raw_spin_lock
      0.75 ± 17%     +10.3       11.04 ± 13%  perf-profile.children.cycles-pp.loop_remove
     19.97 ±  5%     +11.7       31.66 ±  9%  perf-profile.children.cycles-pp.__x64_sys_ioctl
     26.71 ±  4%     +12.7       39.36 ± 11%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     26.68 ±  4%     +12.7       39.34 ± 11%  perf-profile.children.cycles-pp.do_syscall_64
      1.44 ±  9%     +13.0       14.45 ± 13%  perf-profile.children.cycles-pp.loop_control_ioctl
     39.44 ±  7%      -6.5       32.93 ±  9%  perf-profile.self.cycles-pp.intel_idle
      6.97 ±  8%      -3.6        3.41 ± 26%  perf-profile.self.cycles-pp.mutex_spin_on_owner
      2.87 ± 10%      -1.5        1.37 ± 14%  perf-profile.self.cycles-pp.vprintk_emit
      2.20 ± 10%      -1.3        0.92 ± 26%  perf-profile.self.cycles-pp.ktime_get
      0.89 ± 15%      -0.9        0.03 ± 87%  perf-profile.self.cycles-pp.timekeeping_max_deferment
      1.45 ± 12%      -0.7        0.71 ± 11%  perf-profile.self.cycles-pp.cpuidle_enter_state
      1.26 ± 19%      -0.6        0.65 ± 17%  perf-profile.self.cycles-pp.menu_select
      0.65 ±  9%      -0.3        0.31 ±  6%  perf-profile.self.cycles-pp.lapic_next_deadline
      0.57 ±  8%      -0.3        0.30 ± 11%  perf-profile.self.cycles-pp.native_irq_return_iret
      0.37 ± 10%      -0.2        0.14 ± 13%  perf-profile.self.cycles-pp.ktime_get_update_offsets_now
      0.45 ± 13%      -0.2        0.22 ± 10%  perf-profile.self.cycles-pp.read_tsc
      0.41 ± 15%      -0.2        0.21 ±  7%  perf-profile.self.cycles-pp.native_sched_clock
      0.34 ± 14%      -0.2        0.16 ± 15%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.33 ± 10%      -0.2        0.16 ± 17%  perf-profile.self.cycles-pp.rcu_sched_clock_irq
      0.18 ± 16%      -0.1        0.03 ± 87%  perf-profile.self.cycles-pp.__hrtimer_next_event_base
      0.39 ± 11%      -0.1        0.27 ± 16%  perf-profile.self.cycles-pp.tick_nohz_next_event
      0.19 ±  6%      -0.1        0.07 ± 46%  perf-profile.self.cycles-pp.irqtime_account_irq
      0.42 ± 14%      -0.1        0.30 ± 12%  perf-profile.self.cycles-pp.poll_idle
      0.22 ± 20%      -0.1        0.11 ± 16%  perf-profile.self.cycles-pp.__intel_pmu_enable_all
      0.23 ± 13%      -0.1        0.13 ± 19%  perf-profile.self.cycles-pp.do_idle
      0.20 ± 14%      -0.1        0.09 ±  9%  perf-profile.self.cycles-pp._raw_spin_trylock
      0.19 ±  6%      -0.1        0.11 ± 11%  perf-profile.self.cycles-pp.arch_scale_freq_tick
      0.13 ± 18%      -0.1        0.06 ± 43%  perf-profile.self.cycles-pp.hrtimer_interrupt
      0.14 ± 15%      -0.1        0.08 ± 14%  perf-profile.self.cycles-pp.update_irq_load_avg
      0.22 ± 20%      -0.1        0.15 ± 13%  perf-profile.self.cycles-pp.tsc_verify_tsc_adjust
      0.12 ± 14%      -0.1        0.06 ±  8%  perf-profile.self.cycles-pp.asm_sysvec_apic_timer_interrupt
      0.09 ± 22%      -0.1        0.03 ± 87%  perf-profile.self.cycles-pp.perf_mux_hrtimer_handler
      0.10 ± 17%      -0.0        0.06 ± 13%  perf-profile.self.cycles-pp.trigger_load_balance
      0.08 ± 18%      +0.0        0.12 ±  9%  perf-profile.self.cycles-pp.__d_lookup
      0.00            +0.1        0.06 ±  8%  perf-profile.self.cycles-pp.page_counter_cancel
      0.02 ±115%      +0.1        0.08 ± 12%  perf-profile.self.cycles-pp.mutex_unlock
      0.06 ± 18%      +0.1        0.12 ± 16%  perf-profile.self.cycles-pp.memset_erms
      0.14 ± 17%      +0.1        0.21 ± 16%  perf-profile.self.cycles-pp._find_next_bit
      0.07 ± 46%      +0.1        0.14 ± 10%  perf-profile.self.cycles-pp.cpumask_next
      0.00            +0.1        0.07 ± 12%  perf-profile.self.cycles-pp.rcu_cblist_dequeue
      0.00            +0.1        0.07 ± 15%  perf-profile.self.cycles-pp.up_write
      0.00            +0.1        0.07 ± 29%  perf-profile.self.cycles-pp.__list_add_valid
      0.05 ± 66%      +0.1        0.13 ± 13%  perf-profile.self.cycles-pp.___d_drop
      0.02 ±158%      +0.1        0.10 ±  9%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.00            +0.1        0.08 ± 18%  perf-profile.self.cycles-pp.lookup_mnt
      0.02 ±116%      +0.1        0.11 ± 10%  perf-profile.self.cycles-pp.__slab_free
      0.00            +0.1        0.09 ± 14%  perf-profile.self.cycles-pp.__destroy_inode
      0.00            +0.1        0.09 ± 20%  perf-profile.self.cycles-pp.remove_one
      0.01 ±244%      +0.1        0.10 ± 13%  perf-profile.self.cycles-pp.mutex_lock
      0.00            +0.1        0.10 ± 12%  perf-profile.self.cycles-pp.__srcu_read_lock
      0.00            +0.1        0.11 ± 18%  perf-profile.self.cycles-pp.__mutex_lock
      0.00            +0.1        0.14 ± 19%  perf-profile.self.cycles-pp.lockref_put_or_lock
      0.01 ±158%      +0.2        0.20 ± 11%  perf-profile.self.cycles-pp.down_write
      0.00            +0.2        0.21 ± 13%  perf-profile.self.cycles-pp.mntput_no_expire
      0.15 ± 27%      +0.3        0.50 ±  8%  perf-profile.self.cycles-pp.mnt_get_count
      0.54 ± 11%      +0.4        0.93 ±  3%  perf-profile.self.cycles-pp._raw_spin_lock
      0.02 ±161%      +0.5        0.56 ± 16%  perf-profile.self.cycles-pp.path_init
      0.95 ± 10%      +0.7        1.62 ± 11%  perf-profile.self.cycles-pp.io_serial_out
      9.04 ±  4%      +1.5       10.49 ± 10%  perf-profile.self.cycles-pp.io_serial_in
      0.10 ± 74%      +2.6        2.71 ± 38%  perf-profile.self.cycles-pp.osq_lock
      0.00            +8.8        8.82 ± 18%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath


                                                                                
                            stress-ng.loop.ops_per_sec                          
                                                                                
  110 +---------------------------------------------------------------------+   
      |                                                                     |   
  100 |-O O O O  O O O O O   O OO   O                                       |   
   90 |-+      O           O      O                                         |   
      |                                                                     |   
   80 |-+                                                                   |   
      |                               O O O O O O OO O O O O O O            |   
   70 |-+                                                                   |   
      |                                                                     |   
   60 |-+                                                                   |   
   50 |-+                                                                   |   
      |             .+.+. .+.    .+.+                                       |   
   40 |-+      +.+.+     +   +.++    :                                      |   
      |.+.+.   :                     : .+.+.   .+.+           .+. .+     .+.|   
   30 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                            stress-ng.time.system_time                          
                                                                                
  160 +---------------------------------------------------------------------+   
      | O      O O     O       OO       O O          O O                    |   
  140 |-+ O O O    O O   O O O    O         O O O        O O O              |   
      |                                           O            O            |   
  120 |-+                                                                   |   
      |                                                                     |   
  100 |-+                                                                   |   
      |                                                                     |   
   80 |-+                                                                   |   
      |                                                                     |   
   60 |-+                                                                   |   
      |         .+.+.+.+.+.+.   +.+.+.                                      |   
   40 |.+.+.+.++             +.+      +.+.+.+.+.+.++.+.+.+.+.+.+.+.++.+.+.+.|   
      |                                                                     |   
   20 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                    stress-ng.time.percent_of_cpu_this_job_got                  
                                                                                
  260 +---------------------------------------------------------------------+   
  240 |-+                       O     O            O                        |   
      | O   O  O O     O O O O O  O     O O     O    O O     O              |   
  220 |-+ O   O    O O                      O O   O      O O   O            |   
  200 |-+                                                                   |   
  180 |-+                                                                   |   
  160 |-+                                                                   |   
      |                                                                     |   
  140 |-+                                                                   |   
  120 |-+                                                                   |   
  100 |-+                                                                   |   
   80 |-+                                                                   |   
      |        +.+.+.+.+.+.+.+.++.+.+.                                      |   
   60 |.+.+.+.+                       +.+.+.+.+.+.++.+.+.+.+.+.+.+.++.+.+.+.|   
   40 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                      stress-ng.time.voluntary_context_switches                 
                                                                                
  140000 +------------------------------------------------------------------+   
         | O OO   O O OO   O O O OO O O                                     |   
  120000 |-+                            O OO O O O OO O O O O OO            |   
         |                                                                  |   
  100000 |-+                                                                |   
         |                                                                  |   
   80000 |-+                                                                |   
         |                                                                  |   
   60000 |-+                                                                |   
         |                                                                  |   
   40000 |-+                                                                |   
         |                         .+.                                      |   
   20000 |.+.++.+.+.+.++.+.+.+.+.++   +.+.++.+.+.+.++.+.+.+.+.++.+.+.+.++.+.|   
         |                                                                  |   
       0 +------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                    stress-ng.time.involuntary_context_switches                 
                                                                                
  7500 +--------------------------------------------------------------------+   
       | O   OO O O O       OO O                                            |   
  7000 |-+ O          O   O        O O                                      |   
       |                         O     O            O                       |   
  6500 |-+                               OO O O O O   OO O O O O            |   
       |                                                                    |   
  6000 |-+                                                                  |   
       |                                                                    |   
  5500 |-+                                                                  |   
       |                                                                    |   
  5000 |-+                                                                  |   
       |          +.              .+. .+.    .+.              .+            |   
  4500 |.+.+. +. +  +.+.+.+.+   .+   +   ++.+   +.+.+.++.+.+.+  + .+.++.+.+.|   
       | +   +  +            +.+                                 +          |   
  4000 +--------------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


---
0DAY/LKP+ Test Infrastructure                   Open Source Technology Center
https://lists.01.org/hyperkitty/list/lkp@lists.01.org       Intel Corporation

Thanks,
Oliver Sang


View attachment "config-5.14.0-rc5-00006-gacd17464785f" of type "text/plain" (175472 bytes)

View attachment "job-script" of type "text/plain" (8527 bytes)

View attachment "job.yaml" of type "text/plain" (5650 bytes)

View attachment "reproduce" of type "text/plain" (337 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ