lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Message-ID: <20200701090620.GA3874@shao2-debian>
Date:   Wed, 1 Jul 2020 17:06:20 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     Filipe Manana <fdmanana@...e.com>
Cc:     David Sterba <dsterba@...e.com>,
        kernel test robot <rong.a.chen@...el.com>,
        Josef Bacik <josef@...icpanda.com>,
        LKML <linux-kernel@...r.kernel.org>, lkp@...ts.01.org
Subject: [btrfs] e7a79811d0: reaim.jobs_per_min 15.0% improvement

Greeting,

FYI, we noticed a 15.0% improvement of reaim.jobs_per_min due to commit:


commit: e7a79811d0db136dc2d336b56d54cf1b774ce972 ("btrfs: check if a log root exists before locking the log_mutex on unlink")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master

in testcase: reaim
on test machine: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 256G memory
with following parameters:

	runtime: 300s
	nr_task: 100
	disk: 1HDD
	fs: btrfs
	test: disk
	cpufreq_governor: performance
	ucode: 0x5002f01

test-description: REAIM is an updated and improved version of AIM 7 benchmark.
test-url: https://sourceforge.net/projects/re-aim-7/





Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/nr_task/rootfs/runtime/tbox_group/test/testcase/ucode:
  gcc-9/performance/1HDD/btrfs/x86_64-rhel-7.6/100/debian-x86_64-20191114.cgz/300s/lkp-csl-2sp7/disk/reaim/0x5002f01

commit: 
  6bd335b469 ("btrfs: fix bytes_may_use underflow when running balance and scrub in parallel")
  e7a79811d0 ("btrfs: check if a log root exists before locking the log_mutex on unlink")

6bd335b469f945f7 e7a79811d0db136dc2d336b56d5 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          0:4           -1%           0:4     perf-profile.children.cycles-pp.error_entry
         %stddev     %change         %stddev
             \          |                \  
     56.97           +15.4%      65.75 ± 10%  reaim.child_systime
      6.03            -7.0%       5.61        reaim.child_utime
      3785 ±  2%     +15.0%       4352 ±  3%  reaim.jobs_per_min
     37.86 ±  2%     +15.0%      43.52 ±  3%  reaim.jobs_per_min_child
     96.88            +1.2%      98.00        reaim.jti
      3873 ±  2%     +13.6%       4402 ±  4%  reaim.max_jobs_per_min
    158.70 ±  2%     -13.0%     138.03 ±  3%  reaim.parent_time
      2.62 ± 11%     -52.8%       1.23 ±  6%  reaim.std_dev_percent
      4.00 ± 13%     -58.1%       1.67 ±  9%  reaim.std_dev_time
    321.80 ±  2%     +30.7%     420.73 ±  3%  reaim.time.elapsed_time
    321.80 ±  2%     +30.7%     420.73 ±  3%  reaim.time.elapsed_time.max
    135494           +52.5%     206624        reaim.time.file_system_inputs
   5793788           +48.3%    8593432 ±  2%  reaim.time.file_system_outputs
     28941           +37.3%      39723        reaim.time.involuntary_context_switches
   3108447           +49.7%    4652339        reaim.time.minor_page_faults
     38.75 ±  3%     +30.3%      50.50 ±  7%  reaim.time.percent_of_cpu_this_job_got
    114.00           +73.1%     197.32 ± 10%  reaim.time.system_time
     12.08           +39.5%      16.85        reaim.time.user_time
   1355035           +27.6%    1729072        reaim.time.voluntary_context_switches
     20000           +50.0%      30000        reaim.workload
     93.96            -1.8%      92.23        iostat.cpu.idle
      5.50           +29.5%       7.13        iostat.cpu.iowait
      5.54            +1.6        7.16        mpstat.cpu.all.iowait%
      0.01            +0.0        0.01 ± 10%  mpstat.cpu.all.soft%
      0.46 ±  3%      +0.1        0.57 ±  5%  mpstat.cpu.all.sys%
  68147100 ±  3%    +261.2%  2.461e+08 ± 59%  cpuidle.C1.time
    858982          +242.5%    2942166 ± 59%  cpuidle.C1.usage
    873660 ±  2%    +141.5%    2109514 ± 25%  cpuidle.POLL.time
    213950 ±  2%     +60.2%     342736 ± 11%  cpuidle.POLL.usage
   1763954 ±  5%     +44.9%    2555237 ±  4%  numa-numastat.node0.local_node
   1794996 ±  5%     +43.7%    2578568 ±  4%  numa-numastat.node0.numa_hit
   1907557 ±  5%     +37.2%    2618090 ±  4%  numa-numastat.node1.local_node
   1907728 ±  5%     +37.7%    2625994 ±  4%  numa-numastat.node1.numa_hit
     93.50            -1.6%      92.00        vmstat.cpu.id
    208.50 ±  2%     +16.8%     243.50 ±  2%  vmstat.io.bi
     15104 ±  3%     +18.5%      17906        vmstat.io.bo
      5.00           +40.0%       7.00        vmstat.procs.b
    686190 ±  3%     +40.3%     962641        meminfo.Active
    402452 ±  5%     +68.4%     677689        meminfo.Active(file)
    344132           -39.3%     208816 ±  2%  meminfo.Inactive
    325999           -41.6%     190528 ±  3%  meminfo.Inactive(file)
     10986 ±  2%     -19.7%       8821 ±  3%  meminfo.max_used_kB
    369922 ± 12%     +41.3%     522687 ± 12%  numa-meminfo.node0.Active
    210515 ±  6%     +64.5%     346372 ±  2%  numa-meminfo.node0.Active(file)
    173152 ±  2%     -40.0%     103889 ±  4%  numa-meminfo.node0.Inactive
    168266 ±  2%     -42.5%      96719 ±  4%  numa-meminfo.node0.Inactive(file)
      2116 ± 15%    +112.6%       4499 ± 43%  numa-meminfo.node0.PageTables
     95418 ±  4%     +11.5%     106359 ±  4%  numa-meminfo.node0.SUnreclaim
    132099           +12.6%     148799 ±  7%  numa-meminfo.node0.Slab
    316606 ± 18%     +39.0%     440211 ± 16%  numa-meminfo.node1.Active
    192268 ±  5%     +72.5%     331573 ±  3%  numa-meminfo.node1.Active(file)
    170823 ±  2%     -38.7%     104708 ±  5%  numa-meminfo.node1.Inactive
    157576 ±  2%     -40.6%      93590 ±  2%  numa-meminfo.node1.Inactive(file)
      8013 ±  2%     -21.2%       6317 ± 31%  numa-meminfo.node1.PageTables
     52580 ±  6%     +64.7%      86607 ±  2%  numa-vmstat.node0.nr_active_file
    209324 ±  2%     +46.9%     307580 ±  2%  numa-vmstat.node0.nr_dirtied
     42108 ±  2%     -42.7%      24146 ±  4%  numa-vmstat.node0.nr_inactive_file
    532.50 ± 15%    +111.1%       1124 ± 43%  numa-vmstat.node0.nr_page_table_pages
     23853 ±  4%     +11.5%      26588 ±  4%  numa-vmstat.node0.nr_slab_unreclaimable
    207741 ±  2%     +45.0%     301150 ±  2%  numa-vmstat.node0.nr_written
     52580 ±  6%     +64.7%      86607 ±  2%  numa-vmstat.node0.nr_zone_active_file
     42108 ±  2%     -42.7%      24146 ±  4%  numa-vmstat.node0.nr_zone_inactive_file
   1465894 ±  3%     +30.3%    1910510 ±  2%  numa-vmstat.node0.numa_hit
   1279539 ±  4%     +35.4%    1732002 ±  2%  numa-vmstat.node0.numa_local
     48023 ±  5%     +72.6%      82911 ±  3%  numa-vmstat.node1.nr_active_file
    197282 ±  2%     +50.5%     296866 ±  3%  numa-vmstat.node1.nr_dirtied
     39430 ±  2%     -40.7%      23374 ±  2%  numa-vmstat.node1.nr_inactive_file
      1998 ±  2%     -21.1%       1576 ± 31%  numa-vmstat.node1.nr_page_table_pages
    195720 ±  2%     +48.8%     291212 ±  3%  numa-vmstat.node1.nr_written
     48023 ±  5%     +72.6%      82911 ±  3%  numa-vmstat.node1.nr_zone_active_file
     39430 ±  2%     -40.7%      23374 ±  2%  numa-vmstat.node1.nr_zone_inactive_file
   1710353 ±  4%     +19.7%    2047466 ±  2%  numa-vmstat.node1.numa_hit
   1681916 ±  4%     +19.6%    2011220 ±  2%  numa-vmstat.node1.numa_local
     16.26          +117.1%      35.31 ± 53%  perf-stat.i.MPKI
     17.95 ±  3%      -6.6       11.36 ± 50%  perf-stat.i.cache-miss-rate%
  21936400          +109.7%   46000088 ± 49%  perf-stat.i.cache-references
    477.38 ±  2%     +15.1%     549.65        perf-stat.i.cpu-migrations
     35810 ±  3%    +303.4%     144469 ± 71%  perf-stat.i.dTLB-store-misses
     11965           +11.4%      13325 ±  2%  perf-stat.i.minor-faults
     88714            +7.0%      94948        perf-stat.i.node-loads
     86.56            -4.3       82.26 ±  3%  perf-stat.i.node-store-miss-rate%
    245958            +5.5%     259564 ±  2%  perf-stat.i.node-store-misses
     39290 ±  2%     +43.4%      56356 ± 14%  perf-stat.i.node-stores
     11965           +11.4%      13325 ±  2%  perf-stat.i.page-faults
     15.21          +116.5%      32.93 ± 52%  perf-stat.overall.MPKI
     17.90 ±  3%      -6.6       11.28 ± 50%  perf-stat.overall.cache-miss-rate%
      0.32           -17.4%       0.26 ± 20%  perf-stat.overall.ipc
     86.23            -4.0       82.18 ±  3%  perf-stat.overall.node-store-miss-rate%
  23206567 ±  3%     -13.9%   19971317 ±  5%  perf-stat.overall.path-length
  21868281          +109.9%   45892139 ± 49%  perf-stat.ps.cache-references
    475.88 ±  2%     +15.2%     548.34        perf-stat.ps.cpu-migrations
     35700 ±  3%    +303.7%     144131 ± 71%  perf-stat.ps.dTLB-store-misses
     11928           +11.4%      13293 ±  2%  perf-stat.ps.minor-faults
     88439            +7.1%      94723        perf-stat.ps.node-loads
    245191            +5.6%     258947 ±  2%  perf-stat.ps.node-store-misses
     39169 ±  2%     +43.5%      56225 ± 14%  perf-stat.ps.node-stores
     11928           +11.4%      13293 ±  2%  perf-stat.ps.page-faults
 4.641e+11 ±  3%     +29.1%  5.991e+11 ±  5%  perf-stat.total.instructions
    100568 ±  5%     +68.5%     169487        proc-vmstat.nr_active_file
    807685           +47.9%    1194410 ±  2%  proc-vmstat.nr_dirtied
    484069            +7.3%     519483        proc-vmstat.nr_file_pages
      4532            +0.9%       4571        proc-vmstat.nr_inactive_anon
     81496           -41.6%      47558 ±  3%  proc-vmstat.nr_inactive_file
     19103            +2.1%      19505        proc-vmstat.nr_kernel_stack
      2533            +6.7%       2702        proc-vmstat.nr_page_table_pages
      5755            +8.0%       6217        proc-vmstat.nr_shmem
     21922            +4.8%      22965        proc-vmstat.nr_slab_reclaimable
     50417            +3.1%      51958        proc-vmstat.nr_slab_unreclaimable
     57.25 ±  6%     +79.5%     102.75 ±  7%  proc-vmstat.nr_writeback
    800730           +46.1%    1169674 ±  2%  proc-vmstat.nr_written
    100568 ±  5%     +68.5%     169487        proc-vmstat.nr_zone_active_file
      4532            +0.9%       4571        proc-vmstat.nr_zone_inactive_anon
     81496           -41.6%      47558 ±  3%  proc-vmstat.nr_zone_inactive_file
   3728847           +40.3%    5231460        proc-vmstat.numa_hit
   3697622           +40.6%    5200206        proc-vmstat.numa_local
    229874 ±  2%     +24.9%     287222        proc-vmstat.pgactivate
   3906906           +40.5%    5487948        proc-vmstat.pgalloc_normal
    909.00 ±  4%     +47.7%       1342 ±  2%  proc-vmstat.pgdeactivate
   3960854           +45.4%    5760263        proc-vmstat.pgfault
   3609600           +43.7%    5187000        proc-vmstat.pgfree
     67747           +52.5%     103312        proc-vmstat.pgpgin
   4898534           +54.8%    7583021 ±  3%  proc-vmstat.pgpgout
     13702           +48.6%      20363        proc-vmstat.pgrotated
     10025 ±  3%     +30.8%      13110        slabinfo.Acpi-State.active_objs
     10025 ±  3%     +30.8%      13110        slabinfo.Acpi-State.num_objs
     21718           +14.1%      24770        slabinfo.anon_vma.active_objs
     21718           +14.1%      24770        slabinfo.anon_vma.num_objs
     50886           +15.2%      58613 ±  2%  slabinfo.anon_vma_chain.active_objs
    794.50           +15.3%     916.25 ±  2%  slabinfo.anon_vma_chain.active_slabs
     50897           +15.3%      58664 ±  2%  slabinfo.anon_vma_chain.num_objs
    794.50           +15.3%     916.25 ±  2%  slabinfo.anon_vma_chain.num_slabs
      5452 ±  2%     +26.7%       6909        slabinfo.btrfs_inode.active_objs
      5457 ±  2%     +26.6%       6909        slabinfo.btrfs_inode.num_objs
      3276 ± 12%     +29.4%       4239 ± 11%  slabinfo.eventpoll_pwq.active_objs
      3276 ± 12%     +29.4%       4239 ± 11%  slabinfo.eventpoll_pwq.num_objs
      3961           +10.1%       4360 ±  2%  slabinfo.khugepaged_mm_slot.active_objs
      3961           +10.1%       4360 ±  2%  slabinfo.khugepaged_mm_slot.num_objs
      8982 ±  3%     +22.4%      10996 ±  2%  slabinfo.kmalloc-192.active_objs
      9144 ±  3%     +22.3%      11179 ±  2%  slabinfo.kmalloc-192.num_objs
     10846 ±  4%     +35.9%      14737 ±  3%  slabinfo.mnt_cache.active_objs
     10977 ±  4%     +35.9%      14922 ±  3%  slabinfo.mnt_cache.num_objs
     39985 ±  4%     +27.0%      50774        slabinfo.numa_policy.active_objs
    644.25 ±  4%     +27.4%     820.75        slabinfo.numa_policy.active_slabs
     39992 ±  4%     +27.3%      50928        slabinfo.numa_policy.num_objs
    644.25 ±  4%     +27.4%     820.75        slabinfo.numa_policy.num_slabs
     23459            +9.8%      25757        slabinfo.radix_tree_node.active_objs
     23462            +9.8%      25757        slabinfo.radix_tree_node.num_objs
     17792           +10.1%      19590        slabinfo.vmap_area.num_objs
    765.59 ±  5%     +69.7%       1299 ± 16%  sched_debug.cfs_rq:/.exec_clock.avg
    560.98 ±  7%     +88.8%       1059 ± 19%  sched_debug.cfs_rq:/.exec_clock.min
     20789 ± 11%     +23.5%      25672 ± 19%  sched_debug.cfs_rq:/.min_vruntime.avg
      0.02 ± 64%    +440.7%       0.09 ± 59%  sched_debug.cfs_rq:/.nr_spread_over.avg
      0.14 ± 58%    +149.6%       0.34 ± 53%  sched_debug.cfs_rq:/.nr_spread_over.stddev
   1371802 ± 32%     -26.5%    1008707        sched_debug.cpu.avg_idle.max
    156249 ±  8%     +33.5%     208582 ± 10%  sched_debug.cpu.clock.avg
    156255 ±  8%     +33.5%     208587 ± 10%  sched_debug.cpu.clock.max
    156244 ±  8%     +33.5%     208576 ± 10%  sched_debug.cpu.clock.min
      2.98           +14.0%       3.40 ±  8%  sched_debug.cpu.clock.stddev
    156249 ±  8%     +33.5%     208582 ± 10%  sched_debug.cpu.clock_task.avg
    156255 ±  8%     +33.5%     208587 ± 10%  sched_debug.cpu.clock_task.max
    156244 ±  8%     +33.5%     208576 ± 10%  sched_debug.cpu.clock_task.min
      2.98           +14.0%       3.40 ±  8%  sched_debug.cpu.clock_task.stddev
     21560 ±  6%     +51.5%      32664 ±  8%  sched_debug.cpu.curr->pid.max
      2634 ± 19%     +50.9%       3976 ± 22%  sched_debug.cpu.curr->pid.stddev
      0.00 ±  5%     -11.3%       0.00 ±  3%  sched_debug.cpu.next_balance.stddev
     19834 ±  7%     +39.6%      27692 ±  8%  sched_debug.cpu.nr_switches.avg
     45801 ± 19%     +39.4%      63837 ±  7%  sched_debug.cpu.nr_switches.max
     15057 ±  8%     +47.9%      22266 ± 10%  sched_debug.cpu.nr_switches.min
      4211 ± 17%     +42.4%       5997 ±  3%  sched_debug.cpu.nr_switches.stddev
    111.02 ±  8%     +42.7%     158.40 ± 12%  sched_debug.cpu.nr_uninterruptible.max
     22.69 ±  2%     +32.6%      30.08 ±  7%  sched_debug.cpu.nr_uninterruptible.stddev
     18618 ±  8%     +43.4%      26705 ±  9%  sched_debug.cpu.sched_count.avg
     43647 ± 20%     +43.0%      62423 ±  8%  sched_debug.cpu.sched_count.max
     14188 ±  8%     +53.4%      21761 ± 11%  sched_debug.cpu.sched_count.min
      3922 ± 19%     +47.9%       5802 ±  3%  sched_debug.cpu.sched_count.stddev
      8774 ±  8%     +43.0%      12545 ±  9%  sched_debug.cpu.sched_goidle.avg
     20896 ± 22%     +42.4%      29762 ±  7%  sched_debug.cpu.sched_goidle.max
      6594 ±  9%     +53.9%      10147 ± 11%  sched_debug.cpu.sched_goidle.min
      1911 ± 19%     +46.0%       2790 ±  4%  sched_debug.cpu.sched_goidle.stddev
      9004 ±  8%     +43.1%      12887 ±  9%  sched_debug.cpu.ttwu_count.avg
     76782 ±  7%     +58.7%     121825 ±  9%  sched_debug.cpu.ttwu_count.max
      5772 ±  5%     +49.2%       8610 ±  7%  sched_debug.cpu.ttwu_count.min
      7318 ±  9%     +58.4%      11593 ±  8%  sched_debug.cpu.ttwu_count.stddev
      1084 ±  8%     +50.7%       1634 ± 10%  sched_debug.cpu.ttwu_local.avg
     12068 ±  7%     +69.8%      20496 ± 11%  sched_debug.cpu.ttwu_local.max
    535.58 ± 14%     +55.6%     833.15 ± 10%  sched_debug.cpu.ttwu_local.min
      1156 ±  7%     +70.2%       1968 ± 11%  sched_debug.cpu.ttwu_local.stddev
    156245 ±  8%     +33.5%     208576 ± 10%  sched_debug.cpu_clk
    155739 ±  8%     +33.6%     208072 ± 10%  sched_debug.ktime
    156601 ±  8%     +33.4%     208936 ± 10%  sched_debug.sched_clk
    111824           +39.3%     155794        interrupts.315:PCI-MSI.376832-edge.ahci[0000:00:17.0]
    272.75 ±  2%     +40.7%     383.75        interrupts.35:PCI-MSI.31981568-edge.i40e-0000:3d:00.0:misc
     37.00 ±126%    -100.0%       0.00        interrupts.76:PCI-MSI.31981609-edge.i40e-eth0-TxRx-40
    646.00 ±  2%     +30.7%     844.50 ±  3%  interrupts.9:IO-APIC.9-fasteoi.acpi
     99689           +23.7%     123324        interrupts.CAL:Function_call_interrupts
    646.00 ±  2%     +30.7%     844.50 ±  3%  interrupts.CPU1.9:IO-APIC.9-fasteoi.acpi
    114.25 ± 24%     +67.8%     191.75 ± 17%  interrupts.CPU1.NMI:Non-maskable_interrupts
    114.25 ± 24%     +67.8%     191.75 ± 17%  interrupts.CPU1.PMI:Performance_monitoring_interrupts
    592.75 ± 59%     +58.2%     937.75 ± 40%  interrupts.CPU1.RES:Rescheduling_interrupts
    129.00 ±  5%     +62.8%     210.00 ± 31%  interrupts.CPU10.NMI:Non-maskable_interrupts
    129.00 ±  5%     +62.8%     210.00 ± 31%  interrupts.CPU10.PMI:Performance_monitoring_interrupts
    333.75 ±  4%     +47.1%     491.00 ± 12%  interrupts.CPU10.RES:Rescheduling_interrupts
     43.75 ± 12%     +99.4%      87.25 ± 42%  interrupts.CPU10.TLB:TLB_shootdowns
    131.50 ±  7%     +43.9%     189.25 ± 16%  interrupts.CPU11.NMI:Non-maskable_interrupts
    131.50 ±  7%     +43.9%     189.25 ± 16%  interrupts.CPU11.PMI:Performance_monitoring_interrupts
    357.50 ±  5%     +37.1%     490.25 ±  4%  interrupts.CPU11.RES:Rescheduling_interrupts
    359.25 ± 11%     +66.2%     597.00 ± 36%  interrupts.CPU12.RES:Rescheduling_interrupts
    372.25 ± 17%     +30.9%     487.25 ±  8%  interrupts.CPU13.RES:Rescheduling_interrupts
    370.75 ±  7%     +45.2%     538.50 ± 21%  interrupts.CPU14.RES:Rescheduling_interrupts
    378.00 ±  8%     +37.6%     520.00 ±  3%  interrupts.CPU15.RES:Rescheduling_interrupts
    355.25 ±  9%     +31.6%     467.50 ± 10%  interrupts.CPU16.RES:Rescheduling_interrupts
    494.00 ±  6%      +9.1%     538.75 ±  3%  interrupts.CPU19.CAL:Function_call_interrupts
    490.50 ± 30%     +74.8%     857.50 ± 39%  interrupts.CPU19.RES:Rescheduling_interrupts
    114.25 ± 23%     +62.4%     185.50 ± 17%  interrupts.CPU2.NMI:Non-maskable_interrupts
    114.25 ± 23%     +62.4%     185.50 ± 17%  interrupts.CPU2.PMI:Performance_monitoring_interrupts
    388.75 ± 10%     +48.6%     577.50 ± 14%  interrupts.CPU20.RES:Rescheduling_interrupts
    386.00 ±  3%     +33.5%     515.25 ±  5%  interrupts.CPU21.RES:Rescheduling_interrupts
      1537 ±  6%     +27.7%       1962 ±  3%  interrupts.CPU24.CAL:Function_call_interrupts
    389.50 ±  8%     +31.1%     510.50 ±  3%  interrupts.CPU24.RES:Rescheduling_interrupts
      1527 ± 10%     +29.0%       1971 ±  8%  interrupts.CPU25.CAL:Function_call_interrupts
    133.00 ±  8%     +37.8%     183.25 ± 21%  interrupts.CPU25.NMI:Non-maskable_interrupts
    133.00 ±  8%     +37.8%     183.25 ± 21%  interrupts.CPU25.PMI:Performance_monitoring_interrupts
    370.75 ± 15%     +32.0%     489.25 ± 10%  interrupts.CPU25.RES:Rescheduling_interrupts
      1548 ± 12%     +35.9%       2104 ± 11%  interrupts.CPU26.CAL:Function_call_interrupts
    115.75 ± 28%     +66.7%     193.00 ± 19%  interrupts.CPU26.NMI:Non-maskable_interrupts
    115.75 ± 28%     +66.7%     193.00 ± 19%  interrupts.CPU26.PMI:Performance_monitoring_interrupts
    363.00 ± 10%     +35.4%     491.50 ± 13%  interrupts.CPU26.RES:Rescheduling_interrupts
      1484 ±  8%     +33.3%       1978 ±  5%  interrupts.CPU27.CAL:Function_call_interrupts
    112.75 ± 25%     +69.8%     191.50 ± 24%  interrupts.CPU27.NMI:Non-maskable_interrupts
    112.75 ± 25%     +69.8%     191.50 ± 24%  interrupts.CPU27.PMI:Performance_monitoring_interrupts
    351.25 ±  2%     +58.0%     555.00 ± 18%  interrupts.CPU27.RES:Rescheduling_interrupts
      1420           +39.3%       1978 ±  2%  interrupts.CPU28.CAL:Function_call_interrupts
    382.75 ±  4%     +36.5%     522.50 ± 11%  interrupts.CPU28.RES:Rescheduling_interrupts
      1600 ±  4%     +27.2%       2036 ±  6%  interrupts.CPU29.CAL:Function_call_interrupts
    119.50 ± 27%     +59.2%     190.25 ± 17%  interrupts.CPU29.NMI:Non-maskable_interrupts
    119.50 ± 27%     +59.2%     190.25 ± 17%  interrupts.CPU29.PMI:Performance_monitoring_interrupts
    357.25 ±  6%     +44.3%     515.50 ±  3%  interrupts.CPU29.RES:Rescheduling_interrupts
     95.25 ± 36%    +111.5%     201.50 ± 18%  interrupts.CPU3.NMI:Non-maskable_interrupts
     95.25 ± 36%    +111.5%     201.50 ± 18%  interrupts.CPU3.PMI:Performance_monitoring_interrupts
    364.75 ±  9%     +95.6%     713.50 ± 60%  interrupts.CPU3.RES:Rescheduling_interrupts
      1730 ±  2%     +20.3%       2081 ±  7%  interrupts.CPU30.CAL:Function_call_interrupts
    115.50 ± 20%     +71.9%     198.50 ± 17%  interrupts.CPU30.NMI:Non-maskable_interrupts
    115.50 ± 20%     +71.9%     198.50 ± 17%  interrupts.CPU30.PMI:Performance_monitoring_interrupts
      1566 ±  8%     +31.4%       2059 ± 11%  interrupts.CPU31.CAL:Function_call_interrupts
    115.00 ± 24%     +64.6%     189.25 ± 19%  interrupts.CPU31.NMI:Non-maskable_interrupts
    115.00 ± 24%     +64.6%     189.25 ± 19%  interrupts.CPU31.PMI:Performance_monitoring_interrupts
    390.00 ±  9%     +31.2%     511.50 ±  5%  interrupts.CPU31.RES:Rescheduling_interrupts
      1603 ±  6%     +30.8%       2097 ±  2%  interrupts.CPU32.CAL:Function_call_interrupts
    361.00 ±  6%     +40.4%     506.75 ± 13%  interrupts.CPU32.RES:Rescheduling_interrupts
      1648 ±  6%     +26.1%       2078 ±  3%  interrupts.CPU33.CAL:Function_call_interrupts
    362.50 ±  6%     +68.5%     610.75 ± 32%  interrupts.CPU33.RES:Rescheduling_interrupts
    356.25 ±  9%     +49.1%     531.00 ± 14%  interrupts.CPU34.RES:Rescheduling_interrupts
      1538 ±  7%     +36.8%       2104 ±  6%  interrupts.CPU35.CAL:Function_call_interrupts
    356.00 ±  3%     +45.6%     518.25 ±  2%  interrupts.CPU35.RES:Rescheduling_interrupts
      1552 ±  3%     +34.4%       2086 ±  6%  interrupts.CPU36.CAL:Function_call_interrupts
    120.00 ± 25%     +62.1%     194.50 ± 19%  interrupts.CPU36.NMI:Non-maskable_interrupts
    120.00 ± 25%     +62.1%     194.50 ± 19%  interrupts.CPU36.PMI:Performance_monitoring_interrupts
      1729 ±  5%     +22.5%       2118 ±  6%  interrupts.CPU37.CAL:Function_call_interrupts
    338.50 ±  3%     +50.4%     509.00 ±  9%  interrupts.CPU37.RES:Rescheduling_interrupts
    116.75 ± 29%     +57.6%     184.00 ± 20%  interrupts.CPU38.NMI:Non-maskable_interrupts
    116.75 ± 29%     +57.6%     184.00 ± 20%  interrupts.CPU38.PMI:Performance_monitoring_interrupts
    357.25 ± 12%     +30.0%     464.50 ±  4%  interrupts.CPU38.RES:Rescheduling_interrupts
      1640 ±  3%     +22.3%       2005 ±  4%  interrupts.CPU39.CAL:Function_call_interrupts
    119.00 ± 30%     +62.2%     193.00 ± 15%  interrupts.CPU39.NMI:Non-maskable_interrupts
    119.00 ± 30%     +62.2%     193.00 ± 15%  interrupts.CPU39.PMI:Performance_monitoring_interrupts
    362.00 ±  7%     +27.2%     460.50 ±  6%  interrupts.CPU39.RES:Rescheduling_interrupts
     96.75 ± 39%     +92.5%     186.25 ± 19%  interrupts.CPU4.NMI:Non-maskable_interrupts
     96.75 ± 39%     +92.5%     186.25 ± 19%  interrupts.CPU4.PMI:Performance_monitoring_interrupts
    349.25 ±  9%     +43.7%     502.00 ±  3%  interrupts.CPU4.RES:Rescheduling_interrupts
     36.75 ±127%    -100.0%       0.00        interrupts.CPU40.76:PCI-MSI.31981609-edge.i40e-eth0-TxRx-40
    365.25 ±  5%     +34.8%     492.50 ± 10%  interrupts.CPU40.RES:Rescheduling_interrupts
      1755 ±  2%     +19.8%       2103 ±  7%  interrupts.CPU41.CAL:Function_call_interrupts
    119.75 ± 29%     +62.2%     194.25 ± 10%  interrupts.CPU41.NMI:Non-maskable_interrupts
    119.75 ± 29%     +62.2%     194.25 ± 10%  interrupts.CPU41.PMI:Performance_monitoring_interrupts
    372.00 ± 18%     +26.4%     470.25 ±  7%  interrupts.CPU41.RES:Rescheduling_interrupts
      1537 ±  2%     +39.3%       2141        interrupts.CPU42.CAL:Function_call_interrupts
    121.00 ± 29%     +61.6%     195.50 ± 10%  interrupts.CPU42.NMI:Non-maskable_interrupts
    121.00 ± 29%     +61.6%     195.50 ± 10%  interrupts.CPU42.PMI:Performance_monitoring_interrupts
      1628 ±  5%     +28.8%       2096 ±  7%  interrupts.CPU43.CAL:Function_call_interrupts
    106.50 ± 44%     +85.4%     197.50 ± 13%  interrupts.CPU43.NMI:Non-maskable_interrupts
    106.50 ± 44%     +85.4%     197.50 ± 13%  interrupts.CPU43.PMI:Performance_monitoring_interrupts
    334.25 ±  6%     +57.1%     525.00 ± 21%  interrupts.CPU43.RES:Rescheduling_interrupts
      1502 ±  4%     +42.3%       2138 ±  7%  interrupts.CPU44.CAL:Function_call_interrupts
    109.75 ± 43%     +76.1%     193.25 ± 13%  interrupts.CPU44.NMI:Non-maskable_interrupts
    109.75 ± 43%     +76.1%     193.25 ± 13%  interrupts.CPU44.PMI:Performance_monitoring_interrupts
      1585 ±  6%     +36.3%       2161 ±  6%  interrupts.CPU45.CAL:Function_call_interrupts
    131.75 ±  7%     +42.5%     187.75 ± 17%  interrupts.CPU45.NMI:Non-maskable_interrupts
    131.75 ±  7%     +42.5%     187.75 ± 17%  interrupts.CPU45.PMI:Performance_monitoring_interrupts
    316.25 ±  3%     +56.4%     494.50 ±  7%  interrupts.CPU45.RES:Rescheduling_interrupts
      1750 ±  8%     +24.0%       2170 ±  5%  interrupts.CPU46.CAL:Function_call_interrupts
    121.50 ± 28%     +60.5%     195.00 ± 12%  interrupts.CPU46.NMI:Non-maskable_interrupts
    121.50 ± 28%     +60.5%     195.00 ± 12%  interrupts.CPU46.PMI:Performance_monitoring_interrupts
    326.75 ±  5%     +42.5%     465.50 ±  9%  interrupts.CPU46.RES:Rescheduling_interrupts
      1637 ±  9%     +33.5%       2185 ±  4%  interrupts.CPU47.CAL:Function_call_interrupts
    124.00 ± 33%     +58.1%     196.00 ± 17%  interrupts.CPU47.NMI:Non-maskable_interrupts
    124.00 ± 33%     +58.1%     196.00 ± 17%  interrupts.CPU47.PMI:Performance_monitoring_interrupts
    332.25 ±  5%     +52.3%     506.00 ±  6%  interrupts.CPU47.RES:Rescheduling_interrupts
    161.75 ± 20%     +31.5%     212.75 ± 13%  interrupts.CPU48.NMI:Non-maskable_interrupts
    161.75 ± 20%     +31.5%     212.75 ± 13%  interrupts.CPU48.PMI:Performance_monitoring_interrupts
    379.25 ±  6%     +31.2%     497.75 ± 11%  interrupts.CPU48.RES:Rescheduling_interrupts
    136.25 ± 11%     +42.4%     194.00 ± 12%  interrupts.CPU49.NMI:Non-maskable_interrupts
    136.25 ± 11%     +42.4%     194.00 ± 12%  interrupts.CPU49.PMI:Performance_monitoring_interrupts
     97.50 ± 34%    +112.1%     206.75 ± 23%  interrupts.CPU5.NMI:Non-maskable_interrupts
     97.50 ± 34%    +112.1%     206.75 ± 23%  interrupts.CPU5.PMI:Performance_monitoring_interrupts
    394.25 ±  9%     +20.2%     473.75 ±  8%  interrupts.CPU5.RES:Rescheduling_interrupts
    363.00 ±  6%     +47.3%     534.75 ±  8%  interrupts.CPU50.RES:Rescheduling_interrupts
    364.50 ±  8%     +45.7%     531.00 ±  9%  interrupts.CPU51.RES:Rescheduling_interrupts
    327.75 ±  5%     +53.2%     502.25 ±  6%  interrupts.CPU53.RES:Rescheduling_interrupts
    340.00 ±  6%     +40.2%     476.75 ±  4%  interrupts.CPU54.RES:Rescheduling_interrupts
    350.50 ±  5%     +32.6%     464.75 ±  4%  interrupts.CPU55.RES:Rescheduling_interrupts
    354.50 ± 12%     +36.1%     482.50 ±  5%  interrupts.CPU56.RES:Rescheduling_interrupts
    331.75 ±  9%     +42.4%     472.50 ±  9%  interrupts.CPU57.RES:Rescheduling_interrupts
    311.50 ±  7%     +43.6%     447.25 ± 13%  interrupts.CPU58.RES:Rescheduling_interrupts
    334.00 ±  5%     +40.3%     468.50 ±  7%  interrupts.CPU59.RES:Rescheduling_interrupts
    131.00 ± 34%     +52.3%     199.50 ± 13%  interrupts.CPU6.NMI:Non-maskable_interrupts
    131.00 ± 34%     +52.3%     199.50 ± 13%  interrupts.CPU6.PMI:Performance_monitoring_interrupts
    382.25 ±  7%     +29.8%     496.25 ± 12%  interrupts.CPU6.RES:Rescheduling_interrupts
    116.75 ± 25%     +71.3%     200.00 ± 20%  interrupts.CPU60.NMI:Non-maskable_interrupts
    116.75 ± 25%     +71.3%     200.00 ± 20%  interrupts.CPU60.PMI:Performance_monitoring_interrupts
    331.25 ± 12%     +43.7%     476.00 ±  9%  interrupts.CPU60.RES:Rescheduling_interrupts
    121.25 ± 31%     +60.8%     195.00 ± 19%  interrupts.CPU61.NMI:Non-maskable_interrupts
    121.25 ± 31%     +60.8%     195.00 ± 19%  interrupts.CPU61.PMI:Performance_monitoring_interrupts
    308.25 ±  8%     +59.9%     493.00 ±  3%  interrupts.CPU61.RES:Rescheduling_interrupts
    119.25 ± 28%     +63.3%     194.75 ± 14%  interrupts.CPU62.NMI:Non-maskable_interrupts
    119.25 ± 28%     +63.3%     194.75 ± 14%  interrupts.CPU62.PMI:Performance_monitoring_interrupts
    333.00 ± 10%     +40.8%     468.75 ±  7%  interrupts.CPU62.RES:Rescheduling_interrupts
    119.25 ± 29%     +61.4%     192.50 ± 15%  interrupts.CPU63.NMI:Non-maskable_interrupts
    119.25 ± 29%     +61.4%     192.50 ± 15%  interrupts.CPU63.PMI:Performance_monitoring_interrupts
    325.25 ± 16%     +30.4%     424.00 ±  2%  interrupts.CPU63.RES:Rescheduling_interrupts
    122.50 ± 28%     +60.4%     196.50 ± 14%  interrupts.CPU64.NMI:Non-maskable_interrupts
    122.50 ± 28%     +60.4%     196.50 ± 14%  interrupts.CPU64.PMI:Performance_monitoring_interrupts
    315.75 ±  6%     +36.3%     430.50 ±  8%  interrupts.CPU64.RES:Rescheduling_interrupts
    136.25 ± 32%     +50.5%     205.00 ± 17%  interrupts.CPU65.NMI:Non-maskable_interrupts
    136.25 ± 32%     +50.5%     205.00 ± 17%  interrupts.CPU65.PMI:Performance_monitoring_interrupts
    328.75 ±  5%     +34.3%     441.50 ±  5%  interrupts.CPU65.RES:Rescheduling_interrupts
    358.00 ± 10%     +26.6%     453.25 ±  6%  interrupts.CPU66.RES:Rescheduling_interrupts
    361.00 ±  6%     +26.1%     455.25 ±  2%  interrupts.CPU67.RES:Rescheduling_interrupts
    338.00 ±  8%     +31.6%     444.75 ±  3%  interrupts.CPU68.RES:Rescheduling_interrupts
    322.50 ± 10%     +36.4%     440.00 ± 12%  interrupts.CPU69.RES:Rescheduling_interrupts
    272.75 ±  2%     +40.7%     383.75        interrupts.CPU7.35:PCI-MSI.31981568-edge.i40e-0000:3d:00.0:misc
    112.00 ± 24%     +74.8%     195.75 ± 19%  interrupts.CPU7.NMI:Non-maskable_interrupts
    112.00 ± 24%     +74.8%     195.75 ± 19%  interrupts.CPU7.PMI:Performance_monitoring_interrupts
    322.25 ±  6%     +42.2%     458.25 ±  8%  interrupts.CPU7.RES:Rescheduling_interrupts
    335.00           +33.5%     447.25 ±  4%  interrupts.CPU70.RES:Rescheduling_interrupts
      1643 ±  8%     +37.1%       2251 ±  8%  interrupts.CPU72.CAL:Function_call_interrupts
    361.75 ± 10%     +36.1%     492.25 ±  4%  interrupts.CPU72.RES:Rescheduling_interrupts
      1538 ±  7%     +42.4%       2191 ± 10%  interrupts.CPU73.CAL:Function_call_interrupts
    342.25 ± 13%     +51.0%     516.75 ± 16%  interrupts.CPU73.RES:Rescheduling_interrupts
      1540 ± 11%     +38.5%       2134 ±  5%  interrupts.CPU74.CAL:Function_call_interrupts
    299.00 ± 10%     +55.5%     465.00 ±  6%  interrupts.CPU74.RES:Rescheduling_interrupts
      1578 ± 11%     +30.9%       2067 ±  6%  interrupts.CPU75.CAL:Function_call_interrupts
    338.75 ± 13%     +39.1%     471.25 ±  6%  interrupts.CPU75.RES:Rescheduling_interrupts
      1559 ±  3%     +42.3%       2218 ±  6%  interrupts.CPU76.CAL:Function_call_interrupts
    350.25 ±  3%     +33.4%     467.25 ±  5%  interrupts.CPU76.RES:Rescheduling_interrupts
      1689 ±  9%     +30.8%       2209 ±  7%  interrupts.CPU77.CAL:Function_call_interrupts
    344.50 ± 13%     +34.9%     464.75 ±  9%  interrupts.CPU77.RES:Rescheduling_interrupts
      1610 ±  7%     +32.3%       2130 ±  4%  interrupts.CPU78.CAL:Function_call_interrupts
    331.75 ± 16%     +37.6%     456.50 ±  6%  interrupts.CPU78.RES:Rescheduling_interrupts
      1680 ±  5%     +21.8%       2046 ±  4%  interrupts.CPU79.CAL:Function_call_interrupts
    328.50 ± 14%     +45.7%     478.50 ± 12%  interrupts.CPU79.RES:Rescheduling_interrupts
    109.75 ± 25%     +84.5%     202.50 ± 19%  interrupts.CPU8.NMI:Non-maskable_interrupts
    109.75 ± 25%     +84.5%     202.50 ± 19%  interrupts.CPU8.PMI:Performance_monitoring_interrupts
    343.50 ±  6%     +42.2%     488.50 ±  8%  interrupts.CPU8.RES:Rescheduling_interrupts
    311.00 ±  3%     +44.0%     447.75 ±  6%  interrupts.CPU80.RES:Rescheduling_interrupts
     48.25 ± 13%     +49.2%      72.00 ± 25%  interrupts.CPU80.TLB:TLB_shootdowns
      1533 ±  2%     +37.0%       2101 ±  3%  interrupts.CPU81.CAL:Function_call_interrupts
    325.00 ±  4%     +78.6%     580.50 ± 27%  interrupts.CPU81.RES:Rescheduling_interrupts
     52.50 ± 12%     +58.6%      83.25 ± 31%  interrupts.CPU81.TLB:TLB_shootdowns
      1620           +25.5%       2033        interrupts.CPU82.CAL:Function_call_interrupts
    326.75 ±  7%     +83.7%     600.25 ± 28%  interrupts.CPU82.RES:Rescheduling_interrupts
      1525 ±  8%     +34.8%       2056 ±  4%  interrupts.CPU83.CAL:Function_call_interrupts
    318.75 ± 10%     +44.6%     461.00 ±  8%  interrupts.CPU83.RES:Rescheduling_interrupts
      1508 ±  5%     +30.1%       1962 ±  4%  interrupts.CPU84.CAL:Function_call_interrupts
    133.25 ±  5%     +51.6%     202.00 ± 19%  interrupts.CPU84.NMI:Non-maskable_interrupts
    133.25 ±  5%     +51.6%     202.00 ± 19%  interrupts.CPU84.PMI:Performance_monitoring_interrupts
    324.00 ±  4%     +47.2%     477.00 ± 15%  interrupts.CPU85.RES:Rescheduling_interrupts
      1462 ±  4%     +44.5%       2112 ±  7%  interrupts.CPU86.CAL:Function_call_interrupts
    131.25 ±  9%     +50.3%     197.25 ± 14%  interrupts.CPU86.NMI:Non-maskable_interrupts
    131.25 ±  9%     +50.3%     197.25 ± 14%  interrupts.CPU86.PMI:Performance_monitoring_interrupts
    352.25 ±  7%     +39.5%     491.25 ± 10%  interrupts.CPU86.RES:Rescheduling_interrupts
      1640 ± 16%     +20.7%       1980 ±  4%  interrupts.CPU87.CAL:Function_call_interrupts
    133.00 ± 15%     +47.6%     196.25 ± 16%  interrupts.CPU87.NMI:Non-maskable_interrupts
    133.00 ± 15%     +47.6%     196.25 ± 16%  interrupts.CPU87.PMI:Performance_monitoring_interrupts
    342.75 ±  7%     +32.2%     453.00 ±  5%  interrupts.CPU87.RES:Rescheduling_interrupts
      1603 ±  6%     +27.8%       2049 ± 10%  interrupts.CPU88.CAL:Function_call_interrupts
    134.50 ±  5%     +47.4%     198.25 ± 14%  interrupts.CPU88.NMI:Non-maskable_interrupts
    134.50 ±  5%     +47.4%     198.25 ± 14%  interrupts.CPU88.PMI:Performance_monitoring_interrupts
    318.50 ±  5%     +39.6%     444.75 ±  6%  interrupts.CPU88.RES:Rescheduling_interrupts
      1505 ±  6%     +41.7%       2132 ±  3%  interrupts.CPU89.CAL:Function_call_interrupts
    146.25 ± 15%     +37.3%     200.75 ± 12%  interrupts.CPU89.NMI:Non-maskable_interrupts
    146.25 ± 15%     +37.3%     200.75 ± 12%  interrupts.CPU89.PMI:Performance_monitoring_interrupts
     95.75 ± 31%    +112.0%     203.00 ± 14%  interrupts.CPU9.NMI:Non-maskable_interrupts
     95.75 ± 31%    +112.0%     203.00 ± 14%  interrupts.CPU9.PMI:Performance_monitoring_interrupts
      1624           +30.3%       2116 ±  8%  interrupts.CPU90.CAL:Function_call_interrupts
    142.00 ± 11%     +38.7%     197.00 ± 12%  interrupts.CPU90.NMI:Non-maskable_interrupts
    142.00 ± 11%     +38.7%     197.00 ± 12%  interrupts.CPU90.PMI:Performance_monitoring_interrupts
      1660 ±  7%     +25.6%       2085 ±  7%  interrupts.CPU91.CAL:Function_call_interrupts
    138.00 ± 11%     +47.3%     203.25 ± 12%  interrupts.CPU91.NMI:Non-maskable_interrupts
    138.00 ± 11%     +47.3%     203.25 ± 12%  interrupts.CPU91.PMI:Performance_monitoring_interrupts
    366.75 ± 12%     +41.0%     517.25 ± 27%  interrupts.CPU91.RES:Rescheduling_interrupts
      1627 ±  6%     +31.8%       2144 ± 11%  interrupts.CPU92.CAL:Function_call_interrupts
    148.25 ± 14%     +30.2%     193.00 ± 11%  interrupts.CPU92.NMI:Non-maskable_interrupts
    148.25 ± 14%     +30.2%     193.00 ± 11%  interrupts.CPU92.PMI:Performance_monitoring_interrupts
      1484 ±  2%     +35.9%       2018 ±  3%  interrupts.CPU93.CAL:Function_call_interrupts
    134.50 ±  9%     +42.8%     192.00 ± 17%  interrupts.CPU93.NMI:Non-maskable_interrupts
    134.50 ±  9%     +42.8%     192.00 ± 17%  interrupts.CPU93.PMI:Performance_monitoring_interrupts
    331.50 ± 10%     +36.8%     453.50 ± 12%  interrupts.CPU93.RES:Rescheduling_interrupts
      1736 ±  7%     +16.8%       2028 ±  6%  interrupts.CPU94.CAL:Function_call_interrupts
    346.00 ±  4%     +36.3%     471.50 ±  2%  interrupts.CPU94.RES:Rescheduling_interrupts
      1546 ±  3%     +34.5%       2080 ±  2%  interrupts.CPU95.CAL:Function_call_interrupts
    142.25 ± 13%     +44.3%     205.25 ± 18%  interrupts.CPU95.NMI:Non-maskable_interrupts
    142.25 ± 13%     +44.3%     205.25 ± 18%  interrupts.CPU95.PMI:Performance_monitoring_interrupts
    319.50 ± 13%     +35.3%     432.25 ± 11%  interrupts.CPU95.RES:Rescheduling_interrupts
     12290 ±  8%     +44.3%      17731 ± 15%  interrupts.NMI:Non-maskable_interrupts
     12290 ±  8%     +44.3%      17731 ± 15%  interrupts.PMI:Performance_monitoring_interrupts
     36452           +36.3%      49685 ±  3%  interrupts.RES:Rescheduling_interrupts
     85.37 ±  4%     -10.2       75.15 ±  2%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     85.41 ±  4%     -10.2       75.20 ±  2%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
     85.41 ±  4%     -10.2       75.20 ±  2%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
     86.22 ±  4%     -10.0       76.22 ±  3%  perf-profile.calltrace.cycles-pp.secondary_startup_64
     30.92 ±  6%      -9.6       21.30 ± 12%  perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     17.82 ±  9%      -9.0        8.81 ± 35%  perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     16.98 ±  3%      -2.2       14.75 ±  7%  perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle
     10.26 ±  5%      -1.3        8.92 ±  8%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      4.19 ± 14%      -1.3        2.89 ±  8%  perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry.start_secondary
      3.62 ± 18%      -1.3        2.33 ± 10%  perf-profile.calltrace.cycles-pp.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry
      3.31 ±  7%      -1.1        2.21 ± 28%  perf-profile.calltrace.cycles-pp.clockevents_program_event.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
      4.91 ±  9%      -1.1        3.82 ±  7%  perf-profile.calltrace.cycles-pp.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      1.55 ± 17%      -1.0        0.51 ±102%  perf-profile.calltrace.cycles-pp.tick_nohz_irq_exit.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
      1.49 ± 18%      -1.0        0.47 ±102%  perf-profile.calltrace.cycles-pp.ktime_get.tick_nohz_irq_exit.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt
      2.17 ±  9%      -0.8        1.37 ± 40%  perf-profile.calltrace.cycles-pp.ktime_get.clockevents_program_event.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      2.16 ± 10%      -0.6        1.58 ±  9%  perf-profile.calltrace.cycles-pp.get_next_timer_interrupt.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle
      0.70 ±  9%      -0.4        0.27 ±100%  perf-profile.calltrace.cycles-pp._find_next_bit.__next_timer_interrupt.get_next_timer_interrupt.tick_nohz_next_event.tick_nohz_get_sleep_length
      0.91 ±  7%      -0.4        0.49 ± 58%  perf-profile.calltrace.cycles-pp.native_write_msr.lapic_next_deadline.clockevents_program_event.hrtimer_interrupt.smp_apic_timer_interrupt
      1.38 ±  6%      -0.3        1.04 ± 12%  perf-profile.calltrace.cycles-pp.__next_timer_interrupt.get_next_timer_interrupt.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select
      1.05 ±  7%      -0.3        0.74 ± 13%  perf-profile.calltrace.cycles-pp.lapic_next_deadline.clockevents_program_event.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      0.63 ± 71%      +0.6        1.19 ± 11%  perf-profile.calltrace.cycles-pp.exit_mmap.mmput.do_exit.do_group_exit.__x64_sys_exit_group
      0.63 ± 71%      +0.6        1.19 ± 11%  perf-profile.calltrace.cycles-pp.mmput.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      0.92 ± 45%      +0.6        1.51 ±  8%  perf-profile.calltrace.cycles-pp.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.92 ± 45%      +0.6        1.51 ±  8%  perf-profile.calltrace.cycles-pp.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.92 ± 45%      +0.6        1.51 ±  8%  perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +0.6        0.60 ±  7%  perf-profile.calltrace.cycles-pp.do_user_addr_fault.page_fault
      0.00            +0.6        0.62 ±  8%  perf-profile.calltrace.cycles-pp.page_fault
      0.00            +0.7        0.65 ± 13%  perf-profile.calltrace.cycles-pp.prepare_to_wait_event.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent
      0.17 ±173%      +0.8        0.96 ± 14%  perf-profile.calltrace.cycles-pp.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent.__btrfs_drop_extents.insert_reserved_file_extent
      0.00            +0.8        0.79 ± 28%  perf-profile.calltrace.cycles-pp.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.95 ± 20%      +0.9        2.88 ±  9%  perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
      1.95 ± 20%      +0.9        2.89 ±  9%  perf-profile.calltrace.cycles-pp.ret_from_fork
      1.74 ± 23%      +0.9        2.68 ±  9%  perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork
      1.84 ± 22%      +1.0        2.80 ±  9%  perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork
      0.38 ±107%      +1.0        1.35 ± 11%  perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_lookup_file_extent.__btrfs_drop_extents.insert_reserved_file_extent.btrfs_finish_ordered_io
      0.38 ±107%      +1.0        1.35 ± 11%  perf-profile.calltrace.cycles-pp.btrfs_lookup_file_extent.__btrfs_drop_extents.insert_reserved_file_extent.btrfs_finish_ordered_io.btrfs_work_helper
      0.94 ± 83%      +1.0        1.93 ± 41%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.iterate_supers.ksys_sync.__x64_sys_sync
      0.95 ± 83%      +1.0        1.94 ± 41%  perf-profile.calltrace.cycles-pp._raw_spin_lock.iterate_supers.ksys_sync.__x64_sys_sync.do_syscall_64
      0.00            +1.1        1.10 ± 20%  perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_lookup_file_extent.__btrfs_drop_extents.log_one_extent.btrfs_log_changed_extents
      0.00            +1.1        1.10 ± 20%  perf-profile.calltrace.cycles-pp.btrfs_lookup_file_extent.__btrfs_drop_extents.log_one_extent.btrfs_log_changed_extents.btrfs_log_inode
      0.44 ±105%      +1.1        1.56 ± 10%  perf-profile.calltrace.cycles-pp.__btrfs_drop_extents.insert_reserved_file_extent.btrfs_finish_ordered_io.btrfs_work_helper.process_one_work
      0.44 ±105%      +1.2        1.61 ± 11%  perf-profile.calltrace.cycles-pp.insert_reserved_file_extent.btrfs_finish_ordered_io.btrfs_work_helper.process_one_work.worker_thread
      0.00            +1.2        1.17 ± 24%  perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_insert_empty_items.btrfs_csum_file_blocks.log_csums.log_one_extent
      0.00            +1.2        1.18 ± 24%  perf-profile.calltrace.cycles-pp.btrfs_insert_empty_items.btrfs_csum_file_blocks.log_csums.log_one_extent.btrfs_log_changed_extents
      0.00            +1.2        1.19 ± 48%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.prepare_to_wait_event.btrfs_tree_read_lock.btrfs_read_lock_root_node
      0.00            +1.2        1.20 ± 48%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.prepare_to_wait_event.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot
      0.00            +1.2        1.21 ± 20%  perf-profile.calltrace.cycles-pp.__btrfs_drop_extents.log_one_extent.btrfs_log_changed_extents.btrfs_log_inode.btrfs_log_inode_parent
      0.72 ± 72%      +1.2        1.94 ±  9%  perf-profile.calltrace.cycles-pp.btrfs_finish_ordered_io.btrfs_work_helper.process_one_work.worker_thread.kthread
      0.17 ±173%      +1.2        1.42 ± 18%  perf-profile.calltrace.cycles-pp.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent.__btrfs_drop_extents
      0.79 ± 71%      +1.2        2.04 ± 10%  perf-profile.calltrace.cycles-pp.btrfs_work_helper.process_one_work.worker_thread.kthread.ret_from_fork
      0.20 ±173%      +1.3        1.47 ± 27%  perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_del_csums.log_csums.log_one_extent.btrfs_log_changed_extents
      1.88 ± 44%      +1.4        3.26 ± 23%  perf-profile.calltrace.cycles-pp.iterate_supers.ksys_sync.__x64_sys_sync.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.20 ±173%      +1.4        1.58 ± 25%  perf-profile.calltrace.cycles-pp.btrfs_del_csums.log_csums.log_one_extent.btrfs_log_changed_extents.btrfs_log_inode
      0.00            +1.4        1.45 ± 34%  perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_insert_empty_items.btrfs_log_inode.btrfs_log_inode_parent.btrfs_log_dentry_safe
      0.00            +1.5        1.51 ± 33%  perf-profile.calltrace.cycles-pp.btrfs_insert_empty_items.btrfs_log_inode.btrfs_log_inode_parent.btrfs_log_dentry_safe.btrfs_sync_file
      2.16 ± 47%      +1.5        3.70 ± 21%  perf-profile.calltrace.cycles-pp.sync
      2.12 ± 48%      +1.6        3.68 ± 21%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.sync
      2.12 ± 48%      +1.6        3.68 ± 21%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.sync
      2.12 ± 48%      +1.6        3.68 ± 21%  perf-profile.calltrace.cycles-pp.__x64_sys_sync.do_syscall_64.entry_SYSCALL_64_after_hwframe.sync
      2.12 ± 48%      +1.6        3.68 ± 21%  perf-profile.calltrace.cycles-pp.ksys_sync.__x64_sys_sync.do_syscall_64.entry_SYSCALL_64_after_hwframe.sync
      0.44 ±110%      +2.0        2.41 ± 24%  perf-profile.calltrace.cycles-pp.btrfs_csum_file_blocks.log_csums.log_one_extent.btrfs_log_changed_extents.btrfs_log_inode
      0.82 ± 87%      +3.2        4.01 ± 25%  perf-profile.calltrace.cycles-pp.log_csums.log_one_extent.btrfs_log_changed_extents.btrfs_log_inode.btrfs_log_inode_parent
      1.12 ± 85%      +4.3        5.44 ± 22%  perf-profile.calltrace.cycles-pp.btrfs_log_changed_extents.btrfs_log_inode.btrfs_log_inode_parent.btrfs_log_dentry_safe.btrfs_sync_file
      1.12 ± 85%      +4.3        5.44 ± 22%  perf-profile.calltrace.cycles-pp.log_one_extent.btrfs_log_changed_extents.btrfs_log_inode.btrfs_log_inode_parent.btrfs_log_dentry_safe
      1.57 ± 79%      +6.1        7.66 ± 25%  perf-profile.calltrace.cycles-pp.btrfs_log_inode.btrfs_log_inode_parent.btrfs_log_dentry_safe.btrfs_sync_file.btrfs_file_write_iter
      1.74 ± 75%      +6.3        8.05 ± 23%  perf-profile.calltrace.cycles-pp.btrfs_log_dentry_safe.btrfs_sync_file.btrfs_file_write_iter.new_sync_write.vfs_write
      1.74 ± 75%      +6.3        8.05 ± 23%  perf-profile.calltrace.cycles-pp.btrfs_log_inode_parent.btrfs_log_dentry_safe.btrfs_sync_file.btrfs_file_write_iter.new_sync_write
      2.22 ± 76%      +6.8        8.98 ± 20%  perf-profile.calltrace.cycles-pp.btrfs_sync_file.btrfs_file_write_iter.new_sync_write.vfs_write.ksys_write
      2.59 ± 72%      +6.9        9.45 ± 19%  perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.56 ± 72%      +6.9        9.45 ± 19%  perf-profile.calltrace.cycles-pp.btrfs_file_write_iter.new_sync_write.vfs_write.ksys_write.do_syscall_64
     33.17 ±  2%      +8.7       41.90 ± 16%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      0.00            +9.5        9.45 ± 19%  perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +9.5        9.45 ± 19%  perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.63 ± 24%     +12.5       14.10 ± 14%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.63 ± 24%     +12.5       14.11 ± 14%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
     85.41 ±  4%     -10.2       75.20 ±  2%  perf-profile.children.cycles-pp.start_secondary
     86.28 ±  4%     -10.0       76.27 ±  3%  perf-profile.children.cycles-pp.do_idle
     86.22 ±  4%     -10.0       76.22 ±  3%  perf-profile.children.cycles-pp.secondary_startup_64
     86.22 ±  4%     -10.0       76.22 ±  3%  perf-profile.children.cycles-pp.cpu_startup_entry
     17.97 ±  9%      -9.0        8.93 ± 35%  perf-profile.children.cycles-pp.menu_select
     24.33 ±  5%      -5.9       18.46 ±  6%  perf-profile.children.cycles-pp.apic_timer_interrupt
      3.02 ± 58%      -3.0        0.06 ± 59%  perf-profile.children.cycles-pp.write
     17.22 ±  4%      -2.1       15.08 ±  8%  perf-profile.children.cycles-pp.smp_apic_timer_interrupt
      5.49 ±  5%      -2.0        3.51 ± 30%  perf-profile.children.cycles-pp.ktime_get
      4.23 ± 15%      -1.3        2.92 ±  8%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      3.66 ± 18%      -1.3        2.36 ± 10%  perf-profile.children.cycles-pp.tick_nohz_next_event
     10.44 ±  5%      -1.3        9.16 ±  8%  perf-profile.children.cycles-pp.hrtimer_interrupt
      3.37 ±  7%      -1.1        2.27 ± 28%  perf-profile.children.cycles-pp.clockevents_program_event
      5.21 ±  8%      -1.1        4.15 ±  9%  perf-profile.children.cycles-pp.irq_exit
      1.56 ± 17%      -0.9        0.70 ± 47%  perf-profile.children.cycles-pp.tick_nohz_irq_exit
      2.18 ± 11%      -0.6        1.59 ± 10%  perf-profile.children.cycles-pp.get_next_timer_interrupt
      0.95 ± 34%      -0.6        0.37 ± 52%  perf-profile.children.cycles-pp.timekeeping_max_deferment
      1.40 ±  8%      -0.4        0.96 ± 14%  perf-profile.children.cycles-pp.native_write_msr
      1.24 ±  5%      -0.4        0.80 ± 15%  perf-profile.children.cycles-pp.native_irq_return_iret
      1.53 ±  7%      -0.4        1.15 ± 12%  perf-profile.children.cycles-pp.__next_timer_interrupt
      1.08 ±  7%      -0.3        0.77 ± 12%  perf-profile.children.cycles-pp.lapic_next_deadline
      0.97 ±  9%      -0.3        0.70 ± 14%  perf-profile.children.cycles-pp.read_tsc
      1.04 ±  4%      -0.3        0.78 ± 13%  perf-profile.children.cycles-pp._find_next_bit
      0.40 ± 23%      -0.1        0.26 ± 17%  perf-profile.children.cycles-pp.rcu_idle_exit
      0.35 ± 31%      -0.1        0.24 ± 13%  perf-profile.children.cycles-pp.hrtimer_get_next_event
      0.20 ±  9%      -0.1        0.14 ±  8%  perf-profile.children.cycles-pp.call_cpuidle
      0.30 ± 12%      -0.1        0.24 ± 11%  perf-profile.children.cycles-pp.calc_global_load_tick
      0.10 ± 14%      -0.0        0.06 ± 64%  perf-profile.children.cycles-pp.tick_sched_do_timer
      0.09 ±  7%      -0.0        0.05 ± 60%  perf-profile.children.cycles-pp.pagecache_get_page
      0.05 ± 58%      +0.0        0.08 ±  8%  perf-profile.children.cycles-pp.lookup_fast
      0.07 ± 17%      +0.0        0.11 ± 14%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.04 ± 60%      +0.0        0.08 ± 19%  perf-profile.children.cycles-pp.__kernel_text_address
      0.07 ± 41%      +0.0        0.11 ± 31%  perf-profile.children.cycles-pp.poll_idle
      0.12 ± 14%      +0.0        0.15 ±  9%  perf-profile.children.cycles-pp.walk_component
      0.06 ± 26%      +0.0        0.10 ± 18%  perf-profile.children.cycles-pp.read_block_for_search
      0.04 ± 59%      +0.0        0.09 ± 17%  perf-profile.children.cycles-pp.unwind_get_return_address
      0.06 ± 58%      +0.0        0.10 ±  8%  perf-profile.children.cycles-pp.filename_lookup
      0.06 ± 60%      +0.0        0.10 ±  7%  perf-profile.children.cycles-pp.path_lookupat
      0.05 ± 60%      +0.0        0.10 ± 15%  perf-profile.children.cycles-pp.kmem_cache_free
      0.09 ± 26%      +0.0        0.14 ± 13%  perf-profile.children.cycles-pp.elf_map
      0.07 ± 24%      +0.1        0.12 ± 37%  perf-profile.children.cycles-pp.get_cpu_device
      0.08 ± 29%      +0.1        0.13 ± 22%  perf-profile.children.cycles-pp.mmap64
      0.03 ±100%      +0.1        0.09 ±  9%  perf-profile.children.cycles-pp.down_read
      0.01 ±173%      +0.1        0.07 ± 26%  perf-profile.children.cycles-pp.alloc_pages_vma
      0.05 ± 61%      +0.1        0.11 ± 21%  perf-profile.children.cycles-pp.btrfs_map_bio
      0.03 ±102%      +0.1        0.09 ± 28%  perf-profile.children.cycles-pp.sync_inodes_sb
      0.05 ± 59%      +0.1        0.11 ±  9%  perf-profile.children.cycles-pp.btrfs_get_32
      0.03 ±100%      +0.1        0.10 ± 26%  perf-profile.children.cycles-pp.mutex_spin_on_owner
      0.00            +0.1        0.07 ± 17%  perf-profile.children.cycles-pp.generic_bin_search
      0.00            +0.1        0.07 ±  7%  perf-profile.children.cycles-pp.generic_file_buffered_read
      0.08 ± 68%      +0.1        0.15 ± 23%  perf-profile.children.cycles-pp.write_one_eb
      0.08 ± 36%      +0.1        0.15 ±  7%  perf-profile.children.cycles-pp.vma_interval_tree_insert
      0.13 ± 62%      +0.1        0.21 ± 13%  perf-profile.children.cycles-pp.check_leaf
      0.11 ± 34%      +0.1        0.18 ± 11%  perf-profile.children.cycles-pp.vma_link
      0.08 ± 17%      +0.1        0.16 ± 32%  perf-profile.children.cycles-pp.account_process_tick
      0.13 ± 62%      +0.1        0.21 ± 14%  perf-profile.children.cycles-pp.btree_csum_one_bio
      0.08 ± 63%      +0.1        0.17 ±  7%  perf-profile.children.cycles-pp.wait_task_zombie
      0.12 ± 48%      +0.1        0.21 ± 19%  perf-profile.children.cycles-pp.__extent_writepage
      0.03 ±173%      +0.1        0.12 ± 12%  perf-profile.children.cycles-pp.btrfs_delalloc_reserve_metadata
      0.01 ±173%      +0.1        0.11 ± 25%  perf-profile.children.cycles-pp.find_free_extent
      0.07 ± 63%      +0.1        0.17 ±  7%  perf-profile.children.cycles-pp.release_task
      0.01 ±173%      +0.1        0.11 ± 28%  perf-profile.children.cycles-pp.btrfs_reserve_extent
      0.15 ± 44%      +0.1        0.25 ± 18%  perf-profile.children.cycles-pp.extent_write_cache_pages
      0.14 ± 63%      +0.1        0.24 ± 15%  perf-profile.children.cycles-pp.btree_submit_bio_hook
      0.13 ± 41%      +0.1        0.23 ±  7%  perf-profile.children.cycles-pp.schedule_idle
      0.73 ±  7%      +0.1        0.83 ±  3%  perf-profile.children.cycles-pp.find_busiest_group
      0.68 ±  9%      +0.1        0.79 ±  3%  perf-profile.children.cycles-pp.update_sd_lb_stats
      0.17 ± 22%      +0.1        0.29 ±  9%  perf-profile.children.cycles-pp.ksys_mmap_pgoff
      0.20 ± 44%      +0.1        0.32 ±  7%  perf-profile.children.cycles-pp.submit_one_bio
      0.19 ± 43%      +0.1        0.32 ±  9%  perf-profile.children.cycles-pp.extent_writepages
      0.20 ± 28%      +0.1        0.33 ±  7%  perf-profile.children.cycles-pp.mmap_region
      0.10 ± 40%      +0.1        0.23 ± 28%  perf-profile.children.cycles-pp.wait_for_commit
      0.23 ± 20%      +0.1        0.36 ±  9%  perf-profile.children.cycles-pp.vm_mmap_pgoff
      0.23 ± 43%      +0.1        0.37 ± 15%  perf-profile.children.cycles-pp.wb_workfn
      0.15 ± 55%      +0.1        0.30 ±  9%  perf-profile.children.cycles-pp.btrfs_fdatawrite_range
      0.22 ± 46%      +0.1        0.37 ± 16%  perf-profile.children.cycles-pp.wb_writeback
      0.22 ± 45%      +0.1        0.36 ± 16%  perf-profile.children.cycles-pp.__writeback_single_inode
      0.21 ± 25%      +0.1        0.36 ±  9%  perf-profile.children.cycles-pp.do_mmap
      0.08 ± 66%      +0.1        0.23 ± 31%  perf-profile.children.cycles-pp.btrfs_wait_for_commit
      0.03 ±102%      +0.1        0.18 ± 28%  perf-profile.children.cycles-pp.copy_items
      0.22 ± 45%      +0.2        0.37 ± 16%  perf-profile.children.cycles-pp.writeback_sb_inodes
      0.03 ±100%      +0.2        0.18 ± 37%  perf-profile.children.cycles-pp.btrfs_next_old_leaf
      0.19 ± 50%      +0.2        0.35 ± 17%  perf-profile.children.cycles-pp.__do_sys_wait4
      0.19 ± 50%      +0.2        0.35 ± 17%  perf-profile.children.cycles-pp.kernel_wait4
      0.19 ± 49%      +0.2        0.35 ± 17%  perf-profile.children.cycles-pp.do_wait
      0.91 ±  8%      +0.2        1.07 ±  7%  perf-profile.children.cycles-pp.load_balance
      0.18 ± 46%      +0.2        0.34 ±  6%  perf-profile.children.cycles-pp.__filemap_fdatawrite_range
      0.43 ± 27%      +0.2        0.60 ± 11%  perf-profile.children.cycles-pp.unmap_vmas
      0.41 ± 29%      +0.2        0.58 ± 11%  perf-profile.children.cycles-pp.unmap_page_range
      0.50 ± 31%      +0.2        0.67 ±  5%  perf-profile.children.cycles-pp.search_binary_handler
      0.50 ± 31%      +0.2        0.67 ±  5%  perf-profile.children.cycles-pp.load_elf_binary
      0.31 ± 36%      +0.2        0.48 ±  9%  perf-profile.children.cycles-pp.arch_stack_walk
      0.21 ± 50%      +0.2        0.38 ± 15%  perf-profile.children.cycles-pp.btree_write_cache_pages
      0.28 ±  9%      +0.2        0.45 ±  9%  perf-profile.children.cycles-pp.newidle_balance
      0.10 ± 62%      +0.2        0.27 ± 30%  perf-profile.children.cycles-pp.btrfs_free_path
      0.24 ± 39%      +0.2        0.43 ± 15%  perf-profile.children.cycles-pp.sched_ttwu_pending
      0.20 ± 61%      +0.2        0.41 ± 22%  perf-profile.children.cycles-pp.queued_read_lock_slowpath
      0.49 ± 13%      +0.2        0.70 ±  2%  perf-profile.children.cycles-pp.schedule
      0.07 ± 62%      +0.2        0.29 ± 26%  perf-profile.children.cycles-pp.btrfs_unlock_up_safe
      0.29 ± 17%      +0.2        0.52 ±  9%  perf-profile.children.cycles-pp.pick_next_task_fair
      0.77 ± 17%      +0.2        1.00 ±  4%  perf-profile.children.cycles-pp.do_user_addr_fault
      0.84 ± 17%      +0.2        1.08 ±  4%  perf-profile.children.cycles-pp.page_fault
      0.30 ± 49%      +0.2        0.54 ± 14%  perf-profile.children.cycles-pp.release_pages
      0.15 ± 39%      +0.3        0.42 ± 14%  perf-profile.children.cycles-pp.setup_items_for_insert
      0.24 ± 39%      +0.3        0.51 ± 24%  perf-profile.children.cycles-pp.btrfs_attach_transaction_barrier
      0.31 ± 51%      +0.3        0.58 ± 25%  perf-profile.children.cycles-pp.wait_current_trans
      0.34 ± 52%      +0.3        0.62 ± 25%  perf-profile.children.cycles-pp.start_transaction
      0.33 ± 55%      +0.3        0.61 ± 13%  perf-profile.children.cycles-pp.tlb_finish_mmu
      0.33 ± 57%      +0.3        0.61 ± 14%  perf-profile.children.cycles-pp.tlb_flush_mmu
      0.46 ± 38%      +0.3        0.76 ±  9%  perf-profile.children.cycles-pp.autoremove_wake_function
      0.25 ± 39%      +0.3        0.55 ± 24%  perf-profile.children.cycles-pp.btrfs_sync_fs
      0.50 ± 36%      +0.3        0.80 ±  8%  perf-profile.children.cycles-pp.__wake_up_common
      0.40 ± 40%      +0.3        0.70 ± 11%  perf-profile.children.cycles-pp.do_writepages
      0.61 ± 33%      +0.3        0.93 ± 10%  perf-profile.children.cycles-pp.try_to_wake_up
      0.62 ± 19%      +0.3        0.97        perf-profile.children.cycles-pp.__sched_text_start
      0.36 ± 70%      +0.4        0.72 ± 17%  perf-profile.children.cycles-pp.__mutex_lock
      0.97 ± 34%      +0.5        1.50 ±  8%  perf-profile.children.cycles-pp.mmput
      0.96 ± 34%      +0.5        1.49 ±  8%  perf-profile.children.cycles-pp.exit_mmap
      0.47 ± 56%      +0.6        1.05 ± 25%  perf-profile.children.cycles-pp.btrfs_release_path
      0.94 ± 44%      +0.6        1.53 ±  8%  perf-profile.children.cycles-pp.__x64_sys_exit_group
      0.94 ± 44%      +0.6        1.53 ±  8%  perf-profile.children.cycles-pp.do_group_exit
      0.94 ± 44%      +0.6        1.53 ±  8%  perf-profile.children.cycles-pp.do_exit
      0.26 ± 80%      +0.6        0.90 ± 25%  perf-profile.children.cycles-pp.btrfs_lookup_csum
      0.70 ± 47%      +0.9        1.58 ± 12%  perf-profile.children.cycles-pp.__wake_up_common_lock
      1.95 ± 20%      +0.9        2.88 ±  9%  perf-profile.children.cycles-pp.kthread
      1.97 ± 20%      +0.9        2.91 ± 10%  perf-profile.children.cycles-pp.ret_from_fork
      1.74 ± 23%      +0.9        2.68 ±  9%  perf-profile.children.cycles-pp.process_one_work
      1.84 ± 22%      +1.0        2.80 ±  9%  perf-profile.children.cycles-pp.worker_thread
      0.62 ± 50%      +1.0        1.61 ± 11%  perf-profile.children.cycles-pp.insert_reserved_file_extent
      0.80 ± 52%      +1.1        1.94 ±  9%  perf-profile.children.cycles-pp.btrfs_finish_ordered_io
      0.88 ± 51%      +1.2        2.04 ± 10%  perf-profile.children.cycles-pp.btrfs_work_helper
      2.05 ± 44%      +1.3        3.40 ± 25%  perf-profile.children.cycles-pp._raw_spin_lock
      0.29 ±103%      +1.4        1.65 ± 25%  perf-profile.children.cycles-pp.btrfs_del_csums
      1.89 ± 43%      +1.4        3.33 ± 24%  perf-profile.children.cycles-pp.iterate_supers
      2.16 ± 47%      +1.5        3.70 ± 21%  perf-profile.children.cycles-pp.sync
      2.13 ± 47%      +1.7        3.79 ± 21%  perf-profile.children.cycles-pp.__x64_sys_sync
      2.13 ± 47%      +1.7        3.79 ± 21%  perf-profile.children.cycles-pp.ksys_sync
      0.80 ± 50%      +1.7        2.51 ± 11%  perf-profile.children.cycles-pp.btrfs_lookup_file_extent
      0.86 ± 64%      +1.8        2.71 ± 21%  perf-profile.children.cycles-pp.finish_wait
      0.57 ± 75%      +1.9        2.47 ± 23%  perf-profile.children.cycles-pp.btrfs_csum_file_blocks
      0.83 ± 50%      +2.0        2.78 ±  9%  perf-profile.children.cycles-pp.__btrfs_drop_extents
      1.00 ± 50%      +2.4        3.41 ± 25%  perf-profile.children.cycles-pp.btrfs_insert_empty_items
      1.05 ± 58%      +2.4        3.48 ± 27%  perf-profile.children.cycles-pp.btrfs_lock_root_node
      1.07 ± 59%      +2.4        3.52 ± 27%  perf-profile.children.cycles-pp.btrfs_tree_lock
      0.83 ± 86%      +3.2        4.06 ± 25%  perf-profile.children.cycles-pp.log_csums
      1.71 ± 51%      +3.3        4.98 ± 19%  perf-profile.children.cycles-pp.btrfs_tree_read_lock
      1.75 ± 50%      +3.3        5.02 ± 18%  perf-profile.children.cycles-pp.btrfs_read_lock_root_node
      1.74 ± 51%      +3.8        5.58 ± 17%  perf-profile.children.cycles-pp.prepare_to_wait_event
      1.13 ± 83%      +4.3        5.44 ± 22%  perf-profile.children.cycles-pp.btrfs_log_changed_extents
      1.13 ± 83%      +4.3        5.44 ± 22%  perf-profile.children.cycles-pp.log_one_extent
      3.23 ± 50%      +5.9        9.14 ± 20%  perf-profile.children.cycles-pp.btrfs_search_slot
      3.76 ± 41%      +6.0        9.80 ± 17%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      1.60 ± 75%      +6.1        7.66 ± 25%  perf-profile.children.cycles-pp.btrfs_log_inode
      1.77 ± 71%      +6.3        8.05 ± 23%  perf-profile.children.cycles-pp.btrfs_log_dentry_safe
      1.77 ± 71%      +6.3        8.05 ± 23%  perf-profile.children.cycles-pp.btrfs_log_inode_parent
      3.00 ± 58%      +6.5        9.51 ± 19%  perf-profile.children.cycles-pp.ksys_write
      3.00 ± 59%      +6.5        9.51 ± 19%  perf-profile.children.cycles-pp.vfs_write
      2.99 ± 59%      +6.5        9.50 ± 19%  perf-profile.children.cycles-pp.new_sync_write
      2.28 ± 72%      +6.7        8.98 ± 20%  perf-profile.children.cycles-pp.btrfs_sync_file
      2.64 ± 65%      +6.8        9.45 ± 19%  perf-profile.children.cycles-pp.btrfs_file_write_iter
      4.74 ± 56%      +7.8       12.51 ± 19%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
     33.38 ±  2%      +8.7       42.11 ± 16%  perf-profile.children.cycles-pp.intel_idle
     10.22 ± 37%      +9.0       19.20 ± 13%  perf-profile.children.cycles-pp.do_syscall_64
     10.23 ± 37%      +9.0       19.21 ± 13%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     13.49 ±  9%      -7.8        5.64 ± 57%  perf-profile.self.cycles-pp.menu_select
     13.16 ± 10%      -7.5        5.63 ± 57%  perf-profile.self.cycles-pp.cpuidle_enter_state
      4.67 ±  6%      -1.7        2.93 ± 33%  perf-profile.self.cycles-pp.ktime_get
      0.93 ± 34%      -0.6        0.36 ± 55%  perf-profile.self.cycles-pp.timekeeping_max_deferment
      1.24 ±  5%      -0.4        0.80 ± 15%  perf-profile.self.cycles-pp.native_irq_return_iret
      1.40 ±  8%      -0.4        0.96 ± 14%  perf-profile.self.cycles-pp.native_write_msr
      0.95 ±  8%      -0.3        0.67 ± 14%  perf-profile.self.cycles-pp.read_tsc
      1.02 ±  3%      -0.3        0.76 ± 13%  perf-profile.self.cycles-pp._find_next_bit
      0.47 ± 29%      -0.2        0.28 ±  3%  perf-profile.self.cycles-pp.tick_nohz_next_event
      0.59 ± 12%      -0.1        0.47 ± 11%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.22 ± 25%      -0.1        0.15 ± 16%  perf-profile.self.cycles-pp.tick_sched_timer
      0.17 ± 15%      -0.1        0.10 ± 25%  perf-profile.self.cycles-pp.rcu_idle_exit
      0.20 ±  9%      -0.1        0.14 ±  8%  perf-profile.self.cycles-pp.call_cpuidle
      0.29 ± 13%      -0.1        0.23 ±  9%  perf-profile.self.cycles-pp.calc_global_load_tick
      0.12 ±  6%      -0.0        0.09 ± 17%  perf-profile.self.cycles-pp.apic_timer_interrupt
      0.07 ± 31%      +0.0        0.11 ± 30%  perf-profile.self.cycles-pp.poll_idle
      0.01 ±173%      +0.0        0.06 ± 20%  perf-profile.self.cycles-pp.tick_nohz_irq_exit
      0.03 ±102%      +0.1        0.08 ±  8%  perf-profile.self.cycles-pp.down_read
      0.07 ± 28%      +0.1        0.12 ± 16%  perf-profile.self.cycles-pp.try_to_wake_up
      0.07 ± 26%      +0.1        0.12 ± 37%  perf-profile.self.cycles-pp.get_cpu_device
      0.00            +0.1        0.05 ±  9%  perf-profile.self.cycles-pp.generic_bin_search
      0.02 ±173%      +0.1        0.08 ± 14%  perf-profile.self.cycles-pp.tick_nohz_get_sleep_length
      0.03 ±100%      +0.1        0.10 ± 26%  perf-profile.self.cycles-pp.mutex_spin_on_owner
      0.01 ±173%      +0.1        0.08 ± 34%  perf-profile.self.cycles-pp.prepare_to_wait_event
      0.05 ± 62%      +0.1        0.12 ± 27%  perf-profile.self.cycles-pp.__sched_text_start
      0.08 ± 36%      +0.1        0.15 ±  7%  perf-profile.self.cycles-pp.vma_interval_tree_insert
      0.08 ± 23%      +0.1        0.16 ± 32%  perf-profile.self.cycles-pp.account_process_tick
      0.25 ± 16%      +0.1        0.33 ±  5%  perf-profile.self.cycles-pp.filemap_map_pages
      0.07 ± 60%      +0.1        0.15 ± 21%  perf-profile.self.cycles-pp.__account_scheduler_latency
      0.18 ± 15%      +0.1        0.28 ± 13%  perf-profile.self.cycles-pp.unmap_page_range
      0.44 ±  8%      +0.1        0.54 ±  2%  perf-profile.self.cycles-pp.update_sd_lb_stats
      0.18 ± 53%      +0.2        0.38 ± 12%  perf-profile.self.cycles-pp.release_pages
      4.72 ± 56%      +7.7       12.45 ± 19%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
     33.37 ±  2%      +8.7       42.06 ± 16%  perf-profile.self.cycles-pp.intel_idle
     56291           +38.9%      78202        softirqs.BLOCK
     45264 ±  4%     +24.5%      56345 ±  3%  softirqs.CPU0.RCU
     44834 ±  5%     +33.7%      59948 ±  5%  softirqs.CPU0.SCHED
    114870 ±  3%     +24.6%     143114 ±  3%  softirqs.CPU0.TIMER
     45447 ±  2%     +26.6%      57546 ±  4%  softirqs.CPU1.RCU
     41056 ±  4%     +35.3%      55553 ±  4%  softirqs.CPU1.SCHED
    113951 ±  3%     +25.2%     142623 ±  4%  softirqs.CPU1.TIMER
     45256 ±  2%     +30.9%      59221 ±  4%  softirqs.CPU10.RCU
     39701 ±  4%     +38.7%      55084 ±  5%  softirqs.CPU10.SCHED
    111905           +35.8%     151997 ± 10%  softirqs.CPU10.TIMER
     45670 ±  2%     +30.1%      59420 ±  4%  softirqs.CPU11.RCU
     39539 ±  4%     +37.9%      54537 ±  4%  softirqs.CPU11.SCHED
    112377           +33.3%     149762 ±  9%  softirqs.CPU11.TIMER
     45005 ±  2%     +29.4%      58237 ±  4%  softirqs.CPU12.RCU
     39022 ±  5%     +35.7%      52970 ±  8%  softirqs.CPU12.SCHED
    112684 ±  3%     +27.7%     143907 ±  4%  softirqs.CPU12.TIMER
     45086 ±  2%     +29.1%      58197 ±  4%  softirqs.CPU13.RCU
     39411 ±  4%     +38.8%      54692 ±  5%  softirqs.CPU13.SCHED
    111486 ±  2%     +30.6%     145639 ±  5%  softirqs.CPU13.TIMER
     44447 ±  2%     +31.3%      58347 ±  4%  softirqs.CPU14.RCU
     39542 ±  4%     +38.7%      54857 ±  5%  softirqs.CPU14.SCHED
    112206           +34.8%     151252 ±  9%  softirqs.CPU14.TIMER
     42513 ±  4%     +34.5%      57187 ±  5%  softirqs.CPU15.RCU
     41492 ±  8%     +31.7%      54665 ±  7%  softirqs.CPU15.SCHED
     46618 ±  2%     +22.3%      57033 ±  2%  softirqs.CPU16.RCU
     39954 ±  4%     +37.9%      55090 ±  4%  softirqs.CPU16.SCHED
    112008 ±  2%     +36.7%     153124 ± 10%  softirqs.CPU16.TIMER
     46950 ±  2%     +22.4%      57474 ±  3%  softirqs.CPU17.RCU
     39292 ±  5%     +40.5%      55211 ±  5%  softirqs.CPU17.SCHED
    112373 ±  2%     +37.5%     154500 ± 11%  softirqs.CPU17.TIMER
     46213 ±  3%     +17.3%      54195 ±  3%  softirqs.CPU18.RCU
     39548 ±  4%     +40.9%      55741 ±  3%  softirqs.CPU18.SCHED
    111460           +43.4%     159856 ± 17%  softirqs.CPU18.TIMER
     45870 ±  3%     +23.6%      56700 ±  2%  softirqs.CPU19.RCU
     39210 ±  6%     +35.6%      53163 ±  9%  softirqs.CPU19.SCHED
    112341           +32.4%     148753 ±  8%  softirqs.CPU19.TIMER
     44712 ±  2%     +28.9%      57653 ±  4%  softirqs.CPU2.RCU
     40373 ±  4%     +37.1%      55338 ±  5%  softirqs.CPU2.SCHED
    112513 ±  2%     +27.5%     143493 ±  3%  softirqs.CPU2.TIMER
     46291 ±  2%     +23.7%      57280        softirqs.CPU20.RCU
     39687 ±  4%     +39.4%      55306 ±  5%  softirqs.CPU20.SCHED
    111783 ±  2%     +55.1%     173418 ± 20%  softirqs.CPU20.TIMER
     46685 ±  2%     +23.0%      57445        softirqs.CPU21.RCU
     39319 ±  4%     +39.7%      54915 ±  5%  softirqs.CPU21.SCHED
    111851           +35.7%     151740 ± 10%  softirqs.CPU21.TIMER
     46786 ±  2%     +23.4%      57724        softirqs.CPU22.RCU
     39795 ±  3%     +38.3%      55026 ±  5%  softirqs.CPU22.SCHED
    111335 ±  2%     +36.9%     152398 ± 10%  softirqs.CPU22.TIMER
     47036 ±  2%     +22.5%      57639        softirqs.CPU23.RCU
     39905 ±  3%     +38.3%      55209 ±  5%  softirqs.CPU23.SCHED
    112390 ±  3%     +33.7%     150216 ±  9%  softirqs.CPU23.TIMER
     47971 ±  2%     +19.5%      57318 ±  2%  softirqs.CPU24.RCU
     40406 ±  4%     +27.4%      51477 ±  8%  softirqs.CPU24.SCHED
    115760 ±  3%     +39.5%     161516 ±  9%  softirqs.CPU24.TIMER
     46383 ±  3%     +21.1%      56170        softirqs.CPU25.RCU
     39216           +37.2%      53796 ±  4%  softirqs.CPU25.SCHED
    114037           +38.4%     157877 ± 12%  softirqs.CPU25.TIMER
     47550 ±  4%     +17.9%      56082        softirqs.CPU26.RCU
     39021 ±  3%     +36.4%      53220 ±  3%  softirqs.CPU26.SCHED
    113375 ±  3%     +39.2%     157792 ± 11%  softirqs.CPU26.TIMER
     49268 ±  7%     +18.3%      58287 ±  3%  softirqs.CPU27.RCU
     39622 ±  5%     +35.9%      53851 ±  4%  softirqs.CPU27.SCHED
    115832 ±  3%     +38.2%     160102 ± 13%  softirqs.CPU27.TIMER
     46579 ±  2%     +24.1%      57791 ±  2%  softirqs.CPU28.RCU
     39682 ±  4%     +36.8%      54268 ±  3%  softirqs.CPU28.SCHED
    114461           +40.0%     160238 ± 11%  softirqs.CPU28.TIMER
     46993 ±  2%     +22.8%      57698 ±  3%  softirqs.CPU29.RCU
     38792 ±  5%     +39.7%      54202 ±  3%  softirqs.CPU29.SCHED
    114092 ±  2%     +40.6%     160379 ± 11%  softirqs.CPU29.TIMER
     44931 ±  2%     +30.8%      58757 ±  5%  softirqs.CPU3.RCU
     39780 ±  4%     +38.8%      55195 ±  4%  softirqs.CPU3.SCHED
    113069 ±  2%     +36.7%     154564 ± 11%  softirqs.CPU3.TIMER
     46965           +23.8%      58149 ±  2%  softirqs.CPU30.RCU
     39883 ±  4%     +34.5%      53649 ±  3%  softirqs.CPU30.SCHED
    115104 ±  2%     +38.8%     159791 ± 11%  softirqs.CPU30.TIMER
     46418 ±  2%     +23.2%      57203 ±  2%  softirqs.CPU31.RCU
     38608 ±  5%     +40.2%      54110 ±  4%  softirqs.CPU31.SCHED
    113041           +39.9%     158198 ± 12%  softirqs.CPU31.TIMER
     47722 ±  2%     +22.9%      58633 ±  7%  softirqs.CPU32.RCU
     38834 ±  5%     +40.2%      54461 ±  6%  softirqs.CPU32.SCHED
    114989           +55.0%     178238 ± 15%  softirqs.CPU32.TIMER
     47567 ±  2%     +26.7%      60246 ±  4%  softirqs.CPU33.RCU
     39569 ±  5%     +35.5%      53610 ±  4%  softirqs.CPU33.SCHED
    114442 ±  2%     +39.0%     159060 ± 11%  softirqs.CPU33.TIMER
     47872 ±  2%     +28.6%      61552 ±  5%  softirqs.CPU34.RCU
     39857 ±  4%     +34.8%      53743 ±  3%  softirqs.CPU34.SCHED
    114251 ±  2%     +40.2%     160171 ± 11%  softirqs.CPU34.TIMER
     48139 ±  2%     +27.9%      61568 ±  5%  softirqs.CPU35.RCU
     38614 ±  5%     +40.9%      54398 ±  4%  softirqs.CPU35.SCHED
    114933 ±  2%     +38.8%     159564 ± 11%  softirqs.CPU35.TIMER
     47240 ±  2%     +27.5%      60246 ±  5%  softirqs.CPU36.RCU
     38336 ±  4%     +37.4%      52686 ±  4%  softirqs.CPU36.SCHED
    113870           +38.4%     157567 ± 13%  softirqs.CPU36.TIMER
     47214 ±  2%     +27.5%      60177 ±  5%  softirqs.CPU37.RCU
     39272 ±  4%     +35.8%      53314 ±  3%  softirqs.CPU37.SCHED
    114309           +39.4%     159354 ± 11%  softirqs.CPU37.TIMER
     47555 ±  3%     +28.4%      61065 ±  5%  softirqs.CPU38.RCU
     38694 ±  3%     +39.1%      53811 ±  3%  softirqs.CPU38.SCHED
    114703 ±  2%     +38.5%     158823 ± 12%  softirqs.CPU38.TIMER
     47585 ±  2%     +26.3%      60078 ±  4%  softirqs.CPU39.RCU
     39789 ±  4%     +34.3%      53443 ±  5%  softirqs.CPU39.SCHED
    114256           +38.8%     158588 ± 12%  softirqs.CPU39.TIMER
     45288 ±  2%     +31.1%      59368 ±  5%  softirqs.CPU4.RCU
     39555 ±  4%     +40.1%      55412 ±  5%  softirqs.CPU4.SCHED
    112477 ±  2%     +37.2%     154308 ± 11%  softirqs.CPU4.TIMER
     47849 ±  2%     +30.8%      62580 ±  6%  softirqs.CPU40.RCU
     39746 ±  3%     +34.9%      53614 ±  6%  softirqs.CPU40.SCHED
    113623 ±  2%     +40.3%     159378 ± 12%  softirqs.CPU40.TIMER
     47753 ±  2%     +30.6%      62381 ±  6%  softirqs.CPU41.RCU
     39840 ±  4%     +36.2%      54252 ±  4%  softirqs.CPU41.SCHED
    113534 ±  2%     +40.4%     159383 ± 12%  softirqs.CPU41.TIMER
     47052 ±  2%     +22.8%      57796        softirqs.CPU42.RCU
     39715 ±  4%     +33.4%      52993 ±  4%  softirqs.CPU42.SCHED
    114262           +37.4%     157028 ± 11%  softirqs.CPU42.TIMER
     47726 ±  2%     +29.4%      61747 ±  6%  softirqs.CPU43.RCU
     39441 ±  4%     +36.6%      53873 ±  4%  softirqs.CPU43.SCHED
    114759 ±  2%     +38.7%     159182 ± 12%  softirqs.CPU43.TIMER
     47262 ±  2%     +30.7%      61781 ±  6%  softirqs.CPU44.RCU
     39376 ±  4%     +37.8%      54261 ±  4%  softirqs.CPU44.SCHED
    113600 ±  2%     +41.6%     160870 ± 12%  softirqs.CPU44.TIMER
     47604 ±  2%     +29.8%      61790 ±  5%  softirqs.CPU45.RCU
     37834 ±  5%     +43.0%      54114 ±  3%  softirqs.CPU45.SCHED
    113552           +40.5%     159573 ± 11%  softirqs.CPU45.TIMER
     48059 ±  2%     +26.2%      60656 ±  6%  softirqs.CPU46.RCU
     39537 ±  4%     +34.5%      53165 ±  4%  softirqs.CPU46.SCHED
    114736           +39.7%     160292 ± 11%  softirqs.CPU46.TIMER
     47937 ±  2%     +27.4%      61094 ±  5%  softirqs.CPU47.RCU
     39694 ±  4%     +31.2%      52079 ± 10%  softirqs.CPU47.SCHED
    114366 ±  2%     +39.7%     159719 ± 12%  softirqs.CPU47.TIMER
     45451 ±  6%     +27.0%      57729 ±  3%  softirqs.CPU48.RCU
     40921 ±  5%     +36.1%      55702 ±  3%  softirqs.CPU48.SCHED
     46424 ±  2%     +28.0%      59403 ±  3%  softirqs.CPU49.RCU
     40475 ±  3%     +38.3%      55994 ±  3%  softirqs.CPU49.SCHED
    117350 ±  8%     +30.8%     153470 ±  6%  softirqs.CPU49.TIMER
     45191 ±  2%     +31.6%      59464 ±  5%  softirqs.CPU5.RCU
     38964 ±  5%     +41.0%      54939 ±  5%  softirqs.CPU5.SCHED
    112204 ±  2%     +37.4%     154205 ± 10%  softirqs.CPU5.TIMER
     47026 ±  3%     +26.5%      59507 ±  4%  softirqs.CPU50.RCU
     42675 ±  4%     +31.5%      56098 ±  3%  softirqs.CPU50.SCHED
    119333 ±  7%     +29.6%     154679 ±  5%  softirqs.CPU50.TIMER
     46659 ±  3%     +30.6%      60938 ±  4%  softirqs.CPU51.RCU
     42017 ±  4%     +34.2%      56393 ±  3%  softirqs.CPU51.SCHED
    118962 ±  9%     +41.1%     167900 ±  7%  softirqs.CPU51.TIMER
     46210 ±  2%     +31.6%      60797 ±  4%  softirqs.CPU52.RCU
     41772 ±  4%     +31.6%      54987 ±  3%  softirqs.CPU52.SCHED
    117761 ±  8%     +40.9%     165906 ±  7%  softirqs.CPU52.TIMER
     46309 ±  3%     +31.5%      60918 ±  4%  softirqs.CPU53.RCU
     41918 ±  3%     +33.8%      56080 ±  3%  softirqs.CPU53.SCHED
    118371 ±  9%     +40.2%     165946 ±  8%  softirqs.CPU53.TIMER
     46415 ±  4%     +33.1%      61783 ±  3%  softirqs.CPU54.RCU
     41820 ±  4%     +30.7%      54654 ±  7%  softirqs.CPU54.SCHED
    118369 ±  9%     +40.1%     165859 ±  7%  softirqs.CPU54.TIMER
     46580 ±  4%     +29.4%      60284 ±  4%  softirqs.CPU55.RCU
     41121 ±  4%     +36.0%      55941 ±  3%  softirqs.CPU55.SCHED
    117379 ±  9%     +34.8%     158257 ±  6%  softirqs.CPU55.TIMER
     46053 ±  3%     +31.2%      60432 ±  4%  softirqs.CPU56.RCU
     41734 ±  4%     +34.2%      56027 ±  3%  softirqs.CPU56.SCHED
    117812 ±  9%     +40.0%     164939 ±  8%  softirqs.CPU56.TIMER
     45630 ±  3%     +30.2%      59423 ±  4%  softirqs.CPU57.RCU
     41557 ±  4%     +34.2%      55756 ±  3%  softirqs.CPU57.SCHED
    116923 ±  8%     +34.9%     157764 ±  5%  softirqs.CPU57.TIMER
     46169 ±  3%     +31.2%      60561 ±  3%  softirqs.CPU58.RCU
     41898 ±  4%     +33.5%      55916 ±  3%  softirqs.CPU58.SCHED
    118444 ±  9%     +38.8%     164456 ±  7%  softirqs.CPU58.TIMER
     46840 ±  3%     +29.7%      60753 ±  4%  softirqs.CPU59.RCU
     41343 ±  6%     +35.6%      56072 ±  3%  softirqs.CPU59.SCHED
    118703 ±  9%     +36.4%     161889 ±  7%  softirqs.CPU59.TIMER
     45327 ±  2%     +31.0%      59361 ±  4%  softirqs.CPU6.RCU
     39498 ±  4%     +39.0%      54906 ±  5%  softirqs.CPU6.SCHED
    111807 ±  2%     +35.1%     151011 ± 10%  softirqs.CPU6.TIMER
     46285 ±  4%     +29.4%      59880 ±  3%  softirqs.CPU60.RCU
     41682 ±  4%     +33.9%      55818 ±  3%  softirqs.CPU60.SCHED
    117904 ±  9%     +33.1%     156969 ±  7%  softirqs.CPU60.TIMER
     46352 ±  3%     +28.0%      59339 ±  3%  softirqs.CPU61.RCU
     41229 ±  4%     +35.0%      55662 ±  3%  softirqs.CPU61.SCHED
    117869 ±  9%     +33.0%     156799 ±  6%  softirqs.CPU61.TIMER
     45940 ±  4%     +30.0%      59722 ±  4%  softirqs.CPU62.RCU
     41280 ±  4%     +33.7%      55179 ±  4%  softirqs.CPU62.SCHED
    118169 ±  9%     +38.3%     163480 ±  6%  softirqs.CPU62.TIMER
     43014 ±  4%     +33.8%      57558 ±  6%  softirqs.CPU63.RCU
     41504 ±  4%     +34.8%      55941 ±  3%  softirqs.CPU63.SCHED
    117486 ±  9%     +28.9%     151483 ±  8%  softirqs.CPU63.TIMER
     45985 ±  4%     +24.7%      57343 ±  3%  softirqs.CPU64.RCU
     39093 ±  6%     +43.3%      56029 ±  3%  softirqs.CPU64.SCHED
    116906 ±  8%     +39.6%     163244 ±  7%  softirqs.CPU64.TIMER
     46441 ±  3%     +23.6%      57396 ±  3%  softirqs.CPU65.RCU
     40582 ±  6%     +38.5%      56221 ±  3%  softirqs.CPU65.SCHED
    118596 ±  8%     +40.1%     166122 ±  8%  softirqs.CPU65.TIMER
     45721 ±  3%     +18.9%      54380 ±  3%  softirqs.CPU66.RCU
     39594 ±  6%     +32.3%      52371 ± 10%  softirqs.CPU66.SCHED
    115949 ±  7%     +30.7%     151501 ±  5%  softirqs.CPU66.TIMER
     44688 ±  6%     +28.5%      57420 ±  2%  softirqs.CPU67.RCU
     39647 ± 11%     +31.9%      52307 ±  4%  softirqs.CPU67.SCHED
    131027 ± 15%     +23.0%     161208 ±  5%  softirqs.CPU67.TIMER
     46122 ±  3%     +23.1%      56776 ±  2%  softirqs.CPU68.RCU
     41660 ±  4%     +34.7%      56109 ±  3%  softirqs.CPU68.SCHED
    118343 ±  8%     +36.0%     160976 ± 13%  softirqs.CPU68.TIMER
     46283 ±  3%     +25.1%      57920        softirqs.CPU69.RCU
     41472 ±  4%     +34.7%      55866 ±  3%  softirqs.CPU69.SCHED
    118225 ±  9%     +38.9%     164226 ±  6%  softirqs.CPU69.TIMER
     45045 ±  3%     +29.8%      58473 ±  5%  softirqs.CPU7.RCU
     39499 ±  4%     +38.5%      54721 ±  5%  softirqs.CPU7.SCHED
    112239 ±  2%     +30.2%     146137 ±  7%  softirqs.CPU7.TIMER
     46141 ±  3%     +25.7%      58004        softirqs.CPU70.RCU
     41268 ±  3%     +33.8%      55199 ±  5%  softirqs.CPU70.SCHED
    118482 ±  8%     +38.5%     164110 ±  7%  softirqs.CPU70.TIMER
     46453 ±  4%     +25.7%      58394        softirqs.CPU71.RCU
     41915 ±  4%     +29.8%      54395 ±  7%  softirqs.CPU71.SCHED
    118270 ±  9%     +37.6%     162753 ±  6%  softirqs.CPU71.TIMER
     46397 ±  3%     +24.3%      57688 ±  2%  softirqs.CPU72.RCU
     40731 ±  2%     +30.6%      53209 ±  5%  softirqs.CPU72.SCHED
    120984 ±  8%     +42.6%     172469 ±  3%  softirqs.CPU72.TIMER
     45430 ±  4%     +24.6%      56615        softirqs.CPU73.RCU
     41395 ±  4%     +31.2%      54290        softirqs.CPU73.SCHED
    118552 ±  7%     +42.4%     168769 ±  5%  softirqs.CPU73.TIMER
     45427 ±  3%     +24.3%      56455        softirqs.CPU74.RCU
     40302 ±  2%     +31.0%      52791 ±  4%  softirqs.CPU74.SCHED
    118236 ±  7%     +42.3%     168284 ±  4%  softirqs.CPU74.TIMER
     45728 ±  2%     +25.6%      57445 ±  3%  softirqs.CPU75.RCU
     41094 ±  2%     +33.4%      54800 ±  3%  softirqs.CPU75.SCHED
    119194 ±  7%     +42.8%     170157 ±  5%  softirqs.CPU75.TIMER
     46157 ±  2%     +34.1%      61885 ±  5%  softirqs.CPU76.RCU
     40627 ±  5%     +34.6%      54665 ±  5%  softirqs.CPU76.SCHED
    119266 ±  7%     +47.2%     175585 ±  4%  softirqs.CPU76.TIMER
     46008 ±  2%     +25.4%      57681 ±  3%  softirqs.CPU77.RCU
     40697 ±  5%     +36.2%      55420 ±  2%  softirqs.CPU77.SCHED
    119944 ±  8%     +43.4%     171992 ±  5%  softirqs.CPU77.TIMER
     46125 ±  2%     +26.5%      58355 ±  2%  softirqs.CPU78.RCU
     41630 ±  3%     +33.1%      55403 ±  2%  softirqs.CPU78.SCHED
    119171 ±  7%     +44.2%     171839 ±  4%  softirqs.CPU78.TIMER
     45956 ±  3%     +25.5%      57680 ±  2%  softirqs.CPU79.RCU
     41525 ±  4%     +31.5%      54592 ±  2%  softirqs.CPU79.SCHED
    119947 ±  8%     +42.2%     170601 ±  4%  softirqs.CPU79.TIMER
     44572 ±  2%     +31.5%      58627 ±  5%  softirqs.CPU8.RCU
     39550 ±  4%     +38.7%      54867 ±  5%  softirqs.CPU8.SCHED
    111196 ±  2%     +37.4%     152754 ± 11%  softirqs.CPU8.TIMER
     46439 ±  3%     +24.8%      57941 ±  8%  softirqs.CPU80.RCU
     39264 ±  8%     +40.5%      55169 ±  3%  softirqs.CPU80.SCHED
    119477 ±  8%     +43.0%     170836 ±  6%  softirqs.CPU80.TIMER
     45703 ±  2%     +30.8%      59787 ±  5%  softirqs.CPU81.RCU
     41300 ±  3%     +32.5%      54723 ±  2%  softirqs.CPU81.SCHED
    117674 ±  6%     +43.9%     169343 ±  4%  softirqs.CPU81.TIMER
     46668 ±  3%     +32.3%      61763 ±  6%  softirqs.CPU82.RCU
     41574 ±  4%     +32.7%      55158 ±  2%  softirqs.CPU82.SCHED
    119439 ±  8%     +44.0%     172034 ±  4%  softirqs.CPU82.TIMER
     46660 ±  3%     +31.5%      61364 ±  6%  softirqs.CPU83.RCU
     41554 ±  4%     +32.3%      54980 ±  2%  softirqs.CPU83.SCHED
    119441 ±  8%     +43.3%     171122 ±  4%  softirqs.CPU83.TIMER
     46076 ±  2%     +30.7%      60240 ±  5%  softirqs.CPU84.RCU
     41515 ±  3%     +32.5%      55004 ±  2%  softirqs.CPU84.SCHED
    119203 ±  7%     +43.0%     170474 ±  3%  softirqs.CPU84.TIMER
     45923 ±  2%     +30.2%      59805 ±  5%  softirqs.CPU85.RCU
     41491 ±  4%     +28.8%      53421        softirqs.CPU85.SCHED
    119344 ±  8%     +41.5%     168872 ±  5%  softirqs.CPU85.TIMER
     45763 ±  3%     +33.4%      61049 ±  6%  softirqs.CPU86.RCU
     41277 ±  4%     +33.2%      54981 ±  3%  softirqs.CPU86.SCHED
    118599 ±  7%     +44.2%     171049 ±  5%  softirqs.CPU86.TIMER
     46074 ±  3%     +29.4%      59631 ±  4%  softirqs.CPU87.RCU
     41323 ±  4%     +31.5%      54340 ±  2%  softirqs.CPU87.SCHED
    119039 ±  8%     +42.6%     169743 ±  4%  softirqs.CPU87.TIMER
     46622 ±  3%     +33.4%      62174 ±  7%  softirqs.CPU88.RCU
     40827 ±  5%     +34.1%      54760 ±  4%  softirqs.CPU88.SCHED
    119425 ±  8%     +43.6%     171474 ±  4%  softirqs.CPU88.TIMER
     46543 ±  2%     +33.8%      62282 ±  7%  softirqs.CPU89.RCU
     41572 ±  4%     +28.9%      53592 ±  6%  softirqs.CPU89.SCHED
    119621 ±  8%     +43.7%     171937 ±  5%  softirqs.CPU89.TIMER
     44748 ±  2%     +29.8%      58099 ±  4%  softirqs.CPU9.RCU
     39586 ±  4%     +38.4%      54771 ±  5%  softirqs.CPU9.SCHED
    111609           +30.4%     145501 ±  6%  softirqs.CPU9.TIMER
     45597 ±  3%     +26.2%      57532        softirqs.CPU90.RCU
     40653 ±  4%     +35.1%      54920 ±  2%  softirqs.CPU90.SCHED
    119242 ±  7%     +41.6%     168860 ±  4%  softirqs.CPU90.TIMER
     45927 ±  3%     +32.6%      60896 ±  6%  softirqs.CPU91.RCU
     41557 ±  4%     +31.1%      54480 ±  3%  softirqs.CPU91.SCHED
    119538 ±  8%     +42.0%     169702 ±  5%  softirqs.CPU91.TIMER
     45951 ±  3%     +33.7%      61442 ±  6%  softirqs.CPU92.RCU
     40369 ±  6%     +36.6%      55151 ±  2%  softirqs.CPU92.SCHED
    119739 ±  8%     +43.4%     171664 ±  4%  softirqs.CPU92.TIMER
     46248 ±  3%     +33.0%      61496 ±  6%  softirqs.CPU93.RCU
     38968 ±  6%     +40.1%      54605 ±  2%  softirqs.CPU93.SCHED
    119612 ±  8%     +43.0%     171094 ±  4%  softirqs.CPU93.TIMER
     46714 ±  2%     +28.8%      60153 ±  6%  softirqs.CPU94.RCU
     41693 ±  3%     +32.5%      55246 ±  2%  softirqs.CPU94.SCHED
    120188 ±  7%     +42.6%     171434 ±  4%  softirqs.CPU94.TIMER
     46659 ±  3%     +31.0%      61139 ±  5%  softirqs.CPU95.RCU
     41443 ±  3%     +33.6%      55369 ±  2%  softirqs.CPU95.SCHED
    119452 ±  8%     +43.6%     171521 ±  4%  softirqs.CPU95.TIMER
   4443260 ±  2%     +27.6%    5671101 ±  3%  softirqs.RCU
   3876110 ±  3%     +35.5%    5250347 ±  3%  softirqs.SCHED
  11192465 ±  4%     +37.9%   15437794 ±  5%  softirqs.TIMER


                                                                                
                                reaim.jobs_per_min                              
                                                                                
  4600 +--------------------------------------------------------------------+   
       | O      O  O   O    O      O                                        |   
  4400 |-+           O   O    O O                    O                      |   
       |      O                        O    O                               |   
  4200 |-+                                      O  O                        |   
       |                                                                    |   
  4000 |-+                                                                  |   
       |                                               +             +      |   
  3800 |.+        .+. .+                               ::   +.  .+. + +  .+.|   
       | +..+   +.   +  :      .+..+.+.+..    +.      : :   : +.   +   +.   |   
  3600 |-+   : :        :    .+              :  +..   :  : :                |   
       |     : :         +..+             +. :        :  : :                |   
  3400 |-+    +                             +      +.:    :                 |   
       |                                             +    +                 |   
  3200 +--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                             reaim.time.minor_page_faults                       
                                                                                
  4.8e+06 +-----------------------------------------------------------------+   
          | O O  O O O O O  O O O O O  O O O O O  O O O                     |   
  4.6e+06 |-+                                                               |   
  4.4e+06 |-+                                                               |   
          |                                                                 |   
  4.2e+06 |-+                                                               |   
    4e+06 |-+                                                               |   
          |                                                                 |   
  3.8e+06 |-+                                                               |   
  3.6e+06 |-+                                                               |   
          |                                                                 |   
  3.4e+06 |-+                                                               |   
  3.2e+06 |-+                                                               |   
          |.+.+..+.+.+.+.+..+.+.+.+.+..+.+.+.+.+..+.+.+.+.+..+.+.+.+.+..+.+.|   
    3e+06 +-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                         reaim.time.voluntary_context_switches                  
                                                                                
   1.8e+06 +----------------------------------------------------------------+   
  1.75e+06 |-+                         O    O     O                         |   
           |      O       O              O    O O   O O                     |   
   1.7e+06 |-O O    O O O   O  O O O O                                      |   
  1.65e+06 |-+                                                              |   
   1.6e+06 |-+                                                              |   
  1.55e+06 |-+                                                              |   
           |                                                                |   
   1.5e+06 |-+                                                              |   
  1.45e+06 |-+                                                              |   
   1.4e+06 |-+                                       .+..                   |   
  1.35e+06 |-+                      .+. .+..       .+         .+.    .+. .+.|   
           |.+.+.. .+. .+.+.+..+.+.+   +    +.+.+.+      +.+.+   +.+.   +   |   
   1.3e+06 |-+    +   +                                                     |   
  1.25e+06 +----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                       reaim.time.involuntary_context_switches                  
                                                                                
  45000 +-------------------------------------------------------------------+   
        |                                                                   |   
  40000 |-O  O O O O  O O O O  O O O O  O O O    O O O                      |   
        |                                      O                            |   
        |                                                                   |   
  35000 |-+                                                                 |   
        |                                                                   |   
  30000 |.+..+.+.+.+..+.+.+.+..+.+.+.+..+.+.+..+.+       .+.+.+..+.        .|   
        |                                        :      +          +.+.+..+ |   
  25000 |-+                                       :    :                    |   
        |                                         :    :                    |   
        |                                         :   :                     |   
  20000 |-+                                        :.+                      |   
        |                                          +                        |   
  15000 +-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                            reaim.time.file_system_inputs                       
                                                                                
  220000 +------------------------------------------------------------------+   
  200000 |-O O  O O O O  O O O O  O O O O  O O O O O  O                     |   
         |                                                                  |   
  180000 |-+                                                                |   
  160000 |-+                                                                |   
  140000 |.+.+..+.+.+.+..+.+.+.+..+.+.+.+..+.+.+.+      +.+.+..+.+.+.+..+.+.|   
  120000 |-+                                     :      :                   |   
         |                                        :     :                   |   
  100000 |-+                                      :    :                    |   
   80000 |-+                                      :    :                    |   
   60000 |-+                                      :    :                    |   
   40000 |-+                                       :   :                    |   
         |                                         :  :                     |   
   20000 |-+                                       +..+                     |   
       0 +------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                            reaim.time.file_system_outputs                      
                                                                                
  9.5e+06 +-----------------------------------------------------------------+   
          |                                                                 |   
    9e+06 |-+                            O   O    O O                       |   
  8.5e+06 |-O O  O O O   O        O                                         |   
          |            O    O O O   O  O   O   O      O                     |   
    8e+06 |-+                                                               |   
          |                                                                 |   
  7.5e+06 |-+                                                               |   
          |                                                                 |   
    7e+06 |-+                                                               |   
  6.5e+06 |-+                                                               |   
          |                                                                 |   
    6e+06 |-+   .+.        .+.+.          .+.+.  .+.+.+. .+.. .+.        .+.|   
          |.+.+.   +.+.+.+.     +. .+.. .+     +.       +    +   +.+.+..+   |   
  5.5e+06 +-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                               reaim.std_dev_time                               
                                                                                
  16 +----------------------------------------------------------------------+   
     |                                              +                       |   
  14 |-+                                           ::                       |   
  12 |-+                                           : :                      |   
     |                                            +  :                      |   
  10 |-+                                          :  :                      |   
     |                                           :    :                     |   
   8 |-+                                         :    :                     |   
     |                                           :    :                     |   
   6 |-+                                         :    :                     |   
   4 |-+ .+.+.       .+.+.  .+.+.  .+.  .+.+.+..:      :     .+.+..        .|   
     |.+.     +..+.+.     +.     +.   +.        +      +.+.+.      +.+.+..+ |   
   2 |-O  O      O O  O          O    O    O    O O                         |   
     |      O O         O O  O O    O    O   O      O                       |   
   0 +----------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                   reaim.workload                               
                                                                                
  30000 +-------------------------------------------------------------------+   
        |                                                                   |   
        |                                                                   |   
  28000 |-+                                                                 |   
        |                                                                   |   
        |                                                                   |   
  26000 |-+                                                                 |   
        |                                                                   |   
  24000 |-+                                                                 |   
        |                                                                   |   
        |                                                                   |   
  22000 |-+                                                                 |   
        |                                                                   |   
        |                                                                   |   
  20000 +-------------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.7.0-rc7-00142-ge7a79811d0db13" of type "text/plain" (202605 bytes)

View attachment "job-script" of type "text/plain" (7919 bytes)

View attachment "job.yaml" of type "text/plain" (5501 bytes)

View attachment "reproduce" of type "text/plain" (803 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ