lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Message-ID: <20210210021252.GA26362@xsang-OptiPlex-9020>
Date:   Wed, 10 Feb 2021 10:12:52 +0800
From:   kernel test robot <oliver.sang@...el.com>
To:     Josef Bacik <josef@...icpanda.com>
Cc:     David Sterba <dsterba@...e.com>,
        Nikolay Borisov <nborisov@...e.com>,
        Johannes Thumshirn <johannes.thumshirn@....com>,
        LKML <linux-kernel@...r.kernel.org>, lkp@...ts.01.org,
        lkp@...el.com, ying.huang@...el.com, feng.tang@...el.com,
        zhengjun.xing@...el.com, linux-btrfs@...r.kernel.org
Subject: [btrfs]  8698fc4eb7:  fio.write_iops 44.9% improvement


Greeting,

FYI, we noticed a 44.9% improvement of fio.write_iops due to commit:


commit: 8698fc4eb7884eeba29adc4193f9d05fa2bed16b ("btrfs: add btrfs_reserve_data_bytes and use it")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master


in testcase: fio-basic
on test machine: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 256G memory
with following parameters:

	disk: 2pmem
	fs: btrfs
	runtime: 200s
	nr_task: 50%
	time_based: tb
	rw: write
	bs: 2M
	ioengine: sync
	test_size: 100G
	cpufreq_governor: performance
	ucode: 0x5003003

test-description: Fio is a tool that will spawn a number of threads or processes doing a particular type of I/O action as specified by the user.
test-url: https://github.com/axboe/fio





Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install                job.yaml  # job file is attached in this email
        bin/lkp split-job --compatible job.yaml
        bin/lkp run                    compatible-job.yaml

=========================================================================================
bs/compiler/cpufreq_governor/disk/fs/ioengine/kconfig/nr_task/rootfs/runtime/rw/tbox_group/test_size/testcase/time_based/ucode:
  2M/gcc-9/performance/2pmem/btrfs/sync/x86_64-rhel-8.3/50%/debian-10.4-x86_64-20200603.cgz/200s/write/lkp-csl-2sp6/100G/fio-basic/tb/0x5003003

commit: 
  a1ed0a8216 ("btrfs: add the data transaction commit logic into may_commit_transaction")
  8698fc4eb7 ("btrfs: add btrfs_reserve_data_bytes and use it")

a1ed0a8216f7c730 8698fc4eb7884eeba29adc4193f 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
      0.12 ± 26%      +0.1        0.20 ± 25%  fio.latency_100ms%
      0.99 ± 29%      -0.5        0.51 ± 14%  fio.latency_10ms%
      0.03 ± 37%      +0.0        0.06 ±  3%  fio.latency_2000ms%
     26.54 ±  4%      +1.5       28.02 ±  2%  fio.latency_2ms%
      0.06 ± 18%      -0.0        0.01 ±  3%  fio.latency_>=2000ms%
    218.26 ±  7%      -7.7%     201.53        fio.time.elapsed_time
    218.26 ±  7%      -7.7%     201.53        fio.time.elapsed_time.max
      8726 ± 12%     +35.5%      11820 ±  7%  fio.time.file_system_inputs
 1.878e+09 ±  8%     +34.9%  2.534e+09        fio.time.file_system_outputs
     11589 ± 17%     +42.7%      16537 ±  3%  fio.time.involuntary_context_switches
    361.50 ± 13%     +45.0%     524.25 ±  3%  fio.time.percent_of_cpu_this_job_got
    732.88 ±  7%     +35.7%     994.59 ±  3%  fio.time.system_time
     50.30 ±  8%     +25.2%      62.99        fio.time.user_time
    755380 ± 12%     +30.1%     982939 ±  3%  fio.time.voluntary_context_switches
    458230 ±  8%     +35.0%     618393        fio.workload
      4266 ± 14%     +44.9%       6180        fio.write_bw_MBps
  36175872 ±  2%      +9.4%   39583744 ±  5%  fio.write_clat_90%_us
  21504051 ± 11%     -29.3%   15210202        fio.write_clat_mean_us
 3.692e+08 ± 15%     -87.8%   44956878 ±  4%  fio.write_clat_stddev
      2133 ± 14%     +44.9%       3090        fio.write_iops
   1344002 ± 14%     -22.5%    1042231 ±  4%  sched_debug.cpu.avg_idle.max
     16821 ± 10%     -25.0%      12624        uptime.idle
      2260 ± 11%     +23.2%       2784        slabinfo.biovec-max.active_objs
      2322 ± 10%     +22.1%       2834        slabinfo.biovec-max.num_objs
     68.24 ±  5%     -22.9%      52.58        iostat.cpu.idle
     23.71 ± 13%     +53.1%      36.31        iostat.cpu.iowait
      7.80 ±  7%     +38.2%      10.77 ±  2%  iostat.cpu.system
    689544 ±  2%     +18.1%     814587 ±  2%  meminfo.Active
      8688 ±  6%     +20.2%      10446 ±  4%  meminfo.Active(anon)
    680856 ±  2%     +18.1%     804141 ±  2%  meminfo.Active(file)
   2531252 ± 20%     +27.9%    3238547        meminfo.Dirty
   5381391 ±  7%     -26.4%    3959708 ± 10%  meminfo.MemFree
     67.99 ±  5%     -15.8       52.16        mpstat.cpu.all.idle%
     23.93 ± 13%     +12.7       36.67        mpstat.cpu.all.iowait%
      0.97 ± 18%      -0.2        0.75        mpstat.cpu.all.irq%
      6.78 ± 11%      +3.2       10.01 ±  2%  mpstat.cpu.all.sys%
      0.25 ± 12%      +0.1        0.33        mpstat.cpu.all.usr%
  86028645 ± 24%     +63.4%  1.405e+08 ±  2%  numa-numastat.node0.local_node
  16871090 ±102%    +233.7%   56305353 ± 13%  numa-numastat.node0.numa_foreign
  86052133 ± 24%     +63.3%  1.405e+08 ±  2%  numa-numastat.node0.numa_hit
  16871090 ±102%    +233.7%   56305353 ± 13%  numa-numastat.node1.numa_miss
  16879064 ±102%    +233.8%   56336725 ± 13%  numa-numastat.node1.other_node
     67.75 ±  5%     -23.2%      52.00        vmstat.cpu.id
     23.00 ± 13%     +55.4%      35.75        vmstat.cpu.wa
     78.00 ±  6%     +54.8%     120.75 ±  6%  vmstat.io.bi
   4272994 ± 14%     +44.4%    6171693        vmstat.io.bo
   5376736 ±  6%     -27.3%    3910753 ± 10%  vmstat.memory.free
     24.50 ± 11%     +54.1%      37.75        vmstat.procs.b
      6.75 ± 16%     +48.1%      10.00 ±  7%  vmstat.procs.r
     18256 ± 19%     +51.8%      27712 ±  7%  vmstat.system.cs
    336595 ±  9%     +31.2%     441512 ±  4%  numa-meminfo.node0.Active
    335350 ±  9%     +31.2%     439913 ±  4%  numa-meminfo.node0.Active(file)
    838242 ±139%    +251.9%    2949516        numa-meminfo.node0.AnonHugePages
    920317 ±127%    +241.0%    3138453 ±  2%  numa-meminfo.node0.AnonPages
    978742 ±126%    +231.7%    3246203 ±  2%  numa-meminfo.node0.AnonPages.max
   1235416 ± 21%     +21.9%    1505953        numa-meminfo.node0.Dirty
    927141 ±126%    +328.8%    3975328 ± 11%  numa-meminfo.node0.Inactive(anon)
  17344547 ±  4%     -11.4%   15364198 ±  2%  numa-meminfo.node0.Inactive(file)
   2850605 ± 16%     -41.7%    1662734 ±  7%  numa-meminfo.node0.MemFree
      3299 ±109%    +276.1%      12409 ± 10%  numa-meminfo.node0.PageTables
      7437 ± 11%     +19.0%       8849 ±  7%  numa-meminfo.node1.Active(anon)
   2133660 ± 53%     -95.7%      91652 ± 13%  numa-meminfo.node1.AnonHugePages
   2263786 ± 49%     -92.3%     175418 ± 36%  numa-meminfo.node1.AnonPages
   2477918 ± 50%     -92.1%     194686 ± 32%  numa-meminfo.node1.AnonPages.max
   1300078 ± 19%     +34.3%    1745649        numa-meminfo.node1.Dirty
  17695442 ±  5%     +13.2%   20033271        numa-meminfo.node1.FilePages
   3376450 ± 33%     -86.5%     456196 ±102%  numa-meminfo.node1.Inactive(anon)
  15736948 ±  5%     +19.9%   18871868        numa-meminfo.node1.Inactive(file)
    100097 ±  3%     -12.1%      87939 ±  3%  numa-meminfo.node1.KReclaimable
   1124610           -74.1%     291817 ±163%  numa-meminfo.node1.Mapped
     10457 ± 33%     -83.8%       1695 ± 75%  numa-meminfo.node1.PageTables
    100097 ±  3%     -12.1%      87939 ±  3%  numa-meminfo.node1.SReclaimable
   1120336           -74.1%     289793 ±164%  numa-meminfo.node1.Shmem
   1362032 ± 21%     +26.4%    1721076        numa-meminfo.node1.Writeback
     83783 ±  9%     +31.0%     109766 ±  4%  numa-vmstat.node0.nr_active_file
    230069 ±127%    +240.9%     784207        numa-vmstat.node0.nr_anon_pages
    410.25 ±139%    +251.1%       1440        numa-vmstat.node0.nr_anon_transparent_hugepages
  65856162 ±  4%     +13.8%   74952116 ±  4%  numa-vmstat.node0.nr_dirtied
    310696 ± 21%     +19.9%     372408        numa-vmstat.node0.nr_dirty
    709270 ± 16%     -42.1%     410911 ±  6%  numa-vmstat.node0.nr_free_pages
    231775 ±126%    +328.9%     994137 ± 11%  numa-vmstat.node0.nr_inactive_anon
   4339488 ±  4%     -11.4%    3845581 ±  2%  numa-vmstat.node0.nr_inactive_file
    828.00 ±108%    +274.7%       3102 ± 10%  numa-vmstat.node0.nr_page_table_pages
  65162837 ±  4%     +13.8%   74132657 ±  4%  numa-vmstat.node0.nr_written
     83833 ±  9%     +31.0%     109859 ±  4%  numa-vmstat.node0.nr_zone_active_file
    231793 ±126%    +328.9%     994149 ± 11%  numa-vmstat.node0.nr_zone_inactive_anon
   4339897 ±  4%     -11.4%    3846191 ±  2%  numa-vmstat.node0.nr_zone_inactive_file
   9241205 ±102%    +219.1%   29489269 ± 13%  numa-vmstat.node0.numa_foreign
  48410005 ± 20%     +51.1%   73157012 ±  2%  numa-vmstat.node0.numa_hit
  48350526 ± 20%     +51.2%   73121886 ±  2%  numa-vmstat.node0.numa_local
     10.25 ± 42%    +334.1%      44.50 ± 35%  numa-vmstat.node0.workingset_activate_file
      2362 ± 35%     +85.3%       4377 ± 22%  numa-vmstat.node0.workingset_refault_file
      1855 ± 11%     +18.7%       2202 ±  7%  numa-vmstat.node1.nr_active_anon
    565821 ± 49%     -92.3%      43831 ± 36%  numa-vmstat.node1.nr_anon_pages
      1042 ± 53%     -95.8%      44.25 ± 13%  numa-vmstat.node1.nr_anon_transparent_hugepages
  69626168           +21.8%   84787550        numa-vmstat.node1.nr_dirtied
    324236 ± 19%     +33.8%     433763        numa-vmstat.node1.nr_dirty
   4426610 ±  5%     +13.3%    5017487        numa-vmstat.node1.nr_file_pages
    844658 ± 33%     -86.5%     114374 ±102%  numa-vmstat.node1.nr_inactive_anon
   3936351 ±  5%     +20.1%    4726908        numa-vmstat.node1.nr_inactive_file
    281693           -74.0%      73331 ±163%  numa-vmstat.node1.nr_mapped
      2615 ± 33%     -83.8%     424.00 ± 76%  numa-vmstat.node1.nr_page_table_pages
    280751           -74.1%      72788 ±164%  numa-vmstat.node1.nr_shmem
     25007 ±  3%     -12.2%      21952 ±  3%  numa-vmstat.node1.nr_slab_reclaimable
      2197 ±141%     -90.9%     199.25 ± 21%  numa-vmstat.node1.nr_vmscan_immediate_reclaim
    339833 ± 22%     +27.4%     432785 ±  2%  numa-vmstat.node1.nr_writeback
  68914224           +21.7%   83869403        numa-vmstat.node1.nr_written
      1855 ± 11%     +18.8%       2203 ±  7%  numa-vmstat.node1.nr_zone_active_anon
    844673 ± 33%     -86.5%     114408 ±102%  numa-vmstat.node1.nr_zone_inactive_anon
   3936761 ±  5%     +20.1%    4727651        numa-vmstat.node1.nr_zone_inactive_file
    664953 ± 20%     +30.5%     868055        numa-vmstat.node1.nr_zone_write_pending
   9241420 ±102%    +219.1%   29489878 ± 13%  numa-vmstat.node1.numa_miss
   9361134 ±101%    +216.6%   29633955 ± 13%  numa-vmstat.node1.numa_other
     42557 ±  9%     -25.1%      31856 ±  5%  numa-vmstat.node1.workingset_nodes
 2.721e+09 ± 13%     +42.0%  3.864e+09        perf-stat.i.branch-instructions
      0.66 ± 33%      -0.3        0.36 ±  2%  perf-stat.i.branch-miss-rate%
  10669528 ± 10%     +25.4%   13376018 ±  3%  perf-stat.i.branch-misses
     61.83 ±  8%      +7.2       69.02        perf-stat.i.cache-miss-rate%
 2.291e+08 ± 12%     +44.8%  3.318e+08 ±  2%  perf-stat.i.cache-misses
 3.391e+08 ± 13%     +41.6%  4.803e+08        perf-stat.i.cache-references
     18481 ± 19%     +51.9%      28079 ±  7%  perf-stat.i.context-switches
      1.98 ± 29%     -25.1%       1.48        perf-stat.i.cpi
 2.335e+10 ±  9%     +39.4%  3.254e+10 ±  2%  perf-stat.i.cpu-cycles
    129.21           +11.2%     143.65        perf-stat.i.cpu-migrations
    266.83 ± 59%     -58.5%     110.63        perf-stat.i.cycles-between-cache-misses
 4.571e+09 ± 13%     +42.5%  6.514e+09        perf-stat.i.dTLB-loads
      0.02 ±106%      -0.0        0.00 ± 12%  perf-stat.i.dTLB-store-miss-rate%
 2.448e+09 ± 13%     +42.0%  3.476e+09        perf-stat.i.dTLB-stores
   2339776 ±  2%      +6.8%    2498384        perf-stat.i.iTLB-loads
 1.543e+10 ± 13%     +42.1%  2.193e+10        perf-stat.i.instructions
      6685 ± 14%     +38.3%       9245        perf-stat.i.instructions-per-iTLB-miss
      1296 ±  6%      +7.9%       1398        perf-stat.i.major-faults
      0.24 ±  9%     +39.4%       0.34 ±  2%  perf-stat.i.metric.GHz
    105.67 ± 13%     +42.2%     150.29        perf-stat.i.metric.M/sec
     56.64 ±  7%      -6.1       50.57 ±  2%  perf-stat.i.node-load-miss-rate%
  18938854 ± 15%     +43.6%   27197409        perf-stat.i.node-load-misses
  17832878 ± 12%     +50.0%   26746431 ±  6%  perf-stat.i.node-loads
  11322336 ± 11%     +51.1%   17107076        perf-stat.i.node-store-misses
  12402049 ± 10%     +46.2%   18127169 ±  4%  perf-stat.i.node-stores
      0.39 ±  4%      -0.0        0.35 ±  2%  perf-stat.overall.branch-miss-rate%
    102.26 ±  2%      -4.1%      98.10        perf-stat.overall.cycles-between-cache-misses
      0.01 ± 44%      -0.0        0.00 ± 16%  perf-stat.overall.dTLB-store-miss-rate%
      6804 ± 15%     +35.5%       9219        perf-stat.overall.instructions-per-iTLB-miss
   7304862            -1.9%    7162882        perf-stat.overall.path-length
 2.712e+09 ± 12%     +41.8%  3.845e+09        perf-stat.ps.branch-instructions
  10640712 ± 10%     +25.2%   13327057 ±  3%  perf-stat.ps.branch-misses
 2.284e+08 ± 12%     +44.6%  3.302e+08 ±  2%  perf-stat.ps.cache-misses
 3.379e+08 ± 13%     +41.5%   4.78e+08        perf-stat.ps.cache-references
     18391 ± 19%     +51.7%      27903 ±  7%  perf-stat.ps.context-switches
 2.327e+10 ±  9%     +39.2%  3.239e+10 ±  2%  perf-stat.ps.cpu-cycles
    128.65           +11.1%     142.95        perf-stat.ps.cpu-migrations
 4.555e+09 ± 13%     +42.3%  6.482e+09        perf-stat.ps.dTLB-loads
  2.44e+09 ± 13%     +41.8%  3.459e+09        perf-stat.ps.dTLB-stores
   2329066 ±  2%      +6.7%    2485638        perf-stat.ps.iTLB-loads
 1.537e+10 ± 13%     +41.9%  2.182e+10        perf-stat.ps.instructions
      1289 ±  6%      +7.6%       1387        perf-stat.ps.major-faults
  18869282 ± 15%     +43.5%   27069424        perf-stat.ps.node-load-misses
  17779829 ± 12%     +49.7%   26623612 ±  6%  perf-stat.ps.node-loads
  11287982 ± 11%     +50.9%   17029842        perf-stat.ps.node-store-misses
  12375059 ± 10%     +45.9%   18054948 ±  4%  perf-stat.ps.node-stores
 3.344e+12 ±  7%     +32.5%  4.429e+12        perf-stat.total.instructions
      3040 ± 12%    +138.7%       7257 ±  9%  proc-vmstat.compact_daemon_wake
  41325426 ± 12%     -59.9%   16562524 ± 36%  proc-vmstat.compact_free_scanned
  17014766 ± 11%     -43.4%    9637162 ± 19%  proc-vmstat.compact_isolated
  11274632 ± 11%     -35.8%    7238400 ± 16%  proc-vmstat.compact_migrate_scanned
      6552 ±  4%     +58.0%      10351 ±  6%  proc-vmstat.kswapd_low_wmark_hit_quickly
      2170 ±  6%     +20.4%       2612 ±  4%  proc-vmstat.nr_active_anon
    170212 ±  2%     +18.2%     201264 ±  2%  proc-vmstat.nr_active_file
    796021 ±  4%      +4.1%     828460        proc-vmstat.nr_anon_pages
 2.351e+08 ±  8%     +35.0%  3.173e+08        proc-vmstat.nr_dirtied
    634214 ± 20%     +28.5%     815078        proc-vmstat.nr_dirty
   8979144            +3.5%    9296310        proc-vmstat.nr_file_pages
   1339505 ±  6%     -26.2%     988769 ±  9%  proc-vmstat.nr_free_pages
   1075895 ±  3%      +3.0%    1107872        proc-vmstat.nr_inactive_anon
   8273226            +3.5%    8559431        proc-vmstat.nr_inactive_file
      3438 ±  2%      +2.5%       3524        proc-vmstat.nr_page_table_pages
     20452 ± 76%     -94.9%       1035 ± 16%  proc-vmstat.nr_vmscan_immediate_reclaim
    661394 ± 21%     +19.8%     792085        proc-vmstat.nr_writeback
 2.347e+08 ±  8%     +34.7%  3.162e+08        proc-vmstat.nr_written
      2170 ±  6%     +20.3%       2612 ±  4%  proc-vmstat.nr_zone_active_anon
    170307 ±  2%     +18.3%     201408 ±  2%  proc-vmstat.nr_zone_active_file
   1075925 ±  3%      +3.0%    1107915        proc-vmstat.nr_zone_inactive_anon
   8274073            +3.5%    8560820        proc-vmstat.nr_zone_inactive_file
   1297330 ± 20%     +24.1%    1609875        proc-vmstat.nr_zone_write_pending
  45520486 ± 10%     +38.3%   62971594 ±  8%  proc-vmstat.numa_foreign
 1.905e+08 ± 11%     +33.9%  2.551e+08 ±  3%  proc-vmstat.numa_hit
      2004 ±  2%      +8.4%       2172        proc-vmstat.numa_huge_pte_updates
 1.905e+08 ± 11%     +33.9%  2.551e+08 ±  3%  proc-vmstat.numa_local
  45520486 ± 10%     +38.3%   62971594 ±  8%  proc-vmstat.numa_miss
  45551972 ± 10%     +38.3%   63003143 ±  8%  proc-vmstat.numa_other
   1113759 ±  2%      +5.8%    1177902        proc-vmstat.numa_pte_updates
      6808 ±  3%     +56.4%      10646 ±  6%  proc-vmstat.pageoutrun
   7434132 ±  9%     +53.5%   11407725        proc-vmstat.pgalloc_dma32
 2.301e+08 ±  8%     +34.1%  3.084e+08        proc-vmstat.pgalloc_normal
      7531 ± 72%    +150.1%      18837 ± 41%  proc-vmstat.pgdeactivate
   1305993 ±  3%      -5.7%    1231006        proc-vmstat.pgfault
 2.375e+08 ±  8%     +32.7%  3.152e+08        proc-vmstat.pgfree
     30530 ± 22%     +46.5%      44739 ± 11%  proc-vmstat.pgmigrate_fail
   8513244 ± 11%     -43.4%    4815078 ± 19%  proc-vmstat.pgmigrate_success
     17369 ±  4%     +43.0%      24843 ±  6%  proc-vmstat.pgpgin
 9.388e+08 ±  8%     +34.7%  1.265e+09        proc-vmstat.pgpgout
   8698245 ± 26%     +40.3%   12202082 ±  2%  proc-vmstat.pgscan_direct
 1.243e+08 ±  5%     +47.2%  1.831e+08 ±  4%  proc-vmstat.pgscan_file
 1.156e+08 ±  6%     +47.8%  1.709e+08 ±  4%  proc-vmstat.pgscan_kswapd
    107.25 ±173%  +8.3e+05%     894040 ± 89%  proc-vmstat.pgskip_normal
   8684455 ± 26%     +40.4%   12194054 ±  2%  proc-vmstat.pgsteal_direct
 1.243e+08 ±  5%     +46.5%  1.821e+08 ±  4%  proc-vmstat.pgsteal_file
 1.156e+08 ±  6%     +47.0%  1.699e+08 ±  4%  proc-vmstat.pgsteal_kswapd
      3.60 ± 22%      -1.7        1.91 ± 10%  perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      2.98 ± 16%      -1.3        1.71 ± 10%  perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle
      1.88 ± 16%      -0.8        1.09 ± 12%  perf-profile.calltrace.cycles-pp.asm_call_sysvec_on_stack.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      1.88 ± 16%      -0.8        1.09 ± 12%  perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      1.84 ± 16%      -0.8        1.07 ± 11%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      1.54 ± 30%      -0.8        0.77 ± 17%  perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      1.04 ± 16%      -0.4        0.61 ±  8%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack.sysvec_apic_timer_interrupt
      0.81 ± 17%      +0.2        0.98 ±  6%  perf-profile.calltrace.cycles-pp.__pagevec_lru_add_fn.pagevec_lru_move_fn.lru_cache_add.add_to_page_cache_lru.pagecache_get_page
      0.27 ±100%      +0.3        0.57 ±  4%  perf-profile.calltrace.cycles-pp.iov_iter_fault_in_readable.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write.vfs_write
      3.91 ± 15%      -1.6        2.34 ± 12%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
      3.37 ± 14%      -1.3        2.08 ± 12%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
      2.83 ± 13%      -1.0        1.82 ± 12%  perf-profile.children.cycles-pp.asm_call_sysvec_on_stack
      2.21 ± 14%      -0.8        1.38 ± 11%  perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
      2.16 ± 14%      -0.8        1.35 ± 11%  perf-profile.children.cycles-pp.hrtimer_interrupt
      1.56 ± 30%      -0.8        0.78 ± 17%  perf-profile.children.cycles-pp.menu_select
      1.26 ± 14%      -0.5        0.80 ±  9%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.94 ± 17%      -0.4        0.56 ± 16%  perf-profile.children.cycles-pp.ktime_get
      0.72 ± 20%      -0.3        0.42 ± 14%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.88 ± 11%      -0.3        0.59 ±  8%  perf-profile.children.cycles-pp.tick_sched_timer
      0.64 ± 19%      -0.3        0.38 ± 14%  perf-profile.children.cycles-pp.tick_nohz_next_event
      0.71 ± 12%      -0.2        0.48 ±  7%  perf-profile.children.cycles-pp.update_process_times
      0.61 ± 13%      -0.2        0.39 ± 13%  perf-profile.children.cycles-pp.clockevents_program_event
      0.71 ± 12%      -0.2        0.48 ±  6%  perf-profile.children.cycles-pp.tick_sched_handle
      0.58 ± 10%      -0.2        0.41 ± 13%  perf-profile.children.cycles-pp.irq_exit_rcu
      0.36 ± 20%      -0.1        0.21 ± 13%  perf-profile.children.cycles-pp.timekeeping_max_deferment
      0.28 ± 23%      -0.1        0.14 ± 16%  perf-profile.children.cycles-pp.irq_enter_rcu
      0.51 ±  4%      -0.1        0.37 ± 20%  perf-profile.children.cycles-pp.__softirqentry_text_start
      0.52 ± 11%      -0.1        0.39 ± 13%  perf-profile.children.cycles-pp.btrfs_get_chunk_map
      0.48 ±  8%      -0.1        0.35 ± 16%  perf-profile.children.cycles-pp.do_softirq_own_stack
      0.39 ± 14%      -0.1        0.27 ±  9%  perf-profile.children.cycles-pp.scheduler_tick
      0.23 ± 18%      -0.1        0.12 ± 19%  perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
      0.21 ± 22%      -0.1        0.11 ± 21%  perf-profile.children.cycles-pp.tick_irq_enter
      0.23 ± 20%      -0.1        0.13 ± 18%  perf-profile.children.cycles-pp.tick_nohz_irq_exit
      0.24 ± 13%      -0.1        0.14 ± 20%  perf-profile.children.cycles-pp.__etree_search
      0.23 ± 13%      -0.1        0.15 ± 14%  perf-profile.children.cycles-pp.rebalance_domains
      0.21 ± 20%      -0.1        0.12 ± 15%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.16 ± 20%      -0.1        0.08 ± 10%  perf-profile.children.cycles-pp.lapic_next_deadline
      0.15 ± 17%      -0.1        0.08 ± 10%  perf-profile.children.cycles-pp.irqtime_account_irq
      0.16 ± 19%      -0.1        0.10 ± 33%  perf-profile.children.cycles-pp.ktime_get_update_offsets_now
      0.14 ± 24%      -0.1        0.08 ± 27%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.13 ± 20%      -0.1        0.08 ± 24%  perf-profile.children.cycles-pp.sched_clock
      0.10 ± 14%      -0.1        0.05 ± 58%  perf-profile.children.cycles-pp.arch_cpu_idle_enter
      0.10 ± 15%      -0.1        0.04 ± 59%  perf-profile.children.cycles-pp.tsc_verify_tsc_adjust
      0.12 ± 18%      -0.1        0.07 ± 24%  perf-profile.children.cycles-pp.native_sched_clock
      0.29 ±  3%      -0.0        0.24 ±  4%  perf-profile.children.cycles-pp.btrfs_find_delalloc_range
      0.13 ±  9%      -0.0        0.09 ±  7%  perf-profile.children.cycles-pp.calc_global_load_tick
      0.11 ± 13%      -0.0        0.07 ± 17%  perf-profile.children.cycles-pp._raw_spin_trylock
      0.14 ±  6%      -0.0        0.10 ±  4%  perf-profile.children.cycles-pp.run_local_timers
      0.39            +0.0        0.43 ±  6%  perf-profile.children.cycles-pp.run_one_async_start
      0.39 ±  2%      +0.0        0.43 ±  6%  perf-profile.children.cycles-pp.btrfs_csum_one_bio
      0.01 ±173%      +0.0        0.05 ±  9%  perf-profile.children.cycles-pp.__inc_zone_state
      0.42 ±  7%      +0.1        0.50 ± 10%  perf-profile.children.cycles-pp.find_get_pages_range_tag
      0.42 ±  7%      +0.1        0.50 ± 10%  perf-profile.children.cycles-pp.pagevec_lookup_range_tag
      0.48 ± 16%      +0.1        0.58 ±  4%  perf-profile.children.cycles-pp.iov_iter_fault_in_readable
      0.04 ±115%      +0.9        0.94 ±111%  perf-profile.children.cycles-pp.flush_space
      0.00            +0.9        0.91 ±117%  perf-profile.children.cycles-pp.btrfs_reserve_data_bytes
      0.85 ± 34%      -0.5        0.33 ± 12%  perf-profile.self.cycles-pp.cpuidle_enter_state
      0.82 ± 15%      -0.3        0.50 ± 16%  perf-profile.self.cycles-pp.ktime_get
      0.36 ± 20%      -0.1        0.21 ± 13%  perf-profile.self.cycles-pp.timekeeping_max_deferment
      0.23 ± 13%      -0.1        0.14 ± 23%  perf-profile.self.cycles-pp.__etree_search
      0.21 ± 20%      -0.1        0.12 ± 15%  perf-profile.self.cycles-pp.native_irq_return_iret
      0.16 ± 20%      -0.1        0.08 ± 10%  perf-profile.self.cycles-pp.lapic_next_deadline
      0.15 ± 35%      -0.1        0.08 ± 20%  perf-profile.self.cycles-pp.read_tsc
      0.19 ± 15%      -0.1        0.12 ± 16%  perf-profile.self.cycles-pp.tick_nohz_next_event
      0.14 ± 15%      -0.1        0.08 ± 32%  perf-profile.self.cycles-pp.ktime_get_update_offsets_now
      0.10 ± 11%      -0.1        0.04 ± 59%  perf-profile.self.cycles-pp.tsc_verify_tsc_adjust
      0.09 ± 13%      -0.0        0.04 ± 58%  perf-profile.self.cycles-pp.irqtime_account_irq
      0.12 ± 16%      -0.0        0.07 ± 24%  perf-profile.self.cycles-pp.native_sched_clock
      0.13 ±  9%      -0.0        0.09 ±  4%  perf-profile.self.cycles-pp.calc_global_load_tick
      0.11 ± 13%      -0.0        0.07 ± 17%  perf-profile.self.cycles-pp._raw_spin_trylock
      0.13 ±  6%      -0.0        0.09 ±  4%  perf-profile.self.cycles-pp.run_local_timers
      0.34            +0.0        0.38 ±  5%  perf-profile.self.cycles-pp.btrfs_csum_one_bio
      0.01 ±173%      +0.0        0.05 ±  9%  perf-profile.self.cycles-pp.__inc_zone_state
      0.38 ±  7%      +0.1        0.45 ± 10%  perf-profile.self.cycles-pp.find_get_pages_range_tag
      0.46 ± 16%      +0.1        0.57 ±  4%  perf-profile.self.cycles-pp.iov_iter_fault_in_readable
      8745 ± 43%     -42.4%       5036 ±  7%  softirqs.CPU1.RCU
      8100 ± 52%     -47.4%       4262 ±  8%  softirqs.CPU10.RCU
      8481 ± 44%     -46.5%       4541 ±  6%  softirqs.CPU11.RCU
      8191 ± 49%     -48.4%       4230 ±  4%  softirqs.CPU12.RCU
      8087 ± 53%     -45.2%       4430 ±  2%  softirqs.CPU13.RCU
      8302 ± 46%     -45.6%       4519 ±  4%  softirqs.CPU14.RCU
      8070 ± 54%     -48.7%       4139 ±  3%  softirqs.CPU15.RCU
      8233 ± 56%     -46.2%       4428 ± 13%  softirqs.CPU16.RCU
      9639 ± 41%     -52.1%       4618 ±  9%  softirqs.CPU17.RCU
      8406 ± 52%     -47.0%       4453 ±  7%  softirqs.CPU18.RCU
      8120 ± 54%     -46.5%       4347 ±  3%  softirqs.CPU19.RCU
      8871 ± 49%     -47.7%       4637 ±  6%  softirqs.CPU2.RCU
      8451 ± 51%     -48.3%       4369 ±  5%  softirqs.CPU20.RCU
      8405 ± 52%     -50.8%       4134 ±  8%  softirqs.CPU21.RCU
      8366 ± 52%     -51.9%       4025 ±  4%  softirqs.CPU22.RCU
      8639 ± 49%     -50.5%       4275 ±  5%  softirqs.CPU24.RCU
      8483 ± 51%     -50.5%       4203 ±  2%  softirqs.CPU25.RCU
      9608 ± 48%     -55.7%       4252 ±  4%  softirqs.CPU26.RCU
      8246 ± 52%     -49.8%       4141 ±  7%  softirqs.CPU27.RCU
      8739 ± 48%     -52.5%       4149        softirqs.CPU28.RCU
      8469 ± 56%     -51.3%       4125 ±  2%  softirqs.CPU29.RCU
      8480 ± 39%     -45.6%       4614 ±  6%  softirqs.CPU3.RCU
      8538 ± 54%     -52.9%       4023 ±  5%  softirqs.CPU30.RCU
      8380 ± 53%     -52.0%       4018 ±  7%  softirqs.CPU32.RCU
      8274 ± 54%     -51.2%       4040 ±  3%  softirqs.CPU33.RCU
      8401 ± 54%     -52.6%       3984 ±  3%  softirqs.CPU34.RCU
      8238 ± 56%     -53.0%       3869 ±  8%  softirqs.CPU35.RCU
      8283 ± 53%     -54.3%       3785 ±  7%  softirqs.CPU36.RCU
      8554 ± 55%     -56.1%       3753 ±  6%  softirqs.CPU37.RCU
     28156 ±  6%     -12.1%      24755 ±  2%  softirqs.CPU37.SCHED
      8274 ± 52%     -53.8%       3821 ±  3%  softirqs.CPU38.RCU
      8666 ± 52%     -55.7%       3840 ±  2%  softirqs.CPU39.RCU
      8299 ± 49%     -46.6%       4432 ±  4%  softirqs.CPU4.RCU
      8336 ± 52%     -52.4%       3969 ±  5%  softirqs.CPU40.RCU
      8276 ± 52%     -52.1%       3968 ±  4%  softirqs.CPU41.RCU
      8653 ± 47%     -56.3%       3779 ±  5%  softirqs.CPU43.RCU
      8285 ± 53%     -53.6%       3846 ±  7%  softirqs.CPU44.RCU
      8292 ± 55%     -54.4%       3785 ±  5%  softirqs.CPU45.RCU
      8182 ± 57%     -48.7%       4200 ± 20%  softirqs.CPU46.RCU
      8745 ± 54%     -54.9%       3947 ±  8%  softirqs.CPU47.RCU
      8923 ± 45%     -53.2%       4176 ±  4%  softirqs.CPU48.RCU
      8012 ± 54%     -45.4%       4373 ±  8%  softirqs.CPU49.RCU
      8494 ± 50%     -50.9%       4172 ±  3%  softirqs.CPU5.RCU
      7867 ± 50%     -45.8%       4266 ±  3%  softirqs.CPU50.RCU
      7575 ± 41%     -44.4%       4213 ±  5%  softirqs.CPU51.RCU
      8173 ± 54%     -46.1%       4403 ±  3%  softirqs.CPU52.RCU
      8800 ± 45%     -50.2%       4384 ±  2%  softirqs.CPU53.RCU
     27984 ±  5%      -7.3%      25940 ±  3%  softirqs.CPU53.SCHED
      8291 ± 53%     -50.0%       4142 ±  2%  softirqs.CPU54.RCU
      8478 ± 52%     -44.4%       4710 ± 12%  softirqs.CPU55.RCU
      8061 ± 49%     -46.2%       4335 ±  3%  softirqs.CPU56.RCU
      8146 ± 49%     -40.1%       4882 ± 17%  softirqs.CPU57.RCU
      8121 ± 48%     -46.4%       4352 ±  3%  softirqs.CPU58.RCU
      8089 ± 52%     -43.4%       4577 ± 13%  softirqs.CPU59.RCU
      8471 ± 46%     -46.8%       4508 ±  6%  softirqs.CPU6.RCU
     27998 ±  5%     -11.9%      24652 ±  4%  softirqs.CPU6.SCHED
      8131 ± 56%     -48.9%       4151 ±  5%  softirqs.CPU60.RCU
      8302 ± 46%     -47.9%       4329 ±  5%  softirqs.CPU61.RCU
      7887 ± 51%     -43.3%       4475 ± 10%  softirqs.CPU62.RCU
      7899 ± 50%     -47.6%       4137 ±  5%  softirqs.CPU63.RCU
      8257 ± 54%     -50.0%       4131 ±  5%  softirqs.CPU64.RCU
      8253 ± 51%     -49.3%       4180 ±  5%  softirqs.CPU65.RCU
      7951 ± 57%     -44.8%       4390 ±  5%  softirqs.CPU66.RCU
      8029 ± 55%     -45.4%       4380 ±  2%  softirqs.CPU67.RCU
      8282 ± 54%     -50.2%       4122 ±  5%  softirqs.CPU68.RCU
      7952 ± 55%     -48.5%       4098 ±  5%  softirqs.CPU69.RCU
      8289 ± 47%     -46.1%       4468 ±  5%  softirqs.CPU7.RCU
      8304 ± 53%     -49.3%       4207 ±  7%  softirqs.CPU70.RCU
      8223 ± 52%     -48.3%       4255 ±  4%  softirqs.CPU71.RCU
      8132 ± 54%     -54.2%       3720 ±  2%  softirqs.CPU72.RCU
      8222 ± 55%     -55.1%       3694 ±  3%  softirqs.CPU73.RCU
      8020 ± 56%     -51.1%       3921 ±  5%  softirqs.CPU74.RCU
      8425 ± 56%     -52.5%       4002 ±  3%  softirqs.CPU75.RCU
      8206 ± 58%     -55.0%       3692 ±  4%  softirqs.CPU76.RCU
      8265 ± 55%     -53.0%       3887 ±  7%  softirqs.CPU77.RCU
      8159 ± 54%     -53.0%       3833 ± 10%  softirqs.CPU78.RCU
      8106 ± 55%     -53.5%       3770 ±  3%  softirqs.CPU79.RCU
      8074 ± 51%     -45.2%       4424 ±  2%  softirqs.CPU8.RCU
      8816 ± 49%     -54.9%       3976 ±  8%  softirqs.CPU80.RCU
      8218 ± 54%     -53.5%       3821 ±  4%  softirqs.CPU81.RCU
      8270 ± 55%     -53.7%       3831 ±  8%  softirqs.CPU82.RCU
      8246 ± 58%     -50.9%       4049        softirqs.CPU83.RCU
      8248 ± 53%     -55.0%       3708 ±  7%  softirqs.CPU86.RCU
      8183 ± 54%     -55.5%       3645 ±  9%  softirqs.CPU87.RCU
      8353 ± 52%     -54.7%       3784 ±  9%  softirqs.CPU88.RCU
      8009 ± 58%     -53.1%       3759 ± 10%  softirqs.CPU89.RCU
      8257 ± 49%     -47.4%       4343 ±  8%  softirqs.CPU9.RCU
      7869 ± 54%     -54.2%       3608 ±  7%  softirqs.CPU90.RCU
      8122 ± 55%     -55.4%       3622 ±  6%  softirqs.CPU91.RCU
      8281 ± 54%     -53.7%       3831 ±  7%  softirqs.CPU92.RCU
      8381 ± 49%     -53.2%       3918 ± 14%  softirqs.CPU93.RCU
      8234 ± 56%     -53.5%       3829 ±  4%  softirqs.CPU94.RCU
      8151 ± 54%     -51.0%       3990 ± 11%  softirqs.CPU95.RCU
    799899 ± 52%     -49.6%     403266        softirqs.RCU
    128655 ±  5%     +19.5%     153698 ±  2%  softirqs.TIMER
    119151 ± 22%     +34.5%     160281 ±  5%  interrupts.CAL:Function_call_interrupts
    599.75 ± 68%    +151.2%       1506 ± 28%  interrupts.CPU0.NMI:Non-maskable_interrupts
    599.75 ± 68%    +151.2%       1506 ± 28%  interrupts.CPU0.PMI:Performance_monitoring_interrupts
     86.50 ± 35%    +111.6%     183.00 ±  7%  interrupts.CPU0.RES:Rescheduling_interrupts
    111.25 ± 48%     +74.6%     194.25 ± 20%  interrupts.CPU1.RES:Rescheduling_interrupts
    679.25 ±  9%    +105.0%       1392 ± 46%  interrupts.CPU10.CAL:Function_call_interrupts
    610.50 ± 77%    +130.2%       1405 ± 13%  interrupts.CPU10.NMI:Non-maskable_interrupts
    610.50 ± 77%    +130.2%       1405 ± 13%  interrupts.CPU10.PMI:Performance_monitoring_interrupts
     77.75 ±  9%    +153.7%     197.25 ± 20%  interrupts.CPU10.RES:Rescheduling_interrupts
     95.25 ± 32%     +78.2%     169.75 ± 26%  interrupts.CPU11.RES:Rescheduling_interrupts
    347.25 ± 47%    +288.5%       1349 ± 42%  interrupts.CPU12.NMI:Non-maskable_interrupts
    347.25 ± 47%    +288.5%       1349 ± 42%  interrupts.CPU12.PMI:Performance_monitoring_interrupts
     91.50 ± 41%    +107.9%     190.25 ± 11%  interrupts.CPU12.RES:Rescheduling_interrupts
    642.50 ± 13%    +160.6%       1674 ± 33%  interrupts.CPU13.CAL:Function_call_interrupts
    427.00 ± 98%    +249.4%       1491 ± 49%  interrupts.CPU13.NMI:Non-maskable_interrupts
    427.00 ± 98%    +249.4%       1491 ± 49%  interrupts.CPU13.PMI:Performance_monitoring_interrupts
     76.75 ± 31%    +182.7%     217.00 ± 12%  interrupts.CPU13.RES:Rescheduling_interrupts
    109.25 ± 41%     +77.1%     193.50 ± 28%  interrupts.CPU14.RES:Rescheduling_interrupts
    509.75 ± 53%    +200.9%       1533 ± 23%  interrupts.CPU15.NMI:Non-maskable_interrupts
    509.75 ± 53%    +200.9%       1533 ± 23%  interrupts.CPU15.PMI:Performance_monitoring_interrupts
    107.25 ± 30%    +103.7%     218.50 ± 16%  interrupts.CPU15.RES:Rescheduling_interrupts
    468.00 ± 73%    +234.0%       1563 ± 24%  interrupts.CPU16.NMI:Non-maskable_interrupts
    468.00 ± 73%    +234.0%       1563 ± 24%  interrupts.CPU16.PMI:Performance_monitoring_interrupts
     98.75 ± 39%    +155.7%     252.50 ± 23%  interrupts.CPU16.RES:Rescheduling_interrupts
    445.50 ± 77%    +136.2%       1052 ± 34%  interrupts.CPU18.NMI:Non-maskable_interrupts
    445.50 ± 77%    +136.2%       1052 ± 34%  interrupts.CPU18.PMI:Performance_monitoring_interrupts
    654.00 ± 42%    +140.1%       1570 ± 44%  interrupts.CPU20.NMI:Non-maskable_interrupts
    654.00 ± 42%    +140.1%       1570 ± 44%  interrupts.CPU20.PMI:Performance_monitoring_interrupts
     88.25 ± 11%    +162.6%     231.75 ± 56%  interrupts.CPU20.RES:Rescheduling_interrupts
    586.00 ± 29%    +208.2%       1806 ± 27%  interrupts.CPU21.NMI:Non-maskable_interrupts
    586.00 ± 29%    +208.2%       1806 ± 27%  interrupts.CPU21.PMI:Performance_monitoring_interrupts
    102.75 ± 36%     +71.5%     176.25 ± 11%  interrupts.CPU22.RES:Rescheduling_interrupts
     88.25 ± 11%    +141.1%     212.75 ± 25%  interrupts.CPU23.RES:Rescheduling_interrupts
     98.75 ± 12%     -70.9%      28.75 ± 63%  interrupts.CPU25.TLB:TLB_shootdowns
    537.50 ± 30%     +71.2%     920.25 ± 20%  interrupts.CPU3.NMI:Non-maskable_interrupts
    537.50 ± 30%     +71.2%     920.25 ± 20%  interrupts.CPU3.PMI:Performance_monitoring_interrupts
    118.75 ± 28%    +107.2%     246.00 ± 25%  interrupts.CPU3.RES:Rescheduling_interrupts
    676.25 ± 14%     +49.5%       1010 ±  8%  interrupts.CPU36.CAL:Function_call_interrupts
     90.25 ± 13%     +64.8%     148.75 ± 15%  interrupts.CPU36.RES:Rescheduling_interrupts
    756.75 ± 80%    +189.7%       2192 ± 20%  interrupts.CPU4.NMI:Non-maskable_interrupts
    756.75 ± 80%    +189.7%       2192 ± 20%  interrupts.CPU4.PMI:Performance_monitoring_interrupts
    129.50 ± 22%     +65.3%     214.00 ± 34%  interrupts.CPU4.RES:Rescheduling_interrupts
    177.00 ± 14%     -33.5%     117.75 ± 31%  interrupts.CPU40.RES:Rescheduling_interrupts
    142.75 ± 11%     -22.6%     110.50 ± 10%  interrupts.CPU44.RES:Rescheduling_interrupts
    144.25 ± 21%     -27.9%     104.00 ± 16%  interrupts.CPU45.RES:Rescheduling_interrupts
    125.75 ± 30%     +83.3%     230.50 ± 21%  interrupts.CPU49.RES:Rescheduling_interrupts
    418.00 ± 45%    +312.3%       1723 ± 31%  interrupts.CPU5.NMI:Non-maskable_interrupts
    418.00 ± 45%    +312.3%       1723 ± 31%  interrupts.CPU5.PMI:Performance_monitoring_interrupts
    746.50 ± 23%    +235.2%       2502 ± 88%  interrupts.CPU50.CAL:Function_call_interrupts
     92.00 ± 25%    +103.3%     187.00 ± 20%  interrupts.CPU50.RES:Rescheduling_interrupts
    349.50 ± 10%    +146.2%     860.50 ± 46%  interrupts.CPU51.NMI:Non-maskable_interrupts
    349.50 ± 10%    +146.2%     860.50 ± 46%  interrupts.CPU51.PMI:Performance_monitoring_interrupts
    104.50 ± 16%     +87.6%     196.00 ± 17%  interrupts.CPU52.RES:Rescheduling_interrupts
    749.50 ± 11%     +96.9%       1476 ± 49%  interrupts.CPU53.CAL:Function_call_interrupts
    437.50 ± 50%    +164.0%       1155 ± 71%  interrupts.CPU53.NMI:Non-maskable_interrupts
    437.50 ± 50%    +164.0%       1155 ± 71%  interrupts.CPU53.PMI:Performance_monitoring_interrupts
     77.50 ±  8%    +130.3%     178.50 ± 15%  interrupts.CPU53.RES:Rescheduling_interrupts
    777.75 ± 20%    +263.4%       2826 ± 80%  interrupts.CPU54.CAL:Function_call_interrupts
    508.50 ± 27%    +112.4%       1080 ± 19%  interrupts.CPU54.NMI:Non-maskable_interrupts
    508.50 ± 27%    +112.4%       1080 ± 19%  interrupts.CPU54.PMI:Performance_monitoring_interrupts
     99.00 ± 29%    +135.1%     232.75 ± 11%  interrupts.CPU54.RES:Rescheduling_interrupts
    545.75 ± 39%    +118.8%       1194 ± 13%  interrupts.CPU55.NMI:Non-maskable_interrupts
    545.75 ± 39%    +118.8%       1194 ± 13%  interrupts.CPU55.PMI:Performance_monitoring_interrupts
     76.25 ± 12%    +167.5%     204.00 ± 46%  interrupts.CPU55.RES:Rescheduling_interrupts
    527.25 ± 39%    +101.1%       1060 ± 23%  interrupts.CPU56.NMI:Non-maskable_interrupts
    527.25 ± 39%    +101.1%       1060 ± 23%  interrupts.CPU56.PMI:Performance_monitoring_interrupts
    101.50 ± 26%     +96.1%     199.00 ± 32%  interrupts.CPU56.RES:Rescheduling_interrupts
     33.50 ± 49%    +128.4%      76.50 ± 19%  interrupts.CPU56.TLB:TLB_shootdowns
    708.00 ± 10%    +416.7%       3658 ± 90%  interrupts.CPU57.CAL:Function_call_interrupts
    105.25 ± 17%    +175.5%     290.00 ± 60%  interrupts.CPU57.RES:Rescheduling_interrupts
    542.00 ± 33%    +159.0%       1403 ± 25%  interrupts.CPU58.NMI:Non-maskable_interrupts
    542.00 ± 33%    +159.0%       1403 ± 25%  interrupts.CPU58.PMI:Performance_monitoring_interrupts
     97.75 ± 20%    +133.5%     228.25 ± 33%  interrupts.CPU58.RES:Rescheduling_interrupts
    625.25 ± 14%    +153.0%       1582 ± 33%  interrupts.CPU59.CAL:Function_call_interrupts
     96.50 ± 38%    +131.1%     223.00 ± 16%  interrupts.CPU59.RES:Rescheduling_interrupts
    567.25 ± 41%    +137.2%       1345 ± 25%  interrupts.CPU6.NMI:Non-maskable_interrupts
    567.25 ± 41%    +137.2%       1345 ± 25%  interrupts.CPU6.PMI:Performance_monitoring_interrupts
    120.00 ± 25%     +70.0%     204.00 ± 14%  interrupts.CPU6.RES:Rescheduling_interrupts
    574.25 ±  6%    +591.6%       3971 ± 71%  interrupts.CPU60.CAL:Function_call_interrupts
    456.25 ± 56%    +216.5%       1444 ± 23%  interrupts.CPU60.NMI:Non-maskable_interrupts
    456.25 ± 56%    +216.5%       1444 ± 23%  interrupts.CPU60.PMI:Performance_monitoring_interrupts
     95.25 ± 44%    +135.7%     224.50 ± 31%  interrupts.CPU60.RES:Rescheduling_interrupts
    107.75 ± 26%    +102.6%     218.25 ± 14%  interrupts.CPU61.RES:Rescheduling_interrupts
    694.00 ±  2%    +187.1%       1992 ± 68%  interrupts.CPU62.CAL:Function_call_interrupts
    697.75 ± 61%    +169.4%       1879 ± 40%  interrupts.CPU62.NMI:Non-maskable_interrupts
    697.75 ± 61%    +169.4%       1879 ± 40%  interrupts.CPU62.PMI:Performance_monitoring_interrupts
     90.25 ± 26%    +141.8%     218.25 ± 28%  interrupts.CPU62.RES:Rescheduling_interrupts
    119.75 ± 60%    +120.9%     264.50 ± 29%  interrupts.CPU63.RES:Rescheduling_interrupts
    119.50 ± 30%     +50.6%     180.00 ± 19%  interrupts.CPU64.RES:Rescheduling_interrupts
    358.00 ± 65%    +205.8%       1094 ± 32%  interrupts.CPU65.NMI:Non-maskable_interrupts
    358.00 ± 65%    +205.8%       1094 ± 32%  interrupts.CPU65.PMI:Performance_monitoring_interrupts
    105.25 ± 35%    +296.7%     417.50 ± 92%  interrupts.CPU65.RES:Rescheduling_interrupts
     86.50 ± 17%    +291.3%     338.50 ± 99%  interrupts.CPU66.RES:Rescheduling_interrupts
    819.25 ± 28%     +61.1%       1319 ± 38%  interrupts.CPU67.CAL:Function_call_interrupts
    135.75 ± 61%     +83.2%     248.75 ± 13%  interrupts.CPU67.RES:Rescheduling_interrupts
    711.75 ± 40%    +199.4%       2130 ± 52%  interrupts.CPU68.NMI:Non-maskable_interrupts
    711.75 ± 40%    +199.4%       2130 ± 52%  interrupts.CPU68.PMI:Performance_monitoring_interrupts
    112.50 ± 40%     +88.2%     211.75 ± 10%  interrupts.CPU68.RES:Rescheduling_interrupts
    484.50 ± 12%    +320.7%       2038 ± 35%  interrupts.CPU69.NMI:Non-maskable_interrupts
    484.50 ± 12%    +320.7%       2038 ± 35%  interrupts.CPU69.PMI:Performance_monitoring_interrupts
    406.25 ± 30%    +173.6%       1111 ± 28%  interrupts.CPU7.NMI:Non-maskable_interrupts
    406.25 ± 30%    +173.6%       1111 ± 28%  interrupts.CPU7.PMI:Performance_monitoring_interrupts
     90.50 ± 40%    +129.6%     207.75 ± 34%  interrupts.CPU70.RES:Rescheduling_interrupts
     96.75 ± 58%     +83.7%     177.75 ± 13%  interrupts.CPU71.RES:Rescheduling_interrupts
     22.75 ± 37%    +886.8%     224.50 ±126%  interrupts.CPU71.TLB:TLB_shootdowns
    995.00 ± 15%    +452.8%       5500 ± 82%  interrupts.CPU73.CAL:Function_call_interrupts
    822.25 ± 10%     +44.1%       1184 ± 36%  interrupts.CPU75.CAL:Function_call_interrupts
    710.25 ± 28%     +73.0%       1228 ± 10%  interrupts.CPU8.NMI:Non-maskable_interrupts
    710.25 ± 28%     +73.0%       1228 ± 10%  interrupts.CPU8.PMI:Performance_monitoring_interrupts
     90.50 ± 37%    +136.5%     214.00 ± 12%  interrupts.CPU9.RES:Rescheduling_interrupts
    117.25 ±  7%     -55.9%      51.75 ± 68%  interrupts.CPU91.TLB:TLB_shootdowns
      2064 ± 59%     -63.7%     749.75 ±  8%  interrupts.CPU92.CAL:Function_call_interrupts
    730.25 ± 18%     +52.3%       1112 ± 16%  interrupts.CPU93.CAL:Function_call_interrupts
    616.75 ± 30%     +64.7%       1015 ± 24%  interrupts.CPU95.NMI:Non-maskable_interrupts
    616.75 ± 30%     +64.7%       1015 ± 24%  interrupts.CPU95.PMI:Performance_monitoring_interrupts
     77842 ± 15%     +48.2%     115350 ±  7%  interrupts.NMI:Non-maskable_interrupts
     77842 ± 15%     +48.2%     115350 ±  7%  interrupts.PMI:Performance_monitoring_interrupts
     11940 ±  9%     +43.5%      17137 ±  9%  interrupts.RES:Rescheduling_interrupts


                                                                                
                                 fio.write_bw_MBps                              
                                                                                
  6500 +--------------------------------------------------------------------+   
       | O  O O       O O  O    O O  O O    O    O    O O  O O  O O  O      |   
  6000 |-+       O O         O            O   O    O                   O  O |   
       |                                                                    |   
  5500 |-+                                                                  |   
       |                                                                    |   
  5000 |-+                                    +                             |   
       | +       +.+..               +    +   ::   +..+                     |   
  4500 |++:     :     +      +      : :  : : : :   :   +     +              |   
       |  :     :     :     + +     : :  : : :  : :     +    :              |   
  4000 |-+ :   :       :   +   +   :   ::   :   : :      :  :               |   
       |   :  +        :  +     +.+    +    +    :        : :               |   
  3500 |-+  :+          :+                       +        ::                |   
       |    +           +                                  +                |   
  3000 +--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                  fio.write_iops                                
                                                                                
  3200 +--------------------------------------------------------------------+   
       | O  O O    O  O O  O O  O      O  O O    O    O O  O O  O      O  O |   
  3000 |-+       O                            O    O                        |   
  2800 |-+                                                                  |   
       |                                                                    |   
  2600 |-+                                                                  |   
       |                                                                    |   
  2400 |-+       +.                           +    +..+                     |   
       |::      :  +..+      +       +    +   ::   :   +     +              |   
  2200 |:+:     :     :     + +     : :  : : : :  :     +    :              |   
  2000 |-+:    :       :   +   +    : :  : : :  : :      :  :               |   
       |   :  +        :  :     +. :   ::   +   : :      :  :               |   
  1800 |-+ : :         :  :       +    +         :        : :               |   
       |    ::          ::                       +        ::                |   
  1600 +--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                 fio.write_clat_stddev                          
                                                                                
  5.5e+08 +-----------------------------------------------------------------+   
    5e+08 |-+  +.                                 +                         |   
          |    : +         +      +.+    +        :         +               |   
  4.5e+08 |-+ :  :        + +    :   :  : :  +   : :      .. :              |   
    4e+08 |-+ :   :      +   +   :   :  : : : :  : :     +   :              |   
  3.5e+08 |-+:    :    .+     + :     ::   :: :  :  :   +     +             |   
    3e+08 |++:    :   +        +      +    +   : :  +..+                    |   
          | +      :..                         ::                           |   
  2.5e+08 |-+      +                            +                           |   
    2e+08 |-+                                                               |   
  1.5e+08 |-+                                                               |   
    1e+08 |-+                                                               |   
          |                                                                 |   
    5e+07 |-O  O O O  O O  O O O  O O O  O O O  O O O  O O  O O O  O O O  O |   
        0 +-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                               fio.time.elapsed_time                            
                                                                                
  245 +---------------------------------------------------------------------+   
      |    :            :                                  :                |   
  240 |-+  ::          ::                                 ::                |   
  235 |-+ : :          : :                                : :               |   
      |   : :          : :                                : :               |   
  230 |-+ :  :         : :                                : :               |   
  225 |-+ :  :        :  :                               :  :               |   
      |  :   :        :   :                              :  :               |   
  220 |-+:   :        :   :                              :  :               |   
  215 |..+    :       :   +..  +..          +.           :   :              |   
      |       +.+..  :        +           ..  +..+.     :    :              |   
  210 |-+          +.+       +    +.+..+.+         +..+.+    +              |   
  205 |-+                                                                   |   
      |                        O         O  O    O                          |   
  200 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                             fio.time.elapsed_time.max                          
                                                                                
  245 +---------------------------------------------------------------------+   
      |    :            :                                  :                |   
  240 |-+  ::          ::                                 ::                |   
  235 |-+ : :          : :                                : :               |   
      |   : :          : :                                : :               |   
  230 |-+ :  :         : :                                : :               |   
  225 |-+ :  :        :  :                               :  :               |   
      |  :   :        :   :                              :  :               |   
  220 |-+:   :        :   :                              :  :               |   
  215 |..+    :       :   +..  +..          +.           :   :              |   
      |       +.+..  :        +           ..  +..+.     :    :              |   
  210 |-+          +.+       +    +.+..+.+         +..+.+    +              |   
  205 |-+                                                                   |   
      |                        O         O  O    O                          |   
  200 +---------------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Oliver Sang


View attachment "config-5.9.0-rc8-00018-g8698fc4eb7884" of type "text/plain" (171288 bytes)

View attachment "job-script" of type "text/plain" (8103 bytes)

View attachment "job.yaml" of type "text/plain" (5598 bytes)

View attachment "reproduce" of type "text/plain" (901 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ