lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Date:   Sun, 4 Oct 2020 21:32:31 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     Ritesh Harjani <riteshh@...ux.ibm.com>
Cc:     Theodore Ts'o <tytso@....edu>,
        Dan Williams <dan.j.williams@...el.com>,
        Jan Kara <jack@...e.cz>, LKML <linux-kernel@...r.kernel.org>,
        linux-ext4@...r.kernel.org, lkp@...ts.01.org, lkp@...el.com,
        ying.huang@...el.com, feng.tang@...el.com, zhengjun.xing@...el.com
Subject: [ext4] 061113efe9: fio.write_iops 364.6% improvement

Greeting,

FYI, we noticed a 364.6% improvement of fio.write_iops due to commit:


commit: 061113efe99b24ac026db5aa5a72844e16318bd7 ("ext4: optimize file overwrites")
https://git.kernel.org/cgit/linux/kernel/git/tytso/ext4.git dev


in testcase: fio-basic
on test machine: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 256G memory
with following parameters:

	disk: 2pmem
	fs: ext4
	mount_option: dax
	runtime: 200s
	nr_task: 50%
	time_based: tb
	rw: randwrite
	bs: 4k
	ioengine: sync
	test_size: 200G
	cpufreq_governor: performance
	ucode: 0x5002f01

test-description: Fio is a tool that will spawn a number of threads or processes doing a particular type of I/O action as specified by the user.
test-url: https://github.com/axboe/fio





Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
bs/compiler/cpufreq_governor/disk/fs/ioengine/kconfig/mount_option/nr_task/rootfs/runtime/rw/tbox_group/test_size/testcase/time_based/ucode:
  4k/gcc-9/performance/2pmem/ext4/sync/x86_64-rhel-8.3/dax/50%/debian-10.4-x86_64-20200603.cgz/200s/randwrite/lkp-csl-2sp6/200G/fio-basic/tb/0x5002f01

commit: 
  9ffd5728cc ("ext4: remove unused including <linux/version.h>")
  061113efe9 ("ext4: optimize file overwrites")

9ffd5728cca71e4f 061113efe99b24ac026db5aa5a7 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
      0.16 ± 33%      -0.2        0.01        fio.latency_100us%
     48.50 ± 31%     -47.2        1.28 ± 43%  fio.latency_20us%
      0.01           +32.7       32.76 ± 51%  fio.latency_2us%
      0.21 ± 89%     +47.9       48.10 ± 17%  fio.latency_4us%
     25.01 ± 23%     -24.6        0.37 ± 51%  fio.latency_50us%
     15563 ±  4%     -10.4%      13952 ±  4%  fio.time.involuntary_context_switches
     13539 ±  5%     -13.9%      11659 ±  9%  fio.time.minor_page_faults
      9206           -11.5%       8144 ±  3%  fio.time.system_time
    316.40 ±  5%    +334.8%       1375 ± 18%  fio.time.user_time
     24203            -7.2%      22455        fio.time.voluntary_context_switches
  5.93e+08 ±  2%    +364.6%  2.755e+09 ± 18%  fio.workload
     11581 ±  2%    +364.6%      53803 ± 18%  fio.write_bw_MBps
     25088 ±  8%     -80.7%       4832 ± 24%  fio.write_clat_90%_us
     28480 ±  8%     -78.7%       6072 ± 23%  fio.write_clat_95%_us
     37248 ±  6%     -65.5%      12848 ± 21%  fio.write_clat_99%_us
     15694 ±  3%     -80.2%       3100 ± 22%  fio.write_clat_mean_us
      9722 ± 15%     -72.8%       2648 ± 18%  fio.write_clat_stddev
   2964935 ±  2%    +364.6%   13773690 ± 18%  fio.write_iops
      1858            +4.2%       1936        vmstat.system.cs
     47.96           -11.3%      42.52 ±  3%  iostat.cpu.system
      1.69 ±  5%    +321.6%       7.11 ± 18%  iostat.cpu.user
 1.918e+09 ± 36%    +226.2%  6.257e+09 ± 41%  cpuidle.C1E.time
  10452864 ±  3%     +42.6%   14909339 ± 25%  cpuidle.C1E.usage
 7.753e+09 ±  8%     -55.6%   3.44e+09 ± 74%  cpuidle.C6.time
     39387 ±  2%     +15.2%      45385        meminfo.Active(anon)
      7410 ± 10%     +84.6%      13677 ±  4%  meminfo.Dirty
    122418            -9.6%     110640        meminfo.KReclaimable
    122418            -9.6%     110640        meminfo.SReclaimable
      0.01 ± 16%      -0.0        0.00 ± 37%  mpstat.cpu.all.iowait%
      0.03 ±  3%      +0.0        0.03 ±  4%  mpstat.cpu.all.soft%
     47.59            -5.5       42.07 ±  3%  mpstat.cpu.all.sys%
      1.70 ±  5%      +5.5        7.17 ± 18%  mpstat.cpu.all.usr%
    741453 ±  7%     -58.1%     310382 ± 30%  numa-numastat.node0.local_node
    756956 ±  6%     -57.6%     321292 ± 33%  numa-numastat.node0.numa_hit
    247432 ± 24%    +153.0%     626065 ± 15%  numa-numastat.node1.local_node
    263275 ± 19%    +145.6%     646527 ± 17%  numa-numastat.node1.numa_hit
      3830 ±  5%      -7.1%       3558 ±  2%  slabinfo.dmaengine-unmap-16.active_objs
      3830 ±  5%      -7.1%       3558 ±  2%  slabinfo.dmaengine-unmap-16.num_objs
    421181 ±  5%     -32.1%     286015 ±  7%  slabinfo.ext4_extent_status.active_objs
      6662 ±  6%     -42.1%       3855 ±  8%  slabinfo.ext4_extent_status.active_slabs
    679573 ±  6%     -42.1%     393340 ±  8%  slabinfo.ext4_extent_status.num_objs
      6662 ±  6%     -42.1%       3855 ±  8%  slabinfo.ext4_extent_status.num_slabs
      9849 ±  2%     +14.9%      11317        proc-vmstat.nr_active_anon
      1812 ± 10%     +87.2%       3392 ±  3%  proc-vmstat.nr_dirty
      2123            +1.3%       2151        proc-vmstat.nr_page_table_pages
     30595            -9.6%      27662        proc-vmstat.nr_slab_reclaimable
      9849 ±  2%     +14.9%      11317        proc-vmstat.nr_zone_active_anon
      1813 ± 10%     +87.1%       3391 ±  3%  proc-vmstat.nr_zone_write_pending
   1064225            -5.0%    1011161        proc-vmstat.numa_hit
   1032828            -5.1%     979762        proc-vmstat.numa_local
   1104273            -5.0%    1049441        proc-vmstat.pgalloc_normal
    898139 ± 14%     -18.3%     733903        proc-vmstat.pgfree
      6689 ±  9%     +78.3%      11924 ± 15%  numa-meminfo.node0.Dirty
   1787289           -62.9%     662714 ±  3%  numa-meminfo.node0.FilePages
   1308617 ±  2%     -88.8%     146900 ± 15%  numa-meminfo.node0.Inactive
   1308337 ±  2%     -88.8%     146757 ± 15%  numa-meminfo.node0.Inactive(anon)
   1131509           -98.6%      16081 ± 14%  numa-meminfo.node0.Mapped
   4350653           -28.2%    3121909 ±  3%  numa-meminfo.node0.MemUsed
      7561 ±  3%     -37.8%       4704 ± 12%  numa-meminfo.node0.PageTables
   1120696           -99.6%       4832 ± 79%  numa-meminfo.node0.Shmem
     37743           +15.4%      43552        numa-meminfo.node1.Active(anon)
    539135          +209.5%    1668823        numa-meminfo.node1.FilePages
    119869 ± 30%    +969.4%    1281909        numa-meminfo.node1.Inactive
    119766 ± 30%    +970.2%    1281701        numa-meminfo.node1.Inactive(anon)
     17210 ± 16%   +6484.3%    1133150        numa-meminfo.node1.Mapped
   2835108 ±  2%     +43.3%    4061792 ±  2%  numa-meminfo.node1.MemUsed
    941.50 ± 19%    +314.5%       3902 ± 14%  numa-meminfo.node1.PageTables
     44833 ± 10%   +2502.5%    1166807        numa-meminfo.node1.Shmem
      1601 ±  7%     +85.5%       2970 ± 15%  numa-vmstat.node0.nr_dirty
    446809           -62.9%     165679 ±  3%  numa-vmstat.node0.nr_file_pages
    327071 ±  2%     -88.8%      36681 ± 15%  numa-vmstat.node0.nr_inactive_anon
    282962           -98.5%       4157 ± 14%  numa-vmstat.node0.nr_mapped
      1890 ±  3%     -37.9%       1174 ± 12%  numa-vmstat.node0.nr_page_table_pages
    280164           -99.6%       1208 ± 79%  numa-vmstat.node0.nr_shmem
    327071 ±  2%     -88.8%      36681 ± 15%  numa-vmstat.node0.nr_zone_inactive_anon
      1601 ±  7%     +85.5%       2970 ± 15%  numa-vmstat.node0.nr_zone_write_pending
   1917342           -22.9%    1477585 ±  7%  numa-vmstat.node0.numa_hit
   1869725 ±  3%     -21.9%    1461109 ±  7%  numa-vmstat.node0.numa_local
      9474           +15.3%      10920        numa-vmstat.node1.nr_active_anon
    134747          +208.6%     415874        numa-vmstat.node1.nr_file_pages
     29850 ± 30%    +968.9%     319054        numa-vmstat.node1.nr_inactive_anon
      4207 ± 16%   +6601.2%     281918        numa-vmstat.node1.nr_mapped
    235.25 ± 18%    +312.8%     971.00 ± 14%  numa-vmstat.node1.nr_page_table_pages
     11172 ± 10%   +2499.0%     290369        numa-vmstat.node1.nr_shmem
      9474           +15.3%      10920        numa-vmstat.node1.nr_zone_active_anon
     29850 ± 30%    +968.9%     319054        numa-vmstat.node1.nr_zone_inactive_anon
    687971 ±  3%     +56.8%    1078550 ± 10%  numa-vmstat.node1.numa_hit
    557691 ± 11%     +64.5%     917161 ± 12%  numa-vmstat.node1.numa_local
     28318 ± 22%     +31.8%      37317 ± 16%  sched_debug.cfs_rq:/.exec_clock.stddev
    385.22 ±  8%      +9.7%     422.63 ±  9%  sched_debug.cfs_rq:/.load_avg.avg
     59833 ± 14%     +24.5%      74463 ± 10%  sched_debug.cfs_rq:/.min_vruntime.avg
    103784 ± 12%     +19.3%     123853 ± 11%  sched_debug.cfs_rq:/.min_vruntime.max
     29685 ± 20%     +28.7%      38216 ± 16%  sched_debug.cfs_rq:/.min_vruntime.stddev
     29685 ± 20%     +28.8%      38221 ± 16%  sched_debug.cfs_rq:/.spread0.stddev
    673766 ±  5%     +28.1%     862888 ±  2%  sched_debug.cpu.avg_idle.avg
    316122 ±  5%     -41.5%     185066 ±  8%  sched_debug.cpu.avg_idle.stddev
      2928 ±  8%     +10.9%       3248 ±  7%  sched_debug.cpu.nr_switches.avg
     18266 ± 24%     +53.1%      27974 ± 16%  sched_debug.cpu.nr_switches.max
      2854 ± 20%     +37.8%       3934 ± 15%  sched_debug.cpu.nr_switches.stddev
     28.58 ± 24%     +59.5%      45.58 ±  9%  sched_debug.cpu.nr_uninterruptible.max
     14485 ± 33%     +69.6%      24568 ± 21%  sched_debug.cpu.sched_count.max
      2206 ± 27%     +57.9%       3483 ± 19%  sched_debug.cpu.sched_count.stddev
    605.82 ± 19%     +25.4%     759.56 ± 15%  sched_debug.cpu.sched_goidle.avg
      7179 ± 33%     +70.3%      12229 ± 21%  sched_debug.cpu.sched_goidle.max
     34.46 ± 10%     -42.4%      19.83 ± 10%  sched_debug.cpu.sched_goidle.min
      1117 ± 27%     +57.4%       1759 ± 19%  sched_debug.cpu.sched_goidle.stddev
      7717 ± 27%     +53.7%      11861 ± 20%  sched_debug.cpu.ttwu_count.max
      1125 ± 25%     +49.2%       1679 ± 18%  sched_debug.cpu.ttwu_count.stddev
      5318 ± 36%     +75.8%       9350 ± 23%  sched_debug.cpu.ttwu_local.max
     83.33           +33.3%     111.06 ± 14%  sched_debug.cpu.ttwu_local.min
    750.49 ± 30%     +61.3%       1210 ± 21%  sched_debug.cpu.ttwu_local.stddev
     14.37           -79.1%       3.01 ± 55%  perf-stat.i.MPKI
 4.045e+09 ±  2%    +276.2%  1.522e+10 ± 17%  perf-stat.i.branch-instructions
      1.01            -0.1        0.93 ±  2%  perf-stat.i.branch-miss-rate%
  44395538          +205.6%  1.357e+08 ± 14%  perf-stat.i.branch-misses
     35.66 ±  4%      -8.0       27.62 ± 24%  perf-stat.i.cache-miss-rate%
      1814            +4.4%       1894        perf-stat.i.context-switches
      6.02 ±  2%     -73.5%       1.59 ± 19%  perf-stat.i.cpi
      0.00 ± 27%      -0.0        0.00 ± 37%  perf-stat.i.dTLB-load-miss-rate%
 7.109e+09 ±  2%    +276.9%  2.679e+10 ± 17%  perf-stat.i.dTLB-loads
    134633 ± 23%    +202.7%     407497 ± 13%  perf-stat.i.dTLB-store-misses
 4.516e+09 ±  2%    +291.9%   1.77e+10 ± 17%  perf-stat.i.dTLB-stores
     88.04            +6.5       94.54        perf-stat.i.iTLB-load-miss-rate%
  36917229 ± 11%    +180.6%  1.036e+08 ± 13%  perf-stat.i.iTLB-load-misses
   4880282           +16.3%    5678123        perf-stat.i.iTLB-loads
 2.308e+10 ±  2%    +279.2%  8.752e+10 ± 17%  perf-stat.i.instructions
    635.06 ±  8%     +34.3%     852.98 ±  7%  perf-stat.i.instructions-per-iTLB-miss
      0.17 ±  2%    +276.4%       0.65 ± 17%  perf-stat.i.ipc
      0.39 ± 61%     -66.9%       0.13 ±  5%  perf-stat.i.metric.K/sec
    166.97 ±  2%    +274.2%     624.73 ± 17%  perf-stat.i.metric.M/sec
     84.49            +3.4       87.88 ±  2%  perf-stat.i.node-load-miss-rate%
  10345827 ±  2%     -61.8%    3956301 ± 22%  perf-stat.i.node-store-misses
     13.96           -78.8%       2.96 ± 56%  perf-stat.overall.MPKI
      1.10            -0.2        0.90 ±  3%  perf-stat.overall.branch-miss-rate%
     35.81 ±  4%      -7.7       28.08 ± 24%  perf-stat.overall.cache-miss-rate%
      5.84 ±  2%     -72.9%       1.58 ± 19%  perf-stat.overall.cpi
      0.00 ± 36%      -0.0        0.00 ± 49%  perf-stat.overall.dTLB-load-miss-rate%
      0.00 ± 24%      -0.0        0.00 ±  8%  perf-stat.overall.dTLB-store-miss-rate%
     88.21            +6.5       94.73        perf-stat.overall.iTLB-load-miss-rate%
    631.78 ±  8%     +33.0%     840.08 ±  7%  perf-stat.overall.instructions-per-iTLB-miss
      0.17 ±  2%    +281.6%       0.65 ± 17%  perf-stat.overall.ipc
      7818           -18.3%       6384        perf-stat.overall.path-length
 4.025e+09 ±  2%    +276.0%  1.514e+10 ± 17%  perf-stat.ps.branch-instructions
  44199217          +205.6%  1.351e+08 ± 14%  perf-stat.ps.branch-misses
      1805            +4.4%       1885        perf-stat.ps.context-switches
 7.075e+09 ±  2%    +276.8%  2.665e+10 ± 17%  perf-stat.ps.dTLB-loads
    134275 ± 23%    +202.0%     405555 ± 13%  perf-stat.ps.dTLB-store-misses
 4.494e+09 ±  2%    +291.8%   1.76e+10 ± 17%  perf-stat.ps.dTLB-stores
  36727460 ± 11%    +180.5%   1.03e+08 ± 13%  perf-stat.ps.iTLB-load-misses
   4854565           +16.3%    5647668        perf-stat.ps.iTLB-loads
 2.297e+10 ±  2%    +279.1%  8.706e+10 ± 17%  perf-stat.ps.instructions
  10291805 ±  2%     -61.7%    3937017 ± 22%  perf-stat.ps.node-store-misses
 4.636e+12 ±  2%    +279.0%  1.757e+13 ± 17%  perf-stat.total.instructions
      3980 ±  8%    +122.3%       8848 ± 35%  softirqs.CPU1.RCU
      3187 ± 10%    +159.2%       8263 ± 34%  softirqs.CPU12.RCU
      3430 ± 16%    +142.1%       8305 ± 35%  softirqs.CPU13.RCU
      3187 ±  3%    +155.3%       8137 ± 26%  softirqs.CPU14.RCU
      3655 ±  7%    +153.6%       9271 ± 21%  softirqs.CPU2.RCU
      3302 ±  4%    +139.9%       7923 ± 24%  softirqs.CPU23.RCU
      3629 ± 12%    +147.4%       8979 ± 12%  softirqs.CPU24.RCU
      5417 ± 25%     +72.7%       9355 ± 15%  softirqs.CPU25.RCU
      3915 ± 19%    +141.6%       9459 ± 17%  softirqs.CPU26.RCU
      4003 ± 13%    +113.5%       8549 ± 21%  softirqs.CPU27.RCU
      3553 ± 14%    +159.2%       9210 ± 19%  softirqs.CPU28.RCU
      3413 ± 10%    +174.0%       9352 ± 16%  softirqs.CPU29.RCU
      3784 ±  8%    +124.7%       8505 ± 34%  softirqs.CPU3.RCU
      3802 ±  7%    +146.5%       9372 ± 16%  softirqs.CPU30.RCU
      3767 ±  8%    +146.8%       9295 ± 16%  softirqs.CPU31.RCU
      3939 ±  9%    +130.9%       9097 ± 20%  softirqs.CPU32.RCU
      3935 ± 10%    +125.3%       8867 ± 24%  softirqs.CPU33.RCU
      3991          +131.0%       9218 ± 19%  softirqs.CPU34.RCU
      3792 ± 16%    +132.0%       8798 ± 21%  softirqs.CPU35.RCU
      3089 ±  8%    +235.6%      10367 ± 19%  softirqs.CPU36.RCU
     14423 ± 24%     -77.4%       3261 ± 10%  softirqs.CPU36.SCHED
      3965 ± 15%    +134.5%       9298 ± 19%  softirqs.CPU37.RCU
      3548 ± 15%    +195.0%      10468 ± 21%  softirqs.CPU38.RCU
      3596 ± 14%    +198.2%      10723 ± 21%  softirqs.CPU39.RCU
      3815 ±  6%    +169.3%      10275 ± 19%  softirqs.CPU40.RCU
      3468 ±  6%    +170.9%       9396 ± 27%  softirqs.CPU41.RCU
      3712 ± 12%    +179.7%      10381 ± 18%  softirqs.CPU42.RCU
      3573 ± 10%    +166.6%       9527 ± 21%  softirqs.CPU43.RCU
      3866          +164.1%      10210 ± 18%  softirqs.CPU44.RCU
      3902 ±  2%    +142.7%       9470 ± 26%  softirqs.CPU45.RCU
      3529 ± 16%    +157.6%       9091 ± 25%  softirqs.CPU46.RCU
      3669 ± 13%    +162.9%       9647 ± 20%  softirqs.CPU47.RCU
      3060 ± 17%    +211.6%       9538 ± 16%  softirqs.CPU48.RCU
      3776 ± 14%    +160.0%       9815 ± 18%  softirqs.CPU49.RCU
      3525 ± 13%    +143.2%       8574 ± 38%  softirqs.CPU5.RCU
      4257 ±  3%    +135.3%      10014 ± 34%  softirqs.CPU50.RCU
      4323 ±  4%    +124.0%       9684 ± 16%  softirqs.CPU51.RCU
      4218 ± 16%    +156.8%      10830 ± 18%  softirqs.CPU52.RCU
      4049 ± 11%    +148.7%      10072 ± 17%  softirqs.CPU53.RCU
      4376 ± 20%    +153.4%      11088 ± 16%  softirqs.CPU54.RCU
      8728 ±116%     -69.9%       2625 ±  5%  softirqs.CPU54.SCHED
      4460          +122.1%       9903 ± 19%  softirqs.CPU55.RCU
      4369 ±  7%    +124.8%       9823 ± 17%  softirqs.CPU56.RCU
      5078 ± 45%    +112.4%      10788 ± 18%  softirqs.CPU58.RCU
      4510 ± 15%    +143.1%      10964 ± 18%  softirqs.CPU59.RCU
      4504 ±  8%    +118.8%       9854 ± 17%  softirqs.CPU60.RCU
      4252 ±  4%    +121.7%       9428 ± 17%  softirqs.CPU61.RCU
      4241 ±  2%    +152.2%      10695 ± 18%  softirqs.CPU62.RCU
      4388 ±  6%    +132.2%      10189 ± 25%  softirqs.CPU63.RCU
      4937 ± 13%    +122.2%      10971 ± 20%  softirqs.CPU64.RCU
      4411 ±  2%    +155.2%      11259 ± 20%  softirqs.CPU65.RCU
      4435 ± 10%    +148.9%      11038 ± 20%  softirqs.CPU66.RCU
      4047 ± 17%    +179.7%      11320 ± 20%  softirqs.CPU67.RCU
      4238          +166.5%      11294 ± 23%  softirqs.CPU68.RCU
      4241 ±  4%    +170.5%      11473 ± 21%  softirqs.CPU69.RCU
      3567 ±  8%    +132.1%       8278 ± 37%  softirqs.CPU7.RCU
      4301 ±  2%    +156.8%      11043 ± 19%  softirqs.CPU70.RCU
      4360 ±  2%    +132.7%      10146 ± 30%  softirqs.CPU71.RCU
      3460 ± 11%    +149.4%       8631 ± 34%  softirqs.CPU72.RCU
      3248 ± 11%    +143.1%       7897 ± 22%  softirqs.CPU73.RCU
      3484 ± 17%    +140.0%       8362 ± 36%  softirqs.CPU74.RCU
      3223 ± 11%    +167.8%       8634 ± 32%  softirqs.CPU75.RCU
      3120 ± 16%    +130.4%       7189 ± 33%  softirqs.CPU76.RCU
      3493 ±  5%    +135.0%       8210 ± 40%  softirqs.CPU77.RCU
      2908 ± 12%    +165.5%       7720 ± 42%  softirqs.CPU78.RCU
      3348 ± 17%    +135.8%       7897 ± 42%  softirqs.CPU79.RCU
      3380 ±  4%    +146.4%       8329 ± 35%  softirqs.CPU8.RCU
      3136 ± 12%    +131.3%       7255 ± 37%  softirqs.CPU80.RCU
      2708 ±  7%    +164.8%       7172 ± 45%  softirqs.CPU81.RCU
      2888 ±  8%    +163.8%       7619 ± 34%  softirqs.CPU82.RCU
      2795 ± 16%    +171.9%       7599 ± 33%  softirqs.CPU83.RCU
     10700 ± 27%    +144.6%      26171 ±  3%  softirqs.CPU84.SCHED
      2686 ±  8%    +177.9%       7465 ± 37%  softirqs.CPU85.RCU
      2640 ± 10%    +174.3%       7242 ± 35%  softirqs.CPU88.RCU
      2957 ± 15%    +142.1%       7160 ± 36%  softirqs.CPU91.RCU
    354901          +137.8%     843880 ± 17%  softirqs.RCU
     34287 ± 53%     +55.0%      53147        interrupts.CAL:Function_call_interrupts
     84.00 ± 92%     -90.2%       8.25 ± 68%  interrupts.CPU10.RES:Rescheduling_interrupts
     83.25 ± 72%     -88.9%       9.25 ± 31%  interrupts.CPU11.RES:Rescheduling_interrupts
     64.50 ± 37%     -56.6%      28.00 ± 33%  interrupts.CPU14.TLB:TLB_shootdowns
     69.00 ± 33%     -79.7%      14.00 ± 44%  interrupts.CPU18.TLB:TLB_shootdowns
    262.00 ± 73%    +105.9%     539.50 ± 16%  interrupts.CPU24.CAL:Function_call_interrupts
    210.75 ± 70%    +127.9%     480.25 ±  3%  interrupts.CPU25.CAL:Function_call_interrupts
     12.75 ±133%    +639.2%      94.25 ± 31%  interrupts.CPU25.TLB:TLB_shootdowns
     13.50 ± 69%    +525.9%      84.50 ± 29%  interrupts.CPU27.TLB:TLB_shootdowns
     28.00 ± 53%    +223.2%      90.50 ± 19%  interrupts.CPU28.TLB:TLB_shootdowns
     16.00 ± 62%    +393.8%      79.00 ± 27%  interrupts.CPU29.TLB:TLB_shootdowns
    340.00 ± 70%    +217.4%       1079 ± 39%  interrupts.CPU3.CAL:Function_call_interrupts
    242.25 ± 79%     +96.4%     475.75 ±  4%  interrupts.CPU30.CAL:Function_call_interrupts
     12.75 ± 80%    +609.8%      90.50 ± 38%  interrupts.CPU30.TLB:TLB_shootdowns
    251.25 ± 75%     +95.9%     492.25 ±  7%  interrupts.CPU31.CAL:Function_call_interrupts
     23.75 ± 28%    +263.2%      86.25 ± 33%  interrupts.CPU31.TLB:TLB_shootdowns
    174.50 ± 22%     +62.2%     283.00 ± 47%  interrupts.CPU32.RES:Rescheduling_interrupts
     19.75 ± 63%    +432.9%     105.25 ± 28%  interrupts.CPU32.TLB:TLB_shootdowns
    246.75 ± 77%    +130.7%     569.25 ± 29%  interrupts.CPU33.CAL:Function_call_interrupts
      8.75 ± 79%    +900.0%      87.50 ± 32%  interrupts.CPU33.TLB:TLB_shootdowns
    244.75 ± 79%     +99.6%     488.50 ±  3%  interrupts.CPU35.CAL:Function_call_interrupts
      2580 ± 18%    +166.2%       6870 ± 24%  interrupts.CPU36.NMI:Non-maskable_interrupts
      2580 ± 18%    +166.2%       6870 ± 24%  interrupts.CPU36.PMI:Performance_monitoring_interrupts
     84.50 ± 53%    +133.7%     197.50        interrupts.CPU36.RES:Rescheduling_interrupts
     22.25 ± 65%   +1022.5%     249.75 ±103%  interrupts.CPU36.TLB:TLB_shootdowns
     17.50 ± 66%    +578.6%     118.75 ± 33%  interrupts.CPU38.TLB:TLB_shootdowns
    241.25 ± 80%    +101.2%     485.50 ±  3%  interrupts.CPU39.CAL:Function_call_interrupts
     21.00 ± 58%    +385.7%     102.00 ± 28%  interrupts.CPU39.TLB:TLB_shootdowns
     19.75 ± 35%    +370.9%      93.00 ± 32%  interrupts.CPU40.TLB:TLB_shootdowns
     27.50 ± 48%    +278.2%     104.00 ±  9%  interrupts.CPU42.TLB:TLB_shootdowns
     23.00 ± 32%    +241.3%      78.50 ± 24%  interrupts.CPU43.TLB:TLB_shootdowns
    246.00 ± 79%     +94.9%     479.50        interrupts.CPU44.CAL:Function_call_interrupts
     21.75 ± 30%    +388.5%     106.25 ± 21%  interrupts.CPU44.TLB:TLB_shootdowns
     24.75 ± 24%    +168.7%      66.50 ± 51%  interrupts.CPU45.TLB:TLB_shootdowns
     29.00 ± 33%    +200.0%      87.00 ± 39%  interrupts.CPU46.TLB:TLB_shootdowns
     22.00 ± 34%    +352.3%      99.50 ± 24%  interrupts.CPU47.TLB:TLB_shootdowns
     71.50 ± 25%     -62.9%      26.50 ± 61%  interrupts.CPU48.TLB:TLB_shootdowns
     72.00 ± 28%     -70.5%      21.25 ± 60%  interrupts.CPU54.TLB:TLB_shootdowns
     70.50 ± 24%     -59.6%      28.50 ± 43%  interrupts.CPU62.TLB:TLB_shootdowns
    255.75 ± 78%     +89.6%     485.00        interrupts.CPU72.CAL:Function_call_interrupts
     23.75 ± 24%    +307.4%      96.75 ± 13%  interrupts.CPU72.TLB:TLB_shootdowns
     19.25 ± 61%    +366.2%      89.75 ± 12%  interrupts.CPU73.TLB:TLB_shootdowns
    246.75 ± 79%    +105.6%     507.25 ±  8%  interrupts.CPU74.CAL:Function_call_interrupts
     39.75 ± 73%    +190.6%     115.50 ± 30%  interrupts.CPU74.TLB:TLB_shootdowns
    251.25 ± 79%    +193.7%     738.00 ± 53%  interrupts.CPU75.CAL:Function_call_interrupts
     27.00 ± 33%    +408.3%     137.25 ± 55%  interrupts.CPU75.TLB:TLB_shootdowns
    249.00 ± 78%    +243.1%     854.25 ± 42%  interrupts.CPU76.CAL:Function_call_interrupts
     28.00 ± 24%    +248.2%      97.50 ± 21%  interrupts.CPU76.TLB:TLB_shootdowns
      5753 ± 36%     -28.5%       4112 ± 52%  interrupts.CPU77.NMI:Non-maskable_interrupts
      5753 ± 36%     -28.5%       4112 ± 52%  interrupts.CPU77.PMI:Performance_monitoring_interrupts
     26.00 ± 27%    +253.8%      92.00 ± 18%  interrupts.CPU77.TLB:TLB_shootdowns
    248.25 ± 77%     +91.9%     476.50        interrupts.CPU78.CAL:Function_call_interrupts
     21.75 ± 15%    +302.3%      87.50 ± 27%  interrupts.CPU78.TLB:TLB_shootdowns
     22.25 ± 61%    +370.8%     104.75 ± 18%  interrupts.CPU79.TLB:TLB_shootdowns
     30.50 ± 33%    +235.2%     102.25 ±  8%  interrupts.CPU80.TLB:TLB_shootdowns
    251.25 ± 78%    +200.6%     755.25 ± 60%  interrupts.CPU81.CAL:Function_call_interrupts
     30.25 ± 54%    +237.2%     102.00 ± 24%  interrupts.CPU81.TLB:TLB_shootdowns
     32.25 ± 23%    +186.8%      92.50 ± 13%  interrupts.CPU82.TLB:TLB_shootdowns
     26.00 ± 16%    +243.3%      89.25 ± 23%  interrupts.CPU83.TLB:TLB_shootdowns
      7732 ±  2%     -67.6%       2505 ± 24%  interrupts.CPU84.NMI:Non-maskable_interrupts
      7732 ±  2%     -67.6%       2505 ± 24%  interrupts.CPU84.PMI:Performance_monitoring_interrupts
     20.50 ± 39%    +393.9%     101.25 ± 16%  interrupts.CPU84.TLB:TLB_shootdowns
     32.00 ± 31%    +176.6%      88.50 ± 22%  interrupts.CPU85.TLB:TLB_shootdowns
    253.00 ± 78%    +105.9%     521.00 ±  7%  interrupts.CPU86.CAL:Function_call_interrupts
     82.75 ± 82%     -81.0%      15.75 ± 89%  interrupts.CPU86.RES:Rescheduling_interrupts
     26.75 ± 31%    +238.3%      90.50 ± 28%  interrupts.CPU86.TLB:TLB_shootdowns
    270.75 ± 80%    +389.6%       1325 ± 62%  interrupts.CPU87.CAL:Function_call_interrupts
     29.75 ± 33%    +236.1%     100.00 ± 31%  interrupts.CPU87.TLB:TLB_shootdowns
     28.50 ± 16%    +249.1%      99.50 ± 13%  interrupts.CPU88.TLB:TLB_shootdowns
     28.00 ± 55%    +225.9%      91.25 ± 23%  interrupts.CPU89.TLB:TLB_shootdowns
    411.00 ± 81%     -86.0%      57.50 ±148%  interrupts.CPU9.RES:Rescheduling_interrupts
    257.00 ± 77%    +125.5%     579.50 ± 22%  interrupts.CPU90.CAL:Function_call_interrupts
     34.00 ± 29%    +197.8%     101.25 ± 14%  interrupts.CPU90.TLB:TLB_shootdowns
    256.00 ± 76%    +218.2%     814.50 ± 40%  interrupts.CPU91.CAL:Function_call_interrupts
     30.75 ± 28%    +216.3%      97.25 ± 12%  interrupts.CPU91.TLB:TLB_shootdowns
    252.00 ± 77%    +197.7%     750.25 ± 34%  interrupts.CPU92.CAL:Function_call_interrupts
     30.50 ± 27%    +245.1%     105.25 ±  9%  interrupts.CPU92.TLB:TLB_shootdowns
     10.50 ± 51%    +690.5%      83.00 ± 87%  interrupts.CPU93.RES:Rescheduling_interrupts
     33.00 ± 24%    +215.9%     104.25 ± 16%  interrupts.CPU93.TLB:TLB_shootdowns
     96.75 ±  7%     +35.1%     130.75        interrupts.IWI:IRQ_work_interrupts
      4367 ± 18%     +69.1%       7387 ± 31%  interrupts.TLB:TLB_shootdowns
     41.48 ±  2%     -41.5        0.00        perf-profile.calltrace.cycles-pp.__ext4_journal_start_sb.ext4_iomap_begin.iomap_apply.dax_iomap_rw.ext4_file_write_iter
     39.78           -39.8        0.00        perf-profile.calltrace.cycles-pp.jbd2__journal_start.__ext4_journal_start_sb.ext4_iomap_begin.iomap_apply.dax_iomap_rw
     39.54           -39.5        0.00        perf-profile.calltrace.cycles-pp.start_this_handle.jbd2__journal_start.__ext4_journal_start_sb.ext4_iomap_begin.iomap_apply
     65.49 ±  2%     -34.1       31.39 ±  9%  perf-profile.calltrace.cycles-pp.ext4_iomap_begin.iomap_apply.dax_iomap_rw.ext4_file_write_iter.new_sync_write
     69.67           -11.6       58.02 ±  7%  perf-profile.calltrace.cycles-pp.iomap_apply.dax_iomap_rw.ext4_file_write_iter.new_sync_write.vfs_write
     69.69           -11.5       58.19 ±  7%  perf-profile.calltrace.cycles-pp.dax_iomap_rw.ext4_file_write_iter.new_sync_write.vfs_write.ksys_write
      9.33            -9.3        0.00        perf-profile.calltrace.cycles-pp.add_transaction_credits.start_this_handle.jbd2__journal_start.__ext4_journal_start_sb.ext4_iomap_begin
      8.94 ±  7%      -8.9        0.00        perf-profile.calltrace.cycles-pp.__ext4_journal_stop.ext4_iomap_begin.iomap_apply.dax_iomap_rw.ext4_file_write_iter
      8.64 ±  7%      -8.6        0.00        perf-profile.calltrace.cycles-pp.jbd2_journal_stop.__ext4_journal_stop.ext4_iomap_begin.iomap_apply.dax_iomap_rw
     71.18            -8.3       62.88 ±  3%  perf-profile.calltrace.cycles-pp.ext4_file_write_iter.new_sync_write.vfs_write.ksys_write.do_syscall_64
     71.23            -7.5       63.72        perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      6.62 ±  3%      -6.6        0.00        perf-profile.calltrace.cycles-pp._raw_read_lock.start_this_handle.jbd2__journal_start.__ext4_journal_start_sb.ext4_iomap_begin
      6.41 ±  5%      -6.4        0.00        perf-profile.calltrace.cycles-pp.stop_this_handle.jbd2_journal_stop.__ext4_journal_stop.ext4_iomap_begin.iomap_apply
      1.06 ±  5%      +1.0        2.03 ± 55%  perf-profile.calltrace.cycles-pp.file_update_time.ext4_write_checks.ext4_file_write_iter.new_sync_write.vfs_write
      0.00            +1.0        1.04 ± 52%  perf-profile.calltrace.cycles-pp.ext4_inode_block_valid.__check_block_validity.ext4_map_blocks.ext4_iomap_begin.iomap_apply
      0.00            +1.4        1.45 ± 53%  perf-profile.calltrace.cycles-pp.ksys_lseek.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +1.5        1.48 ±  9%  perf-profile.calltrace.cycles-pp.ext4_es_lookup_extent.ext4_map_blocks.ext4_iomap_begin.iomap_apply.dax_iomap_rw
      0.00            +1.6        1.64 ± 28%  perf-profile.calltrace.cycles-pp.__check_block_validity.ext4_map_blocks.ext4_iomap_begin.iomap_apply.dax_iomap_rw
      1.13 ±  6%      +1.7        2.86 ± 67%  perf-profile.calltrace.cycles-pp.ext4_write_checks.ext4_file_write_iter.new_sync_write.vfs_write.ksys_write
     26.94 ±  2%      +2.0       28.91 ±  2%  perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     26.94 ±  2%      +2.0       28.91 ±  2%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
     26.94 ±  2%      +2.0       28.91 ±  2%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
     26.94 ±  2%      +2.0       28.91 ±  2%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     26.94 ±  2%      +2.0       28.91 ±  2%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
     27.09 ±  3%      +2.3       29.41 ±  3%  perf-profile.calltrace.cycles-pp.secondary_startup_64
     27.08 ±  3%      +2.3       29.41 ±  3%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      0.00            +3.3        3.33 ± 11%  perf-profile.calltrace.cycles-pp.ext4_map_blocks.ext4_iomap_begin.iomap_apply.dax_iomap_rw.ext4_file_write_iter
      1.64 ±  8%     +10.0       11.63 ± 25%  perf-profile.calltrace.cycles-pp.__srcu_read_unlock.dax_iomap_actor.iomap_apply.dax_iomap_rw.ext4_file_write_iter
      2.18 ± 15%     +10.9       13.09 ± 31%  perf-profile.calltrace.cycles-pp.__copy_user_nocache.__copy_user_flushcache._copy_from_iter_flushcache.dax_iomap_actor.iomap_apply
      2.19 ± 15%     +11.0       13.16 ± 31%  perf-profile.calltrace.cycles-pp.__copy_user_flushcache._copy_from_iter_flushcache.dax_iomap_actor.iomap_apply.dax_iomap_rw
      2.23 ± 14%     +11.2       13.41 ± 31%  perf-profile.calltrace.cycles-pp._copy_from_iter_flushcache.dax_iomap_actor.iomap_apply.dax_iomap_rw.ext4_file_write_iter
     14.12           +11.6       25.71 ± 10%  perf-profile.calltrace.cycles-pp.jbd2_transaction_committed.ext4_set_iomap.ext4_iomap_begin.iomap_apply.dax_iomap_rw
      7.77 ±  4%     +12.4       20.20 ± 10%  perf-profile.calltrace.cycles-pp._raw_read_lock.jbd2_transaction_committed.ext4_set_iomap.ext4_iomap_begin.iomap_apply
     14.31           +13.3       27.60 ± 10%  perf-profile.calltrace.cycles-pp.ext4_set_iomap.ext4_iomap_begin.iomap_apply.dax_iomap_rw.ext4_file_write_iter
      4.04 ±  6%     +21.9       25.93 ± 22%  perf-profile.calltrace.cycles-pp.dax_iomap_actor.iomap_apply.dax_iomap_rw.ext4_file_write_iter.new_sync_write
     42.24 ±  2%     -41.5        0.77 ±  6%  perf-profile.children.cycles-pp.__ext4_journal_start_sb
     40.51           -39.8        0.73 ±  5%  perf-profile.children.cycles-pp.jbd2__journal_start
     40.28           -39.6        0.72 ±  5%  perf-profile.children.cycles-pp.start_this_handle
     65.50 ±  2%     -34.1       31.40 ±  9%  perf-profile.children.cycles-pp.ext4_iomap_begin
     69.67           -11.6       58.03 ±  7%  perf-profile.children.cycles-pp.iomap_apply
     69.70           -11.5       58.20 ±  7%  perf-profile.children.cycles-pp.dax_iomap_rw
      9.50            -9.3        0.18 ±  6%  perf-profile.children.cycles-pp.add_transaction_credits
      9.02 ±  7%      -9.0        0.04 ± 57%  perf-profile.children.cycles-pp.__ext4_journal_stop
      8.72 ±  7%      -8.7        0.04 ± 58%  perf-profile.children.cycles-pp.jbd2_journal_stop
     71.18            -8.3       62.91 ±  3%  perf-profile.children.cycles-pp.ext4_file_write_iter
     71.25            -7.5       63.75        perf-profile.children.cycles-pp.new_sync_write
      6.43 ±  5%      -6.4        0.00        perf-profile.children.cycles-pp.stop_this_handle
     71.51            -5.8       65.70        perf-profile.children.cycles-pp.vfs_write
     71.58            -5.5       66.11        perf-profile.children.cycles-pp.ksys_write
     71.93            -3.8       68.15        perf-profile.children.cycles-pp.do_syscall_64
     72.05            -3.2       68.81        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.08            +0.0        0.10 ±  4%  perf-profile.children.cycles-pp.scheduler_tick
      0.05            +0.0        0.07 ± 10%  perf-profile.children.cycles-pp.task_tick_fair
      0.06 ±  6%      +0.0        0.09 ± 14%  perf-profile.children.cycles-pp.ext4_reserve_inode_write
      0.00            +0.1        0.06 ± 11%  perf-profile.children.cycles-pp.rcu_all_qs
      0.15 ±  8%      +0.1        0.21 ± 12%  perf-profile.children.cycles-pp.__ext4_mark_inode_dirty
      0.00            +0.1        0.07 ± 17%  perf-profile.children.cycles-pp.rw_verify_area
      0.00            +0.1        0.08 ± 24%  perf-profile.children.cycles-pp.pmem_dax_direct_access
      0.04 ± 57%      +0.1        0.12 ± 18%  perf-profile.children.cycles-pp._cond_resched
      0.00            +0.1        0.08 ±  8%  perf-profile.children.cycles-pp.ktime_get_coarse_real_ts64
      0.00            +0.1        0.09 ± 26%  perf-profile.children.cycles-pp.generic_file_llseek_size
      0.00            +0.1        0.09 ± 20%  perf-profile.children.cycles-pp.file_modified
      0.00            +0.1        0.09 ± 15%  perf-profile.children.cycles-pp.aa_file_perm
      0.00            +0.1        0.10 ± 37%  perf-profile.children.cycles-pp.apparmor_file_permission
      0.00            +0.1        0.11 ± 20%  perf-profile.children.cycles-pp.__pmem_direct_access
      0.03 ±100%      +0.1        0.14 ± 28%  perf-profile.children.cycles-pp.__might_sleep
      0.04 ± 57%      +0.1        0.16 ± 28%  perf-profile.children.cycles-pp.___might_sleep
      0.00            +0.1        0.15 ± 20%  perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
      0.03 ±100%      +0.2        0.21 ± 22%  perf-profile.children.cycles-pp.up_write
      0.00            +0.2        0.19 ±102%  perf-profile.children.cycles-pp.timestamp_truncate
      0.01 ±173%      +0.2        0.21 ± 22%  perf-profile.children.cycles-pp.exit_to_user_mode_prepare
      0.03 ±100%      +0.2        0.22 ± 20%  perf-profile.children.cycles-pp.dax_direct_access
      0.00            +0.2        0.22 ±134%  perf-profile.children.cycles-pp.__sb_end_write
      0.05 ±  9%      +0.2        0.29 ± 18%  perf-profile.children.cycles-pp.__srcu_read_lock
      0.06 ±  7%      +0.3        0.32 ± 21%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
      0.04 ± 58%      +0.3        0.34 ± 54%  perf-profile.children.cycles-pp.current_time
      0.07 ± 11%      +0.3        0.39 ± 22%  perf-profile.children.cycles-pp.down_write
      0.00            +0.3        0.32 ± 83%  perf-profile.children.cycles-pp.__fsnotify_parent
      0.09 ±  7%      +0.3        0.42 ± 19%  perf-profile.children.cycles-pp.common_file_perm
      0.07 ±  6%      +0.3        0.40 ± 22%  perf-profile.children.cycles-pp.syscall_enter_from_user_mode
      0.10 ±  8%      +0.4        0.55 ± 21%  perf-profile.children.cycles-pp.__fget_light
      0.11 ±  9%      +0.5        0.58 ± 21%  perf-profile.children.cycles-pp.security_file_permission
      0.05 ±  9%      +0.5        0.52 ± 46%  perf-profile.children.cycles-pp.__sb_start_write
      0.06 ± 28%      +0.5        0.57 ± 15%  perf-profile.children.cycles-pp.percpu_counter_add_batch
      0.16 ± 30%      +0.5        0.68 ± 20%  perf-profile.children.cycles-pp.entry_SYSCALL_64
      0.12 ±  6%      +0.5        0.67 ± 20%  perf-profile.children.cycles-pp.__fdget_pos
      0.00            +0.5        0.55 ±154%  perf-profile.children.cycles-pp.generic_write_check_limits
      0.00            +0.6        0.64 ±129%  perf-profile.children.cycles-pp.generic_write_checks
      0.01 ±173%      +0.7        0.70 ±117%  perf-profile.children.cycles-pp.ext4_generic_write_checks
      0.10 ± 10%      +0.8        0.91 ± 97%  perf-profile.children.cycles-pp.ext4_llseek
      0.19 ±  6%      +0.8        1.00 ± 20%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.12 ± 42%      +0.9        1.04 ± 52%  perf-profile.children.cycles-pp.ext4_inode_block_valid
      1.06 ±  4%      +1.0        2.04 ± 55%  perf-profile.children.cycles-pp.file_update_time
      0.21 ±  8%      +1.3        1.46 ± 52%  perf-profile.children.cycles-pp.ksys_lseek
      0.21 ± 10%      +1.3        1.50 ±  9%  perf-profile.children.cycles-pp.ext4_es_lookup_extent
      0.17 ± 22%      +1.5        1.65 ± 28%  perf-profile.children.cycles-pp.__check_block_validity
      1.13 ±  6%      +1.7        2.87 ± 67%  perf-profile.children.cycles-pp.ext4_write_checks
     26.94 ±  2%      +2.0       28.91 ±  2%  perf-profile.children.cycles-pp.start_secondary
     27.09 ±  3%      +2.3       29.41 ±  3%  perf-profile.children.cycles-pp.secondary_startup_64
     27.09 ±  3%      +2.3       29.41 ±  3%  perf-profile.children.cycles-pp.cpu_startup_entry
     27.09 ±  3%      +2.3       29.41 ±  3%  perf-profile.children.cycles-pp.do_idle
     27.09 ±  3%      +2.3       29.41 ±  3%  perf-profile.children.cycles-pp.cpuidle_enter
     27.09 ±  3%      +2.3       29.41 ±  3%  perf-profile.children.cycles-pp.cpuidle_enter_state
     27.09 ±  3%      +2.3       29.41 ±  3%  perf-profile.children.cycles-pp.intel_idle
      0.41 ± 11%      +2.9        3.36 ± 11%  perf-profile.children.cycles-pp.ext4_map_blocks
     14.60 ±  2%      +6.1       20.66 ± 10%  perf-profile.children.cycles-pp._raw_read_lock
      1.64 ±  8%     +10.0       11.64 ± 25%  perf-profile.children.cycles-pp.__srcu_read_unlock
      2.18 ± 15%     +10.9       13.12 ± 31%  perf-profile.children.cycles-pp.__copy_user_nocache
      2.19 ± 15%     +11.0       13.17 ± 31%  perf-profile.children.cycles-pp.__copy_user_flushcache
      2.24 ± 15%     +11.2       13.41 ± 31%  perf-profile.children.cycles-pp._copy_from_iter_flushcache
     14.13           +11.6       25.73 ± 10%  perf-profile.children.cycles-pp.jbd2_transaction_committed
     14.31           +13.3       27.62 ± 10%  perf-profile.children.cycles-pp.ext4_set_iomap
      4.04 ±  6%     +21.9       25.97 ± 22%  perf-profile.children.cycles-pp.dax_iomap_actor
     23.93           -23.5        0.42 ±  6%  perf-profile.self.cycles-pp.start_this_handle
      9.46            -9.3        0.18 ±  6%  perf-profile.self.cycles-pp.add_transaction_credits
      6.40 ±  5%      -6.4        0.00        perf-profile.self.cycles-pp.stop_this_handle
      0.00            +0.1        0.08 ± 19%  perf-profile.self.cycles-pp.do_syscall_64
      0.00            +0.1        0.08 ± 24%  perf-profile.self.cycles-pp.current_time
      0.00            +0.1        0.08 ±  8%  perf-profile.self.cycles-pp.ktime_get_coarse_real_ts64
      0.00            +0.1        0.08 ± 17%  perf-profile.self.cycles-pp.aa_file_perm
      0.00            +0.1        0.08 ± 24%  perf-profile.self.cycles-pp.generic_file_llseek_size
      0.00            +0.1        0.09 ± 40%  perf-profile.self.cycles-pp.apparmor_file_permission
      0.00            +0.1        0.09 ± 23%  perf-profile.self.cycles-pp.generic_write_checks
      0.00            +0.1        0.10 ± 18%  perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
      0.01 ±173%      +0.1        0.12 ± 30%  perf-profile.self.cycles-pp.__might_sleep
      0.00            +0.1        0.11 ± 20%  perf-profile.self.cycles-pp.__pmem_direct_access
      0.00            +0.1        0.11 ± 20%  perf-profile.self.cycles-pp.ksys_lseek
      0.00            +0.1        0.11 ± 19%  perf-profile.self.cycles-pp.ksys_write
      0.00            +0.1        0.12 ± 18%  perf-profile.self.cycles-pp.syscall_exit_to_user_mode
      0.00            +0.1        0.13 ± 22%  perf-profile.self.cycles-pp.__fdget_pos
      0.03 ±100%      +0.1        0.16 ± 28%  perf-profile.self.cycles-pp.___might_sleep
      0.00            +0.2        0.16 ± 22%  perf-profile.self.cycles-pp.dax_iomap_rw
      0.00            +0.2        0.18 ±110%  perf-profile.self.cycles-pp.timestamp_truncate
      0.00            +0.2        0.18 ± 24%  perf-profile.self.cycles-pp.exit_to_user_mode_prepare
      0.03 ±100%      +0.2        0.21 ± 22%  perf-profile.self.cycles-pp.up_write
      0.03 ±100%      +0.2        0.22 ± 22%  perf-profile.self.cycles-pp.down_write
      0.00            +0.2        0.22 ± 28%  perf-profile.self.cycles-pp.ext4_map_blocks
      0.00            +0.2        0.22 ±133%  perf-profile.self.cycles-pp.__sb_end_write
      0.05 ±  9%      +0.2        0.29 ± 20%  perf-profile.self.cycles-pp.dax_iomap_actor
      0.04 ± 58%      +0.2        0.28 ± 17%  perf-profile.self.cycles-pp.__srcu_read_lock
      0.00            +0.2        0.24 ± 25%  perf-profile.self.cycles-pp._copy_from_iter_flushcache
      0.03 ±100%      +0.2        0.27 ± 27%  perf-profile.self.cycles-pp.vfs_write
      0.08 ±  5%      +0.2        0.33 ± 20%  perf-profile.self.cycles-pp.common_file_perm
      0.07 ± 10%      +0.3        0.36 ± 21%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.00            +0.3        0.30 ± 95%  perf-profile.self.cycles-pp.__sb_start_write
      0.00            +0.3        0.31 ± 88%  perf-profile.self.cycles-pp.__fsnotify_parent
      0.07 ±  7%      +0.3        0.39 ± 22%  perf-profile.self.cycles-pp.syscall_enter_from_user_mode
      0.10 ±  8%      +0.3        0.43 ± 26%  perf-profile.self.cycles-pp.ext4_iomap_begin
      0.10 ±  4%      +0.4        0.53 ± 20%  perf-profile.self.cycles-pp.__fget_light
      0.04 ± 63%      +0.5        0.53 ± 15%  perf-profile.self.cycles-pp.percpu_counter_add_batch
      0.11 ±  4%      +0.5        0.62 ± 21%  perf-profile.self.cycles-pp.iomap_apply
      0.07 ± 10%      +0.5        0.61 ± 19%  perf-profile.self.cycles-pp.ext4_es_lookup_extent
      0.00            +0.5        0.55 ±154%  perf-profile.self.cycles-pp.generic_write_check_limits
      0.13 ±  6%      +0.6        0.68 ± 20%  perf-profile.self.cycles-pp.entry_SYSCALL_64
      0.04 ± 60%      +0.6        0.60 ± 21%  perf-profile.self.cycles-pp.__check_block_validity
      0.00            +0.7        0.66 ±136%  perf-profile.self.cycles-pp.file_update_time
      0.01 ±173%      +0.8        0.79 ±116%  perf-profile.self.cycles-pp.new_sync_write
      0.10 ±  8%      +0.8        0.89 ± 99%  perf-profile.self.cycles-pp.ext4_llseek
      0.19 ±  6%      +0.8        1.00 ± 20%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.11 ± 39%      +0.9        1.04 ± 52%  perf-profile.self.cycles-pp.ext4_inode_block_valid
      0.23 ± 18%      +1.0        1.26 ± 38%  perf-profile.self.cycles-pp.ext4_file_write_iter
      0.18 ± 16%      +1.7        1.88 ± 67%  perf-profile.self.cycles-pp.ext4_set_iomap
     27.09 ±  3%      +2.3       29.41 ±  3%  perf-profile.self.cycles-pp.intel_idle
     14.53 ±  2%      +6.0       20.54 ± 10%  perf-profile.self.cycles-pp._raw_read_lock
      1.63 ±  8%     +10.0       11.58 ± 25%  perf-profile.self.cycles-pp.__srcu_read_unlock
      2.18 ± 15%     +10.9       13.05 ± 31%  perf-profile.self.cycles-pp.__copy_user_nocache


                                                                                
                                  fio.write_bw_MBps                             
                                                                                
  70000 +-------------------------------------------------------------------+   
        |                                       O                           |   
  60000 |-O       O                                                         |   
        |             O                     O                               |   
        |     O         O                     O                             |   
  50000 |-+ O   O         O O O O O O   O                                   |   
        |           O                 O                                     |   
  40000 |-+                               O                                 |   
        |                                                                   |   
  30000 |-+                                                                 |   
        |                                                                   |   
        |                                                                   |   
  20000 |-+                                                                 |   
        |                .+.                                                |   
  10000 +-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                    fio.write_iops                              
                                                                                
  1.8e+07 +-----------------------------------------------------------------+   
          |                                      O                          |   
  1.6e+07 |-O       O                                                       |   
  1.4e+07 |-+           O                    O                              |   
          |     O         O                    O                            |   
  1.2e+07 |-+ O   O        O O O O O O   O                                  |   
          |           O                O                                    |   
    1e+07 |-+                              O                                |   
          |                                                                 |   
    8e+06 |-+                                                               |   
    6e+06 |-+                                                               |   
          |                                                                 |   
    4e+06 |-+                                                               |   
          |.+.+.+.+.+.+.+.++.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.++.+.+.+.+.+.+.+.|   
    2e+06 +-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                               fio.write_clat_mean_us                           
                                                                                
  18000 +-------------------------------------------------------------------+   
        |    .+.                                   .+.       .+. .+.      +.|   
  16000 |.+.+   +.+.+.+.+   +.+.+.+.+.+.+.+.+.+.+.+   +.+.+.+   +   +.+. +  |   
  14000 |-+              + +                                            +   |   
        |                 +                                                 |   
  12000 |-+                                                                 |   
        |                                                                   |   
  10000 |-+                                                                 |   
        |                                                                   |   
   8000 |-+                                                                 |   
   6000 |-+                                                                 |   
        |                                                                   |   
   4000 |-+         O                 O   O                                 |   
        | O O O O     O O O O O O O O   O   O O                             |   
   2000 +-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                fio.write_clat_stddev                           
                                                                                
  16000 +-------------------------------------------------------------------+   
        |                             +                                     |   
  14000 |-+                           ::                                    |   
  12000 |-+                          : :                              +     |   
        |            .+              :  :.+                          : :    |   
  10000 |.+. .+.    +  +            :   +  +   .+. .+.   .+. .+.     : :    |   
        |   +   +. +    +   +.+.+. .+       +.+   +   +.+   +   +.+.+   +.+.|   
   8000 |-+       +      + +      +                                         |   
        |                 +                                                 |   
   6000 |-+                                                                 |   
   4000 |-+         O                                                       |   
        |   O O   O                   O   O O                               |   
   2000 |-O     O     O O O O O O O O   O     O O                           |   
        |                                                                   |   
      0 +-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                fio.write_clat_90__us                           
                                                                                
  30000 +-------------------------------------------------------------------+   
        |    .+.      +                            .+.       .+.      +.    |   
  25000 |.+.+   +.   : :    +.+.+    .+.+.+.+.+.+.+   +.+.+.+   +.+. +  +.+ |   
        |         +. : :   :     + .+                               +      +|   
        |           +   +. :      +                                         |   
  20000 |-+               +                                                 |   
        |                                                                   |   
  15000 |-+                                                                 |   
        |                                                                   |   
  10000 |-+                                                                 |   
        |                                                                   |   
        |           O     O O O O O O O   O                                 |   
   5000 |-O O O O O   O O               O   O O O                           |   
        |                                                                   |   
      0 +-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                fio.write_clat_95__us                           
                                                                                
  35000 +-------------------------------------------------------------------+   
        |             +                                                     |   
  30000 |.+.+.+.+    : :    +.+.      +.+.+.+.+.+.+.+.+.+.+.+.+.+.   .+.+.  |   
        |        +   : :   :    +.   +                            +.+     +.|   
  25000 |-+       +.+   +. :      +.+                                       |   
        |                 +                                                 |   
  20000 |-+                                                                 |   
        |                                                                   |   
  15000 |-+                                                                 |   
        |                                                                   |   
  10000 |-+                                                                 |   
        |           O   O O O O O O O O O O                                 |   
   5000 |-O O O O O   O                     O O O                           |   
        |                                                                   |   
      0 +-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                fio.write_clat_99__us                           
                                                                                
  45000 +-------------------------------------------------------------------+   
        |             +                                                     |   
  40000 |.+. .+.      ::     .+.      +. .+.+. .+. .+.   .+. .+.     .+.+.  |   
  35000 |-+ +   +.   : :    +   +.   +  +     +   +   +.+   +   +.+.+     +.|   
        |         +. :  :  :      +.+                                       |   
  30000 |-+         +   +. :                                                |   
        |                 +                                                 |   
  25000 |-+                                                                 |   
        |                                                                   |   
  20000 |-+                                                                 |   
  15000 |-+         O                 O   O                                 |   
        |   O O O     O O O O O O O O   O   O O                             |   
  10000 |-O       O                             O                           |   
        |                                                                   |   
   5000 +-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                fio.latency_4us_                                
                                                                                
  70 +----------------------------------------------------------------------+   
     |   O   O                                                              |   
  60 |-+          O                 O                                       |   
     |     O            O O   O O     O    O O                              |   
  50 |-+            O O     O     O      O                                  |   
     | O                                                                    |   
  40 |-+       O                                                            |   
     |                                         O                            |   
  30 |-+                                                                    |   
     |                                                                      |   
  20 |-+                                                                    |   
     |                                                                      |   
  10 |-+                                                                    |   
     |                                                                      |   
   0 +----------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                fio.latency_20us_                               
                                                                                
  80 +----------------------------------------------------------------------+   
     |           .+   +.        +.                                          |   
  70 |-+       +. :   : +      :  +                                         |   
  60 |-+      +    :  :  :     :  :                                        :|   
     |.     .+     : :   :    +    :         +.     .+.+.     .+.. .+      :|   
  50 |-+.+.+       : :    +  :     :        +  +.+.+     +.+.+    +  :    : |   
     |             : :     + :      :   .+.+                         :    + |   
  40 |-+            ::      +       +.+.                              +. +  |   
     |              :                                                   +   |   
  30 |-+            +                                                       |   
  20 |-+                                                                    |   
     |                                                                      |   
  10 |-+                                                                    |   
     |                                                                      |   
   0 +----------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                fio.latency_50us_                               
                                                                                
  35 +----------------------------------------------------------------------+   
     |              +                                                       |   
  30 |-+   +        :                              +         +        +.    |   
     |    : +      : :              +.+..         : +       : +  .+  :  +.+ |   
  25 |.+. :  +     : :    +.+.      :    +.+. .+. :  +. .+. :  +.  + :    : |   
     |   +    :    : :    :   +    :         +   +     +   +        +      :|   
  20 |-+      :   :  :   :     +   :                                       :|   
     |         +..+   :  :      +.+                                         |   
  15 |-+              +  :                                                  |   
     |                 : :                                                  |   
  10 |-+               ::                                                   |   
     |                  +                                                   |   
   5 |-+                                                                    |   
     |                                                                      |   
   0 +----------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                fio.latency_100us_                              
                                                                                
   0.3 +--------------------------------------------------------------------+   
       |                                                                    |   
  0.25 |-+   +       +                              +         +       +     |   
       |     ::      :                    +         ::        ::      :     |   
       |    : :      ::              +   + +       : :       : :     : :    |   
   0.2 |-+. :  :    : :              :+ +   +   +. :  :   +. :  :    : :    |   
       |+  +   +    : :     .+      :  +     + +  +   +. +  +   +.+.:   :   |   
  0.15 |-+      +   : :    +  +     :         +         +           +   +.+ |   
       |         +  :  :   :   +. .+                                       :|   
   0.1 |-+        +:   +  :      +                                         :|   
       |           +    + :                                                 |   
       |                 +                                                  |   
  0.05 |-+                                                                  |   
       |                                                                    |   
     0 +--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                     fio.workload                               
                                                                                
  3.5e+09 +-----------------------------------------------------------------+   
          |                                      O                          |   
    3e+09 |-O       O                                                       |   
          |             O O                  O                              |   
          |     O O                      O     O                            |   
  2.5e+09 |-+ O            O O O O O O                                      |   
          |           O                O                                    |   
    2e+09 |-+                              O                                |   
          |                                                                 |   
  1.5e+09 |-+                                                               |   
          |                                                                 |   
          |                                                                 |   
    1e+09 |-+                                                               |   
          |                +.                                          .+.  |   
    5e+08 +-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                fio.time.user_time                              
                                                                                
  1800 +--------------------------------------------------------------------+   
       | O                                      O                           |   
  1600 |-+       O                                                          |   
  1400 |-+           O                      O                               |   
       |     O O       O                      O                             |   
  1200 |-+ O             O O O O O O   O                                    |   
       |           O                 O                                      |   
  1000 |-+                                O                                 |   
       |                                                                    |   
   800 |-+                                                                  |   
   600 |-+                                                                  |   
       |                                                                    |   
   400 |-+                                                                  |   
       |.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+..+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.|   
   200 +--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                               fio.time.system_time                             
                                                                                
  9400 +--------------------------------------------------------------------+   
       |              .+.                                        .+.        |   
  9200 |.+.+.+.+.+.+.+   +.+.+.+.+.+.+.+..+.+.+.+.+.+.+.+.+.+.+.+   +.+.+.+.|   
  9000 |-+                                                                  |   
       |                                                                    |   
  8800 |-+                                                                  |   
       |                                                                    |   
  8600 |-+                                                                  |   
       |                             O    O                                 |   
  8400 |-+         O                                                        |   
  8200 |-+ O O O         O O O O O O   O      O                             |   
       |               O                                                    |   
  8000 |-+       O   O                      O                               |   
       | O                                                                  |   
  7800 +--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                         fio.time.voluntary_context_switches                    
                                                                                
  25000 +-------------------------------------------------------------------+   
        |                     +.                                            |   
  24500 |-+        .+.+.+    +  +                                           |   
        |.+.+.   .+      + .+    + .+.   .+.   .+.+.     .+.+.   .+.+.+.    |   
        |     +.+         +       +   +.+   +.+     +.+.+     +.+       +.+.|   
  24000 |-+                                                                 |   
        |                                                                   |   
  23500 |-+                                                                 |   
        |                                                                   |   
  23000 |-+                                                                 |   
        |                                                                   |   
        |                             O O O   O                             |   
  22500 |-O                   O     O                                       |   
        |   O O O O O O O O O   O O         O   O                           |   
  22000 +-------------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.9.0-rc7-00013-g061113efe99b2" of type "text/plain" (170148 bytes)

View attachment "job-script" of type "text/plain" (8348 bytes)

View attachment "job.yaml" of type "text/plain" (5848 bytes)

View attachment "reproduce" of type "text/plain" (931 bytes)

Powered by blists - more mailing lists