lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Message-ID: <20200204091347.GI12867@shao2-debian>
Date:   Tue, 4 Feb 2020 17:13:48 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     Matthew Bobrowski <mbobrowski@...browski.org>
Cc:     Theodore Ts'o <tytso@....edu>, Jan Kara <jack@...e.cz>,
        Ritesh Harjani <riteshh@...ux.ibm.com>,
        LKML <linux-kernel@...r.kernel.org>,
        Linus Torvalds <torvalds@...ux-foundation.org>,
        lkp@...ts.01.org
Subject: [ext4] 569342dc24: fio.read_bw_MBps 56.4% improvement

Greeting,

FYI, we noticed a 56.4% improvement of fio.read_bw_MBps due to commit:


commit: 569342dc2485392e95b6a626281708c25014ba37 ("ext4: move inode extension/truncate code out from ->iomap_end() callback")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master

in testcase: fio-basic
on test machine: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 256G memory
with following parameters:

	disk: 2pmem
	fs: ext4
	mount_option: dax
	runtime: 200s
	nr_task: 50%
	time_based: tb
	rw: rw
	bs: 4k
	ioengine: libaio
	test_size: 200G
	cpufreq_governor: performance
	ucode: 0x500002c

test-description: Fio is a tool that will spawn a number of threads or processes doing a particular type of I/O action as specified by the user.
test-url: https://github.com/axboe/fio

In addition to that, the commit also has significant impact on the following tests:

+------------------+----------------------------------------------------------------------+
| testcase: change | fio-basic: fio.write_bw_MBps 54.0% improvement                       |
| test machine     | 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 256G memory |
| test parameters  | bs=4k                                                                |
|                  | cpufreq_governor=performance                                         |
|                  | disk=2pmem                                                           |
|                  | fs=ext4                                                              |
|                  | ioengine=sync                                                        |
|                  | mount_option=dax                                                     |
|                  | nr_task=50%                                                          |
|                  | runtime=200s                                                         |
|                  | rw=randwrite                                                         |
|                  | test_size=200G                                                       |
|                  | time_based=tb                                                        |
|                  | ucode=0x500002c                                                      |
+------------------+----------------------------------------------------------------------+
| testcase: change | fio-basic: fio.write_bw_MBps 68.8% improvement                       |
| test machine     | 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 256G memory |
| test parameters  | bs=4k                                                                |
|                  | cpufreq_governor=performance                                         |
|                  | disk=2pmem                                                           |
|                  | fs=ext4                                                              |
|                  | ioengine=libaio                                                      |
|                  | mount_option=dax                                                     |
|                  | nr_task=50%                                                          |
|                  | runtime=200s                                                         |
|                  | rw=randwrite                                                         |
|                  | test_size=200G                                                       |
|                  | time_based=tb                                                        |
|                  | ucode=0x500002c                                                      |
+------------------+----------------------------------------------------------------------+




Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
bs/compiler/cpufreq_governor/disk/fs/ioengine/kconfig/mount_option/nr_task/rootfs/runtime/rw/tbox_group/test_size/testcase/time_based/ucode:
  4k/gcc-7/performance/2pmem/ext4/libaio/x86_64-rhel-7.6/dax/50%/debian-x86_64-20191114.cgz/200s/rw/lkp-csl-2sp6/200G/fio-basic/tb/0x500002c

commit: 
  b1b4705d54 ("ext4: introduce direct I/O read using iomap infrastructure")
  569342dc24 ("ext4: move inode extension/truncate code out from ->iomap_end() callback")

b1b4705d54abedfd 569342dc2485392e95b6a626281 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :4           25%           1:4     dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x
           :4           25%           1:4     dmesg.WARNING:at_ip__fsnotify_parent/0x
          1:4          -25%            :4     dmesg.WARNING:at_ip_aio_read/0x
           :4           25%           1:4     dmesg.WARNING:stack_recursion
         %stddev     %change         %stddev
             \          |                \  
     14.31 ± 30%     -14.3        0.01        fio.latency_1000us%
      3.05 ± 72%     +20.2       23.21 ± 69%  fio.latency_250us%
      0.62 ± 83%      -0.6        0.01        fio.latency_2ms%
     39.71 ± 26%     -31.5        8.24 ± 83%  fio.latency_750us%
      5456           +56.4%       8534        fio.read_bw_MBps
    794624 ±  5%     -40.5%     472576 ± 10%  fio.read_clat_90%_us
    849920 ±  6%     -40.8%     502784 ± 10%  fio.read_clat_95%_us
    948224 ±  7%     -41.1%     558592 ±  9%  fio.read_clat_99%_us
    532777           -36.1%     340678        fio.read_clat_mean_us
    194664 ± 13%     -49.1%      99057 ± 34%  fio.read_clat_stddev
   1396934           +56.4%    2184757        fio.read_iops
      4719 ±  2%      -5.4%       4464        fio.read_slat_mean_us
      4020 ±  5%     -17.9%       3300 ± 10%  fio.read_slat_stddev
      4567 ±  9%     +11.3%       5084 ±  9%  fio.time.involuntary_context_switches
      9322            -1.7%       9160        fio.time.system_time
    286.59 ±  2%     +56.4%     448.31 ±  2%  fio.time.user_time
 5.588e+08           +56.4%  8.739e+08        fio.workload
      5456           +56.4%       8534        fio.write_bw_MBps
    796672 ±  4%     -40.7%     472576 ± 10%  fio.write_clat_90%_us
    849920 ±  6%     -40.7%     503808 ± 10%  fio.write_clat_95%_us
    948224 ±  7%     -41.1%     558592 ±  9%  fio.write_clat_99%_us
    533381           -36.1%     341017        fio.write_clat_mean_us
    194676 ± 13%     -49.1%      99161 ± 34%  fio.write_clat_stddev
   1396910           +56.4%    2184882        fio.write_iops
     28403           -42.8%      16259        fio.write_slat_mean_us
     12283 ± 11%     -45.7%       6666 ± 22%  fio.write_slat_stddev
      1.50 ±  2%      +0.8        2.29 ±  4%  mpstat.cpu.all.usr%
      5899 ± 79%    +151.9%      14862 ± 51%  softirqs.CPU48.SCHED
     47.80            -3.5%      46.11 ±  2%  iostat.cpu.system
      1.49 ±  2%     +52.9%       2.27 ±  4%  iostat.cpu.user
     18587           -82.9%       3179 ±170%  numa-numastat.node0.other_node
    103.25 ± 33%  +14978.9%      15569 ± 34%  numa-numastat.node1.other_node
     45512 ± 23%     -59.2%      18576 ±144%  sched_debug.cfs_rq:/.spread0.avg
     15.56 ± 11%     +33.3%      20.75 ±  7%  sched_debug.cpu.nr_uninterruptible.max
     38681 ±  3%    +535.2%     245689 ±143%  turbostat.C1
    126.48            +4.2%     131.76        turbostat.RAMWatt
      9311 ±  8%     +20.5%      11221 ±  6%  numa-vmstat.node0.nr_slab_reclaimable
     19397 ±  2%     -79.7%       3942 ±142%  numa-vmstat.node0.numa_other
      9935 ±  7%     -19.6%       7983 ± 10%  numa-vmstat.node1.nr_slab_reclaimable
    156898            +9.7%     172186 ±  3%  numa-vmstat.node1.numa_other
     37246 ±  8%     +20.5%      44887 ±  6%  numa-meminfo.node0.KReclaimable
     37246 ±  8%     +20.5%      44887 ±  6%  numa-meminfo.node0.SReclaimable
    137281 ±  3%      +7.8%     148012 ±  2%  numa-meminfo.node0.Slab
     39742 ±  7%     -19.6%      31935 ± 10%  numa-meminfo.node1.KReclaimable
     39742 ±  7%     -19.6%      31935 ± 10%  numa-meminfo.node1.SReclaimable
     47.25 ±137%    -100.0%       0.00        interrupts.46:PCI-MSI.31981579-edge.i40e-eth0-TxRx-10
     59.50 ± 61%     -71.0%      17.25 ±109%  interrupts.CPU10.TLB:TLB_shootdowns
     83.25 ± 45%     -70.9%      24.25 ± 81%  interrupts.CPU11.TLB:TLB_shootdowns
     70.50 ± 50%     -76.6%      16.50 ± 90%  interrupts.CPU16.TLB:TLB_shootdowns
     87.00 ± 44%     -73.9%      22.75 ± 93%  interrupts.CPU17.TLB:TLB_shootdowns
     75.75 ± 46%     -59.1%      31.00 ± 98%  interrupts.CPU20.TLB:TLB_shootdowns
     78.00 ± 42%     -72.4%      21.50 ± 67%  interrupts.CPU22.TLB:TLB_shootdowns
      7327 ± 12%     -40.0%       4393 ± 46%  interrupts.CPU26.NMI:Non-maskable_interrupts
      7327 ± 12%     -40.0%       4393 ± 46%  interrupts.CPU26.PMI:Performance_monitoring_interrupts
      7.25 ± 65%   +1220.7%      95.75 ± 82%  interrupts.CPU3.RES:Rescheduling_interrupts
      1965 ±  7%      -9.4%       1781 ±  9%  interrupts.CPU30.CAL:Function_call_interrupts
      1951 ±  6%     -10.0%       1756 ±  7%  interrupts.CPU31.CAL:Function_call_interrupts
      1957 ±  7%     -10.2%       1757 ±  7%  interrupts.CPU32.CAL:Function_call_interrupts
      1853 ± 13%     +25.4%       2324 ± 16%  interrupts.CPU43.CAL:Function_call_interrupts
      7367 ± 12%     -45.3%       4027 ± 58%  interrupts.CPU44.NMI:Non-maskable_interrupts
      7367 ± 12%     -45.3%       4027 ± 58%  interrupts.CPU44.PMI:Performance_monitoring_interrupts
     13.25 ± 35%    +300.0%      53.00 ± 83%  interrupts.CPU48.RES:Rescheduling_interrupts
     64.25 ± 64%     -60.3%      25.50 ± 54%  interrupts.CPU55.TLB:TLB_shootdowns
     74.00 ± 57%     -74.7%      18.75 ± 81%  interrupts.CPU7.TLB:TLB_shootdowns
     40.50 ± 50%    +108.0%      84.25 ± 21%  interrupts.CPU75.TLB:TLB_shootdowns
      5558 ± 32%     -52.9%       2617 ± 21%  interrupts.CPU8.NMI:Non-maskable_interrupts
      5558 ± 32%     -52.9%       2617 ± 21%  interrupts.CPU8.PMI:Performance_monitoring_interrupts
      5.50 ± 91%   +1440.9%      84.75 ± 34%  interrupts.CPU8.RES:Rescheduling_interrupts
     40.25 ± 38%     +91.3%      77.00 ± 33%  interrupts.CPU83.TLB:TLB_shootdowns
     41.00 ± 46%     +95.7%      80.25 ± 22%  interrupts.CPU90.TLB:TLB_shootdowns
 4.867e+09           +46.6%  7.133e+09 ±  3%  perf-stat.i.branch-instructions
  35659892 ±  2%     +40.1%   49959300        perf-stat.i.branch-misses
     48.56 ±  2%      +6.7       55.23 ±  2%  perf-stat.i.cache-miss-rate%
 1.806e+08 ±  3%     +24.0%  2.239e+08 ±  3%  perf-stat.i.cache-misses
 3.709e+08 ±  2%      +7.7%  3.993e+08 ±  2%  perf-stat.i.cache-references
      5.12           -31.6%       3.50 ±  3%  perf-stat.i.cpi
    750.82 ±  3%     -16.1%     630.03 ±  9%  perf-stat.i.cycles-between-cache-misses
  7.13e+09 ±  9%     +47.4%  1.051e+10 ±  8%  perf-stat.i.dTLB-loads
 4.827e+09 ±  8%     +45.6%  7.027e+09 ±  3%  perf-stat.i.dTLB-stores
  38590154 ±  4%     +23.4%   47627762 ±  9%  perf-stat.i.iTLB-load-misses
 2.618e+10           +46.5%  3.835e+10 ±  3%  perf-stat.i.instructions
    686.01 ±  3%     +19.6%     820.13 ±  7%  perf-stat.i.instructions-per-iTLB-miss
      0.20           +47.3%       0.29        perf-stat.i.ipc
   1537507 ±  4%     +17.0%    1798882 ±  7%  perf-stat.i.node-loads
  13902646 ±  3%      -8.9%   12667482 ±  3%  perf-stat.i.node-store-misses
     17738 ±  3%     +16.9%      20736 ±  4%  perf-stat.i.node-stores
     14.17 ±  2%     -26.5%      10.41        perf-stat.overall.MPKI
      0.73            -0.0        0.70        perf-stat.overall.branch-miss-rate%
     48.71 ±  2%      +7.4       56.08        perf-stat.overall.cache-miss-rate%
      5.13           -32.9%       3.44        perf-stat.overall.cpi
    744.79 ±  3%     -20.8%     589.98        perf-stat.overall.cycles-between-cache-misses
     86.06            +1.7       87.73        perf-stat.overall.iTLB-load-miss-rate%
    679.44 ±  3%     +19.2%     810.22 ±  6%  perf-stat.overall.instructions-per-iTLB-miss
      0.19           +49.0%       0.29        perf-stat.overall.ipc
      9405            -4.4%       8987        perf-stat.overall.path-length
 4.842e+09           +46.6%  7.097e+09 ±  3%  perf-stat.ps.branch-instructions
  35478652 ±  2%     +40.1%   49709486        perf-stat.ps.branch-misses
 1.797e+08 ±  3%     +24.0%  2.228e+08 ±  3%  perf-stat.ps.cache-misses
  3.69e+08 ±  2%      +7.7%  3.973e+08 ±  2%  perf-stat.ps.cache-references
 7.093e+09 ±  9%     +47.4%  1.046e+10 ±  8%  perf-stat.ps.dTLB-loads
 4.803e+09 ±  8%     +45.6%  6.992e+09 ±  3%  perf-stat.ps.dTLB-stores
  38394613 ±  4%     +23.4%   47389655 ±  9%  perf-stat.ps.iTLB-load-misses
 2.605e+10           +46.5%  3.816e+10 ±  3%  perf-stat.ps.instructions
   1529726 ±  4%     +17.0%    1789899 ±  7%  perf-stat.ps.node-loads
  13832194 ±  3%      -8.9%   12604236 ±  3%  perf-stat.ps.node-store-misses
     17673 ±  3%     +16.9%      20659 ±  4%  perf-stat.ps.node-stores
 5.255e+12           +49.5%  7.854e+12        perf-stat.total.instructions
     20.14 ±  9%     -20.1        0.00        perf-profile.calltrace.cycles-pp.ext4_iomap_end.iomap_apply.dax_iomap_rw.ext4_file_write_iter.aio_write
     14.28 ±  9%     -14.3        0.00        perf-profile.calltrace.cycles-pp.jbd2__journal_start.ext4_iomap_end.iomap_apply.dax_iomap_rw.ext4_file_write_iter
     14.20 ±  9%     -14.2        0.00        perf-profile.calltrace.cycles-pp.start_this_handle.jbd2__journal_start.ext4_iomap_end.iomap_apply.dax_iomap_rw
      0.60 ± 10%      +0.4        1.05 ±  7%  perf-profile.calltrace.cycles-pp.do_io_getevents.__x64_sys_io_getevents.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.64 ± 10%      +0.5        1.11 ±  7%  perf-profile.calltrace.cycles-pp.__x64_sys_io_getevents.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.80 ± 12%      +0.6        1.39 ± 10%  perf-profile.calltrace.cycles-pp.__srcu_read_unlock.dax_iomap_actor.iomap_apply.dax_iomap_rw.ext4_file_write_iter
      0.12 ±173%      +0.6        0.75 ±  7%  perf-profile.calltrace.cycles-pp.read_events.do_io_getevents.__x64_sys_io_getevents.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +0.7        0.68 ±  7%  perf-profile.calltrace.cycles-pp.aio_read_events.read_events.do_io_getevents.__x64_sys_io_getevents.do_syscall_64
      1.02 ± 11%      +0.7        1.74 ±  4%  perf-profile.calltrace.cycles-pp.__copy_user_nocache.__copy_user_flushcache._copy_from_iter_flushcache.dax_iomap_actor.iomap_apply
      1.02 ± 12%      +0.7        1.75 ±  4%  perf-profile.calltrace.cycles-pp.__copy_user_flushcache._copy_from_iter_flushcache.dax_iomap_actor.iomap_apply.dax_iomap_rw
      1.04 ± 11%      +0.7        1.78 ±  4%  perf-profile.calltrace.cycles-pp._copy_from_iter_flushcache.dax_iomap_actor.iomap_apply.dax_iomap_rw.ext4_file_write_iter
      1.09 ±  7%      +0.8        1.87 ±  7%  perf-profile.calltrace.cycles-pp.__memcpy_mcsafe.copyout_mcsafe._copy_to_iter_mcsafe.dax_iomap_actor.iomap_apply
      1.11 ±  8%      +0.8        1.91 ±  8%  perf-profile.calltrace.cycles-pp.copyout_mcsafe._copy_to_iter_mcsafe.dax_iomap_actor.iomap_apply.dax_iomap_rw
      1.16 ±  8%      +0.8        1.98 ±  8%  perf-profile.calltrace.cycles-pp._copy_to_iter_mcsafe.dax_iomap_actor.iomap_apply.dax_iomap_rw.ext4_file_read_iter
      1.24 ±  8%      +0.9        2.14 ±  7%  perf-profile.calltrace.cycles-pp.dax_iomap_actor.iomap_apply.dax_iomap_rw.ext4_file_read_iter.aio_read
      2.00 ±  7%      +1.4        3.38 ±  7%  perf-profile.calltrace.cycles-pp.ext4_journal_check_start.__ext4_journal_start_sb.ext4_iomap_begin.iomap_apply.dax_iomap_rw
      1.91 ±  8%      +1.4        3.29 ±  6%  perf-profile.calltrace.cycles-pp.dax_iomap_actor.iomap_apply.dax_iomap_rw.ext4_file_write_iter.aio_write
      2.01 ±  7%      +1.4        3.40 ±  7%  perf-profile.calltrace.cycles-pp.__ext4_journal_start_sb.ext4_iomap_begin.iomap_apply.dax_iomap_rw.ext4_file_write_iter
      2.81 ±  9%      +1.5        4.34 ±  6%  perf-profile.calltrace.cycles-pp._raw_read_lock.start_this_handle.jbd2__journal_start.ext4_iomap_begin.iomap_apply
      3.50 ±  9%      +2.1        5.63 ±  6%  perf-profile.calltrace.cycles-pp.add_transaction_credits.start_this_handle.jbd2__journal_start.ext4_iomap_begin.iomap_apply
      4.08 ±  8%      +2.9        6.94 ±  7%  perf-profile.calltrace.cycles-pp._raw_read_lock.jbd2_transaction_committed.ext4_set_iomap.ext4_iomap_begin.iomap_apply
      7.23 ±  9%      +3.6       10.86 ±  5%  perf-profile.calltrace.cycles-pp.jbd2_journal_stop.__ext4_journal_stop.ext4_iomap_begin.iomap_apply.dax_iomap_rw
      5.24 ±  8%      +3.8        8.99 ±  7%  perf-profile.calltrace.cycles-pp.ext4_set_iomap.ext4_iomap_begin.iomap_apply.dax_iomap_rw.ext4_file_write_iter
      5.21 ±  7%      +3.8        8.98 ±  8%  perf-profile.calltrace.cycles-pp.ext4_set_iomap.ext4_iomap_begin.iomap_apply.dax_iomap_rw.ext4_file_read_iter
      7.33 ±  9%      +3.8       11.12 ±  5%  perf-profile.calltrace.cycles-pp.__ext4_journal_stop.ext4_iomap_begin.iomap_apply.dax_iomap_rw.ext4_file_write_iter
      5.36 ±  8%      +3.9        9.24 ±  8%  perf-profile.calltrace.cycles-pp.ext4_iomap_begin.iomap_apply.dax_iomap_rw.ext4_file_read_iter.aio_read
      6.67 ±  7%      +4.8       11.48 ±  8%  perf-profile.calltrace.cycles-pp.iomap_apply.dax_iomap_rw.ext4_file_read_iter.aio_read.io_submit_one
      6.68 ±  7%      +4.8       11.51 ±  8%  perf-profile.calltrace.cycles-pp.dax_iomap_rw.ext4_file_read_iter.aio_read.io_submit_one.__x64_sys_io_submit
      7.75 ±  8%      +5.0       12.72 ±  8%  perf-profile.calltrace.cycles-pp.ext4_file_read_iter.aio_read.io_submit_one.__x64_sys_io_submit.do_syscall_64
      7.98 ±  8%      +5.1       13.13 ±  8%  perf-profile.calltrace.cycles-pp.aio_read.io_submit_one.__x64_sys_io_submit.do_syscall_64.entry_SYSCALL_64_after_hwframe
     10.41 ±  8%      +7.5       17.87 ±  7%  perf-profile.calltrace.cycles-pp.jbd2_transaction_committed.ext4_set_iomap.ext4_iomap_begin.iomap_apply.dax_iomap_rw
     14.19 ±  9%      +8.7       22.93 ±  7%  perf-profile.calltrace.cycles-pp.start_this_handle.jbd2__journal_start.ext4_iomap_begin.iomap_apply.dax_iomap_rw
     14.26 ±  9%      +8.8       23.05 ±  7%  perf-profile.calltrace.cycles-pp.jbd2__journal_start.ext4_iomap_begin.iomap_apply.dax_iomap_rw.ext4_file_write_iter
     29.03 ±  8%     +17.9       46.90 ±  6%  perf-profile.calltrace.cycles-pp.ext4_iomap_begin.iomap_apply.dax_iomap_rw.ext4_file_write_iter.aio_write
     20.16 ±  9%     -20.2        0.00        perf-profile.children.cycles-pp.ext4_iomap_end
     29.63 ±  9%      -5.5       24.17 ±  7%  perf-profile.children.cycles-pp.jbd2__journal_start
     29.48 ±  9%      -5.4       24.04 ±  7%  perf-profile.children.cycles-pp.start_this_handle
      7.30 ±  9%      -1.4        5.90 ±  7%  perf-profile.children.cycles-pp.add_transaction_credits
      4.17 ±  7%      -0.6        3.54 ±  7%  perf-profile.children.cycles-pp.ext4_journal_check_start
      4.19 ±  7%      -0.6        3.56 ±  7%  perf-profile.children.cycles-pp.__ext4_journal_start_sb
      0.06 ± 17%      +0.0        0.09 ±  9%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.08 ± 15%      +0.0        0.11 ±  4%  perf-profile.children.cycles-pp.aio_complete_rw
      0.06 ±  6%      +0.0        0.10 ±  4%  perf-profile.children.cycles-pp._cond_resched
      0.05 ±  8%      +0.0        0.09 ±  8%  perf-profile.children.cycles-pp.put_reqs_available
      0.04 ± 58%      +0.0        0.08 ± 10%  perf-profile.children.cycles-pp.__get_user_8
      0.09 ± 13%      +0.0        0.13 ±  6%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.07 ± 10%      +0.0        0.12 ±  7%  perf-profile.children.cycles-pp.__check_object_size
      0.04 ± 58%      +0.0        0.09 ±  7%  perf-profile.children.cycles-pp.__put_user_4
      0.07 ± 14%      +0.1        0.12 ± 12%  perf-profile.children.cycles-pp.__fget
      0.06 ± 11%      +0.1        0.11 ± 11%  perf-profile.children.cycles-pp.mutex_lock
      0.01 ±173%      +0.1        0.06 ± 13%  perf-profile.children.cycles-pp.fsnotify
      0.01 ±173%      +0.1        0.07 ±  7%  perf-profile.children.cycles-pp.fput_many
      0.01 ±173%      +0.1        0.07 ± 13%  perf-profile.children.cycles-pp.__virt_addr_valid
      0.00            +0.1        0.05 ±  8%  perf-profile.children.cycles-pp.__pmem_direct_access
      0.18 ±  8%      +0.1        0.23 ±  8%  perf-profile.children.cycles-pp.kmem_cache_alloc
      0.04 ± 57%      +0.1        0.09 ±  4%  perf-profile.children.cycles-pp.__get_reqs_available
      0.01 ±173%      +0.1        0.07 ±  6%  perf-profile.children.cycles-pp.mutex_unlock
      0.00            +0.1        0.06 ±  7%  perf-profile.children.cycles-pp.refill_reqs_available
      0.01 ±173%      +0.1        0.07 ± 10%  perf-profile.children.cycles-pp.__srcu_read_lock
      0.01 ±173%      +0.1        0.07 ±  5%  perf-profile.children.cycles-pp.__inode_security_revalidate
      0.00            +0.1        0.06 ± 13%  perf-profile.children.cycles-pp.ext4_write_checks
      0.00            +0.1        0.07 ± 17%  perf-profile.children.cycles-pp.ext4_data_block_valid
      0.08 ± 13%      +0.1        0.15 ± 17%  perf-profile.children.cycles-pp.aio_prep_rw
      0.03 ±100%      +0.1        0.09 ± 15%  perf-profile.children.cycles-pp.__check_block_validity
      0.01 ±173%      +0.1        0.08 ± 10%  perf-profile.children.cycles-pp.copy_user_generic_unrolled
      0.00            +0.1        0.07 ± 16%  perf-profile.children.cycles-pp.__sb_start_write
      0.00            +0.1        0.07 ± 17%  perf-profile.children.cycles-pp.current_time
      0.12 ± 10%      +0.1        0.19 ±  9%  perf-profile.children.cycles-pp.__might_sleep
      0.11 ±  7%      +0.1        0.18 ±  8%  perf-profile.children.cycles-pp.__get_user_4
      0.02 ±173%      +0.1        0.10 ± 17%  perf-profile.children.cycles-pp.ext4_meta_trans_blocks
      0.11 ±  7%      +0.1        0.19 ±  6%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      0.00            +0.1        0.08 ±  5%  perf-profile.children.cycles-pp.dax_direct_access
      0.11 ±  9%      +0.1        0.19 ±  9%  perf-profile.children.cycles-pp.selinux_file_permission
      0.10 ± 12%      +0.1        0.18 ± 10%  perf-profile.children.cycles-pp._copy_to_user
      0.12 ±  7%      +0.1        0.20 ±  7%  perf-profile.children.cycles-pp.entry_SYSCALL_64
      0.12 ±  8%      +0.1        0.22 ±  8%  perf-profile.children.cycles-pp._copy_from_user
      0.15 ±  8%      +0.1        0.26 ±  8%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.13 ± 10%      +0.1        0.24 ±  8%  perf-profile.children.cycles-pp.ext4_es_lookup_extent
      0.15 ± 11%      +0.1        0.26 ±  8%  perf-profile.children.cycles-pp.security_file_permission
      0.18 ± 12%      +0.1        0.30 ±  7%  perf-profile.children.cycles-pp.___might_sleep
      0.18 ± 11%      +0.1        0.32 ±  9%  perf-profile.children.cycles-pp.__might_fault
      0.20 ± 11%      +0.2        0.38 ± 10%  perf-profile.children.cycles-pp.ext4_map_blocks
      0.30 ± 10%      +0.2        0.52 ±  7%  perf-profile.children.cycles-pp.lookup_ioctx
      0.39 ± 10%      +0.3        0.69 ±  7%  perf-profile.children.cycles-pp.aio_read_events
      0.43 ± 10%      +0.3        0.76 ±  7%  perf-profile.children.cycles-pp.read_events
      0.61 ± 10%      +0.4        1.05 ±  7%  perf-profile.children.cycles-pp.do_io_getevents
      0.64 ± 10%      +0.5        1.12 ±  7%  perf-profile.children.cycles-pp.__x64_sys_io_getevents
      0.81 ± 12%      +0.6        1.42 ± 10%  perf-profile.children.cycles-pp.__srcu_read_unlock
      1.02 ± 11%      +0.7        1.74 ±  4%  perf-profile.children.cycles-pp.__copy_user_nocache
      1.03 ± 12%      +0.7        1.75 ±  4%  perf-profile.children.cycles-pp.__copy_user_flushcache
      1.04 ± 11%      +0.7        1.78 ±  4%  perf-profile.children.cycles-pp._copy_from_iter_flushcache
      1.09 ±  7%      +0.8        1.88 ±  7%  perf-profile.children.cycles-pp.__memcpy_mcsafe
      1.12 ±  7%      +0.8        1.92 ±  7%  perf-profile.children.cycles-pp.copyout_mcsafe
      1.16 ±  7%      +0.8        1.98 ±  8%  perf-profile.children.cycles-pp._copy_to_iter_mcsafe
      9.94 ±  8%      +1.7       11.60 ±  7%  perf-profile.children.cycles-pp._raw_read_lock
      3.15 ±  8%      +2.3        5.44 ±  7%  perf-profile.children.cycles-pp.dax_iomap_actor
      7.75 ±  8%      +5.0       12.72 ±  8%  perf-profile.children.cycles-pp.ext4_file_read_iter
      7.98 ±  8%      +5.2       13.14 ±  8%  perf-profile.children.cycles-pp.aio_read
     10.41 ±  8%      +7.5       17.88 ±  7%  perf-profile.children.cycles-pp.jbd2_transaction_committed
     10.46 ±  8%      +7.5       17.98 ±  7%  perf-profile.children.cycles-pp.ext4_set_iomap
     34.40 ±  8%     +21.8       56.16 ±  7%  perf-profile.children.cycles-pp.ext4_iomap_begin
     16.32 ±  9%      -2.8       13.53 ±  8%  perf-profile.self.cycles-pp.start_this_handle
      7.28 ±  9%      -1.4        5.87 ±  6%  perf-profile.self.cycles-pp.add_transaction_credits
      4.12 ±  7%      -0.6        3.50 ±  7%  perf-profile.self.cycles-pp.ext4_journal_check_start
      0.07 ± 12%      +0.0        0.10 ±  5%  perf-profile.self.cycles-pp.aio_complete_rw
      0.06 ± 14%      +0.0        0.09 ± 10%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.08 ± 13%      +0.0        0.12 ±  5%  perf-profile.self.cycles-pp.kmem_cache_alloc
      0.05 ±  8%      +0.0        0.09 ±  8%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.05 ±  8%      +0.0        0.09 ±  8%  perf-profile.self.cycles-pp.put_reqs_available
      0.09 ± 13%      +0.0        0.13 ±  9%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.07 ± 13%      +0.0        0.11 ±  7%  perf-profile.self.cycles-pp.__x64_sys_io_submit
      0.04 ± 58%      +0.0        0.08 ±  5%  perf-profile.self.cycles-pp.__put_user_4
      0.04 ± 57%      +0.0        0.08 ± 10%  perf-profile.self.cycles-pp.do_syscall_64
      0.04 ± 57%      +0.0        0.08 ± 10%  perf-profile.self.cycles-pp.__get_user_8
      0.08 ± 15%      +0.0        0.13 ±  8%  perf-profile.self.cycles-pp.ext4_iomap_begin
      0.07 ± 12%      +0.0        0.12 ±  9%  perf-profile.self.cycles-pp.selinux_file_permission
      0.01 ±173%      +0.0        0.06 ± 11%  perf-profile.self.cycles-pp.fsnotify
      0.01 ±173%      +0.0        0.06 ± 11%  perf-profile.self.cycles-pp.kmem_cache_free
      0.04 ± 58%      +0.0        0.09 ±  7%  perf-profile.self.cycles-pp.ext4_es_lookup_extent
      0.07 ± 14%      +0.1        0.12 ± 12%  perf-profile.self.cycles-pp.__fget
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.copyout_mcsafe
      0.04 ± 57%      +0.1        0.09 ±  7%  perf-profile.self.cycles-pp.__get_reqs_available
      0.03 ±100%      +0.1        0.08 ±  8%  perf-profile.self.cycles-pp.dax_iomap_actor
      0.01 ±173%      +0.1        0.07 ±  7%  perf-profile.self.cycles-pp.__srcu_read_lock
      0.01 ±173%      +0.1        0.07 ± 13%  perf-profile.self.cycles-pp.__virt_addr_valid
      0.00            +0.1        0.06 ±  9%  perf-profile.self.cycles-pp.aio_write
      0.00            +0.1        0.06 ±  9%  perf-profile.self.cycles-pp.ext4_map_blocks
      0.01 ±173%      +0.1        0.07 ±  6%  perf-profile.self.cycles-pp.mutex_unlock
      0.00            +0.1        0.06 ±  7%  perf-profile.self.cycles-pp.refill_reqs_available
      0.00            +0.1        0.06 ± 11%  perf-profile.self.cycles-pp.ext4_file_read_iter
      0.01 ±173%      +0.1        0.07 ± 11%  perf-profile.self.cycles-pp.mutex_lock
      0.04 ± 57%      +0.1        0.10 ± 25%  perf-profile.self.cycles-pp.ext4_set_iomap
      0.00            +0.1        0.06 ±  6%  perf-profile.self.cycles-pp.fput_many
      0.09 ± 11%      +0.1        0.16 ±  7%  perf-profile.self.cycles-pp.iomap_apply
      0.01 ±173%      +0.1        0.08 ± 15%  perf-profile.self.cycles-pp.copy_user_generic_unrolled
      0.11 ±  7%      +0.1        0.17 ±  8%  perf-profile.self.cycles-pp.__might_sleep
      0.00            +0.1        0.07 ± 12%  perf-profile.self.cycles-pp.__x64_sys_io_getevents
      0.00            +0.1        0.07 ± 12%  perf-profile.self.cycles-pp.__might_fault
      0.08 ± 15%      +0.1        0.15 ± 17%  perf-profile.self.cycles-pp.aio_prep_rw
      0.11 ±  7%      +0.1        0.18 ±  6%  perf-profile.self.cycles-pp.__get_user_4
      0.00            +0.1        0.07 ± 20%  perf-profile.self.cycles-pp.ext4_file_write_iter
      0.11 ±  7%      +0.1        0.19 ±  6%  perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      0.10 ±  4%      +0.1        0.18 ±  6%  perf-profile.self.cycles-pp.aio_read_events
      0.12 ±  7%      +0.1        0.20 ±  7%  perf-profile.self.cycles-pp.entry_SYSCALL_64
      0.13 ± 14%      +0.1        0.22 ±  7%  perf-profile.self.cycles-pp.lookup_ioctx
      0.15 ±  8%      +0.1        0.26 ±  8%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.18 ± 11%      +0.1        0.29 ±  8%  perf-profile.self.cycles-pp.___might_sleep
      0.19 ±  6%      +0.2        0.35 ±  7%  perf-profile.self.cycles-pp.io_submit_one
      0.81 ± 12%      +0.6        1.41 ± 10%  perf-profile.self.cycles-pp.__srcu_read_unlock
      1.02 ± 11%      +0.7        1.73 ±  4%  perf-profile.self.cycles-pp.__copy_user_nocache
      1.08 ±  7%      +0.8        1.86 ±  8%  perf-profile.self.cycles-pp.__memcpy_mcsafe
      9.89 ±  9%      +1.6       11.54 ±  7%  perf-profile.self.cycles-pp._raw_read_lock
      6.29 ±  8%      +4.6       10.88 ±  7%  perf-profile.self.cycles-pp.jbd2_transaction_committed


                                                                                
                                  fio.read_bw_MBps                              
                                                                                
  10000 +-+-----------------------------------------------------------------+   
   9000 O-OO O   O  O    O    O O OO                                        |   
        |      O  O   O    OO        O O  O  O O O OO                       |   
   8000 +-+                             O   O                               |   
   7000 +-+                                                                 |   
        |                                                                   |   
   6000 +-+     .+ .+.++.+.++.+.+.++.+.++.+.++.+.+.++.+.++.+.++.+.+. +.+. +.|   
   5000 +-++.+.+  +                                                 +    +  |   
   4000 +-+                                                                 |   
        |                                                                   |   
   3000 +-+                                                                 |   
   2000 +-+                                                                 |   
        |                                                                   |   
   1000 +-+                                                                 |   
      0 +-+------------O----------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                     fio.read_iops                              
                                                                                
  2.5e+06 +-+---------------------------------------------------------------+   
          O OO O OO O OO   OO OO O OO O                                     |   
          |                             OO O OO O OO O                      |   
    2e+06 +-+                                                               |   
          |                                                                 |   
          |                                                                 |   
  1.5e+06 +-++.+. +. .++.+.++.++.+.++.+.++.+.++.+.++.+.++.+.++.++.+. +.+.++.|   
          |.+    +  +                                               +       |   
    1e+06 +-+                                                               |   
          |                                                                 |   
          |                                                                 |   
   500000 +-+                                                               |   
          |                                                                 |   
          |                                                                 |   
        0 +-+------------O--------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                fio.read_clat_mean_us                           
                                                                                
  600000 +-+----------------------------------------------------------------+   
         |.++.+  +.+.                                              + :  .+  |   
  500000 +-+         ++.+.++.+.++.+.++.+.+.++.+.++.+.++.+.++.+.++.+  +.+  +.|   
         |                                                                  |   
         |                                                                  |   
  400000 +-+                                                                |   
         |    O O  O  O    O O  O   OO O O OO O OO O O                      |   
  300000 O-OO    O   O    O    O  O                                         |   
         |                                                                  |   
  200000 +-+                                                                |   
         |                                                                  |   
         |                                                                  |   
  100000 +-+                                                                |   
         |                                                                  |   
       0 +-+------------O---------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                  fio.write_bw_MBps                             
                                                                                
  10000 +-+-----------------------------------------------------------------+   
   9000 O-OO O   O  O    O    O O OO                                        |   
        |      O  O   O    OO        O O  O  O O O OO                       |   
   8000 +-+                             O   O                               |   
   7000 +-+                                                                 |   
        |                                                                   |   
   6000 +-+     .+ .+.++.+.++.+.+.++.+.++.+.++.+.+.++.+.++.+.++.+.+. +.+. +.|   
   5000 +-++.+.+  +                                                 +    +  |   
   4000 +-+                                                                 |   
        |                                                                   |   
   3000 +-+                                                                 |   
   2000 +-+                                                                 |   
        |                                                                   |   
   1000 +-+                                                                 |   
      0 +-+------------O----------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                    fio.write_iops                              
                                                                                
  2.5e+06 +-+---------------------------------------------------------------+   
          O OO O OO O OO   OO OO O OO O                                     |   
          |                             OO O OO O OO O                      |   
    2e+06 +-+                                                               |   
          |                                                                 |   
          |                                                                 |   
  1.5e+06 +-++.+. +. .++.+.++.++.+.++.+.++.+.++.+.++.+.++.+.++.++.+. +.+.++.|   
          |.+    +  +                                               +       |   
    1e+06 +-+                                                               |   
          |                                                                 |   
          |                                                                 |   
   500000 +-+                                                               |   
          |                                                                 |   
          |                                                                 |   
        0 +-+------------O--------------------------------------------------+   
                                                                                
                                                                                                                                                                
                               fio.write_clat_mean_us                           
                                                                                
  600000 +-+----------------------------------------------------------------+   
         |.++.+  +  +                                              + :  .+  |   
  500000 +-+         ++.+.++.+.++.+.++.+.+.++.+.++.+.++.+.++.+.++.+  +.+  +.|   
         |                                                                  |   
         |                                                                  |   
  400000 +-+                                                                |   
         |    O O  O  O    O O  O   OO O O OO O OO O O                      |   
  300000 O-OO    O   O    O    O  O                                         |   
         |                                                                  |   
  200000 +-+                                                                |   
         |                                                                  |   
         |                                                                  |   
  100000 +-+                                                                |   
         |                                                                  |   
       0 +-+------------O---------------------------------------------------+   
                                                                                
                                                                                                                                                                
                               fio.write_slat_mean_us                           
                                                                                
  35000 +-+-----------------------------------------------------------------+   
        |                                                                   |   
  30000 +-+   .+. +.                                               .+   .+  |   
        |  +.+   +  +.++.+.++.+.+.++.+.++.+.++.+.+.++. .++.+.++.+.+  +.+  +.|   
  25000 +-+                                           +                     |   
        |                                                                   |   
  20000 +-+                                                                 |   
        |                               O O OO O    O                       |   
  15000 O-OO O O OO O O  O OO O O OO O O         O O                        |   
        |                                                                   |   
  10000 +-+                                                                 |   
        |                                                                   |   
   5000 +-+                                                                 |   
        |                                                                   |   
      0 +-+------------O----------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                    fio.workload                                
                                                                                
  1e+09 +-+-----------------------------------------------------------------+   
  9e+08 O-OO O O OO O O  O OO O O OO O                                      |   
        |                              OO O OO O O OO                       |   
  8e+08 +-+                                                                 |   
  7e+08 +-+                                                                 |   
        |                                                                   |   
  6e+08 +-++.+. .+ .+.++.+.++.+.+.++.+.++.+.++.+.+.++.+.++.+.++.+.+. +.+.++.|   
  5e+08 +-+    +  +                                                 +       |   
  4e+08 +-+                                                                 |   
        |                                                                   |   
  3e+08 +-+                                                                 |   
  2e+08 +-+                                                                 |   
        |                                                                   |   
  1e+08 +-+                                                                 |   
      0 +-+------------O----------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample

***************************************************************************************************
lkp-csl-2sp6: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 256G memory
=========================================================================================
bs/compiler/cpufreq_governor/disk/fs/ioengine/kconfig/mount_option/nr_task/rootfs/runtime/rw/tbox_group/test_size/testcase/time_based/ucode:
  4k/gcc-7/performance/2pmem/ext4/sync/x86_64-rhel-7.6/dax/50%/debian-x86_64-20191114.cgz/200s/randwrite/lkp-csl-2sp6/200G/fio-basic/tb/0x500002c

commit: 
  b1b4705d54 ("ext4: introduce direct I/O read using iomap infrastructure")
  569342dc24 ("ext4: move inode extension/truncate code out from ->iomap_end() callback")

b1b4705d54abedfd 569342dc2485392e95b6a626281 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :4           25%           1:4     dmesg.WARNING:at#for_ip_interrupt_entry/0x
         %stddev     %change         %stddev
             \          |                \  
      4.08 ± 25%      -3.9        0.15 ± 34%  fio.latency_100us%
      0.81 ± 45%     +15.6       16.45 ± 47%  fio.latency_10us%
     63.62 ±  3%     -27.6       36.07 ± 17%  fio.latency_50us%
    175.73           +50.8%     265.02 ±  2%  fio.time.user_time
 3.401e+08 ±  2%     +54.0%  5.237e+08        fio.workload
      6642 ±  2%     +54.0%      10229        fio.write_bw_MBps
     43392 ±  4%     -36.1%      27712 ±  5%  fio.write_clat_90%_us
     48384 ±  4%     -35.7%      31104 ±  5%  fio.write_clat_95%_us
     59136 ±  3%     -33.3%      39424 ±  5%  fio.write_clat_99%_us
     27760 ±  2%     -35.7%      17839        fio.write_clat_mean_us
     12867 ±  9%     -33.9%       8506 ± 18%  fio.write_clat_stddev
   1700412 ±  2%     +54.0%    2618693        fio.write_iops
      2011            +1.2%       2034        boot-time.idle
      0.94           +48.6%       1.39 ±  2%  iostat.cpu.user
      2972 ±  3%     +21.0%       3597        meminfo.Dirty
     66317 ±  2%     +24.3%      82442 ± 20%  softirqs.CPU48.TIMER
      0.00 ±  9%      -0.0        0.00 ± 24%  mpstat.cpu.all.iowait%
      0.94            +0.5        1.40 ±  2%  mpstat.cpu.all.usr%
     13981 ± 56%     -99.2%     109.75 ± 95%  numa-numastat.node0.other_node
      4771 ±167%    +292.0%      18700        numa-numastat.node1.other_node
    745.50 ±  3%     +21.1%     903.00        proc-vmstat.nr_dirty
    743.75 ±  3%     +21.5%     903.75        proc-vmstat.nr_zone_write_pending
    238649 ± 16%     +28.0%     305583 ±  7%  numa-meminfo.node0.Active
    208040 ± 24%     +37.6%     286262 ±  5%  numa-meminfo.node0.Active(anon)
    229500 ± 26%     -36.3%     146157 ±  7%  numa-meminfo.node1.Active(anon)
  19606483           -18.4%   16003967 ± 20%  turbostat.C1E
     49.52           -12.5       37.06 ± 30%  turbostat.C1E%
    124.19            +3.3%     128.34        turbostat.RAMWatt
   9.6e+09           -25.2%  7.183e+09 ± 30%  cpuidle.C1E.time
  19607054           -18.4%   16005122 ± 20%  cpuidle.C1E.usage
  42226648 ± 44%   +5712.0%  2.454e+09 ± 88%  cpuidle.C6.time
     40856 ± 44%   +8800.0%    3636237 ± 90%  cpuidle.C6.usage
     51953 ± 24%     +37.6%      71503 ±  5%  numa-vmstat.node0.nr_active_anon
     51953 ± 24%     +37.6%      71503 ±  5%  numa-vmstat.node0.nr_zone_active_anon
     14639 ± 52%     -95.4%     678.00 ± 85%  numa-vmstat.node0.numa_other
     57365 ± 26%     -36.3%      36564 ±  7%  numa-vmstat.node1.nr_active_anon
     57365 ± 26%     -36.3%      36564 ±  7%  numa-vmstat.node1.nr_zone_active_anon
    161195 ±  4%      +9.1%     175918        numa-vmstat.node1.numa_other
      1476 ±  3%     -14.7%       1259 ±  4%  slabinfo.ext4_extent_status.active_slabs
    150614 ±  3%     -14.7%     128502 ±  4%  slabinfo.ext4_extent_status.num_objs
      1476 ±  3%     -14.7%       1259 ±  4%  slabinfo.ext4_extent_status.num_slabs
      1096 ±  7%     -14.8%     934.25 ±  7%  slabinfo.khugepaged_mm_slot.active_objs
      1096 ±  7%     -14.8%     934.25 ±  7%  slabinfo.khugepaged_mm_slot.num_objs
    400.00 ± 13%     +28.2%     513.00 ±  8%  slabinfo.skbuff_fclone_cache.active_objs
    400.00 ± 13%     +28.2%     513.00 ±  8%  slabinfo.skbuff_fclone_cache.num_objs
      2785 ±  7%     +14.5%       3188 ±  7%  slabinfo.sock_inode_cache.active_objs
      2785 ±  7%     +14.5%       3188 ±  7%  slabinfo.sock_inode_cache.num_objs
 2.393e+09 ±  2%     +35.2%  3.235e+09        perf-stat.i.branch-instructions
      0.98            -0.0        0.93 ±  2%  perf-stat.i.branch-miss-rate%
  23967630 ±  2%     +28.1%   30704745 ±  3%  perf-stat.i.branch-misses
     10.10 ±  2%     -26.6%       7.41        perf-stat.i.cpi
 4.179e+09 ±  4%     +38.3%  5.779e+09 ±  8%  perf-stat.i.dTLB-loads
 2.663e+09           +36.1%  3.625e+09 ±  7%  perf-stat.i.dTLB-stores
     79.97            +3.0       82.93        perf-stat.i.iTLB-load-miss-rate%
  26249647           +31.2%   34434951 ±  4%  perf-stat.i.iTLB-load-misses
   6552634            +8.0%    7076093 ±  3%  perf-stat.i.iTLB-loads
 1.347e+10 ±  2%     +35.8%   1.83e+10        perf-stat.i.instructions
      0.10 ±  2%     +34.3%       0.14 ±  2%  perf-stat.i.ipc
      1.00            -0.1        0.95        perf-stat.overall.branch-miss-rate%
      9.99 ±  2%     -26.4%       7.35        perf-stat.overall.cpi
      0.00 ± 21%      -0.0        0.00 ± 26%  perf-stat.overall.dTLB-store-miss-rate%
     80.02            +2.9       82.94        perf-stat.overall.iTLB-load-miss-rate%
      0.10 ±  2%     +35.9%       0.14        perf-stat.overall.ipc
      7939           -11.9%       6995        perf-stat.overall.path-length
 2.381e+09 ±  2%     +35.2%  3.218e+09        perf-stat.ps.branch-instructions
  23847364 ±  2%     +28.1%   30544365 ±  3%  perf-stat.ps.branch-misses
 4.157e+09 ±  4%     +38.3%  5.749e+09 ±  8%  perf-stat.ps.dTLB-loads
 2.649e+09           +36.1%  3.606e+09 ±  7%  perf-stat.ps.dTLB-stores
  26117405           +31.2%   34259896 ±  4%  perf-stat.ps.iTLB-load-misses
   6519655            +8.0%    7039502 ±  3%  perf-stat.ps.iTLB-loads
 1.341e+10 ±  2%     +35.8%  1.821e+10        perf-stat.ps.instructions
   2.7e+12 ±  2%     +35.7%  3.664e+12        perf-stat.total.instructions
     57.00 ± 88%     -96.5%       2.00 ±173%  interrupts.55:PCI-MSI.31981588-edge.i40e-eth0-TxRx-19
      1760 ±  4%     +17.3%       2065 ±  4%  interrupts.CPU0.CAL:Function_call_interrupts
      5733           -24.1%       4353 ± 21%  interrupts.CPU0.NMI:Non-maskable_interrupts
      5733           -24.1%       4353 ± 21%  interrupts.CPU0.PMI:Performance_monitoring_interrupts
      1761 ±  5%     +26.4%       2226 ± 10%  interrupts.CPU1.CAL:Function_call_interrupts
     14.75 ± 49%   +2852.5%     435.50 ±111%  interrupts.CPU1.RES:Rescheduling_interrupts
      1734 ±  5%     +17.9%       2044 ± 13%  interrupts.CPU12.CAL:Function_call_interrupts
      1681 ±  6%     +23.1%       2068 ±  8%  interrupts.CPU15.CAL:Function_call_interrupts
      1757 ±  5%     +15.9%       2037 ±  2%  interrupts.CPU2.CAL:Function_call_interrupts
      1829 ±  9%     +19.8%       2191 ±  7%  interrupts.CPU31.CAL:Function_call_interrupts
      1757 ±  4%     +19.6%       2101 ±  9%  interrupts.CPU4.CAL:Function_call_interrupts
      1761 ±  4%     +17.6%       2070 ±  4%  interrupts.CPU5.CAL:Function_call_interrupts
      1753 ±  4%     +18.8%       2082 ±  3%  interrupts.CPU6.CAL:Function_call_interrupts
      6375 ± 25%     -39.4%       3861 ± 30%  interrupts.CPU60.NMI:Non-maskable_interrupts
      6375 ± 25%     -39.4%       3861 ± 30%  interrupts.CPU60.PMI:Performance_monitoring_interrupts
      6.50 ± 93%    +807.7%      59.00 ± 90%  interrupts.CPU65.RES:Rescheduling_interrupts
      6333 ± 25%     -31.0%       4370 ± 46%  interrupts.CPU66.NMI:Non-maskable_interrupts
      6333 ± 25%     -31.0%       4370 ± 46%  interrupts.CPU66.PMI:Performance_monitoring_interrupts
      1754 ±  4%     +16.8%       2050 ±  3%  interrupts.CPU7.CAL:Function_call_interrupts
      6364 ± 25%     -30.9%       4399 ± 46%  interrupts.CPU71.NMI:Non-maskable_interrupts
      6364 ± 25%     -30.9%       4399 ± 46%  interrupts.CPU71.PMI:Performance_monitoring_interrupts
     61.50 ± 77%     -68.3%      19.50 ±108%  interrupts.CPU77.RES:Rescheduling_interrupts
      1750 ±  4%     +17.6%       2058 ±  4%  interrupts.CPU8.CAL:Function_call_interrupts
      6785 ± 15%     -36.5%       4311 ± 23%  interrupts.CPU85.NMI:Non-maskable_interrupts
      6785 ± 15%     -36.5%       4311 ± 23%  interrupts.CPU85.PMI:Performance_monitoring_interrupts
      6262 ± 14%     -27.2%       4557 ± 15%  interrupts.CPU86.NMI:Non-maskable_interrupts
      6262 ± 14%     -27.2%       4557 ± 15%  interrupts.CPU86.PMI:Performance_monitoring_interrupts
      1752 ±  4%     +17.8%       2064 ±  4%  interrupts.CPU9.CAL:Function_call_interrupts
     21.45 ±  8%     -21.4        0.00        perf-profile.calltrace.cycles-pp.ext4_iomap_end.iomap_apply.dax_iomap_rw.ext4_file_write_iter.new_sync_write
     14.50 ±  8%     -14.5        0.00        perf-profile.calltrace.cycles-pp.jbd2__journal_start.ext4_iomap_end.iomap_apply.dax_iomap_rw.ext4_file_write_iter
     14.42 ±  8%     -14.4        0.00        perf-profile.calltrace.cycles-pp.start_this_handle.jbd2__journal_start.ext4_iomap_end.iomap_apply.dax_iomap_rw
     45.47 ±  9%      -8.0       37.44 ± 16%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     45.61 ±  9%      -8.0       37.66 ± 16%  perf-profile.calltrace.cycles-pp.secondary_startup_64
     44.50 ±  9%      -7.6       36.90 ± 16%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
     44.50 ±  9%      -7.6       36.91 ± 16%  perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     44.58 ±  9%      -7.6       37.00 ± 16%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     44.58 ±  9%      -7.6       37.00 ± 16%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
     44.58 ±  9%      -7.6       37.00 ± 16%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
      4.87 ±  8%      -4.9        0.00        perf-profile.calltrace.cycles-pp.__ext4_journal_stop.ext4_iomap_end.iomap_apply.dax_iomap_rw.ext4_file_write_iter
      4.74 ±  8%      -4.7        0.00        perf-profile.calltrace.cycles-pp.jbd2_journal_stop.__ext4_journal_stop.ext4_iomap_end.iomap_apply.dax_iomap_rw
      0.77 ±  7%      +0.5        1.31 ±  5%  perf-profile.calltrace.cycles-pp.__srcu_read_unlock.dax_iomap_actor.iomap_apply.dax_iomap_rw.ext4_file_write_iter
      0.80 ± 12%      +0.9        1.70 ± 26%  perf-profile.calltrace.cycles-pp.__copy_user_nocache.__copy_user_flushcache._copy_from_iter_flushcache.dax_iomap_actor.iomap_apply
      0.81 ± 12%      +0.9        1.71 ± 26%  perf-profile.calltrace.cycles-pp.__copy_user_flushcache._copy_from_iter_flushcache.dax_iomap_actor.iomap_apply.dax_iomap_rw
      0.83 ± 12%      +0.9        1.74 ± 25%  perf-profile.calltrace.cycles-pp._copy_from_iter_flushcache.dax_iomap_actor.iomap_apply.dax_iomap_rw.ext4_file_write_iter
      1.66 ±  6%      +1.5        3.18 ± 16%  perf-profile.calltrace.cycles-pp.dax_iomap_actor.iomap_apply.dax_iomap_rw.ext4_file_write_iter.new_sync_write
      2.29 ±  9%      +1.7        3.98 ± 13%  perf-profile.calltrace.cycles-pp._raw_read_lock.jbd2_transaction_committed.ext4_set_iomap.ext4_iomap_begin.iomap_apply
      2.00 ± 11%      +2.0        3.96 ± 12%  perf-profile.calltrace.cycles-pp.ext4_journal_check_start.__ext4_journal_start_sb.ext4_iomap_begin.iomap_apply.dax_iomap_rw
      2.01 ± 11%      +2.0        4.03 ± 12%  perf-profile.calltrace.cycles-pp.__ext4_journal_start_sb.ext4_iomap_begin.iomap_apply.dax_iomap_rw.ext4_file_write_iter
      2.94 ±  9%      +2.5        5.40 ±  9%  perf-profile.calltrace.cycles-pp._raw_read_lock.start_this_handle.jbd2__journal_start.ext4_iomap_begin.iomap_apply
      3.23 ±  9%      +3.6        6.81 ± 14%  perf-profile.calltrace.cycles-pp.add_transaction_credits.start_this_handle.jbd2__journal_start.ext4_iomap_begin.iomap_apply
      5.56 ±  9%      +4.8       10.40 ± 14%  perf-profile.calltrace.cycles-pp.jbd2_transaction_committed.ext4_set_iomap.ext4_iomap_begin.iomap_apply.dax_iomap_rw
      5.58 ±  9%      +5.0       10.54 ± 13%  perf-profile.calltrace.cycles-pp.ext4_set_iomap.ext4_iomap_begin.iomap_apply.dax_iomap_rw.ext4_file_write_iter
      7.53 ±  9%      +6.4       13.92 ±  6%  perf-profile.calltrace.cycles-pp.jbd2_journal_stop.__ext4_journal_stop.ext4_iomap_begin.iomap_apply.dax_iomap_rw
      7.63 ±  9%      +6.5       14.15 ±  6%  perf-profile.calltrace.cycles-pp.__ext4_journal_stop.ext4_iomap_begin.iomap_apply.dax_iomap_rw.ext4_file_write_iter
     54.07 ±  8%      +7.8       61.91 ± 10%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
     54.10 ±  8%      +7.9       61.97 ± 10%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
     13.99 ±  8%     +13.3       27.25 ± 10%  perf-profile.calltrace.cycles-pp.start_this_handle.jbd2__journal_start.ext4_iomap_begin.iomap_apply.dax_iomap_rw
     14.08 ±  8%     +13.3       27.40 ± 10%  perf-profile.calltrace.cycles-pp.jbd2__journal_start.ext4_iomap_begin.iomap_apply.dax_iomap_rw.ext4_file_write_iter
     29.51 ±  8%     +27.1       56.59 ±  9%  perf-profile.calltrace.cycles-pp.ext4_iomap_begin.iomap_apply.dax_iomap_rw.ext4_file_write_iter.new_sync_write
     21.45 ±  8%     -21.4        0.00        perf-profile.children.cycles-pp.ext4_iomap_end
     45.47 ±  9%      -8.0       37.44 ± 16%  perf-profile.children.cycles-pp.intel_idle
     45.53 ±  9%      -8.0       37.55 ± 16%  perf-profile.children.cycles-pp.cpuidle_enter
     45.53 ±  9%      -8.0       37.55 ± 16%  perf-profile.children.cycles-pp.cpuidle_enter_state
     45.61 ±  9%      -8.0       37.66 ± 16%  perf-profile.children.cycles-pp.do_idle
     45.61 ±  9%      -8.0       37.66 ± 16%  perf-profile.children.cycles-pp.secondary_startup_64
     45.61 ±  9%      -8.0       37.66 ± 16%  perf-profile.children.cycles-pp.cpu_startup_entry
     44.58 ±  9%      -7.6       37.00 ± 16%  perf-profile.children.cycles-pp.start_secondary
      0.07 ±  5%      +0.0        0.09 ± 14%  perf-profile.children.cycles-pp.update_process_times
      0.07 ±  5%      +0.0        0.10 ± 15%  perf-profile.children.cycles-pp.tick_sched_handle
      0.09 ±  4%      +0.0        0.12 ± 15%  perf-profile.children.cycles-pp.tick_sched_timer
      0.06 ± 11%      +0.0        0.10 ±  8%  perf-profile.children.cycles-pp.selinux_file_permission
      0.05 ±  8%      +0.0        0.09 ± 11%  perf-profile.children.cycles-pp.__fdget_pos
      0.07 ± 12%      +0.0        0.11 ± 11%  perf-profile.children.cycles-pp.security_file_permission
      0.04 ± 58%      +0.1        0.10 ± 18%  perf-profile.children.cycles-pp.ext4_llseek
      0.04 ± 58%      +0.1        0.10 ±  9%  perf-profile.children.cycles-pp.entry_SYSCALL_64
      0.09 ±  8%      +0.1        0.15 ±  8%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.01 ±173%      +0.1        0.07 ± 10%  perf-profile.children.cycles-pp.__fget_light
      0.06 ± 58%      +0.1        0.12 ± 36%  perf-profile.children.cycles-pp.ktime_get
      0.00            +0.1        0.08 ± 19%  perf-profile.children.cycles-pp.__sb_start_write
      0.10 ±  4%      +0.1        0.18 ± 11%  perf-profile.children.cycles-pp.ksys_lseek
      0.24 ± 13%      +0.1        0.34 ± 15%  perf-profile.children.cycles-pp.hrtimer_interrupt
      0.06 ±  6%      +0.1        0.16 ± 45%  perf-profile.children.cycles-pp.ext4_es_lookup_extent
      0.28 ± 11%      +0.1        0.39 ± 23%  perf-profile.children.cycles-pp.smp_apic_timer_interrupt
      0.31 ±  9%      +0.1        0.44 ± 21%  perf-profile.children.cycles-pp.apic_timer_interrupt
      0.11 ± 17%      +0.2        0.29 ± 52%  perf-profile.children.cycles-pp.ext4_map_blocks
      0.77 ±  7%      +0.5        1.31 ±  5%  perf-profile.children.cycles-pp.__srcu_read_unlock
      0.80 ± 12%      +0.9        1.71 ± 26%  perf-profile.children.cycles-pp.__copy_user_nocache
      0.81 ± 12%      +0.9        1.71 ± 26%  perf-profile.children.cycles-pp.__copy_user_flushcache
      0.83 ± 12%      +0.9        1.74 ± 25%  perf-profile.children.cycles-pp._copy_from_iter_flushcache
      1.66 ±  6%      +1.5        3.19 ± 16%  perf-profile.children.cycles-pp.dax_iomap_actor
     12.68 ±  8%      +1.6       14.32 ±  6%  perf-profile.children.cycles-pp.__ext4_journal_stop
     12.46 ±  8%      +1.6       14.10 ±  6%  perf-profile.children.cycles-pp.jbd2_journal_stop
      5.56 ±  9%      +4.9       10.41 ± 14%  perf-profile.children.cycles-pp.jbd2_transaction_committed
      5.58 ±  9%      +5.0       10.54 ± 13%  perf-profile.children.cycles-pp.ext4_set_iomap
     54.18 ±  8%      +7.8       62.02 ± 10%  perf-profile.children.cycles-pp.do_syscall_64
     54.21 ±  8%      +7.9       62.07 ± 10%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     29.51 ±  8%     +27.1       56.60 ±  9%  perf-profile.children.cycles-pp.ext4_iomap_begin
     45.47 ±  9%      -8.0       37.44 ± 16%  perf-profile.self.cycles-pp.intel_idle
      0.01 ±173%      +0.1        0.06 ± 17%  perf-profile.self.cycles-pp.___might_sleep
      0.00            +0.1        0.05 ±  8%  perf-profile.self.cycles-pp.do_syscall_64
      0.04 ± 58%      +0.1        0.10 ±  9%  perf-profile.self.cycles-pp.entry_SYSCALL_64
      0.09 ±  8%      +0.1        0.15 ±  8%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.00            +0.1        0.06 ± 14%  perf-profile.self.cycles-pp.selinux_file_permission
      0.01 ±173%      +0.1        0.07 ± 10%  perf-profile.self.cycles-pp.__fget_light
      0.05 ± 58%      +0.1        0.11 ± 33%  perf-profile.self.cycles-pp.ktime_get
      0.00            +0.1        0.06 ± 20%  perf-profile.self.cycles-pp.ext4_es_lookup_extent
      0.03 ±100%      +0.1        0.09 ± 16%  perf-profile.self.cycles-pp.iomap_apply
      0.03 ±100%      +0.1        0.10 ± 18%  perf-profile.self.cycles-pp.ext4_llseek
      0.00            +0.1        0.07 ± 15%  perf-profile.self.cycles-pp.ext4_iomap_begin
      0.02 ±173%      +0.1        0.14 ± 56%  perf-profile.self.cycles-pp.ext4_file_write_iter
      0.76 ±  7%      +0.5        1.30 ±  6%  perf-profile.self.cycles-pp.__srcu_read_unlock
      0.80 ± 12%      +0.9        1.69 ± 26%  perf-profile.self.cycles-pp.__copy_user_nocache
      3.25 ±  9%      +3.1        6.40 ± 14%  perf-profile.self.cycles-pp.jbd2_transaction_committed



***************************************************************************************************
lkp-csl-2sp6: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 256G memory
=========================================================================================
bs/compiler/cpufreq_governor/disk/fs/ioengine/kconfig/mount_option/nr_task/rootfs/runtime/rw/tbox_group/test_size/testcase/time_based/ucode:
  4k/gcc-7/performance/2pmem/ext4/libaio/x86_64-rhel-7.6/dax/50%/debian-x86_64-20191114.cgz/200s/randwrite/lkp-csl-2sp6/200G/fio-basic/tb/0x500002c

commit: 
  b1b4705d54 ("ext4: introduce direct I/O read using iomap infrastructure")
  569342dc24 ("ext4: move inode extension/truncate code out from ->iomap_end() callback")

b1b4705d54abedfd 569342dc2485392e95b6a626281 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          1:4          -25%            :4     dmesg.WARNING:at_ip__x64_sys_io_submit/0x
           :4           25%           1:4     dmesg.WARNING:at_ip_aio_write/0x
         %stddev     %change         %stddev
             \          |                \  
     22.50 ± 85%     -16.5        6.00 ±150%  fio.latency_1000us%
      0.01            +3.3        3.29 ±158%  fio.latency_250us%
     43.58 ± 27%     -43.5        0.04 ±104%  fio.latency_2ms%
     22.38 ± 16%     +40.0       62.34 ± 24%  fio.latency_750us%
      9422            -1.2%       9304        fio.time.system_time
    185.76 ±  3%     +63.1%     303.05 ±  2%  fio.time.user_time
 3.191e+08           +68.8%  5.384e+08        fio.workload
      6231           +68.8%      10516        fio.write_bw_MBps
   1327104 ±  7%     -46.3%     712704 ± 10%  fio.write_clat_90%_us
   1380352 ±  6%     -46.6%     737280 ± 10%  fio.write_clat_95%_us
   1503232 ±  7%     -47.4%     790528 ± 10%  fio.write_clat_99%_us
    933218           -40.7%     553062        fio.write_clat_mean_us
    310780 ± 29%     -57.8%     131049 ± 33%  fio.write_clat_stddev
   1595317           +68.8%    2692113        fio.write_iops
     29413           -41.6%      17164        fio.write_slat_mean_us
     14463 ± 30%     -52.9%       6814 ± 13%  fio.write_slat_stddev
   7159296 ± 12%     -11.5%    6334976        meminfo.DirectMap2M
     21181            -1.0%      20962        proc-vmstat.nr_slab_reclaimable
     48.55            -1.4%      47.88        iostat.cpu.system
      0.99 ±  3%     +60.9%       1.59 ±  2%  iostat.cpu.user
      0.00 ± 17%      -0.0        0.00 ± 24%  mpstat.cpu.all.iowait%
      0.99 ±  3%      +0.6        1.60 ±  2%  mpstat.cpu.all.usr%
    383524 ± 37%     -42.5%     220431 ± 57%  numa-numastat.node1.local_node
    392957 ± 35%     -40.3%     234425 ± 50%  numa-numastat.node1.numa_hit
    248.01            +2.5%     254.11        turbostat.PkgWatt
    124.91            +3.3%     129.03        turbostat.RAMWatt
     40962 ± 10%     -16.3%      34274 ±  3%  numa-vmstat.node0.nr_anon_pages
     26868 ± 16%     +25.4%      33684 ±  3%  numa-vmstat.node1.nr_anon_pages
      8600 ± 12%     +17.7%      10124 ± 10%  numa-vmstat.node1.nr_slab_reclaimable
    163884 ± 10%     -16.3%     137230 ±  3%  numa-meminfo.node0.AnonPages
    138488 ± 14%     +22.8%     170029        numa-meminfo.node1.Active
     67242 ± 22%     +25.5%      84358 ±  6%  numa-meminfo.node1.AnonHugePages
    107464 ± 16%     +25.2%     134574 ±  3%  numa-meminfo.node1.AnonPages
     34404 ± 12%     +17.7%      40494 ± 10%  numa-meminfo.node1.KReclaimable
     34404 ± 12%     +17.7%      40494 ± 10%  numa-meminfo.node1.SReclaimable
     70842 ±  6%      -7.7%      65369 ±  3%  softirqs.CPU3.TIMER
      8500 ±105%    +128.3%      19405 ± 45%  softirqs.CPU35.SCHED
      8786 ±103%    +123.1%      19604 ± 46%  softirqs.CPU40.SCHED
      8981 ± 98%    +116.0%      19402 ± 44%  softirqs.CPU42.SCHED
     68254 ±  5%      +6.0%      72342 ±  4%  softirqs.CPU44.TIMER
     65323 ±  3%     +10.5%      72152 ±  2%  softirqs.CPU58.TIMER
     19561 ± 49%     -71.6%       5557 ± 91%  softirqs.CPU82.SCHED
      1744 ±  6%     +23.5%       2154 ±  3%  slabinfo.UNIX.active_objs
      1744 ±  6%     +23.5%       2154 ±  3%  slabinfo.UNIX.num_objs
    111978 ±  4%     -17.4%      92440 ±  4%  slabinfo.ext4_extent_status.active_objs
      1416 ±  4%     -22.0%       1104 ±  4%  slabinfo.ext4_extent_status.active_slabs
    144495 ±  4%     -22.0%     112722 ±  4%  slabinfo.ext4_extent_status.num_objs
      1416 ±  4%     -22.0%       1104 ±  4%  slabinfo.ext4_extent_status.num_slabs
      2737 ±  4%     +20.5%       3298 ±  4%  slabinfo.sock_inode_cache.active_objs
      2737 ±  4%     +20.5%       3298 ±  4%  slabinfo.sock_inode_cache.num_objs
    535573 ±  2%      +8.1%     579164 ±  3%  sched_debug.cfs_rq:/.load.avg
    516.79 ±  2%      +7.8%     557.21 ±  3%  sched_debug.cfs_rq:/.runnable_load_avg.avg
    535510 ±  2%      +8.1%     579108 ±  3%  sched_debug.cfs_rq:/.runnable_weight.avg
      0.01 ± 24%     -50.0%       0.00 ± 24%  sched_debug.cpu.nr_uninterruptible.avg
     14089 ±  4%     -21.0%      11124 ± 12%  sched_debug.cpu.sched_count.max
      2552 ±  9%     -13.5%       2208 ± 12%  sched_debug.cpu.sched_count.stddev
      6997 ±  5%     -21.1%       5521 ± 12%  sched_debug.cpu.sched_goidle.max
      1261 ±  8%     -13.7%       1089 ± 11%  sched_debug.cpu.sched_goidle.stddev
      5749 ±  5%     -23.7%       4386 ± 13%  sched_debug.cpu.ttwu_local.max
     23.50 ± 66%    +293.6%      92.50 ± 39%  interrupts.CPU12.TLB:TLB_shootdowns
      4.75 ± 63%   +3384.2%     165.50 ±162%  interrupts.CPU13.RES:Rescheduling_interrupts
      2382 ± 14%     -24.1%       1809 ± 10%  interrupts.CPU32.CAL:Function_call_interrupts
      2351 ± 11%     -19.1%       1901 ±  9%  interrupts.CPU36.CAL:Function_call_interrupts
      2259 ± 11%     -16.2%       1894 ±  8%  interrupts.CPU38.CAL:Function_call_interrupts
      2428 ± 12%     -17.6%       2001 ± 12%  interrupts.CPU60.CAL:Function_call_interrupts
      5774 ± 25%     -48.5%       2973 ± 56%  interrupts.CPU7.NMI:Non-maskable_interrupts
      5774 ± 25%     -48.5%       2973 ± 56%  interrupts.CPU7.PMI:Performance_monitoring_interrupts
     80.50 ± 38%     +60.2%     129.00 ± 20%  interrupts.CPU73.TLB:TLB_shootdowns
     54.50 ± 84%     -83.9%       8.75 ±100%  interrupts.CPU83.RES:Rescheduling_interrupts
     91.75 ±100%     -86.4%      12.50 ±122%  interrupts.CPU89.RES:Rescheduling_interrupts
     18.16 ± 13%     -36.0%      11.62 ±  5%  perf-stat.i.MPKI
 2.855e+09           +51.8%  4.333e+09        perf-stat.i.branch-instructions
      0.80            -0.1        0.73        perf-stat.i.branch-miss-rate%
  23116045           +37.2%   31705911        perf-stat.i.branch-misses
      8.56           -34.6%       5.59        perf-stat.i.cpi
 4.579e+09           +66.3%  7.613e+09 ± 10%  perf-stat.i.dTLB-loads
 3.192e+09 ±  2%     +45.8%  4.654e+09 ±  8%  perf-stat.i.dTLB-stores
     81.32            +2.0       83.28        perf-stat.i.iTLB-load-miss-rate%
  28448462 ±  4%     +18.0%   33581121 ±  3%  perf-stat.i.iTLB-load-misses
 1.588e+10           +52.2%  2.417e+10        perf-stat.i.instructions
    573.58 ±  3%     +27.9%     733.82 ±  4%  perf-stat.i.instructions-per-iTLB-miss
      0.12           +50.6%       0.18        perf-stat.i.ipc
  14520205           -10.4%   13007330 ± 13%  perf-stat.i.node-store-misses
     17811 ±  3%     +21.2%      21596 ±  3%  perf-stat.i.node-stores
     18.03 ± 13%     -35.8%      11.57 ±  5%  perf-stat.overall.MPKI
      0.81            -0.1        0.73        perf-stat.overall.branch-miss-rate%
      8.48           -34.4%       5.56        perf-stat.overall.cpi
     81.36            +2.0       83.34        perf-stat.overall.iTLB-load-miss-rate%
    559.06 ±  3%     +29.0%     721.05 ±  4%  perf-stat.overall.instructions-per-iTLB-miss
      0.12           +52.4%       0.18        perf-stat.overall.ipc
      9977            -9.8%       8996        perf-stat.overall.path-length
  2.84e+09           +51.8%  4.311e+09        perf-stat.ps.branch-instructions
  23003415           +37.1%   31544372        perf-stat.ps.branch-misses
 4.556e+09           +66.2%  7.574e+09 ± 10%  perf-stat.ps.dTLB-loads
 3.176e+09 ±  2%     +45.8%   4.63e+09 ±  8%  perf-stat.ps.dTLB-stores
  28304001 ±  4%     +18.0%   33410816 ±  3%  perf-stat.ps.iTLB-load-misses
  1.58e+10           +52.2%  2.405e+10        perf-stat.ps.instructions
  14446128           -10.4%   12941633 ± 13%  perf-stat.ps.node-store-misses
     17748 ±  3%     +21.0%      21481 ±  3%  perf-stat.ps.node-stores
 3.184e+12           +52.1%  4.844e+12        perf-stat.total.instructions
     24.52 ±  9%     -24.5        0.00        perf-profile.calltrace.cycles-pp.ext4_iomap_end.iomap_apply.dax_iomap_rw.ext4_file_write_iter.aio_write
     17.04 ± 10%     -17.0        0.00        perf-profile.calltrace.cycles-pp.jbd2__journal_start.ext4_iomap_end.iomap_apply.dax_iomap_rw.ext4_file_write_iter
     16.95 ± 10%     -16.9        0.00        perf-profile.calltrace.cycles-pp.start_this_handle.jbd2__journal_start.ext4_iomap_end.iomap_apply.dax_iomap_rw
      5.24 ±  7%      -5.2        0.00        perf-profile.calltrace.cycles-pp.__ext4_journal_stop.ext4_iomap_end.iomap_apply.dax_iomap_rw.ext4_file_write_iter
      5.09 ±  7%      -5.1        0.00        perf-profile.calltrace.cycles-pp.jbd2_journal_stop.__ext4_journal_stop.ext4_iomap_end.iomap_apply.dax_iomap_rw
      0.00            +0.6        0.58 ±  8%  perf-profile.calltrace.cycles-pp.do_io_getevents.__x64_sys_io_getevents.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +0.6        0.62 ±  8%  perf-profile.calltrace.cycles-pp.__x64_sys_io_getevents.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.97 ±  5%      +0.6        1.60 ± 22%  perf-profile.calltrace.cycles-pp.__srcu_read_unlock.dax_iomap_actor.iomap_apply.dax_iomap_rw.ext4_file_write_iter
      1.13 ± 12%      +1.0        2.08 ± 12%  perf-profile.calltrace.cycles-pp.__copy_user_nocache.__copy_user_flushcache._copy_from_iter_flushcache.dax_iomap_actor.iomap_apply
      1.14 ± 12%      +1.0        2.09 ± 12%  perf-profile.calltrace.cycles-pp.__copy_user_flushcache._copy_from_iter_flushcache.dax_iomap_actor.iomap_apply.dax_iomap_rw
      1.16 ± 12%      +1.0        2.12 ± 12%  perf-profile.calltrace.cycles-pp._copy_from_iter_flushcache.dax_iomap_actor.iomap_apply.dax_iomap_rw.ext4_file_write_iter
      2.20 ±  8%      +1.7        3.86 ±  7%  perf-profile.calltrace.cycles-pp.dax_iomap_actor.iomap_apply.dax_iomap_rw.ext4_file_write_iter.aio_write
      2.18 ± 12%      +1.7        3.90 ±  7%  perf-profile.calltrace.cycles-pp._raw_read_lock.jbd2_transaction_committed.ext4_set_iomap.ext4_iomap_begin.iomap_apply
      2.17 ± 12%      +1.7        3.89 ±  7%  perf-profile.calltrace.cycles-pp.ext4_journal_check_start.__ext4_journal_start_sb.ext4_iomap_begin.iomap_apply.dax_iomap_rw
      2.19 ± 12%      +1.7        3.93 ±  7%  perf-profile.calltrace.cycles-pp.__ext4_journal_start_sb.ext4_iomap_begin.iomap_apply.dax_iomap_rw.ext4_file_write_iter
      3.46 ± 10%      +1.9        5.32 ± 11%  perf-profile.calltrace.cycles-pp._raw_read_lock.start_this_handle.jbd2__journal_start.ext4_iomap_begin.iomap_apply
      4.08 ± 10%      +2.6        6.69 ±  7%  perf-profile.calltrace.cycles-pp.add_transaction_credits.start_this_handle.jbd2__journal_start.ext4_iomap_begin.iomap_apply
      9.21 ±  9%      +4.5       13.74 ± 12%  perf-profile.calltrace.cycles-pp.jbd2_journal_stop.__ext4_journal_stop.ext4_iomap_begin.iomap_apply.dax_iomap_rw
      5.56 ± 12%      +4.6       10.18 ±  7%  perf-profile.calltrace.cycles-pp.jbd2_transaction_committed.ext4_set_iomap.ext4_iomap_begin.iomap_apply.dax_iomap_rw
      5.59 ± 12%      +4.7       10.25 ±  8%  perf-profile.calltrace.cycles-pp.ext4_set_iomap.ext4_iomap_begin.iomap_apply.dax_iomap_rw.ext4_file_write_iter
      9.32 ±  9%      +4.7       14.00 ± 12%  perf-profile.calltrace.cycles-pp.__ext4_journal_stop.ext4_iomap_begin.iomap_apply.dax_iomap_rw.ext4_file_write_iter
     16.80 ± 10%     +10.3       27.15 ± 10%  perf-profile.calltrace.cycles-pp.start_this_handle.jbd2__journal_start.ext4_iomap_begin.iomap_apply.dax_iomap_rw
     16.89 ± 10%     +10.4       27.30 ± 10%  perf-profile.calltrace.cycles-pp.jbd2__journal_start.ext4_iomap_begin.iomap_apply.dax_iomap_rw.ext4_file_write_iter
     34.20 ± 10%     +21.7       55.87 ± 10%  perf-profile.calltrace.cycles-pp.ext4_iomap_begin.iomap_apply.dax_iomap_rw.ext4_file_write_iter.aio_write
     24.52 ±  9%     -24.5        0.00        perf-profile.children.cycles-pp.ext4_iomap_end
     34.47 ± 10%      -6.7       27.80 ± 10%  perf-profile.children.cycles-pp.jbd2__journal_start
     34.30 ± 10%      -6.7       27.65 ± 10%  perf-profile.children.cycles-pp.start_this_handle
      8.29 ± 11%      -1.5        6.81 ±  7%  perf-profile.children.cycles-pp.add_transaction_credits
      0.05 ±  8%      +0.0        0.07 ± 12%  perf-profile.children.cycles-pp._cond_resched
      0.07 ±  7%      +0.0        0.10 ± 13%  perf-profile.children.cycles-pp.selinux_file_permission
      0.08 ± 10%      +0.0        0.12 ±  7%  perf-profile.children.cycles-pp.__might_sleep
      0.07 ± 14%      +0.0        0.11 ± 11%  perf-profile.children.cycles-pp.security_file_permission
      0.03 ±100%      +0.0        0.07 ± 12%  perf-profile.children.cycles-pp.__fget
      0.06 ± 14%      +0.0        0.10 ±  7%  perf-profile.children.cycles-pp._copy_to_user
      0.07 ±  7%      +0.0        0.11 ±  6%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      0.06 ± 13%      +0.0        0.11 ±  7%  perf-profile.children.cycles-pp.__get_user_4
      0.04 ± 57%      +0.0        0.09 ±  8%  perf-profile.children.cycles-pp.aio_prep_rw
      0.07 ±  7%      +0.0        0.11 ±  9%  perf-profile.children.cycles-pp.entry_SYSCALL_64
      0.03 ±100%      +0.1        0.08 ± 14%  perf-profile.children.cycles-pp.__sb_start_write
      0.00            +0.1        0.05 ±  9%  perf-profile.children.cycles-pp.put_reqs_available
      0.00            +0.1        0.05 ±  9%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.04 ± 58%      +0.1        0.10 ± 18%  perf-profile.children.cycles-pp.ext4_meta_trans_blocks
      0.01 ±173%      +0.1        0.07 ± 12%  perf-profile.children.cycles-pp.__check_object_size
      0.07 ± 11%      +0.1        0.13 ±  9%  perf-profile.children.cycles-pp._copy_from_user
      0.09 ± 13%      +0.1        0.14 ± 13%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.07 ± 17%      +0.1        0.12 ± 13%  perf-profile.children.cycles-pp.ext4_es_lookup_extent
      0.12 ± 11%      +0.1        0.18 ±  7%  perf-profile.children.cycles-pp.___might_sleep
      0.00            +0.1        0.06 ± 13%  perf-profile.children.cycles-pp.mutex_lock
      0.10 ± 11%      +0.1        0.17 ±  9%  perf-profile.children.cycles-pp.__might_fault
      0.11 ± 17%      +0.1        0.21 ± 17%  perf-profile.children.cycles-pp.ext4_map_blocks
      0.18 ± 13%      +0.1        0.30 ±  9%  perf-profile.children.cycles-pp.lookup_ioctx
      0.22 ±  9%      +0.2        0.38 ±  8%  perf-profile.children.cycles-pp.aio_read_events
      0.25 ±  9%      +0.2        0.42 ±  8%  perf-profile.children.cycles-pp.read_events
      0.35 ±  9%      +0.2        0.58 ±  8%  perf-profile.children.cycles-pp.do_io_getevents
      0.37 ± 10%      +0.3        0.63 ±  8%  perf-profile.children.cycles-pp.__x64_sys_io_getevents
      0.97 ±  5%      +0.6        1.60 ± 22%  perf-profile.children.cycles-pp.__srcu_read_unlock
      1.14 ± 12%      +1.0        2.09 ± 12%  perf-profile.children.cycles-pp.__copy_user_flushcache
      1.13 ± 12%      +1.0        2.08 ± 12%  perf-profile.children.cycles-pp.__copy_user_nocache
      1.16 ± 12%      +1.0        2.12 ± 12%  perf-profile.children.cycles-pp._copy_from_iter_flushcache
      2.20 ±  8%      +1.7        3.87 ±  7%  perf-profile.children.cycles-pp.dax_iomap_actor
      5.56 ± 12%      +4.6       10.19 ±  7%  perf-profile.children.cycles-pp.jbd2_transaction_committed
      5.59 ± 12%      +4.7       10.26 ±  8%  perf-profile.children.cycles-pp.ext4_set_iomap
     34.20 ± 10%     +21.7       55.87 ± 10%  perf-profile.children.cycles-pp.ext4_iomap_begin
      8.26 ± 10%      -1.5        6.79 ±  7%  perf-profile.self.cycles-pp.add_transaction_credits
      0.08 ±  6%      +0.0        0.11 ±  4%  perf-profile.self.cycles-pp.__might_sleep
      0.05            +0.0        0.09 ±  9%  perf-profile.self.cycles-pp.iomap_apply
      0.06 ± 13%      +0.0        0.10 ± 10%  perf-profile.self.cycles-pp.__get_user_4
      0.03 ±100%      +0.0        0.07 ± 12%  perf-profile.self.cycles-pp.__fget
      0.07 ±  7%      +0.0        0.11 ±  6%  perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      0.03 ±100%      +0.0        0.07 ± 14%  perf-profile.self.cycles-pp.__x64_sys_io_submit
      0.05 ±  9%      +0.0        0.10 ±  7%  perf-profile.self.cycles-pp.aio_read_events
      0.08 ± 13%      +0.0        0.13 ± 12%  perf-profile.self.cycles-pp.lookup_ioctx
      0.04 ± 57%      +0.0        0.09 ±  8%  perf-profile.self.cycles-pp.aio_prep_rw
      0.07 ±  7%      +0.0        0.11 ±  9%  perf-profile.self.cycles-pp.entry_SYSCALL_64
      0.00            +0.1        0.05 ±  9%  perf-profile.self.cycles-pp.aio_write
      0.00            +0.1        0.05 ±  9%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.00            +0.1        0.05 ±  9%  perf-profile.self.cycles-pp.put_reqs_available
      0.09 ± 13%      +0.1        0.14 ± 13%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.12 ± 11%      +0.1        0.18 ±  8%  perf-profile.self.cycles-pp.___might_sleep
      0.01 ±173%      +0.1        0.07 ± 14%  perf-profile.self.cycles-pp.ext4_iomap_begin
      0.11 ± 10%      +0.1        0.18 ±  8%  perf-profile.self.cycles-pp.io_submit_one
      0.03 ±102%      +0.1        0.11 ± 17%  perf-profile.self.cycles-pp.ext4_file_write_iter
      0.97 ±  5%      +0.6        1.59 ± 22%  perf-profile.self.cycles-pp.__srcu_read_unlock
      1.13 ± 12%      +0.9        2.07 ± 12%  perf-profile.self.cycles-pp.__copy_user_nocache
      3.37 ± 12%      +2.9        6.25 ±  8%  perf-profile.self.cycles-pp.jbd2_transaction_committed



***************************************************************************************************
lkp-hsw-d01: 8 threads Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz with 8G memory
=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/rootfs/tbox_group/test/testcase/ucode:
  gcc-7/performance/1HDD/ext4/x86_64-rhel-7.6/debian-x86_64-2019-11-14.cgz/lkp-hsw-d01/fivestreamreaddirect.f/filebench/0x27

commit: 
  b1b4705d54 ("ext4: introduce direct I/O read using iomap infrastructure")
  569342dc24 ("ext4: move inode extension/truncate code out from ->iomap_end() callback")

b1b4705d54abedfd 569342dc2485392e95b6a626281 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          1:4          -25%            :2     dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x
          1:4          -25%            :2     dmesg.WARNING:stack_recursion
          1:4          -25%            :2     kmsg.addr:#vm_flags:#anon_vma:#mapping:#index:da
          1:4          -25%            :2     kmsg.addr:#vm_flags:#anon_vma:#mapping:#index:db
          1:4          -25%            :2     kmsg.addr:#vm_flags:#anon_vma:#mapping:#index:dc
          1:4          -25%            :2     kmsg.addr:#vm_flags:#anon_vma:#mapping:#index:dd
          1:4          -25%            :2     kmsg.addr:#vm_flags:#anon_vma:#mapping:#index:de
          1:4          -25%            :2     kmsg.addr:#vm_flags:#anon_vma:#mapping:#index:df
          1:4          -25%            :2     kmsg.addr:#vm_flags:#anon_vma:#mapping:#index:ea
          1:4          -25%            :2     kmsg.addr:#vm_flags:#anon_vma:#mapping:#index:eb
          1:4          -25%            :2     kmsg.addr:#vm_flags:#anon_vma:#mapping:#index:ec
          1:4          -25%            :2     kmsg.addr:#vm_flags:#anon_vma:#mapping:#index:ed
          1:4          -25%            :2     kmsg.addr:#vm_flags:#anon_vma:#mapping:#index:ee
          1:4          -25%            :2     kmsg.addr:#vm_flags:#anon_vma:#mapping:#index:ef
          1:4          -25%            :2     kmsg.addr:#vm_flags:#anon_vma:#mapping:#index:fa
          1:4          -25%            :2     kmsg.addr:#vm_flags:#anon_vma:#mapping:#index:fb
          1:4          -25%            :2     kmsg.addr:#vm_flags:#anon_vma:#mapping:#index:fc
          1:4          -25%            :2     kmsg.addr:#vm_flags:#anon_vma:#mapping:#index:fd
          1:4          -25%            :2     kmsg.addr:#vm_flags:#anon_vma:#mapping:#index:fe
          1:4          -25%            :2     kmsg.addr:#vm_flags:#anon_vma:#mapping:#index:ff
          1:4          -25%            :2     kmsg.file:libnss_files-#so_fault:filemap_fault_mmap:generic_file_mmap_readpage:simple_readpage
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#a07fffffa0ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#a17fffffa1ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#a27fffffa2ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#a37fffffa3ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#a47fffffa4ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#a57fffffa5ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#a67fffffa6ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#a77fffffa7ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#a7fffff8aff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#a7fffff9aff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#a87fffffa8ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#a97fffffa9ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#aa7fffffaaff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#ab7fffffabff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#ac7fffffacff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#ad7fffffadff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#ae7fffffaeff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#af7fffffafff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#b07fffffb0ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#b17fffffb1ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#b27fffffb2ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#b37fffffb3ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#b47fffffb4ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#b57fffffb5ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#b67fffffb6ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#b77fffffb7ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#b7fffff8bff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#b7fffff9bff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#b87fffffb8ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#b97fffffb9ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#ba7fffffbaff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#bb7fffffbbff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#bc7fffffbcff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#bd7fffffbdff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#be7fffffbeff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#bf7fffffbfff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#c07fffffc0ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#c17fffffc1ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#c27fffffc2ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#c37fffffc3ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#c47fffffc4ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#c57fffffc5ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#c67fffffc6ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#c77fffffc7ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#c7fffff8cff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#c7fffff9cff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#c87fffffc8ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#c97fffffc9ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#ca7fffffcaff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#cb7fffffcbff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#cc7fffffccff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#cd7fffffcdff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#ce7fffffceff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#cf7fffffcfff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#d07fffffd0ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#d17fffffd1ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#d27fffffd2ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#d37fffffd3ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#d47fffffd4ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#d57fffffd5ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#d67fffffd6ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#d77fffffd7ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#d7fffff8dff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#d7fffff9dff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#d87fffffd8ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#d97fffffd9ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#da7fffffdaff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#db7fffffdbff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#dc7fffffdcff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#dd7fffffddff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#de7fffffdeff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#df7fffffdfff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#e07fffffe0ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#e17fffffe1ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#e27fffffe2ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#e37fffffe3ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#e47fffffe4ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#e57fffffe5ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#e67fffffe6ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#e77fffffe7ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#e7fffff8eff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#e7fffff9eff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#e87fffffe8ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#e97fffffe9ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#ea7fffffeaff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#eb7fffffebff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#ec7fffffecff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#ed7fffffedff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#ee7fffffeeff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#ef7fffffefff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#f07ffffff0ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#f17ffffff1ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#f27ffffff2ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#f37ffffff3ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#f47ffffff4ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#f57ffffff5ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#f67ffffff6ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#f77ffffff7ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#f7fffff8fff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#f7fffff9fff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#f87ffffff8ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#f97ffffff9ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#fa7ffffffaff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#fb7ffffffbff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#fc7ffffffcff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#fd7ffffffdff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#fe7ffffffeff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#ff7fffffffff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#fffff80ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#fffff81ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#fffff82ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#fffff83ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#fffff84ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#fffff85ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#fffff86ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#fffff87ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#fffff88ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#fffff89ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#fffff90ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#fffff91ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#fffff92ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#fffff93ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#fffff94ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#fffff95ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#fffff96ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#fffff97ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#fffff98ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#fffff99ff
          1:4          -25%            :2     kmsg.swap_info_get:Bad_swap_file_entry#fffffffffff
         %stddev     %change         %stddev
             \          |                \  
    219770 ± 63%     +70.4%     374538 ±  3%  cpuidle.C7s.usage
    219674 ± 63%     +70.5%     374616 ±  3%  turbostat.C7s
     49.50 ±  3%      +9.1%      54.00        vmstat.cpu.id
     47.75 ±  2%      -7.9%      44.00 ±  2%  vmstat.cpu.wa
     50.18 ±  3%      +8.5%      54.43        iostat.cpu.idle
     48.35 ±  2%      -7.1%      44.91 ±  2%  iostat.cpu.iowait
      0.87 ±137%     -79.7%       0.18        iostat.cpu.user
      6679 ± 14%     -29.6%       4701 ±  4%  slabinfo.anon_vma_chain.active_objs
      7047 ± 12%     -23.5%       5387 ±  2%  slabinfo.anon_vma_chain.num_objs
      2794 ±  4%      +9.1%       3049 ±  2%  slabinfo.filp.active_objs
    332426 ± 16%     +21.2%     402873 ±  3%  interrupts.CPU0.LOC:Local_timer_interrupts
    169.50 ± 15%    +134.8%     398.00 ± 45%  interrupts.CPU0.NMI:Non-maskable_interrupts
    169.50 ± 15%    +134.8%     398.00 ± 45%  interrupts.CPU0.PMI:Performance_monitoring_interrupts
    204.75 ± 21%     -48.7%     105.00        interrupts.CPU0.RES:Rescheduling_interrupts
    307421 ±  3%     +13.9%     350098 ±  2%  interrupts.CPU2.LOC:Local_timer_interrupts
    158.25 ± 54%    +137.6%     376.00 ± 61%  interrupts.CPU4.RES:Rescheduling_interrupts
    786.00 ± 30%     +26.4%     993.50 ±  2%  interrupts.CPU5.CAL:Function_call_interrupts
      1588 ± 14%     +30.7%       2076        interrupts.RES:Rescheduling_interrupts
    331.32 ± 13%     -27.3%     240.78        sched_debug.cfs_rq:/.exec_clock.min
      4972 ±  5%     -13.2%       4317 ±  3%  sched_debug.cfs_rq:/.min_vruntime.min
      0.45 ± 17%     -51.7%       0.22 ± 42%  sched_debug.cfs_rq:/.nr_spread_over.avg
      2.00           -50.0%       1.00        sched_debug.cfs_rq:/.nr_spread_over.max
      0.68 ±  4%     -44.4%       0.38 ± 12%  sched_debug.cfs_rq:/.nr_spread_over.stddev
    564589 ±  4%      +4.8%     591417 ±  3%  sched_debug.cpu.avg_idle.avg
     13.12 ± 19%     +21.9%      16.00 ±  9%  sched_debug.cpu.nr_uninterruptible.max
    133870 ±  6%     -16.3%     112088 ± 10%  sched_debug.cpu.ttwu_count.max
     15029 ±  8%     -33.4%      10005 ±  7%  sched_debug.cpu.ttwu_count.min
    132827 ±  6%     -16.0%     111524 ± 10%  sched_debug.cpu.ttwu_local.max
     10470 ±  9%     -27.5%       7589 ± 14%  sched_debug.cpu.ttwu_local.min
     10.17 ±  2%      +0.3       10.48        perf-stat.i.cache-miss-rate%
    558901 ±  2%      +6.4%     594670 ±  3%  perf-stat.i.cache-misses
   5172535            +1.8%    5265269        perf-stat.i.cache-references
      2.40            +3.8%       2.49 ±  2%  perf-stat.i.cpi
     14.90 ±  6%      +9.7%      16.35 ±  3%  perf-stat.i.cpu-migrations
 1.115e+08 ±  2%      +3.3%  1.152e+08        perf-stat.i.dTLB-loads
     62749 ±  3%      +8.0%      67738 ±  4%  perf-stat.i.dTLB-store-misses
     10.81 ±  2%      +0.5       11.29 ±  3%  perf-stat.overall.cache-miss-rate%
      1.82 ±  2%      +4.2%       1.89        perf-stat.overall.cpi
      0.07 ±  3%      +0.0        0.08 ±  3%  perf-stat.overall.dTLB-store-miss-rate%
      0.55 ±  2%      -4.1%       0.53        perf-stat.overall.ipc
    553610 ±  2%      +6.4%     589043 ±  3%  perf-stat.ps.cache-misses
   5123552            +1.8%    5215383        perf-stat.ps.cache-references
     14.76 ±  6%      +9.7%      16.20 ±  3%  perf-stat.ps.cpu-migrations
 1.104e+08 ±  2%      +3.3%  1.141e+08        perf-stat.ps.dTLB-loads
     62154 ±  3%      +8.0%      67096 ±  4%  perf-stat.ps.dTLB-store-misses
      0.68 ±104%      +0.8        1.53 ± 26%  perf-profile.calltrace.cycles-pp.free_unref_page_list.release_pages.__pagevec_release.truncate_inode_pages_range.ext4_evict_inode
      0.12 ± 17%      -0.1        0.04 ±100%  perf-profile.children.cycles-pp.__delete_from_page_cache
      0.07 ± 14%      +0.0        0.11 ± 18%  perf-profile.children.cycles-pp.irq_work_needs_cpu
      0.01 ±173%      +0.1        0.07 ±  7%  perf-profile.children.cycles-pp.kmem_cache_alloc
      0.01 ±173%      +0.1        0.07 ±  7%  perf-profile.children.cycles-pp.__unwind_start
      0.00            +0.1        0.07 ±  7%  perf-profile.children.cycles-pp.truncate_exceptional_pvec_entries
      0.00            +0.1        0.07 ±  7%  perf-profile.children.cycles-pp.rcu_core_si
      0.04 ±106%      +0.1        0.11 ± 18%  perf-profile.children.cycles-pp.drm_fb_helper_dirty
      0.04 ±102%      +0.1        0.11 ± 18%  perf-profile.children.cycles-pp.ext4_invalidatepage
      0.04 ±102%      +0.1        0.11 ± 18%  perf-profile.children.cycles-pp.xas_find
      0.02 ±173%      +0.1        0.10 ± 36%  perf-profile.children.cycles-pp.__wake_up_common
      0.02 ±173%      +0.1        0.10 ± 30%  perf-profile.children.cycles-pp.intel_pmu_disable_all
      0.00            +0.1        0.08 ± 12%  perf-profile.children.cycles-pp.timerqueue_iterate_next
      0.04 ±110%      +0.1        0.13        perf-profile.children.cycles-pp.free_unref_page_prepare
      0.04 ±102%      +0.1        0.13        perf-profile.children.cycles-pp.xas_start
      0.00            +0.1        0.10 ± 36%  perf-profile.children.cycles-pp.autoremove_wake_function
      0.17 ± 51%      +0.1        0.28 ± 20%  perf-profile.children.cycles-pp.__blk_mq_delay_run_hw_queue
      0.11 ± 44%      +0.1        0.21 ±  4%  perf-profile.children.cycles-pp.ext4_io_submit
      0.03 ±173%      +0.1        0.15 ± 37%  perf-profile.children.cycles-pp.update_rt_rq_load_avg
      0.22 ± 45%      +0.1        0.34 ±  2%  perf-profile.children.cycles-pp.rb_next
      0.11 ± 81%      +0.1        0.23 ± 17%  perf-profile.children.cycles-pp.cpuidle_reflect
      0.32 ± 11%      +0.1        0.46        perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
      0.46 ± 25%      +0.2        0.62 ± 15%  perf-profile.children.cycles-pp._raw_spin_lock
      0.14 ± 93%      +0.2        0.31 ± 48%  perf-profile.children.cycles-pp.workingset_update_node
      0.44 ± 23%      +0.2        0.61 ± 18%  perf-profile.children.cycles-pp.sched_clock
      0.09 ± 59%      +0.2        0.26 ±  3%  perf-profile.children.cycles-pp.__mod_zone_page_state
      0.20 ± 19%      +0.2        0.37 ± 45%  perf-profile.children.cycles-pp.pm_qos_read_value
      0.41 ± 24%      +0.2        0.58 ± 24%  perf-profile.children.cycles-pp.native_sched_clock
      0.15 ± 53%      +0.2        0.34 ± 35%  perf-profile.children.cycles-pp.unaccount_page_cache_page
      0.20 ± 42%      +0.2        0.42 ± 20%  perf-profile.children.cycles-pp.nr_iowait_cpu
      0.25 ± 55%      +0.3        0.52 ± 22%  perf-profile.children.cycles-pp.update_ts_time_stats
      0.56 ± 16%      +0.3        0.83 ± 20%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.95 ± 58%      +0.7        1.61 ± 27%  perf-profile.children.cycles-pp.free_unref_page_list
      0.16 ± 23%      +0.0        0.21 ±  4%  perf-profile.self.cycles-pp.menu_reflect
      0.05 ± 60%      +0.1        0.10 ± 30%  perf-profile.self.cycles-pp.leave_mm
      0.04 ±110%      +0.1        0.10 ± 30%  perf-profile.self.cycles-pp.free_unref_page_prepare
      0.01 ±173%      +0.1        0.08 ± 12%  perf-profile.self.cycles-pp.jbd2_journal_try_to_free_buffers
      0.00            +0.1        0.07 ±  7%  perf-profile.self.cycles-pp.timerqueue_iterate_next
      0.00            +0.1        0.07 ±  7%  perf-profile.self.cycles-pp.truncate_exceptional_pvec_entries
      0.00            +0.1        0.07 ±  7%  perf-profile.self.cycles-pp.rcu_core_si
      0.04 ±102%      +0.1        0.11 ± 18%  perf-profile.self.cycles-pp.ext4_invalidatepage
      0.15 ± 24%      +0.1        0.24 ± 10%  perf-profile.self.cycles-pp.get_next_timer_interrupt
      0.09 ± 67%      +0.1        0.20 ± 35%  perf-profile.self.cycles-pp.try_to_free_buffers
      0.00            +0.1        0.11 ± 18%  perf-profile.self.cycles-pp.xas_start
      0.03 ±173%      +0.1        0.15 ± 37%  perf-profile.self.cycles-pp.update_rt_rq_load_avg
      0.11 ± 81%      +0.1        0.23 ± 17%  perf-profile.self.cycles-pp.cpuidle_reflect
      0.26 ± 20%      +0.1        0.39 ± 14%  perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
      0.39 ± 22%      +0.1        0.52 ± 15%  perf-profile.self.cycles-pp.native_sched_clock
      0.10 ±109%      +0.1        0.23 ± 17%  perf-profile.self.cycles-pp.update_ts_time_stats
      0.10 ± 57%      +0.1        0.24 ± 10%  perf-profile.self.cycles-pp.sched_clock_cpu
      0.07 ± 66%      +0.2        0.22 ± 25%  perf-profile.self.cycles-pp.free_unref_page_commit
      0.46 ± 25%      +0.2        0.62 ± 15%  perf-profile.self.cycles-pp._raw_spin_lock
      0.03 ±173%      +0.2        0.20 ± 35%  perf-profile.self.cycles-pp.unaccount_page_cache_page
      0.09 ± 59%      +0.2        0.26 ±  3%  perf-profile.self.cycles-pp.__mod_zone_page_state
      0.20 ± 19%      +0.2        0.37 ± 45%  perf-profile.self.cycles-pp.pm_qos_read_value
      0.18 ± 40%      +0.2        0.37 ± 27%  perf-profile.self.cycles-pp.timerqueue_del
      0.38 ± 35%      +0.2        0.58 ± 13%  perf-profile.self.cycles-pp.__hrtimer_run_queues
      0.19 ± 36%      +0.2        0.42 ± 20%  perf-profile.self.cycles-pp.nr_iowait_cpu
      0.00            +0.2        0.23 ± 43%  perf-profile.self.cycles-pp.free_unref_page_list





Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.4.0-rc3-00048-g569342dc24853" of type "text/plain" (200550 bytes)

View attachment "job-script" of type "text/plain" (8266 bytes)

View attachment "job.yaml" of type "text/plain" (5796 bytes)

View attachment "reproduce" of type "text/plain" (900 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ