lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Date:   Sat, 27 Feb 2021 20:08:04 +0800
From:   kernel test robot <oliver.sang@...el.com>
To:     Jan Kara <jack@...e.cz>
Cc:     Theodore Ts'o <tytso@....edu>, LKML <linux-kernel@...r.kernel.org>,
        lkp@...ts.01.org, lkp@...el.com, ying.huang@...el.com,
        feng.tang@...el.com, zhengjun.xing@...el.com
Subject: [ext4]  05c2c00f37:  aim7.jobs-per-min -11.8% regression


Greeting,

FYI, we noticed a -11.8% regression of aim7.jobs-per-min due to commit:


commit: 05c2c00f3769abb9e323fcaca70d2de0b48af7ba ("ext4: protect superblock modifications with a buffer lock")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master


in testcase: aim7
on test machine: 88 threads Intel(R) Xeon(R) Gold 6238M CPU @ 2.10GHz with 128G memory
with following parameters:

	disk: 4BRD_12G
	md: RAID1
	fs: ext4
	test: creat-clo
	load: 1000
	cpufreq_governor: performance
	ucode: 0x5003006

test-description: AIM7 is a traditional UNIX system level benchmark suite which is used to test and measure the performance of multiuser system.
test-url: https://sourceforge.net/projects/aimbench/files/aim-suite7/

In addition to that, the commit also has significant impact on the following tests:

+------------------+---------------------------------------------------------------------------+
| testcase: change | aim9: aim9.udp_test.ops_per_sec 2.1% improvement                          |
| test machine     | 256 threads Intel(R) Genuine Intel(R) CPU 0000 @ 1.30GHz with 112G memory |
| test parameters  | cpufreq_governor=performance                                              |
|                  | test=all                                                                  |
|                  | testtime=5s                                                               |
|                  | ucode=0xffff0190                                                          |
+------------------+---------------------------------------------------------------------------+


If you fix the issue, kindly add following tag
Reported-by: kernel test robot <oliver.sang@...el.com>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install                job.yaml  # job file is attached in this email
        bin/lkp split-job --compatible job.yaml
        bin/lkp run                    compatible-job.yaml

=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/load/md/rootfs/tbox_group/test/testcase/ucode:
  gcc-9/performance/4BRD_12G/ext4/x86_64-rhel-8.3/1000/RAID1/debian-10.4-x86_64-20200603.cgz/lkp-csl-2sp9/creat-clo/aim7/0x5003006

commit: 
  4392fbc4ba ("ext4: drop sync argument of ext4_commit_super()")
  05c2c00f37 ("ext4: protect superblock modifications with a buffer lock")

4392fbc4bab57db3 05c2c00f3769abb9e323fcaca70 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
     13342           -11.8%      11771 ±  2%  aim7.jobs-per-min
    449.81           +13.4%     510.10 ±  2%  aim7.time.elapsed_time
    449.81           +13.4%     510.10 ±  2%  aim7.time.elapsed_time.max
   1246834 ±  2%     +13.8%    1418574 ±  3%  aim7.time.involuntary_context_switches
    204562 ±  4%     +12.2%     229478 ±  3%  aim7.time.minor_page_faults
     35777           +12.9%      40390 ±  2%  aim7.time.system_time
   6270530 ±  3%     +34.5%    8435526 ±  2%  aim7.time.voluntary_context_switches
   8686688 ± 12%     +24.1%   10784294 ±  4%  cpuidle.C1E.usage
      8.74            +4.5%       9.14        iostat.cpu.idle
      0.02 ± 12%      -0.0        0.02 ±  9%  mpstat.cpu.all.iowait%
      6988 ± 11%     +43.1%       9999 ±  9%  numa-meminfo.node0.Active(file)
   1440589 ±  7%      +7.8%    1552890 ±  6%  numa-numastat.node1.numa_hit
      1747 ± 11%     +43.0%       2499 ±  9%  numa-vmstat.node0.nr_active_file
      1747 ± 11%     +43.0%       2499 ±  9%  numa-vmstat.node0.nr_zone_active_file
    481.09           +12.6%     541.71 ±  2%  uptime.boot
      5541           +10.8%       6140        uptime.idle
     42743           -10.6%      38207        slabinfo.filp.active_objs
      1430           -10.0%       1287        slabinfo.filp.active_slabs
     45785           -10.0%      41214        slabinfo.filp.num_objs
      1430           -10.0%       1287        slabinfo.filp.num_slabs
     91.00            -1.1%      90.00        vmstat.cpu.sy
      2850 ±  2%     +14.5%       3263        vmstat.io.bo
     13739 ±  3%     +29.2%      17754 ±  2%  vmstat.memory.buff
     25822 ±  2%     +14.8%      29645        vmstat.system.cs
    452981           +11.7%     505788 ±  3%  meminfo.Active
    439264           +11.1%     488079 ±  3%  meminfo.Active(anon)
     13715 ±  3%     +29.1%      17708 ±  2%  meminfo.Active(file)
     13713 ±  3%     +29.2%      17723 ±  2%  meminfo.Buffers
    459294           +10.6%     507955 ±  3%  meminfo.Shmem
    109825           +11.1%     121996 ±  3%  proc-vmstat.nr_active_anon
      3428 ±  3%     +29.1%       4426 ±  2%  proc-vmstat.nr_active_file
      1427           +13.3%       1618 ±  2%  proc-vmstat.nr_dirtied
    376669            +3.5%     389792        proc-vmstat.nr_file_pages
    114820           +10.6%     126987 ±  3%  proc-vmstat.nr_shmem
      1372           +15.1%       1579 ±  4%  proc-vmstat.nr_written
    109825           +11.1%     121996 ±  3%  proc-vmstat.nr_zone_active_anon
      3428 ±  3%     +29.1%       4426 ±  2%  proc-vmstat.nr_zone_active_file
    151153 ±  5%     +16.9%     176679 ±  3%  proc-vmstat.numa_hint_faults
    106816 ±  7%     +17.4%     125437 ±  4%  proc-vmstat.numa_hint_faults_local
   2915186            +5.4%    3071445        proc-vmstat.numa_hit
   2835807            +5.4%    2988976        proc-vmstat.numa_local
    129566 ±  2%      +7.4%     139212 ±  4%  proc-vmstat.pgactivate
   4233617            +2.8%    4353951        proc-vmstat.pgalloc_normal
   1622353           +12.0%    1816326        proc-vmstat.pgfault
   3863258            +2.8%    3970690        proc-vmstat.pgfree
   1290723 ±  3%     +29.8%    1675288 ±  2%  proc-vmstat.pgpgout
     86298           +10.3%      95227 ±  3%  proc-vmstat.pgreuse
    189.67 ±  5%      -9.6%     171.45 ±  5%  sched_debug.cfs_rq:/.load_avg.stddev
  15090675           +13.9%   17181333        sched_debug.cfs_rq:/.min_vruntime.avg
  15410230           +14.0%   17563602        sched_debug.cfs_rq:/.min_vruntime.max
  14179241           +13.8%   16142220        sched_debug.cfs_rq:/.min_vruntime.min
    909405 ±  5%     +13.7%    1034081 ±  6%  sched_debug.cfs_rq:/.spread0.avg
   1229377 ±  4%     +15.2%    1416676 ±  3%  sched_debug.cfs_rq:/.spread0.max
    240213           +12.5%     270308        sched_debug.cpu.clock.avg
    240225           +12.5%     270320        sched_debug.cpu.clock.max
    240200           +12.5%     270295        sched_debug.cpu.clock.min
    237580           +12.5%     267317        sched_debug.cpu.clock_task.avg
    237797           +12.5%     267523        sched_debug.cpu.clock_task.max
    232767           +12.8%     262508        sched_debug.cpu.clock_task.min
     64022 ±  2%     +31.4%      84141        sched_debug.cpu.nr_switches.avg
     77073           +28.8%      99292 ±  3%  sched_debug.cpu.nr_switches.max
     60980 ±  2%     +32.5%      80775        sched_debug.cpu.nr_switches.min
      1399 ±  8%     +23.7%       1731 ±  4%  sched_debug.cpu.nr_uninterruptible.max
    248.99 ±  6%     +21.4%     302.15 ±  9%  sched_debug.cpu.nr_uninterruptible.stddev
    240200           +12.5%     270295        sched_debug.cpu_clk
    239703           +12.6%     269797        sched_debug.ktime
    240543           +12.5%     270629        sched_debug.sched_clk
 1.221e+10            -1.1%  1.208e+10        perf-stat.i.branch-instructions
      0.22 ± 23%      -0.0        0.17        perf-stat.i.branch-miss-rate%
  14793770            -6.0%   13912167        perf-stat.i.branch-misses
     25953 ±  2%     +14.8%      29803        perf-stat.i.context-switches
      4869           +13.6%       5530        perf-stat.i.cpu-migrations
  1.64e+10            -1.3%   1.62e+10        perf-stat.i.dTLB-loads
 1.075e+09            -9.9%  9.682e+08 ±  2%  perf-stat.i.dTLB-stores
    430893 ±  2%     +11.6%     480826        perf-stat.i.iTLB-loads
 6.072e+10            -1.1%  6.005e+10        perf-stat.i.instructions
    338.69            -1.5%     333.53        perf-stat.i.metric.M/sec
   6039682            -7.8%    5565765 ±  2%  perf-stat.i.node-store-misses
     96902 ±  2%      -4.0%      93045 ±  2%  perf-stat.i.node-stores
      0.12            -0.0        0.12        perf-stat.overall.branch-miss-rate%
 1.219e+10            -1.0%  1.206e+10        perf-stat.ps.branch-instructions
  14740466            -5.9%   13872099        perf-stat.ps.branch-misses
     25873 ±  2%     +14.9%      29722        perf-stat.ps.context-switches
      4856           +13.6%       5517        perf-stat.ps.cpu-migrations
 1.637e+10            -1.2%  1.617e+10        perf-stat.ps.dTLB-loads
 1.072e+09            -9.9%  9.665e+08 ±  2%  perf-stat.ps.dTLB-stores
    428747 ±  2%     +11.6%     478487        perf-stat.ps.iTLB-loads
 6.061e+10            -1.1%  5.996e+10        perf-stat.ps.instructions
   6028729            -7.8%    5557348 ±  2%  perf-stat.ps.node-store-misses
     96841            -4.1%      92846 ±  3%  perf-stat.ps.node-stores
 2.734e+13           +12.2%  3.066e+13 ±  2%  perf-stat.total.instructions
      1.54 ± 72%     -86.8%       0.20 ±221%  perf-sched.sch_delay.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
      0.17 ± 19%     -47.8%       0.09 ± 22%  perf-sched.sch_delay.avg.ms.preempt_schedule_common._cond_resched.__ext4_journal_get_write_access.ext4_orphan_add.ext4_truncate
      1.57 ± 11%     +18.6%       1.86 ±  5%  perf-sched.sch_delay.avg.ms.preempt_schedule_common._cond_resched.__getblk_gfp.__ext4_get_inode_loc.ext4_get_inode_loc
      1.45 ± 37%    +126.7%       3.29 ± 34%  perf-sched.sch_delay.avg.ms.preempt_schedule_common._cond_resched.kmem_cache_alloc.jbd2__journal_start.__ext4_journal_start_sb
      0.59 ± 34%     +58.4%       0.93 ± 24%  perf-sched.sch_delay.avg.ms.preempt_schedule_common._cond_resched.stop_one_cpu.migrate_task_to.task_numa_migrate
      3.16 ± 68%     -80.8%       0.61 ±223%  perf-sched.sch_delay.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
      8.52 ± 28%     +80.5%      15.38 ± 25%  perf-sched.sch_delay.max.ms.preempt_schedule_common._cond_resched.kmem_cache_alloc.__alloc_file.alloc_empty_file
    162500 ±  9%     +25.1%     203298 ± 10%  perf-sched.total_wait_and_delay.count.ms
    116.08 ±  9%     -21.7%      90.91 ± 13%  perf-sched.wait_and_delay.avg.ms.preempt_schedule_common._cond_resched.__ext4_handle_dirty_metadata.ext4_do_update_inode.ext4_mark_iloc_dirty
    134.71 ±  9%     -24.4%     101.80 ±  7%  perf-sched.wait_and_delay.avg.ms.preempt_schedule_common._cond_resched.__getblk_gfp.__ext4_get_inode_loc.ext4_get_inode_loc
    145.16 ± 21%     -42.0%      84.13 ± 46%  perf-sched.wait_and_delay.avg.ms.preempt_schedule_common._cond_resched.__mutex_lock.isra.0
    153.37 ± 30%     -68.2%      48.78 ±103%  perf-sched.wait_and_delay.avg.ms.preempt_schedule_common._cond_resched.ext4_journal_check_start.__ext4_journal_start_sb.ext4_truncate
     10.19 ±  5%     -84.0%       1.62 ±223%  perf-sched.wait_and_delay.avg.ms.preempt_schedule_common._cond_resched.stop_one_cpu.__set_cpus_allowed_ptr.sched_setaffinity
     76.08 ± 11%     -23.6%      58.16 ± 12%  perf-sched.wait_and_delay.avg.ms.wait_transaction_locked.add_transaction_credits.start_this_handle.jbd2__journal_start
     69.33 ± 14%     +36.5%      94.67 ± 13%  perf-sched.wait_and_delay.count.jbd2_journal_commit_transaction.kjournald2.kthread.ret_from_fork
    890.17 ±  5%     -30.8%     615.83 ±  4%  perf-sched.wait_and_delay.count.preempt_schedule_common._cond_resched.__ext4_handle_dirty_metadata.ext4_do_update_inode.ext4_mark_iloc_dirty
      3176 ± 15%     +43.3%       4551 ± 17%  perf-sched.wait_and_delay.count.preempt_schedule_common._cond_resched.__ext4_mark_inode_dirty.ext4_dirty_inode.__mark_inode_dirty
    349.17 ±  5%     -17.3%     288.83 ±  5%  perf-sched.wait_and_delay.count.preempt_schedule_common._cond_resched.__getblk_gfp.__ext4_get_inode_loc.ext4_get_inode_loc
    654.17 ± 73%    +108.3%       1362 ± 18%  perf-sched.wait_and_delay.count.preempt_schedule_common._cond_resched.kmem_cache_alloc.start_this_handle.jbd2__journal_start
    953.33 ±  3%     -84.6%     146.67 ±223%  perf-sched.wait_and_delay.count.preempt_schedule_common._cond_resched.stop_one_cpu.__set_cpus_allowed_ptr.sched_setaffinity
     35615 ± 10%     +26.7%      45108 ± 10%  perf-sched.wait_and_delay.count.schedule_preempt_disabled.__mutex_lock.isra.0.ext4_orphan_add
     54444 ± 15%     +44.7%      78784 ± 17%  perf-sched.wait_and_delay.count.wait_transaction_locked.add_transaction_credits.start_this_handle.jbd2__journal_start
    749.81 ±  5%     -18.6%     610.38 ±  3%  perf-sched.wait_and_delay.max.ms.preempt_schedule_common._cond_resched.__ext4_handle_dirty_metadata.ext4_do_update_inode.ext4_mark_iloc_dirty
    751.99 ±  7%     -21.6%     589.37 ±  5%  perf-sched.wait_and_delay.max.ms.preempt_schedule_common._cond_resched.__getblk_gfp.__ext4_get_inode_loc.ext4_get_inode_loc
    545.93 ± 17%     -31.5%     374.21 ± 24%  perf-sched.wait_and_delay.max.ms.preempt_schedule_common._cond_resched.__kmalloc.ext4_ext_remove_space.ext4_ext_truncate
    619.18 ± 17%     -45.6%     337.10 ± 49%  perf-sched.wait_and_delay.max.ms.preempt_schedule_common._cond_resched.__mutex_lock.isra.0
    573.86 ± 20%     -64.0%     206.68 ±104%  perf-sched.wait_and_delay.max.ms.preempt_schedule_common._cond_resched.ext4_journal_check_start.__ext4_journal_start_sb.ext4_truncate
      1067 ± 12%     -84.4%     166.83 ±223%  perf-sched.wait_and_delay.max.ms.preempt_schedule_common._cond_resched.stop_one_cpu.__set_cpus_allowed_ptr.sched_setaffinity
    650.19 ±  8%     -13.1%     564.84 ±  3%  perf-sched.wait_and_delay.max.ms.preempt_schedule_common._cond_resched.stop_one_cpu.migrate_task_to.task_numa_migrate
    676.68 ±  9%     -17.8%     556.38 ±  3%  perf-sched.wait_and_delay.max.ms.preempt_schedule_common._cond_resched.wait_for_completion.stop_two_cpus.migrate_swap
    745.90 ±  4%     -18.2%     609.91 ±  2%  perf-sched.wait_and_delay.max.ms.rwsem_down_write_slowpath.path_openat.do_filp_open.do_sys_openat2
    775.20 ±  6%     -18.9%     628.79        perf-sched.wait_and_delay.max.ms.schedule_preempt_disabled.__mutex_lock.isra.0.ext4_orphan_add
    770.37 ±  6%     -18.9%     624.81        perf-sched.wait_and_delay.max.ms.schedule_preempt_disabled.__mutex_lock.isra.0.ext4_orphan_del
    114.49 ±  9%     -22.1%      89.15 ± 13%  perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.__ext4_handle_dirty_metadata.ext4_do_update_inode.ext4_mark_iloc_dirty
    133.14 ±  9%     -24.9%      99.94 ±  7%  perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.__getblk_gfp.__ext4_get_inode_loc.ext4_get_inode_loc
    142.91 ± 22%     -37.9%      88.74 ± 26%  perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.__mutex_lock.isra.0
    120.05 ± 39%     -61.7%      46.00 ± 96%  perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.down_write.ext4_truncate.ext4_setattr
    151.19 ± 30%     -57.1%      64.88 ± 53%  perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.ext4_journal_check_start.__ext4_journal_start_sb.ext4_truncate
      4.55 ±  4%     +12.8%       5.13 ±  3%  perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.jbd2_journal_commit_transaction.kjournald2.kthread
     74.36 ± 11%     -24.3%      56.28 ± 13%  perf-sched.wait_time.avg.ms.wait_transaction_locked.add_transaction_credits.start_this_handle.jbd2__journal_start
    749.64 ±  5%     -18.6%     610.37 ±  3%  perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.__ext4_handle_dirty_metadata.ext4_do_update_inode.ext4_mark_iloc_dirty
    751.98 ±  7%     -21.6%     589.37 ±  5%  perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.__getblk_gfp.__ext4_get_inode_loc.ext4_get_inode_loc
    545.92 ± 17%     -32.0%     371.40 ± 24%  perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.__kmalloc.ext4_ext_remove_space.ext4_ext_truncate
    619.17 ± 17%     -38.7%     379.32 ± 23%  perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.__mutex_lock.isra.0
    571.69 ± 20%     -52.1%     273.60 ± 56%  perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.ext4_journal_check_start.__ext4_journal_start_sb.ext4_truncate
    650.17 ±  8%     -13.1%     564.83 ±  3%  perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.stop_one_cpu.migrate_task_to.task_numa_migrate
    676.67 ±  9%     -17.8%     556.37 ±  3%  perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.wait_for_completion.stop_two_cpus.migrate_swap
    745.40 ±  4%     -18.3%     609.29 ±  2%  perf-sched.wait_time.max.ms.rwsem_down_write_slowpath.path_openat.do_filp_open.do_sys_openat2
    769.37 ±  6%     -19.0%     622.87 ±  2%  perf-sched.wait_time.max.ms.schedule_preempt_disabled.__mutex_lock.isra.0.ext4_orphan_add
    769.35 ±  6%     -19.1%     622.61 ±  2%  perf-sched.wait_time.max.ms.schedule_preempt_disabled.__mutex_lock.isra.0.ext4_orphan_del
      0.59 ±  4%      -0.2        0.35 ± 70%  perf-profile.calltrace.cycles-pp.ext4_ext_truncate.ext4_truncate.ext4_setattr.notify_change.do_truncate
      0.81 ±  3%      -0.1        0.70 ±  4%  perf-profile.children.cycles-pp.ext4_reserve_inode_write
      0.89 ±  3%      -0.1        0.78 ±  3%  perf-profile.children.cycles-pp.__ext4_mark_inode_dirty
      1.00 ±  2%      -0.1        0.89 ±  3%  perf-profile.children.cycles-pp.ext4_mark_iloc_dirty
      0.59 ±  4%      -0.1        0.52 ±  4%  perf-profile.children.cycles-pp.ext4_ext_truncate
      0.65 ±  2%      -0.1        0.59 ±  4%  perf-profile.children.cycles-pp.ext4_do_update_inode
      0.32 ±  2%      -0.0        0.28 ±  2%  perf-profile.children.cycles-pp.__mutex_unlock_slowpath
      0.35 ±  4%      -0.0        0.30 ±  4%  perf-profile.children.cycles-pp.ext4_ext_remove_space
      0.21 ±  3%      -0.0        0.17 ±  4%  perf-profile.children.cycles-pp.__getblk_gfp
      0.15 ± 13%      -0.0        0.11 ± 15%  perf-profile.children.cycles-pp.run_ksoftirqd
      0.18 ±  3%      -0.0        0.14 ±  5%  perf-profile.children.cycles-pp.crc32c_pcl_intel_update
      0.15 ± 11%      -0.0        0.12 ± 11%  perf-profile.children.cycles-pp.smpboot_thread_fn
      0.18 ±  2%      -0.0        0.15 ±  4%  perf-profile.children.cycles-pp.__find_get_block
      0.12 ±  4%      -0.0        0.09 ±  5%  perf-profile.children.cycles-pp.mutex_unlock
      0.15 ±  3%      -0.0        0.12 ±  5%  perf-profile.children.cycles-pp.ext4_inode_csum_set
      0.24 ±  3%      -0.0        0.21 ±  3%  perf-profile.children.cycles-pp.__close
      0.10 ±  9%      -0.0        0.08 ± 12%  perf-profile.children.cycles-pp.osq_unlock
      0.20 ±  3%      -0.0        0.18 ±  3%  perf-profile.children.cycles-pp.exit_to_user_mode_prepare
      0.14 ±  3%      -0.0        0.12 ±  3%  perf-profile.children.cycles-pp.wake_up_q
      0.10 ±  4%      -0.0        0.08 ±  5%  perf-profile.children.cycles-pp.cap_inode_need_killpriv
      0.12 ±  4%      -0.0        0.11 ±  6%  perf-profile.children.cycles-pp.mutex_lock
      0.08            -0.0        0.06 ±  7%  perf-profile.children.cycles-pp.ext4_xattr_get
      0.11            -0.0        0.09 ±  5%  perf-profile.children.cycles-pp.dentry_needs_remove_privs
      0.10            -0.0        0.08 ±  5%  perf-profile.children.cycles-pp.__vfs_getxattr
      0.11 ±  3%      -0.0        0.10 ±  3%  perf-profile.children.cycles-pp.alloc_empty_file
      0.11            -0.0        0.10 ±  4%  perf-profile.children.cycles-pp.__alloc_file
      0.18 ±  6%      +0.0        0.20 ±  4%  perf-profile.children.cycles-pp.do_softirq_own_stack
      0.19 ±  6%      +0.0        0.21 ±  4%  perf-profile.children.cycles-pp.irq_exit_rcu
      0.03 ±100%      +0.0        0.07 ±  7%  perf-profile.children.cycles-pp.ktime_get_update_offsets_now
      0.06 ± 45%      +0.0        0.10 ± 11%  perf-profile.children.cycles-pp.ktime_get
      0.69            +0.1        0.75        perf-profile.children.cycles-pp.mutex_spin_on_owner
      0.35 ±  6%      +0.1        0.42 ±  6%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.30 ±  5%      +0.1        0.38 ±  8%  perf-profile.children.cycles-pp.tick_sched_timer
      0.47 ±  7%      +0.1        0.57 ±  7%  perf-profile.children.cycles-pp.hrtimer_interrupt
      0.47 ±  7%      +0.1        0.58 ±  7%  perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
      0.68 ±  6%      +0.1        0.80 ±  4%  perf-profile.children.cycles-pp.asm_call_sysvec_on_stack
      0.68 ±  6%      +0.1        0.81 ±  4%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
      0.72 ±  7%      +0.1        0.85 ±  4%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
      0.24            -0.0        0.19 ±  3%  perf-profile.self.cycles-pp.ext4_reserve_inode_write
      0.29 ±  4%      -0.0        0.24 ±  2%  perf-profile.self.cycles-pp.ext4_mark_iloc_dirty
      0.32 ±  3%      -0.0        0.27 ±  4%  perf-profile.self.cycles-pp.__mutex_lock
      0.07 ±  7%      -0.0        0.03 ±100%  perf-profile.self.cycles-pp.ext4_inode_csum_set
      0.12 ±  4%      -0.0        0.09 ±  5%  perf-profile.self.cycles-pp.mutex_unlock
      0.12 ±  5%      -0.0        0.10 ±  7%  perf-profile.self.cycles-pp.mutex_lock
      0.10 ±  6%      -0.0        0.08 ± 12%  perf-profile.self.cycles-pp.osq_unlock
      0.12            -0.0        0.10        perf-profile.self.cycles-pp._raw_spin_lock
      0.09 ±  7%      -0.0        0.07 ±  6%  perf-profile.self.cycles-pp.__find_get_block
      0.07 ±  5%      -0.0        0.05 ±  8%  perf-profile.self.cycles-pp.ext4_xattr_get
      0.09            -0.0        0.08 ±  6%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.05 ± 45%      +0.0        0.09 ± 12%  perf-profile.self.cycles-pp.ktime_get
      0.02 ±141%      +0.0        0.06        perf-profile.self.cycles-pp.ktime_get_update_offsets_now
      0.68            +0.1        0.74 ±  2%  perf-profile.self.cycles-pp.mutex_spin_on_owner
      0.02 ±141%      +0.1        0.08 ±  8%  perf-profile.self.cycles-pp.ext4_orphan_add
     20823 ±  2%     +14.2%      23775 ±  3%  softirqs.CPU0.SCHED
     17539 ±  2%     +22.4%      21460 ±  5%  softirqs.CPU1.SCHED
     17122           +19.9%      20527        softirqs.CPU10.SCHED
     17055 ±  2%     +19.8%      20433        softirqs.CPU11.SCHED
     16919           +21.4%      20538        softirqs.CPU12.SCHED
     17013 ±  2%     +20.6%      20510 ±  2%  softirqs.CPU13.SCHED
     17119           +19.8%      20501        softirqs.CPU14.SCHED
     17158           +19.1%      20431 ±  2%  softirqs.CPU15.SCHED
     17223           +18.7%      20443        softirqs.CPU16.SCHED
     16969 ±  2%     +19.7%      20305        softirqs.CPU17.SCHED
     16997           +20.3%      20441        softirqs.CPU18.SCHED
     17126           +19.1%      20399        softirqs.CPU19.SCHED
     17544 ±  3%     +18.5%      20796 ±  5%  softirqs.CPU2.SCHED
     17207           +19.1%      20489        softirqs.CPU20.SCHED
     17215           +17.4%      20219        softirqs.CPU21.SCHED
     16574           +19.2%      19758 ±  3%  softirqs.CPU22.SCHED
     16901 ±  2%     +17.7%      19893 ±  2%  softirqs.CPU23.SCHED
     16901 ±  3%     +19.4%      20180 ±  2%  softirqs.CPU24.SCHED
     16818           +19.5%      20099 ±  2%  softirqs.CPU25.SCHED
     16892           +18.6%      20039        softirqs.CPU26.SCHED
     16839           +20.0%      20205        softirqs.CPU27.SCHED
     16926           +18.7%      20091        softirqs.CPU28.SCHED
     16762 ±  2%     +20.1%      20125        softirqs.CPU29.SCHED
     17050 ±  2%     +19.7%      20410 ±  2%  softirqs.CPU3.SCHED
     16961           +19.7%      20304 ±  2%  softirqs.CPU30.SCHED
     17013           +19.0%      20238 ±  2%  softirqs.CPU31.SCHED
     16836           +22.4%      20599 ±  3%  softirqs.CPU32.SCHED
     17160           +17.5%      20164        softirqs.CPU33.SCHED
     16658 ±  2%     +21.7%      20268 ±  2%  softirqs.CPU34.SCHED
     16980 ±  2%     +18.2%      20069 ±  3%  softirqs.CPU35.SCHED
     16720           +20.0%      20067 ±  2%  softirqs.CPU36.SCHED
     17043           +19.3%      20338        softirqs.CPU37.SCHED
     16935 ±  2%     +20.0%      20321        softirqs.CPU38.SCHED
     17031           +19.5%      20349        softirqs.CPU39.SCHED
     17457 ±  3%     +17.1%      20438        softirqs.CPU4.SCHED
     16977           +20.1%      20393 ±  2%  softirqs.CPU40.SCHED
     16873           +20.7%      20369        softirqs.CPU41.SCHED
     17216 ±  2%     +17.6%      20238 ±  2%  softirqs.CPU42.SCHED
     17045 ±  2%     +19.3%      20342 ±  2%  softirqs.CPU43.SCHED
     16175 ±  3%     +20.4%      19473 ±  2%  softirqs.CPU44.SCHED
     16913 ±  2%     +18.6%      20051        softirqs.CPU45.SCHED
     16989 ±  2%     +17.6%      19980        softirqs.CPU46.SCHED
     16774 ±  2%     +18.9%      19949        softirqs.CPU47.SCHED
     16922 ±  2%     +19.1%      20162        softirqs.CPU48.SCHED
     16933 ±  3%     +19.8%      20286        softirqs.CPU49.SCHED
     17135 ±  3%     +20.1%      20585        softirqs.CPU5.SCHED
     16998 ±  2%     +16.8%      19847        softirqs.CPU50.SCHED
     16815 ±  2%     +19.4%      20077        softirqs.CPU51.SCHED
     16962           +18.6%      20111        softirqs.CPU52.SCHED
     16853 ±  2%     +20.0%      20224        softirqs.CPU53.SCHED
     17002           +19.5%      20320        softirqs.CPU54.SCHED
     17025 ±  2%     +18.8%      20228 ±  2%  softirqs.CPU55.SCHED
     16851           +19.8%      20190        softirqs.CPU56.SCHED
     17018           +18.3%      20132        softirqs.CPU57.SCHED
     16949 ±  2%     +19.8%      20308 ±  2%  softirqs.CPU58.SCHED
     17189           +18.0%      20287        softirqs.CPU59.SCHED
     16891           +19.2%      20135        softirqs.CPU6.SCHED
     16989           +20.0%      20387        softirqs.CPU60.SCHED
     17075           +17.7%      20102 ±  2%  softirqs.CPU61.SCHED
     16938 ±  2%     +19.8%      20285        softirqs.CPU62.SCHED
     16854 ±  2%     +19.2%      20094 ±  2%  softirqs.CPU63.SCHED
     17063           +18.5%      20216        softirqs.CPU64.SCHED
     17048           +17.2%      19984        softirqs.CPU65.SCHED
     16870           +19.4%      20138        softirqs.CPU66.SCHED
     16817           +20.6%      20274 ±  2%  softirqs.CPU67.SCHED
     16868 ±  2%     +20.1%      20256        softirqs.CPU68.SCHED
     16890           +19.7%      20212 ±  2%  softirqs.CPU69.SCHED
     16842           +20.2%      20252        softirqs.CPU7.SCHED
     17011           +19.8%      20375        softirqs.CPU70.SCHED
     17048           +20.1%      20478 ±  2%  softirqs.CPU71.SCHED
     16881 ±  2%     +19.6%      20194        softirqs.CPU72.SCHED
     16931           +19.9%      20294        softirqs.CPU73.SCHED
     17102           +18.7%      20308        softirqs.CPU74.SCHED
     16926 ±  2%     +19.4%      20206        softirqs.CPU75.SCHED
     16757 ±  2%     +21.6%      20380 ±  2%  softirqs.CPU76.SCHED
     17052           +19.8%      20420        softirqs.CPU77.SCHED
     16998           +19.8%      20368        softirqs.CPU78.SCHED
     17209 ±  2%     +19.3%      20533        softirqs.CPU79.SCHED
     17084 ±  2%     +18.6%      20260        softirqs.CPU8.SCHED
     16973 ±  2%     +20.9%      20525 ±  2%  softirqs.CPU80.SCHED
     17198           +18.9%      20457 ±  2%  softirqs.CPU81.SCHED
     16907 ±  2%     +20.8%      20419 ±  2%  softirqs.CPU82.SCHED
     17023           +19.3%      20316 ±  2%  softirqs.CPU83.SCHED
     16935           +18.8%      20116 ±  2%  softirqs.CPU84.SCHED
     17084           +18.8%      20297        softirqs.CPU85.SCHED
     17086           +19.2%      20363        softirqs.CPU86.SCHED
     16995 ±  3%     +19.7%      20350 ±  2%  softirqs.CPU87.SCHED
     17100 ±  2%     +19.4%      20420        softirqs.CPU9.SCHED
   1498602           +19.3%    1788245        softirqs.SCHED
     80372           +12.5%      90439 ±  2%  softirqs.TIMER
    902.67           +13.6%       1025 ±  2%  interrupts.9:IO-APIC.9-fasteoi.acpi
   1556804 ±  4%     +35.0%    2102423 ±  3%  interrupts.CAL:Function_call_interrupts
     17978 ±  4%     +27.2%      22867 ±  8%  interrupts.CPU0.CAL:Function_call_interrupts
    902210           +13.5%    1024347 ±  2%  interrupts.CPU0.LOC:Local_timer_interrupts
      3865 ±  3%     +22.0%       4714 ±  3%  interrupts.CPU0.RES:Rescheduling_interrupts
    902.67           +13.6%       1025 ±  2%  interrupts.CPU1.9:IO-APIC.9-fasteoi.acpi
     17763 ±  7%     +27.9%      22718 ±  8%  interrupts.CPU1.CAL:Function_call_interrupts
    902064           +13.5%    1024088 ±  2%  interrupts.CPU1.LOC:Local_timer_interrupts
      3156 ±  3%     +22.8%       3874 ±  5%  interrupts.CPU1.RES:Rescheduling_interrupts
     16929 ±  5%     +27.5%      21592 ±  6%  interrupts.CPU10.CAL:Function_call_interrupts
    902163           +13.5%    1024115 ±  2%  interrupts.CPU10.LOC:Local_timer_interrupts
      3174 ±  4%     +20.5%       3825 ±  3%  interrupts.CPU10.RES:Rescheduling_interrupts
     17099 ±  4%     +25.8%      21503 ±  7%  interrupts.CPU11.CAL:Function_call_interrupts
    902105           +13.5%    1024101 ±  2%  interrupts.CPU11.LOC:Local_timer_interrupts
      3166 ±  3%     +22.1%       3865 ±  3%  interrupts.CPU11.RES:Rescheduling_interrupts
     16935 ±  5%     +27.2%      21542 ±  7%  interrupts.CPU12.CAL:Function_call_interrupts
    902122           +13.5%    1024089 ±  2%  interrupts.CPU12.LOC:Local_timer_interrupts
      3134 ±  4%     +22.4%       3837 ±  5%  interrupts.CPU12.RES:Rescheduling_interrupts
     17107 ±  5%     +25.2%      21414 ±  6%  interrupts.CPU13.CAL:Function_call_interrupts
    902138           +13.5%    1024071 ±  2%  interrupts.CPU13.LOC:Local_timer_interrupts
      3154 ±  4%     +21.3%       3825 ±  5%  interrupts.CPU13.RES:Rescheduling_interrupts
     16949 ±  4%     +28.0%      21692 ±  8%  interrupts.CPU14.CAL:Function_call_interrupts
    902142           +13.5%    1024014 ±  2%  interrupts.CPU14.LOC:Local_timer_interrupts
      3154 ±  2%     +20.8%       3809 ±  5%  interrupts.CPU14.RES:Rescheduling_interrupts
     16971 ±  6%     +26.5%      21460 ±  7%  interrupts.CPU15.CAL:Function_call_interrupts
    902125           +13.5%    1024008 ±  2%  interrupts.CPU15.LOC:Local_timer_interrupts
      3159 ±  3%     +21.7%       3847 ±  5%  interrupts.CPU15.RES:Rescheduling_interrupts
     16914 ±  4%     +26.9%      21469 ±  6%  interrupts.CPU16.CAL:Function_call_interrupts
    902263           +13.5%    1024072 ±  2%  interrupts.CPU16.LOC:Local_timer_interrupts
      3176 ±  4%     +20.6%       3831 ±  3%  interrupts.CPU16.RES:Rescheduling_interrupts
     16959 ±  5%     +27.7%      21659 ±  7%  interrupts.CPU17.CAL:Function_call_interrupts
    902131           +13.5%    1024076 ±  2%  interrupts.CPU17.LOC:Local_timer_interrupts
      3164 ±  3%     +21.1%       3834 ±  5%  interrupts.CPU17.RES:Rescheduling_interrupts
     16921 ±  5%     +27.9%      21647 ±  7%  interrupts.CPU18.CAL:Function_call_interrupts
    901992           +13.5%    1024059 ±  2%  interrupts.CPU18.LOC:Local_timer_interrupts
      3114 ±  3%     +23.7%       3851 ±  3%  interrupts.CPU18.RES:Rescheduling_interrupts
     16953 ±  5%     +27.8%      21669 ±  7%  interrupts.CPU19.CAL:Function_call_interrupts
    902113           +13.5%    1024021 ±  2%  interrupts.CPU19.LOC:Local_timer_interrupts
      3186 ±  2%     +20.8%       3850 ±  4%  interrupts.CPU19.RES:Rescheduling_interrupts
     17479 ±  5%     +32.7%      23186 ±  3%  interrupts.CPU2.CAL:Function_call_interrupts
    902148           +13.5%    1024034 ±  2%  interrupts.CPU2.LOC:Local_timer_interrupts
      3163 ±  3%     +21.2%       3835 ±  3%  interrupts.CPU2.RES:Rescheduling_interrupts
     16875 ±  5%     +27.8%      21575 ±  7%  interrupts.CPU20.CAL:Function_call_interrupts
    902126           +13.5%    1024077 ±  2%  interrupts.CPU20.LOC:Local_timer_interrupts
      3125 ±  3%     +23.9%       3874 ±  3%  interrupts.CPU20.RES:Rescheduling_interrupts
     16824 ±  4%     +27.8%      21502 ±  7%  interrupts.CPU21.CAL:Function_call_interrupts
    902123           +13.5%    1024032 ±  2%  interrupts.CPU21.LOC:Local_timer_interrupts
      3119 ±  2%     +23.8%       3861 ±  4%  interrupts.CPU21.RES:Rescheduling_interrupts
     18853 ± 10%     +42.1%      26784 ±  7%  interrupts.CPU22.CAL:Function_call_interrupts
    902129           +13.5%    1024053 ±  2%  interrupts.CPU22.LOC:Local_timer_interrupts
      3076 ±  2%     +15.9%       3564 ±  3%  interrupts.CPU22.RES:Rescheduling_interrupts
     18904 ± 10%     +41.5%      26758 ±  7%  interrupts.CPU23.CAL:Function_call_interrupts
    902141           +13.5%    1024100 ±  2%  interrupts.CPU23.LOC:Local_timer_interrupts
      3055 ±  2%     +13.8%       3478 ±  2%  interrupts.CPU23.RES:Rescheduling_interrupts
     18697 ± 10%     +42.0%      26548 ±  7%  interrupts.CPU24.CAL:Function_call_interrupts
    902132           +13.5%    1024051 ±  2%  interrupts.CPU24.LOC:Local_timer_interrupts
      2996 ±  3%     +15.3%       3454 ±  2%  interrupts.CPU24.RES:Rescheduling_interrupts
     18610 ± 10%     +42.4%      26498 ±  7%  interrupts.CPU25.CAL:Function_call_interrupts
    902099           +13.5%    1024078 ±  2%  interrupts.CPU25.LOC:Local_timer_interrupts
      3061 ±  3%     +15.4%       3531 ±  2%  interrupts.CPU25.RES:Rescheduling_interrupts
     18506 ± 10%     +42.8%      26417 ±  7%  interrupts.CPU26.CAL:Function_call_interrupts
    902116           +13.5%    1024074 ±  2%  interrupts.CPU26.LOC:Local_timer_interrupts
      3016 ±  3%     +15.6%       3488 ±  2%  interrupts.CPU26.RES:Rescheduling_interrupts
     18712 ± 11%     +40.1%      26212 ±  7%  interrupts.CPU27.CAL:Function_call_interrupts
    902131           +13.5%    1023984 ±  2%  interrupts.CPU27.LOC:Local_timer_interrupts
      3032 ±  3%     +16.4%       3528 ±  2%  interrupts.CPU27.RES:Rescheduling_interrupts
     18533 ± 10%     +41.4%      26214 ±  7%  interrupts.CPU28.CAL:Function_call_interrupts
    902108           +13.5%    1024107 ±  2%  interrupts.CPU28.LOC:Local_timer_interrupts
      3022 ±  2%     +17.1%       3539 ±  2%  interrupts.CPU28.RES:Rescheduling_interrupts
     18483 ± 10%     +41.9%      26234 ±  7%  interrupts.CPU29.CAL:Function_call_interrupts
    902262           +13.5%    1024052 ±  2%  interrupts.CPU29.LOC:Local_timer_interrupts
      3034           +16.4%       3533 ±  2%  interrupts.CPU29.RES:Rescheduling_interrupts
     17359 ±  5%     +29.3%      22454 ±  7%  interrupts.CPU3.CAL:Function_call_interrupts
    902230           +13.5%    1024080 ±  2%  interrupts.CPU3.LOC:Local_timer_interrupts
      3160 ±  4%     +22.2%       3863 ±  3%  interrupts.CPU3.RES:Rescheduling_interrupts
     18526 ± 10%     +42.2%      26340 ±  7%  interrupts.CPU30.CAL:Function_call_interrupts
    902146           +13.5%    1024088 ±  2%  interrupts.CPU30.LOC:Local_timer_interrupts
      3003           +16.6%       3503 ±  2%  interrupts.CPU30.RES:Rescheduling_interrupts
     18417 ± 10%     +42.1%      26162 ±  8%  interrupts.CPU31.CAL:Function_call_interrupts
    902108           +13.5%    1024056 ±  2%  interrupts.CPU31.LOC:Local_timer_interrupts
      2992 ±  3%     +17.2%       3506 ±  2%  interrupts.CPU31.RES:Rescheduling_interrupts
     18324 ± 10%     +41.6%      25944 ±  7%  interrupts.CPU32.CAL:Function_call_interrupts
    902092           +13.5%    1024078 ±  2%  interrupts.CPU32.LOC:Local_timer_interrupts
      3037 ±  3%     +15.2%       3498 ±  2%  interrupts.CPU32.RES:Rescheduling_interrupts
     18353 ± 11%     +42.0%      26057 ±  7%  interrupts.CPU33.CAL:Function_call_interrupts
    902085           +13.5%    1024101 ±  2%  interrupts.CPU33.LOC:Local_timer_interrupts
      3055 ±  4%     +14.6%       3502 ±  2%  interrupts.CPU33.RES:Rescheduling_interrupts
     18376 ± 10%     +41.2%      25953 ±  7%  interrupts.CPU34.CAL:Function_call_interrupts
    902095           +13.5%    1024078 ±  2%  interrupts.CPU34.LOC:Local_timer_interrupts
      3015 ±  2%     +14.4%       3448 ±  3%  interrupts.CPU34.RES:Rescheduling_interrupts
     18232 ± 10%     +42.5%      25986 ±  7%  interrupts.CPU35.CAL:Function_call_interrupts
    902197           +13.5%    1024042 ±  2%  interrupts.CPU35.LOC:Local_timer_interrupts
      2990 ±  3%     +16.6%       3487 ±  2%  interrupts.CPU35.RES:Rescheduling_interrupts
     18314 ± 10%     +41.9%      25991 ±  7%  interrupts.CPU36.CAL:Function_call_interrupts
    902046           +13.5%    1024078 ±  2%  interrupts.CPU36.LOC:Local_timer_interrupts
      3002 ±  2%     +15.9%       3479        interrupts.CPU36.RES:Rescheduling_interrupts
     18369 ± 10%     +40.6%      25829 ±  7%  interrupts.CPU37.CAL:Function_call_interrupts
    902117           +13.5%    1024026 ±  2%  interrupts.CPU37.LOC:Local_timer_interrupts
      2999 ±  2%     +16.4%       3490 ±  2%  interrupts.CPU37.RES:Rescheduling_interrupts
     18331 ± 10%     +41.5%      25932 ±  7%  interrupts.CPU38.CAL:Function_call_interrupts
    902118           +13.5%    1024085 ±  2%  interrupts.CPU38.LOC:Local_timer_interrupts
      2995 ±  3%     +16.0%       3474 ±  4%  interrupts.CPU38.RES:Rescheduling_interrupts
     18313 ± 10%     +41.6%      25935 ±  7%  interrupts.CPU39.CAL:Function_call_interrupts
    902082           +13.5%    1024103 ±  2%  interrupts.CPU39.LOC:Local_timer_interrupts
      2973 ±  2%     +18.1%       3510        interrupts.CPU39.RES:Rescheduling_interrupts
     18089 ± 10%     +20.9%      21864 ±  6%  interrupts.CPU4.CAL:Function_call_interrupts
    902120           +13.5%    1024041 ±  2%  interrupts.CPU4.LOC:Local_timer_interrupts
      3126 ±  2%     +22.9%       3842 ±  3%  interrupts.CPU4.RES:Rescheduling_interrupts
     18303 ± 10%     +42.3%      26045 ±  7%  interrupts.CPU40.CAL:Function_call_interrupts
    902120           +13.5%    1024097 ±  2%  interrupts.CPU40.LOC:Local_timer_interrupts
      3019 ±  2%     +14.7%       3462 ±  3%  interrupts.CPU40.RES:Rescheduling_interrupts
     18288 ± 10%     +42.8%      26107 ±  7%  interrupts.CPU41.CAL:Function_call_interrupts
    902064           +13.5%    1024075 ±  2%  interrupts.CPU41.LOC:Local_timer_interrupts
      2992           +16.6%       3489 ±  2%  interrupts.CPU41.RES:Rescheduling_interrupts
     18169 ± 10%     +43.3%      26042 ±  7%  interrupts.CPU42.CAL:Function_call_interrupts
    902151           +13.5%    1024091 ±  2%  interrupts.CPU42.LOC:Local_timer_interrupts
      3010 ±  2%     +16.7%       3512 ±  2%  interrupts.CPU42.RES:Rescheduling_interrupts
     18161 ± 11%     +42.4%      25871 ±  7%  interrupts.CPU43.CAL:Function_call_interrupts
    902074           +13.5%    1024076 ±  2%  interrupts.CPU43.LOC:Local_timer_interrupts
      2946 ±  2%     +18.1%       3478 ±  2%  interrupts.CPU43.RES:Rescheduling_interrupts
     16873 ±  4%     +26.9%      21415 ±  6%  interrupts.CPU44.CAL:Function_call_interrupts
    902175           +13.5%    1024065 ±  2%  interrupts.CPU44.LOC:Local_timer_interrupts
      3067 ±  4%     +21.2%       3719 ±  5%  interrupts.CPU44.RES:Rescheduling_interrupts
     16915 ±  5%     +28.3%      21704 ±  6%  interrupts.CPU45.CAL:Function_call_interrupts
    902101           +13.5%    1024072 ±  2%  interrupts.CPU45.LOC:Local_timer_interrupts
      3056 ±  3%     +22.2%       3734 ±  3%  interrupts.CPU45.RES:Rescheduling_interrupts
     16894 ±  4%     +27.6%      21553 ±  7%  interrupts.CPU46.CAL:Function_call_interrupts
    902132           +13.5%    1024049 ±  2%  interrupts.CPU46.LOC:Local_timer_interrupts
      3069 ±  2%     +22.9%       3773 ±  4%  interrupts.CPU46.RES:Rescheduling_interrupts
     16781 ±  5%     +27.6%      21408 ±  7%  interrupts.CPU47.CAL:Function_call_interrupts
    902147           +13.5%    1024088 ±  2%  interrupts.CPU47.LOC:Local_timer_interrupts
      3086 ±  3%     +20.9%       3732 ±  4%  interrupts.CPU47.RES:Rescheduling_interrupts
     16892 ±  5%     +27.3%      21496 ±  7%  interrupts.CPU48.CAL:Function_call_interrupts
    902150           +13.5%    1024084 ±  2%  interrupts.CPU48.LOC:Local_timer_interrupts
      3123 ±  4%     +19.1%       3721 ±  4%  interrupts.CPU48.RES:Rescheduling_interrupts
     16686 ±  4%     +27.9%      21340 ±  7%  interrupts.CPU49.CAL:Function_call_interrupts
    902113           +13.5%    1024070 ±  2%  interrupts.CPU49.LOC:Local_timer_interrupts
      3039 ±  4%     +22.8%       3733 ±  4%  interrupts.CPU49.RES:Rescheduling_interrupts
     17142 ±  5%     +29.2%      22150 ±  7%  interrupts.CPU5.CAL:Function_call_interrupts
    902296           +13.5%    1024251 ±  2%  interrupts.CPU5.LOC:Local_timer_interrupts
      3178 ±  3%     +21.7%       3866 ±  3%  interrupts.CPU5.RES:Rescheduling_interrupts
     16894 ±  4%     +27.4%      21529 ±  6%  interrupts.CPU50.CAL:Function_call_interrupts
    902077           +13.5%    1024082 ±  2%  interrupts.CPU50.LOC:Local_timer_interrupts
      3106 ±  4%     +21.0%       3759 ±  3%  interrupts.CPU50.RES:Rescheduling_interrupts
     16842 ±  4%     +28.0%      21565 ±  6%  interrupts.CPU51.CAL:Function_call_interrupts
    902143           +13.5%    1024096 ±  2%  interrupts.CPU51.LOC:Local_timer_interrupts
      3106 ±  3%     +20.1%       3730 ±  3%  interrupts.CPU51.RES:Rescheduling_interrupts
     16754 ±  5%     +28.8%      21571 ±  7%  interrupts.CPU52.CAL:Function_call_interrupts
    902098           +13.5%    1024069 ±  2%  interrupts.CPU52.LOC:Local_timer_interrupts
      3050 ±  2%     +21.8%       3715 ±  4%  interrupts.CPU52.RES:Rescheduling_interrupts
     16868 ±  5%     +27.3%      21481 ±  6%  interrupts.CPU53.CAL:Function_call_interrupts
    902150           +13.5%    1024062 ±  2%  interrupts.CPU53.LOC:Local_timer_interrupts
      3054 ±  3%     +22.1%       3729 ±  4%  interrupts.CPU53.RES:Rescheduling_interrupts
     16759 ±  5%     +27.2%      21311 ±  7%  interrupts.CPU54.CAL:Function_call_interrupts
    902146           +13.5%    1024081 ±  2%  interrupts.CPU54.LOC:Local_timer_interrupts
      3061 ±  4%     +23.5%       3780 ±  5%  interrupts.CPU54.RES:Rescheduling_interrupts
     16893 ±  4%     +26.4%      21358 ±  7%  interrupts.CPU55.CAL:Function_call_interrupts
    902113           +13.5%    1024081 ±  2%  interrupts.CPU55.LOC:Local_timer_interrupts
      3073 ±  3%     +21.8%       3742 ±  5%  interrupts.CPU55.RES:Rescheduling_interrupts
     16820 ±  4%     +26.6%      21288 ±  7%  interrupts.CPU56.CAL:Function_call_interrupts
    902136           +13.5%    1024085 ±  2%  interrupts.CPU56.LOC:Local_timer_interrupts
      3052 ±  4%     +22.3%       3734 ±  5%  interrupts.CPU56.RES:Rescheduling_interrupts
     16837 ±  4%     +26.6%      21324 ±  7%  interrupts.CPU57.CAL:Function_call_interrupts
    902135           +13.5%    1024226 ±  2%  interrupts.CPU57.LOC:Local_timer_interrupts
      3075 ±  3%     +21.1%       3723 ±  3%  interrupts.CPU57.RES:Rescheduling_interrupts
     16851 ±  5%     +26.7%      21359 ±  7%  interrupts.CPU58.CAL:Function_call_interrupts
    902145           +13.5%    1024084 ±  2%  interrupts.CPU58.LOC:Local_timer_interrupts
      3067 ±  3%     +22.0%       3742 ±  4%  interrupts.CPU58.RES:Rescheduling_interrupts
     16813 ±  4%     +27.8%      21480 ±  7%  interrupts.CPU59.CAL:Function_call_interrupts
    902135           +13.5%    1024105 ±  2%  interrupts.CPU59.LOC:Local_timer_interrupts
      3075 ±  3%     +23.1%       3784 ±  5%  interrupts.CPU59.RES:Rescheduling_interrupts
     17318 ±  4%     +25.8%      21778 ±  6%  interrupts.CPU6.CAL:Function_call_interrupts
    902058           +13.5%    1024091 ±  2%  interrupts.CPU6.LOC:Local_timer_interrupts
      3167 ±  4%     +22.1%       3866 ±  4%  interrupts.CPU6.RES:Rescheduling_interrupts
     16910 ±  4%     +26.7%      21430 ±  7%  interrupts.CPU60.CAL:Function_call_interrupts
    902154           +13.5%    1024028 ±  2%  interrupts.CPU60.LOC:Local_timer_interrupts
      3097 ±  3%     +21.4%       3760 ±  4%  interrupts.CPU60.RES:Rescheduling_interrupts
     16900 ±  4%     +27.5%      21555 ±  7%  interrupts.CPU61.CAL:Function_call_interrupts
    902139           +13.5%    1024089 ±  2%  interrupts.CPU61.LOC:Local_timer_interrupts
      3063 ±  4%     +22.0%       3738 ±  3%  interrupts.CPU61.RES:Rescheduling_interrupts
     16991 ±  4%     +26.0%      21414 ±  7%  interrupts.CPU62.CAL:Function_call_interrupts
    902132           +13.5%    1024089 ±  2%  interrupts.CPU62.LOC:Local_timer_interrupts
      3125 ±  3%     +19.7%       3742 ±  3%  interrupts.CPU62.RES:Rescheduling_interrupts
     16967 ±  5%     +26.9%      21537 ±  6%  interrupts.CPU63.CAL:Function_call_interrupts
    902111           +13.5%    1024071 ±  2%  interrupts.CPU63.LOC:Local_timer_interrupts
      3110 ±  3%     +22.1%       3799 ±  4%  interrupts.CPU63.RES:Rescheduling_interrupts
     16832 ±  4%     +28.1%      21557 ±  7%  interrupts.CPU64.CAL:Function_call_interrupts
    902165           +13.5%    1024037 ±  2%  interrupts.CPU64.LOC:Local_timer_interrupts
      3095 ±  3%     +22.5%       3791 ±  3%  interrupts.CPU64.RES:Rescheduling_interrupts
     16887 ±  5%     +27.7%      21563 ±  7%  interrupts.CPU65.CAL:Function_call_interrupts
    902159           +13.5%    1024083 ±  2%  interrupts.CPU65.LOC:Local_timer_interrupts
      3079 ±  4%     +23.5%       3802 ±  4%  interrupts.CPU65.RES:Rescheduling_interrupts
     18236 ± 10%     +43.5%      26177 ±  8%  interrupts.CPU66.CAL:Function_call_interrupts
    902097           +13.5%    1023991 ±  2%  interrupts.CPU66.LOC:Local_timer_interrupts
      2952 ±  3%     +16.0%       3423 ±  3%  interrupts.CPU66.RES:Rescheduling_interrupts
     18440 ± 11%     +41.6%      26117 ±  7%  interrupts.CPU67.CAL:Function_call_interrupts
    902050           +13.5%    1024090 ±  2%  interrupts.CPU67.LOC:Local_timer_interrupts
      2948 ±  2%     +16.8%       3443 ±  3%  interrupts.CPU67.RES:Rescheduling_interrupts
     18284 ± 10%     +43.0%      26148 ±  7%  interrupts.CPU68.CAL:Function_call_interrupts
    902119           +13.5%    1024096 ±  2%  interrupts.CPU68.LOC:Local_timer_interrupts
      2960 ±  2%     +17.7%       3485 ±  3%  interrupts.CPU68.RES:Rescheduling_interrupts
     18357 ± 10%     +42.0%      26070 ±  7%  interrupts.CPU69.CAL:Function_call_interrupts
    902096           +13.5%    1024089 ±  2%  interrupts.CPU69.LOC:Local_timer_interrupts
      2977           +16.1%       3455 ±  5%  interrupts.CPU69.RES:Rescheduling_interrupts
     16984 ±  5%     +28.6%      21843 ±  6%  interrupts.CPU7.CAL:Function_call_interrupts
    902119           +13.5%    1024116 ±  2%  interrupts.CPU7.LOC:Local_timer_interrupts
      3122 ±  2%     +22.8%       3835 ±  4%  interrupts.CPU7.RES:Rescheduling_interrupts
     18176 ± 10%     +43.4%      26056 ±  7%  interrupts.CPU70.CAL:Function_call_interrupts
    902106           +13.5%    1024063 ±  2%  interrupts.CPU70.LOC:Local_timer_interrupts
      2937 ±  2%     +18.5%       3479 ±  2%  interrupts.CPU70.RES:Rescheduling_interrupts
     18320 ± 10%     +42.2%      26047 ±  8%  interrupts.CPU71.CAL:Function_call_interrupts
    902142           +13.5%    1024123 ±  2%  interrupts.CPU71.LOC:Local_timer_interrupts
      2999 ±  3%     +16.3%       3487 ±  3%  interrupts.CPU71.RES:Rescheduling_interrupts
     18410 ± 10%     +42.0%      26143 ±  7%  interrupts.CPU72.CAL:Function_call_interrupts
    902129           +13.5%    1024094 ±  2%  interrupts.CPU72.LOC:Local_timer_interrupts
      2965 ±  3%     +17.6%       3488 ±  4%  interrupts.CPU72.RES:Rescheduling_interrupts
     18305 ± 10%     +42.5%      26085 ±  8%  interrupts.CPU73.CAL:Function_call_interrupts
    902133           +13.5%    1024041 ±  2%  interrupts.CPU73.LOC:Local_timer_interrupts
      2977 ±  3%     +16.6%       3471 ±  3%  interrupts.CPU73.RES:Rescheduling_interrupts
     18344 ± 10%     +42.6%      26152 ±  7%  interrupts.CPU74.CAL:Function_call_interrupts
    902128           +13.5%    1024065 ±  2%  interrupts.CPU74.LOC:Local_timer_interrupts
      3002 ±  2%     +17.0%       3511 ±  3%  interrupts.CPU74.RES:Rescheduling_interrupts
     18310 ± 10%     +41.9%      25976 ±  7%  interrupts.CPU75.CAL:Function_call_interrupts
    902143           +13.5%    1024072 ±  2%  interrupts.CPU75.LOC:Local_timer_interrupts
      2922 ±  3%     +18.5%       3463 ±  3%  interrupts.CPU75.RES:Rescheduling_interrupts
     18275 ± 10%     +41.6%      25883 ±  7%  interrupts.CPU76.CAL:Function_call_interrupts
    902099           +13.5%    1024098 ±  2%  interrupts.CPU76.LOC:Local_timer_interrupts
      2921 ±  4%     +18.3%       3457 ±  2%  interrupts.CPU76.RES:Rescheduling_interrupts
     18330 ± 10%     +42.1%      26054 ±  7%  interrupts.CPU77.CAL:Function_call_interrupts
    902158           +13.5%    1024103 ±  2%  interrupts.CPU77.LOC:Local_timer_interrupts
      2960 ±  3%     +16.4%       3447 ±  3%  interrupts.CPU77.RES:Rescheduling_interrupts
     18199 ± 10%     +43.9%      26183 ±  8%  interrupts.CPU78.CAL:Function_call_interrupts
    902127           +13.5%    1024113 ±  2%  interrupts.CPU78.LOC:Local_timer_interrupts
      2936 ±  3%     +17.8%       3460 ±  2%  interrupts.CPU78.RES:Rescheduling_interrupts
     18302 ±  9%     +42.2%      26018 ±  7%  interrupts.CPU79.CAL:Function_call_interrupts
    902128           +13.5%    1024093 ±  2%  interrupts.CPU79.LOC:Local_timer_interrupts
      2933 ±  3%     +16.1%       3406 ±  2%  interrupts.CPU79.RES:Rescheduling_interrupts
     17257 ±  4%     +26.1%      21766 ±  6%  interrupts.CPU8.CAL:Function_call_interrupts
    902064           +13.5%    1024249 ±  2%  interrupts.CPU8.LOC:Local_timer_interrupts
      3148 ±  4%     +22.8%       3867 ±  4%  interrupts.CPU8.RES:Rescheduling_interrupts
     18351 ± 10%     +41.6%      25983 ±  8%  interrupts.CPU80.CAL:Function_call_interrupts
    902119           +13.5%    1024104 ±  2%  interrupts.CPU80.LOC:Local_timer_interrupts
      2950 ±  2%     +15.0%       3393 ±  2%  interrupts.CPU80.RES:Rescheduling_interrupts
     18351 ± 10%     +42.5%      26147 ±  8%  interrupts.CPU81.CAL:Function_call_interrupts
    902138           +13.5%    1024087 ±  2%  interrupts.CPU81.LOC:Local_timer_interrupts
      2952 ±  2%     +16.0%       3424 ±  2%  interrupts.CPU81.RES:Rescheduling_interrupts
     18259 ± 10%     +42.4%      26004 ±  7%  interrupts.CPU82.CAL:Function_call_interrupts
    902149           +13.5%    1024272 ±  2%  interrupts.CPU82.LOC:Local_timer_interrupts
      2931 ±  2%     +17.0%       3429 ±  2%  interrupts.CPU82.RES:Rescheduling_interrupts
     18210 ± 10%     +42.8%      26012 ±  8%  interrupts.CPU83.CAL:Function_call_interrupts
    902111           +13.5%    1024099 ±  2%  interrupts.CPU83.LOC:Local_timer_interrupts
      2933 ±  2%     +16.6%       3419 ±  2%  interrupts.CPU83.RES:Rescheduling_interrupts
     18085 ± 10%     +44.2%      26086 ±  8%  interrupts.CPU84.CAL:Function_call_interrupts
    902118           +13.5%    1024109 ±  2%  interrupts.CPU84.LOC:Local_timer_interrupts
      2981 ±  2%     +13.9%       3396 ±  2%  interrupts.CPU84.RES:Rescheduling_interrupts
     18294 ± 10%     +43.0%      26167 ±  7%  interrupts.CPU85.CAL:Function_call_interrupts
    902135           +13.5%    1024098 ±  2%  interrupts.CPU85.LOC:Local_timer_interrupts
      2937 ±  2%     +16.8%       3430 ±  3%  interrupts.CPU85.RES:Rescheduling_interrupts
     18253 ± 10%     +42.7%      26056 ±  8%  interrupts.CPU86.CAL:Function_call_interrupts
    902302           +13.5%    1024102 ±  2%  interrupts.CPU86.LOC:Local_timer_interrupts
      2999 ±  2%     +13.5%       3403 ±  3%  interrupts.CPU86.RES:Rescheduling_interrupts
     18248 ± 10%     +41.5%      25817 ±  8%  interrupts.CPU87.CAL:Function_call_interrupts
    902127           +13.5%    1024064 ±  2%  interrupts.CPU87.LOC:Local_timer_interrupts
      2954 ±  2%     +14.3%       3377 ±  2%  interrupts.CPU87.RES:Rescheduling_interrupts
     16991 ±  4%     +26.9%      21557 ±  6%  interrupts.CPU9.CAL:Function_call_interrupts
    902162           +13.5%    1024110 ±  2%  interrupts.CPU9.LOC:Local_timer_interrupts
      3164 ±  3%     +22.5%       3876 ±  5%  interrupts.CPU9.RES:Rescheduling_interrupts
  79387471           +13.5%   90119618 ±  2%  interrupts.LOC:Local_timer_interrupts
    269188           +19.2%     320766 ±  2%  interrupts.RES:Rescheduling_interrupts


                                                                                
                                  aim7.jobs-per-min                             
                                                                                
  14000 +-------------------------------------------------------------------+   
        |        +                                                          |   
  13500 |-+     + +   ++   +  +.+  +   +                     +        .+    |   
        |     .+   + +  : :: :  :  :: : :.+.       +    +    :     .++  + +.|   
        |. +.+      +   : : ::   : :: : +   ++.+. + +. + +  ::   .+      +  |   
  13000 |-+              +  +    ::  +           +    +   + ::  +           |   
        |                         :                        +  : :           |   
  12500 |-+                       +                           ::            |   
        |                                                     ::            |   
  12000 |-+    O           O  O    O    O      O   O          +             |   
        | O       O O       O               O           O                   |   
        |    O        O  O        O    O  O      O  O                       |   
  11500 |-+O                    O    O       O        O                     |   
        |        O     O                                                    |   
  11000 +-------------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample

***************************************************************************************************
lkp-knl-f1: 256 threads Intel(R) Genuine Intel(R) CPU 0000 @ 1.30GHz with 112G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/rootfs/tbox_group/test/testcase/testtime/ucode:
  gcc-9/performance/x86_64-rhel-8.3/debian-10.4-x86_64-20200603.cgz/lkp-knl-f1/all/aim9/5s/0xffff0190

commit: 
  4392fbc4ba ("ext4: drop sync argument of ext4_commit_super()")
  05c2c00f37 ("ext4: protect superblock modifications with a buffer lock")

4392fbc4bab57db3 05c2c00f3769abb9e323fcaca70 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :5           20%           1:6     dmesg.WARNING:missing_R10_value_at__fsnotify_parent/0x
          0:5            4%           0:6     perf-profile.children.cycles-pp.error_return
          2:5            8%           2:6     perf-profile.self.cycles-pp.error_entry
         %stddev     %change         %stddev
             \          |                \  
     74384            +3.4%      76883        aim9.dgram_pipe.ops_per_sec
    121865            +2.6%     125020        aim9.disk_cp.ops_per_sec
    198997            +1.9%     202688        aim9.disk_wrt.ops_per_sec
     52029            -3.9%      49990        aim9.link_test.ops_per_sec
     95645            +7.5%     102865        aim9.signal_test.ops_per_sec
     80028            +5.0%      84023        aim9.stream_pipe.ops_per_sec
    101888            +3.9%     105872        aim9.sync_disk_rw.ops_per_sec
     17794            +1.5%      18069        aim9.tcp_test.ops_per_sec
     35329            +2.1%      36070        aim9.udp_test.ops_per_sec
     84505            +1.3%      85562        proc-vmstat.pgactivate
 1.037e+08            -6.0%   97479544 ±  6%  perf-stat.i.cache-references
 1.034e+08            -6.0%   97161961 ±  6%  perf-stat.ps.cache-references
    412.50 ±  3%     +25.8%     519.03 ±  9%  sched_debug.cfs_rq:/.util_est_enqueued.max
     45.99 ± 10%     +31.6%      60.51 ± 11%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
     20873 ± 13%     -16.2%      17493 ±  6%  softirqs.CPU134.RCU
     10054 ± 56%     -81.3%       1880 ±126%  softirqs.CPU14.NET_RX
      0.05 ±  6%     +45.0%       0.08 ± 28%  perf-sched.sch_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      0.04 ±  4%     +22.3%       0.05 ± 10%  perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      0.01 ±123%    +153.5%       0.04 ± 24%  perf-sched.sch_delay.avg.ms.schedule_timeout.wait_for_completion.__flush_work.lru_add_drain_all
      0.09 ± 18%     +82.9%       0.17 ± 52%  perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
      0.01 ±123%    +175.8%       0.04 ± 12%  perf-sched.sch_delay.max.ms.schedule_timeout.wait_for_completion.__flush_work.lru_add_drain_all
    141.15 ± 21%     -41.1%      83.12 ± 41%  perf-sched.wait_and_delay.max.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
      0.18 ± 40%     +70.5%       0.31 ± 41%  perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.dput.path_put.set_fs_pwd
      0.23 ± 33%     +59.8%       0.37 ± 26%  perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.kmem_cache_alloc.getname_flags.__x64_sys_unlink
    141.12 ± 21%     -41.1%      83.09 ± 41%  perf-sched.wait_time.max.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
    305.60 ± 29%     -43.3%     173.17 ± 21%  interrupts.CPU114.NMI:Non-maskable_interrupts
    305.60 ± 29%     -43.3%     173.17 ± 21%  interrupts.CPU114.PMI:Performance_monitoring_interrupts
    335.00 ± 20%     -36.6%     212.33 ± 32%  interrupts.CPU239.NMI:Non-maskable_interrupts
    335.00 ± 20%     -36.6%     212.33 ± 32%  interrupts.CPU239.PMI:Performance_monitoring_interrupts
    314.80 ± 32%     -37.0%     198.33 ± 46%  interrupts.CPU251.NMI:Non-maskable_interrupts
    314.80 ± 32%     -37.0%     198.33 ± 46%  interrupts.CPU251.PMI:Performance_monitoring_interrupts
     23.20 ± 44%    +117.7%      50.50 ± 57%  interrupts.CPU53.RES:Rescheduling_interrupts
      1417 ± 10%     +24.6%       1766 ± 13%  interrupts.CPU6.CAL:Function_call_interrupts
     30.40 ± 59%    +198.8%      90.83 ± 23%  interrupts.CPU6.RES:Rescheduling_interrupts
      1260 ±  5%      +9.3%       1377 ±  7%  interrupts.CPU7.CAL:Function_call_interrupts
    253.40 ± 29%     +46.7%     371.83 ± 27%  interrupts.CPU90.NMI:Non-maskable_interrupts
    253.40 ± 29%     +46.7%     371.83 ± 27%  interrupts.CPU90.PMI:Performance_monitoring_interrupts
      1.07 ±  2%      -0.2        0.86 ± 24%  perf-profile.calltrace.cycles-pp.irqtime_account_irq.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
      0.75 ±  4%      +0.3        1.03 ± 34%  perf-profile.calltrace.cycles-pp.timerqueue_add.enqueue_hrtimer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
      0.90 ±  5%      +0.3        1.21 ± 33%  perf-profile.calltrace.cycles-pp.enqueue_hrtimer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack
      2.40 ±  6%      +0.4        2.76 ±  9%  perf-profile.calltrace.cycles-pp.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack.sysvec_apic_timer_interrupt
      1.42 ±  7%      +0.4        1.84 ± 14%  perf-profile.calltrace.cycles-pp.ktime_get.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack
      0.33 ±  7%      -0.1        0.25 ± 11%  perf-profile.children.cycles-pp.irq_work_needs_cpu
      0.09 ± 13%      +0.0        0.11 ±  8%  perf-profile.children.cycles-pp.hrtimer_run_queues
      0.09 ±  9%      +0.0        0.12 ± 19%  perf-profile.children.cycles-pp.__set_cpus_allowed_ptr
      0.06 ± 12%      +0.0        0.09 ± 13%  perf-profile.children.cycles-pp.irq_work_single
      0.06 ± 12%      +0.0        0.09 ± 13%  perf-profile.children.cycles-pp.asm_sysvec_irq_work
      0.06 ± 12%      +0.0        0.09 ± 13%  perf-profile.children.cycles-pp.sysvec_irq_work
      0.06 ± 12%      +0.0        0.09 ± 13%  perf-profile.children.cycles-pp.__sysvec_irq_work
      0.06 ± 12%      +0.0        0.09 ± 13%  perf-profile.children.cycles-pp.irq_work_run
      0.11 ±  8%      +0.0        0.15 ± 22%  perf-profile.children.cycles-pp.__x64_sys_sched_setaffinity
      0.14 ±  5%      +0.1        0.19 ± 17%  perf-profile.children.cycles-pp.irq_work_run_list
      0.29 ±  6%      +0.1        0.39 ± 28%  perf-profile.children.cycles-pp.rb_insert_color
      0.77 ±  4%      +0.3        1.07 ± 34%  perf-profile.children.cycles-pp.timerqueue_add
      0.92 ±  5%      +0.3        1.23 ± 32%  perf-profile.children.cycles-pp.enqueue_hrtimer
      2.45 ±  6%      +0.4        2.82 ±  9%  perf-profile.children.cycles-pp.clockevents_program_event
      4.23 ±  2%      +0.4        4.61 ±  7%  perf-profile.children.cycles-pp.scheduler_tick
      0.97 ±  2%      -0.1        0.83 ± 15%  perf-profile.self.cycles-pp.irqtime_account_irq
      0.75 ±  4%      -0.1        0.67 ±  6%  perf-profile.self.cycles-pp.__sysvec_apic_timer_interrupt
      0.30 ±  8%      -0.1        0.23 ± 15%  perf-profile.self.cycles-pp.irq_work_needs_cpu
      0.19 ±  3%      +0.0        0.21 ±  6%  perf-profile.self.cycles-pp.trigger_load_balance
      0.09 ± 13%      +0.0        0.11 ±  8%  perf-profile.self.cycles-pp.hrtimer_run_queues
      0.02 ±122%      +0.0        0.06 ± 14%  perf-profile.self.cycles-pp.io_serial_in
      0.14 ± 11%      +0.0        0.19 ± 22%  perf-profile.self.cycles-pp.tick_sched_handle
      0.27 ±  4%      +0.1        0.37 ± 27%  perf-profile.self.cycles-pp.rb_insert_color





Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Oliver Sang


View attachment "config-5.10.0-rc5-00032-g05c2c00f3769" of type "text/plain" (171050 bytes)

View attachment "job-script" of type "text/plain" (8055 bytes)

View attachment "job.yaml" of type "text/plain" (5554 bytes)

View attachment "reproduce" of type "text/plain" (1011 bytes)

Powered by blists - more mailing lists