lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Date:   Thu, 27 Feb 2020 11:08:26 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     Theodore Ts'o <tytso@....edu>
Cc:     LKML <linux-kernel@...r.kernel.org>,
        Linus Torvalds <torvalds@...ux-foundation.org>,
        lkp@...ts.01.org
Subject: [ext4] 244adf6426: fsmark.files_per_sec 17.5% improvement

Greeting,

FYI, we noticed a 17.5% improvement of fsmark.files_per_sec due to commit:


commit: 244adf6426ee31a83f397b700d964cff12a247d3 ("ext4: make dioread_nolock the default")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master

in testcase: fsmark
on test machine: 192 threads Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory
with following parameters:

	iterations: 1x
	nr_threads: 64t
	disk: 1BRD_48G
	fs: ext4
	fs2: nfsv4
	filesize: 4M
	test_size: 40G
	sync_method: NoSync
	cpufreq_governor: performance
	ucode: 0x500002c

test-description: The fsmark is a file system benchmark to test synchronous write workloads, for example, mail servers workload.
test-url: https://sourceforge.net/projects/fsmark/





Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
compiler/cpufreq_governor/disk/filesize/fs2/fs/iterations/kconfig/nr_threads/rootfs/sync_method/tbox_group/test_size/testcase/ucode:
  gcc-7/performance/1BRD_48G/4M/nfsv4/ext4/1x/x86_64-rhel-7.6/64t/debian-x86_64-20191114.cgz/NoSync/lkp-csl-2ap2/40G/fsmark/0x500002c

commit: 
  4068664e3c ("ext4: fix extent_status fragmentation for plain files")
  244adf6426 ("ext4: make dioread_nolock the default")

4068664e3cd23126 244adf6426ee31a83f397b700d9 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :4           25%           1:4     dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x
           :4           25%           1:4     kmsg.ipmi_si_dmi-ipmi-si.#:IRQ_index#not_found
         %stddev     %change         %stddev
             \          |                \  
    481.65           +17.5%     565.93        fsmark.files_per_sec
     23.95           -13.3%      20.77        fsmark.time.elapsed_time
     23.95           -13.3%      20.77        fsmark.time.elapsed_time.max
    175.00           +16.2%     203.33        fsmark.time.percent_of_cpu_this_job_got
   5233562           -12.7%    4570503        turbostat.IRQ
   3643143           -11.3%    3232826        cpuidle.POLL.time
   2028059 ±  2%     -16.8%    1687836        cpuidle.POLL.usage
      1.31 ±  4%     +23.7%       1.62 ±  2%  iostat.cpu.iowait
      3.38 ±  2%     +13.4%       3.83        iostat.cpu.system
      1.40 ±  4%      +0.4        1.75 ±  2%  mpstat.cpu.all.iowait%
      3.54            +0.5        4.05        mpstat.cpu.all.sys%
   1603302           +15.7%    1854829        vmstat.io.bo
     58893           +25.3%      73786        vmstat.memory.buff
     59287           +23.5%      73243        meminfo.Active(file)
     57225           +24.9%      71463        meminfo.Buffers
   4992907 ±  2%     +16.4%    5809826 ±  3%  meminfo.max_used_kB
  12229730 ± 26%     -38.8%    7486409 ± 48%  numa-numastat.node1.local_node
  12245297 ± 26%     -38.6%    7517445 ± 48%  numa-numastat.node1.numa_hit
     27932 ± 19%     -63.0%      10345 ±141%  numa-numastat.node2.other_node
      6244 ± 11%     +34.2%       8381 ±  9%  slabinfo.eventpoll_pwq.active_objs
      6244 ± 11%     +34.2%       8381 ±  9%  slabinfo.eventpoll_pwq.num_objs
      1270 ±  4%     -17.4%       1049 ± 19%  slabinfo.task_group.active_objs
      1270 ±  4%     -17.4%       1049 ± 19%  slabinfo.task_group.num_objs
     14831           +23.6%      18328        proc-vmstat.nr_active_file
      4388 ± 10%     +28.7%       5645 ± 10%  proc-vmstat.nr_dirty
      4231 ±  9%     +37.1%       5802 ± 17%  proc-vmstat.nr_writeback
     14831           +23.6%      18328        proc-vmstat.nr_zone_active_file
      9891 ±  7%     +32.2%      13074 ± 10%  proc-vmstat.nr_zone_write_pending
      6462 ±  2%     +30.3%       8421 ±  2%  proc-vmstat.pgactivate
    224362 ±  3%      -5.7%     211612        proc-vmstat.pgfault
     14675 ± 96%    +236.8%      49428 ±  4%  numa-meminfo.node0.Active(file)
      6128 ± 59%     -82.0%       1101 ± 54%  numa-meminfo.node0.Inactive(anon)
      6623 ± 56%     -79.0%       1389 ± 38%  numa-meminfo.node0.Shmem
      3360 ± 16%     +67.1%       5615 ± 18%  numa-meminfo.node0.Writeback
     90095 ± 53%    +111.9%     190926 ± 33%  numa-meminfo.node1.Active
     80047 ± 68%    +134.5%     187704 ± 33%  numa-meminfo.node1.Active(anon)
     79814 ± 68%    +135.2%     187754 ± 33%  numa-meminfo.node1.AnonPages
     16510 ± 17%     -49.1%       8405 ± 84%  numa-meminfo.node2.Active(file)
      2785 ±173%    +206.7%       8543 ± 58%  numa-meminfo.node2.AnonHugePages
      1.22 ± 13%     +29.3%       1.58 ±  2%  sched_debug.cfs_rq:/.exec_clock.stddev
      2566 ± 84%    +263.1%       9319 ± 57%  sched_debug.cfs_rq:/.load.avg
     23092 ±131%    +297.7%      91842 ± 67%  sched_debug.cfs_rq:/.load.stddev
      0.22 ± 13%     +24.2%       0.28 ±  5%  sched_debug.cfs_rq:/.nr_running.stddev
      0.92 ± 21%     +37.9%       1.26 ±  2%  sched_debug.cfs_rq:/.runnable_load_avg.avg
      4.34 ± 28%     +36.7%       5.93 ± 21%  sched_debug.cfs_rq:/.runnable_load_avg.stddev
      2304 ± 94%    +291.8%       9027 ± 60%  sched_debug.cfs_rq:/.runnable_weight.avg
     22073 ±140%    +312.4%      91023 ± 68%  sched_debug.cfs_rq:/.runnable_weight.stddev
     -2017          -481.0%       7684 ± 56%  sched_debug.cfs_rq:/.spread0.avg
     30572 ± 21%     +56.4%      47811 ± 16%  sched_debug.cfs_rq:/.spread0.max
      1130 ±  5%     -16.8%     940.29 ±  2%  sched_debug.cpu.nr_switches.stddev
      3674 ± 96%    +237.0%      12382 ±  5%  numa-vmstat.node0.nr_active_file
    952.50 ± 46%     +66.7%       1588 ±  5%  numa-vmstat.node0.nr_dirty
      1532 ± 59%     -82.0%     275.00 ± 55%  numa-vmstat.node0.nr_inactive_anon
      1655 ± 56%     -79.0%     347.33 ± 38%  numa-vmstat.node0.nr_shmem
    783.75 ± 12%     +72.8%       1354 ±  4%  numa-vmstat.node0.nr_writeback
      3674 ± 96%    +237.0%      12382 ±  5%  numa-vmstat.node0.nr_zone_active_file
      1532 ± 59%     -82.0%     275.00 ± 55%  numa-vmstat.node0.nr_zone_inactive_anon
      2105 ± 20%     +53.5%       3232 ±  7%  numa-vmstat.node0.nr_zone_write_pending
     20019 ± 68%    +134.5%      46954 ± 33%  numa-vmstat.node1.nr_active_anon
     19959 ± 68%    +135.4%      46973 ± 33%  numa-vmstat.node1.nr_anon_pages
     20019 ± 68%    +134.5%      46954 ± 33%  numa-vmstat.node1.nr_zone_active_anon
      4136 ± 17%     -49.1%       2103 ± 84%  numa-vmstat.node2.nr_active_file
      4136 ± 17%     -49.1%       2103 ± 84%  numa-vmstat.node2.nr_zone_active_file
      2358 ± 10%     +24.9%       2945 ±  8%  numa-vmstat.node2.nr_zone_write_pending
  10683551 ± 14%     -29.6%    7523475 ± 36%  numa-vmstat.node2.numa_hit
  10568791 ± 14%     -29.7%    7425319 ± 36%  numa-vmstat.node2.numa_local
    114783 ±  4%     -14.5%      98177 ± 13%  numa-vmstat.node2.numa_other
 3.936e+09            +9.4%  4.305e+09        perf-stat.i.branch-instructions
      2.25 ±  3%      -0.1        2.14 ±  2%  perf-stat.i.branch-miss-rate%
  86221130            +3.1%   88906985        perf-stat.i.branch-misses
 1.575e+08 ±  2%     +18.9%  1.872e+08        perf-stat.i.cache-misses
 3.208e+08           +14.3%  3.667e+08        perf-stat.i.cache-references
 3.647e+10            +9.6%  3.998e+10        perf-stat.i.cpu-cycles
    693.89 ±  2%     +34.8%     935.60 ±  3%  perf-stat.i.cpu-migrations
 4.983e+09           +10.7%  5.518e+09 ±  2%  perf-stat.i.dTLB-loads
 2.496e+09           +11.6%  2.786e+09        perf-stat.i.dTLB-stores
     54.41            +0.8       55.19        perf-stat.i.iTLB-load-miss-rate%
   5583686            +7.1%    5978591        perf-stat.i.iTLB-load-misses
   4621794            +3.4%    4776662        perf-stat.i.iTLB-loads
  1.95e+10            +9.6%  2.138e+10        perf-stat.i.instructions
      8743 ±  4%      +8.7%       9501        perf-stat.i.minor-faults
     83.77            -2.1       81.69 ±  2%  perf-stat.i.node-load-miss-rate%
  36422751 ±  2%     +19.3%   43442066        perf-stat.i.node-load-misses
   7033122 ±  8%     +29.0%    9069997 ±  9%  perf-stat.i.node-loads
      8743 ±  4%      +8.7%       9501        perf-stat.i.page-faults
     16.46 ±  2%      +4.2%      17.15        perf-stat.overall.MPKI
      2.19 ±  2%      -0.1        2.06        perf-stat.overall.branch-miss-rate%
     49.09 ±  2%      +2.0       51.05        perf-stat.overall.cache-miss-rate%
    231.69 ±  2%      -7.8%     213.56        perf-stat.overall.cycles-between-cache-misses
     54.72            +0.9       55.59        perf-stat.overall.iTLB-load-miss-rate%
      3491            +2.4%       3576        perf-stat.overall.instructions-per-iTLB-miss
 3.774e+09            +8.7%  4.103e+09        perf-stat.ps.branch-instructions
  1.51e+08 ±  2%     +18.1%  1.784e+08        perf-stat.ps.cache-misses
 3.076e+08 ±  2%     +13.6%  3.495e+08        perf-stat.ps.cache-references
 3.497e+10            +8.9%   3.81e+10        perf-stat.ps.cpu-cycles
    665.73 ±  2%     +34.0%     892.00 ±  3%  perf-stat.ps.cpu-migrations
 4.777e+09           +10.1%  5.259e+09 ±  2%  perf-stat.ps.dTLB-loads
 2.393e+09           +11.0%  2.655e+09        perf-stat.ps.dTLB-stores
   5354036            +6.4%    5697568        perf-stat.ps.iTLB-load-misses
   4431530            +2.7%    4551699        perf-stat.ps.iTLB-loads
  1.87e+10            +9.0%  2.038e+10        perf-stat.ps.instructions
  34920927 ±  2%     +18.6%   41404907        perf-stat.ps.node-load-misses
   6743930 ±  8%     +28.2%    8645491 ±  9%  perf-stat.ps.node-loads
   4.6e+11            -5.4%   4.35e+11        perf-stat.total.instructions
     16.26 ±  3%      -1.6       14.61        perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     12.22            -1.0       11.20        perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle
      1.71 ± 18%      -0.6        1.11 ± 20%  perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.ext4_buffered_write_iter.ext4_file_write_iter.do_iter_readv_writev.do_iter_write
      1.70 ± 18%      -0.6        1.11 ± 19%  perf-profile.calltrace.cycles-pp.rwsem_optimistic_spin.rwsem_down_write_slowpath.ext4_buffered_write_iter.ext4_file_write_iter.do_iter_readv_writev
      6.26 ±  2%      -0.5        5.72 ±  4%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      1.34 ± 15%      -0.4        0.94 ± 16%  perf-profile.calltrace.cycles-pp.rwsem_spin_on_owner.rwsem_optimistic_spin.rwsem_down_write_slowpath.ext4_buffered_write_iter.ext4_file_write_iter
      4.66 ±  2%      -0.3        4.31        perf-profile.calltrace.cycles-pp.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      3.40 ±  2%      -0.3        3.11 ±  2%  perf-profile.calltrace.cycles-pp.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
      3.20 ±  2%      -0.2        2.97 ±  4%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
      2.33            -0.2        2.11 ±  7%  perf-profile.calltrace.cycles-pp.clockevents_program_event.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
      2.10 ±  2%      -0.2        1.89 ±  4%  perf-profile.calltrace.cycles-pp.rcu_core.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt
      1.97 ±  2%      -0.2        1.77 ±  9%  perf-profile.calltrace.cycles-pp.ktime_get.clockevents_program_event.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      1.49 ±  3%      -0.2        1.31 ±  6%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.rcu_core.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt
      1.16 ±  3%      -0.1        1.01 ±  7%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.rcu_core.__softirqentry_text_start.irq_exit
      0.64            +0.1        0.70 ±  8%  perf-profile.calltrace.cycles-pp.end_page_writeback.nfs_end_page_writeback.nfs_write_completion.rpc_free_task.rpc_async_release
      0.80            +0.1        0.87 ±  6%  perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.wake_up_page_bit.nfs_end_page_writeback.nfs_write_completion
      0.80 ±  2%      +0.1        0.87 ±  6%  perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.wake_up_page_bit.nfs_end_page_writeback
      0.83            +0.1        0.91 ±  5%  perf-profile.calltrace.cycles-pp.__wake_up_common.wake_up_page_bit.nfs_end_page_writeback.nfs_write_completion.rpc_free_task
      0.92 ±  3%      +0.1        1.02 ±  5%  perf-profile.calltrace.cycles-pp.pagecache_get_page.grab_cache_page_write_begin.nfs_write_begin.generic_perform_write.nfs_file_write
      0.94 ±  2%      +0.1        1.04 ±  5%  perf-profile.calltrace.cycles-pp.wake_up_page_bit.nfs_end_page_writeback.nfs_write_completion.rpc_free_task.rpc_async_release
      0.93 ±  3%      +0.1        1.03 ±  5%  perf-profile.calltrace.cycles-pp.grab_cache_page_write_begin.nfs_write_begin.generic_perform_write.nfs_file_write.new_sync_write
      0.98 ±  3%      +0.1        1.09 ±  5%  perf-profile.calltrace.cycles-pp.nfs_write_begin.generic_perform_write.nfs_file_write.new_sync_write.vfs_write
      1.56 ±  2%      +0.2        1.71        perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyin.iov_iter_copy_from_user_atomic.generic_perform_write.nfs_file_write
      1.57            +0.2        1.72        perf-profile.calltrace.cycles-pp.copyin.iov_iter_copy_from_user_atomic.generic_perform_write.nfs_file_write.new_sync_write
      1.59            +0.2        1.75        perf-profile.calltrace.cycles-pp.iov_iter_copy_from_user_atomic.generic_perform_write.nfs_file_write.new_sync_write.vfs_write
      1.97 ±  2%      +0.2        2.21 ±  7%  perf-profile.calltrace.cycles-pp.nfs_end_page_writeback.nfs_write_completion.rpc_free_task.rpc_async_release.process_one_work
      2.50 ±  3%      +0.3        2.78 ±  5%  perf-profile.calltrace.cycles-pp.memcpy_from_page._copy_from_iter_full.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg
      2.54 ±  3%      +0.3        2.83 ±  5%  perf-profile.calltrace.cycles-pp._copy_from_iter_full.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.xs_sendpages
      2.08 ±  3%      +0.3        2.37 ±  9%  perf-profile.calltrace.cycles-pp.nfs_write_end.generic_perform_write.nfs_file_write.new_sync_write.vfs_write
      2.49 ±  2%      +0.3        2.81 ±  6%  perf-profile.calltrace.cycles-pp.nfs_write_completion.rpc_free_task.rpc_async_release.process_one_work.worker_thread
      2.94 ±  3%      +0.3        3.29 ±  7%  perf-profile.calltrace.cycles-pp.rpc_free_task.rpc_async_release.process_one_work.worker_thread.kthread
      2.94 ±  3%      +0.3        3.29 ±  7%  perf-profile.calltrace.cycles-pp.rpc_async_release.process_one_work.worker_thread.kthread.ret_from_fork
      3.19 ±  3%      +0.3        3.54 ±  5%  perf-profile.calltrace.cycles-pp.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.xs_sendpages.xs_tcp_send_request
      3.49 ±  3%      +0.4        3.86 ±  5%  perf-profile.calltrace.cycles-pp.xs_sendpages.xs_tcp_send_request.xprt_transmit.call_transmit.__rpc_execute
      3.48 ±  3%      +0.4        3.85 ±  5%  perf-profile.calltrace.cycles-pp.sock_sendmsg.xs_sendpages.xs_tcp_send_request.xprt_transmit.call_transmit
      3.47 ±  3%      +0.4        3.85 ±  5%  perf-profile.calltrace.cycles-pp.tcp_sendmsg.sock_sendmsg.xs_sendpages.xs_tcp_send_request.xprt_transmit
      3.49 ±  3%      +0.4        3.87 ±  5%  perf-profile.calltrace.cycles-pp.xs_tcp_send_request.xprt_transmit.call_transmit.__rpc_execute.rpc_async_schedule
      3.53 ±  3%      +0.4        3.91 ±  5%  perf-profile.calltrace.cycles-pp.xprt_transmit.call_transmit.__rpc_execute.rpc_async_schedule.process_one_work
      3.53 ±  3%      +0.4        3.92 ±  5%  perf-profile.calltrace.cycles-pp.call_transmit.__rpc_execute.rpc_async_schedule.process_one_work.worker_thread
      3.75 ±  3%      +0.4        4.17 ±  5%  perf-profile.calltrace.cycles-pp.__rpc_execute.rpc_async_schedule.process_one_work.worker_thread.kthread
      3.75 ±  3%      +0.4        4.17 ±  5%  perf-profile.calltrace.cycles-pp.rpc_async_schedule.process_one_work.worker_thread.kthread.ret_from_fork
      0.00            +0.5        0.55 ±  6%  perf-profile.calltrace.cycles-pp.ext4_release_io_end.ext4_put_io_end.ext4_writepages.do_writepages.__filemap_fdatawrite_range
      0.00            +0.5        0.55 ±  6%  perf-profile.calltrace.cycles-pp.ext4_finish_bio.ext4_release_io_end.ext4_put_io_end.ext4_writepages.do_writepages
      0.00            +0.6        0.58 ±  5%  perf-profile.calltrace.cycles-pp.ext4_put_io_end.ext4_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range
      4.87 ±  2%      +0.6        5.46 ±  5%  perf-profile.calltrace.cycles-pp.generic_perform_write.nfs_file_write.new_sync_write.vfs_write.ksys_write
      5.01 ±  2%      +0.6        5.61 ±  5%  perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.06 ±  2%      +0.6        5.66 ±  5%  perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
      4.99 ±  2%      +0.6        5.60 ±  5%  perf-profile.calltrace.cycles-pp.nfs_file_write.new_sync_write.vfs_write.ksys_write.do_syscall_64
      5.07 ±  2%      +0.6        5.67 ±  5%  perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
      5.08 ±  2%      +0.6        5.68 ±  5%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
      5.11 ±  2%      +0.6        5.72 ±  5%  perf-profile.calltrace.cycles-pp.write
      5.08 ±  2%      +0.6        5.69 ±  5%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
      7.48 ±  3%      +0.7        8.21 ±  5%  perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork
      7.14 ±  3%      +0.8        7.91 ±  5%  perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork
     14.78 ±  2%      -1.3       13.46        perf-profile.children.cycles-pp.apic_timer_interrupt
     12.69            -1.0       11.68        perf-profile.children.cycles-pp.smp_apic_timer_interrupt
      1.74 ± 17%      -0.6        1.14 ± 19%  perf-profile.children.cycles-pp.rwsem_down_write_slowpath
      1.72 ± 17%      -0.6        1.13 ± 19%  perf-profile.children.cycles-pp.rwsem_optimistic_spin
      6.56 ±  2%      -0.5        6.03 ±  5%  perf-profile.children.cycles-pp.hrtimer_interrupt
      1.36 ± 15%      -0.4        0.96 ± 16%  perf-profile.children.cycles-pp.rwsem_spin_on_owner
      4.15 ±  3%      -0.4        3.75 ±  4%  perf-profile.children.cycles-pp.ktime_get
      4.83 ±  2%      -0.3        4.48        perf-profile.children.cycles-pp.irq_exit
      3.90 ±  2%      -0.3        3.63        perf-profile.children.cycles-pp.__softirqentry_text_start
      3.38 ±  2%      -0.2        3.16 ±  4%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      2.41 ±  2%      -0.2        2.20 ±  7%  perf-profile.children.cycles-pp.clockevents_program_event
      2.19 ±  2%      -0.2        1.98 ±  3%  perf-profile.children.cycles-pp.rcu_core
      0.75 ±  4%      -0.2        0.55 ±  6%  perf-profile.children.cycles-pp.ext4_finish_bio
      0.37 ± 27%      -0.2        0.17 ± 38%  perf-profile.children.cycles-pp.osq_lock
      1.24            -0.2        1.06 ±  5%  perf-profile.children.cycles-pp.__filemap_fdatawait_range
      2.48 ±  2%      -0.1        2.34 ±  2%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.44 ±  7%      -0.1        0.31 ± 10%  perf-profile.children.cycles-pp.jbd2_journal_commit_transaction
      0.45 ±  6%      -0.1        0.33 ±  8%  perf-profile.children.cycles-pp.kjournald2
      0.39 ±  6%      -0.1        0.32 ±  7%  perf-profile.children.cycles-pp.pagevec_lookup_range_tag
      0.39 ±  6%      -0.1        0.32 ±  7%  perf-profile.children.cycles-pp.find_get_pages_range_tag
      0.51            -0.1        0.46 ±  3%  perf-profile.children.cycles-pp.get_next_timer_interrupt
      0.43 ±  3%      -0.0        0.38 ±  4%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.23 ± 11%      -0.0        0.19 ±  6%  perf-profile.children.cycles-pp.__might_sleep
      0.36 ±  3%      -0.0        0.33 ±  3%  perf-profile.children.cycles-pp.update_rq_clock
      0.21 ±  2%      -0.0        0.19 ±  2%  perf-profile.children.cycles-pp.drm_fb_helper_dirty_work
      0.05            +0.0        0.06 ±  7%  perf-profile.children.cycles-pp.__unwind_start
      0.09 ±  4%      +0.0        0.11 ±  4%  perf-profile.children.cycles-pp.sk_page_frag_refill
      0.09 ±  7%      +0.0        0.11 ±  4%  perf-profile.children.cycles-pp.skb_page_frag_refill
      0.07 ± 10%      +0.0        0.09 ±  5%  perf-profile.children.cycles-pp.__free_pages_ok
      0.05            +0.0        0.07 ± 14%  perf-profile.children.cycles-pp.xs_read_stream_request
      0.04 ± 57%      +0.0        0.06        perf-profile.children.cycles-pp.vfs_create
      0.17 ±  2%      +0.0        0.20 ±  4%  perf-profile.children.cycles-pp.skb_release_data
      0.23 ±  3%      +0.0        0.26        perf-profile.children.cycles-pp.nfs_initiate_commit
      0.07 ± 10%      +0.0        0.10 ±  8%  perf-profile.children.cycles-pp.jbd2_journal_get_write_access
      0.10 ±  7%      +0.0        0.13 ± 10%  perf-profile.children.cycles-pp.__ext4_journal_get_write_access
      0.06 ± 11%      +0.0        0.09        perf-profile.children.cycles-pp.ext4_map_blocks
      0.17 ±  6%      +0.0        0.20 ±  8%  perf-profile.children.cycles-pp.nfs_io_completion_put
      0.32 ±  5%      +0.0        0.35 ±  2%  perf-profile.children.cycles-pp.nfs_page_group_sync_on_bit
      0.24 ±  4%      +0.0        0.27 ±  5%  perf-profile.children.cycles-pp.__kfree_skb
      0.04 ± 58%      +0.0        0.07 ±  6%  perf-profile.children.cycles-pp.ext4_ext_map_blocks
      0.23            +0.0        0.27 ±  3%  perf-profile.children.cycles-pp.nfs_request_add_commit_list
      0.06 ± 20%      +0.0        0.10 ± 12%  perf-profile.children.cycles-pp.submit_bh_wbc
      0.23 ±  6%      +0.1        0.28 ± 11%  perf-profile.children.cycles-pp.__wake_up_common_lock
      0.18 ± 12%      +0.1        0.23 ±  9%  perf-profile.children.cycles-pp.nfs_ctx_key_to_expire
      0.40 ±  6%      +0.1        0.46 ± 10%  perf-profile.children.cycles-pp.__set_page_dirty_nobuffers
      0.22 ± 11%      +0.1        0.28 ± 11%  perf-profile.children.cycles-pp.stop_this_handle
      0.55 ±  3%      +0.1        0.61 ±  5%  perf-profile.children.cycles-pp.__nfs_commit_inode
      0.98 ±  3%      +0.1        1.09 ±  5%  perf-profile.children.cycles-pp.nfs_write_begin
      1.57 ±  2%      +0.1        1.72        perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      1.57            +0.2        1.73        perf-profile.children.cycles-pp.copyin
      1.98 ±  2%      +0.2        2.21 ±  6%  perf-profile.children.cycles-pp.nfs_end_page_writeback
      3.83            +0.3        4.10 ±  4%  perf-profile.children.cycles-pp.iov_iter_copy_from_user_atomic
      2.51 ±  3%      +0.3        2.79 ±  5%  perf-profile.children.cycles-pp.memcpy_from_page
      2.55 ±  3%      +0.3        2.84 ±  5%  perf-profile.children.cycles-pp._copy_from_iter_full
      2.08 ±  3%      +0.3        2.37 ±  9%  perf-profile.children.cycles-pp.nfs_write_end
      2.50 ±  2%      +0.3        2.82 ±  6%  perf-profile.children.cycles-pp.nfs_write_completion
      2.94 ±  3%      +0.3        3.29 ±  7%  perf-profile.children.cycles-pp.rpc_async_release
      3.23 ±  3%      +0.4        3.58 ±  5%  perf-profile.children.cycles-pp.tcp_sendmsg_locked
      3.17 ±  2%      +0.4        3.54 ±  6%  perf-profile.children.cycles-pp.rpc_free_task
      3.52 ±  3%      +0.4        3.89 ±  5%  perf-profile.children.cycles-pp.sock_sendmsg
      3.53 ±  3%      +0.4        3.91 ±  5%  perf-profile.children.cycles-pp.xs_sendpages
      3.51 ±  3%      +0.4        3.89 ±  5%  perf-profile.children.cycles-pp.tcp_sendmsg
      3.54 ±  2%      +0.4        3.92 ±  5%  perf-profile.children.cycles-pp.xs_tcp_send_request
      3.57 ±  3%      +0.4        3.97 ±  5%  perf-profile.children.cycles-pp.xprt_transmit
      3.58 ±  3%      +0.4        3.97 ±  5%  perf-profile.children.cycles-pp.call_transmit
      3.75 ±  3%      +0.4        4.17 ±  5%  perf-profile.children.cycles-pp.rpc_async_schedule
      3.85 ±  3%      +0.4        4.28 ±  5%  perf-profile.children.cycles-pp.__rpc_execute
      4.62 ±  7%      +0.5        5.14 ±  6%  perf-profile.children.cycles-pp.brd_do_bvec
      0.00            +0.5        0.55 ±  6%  perf-profile.children.cycles-pp.ext4_release_io_end
      0.00            +0.6        0.58 ±  5%  perf-profile.children.cycles-pp.ext4_put_io_end
      5.21 ±  2%      +0.6        5.80 ±  5%  perf-profile.children.cycles-pp.vfs_write
      5.16 ±  2%      +0.6        5.75 ±  5%  perf-profile.children.cycles-pp.new_sync_write
      5.22 ±  2%      +0.6        5.82 ±  5%  perf-profile.children.cycles-pp.ksys_write
      4.99 ±  2%      +0.6        5.60 ±  5%  perf-profile.children.cycles-pp.nfs_file_write
      5.26 ±  2%      +0.6        5.87 ±  5%  perf-profile.children.cycles-pp.write
      7.48 ±  3%      +0.7        8.21 ±  5%  perf-profile.children.cycles-pp.worker_thread
      7.14 ±  3%      +0.8        7.91 ±  5%  perf-profile.children.cycles-pp.process_one_work
      8.59 ±  2%      +0.8        9.41 ±  6%  perf-profile.children.cycles-pp.do_syscall_64
      8.60 ±  2%      +0.8        9.42 ±  6%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     13.61 ±  7%      +1.6       15.23 ±  3%  perf-profile.children.cycles-pp.generic_perform_write
      1.35 ± 15%      -0.4        0.95 ± 16%  perf-profile.self.cycles-pp.rwsem_spin_on_owner
      3.90 ±  4%      -0.4        3.51 ±  4%  perf-profile.self.cycles-pp.ktime_get
      0.36 ± 26%      -0.2        0.17 ± 38%  perf-profile.self.cycles-pp.osq_lock
      0.31 ±  7%      -0.1        0.25 ± 11%  perf-profile.self.cycles-pp.find_get_pages_range_tag
      0.43 ±  3%      -0.0        0.38 ±  4%  perf-profile.self.cycles-pp.native_irq_return_iret
      0.23 ± 11%      -0.0        0.18 ±  6%  perf-profile.self.cycles-pp.__might_sleep
      0.17 ±  3%      -0.0        0.14 ±  3%  perf-profile.self.cycles-pp.__filemap_fdatawait_range
      0.36 ±  2%      -0.0        0.34 ±  3%  perf-profile.self.cycles-pp.note_gp_changes
      0.12 ±  3%      -0.0        0.10 ±  4%  perf-profile.self.cycles-pp.rcu_core
      0.07 ±  7%      -0.0        0.05 ±  8%  perf-profile.self.cycles-pp.wait_on_page_writeback
      0.06 ±  9%      +0.0        0.07 ±  6%  perf-profile.self.cycles-pp.nfs_request_add_commit_list
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.balance_dirty_pages
      0.17 ± 11%      +0.1        0.22 ±  9%  perf-profile.self.cycles-pp.nfs_ctx_key_to_expire
      0.16 ± 13%      +0.1        0.22 ± 10%  perf-profile.self.cycles-pp.stop_this_handle
      0.56            +0.1        0.64 ±  6%  perf-profile.self.cycles-pp.nfs_write_end
      1.55 ±  2%      +0.2        1.71        perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      1.55 ±  5%      +0.2        1.77 ±  8%  perf-profile.self.cycles-pp.brd_do_bvec
      2.49 ±  3%      +0.3        2.77 ±  5%  perf-profile.self.cycles-pp.memcpy_from_page
     15916 ±  8%     -14.7%      13584        softirqs.CPU0.RCU
     20703 ±  4%      -8.3%      18981 ±  2%  softirqs.CPU0.TIMER
     21847           -35.8%      14016 ±  2%  softirqs.CPU1.RCU
     19251 ±  6%     -16.8%      16009 ±  9%  softirqs.CPU1.TIMER
     15208 ±  2%     -11.7%      13427        softirqs.CPU10.RCU
     14871 ±  7%     -15.3%      12602        softirqs.CPU100.RCU
     18798 ±  3%     -11.0%      16735        softirqs.CPU100.TIMER
     14793 ±  7%     -14.6%      12631        softirqs.CPU101.RCU
     18639 ±  2%      -9.7%      16831        softirqs.CPU101.TIMER
     14721 ±  3%     -11.6%      13018 ±  3%  softirqs.CPU102.RCU
     18681 ±  2%      -8.4%      17120 ±  2%  softirqs.CPU102.TIMER
     14784 ±  2%     -14.3%      12677        softirqs.CPU103.RCU
     18605 ±  3%     -10.0%      16745        softirqs.CPU103.TIMER
     15405 ±  7%     -17.1%      12766        softirqs.CPU104.RCU
     19167 ±  4%     -12.0%      16873        softirqs.CPU104.TIMER
     14534 ±  3%      -8.8%      13254 ±  4%  softirqs.CPU105.RCU
     15286 ±  7%     -14.8%      13017 ±  3%  softirqs.CPU106.RCU
     18975 ±  7%     -10.0%      17070 ±  3%  softirqs.CPU106.TIMER
     14273 ±  3%     -10.7%      12742 ±  2%  softirqs.CPU107.RCU
     14274 ±  2%     -10.3%      12804 ±  2%  softirqs.CPU108.RCU
     18222 ±  2%      -7.9%      16786        softirqs.CPU108.TIMER
     14970 ±  5%     -16.2%      12543        softirqs.CPU109.RCU
     18510 ±  2%      -9.8%      16697        softirqs.CPU109.TIMER
     21308 ± 47%     -35.9%      13660 ±  4%  softirqs.CPU11.RCU
      7182 ± 33%     -29.0%       5096 ±  3%  softirqs.CPU11.SCHED
     25129 ± 38%     -29.1%      17824        softirqs.CPU11.TIMER
     15063 ±  4%     -13.8%      12984        softirqs.CPU110.RCU
     14296 ±  3%     -11.4%      12663        softirqs.CPU111.RCU
     18304 ±  2%      -9.1%      16643        softirqs.CPU111.TIMER
     14544           -14.6%      12415        softirqs.CPU112.RCU
     14291 ±  2%     -11.6%      12629 ±  4%  softirqs.CPU114.RCU
     14546 ±  2%     -13.0%      12649        softirqs.CPU115.RCU
     18165 ±  2%      -7.8%      16747        softirqs.CPU115.TIMER
     14538           -10.9%      12956 ±  3%  softirqs.CPU116.RCU
     14627           -12.9%      12734        softirqs.CPU117.RCU
     18194 ±  2%      -7.8%      16781        softirqs.CPU117.TIMER
     14897 ±  2%     -15.3%      12613 ±  3%  softirqs.CPU119.RCU
     18247 ±  2%      -7.3%      16906        softirqs.CPU119.TIMER
     18474 ± 26%     -28.7%      13165 ±  2%  softirqs.CPU12.RCU
     22339 ± 22%     -21.5%      17533        softirqs.CPU12.TIMER
     14352           -12.6%      12548 ±  2%  softirqs.CPU120.RCU
     14340           -10.8%      12784        softirqs.CPU121.RCU
     18104            -9.4%      16396 ±  3%  softirqs.CPU121.TIMER
     14243 ±  3%      -8.7%      13007 ±  3%  softirqs.CPU123.RCU
     14434            -7.2%      13395 ±  4%  softirqs.CPU124.RCU
     14394 ±  3%     -10.2%      12920 ±  2%  softirqs.CPU126.RCU
     18107            -8.9%      16489        softirqs.CPU126.TIMER
     14361 ±  3%      -8.2%      13187 ±  3%  softirqs.CPU127.RCU
     14891 ± 10%     -15.2%      12632        softirqs.CPU128.RCU
     18664 ±  6%     -12.2%      16385 ±  2%  softirqs.CPU128.TIMER
     14826           -10.7%      13244 ±  2%  softirqs.CPU13.RCU
     18128 ±  2%      -9.9%      16326 ±  2%  softirqs.CPU130.TIMER
     15073 ±  9%     -13.3%      13075 ±  4%  softirqs.CPU131.RCU
     19277 ±  9%     -12.2%      16927        softirqs.CPU131.TIMER
     14165 ±  3%     -10.7%      12645        softirqs.CPU132.RCU
     18091            -9.4%      16387 ±  4%  softirqs.CPU132.TIMER
     14324 ±  3%     -11.6%      12667        softirqs.CPU133.RCU
     18014           -10.0%      16215 ±  3%  softirqs.CPU133.TIMER
     14225           -10.1%      12792        softirqs.CPU134.RCU
     17678            -7.6%      16328 ±  3%  softirqs.CPU134.TIMER
     14428 ±  3%     -14.5%      12338 ±  2%  softirqs.CPU135.RCU
     18337 ±  3%     -11.5%      16237 ±  2%  softirqs.CPU135.TIMER
     14080            -9.6%      12735 ±  2%  softirqs.CPU136.RCU
     17897            -7.5%      16548 ±  4%  softirqs.CPU136.TIMER
     13825           -10.8%      12336        softirqs.CPU138.RCU
     17943            -9.1%      16312 ±  3%  softirqs.CPU138.TIMER
     14202           -10.5%      12715        softirqs.CPU139.RCU
     18027            -9.4%      16324 ±  3%  softirqs.CPU139.TIMER
     15563 ±  5%     -15.4%      13171 ±  2%  softirqs.CPU14.RCU
     14289 ±  3%     -13.8%      12311        softirqs.CPU140.RCU
     18156 ±  3%     -10.2%      16312 ±  3%  softirqs.CPU140.TIMER
     13794           -10.2%      12386 ±  3%  softirqs.CPU141.RCU
     17819            -8.2%      16363 ±  2%  softirqs.CPU141.TIMER
     14058           -12.3%      12331        softirqs.CPU142.RCU
     17960           -10.1%      16145 ±  3%  softirqs.CPU142.TIMER
     15036 ±  8%     -14.2%      12906 ±  6%  softirqs.CPU143.RCU
     19060 ±  6%     -12.2%      16733 ±  4%  softirqs.CPU143.TIMER
     14124 ±  5%     -11.5%      12505 ±  3%  softirqs.CPU144.RCU
     14073 ±  3%     -11.4%      12473 ±  2%  softirqs.CPU145.RCU
     14198 ±  4%     -11.5%      12558 ±  3%  softirqs.CPU146.RCU
     14284 ±  5%     -13.3%      12386        softirqs.CPU147.RCU
     13717 ±  4%     -11.3%      12164        softirqs.CPU148.RCU
     14177 ±  8%     -14.0%      12196 ±  2%  softirqs.CPU149.RCU
     18096 ±  3%      -8.1%      16629 ±  2%  softirqs.CPU149.TIMER
     15305 ±  2%     -13.7%      13208        softirqs.CPU15.RCU
     13692 ±  3%      -8.3%      12553 ±  2%  softirqs.CPU150.RCU
     13873 ±  2%     -11.0%      12345 ±  2%  softirqs.CPU151.RCU
     13682 ±  4%     -10.4%      12257 ±  4%  softirqs.CPU152.RCU
     13526 ±  4%      -5.8%      12748 ±  2%  softirqs.CPU153.RCU
     18770 ±  7%     -11.5%      16608 ±  2%  softirqs.CPU155.TIMER
     14152 ±  2%     -11.5%      12523 ±  3%  softirqs.CPU156.RCU
     13955 ±  2%     -10.9%      12438 ±  3%  softirqs.CPU157.RCU
     15265           -13.6%      13191        softirqs.CPU16.RCU
     13871 ±  3%     -10.6%      12400        softirqs.CPU160.RCU
     14330 ±  4%     -13.3%      12430 ±  3%  softirqs.CPU161.RCU
     18024 ±  5%      -8.0%      16590        softirqs.CPU161.TIMER
     13818            -9.2%      12547        softirqs.CPU162.RCU
     14091 ±  2%      -9.7%      12719        softirqs.CPU163.RCU
     20052 ± 18%     -15.9%      16868        softirqs.CPU163.TIMER
     14317 ±  2%     -12.8%      12488 ±  3%  softirqs.CPU165.RCU
     18012 ±  3%      -8.6%      16470        softirqs.CPU165.TIMER
     14400 ±  4%     -16.8%      11987        softirqs.CPU166.RCU
     17882 ±  2%      -9.1%      16251        softirqs.CPU166.TIMER
     15164 ±  9%     -17.9%      12450 ±  3%  softirqs.CPU167.RCU
     18838 ±  6%     -12.9%      16416        softirqs.CPU167.TIMER
     13776           -12.5%      12048 ±  2%  softirqs.CPU168.RCU
     14359 ±  6%     -12.5%      12567 ±  4%  softirqs.CPU169.RCU
     18710 ± 11%     -13.2%      16239        softirqs.CPU169.TIMER
     15244 ±  2%     -14.4%      13043 ±  2%  softirqs.CPU17.RCU
     14285 ±  4%     -15.2%      12112 ±  3%  softirqs.CPU172.RCU
     18156 ±  2%      -9.8%      16378        softirqs.CPU172.TIMER
     14117           -10.2%      12673 ±  6%  softirqs.CPU173.RCU
     13730           -10.6%      12275 ±  3%  softirqs.CPU174.RCU
     14886 ±  8%     -14.8%      12678 ±  7%  softirqs.CPU177.RCU
     14065            -9.2%      12770 ±  3%  softirqs.CPU178.RCU
     13938           -11.2%      12379        softirqs.CPU179.RCU
     15081            -8.5%      13800 ±  6%  softirqs.CPU18.RCU
     14232           -11.3%      12624 ±  2%  softirqs.CPU180.RCU
     17884            -8.7%      16323        softirqs.CPU180.TIMER
     13844           -11.0%      12327 ±  3%  softirqs.CPU182.RCU
     14588 ±  8%     -16.4%      12189 ±  4%  softirqs.CPU183.RCU
     18383 ±  7%     -11.2%      16326        softirqs.CPU183.TIMER
     14466 ±  3%     -14.7%      12335 ±  4%  softirqs.CPU185.RCU
     17939 ±  3%      -9.6%      16218        softirqs.CPU185.TIMER
     13684            -9.8%      12349 ±  3%  softirqs.CPU186.RCU
     13850           -11.9%      12202 ±  2%  softirqs.CPU187.RCU
     13943           -12.4%      12213 ±  2%  softirqs.CPU188.RCU
     13468           -10.5%      12048        softirqs.CPU189.RCU
     14921 ±  3%     -12.2%      13103        softirqs.CPU19.RCU
     14381 ±  9%     -17.4%      11874 ±  4%  softirqs.CPU190.RCU
     18131 ±  4%     -11.7%      16004        softirqs.CPU190.TIMER
     15083 ±  8%     -20.6%      11970 ±  3%  softirqs.CPU191.RCU
     18836 ±  5%     -12.3%      16517 ±  3%  softirqs.CPU191.TIMER
     15266 ±  2%     -15.2%      12945 ±  2%  softirqs.CPU20.RCU
     15094 ±  2%     -12.9%      13143 ±  2%  softirqs.CPU21.RCU
     15129           -13.9%      13020        softirqs.CPU22.RCU
     15228           -11.0%      13545        softirqs.CPU23.RCU
     15024           -11.6%      13280        softirqs.CPU24.RCU
     18625            -7.8%      17172 ±  2%  softirqs.CPU24.TIMER
     15374 ±  3%     -12.4%      13467        softirqs.CPU25.RCU
     19007 ±  2%      -8.6%      17375 ±  4%  softirqs.CPU25.TIMER
     15359           -14.1%      13196 ±  2%  softirqs.CPU26.RCU
     15292           -10.7%      13657 ±  3%  softirqs.CPU27.RCU
     15247 ±  7%     -11.2%      13541        softirqs.CPU31.RCU
     18981 ±  3%      -8.1%      17435 ±  5%  softirqs.CPU31.TIMER
     15175           -10.4%      13596 ±  2%  softirqs.CPU32.RCU
     16025 ±  4%     -14.1%      13770 ±  4%  softirqs.CPU33.RCU
     19786 ±  2%     -10.1%      17796 ±  5%  softirqs.CPU33.TIMER
     15164 ±  5%     -11.0%      13501 ±  5%  softirqs.CPU35.RCU
     15611 ±  6%     -16.1%      13101        softirqs.CPU36.RCU
     19551 ±  5%     -11.4%      17322 ±  2%  softirqs.CPU36.TIMER
     14513            -9.1%      13189        softirqs.CPU37.RCU
     15374 ±  9%     -14.8%      13101        softirqs.CPU38.RCU
     14678           -10.4%      13151 ±  3%  softirqs.CPU39.RCU
     15384 ±  2%     -12.3%      13495        softirqs.CPU4.RCU
     19453 ±  2%      -8.2%      17864        softirqs.CPU4.TIMER
     14926           -11.9%      13154 ±  2%  softirqs.CPU40.RCU
     14836 ±  2%     -10.1%      13339        softirqs.CPU41.RCU
     14575           -11.1%      12959        softirqs.CPU42.RCU
     14643 ±  2%     -10.4%      13127 ±  2%  softirqs.CPU43.RCU
     18512            -7.7%      17083 ±  2%  softirqs.CPU43.TIMER
     15079 ±  2%     -12.7%      13169        softirqs.CPU44.RCU
     14806 ±  2%     -11.1%      13165 ±  2%  softirqs.CPU45.RCU
     14980 ±  3%     -12.3%      13138        softirqs.CPU46.RCU
     18820 ±  2%      -9.3%      17071 ±  3%  softirqs.CPU46.TIMER
     15411 ±  8%     -12.5%      13485 ±  3%  softirqs.CPU47.RCU
     15269           -12.4%      13376 ±  3%  softirqs.CPU5.RCU
     19092 ±  3%      -7.4%      17683        softirqs.CPU5.TIMER
     15092           -13.3%      13086        softirqs.CPU50.RCU
     14444 ±  4%     -10.4%      12948        softirqs.CPU52.RCU
     14969 ±  3%      -9.3%      13572 ±  3%  softirqs.CPU54.RCU
     14667 ±  2%      -9.1%      13328 ±  2%  softirqs.CPU55.RCU
     15171 ± 10%     -15.3%      12853 ±  2%  softirqs.CPU56.RCU
     15778 ± 11%     -16.1%      13242 ±  2%  softirqs.CPU57.RCU
     15003 ±  4%     -14.3%      12856        softirqs.CPU58.RCU
     18918 ±  2%      -8.5%      17304 ±  3%  softirqs.CPU58.TIMER
     15198 ±  6%     -13.8%      13104 ±  2%  softirqs.CPU59.RCU
     14998 ±  2%     -10.0%      13501 ±  2%  softirqs.CPU6.RCU
     15501 ±  9%     -16.4%      12967 ±  2%  softirqs.CPU60.RCU
     15293 ±  9%     -15.6%      12911        softirqs.CPU61.RCU
     14799 ±  5%     -11.4%      13108 ±  2%  softirqs.CPU62.RCU
     14691 ±  2%     -11.7%      12978        softirqs.CPU63.RCU
     15170 ±  4%     -10.5%      13575 ±  4%  softirqs.CPU64.RCU
     14713 ±  4%      -9.2%      13357 ±  3%  softirqs.CPU65.RCU
     15002 ±  2%      -8.0%      13795 ±  2%  softirqs.CPU66.RCU
     14687           -11.2%      13038 ±  2%  softirqs.CPU67.RCU
     14759 ±  3%      -8.6%      13485        softirqs.CPU68.RCU
     14683            -9.3%      13313 ±  3%  softirqs.CPU69.RCU
     15115            -9.5%      13682 ±  2%  softirqs.CPU7.RCU
     14543 ±  3%      -9.1%      13214        softirqs.CPU70.RCU
     14756 ±  2%      -9.6%      13338        softirqs.CPU71.RCU
     15613 ±  8%     -15.4%      13212 ±  2%  softirqs.CPU72.RCU
     19744 ±  3%     -11.3%      17513 ±  4%  softirqs.CPU72.TIMER
     15370 ±  4%      -8.2%      14111 ±  4%  softirqs.CPU73.RCU
     15181 ±  4%     -13.7%      13098        softirqs.CPU75.RCU
     15880 ±  4%     -16.8%      13210        softirqs.CPU76.RCU
     14970 ±  2%     -12.0%      13180 ±  2%  softirqs.CPU77.RCU
     14923           -12.3%      13090        softirqs.CPU78.RCU
     16686 ± 10%     -16.3%      13969 ±  9%  softirqs.CPU79.RCU
     15663 ±  5%     -10.5%      14025 ±  5%  softirqs.CPU8.RCU
     15545 ±  7%     -12.1%      13658 ±  6%  softirqs.CPU80.RCU
     18965 ±  5%      -6.6%      17707 ±  4%  softirqs.CPU80.TIMER
     14701           -11.8%      12963 ±  2%  softirqs.CPU81.RCU
     14916           -12.0%      13132 ±  3%  softirqs.CPU82.RCU
     15615 ±  8%     -14.1%      13409 ±  2%  softirqs.CPU83.RCU
     14898           -13.5%      12883 ±  2%  softirqs.CPU84.RCU
     14951 ±  6%     -13.0%      13012 ±  5%  softirqs.CPU85.RCU
     18401 ±  3%      -6.0%      17297        softirqs.CPU85.TIMER
     15478 ±  9%     -15.8%      13028 ±  3%  softirqs.CPU86.RCU
     14338 ±  2%     -10.1%      12891 ±  2%  softirqs.CPU87.RCU
     14544           -10.7%      12987 ±  2%  softirqs.CPU88.RCU
     14709 ±  2%     -12.9%      12805        softirqs.CPU89.RCU
     15806 ±  4%     -15.4%      13377        softirqs.CPU9.RCU
     19705 ±  2%     -10.2%      17700        softirqs.CPU9.TIMER
     14435           -10.2%      12957        softirqs.CPU90.RCU
     14239            -9.2%      12936        softirqs.CPU92.RCU
     15456 ±  8%     -13.8%      13320 ±  3%  softirqs.CPU93.RCU
     14326            -9.6%      12949        softirqs.CPU94.RCU
     14566 ±  2%     -11.6%      12882        softirqs.CPU96.RCU
     18119           -16.5%      15136 ± 14%  softirqs.CPU96.TIMER
     14358 ±  2%     -11.4%      12728        softirqs.CPU99.RCU
     18474 ±  2%      -8.1%      16978        softirqs.CPU99.TIMER
   2840660           -11.6%    2510022        softirqs.RCU
    119.00 ± 39%     -39.8%      71.67 ± 69%  interrupts.31:PCI-MSI.524289-edge.eth0-TxRx-0
     50502           -13.6%      43637        interrupts.CPU0.LOC:Local_timer_interrupts
     50481           -13.6%      43634        interrupts.CPU1.LOC:Local_timer_interrupts
     50468           -13.5%      43667        interrupts.CPU10.LOC:Local_timer_interrupts
     50438           -13.5%      43615        interrupts.CPU100.LOC:Local_timer_interrupts
     50450           -13.5%      43633        interrupts.CPU101.LOC:Local_timer_interrupts
     50439           -13.5%      43606        interrupts.CPU102.LOC:Local_timer_interrupts
     50421           -13.5%      43614        interrupts.CPU103.LOC:Local_timer_interrupts
     50434           -13.5%      43626        interrupts.CPU104.LOC:Local_timer_interrupts
     50430           -13.4%      43651        interrupts.CPU105.LOC:Local_timer_interrupts
     50427           -13.6%      43593        interrupts.CPU106.LOC:Local_timer_interrupts
     50425           -13.5%      43601        interrupts.CPU107.LOC:Local_timer_interrupts
     50414           -13.5%      43609        interrupts.CPU108.LOC:Local_timer_interrupts
     50417           -13.5%      43597        interrupts.CPU109.LOC:Local_timer_interrupts
     50503           -13.6%      43613        interrupts.CPU11.LOC:Local_timer_interrupts
      1001 ± 29%     +35.5%       1357 ± 10%  interrupts.CPU11.NMI:Non-maskable_interrupts
      1001 ± 29%     +35.5%       1357 ± 10%  interrupts.CPU11.PMI:Performance_monitoring_interrupts
     50420           -13.5%      43605        interrupts.CPU110.LOC:Local_timer_interrupts
     50421           -13.5%      43595        interrupts.CPU111.LOC:Local_timer_interrupts
     50419           -13.5%      43599        interrupts.CPU112.LOC:Local_timer_interrupts
     50426           -13.5%      43599        interrupts.CPU113.LOC:Local_timer_interrupts
     50429           -13.5%      43617        interrupts.CPU114.LOC:Local_timer_interrupts
     50418           -13.5%      43600        interrupts.CPU115.LOC:Local_timer_interrupts
     50428           -13.6%      43591        interrupts.CPU116.LOC:Local_timer_interrupts
     50424           -13.6%      43591        interrupts.CPU117.LOC:Local_timer_interrupts
     50429           -13.4%      43648        interrupts.CPU118.LOC:Local_timer_interrupts
     50395           -13.4%      43629        interrupts.CPU119.LOC:Local_timer_interrupts
     50493           -13.6%      43610        interrupts.CPU12.LOC:Local_timer_interrupts
     50487           -13.7%      43587        interrupts.CPU120.LOC:Local_timer_interrupts
     50394           -13.5%      43591        interrupts.CPU121.LOC:Local_timer_interrupts
     50467           -13.9%      43437        interrupts.CPU122.LOC:Local_timer_interrupts
     50486           -13.7%      43557        interrupts.CPU123.LOC:Local_timer_interrupts
    781.75 ±101%     -79.4%     161.00 ± 21%  interrupts.CPU123.RES:Rescheduling_interrupts
     50489           -13.7%      43580        interrupts.CPU124.LOC:Local_timer_interrupts
    166.25 ± 53%    +157.6%     428.33 ± 58%  interrupts.CPU124.RES:Rescheduling_interrupts
     50482           -13.6%      43592        interrupts.CPU125.LOC:Local_timer_interrupts
     50484           -13.6%      43598        interrupts.CPU126.LOC:Local_timer_interrupts
     50477           -13.6%      43588        interrupts.CPU127.LOC:Local_timer_interrupts
     50484           -13.6%      43595        interrupts.CPU128.LOC:Local_timer_interrupts
     50480           -13.6%      43600        interrupts.CPU129.LOC:Local_timer_interrupts
     50453           -13.6%      43614        interrupts.CPU13.LOC:Local_timer_interrupts
     50481           -13.7%      43589        interrupts.CPU130.LOC:Local_timer_interrupts
     50484           -13.8%      43514        interrupts.CPU131.LOC:Local_timer_interrupts
     50361           -13.7%      43466        interrupts.CPU132.LOC:Local_timer_interrupts
     50476           -13.6%      43594        interrupts.CPU133.LOC:Local_timer_interrupts
     50479           -13.6%      43605        interrupts.CPU134.LOC:Local_timer_interrupts
     50491           -13.6%      43609        interrupts.CPU135.LOC:Local_timer_interrupts
    527.25 ± 25%    +124.5%       1183 ± 27%  interrupts.CPU135.NMI:Non-maskable_interrupts
    527.25 ± 25%    +124.5%       1183 ± 27%  interrupts.CPU135.PMI:Performance_monitoring_interrupts
     50479           -13.6%      43607        interrupts.CPU136.LOC:Local_timer_interrupts
     50462           -13.5%      43630        interrupts.CPU137.LOC:Local_timer_interrupts
     50482           -13.6%      43606        interrupts.CPU138.LOC:Local_timer_interrupts
     50473           -13.6%      43606        interrupts.CPU139.LOC:Local_timer_interrupts
     50446           -13.6%      43607        interrupts.CPU14.LOC:Local_timer_interrupts
    683.25 ± 25%    +101.8%       1378 ± 16%  interrupts.CPU14.NMI:Non-maskable_interrupts
    683.25 ± 25%    +101.8%       1378 ± 16%  interrupts.CPU14.PMI:Performance_monitoring_interrupts
     50498           -13.7%      43601        interrupts.CPU140.LOC:Local_timer_interrupts
     50484           -13.6%      43605        interrupts.CPU141.LOC:Local_timer_interrupts
     50484           -13.6%      43605        interrupts.CPU142.LOC:Local_timer_interrupts
     50491           -13.6%      43602        interrupts.CPU143.LOC:Local_timer_interrupts
     50468           -13.5%      43645        interrupts.CPU144.LOC:Local_timer_interrupts
     50425           -13.5%      43611        interrupts.CPU145.LOC:Local_timer_interrupts
      1933 ± 68%     -56.4%     843.00 ± 14%  interrupts.CPU145.NMI:Non-maskable_interrupts
      1933 ± 68%     -56.4%     843.00 ± 14%  interrupts.CPU145.PMI:Performance_monitoring_interrupts
     50432           -13.5%      43627        interrupts.CPU146.LOC:Local_timer_interrupts
     50432           -13.5%      43623        interrupts.CPU147.LOC:Local_timer_interrupts
     50422           -13.5%      43616        interrupts.CPU148.LOC:Local_timer_interrupts
     50427           -13.5%      43623        interrupts.CPU149.LOC:Local_timer_interrupts
     50444           -13.5%      43616        interrupts.CPU15.LOC:Local_timer_interrupts
     50439           -13.5%      43619        interrupts.CPU150.LOC:Local_timer_interrupts
     50434           -13.5%      43622        interrupts.CPU151.LOC:Local_timer_interrupts
    255.75 ± 12%     -47.7%     133.67 ±  7%  interrupts.CPU151.RES:Rescheduling_interrupts
     50451           -13.6%      43607        interrupts.CPU152.LOC:Local_timer_interrupts
     50444           -13.5%      43628        interrupts.CPU153.LOC:Local_timer_interrupts
     50429           -13.5%      43627        interrupts.CPU154.LOC:Local_timer_interrupts
     50421           -13.5%      43624        interrupts.CPU155.LOC:Local_timer_interrupts
     50329           -13.3%      43613        interrupts.CPU156.LOC:Local_timer_interrupts
     50431           -13.5%      43613        interrupts.CPU157.LOC:Local_timer_interrupts
    266.50 ± 21%     -43.3%     151.00 ± 35%  interrupts.CPU157.RES:Rescheduling_interrupts
     50422           -13.5%      43625        interrupts.CPU158.LOC:Local_timer_interrupts
     50423           -13.5%      43612        interrupts.CPU159.LOC:Local_timer_interrupts
     50445           -13.6%      43609        interrupts.CPU16.LOC:Local_timer_interrupts
     50426           -13.5%      43613        interrupts.CPU160.LOC:Local_timer_interrupts
     50446           -13.4%      43698        interrupts.CPU161.LOC:Local_timer_interrupts
     50451           -13.5%      43640        interrupts.CPU162.LOC:Local_timer_interrupts
     50424           -13.5%      43614        interrupts.CPU163.LOC:Local_timer_interrupts
    699.50 ±  4%     +47.0%       1028 ± 13%  interrupts.CPU163.NMI:Non-maskable_interrupts
    699.50 ±  4%     +47.0%       1028 ± 13%  interrupts.CPU163.PMI:Performance_monitoring_interrupts
     50424           -13.5%      43619        interrupts.CPU164.LOC:Local_timer_interrupts
     50444           -13.4%      43697        interrupts.CPU165.LOC:Local_timer_interrupts
     50423           -13.5%      43639        interrupts.CPU166.LOC:Local_timer_interrupts
    295.50 ± 14%     -28.0%     212.67 ± 20%  interrupts.CPU166.RES:Rescheduling_interrupts
     50419           -13.6%      43580        interrupts.CPU167.LOC:Local_timer_interrupts
     50484           -13.6%      43607        interrupts.CPU168.LOC:Local_timer_interrupts
     50503           -13.7%      43609        interrupts.CPU169.LOC:Local_timer_interrupts
     50454           -13.5%      43640        interrupts.CPU17.LOC:Local_timer_interrupts
    569.50 ± 30%     -46.0%     307.67 ± 61%  interrupts.CPU17.RES:Rescheduling_interrupts
     50490           -13.7%      43598        interrupts.CPU170.LOC:Local_timer_interrupts
      1290 ± 14%     -39.6%     779.67 ± 24%  interrupts.CPU170.NMI:Non-maskable_interrupts
      1290 ± 14%     -39.6%     779.67 ± 24%  interrupts.CPU170.PMI:Performance_monitoring_interrupts
     50492           -13.6%      43600        interrupts.CPU171.LOC:Local_timer_interrupts
     50480           -13.6%      43602        interrupts.CPU172.LOC:Local_timer_interrupts
     50456           -13.5%      43629        interrupts.CPU173.LOC:Local_timer_interrupts
    303.00 ± 45%     -60.3%     120.33 ± 16%  interrupts.CPU173.RES:Rescheduling_interrupts
     50454           -13.6%      43610        interrupts.CPU174.LOC:Local_timer_interrupts
    601.50 ± 72%     -76.4%     141.67 ± 17%  interrupts.CPU174.RES:Rescheduling_interrupts
     50453           -13.4%      43708        interrupts.CPU175.LOC:Local_timer_interrupts
     50453           -13.2%      43805        interrupts.CPU176.LOC:Local_timer_interrupts
     50483           -13.4%      43722        interrupts.CPU177.LOC:Local_timer_interrupts
     50464           -13.6%      43620        interrupts.CPU178.LOC:Local_timer_interrupts
    713.00 ± 16%    +159.6%       1850 ± 70%  interrupts.CPU178.NMI:Non-maskable_interrupts
    713.00 ± 16%    +159.6%       1850 ± 70%  interrupts.CPU178.PMI:Performance_monitoring_interrupts
     50460           -13.6%      43606        interrupts.CPU179.LOC:Local_timer_interrupts
    294.00 ± 16%     -43.7%     165.67 ± 31%  interrupts.CPU179.RES:Rescheduling_interrupts
     50443           -13.5%      43609        interrupts.CPU18.LOC:Local_timer_interrupts
    698.00 ± 37%    +102.6%       1414 ±  3%  interrupts.CPU18.NMI:Non-maskable_interrupts
    698.00 ± 37%    +102.6%       1414 ±  3%  interrupts.CPU18.PMI:Performance_monitoring_interrupts
     50450           -13.6%      43595        interrupts.CPU180.LOC:Local_timer_interrupts
     50449           -13.6%      43600        interrupts.CPU181.LOC:Local_timer_interrupts
     50457           -13.6%      43600        interrupts.CPU182.LOC:Local_timer_interrupts
     50453           -13.6%      43594        interrupts.CPU183.LOC:Local_timer_interrupts
    256.50 ± 15%     -16.4%     214.33 ± 16%  interrupts.CPU183.RES:Rescheduling_interrupts
     50448           -13.6%      43594        interrupts.CPU184.LOC:Local_timer_interrupts
     50470           -13.6%      43594        interrupts.CPU185.LOC:Local_timer_interrupts
     50451           -13.6%      43595        interrupts.CPU186.LOC:Local_timer_interrupts
     50456           -13.6%      43598        interrupts.CPU187.LOC:Local_timer_interrupts
     50458           -13.6%      43597        interrupts.CPU188.LOC:Local_timer_interrupts
     50475           -13.6%      43593        interrupts.CPU189.LOC:Local_timer_interrupts
     50455           -13.6%      43606        interrupts.CPU19.LOC:Local_timer_interrupts
     50466           -13.6%      43586        interrupts.CPU190.LOC:Local_timer_interrupts
     50491           -13.6%      43632        interrupts.CPU191.LOC:Local_timer_interrupts
    712.25 ± 28%    +223.7%       2305 ± 56%  interrupts.CPU191.NMI:Non-maskable_interrupts
    712.25 ± 28%    +223.7%       2305 ± 56%  interrupts.CPU191.PMI:Performance_monitoring_interrupts
     50469           -13.6%      43626        interrupts.CPU2.LOC:Local_timer_interrupts
    837.75 ± 45%     +97.8%       1656 ± 13%  interrupts.CPU2.NMI:Non-maskable_interrupts
    837.75 ± 45%     +97.8%       1656 ± 13%  interrupts.CPU2.PMI:Performance_monitoring_interrupts
     50441           -13.5%      43625        interrupts.CPU20.LOC:Local_timer_interrupts
     50443           -13.5%      43624        interrupts.CPU21.LOC:Local_timer_interrupts
    710.75 ± 26%     +48.1%       1052 ± 11%  interrupts.CPU21.NMI:Non-maskable_interrupts
    710.75 ± 26%     +48.1%       1052 ± 11%  interrupts.CPU21.PMI:Performance_monitoring_interrupts
     50463           -13.5%      43638        interrupts.CPU22.LOC:Local_timer_interrupts
     50447           -13.6%      43608        interrupts.CPU23.LOC:Local_timer_interrupts
     50503           -13.7%      43609        interrupts.CPU24.LOC:Local_timer_interrupts
     50514           -13.7%      43613        interrupts.CPU25.LOC:Local_timer_interrupts
     50480           -13.6%      43610        interrupts.CPU26.LOC:Local_timer_interrupts
     50492           -13.8%      43530        interrupts.CPU27.LOC:Local_timer_interrupts
     50376           -13.4%      43612        interrupts.CPU28.LOC:Local_timer_interrupts
     50437           -13.5%      43607        interrupts.CPU29.LOC:Local_timer_interrupts
     50465           -13.6%      43622        interrupts.CPU3.LOC:Local_timer_interrupts
     50396           -13.5%      43607        interrupts.CPU30.LOC:Local_timer_interrupts
     50501           -13.7%      43605        interrupts.CPU31.LOC:Local_timer_interrupts
     50492           -13.6%      43608        interrupts.CPU32.LOC:Local_timer_interrupts
    388.75 ± 29%     -50.5%     192.33 ± 48%  interrupts.CPU32.RES:Rescheduling_interrupts
     50499           -13.6%      43618        interrupts.CPU33.LOC:Local_timer_interrupts
     50503           -13.7%      43599        interrupts.CPU34.LOC:Local_timer_interrupts
     50500           -13.6%      43655        interrupts.CPU35.LOC:Local_timer_interrupts
     50499           -13.6%      43611        interrupts.CPU36.LOC:Local_timer_interrupts
     50496           -13.6%      43609        interrupts.CPU37.LOC:Local_timer_interrupts
     50496           -13.6%      43612        interrupts.CPU38.LOC:Local_timer_interrupts
     50505           -13.6%      43611        interrupts.CPU39.LOC:Local_timer_interrupts
    582.75 ± 24%    +421.3%       3038 ± 77%  interrupts.CPU39.NMI:Non-maskable_interrupts
    582.75 ± 24%    +421.3%       3038 ± 77%  interrupts.CPU39.PMI:Performance_monitoring_interrupts
     50451           -13.5%      43624        interrupts.CPU4.LOC:Local_timer_interrupts
     50501           -13.6%      43610        interrupts.CPU40.LOC:Local_timer_interrupts
    465.25 ± 23%     +86.8%     869.00 ± 17%  interrupts.CPU40.NMI:Non-maskable_interrupts
    465.25 ± 23%     +86.8%     869.00 ± 17%  interrupts.CPU40.PMI:Performance_monitoring_interrupts
     50459           -13.7%      43570        interrupts.CPU41.LOC:Local_timer_interrupts
     50488           -13.6%      43614        interrupts.CPU42.LOC:Local_timer_interrupts
    379.00 ± 14%     -30.0%     265.33 ± 33%  interrupts.CPU42.RES:Rescheduling_interrupts
     50511           -13.6%      43617        interrupts.CPU43.LOC:Local_timer_interrupts
     50532           -13.6%      43650        interrupts.CPU44.LOC:Local_timer_interrupts
     50511           -13.6%      43620        interrupts.CPU45.LOC:Local_timer_interrupts
     50503           -13.6%      43622        interrupts.CPU46.LOC:Local_timer_interrupts
     50506           -13.6%      43619        interrupts.CPU47.LOC:Local_timer_interrupts
     50445           -13.5%      43638        interrupts.CPU48.LOC:Local_timer_interrupts
     50449           -13.5%      43637        interrupts.CPU49.LOC:Local_timer_interrupts
      2568 ± 81%     -76.9%     594.33 ± 27%  interrupts.CPU49.NMI:Non-maskable_interrupts
      2568 ± 81%     -76.9%     594.33 ± 27%  interrupts.CPU49.PMI:Performance_monitoring_interrupts
     50447           -13.4%      43684        interrupts.CPU5.LOC:Local_timer_interrupts
     50460           -13.5%      43636        interrupts.CPU50.LOC:Local_timer_interrupts
     50450           -13.5%      43634        interrupts.CPU51.LOC:Local_timer_interrupts
     50423           -13.7%      43495        interrupts.CPU52.LOC:Local_timer_interrupts
     50449           -13.5%      43621        interrupts.CPU53.LOC:Local_timer_interrupts
     50450           -13.5%      43636        interrupts.CPU54.LOC:Local_timer_interrupts
     50447           -13.6%      43607        interrupts.CPU55.LOC:Local_timer_interrupts
     50450           -13.5%      43634        interrupts.CPU56.LOC:Local_timer_interrupts
     50472           -13.6%      43618        interrupts.CPU57.LOC:Local_timer_interrupts
     50467           -13.5%      43650        interrupts.CPU58.LOC:Local_timer_interrupts
     50468           -13.5%      43630        interrupts.CPU59.LOC:Local_timer_interrupts
     50446           -13.5%      43641        interrupts.CPU6.LOC:Local_timer_interrupts
     50423           -13.5%      43634        interrupts.CPU60.LOC:Local_timer_interrupts
     50449           -13.5%      43636        interrupts.CPU61.LOC:Local_timer_interrupts
     50460           -13.6%      43622        interrupts.CPU62.LOC:Local_timer_interrupts
    645.25 ± 12%     +31.2%     846.33 ± 14%  interrupts.CPU62.NMI:Non-maskable_interrupts
    645.25 ± 12%     +31.2%     846.33 ± 14%  interrupts.CPU62.PMI:Performance_monitoring_interrupts
     50466           -13.6%      43628        interrupts.CPU63.LOC:Local_timer_interrupts
     50462           -13.5%      43630        interrupts.CPU64.LOC:Local_timer_interrupts
     50462           -13.6%      43614        interrupts.CPU65.LOC:Local_timer_interrupts
     50488           -13.5%      43653        interrupts.CPU66.LOC:Local_timer_interrupts
     50456           -13.5%      43633        interrupts.CPU67.LOC:Local_timer_interrupts
    629.50 ± 27%     +89.5%       1193 ± 21%  interrupts.CPU67.NMI:Non-maskable_interrupts
    629.50 ± 27%     +89.5%       1193 ± 21%  interrupts.CPU67.PMI:Performance_monitoring_interrupts
     50446           -13.5%      43627        interrupts.CPU68.LOC:Local_timer_interrupts
     50444           -13.5%      43629        interrupts.CPU69.LOC:Local_timer_interrupts
     50448           -13.5%      43643        interrupts.CPU7.LOC:Local_timer_interrupts
     50457           -13.5%      43636        interrupts.CPU70.LOC:Local_timer_interrupts
     50469           -13.5%      43649        interrupts.CPU71.LOC:Local_timer_interrupts
    226.75 ± 35%     -51.3%     110.33 ± 29%  interrupts.CPU71.RES:Rescheduling_interrupts
     50490           -13.9%      43476        interrupts.CPU72.LOC:Local_timer_interrupts
    509.00 ± 58%     -72.2%     141.67 ± 35%  interrupts.CPU72.RES:Rescheduling_interrupts
     50459           -13.5%      43626        interrupts.CPU73.LOC:Local_timer_interrupts
     50349           -13.4%      43609        interrupts.CPU74.LOC:Local_timer_interrupts
      1832 ± 44%     -62.3%     691.33 ± 13%  interrupts.CPU74.NMI:Non-maskable_interrupts
      1832 ± 44%     -62.3%     691.33 ± 13%  interrupts.CPU74.PMI:Performance_monitoring_interrupts
     50470           -13.6%      43622        interrupts.CPU75.LOC:Local_timer_interrupts
     50480           -13.8%      43534        interrupts.CPU76.LOC:Local_timer_interrupts
     50478           -13.5%      43643        interrupts.CPU77.LOC:Local_timer_interrupts
     50472           -13.5%      43637        interrupts.CPU78.LOC:Local_timer_interrupts
     50465           -13.5%      43628        interrupts.CPU79.LOC:Local_timer_interrupts
     50339           -13.4%      43615        interrupts.CPU8.LOC:Local_timer_interrupts
     50469           -13.6%      43628        interrupts.CPU80.LOC:Local_timer_interrupts
     50469           -13.8%      43527        interrupts.CPU81.LOC:Local_timer_interrupts
     50474           -13.6%      43613        interrupts.CPU82.LOC:Local_timer_interrupts
     50484           -13.5%      43669        interrupts.CPU83.LOC:Local_timer_interrupts
     50474           -13.6%      43625        interrupts.CPU84.LOC:Local_timer_interrupts
    324.25 ± 37%     -55.8%     143.33 ± 14%  interrupts.CPU84.RES:Rescheduling_interrupts
     50464           -13.6%      43607        interrupts.CPU85.LOC:Local_timer_interrupts
     50476           -13.6%      43626        interrupts.CPU86.LOC:Local_timer_interrupts
     50477           -13.6%      43608        interrupts.CPU87.LOC:Local_timer_interrupts
     50506           -13.6%      43632        interrupts.CPU88.LOC:Local_timer_interrupts
     50476           -13.6%      43601        interrupts.CPU89.LOC:Local_timer_interrupts
    119.00 ± 39%     -39.8%      71.67 ± 69%  interrupts.CPU9.31:PCI-MSI.524289-edge.eth0-TxRx-0
     50472           -13.6%      43618        interrupts.CPU9.LOC:Local_timer_interrupts
     50473           -13.6%      43627        interrupts.CPU90.LOC:Local_timer_interrupts
     50478           -13.6%      43604        interrupts.CPU91.LOC:Local_timer_interrupts
     50484           -13.6%      43609        interrupts.CPU92.LOC:Local_timer_interrupts
     50426           -13.5%      43595        interrupts.CPU93.LOC:Local_timer_interrupts
     50482           -13.6%      43607        interrupts.CPU94.LOC:Local_timer_interrupts
     50531           -13.6%      43655        interrupts.CPU95.LOC:Local_timer_interrupts
     50421           -13.5%      43611        interrupts.CPU96.LOC:Local_timer_interrupts
     50416           -13.5%      43628        interrupts.CPU97.LOC:Local_timer_interrupts
     50420           -13.5%      43600        interrupts.CPU98.LOC:Local_timer_interrupts
     50437           -13.6%      43601        interrupts.CPU99.LOC:Local_timer_interrupts
   9687948           -13.6%    8373919        interrupts.LOC:Local_timer_interrupts
     62344 ±  4%     -20.1%      49826        interrupts.RES:Rescheduling_interrupts


                                                                                
                      fsmark.time.percent_of_cpu_this_job_got                   
                                                                                
  250 +---------------------------------------------------------------------+   
      |                                                                     |   
      |    O             O O O  O O O  O        O                           |   
  200 |-O    O O  O O O                  O O                                |   
      |.+..+.+    +.+.+..+.+.+..+.+.+.. .+.+..+.+.+..+.+.+..+.+.+..+.+.+..+.|   
      |      :    :                    +                                    |   
  150 |-+    :   :                                                          |   
      |       :  :                                                          |   
  100 |-+     :  :                                                          |   
      |       :  :                                                          |   
      |       : :                                                           |   
   50 |-+     : :                                                           |   
      |        ::                                                           |   
      |        :                                                            |   
    0 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                               fsmark.files_per_sec                             
                                                                                
  600 +---------------------------------------------------------------------+   
      | O  O O O  O O O  O O O  O O O  O O O    O                           |   
  500 |-+                                                                   |   
      |.+..+.+    +.+.+..+.+.+..+.+.+..+.+.+..+.+.+..+.+.+..+.+.+..+.+.+..+.|   
      |      :    :                                                         |   
  400 |-+    :   :                                                          |   
      |       :  :                                                          |   
  300 |-+     :  :                                                          |   
      |       :  :                                                          |   
  200 |-+     : :                                                           |   
      |       : :                                                           |   
      |       : :                                                           |   
  100 |-+      ::                                                           |   
      |        :                                                            |   
    0 +---------------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.5.0-rc3-00038-g244adf6426ee3" of type "text/plain" (202322 bytes)

View attachment "job-script" of type "text/plain" (8152 bytes)

View attachment "job.yaml" of type "text/plain" (5726 bytes)

View attachment "reproduce" of type "text/plain" (2200 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ