lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite for Android: free password hash cracker in your pocket
[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Message-ID: <20191208154145.GM32275@shao2-debian>
Date:   Sun, 8 Dec 2019 23:41:46 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     Chao Yu <yuchao0@...wei.com>
Cc:     Jaegeuk Kim <jaegeuk@...nel.org>,
        LKML <linux-kernel@...r.kernel.org>,
        Linus Torvalds <torvalds@...ux-foundation.org>,
        lkp@...ts.01.org
Subject: [f2fs] fe1897eaa6:  aim7.jobs-per-min -60.9% regression

Greeting,

FYI, we noticed a -60.9% regression of aim7.jobs-per-min due to commit:


commit: fe1897eaa6646f5a64a4cee0e6473ed9887d324b ("f2fs: fix to update time in lazytime mode")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master

in testcase: aim7
on test machine: 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 64G memory
with following parameters:

	disk: 4BRD_12G
	md: RAID1
	fs: f2fs
	test: sync_disk_rw
	load: 200
	cpufreq_governor: performance
	ucode: 0x42e

test-description: AIM7 is a traditional UNIX system level benchmark suite which is used to test and measure the performance of multiuser system.
test-url: https://sourceforge.net/projects/aimbench/files/aim-suite7/



If you fix the issue, kindly add following tag
Reported-by: kernel test robot <rong.a.chen@...el.com>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/load/md/rootfs/tbox_group/test/testcase/ucode:
  gcc-7/performance/4BRD_12G/f2fs/x86_64-rhel-7.6/200/RAID1/debian-x86_64-2019-11-14.cgz/lkp-ivb-2ep1/sync_disk_rw/aim7/0x42e

commit: 
  b145b0eb20 (" ARM and x86 bugfixes of all kinds.  The most visible one is that migrating")
  fe1897eaa6 ("f2fs: fix to update time in lazytime mode")

b145b0eb2031a620 fe1897eaa6646f5a64a4cee0e64 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          1:4          -25%            :4     dmesg.WARNING:at#for_ip_interrupt_entry/0x
         %stddev     %change         %stddev
             \          |                \  
      1208           -60.9%     473.12        aim7.jobs-per-min
    992.70          +155.6%       2537        aim7.time.elapsed_time
    992.70          +155.6%       2537        aim7.time.elapsed_time.max
 1.078e+08           +60.0%  1.724e+08        aim7.time.file_system_outputs
   9646301            -4.2%    9238717        aim7.time.involuntary_context_switches
     24107 ±  3%     +58.2%      38130 ± 17%  aim7.time.minor_page_faults
      3295          +202.3%       9963        aim7.time.system_time
     26.75           +51.4%      40.49        aim7.time.user_time
  5.02e+08          +210.5%  1.559e+09        aim7.time.voluntary_context_switches
   9294452 ±  4%     -15.7%    7835497 ± 16%  numa-numastat.node0.local_node
   9298417 ±  4%     -15.7%    7840991 ± 16%  numa-numastat.node0.numa_hit
      3.66 ±  3%      +4.2        7.82 ± 18%  mpstat.cpu.all.idle%
      0.01 ± 32%      -0.0        0.00 ± 31%  mpstat.cpu.all.soft%
      0.06            -0.0        0.04        mpstat.cpu.all.usr%
      3.85 ±  3%    +105.0%       7.89 ± 17%  iostat.cpu.idle
     89.80            -4.9%      85.36        iostat.cpu.iowait
      6.29            +6.6%       6.70        iostat.cpu.system
     23968           -48.4%      12356        iostat.md0.w/s
     53780           -37.1%      33824        iostat.md0.wkB/s
     89.00            -4.8%      84.75        vmstat.cpu.wa
     53779           -37.1%      33833        vmstat.io.bo
  29311046           -28.6%   20935254        vmstat.memory.free
   1026388           +19.4%    1225080        vmstat.system.cs
    103289            +8.5%     112038        vmstat.system.in
     24013 ± 29%     -58.5%       9976 ±120%  numa-meminfo.node0.Dirty
    119019 ± 10%     +44.6%     172139 ± 13%  numa-meminfo.node1.Active
      3503 ± 56%    +464.8%      19786 ± 38%  numa-meminfo.node1.Active(file)
     40477 ± 16%     -38.5%      24896 ± 47%  numa-meminfo.node1.Dirty
  19966008 ±  2%     -29.5%   14084594 ± 44%  numa-meminfo.node1.MemFree
  13014643 ±  4%     +45.2%   18896052 ± 32%  numa-meminfo.node1.MemUsed
     12018 ± 41%    +184.8%      34228 ± 35%  numa-meminfo.node1.Shmem
  2.09e+10          +127.4%  4.752e+10 ±  4%  cpuidle.C1.time
 2.558e+08          +140.1%  6.143e+08 ±  4%  cpuidle.C1.usage
 3.017e+09 ± 10%    +301.2%  1.211e+10 ±  9%  cpuidle.C1E.time
  18225961 ±  9%    +430.1%   96624314 ±  6%  cpuidle.C1E.usage
 4.627e+08 ± 14%    +147.5%  1.145e+09 ± 20%  cpuidle.C6.time
    802166 ± 17%    +186.9%    2301023 ± 26%  cpuidle.C6.usage
 1.799e+10          +148.9%  4.476e+10        cpuidle.POLL.time
 2.616e+08 ±  2%    +246.4%  9.063e+08 ±  2%  cpuidle.POLL.usage
      6017 ± 29%     -58.5%       2496 ±120%  numa-vmstat.node0.nr_dirty
      5742 ± 29%     -65.0%       2011 ±120%  numa-vmstat.node0.nr_zone_write_pending
    874.50 ± 56%    +465.6%       4946 ± 38%  numa-vmstat.node1.nr_active_file
     10139 ± 16%     -38.6%       6224 ± 47%  numa-vmstat.node1.nr_dirty
   4992144 ±  2%     -29.5%    3521200 ± 44%  numa-vmstat.node1.nr_free_pages
      3003 ± 41%    +184.9%       8556 ± 35%  numa-vmstat.node1.nr_shmem
    874.50 ± 56%    +465.6%       4946 ± 38%  numa-vmstat.node1.nr_zone_active_file
      9775 ± 16%     -47.9%       5089 ± 47%  numa-vmstat.node1.nr_zone_write_pending
 2.558e+08          +140.1%  6.143e+08 ±  4%  turbostat.C1
     43.77            -4.8       38.98 ±  3%  turbostat.C1%
  18225642 ±  9%    +430.2%   96624037 ±  6%  turbostat.C1E
      6.31 ± 10%      +3.6        9.94 ±  9%  turbostat.C1E%
    795117 ± 17%    +188.4%    2292823 ± 26%  turbostat.C6
      0.38 ± 42%     -63.6%       0.14 ± 13%  turbostat.CPU%c6
 1.028e+08          +176.7%  2.844e+08        turbostat.IRQ
      0.23 ± 33%     -69.1%       0.07 ± 22%  turbostat.Pkg%pc2
    275064           +12.3%     308846 ±  4%  meminfo.Active
      8036 ± 21%    +261.7%      29072 ± 52%  meminfo.Active(file)
     64400           -45.9%      34850        meminfo.Dirty
     16481           +10.8%      18264        meminfo.Inactive(anon)
    147067           +12.8%     165858 ±  2%  meminfo.KReclaimable
  29120417           -28.8%   20740075        meminfo.MemAvailable
  29335165           -28.6%   20936804        meminfo.MemFree
  36511133           +23.0%   44909491        meminfo.Memused
    147067           +12.8%     165858 ±  2%  meminfo.SReclaimable
     36567 ±  6%     +37.2%      50173        meminfo.Shmem
     51580           -60.8%      20220        meminfo.max_used_kB
      4.27 ±  7%     -13.9%       3.67 ±  4%  perf-stat.i.MPKI
      0.71 ±  3%      -0.0        0.67        perf-stat.i.branch-miss-rate%
     17.93            +3.5       21.41 ±  2%  perf-stat.i.cache-miss-rate%
   1029337           +19.1%    1226025        perf-stat.i.context-switches
     11421 ±  4%     +41.1%      16120 ±  6%  perf-stat.i.cpu-migrations
      0.42 ±  2%      +0.0        0.45 ±  6%  perf-stat.i.dTLB-load-miss-rate%
      0.11 ± 27%      -0.0        0.08 ±  3%  perf-stat.i.dTLB-store-miss-rate%
     84.04 ±  2%      -2.3       81.75 ±  2%  perf-stat.i.iTLB-load-miss-rate%
   7375571            +7.3%    7917282        perf-stat.i.iTLB-load-misses
      3.91 ±  7%      -9.2%       3.56 ±  4%  perf-stat.overall.MPKI
     17.90            +4.1       21.96 ±  3%  perf-stat.overall.cache-miss-rate%
      0.41 ±  2%      +0.0        0.45 ±  6%  perf-stat.overall.dTLB-load-miss-rate%
      0.10 ± 27%      -0.0        0.08 ±  3%  perf-stat.overall.dTLB-store-miss-rate%
     83.99 ±  2%      -2.3       81.72 ±  2%  perf-stat.overall.iTLB-load-miss-rate%
   1028292           +19.2%    1225532        perf-stat.ps.context-switches
     11410 ±  4%     +41.2%      16114 ±  6%  perf-stat.ps.cpu-migrations
   7368087            +7.4%    7914119        perf-stat.ps.iTLB-load-misses
 2.558e+13 ±  2%    +157.0%  6.575e+13 ±  3%  perf-stat.total.instructions
      9128 ±  2%     -25.3%       6821 ±  4%  slabinfo.dmaengine-unmap-16.active_objs
      9175 ±  2%     -24.4%       6938 ±  4%  slabinfo.dmaengine-unmap-16.num_objs
     28309           +12.5%      31840        slabinfo.f2fs_extent_node.active_objs
     28704           +13.0%      32429        slabinfo.f2fs_extent_node.num_objs
    964.00 ±  5%     -13.3%     836.00 ±  6%  slabinfo.kmem_cache_node.active_objs
      1008 ±  5%     -12.7%     880.00 ±  6%  slabinfo.kmem_cache_node.num_objs
     13017           -33.1%       8707        slabinfo.nat_entry.active_objs
     13036           -33.2%       8707        slabinfo.nat_entry.num_objs
    433.75 ±  9%     +32.4%     574.50 ±  2%  slabinfo.nfs_read_data.active_objs
    433.75 ±  9%     +32.4%     574.50 ±  2%  slabinfo.nfs_read_data.num_objs
    697.50 ±  3%     +48.6%       1036 ± 33%  slabinfo.numa_policy.active_objs
    697.50 ±  3%     +48.6%       1036 ± 33%  slabinfo.numa_policy.num_objs
    153098           +22.3%     187296        slabinfo.radix_tree_node.active_objs
      2733           +22.3%       3344        slabinfo.radix_tree_node.active_slabs
    153098           +22.3%     187296        slabinfo.radix_tree_node.num_objs
      2733           +22.3%       3344        slabinfo.radix_tree_node.num_slabs
    542.00 ± 10%     +29.5%     702.00 ±  6%  slabinfo.xfrm_state.active_objs
    542.00 ± 10%     +29.5%     702.00 ±  6%  slabinfo.xfrm_state.num_objs
     66757            +4.8%      69942        proc-vmstat.nr_active_anon
      2009 ± 21%    +261.7%       7267 ± 52%  proc-vmstat.nr_active_file
     96.00            +3.9%      99.75        proc-vmstat.nr_anon_transparent_hugepages
  13466177           +60.0%   21542226        proc-vmstat.nr_dirtied
     16154           -46.0%       8719        proc-vmstat.nr_dirty
    726643           -28.8%     517541        proc-vmstat.nr_dirty_background_threshold
   1455064           -28.8%    1036348        proc-vmstat.nr_dirty_threshold
    311402            +2.4%     318957        proc-vmstat.nr_file_pages
   7332853           -28.6%    5234441        proc-vmstat.nr_free_pages
      4120           +10.8%       4565        proc-vmstat.nr_inactive_anon
     13939            -1.2%      13772        proc-vmstat.nr_kernel_stack
      2845            -1.5%       2803        proc-vmstat.nr_page_table_pages
      9142 ±  6%     +37.2%      12542        proc-vmstat.nr_shmem
     36767           +12.8%      41463 ±  2%  proc-vmstat.nr_slab_reclaimable
  13387539           +60.3%   21463781        proc-vmstat.nr_written
     66757            +4.8%      69942        proc-vmstat.nr_zone_active_anon
      2009 ± 21%    +261.7%       7267 ± 52%  proc-vmstat.nr_zone_active_file
      4120           +10.8%       4565        proc-vmstat.nr_zone_inactive_anon
     15494           -54.2%       7089        proc-vmstat.nr_zone_write_pending
    582146 ±102%    +628.1%    4238850 ±  6%  proc-vmstat.numa_foreign
     13506 ±  3%    +114.5%      28974 ± 31%  proc-vmstat.numa_hint_faults
      8913 ±  6%    +148.6%      22154 ± 35%  proc-vmstat.numa_hint_faults_local
    582146 ±102%    +628.1%    4238850 ±  6%  proc-vmstat.numa_miss
    597885 ± 99%    +611.6%    4254585 ±  6%  proc-vmstat.numa_other
   1014174 ± 20%     -46.1%     547135 ± 57%  proc-vmstat.pgalloc_dma32
  16157005           +22.4%   19769138        proc-vmstat.pgalloc_normal
   2516271          +152.8%    6360808        proc-vmstat.pgfault
   4866589           +64.4%    8002519        proc-vmstat.pgfree
  53553588           +60.3%   85856803        proc-vmstat.pgpgout
      1.55 ±  4%      +0.1        1.65        perf-profile.calltrace.cycles-pp.__schedule.schedule.io_schedule.wait_on_page_bit.f2fs_wait_on_page_writeback
      1.58 ±  4%      +0.1        1.69        perf-profile.calltrace.cycles-pp.schedule.io_schedule.wait_on_page_bit.f2fs_wait_on_page_writeback.f2fs_wait_on_node_pages_writeback
      1.60 ±  4%      +0.1        1.71        perf-profile.calltrace.cycles-pp.io_schedule.wait_on_page_bit.f2fs_wait_on_page_writeback.f2fs_wait_on_node_pages_writeback.f2fs_do_sync_file
      0.75 ±  2%      +0.2        0.91 ±  2%  perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      0.70 ±  4%      -0.3        0.36 ±  7%  perf-profile.children.cycles-pp.__generic_file_write_iter
      0.69 ±  4%      -0.3        0.35 ±  7%  perf-profile.children.cycles-pp.generic_perform_write
      0.57 ±  5%      -0.3        0.25 ±  6%  perf-profile.children.cycles-pp.__filemap_fdatawrite_range
      0.58 ±  5%      -0.3        0.26 ±  7%  perf-profile.children.cycles-pp.file_write_and_wait_range
      0.57 ±  6%      -0.3        0.25 ±  6%  perf-profile.children.cycles-pp.do_writepages
      0.57 ±  5%      -0.3        0.25 ±  6%  perf-profile.children.cycles-pp.f2fs_write_data_pages
      0.64 ±  5%      -0.3        0.34 ±  6%  perf-profile.children.cycles-pp.f2fs_write_begin
      0.57 ±  5%      -0.2        0.32 ±  7%  perf-profile.children.cycles-pp.f2fs_put_page
      0.31 ± 10%      -0.2        0.11 ±  7%  perf-profile.children.cycles-pp.blk_finish_plug
      0.31 ± 10%      -0.2        0.11 ±  7%  perf-profile.children.cycles-pp.blk_flush_plug_list
      0.30 ± 11%      -0.2        0.11 ±  6%  perf-profile.children.cycles-pp.raid1_unplug
      0.26            -0.1        0.14 ± 12%  perf-profile.children.cycles-pp.f2fs_write_cache_pages
      0.22 ± 20%      -0.1        0.11 ± 41%  perf-profile.children.cycles-pp.brd_do_bvec
      0.14 ± 12%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.submit_bio_wait
      0.21 ± 10%      -0.1        0.11 ± 10%  perf-profile.children.cycles-pp.submit_bio
      0.23 ± 17%      -0.1        0.13 ± 31%  perf-profile.children.cycles-pp.worker_thread
      0.20 ±  3%      -0.1        0.11 ± 14%  perf-profile.children.cycles-pp.__write_data_page
      0.14 ± 15%      -0.1        0.05 ±  9%  perf-profile.children.cycles-pp.__submit_flush_wait
      0.17 ± 12%      -0.1        0.09 ± 12%  perf-profile.children.cycles-pp.md_make_request
      0.16 ± 11%      -0.1        0.08 ± 14%  perf-profile.children.cycles-pp.md_handle_request
      0.15 ±  9%      -0.1        0.07 ± 11%  perf-profile.children.cycles-pp.raid1_make_request
      0.14 ± 17%      -0.1        0.06 ± 11%  perf-profile.children.cycles-pp.f2fs_issue_flush
      0.14 ±  6%      -0.1        0.08 ± 19%  perf-profile.children.cycles-pp.f2fs_do_write_data_page
      0.10 ± 33%      -0.0        0.06 ± 11%  perf-profile.children.cycles-pp.reschedule_interrupt
      0.09 ± 37%      -0.0        0.05 ±  8%  perf-profile.children.cycles-pp.scheduler_ipi
      0.06 ±  6%      +0.0        0.08 ± 10%  perf-profile.children.cycles-pp.up_read
      0.10 ±  4%      +0.0        0.12 ± 13%  perf-profile.children.cycles-pp.pick_next_task_idle
      0.20 ±  2%      +0.0        0.23 ±  5%  perf-profile.children.cycles-pp.get_next_timer_interrupt
      0.16 ±  6%      +0.0        0.19 ±  4%  perf-profile.children.cycles-pp.irq_exit
      0.01 ±173%      +0.1        0.07 ±  7%  perf-profile.children.cycles-pp.put_prev_task_fair
      0.27 ±  3%      +0.1        0.33 ±  4%  perf-profile.children.cycles-pp.tick_nohz_next_event
      0.35 ±  3%      +0.1        0.42 ±  3%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      1.60 ±  4%      +0.1        1.72        perf-profile.children.cycles-pp.io_schedule
      0.78 ±  2%      +0.1        0.92 ±  2%  perf-profile.children.cycles-pp.menu_select
      0.06 ±  6%      +0.0        0.08 ± 10%  perf-profile.self.cycles-pp.up_read
      0.08 ±  5%      +0.0        0.11 ± 11%  perf-profile.self.cycles-pp.cpuidle_enter_state
      0.35 ±  5%      +0.0        0.39 ±  5%  perf-profile.self.cycles-pp.f2fs_wait_on_node_pages_writeback
      0.35 ±  4%      +0.1        0.41 ±  3%  perf-profile.self.cycles-pp.menu_select
      8096 ± 28%    +278.9%      30679 ± 20%  sched_debug.cfs_rq:/.MIN_vruntime.avg
    158062 ± 17%    +299.5%     631442 ±  8%  sched_debug.cfs_rq:/.MIN_vruntime.max
     32629 ± 22%    +289.7%     127148 ± 13%  sched_debug.cfs_rq:/.MIN_vruntime.stddev
     44053          +192.3%     128774        sched_debug.cfs_rq:/.exec_clock.avg
    140530 ±  9%    +377.8%     671471 ± 29%  sched_debug.cfs_rq:/.exec_clock.max
     33108 ±  2%    +109.9%      69484 ±  2%  sched_debug.cfs_rq:/.exec_clock.min
     20291 ±  7%    +386.1%      98642 ± 21%  sched_debug.cfs_rq:/.exec_clock.stddev
      0.25 ± 82%     -88.4%       0.03 ±131%  sched_debug.cfs_rq:/.load_avg.min
      8096 ± 28%    +278.9%      30679 ± 20%  sched_debug.cfs_rq:/.max_vruntime.avg
    158062 ± 17%    +299.5%     631442 ±  8%  sched_debug.cfs_rq:/.max_vruntime.max
     32629 ± 22%    +289.7%     127148 ± 13%  sched_debug.cfs_rq:/.max_vruntime.stddev
    184170          +218.6%     586859 ±  2%  sched_debug.cfs_rq:/.min_vruntime.avg
    251516 ±  4%    +254.5%     891702 ± 11%  sched_debug.cfs_rq:/.min_vruntime.max
    165614          +131.3%     383025 ±  2%  sched_debug.cfs_rq:/.min_vruntime.min
     15286 ±  9%   +1057.3%     176903 ± 10%  sched_debug.cfs_rq:/.min_vruntime.stddev
      0.67 ±  6%     +45.5%       0.98 ±  9%  sched_debug.cfs_rq:/.nr_spread_over.avg
      0.74 ± 20%     +32.4%       0.98 ± 13%  sched_debug.cfs_rq:/.nr_spread_over.stddev
      3.43 ± 29%     -60.7%       1.35 ±100%  sched_debug.cfs_rq:/.removed.load_avg.avg
     75.10 ± 34%     -84.1%      11.91 ±100%  sched_debug.cfs_rq:/.removed.load_avg.max
     15.24 ± 23%     -75.5%       3.74 ±100%  sched_debug.cfs_rq:/.removed.load_avg.stddev
    157.58 ± 29%     -60.5%      62.19 ±100%  sched_debug.cfs_rq:/.removed.runnable_sum.avg
      3459 ± 34%     -84.0%     552.75 ± 99%  sched_debug.cfs_rq:/.removed.runnable_sum.max
    700.79 ± 23%     -75.4%     172.66 ±100%  sched_debug.cfs_rq:/.removed.runnable_sum.stddev
     34.81 ± 42%     -87.0%       4.53 ±100%  sched_debug.cfs_rq:/.removed.util_avg.max
      5.89 ± 31%     -79.0%       1.23 ±100%  sched_debug.cfs_rq:/.removed.util_avg.stddev
     59642 ± 58%    +506.4%     361657 ± 61%  sched_debug.cfs_rq:/.spread0.max
     15286 ±  9%   +1057.2%     176904 ± 10%  sched_debug.cfs_rq:/.spread0.stddev
     15.81 ± 15%     +28.7%      20.34 ±  2%  sched_debug.cfs_rq:/.util_est_enqueued.avg
    553.04 ± 15%     +42.8%     789.60 ±  2%  sched_debug.cfs_rq:/.util_est_enqueued.max
     83.37 ± 15%     +37.7%     114.78 ±  2%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
    130721           -17.9%     107295        sched_debug.cpu.avg_idle.avg
     32650 ±  7%     -16.9%      27145 ±  6%  sched_debug.cpu.avg_idle.min
    509212          +151.7%    1281639        sched_debug.cpu.clock.avg
    509215          +151.7%    1281642        sched_debug.cpu.clock.max
    509209          +151.7%    1281636        sched_debug.cpu.clock.min
    509212          +151.7%    1281639        sched_debug.cpu.clock_task.avg
    509215          +151.7%    1281642        sched_debug.cpu.clock_task.max
    509209          +151.7%    1281636        sched_debug.cpu.clock_task.min
    480.69 ±  2%     +89.2%     909.42 ±  3%  sched_debug.cpu.curr->pid.avg
     14037          +135.4%      33040        sched_debug.cpu.curr->pid.max
      2146 ±  2%    +125.6%       4843        sched_debug.cpu.curr->pid.stddev
      0.12 ±  5%     +10.2%       0.13 ±  3%  sched_debug.cpu.nr_running.avg
  10174129          +208.2%   31360192        sched_debug.cpu.nr_switches.avg
  10849045          +290.1%   42321541 ±  3%  sched_debug.cpu.nr_switches.max
    564008 ± 15%   +1719.4%   10261545 ±  8%  sched_debug.cpu.nr_switches.stddev
    -87.16          +262.4%    -315.83        sched_debug.cpu.nr_uninterruptible.min
     66.84 ± 14%    +143.1%     162.48 ± 21%  sched_debug.cpu.nr_uninterruptible.stddev
  10172460          +208.3%   31358619        sched_debug.cpu.sched_count.avg
  10847113          +290.1%   42319816 ±  3%  sched_debug.cpu.sched_count.max
    563945 ± 15%   +1719.6%   10261603 ±  8%  sched_debug.cpu.sched_count.stddev
   4957584          +211.1%   15425344        sched_debug.cpu.sched_goidle.avg
   5286410          +291.7%   20709361 ±  3%  sched_debug.cpu.sched_goidle.max
    272603 ± 14%   +1717.5%    4954528 ±  7%  sched_debug.cpu.sched_goidle.stddev
   5120024          +209.4%   15841794        sched_debug.cpu.ttwu_count.avg
  58965381 ± 14%    +567.2%  3.934e+08 ± 33%  sched_debug.cpu.ttwu_count.max
    132694 ±  2%     -41.0%      78274 ±  5%  sched_debug.cpu.ttwu_count.min
  11266172 ±  8%    +482.6%   65635267 ± 21%  sched_debug.cpu.ttwu_count.stddev
    114955           +26.1%     144930 ±  3%  sched_debug.cpu.ttwu_local.max
     83595 ±  4%     -53.0%      39284 ± 30%  sched_debug.cpu.ttwu_local.min
      5967 ± 16%    +460.9%      33468 ±  9%  sched_debug.cpu.ttwu_local.stddev
    509209          +151.7%    1281636        sched_debug.cpu_clk
    506629          +152.5%    1279056        sched_debug.ktime
    509627          +151.6%    1282053        sched_debug.sched_clk
     18876          +138.7%      45051 ±  4%  softirqs.CPU0.RCU
    131251 ±  5%    +153.2%     332324 ±  8%  softirqs.CPU0.SCHED
    327888 ±  3%    +181.5%     923135 ± 10%  softirqs.CPU0.TIMER
     19003 ±  4%    +121.7%      42132 ± 12%  softirqs.CPU1.RCU
    125587 ±  4%    +130.3%     289267 ± 22%  softirqs.CPU1.SCHED
    331057 ±  2%    +205.9%    1012636 ± 18%  softirqs.CPU1.TIMER
     19567 ±  5%    +137.8%      46538 ±  2%  softirqs.CPU10.RCU
    131681          +161.3%     344047        softirqs.CPU10.SCHED
    318774          +177.5%     884714 ±  8%  softirqs.CPU10.TIMER
     17876 ±  5%    +158.8%      46258        softirqs.CPU11.RCU
    128299 ±  4%    +167.5%     343196        softirqs.CPU11.SCHED
    320847 ±  3%    +166.4%     854607 ±  8%  softirqs.CPU11.TIMER
     19129 ±  4%    +121.1%      42292 ±  4%  softirqs.CPU12.RCU
    135251          +172.0%     367942        softirqs.CPU12.SCHED
    310663          +201.8%     937514 ± 24%  softirqs.CPU12.TIMER
     17993          +143.9%      43882 ±  3%  softirqs.CPU13.RCU
    137747 ±  4%    +157.5%     354715 ±  4%  softirqs.CPU13.SCHED
    359507 ± 23%    +126.2%     813048 ± 13%  softirqs.CPU13.TIMER
     18173          +141.9%      43961 ±  3%  softirqs.CPU14.RCU
    134564          +163.8%     355019 ±  4%  softirqs.CPU14.SCHED
    310599 ±  2%    +160.3%     808494 ± 13%  softirqs.CPU14.TIMER
     18180          +139.4%      43524 ±  3%  softirqs.CPU15.RCU
    134993          +160.8%     352061 ±  6%  softirqs.CPU15.SCHED
    308373          +164.7%     816152 ± 12%  softirqs.CPU15.TIMER
     19918          +144.6%      48716        softirqs.CPU16.RCU
    134609          +162.6%     353418 ±  4%  softirqs.CPU16.SCHED
    307916          +163.0%     809772 ± 13%  softirqs.CPU16.TIMER
     19672 ±  2%    +148.8%      48938 ±  3%  softirqs.CPU17.RCU
    132926 ±  2%    +165.9%     353452 ±  4%  softirqs.CPU17.SCHED
    314316          +157.0%     807880 ± 13%  softirqs.CPU17.TIMER
     20021 ±  3%    +141.2%      48298 ±  2%  softirqs.CPU18.RCU
    133347 ±  2%    +164.0%     352038 ±  5%  softirqs.CPU18.SCHED
    312605          +158.6%     808358 ± 13%  softirqs.CPU18.TIMER
     19458          +139.3%      46570 ±  6%  softirqs.CPU19.RCU
    134001          +150.1%     335170 ± 14%  softirqs.CPU19.SCHED
    310521          +171.0%     841639 ± 14%  softirqs.CPU19.TIMER
     19131 ±  3%    +138.3%      45586 ±  3%  softirqs.CPU2.RCU
    129111 ±  4%    +154.4%     328419 ±  9%  softirqs.CPU2.SCHED
    329064 ±  3%    +172.7%     897239 ± 10%  softirqs.CPU2.TIMER
     19249 ±  2%    +145.5%      47257 ±  3%  softirqs.CPU20.RCU
    132532          +167.8%     354983 ±  3%  softirqs.CPU20.SCHED
    310659          +163.4%     818268 ± 13%  softirqs.CPU20.TIMER
     19473          +142.2%      47163 ±  4%  softirqs.CPU21.RCU
    133856          +166.0%     356005 ±  4%  softirqs.CPU21.SCHED
    308960          +164.3%     816637 ± 13%  softirqs.CPU21.TIMER
     19216 ±  2%    +146.1%      47292 ±  3%  softirqs.CPU22.RCU
    133039          +165.4%     353140 ±  4%  softirqs.CPU22.SCHED
    307621          +166.0%     818260 ± 13%  softirqs.CPU22.TIMER
     19244 ±  2%    +145.9%      47314 ±  2%  softirqs.CPU23.RCU
    133897          +163.1%     352326 ±  5%  softirqs.CPU23.SCHED
    310036 ±  2%    +161.6%     811061 ± 13%  softirqs.CPU23.TIMER
     18954 ±  4%    +143.9%      46235 ±  4%  softirqs.CPU24.RCU
    128754 ±  3%    +157.8%     331927 ±  7%  softirqs.CPU24.SCHED
    323118          +169.7%     871609 ±  8%  softirqs.CPU24.TIMER
     18942 ±  6%    +136.0%      44695 ±  3%  softirqs.CPU25.RCU
    129580 ±  3%    +162.2%     339786 ±  3%  softirqs.CPU25.SCHED
    328907          +168.0%     881566 ±  7%  softirqs.CPU25.TIMER
     18913 ±  2%    +147.2%      46757 ±  2%  softirqs.CPU26.RCU
    124714 ±  2%    +176.4%     344692        softirqs.CPU26.SCHED
    328463 ±  4%    +170.2%     887463 ±  8%  softirqs.CPU26.TIMER
     19015          +137.2%      45107 ±  5%  softirqs.CPU27.RCU
    128551 ±  6%    +152.0%     323916 ± 13%  softirqs.CPU27.SCHED
    325384 ±  2%    +193.3%     954266 ± 13%  softirqs.CPU27.TIMER
     19094          +149.9%      47725 ±  2%  softirqs.CPU28.RCU
    131563          +161.9%     344579 ±  2%  softirqs.CPU28.SCHED
    319151          +176.7%     883105 ±  9%  softirqs.CPU28.TIMER
     19050 ±  4%    +145.4%      46756 ±  2%  softirqs.CPU29.RCU
    124211 ±  6%    +176.0%     342763 ±  2%  softirqs.CPU29.SCHED
    329526 ±  4%    +158.9%     853085 ±  8%  softirqs.CPU29.TIMER
     22721 ±  8%    +121.7%      50370 ±  4%  softirqs.CPU3.RCU
    132516 ±  2%    +156.9%     340453 ±  4%  softirqs.CPU3.SCHED
    323552 ±  2%    +174.3%     887612 ±  9%  softirqs.CPU3.TIMER
     19911 ±  3%    +137.4%      47276        softirqs.CPU30.RCU
    130778          +161.4%     341879        softirqs.CPU30.SCHED
    317264          +181.0%     891407 ±  8%  softirqs.CPU30.TIMER
     19232 ±  4%    +144.4%      47012        softirqs.CPU31.RCU
    126109 ± 10%    +174.9%     346628 ±  2%  softirqs.CPU31.SCHED
    329439 ±  4%    +169.4%     887674 ±  9%  softirqs.CPU31.TIMER
     18139 ±  4%    +141.4%      43795 ±  4%  softirqs.CPU32.RCU
    125566 ±  4%    +170.9%     340112 ±  4%  softirqs.CPU32.SCHED
    328833          +184.0%     934027 ±  9%  softirqs.CPU32.TIMER
     19309 ± 10%    +128.3%      44077 ±  3%  softirqs.CPU33.RCU
    128438 ±  3%    +167.7%     343879 ±  2%  softirqs.CPU33.SCHED
    320949          +172.2%     873631 ±  9%  softirqs.CPU33.TIMER
     18041 ±  4%    +146.1%      44399 ±  2%  softirqs.CPU34.RCU
    129791          +165.0%     343914 ±  2%  softirqs.CPU34.SCHED
    319403 ±  2%    +177.3%     885711 ±  8%  softirqs.CPU34.TIMER
     17991 ±  5%    +146.2%      44290 ±  2%  softirqs.CPU35.RCU
    137112 ±  7%    +148.6%     340824 ±  3%  softirqs.CPU35.SCHED
    416552 ± 21%    +119.5%     914435 ±  8%  softirqs.CPU35.TIMER
     20036 ±  4%    +138.7%      47823 ±  4%  softirqs.CPU36.RCU
    138048 ±  3%    +165.7%     366854 ±  7%  softirqs.CPU36.SCHED
    359442 ± 22%    +196.1%    1064343 ± 19%  softirqs.CPU36.TIMER
     19421 ±  5%    +153.4%      49203 ±  2%  softirqs.CPU37.RCU
    135462          +158.7%     350500 ±  6%  softirqs.CPU37.SCHED
    305531          +166.8%     815287 ± 12%  softirqs.CPU37.TIMER
     19963          +150.5%      50014        softirqs.CPU38.RCU
    132257          +167.5%     353848 ±  5%  softirqs.CPU38.SCHED
    311979 ±  2%    +160.3%     812118 ± 12%  softirqs.CPU38.TIMER
     20119 ±  2%    +143.6%      49011 ±  2%  softirqs.CPU39.RCU
    135450          +161.9%     354713 ±  4%  softirqs.CPU39.SCHED
    307797          +164.3%     813377 ± 12%  softirqs.CPU39.TIMER
     20593 ±  9%    +132.1%      47796 ±  5%  softirqs.CPU4.RCU
    129881          +159.2%     336643 ±  3%  softirqs.CPU4.SCHED
    321940          +175.0%     885297 ±  8%  softirqs.CPU4.TIMER
     19874 ±  2%    +149.6%      49611        softirqs.CPU40.RCU
    134660          +161.9%     352636 ±  5%  softirqs.CPU40.SCHED
    309870          +160.6%     807374 ± 13%  softirqs.CPU40.TIMER
     19897 ±  2%    +147.2%      49186 ±  2%  softirqs.CPU41.RCU
    131978 ±  2%    +167.4%     352864 ±  5%  softirqs.CPU41.SCHED
    315198          +157.6%     812013 ± 12%  softirqs.CPU41.TIMER
     19988 ±  2%    +143.9%      48749 ±  2%  softirqs.CPU42.RCU
    133326 ±  2%    +165.6%     354135 ±  4%  softirqs.CPU42.SCHED
    313042          +159.5%     812298 ± 12%  softirqs.CPU42.TIMER
     19723 ±  2%    +141.9%      47704 ±  4%  softirqs.CPU43.RCU
    131944          +165.7%     350566 ±  5%  softirqs.CPU43.SCHED
    314428          +162.9%     826595 ± 13%  softirqs.CPU43.TIMER
     19824 ±  2%    +143.0%      48177 ±  4%  softirqs.CPU44.RCU
    132047 ±  2%    +157.8%     340362 ± 12%  softirqs.CPU44.SCHED
    312615 ±  2%    +165.2%     829094 ± 12%  softirqs.CPU44.TIMER
     20140 ±  2%    +138.6%      48049 ±  7%  softirqs.CPU45.RCU
    133846          +151.7%     336901 ± 14%  softirqs.CPU45.SCHED
    309086 ±  2%    +170.8%     837094 ± 12%  softirqs.CPU45.TIMER
     20103 ±  3%    +138.8%      48009 ±  3%  softirqs.CPU46.RCU
    133860          +154.1%     340108 ± 11%  softirqs.CPU46.SCHED
    311323          +166.1%     828498 ± 13%  softirqs.CPU46.TIMER
     19628 ±  2%    +149.0%      48881 ±  2%  softirqs.CPU47.RCU
    132835 ±  2%    +164.6%     351487 ±  5%  softirqs.CPU47.SCHED
    309264          +162.8%     812666 ± 13%  softirqs.CPU47.TIMER
     19773 ±  2%    +139.0%      47263 ±  2%  softirqs.CPU5.RCU
    130891 ±  2%    +163.7%     345185        softirqs.CPU5.SCHED
    323979 ±  2%    +182.2%     914392 ±  8%  softirqs.CPU5.TIMER
     19904 ±  3%    +141.4%      48045 ±  4%  softirqs.CPU6.RCU
    131836          +160.9%     344009        softirqs.CPU6.SCHED
    316150          +181.7%     890614 ±  8%  softirqs.CPU6.TIMER
     19077 ±  3%    +145.1%      46751 ±  3%  softirqs.CPU7.RCU
    127906 ±  2%    +157.6%     329548 ±  5%  softirqs.CPU7.SCHED
    325087          +181.6%     915494 ±  9%  softirqs.CPU7.TIMER
     20183 ±  2%    +114.9%      43380 ± 10%  softirqs.CPU8.RCU
    131974          +118.3%     288124 ± 30%  softirqs.CPU8.SCHED
    319626          +190.1%     927161 ± 13%  softirqs.CPU8.TIMER
     19292 ±  3%    +137.6%      45839 ±  3%  softirqs.CPU9.RCU
    129715          +158.9%     335788 ±  4%  softirqs.CPU9.SCHED
    321209          +178.0%     893035 ±  8%  softirqs.CPU9.TIMER
    931061          +140.9%    2242787        softirqs.RCU
   6322319          +160.8%   16487173        softirqs.SCHED
  15417545          +169.7%   41581783 ±  8%  softirqs.TIMER
    853.75 ± 12%    +139.9%       2048 ±  7%  interrupts.35:PCI-MSI.2621441-edge.eth0-TxRx-0
    586.00 ± 14%    +130.1%       1348 ±  5%  interrupts.36:PCI-MSI.2621442-edge.eth0-TxRx-1
    651.00 ±  9%    +136.3%       1538 ±  9%  interrupts.37:PCI-MSI.2621443-edge.eth0-TxRx-2
    565.25 ± 14%    +129.7%       1298        interrupts.38:PCI-MSI.2621444-edge.eth0-TxRx-3
    591.00 ± 14%    +123.8%       1322 ±  2%  interrupts.39:PCI-MSI.2621445-edge.eth0-TxRx-4
    660.25 ± 13%    +111.4%       1396 ±  7%  interrupts.40:PCI-MSI.2621446-edge.eth0-TxRx-5
    529.50 ±  4%    +227.1%       1732 ± 14%  interrupts.41:PCI-MSI.2621447-edge.eth0-TxRx-6
    585.75 ± 16%    +163.2%       1541 ± 10%  interrupts.42:PCI-MSI.2621448-edge.eth0-TxRx-7
    404968          +145.6%     994665 ±  3%  interrupts.CAL:Function_call_interrupts
      8396 ±  6%    +133.7%      19627 ±  6%  interrupts.CPU0.CAL:Function_call_interrupts
   1988743          +155.3%    5077644        interrupts.CPU0.LOC:Local_timer_interrupts
      8698 ±  5%    +121.4%      19255 ±  2%  interrupts.CPU1.CAL:Function_call_interrupts
   1986989          +155.5%    5077206        interrupts.CPU1.LOC:Local_timer_interrupts
      8487 ±  5%    +132.3%      19713 ±  8%  interrupts.CPU10.CAL:Function_call_interrupts
   1988375          +155.4%    5077322        interrupts.CPU10.LOC:Local_timer_interrupts
      8770 ±  5%    +149.6%      21889 ±  3%  interrupts.CPU11.CAL:Function_call_interrupts
   1987964          +155.4%    5077484        interrupts.CPU11.LOC:Local_timer_interrupts
     78678 ±  8%    +541.4%     504605 ±122%  interrupts.CPU11.RES:Rescheduling_interrupts
      8731 ±  9%    +119.5%      19167 ±  6%  interrupts.CPU12.CAL:Function_call_interrupts
   1988222          +155.4%    5077524        interrupts.CPU12.LOC:Local_timer_interrupts
      7960 ±  3%    +153.5%      20176 ±  9%  interrupts.CPU13.CAL:Function_call_interrupts
   1988667          +155.3%    5076359        interrupts.CPU13.LOC:Local_timer_interrupts
      8518 ±  6%    +155.0%      21723 ±  7%  interrupts.CPU14.CAL:Function_call_interrupts
   1987350          +155.5%    5077482        interrupts.CPU14.LOC:Local_timer_interrupts
     37.00 ± 31%    +124.3%      83.00 ± 24%  interrupts.CPU14.TLB:TLB_shootdowns
      8831 ±  4%    +137.4%      20964 ± 10%  interrupts.CPU15.CAL:Function_call_interrupts
   1987501          +155.5%    5077129        interrupts.CPU15.LOC:Local_timer_interrupts
      3708 ± 27%     -63.0%       1370 ±103%  interrupts.CPU15.NMI:Non-maskable_interrupts
      3708 ± 27%     -63.0%       1370 ±103%  interrupts.CPU15.PMI:Performance_monitoring_interrupts
     23.25 ± 24%    +253.8%      82.25 ± 28%  interrupts.CPU15.TLB:TLB_shootdowns
      8708 ±  8%    +132.7%      20268 ±  7%  interrupts.CPU16.CAL:Function_call_interrupts
   1988418          +155.3%    5076923        interrupts.CPU16.LOC:Local_timer_interrupts
      8548 ±  7%    +148.3%      21226 ±  7%  interrupts.CPU17.CAL:Function_call_interrupts
   1988416          +155.3%    5077317        interrupts.CPU17.LOC:Local_timer_interrupts
     33.00 ± 18%    +147.0%      81.50 ± 21%  interrupts.CPU17.TLB:TLB_shootdowns
      8456 ±  9%    +157.0%      21731 ±  6%  interrupts.CPU18.CAL:Function_call_interrupts
   1988574          +155.3%    5077430        interrupts.CPU18.LOC:Local_timer_interrupts
     30.25 ± 15%    +181.8%      85.25 ± 25%  interrupts.CPU18.TLB:TLB_shootdowns
      8884 ±  6%    +144.5%      21720 ±  3%  interrupts.CPU19.CAL:Function_call_interrupts
   1987747          +155.4%    5077260        interrupts.CPU19.LOC:Local_timer_interrupts
     37.00 ± 19%    +127.0%      84.00 ± 26%  interrupts.CPU19.TLB:TLB_shootdowns
      8298 ± 11%    +145.3%      20352 ±  9%  interrupts.CPU2.CAL:Function_call_interrupts
   1988446          +155.4%    5077740        interrupts.CPU2.LOC:Local_timer_interrupts
      8594 ±  5%    +146.1%      21149 ±  5%  interrupts.CPU20.CAL:Function_call_interrupts
   1987855          +155.4%    5077226        interrupts.CPU20.LOC:Local_timer_interrupts
      8376 ±  8%    +151.7%      21084 ±  9%  interrupts.CPU21.CAL:Function_call_interrupts
   1988171          +155.4%    5077212        interrupts.CPU21.LOC:Local_timer_interrupts
     41.25 ± 20%    +106.1%      85.00 ± 27%  interrupts.CPU21.TLB:TLB_shootdowns
      9024 ±  2%    +141.1%      21757 ±  3%  interrupts.CPU22.CAL:Function_call_interrupts
   1987613          +155.4%    5077125        interrupts.CPU22.LOC:Local_timer_interrupts
      8095 ±  7%    +154.9%      20633 ±  6%  interrupts.CPU23.CAL:Function_call_interrupts
   1989315          +155.2%    5077451        interrupts.CPU23.LOC:Local_timer_interrupts
    853.75 ± 12%    +139.9%       2048 ±  7%  interrupts.CPU24.35:PCI-MSI.2621441-edge.eth0-TxRx-0
      8463 ±  5%    +147.0%      20901 ±  7%  interrupts.CPU24.CAL:Function_call_interrupts
   1988251          +155.4%    5077879        interrupts.CPU24.LOC:Local_timer_interrupts
     74848 ±  8%    +525.2%     467985 ±128%  interrupts.CPU24.RES:Rescheduling_interrupts
    586.00 ± 14%    +130.1%       1348 ±  5%  interrupts.CPU25.36:PCI-MSI.2621442-edge.eth0-TxRx-1
      8156 ±  7%    +170.0%      22019 ±  2%  interrupts.CPU25.CAL:Function_call_interrupts
   1988295          +155.3%    5077010        interrupts.CPU25.LOC:Local_timer_interrupts
    930.75 ±123%    +260.9%       3359 ± 15%  interrupts.CPU25.NMI:Non-maskable_interrupts
    930.75 ±123%    +260.9%       3359 ± 15%  interrupts.CPU25.PMI:Performance_monitoring_interrupts
     72678 ±  7%    +575.9%     491267 ±120%  interrupts.CPU25.RES:Rescheduling_interrupts
    651.00 ±  9%    +136.3%       1538 ±  9%  interrupts.CPU26.37:PCI-MSI.2621443-edge.eth0-TxRx-2
      8538 ±  6%    +141.5%      20625 ±  6%  interrupts.CPU26.CAL:Function_call_interrupts
   1988053          +155.4%    5077342        interrupts.CPU26.LOC:Local_timer_interrupts
     75367 ±  8%    +522.0%     468794 ±133%  interrupts.CPU26.RES:Rescheduling_interrupts
    565.25 ± 14%    +129.7%       1298        interrupts.CPU27.38:PCI-MSI.2621444-edge.eth0-TxRx-3
      8157 ±  5%    +132.5%      18969 ±  3%  interrupts.CPU27.CAL:Function_call_interrupts
   1988437          +155.3%    5077428        interrupts.CPU27.LOC:Local_timer_interrupts
    591.00 ± 14%    +123.8%       1322 ±  2%  interrupts.CPU28.39:PCI-MSI.2621445-edge.eth0-TxRx-4
      8181 ±  6%    +148.1%      20300 ±  3%  interrupts.CPU28.CAL:Function_call_interrupts
   1987807          +155.4%    5077738        interrupts.CPU28.LOC:Local_timer_interrupts
     71071 ±  7%    +581.0%     483987 ±128%  interrupts.CPU28.RES:Rescheduling_interrupts
    660.25 ± 13%    +111.4%       1396 ±  7%  interrupts.CPU29.40:PCI-MSI.2621446-edge.eth0-TxRx-5
      8178 ±  5%    +150.4%      20475 ±  7%  interrupts.CPU29.CAL:Function_call_interrupts
   1987852          +155.4%    5077382        interrupts.CPU29.LOC:Local_timer_interrupts
     70205 ±  8%    +585.5%     481286 ±127%  interrupts.CPU29.RES:Rescheduling_interrupts
     17.75 ± 36%    +163.4%      46.75 ± 80%  interrupts.CPU29.TLB:TLB_shootdowns
      8684 ±  5%    +151.7%      21860 ±  4%  interrupts.CPU3.CAL:Function_call_interrupts
   1988786          +155.3%    5077436        interrupts.CPU3.LOC:Local_timer_interrupts
     80407 ±  4%    +518.6%     497419 ±122%  interrupts.CPU3.RES:Rescheduling_interrupts
    529.50 ±  4%    +227.1%       1732 ± 14%  interrupts.CPU30.41:PCI-MSI.2621447-edge.eth0-TxRx-6
      8415 ±  4%    +138.0%      20031 ±  3%  interrupts.CPU30.CAL:Function_call_interrupts
   1988609          +155.3%    5077494        interrupts.CPU30.LOC:Local_timer_interrupts
     74875 ±  8%    +518.8%     463337 ±134%  interrupts.CPU30.RES:Rescheduling_interrupts
    585.75 ± 16%    +163.2%       1541 ± 10%  interrupts.CPU31.42:PCI-MSI.2621448-edge.eth0-TxRx-7
      7917          +151.5%      19913 ±  3%  interrupts.CPU31.CAL:Function_call_interrupts
   1988677          +155.3%    5077502        interrupts.CPU31.LOC:Local_timer_interrupts
     67596 ±  3%    +596.5%     470844 ±134%  interrupts.CPU31.RES:Rescheduling_interrupts
      8230 ±  4%    +161.9%      21551 ±  8%  interrupts.CPU32.CAL:Function_call_interrupts
   1988317          +155.4%    5077410        interrupts.CPU32.LOC:Local_timer_interrupts
     70449 ±  5%    +597.2%     491174 ±127%  interrupts.CPU32.RES:Rescheduling_interrupts
      8684 ±  6%    +137.3%      20610 ±  6%  interrupts.CPU33.CAL:Function_call_interrupts
   1988440          +155.3%    5077281        interrupts.CPU33.LOC:Local_timer_interrupts
     77401 ±  5%    +495.8%     461151 ±134%  interrupts.CPU33.RES:Rescheduling_interrupts
      8412 ±  8%    +150.3%      21061 ±  4%  interrupts.CPU34.CAL:Function_call_interrupts
   1988447          +155.4%    5077546        interrupts.CPU34.LOC:Local_timer_interrupts
     75351 ±  8%    +537.5%     480376 ±128%  interrupts.CPU34.RES:Rescheduling_interrupts
      7890          +147.7%      19543 ±  5%  interrupts.CPU35.CAL:Function_call_interrupts
   1988875          +155.3%    5077717        interrupts.CPU35.LOC:Local_timer_interrupts
     70158 ±  2%    +577.1%     475011 ±135%  interrupts.CPU35.RES:Rescheduling_interrupts
      7908 ±  2%    +184.9%      22530 ±  3%  interrupts.CPU36.CAL:Function_call_interrupts
   1988403          +155.4%    5077694        interrupts.CPU36.LOC:Local_timer_interrupts
     31.75 ± 20%    +178.0%      88.25 ± 28%  interrupts.CPU36.TLB:TLB_shootdowns
      9078 ±  2%    +133.8%      21229 ±  6%  interrupts.CPU37.CAL:Function_call_interrupts
   1986971          +155.5%    5076871        interrupts.CPU37.LOC:Local_timer_interrupts
     34.75 ± 36%    +154.7%      88.50 ± 25%  interrupts.CPU37.TLB:TLB_shootdowns
      8574 ±  7%    +131.7%      19864 ±  3%  interrupts.CPU38.CAL:Function_call_interrupts
   1986780          +155.5%    5076769        interrupts.CPU38.LOC:Local_timer_interrupts
     29.25 ± 35%    +195.7%      86.50 ± 18%  interrupts.CPU38.TLB:TLB_shootdowns
      8184 ±  7%    +158.0%      21112 ±  4%  interrupts.CPU39.CAL:Function_call_interrupts
   1986972          +155.5%    5077337        interrupts.CPU39.LOC:Local_timer_interrupts
      8368 ±  8%    +144.7%      20481 ± 12%  interrupts.CPU4.CAL:Function_call_interrupts
   1987774          +155.4%    5077506        interrupts.CPU4.LOC:Local_timer_interrupts
      8165 ±  3%    +163.7%      21528 ±  6%  interrupts.CPU40.CAL:Function_call_interrupts
   1987659          +155.4%    5077421        interrupts.CPU40.LOC:Local_timer_interrupts
     32.00 ± 12%    +154.7%      81.50 ± 29%  interrupts.CPU40.TLB:TLB_shootdowns
      8454 ±  6%    +158.4%      21847 ±  8%  interrupts.CPU41.CAL:Function_call_interrupts
   1987504          +155.5%    5077467        interrupts.CPU41.LOC:Local_timer_interrupts
     37.75 ± 45%    +122.5%      84.00 ± 17%  interrupts.CPU41.TLB:TLB_shootdowns
      8529 ±  7%    +136.4%      20164 ±  8%  interrupts.CPU42.CAL:Function_call_interrupts
   1988401          +155.4%    5077407        interrupts.CPU42.LOC:Local_timer_interrupts
     32.75 ± 39%    +171.8%      89.00 ± 37%  interrupts.CPU42.TLB:TLB_shootdowns
      8102 ±  5%    +150.4%      20283 ±  9%  interrupts.CPU43.CAL:Function_call_interrupts
   1987414          +155.5%    5077903        interrupts.CPU43.LOC:Local_timer_interrupts
     32.50 ± 18%    +150.8%      81.50 ± 34%  interrupts.CPU43.TLB:TLB_shootdowns
      8206 ±  7%    +157.3%      21112 ±  8%  interrupts.CPU44.CAL:Function_call_interrupts
   1987278          +155.5%    5077309        interrupts.CPU44.LOC:Local_timer_interrupts
     37.00 ± 25%    +119.6%      81.25 ± 24%  interrupts.CPU44.TLB:TLB_shootdowns
      8467 ±  7%    +139.3%      20261 ±  5%  interrupts.CPU45.CAL:Function_call_interrupts
   1987858          +155.4%    5077234        interrupts.CPU45.LOC:Local_timer_interrupts
     38.25 ± 22%    +113.1%      81.50 ± 27%  interrupts.CPU45.TLB:TLB_shootdowns
      7773 ±  4%    +162.0%      20367 ±  9%  interrupts.CPU46.CAL:Function_call_interrupts
   1988584          +155.3%    5077417        interrupts.CPU46.LOC:Local_timer_interrupts
     37.50 ± 14%    +142.7%      91.00 ± 34%  interrupts.CPU46.TLB:TLB_shootdowns
      8746 ±  4%    +138.7%      20880 ±  8%  interrupts.CPU47.CAL:Function_call_interrupts
   1988495          +155.3%    5077204        interrupts.CPU47.LOC:Local_timer_interrupts
      8722 ±  7%    +134.1%      20414 ± 10%  interrupts.CPU5.CAL:Function_call_interrupts
   1987875          +155.4%    5077418        interrupts.CPU5.LOC:Local_timer_interrupts
      8435 ±  8%    +143.9%      20570 ± 11%  interrupts.CPU6.CAL:Function_call_interrupts
   1988389          +155.4%    5077630        interrupts.CPU6.LOC:Local_timer_interrupts
      8837 ±  5%    +139.1%      21133 ±  9%  interrupts.CPU7.CAL:Function_call_interrupts
   1987853          +155.4%    5077370        interrupts.CPU7.LOC:Local_timer_interrupts
      8852 ±  5%    +123.8%      19810 ±  3%  interrupts.CPU8.CAL:Function_call_interrupts
   1988702          +155.3%    5077569        interrupts.CPU8.LOC:Local_timer_interrupts
      8273 ±  6%    +151.1%      20771 ±  9%  interrupts.CPU9.CAL:Function_call_interrupts
   1988033          +155.4%    5077588        interrupts.CPU9.LOC:Local_timer_interrupts
     70739 ±  6%    +578.9%     480243 ±129%  interrupts.CPU9.RES:Rescheduling_interrupts
  95428175          +155.4%  2.437e+08        interrupts.LOC:Local_timer_interrupts
    144.00          +133.3%     336.00        interrupts.MCP:Machine_check_polls
   6773889 ±  8%    +482.9%   39483682 ±  7%  interrupts.RES:Rescheduling_interrupts
      1544 ±  5%     +91.5%       2957 ± 11%  interrupts.TLB:TLB_shootdowns


                                                                                
                                 aim7.jobs-per-min                              
                                                                                
  1300 +-+------------------------------------------------------------------+   
  1200 +-+.++.+.+.+.+. +.+.+.+. +.+.+   +.++.+.+.+.++.+.+.+.++.+.+.+.+.++.+.|   
       | +            +        +                                            |   
  1100 +-+                                                                  |   
  1000 +-+                                                                  |   
       |                                                                    |   
   900 +-+                                                                  |   
   800 +-+                                                                  |   
   700 +-+                                                                  |   
       |                                                                    |   
   600 +-+                                                                  |   
   500 +-+                                                  O               |   
       |                                            O O O O  O O O          |   
   400 O-O OO O O O O OO O O O OO O O O O OO O O O O                        |   
   300 +-+------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                               aim7.time.user_time                              
                                                                                
  55 +-+--------------------------------------------------------------------+   
     |                                                                      |   
  50 +-+          O                                                         |   
     |                O                                                     |   
     |                                                                      |   
  45 +-+                                                                    |   
     |                  O O  O O O   O   O    O O O                         |   
  40 O-O O OO O O   O      O       O   O  O O       O OO O O O O O          |   
     |                                                                      |   
  35 +-+                                                                    |   
     |                                                                      |   
     |                                                                      |   
  30 +-+                                                                    |   
     |.+.+.++.+.+.+.+.+.+.++.+.+.+.+.+.+.++.+.+.+.+.+.++.+. .+.+.+.+.++.+.+.|   
  25 +-+--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                aim7.time.system_time                           
                                                                                
  11000 +-+-----------------------------------------------------------------+   
        O O OO O O O OO O O OO O O O OO O O O OO O O                        |   
  10000 +-+                                          O OO O O OO O          |   
   9000 +-+                                                                 |   
        |                                                                   |   
   8000 +-+                                                                 |   
        |                                                                   |   
   7000 +-+                                                                 |   
        |                                                                   |   
   6000 +-+                                                                 |   
   5000 +-+                                                                 |   
        |                                                                   |   
   4000 +-+                                                                 |   
        |  .++. .+.    .+.+.    .+.+.        .++.     .+ .+.+.+ .+.   .++.+.|   
   3000 +-+-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                              aim7.time.elapsed_time                            
                                                                                
  3500 +-+------------------------------------------------------------------+   
       O O OO O O O O OO O O   OO O O O O OO O O O O                        |   
  3000 +-+                   O                                              |   
       |                                                                    |   
       |                                            O O O O    O O          |   
  2500 +-+                                                  OO              |   
       |                                                                    |   
  2000 +-+                                                                  |   
       |                                                                    |   
  1500 +-+                                                                  |   
       |                                                                    |   
       |                                                                    |   
  1000 +-+.++.+.+.+.+.++.+.+.+.++.+.+.+.+.++.+.+.+.++.+.+.+.++.+.+.+.+.++.+.|   
       |                                                                    |   
   500 +-+------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                            aim7.time.elapsed_time.max                          
                                                                                
  3500 +-+------------------------------------------------------------------+   
       O O OO O O O O OO O O   OO O O O O OO O O O O                        |   
  3000 +-+                   O                                              |   
       |                                                                    |   
       |                                            O O O O    O O          |   
  2500 +-+                                                  OO              |   
       |                                                                    |   
  2000 +-+                                                                  |   
       |                                                                    |   
  1500 +-+                                                                  |   
       |                                                                    |   
       |                                                                    |   
  1000 +-+.++.+.+.+.+.++.+.+.+.++.+.+.+.+.++.+.+.+.++.+.+.+.++.+.+.+.+.++.+.|   
       |                                                                    |   
   500 +-+------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                         aim7.time.voluntary_context_switches                   
                                                                                
  1.8e+09 +-+---------------------------------------------------------------+   
          O  O O O OO   O OO O O OO O O OO O OO O   O                       |   
  1.6e+09 +-O         O                           O  O O O OO O O O         |   
          |                                                                 |   
  1.4e+09 +-+                                                               |   
          |                                                                 |   
  1.2e+09 +-+                                                               |   
          |                                                                 |   
    1e+09 +-+                                                               |   
          |                                                                 |   
    8e+08 +-+                                                               |   
          |                                                                 |   
    6e+08 +-+                                                               |   
          |.++.+.+.++.+.+.++.+.+.++.+.+.++.+.++.+.+.++.+.+.++.+.+.++.+.+.++.|   
    4e+08 +-+---------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                             aim7.time.file_system_outputs                      
                                                                                
  1.8e+08 +-+---------------------------------------------------------------+   
          O OO O O OO O O OO O O OO O O OO O OO O O OO O O OO O O O         |   
  1.7e+08 +-+                                                               |   
  1.6e+08 +-+                                                               |   
          |                                                                 |   
  1.5e+08 +-+                                                               |   
          |                                                                 |   
  1.4e+08 +-+                                                               |   
          |                                                                 |   
  1.3e+08 +-+                                                               |   
  1.2e+08 +-+                                                               |   
          |                                                                 |   
  1.1e+08 +-+                                                               |   
          |.++.+.+.++.+.+.++.+.+.++.+.+.++.+.++.+.+.++.+.+.++.+.+.++.+.+.++.|   
    1e+08 +-+---------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.4.0-rc1-00111-gfe1897eaa6646" of type "text/plain" (200619 bytes)

View attachment "job-script" of type "text/plain" (8066 bytes)

View attachment "job.yaml" of type "text/plain" (5696 bytes)

View attachment "reproduce" of type "text/plain" (1043 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ