lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Message-ID: <87h9hxm5rf.fsf@yhuang-dev.intel.com>
Date:	Fri, 29 Jan 2016 11:19:48 +0800
From:	kernel test robot <ying.huang@...ux.intel.com>
TO:	Willy Tarreau <w@....eu>
CC:	Al Viro <viro@...iv.linux.org.uk>
Subject: [lkp] [pipe] 759c01142a: -51.5% hackbench.throughput

FYI, we noticed the below changes on

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master
commit 759c01142a5d0f364a462346168a56de28a80f52 ("pipe: limit the per-user amount of pages allocated in pipes")


=========================================================================================
compiler/cpufreq_governor/ipc/kconfig/mode/nr_threads/rootfs/tbox_group/testcase:
  gcc-4.9/performance/pipe/x86_64-rhel/threads/1600%/debian-x86_64-2015-02-07.cgz/lkp-wsx02/hackbench

commit: 
  558041d8d21b48287224dd0e32cf19563c77607c
  759c01142a5d0f364a462346168a56de28a80f52

558041d8d21b4828 759c01142a5d0f364a46234616 
---------------- -------------------------- 
         %stddev     %change         %stddev
             \          |                \  
    219967 ±  4%     -51.5%     106583 ±  1%  hackbench.throughput
 5.319e+08 ± 11%     -19.7%  4.273e+08 ±  0%  hackbench.time.involuntary_context_switches
   2422668 ±  9%     -36.4%    1540731 ±  2%  hackbench.time.minor_page_faults
      7127 ± 11%      +9.4%       7800 ±  0%  hackbench.time.percent_of_cpu_this_job_got
     42924 ±  9%     +11.9%      48049 ±  1%  hackbench.time.system_time
      1665 ± 11%     -49.3%     844.05 ±  1%  hackbench.time.user_time
    534.25 ± 55%    +298.5%       2129 ± 67%  numa-meminfo.node3.AnonHugePages
      8581 ± 57%     -48.4%       4427 ±  1%  uptime.idle
   5705345 ± 13%     +58.8%    9061490 ±  5%  softirqs.RCU
    279868 ±  8%     +20.4%     336916 ±  1%  softirqs.SCHED
     26.75 ± 23%    +245.8%      92.50 ± 39%  vmstat.procs.b
      9809 ± 13%     -64.7%       3463 ±  7%  vmstat.procs.r
    284320 ± 13%     -45.9%     153686 ±  1%  vmstat.system.in
    114.00 ± 10%     -50.9%      56.00 ±  0%  time.file_system_outputs
 5.319e+08 ± 11%     -19.7%  4.273e+08 ±  0%  time.involuntary_context_switches
   2422668 ±  9%     -36.4%    1540731 ±  2%  time.minor_page_faults
      1665 ± 11%     -49.3%     844.05 ±  1%  time.user_time
     89.86 ± 10%      +8.9%      97.84 ±  0%  turbostat.%Busy
      2270 ± 10%      +8.9%       2471 ±  0%  turbostat.Avg_MHz
      1.45 ±  8%     -52.6%       0.69 ±  9%  turbostat.CPU%c1
      0.17 ± 13%     -52.2%       0.08 ± 34%  turbostat.CPU%c3
      8.53 ±116%     -83.7%       1.39 ±  3%  turbostat.CPU%c6
  63583716 ± 16%     -98.1%    1183913 ± 53%  numa-numastat.node0.local_node
  63587769 ± 16%     -98.1%    1189061 ± 53%  numa-numastat.node0.numa_hit
  69989955 ±  6%     -96.8%    2239323 ±  8%  numa-numastat.node1.local_node
  69995282 ±  6%     -96.8%    2244474 ±  8%  numa-numastat.node1.numa_hit
  56620071 ± 22%     -97.3%    1520012 ± 32%  numa-numastat.node2.local_node
  56625301 ± 22%     -97.3%    1522594 ± 31%  numa-numastat.node2.numa_hit
  63753303 ± 22%     -96.1%    2508634 ± 14%  numa-numastat.node3.local_node
  63754612 ± 22%     -96.1%    2511213 ± 14%  numa-numastat.node3.numa_hit
  32233121 ± 15%     -97.4%     830659 ± 60%  numa-vmstat.node0.numa_hit
  32193958 ± 15%     -97.5%     790185 ± 63%  numa-vmstat.node0.numa_local
  35020615 ± 10%     -96.4%    1253289 ±  8%  numa-vmstat.node1.numa_hit
  34968563 ± 10%     -96.6%    1201078 ±  9%  numa-vmstat.node1.numa_local
  30394713 ± 14%     -97.2%     843134 ± 31%  numa-vmstat.node2.numa_hit
  30373607 ± 14%     -97.3%     824676 ± 31%  numa-vmstat.node2.numa_local
  32334081 ± 21%     -95.5%    1469250 ± 16%  numa-vmstat.node3.numa_hit
  32286373 ± 21%     -95.6%    1420300 ± 16%  numa-vmstat.node3.numa_local
 1.868e+08 ± 11%     -51.0%   91569694 ± 13%  cpuidle.C1-NHM.time
   1743049 ± 10%     -34.6%    1139878 ± 11%  cpuidle.C1-NHM.usage
 1.918e+08 ± 12%     -49.7%   96433830 ± 11%  cpuidle.C1E-NHM.time
   1243978 ± 16%     -40.7%     738168 ± 12%  cpuidle.C1E-NHM.usage
  70404853 ±  8%     -47.4%   37040591 ± 19%  cpuidle.C3-NHM.time
 4.723e+09 ±108%     -81.6%  8.672e+08 ±  3%  cpuidle.C6-NHM.time
    311666 ± 18%     -47.9%     162252 ± 15%  cpuidle.C6-NHM.usage
 2.456e+08 ± 18%     -53.2%   1.15e+08 ±  9%  cpuidle.POLL.time
    260576 ± 21%     -58.6%     107765 ± 17%  cpuidle.POLL.usage
    906770 ±  8%     -19.4%     730530 ±  5%  proc-vmstat.numa_hint_faults
  2.54e+08 ± 11%     -97.1%    7464740 ±  4%  proc-vmstat.numa_hit
 2.539e+08 ± 11%     -97.1%    7449281 ±  4%  proc-vmstat.numa_local
    662291 ± 10%     -33.4%     441016 ±  5%  proc-vmstat.numa_pages_migrated
   1203460 ±  6%     -16.7%    1002984 ±  5%  proc-vmstat.numa_pte_updates
   5716982 ± 17%     -97.2%     158482 ± 40%  proc-vmstat.pgalloc_dma32
 2.521e+08 ± 11%     -96.3%    9338972 ±  3%  proc-vmstat.pgalloc_normal
   3627983 ±  4%     -26.1%    2682860 ±  1%  proc-vmstat.pgfault
 2.576e+08 ± 11%     -96.4%    9352801 ±  3%  proc-vmstat.pgfree
    662291 ± 10%     -33.4%     441016 ±  5%  proc-vmstat.pgmigrate_success
     27845 ± 11%     -79.8%       5624 ±  1%  slabinfo.kmalloc-1024.active_objs
    879.00 ± 11%     -79.2%     183.00 ±  1%  slabinfo.kmalloc-1024.active_slabs
     28142 ± 11%     -79.1%       5879 ±  0%  slabinfo.kmalloc-1024.num_objs
    879.00 ± 11%     -79.2%     183.00 ±  1%  slabinfo.kmalloc-1024.num_slabs
     20288 ±  0%    +129.9%      46639 ±  0%  slabinfo.kmalloc-64.active_objs
    317.00 ±  0%    +131.4%     733.50 ±  0%  slabinfo.kmalloc-64.active_slabs
     20288 ±  0%    +131.6%      46980 ±  0%  slabinfo.kmalloc-64.num_objs
    317.00 ±  0%    +131.4%     733.50 ±  0%  slabinfo.kmalloc-64.num_slabs
    335.50 ±  9%     -44.8%     185.25 ±  4%  slabinfo.taskstats.active_objs
    335.50 ±  9%     -44.8%     185.25 ±  4%  slabinfo.taskstats.num_objs
     38.97 ± 29%     +51.7%      59.11 ±  7%  perf-profile.cycles-pp.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate
      3.89 ± 60%    +334.7%      16.93 ± 13%  perf-profile.cycles-pp.__vfs_read.vfs_read.sys_read.entry_SYSCALL_64_fastpath
      0.70 ± 30%   +1969.3%      14.49 ± 11%  perf-profile.cycles-pp.__wake_up_common.__wake_up_sync_key.pipe_read.__vfs_read.vfs_read
      0.71 ± 30%   +1971.0%      14.65 ± 11%  perf-profile.cycles-pp.__wake_up_sync_key.pipe_read.__vfs_read.vfs_read.sys_read
      0.69 ± 84%     -92.3%       0.05 ± 63%  perf-profile.cycles-pp._raw_spin_lock.__schedule.schedule.exit_to_usermode_loop.syscall_return_slowpath
     34.58 ± 33%     +57.1%      54.33 ±  7%  perf-profile.cycles-pp._raw_spin_lock_irqsave.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.activate_task
     39.61 ± 28%     +53.7%      60.89 ±  7%  perf-profile.cycles-pp.activate_task.ttwu_do_activate.try_to_wake_up.default_wake_function.autoremove_wake_function
      0.70 ± 30%   +1967.9%      14.47 ± 11%  perf-profile.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_sync_key.pipe_read.__vfs_read
      0.70 ± 30%   +1965.0%      14.46 ± 11%  perf-profile.cycles-pp.default_wake_function.autoremove_wake_function.__wake_up_common.__wake_up_sync_key.pipe_read
     38.99 ± 28%     +54.5%      60.25 ±  7%  perf-profile.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up
     39.45 ± 28%     +53.9%      60.72 ±  7%  perf-profile.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up.default_wake_function
     57.97 ± 20%     +40.4%      81.35 ±  8%  perf-profile.cycles-pp.entry_SYSCALL_64_fastpath
      1.31 ± 60%     -92.0%       0.10 ± 67%  perf-profile.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.__schedule.schedule.exit_to_usermode_loop
      2.54 ± 68%     -81.8%       0.46 ± 60%  perf-profile.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.try_to_wake_up.default_wake_function.autoremove_wake_function
     34.38 ± 33%     +55.7%      53.54 ±  6%  perf-profile.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.__account_scheduler_latency.enqueue_entity.enqueue_task_fair
      3.67 ± 65%    +354.5%      16.69 ± 11%  perf-profile.cycles-pp.pipe_read.__vfs_read.vfs_read.sys_read.entry_SYSCALL_64_fastpath
      5.22 ± 42%    +250.6%      18.30 ± 21%  perf-profile.cycles-pp.sys_read.entry_SYSCALL_64_fastpath
     39.95 ± 28%     +53.0%      61.10 ±  7%  perf-profile.cycles-pp.ttwu_do_activate.constprop.86.try_to_wake_up.default_wake_function.autoremove_wake_function.__wake_up_common
      4.70 ± 48%    +282.5%      17.96 ± 19%  perf-profile.cycles-pp.vfs_read.sys_read.entry_SYSCALL_64_fastpath
  16323355 ± 13%    -100.0%       0.00 ± -1%  latency_stats.avg.nfs_wait_on_request.nfs_updatepage.nfs_write_end.generic_perform_write.__generic_file_write_iter.generic_file_write_iter.nfs_file_write.__vfs_write.vfs_write.SyS_write.entry_SYSCALL_64_fastpath
   1702613 ± 26%    +288.1%    6607185 ± 61%  latency_stats.avg.pipe_read.__vfs_read.vfs_read.SyS_read.entry_SYSCALL_64_fastpath
     31.67 ± 74%  +63999.7%      20298 ±149%  latency_stats.avg.stop_two_cpus.migrate_swap.task_numa_migrate.numa_migrate_preferred.task_numa_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
   3607008 ±163%    -100.0%     384.25 ± 14%  latency_stats.avg.wait_on_page_bit.__filemap_fdatawait_range.filemap_fdatawait_range.filemap_write_and_wait_range.nfs4_file_fsync.[nfsv4].vfs_fsync_range.vfs_fsync.nfs4_file_flush.[nfsv4].filp_close.do_dup2.SyS_dup2.entry_SYSCALL_64_fastpath
   1042512 ± 15%    +145.7%    2561649 ± 35%  latency_stats.avg.wait_on_page_bit.__migration_entry_wait.migration_entry_wait.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
    384229 ±  2%     -76.9%      88748 ± 24%  latency_stats.hits.call_rwsem_down_read_failed.do_exit.SyS_exit.entry_SYSCALL_64_fastpath
  22676827 ± 14%    +762.3%  1.955e+08 ±  0%  latency_stats.hits.pipe_wait.pipe_write.__vfs_write.vfs_write.SyS_write.entry_SYSCALL_64_fastpath
    108077 ± 11%     -96.9%       3361 ±  8%  latency_stats.hits.pipe_write.__vfs_write.vfs_write.SyS_write.entry_SYSCALL_64_fastpath
   9791700 ± 80%    +173.5%   26782186 ± 72%  latency_stats.max.call_rwsem_down_read_failed.__do_page_fault.do_page_fault.page_fault.copy_page_from_iter.pipe_write.__vfs_write.vfs_write.SyS_write.entry_SYSCALL_64_fastpath
   8696153 ± 89%    +196.7%   25800599 ± 75%  latency_stats.max.call_rwsem_down_read_failed.__do_page_fault.do_page_fault.page_fault.copy_page_to_iter.pipe_read.__vfs_read.vfs_read.SyS_read.entry_SYSCALL_64_fastpath
     13849 ±173%     -64.1%       4975 ±102%  latency_stats.max.call_rwsem_down_write_failed.unlink_file_vma.free_pgtables.exit_mmap.mmput.flush_old_exec.load_elf_binary.search_binary_handler.do_execveat_common.SyS_execve.return_from_execve
  42431494 ± 56%    -100.0%       0.00 ± -1%  latency_stats.max.nfs_wait_on_request.nfs_updatepage.nfs_write_end.generic_perform_write.__generic_file_write_iter.generic_file_write_iter.nfs_file_write.__vfs_write.vfs_write.SyS_write.entry_SYSCALL_64_fastpath
  11940618 ± 27%    +124.4%   26793707 ± 72%  latency_stats.max.pipe_wait.pipe_write.__vfs_write.vfs_write.SyS_write.entry_SYSCALL_64_fastpath
     31.67 ± 74%  +73210.5%      23215 ±124%  latency_stats.max.stop_two_cpus.migrate_swap.task_numa_migrate.numa_migrate_preferred.task_numa_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
  27276522 ±155%    -100.0%     473.50 ± 18%  latency_stats.max.wait_on_page_bit.__filemap_fdatawait_range.filemap_fdatawait_range.filemap_write_and_wait_range.nfs4_file_fsync.[nfsv4].vfs_fsync_range.vfs_fsync.nfs4_file_flush.[nfsv4].filp_close.do_dup2.SyS_dup2.entry_SYSCALL_64_fastpath
  86431834 ±106%    +708.2%  6.985e+08 ± 40%  latency_stats.sum.call_rwsem_down_read_failed.__do_page_fault.do_page_fault.page_fault.copy_page_to_iter.pipe_read.__vfs_read.vfs_read.SyS_read.entry_SYSCALL_64_fastpath
     14619 ±173%     -65.9%       4980 ±102%  latency_stats.sum.call_rwsem_down_write_failed.unlink_file_vma.free_pgtables.exit_mmap.mmput.flush_old_exec.load_elf_binary.search_binary_handler.do_execveat_common.SyS_execve.return_from_execve
  44595961 ± 58%    -100.0%       0.00 ± -1%  latency_stats.sum.nfs_wait_on_request.nfs_updatepage.nfs_write_end.generic_perform_write.__generic_file_write_iter.generic_file_write_iter.nfs_file_write.__vfs_write.vfs_write.SyS_write.entry_SYSCALL_64_fastpath
 4.608e+08 ± 37%    +202.9%  1.396e+09 ± 59%  latency_stats.sum.pipe_read.__vfs_read.vfs_read.SyS_read.entry_SYSCALL_64_fastpath
 2.164e+10 ± 13%    +960.4%  2.294e+11 ±  5%  latency_stats.sum.pipe_wait.pipe_write.__vfs_write.vfs_write.SyS_write.entry_SYSCALL_64_fastpath
 1.117e+11 ± 16%     -97.9%  2.338e+09 ± 29%  latency_stats.sum.pipe_write.__vfs_write.vfs_write.SyS_write.entry_SYSCALL_64_fastpath
     35826 ± 54%     -70.9%      10422 ±  6%  latency_stats.sum.rpc_wait_bit_killable.__rpc_execute.rpc_execute.rpc_run_task.nfs4_call_sync_sequence.[nfsv4]._nfs4_proc_access.[nfsv4].nfs4_proc_access.[nfsv4].nfs_do_access.nfs_permission.__inode_permission.inode_permission.link_path_walk
     31.67 ± 74%  +1.3e+05%      39883 ± 89%  latency_stats.sum.stop_two_cpus.migrate_swap.task_numa_migrate.numa_migrate_preferred.task_numa_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
  40133821 ±161%    -100.0%       2693 ± 14%  latency_stats.sum.wait_on_page_bit.__filemap_fdatawait_range.filemap_fdatawait_range.filemap_write_and_wait_range.nfs4_file_fsync.[nfsv4].vfs_fsync_range.vfs_fsync.nfs4_file_flush.[nfsv4].filp_close.do_dup2.SyS_dup2.entry_SYSCALL_64_fastpath
 9.857e+09 ± 19%    +462.0%  5.539e+10 ± 40%  latency_stats.sum.wait_on_page_bit.__migration_entry_wait.migration_entry_wait.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
      7.50 ± 54%     +73.3%      13.00 ±  9%  sched_debug.cfs_rq:/.load.17
     12.00 ± 17%     -35.4%       7.75 ± 14%  sched_debug.cfs_rq:/.load.22
      5.50 ± 41%    +140.9%      13.25 ± 33%  sched_debug.cfs_rq:/.load.23
      6.25 ± 42%    +100.0%      12.50 ± 23%  sched_debug.cfs_rq:/.load.25
      6.00 ± 51%    +145.8%      14.75 ± 28%  sched_debug.cfs_rq:/.load.29
      5.75 ± 56%    +121.7%      12.75 ± 29%  sched_debug.cfs_rq:/.load.35
      7.50 ± 74%     +86.7%      14.00 ± 44%  sched_debug.cfs_rq:/.load.43
     11.75 ± 20%     -34.0%       7.75 ± 34%  sched_debug.cfs_rq:/.load.46
      5.75 ± 39%    +113.0%      12.25 ± 18%  sched_debug.cfs_rq:/.load.53
     12.75 ± 12%     -31.4%       8.75 ±  9%  sched_debug.cfs_rq:/.load.54
     13.75 ± 31%     -40.0%       8.25 ± 30%  sched_debug.cfs_rq:/.load.58
     27.00 ± 49%     -58.3%      11.25 ± 20%  sched_debug.cfs_rq:/.load_avg.24
     28.25 ± 58%     -61.9%      10.75 ± 20%  sched_debug.cfs_rq:/.load_avg.26
     22.25 ± 51%     -51.7%      10.75 ± 13%  sched_debug.cfs_rq:/.load_avg.28
     20.75 ± 48%     -53.0%       9.75 ± 15%  sched_debug.cfs_rq:/.load_avg.62
    182.64 ± 15%     -41.5%     106.77 ± 18%  sched_debug.cfs_rq:/.load_avg.max
      1.14 ± 30%     +65.7%       1.90 ± 18%  sched_debug.cfs_rq:/.load_avg.min
     30.53 ±  7%     -39.5%      18.46 ± 20%  sched_debug.cfs_rq:/.load_avg.stddev
  61751943 ±  3%     -29.5%   43522729 ± 13%  sched_debug.cfs_rq:/.min_vruntime.1
  63695496 ±  3%     -25.5%   47470305 ± 13%  sched_debug.cfs_rq:/.min_vruntime.13
  64969515 ±  3%     -27.9%   46847510 ± 14%  sched_debug.cfs_rq:/.min_vruntime.17
  65957991 ±  1%     -27.8%   47648514 ± 13%  sched_debug.cfs_rq:/.min_vruntime.21
  64485085 ±  3%     -26.1%   47642410 ± 13%  sched_debug.cfs_rq:/.min_vruntime.25
  64721403 ±  3%     -28.9%   45992467 ± 16%  sched_debug.cfs_rq:/.min_vruntime.29
  65138437 ±  4%     -29.3%   46039876 ± 12%  sched_debug.cfs_rq:/.min_vruntime.33
  66399877 ±  4%     -26.6%   48722623 ± 12%  sched_debug.cfs_rq:/.min_vruntime.37
  67958189 ±  4%     -29.4%   47968815 ± 16%  sched_debug.cfs_rq:/.min_vruntime.41
  67595662 ±  6%     -27.4%   49096370 ± 13%  sched_debug.cfs_rq:/.min_vruntime.45
  67690444 ±  3%     -27.6%   49000109 ± 12%  sched_debug.cfs_rq:/.min_vruntime.49
  64098981 ±  3%     -26.2%   47277630 ± 14%  sched_debug.cfs_rq:/.min_vruntime.5
  67783223 ±  3%     -27.4%   49216714 ± 12%  sched_debug.cfs_rq:/.min_vruntime.53
  66220648 ±  3%     -25.3%   49481358 ± 11%  sched_debug.cfs_rq:/.min_vruntime.57
  68140688 ±  4%     -30.7%   47207825 ± 16%  sched_debug.cfs_rq:/.min_vruntime.61
  65498493 ±  5%     -25.8%   48590023 ± 13%  sched_debug.cfs_rq:/.min_vruntime.65
  65049958 ±  4%     -28.2%   46691564 ± 18%  sched_debug.cfs_rq:/.min_vruntime.69
  68046848 ±  3%     -26.8%   49835973 ± 13%  sched_debug.cfs_rq:/.min_vruntime.73
  66382113 ±  4%     -26.9%   48557976 ± 18%  sched_debug.cfs_rq:/.min_vruntime.77
  65055617 ±  3%     -28.7%   46408287 ± 15%  sched_debug.cfs_rq:/.min_vruntime.9
  64257203 ± 11%     -19.4%   51816609 ±  3%  sched_debug.cfs_rq:/.min_vruntime.avg
  81981181 ±  6%     -14.9%   69761399 ±  5%  sched_debug.cfs_rq:/.min_vruntime.max
      7.25 ± 39%    +348.3%      32.50 ± 68%  sched_debug.cfs_rq:/.nr_spread_over.0
      6.75 ± 28%    +622.2%      48.75 ± 58%  sched_debug.cfs_rq:/.nr_spread_over.1
      3.50 ± 51%   +1342.9%      50.50 ± 45%  sched_debug.cfs_rq:/.nr_spread_over.10
      4.25 ±100%   +4358.8%     189.50 ±143%  sched_debug.cfs_rq:/.nr_spread_over.13
      7.25 ± 86%   +2000.0%     152.25 ±109%  sched_debug.cfs_rq:/.nr_spread_over.14
      9.00 ± 39%    +933.3%      93.00 ±105%  sched_debug.cfs_rq:/.nr_spread_over.17
      6.50 ± 77%    +826.9%      60.25 ± 67%  sched_debug.cfs_rq:/.nr_spread_over.18
      9.75 ± 67%    +220.5%      31.25 ± 79%  sched_debug.cfs_rq:/.nr_spread_over.20
      9.25 ± 67%    +281.1%      35.25 ± 65%  sched_debug.cfs_rq:/.nr_spread_over.21
      4.25 ± 45%   +1517.6%      68.75 ± 53%  sched_debug.cfs_rq:/.nr_spread_over.22
      5.50 ± 68%    +581.8%      37.50 ± 92%  sched_debug.cfs_rq:/.nr_spread_over.24
      5.50 ± 30%    +754.5%      47.00 ± 73%  sched_debug.cfs_rq:/.nr_spread_over.25
      6.25 ± 23%   +1660.0%     110.00 ± 39%  sched_debug.cfs_rq:/.nr_spread_over.26
      8.00 ± 12%    +343.8%      35.50 ± 39%  sched_debug.cfs_rq:/.nr_spread_over.29
      3.00 ± 78%    +600.0%      21.00 ± 25%  sched_debug.cfs_rq:/.nr_spread_over.32
      5.75 ± 93%    +856.5%      55.00 ± 67%  sched_debug.cfs_rq:/.nr_spread_over.36
      4.25 ± 65%    +670.6%      32.75 ± 30%  sched_debug.cfs_rq:/.nr_spread_over.37
      4.75 ± 47%   +1300.0%      66.50 ± 63%  sched_debug.cfs_rq:/.nr_spread_over.38
      6.25 ± 81%    +812.0%      57.00 ± 43%  sched_debug.cfs_rq:/.nr_spread_over.4
      8.25 ± 89%    +169.7%      22.25 ± 69%  sched_debug.cfs_rq:/.nr_spread_over.41
      3.00 ±102%   +5608.3%     171.25 ±140%  sched_debug.cfs_rq:/.nr_spread_over.42
      5.25 ± 47%    +566.7%      35.00 ± 27%  sched_debug.cfs_rq:/.nr_spread_over.44
      4.00 ± 46%    +662.5%      30.50 ± 28%  sched_debug.cfs_rq:/.nr_spread_over.45
      3.25 ±121%   +2761.5%      93.00 ± 87%  sched_debug.cfs_rq:/.nr_spread_over.46
      4.25 ± 58%    +758.8%      36.50 ±104%  sched_debug.cfs_rq:/.nr_spread_over.47
      6.50 ± 60%    +323.1%      27.50 ± 52%  sched_debug.cfs_rq:/.nr_spread_over.48
      7.50 ± 38%    +800.0%      67.50 ± 81%  sched_debug.cfs_rq:/.nr_spread_over.49
      6.75 ± 21%    +303.7%      27.25 ± 43%  sched_debug.cfs_rq:/.nr_spread_over.5
      2.75 ± 30%   +1736.4%      50.50 ± 56%  sched_debug.cfs_rq:/.nr_spread_over.50
      3.75 ± 66%   +1220.0%      49.50 ±117%  sched_debug.cfs_rq:/.nr_spread_over.52
      2.75 ± 64%    +854.5%      26.25 ± 79%  sched_debug.cfs_rq:/.nr_spread_over.53
      2.75 ± 97%   +3218.2%      91.25 ± 57%  sched_debug.cfs_rq:/.nr_spread_over.54
      3.00 ± 40%    +900.0%      30.00 ± 66%  sched_debug.cfs_rq:/.nr_spread_over.56
      2.25 ± 57%   +1700.0%      40.50 ± 77%  sched_debug.cfs_rq:/.nr_spread_over.57
      3.50 ± 58%   +2771.4%     100.50 ± 69%  sched_debug.cfs_rq:/.nr_spread_over.58
      4.00 ±107%    +893.8%      39.75 ± 82%  sched_debug.cfs_rq:/.nr_spread_over.59
      6.00 ± 72%   +1283.3%      83.00 ± 36%  sched_debug.cfs_rq:/.nr_spread_over.6
      6.50 ± 77%    +384.6%      31.50 ± 78%  sched_debug.cfs_rq:/.nr_spread_over.61
      3.50 ± 51%   +1571.4%      58.50 ± 73%  sched_debug.cfs_rq:/.nr_spread_over.62
      6.25 ± 52%    +356.0%      28.50 ± 45%  sched_debug.cfs_rq:/.nr_spread_over.64
      7.75 ± 84%    +106.5%      16.00 ± 41%  sched_debug.cfs_rq:/.nr_spread_over.65
      6.50 ± 72%    +319.2%      27.25 ± 83%  sched_debug.cfs_rq:/.nr_spread_over.68
      8.50 ± 59%    +329.4%      36.50 ± 57%  sched_debug.cfs_rq:/.nr_spread_over.69
      5.75 ± 56%    +308.7%      23.50 ± 60%  sched_debug.cfs_rq:/.nr_spread_over.72
      3.50 ± 65%    +400.0%      17.50 ± 52%  sched_debug.cfs_rq:/.nr_spread_over.73
      3.25 ± 25%    +876.9%      31.75 ± 68%  sched_debug.cfs_rq:/.nr_spread_over.75
      4.00 ± 63%  +23906.2%     960.25 ±169%  sched_debug.cfs_rq:/.nr_spread_over.76
      5.00 ± 70%    +720.0%      41.00 ± 54%  sched_debug.cfs_rq:/.nr_spread_over.77
      3.50 ± 47%    +600.0%      24.50 ± 73%  sched_debug.cfs_rq:/.nr_spread_over.8
      6.75 ± 48%    +370.4%      31.75 ± 83%  sched_debug.cfs_rq:/.nr_spread_over.9
     19.08 ± 14%    +232.7%      63.48 ± 32%  sched_debug.cfs_rq:/.nr_spread_over.avg
     13.50 ± 21%     -38.9%       8.25 ± 21%  sched_debug.cfs_rq:/.runnable_load_avg.62
     17.34 ± 33%     -48.7%       8.90 ± 22%  sched_debug.cfs_rq:/.runnable_load_avg.stddev
   1650786 ± 96%    +215.7%    5211288 ± 32%  sched_debug.cfs_rq:/.spread0.24
   3735888 ± 25%     -66.2%    1261722 ± 76%  sched_debug.cfs_rq:/.spread0.32
   6947361 ±181%    -153.7%   -3730893 ±-158%  sched_debug.cfs_rq:/.spread0.41
    822224 ±  6%    +262.2%    2978378 ± 55%  sched_debug.cpu.avg_idle.11
    899425 ± 22%    +115.1%    1934526 ± 63%  sched_debug.cpu.avg_idle.27
    902557 ± 11%    +161.2%    2357181 ± 71%  sched_debug.cpu.avg_idle.50
    855811 ±  2%     +90.8%    1632603 ± 74%  sched_debug.cpu.avg_idle.51
    968895 ±  7%    +160.6%    2524849 ± 36%  sched_debug.cpu.avg_idle.52
   1175701 ± 48%    +111.3%    2484589 ± 71%  sched_debug.cpu.avg_idle.69
      5.00 ± 44%    +120.0%      11.00 ± 33%  sched_debug.cpu.cpu_load[0].23
     13.50 ± 21%     -38.9%       8.25 ± 21%  sched_debug.cpu.cpu_load[0].62
      0.94 ± 25%     +74.1%       1.64 ± 20%  sched_debug.cpu.cpu_load[0].min
      5.00 ± 44%    +120.0%      11.00 ± 33%  sched_debug.cpu.cpu_load[1].23
      5.75 ± 31%     +91.3%      11.00 ± 23%  sched_debug.cpu.cpu_load[1].45
     13.50 ± 21%     -38.9%       8.25 ± 21%  sched_debug.cpu.cpu_load[1].62
      0.94 ± 25%     +74.1%       1.64 ± 20%  sched_debug.cpu.cpu_load[1].min
     17.26 ± 33%     -40.2%      10.33 ± 41%  sched_debug.cpu.cpu_load[1].stddev
      5.50 ± 32%    +100.0%      11.00 ± 23%  sched_debug.cpu.cpu_load[2].45
     13.50 ± 21%     -38.9%       8.25 ± 21%  sched_debug.cpu.cpu_load[2].62
     17.07 ± 33%     -43.4%       9.66 ± 32%  sched_debug.cpu.cpu_load[2].stddev
      5.00 ± 42%    +120.0%      11.00 ± 23%  sched_debug.cpu.cpu_load[3].45
     13.75 ± 18%     -40.0%       8.25 ± 21%  sched_debug.cpu.cpu_load[3].62
     16.85 ± 31%     -42.2%       9.75 ± 32%  sched_debug.cpu.cpu_load[3].stddev
      4.75 ± 50%    +131.6%      11.00 ± 23%  sched_debug.cpu.cpu_load[4].45
     13.75 ± 18%     -40.0%       8.25 ± 21%  sched_debug.cpu.cpu_load[4].62
     13.50 ± 19%     -37.0%       8.50 ± 24%  sched_debug.cpu.cpu_load[4].78
     16.46 ± 29%     -40.2%       9.85 ± 33%  sched_debug.cpu.cpu_load[4].stddev
     45774 ± 14%     -26.5%      33656 ± 21%  sched_debug.cpu.curr->pid.67
      7.50 ± 52%     +76.7%      13.25 ±  8%  sched_debug.cpu.load.17
     12.25 ± 25%     -36.7%       7.75 ± 14%  sched_debug.cpu.load.20
     11.75 ± 18%     -36.2%       7.50 ± 14%  sched_debug.cpu.load.22
      5.75 ± 37%    +126.1%      13.00 ± 32%  sched_debug.cpu.load.23
      6.00 ± 42%    +104.2%      12.25 ± 26%  sched_debug.cpu.load.25
      5.75 ± 56%    +126.1%      13.00 ± 30%  sched_debug.cpu.load.35
     12.00 ± 17%     -35.4%       7.75 ± 34%  sched_debug.cpu.load.46
      5.75 ± 39%    +117.4%      12.50 ± 18%  sched_debug.cpu.load.53
     12.50 ± 12%     -30.0%       8.75 ±  9%  sched_debug.cpu.load.54
    505539 ±  1%    +644.1%    3761938 ± 74%  sched_debug.cpu.max_idle_balance_cost.11
    719417 ± 52%    +395.3%    3563254 ± 71%  sched_debug.cpu.max_idle_balance_cost.37
    744737 ± 44%    +202.3%    2251473 ± 75%  sched_debug.cpu.max_idle_balance_cost.40
    534942 ±  7%    +266.8%    1962058 ± 46%  sched_debug.cpu.max_idle_balance_cost.52
   2113103 ± 65%     -68.9%     656815 ± 41%  sched_debug.cpu.max_idle_balance_cost.65
    630785 ± 17%     -19.5%     507892 ±  1%  sched_debug.cpu.max_idle_balance_cost.7
    148.50 ± 45%     -83.5%      24.50 ± 19%  sched_debug.cpu.nr_running.0
    165.00 ± 45%     -85.8%      23.50 ± 38%  sched_debug.cpu.nr_running.10
     75.75 ± 29%     -71.6%      21.50 ± 25%  sched_debug.cpu.nr_running.11
    124.50 ± 85%     -82.9%      21.25 ± 19%  sched_debug.cpu.nr_running.12
    141.75 ± 29%     -84.0%      22.75 ± 26%  sched_debug.cpu.nr_running.14
    102.25 ± 78%     -71.1%      29.50 ± 33%  sched_debug.cpu.nr_running.15
    125.25 ± 83%     -82.6%      21.75 ± 21%  sched_debug.cpu.nr_running.16
    157.25 ± 39%     -85.2%      23.25 ± 34%  sched_debug.cpu.nr_running.18
    159.00 ± 66%     -83.6%      26.00 ± 27%  sched_debug.cpu.nr_running.19
    148.00 ± 24%     -80.9%      28.25 ± 29%  sched_debug.cpu.nr_running.2
    117.75 ± 49%     -81.7%      21.50 ± 28%  sched_debug.cpu.nr_running.20
    135.00 ± 32%     -86.1%      18.75 ± 27%  sched_debug.cpu.nr_running.22
     71.25 ± 27%     -55.1%      32.00 ± 27%  sched_debug.cpu.nr_running.23
    243.50 ± 71%     -89.4%      25.75 ±  8%  sched_debug.cpu.nr_running.24
    186.00 ± 51%     -89.1%      20.25 ± 24%  sched_debug.cpu.nr_running.26
     95.75 ± 59%     -73.4%      25.50 ± 29%  sched_debug.cpu.nr_running.27
    239.25 ± 99%     -92.1%      19.00 ± 15%  sched_debug.cpu.nr_running.28
     98.25 ± 54%     -66.9%      32.50 ± 22%  sched_debug.cpu.nr_running.3
    172.50 ± 44%     -87.5%      21.50 ± 43%  sched_debug.cpu.nr_running.30
     66.50 ± 25%     -63.9%      24.00 ± 25%  sched_debug.cpu.nr_running.31
    133.25 ± 48%     -82.2%      23.75 ± 18%  sched_debug.cpu.nr_running.32
    139.75 ± 37%     -82.3%      24.75 ± 27%  sched_debug.cpu.nr_running.34
     86.25 ± 75%     -75.4%      21.25 ± 23%  sched_debug.cpu.nr_running.36
     62.75 ± 31%     -58.2%      26.25 ± 30%  sched_debug.cpu.nr_running.39
    100.00 ± 72%     -79.0%      21.00 ± 21%  sched_debug.cpu.nr_running.4
     92.00 ± 64%     -78.0%      20.25 ± 15%  sched_debug.cpu.nr_running.40
    155.50 ± 46%     -83.8%      25.25 ± 45%  sched_debug.cpu.nr_running.42
     62.25 ± 39%     -55.8%      27.50 ± 30%  sched_debug.cpu.nr_running.43
    113.50 ±102%     -79.5%      23.25 ± 24%  sched_debug.cpu.nr_running.44
    140.00 ± 29%     -86.4%      19.00 ± 31%  sched_debug.cpu.nr_running.46
    311.50 ±114%     -91.3%      27.25 ± 20%  sched_debug.cpu.nr_running.47
    112.75 ± 91%     -81.2%      21.25 ±  5%  sched_debug.cpu.nr_running.48
    138.50 ± 34%     -84.7%      21.25 ± 31%  sched_debug.cpu.nr_running.50
     88.75 ± 67%     -72.7%      24.25 ± 33%  sched_debug.cpu.nr_running.51
    105.50 ± 51%     -79.9%      21.25 ± 14%  sched_debug.cpu.nr_running.52
    129.50 ± 29%     -84.6%      20.00 ± 21%  sched_debug.cpu.nr_running.54
    122.25 ± 51%     -80.2%      24.25 ± 12%  sched_debug.cpu.nr_running.56
    163.25 ± 45%     -87.7%      20.00 ± 36%  sched_debug.cpu.nr_running.58
     92.25 ± 66%     -71.5%      26.25 ± 13%  sched_debug.cpu.nr_running.59
    230.75 ±101%     -91.9%      18.75 ± 23%  sched_debug.cpu.nr_running.60
    144.75 ± 42%     -82.6%      25.25 ± 41%  sched_debug.cpu.nr_running.62
     58.25 ± 32%     -55.4%      26.00 ± 31%  sched_debug.cpu.nr_running.63
    122.25 ± 47%     -81.6%      22.50 ±  9%  sched_debug.cpu.nr_running.64
    150.00 ± 38%     -74.5%      38.25 ± 76%  sched_debug.cpu.nr_running.66
     89.25 ± 62%     -75.1%      22.25 ± 43%  sched_debug.cpu.nr_running.67
    116.00 ± 60%     -78.7%      24.75 ± 10%  sched_debug.cpu.nr_running.68
     88.25 ± 22%     -67.4%      28.75 ± 25%  sched_debug.cpu.nr_running.7
    134.25 ± 40%     -81.8%      24.50 ± 40%  sched_debug.cpu.nr_running.70
     69.00 ± 38%     -59.8%      27.75 ± 22%  sched_debug.cpu.nr_running.71
    119.75 ± 42%     -80.4%      23.50 ± 16%  sched_debug.cpu.nr_running.72
    138.50 ± 38%     -74.5%      35.25 ± 93%  sched_debug.cpu.nr_running.74
     86.50 ± 61%     -73.7%      22.75 ± 32%  sched_debug.cpu.nr_running.75
    115.50 ± 62%     -81.8%      21.00 ± 10%  sched_debug.cpu.nr_running.76
    127.00 ± 35%     -83.5%      21.00 ± 37%  sched_debug.cpu.nr_running.78
     59.25 ± 40%     -66.2%      20.00 ± 44%  sched_debug.cpu.nr_running.79
     97.50 ± 54%     -74.1%      25.25 ± 10%  sched_debug.cpu.nr_running.8
    106.72 ± 14%     -69.9%      32.15 ± 23%  sched_debug.cpu.nr_running.avg
    975.12 ± 47%     -72.2%     271.24 ± 72%  sched_debug.cpu.nr_running.max
      0.83 ± 26%     +65.9%       1.38 ± 26%  sched_debug.cpu.nr_running.min
    189.39 ± 34%     -74.1%      49.14 ± 62%  sched_debug.cpu.nr_running.stddev
   8049365 ±  5%      -9.5%    7281434 ±  5%  sched_debug.cpu.nr_switches.21
   8222072 ±  4%     -11.3%    7293398 ±  2%  sched_debug.cpu.nr_switches.25
   8355303 ±  5%     -13.3%    7241351 ±  4%  sched_debug.cpu.nr_switches.29
   8377252 ±  5%     -12.2%    7358538 ±  3%  sched_debug.cpu.nr_switches.33
   8646522 ±  6%     -12.8%    7543084 ±  3%  sched_debug.cpu.nr_switches.37
   8680679 ±  6%     -13.5%    7512566 ±  3%  sched_debug.cpu.nr_switches.41
   8862414 ±  6%     -15.0%    7533561 ±  4%  sched_debug.cpu.nr_switches.45
   8513418 ±  5%     -12.4%    7453546 ±  3%  sched_debug.cpu.nr_switches.49
   8554847 ±  6%     -12.6%    7480415 ±  4%  sched_debug.cpu.nr_switches.53
   8818690 ±  3%     -15.0%    7492325 ±  2%  sched_debug.cpu.nr_switches.57
   8446431 ±  5%     -12.2%    7416662 ±  3%  sched_debug.cpu.nr_switches.61
   8424264 ±  6%     -11.2%    7483465 ±  3%  sched_debug.cpu.nr_switches.65
   8874420 ±  6%     -14.6%    7581001 ±  4%  sched_debug.cpu.nr_switches.73
   8548326 ±  3%     -12.7%    7462215 ±  3%  sched_debug.cpu.nr_switches.77
   9638903 ±  5%     -17.4%    7963391 ±  4%  sched_debug.cpu.nr_switches.max
    896810 ± 21%     -68.9%     279227 ± 18%  sched_debug.cpu.nr_switches.stddev
    116.00 ± 29%    -115.7%     -18.25 ±-446%  sched_debug.cpu.nr_uninterruptible.10
    333.00 ± 85%    -104.9%     -16.25 ±-454%  sched_debug.cpu.nr_uninterruptible.14
    132.00 ± 65%     -76.7%      30.75 ±119%  sched_debug.cpu.nr_uninterruptible.16
    170.00 ± 57%    -100.9%      -1.50 ±-8373%  sched_debug.cpu.nr_uninterruptible.20
    201.00 ± 66%     -81.1%      38.00 ±318%  sched_debug.cpu.nr_uninterruptible.24
    277.00 ± 44%     -74.4%      71.00 ± 96%  sched_debug.cpu.nr_uninterruptible.25
    213.75 ± 60%     -76.8%      49.50 ± 63%  sched_debug.cpu.nr_uninterruptible.33
     80.25 ± 41%    -109.7%      -7.75 ±-722%  sched_debug.cpu.nr_uninterruptible.40
    350.00 ±100%    -107.5%     -26.25 ±-523%  sched_debug.cpu.nr_uninterruptible.46
    177.00 ± 35%     -60.7%      69.50 ± 65%  sched_debug.cpu.nr_uninterruptible.56
    223.00 ± 55%     -58.6%      92.25 ± 84%  sched_debug.cpu.nr_uninterruptible.58
    260.50 ± 67%    -313.2%    -555.50 ±-124%  sched_debug.cpu.nr_uninterruptible.59
    290.25 ± 78%     -98.7%       3.75 ±246%  sched_debug.cpu.nr_uninterruptible.6
    139.00 ± 40%     -87.4%      17.50 ±226%  sched_debug.cpu.nr_uninterruptible.66
     94.25 ± 96%    -136.1%     -34.00 ±-331%  sched_debug.cpu.nr_uninterruptible.68
    134.50 ± 69%    -128.6%     -38.50 ±-288%  sched_debug.cpu.nr_uninterruptible.69
    160.25 ± 60%     -84.9%      24.25 ±377%  sched_debug.cpu.nr_uninterruptible.73
    224.00 ± 30%     -71.5%      63.75 ± 24%  sched_debug.cpu.nr_uninterruptible.78
    104.50 ±112%    -128.7%     -30.00 ±-179%  sched_debug.cpu.nr_uninterruptible.8
     93.00 ± 30%    -128.2%     -26.25 ±-149%  sched_debug.cpu.nr_uninterruptible.9
     68.65 ± 25%     -77.4%      15.53 ±150%  sched_debug.cpu.nr_uninterruptible.avg
   8050079 ±  5%      -9.5%    7282381 ±  5%  sched_debug.cpu.sched_count.21
   8223158 ±  4%     -11.3%    7293920 ±  2%  sched_debug.cpu.sched_count.25
   8355745 ±  5%     -13.3%    7241867 ±  4%  sched_debug.cpu.sched_count.29
   8377564 ±  5%     -12.2%    7358616 ±  3%  sched_debug.cpu.sched_count.33
   8647489 ±  6%     -12.8%    7543179 ±  3%  sched_debug.cpu.sched_count.37
   8680933 ±  6%     -13.5%    7512654 ±  3%  sched_debug.cpu.sched_count.41
   8862638 ±  6%     -15.0%    7533639 ±  4%  sched_debug.cpu.sched_count.45
   8513659 ±  5%     -12.5%    7453634 ±  3%  sched_debug.cpu.sched_count.49
   8555109 ±  6%     -12.6%    7480502 ±  4%  sched_debug.cpu.sched_count.53
   8818934 ±  3%     -15.0%    7492403 ±  2%  sched_debug.cpu.sched_count.57
   8446695 ±  5%     -12.2%    7416735 ±  3%  sched_debug.cpu.sched_count.61
   8424736 ±  6%     -11.2%    7483931 ±  3%  sched_debug.cpu.sched_count.65
   8874653 ±  6%     -14.6%    7581099 ±  4%  sched_debug.cpu.sched_count.73
   8548563 ±  3%     -12.7%    7462306 ±  3%  sched_debug.cpu.sched_count.77
   1277740 ± 13%     -28.3%     915765 ±  1%  sched_debug.cpu.sched_count.stddev
     23897 ± 33%     -52.1%      11437 ± 45%  sched_debug.cpu.sched_goidle.0
     24028 ± 11%     -55.3%      10738 ± 19%  sched_debug.cpu.sched_goidle.10
     23579 ± 16%     -50.5%      11679 ± 45%  sched_debug.cpu.sched_goidle.12
     25676 ± 27%     -55.2%      11511 ± 21%  sched_debug.cpu.sched_goidle.13
     29635 ± 45%     -54.6%      13447 ± 30%  sched_debug.cpu.sched_goidle.15
     24179 ± 31%     -45.1%      13266 ± 33%  sched_debug.cpu.sched_goidle.16
     19085 ± 16%     -50.5%       9456 ± 29%  sched_debug.cpu.sched_goidle.2
     26332 ± 24%     -51.3%      12826 ± 23%  sched_debug.cpu.sched_goidle.20
     19407 ± 23%     -26.8%      14211 ± 18%  sched_debug.cpu.sched_goidle.21
     23725 ± 10%     -60.5%       9362 ± 22%  sched_debug.cpu.sched_goidle.22
     20885 ±  8%     -30.4%      14541 ± 33%  sched_debug.cpu.sched_goidle.23
     26971 ± 35%     -61.1%      10498 ± 25%  sched_debug.cpu.sched_goidle.24
     21959 ± 16%     -41.4%      12865 ± 22%  sched_debug.cpu.sched_goidle.25
     21776 ± 12%     -46.1%      11742 ± 23%  sched_debug.cpu.sched_goidle.27
     21108 ± 27%     -56.6%       9163 ± 29%  sched_debug.cpu.sched_goidle.30
     18235 ± 46%     -57.1%       7826 ± 51%  sched_debug.cpu.sched_goidle.32
     18074 ± 14%     -44.3%      10059 ± 40%  sched_debug.cpu.sched_goidle.33
     15392 ± 23%     -37.3%       9657 ± 13%  sched_debug.cpu.sched_goidle.34
     21667 ± 26%     -59.5%       8783 ± 13%  sched_debug.cpu.sched_goidle.38
     19652 ± 10%     -42.0%      11404 ± 25%  sched_debug.cpu.sched_goidle.4
     16821 ± 13%     -57.7%       7113 ± 14%  sched_debug.cpu.sched_goidle.42
     17884 ± 27%     -62.7%       6664 ± 72%  sched_debug.cpu.sched_goidle.44
     22708 ± 46%     -63.9%       8192 ± 58%  sched_debug.cpu.sched_goidle.48
     19401 ± 26%     -42.1%      11228 ±  5%  sched_debug.cpu.sched_goidle.49
     16767 ± 19%     -44.3%       9332 ± 20%  sched_debug.cpu.sched_goidle.50
     15964 ± 27%     -48.2%       8269 ± 29%  sched_debug.cpu.sched_goidle.52
     14561 ± 17%     -31.7%       9951 ±  8%  sched_debug.cpu.sched_goidle.53
     16277 ± 22%     -67.9%       5232 ± 35%  sched_debug.cpu.sched_goidle.54
     23015 ± 35%     -51.2%      11236 ± 24%  sched_debug.cpu.sched_goidle.6
     15897 ± 14%     -48.0%       8264 ± 27%  sched_debug.cpu.sched_goidle.62
     22641 ± 16%     -46.0%      12217 ± 20%  sched_debug.cpu.sched_goidle.64
     23159 ± 27%     -47.9%      12060 ± 15%  sched_debug.cpu.sched_goidle.68
     20571 ± 10%     -30.7%      14246 ± 29%  sched_debug.cpu.sched_goidle.69
     21750 ± 25%     -54.6%       9880 ± 38%  sched_debug.cpu.sched_goidle.70
     13572 ± 12%     -35.5%       8757 ± 29%  sched_debug.cpu.sched_goidle.73
     15307 ± 20%     -36.8%       9677 ± 40%  sched_debug.cpu.sched_goidle.78
     17866 ±  7%     -33.2%      11942 ± 29%  sched_debug.cpu.sched_goidle.9
     18857 ± 12%     -40.4%      11240 ± 18%  sched_debug.cpu.sched_goidle.avg
     43849 ± 13%     -36.0%      28085 ± 10%  sched_debug.cpu.sched_goidle.max
      6115 ± 23%     -50.4%       3034 ± 27%  sched_debug.cpu.sched_goidle.min
      7313 ±  9%     -32.8%       4915 ±  8%  sched_debug.cpu.sched_goidle.stddev
   4341077 ±  9%     +14.6%    4976682 ±  4%  sched_debug.cpu.ttwu_count.0
    372672 ± 13%     -70.4%     110439 ±  8%  sched_debug.cpu.ttwu_count.stddev
   2955563 ±  9%     +53.1%    4523498 ±  4%  sched_debug.cpu.ttwu_local.0
   3131208 ±  7%     +43.3%    4486241 ±  5%  sched_debug.cpu.ttwu_local.1
   3001199 ± 15%     +44.8%    4346826 ±  4%  sched_debug.cpu.ttwu_local.10
   3166403 ±  7%     +35.4%    4288123 ±  7%  sched_debug.cpu.ttwu_local.11
   3100602 ± 13%     +38.3%    4288026 ±  4%  sched_debug.cpu.ttwu_local.12
   3391546 ±  5%     +30.8%    4435978 ±  5%  sched_debug.cpu.ttwu_local.13
   2959588 ± 16%     +44.0%    4261936 ±  4%  sched_debug.cpu.ttwu_local.14
   3267265 ±  9%     +32.4%    4326510 ±  7%  sched_debug.cpu.ttwu_local.15
   3137496 ± 18%     +38.5%    4345588 ±  5%  sched_debug.cpu.ttwu_local.16
   3356134 ±  7%     +31.1%    4399395 ±  4%  sched_debug.cpu.ttwu_local.17
   3108355 ± 17%     +36.7%    4248516 ±  4%  sched_debug.cpu.ttwu_local.18
   3303788 ± 10%     +31.6%    4346891 ±  6%  sched_debug.cpu.ttwu_local.19
   2790792 ± 16%     +60.4%    4476856 ±  4%  sched_debug.cpu.ttwu_local.2
   3208488 ± 16%     +36.0%    4363319 ±  5%  sched_debug.cpu.ttwu_local.20
   3323090 ±  6%     +33.4%    4432674 ±  8%  sched_debug.cpu.ttwu_local.21
   2920237 ± 17%     +50.4%    4391940 ±  3%  sched_debug.cpu.ttwu_local.22
   3133540 ±  6%     +38.6%    4342196 ± 10%  sched_debug.cpu.ttwu_local.23
   3082052 ± 15%     +40.8%    4340369 ±  5%  sched_debug.cpu.ttwu_local.24
   3410451 ±  2%     +30.2%    4440794 ±  5%  sched_debug.cpu.ttwu_local.25
   2965013 ± 15%     +44.9%    4295580 ±  5%  sched_debug.cpu.ttwu_local.26
   3140505 ±  7%     +40.7%    4420128 ±  7%  sched_debug.cpu.ttwu_local.27
   3107859 ± 16%     +40.0%    4352409 ±  5%  sched_debug.cpu.ttwu_local.28
   3558229 ±  8%     +23.2%    4383658 ±  5%  sched_debug.cpu.ttwu_local.29
   3056265 ±  8%     +48.4%    4534490 ±  5%  sched_debug.cpu.ttwu_local.3
   3064582 ± 16%     +41.9%    4348948 ±  2%  sched_debug.cpu.ttwu_local.30
   3209662 ±  6%     +36.0%    4366452 ±  6%  sched_debug.cpu.ttwu_local.31
   3525265 ± 14%     +29.9%    4578608 ±  2%  sched_debug.cpu.ttwu_local.32
   3649268 ±  5%     +25.8%    4592214 ±  5%  sched_debug.cpu.ttwu_local.33
   3181985 ± 18%     +44.3%    4590269 ±  4%  sched_debug.cpu.ttwu_local.34
   3503558 ±  6%     +34.7%    4719199 ±  5%  sched_debug.cpu.ttwu_local.35
   3510966 ± 13%     +24.9%    4384477 ±  4%  sched_debug.cpu.ttwu_local.36
   3801989 ±  7%     +19.7%    4550261 ±  5%  sched_debug.cpu.ttwu_local.37
   3197749 ± 14%     +34.7%    4306364 ±  0%  sched_debug.cpu.ttwu_local.38
   3452371 ±  6%     +27.1%    4387164 ±  7%  sched_debug.cpu.ttwu_local.39
   3106534 ± 14%     +38.5%    4301950 ±  3%  sched_debug.cpu.ttwu_local.4
   3410508 ± 14%     +27.4%    4345186 ±  5%  sched_debug.cpu.ttwu_local.40
   3854125 ±  6%     +18.4%    4562285 ±  6%  sched_debug.cpu.ttwu_local.41
   3210250 ± 14%     +38.7%    4453936 ±  3%  sched_debug.cpu.ttwu_local.42
   3393670 ±  7%     +30.2%    4419376 ±  6%  sched_debug.cpu.ttwu_local.43
   3504231 ± 13%     +24.3%    4356255 ±  4%  sched_debug.cpu.ttwu_local.44
   3828003 ±  6%     +18.7%    4542494 ±  6%  sched_debug.cpu.ttwu_local.45
   3285330 ± 14%     +31.1%    4305974 ±  3%  sched_debug.cpu.ttwu_local.46
   3457275 ± 11%     +28.7%    4449353 ±  6%  sched_debug.cpu.ttwu_local.47
   3352884 ± 17%     +32.3%    4436488 ±  4%  sched_debug.cpu.ttwu_local.48
   3674474 ±  4%     +22.3%    4495693 ±  5%  sched_debug.cpu.ttwu_local.49
   3448117 ±  5%     +27.9%    4408576 ±  7%  sched_debug.cpu.ttwu_local.5
   3300328 ± 15%     +32.4%    4368728 ±  4%  sched_debug.cpu.ttwu_local.50
   3359540 ±  6%     +31.7%    4426052 ±  6%  sched_debug.cpu.ttwu_local.51
   3398675 ± 15%     +32.1%    4488804 ±  5%  sched_debug.cpu.ttwu_local.52
   3744113 ±  7%     +22.3%    4579796 ±  5%  sched_debug.cpu.ttwu_local.53
   3186989 ± 15%     +40.5%    4477419 ±  2%  sched_debug.cpu.ttwu_local.54
   3578762 ±  8%     +25.2%    4478923 ±  9%  sched_debug.cpu.ttwu_local.55
   3390671 ± 15%     +30.9%    4437633 ±  4%  sched_debug.cpu.ttwu_local.56
   3860784 ±  2%     +17.4%    4534386 ±  4%  sched_debug.cpu.ttwu_local.57
   3205789 ± 12%     +37.7%    4414254 ±  5%  sched_debug.cpu.ttwu_local.58
   3395352 ±  7%     +32.7%    4506999 ±  7%  sched_debug.cpu.ttwu_local.59
   2915811 ± 16%     +45.4%    4240300 ±  1%  sched_debug.cpu.ttwu_local.6
   3562805 ± 14%     +23.5%    4401745 ±  6%  sched_debug.cpu.ttwu_local.60
   3715936 ±  6%     +21.2%    4503014 ±  5%  sched_debug.cpu.ttwu_local.61
   3204936 ± 14%     +39.2%    4461506 ±  2%  sched_debug.cpu.ttwu_local.62
   3507592 ±  4%     +28.1%    4492218 ±  6%  sched_debug.cpu.ttwu_local.63
   3202967 ± 13%     +37.3%    4398303 ±  4%  sched_debug.cpu.ttwu_local.64
   3483280 ±  7%     +30.7%    4554002 ±  5%  sched_debug.cpu.ttwu_local.65
   3138219 ± 15%     +39.9%    4389905 ±  4%  sched_debug.cpu.ttwu_local.66
   3153951 ±  6%     +41.6%    4466967 ±  5%  sched_debug.cpu.ttwu_local.67
   3229444 ± 12%     +37.5%    4441388 ±  5%  sched_debug.cpu.ttwu_local.68
   3394709 ±  3%     +32.3%    4489810 ±  5%  sched_debug.cpu.ttwu_local.69
   3230002 ±  7%     +33.4%    4308068 ±  6%  sched_debug.cpu.ttwu_local.7
   3056815 ± 13%     +43.4%    4382407 ±  3%  sched_debug.cpu.ttwu_local.70
   3223738 ±  6%     +37.6%    4437307 ±  7%  sched_debug.cpu.ttwu_local.71
   3460274 ± 13%     +29.3%    4473753 ±  4%  sched_debug.cpu.ttwu_local.72
   3860628 ±  6%     +19.2%    4600629 ±  4%  sched_debug.cpu.ttwu_local.73
   3361754 ± 15%     +31.7%    4427085 ±  4%  sched_debug.cpu.ttwu_local.74
   3375963 ±  9%     +32.4%    4468587 ±  5%  sched_debug.cpu.ttwu_local.75
   3416899 ± 13%     +30.6%    4462381 ±  4%  sched_debug.cpu.ttwu_local.76
   3725021 ±  2%     +22.7%    4571798 ±  4%  sched_debug.cpu.ttwu_local.77
   3301968 ±  9%     +35.0%    4458653 ±  2%  sched_debug.cpu.ttwu_local.78
   3438172 ± 10%     +31.5%    4519809 ±  8%  sched_debug.cpu.ttwu_local.79
   3296672 ± 16%     +29.4%    4267155 ±  6%  sched_debug.cpu.ttwu_local.8
   3480776 ±  7%     +27.5%    4437332 ±  5%  sched_debug.cpu.ttwu_local.9
   3329473 ±  7%     +33.0%    4429258 ±  4%  sched_debug.cpu.ttwu_local.avg
   2541283 ± 13%     +56.8%    3983513 ±  5%  sched_debug.cpu.ttwu_local.min
    414648 ± 17%     -57.3%     176872 ±  3%  sched_debug.cpu.ttwu_local.stddev
      0.00 ± 85%     -79.4%       0.00 ± 10%  sched_debug.rt_rq:/.rt_time.24
      0.03 ±149%     -96.4%       0.00 ± 27%  sched_debug.rt_rq:/.rt_time.4
      0.00 ± 58%     -80.4%       0.00 ± 27%  sched_debug.rt_rq:/.rt_time.9


lkp-wsx02: Westmere-EX
Memory: 128G





    5e+11 ++----------------------------------------------------------------+
  4.5e+11 ++        O                                                       |
          |                                                                 |
    4e+11 ++                                                                |
  3.5e+11 ++                                                                |
          |                                                                 |
    3e+11 ++                                                                |
  2.5e+11 +O    O      O  O  O  OO                                          |
    2e+11 O+ OO  O O O        O   O                                         |
          |                                                                 |
  1.5e+11 ++                                                                |
    1e+11 ++                                                                |
          |                                                                 |
    5e+10 +*. *.* .***. *.**.**.** .* .* .** .* .* .** .* .**.**.***.**.* .**
        0 *+-*---*-----*O--O------*--*--*---*--*--*---*--*---------------*--+


                                    softirqs.RCU

  1e+07 ++--------------------O---------------------------------------------+
  9e+06 OO O  OO O      O  O     O                                          |
        |   O       OO       O  O                                           |
  8e+06 ++        O                                                         |
  7e+06 ++                                                                  |
        |*.**.**.*      *.* .* .*  *  **.***.*  **.**.**.**.**.**.**.      **
  6e+06 ++        *.*  *   *  * :  :  :      :  :                    **.* : |
  5e+06 ++          :  :         :: ::        ::                         :: |
  4e+06 ++          :  :         :: ::        ::                         *  |
        *           : :          *  *         *                             |
  3e+06 ++           ::                                                     |
  2e+06 ++           ::                                                     |
        |            ::                                                     |
  1e+06 ++           :                                                      |
      0 ++-----------*-O--O-------------------------------------------------+


                                  cpuidle.C6-NHM.time

    3e+10 *+-------------------------*---------*----------------------------+
          |                          :         :                            |
  2.5e+10 ++                         :         :                            |
          |                          :         :                            |
          |                       *  ::        ::                           |
    2e+10 ++                      : : :        ::                           |
          |                       ::: :        ::                           |
  1.5e+10 ++                      ::: :       : :                           |
          |:                      : : :       : :                        *  |
    1e+10 ++                     :  * :       : :                        :  |
          |:                     :     :      :  :                       :: |
          |:                     :     :      :  :                      : : |
    5e+09 ++                .*   :     :      :  :                      :  :|
          |*.**.**.***. *.**  *.**     **.***.*  **.***.**.**.**.***.**.*  **
        0 OO-OO-OO-OOO-OO-OO-OO-OOO-----------------------------------------+


                                 turbostat.Avg_MHz

  2500 O+OO-OO-OO-OO-O*-O*-OO-OO-O--------------------------**-**-**-**----**
       | **.**.**  :  :    *   :     *.**.**.*  **.**.**.**             *  :|
       | :         :  :         :    :       :  :                       : : |
  2000 ++          :  :         : *  :       : :                         :: |
       |:           : :         : :: :        ::                         *  |
       |:           : :          : ::         ::                            |
  1500 ++           : :          *  :         :                             |
       *            : :             *         *                             |
  1000 ++           ::                                                      |
       |            ::                                                      |
       |            ::                                                      |
   500 ++            :                                                      |
       |             :                                                      |
       |             :                                                      |
     0 ++------------*O--O--------------------------------------------------+


                                  turbostat._Busy

  100 O+OO-OO-OO-OO-O--O--OO-OO-O-------------------------------------------+
   90 ++**.**.**.**  *.**.**.**     *.**.*.**  *.**.**.**.**.**.**.**.**  *.*
      | :         :  :        :     :       :  :                        : : |
   80 ++          :  :         : *  :        : :                        ::  |
   70 ++          :  :         : :: :        : :                         *  |
      |:           : :          : ::         ::                             |
   60 ++           : :          *  :          :                             |
   50 *+           : :             *          *                             |
   40 ++           ::                                                       |
      |            ::                                                       |
   30 ++           ::                                                       |
   20 ++            :                                                       |
      |             :                                                       |
   10 ++            :                                                       |
    0 ++------------*O--O---------------------------------------------------+


                                turbostat.CPU_c6

  60 ++---------------------------------------------------------------------+
     |                                                                      |
  50 *+                           *          *                              |
     |                            :          :                              |
     |                         *  :          :                              |
  40 ++                        :  ::         ::                             |
     |:                        :: ::         ::                             |
  30 ++                        ::: :         ::                             |
     |:                        ::: :        : :                          *  |
  20 ++                       :  : :        : :                          :  |
     |:                       :  * :        : :                         ::  |
     | :                      :     :       :  :                        : : |
  10 ++:                 *.   :     :       :  :                       :  : |
     | **.**.**.**. *.*.*  **.*     **.*.**.*  **.**.**.**.*.**.**.**.**  *.*
   0 O+OO-OO-OO-OO-OO-O-OO-OO-OO--------------------------------------------+




  18 ++---------------------------------------------------------------------+
     |                      O                                               |
  16 O+ O     O       O  O                                                  |
  14 ++    O       O       O   O                                            |
     |       O                O                                             |
  12 ++O        O                                                           |
  10 ++   O      O                                                          |
     |                                                                      |
   8 ++                                                                     |
   6 ++                                                                     |
     |                                                                      |
   4 ++                                                                     |
   2 ++                                                                     |
     |  *.**.  .**.    .*            *.*.*      *.**.       .*      *.* .* .*
   0 *+*-----**----*O-*-O*-**-**-**-*-----*-**-*-----**-**-*--*-**-*---*--*-+




  18 ++---------------------------------------------------------------------+
     |                      O                                               |
  16 O+ O     O       O  O                                                  |
  14 ++    O       O       O   O                                            |
     |       O                O                                             |
  12 ++         O                                                           |
  10 ++O  O      O                                                          |
     |                                                                      |
   8 ++                                                                     |
   6 ++                                                                     |
     |                                                                      |
   4 ++                                                                     |
   2 ++                                                                     |
     |  *.**.  .**.    .*            *.*.*      *.**.       .*      *.* .* .*
   0 *+*-----**----*O-*-O*-**-**-**-*-----*-**-*-----**-**-*--*-**-*---*--*-+




  18 ++---------------------------------------------------------------------+
     |                      O                                               |
  16 O+ O     O       O  O                                                  |
  14 ++    O       O       O   O                                            |
     |       O                O                                             |
  12 ++         O                                                           |
  10 ++O  O      O                                                          |
     |                                                                      |
   8 ++                                                                     |
   6 ++                                                                     |
     |                                                                      |
   4 ++                                                                     |
   2 ++                                                                     |
     |  *.**.  .**.    .*            *.*.*      *.**.       .*      *.* .* .*
   0 *+*-----**----*O-*-O*-**-**-**-*-----*-**-*-----**-**-*--*-**-*---*--*-+




  18 ++---------------------------------------------------------------------+
     |                      O                                               |
  16 O+ O     O       O  O                                                  |
  14 ++    O       O       O   O                                            |
     |       O                O                                             |
  12 ++         O                                                           |
  10 ++O  O      O                                                          |
     |                                                                      |
   8 ++                                                                     |
   6 ++                                                                     |
     |                                                                      |
   4 ++                                                                     |
   2 ++                                                                     |
     |  *.**.  .**.    .*            *.*.*      *.**.       .*      *.* .* .*
   0 *+*-----**----*O-*-O*-**-**-**-*-----*-**-*-----**-**-*--*-**-*---*--*-+


                                hackbench.throughput

  300000 ++-----------------------------------------------------------------+
         |                                                                  |
  250000 ++                                              *.*                |
         |*.**.**. *.*        .*      *.**.**   *.**.**. :  :       .**.  .*|
         *        *  :  ***.**  *.**.*       *.*        *   **. *.**    **  |
  200000 ++          :  :                                      *            *
         |           :  :                                                   |
  150000 ++          : :                                                    |
         |           : :                                                    |
  100000 OO OO OO OO OO: O  OO OO O                                         |
         |            ::                                                    |
         |            ::                                                    |
   50000 ++           :                                                     |
         |            :                                                     |
       0 ++-----------*-O-O-------------------------------------------------+


                    hackbench.time.percent_of_cpu_this_job_got

  8000 O+OO-OO-OO-OO-O--O--OO-OO-O------------------------------------------+
       | **.**.**.**  *.**. *.**     *.**.**.*  **.**.**.**.**.**.**.**.*  **
  7000 ++:         :  :    *   :     :       :  :                       : : |
  6000 ++          :  :         : *  :       : :                         :: |
       |:          :  :         : :: :        ::                         *  |
  5000 ++           : :          : ::         ::                            |
       |            : :          *  :         :                             |
  4000 *+           : :             *         *                             |
       |            ::                                                      |
  3000 ++           ::                                                      |
  2000 ++           ::                                                      |
       |             :                                                      |
  1000 ++            :                                                      |
       |             :                                                      |
     0 ++------------*O--O--------------------------------------------------+


                             numa-numastat.node1.numa_hit

  1.2e+08 ++----------------------------------------------------------------+
          |                                                                 |
    1e+08 ++       *                          *         *                   |
          |       + :*                       +:        + :.**         *     |
          |*    **  *:    *     *      *   ** :  *   **  *   : *.    * :    |
    8e+07 +++  +     :   ::  * + :  *  ::  :  :  ::  :       : : ** +  : *  *
          |: **       :  : :: *  * :: : *.*    :: *.*         :    *    * + |
    6e+07 ++          : *  ::     :: ::        ::             *            *|
          |           : :  *      *  :         :                            |
    4e+07 *+          : :            *         *                            |
          |           : :                                                   |
          |           ::                                                    |
    2e+07 ++           :                                                    |
          O            :                                                    |
        0 +O-OO-OO-OOO-OO-OO-OO-OOO-----------------------------------------+


                            numa-numastat.node1.local_node

  1.2e+08 ++----------------------------------------------------------------+
          |                                                                 |
    1e+08 ++       *                          *         *                   |
          |       + :*                       +:        + :.**         *     |
          |*    **  *:    *     *      *   ** :  *   **  *   : *.    * :    |
    8e+07 +++  +     :   ::  * + :  *  ::  :  :  ::  :       : : ** +  : *  *
          |: **       :  : :: *  * :: : *.*    :: *.*         :    *    * + |
    6e+07 ++          : *  ::     :: ::        ::             *            *|
          |           : :  *      *  :         :                            |
    4e+07 *+          : :            *         *                            |
          |           : :                                                   |
          |           ::                                                    |
    2e+07 ++           :                                                    |
          O            :                                                    |
        0 +O-OO-OO-OOO-OO-OO-OO-OOO-----------------------------------------+


                            numa-numastat.node2.numa_hit

  9e+07 ++------------------------------------------------------------------+
        |     *                           *         *    *                  |
  8e+07 +*.* + :                *     **. ::    **. :+   :                  |
  7e+07 ++  *  *  *.            :     :  * *    :  *  * : :        *.* .*   |
        |:      : : *  *   *.* ::    :      :  :      : : *        :  * :  *|
  6e+07 ++      ::  :  ::  : : ::    :      :  :       :   + *. *.*     :  :*
  5e+07 ++       *  :  :*.*   :  :  :        *:        *    *  *         :: |
        *           : :       *  : **         *                          :: |
  4e+07 ++          : :          :+                                      *  |
  3e+07 ++           ::          *                                          |
        |            ::                                                     |
  2e+07 ++           ::                                                     |
  1e+07 ++           :                                                      |
        |            :                                                      |
      0 OO-OO-OO-OO-OO-OO-OO-OO-OO------------------------------------------+


                           numa-numastat.node2.local_node

  9e+07 ++------------------------------------------------------------------+
        |     *                           *         *    *                  |
  8e+07 +*.* + :                *     **. ::    **. :+   :                  |
  7e+07 ++  *  *  *.            :     :  * *    :  *  * : :        *.* .*   |
        |:      : : *  *   *.* ::    :      :  :      : : *        :  * :  *|
  6e+07 ++      ::  :  ::  : : ::    :      :  :       :   + *. *.*     :  :*
  5e+07 ++       *  :  :*.*   :  :  :        *:        *    *  *         :: |
        *           : :       *  : **         *                          :: |
  4e+07 ++          : :          :+                                      *  |
  3e+07 ++           ::          *                                          |
        |            ::                                                     |
  2e+07 ++           ::                                                     |
  1e+07 ++           :                                                      |
        |            :                                                      |
      0 OO-OO-OO-OO-OO-OO-OO-OO-OO------------------------------------------+


                            numa-numastat.node3.numa_hit

  1e+08 ++------------------------------------------------*-----------------+
  9e+07 ++                                                ::                |
        |*                            *         *        : :               *|
  8e+07 +++       *                   ::        ::       *  :     *  *     :|
  7e+07 ++ *   *  ::      * .*     * : *.  *   : *.  .*  :  **   : :+:    : |
        |:  *.* : ::   * + *  :    : :   ** :  :   ** : :     +  : *  :   : |
  6e+07 ++      ::  *  :*     *.* :: :      :  :       ::      **     *.**  *
  5e+07 ++       :  :  :         :: ::       : :       :                    |
  4e+07 ++       *  : :          *  :        *:        *                    |
        *           : :             *         *                             |
  3e+07 ++           ::                                                     |
  2e+07 ++           ::                                                     |
        |            :                                                      |
  1e+07 ++           :                                                      |
      0 OO-OO-OO-OO-OO-OO-OO-OO-OO------------------------------------------+


                           numa-numastat.node3.local_node

  1e+08 ++------------------------------------------------*-----------------+
  9e+07 ++                                                ::                |
        |*                            *         *        : :               *|
  8e+07 +++       *                   ::        ::       *  :     *  *     :|
  7e+07 ++ *   *  ::      * .*     * : *.  *   : *.  .*  :  **   : :+:    : |
        |:  *.* : ::   * + *  :    : :   ** :  :   ** : :     +  : *  :   : |
  6e+07 ++      ::  *  :*     *.* :: :      :  :       ::      **     *.**  *
  5e+07 ++       :  :  :         :: ::       : :       :                    |
  4e+07 ++       *  : :          *  :        *:        *                    |
        *           : :             *         *                             |
  3e+07 ++           ::                                                     |
  2e+07 ++           ::                                                     |
        |            :                                                      |
  1e+07 ++           :                                                      |
      0 OO-OO-OO-OO-OO-OO-OO-OO-OO------------------------------------------+


                              numa-vmstat.node0.numa_hit

    5e+07 ++---------------*------------------------------------------------+
  4.5e+07 ++               :                                                |
          |   *           : :            .*        .*                       |
    4e+07 ++ * :   *      * :   *       * :   *   * :   *               *   |
  3.5e+07 +++  :  +:      : :  ::   *   :  : +:   :  : + *       * *   + :  |
          |*    **  :*   :   : : :  :  *   ** :  *   **   +     : * : *  *. |
    3e+07 ++        ::   :   **  * : ::        ::          **.  :   : :    *|
  2.5e+07 ++        *:  *         :: ::        ::             **     *      *
    2e+07 *+          : :         *  *         *                            |
          |           : :                                                   |
  1.5e+07 ++          : :                                                   |
    1e+07 ++          ::                                                    |
          |            :                                                    |
    5e+06 ++           :                                                    |
        0 OO-OO-OO-OOO-OO-OO-OO-OOO-----------------------------------------+


                             numa-vmstat.node0.numa_local

    5e+07 ++---------------*------------------------------------------------+
  4.5e+07 ++               :                                                |
          |   *           : :            .*        .*                       |
    4e+07 ++ * :   *      * :   *       * :   *   * :   *               *   |
  3.5e+07 +++  :  +:      : :  ::   *   :  : +:   :  : + *       * *   + :  |
          |*    **  :*   :   : : :  :  *   ** :  *   **   +     : * : *  *. |
    3e+07 ++        ::   :   **  * : ::        ::          **.  :   : :    *|
  2.5e+07 ++        *:  *         :: ::        ::             **     *      *
    2e+07 *+          : :         *  *         *                            |
          |           : :                                                   |
  1.5e+07 ++          : :                                                   |
    1e+07 ++          ::                                                    |
          |            :                                                    |
    5e+06 ++           :                                                    |
        0 OO-OO-OO-OOO-OO-OO-OO-OOO-----------------------------------------+


                             numa-vmstat.node1.numa_hit

  6e+07 ++------------------------------------------------------------------+
        |       .*                          .*         *                    |
  5e+07 ++     * :                         * :        * :                   |
        |      :  :                        : :       +  :                   |
        |*   .*   :     *  *       *  *   *   : *   *    :*.*  *            |
  4e+07 ++: *     *.*   :  ::  .*  : ::  *    :::  *     *   :+ :  *.**  *. |
        | : :       :  * : :: * : : :: :+     :: :+          *  *.*    + : **
  3e+07 *+ *        :  : ::  *   :: *  *      *  *                      *   |
        |           :  :  :      :                                          |
  2e+07 ++          : :   *      *                                          |
        |            ::                                                     |
        |            ::                                                     |
  1e+07 ++           ::                                                     |
        O            :                                                      |
      0 +O-OO-OO-OO-OO-OO-OO-OO-OO------------------------------------------+


                            numa-vmstat.node1.numa_local

  6e+07 ++------------------------------------------------------------------+
        |       .*                          .*         *                    |
  5e+07 ++     * :                         * :        * :                   |
        |      :  :                        : :       +  :                   |
        |*   .*   :     *  *       *  *   *   : *   *    :*.*  *            |
  4e+07 ++: *     *.*   :  ::  .*  : ::  *    :::  *     *   :+ :  *.**  *. |
        | : :       :  * : :: * : : :: :+     :: :+          *  *.*    + : **
  3e+07 *+ *        :  : ::  *   :: *  *      *  *                      *   |
        |           :  :  :      :                                          |
  2e+07 ++          : :   *      *                                          |
        |            ::                                                     |
        |            ::                                                     |
  1e+07 ++           ::                                                     |
        O            :                                                      |
      0 +O-OO-OO-OO-OO-OO-OO-OO-OO------------------------------------------+


                              numa-vmstat.node2.numa_hit

    5e+07 ++----------------------------------------------------------------+
  4.5e+07 ++    *                *         *         *                      |
          |*.  + :               :     *   ::    *   ::  *                  |
    4e+07 ++ **  *   *           :    + *.* *   + *.* *  :+           *.    |
  3.5e+07 *+      : *:       *   :   *       : *       : : *    .***.*  *   |
          |       : ::  *    :* : :  :       : :       ::  :   *        :  **
    3e+07 ++       *  : :+  :  +: : *         *         *   :  :         :: |
  2.5e+07 ++          : : * :   * ::                        :.*          :: |
    2e+07 ++          : :  *      ::                        *            *  |
          |           : :         *                                         |
  1.5e+07 ++          ::                                                    |
    1e+07 ++          ::                                                    |
          |            :                                                    |
    5e+06 ++           :                                                    |
        0 OO-OO-OO-OOO-OO-OO-OO-OOO-----------------------------------------+


                             numa-vmstat.node2.numa_local

    5e+07 ++----------------------------------------------------------------+
  4.5e+07 ++    *                *         *         *                      |
          |*.  + :               :     *   ::    *   ::  *                  |
    4e+07 ++ **  *               :    + *.* *   + *.* *  :+                 |
  3.5e+07 *+      : **       *   :   *       : *       : : *    .***.**.*   |
          |       : ::  *    :* : :  :       : :       ::  :   *        :  **
    3e+07 ++       * :  :+  :  +: : *         *         *   :  :         :: |
  2.5e+07 ++          : : * :   * ::                        :.*          :: |
    2e+07 ++          : :  *      ::                        *            *  |
          |           : :         *                                         |
  1.5e+07 ++          ::                                                    |
    1e+07 ++          ::                                                    |
          |            :                                                    |
    5e+06 ++           :                                                    |
        0 OO-OO-OO-OOO-OO-OO-OO-OOO-----------------------------------------+


                             numa-vmstat.node3.numa_hit

  6e+07 ++------------------------------------------------------------------+
        |                                                                   |
  5e+07 ++                                                *                 |
        |*                            *         *         ::                |
        |::                  *        :         :        : :      *        *|
  4e+07 ++:  .**  *         ::     * : :  **   : :  *.*  *  :*    :  *    ::|
        |  **  :  :+   *    : :    ::: *.* :   : *.*  :  :  * +  : :: :   : |
  3e+07 *+      : : *  ::.**  *.* : *       : *       : :      * : :: *.**  *
        |       ::  :  :*        ::         : :        ::       *  *        |
  2e+07 ++       :  : :          *           :         :                    |
        |        *  : :                      *         *                    |
        |            ::                                                     |
  1e+07 ++           ::                                                     |
        |            :                                                      |
      0 OO-OO-OO-OO-OO-OO-OO-OO-OO------------------------------------------+


                            numa-vmstat.node3.numa_local

  6e+07 ++------------------------------------------------------------------+
        |                                                                   |
  5e+07 ++                                                *                 |
        |*                            *         *         :                 |
        |::                  *        :         :        : :      *        *|
  4e+07 ++:  .**  *         ::     * : :  **   : :  *.*  * : *    :  *     :|
        |  **  :  :+   *    : :    ::: *.* :   : *.*  :  :  ::+  : :: :   : |
  3e+07 *+      : : *  ::.**  :.* : *       : *       : :   *  * : :: *.* : *
        |       ::  :  :*     *  ::         : :        ::       *  *     *  |
  2e+07 ++       :  : :          *           :         :                    |
        |        *  : :                      *         *                    |
        |            ::                                                     |
  1e+07 ++           ::                                                     |
        |            :                                                      |
      0 OO-OO-OO-OO-OO-OO-OO-OO-OO------------------------------------------+


                                 proc-vmstat.numa_hit

  3.5e+08 ++----------------------------------------------------------------+
          |                                              *.                 |
    3e+08 +*.**.**. *                  **.***.   **.***. : *        .*     *|
          |:       * *   .**. *.**     :      *  :      *   *.  .***  *.*  :|
  2.5e+08 ++         :  *    *   :  * :       : :             **        : : *
          |:         :  :        :  : :       : :                        :: |
    2e+08 ++          : :         :: ::        ::                        *  |
          |           : :         :: :         :                            |
  1.5e+08 *+          : :         *  *         *                            |
          |           : :                                                   |
    1e+08 ++          ::                                                    |
          |           ::                                                    |
    5e+07 ++           :                                                    |
          |            :                                                    |
        0 OO-OO-OO-OOO-OO-OO-OO-OOO-----------------------------------------+


                                proc-vmstat.numa_local

  3.5e+08 ++----------------------------------------------------------------+
          |                                              *.                 |
    3e+08 +*.**.**. *                  **.***.   **.***. : *        .*     *|
          |:       * *   .**. *.**     :      *  :      *   *.  .***  *.*  :|
  2.5e+08 ++         :  *    *   :  * :       : :             **        : : *
          |:         :  :        :  : :       : :                        :: |
    2e+08 ++          : :         :: ::        ::                        *  |
          |           : :         :: :         :                            |
  1.5e+08 *+          : :         *  *         *                            |
          |           : :                                                   |
    1e+08 ++          ::                                                    |
          |           ::                                                    |
    5e+07 ++           :                                                    |
          |            :                                                    |
        0 OO-OO-OO-OOO-OO-OO-OO-OOO-----------------------------------------+


                              proc-vmstat.pgalloc_normal

  3.5e+08 ++----------------------------------------------------------------+
          |                                                                 |
    3e+08 +*.  .**.                    *   **.   *   **. *.*               *|
          |: **    ***   .*   *.**     :*.*   *  :*.*   *   *.  .***.**.*  :|
  2.5e+08 ++         :  *  *.*   :  * :       : :             **        : : *
          |:         :  :        :  : :       : :                        :: |
    2e+08 ++          : :         :: ::        ::                        *  |
          |           : :         :: :         :                            |
  1.5e+08 *+          : :         *  *         *                            |
          |           : :                                                   |
    1e+08 ++          ::                                                    |
          |           ::                                                    |
    5e+07 ++           :                                                    |
          |            :                                                    |
        0 OO-OO-OO-OOO-OO-OO-OO-OOO-----------------------------------------+


                                  proc-vmstat.pgfree

  3.5e+08 ++----------------------------------------------------------------+
          |*.                          *         *       *.*                |
    3e+08 ++ **.**. *                  :*.***.   :*.***. :  :       .*     *|
          |:       * *   .**. *.**    :       * :       *   *.* .***  *.* : |
  2.5e+08 ++         :  *    *   :  * :       : :              *         :: *
          |          :  :        :  : :       : :                        *  |
    2e+08 ++          : :         :: ::        ::                           |
          |           : :         :: :         :                            |
  1.5e+08 *+          : :         *  *         *                            |
          |           : :                                                   |
    1e+08 ++          ::                                                    |
          |           ::                                                    |
    5e+07 ++           :                                                    |
          |            :                                                    |
        0 OO-OO-OO-OOO-OO-OO-OO-OOO-----------------------------------------+


                          slabinfo.taskstats.active_objs

  450 ++--------------------------------------------------------------------+
      |                                                  .*                 |
  400 ++*   *.* .*                  *.  .*.*   *.  .**. * :         *.    * |
  350 ++ *.*   *  *   .* .*  **     : **    *  : **    *   :.*  **. : **. ::|
      |:          :  *  *  :+ :  *.:         +:            *  :+   *     * :|
  300 *+          :  :     *   : : *          *               *             *
  250 ++           : :         : :                                          |
      |            : :          :                                           |
  200 ++OO OO OO OO:O: O  OO  O O                                           |
  150 O+           : :       O                                              |
      |            ::                                                       |
  100 ++           ::                                                       |
   50 ++            :                                                       |
      |             :                                                       |
    0 ++------------*O--O---------------------------------------------------+


                            slabinfo.taskstats.num_objs

  450 ++--------------------------------------------------------------------+
      |                                                  .*                 |
  400 ++*   *.* .*                  *.  .*.*   *.  .**. * :         *.    * |
  350 ++ *.*   *  *   .* .*  **     : **    *  : **    *   :.*  **. : **. ::|
      |:          :  *  *  :+ :  *.:         +:            *  :+   *     * :|
  300 *+          :  :     *   : : *          *               *             *
  250 ++           : :         : :                                          |
      |            : :          :                                           |
  200 ++OO OO OO OO:O: O  OO  O O                                           |
  150 O+           : :       O                                              |
      |            ::                                                       |
  100 ++           ::                                                       |
   50 ++            :                                                       |
      |             :                                                       |
    0 ++------------*O--O---------------------------------------------------+


        kmsg.DMAR-IR:Failed_to_copy_IR_table_for_dmar0_from_previous_kernel

    1 *+-*-*------------------------------------------*-*--*-----------*--*-*
      |    :                                          :    :           :    |
      |    :                                          :    :           :    |
  0.8 ++    :                                        :     :           :    |
      |     :                                        :      :         :     |
      |     :                                        :      :         :     |
  0.6 ++    :                                        :      :         :     |
      |      :                                       :      :         :     |
  0.4 ++     :                                      :       :         :     |
      |      :                                      :       :         :     |
      |      :                                      :       :         :     |
  0.2 ++     :                                      :        :       :      |
      |       :                                    :         :       :      |
      |       :                                    :         :       :      |
    0 ++------*-*--*-*--*-*--*-*--*-*--*-*--*-*--*-*---------*--*-*--*------+


        kmsg.DMAR-IR:Failed_to_copy_IR_table_for_dmar1_from_previous_kernel

    1 *+-*-*------------------------------------------*-*--*-----------*--*-*
      |    :                                          :    :           :    |
      |    :                                          :    :           :    |
  0.8 ++    :                                        :     :           :    |
      |     :                                        :      :         :     |
      |     :                                        :      :         :     |
  0.6 ++    :                                        :      :         :     |
      |      :                                       :      :         :     |
  0.4 ++     :                                      :       :         :     |
      |      :                                      :       :         :     |
      |      :                                      :       :         :     |
  0.2 ++     :                                      :        :       :      |
      |       :                                    :         :       :      |
      |       :                                    :         :       :      |
    0 ++------*-*--*-*--*-*--*-*--*-*--*-*--*-*--*-*---------*--*-*--*------+


   kmsg.DMAR-IR:IRQ_remapping_was_enabled_on_dmar0_but_we_are_not_in_kdump_mode

    1 ++-*-*--------------------------------------------*--*--------------*-*
      |  : :                                            :  :              : |
      |  : :                                            :  :              : |
  0.8 ++:   :                                           :  :             :  |
      | :   :                                          :    :            :  |
      | :   :                                          :    :            :  |
  0.6 ++:   :                                          :    :            :  |
      | :    :                                         :    :            :  |
  0.4 ++     :                                         :    :           :   |
      |:     :                                         :    :           :   |
      |:     :                                         :    :           :   |
  0.2 ++     :                                        :      :          :   |
      |       :                                       :      :         :    |
      |       :                                       :      :         :    |
    0 *+------*-*--*-*--*-*--*-*--*-*--*-*--*-*--*-*--*------*--*-*--*-*----+


   kmsg.DMAR-IR:IRQ_remapping_was_enabled_on_dmar1_but_we_are_not_in_kdump_mode

    1 ++-*-*--------------------------------------------*--*--------------*-*
      |  : :                                            :  :              : |
      |  : :                                            :  :              : |
  0.8 ++:   :                                           :  :             :  |
      | :   :                                          :    :            :  |
      | :   :                                          :    :            :  |
  0.6 ++:   :                                          :    :            :  |
      | :    :                                         :    :            :  |
  0.4 ++     :                                         :    :           :   |
      |:     :                                         :    :           :   |
      |:     :                                         :    :           :   |
  0.2 ++     :                                        :      :          :   |
      |       :                                       :      :         :    |
      |       :                                       :      :         :    |
    0 *+------*-*--*-*--*-*--*-*--*-*--*-*--*-*--*-*--*------*--*-*--*-*----+


                   kmsg.DHCP_BOOTP:Reply_not_for_us_op___xid___

    1 ++--------------------------------------------------------------------*
      |                                                                     |
      |                                                                    :|
  0.8 ++                                                                   :|
      |                                                                    :|
      |                                                                   : |
  0.6 ++                                                                  : |
      |                                                                   : |
  0.4 ++                                                                 :  |
      |                                                                  :  |
      |                                                                 :   |
  0.2 ++                                                                :   |
      |                                                                 :   |
      |                                                                :    |
    0 *+---*---*----*----*---*----*----*---*----*----*---*----*----*---*----+

	[*] bisect-good sample
	[O] bisect-bad  sample

To reproduce:

        git clone git://git.kernel.org/pub/scm/linux/kernel/git/wfg/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml


Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Ying Huang

View attachment "job.yaml" of type "text/plain" (3401 bytes)

Download attachment "reproduce.sh" of type "application/x-sh" (8014 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ