lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Message-ID: <20160807141857.GB22624@yexl-desktop>
Date:	Sun, 7 Aug 2016 22:18:57 +0800
From:	kernel test robot <xiaolong.ye@...el.com>
To:	Ilya Lesokhin <ilyal@...lanox.com>
Cc:	0day robot <fengguang.wu@...el.com>,
	LKML <linux-kernel@...r.kernel.org>, lkp@...org
Subject: [lkp] [net]  9863a93605:  netpipe.bigger_5M_Mbps.avg -80.0%
 regression


FYI, we noticed a -80.0% regression of netpipe.bigger_5M_Mbps.avg due to commit:

commit 9863a936051349cd921735b4c84eaa209de34c67 ("net: use non-compound pages in frag allocator")
https://github.com/0day-ci/linux Ilya-Lesokhin/net-use-non-compound-pages-in-frag-allocator/20160804-205029

in testcase: netpipe
on test machine: 48 threads 2 sockets Xeon E5 (Ivytown Ivy Bridge-EP) with 64G memory
with following parameters:

	test: tcp
	cluster: cs-localhost
	cpufreq_governor: performance



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.

Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone git://git.kernel.org/pub/scm/linux/kernel/git/wfg/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
cluster/compiler/cpufreq_governor/kconfig/rootfs/tbox_group/test/testcase:
  cs-localhost/gcc-6/performance/x86_64-rhel/debian-x86_64-2015-02-07.cgz/ivb42/tcp/netpipe

commit: 
  6a492b0f23 (" SCSI misc on 20160727")
  9863a93605 ("net: use non-compound pages in frag allocator")

6a492b0f23d28e1f 9863a936051349cd921735b4c8 
---------------- -------------------------- 
         %stddev     %change         %stddev
             \          |                \  
     40585 ±  0%     -80.0%       8101 ±  6%  netpipe.bigger_5M_Mbps.avg
    107.21 ±  0%    +327.3%     458.16 ±  3%  netpipe.time.elapsed_time
    107.21 ±  0%    +327.3%     458.16 ±  3%  netpipe.time.elapsed_time.max
     25226 ± 23%     -39.1%      15364 ± 32%  netpipe.time.minor_page_faults
     92.00 ±  0%     -27.7%      66.50 ±  1%  netpipe.time.percent_of_cpu_this_job_got
     98.52 ±  0%    +211.0%     306.41 ±  2%  netpipe.time.system_time
   2017738 ±  1%    +110.4%    4244933 ±  0%  netpipe.time.voluntary_context_switches
     60026 ±  0%    +256.6%     214043 ±  3%  interrupts.CAL:Function_call_interrupts
     38034 ± 27%    +324.1%     161283 ±  2%  softirqs.RCU
    143833 ±  0%    +212.0%     448816 ±  3%  softirqs.SCHED
    330108 ±  0%    +273.9%    1234413 ±  2%  softirqs.TIMER
      4650 ± 74%  +16228.4%     759271 ±  8%  numa-numastat.node0.numa_miss
   8561476 ± 24%   +1137.5%  1.059e+08 ±  5%  numa-numastat.node1.local_node
      4658 ± 74%  +16166.7%     757742 ±  8%  numa-numastat.node1.numa_foreign
   8561481 ± 24%   +1137.5%  1.059e+08 ±  5%  numa-numastat.node1.numa_hit
  64824003 ±  0%     -83.2%   10874012 ±  0%  vmstat.memory.free
      1.00 ±  0%    +300.0%       4.00 ±  0%  vmstat.procs.r
     74306 ±  0%    +563.4%     492954 ±  0%  vmstat.system.cs
     49094 ±  0%     +20.2%      58995 ±  2%  vmstat.system.in
     17393 ±  3%      +7.1%      18637 ±  3%  slabinfo.anon_vma_chain.active_objs
     17393 ±  3%      +7.1%      18637 ±  3%  slabinfo.anon_vma_chain.num_objs
    601.00 ±  0%     -29.1%     426.25 ±  0%  slabinfo.kmalloc-8192.active_objs
    601.00 ±  0%     -23.1%     462.25 ±  0%  slabinfo.kmalloc-8192.num_objs
     15508 ±  5%     +11.3%      17261 ±  1%  slabinfo.vm_area_struct.active_objs
     15509 ±  5%     +11.3%      17261 ±  1%  slabinfo.vm_area_struct.num_objs
     55688 ±  1%     +10.4%      61503 ±  2%  meminfo.Active(anon)
     27090 ±  2%     -66.4%       9106 ± 60%  meminfo.Active(file)
    498088 ±  0%     -69.1%     153856 ± 61%  meminfo.Inactive
    488930 ±  0%     -70.4%     144542 ± 64%  meminfo.Inactive(file)
  64854901 ±  0%     -83.6%   10636727 ±  0%  meminfo.MemAvailable
  64822934 ±  0%     -83.2%   10881925 ±  0%  meminfo.MemFree
      9737 ±  0%     +62.9%      15866 ±  5%  meminfo.Shmem
  25915271 ±  2%   +2220.9%  6.015e+08 ±  5%  cpuidle.C1-IVT.time
   4060493 ±  0%   +2278.2%   96565345 ±  4%  cpuidle.C1-IVT.usage
   1098687 ±  9%    +468.9%    6250736 ±  7%  cpuidle.C1E-IVT.time
      8773 ±  6%    +413.3%      45034 ±  8%  cpuidle.C1E-IVT.usage
   3207505 ± 11%    +355.1%   14598780 ±  6%  cpuidle.C3-IVT.time
     14605 ±  5%    +347.5%      65353 ±  7%  cpuidle.C3-IVT.usage
 4.807e+09 ±  0%    +294.7%  1.897e+10 ±  3%  cpuidle.C6-IVT.time
   5191216 ±  0%    +313.4%   21458441 ±  3%  cpuidle.C6-IVT.usage
  25722007 ± 21%    +181.1%   72302915 ± 22%  cpuidle.POLL.time
      2864 ±  2%   +6446.9%     187502 ±  4%  cpuidle.POLL.usage
      2548 ±139%    +164.9%       6751 ± 52%  numa-meminfo.node0.Inactive(anon)
    244472 ±  0%     -48.7%     125471 ± 75%  numa-meminfo.node0.Inactive(file)
  32341249 ±  0%     -82.3%    5734651 ±  0%  numa-meminfo.node0.MemFree
    524982 ±  0%   +5068.1%   27131580 ±  0%  numa-meminfo.node0.MemUsed
      2835 ±124%    +205.8%       8669 ± 61%  numa-meminfo.node0.Shmem
     13549 ±  2%     -92.2%       1062 ±  2%  numa-meminfo.node1.Active(file)
    251064 ±  1%     -91.4%      21662 ± 18%  numa-meminfo.node1.Inactive
    244456 ±  0%     -92.2%      19100 ±  2%  numa-meminfo.node1.Inactive(file)
     11336 ± 21%     -29.2%       8024 ± 30%  numa-meminfo.node1.Mapped
  32481701 ±  0%     -84.2%    5146334 ±  1%  numa-meminfo.node1.MemFree
    511210 ±  0%   +5347.2%   27846557 ±  0%  numa-meminfo.node1.MemUsed
      7.06 ±  0%     +56.3%      11.03 ±  0%  turbostat.%Busy
    191.00 ±  1%     +80.6%     345.00 ±  0%  turbostat.Avg_MHz
      2707 ±  0%     +15.5%       3128 ±  0%  turbostat.Bzy_MHz
     32.43 ±  0%     +15.4%      37.43 ±  0%  turbostat.CPU%c1
     60.49 ±  0%     -14.8%      51.52 ±  0%  turbostat.CPU%c6
     71.09 ±  0%     +28.0%      91.00 ±  0%  turbostat.CorWatt
     64.25 ±  7%     +16.0%      74.50 ±  3%  turbostat.CoreTmp
     23.91 ±  2%     -93.3%       1.59 ±  8%  turbostat.Pkg%pc2
      0.03 ± 50%     -61.5%       0.01 ± 34%  turbostat.Pkg%pc6
     63.50 ±  7%     +16.9%      74.25 ±  3%  turbostat.PkgTmp
     99.87 ±  0%     +21.8%     121.68 ±  0%  turbostat.PkgWatt
      4.23 ±  0%    +135.2%       9.96 ±  0%  turbostat.RAMWatt
   8085307 ±  0%     -82.3%    1433827 ±  0%  numa-vmstat.node0.nr_free_pages
    637.75 ±139%    +164.5%       1686 ± 52%  numa-vmstat.node0.nr_inactive_anon
     61117 ±  0%     -48.7%      31371 ± 75%  numa-vmstat.node0.nr_inactive_file
    709.75 ±123%    +205.3%       2167 ± 61%  numa-vmstat.node0.nr_shmem
     73400 ±  4%    +964.5%     781320 ±  7%  numa-vmstat.node0.numa_miss
      3386 ±  2%     -92.2%     263.75 ±  2%  numa-vmstat.node1.nr_active_file
      4152 ±  5%      +7.8%       4475 ±  2%  numa-vmstat.node1.nr_alloc_batch
   8120415 ±  0%     -84.2%    1286672 ±  1%  numa-vmstat.node1.nr_free_pages
     61113 ±  0%     -92.2%       4749 ±  3%  numa-vmstat.node1.nr_inactive_file
      2936 ± 20%     -28.7%       2092 ± 28%  numa-vmstat.node1.nr_mapped
     20750 ± 17%   +3398.3%     725897 ±  8%  numa-vmstat.node1.numa_foreign
   4215737 ± 28%   +1119.2%   51399128 ±  4%  numa-vmstat.node1.numa_hit
   4215735 ± 28%   +1119.2%   51399123 ±  4%  numa-vmstat.node1.numa_local
     18130 ± 51%     -60.3%       7201 ± 23%  latency_stats.avg.perf_event_alloc.SYSC_perf_event_open.SyS_perf_event_open.entry_SYSCALL_64_fastpath
     22466 ±  2%    +362.9%     104006 ±  3%  latency_stats.hits.pipe_wait.pipe_read.__vfs_read.vfs_read.SyS_read.entry_SYSCALL_64_fastpath
     18130 ± 51%     -60.3%       7201 ± 23%  latency_stats.max.perf_event_alloc.SYSC_perf_event_open.SyS_perf_event_open.entry_SYSCALL_64_fastpath
      0.00 ± -1%      +Inf%      32812 ± 14%  latency_stats.sum.call_rwsem_down_write_failed.copy_process._do_fork.SyS_clone.do_syscall_64.return_from_SYSCALL_64
      0.00 ± -1%      +Inf%      31399 ± 24%  latency_stats.sum.call_rwsem_down_write_failed.unlink_file_vma.free_pgtables.exit_mmap.mmput.flush_old_exec.load_elf_binary.search_binary_handler.do_execveat_common.SyS_execve.do_syscall_64.return_from_SYSCALL_64
      0.00 ± -1%      +Inf%      14303 ± 10%  latency_stats.sum.call_rwsem_down_write_failed.unlink_file_vma.free_pgtables.unmap_region.do_munmap.mmap_region.do_mmap.vm_mmap_pgoff.SyS_mmap_pgoff.SyS_mmap.entry_SYSCALL_64_fastpath
      0.00 ± -1%      +Inf%      20756 ± 14%  latency_stats.sum.call_rwsem_down_write_failed.vma_adjust.__split_vma.split_vma.mprotect_fixup.SyS_mprotect.entry_SYSCALL_64_fastpath
      0.00 ± -1%      +Inf%      12369 ± 12%  latency_stats.sum.call_rwsem_down_write_failed.vma_link.mmap_region.do_mmap.vm_mmap_pgoff.SyS_mmap_pgoff.SyS_mmap.entry_SYSCALL_64_fastpath
    846532 ±  1%    +434.7%    4526424 ±  3%  latency_stats.sum.do_wait.SyS_wait4.entry_SYSCALL_64_fastpath
    538496 ±  2%    +324.6%    2286293 ±  3%  latency_stats.sum.msleep_interruptible.uart_wait_until_sent.tty_wait_until_sent.tty_port_close_start.tty_port_close_start.uart_close.tty_release.__fput.____fput.task_work_run.exit_to_usermode_loop.syscall_return_slowpath
     18130 ± 51%     -60.3%       7201 ± 23%  latency_stats.sum.perf_event_alloc.SYSC_perf_event_open.SyS_perf_event_open.entry_SYSCALL_64_fastpath
   1256542 ±  1%    +448.2%    6888859 ±  3%  latency_stats.sum.pipe_wait.pipe_read.__vfs_read.vfs_read.SyS_read.entry_SYSCALL_64_fastpath
      1291 ± 42%    +580.4%       8783 ± 26%  latency_stats.sum.pipe_wait.pipe_write.__vfs_write.vfs_write.SyS_write.entry_SYSCALL_64_fastpath
      4864 ± 96%    -100.0%       0.00 ± -1%  latency_stats.sum.sk_stream_wait_memory.tcp_sendmsg.inet_sendmsg.sock_sendmsg.sock_write_iter.__vfs_write.vfs_write.SyS_write.entry_SYSCALL_64_fastpath
  13366050 ±  1%   +2056.5%  2.882e+08 ±  5%  latency_stats.sum.sk_wait_data.tcp_recvmsg.inet_recvmsg.sock_recvmsg.sock_read_iter.__vfs_read.vfs_read.SyS_read.entry_SYSCALL_64_fastpath
   1709389 ±  0%    +348.0%    7658418 ±  3%  latency_stats.sum.tty_wait_until_sent.tty_port_close_start.tty_port_close_start.uart_close.tty_release.__fput.____fput.task_work_run.exit_to_usermode_loop.syscall_return_slowpath.entry_SYSCALL_64_fastpath
     13918 ±  1%     +10.6%      15387 ±  2%  proc-vmstat.nr_active_anon
      6772 ±  2%     -66.4%       2277 ± 60%  proc-vmstat.nr_active_file
   1622246 ±  0%     -83.6%     266614 ±  0%  proc-vmstat.nr_dirty_background_threshold
   3248460 ±  0%     -83.6%     533881 ±  0%  proc-vmstat.nr_dirty_threshold
  16205752 ±  0%     -83.2%    2720129 ±  0%  proc-vmstat.nr_free_pages
    122232 ±  0%     -70.4%      36142 ± 64%  proc-vmstat.nr_inactive_file
      2432 ±  0%     +63.1%       3966 ±  5%  proc-vmstat.nr_shmem
      9308 ±  0%   +8121.3%     765255 ±  8%  proc-vmstat.numa_foreign
     44068 ± 16%     -59.9%      17665 ± 21%  proc-vmstat.numa_hint_faults
     42800 ± 19%     -60.6%      16850 ± 18%  proc-vmstat.numa_hint_faults_local
  15663734 ±  0%    +605.4%  1.105e+08 ±  0%  proc-vmstat.numa_hit
  15663725 ±  0%    +605.4%  1.105e+08 ±  0%  proc-vmstat.numa_local
      9308 ±  0%   +8121.3%     765255 ±  8%  proc-vmstat.numa_miss
     81156 ± 11%     -56.9%      34998 ± 29%  proc-vmstat.numa_pte_updates
      0.00 ±  0%      +Inf%  1.477e+08 ±  2%  proc-vmstat.pageoutrun
      1591 ±  3%    +120.6%       3510 ±  9%  proc-vmstat.pgactivate
   5347312 ± 30%     -85.3%     784900 ±  2%  proc-vmstat.pgalloc_dma32
    331816 ±  1%    +239.9%    1127689 ±  3%  proc-vmstat.pgfault
 1.236e+08 ±  0%      -9.8%  1.115e+08 ±  0%  proc-vmstat.pgfree
      1332 ± 76%   +3006.4%      41384 ±  6%  proc-vmstat.pgmigrate_success
 8.468e+10 ± 16%   +1962.0%  1.746e+12 ±  4%  perf-stat.branch-instructions
      1.60 ± 10%     -76.2%       0.38 ±  1%  perf-stat.branch-miss-rate
 1.329e+09 ±  7%    +398.0%   6.62e+09 ±  3%  perf-stat.branch-misses
      1.39 ± 17%   +1926.4%      28.10 ±  1%  perf-stat.cache-miss-rate
  7.48e+08 ±  1%   +5011.7%  3.824e+10 ±  4%  perf-stat.cache-misses
 5.577e+10 ± 19%    +143.9%   1.36e+11 ±  3%  perf-stat.cache-references
   8144248 ±  0%   +2687.4%   2.27e+08 ±  3%  perf-stat.context-switches
 1.397e+12 ± 14%    +512.6%   8.56e+12 ±  4%  perf-stat.cpu-cycles
      6747 ±  3%    +342.9%      29883 ±  3%  perf-stat.cpu-migrations
      0.29 ±  3%     -50.6%       0.14 ±  5%  perf-stat.dTLB-load-miss-rate
 3.623e+08 ±  7%    +458.2%  2.022e+09 ± 10%  perf-stat.dTLB-load-misses
 1.244e+11 ±  6%   +1028.5%  1.404e+12 ±  6%  perf-stat.dTLB-loads
      0.12 ±  4%     -55.2%       0.05 ±  4%  perf-stat.dTLB-store-miss-rate
 1.303e+08 ±  4%    +133.9%  3.049e+08 ± 10%  perf-stat.dTLB-store-misses
 1.081e+11 ±  8%    +420.4%  5.625e+11 ±  7%  perf-stat.dTLB-stores
     33.41 ±  3%     +24.7%      41.64 ±  0%  perf-stat.iTLB-load-miss-rate
  77510897 ±  4%    +597.7%  5.408e+08 ±  4%  perf-stat.iTLB-load-misses
  1.55e+08 ±  8%    +388.6%  7.575e+08 ±  2%  perf-stat.iTLB-loads
 4.101e+11 ± 17%   +1729.7%  7.503e+12 ±  4%  perf-stat.instructions
      5276 ± 14%    +163.0%      13874 ±  0%  perf-stat.instructions-per-iTLB-miss
      0.29 ±  2%    +199.8%       0.88 ±  1%  perf-stat.ipc
    311496 ±  2%    +252.1%    1096914 ±  3%  perf-stat.minor-faults
     47.30 ±  0%     -90.1%       4.68 ±  6%  perf-stat.node-load-miss-rate
  23272241 ±  6%  +12450.7%  2.921e+09 ±  2%  perf-stat.node-load-misses
  25935066 ±  6%  +2.3e+05%  5.966e+10 ±  4%  perf-stat.node-loads
     42.56 ±  2%     -59.3%      17.31 ± 14%  perf-stat.node-store-miss-rate
  26140830 ± 19%   +4276.4%  1.144e+09 ± 16%  perf-stat.node-store-misses
  35067504 ± 16%  +15454.0%  5.454e+09 ±  3%  perf-stat.node-stores
    311445 ±  2%    +252.2%    1096835 ±  3%  perf-stat.page-faults
      1169 ±  0%   +1453.5%      18164 ±  0%  sched_debug.cfs_rq:/.exec_clock.avg
     13551 ± 15%    +744.7%     114462 ± 14%  sched_debug.cfs_rq:/.exec_clock.max
      5.37 ± 21%   +1029.9%      60.64 ± 14%  sched_debug.cfs_rq:/.exec_clock.min
      2896 ± 12%    +738.1%      24271 ±  6%  sched_debug.cfs_rq:/.exec_clock.stddev
     16586 ± 11%    +349.5%      74549 ±  7%  sched_debug.cfs_rq:/.load.avg
    202788 ±  5%    +340.2%     892596 ±  4%  sched_debug.cfs_rq:/.load.max
     48416 ±  4%    +352.2%     218928 ±  4%  sched_debug.cfs_rq:/.load.stddev
    860.18 ±  0%     -66.2%     290.88 ±  3%  sched_debug.cfs_rq:/.load_avg.avg
    755.12 ±  3%     -73.4%     201.22 ±  1%  sched_debug.cfs_rq:/.load_avg.min
     82.58 ± 56%    +167.9%     221.20 ±  3%  sched_debug.cfs_rq:/.load_avg.stddev
      7448 ± 11%    +311.4%      30641 ±  2%  sched_debug.cfs_rq:/.min_vruntime.avg
     41459 ±  9%    +209.5%     128306 ±  9%  sched_debug.cfs_rq:/.min_vruntime.max
      1341 ± 49%     +91.8%       2574 ± 17%  sched_debug.cfs_rq:/.min_vruntime.min
      8946 ±  5%    +269.7%      33077 ±  2%  sched_debug.cfs_rq:/.min_vruntime.stddev
      0.18 ± 15%     -20.8%       0.15 ±  7%  sched_debug.cfs_rq:/.nr_running.avg
      0.51 ±  1%     +74.1%       0.88 ±  1%  sched_debug.cfs_rq:/.nr_spread_over.avg
      0.50 ±  0%     +75.0%       0.88 ±  0%  sched_debug.cfs_rq:/.nr_spread_over.min
      3.63 ± 16%   +1404.7%      54.62 ±  3%  sched_debug.cfs_rq:/.runnable_load_avg.avg
     81.50 ± 15%    +923.8%     834.44 ±  6%  sched_debug.cfs_rq:/.runnable_load_avg.max
     15.72 ± 16%   +1061.2%     182.51 ±  5%  sched_debug.cfs_rq:/.runnable_load_avg.stddev
     14157 ± 39%    +510.6%      86447 ± 38%  sched_debug.cfs_rq:/.spread0.max
      8949 ±  5%    +269.6%      33077 ±  2%  sched_debug.cfs_rq:/.spread0.stddev
    368.33 ±  0%     -50.5%     182.19 ±  3%  sched_debug.cfs_rq:/.util_avg.avg
    297.75 ±  6%     -73.3%      79.53 ±  2%  sched_debug.cfs_rq:/.util_avg.min
    118.23 ± 12%    +103.2%     240.26 ±  3%  sched_debug.cfs_rq:/.util_avg.stddev
    249308 ±  1%     +32.4%     329978 ±  3%  sched_debug.cpu.avg_idle.stddev
     56195 ±  0%    +333.6%     243688 ±  1%  sched_debug.cpu.clock.avg
     56199 ±  0%    +333.6%     243692 ±  1%  sched_debug.cpu.clock.max
     56191 ±  0%    +333.7%     243684 ±  1%  sched_debug.cpu.clock.min
     56195 ±  0%    +333.6%     243688 ±  1%  sched_debug.cpu.clock_task.avg
     56199 ±  0%    +333.6%     243692 ±  1%  sched_debug.cpu.clock_task.max
     56191 ±  0%    +333.7%     243684 ±  1%  sched_debug.cpu.clock_task.min
      1.28 ± 66%   +2915.2%      38.63 ±  8%  sched_debug.cpu.cpu_load[0].avg
     39.62 ± 85%   +2006.0%     834.50 ±  6%  sched_debug.cpu.cpu_load[0].max
      6.01 ± 79%   +2594.9%     161.99 ±  9%  sched_debug.cpu.cpu_load[0].stddev
      4.36 ±  2%   +1394.4%      65.11 ±  2%  sched_debug.cpu.cpu_load[1].avg
     88.50 ±  4%    +892.7%     878.50 ±  1%  sched_debug.cpu.cpu_load[1].max
     17.04 ±  4%   +1043.5%     194.90 ±  3%  sched_debug.cpu.cpu_load[1].stddev
      4.28 ±  2%   +1437.1%      65.73 ±  1%  sched_debug.cpu.cpu_load[2].avg
     88.25 ±  4%    +891.3%     874.84 ±  2%  sched_debug.cpu.cpu_load[2].max
     16.91 ±  5%   +1053.7%     195.12 ±  2%  sched_debug.cpu.cpu_load[2].stddev
      4.18 ±  1%   +1490.5%      66.56 ±  1%  sched_debug.cpu.cpu_load[3].avg
     87.38 ±  4%    +904.4%     877.56 ±  1%  sched_debug.cpu.cpu_load[3].max
     16.74 ±  6%   +1078.0%     197.15 ±  1%  sched_debug.cpu.cpu_load[3].stddev
      4.09 ±  2%   +1542.5%      67.15 ±  2%  sched_debug.cpu.cpu_load[4].avg
     86.50 ±  3%    +917.3%     880.00 ±  0%  sched_debug.cpu.cpu_load[4].max
     16.45 ±  8%   +1109.1%     198.88 ±  1%  sched_debug.cpu.cpu_load[4].stddev
      1916 ± 16%    +221.0%       6153 ± 11%  sched_debug.cpu.curr->pid.max
    531.59 ± 13%     +90.6%       1013 ±  9%  sched_debug.cpu.curr->pid.stddev
     15879 ±  7%    +376.5%      75669 ±  5%  sched_debug.cpu.load.avg
    202788 ±  5%    +340.0%     892316 ±  4%  sched_debug.cpu.load.max
     47563 ±  4%    +365.6%     221441 ±  4%  sched_debug.cpu.load.stddev
     13106 ±  1%    +482.6%      76364 ±  0%  sched_debug.cpu.nr_load_updates.avg
     25023 ±  8%    +477.3%     144459 ±  8%  sched_debug.cpu.nr_load_updates.max
      8853 ± 11%    +287.4%      34298 ± 42%  sched_debug.cpu.nr_load_updates.min
      3104 ±  8%    +623.9%      22476 ±  6%  sched_debug.cpu.nr_load_updates.stddev
      0.22 ±  7%     -31.6%       0.15 ±  7%  sched_debug.cpu.nr_running.avg
      0.44 ±  8%     -16.6%       0.36 ±  5%  sched_debug.cpu.nr_running.stddev
     49380 ±  0%   +4060.4%    2054420 ±  0%  sched_debug.cpu.nr_switches.avg
    525761 ± 16%   +2710.4%   14776083 ± 30%  sched_debug.cpu.nr_switches.max
    346.12 ±  8%    +224.5%       1123 ± 12%  sched_debug.cpu.nr_switches.min
    113010 ±  9%   +2661.2%    3120489 ± 14%  sched_debug.cpu.nr_switches.stddev
     14.00 ± 50%    +136.6%      33.12 ± 13%  sched_debug.cpu.nr_uninterruptible.max
    -12.62 ±-21%    +380.4%     -60.66 ± -9%  sched_debug.cpu.nr_uninterruptible.min
      4.89 ± 16%    +270.7%      18.12 ±  6%  sched_debug.cpu.nr_uninterruptible.stddev
     52603 ±  0%   +4584.2%    2464026 ±  0%  sched_debug.cpu.sched_count.avg
    528444 ± 16%   +2856.5%   15623669 ± 26%  sched_debug.cpu.sched_count.max
     82.12 ±  8%    +836.6%     769.22 ± 13%  sched_debug.cpu.sched_count.min
    116264 ±  8%   +2828.5%    3404766 ± 15%  sched_debug.cpu.sched_count.stddev
     24230 ±  0%   +4136.7%    1026597 ±  0%  sched_debug.cpu.sched_goidle.avg
    260879 ± 16%   +2731.7%    7387274 ± 30%  sched_debug.cpu.sched_goidle.max
     33.75 ±  6%    +839.6%     317.12 ± 13%  sched_debug.cpu.sched_goidle.min
     56150 ± 10%   +2678.6%    1560212 ± 14%  sched_debug.cpu.sched_goidle.stddev
     24231 ±  0%   +5803.5%    1430488 ±  1%  sched_debug.cpu.ttwu_count.avg
    259165 ± 15%   +3435.2%    9161927 ±  9%  sched_debug.cpu.ttwu_count.max
     43.38 ± 16%    +880.9%     425.47 ±  9%  sched_debug.cpu.ttwu_count.min
     55941 ± 10%   +4327.1%    2476569 ±  3%  sched_debug.cpu.ttwu_count.stddev
     73.83 ±  1%   +1382.0%       1094 ±  2%  sched_debug.cpu.ttwu_local.avg
    570.75 ± 13%    +728.8%       4730 ± 19%  sched_debug.cpu.ttwu_local.max
     20.62 ±  9%    +622.9%     149.09 ±  5%  sched_debug.cpu.ttwu_local.min
     82.38 ±  9%   +1359.8%       1202 ± 11%  sched_debug.cpu.ttwu_local.stddev
     56193 ±  0%    +333.6%     243633 ±  1%  sched_debug.cpu_clk
     55064 ±  0%    +340.4%     242512 ±  1%  sched_debug.ktime
      0.07 ± 37%     -69.8%       0.02 ± 29%  sched_debug.rt_rq:/.rt_time.avg
      2.24 ± 26%     -75.1%       0.56 ± 21%  sched_debug.rt_rq:/.rt_time.max
      0.35 ± 31%     -74.3%       0.09 ± 21%  sched_debug.rt_rq:/.rt_time.stddev
     56193 ±  0%    +333.6%     243633 ±  1%  sched_debug.sched_clk
      2.53 ±  6%     +31.8%       3.34 ±  9%  perf-profile.cycles-pp.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate
      0.00 ± -1%      +Inf%       8.26 ± 13%  perf-profile.cycles-pp.__alloc_pages_nodemask.alloc_pages_current.skb_page_frag_refill.sk_page_frag_refill.tcp_sendmsg
      0.00 ± -1%      +Inf%       7.33 ± 13%  perf-profile.cycles-pp.__alloc_pages_slowpath.__alloc_pages_nodemask.alloc_pages_current.skb_page_frag_refill.sk_page_frag_refill
      1.14 ±  8%     -89.1%       0.12 ±173%  perf-profile.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.local_apic_timer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      6.46 ±  7%     -85.0%       0.97 ± 57%  perf-profile.cycles-pp.__local_bh_enable_ip.ip_finish_output2.ip_finish_output.ip_output.ip_local_out
      5.95 ±  6%     -84.6%       0.92 ± 57%  perf-profile.cycles-pp.__netif_receive_skb.process_backlog.net_rx_action.__softirqentry_text_start.do_softirq_own_stack
      5.93 ±  7%     -84.6%       0.91 ± 57%  perf-profile.cycles-pp.__netif_receive_skb_core.__netif_receive_skb.process_backlog.net_rx_action.__softirqentry_text_start
      1.85 ±  4%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.__release_sock.__sk_flush_backlog.tcp_sendmsg.inet_sendmsg.sock_sendmsg
      7.55 ±  6%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.__release_sock.release_sock.tcp_recvmsg.inet_recvmsg.sock_recvmsg
      0.31 ±100%    +395.9%       1.53 ± 13%  perf-profile.cycles-pp.__schedule.schedule.schedule_preempt_disabled.cpu_startup_entry.start_secondary
      0.00 ± -1%      +Inf%       2.74 ± 11%  perf-profile.cycles-pp.__schedule.schedule.schedule_timeout.kswapd.kthread
      1.88 ±  5%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.__sk_flush_backlog.tcp_sendmsg.inet_sendmsg.sock_sendmsg.sock_write_iter
      6.35 ±  7%     -85.0%       0.96 ± 57%  perf-profile.cycles-pp.__softirqentry_text_start.do_softirq_own_stack.do_softirq.part.16.__local_bh_enable_ip.ip_finish_output2
      1.20 ±  7%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.__tcp_ack_snd_check.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock
      2.08 ±  4%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.__tcp_ack_snd_check.tcp_rcv_established.tcp_v4_do_rcv.tcp_prequeue_process.tcp_recvmsg
      0.32 ±100%    +327.0%       1.34 ± 11%  perf-profile.cycles-pp.__tick_nohz_idle_enter.tick_nohz_idle_enter.cpu_startup_entry.start_secondary
      1.26 ±  3%     -78.4%       0.27 ±100%  perf-profile.cycles-pp.__tick_nohz_idle_enter.tick_nohz_irq_exit.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt
     10.27 ±  3%     -87.2%       1.31 ± 14%  perf-profile.cycles-pp.__vfs_read.vfs_read.sys_read.entry_SYSCALL_64_fastpath
      9.95 ±  6%     -87.2%       1.28 ± 11%  perf-profile.cycles-pp.__vfs_read.vfs_read.sys_read.entry_SYSCALL_64_fastpath.read
     15.00 ±  5%     -58.3%       6.26 ± 13%  perf-profile.cycles-pp.__vfs_write.vfs_write.sys_write.entry_SYSCALL_64_fastpath
     14.59 ±  3%     -57.3%       6.23 ± 13%  perf-profile.cycles-pp.__vfs_write.vfs_write.sys_write.entry_SYSCALL_64_fastpath.write
      0.00 ± -1%      +Inf%       6.02 ± 13%  perf-profile.cycles-pp.__wake_up.wakeup_kswapd.__alloc_pages_slowpath.__alloc_pages_nodemask.alloc_pages_current
      0.00 ± -1%      +Inf%       5.77 ± 13%  perf-profile.cycles-pp.__wake_up_common.__wake_up.wakeup_kswapd.__alloc_pages_slowpath.__alloc_pages_nodemask
      4.02 ±  7%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.__wake_up_common.__wake_up_sync_key.tcp_prequeue.tcp_v4_rcv.ip_local_deliver_finish
      4.18 ±  8%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.__wake_up_sync_key.tcp_prequeue.tcp_v4_rcv.ip_local_deliver_finish.ip_local_deliver
      0.00 ± -1%      +Inf%       1.52 ±  7%  perf-profile.cycles-pp.activate_task.ttwu_do_activate.sched_ttwu_pending.cpu_startup_entry.start_secondary
      3.04 ±  6%     -15.8%       2.56 ± 14%  perf-profile.cycles-pp.activate_task.ttwu_do_activate.try_to_wake_up.default_wake_function.autoremove_wake_function
      0.13 ±173%   +6203.8%       8.35 ± 13%  perf-profile.cycles-pp.alloc_pages_current.skb_page_frag_refill.sk_page_frag_refill.tcp_sendmsg.inet_sendmsg
      4.54 ±  6%     -54.8%       2.06 ±  2%  perf-profile.cycles-pp.apic_timer_interrupt.cpuidle_enter.call_cpuidle.cpu_startup_entry.start_secondary
      0.00 ± -1%      +Inf%       5.43 ± 13%  perf-profile.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up.wakeup_kswapd.__alloc_pages_slowpath
      3.97 ±  7%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_sync_key.tcp_prequeue.tcp_v4_rcv
     43.55 ±  5%     -14.7%      37.13 ±  3%  perf-profile.cycles-pp.call_cpuidle.cpu_startup_entry.start_secondary
      0.00 ± -1%      +Inf%      25.00 ± 12%  perf-profile.cycles-pp.compact_zone.kcompactd_do_work.kcompactd.kthread.ret_from_fork
      5.77 ±  6%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.copy_page_to_iter.skb_copy_datagram_iter.tcp_rcv_established.tcp_v4_do_rcv.__release_sock
      6.25 ±  2%     -75.1%       1.55 ± 13%  perf-profile.cycles-pp.copy_page_to_iter.skb_copy_datagram_iter.tcp_rcv_established.tcp_v4_do_rcv.tcp_prequeue_process
     11.48 ±  3%     -87.3%       1.45 ± 12%  perf-profile.cycles-pp.copy_user_enhanced_fast_string.copy_page_to_iter.skb_copy_datagram_iter.tcp_rcv_established.tcp_v4_do_rcv
     14.72 ±  4%     -91.3%       1.28 ± 33%  perf-profile.cycles-pp.copy_user_enhanced_fast_string.tcp_sendmsg.inet_sendmsg.sock_sendmsg.sock_write_iter
     43.48 ±  5%     -14.8%      37.04 ±  3%  perf-profile.cycles-pp.cpuidle_enter.call_cpuidle.cpu_startup_entry.start_secondary
      0.00 ± -1%      +Inf%       5.40 ± 13%  perf-profile.cycles-pp.default_wake_function.autoremove_wake_function.__wake_up_common.__wake_up.wakeup_kswapd
      3.96 ±  8%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.default_wake_function.autoremove_wake_function.__wake_up_common.__wake_up_sync_key.tcp_prequeue
      6.42 ±  7%     -85.0%       0.96 ± 57%  perf-profile.cycles-pp.do_softirq.part.16.__local_bh_enable_ip.ip_finish_output2.ip_finish_output.ip_output
      6.37 ±  7%     -84.9%       0.96 ± 57%  perf-profile.cycles-pp.do_softirq_own_stack.do_softirq.part.16.__local_bh_enable_ip.ip_finish_output2.ip_finish_output
      2.38 ±  6%     +23.3%       2.94 ±  9%  perf-profile.cycles-pp.dump_trace.save_stack_trace_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair
      0.00 ± -1%      +Inf%       1.44 ±  8%  perf-profile.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.sched_ttwu_pending
      2.93 ±  6%     -20.2%       2.34 ± 15%  perf-profile.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up
      0.00 ± -1%      +Inf%       1.47 ±  8%  perf-profile.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.sched_ttwu_pending.cpu_startup_entry
      2.98 ±  6%     -19.5%       2.40 ± 15%  perf-profile.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up.default_wake_function
     25.58 ±  4%     -70.1%       7.64 ± 13%  perf-profile.cycles-pp.entry_SYSCALL_64_fastpath
     10.14 ±  6%     -87.2%       1.30 ± 11%  perf-profile.cycles-pp.entry_SYSCALL_64_fastpath.read
     14.60 ±  3%     -57.3%       6.23 ± 13%  perf-profile.cycles-pp.entry_SYSCALL_64_fastpath.write
      1.66 ±  9%     -55.6%       0.74 ±  2%  perf-profile.cycles-pp.hrtimer_interrupt.local_apic_timer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter
     19.95 ±  4%     -87.3%       2.54 ± 13%  perf-profile.cycles-pp.inet_recvmsg.sock_recvmsg.sock_read_iter.__vfs_read.vfs_read
     29.58 ±  4%     -57.8%      12.47 ± 13%  perf-profile.cycles-pp.inet_sendmsg.sock_sendmsg.sock_write_iter.__vfs_write.vfs_write
      8.12 ± 10%     -83.6%       1.33 ± 12%  perf-profile.cycles-pp.ip_finish_output.ip_output.ip_local_out.ip_queue_xmit.tcp_transmit_skb
      8.00 ± 10%     -83.5%       1.32 ± 12%  perf-profile.cycles-pp.ip_finish_output2.ip_finish_output.ip_output.ip_local_out.ip_queue_xmit
      5.60 ±  7%     -84.6%       0.86 ± 57%  perf-profile.cycles-pp.ip_local_deliver.ip_rcv_finish.ip_rcv.__netif_receive_skb_core.__netif_receive_skb
      5.55 ±  7%     -84.5%       0.86 ± 57%  perf-profile.cycles-pp.ip_local_deliver_finish.ip_local_deliver.ip_rcv_finish.ip_rcv.__netif_receive_skb_core
      1.34 ±  9%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.ip_local_out.ip_queue_xmit.tcp_transmit_skb.tcp_send_ack.__tcp_ack_snd_check
      7.87 ±  6%     -82.0%       1.42 ± 13%  perf-profile.cycles-pp.ip_local_out.ip_queue_xmit.tcp_transmit_skb.tcp_write_xmit.tcp_push_one
      1.25 ± 10%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.ip_output.ip_local_out.ip_queue_xmit.tcp_transmit_skb.tcp_send_ack
      7.68 ±  6%     -81.8%       1.40 ± 13%  perf-profile.cycles-pp.ip_output.ip_local_out.ip_queue_xmit.tcp_transmit_skb.tcp_write_xmit
      1.57 ± 14%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.ip_queue_xmit.tcp_transmit_skb.tcp_send_ack.__tcp_ack_snd_check.tcp_rcv_established
      8.06 ±  6%     -82.0%       1.45 ± 12%  perf-profile.cycles-pp.ip_queue_xmit.tcp_transmit_skb.tcp_write_xmit.tcp_push_one.tcp_sendmsg
      5.77 ±  7%     -84.7%       0.88 ± 57%  perf-profile.cycles-pp.ip_rcv.__netif_receive_skb_core.__netif_receive_skb.process_backlog.net_rx_action
      5.65 ±  7%     -84.6%       0.87 ± 57%  perf-profile.cycles-pp.ip_rcv_finish.ip_rcv.__netif_receive_skb_core.__netif_receive_skb.process_backlog
      1.60 ±  2%     -59.2%       0.66 ±  6%  perf-profile.cycles-pp.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter.call_cpuidle
      0.00 ± -1%      +Inf%      20.52 ± 11%  perf-profile.cycles-pp.isolate_migratepages_block.compact_zone.kcompactd_do_work.kcompactd.kthread
      0.00 ± -1%      +Inf%      25.02 ± 12%  perf-profile.cycles-pp.kcompactd.kthread.ret_from_fork
      0.00 ± -1%      +Inf%      25.01 ± 12%  perf-profile.cycles-pp.kcompactd_do_work.kcompactd.kthread.ret_from_fork
      0.00 ± -1%      +Inf%      11.06 ± 12%  perf-profile.cycles-pp.kswapd.kthread.ret_from_fork
      0.14 ±173%  +25600.0%      36.62 ±  4%  perf-profile.cycles-pp.kthread.ret_from_fork
      1.76 ±  9%     -55.3%       0.79 ±  1%  perf-profile.cycles-pp.local_apic_timer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter.call_cpuidle
      0.00 ± -1%      +Inf%       3.16 ± 15%  perf-profile.cycles-pp.migrate_pages.compact_zone.kcompactd_do_work.kcompactd.kthread
      6.16 ±  6%     -84.7%       0.94 ± 57%  perf-profile.cycles-pp.net_rx_action.__softirqentry_text_start.do_softirq_own_stack.do_softirq.part.16.__local_bh_enable_ip
      0.00 ± -1%      +Inf%       3.93 ± 12%  perf-profile.cycles-pp.pgdat_balanced.kswapd.kthread.ret_from_fork
      0.00 ± -1%      +Inf%       1.04 ± 12%  perf-profile.cycles-pp.prepare_to_wait.kswapd.kthread.ret_from_fork
      1.99 ±  6%     +23.8%       2.47 ±  9%  perf-profile.cycles-pp.print_context_stack.dump_trace.save_stack_trace_tsk.__account_scheduler_latency.enqueue_entity
      6.01 ±  7%     -84.6%       0.93 ± 57%  perf-profile.cycles-pp.process_backlog.net_rx_action.__softirqentry_text_start.do_softirq_own_stack.do_softirq.part.16
     10.24 ±  6%     -87.1%       1.32 ± 11%  perf-profile.cycles-pp.read
      7.62 ±  6%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.release_sock.tcp_recvmsg.inet_recvmsg.sock_recvmsg.sock_read_iter
      0.14 ±173%  +25600.0%      36.62 ±  4%  perf-profile.cycles-pp.ret_from_fork
      2.42 ±  6%     +21.9%       2.94 ±  9%  perf-profile.cycles-pp.save_stack_trace_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.activate_task
      0.00 ± -1%      +Inf%       1.81 ±  7%  perf-profile.cycles-pp.sched_ttwu_pending.cpu_startup_entry.start_secondary
      0.34 ±100%    +381.9%       1.66 ± 13%  perf-profile.cycles-pp.schedule.schedule_preempt_disabled.cpu_startup_entry.start_secondary
      0.00 ± -1%      +Inf%       3.13 ± 11%  perf-profile.cycles-pp.schedule.schedule_timeout.kswapd.kthread.ret_from_fork
      0.96 ± 19%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.schedule.schedule_timeout.sk_wait_data.tcp_recvmsg.inet_recvmsg
      0.62 ± 16%    +174.6%       1.70 ± 13%  perf-profile.cycles-pp.schedule_preempt_disabled.cpu_startup_entry.start_secondary
      0.00 ± -1%      +Inf%       3.91 ± 10%  perf-profile.cycles-pp.schedule_timeout.kswapd.kthread.ret_from_fork
      1.01 ± 20%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.schedule_timeout.sk_wait_data.tcp_recvmsg.inet_recvmsg.sock_recvmsg
      0.40 ± 58%   +2000.6%       8.46 ± 13%  perf-profile.cycles-pp.sk_page_frag_refill.tcp_sendmsg.inet_sendmsg.sock_sendmsg.sock_write_iter
      1.81 ±  6%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.sk_wait_data.tcp_recvmsg.inet_recvmsg.sock_recvmsg.sock_read_iter
      5.99 ±  6%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.skb_copy_datagram_iter.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock
      6.52 ±  2%     -75.1%       1.62 ± 13%  perf-profile.cycles-pp.skb_copy_datagram_iter.tcp_rcv_established.tcp_v4_do_rcv.tcp_prequeue_process.tcp_recvmsg
      0.28 ±100%   +2969.1%       8.44 ± 13%  perf-profile.cycles-pp.skb_page_frag_refill.sk_page_frag_refill.tcp_sendmsg.inet_sendmsg.sock_sendmsg
      4.40 ±  6%     -54.3%       2.01 ±  2%  perf-profile.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter.call_cpuidle.cpu_startup_entry
     20.12 ±  4%     -87.3%       2.56 ± 13%  perf-profile.cycles-pp.sock_read_iter.__vfs_read.vfs_read.sys_read.entry_SYSCALL_64_fastpath
     20.06 ±  4%     -87.2%       2.56 ± 13%  perf-profile.cycles-pp.sock_recvmsg.sock_read_iter.__vfs_read.vfs_read.sys_read
     29.59 ±  4%     -57.8%      12.47 ± 13%  perf-profile.cycles-pp.sock_sendmsg.sock_write_iter.__vfs_write.vfs_write.sys_write
     29.59 ±  4%     -57.8%      12.47 ± 13%  perf-profile.cycles-pp.sock_write_iter.__vfs_write.vfs_write.sys_write.entry_SYSCALL_64_fastpath
     10.46 ±  3%     -87.2%       1.33 ± 14%  perf-profile.cycles-pp.sys_read.entry_SYSCALL_64_fastpath
     10.12 ±  6%     -87.2%       1.30 ± 11%  perf-profile.cycles-pp.sys_read.entry_SYSCALL_64_fastpath.read
     15.01 ±  5%     -58.3%       6.26 ± 13%  perf-profile.cycles-pp.sys_write.entry_SYSCALL_64_fastpath
     14.60 ±  3%     -57.3%       6.23 ± 13%  perf-profile.cycles-pp.sys_write.entry_SYSCALL_64_fastpath.write
      1.41 ±  4%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.__sk_flush_backlog
      1.15 ±  4%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.__release_sock
      4.38 ±  8%     -94.1%       0.26 ±100%  perf-profile.cycles-pp.tcp_prequeue.tcp_v4_rcv.ip_local_deliver_finish.ip_local_deliver.ip_rcv_finish
      9.34 ±  1%     -76.4%       2.21 ± 12%  perf-profile.cycles-pp.tcp_prequeue_process.tcp_recvmsg.inet_recvmsg.sock_recvmsg.sock_read_iter
      9.39 ±  5%     -82.8%       1.62 ± 12%  perf-profile.cycles-pp.tcp_push_one.tcp_sendmsg.inet_sendmsg.sock_sendmsg.sock_write_iter
      1.70 ±  5%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.__sk_flush_backlog.tcp_sendmsg
      7.39 ±  6%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock.tcp_recvmsg
      9.01 ±  1%     -76.4%       2.13 ± 12%  perf-profile.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.tcp_prequeue_process.tcp_recvmsg.inet_recvmsg
     19.87 ±  4%     -87.3%       2.52 ± 13%  perf-profile.cycles-pp.tcp_recvmsg.inet_recvmsg.sock_recvmsg.sock_read_iter.__vfs_read
      0.89 ± 35%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.tcp_send_ack.__tcp_ack_snd_check.tcp_rcv_established.tcp_v4_do_rcv.__release_sock
      1.84 ±  3%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.tcp_send_ack.__tcp_ack_snd_check.tcp_rcv_established.tcp_v4_do_rcv.tcp_prequeue_process
     29.44 ±  4%     -57.8%      12.44 ± 13%  perf-profile.cycles-pp.tcp_sendmsg.inet_sendmsg.sock_sendmsg.sock_write_iter.__vfs_write
      2.05 ± 20%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.tcp_transmit_skb.tcp_send_ack.__tcp_ack_snd_check.tcp_rcv_established.tcp_v4_do_rcv
      8.65 ±  5%     -82.3%       1.53 ± 12%  perf-profile.cycles-pp.tcp_transmit_skb.tcp_write_xmit.tcp_push_one.tcp_sendmsg.inet_sendmsg
      1.73 ±  5%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.tcp_v4_do_rcv.__release_sock.__sk_flush_backlog.tcp_sendmsg.inet_sendmsg
      7.43 ±  6%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.tcp_v4_do_rcv.__release_sock.release_sock.tcp_recvmsg.inet_recvmsg
      9.19 ±  1%     -76.4%       2.17 ± 13%  perf-profile.cycles-pp.tcp_v4_do_rcv.tcp_prequeue_process.tcp_recvmsg.inet_recvmsg.sock_recvmsg
      5.14 ±  8%     -83.9%       0.83 ± 57%  perf-profile.cycles-pp.tcp_v4_rcv.ip_local_deliver_finish.ip_local_deliver.ip_rcv_finish.ip_rcv
      9.37 ±  5%     -82.9%       1.60 ± 12%  perf-profile.cycles-pp.tcp_write_xmit.tcp_push_one.tcp_sendmsg.inet_sendmsg.sock_sendmsg
      0.33 ±100%    +329.0%       1.41 ± 12%  perf-profile.cycles-pp.tick_nohz_idle_enter.cpu_startup_entry.start_secondary
      0.00 ± -1%      +Inf%       0.89 ± 10%  perf-profile.cycles-pp.tick_nohz_idle_exit.cpu_startup_entry.start_secondary
      1.31 ±  3%     -69.0%       0.41 ± 58%  perf-profile.cycles-pp.tick_nohz_irq_exit.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter
      0.26 ±100%    +354.4%       1.17 ± 11%  perf-profile.cycles-pp.tick_nohz_stop_sched_tick.__tick_nohz_idle_enter.tick_nohz_idle_enter.cpu_startup_entry.start_secondary
      1.03 ±  3%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.tick_nohz_stop_sched_tick.__tick_nohz_idle_enter.tick_nohz_irq_exit.irq_exit.smp_apic_timer_interrupt
      0.00 ± -1%      +Inf%       5.21 ± 13%  perf-profile.cycles-pp.try_to_wake_up.default_wake_function.autoremove_wake_function.__wake_up_common.__wake_up
      3.90 ±  7%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.try_to_wake_up.default_wake_function.autoremove_wake_function.__wake_up_common.__wake_up_sync_key
      0.00 ± -1%      +Inf%       1.64 ±  7%  perf-profile.cycles-pp.ttwu_do_activate.sched_ttwu_pending.cpu_startup_entry.start_secondary
     10.42 ±  3%     -87.2%       1.33 ± 14%  perf-profile.cycles-pp.vfs_read.sys_read.entry_SYSCALL_64_fastpath
     10.08 ±  6%     -87.2%       1.30 ± 11%  perf-profile.cycles-pp.vfs_read.sys_read.entry_SYSCALL_64_fastpath.read
     15.01 ±  5%     -58.3%       6.26 ± 13%  perf-profile.cycles-pp.vfs_write.sys_write.entry_SYSCALL_64_fastpath
     14.60 ±  3%     -57.3%       6.23 ± 13%  perf-profile.cycles-pp.vfs_write.sys_write.entry_SYSCALL_64_fastpath.write
      0.00 ± -1%      +Inf%       6.67 ± 13%  perf-profile.cycles-pp.wakeup_kswapd.__alloc_pages_slowpath.__alloc_pages_nodemask.alloc_pages_current.skb_page_frag_refill
     14.60 ±  3%     -57.3%       6.23 ± 13%  perf-profile.cycles-pp.write
      0.00 ± -1%      +Inf%       3.78 ± 11%  perf-profile.cycles-pp.zone_reclaimable.pgdat_balanced.kswapd.kthread.ret_from_fork
      0.00 ± -1%      +Inf%       2.55 ±  8%  perf-profile.cycles-pp.zone_reclaimable_pages.zone_reclaimable.pgdat_balanced.kswapd.kthread
      0.65 ±  9%     +43.8%       0.93 ±  8%  perf-profile.func.cycles-pp.__kernel_text_address
      0.34 ± 13%    +435.3%       1.82 ± 12%  perf-profile.func.cycles-pp.__schedule
      0.10 ± 31%   +1000.0%       1.05 ± 10%  perf-profile.func.cycles-pp.__zone_watermark_ok
      0.17 ± 28%    +637.3%       1.23 ±  9%  perf-profile.func.cycles-pp._find_next_bit.part.0
      0.49 ± 10%    +276.8%       1.86 ±  6%  perf-profile.func.cycles-pp._raw_spin_lock_irqsave
     27.29 ±  4%     -89.3%       2.91 ± 13%  perf-profile.func.cycles-pp.copy_user_enhanced_fast_string
      0.00 ± -1%      +Inf%      20.19 ± 11%  perf-profile.func.cycles-pp.isolate_migratepages_block
      0.00 ± -1%      +Inf%       1.44 ± 14%  perf-profile.func.cycles-pp.kswapd
      0.00 ± -1%      +Inf%       1.69 ±  9%  perf-profile.func.cycles-pp.zone_reclaimable_pages




                             netpipe.bigger_5M_Mbps.avg

  45000 ++------------------------------------------------------------------+
        |     *.   *.*.**.**.      .*         .*                        .*  |
  40000 *+**.*  *.*          *.**.*  *.**.*.**  *.*.**.*.**.**.*.**.*.**  *.*
  35000 ++                                                                  |
        |                                                                   |
  30000 ++                                                                  |
        |                                                                   |
  25000 ++                                                                  |
        | OO O  O  O O  O O  O  O                                           |
  20000 O+    O   O    O   O   O  O                                         |
  15000 ++                                                                  |
        |                                                                   |
  10000 ++                                                                  |
        |                           OO OO O OO O                            |
   5000 ++------------------------------------------------------------------+


	[*] bisect-good sample
	[O] bisect-bad  sample




Thanks,
Xiaolong

View attachment "config-4.7.0-05542-g9863a93" of type "text/plain" (152075 bytes)

View attachment "job.yaml" of type "text/plain" (3852 bytes)

View attachment "reproduce" of type "text/plain" (103 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ