lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Message-ID: <ed06ccf4ddb98ec9c31d30d5f92f637bdb890326.camel@surriel.com>
Date:   Wed, 26 May 2021 22:00:55 -0400
From:   Rik van Riel <riel@...riel.com>
To:     "Xing, Zhengjun" <zhengjun.xing@...el.com>,
        kernel test robot <oliver.sang@...el.com>
Cc:     Peter Zijlstra <peterz@...radead.org>,
        Mel Gorman <mgorman@...hsingularity.net>,
        LKML <linux-kernel@...r.kernel.org>, lkp@...ts.01.org,
        lkp@...el.com, aubrey.li@...ux.intel.com, yu.c.chen@...el.com
Subject: Re: [LKP] [sched/fair] c722f35b51: tbench.throughput-MB/sec -29.1%
 regression

On Thu, 2021-05-27 at 09:27 +0800, Xing, Zhengjun wrote:
> Hi Rik,
> 
>    Do you have time to look at this? Thanks.

Hello,

I will try to take a look at this on Friday.

However, even if I manage to reproduce it on one of
the systems I have access to, I'm still not sure how
exactly we would root cause the issue.

Is it due to
select_idle_sibling() doing a little bit
more work?

Is it because we invoke test_idle_cores() a little
earlier, widening the race window with CPUs going idle,
causing select_idle_cpu to do a lot more work?

Is it a locality thing where random placement on any
core in the LLC is somehow better than placement on
the same core as "prev" when there is no idle core?

Is it tbench running
faster when the woken up task is
placed on the runqueue behind the current task on the
"target" cpu, even though that CPU isn't currently
idle, because tbench happens to go to sleep fast?

In other words, I'm
not quite sure whether this is
a tbench (and other similar benchmark) specific thing,
or a kernel thing, or what instrumentation we would
want in select_idle_sibling / select_idle_cpu for us
to root cause issues like this more easily in the
future...

> On 5/19/2021 4:22 PM, kernel test robot wrote:
> > Greeting,
> > 
> > FYI, we noticed a -29.1% regression of tbench.throughput-MB/sec due
> > to commit:
> > 
> > 
> > commit: c722f35b513f807629603bbf24640b1a48be21b5 ("sched/fair:
> > Bring back select_idle_smt(), but differently")
> > https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git
> > master
> > 
> > 
> > in testcase: tbench
> > on test machine: 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252
> > CPU @ 2.10GHz with 128G memory
> > with following parameters:
> > 
> > 	nr_threads: 100%
> > 	cluster: cs-localhost
> > 	cpufreq_governor: performance
> > 	ucode: 0x5003006
> > 
> > 
> > 
> > 
> > If you fix the issue, kindly add following tag
> > Reported-by: kernel test robot <oliver.sang@...el.com>
> > 
> > 
> > Details are as below:
> > -----------------------------------------------------------------
> > --------------------------------->
> > 
> > 
> > To reproduce:
> > 
> >         git clone https://github.com/intel/lkp-tests.git
> >         cd lkp-tests
> >         bin/lkp install                job.yaml  # job file is
> > attached in this email
> >         bin/lkp split-job --compatible job.yaml  # generate the
> > yaml file for lkp run
> >         bin/lkp run                    generated-yaml-file
> > 
> > ===================================================================
> > ======================
> > cluster/compiler/cpufreq_governor/kconfig/nr_threads/rootfs/tbox_gr
> > oup/testcase/ucode:
> >   cs-localhost/gcc-9/performance/x86_64-rhel-8.3/100%/debian-10.4-
> > x86_64-20200603.cgz/lkp-csl-2sp8/tbench/0x5003006
> > 
> > commit: 
> >   6db12ee045 ("psi: allow unprivileged users with CAP_SYS_RESOURCE
> > to write psi files")
> >   c722f35b51 ("sched/fair: Bring back select_idle_smt(), but
> > differently")
> > 
> > 6db12ee0456d0e36 c722f35b513f807629603bbf246 
> > ---------------- --------------------------- 
> >          %stddev     %change         %stddev
> >              \          |                \  
> >      15027           -29.1%      10651 ±  5%  tbench.throughput-
> > MB/sec
> >  1.893e+09 ±  2%     -87.5%   2.36e+08 ±
> > 73%  tbench.time.involuntary_context_switches
> >      51693 ± 21%     +65.7%      85645
> > ±  7%  tbench.time.minor_page_faults
> >       5200           -45.9%       2815
> > ±  9%  tbench.time.percent_of_cpu_this_job_got
> >      32467           -48.9%      16595 ±
> > 10%  tbench.time.system_time
> >       4990           -26.1%       3686 ±  6%  tbench.time.user_time
> >  1.387e+08 ± 25%    +760.7%  1.194e+09
> > ±  8%  tbench.time.voluntary_context_switches
> >       3954 ±  8%    +307.9%      16131 ±  8%  uptime.idle
> >   93556034 ±  4%     -24.5%   70597092 ±  5%  numa-
> > numastat.node0.local_node
> >   93594173 ±  4%     -24.5%   70643519 ±  5%  numa-
> > numastat.node0.numa_hit
> >  1.011e+08 ±  2%     -29.7%   71069877 ±  6%  numa-
> > numastat.node1.local_node
> >  1.012e+08 ±  2%     -29.7%   71110181 ±  6%  numa-
> > numastat.node1.numa_hit
> >   47900004 ±  3%     -21.4%   37635451 ±  6%  numa-
> > vmstat.node0.numa_hit
> >   47817433 ±  3%     -21.5%   37543255 ±  6%  numa-
> > vmstat.node0.numa_local
> >   51712415 ±  2%     -26.3%   38110581 ±  8%  numa-
> > vmstat.node1.numa_hit
> >   51561304 ±  2%     -26.4%   37969236 ±  8%  numa-
> > vmstat.node1.numa_local
> >       2.17 ± 31%    +861.5%      20.83 ±  9%  vmstat.cpu.id
> >      88.67           -19.5%      71.33 ±  2%  vmstat.cpu.sy
> >     132.50           -16.2%     111.00        vmstat.procs.r
> >     197679            +6.4%     210333        vmstat.system.in
> >  1.445e+09 ± 24%    +838.9%  1.357e+10 ± 10%  cpuidle.C1.time
> >  1.585e+08 ± 24%    +803.8%  1.433e+09 ±  8%  cpuidle.C1.usage
> >     238575 ± 17%    +101.0%     479497 ±  9%  cpuidle.C1E.usage
> >   16003682 ± 25%    +636.0%  1.178e+08 ±  7%  cpuidle.POLL.time
> >    1428997 ± 25%    +572.1%    9604232 ±  8%  cpuidle.POLL.usage
> >      22.00           +14.7       36.65 ±  4%  mpstat.cpu.all.idle%
> >       0.81            +0.1        0.90        mpstat.cpu.all.irq%
> >      15.57            +4.6       20.20 ±  2%  mpstat.cpu.all.soft%
> >      54.87           -18.2       36.68 ±  5%  mpstat.cpu.all.sys%
> >       6.75            -1.2        5.57 ±  3%  mpstat.cpu.all.usr%
> >  1.948e+08           -27.3%  1.417e+08 ±  4%  proc-vmstat.numa_hit
> >  1.947e+08           -27.3%  1.416e+08 ±  4%  proc-
> > vmstat.numa_local
> >  1.516e+09           -28.0%  1.092e+09 ±  5%  proc-
> > vmstat.pgalloc_normal
> >    2867654 ±  2%      +5.3%    3018753        proc-vmstat.pgfault
> >  1.512e+09           -28.1%  1.087e+09 ±  5%  proc-vmstat.pgfree
> >       2663           -15.7%       2244        turbostat.Avg_MHz
> >      97.59           -15.8       81.80 ±  2%  turbostat.Busy%
> >  1.585e+08 ± 24%    +803.8%  1.433e+09 ±  8%  turbostat.C1
> >       2.09 ± 24%     +17.5       19.58 ± 10%  turbostat.C1%
> >     237770 ± 17%    +101.5%     479071 ±  9%  turbostat.C1E
> >       2.38 ± 17%    +663.9%      18.18 ± 10%  turbostat.CPU%c1
> >      74.39            -6.2%      69.77        turbostat.RAMWatt
> >       0.01 ± 59%    +886.2%       0.10 ± 48%  perf-
> > sched.sch_delay.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_us
> > er_mode.entry_SYSCALL_64_after_hwframe.[unknown]
> >       0.00 ± 62%   +2215.4%       0.05 ± 40%  perf-
> > sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.__rel
> > ease_sock.release_sock.tcp_sendmsg
> >       0.01 ± 24%    +378.9%       0.05 ± 22%  perf-
> > sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.kmem_
> > cache_alloc_node.__alloc_skb.sk_stream_alloc_skb
> >       0.05 ± 24%     +47.3%       0.07 ± 12%  perf-
> > sched.sch_delay.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do
> > _open.isra
> >       2347 ± 20%     +28.2%       3010        perf-
> > sched.sch_delay.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
> >       1.41 ± 25%     -34.8%       0.92 ± 19%  perf-
> > sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.aa_sk
> > _perm.security_socket_sendmsg.sock_sendmsg
> >     790.43 ± 10%     +30.2%       1029        perf-
> > sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
> >     215752 ± 29%     -94.4%      12122 ± 46%  perf-
> > sched.wait_and_delay.count.exit_to_user_mode_prepare.syscall_exit_t
> > o_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
> >       2004 ± 12%     -37.7%       1248 ±  6%  perf-
> > sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork
> >       0.06 ± 23%    +762.8%       0.53 ± 42%  perf-
> > sched.wait_time.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_us
> > er_mode.entry_SYSCALL_64_after_hwframe.[unknown]
> >       0.02 ±  8%    +421.8%       0.10 ± 32%  perf-
> > sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.__rel
> > ease_sock.release_sock.tcp_sendmsg
> >       0.03 ± 12%    +191.3%       0.09 ± 18%  perf-
> > sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.kmem_
> > cache_alloc_node.__alloc_skb.sk_stream_alloc_skb
> >     685.22 ±  7%     +17.4%     804.65 ± 12%  perf-
> > sched.wait_time.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_
> > fork
> >     785.50 ± 10%     +30.3%       1023        perf-
> > sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
> >       1.18 ± 35%     -33.6%       0.78 ± 18%  perf-
> > sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_u
> > ser_mode.asm_sysvec_apic_timer_interrupt.[unknown]
> >       1.58 ± 15%     -34.9%       1.03 ± 14%  perf-
> > sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.aa_sk
> > _perm.security_socket_sendmsg.sock_sendmsg
> >      51178 ± 93%   +1451.5%     794049
> > ±  9%  sched_debug.cfs_rq:/.MIN_vruntime.avg
> >    2407970 ± 59%    +466.9%   13650279
> > ±  4%  sched_debug.cfs_rq:/.MIN_vruntime.max
> >     330425 ± 71%    +821.3%    3044331
> > ±  6%  sched_debug.cfs_rq:/.MIN_vruntime.stddev
> >       6186 ±  5%     -91.9%     499.13
> > ±165%  sched_debug.cfs_rq:/.load.min
> >       4.01 ±  7%     -39.0%       2.45 ±
> > 15%  sched_debug.cfs_rq:/.load_avg.min
> >      51178 ± 93%   +1451.5%     794049
> > ±  9%  sched_debug.cfs_rq:/.max_vruntime.avg
> >    2407973 ± 59%    +466.9%   13650279
> > ±  4%  sched_debug.cfs_rq:/.max_vruntime.max
> >     330425 ± 71%    +821.3%    3044331
> > ±  6%  sched_debug.cfs_rq:/.max_vruntime.stddev
> >   23445823           -41.9%   13622977 ±
> > 11%  sched_debug.cfs_rq:/.min_vruntime.avg
> >   25062917 ±  2%     -32.7%   16866064
> > ±  8%  sched_debug.cfs_rq:/.min_vruntime.max
> >   22902926 ±  2%     -42.8%   13102629 ±
> > 11%  sched_debug.cfs_rq:/.min_vruntime.min
> >       0.92           -19.0%       0.75
> > ±  3%  sched_debug.cfs_rq:/.nr_running.avg
> >       0.74 ± 12%     -89.7%       0.08
> > ±115%  sched_debug.cfs_rq:/.nr_running.min
> >       0.07 ± 32%    +432.3%       0.38 ±
> > 10%  sched_debug.cfs_rq:/.nr_running.stddev
> >       1096 ±  2%     -
> > 18.1%     898.07        sched_debug.cfs_rq:/.runnable_avg.avg
> >     203.00 ±  4%      +8.1%     219.36
> > ±  2%  sched_debug.cfs_rq:/.runnable_avg.stddev
> >     996603 ± 48%    +225.1%    3240056 ±
> > 25%  sched_debug.cfs_rq:/.spread0.max
> >     855.14           -35.7%     549.90
> > ±  6%  sched_debug.cfs_rq:/.util_avg.avg
> >       1230 ±  6%     -21.1%     971.21
> > ±  6%  sched_debug.cfs_rq:/.util_avg.max
> >     142.21 ±  3%     -38.9%      86.89 ±
> > 17%  sched_debug.cfs_rq:/.util_avg.stddev
> >     630.23 ±  5%     -86.6%      84.51 ±
> > 65%  sched_debug.cfs_rq:/.util_est_enqueued.avg
> >       1148 ±  5%     -57.8%     484.29 ±
> > 17%  sched_debug.cfs_rq:/.util_est_enqueued.max
> >     162.67 ± 21%     -92.9%      11.56
> > ±138%  sched_debug.cfs_rq:/.util_est_enqueued.min
> >     189.55 ±  5%     -60.4%      75.05 ±
> > 31%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
> >      13.27 ±  7%     -56.2%       5.81 ±
> > 16%  sched_debug.cpu.clock.stddev
> >       2627           -22.2%       2042 ±  3%  sched_debug.cpu.curr-
> > >pid.avg
> >       1954 ± 10%     -92.0%     157.14 ±137%  sched_debug.cpu.curr-
> > >pid.min
> >     963.90 ±  6%     +58.2%       1525 ±  4%  sched_debug.cpu.curr-
> > >pid.stddev
> >       0.00 ±  7%     -47.7%       0.00 ±
> > 15%  sched_debug.cpu.next_balance.stddev
> >       1.30           -14.2%       1.12
> > ±  2%  sched_debug.cpu.nr_running.avg
> >       2.50 ±  6%     +79.0%       4.47
> > ±  6%  sched_debug.cpu.nr_running.max
> >       0.76 ± 10%     -89.8%       0.08
> > ±115%  sched_debug.cpu.nr_running.min
> >       0.51 ±  3%     +73.6%       0.89
> > ±  6%  sched_debug.cpu.nr_running.stddev
> >       8.53 ±  3%    +154.8%      21.73 ±  6%  perf-stat.i.MPKI
> >  2.178e+10           -13.8%  1.878e+10 ±  3%  perf-stat.i.branch-
> > instructions
> >       1.69            +0.2        1.90        perf-stat.i.branch-
> > miss-rate%
> >  3.675e+08            -3.5%  3.545e+08        perf-stat.i.branch-
> > misses
> >      18.30 ±  7%     -13.5        4.80 ± 36%  perf-stat.i.cache-
> > miss-rate%
> >  1.546e+08 ±  9%     -48.7%   79255656 ± 18%  perf-stat.i.cache-
> > misses
> >  9.224e+08 ±  2%    +113.6%  1.971e+09 ±  4%  perf-stat.i.cache-
> > references
> >  2.553e+11           -14.5%  2.183e+11        perf-stat.i.cpu-
> > cycles
> >     240694 ± 23%    +737.9%    2016725 ±  7%  perf-stat.i.cpu-
> > migrations
> >       1700 ± 10%     +78.2%       3029 ± 11%  perf-stat.i.cycles-
> > between-cache-misses
> >       0.02 ± 19%      +0.2        0.18 ±  7%  perf-stat.i.dTLB-
> > load-miss-rate%
> >    6572698 ± 19%    +582.9%   44886023 ±  7%  perf-stat.i.dTLB-
> > load-misses
> >  3.024e+10           -14.7%   2.58e+10 ±  3%  perf-stat.i.dTLB-
> > loads
> >       0.01 ± 16%      +0.1        0.06 ±  8%  perf-stat.i.dTLB-
> > store-miss-rate%
> >    1539288 ± 16%    +464.4%    8687418 ±  8%  perf-stat.i.dTLB-
> > store-misses
> >  1.709e+10           -14.7%  1.457e+10 ±  3%  perf-stat.i.dTLB-
> > stores
> >      73.36           -12.9       60.46 ±  2%  perf-stat.i.iTLB-
> > load-miss-rate%
> >  1.847e+08 ±  2%     -10.9%  1.645e+08 ±  2%  perf-stat.i.iTLB-
> > load-misses
> >   67816199 ±  2%     +59.3%   1.08e+08 ±  3%  perf-stat.i.iTLB-
> > loads
> >  1.073e+11           -13.7%  9.264e+10 ±  3%  perf-
> > stat.i.instructions
> >       2.66           -14.5%       2.27        perf-
> > stat.i.metric.GHz
> >       1188 ±  4%     -48.1%     616.51 ± 13%  perf-
> > stat.i.metric.K/sec
> >     729.51           -12.6%     637.79 ±  2%  perf-
> > stat.i.metric.M/sec
> >       3886 ±  2%      +5.4%       4097        perf-stat.i.minor-
> > faults
> >   31516592 ±  8%     -29.4%   22240687 ± 13%  perf-stat.i.node-
> > load-misses
> >    5444441 ±  5%     -16.3%    4559306 ±  4%  perf-stat.i.node-
> > loads
> >      96.59            -4.9       91.73        perf-stat.i.node-
> > store-miss-rate%
> >     432854 ±  9%     +96.5%     850464 ± 17%  perf-stat.i.node-
> > stores
> >       3887 ±  2%      +5.4%       4098        perf-stat.i.page-
> > faults
> >       8.60 ±  3%    +147.7%      21.30 ±  6%  perf-
> > stat.overall.MPKI
> >       1.69            +0.2        1.89        perf-
> > stat.overall.branch-miss-rate%
> >      16.76 ±  9%     -12.7        4.07 ± 23%  perf-
> > stat.overall.cache-miss-rate%
> >       1666 ±  9%     +69.9%       2830 ± 14%  perf-
> > stat.overall.cycles-between-cache-misses
> >       0.02 ± 18%      +0.2        0.17 ±  8%  perf-
> > stat.overall.dTLB-load-miss-rate%
> >       0.01 ± 16%      +0.1        0.06 ±  9%  perf-
> > stat.overall.dTLB-store-miss-rate%
> >      73.13           -12.8       60.37 ±  2%  perf-
> > stat.overall.iTLB-load-miss-rate%
> >      95.70            -4.9       90.84        perf-
> > stat.overall.node-store-miss-rate%
> >  2.175e+10           -13.7%  1.876e+10 ±  3%  perf-stat.ps.branch-
> > instructions
> >   3.67e+08            -3.5%   3.54e+08        perf-stat.ps.branch-
> > misses
> >  1.543e+08 ±  9%     -48.7%   79211813 ± 18%  perf-stat.ps.cache-
> > misses
> >  9.214e+08 ±  2%    +113.5%  1.967e+09 ±  4%  perf-stat.ps.cache-
> > references
> >  2.549e+11           -14.5%   2.18e+11        perf-stat.ps.cpu-
> > cycles
> >     240897 ± 23%    +735.5%    2012779 ±  7%  perf-stat.ps.cpu-
> > migrations
> >    6575045 ± 19%    +581.3%   44797852 ±  7%  perf-stat.ps.dTLB-
> > load-misses
> >   3.02e+10           -14.7%  2.577e+10 ±  3%  perf-stat.ps.dTLB-
> > loads
> >    1539214 ± 16%    +463.3%    8670426 ±  8%  perf-stat.ps.dTLB-
> > store-misses
> >  1.707e+10           -14.7%  1.456e+10 ±  3%  perf-stat.ps.dTLB-
> > stores
> >  1.844e+08 ±  2%     -10.9%  1.643e+08 ±  2%  perf-stat.ps.iTLB-
> > load-misses
> >   67743849 ±  2%     +59.2%  1.079e+08 ±  3%  perf-stat.ps.iTLB-
> > loads
> >  1.072e+11           -13.7%  9.253e+10 ±  3%  perf-
> > stat.ps.instructions
> >       3886 ±  2%      +5.5%       4098        perf-stat.ps.minor-
> > faults
> >   31470314 ±  8%     -29.4%   22222434 ± 13%  perf-stat.ps.node-
> > load-misses
> >    5442735 ±  5%     -16.2%    4560730 ±  3%  perf-stat.ps.node-
> > loads
> >     432399 ±  9%     +96.4%     849120 ± 17%  perf-stat.ps.node-
> > stores
> >       3886 ±  2%      +5.5%       4098        perf-stat.ps.page-
> > faults
> >   7.73e+13           -13.7%  6.672e+13 ±  3%  perf-
> > stat.total.instructions
> >       0.00       +4.1e+103%      40.83 ±117%  interrupts.102:PCI-
> > MSI.31981635-edge.i40e-eth0-TxRx-66
> >     163522 ±  5%    +659.4%    1241719 ±
> > 40%  interrupts.CAL:Function_call_interrupts
> >       2717 ± 28%    +275.9%      10215 ±
> > 52%  interrupts.CPU0.CAL:Function_call_interrupts
> >      30618 ± 41%    +290.0%     119405
> > ±  9%  interrupts.CPU0.RES:Rescheduling_interrupts
> >       1686 ± 25%    +483.4%       9836 ±
> > 44%  interrupts.CPU1.CAL:Function_call_interrupts
> >      26633 ± 50%    +342.3%     117798 ±
> > 10%  interrupts.CPU1.RES:Rescheduling_interrupts
> >       1633 ± 25%    +563.1%      10833 ±
> > 50%  interrupts.CPU10.CAL:Function_call_interrupts
> >      27398 ± 51%    +319.3%     114895 ±
> > 10%  interrupts.CPU10.RES:Rescheduling_interrupts
> >       1857 ± 21%    +446.8%      10154 ±
> > 50%  interrupts.CPU11.CAL:Function_call_interrupts
> >      26206 ± 53%    +312.9%     108207
> > ±  8%  interrupts.CPU11.RES:Rescheduling_interrupts
> >       1423 ± 15%    +614.7%      10176 ±
> > 52%  interrupts.CPU12.CAL:Function_call_interrupts
> >      26880 ± 58%    +322.7%     113610
> > ±  8%  interrupts.CPU12.RES:Rescheduling_interrupts
> >       1457 ± 18%    +585.6%       9988 ±
> > 52%  interrupts.CPU13.CAL:Function_call_interrupts
> >      26741 ± 57%    +330.3%     115064 ±
> > 10%  interrupts.CPU13.RES:Rescheduling_interrupts
> >       1440 ±  8%    +626.7%      10466 ±
> > 52%  interrupts.CPU14.CAL:Function_call_interrupts
> >      27280 ± 58%    +321.3%     114920 ±
> > 10%  interrupts.CPU14.RES:Rescheduling_interrupts
> >       1547 ± 13%    +550.6%      10065 ±
> > 51%  interrupts.CPU15.CAL:Function_call_interrupts
> >      24652 ± 60%    +366.8%     115077 ±
> > 10%  interrupts.CPU15.RES:Rescheduling_interrupts
> >       1613 ± 14%    +518.7%       9981 ±
> > 52%  interrupts.CPU16.CAL:Function_call_interrupts
> >      26360 ± 60%    +338.3%     115544
> > ±  9%  interrupts.CPU16.RES:Rescheduling_interrupts
> >       1783 ± 22%    +476.1%      10273 ±
> > 50%  interrupts.CPU17.CAL:Function_call_interrupts
> >      29465 ± 57%    +290.9%     115190 ±
> > 10%  interrupts.CPU17.RES:Rescheduling_interrupts
> >       1692 ± 15%    +539.1%      10816 ±
> > 49%  interrupts.CPU18.CAL:Function_call_interrupts
> >      25242 ± 56%    +355.0%     114843
> > ±  8%  interrupts.CPU18.RES:Rescheduling_interrupts
> >       1546 ± 14%    +576.6%      10463 ±
> > 48%  interrupts.CPU19.CAL:Function_call_interrupts
> >      27541 ± 53%    +318.0%     115135 ±
> > 10%  interrupts.CPU19.RES:Rescheduling_interrupts
> >       1539 ± 20%    +597.8%      10744 ±
> > 50%  interrupts.CPU2.CAL:Function_call_interrupts
> >      26770 ± 52%    +333.8%     116123 ±
> > 10%  interrupts.CPU2.RES:Rescheduling_interrupts
> >       1904 ± 25%    +432.4%      10138 ±
> > 52%  interrupts.CPU20.CAL:Function_call_interrupts
> >      25460 ± 61%    +353.1%     115361
> > ±  8%  interrupts.CPU20.RES:Rescheduling_interrupts
> >       1762 ± 23%    +469.0%      10028 ±
> > 47%  interrupts.CPU21.CAL:Function_call_interrupts
> >      26522 ± 54%    +331.6%     114472
> > ±  8%  interrupts.CPU21.RES:Rescheduling_interrupts
> >       1684 ± 19%    +506.5%      10214 ±
> > 49%  interrupts.CPU22.CAL:Function_call_interrupts
> >      26964 ± 52%    +320.4%     113356
> > ±  9%  interrupts.CPU22.RES:Rescheduling_interrupts
> >       1664 ± 16%    +552.7%      10863 ±
> > 47%  interrupts.CPU23.CAL:Function_call_interrupts
> >      25881 ± 53%    +331.9%     111768
> > ±  9%  interrupts.CPU23.RES:Rescheduling_interrupts
> >       2767 ± 70%    +456.6%      15402 ±
> > 58%  interrupts.CPU24.CAL:Function_call_interrupts
> >      34637 ± 23%    +245.6%     119696
> > ±  7%  interrupts.CPU24.RES:Rescheduling_interrupts
> >       1644 ± 19%    +790.7%      14649 ±
> > 62%  interrupts.CPU25.CAL:Function_call_interrupts
> >      31698 ± 34%    +271.3%     117692
> > ±  9%  interrupts.CPU25.RES:Rescheduling_interrupts
> >       1600 ± 20%    +885.1%      15766 ±
> > 63%  interrupts.CPU26.CAL:Function_call_interrupts
> >      32053 ± 35%    +263.6%     116533
> > ±  8%  interrupts.CPU26.RES:Rescheduling_interrupts
> >       1906 ± 40%    +773.1%      16646 ±
> > 65%  interrupts.CPU27.CAL:Function_call_interrupts
> >      32311 ± 37%    +257.4%     115489
> > ±  8%  interrupts.CPU27.RES:Rescheduling_interrupts
> >       1847 ± 15%    +729.0%      15312 ±
> > 64%  interrupts.CPU28.CAL:Function_call_interrupts
> >      30973 ± 38%    +271.9%     115189
> > ±  8%  interrupts.CPU28.RES:Rescheduling_interrupts
> >       1637 ± 24%    +825.0%      15146 ±
> > 64%  interrupts.CPU29.CAL:Function_call_interrupts
> >      32830 ± 34%    +250.8%     115184
> > ±  6%  interrupts.CPU29.RES:Rescheduling_interrupts
> >       1433 ± 14%    +603.8%      10090 ±
> > 50%  interrupts.CPU3.CAL:Function_call_interrupts
> >      26983 ± 49%    +320.2%     113392 ±
> > 10%  interrupts.CPU3.RES:Rescheduling_interrupts
> >       1597 ± 14%    +863.9%      15400 ±
> > 62%  interrupts.CPU30.CAL:Function_call_interrupts
> >      31376 ± 38%    +260.7%     113167
> > ±  8%  interrupts.CPU30.RES:Rescheduling_interrupts
> >       1648 ± 16%    +828.7%      15304 ±
> > 62%  interrupts.CPU31.CAL:Function_call_interrupts
> >      32435 ± 33%    +258.4%     116241
> > ±  8%  interrupts.CPU31.RES:Rescheduling_interrupts
> >       2081 ± 28%    +635.8%      15314 ±
> > 64%  interrupts.CPU32.CAL:Function_call_interrupts
> >      30678 ± 34%    +276.2%     115409
> > ±  8%  interrupts.CPU32.RES:Rescheduling_interrupts
> >       1508 ± 19%    +951.6%      15860 ±
> > 65%  interrupts.CPU33.CAL:Function_call_interrupts
> >      32920 ± 39%    +251.5%     115730
> > ±  8%  interrupts.CPU33.RES:Rescheduling_interrupts
> >       1529 ± 22%    +936.7%      15857 ±
> > 65%  interrupts.CPU34.CAL:Function_call_interrupts
> >      31214 ± 27%    +266.4%     114381
> > ±  8%  interrupts.CPU34.RES:Rescheduling_interrupts
> >       1543 ± 19%    +933.2%      15950 ±
> > 60%  interrupts.CPU35.CAL:Function_call_interrupts
> >      30887 ± 33%    +269.9%     114244
> > ±  7%  interrupts.CPU35.RES:Rescheduling_interrupts
> >       1566 ± 17%    +890.1%      15505 ±
> > 64%  interrupts.CPU36.CAL:Function_call_interrupts
> >      31545 ± 41%    +254.5%     111828
> > ±  9%  interrupts.CPU36.RES:Rescheduling_interrupts
> >       1518 ± 17%    +889.4%      15021 ±
> > 61%  interrupts.CPU37.CAL:Function_call_interrupts
> >      31749 ± 38%    +266.0%     116188
> > ±  9%  interrupts.CPU37.RES:Rescheduling_interrupts
> >       1507 ± 18%    +946.1%      15770 ±
> > 62%  interrupts.CPU38.CAL:Function_call_interrupts
> >      32670 ± 36%    +257.7%     116863
> > ±  7%  interrupts.CPU38.RES:Rescheduling_interrupts
> >       1662 ± 22%    +818.7%      15277 ±
> > 63%  interrupts.CPU39.CAL:Function_call_interrupts
> >      32667 ± 37%    +255.3%     116081
> > ±  9%  interrupts.CPU39.RES:Rescheduling_interrupts
> >       1908 ± 49%    +447.9%      10454 ±
> > 49%  interrupts.CPU4.CAL:Function_call_interrupts
> >      26121 ± 56%    +341.3%     115279
> > ±  7%  interrupts.CPU4.RES:Rescheduling_interrupts
> >       1599 ± 30%    +883.5%      15734 ±
> > 66%  interrupts.CPU40.CAL:Function_call_interrupts
> >      31759 ± 37%    +259.8%     114285
> > ±  8%  interrupts.CPU40.RES:Rescheduling_interrupts
> >       1535 ± 19%    +889.5%      15194 ±
> > 62%  interrupts.CPU41.CAL:Function_call_interrupts
> >      31350 ± 37%    +263.1%     113826
> > ±  9%  interrupts.CPU41.RES:Rescheduling_interrupts
> >       1565 ± 19%    +959.8%      16588 ±
> > 63%  interrupts.CPU42.CAL:Function_call_interrupts
> >      30781 ± 33%    +269.8%     113842
> > ±  7%  interrupts.CPU42.RES:Rescheduling_interrupts
> >       1581 ± 11%    +929.0%      16272 ±
> > 63%  interrupts.CPU43.CAL:Function_call_interrupts
> >      31679 ± 33%    +268.0%     116568
> > ±  7%  interrupts.CPU43.RES:Rescheduling_interrupts
> >       1520 ± 22%    +914.5%      15422 ±
> > 65%  interrupts.CPU44.CAL:Function_call_interrupts
> >      31967 ± 41%    +264.6%     116553
> > ±  9%  interrupts.CPU44.RES:Rescheduling_interrupts
> >       1503 ± 16%    +931.3%      15503 ±
> > 63%  interrupts.CPU45.CAL:Function_call_interrupts
> >      30829 ± 32%    +272.3%     114770
> > ±  8%  interrupts.CPU45.RES:Rescheduling_interrupts
> >       1492 ± 15%    +948.3%      15643 ±
> > 64%  interrupts.CPU46.CAL:Function_call_interrupts
> >      32136 ± 37%    +256.1%     114426
> > ±  8%  interrupts.CPU46.RES:Rescheduling_interrupts
> >       6530 ±  4%    +186.1%      18685 ±
> > 49%  interrupts.CPU47.CAL:Function_call_interrupts
> >      32843 ± 36%    +239.2%     111397
> > ±  8%  interrupts.CPU47.RES:Rescheduling_interrupts
> >       1501 ± 21%    +552.3%       9795 ±
> > 49%  interrupts.CPU48.CAL:Function_call_interrupts
> >      27770 ± 49%    +306.1%     112779 ±
> > 10%  interrupts.CPU48.RES:Rescheduling_interrupts
> >       1716 ± 12%    +479.8%       9951 ±
> > 47%  interrupts.CPU49.CAL:Function_call_interrupts
> >      27441 ± 50%    +306.8%     111620 ±
> > 12%  interrupts.CPU49.RES:Rescheduling_interrupts
> >       1536 ± 20%    +561.4%      10164 ±
> > 45%  interrupts.CPU5.CAL:Function_call_interrupts
> >      26043 ± 58%    +336.2%     113601
> > ±  8%  interrupts.CPU5.RES:Rescheduling_interrupts
> >       1466 ± 15%    +652.1%      11028 ±
> > 52%  interrupts.CPU50.CAL:Function_call_interrupts
> >      25157 ± 58%    +346.9%     112432 ±
> > 11%  interrupts.CPU50.RES:Rescheduling_interrupts
> >       1480 ± 14%    +607.5%      10472 ±
> > 49%  interrupts.CPU51.CAL:Function_call_interrupts
> >      25280 ± 53%    +326.7%     107877 ±
> > 10%  interrupts.CPU51.RES:Rescheduling_interrupts
> >       1718 ± 28%    +513.8%      10546 ±
> > 49%  interrupts.CPU52.CAL:Function_call_interrupts
> >      25402 ± 61%    +343.7%     112698
> > ±  9%  interrupts.CPU52.RES:Rescheduling_interrupts
> >       1471 ± 18%    +584.4%      10067 ±
> > 51%  interrupts.CPU53.CAL:Function_call_interrupts
> >      26502 ± 51%    +317.7%     110702
> > ±  9%  interrupts.CPU53.RES:Rescheduling_interrupts
> >       1520 ± 10%    +577.5%      10301 ±
> > 49%  interrupts.CPU54.CAL:Function_call_interrupts
> >      26868 ± 53%    +311.4%     110540 ±
> > 11%  interrupts.CPU54.RES:Rescheduling_interrupts
> >       1378 ± 15%    +629.6%      10059 ±
> > 48%  interrupts.CPU55.CAL:Function_call_interrupts
> >      26538 ± 55%    +328.1%     113615
> > ±  9%  interrupts.CPU55.RES:Rescheduling_interrupts
> >       1432 ± 12%    +613.7%      10226 ±
> > 50%  interrupts.CPU56.CAL:Function_call_interrupts
> >      27378 ± 54%    +318.3%     114536
> > ±  9%  interrupts.CPU56.RES:Rescheduling_interrupts
> >       1578 ± 23%    +551.4%      10283 ±
> > 51%  interrupts.CPU57.CAL:Function_call_interrupts
> >      25737 ± 62%    +337.5%     112612 ±
> > 10%  interrupts.CPU57.RES:Rescheduling_interrupts
> >       1483 ± 22%    +630.0%      10829 ±
> > 52%  interrupts.CPU58.CAL:Function_call_interrupts
> >      27343 ± 53%    +315.8%     113705
> > ±  9%  interrupts.CPU58.RES:Rescheduling_interrupts
> >       1460 ± 18%    +602.5%      10261 ±
> > 51%  interrupts.CPU59.CAL:Function_call_interrupts
> >      25911 ± 58%    +298.8%     103334
> > ±  7%  interrupts.CPU59.RES:Rescheduling_interrupts
> >       1569 ±  8%    +530.0%       9889 ±
> > 50%  interrupts.CPU6.CAL:Function_call_interrupts
> >      27162 ± 52%    +310.3%     111462 ±
> > 10%  interrupts.CPU6.RES:Rescheduling_interrupts
> >       1587 ± 31%    +530.4%      10005 ±
> > 51%  interrupts.CPU60.CAL:Function_call_interrupts
> >      26060 ± 56%    +330.3%     112129
> > ±  9%  interrupts.CPU60.RES:Rescheduling_interrupts
> >       1414 ± 16%    +608.1%      10012 ±
> > 49%  interrupts.CPU61.CAL:Function_call_interrupts
> >      25916 ± 59%    +345.2%     115375 ±
> > 10%  interrupts.CPU61.RES:Rescheduling_interrupts
> >       1501 ± 18%    +601.8%      10534 ±
> > 53%  interrupts.CPU62.CAL:Function_call_interrupts
> >      27209 ± 60%    +317.1%     113499
> > ±  9%  interrupts.CPU62.RES:Rescheduling_interrupts
> >       1411 ± 18%    +612.3%      10052 ±
> > 51%  interrupts.CPU63.CAL:Function_call_interrupts
> >      25584 ± 59%    +345.4%     113944
> > ±  9%  interrupts.CPU63.RES:Rescheduling_interrupts
> >       1890 ± 17%    +437.8%      10169 ±
> > 50%  interrupts.CPU64.CAL:Function_call_interrupts
> >      26884 ± 60%    +322.0%     113444 ±
> > 10%  interrupts.CPU64.RES:Rescheduling_interrupts
> >       1729 ± 25%    +475.6%       9955 ±
> > 48%  interrupts.CPU65.CAL:Function_call_interrupts
> >      26952 ± 54%    +321.5%     113608 ±
> > 10%  interrupts.CPU65.RES:Rescheduling_interrupts
> >       1897 ± 16%    +481.0%      11020 ±
> > 46%  interrupts.CPU66.CAL:Function_call_interrupts
> >      25947 ± 58%    +327.7%     110971
> > ±  9%  interrupts.CPU66.RES:Rescheduling_interrupts
> >       1761 ± 18%    +467.0%       9987 ±
> > 50%  interrupts.CPU67.CAL:Function_call_interrupts
> >      27578 ± 52%    +307.5%     112380 ±
> > 10%  interrupts.CPU67.RES:Rescheduling_interrupts
> >       1784 ± 18%    +451.1%       9835 ±
> > 49%  interrupts.CPU68.CAL:Function_call_interrupts
> >      25578 ± 60%    +340.7%     112722 ±
> > 10%  interrupts.CPU68.RES:Rescheduling_interrupts
> >       1837 ± 17%    +436.5%       9857 ±
> > 49%  interrupts.CPU69.CAL:Function_call_interrupts
> >      26266 ± 59%    +329.5%     112809
> > ±  9%  interrupts.CPU69.RES:Rescheduling_interrupts
> >       1465 ± 15%    +583.8%      10020 ±
> > 49%  interrupts.CPU7.CAL:Function_call_interrupts
> >      25720 ± 57%    +351.1%     116026
> > ±  8%  interrupts.CPU7.RES:Rescheduling_interrupts
> >       1810 ± 21%    +425.7%       9515 ±
> > 44%  interrupts.CPU70.CAL:Function_call_interrupts
> >      27634 ± 51%    +304.4%     111767 ±
> > 10%  interrupts.CPU70.RES:Rescheduling_interrupts
> >       1891 ± 12%    +415.1%       9740 ±
> > 48%  interrupts.CPU71.CAL:Function_call_interrupts
> >      26205 ± 55%    +323.3%     110934
> > ±  9%  interrupts.CPU71.RES:Rescheduling_interrupts
> >       2946 ± 93%    +429.1%      15592 ±
> > 63%  interrupts.CPU72.CAL:Function_call_interrupts
> >      33476 ± 29%    +241.2%     114214
> > ±  8%  interrupts.CPU72.RES:Rescheduling_interrupts
> >       1797 ± 28%    +730.7%      14931 ±
> > 64%  interrupts.CPU73.CAL:Function_call_interrupts
> >      31188 ± 31%    +269.5%     115229
> > ±  9%  interrupts.CPU73.RES:Rescheduling_interrupts
> >       1649 ± 23%    +918.0%      16794 ±
> > 64%  interrupts.CPU74.CAL:Function_call_interrupts
> >      30456 ± 37%    +274.1%     113927
> > ±  9%  interrupts.CPU74.RES:Rescheduling_interrupts
> >       1875 ± 31%    +785.5%      16610 ±
> > 62%  interrupts.CPU75.CAL:Function_call_interrupts
> >      31503 ± 38%    +259.7%     113310
> > ±  8%  interrupts.CPU75.RES:Rescheduling_interrupts
> >       1833 ± 17%    +766.5%      15885 ±
> > 64%  interrupts.CPU76.CAL:Function_call_interrupts
> >      29718 ± 38%    +278.7%     112554
> > ±  9%  interrupts.CPU76.RES:Rescheduling_interrupts
> >       1650 ± 15%    +857.9%      15811 ±
> > 66%  interrupts.CPU77.CAL:Function_call_interrupts
> >      31625 ± 29%    +252.2%     111391
> > ±  7%  interrupts.CPU77.RES:Rescheduling_interrupts
> >       1740 ± 16%    +798.0%      15632 ±
> > 62%  interrupts.CPU78.CAL:Function_call_interrupts
> >      32549 ± 34%    +239.6%     110549
> > ±  9%  interrupts.CPU78.RES:Rescheduling_interrupts
> >       1675 ± 19%    +830.7%      15595 ±
> > 64%  interrupts.CPU79.CAL:Function_call_interrupts
> >      31016 ± 36%    +266.4%     113656
> > ±  8%  interrupts.CPU79.RES:Rescheduling_interrupts
> >       1437 ± 11%    +635.4%      10570 ±
> > 49%  interrupts.CPU8.CAL:Function_call_interrupts
> >      27602 ± 60%    +322.8%     116699 ±
> > 10%  interrupts.CPU8.RES:Rescheduling_interrupts
> >       1683 ± 27%    +802.2%      15187 ±
> > 65%  interrupts.CPU80.CAL:Function_call_interrupts
> >      31149 ± 43%    +268.5%     114795
> > ±  8%  interrupts.CPU80.RES:Rescheduling_interrupts
> >       1583 ± 24%    +911.9%      16020 ±
> > 66%  interrupts.CPU81.CAL:Function_call_interrupts
> >      31699 ± 38%    +260.9%     114418
> > ±  8%  interrupts.CPU81.RES:Rescheduling_interrupts
> >       1752 ± 36%    +827.9%      16262 ±
> > 66%  interrupts.CPU82.CAL:Function_call_interrupts
> >      29360 ± 36%    +288.9%     114167
> > ±  8%  interrupts.CPU82.RES:Rescheduling_interrupts
> >       1558 ± 13%    +932.4%      16090 ±
> > 64%  interrupts.CPU83.CAL:Function_call_interrupts
> >      30230 ± 36%    +270.2%     111904
> > ±  8%  interrupts.CPU83.RES:Rescheduling_interrupts
> >       1455 ± 17%    +958.3%      15407 ±
> > 64%  interrupts.CPU84.CAL:Function_call_interrupts
> >      30911 ± 34%    +254.7%     109645 ±
> > 10%  interrupts.CPU84.RES:Rescheduling_interrupts
> >       1603 ± 10%    +827.8%      14881 ±
> > 62%  interrupts.CPU85.CAL:Function_call_interrupts
> >      31085 ± 37%    +265.4%     113578
> > ±  8%  interrupts.CPU85.RES:Rescheduling_interrupts
> >       1575 ± 17%    +904.7%      15827 ±
> > 64%  interrupts.CPU86.CAL:Function_call_interrupts
> >      33065 ± 38%    +248.7%     115301
> > ±  8%  interrupts.CPU86.RES:Rescheduling_interrupts
> >       1498 ± 18%    +920.9%      15295 ±
> > 64%  interrupts.CPU87.CAL:Function_call_interrupts
> >      30477 ± 42%    +274.6%     114181
> > ±  9%  interrupts.CPU87.RES:Rescheduling_interrupts
> >       1418 ± 17%    +960.6%      15044 ±
> > 64%  interrupts.CPU88.CAL:Function_call_interrupts
> >      31735 ± 32%    +255.3%     112741
> > ±  9%  interrupts.CPU88.RES:Rescheduling_interrupts
> >       1459 ± 21%    +984.3%      15826 ±
> > 65%  interrupts.CPU89.CAL:Function_call_interrupts
> >      31037 ± 41%    +260.3%     111826
> > ±  9%  interrupts.CPU89.RES:Rescheduling_interrupts
> >       1505 ± 18%    +562.6%       9975 ±
> > 47%  interrupts.CPU9.CAL:Function_call_interrupts
> >      27015 ± 64%    +327.1%     115392
> > ±  9%  interrupts.CPU9.RES:Rescheduling_interrupts
> >       1592 ± 24%    +935.1%      16487 ±
> > 66%  interrupts.CPU90.CAL:Function_call_interrupts
> >      31256 ± 37%    +257.9%     111877
> > ±  8%  interrupts.CPU90.RES:Rescheduling_interrupts
> >       1576 ± 19%    +882.7%      15489 ±
> > 66%  interrupts.CPU91.CAL:Function_call_interrupts
> >      31276 ± 37%    +266.3%     114573
> > ±  8%  interrupts.CPU91.RES:Rescheduling_interrupts
> >       1706 ± 18%    +786.3%      15121 ±
> > 66%  interrupts.CPU92.CAL:Function_call_interrupts
> >      31400 ± 43%    +260.4%     113170
> > ±  8%  interrupts.CPU92.RES:Rescheduling_interrupts
> >       1347 ± 18%   +1012.1%      14981 ±
> > 66%  interrupts.CPU93.CAL:Function_call_interrupts
> >      29869 ± 35%    +277.4%     112727
> > ±  8%  interrupts.CPU93.RES:Rescheduling_interrupts
> >       1372 ± 12%    +937.5%      14243 ±
> > 67%  interrupts.CPU94.CAL:Function_call_interrupts
> >      30286 ± 40%    +271.0%     112366
> > ±  7%  interrupts.CPU94.RES:Rescheduling_interrupts
> >       1739 ± 27%    +735.8%      14538 ±
> > 64%  interrupts.CPU95.CAL:Function_call_interrupts
> >      30315 ± 39%    +266.8%     111198
> > ±  8%  interrupts.CPU95.RES:Rescheduling_interrupts
> >    2791109 ± 14%    +291.3%   10921566
> > ±  8%  interrupts.RES:Rescheduling_interrupts
> >       1.73 ± 27%      -1.5        0.26 ±100%  perf-
> > profile.calltrace.cycles-
> > pp.kill_pid_info.kill_something_info.__x64_sys_kill.do_syscall_64.e
> > ntry_SYSCALL_64_after_hwframe
> >       2.22 ± 22%      -1.2        0.97        perf-
> > profile.calltrace.cycles-pp.kill
> >       2.08 ± 23%      -1.2        0.84        perf-
> > profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.kill
> >       2.03 ± 24%      -1.2        0.80        perf-
> > profile.calltrace.cycles-
> > pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.kill
> >       1.99 ± 24%      -1.2        0.76        perf-
> > profile.calltrace.cycles-
> > pp.__x64_sys_kill.do_syscall_64.entry_SYSCALL_64_after_hwframe.kill
> >       1.85 ± 26%      -1.2        0.62        perf-
> > profile.calltrace.cycles-
> > pp.kill_something_info.__x64_sys_kill.do_syscall_64.entry_SYSCALL_6
> > 4_after_hwframe.kill
> >      45.78            -0.9       44.84        perf-
> > profile.calltrace.cycles-pp.__send
> >      45.45            -0.9       44.56        perf-
> > profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__send
> >      44.92            -0.6       44.37        perf-
> > profile.calltrace.cycles-
> > pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send
> >      44.85            -0.5       44.31        perf-
> > profile.calltrace.cycles-
> > pp.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe.__
> > send
> >      44.78            -0.5       44.25        perf-
> > profile.calltrace.cycles-
> > pp.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_aft
> > er_hwframe.__send
> >      44.54            -0.5       44.03        perf-
> > profile.calltrace.cycles-
> > pp.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_S
> > YSCALL_64_after_hwframe
> >       1.47 ±  4%      -0.2        1.29 ±  2%  perf-
> > profile.calltrace.cycles-
> > pp.__dev_queue_xmit.ip_finish_output2.ip_output.__ip_queue_xmit.__t
> > cp_transmit_skb
> >       2.48            -0.1        2.33        perf-
> > profile.calltrace.cycles-
> > pp.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol
> > _deliver_rcu
> >       1.66            -0.1        1.57        perf-
> > profile.calltrace.cycles-
> > pp.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.tc
> > p_v4_rcv
> >       0.74 ±  3%      -0.1        0.64 ±  2%  perf-
> > profile.calltrace.cycles-
> > pp.__kfree_skb.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg.__sys_re
> > cvfrom
> >       0.62 ±  5%      -0.1        0.54 ±  2%  perf-
> > profile.calltrace.cycles-
> > pp.skb_release_all.__kfree_skb.tcp_recvmsg_locked.tcp_recvmsg.inet_
> > recvmsg
> >       0.61 ±  4%      -0.1        0.53 ±  2%  perf-
> > profile.calltrace.cycles-
> > pp.skb_release_head_state.skb_release_all.__kfree_skb.tcp_recvmsg_l
> > ocked.tcp_recvmsg
> >       0.88 ±  3%      -0.1        0.79        perf-
> > profile.calltrace.cycles-
> > pp.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2.ip_output
> > .__ip_queue_xmit
> >       0.80 ±  4%      -0.1        0.72        perf-
> > profile.calltrace.cycles-
> > pp.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_out
> > put2.ip_output
> >       1.27 ±  3%      -0.1        1.20        perf-
> > profile.calltrace.cycles-
> > pp._copy_from_iter_full.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg
> > .__sys_sendto
> >       1.14 ±  3%      -0.1        1.08        perf-
> > profile.calltrace.cycles-
> > pp.copyin._copy_from_iter_full.tcp_sendmsg_locked.tcp_sendmsg.sock_
> > sendmsg
> >       0.96 ±  2%      -0.1        0.90        perf-
> > profile.calltrace.cycles-
> > pp.sk_stream_alloc_skb.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.
> > __sys_sendto
> >       1.59            -0.1        1.53        perf-
> > profile.calltrace.cycles-
> > pp.skb_copy_datagram_iter.tcp_recvmsg_locked.tcp_recvmsg.inet_recvm
> > sg.__sys_recvfrom
> >       1.57            -0.1        1.51        perf-
> > profile.calltrace.cycles-
> > pp.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg_locked.tc
> > p_recvmsg.inet_recvmsg
> >       1.06 ±  3%      -0.1        1.01        perf-
> > profile.calltrace.cycles-
> > pp.copy_user_enhanced_fast_string.copyin._copy_from_iter_full.tcp_s
> > endmsg_locked.tcp_sendmsg
> >       0.90 ±  2%      -0.1        0.85        perf-
> > profile.calltrace.cycles-
> > pp.__alloc_skb.sk_stream_alloc_skb.tcp_sendmsg_locked.tcp_sendmsg.s
> > ock_sendmsg
> >       0.95            -0.0        0.91        perf-
> > profile.calltrace.cycles-
> > pp._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_rec
> > vmsg_locked.tcp_recvmsg
> >       1.76 ±  3%      +0.1        1.82        perf-
> > profile.calltrace.cycles-
> > pp.perf_swevent_overflow.perf_tp_event.perf_trace_sched_switch.__sc
> > hedule.schedule_idle
> >       1.81 ±  3%      +0.1        1.88        perf-
> > profile.calltrace.cycles-
> > pp.perf_tp_event.perf_trace_sched_switch.__schedule.schedule_idle.d
> > o_idle
> >       0.61 ±  2%      +0.1        0.68        perf-
> > profile.calltrace.cycles-
> > pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_
> > startup_64_no_verify
> >       1.94 ±  3%      +0.1        2.01        perf-
> > profile.calltrace.cycles-
> > pp.perf_trace_sched_switch.__schedule.schedule_idle.do_idle.cpu_sta
> > rtup_entry
> >       0.58 ±  2%      +0.1        0.67 ±  2%  perf-
> > profile.calltrace.cycles-
> > pp.pick_next_task_fair.__schedule.schedule_idle.do_idle.cpu_startup
> > _entry
> >       0.89 ±  6%      +0.2        1.07 ±  5%  perf-
> > profile.calltrace.cycles-
> > pp.update_load_avg.enqueue_entity.enqueue_task_fair.ttwu_do_activat
> > e.try_to_wake_up
> >       0.92 ±  3%      +0.2        1.10 ±  5%  perf-
> > profile.calltrace.cycles-
> > pp.update_load_avg.dequeue_entity.dequeue_task_fair.__schedule.sche
> > dule
> >       2.55            +0.2        2.78        perf-
> > profile.calltrace.cycles-
> > pp.select_task_rq_fair.try_to_wake_up.__wake_up_common.__wake_up_co
> > mmon_lock.sock_def_readable
> >       6.25 ±  3%      +0.2        6.48        perf-
> > profile.calltrace.cycles-
> > pp.update_curr.dequeue_entity.dequeue_task_fair.__schedule.schedule
> >       2.10 ±  3%      +0.2        2.34        perf-
> > profile.calltrace.cycles-
> > pp._raw_spin_lock.try_to_wake_up.__wake_up_common.__wake_up_common_
> > lock.sock_def_readable
> >       1.96 ±  3%      +0.2        2.20        perf-
> > profile.calltrace.cycles-
> > pp.native_queued_spin_lock_slowpath._raw_spin_lock.try_to_wake_up._
> > _wake_up_common.__wake_up_common_lock
> >       3.94 ±  2%      +0.2        4.19        perf-
> > profile.calltrace.cycles-
> > pp.enqueue_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up
> > .__wake_up_common
> >       4.83 ±  2%      +0.3        5.11        perf-
> > profile.calltrace.cycles-
> > pp.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.__wake_up_comm
> > on.__wake_up_common_lock
> >       4.86 ±  2%      +0.3        5.15        perf-
> > profile.calltrace.cycles-
> > pp.ttwu_do_activate.try_to_wake_up.__wake_up_common.__wake_up_commo
> > n_lock.sock_def_readable
> >       5.42 ±  2%      +0.3        5.72        perf-
> > profile.calltrace.cycles-
> > pp.set_task_cpu.try_to_wake_up.__wake_up_common.__wake_up_common_lo
> > ck.sock_def_readable
> >       3.86 ±  2%      +0.5        4.31        perf-
> > profile.calltrace.cycles-
> > pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_seconda
> > ry
> >       3.90 ±  2%      +0.5        4.36        perf-
> > profile.calltrace.cycles-
> > pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondar
> > y_startup_64_no_verify
> >       0.18 ±141%      +0.5        0.66 ±  5%  perf-
> > profile.calltrace.cycles-
> > pp._raw_spin_lock.__schedule.schedule_idle.do_idle.cpu_startup_entr
> > y
> >       0.00            +0.5        0.53 ±  2%  perf-
> > profile.calltrace.cycles-
> > pp.set_next_entity.pick_next_task_fair.__schedule.schedule_idle.do_
> > idle
> >       7.74 ±  2%      +0.6        8.32        perf-
> > profile.calltrace.cycles-
> > pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.schedule_ti
> > meout
> >      33.75            +0.6       34.34        perf-
> > profile.calltrace.cycles-
> > pp.__softirqentry_text_start.do_softirq.__local_bh_enable_ip.ip_fin
> > ish_output2.ip_output
> >       0.00            +0.6        0.59 ±  6%  perf-
> > profile.calltrace.cycles-
> > pp.native_queued_spin_lock_slowpath._raw_spin_lock.__schedule.sched
> > ule_idle.do_idle
> >      23.16            +0.6       23.75        perf-
> > profile.calltrace.cycles-
> > pp.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.ent
> > ry_SYSCALL_64_after_hwframe
> >      23.06            +0.6       23.66        perf-
> > profile.calltrace.cycles-
> > pp.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_sy
> > scall_64
> >       8.59 ±  2%      +0.6        9.21        perf-
> > profile.calltrace.cycles-
> > pp.dequeue_task_fair.__schedule.schedule.schedule_timeout.wait_woke
> > n
> >      22.39            +0.6       23.03        perf-
> > profile.calltrace.cycles-
> > pp.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64
> > _sys_recvfrom
> >      33.98            +0.7       34.64        perf-
> > profile.calltrace.cycles-
> > pp.__local_bh_enable_ip.ip_finish_output2.ip_output.__ip_queue_xmit
> > .__tcp_transmit_skb
> >      33.89            +0.7       34.56        perf-
> > profile.calltrace.cycles-
> > pp.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_output.__ip
> > _queue_xmit
> >      16.58 ±  2%      +0.7       17.32        perf-
> > profile.calltrace.cycles-
> > pp.__schedule.schedule.schedule_timeout.wait_woken.sk_wait_data
> >      31.96            +0.7       32.70        perf-
> > profile.calltrace.cycles-
> > pp.ip_rcv.__netif_receive_skb_one_core.process_backlog.__napi_poll.
> > net_rx_action
> >      16.76 ±  2%      +0.8       17.51        perf-
> > profile.calltrace.cycles-
> > pp.schedule.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_lo
> > cked
> >      16.91 ±  2%      +0.8       17.66        perf-
> > profile.calltrace.cycles-
> > pp.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_locked.tcp_
> > recvmsg
> >      17.24 ±  2%      +0.8       18.00        perf-
> > profile.calltrace.cycles-
> > pp.wait_woken.sk_wait_data.tcp_recvmsg_locked.tcp_recvmsg.inet_recv
> > msg
> >      18.29            +0.8       19.07        perf-
> > profile.calltrace.cycles-
> > pp.sk_wait_data.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg.__sys_r
> > ecvfrom
> >      31.40            +0.9       32.29        perf-
> > profile.calltrace.cycles-
> > pp.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_bac
> > klog.__napi_poll
> >      31.33            +0.9       32.22        perf-
> > profile.calltrace.cycles-
> > pp.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_
> > skb_one_core.process_backlog
> >      31.29            +0.9       32.18        perf-
> > profile.calltrace.cycles-
> > pp.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver
> > .ip_rcv.__netif_receive_skb_one_core
> >      31.15            +0.9       32.05        perf-
> > profile.calltrace.cycles-
> > pp.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_lo
> > cal_deliver.ip_rcv
> >      29.63            +1.0       30.61        perf-
> > profile.calltrace.cycles-
> > pp.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_delive
> > r_finish.ip_local_deliver
> >      29.48            +1.0       30.49        perf-
> > profile.calltrace.cycles-
> > pp.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver
> > _rcu.ip_local_deliver_finish
> >       0.00            +1.1        1.05        perf-
> > profile.calltrace.cycles-
> > pp.available_idle_cpu.select_idle_cpu.select_idle_sibling.select_ta
> > sk_rq_fair.try_to_wake_up
> >      24.86 ±  2%      +1.2       26.07        perf-
> > profile.calltrace.cycles-
> > pp.__wake_up_common.__wake_up_common_lock.sock_def_readable.tcp_rcv
> > _established.tcp_v4_do_rcv
> >      24.61 ±  2%      +1.2       25.82        perf-
> > profile.calltrace.cycles-
> > pp.try_to_wake_up.__wake_up_common.__wake_up_common_lock.sock_def_r
> > eadable.tcp_rcv_established
> >      25.65 ±  2%      +1.2       26.87        perf-
> > profile.calltrace.cycles-
> > pp.sock_def_readable.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.i
> > p_protocol_deliver_rcu
> >      25.34 ±  2%      +1.2       26.58        perf-
> > profile.calltrace.cycles-
> > pp.__wake_up_common_lock.sock_def_readable.tcp_rcv_established.tcp_
> > v4_do_rcv.tcp_v4_rcv
> >      14.20 ±  2%      +1.6       15.78        perf-
> > profile.calltrace.cycles-
> > pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup
> > _entry
> >      14.62 ±  2%      +1.6       16.26        perf-
> > profile.calltrace.cycles-
> > pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.star
> > t_secondary
> >      14.64 ±  2%      +1.6       16.28        perf-
> > profile.calltrace.cycles-
> > pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondar
> > y_startup_64_no_verify
> >       0.00            +2.0        2.03        perf-
> > profile.calltrace.cycles-
> > pp.select_idle_cpu.select_idle_sibling.select_task_rq_fair.try_to_w
> > ake_up.__wake_up_common
> >      19.87 ±  2%      +2.3       22.16        perf-
> > profile.calltrace.cycles-
> > pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_n
> > o_verify
> >      19.89 ±  2%      +2.3       22.19        perf-
> > profile.calltrace.cycles-
> > pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
> >      19.89 ±  2%      +2.3       22.19        perf-
> > profile.calltrace.cycles-
> > pp.start_secondary.secondary_startup_64_no_verify
> >      20.12 ±  2%      +2.3       22.45        perf-
> > profile.calltrace.cycles-pp.secondary_startup_64_no_verify
> >       0.00            +2.4        2.36        perf-
> > profile.calltrace.cycles-
> > pp.select_idle_sibling.select_task_rq_fair.try_to_wake_up.__wake_up
> > _common.__wake_up_common_lock
> >      74.35            -2.1       72.23        perf-
> > profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
> >      73.04            -1.8       71.27        perf-
> > profile.children.cycles-pp.do_syscall_64
> >       2.25 ± 22%      -1.3        0.99        perf-
> > profile.children.cycles-pp.kill
> >       1.99 ± 24%      -1.2        0.76        perf-
> > profile.children.cycles-pp.__x64_sys_kill
> >       1.85 ± 26%      -1.2        0.62        perf-
> > profile.children.cycles-pp.kill_something_info
> >       1.73 ± 27%      -1.2        0.50        perf-
> > profile.children.cycles-pp.kill_pid_info
> >       1.71 ± 28%      -1.2        0.48        perf-
> > profile.children.cycles-pp.group_send_sig_info
> >       1.59 ± 30%      -1.2        0.38        perf-
> > profile.children.cycles-pp.security_task_kill
> >       1.58 ± 30%      -1.2        0.37        perf-
> > profile.children.cycles-pp.apparmor_task_kill
> >      45.86            -1.0       44.90        perf-
> > profile.children.cycles-pp.__send
> >       1.28 ± 21%      -0.9        0.39 ±  2%  perf-
> > profile.children.cycles-pp.aa_sk_perm
> >       0.85 ± 30%      -0.6        0.24 ±  3%  perf-
> > profile.children.cycles-pp.aa_get_task_label
> >       0.91 ± 19%      -0.6        0.32 ±  3%  perf-
> > profile.children.cycles-pp.security_socket_recvmsg
> >       0.93 ± 19%      -0.6        0.34 ±  3%  perf-
> > profile.children.cycles-pp.sock_recvmsg
> >      44.86            -0.5       44.31        perf-
> > profile.children.cycles-pp.__x64_sys_sendto
> >      44.79            -0.5       44.25        perf-
> > profile.children.cycles-pp.__sys_sendto
> >      44.54            -0.5       44.03        perf-
> > profile.children.cycles-pp.sock_sendmsg
> >       1.09 ±  8%      -0.3        0.75        perf-
> > profile.children.cycles-pp.syscall_exit_to_user_mode
> >       0.91 ±  9%      -0.3        0.59        perf-
> > profile.children.cycles-pp.exit_to_user_mode_prepare
> >       0.44 ± 22%      -0.3        0.14 ±  2%  perf-
> > profile.children.cycles-pp.security_socket_sendmsg
> >       1.53 ±  4%      -0.2        1.35 ±  2%  perf-
> > profile.children.cycles-pp.__dev_queue_xmit
> >       2.59            -0.1        2.44        perf-
> > profile.children.cycles-pp.tcp_ack
> >       1.46 ±  2%      -0.1        1.34        perf-
> > profile.children.cycles-pp.__kfree_skb
> >       1.74 ±  2%      -0.1        1.64        perf-
> > profile.children.cycles-pp.tcp_clean_rtx_queue
> >       0.72 ±  4%      -0.1        0.62 ±  2%  perf-
> > profile.children.cycles-pp.skb_release_all
> >       0.70 ±  4%      -0.1        0.61 ±  2%  perf-
> > profile.children.cycles-pp.skb_release_head_state
> >       0.82 ±  4%      -0.1        0.74        perf-
> > profile.children.cycles-pp.loopback_xmit
> >       0.90 ±  4%      -0.1        0.82        perf-
> > profile.children.cycles-pp.dev_hard_start_xmit
> >       1.28 ±  3%      -0.1        1.20        perf-
> > profile.children.cycles-pp._copy_from_iter_full
> >       0.69 ±  4%      -0.1        0.62 ±  2%  perf-
> > profile.children.cycles-pp._raw_spin_lock_bh
> >       0.54 ±  6%      -0.1        0.46 ±  2%  perf-
> > profile.children.cycles-pp.switch_mm_irqs_off
> >       0.53 ±  4%      -0.1        0.45 ±  2%  perf-
> > profile.children.cycles-pp.dst_release
> >       1.58            -0.1        1.51        perf-
> > profile.children.cycles-pp.__skb_datagram_iter
> >       0.96 ±  2%      -0.1        0.90        perf-
> > profile.children.cycles-pp.sk_stream_alloc_skb
> >       1.60            -0.1        1.53        perf-
> > profile.children.cycles-pp.skb_copy_datagram_iter
> >       0.40 ±  3%      -0.1        0.34 ±  4%  perf-
> > profile.children.cycles-pp.tcp_send_mss
> >       0.36 ±  4%      -0.1        0.31 ±  3%  perf-
> > profile.children.cycles-pp.tcp_current_mss
> >       0.91 ±  2%      -0.1        0.86        perf-
> > profile.children.cycles-pp.__alloc_skb
> >       0.23 ±  5%      -0.0        0.18 ±  4%  perf-
> > profile.children.cycles-pp.irqtime_account_irq
> >       0.27 ±  5%      -0.0        0.23        perf-
> > profile.children.cycles-pp.load_new_mm_cr3
> >       0.17 ±  2%      -0.0        0.13 ±  4%  perf-
> > profile.children.cycles-pp.tcp_ack_update_rtt
> >       0.53            -0.0        0.49 ±  2%  perf-
> > profile.children.cycles-pp.sched_clock_cpu
> >       0.61 ±  4%      -0.0        0.58 ±  3%  perf-
> > profile.children.cycles-pp.release_sock
> >       0.33 ±  8%      -0.0        0.29 ±  2%  perf-
> > profile.children.cycles-pp.next_token
> >       0.95            -0.0        0.92        perf-
> > profile.children.cycles-pp._copy_to_iter
> >       0.49 ±  2%      -0.0        0.46 ±  2%  perf-
> > profile.children.cycles-pp.switch_fpu_return
> >       0.41 ±  3%      -0.0        0.38 ±  3%  perf-
> > profile.children.cycles-pp.tcp_event_new_data_sent
> >       0.39 ±  3%      -0.0        0.36 ±  2%  perf-
> > profile.children.cycles-pp.netif_rx
> >       0.31 ±  2%      -0.0        0.28        perf-
> > profile.children.cycles-pp.reweight_entity
> >       0.21 ±  4%      -0.0        0.18 ±  4%  perf-
> > profile.children.cycles-pp.kmem_cache_alloc_node
> >       0.33 ±  2%      -0.0        0.30 ±  3%  perf-
> > profile.children.cycles-pp.__entry_text_start
> >       0.20 ±  4%      -0.0        0.17 ±  3%  perf-
> > profile.children.cycles-pp.enqueue_to_backlog
> >       0.38            -0.0        0.35 ±  2%  perf-
> > profile.children.cycles-pp.__check_object_size
> >       0.38 ±  3%      -0.0        0.35        perf-
> > profile.children.cycles-pp.netif_rx_internal
> >       0.32 ±  3%      -0.0        0.29        perf-
> > profile.children.cycles-pp.syscall_return_via_sysret
> >       0.16 ±  4%      -0.0        0.13 ±  5%  perf-
> > profile.children.cycles-pp.ip_local_out
> >       0.28 ±  3%      -0.0        0.25 ±  2%  perf-
> > profile.children.cycles-pp.ttwu_do_wakeup
> >       0.13 ±  5%      -0.0        0.11 ±  6%  perf-
> > profile.children.cycles-pp.__virt_addr_valid
> >       0.13 ±  3%      -0.0        0.11 ±  6%  perf-
> > profile.children.cycles-pp.nf_hook_slow
> >       0.31            -0.0        0.29 ±  2%  perf-
> > profile.children.cycles-pp.simple_copy_to_iter
> >       0.11 ±  6%      -0.0        0.09 ±  5%  perf-
> > profile.children.cycles-pp.import_single_range
> >       0.11 ±  9%      -0.0        0.09 ±  4%  perf-
> > profile.children.cycles-pp.sock_put
> >       0.26 ±  2%      -0.0        0.24 ±  3%  perf-
> > profile.children.cycles-pp.check_preempt_curr
> >       0.09 ±  5%      -0.0        0.07 ±  7%  perf-
> > profile.children.cycles-pp.__might_fault
> >       0.13 ±  5%      -0.0        0.11 ±  4%  perf-
> > profile.children.cycles-pp.copy_fpregs_to_fpstate
> >       0.17 ±  2%      -0.0        0.15 ±  3%  perf-
> > profile.children.cycles-pp.tcp_rearm_rto
> >       0.13 ±  6%      -0.0        0.11 ±  5%  perf-
> > profile.children.cycles-pp.__cond_resched
> >       0.10 ±  7%      -0.0        0.09 ±  4%  perf-
> > profile.children.cycles-pp.tcp_event_data_recv
> >       0.11 ±  6%      -0.0        0.09 ±  5%  perf-
> > profile.children.cycles-pp.check_kill_permission
> >       0.23 ±  3%      -0.0        0.22        perf-
> > profile.children.cycles-pp.__ksize
> >       0.10 ±  4%      -0.0        0.08 ±  5%  perf-
> > profile.children.cycles-pp.tcp_update_pacing_rate
> >       0.07 ±  7%      -0.0        0.05 ±  8%  perf-
> > profile.children.cycles-pp.server
> >       0.08 ±  4%      +0.0        0.09 ±  4%  perf-
> > profile.children.cycles-pp.attach_entity_load_avg
> >       0.07 ±  5%      +0.0        0.09 ±  5%  perf-
> > profile.children.cycles-pp.is_module_text_address
> >       0.10 ±  8%      +0.0        0.12 ±  5%  perf-
> > profile.children.cycles-pp.set_next_task_idle
> >       0.09 ±  5%      +0.0        0.11 ±  4%  perf-
> > profile.children.cycles-pp.__update_idle_core
> >       0.29            +0.0        0.32 ±  2%  perf-
> > profile.children.cycles-pp.tcp_cleanup_rbuf
> >       0.30            +0.0        0.33        perf-
> > profile.children.cycles-pp.kmem_cache_free
> >       0.13 ±  5%      +0.0        0.15 ±  3%  perf-
> > profile.children.cycles-pp.perf_instruction_pointer
> >       0.08 ±  7%      +0.0        0.10 ±  6%  perf-
> > profile.children.cycles-pp.cpus_share_cache
> >       0.10 ±  4%      +0.0        0.13 ±  5%  perf-
> > profile.children.cycles-pp.pick_next_task_idle
> >       0.20 ±  3%      +0.0        0.23 ±  2%  perf-
> > profile.children.cycles-pp.tick_nohz_idle_exit
> >       0.55 ±  2%      +0.0        0.58        perf-
> > profile.children.cycles-pp.update_rq_clock
> >       0.32 ±  4%      +0.0        0.34 ±  2%  perf-
> > profile.children.cycles-pp.tick_nohz_get_sleep_length
> >       0.23 ±  2%      +0.0        0.26 ±  2%  perf-
> > profile.children.cycles-pp._find_next_bit
> >       0.19 ±  5%      +0.0        0.22 ±  3%  perf-
> > profile.children.cycles-pp.tick_nohz_next_event
> >       0.49 ±  2%      +0.0        0.52 ±  2%  perf-
> > profile.children.cycles-pp.finish_task_switch
> >       0.33 ±  3%      +0.0        0.37        perf-
> > profile.children.cycles-pp.migrate_task_rq_fair
> >       0.22 ±  5%      +0.0        0.26 ±  4%  perf-
> > profile.children.cycles-pp.start_kernel
> >       0.19 ±  2%      +0.0        0.23 ±  2%  perf-
> > profile.children.cycles-pp.cpumask_next
> >       0.29 ±  4%      +0.0        0.34 ±  2%  perf-
> > profile.children.cycles-pp.__slab_free
> >       0.60            +0.0        0.65 ±  2%  perf-
> > profile.children.cycles-pp.set_next_entity
> >       0.90            +0.0        0.94        perf-
> > profile.children.cycles-pp._raw_spin_lock_irqsave
> >       0.62 ±  2%      +0.1        0.69        perf-
> > profile.children.cycles-pp.menu_select
> >       2.01            +0.1        2.10        perf-
> > profile.children.cycles-pp.select_idle_cpu
> >       1.16            +0.1        1.27        perf-
> > profile.children.cycles-pp.available_idle_cpu
> >       2.62            +0.2        2.86        perf-
> > profile.children.cycles-pp.select_task_rq_fair
> >       4.07 ±  2%      +0.2        4.32        perf-
> > profile.children.cycles-pp.enqueue_entity
> >       5.00 ±  2%      +0.3        5.28        perf-
> > profile.children.cycles-pp.ttwu_do_activate
> >       4.97 ±  2%      +0.3        5.26        perf-
> > profile.children.cycles-pp.enqueue_task_fair
> >       5.60 ±  3%      +0.3        5.90        perf-
> > profile.children.cycles-pp.set_task_cpu
> >       2.54 ±  4%      +0.4        2.91 ±  4%  perf-
> > profile.children.cycles-pp.update_load_avg
> >       3.37 ±  2%      +0.4        3.76        perf-
> > profile.children.cycles-pp._raw_spin_lock
> >       2.61 ±  3%      +0.4        3.02 ±  2%  perf-
> > profile.children.cycles-pp.native_queued_spin_lock_slowpath
> >       1.46 ±  8%      +0.4        1.88 ±  8%  perf-
> > profile.children.cycles-pp.update_cfs_group
> >       3.95 ±  2%      +0.5        4.42        perf-
> > profile.children.cycles-pp.schedule_idle
> >      17.05            +0.5       17.54        perf-
> > profile.children.cycles-pp.schedule
> >       7.76 ±  2%      +0.6        8.34        perf-
> > profile.children.cycles-pp.dequeue_entity
> >      23.16            +0.6       23.76        perf-
> > profile.children.cycles-pp.inet_recvmsg
> >      23.07            +0.6       23.67        perf-
> > profile.children.cycles-pp.tcp_recvmsg
> >       8.61 ±  2%      +0.6        9.23        perf-
> > profile.children.cycles-pp.dequeue_task_fair
> >      22.41            +0.6       23.05        perf-
> > profile.children.cycles-pp.tcp_recvmsg_locked
> >      34.18            +0.7       34.86        perf-
> > profile.children.cycles-pp.__local_bh_enable_ip
> >      33.99            +0.7       34.68        perf-
> > profile.children.cycles-pp.do_softirq
> >      34.00            +0.7       34.69        perf-
> > profile.children.cycles-pp.__softirqentry_text_start
> >      33.40            +0.7       34.14        perf-
> > profile.children.cycles-pp.net_rx_action
> >      33.18            +0.8       33.94        perf-
> > profile.children.cycles-pp.__napi_poll
> >      16.91 ±  2%      +0.8       17.66        perf-
> > profile.children.cycles-pp.schedule_timeout
> >      17.24 ±  2%      +0.8       18.00        perf-
> > profile.children.cycles-pp.wait_woken
> >      33.09            +0.8       33.86        perf-
> > profile.children.cycles-pp.process_backlog
> >      18.30            +0.8       19.07        perf-
> > profile.children.cycles-pp.sk_wait_data
> >      32.81            +0.8       33.61        perf-
> > profile.children.cycles-pp.__netif_receive_skb_one_core
> >      32.47            +0.8       33.30        perf-
> > profile.children.cycles-pp.ip_rcv
> >      31.91            +0.9       32.81        perf-
> > profile.children.cycles-pp.ip_local_deliver_finish
> >      31.97            +0.9       32.87        perf-
> > profile.children.cycles-pp.ip_local_deliver
> >      31.86            +0.9       32.77        perf-
> > profile.children.cycles-pp.ip_protocol_deliver_rcu
> >      31.72            +0.9       32.63        perf-
> > profile.children.cycles-pp.tcp_v4_rcv
> >      20.81            +0.9       21.74        perf-
> > profile.children.cycles-pp.__schedule
> >      30.18            +1.0       31.16        perf-
> > profile.children.cycles-pp.tcp_v4_do_rcv
> >      30.03            +1.0       31.04        perf-
> > profile.children.cycles-pp.tcp_rcv_established
> >      25.54 ±  2%      +1.2       26.76        perf-
> > profile.children.cycles-pp.__wake_up_common
> >      25.29 ±  2%      +1.2       26.52        perf-
> > profile.children.cycles-pp.try_to_wake_up
> >      26.34 ±  2%      +1.2       27.58        perf-
> > profile.children.cycles-pp.sock_def_readable
> >      26.04 ±  2%      +1.2       27.29        perf-
> > profile.children.cycles-pp.__wake_up_common_lock
> >      14.36 ±  2%      +1.6       15.96        perf-
> > profile.children.cycles-pp.intel_idle
> >      14.80 ±  2%      +1.7       16.46        perf-
> > profile.children.cycles-pp.cpuidle_enter_state
> >      14.80 ±  2%      +1.7       16.47        perf-
> > profile.children.cycles-pp.cpuidle_enter
> >      19.89 ±  2%      +2.3       22.19        perf-
> > profile.children.cycles-pp.start_secondary
> >      20.10 ±  2%      +2.3       22.43        perf-
> > profile.children.cycles-pp.do_idle
> >      20.12 ±  2%      +2.3       22.45        perf-
> > profile.children.cycles-pp.secondary_startup_64_no_verify
> >      20.12 ±  2%      +2.3       22.45        perf-
> > profile.children.cycles-pp.cpu_startup_entry
> >       0.00            +2.4        2.43        perf-
> > profile.children.cycles-pp.select_idle_sibling
> >       1.18 ± 22%      -0.9        0.30 ±  2%  perf-
> > profile.self.cycles-pp.aa_sk_perm
> >       0.84 ± 30%      -0.6        0.23 ±  4%  perf-
> > profile.self.cycles-pp.aa_get_task_label
> >       0.66 ± 31%      -0.6        0.10 ±  4%  perf-
> > profile.self.cycles-pp.apparmor_task_kill
> >       0.61 ±  5%      -0.1        0.55 ±  3%  perf-
> > profile.self.cycles-pp._raw_spin_lock_bh
> >       0.52 ±  4%      -0.1        0.45 ±  2%  perf-
> > profile.self.cycles-pp.dst_release
> >       0.33 ±  2%      -0.1        0.26 ±  2%  perf-
> > profile.self.cycles-pp.select_task_rq_fair
> >       0.43 ±  6%      -0.1        0.38 ±  2%  perf-
> > profile.self.cycles-pp.__tcp_transmit_skb
> >       0.70 ±  3%      -0.1        0.65 ±  2%  perf-
> > profile.self.cycles-pp.tcp_sendmsg_locked
> >       0.27 ±  5%      -0.0        0.23        perf-
> > profile.self.cycles-pp.load_new_mm_cr3
> >       0.49 ±  2%      -0.0        0.45 ±  2%  perf-
> > profile.self.cycles-pp.switch_fpu_return
> >       0.16 ±  4%      -0.0        0.12 ±  3%  perf-
> > profile.self.cycles-pp.tcp_ack_update_rtt
> >       0.33 ±  3%      -0.0        0.30 ±  3%  perf-
> > profile.self.cycles-pp.tcp_clean_rtx_queue
> >       0.18 ±  6%      -0.0        0.15 ±  2%  perf-
> > profile.self.cycles-pp.__ip_queue_xmit
> >       0.34 ±  2%      -0.0        0.30 ±  3%  perf-
> > profile.self.cycles-pp.tcp_write_xmit
> >       0.25 ±  7%      -0.0        0.22 ±  6%  perf-
> > profile.self.cycles-pp.switch_mm_irqs_off
> >       0.13 ±  7%      -0.0        0.10 ± 12%  perf-
> > profile.self.cycles-pp.tcp_current_mss
> >       0.25 ±  5%      -0.0        0.22 ±  3%  perf-
> > profile.self.cycles-pp.ip_finish_output2
> >       0.10 ±  7%      -0.0        0.07 ±  8%  perf-
> > profile.self.cycles-pp.exit_to_user_mode_prepare
> >       0.13 ±  5%      -0.0        0.10 ±  3%  perf-
> > profile.self.cycles-pp.do_user_addr_fault
> >       0.13 ±  5%      -0.0        0.11 ±  6%  perf-
> > profile.self.cycles-pp.__virt_addr_valid
> >       0.21 ±  6%      -0.0        0.18 ±  3%  perf-
> > profile.self.cycles-pp.tcp_wfree
> >       0.32 ±  3%      -0.0        0.29 ±  2%  perf-
> > profile.self.cycles-pp.syscall_return_via_sysret
> >       0.23 ±  4%      -0.0        0.21 ±  2%  perf-
> > profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
> >       0.11 ±  8%      -0.0        0.09        perf-
> > profile.self.cycles-pp.import_single_range
> >       0.15 ±  3%      -0.0        0.14 ±  3%  perf-
> > profile.self.cycles-pp.tcp_rearm_rto
> >       0.09 ±  8%      -0.0        0.07 ±  5%  perf-
> > profile.self.cycles-pp.sched_clock_cpu
> >       0.22            -0.0        0.20 ±  3%  perf-
> > profile.self.cycles-pp.__softirqentry_text_start
> >       0.11 ±  5%      -0.0        0.09 ±  5%  perf-
> > profile.self.cycles-pp.__netif_receive_skb_one_core
> >       0.07 ±  8%      -0.0        0.05 ±  8%  perf-
> > profile.self.cycles-pp.tcp_v4_do_rcv
> >       0.10 ±  4%      -0.0        0.08 ±  7%  perf-
> > profile.self.cycles-pp.perf_callchain
> >       0.13 ±  5%      -0.0        0.11 ±  3%  perf-
> > profile.self.cycles-pp.copy_fpregs_to_fpstate
> >       0.13 ±  5%      -0.0        0.11 ±  4%  perf-
> > profile.self.cycles-pp.kmem_cache_alloc_node
> >       0.09            -0.0        0.08 ±  6%  perf-
> > profile.self.cycles-pp.__napi_poll
> >       0.08 ±  6%      -0.0        0.06 ±  6%  perf-
> > profile.self.cycles-pp.__x64_sys_sendto
> >       0.12 ±  3%      -0.0        0.11 ±  3%  perf-
> > profile.self.cycles-pp.__sys_sendto
> >       0.11 ±  6%      -0.0        0.09 ±  5%  perf-
> > profile.self.cycles-pp.__x64_sys_recvfrom
> >       0.14 ±  3%      -0.0        0.12 ±  3%  perf-
> > profile.self.cycles-pp.__kmalloc_node_track_caller
> >       0.07 ±  7%      -0.0        0.05 ±  8%  perf-
> > profile.self.cycles-pp.ftrace_ops_trampoline
> >       0.09 ±  5%      -0.0        0.08 ±  5%  perf-
> > profile.self.cycles-pp.tcp_update_pacing_rate
> >       0.06 ±  8%      +0.0        0.07        perf-
> > profile.self.cycles-pp.cpuidle_enter_state
> >       0.06            +0.0        0.07 ±  6%  perf-
> > profile.self.cycles-pp.rcu_nocb_flush_deferred_wakeup
> >       0.08 ±  7%      +0.0        0.10 ±  4%  perf-
> > profile.self.cycles-pp.cpus_share_cache
> >       0.05 ±  8%      +0.0        0.07 ±  5%  perf-
> > profile.self.cycles-pp.perf_instruction_pointer
> >       0.28            +0.0        0.30 ±  3%  perf-
> > profile.self.cycles-pp.tcp_cleanup_rbuf
> >       0.18 ±  5%      +0.0        0.20        perf-
> > profile.self.cycles-pp.migrate_task_rq_fair
> >       0.24 ±  3%      +0.0        0.27 ±  2%  perf-
> > profile.self.cycles-pp.menu_select
> >       0.41 ±  3%      +0.0        0.44 ±  2%  perf-
> > profile.self.cycles-pp.update_rq_clock
> >       0.22 ±  4%      +0.0        0.26 ±  2%  perf-
> > profile.self.cycles-pp._find_next_bit
> >       0.16 ±  2%      +0.0        0.20 ±  3%  perf-
> > profile.self.cycles-pp.do_idle
> >       0.29 ±  3%      +0.0        0.34 ±  2%  perf-
> > profile.self.cycles-pp.__slab_free
> >       0.87            +0.0        0.92        perf-
> > profile.self.cycles-pp._raw_spin_lock_irqsave
> >       1.24            +0.0        1.29 ±  2%  perf-
> > profile.self.cycles-pp.__schedule
> >       1.15            +0.1        1.26        perf-
> > profile.self.cycles-pp.available_idle_cpu
> >       0.00            +0.1        0.14 ±  3%  perf-
> > profile.self.cycles-pp.select_idle_sibling
> >       0.43 ±  7%      +0.2        0.61 ±  8%  perf-
> > profile.self.cycles-pp.set_task_cpu
> >       1.78 ±  4%      +0.3        2.12 ±  5%  perf-
> > profile.self.cycles-pp.update_load_avg
> >       2.61 ±  3%      +0.4        3.01 ±  2%  perf-
> > profile.self.cycles-pp.native_queued_spin_lock_slowpath
> >       1.46 ±  8%      +0.4        1.88 ±  8%  perf-
> > profile.self.cycles-pp.update_cfs_group
> >      14.36 ±  2%      +1.6       15.96        perf-
> > profile.self.cycles-pp.intel_idle
> >   20074513 ±  7%     -24.9%   15075828 ±  7%  softirqs.CPU0.NET_RX
> >      38385 ± 66%    +476.2%     221177 ±  9%  softirqs.CPU0.SCHED
> >      32352 ± 12%     -78.4%       6999 ± 45%  softirqs.CPU0.TIMER
> >   20359880 ±  5%     -26.9%   14879840 ±  7%  softirqs.CPU1.NET_RX
> >      36748 ± 67%    +492.7%     217810 ± 10%  softirqs.CPU1.SCHED
> >      31252 ± 13%     -85.5%       4524 ± 59%  softirqs.CPU1.TIMER
> >   20405448 ±  5%     -25.5%   15210284 ±  7%  softirqs.CPU10.NET_RX
> >      36443 ± 69%    +499.5%     218467 ±  9%  softirqs.CPU10.SCHED
> >      30609 ± 11%     -84.5%       4741 ± 63%  softirqs.CPU10.TIMER
> >   20394573 ±  4%     -29.0%   14484845 ± 10%  softirqs.CPU11.NET_RX
> >      36636 ± 70%    +464.1%     206683 ±  7%  softirqs.CPU11.SCHED
> >      30678 ± 11%     -84.8%       4653 ± 67%  softirqs.CPU11.TIMER
> >   20405829 ±  4%     -26.1%   15089688 ±  7%  softirqs.CPU12.NET_RX
> >      37061 ± 70%    +494.1%     220185 ±  8%  softirqs.CPU12.SCHED
> >      31218 ± 10%     -84.2%       4919 ± 66%  softirqs.CPU12.TIMER
> >   20444043 ±  5%     -26.2%   15095203 ±  7%  softirqs.CPU13.NET_RX
> >      36503 ± 70%    +497.6%     218132 ±  9%  softirqs.CPU13.SCHED
> >      30651 ± 11%     -84.6%       4710 ± 62%  softirqs.CPU13.TIMER
> >   20419827 ±  5%     -25.6%   15186252 ±  7%  softirqs.CPU14.NET_RX
> >      36704 ± 70%    +494.3%     218149 ±  9%  softirqs.CPU14.SCHED
> >      30681 ± 11%     -84.3%       4806 ± 65%  softirqs.CPU14.TIMER
> >   20510872 ±  5%     -25.7%   15232936 ±  7%  softirqs.CPU15.NET_RX
> >      36871 ± 69%    +493.3%     218761 ±  9%  softirqs.CPU15.SCHED
> >      31009 ± 12%     -84.5%       4793 ± 63%  softirqs.CPU15.TIMER
> >   20418436 ±  5%     -25.4%   15224178 ±  7%  softirqs.CPU16.NET_RX
> >      36854 ± 69%    +495.1%     219327 ±  9%  softirqs.CPU16.SCHED
> >      31013 ± 11%     -84.7%       4758 ± 64%  softirqs.CPU16.TIMER
> >   20372676 ±  4%     -25.9%   15095272 ±  7%  softirqs.CPU17.NET_RX
> >      36811 ± 70%    +494.0%     218669 ±  9%  softirqs.CPU17.SCHED
> >      31390 ± 13%     -77.9%       6929 ± 67%  softirqs.CPU17.TIMER
> >   20416587 ±  5%     -26.0%   15118161 ±  7%  softirqs.CPU18.NET_RX
> >      36394 ± 70%    +502.6%     219293 ±  9%  softirqs.CPU18.SCHED
> >      30912 ± 11%     -84.5%       4784 ± 63%  softirqs.CPU18.TIMER
> >   20432227 ±  4%     -25.8%   15168258 ±  7%  softirqs.CPU19.NET_RX
> >      36602 ± 70%    +497.8%     218796 ±  9%  softirqs.CPU19.SCHED
> >      30843 ± 11%     -82.9%       5259 ± 56%  softirqs.CPU19.TIMER
> >   20304573 ±  6%     -26.0%   15034216 ±  7%  softirqs.CPU2.NET_RX
> >      36968 ± 69%    +495.1%     220012 ±  9%  softirqs.CPU2.SCHED
> >      30526 ± 12%     -84.4%       4764 ± 62%  softirqs.CPU2.TIMER
> >   20444514 ±  5%     -25.5%   15240754 ±  7%  softirqs.CPU20.NET_RX
> >      36759 ± 70%    +495.9%     219056 ±  9%  softirqs.CPU20.SCHED
> >      30846 ± 11%     -84.3%       4831 ± 61%  softirqs.CPU20.TIMER
> >   20253081 ±  6%     -25.1%   15174237 ±  7%  softirqs.CPU21.NET_RX
> >      36699 ± 70%    +495.2%     218452 ±  8%  softirqs.CPU21.SCHED
> >      30719 ± 12%     -83.5%       5060 ± 65%  softirqs.CPU21.TIMER
> >   20460332 ±  4%     -25.8%   15186744 ±  7%  softirqs.CPU22.NET_RX
> >      36652 ± 70%    +497.1%     218854 ±  9%  softirqs.CPU22.SCHED
> >      30899 ± 11%     -84.4%       4811 ± 63%  softirqs.CPU22.TIMER
> >   20389322 ±  5%     -26.5%   14979439 ±  8%  softirqs.CPU23.NET_RX
> >      36717 ± 70%    +492.5%     217548 ±  8%  softirqs.CPU23.SCHED
> >      30864 ± 11%     -83.1%       5216 ± 60%  softirqs.CPU23.TIMER
> >   21350836 ±  3%     -30.0%   14945967 ±  6%  softirqs.CPU24.NET_RX
> >      47117 ± 46%    +359.7%     216593 ±  8%  softirqs.CPU24.SCHED
> >      31954 ± 20%     -84.0%       5123 ± 66%  softirqs.CPU24.TIMER
> >   21494795 ±  3%     -28.6%   15347973 ±  6%  softirqs.CPU25.NET_RX
> >      46755 ± 49%    +362.6%     216302 ±  8%  softirqs.CPU25.SCHED
> >      30392 ±  9%     -82.7%       5245 ± 47%  softirqs.CPU25.TIMER
> >   21529925 ±  3%     -28.8%   15338064 ±  6%  softirqs.CPU26.NET_RX
> >      46989 ± 48%    +360.6%     216437 ±  8%  softirqs.CPU26.SCHED
> >      30126 ±  9%     -84.3%       4715 ± 60%  softirqs.CPU26.TIMER
> >   21609177 ±  3%     -28.9%   15374089 ±  6%  softirqs.CPU27.NET_RX
> >      47211 ± 48%    +358.1%     216275 ±  8%  softirqs.CPU27.SCHED
> >      30806 ±  9%     -84.4%       4806 ± 59%  softirqs.CPU27.TIMER
> >   21434518 ±  3%     -28.0%   15426523 ±  6%  softirqs.CPU28.NET_RX
> >      46072 ± 49%    +369.3%     216211 ±  8%  softirqs.CPU28.SCHED
> >      29926 ± 10%     -83.7%       4871 ± 60%  softirqs.CPU28.TIMER
> >   21436007 ±  3%     -28.3%   15360927 ±  6%  softirqs.CPU29.NET_RX
> >      46751 ± 48%    +363.2%     216561 ±  8%  softirqs.CPU29.SCHED
> >      29983 ±  9%     -83.7%       4876 ± 58%  softirqs.CPU29.TIMER
> >   20556029 ±  5%     -27.1%   14991636 ±  9%  softirqs.CPU3.NET_RX
> >      36617 ± 69%    +485.7%     214485 ±  9%  softirqs.CPU3.SCHED
> >      30777 ± 11%     -82.7%       5311 ± 54%  softirqs.CPU3.TIMER
> >   21514774 ±  3%     -29.0%   15275932 ±  6%  softirqs.CPU30.NET_RX
> >      46934 ± 48%    +361.6%     216667 ±  8%  softirqs.CPU30.SCHED
> >      30153 ±  9%     -84.1%       4799 ± 58%  softirqs.CPU30.TIMER
> >   21317864 ±  4%     -28.1%   15318620 ±  6%  softirqs.CPU31.NET_RX
> >      46511 ± 49%    +365.4%     216450 ±  8%  softirqs.CPU31.SCHED
> >      29862 ± 10%     -83.9%       4819 ± 60%  softirqs.CPU31.TIMER
> >   21191794 ±  5%     -27.4%   15375714 ±  6%  softirqs.CPU32.NET_RX
> >      45002 ± 47%    +380.8%     216377 ±  8%  softirqs.CPU32.SCHED
> >      30188 ± 13%     -84.0%       4825 ± 60%  softirqs.CPU32.TIMER
> >   21594693 ±  3%     -28.5%   15432125 ±  6%  softirqs.CPU33.NET_RX
> >      46875 ± 48%    +361.2%     216187 ±  8%  softirqs.CPU33.SCHED
> >      30026 ± 10%     -84.0%       4803 ± 60%  softirqs.CPU33.TIMER
> >   21459214 ±  3%     -28.2%   15402703 ±  6%  softirqs.CPU34.NET_RX
> >      45836 ± 47%    +371.2%     215962 ±  8%  softirqs.CPU34.SCHED
> >      29899 ± 10%     -84.1%       4764 ± 58%  softirqs.CPU34.TIMER
> >   21334265 ±  4%     -28.2%   15311475 ±  6%  softirqs.CPU35.NET_RX
> >      46008 ± 47%    +369.7%     216108 ±  8%  softirqs.CPU35.SCHED
> >      30033 ±  9%     -83.8%       4872 ± 61%  softirqs.CPU35.TIMER
> >   21434454 ±  3%     -29.8%   15051817 ±  6%  softirqs.CPU36.NET_RX
> >      47102 ± 48%    +353.6%     213647 ±  8%  softirqs.CPU36.SCHED
> >      30159 ±  9%     -84.3%       4724 ± 59%  softirqs.CPU36.TIMER
> >   21446302 ±  3%     -28.5%   15324037 ±  6%  softirqs.CPU37.NET_RX
> >      46540 ± 48%    +365.2%     216506 ±  8%  softirqs.CPU37.SCHED
> >      29955 ±  9%     -84.3%       4688 ± 60%  softirqs.CPU37.TIMER
> >   21408080 ±  4%     -28.1%   15398099 ±  6%  softirqs.CPU38.NET_RX
> >      46514 ± 47%    +364.5%     216055 ±  8%  softirqs.CPU38.SCHED
> >      30224 ± 10%     -84.2%       4763 ± 59%  softirqs.CPU38.TIMER
> >   21488127 ±  3%     -28.2%   15418075 ±  6%  softirqs.CPU39.NET_RX
> >      46334 ± 49%    +366.6%     216186 ±  8%  softirqs.CPU39.SCHED
> >      29912 ± 10%     -84.1%       4757 ± 59%  softirqs.CPU39.TIMER
> >   20494991 ±  5%     -25.8%   15203629 ±  7%  softirqs.CPU4.NET_RX
> >      36606 ± 70%    +500.9%     219950 ±  8%  softirqs.CPU4.SCHED
> >      30735 ± 11%     -83.9%       4963 ± 64%  softirqs.CPU4.TIMER
> >   21463890 ±  3%     -28.3%   15380723 ±  6%  softirqs.CPU40.NET_RX
> >      46863 ± 48%    +361.7%     216368 ±  8%  softirqs.CPU40.SCHED
> >      30014 ±  9%     -84.1%       4763 ± 59%  softirqs.CPU40.TIMER
> >   21482678 ±  3%     -28.7%   15323146 ±  6%  softirqs.CPU41.NET_RX
> >      46944 ± 48%    +361.1%     216451 ±  8%  softirqs.CPU41.SCHED
> >      30167 ±  9%     -83.9%       4864 ± 59%  softirqs.CPU41.TIMER
> >   21294399 ±  4%     -28.3%   15278181 ±  6%  softirqs.CPU42.NET_RX
> >      45452 ± 47%    +376.3%     216501 ±  8%  softirqs.CPU42.SCHED
> >      29866 ± 10%     -82.6%       5185 ± 54%  softirqs.CPU42.TIMER
> >   21365880 ±  3%     -28.1%   15358507 ±  6%  softirqs.CPU43.NET_RX
> >      46603 ± 49%    +364.4%     216428 ±  8%  softirqs.CPU43.SCHED
> >      30073 ±  9%     -84.0%       4809 ± 59%  softirqs.CPU43.TIMER
> >   21520046 ±  3%     -28.2%   15449623 ±  6%  softirqs.CPU44.NET_RX
> >      46912 ± 48%    +361.0%     216249 ±  8%  softirqs.CPU44.SCHED
> >      29983 ± 10%     -83.8%       4865 ± 58%  softirqs.CPU44.TIMER
> >   21519719 ±  3%     -28.4%   15408388 ±  6%  softirqs.CPU45.NET_RX
> >      46866 ± 48%    +361.2%     216163 ±  8%  softirqs.CPU45.SCHED
> >      29935 ±  9%     -83.0%       5082 ± 55%  softirqs.CPU45.TIMER
> >   21419897 ±  3%     -28.3%   15362156 ±  6%  softirqs.CPU46.NET_RX
> >      46939 ± 48%    +361.6%     216649 ±  8%  softirqs.CPU46.SCHED
> >      29931 ± 10%     -83.4%       4981 ± 56%  softirqs.CPU46.TIMER
> >   21420535 ±  3%     -28.7%   15271884 ±  6%  softirqs.CPU47.NET_RX
> >      45846 ± 47%    +370.1%     215519 ±  8%  softirqs.CPU47.SCHED
> >      29858 ± 10%     -84.1%       4740 ± 60%  softirqs.CPU47.TIMER
> >   20312914 ±  5%     -25.7%   15093130 ±  7%  softirqs.CPU48.NET_RX
> >      36150 ± 69%    +505.3%     218836 ±  9%  softirqs.CPU48.SCHED
> >      30533 ± 12%     -84.7%       4657 ± 65%  softirqs.CPU48.TIMER
> >   20470745 ±  4%     -28.8%   14574011 ±  8%  softirqs.CPU49.NET_RX
> >      36483 ± 70%    +488.7%     214762 ± 11%  softirqs.CPU49.SCHED
> >      30893 ± 11%     -85.0%       4630 ± 54%  softirqs.CPU49.TIMER
> >   20156700 ±  7%     -24.7%   15172321 ±  7%  softirqs.CPU5.NET_RX
> >      35518 ± 68%    +517.8%     219449 ±  9%  softirqs.CPU5.SCHED
> >      30265 ± 13%     -83.0%       5157 ± 53%  softirqs.CPU5.TIMER
> >   20423756 ±  5%     -26.0%   15104088 ±  7%  softirqs.CPU50.NET_RX
> >      36375 ± 70%    +499.0%     217904 ±  9%  softirqs.CPU50.SCHED
> >      30692 ± 11%     -83.8%       4974 ± 56%  softirqs.CPU50.TIMER
> >   20528259 ±  5%     -27.9%   14802498 ± 11%  softirqs.CPU51.NET_RX
> >      36471 ± 71%    +481.1%     211940 ±  9%  softirqs.CPU51.SCHED
> >      30830 ± 11%     -84.8%       4699 ± 64%  softirqs.CPU51.TIMER
> >   20406992 ±  4%     -25.5%   15212005 ±  7%  softirqs.CPU52.NET_RX
> >      36584 ± 70%    +498.1%     218812 ±  9%  softirqs.CPU52.SCHED
> >      30640 ± 11%     -83.2%       5158 ± 60%  softirqs.CPU52.TIMER
> >   20215189 ±  6%     -25.1%   15148729 ±  7%  softirqs.CPU53.NET_RX
> >      35623 ± 68%    +515.2%     219165 ±  9%  softirqs.CPU53.SCHED
> >      30368 ± 13%     -82.7%       5247 ± 54%  softirqs.CPU53.TIMER
> >   20363531 ±  5%     -25.9%   15088299 ±  7%  softirqs.CPU54.NET_RX
> >      36829 ± 70%    +495.7%     219407 ±  9%  softirqs.CPU54.SCHED
> >      30604 ± 12%     -84.6%       4718 ± 63%  softirqs.CPU54.TIMER
> >   20446122 ±  5%     -26.0%   15123333 ±  7%  softirqs.CPU55.NET_RX
> >      36648 ± 70%    +496.4%     218572 ±  9%  softirqs.CPU55.SCHED
> >      30692 ± 11%     -84.3%       4807 ± 61%  softirqs.CPU55.TIMER
> >   20460809 ±  5%     -26.2%   15103440 ±  6%  softirqs.CPU56.NET_RX
> >      36546 ± 71%    +499.2%     218984 ±  9%  softirqs.CPU56.SCHED
> >      30796 ± 11%     -85.0%       4612 ± 60%  softirqs.CPU56.TIMER
> >   20579289 ±  5%     -25.7%   15290133 ±  7%  softirqs.CPU57.NET_RX
> >      36635 ± 70%    +497.9%     219057 ±  9%  softirqs.CPU57.SCHED
> >      30824 ± 11%     -84.1%       4887 ± 61%  softirqs.CPU57.TIMER
> >   20340209 ±  5%     -25.1%   15226056 ±  7%  softirqs.CPU58.NET_RX
> >      36661 ± 70%    +497.4%     219034 ±  9%  softirqs.CPU58.SCHED
> >      30643 ± 12%     -83.8%       4958 ± 67%  softirqs.CPU58.TIMER
> >   20407456 ±  5%     -31.2%   14032788 ± 14%  softirqs.CPU59.NET_RX
> >      36581 ± 70%    +445.7%     199631 ±  7%  softirqs.CPU59.SCHED
> >      30705 ± 11%     -84.8%       4662 ± 66%  softirqs.CPU59.TIMER
> >   20391632 ±  5%     -26.0%   15081271 ±  7%  softirqs.CPU6.NET_RX
> >      36990 ± 70%    +492.1%     219031 ±  9%  softirqs.CPU6.SCHED
> >      31051 ± 10%     -84.9%       4685 ± 66%  softirqs.CPU6.TIMER
> >   20313212 ±  5%     -25.8%   15082404 ±  7%  softirqs.CPU60.NET_RX
> >      36711 ± 70%    +497.9%     219487 ±  9%  softirqs.CPU60.SCHED
> >      30611 ± 12%     -77.9%       6758 ± 62%  softirqs.CPU60.TIMER
> >   20370636 ±  5%     -25.5%   15182632 ±  7%  softirqs.CPU61.NET_RX
> >      36456 ± 69%    +499.3%     218484 ±  9%  softirqs.CPU61.SCHED
> >      30655 ± 11%     -83.6%       5031 ± 56%  softirqs.CPU61.TIMER
> >   20447745 ±  5%     -25.8%   15175825 ±  7%  softirqs.CPU62.NET_RX
> >      36878 ± 70%    +491.8%     218266 ±  9%  softirqs.CPU62.SCHED
> >      30914 ± 12%     -83.8%       5002 ± 63%  softirqs.CPU62.TIMER
> >   20585734 ±  4%     -26.3%   15163240 ±  7%  softirqs.CPU63.NET_RX
> >      36766 ± 69%    +493.6%     218247 ±  9%  softirqs.CPU63.SCHED
> >      30783 ± 11%     -84.3%       4848 ± 63%  softirqs.CPU63.TIMER
> >   20529909 ±  5%     -26.0%   15201926 ±  7%  softirqs.CPU64.NET_RX
> >      36996 ± 69%    +492.3%     219138 ±  9%  softirqs.CPU64.SCHED
> >      31589 ± 12%     -84.3%       4944 ± 59%  softirqs.CPU64.TIMER
> >   20469049 ±  5%     -26.2%   15105347 ±  7%  softirqs.CPU65.NET_RX
> >      36816 ± 70%    +495.3%     219168 ±  9%  softirqs.CPU65.SCHED
> >      31163 ± 11%     -84.5%       4843 ± 62%  softirqs.CPU65.TIMER
> >   20451275 ±  5%     -26.1%   15117422 ±  7%  softirqs.CPU66.NET_RX
> >      36621 ± 70%    +498.4%     219158 ±  9%  softirqs.CPU66.SCHED
> >      31143 ± 11%     -84.5%       4812 ± 63%  softirqs.CPU66.TIMER
> >   20535600 ±  4%     -26.1%   15183175 ±  7%  softirqs.CPU67.NET_RX
> >      36716 ± 70%    +496.0%     218827 ±  9%  softirqs.CPU67.SCHED
> >      31123 ± 11%     -84.5%       4811 ± 62%  softirqs.CPU67.TIMER
> >   20606503 ±  4%     -25.9%   15263709 ±  7%  softirqs.CPU68.NET_RX
> >      36843 ± 71%    +493.6%     218696 ±  9%  softirqs.CPU68.SCHED
> >      31375 ± 10%     -84.5%       4850 ± 61%  softirqs.CPU68.TIMER
> >   20565997 ±  4%     -26.1%   15207162 ±  7%  softirqs.CPU69.NET_RX
> >      36903 ± 70%    +493.4%     218980 ±  9%  softirqs.CPU69.SCHED
> >      31377 ± 10%     -84.6%       4840 ± 63%  softirqs.CPU69.TIMER
> >   20398774 ±  5%     -26.3%   15029635 ±  7%  softirqs.CPU7.NET_RX
> >      36601 ± 69%    +495.4%     217915 ±  9%  softirqs.CPU7.SCHED
> >      30791 ± 11%     -84.9%       4640 ± 66%  softirqs.CPU7.TIMER
> >   20433298 ±  4%     -26.2%   15079571 ±  7%  softirqs.CPU70.NET_RX
> >      36471 ± 70%    +499.8%     218755 ±  9%  softirqs.CPU70.SCHED
> >      31125 ± 11%     -84.6%       4791 ± 61%  softirqs.CPU70.TIMER
> >   20514519 ±  4%     -26.6%   15064099 ±  7%  softirqs.CPU71.NET_RX
> >      36787 ± 70%    +493.4%     218283 ±  9%  softirqs.CPU71.SCHED
> >      31230 ± 11%     -84.3%       4914 ± 64%  softirqs.CPU71.TIMER
> >   21343935 ±  3%     -29.8%   14976558 ±  6%  softirqs.CPU72.NET_RX
> >      47439 ± 47%    +356.6%     216628 ±  8%  softirqs.CPU72.SCHED
> >      33558 ± 25%     -86.3%       4611 ± 62%  softirqs.CPU72.TIMER
> >   21417052 ±  3%     -28.3%   15349903 ±  6%  softirqs.CPU73.NET_RX
> >      46794 ± 48%    +362.8%     216549 ±  8%  softirqs.CPU73.SCHED
> >      30149 ± 10%     -83.8%       4893 ± 57%  softirqs.CPU73.TIMER
> >   21398667 ±  3%     -28.4%   15321598 ±  6%  softirqs.CPU74.NET_RX
> >     117019 ± 29%     -36.5%      74286 ± 24%  softirqs.CPU74.RCU
> >      46710 ± 47%    +363.0%     216256 ±  8%  softirqs.CPU74.SCHED
> >      30250 ± 10%     -84.3%       4756 ± 61%  softirqs.CPU74.TIMER
> >   21158934 ±  6%     -27.3%   15387291 ±  6%  softirqs.CPU75.NET_RX
> >     116793 ± 29%     -36.3%      74379 ± 24%  softirqs.CPU75.RCU
> >      46862 ± 48%    +361.5%     216265 ±  8%  softirqs.CPU75.SCHED
> >      29925 ± 11%     -83.8%       4847 ± 57%  softirqs.CPU75.TIMER
> >   21327908 ±  4%     -27.8%   15394469 ±  6%  softirqs.CPU76.NET_RX
> >      45803 ± 47%    +372.3%     216338 ±  8%  softirqs.CPU76.SCHED
> >      30109 ± 10%     -83.7%       4920 ± 57%  softirqs.CPU76.TIMER
> >   21477994 ±  4%     -28.6%   15330910 ±  6%  softirqs.CPU77.NET_RX
> >      46487 ± 47%    +365.6%     216466 ±  8%  softirqs.CPU77.SCHED
> >      30216 ± 10%     -84.4%       4726 ± 59%  softirqs.CPU77.TIMER
> >   21460057 ±  4%     -29.1%   15225183 ±  6%  softirqs.CPU78.NET_RX
> >      46884 ± 48%    +362.1%     216676 ±  8%  softirqs.CPU78.SCHED
> >      30421 ± 10%     -84.4%       4730 ± 60%  softirqs.CPU78.TIMER
> >   21541321 ±  3%     -28.9%   15319139 ±  6%  softirqs.CPU79.NET_RX
> >      46875 ± 48%    +362.6%     216857 ±  8%  softirqs.CPU79.SCHED
> >      30230 ±  9%     -82.5%       5291 ± 76%  softirqs.CPU79.TIMER
> >   20493021 ±  4%     -26.0%   15160469 ±  7%  softirqs.CPU8.NET_RX
> >      36579 ± 70%    +498.1%     218781 ±  9%  softirqs.CPU8.SCHED
> >      30682 ± 11%     -84.4%       4790 ± 59%  softirqs.CPU8.TIMER
> >   21172364 ±  7%     -27.3%   15402317 ±  6%  softirqs.CPU80.NET_RX
> >      46254 ± 47%    +368.0%     216468 ±  8%  softirqs.CPU80.SCHED
> >      29465 ± 12%     -83.2%       4953 ± 56%  softirqs.CPU80.TIMER
> >   21567300 ±  3%     -28.5%   15430307 ±  6%  softirqs.CPU81.NET_RX
> >      46695 ± 48%    +362.7%     216072 ±  8%  softirqs.CPU81.SCHED
> >      30093 ±  9%     -84.0%       4801 ± 60%  softirqs.CPU81.TIMER
> >   21491215 ±  3%     -28.3%   15417714 ±  6%  softirqs.CPU82.NET_RX
> >      46133 ± 47%    +368.7%     216210 ±  8%  softirqs.CPU82.SCHED
> >      29921 ± 10%     -76.8%       6934 ± 67%  softirqs.CPU82.TIMER
> >   21460124 ±  3%     -28.5%   15348172 ±  6%  softirqs.CPU83.NET_RX
> >      46491 ± 48%    +365.6%     216476 ±  8%  softirqs.CPU83.SCHED
> >      29970 ± 10%     -83.7%       4888 ± 56%  softirqs.CPU83.TIMER
> >   21445385 ±  4%     -30.8%   14839411 ±  9%  softirqs.CPU84.NET_RX
> >      46941 ± 48%    +350.7%     211545 ±  9%  softirqs.CPU84.SCHED
> >      30037 ± 10%     -84.3%       4718 ± 60%  softirqs.CPU84.TIMER
> >   21551635 ±  3%     -28.9%   15333781 ±  6%  softirqs.CPU85.NET_RX
> >      46814 ± 48%    +362.1%     216323 ±  8%  softirqs.CPU85.SCHED
> >      29996 ± 10%     -83.9%       4817 ± 59%  softirqs.CPU85.TIMER
> >   21556404 ±  3%     -28.6%   15382397 ±  6%  softirqs.CPU86.NET_RX
> >      46934 ± 48%    +360.7%     216231 ±  8%  softirqs.CPU86.SCHED
> >      30053 ±  9%     -83.7%       4900 ± 55%  softirqs.CPU86.TIMER
> >   21549266 ±  3%     -28.5%   15417462 ±  6%  softirqs.CPU87.NET_RX
> >      46429 ± 49%    +366.0%     216351 ±  8%  softirqs.CPU87.SCHED
> >      29937 ± 10%     -83.9%       4806 ± 58%  softirqs.CPU87.TIMER
> >   21537823 ±  3%     -28.7%   15361608 ±  6%  softirqs.CPU88.NET_RX
> >      46887 ± 48%    +360.1%     215706 ±  8%  softirqs.CPU88.SCHED
> >      29981 ±  9%     -84.1%       4781 ± 59%  softirqs.CPU88.TIMER
> >   21559950 ±  3%     -29.0%   15300783 ±  6%  softirqs.CPU89.NET_RX
> >      46878 ± 48%    +361.6%     216403 ±  8%  softirqs.CPU89.SCHED
> >      30017 ±  9%     -84.1%       4783 ± 59%  softirqs.CPU89.TIMER
> >   20571529 ±  5%     -25.7%   15286193 ±  7%  softirqs.CPU9.NET_RX
> >      36585 ± 70%    +498.6%     219007 ±  9%  softirqs.CPU9.SCHED
> >      30724 ± 11%     -83.7%       5009 ± 58%  softirqs.CPU9.TIMER
> >   21116816 ±  5%     -27.5%   15303522 ±  6%  softirqs.CPU90.NET_RX
> >      46158 ± 48%    +368.6%     216275 ±  8%  softirqs.CPU90.SCHED
> >      29583 ± 11%     -83.6%       4842 ± 58%  softirqs.CPU90.TIMER
> >   21247319 ±  4%     -27.6%   15382623 ±  6%  softirqs.CPU91.NET_RX
> >      46803 ± 48%    +362.0%     216238 ±  8%  softirqs.CPU91.SCHED
> >      29690 ± 10%     -83.6%       4856 ± 58%  softirqs.CPU91.TIMER
> >   21660224 ±  3%     -28.6%   15460344 ±  6%  softirqs.CPU92.NET_RX
> >      46935 ± 48%    +359.9%     215852 ±  8%  softirqs.CPU92.SCHED
> >      30143 ±  9%     -84.0%       4814 ± 59%  softirqs.CPU92.TIMER
> >   21653985 ±  3%     -28.9%   15405532 ±  6%  softirqs.CPU93.NET_RX
> >      46972 ± 48%    +359.5%     215821 ±  8%  softirqs.CPU93.SCHED
> >      30034 ± 10%     -83.5%       4964 ± 59%  softirqs.CPU93.TIMER
> >   21602751 ±  3%     -28.8%   15385232 ±  6%  softirqs.CPU94.NET_RX
> >      46892 ± 48%    +362.2%     216728 ±  8%  softirqs.CPU94.SCHED
> >      29959 ± 10%     -83.6%       4921 ± 58%  softirqs.CPU94.TIMER
> >   21277877 ±  4%     -28.4%   15224899 ±  6%  softirqs.CPU95.NET_RX
> >      45607 ± 48%    +373.5%     215952 ±  8%  softirqs.CPU95.SCHED
> >      29711 ± 10%     -83.0%       5065 ± 55%  softirqs.CPU95.TIMER
> >  2.009e+09           -27.4%   1.46e+09 ±  5%  softirqs.NET_RX
> >    3996567 ± 14%    +421.3%   20835185 ±  8%  softirqs.SCHED
> >    2929735 ±  3%     -83.8%     474988 ± 53%  softirqs.TIMER
> > 
> > 
> >                                                                    
> >              
> >                               tbench.throughput-
> > MB_sec                          
> >                                                                    
> >              
> >   16000 +--------------------------------------------------------
> > -----------+   
> >         |                  .+.          .+.                        
> >          |   
> >   15000 |.+.+..+. .+.+.+..+   +.+.+..+.+   +. .+..+.
> > .+.+.    .+.+.+.+..+.+.|   
> >         |        +                           +      +     +..+     
> >          |   
> >   14000 |-
> > +                                                                 |
> >    
> >         |                                                          
> >          |   
> >   13000 |-
> > +                                                                 |
> >    
> >         |                                                          
> >          |   
> >   12000 |-
> > +                                                                 |
> >    
> >         | O                     O O    O
> > O                           O      |   
> >   11000 |-+            O    O
> > O                     O          O        O   |   
> >         |      O O O O    O          O     O O O  O   O O
> > O      O        O |   
> >   10000 |-+
> > O                                                O     O        |  
> >  
> >         |                                                          
> >          |   
> >    9000 +--------------------------------------------------------
> > -----------+   
> >                                                                    
> >              
> >                                                                    
> >                                                                    
> >                           
> >                                tbench.time.user_time               
> >              
> >                                                                    
> >              
> >   5200 +-----------------------------------------------------------
> > ---------+   
> >   5000 |-+.+..          .+.+   +..   .+.+..     + +    + :      .+.
> > .+..   .|   
> >        |.+    +.+. .+..+    + +   +.+      +. .+   +
> > .+  :    .+   +    +.+ |   
> >   4800 |-
> > +        +          +               +      +     +.+.              
> > |   
> >   4600 |-
> > +                                                                  
> > |   
> >        |                                                           
> >          |   
> >   4400 |-
> > +                                                                  
> > |   
> >   4200 |-
> > +                                                                  
> > |   
> >   4000 |-+                     O  O   O
> > O                            O      |   
> >        |
> > O                   O                      O          O        O   
> > |   
> >   3800 |-+             O   O                   O
> > O                          |   
> >   3600 |-+    O O   O    O                   O        O
> > O                 O |   
> >        |   O      O                                       O
> > O    O          |   
> >   3400 |-
> > +                          O      O                       O        
> > |   
> >   3200 +-----------------------------------------------------------
> > ---------+   
> >                                                                    
> >              
> >                                                                    
> >                                                                    
> >                           
> >                                tbench.time.system_time             
> >              
> >                                                                    
> >              
> >   34000 +--------------------------------------------------------
> > -----------+   
> >   32000 |.+     .+  :            .+.. .+.+.
> > .+.    .+.        .+   +.+..+.+.|   
> >         | +.+..+    :
> > .+..+.+.+.+    +     +   +..+   +.+.+..+              |   
> >   30000 |-
> > +          +                                                      |
> >    
> >   28000 |-
> > +                                                                 |
> >    
> >         |                                                          
> >          |   
> >   26000 |-
> > +                                                                 |
> >    
> >   24000 |-
> > +                                                                 |
> >    
> >   22000 |-
> > +                                                                 |
> >    
> >         |                                                          
> >          |   
> >   20000 |-
> > +                       O      O                           O      |
> >    
> >   18000 |-
> > O                 O   O      O            O          O            |
> >    
> >         |              O      O      O     O O
> > O  O     O               O O |   
> >   16000 |-+ O  O O O
> > O    O                           O   O  O   O          |   
> >   14000 +--------------------------------------------------------
> > -----------+   
> >                                                                    
> >              
> >                                                                    
> >                                                                    
> >                           
> >                       tbench.time.percent_of_cpu_this_job_got      
> >              
> >                                                                    
> >              
> >   5500 +-----------------------------------------------------------
> > ---------+   
> >        |.       +.+              .+. .+.+..
> > .+     .+         .+.+.    .+.+.|   
> >   5000 |-+.    +   +  .+.+.+.+.+.   +      +  + .+.  +
> > .+.+.+.     +.+.     |   
> >        |   +..+     +.                         +      +            
> >          |   
> >        |                                                           
> >          |   
> >   4500 |-
> > +                                                                  
> > |   
> >        |                                                           
> >          |   
> >   4000 |-
> > +                                                                  
> > |   
> >        |                                                           
> >          |   
> >   3500 |-
> > +                                                                  
> > |   
> >        |                                                           
> >   O      |   
> >        |                       O  O   O
> > O                      O            |   
> >   3000 |-O                 O O                 O
> > O  O                   O   |   
> >        |      O O O O  O O          O      O O        O O O
> > O    O        O |   
> >   2500 +-----------------------------------------------------------
> > ---------+   
> >                                                                    
> >              
> >                                                                    
> >                                                                    
> >                           
> >                         tbench.time.voluntary_context_switches     
> >              
> >                                                                    
> >              
> >   1.4e+09 +------------------------------------------------------
> > -----------+   
> >           |   O
> > O    O     O                                 O     O        |   
> >   1.2e+09 |-O      O   O O     O      O     O O O O    O O
> > O     O      O O |   
> >           |                  O   O      O            O         O   
> >          |   
> >     1e+09 |-
> > +                       O     O                           O     |  
> >  
> >           |                                                        
> >          |   
> >     8e+08 |-
> > +                                                               |  
> >  
> >           |                                                        
> >          |   
> >     6e+08 |-
> > +                                                               |  
> >  
> >           |                                                        
> >          |   
> >     4e+08 |-
> > +                                                               |  
> >  
> >           |  .+.+..    +.                       +.     +.          
> >          |   
> >     2e+08 |.+         +  +.+.+.+.+.. .+.   .+. +  +..
> > +  +.+.+.   .+..+.   .|   
> >           |        +.+              +   +.+   +      +         +.+ 
> >      +.+ |   
> >         0 +------------------------------------------------------
> > -----------+   
> >                                                                    
> >              
> >                                                                    
> >                                                                    
> >                           
> >                        tbench.time.involuntary_context_switches    
> >              
> >                                                                    
> >              
> >     2e+09 +------------------------------------------------------
> > -----------+   
> >   1.8e+09 |.+.    .+  + .+.+.+.+.+..+.+.+   +.+. .+..+.
> > .+.   .+   +..+.+.+.|   
> >           |   +.+.     +                        +      +   +.+     
> >          |   
> >   1.6e+09 |-
> > +                                                               |  
> >  
> >   1.4e+09 |-
> > +                                                               |  
> >  
> >           |                                                        
> >          |   
> >   1.2e+09 |-
> > +                                                               |  
> >  
> >     1e+09 |-
> > +                                                               |  
> >  
> >     8e+08 |-
> > +                                                               |  
> >  
> >           |                                                        
> >          |   
> >     6e+08 |-
> > +                       O     O                           O     |  
> >  
> >     4e+08 |-
> > O                O   O      O            O         O            |  
> >  
> >           |              O     O      O     O   O
> > O      O              O   |   
> >     2e+08 |-+ O O  O O O   O                  O        O   O
> > O   O        O |   
> >         0 +------------------------------------------------------
> > -----------+   
> >                                                                    
> >              
> >                                                                    
> >              
> > [*] bisect-good sample
> > [O] bisect-bad  sample
> > 
> > 
> > 
> > Disclaimer:
> > Results have been estimated based on internal Intel analysis and
> > are provided
> > for informational purposes only. Any difference in system hardware
> > or software
> > design or configuration may affect actual performance.
> > 
> > 
> > ---
> > 0DAY/LKP+ Test Infrastructure                   Open Source
> > Technology Center
> > https://lists.01.org/hyperkitty/list/lkp@lists.01.org       Intel
> > Corporation
> > 
> > Thanks,
> > Oliver Sang
> > 
> > 
> > 
> > _______________________________________________
> > LKP mailing list -- lkp@...ts.01.org
> > To unsubscribe send an email to lkp-leave@...ts.01.org
>  
-- 
All Rights Reversed.

Download attachment "signature.asc" of type "application/pgp-signature" (489 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ