| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <CAHbLzkpF4tuuukmhqC1soeVUaS4D5XtDjNDugy9t9D77s_xG8g@mail.gmail.com>
Date: Wed, 21 Dec 2022 08:45:39 -0800
From: Yang Shi <shy828301@...il.com>
To: Yujie Liu <yujie.liu@...el.com>
Cc: Liam Howlett <liam.howlett@...cle.com>,
Linus Torvalds <torvalds@...ux-foundation.org>,
"oe-lkp@...ts.linux.dev" <oe-lkp@...ts.linux.dev>,
"lkp@...el.com" <lkp@...el.com>,
Nathan Chancellor <nathan@...nel.org>,
"Huang, Ying" <ying.huang@...el.com>,
Rik van Riel <riel@...riel.com>,
Andrew Morton <akpm@...ux-foundation.org>,
"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
"linux-mm@...ck.org" <linux-mm@...ck.org>,
"feng.tang@...el.com" <feng.tang@...el.com>,
"zhengjun.xing@...ux.intel.com" <zhengjun.xing@...ux.intel.com>,
"Yin, Fengwei" <fengwei.yin@...el.com>
Subject: Re: [linus:master] [mm] 0ba09b1733: will-it-scale.per_thread_ops
-21.1% regression in mmap1 benchmark
On Tue, Dec 20, 2022 at 6:36 PM Yujie Liu <yujie.liu@...el.com> wrote:
>
> On Tue, Dec 20, 2022 at 08:48:53PM +0000, Liam Howlett wrote:
> > * Yang Shi <shy828301@...il.com> [221220 13:04]:
> > > On Mon, Dec 19, 2022 at 3:30 AM kernel test robot <yujie.liu@...el.com> wrote:
> > > >
> > > > Greetings,
> > > >
> > > > Please note that we reported a regression in will-it-scale malloc1
> > > > benchmark on below commit
> > > > f35b5d7d676e ("mm: align larger anonymous mappings on THP boundaries")
> > > > at
> > > > https://lore.kernel.org/all/202210181535.7144dd15-yujie.liu@intel.com/
> > > > and Nathan reported a kbuild slowdown under clang toolchain at
> > > > https://lore.kernel.org/all/Y1DNQaoPWxE+rGce@dev-arch.thelio-3990X/
> > > > That commit was finally reverted.
> > > >
> > > > When we tested the revert commit, the score in malloc1 benchmark
> > > > recovered, but we observed another regression in mmap1 benchmark.
> > > >
> > > > "Yin, Fengwei" helped to check and got below clues:
> > > >
> > > > 1. The regression is related with the VMA merge with prev/next
> > > > VMA when doing mmap.
> > > >
> > > > 2. Before the patch reverted, almost all the VMA for 128M mapping
> > > > can't be merged with prev/next VMA. So always create new VMA.
> > > > With the patch reverted, most VMA for 128 mapping can be merged.
> > > >
> > > > It looks like VMA merging introduce more latency comparing to
> > > > creating new VMA.
> > > >
> > > > 3. If force to create new VMA with patch reverted, the result of
> > > > mmap1_thread is restored.
> > > >
> > > > 4. The thp_get_unmapped_area() adds a padding to request mapping
> > > > length. The padding is 2M in general. I believe this padding
> > > > break VMA merging behavior.
> > > >
> > > > 5. No idea about why the difference of the two path (VMA merging
> > > > vs New VMA) is not shown in perf data
> > >
> > > IIRC thp_get_unmapped_area() has been behaving like that for years.
> > > The other change between the problematic commit and the revert commit,
> > > which might have an impact to VMA merging, is maple tree. Did you try to
> > > bisect further?
> > >
> >
> > There was also the work done to vma_merge(). Vlastimil (added to Cc)
> > tracked down an issue with mremap() quite recently [1], which sounds a
> > lot like what is happening here - especially with the padding.
> >
> > >
> > > BTW, is this similar to
> > > https://lore.kernel.org/linux-mm/20221219180857.u6opzhqqbbfxdj3h@revolver/T/#t
> > > ?
> >
> > Yes, it looks to be similar. I'm surprised the mmap1 benchmark was
> > altered with this commit, or am I reading this email incorrectly?
>
> We caught two mmap1 regressions on mailine, please see the data below:
>
> 830b3c68c1fb1 Linux 6.1 2085 2355 2088
> 76dcd734eca23 Linux 6.1-rc8 2093 2082 2094 2073 2304 2088
> 0ba09b1733878 Revert "mm: align larger anonymous mappings on THP boundaries" 2124 2286 2086 2114 2065 2081
> 23393c6461422 char: tpm: Protect tpm_pm_suspend with locks 2756 2711 2689 2696 2660 2665
> b7b275e60bcd5 Linux 6.1-rc7 2670 2656 2720 2691 2667
> ...
> 9abf2313adc1c Linux 6.1-rc1 2725 2717 2690 2691 2710
> 3b0e81a1cdc9a mmap: change zeroing of maple tree in __vma_adjust() 2736 2781 2748
> 524e00b36e8c5 mm: remove rb tree. 2747 2744 2747
> 0c563f1480435 proc: remove VMA rbtree use from nommu
> d0cf3dd47f0d5 damon: convert __damon_va_three_regions to use the VMA iterator
> 3499a13168da6 mm/mmap: use maple tree for unmapped_area{_topdown}
> 7fdbd37da5c6f mm/mmap: use the maple tree for find_vma_prev() instead of the rbtree
> f39af05949a42 mm: add VMA iterator
> d4af56c5c7c67 mm: start tracking VMAs with maple tree
> e15e06a839232 lib/test_maple_tree: add testing for maple tree 4638 4628 4502
> 9832fb87834e2 mm/demotion: expose memory tier details via sysfs 4625 4509 4548
> 4fe89d07dcc28 Linux 6.0 4385 4205 4348 4228 4504
>
>
> The first regression was between v6.0 and v6.1-rc1. The score dropped
> from 4600 to 2700, and bisected to the patches switching from rb tree to
> maple tree. This was reported at
> https://lore.kernel.org/oe-lkp/202212191714.524e00b3-yujie.liu@intel.com/
> Thanks for the explanation that it is an expected regression as a trade
> off to benefit read performance.
>
> The second regression was between v6.1-rc7 and v6.1-rc8. The score
> dropped from 2700 to 2100, and bisected to this "Revert "mm: align larger
> anonymous mappings on THP boundaries"" commit.
So it means "mm: align larger anonymous mappings on THP boundaries"
actually improved the mmap1 benchmark? But it caused regression for
other usecase, for example, building kernel with clang, which is a
regression for a real life usecase.
>
> > The trace below does not seem to show what RedHad [2] found in its
> > testing.
> >
> > [1]. https://lore.kernel.org/all/20221216163227.24648-1-vbabka@suse.cz/T/#u
> > [2]. https://bugzilla.redhat.com/show_bug.cgi?id=2149636
>
> Thanks for the info in [2], looks it is an overall regression between
> v6.0 and v6.1, not sure if it could be the result of adding up the two
> regressions above?
>
>
> --
> Best Regards,
> Yujie
>
> > >
> > > >
> > > > Please check below report for details.
> > > >
> > > >
> > > > FYI, we noticed a -21.1% regression of will-it-scale.per_thread_ops due to commit:
> > > >
> > > > commit: 0ba09b1733878afe838fe35c310715fda3d46428 ("Revert "mm: align larger anonymous mappings on THP boundaries"")
> > > > https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
> > > >
> > > > in testcase: will-it-scale
> > > > on test machine: 104 threads 2 sockets (Skylake) with 192G memory
> > > > with following parameters:
> > > >
> > > > nr_task: 50%
> > > > mode: thread
> > > > test: mmap1
> > > > cpufreq_governor: performance
> > > >
> > > > test-description: Will It Scale takes a testcase and runs it from 1 through to n parallel copies to see if the testcase will scale. It builds both a process and threads based test in order to see any differences between the two.
> > > > test-url: https://github.com/antonblanchard/will-it-scale
> > > >
> > > > In addition to that, the commit also has significant impact on the following tests:
> > > >
> > > > +------------------+------------------------------------------------------------------------------------------------+
> > > > | testcase: change | will-it-scale: will-it-scale.per_process_ops 1943.6% improvement |
> > > > | test machine | 128 threads 4 sockets Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz (Ice Lake) with 256G memory |
> > > > | test parameters | cpufreq_governor=performance |
> > > > | | mode=process |
> > > > | | nr_task=50% |
> > > > | | test=malloc1 |
> > > > +------------------+------------------------------------------------------------------------------------------------+
> > > > | testcase: change | unixbench: unixbench.score 2.6% improvement |
> > > > | test machine | 16 threads 1 sockets Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz (Coffee Lake) with 32G memory |
> > > > | test parameters | cpufreq_governor=performance |
> > > > | | nr_task=30% |
> > > > | | runtime=300s |
> > > > | | test=shell8 |
> > > > +------------------+------------------------------------------------------------------------------------------------+
> > > > | testcase: change | phoronix-test-suite: phoronix-test-suite.build-eigen.0.seconds 9.1% regression |
> > > > | test machine | 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz (Cascade Lake) with 512G memory |
> > > > | test parameters | cpufreq_governor=performance |
> > > > | | test=build-eigen-1.1.0 |
> > > > +------------------+------------------------------------------------------------------------------------------------+
> > > > | testcase: change | will-it-scale: will-it-scale.per_process_ops 2882.9% improvement |
> > > > | test machine | 88 threads 2 sockets Intel(R) Xeon(R) Gold 6238M CPU @ 2.10GHz (Cascade Lake) with 128G memory |
> > > > | test parameters | cpufreq_governor=performance |
> > > > | | mode=process |
> > > > | | nr_task=100% |
> > > > | | test=malloc1 |
> > > > +------------------+------------------------------------------------------------------------------------------------+
> > > > | testcase: change | will-it-scale: will-it-scale.per_process_ops 12.7% improvement |
> > > > | test machine | 88 threads 2 sockets Intel(R) Xeon(R) Gold 6238M CPU @ 2.10GHz (Cascade Lake) with 128G memory |
> > > > | test parameters | cpufreq_governor=performance |
> > > > | | mode=process |
> > > > | | nr_task=50% |
> > > > | | test=mmap1 |
> > > > +------------------+------------------------------------------------------------------------------------------------+
> > > > | testcase: change | stress-ng: stress-ng.pthread.ops_per_sec 600.6% improvement |
> > > > | test machine | 128 threads 2 sockets Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz (Ice Lake) with 128G memory |
> > > > | test parameters | class=scheduler |
> > > > | | cpufreq_governor=performance |
> > > > | | nr_threads=100% |
> > > > | | sc_pid_max=4194304 |
> > > > | | test=pthread |
> > > > | | testtime=60s |
> > > > +------------------+------------------------------------------------------------------------------------------------+
> > > > | testcase: change | will-it-scale: will-it-scale.per_process_ops 601.0% improvement |
> > > > | test machine | 104 threads 2 sockets (Skylake) with 192G memory |
> > > > | test parameters | cpufreq_governor=performance |
> > > > | | mode=process |
> > > > | | nr_task=50% |
> > > > | | test=malloc1 |
> > > > +------------------+------------------------------------------------------------------------------------------------+
> > > >
> > > >
> > > > Details are as below:
> > > >
> > > > =========================================================================================
> > > > compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase:
> > > > gcc-11/performance/x86_64-rhel-8.3/thread/50%/debian-11.1-x86_64-20220510.cgz/lkp-skl-fpga01/mmap1/will-it-scale
> > > >
> > > > commit:
> > > > 23393c6461 ("char: tpm: Protect tpm_pm_suspend with locks")
> > > > 0ba09b1733 ("Revert "mm: align larger anonymous mappings on THP boundaries"")
> > > >
> > > > 23393c6461422df5 0ba09b1733878afe838fe35c310
> > > > ---------------- ---------------------------
> > > > %stddev %change %stddev
> > > > \ | \
> > > > 140227 -21.1% 110582 ą 3% will-it-scale.52.threads
> > > > 49.74 +0.1% 49.78 will-it-scale.52.threads_idle
> > > > 2696 -21.1% 2126 ą 3% will-it-scale.per_thread_ops
> > > > 301.30 -0.0% 301.26 will-it-scale.time.elapsed_time
> > > > 301.30 -0.0% 301.26 will-it-scale.time.elapsed_time.max
> > > > 3.67 ą 71% -22.7% 2.83 ą 47% will-it-scale.time.involuntary_context_switches
> > > > 0.67 ą165% -75.0% 0.17 ą223% will-it-scale.time.major_page_faults
> > > > 9772 -0.7% 9702 will-it-scale.time.maximum_resident_set_size
> > > > 7274 -0.3% 7254 will-it-scale.time.minor_page_faults
> > > > 4096 +0.0% 4096 will-it-scale.time.page_size
> > > > 0.04 ą 16% -4.0% 0.04 will-it-scale.time.system_time
> > > > 0.06 ą 24% -11.8% 0.05 ą 16% will-it-scale.time.user_time
> > > > 102.83 +1.9% 104.83 ą 2% will-it-scale.time.voluntary_context_switches
> > > > 140227 -21.1% 110582 ą 3% will-it-scale.workload
> > > > 1.582e+10 +0.1% 1.584e+10 cpuidle..time
> > > > 33034032 -0.0% 33021393 cpuidle..usage
> > > > 10.00 +0.0% 10.00 dmesg.bootstage:last
> > > > 172.34 +0.1% 172.58 dmesg.timestamp:last
> > > > 10.00 +0.0% 10.00 kmsg.bootstage:last
> > > > 172.34 +0.1% 172.58 kmsg.timestamp:last
> > > > 362.22 +0.0% 362.25 uptime.boot
> > > > 21363 +0.1% 21389 uptime.idle
> > > > 55.94 +0.2% 56.06 boot-time.boot
> > > > 38.10 +0.2% 38.19 boot-time.dhcp
> > > > 5283 +0.2% 5295 boot-time.idle
> > > > 1.11 -0.1% 1.11 boot-time.smp_boot
> > > > 50.14 +0.0 50.16 mpstat.cpu.all.idle%
> > > > 0.03 ą223% -0.0 0.00 ą223% mpstat.cpu.all.iowait%
> > > > 1.02 +0.0 1.03 mpstat.cpu.all.irq%
> > > > 0.03 ą 4% -0.0 0.02 mpstat.cpu.all.soft%
> > > > 48.59 +0.0 48.61 mpstat.cpu.all.sys%
> > > > 0.20 ą 2% -0.0 0.17 ą 4% mpstat.cpu.all.usr%
> > > > 0.00 -100.0% 0.00 numa-numastat.node0.interleave_hit
> > > > 328352 ą 15% -7.2% 304842 ą 20% numa-numastat.node0.local_node
> > > > 374230 ą 6% -4.2% 358578 ą 7% numa-numastat.node0.numa_hit
> > > > 45881 ą 75% +17.1% 53735 ą 69% numa-numastat.node0.other_node
> > > > 0.00 -100.0% 0.00 numa-numastat.node1.interleave_hit
> > > > 381812 ą 13% +5.9% 404461 ą 14% numa-numastat.node1.local_node
> > > > 430007 ą 5% +3.4% 444810 ą 5% numa-numastat.node1.numa_hit
> > > > 48195 ą 71% -16.3% 40348 ą 92% numa-numastat.node1.other_node
> > > > 301.30 -0.0% 301.26 time.elapsed_time
> > > > 301.30 -0.0% 301.26 time.elapsed_time.max
> > > > 3.67 ą 71% -22.7% 2.83 ą 47% time.involuntary_context_switches
> > > > 0.67 ą165% -75.0% 0.17 ą223% time.major_page_faults
> > > > 9772 -0.7% 9702 time.maximum_resident_set_size
> > > > 7274 -0.3% 7254 time.minor_page_faults
> > > > 4096 +0.0% 4096 time.page_size
> > > > 0.04 ą 16% -4.0% 0.04 time.system_time
> > > > 0.06 ą 24% -11.8% 0.05 ą 16% time.user_time
> > > > 102.83 +1.9% 104.83 ą 2% time.voluntary_context_switches
> > > > 50.00 +0.0% 50.00 vmstat.cpu.id
> > > > 49.00 +0.0% 49.00 vmstat.cpu.sy
> > > > 0.00 -100.0% 0.00 vmstat.cpu.us
> > > > 0.00 -100.0% 0.00 vmstat.cpu.wa
> > > > 12.50 ą100% -66.7% 4.17 ą223% vmstat.io.bi
> > > > 3.33 ą141% -55.0% 1.50 ą223% vmstat.io.bo
> > > > 6.00 ą 47% -16.7% 5.00 ą 44% vmstat.memory.buff
> > > > 4150651 -0.1% 4148516 vmstat.memory.cache
> > > > 1.912e+08 +0.1% 1.913e+08 vmstat.memory.free
> > > > 0.00 -100.0% 0.00 vmstat.procs.b
> > > > 50.50 -0.3% 50.33 vmstat.procs.r
> > > > 8274 ą 2% +1.2% 8371 ą 4% vmstat.system.cs
> > > > 211078 -0.1% 210826 vmstat.system.in
> > > > 1399 +0.0% 1399 turbostat.Avg_MHz
> > > > 50.12 +0.0 50.13 turbostat.Busy%
> > > > 2799 -0.0% 2798 turbostat.Bzy_MHz
> > > > 208677 ą 13% +1112.3% 2529776 ą194% turbostat.C1
> > > > 0.03 ą 89% +0.3 0.36 ą203% turbostat.C1%
> > > > 27078371 ą 15% -22.0% 21125809 ą 51% turbostat.C1E
> > > > 37.41 ą 33% -9.4 28.04 ą 62% turbostat.C1E%
> > > > 5088326 ą 84% +63.1% 8298766 ą 77% turbostat.C6
> > > > 12.59 ą 99% +9.1 21.69 ą 78% turbostat.C6%
> > > > 49.79 -0.1% 49.75 turbostat.CPU%c1
> > > > 0.08 ą 71% +37.3% 0.12 ą 78% turbostat.CPU%c6
> > > > 43.67 -0.4% 43.50 turbostat.CoreTmp
> > > > 0.03 +0.0% 0.03 turbostat.IPC
> > > > 64483530 -0.2% 64338768 turbostat.IRQ
> > > > 647657 ą 2% +63.2% 1057048 ą 98% turbostat.POLL
> > > > 0.01 +0.0 0.05 ą178% turbostat.POLL%
> > > > 0.01 ą223% +200.0% 0.04 ą147% turbostat.Pkg%pc2
> > > > 0.01 ą223% +140.0% 0.02 ą165% turbostat.Pkg%pc6
> > > > 44.17 +0.4% 44.33 turbostat.PkgTmp
> > > > 284.98 +0.1% 285.28 turbostat.PkgWatt
> > > > 26.78 +0.4% 26.89 turbostat.RAMWatt
> > > > 2095 +0.0% 2095 turbostat.TSC_MHz
> > > > 49585 ą 7% +1.1% 50139 ą 7% meminfo.Active
> > > > 49182 ą 7% +1.4% 49889 ą 7% meminfo.Active(anon)
> > > > 402.33 ą 99% -37.9% 250.00 ą123% meminfo.Active(file)
> > > > 290429 -33.7% 192619 meminfo.AnonHugePages
> > > > 419654 -25.9% 311054 meminfo.AnonPages
> > > > 6.00 ą 47% -16.7% 5.00 ą 44% meminfo.Buffers
> > > > 4026046 -0.1% 4023990 meminfo.Cached
> > > > 98360160 +0.0% 98360160 meminfo.CommitLimit
> > > > 4319751 +0.4% 4337801 meminfo.Committed_AS
> > > > 1.877e+08 -0.1% 1.875e+08 meminfo.DirectMap1G
> > > > 14383445 ą 12% +0.7% 14491306 ą 4% meminfo.DirectMap2M
> > > > 1042426 ą 9% +6.4% 1109328 ą 7% meminfo.DirectMap4k
> > > > 4.00 ą141% -50.0% 2.00 ą223% meminfo.Dirty
> > > > 2048 +0.0% 2048 meminfo.Hugepagesize
> > > > 434675 -26.3% 320518 meminfo.Inactive
> > > > 431330 -26.0% 319346 meminfo.Inactive(anon)
> > > > 3344 ą 95% -65.0% 1171 ą186% meminfo.Inactive(file)
> > > > 124528 -0.1% 124460 meminfo.KReclaimable
> > > > 18433 +0.7% 18559 meminfo.KernelStack
> > > > 40185 ą 2% -0.9% 39837 meminfo.Mapped
> > > > 1.903e+08 +0.1% 1.904e+08 meminfo.MemAvailable
> > > > 1.912e+08 +0.1% 1.913e+08 meminfo.MemFree
> > > > 1.967e+08 +0.0% 1.967e+08 meminfo.MemTotal
> > > > 5569412 -1.8% 5466754 meminfo.Memused
> > > > 4763 -5.7% 4489 meminfo.PageTables
> > > > 51956 +0.0% 51956 meminfo.Percpu
> > > > 124528 -0.1% 124460 meminfo.SReclaimable
> > > > 197128 +0.1% 197293 meminfo.SUnreclaim
> > > > 57535 ą 7% +0.8% 57986 ą 6% meminfo.Shmem
> > > > 321657 +0.0% 321754 meminfo.Slab
> > > > 3964769 -0.0% 3964586 meminfo.Unevictable
> > > > 3.436e+10 +0.0% 3.436e+10 meminfo.VmallocTotal
> > > > 280612 +0.1% 280841 meminfo.VmallocUsed
> > > > 6194619 -2.0% 6071944 meminfo.max_used_kB
> > > > 2626 ą 28% -7.7% 2423 ą 11% numa-meminfo.node0.Active
> > > > 2361 ą 20% -5.3% 2236 ą 10% numa-meminfo.node0.Active(anon)
> > > > 264.67 ą117% -29.5% 186.67 ą152% numa-meminfo.node0.Active(file)
> > > > 135041 ą 20% -22.4% 104774 ą 42% numa-meminfo.node0.AnonHugePages
> > > > 197759 ą 18% -20.4% 157470 ą 35% numa-meminfo.node0.AnonPages
> > > > 235746 ą 19% -11.8% 207988 ą 29% numa-meminfo.node0.AnonPages.max
> > > > 2.00 ą223% +0.0% 2.00 ą223% numa-meminfo.node0.Dirty
> > > > 1386137 ą123% +89.5% 2626100 ą 67% numa-meminfo.node0.FilePages
> > > > 202317 ą 19% -21.0% 159846 ą 36% numa-meminfo.node0.Inactive
> > > > 200223 ą 19% -20.7% 158765 ą 35% numa-meminfo.node0.Inactive(anon)
> > > > 2093 ą129% -48.4% 1080 ą200% numa-meminfo.node0.Inactive(file)
> > > > 46369 ą 57% +43.5% 66525 ą 41% numa-meminfo.node0.KReclaimable
> > > > 9395 ą 4% +4.6% 9822 ą 5% numa-meminfo.node0.KernelStack
> > > > 14343 ą101% +65.1% 23681 ą 58% numa-meminfo.node0.Mapped
> > > > 95532160 -1.3% 94306066 numa-meminfo.node0.MemFree
> > > > 97681544 +0.0% 97681544 numa-meminfo.node0.MemTotal
> > > > 2149382 ą 82% +57.0% 3375476 ą 53% numa-meminfo.node0.MemUsed
> > > > 2356 ą 21% -9.9% 2122 ą 9% numa-meminfo.node0.PageTables
> > > > 46369 ą 57% +43.5% 66525 ą 41% numa-meminfo.node0.SReclaimable
> > > > 109141 ą 6% +1.5% 110817 ą 7% numa-meminfo.node0.SUnreclaim
> > > > 4514 ą 34% -22.4% 3505 ą 30% numa-meminfo.node0.Shmem
> > > > 155511 ą 18% +14.0% 177344 ą 14% numa-meminfo.node0.Slab
> > > > 1379264 ą124% +90.1% 2621327 ą 67% numa-meminfo.node0.Unevictable
> > > > 46974 ą 8% +1.5% 47665 ą 7% numa-meminfo.node1.Active
> > > > 46837 ą 8% +1.6% 47601 ą 7% numa-meminfo.node1.Active(anon)
> > > > 137.33 ą219% -54.0% 63.17 ą 85% numa-meminfo.node1.Active(file)
> > > > 155559 ą 18% -43.5% 87865 ą 52% numa-meminfo.node1.AnonHugePages
> > > > 222077 ą 16% -30.8% 153725 ą 36% numa-meminfo.node1.AnonPages
> > > > 304080 ą 17% -27.5% 220544 ą 28% numa-meminfo.node1.AnonPages.max
> > > > 2.00 ą223% -100.0% 0.00 numa-meminfo.node1.Dirty
> > > > 2639873 ą 65% -47.0% 1397913 ą126% numa-meminfo.node1.FilePages
> > > > 232481 ą 17% -30.8% 160887 ą 34% numa-meminfo.node1.Inactive
> > > > 231228 ą 16% -30.5% 160796 ą 34% numa-meminfo.node1.Inactive(anon)
> > > > 1252 ą213% -92.8% 90.33 ą 96% numa-meminfo.node1.Inactive(file)
> > > > 78155 ą 34% -25.9% 57927 ą 47% numa-meminfo.node1.KReclaimable
> > > > 9041 ą 4% -3.3% 8740 ą 5% numa-meminfo.node1.KernelStack
> > > > 25795 ą 55% -37.5% 16118 ą 85% numa-meminfo.node1.Mapped
> > > > 95619356 +1.4% 96947357 numa-meminfo.node1.MemFree
> > > > 99038776 +0.0% 99038776 numa-meminfo.node1.MemTotal
> > > > 3419418 ą 52% -38.8% 2091417 ą 85% numa-meminfo.node1.MemUsed
> > > > 2405 ą 21% -1.5% 2369 ą 7% numa-meminfo.node1.PageTables
> > > > 78155 ą 34% -25.9% 57927 ą 47% numa-meminfo.node1.SReclaimable
> > > > 87984 ą 7% -1.7% 86475 ą 9% numa-meminfo.node1.SUnreclaim
> > > > 52978 ą 9% +2.9% 54500 ą 8% numa-meminfo.node1.Shmem
> > > > 166140 ą 16% -13.1% 144403 ą 17% numa-meminfo.node1.Slab
> > > > 2585504 ą 66% -48.0% 1343258 ą131% numa-meminfo.node1.Unevictable
> > > > 486.17 ą 9% +6.8% 519.17 ą 7% proc-vmstat.direct_map_level2_splits
> > > > 8.00 ą 22% +2.1% 8.17 ą 8% proc-vmstat.direct_map_level3_splits
> > > > 12303 ą 7% +1.3% 12461 ą 7% proc-vmstat.nr_active_anon
> > > > 100.50 ą 99% -37.8% 62.50 ą123% proc-vmstat.nr_active_file
> > > > 104906 -25.9% 77785 proc-vmstat.nr_anon_pages
> > > > 141.00 -33.6% 93.67 proc-vmstat.nr_anon_transparent_hugepages
> > > > 264.00 ą141% -54.3% 120.67 ą223% proc-vmstat.nr_dirtied
> > > > 1.00 ą141% -50.0% 0.50 ą223% proc-vmstat.nr_dirty
> > > > 4750146 +0.1% 4752612 proc-vmstat.nr_dirty_background_threshold
> > > > 9511907 +0.1% 9516846 proc-vmstat.nr_dirty_threshold
> > > > 1006517 -0.1% 1005995 proc-vmstat.nr_file_pages
> > > > 47787985 +0.1% 47813269 proc-vmstat.nr_free_pages
> > > > 107821 -25.9% 79869 proc-vmstat.nr_inactive_anon
> > > > 836.17 ą 95% -65.1% 292.17 ą186% proc-vmstat.nr_inactive_file
> > > > 18434 +0.7% 18563 proc-vmstat.nr_kernel_stack
> > > > 10033 ą 2% -1.1% 9924 proc-vmstat.nr_mapped
> > > > 1190 -5.7% 1122 proc-vmstat.nr_page_table_pages
> > > > 14387 ą 7% +0.7% 14493 ą 6% proc-vmstat.nr_shmem
> > > > 31131 -0.1% 31114 proc-vmstat.nr_slab_reclaimable
> > > > 49281 +0.1% 49323 proc-vmstat.nr_slab_unreclaimable
> > > > 991192 -0.0% 991146 proc-vmstat.nr_unevictable
> > > > 264.00 ą141% -54.3% 120.67 ą223% proc-vmstat.nr_written
> > > > 12303 ą 7% +1.3% 12461 ą 7% proc-vmstat.nr_zone_active_anon
> > > > 100.50 ą 99% -37.8% 62.50 ą123% proc-vmstat.nr_zone_active_file
> > > > 107821 -25.9% 79869 proc-vmstat.nr_zone_inactive_anon
> > > > 836.17 ą 95% -65.1% 292.17 ą186% proc-vmstat.nr_zone_inactive_file
> > > > 991192 -0.0% 991146 proc-vmstat.nr_zone_unevictable
> > > > 1.00 ą141% -50.0% 0.50 ą223% proc-vmstat.nr_zone_write_pending
> > > > 17990 ą 21% -17.6% 14820 ą 46% proc-vmstat.numa_hint_faults
> > > > 7847 ą 37% -41.5% 4588 ą 26% proc-vmstat.numa_hint_faults_local
> > > > 806662 +0.3% 809070 proc-vmstat.numa_hit
> > > > 488.50 ą 13% -73.4% 130.17 ą 22% proc-vmstat.numa_huge_pte_updates
> > > > 0.00 -100.0% 0.00 proc-vmstat.numa_interleave
> > > > 712588 -0.2% 711419 proc-vmstat.numa_local
> > > > 94077 +0.0% 94084 proc-vmstat.numa_other
> > > > 18894 ą 67% -3.1% 18303 ą 41% proc-vmstat.numa_pages_migrated
> > > > 337482 ą 10% -59.0% 138314 ą 10% proc-vmstat.numa_pte_updates
> > > > 61815 -1.6% 60823 proc-vmstat.pgactivate
> > > > 0.00 -100.0% 0.00 proc-vmstat.pgalloc_dma32
> > > > 933601 -3.8% 898485 proc-vmstat.pgalloc_normal
> > > > 899579 -0.5% 895253 proc-vmstat.pgfault
> > > > 896972 -3.9% 861819 proc-vmstat.pgfree
> > > > 18894 ą 67% -3.1% 18303 ą 41% proc-vmstat.pgmigrate_success
> > > > 3845 ą100% -66.8% 1277 ą223% proc-vmstat.pgpgin
> > > > 1064 ą141% -54.3% 486.67 ą223% proc-vmstat.pgpgout
> > > > 40396 -0.6% 40172 proc-vmstat.pgreuse
> > > > 105.50 -9.2% 95.83 ą 5% proc-vmstat.thp_collapse_alloc
> > > > 57.00 -87.4% 7.17 ą 5% proc-vmstat.thp_deferred_split_page
> > > > 74.83 -72.4% 20.67 ą 4% proc-vmstat.thp_fault_alloc
> > > > 19.50 ą105% -15.4% 16.50 ą 71% proc-vmstat.thp_migration_success
> > > > 57.00 -87.4% 7.17 ą 5% proc-vmstat.thp_split_pmd
> > > > 0.00 -100.0% 0.00 proc-vmstat.thp_zero_page_alloc
> > > > 17.00 +0.0% 17.00 proc-vmstat.unevictable_pgs_culled
> > > > 589.83 ą 21% -5.2% 559.00 ą 10% numa-vmstat.node0.nr_active_anon
> > > > 66.00 ą117% -29.3% 46.67 ą152% numa-vmstat.node0.nr_active_file
> > > > 49406 ą 18% -20.3% 39355 ą 35% numa-vmstat.node0.nr_anon_pages
> > > > 65.17 ą 21% -22.0% 50.83 ą 42% numa-vmstat.node0.nr_anon_transparent_hugepages
> > > > 132.00 ą223% -8.6% 120.67 ą223% numa-vmstat.node0.nr_dirtied
> > > > 0.50 ą223% +0.0% 0.50 ą223% numa-vmstat.node0.nr_dirty
> > > > 346534 ą123% +89.5% 656525 ą 67% numa-vmstat.node0.nr_file_pages
> > > > 23883055 -1.3% 23576561 numa-vmstat.node0.nr_free_pages
> > > > 50051 ą 19% -20.7% 39679 ą 35% numa-vmstat.node0.nr_inactive_anon
> > > > 522.67 ą129% -48.4% 269.67 ą200% numa-vmstat.node0.nr_inactive_file
> > > > 0.00 -100.0% 0.00 numa-vmstat.node0.nr_isolated_anon
> > > > 9392 ą 4% +4.6% 9823 ą 5% numa-vmstat.node0.nr_kernel_stack
> > > > 3594 ą101% +64.8% 5922 ą 58% numa-vmstat.node0.nr_mapped
> > > > 587.83 ą 21% -9.8% 530.00 ą 9% numa-vmstat.node0.nr_page_table_pages
> > > > 1129 ą 34% -22.4% 876.67 ą 30% numa-vmstat.node0.nr_shmem
> > > > 11591 ą 57% +43.5% 16631 ą 41% numa-vmstat.node0.nr_slab_reclaimable
> > > > 27285 ą 6% +1.5% 27704 ą 7% numa-vmstat.node0.nr_slab_unreclaimable
> > > > 344815 ą124% +90.1% 655331 ą 67% numa-vmstat.node0.nr_unevictable
> > > > 132.00 ą223% -8.6% 120.67 ą223% numa-vmstat.node0.nr_written
> > > > 589.83 ą 21% -5.2% 559.00 ą 10% numa-vmstat.node0.nr_zone_active_anon
> > > > 66.00 ą117% -29.3% 46.67 ą152% numa-vmstat.node0.nr_zone_active_file
> > > > 50051 ą 19% -20.7% 39679 ą 35% numa-vmstat.node0.nr_zone_inactive_anon
> > > > 522.67 ą129% -48.4% 269.67 ą200% numa-vmstat.node0.nr_zone_inactive_file
> > > > 344815 ą124% +90.1% 655331 ą 67% numa-vmstat.node0.nr_zone_unevictable
> > > > 0.50 ą223% +0.0% 0.50 ą223% numa-vmstat.node0.nr_zone_write_pending
> > > > 374134 ą 6% -4.1% 358690 ą 7% numa-vmstat.node0.numa_hit
> > > > 0.00 -100.0% 0.00 numa-vmstat.node0.numa_interleave
> > > > 328256 ą 15% -7.1% 304955 ą 20% numa-vmstat.node0.numa_local
> > > > 45881 ą 75% +17.1% 53735 ą 69% numa-vmstat.node0.numa_other
> > > > 11706 ą 8% +1.7% 11901 ą 7% numa-vmstat.node1.nr_active_anon
> > > > 34.17 ą219% -54.1% 15.67 ą 84% numa-vmstat.node1.nr_active_file
> > > > 55500 ą 16% -30.8% 38424 ą 36% numa-vmstat.node1.nr_anon_pages
> > > > 75.50 ą 18% -43.7% 42.50 ą 53% numa-vmstat.node1.nr_anon_transparent_hugepages
> > > > 132.00 ą223% -100.0% 0.00 numa-vmstat.node1.nr_dirtied
> > > > 0.50 ą223% -100.0% 0.00 numa-vmstat.node1.nr_dirty
> > > > 659985 ą 65% -47.0% 349484 ą126% numa-vmstat.node1.nr_file_pages
> > > > 23904828 +1.4% 24236871 numa-vmstat.node1.nr_free_pages
> > > > 57826 ą 16% -30.5% 40197 ą 34% numa-vmstat.node1.nr_inactive_anon
> > > > 313.00 ą213% -92.9% 22.33 ą 96% numa-vmstat.node1.nr_inactive_file
> > > > 9043 ą 4% -3.3% 8740 ą 5% numa-vmstat.node1.nr_kernel_stack
> > > > 6467 ą 55% -37.6% 4038 ą 85% numa-vmstat.node1.nr_mapped
> > > > 601.50 ą 21% -1.6% 591.83 ą 7% numa-vmstat.node1.nr_page_table_pages
> > > > 13261 ą 9% +2.8% 13630 ą 8% numa-vmstat.node1.nr_shmem
> > > > 19538 ą 34% -25.9% 14481 ą 47% numa-vmstat.node1.nr_slab_reclaimable
> > > > 21995 ą 7% -1.7% 21618 ą 9% numa-vmstat.node1.nr_slab_unreclaimable
> > > > 646375 ą 66% -48.0% 335813 ą131% numa-vmstat.node1.nr_unevictable
> > > > 132.00 ą223% -100.0% 0.00 numa-vmstat.node1.nr_written
> > > > 11706 ą 8% +1.7% 11901 ą 7% numa-vmstat.node1.nr_zone_active_anon
> > > > 34.17 ą219% -54.1% 15.67 ą 84% numa-vmstat.node1.nr_zone_active_file
> > > > 57826 ą 16% -30.5% 40197 ą 34% numa-vmstat.node1.nr_zone_inactive_anon
> > > > 313.00 ą213% -92.9% 22.33 ą 96% numa-vmstat.node1.nr_zone_inactive_file
> > > > 646375 ą 66% -48.0% 335813 ą131% numa-vmstat.node1.nr_zone_unevictable
> > > > 0.50 ą223% -100.0% 0.00 numa-vmstat.node1.nr_zone_write_pending
> > > > 429997 ą 5% +3.5% 444962 ą 5% numa-vmstat.node1.numa_hit
> > > > 0.00 -100.0% 0.00 numa-vmstat.node1.numa_interleave
> > > > 381801 ą 13% +6.0% 404613 ą 14% numa-vmstat.node1.numa_local
> > > > 48195 ą 71% -16.3% 40348 ą 92% numa-vmstat.node1.numa_other
> > > > 2.47 ą 2% -2.0% 2.42 ą 5% perf-stat.i.MPKI
> > > > 3.282e+09 +0.7% 3.305e+09 perf-stat.i.branch-instructions
> > > > 0.41 -0.1 0.33 perf-stat.i.branch-miss-rate%
> > > > 13547319 -16.6% 11300609 perf-stat.i.branch-misses
> > > > 42.88 +0.7 43.53 perf-stat.i.cache-miss-rate%
> > > > 17114713 ą 3% +1.4% 17346470 ą 5% perf-stat.i.cache-misses
> > > > 40081707 ą 2% -0.0% 40073189 ą 5% perf-stat.i.cache-references
> > > > 8192 ą 2% +1.4% 8311 ą 4% perf-stat.i.context-switches
> > > > 8.84 -0.8% 8.77 perf-stat.i.cpi
> > > > 104007 +0.0% 104008 perf-stat.i.cpu-clock
> > > > 1.446e+11 +0.1% 1.447e+11 perf-stat.i.cpu-cycles
> > > > 140.10 -1.0% 138.76 perf-stat.i.cpu-migrations
> > > > 8487 ą 3% -0.9% 8412 ą 6% perf-stat.i.cycles-between-cache-misses
> > > > 0.01 ą 6% -0.0 0.01 perf-stat.i.dTLB-load-miss-rate%
> > > > 434358 ą 3% -16.9% 360889 perf-stat.i.dTLB-load-misses
> > > > 4.316e+09 +1.3% 4.373e+09 perf-stat.i.dTLB-loads
> > > > 0.00 ą 15% -0.0 0.00 ą 9% perf-stat.i.dTLB-store-miss-rate%
> > > > 10408 ą 11% -2.6% 10135 ą 8% perf-stat.i.dTLB-store-misses
> > > > 4.302e+08 +5.5% 4.539e+08 perf-stat.i.dTLB-stores
> > > > 16.21 ą 2% -2.5 13.73 ą 18% perf-stat.i.iTLB-load-miss-rate%
> > > > 394805 ą 5% -26.0% 292089 ą 8% perf-stat.i.iTLB-load-misses
> > > > 2041963 ą 3% -8.3% 1872405 ą 12% perf-stat.i.iTLB-loads
> > > > 1.638e+10 +1.0% 1.654e+10 perf-stat.i.instructions
> > > > 41729 ą 6% +37.4% 57323 ą 8% perf-stat.i.instructions-per-iTLB-miss
> > > > 0.11 +0.8% 0.11 perf-stat.i.ipc
> > > > 0.01 ą 55% -1.5% 0.01 ą 85% perf-stat.i.major-faults
> > > > 1.39 +0.1% 1.39 perf-stat.i.metric.GHz
> > > > 468.46 ą 2% -1.5% 461.59 ą 4% perf-stat.i.metric.K/sec
> > > > 77.18 +1.3% 78.18 perf-stat.i.metric.M/sec
> > > > 2473 -0.0% 2472 perf-stat.i.minor-faults
> > > > 89.67 -0.5 89.18 perf-stat.i.node-load-miss-rate%
> > > > 5070484 -10.3% 4547670 perf-stat.i.node-load-misses
> > > > 585336 ą 2% -5.5% 553260 ą 8% perf-stat.i.node-loads
> > > > 98.73 +0.2 98.91 perf-stat.i.node-store-miss-rate%
> > > > 935187 +2.2% 955923 ą 3% perf-stat.i.node-store-misses
> > > > 13301 ą 8% -12.6% 11631 ą 5% perf-stat.i.node-stores
> > > > 2473 -0.0% 2472 perf-stat.i.page-faults
> > > > 104007 +0.0% 104008 perf-stat.i.task-clock
> > > > 2.45 ą 2% -1.0% 2.42 ą 5% perf-stat.overall.MPKI
> > > > 0.41 -0.1 0.34 perf-stat.overall.branch-miss-rate%
> > > > 42.68 +0.6 43.26 perf-stat.overall.cache-miss-rate%
> > > > 8.83 -0.9% 8.75 perf-stat.overall.cpi
> > > > 8459 ą 3% -1.0% 8372 ą 6% perf-stat.overall.cycles-between-cache-misses
> > > > 0.01 ą 3% -0.0 0.01 perf-stat.overall.dTLB-load-miss-rate%
> > > > 0.00 ą 11% -0.0 0.00 ą 8% perf-stat.overall.dTLB-store-miss-rate%
> > > > 16.19 ą 2% -2.5 13.73 ą 18% perf-stat.overall.iTLB-load-miss-rate%
> > > > 41644 ą 6% +37.0% 57047 ą 8% perf-stat.overall.instructions-per-iTLB-miss
> > > > 0.11 +0.9% 0.11 perf-stat.overall.ipc
> > > > 89.65 -0.5 89.15 perf-stat.overall.node-load-miss-rate%
> > > > 98.59 +0.2 98.78 perf-stat.overall.node-store-miss-rate%
> > > > 35314961 +28.0% 45213422 ą 3% perf-stat.overall.path-length
> > > > 3.272e+09 +0.7% 3.295e+09 perf-stat.ps.branch-instructions
> > > > 13563215 -16.5% 11329031 perf-stat.ps.branch-misses
> > > > 17059170 ą 3% +1.3% 17288798 ą 5% perf-stat.ps.cache-misses
> > > > 39960738 ą 2% -0.0% 39951411 ą 5% perf-stat.ps.cache-references
> > > > 8205 ą 2% +1.4% 8320 ą 4% perf-stat.ps.context-switches
> > > > 103658 -0.0% 103657 perf-stat.ps.cpu-clock
> > > > 1.441e+11 +0.1% 1.442e+11 perf-stat.ps.cpu-cycles
> > > > 140.16 -1.0% 138.77 perf-stat.ps.cpu-migrations
> > > > 433133 ą 3% -16.9% 359910 perf-stat.ps.dTLB-load-misses
> > > > 4.302e+09 +1.3% 4.359e+09 perf-stat.ps.dTLB-loads
> > > > 10392 ą 11% -2.6% 10120 ą 8% perf-stat.ps.dTLB-store-misses
> > > > 4.29e+08 +5.5% 4.527e+08 perf-stat.ps.dTLB-stores
> > > > 393499 ą 5% -26.0% 291118 ą 8% perf-stat.ps.iTLB-load-misses
> > > > 2035052 ą 3% -8.3% 1866106 ą 12% perf-stat.ps.iTLB-loads
> > > > 1.633e+10 +1.0% 1.649e+10 perf-stat.ps.instructions
> > > > 0.01 ą 55% +0.1% 0.01 ą 85% perf-stat.ps.major-faults
> > > > 2466 +0.0% 2466 perf-stat.ps.minor-faults
> > > > 5053378 -10.3% 4532205 perf-stat.ps.node-load-misses
> > > > 583428 ą 2% -5.5% 551516 ą 8% perf-stat.ps.node-loads
> > > > 932227 +2.2% 952780 ą 3% perf-stat.ps.node-store-misses
> > > > 13342 ą 8% -12.1% 11729 ą 6% perf-stat.ps.node-stores
> > > > 2466 +0.0% 2466 perf-stat.ps.page-faults
> > > > 103658 -0.0% 103657 perf-stat.ps.task-clock
> > > > 4.952e+12 +0.9% 4.994e+12 perf-stat.total.instructions
> > > > 10.88 ą223% -100.0% 0.00 sched_debug.cfs_rq:/.MIN_vruntime.avg
> > > > 1132 ą223% -100.0% 0.00 sched_debug.cfs_rq:/.MIN_vruntime.max
> > > > 0.00 +0.0% 0.00 sched_debug.cfs_rq:/.MIN_vruntime.min
> > > > 110.47 ą223% -100.0% 0.00 sched_debug.cfs_rq:/.MIN_vruntime.stddev
> > > > 0.53 ą 4% +7.4% 0.57 ą 4% sched_debug.cfs_rq:/.h_nr_running.avg
> > > > 1.03 ą 7% -3.2% 1.00 sched_debug.cfs_rq:/.h_nr_running.max
> > > > 0.45 ą 2% -1.9% 0.44 ą 3% sched_debug.cfs_rq:/.h_nr_running.stddev
> > > > 11896 ą 12% -0.1% 11883 ą 13% sched_debug.cfs_rq:/.load.avg
> > > > 123097 ą123% -80.1% 24487 ą 18% sched_debug.cfs_rq:/.load.max
> > > > 19029 ą 74% -49.9% 9525 ą 13% sched_debug.cfs_rq:/.load.stddev
> > > > 22.63 ą 23% +1.4% 22.93 ą 16% sched_debug.cfs_rq:/.load_avg.avg
> > > > 530.85 ą 73% -13.1% 461.19 ą 43% sched_debug.cfs_rq:/.load_avg.max
> > > > 73.53 ą 46% -7.1% 68.30 ą 33% sched_debug.cfs_rq:/.load_avg.stddev
> > > > 10.88 ą223% -100.0% 0.00 sched_debug.cfs_rq:/.max_vruntime.avg
> > > > 1132 ą223% -100.0% 0.00 sched_debug.cfs_rq:/.max_vruntime.max
> > > > 0.00 +0.0% 0.00 sched_debug.cfs_rq:/.max_vruntime.min
> > > > 110.47 ą223% -100.0% 0.00 sched_debug.cfs_rq:/.max_vruntime.stddev
> > > > 3883756 ą 13% +12.7% 4377466 ą 4% sched_debug.cfs_rq:/.min_vruntime.avg
> > > > 6993455 ą 10% +6.5% 7445221 ą 2% sched_debug.cfs_rq:/.min_vruntime.max
> > > > 219925 ą 60% +43.7% 315970 ą 71% sched_debug.cfs_rq:/.min_vruntime.min
> > > > 2240239 ą 11% +14.0% 2554847 ą 14% sched_debug.cfs_rq:/.min_vruntime.stddev
> > > > 0.53 ą 5% +7.5% 0.57 ą 4% sched_debug.cfs_rq:/.nr_running.avg
> > > > 1.03 ą 7% -3.2% 1.00 sched_debug.cfs_rq:/.nr_running.max
> > > > 0.45 ą 2% -1.9% 0.44 ą 3% sched_debug.cfs_rq:/.nr_running.stddev
> > > > 6.96 ą 55% +26.9% 8.83 ą 45% sched_debug.cfs_rq:/.removed.load_avg.avg
> > > > 305.28 ą 32% +39.3% 425.39 ą 44% sched_debug.cfs_rq:/.removed.load_avg.max
> > > > 42.94 ą 36% +34.4% 57.70 ą 42% sched_debug.cfs_rq:/.removed.load_avg.stddev
> > > > 2.96 ą 58% +39.1% 4.12 ą 48% sched_debug.cfs_rq:/.removed.runnable_avg.avg
> > > > 150.06 ą 34% +44.0% 216.03 ą 45% sched_debug.cfs_rq:/.removed.runnable_avg.max
> > > > 19.33 ą 42% +42.6% 27.56 ą 45% sched_debug.cfs_rq:/.removed.runnable_avg.stddev
> > > > 2.96 ą 58% +39.1% 4.12 ą 48% sched_debug.cfs_rq:/.removed.util_avg.avg
> > > > 150.06 ą 34% +44.0% 216.03 ą 45% sched_debug.cfs_rq:/.removed.util_avg.max
> > > > 19.33 ą 42% +42.6% 27.56 ą 45% sched_debug.cfs_rq:/.removed.util_avg.stddev
> > > > 540.76 ą 6% +7.5% 581.25 ą 5% sched_debug.cfs_rq:/.runnable_avg.avg
> > > > 1060 ą 2% +2.5% 1087 ą 3% sched_debug.cfs_rq:/.runnable_avg.max
> > > > 442.07 ą 4% -0.1% 441.69 ą 5% sched_debug.cfs_rq:/.runnable_avg.stddev
> > > > 3123464 ą 14% +10.0% 3436745 ą 3% sched_debug.cfs_rq:/.spread0.avg
> > > > 6233151 ą 10% +4.4% 6504505 ą 3% sched_debug.cfs_rq:/.spread0.max
> > > > -540338 +15.6% -624739 sched_debug.cfs_rq:/.spread0.min
> > > > 2240217 ą 11% +14.0% 2554844 ą 14% sched_debug.cfs_rq:/.spread0.stddev
> > > > 540.71 ą 6% +7.5% 581.22 ą 5% sched_debug.cfs_rq:/.util_avg.avg
> > > > 1060 ą 2% +2.5% 1086 ą 3% sched_debug.cfs_rq:/.util_avg.max
> > > > 442.07 ą 4% -0.1% 441.67 ą 5% sched_debug.cfs_rq:/.util_avg.stddev
> > > > 454.69 ą 6% +7.0% 486.47 ą 8% sched_debug.cfs_rq:/.util_est_enqueued.avg
> > > > 1024 -0.0% 1023 sched_debug.cfs_rq:/.util_est_enqueued.max
> > > > 396.02 ą 2% -0.1% 395.79 sched_debug.cfs_rq:/.util_est_enqueued.stddev
> > > > 642171 ą 4% +16.6% 748912 ą 2% sched_debug.cpu.avg_idle.avg
> > > > 1051166 -1.2% 1038098 sched_debug.cpu.avg_idle.max
> > > > 2402 ą 5% +28.5% 3088 ą 9% sched_debug.cpu.avg_idle.min
> > > > 384501 ą 3% -12.3% 337306 ą 5% sched_debug.cpu.avg_idle.stddev
> > > > 198632 ą 7% +5.1% 208788 sched_debug.cpu.clock.avg
> > > > 198638 ą 7% +5.1% 208794 sched_debug.cpu.clock.max
> > > > 198626 ą 7% +5.1% 208783 sched_debug.cpu.clock.min
> > > > 3.25 +2.3% 3.32 ą 5% sched_debug.cpu.clock.stddev
> > > > 196832 ą 7% +5.1% 206882 sched_debug.cpu.clock_task.avg
> > > > 197235 ą 7% +5.1% 207282 sched_debug.cpu.clock_task.max
> > > > 181004 ą 7% +5.7% 191329 sched_debug.cpu.clock_task.min
> > > > 1575 ą 3% -1.8% 1546 sched_debug.cpu.clock_task.stddev
> > > > 2411 ą 4% +2.8% 2478 sched_debug.cpu.curr->pid.avg
> > > > 8665 ą 4% +3.1% 8935 sched_debug.cpu.curr->pid.max
> > > > 2522 ą 2% +1.0% 2548 sched_debug.cpu.curr->pid.stddev
> > > > 501318 -0.0% 501249 sched_debug.cpu.max_idle_balance_cost.avg
> > > > 528365 +0.5% 531236 ą 2% sched_debug.cpu.max_idle_balance_cost.max
> > > > 500000 +0.0% 500000 sched_debug.cpu.max_idle_balance_cost.min
> > > > 5157 ą 19% -4.2% 4941 ą 23% sched_debug.cpu.max_idle_balance_cost.stddev
> > > > 4294 +0.0% 4294 sched_debug.cpu.next_balance.avg
> > > > 4294 +0.0% 4294 sched_debug.cpu.next_balance.max
> > > > 4294 +0.0% 4294 sched_debug.cpu.next_balance.min
> > > > 0.00 ą 41% -40.0% 0.00 ą 13% sched_debug.cpu.next_balance.stddev
> > > > 0.44 ą 4% +2.4% 0.45 sched_debug.cpu.nr_running.avg
> > > > 1.00 +0.0% 1.00 sched_debug.cpu.nr_running.max
> > > > 0.47 +0.5% 0.47 sched_debug.cpu.nr_running.stddev
> > > > 14345 ą 8% +6.7% 15305 ą 4% sched_debug.cpu.nr_switches.avg
> > > > 30800 ą 8% +34.5% 41437 ą 10% sched_debug.cpu.nr_switches.max
> > > > 4563 ą 28% +5.7% 4822 ą 25% sched_debug.cpu.nr_switches.min
> > > > 5491 ą 8% +26.4% 6941 ą 10% sched_debug.cpu.nr_switches.stddev
> > > > 2.111e+09 ą 7% +1.5% 2.142e+09 ą 6% sched_debug.cpu.nr_uninterruptible.avg
> > > > 4.295e+09 +0.0% 4.295e+09 sched_debug.cpu.nr_uninterruptible.max
> > > > 2.14e+09 +0.1% 2.143e+09 sched_debug.cpu.nr_uninterruptible.stddev
> > > > 198627 ą 7% +5.1% 208783 sched_debug.cpu_clk
> > > > 996147 +0.0% 996147 sched_debug.dl_rq:.dl_bw->bw.avg
> > > > 996147 +0.0% 996147 sched_debug.dl_rq:.dl_bw->bw.max
> > > > 996147 +0.0% 996147 sched_debug.dl_rq:.dl_bw->bw.min
> > > > 4.295e+09 +0.0% 4.295e+09 sched_debug.jiffies
> > > > 198022 ą 7% +5.1% 208178 sched_debug.ktime
> > > > 950.00 +0.0% 950.00 sched_debug.rt_rq:.rt_runtime.avg
> > > > 950.00 +0.0% 950.00 sched_debug.rt_rq:.rt_runtime.max
> > > > 950.00 +0.0% 950.00 sched_debug.rt_rq:.rt_runtime.min
> > > > 199377 ą 7% +5.1% 209531 sched_debug.sched_clk
> > > > 1.00 +0.0% 1.00 sched_debug.sched_clock_stable()
> > > > 58611259 +0.0% 58611259 sched_debug.sysctl_sched.sysctl_sched_features
> > > > 0.75 +0.0% 0.75 sched_debug.sysctl_sched.sysctl_sched_idle_min_granularity
> > > > 24.00 +0.0% 24.00 sched_debug.sysctl_sched.sysctl_sched_latency
> > > > 3.00 +0.0% 3.00 sched_debug.sysctl_sched.sysctl_sched_min_granularity
> > > > 1.00 +0.0% 1.00 sched_debug.sysctl_sched.sysctl_sched_tunable_scaling
> > > > 4.00 +0.0% 4.00 sched_debug.sysctl_sched.sysctl_sched_wakeup_granularity
> > > > 20.90 ą 47% -6.4 14.49 ą100% perf-profile.calltrace.cycles-pp.mwait_idle_with_hints.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
> > > > 20.90 ą 47% -6.4 14.49 ą100% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
> > > > 0.48 ą 44% -0.5 0.00 perf-profile.calltrace.cycles-pp.do_mmap.vm_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe.__mmap
> > > > 29.41 ą 19% -0.2 29.23 ą 18% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
> > > > 35.02 ą 8% -0.2 34.86 ą 7% perf-profile.calltrace.cycles-pp.__mmap
> > > > 34.95 ą 8% -0.1 34.81 ą 7% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__mmap
> > > > 34.92 ą 8% -0.1 34.79 ą 7% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__mmap
> > > > 34.87 ą 8% -0.1 34.74 ą 7% perf-profile.calltrace.cycles-pp.vm_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe.__mmap
> > > > 0.41 ą 74% -0.1 0.30 ą156% perf-profile.calltrace.cycles-pp.cpu_startup_entry.rest_init.arch_call_rest_init.start_kernel.secondary_startup_64_no_verify
> > > > 0.41 ą 74% -0.1 0.30 ą156% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.rest_init.arch_call_rest_init.start_kernel
> > > > 0.41 ą 74% -0.1 0.30 ą156% perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.rest_init.arch_call_rest_init
> > > > 0.41 ą 74% -0.1 0.30 ą156% perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.rest_init
> > > > 0.41 ą 74% -0.1 0.30 ą156% perf-profile.calltrace.cycles-pp.start_kernel.secondary_startup_64_no_verify
> > > > 0.41 ą 74% -0.1 0.30 ą156% perf-profile.calltrace.cycles-pp.arch_call_rest_init.start_kernel.secondary_startup_64_no_verify
> > > > 0.41 ą 74% -0.1 0.30 ą156% perf-profile.calltrace.cycles-pp.rest_init.arch_call_rest_init.start_kernel.secondary_startup_64_no_verify
> > > > 29.59 ą 19% -0.1 29.50 ą 17% perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
> > > > 29.03 ą 19% -0.1 28.95 ą 17% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
> > > > 29.03 ą 19% -0.1 28.95 ą 17% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
> > > > 29.03 ą 19% -0.1 28.95 ą 17% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
> > > > 29.00 ą 19% -0.1 28.93 ą 17% perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
> > > > 29.00 ą 19% -0.1 28.93 ą 17% perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
> > > > 33.56 ą 8% -0.0 33.53 ą 7% perf-profile.calltrace.cycles-pp.osq_lock.rwsem_optimistic_spin.rwsem_down_write_slowpath.down_write_killable.vm_mmap_pgoff
> > > > 34.26 ą 8% -0.0 34.24 ą 7% perf-profile.calltrace.cycles-pp.down_write_killable.vm_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe.__mmap
> > > > 34.23 ą 8% -0.0 34.21 ą 7% perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write_killable.vm_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
> > > > 34.19 ą 8% -0.0 34.18 ą 7% perf-profile.calltrace.cycles-pp.rwsem_optimistic_spin.rwsem_down_write_slowpath.down_write_killable.vm_mmap_pgoff.do_syscall_64
> > > > 0.44 ą 44% +0.0 0.48 ą 44% perf-profile.calltrace.cycles-pp.rwsem_spin_on_owner.rwsem_optimistic_spin.rwsem_down_write_slowpath.down_write_killable.__vm_munmap
> > > > 0.45 ą 44% +0.0 0.48 ą 44% perf-profile.calltrace.cycles-pp.rwsem_spin_on_owner.rwsem_optimistic_spin.rwsem_down_write_slowpath.down_write_killable.vm_mmap_pgoff
> > > > 33.62 ą 8% +0.1 33.71 ą 7% perf-profile.calltrace.cycles-pp.osq_lock.rwsem_optimistic_spin.rwsem_down_write_slowpath.down_write_killable.__vm_munmap
> > > > 34.32 ą 8% +0.1 34.42 ą 7% perf-profile.calltrace.cycles-pp.down_write_killable.__vm_munmap.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe
> > > > 34.29 ą 8% +0.1 34.39 ą 7% perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write_killable.__vm_munmap.__x64_sys_munmap.do_syscall_64
> > > > 34.25 ą 8% +0.1 34.36 ą 7% perf-profile.calltrace.cycles-pp.rwsem_optimistic_spin.rwsem_down_write_slowpath.down_write_killable.__vm_munmap.__x64_sys_munmap
> > > > 35.11 ą 8% +0.2 35.31 ą 7% perf-profile.calltrace.cycles-pp.__munmap
> > > > 35.04 ą 8% +0.2 35.25 ą 7% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__munmap
> > > > 35.02 ą 8% +0.2 35.24 ą 7% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__munmap
> > > > 0.00 +0.2 0.22 ą223% perf-profile.calltrace.cycles-pp.intel_idle_irq.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
> > > > 34.97 ą 8% +0.2 35.20 ą 7% perf-profile.calltrace.cycles-pp.__vm_munmap.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe.__munmap
> > > > 34.97 ą 8% +0.2 35.20 ą 7% perf-profile.calltrace.cycles-pp.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe.__munmap
> > > > 0.47 ą 44% +0.2 0.70 ą 7% perf-profile.calltrace.cycles-pp.do_mas_align_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe
> > > > 0.00 +0.4 0.44 ą223% perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.mwait_idle_with_hints.intel_idle_irq.cpuidle_enter_state.cpuidle_enter
> > > > 8.27 ą 91% +6.2 14.46 ą 77% perf-profile.calltrace.cycles-pp.mwait_idle_with_hints.intel_idle_ibrs.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
> > > > 8.27 ą 91% +6.2 14.46 ą 77% perf-profile.calltrace.cycles-pp.intel_idle_ibrs.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
> > > > 21.09 ą 47% -6.5 14.62 ą 99% perf-profile.children.cycles-pp.intel_idle
> > > > 35.02 ą 8% -0.2 34.86 ą 7% perf-profile.children.cycles-pp.__mmap
> > > > 0.14 ą 9% -0.1 0.00 perf-profile.children.cycles-pp.thp_get_unmapped_area
> > > > 34.87 ą 8% -0.1 34.74 ą 7% perf-profile.children.cycles-pp.vm_mmap_pgoff
> > > > 0.55 ą 9% -0.1 0.46 ą 7% perf-profile.children.cycles-pp.do_mmap
> > > > 29.59 ą 19% -0.1 29.50 ą 17% perf-profile.children.cycles-pp.secondary_startup_64_no_verify
> > > > 29.59 ą 19% -0.1 29.50 ą 17% perf-profile.children.cycles-pp.cpu_startup_entry
> > > > 29.59 ą 19% -0.1 29.50 ą 17% perf-profile.children.cycles-pp.do_idle
> > > > 29.03 ą 19% -0.1 28.95 ą 17% perf-profile.children.cycles-pp.start_secondary
> > > > 29.56 ą 19% -0.1 29.49 ą 17% perf-profile.children.cycles-pp.cpuidle_idle_call
> > > > 29.56 ą 19% -0.1 29.48 ą 17% perf-profile.children.cycles-pp.cpuidle_enter
> > > > 29.56 ą 19% -0.1 29.48 ą 17% perf-profile.children.cycles-pp.cpuidle_enter_state
> > > > 29.52 ą 19% -0.1 29.45 ą 17% perf-profile.children.cycles-pp.mwait_idle_with_hints
> > > > 0.38 ą 9% -0.1 0.32 ą 6% perf-profile.children.cycles-pp.mmap_region
> > > > 0.05 ą 7% -0.1 0.00 perf-profile.children.cycles-pp.unmap_vmas
> > > > 0.11 ą 8% -0.1 0.06 ą 13% perf-profile.children.cycles-pp.unmap_region
> > > > 0.16 ą 10% -0.0 0.13 ą 9% perf-profile.children.cycles-pp.get_unmapped_area
> > > > 0.07 ą 7% -0.0 0.03 ą 70% perf-profile.children.cycles-pp.mas_find
> > > > 0.05 ą 44% -0.0 0.02 ą141% perf-profile.children.cycles-pp.mas_wr_node_store
> > > > 0.10 ą 10% -0.0 0.07 ą 14% perf-profile.children.cycles-pp.mas_spanning_rebalance
> > > > 0.14 ą 9% -0.0 0.11 ą 9% perf-profile.children.cycles-pp.arch_get_unmapped_area_topdown
> > > > 0.06 ą 11% -0.0 0.04 ą 72% perf-profile.children.cycles-pp.__schedule
> > > > 0.14 ą 10% -0.0 0.11 ą 9% perf-profile.children.cycles-pp.vm_unmapped_area
> > > > 0.07 ą 10% -0.0 0.04 ą 45% perf-profile.children.cycles-pp.do_mas_munmap
> > > > 0.02 ą 99% -0.0 0.00 perf-profile.children.cycles-pp.mas_next_entry
> > > > 0.04 ą 44% -0.0 0.02 ą141% perf-profile.children.cycles-pp.schedule
> > > > 0.06 ą 9% -0.0 0.04 ą 71% perf-profile.children.cycles-pp.mas_wr_modify
> > > > 0.10 ą 8% -0.0 0.08 ą 11% perf-profile.children.cycles-pp.mas_rev_awalk
> > > > 0.10 ą 12% -0.0 0.08 ą 16% perf-profile.children.cycles-pp.mas_wr_spanning_store
> > > > 0.06 ą 7% -0.0 0.04 ą 45% perf-profile.children.cycles-pp.mas_walk
> > > > 0.09 ą 11% -0.0 0.08 ą 16% perf-profile.children.cycles-pp.syscall_exit_to_user_mode
> > > > 0.02 ą141% -0.0 0.00 perf-profile.children.cycles-pp.perf_event_mmap
> > > > 0.02 ą141% -0.0 0.00 perf-profile.children.cycles-pp.unmap_page_range
> > > > 0.11 ą 26% -0.0 0.10 ą 10% perf-profile.children.cycles-pp.__get_user_nocheck_8
> > > > 0.35 ą 19% -0.0 0.34 ą 11% perf-profile.children.cycles-pp.perf_tp_event
> > > > 0.11 ą 26% -0.0 0.10 ą 11% perf-profile.children.cycles-pp.perf_callchain_user
> > > > 0.34 ą 19% -0.0 0.33 ą 10% perf-profile.children.cycles-pp.__perf_event_overflow
> > > > 0.34 ą 19% -0.0 0.33 ą 10% perf-profile.children.cycles-pp.perf_event_output_forward
> > > > 0.31 ą 19% -0.0 0.30 ą 12% perf-profile.children.cycles-pp.perf_prepare_sample
> > > > 0.30 ą 19% -0.0 0.29 ą 10% perf-profile.children.cycles-pp.perf_callchain
> > > > 0.30 ą 19% -0.0 0.29 ą 10% perf-profile.children.cycles-pp.get_perf_callchain
> > > > 0.12 ą 9% -0.0 0.11 ą 9% perf-profile.children.cycles-pp.mas_empty_area_rev
> > > > 0.08 ą 7% -0.0 0.07 ą 8% perf-profile.children.cycles-pp.syscall_return_via_sysret
> > > > 0.01 ą223% -0.0 0.00 perf-profile.children.cycles-pp.mas_wr_bnode
> > > > 0.01 ą223% -0.0 0.00 perf-profile.children.cycles-pp.perf_event_mmap_event
> > > > 0.01 ą223% -0.0 0.00 perf-profile.children.cycles-pp.__entry_text_start
> > > > 0.33 ą 10% -0.0 0.32 ą 7% perf-profile.children.cycles-pp.mas_store_prealloc
> > > > 0.32 ą 20% -0.0 0.32 ą 10% perf-profile.children.cycles-pp.update_curr
> > > > 0.32 ą 19% -0.0 0.31 ą 11% perf-profile.children.cycles-pp.perf_trace_sched_stat_runtime
> > > > 0.56 ą 22% -0.0 0.56 ą 58% perf-profile.children.cycles-pp.start_kernel
> > > > 0.56 ą 22% -0.0 0.56 ą 58% perf-profile.children.cycles-pp.arch_call_rest_init
> > > > 0.56 ą 22% -0.0 0.56 ą 58% perf-profile.children.cycles-pp.rest_init
> > > > 0.07 ą 45% -0.0 0.07 ą 11% perf-profile.children.cycles-pp.native_irq_return_iret
> > > > 0.01 ą223% +0.0 0.01 ą223% perf-profile.children.cycles-pp.ktime_get_update_offsets_now
> > > > 0.06 ą 45% +0.0 0.06 ą 8% perf-profile.children.cycles-pp.asm_exc_page_fault
> > > > 0.18 ą 16% +0.0 0.18 ą 14% perf-profile.children.cycles-pp.perf_callchain_kernel
> > > > 0.12 ą 16% +0.0 0.12 ą 12% perf-profile.children.cycles-pp.unwind_next_frame
> > > > 0.36 ą 18% +0.0 0.37 ą 10% perf-profile.children.cycles-pp.task_tick_fair
> > > > 0.58 ą 14% +0.0 0.58 ą 10% perf-profile.children.cycles-pp.hrtimer_interrupt
> > > > 0.49 ą 14% +0.0 0.50 ą 11% perf-profile.children.cycles-pp.__hrtimer_run_queues
> > > > 0.05 ą 46% +0.0 0.05 ą 45% perf-profile.children.cycles-pp.__unwind_start
> > > > 0.45 ą 14% +0.0 0.46 ą 11% perf-profile.children.cycles-pp.tick_sched_handle
> > > > 0.46 ą 14% +0.0 0.46 ą 11% perf-profile.children.cycles-pp.tick_sched_timer
> > > > 0.45 ą 15% +0.0 0.45 ą 11% perf-profile.children.cycles-pp.update_process_times
> > > > 0.06 ą 11% +0.0 0.07 ą 12% perf-profile.children.cycles-pp.kmem_cache_free_bulk
> > > > 0.58 ą 14% +0.0 0.58 ą 10% perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
> > > > 0.00 +0.0 0.01 ą223% perf-profile.children.cycles-pp.record__mmap_read_evlist
> > > > 0.00 +0.0 0.01 ą223% perf-profile.children.cycles-pp.perf_mmap__push
> > > > 0.00 +0.0 0.01 ą223% perf-profile.children.cycles-pp.record__pushfn
> > > > 0.00 +0.0 0.01 ą223% perf-profile.children.cycles-pp.ksys_write
> > > > 0.00 +0.0 0.01 ą223% perf-profile.children.cycles-pp.vfs_write
> > > > 0.00 +0.0 0.01 ą223% perf-profile.children.cycles-pp.__libc_write
> > > > 0.00 +0.0 0.01 ą223% perf-profile.children.cycles-pp.generic_file_write_iter
> > > > 0.00 +0.0 0.01 ą223% perf-profile.children.cycles-pp.__generic_file_write_iter
> > > > 0.00 +0.0 0.01 ą223% perf-profile.children.cycles-pp.generic_perform_write
> > > > 0.00 +0.0 0.01 ą223% perf-profile.children.cycles-pp.build_id__mark_dso_hit
> > > > 0.39 ą 17% +0.0 0.40 ą 10% perf-profile.children.cycles-pp.scheduler_tick
> > > > 0.00 +0.0 0.01 ą223% perf-profile.children.cycles-pp.clockevents_program_event
> > > > 0.05 ą 45% +0.0 0.06 ą 11% perf-profile.children.cycles-pp.mas_wr_store_entry
> > > > 0.60 ą 14% +0.0 0.61 ą 9% perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
> > > > 0.08 ą 8% +0.0 0.10 ą 12% perf-profile.children.cycles-pp.mas_destroy
> > > > 0.08 ą 9% +0.0 0.09 ą 21% perf-profile.children.cycles-pp.perf_session__deliver_event
> > > > 0.08 ą 12% +0.0 0.09 ą 33% perf-profile.children.cycles-pp.ordered_events__queue
> > > > 0.08 ą 11% +0.0 0.10 ą 22% perf-profile.children.cycles-pp.__ordered_events__flush
> > > > 0.08 ą 9% +0.0 0.10 ą 22% perf-profile.children.cycles-pp.perf_session__process_user_event
> > > > 0.06 ą 13% +0.0 0.08 ą 14% perf-profile.children.cycles-pp.kmem_cache_alloc
> > > > 0.07 ą 9% +0.0 0.09 ą 33% perf-profile.children.cycles-pp.queue_event
> > > > 0.08 ą 8% +0.0 0.10 ą 31% perf-profile.children.cycles-pp.process_simple
> > > > 0.00 +0.0 0.03 ą100% perf-profile.children.cycles-pp.evlist__parse_sample
> > > > 0.06 ą 6% +0.0 0.08 ą 8% perf-profile.children.cycles-pp.memset_erms
> > > > 0.22 ą 7% +0.0 0.26 ą 23% perf-profile.children.cycles-pp.__libc_start_main
> > > > 0.22 ą 7% +0.0 0.26 ą 23% perf-profile.children.cycles-pp.main
> > > > 0.22 ą 7% +0.0 0.26 ą 23% perf-profile.children.cycles-pp.run_builtin
> > > > 0.21 ą 9% +0.0 0.25 ą 23% perf-profile.children.cycles-pp.cmd_record
> > > > 0.21 ą 9% +0.0 0.25 ą 23% perf-profile.children.cycles-pp.__cmd_record
> > > > 0.20 ą 9% +0.0 0.24 ą 24% perf-profile.children.cycles-pp.cmd_sched
> > > > 0.17 ą 11% +0.0 0.21 ą 25% perf-profile.children.cycles-pp.reader__read_event
> > > > 0.17 ą 11% +0.0 0.21 ą 26% perf-profile.children.cycles-pp.record__finish_output
> > > > 0.17 ą 11% +0.0 0.21 ą 26% perf-profile.children.cycles-pp.perf_session__process_events
> > > > 0.00 +0.0 0.04 ą 45% perf-profile.children.cycles-pp.kmem_cache_free
> > > > 0.17 ą 7% +0.1 0.22 ą 8% perf-profile.children.cycles-pp.mas_alloc_nodes
> > > > 0.11 ą 9% +0.1 0.17 ą 6% perf-profile.children.cycles-pp.kmem_cache_alloc_bulk
> > > > 0.00 +0.1 0.06 ą 13% perf-profile.children.cycles-pp.vm_area_dup
> > > > 0.16 ą 8% +0.1 0.22 ą 6% perf-profile.children.cycles-pp.mas_preallocate
> > > > 67.20 ą 8% +0.1 67.28 ą 7% perf-profile.children.cycles-pp.osq_lock
> > > > 68.59 ą 8% +0.1 68.66 ą 7% perf-profile.children.cycles-pp.down_write_killable
> > > > 1.04 ą 8% +0.1 1.12 ą 7% perf-profile.children.cycles-pp.rwsem_spin_on_owner
> > > > 70.08 ą 8% +0.1 70.15 ą 7% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
> > > > 68.52 ą 8% +0.1 68.60 ą 7% perf-profile.children.cycles-pp.rwsem_down_write_slowpath
> > > > 70.03 ą 8% +0.1 70.11 ą 7% perf-profile.children.cycles-pp.do_syscall_64
> > > > 68.46 ą 8% +0.1 68.55 ą 7% perf-profile.children.cycles-pp.rwsem_optimistic_spin
> > > > 0.55 ą 8% +0.2 0.71 ą 8% perf-profile.children.cycles-pp.do_mas_align_munmap
> > > > 35.12 ą 8% +0.2 35.31 ą 7% perf-profile.children.cycles-pp.__munmap
> > > > 0.00 +0.2 0.22 ą 7% perf-profile.children.cycles-pp.vma_expand
> > > > 0.00 +0.2 0.22 ą223% perf-profile.children.cycles-pp.intel_idle_irq
> > > > 34.98 ą 8% +0.2 35.20 ą 7% perf-profile.children.cycles-pp.__x64_sys_munmap
> > > > 34.97 ą 8% +0.2 35.20 ą 7% perf-profile.children.cycles-pp.__vm_munmap
> > > > 0.64 ą 13% +0.2 0.88 ą 55% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
> > > > 0.00 +0.3 0.30 ą 7% perf-profile.children.cycles-pp.__vma_adjust
> > > > 0.00 +0.4 0.36 ą 6% perf-profile.children.cycles-pp.__split_vma
> > > > 8.42 ą 91% +6.2 14.60 ą 77% perf-profile.children.cycles-pp.intel_idle_ibrs
> > > > 29.52 ą 19% -0.1 29.45 ą 17% perf-profile.self.cycles-pp.mwait_idle_with_hints
> > > > 0.18 ą 9% -0.1 0.12 ą 10% perf-profile.self.cycles-pp.rwsem_optimistic_spin
> > > > 0.04 ą 45% -0.0 0.00 perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
> > > > 0.04 ą 44% -0.0 0.00 perf-profile.self.cycles-pp.mmap_region
> > > > 0.10 ą 5% -0.0 0.08 ą 9% perf-profile.self.cycles-pp.mas_rev_awalk
> > > > 0.06 ą 7% -0.0 0.04 ą 45% perf-profile.self.cycles-pp.mas_walk
> > > > 0.06 ą 11% -0.0 0.04 ą 45% perf-profile.self.cycles-pp.do_mas_align_munmap
> > > > 0.08 ą 8% -0.0 0.07 ą 14% perf-profile.self.cycles-pp.syscall_exit_to_user_mode
> > > > 0.08 ą 7% -0.0 0.07 ą 8% perf-profile.self.cycles-pp.syscall_return_via_sysret
> > > > 0.06 ą 13% -0.0 0.05 ą 7% perf-profile.self.cycles-pp.down_write_killable
> > > > 0.07 ą 45% -0.0 0.07 ą 11% perf-profile.self.cycles-pp.native_irq_return_iret
> > > > 0.05 ą 45% -0.0 0.05 ą 47% perf-profile.self.cycles-pp.unwind_next_frame
> > > > 0.00 +0.0 0.01 ą223% perf-profile.self.cycles-pp.ktime_get_update_offsets_now
> > > > 0.05 ą 45% +0.0 0.06 ą 11% perf-profile.self.cycles-pp.kmem_cache_free_bulk
> > > > 0.00 +0.0 0.02 ą141% perf-profile.self.cycles-pp.kmem_cache_free
> > > > 0.07 ą 8% +0.0 0.09 ą 33% perf-profile.self.cycles-pp.queue_event
> > > > 0.06 ą 8% +0.0 0.08 ą 8% perf-profile.self.cycles-pp.memset_erms
> > > > 0.04 ą 45% +0.0 0.08 ą 6% perf-profile.self.cycles-pp.kmem_cache_alloc_bulk
> > > > 66.61 ą 8% +0.1 66.68 ą 7% perf-profile.self.cycles-pp.osq_lock
> > > > 1.02 ą 8% +0.1 1.10 ą 7% perf-profile.self.cycles-pp.rwsem_spin_on_owner
> > > >
> > > >
> > > >
> > > > If you fix the issue, kindly add following tag
> > > > | Reported-by: kernel test robot <yujie.liu@...el.com>
> > > > | Link: https://lore.kernel.org/oe-lkp/202212151657.5d11a672-yujie.liu@intel.com
> > > >
> > > >
> > > > To reproduce:
> > > >
> > > > git clone https://github.com/intel/lkp-tests.git
> > > > cd lkp-tests
> > > > sudo bin/lkp install job.yaml # job file is attached in this email
> > > > bin/lkp split-job --compatible job.yaml # generate the yaml file for lkp run
> > > > sudo bin/lkp run generated-yaml-file
> > > >
> > > > # if come across any failure that blocks the test,
> > > > # please remove ~/.lkp and /lkp dir to run from a clean state.
> > > >
> > > >
> > > > Disclaimer:
> > > > Results have been estimated based on internal Intel analysis and are provided
> > > > for informational purposes only. Any difference in system hardware or software
> > > > design or configuration may affect actual performance.
> > > >
> > > >
> > > > --
> > > > 0-DAY CI Kernel Test Service
> > > > https://01.org/lkp
Powered by blists - more mailing lists