| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <Y9FNm50H+w1B26yR@hyeyoo>
Date: Thu, 26 Jan 2023 00:41:15 +0900
From: Hyeonggon Yoo <42.hyeyoo@...il.com>
To: Michal Hocko <mhocko@...e.com>
Cc: Vlastimil Babka <vbabka@...e.cz>,
Andrew Morton <akpm@...ux-foundation.org>,
Christoph Lameter <cl@...ux.com>,
Pekka Enberg <penberg@...nel.org>,
David Rientjes <rientjes@...gle.com>,
Joonsoo Kim <iamjoonsoo.kim@....com>,
Roman Gushchin <roman.gushchin@...ux.dev>,
Ingo Molnar <mingo@...hat.com>,
Johannes Weiner <hannes@...xchg.org>,
Shakeel Butt <shakeelb@...gle.com>,
Muchun Song <muchun.song@...ux.dev>,
Matthew Wilcox <willy@...radead.org>, linux-mm@...ck.org,
linux-kernel@...r.kernel.org,
Peter Zijlstra <peterz@...radead.org>,
Juri Lelli <juri.lelli@...hat.com>,
Vincent Guittot <vincent.guittot@...aro.org>,
Dietmar Eggemann <dietmar.eggemann@....com>,
Steven Rostedt <rostedt@...dmis.org>,
Ben Segall <bsegall@...gle.com>,
Mel Gorman <mgorman@...hsingularity.net>,
Daniel Bristot de Oliveira <bristot@...hat.com>,
Valentin Schneider <vschneid@...hat.com>,
Dennis Zhou <dennis@...nel.org>, Tejun Heo <tj@...nel.org>
Subject: Re: [PATCH mm-unstable] lib/Kconfig.debug: do not enable
DEBUG_PREEMPT by default
From: Hyeonggon Yoo <42.hyeyoo@...il.com>
To: Michal Hocko <mhocko@...e.com>
Cc: Vlastimil Babka <vbabka@...e.cz>,
Andrew Morton <akpm@...ux-foundation.org>,
Christoph Lameter <cl@...ux.com>, Pekka Enberg <penberg@...nel.org>,
David Rientjes <rientjes@...gle.com>,
Joonsoo Kim <iamjoonsoo.kim@....com>,
Roman Gushchin <roman.gushchin@...ux.dev>,
Ingo Molnar <mingo@...hat.com>,
Johannes Weiner <hannes@...xchg.org>,
Shakeel Butt <shakeelb@...gle.com>,
Muchun Song <muchun.song@...ux.dev>,
Matthew Wilcox <willy@...radead.org>, linux-mm@...ck.org,
linux-kernel@...r.kernel.org, Peter Zijlstra <peterz@...radead.org>,
Juri Lelli <juri.lelli@...hat.com>,
Vincent Guittot <vincent.guittot@...aro.org>,
Dietmar Eggemann <dietmar.eggemann@....com>,
Steven Rostedt <rostedt@...dmis.org>,
Ben Segall <bsegall@...gle.com>,
Mel Gorman <mgorman@...hsingularity.net>,
Daniel Bristot de Oliveira <bristot@...hat.com>,
Valentin Schneider <vschneid@...hat.com>,
Dennis Zhou <dennis@...nel.org>, Tejun Heo <tj@...nel.org>
Bcc:
Subject: Re: [PATCH mm-unstable] lib/Kconfig.debug: do not enable
DEBUG_PREEMPT by default
Reply-To:
In-Reply-To: <Y85MNmZDc5czMRUJ@...p22.suse.cz>
On Mon, Jan 23, 2023 at 09:58:30AM +0100, Michal Hocko wrote:
> On Sat 21-01-23 20:54:15, Hyeonggon Yoo wrote:
> > On Sat, Jan 21, 2023 at 12:29:44PM +0100, Vlastimil Babka wrote:
> > > On 1/21/23 04:39, Hyeonggon Yoo wrote:
> > > > In workloads where this_cpu operations are frequently performed,
> > > > enabling DEBUG_PREEMPT may result in significant increase in
> > > > runtime overhead due to frequent invocation of
> > > > __this_cpu_preempt_check() function.
> > > >
> > > > This can be demonstrated through benchmarks such as hackbench where this
> > > > configuration results in a 10% reduction in performance, primarily due to
> > > > the added overhead within memcg charging path.
> > > >
> > > > Therefore, do not to enable DEBUG_PREEMPT by default and make users aware
> > > > of its potential impact on performance in some workloads.
> > > >
> > > > hackbench-process-sockets
> > > > debug_preempt no_debug_preempt
> > > > Amean 1 0.4743 ( 0.00%) 0.4295 * 9.45%*
> > > > Amean 4 1.4191 ( 0.00%) 1.2650 * 10.86%*
> > > > Amean 7 2.2677 ( 0.00%) 2.0094 * 11.39%*
> > > > Amean 12 3.6821 ( 0.00%) 3.2115 * 12.78%*
> > > > Amean 21 6.6752 ( 0.00%) 5.7956 * 13.18%*
> > > > Amean 30 9.6646 ( 0.00%) 8.5197 * 11.85%*
> > > > Amean 48 15.3363 ( 0.00%) 13.5559 * 11.61%*
> > > > Amean 79 24.8603 ( 0.00%) 22.0597 * 11.27%*
> > > > Amean 96 30.1240 ( 0.00%) 26.8073 * 11.01%*
Hello Michal, thanks for looking at this.
> Do you happen to have any perf data collected during those runs? I
> would be interested in the memcg side of things. Maybe we can do
> something better there.
Yes, below is performance data I've collected.
6.1.8-debug-preempt-dirty
=========================
Overhead Command Shared Object Symbol
+ 9.14% hackbench [kernel.vmlinux] [k] check_preemption_disabled
+ 7.33% hackbench [kernel.vmlinux] [k] copy_user_enhanced_fast_string
+ 7.32% hackbench [kernel.vmlinux] [k] mod_objcg_state
3.55% hackbench [kernel.vmlinux] [k] refill_obj_stock
3.39% hackbench [kernel.vmlinux] [k] debug_smp_processor_id
2.97% hackbench [kernel.vmlinux] [k] memset_erms
2.55% hackbench [kernel.vmlinux] [k] __check_object_size
+ 2.36% hackbench [kernel.vmlinux] [k] native_queued_spin_lock_slowpath
1.76% hackbench [kernel.vmlinux] [k] unix_stream_read_generic
1.64% hackbench [kernel.vmlinux] [k] __slab_free
1.58% hackbench [kernel.vmlinux] [k] unix_stream_sendmsg
1.46% hackbench [kernel.vmlinux] [k] memcg_slab_post_alloc_hook
1.35% hackbench [kernel.vmlinux] [k] vfs_write
1.33% hackbench [kernel.vmlinux] [k] vfs_read
1.28% hackbench [kernel.vmlinux] [k] __alloc_skb
1.18% hackbench [kernel.vmlinux] [k] sock_read_iter
1.16% hackbench [kernel.vmlinux] [k] obj_cgroup_charge
1.16% hackbench [kernel.vmlinux] [k] entry_SYSCALL_64
1.14% hackbench [kernel.vmlinux] [k] sock_write_iter
1.12% hackbench [kernel.vmlinux] [k] skb_release_data
1.08% hackbench [kernel.vmlinux] [k] sock_wfree
1.07% hackbench [kernel.vmlinux] [k] cache_from_obj
0.96% hackbench [kernel.vmlinux] [k] unix_destruct_scm
0.95% hackbench [kernel.vmlinux] [k] kmem_cache_free
0.94% hackbench [kernel.vmlinux] [k] __kmem_cache_alloc_node
0.92% hackbench [kernel.vmlinux] [k] kmem_cache_alloc_node
0.89% hackbench [kernel.vmlinux] [k] _raw_spin_lock_irqsave
0.84% hackbench [kernel.vmlinux] [k] __x86_indirect_thunk_array
0.84% hackbench libc.so.6 [.] write
0.81% hackbench [kernel.vmlinux] [k] exit_to_user_mode_prepare
0.76% hackbench libc.so.6 [.] read
0.75% hackbench [kernel.vmlinux] [k] syscall_trace_enter.constprop.0
0.75% hackbench [kernel.vmlinux] [k] preempt_count_add
0.74% hackbench [kernel.vmlinux] [k] cmpxchg_double_slab.constprop.0.isra.0
0.69% hackbench [kernel.vmlinux] [k] get_partial_node
0.69% hackbench [kernel.vmlinux] [k] __virt_addr_valid
0.69% hackbench [kernel.vmlinux] [k] __rcu_read_unlock
0.65% hackbench [kernel.vmlinux] [k] get_obj_cgroup_from_current
0.63% hackbench [kernel.vmlinux] [k] __kmem_cache_free
0.62% hackbench [kernel.vmlinux] [k] entry_SYSRETQ_unsafe_stack
0.60% hackbench [kernel.vmlinux] [k] __rcu_read_lock
0.59% hackbench [kernel.vmlinux] [k] syscall_exit_to_user_mode_prepare
0.54% hackbench [kernel.vmlinux] [k] __unfreeze_partials
0.53% hackbench [kernel.vmlinux] [k] check_stack_object
0.52% hackbench [kernel.vmlinux] [k] entry_SYSCALL_64_after_hwframe
0.51% hackbench [kernel.vmlinux] [k] security_file_permission
0.50% hackbench [kernel.vmlinux] [k] __x64_sys_write
0.49% hackbench [kernel.vmlinux] [k] bpf_lsm_file_permission
0.48% hackbench [kernel.vmlinux] [k] ___slab_alloc
0.46% hackbench [kernel.vmlinux] [k] __check_heap_object
and attached flamegraph-6.1.8-debug-preempt-dirty.svg.
6.1.8 (no debug preempt)
========================
Overhead Command Shared Object Symbol
+ 10.96% hackbench [kernel.vmlinux] [k] mod_objcg_state
+ 8.16% hackbench [kernel.vmlinux] [k] copy_user_enhanced_fast_string
3.29% hackbench [kernel.vmlinux] [k] memset_erms
3.07% hackbench [kernel.vmlinux] [k] __slab_free
2.89% hackbench [kernel.vmlinux] [k] refill_obj_stock
2.82% hackbench [kernel.vmlinux] [k] __check_object_size
+ 2.72% hackbench [kernel.vmlinux] [k] native_queued_spin_lock_slowpath
1.96% hackbench [kernel.vmlinux] [k] __x86_indirect_thunk_rax
1.88% hackbench [kernel.vmlinux] [k] memcg_slab_post_alloc_hook
1.69% hackbench [kernel.vmlinux] [k] __rcu_read_unlock
1.54% hackbench [kernel.vmlinux] [k] __alloc_skb
1.53% hackbench [kernel.vmlinux] [k] unix_stream_sendmsg
1.46% hackbench [kernel.vmlinux] [k] kmem_cache_free
1.44% hackbench [kernel.vmlinux] [k] vfs_write
1.43% hackbench [kernel.vmlinux] [k] vfs_read
1.33% hackbench [kernel.vmlinux] [k] unix_stream_read_generic
1.31% hackbench [kernel.vmlinux] [k] sock_write_iter
1.27% hackbench [kernel.vmlinux] [k] kmalloc_slab
1.22% hackbench [kernel.vmlinux] [k] __rcu_read_lock
1.20% hackbench [kernel.vmlinux] [k] sock_read_iter
1.18% hackbench [kernel.vmlinux] [k] __entry_text_start
1.15% hackbench [kernel.vmlinux] [k] kmem_cache_alloc_node
1.12% hackbench [kernel.vmlinux] [k] unix_stream_recvmsg
1.10% hackbench [kernel.vmlinux] [k] obj_cgroup_charge
0.98% hackbench [kernel.vmlinux] [k] __kmem_cache_alloc_node
0.97% hackbench libc.so.6 [.] write
0.91% hackbench [kernel.vmlinux] [k] exit_to_user_mode_prepare
0.88% hackbench [kernel.vmlinux] [k] __kmem_cache_free
0.87% hackbench [kernel.vmlinux] [k] syscall_trace_enter.constprop.0
0.86% hackbench [kernel.vmlinux] [k] __kmalloc_node_track_caller
0.84% hackbench libc.so.6 [.] read
0.81% hackbench [kernel.vmlinux] [k] __lock_text_start
0.80% hackbench [kernel.vmlinux] [k] cache_from_obj
0.74% hackbench [kernel.vmlinux] [k] get_obj_cgroup_from_current
0.73% hackbench [kernel.vmlinux] [k] entry_SYSRETQ_unsafe_stack
0.72% hackbench [kernel.vmlinux] [k] unix_destruct_scm
0.70% hackbench [kernel.vmlinux] [k] get_partial_node
0.69% hackbench [kernel.vmlinux] [k] syscall_exit_to_user_mode_prepare
0.65% hackbench [kernel.vmlinux] [k] kfree
0.63% hackbench [kernel.vmlinux] [k] __unfreeze_partials
0.60% hackbench [kernel.vmlinux] [k] cmpxchg_double_slab.constprop.0.isra.0
0.58% hackbench [kernel.vmlinux] [k] skb_release_data
0.56% hackbench [kernel.vmlinux] [k] __virt_addr_valid
0.56% hackbench [kernel.vmlinux] [k] entry_SYSCALL_64_after_hwframe
0.56% hackbench [kernel.vmlinux] [k] __check_heap_object
0.55% hackbench [kernel.vmlinux] [k] sock_wfree
0.54% hackbench [kernel.vmlinux] [k] __audit_syscall_entry
0.53% hackbench [kernel.vmlinux] [k] ___slab_alloc
0.53% hackbench [kernel.vmlinux] [k] check_stack_object
0.52% hackbench [kernel.vmlinux] [k] bpf_lsm_file_permission
and attached flamegraph-6.1.8.svg.
If you need more information, feel free to ask.
--
Thanks,
Hyeonggon
> --
> Michal Hocko
> SUSE Labs
Download attachment "flamegraph-6.1.8-debug-preempt-dirty.svg" of type "image/svg+xml" (126637 bytes)
Download attachment "flamegraph-6.1.8.svg" of type "image/svg+xml" (112745 bytes)
Powered by blists - more mailing lists