linux-kernel - Re: [PATCH v2] x86/alternatives: remove false sharing in poke_int3

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20250324111817.GA14944@noisy.programming.kicks-ass.net>
Date: Mon, 24 Mar 2025 12:18:17 +0100
From: Peter Zijlstra <peterz@...radead.org>
To: Eric Dumazet <edumazet@...gle.com>
Cc: Thomas Gleixner <tglx@...utronix.de>, Ingo Molnar <mingo@...hat.com>,
	Borislav Petkov <bp@...en8.de>,
	Dave Hansen <dave.hansen@...ux.intel.com>,
	"H . Peter Anvin" <hpa@...or.com>,
	Steven Rostedt <rostedt@...dmis.org>,
	linux-kernel <linux-kernel@...r.kernel.org>,
	Alexei Starovoitov <ast@...nel.org>,
	Daniel Borkmann <daniel@...earbox.net>,
	Masami Hiramatsu <mhiramat@...nel.org>, x86@...nel.org,
	bpf@...r.kernel.org, Eric Dumazet <eric.dumazet@...il.com>,
	Greg Thelen <gthelen@...gle.com>,
	Stephane Eranian <eranian@...gle.com>
Subject: Re: [PATCH v2] x86/alternatives: remove false sharing in
 poke_int3_handler()

On Mon, Mar 24, 2025 at 08:30:51AM +0000, Eric Dumazet wrote:
> eBPF programs can be run 50,000,000 times per second on busy servers.
> 
> Whenever /proc/sys/kernel/bpf_stats_enabled is turned off,
> hundreds of calls sites are patched from text_poke_bp_batch()
> and we see a huge loss of performance due to false sharing
> on bp_desc.refs lasting up to three seconds.
> 
>    51.30%  server_bin       [kernel.kallsyms]           [k] poke_int3_handler
>             |
>             |--46.45%--poke_int3_handler
>             |          exc_int3
>             |          asm_exc_int3
>             |          |
>             |          |--24.26%--cls_bpf_classify
>             |          |          tcf_classify
>             |          |          __dev_queue_xmit
>             |          |          ip6_finish_output2
>             |          |          ip6_output
>             |          |          ip6_xmit
>             |          |          inet6_csk_xmit
>             |          |          __tcp_transmit_skb
>             |          |          |
>             |          |          |--9.00%--tcp_v6_do_rcv
>             |          |          |          tcp_v6_rcv
>             |          |          |          ip6_protocol_deliver_rcu
>             |          |          |          ip6_rcv_finish
>             |          |          |          ipv6_rcv
>             |          |          |          __netif_receive_skb
>             |          |          |          process_backlog
>             |          |          |          __napi_poll
>             |          |          |          net_rx_action
>             |          |          |          __softirqentry_text_start
>             |          |          |          asm_call_sysvec_on_stack
>             |          |          |          do_softirq_own_stack
> 
> Fix this by replacing bp_desc.refs with a per-cpu bp_refs.
> 
> Before the patch, on a host with 240 cores (480 threads):
> 
> $ bpftool prog | grep run_time_ns
> ...
> 105: sched_cls  name hn_egress  tag 699fc5eea64144e3  gpl run_time_ns
> 3009063719 run_cnt 82757845
> 
> -> average cost is 36 nsec per call
> 
> echo 0 >/proc/sys/kernel/bpf_stats_enabled
> text_poke_bp_batch(nr_entries=2)
>         text_poke_bp_batch+1
>         text_poke_finish+27
>         arch_jump_label_transform_apply+22
>         jump_label_update+98
>         __static_key_slow_dec_cpuslocked+64
>         static_key_slow_dec+31
>         bpf_stats_handler+236
>         proc_sys_call_handler+396
>         vfs_write+761
>         ksys_write+102
>         do_syscall_64+107
>         entry_SYSCALL_64_after_hwframe+103
> Took 324 usec
> 
> text_poke_bp_batch(nr_entries=164)
>         text_poke_bp_batch+1
>         text_poke_finish+27
>         arch_jump_label_transform_apply+22
>         jump_label_update+98
>         __static_key_slow_dec_cpuslocked+64
>         static_key_slow_dec+31
>         bpf_stats_handler+236
>         proc_sys_call_handler+396
>         vfs_write+761
>         ksys_write+102
>         do_syscall_64+107
>         entry_SYSCALL_64_after_hwframe+103
> Took 2655300 usec
> 
> After this patch:
> 
> $ bpftool prog | grep run_time_ns
> ...
> 105: sched_cls  name hn_egress  tag 699fc5eea64144e3  gpl run_time_ns
> 1928223019 run_cnt 67682728
> 
>  -> average cost is 28 nsec per call
> 
> echo 0 >/proc/sys/kernel/bpf_stats_enabled
> text_poke_bp_batch(nr_entries=2)
>         text_poke_bp_batch+1
>         text_poke_finish+27
>         arch_jump_label_transform_apply+22
>         jump_label_update+98
>         __static_key_slow_dec_cpuslocked+64
>         static_key_slow_dec+31
>         bpf_stats_handler+236
>         proc_sys_call_handler+396
>         vfs_write+761
>         ksys_write+102
>         do_syscall_64+107
>         entry_SYSCALL_64_after_hwframe+103
> Took 519 usec
> 
> text_poke_bp_batch(nr_entries=164)
>         text_poke_bp_batch+1
>         text_poke_finish+27
>         arch_jump_label_transform_apply+22
>         jump_label_update+98
>         __static_key_slow_dec_cpuslocked+64
>         static_key_slow_dec+31
>         bpf_stats_handler+236
>         proc_sys_call_handler+396
>         vfs_write+761
>         ksys_write+102
>         do_syscall_64+107
>         entry_SYSCALL_64_after_hwframe+103
> Took 702 usec

This is unreadable due to the amount of pointless repetition.

Also, urgh.