linux-kernel - [PATCH tip 0/3] sched: make migrate_enable/migrate

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <20250810030442.246974-1-dongml2@chinatelecom.cn>
Date: Sun, 10 Aug 2025 11:04:39 +0800
From: Menglong Dong <menglong8.dong@...il.com>
To: peterz@...radead.org,
	alexei.starovoitov@...il.com
Cc: mingo@...hat.com,
	juri.lelli@...hat.com,
	vincent.guittot@...aro.org,
	dietmar.eggemann@....com,
	rostedt@...dmis.org,
	bsegall@...gle.com,
	mgorman@...e.de,
	vschneid@...hat.com,
	ast@...nel.org,
	daniel@...earbox.net,
	john.fastabend@...il.com,
	andrii@...nel.org,
	martin.lau@...ux.dev,
	eddyz87@...il.com,
	song@...nel.org,
	yonghong.song@...ux.dev,
	kpsingh@...nel.org,
	sdf@...ichev.me,
	haoluo@...gle.com,
	jolsa@...nel.org,
	jani.nikula@...el.com,
	simona.vetter@...ll.ch,
	tzimmermann@...e.de,
	linux-kernel@...r.kernel.org,
	bpf@...r.kernel.org
Subject: [PATCH tip 0/3] sched: make migrate_enable/migrate_disable inline

In this series, we make migrate_enable/migrate_disable inline to obtain
better performance in some case.

In the first patch, we add the macro "COMPILE_OFFSETS" to all the
asm-offset.c to avoid circular dependency in the 2nd patch.

In the 2nd patch, we generate the offset of nr_pinned in "struct rq" with
rq-offsets.c, as the "struct rq" is defined internally and we need to
access the "nr_pinned" field in migrate_enable and migrate_disable. Then,
we move the definition of migrate_enable/migrate_disable from
kernel/sched/core.c to include/linux/sched.h.

In the 3rd patch, we fix some typos in include/linux/preempt.h.

One of the beneficiaries of this series is BPF trampoline. Without this
series, the migrate_enable/migrate_disable is hot when we run the
benchmark for FENTRY, FEXIT, MODIFY_RETURN, etc:

  54.63% bpf_prog_2dcccf652aac1793_bench_trigger_fentry [k]
                 bpf_prog_2dcccf652aac1793_bench_trigger_fentry
  10.43% [kernel] [k] migrate_enable
  10.07% bpf_trampoline_6442517037 [k] bpf_trampoline_6442517037
  8.06% [kernel] [k] __bpf_prog_exit_recur
  4.11% libc.so.6 [.] syscall
  2.15% [kernel] [k] entry_SYSCALL_64
  1.48% [kernel] [k] memchr_inv
  1.32% [kernel] [k] fput
  1.16% [kernel] [k] _copy_to_user
  0.73% [kernel] [k] bpf_prog_test_run_raw_tp

Before this patch, the performance of BPF FENTRY is:

  fentry         :  113.030 ± 0.149M/s
  fentry         :  112.501 ± 0.187M/s
  fentry         :  112.828 ± 0.267M/s
  fentry         :  115.287 ± 0.241M/s

After this patch, the performance of BPF FENTRY increases to:

  fentry         :  143.644 ± 0.670M/s
  fentry         :  149.764 ± 0.362M/s
  fentry         :  149.642 ± 0.156M/s
  fentry         :  145.263 ± 0.221M/s

Menglong Dong (3):
  arch: add the macro COMPILE_OFFSETS to all the asm-offsets.c
  sched: make migrate_enable/migrate_disable inline
  sched: fix some typos in include/linux/preempt.h

 Kbuild                               | 13 ++++-
 arch/alpha/kernel/asm-offsets.c      |  1 +
 arch/arc/kernel/asm-offsets.c        |  1 +
 arch/arm/kernel/asm-offsets.c        |  2 +
 arch/arm64/kernel/asm-offsets.c      |  1 +
 arch/csky/kernel/asm-offsets.c       |  1 +
 arch/hexagon/kernel/asm-offsets.c    |  1 +
 arch/loongarch/kernel/asm-offsets.c  |  2 +
 arch/m68k/kernel/asm-offsets.c       |  1 +
 arch/microblaze/kernel/asm-offsets.c |  1 +
 arch/mips/kernel/asm-offsets.c       |  2 +
 arch/nios2/kernel/asm-offsets.c      |  1 +
 arch/openrisc/kernel/asm-offsets.c   |  1 +
 arch/parisc/kernel/asm-offsets.c     |  1 +
 arch/powerpc/kernel/asm-offsets.c    |  1 +
 arch/riscv/kernel/asm-offsets.c      |  1 +
 arch/s390/kernel/asm-offsets.c       |  1 +
 arch/sh/kernel/asm-offsets.c         |  1 +
 arch/sparc/kernel/asm-offsets.c      |  1 +
 arch/um/kernel/asm-offsets.c         |  2 +
 arch/xtensa/kernel/asm-offsets.c     |  1 +
 include/linux/preempt.h              | 11 ++---
 include/linux/sched.h                | 72 ++++++++++++++++++++++++++++
 kernel/bpf/verifier.c                |  3 +-
 kernel/sched/core.c                  | 56 ++--------------------
 kernel/sched/rq-offsets.c            | 12 +++++
 26 files changed, 129 insertions(+), 62 deletions(-)
 create mode 100644 kernel/sched/rq-offsets.c

-- 
2.50.1