lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening PHC | |
Open Source and information security mailing list archives
| ||
|
Date: Wed, 5 Aug 2020 17:08:58 -0700 From: Peter Oskolkov <posk@...gle.com> To: Mathieu Desnoyers <mathieu.desnoyers@...icios.com>, "Paul E . McKenney" <paulmck@...nel.org>, linux-kernel@...r.kernel.org Cc: Paul Turner <pjt@...gle.com>, Chris Kennelly <ckennelly@...gle.com>, Peter Oskolkov <posk@...k.io>, Peter Oskolkov <posk@...gle.com> Subject: [PATCH 1/2] membarrier: add MEMBARRIER_CMD_PRIVATE_RESTART_RSEQ_ON_CPU This patchset is based on Google-internal RSEQ work done by Paul Turner and Andrew Hunter. When working with per-CPU RSEQ-based memory allocations, it is sometimes important to make sure that a global memory location is no longer accessed from RSEQ critical sections. For example, there can be two per-CPU lists, one is "active" and accessed per-CPU, while another one is inactive and worked on asynchronously "off CPU" (e.g. garbage collection is performed). Then at some point the two lists are swapped, and a fast RCU-like mechanism is required to make sure that the previously active list is no longer accessed. This patch introduces such a mechanism: in short, membarrier() syscall issues an IPI to a CPU, restarting any potentially active RSEQ critical sections on the CPU, with an option to restart RSEQ CSs on all CPUs. The second patch in the patchset adds a selftest of this feature. Signed-off-by: Peter Oskolkov <posk@...gle.com> --- include/uapi/linux/membarrier.h | 8 ++++++ kernel/sched/membarrier.c | 51 +++++++++++++++++++++++++++++++-- 2 files changed, 57 insertions(+), 2 deletions(-) diff --git a/include/uapi/linux/membarrier.h b/include/uapi/linux/membarrier.h index 5891d7614c8c..169ffb613397 100644 --- a/include/uapi/linux/membarrier.h +++ b/include/uapi/linux/membarrier.h @@ -114,6 +114,13 @@ * If this command is not implemented by an * architecture, -EINVAL is returned. * Returns 0 on success. + * @MEMBARRIER_CMD_PRIVATE_RESTART_RSEQ_ON_CPU: + * If a thread belonging to the current process + * is currently in an RSEQ critical section on the + * CPU identified by flags parameter, restart it. + * @flags: if @flags >= 0, identifies the CPU to + * restart RSEQ CS on; if == -1, restarts + * RSEQ CSs on all CPUs. * @MEMBARRIER_CMD_SHARED: * Alias to MEMBARRIER_CMD_GLOBAL. Provided for * header backward compatibility. @@ -131,6 +138,7 @@ enum membarrier_cmd { MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED = (1 << 4), MEMBARRIER_CMD_PRIVATE_EXPEDITED_SYNC_CORE = (1 << 5), MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_SYNC_CORE = (1 << 6), + MEMBARRIER_CMD_PRIVATE_RESTART_RSEQ_ON_CPU = (1 << 7), /* Alias for header backward compatibility. */ MEMBARRIER_CMD_SHARED = MEMBARRIER_CMD_GLOBAL, diff --git a/kernel/sched/membarrier.c b/kernel/sched/membarrier.c index 168479a7d61b..edcc1386daf5 100644 --- a/kernel/sched/membarrier.c +++ b/kernel/sched/membarrier.c @@ -18,11 +18,19 @@ #define MEMBARRIER_PRIVATE_EXPEDITED_SYNC_CORE_BITMASK 0 #endif +#ifdef CONFIG_RSEQ +#define MEMBARRIER_CMD_PRIVATE_RESTART_RSEQ_ON_CPU_BITMASK \ + MEMBARRIER_CMD_PRIVATE_RESTART_RSEQ_ON_CPU +#else +#define MEMBARRIER_CMD_PRIVATE_RESTART_RSEQ_ON_CPU_BITMASK 0 +#endif + #define MEMBARRIER_CMD_BITMASK \ (MEMBARRIER_CMD_GLOBAL | MEMBARRIER_CMD_GLOBAL_EXPEDITED \ | MEMBARRIER_CMD_REGISTER_GLOBAL_EXPEDITED \ | MEMBARRIER_CMD_PRIVATE_EXPEDITED \ | MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED \ + | MEMBARRIER_CMD_PRIVATE_RESTART_RSEQ_ON_CPU_BITMASK \ | MEMBARRIER_PRIVATE_EXPEDITED_SYNC_CORE_BITMASK) static void ipi_mb(void *info) @@ -308,10 +316,47 @@ static int membarrier_register_private_expedited(int flags) return 0; } +#ifdef CONFIG_RSEQ +static void membarrier_rseq_ipi(void *arg) +{ + struct task_struct *leader = (struct task_struct *)arg; + + if (current->group_leader != leader) /* Not our process. */ + return; + if (!current->rseq) /* RSEQ not set up for current task/thread. */ + return; + + rseq_preempt(current); +} +#endif + +static int membarrier_private_restart_rseq_on_cpu(int cpu_id) +{ +#ifdef CONFIG_RSEQ + /* syscalls are not allowed inside rseq critical sections. */ + if (cpu_id == raw_smp_processor_id()) + return 0; + + if (cpu_id >= 0) { + return smp_call_function_single(cpu_id, membarrier_rseq_ipi, + current->group_leader, true); + } else if (cpu_id == -1) { + on_each_cpu(membarrier_rseq_ipi, + current->group_leader, true); + } else { + return -EINVAL; + } +#endif + return 0; +} + /** * sys_membarrier - issue memory barriers on a set of threads * @cmd: Takes command values defined in enum membarrier_cmd. - * @flags: Currently needs to be 0. For future extensions. + * @flags: Currently needs to be 0 for all commands other than + * MEMBARRIER_CMD_PRIVATE_RESTART_RSEQ_ON_CPU: in the latter + * case it indicates the CPU on which to interrupt (= restart) + * the RSEQ critical section, or -1 to restart RSEQ CSs on all CPUs. * * If this system call is not implemented, -ENOSYS is returned. If the * command specified does not exist, not available on the running @@ -339,7 +384,7 @@ static int membarrier_register_private_expedited(int flags) */ SYSCALL_DEFINE2(membarrier, int, cmd, int, flags) { - if (unlikely(flags)) + if (unlikely(flags && cmd != MEMBARRIER_CMD_PRIVATE_RESTART_RSEQ_ON_CPU)) return -EINVAL; switch (cmd) { case MEMBARRIER_CMD_QUERY: @@ -369,6 +414,8 @@ SYSCALL_DEFINE2(membarrier, int, cmd, int, flags) return membarrier_private_expedited(MEMBARRIER_FLAG_SYNC_CORE); case MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_SYNC_CORE: return membarrier_register_private_expedited(MEMBARRIER_FLAG_SYNC_CORE); + case MEMBARRIER_CMD_PRIVATE_RESTART_RSEQ_ON_CPU: + return membarrier_private_restart_rseq_on_cpu(flags); default: return -EINVAL; } -- 2.28.0.163.g6104cc2f0b6-goog
Powered by blists - more mailing lists