Date:   Wed, 17 Feb 2021 23:47:59 +0900
From:   Masami Hiramatsu <mhiramat@...nel.org>
To:     paulmck@...nel.org
Cc:     Sebastian Andrzej Siewior <bigeasy@...utronix.de>,
        Peter Zijlstra <peterz@...radead.org>,
        Thomas Gleixner <tglx@...utronix.de>,
        Uladzislau Rezki <urezki@...il.com>,
        LKML <linux-kernel@...r.kernel.org>, RCU <rcu@...r.kernel.org>,
        Michael Ellerman <mpe@...erman.id.au>,
        Andrew Morton <akpm@...ux-foundation.org>,
        Daniel Axtens <dja@...ens.net>,
        Frederic Weisbecker <frederic@...nel.org>,
        Neeraj Upadhyay <neeraju@...eaurora.org>,
        Joel Fernandes <joel@...lfernandes.org>,
        Michal Hocko <mhocko@...e.com>,
        "Theodore Y . Ts'o" <tytso@....edu>,
        Oleksiy Avramchenko <oleksiy.avramchenko@...ymobile.com>,
        mhiramat@...nel.org, rostedt@...dmis.org
Subject: Re: [PATCH 2/2] rcu-tasks: add RCU-tasks self tests

On Tue, 16 Feb 2021 09:30:03 -0800
"Paul E. McKenney" <paulmck@...nel.org> wrote:

> On Mon, Feb 15, 2021 at 12:28:26PM +0100, Sebastian Andrzej Siewior wrote:
> > On 2021-02-13 08:45:54 [-0800], Paul E. McKenney wrote:
> > > Glad you like it!  But let's see which (if any) of these patches solves
> > > the problem for Sebastian.
> > 
> > Looking at that, is there any reason for doing this that can not be
> > solved by moving the self-test a little later? Maybe once we reached at
> > least SYSTEM_SCHEDULING?
> 
> One problem is that ksoftirqd and the kprobes use are early_initcall(),
> so we cannot count on ksoftirqd being spawned when kprobes first uses
> synchronize_rcu_tasks().  Moving the selftest later won't fix this
> problem, but rather just paper it over.
> 
> > This happens now even before lockdep is up or the console is registered.
> > So if something bad happens, you end up with a blank terminal.
> 
> I was getting a splat, but I could easily believe that there are
> configurations where the hang is totally silent.  In other words, I do
> agree that this needs a proper fix.  All we need do is work out an
> agreeable value of "proper".  ;-)
> 
> > There is nothing else that early in the boot process that requires
> > working softirq. The only exception to this is wait_task_inactive()
> > which is used while starting a new thread (including the ksoftirqd)
> > which is why it was moved to schedule_hrtimeout().
> 
> Moving kprobes initialization to early_initcall() [1] means that there
> can be a call to synchronize_rcu_tasks() before the current spawning of
> ksoftirqd.  Because synchronize_rcu_tasks() needs timers to work, it needs
> softirq to work.  I know two straightforward ways to make that happen:
> 
> 1.	Spawn ksoftirqd earlier.
> 
> 2.	Suppress attempts to awaken ksoftirqd before it exists,
> 	forcing all ksoftirq execution on the back of interrupts.
> 
> Uladzislau and I each produced patches for #1, and I produced a patch
> for #2.
> 
> The only other option I know of is to push the call to init_kprobes()
> later in the boot sequence, perhaps to its original subsys_initcall(),
> or maybe only as late as core_initcall().  I added Masami and Steve on
> CC for their thoughts on this.
> 
> Is there some other proper fix that I am missing?

Oh, I missed that the synchronize_rcu_tasks() will be involved the kprobes
in early stage. Does the problem only exist in the synchronize_rcu_tasks()
instead of synchronize_rcu()? If so I can just stop optimizer in early stage
because I just want to enable kprobes in early stage, but not optprobes.

Does the following patch help?

>From e5fafcda3ff918cd52619f795a3f22fb95c72b11 Mon Sep 17 00:00:00 2001
From: Masami Hiramatsu <mhiramat@...nel.org>
Date: Wed, 17 Feb 2021 23:35:20 +0900
Subject: [PATCH] kprobes: Fix to delay the kprobes jump optimization

Since the kprobes jump optimization involves synchronize_rcu_tasks()
which depends on the ksoftirqd, that can not be enabled at the
early_initcall() boot stage. So this makes the kprobe optimization
disabled in the early_initcall() and enables it in subsys_initcall().

Note that non-optimized kprobes is still available after
early_initcall(). Only jump optimization is delayed.

Fixes: 36dadef23fcc ("kprobes: Init kprobes in early_initcall")
Signed-off-by: Masami Hiramatsu <mhiramat@...nel.org>
---
 kernel/kprobes.c | 31 +++++++++++++++++++++----------
 1 file changed, 21 insertions(+), 10 deletions(-)

diff --git a/kernel/kprobes.c b/kernel/kprobes.c
index d5a3eb74a657..779d8322e307 100644
--- a/kernel/kprobes.c
+++ b/kernel/kprobes.c
@@ -861,7 +861,6 @@ static void try_to_optimize_kprobe(struct kprobe *p)
 	cpus_read_unlock();
 }
 
-#ifdef CONFIG_SYSCTL
 static void optimize_all_kprobes(void)
 {
 	struct hlist_head *head;
@@ -887,6 +886,7 @@ static void optimize_all_kprobes(void)
 	mutex_unlock(&kprobe_mutex);
 }
 
+#ifdef CONFIG_SYSCTL
 static void unoptimize_all_kprobes(void)
 {
 	struct hlist_head *head;
@@ -2497,18 +2497,14 @@ static int __init init_kprobes(void)
 		}
 	}
 
-#if defined(CONFIG_OPTPROBES)
-#if defined(__ARCH_WANT_KPROBES_INSN_SLOT)
-	/* Init kprobe_optinsn_slots */
-	kprobe_optinsn_slots.insn_size = MAX_OPTINSN_SIZE;
-#endif
-	/* By default, kprobes can be optimized */
-	kprobes_allow_optimization = true;
-#endif
-
 	/* By default, kprobes are armed */
 	kprobes_all_disarmed = false;
 
+#if defined(CONFIG_OPTPROBES) && defined(__ARCH_WANT_KPROBES_INSN_SLOT)
+	/* Init kprobe_optinsn_slots for allocation */
+	kprobe_optinsn_slots.insn_size = MAX_OPTINSN_SIZE;
+#endif
+
 	err = arch_init_kprobes();
 	if (!err)
 		err = register_die_notifier(&kprobe_exceptions_nb);
@@ -2523,6 +2519,21 @@ static int __init init_kprobes(void)
 }
 early_initcall(init_kprobes);
 
+#if defined(CONFIG_OPTPROBES)
+static int __init init_optprobes(void)
+{
+	/*
+	 * Enable kprobe optimization - this kicks the optimizer which
+	 * depends on synchronize_rcu_tasks() and ksoftirqd, that is
+	 * not spawned in early initcall. So delay the optimization.
+	 */
+	optimize_all_kprobes();
+
+	return 0;
+}
+subsys_initcall(init_optprobes);
+#endif
+
 #ifdef CONFIG_DEBUG_FS
 static void report_probe(struct seq_file *pi, struct kprobe *p,
 		const char *sym, int offset, char *modname, struct kprobe *pp)
-- 
2.25.1


-- 
Masami Hiramatsu <mhiramat@...nel.org>

Powered by blists - more mailing lists