Date:   Mon, 22 Feb 2021 10:16:08 -0800
From:   "Paul E. McKenney" <paulmck@...nel.org>
To:     Uladzislau Rezki <urezki@...il.com>
Cc:     Sebastian Andrzej Siewior <bigeasy@...utronix.de>,
        Masami Hiramatsu <mhiramat@...nel.org>,
        Ingo Molnar <mingo@...nel.org>,
        Steven Rostedt <rostedt@...dmis.org>,
        Peter Zijlstra <peterz@...radead.org>,
        Thomas Gleixner <tglx@...utronix.de>,
        LKML <linux-kernel@...r.kernel.org>, RCU <rcu@...r.kernel.org>,
        Michael Ellerman <mpe@...erman.id.au>,
        Andrew Morton <akpm@...ux-foundation.org>,
        Daniel Axtens <dja@...ens.net>,
        Frederic Weisbecker <frederic@...nel.org>,
        Neeraj Upadhyay <neeraju@...eaurora.org>,
        Joel Fernandes <joel@...lfernandes.org>,
        Michal Hocko <mhocko@...e.com>,
        "Theodore Y . Ts'o" <tytso@....edu>,
        Oleksiy Avramchenko <oleksiy.avramchenko@...ymobile.com>
Subject: Re: [PATCH] kprobes: Fix to delay the kprobes jump optimization

On Mon, Feb 22, 2021 at 06:16:05PM +0100, Uladzislau Rezki wrote:
> On Mon, Feb 22, 2021 at 07:09:03AM -0800, Paul E. McKenney wrote:
> > On Mon, Feb 22, 2021 at 01:54:31PM +0100, Uladzislau Rezki wrote:
> > > On Mon, Feb 22, 2021 at 11:21:04AM +0100, Sebastian Andrzej Siewior wrote:
> > > > On 2021-02-19 10:33:36 [-0800], Paul E. McKenney wrote:
> > > > > For definiteness, here is the first part of the change, posted earlier.
> > > > > The commit log needs to be updated.  I will post the change that keeps
> > > > > the tick going as a reply to this email.
> > > > …
> > > > > diff --git a/kernel/softirq.c b/kernel/softirq.c
> > > > > index 9d71046..ba78e63 100644
> > > > > --- a/kernel/softirq.c
> > > > > +++ b/kernel/softirq.c
> > > > > @@ -209,7 +209,7 @@ static inline void invoke_softirq(void)
> > > > >  	if (ksoftirqd_running(local_softirq_pending()))
> > > > >  		return;
> > > > >  
> > > > > -	if (!force_irqthreads) {
> > > > > +	if (!force_irqthreads || !__this_cpu_read(ksoftirqd)) {
> > > > >  #ifdef CONFIG_HAVE_IRQ_EXIT_ON_IRQ_STACK
> > > > >  		/*
> > > > >  		 * We can safely execute softirq on the current stack if
> > > > > @@ -358,8 +358,8 @@ asmlinkage __visible void __softirq_entry __do_softirq(void)
> > > > >  
> > > > >  	pending = local_softirq_pending();
> > > > >  	if (pending) {
> > > > > -		if (time_before(jiffies, end) && !need_resched() &&
> > > > > -		    --max_restart)
> > > > > +		if (!__this_cpu_read(ksoftirqd) ||
> > > > > +		    (time_before(jiffies, end) && !need_resched() && --max_restart))
> > > > >  			goto restart;
> > > > 
> > > > This is hunk shouldn't be needed. The reason for it is probably that the
> > > > following wakeup_softirqd() would avoid further invoke_softirq()
> > > > performing the actual softirq work. It would leave early due to
> > > > ksoftirqd_running(). Unless I'm wrong, any raise_softirq() invocation
> > > > outside of an interrupt would do the same. 
> > 
> > And it does pass the rcutorture test without that hunk:
> > 
> > tools/testing/selftests/rcutorture/bin/kvm.sh --allcpus --duration 2 --configs "TREE03" --kconfig "CONFIG_DEBUG_LOCK_ALLOC=y CONFIG_PROVE_LOCKING=y" --bootargs "threadirqs=1" --trust-make
> > 
> Yep. I have tested that patch also. It works for me as well. So
> technically i do not see any issues from the first glance but of
> course it should be reviewed by the softirq people to hear their
> opinion.
> 
> IRQs are enabled, so it can be handled from an IRQ tail until
> ksoftirqd threads are spawned.

And if I add "CONFIG_NO_HZ_IDLE=y CONFIG_HZ_PERIODIC=n" it still works,
even if I revert my changes to rcu_needs_cpu().  Should I rely on this
working globally?  ;-)

							Thanx, Paul

> > > > I would like PeterZ / tglx to comment on this one. Basically I'm not
> > > > sure if it is okay to expect softirqs beeing served and waited on that
> > > > early in the boot.
> > 
> > It would be good to get other eyes on this.
> > 
> > I do agree that "don't wait on softirq handlers until after completion
> > of all early_initcall() handlers" is a nice simple rule, but debugging
> > violations of it is not so simple.  Adding warnings to ease debugging
> > of violations of this rule is quite a bit more complex than is either of
> > the methods of making the rule unnecessary, at least from what I can see
> > at this point.  The complexity of the warnings is exactly what Sebastian
> > pointed out earlier, that it is currently legal to raise_softirq() but
> > not to wait on the resulting handlers.  But even waiting is OK if that
> > waiting does not delay the boot sequence.  But if the boot kthread waits
> > on the kthread that does the waiting, it is once again not OK.
> > 
> > So am I missing something subtle here?
> >
> I agree here. Seems like we are on the same page in understanding :)
> 
> > > The ksoftirqd threads get spawned during early_initcall() phase. Why not
> > > just spawn them one step earlier what is totally safe? I mean before
> > > do_pre_smp_initcalls() that calls early callbacks.
> > > 
> > > +       spawn_ksoftirqd();
> > >         rcu_init_tasks_generic();
> > >         do_pre_smp_initcalls();
> > > 
> > > With such change the spawning will not be depended on linker/compiler
> > > i.e. when and in which order an early_initcall(spawn_ksoftirqd) callback
> > > is executed.
> > 
> > We both posted patches similar to this, so I am not opposed.  One caveat,
> > though, namely that this narrows the window quite a bit but does not
> > entirely close it.  But it does allow the early_initcall()s to wait on
> > softirq handlers.
> > 
> Yep, that was an intention. At least to provide such functionality for early
> callbacks. What happens before it(init/main.c) is pretty controllable.
> 
> --
> Vlad Rezki

Powered by blists - more mailing lists